2024-04-23T15:54:35.873 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-23T15:54:35.873 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} --archive /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669729 --owner scheduled_yuriw@teuthology --verbose -- /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669729/orig.config.yaml 2024-04-23T15:54:35.920 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T15:54:36.011 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669729 branch: wip-yuri3-testing-2024-04-05-0825 description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} email: yweinste@redhat.com first_in_suite: false job_id: '7669729' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri3-testing-2024-04-05-0825 ceph: conf: client.2: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd heartbeat use min delay socket: true osd map cache size: 1 osd max backfills: 6 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub during recovery: false osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - osd_map_cache_size - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum mon_bind_msgr2: false sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri3-testing-2024-04-05-0825 sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 3753 sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri3-testing-2024-04-05-0825 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb targets: smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKcLq1XqhET9DWVFdAMWm2N9zEMTbTkRbDjXk5DXARzJ3Oud6Xg4iR1GIBZ7N5Pp3eEJurCAdxrPgVpFrYEKCXk= smithi148.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAJOtPzIJn6KzU2crEJ1/mEe1H325BE8DH6ASuMkeG+o9iUxizLCWs66qVk1YW6Hf+yWvUDSo7bX+vvx+94M3hE= smithi188.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMHm2V0pB3G6ulATzdlCK83Lwo819iCQpzyiGN7s52sIdj7c32tT5nrv7NCAt9rgLnsX59tJAWsNyBRtS0pd008= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: nautilus exclude_packages: - cephadm - ceph-mgr-cephadm - ceph-immutable-object-cache - python3-rados - python3-rgw - python3-rbd - python3-cephfs - ceph-volume extra_packages: - python-rados - python-rgw - python-rbd - python-cephfs - cephadm: conf: mon: auth allow insecure global id reclaim: true - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: chance_pgnum_grow: 0.25 chance_pgpnum_fix: 0.25 chance_test_map_discontinuity: 2 timeout: 1800 - full_sequential: - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-23_14:14:08 tube: smithi user: yuriw verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-23T15:54:36.012 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa; will attempt to use it 2024-04-23T15:54:36.012 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa/tasks 2024-04-23T15:54:36.013 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-23T15:54:36.015 INFO:teuthology.task.internal:Saving configuration 2024-04-23T15:54:36.038 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-23T15:54:36.045 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T15:54:36.071 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669729', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 15:49:06.902165', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKcLq1XqhET9DWVFdAMWm2N9zEMTbTkRbDjXk5DXARzJ3Oud6Xg4iR1GIBZ7N5Pp3eEJurCAdxrPgVpFrYEKCXk='} 2024-04-23T15:54:36.093 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669729', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 15:49:06.905872', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAJOtPzIJn6KzU2crEJ1/mEe1H325BE8DH6ASuMkeG+o9iUxizLCWs66qVk1YW6Hf+yWvUDSo7bX+vvx+94M3hE='} 2024-04-23T15:54:36.114 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669729', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 15:49:06.904114', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMHm2V0pB3G6ulATzdlCK83Lwo819iCQpzyiGN7s52sIdj7c32tT5nrv7NCAt9rgLnsX59tJAWsNyBRtS0pd008='} 2024-04-23T15:54:36.114 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-23T15:54:36.121 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-23T15:54:36.121 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-04-23T15:54:36.121 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-04-23T15:54:36.121 INFO:teuthology.run_tasks:Running task console_log... 2024-04-23T15:54:36.223 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa96b845f70>, signals=[15]) 2024-04-23T15:54:36.224 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-23T15:54:36.229 INFO:teuthology.task.internal:Opening connections... 2024-04-23T15:54:36.229 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-04-23T15:54:36.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T15:54:36.313 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi148.front.sepia.ceph.com 2024-04-23T15:54:36.314 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T15:54:36.393 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi188.front.sepia.ceph.com 2024-04-23T15:54:36.394 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T15:54:36.470 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-23T15:54:36.477 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-04-23T15:54:36.499 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-04-23T15:54:36.499 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-04-23T15:54:36.559 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2024-04-23T15:54:36.559 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="8" 2024-04-23T15:54:36.559 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2024-04-23T15:54:36.559 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2024-04-23T15:54:36.559 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="8" 2024-04-23T15:54:36.559 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el8" 2024-04-23T15:54:36.559 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-23T15:54:36.559 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2024-04-23T15:54:36.560 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-23T15:54:36.560 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2024-04-23T15:54:36.560 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T15:54:36.560 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-23T15:54:36.560 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T15:54:36.560 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-04-23T15:54:36.592 DEBUG:teuthology.orchestra.run.smithi148:> uname -m 2024-04-23T15:54:36.616 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2024-04-23T15:54:36.616 DEBUG:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2024-04-23T15:54:36.677 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Stream" 2024-04-23T15:54:36.678 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="8" 2024-04-23T15:54:36.678 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos" 2024-04-23T15:54:36.678 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora" 2024-04-23T15:54:36.678 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="8" 2024-04-23T15:54:36.678 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el8" 2024-04-23T15:54:36.678 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-23T15:54:36.678 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2024-04-23T15:54:36.678 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-23T15:54:36.678 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://centos.org/" 2024-04-23T15:54:36.678 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T15:54:36.678 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-23T15:54:36.678 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T15:54:36.679 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2024-04-23T15:54:36.699 DEBUG:teuthology.orchestra.run.smithi188:> uname -m 2024-04-23T15:54:36.722 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2024-04-23T15:54:36.722 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2024-04-23T15:54:36.783 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="CentOS Stream" 2024-04-23T15:54:36.783 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="8" 2024-04-23T15:54:36.783 INFO:teuthology.orchestra.run.smithi188.stdout:ID="centos" 2024-04-23T15:54:36.783 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="rhel fedora" 2024-04-23T15:54:36.783 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="8" 2024-04-23T15:54:36.784 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el8" 2024-04-23T15:54:36.784 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-23T15:54:36.784 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2024-04-23T15:54:36.784 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-23T15:54:36.784 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://centos.org/" 2024-04-23T15:54:36.784 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T15:54:36.784 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-23T15:54:36.784 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T15:54:36.784 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2024-04-23T15:54:36.816 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-23T15:54:36.825 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-23T15:54:36.833 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-23T15:54:36.833 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-04-23T15:54:36.838 DEBUG:teuthology.orchestra.run.smithi148:> test '!' -e /home/ubuntu/cephtest 2024-04-23T15:54:36.841 DEBUG:teuthology.orchestra.run.smithi188:> test '!' -e /home/ubuntu/cephtest 2024-04-23T15:54:36.861 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-23T15:54:36.867 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-23T15:54:36.868 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-04-23T15:54:36.898 DEBUG:teuthology.orchestra.run.smithi148:> test -z $(ls -A /var/lib/ceph) 2024-04-23T15:54:36.901 DEBUG:teuthology.orchestra.run.smithi188:> test -z $(ls -A /var/lib/ceph) 2024-04-23T15:54:36.935 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-23T15:54:37.015 INFO:teuthology.run_tasks:Running task kernel... 2024-04-23T15:54:37.036 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-23T15:54:37.036 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-23T15:54:37.036 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T15:54:37.037 DEBUG:teuthology.orchestra.run.smithi148:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T15:54:37.037 DEBUG:teuthology.orchestra.run.smithi188:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T15:54:37.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T15:54:37.058 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2024-04-23T15:54:37.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T15:54:37.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T15:54:37.060 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-04-23T15:54:37.060 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2024-04-23T15:54:37.117 INFO:teuthology.orchestra.run.smithi188.stdout:4.18.0-489.el8.x86_64 2024-04-23T15:54:37.119 INFO:teuthology.task.kernel:Running kernel on smithi188: 4.18.0-489.el8.x86_64 2024-04-23T15:54:37.119 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum install -y kernel 2024-04-23T15:54:37.121 INFO:teuthology.orchestra.run.smithi148.stdout:4.18.0-489.el8.x86_64 2024-04-23T15:54:37.121 INFO:teuthology.task.kernel:Running kernel on smithi148: 4.18.0-489.el8.x86_64 2024-04-23T15:54:37.121 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum install -y kernel 2024-04-23T15:54:37.122 INFO:teuthology.orchestra.run.smithi103.stdout:4.18.0-489.el8.x86_64 2024-04-23T15:54:37.122 INFO:teuthology.task.kernel:Running kernel on smithi103: 4.18.0-489.el8.x86_64 2024-04-23T15:54:37.122 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum install -y kernel 2024-04-23T15:54:40.896 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - AppStream 23 MB/s | 28 MB 00:01 2024-04-23T15:54:41.610 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 8 - AppStream 15 MB/s | 28 MB 00:01 2024-04-23T15:54:42.213 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - AppStream 11 MB/s | 28 MB 00:02 2024-04-23T15:54:47.096 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - BaseOS 8.6 MB/s | 10 MB 00:01 2024-04-23T15:54:47.827 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 8 - BaseOS 8.2 MB/s | 10 MB 00:01 2024-04-23T15:54:49.105 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras 44 kB/s | 18 kB 00:00 2024-04-23T15:54:49.505 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras common packages 26 kB/s | 7.7 kB 00:00 2024-04-23T15:54:49.773 INFO:teuthology.orchestra.run.smithi188.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-04-23T15:54:49.827 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 8 - Extras 45 kB/s | 18 kB 00:00 2024-04-23T15:54:50.071 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - BaseOS 3.6 MB/s | 10 MB 00:02 2024-04-23T15:54:50.345 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 7.7 kB 00:00 2024-04-23T15:54:50.571 INFO:teuthology.orchestra.run.smithi103.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-04-23T15:54:51.368 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-04-23T15:54:52.138 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - Extras 41 kB/s | 18 kB 00:00 2024-04-23T15:54:52.380 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 9.5 MB/s | 16 MB 00:01 2024-04-23T15:54:52.797 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - Extras common packages 14 kB/s | 7.7 kB 00:00 2024-04-23T15:54:53.090 INFO:teuthology.orchestra.run.smithi148.stdout:Copr repo for python3-asyncssh owned by ceph 17 kB/s | 3.5 kB 00:00 2024-04-23T15:54:55.148 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 585 kB/s | 24 kB 00:00 2024-04-23T15:54:56.214 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 428 kB/s | 24 kB 00:00 2024-04-23T15:54:58.129 INFO:teuthology.orchestra.run.smithi188.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-23T15:54:58.129 INFO:teuthology.orchestra.run.smithi188.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-23T15:54:58.275 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T15:54:58.276 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T15:54:58.276 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-23T15:54:58.276 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T15:54:58.276 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-23T15:54:58.276 INFO:teuthology.orchestra.run.smithi188.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-23T15:54:58.276 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-23T15:54:58.277 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-23T15:54:58.277 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-23T15:54:58.277 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T15:54:58.277 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-23T15:54:58.277 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T15:54:58.277 INFO:teuthology.orchestra.run.smithi188.stdout:Install 3 Packages 2024-04-23T15:54:58.277 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T15:54:58.278 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 90 M 2024-04-23T15:54:58.278 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 96 M 2024-04-23T15:54:58.278 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-23T15:54:59.211 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-23T15:54:59.211 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-23T15:54:59.344 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T15:54:59.345 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T15:54:59.345 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-04-23T15:54:59.345 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T15:54:59.345 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-23T15:54:59.345 INFO:teuthology.orchestra.run.smithi103.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-23T15:54:59.345 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-23T15:54:59.345 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-23T15:54:59.345 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-23T15:54:59.345 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T15:54:59.345 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-23T15:54:59.345 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T15:54:59.346 INFO:teuthology.orchestra.run.smithi103.stdout:Install 3 Packages 2024-04-23T15:54:59.346 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T15:54:59.346 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 90 M 2024-04-23T15:54:59.346 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 96 M 2024-04-23T15:54:59.347 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-23T15:55:02.357 INFO:teuthology.orchestra.run.smithi103.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 3.5 MB/s | 10 MB 00:02 2024-04-23T15:55:02.849 INFO:teuthology.orchestra.run.smithi188.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 2.4 MB/s | 10 MB 00:04 2024-04-23T15:55:03.499 INFO:teuthology.orchestra.run.smithi188.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 7.3 MB/s | 36 MB 00:05 2024-04-23T15:55:03.907 INFO:teuthology.orchestra.run.smithi188.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 8.1 MB/s | 44 MB 00:05 2024-04-23T15:55:03.907 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-23T15:55:03.907 INFO:teuthology.orchestra.run.smithi188.stdout:Total 16 MB/s | 90 MB 00:05 2024-04-23T15:55:04.596 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-23T15:55:04.665 INFO:teuthology.orchestra.run.smithi103.stdout:(2/3): kernel-core-4.18.0-552.el8.x86_64.rpm 8.3 MB/s | 44 MB 00:05 2024-04-23T15:55:05.042 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-23T15:55:05.042 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-23T15:55:06.298 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-23T15:55:06.330 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-23T15:55:09.090 INFO:teuthology.orchestra.run.smithi103.stdout:(3/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 3.8 MB/s | 36 MB 00:09 2024-04-23T15:55:09.090 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-23T15:55:09.091 INFO:teuthology.orchestra.run.smithi103.stdout:Total 9.3 MB/s | 90 MB 00:09 2024-04-23T15:55:09.174 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-23T15:55:09.698 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-23T15:55:09.774 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-23T15:55:09.974 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 993 kB/s | 16 MB 00:16 2024-04-23T15:55:10.245 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-23T15:55:10.246 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-23T15:55:11.485 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-23T15:55:11.511 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-23T15:55:13.238 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-23T15:55:13.631 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-23T15:55:13.714 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 409 kB/s | 24 kB 00:00 2024-04-23T15:55:14.289 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-23T15:55:14.820 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-23T15:55:16.769 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-23T15:55:16.769 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-23T15:55:16.884 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T15:55:16.884 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T15:55:16.885 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2024-04-23T15:55:16.885 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T15:55:16.885 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-04-23T15:55:16.885 INFO:teuthology.orchestra.run.smithi148.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-23T15:55:16.885 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-04-23T15:55:16.885 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-23T15:55:16.885 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-23T15:55:16.885 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T15:55:16.885 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-04-23T15:55:16.885 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T15:55:16.885 INFO:teuthology.orchestra.run.smithi148.stdout:Install 3 Packages 2024-04-23T15:55:16.886 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T15:55:16.886 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 90 M 2024-04-23T15:55:16.886 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 96 M 2024-04-23T15:55:16.886 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-04-23T15:55:17.068 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-23T15:55:17.336 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-23T15:55:17.940 INFO:teuthology.orchestra.run.smithi148.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 10 MB/s | 10 MB 00:01 2024-04-23T15:55:18.419 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-23T15:55:18.819 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-23T15:55:19.265 INFO:teuthology.orchestra.run.smithi148.stdout:(2/3): kernel-core-4.18.0-552.el8.x86_64.rpm 19 MB/s | 44 MB 00:02 2024-04-23T15:55:20.099 INFO:teuthology.orchestra.run.smithi148.stdout:(3/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 12 MB/s | 36 MB 00:03 2024-04-23T15:55:20.099 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-04-23T15:55:20.099 INFO:teuthology.orchestra.run.smithi148.stdout:Total 28 MB/s | 90 MB 00:03 2024-04-23T15:55:20.788 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-04-23T15:55:21.194 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-04-23T15:55:21.194 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-04-23T15:55:22.258 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-23T15:55:22.434 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-04-23T15:55:22.458 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-04-23T15:55:22.660 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-23T15:55:25.214 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-04-23T15:55:25.721 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-23T15:55:29.282 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-23T15:55:29.637 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-23T15:55:33.080 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-23T15:55:33.343 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-23T15:55:39.242 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-23T15:55:39.266 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-23T15:55:39.740 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-23T15:55:39.740 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-23T15:55:39.741 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-23T15:55:40.491 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-23T15:55:40.491 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T15:55:40.491 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-23T15:55:40.492 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-23T15:55:40.492 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-23T15:55:40.492 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T15:55:40.492 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T15:55:40.629 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q kernel | sort -rV | head -n 1 2024-04-23T15:55:40.904 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-23T15:55:40.905 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-23T15:55:40.905 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-23T15:55:40.905 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-23T15:55:40.905 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-23T15:55:40.905 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-04-23T15:55:40.905 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi188.front.sepia.ceph.com, path=None, version=distro) 2024-04-23T15:55:40.906 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q kernel | sort -rV | head -n 1 2024-04-23T15:55:41.117 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-23T15:55:41.117 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-23T15:55:41.117 DEBUG:teuthology.orchestra.run.smithi188:> sudo rpm -qi grub2-tools 2024-04-23T15:55:41.152 INFO:teuthology.orchestra.run.smithi188.stdout:Name : grub2-tools 2024-04-23T15:55:41.152 INFO:teuthology.orchestra.run.smithi188.stdout:Epoch : 1 2024-04-23T15:55:41.152 INFO:teuthology.orchestra.run.smithi188.stdout:Version : 2.02 2024-04-23T15:55:41.152 INFO:teuthology.orchestra.run.smithi188.stdout:Release : 129.el8 2024-04-23T15:55:41.152 INFO:teuthology.orchestra.run.smithi188.stdout:Architecture: x86_64 2024-04-23T15:55:41.152 INFO:teuthology.orchestra.run.smithi188.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-23T15:55:41.153 INFO:teuthology.orchestra.run.smithi188.stdout:Group : System Environment/Base 2024-04-23T15:55:41.153 INFO:teuthology.orchestra.run.smithi188.stdout:Size : 9265586 2024-04-23T15:55:41.153 INFO:teuthology.orchestra.run.smithi188.stdout:License : GPLv3+ 2024-04-23T15:55:41.153 INFO:teuthology.orchestra.run.smithi188.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-23T15:55:41.153 INFO:teuthology.orchestra.run.smithi188.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-23T15:55:41.153 INFO:teuthology.orchestra.run.smithi188.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-23T15:55:41.153 INFO:teuthology.orchestra.run.smithi188.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-23T15:55:41.153 INFO:teuthology.orchestra.run.smithi188.stdout:Relocations : (not relocatable) 2024-04-23T15:55:41.153 INFO:teuthology.orchestra.run.smithi188.stdout:Packager : CentOS BuildSystem 2024-04-23T15:55:41.153 INFO:teuthology.orchestra.run.smithi188.stdout:Vendor : CentOS 2024-04-23T15:55:41.153 INFO:teuthology.orchestra.run.smithi188.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-23T15:55:41.153 INFO:teuthology.orchestra.run.smithi188.stdout:Summary : Support tools for GRUB. 2024-04-23T15:55:41.153 INFO:teuthology.orchestra.run.smithi188.stdout:Description : 2024-04-23T15:55:41.154 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T15:55:41.154 INFO:teuthology.orchestra.run.smithi188.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-23T15:55:41.154 INFO:teuthology.orchestra.run.smithi188.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-23T15:55:41.154 INFO:teuthology.orchestra.run.smithi188.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-23T15:55:41.154 INFO:teuthology.orchestra.run.smithi188.stdout:hardware devices. 2024-04-23T15:55:41.154 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T15:55:41.154 INFO:teuthology.orchestra.run.smithi188.stdout:This subpackage provides tools for support of all platforms. 2024-04-23T15:55:41.155 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-23T15:55:41.155 INFO:teuthology.task.kernel:Updating grub on smithi188 to boot 4.18.0-552.el8.x86_64 2024-04-23T15:55:41.155 DEBUG:teuthology.orchestra.run.smithi188:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-23T15:55:42.497 INFO:teuthology.orchestra.run.smithi188.stderr:Generating grub configuration file ... 2024-04-23T15:55:44.500 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-23T15:55:44.524 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-23T15:55:44.870 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-23T15:55:44.870 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-23T15:55:44.870 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-23T15:55:45.280 INFO:teuthology.orchestra.run.smithi188.stderr:done 2024-04-23T15:55:45.282 DEBUG:teuthology.orchestra.run.smithi188:> mktemp 2024-04-23T15:55:45.314 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.0kpCgAkmoO 2024-04-23T15:55:45.314 DEBUG:teuthology.orchestra.run.smithi188:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0kpCgAkmoO 2024-04-23T15:55:45.387 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.0kpCgAkmoO 2024-04-23T15:55:45.392 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-23T15:55:45.392 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T15:55:45.392 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-23T15:55:45.392 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-23T15:55:45.393 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-23T15:55:45.393 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T15:55:45.393 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T15:55:45.523 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.0kpCgAkmoO is 6KB 2024-04-23T15:55:45.545 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-04-23T15:55:45.574 DEBUG:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.0kpCgAkmoO 2024-04-23T15:55:45.592 DEBUG:teuthology.orchestra.run.smithi188:> sudo /bin/ls /boot/loader/entries || true 2024-04-23T15:55:45.660 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-23T15:55:45.661 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-23T15:55:45.661 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-23T15:55:45.661 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-23T15:55:45.662 DEBUG:teuthology.orchestra.run.smithi188:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-23T15:55:45.765 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-23T15:55:45.765 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-23T15:55:45.765 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-23T15:55:45.766 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-23T15:55:45.766 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-23T15:55:45.766 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-23T15:55:45.766 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi103.front.sepia.ceph.com, path=None, version=distro) 2024-04-23T15:55:45.766 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-04-23T15:55:45.802 DEBUG:teuthology.orchestra.run.smithi188:> sudo shutdown -r now 2024-04-23T15:55:45.976 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-23T15:55:45.977 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-23T15:55:45.977 DEBUG:teuthology.orchestra.run.smithi103:> sudo rpm -qi grub2-tools 2024-04-23T15:55:46.011 INFO:teuthology.orchestra.run.smithi103.stdout:Name : grub2-tools 2024-04-23T15:55:46.011 INFO:teuthology.orchestra.run.smithi103.stdout:Epoch : 1 2024-04-23T15:55:46.011 INFO:teuthology.orchestra.run.smithi103.stdout:Version : 2.02 2024-04-23T15:55:46.011 INFO:teuthology.orchestra.run.smithi103.stdout:Release : 129.el8 2024-04-23T15:55:46.011 INFO:teuthology.orchestra.run.smithi103.stdout:Architecture: x86_64 2024-04-23T15:55:46.012 INFO:teuthology.orchestra.run.smithi103.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-23T15:55:46.012 INFO:teuthology.orchestra.run.smithi103.stdout:Group : System Environment/Base 2024-04-23T15:55:46.012 INFO:teuthology.orchestra.run.smithi103.stdout:Size : 9265586 2024-04-23T15:55:46.012 INFO:teuthology.orchestra.run.smithi103.stdout:License : GPLv3+ 2024-04-23T15:55:46.012 INFO:teuthology.orchestra.run.smithi103.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-23T15:55:46.012 INFO:teuthology.orchestra.run.smithi103.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-23T15:55:46.012 INFO:teuthology.orchestra.run.smithi103.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-23T15:55:46.012 INFO:teuthology.orchestra.run.smithi103.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-23T15:55:46.012 INFO:teuthology.orchestra.run.smithi103.stdout:Relocations : (not relocatable) 2024-04-23T15:55:46.012 INFO:teuthology.orchestra.run.smithi103.stdout:Packager : CentOS BuildSystem 2024-04-23T15:55:46.012 INFO:teuthology.orchestra.run.smithi103.stdout:Vendor : CentOS 2024-04-23T15:55:46.012 INFO:teuthology.orchestra.run.smithi103.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-23T15:55:46.012 INFO:teuthology.orchestra.run.smithi103.stdout:Summary : Support tools for GRUB. 2024-04-23T15:55:46.012 INFO:teuthology.orchestra.run.smithi103.stdout:Description : 2024-04-23T15:55:46.013 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T15:55:46.013 INFO:teuthology.orchestra.run.smithi103.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-23T15:55:46.013 INFO:teuthology.orchestra.run.smithi103.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-23T15:55:46.013 INFO:teuthology.orchestra.run.smithi103.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-23T15:55:46.013 INFO:teuthology.orchestra.run.smithi103.stdout:hardware devices. 2024-04-23T15:55:46.013 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T15:55:46.013 INFO:teuthology.orchestra.run.smithi103.stdout:This subpackage provides tools for support of all platforms. 2024-04-23T15:55:46.014 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-23T15:55:46.014 INFO:teuthology.task.kernel:Updating grub on smithi103 to boot 4.18.0-552.el8.x86_64 2024-04-23T15:55:46.014 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-23T15:55:47.626 INFO:teuthology.orchestra.run.smithi103.stderr:Generating grub configuration file ... 2024-04-23T15:55:49.822 INFO:teuthology.orchestra.run.smithi103.stderr:done 2024-04-23T15:55:49.828 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-04-23T15:55:49.867 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.oEwnHn26hL 2024-04-23T15:55:49.867 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.oEwnHn26hL 2024-04-23T15:55:49.942 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.oEwnHn26hL 2024-04-23T15:55:50.086 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.oEwnHn26hL is 6KB 2024-04-23T15:55:50.134 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.oEwnHn26hL 2024-04-23T15:55:50.151 DEBUG:teuthology.orchestra.run.smithi103:> sudo /bin/ls /boot/loader/entries || true 2024-04-23T15:55:50.228 INFO:teuthology.orchestra.run.smithi103.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-23T15:55:50.228 INFO:teuthology.orchestra.run.smithi103.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-23T15:55:50.228 INFO:teuthology.orchestra.run.smithi103.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-23T15:55:50.228 INFO:teuthology.orchestra.run.smithi103.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-23T15:55:50.229 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-23T15:55:50.373 DEBUG:teuthology.orchestra.run.smithi103:> sudo shutdown -r now 2024-04-23T15:55:55.443 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-23T15:55:55.462 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-23T15:55:55.799 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-23T15:55:55.799 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-23T15:55:55.799 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-23T15:55:56.311 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-23T15:55:56.311 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T15:55:56.311 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-04-23T15:55:56.311 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-23T15:55:56.311 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-23T15:55:56.311 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T15:55:56.312 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T15:55:56.465 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2024-04-23T15:55:56.686 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-23T15:55:56.686 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-23T15:55:56.686 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-23T15:55:56.686 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-23T15:55:56.686 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-23T15:55:56.686 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-23T15:55:56.687 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi148.front.sepia.ceph.com, path=None, version=distro) 2024-04-23T15:55:56.687 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2024-04-23T15:55:56.898 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-23T15:55:56.898 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-23T15:55:56.898 DEBUG:teuthology.orchestra.run.smithi148:> sudo rpm -qi grub2-tools 2024-04-23T15:55:56.932 INFO:teuthology.orchestra.run.smithi148.stdout:Name : grub2-tools 2024-04-23T15:55:56.932 INFO:teuthology.orchestra.run.smithi148.stdout:Epoch : 1 2024-04-23T15:55:56.932 INFO:teuthology.orchestra.run.smithi148.stdout:Version : 2.02 2024-04-23T15:55:56.932 INFO:teuthology.orchestra.run.smithi148.stdout:Release : 129.el8 2024-04-23T15:55:56.932 INFO:teuthology.orchestra.run.smithi148.stdout:Architecture: x86_64 2024-04-23T15:55:56.932 INFO:teuthology.orchestra.run.smithi148.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-23T15:55:56.932 INFO:teuthology.orchestra.run.smithi148.stdout:Group : System Environment/Base 2024-04-23T15:55:56.932 INFO:teuthology.orchestra.run.smithi148.stdout:Size : 9265586 2024-04-23T15:55:56.932 INFO:teuthology.orchestra.run.smithi148.stdout:License : GPLv3+ 2024-04-23T15:55:56.932 INFO:teuthology.orchestra.run.smithi148.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-23T15:55:56.932 INFO:teuthology.orchestra.run.smithi148.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-23T15:55:56.933 INFO:teuthology.orchestra.run.smithi148.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-23T15:55:56.933 INFO:teuthology.orchestra.run.smithi148.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-23T15:55:56.933 INFO:teuthology.orchestra.run.smithi148.stdout:Relocations : (not relocatable) 2024-04-23T15:55:56.933 INFO:teuthology.orchestra.run.smithi148.stdout:Packager : CentOS BuildSystem 2024-04-23T15:55:56.933 INFO:teuthology.orchestra.run.smithi148.stdout:Vendor : CentOS 2024-04-23T15:55:56.933 INFO:teuthology.orchestra.run.smithi148.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-23T15:55:56.933 INFO:teuthology.orchestra.run.smithi148.stdout:Summary : Support tools for GRUB. 2024-04-23T15:55:56.933 INFO:teuthology.orchestra.run.smithi148.stdout:Description : 2024-04-23T15:55:56.933 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T15:55:56.933 INFO:teuthology.orchestra.run.smithi148.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-23T15:55:56.933 INFO:teuthology.orchestra.run.smithi148.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-23T15:55:56.933 INFO:teuthology.orchestra.run.smithi148.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-23T15:55:56.933 INFO:teuthology.orchestra.run.smithi148.stdout:hardware devices. 2024-04-23T15:55:56.933 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T15:55:56.933 INFO:teuthology.orchestra.run.smithi148.stdout:This subpackage provides tools for support of all platforms. 2024-04-23T15:55:56.934 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-23T15:55:56.934 INFO:teuthology.task.kernel:Updating grub on smithi148 to boot 4.18.0-552.el8.x86_64 2024-04-23T15:55:56.935 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-23T15:55:58.468 INFO:teuthology.orchestra.run.smithi148.stderr:Generating grub configuration file ... 2024-04-23T15:56:00.322 INFO:teuthology.orchestra.run.smithi148.stderr:done 2024-04-23T15:56:00.323 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2024-04-23T15:56:00.352 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.BbJ0VAj0Cu 2024-04-23T15:56:00.352 DEBUG:teuthology.orchestra.run.smithi148:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.BbJ0VAj0Cu 2024-04-23T15:56:00.435 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.BbJ0VAj0Cu 2024-04-23T15:56:00.580 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.BbJ0VAj0Cu is 6KB 2024-04-23T15:56:00.629 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.BbJ0VAj0Cu 2024-04-23T15:56:00.647 DEBUG:teuthology.orchestra.run.smithi148:> sudo /bin/ls /boot/loader/entries || true 2024-04-23T15:56:00.732 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-23T15:56:00.732 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-23T15:56:00.733 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-23T15:56:00.733 INFO:teuthology.orchestra.run.smithi148.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-23T15:56:00.734 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-23T15:56:00.842 DEBUG:teuthology.orchestra.run.smithi148:> sudo shutdown -r now 2024-04-23T15:56:15.819 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-04-23T15:56:15.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-23T15:56:15.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T15:56:20.378 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-23T15:56:20.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-23T15:56:20.379 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T15:56:30.854 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-23T15:56:30.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-04-23T15:56:30.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T15:56:54.947 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-04-23T15:57:03.955 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-23T15:57:03.956 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T15:57:05.438 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2024-04-23T15:57:14.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-04-23T15:57:14.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T15:57:15.822 DEBUG:teuthology.orchestra.remote:timed out 2024-04-23T15:57:24.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-23T15:57:24.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T15:57:25.231 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-04-23T15:57:25.503 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-23T15:57:25.503 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-552.el8.x86_64"... 2024-04-23T15:57:25.504 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2024-04-23T15:57:25.520 INFO:teuthology.orchestra.run.smithi188.stdout:4.18.0-552.el8.x86_64 2024-04-23T15:57:25.521 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-23T15:57:25.521 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-23T15:57:25.521 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-23T15:57:26.521 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-23T15:57:26.522 DEBUG:teuthology.orchestra.run.smithi188:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-23T15:57:26.579 INFO:teuthology.orchestra.run.smithi188.stdout:ttyS1 2024-04-23T15:57:26.595 DEBUG:teuthology.parallel:result is None 2024-04-23T15:57:46.358 DEBUG:teuthology.orchestra.run.smithi148:> true 2024-04-23T15:57:46.654 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-04-23T15:57:46.654 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-552.el8.x86_64"... 2024-04-23T15:57:46.654 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2024-04-23T15:57:46.673 INFO:teuthology.orchestra.run.smithi148.stdout:4.18.0-552.el8.x86_64 2024-04-23T15:57:46.673 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-23T15:57:46.673 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-23T15:57:46.673 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-23T15:57:47.674 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-23T15:57:47.674 DEBUG:teuthology.orchestra.run.smithi148:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-23T15:57:47.770 INFO:teuthology.orchestra.run.smithi148.stdout:ttyS1 2024-04-23T15:57:47.781 DEBUG:teuthology.parallel:result is None 2024-04-23T15:58:03.959 DEBUG:teuthology.orchestra.remote:timed out 2024-04-23T15:58:15.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-23T15:58:15.961 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T15:58:16.290 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-04-23T15:58:16.557 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-23T15:58:16.557 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-552.el8.x86_64"... 2024-04-23T15:58:16.558 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-04-23T15:58:16.617 INFO:teuthology.orchestra.run.smithi103.stdout:4.18.0-552.el8.x86_64 2024-04-23T15:58:16.618 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-23T15:58:16.618 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-23T15:58:16.618 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-23T15:58:17.619 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-23T15:58:17.619 DEBUG:teuthology.orchestra.run.smithi103:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-23T15:58:17.695 INFO:teuthology.orchestra.run.smithi103.stdout:ttyS1 2024-04-23T15:58:17.710 DEBUG:teuthology.parallel:result is None 2024-04-23T15:58:17.710 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-23T15:58:17.720 INFO:teuthology.task.internal:Creating test directory... 2024-04-23T15:58:17.720 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T15:58:17.724 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T15:58:17.728 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T15:58:17.753 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-23T15:58:17.762 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-23T15:58:17.768 INFO:teuthology.task.internal:Creating archive directory... 2024-04-23T15:58:17.768 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T15:58:17.787 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T15:58:17.792 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T15:58:17.825 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-23T15:58:17.832 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-23T15:58:17.832 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T15:58:17.856 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T15:58:17.859 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T15:58:17.888 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T15:58:17.899 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T15:58:17.899 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T15:58:17.905 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T15:58:17.910 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T15:58:17.916 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T15:58:17.918 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-23T15:58:17.937 INFO:teuthology.task.internal:Configuring sudo... 2024-04-23T15:58:17.937 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T15:58:17.943 DEBUG:teuthology.orchestra.run.smithi148:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T15:58:17.955 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T15:58:17.986 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-23T15:58:17.996 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-23T15:58:17.996 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T15:58:18.014 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T15:58:18.025 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T15:58:18.046 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T15:58:18.117 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T15:58:18.169 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T15:58:18.254 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T15:58:18.297 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-23T15:58:18.297 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T15:58:18.363 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T15:58:18.409 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T15:58:18.464 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T15:58:18.547 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T15:58:18.594 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-04-23T15:58:18.594 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T15:58:18.661 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T15:58:18.709 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T15:58:18.769 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T15:58:18.856 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T15:58:18.901 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T15:58:18.902 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T15:58:18.970 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-04-23T15:58:18.977 DEBUG:teuthology.orchestra.run.smithi148:> sudo service rsyslog restart 2024-04-23T15:58:18.982 DEBUG:teuthology.orchestra.run.smithi188:> sudo service rsyslog restart 2024-04-23T15:58:19.024 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T15:58:19.042 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T15:58:19.087 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T15:58:19.484 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-23T15:58:19.492 INFO:teuthology.task.internal:Starting timer... 2024-04-23T15:58:19.492 INFO:teuthology.run_tasks:Running task pcp... 2024-04-23T15:58:19.512 INFO:teuthology.run_tasks:Running task selinux... 2024-04-23T15:58:19.522 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-23T15:58:19.524 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2024-04-23T15:58:19.601 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2024-04-23T15:58:19.603 DEBUG:teuthology.orchestra.run.smithi148:> sudo service auditd rotate 2024-04-23T15:58:19.674 INFO:teuthology.orchestra.run.smithi148.stdout:Rotating logs: 2024-04-23T15:58:19.676 DEBUG:teuthology.orchestra.run.smithi188:> sudo service auditd rotate 2024-04-23T15:58:19.753 INFO:teuthology.orchestra.run.smithi188.stdout:Rotating logs: 2024-04-23T15:58:19.755 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-23T15:58:19.755 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2024-04-23T15:58:19.781 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2024-04-23T15:58:19.782 DEBUG:teuthology.orchestra.run.smithi148:> /usr/sbin/getenforce 2024-04-23T15:58:19.805 INFO:teuthology.orchestra.run.smithi148.stdout:Permissive 2024-04-23T15:58:19.805 DEBUG:teuthology.orchestra.run.smithi188:> /usr/sbin/getenforce 2024-04-23T15:58:19.834 INFO:teuthology.orchestra.run.smithi188.stdout:Permissive 2024-04-23T15:58:19.834 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi103.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi148.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi188.front.sepia.ceph.com': 'permissive'} 2024-04-23T15:58:19.835 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-23T15:58:19.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T15:58:19.862 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-23T15:58:19.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T15:58:19.889 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-23T15:58:19.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T15:58:19.917 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-23T15:58:19.917 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2024-04-23T15:58:19.944 DEBUG:teuthology.orchestra.run.smithi148:> sudo /usr/sbin/setenforce permissive 2024-04-23T15:58:19.970 DEBUG:teuthology.orchestra.run.smithi188:> sudo /usr/sbin/setenforce permissive 2024-04-23T15:58:19.998 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-23T15:58:20.007 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-23T15:58:20.023 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-23T15:58:20.025 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi103.front.sepia.ceph.com,smithi148.front.sepia.ceph.com,smithi188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-23T16:04:17.210 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi103.front.sepia.ceph.com'), Remote(name='ubuntu@smithi148.front.sepia.ceph.com'), Remote(name='ubuntu@smithi188.front.sepia.ceph.com')] 2024-04-23T16:04:17.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-23T16:04:17.212 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T16:04:17.287 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-04-23T16:04:17.361 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-23T16:04:17.362 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-04-23T16:04:17.362 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T16:04:17.438 DEBUG:teuthology.orchestra.run.smithi148:> true 2024-04-23T16:04:17.508 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-04-23T16:04:17.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-23T16:04:17.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T16:04:17.581 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-04-23T16:04:17.654 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-23T16:04:17.654 INFO:teuthology.run_tasks:Running task clock... 2024-04-23T16:04:17.664 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-23T16:04:17.664 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T16:04:17.665 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T16:04:17.667 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T16:04:17.667 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T16:04:17.669 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T16:04:17.670 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T16:04:17.699 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T16:04:17.699 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T16:04:17.712 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T16:04:17.713 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T16:04:17.727 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T16:04:17.742 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T16:04:17.779 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2024-04-23T16:04:17.779 INFO:teuthology.orchestra.run.smithi148.stderr:sudo: ntpd: command not found 2024-04-23T16:04:17.789 INFO:teuthology.orchestra.run.smithi148.stdout:506 Cannot talk to daemon 2024-04-23T16:04:17.790 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2024-04-23T16:04:17.803 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T16:04:17.804 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T16:04:17.816 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T16:04:17.817 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T16:04:17.818 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: ntpd: command not found 2024-04-23T16:04:17.829 INFO:teuthology.orchestra.run.smithi188.stdout:506 Cannot talk to daemon 2024-04-23T16:04:17.842 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T16:04:17.857 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T16:04:17.878 INFO:teuthology.orchestra.run.smithi148.stderr:bash: ntpq: command not found 2024-04-23T16:04:17.881 INFO:teuthology.orchestra.run.smithi148.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T16:04:17.881 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================== 2024-04-23T16:04:17.881 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T16:04:17.881 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T16:04:17.881 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T16:04:17.881 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T16:04:17.881 INFO:teuthology.orchestra.run.smithi103.stderr:bash: ntpq: command not found 2024-04-23T16:04:17.885 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T16:04:17.885 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-04-23T16:04:17.885 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T16:04:17.885 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T16:04:17.885 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T16:04:17.885 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T16:04:17.922 INFO:teuthology.orchestra.run.smithi188.stderr:bash: ntpq: command not found 2024-04-23T16:04:17.925 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T16:04:17.925 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-04-23T16:04:17.926 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T16:04:17.926 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T16:04:17.926 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T16:04:17.926 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T16:04:17.927 INFO:teuthology.run_tasks:Running task pexec... 2024-04-23T16:04:17.936 INFO:teuthology.task.pexec:Executing custom commands... 2024-04-23T16:04:17.936 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi103.front.sepia.ceph.com 2024-04-23T16:04:17.936 DEBUG:teuthology.orchestra.run.smithi103:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-23T16:04:17.937 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi148.front.sepia.ceph.com 2024-04-23T16:04:17.937 DEBUG:teuthology.orchestra.run.smithi148:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-23T16:04:17.937 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi188.front.sepia.ceph.com 2024-04-23T16:04:17.938 DEBUG:teuthology.orchestra.run.smithi188:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-23T16:04:18.695 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:26 ago on Tue 23 Apr 2024 04:02:52 PM UTC. 2024-04-23T16:04:18.697 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:22 ago on Tue 23 Apr 2024 04:02:56 PM UTC. 2024-04-23T16:04:18.722 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:25 ago on Tue 23 Apr 2024 04:02:53 PM UTC. 2024-04-23T16:04:20.762 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:04:20.763 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:04:20.763 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2024-04-23T16:04:20.763 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:04:20.763 INFO:teuthology.orchestra.run.smithi148.stdout:Resetting modules: 2024-04-23T16:04:20.763 INFO:teuthology.orchestra.run.smithi148.stdout: container-tools 2024-04-23T16:04:20.763 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:04:20.763 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-04-23T16:04:20.763 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:04:20.763 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:04:20.860 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:04:20.861 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:04:20.861 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-04-23T16:04:20.862 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:04:20.862 INFO:teuthology.orchestra.run.smithi103.stdout:Resetting modules: 2024-04-23T16:04:20.862 INFO:teuthology.orchestra.run.smithi103.stdout: container-tools 2024-04-23T16:04:20.862 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:04:20.862 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-23T16:04:20.862 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:04:20.862 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:04:20.880 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:04:20.881 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:04:20.881 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-23T16:04:20.881 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:04:20.881 INFO:teuthology.orchestra.run.smithi188.stdout:Resetting modules: 2024-04-23T16:04:20.881 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools 2024-04-23T16:04:20.881 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:04:20.881 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-23T16:04:20.881 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:04:20.881 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:04:21.025 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:04:21.130 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:04:21.133 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:04:21.808 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:25 ago on Tue 23 Apr 2024 04:02:56 PM UTC. 2024-04-23T16:04:21.924 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:28 ago on Tue 23 Apr 2024 04:02:53 PM UTC. 2024-04-23T16:04:21.931 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:29 ago on Tue 23 Apr 2024 04:02:52 PM UTC. 2024-04-23T16:04:23.943 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:04:23.980 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:04:23.980 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2024-04-23T16:04:23.980 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:04:23.980 INFO:teuthology.orchestra.run.smithi148.stdout:Installing group/module packages: 2024-04-23T16:04:23.980 INFO:teuthology.orchestra.run.smithi148.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-23T16:04:23.980 INFO:teuthology.orchestra.run.smithi148.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-23T16:04:23.980 INFO:teuthology.orchestra.run.smithi148.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-23T16:04:23.980 INFO:teuthology.orchestra.run.smithi148.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-23T16:04:23.981 INFO:teuthology.orchestra.run.smithi148.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-23T16:04:23.981 INFO:teuthology.orchestra.run.smithi148.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-23T16:04:23.981 INFO:teuthology.orchestra.run.smithi148.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-23T16:04:23.981 INFO:teuthology.orchestra.run.smithi148.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-23T16:04:23.981 INFO:teuthology.orchestra.run.smithi148.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-23T16:04:23.981 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-04-23T16:04:23.981 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-23T16:04:23.981 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-23T16:04:23.981 INFO:teuthology.orchestra.run.smithi148.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-23T16:04:23.981 INFO:teuthology.orchestra.run.smithi148.stdout:Installing weak dependencies: 2024-04-23T16:04:23.981 INFO:teuthology.orchestra.run.smithi148.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-23T16:04:23.981 INFO:teuthology.orchestra.run.smithi148.stdout:Installing module profiles: 2024-04-23T16:04:23.981 INFO:teuthology.orchestra.run.smithi148.stdout: container-tools/common 2024-04-23T16:04:23.981 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:04:23.982 INFO:teuthology.orchestra.run.smithi148.stdout:Enabling module streams: 2024-04-23T16:04:23.982 INFO:teuthology.orchestra.run.smithi148.stdout: container-tools rhel8 2024-04-23T16:04:23.982 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:04:23.982 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-04-23T16:04:23.982 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:04:23.982 INFO:teuthology.orchestra.run.smithi148.stdout:Install 13 Packages 2024-04-23T16:04:23.982 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:04:23.982 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 27 M 2024-04-23T16:04:23.983 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 91 M 2024-04-23T16:04:23.983 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-04-23T16:04:24.052 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:04:24.079 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:04:24.091 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:04:24.091 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-23T16:04:24.091 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:04:24.091 INFO:teuthology.orchestra.run.smithi103.stdout:Installing group/module packages: 2024-04-23T16:04:24.091 INFO:teuthology.orchestra.run.smithi103.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-23T16:04:24.091 INFO:teuthology.orchestra.run.smithi103.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-23T16:04:24.092 INFO:teuthology.orchestra.run.smithi103.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-23T16:04:24.092 INFO:teuthology.orchestra.run.smithi103.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-23T16:04:24.092 INFO:teuthology.orchestra.run.smithi103.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-23T16:04:24.092 INFO:teuthology.orchestra.run.smithi103.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-23T16:04:24.092 INFO:teuthology.orchestra.run.smithi103.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-23T16:04:24.092 INFO:teuthology.orchestra.run.smithi103.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-23T16:04:24.092 INFO:teuthology.orchestra.run.smithi103.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-23T16:04:24.092 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-23T16:04:24.092 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-23T16:04:24.092 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-23T16:04:24.092 INFO:teuthology.orchestra.run.smithi103.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-23T16:04:24.092 INFO:teuthology.orchestra.run.smithi103.stdout:Installing weak dependencies: 2024-04-23T16:04:24.092 INFO:teuthology.orchestra.run.smithi103.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-23T16:04:24.093 INFO:teuthology.orchestra.run.smithi103.stdout:Installing module profiles: 2024-04-23T16:04:24.093 INFO:teuthology.orchestra.run.smithi103.stdout: container-tools/common 2024-04-23T16:04:24.093 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:04:24.093 INFO:teuthology.orchestra.run.smithi103.stdout:Enabling module streams: 2024-04-23T16:04:24.093 INFO:teuthology.orchestra.run.smithi103.stdout: container-tools rhel8 2024-04-23T16:04:24.093 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:04:24.093 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-23T16:04:24.093 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:04:24.093 INFO:teuthology.orchestra.run.smithi103.stdout:Install 13 Packages 2024-04-23T16:04:24.093 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:04:24.094 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 27 M 2024-04-23T16:04:24.094 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 91 M 2024-04-23T16:04:24.094 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-23T16:04:24.117 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:04:24.117 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-04-23T16:04:24.117 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:04:24.117 INFO:teuthology.orchestra.run.smithi188.stdout:Installing group/module packages: 2024-04-23T16:04:24.117 INFO:teuthology.orchestra.run.smithi188.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-23T16:04:24.117 INFO:teuthology.orchestra.run.smithi188.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-23T16:04:24.117 INFO:teuthology.orchestra.run.smithi188.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-23T16:04:24.117 INFO:teuthology.orchestra.run.smithi188.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-23T16:04:24.117 INFO:teuthology.orchestra.run.smithi188.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-23T16:04:24.117 INFO:teuthology.orchestra.run.smithi188.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-23T16:04:24.117 INFO:teuthology.orchestra.run.smithi188.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-23T16:04:24.118 INFO:teuthology.orchestra.run.smithi188.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-23T16:04:24.118 INFO:teuthology.orchestra.run.smithi188.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-23T16:04:24.118 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-23T16:04:24.118 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-23T16:04:24.118 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-23T16:04:24.118 INFO:teuthology.orchestra.run.smithi188.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-23T16:04:24.118 INFO:teuthology.orchestra.run.smithi188.stdout:Installing weak dependencies: 2024-04-23T16:04:24.118 INFO:teuthology.orchestra.run.smithi188.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-23T16:04:24.118 INFO:teuthology.orchestra.run.smithi188.stdout:Installing module profiles: 2024-04-23T16:04:24.118 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools/common 2024-04-23T16:04:24.118 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:04:24.118 INFO:teuthology.orchestra.run.smithi188.stdout:Enabling module streams: 2024-04-23T16:04:24.118 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools rhel8 2024-04-23T16:04:24.118 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:04:24.119 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-23T16:04:24.119 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:04:24.119 INFO:teuthology.orchestra.run.smithi188.stdout:Install 13 Packages 2024-04-23T16:04:24.119 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:04:24.119 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 27 M 2024-04-23T16:04:24.119 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 91 M 2024-04-23T16:04:24.120 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-23T16:04:24.392 INFO:teuthology.orchestra.run.smithi148.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 3.0 MB/s | 689 kB 00:00 2024-04-23T16:04:24.414 INFO:teuthology.orchestra.run.smithi188.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 2.8 MB/s | 689 kB 00:00 2024-04-23T16:04:24.451 INFO:teuthology.orchestra.run.smithi148.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 649 kB/s | 38 kB 00:00 2024-04-23T16:04:24.453 INFO:teuthology.orchestra.run.smithi188.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 3.6 MB/s | 1.0 MB 00:00 2024-04-23T16:04:24.478 INFO:teuthology.orchestra.run.smithi188.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 589 kB/s | 38 kB 00:00 2024-04-23T16:04:24.518 INFO:teuthology.orchestra.run.smithi148.stdout:(3/13): aardvark-dns-1.10.0-1.module_el8+871+55 2.9 MB/s | 1.0 MB 00:00 2024-04-23T16:04:24.537 INFO:teuthology.orchestra.run.smithi188.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 3.0 MB/s | 258 kB 00:00 2024-04-23T16:04:24.551 INFO:teuthology.orchestra.run.smithi148.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 2.5 MB/s | 258 kB 00:00 2024-04-23T16:04:24.577 INFO:teuthology.orchestra.run.smithi103.stdout:(1/13): aardvark-dns-1.10.0-1.module_el8+871+55 3.5 MB/s | 1.0 MB 00:00 2024-04-23T16:04:24.602 INFO:teuthology.orchestra.run.smithi103.stdout:(2/13): cockpit-podman-84.1-1.module_el8+977+8a 2.2 MB/s | 689 kB 00:00 2024-04-23T16:04:24.644 INFO:teuthology.orchestra.run.smithi103.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 565 kB/s | 38 kB 00:00 2024-04-23T16:04:24.645 INFO:teuthology.orchestra.run.smithi188.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.6 MB/s | 180 kB 00:00 2024-04-23T16:04:24.651 INFO:teuthology.orchestra.run.smithi148.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.8 MB/s | 180 kB 00:00 2024-04-23T16:04:24.688 INFO:teuthology.orchestra.run.smithi103.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 2.9 MB/s | 258 kB 00:00 2024-04-23T16:04:24.718 INFO:teuthology.orchestra.run.smithi148.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 371 kB/s | 25 kB 00:00 2024-04-23T16:04:24.720 INFO:teuthology.orchestra.run.smithi188.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 330 kB/s | 25 kB 00:00 2024-04-23T16:04:24.770 INFO:teuthology.orchestra.run.smithi103.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 2.2 MB/s | 180 kB 00:00 2024-04-23T16:04:24.785 INFO:teuthology.orchestra.run.smithi148.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.4 MB/s | 94 kB 00:00 2024-04-23T16:04:24.787 INFO:teuthology.orchestra.run.smithi188.stdout:(7/13): netavark-1.10.3-1.module_el8+951+32019c 13 MB/s | 4.1 MB 00:00 2024-04-23T16:04:24.813 INFO:teuthology.orchestra.run.smithi188.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.0 MB/s | 94 kB 00:00 2024-04-23T16:04:24.837 INFO:teuthology.orchestra.run.smithi103.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 371 kB/s | 25 kB 00:00 2024-04-23T16:04:24.954 INFO:teuthology.orchestra.run.smithi103.stdout:(7/13): netavark-1.10.3-1.module_el8+951+32019c 13 MB/s | 4.1 MB 00:00 2024-04-23T16:04:24.971 INFO:teuthology.orchestra.run.smithi103.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 701 kB/s | 94 kB 00:00 2024-04-23T16:04:25.096 INFO:teuthology.orchestra.run.smithi188.stdout:(9/13): toolbox-0.0.99.5-2.module_el8+971+3d3df 8.9 MB/s | 2.5 MB 00:00 2024-04-23T16:04:25.180 INFO:teuthology.orchestra.run.smithi188.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 633 kB/s | 53 kB 00:00 2024-04-23T16:04:25.255 INFO:teuthology.orchestra.run.smithi188.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 546 kB/s | 41 kB 00:00 2024-04-23T16:04:25.304 INFO:teuthology.orchestra.run.smithi103.stdout:(9/13): buildah-1.33.6-2.module_el8+976+1136073 9.5 MB/s | 9.6 MB 00:01 2024-04-23T16:04:25.337 INFO:teuthology.orchestra.run.smithi148.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 5.0 MB/s | 4.1 MB 00:00 2024-04-23T16:04:25.363 INFO:teuthology.orchestra.run.smithi103.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 904 kB/s | 53 kB 00:00 2024-04-23T16:04:25.389 INFO:teuthology.orchestra.run.smithi188.stdout:(12/13): buildah-1.33.6-2.module_el8+976+113607 7.9 MB/s | 9.6 MB 00:01 2024-04-23T16:04:25.422 INFO:teuthology.orchestra.run.smithi103.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 705 kB/s | 41 kB 00:00 2024-04-23T16:04:25.445 INFO:teuthology.orchestra.run.smithi148.stdout:(9/13): buildah-1.33.6-2.module_el8+976+1136073 7.6 MB/s | 9.6 MB 00:01 2024-04-23T16:04:25.486 INFO:teuthology.orchestra.run.smithi103.stdout:(12/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 4.9 MB/s | 2.5 MB 00:00 2024-04-23T16:04:25.503 INFO:teuthology.orchestra.run.smithi148.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 909 kB/s | 53 kB 00:00 2024-04-23T16:04:25.547 INFO:teuthology.orchestra.run.smithi188.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 12 MB/s | 8.8 MB 00:00 2024-04-23T16:04:25.548 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:04:25.548 INFO:teuthology.orchestra.run.smithi188.stdout:Total 19 MB/s | 27 MB 00:01 2024-04-23T16:04:25.562 INFO:teuthology.orchestra.run.smithi148.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 701 kB/s | 41 kB 00:00 2024-04-23T16:04:25.589 INFO:teuthology.orchestra.run.smithi103.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 14 MB/s | 8.8 MB 00:00 2024-04-23T16:04:25.589 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:04:25.589 INFO:teuthology.orchestra.run.smithi103.stdout:Total 18 MB/s | 27 MB 00:01 2024-04-23T16:04:25.670 INFO:teuthology.orchestra.run.smithi148.stdout:(12/13): skopeo-1.14.3-0.1.module_el8+885+7da14 10 MB/s | 8.8 MB 00:00 2024-04-23T16:04:25.817 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-23T16:04:25.838 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-23T16:04:25.838 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-23T16:04:25.854 INFO:teuthology.orchestra.run.smithi148.stdout:(13/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 4.9 MB/s | 2.5 MB 00:00 2024-04-23T16:04:25.854 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:04:25.854 INFO:teuthology.orchestra.run.smithi148.stdout:Total 15 MB/s | 27 MB 00:01 2024-04-23T16:04:25.857 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-23T16:04:25.879 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-23T16:04:25.879 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-23T16:04:26.094 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-23T16:04:26.098 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-23T16:04:26.124 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-04-23T16:04:26.133 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-23T16:04:26.136 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-23T16:04:26.145 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-04-23T16:04:26.145 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-04-23T16:04:26.410 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-04-23T16:04:26.413 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-04-23T16:04:26.718 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-23T16:04:26.780 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-23T16:04:27.023 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-04-23T16:04:27.703 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-23T16:04:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-23T16:04:27.909 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-23T16:04:27.961 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-23T16:04:27.980 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-23T16:04:28.070 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-23T16:04:28.115 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-23T16:04:28.151 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-23T16:04:28.281 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-23T16:04:28.318 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-23T16:04:28.322 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-23T16:04:28.514 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-23T16:04:28.520 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-23T16:04:28.525 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-23T16:04:28.765 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-23T16:04:29.056 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-23T16:04:29.075 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-23T16:04:29.301 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-23T16:04:29.998 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-23T16:04:30.030 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-23T16:04:30.238 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-23T16:04:30.268 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-23T16:04:30.289 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-23T16:04:30.456 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-23T16:04:30.485 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-23T16:04:30.499 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-23T16:04:30.697 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-23T16:04:30.869 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-23T16:04:30.955 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-23T16:04:31.051 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-23T16:04:31.128 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-23T16:04:31.181 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-23T16:04:31.287 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-23T16:04:31.358 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-23T16:04:31.361 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-23T16:04:31.486 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-23T16:04:31.579 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-23T16:04:31.614 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-23T16:04:31.787 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-23T16:04:31.917 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-23T16:04:31.917 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-23T16:04:31.918 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-23T16:04:31.918 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-23T16:04:31.918 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-23T16:04:31.918 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-23T16:04:31.918 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-23T16:04:31.918 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-23T16:04:31.918 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-23T16:04:31.918 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-23T16:04:31.918 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-23T16:04:31.918 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-23T16:04:31.918 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-23T16:04:32.018 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-23T16:04:32.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-23T16:04:32.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-23T16:04:32.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-23T16:04:32.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-23T16:04:32.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-23T16:04:32.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-23T16:04:32.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-23T16:04:32.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-23T16:04:32.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-23T16:04:32.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-23T16:04:32.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-23T16:04:32.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-23T16:04:32.213 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-23T16:04:32.213 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-23T16:04:32.213 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-23T16:04:32.213 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-23T16:04:32.213 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-23T16:04:32.213 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-23T16:04:32.213 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-23T16:04:32.213 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-23T16:04:32.213 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-23T16:04:32.213 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-23T16:04:32.213 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-23T16:04:32.214 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-23T16:04:32.214 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-23T16:04:32.442 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-23T16:04:32.442 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:04:32.442 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-23T16:04:32.442 INFO:teuthology.orchestra.run.smithi188.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-23T16:04:32.442 INFO:teuthology.orchestra.run.smithi188.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-23T16:04:32.442 INFO:teuthology.orchestra.run.smithi188.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-23T16:04:32.442 INFO:teuthology.orchestra.run.smithi188.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-23T16:04:32.442 INFO:teuthology.orchestra.run.smithi188.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-23T16:04:32.442 INFO:teuthology.orchestra.run.smithi188.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-23T16:04:32.443 INFO:teuthology.orchestra.run.smithi188.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-23T16:04:32.443 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-23T16:04:32.443 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-23T16:04:32.443 INFO:teuthology.orchestra.run.smithi188.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-23T16:04:32.443 INFO:teuthology.orchestra.run.smithi188.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-23T16:04:32.443 INFO:teuthology.orchestra.run.smithi188.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-23T16:04:32.443 INFO:teuthology.orchestra.run.smithi188.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-23T16:04:32.443 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:04:32.443 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:04:32.519 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-23T16:04:32.519 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:04:32.519 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-23T16:04:32.519 INFO:teuthology.orchestra.run.smithi103.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-23T16:04:32.519 INFO:teuthology.orchestra.run.smithi103.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-23T16:04:32.519 INFO:teuthology.orchestra.run.smithi103.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-23T16:04:32.519 INFO:teuthology.orchestra.run.smithi103.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-23T16:04:32.519 INFO:teuthology.orchestra.run.smithi103.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-23T16:04:32.519 INFO:teuthology.orchestra.run.smithi103.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-23T16:04:32.519 INFO:teuthology.orchestra.run.smithi103.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-23T16:04:32.520 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-23T16:04:32.520 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-23T16:04:32.520 INFO:teuthology.orchestra.run.smithi103.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-23T16:04:32.520 INFO:teuthology.orchestra.run.smithi103.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-23T16:04:32.520 INFO:teuthology.orchestra.run.smithi103.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-23T16:04:32.520 INFO:teuthology.orchestra.run.smithi103.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-23T16:04:32.520 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:04:32.520 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:04:32.721 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-23T16:04:32.721 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:04:32.721 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-04-23T16:04:32.721 INFO:teuthology.orchestra.run.smithi148.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-23T16:04:32.721 INFO:teuthology.orchestra.run.smithi148.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-23T16:04:32.721 INFO:teuthology.orchestra.run.smithi148.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-23T16:04:32.721 INFO:teuthology.orchestra.run.smithi148.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-23T16:04:32.721 INFO:teuthology.orchestra.run.smithi148.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-23T16:04:32.721 INFO:teuthology.orchestra.run.smithi148.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-23T16:04:32.721 INFO:teuthology.orchestra.run.smithi148.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-23T16:04:32.721 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-23T16:04:32.722 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-23T16:04:32.722 INFO:teuthology.orchestra.run.smithi148.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-23T16:04:32.722 INFO:teuthology.orchestra.run.smithi148.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-23T16:04:32.722 INFO:teuthology.orchestra.run.smithi148.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-23T16:04:32.722 INFO:teuthology.orchestra.run.smithi148.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-23T16:04:32.722 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:04:32.722 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:04:32.734 DEBUG:teuthology.parallel:result is None 2024-04-23T16:04:32.815 DEBUG:teuthology.parallel:result is None 2024-04-23T16:04:33.013 DEBUG:teuthology.parallel:result is None 2024-04-23T16:04:33.014 INFO:teuthology.run_tasks:Running task install... 2024-04-23T16:04:33.022 DEBUG:teuthology.task.install:project ceph 2024-04-23T16:04:33.022 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'}} 2024-04-23T16:04:33.022 DEBUG:teuthology.task.install:config {'branch': 'nautilus', 'exclude_packages': ['cephadm', 'ceph-mgr-cephadm', 'ceph-immutable-object-cache', 'python3-rados', 'python3-rgw', 'python3-rbd', 'python3-cephfs', 'ceph-volume'], 'extra_packages': ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'], 'flavor': 'default', 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'} 2024-04-23T16:04:33.023 INFO:teuthology.task.install:Using flavor: default 2024-04-23T16:04:33.029 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-23T16:04:33.029 INFO:teuthology.task.install:extra packages: ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'] 2024-04-23T16:04:33.029 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'nautilus', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['cephadm', 'ceph-mgr-cephadm', 'ceph-immutable-object-cache', 'python3-rados', 'python3-rgw', 'python3-rbd', 'python3-cephfs', 'ceph-volume'], 'extra_packages': ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb', 'tag': None, 'wait_for_package': False} 2024-04-23T16:04:33.029 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-23T16:04:33.029 INFO:teuthology.packaging:ref: None 2024-04-23T16:04:33.030 INFO:teuthology.packaging:tag: None 2024-04-23T16:04:33.030 INFO:teuthology.packaging:branch: nautilus 2024-04-23T16:04:33.030 INFO:teuthology.packaging:sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T16:04:33.030 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-04-23T16:04:33.032 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'nautilus', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['cephadm', 'ceph-mgr-cephadm', 'ceph-immutable-object-cache', 'python3-rados', 'python3-rgw', 'python3-rbd', 'python3-cephfs', 'ceph-volume'], 'extra_packages': ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb', 'tag': None, 'wait_for_package': False} 2024-04-23T16:04:33.032 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-23T16:04:33.032 INFO:teuthology.packaging:ref: None 2024-04-23T16:04:33.032 INFO:teuthology.packaging:tag: None 2024-04-23T16:04:33.032 INFO:teuthology.packaging:branch: nautilus 2024-04-23T16:04:33.032 INFO:teuthology.packaging:sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T16:04:33.033 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-04-23T16:04:33.033 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'nautilus', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['cephadm', 'ceph-mgr-cephadm', 'ceph-immutable-object-cache', 'python3-rados', 'python3-rgw', 'python3-rbd', 'python3-cephfs', 'ceph-volume'], 'extra_packages': ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb', 'tag': None, 'wait_for_package': False} 2024-04-23T16:04:33.034 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-23T16:04:33.034 INFO:teuthology.packaging:ref: None 2024-04-23T16:04:33.034 INFO:teuthology.packaging:tag: None 2024-04-23T16:04:33.034 INFO:teuthology.packaging:branch: nautilus 2024-04-23T16:04:33.034 INFO:teuthology.packaging:sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T16:04:33.034 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-04-23T16:04:33.212 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/ 2024-04-23T16:04:33.212 INFO:teuthology.task.install.rpm:Package version is 14.2.22-0 2024-04-23T16:04:33.228 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/ 2024-04-23T16:04:33.228 INFO:teuthology.task.install.rpm:Package version is 14.2.22-0 2024-04-23T16:04:33.239 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/ 2024-04-23T16:04:33.240 INFO:teuthology.task.install.rpm:Package version is 14.2.22-0 2024-04-23T16:04:33.252 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T16:04:33.252 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T16:04:33.253 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T16:04:33.263 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T16:04:33.263 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-04-23T16:04:33.264 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T16:04:33.276 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T16:04:33.276 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-23T16:04:33.276 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T16:04:33.286 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, rbd-fuse, rbd-mirror, rbd-nbd, python-rados, python-rgw, python-rbd, python-cephfs on remote rpm x86_64 2024-04-23T16:04:33.286 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-23T16:04:33.286 INFO:teuthology.packaging:ref: None 2024-04-23T16:04:33.286 INFO:teuthology.packaging:tag: None 2024-04-23T16:04:33.286 INFO:teuthology.packaging:branch: nautilus 2024-04-23T16:04:33.286 INFO:teuthology.packaging:sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T16:04:33.287 DEBUG:teuthology.orchestra.run.smithi188:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/nautilus/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T16:04:33.291 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, rbd-fuse, rbd-mirror, rbd-nbd, python-rados, python-rgw, python-rbd, python-cephfs on remote rpm x86_64 2024-04-23T16:04:33.291 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-23T16:04:33.291 INFO:teuthology.packaging:ref: None 2024-04-23T16:04:33.291 INFO:teuthology.packaging:tag: None 2024-04-23T16:04:33.291 INFO:teuthology.packaging:branch: nautilus 2024-04-23T16:04:33.291 INFO:teuthology.packaging:sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T16:04:33.291 DEBUG:teuthology.orchestra.run.smithi148:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/nautilus/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T16:04:33.303 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, rbd-fuse, rbd-mirror, rbd-nbd, python-rados, python-rgw, python-rbd, python-cephfs on remote rpm x86_64 2024-04-23T16:04:33.303 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-23T16:04:33.303 INFO:teuthology.packaging:ref: None 2024-04-23T16:04:33.303 INFO:teuthology.packaging:tag: None 2024-04-23T16:04:33.303 INFO:teuthology.packaging:branch: nautilus 2024-04-23T16:04:33.303 INFO:teuthology.packaging:sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T16:04:33.303 DEBUG:teuthology.orchestra.run.smithi103:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/nautilus/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T16:04:33.357 DEBUG:teuthology.orchestra.run.smithi188:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T16:04:33.358 DEBUG:teuthology.orchestra.run.smithi148:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T16:04:33.371 DEBUG:teuthology.orchestra.run.smithi103:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T16:04:33.435 DEBUG:teuthology.orchestra.run.smithi148:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T16:04:33.437 DEBUG:teuthology.orchestra.run.smithi188:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T16:04:33.447 DEBUG:teuthology.orchestra.run.smithi103:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T16:04:33.501 INFO:teuthology.orchestra.run.smithi148.stdout:check_obsoletes = 1 2024-04-23T16:04:33.503 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum clean all 2024-04-23T16:04:33.505 INFO:teuthology.orchestra.run.smithi188.stdout:check_obsoletes = 1 2024-04-23T16:04:33.506 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum clean all 2024-04-23T16:04:33.512 INFO:teuthology.orchestra.run.smithi103.stdout:check_obsoletes = 1 2024-04-23T16:04:33.514 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum clean all 2024-04-23T16:04:33.768 INFO:teuthology.orchestra.run.smithi188.stdout:78 files removed 2024-04-23T16:04:33.772 INFO:teuthology.orchestra.run.smithi148.stdout:78 files removed 2024-04-23T16:04:33.788 INFO:teuthology.orchestra.run.smithi103.stdout:78 files removed 2024-04-23T16:04:33.795 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-radosgw 2024-04-23T16:04:33.799 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-radosgw 2024-04-23T16:04:33.817 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-radosgw 2024-04-23T16:04:34.349 INFO:teuthology.orchestra.run.smithi188.stdout:ceph packages for x86_64 437 kB/s | 65 kB 00:00 2024-04-23T16:04:34.359 INFO:teuthology.orchestra.run.smithi103.stdout:ceph packages for x86_64 481 kB/s | 65 kB 00:00 2024-04-23T16:04:34.370 INFO:teuthology.orchestra.run.smithi148.stdout:ceph packages for x86_64 399 kB/s | 65 kB 00:00 2024-04-23T16:04:34.525 INFO:teuthology.orchestra.run.smithi188.stdout:ceph noarch packages 82 kB/s | 11 kB 00:00 2024-04-23T16:04:34.526 INFO:teuthology.orchestra.run.smithi103.stdout:ceph noarch packages 88 kB/s | 11 kB 00:00 2024-04-23T16:04:34.529 INFO:teuthology.orchestra.run.smithi148.stdout:ceph noarch packages 94 kB/s | 11 kB 00:00 2024-04-23T16:04:34.688 INFO:teuthology.orchestra.run.smithi148.stdout:ceph source packages 13 kB/s | 1.7 kB 00:00 2024-04-23T16:04:34.703 INFO:teuthology.orchestra.run.smithi103.stdout:ceph source packages 11 kB/s | 1.7 kB 00:00 2024-04-23T16:04:34.893 INFO:teuthology.orchestra.run.smithi188.stdout:ceph source packages 4.9 kB/s | 1.7 kB 00:00 2024-04-23T16:04:37.239 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - AppStream 11 MB/s | 28 MB 00:02 2024-04-23T16:04:37.853 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 8 - AppStream 9.0 MB/s | 28 MB 00:03 2024-04-23T16:04:37.884 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - AppStream 9.4 MB/s | 28 MB 00:02 2024-04-23T16:04:43.514 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - BaseOS 8.0 MB/s | 10 MB 00:01 2024-04-23T16:04:43.654 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 8 - BaseOS 12 MB/s | 10 MB 00:00 2024-04-23T16:04:44.419 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - BaseOS 6.7 MB/s | 10 MB 00:01 2024-04-23T16:04:45.381 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - Extras 61 kB/s | 18 kB 00:00 2024-04-23T16:04:45.666 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 8 - Extras 41 kB/s | 18 kB 00:00 2024-04-23T16:04:45.782 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - Extras common packages 26 kB/s | 7.7 kB 00:00 2024-04-23T16:04:46.105 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 8 - Extras common packages 23 kB/s | 7.7 kB 00:00 2024-04-23T16:04:46.403 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras 45 kB/s | 18 kB 00:00 2024-04-23T16:04:46.836 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras common packages 24 kB/s | 7.7 kB 00:00 2024-04-23T16:04:47.042 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - PowerTools 4.2 MB/s | 5.0 MB 00:01 2024-04-23T16:04:47.239 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 8 - PowerTools 4.7 MB/s | 5.0 MB 00:01 2024-04-23T16:04:47.812 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - PowerTools 5.6 MB/s | 5.0 MB 00:00 2024-04-23T16:04:48.110 INFO:teuthology.orchestra.run.smithi148.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-04-23T16:04:48.300 INFO:teuthology.orchestra.run.smithi103.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-04-23T16:04:48.889 INFO:teuthology.orchestra.run.smithi188.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-04-23T16:04:50.093 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 16 MB 00:01 2024-04-23T16:04:50.688 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 6.6 MB/s | 16 MB 00:02 2024-04-23T16:04:51.340 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 6.9 MB/s | 16 MB 00:02 2024-04-23T16:04:53.944 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 446 kB/s | 24 kB 00:00 2024-04-23T16:04:54.503 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 342 kB/s | 24 kB 00:00 2024-04-23T16:04:55.148 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 391 kB/s | 24 kB 00:00 2024-04-23T16:04:57.590 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:04:57.592 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:04:57.592 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-23T16:04:57.592 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:04:57.592 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-23T16:04:57.592 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw x86_64 2:14.2.22-0.el8 ceph 6.0 M 2024-04-23T16:04:57.592 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrading: 2024-04-23T16:04:57.593 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 x86_64 2:14.2.22-0.el8 ceph 3.5 M 2024-04-23T16:04:57.593 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1 x86_64 2:14.2.22-0.el8 ceph 1.7 M 2024-04-23T16:04:57.593 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-23T16:04:57.593 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base x86_64 2:14.2.22-0.el8 ceph 5.4 M 2024-04-23T16:04:57.593 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common x86_64 2:14.2.22-0.el8 ceph 19 M 2024-04-23T16:04:57.593 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux x86_64 2:14.2.22-0.el8 ceph 25 k 2024-04-23T16:04:57.593 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-23T16:04:57.593 INFO:teuthology.orchestra.run.smithi103.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-04-23T16:04:57.593 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2 x86_64 2:14.2.22-0.el8 ceph 520 k 2024-04-23T16:04:57.593 INFO:teuthology.orchestra.run.smithi103.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-23T16:04:57.593 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-23T16:04:57.594 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1 x86_64 2:14.2.22-0.el8 ceph 376 k 2024-04-23T16:04:57.594 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-23T16:04:57.594 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2 x86_64 2:14.2.22-0.el8 ceph 5.3 M 2024-04-23T16:04:57.594 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-23T16:04:57.594 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-23T16:04:57.594 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse x86_64 2:14.2.22-0.el8 ceph 39 k 2024-04-23T16:04:57.594 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs x86_64 2:14.2.22-0.el8 ceph 159 k 2024-04-23T16:04:57.594 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-23T16:04:57.595 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados x86_64 2:14.2.22-0.el8 ceph 323 k 2024-04-23T16:04:57.595 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd x86_64 2:14.2.22-0.el8 ceph 291 k 2024-04-23T16:04:57.595 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw x86_64 2:14.2.22-0.el8 ceph 113 k 2024-04-23T16:04:57.595 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:04:57.595 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-23T16:04:57.595 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:04:57.595 INFO:teuthology.orchestra.run.smithi103.stdout:Install 20 Packages 2024-04-23T16:04:57.596 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrade 2 Packages 2024-04-23T16:04:57.596 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:04:57.596 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 45 M 2024-04-23T16:04:57.596 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-23T16:04:58.129 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:04:58.131 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:04:58.131 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2024-04-23T16:04:58.131 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:04:58.131 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-04-23T16:04:58.131 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw x86_64 2:14.2.22-0.el8 ceph 6.0 M 2024-04-23T16:04:58.131 INFO:teuthology.orchestra.run.smithi148.stdout:Upgrading: 2024-04-23T16:04:58.132 INFO:teuthology.orchestra.run.smithi148.stdout: librados2 x86_64 2:14.2.22-0.el8 ceph 3.5 M 2024-04-23T16:04:58.132 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1 x86_64 2:14.2.22-0.el8 ceph 1.7 M 2024-04-23T16:04:58.132 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-04-23T16:04:58.132 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base x86_64 2:14.2.22-0.el8 ceph 5.4 M 2024-04-23T16:04:58.132 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common x86_64 2:14.2.22-0.el8 ceph 19 M 2024-04-23T16:04:58.132 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-selinux x86_64 2:14.2.22-0.el8 ceph 25 k 2024-04-23T16:04:58.132 INFO:teuthology.orchestra.run.smithi148.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-23T16:04:58.132 INFO:teuthology.orchestra.run.smithi148.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-04-23T16:04:58.132 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2 x86_64 2:14.2.22-0.el8 ceph 520 k 2024-04-23T16:04:58.132 INFO:teuthology.orchestra.run.smithi148.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-23T16:04:58.132 INFO:teuthology.orchestra.run.smithi148.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-23T16:04:58.132 INFO:teuthology.orchestra.run.smithi148.stdout: libradosstriper1 x86_64 2:14.2.22-0.el8 ceph 376 k 2024-04-23T16:04:58.132 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-23T16:04:58.133 INFO:teuthology.orchestra.run.smithi148.stdout: librgw2 x86_64 2:14.2.22-0.el8 ceph 5.3 M 2024-04-23T16:04:58.133 INFO:teuthology.orchestra.run.smithi148.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-23T16:04:58.133 INFO:teuthology.orchestra.run.smithi148.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-23T16:04:58.133 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse x86_64 2:14.2.22-0.el8 ceph 39 k 2024-04-23T16:04:58.133 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs x86_64 2:14.2.22-0.el8 ceph 159 k 2024-04-23T16:04:58.133 INFO:teuthology.orchestra.run.smithi148.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-23T16:04:58.133 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados x86_64 2:14.2.22-0.el8 ceph 323 k 2024-04-23T16:04:58.133 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd x86_64 2:14.2.22-0.el8 ceph 291 k 2024-04-23T16:04:58.133 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw x86_64 2:14.2.22-0.el8 ceph 113 k 2024-04-23T16:04:58.133 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:04:58.133 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-04-23T16:04:58.133 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:04:58.133 INFO:teuthology.orchestra.run.smithi148.stdout:Install 20 Packages 2024-04-23T16:04:58.133 INFO:teuthology.orchestra.run.smithi148.stdout:Upgrade 2 Packages 2024-04-23T16:04:58.134 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:04:58.134 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 45 M 2024-04-23T16:04:58.134 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-04-23T16:04:58.353 INFO:teuthology.orchestra.run.smithi103.stdout:(1/22): ceph-base-14.2.22-0.el8.x86_64.rpm 18 MB/s | 5.4 MB 00:00 2024-04-23T16:04:58.395 INFO:teuthology.orchestra.run.smithi103.stdout:(2/22): ceph-selinux-14.2.22-0.el8.x86_64.rpm 600 kB/s | 25 kB 00:00 2024-04-23T16:04:58.437 INFO:teuthology.orchestra.run.smithi103.stdout:(3/22): libcephfs2-14.2.22-0.el8.x86_64.rpm 12 MB/s | 520 kB 00:00 2024-04-23T16:04:58.546 INFO:teuthology.orchestra.run.smithi103.stdout:(4/22): ceph-radosgw-14.2.22-0.el8.x86_64.rpm 12 MB/s | 6.0 MB 00:00 2024-04-23T16:04:58.571 INFO:teuthology.orchestra.run.smithi103.stdout:(5/22): libradosstriper1-14.2.22-0.el8.x86_64.r 2.8 MB/s | 376 kB 00:00 2024-04-23T16:04:58.670 INFO:teuthology.orchestra.run.smithi148.stdout:(1/22): ceph-base-14.2.22-0.el8.x86_64.rpm 34 MB/s | 5.4 MB 00:00 2024-04-23T16:04:58.729 INFO:teuthology.orchestra.run.smithi148.stdout:(2/22): ceph-selinux-14.2.22-0.el8.x86_64.rpm 428 kB/s | 25 kB 00:00 2024-04-23T16:04:58.754 INFO:teuthology.orchestra.run.smithi103.stdout:(6/22): librgw2-14.2.22-0.el8.x86_64.rpm 25 MB/s | 5.3 MB 00:00 2024-04-23T16:04:58.762 INFO:teuthology.orchestra.run.smithi148.stdout:(3/22): libcephfs2-14.2.22-0.el8.x86_64.rpm 15 MB/s | 520 kB 00:00 2024-04-23T16:04:58.771 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:04:58.774 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:04:58.774 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-04-23T16:04:58.774 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:04:58.774 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-23T16:04:58.774 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw x86_64 2:14.2.22-0.el8 ceph 6.0 M 2024-04-23T16:04:58.775 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrading: 2024-04-23T16:04:58.775 INFO:teuthology.orchestra.run.smithi188.stdout: librados2 x86_64 2:14.2.22-0.el8 ceph 3.5 M 2024-04-23T16:04:58.775 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1 x86_64 2:14.2.22-0.el8 ceph 1.7 M 2024-04-23T16:04:58.775 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-23T16:04:58.775 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base x86_64 2:14.2.22-0.el8 ceph 5.4 M 2024-04-23T16:04:58.775 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common x86_64 2:14.2.22-0.el8 ceph 19 M 2024-04-23T16:04:58.775 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux x86_64 2:14.2.22-0.el8 ceph 25 k 2024-04-23T16:04:58.775 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-23T16:04:58.775 INFO:teuthology.orchestra.run.smithi188.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-04-23T16:04:58.775 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2 x86_64 2:14.2.22-0.el8 ceph 520 k 2024-04-23T16:04:58.775 INFO:teuthology.orchestra.run.smithi188.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-23T16:04:58.775 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-23T16:04:58.775 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 x86_64 2:14.2.22-0.el8 ceph 376 k 2024-04-23T16:04:58.776 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-23T16:04:58.776 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2 x86_64 2:14.2.22-0.el8 ceph 5.3 M 2024-04-23T16:04:58.776 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-23T16:04:58.776 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-23T16:04:58.776 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse x86_64 2:14.2.22-0.el8 ceph 39 k 2024-04-23T16:04:58.776 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs x86_64 2:14.2.22-0.el8 ceph 159 k 2024-04-23T16:04:58.776 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-23T16:04:58.776 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados x86_64 2:14.2.22-0.el8 ceph 323 k 2024-04-23T16:04:58.776 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd x86_64 2:14.2.22-0.el8 ceph 291 k 2024-04-23T16:04:58.776 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw x86_64 2:14.2.22-0.el8 ceph 113 k 2024-04-23T16:04:58.776 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:04:58.776 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-23T16:04:58.776 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:04:58.776 INFO:teuthology.orchestra.run.smithi188.stdout:Install 20 Packages 2024-04-23T16:04:58.777 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrade 2 Packages 2024-04-23T16:04:58.777 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:04:58.777 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 45 M 2024-04-23T16:04:58.777 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-23T16:04:58.780 INFO:teuthology.orchestra.run.smithi103.stdout:(7/22): python3-ceph-argparse-14.2.22-0.el8.x86 188 kB/s | 39 kB 00:00 2024-04-23T16:04:58.805 INFO:teuthology.orchestra.run.smithi103.stdout:(8/22): python3-cephfs-14.2.22-0.el8.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-04-23T16:04:58.821 INFO:teuthology.orchestra.run.smithi148.stdout:(4/22): libradosstriper1-14.2.22-0.el8.x86_64.r 6.3 MB/s | 376 kB 00:00 2024-04-23T16:04:58.847 INFO:teuthology.orchestra.run.smithi103.stdout:(9/22): python3-rados-14.2.22-0.el8.x86_64.rpm 4.7 MB/s | 323 kB 00:00 2024-04-23T16:04:58.881 INFO:teuthology.orchestra.run.smithi103.stdout:(10/22): python3-rbd-14.2.22-0.el8.x86_64.rpm 3.8 MB/s | 291 kB 00:00 2024-04-23T16:04:58.905 INFO:teuthology.orchestra.run.smithi148.stdout:(5/22): ceph-radosgw-14.2.22-0.el8.x86_64.rpm 15 MB/s | 6.0 MB 00:00 2024-04-23T16:04:58.906 INFO:teuthology.orchestra.run.smithi103.stdout:(11/22): python3-rgw-14.2.22-0.el8.x86_64.rpm 1.9 MB/s | 113 kB 00:00 2024-04-23T16:04:58.930 INFO:teuthology.orchestra.run.smithi148.stdout:(6/22): python3-ceph-argparse-14.2.22-0.el8.x86 1.6 MB/s | 39 kB 00:00 2024-04-23T16:04:58.964 INFO:teuthology.orchestra.run.smithi148.stdout:(7/22): python3-cephfs-14.2.22-0.el8.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-04-23T16:04:58.997 INFO:teuthology.orchestra.run.smithi148.stdout:(8/22): python3-rados-14.2.22-0.el8.x86_64.rpm 9.6 MB/s | 323 kB 00:00 2024-04-23T16:04:59.031 INFO:teuthology.orchestra.run.smithi148.stdout:(9/22): python3-rbd-14.2.22-0.el8.x86_64.rpm 8.6 MB/s | 291 kB 00:00 2024-04-23T16:04:59.048 INFO:teuthology.orchestra.run.smithi148.stdout:(10/22): python3-rgw-14.2.22-0.el8.x86_64.rpm 6.6 MB/s | 113 kB 00:00 2024-04-23T16:04:59.132 INFO:teuthology.orchestra.run.smithi148.stdout:(11/22): librgw2-14.2.22-0.el8.x86_64.rpm 17 MB/s | 5.3 MB 00:00 2024-04-23T16:04:59.149 INFO:teuthology.orchestra.run.smithi188.stdout:(1/22): ceph-base-14.2.22-0.el8.x86_64.rpm 40 MB/s | 5.4 MB 00:00 2024-04-23T16:04:59.191 INFO:teuthology.orchestra.run.smithi188.stdout:(2/22): ceph-selinux-14.2.22-0.el8.x86_64.rpm 599 kB/s | 25 kB 00:00 2024-04-23T16:04:59.224 INFO:teuthology.orchestra.run.smithi188.stdout:(3/22): libcephfs2-14.2.22-0.el8.x86_64.rpm 15 MB/s | 520 kB 00:00 2024-04-23T16:04:59.258 INFO:teuthology.orchestra.run.smithi188.stdout:(4/22): libradosstriper1-14.2.22-0.el8.x86_64.r 11 MB/s | 376 kB 00:00 2024-04-23T16:04:59.332 INFO:teuthology.orchestra.run.smithi148.stdout:(12/22): ceph-common-14.2.22-0.el8.x86_64.rpm 24 MB/s | 19 MB 00:00 2024-04-23T16:04:59.350 INFO:teuthology.orchestra.run.smithi188.stdout:(5/22): ceph-radosgw-14.2.22-0.el8.x86_64.rpm 18 MB/s | 6.0 MB 00:00 2024-04-23T16:04:59.365 INFO:teuthology.orchestra.run.smithi103.stdout:(12/22): ceph-common-14.2.22-0.el8.x86_64.rpm 15 MB/s | 19 MB 00:01 2024-04-23T16:04:59.367 INFO:teuthology.orchestra.run.smithi188.stdout:(6/22): python3-ceph-argparse-14.2.22-0.el8.x86 2.4 MB/s | 39 kB 00:00 2024-04-23T16:04:59.392 INFO:teuthology.orchestra.run.smithi188.stdout:(7/22): python3-cephfs-14.2.22-0.el8.x86_64.rpm 6.2 MB/s | 159 kB 00:00 2024-04-23T16:04:59.426 INFO:teuthology.orchestra.run.smithi188.stdout:(8/22): python3-rados-14.2.22-0.el8.x86_64.rpm 9.4 MB/s | 323 kB 00:00 2024-04-23T16:04:59.448 INFO:teuthology.orchestra.run.smithi148.stdout:(13/22): librdkafka-1.6.1-1.el8.x86_64.rpm 1.7 MB/s | 677 kB 00:00 2024-04-23T16:04:59.459 INFO:teuthology.orchestra.run.smithi188.stdout:(9/22): python3-rbd-14.2.22-0.el8.x86_64.rpm 8.6 MB/s | 291 kB 00:00 2024-04-23T16:04:59.485 INFO:teuthology.orchestra.run.smithi188.stdout:(10/22): python3-rgw-14.2.22-0.el8.x86_64.rpm 4.5 MB/s | 113 kB 00:00 2024-04-23T16:04:59.491 INFO:teuthology.orchestra.run.smithi148.stdout:(14/22): python3-prettytable-0.7.2-14.el8.noarc 277 kB/s | 44 kB 00:00 2024-04-23T16:04:59.549 INFO:teuthology.orchestra.run.smithi148.stdout:(15/22): lttng-ust-2.8.1-11.el8.x86_64.rpm 622 kB/s | 259 kB 00:00 2024-04-23T16:04:59.568 INFO:teuthology.orchestra.run.smithi188.stdout:(11/22): librgw2-14.2.22-0.el8.x86_64.rpm 17 MB/s | 5.3 MB 00:00 2024-04-23T16:04:59.591 INFO:teuthology.orchestra.run.smithi148.stdout:(16/22): librabbitmq-0.9.0-5.el8.x86_64.rpm 330 kB/s | 47 kB 00:00 2024-04-23T16:04:59.659 INFO:teuthology.orchestra.run.smithi148.stdout:(17/22): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2024-04-23T16:04:59.683 INFO:teuthology.orchestra.run.smithi148.stdout:(18/22): leveldb-1.22-1.el8.x86_64.rpm 1.3 MB/s | 181 kB 00:00 2024-04-23T16:04:59.701 INFO:teuthology.orchestra.run.smithi148.stdout:(19/22): liboath-2.6.2-3.el8.x86_64.rpm 540 kB/s | 59 kB 00:00 2024-04-23T16:04:59.716 INFO:teuthology.orchestra.run.smithi103.stdout:(13/22): lttng-ust-2.8.1-11.el8.x86_64.rpm 321 kB/s | 259 kB 00:00 2024-04-23T16:04:59.726 INFO:teuthology.orchestra.run.smithi148.stdout:(20/22): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2024-04-23T16:04:59.785 INFO:teuthology.orchestra.run.smithi188.stdout:(12/22): ceph-common-14.2.22-0.el8.x86_64.rpm 25 MB/s | 19 MB 00:00 2024-04-23T16:04:59.868 INFO:teuthology.orchestra.run.smithi148.stdout:(21/22): librados2-14.2.22-0.el8.x86_64.rpm 19 MB/s | 3.5 MB 00:00 2024-04-23T16:04:59.902 INFO:teuthology.orchestra.run.smithi188.stdout:(13/22): librdkafka-1.6.1-1.el8.x86_64.rpm 1.6 MB/s | 677 kB 00:00 2024-04-23T16:04:59.926 INFO:teuthology.orchestra.run.smithi148.stdout:(22/22): librbd1-14.2.22-0.el8.x86_64.rpm 7.6 MB/s | 1.7 MB 00:00 2024-04-23T16:04:59.928 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:04:59.928 INFO:teuthology.orchestra.run.smithi148.stdout:Total 25 MB/s | 45 MB 00:01 2024-04-23T16:04:59.941 INFO:teuthology.orchestra.run.smithi103.stdout:(14/22): python3-prettytable-0.7.2-14.el8.noarc 76 kB/s | 44 kB 00:00 2024-04-23T16:04:59.944 INFO:teuthology.orchestra.run.smithi188.stdout:(14/22): python3-prettytable-0.7.2-14.el8.noarc 276 kB/s | 44 kB 00:00 2024-04-23T16:04:59.966 INFO:teuthology.orchestra.run.smithi103.stdout:(15/22): librdkafka-1.6.1-1.el8.x86_64.rpm 623 kB/s | 677 kB 00:01 2024-04-23T16:04:59.968 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-04-23T16:04:59.978 INFO:teuthology.orchestra.run.smithi188.stdout:(15/22): lttng-ust-2.8.1-11.el8.x86_64.rpm 634 kB/s | 259 kB 00:00 2024-04-23T16:05:00.014 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-04-23T16:05:00.014 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-04-23T16:05:00.045 INFO:teuthology.orchestra.run.smithi188.stdout:(16/22): librabbitmq-0.9.0-5.el8.x86_64.rpm 330 kB/s | 47 kB 00:00 2024-04-23T16:05:00.067 INFO:teuthology.orchestra.run.smithi103.stdout:(16/22): librabbitmq-0.9.0-5.el8.x86_64.rpm 134 kB/s | 47 kB 00:00 2024-04-23T16:05:00.309 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-04-23T16:05:00.312 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-04-23T16:05:00.320 INFO:teuthology.orchestra.run.smithi188.stdout:(17/22): liboath-2.6.2-3.el8.x86_64.rpm 221 kB/s | 59 kB 00:00 2024-04-23T16:05:00.346 INFO:teuthology.orchestra.run.smithi188.stdout:(18/22): leveldb-1.22-1.el8.x86_64.rpm 493 kB/s | 181 kB 00:00 2024-04-23T16:05:00.359 INFO:teuthology.orchestra.run.smithi103.stdout:(17/22): leveldb-1.22-1.el8.x86_64.rpm 461 kB/s | 181 kB 00:00 2024-04-23T16:05:00.371 INFO:teuthology.orchestra.run.smithi188.stdout:(19/22): gperftools-libs-2.7-9.el8.x86_64.rpm 719 kB/s | 306 kB 00:00 2024-04-23T16:05:00.386 INFO:teuthology.orchestra.run.smithi103.stdout:(18/22): liboath-2.6.2-3.el8.x86_64.rpm 185 kB/s | 59 kB 00:00 2024-04-23T16:05:00.409 INFO:teuthology.orchestra.run.smithi103.stdout:(19/22): gperftools-libs-2.7-9.el8.x86_64.rpm 654 kB/s | 306 kB 00:00 2024-04-23T16:05:00.421 INFO:teuthology.orchestra.run.smithi188.stdout:(20/22): librbd1-14.2.22-0.el8.x86_64.rpm 35 MB/s | 1.7 MB 00:00 2024-04-23T16:05:00.446 INFO:teuthology.orchestra.run.smithi188.stdout:(21/22): libunwind-1.3.1-3.el8.x86_64.rpm 597 kB/s | 75 kB 00:00 2024-04-23T16:05:00.468 INFO:teuthology.orchestra.run.smithi103.stdout:(20/22): librbd1-14.2.22-0.el8.x86_64.rpm 30 MB/s | 1.7 MB 00:00 2024-04-23T16:05:00.493 INFO:teuthology.orchestra.run.smithi103.stdout:(21/22): libunwind-1.3.1-3.el8.x86_64.rpm 559 kB/s | 75 kB 00:00 2024-04-23T16:05:00.505 INFO:teuthology.orchestra.run.smithi188.stdout:(22/22): librados2-14.2.22-0.el8.x86_64.rpm 22 MB/s | 3.5 MB 00:00 2024-04-23T16:05:00.509 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:05:00.510 INFO:teuthology.orchestra.run.smithi188.stdout:Total 26 MB/s | 45 MB 00:01 2024-04-23T16:05:00.552 INFO:teuthology.orchestra.run.smithi103.stdout:(22/22): librados2-14.2.22-0.el8.x86_64.rpm 21 MB/s | 3.5 MB 00:00 2024-04-23T16:05:00.552 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-23T16:05:00.554 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:05:00.554 INFO:teuthology.orchestra.run.smithi103.stdout:Total 15 MB/s | 45 MB 00:02 2024-04-23T16:05:00.597 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-23T16:05:00.600 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-23T16:05:00.600 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-23T16:05:00.644 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-23T16:05:00.644 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-23T16:05:00.894 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-23T16:05:00.897 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-23T16:05:00.944 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-23T16:05:00.948 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-23T16:05:01.220 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-04-23T16:05:01.404 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/24 2024-04-23T16:05:01.625 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-04-23T16:05:01.911 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-23T16:05:01.987 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-04-23T16:05:02.119 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/24 2024-04-23T16:05:02.200 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : librados2-2:14.2.22-0.el8.x86_64 3/24 2024-04-23T16:05:02.278 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados2-2:14.2.22-0.el8.x86_64 3/24 2024-04-23T16:05:02.361 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-04-23T16:05:02.500 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rados-2:14.2.22-0.el8.x86_64 4/24 2024-04-23T16:05:02.684 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-23T16:05:02.737 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-23T16:05:02.750 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-04-23T16:05:02.893 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-23T16:05:02.929 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/24 2024-04-23T16:05:02.936 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librados2-2:14.2.22-0.el8.x86_64 3/24 2024-04-23T16:05:03.020 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:14.2.22-0.el8.x86_64 3/24 2024-04-23T16:05:03.116 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-04-23T16:05:03.129 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-04-23T16:05:03.184 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-04-23T16:05:03.219 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rados-2:14.2.22-0.el8.x86_64 4/24 2024-04-23T16:05:03.360 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-23T16:05:03.546 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-04-23T16:05:03.580 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-23T16:05:03.754 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librados2-2:14.2.22-0.el8.x86_64 3/24 2024-04-23T16:05:03.777 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/24 2024-04-23T16:05:03.819 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-04-23T16:05:03.849 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:14.2.22-0.el8.x86_64 3/24 2024-04-23T16:05:03.891 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-04-23T16:05:04.062 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rados-2:14.2.22-0.el8.x86_64 4/24 2024-04-23T16:05:04.212 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-23T16:05:04.418 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-23T16:05:04.482 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/24 2024-04-23T16:05:04.508 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-04-23T16:05:04.589 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-04-23T16:05:04.637 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-04-23T16:05:04.660 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-04-23T16:05:04.685 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-04-23T16:05:04.721 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-04-23T16:05:04.763 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : leveldb-1.22-1.el8.x86_64 9/24 2024-04-23T16:05:04.873 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : leveldb-1.22-1.el8.x86_64 9/24 2024-04-23T16:05:04.918 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 10/24 2024-04-23T16:05:05.044 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 10/24 2024-04-23T16:05:05.070 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cephfs-2:14.2.22-0.el8.x86_64 11/24 2024-04-23T16:05:05.253 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-04-23T16:05:05.284 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/24 2024-04-23T16:05:05.437 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-04-23T16:05:05.466 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rbd-2:14.2.22-0.el8.x86_64 13/24 2024-04-23T16:05:05.516 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-04-23T16:05:05.590 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cephfs-2:14.2.22-0.el8.x86_64 11/24 2024-04-23T16:05:05.641 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-04-23T16:05:05.691 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : leveldb-1.22-1.el8.x86_64 9/24 2024-04-23T16:05:05.720 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-04-23T16:05:05.838 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 10/24 2024-04-23T16:05:05.889 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-04-23T16:05:05.944 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/24 2024-04-23T16:05:06.023 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cephfs-2:14.2.22-0.el8.x86_64 11/24 2024-04-23T16:05:06.076 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rbd-2:14.2.22-0.el8.x86_64 13/24 2024-04-23T16:05:06.158 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/24 2024-04-23T16:05:06.223 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-04-23T16:05:06.235 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-04-23T16:05:06.294 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-04-23T16:05:06.369 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/24 2024-04-23T16:05:06.443 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rbd-2:14.2.22-0.el8.x86_64 13/24 2024-04-23T16:05:06.513 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/24 2024-04-23T16:05:06.542 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 18/24 2024-04-23T16:05:06.620 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-04-23T16:05:06.697 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-04-23T16:05:06.722 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/24 2024-04-23T16:05:06.887 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/24 2024-04-23T16:05:06.914 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/24 2024-04-23T16:05:07.069 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 18/24 2024-04-23T16:05:07.112 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/24 2024-04-23T16:05:07.331 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/24 2024-04-23T16:05:07.504 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 18/24 2024-04-23T16:05:08.626 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-23T16:05:08.816 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-23T16:05:09.181 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-23T16:05:09.402 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-23T16:05:09.442 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-23T16:05:09.604 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-23T16:05:09.715 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-04-23T16:05:09.820 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-23T16:05:09.822 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-04-23T16:05:09.966 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-04-23T16:05:10.017 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-23T16:05:10.298 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-04-23T16:05:10.389 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-04-23T16:05:10.453 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-23T16:05:10.576 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-04-23T16:05:10.738 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-04-23T16:05:10.841 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-04-23T16:05:11.023 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-04-23T16:05:28.884 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-04-23T16:05:28.884 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /sys 2024-04-23T16:05:28.884 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /proc 2024-04-23T16:05:28.884 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /dev 2024-04-23T16:05:28.884 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /run 2024-04-23T16:05:28.884 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /mnt 2024-04-23T16:05:28.884 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /var/tmp 2024-04-23T16:05:28.885 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /home 2024-04-23T16:05:28.885 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /tmp 2024-04-23T16:05:28.885 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /dev 2024-04-23T16:05:28.885 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:05:29.520 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-04-23T16:05:29.521 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /sys 2024-04-23T16:05:29.521 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /proc 2024-04-23T16:05:29.521 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /dev 2024-04-23T16:05:29.521 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /run 2024-04-23T16:05:29.521 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /mnt 2024-04-23T16:05:29.521 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /var/tmp 2024-04-23T16:05:29.521 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /home 2024-04-23T16:05:29.521 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /tmp 2024-04-23T16:05:29.521 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /dev 2024-04-23T16:05:29.522 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:05:29.571 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-04-23T16:05:29.590 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-04-23T16:05:29.591 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-04-23T16:05:29.797 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-04-23T16:05:29.797 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 24/24 2024-04-23T16:05:29.844 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-04-23T16:05:29.844 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /sys 2024-04-23T16:05:29.844 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /proc 2024-04-23T16:05:29.844 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /dev 2024-04-23T16:05:29.844 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /run 2024-04-23T16:05:29.844 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /mnt 2024-04-23T16:05:29.845 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /var/tmp 2024-04-23T16:05:29.845 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /home 2024-04-23T16:05:29.845 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /tmp 2024-04-23T16:05:29.845 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /dev 2024-04-23T16:05:29.845 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:05:30.203 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-04-23T16:05:30.223 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-04-23T16:05:30.223 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-04-23T16:05:30.475 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-04-23T16:05:30.475 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 24/24 2024-04-23T16:05:30.488 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 24/24 2024-04-23T16:05:30.488 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-base-2:14.2.22-0.el8.x86_64 1/24 2024-04-23T16:05:30.488 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-common-2:14.2.22-0.el8.x86_64 2/24 2024-04-23T16:05:30.488 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-radosgw-2:14.2.22-0.el8.x86_64 3/24 2024-04-23T16:05:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-selinux-2:14.2.22-0.el8.x86_64 4/24 2024-04-23T16:05:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-23T16:05:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libradosstriper1-2:14.2.22-0.el8.x86_64 6/24 2024-04-23T16:05:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librgw2-2:14.2.22-0.el8.x86_64 7/24 2024-04-23T16:05:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 8/24 2024-04-23T16:05:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cephfs-2:14.2.22-0.el8.x86_64 9/24 2024-04-23T16:05:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rados-2:14.2.22-0.el8.x86_64 10/24 2024-04-23T16:05:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rbd-2:14.2.22-0.el8.x86_64 11/24 2024-04-23T16:05:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-04-23T16:05:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 13/24 2024-04-23T16:05:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 14/24 2024-04-23T16:05:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 15/24 2024-04-23T16:05:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 16/24 2024-04-23T16:05:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 17/24 2024-04-23T16:05:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : leveldb-1.22-1.el8.x86_64 18/24 2024-04-23T16:05:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/24 2024-04-23T16:05:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/24 2024-04-23T16:05:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-2:14.2.22-0.el8.x86_64 21/24 2024-04-23T16:05:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 22/24 2024-04-23T16:05:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-2:14.2.22-0.el8.x86_64 23/24 2024-04-23T16:05:30.525 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-04-23T16:05:30.545 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-04-23T16:05:30.545 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-04-23T16:05:30.741 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-04-23T16:05:30.741 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 24/24 2024-04-23T16:05:31.129 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 24/24 2024-04-23T16:05:31.129 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-base-2:14.2.22-0.el8.x86_64 1/24 2024-04-23T16:05:31.129 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-common-2:14.2.22-0.el8.x86_64 2/24 2024-04-23T16:05:31.129 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-radosgw-2:14.2.22-0.el8.x86_64 3/24 2024-04-23T16:05:31.129 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-selinux-2:14.2.22-0.el8.x86_64 4/24 2024-04-23T16:05:31.129 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-23T16:05:31.129 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libradosstriper1-2:14.2.22-0.el8.x86_64 6/24 2024-04-23T16:05:31.130 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librgw2-2:14.2.22-0.el8.x86_64 7/24 2024-04-23T16:05:31.130 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 8/24 2024-04-23T16:05:31.130 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cephfs-2:14.2.22-0.el8.x86_64 9/24 2024-04-23T16:05:31.130 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rados-2:14.2.22-0.el8.x86_64 10/24 2024-04-23T16:05:31.130 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rbd-2:14.2.22-0.el8.x86_64 11/24 2024-04-23T16:05:31.130 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-04-23T16:05:31.130 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 13/24 2024-04-23T16:05:31.130 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 14/24 2024-04-23T16:05:31.130 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 15/24 2024-04-23T16:05:31.130 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 16/24 2024-04-23T16:05:31.130 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 17/24 2024-04-23T16:05:31.130 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : leveldb-1.22-1.el8.x86_64 18/24 2024-04-23T16:05:31.130 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/24 2024-04-23T16:05:31.130 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/24 2024-04-23T16:05:31.131 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:14.2.22-0.el8.x86_64 21/24 2024-04-23T16:05:31.131 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 22/24 2024-04-23T16:05:31.131 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:14.2.22-0.el8.x86_64 23/24 2024-04-23T16:05:31.347 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 24/24 2024-04-23T16:05:31.347 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:05:31.347 INFO:teuthology.orchestra.run.smithi148.stdout:Upgraded: 2024-04-23T16:05:31.347 INFO:teuthology.orchestra.run.smithi148.stdout: librados2-2:14.2.22-0.el8.x86_64 librbd1-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.347 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-04-23T16:05:31.347 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.347 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.348 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.348 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-selinux-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.348 INFO:teuthology.orchestra.run.smithi148.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-23T16:05:31.348 INFO:teuthology.orchestra.run.smithi148.stdout: leveldb-1.22-1.el8.x86_64 2024-04-23T16:05:31.348 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.348 INFO:teuthology.orchestra.run.smithi148.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-23T16:05:31.348 INFO:teuthology.orchestra.run.smithi148.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-23T16:05:31.348 INFO:teuthology.orchestra.run.smithi148.stdout: libradosstriper1-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.348 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-23T16:05:31.348 INFO:teuthology.orchestra.run.smithi148.stdout: librgw2-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.349 INFO:teuthology.orchestra.run.smithi148.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-23T16:05:31.349 INFO:teuthology.orchestra.run.smithi148.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-23T16:05:31.349 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.349 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.349 INFO:teuthology.orchestra.run.smithi148.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-23T16:05:31.349 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.349 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.349 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.349 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:05:31.349 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:05:31.407 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 24/24 2024-04-23T16:05:31.407 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-base-2:14.2.22-0.el8.x86_64 1/24 2024-04-23T16:05:31.407 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-common-2:14.2.22-0.el8.x86_64 2/24 2024-04-23T16:05:31.407 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-radosgw-2:14.2.22-0.el8.x86_64 3/24 2024-04-23T16:05:31.407 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-selinux-2:14.2.22-0.el8.x86_64 4/24 2024-04-23T16:05:31.407 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-23T16:05:31.407 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libradosstriper1-2:14.2.22-0.el8.x86_64 6/24 2024-04-23T16:05:31.408 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librgw2-2:14.2.22-0.el8.x86_64 7/24 2024-04-23T16:05:31.408 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 8/24 2024-04-23T16:05:31.408 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cephfs-2:14.2.22-0.el8.x86_64 9/24 2024-04-23T16:05:31.408 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rados-2:14.2.22-0.el8.x86_64 10/24 2024-04-23T16:05:31.408 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rbd-2:14.2.22-0.el8.x86_64 11/24 2024-04-23T16:05:31.408 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-04-23T16:05:31.408 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 13/24 2024-04-23T16:05:31.408 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 14/24 2024-04-23T16:05:31.408 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 15/24 2024-04-23T16:05:31.408 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 16/24 2024-04-23T16:05:31.408 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 17/24 2024-04-23T16:05:31.408 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : leveldb-1.22-1.el8.x86_64 18/24 2024-04-23T16:05:31.408 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/24 2024-04-23T16:05:31.408 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/24 2024-04-23T16:05:31.409 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:14.2.22-0.el8.x86_64 21/24 2024-04-23T16:05:31.409 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 22/24 2024-04-23T16:05:31.409 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:14.2.22-0.el8.x86_64 23/24 2024-04-23T16:05:31.606 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-test 2024-04-23T16:05:31.972 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 24/24 2024-04-23T16:05:31.973 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:05:31.973 INFO:teuthology.orchestra.run.smithi103.stdout:Upgraded: 2024-04-23T16:05:31.973 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-2:14.2.22-0.el8.x86_64 librbd1-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.973 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-23T16:05:31.973 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.973 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.973 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.973 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.973 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-23T16:05:31.973 INFO:teuthology.orchestra.run.smithi103.stdout: leveldb-1.22-1.el8.x86_64 2024-04-23T16:05:31.973 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.973 INFO:teuthology.orchestra.run.smithi103.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-23T16:05:31.973 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-23T16:05:31.974 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.974 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-23T16:05:31.974 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.974 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-23T16:05:31.974 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-23T16:05:31.974 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.974 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.974 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-23T16:05:31.974 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.974 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.974 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:31.974 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:05:31.975 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:05:32.273 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-test 2024-04-23T16:05:32.317 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 24/24 2024-04-23T16:05:32.317 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:05:32.317 INFO:teuthology.orchestra.run.smithi188.stdout:Upgraded: 2024-04-23T16:05:32.318 INFO:teuthology.orchestra.run.smithi188.stdout: librados2-2:14.2.22-0.el8.x86_64 librbd1-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:32.318 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-23T16:05:32.318 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:32.318 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:32.318 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:32.318 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:32.318 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-23T16:05:32.318 INFO:teuthology.orchestra.run.smithi188.stdout: leveldb-1.22-1.el8.x86_64 2024-04-23T16:05:32.318 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:32.318 INFO:teuthology.orchestra.run.smithi188.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-23T16:05:32.318 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-23T16:05:32.319 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:32.319 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-23T16:05:32.319 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:32.319 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-23T16:05:32.319 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-23T16:05:32.319 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:32.319 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:32.319 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-23T16:05:32.319 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:32.319 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:32.319 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:32.319 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:05:32.319 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:05:32.335 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:38 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:05:32.590 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-test 2024-04-23T16:05:33.001 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:39 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:05:33.323 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:00:38 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:05:33.985 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:05:33.986 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:05:33.986 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2024-04-23T16:05:33.986 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:05:33.986 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-04-23T16:05:33.986 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test x86_64 2:14.2.22-0.el8 ceph 35 M 2024-04-23T16:05:33.987 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-04-23T16:05:33.987 INFO:teuthology.orchestra.run.smithi148.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-23T16:05:33.987 INFO:teuthology.orchestra.run.smithi148.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-23T16:05:33.987 INFO:teuthology.orchestra.run.smithi148.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-23T16:05:33.987 INFO:teuthology.orchestra.run.smithi148.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-23T16:05:33.987 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:05:33.987 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-04-23T16:05:33.987 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:05:33.987 INFO:teuthology.orchestra.run.smithi148.stdout:Install 5 Packages 2024-04-23T16:05:33.987 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:05:33.988 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 36 M 2024-04-23T16:05:33.988 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 127 M 2024-04-23T16:05:33.988 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-04-23T16:05:34.231 INFO:teuthology.orchestra.run.smithi148.stdout:(1/5): jq-1.6-8.el8.x86_64.rpm 1.1 MB/s | 203 kB 00:00 2024-04-23T16:05:34.256 INFO:teuthology.orchestra.run.smithi148.stdout:(2/5): oniguruma-6.8.2-3.el8.x86_64.rpm 883 kB/s | 188 kB 00:00 2024-04-23T16:05:34.349 INFO:teuthology.orchestra.run.smithi148.stdout:(3/5): socat-1.7.4.1-1.el8.x86_64.rpm 2.7 MB/s | 323 kB 00:00 2024-04-23T16:05:34.374 INFO:teuthology.orchestra.run.smithi148.stdout:(4/5): xmlstarlet-1.6.1-20.el8.x86_64.rpm 604 kB/s | 71 kB 00:00 2024-04-23T16:05:34.631 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:05:34.632 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:05:34.632 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-04-23T16:05:34.632 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:05:34.633 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-23T16:05:34.633 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test x86_64 2:14.2.22-0.el8 ceph 35 M 2024-04-23T16:05:34.633 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-23T16:05:34.633 INFO:teuthology.orchestra.run.smithi103.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-23T16:05:34.633 INFO:teuthology.orchestra.run.smithi103.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-23T16:05:34.633 INFO:teuthology.orchestra.run.smithi103.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-23T16:05:34.633 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-23T16:05:34.633 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:05:34.633 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-23T16:05:34.633 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:05:34.633 INFO:teuthology.orchestra.run.smithi103.stdout:Install 5 Packages 2024-04-23T16:05:34.633 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:05:34.634 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 36 M 2024-04-23T16:05:34.634 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 127 M 2024-04-23T16:05:34.635 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-23T16:05:34.942 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:05:34.943 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:05:34.943 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-23T16:05:34.943 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:05:34.943 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-23T16:05:34.943 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test x86_64 2:14.2.22-0.el8 ceph 35 M 2024-04-23T16:05:34.943 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-23T16:05:34.943 INFO:teuthology.orchestra.run.smithi188.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-23T16:05:34.943 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-23T16:05:34.943 INFO:teuthology.orchestra.run.smithi188.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-23T16:05:34.943 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-23T16:05:34.944 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:05:34.944 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-23T16:05:34.944 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:05:34.944 INFO:teuthology.orchestra.run.smithi188.stdout:Install 5 Packages 2024-04-23T16:05:34.944 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:05:34.945 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 36 M 2024-04-23T16:05:34.945 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 127 M 2024-04-23T16:05:34.945 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-23T16:05:35.066 INFO:teuthology.orchestra.run.smithi148.stdout:(5/5): ceph-test-14.2.22-0.el8.x86_64.rpm 34 MB/s | 35 MB 00:01 2024-04-23T16:05:35.067 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:05:35.067 INFO:teuthology.orchestra.run.smithi148.stdout:Total 33 MB/s | 36 MB 00:01 2024-04-23T16:05:35.108 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-04-23T16:05:35.128 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-04-23T16:05:35.129 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-04-23T16:05:35.163 INFO:teuthology.orchestra.run.smithi103.stdout:(1/5): ceph-test-14.2.22-0.el8.x86_64.rpm 73 MB/s | 35 MB 00:00 2024-04-23T16:05:35.188 INFO:teuthology.orchestra.run.smithi103.stdout:(2/5): oniguruma-6.8.2-3.el8.x86_64.rpm 376 kB/s | 188 kB 00:00 2024-04-23T16:05:35.221 INFO:teuthology.orchestra.run.smithi103.stdout:(3/5): jq-1.6-8.el8.x86_64.rpm 380 kB/s | 203 kB 00:00 2024-04-23T16:05:35.238 INFO:teuthology.orchestra.run.smithi103.stdout:(4/5): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-04-23T16:05:35.352 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-04-23T16:05:35.355 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-04-23T16:05:35.422 INFO:teuthology.orchestra.run.smithi103.stdout:(5/5): socat-1.7.4.1-1.el8.x86_64.rpm 1.2 MB/s | 323 kB 00:00 2024-04-23T16:05:35.423 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:05:35.423 INFO:teuthology.orchestra.run.smithi103.stdout:Total 45 MB/s | 36 MB 00:00 2024-04-23T16:05:35.466 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-23T16:05:35.487 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-23T16:05:35.487 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-23T16:05:35.610 INFO:teuthology.orchestra.run.smithi188.stdout:(1/5): ceph-test-14.2.22-0.el8.x86_64.rpm 74 MB/s | 35 MB 00:00 2024-04-23T16:05:35.635 INFO:teuthology.orchestra.run.smithi188.stdout:(2/5): oniguruma-6.8.2-3.el8.x86_64.rpm 378 kB/s | 188 kB 00:00 2024-04-23T16:05:35.660 INFO:teuthology.orchestra.run.smithi188.stdout:(3/5): jq-1.6-8.el8.x86_64.rpm 388 kB/s | 203 kB 00:00 2024-04-23T16:05:35.694 INFO:teuthology.orchestra.run.smithi188.stdout:(4/5): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-04-23T16:05:35.708 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-23T16:05:35.711 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-23T16:05:35.771 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-04-23T16:05:35.844 INFO:teuthology.orchestra.run.smithi188.stdout:(5/5): socat-1.7.4.1-1.el8.x86_64.rpm 1.4 MB/s | 323 kB 00:00 2024-04-23T16:05:35.845 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:05:35.845 INFO:teuthology.orchestra.run.smithi188.stdout:Total 39 MB/s | 36 MB 00:00 2024-04-23T16:05:35.888 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-23T16:05:35.908 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-23T16:05:35.908 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-23T16:05:35.978 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/5 2024-04-23T16:05:36.096 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-23T16:05:36.130 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-23T16:05:36.134 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-23T16:05:36.156 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/5 2024-04-23T16:05:36.278 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-23T16:05:36.295 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/5 2024-04-23T16:05:36.379 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-23T16:05:36.518 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/5 2024-04-23T16:05:36.560 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-23T16:05:36.673 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-23T16:05:36.767 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/5 2024-04-23T16:05:36.776 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-23T16:05:36.966 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/5 2024-04-23T16:05:37.105 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-23T16:05:37.182 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-23T16:05:39.598 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : jq-1.6-8.el8.x86_64 4/5 2024-04-23T16:05:39.851 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-04-23T16:05:39.990 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : jq-1.6-8.el8.x86_64 4/5 2024-04-23T16:05:40.141 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-04-23T16:05:40.141 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-test-2:14.2.22-0.el8.x86_64 1/5 2024-04-23T16:05:40.141 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : jq-1.6-8.el8.x86_64 2/5 2024-04-23T16:05:40.141 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-23T16:05:40.141 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/5 2024-04-23T16:05:40.196 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-04-23T16:05:40.529 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-04-23T16:05:40.529 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-test-2:14.2.22-0.el8.x86_64 1/5 2024-04-23T16:05:40.529 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : jq-1.6-8.el8.x86_64 2/5 2024-04-23T16:05:40.529 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-23T16:05:40.529 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/5 2024-04-23T16:05:40.608 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 5/5 2024-04-23T16:05:40.608 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:05:40.608 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-04-23T16:05:40.608 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test-2:14.2.22-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-04-23T16:05:40.608 INFO:teuthology.orchestra.run.smithi148.stdout: oniguruma-6.8.2-3.el8.x86_64 socat-1.7.4.1-1.el8.x86_64 2024-04-23T16:05:40.608 INFO:teuthology.orchestra.run.smithi148.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-23T16:05:40.608 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:05:40.608 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:05:40.634 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : jq-1.6-8.el8.x86_64 4/5 2024-04-23T16:05:40.870 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-04-23T16:05:40.928 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph 2024-04-23T16:05:41.003 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 5/5 2024-04-23T16:05:41.003 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:05:41.003 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-23T16:05:41.004 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-2:14.2.22-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-04-23T16:05:41.004 INFO:teuthology.orchestra.run.smithi103.stdout: oniguruma-6.8.2-3.el8.x86_64 socat-1.7.4.1-1.el8.x86_64 2024-04-23T16:05:41.004 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-23T16:05:41.004 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:05:41.004 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:05:41.154 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-04-23T16:05:41.154 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-test-2:14.2.22-0.el8.x86_64 1/5 2024-04-23T16:05:41.154 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : jq-1.6-8.el8.x86_64 2/5 2024-04-23T16:05:41.154 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-23T16:05:41.154 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/5 2024-04-23T16:05:41.237 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph 2024-04-23T16:05:41.574 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 5/5 2024-04-23T16:05:41.574 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:05:41.574 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-23T16:05:41.574 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test-2:14.2.22-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-04-23T16:05:41.575 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma-6.8.2-3.el8.x86_64 socat-1.7.4.1-1.el8.x86_64 2024-04-23T16:05:41.575 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-23T16:05:41.575 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:05:41.575 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:05:41.658 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:47 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:05:41.799 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph 2024-04-23T16:05:41.968 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:48 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:05:42.537 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:00:47 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:05:43.318 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:05:43.322 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:05:43.322 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2024-04-23T16:05:43.322 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:05:43.322 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-04-23T16:05:43.322 INFO:teuthology.orchestra.run.smithi148.stdout: ceph x86_64 2:14.2.22-0.el8 ceph 6.3 k 2024-04-23T16:05:43.322 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-04-23T16:05:43.322 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds x86_64 2:14.2.22-0.el8 ceph 2.0 M 2024-04-23T16:05:43.322 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr x86_64 2:14.2.22-0.el8 ceph 1.6 M 2024-04-23T16:05:43.322 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon x86_64 2:14.2.22-0.el8 ceph 4.5 M 2024-04-23T16:05:43.322 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd x86_64 2:14.2.22-0.el8 ceph 17 M 2024-04-23T16:05:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-23T16:05:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-23T16:05:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-23T16:05:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-23T16:05:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-23T16:05:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-23T16:05:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-23T16:05:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-23T16:05:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-23T16:05:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-23T16:05:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-23T16:05:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-23T16:05:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-23T16:05:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-23T16:05:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-23T16:05:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-23T16:05:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-23T16:05:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-23T16:05:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-23T16:05:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-23T16:05:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-23T16:05:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-23T16:05:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-23T16:05:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-23T16:05:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-23T16:05:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-23T16:05:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:05:43.324 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-04-23T16:05:43.324 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:05:43.325 INFO:teuthology.orchestra.run.smithi148.stdout:Install 31 Packages 2024-04-23T16:05:43.325 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:05:43.325 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 28 M 2024-04-23T16:05:43.325 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 93 M 2024-04-23T16:05:43.325 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-04-23T16:05:43.591 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:05:43.594 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:05:43.594 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-23T16:05:43.595 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:05:43.595 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-23T16:05:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: ceph x86_64 2:14.2.22-0.el8 ceph 6.3 k 2024-04-23T16:05:43.595 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-23T16:05:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds x86_64 2:14.2.22-0.el8 ceph 2.0 M 2024-04-23T16:05:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr x86_64 2:14.2.22-0.el8 ceph 1.6 M 2024-04-23T16:05:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon x86_64 2:14.2.22-0.el8 ceph 4.5 M 2024-04-23T16:05:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd x86_64 2:14.2.22-0.el8 ceph 17 M 2024-04-23T16:05:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-23T16:05:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-23T16:05:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-23T16:05:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-23T16:05:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-23T16:05:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-23T16:05:43.595 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-23T16:05:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-23T16:05:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-23T16:05:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-23T16:05:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-23T16:05:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-23T16:05:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-23T16:05:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-23T16:05:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-23T16:05:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-23T16:05:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-23T16:05:43.596 INFO:teuthology.orchestra.run.smithi103.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-23T16:05:43.597 INFO:teuthology.orchestra.run.smithi103.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-23T16:05:43.597 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-23T16:05:43.597 INFO:teuthology.orchestra.run.smithi103.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-23T16:05:43.597 INFO:teuthology.orchestra.run.smithi103.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-23T16:05:43.597 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-23T16:05:43.597 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-23T16:05:43.597 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-23T16:05:43.597 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-23T16:05:43.597 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:05:43.597 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-23T16:05:43.597 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:05:43.597 INFO:teuthology.orchestra.run.smithi103.stdout:Install 31 Packages 2024-04-23T16:05:43.597 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:05:43.598 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 28 M 2024-04-23T16:05:43.598 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 93 M 2024-04-23T16:05:43.598 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-23T16:05:43.974 INFO:teuthology.orchestra.run.smithi148.stdout:(1/31): ceph-14.2.22-0.el8.x86_64.rpm 97 kB/s | 6.3 kB 00:00 2024-04-23T16:05:44.033 INFO:teuthology.orchestra.run.smithi148.stdout:(2/31): ceph-mds-14.2.22-0.el8.x86_64.rpm 16 MB/s | 2.0 MB 00:00 2024-04-23T16:05:44.157 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:05:44.160 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:05:44.160 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-04-23T16:05:44.160 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:05:44.160 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-23T16:05:44.160 INFO:teuthology.orchestra.run.smithi188.stdout: ceph x86_64 2:14.2.22-0.el8 ceph 6.3 k 2024-04-23T16:05:44.160 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-23T16:05:44.160 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds x86_64 2:14.2.22-0.el8 ceph 2.0 M 2024-04-23T16:05:44.160 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr x86_64 2:14.2.22-0.el8 ceph 1.6 M 2024-04-23T16:05:44.160 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon x86_64 2:14.2.22-0.el8 ceph 4.5 M 2024-04-23T16:05:44.160 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd x86_64 2:14.2.22-0.el8 ceph 17 M 2024-04-23T16:05:44.161 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-23T16:05:44.161 INFO:teuthology.orchestra.run.smithi188.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-23T16:05:44.161 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-23T16:05:44.161 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-23T16:05:44.161 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-23T16:05:44.161 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-23T16:05:44.161 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-23T16:05:44.161 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-23T16:05:44.161 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-23T16:05:44.161 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-23T16:05:44.161 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-23T16:05:44.161 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-23T16:05:44.161 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-23T16:05:44.162 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-23T16:05:44.162 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-23T16:05:44.162 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-23T16:05:44.162 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-23T16:05:44.162 INFO:teuthology.orchestra.run.smithi188.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-23T16:05:44.162 INFO:teuthology.orchestra.run.smithi188.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-23T16:05:44.162 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-23T16:05:44.162 INFO:teuthology.orchestra.run.smithi188.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-23T16:05:44.162 INFO:teuthology.orchestra.run.smithi188.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-23T16:05:44.162 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-23T16:05:44.162 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-23T16:05:44.162 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-23T16:05:44.162 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-23T16:05:44.162 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:05:44.163 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-23T16:05:44.163 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:05:44.163 INFO:teuthology.orchestra.run.smithi188.stdout:Install 31 Packages 2024-04-23T16:05:44.163 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:05:44.163 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 28 M 2024-04-23T16:05:44.164 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 93 M 2024-04-23T16:05:44.164 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-23T16:05:44.383 INFO:teuthology.orchestra.run.smithi103.stdout:(1/31): ceph-14.2.22-0.el8.x86_64.rpm 116 kB/s | 6.3 kB 00:00 2024-04-23T16:05:44.386 INFO:teuthology.orchestra.run.smithi188.stdout:(1/31): ceph-14.2.22-0.el8.x86_64.rpm 134 kB/s | 6.3 kB 00:00 2024-04-23T16:05:44.434 INFO:teuthology.orchestra.run.smithi103.stdout:(2/31): ceph-mgr-14.2.22-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-23T16:05:44.445 INFO:teuthology.orchestra.run.smithi188.stdout:(2/31): ceph-mgr-14.2.22-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-23T16:05:44.466 INFO:teuthology.orchestra.run.smithi148.stdout:(3/31): ceph-mon-14.2.22-0.el8.x86_64.rpm 9.1 MB/s | 4.5 MB 00:00 2024-04-23T16:05:44.484 INFO:teuthology.orchestra.run.smithi103.stdout:(3/31): ceph-mds-14.2.22-0.el8.x86_64.rpm 13 MB/s | 2.0 MB 00:00 2024-04-23T16:05:44.495 INFO:teuthology.orchestra.run.smithi188.stdout:(3/31): ceph-mds-14.2.22-0.el8.x86_64.rpm 13 MB/s | 2.0 MB 00:00 2024-04-23T16:05:44.508 INFO:teuthology.orchestra.run.smithi148.stdout:(4/31): ceph-mgr-14.2.22-0.el8.x86_64.rpm 2.6 MB/s | 1.6 MB 00:00 2024-04-23T16:05:44.587 INFO:teuthology.orchestra.run.smithi188.stdout:(4/31): ceph-mon-14.2.22-0.el8.x86_64.rpm 22 MB/s | 4.5 MB 00:00 2024-04-23T16:05:44.618 INFO:teuthology.orchestra.run.smithi103.stdout:(4/31): ceph-mon-14.2.22-0.el8.x86_64.rpm 19 MB/s | 4.5 MB 00:00 2024-04-23T16:05:44.701 INFO:teuthology.orchestra.run.smithi148.stdout:(5/31): ceph-osd-14.2.22-0.el8.x86_64.rpm 25 MB/s | 17 MB 00:00 2024-04-23T16:05:44.726 INFO:teuthology.orchestra.run.smithi148.stdout:(6/31): python3-mako-1.0.6-14.el8.noarch.rpm 607 kB/s | 157 kB 00:00 2024-04-23T16:05:44.784 INFO:teuthology.orchestra.run.smithi148.stdout:(7/31): python3-pytz-2017.2-11.el8.noarch.rpm 978 kB/s | 57 kB 00:00 2024-04-23T16:05:44.796 INFO:teuthology.orchestra.run.smithi188.stdout:(5/31): ceph-osd-14.2.22-0.el8.x86_64.rpm 48 MB/s | 17 MB 00:00 2024-04-23T16:05:44.810 INFO:teuthology.orchestra.run.smithi148.stdout:(8/31): python3-markupsafe-0.23-19.el8.x86_64.r 129 kB/s | 39 kB 00:00 2024-04-23T16:05:44.818 INFO:teuthology.orchestra.run.smithi103.stdout:(5/31): ceph-osd-14.2.22-0.el8.x86_64.rpm 44 MB/s | 17 MB 00:00 2024-04-23T16:05:44.821 INFO:teuthology.orchestra.run.smithi188.stdout:(6/31): python3-markupsafe-0.23-19.el8.x86_64.r 166 kB/s | 39 kB 00:00 2024-04-23T16:05:44.843 INFO:teuthology.orchestra.run.smithi103.stdout:(6/31): python3-markupsafe-0.23-19.el8.x86_64.r 172 kB/s | 39 kB 00:00 2024-04-23T16:05:44.846 INFO:teuthology.orchestra.run.smithi188.stdout:(7/31): python3-mako-1.0.6-14.el8.noarch.rpm 448 kB/s | 157 kB 00:00 2024-04-23T16:05:44.868 INFO:teuthology.orchestra.run.smithi103.stdout:(7/31): python3-mako-1.0.6-14.el8.noarch.rpm 408 kB/s | 157 kB 00:00 2024-04-23T16:05:44.885 INFO:teuthology.orchestra.run.smithi148.stdout:(9/31): python3-werkzeug-0.12.2-4.el8.noarch.rp 4.5 MB/s | 457 kB 00:00 2024-04-23T16:05:44.910 INFO:teuthology.orchestra.run.smithi148.stdout:(10/31): python3-pyOpenSSL-19.0.0-1.el8.noarch. 494 kB/s | 103 kB 00:00 2024-04-23T16:05:44.913 INFO:teuthology.orchestra.run.smithi188.stdout:(8/31): python3-pytz-2017.2-11.el8.noarch.rpm 621 kB/s | 57 kB 00:00 2024-04-23T16:05:44.963 INFO:teuthology.orchestra.run.smithi103.stdout:(8/31): python3-pytz-2017.2-11.el8.noarch.rpm 480 kB/s | 57 kB 00:00 2024-04-23T16:05:44.964 INFO:teuthology.orchestra.run.smithi188.stdout:(9/31): python3-werkzeug-0.12.2-4.el8.noarch.rp 3.8 MB/s | 457 kB 00:00 2024-04-23T16:05:45.002 INFO:teuthology.orchestra.run.smithi148.stdout:(11/31): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.3 MB/s | 249 kB 00:00 2024-04-23T16:05:45.023 INFO:teuthology.orchestra.run.smithi188.stdout:(10/31): python3-pyOpenSSL-19.0.0-1.el8.noarch. 458 kB/s | 103 kB 00:00 2024-04-23T16:05:45.023 INFO:teuthology.orchestra.run.smithi103.stdout:(9/31): python3-werkzeug-0.12.2-4.el8.noarch.rp 3.0 MB/s | 457 kB 00:00 2024-04-23T16:05:45.061 INFO:teuthology.orchestra.run.smithi103.stdout:(10/31): python3-pyOpenSSL-19.0.0-1.el8.noarch. 426 kB/s | 103 kB 00:00 2024-04-23T16:05:45.111 INFO:teuthology.orchestra.run.smithi148.stdout:(12/31): python3-pycparser-2.14-14.el8.noarch.r 544 kB/s | 109 kB 00:00 2024-04-23T16:05:45.156 INFO:teuthology.orchestra.run.smithi188.stdout:(11/31): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.0 MB/s | 249 kB 00:00 2024-04-23T16:05:45.206 INFO:teuthology.orchestra.run.smithi188.stdout:(12/31): python3-pycparser-2.14-14.el8.noarch.r 593 kB/s | 109 kB 00:00 2024-04-23T16:05:45.218 INFO:teuthology.orchestra.run.smithi148.stdout:(13/31): python3-cryptography-3.2.1-7.el8.x86_6 1.9 MB/s | 641 kB 00:00 2024-04-23T16:05:45.221 INFO:teuthology.orchestra.run.smithi103.stdout:(11/31): python3-cffi-1.11.5-6.el8.x86_64.rpm 967 kB/s | 249 kB 00:00 2024-04-23T16:05:45.257 INFO:teuthology.orchestra.run.smithi188.stdout:(13/31): python3-cryptography-3.2.1-7.el8.x86_6 2.1 MB/s | 641 kB 00:00 2024-04-23T16:05:45.270 INFO:teuthology.orchestra.run.smithi148.stdout:(14/31): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 163 kB/s | 44 kB 00:00 2024-04-23T16:05:45.279 INFO:teuthology.orchestra.run.smithi103.stdout:(12/31): python3-pycparser-2.14-14.el8.noarch.r 501 kB/s | 109 kB 00:00 2024-04-23T16:05:45.379 INFO:teuthology.orchestra.run.smithi103.stdout:(13/31): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 274 kB/s | 44 kB 00:00 2024-04-23T16:05:45.400 INFO:teuthology.orchestra.run.smithi188.stdout:(14/31): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 180 kB/s | 44 kB 00:00 2024-04-23T16:05:45.405 INFO:teuthology.orchestra.run.smithi103.stdout:(14/31): python3-cryptography-3.2.1-7.el8.x86_6 1.6 MB/s | 641 kB 00:00 2024-04-23T16:05:45.446 INFO:teuthology.orchestra.run.smithi103.stdout:(15/31): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.1 MB/s | 185 kB 00:00 2024-04-23T16:05:45.458 INFO:teuthology.orchestra.run.smithi188.stdout:(15/31): python3-beautifulsoup4-4.6.3-2.el8.1.n 738 kB/s | 185 kB 00:00 2024-04-23T16:05:45.482 INFO:teuthology.orchestra.run.smithi188.stdout:(16/31): python3-cheroot-8.5.2-1.el8.noarch.rpm 798 kB/s | 173 kB 00:00 2024-04-23T16:05:45.487 INFO:teuthology.orchestra.run.smithi148.stdout:(15/31): python3-beautifulsoup4-4.6.3-2.el8.1.n 494 kB/s | 185 kB 00:00 2024-04-23T16:05:45.497 INFO:teuthology.orchestra.run.smithi103.stdout:(16/31): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.4 MB/s | 173 kB 00:00 2024-04-23T16:05:45.508 INFO:teuthology.orchestra.run.smithi188.stdout:(17/31): python3-jaraco-6.2-6.el8.noarch.rpm 222 kB/s | 11 kB 00:00 2024-04-23T16:05:45.512 INFO:teuthology.orchestra.run.smithi148.stdout:(16/31): python3-cherrypy-18.4.0-1.el8.noarch.r 1.6 MB/s | 384 kB 00:00 2024-04-23T16:05:45.514 INFO:teuthology.orchestra.run.smithi103.stdout:(17/31): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2024-04-23T16:05:45.533 INFO:teuthology.orchestra.run.smithi188.stdout:(18/31): python3-jaraco-functools-2.0-4.el8.noa 349 kB/s | 18 kB 00:00 2024-04-23T16:05:45.547 INFO:teuthology.orchestra.run.smithi103.stdout:(18/31): python3-jaraco-functools-2.0-4.el8.noa 350 kB/s | 18 kB 00:00 2024-04-23T16:05:45.559 INFO:teuthology.orchestra.run.smithi188.stdout:(19/31): python3-logutils-0.3.5-11.el8.noarch.r 968 kB/s | 49 kB 00:00 2024-04-23T16:05:45.562 INFO:teuthology.orchestra.run.smithi148.stdout:(17/31): python3-jaraco-6.2-6.el8.noarch.rpm 148 kB/s | 11 kB 00:00 2024-04-23T16:05:45.573 INFO:teuthology.orchestra.run.smithi103.stdout:(19/31): python3-logutils-0.3.5-11.el8.noarch.r 830 kB/s | 49 kB 00:00 2024-04-23T16:05:45.584 INFO:teuthology.orchestra.run.smithi188.stdout:(20/31): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2024-04-23T16:05:45.596 INFO:teuthology.orchestra.run.smithi148.stdout:(18/31): python3-jaraco-functools-2.0-4.el8.noa 210 kB/s | 18 kB 00:00 2024-04-23T16:05:45.598 INFO:teuthology.orchestra.run.smithi103.stdout:(20/31): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2024-04-23T16:05:45.609 INFO:teuthology.orchestra.run.smithi188.stdout:(21/31): python3-cherrypy-18.4.0-1.el8.noarch.r 1.8 MB/s | 384 kB 00:00 2024-04-23T16:05:45.621 INFO:teuthology.orchestra.run.smithi148.stdout:(19/31): python3-cheroot-8.5.2-1.el8.noarch.rpm 421 kB/s | 173 kB 00:00 2024-04-23T16:05:45.623 INFO:teuthology.orchestra.run.smithi103.stdout:(21/31): python3-cherrypy-18.4.0-1.el8.noarch.r 1.7 MB/s | 384 kB 00:00 2024-04-23T16:05:45.634 INFO:teuthology.orchestra.run.smithi188.stdout:(22/31): python3-portend-2.6-1.el8.noarch.rpm 309 kB/s | 16 kB 00:00 2024-04-23T16:05:45.647 INFO:teuthology.orchestra.run.smithi148.stdout:(20/31): python3-logutils-0.3.5-11.el8.noarch.r 580 kB/s | 49 kB 00:00 2024-04-23T16:05:45.649 INFO:teuthology.orchestra.run.smithi103.stdout:(22/31): python3-portend-2.6-1.el8.noarch.rpm 309 kB/s | 16 kB 00:00 2024-04-23T16:05:45.660 INFO:teuthology.orchestra.run.smithi188.stdout:(23/31): python3-simplegeneric-0.8.1-17.el8.noa 372 kB/s | 19 kB 00:00 2024-04-23T16:05:45.672 INFO:teuthology.orchestra.run.smithi148.stdout:(21/31): python3-more-itertools-7.2.0-3.el8.noa 777 kB/s | 59 kB 00:00 2024-04-23T16:05:45.674 INFO:teuthology.orchestra.run.smithi103.stdout:(23/31): python3-pecan-1.3.2-9.el8.noarch.rpm 2.7 MB/s | 283 kB 00:00 2024-04-23T16:05:45.685 INFO:teuthology.orchestra.run.smithi188.stdout:(24/31): python3-singledispatch-3.4.0.3-18.el8. 476 kB/s | 24 kB 00:00 2024-04-23T16:05:45.699 INFO:teuthology.orchestra.run.smithi103.stdout:(24/31): python3-simplegeneric-0.8.1-17.el8.noa 247 kB/s | 19 kB 00:00 2024-04-23T16:05:45.710 INFO:teuthology.orchestra.run.smithi188.stdout:(25/31): python3-tempora-1.14.1-5.el8.noarch.rp 583 kB/s | 29 kB 00:00 2024-04-23T16:05:45.722 INFO:teuthology.orchestra.run.smithi148.stdout:(22/31): python3-portend-2.6-1.el8.noarch.rpm 207 kB/s | 16 kB 00:00 2024-04-23T16:05:45.736 INFO:teuthology.orchestra.run.smithi188.stdout:(26/31): python3-trustme-0.6.0-4.el8.noarch.rpm 549 kB/s | 27 kB 00:00 2024-04-23T16:05:45.747 INFO:teuthology.orchestra.run.smithi148.stdout:(23/31): python3-simplegeneric-0.8.1-17.el8.noa 249 kB/s | 19 kB 00:00 2024-04-23T16:05:45.758 INFO:teuthology.orchestra.run.smithi103.stdout:(25/31): python3-singledispatch-3.4.0.3-18.el8. 220 kB/s | 24 kB 00:00 2024-04-23T16:05:45.773 INFO:teuthology.orchestra.run.smithi148.stdout:(24/31): python3-pecan-1.3.2-9.el8.noarch.rpm 1.8 MB/s | 283 kB 00:00 2024-04-23T16:05:45.783 INFO:teuthology.orchestra.run.smithi103.stdout:(26/31): python3-tempora-1.14.1-5.el8.noarch.rp 268 kB/s | 29 kB 00:00 2024-04-23T16:05:45.786 INFO:teuthology.orchestra.run.smithi188.stdout:(27/31): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2024-04-23T16:05:45.798 INFO:teuthology.orchestra.run.smithi148.stdout:(25/31): python3-singledispatch-3.4.0.3-18.el8. 317 kB/s | 24 kB 00:00 2024-04-23T16:05:45.808 INFO:teuthology.orchestra.run.smithi103.stdout:(27/31): python3-trustme-0.6.0-4.el8.noarch.rpm 253 kB/s | 27 kB 00:00 2024-04-23T16:05:45.811 INFO:teuthology.orchestra.run.smithi188.stdout:(28/31): python3-waitress-1.4.3-1.el8.noarch.rp 2.5 MB/s | 255 kB 00:00 2024-04-23T16:05:45.823 INFO:teuthology.orchestra.run.smithi148.stdout:(26/31): python3-tempora-1.14.1-5.el8.noarch.rp 387 kB/s | 29 kB 00:00 2024-04-23T16:05:45.837 INFO:teuthology.orchestra.run.smithi188.stdout:(29/31): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2024-04-23T16:05:45.842 INFO:teuthology.orchestra.run.smithi103.stdout:(28/31): python3-waitress-1.4.3-1.el8.noarch.rp 3.0 MB/s | 255 kB 00:00 2024-04-23T16:05:45.848 INFO:teuthology.orchestra.run.smithi148.stdout:(27/31): python3-trustme-0.6.0-4.el8.noarch.rpm 365 kB/s | 27 kB 00:00 2024-04-23T16:05:45.862 INFO:teuthology.orchestra.run.smithi188.stdout:(30/31): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.9 MB/s | 251 kB 00:00 2024-04-23T16:05:45.879 INFO:teuthology.orchestra.run.smithi188.stdout:(31/31): python3-zc-lockfile-2.0-2.el8.noarch.r 342 kB/s | 23 kB 00:00 2024-04-23T16:05:45.883 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:05:45.883 INFO:teuthology.orchestra.run.smithi188.stdout:Total 17 MB/s | 28 MB 00:01 2024-04-23T16:05:45.884 INFO:teuthology.orchestra.run.smithi103.stdout:(29/31): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.3 MB/s | 251 kB 00:00 2024-04-23T16:05:45.909 INFO:teuthology.orchestra.run.smithi103.stdout:(30/31): python3-zc-lockfile-2.0-2.el8.noarch.r 343 kB/s | 23 kB 00:00 2024-04-23T16:05:45.915 INFO:teuthology.orchestra.run.smithi148.stdout:(28/31): python3-waitress-1.4.3-1.el8.noarch.rp 2.1 MB/s | 255 kB 00:00 2024-04-23T16:05:45.934 INFO:teuthology.orchestra.run.smithi103.stdout:(31/31): python3-webtest-2.0.33-1.el8.noarch.rp 681 kB/s | 85 kB 00:00 2024-04-23T16:05:45.936 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:05:45.936 INFO:teuthology.orchestra.run.smithi103.stdout:Total 12 MB/s | 28 MB 00:02 2024-04-23T16:05:45.941 INFO:teuthology.orchestra.run.smithi148.stdout:(29/31): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.1 MB/s | 251 kB 00:00 2024-04-23T16:05:45.965 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-23T16:05:45.966 INFO:teuthology.orchestra.run.smithi148.stdout:(30/31): python3-webtest-2.0.33-1.el8.noarch.rp 729 kB/s | 85 kB 00:00 2024-04-23T16:05:45.991 INFO:teuthology.orchestra.run.smithi148.stdout:(31/31): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2024-04-23T16:05:45.995 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:05:45.996 INFO:teuthology.orchestra.run.smithi148.stdout:Total 11 MB/s | 28 MB 00:02 2024-04-23T16:05:46.005 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-23T16:05:46.005 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-23T16:05:46.018 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-23T16:05:46.057 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-23T16:05:46.057 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-23T16:05:46.077 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-04-23T16:05:46.117 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-04-23T16:05:46.117 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-04-23T16:05:46.307 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-23T16:05:46.310 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-23T16:05:46.367 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-23T16:05:46.370 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-23T16:05:46.425 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-04-23T16:05:46.428 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-04-23T16:05:47.373 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-23T16:05:47.471 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-23T16:05:47.492 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-04-23T16:05:47.597 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/31 2024-04-23T16:05:47.686 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/31 2024-04-23T16:05:47.723 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/31 2024-04-23T16:05:47.848 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/31 2024-04-23T16:05:47.961 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/31 2024-04-23T16:05:47.971 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/31 2024-04-23T16:05:48.117 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/31 2024-04-23T16:05:48.242 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/31 2024-04-23T16:05:48.246 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/31 2024-04-23T16:05:48.357 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/31 2024-04-23T16:05:48.468 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/31 2024-04-23T16:05:48.470 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/31 2024-04-23T16:05:48.540 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/31 2024-04-23T16:05:48.680 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/31 2024-04-23T16:05:48.688 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/31 2024-04-23T16:05:48.692 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/31 2024-04-23T16:05:48.848 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/31 2024-04-23T16:05:48.880 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/31 2024-04-23T16:05:48.885 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/31 2024-04-23T16:05:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/31 2024-04-23T16:05:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/31 2024-04-23T16:05:49.087 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/31 2024-04-23T16:05:49.211 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/31 2024-04-23T16:05:49.240 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/31 2024-04-23T16:05:49.242 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/31 2024-04-23T16:05:49.447 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/31 2024-04-23T16:05:49.462 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/31 2024-04-23T16:05:49.471 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/31 2024-04-23T16:05:49.690 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/31 2024-04-23T16:05:49.701 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/31 2024-04-23T16:05:49.708 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/31 2024-04-23T16:05:49.939 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/31 2024-04-23T16:05:49.941 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/31 2024-04-23T16:05:49.962 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/31 2024-04-23T16:05:50.180 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/31 2024-04-23T16:05:50.219 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/31 2024-04-23T16:05:50.351 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/31 2024-04-23T16:05:50.558 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/31 2024-04-23T16:05:50.581 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/31 2024-04-23T16:05:50.628 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/31 2024-04-23T16:05:50.776 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/31 2024-04-23T16:05:50.812 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/31 2024-04-23T16:05:50.866 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/31 2024-04-23T16:05:51.012 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/31 2024-04-23T16:05:51.073 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/31 2024-04-23T16:05:51.074 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/31 2024-04-23T16:05:51.249 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/31 2024-04-23T16:05:51.295 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/31 2024-04-23T16:05:51.345 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/31 2024-04-23T16:05:51.485 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/31 2024-04-23T16:05:51.597 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/31 2024-04-23T16:05:51.617 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/31 2024-04-23T16:05:51.809 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/31 2024-04-23T16:05:51.858 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/31 2024-04-23T16:05:51.931 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/31 2024-04-23T16:05:52.039 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/31 2024-04-23T16:05:52.046 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/31 2024-04-23T16:05:52.185 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/31 2024-04-23T16:05:52.228 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/31 2024-04-23T16:05:52.290 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/31 2024-04-23T16:05:52.323 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/31 2024-04-23T16:05:52.405 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/31 2024-04-23T16:05:52.501 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/31 2024-04-23T16:05:52.564 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/31 2024-04-23T16:05:52.664 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/31 2024-04-23T16:05:52.744 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/31 2024-04-23T16:05:52.764 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/31 2024-04-23T16:05:52.871 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/31 2024-04-23T16:05:52.967 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/31 2024-04-23T16:05:52.983 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/31 2024-04-23T16:05:53.120 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/31 2024-04-23T16:05:53.199 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/31 2024-04-23T16:05:53.309 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/31 2024-04-23T16:05:53.449 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/31 2024-04-23T16:05:53.568 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/31 2024-04-23T16:05:53.746 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/31 2024-04-23T16:05:53.896 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/31 2024-04-23T16:05:54.004 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-04-23T16:05:54.016 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/31 2024-04-23T16:05:54.126 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-04-23T16:05:54.267 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-04-23T16:05:55.376 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-04-23T16:05:55.495 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-04-23T16:05:55.554 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-04-23T16:05:55.632 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-04-23T16:05:55.683 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-04-23T16:05:55.808 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-04-23T16:05:55.939 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-04-23T16:05:56.074 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-04-23T16:05:56.101 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-04-23T16:05:56.206 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-04-23T16:05:56.250 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-04-23T16:05:56.288 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-04-23T16:05:56.379 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-04-23T16:05:56.434 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-04-23T16:05:56.467 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-04-23T16:05:56.485 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-04-23T16:05:56.576 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-04-23T16:05:56.608 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-2:14.2.22-0.el8.x86_64 31/31 2024-04-23T16:05:56.617 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-04-23T16:05:56.635 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-04-23T16:05:56.758 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-2:14.2.22-0.el8.x86_64 31/31 2024-04-23T16:05:56.771 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-04-23T16:05:56.788 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-04-23T16:05:56.953 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-2:14.2.22-0.el8.x86_64 31/31 2024-04-23T16:05:57.039 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-2:14.2.22-0.el8.x86_64 31/31 2024-04-23T16:05:57.039 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-2:14.2.22-0.el8.x86_64 1/31 2024-04-23T16:05:57.039 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mds-2:14.2.22-0.el8.x86_64 2/31 2024-04-23T16:05:57.039 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-2:14.2.22-0.el8.x86_64 3/31 2024-04-23T16:05:57.039 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mon-2:14.2.22-0.el8.x86_64 4/31 2024-04-23T16:05:57.039 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-osd-2:14.2.22-0.el8.x86_64 5/31 2024-04-23T16:05:57.040 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 6/31 2024-04-23T16:05:57.040 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 7/31 2024-04-23T16:05:57.040 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 8/31 2024-04-23T16:05:57.040 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 9/31 2024-04-23T16:05:57.040 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 10/31 2024-04-23T16:05:57.040 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 11/31 2024-04-23T16:05:57.040 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 12/31 2024-04-23T16:05:57.040 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 13/31 2024-04-23T16:05:57.040 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 14/31 2024-04-23T16:05:57.040 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 15/31 2024-04-23T16:05:57.040 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 16/31 2024-04-23T16:05:57.040 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 17/31 2024-04-23T16:05:57.040 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 18/31 2024-04-23T16:05:57.041 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 19/31 2024-04-23T16:05:57.041 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 20/31 2024-04-23T16:05:57.041 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 21/31 2024-04-23T16:05:57.041 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 22/31 2024-04-23T16:05:57.041 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-portend-2.6-1.el8.noarch 23/31 2024-04-23T16:05:57.041 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 24/31 2024-04-23T16:05:57.041 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 25/31 2024-04-23T16:05:57.041 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 26/31 2024-04-23T16:05:57.041 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 27/31 2024-04-23T16:05:57.041 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 28/31 2024-04-23T16:05:57.041 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 29/31 2024-04-23T16:05:57.041 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 30/31 2024-04-23T16:05:57.169 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-2:14.2.22-0.el8.x86_64 31/31 2024-04-23T16:05:57.170 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-2:14.2.22-0.el8.x86_64 1/31 2024-04-23T16:05:57.170 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mds-2:14.2.22-0.el8.x86_64 2/31 2024-04-23T16:05:57.170 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-2:14.2.22-0.el8.x86_64 3/31 2024-04-23T16:05:57.170 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mon-2:14.2.22-0.el8.x86_64 4/31 2024-04-23T16:05:57.170 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-osd-2:14.2.22-0.el8.x86_64 5/31 2024-04-23T16:05:57.170 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 6/31 2024-04-23T16:05:57.170 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 7/31 2024-04-23T16:05:57.170 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 8/31 2024-04-23T16:05:57.170 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 9/31 2024-04-23T16:05:57.170 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 10/31 2024-04-23T16:05:57.170 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 11/31 2024-04-23T16:05:57.170 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 12/31 2024-04-23T16:05:57.171 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 13/31 2024-04-23T16:05:57.171 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 14/31 2024-04-23T16:05:57.171 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 15/31 2024-04-23T16:05:57.171 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 16/31 2024-04-23T16:05:57.171 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 17/31 2024-04-23T16:05:57.171 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 18/31 2024-04-23T16:05:57.171 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 19/31 2024-04-23T16:05:57.171 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 20/31 2024-04-23T16:05:57.171 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 21/31 2024-04-23T16:05:57.171 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 22/31 2024-04-23T16:05:57.171 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-portend-2.6-1.el8.noarch 23/31 2024-04-23T16:05:57.171 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 24/31 2024-04-23T16:05:57.171 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 25/31 2024-04-23T16:05:57.172 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 26/31 2024-04-23T16:05:57.172 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 27/31 2024-04-23T16:05:57.172 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 28/31 2024-04-23T16:05:57.172 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 29/31 2024-04-23T16:05:57.172 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 30/31 2024-04-23T16:05:57.343 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-2:14.2.22-0.el8.x86_64 31/31 2024-04-23T16:05:57.343 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-2:14.2.22-0.el8.x86_64 1/31 2024-04-23T16:05:57.343 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mds-2:14.2.22-0.el8.x86_64 2/31 2024-04-23T16:05:57.343 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-2:14.2.22-0.el8.x86_64 3/31 2024-04-23T16:05:57.344 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mon-2:14.2.22-0.el8.x86_64 4/31 2024-04-23T16:05:57.344 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-osd-2:14.2.22-0.el8.x86_64 5/31 2024-04-23T16:05:57.344 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 6/31 2024-04-23T16:05:57.344 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 7/31 2024-04-23T16:05:57.344 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 8/31 2024-04-23T16:05:57.344 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 9/31 2024-04-23T16:05:57.344 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 10/31 2024-04-23T16:05:57.344 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 11/31 2024-04-23T16:05:57.344 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 12/31 2024-04-23T16:05:57.344 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 13/31 2024-04-23T16:05:57.344 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 14/31 2024-04-23T16:05:57.344 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 15/31 2024-04-23T16:05:57.344 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 16/31 2024-04-23T16:05:57.344 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 17/31 2024-04-23T16:05:57.345 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 18/31 2024-04-23T16:05:57.345 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 19/31 2024-04-23T16:05:57.345 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 20/31 2024-04-23T16:05:57.345 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 21/31 2024-04-23T16:05:57.345 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 22/31 2024-04-23T16:05:57.345 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-portend-2.6-1.el8.noarch 23/31 2024-04-23T16:05:57.345 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 24/31 2024-04-23T16:05:57.345 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 25/31 2024-04-23T16:05:57.345 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 26/31 2024-04-23T16:05:57.345 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 27/31 2024-04-23T16:05:57.345 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 28/31 2024-04-23T16:05:57.345 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 29/31 2024-04-23T16:05:57.345 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 30/31 2024-04-23T16:05:57.895 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 31/31 2024-04-23T16:05:57.895 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:05:57.895 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-23T16:05:57.895 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-23T16:05:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-23T16:05:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-23T16:05:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-23T16:05:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-23T16:05:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-23T16:05:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-23T16:05:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-23T16:05:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-23T16:05:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-23T16:05:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-23T16:05:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-23T16:05:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-23T16:05:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend-2.6-1.el8.noarch 2024-04-23T16:05:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-23T16:05:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-23T16:05:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-23T16:05:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-23T16:05:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-23T16:05:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-23T16:05:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-23T16:05:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-23T16:05:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-23T16:05:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-23T16:05:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-23T16:05:57.898 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-23T16:05:57.898 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:05:57.898 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:05:57.929 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 31/31 2024-04-23T16:05:57.929 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:05:57.930 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-04-23T16:05:57.930 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:57.930 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:57.930 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:57.930 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:57.931 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:57.931 INFO:teuthology.orchestra.run.smithi148.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-23T16:05:57.931 INFO:teuthology.orchestra.run.smithi148.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-23T16:05:57.931 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-23T16:05:57.931 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-23T16:05:57.931 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-23T16:05:57.931 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-23T16:05:57.931 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-23T16:05:57.931 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-23T16:05:57.931 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-23T16:05:57.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-23T16:05:57.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-23T16:05:57.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-23T16:05:57.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-23T16:05:57.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-portend-2.6-1.el8.noarch 2024-04-23T16:05:57.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-23T16:05:57.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-23T16:05:57.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-23T16:05:57.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-23T16:05:57.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-23T16:05:57.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-23T16:05:57.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-23T16:05:57.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-23T16:05:57.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-23T16:05:57.933 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-23T16:05:57.933 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-23T16:05:57.933 INFO:teuthology.orchestra.run.smithi148.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-23T16:05:57.933 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:05:57.933 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:05:58.172 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-base 2024-04-23T16:05:58.233 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-base 2024-04-23T16:05:58.299 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 31/31 2024-04-23T16:05:58.299 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:05:58.299 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-23T16:05:58.300 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:58.300 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:58.300 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:58.300 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:58.300 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-2:14.2.22-0.el8.x86_64 2024-04-23T16:05:58.300 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-23T16:05:58.300 INFO:teuthology.orchestra.run.smithi103.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-23T16:05:58.300 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-23T16:05:58.300 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-23T16:05:58.300 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-23T16:05:58.300 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-23T16:05:58.300 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-23T16:05:58.300 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-23T16:05:58.300 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-23T16:05:58.301 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-23T16:05:58.301 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-23T16:05:58.301 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-23T16:05:58.301 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-23T16:05:58.301 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend-2.6-1.el8.noarch 2024-04-23T16:05:58.301 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-23T16:05:58.301 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-23T16:05:58.301 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-23T16:05:58.301 INFO:teuthology.orchestra.run.smithi103.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-23T16:05:58.301 INFO:teuthology.orchestra.run.smithi103.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-23T16:05:58.301 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-23T16:05:58.301 INFO:teuthology.orchestra.run.smithi103.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-23T16:05:58.301 INFO:teuthology.orchestra.run.smithi103.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-23T16:05:58.301 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-23T16:05:58.301 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-23T16:05:58.302 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-23T16:05:58.302 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-23T16:05:58.302 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:05:58.302 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:05:58.613 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-base 2024-04-23T16:05:58.906 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:03 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:05:58.969 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:04 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:05:59.341 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:06 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:06:00.496 INFO:teuthology.orchestra.run.smithi188.stdout:Package ceph-base-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:06:00.545 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:06:00.920 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-base-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:06:00.921 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-04-23T16:06:00.922 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:06:00.922 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr 2024-04-23T16:06:00.922 INFO:teuthology.orchestra.run.smithi148.stdout:Package ceph-base-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:06:00.923 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:06:00.923 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2024-04-23T16:06:00.923 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:06:00.923 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr 2024-04-23T16:06:00.963 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:06:00.964 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-23T16:06:00.964 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:06:01.088 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr 2024-04-23T16:06:01.651 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:07 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:06:01.651 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:06 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:06:01.817 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:08 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:06:03.226 INFO:teuthology.orchestra.run.smithi188.stdout:Package ceph-mgr-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:06:03.258 INFO:teuthology.orchestra.run.smithi148.stdout:Package ceph-mgr-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:06:03.274 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:06:03.276 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-04-23T16:06:03.276 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:06:03.309 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:06:03.310 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2024-04-23T16:06:03.310 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:06:03.383 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-dashboard 2024-04-23T16:06:03.413 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-mgr-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:06:03.423 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-dashboard 2024-04-23T16:06:03.461 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:06:03.463 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-23T16:06:03.463 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:06:03.582 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-dashboard 2024-04-23T16:06:04.104 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:09 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:06:04.154 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:10 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:06:04.302 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:11 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:06:05.758 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:06:05.758 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:05.758 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-04-23T16:06:05.758 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:05.759 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-23T16:06:05.759 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard noarch 2:14.2.22-0.el8 ceph-noarch 3.7 M 2024-04-23T16:06:05.759 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-23T16:06:05.759 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards noarch 2:14.2.22-0.el8 ceph-noarch 24 k 2024-04-23T16:06:05.759 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-23T16:06:05.759 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-23T16:06:05.759 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-23T16:06:05.759 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:05.759 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-23T16:06:05.759 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:05.759 INFO:teuthology.orchestra.run.smithi188.stdout:Install 5 Packages 2024-04-23T16:06:05.759 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:05.760 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 4.0 M 2024-04-23T16:06:05.760 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 17 M 2024-04-23T16:06:05.760 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-23T16:06:05.765 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:06:05.766 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:05.766 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2024-04-23T16:06:05.766 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:05.767 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-04-23T16:06:05.767 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard noarch 2:14.2.22-0.el8 ceph-noarch 3.7 M 2024-04-23T16:06:05.767 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-04-23T16:06:05.767 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-grafana-dashboards noarch 2:14.2.22-0.el8 ceph-noarch 24 k 2024-04-23T16:06:05.767 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-23T16:06:05.767 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-23T16:06:05.767 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-23T16:06:05.767 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:05.767 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-04-23T16:06:05.767 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:05.767 INFO:teuthology.orchestra.run.smithi148.stdout:Install 5 Packages 2024-04-23T16:06:05.767 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:05.768 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 4.0 M 2024-04-23T16:06:05.768 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 17 M 2024-04-23T16:06:05.768 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-04-23T16:06:05.932 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:06:05.933 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:05.933 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-23T16:06:05.933 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:05.933 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-23T16:06:05.933 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard noarch 2:14.2.22-0.el8 ceph-noarch 3.7 M 2024-04-23T16:06:05.933 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-23T16:06:05.933 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards noarch 2:14.2.22-0.el8 ceph-noarch 24 k 2024-04-23T16:06:05.933 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-23T16:06:05.933 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-23T16:06:05.933 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-23T16:06:05.933 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:05.933 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-23T16:06:05.934 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:05.934 INFO:teuthology.orchestra.run.smithi103.stdout:Install 5 Packages 2024-04-23T16:06:05.934 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:05.934 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 4.0 M 2024-04-23T16:06:05.934 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 17 M 2024-04-23T16:06:05.934 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-23T16:06:06.228 INFO:teuthology.orchestra.run.smithi188.stdout:(1/5): ceph-grafana-dashboards-14.2.22-0.el8.no 427 kB/s | 24 kB 00:00 2024-04-23T16:06:06.337 INFO:teuthology.orchestra.run.smithi188.stdout:(2/5): ceph-mgr-dashboard-14.2.22-0.el8.noarch. 23 MB/s | 3.7 MB 00:00 2024-04-23T16:06:06.339 INFO:teuthology.orchestra.run.smithi148.stdout:(1/5): ceph-grafana-dashboards-14.2.22-0.el8.no 544 kB/s | 24 kB 00:00 2024-04-23T16:06:06.362 INFO:teuthology.orchestra.run.smithi188.stdout:(3/5): python3-jwt-1.6.1-2.el8.noarch.rpm 224 kB/s | 43 kB 00:00 2024-04-23T16:06:06.404 INFO:teuthology.orchestra.run.smithi188.stdout:(4/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 191 kB/s | 34 kB 00:00 2024-04-23T16:06:06.431 INFO:teuthology.orchestra.run.smithi148.stdout:(2/5): ceph-mgr-dashboard-14.2.22-0.el8.noarch. 27 MB/s | 3.7 MB 00:00 2024-04-23T16:06:06.448 INFO:teuthology.orchestra.run.smithi148.stdout:(3/5): python3-jwt-1.6.1-2.el8.noarch.rpm 278 kB/s | 43 kB 00:00 2024-04-23T16:06:06.524 INFO:teuthology.orchestra.run.smithi103.stdout:(1/5): ceph-grafana-dashboards-14.2.22-0.el8.no 496 kB/s | 24 kB 00:00 2024-04-23T16:06:06.537 INFO:teuthology.orchestra.run.smithi188.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 983 kB/s | 196 kB 00:00 2024-04-23T16:06:06.540 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:06.540 INFO:teuthology.orchestra.run.smithi188.stdout:Total 5.2 MB/s | 4.0 MB 00:00 2024-04-23T16:06:06.552 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-23T16:06:06.562 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-23T16:06:06.562 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-23T16:06:06.599 INFO:teuthology.orchestra.run.smithi103.stdout:(2/5): ceph-mgr-dashboard-14.2.22-0.el8.noarch. 30 MB/s | 3.7 MB 00:00 2024-04-23T16:06:06.606 INFO:teuthology.orchestra.run.smithi148.stdout:(4/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 126 kB/s | 34 kB 00:00 2024-04-23T16:06:06.626 INFO:teuthology.orchestra.run.smithi103.stdout:(3/5): python3-jwt-1.6.1-2.el8.noarch.rpm 286 kB/s | 43 kB 00:00 2024-04-23T16:06:06.658 INFO:teuthology.orchestra.run.smithi103.stdout:(4/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 251 kB/s | 34 kB 00:00 2024-04-23T16:06:06.725 INFO:teuthology.orchestra.run.smithi103.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 1.5 MB/s | 196 kB 00:00 2024-04-23T16:06:06.726 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:06.727 INFO:teuthology.orchestra.run.smithi103.stdout:Total 5.1 MB/s | 4.0 MB 00:00 2024-04-23T16:06:06.733 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-23T16:06:06.736 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-23T16:06:06.737 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-23T16:06:06.746 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-23T16:06:06.746 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-23T16:06:06.873 INFO:teuthology.orchestra.run.smithi148.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 445 kB/s | 196 kB 00:00 2024-04-23T16:06:06.875 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:06.875 INFO:teuthology.orchestra.run.smithi148.stdout:Total 3.6 MB/s | 4.0 MB 00:01 2024-04-23T16:06:06.887 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-04-23T16:06:06.898 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-04-23T16:06:06.898 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-04-23T16:06:06.903 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-23T16:06:06.906 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-23T16:06:06.995 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-23T16:06:07.070 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-04-23T16:06:07.074 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-04-23T16:06:07.158 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-23T16:06:07.204 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-23T16:06:07.359 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-04-23T16:06:07.359 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-23T16:06:07.440 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-23T16:06:07.568 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-23T16:06:07.647 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-04-23T16:06:07.653 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-23T16:06:07.811 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-23T16:06:07.854 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-04-23T16:06:08.007 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-04-23T16:06:08.553 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 4/5 2024-04-23T16:06:08.754 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 4/5 2024-04-23T16:06:08.841 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-04-23T16:06:09.006 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-04-23T16:06:09.052 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 4/5 2024-04-23T16:06:09.125 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-04-23T16:06:09.125 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 1/5 2024-04-23T16:06:09.125 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2/5 2024-04-23T16:06:09.125 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-04-23T16:06:09.126 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-23T16:06:09.320 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-04-23T16:06:09.320 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 1/5 2024-04-23T16:06:09.320 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2/5 2024-04-23T16:06:09.320 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-04-23T16:06:09.320 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-23T16:06:09.713 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-23T16:06:09.713 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:09.713 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-23T16:06:09.713 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 2024-04-23T16:06:09.713 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2024-04-23T16:06:09.713 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-23T16:06:09.713 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-23T16:06:09.713 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-23T16:06:09.714 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:09.714 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:06:09.982 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-23T16:06:09.982 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:09.982 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-23T16:06:09.982 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 2024-04-23T16:06:09.982 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2024-04-23T16:06:09.982 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-23T16:06:09.982 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-23T16:06:09.983 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-23T16:06:09.983 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:09.983 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:06:10.239 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-04-23T16:06:10.533 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-04-23T16:06:10.533 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 1/5 2024-04-23T16:06:10.533 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2/5 2024-04-23T16:06:10.533 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-04-23T16:06:10.533 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-23T16:06:10.850 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-23T16:06:10.903 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-23T16:06:10.903 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:10.903 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-04-23T16:06:10.903 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 2024-04-23T16:06:10.903 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2024-04-23T16:06:10.903 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-23T16:06:10.903 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-23T16:06:10.903 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-23T16:06:10.903 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:10.903 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:06:11.038 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-23T16:06:11.507 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-23T16:06:11.585 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:16 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:06:11.766 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:17 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:06:12.236 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:19 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:06:13.220 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:06:13.221 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================ 2024-04-23T16:06:13.221 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-04-23T16:06:13.221 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================ 2024-04-23T16:06:13.221 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-23T16:06:13.221 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local noarch 2:14.2.22-0.el8 ceph-noarch 1.1 M 2024-04-23T16:06:13.221 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-23T16:06:13.222 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-23T16:06:13.222 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-23T16:06:13.222 INFO:teuthology.orchestra.run.smithi188.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-23T16:06:13.222 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-23T16:06:13.222 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-23T16:06:13.222 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-23T16:06:13.222 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-23T16:06:13.222 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:13.222 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-23T16:06:13.222 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================ 2024-04-23T16:06:13.222 INFO:teuthology.orchestra.run.smithi188.stdout:Install 8 Packages 2024-04-23T16:06:13.222 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:13.223 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 30 M 2024-04-23T16:06:13.223 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 172 M 2024-04-23T16:06:13.223 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-23T16:06:13.407 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:06:13.408 INFO:teuthology.orchestra.run.smithi148.stdout:============================================================================================ 2024-04-23T16:06:13.408 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2024-04-23T16:06:13.409 INFO:teuthology.orchestra.run.smithi148.stdout:============================================================================================ 2024-04-23T16:06:13.409 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-04-23T16:06:13.409 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-diskprediction-local noarch 2:14.2.22-0.el8 ceph-noarch 1.1 M 2024-04-23T16:06:13.409 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-04-23T16:06:13.409 INFO:teuthology.orchestra.run.smithi148.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-23T16:06:13.409 INFO:teuthology.orchestra.run.smithi148.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-23T16:06:13.409 INFO:teuthology.orchestra.run.smithi148.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-23T16:06:13.409 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-23T16:06:13.409 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-23T16:06:13.409 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-23T16:06:13.409 INFO:teuthology.orchestra.run.smithi148.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-23T16:06:13.409 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:13.410 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-04-23T16:06:13.410 INFO:teuthology.orchestra.run.smithi148.stdout:============================================================================================ 2024-04-23T16:06:13.410 INFO:teuthology.orchestra.run.smithi148.stdout:Install 8 Packages 2024-04-23T16:06:13.410 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:13.410 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 30 M 2024-04-23T16:06:13.410 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 172 M 2024-04-23T16:06:13.411 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-04-23T16:06:13.416 INFO:teuthology.orchestra.run.smithi188.stdout:(1/8): ceph-mgr-diskprediction-local-14.2.22-0. 13 MB/s | 1.1 MB 00:00 2024-04-23T16:06:13.709 INFO:teuthology.orchestra.run.smithi148.stdout:(1/8): ceph-mgr-diskprediction-local-14.2.22-0. 20 MB/s | 1.1 MB 00:00 2024-04-23T16:06:13.816 INFO:teuthology.orchestra.run.smithi188.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 9.7 MB/s | 4.7 MB 00:00 2024-04-23T16:06:13.868 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:06:13.869 INFO:teuthology.orchestra.run.smithi103.stdout:============================================================================================ 2024-04-23T16:06:13.869 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-23T16:06:13.869 INFO:teuthology.orchestra.run.smithi103.stdout:============================================================================================ 2024-04-23T16:06:13.870 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-23T16:06:13.870 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local noarch 2:14.2.22-0.el8 ceph-noarch 1.1 M 2024-04-23T16:06:13.870 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-23T16:06:13.870 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-23T16:06:13.870 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-23T16:06:13.870 INFO:teuthology.orchestra.run.smithi103.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-23T16:06:13.870 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-23T16:06:13.870 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-23T16:06:13.870 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-23T16:06:13.870 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-23T16:06:13.870 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:13.871 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-23T16:06:13.871 INFO:teuthology.orchestra.run.smithi103.stdout:============================================================================================ 2024-04-23T16:06:13.871 INFO:teuthology.orchestra.run.smithi103.stdout:Install 8 Packages 2024-04-23T16:06:13.871 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:13.871 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 30 M 2024-04-23T16:06:13.871 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 172 M 2024-04-23T16:06:13.872 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-23T16:06:13.908 INFO:teuthology.orchestra.run.smithi188.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 8.4 MB/s | 4.9 MB 00:00 2024-04-23T16:06:13.933 INFO:teuthology.orchestra.run.smithi188.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.9 MB/s | 225 kB 00:00 2024-04-23T16:06:14.001 INFO:teuthology.orchestra.run.smithi188.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 6.4 MB/s | 3.7 MB 00:00 2024-04-23T16:06:14.185 INFO:teuthology.orchestra.run.smithi148.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 8.9 MB/s | 4.7 MB 00:00 2024-04-23T16:06:14.200 INFO:teuthology.orchestra.run.smithi188.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 863 kB/s | 172 kB 00:00 2024-04-23T16:06:14.226 INFO:teuthology.orchestra.run.smithi188.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.2 MB/s | 645 kB 00:00 2024-04-23T16:06:14.252 INFO:teuthology.orchestra.run.smithi148.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 3.3 MB/s | 225 kB 00:00 2024-04-23T16:06:14.346 INFO:teuthology.orchestra.run.smithi103.stdout:(1/8): ceph-mgr-diskprediction-local-14.2.22-0. 19 MB/s | 1.1 MB 00:00 2024-04-23T16:06:14.352 INFO:teuthology.orchestra.run.smithi148.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 6.9 MB/s | 4.9 MB 00:00 2024-04-23T16:06:14.444 INFO:teuthology.orchestra.run.smithi148.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.1 MB/s | 3.7 MB 00:00 2024-04-23T16:06:14.669 INFO:teuthology.orchestra.run.smithi148.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 767 kB/s | 172 kB 00:00 2024-04-23T16:06:14.709 INFO:teuthology.orchestra.run.smithi188.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 18 MB/s | 14 MB 00:00 2024-04-23T16:06:14.710 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:14.710 INFO:teuthology.orchestra.run.smithi188.stdout:Total 20 MB/s | 30 MB 00:01 2024-04-23T16:06:14.811 INFO:teuthology.orchestra.run.smithi148.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.4 MB/s | 645 kB 00:00 2024-04-23T16:06:14.941 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-23T16:06:14.959 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-23T16:06:14.959 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-23T16:06:15.112 INFO:teuthology.orchestra.run.smithi148.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 16 MB/s | 14 MB 00:00 2024-04-23T16:06:15.112 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:15.112 INFO:teuthology.orchestra.run.smithi148.stdout:Total 17 MB/s | 30 MB 00:01 2024-04-23T16:06:15.283 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-23T16:06:15.286 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-23T16:06:15.295 INFO:teuthology.orchestra.run.smithi103.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 4.7 MB/s | 4.7 MB 00:01 2024-04-23T16:06:15.343 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-04-23T16:06:15.361 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-04-23T16:06:15.361 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-04-23T16:06:15.389 INFO:teuthology.orchestra.run.smithi103.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.4 MB/s | 4.9 MB 00:01 2024-04-23T16:06:15.414 INFO:teuthology.orchestra.run.smithi103.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.9 MB/s | 225 kB 00:00 2024-04-23T16:06:15.481 INFO:teuthology.orchestra.run.smithi103.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.3 MB/s | 3.7 MB 00:01 2024-04-23T16:06:15.648 INFO:teuthology.orchestra.run.smithi103.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.0 MB/s | 172 kB 00:00 2024-04-23T16:06:15.683 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-04-23T16:06:15.686 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-04-23T16:06:15.706 INFO:teuthology.orchestra.run.smithi103.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.2 MB/s | 645 kB 00:00 2024-04-23T16:06:15.806 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-23T16:06:15.946 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-23T16:06:16.085 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-23T16:06:16.208 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-04-23T16:06:16.238 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-23T16:06:16.331 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-23T16:06:16.447 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-23T16:06:16.597 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-23T16:06:16.839 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-23T16:06:16.996 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-23T16:06:17.178 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-23T16:06:17.347 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-23T16:06:17.614 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-23T16:06:17.690 INFO:teuthology.orchestra.run.smithi103.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 6.2 MB/s | 14 MB 00:02 2024-04-23T16:06:17.690 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:17.691 INFO:teuthology.orchestra.run.smithi103.stdout:Total 7.8 MB/s | 30 MB 00:03 2024-04-23T16:06:17.737 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-23T16:06:17.923 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-23T16:06:17.941 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-23T16:06:17.942 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-23T16:06:17.966 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-23T16:06:18.130 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-23T16:06:18.251 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-23T16:06:18.254 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-23T16:06:18.422 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-23T16:06:18.740 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-23T16:06:18.810 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-23T16:06:18.828 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-23T16:06:18.954 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-23T16:06:18.984 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-23T16:06:19.127 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-23T16:06:19.175 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-23T16:06:19.309 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-23T16:06:19.422 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-23T16:06:19.914 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-23T16:06:20.109 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-23T16:06:20.699 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-23T16:06:20.725 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-23T16:06:20.884 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-23T16:06:21.171 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-23T16:06:21.292 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-23T16:06:21.462 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-04-23T16:06:21.575 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-23T16:06:21.765 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-23T16:06:21.804 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-04-23T16:06:21.805 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 1/8 2024-04-23T16:06:21.805 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-23T16:06:21.805 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-23T16:06:21.805 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-23T16:06:21.805 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-23T16:06:21.805 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-23T16:06:21.805 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-04-23T16:06:21.937 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-23T16:06:21.946 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-04-23T16:06:22.201 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-23T16:06:22.206 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-04-23T16:06:22.206 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:22.206 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-23T16:06:22.206 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 2024-04-23T16:06:22.206 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-23T16:06:22.206 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-23T16:06:22.207 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-23T16:06:22.207 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-23T16:06:22.207 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-23T16:06:22.207 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-23T16:06:22.207 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-23T16:06:22.207 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:22.207 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:06:22.267 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-04-23T16:06:22.267 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 1/8 2024-04-23T16:06:22.267 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-23T16:06:22.267 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-23T16:06:22.267 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-23T16:06:22.267 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-23T16:06:22.267 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-23T16:06:22.267 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-04-23T16:06:22.339 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-rook 2024-04-23T16:06:22.672 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-04-23T16:06:22.672 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:22.672 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-04-23T16:06:22.672 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 2024-04-23T16:06:22.672 INFO:teuthology.orchestra.run.smithi148.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-23T16:06:22.672 INFO:teuthology.orchestra.run.smithi148.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-23T16:06:22.673 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-23T16:06:22.673 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-23T16:06:22.673 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-23T16:06:22.673 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-23T16:06:22.673 INFO:teuthology.orchestra.run.smithi148.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-23T16:06:22.673 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:22.673 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:06:22.804 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-rook 2024-04-23T16:06:23.072 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:28 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:06:23.539 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:29 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:06:23.954 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-23T16:06:24.504 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-23T16:06:24.699 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:06:24.700 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-04-23T16:06:24.701 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:24.701 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-04-23T16:06:24.701 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:24.701 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-23T16:06:24.701 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook noarch 2:14.2.22-0.el8 ceph-noarch 15 k 2024-04-23T16:06:24.701 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-23T16:06:24.701 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-23T16:06:24.701 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-23T16:06:24.701 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-23T16:06:24.702 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-23T16:06:24.702 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-23T16:06:24.702 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-23T16:06:24.702 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-23T16:06:24.702 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-23T16:06:24.702 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-23T16:06:24.702 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-23T16:06:24.702 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:24.702 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-23T16:06:24.702 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:24.702 INFO:teuthology.orchestra.run.smithi188.stdout:Install 11 Packages 2024-04-23T16:06:24.702 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:24.703 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 1.9 M 2024-04-23T16:06:24.703 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 24 M 2024-04-23T16:06:24.703 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-23T16:06:25.036 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-04-23T16:06:25.036 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 1/8 2024-04-23T16:06:25.036 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-23T16:06:25.036 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-23T16:06:25.037 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-23T16:06:25.037 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-23T16:06:25.037 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-23T16:06:25.037 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-04-23T16:06:25.176 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:06:25.177 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:25.177 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2024-04-23T16:06:25.177 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:25.177 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-04-23T16:06:25.177 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-rook noarch 2:14.2.22-0.el8 ceph-noarch 15 k 2024-04-23T16:06:25.177 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-04-23T16:06:25.177 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-23T16:06:25.178 INFO:teuthology.orchestra.run.smithi148.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-23T16:06:25.178 INFO:teuthology.orchestra.run.smithi148.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-23T16:06:25.178 INFO:teuthology.orchestra.run.smithi148.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-23T16:06:25.178 INFO:teuthology.orchestra.run.smithi148.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-23T16:06:25.178 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-23T16:06:25.178 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-23T16:06:25.178 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-23T16:06:25.178 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-23T16:06:25.178 INFO:teuthology.orchestra.run.smithi148.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-23T16:06:25.178 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:25.178 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-04-23T16:06:25.178 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:25.178 INFO:teuthology.orchestra.run.smithi148.stdout:Install 11 Packages 2024-04-23T16:06:25.178 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:25.179 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 1.9 M 2024-04-23T16:06:25.179 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 24 M 2024-04-23T16:06:25.179 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-04-23T16:06:25.226 INFO:teuthology.orchestra.run.smithi188.stdout:(1/11): ceph-mgr-rook-14.2.22-0.el8.noarch.rpm 243 kB/s | 15 kB 00:00 2024-04-23T16:06:25.368 INFO:teuthology.orchestra.run.smithi188.stdout:(2/11): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.1 MB/s | 155 kB 00:00 2024-04-23T16:06:25.384 INFO:teuthology.orchestra.run.smithi188.stdout:(3/11): python3-pyasn1-modules-0.3.7-6.el8.noar 497 kB/s | 110 kB 00:00 2024-04-23T16:06:25.410 INFO:teuthology.orchestra.run.smithi188.stdout:(4/11): python3-pyasn1-0.3.7-6.el8.noarch.rpm 511 kB/s | 126 kB 00:00 2024-04-23T16:06:25.443 INFO:teuthology.orchestra.run.smithi188.stdout:(5/11): python3-requests-oauthlib-1.0.0-1.el8.n 565 kB/s | 43 kB 00:00 2024-04-23T16:06:25.462 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-04-23T16:06:25.462 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:25.463 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-23T16:06:25.463 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 2024-04-23T16:06:25.463 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-23T16:06:25.463 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-23T16:06:25.463 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-23T16:06:25.463 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-23T16:06:25.463 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-23T16:06:25.463 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-23T16:06:25.463 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-23T16:06:25.463 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:25.463 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:06:25.552 INFO:teuthology.orchestra.run.smithi188.stdout:(6/11): python3-google-auth-1.1.1-10.el8.noarch 750 kB/s | 81 kB 00:00 2024-04-23T16:06:25.594 INFO:teuthology.orchestra.run.smithi188.stdout:(7/11): python3-cachetools-3.1.1-4.el8.noarch.r 157 kB/s | 33 kB 00:00 2024-04-23T16:06:25.610 INFO:teuthology.orchestra.run.smithi148.stdout:(1/11): ceph-mgr-rook-14.2.22-0.el8.noarch.rpm 312 kB/s | 15 kB 00:00 2024-04-23T16:06:25.619 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-rook 2024-04-23T16:06:25.661 INFO:teuthology.orchestra.run.smithi188.stdout:(8/11): python3-certifi-2018.10.15-7.el8.noarch 65 kB/s | 16 kB 00:00 2024-04-23T16:06:25.686 INFO:teuthology.orchestra.run.smithi188.stdout:(9/11): python3-rsa-4.9-2.el8.noarch.rpm 699 kB/s | 64 kB 00:00 2024-04-23T16:06:25.720 INFO:teuthology.orchestra.run.smithi188.stdout:(10/11): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2024-04-23T16:06:25.910 INFO:teuthology.orchestra.run.smithi148.stdout:(2/11): python3-pyasn1-0.3.7-6.el8.noarch.rpm 362 kB/s | 126 kB 00:00 2024-04-23T16:06:25.987 INFO:teuthology.orchestra.run.smithi188.stdout:(11/11): python3-kubernetes-11.0.0-6.el8.noarch 2.7 MB/s | 1.2 MB 00:00 2024-04-23T16:06:25.987 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:25.987 INFO:teuthology.orchestra.run.smithi188.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2024-04-23T16:06:26.020 INFO:teuthology.orchestra.run.smithi148.stdout:(3/11): python3-pyasn1-modules-0.3.7-6.el8.noar 240 kB/s | 110 kB 00:00 2024-04-23T16:06:26.027 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-23T16:06:26.043 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-23T16:06:26.043 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-23T16:06:26.119 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-23T16:06:26.122 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-23T16:06:26.133 INFO:teuthology.orchestra.run.smithi148.stdout:(4/11): python3-oauthlib-2.1.0-1.el8.noarch.rpm 296 kB/s | 155 kB 00:00 2024-04-23T16:06:26.290 INFO:teuthology.orchestra.run.smithi148.stdout:(5/11): python3-requests-oauthlib-1.0.0-1.el8.n 112 kB/s | 43 kB 00:00 2024-04-23T16:06:26.346 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:33 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:06:26.707 INFO:teuthology.orchestra.run.smithi148.stdout:(6/11): python3-certifi-2018.10.15-7.el8.noarch 28 kB/s | 16 kB 00:00 2024-04-23T16:06:26.947 INFO:teuthology.orchestra.run.smithi148.stdout:(7/11): python3-cachetools-3.1.1-4.el8.noarch.r 36 kB/s | 33 kB 00:00 2024-04-23T16:06:26.972 INFO:teuthology.orchestra.run.smithi148.stdout:(8/11): python3-google-auth-1.1.1-10.el8.noarch 119 kB/s | 81 kB 00:00 2024-04-23T16:06:27.072 INFO:teuthology.orchestra.run.smithi148.stdout:(9/11): python3-rsa-4.9-2.el8.noarch.rpm 513 kB/s | 64 kB 00:00 2024-04-23T16:06:27.089 INFO:teuthology.orchestra.run.smithi148.stdout:(10/11): python3-websocket-client-0.56.0-5.el8. 525 kB/s | 61 kB 00:00 2024-04-23T16:06:27.306 INFO:teuthology.orchestra.run.smithi148.stdout:(11/11): python3-kubernetes-11.0.0-6.el8.noarch 1.9 MB/s | 1.2 MB 00:00 2024-04-23T16:06:27.307 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:27.307 INFO:teuthology.orchestra.run.smithi148.stdout:Total 892 kB/s | 1.9 MB 00:02 2024-04-23T16:06:27.348 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-04-23T16:06:27.364 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-04-23T16:06:27.364 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-04-23T16:06:27.440 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-04-23T16:06:27.444 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-04-23T16:06:27.896 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-23T16:06:27.922 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-04-23T16:06:27.995 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:06:27.996 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:27.996 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-23T16:06:27.996 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:27.996 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-23T16:06:27.997 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook noarch 2:14.2.22-0.el8 ceph-noarch 15 k 2024-04-23T16:06:27.997 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-23T16:06:27.997 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-23T16:06:27.997 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-23T16:06:27.997 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-23T16:06:27.997 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-23T16:06:27.997 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-23T16:06:27.997 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-23T16:06:27.997 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-23T16:06:27.997 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-23T16:06:27.997 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-23T16:06:27.997 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-23T16:06:27.997 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:27.998 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-23T16:06:27.998 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:27.998 INFO:teuthology.orchestra.run.smithi103.stdout:Install 11 Packages 2024-04-23T16:06:27.998 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:27.998 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 1.9 M 2024-04-23T16:06:27.999 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 24 M 2024-04-23T16:06:27.999 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-23T16:06:28.129 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/11 2024-04-23T16:06:28.190 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/11 2024-04-23T16:06:28.301 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/11 2024-04-23T16:06:28.379 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/11 2024-04-23T16:06:28.409 INFO:teuthology.orchestra.run.smithi103.stdout:(1/11): ceph-mgr-rook-14.2.22-0.el8.noarch.rpm 364 kB/s | 15 kB 00:00 2024-04-23T16:06:28.520 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/11 2024-04-23T16:06:28.585 INFO:teuthology.orchestra.run.smithi103.stdout:(2/11): python3-pyasn1-modules-0.3.7-6.el8.noar 507 kB/s | 110 kB 00:00 2024-04-23T16:06:28.610 INFO:teuthology.orchestra.run.smithi103.stdout:(3/11): python3-pyasn1-0.3.7-6.el8.noarch.rpm 521 kB/s | 126 kB 00:00 2024-04-23T16:06:28.631 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/11 2024-04-23T16:06:28.635 INFO:teuthology.orchestra.run.smithi103.stdout:(4/11): python3-oauthlib-2.1.0-1.el8.noarch.rpm 686 kB/s | 155 kB 00:00 2024-04-23T16:06:28.719 INFO:teuthology.orchestra.run.smithi103.stdout:(5/11): python3-requests-oauthlib-1.0.0-1.el8.n 318 kB/s | 43 kB 00:00 2024-04-23T16:06:28.724 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/11 2024-04-23T16:06:28.787 INFO:teuthology.orchestra.run.smithi103.stdout:(6/11): python3-cachetools-3.1.1-4.el8.noarch.r 187 kB/s | 33 kB 00:00 2024-04-23T16:06:28.812 INFO:teuthology.orchestra.run.smithi103.stdout:(7/11): python3-certifi-2018.10.15-7.el8.noarch 92 kB/s | 16 kB 00:00 2024-04-23T16:06:28.860 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/11 2024-04-23T16:06:28.871 INFO:teuthology.orchestra.run.smithi103.stdout:(8/11): python3-rsa-4.9-2.el8.noarch.rpm 1.1 MB/s | 64 kB 00:00 2024-04-23T16:06:28.904 INFO:teuthology.orchestra.run.smithi103.stdout:(9/11): python3-google-auth-1.1.1-10.el8.noarch 440 kB/s | 81 kB 00:00 2024-04-23T16:06:28.929 INFO:teuthology.orchestra.run.smithi103.stdout:(10/11): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2024-04-23T16:06:28.963 INFO:teuthology.orchestra.run.smithi103.stdout:(11/11): python3-kubernetes-11.0.0-6.el8.noarch 6.8 MB/s | 1.2 MB 00:00 2024-04-23T16:06:28.965 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/11 2024-04-23T16:06:28.965 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:28.965 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.9 MB/s | 1.9 MB 00:00 2024-04-23T16:06:29.002 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-23T16:06:29.017 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-23T16:06:29.017 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-23T16:06:29.090 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-23T16:06:29.093 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-23T16:06:29.101 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/11 2024-04-23T16:06:29.187 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-04-23T16:06:29.332 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-04-23T16:06:29.503 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/11 2024-04-23T16:06:29.607 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-23T16:06:29.615 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/11 2024-04-23T16:06:29.759 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/11 2024-04-23T16:06:29.844 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/11 2024-04-23T16:06:29.880 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/11 2024-04-23T16:06:30.067 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/11 2024-04-23T16:06:30.294 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/11 2024-04-23T16:06:30.531 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/11 2024-04-23T16:06:30.789 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/11 2024-04-23T16:06:30.798 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/11 2024-04-23T16:06:30.928 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/11 2024-04-23T16:06:31.022 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-04-23T16:06:31.053 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/11 2024-04-23T16:06:31.172 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/11 2024-04-23T16:06:31.201 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-04-23T16:06:31.328 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-04-23T16:06:31.340 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/11 2024-04-23T16:06:31.494 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-04-23T16:06:31.494 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-rook-2:14.2.22-0.el8.noarch 1/11 2024-04-23T16:06:31.494 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 2/11 2024-04-23T16:06:31.494 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/11 2024-04-23T16:06:31.495 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 4/11 2024-04-23T16:06:31.495 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/11 2024-04-23T16:06:31.495 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-04-23T16:06:31.495 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 7/11 2024-04-23T16:06:31.495 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 8/11 2024-04-23T16:06:31.495 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 9/11 2024-04-23T16:06:31.495 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 10/11 2024-04-23T16:06:31.613 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/11 2024-04-23T16:06:31.633 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-04-23T16:06:31.634 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-rook-2:14.2.22-0.el8.noarch 1/11 2024-04-23T16:06:31.634 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 2/11 2024-04-23T16:06:31.634 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/11 2024-04-23T16:06:31.634 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 4/11 2024-04-23T16:06:31.634 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/11 2024-04-23T16:06:31.634 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-04-23T16:06:31.634 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 7/11 2024-04-23T16:06:31.634 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 8/11 2024-04-23T16:06:31.634 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 9/11 2024-04-23T16:06:31.634 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 10/11 2024-04-23T16:06:31.937 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 11/11 2024-04-23T16:06:31.937 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:31.937 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-23T16:06:31.937 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook-2:14.2.22-0.el8.noarch 2024-04-23T16:06:31.937 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-23T16:06:31.937 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-23T16:06:31.937 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-23T16:06:31.938 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-23T16:06:31.938 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-23T16:06:31.938 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-23T16:06:31.938 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-23T16:06:31.938 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-23T16:06:31.938 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-23T16:06:31.938 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-23T16:06:31.938 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:31.938 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:06:32.061 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 11/11 2024-04-23T16:06:32.061 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:32.061 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-04-23T16:06:32.061 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-rook-2:14.2.22-0.el8.noarch 2024-04-23T16:06:32.061 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-23T16:06:32.061 INFO:teuthology.orchestra.run.smithi148.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-23T16:06:32.062 INFO:teuthology.orchestra.run.smithi148.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-23T16:06:32.062 INFO:teuthology.orchestra.run.smithi148.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-23T16:06:32.062 INFO:teuthology.orchestra.run.smithi148.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-23T16:06:32.062 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-23T16:06:32.062 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-23T16:06:32.062 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-23T16:06:32.062 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-23T16:06:32.062 INFO:teuthology.orchestra.run.smithi148.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-23T16:06:32.062 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:32.062 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:06:32.063 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-fuse 2024-04-23T16:06:32.172 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-fuse 2024-04-23T16:06:32.672 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/11 2024-04-23T16:06:32.787 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:37 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:06:32.900 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:38 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:06:32.968 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/11 2024-04-23T16:06:33.157 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-04-23T16:06:33.447 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-04-23T16:06:33.447 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-rook-2:14.2.22-0.el8.noarch 1/11 2024-04-23T16:06:33.447 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 2/11 2024-04-23T16:06:33.447 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/11 2024-04-23T16:06:33.447 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 4/11 2024-04-23T16:06:33.447 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/11 2024-04-23T16:06:33.447 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-04-23T16:06:33.447 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 7/11 2024-04-23T16:06:33.447 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 8/11 2024-04-23T16:06:33.448 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 9/11 2024-04-23T16:06:33.448 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 10/11 2024-04-23T16:06:33.927 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 11/11 2024-04-23T16:06:33.927 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:33.928 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-23T16:06:33.928 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook-2:14.2.22-0.el8.noarch 2024-04-23T16:06:33.928 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-23T16:06:33.928 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-23T16:06:33.928 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-23T16:06:33.928 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-23T16:06:33.928 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-23T16:06:33.928 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-23T16:06:33.928 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-23T16:06:33.928 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-23T16:06:33.928 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-23T16:06:33.928 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-23T16:06:33.929 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:33.929 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:06:34.063 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-fuse 2024-04-23T16:06:34.397 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:06:34.398 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:34.398 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-23T16:06:34.398 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:34.398 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-23T16:06:34.398 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse x86_64 2:14.2.22-0.el8 ceph 590 k 2024-04-23T16:06:34.399 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-23T16:06:34.399 INFO:teuthology.orchestra.run.smithi188.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-23T16:06:34.399 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:34.399 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-23T16:06:34.399 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:34.399 INFO:teuthology.orchestra.run.smithi188.stdout:Install 2 Packages 2024-04-23T16:06:34.399 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:34.400 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 673 k 2024-04-23T16:06:34.400 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 1.9 M 2024-04-23T16:06:34.400 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-23T16:06:34.519 INFO:teuthology.orchestra.run.smithi188.stdout:(1/2): ceph-fuse-14.2.22-0.el8.x86_64.rpm 8.3 MB/s | 590 kB 00:00 2024-04-23T16:06:34.545 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:06:34.546 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:34.546 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2024-04-23T16:06:34.547 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:34.547 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-04-23T16:06:34.547 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse x86_64 2:14.2.22-0.el8 ceph 590 k 2024-04-23T16:06:34.547 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-04-23T16:06:34.547 INFO:teuthology.orchestra.run.smithi148.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-23T16:06:34.547 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:34.547 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-04-23T16:06:34.547 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:34.547 INFO:teuthology.orchestra.run.smithi148.stdout:Install 2 Packages 2024-04-23T16:06:34.547 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:34.547 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 673 k 2024-04-23T16:06:34.547 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 1.9 M 2024-04-23T16:06:34.548 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-04-23T16:06:34.561 INFO:teuthology.orchestra.run.smithi188.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 748 kB/s | 83 kB 00:00 2024-04-23T16:06:34.561 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:34.562 INFO:teuthology.orchestra.run.smithi188.stdout:Total 4.0 MB/s | 673 kB 00:00 2024-04-23T16:06:34.571 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-23T16:06:34.593 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-23T16:06:34.593 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-23T16:06:34.679 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-23T16:06:34.682 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-23T16:06:34.791 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:41 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:06:34.795 INFO:teuthology.orchestra.run.smithi148.stdout:(1/2): ceph-fuse-14.2.22-0.el8.x86_64.rpm 11 MB/s | 590 kB 00:00 2024-04-23T16:06:34.863 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-23T16:06:34.904 INFO:teuthology.orchestra.run.smithi148.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 517 kB/s | 83 kB 00:00 2024-04-23T16:06:34.904 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:34.905 INFO:teuthology.orchestra.run.smithi148.stdout:Total 1.8 MB/s | 673 kB 00:00 2024-04-23T16:06:34.914 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-04-23T16:06:34.936 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-04-23T16:06:34.936 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-04-23T16:06:35.021 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-04-23T16:06:35.025 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-04-23T16:06:35.090 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-23T16:06:35.200 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-04-23T16:06:35.300 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-04-23T16:06:35.443 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-23T16:06:35.653 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-04-23T16:06:35.703 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-04-23T16:06:35.703 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-fuse-2:14.2.22-0.el8.x86_64 1/2 2024-04-23T16:06:36.048 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-04-23T16:06:36.048 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-fuse-2:14.2.22-0.el8.x86_64 1/2 2024-04-23T16:06:36.110 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-23T16:06:36.110 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:36.110 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-23T16:06:36.110 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse-2:14.2.22-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-23T16:06:36.110 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:36.110 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:06:36.366 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librados-devel 2024-04-23T16:06:36.432 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-23T16:06:36.433 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:36.433 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-04-23T16:06:36.433 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse-2:14.2.22-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-23T16:06:36.433 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:36.433 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:06:36.440 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:06:36.441 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:36.441 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-04-23T16:06:36.441 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:36.441 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-23T16:06:36.441 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse x86_64 2:14.2.22-0.el8 ceph 590 k 2024-04-23T16:06:36.441 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-23T16:06:36.442 INFO:teuthology.orchestra.run.smithi103.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-23T16:06:36.442 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:36.442 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-23T16:06:36.442 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:36.442 INFO:teuthology.orchestra.run.smithi103.stdout:Install 2 Packages 2024-04-23T16:06:36.442 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:36.443 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 673 k 2024-04-23T16:06:36.443 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 1.9 M 2024-04-23T16:06:36.443 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-23T16:06:36.545 INFO:teuthology.orchestra.run.smithi103.stdout:(1/2): ceph-fuse-14.2.22-0.el8.x86_64.rpm 14 MB/s | 590 kB 00:00 2024-04-23T16:06:36.649 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install librados-devel 2024-04-23T16:06:36.662 INFO:teuthology.orchestra.run.smithi103.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 523 kB/s | 83 kB 00:00 2024-04-23T16:06:36.663 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:36.663 INFO:teuthology.orchestra.run.smithi103.stdout:Total 3.0 MB/s | 673 kB 00:00 2024-04-23T16:06:36.672 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-23T16:06:36.694 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-23T16:06:36.694 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-23T16:06:36.778 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-23T16:06:36.781 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-23T16:06:36.948 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-23T16:06:37.107 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:42 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:06:37.222 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-23T16:06:37.391 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:43 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:06:37.557 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-04-23T16:06:37.943 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-04-23T16:06:37.944 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-fuse-2:14.2.22-0.el8.x86_64 1/2 2024-04-23T16:06:38.321 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-23T16:06:38.321 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:38.322 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-23T16:06:38.322 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-2:14.2.22-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-23T16:06:38.322 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:38.322 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:06:38.561 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados-devel 2024-04-23T16:06:38.712 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:06:38.713 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:38.713 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-23T16:06:38.713 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:38.713 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-23T16:06:38.713 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel x86_64 2:14.2.22-0.el8 ceph 90 k 2024-04-23T16:06:38.713 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:38.713 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-23T16:06:38.714 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:38.714 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-04-23T16:06:38.714 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:38.714 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 90 k 2024-04-23T16:06:38.714 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 310 k 2024-04-23T16:06:38.714 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-23T16:06:38.774 INFO:teuthology.orchestra.run.smithi188.stdout:librados-devel-14.2.22-0.el8.x86_64.rpm 1.5 MB/s | 90 kB 00:00 2024-04-23T16:06:38.775 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:38.775 INFO:teuthology.orchestra.run.smithi188.stdout:Total 1.4 MB/s | 90 kB 00:00 2024-04-23T16:06:38.775 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-23T16:06:38.786 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-23T16:06:38.786 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-23T16:06:38.834 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-23T16:06:38.837 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-23T16:06:38.949 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-23T16:06:39.015 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:06:39.015 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:39.016 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2024-04-23T16:06:39.016 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:39.016 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-04-23T16:06:39.016 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel x86_64 2:14.2.22-0.el8 ceph 90 k 2024-04-23T16:06:39.016 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:39.016 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-04-23T16:06:39.016 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:39.016 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2024-04-23T16:06:39.016 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:39.017 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 90 k 2024-04-23T16:06:39.017 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 310 k 2024-04-23T16:06:39.017 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-04-23T16:06:39.057 INFO:teuthology.orchestra.run.smithi148.stdout:librados-devel-14.2.22-0.el8.x86_64.rpm 2.2 MB/s | 90 kB 00:00 2024-04-23T16:06:39.058 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:39.058 INFO:teuthology.orchestra.run.smithi148.stdout:Total 2.1 MB/s | 90 kB 00:00 2024-04-23T16:06:39.058 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-04-23T16:06:39.069 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-04-23T16:06:39.069 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-04-23T16:06:39.118 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-04-23T16:06:39.121 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-04-23T16:06:39.150 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:39.234 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-04-23T16:06:39.296 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:46 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:06:39.409 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:39.444 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:39.711 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:39.839 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:39.839 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:39.839 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-23T16:06:39.839 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel-2:14.2.22-0.el8.x86_64 2024-04-23T16:06:39.839 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:39.839 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:06:40.062 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install libcephfs2 2024-04-23T16:06:40.120 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:40.120 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:40.120 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-04-23T16:06:40.120 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel-2:14.2.22-0.el8.x86_64 2024-04-23T16:06:40.120 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:40.120 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:06:40.336 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install libcephfs2 2024-04-23T16:06:40.803 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:45 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:06:40.904 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:06:40.904 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:40.904 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-04-23T16:06:40.904 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:40.904 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-23T16:06:40.904 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel x86_64 2:14.2.22-0.el8 ceph 90 k 2024-04-23T16:06:40.904 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:40.905 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-23T16:06:40.905 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:40.905 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-04-23T16:06:40.905 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:40.905 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 90 k 2024-04-23T16:06:40.905 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 310 k 2024-04-23T16:06:40.905 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-23T16:06:40.951 INFO:teuthology.orchestra.run.smithi103.stdout:librados-devel-14.2.22-0.el8.x86_64.rpm 1.9 MB/s | 90 kB 00:00 2024-04-23T16:06:40.952 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:40.952 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.9 MB/s | 90 kB 00:00 2024-04-23T16:06:40.952 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-23T16:06:40.963 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-23T16:06:40.963 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-23T16:06:41.012 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-23T16:06:41.016 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-23T16:06:41.080 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:47 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:06:41.153 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-23T16:06:41.363 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:41.683 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:42.070 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:42.070 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:42.071 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-23T16:06:42.071 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-2:14.2.22-0.el8.x86_64 2024-04-23T16:06:42.071 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:42.071 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:06:42.338 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs2 2024-04-23T16:06:42.369 INFO:teuthology.orchestra.run.smithi188.stdout:Package libcephfs2-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:06:42.417 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:06:42.418 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-04-23T16:06:42.418 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:06:42.529 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install libcephfs-devel 2024-04-23T16:06:42.652 INFO:teuthology.orchestra.run.smithi148.stdout:Package libcephfs2-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:06:42.701 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:06:42.702 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2024-04-23T16:06:42.703 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:06:42.913 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install libcephfs-devel 2024-04-23T16:06:43.079 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:50 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:06:43.262 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:48 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:06:43.646 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:49 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:06:44.673 INFO:teuthology.orchestra.run.smithi103.stdout:Package libcephfs2-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:06:44.721 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:06:44.722 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-23T16:06:44.722 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:06:44.844 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs-devel 2024-04-23T16:06:44.886 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:06:44.886 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:44.886 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-23T16:06:44.886 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:44.886 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-23T16:06:44.886 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel x86_64 2:14.2.22-0.el8 ceph 22 k 2024-04-23T16:06:44.886 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:44.886 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-23T16:06:44.887 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:44.887 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-04-23T16:06:44.887 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:44.887 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 22 k 2024-04-23T16:06:44.887 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 77 k 2024-04-23T16:06:44.887 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-23T16:06:44.962 INFO:teuthology.orchestra.run.smithi188.stdout:libcephfs-devel-14.2.22-0.el8.x86_64.rpm 281 kB/s | 22 kB 00:00 2024-04-23T16:06:44.963 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:44.963 INFO:teuthology.orchestra.run.smithi188.stdout:Total 277 kB/s | 22 kB 00:00 2024-04-23T16:06:44.963 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-23T16:06:44.966 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-23T16:06:44.967 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-23T16:06:44.968 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-23T16:06:44.972 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-23T16:06:45.019 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-23T16:06:45.163 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:45.276 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:06:45.276 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:45.276 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2024-04-23T16:06:45.277 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:45.277 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-04-23T16:06:45.277 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-devel x86_64 2:14.2.22-0.el8 ceph 22 k 2024-04-23T16:06:45.277 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:45.277 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-04-23T16:06:45.277 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:45.277 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2024-04-23T16:06:45.277 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:45.278 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 22 k 2024-04-23T16:06:45.278 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 77 k 2024-04-23T16:06:45.278 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-04-23T16:06:45.328 INFO:teuthology.orchestra.run.smithi148.stdout:libcephfs-devel-14.2.22-0.el8.x86_64.rpm 428 kB/s | 22 kB 00:00 2024-04-23T16:06:45.329 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:45.329 INFO:teuthology.orchestra.run.smithi148.stdout:Total 416 kB/s | 22 kB 00:00 2024-04-23T16:06:45.329 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-04-23T16:06:45.333 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-04-23T16:06:45.333 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-04-23T16:06:45.335 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-04-23T16:06:45.338 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-04-23T16:06:45.386 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-04-23T16:06:45.428 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:45.566 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:45.574 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:52 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:06:45.781 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:45.781 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:45.781 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-23T16:06:45.781 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel-2:14.2.22-0.el8.x86_64 2024-04-23T16:06:45.781 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:45.781 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:06:45.835 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:45.909 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librados2 2024-04-23T16:06:46.182 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:46.182 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:46.182 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-04-23T16:06:46.182 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-devel-2:14.2.22-0.el8.x86_64 2024-04-23T16:06:46.182 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:46.182 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:06:46.293 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install librados2 2024-04-23T16:06:46.640 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:51 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:06:47.023 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:53 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:06:47.205 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:06:47.206 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:47.206 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-04-23T16:06:47.206 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:47.206 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-23T16:06:47.206 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel x86_64 2:14.2.22-0.el8 ceph 22 k 2024-04-23T16:06:47.206 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:47.206 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-23T16:06:47.206 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:47.206 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-04-23T16:06:47.206 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:47.206 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 22 k 2024-04-23T16:06:47.207 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 77 k 2024-04-23T16:06:47.207 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-23T16:06:47.250 INFO:teuthology.orchestra.run.smithi103.stdout:libcephfs-devel-14.2.22-0.el8.x86_64.rpm 481 kB/s | 22 kB 00:00 2024-04-23T16:06:47.254 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:47.255 INFO:teuthology.orchestra.run.smithi103.stdout:Total 468 kB/s | 22 kB 00:00 2024-04-23T16:06:47.255 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-23T16:06:47.256 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-23T16:06:47.256 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-23T16:06:47.256 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-23T16:06:47.259 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-23T16:06:47.306 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-23T16:06:47.505 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:47.761 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:48.129 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:48.129 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:48.129 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-23T16:06:48.129 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel-2:14.2.22-0.el8.x86_64 2024-04-23T16:06:48.129 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:48.129 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:06:48.235 INFO:teuthology.orchestra.run.smithi188.stdout:Package librados2-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:06:48.284 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:06:48.285 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-04-23T16:06:48.285 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:06:48.525 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librbd1 2024-04-23T16:06:48.532 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados2 2024-04-23T16:06:48.611 INFO:teuthology.orchestra.run.smithi148.stdout:Package librados2-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:06:48.659 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:06:48.660 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2024-04-23T16:06:48.660 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:06:48.772 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install librbd1 2024-04-23T16:06:49.255 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:56 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:06:49.257 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:54 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:06:49.498 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:55 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:06:50.829 INFO:teuthology.orchestra.run.smithi188.stdout:Package librbd1-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:06:50.844 INFO:teuthology.orchestra.run.smithi103.stdout:Package librados2-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:06:50.876 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:06:50.877 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-04-23T16:06:50.877 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:06:50.892 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:06:50.893 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-23T16:06:50.893 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:06:50.985 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-fuse 2024-04-23T16:06:51.018 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librbd1 2024-04-23T16:06:51.083 INFO:teuthology.orchestra.run.smithi148.stdout:Package librbd1-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:06:51.131 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:06:51.132 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2024-04-23T16:06:51.132 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:06:51.230 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-fuse 2024-04-23T16:06:51.710 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:56 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:06:51.743 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:58 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:06:51.957 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:57 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:06:53.330 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:06:53.331 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:53.331 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-23T16:06:53.331 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:53.331 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-23T16:06:53.331 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse x86_64 2:14.2.22-0.el8 ceph 73 k 2024-04-23T16:06:53.331 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:53.331 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-23T16:06:53.332 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:53.332 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-04-23T16:06:53.332 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:53.332 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 73 k 2024-04-23T16:06:53.332 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 188 k 2024-04-23T16:06:53.332 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-23T16:06:53.398 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-fuse-14.2.22-0.el8.x86_64.rpm 1.1 MB/s | 73 kB 00:00 2024-04-23T16:06:53.399 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:53.399 INFO:teuthology.orchestra.run.smithi188.stdout:Total 1.1 MB/s | 73 kB 00:00 2024-04-23T16:06:53.399 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-23T16:06:53.409 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-23T16:06:53.410 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-23T16:06:53.419 INFO:teuthology.orchestra.run.smithi103.stdout:Package librbd1-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:06:53.460 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-23T16:06:53.466 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-23T16:06:53.468 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:06:53.468 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-23T16:06:53.468 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:06:53.560 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-23T16:06:53.579 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-fuse 2024-04-23T16:06:53.600 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:06:53.601 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:53.601 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2024-04-23T16:06:53.601 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:53.602 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-04-23T16:06:53.602 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse x86_64 2:14.2.22-0.el8 ceph 73 k 2024-04-23T16:06:53.602 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:53.602 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-04-23T16:06:53.602 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:53.602 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2024-04-23T16:06:53.602 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:53.602 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 73 k 2024-04-23T16:06:53.602 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 188 k 2024-04-23T16:06:53.602 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-04-23T16:06:53.778 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:54.058 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-fuse-14.2.22-0.el8.x86_64.rpm 160 kB/s | 73 kB 00:00 2024-04-23T16:06:54.059 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:54.059 INFO:teuthology.orchestra.run.smithi148.stdout:Total 159 kB/s | 73 kB 00:00 2024-04-23T16:06:54.060 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-04-23T16:06:54.071 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:54.071 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-04-23T16:06:54.071 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-04-23T16:06:54.123 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-04-23T16:06:54.126 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-04-23T16:06:54.319 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:01 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:06:54.347 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-04-23T16:06:54.439 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:54.439 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:54.439 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-23T16:06:54.440 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse-2:14.2.22-0.el8.x86_64 2024-04-23T16:06:54.440 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:54.440 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:06:54.582 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:54.880 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:55.283 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:55.283 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:55.283 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-04-23T16:06:55.283 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse-2:14.2.22-0.el8.x86_64 2024-04-23T16:06:55.283 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:55.283 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:06:55.449 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-mirror 2024-04-23T16:06:55.500 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-mirror 2024-04-23T16:06:55.952 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:06:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:55.953 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-04-23T16:06:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-23T16:06:55.953 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse x86_64 2:14.2.22-0.el8 ceph 73 k 2024-04-23T16:06:55.953 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-23T16:06:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:55.954 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-04-23T16:06:55.954 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:55.954 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 73 k 2024-04-23T16:06:55.954 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 188 k 2024-04-23T16:06:55.954 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-23T16:06:56.006 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-fuse-14.2.22-0.el8.x86_64.rpm 1.4 MB/s | 73 kB 00:00 2024-04-23T16:06:56.007 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:56.007 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.3 MB/s | 73 kB 00:00 2024-04-23T16:06:56.008 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-23T16:06:56.018 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-23T16:06:56.018 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-23T16:06:56.066 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-23T16:06:56.069 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-23T16:06:56.171 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-23T16:06:56.187 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:01 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:06:56.239 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:02 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:06:56.403 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:56.691 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:57.092 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:57.092 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:57.092 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-23T16:06:57.092 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-2:14.2.22-0.el8.x86_64 2024-04-23T16:06:57.092 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:57.092 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:06:57.332 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-mirror 2024-04-23T16:06:57.814 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:06:57.815 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:57.815 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-23T16:06:57.815 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:57.815 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-23T16:06:57.815 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror x86_64 2:14.2.22-0.el8 ceph 2.3 M 2024-04-23T16:06:57.815 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:57.815 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-23T16:06:57.815 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:06:57.816 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-04-23T16:06:57.816 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:57.816 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 2.3 M 2024-04-23T16:06:57.816 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 8.4 M 2024-04-23T16:06:57.816 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-23T16:06:57.881 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:06:57.881 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:57.881 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2024-04-23T16:06:57.881 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:57.881 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-04-23T16:06:57.882 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror x86_64 2:14.2.22-0.el8 ceph 2.3 M 2024-04-23T16:06:57.882 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:57.882 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-04-23T16:06:57.882 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:06:57.882 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2024-04-23T16:06:57.882 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:57.883 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 2.3 M 2024-04-23T16:06:57.883 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 8.4 M 2024-04-23T16:06:57.883 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-04-23T16:06:57.970 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-mirror-14.2.22-0.el8.x86_64.rpm 27 MB/s | 2.3 MB 00:00 2024-04-23T16:06:57.970 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:57.971 INFO:teuthology.orchestra.run.smithi148.stdout:Total 26 MB/s | 2.3 MB 00:00 2024-04-23T16:06:57.971 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-04-23T16:06:57.983 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-04-23T16:06:57.983 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-04-23T16:06:57.989 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-mirror-14.2.22-0.el8.x86_64.rpm 13 MB/s | 2.3 MB 00:00 2024-04-23T16:06:57.990 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:57.990 INFO:teuthology.orchestra.run.smithi188.stdout:Total 13 MB/s | 2.3 MB 00:00 2024-04-23T16:06:57.990 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-23T16:06:58.002 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-23T16:06:58.002 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-23T16:06:58.041 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-04-23T16:06:58.043 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-04-23T16:06:58.061 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-23T16:06:58.064 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-23T16:06:58.069 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:05 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:06:58.346 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-04-23T16:06:58.369 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-23T16:06:58.533 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:58.587 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:58.951 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:58.989 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:59.351 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:59.352 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:59.352 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-04-23T16:06:59.352 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror-2:14.2.22-0.el8.x86_64 2024-04-23T16:06:59.352 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:06:59.352 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:06:59.377 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:06:59.377 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:59.377 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-23T16:06:59.378 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror-2:14.2.22-0.el8.x86_64 2024-04-23T16:06:59.378 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:06:59.378 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:06:59.588 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-nbd 2024-04-23T16:06:59.665 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-nbd 2024-04-23T16:06:59.679 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:06:59.680 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:59.680 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-04-23T16:06:59.680 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:59.680 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-23T16:06:59.680 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror x86_64 2:14.2.22-0.el8 ceph 2.3 M 2024-04-23T16:06:59.680 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:59.680 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-23T16:06:59.680 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:06:59.680 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-04-23T16:06:59.681 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:06:59.681 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 2.3 M 2024-04-23T16:06:59.681 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 8.4 M 2024-04-23T16:06:59.681 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-23T16:06:59.753 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-mirror-14.2.22-0.el8.x86_64.rpm 33 MB/s | 2.3 MB 00:00 2024-04-23T16:06:59.754 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:06:59.754 INFO:teuthology.orchestra.run.smithi103.stdout:Total 32 MB/s | 2.3 MB 00:00 2024-04-23T16:06:59.754 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-23T16:06:59.765 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-23T16:06:59.766 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-23T16:06:59.824 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-23T16:06:59.827 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-23T16:07:00.133 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-23T16:07:00.325 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:05 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:07:00.404 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:07:00.407 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:06 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:07:00.823 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:07:01.210 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:07:01.210 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:07:01.211 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-23T16:07:01.211 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-2:14.2.22-0.el8.x86_64 2024-04-23T16:07:01.211 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:07:01.211 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:07:01.499 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-nbd 2024-04-23T16:07:01.979 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:07:01.979 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:07:01.980 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-23T16:07:01.980 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:07:01.980 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-23T16:07:01.980 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd x86_64 2:14.2.22-0.el8 ceph 142 k 2024-04-23T16:07:01.980 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:07:01.980 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-23T16:07:01.980 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-23T16:07:01.980 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-04-23T16:07:01.980 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:07:01.981 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 142 k 2024-04-23T16:07:01.981 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 422 k 2024-04-23T16:07:01.981 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-23T16:07:02.037 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-nbd-14.2.22-0.el8.x86_64.rpm 2.4 MB/s | 142 kB 00:00 2024-04-23T16:07:02.038 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:07:02.038 INFO:teuthology.orchestra.run.smithi188.stdout:Total 2.4 MB/s | 142 kB 00:00 2024-04-23T16:07:02.039 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-23T16:07:02.041 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:07:02.041 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:07:02.041 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2024-04-23T16:07:02.042 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:07:02.042 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-04-23T16:07:02.042 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd x86_64 2:14.2.22-0.el8 ceph 142 k 2024-04-23T16:07:02.042 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:07:02.042 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-04-23T16:07:02.042 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-04-23T16:07:02.042 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2024-04-23T16:07:02.042 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:07:02.043 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 142 k 2024-04-23T16:07:02.043 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 422 k 2024-04-23T16:07:02.043 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-04-23T16:07:02.049 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-23T16:07:02.049 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-23T16:07:02.090 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-nbd-14.2.22-0.el8.x86_64.rpm 3.0 MB/s | 142 kB 00:00 2024-04-23T16:07:02.091 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:07:02.091 INFO:teuthology.orchestra.run.smithi148.stdout:Total 2.9 MB/s | 142 kB 00:00 2024-04-23T16:07:02.091 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-04-23T16:07:02.102 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-04-23T16:07:02.102 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-04-23T16:07:02.109 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-23T16:07:02.112 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-23T16:07:02.160 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-04-23T16:07:02.163 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-04-23T16:07:02.231 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-23T16:07:02.236 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:09 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:07:02.283 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-04-23T16:07:02.451 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:07:02.472 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:07:02.743 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:07:02.869 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:07:03.121 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:07:03.121 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:07:03.121 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-23T16:07:03.121 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd-2:14.2.22-0.el8.x86_64 2024-04-23T16:07:03.121 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:07:03.121 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:07:03.143 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:07:03.144 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:07:03.144 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-04-23T16:07:03.144 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd-2:14.2.22-0.el8.x86_64 2024-04-23T16:07:03.144 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-04-23T16:07:03.144 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:07:03.373 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python-rados 2024-04-23T16:07:03.376 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python-rados 2024-04-23T16:07:03.877 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:07:03.877 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:07:03.877 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-04-23T16:07:03.877 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:07:03.877 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-23T16:07:03.878 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd x86_64 2:14.2.22-0.el8 ceph 142 k 2024-04-23T16:07:03.878 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:07:03.878 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-23T16:07:03.878 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-23T16:07:03.878 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-04-23T16:07:03.878 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:07:03.878 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 142 k 2024-04-23T16:07:03.879 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 422 k 2024-04-23T16:07:03.879 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-23T16:07:03.940 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-nbd-14.2.22-0.el8.x86_64.rpm 2.3 MB/s | 142 kB 00:00 2024-04-23T16:07:03.941 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:07:03.941 INFO:teuthology.orchestra.run.smithi103.stdout:Total 2.2 MB/s | 142 kB 00:00 2024-04-23T16:07:03.941 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-23T16:07:03.952 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-23T16:07:03.952 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-23T16:07:04.010 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-23T16:07:04.013 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-23T16:07:04.117 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:10 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:07:04.118 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:09 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:07:04.126 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-23T16:07:04.335 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:07:04.608 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:07:04.989 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-23T16:07:04.989 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:07:04.989 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-23T16:07:04.989 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-2:14.2.22-0.el8.x86_64 2024-04-23T16:07:04.989 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:07:04.989 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:07:05.223 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python-rados 2024-04-23T16:07:05.700 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rados-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:07:05.708 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-rados-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:07:05.751 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:07:05.752 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-04-23T16:07:05.752 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:07:05.756 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:07:05.758 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2024-04-23T16:07:05.758 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:07:05.850 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python-rgw 2024-04-23T16:07:05.878 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python-rgw 2024-04-23T16:07:05.959 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:12 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:07:06.581 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:11 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:07:06.603 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:12 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:07:07.548 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rados-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:07:07.598 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:07:07.599 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-23T16:07:07.599 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:07:07.710 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python-rgw 2024-04-23T16:07:08.163 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rgw-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:07:08.192 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-rgw-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:07:08.212 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:07:08.213 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-04-23T16:07:08.214 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:07:08.241 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:07:08.243 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2024-04-23T16:07:08.243 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:07:08.319 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python-rbd 2024-04-23T16:07:08.365 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python-rbd 2024-04-23T16:07:08.443 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:15 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:07:09.062 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:14 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:07:09.097 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:15 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:07:10.037 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rgw-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:07:10.088 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:07:10.089 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-23T16:07:10.089 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:07:10.202 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python-rbd 2024-04-23T16:07:10.638 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rbd-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:07:10.684 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-rbd-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:07:10.688 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:07:10.689 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-04-23T16:07:10.689 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:07:10.733 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:07:10.734 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2024-04-23T16:07:10.735 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:07:10.800 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python-cephfs 2024-04-23T16:07:10.849 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python-cephfs 2024-04-23T16:07:10.929 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:17 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:07:11.537 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:16 ago on Tue 23 Apr 2024 04:04:55 PM UTC. 2024-04-23T16:07:11.581 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:02:17 ago on Tue 23 Apr 2024 04:04:54 PM UTC. 2024-04-23T16:07:12.519 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rbd-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:07:12.567 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:07:12.568 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-23T16:07:12.569 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:07:12.685 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python-cephfs 2024-04-23T16:07:13.118 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-cephfs-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:07:13.167 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T16:07:13.169 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-04-23T16:07:13.169 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T16:07:13.178 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-cephfs-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:07:13.228 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-04-23T16:07:13.229 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2024-04-23T16:07:13.229 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-04-23T16:07:13.282 DEBUG:teuthology.parallel:result is None 2024-04-23T16:07:13.337 DEBUG:teuthology.parallel:result is None 2024-04-23T16:07:13.411 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:20 ago on Tue 23 Apr 2024 04:04:53 PM UTC. 2024-04-23T16:07:14.997 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-cephfs-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-23T16:07:15.046 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-23T16:07:15.047 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-23T16:07:15.048 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-23T16:07:15.168 DEBUG:teuthology.parallel:result is None 2024-04-23T16:07:15.169 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-23T16:07:15.169 INFO:teuthology.packaging:ref: None 2024-04-23T16:07:15.169 INFO:teuthology.packaging:tag: None 2024-04-23T16:07:15.169 INFO:teuthology.packaging:branch: nautilus 2024-04-23T16:07:15.169 INFO:teuthology.packaging:sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T16:07:15.170 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-04-23T16:07:15.444 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T16:07:15.466 INFO:teuthology.orchestra.run.smithi103.stdout:14.2.22-0.el8 2024-04-23T16:07:15.467 INFO:teuthology.packaging:The installed version of ceph is 14.2.22-0.el8 2024-04-23T16:07:15.467 INFO:teuthology.task.install:The correct ceph version 14.2.22-0 is installed. 2024-04-23T16:07:15.469 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-23T16:07:15.469 INFO:teuthology.packaging:ref: None 2024-04-23T16:07:15.469 INFO:teuthology.packaging:tag: None 2024-04-23T16:07:15.469 INFO:teuthology.packaging:branch: nautilus 2024-04-23T16:07:15.469 INFO:teuthology.packaging:sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T16:07:15.469 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-04-23T16:07:15.635 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T16:07:15.662 INFO:teuthology.orchestra.run.smithi148.stdout:14.2.22-0.el8 2024-04-23T16:07:15.662 INFO:teuthology.packaging:The installed version of ceph is 14.2.22-0.el8 2024-04-23T16:07:15.662 INFO:teuthology.task.install:The correct ceph version 14.2.22-0 is installed. 2024-04-23T16:07:15.664 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-23T16:07:15.664 INFO:teuthology.packaging:ref: None 2024-04-23T16:07:15.664 INFO:teuthology.packaging:tag: None 2024-04-23T16:07:15.664 INFO:teuthology.packaging:branch: nautilus 2024-04-23T16:07:15.664 INFO:teuthology.packaging:sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T16:07:15.665 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-04-23T16:07:15.833 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T16:07:15.859 INFO:teuthology.orchestra.run.smithi188.stdout:14.2.22-0.el8 2024-04-23T16:07:15.859 INFO:teuthology.packaging:The installed version of ceph is 14.2.22-0.el8 2024-04-23T16:07:15.860 INFO:teuthology.task.install:The correct ceph version 14.2.22-0 is installed. 2024-04-23T16:07:15.861 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-23T16:07:15.861 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-23T16:07:15.861 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T16:07:15.890 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-04-23T16:07:15.891 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T16:07:15.923 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T16:07:15.923 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T16:07:15.954 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-23T16:07:15.954 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-23T16:07:15.955 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T16:07:15.982 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T16:07:16.049 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-04-23T16:07:16.049 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T16:07:16.080 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T16:07:16.150 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T16:07:16.151 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T16:07:16.183 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T16:07:16.254 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-23T16:07:16.255 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-23T16:07:16.255 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T16:07:16.282 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T16:07:16.349 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-04-23T16:07:16.350 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T16:07:16.382 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T16:07:16.452 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T16:07:16.452 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T16:07:16.484 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T16:07:16.555 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-23T16:07:16.555 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-23T16:07:16.556 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T16:07:16.582 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T16:07:16.649 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-04-23T16:07:16.649 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T16:07:16.680 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T16:07:16.751 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T16:07:16.751 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T16:07:16.782 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T16:07:16.853 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-23T16:07:16.972 INFO:tasks.cephadm:Config: {'conf': {'mon': {'auth allow insecure global id reclaim': True, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd heartbeat use min delay socket': True, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'mon_bind_msgr2': False, 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'} 2024-04-23T16:07:16.972 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T16:07:16.973 INFO:tasks.cephadm:Cluster fsid is 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:07:16.973 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-23T16:07:16.973 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.103:6789]', 'mon.b': '[v1:172.21.15.148:6789]', 'mon.c': '[v1:172.21.15.188:6789]'} 2024-04-23T16:07:16.974 INFO:tasks.cephadm:First mon is mon.a on smithi103 2024-04-23T16:07:16.974 INFO:tasks.cephadm:First mgr is y 2024-04-23T16:07:16.974 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-23T16:07:16.974 DEBUG:teuthology.orchestra.run.smithi103:> sudo hostname $(hostname -s) 2024-04-23T16:07:17.003 DEBUG:teuthology.orchestra.run.smithi148:> sudo hostname $(hostname -s) 2024-04-23T16:07:17.037 DEBUG:teuthology.orchestra.run.smithi188:> sudo hostname $(hostname -s) 2024-04-23T16:07:17.069 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-23T16:07:17.070 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T16:07:17.210 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/78805/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3211-g5d349943', 'node_name': '172.21.5.31+adami01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3211.g5d349943'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2024-04-22 19:38:22.439018', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-yuri3-testing-2024-04-05-0825', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/centos/8/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-23T16:07:17.347 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-yuri3-testing-2024-04-05-0825, sha1 5d349943c59c9485df060d6adb0594f3940ec0eb from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T16:07:17.348 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/centos/8/x86_64/flavors/default/cephadm 2024-04-23T16:07:17.349 DEBUG:teuthology.orchestra.run.smithi103:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T16:07:18.231 INFO:teuthology.orchestra.run.smithi103.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2294149 Apr 23 16:07 /home/ubuntu/cephtest/cephadm 2024-04-23T16:07:18.231 DEBUG:teuthology.orchestra.run.smithi148:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T16:07:18.797 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2294149 Apr 23 16:07 /home/ubuntu/cephtest/cephadm 2024-04-23T16:07:18.797 DEBUG:teuthology.orchestra.run.smithi188:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T16:07:19.354 INFO:teuthology.orchestra.run.smithi188.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2294149 Apr 23 16:07 /home/ubuntu/cephtest/cephadm 2024-04-23T16:07:19.355 DEBUG:teuthology.orchestra.run.smithi103:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T16:07:19.378 DEBUG:teuthology.orchestra.run.smithi148:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T16:07:19.401 DEBUG:teuthology.orchestra.run.smithi188:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T16:07:19.437 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb on all hosts... 2024-04-23T16:07:19.437 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb pull 2024-04-23T16:07:19.440 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb pull 2024-04-23T16:07:19.443 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb pull 2024-04-23T16:07:20.568 INFO:teuthology.orchestra.run.smithi148.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb... 2024-04-23T16:07:20.606 INFO:teuthology.orchestra.run.smithi188.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb... 2024-04-23T16:07:20.765 INFO:teuthology.orchestra.run.smithi103.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb... 2024-04-23T16:07:48.826 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:07:48.827 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_version": "ceph version 19.0.0-3211-g5d349943 (5d349943c59c9485df060d6adb0594f3940ec0eb) squid (dev)", 2024-04-23T16:07:48.827 INFO:teuthology.orchestra.run.smithi103.stdout: "image_id": "a2b4ee7af118c4f19dfec4d7742042324c1087dcf2dc4691a617e25542e09461", 2024-04-23T16:07:48.827 INFO:teuthology.orchestra.run.smithi103.stdout: "repo_digests": [ 2024-04-23T16:07:48.827 INFO:teuthology.orchestra.run.smithi103.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:4b8fadbb2850222b1a0d53f94662568f540a1d3f86861435d9eb9005f9a5b779" 2024-04-23T16:07:48.827 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:07:48.827 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:07:51.313 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:07:51.313 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_version": "ceph version 19.0.0-3211-g5d349943 (5d349943c59c9485df060d6adb0594f3940ec0eb) squid (dev)", 2024-04-23T16:07:51.313 INFO:teuthology.orchestra.run.smithi148.stdout: "image_id": "a2b4ee7af118c4f19dfec4d7742042324c1087dcf2dc4691a617e25542e09461", 2024-04-23T16:07:51.313 INFO:teuthology.orchestra.run.smithi148.stdout: "repo_digests": [ 2024-04-23T16:07:51.313 INFO:teuthology.orchestra.run.smithi148.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:4b8fadbb2850222b1a0d53f94662568f540a1d3f86861435d9eb9005f9a5b779" 2024-04-23T16:07:51.314 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:07:51.314 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:07:57.989 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:07:57.989 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 19.0.0-3211-g5d349943 (5d349943c59c9485df060d6adb0594f3940ec0eb) squid (dev)", 2024-04-23T16:07:57.989 INFO:teuthology.orchestra.run.smithi188.stdout: "image_id": "a2b4ee7af118c4f19dfec4d7742042324c1087dcf2dc4691a617e25542e09461", 2024-04-23T16:07:57.989 INFO:teuthology.orchestra.run.smithi188.stdout: "repo_digests": [ 2024-04-23T16:07:57.989 INFO:teuthology.orchestra.run.smithi188.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:4b8fadbb2850222b1a0d53f94662568f540a1d3f86861435d9eb9005f9a5b779" 2024-04-23T16:07:57.989 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:07:57.989 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:07:58.015 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph 2024-04-23T16:07:58.053 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /etc/ceph 2024-04-23T16:07:58.090 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /etc/ceph 2024-04-23T16:07:58.120 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /etc/ceph 2024-04-23T16:07:58.152 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 777 /etc/ceph 2024-04-23T16:07:58.182 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 777 /etc/ceph 2024-04-23T16:07:58.210 INFO:tasks.cephadm:Writing seed config... 2024-04-23T16:07:58.211 INFO:tasks.cephadm: override: [mon] auth allow insecure global id reclaim = True 2024-04-23T16:07:58.211 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-23T16:07:58.211 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-23T16:07:58.211 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-23T16:07:58.211 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-04-23T16:07:58.211 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-04-23T16:07:58.211 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-04-23T16:07:58.211 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-04-23T16:07:58.211 INFO:tasks.cephadm: override: [mon] mon scrub interval = 300 2024-04-23T16:07:58.211 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-04-23T16:07:58.212 INFO:tasks.cephadm: override: [client.2] debug ms = 1 2024-04-23T16:07:58.212 INFO:tasks.cephadm: override: [client.2] debug objecter = 20 2024-04-23T16:07:58.212 INFO:tasks.cephadm: override: [client.2] debug rados = 20 2024-04-23T16:07:58.212 INFO:tasks.cephadm: override: [global] mon client directed command retry = 5 2024-04-23T16:07:58.212 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-04-23T16:07:58.212 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-04-23T16:07:58.212 INFO:tasks.cephadm: override: [global] ms inject socket failures = 5000 2024-04-23T16:07:58.212 INFO:tasks.cephadm: override: [global] ms type = async 2024-04-23T16:07:58.212 INFO:tasks.cephadm: override: [global] osd_pool_default_min_size = 2 2024-04-23T16:07:58.212 INFO:tasks.cephadm: override: [global] osd_pool_default_size = 2 2024-04-23T16:07:58.212 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-23T16:07:58.212 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-23T16:07:58.212 INFO:tasks.cephadm: override: [osd] bluestore zero block detection = True 2024-04-23T16:07:58.212 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-23T16:07:58.212 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-23T16:07:58.213 INFO:tasks.cephadm: override: [osd] osd backoff on peering = True 2024-04-23T16:07:58.213 INFO:tasks.cephadm: override: [osd] osd blocked scrub grace period = 3600 2024-04-23T16:07:58.213 INFO:tasks.cephadm: override: [osd] osd debug verify cached snaps = True 2024-04-23T16:07:58.213 INFO:tasks.cephadm: override: [osd] osd debug verify missing on start = True 2024-04-23T16:07:58.213 INFO:tasks.cephadm: override: [osd] osd heartbeat use min delay socket = True 2024-04-23T16:07:58.213 INFO:tasks.cephadm: override: [osd] osd map cache size = 1 2024-04-23T16:07:58.213 INFO:tasks.cephadm: override: [osd] osd max backfills = 6 2024-04-23T16:07:58.213 INFO:tasks.cephadm: override: [osd] osd max markdown count = 1000 2024-04-23T16:07:58.213 INFO:tasks.cephadm: override: [osd] osd mclock override recovery settings = True 2024-04-23T16:07:58.213 INFO:tasks.cephadm: override: [osd] osd mclock profile = high_recovery_ops 2024-04-23T16:07:58.213 INFO:tasks.cephadm: override: [osd] osd op queue = debug_random 2024-04-23T16:07:58.213 INFO:tasks.cephadm: override: [osd] osd op queue cut off = debug_random 2024-04-23T16:07:58.213 INFO:tasks.cephadm: override: [osd] osd scrub during recovery = False 2024-04-23T16:07:58.213 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-04-23T16:07:58.213 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-04-23T16:07:58.214 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-23T16:07:58.214 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-23T16:07:58.214 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-23T16:07:58.235 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 8e3594f2-018b-11ef-bc93-c7b262605968 mon client directed command retry = 5 mon election default strategy = 1 ms bind msgr2 = False ms inject socket failures = 5000 ms type = async osd_pool_default_min_size = 2 osd_pool_default_size = 2 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = True osd debug verify missing on start = True osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore zero block detection = True debug ms = 1 debug osd = 20 osd backoff on peering = True osd blocked scrub grace period = 3600 osd heartbeat use min delay socket = True osd map cache size = 1 osd max backfills = 6 osd max markdown count = 1000 osd mclock override recovery settings = True osd scrub during recovery = False osd scrub min interval = 60 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false auth allow insecure global id reclaim = True debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon scrub interval = 300 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.2] debug ms = 1 debug objecter = 20 debug rados = 20 2024-04-23T16:07:58.236 DEBUG:teuthology.orchestra.run.smithi103:mon.a> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mon.a.service 2024-04-23T16:07:58.278 DEBUG:teuthology.orchestra.run.smithi103:mgr.y> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mgr.y.service 2024-04-23T16:07:58.307 INFO:journalctl@ceph.mon.a.smithi103.stdout:-- Logs begin at Tue 2024-04-23 15:57:24 UTC. -- 2024-04-23T16:07:58.308 INFO:tasks.cephadm:Bootstrapping... 2024-04-23T16:07:58.309 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb -v bootstrap --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.103:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-23T16:07:58.335 INFO:journalctl@ceph.mgr.y.smithi103.stdout:-- Logs begin at Tue 2024-04-23 15:57:24 UTC. -- 2024-04-23T16:07:58.518 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-23T16:07:58.519 INFO:teuthology.orchestra.run.smithi103.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb', '-v', 'bootstrap', '--fsid', '8e3594f2-018b-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.103:6789]', '--skip-admin-label'] 2024-04-23T16:07:58.519 INFO:teuthology.orchestra.run.smithi103.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-23T16:07:58.519 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying podman|docker is present... 2024-04-23T16:07:58.556 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-23T16:07:58.556 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying lvm2 is present... 2024-04-23T16:07:58.556 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying time synchronization is in place... 2024-04-23T16:07:58.563 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-23T16:07:58.563 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-23T16:07:58.569 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-23T16:07:58.569 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-04-23T16:07:58.575 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout enabled 2024-04-23T16:07:58.580 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout active 2024-04-23T16:07:58.581 INFO:teuthology.orchestra.run.smithi103.stdout:Unit chronyd.service is enabled and running 2024-04-23T16:07:58.581 INFO:teuthology.orchestra.run.smithi103.stdout:Repeating the final host check... 2024-04-23T16:07:58.613 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-23T16:07:58.613 INFO:teuthology.orchestra.run.smithi103.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-23T16:07:58.613 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl is present 2024-04-23T16:07:58.613 INFO:teuthology.orchestra.run.smithi103.stdout:lvcreate is present 2024-04-23T16:07:58.619 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-23T16:07:58.619 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-23T16:07:58.625 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-23T16:07:58.625 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-04-23T16:07:58.630 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout enabled 2024-04-23T16:07:58.636 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout active 2024-04-23T16:07:58.636 INFO:teuthology.orchestra.run.smithi103.stdout:Unit chronyd.service is enabled and running 2024-04-23T16:07:58.636 INFO:teuthology.orchestra.run.smithi103.stdout:Host looks OK 2024-04-23T16:07:58.636 INFO:teuthology.orchestra.run.smithi103.stdout:Cluster fsid: 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:07:58.637 INFO:teuthology.orchestra.run.smithi103.stdout:Acquiring lock 140484907040272 on /run/cephadm/8e3594f2-018b-11ef-bc93-c7b262605968.lock 2024-04-23T16:07:58.637 INFO:teuthology.orchestra.run.smithi103.stdout:Lock 140484907040272 acquired on /run/cephadm/8e3594f2-018b-11ef-bc93-c7b262605968.lock 2024-04-23T16:07:58.637 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying IP 172.21.15.103 port 6789 ... 2024-04-23T16:07:58.637 INFO:teuthology.orchestra.run.smithi103.stdout:Base mon IP(s) is [172.21.15.103:6789], mon addrv is [v1:172.21.15.103:6789] 2024-04-23T16:07:58.641 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.103 metric 100 2024-04-23T16:07:58.641 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.103 metric 100 2024-04-23T16:07:58.644 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-23T16:07:58.644 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-23T16:07:58.644 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1799sec hoplimit 64 pref medium 2024-04-23T16:07:58.647 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-23T16:07:58.647 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-23T16:07:58.647 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-23T16:07:58.647 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-23T16:07:58.647 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-04-23T16:07:58.647 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-23T16:07:58.647 INFO:teuthology.orchestra.run.smithi103.stdout:Mon IP `172.21.15.103` is in CIDR network `172.21.0.0/20` 2024-04-23T16:07:58.647 INFO:teuthology.orchestra.run.smithi103.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-04-23T16:07:58.648 INFO:teuthology.orchestra.run.smithi103.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-23T16:07:58.649 INFO:teuthology.orchestra.run.smithi103.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb... 2024-04-23T16:08:01.687 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stdout a2b4ee7af118c4f19dfec4d7742042324c1087dcf2dc4691a617e25542e09461 2024-04-23T16:08:01.687 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb... 2024-04-23T16:08:01.687 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Getting image source signatures 2024-04-23T16:08:01.688 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Copying blob sha256:81d769813be2554f547c4799c95baf1c9efd7f6096ecb47001102f9c95a92204 2024-04-23T16:08:01.688 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-23T16:08:01.688 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Copying config sha256:a2b4ee7af118c4f19dfec4d7742042324c1087dcf2dc4691a617e25542e09461 2024-04-23T16:08:01.688 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-23T16:08:05.460 INFO:teuthology.orchestra.run.smithi103.stdout:ceph: stdout ceph version 19.0.0-3211-g5d349943 (5d349943c59c9485df060d6adb0594f3940ec0eb) squid (dev) 2024-04-23T16:08:05.460 INFO:teuthology.orchestra.run.smithi103.stdout:Ceph version: ceph version 19.0.0-3211-g5d349943 (5d349943c59c9485df060d6adb0594f3940ec0eb) squid (dev) 2024-04-23T16:08:05.460 INFO:teuthology.orchestra.run.smithi103.stdout:Extracting ceph user uid/gid from container image... 2024-04-23T16:08:06.574 INFO:teuthology.orchestra.run.smithi103.stdout:stat: stdout 167 167 2024-04-23T16:08:06.574 INFO:teuthology.orchestra.run.smithi103.stdout:Creating initial keys... 2024-04-23T16:08:07.743 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph-authtool: stdout AQDn3Cdm5+fzAhAA+z1ZB5nRxmbluNmvt/mrMw== 2024-04-23T16:08:08.806 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph-authtool: stdout AQDo3CdmADySCRAAP+PrD82sSE1Xww94ZGvHvA== 2024-04-23T16:08:09.963 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph-authtool: stdout AQDp3Cdmk7LiDxAAqQA6M2Es4lc7mM6budNGPg== 2024-04-23T16:08:09.964 INFO:teuthology.orchestra.run.smithi103.stdout:Creating initial monmap... 2024-04-23T16:08:11.201 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-23T16:08:11.201 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-23T16:08:11.201 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:08:11.201 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-23T16:08:11.201 INFO:teuthology.orchestra.run.smithi103.stdout:monmaptool for a [v1:172.21.15.103:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-23T16:08:11.201 INFO:teuthology.orchestra.run.smithi103.stdout:setting min_mon_release = quincy 2024-04-23T16:08:11.202 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: set fsid to 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:08:11.202 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-23T16:08:11.202 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:08:11.202 INFO:teuthology.orchestra.run.smithi103.stdout:Creating mon... 2024-04-23T16:08:12.329 INFO:teuthology.orchestra.run.smithi103.stdout:create mon.a on 2024-04-23T16:08:12.457 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-04-23T16:08:12.548 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-23T16:08:12.672 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-8e3594f2-018b-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-8e3594f2-018b-11ef-bc93-c7b262605968.target. 2024-04-23T16:08:12.672 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-8e3594f2-018b-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-8e3594f2-018b-11ef-bc93-c7b262605968.target. 2024-04-23T16:08:12.851 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mon.a 2024-04-23T16:08:12.851 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-23T16:08:12.938 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8e3594f2-018b-11ef-bc93-c7b262605968.target.wants/ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-8e3594f2-018b-11ef-bc93-c7b262605968@.service. 2024-04-23T16:08:13.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:12 smithi103 systemd[1]: Starting Ceph mon.a for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T16:08:13.464 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T16:08:13.464 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout disabled 2024-04-23T16:08:13.471 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T16:08:13.472 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-04-23T16:08:13.472 INFO:teuthology.orchestra.run.smithi103.stdout:firewalld.service is not enabled 2024-04-23T16:08:13.472 INFO:teuthology.orchestra.run.smithi103.stdout:Not possible to enable service . firewalld.service is not available 2024-04-23T16:08:13.472 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mon to start... 2024-04-23T16:08:13.472 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mon... 2024-04-23T16:08:13.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:13 smithi103 bash[39106]: 3c21130fd0b9f6e5f83f9569d5284b744ac9faa75a81c299255fbef24b4acd66 2024-04-23T16:08:13.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:13 smithi103 systemd[1]: Started Ceph mon.a for 8e3594f2-018b-11ef-bc93-c7b262605968. 2024-04-23T16:08:13.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:13 smithi103 ceph-mon[39143]: mkfs 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:08:13.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:13 smithi103 ceph-mon[39143]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T16:08:14.679 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout cluster: 2024-04-23T16:08:14.679 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout id: 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:08:14.679 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-23T16:08:14.680 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-04-23T16:08:14.680 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout services: 2024-04-23T16:08:14.680 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.668692s) 2024-04-23T16:08:14.680 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-23T16:08:14.680 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-23T16:08:14.680 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-04-23T16:08:14.680 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout data: 2024-04-23T16:08:14.680 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-23T16:08:14.680 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-23T16:08:14.680 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-23T16:08:14.680 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout pgs: 2024-04-23T16:08:14.680 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-04-23T16:08:14.680 INFO:teuthology.orchestra.run.smithi103.stdout:mon is available 2024-04-23T16:08:14.680 INFO:teuthology.orchestra.run.smithi103.stdout:Assimilating anything we can from ceph.conf... 2024-04-23T16:08:14.681 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:14 smithi103 ceph-mon[39143]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T16:08:14.681 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:14 smithi103 ceph-mon[39143]: monmap epoch 1 2024-04-23T16:08:14.681 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:14 smithi103 ceph-mon[39143]: fsid 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:08:14.681 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:14 smithi103 ceph-mon[39143]: last_changed 2024-04-23T16:08:10.387384+0000 2024-04-23T16:08:14.681 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:14 smithi103 ceph-mon[39143]: created 2024-04-23T16:08:10.387384+0000 2024-04-23T16:08:14.681 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:14 smithi103 ceph-mon[39143]: min_mon_release 19 (squid) 2024-04-23T16:08:14.681 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:14 smithi103 ceph-mon[39143]: election_strategy: 1 2024-04-23T16:08:14.681 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:14 smithi103 ceph-mon[39143]: 0: v1:172.21.15.103:6789/0 mon.a 2024-04-23T16:08:14.681 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:14 smithi103 ceph-mon[39143]: fsmap 2024-04-23T16:08:14.681 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:14 smithi103 ceph-mon[39143]: osdmap e1: 0 total, 0 up, 0 in 2024-04-23T16:08:14.681 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:14 smithi103 ceph-mon[39143]: mgrmap e1: no daemons active 2024-04-23T16:08:14.681 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:14 smithi103 ceph-mon[39143]: from='client.? v1:172.21.15.103:0/133120956' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-23T16:08:15.597 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:15 smithi103 ceph-mon[39143]: from='client.? v1:172.21.15.103:0/633515590' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-23T16:08:15.598 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:15 smithi103 ceph-mon[39143]: from='client.? v1:172.21.15.103:0/633515590' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-23T16:08:15.835 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-04-23T16:08:15.835 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [global] 2024-04-23T16:08:15.835 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout fsid = 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:08:15.835 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-23T16:08:15.836 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.103:6789] 2024-04-23T16:08:15.836 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-23T16:08:15.836 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-23T16:08:15.836 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-23T16:08:15.836 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-23T16:08:15.836 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-04-23T16:08:15.836 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-23T16:08:15.836 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-23T16:08:15.836 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-04-23T16:08:15.836 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [osd] 2024-04-23T16:08:15.836 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-23T16:08:15.836 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-23T16:08:15.836 INFO:teuthology.orchestra.run.smithi103.stdout:Generating new minimal ceph.conf... 2024-04-23T16:08:16.778 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:16 smithi103 ceph-mon[39143]: from='client.? v1:172.21.15.103:0/2888079151' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:08:17.074 INFO:teuthology.orchestra.run.smithi103.stdout:Restarting the monitor... 2024-04-23T16:08:17.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:17 smithi103 systemd[1]: Stopping Ceph mon.a for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T16:08:17.624 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:17 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39120]: 2024-04-23T16:08:17.353+0000 7fc6cdf42700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T16:08:17.624 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:17 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39120]: 2024-04-23T16:08:17.353+0000 7fc6cdf42700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-23T16:08:18.017 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:17 smithi103 bash[39472]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a 2024-04-23T16:08:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 systemd[1]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mon.a.service: Succeeded. 2024-04-23T16:08:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 systemd[1]: Stopped Ceph mon.a for 8e3594f2-018b-11ef-bc93-c7b262605968. 2024-04-23T16:08:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 systemd[1]: Starting Ceph mon.a for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T16:08:18.724 INFO:teuthology.orchestra.run.smithi103.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-23T16:08:18.973 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T16:08:18.973 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: ceph version 19.0.0-3211-g5d349943 (5d349943c59c9485df060d6adb0594f3940ec0eb) squid (dev), process ceph-mon, pid 7 2024-04-23T16:08:18.973 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: pidfile_write: ignore empty --pid-file 2024-04-23T16:08:18.973 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: load: jerasure load: lrc 2024-04-23T16:08:18.974 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: RocksDB version: 7.9.2 2024-04-23T16:08:18.974 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Git sha 0 2024-04-23T16:08:18.974 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Compile date 2024-04-22 18:35:14 2024-04-23T16:08:18.974 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: DB SUMMARY 2024-04-23T16:08:18.974 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: DB Session ID: U1PSORSLU1ZAPJJ90DPW 2024-04-23T16:08:18.974 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: CURRENT file: CURRENT 2024-04-23T16:08:18.974 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T16:08:18.974 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-23T16:08:18.974 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-23T16:08:18.974 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 98293 ; 2024-04-23T16:08:18.974 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.error_if_exists: 0 2024-04-23T16:08:18.974 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.create_if_missing: 0 2024-04-23T16:08:18.975 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.paranoid_checks: 1 2024-04-23T16:08:18.975 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-23T16:08:18.975 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T16:08:18.975 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-23T16:08:18.975 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.env: 0x5580b8158cc0 2024-04-23T16:08:18.975 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.fs: PosixFileSystem 2024-04-23T16:08:18.975 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.info_log: 0x5580b8bdfc40 2024-04-23T16:08:18.975 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T16:08:18.975 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.statistics: (nil) 2024-04-23T16:08:18.975 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.use_fsync: 0 2024-04-23T16:08:18.976 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_log_file_size: 0 2024-04-23T16:08:18.976 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T16:08:18.976 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T16:08:18.976 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T16:08:18.976 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T16:08:18.976 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.allow_fallocate: 1 2024-04-23T16:08:18.976 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T16:08:18.976 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T16:08:18.976 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.use_direct_reads: 0 2024-04-23T16:08:18.977 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T16:08:18.977 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T16:08:18.977 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.db_log_dir: 2024-04-23T16:08:18.977 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.wal_dir: 2024-04-23T16:08:18.977 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T16:08:18.977 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T16:08:18.977 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T16:08:18.977 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T16:08:18.977 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T16:08:18.977 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T16:08:18.977 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T16:08:18.977 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T16:08:18.977 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.write_buffer_manager: 0x5580b8b730e0 2024-04-23T16:08:18.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T16:08:18.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T16:08:18.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T16:08:18.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T16:08:18.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T16:08:18.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T16:08:18.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T16:08:18.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T16:08:18.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.unordered_write: 0 2024-04-23T16:08:18.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T16:08:18.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T16:08:18.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T16:08:18.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T16:08:18.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.row_cache: None 2024-04-23T16:08:18.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.wal_filter: None 2024-04-23T16:08:18.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T16:08:18.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T16:08:18.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.two_write_queues: 0 2024-04-23T16:08:18.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T16:08:18.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.wal_compression: 0 2024-04-23T16:08:18.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.atomic_flush: 0 2024-04-23T16:08:18.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T16:08:18.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T16:08:18.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T16:08:18.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.log_readahead_size: 0 2024-04-23T16:08:18.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T16:08:18.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T16:08:18.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T16:08:18.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T16:08:18.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T16:08:18.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T16:08:18.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-23T16:08:18.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_background_jobs: 2 2024-04-23T16:08:18.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_background_compactions: -1 2024-04-23T16:08:18.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_subcompactions: 1 2024-04-23T16:08:18.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T16:08:18.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T16:08:18.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T16:08:18.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T16:08:18.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T16:08:18.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T16:08:18.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T16:08:18.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T16:08:18.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_open_files: -1 2024-04-23T16:08:18.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T16:08:18.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T16:08:18.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T16:08:18.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T16:08:18.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_background_flushes: -1 2024-04-23T16:08:18.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Compression algorithms supported: 2024-04-23T16:08:18.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T16:08:18.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: kZSTD supported: 0 2024-04-23T16:08:18.982 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: kXpressCompression supported: 0 2024-04-23T16:08:18.982 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T16:08:18.982 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: kZlibCompression supported: 1 2024-04-23T16:08:18.982 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: kSnappyCompression supported: 1 2024-04-23T16:08:18.982 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: kLZ4Compression supported: 1 2024-04-23T16:08:18.982 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: kBZip2Compression supported: 0 2024-04-23T16:08:18.982 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T16:08:18.982 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-23T16:08:18.982 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-23T16:08:18.982 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-23T16:08:18.982 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T16:08:18.983 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.merge_operator: 2024-04-23T16:08:18.983 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compaction_filter: None 2024-04-23T16:08:18.983 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T16:08:18.983 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T16:08:18.983 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T16:08:18.983 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T16:08:18.983 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5580b8bdfe00) 2024-04-23T16:08:18.983 INFO:journalctl@ceph.mon.a.smithi103.stdout: cache_index_and_filter_blocks: 1 2024-04-23T16:08:18.983 INFO:journalctl@ceph.mon.a.smithi103.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T16:08:18.983 INFO:journalctl@ceph.mon.a.smithi103.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T16:08:18.983 INFO:journalctl@ceph.mon.a.smithi103.stdout: pin_top_level_index_and_filter: 1 2024-04-23T16:08:18.983 INFO:journalctl@ceph.mon.a.smithi103.stdout: index_type: 0 2024-04-23T16:08:18.983 INFO:journalctl@ceph.mon.a.smithi103.stdout: data_block_index_type: 0 2024-04-23T16:08:18.983 INFO:journalctl@ceph.mon.a.smithi103.stdout: index_shortening: 1 2024-04-23T16:08:18.984 INFO:journalctl@ceph.mon.a.smithi103.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T16:08:18.984 INFO:journalctl@ceph.mon.a.smithi103.stdout: checksum: 4 2024-04-23T16:08:18.984 INFO:journalctl@ceph.mon.a.smithi103.stdout: no_block_cache: 0 2024-04-23T16:08:18.984 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_cache: 0x5580b8ba51f0 2024-04-23T16:08:18.984 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_cache_name: BinnedLRUCache 2024-04-23T16:08:18.984 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_cache_options: 2024-04-23T16:08:18.984 INFO:journalctl@ceph.mon.a.smithi103.stdout: capacity : 536870912 2024-04-23T16:08:18.984 INFO:journalctl@ceph.mon.a.smithi103.stdout: num_shard_bits : 4 2024-04-23T16:08:18.984 INFO:journalctl@ceph.mon.a.smithi103.stdout: strict_capacity_limit : 0 2024-04-23T16:08:18.984 INFO:journalctl@ceph.mon.a.smithi103.stdout: high_pri_pool_ratio: 0.000 2024-04-23T16:08:18.984 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_cache_compressed: (nil) 2024-04-23T16:08:18.984 INFO:journalctl@ceph.mon.a.smithi103.stdout: persistent_cache: (nil) 2024-04-23T16:08:18.984 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_size: 4096 2024-04-23T16:08:18.984 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_size_deviation: 10 2024-04-23T16:08:18.985 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_restart_interval: 16 2024-04-23T16:08:18.985 INFO:journalctl@ceph.mon.a.smithi103.stdout: index_block_restart_interval: 1 2024-04-23T16:08:18.985 INFO:journalctl@ceph.mon.a.smithi103.stdout: metadata_block_size: 4096 2024-04-23T16:08:18.985 INFO:journalctl@ceph.mon.a.smithi103.stdout: partition_filters: 0 2024-04-23T16:08:18.985 INFO:journalctl@ceph.mon.a.smithi103.stdout: use_delta_encoding: 1 2024-04-23T16:08:18.985 INFO:journalctl@ceph.mon.a.smithi103.stdout: filter_policy: bloomfilter 2024-04-23T16:08:18.985 INFO:journalctl@ceph.mon.a.smithi103.stdout: whole_key_filtering: 1 2024-04-23T16:08:18.985 INFO:journalctl@ceph.mon.a.smithi103.stdout: verify_compression: 0 2024-04-23T16:08:18.985 INFO:journalctl@ceph.mon.a.smithi103.stdout: read_amp_bytes_per_bit: 0 2024-04-23T16:08:18.985 INFO:journalctl@ceph.mon.a.smithi103.stdout: format_version: 5 2024-04-23T16:08:18.985 INFO:journalctl@ceph.mon.a.smithi103.stdout: enable_index_compression: 1 2024-04-23T16:08:18.985 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_align: 0 2024-04-23T16:08:18.985 INFO:journalctl@ceph.mon.a.smithi103.stdout: max_auto_readahead_size: 262144 2024-04-23T16:08:18.985 INFO:journalctl@ceph.mon.a.smithi103.stdout: prepopulate_block_cache: 0 2024-04-23T16:08:18.986 INFO:journalctl@ceph.mon.a.smithi103.stdout: initial_auto_readahead_size: 8192 2024-04-23T16:08:18.986 INFO:journalctl@ceph.mon.a.smithi103.stdout: num_file_reads_for_auto_readahead: 2 2024-04-23T16:08:18.986 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T16:08:18.986 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T16:08:18.986 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compression: NoCompression 2024-04-23T16:08:18.986 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T16:08:18.986 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T16:08:18.986 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T16:08:18.986 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.num_levels: 7 2024-04-23T16:08:18.986 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T16:08:18.986 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T16:08:18.986 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T16:08:18.987 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T16:08:18.987 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T16:08:18.987 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T16:08:18.987 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T16:08:18.987 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T16:08:18.988 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T16:08:18.988 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T16:08:18.988 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-23T16:08:18.988 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-23T16:08:18.988 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T16:08:18.988 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T16:08:18.988 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T16:08:18.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T16:08:18.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T16:08:18.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-23T16:08:18.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T16:08:18.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T16:08:18.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-23T16:08:18.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T16:08:18.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T16:08:18.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T16:08:18.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T16:08:18.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T16:08:18.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T16:08:18.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T16:08:18.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T16:08:18.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T16:08:18.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T16:08:18.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T16:08:18.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T16:08:18.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T16:08:18.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T16:08:18.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T16:08:18.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T16:08:18.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T16:08:18.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-23T16:08:18.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.arena_block_size: 1048576 2024-04-23T16:08:18.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T16:08:18.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T16:08:18.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T16:08:18.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T16:08:18.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T16:08:18.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T16:08:18.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T16:08:18.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T16:08:18.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T16:08:18.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T16:08:18.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T16:08:18.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T16:08:18.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T16:08:18.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-23T16:08:18.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.inplace_update_support: 0 2024-04-23T16:08:18.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T16:08:18.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T16:08:18.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T16:08:18.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T16:08:18.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.bloom_locality: 0 2024-04-23T16:08:18.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.max_successive_merges: 0 2024-04-23T16:08:18.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T16:08:18.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T16:08:18.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T16:08:18.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T16:08:18.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.ttl: 2592000 2024-04-23T16:08:18.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T16:08:18.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-23T16:08:18.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-23T16:08:18.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.enable_blob_files: false 2024-04-23T16:08:18.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.min_blob_size: 0 2024-04-23T16:08:18.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T16:08:18.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T16:08:18.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T16:08:18.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T16:08:18.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-23T16:08:18.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-23T16:08:18.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.blob_file_starting_level: 0 2024-04-23T16:08:18.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-23T16:08:18.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-23T16:08:18.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-23T16:08:18.994 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b8b2bd1e-6f3f-4a04-a6c3-c3c156b5ef31 2024-04-23T16:08:18.994 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713888498662609, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-23T16:08:18.994 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-23T16:08:18.994 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713888498663538, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93679, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 306, "table_properties": {"data_size": 91680, "index_size": 255, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13221, "raw_average_key_size": 49, "raw_value_size": 84490, "raw_average_value_size": 318, "num_data_blocks": 11, "num_entries": 265, "num_filter_entries": 265, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713888498, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b8b2bd1e-6f3f-4a04-a6c3-c3c156b5ef31", "db_session_id": "U1PSORSLU1ZAPJJ90DPW", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-23T16:08:18.994 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713888498663618, "job": 1, "event": "recovery_finished"} 2024-04-23T16:08:18.994 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-23T16:08:18.994 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T16:08:18.994 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5580b8cba000 2024-04-23T16:08:18.994 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: DB pointer 0x5580b8ca4000 2024-04-23T16:08:18.994 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-23T16:08:18.994 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-23T16:08:18.994 INFO:journalctl@ceph.mon.a.smithi103.stdout: ** DB Stats ** 2024-04-23T16:08:18.995 INFO:journalctl@ceph.mon.a.smithi103.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T16:08:18.995 INFO:journalctl@ceph.mon.a.smithi103.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T16:08:18.995 INFO:journalctl@ceph.mon.a.smithi103.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T16:08:18.995 INFO:journalctl@ceph.mon.a.smithi103.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T16:08:18.995 INFO:journalctl@ceph.mon.a.smithi103.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T16:08:18.995 INFO:journalctl@ceph.mon.a.smithi103.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T16:08:18.995 INFO:journalctl@ceph.mon.a.smithi103.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T16:08:18.995 INFO:journalctl@ceph.mon.a.smithi103.stdout: 2024-04-23T16:08:18.995 INFO:journalctl@ceph.mon.a.smithi103.stdout: ** Compaction Stats [default] ** 2024-04-23T16:08:18.995 INFO:journalctl@ceph.mon.a.smithi103.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T16:08:18.995 INFO:journalctl@ceph.mon.a.smithi103.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-23T16:08:18.995 INFO:journalctl@ceph.mon.a.smithi103.stdout: L0 2/0 93.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T16:08:18.995 INFO:journalctl@ceph.mon.a.smithi103.stdout: Sum 2/0 93.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T16:08:18.996 INFO:journalctl@ceph.mon.a.smithi103.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T16:08:18.996 INFO:journalctl@ceph.mon.a.smithi103.stdout: 2024-04-23T16:08:18.996 INFO:journalctl@ceph.mon.a.smithi103.stdout: ** Compaction Stats [default] ** 2024-04-23T16:08:18.996 INFO:journalctl@ceph.mon.a.smithi103.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T16:08:18.996 INFO:journalctl@ceph.mon.a.smithi103.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T16:08:18.996 INFO:journalctl@ceph.mon.a.smithi103.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 156.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T16:08:18.996 INFO:journalctl@ceph.mon.a.smithi103.stdout: 2024-04-23T16:08:18.996 INFO:journalctl@ceph.mon.a.smithi103.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-23T16:08:18.996 INFO:journalctl@ceph.mon.a.smithi103.stdout: 2024-04-23T16:08:18.996 INFO:journalctl@ceph.mon.a.smithi103.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T16:08:18.996 INFO:journalctl@ceph.mon.a.smithi103.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T16:08:18.996 INFO:journalctl@ceph.mon.a.smithi103.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T16:08:18.996 INFO:journalctl@ceph.mon.a.smithi103.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T16:08:18.996 INFO:journalctl@ceph.mon.a.smithi103.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T16:08:18.997 INFO:journalctl@ceph.mon.a.smithi103.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T16:08:18.997 INFO:journalctl@ceph.mon.a.smithi103.stdout: Cumulative compaction: 0.00 GB write, 28.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T16:08:18.997 INFO:journalctl@ceph.mon.a.smithi103.stdout: Interval compaction: 0.00 GB write, 28.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T16:08:18.997 INFO:journalctl@ceph.mon.a.smithi103.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T16:08:18.997 INFO:journalctl@ceph.mon.a.smithi103.stdout: Block cache BinnedLRUCache@0x5580b8ba51f0#7 capacity: 512.00 MB usage: 1.33 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-23T16:08:18.997 INFO:journalctl@ceph.mon.a.smithi103.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-04-23T16:08:18.997 INFO:journalctl@ceph.mon.a.smithi103.stdout: 2024-04-23T16:08:18.997 INFO:journalctl@ceph.mon.a.smithi103.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T16:08:18.997 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: starting mon.a rank 0 at public addrs v1:172.21.15.103:6789/0 at bind addrs v1:172.21.15.103:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:08:18.997 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: mon.a@-1(???) e1 preinit fsid 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:08:18.997 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: mon.a@-1(???).mds e1 new map 2024-04-23T16:08:18.997 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: mon.a@-1(???).mds e1 print_map 2024-04-23T16:08:18.997 INFO:journalctl@ceph.mon.a.smithi103.stdout: e1 2024-04-23T16:08:18.997 INFO:journalctl@ceph.mon.a.smithi103.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T16:08:18.998 INFO:journalctl@ceph.mon.a.smithi103.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T16:08:18.998 INFO:journalctl@ceph.mon.a.smithi103.stdout: legacy client fscid: -1 2024-04-23T16:08:18.998 INFO:journalctl@ceph.mon.a.smithi103.stdout: 2024-04-23T16:08:18.998 INFO:journalctl@ceph.mon.a.smithi103.stdout: No filesystems configured 2024-04-23T16:08:18.998 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-23T16:08:18.998 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T16:08:18.998 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T16:08:18.998 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T16:08:18.998 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-23T16:08:18.998 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 bash[39592]: b0d425db83deb36d7a43a9a3ce8e3f5794c855e1a441181050fccfb1faf71c8e 2024-04-23T16:08:18.998 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 systemd[1]: Started Ceph mon.a for 8e3594f2-018b-11ef-bc93-c7b262605968. 2024-04-23T16:08:18.998 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T16:08:18.998 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: monmap epoch 1 2024-04-23T16:08:18.999 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: fsid 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:08:18.999 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: last_changed 2024-04-23T16:08:10.387384+0000 2024-04-23T16:08:18.999 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: created 2024-04-23T16:08:10.387384+0000 2024-04-23T16:08:18.999 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: min_mon_release 19 (squid) 2024-04-23T16:08:18.999 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: election_strategy: 1 2024-04-23T16:08:18.999 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: 0: v1:172.21.15.103:6789/0 mon.a 2024-04-23T16:08:18.999 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: fsmap 2024-04-23T16:08:18.999 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: osdmap e1: 0 total, 0 up, 0 in 2024-04-23T16:08:18.999 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:18 smithi103 ceph-mon[39629]: mgrmap e1: no daemons active 2024-04-23T16:08:19.878 INFO:teuthology.orchestra.run.smithi103.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-23T16:08:19.901 INFO:teuthology.orchestra.run.smithi103.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-23T16:08:19.901 INFO:teuthology.orchestra.run.smithi103.stdout:Creating mgr... 2024-04-23T16:08:19.901 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-23T16:08:19.902 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-23T16:08:20.098 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mgr.y 2024-04-23T16:08:20.098 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mgr.y.service: Unit ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mgr.y.service not loaded. 2024-04-23T16:08:20.186 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8e3594f2-018b-11ef-bc93-c7b262605968.target.wants/ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-8e3594f2-018b-11ef-bc93-c7b262605968@.service. 2024-04-23T16:08:20.349 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:20 smithi103 systemd[1]: Starting Ceph mgr.y for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T16:08:20.671 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:20 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1196450260' entity='client.admin' 2024-04-23T16:08:20.913 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T16:08:20.914 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout disabled 2024-04-23T16:08:20.921 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T16:08:20.921 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-04-23T16:08:20.921 INFO:teuthology.orchestra.run.smithi103.stdout:firewalld.service is not enabled 2024-04-23T16:08:20.921 INFO:teuthology.orchestra.run.smithi103.stdout:Not possible to enable service . firewalld.service is not available 2024-04-23T16:08:20.929 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T16:08:20.930 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout disabled 2024-04-23T16:08:20.938 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T16:08:20.938 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-04-23T16:08:20.938 INFO:teuthology.orchestra.run.smithi103.stdout:firewalld.service is not enabled 2024-04-23T16:08:20.938 INFO:teuthology.orchestra.run.smithi103.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-23T16:08:20.939 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mgr to start... 2024-04-23T16:08:20.939 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mgr... 2024-04-23T16:08:21.177 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:20 smithi103 bash[39862]: cfcf28af4dc5dbdd42b28fcca0873843c95bc451b3622ab5a510c1e830491619 2024-04-23T16:08:21.178 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:20 smithi103 systemd[1]: Started Ceph mgr.y for 8e3594f2-018b-11ef-bc93-c7b262605968. 2024-04-23T16:08:21.178 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:20 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:20.960+0000 7f0eccf6a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T16:08:21.178 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:21 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:21.127+0000 7f0eccf6a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T16:08:21.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:21 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4100287020' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T16:08:21.850 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:21 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:21.469+0000 7f0eccf6a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T16:08:22.234 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-04-23T16:08:22.237 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-04-23T16:08:22.237 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsid": "8e3594f2-018b-11ef-bc93-c7b262605968", 2024-04-23T16:08:22.237 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T16:08:22.237 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T16:08:22.237 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T16:08:22.237 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T16:08:22.237 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:22.237 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T16:08:22.238 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T16:08:22.238 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 0 2024-04-23T16:08:22.238 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-04-23T16:08:22.238 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T16:08:22.238 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T16:08:22.238 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-04-23T16:08:22.238 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-23T16:08:22.238 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T16:08:22.238 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T16:08:22.238 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T16:08:22.238 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T16:08:22.238 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:22.238 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T16:08:22.238 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T16:08:22.239 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T16:08:22.239 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T16:08:22.239 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T16:08:22.239 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T16:08:22.239 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T16:08:22.239 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T16:08:22.239 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:22.240 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:22 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:22.000+0000 7f0eccf6a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T16:08:22.243 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T16:08:22.243 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T16:08:22.243 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T16:08:22.244 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T16:08:22.244 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T16:08:22.244 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T16:08:22.244 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T16:08:22.244 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T16:08:22.244 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T16:08:22.244 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:22.244 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T16:08:22.244 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T16:08:22.244 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T16:08:22.244 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T16:08:22.244 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:22.244 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T16:08:22.244 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-23T16:08:22.245 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T16:08:22.245 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T16:08:22.245 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T16:08:22.245 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T16:08:22.245 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T16:08:22.245 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-04-23T16:08:22.245 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T16:08:22.245 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:22.245 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T16:08:22.245 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T16:08:22.245 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T16:08:13.478486+0000", 2024-04-23T16:08:22.245 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T16:08:22.245 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:22.245 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T16:08:22.245 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-04-23T16:08:22.246 INFO:teuthology.orchestra.run.smithi103.stdout:mgr not available, waiting (1/15)... 2024-04-23T16:08:22.599 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:22 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:22.243+0000 7f0eccf6a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T16:08:23.099 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:22 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:22.674+0000 7f0eccf6a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T16:08:23.849 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:23 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:23.391+0000 7f0eccf6a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T16:08:23.850 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:23 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:23.477+0000 7f0eccf6a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T16:08:24.349 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:24 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:24.082+0000 7f0eccf6a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T16:08:24.349 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:24 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:24.167+0000 7f0eccf6a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T16:08:24.350 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:24 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:24.257+0000 7f0eccf6a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T16:08:24.849 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:24 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:24.492+0000 7f0eccf6a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T16:08:24.849 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:24 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:24.586+0000 7f0eccf6a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T16:08:24.850 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:24 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:24.679+0000 7f0eccf6a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T16:08:25.209 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3551709842' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T16:08:25.209 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:24 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:24.922+0000 7f0eccf6a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T16:08:25.209 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:25 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:25.020+0000 7f0eccf6a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T16:08:25.462 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:25 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:25.208+0000 7f0eccf6a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T16:08:25.522 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-04-23T16:08:25.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-04-23T16:08:25.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsid": "8e3594f2-018b-11ef-bc93-c7b262605968", 2024-04-23T16:08:25.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T16:08:25.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T16:08:25.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T16:08:25.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T16:08:25.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:25.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T16:08:25.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T16:08:25.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 0 2024-04-23T16:08:25.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-04-23T16:08:25.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T16:08:25.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T16:08:25.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-04-23T16:08:25.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-23T16:08:25.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T16:08:25.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T16:08:25.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T16:08:25.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T16:08:25.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:25.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T16:08:25.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T16:08:25.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T16:08:25.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T16:08:25.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T16:08:25.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T16:08:25.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T16:08:25.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T16:08:25.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:25.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T16:08:25.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T16:08:25.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T16:08:25.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T16:08:25.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T16:08:25.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T16:08:25.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T16:08:25.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T16:08:25.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T16:08:25.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:25.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T16:08:25.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T16:08:25.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T16:08:25.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T16:08:25.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:25.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T16:08:25.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-23T16:08:25.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T16:08:25.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T16:08:25.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T16:08:25.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T16:08:25.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T16:08:25.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-04-23T16:08:25.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T16:08:25.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:25.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T16:08:25.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T16:08:25.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T16:08:13.478486+0000", 2024-04-23T16:08:25.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T16:08:25.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:25.527 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T16:08:25.527 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-04-23T16:08:25.527 INFO:teuthology.orchestra.run.smithi103.stdout:mgr not available, waiting (2/15)... 2024-04-23T16:08:25.849 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:25 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:25.461+0000 7f0eccf6a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T16:08:26.349 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:26 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:26.043+0000 7f0eccf6a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T16:08:26.350 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:26 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:26.251+0000 7f0eccf6a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T16:08:26.849 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:26 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:26.445+0000 7f0eccf6a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T16:08:27.349 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:27.098+0000 7f0eccf6a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T16:08:27.349 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:27.295+0000 7f0eccf6a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T16:08:27.739 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:27.475+0000 7f0eccf6a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T16:08:27.739 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:27.592+0000 7f0eccf6a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T16:08:27.739 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-mon[39629]: Activating manager daemon y 2024-04-23T16:08:27.739 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-mon[39629]: mgrmap e2: y(active, starting, since 0.00328726s) 2024-04-23T16:08:27.739 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-mon[39629]: from='mgr.14100 v1:172.21.15.103:0/3420624011' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T16:08:27.740 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-mon[39629]: from='mgr.14100 v1:172.21.15.103:0/3420624011' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T16:08:27.740 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-mon[39629]: from='mgr.14100 v1:172.21.15.103:0/3420624011' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T16:08:27.740 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-mon[39629]: from='mgr.14100 v1:172.21.15.103:0/3420624011' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T16:08:27.740 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-mon[39629]: from='mgr.14100 v1:172.21.15.103:0/3420624011' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T16:08:27.740 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-mon[39629]: from='mgr.14100 v1:172.21.15.103:0/3420624011' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T16:08:27.740 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-mon[39629]: from='mgr.14100 v1:172.21.15.103:0/3420624011' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:08:27.740 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-mon[39629]: from='mgr.14100 v1:172.21.15.103:0/3420624011' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T16:08:27.740 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-mon[39629]: Manager daemon y is now available 2024-04-23T16:08:27.740 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-mon[39629]: from='mgr.14100 v1:172.21.15.103:0/3420624011' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T16:08:27.740 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-mon[39629]: from='mgr.14100 v1:172.21.15.103:0/3420624011' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T16:08:27.740 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-mon[39629]: from='mgr.14100 v1:172.21.15.103:0/3420624011' entity='mgr.y' 2024-04-23T16:08:27.740 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-mon[39629]: from='mgr.14100 v1:172.21.15.103:0/3420624011' entity='mgr.y' 2024-04-23T16:08:27.740 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:27 smithi103 ceph-mon[39629]: from='mgr.14100 v1:172.21.15.103:0/3420624011' entity='mgr.y' 2024-04-23T16:08:28.783 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-04-23T16:08:28.783 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-04-23T16:08:28.783 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsid": "8e3594f2-018b-11ef-bc93-c7b262605968", 2024-04-23T16:08:28.783 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T16:08:28.783 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T16:08:28.784 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T16:08:28.784 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T16:08:28.784 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:28.784 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T16:08:28.784 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T16:08:28.784 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 0 2024-04-23T16:08:28.784 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-04-23T16:08:28.784 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T16:08:28.784 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T16:08:28.784 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-04-23T16:08:28.784 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-04-23T16:08:28.784 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T16:08:28.784 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T16:08:28.784 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T16:08:28.784 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T16:08:28.785 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:28.785 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T16:08:28.785 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T16:08:28.785 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T16:08:28.785 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T16:08:28.785 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T16:08:28.785 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T16:08:28.785 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T16:08:28.785 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T16:08:28.787 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:28.788 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T16:08:28.788 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T16:08:28.788 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T16:08:28.788 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T16:08:28.788 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T16:08:28.788 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T16:08:28.788 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T16:08:28.788 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T16:08:28.788 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T16:08:28.788 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:28.788 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T16:08:28.788 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T16:08:28.788 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T16:08:28.788 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T16:08:28.789 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:28.789 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T16:08:28.789 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-23T16:08:28.789 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T16:08:28.789 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T16:08:28.789 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T16:08:28.789 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T16:08:28.789 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:28 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3356848766' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T16:08:28.789 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:28 smithi103 ceph-mon[39629]: mgrmap e3: y(active, since 1.00636s) 2024-04-23T16:08:28.791 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T16:08:28.791 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-04-23T16:08:28.791 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T16:08:28.791 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:28.791 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T16:08:28.791 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T16:08:28.791 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T16:08:13.478486+0000", 2024-04-23T16:08:28.791 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T16:08:28.791 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:28.791 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T16:08:28.791 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-04-23T16:08:28.792 INFO:teuthology.orchestra.run.smithi103.stdout:mgr not available, waiting (3/15)... 2024-04-23T16:08:31.013 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:30 smithi103 ceph-mon[39629]: mgrmap e4: y(active, since 2s) 2024-04-23T16:08:34.941 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:34 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/769455727' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T16:08:35.213 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-04-23T16:08:35.214 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-04-23T16:08:35.214 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsid": "8e3594f2-018b-11ef-bc93-c7b262605968", 2024-04-23T16:08:35.214 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T16:08:35.214 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T16:08:35.214 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T16:08:35.214 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T16:08:35.214 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:35.214 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T16:08:35.214 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T16:08:35.214 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 0 2024-04-23T16:08:35.214 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-04-23T16:08:35.214 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T16:08:35.214 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T16:08:35.215 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-04-23T16:08:35.215 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-04-23T16:08:35.215 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T16:08:35.215 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T16:08:35.215 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T16:08:35.215 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T16:08:35.215 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:35.215 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T16:08:35.215 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T16:08:35.215 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T16:08:35.215 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T16:08:35.215 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T16:08:35.215 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T16:08:35.215 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T16:08:35.215 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T16:08:35.219 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:35.219 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T16:08:35.219 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T16:08:35.219 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T16:08:35.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T16:08:35.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T16:08:35.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T16:08:35.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T16:08:35.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T16:08:35.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T16:08:35.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:35.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T16:08:35.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T16:08:35.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T16:08:35.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T16:08:35.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:35.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T16:08:35.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-23T16:08:35.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T16:08:35.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T16:08:35.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T16:08:35.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T16:08:35.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T16:08:35.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-04-23T16:08:35.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T16:08:35.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:35.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T16:08:35.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T16:08:35.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T16:08:13.478486+0000", 2024-04-23T16:08:35.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T16:08:35.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-04-23T16:08:35.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T16:08:35.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-04-23T16:08:35.222 INFO:teuthology.orchestra.run.smithi103.stdout:mgr is available 2024-04-23T16:08:36.540 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/539315206' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-23T16:08:36.812 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-04-23T16:08:36.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [global] 2024-04-23T16:08:36.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout fsid = 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:08:36.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-23T16:08:36.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-23T16:08:36.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-23T16:08:36.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-23T16:08:36.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-23T16:08:36.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-04-23T16:08:36.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-23T16:08:36.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-23T16:08:36.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-04-23T16:08:36.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [osd] 2024-04-23T16:08:36.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-23T16:08:36.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-23T16:08:36.814 INFO:teuthology.orchestra.run.smithi103.stdout:Enabling cephadm module... 2024-04-23T16:08:37.961 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:37 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/69909937' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-23T16:08:37.961 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:37 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: ignoring --setuser ceph since I am not root 2024-04-23T16:08:37.962 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:37 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: ignoring --setgroup ceph since I am not root 2024-04-23T16:08:37.962 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:37 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:37.784+0000 7f3950140200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T16:08:38.278 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:37 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:37.950+0000 7f3950140200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T16:08:38.536 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:38 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:38.277+0000 7f3950140200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T16:08:38.793 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:38 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/69909937' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-23T16:08:38.793 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:38 smithi103 ceph-mon[39629]: mgrmap e5: y(active, since 10s) 2024-04-23T16:08:39.057 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:38 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:38.792+0000 7f3950140200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T16:08:39.349 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:39 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:39.057+0000 7f3950140200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T16:08:39.740 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-04-23T16:08:39.740 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-23T16:08:39.741 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-23T16:08:39.741 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-23T16:08:39.741 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-23T16:08:39.741 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-04-23T16:08:39.741 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for the mgr to restart... 2024-04-23T16:08:39.741 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mgr epoch 5... 2024-04-23T16:08:39.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:39 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/891070947' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-23T16:08:39.850 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:39 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:39.505+0000 7f3950140200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T16:08:40.599 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:40 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:40.243+0000 7f3950140200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T16:08:40.600 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:40 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:40.335+0000 7f3950140200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T16:08:41.345 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:40 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:40.935+0000 7f3950140200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T16:08:41.345 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:41 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:41.020+0000 7f3950140200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T16:08:41.345 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:41 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:41.109+0000 7f3950140200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T16:08:41.601 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:41 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:41.344+0000 7f3950140200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T16:08:41.601 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:41 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:41.436+0000 7f3950140200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T16:08:41.601 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:41 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:41.530+0000 7f3950140200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T16:08:42.042 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:41 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:41.761+0000 7f3950140200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T16:08:42.042 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:41 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:41.856+0000 7f3950140200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T16:08:42.349 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:42 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:42.041+0000 7f3950140200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T16:08:42.350 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:42 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:42.289+0000 7f3950140200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T16:08:43.281 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:42 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:42.875+0000 7f3950140200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T16:08:43.281 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:43 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:43.084+0000 7f3950140200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T16:08:43.599 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:43 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:43.279+0000 7f3950140200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T16:08:44.320 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:43 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:43.939+0000 7f3950140200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T16:08:44.320 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:44 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:44.136+0000 7f3950140200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T16:08:44.599 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:44 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:44.319+0000 7f3950140200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T16:08:44.600 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:44 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:08:44.437+0000 7f3950140200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T16:08:44.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:44 smithi103 ceph-mon[39629]: Active manager daemon y restarted 2024-04-23T16:08:44.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:44 smithi103 ceph-mon[39629]: Activating manager daemon y 2024-04-23T16:08:44.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:44 smithi103 ceph-mon[39629]: osdmap e2: 0 total, 0 up, 0 in 2024-04-23T16:08:44.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:44 smithi103 ceph-mon[39629]: mgrmap e6: y(active, starting, since 0.00335743s) 2024-04-23T16:08:44.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:44 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:08:44.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:44 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T16:08:44.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:44 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T16:08:44.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:44 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T16:08:44.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:44 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T16:08:44.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:44 smithi103 ceph-mon[39629]: Manager daemon y is now available 2024-04-23T16:08:44.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:44 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' 2024-04-23T16:08:44.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:44 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' 2024-04-23T16:08:44.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:44 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:08:44.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:44 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:08:45.793 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:45 smithi103 ceph-mon[39629]: Found migration_current of "None". Setting to last migration. 2024-04-23T16:08:45.793 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:45 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T16:08:45.793 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:45 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T16:08:45.793 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:45 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' 2024-04-23T16:08:45.793 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:45 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' 2024-04-23T16:08:45.793 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:45 smithi103 ceph-mon[39629]: mgrmap e7: y(active, since 1.00536s) 2024-04-23T16:08:46.164 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-04-23T16:08:46.164 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-23T16:08:46.164 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-23T16:08:46.164 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-04-23T16:08:46.164 INFO:teuthology.orchestra.run.smithi103.stdout:mgr epoch 5 is available 2024-04-23T16:08:46.164 INFO:teuthology.orchestra.run.smithi103.stdout:Setting orchestrator backend to cephadm... 2024-04-23T16:08:46.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:46 smithi103 ceph-mon[39629]: from='client.14125 v1:172.21.15.103:0/3105167596' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-23T16:08:46.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:46 smithi103 ceph-mon[39629]: from='client.14125 v1:172.21.15.103:0/3105167596' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-23T16:08:46.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:46 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:08:47.793 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:47 smithi103 ceph-mon[39629]: [23/Apr/2024:16:08:46] ENGINE Bus STARTING 2024-04-23T16:08:47.794 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:47 smithi103 ceph-mon[39629]: [23/Apr/2024:16:08:46] ENGINE Serving on https://172.21.15.103:7150 2024-04-23T16:08:47.794 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:47 smithi103 ceph-mon[39629]: [23/Apr/2024:16:08:46] ENGINE Serving on http://172.21.15.103:8765 2024-04-23T16:08:47.794 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:47 smithi103 ceph-mon[39629]: [23/Apr/2024:16:08:46] ENGINE Bus STARTED 2024-04-23T16:08:47.794 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:47 smithi103 ceph-mon[39629]: mgrmap e8: y(active, since 2s) 2024-04-23T16:08:47.794 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:47 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' 2024-04-23T16:08:47.794 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:47 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:08:48.673 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:48 smithi103 ceph-mon[39629]: from='client.14133 v1:172.21.15.103:0/2662941723' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:08:48.951 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-23T16:08:48.951 INFO:teuthology.orchestra.run.smithi103.stdout:Generating ssh key... 2024-04-23T16:08:49.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:49 smithi103 ceph-mon[39629]: from='client.14136 v1:172.21.15.103:0/342789805' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:08:49.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:49 smithi103 ceph-mon[39629]: mgrmap e9: y(active, since 4s) 2024-04-23T16:08:51.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:50 smithi103 ceph-mon[39629]: from='client.14138 v1:172.21.15.103:0/2130372239' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:08:51.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:50 smithi103 ceph-mon[39629]: Generating ssh key... 2024-04-23T16:08:51.600 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:51 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: Generating public/private rsa key pair. 2024-04-23T16:08:51.600 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:51 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: Your identification has been saved in /tmp/tmpsheljg_q/key. 2024-04-23T16:08:51.600 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:51 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: Your public key has been saved in /tmp/tmpsheljg_q/key.pub. 2024-04-23T16:08:51.601 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:51 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: The key fingerprint is: 2024-04-23T16:08:51.601 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:51 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: SHA256:TaWMfxjxK6S/M5iuLLMJ8M5+mp6aaisco9f6N4oMSwE ceph-8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:08:51.601 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:51 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: The key's randomart image is: 2024-04-23T16:08:51.601 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:51 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: +---[RSA 3072]----+ 2024-04-23T16:08:51.601 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:51 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: | . . | 2024-04-23T16:08:51.601 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:51 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: | o = | 2024-04-23T16:08:51.602 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:51 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: |E . * . | 2024-04-23T16:08:51.602 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:51 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: |. * o . | 2024-04-23T16:08:51.602 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:51 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: |.. S = o | 2024-04-23T16:08:51.602 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:51 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: | =. . o | 2024-04-23T16:08:51.602 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:51 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: |+.=. o. | 2024-04-23T16:08:51.602 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:51 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: |+X.==oo o o. | 2024-04-23T16:08:51.603 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:51 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: |O*&==*o+. .o | 2024-04-23T16:08:51.603 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:08:51 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: +----[SHA256]-----+ 2024-04-23T16:08:52.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:52 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' 2024-04-23T16:08:52.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:52 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' 2024-04-23T16:08:53.259 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBZUJpx0PoOPYjok05paDfuYo1SoTjt1ayFP4zDLeeWzcqAOQeTOolU5AiuEY123BcePtO00UFRzBfXhwTS2vm/Vtrn5Fg4N8c183fBt5ZzVmIqu1HflqR3uU5A+aQL431lEmPeNNTXy9jXXAxWH4ve2/Er7aynug4x7gNIgsWHlLWRszdmezogoFsz7X/gq0hLNoUWRYPsNhPt0hEq4D10CtelbPqP8FOGU1ldzjtzTLXeW4JIwKvWKdnjF0nohTgNGYGegV0fj9hkyj5NUgTGfxs+cIucI9n0KWX6TdpXv+yXckcPg/OGlq25Oq+0d4kDH57lxp0HF+6NAds4PWdWtzqaPqCSCNhipzkxHpQXDsDevELyDsmsNQjKmugp7OHpVtO3ihCPM2VWlMVq0MbvNL2AYIdfql3K4HjtAGSMmIsDrYz3bav9LV2L+xyIm+gAgZX3hlupMn3gv/KoG3K6HneDm7osTp9rVIDL8+ugWl56Kr7oyziMgYwP7gobIM= ceph-8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:08:53.260 INFO:teuthology.orchestra.run.smithi103.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-23T16:08:53.260 INFO:teuthology.orchestra.run.smithi103.stdout:Adding key to root@localhost authorized_keys... 2024-04-23T16:08:53.260 INFO:teuthology.orchestra.run.smithi103.stdout:Adding host smithi103... 2024-04-23T16:08:53.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:53 smithi103 ceph-mon[39629]: from='client.14140 v1:172.21.15.103:0/2317174387' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:08:54.912 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:54 smithi103 ceph-mon[39629]: from='client.14142 v1:172.21.15.103:0/2532818863' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi103", "addr": "172.21.15.103", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:08:54.912 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:54 smithi103 ceph-mon[39629]: Deploying cephadm binary to smithi103 2024-04-23T16:08:58.359 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Added host 'smithi103' with addr '172.21.15.103' 2024-04-23T16:08:58.359 INFO:teuthology.orchestra.run.smithi103.stdout:Deploying unmanaged mon service... 2024-04-23T16:08:59.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:58 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' 2024-04-23T16:08:59.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:58 smithi103 ceph-mon[39629]: Added host smithi103 2024-04-23T16:08:59.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:08:58 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:09:00.499 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-23T16:09:00.499 INFO:teuthology.orchestra.run.smithi103.stdout:Deploying unmanaged mgr service... 2024-04-23T16:09:00.502 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:00 smithi103 ceph-mon[39629]: from='client.14144 v1:172.21.15.103:0/1102777506' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:09:00.502 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:00 smithi103 ceph-mon[39629]: Saving service mon spec with placement count:5 2024-04-23T16:09:00.502 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:00 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' 2024-04-23T16:09:02.069 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:01 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' 2024-04-23T16:09:02.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:01 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' 2024-04-23T16:09:02.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:01 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' 2024-04-23T16:09:02.551 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-23T16:09:03.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:02 smithi103 ceph-mon[39629]: from='client.14146 v1:172.21.15.103:0/4219970968' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:09:03.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:02 smithi103 ceph-mon[39629]: Saving service mgr spec with placement count:2 2024-04-23T16:09:04.453 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:04 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4284734273' entity='client.admin' 2024-04-23T16:09:04.454 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:04 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' 2024-04-23T16:09:04.454 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:04 smithi103 ceph-mon[39629]: from='mgr.14121 v1:172.21.15.103:0/2895099113' entity='mgr.y' 2024-04-23T16:09:05.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:05 smithi103 ceph-mon[39629]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:09:05.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:05 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1373361288' entity='client.admin' 2024-04-23T16:09:06.616 INFO:teuthology.orchestra.run.smithi103.stdout:Enabling the dashboard module... 2024-04-23T16:09:07.796 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:07 smithi103 ceph-mon[39629]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:09:07.796 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:07 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2859924471' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-23T16:09:07.796 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:07 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: ignoring --setuser ceph since I am not root 2024-04-23T16:09:07.797 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:07 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: ignoring --setgroup ceph since I am not root 2024-04-23T16:09:07.797 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:07 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:07.639+0000 7fa2f8af7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T16:09:08.099 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:07 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:07.807+0000 7fa2f8af7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T16:09:08.397 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:08 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:08.141+0000 7fa2f8af7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T16:09:08.661 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:08 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2859924471' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-23T16:09:08.661 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:08 smithi103 ceph-mon[39629]: mgrmap e10: y(active, since 23s) 2024-04-23T16:09:08.916 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:08 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:08.659+0000 7fa2f8af7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T16:09:09.334 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:08 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:08.914+0000 7fa2f8af7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T16:09:09.571 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-04-23T16:09:09.572 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 10, 2024-04-23T16:09:09.572 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-23T16:09:09.572 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-23T16:09:09.572 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-23T16:09:09.572 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-04-23T16:09:09.572 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for the mgr to restart... 2024-04-23T16:09:09.572 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mgr epoch 10... 2024-04-23T16:09:09.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:09 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2629312703' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-23T16:09:09.600 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:09 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:09.362+0000 7fa2f8af7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T16:09:10.599 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:10 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:10.105+0000 7fa2f8af7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T16:09:10.599 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:10 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:10.195+0000 7fa2f8af7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T16:09:11.099 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:10 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:10.796+0000 7fa2f8af7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T16:09:11.100 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:10 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:10.881+0000 7fa2f8af7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T16:09:11.100 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:10 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:10.970+0000 7fa2f8af7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T16:09:11.599 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:11 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:11.199+0000 7fa2f8af7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T16:09:11.600 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:11 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:11.291+0000 7fa2f8af7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T16:09:11.600 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:11 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:11.386+0000 7fa2f8af7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T16:09:11.904 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:11 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:11.619+0000 7fa2f8af7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T16:09:11.904 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:11 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:11.715+0000 7fa2f8af7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T16:09:12.154 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:11 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:11.902+0000 7fa2f8af7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T16:09:12.599 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:12 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:12.152+0000 7fa2f8af7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T16:09:13.099 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:12 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:12.737+0000 7fa2f8af7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T16:09:13.100 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:12 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:12.949+0000 7fa2f8af7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T16:09:13.600 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:13 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:13.146+0000 7fa2f8af7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T16:09:14.099 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:13 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:13.809+0000 7fa2f8af7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T16:09:14.100 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:14 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:14.010+0000 7fa2f8af7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T16:09:14.599 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:14 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:14.193+0000 7fa2f8af7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T16:09:14.600 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 16:09:14 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T16:09:14.309+0000 7fa2f8af7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T16:09:14.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:14 smithi103 ceph-mon[39629]: Active manager daemon y restarted 2024-04-23T16:09:14.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:14 smithi103 ceph-mon[39629]: Activating manager daemon y 2024-04-23T16:09:14.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:14 smithi103 ceph-mon[39629]: osdmap e3: 0 total, 0 up, 0 in 2024-04-23T16:09:14.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:14 smithi103 ceph-mon[39629]: mgrmap e11: y(active, starting, since 0.00552499s) 2024-04-23T16:09:14.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:09:14.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T16:09:14.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T16:09:14.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T16:09:14.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T16:09:14.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:14 smithi103 ceph-mon[39629]: Manager daemon y is now available 2024-04-23T16:09:14.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:09:15.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T16:09:15.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T16:09:15.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:15 smithi103 ceph-mon[39629]: mgrmap e12: y(active, since 1.00588s) 2024-04-23T16:09:16.099 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-04-23T16:09:16.099 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 12, 2024-04-23T16:09:16.099 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-23T16:09:16.099 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-04-23T16:09:16.099 INFO:teuthology.orchestra.run.smithi103.stdout:mgr epoch 10 is available 2024-04-23T16:09:16.099 INFO:teuthology.orchestra.run.smithi103.stdout:Generating a dashboard self-signed certificate... 2024-04-23T16:09:16.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:16 smithi103 ceph-mon[39629]: [23/Apr/2024:16:09:14] ENGINE Bus STARTING 2024-04-23T16:09:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:16 smithi103 ceph-mon[39629]: [23/Apr/2024:16:09:14] ENGINE Serving on http://172.21.15.103:8765 2024-04-23T16:09:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:16 smithi103 ceph-mon[39629]: [23/Apr/2024:16:09:14] ENGINE Serving on https://172.21.15.103:7150 2024-04-23T16:09:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:16 smithi103 ceph-mon[39629]: [23/Apr/2024:16:09:14] ENGINE Bus STARTED 2024-04-23T16:09:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:16 smithi103 ceph-mon[39629]: from='client.14158 v1:172.21.15.103:0/810226520' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-23T16:09:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:16 smithi103 ceph-mon[39629]: from='client.14158 v1:172.21.15.103:0/810226520' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-23T16:09:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:17.824 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-23T16:09:17.824 INFO:teuthology.orchestra.run.smithi103.stdout:Creating initial admin user... 2024-04-23T16:09:17.826 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:17.826 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:17 smithi103 ceph-mon[39629]: mgrmap e13: y(active, since 2s) 2024-04-23T16:09:17.826 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:17 smithi103 ceph-mon[39629]: from='client.14166 v1:172.21.15.103:0/4018332592' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:09:17.827 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:17.827 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:19.627 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$vdVIpu6PWXCScHj25IEz4u13r1pQ4qkkfAM6t8FkCjaykOGOIYpKi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713888558, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-23T16:09:19.627 INFO:teuthology.orchestra.run.smithi103.stdout:Fetching dashboard port number... 2024-04-23T16:09:19.900 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:19 smithi103 ceph-mon[39629]: from='client.14168 v1:172.21.15.103:0/2857144596' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:09:19.900 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:19 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:21.026 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 8443 2024-04-23T16:09:21.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:20 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3075780038' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-23T16:09:21.037 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T16:09:21.037 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout disabled 2024-04-23T16:09:21.044 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T16:09:21.044 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-04-23T16:09:21.044 INFO:teuthology.orchestra.run.smithi103.stdout:firewalld.service is not enabled 2024-04-23T16:09:21.044 INFO:teuthology.orchestra.run.smithi103.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-23T16:09:21.046 INFO:teuthology.orchestra.run.smithi103.stdout:Ceph Dashboard is now available at: 2024-04-23T16:09:21.047 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:09:21.047 INFO:teuthology.orchestra.run.smithi103.stdout: URL: https://smithi103.front.sepia.ceph.com:8443/ 2024-04-23T16:09:21.047 INFO:teuthology.orchestra.run.smithi103.stdout: User: admin 2024-04-23T16:09:21.047 INFO:teuthology.orchestra.run.smithi103.stdout: Password: 7mpznz7paw 2024-04-23T16:09:21.047 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:09:21.047 INFO:teuthology.orchestra.run.smithi103.stdout:Saving cluster configuration to /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config directory 2024-04-23T16:09:21.047 INFO:teuthology.orchestra.run.smithi103.stdout:Enabling autotune for osd_memory_target 2024-04-23T16:09:22.096 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:21 smithi103 ceph-mon[39629]: mgrmap e14: y(active, since 6s) 2024-04-23T16:09:23.907 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-23T16:09:23.907 INFO:teuthology.orchestra.run.smithi103.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-23T16:09:23.907 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:09:23.907 INFO:teuthology.orchestra.run.smithi103.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-23T16:09:23.907 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:09:23.907 INFO:teuthology.orchestra.run.smithi103.stdout:Or, if you are only running a single cluster on this host: 2024-04-23T16:09:23.907 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:09:23.907 INFO:teuthology.orchestra.run.smithi103.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-23T16:09:23.907 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:09:23.907 INFO:teuthology.orchestra.run.smithi103.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-23T16:09:23.908 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:09:23.908 INFO:teuthology.orchestra.run.smithi103.stdout: ceph telemetry on 2024-04-23T16:09:23.908 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:09:23.908 INFO:teuthology.orchestra.run.smithi103.stdout:For more information see: 2024-04-23T16:09:23.908 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:09:23.908 INFO:teuthology.orchestra.run.smithi103.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-23T16:09:23.908 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:09:23.908 INFO:teuthology.orchestra.run.smithi103.stdout:Bootstrap complete. 2024-04-23T16:09:23.954 INFO:tasks.cephadm:Fetching config... 2024-04-23T16:09:23.955 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-23T16:09:23.955 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-23T16:09:23.974 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-23T16:09:23.974 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-23T16:09:23.974 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-23T16:09:24.035 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-23T16:09:24.035 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-23T16:09:24.036 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-23T16:09:24.101 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-23T16:09:24.101 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-23T16:09:24.101 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-23T16:09:24.159 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-23T16:09:24.159 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBZUJpx0PoOPYjok05paDfuYo1SoTjt1ayFP4zDLeeWzcqAOQeTOolU5AiuEY123BcePtO00UFRzBfXhwTS2vm/Vtrn5Fg4N8c183fBt5ZzVmIqu1HflqR3uU5A+aQL431lEmPeNNTXy9jXXAxWH4ve2/Er7aynug4x7gNIgsWHlLWRszdmezogoFsz7X/gq0hLNoUWRYPsNhPt0hEq4D10CtelbPqP8FOGU1ldzjtzTLXeW4JIwKvWKdnjF0nohTgNGYGegV0fj9hkyj5NUgTGfxs+cIucI9n0KWX6TdpXv+yXckcPg/OGlq25Oq+0d4kDH57lxp0HF+6NAds4PWdWtzqaPqCSCNhipzkxHpQXDsDevELyDsmsNQjKmugp7OHpVtO3ihCPM2VWlMVq0MbvNL2AYIdfql3K4HjtAGSMmIsDrYz3bav9LV2L+xyIm+gAgZX3hlupMn3gv/KoG3K6HneDm7osTp9rVIDL8+ugWl56Kr7oyziMgYwP7gobIM= ceph-8e3594f2-018b-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T16:09:24.236 INFO:teuthology.orchestra.run.smithi103.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBZUJpx0PoOPYjok05paDfuYo1SoTjt1ayFP4zDLeeWzcqAOQeTOolU5AiuEY123BcePtO00UFRzBfXhwTS2vm/Vtrn5Fg4N8c183fBt5ZzVmIqu1HflqR3uU5A+aQL431lEmPeNNTXy9jXXAxWH4ve2/Er7aynug4x7gNIgsWHlLWRszdmezogoFsz7X/gq0hLNoUWRYPsNhPt0hEq4D10CtelbPqP8FOGU1ldzjtzTLXeW4JIwKvWKdnjF0nohTgNGYGegV0fj9hkyj5NUgTGfxs+cIucI9n0KWX6TdpXv+yXckcPg/OGlq25Oq+0d4kDH57lxp0HF+6NAds4PWdWtzqaPqCSCNhipzkxHpQXDsDevELyDsmsNQjKmugp7OHpVtO3ihCPM2VWlMVq0MbvNL2AYIdfql3K4HjtAGSMmIsDrYz3bav9LV2L+xyIm+gAgZX3hlupMn3gv/KoG3K6HneDm7osTp9rVIDL8+ugWl56Kr7oyziMgYwP7gobIM= ceph-8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:09:24.247 DEBUG:teuthology.orchestra.run.smithi148:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBZUJpx0PoOPYjok05paDfuYo1SoTjt1ayFP4zDLeeWzcqAOQeTOolU5AiuEY123BcePtO00UFRzBfXhwTS2vm/Vtrn5Fg4N8c183fBt5ZzVmIqu1HflqR3uU5A+aQL431lEmPeNNTXy9jXXAxWH4ve2/Er7aynug4x7gNIgsWHlLWRszdmezogoFsz7X/gq0hLNoUWRYPsNhPt0hEq4D10CtelbPqP8FOGU1ldzjtzTLXeW4JIwKvWKdnjF0nohTgNGYGegV0fj9hkyj5NUgTGfxs+cIucI9n0KWX6TdpXv+yXckcPg/OGlq25Oq+0d4kDH57lxp0HF+6NAds4PWdWtzqaPqCSCNhipzkxHpQXDsDevELyDsmsNQjKmugp7OHpVtO3ihCPM2VWlMVq0MbvNL2AYIdfql3K4HjtAGSMmIsDrYz3bav9LV2L+xyIm+gAgZX3hlupMn3gv/KoG3K6HneDm7osTp9rVIDL8+ugWl56Kr7oyziMgYwP7gobIM= ceph-8e3594f2-018b-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T16:09:24.291 INFO:teuthology.orchestra.run.smithi148.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBZUJpx0PoOPYjok05paDfuYo1SoTjt1ayFP4zDLeeWzcqAOQeTOolU5AiuEY123BcePtO00UFRzBfXhwTS2vm/Vtrn5Fg4N8c183fBt5ZzVmIqu1HflqR3uU5A+aQL431lEmPeNNTXy9jXXAxWH4ve2/Er7aynug4x7gNIgsWHlLWRszdmezogoFsz7X/gq0hLNoUWRYPsNhPt0hEq4D10CtelbPqP8FOGU1ldzjtzTLXeW4JIwKvWKdnjF0nohTgNGYGegV0fj9hkyj5NUgTGfxs+cIucI9n0KWX6TdpXv+yXckcPg/OGlq25Oq+0d4kDH57lxp0HF+6NAds4PWdWtzqaPqCSCNhipzkxHpQXDsDevELyDsmsNQjKmugp7OHpVtO3ihCPM2VWlMVq0MbvNL2AYIdfql3K4HjtAGSMmIsDrYz3bav9LV2L+xyIm+gAgZX3hlupMn3gv/KoG3K6HneDm7osTp9rVIDL8+ugWl56Kr7oyziMgYwP7gobIM= ceph-8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:09:24.303 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBZUJpx0PoOPYjok05paDfuYo1SoTjt1ayFP4zDLeeWzcqAOQeTOolU5AiuEY123BcePtO00UFRzBfXhwTS2vm/Vtrn5Fg4N8c183fBt5ZzVmIqu1HflqR3uU5A+aQL431lEmPeNNTXy9jXXAxWH4ve2/Er7aynug4x7gNIgsWHlLWRszdmezogoFsz7X/gq0hLNoUWRYPsNhPt0hEq4D10CtelbPqP8FOGU1ldzjtzTLXeW4JIwKvWKdnjF0nohTgNGYGegV0fj9hkyj5NUgTGfxs+cIucI9n0KWX6TdpXv+yXckcPg/OGlq25Oq+0d4kDH57lxp0HF+6NAds4PWdWtzqaPqCSCNhipzkxHpQXDsDevELyDsmsNQjKmugp7OHpVtO3ihCPM2VWlMVq0MbvNL2AYIdfql3K4HjtAGSMmIsDrYz3bav9LV2L+xyIm+gAgZX3hlupMn3gv/KoG3K6HneDm7osTp9rVIDL8+ugWl56Kr7oyziMgYwP7gobIM= ceph-8e3594f2-018b-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T16:09:24.348 INFO:teuthology.orchestra.run.smithi188.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBZUJpx0PoOPYjok05paDfuYo1SoTjt1ayFP4zDLeeWzcqAOQeTOolU5AiuEY123BcePtO00UFRzBfXhwTS2vm/Vtrn5Fg4N8c183fBt5ZzVmIqu1HflqR3uU5A+aQL431lEmPeNNTXy9jXXAxWH4ve2/Er7aynug4x7gNIgsWHlLWRszdmezogoFsz7X/gq0hLNoUWRYPsNhPt0hEq4D10CtelbPqP8FOGU1ldzjtzTLXeW4JIwKvWKdnjF0nohTgNGYGegV0fj9hkyj5NUgTGfxs+cIucI9n0KWX6TdpXv+yXckcPg/OGlq25Oq+0d4kDH57lxp0HF+6NAds4PWdWtzqaPqCSCNhipzkxHpQXDsDevELyDsmsNQjKmugp7OHpVtO3ihCPM2VWlMVq0MbvNL2AYIdfql3K4HjtAGSMmIsDrYz3bav9LV2L+xyIm+gAgZX3hlupMn3gv/KoG3K6HneDm7osTp9rVIDL8+ugWl56Kr7oyziMgYwP7gobIM= ceph-8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:09:24.360 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-23T16:09:24.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3753009030' entity='client.admin' 2024-04-23T16:09:24.630 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:09:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:09:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:09:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:28.606 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-23T16:09:28.607 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-23T16:09:28.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:28 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1616098398' entity='client.admin' 2024-04-23T16:09:28.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:28 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:09:28.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:28 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:29.202 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:09:31.949 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi148 2024-04-23T16:09:31.949 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-04-23T16:09:31.949 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/etc/ceph/ceph.conf 2024-04-23T16:09:31.972 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-04-23T16:09:31.972 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T16:09:32.033 INFO:tasks.cephadm:Adding host smithi148 to orchestrator... 2024-04-23T16:09:32.033 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph orch host add smithi148 2024-04-23T16:09:32.294 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:09:32.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:32 smithi103 ceph-mon[39629]: from='client.14178 v1:172.21.15.103:0/1605990513' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:09:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:09:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:09:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:09:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:32 smithi103 ceph-mon[39629]: Updating smithi103:/etc/ceph/ceph.conf 2024-04-23T16:09:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:32 smithi103 ceph-mon[39629]: Updating smithi103:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T16:09:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:33.514 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:33 smithi103 ceph-mon[39629]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-04-23T16:09:33.514 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:33 smithi103 ceph-mon[39629]: Updating smithi103:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T16:09:35.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:35 smithi103 ceph-mon[39629]: from='client.14180 v1:172.21.15.103:0/1903854776' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi148", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:09:35.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:35 smithi103 ceph-mon[39629]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:09:36.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:36 smithi103 ceph-mon[39629]: Deploying cephadm binary to smithi148 2024-04-23T16:09:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:37 smithi103 ceph-mon[39629]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:09:38.022 INFO:teuthology.orchestra.run.smithi103.stdout:Added host 'smithi148' with addr '172.21.15.148' 2024-04-23T16:09:38.698 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-23T16:09:38.960 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:09:39.294 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:39 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:39.294 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:39 smithi103 ceph-mon[39629]: Added host smithi148 2024-04-23T16:09:39.294 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:39 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:09:39.294 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:39 smithi103 ceph-mon[39629]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:09:39.294 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:39 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:39.294 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:39 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:40.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:40 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:40.899 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:09:40.899 INFO:teuthology.orchestra.run.smithi103.stdout:[{"addr": "172.21.15.103", "hostname": "smithi103", "labels": [], "status": ""}, {"addr": "172.21.15.148", "hostname": "smithi148", "labels": [], "status": ""}] 2024-04-23T16:09:41.543 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:41 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:09:41.573 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi188 2024-04-23T16:09:41.573 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T16:09:41.573 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.conf 2024-04-23T16:09:41.595 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T16:09:41.595 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T16:09:41.658 INFO:tasks.cephadm:Adding host smithi188 to orchestrator... 2024-04-23T16:09:41.658 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph orch host add smithi188 2024-04-23T16:09:41.919 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:09:42.744 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:42 smithi103 ceph-mon[39629]: from='client.14182 v1:172.21.15.103:0/409889851' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:09:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:43 smithi103 ceph-mon[39629]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:09:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:43 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:43 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:43 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:43 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:43 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi148", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:09:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:43 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:09:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:43 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:09:44.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:44 smithi103 ceph-mon[39629]: Updating smithi148:/etc/ceph/ceph.conf 2024-04-23T16:09:44.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:44 smithi103 ceph-mon[39629]: Updating smithi148:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T16:09:44.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:44 smithi103 ceph-mon[39629]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-04-23T16:09:44.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:44 smithi103 ceph-mon[39629]: Updating smithi148:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T16:09:44.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:44 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:44.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:44 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:44.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:44 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:44.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:44 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:09:44.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:44 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:09:44.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:44 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:09:44.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:44 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:45.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:45 smithi103 ceph-mon[39629]: from='client.14184 v1:172.21.15.103:0/4232709072' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:09:45.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:45 smithi103 ceph-mon[39629]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:09:46.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:46 smithi103 ceph-mon[39629]: Deploying cephadm binary to smithi188 2024-04-23T16:09:47.785 INFO:teuthology.orchestra.run.smithi103.stdout:Added host 'smithi188' with addr '172.21.15.188' 2024-04-23T16:09:47.811 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:47 smithi103 ceph-mon[39629]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:09:48.531 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-23T16:09:48.794 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:09:49.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:48 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:49.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:48 smithi103 ceph-mon[39629]: Added host smithi188 2024-04-23T16:09:49.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:48 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:09:49.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:48 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:49.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:48 smithi103 ceph-mon[39629]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:09:49.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:48 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:50.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:50 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:50.676 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:09:50.677 INFO:teuthology.orchestra.run.smithi103.stdout:[{"addr": "172.21.15.103", "hostname": "smithi103", "labels": [], "status": ""}, {"addr": "172.21.15.148", "hostname": "smithi148", "labels": [], "status": ""}, {"addr": "172.21.15.188", "hostname": "smithi188", "labels": [], "status": ""}] 2024-04-23T16:09:51.279 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-23T16:09:51.280 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-23T16:09:51.538 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:09:51.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:51 smithi103 ceph-mon[39629]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:09:52.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:52 smithi103 ceph-mon[39629]: from='client.14186 v1:172.21.15.103:0/97525693' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:09:53.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:53 smithi103 ceph-mon[39629]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:09:53.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:53.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:53.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:53.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:53.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:09:53.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:09:53.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:09:53.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:53 smithi103 ceph-mon[39629]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-23T16:09:53.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:53.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:53.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:54.381 INFO:teuthology.orchestra.run.smithi103.stderr:adjusted tunables profile to default 2024-04-23T16:09:54.730 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:54 smithi103 ceph-mon[39629]: Updating smithi188:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T16:09:54.730 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:54 smithi103 ceph-mon[39629]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-23T16:09:54.731 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:54 smithi103 ceph-mon[39629]: Updating smithi188:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T16:09:54.731 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:54 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/912320622' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-23T16:09:55.061 INFO:tasks.cephadm:Adding mon.a on smithi103 2024-04-23T16:09:55.061 INFO:tasks.cephadm:Adding mon.b on smithi148 2024-04-23T16:09:55.061 INFO:tasks.cephadm:Adding mon.c on smithi188 2024-04-23T16:09:55.061 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi103:[v1:172.21.15.103:6789]=a;smithi148:[v1:172.21.15.148:6789]=b;smithi188:[v1:172.21.15.188:6789]=c' 2024-04-23T16:09:55.357 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T16:09:55.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:55 smithi103 ceph-mon[39629]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:09:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:55 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/912320622' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-23T16:09:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:55 smithi103 ceph-mon[39629]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T16:09:57.187 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mon update... 2024-04-23T16:09:57.817 DEBUG:teuthology.orchestra.run.smithi148:mon.b> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mon.b.service 2024-04-23T16:09:57.819 DEBUG:teuthology.orchestra.run.smithi188:mon.c> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mon.c.service 2024-04-23T16:09:57.822 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T16:09:57.822 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T16:09:57.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:57 smithi103 ceph-mon[39629]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:09:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:09:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:09:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:09:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:09:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T16:09:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:09:57.851 INFO:journalctl@ceph.mon.c.smithi188.stdout:-- Logs begin at Tue 2024-04-23 15:57:06 UTC. -- 2024-04-23T16:09:57.855 INFO:journalctl@ceph.mon.b.smithi148.stdout:-- Logs begin at Tue 2024-04-23 15:57:31 UTC. -- 2024-04-23T16:09:58.543 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T16:09:58.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:58 smithi103 ceph-mon[39629]: from='client.14190 v1:172.21.15.188:0/3108739857' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi103:[v1:172.21.15.103:6789]=a;smithi148:[v1:172.21.15.148:6789]=b;smithi188:[v1:172.21.15.188:6789]=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:09:58.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:58 smithi103 ceph-mon[39629]: Saving service mon spec with placement smithi103:[v1:172.21.15.103:6789]=a;smithi148:[v1:172.21.15.148:6789]=b;smithi188:[v1:172.21.15.188:6789]=c;count:3 2024-04-23T16:09:58.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:58 smithi103 ceph-mon[39629]: Deploying daemon mon.c on smithi188 2024-04-23T16:09:59.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:09:59 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:10:01.107 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:00 smithi188 systemd[1]: Starting Ceph mon.c for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T16:10:01.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:01 smithi103 ceph-mon[39629]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:01.965 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:10:01.966 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","modified":"2024-04-23T16:08:10.387384Z","created":"2024-04-23T16:08:10.387384Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-23T16:10:01.969 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-04-23T16:10:02.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T16:10:02.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: ceph version 19.0.0-3211-g5d349943 (5d349943c59c9485df060d6adb0594f3940ec0eb) squid (dev), process ceph-mon, pid 7 2024-04-23T16:10:02.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: pidfile_write: ignore empty --pid-file 2024-04-23T16:10:02.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: load: jerasure load: lrc 2024-04-23T16:10:02.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: RocksDB version: 7.9.2 2024-04-23T16:10:02.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Git sha 0 2024-04-23T16:10:02.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Compile date 2024-04-22 18:35:14 2024-04-23T16:10:02.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: DB SUMMARY 2024-04-23T16:10:02.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: DB Session ID: J895XKM4VZHM61JOENSM 2024-04-23T16:10:02.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: CURRENT file: CURRENT 2024-04-23T16:10:02.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T16:10:02.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-23T16:10:02.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-23T16:10:02.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 476 ; 2024-04-23T16:10:02.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.error_if_exists: 0 2024-04-23T16:10:02.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.create_if_missing: 0 2024-04-23T16:10:02.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.paranoid_checks: 1 2024-04-23T16:10:02.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-23T16:10:02.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T16:10:02.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-23T16:10:02.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.env: 0x556cb2e53cc0 2024-04-23T16:10:02.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.fs: PosixFileSystem 2024-04-23T16:10:02.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.info_log: 0x556cb46e3c60 2024-04-23T16:10:02.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T16:10:02.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.statistics: (nil) 2024-04-23T16:10:02.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.use_fsync: 0 2024-04-23T16:10:02.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_log_file_size: 0 2024-04-23T16:10:02.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T16:10:02.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T16:10:02.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T16:10:02.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T16:10:02.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.allow_fallocate: 1 2024-04-23T16:10:02.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T16:10:02.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T16:10:02.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.use_direct_reads: 0 2024-04-23T16:10:02.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T16:10:02.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T16:10:02.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.db_log_dir: 2024-04-23T16:10:02.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.wal_dir: 2024-04-23T16:10:02.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T16:10:02.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T16:10:02.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T16:10:02.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T16:10:02.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T16:10:02.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T16:10:02.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T16:10:02.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T16:10:02.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.write_buffer_manager: 0x556cb46770e0 2024-04-23T16:10:02.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T16:10:02.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T16:10:02.396 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T16:10:02.396 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T16:10:02.396 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T16:10:02.396 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T16:10:02.396 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T16:10:02.396 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T16:10:02.396 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.unordered_write: 0 2024-04-23T16:10:02.396 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T16:10:02.396 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T16:10:02.396 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T16:10:02.396 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T16:10:02.397 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.row_cache: None 2024-04-23T16:10:02.397 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.wal_filter: None 2024-04-23T16:10:02.397 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T16:10:02.397 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T16:10:02.397 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.two_write_queues: 0 2024-04-23T16:10:02.397 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T16:10:02.397 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.wal_compression: 0 2024-04-23T16:10:02.397 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.atomic_flush: 0 2024-04-23T16:10:02.397 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T16:10:02.397 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T16:10:02.397 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T16:10:02.397 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.log_readahead_size: 0 2024-04-23T16:10:02.397 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T16:10:02.398 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T16:10:02.398 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T16:10:02.398 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T16:10:02.398 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T16:10:02.398 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T16:10:02.398 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-23T16:10:02.400 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_background_jobs: 2 2024-04-23T16:10:02.400 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_background_compactions: -1 2024-04-23T16:10:02.400 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_subcompactions: 1 2024-04-23T16:10:02.400 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T16:10:02.400 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T16:10:02.400 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T16:10:02.400 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T16:10:02.400 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T16:10:02.401 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T16:10:02.401 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T16:10:02.401 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T16:10:02.401 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_open_files: -1 2024-04-23T16:10:02.401 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T16:10:02.401 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T16:10:02.401 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T16:10:02.401 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T16:10:02.401 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_background_flushes: -1 2024-04-23T16:10:02.401 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Compression algorithms supported: 2024-04-23T16:10:02.401 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T16:10:02.402 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: kZSTD supported: 0 2024-04-23T16:10:02.402 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: kXpressCompression supported: 0 2024-04-23T16:10:02.402 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T16:10:02.402 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: kZlibCompression supported: 1 2024-04-23T16:10:02.402 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: kSnappyCompression supported: 1 2024-04-23T16:10:02.402 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: kLZ4Compression supported: 1 2024-04-23T16:10:02.402 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: kBZip2Compression supported: 0 2024-04-23T16:10:02.402 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T16:10:02.402 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-23T16:10:02.402 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-23T16:10:02.402 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-23T16:10:02.403 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T16:10:02.403 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.merge_operator: 2024-04-23T16:10:02.403 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compaction_filter: None 2024-04-23T16:10:02.403 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T16:10:02.403 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T16:10:02.403 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T16:10:02.403 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T16:10:02.403 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556cb46e3e20) 2024-04-23T16:10:02.403 INFO:journalctl@ceph.mon.c.smithi188.stdout: cache_index_and_filter_blocks: 1 2024-04-23T16:10:02.403 INFO:journalctl@ceph.mon.c.smithi188.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T16:10:02.404 INFO:journalctl@ceph.mon.c.smithi188.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T16:10:02.404 INFO:journalctl@ceph.mon.c.smithi188.stdout: pin_top_level_index_and_filter: 1 2024-04-23T16:10:02.404 INFO:journalctl@ceph.mon.c.smithi188.stdout: index_type: 0 2024-04-23T16:10:02.404 INFO:journalctl@ceph.mon.c.smithi188.stdout: data_block_index_type: 0 2024-04-23T16:10:02.404 INFO:journalctl@ceph.mon.c.smithi188.stdout: index_shortening: 1 2024-04-23T16:10:02.404 INFO:journalctl@ceph.mon.c.smithi188.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T16:10:02.404 INFO:journalctl@ceph.mon.c.smithi188.stdout: checksum: 4 2024-04-23T16:10:02.404 INFO:journalctl@ceph.mon.c.smithi188.stdout: no_block_cache: 0 2024-04-23T16:10:02.404 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache: 0x556cb46a91f0 2024-04-23T16:10:02.404 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache_name: BinnedLRUCache 2024-04-23T16:10:02.404 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache_options: 2024-04-23T16:10:02.404 INFO:journalctl@ceph.mon.c.smithi188.stdout: capacity : 536870912 2024-04-23T16:10:02.405 INFO:journalctl@ceph.mon.c.smithi188.stdout: num_shard_bits : 4 2024-04-23T16:10:02.405 INFO:journalctl@ceph.mon.c.smithi188.stdout: strict_capacity_limit : 0 2024-04-23T16:10:02.405 INFO:journalctl@ceph.mon.c.smithi188.stdout: high_pri_pool_ratio: 0.000 2024-04-23T16:10:02.405 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache_compressed: (nil) 2024-04-23T16:10:02.405 INFO:journalctl@ceph.mon.c.smithi188.stdout: persistent_cache: (nil) 2024-04-23T16:10:02.405 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_size: 4096 2024-04-23T16:10:02.405 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_size_deviation: 10 2024-04-23T16:10:02.405 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_restart_interval: 16 2024-04-23T16:10:02.405 INFO:journalctl@ceph.mon.c.smithi188.stdout: index_block_restart_interval: 1 2024-04-23T16:10:02.405 INFO:journalctl@ceph.mon.c.smithi188.stdout: metadata_block_size: 4096 2024-04-23T16:10:02.405 INFO:journalctl@ceph.mon.c.smithi188.stdout: partition_filters: 0 2024-04-23T16:10:02.405 INFO:journalctl@ceph.mon.c.smithi188.stdout: use_delta_encoding: 1 2024-04-23T16:10:02.406 INFO:journalctl@ceph.mon.c.smithi188.stdout: filter_policy: bloomfilter 2024-04-23T16:10:02.406 INFO:journalctl@ceph.mon.c.smithi188.stdout: whole_key_filtering: 1 2024-04-23T16:10:02.406 INFO:journalctl@ceph.mon.c.smithi188.stdout: verify_compression: 0 2024-04-23T16:10:02.406 INFO:journalctl@ceph.mon.c.smithi188.stdout: read_amp_bytes_per_bit: 0 2024-04-23T16:10:02.406 INFO:journalctl@ceph.mon.c.smithi188.stdout: format_version: 5 2024-04-23T16:10:02.406 INFO:journalctl@ceph.mon.c.smithi188.stdout: enable_index_compression: 1 2024-04-23T16:10:02.406 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_align: 0 2024-04-23T16:10:02.406 INFO:journalctl@ceph.mon.c.smithi188.stdout: max_auto_readahead_size: 262144 2024-04-23T16:10:02.406 INFO:journalctl@ceph.mon.c.smithi188.stdout: prepopulate_block_cache: 0 2024-04-23T16:10:02.406 INFO:journalctl@ceph.mon.c.smithi188.stdout: initial_auto_readahead_size: 8192 2024-04-23T16:10:02.406 INFO:journalctl@ceph.mon.c.smithi188.stdout: num_file_reads_for_auto_readahead: 2 2024-04-23T16:10:02.406 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T16:10:02.407 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T16:10:02.407 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compression: NoCompression 2024-04-23T16:10:02.407 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T16:10:02.407 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T16:10:02.407 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T16:10:02.407 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.num_levels: 7 2024-04-23T16:10:02.407 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T16:10:02.407 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T16:10:02.407 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T16:10:02.407 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T16:10:02.407 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T16:10:02.407 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T16:10:02.408 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T16:10:02.408 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T16:10:02.408 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T16:10:02.408 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T16:10:02.408 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-23T16:10:02.408 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-23T16:10:02.408 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T16:10:02.408 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T16:10:02.408 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T16:10:02.408 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T16:10:02.408 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T16:10:02.408 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-23T16:10:02.408 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T16:10:02.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T16:10:02.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-23T16:10:02.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T16:10:02.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T16:10:02.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T16:10:02.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T16:10:02.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T16:10:02.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T16:10:02.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T16:10:02.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T16:10:02.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T16:10:02.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T16:10:02.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T16:10:02.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T16:10:02.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T16:10:02.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T16:10:02.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T16:10:02.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T16:10:02.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T16:10:02.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-23T16:10:02.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.arena_block_size: 1048576 2024-04-23T16:10:02.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T16:10:02.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T16:10:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T16:10:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T16:10:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T16:10:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T16:10:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T16:10:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T16:10:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T16:10:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T16:10:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T16:10:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T16:10:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T16:10:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-23T16:10:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.inplace_update_support: 0 2024-04-23T16:10:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T16:10:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T16:10:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T16:10:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T16:10:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.bloom_locality: 0 2024-04-23T16:10:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.max_successive_merges: 0 2024-04-23T16:10:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T16:10:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T16:10:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T16:10:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T16:10:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.ttl: 2592000 2024-04-23T16:10:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T16:10:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-23T16:10:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-23T16:10:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.enable_blob_files: false 2024-04-23T16:10:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.min_blob_size: 0 2024-04-23T16:10:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T16:10:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T16:10:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T16:10:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T16:10:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-23T16:10:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-23T16:10:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.blob_file_starting_level: 0 2024-04-23T16:10:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-23T16:10:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-23T16:10:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-23T16:10:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 08610977-5484-4fe7-940a-2145dce862a8 2024-04-23T16:10:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713888602014925, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-23T16:10:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-23T16:10:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713888602015377, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1613, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713888602, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "08610977-5484-4fe7-940a-2145dce862a8", "db_session_id": "J895XKM4VZHM61JOENSM", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-23T16:10:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713888602015453, "job": 1, "event": "recovery_finished"} 2024-04-23T16:10:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-23T16:10:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T16:10:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556cb47be000 2024-04-23T16:10:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: DB pointer 0x556cb47a8000 2024-04-23T16:10:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-23T16:10:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-23T16:10:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** DB Stats ** 2024-04-23T16:10:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T16:10:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T16:10:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T16:10:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T16:10:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T16:10:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T16:10:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T16:10:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-23T16:10:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** Compaction Stats [default] ** 2024-04-23T16:10:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T16:10:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-23T16:10:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T16:10:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T16:10:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T16:10:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-23T16:10:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** Compaction Stats [default] ** 2024-04-23T16:10:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T16:10:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T16:10:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T16:10:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-23T16:10:02.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-23T16:10:02.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-23T16:10:02.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T16:10:02.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T16:10:02.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T16:10:02.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T16:10:02.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T16:10:02.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T16:10:02.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T16:10:02.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T16:10:02.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T16:10:02.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Block cache BinnedLRUCache@0x556cb46a91f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-23T16:10:02.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-23T16:10:02.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-23T16:10:02.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T16:10:02.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-23T16:10:02.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: using public_addrv v1:172.21.15.188:6789/0 2024-04-23T16:10:02.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: starting mon.c rank -1 at public addrs v1:172.21.15.188:6789/0 at bind addrs v1:172.21.15.188:6789/0 mon_data /var/lib/ceph/mon/ceph-c fsid 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:10:02.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: mon.c@-1(???) e0 preinit fsid 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:10:02.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: mon.c@-1(synchronizing).mds e1 new map 2024-04-23T16:10:02.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-23T16:10:02.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: e1 2024-04-23T16:10:02.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T16:10:02.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T16:10:02.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: legacy client fscid: -1 2024-04-23T16:10:02.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-23T16:10:02.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: No filesystems configured 2024-04-23T16:10:02.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-04-23T16:10:02.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-23T16:10:02.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-23T16:10:02.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: (Original Log Time 2024/04/23-16:10:02.036875) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-04-23T16:10:02.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-04-23T16:10:02.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713888602037053, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1145, "num_deletes": 23, "total_data_size": 2365966, "memory_usage": 2439536, "flush_reason": "Manual Compaction"} 2024-04-23T16:10:02.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-04-23T16:10:02.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-23T16:10:02.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-23T16:10:02.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-23T16:10:02.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-23T16:10:02.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-23T16:10:02.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T16:10:02.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T16:10:02.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T16:10:02.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: Added host smithi148 2024-04-23T16:10:02.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:02.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:02.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:10:02.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='client.14182 v1:172.21.15.103:0/409889851' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:10:02.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:02.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi148", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:10:02.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:02.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:10:02.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: Updating smithi148:/etc/ceph/ceph.conf 2024-04-23T16:10:02.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: Updating smithi148:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T16:10:02.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-04-23T16:10:02.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: Updating smithi148:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T16:10:02.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:02.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:02.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:10:02.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='client.14184 v1:172.21.15.103:0/4232709072' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:10:02.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:02.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: Deploying cephadm binary to smithi188 2024-04-23T16:10:02.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:02.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: Added host smithi188 2024-04-23T16:10:02.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:02.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:02.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:02.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='client.14186 v1:172.21.15.103:0/97525693' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:10:02.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:02.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:10:02.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:02.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:10:02.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-23T16:10:02.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: Updating smithi188:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T16:10:02.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-23T16:10:02.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: Updating smithi188:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T16:10:02.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/912320622' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-23T16:10:02.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:02.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/912320622' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-23T16:10:02.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T16:10:02.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:02.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:02.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:02.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:10:02.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:02.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T16:10:02.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:02.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='client.14190 v1:172.21.15.188:0/3108739857' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi103:[v1:172.21.15.103:6789]=a;smithi148:[v1:172.21.15.148:6789]=b;smithi188:[v1:172.21.15.188:6789]=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:10:02.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: Saving service mon spec with placement smithi103:[v1:172.21.15.103:6789]=a;smithi148:[v1:172.21.15.148:6789]=b;smithi188:[v1:172.21.15.188:6789]=c;count:3 2024-04-23T16:10:02.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: Deploying daemon mon.c on smithi188 2024-04-23T16:10:02.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:10:02.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:02.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 ceph-mon[44101]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-23T16:10:02.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 bash[44034]: a4c2ead8543969afb7ee4ddf3e3199a6db4ab1c77adbd737f2eeab54438b55e6 2024-04-23T16:10:02.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:02 smithi188 systemd[1]: Started Ceph mon.c for 8e3594f2-018b-11ef-bc93-c7b262605968. 2024-04-23T16:10:03.673 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T16:10:03.674 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T16:10:03.946 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:10:07.385 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:10:07.386 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":2,"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","modified":"2024-04-23T16:10:02.041720Z","created":"2024-04-23T16:08:10.387384Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6789","nonce":0}]},"addr":"172.21.15.188:6789/0","public_addr":"172.21.15.188:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-23T16:10:07.388 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 2 2024-04-23T16:10:07.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:10:07.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: mon.a calling monitor election 2024-04-23T16:10:07.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:10:07.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:07.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:10:07.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:10:07.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: mon.c calling monitor election 2024-04-23T16:10:07.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:07.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:10:07.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:10:07.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:07.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:10:07.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T16:10:07.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: monmap epoch 2 2024-04-23T16:10:07.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: fsid 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:10:07.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: last_changed 2024-04-23T16:10:02.041720+0000 2024-04-23T16:10:07.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: created 2024-04-23T16:08:10.387384+0000 2024-04-23T16:10:07.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: min_mon_release 19 (squid) 2024-04-23T16:10:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: election_strategy: 1 2024-04-23T16:10:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: 0: v1:172.21.15.103:6789/0 mon.a 2024-04-23T16:10:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: 1: v1:172.21.15.188:6789/0 mon.c 2024-04-23T16:10:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: fsmap 2024-04-23T16:10:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T16:10:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: mgrmap e14: y(active, since 52s) 2024-04-23T16:10:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: overall HEALTH_OK 2024-04-23T16:10:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T16:10:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:10:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: mon.a calling monitor election 2024-04-23T16:10:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:10:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:10:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:10:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: mon.c calling monitor election 2024-04-23T16:10:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:10:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:10:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:07.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:10:07.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T16:10:07.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: monmap epoch 2 2024-04-23T16:10:07.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: fsid 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:10:07.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: last_changed 2024-04-23T16:10:02.041720+0000 2024-04-23T16:10:07.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: created 2024-04-23T16:08:10.387384+0000 2024-04-23T16:10:07.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: min_mon_release 19 (squid) 2024-04-23T16:10:07.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: election_strategy: 1 2024-04-23T16:10:07.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: 0: v1:172.21.15.103:6789/0 mon.a 2024-04-23T16:10:07.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: 1: v1:172.21.15.188:6789/0 mon.c 2024-04-23T16:10:07.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: fsmap 2024-04-23T16:10:07.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T16:10:07.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: mgrmap e14: y(active, since 52s) 2024-04-23T16:10:07.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: overall HEALTH_OK 2024-04-23T16:10:07.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:07.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:07.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:07.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T16:10:07.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:08.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:08 smithi188 ceph-mon[44101]: Deploying daemon mon.b on smithi148 2024-04-23T16:10:08.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:08 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.188:0/2548290675' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T16:10:08.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:08 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:10:08.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:08 smithi103 ceph-mon[39629]: Deploying daemon mon.b on smithi148 2024-04-23T16:10:08.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:08 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.188:0/2548290675' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T16:10:08.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:08 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:10:08.852 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T16:10:08.852 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T16:10:09.125 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:10:09.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:09 smithi188 ceph-mon[44101]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:09.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:09 smithi103 ceph-mon[39629]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:11.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:10 smithi148 bash[44481]: 231afe9eff93fdb93afd08bf51257b08f636ced72c4c048946834962c9983de6 2024-04-23T16:10:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:10 smithi148 systemd[1]: Started Ceph mon.b for 8e3594f2-018b-11ef-bc93-c7b262605968. 2024-04-23T16:10:15.851 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T16:10:15.852 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":3,"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","modified":"2024-04-23T16:10:10.820109Z","created":"2024-04-23T16:08:10.387384Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6789","nonce":0}]},"addr":"172.21.15.188:6789/0","public_addr":"172.21.15.188:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-23T16:10:15.854 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 3 2024-04-23T16:10:16.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:10:16.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: mon.c calling monitor election 2024-04-23T16:10:16.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: mon.a calling monitor election 2024-04-23T16:10:16.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:16.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:10:16.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.188:0/2337114115' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T16:10:16.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:16.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:16.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:16.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:16.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:16.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:16.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:16.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T16:10:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: monmap epoch 3 2024-04-23T16:10:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: fsid 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:10:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: last_changed 2024-04-23T16:10:10.820109+0000 2024-04-23T16:10:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: created 2024-04-23T16:08:10.387384+0000 2024-04-23T16:10:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: min_mon_release 19 (squid) 2024-04-23T16:10:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: election_strategy: 1 2024-04-23T16:10:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: 0: v1:172.21.15.103:6789/0 mon.a 2024-04-23T16:10:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: 1: v1:172.21.15.188:6789/0 mon.c 2024-04-23T16:10:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: 2: v1:172.21.15.148:6789/0 mon.b 2024-04-23T16:10:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: fsmap 2024-04-23T16:10:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T16:10:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: mgrmap e14: y(active, since 61s) 2024-04-23T16:10:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-23T16:10:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-23T16:10:16.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-23T16:10:16.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: mon.b (rank 2) addr v1:172.21.15.148:6789/0 is down (out of quorum) 2024-04-23T16:10:16.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:16.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:16.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:10:16.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: mon.c calling monitor election 2024-04-23T16:10:16.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: mon.a calling monitor election 2024-04-23T16:10:16.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:16.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:10:16.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.188:0/2337114115' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T16:10:16.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:16.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:16.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:16.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:16.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:16.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:16.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:16.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T16:10:16.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: monmap epoch 3 2024-04-23T16:10:16.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: fsid 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:10:16.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: last_changed 2024-04-23T16:10:10.820109+0000 2024-04-23T16:10:16.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: created 2024-04-23T16:08:10.387384+0000 2024-04-23T16:10:16.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: min_mon_release 19 (squid) 2024-04-23T16:10:16.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: election_strategy: 1 2024-04-23T16:10:16.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: 0: v1:172.21.15.103:6789/0 mon.a 2024-04-23T16:10:16.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: 1: v1:172.21.15.188:6789/0 mon.c 2024-04-23T16:10:16.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: 2: v1:172.21.15.148:6789/0 mon.b 2024-04-23T16:10:16.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: fsmap 2024-04-23T16:10:16.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T16:10:16.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: mgrmap e14: y(active, since 61s) 2024-04-23T16:10:16.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-23T16:10:16.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-23T16:10:16.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-23T16:10:16.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: mon.b (rank 2) addr v1:172.21.15.148:6789/0 is down (out of quorum) 2024-04-23T16:10:16.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:16.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:16.537 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-23T16:10:16.538 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-23T16:10:16.814 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:10:17.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:17.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:16 smithi188 ceph-mon[44101]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:17.293 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:17.293 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:17.293 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:17.293 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:16 smithi103 ceph-mon[39629]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:17.293 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:10:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: mon.c calling monitor election 2024-04-23T16:10:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: mon.a calling monitor election 2024-04-23T16:10:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:10:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.188:0/2337114115' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T16:10:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:18.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:18.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:18.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:18.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T16:10:18.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: monmap epoch 3 2024-04-23T16:10:18.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: fsid 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:10:18.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: last_changed 2024-04-23T16:10:10.820109+0000 2024-04-23T16:10:18.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: created 2024-04-23T16:08:10.387384+0000 2024-04-23T16:10:18.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: min_mon_release 19 (squid) 2024-04-23T16:10:18.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: election_strategy: 1 2024-04-23T16:10:18.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: 0: v1:172.21.15.103:6789/0 mon.a 2024-04-23T16:10:18.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: 1: v1:172.21.15.188:6789/0 mon.c 2024-04-23T16:10:18.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: 2: v1:172.21.15.148:6789/0 mon.b 2024-04-23T16:10:18.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: fsmap 2024-04-23T16:10:18.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T16:10:18.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: mgrmap e14: y(active, since 61s) 2024-04-23T16:10:18.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-23T16:10:18.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-23T16:10:18.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-23T16:10:18.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: mon.b (rank 2) addr v1:172.21.15.148:6789/0 is down (out of quorum) 2024-04-23T16:10:18.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:18.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:18.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:18.450 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: mon.b calling monitor election 2024-04-23T16:10:18.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: mon.b calling monitor election 2024-04-23T16:10:18.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: mon.c calling monitor election 2024-04-23T16:10:18.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: mon.a calling monitor election 2024-04-23T16:10:18.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T16:10:18.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: monmap epoch 3 2024-04-23T16:10:18.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: fsid 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:10:18.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: last_changed 2024-04-23T16:10:10.820109+0000 2024-04-23T16:10:18.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: created 2024-04-23T16:08:10.387384+0000 2024-04-23T16:10:18.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: min_mon_release 19 (squid) 2024-04-23T16:10:18.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: election_strategy: 1 2024-04-23T16:10:18.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: 0: v1:172.21.15.103:6789/0 mon.a 2024-04-23T16:10:18.452 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: 1: v1:172.21.15.188:6789/0 mon.c 2024-04-23T16:10:18.452 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: 2: v1:172.21.15.148:6789/0 mon.b 2024-04-23T16:10:18.452 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: fsmap 2024-04-23T16:10:18.452 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T16:10:18.452 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: mgrmap e14: y(active, since 63s) 2024-04-23T16:10:18.452 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-23T16:10:18.452 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:10:18.452 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: overall HEALTH_OK 2024-04-23T16:10:18.452 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.452 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.452 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.452 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.452 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.452 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.452 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.453 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.453 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.453 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.453 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.453 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T16:10:18.453 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T16:10:18.453 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:18.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: mon.b calling monitor election 2024-04-23T16:10:18.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: mon.b calling monitor election 2024-04-23T16:10:18.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: mon.c calling monitor election 2024-04-23T16:10:18.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: mon.a calling monitor election 2024-04-23T16:10:18.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T16:10:18.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: monmap epoch 3 2024-04-23T16:10:18.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: fsid 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:10:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: last_changed 2024-04-23T16:10:10.820109+0000 2024-04-23T16:10:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: created 2024-04-23T16:08:10.387384+0000 2024-04-23T16:10:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: min_mon_release 19 (squid) 2024-04-23T16:10:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: election_strategy: 1 2024-04-23T16:10:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: 0: v1:172.21.15.103:6789/0 mon.a 2024-04-23T16:10:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: 1: v1:172.21.15.188:6789/0 mon.c 2024-04-23T16:10:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: 2: v1:172.21.15.148:6789/0 mon.b 2024-04-23T16:10:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: fsmap 2024-04-23T16:10:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T16:10:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: mgrmap e14: y(active, since 63s) 2024-04-23T16:10:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-23T16:10:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:10:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: overall HEALTH_OK 2024-04-23T16:10:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T16:10:18.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T16:10:18.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:18.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: mon.b calling monitor election 2024-04-23T16:10:18.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: mon.b calling monitor election 2024-04-23T16:10:18.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: mon.c calling monitor election 2024-04-23T16:10:18.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: mon.a calling monitor election 2024-04-23T16:10:18.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T16:10:18.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: monmap epoch 3 2024-04-23T16:10:18.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: fsid 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:10:18.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: last_changed 2024-04-23T16:10:10.820109+0000 2024-04-23T16:10:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: created 2024-04-23T16:08:10.387384+0000 2024-04-23T16:10:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: min_mon_release 19 (squid) 2024-04-23T16:10:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: election_strategy: 1 2024-04-23T16:10:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: 0: v1:172.21.15.103:6789/0 mon.a 2024-04-23T16:10:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: 1: v1:172.21.15.188:6789/0 mon.c 2024-04-23T16:10:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: 2: v1:172.21.15.148:6789/0 mon.b 2024-04-23T16:10:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: fsmap 2024-04-23T16:10:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T16:10:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: mgrmap e14: y(active, since 63s) 2024-04-23T16:10:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-23T16:10:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:10:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: overall HEALTH_OK 2024-04-23T16:10:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:18.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T16:10:18.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T16:10:18.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:18.899 INFO:teuthology.orchestra.run.smithi103.stdout:# minimal ceph.conf for 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:10:18.900 INFO:teuthology.orchestra.run.smithi103.stdout:[global] 2024-04-23T16:10:18.900 INFO:teuthology.orchestra.run.smithi103.stdout: fsid = 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:10:18.900 INFO:teuthology.orchestra.run.smithi103.stdout: mon_host = 172.21.15.103:6789/0 172.21.15.148:6789/0 172.21.15.188:6789/0 2024-04-23T16:10:19.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:19 smithi103 ceph-mon[39629]: Updating smithi103:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T16:10:19.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:19 smithi103 ceph-mon[39629]: Updating smithi148:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T16:10:19.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:19 smithi103 ceph-mon[39629]: Updating smithi188:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T16:10:19.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:19 smithi103 ceph-mon[39629]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T16:10:19.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:19 smithi103 ceph-mon[39629]: Reconfiguring daemon mon.a on smithi103 2024-04-23T16:10:19.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:19 smithi103 ceph-mon[39629]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:19.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:19 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:19.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:19 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1402203085' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:19 smithi148 ceph-mon[44520]: Updating smithi103:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T16:10:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:19 smithi148 ceph-mon[44520]: Updating smithi148:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T16:10:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:19 smithi148 ceph-mon[44520]: Updating smithi188:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T16:10:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:19 smithi148 ceph-mon[44520]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T16:10:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:19 smithi148 ceph-mon[44520]: Reconfiguring daemon mon.a on smithi103 2024-04-23T16:10:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:19 smithi148 ceph-mon[44520]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:19 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:19 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1402203085' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:19.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:19 smithi188 ceph-mon[44101]: Updating smithi103:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T16:10:19.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:19 smithi188 ceph-mon[44101]: Updating smithi148:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T16:10:19.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:19 smithi188 ceph-mon[44101]: Updating smithi188:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T16:10:19.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:19 smithi188 ceph-mon[44101]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T16:10:19.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:19 smithi188 ceph-mon[44101]: Reconfiguring daemon mon.a on smithi103 2024-04-23T16:10:19.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:19 smithi188 ceph-mon[44101]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:19.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:19 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:10:19.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:19 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1402203085' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:20.077 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-23T16:10:20.078 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-23T16:10:20.078 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T16:10:20.111 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-23T16:10:20.111 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T16:10:20.180 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-04-23T16:10:20.180 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T16:10:20.217 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-04-23T16:10:20.218 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T16:10:20.290 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T16:10:20.290 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T16:10:20.326 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T16:10:20.327 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T16:10:20.398 INFO:tasks.cephadm:Adding mgr.y on smithi103 2024-04-23T16:10:20.398 INFO:tasks.cephadm:Adding mgr.x on smithi148 2024-04-23T16:10:20.398 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph orch apply mgr '2;smithi103=y;smithi148=x' 2024-04-23T16:10:20.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:20 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:20.698 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:10:20.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:20 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:20.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:20 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:21.613 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:21 smithi188 ceph-mon[44101]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:21.614 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:21 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:21.614 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:21 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:21.614 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:21 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T16:10:21.614 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:21 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T16:10:21.614 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:21 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:21.725 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:21 smithi148 ceph-mon[44520]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:21.725 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:21 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:21.725 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:21 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:21.725 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:21 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T16:10:21.725 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:21 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T16:10:21.725 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:21 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:21.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:21 smithi103 ceph-mon[39629]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:21.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:21 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:21.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:21 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:21.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:21 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T16:10:21.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:21 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T16:10:21.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:21 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:22.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:22 smithi188 ceph-mon[44101]: Reconfiguring mon.b (monmap changed)... 2024-04-23T16:10:22.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:22 smithi188 ceph-mon[44101]: Reconfiguring daemon mon.b on smithi148 2024-04-23T16:10:22.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:22 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:22 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:22 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T16:10:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:22 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T16:10:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:22 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:22.633 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mgr update... 2024-04-23T16:10:22.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:22 smithi103 ceph-mon[39629]: Reconfiguring mon.b (monmap changed)... 2024-04-23T16:10:22.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:22 smithi103 ceph-mon[39629]: Reconfiguring daemon mon.b on smithi148 2024-04-23T16:10:22.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:22 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:22.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:22 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:22.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:22 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T16:10:22.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:22 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T16:10:22.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:22 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:22.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:22 smithi148 ceph-mon[44520]: Reconfiguring mon.b (monmap changed)... 2024-04-23T16:10:22.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:22 smithi148 ceph-mon[44520]: Reconfiguring daemon mon.b on smithi148 2024-04-23T16:10:22.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:22 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:22.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:22 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:22.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:22 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T16:10:22.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:22 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T16:10:22.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:22 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:23.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:23 smithi188 ceph-mon[44101]: Reconfiguring mon.c (monmap changed)... 2024-04-23T16:10:23.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:23 smithi188 ceph-mon[44101]: Reconfiguring daemon mon.c on smithi188 2024-04-23T16:10:23.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:23 smithi188 ceph-mon[44101]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:23.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:23.698 DEBUG:teuthology.orchestra.run.smithi148:mgr.x> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mgr.x.service 2024-04-23T16:10:23.701 INFO:tasks.cephadm:Deploying OSDs... 2024-04-23T16:10:23.701 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-23T16:10:23.701 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T16:10:23.724 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T16:10:23.724 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-04-23T16:10:23.731 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:23 smithi148 ceph-mon[44520]: Reconfiguring mon.c (monmap changed)... 2024-04-23T16:10:23.731 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:23 smithi148 ceph-mon[44520]: Reconfiguring daemon mon.c on smithi188 2024-04-23T16:10:23.731 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:23 smithi148 ceph-mon[44520]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:23.731 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:23.736 INFO:journalctl@ceph.mgr.x.smithi148.stdout:-- Logs begin at Tue 2024-04-23 15:57:31 UTC. -- 2024-04-23T16:10:23.785 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T16:10:23.785 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:10:23.785 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 167503 Links: 1 2024-04-23T16:10:23.785 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:10:23.785 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T16:10:23.785 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-23 16:09:27.513265573 +0000 2024-04-23T16:10:23.785 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-23 16:03:41.994786940 +0000 2024-04-23T16:10:23.786 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-23 16:03:41.994786940 +0000 2024-04-23T16:10:23.786 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: - 2024-04-23T16:10:23.786 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T16:10:23.854 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:23 smithi103 ceph-mon[39629]: Reconfiguring mon.c (monmap changed)... 2024-04-23T16:10:23.854 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:23 smithi103 ceph-mon[39629]: Reconfiguring daemon mon.c on smithi188 2024-04-23T16:10:23.854 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:23 smithi103 ceph-mon[39629]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:23.854 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:23.858 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-23T16:10:23.858 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-23T16:10:23.858 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.00023847 s, 2.1 MB/s 2024-04-23T16:10:23.859 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T16:10:23.920 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-04-23T16:10:23.980 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T16:10:23.980 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:10:23.980 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 165797 Links: 1 2024-04-23T16:10:23.980 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:10:23.980 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T16:10:23.980 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-23 16:09:27.569264525 +0000 2024-04-23T16:10:23.980 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-23 16:03:42.322780788 +0000 2024-04-23T16:10:23.980 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-23 16:03:42.322780788 +0000 2024-04-23T16:10:23.980 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: - 2024-04-23T16:10:23.981 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T16:10:24.049 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-23T16:10:24.049 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-23T16:10:24.049 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000225079 s, 2.3 MB/s 2024-04-23T16:10:24.050 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T16:10:24.111 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-04-23T16:10:24.170 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T16:10:24.171 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:10:24.171 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 165836 Links: 1 2024-04-23T16:10:24.171 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:10:24.171 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T16:10:24.171 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-23 16:09:27.625263477 +0000 2024-04-23T16:10:24.171 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-23 16:03:42.665774355 +0000 2024-04-23T16:10:24.171 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-23 16:03:42.665774355 +0000 2024-04-23T16:10:24.171 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: - 2024-04-23T16:10:24.171 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T16:10:24.238 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-23T16:10:24.238 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-23T16:10:24.238 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000188661 s, 2.7 MB/s 2024-04-23T16:10:24.239 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T16:10:24.298 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-04-23T16:10:24.359 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T16:10:24.359 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:10:24.359 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 171078 Links: 1 2024-04-23T16:10:24.359 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:10:24.359 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T16:10:24.359 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-23 16:09:27.684262373 +0000 2024-04-23T16:10:24.359 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-23 16:03:43.010767884 +0000 2024-04-23T16:10:24.359 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-23 16:03:43.010767884 +0000 2024-04-23T16:10:24.359 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: - 2024-04-23T16:10:24.360 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T16:10:24.427 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-23T16:10:24.427 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-23T16:10:24.427 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000235764 s, 2.2 MB/s 2024-04-23T16:10:24.428 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T16:10:24.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:24 smithi188 ceph-mon[44101]: from='client.14211 v1:172.21.15.188:0/1131559750' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi103=y;smithi148=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:10:24.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:24 smithi188 ceph-mon[44101]: Saving service mgr spec with placement smithi103=y;smithi148=x;count:2 2024-04-23T16:10:24.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:24 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:24.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:24 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:24.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:24 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:24.487 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-04-23T16:10:24.487 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T16:10:24.510 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T16:10:24.510 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_1 2024-04-23T16:10:24.571 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T16:10:24.572 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:10:24.572 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 6h/6d Inode: 169756 Links: 1 2024-04-23T16:10:24.572 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:10:24.572 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T16:10:24.572 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-04-23 16:10:10.224939265 +0000 2024-04-23T16:10:24.572 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-04-23 16:03:45.262688897 +0000 2024-04-23T16:10:24.572 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-04-23 16:03:45.262688897 +0000 2024-04-23T16:10:24.572 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2024-04-23T16:10:24.572 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T16:10:24.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:24 smithi103 ceph-mon[39629]: from='client.14211 v1:172.21.15.188:0/1131559750' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi103=y;smithi148=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:10:24.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:24 smithi103 ceph-mon[39629]: Saving service mgr spec with placement smithi103=y;smithi148=x;count:2 2024-04-23T16:10:24.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:24 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:24.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:24 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:24.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:24 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:24.642 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-04-23T16:10:24.642 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-04-23T16:10:24.642 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000219063 s, 2.3 MB/s 2024-04-23T16:10:24.643 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T16:10:24.706 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_2 2024-04-23T16:10:24.766 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T16:10:24.766 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:10:24.766 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 6h/6d Inode: 170328 Links: 1 2024-04-23T16:10:24.766 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:10:24.766 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T16:10:24.766 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-04-23 16:10:10.224939265 +0000 2024-04-23T16:10:24.766 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-04-23 16:03:45.616680597 +0000 2024-04-23T16:10:24.766 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-04-23 16:03:45.616680597 +0000 2024-04-23T16:10:24.766 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2024-04-23T16:10:24.767 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T16:10:24.833 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:24 smithi148 ceph-mon[44520]: from='client.14211 v1:172.21.15.188:0/1131559750' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi103=y;smithi148=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:10:24.833 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:24 smithi148 ceph-mon[44520]: Saving service mgr spec with placement smithi103=y;smithi148=x;count:2 2024-04-23T16:10:24.833 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:24 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:24.833 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:24 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:24.833 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:24 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:24.835 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-04-23T16:10:24.835 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-04-23T16:10:24.835 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000210683 s, 2.4 MB/s 2024-04-23T16:10:24.836 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T16:10:24.897 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_3 2024-04-23T16:10:24.957 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T16:10:24.957 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:10:24.958 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 6h/6d Inode: 170375 Links: 1 2024-04-23T16:10:24.958 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:10:24.958 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T16:10:24.958 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-04-23 16:10:10.224939265 +0000 2024-04-23T16:10:24.958 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-04-23 16:03:45.971672274 +0000 2024-04-23T16:10:24.958 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-04-23 16:03:45.971672274 +0000 2024-04-23T16:10:24.958 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2024-04-23T16:10:24.958 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T16:10:25.024 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-04-23T16:10:25.024 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-04-23T16:10:25.024 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000185463 s, 2.8 MB/s 2024-04-23T16:10:25.025 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T16:10:25.084 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_4 2024-04-23T16:10:25.142 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T16:10:25.142 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:10:25.142 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 6h/6d Inode: 165428 Links: 1 2024-04-23T16:10:25.142 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:10:25.142 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T16:10:25.142 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-04-23 16:10:10.224939265 +0000 2024-04-23T16:10:25.142 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-04-23 16:03:46.305664443 +0000 2024-04-23T16:10:25.142 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-04-23 16:03:46.305664443 +0000 2024-04-23T16:10:25.143 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2024-04-23T16:10:25.143 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T16:10:25.208 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-04-23T16:10:25.208 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-04-23T16:10:25.208 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000231577 s, 2.2 MB/s 2024-04-23T16:10:25.209 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T16:10:25.268 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T16:10:25.268 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T16:10:25.291 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T16:10:25.291 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_1 2024-04-23T16:10:25.352 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T16:10:25.352 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:10:25.352 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 163450 Links: 1 2024-04-23T16:10:25.352 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:10:25.352 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T16:10:25.352 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-23 16:10:00.799260163 +0000 2024-04-23T16:10:25.352 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-23 16:03:43.342948057 +0000 2024-04-23T16:10:25.352 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-23 16:03:43.342948057 +0000 2024-04-23T16:10:25.353 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2024-04-23T16:10:25.353 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T16:10:25.425 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-23T16:10:25.425 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-23T16:10:25.425 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000209848 s, 2.4 MB/s 2024-04-23T16:10:25.426 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T16:10:25.487 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_2 2024-04-23T16:10:25.512 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:25 smithi148 ceph-mon[44520]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:25.512 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:25.512 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:10:25.512 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:25.513 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T16:10:25.513 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T16:10:25.513 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T16:10:25.513 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:25.547 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T16:10:25.547 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:10:25.547 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 168516 Links: 1 2024-04-23T16:10:25.547 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:10:25.547 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T16:10:25.547 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-23 16:10:00.800260146 +0000 2024-04-23T16:10:25.548 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-23 16:03:43.670942065 +0000 2024-04-23T16:10:25.548 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-23 16:03:43.670942065 +0000 2024-04-23T16:10:25.548 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2024-04-23T16:10:25.548 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T16:10:25.614 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-23T16:10:25.614 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-23T16:10:25.614 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000196782 s, 2.6 MB/s 2024-04-23T16:10:25.615 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T16:10:25.674 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_3 2024-04-23T16:10:25.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:25 smithi188 ceph-mon[44101]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:25.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:25.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:10:25.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:25.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T16:10:25.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T16:10:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T16:10:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:25.732 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T16:10:25.732 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:10:25.732 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 165499 Links: 1 2024-04-23T16:10:25.733 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:10:25.733 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T16:10:25.733 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-23 16:10:00.800260146 +0000 2024-04-23T16:10:25.733 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-23 16:03:44.018935707 +0000 2024-04-23T16:10:25.733 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-23 16:03:44.018935707 +0000 2024-04-23T16:10:25.733 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2024-04-23T16:10:25.733 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T16:10:25.799 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-23T16:10:25.799 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-23T16:10:25.800 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000186928 s, 2.7 MB/s 2024-04-23T16:10:25.800 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T16:10:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:25 smithi103 ceph-mon[39629]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:10:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T16:10:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T16:10:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T16:10:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:25.860 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_4 2024-04-23T16:10:25.918 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T16:10:25.918 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T16:10:25.918 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 171074 Links: 1 2024-04-23T16:10:25.918 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T16:10:25.918 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T16:10:25.918 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-23 16:10:00.800260146 +0000 2024-04-23T16:10:25.918 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-23 16:03:44.351929623 +0000 2024-04-23T16:10:25.918 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-23 16:03:44.351929623 +0000 2024-04-23T16:10:25.918 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2024-04-23T16:10:25.919 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T16:10:25.983 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-23T16:10:25.984 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-23T16:10:25.984 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.00023296 s, 2.2 MB/s 2024-04-23T16:10:25.985 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T16:10:26.043 INFO:tasks.cephadm:Deploying osd.0 on smithi103 with /dev/vg_nvme/lv_4... 2024-04-23T16:10:26.043 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T16:10:26.312 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:10:26.558 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:26 smithi148 ceph-mon[44520]: Deploying daemon mgr.x on smithi148 2024-04-23T16:10:26.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:26 smithi188 ceph-mon[44101]: Deploying daemon mgr.x on smithi148 2024-04-23T16:10:26.803 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:26 smithi103 ceph-mon[39629]: Deploying daemon mgr.x on smithi148 2024-04-23T16:10:26.873 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:26 smithi148 systemd[1]: Starting Ceph mgr.x for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T16:10:27.400 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:27 smithi148 bash[46141]: 015709556a26d4fb736d412db668d213620be1a1063ecff6acfc6661722124fe 2024-04-23T16:10:27.400 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:27 smithi148 systemd[1]: Started Ceph mgr.x for 8e3594f2-018b-11ef-bc93-c7b262605968. 2024-04-23T16:10:27.400 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:27 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:27.230+0000 7f427ce29200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T16:10:27.400 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:27 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:27.397+0000 7f427ce29200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T16:10:27.525 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:27 smithi103 ceph-mon[39629]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:27.525 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:27.525 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:27.525 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:27.525 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:27.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:27 smithi188 ceph-mon[44101]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:27.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:27 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:27.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:27 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:27.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:27 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:27.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:27 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:27.756 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:27 smithi148 ceph-mon[44520]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:27.756 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:27 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:27.757 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:27 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:27.757 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:27 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:27.757 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:27 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:28.084 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:27 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:27.755+0000 7f427ce29200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T16:10:28.373 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:28 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:28.283+0000 7f427ce29200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T16:10:28.406 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T16:10:28.406 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T16:10:28.406 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-04-23T16:10:28.406 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-04-23T16:10:28.406 INFO:teuthology.orchestra.run.smithi103.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.013023 s, 805 MB/s 2024-04-23T16:10:28.406 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 2024-04-23T16:10:28.406 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-04-23T16:10:28.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:28 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:28.777 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:28 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:28.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:28 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:28.874 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:28 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:28.540+0000 7f427ce29200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T16:10:29.126 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_4 2024-04-23T16:10:29.373 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:28 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:28.977+0000 7f427ce29200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T16:10:29.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:29 smithi103 ceph-mon[39629]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:29.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:29.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:29.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:29.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:10:29.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:29.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T16:10:29.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T16:10:29.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:29.609 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:29.693 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:29 smithi148 ceph-mon[44520]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:29.694 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:29.694 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:29.694 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:29.694 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:10:29.694 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:29.694 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T16:10:29.694 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T16:10:29.694 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:29.694 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:29 smithi188 ceph-mon[44101]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:10:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T16:10:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T16:10:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:29.938 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:10:30.123 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:29 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:29.692+0000 7f427ce29200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T16:10:30.124 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:29 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:29.777+0000 7f427ce29200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T16:10:30.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:30 smithi188 ceph-mon[44101]: Reconfiguring mgr.y (unknown last config time)... 2024-04-23T16:10:30.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:30 smithi188 ceph-mon[44101]: Reconfiguring daemon mgr.y on smithi103 2024-04-23T16:10:30.777 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:30 smithi148 ceph-mon[44520]: Reconfiguring mgr.y (unknown last config time)... 2024-04-23T16:10:30.777 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:30 smithi148 ceph-mon[44520]: Reconfiguring daemon mgr.y on smithi103 2024-04-23T16:10:30.777 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:30 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:30.375+0000 7f427ce29200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T16:10:30.778 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:30 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:30.461+0000 7f427ce29200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T16:10:30.778 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:30 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:30.547+0000 7f427ce29200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T16:10:30.817 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:30 smithi103 ceph-mon[39629]: Reconfiguring mgr.y (unknown last config time)... 2024-04-23T16:10:30.818 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:30 smithi103 ceph-mon[39629]: Reconfiguring daemon mgr.y on smithi103 2024-04-23T16:10:31.123 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:30 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:30.775+0000 7f427ce29200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T16:10:31.123 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:30 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:30.867+0000 7f427ce29200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T16:10:31.124 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:30 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:30.961+0000 7f427ce29200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T16:10:31.469 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:31 smithi148 ceph-mon[44520]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:31.469 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:31 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:31.189+0000 7f427ce29200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T16:10:31.469 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:31 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:31.283+0000 7f427ce29200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T16:10:31.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:31 smithi188 ceph-mon[44101]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:31.732 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:31 smithi103 ceph-mon[39629]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:32.123 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:31 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:31.468+0000 7f427ce29200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T16:10:32.124 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:31 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:31.721+0000 7f427ce29200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T16:10:32.623 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:32 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:32.304+0000 7f427ce29200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T16:10:32.623 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:32 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:32.516+0000 7f427ce29200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T16:10:33.032 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:33.032 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:33.032 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:33.032 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:33.032 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:10:33.032 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:33.032 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:32 smithi103 ceph-mon[39629]: from='client.14223 v1:172.21.15.103:0/3677362226' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:10:33.033 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:10:33.033 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:10:33.033 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:33.033 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:32 smithi103 ceph-mon[39629]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:33.123 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:32 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:32.710+0000 7f427ce29200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T16:10:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:32 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:32 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:32 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:32 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:32 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:10:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:32 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:32 smithi148 ceph-mon[44520]: from='client.14223 v1:172.21.15.103:0/3677362226' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:10:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:32 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:10:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:32 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:10:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:32 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:33.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:32 smithi148 ceph-mon[44520]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:33.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:32 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:32 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:32 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:32 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:32 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:10:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:32 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:32 smithi188 ceph-mon[44101]: from='client.14223 v1:172.21.15.103:0/3677362226' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:10:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:32 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:10:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:32 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:10:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:32 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:32 smithi188 ceph-mon[44101]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:33.752 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:33 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:33.373+0000 7f427ce29200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T16:10:33.752 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:33 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:33.570+0000 7f427ce29200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T16:10:34.123 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:33 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:33.751+0000 7f427ce29200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T16:10:34.123 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 16:10:33 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x[46156]: 2024-04-23T16:10:33.865+0000 7f427ce29200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T16:10:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:33 smithi148 ceph-mon[44520]: Standby manager daemon x started 2024-04-23T16:10:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:33 smithi148 ceph-mon[44520]: from='mgr.? v1:172.21.15.148:0/4131867583' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T16:10:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:33 smithi148 ceph-mon[44520]: from='mgr.? v1:172.21.15.148:0/4131867583' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T16:10:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:33 smithi148 ceph-mon[44520]: from='mgr.? v1:172.21.15.148:0/4131867583' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T16:10:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:33 smithi148 ceph-mon[44520]: from='mgr.? v1:172.21.15.148:0/4131867583' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T16:10:34.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:33 smithi188 ceph-mon[44101]: Standby manager daemon x started 2024-04-23T16:10:34.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:33 smithi188 ceph-mon[44101]: from='mgr.? v1:172.21.15.148:0/4131867583' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T16:10:34.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:33 smithi188 ceph-mon[44101]: from='mgr.? v1:172.21.15.148:0/4131867583' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T16:10:34.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:33 smithi188 ceph-mon[44101]: from='mgr.? v1:172.21.15.148:0/4131867583' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T16:10:34.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:33 smithi188 ceph-mon[44101]: from='mgr.? v1:172.21.15.148:0/4131867583' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T16:10:34.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:33 smithi103 ceph-mon[39629]: Standby manager daemon x started 2024-04-23T16:10:34.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:33 smithi103 ceph-mon[39629]: from='mgr.? v1:172.21.15.148:0/4131867583' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T16:10:34.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:33 smithi103 ceph-mon[39629]: from='mgr.? v1:172.21.15.148:0/4131867583' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T16:10:34.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:33 smithi103 ceph-mon[39629]: from='mgr.? v1:172.21.15.148:0/4131867583' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T16:10:34.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:33 smithi103 ceph-mon[39629]: from='mgr.? v1:172.21.15.148:0/4131867583' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T16:10:35.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:34 smithi188 ceph-mon[44101]: mgrmap e15: y(active, since 79s), standbys: x 2024-04-23T16:10:35.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:34 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T16:10:35.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:34 smithi188 ceph-mon[44101]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:35.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:34 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2413196370' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c084047-8f01-4b86-a60c-5592735b5862"}]: dispatch 2024-04-23T16:10:35.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:34 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2413196370' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c084047-8f01-4b86-a60c-5592735b5862"}]': finished 2024-04-23T16:10:35.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:34 smithi188 ceph-mon[44101]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T16:10:35.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:34 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T16:10:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:34 smithi103 ceph-mon[39629]: mgrmap e15: y(active, since 79s), standbys: x 2024-04-23T16:10:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:34 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T16:10:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:34 smithi103 ceph-mon[39629]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:34 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2413196370' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c084047-8f01-4b86-a60c-5592735b5862"}]: dispatch 2024-04-23T16:10:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:34 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2413196370' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c084047-8f01-4b86-a60c-5592735b5862"}]': finished 2024-04-23T16:10:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:34 smithi103 ceph-mon[39629]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T16:10:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:34 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T16:10:35.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:34 smithi148 ceph-mon[44520]: mgrmap e15: y(active, since 79s), standbys: x 2024-04-23T16:10:35.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:34 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T16:10:35.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:34 smithi148 ceph-mon[44520]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:35.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:34 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2413196370' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c084047-8f01-4b86-a60c-5592735b5862"}]: dispatch 2024-04-23T16:10:35.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:34 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2413196370' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c084047-8f01-4b86-a60c-5592735b5862"}]': finished 2024-04-23T16:10:35.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:34 smithi148 ceph-mon[44520]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T16:10:35.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:34 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T16:10:36.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:35 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3914399648' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:10:36.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:35 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3914399648' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:10:36.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:35 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3914399648' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:10:37.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:36 smithi188 ceph-mon[44101]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:37.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:36 smithi103 ceph-mon[39629]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:37.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:36 smithi148 ceph-mon[44520]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:39.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:39 smithi188 ceph-mon[44101]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:39.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:39 smithi103 ceph-mon[39629]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:39.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:39 smithi148 ceph-mon[44520]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:41.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:41 smithi103 ceph-mon[39629]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:41.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:41 smithi188 ceph-mon[44101]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:41.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:41 smithi148 ceph-mon[44520]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:42.722 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:42 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T16:10:42.723 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:42 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:42.723 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:42 smithi103 ceph-mon[39629]: Deploying daemon osd.0 on smithi103 2024-04-23T16:10:42.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:42 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T16:10:42.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:42 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:42.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:42 smithi188 ceph-mon[44101]: Deploying daemon osd.0 on smithi103 2024-04-23T16:10:42.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:42 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T16:10:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:42 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:42 smithi148 ceph-mon[44520]: Deploying daemon osd.0 on smithi103 2024-04-23T16:10:43.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:43 smithi188 ceph-mon[44101]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:43.772 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:43 smithi103 ceph-mon[39629]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:43.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:43 smithi148 ceph-mon[44520]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:45.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:45 smithi103 ceph-mon[39629]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:45.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:45 smithi188 ceph-mon[44101]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:45.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:45 smithi148 ceph-mon[44520]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:47.674 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:47 smithi103 ceph-mon[39629]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:47.675 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:47 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:47.675 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:47 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:47.675 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:47 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:47.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:47 smithi188 ceph-mon[44101]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:47.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:47 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:47.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:47 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:47.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:47 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:47.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:47 smithi148 ceph-mon[44520]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:47 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:47 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:47 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:49.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:49 smithi103 ceph-mon[39629]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:49.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:49 smithi188 ceph-mon[44101]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:49.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:49 smithi148 ceph-mon[44520]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:50.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:50 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:50.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:50 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:50.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:50 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:50.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:50 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:10:50.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:50 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:50.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:50 smithi103 ceph-mon[39629]: from='osd.0 v1:172.21.15.103:6801/4274040097' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T16:10:50.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:50 smithi103 ceph-mon[39629]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:50.955 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 0 on host 'smithi103' 2024-04-23T16:10:50.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:50 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:50.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:50 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:50.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:50 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:50.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:50 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:10:50.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:50 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:50.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:50 smithi188 ceph-mon[44101]: from='osd.0 v1:172.21.15.103:6801/4274040097' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T16:10:50.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:50 smithi188 ceph-mon[44101]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:51.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:50 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:51.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:50 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:51.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:50 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:51.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:50 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:10:51.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:50 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:51.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:50 smithi148 ceph-mon[44520]: from='osd.0 v1:172.21.15.103:6801/4274040097' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T16:10:51.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:50 smithi148 ceph-mon[44520]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:51.598 DEBUG:teuthology.orchestra.run.smithi103:osd.0> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.0.service 2024-04-23T16:10:51.601 INFO:tasks.cephadm:Deploying osd.1 on smithi103 with /dev/vg_nvme/lv_3... 2024-04-23T16:10:51.601 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T16:10:51.632 INFO:journalctl@ceph.osd.0.smithi103.stdout:-- Logs begin at Tue 2024-04-23 15:57:24 UTC. -- 2024-04-23T16:10:51.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:51 smithi103 ceph-mon[39629]: from='osd.0 v1:172.21.15.103:6801/4274040097' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T16:10:51.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:51 smithi103 ceph-mon[39629]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T16:10:51.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:51 smithi103 ceph-mon[39629]: from='osd.0 v1:172.21.15.103:6801/4274040097' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:10:51.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:51 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T16:10:51.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:51 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:51.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:51 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:51.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:51 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:52.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:51 smithi148 ceph-mon[44520]: from='osd.0 v1:172.21.15.103:6801/4274040097' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T16:10:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:51 smithi148 ceph-mon[44520]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T16:10:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:51 smithi148 ceph-mon[44520]: from='osd.0 v1:172.21.15.103:6801/4274040097' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:10:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:51 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T16:10:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:51 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:51 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:51 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:52.180 INFO:journalctl@ceph.osd.0.smithi103.stdout:Apr 23 16:10:52 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-0[58506]: 2024-04-23T16:10:52.139+0000 7f24886fe700 -1 osd.0 0 waiting for initial osdmap 2024-04-23T16:10:52.180 INFO:journalctl@ceph.osd.0.smithi103.stdout:Apr 23 16:10:52 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-0[58506]: 2024-04-23T16:10:52.141+0000 7f24804e9700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:10:52.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:51 smithi188 ceph-mon[44101]: from='osd.0 v1:172.21.15.103:6801/4274040097' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T16:10:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:51 smithi188 ceph-mon[44101]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T16:10:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:51 smithi188 ceph-mon[44101]: from='osd.0 v1:172.21.15.103:6801/4274040097' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:10:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:51 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T16:10:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:51 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:51 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:51 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:10:52.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:52 smithi103 ceph-mon[39629]: from='osd.0 v1:172.21.15.103:6801/4274040097' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-23T16:10:52.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:52 smithi103 ceph-mon[39629]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T16:10:52.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T16:10:52.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T16:10:52.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:52 smithi103 ceph-mon[39629]: from='osd.0 v1:172.21.15.103:6801/4274040097' entity='osd.0' 2024-04-23T16:10:52.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:52 smithi103 ceph-mon[39629]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:52.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T16:10:52.860 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:10:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:52 smithi148 ceph-mon[44520]: from='osd.0 v1:172.21.15.103:6801/4274040097' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-23T16:10:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:52 smithi148 ceph-mon[44520]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T16:10:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T16:10:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T16:10:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:52 smithi148 ceph-mon[44520]: from='osd.0 v1:172.21.15.103:6801/4274040097' entity='osd.0' 2024-04-23T16:10:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:52 smithi148 ceph-mon[44520]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T16:10:53.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:52 smithi188 ceph-mon[44101]: from='osd.0 v1:172.21.15.103:6801/4274040097' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-23T16:10:53.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:52 smithi188 ceph-mon[44101]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T16:10:53.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T16:10:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T16:10:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:52 smithi188 ceph-mon[44101]: from='osd.0 v1:172.21.15.103:6801/4274040097' entity='osd.0' 2024-04-23T16:10:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:52 smithi188 ceph-mon[44101]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T16:10:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T16:10:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:54 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:10:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:54 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:10:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:10:54.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:10:54.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:54 smithi103 ceph-mon[39629]: osd.0 v1:172.21.15.103:6801/4274040097 boot 2024-04-23T16:10:54.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:54 smithi103 ceph-mon[39629]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T16:10:54.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:54 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T16:10:54.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:54 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:10:54.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:54 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:10:54.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:10:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:10:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:54 smithi188 ceph-mon[44101]: osd.0 v1:172.21.15.103:6801/4274040097 boot 2024-04-23T16:10:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:54 smithi188 ceph-mon[44101]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T16:10:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:54 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T16:10:54.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:54 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:10:54.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:54 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:10:54.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:10:54.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:10:54.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:54 smithi148 ceph-mon[44520]: osd.0 v1:172.21.15.103:6801/4274040097 boot 2024-04-23T16:10:54.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:54 smithi148 ceph-mon[44520]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T16:10:54.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:54 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T16:10:55.433 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:55 smithi103 ceph-mon[39629]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T16:10:55.434 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:55 smithi103 ceph-mon[39629]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:10:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:55 smithi188 ceph-mon[44101]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T16:10:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:55 smithi188 ceph-mon[44101]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:10:55.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:55 smithi148 ceph-mon[44520]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T16:10:55.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:55 smithi148 ceph-mon[44520]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:10:56.031 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T16:10:56.032 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T16:10:56.032 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-04-23T16:10:56.032 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-04-23T16:10:56.032 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112937 s, 928 MB/s 2024-04-23T16:10:56.032 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-04-23T16:10:56.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:56 smithi103 ceph-mon[39629]: Detected new or changed devices on smithi103 2024-04-23T16:10:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:56 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:56 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:56 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:10:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:56 smithi103 ceph-mon[39629]: Adjusting osd_memory_target on smithi103 to 17122M 2024-04-23T16:10:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:56 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:56 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:56 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:10:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:56 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:56.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:56 smithi148 ceph-mon[44520]: Detected new or changed devices on smithi103 2024-04-23T16:10:56.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:56 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:56.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:56 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:56.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:56 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:10:56.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:56 smithi148 ceph-mon[44520]: Adjusting osd_memory_target on smithi103 to 17122M 2024-04-23T16:10:56.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:56 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:56.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:56 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:56.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:56 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:10:56.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:56 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:56 smithi188 ceph-mon[44101]: Detected new or changed devices on smithi103 2024-04-23T16:10:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:56 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:56 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:56 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:10:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:56 smithi188 ceph-mon[44101]: Adjusting osd_memory_target on smithi103 to 17122M 2024-04-23T16:10:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:56 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:56 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:56 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:10:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:56 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:10:56.771 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_3 2024-04-23T16:10:57.036 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:10:57.552 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:57 smithi103 ceph-mon[39629]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:10:57.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:57 smithi148 ceph-mon[44520]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:10:57.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:57 smithi188 ceph-mon[44101]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:10:59.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:59 smithi188 ceph-mon[44101]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:10:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:59 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:10:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:59 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:10:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:10:59 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:59.762 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:59 smithi103 ceph-mon[39629]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:10:59.762 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:59 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:10:59.762 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:59 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:10:59.762 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:10:59 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:10:59.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:59 smithi148 ceph-mon[44520]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:10:59.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:59 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:10:59.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:59 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:10:59.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:10:59 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:00.509 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:00 smithi103 ceph-mon[39629]: from='client.14250 v1:172.21.15.103:0/714788926' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:11:00.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:00 smithi188 ceph-mon[44101]: from='client.14250 v1:172.21.15.103:0/714788926' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:11:00.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:00 smithi148 ceph-mon[44520]: from='client.14250 v1:172.21.15.103:0/714788926' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:11:01.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:01 smithi188 ceph-mon[44101]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:01.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:01 smithi103 ceph-mon[39629]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:01 smithi148 ceph-mon[44520]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:02.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:02 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1783383301' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3438f2bd-5fb7-432c-b383-29e18c442f20"}]: dispatch 2024-04-23T16:11:02.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:02 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1783383301' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3438f2bd-5fb7-432c-b383-29e18c442f20"}]': finished 2024-04-23T16:11:02.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:02 smithi188 ceph-mon[44101]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T16:11:02.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:11:02.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:02 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2204978976' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:11:02.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:02 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1783383301' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3438f2bd-5fb7-432c-b383-29e18c442f20"}]: dispatch 2024-04-23T16:11:02.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:02 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1783383301' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3438f2bd-5fb7-432c-b383-29e18c442f20"}]': finished 2024-04-23T16:11:02.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:02 smithi103 ceph-mon[39629]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T16:11:02.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:02 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:11:02.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:02 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2204978976' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:11:02.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:02 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1783383301' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3438f2bd-5fb7-432c-b383-29e18c442f20"}]: dispatch 2024-04-23T16:11:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:02 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1783383301' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3438f2bd-5fb7-432c-b383-29e18c442f20"}]': finished 2024-04-23T16:11:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:02 smithi148 ceph-mon[44520]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T16:11:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:02 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:11:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:02 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2204978976' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:11:03.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:03 smithi188 ceph-mon[44101]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:03 smithi103 ceph-mon[39629]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:03.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:03 smithi148 ceph-mon[44520]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:05.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:05 smithi188 ceph-mon[44101]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:05.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:05 smithi103 ceph-mon[39629]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:05.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:05 smithi148 ceph-mon[44520]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:07.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:07 smithi103 ceph-mon[39629]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:07.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:07 smithi188 ceph-mon[44101]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:07.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:07 smithi148 ceph-mon[44520]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:09.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:09 smithi103 ceph-mon[39629]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:09.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:09 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T16:11:09.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:09 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:09.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:09 smithi103 ceph-mon[39629]: Deploying daemon osd.1 on smithi103 2024-04-23T16:11:09.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:09 smithi188 ceph-mon[44101]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:09.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:09 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T16:11:09.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:09 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:09 smithi188 ceph-mon[44101]: Deploying daemon osd.1 on smithi103 2024-04-23T16:11:09.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:09 smithi148 ceph-mon[44520]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:09.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:09 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T16:11:09.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:09 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:09.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:09 smithi148 ceph-mon[44520]: Deploying daemon osd.1 on smithi103 2024-04-23T16:11:11.700 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:11 smithi103 ceph-mon[39629]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:11.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:11 smithi188 ceph-mon[44101]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:11.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:11 smithi148 ceph-mon[44520]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:13.716 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:13 smithi103 ceph-mon[39629]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:13.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:13 smithi188 ceph-mon[44101]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:13.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:13 smithi148 ceph-mon[44520]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:15.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:15.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:15.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:11:15.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:15 smithi103 ceph-mon[39629]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:15.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:15 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:15 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:15 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:11:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:15 smithi148 ceph-mon[44520]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:15.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:15.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:15.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:11:15.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:15 smithi188 ceph-mon[44101]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:11:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:16 smithi103 ceph-mon[39629]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:17.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:11:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:16 smithi148 ceph-mon[44520]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:11:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:16 smithi188 ceph-mon[44101]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:17.779 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 1 on host 'smithi103' 2024-04-23T16:11:18.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:17 smithi103 ceph-mon[39629]: from='osd.1 v1:172.21.15.103:6805/3716921156' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T16:11:18.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:11:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:17 smithi148 ceph-mon[44520]: from='osd.1 v1:172.21.15.103:6805/3716921156' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T16:11:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:11:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:18.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:17 smithi188 ceph-mon[44101]: from='osd.1 v1:172.21.15.103:6805/3716921156' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T16:11:18.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:11:18.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:18.541 DEBUG:teuthology.orchestra.run.smithi103:osd.1> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.1.service 2024-04-23T16:11:18.544 INFO:tasks.cephadm:Deploying osd.2 on smithi103 with /dev/vg_nvme/lv_2... 2024-04-23T16:11:18.544 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T16:11:18.574 INFO:journalctl@ceph.osd.1.smithi103.stdout:-- Logs begin at Tue 2024-04-23 15:57:24 UTC. -- 2024-04-23T16:11:19.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:19 smithi103 ceph-mon[39629]: from='osd.1 v1:172.21.15.103:6805/3716921156' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T16:11:19.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:19 smithi103 ceph-mon[39629]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T16:11:19.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:19 smithi103 ceph-mon[39629]: from='osd.1 v1:172.21.15.103:6805/3716921156' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:11:19.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:19 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:11:19.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:19 smithi103 ceph-mon[39629]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:19.372 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:11:19.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:19 smithi148 ceph-mon[44520]: from='osd.1 v1:172.21.15.103:6805/3716921156' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T16:11:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:19 smithi148 ceph-mon[44520]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T16:11:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:19 smithi148 ceph-mon[44520]: from='osd.1 v1:172.21.15.103:6805/3716921156' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:11:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:19 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:11:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:19 smithi148 ceph-mon[44520]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:19.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:19 smithi188 ceph-mon[44101]: from='osd.1 v1:172.21.15.103:6805/3716921156' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T16:11:19.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:19 smithi188 ceph-mon[44101]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T16:11:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:19 smithi188 ceph-mon[44101]: from='osd.1 v1:172.21.15.103:6805/3716921156' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:11:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:19 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:11:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:19 smithi188 ceph-mon[44101]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:19.754 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:11:19 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-1[65562]: 2024-04-23T16:11:19.463+0000 7fa721e5a700 -1 osd.1 0 waiting for initial osdmap 2024-04-23T16:11:19.754 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:11:19 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-1[65562]: 2024-04-23T16:11:19.466+0000 7fa719c45700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:11:20.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:20 smithi103 ceph-mon[39629]: from='osd.1 v1:172.21.15.103:6805/3716921156' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-23T16:11:20.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:20 smithi103 ceph-mon[39629]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T16:11:20.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:20 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:11:20.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:20 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:11:20.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:20 smithi103 ceph-mon[39629]: from='osd.1 v1:172.21.15.103:6805/3716921156' entity='osd.1' 2024-04-23T16:11:20.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:20 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:11:20.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:20 smithi148 ceph-mon[44520]: from='osd.1 v1:172.21.15.103:6805/3716921156' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-23T16:11:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:20 smithi148 ceph-mon[44520]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T16:11:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:20 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:11:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:20 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:11:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:20 smithi148 ceph-mon[44520]: from='osd.1 v1:172.21.15.103:6805/3716921156' entity='osd.1' 2024-04-23T16:11:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:20 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:11:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:20 smithi188 ceph-mon[44101]: from='osd.1 v1:172.21.15.103:6805/3716921156' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-23T16:11:20.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:20 smithi188 ceph-mon[44101]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T16:11:20.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:20 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:11:20.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:20 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:11:20.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:20 smithi188 ceph-mon[44101]: from='osd.1 v1:172.21.15.103:6805/3716921156' entity='osd.1' 2024-04-23T16:11:20.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:20 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:11:21.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:21 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:11:21.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:21 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:11:21.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:11:21.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:11:21.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:21 smithi103 ceph-mon[39629]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:21.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:21 smithi103 ceph-mon[39629]: osd.1 v1:172.21.15.103:6805/3716921156 boot 2024-04-23T16:11:21.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:21 smithi103 ceph-mon[39629]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T16:11:21.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:21 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:11:21.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:21 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:11:21.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:21 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:11:21.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:11:21.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:11:21.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:21 smithi148 ceph-mon[44520]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:21.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:21 smithi148 ceph-mon[44520]: osd.1 v1:172.21.15.103:6805/3716921156 boot 2024-04-23T16:11:21.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:21 smithi148 ceph-mon[44520]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T16:11:21.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:21 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:11:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:21 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:11:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:21 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:11:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:11:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:11:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:21 smithi188 ceph-mon[44101]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-23T16:11:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:21 smithi188 ceph-mon[44101]: osd.1 v1:172.21.15.103:6805/3716921156 boot 2024-04-23T16:11:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:21 smithi188 ceph-mon[44101]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T16:11:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:21 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:11:23.288 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T16:11:23.288 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T16:11:23.288 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-04-23T16:11:23.288 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-04-23T16:11:23.289 INFO:teuthology.orchestra.run.smithi103.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125653 s, 835 MB/s 2024-04-23T16:11:23.289 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-04-23T16:11:23.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:23 smithi103 ceph-mon[39629]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T16:11:23.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:23 smithi103 ceph-mon[39629]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:23.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:23 smithi103 ceph-mon[39629]: Detected new or changed devices on smithi103 2024-04-23T16:11:23.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:23.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:23.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:11:23.326 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:11:23.326 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:23 smithi103 ceph-mon[39629]: Adjusting osd_memory_target on smithi103 to 8561M 2024-04-23T16:11:23.326 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:23.326 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:23.326 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:11:23.326 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:23.326 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:11:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:23 smithi148 ceph-mon[44520]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T16:11:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:23 smithi148 ceph-mon[44520]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:23 smithi148 ceph-mon[44520]: Detected new or changed devices on smithi103 2024-04-23T16:11:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:11:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:11:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:23 smithi148 ceph-mon[44520]: Adjusting osd_memory_target on smithi103 to 8561M 2024-04-23T16:11:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:23.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:23.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:11:23.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:23.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:11:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:23 smithi188 ceph-mon[44101]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T16:11:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:23 smithi188 ceph-mon[44101]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:23 smithi188 ceph-mon[44101]: Detected new or changed devices on smithi103 2024-04-23T16:11:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:11:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:11:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:23 smithi188 ceph-mon[44101]: Adjusting osd_memory_target on smithi103 to 8561M 2024-04-23T16:11:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:11:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:11:24.073 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_2 2024-04-23T16:11:24.339 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:11:24.703 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:24 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-23T16:11:24.703 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:24 smithi103 ceph-mon[39629]: osdmap e15: 2 total, 2 up, 2 in 2024-04-23T16:11:24.703 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:24 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:11:24.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:24 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-23T16:11:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:24 smithi188 ceph-mon[44101]: osdmap e15: 2 total, 2 up, 2 in 2024-04-23T16:11:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:24 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:11:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:24 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-23T16:11:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:24 smithi148 ceph-mon[44520]: osdmap e15: 2 total, 2 up, 2 in 2024-04-23T16:11:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:24 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:11:25.100 INFO:journalctl@ceph.osd.0.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70399]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T16:11:25.100 INFO:journalctl@ceph.osd.0.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70399]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T16:11:25.100 INFO:journalctl@ceph.osd.0.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70399]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T16:11:25.100 INFO:journalctl@ceph.osd.0.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70399]: pam_unix(sudo:session): session closed for user root 2024-04-23T16:11:25.100 INFO:journalctl@ceph.osd.0.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70402]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T16:11:25.101 INFO:journalctl@ceph.osd.0.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70402]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T16:11:25.101 INFO:journalctl@ceph.osd.0.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70402]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T16:11:25.101 INFO:journalctl@ceph.osd.0.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70402]: pam_unix(sudo:session): session closed for user root 2024-04-23T16:11:25.101 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70405]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T16:11:25.101 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70405]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T16:11:25.102 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70405]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T16:11:25.102 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70405]: pam_unix(sudo:session): session closed for user root 2024-04-23T16:11:25.102 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70414]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T16:11:25.102 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70414]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T16:11:25.102 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70414]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T16:11:25.102 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70414]: pam_unix(sudo:session): session closed for user root 2024-04-23T16:11:25.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70418]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T16:11:25.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70418]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T16:11:25.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70418]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T16:11:25.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70418]: pam_unix(sudo:session): session closed for user root 2024-04-23T16:11:25.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70422]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T16:11:25.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70422]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T16:11:25.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70422]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T16:11:25.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:24 smithi103 sudo[70422]: pam_unix(sudo:session): session closed for user root 2024-04-23T16:11:25.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:24 smithi188 sudo[46447]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T16:11:25.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:24 smithi188 sudo[46447]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T16:11:25.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:24 smithi188 sudo[46447]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T16:11:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:24 smithi188 sudo[46447]: pam_unix(sudo:session): session closed for user root 2024-04-23T16:11:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:24 smithi188 sudo[46450]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T16:11:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:24 smithi188 sudo[46450]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T16:11:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:24 smithi188 sudo[46450]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T16:11:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:24 smithi188 sudo[46450]: pam_unix(sudo:session): session closed for user root 2024-04-23T16:11:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:24 smithi148 sudo[46867]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T16:11:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:24 smithi148 sudo[46867]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T16:11:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:24 smithi148 sudo[46867]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T16:11:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:24 smithi148 sudo[46867]: pam_unix(sudo:session): session closed for user root 2024-04-23T16:11:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:24 smithi148 sudo[46870]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T16:11:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:24 smithi148 sudo[46870]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T16:11:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:24 smithi148 sudo[46870]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T16:11:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:24 smithi148 sudo[46870]: pam_unix(sudo:session): session closed for user root 2024-04-23T16:11:25.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:25.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:11:25.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:11:25.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: osdmap e16: 2 total, 2 up, 2 in 2024-04-23T16:11:25.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:25.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:11:25.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:11:25.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T16:11:25.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T16:11:25.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:11:25.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:11:25.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:11:25.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T16:11:25.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:11:25.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T16:11:25.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:25.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:11:25.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:11:25.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T16:11:25.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:11:25.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T16:11:25.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:11:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:11:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: osdmap e16: 2 total, 2 up, 2 in 2024-04-23T16:11:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:11:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:11:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T16:11:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T16:11:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:11:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:11:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:11:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T16:11:25.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:11:25.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T16:11:25.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:25.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:11:25.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:11:25.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T16:11:25.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:11:25.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T16:11:25.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:11:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:11:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: osdmap e16: 2 total, 2 up, 2 in 2024-04-23T16:11:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:11:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:11:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T16:11:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T16:11:25.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:11:25.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:11:25.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:11:25.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T16:11:25.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:11:25.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T16:11:25.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:25.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:11:25.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:11:25.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T16:11:25.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:11:25.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T16:11:25.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:26.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:26 smithi188 ceph-mon[44101]: osdmap e17: 2 total, 2 up, 2 in 2024-04-23T16:11:26.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:26 smithi188 ceph-mon[44101]: mgrmap e16: y(active, since 2m), standbys: x 2024-04-23T16:11:26.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:26 smithi103 ceph-mon[39629]: osdmap e17: 2 total, 2 up, 2 in 2024-04-23T16:11:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:26 smithi103 ceph-mon[39629]: mgrmap e16: y(active, since 2m), standbys: x 2024-04-23T16:11:26.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:26 smithi148 ceph-mon[44520]: osdmap e17: 2 total, 2 up, 2 in 2024-04-23T16:11:26.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:26 smithi148 ceph-mon[44520]: mgrmap e16: y(active, since 2m), standbys: x 2024-04-23T16:11:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:27 smithi103 ceph-mon[39629]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:27 smithi103 ceph-mon[39629]: from='client.14286 v1:172.21.15.103:0/689334279' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:11:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:11:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:11:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:27 smithi103 ceph-mon[39629]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:11:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:27 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:11:27.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:27 smithi188 ceph-mon[44101]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:27.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:27 smithi188 ceph-mon[44101]: from='client.14286 v1:172.21.15.103:0/689334279' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:11:27.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:27 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:11:27.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:27 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:11:27.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:27 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:27.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:27 smithi188 ceph-mon[44101]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:11:27.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:27 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:11:27.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:27 smithi148 ceph-mon[44520]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:27.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:27 smithi148 ceph-mon[44520]: from='client.14286 v1:172.21.15.103:0/689334279' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:11:27.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:27 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:11:27.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:27 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:11:27.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:27 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:27.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:27 smithi148 ceph-mon[44520]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:11:27.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:27 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:11:29.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:29 smithi188 ceph-mon[44101]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:29.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:29 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3177236866' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c17fffb4-8d17-4546-ba24-e5951a2a37fe"}]: dispatch 2024-04-23T16:11:29.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:29 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3177236866' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c17fffb4-8d17-4546-ba24-e5951a2a37fe"}]': finished 2024-04-23T16:11:29.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:29 smithi188 ceph-mon[44101]: osdmap e18: 3 total, 2 up, 3 in 2024-04-23T16:11:29.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:11:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:29 smithi103 ceph-mon[39629]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:29 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3177236866' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c17fffb4-8d17-4546-ba24-e5951a2a37fe"}]: dispatch 2024-04-23T16:11:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:29 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3177236866' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c17fffb4-8d17-4546-ba24-e5951a2a37fe"}]': finished 2024-04-23T16:11:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:29 smithi103 ceph-mon[39629]: osdmap e18: 3 total, 2 up, 3 in 2024-04-23T16:11:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:11:29.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:29 smithi148 ceph-mon[44520]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:29 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3177236866' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c17fffb4-8d17-4546-ba24-e5951a2a37fe"}]: dispatch 2024-04-23T16:11:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:29 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3177236866' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c17fffb4-8d17-4546-ba24-e5951a2a37fe"}]': finished 2024-04-23T16:11:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:29 smithi148 ceph-mon[44520]: osdmap e18: 3 total, 2 up, 3 in 2024-04-23T16:11:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:11:30.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:30 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1804622293' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:11:30.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:30 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1804622293' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:11:30.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:30 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1804622293' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:11:31.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:31 smithi188 ceph-mon[44101]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:31.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:31 smithi103 ceph-mon[39629]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:31.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:31 smithi148 ceph-mon[44520]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:33.651 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:33 smithi103 ceph-mon[39629]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:33.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:33 smithi188 ceph-mon[44101]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:33.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:33 smithi148 ceph-mon[44520]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:35.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:35 smithi188 ceph-mon[44101]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:35.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:35 smithi103 ceph-mon[39629]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:35.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:35 smithi148 ceph-mon[44520]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:36.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:36 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T16:11:36.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:36 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:36.742 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:36 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T16:11:36.742 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:36 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:36.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:36 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T16:11:36.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:36 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:37.576 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:37 smithi103 ceph-mon[39629]: Deploying daemon osd.2 on smithi103 2024-04-23T16:11:37.576 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:37 smithi103 ceph-mon[39629]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:37.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:37 smithi188 ceph-mon[44101]: Deploying daemon osd.2 on smithi103 2024-04-23T16:11:37.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:37 smithi188 ceph-mon[44101]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:37.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:37 smithi148 ceph-mon[44520]: Deploying daemon osd.2 on smithi103 2024-04-23T16:11:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:37 smithi148 ceph-mon[44520]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:39.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:39 smithi103 ceph-mon[39629]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:39.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:39 smithi188 ceph-mon[44101]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:39.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:39 smithi148 ceph-mon[44520]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:41.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:41 smithi188 ceph-mon[44101]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:41.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:41 smithi103 ceph-mon[39629]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:41.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:41 smithi148 ceph-mon[44520]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:42.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:42 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:42.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:42 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:42.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:42 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:11:42.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:42 smithi103 ceph-mon[39629]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:42.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:42 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:42.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:42 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:42.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:42 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:11:42.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:42 smithi188 ceph-mon[44101]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:43.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:42 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:43.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:42 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:43.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:42 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:11:43.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:42 smithi148 ceph-mon[44520]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:45 smithi103 ceph-mon[39629]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:11:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:45 smithi103 ceph-mon[39629]: from='osd.2 v1:172.21.15.103:6809/424240088' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T16:11:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:45 smithi148 ceph-mon[44520]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:11:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:45 smithi148 ceph-mon[44520]: from='osd.2 v1:172.21.15.103:6809/424240088' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T16:11:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:45 smithi188 ceph-mon[44101]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:11:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:45 smithi188 ceph-mon[44101]: from='osd.2 v1:172.21.15.103:6809/424240088' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T16:11:45.705 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 2 on host 'smithi103' 2024-04-23T16:11:46.402 DEBUG:teuthology.orchestra.run.smithi103:osd.2> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.2.service 2024-04-23T16:11:46.404 INFO:tasks.cephadm:Deploying osd.3 on smithi103 with /dev/vg_nvme/lv_1... 2024-04-23T16:11:46.404 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T16:11:46.434 INFO:journalctl@ceph.osd.2.smithi103.stdout:-- Logs begin at Tue 2024-04-23 15:57:24 UTC. -- 2024-04-23T16:11:46.613 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:46 smithi103 ceph-mon[39629]: from='osd.2 v1:172.21.15.103:6809/424240088' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T16:11:46.613 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:46 smithi103 ceph-mon[39629]: osdmap e19: 3 total, 2 up, 3 in 2024-04-23T16:11:46.613 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:46 smithi103 ceph-mon[39629]: from='osd.2 v1:172.21.15.103:6809/424240088' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:11:46.613 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:46 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:11:46.613 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:46 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:46.613 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:46 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:46.614 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:46 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:11:46.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:46 smithi148 ceph-mon[44520]: from='osd.2 v1:172.21.15.103:6809/424240088' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T16:11:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:46 smithi148 ceph-mon[44520]: osdmap e19: 3 total, 2 up, 3 in 2024-04-23T16:11:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:46 smithi148 ceph-mon[44520]: from='osd.2 v1:172.21.15.103:6809/424240088' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:11:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:46 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:11:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:46 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:46 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:46 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:11:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:46 smithi148 ceph-mon[44520]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:46.923 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:46 smithi103 ceph-mon[39629]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:46.923 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:11:46 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[72723]: 2024-04-23T16:11:46.611+0000 7f2db5324700 -1 osd.2 0 waiting for initial osdmap 2024-04-23T16:11:46.923 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:11:46 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[72723]: 2024-04-23T16:11:46.618+0000 7f2db1117700 -1 osd.2 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:11:46.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:46 smithi188 ceph-mon[44101]: from='osd.2 v1:172.21.15.103:6809/424240088' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T16:11:46.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:46 smithi188 ceph-mon[44101]: osdmap e19: 3 total, 2 up, 3 in 2024-04-23T16:11:46.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:46 smithi188 ceph-mon[44101]: from='osd.2 v1:172.21.15.103:6809/424240088' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:11:46.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:46 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:11:46.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:46 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:46.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:46 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:46.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:46 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:11:46.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:46 smithi188 ceph-mon[44101]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:47.240 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:11:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:47 smithi103 ceph-mon[39629]: from='osd.2 v1:172.21.15.103:6809/424240088' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-23T16:11:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:47 smithi103 ceph-mon[39629]: osdmap e20: 3 total, 2 up, 3 in 2024-04-23T16:11:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:47 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:11:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:47 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:11:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:47 smithi188 ceph-mon[44101]: from='osd.2 v1:172.21.15.103:6809/424240088' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-23T16:11:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:47 smithi188 ceph-mon[44101]: osdmap e20: 3 total, 2 up, 3 in 2024-04-23T16:11:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:47 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:11:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:47 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:11:48.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:47 smithi148 ceph-mon[44520]: from='osd.2 v1:172.21.15.103:6809/424240088' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-23T16:11:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:47 smithi148 ceph-mon[44520]: osdmap e20: 3 total, 2 up, 3 in 2024-04-23T16:11:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:47 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:11:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:47 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:11:48.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:48 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:11:48.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:48 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:11:48.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:11:48.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:48 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:11:48.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:48 smithi103 ceph-mon[39629]: osd.2 v1:172.21.15.103:6809/424240088 boot 2024-04-23T16:11:48.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:48 smithi103 ceph-mon[39629]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T16:11:48.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:48 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:11:48.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:48 smithi103 ceph-mon[39629]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:48.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:48 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:11:48.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:48 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:11:48.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:11:48.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:48 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:11:48.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:48 smithi188 ceph-mon[44101]: osd.2 v1:172.21.15.103:6809/424240088 boot 2024-04-23T16:11:48.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:48 smithi188 ceph-mon[44101]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T16:11:48.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:48 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:11:48.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:48 smithi188 ceph-mon[44101]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:49.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:48 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:11:49.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:48 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:11:49.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:11:49.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:48 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:11:49.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:48 smithi148 ceph-mon[44520]: osd.2 v1:172.21.15.103:6809/424240088 boot 2024-04-23T16:11:49.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:48 smithi148 ceph-mon[44520]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T16:11:49.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:48 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:11:49.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:48 smithi148 ceph-mon[44520]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-23T16:11:49.821 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T16:11:49.821 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T16:11:49.821 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-04-23T16:11:49.821 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-04-23T16:11:49.821 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.013578 s, 772 MB/s 2024-04-23T16:11:49.821 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-04-23T16:11:49.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:49 smithi188 ceph-mon[44101]: osdmap e22: 3 total, 3 up, 3 in 2024-04-23T16:11:50.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:49 smithi103 ceph-mon[39629]: osdmap e22: 3 total, 3 up, 3 in 2024-04-23T16:11:50.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:49 smithi148 ceph-mon[44520]: osdmap e22: 3 total, 3 up, 3 in 2024-04-23T16:11:50.680 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_1 2024-04-23T16:11:50.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:50 smithi188 ceph-mon[44101]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:11:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:50 smithi103 ceph-mon[39629]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:11:51.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:50 smithi148 ceph-mon[44520]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:11:51.486 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:11:52.802 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:52 smithi103 ceph-mon[39629]: Detected new or changed devices on smithi103 2024-04-23T16:11:52.803 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:52.803 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:52.803 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:11:52.803 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:11:52.803 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:11:52.803 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:52 smithi103 ceph-mon[39629]: Adjusting osd_memory_target on smithi103 to 5707M 2024-04-23T16:11:52.803 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:52.803 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:52.804 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:11:52.804 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:52 smithi148 ceph-mon[44520]: Detected new or changed devices on smithi103 2024-04-23T16:11:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:11:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:11:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:11:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:52 smithi148 ceph-mon[44520]: Adjusting osd_memory_target on smithi103 to 5707M 2024-04-23T16:11:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:11:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:52.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:52 smithi188 ceph-mon[44101]: Detected new or changed devices on smithi103 2024-04-23T16:11:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:11:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:11:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:11:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:52 smithi188 ceph-mon[44101]: Adjusting osd_memory_target on smithi103 to 5707M 2024-04-23T16:11:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:11:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:11:53.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:53 smithi103 ceph-mon[39629]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:11:53.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:53 smithi148 ceph-mon[44520]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:11:53.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:53 smithi188 ceph-mon[44101]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:11:54.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:54 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:11:54.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:54 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:11:54.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:54 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:54.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:54 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:11:54.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:54 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:11:54.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:54 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:54.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:54 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:11:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:54 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:11:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:54 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:11:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:55 smithi103 ceph-mon[39629]: from='client.14313 v1:172.21.15.103:0/2472014850' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:11:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:55 smithi103 ceph-mon[39629]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:11:55.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:55 smithi148 ceph-mon[44520]: from='client.14313 v1:172.21.15.103:0/2472014850' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:11:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:55 smithi148 ceph-mon[44520]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:11:55.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:55 smithi188 ceph-mon[44101]: from='client.14313 v1:172.21.15.103:0/2472014850' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:11:55.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:55 smithi188 ceph-mon[44101]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:11:56.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:56 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2367634752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cee91a3d-55dc-4e89-b568-86821632bb1c"}]: dispatch 2024-04-23T16:11:56.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:56 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2367634752' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cee91a3d-55dc-4e89-b568-86821632bb1c"}]': finished 2024-04-23T16:11:56.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:56 smithi103 ceph-mon[39629]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T16:11:56.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:56 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:11:56.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:56 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2367634752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cee91a3d-55dc-4e89-b568-86821632bb1c"}]: dispatch 2024-04-23T16:11:56.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:56 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2367634752' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cee91a3d-55dc-4e89-b568-86821632bb1c"}]': finished 2024-04-23T16:11:56.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:56 smithi148 ceph-mon[44520]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T16:11:56.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:56 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:11:56.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:56 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2367634752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cee91a3d-55dc-4e89-b568-86821632bb1c"}]: dispatch 2024-04-23T16:11:56.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:56 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2367634752' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cee91a3d-55dc-4e89-b568-86821632bb1c"}]': finished 2024-04-23T16:11:56.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:56 smithi188 ceph-mon[44101]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T16:11:56.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:56 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:11:57.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:57 smithi103 ceph-mon[39629]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:11:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:57 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2148543765' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:11:57.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:57 smithi148 ceph-mon[44520]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:11:57.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:57 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2148543765' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:11:57.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:57 smithi188 ceph-mon[44101]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:11:57.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:57 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2148543765' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:11:59.755 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:11:59 smithi103 ceph-mon[39629]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:11:59.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:11:59 smithi148 ceph-mon[44520]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:11:59.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:11:59 smithi188 ceph-mon[44101]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:01.569 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:01 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:12:01.569 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:01 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:12:01.569 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:01 smithi103 ceph-mon[39629]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:01.570 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:01 smithi103 ceph-mon[39629]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T16:12:01.570 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:01 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:12:01.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:01 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:12:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:01 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:12:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:01 smithi148 ceph-mon[44520]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:01 smithi148 ceph-mon[44520]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T16:12:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:01 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:12:01.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:01 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:12:01.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:01 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:12:01.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:01 smithi188 ceph-mon[44101]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:01.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:01 smithi188 ceph-mon[44101]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T16:12:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:01 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:12:03.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:03 smithi103 ceph-mon[39629]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:03.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:03 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T16:12:03.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:03 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:03 smithi148 ceph-mon[44520]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:03 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T16:12:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:03 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:03.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:03 smithi188 ceph-mon[44101]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:03 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T16:12:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:03 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:04.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:04 smithi103 ceph-mon[39629]: Deploying daemon osd.3 on smithi103 2024-04-23T16:12:04.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:04 smithi148 ceph-mon[44520]: Deploying daemon osd.3 on smithi103 2024-04-23T16:12:04.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:04 smithi188 ceph-mon[44101]: Deploying daemon osd.3 on smithi103 2024-04-23T16:12:05.758 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:05 smithi103 ceph-mon[39629]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:05.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:05 smithi148 ceph-mon[44520]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:05.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:05 smithi188 ceph-mon[44101]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:07.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:07 smithi103 ceph-mon[39629]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:07.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:07 smithi148 ceph-mon[44520]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:07.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:07 smithi188 ceph-mon[44101]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:09.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:09 smithi103 ceph-mon[39629]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:09.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:09 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:09.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:09 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:12:09.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:09 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:09.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:09 smithi148 ceph-mon[44520]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:09.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:09 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:09.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:09 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:12:09.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:09 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:09.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:09 smithi188 ceph-mon[44101]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:09 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:09 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:12:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:09 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:10.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:10 smithi103 ceph-mon[39629]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:10.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:10 smithi148 ceph-mon[44520]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:10.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:10 smithi188 ceph-mon[44101]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:12.276 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 3 on host 'smithi103' 2024-04-23T16:12:12.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:12.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:12.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:12.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:12:12.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:12.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:12 smithi103 ceph-mon[39629]: from='osd.3 v1:172.21.15.103:6813/1895183074' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T16:12:12.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:12.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:12.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:12:12.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:12.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:12.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:12.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:12:12.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:12.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:12 smithi188 ceph-mon[44101]: from='osd.3 v1:172.21.15.103:6813/1895183074' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T16:12:12.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:12.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:12.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:12:12.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:12.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:12.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:12.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:12:12.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:12.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:12 smithi148 ceph-mon[44520]: from='osd.3 v1:172.21.15.103:6813/1895183074' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T16:12:12.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:12.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:12.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:12:13.081 DEBUG:teuthology.orchestra.run.smithi103:osd.3> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.3.service 2024-04-23T16:12:13.083 INFO:tasks.cephadm:Deploying osd.4 on smithi148 with /dev/vg_nvme/lv_4... 2024-04-23T16:12:13.084 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T16:12:13.115 INFO:journalctl@ceph.osd.3.smithi103.stdout:-- Logs begin at Tue 2024-04-23 15:57:24 UTC. -- 2024-04-23T16:12:13.355 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:12:13.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:13 smithi103 ceph-mon[39629]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:13.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:13 smithi103 ceph-mon[39629]: from='osd.3 v1:172.21.15.103:6813/1895183074' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T16:12:13.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:13 smithi103 ceph-mon[39629]: osdmap e25: 4 total, 3 up, 4 in 2024-04-23T16:12:13.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:13 smithi103 ceph-mon[39629]: from='osd.3 v1:172.21.15.103:6813/1895183074' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:12:13.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:12:13.600 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:12:13 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-3[80767]: 2024-04-23T16:12:13.466+0000 7f23a862b700 -1 osd.3 0 waiting for initial osdmap 2024-04-23T16:12:13.600 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:12:13 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-3[80767]: 2024-04-23T16:12:13.470+0000 7f23a441e700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:12:13.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:13 smithi188 ceph-mon[44101]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:13.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:13 smithi188 ceph-mon[44101]: from='osd.3 v1:172.21.15.103:6813/1895183074' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T16:12:13.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:13 smithi188 ceph-mon[44101]: osdmap e25: 4 total, 3 up, 4 in 2024-04-23T16:12:13.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:13 smithi188 ceph-mon[44101]: from='osd.3 v1:172.21.15.103:6813/1895183074' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:12:13.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:12:13.808 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:13 smithi148 ceph-mon[44520]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:13.808 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:13 smithi148 ceph-mon[44520]: from='osd.3 v1:172.21.15.103:6813/1895183074' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T16:12:13.808 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:13 smithi148 ceph-mon[44520]: osdmap e25: 4 total, 3 up, 4 in 2024-04-23T16:12:13.808 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:13 smithi148 ceph-mon[44520]: from='osd.3 v1:172.21.15.103:6813/1895183074' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:12:13.808 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:12:14.523 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:14 smithi148 ceph-mon[44520]: from='osd.3 v1:172.21.15.103:6813/1895183074' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-23T16:12:14.523 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:14 smithi148 ceph-mon[44520]: osdmap e26: 4 total, 3 up, 4 in 2024-04-23T16:12:14.524 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:12:14.524 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:12:14.524 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:12:14.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:14 smithi103 ceph-mon[39629]: from='osd.3 v1:172.21.15.103:6813/1895183074' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-23T16:12:14.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:14 smithi103 ceph-mon[39629]: osdmap e26: 4 total, 3 up, 4 in 2024-04-23T16:12:14.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:12:14.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:12:14.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:12:14.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:14 smithi188 ceph-mon[44101]: from='osd.3 v1:172.21.15.103:6813/1895183074' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-23T16:12:14.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:14 smithi188 ceph-mon[44101]: osdmap e26: 4 total, 3 up, 4 in 2024-04-23T16:12:14.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:12:14.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:12:14.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:12:15.355 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T16:12:15.355 INFO:teuthology.orchestra.run.smithi148.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T16:12:15.355 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10+0 records in 2024-04-23T16:12:15.355 INFO:teuthology.orchestra.run.smithi148.stderr:10+0 records out 2024-04-23T16:12:15.355 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111785 s, 938 MB/s 2024-04-23T16:12:15.355 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping successful for: 2024-04-23T16:12:15.716 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:15 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:12:15.716 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:15 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:12:15.716 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:12:15.716 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:15 smithi148 ceph-mon[44520]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:15.716 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:15 smithi148 ceph-mon[44520]: osd.3 v1:172.21.15.103:6813/1895183074 boot 2024-04-23T16:12:15.716 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:15 smithi148 ceph-mon[44520]: osdmap e27: 4 total, 4 up, 4 in 2024-04-23T16:12:15.716 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:15 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:12:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:15 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:12:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:15 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:12:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:12:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:15 smithi103 ceph-mon[39629]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:15 smithi103 ceph-mon[39629]: osd.3 v1:172.21.15.103:6813/1895183074 boot 2024-04-23T16:12:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:15 smithi103 ceph-mon[39629]: osdmap e27: 4 total, 4 up, 4 in 2024-04-23T16:12:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:12:15.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:15 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:12:15.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:15 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:12:15.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:12:15.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:15 smithi188 ceph-mon[44101]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-23T16:12:15.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:15 smithi188 ceph-mon[44101]: osd.3 v1:172.21.15.103:6813/1895183074 boot 2024-04-23T16:12:15.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:15 smithi188 ceph-mon[44101]: osdmap e27: 4 total, 4 up, 4 in 2024-04-23T16:12:15.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:12:16.067 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi148:vg_nvme/lv_4 2024-04-23T16:12:16.327 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:12:16.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:16 smithi103 ceph-mon[39629]: osdmap e28: 4 total, 4 up, 4 in 2024-04-23T16:12:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:12:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:12:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:12:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:12:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:12:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:16.783 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:16 smithi148 ceph-mon[44520]: osdmap e28: 4 total, 4 up, 4 in 2024-04-23T16:12:16.784 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:16.784 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:16.784 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:12:16.784 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:12:16.784 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:12:16.784 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:12:16.784 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:16.784 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:16.784 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:12:16.784 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:16.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:16 smithi188 ceph-mon[44101]: osdmap e28: 4 total, 4 up, 4 in 2024-04-23T16:12:16.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:16.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:16.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:12:16.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:12:16.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:12:16.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:12:16.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:16.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:16.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:12:16.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:17.684 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:17 smithi148 ceph-mon[44520]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:17.684 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:17 smithi148 ceph-mon[44520]: Detected new or changed devices on smithi103 2024-04-23T16:12:17.684 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:17 smithi148 ceph-mon[44520]: Adjusting osd_memory_target on smithi103 to 4280M 2024-04-23T16:12:17.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:17 smithi103 ceph-mon[39629]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:17.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:17 smithi103 ceph-mon[39629]: Detected new or changed devices on smithi103 2024-04-23T16:12:17.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:17 smithi103 ceph-mon[39629]: Adjusting osd_memory_target on smithi103 to 4280M 2024-04-23T16:12:17.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:17 smithi188 ceph-mon[44101]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:17 smithi188 ceph-mon[44101]: Detected new or changed devices on smithi103 2024-04-23T16:12:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:17 smithi188 ceph-mon[44101]: Adjusting osd_memory_target on smithi103 to 4280M 2024-04-23T16:12:18.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:12:18.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:12:18.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:12:18.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:12:18.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:18.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:12:18.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:12:18.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:19.537 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:19 smithi148 ceph-mon[44520]: from='client.34226 v1:172.21.15.148:0/2433641682' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:12:19.537 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:19 smithi148 ceph-mon[44520]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:19.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:19 smithi103 ceph-mon[39629]: from='client.34226 v1:172.21.15.148:0/2433641682' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:12:19.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:19 smithi103 ceph-mon[39629]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:19.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:19 smithi188 ceph-mon[44101]: from='client.34226 v1:172.21.15.148:0/2433641682' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:12:19.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:19 smithi188 ceph-mon[44101]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:21.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:21 smithi103 ceph-mon[39629]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:21.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:21 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.148:0/3043897244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61d5385b-2c82-42f8-93c3-8f1baf71c65e"}]: dispatch 2024-04-23T16:12:21.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:21 smithi103 ceph-mon[39629]: from='client.34232 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61d5385b-2c82-42f8-93c3-8f1baf71c65e"}]: dispatch 2024-04-23T16:12:21.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:21 smithi103 ceph-mon[39629]: from='client.34232 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61d5385b-2c82-42f8-93c3-8f1baf71c65e"}]': finished 2024-04-23T16:12:21.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:21 smithi103 ceph-mon[39629]: osdmap e29: 5 total, 4 up, 5 in 2024-04-23T16:12:21.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:21 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:12:21.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:21 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.148:0/4067678090' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:12:21.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:21 smithi148 ceph-mon[44520]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:21.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:21 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.148:0/3043897244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61d5385b-2c82-42f8-93c3-8f1baf71c65e"}]: dispatch 2024-04-23T16:12:21.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:21 smithi148 ceph-mon[44520]: from='client.34232 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61d5385b-2c82-42f8-93c3-8f1baf71c65e"}]: dispatch 2024-04-23T16:12:21.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:21 smithi148 ceph-mon[44520]: from='client.34232 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61d5385b-2c82-42f8-93c3-8f1baf71c65e"}]': finished 2024-04-23T16:12:21.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:21 smithi148 ceph-mon[44520]: osdmap e29: 5 total, 4 up, 5 in 2024-04-23T16:12:21.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:21 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:12:21.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:21 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.148:0/4067678090' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:12:21.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:21 smithi188 ceph-mon[44101]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:21.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:21 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.148:0/3043897244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61d5385b-2c82-42f8-93c3-8f1baf71c65e"}]: dispatch 2024-04-23T16:12:21.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:21 smithi188 ceph-mon[44101]: from='client.34232 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61d5385b-2c82-42f8-93c3-8f1baf71c65e"}]: dispatch 2024-04-23T16:12:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:21 smithi188 ceph-mon[44101]: from='client.34232 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61d5385b-2c82-42f8-93c3-8f1baf71c65e"}]': finished 2024-04-23T16:12:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:21 smithi188 ceph-mon[44101]: osdmap e29: 5 total, 4 up, 5 in 2024-04-23T16:12:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:21 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:12:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:21 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.148:0/4067678090' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:12:23.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:23 smithi103 ceph-mon[39629]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:23.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:23 smithi148 ceph-mon[44520]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:23.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:23 smithi188 ceph-mon[44101]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:25.596 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:25 smithi148 ceph-mon[44520]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:25.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:25 smithi103 ceph-mon[39629]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:25.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:25 smithi188 ceph-mon[44101]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:27.799 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:27 smithi148 ceph-mon[44520]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:27.800 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:27 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:12:27.800 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:27 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:12:27.800 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:27 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T16:12:27.800 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:27 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:27.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:27 smithi103 ceph-mon[39629]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:27.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:27 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:12:27.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:27 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:12:27.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T16:12:27.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:27.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:27 smithi188 ceph-mon[44101]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:27.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:27 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:12:27.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:27 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:12:27.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:27 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T16:12:27.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:27 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:28.826 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:28 smithi148 ceph-mon[44520]: Deploying daemon osd.4 on smithi148 2024-04-23T16:12:28.826 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:28 smithi148 ceph-mon[44520]: osdmap e30: 5 total, 4 up, 5 in 2024-04-23T16:12:28.826 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:28 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:12:28.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:28 smithi103 ceph-mon[39629]: Deploying daemon osd.4 on smithi148 2024-04-23T16:12:28.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:28 smithi103 ceph-mon[39629]: osdmap e30: 5 total, 4 up, 5 in 2024-04-23T16:12:28.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:28 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:12:28.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:28 smithi188 ceph-mon[44101]: Deploying daemon osd.4 on smithi148 2024-04-23T16:12:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:28 smithi188 ceph-mon[44101]: osdmap e30: 5 total, 4 up, 5 in 2024-04-23T16:12:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:28 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:12:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:29 smithi103 ceph-mon[39629]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:29.852 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:29 smithi148 ceph-mon[44520]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:29.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:29 smithi188 ceph-mon[44101]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:31.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:31 smithi148 ceph-mon[44520]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:31.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:31 smithi103 ceph-mon[39629]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:31.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:31 smithi188 ceph-mon[44101]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:33 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:33 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:33 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:12:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:33 smithi148 ceph-mon[44520]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:33.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:33 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:33.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:33 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:33.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:33 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:12:33.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:33 smithi188 ceph-mon[44101]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:33.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:33 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:33.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:33 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:33.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:33 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:12:33.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:33 smithi103 ceph-mon[39629]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:34.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:34 smithi188 ceph-mon[44101]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:34 smithi103 ceph-mon[39629]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:35.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:34 smithi148 ceph-mon[44520]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:35.776 INFO:teuthology.orchestra.run.smithi148.stdout:Created osd(s) 4 on host 'smithi148' 2024-04-23T16:12:36.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:35 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:35 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:35 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:35 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:12:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:35 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:35 smithi148 ceph-mon[44520]: from='osd.4 v1:172.21.15.148:6800/794945071' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T16:12:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:35 smithi148 ceph-mon[44520]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T16:12:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:35 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:35 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:35 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:12:36.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:35 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:36.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:35 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:36.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:35 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:36.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:35 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:12:36.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:35 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:36.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:35 smithi188 ceph-mon[44101]: from='osd.4 v1:172.21.15.148:6800/794945071' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T16:12:36.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:35 smithi188 ceph-mon[44101]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T16:12:36.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:35 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:36.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:35 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:36.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:35 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:12:36.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:35 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:35 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:35 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:35 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:12:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:35 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:35 smithi103 ceph-mon[39629]: from='osd.4 v1:172.21.15.148:6800/794945071' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T16:12:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:35 smithi103 ceph-mon[39629]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T16:12:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:35 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:35 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:35 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:12:36.710 DEBUG:teuthology.orchestra.run.smithi148:osd.4> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.4.service 2024-04-23T16:12:36.713 INFO:tasks.cephadm:Deploying osd.5 on smithi148 with /dev/vg_nvme/lv_3... 2024-04-23T16:12:36.713 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T16:12:36.745 INFO:journalctl@ceph.osd.4.smithi148.stdout:-- Logs begin at Tue 2024-04-23 15:57:31 UTC. -- 2024-04-23T16:12:37.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:36 smithi148 ceph-mon[44520]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T16:12:37.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:36 smithi148 ceph-mon[44520]: osdmap e31: 5 total, 4 up, 5 in 2024-04-23T16:12:37.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:36 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:12:37.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:36 smithi148 ceph-mon[44520]: from='osd.4 v1:172.21.15.148:6800/794945071' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:12:37.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:36 smithi148 ceph-mon[44520]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:12:37.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:36 smithi148 ceph-mon[44520]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:37.124 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:12:36 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[49826]: 2024-04-23T16:12:36.918+0000 7f98b9b65700 -1 osd.4 0 waiting for initial osdmap 2024-04-23T16:12:37.124 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:12:36 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[49826]: 2024-04-23T16:12:36.923+0000 7f98b5958700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:12:37.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:36 smithi188 ceph-mon[44101]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T16:12:37.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:36 smithi188 ceph-mon[44101]: osdmap e31: 5 total, 4 up, 5 in 2024-04-23T16:12:37.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:36 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:12:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:36 smithi188 ceph-mon[44101]: from='osd.4 v1:172.21.15.148:6800/794945071' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:12:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:36 smithi188 ceph-mon[44101]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:12:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:36 smithi188 ceph-mon[44101]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:36 smithi103 ceph-mon[39629]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T16:12:37.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:36 smithi103 ceph-mon[39629]: osdmap e31: 5 total, 4 up, 5 in 2024-04-23T16:12:37.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:36 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:12:37.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:36 smithi103 ceph-mon[39629]: from='osd.4 v1:172.21.15.148:6800/794945071' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:12:37.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:36 smithi103 ceph-mon[39629]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:12:37.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:36 smithi103 ceph-mon[39629]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:37.542 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:12:38.031 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:37 smithi148 ceph-mon[44520]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-04-23T16:12:38.031 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:37 smithi148 ceph-mon[44520]: osdmap e32: 5 total, 4 up, 5 in 2024-04-23T16:12:38.032 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:37 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:12:38.032 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:37 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:12:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:37 smithi188 ceph-mon[44101]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-04-23T16:12:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:37 smithi188 ceph-mon[44101]: osdmap e32: 5 total, 4 up, 5 in 2024-04-23T16:12:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:37 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:12:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:37 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:12:38.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:37 smithi103 ceph-mon[39629]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-04-23T16:12:38.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:37 smithi103 ceph-mon[39629]: osdmap e32: 5 total, 4 up, 5 in 2024-04-23T16:12:38.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:37 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:12:38.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:37 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:12:38.964 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:38 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:12:38.964 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:38 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:12:38.964 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:12:38.964 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:38 smithi148 ceph-mon[44520]: osd.4 v1:172.21.15.148:6800/794945071 boot 2024-04-23T16:12:38.964 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:38 smithi148 ceph-mon[44520]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T16:12:38.965 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:38 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:12:38.965 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:38 smithi148 ceph-mon[44520]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:39.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:38 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:12:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:38 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:12:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:12:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:38 smithi188 ceph-mon[44101]: osd.4 v1:172.21.15.148:6800/794945071 boot 2024-04-23T16:12:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:38 smithi188 ceph-mon[44101]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T16:12:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:38 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:12:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:38 smithi188 ceph-mon[44101]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:38 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:12:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:38 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:12:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:12:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:38 smithi103 ceph-mon[39629]: osd.4 v1:172.21.15.148:6800/794945071 boot 2024-04-23T16:12:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:38 smithi103 ceph-mon[39629]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T16:12:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:38 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:12:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:38 smithi103 ceph-mon[39629]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-23T16:12:40.055 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T16:12:40.055 INFO:teuthology.orchestra.run.smithi148.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T16:12:40.055 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10+0 records in 2024-04-23T16:12:40.055 INFO:teuthology.orchestra.run.smithi148.stderr:10+0 records out 2024-04-23T16:12:40.055 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112937 s, 928 MB/s 2024-04-23T16:12:40.055 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping successful for: 2024-04-23T16:12:40.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:39 smithi188 ceph-mon[44101]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T16:12:40.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:39 smithi103 ceph-mon[39629]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T16:12:40.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:39 smithi148 ceph-mon[44520]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T16:12:40.647 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi148:vg_nvme/lv_3 2024-04-23T16:12:41.142 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:12:41.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:40 smithi188 ceph-mon[44101]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:41.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:40 smithi148 ceph-mon[44520]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:41.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:40 smithi103 ceph-mon[39629]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:42.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:42 smithi148 ceph-mon[44520]: Detected new or changed devices on smithi148 2024-04-23T16:12:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:42 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:42 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:42 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:12:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:42 smithi148 ceph-mon[44520]: Adjusting osd_memory_target on smithi148 to 17122M 2024-04-23T16:12:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:42 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:42 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:42 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:12:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:42 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:42.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:42 smithi188 ceph-mon[44101]: Detected new or changed devices on smithi148 2024-04-23T16:12:42.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:42 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:42.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:42 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:42.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:42 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:12:42.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:42 smithi188 ceph-mon[44101]: Adjusting osd_memory_target on smithi148 to 17122M 2024-04-23T16:12:42.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:42 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:42.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:42 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:42.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:42 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:12:42.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:42 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:42.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:42 smithi103 ceph-mon[39629]: Detected new or changed devices on smithi148 2024-04-23T16:12:42.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:42 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:42.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:42 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:42.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:42 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:12:42.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:42 smithi103 ceph-mon[39629]: Adjusting osd_memory_target on smithi148 to 17122M 2024-04-23T16:12:42.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:42 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:42.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:42 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:42.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:42 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:12:42.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:42 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:43.416 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:43 smithi148 ceph-mon[44520]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:43.416 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:43 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:12:43.416 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:43 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:12:43.416 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:43 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:43.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:43 smithi188 ceph-mon[44101]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:43.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:43 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:12:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:43 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:12:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:43 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:43 smithi103 ceph-mon[39629]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:43 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:12:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:43 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:12:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:43 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:44.266 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:44 smithi148 ceph-mon[44520]: from='client.34253 v1:172.21.15.148:0/3652456785' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:12:44.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:44 smithi188 ceph-mon[44101]: from='client.34253 v1:172.21.15.148:0/3652456785' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:12:44.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:44 smithi103 ceph-mon[39629]: from='client.34253 v1:172.21.15.148:0/3652456785' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:12:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:45 smithi188 ceph-mon[44101]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:45.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:45 smithi103 ceph-mon[39629]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:45.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:45 smithi148 ceph-mon[44520]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:46.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:46 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.148:0/2076370426' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d492189-c3cd-47ad-a12a-ed4d275bcfe3"}]: dispatch 2024-04-23T16:12:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:46 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.148:0/2076370426' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d492189-c3cd-47ad-a12a-ed4d275bcfe3"}]': finished 2024-04-23T16:12:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:46 smithi188 ceph-mon[44101]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T16:12:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:46 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T16:12:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:46 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.148:0/389101690' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:12:46.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:46 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.148:0/2076370426' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d492189-c3cd-47ad-a12a-ed4d275bcfe3"}]: dispatch 2024-04-23T16:12:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:46 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.148:0/2076370426' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d492189-c3cd-47ad-a12a-ed4d275bcfe3"}]': finished 2024-04-23T16:12:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:46 smithi103 ceph-mon[39629]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T16:12:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:46 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T16:12:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:46 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.148:0/389101690' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:12:46.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:46 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.148:0/2076370426' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d492189-c3cd-47ad-a12a-ed4d275bcfe3"}]: dispatch 2024-04-23T16:12:46.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:46 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.148:0/2076370426' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d492189-c3cd-47ad-a12a-ed4d275bcfe3"}]': finished 2024-04-23T16:12:46.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:46 smithi148 ceph-mon[44520]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T16:12:46.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:46 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T16:12:46.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:46 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.148:0/389101690' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:12:47.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:47 smithi188 ceph-mon[44101]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:47.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:47 smithi103 ceph-mon[39629]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:47.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:47 smithi148 ceph-mon[44520]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:49.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:49 smithi188 ceph-mon[44101]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:49.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:49 smithi103 ceph-mon[39629]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:49.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:49 smithi148 ceph-mon[44520]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:51.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:51 smithi148 ceph-mon[44520]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:51.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:51 smithi188 ceph-mon[44101]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:51.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:51 smithi103 ceph-mon[39629]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:52.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T16:12:52.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:52.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T16:12:52.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:52.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T16:12:52.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:53.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:53 smithi148 ceph-mon[44520]: Deploying daemon osd.5 on smithi148 2024-04-23T16:12:53.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:53 smithi148 ceph-mon[44520]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:53.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:53 smithi188 ceph-mon[44101]: Deploying daemon osd.5 on smithi148 2024-04-23T16:12:53.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:53 smithi188 ceph-mon[44101]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:53.854 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:53 smithi103 ceph-mon[39629]: Deploying daemon osd.5 on smithi148 2024-04-23T16:12:53.854 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:53 smithi103 ceph-mon[39629]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:55.693 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:55 smithi148 ceph-mon[44520]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:55.693 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:55 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:12:55.693 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:55 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:12:55.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:55 smithi188 ceph-mon[44101]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:55.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:55 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:12:55.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:55 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:12:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:55 smithi103 ceph-mon[39629]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:55 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:12:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:55 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:12:57.469 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:57 smithi148 ceph-mon[44520]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:57.469 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:57 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:57.469 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:57 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:12:57.469 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:57 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:57.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:57 smithi188 ceph-mon[44101]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:57.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:57 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:57.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:57 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:12:57.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:57 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:57 smithi103 ceph-mon[39629]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:12:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:59.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:59 smithi188 ceph-mon[44101]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:59 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:59 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:59 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:59 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:12:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:12:59 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:59.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:59 smithi103 ceph-mon[39629]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:59.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:59 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:59.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:59 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:59.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:59 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:59.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:59 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:12:59.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:12:59 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:59.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:59 smithi148 ceph-mon[44520]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:12:59.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:59 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:59.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:59 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:12:59.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:59 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:12:59.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:59 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:12:59.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:12:59 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:00.287 INFO:teuthology.orchestra.run.smithi148.stdout:Created osd(s) 5 on host 'smithi148' 2024-04-23T16:13:00.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:00 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:13:00.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:00 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:13:00.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:00 smithi148 ceph-mon[44520]: from='osd.5 v1:172.21.15.148:6804/3547200711' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T16:13:00.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:00 smithi148 ceph-mon[44520]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T16:13:00.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:00 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:00.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:00 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:00.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:00 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:13:00.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:00 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:13:00.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:00 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:13:00.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:00 smithi188 ceph-mon[44101]: from='osd.5 v1:172.21.15.148:6804/3547200711' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T16:13:00.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:00 smithi188 ceph-mon[44101]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T16:13:00.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:00 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:00.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:00 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:00.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:00 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:13:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:00 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:13:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:00 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:13:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:00 smithi103 ceph-mon[39629]: from='osd.5 v1:172.21.15.148:6804/3547200711' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T16:13:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:00 smithi103 ceph-mon[39629]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T16:13:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:00 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:00 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:00 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:13:00.923 DEBUG:teuthology.orchestra.run.smithi148:osd.5> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.5.service 2024-04-23T16:13:00.925 INFO:tasks.cephadm:Deploying osd.6 on smithi148 with /dev/vg_nvme/lv_2... 2024-04-23T16:13:00.926 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T16:13:00.960 INFO:journalctl@ceph.osd.5.smithi148.stdout:-- Logs begin at Tue 2024-04-23 15:57:31 UTC. -- 2024-04-23T16:13:01.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:01 smithi188 ceph-mon[44101]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:13:01.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:01 smithi188 ceph-mon[44101]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T16:13:01.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:01 smithi188 ceph-mon[44101]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T16:13:01.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:01 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T16:13:01.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:01 smithi188 ceph-mon[44101]: from='osd.5 v1:172.21.15.148:6804/3547200711' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:13:01.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:01 smithi188 ceph-mon[44101]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:13:01.764 INFO:journalctl@ceph.osd.5.smithi148.stdout:Apr 23 16:13:01 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-5[56866]: 2024-04-23T16:13:01.455+0000 7f8e0c891700 -1 osd.5 0 waiting for initial osdmap 2024-04-23T16:13:01.764 INFO:journalctl@ceph.osd.5.smithi148.stdout:Apr 23 16:13:01 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-5[56866]: 2024-04-23T16:13:01.460+0000 7f8e08684700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:13:01.765 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:01 smithi148 ceph-mon[44520]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:13:01.765 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:01 smithi148 ceph-mon[44520]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T16:13:01.765 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:01 smithi148 ceph-mon[44520]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T16:13:01.765 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:01 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T16:13:01.765 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:01 smithi148 ceph-mon[44520]: from='osd.5 v1:172.21.15.148:6804/3547200711' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:13:01.765 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:01 smithi148 ceph-mon[44520]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:13:01.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:01 smithi103 ceph-mon[39629]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:13:01.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:01 smithi103 ceph-mon[39629]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T16:13:01.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:01 smithi103 ceph-mon[39629]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T16:13:01.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:01 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T16:13:01.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:01 smithi103 ceph-mon[39629]: from='osd.5 v1:172.21.15.148:6804/3547200711' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:13:01.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:01 smithi103 ceph-mon[39629]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:13:02.573 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:13:02.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:02 smithi148 ceph-mon[44520]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-04-23T16:13:02.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:02 smithi148 ceph-mon[44520]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T16:13:02.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:02 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T16:13:02.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:02 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T16:13:02.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:02 smithi188 ceph-mon[44101]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-04-23T16:13:02.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:02 smithi188 ceph-mon[44101]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T16:13:02.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T16:13:02.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:02 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T16:13:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:02 smithi103 ceph-mon[39629]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-04-23T16:13:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:02 smithi103 ceph-mon[39629]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T16:13:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:02 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T16:13:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:02 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T16:13:03.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:03 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:13:03.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:03 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:13:03.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:13:03.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:03 smithi188 ceph-mon[44101]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:13:03.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:03 smithi188 ceph-mon[44101]: osd.5 v1:172.21.15.148:6804/3547200711 boot 2024-04-23T16:13:03.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:03 smithi188 ceph-mon[44101]: osdmap e38: 6 total, 6 up, 6 in 2024-04-23T16:13:03.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:03 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T16:13:03.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:03 smithi188 ceph-mon[44101]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T16:13:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:03 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:13:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:03 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:13:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:13:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:03 smithi103 ceph-mon[39629]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:13:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:03 smithi103 ceph-mon[39629]: osd.5 v1:172.21.15.148:6804/3547200711 boot 2024-04-23T16:13:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:03 smithi103 ceph-mon[39629]: osdmap e38: 6 total, 6 up, 6 in 2024-04-23T16:13:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:03 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T16:13:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:03 smithi103 ceph-mon[39629]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T16:13:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:03 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:13:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:03 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:13:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:13:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:03 smithi148 ceph-mon[44520]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-23T16:13:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:03 smithi148 ceph-mon[44520]: osd.5 v1:172.21.15.148:6804/3547200711 boot 2024-04-23T16:13:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:03 smithi148 ceph-mon[44520]: osdmap e38: 6 total, 6 up, 6 in 2024-04-23T16:13:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:03 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T16:13:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:03 smithi148 ceph-mon[44520]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T16:13:05.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:04 smithi188 ceph-mon[44101]: Detected new or changed devices on smithi148 2024-04-23T16:13:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:04 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:04 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:04 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:04 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:04 smithi188 ceph-mon[44101]: Adjusting osd_memory_target on smithi148 to 8561M 2024-04-23T16:13:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:04 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:04 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:04 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:13:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:04 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:04 smithi188 ceph-mon[44101]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:04 smithi188 ceph-mon[44101]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T16:13:05.256 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T16:13:05.256 INFO:teuthology.orchestra.run.smithi148.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T16:13:05.256 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10+0 records in 2024-04-23T16:13:05.256 INFO:teuthology.orchestra.run.smithi148.stderr:10+0 records out 2024-04-23T16:13:05.256 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111866 s, 937 MB/s 2024-04-23T16:13:05.257 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping successful for: 2024-04-23T16:13:05.307 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:04 smithi148 ceph-mon[44520]: Detected new or changed devices on smithi148 2024-04-23T16:13:05.307 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:04 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:05.307 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:04 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:05.307 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:04 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:05.307 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:04 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:05.307 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:04 smithi148 ceph-mon[44520]: Adjusting osd_memory_target on smithi148 to 8561M 2024-04-23T16:13:05.307 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:04 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:05.307 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:04 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:05.308 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:04 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:13:05.308 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:04 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:05.308 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:04 smithi148 ceph-mon[44520]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:05.308 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:04 smithi148 ceph-mon[44520]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T16:13:05.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:04 smithi103 ceph-mon[39629]: Detected new or changed devices on smithi148 2024-04-23T16:13:05.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:04 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:05.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:04 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:05.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:04 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:05.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:04 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:05.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:04 smithi103 ceph-mon[39629]: Adjusting osd_memory_target on smithi148 to 8561M 2024-04-23T16:13:05.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:04 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:05.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:04 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:05.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:04 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:13:05.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:04 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:05.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:04 smithi103 ceph-mon[39629]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:05.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:04 smithi103 ceph-mon[39629]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T16:13:05.919 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi148:vg_nvme/lv_2 2024-04-23T16:13:06.184 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:13:06.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:05 smithi188 ceph-mon[44101]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-23T16:13:06.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:05 smithi188 ceph-mon[44101]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T16:13:06.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:05 smithi103 ceph-mon[39629]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-23T16:13:06.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:05 smithi103 ceph-mon[39629]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T16:13:06.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:05 smithi148 ceph-mon[44520]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-23T16:13:06.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:05 smithi148 ceph-mon[44520]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T16:13:07.569 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:07 smithi148 ceph-mon[44520]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:07.569 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:07 smithi148 ceph-mon[44520]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T16:13:07.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:07 smithi188 ceph-mon[44101]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:07.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:07 smithi188 ceph-mon[44101]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T16:13:07.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:07 smithi103 ceph-mon[39629]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:07.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:07 smithi103 ceph-mon[39629]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T16:13:08.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:08 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:13:08.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:08 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:13:08.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:08 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:08.777 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:08 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:13:08.777 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:08 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:13:08.777 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:08 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:08.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:08 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:13:08.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:08 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:13:08.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:08 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:09.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:09 smithi188 ceph-mon[44101]: from='client.14394 v1:172.21.15.148:0/2796029297' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:13:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:09 smithi188 ceph-mon[44101]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:09 smithi188 ceph-mon[44101]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-23T16:13:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:09 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:13:09.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:09 smithi103 ceph-mon[39629]: from='client.14394 v1:172.21.15.148:0/2796029297' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:13:09.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:09 smithi103 ceph-mon[39629]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:09.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:09 smithi103 ceph-mon[39629]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-23T16:13:09.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:09 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:13:09.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:09 smithi148 ceph-mon[44520]: from='client.14394 v1:172.21.15.148:0/2796029297' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:13:09.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:09 smithi148 ceph-mon[44520]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:09.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:09 smithi148 ceph-mon[44520]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-23T16:13:09.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:09 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:13:11.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:11 smithi188 ceph-mon[44101]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:11.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:11 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.148:0/1967886660' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89f2379e-cf14-4e69-8747-7535db81b70b"}]: dispatch 2024-04-23T16:13:11.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:11 smithi188 ceph-mon[44101]: from='client.34280 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89f2379e-cf14-4e69-8747-7535db81b70b"}]: dispatch 2024-04-23T16:13:11.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:11 smithi188 ceph-mon[44101]: from='client.34280 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89f2379e-cf14-4e69-8747-7535db81b70b"}]': finished 2024-04-23T16:13:11.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:11 smithi188 ceph-mon[44101]: osdmap e43: 7 total, 6 up, 7 in 2024-04-23T16:13:11.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:11 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:11.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:11 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.148:0/4028239327' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:13:11.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:11 smithi103 ceph-mon[39629]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:11 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.148:0/1967886660' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89f2379e-cf14-4e69-8747-7535db81b70b"}]: dispatch 2024-04-23T16:13:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:11 smithi103 ceph-mon[39629]: from='client.34280 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89f2379e-cf14-4e69-8747-7535db81b70b"}]: dispatch 2024-04-23T16:13:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:11 smithi103 ceph-mon[39629]: from='client.34280 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89f2379e-cf14-4e69-8747-7535db81b70b"}]': finished 2024-04-23T16:13:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:11 smithi103 ceph-mon[39629]: osdmap e43: 7 total, 6 up, 7 in 2024-04-23T16:13:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:11 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:11 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.148:0/4028239327' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:13:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:11 smithi148 ceph-mon[44520]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:11 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.148:0/1967886660' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89f2379e-cf14-4e69-8747-7535db81b70b"}]: dispatch 2024-04-23T16:13:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:11 smithi148 ceph-mon[44520]: from='client.34280 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89f2379e-cf14-4e69-8747-7535db81b70b"}]: dispatch 2024-04-23T16:13:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:11 smithi148 ceph-mon[44520]: from='client.34280 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89f2379e-cf14-4e69-8747-7535db81b70b"}]': finished 2024-04-23T16:13:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:11 smithi148 ceph-mon[44520]: osdmap e43: 7 total, 6 up, 7 in 2024-04-23T16:13:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:11 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:11 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.148:0/4028239327' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:13:13.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:13 smithi103 ceph-mon[39629]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:13 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:13:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:13 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:13:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:13 smithi148 ceph-mon[44520]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:13 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:13:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:13 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:13:13.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:13 smithi188 ceph-mon[44101]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:13 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:13:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:13 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:13:14.487 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:14 smithi148 ceph-mon[44520]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T16:13:14.487 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:14.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:14 smithi103 ceph-mon[39629]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T16:13:14.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:14.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:14 smithi188 ceph-mon[44101]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T16:13:14.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:15.551 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:15 smithi148 ceph-mon[44520]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:15.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:15 smithi103 ceph-mon[39629]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:15.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:15 smithi188 ceph-mon[44101]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:17.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:17 smithi103 ceph-mon[39629]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:17.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T16:13:17.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:17.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:17 smithi148 ceph-mon[44520]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:17.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T16:13:17.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:17.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:17 smithi188 ceph-mon[44101]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T16:13:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:18.612 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:18 smithi148 ceph-mon[44520]: Deploying daemon osd.6 on smithi148 2024-04-23T16:13:18.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:18 smithi103 ceph-mon[39629]: Deploying daemon osd.6 on smithi148 2024-04-23T16:13:18.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:18 smithi188 ceph-mon[44101]: Deploying daemon osd.6 on smithi148 2024-04-23T16:13:19.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:19 smithi148 ceph-mon[44520]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:19.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:19 smithi103 ceph-mon[39629]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:19.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:19 smithi188 ceph-mon[44101]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:21.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:21 smithi148 ceph-mon[44520]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:21.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:21 smithi103 ceph-mon[39629]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:21.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:21 smithi188 ceph-mon[44101]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:13:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:23 smithi148 ceph-mon[44520]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:13:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:23 smithi188 ceph-mon[44101]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:23.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:23.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:23.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:13:23.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:23 smithi103 ceph-mon[39629]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:25.529 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:25 smithi148 ceph-mon[44520]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:25.529 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:25.529 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:25.529 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:25.529 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:13:25.529 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:25.529 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:25 smithi148 ceph-mon[44520]: from='osd.6 v1:172.21.15.148:6808/366919758' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T16:13:25.529 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:25 smithi148 ceph-mon[44520]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T16:13:25.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:25 smithi188 ceph-mon[44101]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:13:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:25 smithi188 ceph-mon[44101]: from='osd.6 v1:172.21.15.148:6808/366919758' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T16:13:25.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:25 smithi188 ceph-mon[44101]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T16:13:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:25 smithi103 ceph-mon[39629]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:25.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:25.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:13:25.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:25.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:25 smithi103 ceph-mon[39629]: from='osd.6 v1:172.21.15.148:6808/366919758' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T16:13:25.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:25 smithi103 ceph-mon[39629]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T16:13:25.907 INFO:teuthology.orchestra.run.smithi148.stdout:Created osd(s) 6 on host 'smithi148' 2024-04-23T16:13:26.599 DEBUG:teuthology.orchestra.run.smithi148:osd.6> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.6.service 2024-04-23T16:13:26.602 INFO:tasks.cephadm:Deploying osd.7 on smithi148 with /dev/vg_nvme/lv_1... 2024-04-23T16:13:26.602 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T16:13:26.634 INFO:journalctl@ceph.osd.6.smithi148.stdout:-- Logs begin at Tue 2024-04-23 15:57:31 UTC. -- 2024-04-23T16:13:27.083 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:26 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:27.083 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:26 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:27.084 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:26 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:13:27.084 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:26 smithi148 ceph-mon[44520]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T16:13:27.084 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:26 smithi148 ceph-mon[44520]: from='osd.6 v1:172.21.15.148:6808/366919758' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:13:27.084 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:26 smithi148 ceph-mon[44520]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T16:13:27.084 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:26 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:27.084 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:26 smithi148 ceph-mon[44520]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:13:27.084 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:26 smithi148 ceph-mon[44520]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:27.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:26 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:27.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:26 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:26 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:13:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:26 smithi188 ceph-mon[44101]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T16:13:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:26 smithi188 ceph-mon[44101]: from='osd.6 v1:172.21.15.148:6808/366919758' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:13:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:26 smithi188 ceph-mon[44101]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T16:13:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:26 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:26 smithi188 ceph-mon[44101]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:13:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:26 smithi188 ceph-mon[44101]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:26 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:26 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:26 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:13:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:26 smithi103 ceph-mon[39629]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T16:13:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:26 smithi103 ceph-mon[39629]: from='osd.6 v1:172.21.15.148:6808/366919758' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:13:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:26 smithi103 ceph-mon[39629]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T16:13:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:26 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:26 smithi103 ceph-mon[39629]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:13:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:26 smithi103 ceph-mon[39629]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:27.770 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 16:13:27 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T16:13:27.392+0000 7f5d5cf6c700 -1 osd.6 0 waiting for initial osdmap 2024-04-23T16:13:27.770 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 16:13:27 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T16:13:27.400+0000 7f5d54d57700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:13:27.771 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:13:28.029 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:27 smithi148 ceph-mon[44520]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-04-23T16:13:28.030 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:27 smithi148 ceph-mon[44520]: osdmap e46: 7 total, 6 up, 7 in 2024-04-23T16:13:28.030 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:27 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:28.030 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:27 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:28.030 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:27 smithi148 ceph-mon[44520]: from='osd.6 ' entity='osd.6' 2024-04-23T16:13:28.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:27 smithi188 ceph-mon[44101]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-04-23T16:13:28.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:27 smithi188 ceph-mon[44101]: osdmap e46: 7 total, 6 up, 7 in 2024-04-23T16:13:28.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:27 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:28.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:27 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:28.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:27 smithi188 ceph-mon[44101]: from='osd.6 ' entity='osd.6' 2024-04-23T16:13:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:27 smithi103 ceph-mon[39629]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-04-23T16:13:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:27 smithi103 ceph-mon[39629]: osdmap e46: 7 total, 6 up, 7 in 2024-04-23T16:13:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:27 smithi103 ceph-mon[39629]: from='osd.6 ' entity='osd.6' 2024-04-23T16:13:29.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:28 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:13:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:28 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:13:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:13:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:13:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:28 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:28 smithi188 ceph-mon[44101]: osd.6 v1:172.21.15.148:6808/366919758 boot 2024-04-23T16:13:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:28 smithi188 ceph-mon[44101]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T16:13:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:28 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:28 smithi188 ceph-mon[44101]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:28 smithi188 ceph-mon[44101]: osdmap e48: 7 total, 7 up, 7 in 2024-04-23T16:13:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:28 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:13:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:28 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:13:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:13:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:13:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:28 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:28 smithi103 ceph-mon[39629]: osd.6 v1:172.21.15.148:6808/366919758 boot 2024-04-23T16:13:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:28 smithi103 ceph-mon[39629]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T16:13:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:28 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:28 smithi103 ceph-mon[39629]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:28 smithi103 ceph-mon[39629]: osdmap e48: 7 total, 7 up, 7 in 2024-04-23T16:13:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:28 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:13:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:28 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:13:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:13:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:13:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:28 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:28 smithi148 ceph-mon[44520]: osd.6 v1:172.21.15.148:6808/366919758 boot 2024-04-23T16:13:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:28 smithi148 ceph-mon[44520]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T16:13:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:28 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T16:13:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:28 smithi148 ceph-mon[44520]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-23T16:13:29.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:28 smithi148 ceph-mon[44520]: osdmap e48: 7 total, 7 up, 7 in 2024-04-23T16:13:30.590 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T16:13:30.590 INFO:teuthology.orchestra.run.smithi148.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T16:13:30.590 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10+0 records in 2024-04-23T16:13:30.590 INFO:teuthology.orchestra.run.smithi148.stderr:10+0 records out 2024-04-23T16:13:30.590 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111459 s, 941 MB/s 2024-04-23T16:13:30.590 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping successful for: 2024-04-23T16:13:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:30 smithi103 ceph-mon[39629]: osdmap e49: 7 total, 7 up, 7 in 2024-04-23T16:13:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:30 smithi103 ceph-mon[39629]: pgmap v167: 1 pgs: 1 remapped+peering; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-23T16:13:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:30 smithi103 ceph-mon[39629]: Detected new or changed devices on smithi148 2024-04-23T16:13:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:30 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:30 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:30 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:30 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:31.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:30 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:31.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:30 smithi103 ceph-mon[39629]: Adjusting osd_memory_target on smithi148 to 5707M 2024-04-23T16:13:31.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:30 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:31.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:30 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:31.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:30 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:13:31.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:30 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:31.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:30 smithi148 ceph-mon[44520]: osdmap e49: 7 total, 7 up, 7 in 2024-04-23T16:13:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:30 smithi148 ceph-mon[44520]: pgmap v167: 1 pgs: 1 remapped+peering; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-23T16:13:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:30 smithi148 ceph-mon[44520]: Detected new or changed devices on smithi148 2024-04-23T16:13:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:30 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:30 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:30 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:30 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:30 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:30 smithi148 ceph-mon[44520]: Adjusting osd_memory_target on smithi148 to 5707M 2024-04-23T16:13:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:30 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:30 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:30 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:13:31.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:30 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:31.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:30 smithi188 ceph-mon[44101]: osdmap e49: 7 total, 7 up, 7 in 2024-04-23T16:13:31.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:30 smithi188 ceph-mon[44101]: pgmap v167: 1 pgs: 1 remapped+peering; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-23T16:13:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:30 smithi188 ceph-mon[44101]: Detected new or changed devices on smithi148 2024-04-23T16:13:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:30 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:30 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:30 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:30 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:30 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:30 smithi188 ceph-mon[44101]: Adjusting osd_memory_target on smithi148 to 5707M 2024-04-23T16:13:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:30 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:30 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:30 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:13:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:30 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:31.259 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi148:vg_nvme/lv_1 2024-04-23T16:13:31.522 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:13:31.989 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:31 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:13:31.989 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:31 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:13:31.989 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:31 smithi148 ceph-mon[44520]: osdmap e50: 7 total, 7 up, 7 in 2024-04-23T16:13:32.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:31 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:13:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:31 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:13:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:31 smithi103 ceph-mon[39629]: osdmap e50: 7 total, 7 up, 7 in 2024-04-23T16:13:32.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:31 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:13:32.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:31 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:13:32.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:31 smithi188 ceph-mon[44101]: osdmap e50: 7 total, 7 up, 7 in 2024-04-23T16:13:32.947 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:32 smithi148 ceph-mon[44520]: osdmap e51: 7 total, 7 up, 7 in 2024-04-23T16:13:32.948 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:32 smithi148 ceph-mon[44520]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-23T16:13:33.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:32 smithi103 ceph-mon[39629]: osdmap e51: 7 total, 7 up, 7 in 2024-04-23T16:13:33.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:32 smithi103 ceph-mon[39629]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-23T16:13:33.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:32 smithi188 ceph-mon[44101]: osdmap e51: 7 total, 7 up, 7 in 2024-04-23T16:13:33.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:32 smithi188 ceph-mon[44101]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-23T16:13:34.023 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:33 smithi148 ceph-mon[44520]: from='client.34301 v1:172.21.15.148:0/3172764084' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:13:34.024 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:33 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:13:34.024 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:33 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:13:34.024 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:33 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:33 smithi103 ceph-mon[39629]: from='client.34301 v1:172.21.15.148:0/3172764084' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:13:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:33 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:13:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:33 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:13:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:33 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:34.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:33 smithi188 ceph-mon[44101]: from='client.34301 v1:172.21.15.148:0/3172764084' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:13:34.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:33 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:13:34.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:33 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:13:34.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:33 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:34.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:34 smithi148 ceph-mon[44520]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-23T16:13:35.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:34 smithi103 ceph-mon[39629]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-23T16:13:35.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:34 smithi188 ceph-mon[44101]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-23T16:13:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.148:0/3253898531' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f27b035a-558f-4528-80ea-2a00d03f6539"}]: dispatch 2024-04-23T16:13:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:36 smithi103 ceph-mon[39629]: from='client.34307 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f27b035a-558f-4528-80ea-2a00d03f6539"}]: dispatch 2024-04-23T16:13:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:36 smithi103 ceph-mon[39629]: from='client.34307 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f27b035a-558f-4528-80ea-2a00d03f6539"}]': finished 2024-04-23T16:13:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:36 smithi103 ceph-mon[39629]: osdmap e52: 8 total, 7 up, 8 in 2024-04-23T16:13:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:36 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.148:0/3253898531' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f27b035a-558f-4528-80ea-2a00d03f6539"}]: dispatch 2024-04-23T16:13:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:36 smithi148 ceph-mon[44520]: from='client.34307 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f27b035a-558f-4528-80ea-2a00d03f6539"}]: dispatch 2024-04-23T16:13:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:36 smithi148 ceph-mon[44520]: from='client.34307 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f27b035a-558f-4528-80ea-2a00d03f6539"}]': finished 2024-04-23T16:13:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:36 smithi148 ceph-mon[44520]: osdmap e52: 8 total, 7 up, 8 in 2024-04-23T16:13:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:36 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:36.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.148:0/3253898531' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f27b035a-558f-4528-80ea-2a00d03f6539"}]: dispatch 2024-04-23T16:13:36.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:36 smithi188 ceph-mon[44101]: from='client.34307 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f27b035a-558f-4528-80ea-2a00d03f6539"}]: dispatch 2024-04-23T16:13:36.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:36 smithi188 ceph-mon[44101]: from='client.34307 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f27b035a-558f-4528-80ea-2a00d03f6539"}]': finished 2024-04-23T16:13:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:36 smithi188 ceph-mon[44101]: osdmap e52: 8 total, 7 up, 8 in 2024-04-23T16:13:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:36 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:37 smithi103 ceph-mon[39629]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:37 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.148:0/2857408452' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:13:37.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:37 smithi148 ceph-mon[44520]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:37 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.148:0/2857408452' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:13:37.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:37 smithi188 ceph-mon[44101]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:37.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:37 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.148:0/2857408452' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:13:39.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:39 smithi148 ceph-mon[44520]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:39.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:39 smithi188 ceph-mon[44101]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:39 smithi103 ceph-mon[39629]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:41.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:41 smithi188 ceph-mon[44101]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:41.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:41 smithi103 ceph-mon[39629]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:41.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:41 smithi148 ceph-mon[44520]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:43.706 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:43 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:13:43.707 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:43 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:13:43.707 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:43 smithi148 ceph-mon[44520]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:43.707 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:43 smithi148 ceph-mon[44520]: osdmap e53: 8 total, 7 up, 8 in 2024-04-23T16:13:43.707 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:43 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:43.707 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:43 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T16:13:43.707 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:43 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:43.707 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:43 smithi148 ceph-mon[44520]: Deploying daemon osd.7 on smithi148 2024-04-23T16:13:43.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:43 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:13:43.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:43 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:13:43.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:43 smithi188 ceph-mon[44101]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:43.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:43 smithi188 ceph-mon[44101]: osdmap e53: 8 total, 7 up, 8 in 2024-04-23T16:13:43.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:43 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:43.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:43 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T16:13:43.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:43 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:43.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:43 smithi188 ceph-mon[44101]: Deploying daemon osd.7 on smithi148 2024-04-23T16:13:43.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:43 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:13:43.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:43 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:13:43.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:43 smithi103 ceph-mon[39629]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:43.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:43 smithi103 ceph-mon[39629]: osdmap e53: 8 total, 7 up, 8 in 2024-04-23T16:13:43.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:43 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:43.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:43 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T16:13:43.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:43 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:43.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:43 smithi103 ceph-mon[39629]: Deploying daemon osd.7 on smithi148 2024-04-23T16:13:45.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:45 smithi148 ceph-mon[44520]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:45.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:45 smithi188 ceph-mon[44101]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:45.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:45 smithi103 ceph-mon[39629]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:47.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:47 smithi188 ceph-mon[44101]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:47.775 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:47 smithi148 ceph-mon[44520]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:47.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:47 smithi103 ceph-mon[39629]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:48 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:48 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:48 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:13:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:48 smithi103 ceph-mon[39629]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:49.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:48 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:49.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:48 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:49.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:48 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:13:49.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:48 smithi148 ceph-mon[44520]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:49.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:48 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:49.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:48 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:49.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:48 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:13:49.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:48 smithi188 ceph-mon[44101]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:51.443 INFO:teuthology.orchestra.run.smithi148.stdout:Created osd(s) 7 on host 'smithi148' 2024-04-23T16:13:51.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:51 smithi188 ceph-mon[44101]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:51.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:51 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:51.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:51 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:51.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:51 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:51.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:51 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:13:51.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:51 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:51.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:51 smithi188 ceph-mon[44101]: from='osd.7 v1:172.21.15.148:6812/3719550888' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T16:13:51.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:51 smithi188 ceph-mon[44101]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T16:13:51.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:51 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:51.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:51 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:51.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:51 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:13:51.795 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:51 smithi148 ceph-mon[44520]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:51.795 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:51 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:51.795 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:51 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:51.795 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:51 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:51.795 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:51 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:13:51.795 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:51 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:51.795 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:51 smithi148 ceph-mon[44520]: from='osd.7 v1:172.21.15.148:6812/3719550888' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T16:13:51.795 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:51 smithi148 ceph-mon[44520]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T16:13:51.795 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:51 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:51.796 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:51 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:51.796 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:51 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:13:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:51 smithi103 ceph-mon[39629]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:51 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:51 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:51 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:51 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:13:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:51 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:51 smithi103 ceph-mon[39629]: from='osd.7 v1:172.21.15.148:6812/3719550888' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T16:13:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:51 smithi103 ceph-mon[39629]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T16:13:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:51 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:51 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:51 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:13:52.117 DEBUG:teuthology.orchestra.run.smithi148:osd.7> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.7.service 2024-04-23T16:13:52.120 INFO:tasks.cephadm:Deploying osd.8 on smithi188 with /dev/vg_nvme/lv_4... 2024-04-23T16:13:52.120 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T16:13:52.151 INFO:journalctl@ceph.osd.7.smithi148.stdout:-- Logs begin at Tue 2024-04-23 15:57:31 UTC. -- 2024-04-23T16:13:52.390 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:13:52.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:52 smithi103 ceph-mon[39629]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T16:13:52.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:52 smithi103 ceph-mon[39629]: osdmap e54: 8 total, 7 up, 8 in 2024-04-23T16:13:52.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:52.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:52 smithi103 ceph-mon[39629]: from='osd.7 v1:172.21.15.148:6812/3719550888' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:13:52.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:52 smithi103 ceph-mon[39629]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:13:52.854 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:52 smithi188 ceph-mon[44101]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T16:13:52.854 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:52 smithi188 ceph-mon[44101]: osdmap e54: 8 total, 7 up, 8 in 2024-04-23T16:13:52.854 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:52.854 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:52 smithi188 ceph-mon[44101]: from='osd.7 v1:172.21.15.148:6812/3719550888' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:13:52.854 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:52 smithi188 ceph-mon[44101]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:13:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:52 smithi148 ceph-mon[44520]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T16:13:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:52 smithi148 ceph-mon[44520]: osdmap e54: 8 total, 7 up, 8 in 2024-04-23T16:13:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:52 smithi148 ceph-mon[44520]: from='osd.7 v1:172.21.15.148:6812/3719550888' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:13:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:52 smithi148 ceph-mon[44520]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:13:52.874 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 16:13:52 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T16:13:52.487+0000 7f53a5490700 -1 osd.7 0 waiting for initial osdmap 2024-04-23T16:13:52.875 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 16:13:52 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T16:13:52.496+0000 7f53a1283700 -1 osd.7 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:13:53.596 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:53 smithi148 ceph-mon[44520]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:53.596 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:53 smithi148 ceph-mon[44520]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-04-23T16:13:53.596 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:53 smithi148 ceph-mon[44520]: osdmap e55: 8 total, 7 up, 8 in 2024-04-23T16:13:53.596 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:53 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:53.596 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:53 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:53.597 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:53 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:53 smithi103 ceph-mon[39629]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:53 smithi103 ceph-mon[39629]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-04-23T16:13:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:53 smithi103 ceph-mon[39629]: osdmap e55: 8 total, 7 up, 8 in 2024-04-23T16:13:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:53.855 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:53 smithi188 ceph-mon[44101]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-23T16:13:53.855 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:53 smithi188 ceph-mon[44101]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-04-23T16:13:53.855 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:53 smithi188 ceph-mon[44101]: osdmap e55: 8 total, 7 up, 8 in 2024-04-23T16:13:53.855 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:53 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:53.856 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:53 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:53.856 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:53 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:54.456 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T16:13:54.456 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T16:13:54.456 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-23T16:13:54.456 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-23T16:13:54.456 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128643 s, 815 MB/s 2024-04-23T16:13:54.457 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-23T16:13:54.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:54 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:13:54.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:54 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:13:54.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:13:54.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:54 smithi188 ceph-mon[44101]: osd.7 v1:172.21.15.148:6812/3719550888 boot 2024-04-23T16:13:54.839 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:54 smithi188 ceph-mon[44101]: osdmap e56: 8 total, 8 up, 8 in 2024-04-23T16:13:54.839 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:54 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:54.839 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:54 smithi188 ceph-mon[44101]: osdmap e57: 8 total, 8 up, 8 in 2024-04-23T16:13:54.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:54 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:13:54.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:54 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:13:54.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:13:54.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:54 smithi103 ceph-mon[39629]: osd.7 v1:172.21.15.148:6812/3719550888 boot 2024-04-23T16:13:54.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:54 smithi103 ceph-mon[39629]: osdmap e56: 8 total, 8 up, 8 in 2024-04-23T16:13:54.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:54 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:54.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:54 smithi103 ceph-mon[39629]: osdmap e57: 8 total, 8 up, 8 in 2024-04-23T16:13:54.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:54 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:13:54.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:54 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:13:54.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:13:54.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:54 smithi148 ceph-mon[44520]: osd.7 v1:172.21.15.148:6812/3719550888 boot 2024-04-23T16:13:54.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:54 smithi148 ceph-mon[44520]: osdmap e56: 8 total, 8 up, 8 in 2024-04-23T16:13:54.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:54 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T16:13:54.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:54 smithi148 ceph-mon[44520]: osdmap e57: 8 total, 8 up, 8 in 2024-04-23T16:13:55.171 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_4 2024-04-23T16:13:55.436 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:13:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:55 smithi103 ceph-mon[39629]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-23T16:13:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:55 smithi103 ceph-mon[39629]: osdmap e58: 8 total, 8 up, 8 in 2024-04-23T16:13:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:55 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:55 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:55 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:55 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:55 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:55 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:55.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:55 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:55.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:55 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:55.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:55 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:13:55.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:55 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:55.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:55 smithi148 ceph-mon[44520]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-23T16:13:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:55 smithi148 ceph-mon[44520]: osdmap e58: 8 total, 8 up, 8 in 2024-04-23T16:13:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:55 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:55 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:55 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:55 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:55 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:55 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:55 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:55 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:55 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:13:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:55 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:55.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:55 smithi188 ceph-mon[44101]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-23T16:13:55.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:55 smithi188 ceph-mon[44101]: osdmap e58: 8 total, 8 up, 8 in 2024-04-23T16:13:55.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:55 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:55.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:55 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:55.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:55 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:55.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:55 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:55.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:55 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:55.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:55 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:13:55.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:55 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:55.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:55 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:55.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:55 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:13:55.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:55 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:13:56.633 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:56 smithi188 ceph-mon[44101]: Detected new or changed devices on smithi148 2024-04-23T16:13:56.634 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:56 smithi188 ceph-mon[44101]: Adjusting osd_memory_target on smithi148 to 4280M 2024-04-23T16:13:56.634 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:56 smithi188 ceph-mon[44101]: osdmap e59: 8 total, 8 up, 8 in 2024-04-23T16:13:56.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:56 smithi103 ceph-mon[39629]: Detected new or changed devices on smithi148 2024-04-23T16:13:56.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:56 smithi103 ceph-mon[39629]: Adjusting osd_memory_target on smithi148 to 4280M 2024-04-23T16:13:56.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:56 smithi103 ceph-mon[39629]: osdmap e59: 8 total, 8 up, 8 in 2024-04-23T16:13:56.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:56 smithi148 ceph-mon[44520]: Detected new or changed devices on smithi148 2024-04-23T16:13:56.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:56 smithi148 ceph-mon[44520]: Adjusting osd_memory_target on smithi148 to 4280M 2024-04-23T16:13:56.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:56 smithi148 ceph-mon[44520]: osdmap e59: 8 total, 8 up, 8 in 2024-04-23T16:13:57.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:57 smithi103 ceph-mon[39629]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-04-23T16:13:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:57 smithi103 ceph-mon[39629]: osdmap e60: 8 total, 8 up, 8 in 2024-04-23T16:13:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:13:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:13:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:57.856 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:57 smithi188 ceph-mon[44101]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-04-23T16:13:57.856 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:57 smithi188 ceph-mon[44101]: osdmap e60: 8 total, 8 up, 8 in 2024-04-23T16:13:57.856 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:57 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:13:57.856 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:57 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:13:57.856 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:57 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:57.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:57 smithi148 ceph-mon[44520]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-04-23T16:13:57.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:57 smithi148 ceph-mon[44520]: osdmap e60: 8 total, 8 up, 8 in 2024-04-23T16:13:57.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:57 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:13:57.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:57 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:13:57.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:57 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:13:58.708 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:58 smithi188 ceph-mon[44101]: from='client.24331 v1:172.21.15.188:0/2081088088' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:13:58.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:58 smithi103 ceph-mon[39629]: from='client.24331 v1:172.21.15.188:0/2081088088' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:13:58.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:58 smithi148 ceph-mon[44520]: from='client.24331 v1:172.21.15.188:0/2081088088' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:13:59.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:13:59 smithi103 ceph-mon[39629]: pgmap v192: 1 pgs: 1 remapped+peering; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:13:59.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:13:59 smithi148 ceph-mon[44520]: pgmap v192: 1 pgs: 1 remapped+peering; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:13:59.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:13:59 smithi188 ceph-mon[44101]: pgmap v192: 1 pgs: 1 remapped+peering; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:00 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.188:0/1393906350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed"}]: dispatch 2024-04-23T16:14:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:00 smithi103 ceph-mon[39629]: from='client.24337 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed"}]: dispatch 2024-04-23T16:14:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:00 smithi103 ceph-mon[39629]: from='client.24337 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed"}]': finished 2024-04-23T16:14:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:00 smithi103 ceph-mon[39629]: osdmap e61: 9 total, 8 up, 9 in 2024-04-23T16:14:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:00 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:00 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.188:0/3250177988' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:14:00.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:00 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.188:0/1393906350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed"}]: dispatch 2024-04-23T16:14:00.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:00 smithi148 ceph-mon[44520]: from='client.24337 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed"}]: dispatch 2024-04-23T16:14:00.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:00 smithi148 ceph-mon[44520]: from='client.24337 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed"}]': finished 2024-04-23T16:14:00.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:00 smithi148 ceph-mon[44520]: osdmap e61: 9 total, 8 up, 9 in 2024-04-23T16:14:00.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:00 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:00.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:00 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.188:0/3250177988' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:14:00.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:00 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.188:0/1393906350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed"}]: dispatch 2024-04-23T16:14:00.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:00 smithi188 ceph-mon[44101]: from='client.24337 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed"}]: dispatch 2024-04-23T16:14:00.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:00 smithi188 ceph-mon[44101]: from='client.24337 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed"}]': finished 2024-04-23T16:14:00.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:00 smithi188 ceph-mon[44101]: osdmap e61: 9 total, 8 up, 9 in 2024-04-23T16:14:00.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:00 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:00.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:00 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.188:0/3250177988' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:14:01.785 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:01 smithi103 ceph-mon[39629]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:01.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:01 smithi148 ceph-mon[44520]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:01.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:01 smithi188 ceph-mon[44101]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:02 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:14:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:02 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:14:02.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:02 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:14:02.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:02 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:14:02.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:02 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:14:02.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:02 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:14:03.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:03 smithi188 ceph-mon[44101]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:03.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:03 smithi188 ceph-mon[44101]: osdmap e62: 9 total, 8 up, 9 in 2024-04-23T16:14:03.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:03 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:03 smithi103 ceph-mon[39629]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:03 smithi103 ceph-mon[39629]: osdmap e62: 9 total, 8 up, 9 in 2024-04-23T16:14:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:03 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:03.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:03 smithi148 ceph-mon[44520]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:03.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:03 smithi148 ceph-mon[44520]: osdmap e62: 9 total, 8 up, 9 in 2024-04-23T16:14:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:03 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:05.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:04 smithi188 ceph-mon[44101]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:05.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:04 smithi103 ceph-mon[39629]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:05.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:04 smithi148 ceph-mon[44520]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:06.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:06 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-23T16:14:06.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:06 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:06.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:06 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-23T16:14:06.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:06 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:06.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:06 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-23T16:14:06.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:06 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:07.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:07 smithi188 ceph-mon[44101]: Deploying daemon osd.8 on smithi188 2024-04-23T16:14:07.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:07 smithi188 ceph-mon[44101]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:07.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:07 smithi103 ceph-mon[39629]: Deploying daemon osd.8 on smithi188 2024-04-23T16:14:07.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:07 smithi103 ceph-mon[39629]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:07.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:07 smithi148 ceph-mon[44520]: Deploying daemon osd.8 on smithi188 2024-04-23T16:14:07.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:07 smithi148 ceph-mon[44520]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:09.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:09 smithi188 ceph-mon[44101]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:09.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:09 smithi103 ceph-mon[39629]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:09.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:09 smithi148 ceph-mon[44520]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:10.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:10 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:14:10.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:10 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:14:10.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:10 smithi188 ceph-mon[44101]: osdmap e63: 9 total, 8 up, 9 in 2024-04-23T16:14:10.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:10 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:10.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:10 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:14:10.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:10 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:14:10.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:10 smithi103 ceph-mon[39629]: osdmap e63: 9 total, 8 up, 9 in 2024-04-23T16:14:10.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:10 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:10.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:10 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:14:10.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:10 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:14:10.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:10 smithi148 ceph-mon[44520]: osdmap e63: 9 total, 8 up, 9 in 2024-04-23T16:14:10.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:10 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:11.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:11 smithi188 ceph-mon[44101]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:11.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:11 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:11.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:11 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:14:11.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:11 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:11.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:11 smithi103 ceph-mon[39629]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:11 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:11 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:14:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:11 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:11.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:11 smithi148 ceph-mon[44520]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:11 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:11 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:14:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:11 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:13.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:13 smithi188 ceph-mon[44101]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:13.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:13.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:13.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:13.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:14:13.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:13 smithi103 ceph-mon[39629]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:14:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:13 smithi148 ceph-mon[44520]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:14:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:14.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:14 smithi188 ceph-mon[44101]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:14.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:14 smithi188 ceph-mon[44101]: from='osd.8 v1:172.21.15.188:6800/1151408595' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T16:14:14.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:14 smithi188 ceph-mon[44101]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T16:14:14.985 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 8 on host 'smithi188' 2024-04-23T16:14:15.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:14 smithi103 ceph-mon[39629]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:15.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:14 smithi103 ceph-mon[39629]: from='osd.8 v1:172.21.15.188:6800/1151408595' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T16:14:15.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:14 smithi103 ceph-mon[39629]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T16:14:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:14 smithi148 ceph-mon[44520]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:14 smithi148 ceph-mon[44520]: from='osd.8 v1:172.21.15.188:6800/1151408595' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T16:14:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:14 smithi148 ceph-mon[44520]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T16:14:15.723 DEBUG:teuthology.orchestra.run.smithi188:osd.8> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.8.service 2024-04-23T16:14:15.726 INFO:tasks.cephadm:Deploying osd.9 on smithi188 with /dev/vg_nvme/lv_3... 2024-04-23T16:14:15.726 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T16:14:15.758 INFO:journalctl@ceph.osd.8.smithi188.stdout:-- Logs begin at Tue 2024-04-23 15:57:06 UTC. -- 2024-04-23T16:14:15.937 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:15 smithi188 ceph-mon[44101]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-23T16:14:16.323 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:15 smithi188 ceph-mon[44101]: from='osd.8 v1:172.21.15.188:6800/1151408595' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:14:16.323 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:15 smithi188 ceph-mon[44101]: osdmap e64: 9 total, 8 up, 9 in 2024-04-23T16:14:16.323 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:16.323 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:15 smithi188 ceph-mon[44101]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:14:16.323 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:16.323 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:16.323 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:14:16.324 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:15 smithi148 ceph-mon[44520]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-23T16:14:16.324 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:15 smithi148 ceph-mon[44520]: from='osd.8 v1:172.21.15.188:6800/1151408595' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:14:16.325 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:15 smithi148 ceph-mon[44520]: osdmap e64: 9 total, 8 up, 9 in 2024-04-23T16:14:16.325 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:15 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:16.325 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:15 smithi148 ceph-mon[44520]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:14:16.325 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:15 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:16.325 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:15 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:16.325 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:15 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:14:16.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:15 smithi103 ceph-mon[39629]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-23T16:14:16.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:15 smithi103 ceph-mon[39629]: from='osd.8 v1:172.21.15.188:6800/1151408595' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:14:16.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:15 smithi103 ceph-mon[39629]: osdmap e64: 9 total, 8 up, 9 in 2024-04-23T16:14:16.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:16.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:15 smithi103 ceph-mon[39629]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:14:16.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:16.326 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:16.326 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:14:16.326 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 16:14:16 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T16:14:16.176+0000 7fe9a446c700 -1 osd.8 0 waiting for initial osdmap 2024-04-23T16:14:16.326 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 16:14:16 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T16:14:16.177+0000 7fe997be2700 -1 osd.8 0 failed to load OSD map for epoch 11, got 0 bytes 2024-04-23T16:14:16.326 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 16:14:16 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T16:14:16.186+0000 7fe99c257700 -1 osd.8 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:14:16.760 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:14:17.002 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:16 smithi188 ceph-mon[44101]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T16:14:17.002 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:16 smithi188 ceph-mon[44101]: osdmap e65: 9 total, 8 up, 9 in 2024-04-23T16:14:17.002 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:17.002 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:17.002 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:16 smithi188 ceph-mon[44101]: from='osd.8 ' entity='osd.8' 2024-04-23T16:14:17.002 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:16 smithi188 ceph-mon[44101]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:17.003 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:17.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:16 smithi103 ceph-mon[39629]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T16:14:17.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:16 smithi103 ceph-mon[39629]: osdmap e65: 9 total, 8 up, 9 in 2024-04-23T16:14:17.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:17.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:17.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:16 smithi103 ceph-mon[39629]: from='osd.8 ' entity='osd.8' 2024-04-23T16:14:17.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:16 smithi103 ceph-mon[39629]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:17.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:17.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:16 smithi148 ceph-mon[44520]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T16:14:17.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:16 smithi148 ceph-mon[44520]: osdmap e65: 9 total, 8 up, 9 in 2024-04-23T16:14:17.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:17.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:17.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:16 smithi148 ceph-mon[44520]: from='osd.8 ' entity='osd.8' 2024-04-23T16:14:17.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:16 smithi148 ceph-mon[44520]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-23T16:14:17.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:18 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:14:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:18 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:14:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:14:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:14:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:18 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:14:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:18 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:14:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:18 smithi188 ceph-mon[44101]: osd.8 v1:172.21.15.188:6800/1151408595 boot 2024-04-23T16:14:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:18 smithi188 ceph-mon[44101]: osdmap e66: 9 total, 9 up, 9 in 2024-04-23T16:14:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:18.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:18 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:14:18.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:18 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:14:18.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:14:18.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:14:18.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:18 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:14:18.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:18 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:14:18.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:18 smithi103 ceph-mon[39629]: osd.8 v1:172.21.15.188:6800/1151408595 boot 2024-04-23T16:14:18.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:18 smithi103 ceph-mon[39629]: osdmap e66: 9 total, 9 up, 9 in 2024-04-23T16:14:18.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:18.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:18 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:14:18.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:18 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:14:18.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:14:18.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:14:18.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:18 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:14:18.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:18 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:14:18.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:18 smithi148 ceph-mon[44520]: osd.8 v1:172.21.15.188:6800/1151408595 boot 2024-04-23T16:14:18.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:18 smithi148 ceph-mon[44520]: osdmap e66: 9 total, 9 up, 9 in 2024-04-23T16:14:18.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T16:14:19.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:19 smithi188 ceph-mon[44101]: osdmap e67: 9 total, 9 up, 9 in 2024-04-23T16:14:19.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:19 smithi188 ceph-mon[44101]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:19.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:19 smithi103 ceph-mon[39629]: osdmap e67: 9 total, 9 up, 9 in 2024-04-23T16:14:19.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:19 smithi103 ceph-mon[39629]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:19.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:19 smithi148 ceph-mon[44520]: osdmap e67: 9 total, 9 up, 9 in 2024-04-23T16:14:19.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:19 smithi148 ceph-mon[44520]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:19.724 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T16:14:19.724 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T16:14:19.725 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-23T16:14:19.725 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-23T16:14:19.725 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110636 s, 948 MB/s 2024-04-23T16:14:19.725 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-23T16:14:20.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:20 smithi188 ceph-mon[44101]: osdmap e68: 9 total, 9 up, 9 in 2024-04-23T16:14:20.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:20 smithi188 ceph-mon[44101]: Detected new or changed devices on smithi188 2024-04-23T16:14:20.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:20 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:20.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:20 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:20.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:20 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:14:20.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:20 smithi188 ceph-mon[44101]: Adjusting osd_memory_target on smithi188 to 21218M 2024-04-23T16:14:20.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:20 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:20.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:20 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:20.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:20 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:14:20.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:20 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:20.367 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_3 2024-04-23T16:14:20.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:20 smithi103 ceph-mon[39629]: osdmap e68: 9 total, 9 up, 9 in 2024-04-23T16:14:20.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:20 smithi103 ceph-mon[39629]: Detected new or changed devices on smithi188 2024-04-23T16:14:20.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:20 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:20.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:20 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:20.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:20 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:14:20.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:20 smithi103 ceph-mon[39629]: Adjusting osd_memory_target on smithi188 to 21218M 2024-04-23T16:14:20.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:20 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:20.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:20 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:20.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:20 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:14:20.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:20 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:20.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:20 smithi148 ceph-mon[44520]: osdmap e68: 9 total, 9 up, 9 in 2024-04-23T16:14:20.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:20 smithi148 ceph-mon[44520]: Detected new or changed devices on smithi188 2024-04-23T16:14:20.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:20 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:20.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:20 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:20.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:20 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:14:20.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:20 smithi148 ceph-mon[44520]: Adjusting osd_memory_target on smithi188 to 21218M 2024-04-23T16:14:20.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:20 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:20.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:20 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:20.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:20 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:14:20.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:20 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:20.628 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:14:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:21 smithi188 ceph-mon[44101]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-23T16:14:21.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:21 smithi103 ceph-mon[39629]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-23T16:14:21.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:21 smithi148 ceph-mon[44520]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-23T16:14:22.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:22 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:14:22.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:22 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:14:22.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:22 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:22.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:22 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:14:22.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:22 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:14:22.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:22 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:22.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:22 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:14:22.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:22 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:14:22.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:22 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:23.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:23 smithi103 ceph-mon[39629]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-23T16:14:23.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:23 smithi103 ceph-mon[39629]: from='client.24358 v1:172.21.15.188:0/3016294346' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:14:23.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:23 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:14:23.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:23 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:14:23.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:23 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:14:23.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:23 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:14:23.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:23 smithi148 ceph-mon[44520]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-23T16:14:23.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:23 smithi148 ceph-mon[44520]: from='client.24358 v1:172.21.15.188:0/3016294346' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:14:23.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:23 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:14:23.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:23 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:14:23.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:23 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:14:23.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:23 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:14:23.876 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:23 smithi188 ceph-mon[44101]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-23T16:14:23.876 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:23 smithi188 ceph-mon[44101]: from='client.24358 v1:172.21.15.188:0/3016294346' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:14:23.876 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:23 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:14:23.876 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:23 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:14:23.876 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:23 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:14:23.876 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:23 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:14:24.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:24 smithi103 ceph-mon[39629]: osdmap e69: 9 total, 9 up, 9 in 2024-04-23T16:14:24.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:24 smithi148 ceph-mon[44520]: osdmap e69: 9 total, 9 up, 9 in 2024-04-23T16:14:24.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:24 smithi188 ceph-mon[44101]: osdmap e69: 9 total, 9 up, 9 in 2024-04-23T16:14:25.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:25 smithi103 ceph-mon[39629]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-23T16:14:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:25 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.188:0/1610293773' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "124d9b05-11eb-4622-b6a2-c4d5bd60b7e6"}]: dispatch 2024-04-23T16:14:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:25 smithi103 ceph-mon[39629]: from='client.24364 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "124d9b05-11eb-4622-b6a2-c4d5bd60b7e6"}]: dispatch 2024-04-23T16:14:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:25 smithi103 ceph-mon[39629]: from='client.24364 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "124d9b05-11eb-4622-b6a2-c4d5bd60b7e6"}]': finished 2024-04-23T16:14:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:25 smithi103 ceph-mon[39629]: osdmap e70: 10 total, 9 up, 10 in 2024-04-23T16:14:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T16:14:25.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:25 smithi148 ceph-mon[44520]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-23T16:14:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:25 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.188:0/1610293773' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "124d9b05-11eb-4622-b6a2-c4d5bd60b7e6"}]: dispatch 2024-04-23T16:14:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:25 smithi148 ceph-mon[44520]: from='client.24364 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "124d9b05-11eb-4622-b6a2-c4d5bd60b7e6"}]: dispatch 2024-04-23T16:14:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:25 smithi148 ceph-mon[44520]: from='client.24364 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "124d9b05-11eb-4622-b6a2-c4d5bd60b7e6"}]': finished 2024-04-23T16:14:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:25 smithi148 ceph-mon[44520]: osdmap e70: 10 total, 9 up, 10 in 2024-04-23T16:14:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T16:14:25.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:25 smithi188 ceph-mon[44101]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-23T16:14:25.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:25 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.188:0/1610293773' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "124d9b05-11eb-4622-b6a2-c4d5bd60b7e6"}]: dispatch 2024-04-23T16:14:25.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:25 smithi188 ceph-mon[44101]: from='client.24364 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "124d9b05-11eb-4622-b6a2-c4d5bd60b7e6"}]: dispatch 2024-04-23T16:14:25.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:25 smithi188 ceph-mon[44101]: from='client.24364 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "124d9b05-11eb-4622-b6a2-c4d5bd60b7e6"}]': finished 2024-04-23T16:14:25.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:25 smithi188 ceph-mon[44101]: osdmap e70: 10 total, 9 up, 10 in 2024-04-23T16:14:25.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T16:14:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:26 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.188:0/1938571264' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:14:26.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:26 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.188:0/1938571264' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:14:26.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:26 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.188:0/1938571264' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:14:27.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:27 smithi103 ceph-mon[39629]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T16:14:27.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:27 smithi148 ceph-mon[44520]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T16:14:27.950 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:27 smithi188 ceph-mon[44101]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T16:14:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:29 smithi103 ceph-mon[39629]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:29.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:29 smithi148 ceph-mon[44520]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:29 smithi188 ceph-mon[44101]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:31.778 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:31 smithi188 ceph-mon[44101]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:31.779 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:31 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-23T16:14:31.779 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:31 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:31.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:31 smithi103 ceph-mon[39629]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:31.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:31 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-23T16:14:31.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:31 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:31.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:31 smithi148 ceph-mon[44520]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:31 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-23T16:14:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:31 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:32.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:32 smithi188 ceph-mon[44101]: Deploying daemon osd.9 on smithi188 2024-04-23T16:14:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:32 smithi103 ceph-mon[39629]: Deploying daemon osd.9 on smithi188 2024-04-23T16:14:32.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:32 smithi148 ceph-mon[44520]: Deploying daemon osd.9 on smithi188 2024-04-23T16:14:33.673 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:33 smithi188 ceph-mon[44101]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:33 smithi103 ceph-mon[39629]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:33.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:33 smithi148 ceph-mon[44520]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:34.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:34 smithi188 ceph-mon[44101]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:34 smithi103 ceph-mon[39629]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:35.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:34 smithi148 ceph-mon[44520]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:37 smithi188 ceph-mon[44101]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:37 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:37 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:37 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:14:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:37 smithi103 ceph-mon[39629]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:37 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:37 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:37 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:14:37.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:37 smithi148 ceph-mon[44520]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:37.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:37 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:37.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:37 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:37.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:37 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:14:39.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:39 smithi188 ceph-mon[44101]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:39.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:39 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:39.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:39 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:39 smithi103 ceph-mon[39629]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:39 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:39 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:39.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:39 smithi148 ceph-mon[44520]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:39.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:39 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:39.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:39 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:40.343 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 9 on host 'smithi188' 2024-04-23T16:14:40.707 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:40 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:40.707 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:40 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:14:40.707 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:40 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:40.707 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:40 smithi188 ceph-mon[44101]: from='osd.9 v1:172.21.15.188:6804/306501862' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T16:14:40.707 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:40 smithi188 ceph-mon[44101]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T16:14:40.707 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:40 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:40.707 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:40 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:40.707 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:40 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:14:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:40 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:40 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:14:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:40 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:40 smithi103 ceph-mon[39629]: from='osd.9 v1:172.21.15.188:6804/306501862' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T16:14:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:40 smithi103 ceph-mon[39629]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T16:14:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:40 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:40 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:40 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:14:40.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:40 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:40.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:40 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:14:40.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:40 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:40.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:40 smithi148 ceph-mon[44520]: from='osd.9 v1:172.21.15.188:6804/306501862' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T16:14:40.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:40 smithi148 ceph-mon[44520]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T16:14:40.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:40 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:40.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:40 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:40.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:40 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:14:41.038 DEBUG:teuthology.orchestra.run.smithi188:osd.9> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.9.service 2024-04-23T16:14:41.041 INFO:tasks.cephadm:Deploying osd.10 on smithi188 with /dev/vg_nvme/lv_2... 2024-04-23T16:14:41.041 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T16:14:41.072 INFO:journalctl@ceph.osd.9.smithi188.stdout:-- Logs begin at Tue 2024-04-23 15:57:06 UTC. -- 2024-04-23T16:14:41.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:41 smithi188 ceph-mon[44101]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:41.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:41 smithi188 ceph-mon[44101]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-23T16:14:41.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:41 smithi188 ceph-mon[44101]: from='osd.9 v1:172.21.15.188:6804/306501862' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:14:41.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:41 smithi188 ceph-mon[44101]: osdmap e71: 10 total, 9 up, 10 in 2024-04-23T16:14:41.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:41 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T16:14:41.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:41 smithi188 ceph-mon[44101]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:14:41.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:41 smithi103 ceph-mon[39629]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:41.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:41 smithi103 ceph-mon[39629]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-23T16:14:41.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:41 smithi103 ceph-mon[39629]: from='osd.9 v1:172.21.15.188:6804/306501862' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:14:41.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:41 smithi103 ceph-mon[39629]: osdmap e71: 10 total, 9 up, 10 in 2024-04-23T16:14:41.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:41 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T16:14:41.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:41 smithi103 ceph-mon[39629]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:14:41.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:41 smithi148 ceph-mon[44520]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:41.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:41 smithi148 ceph-mon[44520]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-23T16:14:41.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:41 smithi148 ceph-mon[44520]: from='osd.9 v1:172.21.15.188:6804/306501862' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:14:41.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:41 smithi148 ceph-mon[44520]: osdmap e71: 10 total, 9 up, 10 in 2024-04-23T16:14:41.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:41 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T16:14:41.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:41 smithi148 ceph-mon[44520]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:14:41.877 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:14:42.386 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 16:14:42 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T16:14:42.068+0000 7efcd4fcf700 -1 osd.9 0 waiting for initial osdmap 2024-04-23T16:14:42.386 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 16:14:42 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T16:14:42.069+0000 7efcc8745700 -1 osd.9 0 failed to load OSD map for epoch 11, got 0 bytes 2024-04-23T16:14:42.387 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 16:14:42 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T16:14:42.077+0000 7efcccdba700 -1 osd.9 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:14:42.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:42 smithi188 ceph-mon[44101]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T16:14:42.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:42 smithi188 ceph-mon[44101]: osdmap e72: 10 total, 9 up, 10 in 2024-04-23T16:14:42.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:42 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T16:14:42.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:42 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T16:14:42.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:42 smithi188 ceph-mon[44101]: from='osd.9 ' entity='osd.9' 2024-04-23T16:14:42.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:42 smithi188 ceph-mon[44101]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:42.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:42 smithi103 ceph-mon[39629]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T16:14:42.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:42 smithi103 ceph-mon[39629]: osdmap e72: 10 total, 9 up, 10 in 2024-04-23T16:14:42.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:42 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T16:14:42.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:42 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T16:14:42.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:42 smithi103 ceph-mon[39629]: from='osd.9 ' entity='osd.9' 2024-04-23T16:14:42.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:42 smithi103 ceph-mon[39629]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:42.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:42 smithi148 ceph-mon[44520]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T16:14:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:42 smithi148 ceph-mon[44520]: osdmap e72: 10 total, 9 up, 10 in 2024-04-23T16:14:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:42 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T16:14:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:42 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T16:14:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:42 smithi148 ceph-mon[44520]: from='osd.9 ' entity='osd.9' 2024-04-23T16:14:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:42 smithi148 ceph-mon[44520]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-23T16:14:43.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:43 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:14:43.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:43 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:14:43.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:14:43.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:14:43.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:43 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T16:14:43.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:43 smithi148 ceph-mon[44520]: osd.9 v1:172.21.15.188:6804/306501862 boot 2024-04-23T16:14:43.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:43 smithi148 ceph-mon[44520]: osdmap e73: 10 total, 10 up, 10 in 2024-04-23T16:14:43.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:43 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T16:14:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:43 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:14:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:43 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:14:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:14:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:14:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:43 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T16:14:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:43 smithi188 ceph-mon[44101]: osd.9 v1:172.21.15.188:6804/306501862 boot 2024-04-23T16:14:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:43 smithi188 ceph-mon[44101]: osdmap e73: 10 total, 10 up, 10 in 2024-04-23T16:14:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:43 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T16:14:44.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:43 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:14:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:43 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:14:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:14:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:14:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:43 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T16:14:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:43 smithi103 ceph-mon[39629]: osd.9 v1:172.21.15.188:6804/306501862 boot 2024-04-23T16:14:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:43 smithi103 ceph-mon[39629]: osdmap e73: 10 total, 10 up, 10 in 2024-04-23T16:14:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:43 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T16:14:44.270 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T16:14:44.270 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T16:14:44.270 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-23T16:14:44.270 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-23T16:14:44.270 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124442 s, 843 MB/s 2024-04-23T16:14:44.270 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-23T16:14:45.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:44 smithi103 ceph-mon[39629]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:45.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:44 smithi148 ceph-mon[44520]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:45.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:44 smithi188 ceph-mon[44101]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:45.202 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_2 2024-04-23T16:14:45.800 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:14:45.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:45 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:14:45.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:45 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:14:45.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:45 smithi188 ceph-mon[44101]: osdmap e74: 10 total, 10 up, 10 in 2024-04-23T16:14:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:45 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:14:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:45 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:14:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:45 smithi103 ceph-mon[39629]: osdmap e74: 10 total, 10 up, 10 in 2024-04-23T16:14:46.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:45 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:14:46.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:45 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:14:46.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:45 smithi148 ceph-mon[44520]: osdmap e74: 10 total, 10 up, 10 in 2024-04-23T16:14:46.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:46 smithi188 ceph-mon[44101]: Detected new or changed devices on smithi188 2024-04-23T16:14:47.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:46 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:47.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:46 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:47.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:46 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:14:47.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:46 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:14:47.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:46 smithi188 ceph-mon[44101]: Adjusting osd_memory_target on smithi188 to 10609M 2024-04-23T16:14:47.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:46 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:47.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:46 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:47.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:46 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:14:47.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:46 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:47.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:46 smithi188 ceph-mon[44101]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:47.543 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:46 smithi148 ceph-mon[44520]: Detected new or changed devices on smithi188 2024-04-23T16:14:47.543 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:46 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:47.543 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:46 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:47.543 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:46 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:14:47.543 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:46 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:14:47.543 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:46 smithi148 ceph-mon[44520]: Adjusting osd_memory_target on smithi188 to 10609M 2024-04-23T16:14:47.543 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:46 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:47.543 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:46 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:47.543 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:46 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:14:47.543 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:46 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:47.543 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:46 smithi148 ceph-mon[44520]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:47.544 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:46 smithi103 ceph-mon[39629]: Detected new or changed devices on smithi188 2024-04-23T16:14:47.544 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:46 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:47.544 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:46 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:47.544 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:46 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:14:47.544 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:46 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:14:47.544 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:46 smithi103 ceph-mon[39629]: Adjusting osd_memory_target on smithi188 to 10609M 2024-04-23T16:14:47.544 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:46 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:47.544 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:46 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:47.544 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:46 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:14:47.544 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:46 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:14:47.544 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:46 smithi103 ceph-mon[39629]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:48.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:47 smithi148 ceph-mon[44520]: from='client.24385 v1:172.21.15.188:0/3282953521' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:14:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:47 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:14:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:47 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:14:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:47 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:47 smithi188 ceph-mon[44101]: from='client.24385 v1:172.21.15.188:0/3282953521' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:14:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:47 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:14:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:47 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:14:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:47 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:48.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:47 smithi103 ceph-mon[39629]: from='client.24385 v1:172.21.15.188:0/3282953521' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:14:48.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:47 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:14:48.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:47 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:14:48.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:47 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:49.115 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:48 smithi188 ceph-mon[44101]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:49.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:48 smithi148 ceph-mon[44520]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:49.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:48 smithi103 ceph-mon[39629]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:50.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:49 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:14:50.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:49 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:14:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:49 smithi148 ceph-mon[44520]: osdmap e75: 10 total, 10 up, 10 in 2024-04-23T16:14:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:49 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:14:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:49 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:14:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:49 smithi188 ceph-mon[44101]: osdmap e75: 10 total, 10 up, 10 in 2024-04-23T16:14:50.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:49 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:14:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:49 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:14:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:49 smithi103 ceph-mon[39629]: osdmap e75: 10 total, 10 up, 10 in 2024-04-23T16:14:51.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:50 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.188:0/2444381658' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3dfb9070-e534-4ea4-a25e-ee1c51297447"}]: dispatch 2024-04-23T16:14:51.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:50 smithi148 ceph-mon[44520]: from='client.24391 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3dfb9070-e534-4ea4-a25e-ee1c51297447"}]: dispatch 2024-04-23T16:14:51.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:50 smithi148 ceph-mon[44520]: from='client.24391 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3dfb9070-e534-4ea4-a25e-ee1c51297447"}]': finished 2024-04-23T16:14:51.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:50 smithi148 ceph-mon[44520]: osdmap e76: 11 total, 10 up, 11 in 2024-04-23T16:14:51.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:50 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:14:51.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:50 smithi148 ceph-mon[44520]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:51.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:50 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.188:0/2444381658' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3dfb9070-e534-4ea4-a25e-ee1c51297447"}]: dispatch 2024-04-23T16:14:51.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:50 smithi188 ceph-mon[44101]: from='client.24391 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3dfb9070-e534-4ea4-a25e-ee1c51297447"}]: dispatch 2024-04-23T16:14:51.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:50 smithi188 ceph-mon[44101]: from='client.24391 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3dfb9070-e534-4ea4-a25e-ee1c51297447"}]': finished 2024-04-23T16:14:51.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:50 smithi188 ceph-mon[44101]: osdmap e76: 11 total, 10 up, 11 in 2024-04-23T16:14:51.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:50 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:14:51.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:50 smithi188 ceph-mon[44101]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:51.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:50 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.188:0/2444381658' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3dfb9070-e534-4ea4-a25e-ee1c51297447"}]: dispatch 2024-04-23T16:14:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:50 smithi103 ceph-mon[39629]: from='client.24391 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3dfb9070-e534-4ea4-a25e-ee1c51297447"}]: dispatch 2024-04-23T16:14:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:50 smithi103 ceph-mon[39629]: from='client.24391 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3dfb9070-e534-4ea4-a25e-ee1c51297447"}]': finished 2024-04-23T16:14:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:50 smithi103 ceph-mon[39629]: osdmap e76: 11 total, 10 up, 11 in 2024-04-23T16:14:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:50 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:14:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:50 smithi103 ceph-mon[39629]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:52.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:51 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.188:0/1367121676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:14:52.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:51 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.188:0/1367121676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:14:52.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:51 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.188:0/1367121676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:14:53.185 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:52 smithi188 ceph-mon[44101]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:53.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:52 smithi103 ceph-mon[39629]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:53.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:52 smithi148 ceph-mon[44520]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:54.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:54 smithi188 ceph-mon[44101]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:55.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:54 smithi103 ceph-mon[39629]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:55.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:54 smithi148 ceph-mon[44520]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:57.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:57 smithi188 ceph-mon[44101]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:57.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:57 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-23T16:14:57.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:57 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:57.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:57 smithi103 ceph-mon[39629]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-23T16:14:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:57.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:57 smithi148 ceph-mon[44520]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:57.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:57 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-23T16:14:57.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:57 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:14:58.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:58 smithi188 ceph-mon[44101]: Deploying daemon osd.10 on smithi188 2024-04-23T16:14:58.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:58 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:14:58.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:58 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:14:58.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:58 smithi188 ceph-mon[44101]: osdmap e77: 11 total, 10 up, 11 in 2024-04-23T16:14:58.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:58 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:14:58.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:58 smithi103 ceph-mon[39629]: Deploying daemon osd.10 on smithi188 2024-04-23T16:14:58.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:58 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:14:58.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:58 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:14:58.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:58 smithi103 ceph-mon[39629]: osdmap e77: 11 total, 10 up, 11 in 2024-04-23T16:14:58.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:58 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:14:58.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:58 smithi148 ceph-mon[44520]: Deploying daemon osd.10 on smithi188 2024-04-23T16:14:58.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:58 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:14:58.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:58 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:14:58.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:58 smithi148 ceph-mon[44520]: osdmap e77: 11 total, 10 up, 11 in 2024-04-23T16:14:58.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:58 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:14:59.577 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:59 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:14:59.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:59 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:14:59.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:59 smithi188 ceph-mon[44101]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:59.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:59 smithi188 ceph-mon[44101]: osdmap e78: 11 total, 10 up, 11 in 2024-04-23T16:14:59.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:14:59 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:14:59.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:59 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:14:59.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:59 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:14:59.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:59 smithi103 ceph-mon[39629]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:59.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:59 smithi103 ceph-mon[39629]: osdmap e78: 11 total, 10 up, 11 in 2024-04-23T16:14:59.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:14:59 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:14:59.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:59 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:14:59.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:59 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:14:59.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:59 smithi148 ceph-mon[44520]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:14:59.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:59 smithi148 ceph-mon[44520]: osdmap e78: 11 total, 10 up, 11 in 2024-04-23T16:14:59.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:14:59 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:15:01.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:01 smithi188 ceph-mon[44101]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:15:01.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:01 smithi103 ceph-mon[39629]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:15:01.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:01 smithi148 ceph-mon[44520]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:15:03.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:03 smithi188 ceph-mon[44101]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:15:03.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:03 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:03.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:03 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:03.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:03 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:15:03.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:03 smithi103 ceph-mon[39629]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:15:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:03 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:03 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:03 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:15:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:03 smithi148 ceph-mon[44520]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:15:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:03 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:03 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:03 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:15:05.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:05 smithi188 ceph-mon[44101]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:15:05.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:05 smithi103 ceph-mon[39629]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:15:05.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:05 smithi148 ceph-mon[44520]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:15:06.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:06 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:06.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:06 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:06.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:06 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:15:06.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:06 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:15:06.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:06 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:06.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:06 smithi188 ceph-mon[44101]: from='osd.10 v1:172.21.15.188:6808/1231263138' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T16:15:06.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:06 smithi188 ceph-mon[44101]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T16:15:06.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:06 smithi188 ceph-mon[44101]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:15:06.842 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 10 on host 'smithi188' 2024-04-23T16:15:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:06 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:06 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:06 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:15:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:06 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:15:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:06 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:06 smithi103 ceph-mon[39629]: from='osd.10 v1:172.21.15.188:6808/1231263138' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T16:15:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:06 smithi103 ceph-mon[39629]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T16:15:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:06 smithi103 ceph-mon[39629]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:15:07.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:06 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:07.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:06 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:06 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:15:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:06 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:15:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:06 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:06 smithi148 ceph-mon[44520]: from='osd.10 v1:172.21.15.188:6808/1231263138' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T16:15:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:06 smithi148 ceph-mon[44520]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T16:15:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:06 smithi148 ceph-mon[44520]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:15:07.510 DEBUG:teuthology.orchestra.run.smithi188:osd.10> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.10.service 2024-04-23T16:15:07.512 INFO:tasks.cephadm:Deploying osd.11 on smithi188 with /dev/vg_nvme/lv_1... 2024-04-23T16:15:07.512 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T16:15:07.545 INFO:journalctl@ceph.osd.10.smithi188.stdout:-- Logs begin at Tue 2024-04-23 15:57:06 UTC. -- 2024-04-23T16:15:07.758 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 16:15:07 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T16:15:07.667+0000 7f3dd4cdc700 -1 osd.10 0 waiting for initial osdmap 2024-04-23T16:15:07.758 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 16:15:07 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T16:15:07.667+0000 7f3dcc45a700 -1 osd.10 0 failed to load OSD map for epoch 22, got 0 bytes 2024-04-23T16:15:07.758 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 16:15:07 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T16:15:07.675+0000 7f3dd0acf700 -1 osd.10 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:15:07.759 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:07 smithi188 ceph-mon[44101]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-23T16:15:07.759 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:07 smithi188 ceph-mon[44101]: osdmap e79: 11 total, 10 up, 11 in 2024-04-23T16:15:07.759 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:15:07.759 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:07 smithi188 ceph-mon[44101]: from='osd.10 v1:172.21.15.188:6808/1231263138' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:15:07.759 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:07 smithi188 ceph-mon[44101]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:15:07.759 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:07.759 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:07.759 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:15:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:07 smithi103 ceph-mon[39629]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-23T16:15:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:07 smithi103 ceph-mon[39629]: osdmap e79: 11 total, 10 up, 11 in 2024-04-23T16:15:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:15:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:07 smithi103 ceph-mon[39629]: from='osd.10 v1:172.21.15.188:6808/1231263138' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:15:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:07 smithi103 ceph-mon[39629]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:15:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:15:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:07 smithi148 ceph-mon[44520]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-23T16:15:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:07 smithi148 ceph-mon[44520]: osdmap e79: 11 total, 10 up, 11 in 2024-04-23T16:15:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:07 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:15:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:07 smithi148 ceph-mon[44520]: from='osd.10 v1:172.21.15.188:6808/1231263138' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:15:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:07 smithi148 ceph-mon[44520]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:15:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:07 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:07 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:07 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:15:08.316 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:15:08.814 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:08 smithi188 ceph-mon[44101]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T16:15:08.814 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:08 smithi188 ceph-mon[44101]: osdmap e80: 11 total, 10 up, 11 in 2024-04-23T16:15:08.814 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:08 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:15:08.814 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:08 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:15:08.814 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:08 smithi188 ceph-mon[44101]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:15:09.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:08 smithi103 ceph-mon[39629]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T16:15:09.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:08 smithi103 ceph-mon[39629]: osdmap e80: 11 total, 10 up, 11 in 2024-04-23T16:15:09.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:08 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:15:09.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:08 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:15:09.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:08 smithi103 ceph-mon[39629]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:15:09.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:08 smithi148 ceph-mon[44520]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T16:15:09.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:08 smithi148 ceph-mon[44520]: osdmap e80: 11 total, 10 up, 11 in 2024-04-23T16:15:09.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:08 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:15:09.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:08 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:15:09.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:08 smithi148 ceph-mon[44520]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-23T16:15:09.882 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:09 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:15:09.882 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:09 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:15:09.882 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:15:09.882 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:09 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:15:09.882 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:09 smithi188 ceph-mon[44101]: osd.10 v1:172.21.15.188:6808/1231263138 boot 2024-04-23T16:15:09.882 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:09 smithi188 ceph-mon[44101]: osdmap e81: 11 total, 11 up, 11 in 2024-04-23T16:15:09.883 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:09 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:15:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:09 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:15:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:09 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:15:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:15:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:09 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:15:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:09 smithi103 ceph-mon[39629]: osd.10 v1:172.21.15.188:6808/1231263138 boot 2024-04-23T16:15:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:09 smithi103 ceph-mon[39629]: osdmap e81: 11 total, 11 up, 11 in 2024-04-23T16:15:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:09 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:15:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:09 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:15:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:09 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:15:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:15:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:09 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:15:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:09 smithi148 ceph-mon[44520]: osd.10 v1:172.21.15.188:6808/1231263138 boot 2024-04-23T16:15:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:09 smithi148 ceph-mon[44520]: osdmap e81: 11 total, 11 up, 11 in 2024-04-23T16:15:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:09 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T16:15:10.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:10 smithi188 ceph-mon[44101]: osdmap e82: 11 total, 11 up, 11 in 2024-04-23T16:15:10.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:10 smithi188 ceph-mon[44101]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:11.018 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T16:15:11.018 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T16:15:11.018 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-23T16:15:11.018 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-23T16:15:11.018 INFO:teuthology.orchestra.run.smithi188.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0144669 s, 725 MB/s 2024-04-23T16:15:11.018 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-23T16:15:11.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:10 smithi103 ceph-mon[39629]: osdmap e82: 11 total, 11 up, 11 in 2024-04-23T16:15:11.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:10 smithi103 ceph-mon[39629]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:11.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:10 smithi148 ceph-mon[44520]: osdmap e82: 11 total, 11 up, 11 in 2024-04-23T16:15:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:10 smithi148 ceph-mon[44520]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:11.859 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_1 2024-04-23T16:15:12.515 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:15:13.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:13 smithi188 ceph-mon[44101]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:13.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:13.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:13.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:13.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:13.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:13.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:13.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:15:13.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:15:13.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:13 smithi103 ceph-mon[39629]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:15:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:15:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:13 smithi148 ceph-mon[44520]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:15:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:15:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:14.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:14 smithi188 ceph-mon[44101]: Detected new or changed devices on smithi188 2024-04-23T16:15:14.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:14 smithi188 ceph-mon[44101]: Adjusting osd_memory_target on smithi188 to 7072M 2024-04-23T16:15:14.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:14 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:15:14.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:14 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:15:14.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:14 smithi188 ceph-mon[44101]: osdmap e83: 11 total, 11 up, 11 in 2024-04-23T16:15:14.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:14 smithi103 ceph-mon[39629]: Detected new or changed devices on smithi188 2024-04-23T16:15:14.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:14 smithi103 ceph-mon[39629]: Adjusting osd_memory_target on smithi188 to 7072M 2024-04-23T16:15:14.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:14 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:15:14.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:14 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:15:14.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:14 smithi103 ceph-mon[39629]: osdmap e83: 11 total, 11 up, 11 in 2024-04-23T16:15:14.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:14 smithi148 ceph-mon[44520]: Detected new or changed devices on smithi188 2024-04-23T16:15:14.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:14 smithi148 ceph-mon[44520]: Adjusting osd_memory_target on smithi188 to 7072M 2024-04-23T16:15:14.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:14 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:15:14.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:14 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:15:14.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:14 smithi148 ceph-mon[44520]: osdmap e83: 11 total, 11 up, 11 in 2024-04-23T16:15:15.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:15 smithi188 ceph-mon[44101]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:15.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:15 smithi188 ceph-mon[44101]: from='client.24412 v1:172.21.15.188:0/2815923648' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:15:15.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:15:15.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:15:15.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:15:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:15 smithi103 ceph-mon[39629]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:15 smithi103 ceph-mon[39629]: from='client.24412 v1:172.21.15.188:0/2815923648' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:15:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:15:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:15:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:15:15.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:15 smithi148 ceph-mon[44520]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:15.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:15 smithi148 ceph-mon[44520]: from='client.24412 v1:172.21.15.188:0/2815923648' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T16:15:15.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:15 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T16:15:15.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:15 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T16:15:15.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:15 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:15:17.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:17 smithi188 ceph-mon[44101]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:17.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:17 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.188:0/1700526173' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc163925-f623-4f63-ab89-35e2d511a730"}]: dispatch 2024-04-23T16:15:17.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:17 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.188:0/1700526173' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc163925-f623-4f63-ab89-35e2d511a730"}]': finished 2024-04-23T16:15:17.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:17 smithi188 ceph-mon[44101]: osdmap e84: 12 total, 11 up, 12 in 2024-04-23T16:15:17.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:17.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:17 smithi103 ceph-mon[39629]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:18.165 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:17 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.188:0/1700526173' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc163925-f623-4f63-ab89-35e2d511a730"}]: dispatch 2024-04-23T16:15:18.165 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:17 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.188:0/1700526173' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc163925-f623-4f63-ab89-35e2d511a730"}]': finished 2024-04-23T16:15:18.165 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:17 smithi103 ceph-mon[39629]: osdmap e84: 12 total, 11 up, 12 in 2024-04-23T16:15:18.165 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:18.166 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:17 smithi148 ceph-mon[44520]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:18.166 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:17 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.188:0/1700526173' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc163925-f623-4f63-ab89-35e2d511a730"}]: dispatch 2024-04-23T16:15:18.166 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:17 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.188:0/1700526173' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc163925-f623-4f63-ab89-35e2d511a730"}]': finished 2024-04-23T16:15:18.166 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:17 smithi148 ceph-mon[44520]: osdmap e84: 12 total, 11 up, 12 in 2024-04-23T16:15:18.166 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:18.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:15:18.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:15:18.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {auth=23,config=2,health=12,kv=53,logm=10} crc {auth=3928536605,config=359415709,health=3897446751,kv=2644210570,logm=3341531434}) 2024-04-23T16:15:18.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3563426041}) 2024-04-23T16:15:18.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1206993574}) 2024-04-23T16:15:18.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1026332952}) 2024-04-23T16:15:18.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3828671572}) 2024-04-23T16:15:18.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2567669340}) 2024-04-23T16:15:18.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2231433387}) 2024-04-23T16:15:18.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3887829274}) 2024-04-23T16:15:18.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=939371083}) 2024-04-23T16:15:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=394993905}) 2024-04-23T16:15:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3228139200}) 2024-04-23T16:15:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=293730920}) 2024-04-23T16:15:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2893628611}) 2024-04-23T16:15:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3706921475}) 2024-04-23T16:15:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1951226849}) 2024-04-23T16:15:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=452278661}) 2024-04-23T16:15:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=49,mdsmap=3,mgr=19,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=16} crc {logm=2830284699,mdsmap=2840952586,mgr=1765502803,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=2030111494,mon_config_key=593643376}) 2024-04-23T16:15:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2371546508}) 2024-04-23T16:15:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4220894326}) 2024-04-23T16:15:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=59,monmap=5,osd_metadata=11,osd_pg_creating=1,osdmap=24} crc {mon_config_key=764650754,monmap=2133572193,osd_metadata=3605335888,osd_pg_creating=1587152217,osdmap=3475142434}) 2024-04-23T16:15:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=83} crc {osdmap=1683482618}) 2024-04-23T16:15:18.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:18 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.188:0/1008705516' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:15:18.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:15:18.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:15:18.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {auth=23,config=2,health=12,kv=53,logm=10} crc {auth=3928536605,config=359415709,health=3897446751,kv=2644210570,logm=3341531434}) 2024-04-23T16:15:18.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3563426041}) 2024-04-23T16:15:18.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1206993574}) 2024-04-23T16:15:18.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1026332952}) 2024-04-23T16:15:18.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3828671572}) 2024-04-23T16:15:18.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2567669340}) 2024-04-23T16:15:18.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2231433387}) 2024-04-23T16:15:18.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3887829274}) 2024-04-23T16:15:18.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=939371083}) 2024-04-23T16:15:18.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=394993905}) 2024-04-23T16:15:18.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3228139200}) 2024-04-23T16:15:18.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=293730920}) 2024-04-23T16:15:18.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2893628611}) 2024-04-23T16:15:18.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3706921475}) 2024-04-23T16:15:18.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1951226849}) 2024-04-23T16:15:18.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=452278661}) 2024-04-23T16:15:18.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=49,mdsmap=3,mgr=19,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=16} crc {logm=2830284699,mdsmap=2840952586,mgr=1765502803,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=2030111494,mon_config_key=593643376}) 2024-04-23T16:15:18.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2371546508}) 2024-04-23T16:15:18.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4220894326}) 2024-04-23T16:15:18.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=59,monmap=5,osd_metadata=11,osd_pg_creating=1,osdmap=24} crc {mon_config_key=764650754,monmap=2133572193,osd_metadata=3605335888,osd_pg_creating=1587152217,osdmap=3475142434}) 2024-04-23T16:15:18.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=83} crc {osdmap=1683482618}) 2024-04-23T16:15:18.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:18 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.188:0/1008705516' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:15:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:15:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:15:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {auth=23,config=2,health=12,kv=53,logm=10} crc {auth=3928536605,config=359415709,health=3897446751,kv=2644210570,logm=3341531434}) 2024-04-23T16:15:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3563426041}) 2024-04-23T16:15:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1206993574}) 2024-04-23T16:15:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1026332952}) 2024-04-23T16:15:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3828671572}) 2024-04-23T16:15:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2567669340}) 2024-04-23T16:15:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2231433387}) 2024-04-23T16:15:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3887829274}) 2024-04-23T16:15:18.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=939371083}) 2024-04-23T16:15:18.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=394993905}) 2024-04-23T16:15:18.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3228139200}) 2024-04-23T16:15:18.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=293730920}) 2024-04-23T16:15:18.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2893628611}) 2024-04-23T16:15:18.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3706921475}) 2024-04-23T16:15:18.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1951226849}) 2024-04-23T16:15:18.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=452278661}) 2024-04-23T16:15:18.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=49,mdsmap=3,mgr=19,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=16} crc {logm=2830284699,mdsmap=2840952586,mgr=1765502803,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=2030111494,mon_config_key=593643376}) 2024-04-23T16:15:18.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2371546508}) 2024-04-23T16:15:18.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4220894326}) 2024-04-23T16:15:18.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=59,monmap=5,osd_metadata=11,osd_pg_creating=1,osdmap=24} crc {mon_config_key=764650754,monmap=2133572193,osd_metadata=3605335888,osd_pg_creating=1587152217,osdmap=3475142434}) 2024-04-23T16:15:18.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=83} crc {osdmap=1683482618}) 2024-04-23T16:15:18.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:18 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.188:0/1008705516' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T16:15:19.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:19 smithi188 ceph-mon[44101]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:19.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:19 smithi188 ceph-mon[44101]: osdmap e85: 12 total, 11 up, 12 in 2024-04-23T16:15:19.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:19 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:19.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:19 smithi103 ceph-mon[39629]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:19.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:19 smithi103 ceph-mon[39629]: osdmap e85: 12 total, 11 up, 12 in 2024-04-23T16:15:19.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:19 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:19.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:19 smithi148 ceph-mon[44520]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:19.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:19 smithi148 ceph-mon[44520]: osdmap e85: 12 total, 11 up, 12 in 2024-04-23T16:15:19.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:19 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:21.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:21 smithi188 ceph-mon[44101]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:21.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:21 smithi103 ceph-mon[39629]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:21.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:21 smithi148 ceph-mon[44520]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:22.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:22 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:15:22.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:22 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:15:22.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:22 smithi188 ceph-mon[44101]: osdmap e86: 12 total, 11 up, 12 in 2024-04-23T16:15:22.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:22 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:22.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:22 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:15:22.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:22 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:15:22.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:22 smithi103 ceph-mon[39629]: osdmap e86: 12 total, 11 up, 12 in 2024-04-23T16:15:22.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:22 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:22.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:22 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:15:22.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:22 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:15:22.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:22 smithi148 ceph-mon[44520]: osdmap e86: 12 total, 11 up, 12 in 2024-04-23T16:15:22.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:22 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:23.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:23 smithi188 ceph-mon[44101]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:23.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:23 smithi103 ceph-mon[39629]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:23.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:23 smithi148 ceph-mon[44520]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:24.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:24 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-23T16:15:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:24 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:15:24.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:24 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-23T16:15:24.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:24 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:15:24.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:24 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-23T16:15:24.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:24 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:15:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:25 smithi188 ceph-mon[44101]: Deploying daemon osd.11 on smithi188 2024-04-23T16:15:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:25 smithi188 ceph-mon[44101]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:25.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:25 smithi103 ceph-mon[39629]: Deploying daemon osd.11 on smithi188 2024-04-23T16:15:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:25 smithi103 ceph-mon[39629]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:25.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:25 smithi148 ceph-mon[44520]: Deploying daemon osd.11 on smithi188 2024-04-23T16:15:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:25 smithi148 ceph-mon[44520]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:27.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:27 smithi188 ceph-mon[44101]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:27.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:27 smithi103 ceph-mon[39629]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:27.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:27 smithi148 ceph-mon[44520]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:29.752 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:29 smithi188 ceph-mon[44101]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:29.752 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:29.752 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:29.752 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:15:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:29 smithi103 ceph-mon[39629]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:15:29.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:29 smithi148 ceph-mon[44520]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:15:31.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:31 smithi103 ceph-mon[39629]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:31.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:31 smithi148 ceph-mon[44520]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:31.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:31 smithi188 ceph-mon[44101]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:32.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:32 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:32.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:32 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:32.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:32 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:15:32.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:32 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:15:32.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:32 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:32.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:32 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:15:32.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:32 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:15:32.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:32 smithi188 ceph-mon[44101]: from='osd.11 v1:172.21.15.188:6812/457992661' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T16:15:32.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:32 smithi188 ceph-mon[44101]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T16:15:32.881 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:32 smithi188 ceph-mon[44101]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:32.970 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 11 on host 'smithi188' 2024-04-23T16:15:33.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:33.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:33.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:15:33.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:15:33.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:33.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:32 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:15:33.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:32 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:15:33.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:32 smithi103 ceph-mon[39629]: from='osd.11 v1:172.21.15.188:6812/457992661' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T16:15:33.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:32 smithi103 ceph-mon[39629]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T16:15:33.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:32 smithi103 ceph-mon[39629]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:32 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:32 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:32 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:15:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:32 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:15:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:32 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:32 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:15:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:32 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:15:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:32 smithi148 ceph-mon[44520]: from='osd.11 v1:172.21.15.188:6812/457992661' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T16:15:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:32 smithi148 ceph-mon[44520]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T16:15:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:32 smithi148 ceph-mon[44520]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:33.626 DEBUG:teuthology.orchestra.run.smithi188:osd.11> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.11.service 2024-04-23T16:15:33.628 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-23T16:15:33.629 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-23T16:15:33.661 INFO:journalctl@ceph.osd.11.smithi188.stdout:-- Logs begin at Tue 2024-04-23 15:57:06 UTC. -- 2024-04-23T16:15:33.871 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:33 smithi188 ceph-mon[44101]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-23T16:15:33.871 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:33 smithi188 ceph-mon[44101]: from='osd.11 v1:172.21.15.188:6812/457992661' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:15:33.871 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:33 smithi188 ceph-mon[44101]: osdmap e87: 12 total, 11 up, 12 in 2024-04-23T16:15:33.871 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:33 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:33.871 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:33 smithi188 ceph-mon[44101]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:15:33.871 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:33 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:33.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:33 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:15:33.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:33 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:33.910 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:15:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:33 smithi103 ceph-mon[39629]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-23T16:15:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:33 smithi103 ceph-mon[39629]: from='osd.11 v1:172.21.15.188:6812/457992661' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:15:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:33 smithi103 ceph-mon[39629]: osdmap e87: 12 total, 11 up, 12 in 2024-04-23T16:15:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:33 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:33 smithi103 ceph-mon[39629]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:15:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:33 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:33 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:15:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:33 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:34.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:33 smithi148 ceph-mon[44520]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-23T16:15:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:33 smithi148 ceph-mon[44520]: from='osd.11 v1:172.21.15.188:6812/457992661' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:15:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:33 smithi148 ceph-mon[44520]: osdmap e87: 12 total, 11 up, 12 in 2024-04-23T16:15:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:33 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:33 smithi148 ceph-mon[44520]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T16:15:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:33 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:33 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:15:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:33 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:34.473 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 16:15:34 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T16:15:34.162+0000 7fa9fa7a7700 -1 osd.11 0 waiting for initial osdmap 2024-04-23T16:15:34.474 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 16:15:34 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T16:15:34.162+0000 7fa9edf1d700 -1 osd.11 0 failed to load OSD map for epoch 32, got 0 bytes 2024-04-23T16:15:34.474 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 16:15:34 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T16:15:34.172+0000 7fa9f2592700 -1 osd.11 89 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:15:34.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:34 smithi188 ceph-mon[44101]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T16:15:34.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:34 smithi188 ceph-mon[44101]: osdmap e88: 12 total, 11 up, 12 in 2024-04-23T16:15:34.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:34 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:34.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:34 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:34.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:34 smithi188 ceph-mon[44101]: osdmap e89: 12 total, 11 up, 12 in 2024-04-23T16:15:34.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:34 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:34.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:34 smithi188 ceph-mon[44101]: from='osd.11 ' entity='osd.11' 2024-04-23T16:15:34.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:34 smithi188 ceph-mon[44101]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:34 smithi103 ceph-mon[39629]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T16:15:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:34 smithi103 ceph-mon[39629]: osdmap e88: 12 total, 11 up, 12 in 2024-04-23T16:15:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:34 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:34 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:34 smithi103 ceph-mon[39629]: osdmap e89: 12 total, 11 up, 12 in 2024-04-23T16:15:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:34 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:34 smithi103 ceph-mon[39629]: from='osd.11 ' entity='osd.11' 2024-04-23T16:15:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:34 smithi103 ceph-mon[39629]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:35.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:34 smithi148 ceph-mon[44520]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T16:15:35.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:34 smithi148 ceph-mon[44520]: osdmap e88: 12 total, 11 up, 12 in 2024-04-23T16:15:35.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:34 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:35.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:34 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:35.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:34 smithi148 ceph-mon[44520]: osdmap e89: 12 total, 11 up, 12 in 2024-04-23T16:15:35.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:34 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:35.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:34 smithi148 ceph-mon[44520]: from='osd.11 ' entity='osd.11' 2024-04-23T16:15:35.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:34 smithi148 ceph-mon[44520]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-23T16:15:35.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:35 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:15:35.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:35 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:15:35.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:15:35.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:15:35.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:35 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:35.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:35 smithi188 ceph-mon[44101]: osd.11 v1:172.21.15.188:6812/457992661 boot 2024-04-23T16:15:35.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:35 smithi188 ceph-mon[44101]: osdmap e90: 12 total, 12 up, 12 in 2024-04-23T16:15:35.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:35 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:36.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:35 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:15:36.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:35 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:15:36.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:15:36.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:15:36.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:35 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:36.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:35 smithi103 ceph-mon[39629]: osd.11 v1:172.21.15.188:6812/457992661 boot 2024-04-23T16:15:36.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:35 smithi103 ceph-mon[39629]: osdmap e90: 12 total, 12 up, 12 in 2024-04-23T16:15:36.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:35 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:35 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:15:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:35 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:15:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:15:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:15:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:35 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:35 smithi148 ceph-mon[44520]: osd.11 v1:172.21.15.188:6812/457992661 boot 2024-04-23T16:15:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:35 smithi148 ceph-mon[44520]: osdmap e90: 12 total, 12 up, 12 in 2024-04-23T16:15:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:35 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T16:15:36.139 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:15:36.853 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":91,"num_osds":12,"num_up_osds":12,"osd_up_since":1713888934,"num_in_osds":12,"osd_in_since":1713888917,"num_remapped_pgs":0} 2024-04-23T16:15:36.854 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T16:15:37.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:36 smithi103 ceph-mon[39629]: osdmap e91: 12 total, 12 up, 12 in 2024-04-23T16:15:37.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/493516370' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T16:15:37.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:36 smithi103 ceph-mon[39629]: pgmap v272: 1 pgs: 1 remapped+peering; 577 KiB data, 332 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:37.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:36 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:37.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:36 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:37.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:36 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:37.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:36 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:37.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:36 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:37.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:36 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:37.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:36 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:37.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:36 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:15:37.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:36 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:15:37.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:36 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:37.123 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:15:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:36 smithi188 ceph-mon[44101]: osdmap e91: 12 total, 12 up, 12 in 2024-04-23T16:15:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/493516370' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T16:15:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:36 smithi188 ceph-mon[44101]: pgmap v272: 1 pgs: 1 remapped+peering; 577 KiB data, 332 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:36 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:36 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:36 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:36 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:36 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:36 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:36 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:36 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:15:37.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:36 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:15:37.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:36 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:36 smithi148 ceph-mon[44520]: osdmap e91: 12 total, 12 up, 12 in 2024-04-23T16:15:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/493516370' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T16:15:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:36 smithi148 ceph-mon[44520]: pgmap v272: 1 pgs: 1 remapped+peering; 577 KiB data, 332 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:36 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:36 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:36 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:36 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:36 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:36 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T16:15:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:36 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:36 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:15:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:36 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:15:37.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:36 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:15:38.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:37 smithi188 ceph-mon[44101]: Detected new or changed devices on smithi188 2024-04-23T16:15:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:37 smithi188 ceph-mon[44101]: Adjusting osd_memory_target on smithi188 to 5304M 2024-04-23T16:15:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:37 smithi188 ceph-mon[44101]: osdmap e92: 12 total, 12 up, 12 in 2024-04-23T16:15:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:37 smithi103 ceph-mon[39629]: Detected new or changed devices on smithi188 2024-04-23T16:15:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:37 smithi103 ceph-mon[39629]: Adjusting osd_memory_target on smithi188 to 5304M 2024-04-23T16:15:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:37 smithi103 ceph-mon[39629]: osdmap e92: 12 total, 12 up, 12 in 2024-04-23T16:15:38.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:37 smithi148 ceph-mon[44520]: Detected new or changed devices on smithi188 2024-04-23T16:15:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:37 smithi148 ceph-mon[44520]: Adjusting osd_memory_target on smithi188 to 5304M 2024-04-23T16:15:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:37 smithi148 ceph-mon[44520]: osdmap e92: 12 total, 12 up, 12 in 2024-04-23T16:15:39.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:38 smithi103 ceph-mon[39629]: pgmap v274: 1 pgs: 1 remapped+peering; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:38 smithi188 ceph-mon[44101]: pgmap v274: 1 pgs: 1 remapped+peering; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:39.254 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:15:39.254 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":92,"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","created":"2024-04-23T16:08:13.477590+0000","modified":"2024-04-23T16:15:36.878528+0000","last_up_change":"2024-04-23T16:15:34.871303+0000","last_in_change":"2024-04-23T16:15:17.238203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T16:11:22.416777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4c084047-8f01-4b86-a60c-5592735b5862","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6801","nonce":4274040097}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6802","nonce":4274040097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6804","nonce":4274040097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6803","nonce":4274040097}]},"public_addr":"172.21.15.103:6801/4274040097","cluster_addr":"172.21.15.103:6802/4274040097","heartbeat_back_addr":"172.21.15.103:6804/4274040097","heartbeat_front_addr":"172.21.15.103:6803/4274040097","state":["exists","up"]},{"osd":1,"uuid":"3438f2bd-5fb7-432c-b383-29e18c442f20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6805","nonce":3716921156}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6806","nonce":3716921156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6808","nonce":3716921156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6807","nonce":3716921156}]},"public_addr":"172.21.15.103:6805/3716921156","cluster_addr":"172.21.15.103:6806/3716921156","heartbeat_back_addr":"172.21.15.103:6808/3716921156","heartbeat_front_addr":"172.21.15.103:6807/3716921156","state":["exists","up"]},{"osd":2,"uuid":"c17fffb4-8d17-4546-ba24-e5951a2a37fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6809","nonce":424240088}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6810","nonce":424240088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6812","nonce":424240088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6811","nonce":424240088}]},"public_addr":"172.21.15.103:6809/424240088","cluster_addr":"172.21.15.103:6810/424240088","heartbeat_back_addr":"172.21.15.103:6812/424240088","heartbeat_front_addr":"172.21.15.103:6811/424240088","state":["exists","up"]},{"osd":3,"uuid":"cee91a3d-55dc-4e89-b568-86821632bb1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6813","nonce":1895183074}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6814","nonce":1895183074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6816","nonce":1895183074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6815","nonce":1895183074}]},"public_addr":"172.21.15.103:6813/1895183074","cluster_addr":"172.21.15.103:6814/1895183074","heartbeat_back_addr":"172.21.15.103:6816/1895183074","heartbeat_front_addr":"172.21.15.103:6815/1895183074","state":["exists","up"]},{"osd":4,"uuid":"61d5385b-2c82-42f8-93c3-8f1baf71c65e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6800","nonce":794945071}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6801","nonce":794945071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6803","nonce":794945071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6802","nonce":794945071}]},"public_addr":"172.21.15.148:6800/794945071","cluster_addr":"172.21.15.148:6801/794945071","heartbeat_back_addr":"172.21.15.148:6803/794945071","heartbeat_front_addr":"172.21.15.148:6802/794945071","state":["exists","up"]},{"osd":5,"uuid":"5d492189-c3cd-47ad-a12a-ed4d275bcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6804","nonce":3547200711}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6805","nonce":3547200711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6807","nonce":3547200711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6806","nonce":3547200711}]},"public_addr":"172.21.15.148:6804/3547200711","cluster_addr":"172.21.15.148:6805/3547200711","heartbeat_back_addr":"172.21.15.148:6807/3547200711","heartbeat_front_addr":"172.21.15.148:6806/3547200711","state":["exists","up"]},{"osd":6,"uuid":"89f2379e-cf14-4e69-8747-7535db81b70b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6808","nonce":366919758}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6809","nonce":366919758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6811","nonce":366919758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6810","nonce":366919758}]},"public_addr":"172.21.15.148:6808/366919758","cluster_addr":"172.21.15.148:6809/366919758","heartbeat_back_addr":"172.21.15.148:6811/366919758","heartbeat_front_addr":"172.21.15.148:6810/366919758","state":["exists","up"]},{"osd":7,"uuid":"f27b035a-558f-4528-80ea-2a00d03f6539","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6812","nonce":3719550888}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6813","nonce":3719550888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6815","nonce":3719550888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6814","nonce":3719550888}]},"public_addr":"172.21.15.148:6812/3719550888","cluster_addr":"172.21.15.148:6813/3719550888","heartbeat_back_addr":"172.21.15.148:6815/3719550888","heartbeat_front_addr":"172.21.15.148:6814/3719550888","state":["exists","up"]},{"osd":8,"uuid":"a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6800","nonce":1151408595}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6801","nonce":1151408595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6803","nonce":1151408595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6802","nonce":1151408595}]},"public_addr":"172.21.15.188:6800/1151408595","cluster_addr":"172.21.15.188:6801/1151408595","heartbeat_back_addr":"172.21.15.188:6803/1151408595","heartbeat_front_addr":"172.21.15.188:6802/1151408595","state":["exists","up"]},{"osd":9,"uuid":"124d9b05-11eb-4622-b6a2-c4d5bd60b7e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6804","nonce":306501862}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6805","nonce":306501862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6807","nonce":306501862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6806","nonce":306501862}]},"public_addr":"172.21.15.188:6804/306501862","cluster_addr":"172.21.15.188:6805/306501862","heartbeat_back_addr":"172.21.15.188:6807/306501862","heartbeat_front_addr":"172.21.15.188:6806/306501862","state":["exists","up"]},{"osd":10,"uuid":"3dfb9070-e534-4ea4-a25e-ee1c51297447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6808","nonce":1231263138}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6809","nonce":1231263138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6811","nonce":1231263138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6810","nonce":1231263138}]},"public_addr":"172.21.15.188:6808/1231263138","cluster_addr":"172.21.15.188:6809/1231263138","heartbeat_back_addr":"172.21.15.188:6811/1231263138","heartbeat_front_addr":"172.21.15.188:6810/1231263138","state":["exists","up"]},{"osd":11,"uuid":"bc163925-f623-4f63-ab89-35e2d511a730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6812","nonce":457992661}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6813","nonce":457992661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6815","nonce":457992661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6814","nonce":457992661}]},"public_addr":"172.21.15.188:6812/457992661","cluster_addr":"172.21.15.188:6813/457992661","heartbeat_back_addr":"172.21.15.188:6815/457992661","heartbeat_front_addr":"172.21.15.188:6814/457992661","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:14:22.900398+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:14:44.525385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:17.603586+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:14:58.293416+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:32.124668+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:13.417824+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:14:48.622952+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:14:57.091261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:20.600769+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:14:40.751184+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:06.949705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:33.359610+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6800/3535470771":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1188058303":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1510553721":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/281182801":"2024-04-24T16:08:44.450632+0000","172.21.15.103:6800/2049124518":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/466241963":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/2726838698":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/596230573":"2024-04-24T16:09:14.324448+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T16:15:39.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:38 smithi148 ceph-mon[44520]: pgmap v274: 1 pgs: 1 remapped+peering; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:40.092 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:39 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:15:40.092 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:39 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:15:40.092 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:39 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3434552719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:15:40.122 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T16:11:22.416777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T16:15:40.123 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-23T16:15:40.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:39 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:15:40.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:39 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:15:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:39 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3434552719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:15:40.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:39 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:15:40.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:39 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:15:40.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:39 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3434552719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:15:40.429 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:15:40.943 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:40 smithi103 ceph-mon[39629]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-23T16:15:41.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:40 smithi188 ceph-mon[44101]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-23T16:15:41.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:40 smithi148 ceph-mon[44520]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-23T16:15:42.503 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-04-23T16:15:43.295 INFO:tasks.cephadm:Setting up client nodes... 2024-04-23T16:15:43.296 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-23T16:15:43.563 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:15:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:43 smithi103 ceph-mon[39629]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-23T16:15:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:43 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/480682430' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T16:15:43.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:43 smithi188 ceph-mon[44101]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-23T16:15:43.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:43 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/480682430' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T16:15:43.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:43 smithi148 ceph-mon[44520]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-23T16:15:43.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:43 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/480682430' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T16:15:45.683 INFO:teuthology.orchestra.run.smithi103.stdout:[client.0] 2024-04-23T16:15:45.683 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCx3idmQ9NRKBAAK2Kwu65I9lD7Qb4yrxXSbw== 2024-04-23T16:15:45.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:45 smithi188 ceph-mon[44101]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-23T16:15:45.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:45 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:15:45.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:45 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:15:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:45 smithi103 ceph-mon[39629]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-23T16:15:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:45 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:15:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:45 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:15:45.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:45 smithi148 ceph-mon[44520]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-23T16:15:45.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:45 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:15:45.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:45 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:15:46.485 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-23T16:15:46.485 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-23T16:15:46.485 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-23T16:15:46.525 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-23T16:15:46.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:46 smithi188 ceph-mon[44101]: osdmap e93: 12 total, 12 up, 12 in 2024-04-23T16:15:46.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:46 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3484508841' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T16:15:46.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:46 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3484508841' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T16:15:46.803 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:15:46.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:46 smithi103 ceph-mon[39629]: osdmap e93: 12 total, 12 up, 12 in 2024-04-23T16:15:46.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:46 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3484508841' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T16:15:46.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:46 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3484508841' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T16:15:46.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:46 smithi148 ceph-mon[44520]: osdmap e93: 12 total, 12 up, 12 in 2024-04-23T16:15:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:46 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3484508841' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T16:15:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:46 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3484508841' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T16:15:47.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:47 smithi188 ceph-mon[44101]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-23T16:15:47.772 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:47 smithi148 ceph-mon[44520]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-23T16:15:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:47 smithi103 ceph-mon[39629]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-23T16:15:49.013 INFO:teuthology.orchestra.run.smithi148.stdout:[client.1] 2024-04-23T16:15:49.013 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQC13idmJZl4ABAA0tS+siSxtF8T2CGPWMBtJQ== 2024-04-23T16:15:49.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:49 smithi148 ceph-mon[44520]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T16:15:49.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:49 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.148:0/4257807287' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T16:15:49.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:49 smithi148 ceph-mon[44520]: from='client.34457 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T16:15:49.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:49 smithi148 ceph-mon[44520]: from='client.34457 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T16:15:49.723 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-04-23T16:15:49.723 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-23T16:15:49.723 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-23T16:15:49.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:49 smithi188 ceph-mon[44101]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T16:15:49.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:49 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.148:0/4257807287' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T16:15:49.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:49 smithi188 ceph-mon[44101]: from='client.34457 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T16:15:49.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:49 smithi188 ceph-mon[44101]: from='client.34457 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T16:15:49.766 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-23T16:15:49.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:49 smithi103 ceph-mon[39629]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T16:15:49.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:49 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.148:0/4257807287' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T16:15:49.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:49 smithi103 ceph-mon[39629]: from='client.34457 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T16:15:49.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:49 smithi103 ceph-mon[39629]: from='client.34457 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T16:15:50.044 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:15:51.590 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:51 smithi188 ceph-mon[44101]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:51 smithi103 ceph-mon[39629]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:51.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:51 smithi148 ceph-mon[44520]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:52.147 INFO:teuthology.orchestra.run.smithi188.stdout:[client.2] 2024-04-23T16:15:52.147 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQC43idmzJGFCBAAqJtoZJtBhiKvE/vwW/Tu8A== 2024-04-23T16:15:52.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:52 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.188:0/1231509369' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T16:15:52.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:52 smithi188 ceph-mon[44101]: from='client.24466 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T16:15:52.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:52 smithi188 ceph-mon[44101]: from='client.24466 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T16:15:52.828 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T16:15:52.828 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-04-23T16:15:52.828 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-23T16:15:52.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:52 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.188:0/1231509369' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T16:15:52.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:52 smithi103 ceph-mon[39629]: from='client.24466 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T16:15:52.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:52 smithi103 ceph-mon[39629]: from='client.24466 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T16:15:52.868 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-23T16:15:52.868 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-23T16:15:52.869 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-23T16:15:52.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:52 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.188:0/1231509369' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T16:15:52.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:52 smithi148 ceph-mon[44520]: from='client.24466 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T16:15:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:52 smithi148 ceph-mon[44520]: from='client.24466 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T16:15:53.143 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:15:53.635 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:53 smithi103 ceph-mon[39629]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:53 smithi148 ceph-mon[44520]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:53.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:53 smithi188 ceph-mon[44101]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:55.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:54 smithi103 ceph-mon[39629]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:55.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:54 smithi148 ceph-mon[44520]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:55.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:54 smithi188 ceph-mon[44101]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:55.331 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:15:55.975 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:55 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2845468981' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T16:15:56.001 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":16,"flags":0,"active_gid":14154,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6800","nonce":3528736114}]},"active_addr":"172.21.15.103:6800/3528736114","active_change":"2024-04-23T16:09:14.324566+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":34106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.103:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":3799852283}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":3972993134}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":2749250206}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":2874682003}]}]} 2024-04-23T16:15:56.006 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-23T16:15:56.006 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-23T16:15:56.006 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T16:15:56.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:55 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2845468981' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T16:15:56.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:55 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2845468981' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T16:15:56.270 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:15:56.844 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:56 smithi103 ceph-mon[39629]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:57.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:56 smithi148 ceph-mon[44520]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:57.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:56 smithi188 ceph-mon[44101]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:58.484 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:15:58.484 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":93,"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","created":"2024-04-23T16:08:13.477590+0000","modified":"2024-04-23T16:15:45.457697+0000","last_up_change":"2024-04-23T16:15:34.871303+0000","last_in_change":"2024-04-23T16:15:17.238203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T16:11:22.416777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4c084047-8f01-4b86-a60c-5592735b5862","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6801","nonce":4274040097}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6802","nonce":4274040097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6804","nonce":4274040097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6803","nonce":4274040097}]},"public_addr":"172.21.15.103:6801/4274040097","cluster_addr":"172.21.15.103:6802/4274040097","heartbeat_back_addr":"172.21.15.103:6804/4274040097","heartbeat_front_addr":"172.21.15.103:6803/4274040097","state":["exists","up"]},{"osd":1,"uuid":"3438f2bd-5fb7-432c-b383-29e18c442f20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6805","nonce":3716921156}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6806","nonce":3716921156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6808","nonce":3716921156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6807","nonce":3716921156}]},"public_addr":"172.21.15.103:6805/3716921156","cluster_addr":"172.21.15.103:6806/3716921156","heartbeat_back_addr":"172.21.15.103:6808/3716921156","heartbeat_front_addr":"172.21.15.103:6807/3716921156","state":["exists","up"]},{"osd":2,"uuid":"c17fffb4-8d17-4546-ba24-e5951a2a37fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6809","nonce":424240088}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6810","nonce":424240088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6812","nonce":424240088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6811","nonce":424240088}]},"public_addr":"172.21.15.103:6809/424240088","cluster_addr":"172.21.15.103:6810/424240088","heartbeat_back_addr":"172.21.15.103:6812/424240088","heartbeat_front_addr":"172.21.15.103:6811/424240088","state":["exists","up"]},{"osd":3,"uuid":"cee91a3d-55dc-4e89-b568-86821632bb1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6813","nonce":1895183074}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6814","nonce":1895183074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6816","nonce":1895183074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6815","nonce":1895183074}]},"public_addr":"172.21.15.103:6813/1895183074","cluster_addr":"172.21.15.103:6814/1895183074","heartbeat_back_addr":"172.21.15.103:6816/1895183074","heartbeat_front_addr":"172.21.15.103:6815/1895183074","state":["exists","up"]},{"osd":4,"uuid":"61d5385b-2c82-42f8-93c3-8f1baf71c65e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6800","nonce":794945071}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6801","nonce":794945071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6803","nonce":794945071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6802","nonce":794945071}]},"public_addr":"172.21.15.148:6800/794945071","cluster_addr":"172.21.15.148:6801/794945071","heartbeat_back_addr":"172.21.15.148:6803/794945071","heartbeat_front_addr":"172.21.15.148:6802/794945071","state":["exists","up"]},{"osd":5,"uuid":"5d492189-c3cd-47ad-a12a-ed4d275bcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6804","nonce":3547200711}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6805","nonce":3547200711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6807","nonce":3547200711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6806","nonce":3547200711}]},"public_addr":"172.21.15.148:6804/3547200711","cluster_addr":"172.21.15.148:6805/3547200711","heartbeat_back_addr":"172.21.15.148:6807/3547200711","heartbeat_front_addr":"172.21.15.148:6806/3547200711","state":["exists","up"]},{"osd":6,"uuid":"89f2379e-cf14-4e69-8747-7535db81b70b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6808","nonce":366919758}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6809","nonce":366919758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6811","nonce":366919758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6810","nonce":366919758}]},"public_addr":"172.21.15.148:6808/366919758","cluster_addr":"172.21.15.148:6809/366919758","heartbeat_back_addr":"172.21.15.148:6811/366919758","heartbeat_front_addr":"172.21.15.148:6810/366919758","state":["exists","up"]},{"osd":7,"uuid":"f27b035a-558f-4528-80ea-2a00d03f6539","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6812","nonce":3719550888}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6813","nonce":3719550888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6815","nonce":3719550888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6814","nonce":3719550888}]},"public_addr":"172.21.15.148:6812/3719550888","cluster_addr":"172.21.15.148:6813/3719550888","heartbeat_back_addr":"172.21.15.148:6815/3719550888","heartbeat_front_addr":"172.21.15.148:6814/3719550888","state":["exists","up"]},{"osd":8,"uuid":"a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6800","nonce":1151408595}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6801","nonce":1151408595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6803","nonce":1151408595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6802","nonce":1151408595}]},"public_addr":"172.21.15.188:6800/1151408595","cluster_addr":"172.21.15.188:6801/1151408595","heartbeat_back_addr":"172.21.15.188:6803/1151408595","heartbeat_front_addr":"172.21.15.188:6802/1151408595","state":["exists","up"]},{"osd":9,"uuid":"124d9b05-11eb-4622-b6a2-c4d5bd60b7e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6804","nonce":306501862}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6805","nonce":306501862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6807","nonce":306501862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6806","nonce":306501862}]},"public_addr":"172.21.15.188:6804/306501862","cluster_addr":"172.21.15.188:6805/306501862","heartbeat_back_addr":"172.21.15.188:6807/306501862","heartbeat_front_addr":"172.21.15.188:6806/306501862","state":["exists","up"]},{"osd":10,"uuid":"3dfb9070-e534-4ea4-a25e-ee1c51297447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6808","nonce":1231263138}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6809","nonce":1231263138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6811","nonce":1231263138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6810","nonce":1231263138}]},"public_addr":"172.21.15.188:6808/1231263138","cluster_addr":"172.21.15.188:6809/1231263138","heartbeat_back_addr":"172.21.15.188:6811/1231263138","heartbeat_front_addr":"172.21.15.188:6810/1231263138","state":["exists","up"]},{"osd":11,"uuid":"bc163925-f623-4f63-ab89-35e2d511a730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6812","nonce":457992661}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6813","nonce":457992661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6815","nonce":457992661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6814","nonce":457992661}]},"public_addr":"172.21.15.188:6812/457992661","cluster_addr":"172.21.15.188:6813/457992661","heartbeat_back_addr":"172.21.15.188:6815/457992661","heartbeat_front_addr":"172.21.15.188:6814/457992661","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:44.922838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:14:44.525385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:17.603586+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:14:58.293416+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:32.124668+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:13.417824+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:14:48.622952+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:14:57.091261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:20.600769+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:14:40.751184+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:06.949705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:33.359610+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6800/3535470771":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1188058303":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1510553721":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/281182801":"2024-04-24T16:08:44.450632+0000","172.21.15.103:6800/2049124518":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/466241963":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/2726838698":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/596230573":"2024-04-24T16:09:14.324448+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T16:15:59.201 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-23T16:15:59.201 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T16:15:59.460 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:59 smithi103 ceph-mon[39629]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:59.460 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:15:59 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/368359075' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:15:59.466 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:15:59.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:59 smithi188 ceph-mon[44101]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:59.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:15:59 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/368359075' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:15:59.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:59 smithi148 ceph-mon[44520]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:15:59.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:15:59 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/368359075' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:16:01.641 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:16:01.642 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":93,"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","created":"2024-04-23T16:08:13.477590+0000","modified":"2024-04-23T16:15:45.457697+0000","last_up_change":"2024-04-23T16:15:34.871303+0000","last_in_change":"2024-04-23T16:15:17.238203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T16:11:22.416777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4c084047-8f01-4b86-a60c-5592735b5862","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6801","nonce":4274040097}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6802","nonce":4274040097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6804","nonce":4274040097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6803","nonce":4274040097}]},"public_addr":"172.21.15.103:6801/4274040097","cluster_addr":"172.21.15.103:6802/4274040097","heartbeat_back_addr":"172.21.15.103:6804/4274040097","heartbeat_front_addr":"172.21.15.103:6803/4274040097","state":["exists","up"]},{"osd":1,"uuid":"3438f2bd-5fb7-432c-b383-29e18c442f20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6805","nonce":3716921156}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6806","nonce":3716921156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6808","nonce":3716921156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6807","nonce":3716921156}]},"public_addr":"172.21.15.103:6805/3716921156","cluster_addr":"172.21.15.103:6806/3716921156","heartbeat_back_addr":"172.21.15.103:6808/3716921156","heartbeat_front_addr":"172.21.15.103:6807/3716921156","state":["exists","up"]},{"osd":2,"uuid":"c17fffb4-8d17-4546-ba24-e5951a2a37fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6809","nonce":424240088}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6810","nonce":424240088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6812","nonce":424240088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6811","nonce":424240088}]},"public_addr":"172.21.15.103:6809/424240088","cluster_addr":"172.21.15.103:6810/424240088","heartbeat_back_addr":"172.21.15.103:6812/424240088","heartbeat_front_addr":"172.21.15.103:6811/424240088","state":["exists","up"]},{"osd":3,"uuid":"cee91a3d-55dc-4e89-b568-86821632bb1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6813","nonce":1895183074}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6814","nonce":1895183074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6816","nonce":1895183074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6815","nonce":1895183074}]},"public_addr":"172.21.15.103:6813/1895183074","cluster_addr":"172.21.15.103:6814/1895183074","heartbeat_back_addr":"172.21.15.103:6816/1895183074","heartbeat_front_addr":"172.21.15.103:6815/1895183074","state":["exists","up"]},{"osd":4,"uuid":"61d5385b-2c82-42f8-93c3-8f1baf71c65e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6800","nonce":794945071}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6801","nonce":794945071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6803","nonce":794945071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6802","nonce":794945071}]},"public_addr":"172.21.15.148:6800/794945071","cluster_addr":"172.21.15.148:6801/794945071","heartbeat_back_addr":"172.21.15.148:6803/794945071","heartbeat_front_addr":"172.21.15.148:6802/794945071","state":["exists","up"]},{"osd":5,"uuid":"5d492189-c3cd-47ad-a12a-ed4d275bcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6804","nonce":3547200711}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6805","nonce":3547200711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6807","nonce":3547200711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6806","nonce":3547200711}]},"public_addr":"172.21.15.148:6804/3547200711","cluster_addr":"172.21.15.148:6805/3547200711","heartbeat_back_addr":"172.21.15.148:6807/3547200711","heartbeat_front_addr":"172.21.15.148:6806/3547200711","state":["exists","up"]},{"osd":6,"uuid":"89f2379e-cf14-4e69-8747-7535db81b70b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6808","nonce":366919758}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6809","nonce":366919758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6811","nonce":366919758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6810","nonce":366919758}]},"public_addr":"172.21.15.148:6808/366919758","cluster_addr":"172.21.15.148:6809/366919758","heartbeat_back_addr":"172.21.15.148:6811/366919758","heartbeat_front_addr":"172.21.15.148:6810/366919758","state":["exists","up"]},{"osd":7,"uuid":"f27b035a-558f-4528-80ea-2a00d03f6539","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6812","nonce":3719550888}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6813","nonce":3719550888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6815","nonce":3719550888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6814","nonce":3719550888}]},"public_addr":"172.21.15.148:6812/3719550888","cluster_addr":"172.21.15.148:6813/3719550888","heartbeat_back_addr":"172.21.15.148:6815/3719550888","heartbeat_front_addr":"172.21.15.148:6814/3719550888","state":["exists","up"]},{"osd":8,"uuid":"a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6800","nonce":1151408595}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6801","nonce":1151408595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6803","nonce":1151408595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6802","nonce":1151408595}]},"public_addr":"172.21.15.188:6800/1151408595","cluster_addr":"172.21.15.188:6801/1151408595","heartbeat_back_addr":"172.21.15.188:6803/1151408595","heartbeat_front_addr":"172.21.15.188:6802/1151408595","state":["exists","up"]},{"osd":9,"uuid":"124d9b05-11eb-4622-b6a2-c4d5bd60b7e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6804","nonce":306501862}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6805","nonce":306501862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6807","nonce":306501862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6806","nonce":306501862}]},"public_addr":"172.21.15.188:6804/306501862","cluster_addr":"172.21.15.188:6805/306501862","heartbeat_back_addr":"172.21.15.188:6807/306501862","heartbeat_front_addr":"172.21.15.188:6806/306501862","state":["exists","up"]},{"osd":10,"uuid":"3dfb9070-e534-4ea4-a25e-ee1c51297447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6808","nonce":1231263138}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6809","nonce":1231263138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6811","nonce":1231263138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6810","nonce":1231263138}]},"public_addr":"172.21.15.188:6808/1231263138","cluster_addr":"172.21.15.188:6809/1231263138","heartbeat_back_addr":"172.21.15.188:6811/1231263138","heartbeat_front_addr":"172.21.15.188:6810/1231263138","state":["exists","up"]},{"osd":11,"uuid":"bc163925-f623-4f63-ab89-35e2d511a730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6812","nonce":457992661}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6813","nonce":457992661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6815","nonce":457992661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6814","nonce":457992661}]},"public_addr":"172.21.15.188:6812/457992661","cluster_addr":"172.21.15.188:6813/457992661","heartbeat_back_addr":"172.21.15.188:6815/457992661","heartbeat_front_addr":"172.21.15.188:6814/457992661","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:44.922838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:14:44.525385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:17.603586+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:14:58.293416+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:32.124668+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:13.417824+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:14:48.622952+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:14:57.091261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:20.600769+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:14:40.751184+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:06.949705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:15:33.359610+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6800/3535470771":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1188058303":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1510553721":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/281182801":"2024-04-24T16:08:44.450632+0000","172.21.15.103:6800/2049124518":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/466241963":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/2726838698":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/596230573":"2024-04-24T16:09:14.324448+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T16:16:01.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:01 smithi188 ceph-mon[44101]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:01.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:01 smithi103 ceph-mon[39629]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:01.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:01 smithi148 ceph-mon[44520]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:02.352 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-23T16:16:02.352 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-23T16:16:02.353 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-23T16:16:02.353 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-23T16:16:02.353 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-23T16:16:02.354 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-23T16:16:02.354 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-23T16:16:02.355 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-23T16:16:02.355 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-23T16:16:02.356 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-23T16:16:02.356 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-23T16:16:02.356 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-23T16:16:02.610 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:02 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3415530116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:16:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:02 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3415530116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:16:02.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:02 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3415530116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:16:03.078 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:03.078 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:03.078 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:03.079 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:03.082 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:03.088 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:03.651 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:03 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:16:03.651 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:03 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:16:03.652 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:03 smithi103 ceph-mon[39629]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:03.652 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:03 smithi103 ceph-mon[39629]: osdmap e94: 12 total, 12 up, 12 in 2024-04-23T16:16:03.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:03 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:16:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:03 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:16:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:03 smithi148 ceph-mon[44520]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:03 smithi148 ceph-mon[44520]: osdmap e94: 12 total, 12 up, 12 in 2024-04-23T16:16:03.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:03 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:16:03.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:03 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:16:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:03 smithi188 ceph-mon[44101]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:03 smithi188 ceph-mon[44101]: osdmap e94: 12 total, 12 up, 12 in 2024-04-23T16:16:05.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:05 smithi103 ceph-mon[39629]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:05.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:05 smithi148 ceph-mon[44520]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:05.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:05 smithi188 ceph-mon[44101]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:06.728 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:06.729 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:06.730 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:06.760 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:06 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:16:06.760 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:06 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:16:06.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:06 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:16:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:06 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:16:06.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:06 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:16:06.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:06 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:16:07.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:07 smithi103 ceph-mon[39629]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:07.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:07 smithi103 ceph-mon[39629]: osdmap e95: 12 total, 12 up, 12 in 2024-04-23T16:16:07.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:07 smithi148 ceph-mon[44520]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:07 smithi148 ceph-mon[44520]: osdmap e95: 12 total, 12 up, 12 in 2024-04-23T16:16:07.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:07 smithi188 ceph-mon[44101]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:07.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:07 smithi188 ceph-mon[44101]: osdmap e95: 12 total, 12 up, 12 in 2024-04-23T16:16:08.863 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:09.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:09 smithi103 ceph-mon[39629]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:09.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:09 smithi148 ceph-mon[44520]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:09.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:09 smithi188 ceph-mon[44101]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:10.174 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:10.669 INFO:teuthology.orchestra.run.smithi103.stdout:55834574909 2024-04-23T16:16:10.669 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-23T16:16:11.518 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:11 smithi103 ceph-mon[39629]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:11.518 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:11 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:16:11.518 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:11 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:16:11.518 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:11 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:16:11.519 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:11 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:16:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:11 smithi148 ceph-mon[44520]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:11 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:16:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:11 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:16:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:11 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:16:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:11 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:16:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:11 smithi188 ceph-mon[44101]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:11 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:16:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:11 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:16:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:11 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:16:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:11 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:16:12.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:12 smithi103 ceph-mon[39629]: osdmap e96: 12 total, 12 up, 12 in 2024-04-23T16:16:12.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:12 smithi148 ceph-mon[44520]: osdmap e96: 12 total, 12 up, 12 in 2024-04-23T16:16:12.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:12 smithi188 ceph-mon[44101]: osdmap e96: 12 total, 12 up, 12 in 2024-04-23T16:16:13.765 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:13 smithi103 ceph-mon[39629]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:13.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:13 smithi148 ceph-mon[44520]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:13.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:13 smithi188 ceph-mon[44101]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:14.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:14 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:16:14.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:14 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:16:14.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:14 smithi103 ceph-mon[39629]: osdmap e97: 12 total, 12 up, 12 in 2024-04-23T16:16:14.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:14 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:16:14.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:14 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:16:14.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:14 smithi148 ceph-mon[44520]: osdmap e97: 12 total, 12 up, 12 in 2024-04-23T16:16:14.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:14 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:16:14.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:14 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:16:14.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:14 smithi188 ceph-mon[44101]: osdmap e97: 12 total, 12 up, 12 in 2024-04-23T16:16:15.795 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:15 smithi103 ceph-mon[39629]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:15.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:15 smithi148 ceph-mon[44520]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:15.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:15 smithi188 ceph-mon[44101]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:16.263 INFO:teuthology.orchestra.run.smithi103.stdout:386547056650 2024-04-23T16:16:16.263 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-23T16:16:17.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:17 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:16:17.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:17 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:16:17.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:17 smithi103 ceph-mon[39629]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:17.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:17 smithi103 ceph-mon[39629]: osdmap e98: 12 total, 12 up, 12 in 2024-04-23T16:16:17.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:17 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:16:17.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:17 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:16:17.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:17 smithi148 ceph-mon[44520]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:17.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:17 smithi148 ceph-mon[44520]: osdmap e98: 12 total, 12 up, 12 in 2024-04-23T16:16:17.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:17 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:16:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:17 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:16:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:17 smithi188 ceph-mon[44101]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:17 smithi188 ceph-mon[44101]: osdmap e98: 12 total, 12 up, 12 in 2024-04-23T16:16:18.136 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:19.158 INFO:teuthology.orchestra.run.smithi103.stdout:90194313271 2024-04-23T16:16:19.159 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-23T16:16:19.702 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:19 smithi103 ceph-mon[39629]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:19.703 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:19 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:16:19.703 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:19 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:16:19.703 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:19 smithi103 ceph-mon[39629]: osdmap e99: 12 total, 12 up, 12 in 2024-04-23T16:16:19.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:19 smithi148 ceph-mon[44520]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:19.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:19 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:16:19.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:19 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:16:19.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:19 smithi148 ceph-mon[44520]: osdmap e99: 12 total, 12 up, 12 in 2024-04-23T16:16:19.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:19 smithi188 ceph-mon[44101]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:19.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:19 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:16:19.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:19 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:16:19.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:19 smithi188 ceph-mon[44101]: osdmap e99: 12 total, 12 up, 12 in 2024-04-23T16:16:21.802 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:21 smithi103 ceph-mon[39629]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:21.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:21 smithi148 ceph-mon[44520]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:21.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:21 smithi188 ceph-mon[44101]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:23.659 INFO:teuthology.orchestra.run.smithi103.stdout:141733920814 2024-04-23T16:16:23.660 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-23T16:16:23.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:23 smithi103 ceph-mon[39629]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:23.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:23 smithi148 ceph-mon[44520]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:23.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:23 smithi188 ceph-mon[44101]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:25.786 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:25 smithi103 ceph-mon[39629]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:25.812 INFO:teuthology.orchestra.run.smithi103.stdout:34359738437 2024-04-23T16:16:25.812 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-23T16:16:25.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:25 smithi148 ceph-mon[44520]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:25.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:25 smithi188 ceph-mon[44101]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:26.213 INFO:teuthology.orchestra.run.smithi103.stdout:283467841564 2024-04-23T16:16:26.214 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-23T16:16:27.448 INFO:teuthology.orchestra.run.smithi103.stdout:313532612632 2024-04-23T16:16:27.448 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-23T16:16:27.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:27 smithi103 ceph-mon[39629]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:27.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:27 smithi148 ceph-mon[44520]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:27.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:27 smithi188 ceph-mon[44101]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:28.311 INFO:teuthology.orchestra.run.smithi103.stdout:201863462950 2024-04-23T16:16:28.312 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-23T16:16:28.656 INFO:teuthology.orchestra.run.smithi103.stdout:163208757291 2024-04-23T16:16:28.656 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-23T16:16:29.746 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:29 smithi103 ceph-mon[39629]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:29.775 INFO:teuthology.orchestra.run.smithi103.stdout:115964117046 2024-04-23T16:16:29.775 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-23T16:16:29.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:29 smithi148 ceph-mon[44520]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:29.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:29 smithi188 ceph-mon[44101]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:30.840 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:30.882 INFO:teuthology.orchestra.run.smithi103.stdout:347892350993 2024-04-23T16:16:30.882 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-23T16:16:31.315 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:31.321 INFO:teuthology.orchestra.run.smithi103.stdout:240518168611 2024-04-23T16:16:31.321 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-23T16:16:31.322 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:31.327 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:31.354 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:31.815 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:31 smithi103 ceph-mon[39629]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:31 smithi148 ceph-mon[44520]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:31.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:31 smithi188 ceph-mon[44101]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:33 smithi103 ceph-mon[39629]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:33.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:33 smithi148 ceph-mon[44520]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:33.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:33 smithi188 ceph-mon[44101]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:33.991 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:33.996 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:33.997 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:34.003 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:34.003 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:34.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:34 smithi103 ceph-mon[39629]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:35.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:34 smithi148 ceph-mon[44520]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:35.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:34 smithi188 ceph-mon[44101]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:37.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:37 smithi103 ceph-mon[39629]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:37 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:16:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:37 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:16:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:37 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:16:37.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:37 smithi148 ceph-mon[44520]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:37.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:37 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:16:37.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:37 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:16:37.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:37 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:16:37.907 INFO:teuthology.orchestra.run.smithi103.stdout:55834574915 2024-04-23T16:16:37.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:37 smithi188 ceph-mon[44101]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:37.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:37 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:16:37.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:37 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:16:37.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:37 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:16:38.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:38 smithi103 ceph-mon[39629]: osdmap e100: 12 total, 12 up, 12 in 2024-04-23T16:16:38.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:38 smithi103 ceph-mon[39629]: from='client.14679 v1:172.21.15.103:0/49771790' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T16:16:38.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:38 smithi148 ceph-mon[44520]: osdmap e100: 12 total, 12 up, 12 in 2024-04-23T16:16:38.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:38 smithi148 ceph-mon[44520]: from='client.14679 v1:172.21.15.103:0/49771790' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T16:16:38.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:38 smithi188 ceph-mon[44101]: osdmap e100: 12 total, 12 up, 12 in 2024-04-23T16:16:38.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:38 smithi188 ceph-mon[44101]: from='client.14679 v1:172.21.15.103:0/49771790' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T16:16:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:39 smithi103 ceph-mon[39629]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:39 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:16:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:39 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:16:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:39 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:16:39.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:39 smithi148 ceph-mon[44520]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:39 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:16:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:39 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:16:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:39 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:16:39.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:39 smithi188 ceph-mon[44101]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:39 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:16:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:39 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:16:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:39 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:16:41.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:41 smithi103 ceph-mon[39629]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:41.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:41 smithi148 ceph-mon[44520]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:41.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:41 smithi188 ceph-mon[44101]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:42.743 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:43.687 INFO:teuthology.orchestra.run.smithi103.stdout:141733920819 2024-04-23T16:16:43.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:43 smithi103 ceph-mon[39629]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:43.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:43 smithi148 ceph-mon[44520]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:43.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:43 smithi188 ceph-mon[44101]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:44.041 INFO:teuthology.orchestra.run.smithi103.stdout:201863462954 2024-04-23T16:16:44.806 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:16:44.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:44 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3113819792' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T16:16:44.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:44 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1642851081' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T16:16:44.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:44 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3113819792' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T16:16:44.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:44 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1642851081' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T16:16:44.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:44 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3113819792' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T16:16:44.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:44 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1642851081' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T16:16:45.764 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:45 smithi103 ceph-mon[39629]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:45.765 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920814 got 141733920819 for osd.4 2024-04-23T16:16:45.765 DEBUG:teuthology.parallel:result is None 2024-04-23T16:16:45.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:45 smithi148 ceph-mon[44520]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:45.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:45 smithi188 ceph-mon[44101]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:46.705 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574909 got 55834574915 for osd.1 2024-04-23T16:16:46.705 DEBUG:teuthology.parallel:result is None 2024-04-23T16:16:47.276 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462950 got 201863462954 for osd.6 2024-04-23T16:16:47.277 DEBUG:teuthology.parallel:result is None 2024-04-23T16:16:47.573 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:47 smithi103 ceph-mon[39629]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:47.573 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:47 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2350308146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T16:16:47.573 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:47 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:16:47.573 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:47 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:16:47.573 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:47 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2755397871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T16:16:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:47 smithi148 ceph-mon[44520]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:47 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2350308146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T16:16:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:47 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:16:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:47 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:16:47.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:47 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2755397871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T16:16:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:47 smithi188 ceph-mon[44101]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:47 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2350308146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T16:16:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:47 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:16:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:47 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:16:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:47 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2755397871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T16:16:48.302 INFO:teuthology.orchestra.run.smithi103.stdout:90194313278 2024-04-23T16:16:48.730 INFO:teuthology.orchestra.run.smithi103.stdout:34359738441 2024-04-23T16:16:48.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:48 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:16:48.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:48 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:16:48.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:48 smithi103 ceph-mon[39629]: osdmap e101: 12 total, 12 up, 12 in 2024-04-23T16:16:48.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:48 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:16:48.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:48 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:16:48.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:48 smithi148 ceph-mon[44520]: osdmap e101: 12 total, 12 up, 12 in 2024-04-23T16:16:48.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:48 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:16:48.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:48 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:16:48.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:48 smithi188 ceph-mon[44101]: osdmap e101: 12 total, 12 up, 12 in 2024-04-23T16:16:49.703 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:49 smithi103 ceph-mon[39629]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:49.703 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:49 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/954167467' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T16:16:49.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:49 smithi148 ceph-mon[44520]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:49.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:49 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/954167467' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T16:16:49.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:49 smithi188 ceph-mon[44101]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:49.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:49 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/954167467' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T16:16:50.358 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738437 got 34359738441 for osd.0 2024-04-23T16:16:50.358 DEBUG:teuthology.parallel:result is None 2024-04-23T16:16:50.717 INFO:teuthology.orchestra.run.smithi103.stdout:163208757296 2024-04-23T16:16:50.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:50 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3135915448' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T16:16:50.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:50 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3135915448' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T16:16:50.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:50 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3135915448' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T16:16:51.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:51 smithi103 ceph-mon[39629]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:51.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:51 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3385810118' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T16:16:51.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:51 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:16:51.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:51 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:16:51.709 INFO:tasks.cephadm.ceph_manager.ceph:need seq 90194313271 got 90194313278 for osd.2 2024-04-23T16:16:51.709 DEBUG:teuthology.parallel:result is None 2024-04-23T16:16:51.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:51 smithi148 ceph-mon[44520]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:51.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:51 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3385810118' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T16:16:51.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:51 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:16:51.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:51 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:16:51.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:51 smithi188 ceph-mon[44101]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:51 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3385810118' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T16:16:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:51 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:16:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:51 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:16:52.112 INFO:teuthology.orchestra.run.smithi103.stdout:115964117050 2024-04-23T16:16:52.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:52 smithi103 ceph-mon[39629]: osdmap e102: 12 total, 12 up, 12 in 2024-04-23T16:16:52.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:52 smithi148 ceph-mon[44520]: osdmap e102: 12 total, 12 up, 12 in 2024-04-23T16:16:52.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:52 smithi188 ceph-mon[44101]: osdmap e102: 12 total, 12 up, 12 in 2024-04-23T16:16:53.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:53 smithi103 ceph-mon[39629]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:53 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1860696059' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T16:16:53.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:53 smithi148 ceph-mon[44520]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:53 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1860696059' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T16:16:53.940 INFO:teuthology.orchestra.run.smithi103.stdout:283467841569 2024-04-23T16:16:53.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:53 smithi188 ceph-mon[44101]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:53.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:53 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1860696059' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T16:16:54.631 INFO:tasks.cephadm.ceph_manager.ceph:need seq 283467841564 got 283467841569 for osd.8 2024-04-23T16:16:54.631 DEBUG:teuthology.parallel:result is None 2024-04-23T16:16:55.064 INFO:teuthology.orchestra.run.smithi103.stdout:313532612637 2024-04-23T16:16:55.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:54 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/314973194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T16:16:55.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:54 smithi103 ceph-mon[39629]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:55.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:54 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/314973194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T16:16:55.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:54 smithi148 ceph-mon[44520]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:55.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:54 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/314973194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T16:16:55.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:54 smithi188 ceph-mon[44101]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:55.645 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117046 got 115964117050 for osd.3 2024-04-23T16:16:55.645 DEBUG:teuthology.parallel:result is None 2024-04-23T16:16:56.051 INFO:teuthology.orchestra.run.smithi103.stdout:240518168615 2024-04-23T16:16:56.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:56 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4067620525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T16:16:56.353 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612632 got 313532612637 for osd.9 2024-04-23T16:16:56.353 DEBUG:teuthology.parallel:result is None 2024-04-23T16:16:56.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:56 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4067620525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T16:16:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:56 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4067620525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T16:16:57.016 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757291 got 163208757296 for osd.5 2024-04-23T16:16:57.017 DEBUG:teuthology.parallel:result is None 2024-04-23T16:16:57.111 INFO:teuthology.orchestra.run.smithi103.stdout:386547056658 2024-04-23T16:16:57.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:57 smithi103 ceph-mon[39629]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:57 smithi148 ceph-mon[44520]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:57.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:57 smithi188 ceph-mon[44101]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:58.057 INFO:teuthology.orchestra.run.smithi103.stdout:347892351001 2024-04-23T16:16:58.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:58 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3736464914' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T16:16:58.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:58 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3736464914' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T16:16:58.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:58 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3736464914' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T16:16:58.770 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350993 got 347892351001 for osd.10 2024-04-23T16:16:58.770 DEBUG:teuthology.parallel:result is None 2024-04-23T16:16:58.902 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168611 got 240518168615 for osd.7 2024-04-23T16:16:58.902 DEBUG:teuthology.parallel:result is None 2024-04-23T16:16:59.373 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:16:59 smithi103 ceph-mon[39629]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:59.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:16:59 smithi148 ceph-mon[44520]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:59.398 INFO:tasks.cephadm.ceph_manager.ceph:need seq 386547056650 got 386547056658 for osd.11 2024-04-23T16:16:59.398 DEBUG:teuthology.parallel:result is None 2024-04-23T16:16:59.398 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-23T16:16:59.399 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:16:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:16:59 smithi188 ceph-mon[44101]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:16:59.665 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:00.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:00 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:17:00.156 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:00 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:17:00.156 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:00 smithi103 ceph-mon[39629]: osdmap e103: 12 total, 12 up, 12 in 2024-04-23T16:17:00.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:00 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:17:00.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:00 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:17:00.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:00 smithi148 ceph-mon[44520]: osdmap e103: 12 total, 12 up, 12 in 2024-04-23T16:17:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:00 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:17:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:00 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:17:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:00 smithi188 ceph-mon[44101]: osdmap e103: 12 total, 12 up, 12 in 2024-04-23T16:17:01.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:01 smithi103 ceph-mon[39629]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:01.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:01 smithi188 ceph-mon[44101]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:01.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:01 smithi148 ceph-mon[44520]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:01.852 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:17:01.854 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:17:02.257 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:02 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:17:02.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:02 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:17:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:02 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:17:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:02 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:17:02.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:02 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:17:02.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:02 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:17:02.706 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-04-23T16:17:00.420730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":341472,"kb_used_data":8912,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124600352,"statfs":{"total":1151940427776,"available":1151590760448,"internally_reserved":0,"allocated":9125888,"data_stored":4738708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":340374841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.897031"},"pg_stats":[{"pgid":"1.0","version":"16'137","reported_seq":343,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-23T16:16:52.797914+0000","last_change":"2024-04-23T16:15:38.785806+0000","last_active":"2024-04-23T16:16:52.797914+0000","last_peered":"2024-04-23T16:16:52.797914+0000","last_clean":"2024-04-23T16:16:52.797914+0000","last_became_active":"2024-04-23T16:15:36.918552+0000","last_became_peered":"2024-04-23T16:15:36.918552+0000","last_unstale":"2024-04-23T16:16:52.797914+0000","last_undegraded":"2024-04-23T16:16:52.797914+0000","last_fullsized":"2024-04-23T16:16:52.797914+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"16'137","last_scrub_stamp":"2024-04-23T16:15:38.785690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:11:23.401899+0000","last_clean_scrub_stamp":"2024-04-23T16:15:38.785690+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:17:00.623356+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056659,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1204224,"data_stored":835649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":306721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612640,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":306741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841571,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":306741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168617,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1204224,"data_stored":835669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462958,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":306741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31836,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713316,"statfs":{"total":95995035648,"available":95962435584,"internally_reserved":0,"allocated":671744,"data_stored":306741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31836,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713316,"statfs":{"total":95995035648,"available":95962435584,"internally_reserved":0,"allocated":671744,"data_stored":306741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313281,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":306741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117052,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":306741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":306741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757298,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":306741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:17:02.707 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:17:02.971 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:03 smithi188 ceph-mon[44101]: from='client.14751 v1:172.21.15.103:0/598380168' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:17:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:03 smithi188 ceph-mon[44101]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:03.526 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:03 smithi103 ceph-mon[39629]: from='client.14751 v1:172.21.15.103:0/598380168' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:17:03.526 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:03 smithi103 ceph-mon[39629]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:03.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:03 smithi148 ceph-mon[44520]: from='client.14751 v1:172.21.15.103:0/598380168' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:17:03.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:03 smithi148 ceph-mon[44520]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:05.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:04 smithi103 ceph-mon[39629]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:05.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:04 smithi148 ceph-mon[44520]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:05.191 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:17:05.193 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:17:05.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:04 smithi188 ceph-mon[44101]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:05.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:05 smithi103 ceph-mon[39629]: from='client.14757 v1:172.21.15.103:0/24601476' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:17:06.041 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-04-23T16:17:04.421479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":341592,"kb_used_data":9032,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124600232,"statfs":{"total":1151940427776,"available":1151590637568,"internally_reserved":0,"allocated":9248768,"data_stored":4794818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":340374841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002067"},"pg_stats":[{"pgid":"1.0","version":"16'137","reported_seq":343,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-23T16:16:52.797914+0000","last_change":"2024-04-23T16:15:38.785806+0000","last_active":"2024-04-23T16:16:52.797914+0000","last_peered":"2024-04-23T16:16:52.797914+0000","last_clean":"2024-04-23T16:16:52.797914+0000","last_became_active":"2024-04-23T16:15:36.918552+0000","last_became_peered":"2024-04-23T16:15:36.918552+0000","last_unstale":"2024-04-23T16:16:52.797914+0000","last_undegraded":"2024-04-23T16:16:52.797914+0000","last_fullsized":"2024-04-23T16:16:52.797914+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"16'137","last_scrub_stamp":"2024-04-23T16:15:38.785690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:11:23.401899+0000","last_clean_scrub_stamp":"2024-04-23T16:15:38.785690+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:17:00.623356+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056660,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28208,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716944,"statfs":{"total":95995035648,"available":95966150656,"internally_reserved":0,"allocated":1216512,"data_stored":841260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":306721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612641,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":312352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841572,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":312352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168617,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1204224,"data_stored":835669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462959,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":312352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713304,"statfs":{"total":95995035648,"available":95962423296,"internally_reserved":0,"allocated":684032,"data_stored":312352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713304,"statfs":{"total":95995035648,"available":95962423296,"internally_reserved":0,"allocated":684032,"data_stored":312352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313282,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":312352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117053,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":312352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920824,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":312352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757299,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":312352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:17:06.042 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-23T16:17:06.042 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-23T16:17:06.042 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-23T16:17:06.042 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-23T16:17:06.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:05 smithi148 ceph-mon[44520]: from='client.14757 v1:172.21.15.103:0/24601476' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:17:06.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:05 smithi188 ceph-mon[44101]: from='client.14757 v1:172.21.15.103:0/24601476' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:17:06.306 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:06 smithi103 ceph-mon[39629]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:07.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:06 smithi148 ceph-mon[44520]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:07.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:06 smithi188 ceph-mon[44101]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:08.447 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:17:08.447 INFO:teuthology.orchestra.run.smithi103.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-23T16:17:08.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:08 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4198049725' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T16:17:08.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:08 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4198049725' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T16:17:08.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:08 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4198049725' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T16:17:09.257 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-23T16:17:09.258 INFO:tasks.cephadm:Setup complete, yielding 2024-04-23T16:17:09.258 INFO:teuthology.run_tasks:Running task exec... 2024-04-23T16:17:09.268 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T16:17:09.269 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi103.front.sepia.ceph.com 2024-04-23T16:17:09.269 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! ceph balancer status ; do sleep 1 ; done' 2024-04-23T16:17:09.829 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:17:09.829 INFO:teuthology.orchestra.run.smithi103.stdout: "active": true, 2024-04-23T16:17:09.829 INFO:teuthology.orchestra.run.smithi103.stdout: "last_optimize_duration": "0:00:00.000278", 2024-04-23T16:17:09.829 INFO:teuthology.orchestra.run.smithi103.stdout: "last_optimize_started": "Tue Apr 23 16:16:14 2024", 2024-04-23T16:17:09.829 INFO:teuthology.orchestra.run.smithi103.stdout: "mode": "upmap", 2024-04-23T16:17:09.829 INFO:teuthology.orchestra.run.smithi103.stdout: "no_optimization_needed": true, 2024-04-23T16:17:09.829 INFO:teuthology.orchestra.run.smithi103.stdout: "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", 2024-04-23T16:17:09.829 INFO:teuthology.orchestra.run.smithi103.stdout: "plans": [] 2024-04-23T16:17:09.829 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:17:09.841 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:09 smithi103 ceph-mon[39629]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:09.842 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer mode crush-compat' 2024-04-23T16:17:09.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:09 smithi148 ceph-mon[44520]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:09.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:09 smithi188 ceph-mon[44101]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:10.527 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer on' 2024-04-23T16:17:10.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:10 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set create-compat", "format": "json"}]: dispatch 2024-04-23T16:17:10.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:10 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set create-compat", "format": "json"}]: dispatch 2024-04-23T16:17:10.897 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-23T16:17:10.907 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'timeout': 1800, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-23T16:17:10.907 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-yuri3-testing-2024-04-05-0825'}, 'ceph': {'conf': {'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd heartbeat use min delay socket': True, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'mon_bind_msgr2': False, 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuri3-testing-2024-04-05-0825', 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'}} 2024-04-23T16:17:10.907 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'timeout': 1800, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-23T16:17:10.907 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-23T16:17:10.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:10 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set create-compat", "format": "json"}]: dispatch 2024-04-23T16:17:11.169 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:11.642 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:11 smithi103 ceph-mon[39629]: from='client.14769 v1:172.21.15.103:0/1590612991' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mgr", ""]}]: dispatch 2024-04-23T16:17:11.643 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:11 smithi103 ceph-mon[39629]: from='client.14775 v1:172.21.15.103:0/1671018180' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mgr", ""]}]: dispatch 2024-04-23T16:17:11.643 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:11 smithi103 ceph-mon[39629]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:11.643 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:11 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-04-23T16:17:11.643 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:11 smithi103 ceph-mon[39629]: osdmap e104: 12 total, 12 up, 12 in 2024-04-23T16:17:11.643 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:11 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush dump", "format": "json"}]: dispatch 2024-04-23T16:17:11.643 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:11 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:17:11.643 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:11 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:17:11.643 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:11 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:17:11.643 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:11 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:17:11.643 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:11 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:17:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:11 smithi148 ceph-mon[44520]: from='client.14769 v1:172.21.15.103:0/1590612991' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mgr", ""]}]: dispatch 2024-04-23T16:17:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:11 smithi148 ceph-mon[44520]: from='client.14775 v1:172.21.15.103:0/1671018180' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mgr", ""]}]: dispatch 2024-04-23T16:17:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:11 smithi148 ceph-mon[44520]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:11 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-04-23T16:17:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:11 smithi148 ceph-mon[44520]: osdmap e104: 12 total, 12 up, 12 in 2024-04-23T16:17:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:11 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush dump", "format": "json"}]: dispatch 2024-04-23T16:17:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:11 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:17:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:11 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:17:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:11 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:17:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:11 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:17:11.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:11 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:17:11.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:11 smithi188 ceph-mon[44101]: from='client.14769 v1:172.21.15.103:0/1590612991' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mgr", ""]}]: dispatch 2024-04-23T16:17:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:11 smithi188 ceph-mon[44101]: from='client.14775 v1:172.21.15.103:0/1671018180' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mgr", ""]}]: dispatch 2024-04-23T16:17:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:11 smithi188 ceph-mon[44101]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:11 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-04-23T16:17:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:11 smithi188 ceph-mon[44101]: osdmap e104: 12 total, 12 up, 12 in 2024-04-23T16:17:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:11 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush dump", "format": "json"}]: dispatch 2024-04-23T16:17:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:11 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:17:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:11 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:17:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:11 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:17:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:11 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:17:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:11 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:17:12.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:12 smithi103 ceph-mon[39629]: from='client.14781 v1:172.21.15.103:0/2347261986' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mgr", ""]}]: dispatch 2024-04-23T16:17:12.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:12 smithi148 ceph-mon[44520]: from='client.14781 v1:172.21.15.103:0/2347261986' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mgr", ""]}]: dispatch 2024-04-23T16:17:12.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:12 smithi188 ceph-mon[44101]: from='client.14781 v1:172.21.15.103:0/2347261986' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mgr", ""]}]: dispatch 2024-04-23T16:17:13.724 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:13 smithi103 ceph-mon[39629]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:13.724 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:13 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/780640338' entity='client.admin' 2024-04-23T16:17:13.724 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:17:13.724 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:17:13.724 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:17:13.724 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:17:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:13 smithi148 ceph-mon[44520]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:13 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/780640338' entity='client.admin' 2024-04-23T16:17:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:17:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:17:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:17:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:17:13.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:13 smithi188 ceph-mon[44101]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:13 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/780640338' entity='client.admin' 2024-04-23T16:17:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:17:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:17:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:17:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:17:14.139 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-23T16:17:14.140 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-23T16:17:14.140 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:17:14.405 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:14.974 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:14 smithi103 ceph-mon[39629]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:15.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:14 smithi148 ceph-mon[44520]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:14 smithi188 ceph-mon[44101]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:16.055 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:15 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:17:16.055 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:15 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:17:16.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:15 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:17:16.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:15 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:17:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:15 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:17:16.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:15 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:17:16.535 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:17:16.537 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:17:16.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:16 smithi103 ceph-mon[39629]: osdmap e105: 12 total, 12 up, 12 in 2024-04-23T16:17:16.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:16 smithi103 ceph-mon[39629]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:16.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:16 smithi103 ceph-mon[39629]: from='client.14793 v1:172.21.15.103:0/1878888067' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:17:17.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:16 smithi148 ceph-mon[44520]: osdmap e105: 12 total, 12 up, 12 in 2024-04-23T16:17:17.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:16 smithi148 ceph-mon[44520]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:17.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:16 smithi148 ceph-mon[44520]: from='client.14793 v1:172.21.15.103:0/1878888067' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:17:17.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:16 smithi188 ceph-mon[44101]: osdmap e105: 12 total, 12 up, 12 in 2024-04-23T16:17:17.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:16 smithi188 ceph-mon[44101]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:16 smithi188 ceph-mon[44101]: from='client.14793 v1:172.21.15.103:0/1878888067' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:17:17.258 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-04-23T16:17:16.423732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":341760,"kb_used_data":8240,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124600064,"statfs":{"total":1151940427776,"available":1151590465536,"internally_reserved":0,"allocated":8437760,"data_stored":4391404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":340374841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001443"},"pg_stats":[{"pgid":"1.0","version":"16'137","reported_seq":355,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-23T16:17:12.275923+0000","last_change":"2024-04-23T16:17:01.874495+0000","last_active":"2024-04-23T16:17:12.275923+0000","last_peered":"2024-04-23T16:17:12.275923+0000","last_clean":"2024-04-23T16:17:12.275923+0000","last_became_active":"2024-04-23T16:15:36.918552+0000","last_became_peered":"2024-04-23T16:15:36.918552+0000","last_unstale":"2024-04-23T16:17:12.275923+0000","last_undegraded":"2024-04-23T16:17:12.275923+0000","last_fullsized":"2024-04-23T16:17:12.275923+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"16'137","last_scrub_stamp":"2024-04-23T16:17:01.874364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:11:23.401899+0000","last_clean_scrub_stamp":"2024-04-23T16:17:01.874364+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:18:30.555199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056663,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28220,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716932,"statfs":{"total":95995035648,"available":95966138368,"internally_reserved":0,"allocated":1146880,"data_stored":806707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351005,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":614400,"data_stored":277779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612643,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":614400,"data_stored":277799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841574,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":614400,"data_stored":277799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168620,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28220,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716932,"statfs":{"total":95995035648,"available":95966138368,"internally_reserved":0,"allocated":1146880,"data_stored":806727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":614400,"data_stored":277799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31860,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713292,"statfs":{"total":95995035648,"available":95962411008,"internally_reserved":0,"allocated":614400,"data_stored":277799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31860,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713292,"statfs":{"total":95995035648,"available":95962411008,"internally_reserved":0,"allocated":614400,"data_stored":277799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313284,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":614400,"data_stored":277799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117056,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":614400,"data_stored":277799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920826,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":614400,"data_stored":277799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757301,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":614400,"data_stored":277799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:17:17.259 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:17:17.523 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:19.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:19 smithi103 ceph-mon[39629]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:19.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:19 smithi148 ceph-mon[44520]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:19.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:19 smithi188 ceph-mon[44101]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:20.419 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:17:20.422 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:17:21.203 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-04-23T16:17:18.424047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":341624,"kb_used_data":8264,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124600200,"statfs":{"total":1151940427776,"available":1151590604800,"internally_reserved":0,"allocated":8462336,"data_stored":4403146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":340374841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001343"},"pg_stats":[{"pgid":"1.0","version":"16'137","reported_seq":355,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-23T16:17:12.275923+0000","last_change":"2024-04-23T16:17:01.874495+0000","last_active":"2024-04-23T16:17:12.275923+0000","last_peered":"2024-04-23T16:17:12.275923+0000","last_clean":"2024-04-23T16:17:12.275923+0000","last_became_active":"2024-04-23T16:15:36.918552+0000","last_became_peered":"2024-04-23T16:15:36.918552+0000","last_unstale":"2024-04-23T16:17:12.275923+0000","last_undegraded":"2024-04-23T16:17:12.275923+0000","last_fullsized":"2024-04-23T16:17:12.275923+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"16'137","last_scrub_stamp":"2024-04-23T16:17:01.874364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:11:23.401899+0000","last_clean_scrub_stamp":"2024-04-23T16:17:01.874364+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:18:30.555199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056663,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28220,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716932,"statfs":{"total":95995035648,"available":95966138368,"internally_reserved":0,"allocated":1146880,"data_stored":806707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351005,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":614400,"data_stored":277779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612643,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":614400,"data_stored":277799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841575,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717520,"statfs":{"total":95995035648,"available":95966740480,"internally_reserved":0,"allocated":626688,"data_stored":283670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168620,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28220,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716932,"statfs":{"total":95995035648,"available":95966138368,"internally_reserved":0,"allocated":1146880,"data_stored":806727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":614400,"data_stored":277799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31860,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713292,"statfs":{"total":95995035648,"available":95962411008,"internally_reserved":0,"allocated":614400,"data_stored":277799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31860,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713292,"statfs":{"total":95995035648,"available":95962411008,"internally_reserved":0,"allocated":614400,"data_stored":277799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313284,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":614400,"data_stored":277799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117056,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":614400,"data_stored":277799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920826,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":614400,"data_stored":277799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757302,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717520,"statfs":{"total":95995035648,"available":95966740480,"internally_reserved":0,"allocated":626688,"data_stored":283670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:17:21.204 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-23T16:17:21.204 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd dump 2024-04-23T16:17:21.472 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:21.762 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:21 smithi103 ceph-mon[39629]: from='client.14799 v1:172.21.15.103:0/3867941790' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:17:21.762 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:21 smithi103 ceph-mon[39629]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:21.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:21 smithi148 ceph-mon[44520]: from='client.14799 v1:172.21.15.103:0/3867941790' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:17:21.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:21 smithi148 ceph-mon[44520]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:21.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:21 smithi188 ceph-mon[44101]: from='client.14799 v1:172.21.15.103:0/3867941790' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:17:21.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:21 smithi188 ceph-mon[44101]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:23.524 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 105 2024-04-23T16:17:23.524 INFO:teuthology.orchestra.run.smithi103.stdout:fsid 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T16:17:23.525 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-04-23T16:08:13.477590+0000 2024-04-23T16:17:23.525 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-04-23T16:17:15.774861+0000 2024-04-23T16:17:23.525 INFO:teuthology.orchestra.run.smithi103.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-23T16:17:23.525 INFO:teuthology.orchestra.run.smithi103.stdout:crush_version 27 2024-04-23T16:17:23.525 INFO:teuthology.orchestra.run.smithi103.stdout:full_ratio 0.95 2024-04-23T16:17:23.525 INFO:teuthology.orchestra.run.smithi103.stdout:backfillfull_ratio 0.9 2024-04-23T16:17:23.525 INFO:teuthology.orchestra.run.smithi103.stdout:nearfull_ratio 0.85 2024-04-23T16:17:23.525 INFO:teuthology.orchestra.run.smithi103.stdout:require_min_compat_client luminous 2024-04-23T16:17:23.525 INFO:teuthology.orchestra.run.smithi103.stdout:min_compat_client jewel 2024-04-23T16:17:23.525 INFO:teuthology.orchestra.run.smithi103.stdout:require_osd_release squid 2024-04-23T16:17:23.525 INFO:teuthology.orchestra.run.smithi103.stdout:stretch_mode_enabled false 2024-04-23T16:17:23.525 INFO:teuthology.orchestra.run.smithi103.stdout:pool 1 '.mgr' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 17 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 11.76 2024-04-23T16:17:23.525 INFO:teuthology.orchestra.run.smithi103.stdout:max_osd 12 2024-04-23T16:17:23.525 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 up in weight 1 up_from 8 up_thru 58 down_at 0 last_clean_interval [0,0) v1:172.21.15.103:6801/4274040097 v1:172.21.15.103:6802/4274040097 exists,up 4c084047-8f01-4b86-a60c-5592735b5862 2024-04-23T16:17:23.525 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.103:6805/3716921156 v1:172.21.15.103:6806/3716921156 exists,up 3438f2bd-5fb7-432c-b383-29e18c442f20 2024-04-23T16:17:23.526 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.103:6809/424240088 v1:172.21.15.103:6810/424240088 exists,up c17fffb4-8d17-4546-ba24-e5951a2a37fe 2024-04-23T16:17:23.526 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.103:6813/1895183074 v1:172.21.15.103:6814/1895183074 exists,up cee91a3d-55dc-4e89-b568-86821632bb1c 2024-04-23T16:17:23.526 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.148:6800/794945071 v1:172.21.15.148:6801/794945071 exists,up 61d5385b-2c82-42f8-93c3-8f1baf71c65e 2024-04-23T16:17:23.526 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 up in weight 1 up_from 38 up_thru 41 down_at 0 last_clean_interval [0,0) v1:172.21.15.148:6804/3547200711 v1:172.21.15.148:6805/3547200711 exists,up 5d492189-c3cd-47ad-a12a-ed4d275bcfe3 2024-04-23T16:17:23.526 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6 up in weight 1 up_from 47 up_thru 50 down_at 0 last_clean_interval [0,0) v1:172.21.15.148:6808/366919758 v1:172.21.15.148:6809/366919758 exists,up 89f2379e-cf14-4e69-8747-7535db81b70b 2024-04-23T16:17:23.526 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7 up in weight 1 up_from 56 up_thru 91 down_at 0 last_clean_interval [0,0) v1:172.21.15.148:6812/3719550888 v1:172.21.15.148:6813/3719550888 exists,up f27b035a-558f-4528-80ea-2a00d03f6539 2024-04-23T16:17:23.526 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8 up in weight 1 up_from 66 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.188:6800/1151408595 v1:172.21.15.188:6801/1151408595 exists,up a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed 2024-04-23T16:17:23.526 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9 up in weight 1 up_from 73 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.188:6804/306501862 v1:172.21.15.188:6805/306501862 exists,up 124d9b05-11eb-4622-b6a2-c4d5bd60b7e6 2024-04-23T16:17:23.526 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10 up in weight 1 up_from 81 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.188:6808/1231263138 v1:172.21.15.188:6809/1231263138 exists,up 3dfb9070-e534-4ea4-a25e-ee1c51297447 2024-04-23T16:17:23.526 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11 up in weight 1 up_from 90 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.188:6812/457992661 v1:172.21.15.188:6813/457992661 exists,up bc163925-f623-4f63-ab89-35e2d511a730 2024-04-23T16:17:23.526 INFO:teuthology.orchestra.run.smithi103.stdout:blocklist 172.21.15.103:6800/3535470771 expires 2024-04-24T16:09:14.324448+0000 2024-04-23T16:17:23.526 INFO:teuthology.orchestra.run.smithi103.stdout:blocklist 172.21.15.103:0/1188058303 expires 2024-04-24T16:09:14.324448+0000 2024-04-23T16:17:23.526 INFO:teuthology.orchestra.run.smithi103.stdout:blocklist 172.21.15.103:0/1510553721 expires 2024-04-24T16:08:44.450632+0000 2024-04-23T16:17:23.526 INFO:teuthology.orchestra.run.smithi103.stdout:blocklist 172.21.15.103:0/281182801 expires 2024-04-24T16:08:44.450632+0000 2024-04-23T16:17:23.527 INFO:teuthology.orchestra.run.smithi103.stdout:blocklist 172.21.15.103:6800/2049124518 expires 2024-04-24T16:08:44.450632+0000 2024-04-23T16:17:23.527 INFO:teuthology.orchestra.run.smithi103.stdout:blocklist 172.21.15.103:0/466241963 expires 2024-04-24T16:08:44.450632+0000 2024-04-23T16:17:23.527 INFO:teuthology.orchestra.run.smithi103.stdout:blocklist 172.21.15.103:0/2726838698 expires 2024-04-24T16:09:14.324448+0000 2024-04-23T16:17:23.527 INFO:teuthology.orchestra.run.smithi103.stdout:blocklist 172.21.15.103:0/596230573 expires 2024-04-24T16:09:14.324448+0000 2024-04-23T16:17:23.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:23 smithi103 ceph-mon[39629]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:23.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:23 smithi148 ceph-mon[44520]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:23.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:23 smithi188 ceph-mon[44101]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:24.329 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 58 down_at 0 last_clean_interval [0,0) v1:172.21.15.103:6801/4274040097 v1:172.21.15.103:6802/4274040097 exists,up 4c084047-8f01-4b86-a60c-5592735b5862', 'osd.1 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.103:6805/3716921156 v1:172.21.15.103:6806/3716921156 exists,up 3438f2bd-5fb7-432c-b383-29e18c442f20', 'osd.2 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.103:6809/424240088 v1:172.21.15.103:6810/424240088 exists,up c17fffb4-8d17-4546-ba24-e5951a2a37fe', 'osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.103:6813/1895183074 v1:172.21.15.103:6814/1895183074 exists,up cee91a3d-55dc-4e89-b568-86821632bb1c', 'osd.4 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.148:6800/794945071 v1:172.21.15.148:6801/794945071 exists,up 61d5385b-2c82-42f8-93c3-8f1baf71c65e', 'osd.5 up in weight 1 up_from 38 up_thru 41 down_at 0 last_clean_interval [0,0) v1:172.21.15.148:6804/3547200711 v1:172.21.15.148:6805/3547200711 exists,up 5d492189-c3cd-47ad-a12a-ed4d275bcfe3', 'osd.6 up in weight 1 up_from 47 up_thru 50 down_at 0 last_clean_interval [0,0) v1:172.21.15.148:6808/366919758 v1:172.21.15.148:6809/366919758 exists,up 89f2379e-cf14-4e69-8747-7535db81b70b', 'osd.7 up in weight 1 up_from 56 up_thru 91 down_at 0 last_clean_interval [0,0) v1:172.21.15.148:6812/3719550888 v1:172.21.15.148:6813/3719550888 exists,up f27b035a-558f-4528-80ea-2a00d03f6539', 'osd.8 up in weight 1 up_from 66 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.188:6800/1151408595 v1:172.21.15.188:6801/1151408595 exists,up a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed', 'osd.9 up in weight 1 up_from 73 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.188:6804/306501862 v1:172.21.15.188:6805/306501862 exists,up 124d9b05-11eb-4622-b6a2-c4d5bd60b7e6', 'osd.10 up in weight 1 up_from 81 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.188:6808/1231263138 v1:172.21.15.188:6809/1231263138 exists,up 3dfb9070-e534-4ea4-a25e-ee1c51297447', 'osd.11 up in weight 1 up_from 90 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.188:6812/457992661 v1:172.21.15.188:6813/457992661 exists,up bc163925-f623-4f63-ab89-35e2d511a730'] 2024-04-23T16:17:24.330 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon mon.a config show 2024-04-23T16:17:24.596 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:24.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3210278961' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-23T16:17:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:24 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3210278961' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-23T16:17:24.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3210278961' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-23T16:17:25.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:25 smithi103 ceph-mon[39629]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:25 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:17:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:25 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:17:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:25 smithi103 ceph-mon[39629]: osdmap e106: 12 total, 12 up, 12 in 2024-04-23T16:17:25.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:25 smithi148 ceph-mon[44520]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:25.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:25 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:17:25.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:25 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:17:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:25 smithi148 ceph-mon[44520]: osdmap e106: 12 total, 12 up, 12 in 2024-04-23T16:17:25.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:25 smithi188 ceph-mon[44101]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:25.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:25 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:17:25.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:25 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:17:25.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:25 smithi188 ceph-mon[44101]: osdmap e106: 12 total, 12 up, 12 in 2024-04-23T16:17:26.607 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:17:26.607 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "mon.a", 2024-04-23T16:17:26.607 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster": "ceph", 2024-04-23T16:17:26.607 INFO:teuthology.orchestra.run.smithi103.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-23T16:17:26.607 INFO:teuthology.orchestra.run.smithi103.stdout: "admin_socket_mode": "", 2024-04-23T16:17:26.607 INFO:teuthology.orchestra.run.smithi103.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-23T16:17:26.607 INFO:teuthology.orchestra.run.smithi103.stdout: "auth_client_required": "cephx, none", 2024-04-23T16:17:26.608 INFO:teuthology.orchestra.run.smithi103.stdout: "auth_cluster_required": "cephx", 2024-04-23T16:17:26.608 INFO:teuthology.orchestra.run.smithi103.stdout: "auth_debug": "true", 2024-04-23T16:17:26.608 INFO:teuthology.orchestra.run.smithi103.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-23T16:17:26.608 INFO:teuthology.orchestra.run.smithi103.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-23T16:17:26.608 INFO:teuthology.orchestra.run.smithi103.stdout: "auth_service_required": "cephx", 2024-04-23T16:17:26.608 INFO:teuthology.orchestra.run.smithi103.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-23T16:17:26.608 INFO:teuthology.orchestra.run.smithi103.stdout: "auth_supported": "", 2024-04-23T16:17:26.608 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_aio": "true", 2024-04-23T16:17:26.608 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-23T16:17:26.608 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_aio_poll_ms": "250", 2024-04-23T16:17:26.608 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_aio_reap_max": "16", 2024-04-23T16:17:26.608 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_async_discard_threads": "0", 2024-04-23T16:17:26.608 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_block_size": "4096", 2024-04-23T16:17:26.609 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_debug_aio": "false", 2024-04-23T16:17:26.609 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-23T16:17:26.609 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-23T16:17:26.609 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_debug_inflight_ios": "false", 2024-04-23T16:17:26.609 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_enable_discard": "false", 2024-04-23T16:17:26.609 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_flock_retry": "3", 2024-04-23T16:17:26.609 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-23T16:17:26.610 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_inject_crash": "0", 2024-04-23T16:17:26.610 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-23T16:17:26.610 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_ioring": "false", 2024-04-23T16:17:26.610 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_ioring_hipri": "false", 2024-04-23T16:17:26.610 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-23T16:17:26.610 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-23T16:17:26.610 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-23T16:17:26.610 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-23T16:17:26.611 INFO:teuthology.orchestra.run.smithi103.stdout: "bdev_type": "", 2024-04-23T16:17:26.611 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_alloc_size": "1048576", 2024-04-23T16:17:26.611 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_allocator": "hybrid", 2024-04-23T16:17:26.611 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_buffered_io": "true", 2024-04-23T16:17:26.611 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_check_for_zeros": "false", 2024-04-23T16:17:26.611 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-23T16:17:26.611 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-23T16:17:26.611 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_compact_log_sync": "false", 2024-04-23T16:17:26.611 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-23T16:17:26.611 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-23T16:17:26.611 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-23T16:17:26.611 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-23T16:17:26.611 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_max_log_runway": "4194304", 2024-04-23T16:17:26.611 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_max_prefetch": "1048576", 2024-04-23T16:17:26.612 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_min_flush_size": "524288", 2024-04-23T16:17:26.612 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_min_log_runway": "1048576", 2024-04-23T16:17:26.612 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_replay_recovery": "false", 2024-04-23T16:17:26.612 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-23T16:17:26.612 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-23T16:17:26.612 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_sync_write": "false", 2024-04-23T16:17:26.612 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-23T16:17:26.612 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-23T16:17:26.612 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-23T16:17:26.612 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_allocation_from_file": "true", 2024-04-23T16:17:26.612 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_allocator": "hybrid", 2024-04-23T16:17:26.612 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-23T16:17:26.612 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-23T16:17:26.613 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-23T16:17:26.613 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-23T16:17:26.613 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-23T16:17:26.613 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-23T16:17:26.613 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-23T16:17:26.613 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_block_create": "true", 2024-04-23T16:17:26.613 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_block_db_create": "false", 2024-04-23T16:17:26.613 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_block_db_path": "", 2024-04-23T16:17:26.613 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_block_db_size": "0", 2024-04-23T16:17:26.613 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_block_path": "", 2024-04-23T16:17:26.613 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_block_preallocate_file": "false", 2024-04-23T16:17:26.613 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_block_size": "107374182400", 2024-04-23T16:17:26.613 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_block_wal_create": "false", 2024-04-23T16:17:26.613 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_block_wal_path": "", 2024-04-23T16:17:26.613 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_block_wal_size": "100663296", 2024-04-23T16:17:26.614 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_bluefs": "true", 2024-04-23T16:17:26.614 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-23T16:17:26.614 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-23T16:17:26.614 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-23T16:17:26.614 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-23T16:17:26.614 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-23T16:17:26.614 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-23T16:17:26.614 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-23T16:17:26.614 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-23T16:17:26.614 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_autotune": "true", 2024-04-23T16:17:26.614 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-23T16:17:26.614 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-23T16:17:26.615 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-23T16:17:26.615 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-23T16:17:26.615 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_size": "0", 2024-04-23T16:17:26.615 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-23T16:17:26.615 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-23T16:17:26.615 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-23T16:17:26.615 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-23T16:17:26.615 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_type": "2q", 2024-04-23T16:17:26.615 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-23T16:17:26.615 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_clone_cow": "true", 2024-04-23T16:17:26.615 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-23T16:17:26.615 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-23T16:17:26.615 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-23T16:17:26.616 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-23T16:17:26.616 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-23T16:17:26.616 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-23T16:17:26.616 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-23T16:17:26.616 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_compression_mode": "none", 2024-04-23T16:17:26.616 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-23T16:17:26.616 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_csum_type": "crc32c", 2024-04-23T16:17:26.616 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-04-23T16:17:26.616 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-23T16:17:26.616 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_debug_freelist": "false", 2024-04-23T16:17:26.616 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-23T16:17:26.616 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-23T16:17:26.616 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-23T16:17:26.616 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-23T16:17:26.617 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-23T16:17:26.617 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-23T16:17:26.617 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-23T16:17:26.617 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-23T16:17:26.617 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-23T16:17:26.617 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-23T16:17:26.617 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-23T16:17:26.617 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-23T16:17:26.617 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-23T16:17:26.617 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_debug_small_allocations": "0", 2024-04-23T16:17:26.617 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-23T16:17:26.617 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_default_buffered_read": "true", 2024-04-23T16:17:26.617 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_default_buffered_write": "false", 2024-04-23T16:17:26.618 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-23T16:17:26.618 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-23T16:17:26.618 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-23T16:17:26.618 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-23T16:17:26.618 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-23T16:17:26.618 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-23T16:17:26.618 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-23T16:17:26.618 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-23T16:17:26.618 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-23T16:17:26.618 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fail_eio": "false", 2024-04-23T16:17:26.618 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-23T16:17:26.618 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-23T16:17:26.618 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-23T16:17:26.619 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-23T16:17:26.619 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-23T16:17:26.619 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-23T16:17:26.619 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_on_mount": "false", 2024-04-23T16:17:26.619 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-23T16:17:26.619 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_on_umount": "false", 2024-04-23T16:17:26.619 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-23T16:17:26.619 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-23T16:17:26.619 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-23T16:17:26.619 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-23T16:17:26.619 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-23T16:17:26.619 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-23T16:17:26.619 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-23T16:17:26.620 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-23T16:17:26.620 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_ignore_data_csum": "false", 2024-04-23T16:17:26.620 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-23T16:17:26.620 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-23T16:17:26.620 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-23T16:17:26.620 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-23T16:17:26.620 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_log_op_age": "5.000000", 2024-04-23T16:17:26.620 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_max_alloc_size": "0", 2024-04-23T16:17:26.620 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_max_blob_size": "0", 2024-04-23T16:17:26.620 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-23T16:17:26.620 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-23T16:17:26.620 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-23T16:17:26.620 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_max_deferred_txc": "32", 2024-04-23T16:17:26.620 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_min_alloc_size": "0", 2024-04-23T16:17:26.621 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-23T16:17:26.621 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-23T16:17:26.621 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_nid_prealloc": "1024", 2024-04-23T16:17:26.621 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-23T16:17:26.621 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-23T16:17:26.621 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-23T16:17:26.621 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-23T16:17:26.621 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_retry_disk_reads": "3", 2024-04-23T16:17:26.621 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_rocksdb_cf": "true", 2024-04-23T16:17:26.621 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-23T16:17:26.621 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-23T16:17:26.621 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-23T16:17:26.621 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-23T16:17:26.622 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-23T16:17:26.622 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-23T16:17:26.622 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_spdk_mem": "512", 2024-04-23T16:17:26.622 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-23T16:17:26.622 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-23T16:17:26.622 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-23T16:17:26.622 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-23T16:17:26.622 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-23T16:17:26.622 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-23T16:17:26.622 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-23T16:17:26.622 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_tracing": "false", 2024-04-23T16:17:26.622 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-23T16:17:26.622 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-23T16:17:26.623 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-23T16:17:26.623 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-23T16:17:26.623 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-23T16:17:26.623 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-23T16:17:26.623 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-23T16:17:26.623 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-23T16:17:26.623 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-23T16:17:26.623 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_zero_block_detection": "false", 2024-04-23T16:17:26.623 INFO:teuthology.orchestra.run.smithi103.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-23T16:17:26.623 INFO:teuthology.orchestra.run.smithi103.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-23T16:17:26.623 INFO:teuthology.orchestra.run.smithi103.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-23T16:17:26.623 INFO:teuthology.orchestra.run.smithi103.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-23T16:17:26.623 INFO:teuthology.orchestra.run.smithi103.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-23T16:17:26.624 INFO:teuthology.orchestra.run.smithi103.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-23T16:17:26.624 INFO:teuthology.orchestra.run.smithi103.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-23T16:17:26.624 INFO:teuthology.orchestra.run.smithi103.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-23T16:17:26.624 INFO:teuthology.orchestra.run.smithi103.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-23T16:17:26.624 INFO:teuthology.orchestra.run.smithi103.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-23T16:17:26.624 INFO:teuthology.orchestra.run.smithi103.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-23T16:17:26.624 INFO:teuthology.orchestra.run.smithi103.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-23T16:17:26.624 INFO:teuthology.orchestra.run.smithi103.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-23T16:17:26.624 INFO:teuthology.orchestra.run.smithi103.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-23T16:17:26.624 INFO:teuthology.orchestra.run.smithi103.stdout: "cephx_cluster_require_signatures": "false", 2024-04-23T16:17:26.624 INFO:teuthology.orchestra.run.smithi103.stdout: "cephx_cluster_require_version": "2", 2024-04-23T16:17:26.624 INFO:teuthology.orchestra.run.smithi103.stdout: "cephx_require_signatures": "false", 2024-04-23T16:17:26.624 INFO:teuthology.orchestra.run.smithi103.stdout: "cephx_require_version": "2", 2024-04-23T16:17:26.624 INFO:teuthology.orchestra.run.smithi103.stdout: "cephx_service_require_signatures": "false", 2024-04-23T16:17:26.625 INFO:teuthology.orchestra.run.smithi103.stdout: "cephx_service_require_version": "2", 2024-04-23T16:17:26.625 INFO:teuthology.orchestra.run.smithi103.stdout: "cephx_sign_messages": "true", 2024-04-23T16:17:26.625 INFO:teuthology.orchestra.run.smithi103.stdout: "chdir": "", 2024-04-23T16:17:26.625 INFO:teuthology.orchestra.run.smithi103.stdout: "client_acl_type": "", 2024-04-23T16:17:26.625 INFO:teuthology.orchestra.run.smithi103.stdout: "client_asio_thread_count": "2", 2024-04-23T16:17:26.625 INFO:teuthology.orchestra.run.smithi103.stdout: "client_cache_mid": "0.750000", 2024-04-23T16:17:26.625 INFO:teuthology.orchestra.run.smithi103.stdout: "client_cache_size": "16384", 2024-04-23T16:17:26.625 INFO:teuthology.orchestra.run.smithi103.stdout: "client_caps_release_delay": "5", 2024-04-23T16:17:26.625 INFO:teuthology.orchestra.run.smithi103.stdout: "client_check_pool_perm": "true", 2024-04-23T16:17:26.625 INFO:teuthology.orchestra.run.smithi103.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-23T16:17:26.625 INFO:teuthology.orchestra.run.smithi103.stdout: "client_debug_force_sync_read": "false", 2024-04-23T16:17:26.625 INFO:teuthology.orchestra.run.smithi103.stdout: "client_debug_getattr_caps": "false", 2024-04-23T16:17:26.625 INFO:teuthology.orchestra.run.smithi103.stdout: "client_debug_inject_tick_delay": "0", 2024-04-23T16:17:26.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-23T16:17:26.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client_die_on_failed_remount": "false", 2024-04-23T16:17:26.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client_dirsize_rbytes": "true", 2024-04-23T16:17:26.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client_force_lazyio": "false", 2024-04-23T16:17:26.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client_fs": "", 2024-04-23T16:17:26.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-23T16:17:26.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client_inject_release_failure": "false", 2024-04-23T16:17:26.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client_max_inline_size": "4096", 2024-04-23T16:17:26.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-23T16:17:26.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client_mds_namespace": "", 2024-04-23T16:17:26.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client_metadata": "", 2024-04-23T16:17:26.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client_mount_gid": "-1", 2024-04-23T16:17:26.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client_mount_timeout": "300", 2024-04-23T16:17:26.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client_mount_uid": "-1", 2024-04-23T16:17:26.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client_mountpoint": "/", 2024-04-23T16:17:26.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client_notify_timeout": "10", 2024-04-23T16:17:26.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client_oc": "true", 2024-04-23T16:17:26.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client_oc_max_dirty": "104857600", 2024-04-23T16:17:26.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-23T16:17:26.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client_oc_max_objects": "1000", 2024-04-23T16:17:26.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client_oc_size": "209715200", 2024-04-23T16:17:26.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client_oc_target_dirty": "8388608", 2024-04-23T16:17:26.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client_permissions": "true", 2024-04-23T16:17:26.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client_quota": "true", 2024-04-23T16:17:26.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client_quota_df": "true", 2024-04-23T16:17:26.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client_readahead_max_bytes": "0", 2024-04-23T16:17:26.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client_readahead_max_periods": "4", 2024-04-23T16:17:26.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client_readahead_min": "131072", 2024-04-23T16:17:26.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client_reconnect_stale": "false", 2024-04-23T16:17:26.628 INFO:teuthology.orchestra.run.smithi103.stdout: "client_shutdown_timeout": "30", 2024-04-23T16:17:26.628 INFO:teuthology.orchestra.run.smithi103.stdout: "client_snapdir": ".snap", 2024-04-23T16:17:26.628 INFO:teuthology.orchestra.run.smithi103.stdout: "client_tick_interval": "1", 2024-04-23T16:17:26.628 INFO:teuthology.orchestra.run.smithi103.stdout: "client_trace": "", 2024-04-23T16:17:26.628 INFO:teuthology.orchestra.run.smithi103.stdout: "client_try_dentry_invalidate": "false", 2024-04-23T16:17:26.628 INFO:teuthology.orchestra.run.smithi103.stdout: "client_use_faked_inos": "false", 2024-04-23T16:17:26.628 INFO:teuthology.orchestra.run.smithi103.stdout: "client_use_random_mds": "false", 2024-04-23T16:17:26.628 INFO:teuthology.orchestra.run.smithi103.stdout: "clog_to_graylog": "false", 2024-04-23T16:17:26.628 INFO:teuthology.orchestra.run.smithi103.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-23T16:17:26.628 INFO:teuthology.orchestra.run.smithi103.stdout: "clog_to_graylog_port": "12201", 2024-04-23T16:17:26.628 INFO:teuthology.orchestra.run.smithi103.stdout: "clog_to_monitors": "default=true", 2024-04-23T16:17:26.628 INFO:teuthology.orchestra.run.smithi103.stdout: "clog_to_syslog": "false", 2024-04-23T16:17:26.628 INFO:teuthology.orchestra.run.smithi103.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-23T16:17:26.628 INFO:teuthology.orchestra.run.smithi103.stdout: "clog_to_syslog_level": "info", 2024-04-23T16:17:26.629 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "-", 2024-04-23T16:17:26.629 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_network": "", 2024-04-23T16:17:26.629 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_network_interface": "", 2024-04-23T16:17:26.629 INFO:teuthology.orchestra.run.smithi103.stdout: "compressor_zlib_isal": "false", 2024-04-23T16:17:26.629 INFO:teuthology.orchestra.run.smithi103.stdout: "compressor_zlib_level": "5", 2024-04-23T16:17:26.629 INFO:teuthology.orchestra.run.smithi103.stdout: "compressor_zlib_winsize": "-15", 2024-04-23T16:17:26.629 INFO:teuthology.orchestra.run.smithi103.stdout: "compressor_zstd_level": "1", 2024-04-23T16:17:26.629 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:4b8fadbb2850222b1a0d53f94662568f540a1d3f86861435d9eb9005f9a5b779", 2024-04-23T16:17:26.629 INFO:teuthology.orchestra.run.smithi103.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-23T16:17:26.629 INFO:teuthology.orchestra.run.smithi103.stdout: "crimson_alien_op_num_threads": "6", 2024-04-23T16:17:26.629 INFO:teuthology.orchestra.run.smithi103.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-23T16:17:26.629 INFO:teuthology.orchestra.run.smithi103.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-23T16:17:26.629 INFO:teuthology.orchestra.run.smithi103.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-23T16:17:26.629 INFO:teuthology.orchestra.run.smithi103.stdout: "crimson_osd_stat_interval": "0", 2024-04-23T16:17:26.629 INFO:teuthology.orchestra.run.smithi103.stdout: "crimson_seastar_cpu_cores": "", 2024-04-23T16:17:26.630 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_location": "", 2024-04-23T16:17:26.630 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_location_hook": "", 2024-04-23T16:17:26.630 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_location_hook_timeout": "10", 2024-04-23T16:17:26.630 INFO:teuthology.orchestra.run.smithi103.stdout: "daemonize": "false", 2024-04-23T16:17:26.630 INFO:teuthology.orchestra.run.smithi103.stdout: "daos_pool": "tank", 2024-04-23T16:17:26.630 INFO:teuthology.orchestra.run.smithi103.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-23T16:17:26.630 INFO:teuthology.orchestra.run.smithi103.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-23T16:17:26.630 INFO:teuthology.orchestra.run.smithi103.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-23T16:17:26.630 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_alienstore": "0/5", 2024-04-23T16:17:26.630 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_allow_any_pool_priority": "false", 2024-04-23T16:17:26.630 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_asok": "1/5", 2024-04-23T16:17:26.630 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_asok_assert_abort": "false", 2024-04-23T16:17:26.630 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_asserts_on_shutdown": "true", 2024-04-23T16:17:26.630 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_auth": "1/5", 2024-04-23T16:17:26.631 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_bdev": "1/3", 2024-04-23T16:17:26.631 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_bluefs": "1/5", 2024-04-23T16:17:26.631 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_bluestore": "1/5", 2024-04-23T16:17:26.631 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_buffer": "0/1", 2024-04-23T16:17:26.631 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_ceph_exporter": "1/5", 2024-04-23T16:17:26.631 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_cephfs_mirror": "0/5", 2024-04-23T16:17:26.631 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_cephsqlite": "0/5", 2024-04-23T16:17:26.631 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_client": "0/5", 2024-04-23T16:17:26.631 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_compressor": "1/5", 2024-04-23T16:17:26.631 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_context": "0/1", 2024-04-23T16:17:26.631 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_crimson_interrupt": "0/5", 2024-04-23T16:17:26.631 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_crush": "1/1", 2024-04-23T16:17:26.631 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_crypto": "1/5", 2024-04-23T16:17:26.631 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_cyanstore": "0/5", 2024-04-23T16:17:26.631 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_deliberately_leak_memory": "false", 2024-04-23T16:17:26.632 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_disable_randomized_ping": "false", 2024-04-23T16:17:26.632 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_dpdk": "1/5", 2024-04-23T16:17:26.632 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_eventtrace": "1/5", 2024-04-23T16:17:26.632 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_filer": "0/1", 2024-04-23T16:17:26.632 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_filestore": "1/3", 2024-04-23T16:17:26.632 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_finisher": "1/1", 2024-04-23T16:17:26.632 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_fuse": "1/5", 2024-04-23T16:17:26.632 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_heartbeat_testing_span": "0", 2024-04-23T16:17:26.632 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_heartbeatmap": "1/5", 2024-04-23T16:17:26.632 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-23T16:17:26.632 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_javaclient": "1/5", 2024-04-23T16:17:26.632 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_journal": "1/3", 2024-04-23T16:17:26.632 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_journaler": "0/5", 2024-04-23T16:17:26.632 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_kstore": "1/5", 2024-04-23T16:17:26.633 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_lockdep": "0/1", 2024-04-23T16:17:26.633 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_mclock": "1/5", 2024-04-23T16:17:26.633 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_mds": "1/5", 2024-04-23T16:17:26.633 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_mds_balancer": "1/5", 2024-04-23T16:17:26.633 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_mds_locker": "1/5", 2024-04-23T16:17:26.633 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_mds_log": "1/5", 2024-04-23T16:17:26.633 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_mds_log_expire": "1/5", 2024-04-23T16:17:26.633 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_mds_migrator": "1/5", 2024-04-23T16:17:26.633 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_mds_quiesce": "3/5", 2024-04-23T16:17:26.633 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_memstore": "1/5", 2024-04-23T16:17:26.633 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_mgr": "2/5", 2024-04-23T16:17:26.633 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_mgrc": "1/5", 2024-04-23T16:17:26.633 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_mon": "20/20", 2024-04-23T16:17:26.634 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_monc": "0/10", 2024-04-23T16:17:26.634 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_ms": "1/1", 2024-04-23T16:17:26.634 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_none": "0/5", 2024-04-23T16:17:26.634 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_objclass": "0/5", 2024-04-23T16:17:26.634 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_objectcacher": "0/5", 2024-04-23T16:17:26.634 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_objecter": "0/1", 2024-04-23T16:17:26.634 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_optracker": "0/5", 2024-04-23T16:17:26.634 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_osd": "1/5", 2024-04-23T16:17:26.634 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_paxos": "20/20", 2024-04-23T16:17:26.634 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_perfcounter": "1/5", 2024-04-23T16:17:26.634 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_prioritycache": "1/5", 2024-04-23T16:17:26.634 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_rados": "0/5", 2024-04-23T16:17:26.634 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_rbd": "0/5", 2024-04-23T16:17:26.634 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_rbd_mirror": "0/5", 2024-04-23T16:17:26.635 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_rbd_pwl": "0/5", 2024-04-23T16:17:26.635 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_rbd_replay": "0/5", 2024-04-23T16:17:26.635 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_refs": "0/0", 2024-04-23T16:17:26.635 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_reserver": "1/1", 2024-04-23T16:17:26.635 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_rgw": "1/5", 2024-04-23T16:17:26.635 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_rgw_access": "1/5", 2024-04-23T16:17:26.635 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_rgw_datacache": "1/5", 2024-04-23T16:17:26.635 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_rgw_dbstore": "1/5", 2024-04-23T16:17:26.635 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_rgw_flight": "1/5", 2024-04-23T16:17:26.635 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-23T16:17:26.635 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_rgw_notification": "1/5", 2024-04-23T16:17:26.635 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_rgw_sync": "1/5", 2024-04-23T16:17:26.635 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_rocksdb": "4/5", 2024-04-23T16:17:26.635 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_seastore": "0/5", 2024-04-23T16:17:26.636 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_seastore_backref": "0/5", 2024-04-23T16:17:26.636 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_seastore_cache": "0/5", 2024-04-23T16:17:26.636 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_seastore_cleaner": "0/5", 2024-04-23T16:17:26.636 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_seastore_device": "0/5", 2024-04-23T16:17:26.636 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_seastore_epm": "0/5", 2024-04-23T16:17:26.636 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-23T16:17:26.636 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_seastore_journal": "0/5", 2024-04-23T16:17:26.636 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_seastore_lba": "0/5", 2024-04-23T16:17:26.636 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_seastore_odata": "0/5", 2024-04-23T16:17:26.636 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_seastore_omap": "0/5", 2024-04-23T16:17:26.636 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_seastore_onode": "0/5", 2024-04-23T16:17:26.636 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_seastore_t": "0/5", 2024-04-23T16:17:26.636 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_seastore_tm": "0/5", 2024-04-23T16:17:26.636 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_striper": "0/1", 2024-04-23T16:17:26.637 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_test": "0/5", 2024-04-23T16:17:26.637 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_throttle": "1/1", 2024-04-23T16:17:26.637 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_timer": "0/1", 2024-04-23T16:17:26.637 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_tp": "0/5", 2024-04-23T16:17:26.637 INFO:teuthology.orchestra.run.smithi103.stdout: "debug_trace": "1/5", 2024-04-23T16:17:26.637 INFO:teuthology.orchestra.run.smithi103.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-23T16:17:26.637 INFO:teuthology.orchestra.run.smithi103.stdout: "device_failure_prediction_mode": "none", 2024-04-23T16:17:26.637 INFO:teuthology.orchestra.run.smithi103.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-23T16:17:26.637 INFO:teuthology.orchestra.run.smithi103.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-23T16:17:26.637 INFO:teuthology.orchestra.run.smithi103.stdout: "err_to_graylog": "false", 2024-04-23T16:17:26.637 INFO:teuthology.orchestra.run.smithi103.stdout: "err_to_journald": "false", 2024-04-23T16:17:26.637 INFO:teuthology.orchestra.run.smithi103.stdout: "err_to_stderr": "true", 2024-04-23T16:17:26.637 INFO:teuthology.orchestra.run.smithi103.stdout: "err_to_syslog": "false", 2024-04-23T16:17:26.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event_tracing": "false", 2024-04-23T16:17:26.638 INFO:teuthology.orchestra.run.smithi103.stdout: "exporter_addr": "0.0.0.0", 2024-04-23T16:17:26.638 INFO:teuthology.orchestra.run.smithi103.stdout: "exporter_http_port": "9926", 2024-04-23T16:17:26.638 INFO:teuthology.orchestra.run.smithi103.stdout: "exporter_prio_limit": "5", 2024-04-23T16:17:26.638 INFO:teuthology.orchestra.run.smithi103.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-23T16:17:26.638 INFO:teuthology.orchestra.run.smithi103.stdout: "exporter_sort_metrics": "true", 2024-04-23T16:17:26.638 INFO:teuthology.orchestra.run.smithi103.stdout: "exporter_stats_period": "5", 2024-04-23T16:17:26.638 INFO:teuthology.orchestra.run.smithi103.stdout: "fake_statfs_for_testing": "0", 2024-04-23T16:17:26.638 INFO:teuthology.orchestra.run.smithi103.stdout: "fatal_signal_handlers": "true", 2024-04-23T16:17:26.638 INFO:teuthology.orchestra.run.smithi103.stdout: "filer_max_purge_ops": "10", 2024-04-23T16:17:26.638 INFO:teuthology.orchestra.run.smithi103.stdout: "filer_max_truncate_ops": "128", 2024-04-23T16:17:26.638 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_apply_finisher_threads": "1", 2024-04-23T16:17:26.638 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_blackhole": "false", 2024-04-23T16:17:26.638 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_btrfs_clone_range": "true", 2024-04-23T16:17:26.639 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_btrfs_snap": "true", 2024-04-23T16:17:26.639 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_caller_concurrency": "10", 2024-04-23T16:17:26.639 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_collect_device_partition_information": "true", 2024-04-23T16:17:26.639 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_commit_timeout": "600.000000", 2024-04-23T16:17:26.639 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_debug_inject_read_err": "false", 2024-04-23T16:17:26.639 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_debug_omap_check": "false", 2024-04-23T16:17:26.639 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_debug_verify_split": "false", 2024-04-23T16:17:26.639 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_dump_file": "", 2024-04-23T16:17:26.639 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-23T16:17:26.639 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-23T16:17:26.639 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_fadvise": "true", 2024-04-23T16:17:26.639 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_fail_eio": "true", 2024-04-23T16:17:26.639 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_fd_cache_shards": "16", 2024-04-23T16:17:26.640 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_fd_cache_size": "128", 2024-04-23T16:17:26.640 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_fiemap": "false", 2024-04-23T16:17:26.640 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_fiemap_threshold": "4096", 2024-04-23T16:17:26.640 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-23T16:17:26.640 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-23T16:17:26.640 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_inject_stall": "0", 2024-04-23T16:17:26.640 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_journal_parallel": "false", 2024-04-23T16:17:26.640 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_journal_trailing": "false", 2024-04-23T16:17:26.640 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_journal_writeahead": "false", 2024-04-23T16:17:26.640 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_kill_at": "0", 2024-04-23T16:17:26.640 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-23T16:17:26.640 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-23T16:17:26.640 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-23T16:17:26.640 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-23T16:17:26.641 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-23T16:17:26.641 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_max_inline_xattrs": "0", 2024-04-23T16:17:26.641 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-23T16:17:26.641 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-23T16:17:26.641 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-23T16:17:26.641 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-23T16:17:26.641 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_max_xattr_value_size": "0", 2024-04-23T16:17:26.641 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-23T16:17:26.641 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-23T16:17:26.641 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-23T16:17:26.641 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_merge_threshold": "-10", 2024-04-23T16:17:26.641 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-23T16:17:26.641 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_odsync_write": "false", 2024-04-23T16:17:26.641 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_omap_backend": "rocksdb", 2024-04-23T16:17:26.642 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_omap_backend_path": "", 2024-04-23T16:17:26.642 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-23T16:17:26.642 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-23T16:17:26.642 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-23T16:17:26.642 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_op_thread_timeout": "60", 2024-04-23T16:17:26.642 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_op_threads": "2", 2024-04-23T16:17:26.642 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_punch_hole": "false", 2024-04-23T16:17:26.642 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-23T16:17:26.642 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-23T16:17:26.642 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-23T16:17:26.642 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-23T16:17:26.642 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-23T16:17:26.642 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-23T16:17:26.642 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-23T16:17:26.643 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-23T16:17:26.643 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-23T16:17:26.643 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_queue_max_ops": "50", 2024-04-23T16:17:26.643 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-23T16:17:26.643 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_seek_data_hole": "false", 2024-04-23T16:17:26.643 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_sloppy_crc": "false", 2024-04-23T16:17:26.643 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-23T16:17:26.643 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_splice": "false", 2024-04-23T16:17:26.643 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_split_multiple": "2", 2024-04-23T16:17:26.643 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_split_rand_factor": "20", 2024-04-23T16:17:26.643 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_update_to": "1000", 2024-04-23T16:17:26.643 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-23T16:17:26.643 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-23T16:17:26.643 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-23T16:17:26.643 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-23T16:17:26.644 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-23T16:17:26.644 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-23T16:17:26.644 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_wbthrottle_enable": "true", 2024-04-23T16:17:26.644 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-23T16:17:26.644 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-23T16:17:26.644 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-23T16:17:26.644 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-23T16:17:26.644 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-23T16:17:26.644 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-23T16:17:26.644 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_xfs_extsize": "false", 2024-04-23T16:17:26.644 INFO:teuthology.orchestra.run.smithi103.stdout: "filestore_zfs_snap": "false", 2024-04-23T16:17:26.644 INFO:teuthology.orchestra.run.smithi103.stdout: "fio_dir": "/tmp/fio", 2024-04-23T16:17:26.644 INFO:teuthology.orchestra.run.smithi103.stdout: "fsid": "8e3594f2-018b-11ef-bc93-c7b262605968", 2024-04-23T16:17:26.644 INFO:teuthology.orchestra.run.smithi103.stdout: "fuse_allow_other": "true", 2024-04-23T16:17:26.644 INFO:teuthology.orchestra.run.smithi103.stdout: "fuse_atomic_o_trunc": "true", 2024-04-23T16:17:26.644 INFO:teuthology.orchestra.run.smithi103.stdout: "fuse_big_writes": "true", 2024-04-23T16:17:26.645 INFO:teuthology.orchestra.run.smithi103.stdout: "fuse_debug": "false", 2024-04-23T16:17:26.645 INFO:teuthology.orchestra.run.smithi103.stdout: "fuse_default_permissions": "false", 2024-04-23T16:17:26.645 INFO:teuthology.orchestra.run.smithi103.stdout: "fuse_disable_pagecache": "false", 2024-04-23T16:17:26.645 INFO:teuthology.orchestra.run.smithi103.stdout: "fuse_max_write": "0", 2024-04-23T16:17:26.645 INFO:teuthology.orchestra.run.smithi103.stdout: "fuse_multithreaded": "true", 2024-04-23T16:17:26.645 INFO:teuthology.orchestra.run.smithi103.stdout: "fuse_require_active_mds": "true", 2024-04-23T16:17:26.645 INFO:teuthology.orchestra.run.smithi103.stdout: "fuse_set_user_groups": "true", 2024-04-23T16:17:26.645 INFO:teuthology.orchestra.run.smithi103.stdout: "fuse_splice_move": "true", 2024-04-23T16:17:26.645 INFO:teuthology.orchestra.run.smithi103.stdout: "fuse_splice_read": "true", 2024-04-23T16:17:26.645 INFO:teuthology.orchestra.run.smithi103.stdout: "fuse_splice_write": "true", 2024-04-23T16:17:26.645 INFO:teuthology.orchestra.run.smithi103.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-23T16:17:26.645 INFO:teuthology.orchestra.run.smithi103.stdout: "fuse_use_invalidate_cb": "true", 2024-04-23T16:17:26.645 INFO:teuthology.orchestra.run.smithi103.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-23T16:17:26.645 INFO:teuthology.orchestra.run.smithi103.stdout: "gss_target_name": "ceph", 2024-04-23T16:17:26.645 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_file": "", 2024-04-23T16:17:26.646 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_inject_failure": "0", 2024-04-23T16:17:26.646 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_interval": "5", 2024-04-23T16:17:26.646 INFO:teuthology.orchestra.run.smithi103.stdout: "host": "smithi103", 2024-04-23T16:17:26.646 INFO:teuthology.orchestra.run.smithi103.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-23T16:17:26.646 INFO:teuthology.orchestra.run.smithi103.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-23T16:17:26.646 INFO:teuthology.orchestra.run.smithi103.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-23T16:17:26.646 INFO:teuthology.orchestra.run.smithi103.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-23T16:17:26.646 INFO:teuthology.orchestra.run.smithi103.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-23T16:17:26.646 INFO:teuthology.orchestra.run.smithi103.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-23T16:17:26.646 INFO:teuthology.orchestra.run.smithi103.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-23T16:17:26.646 INFO:teuthology.orchestra.run.smithi103.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-23T16:17:26.646 INFO:teuthology.orchestra.run.smithi103.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-23T16:17:26.646 INFO:teuthology.orchestra.run.smithi103.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-23T16:17:26.646 INFO:teuthology.orchestra.run.smithi103.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-23T16:17:26.647 INFO:teuthology.orchestra.run.smithi103.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-23T16:17:26.647 INFO:teuthology.orchestra.run.smithi103.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-23T16:17:26.647 INFO:teuthology.orchestra.run.smithi103.stdout: "inject_early_sigterm": "false", 2024-04-23T16:17:26.647 INFO:teuthology.orchestra.run.smithi103.stdout: "jaeger_agent_port": "6799", 2024-04-23T16:17:26.647 INFO:teuthology.orchestra.run.smithi103.stdout: "jaeger_tracing_enable": "false", 2024-04-23T16:17:26.647 INFO:teuthology.orchestra.run.smithi103.stdout: "journal_aio": "true", 2024-04-23T16:17:26.647 INFO:teuthology.orchestra.run.smithi103.stdout: "journal_align_min_size": "65536", 2024-04-23T16:17:26.647 INFO:teuthology.orchestra.run.smithi103.stdout: "journal_block_align": "true", 2024-04-23T16:17:26.647 INFO:teuthology.orchestra.run.smithi103.stdout: "journal_block_size": "4096", 2024-04-23T16:17:26.647 INFO:teuthology.orchestra.run.smithi103.stdout: "journal_dio": "true", 2024-04-23T16:17:26.647 INFO:teuthology.orchestra.run.smithi103.stdout: "journal_discard": "false", 2024-04-23T16:17:26.647 INFO:teuthology.orchestra.run.smithi103.stdout: "journal_force_aio": "false", 2024-04-23T16:17:26.647 INFO:teuthology.orchestra.run.smithi103.stdout: "journal_ignore_corruption": "false", 2024-04-23T16:17:26.647 INFO:teuthology.orchestra.run.smithi103.stdout: "journal_max_write_bytes": "10485760", 2024-04-23T16:17:26.647 INFO:teuthology.orchestra.run.smithi103.stdout: "journal_max_write_entries": "100", 2024-04-23T16:17:26.647 INFO:teuthology.orchestra.run.smithi103.stdout: "journal_replay_from": "0", 2024-04-23T16:17:26.648 INFO:teuthology.orchestra.run.smithi103.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-23T16:17:26.648 INFO:teuthology.orchestra.run.smithi103.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-23T16:17:26.648 INFO:teuthology.orchestra.run.smithi103.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-23T16:17:26.648 INFO:teuthology.orchestra.run.smithi103.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-23T16:17:26.648 INFO:teuthology.orchestra.run.smithi103.stdout: "journal_write_header_frequency": "0", 2024-04-23T16:17:26.648 INFO:teuthology.orchestra.run.smithi103.stdout: "journal_zero_on_create": "false", 2024-04-23T16:17:26.648 INFO:teuthology.orchestra.run.smithi103.stdout: "journaler_prefetch_periods": "10", 2024-04-23T16:17:26.648 INFO:teuthology.orchestra.run.smithi103.stdout: "journaler_prezero_periods": "5", 2024-04-23T16:17:26.648 INFO:teuthology.orchestra.run.smithi103.stdout: "journaler_write_head_interval": "15", 2024-04-23T16:17:26.648 INFO:teuthology.orchestra.run.smithi103.stdout: "key": "", 2024-04-23T16:17:26.648 INFO:teuthology.orchestra.run.smithi103.stdout: "keyfile": "", 2024-04-23T16:17:26.648 INFO:teuthology.orchestra.run.smithi103.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-23T16:17:26.648 INFO:teuthology.orchestra.run.smithi103.stdout: "kstore_backend": "rocksdb", 2024-04-23T16:17:26.648 INFO:teuthology.orchestra.run.smithi103.stdout: "kstore_default_stripe_size": "65536", 2024-04-23T16:17:26.649 INFO:teuthology.orchestra.run.smithi103.stdout: "kstore_fsck_on_mount": "false", 2024-04-23T16:17:26.649 INFO:teuthology.orchestra.run.smithi103.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-23T16:17:26.649 INFO:teuthology.orchestra.run.smithi103.stdout: "kstore_max_bytes": "67108864", 2024-04-23T16:17:26.649 INFO:teuthology.orchestra.run.smithi103.stdout: "kstore_max_ops": "512", 2024-04-23T16:17:26.649 INFO:teuthology.orchestra.run.smithi103.stdout: "kstore_nid_prealloc": "1024", 2024-04-23T16:17:26.649 INFO:teuthology.orchestra.run.smithi103.stdout: "kstore_onode_map_size": "1024", 2024-04-23T16:17:26.649 INFO:teuthology.orchestra.run.smithi103.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-23T16:17:26.649 INFO:teuthology.orchestra.run.smithi103.stdout: "kstore_sync_submit_transaction": "false", 2024-04-23T16:17:26.649 INFO:teuthology.orchestra.run.smithi103.stdout: "kstore_sync_transaction": "false", 2024-04-23T16:17:26.649 INFO:teuthology.orchestra.run.smithi103.stdout: "librados_thread_count": "2", 2024-04-23T16:17:26.649 INFO:teuthology.orchestra.run.smithi103.stdout: "lockdep": "false", 2024-04-23T16:17:26.649 INFO:teuthology.orchestra.run.smithi103.stdout: "lockdep_force_backtrace": "false", 2024-04-23T16:17:26.649 INFO:teuthology.orchestra.run.smithi103.stdout: "log_coarse_timestamps": "true", 2024-04-23T16:17:26.649 INFO:teuthology.orchestra.run.smithi103.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-23T16:17:26.649 INFO:teuthology.orchestra.run.smithi103.stdout: "log_flush_on_exit": "false", 2024-04-23T16:17:26.650 INFO:teuthology.orchestra.run.smithi103.stdout: "log_graylog_host": "127.0.0.1", 2024-04-23T16:17:26.650 INFO:teuthology.orchestra.run.smithi103.stdout: "log_graylog_port": "12201", 2024-04-23T16:17:26.650 INFO:teuthology.orchestra.run.smithi103.stdout: "log_max_new": "1000", 2024-04-23T16:17:26.650 INFO:teuthology.orchestra.run.smithi103.stdout: "log_max_recent": "10000", 2024-04-23T16:17:26.650 INFO:teuthology.orchestra.run.smithi103.stdout: "log_stderr_prefix": "", 2024-04-23T16:17:26.650 INFO:teuthology.orchestra.run.smithi103.stdout: "log_stop_at_utilization": "0.970000", 2024-04-23T16:17:26.650 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": "true", 2024-04-23T16:17:26.650 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_graylog": "false", 2024-04-23T16:17:26.650 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_journald": "false", 2024-04-23T16:17:26.650 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_stderr": "false", 2024-04-23T16:17:26.650 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_syslog": "false", 2024-04-23T16:17:26.650 INFO:teuthology.orchestra.run.smithi103.stdout: "max_rotating_auth_attempts": "10", 2024-04-23T16:17:26.650 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-23T16:17:26.650 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_action_on_write_error": "1", 2024-04-23T16:17:26.651 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_alternate_name_max": "8192", 2024-04-23T16:17:26.651 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_asio_thread_count": "2", 2024-04-23T16:17:26.651 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_export_pin": "true", 2024-04-23T16:17:26.651 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_fragment_dirs": "true", 2024-04-23T16:17:26.651 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-23T16:17:26.651 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_fragment_interval": "5", 2024-04-23T16:17:26.651 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-23T16:17:26.651 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-23T16:17:26.651 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_interval": "10", 2024-04-23T16:17:26.651 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_max": "-1", 2024-04-23T16:17:26.651 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_max_until": "-1", 2024-04-23T16:17:26.651 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_merge_size": "50", 2024-04-23T16:17:26.651 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_midchunk": "0.300000", 2024-04-23T16:17:26.651 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-23T16:17:26.651 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_min_start": "0.200000", 2024-04-23T16:17:26.651 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_minchunk": "0.001000", 2024-04-23T16:17:26.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_mode": "0", 2024-04-23T16:17:26.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_need_max": "1.200000", 2024-04-23T16:17:26.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_need_min": "0.800000", 2024-04-23T16:17:26.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_overload_epochs": "2", 2024-04-23T16:17:26.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-23T16:17:26.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-23T16:17:26.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_split_bits": "3", 2024-04-23T16:17:26.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-23T16:17:26.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_split_size": "10000", 2024-04-23T16:17:26.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-23T16:17:26.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_target_decay": "10.000000", 2024-04-23T16:17:26.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-23T16:17:26.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_beacon_grace": "15.000000", 2024-04-23T16:17:26.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_beacon_interval": "4.000000", 2024-04-23T16:17:26.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-23T16:17:26.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-23T16:17:26.653 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache_mid": "0.700000", 2024-04-23T16:17:26.653 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-23T16:17:26.653 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache_quiesce_delay": "0", 2024-04-23T16:17:26.653 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-23T16:17:26.653 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-23T16:17:26.653 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-23T16:17:26.653 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache_release_free_interval": "10", 2024-04-23T16:17:26.653 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache_reservation": "0.050000", 2024-04-23T16:17:26.653 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-23T16:17:26.653 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache_trim_interval": "1", 2024-04-23T16:17:26.653 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache_trim_threshold": "262144", 2024-04-23T16:17:26.653 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-23T16:17:26.653 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-23T16:17:26.654 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-23T16:17:26.654 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_client_prealloc_inos": "1000", 2024-04-23T16:17:26.654 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-23T16:17:26.654 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_connect_bootstrapping": "false", 2024-04-23T16:17:26.654 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_damage_table_max_entries": "10000", 2024-04-23T16:17:26.654 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-23T16:17:26.654 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_debug_auth_pins": "false", 2024-04-23T16:17:26.654 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_debug_frag": "false", 2024-04-23T16:17:26.654 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_debug_scatterstat": "false", 2024-04-23T16:17:26.654 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_debug_subtrees": "false", 2024-04-23T16:17:26.654 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_decay_halflife": "5.000000", 2024-04-23T16:17:26.654 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_default_dir_hash": "2", 2024-04-23T16:17:26.654 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_defer_session_stale": "true", 2024-04-23T16:17:26.655 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_deny_all_reconnect": "false", 2024-04-23T16:17:26.655 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_dir_keys_per_op": "16384", 2024-04-23T16:17:26.655 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_dir_max_commit_size": "10", 2024-04-23T16:17:26.655 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_dir_max_entries": "0", 2024-04-23T16:17:26.655 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_dir_prefetch": "true", 2024-04-23T16:17:26.655 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-23T16:17:26.655 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-23T16:17:26.655 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_dump_cache_on_map": "false", 2024-04-23T16:17:26.655 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-23T16:17:26.655 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-23T16:17:26.655 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_early_reply": "true", 2024-04-23T16:17:26.655 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_enable_op_tracker": "true", 2024-04-23T16:17:26.655 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_enforce_unique_name": "true", 2024-04-23T16:17:26.655 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-23T16:17:26.656 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-23T16:17:26.656 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_export_ephemeral_random": "true", 2024-04-23T16:17:26.656 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-23T16:17:26.656 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-23T16:17:26.656 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-23T16:17:26.656 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-23T16:17:26.656 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-23T16:17:26.656 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-23T16:17:26.656 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-23T16:17:26.656 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-23T16:17:26.656 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_health_summarize_threshold": "10", 2024-04-23T16:17:26.656 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-23T16:17:26.656 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-23T16:17:26.656 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_inject_health_dummy": "false", 2024-04-23T16:17:26.657 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-23T16:17:26.657 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_inject_migrator_session_race": "false", 2024-04-23T16:17:26.657 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-23T16:17:26.657 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-23T16:17:26.657 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-23T16:17:26.657 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_join_fs": "", 2024-04-23T16:17:26.657 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_journal_format": "1", 2024-04-23T16:17:26.657 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_kill_create_at": "0", 2024-04-23T16:17:26.657 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_kill_export_at": "0", 2024-04-23T16:17:26.657 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_kill_import_at": "0", 2024-04-23T16:17:26.657 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_kill_journal_at": "0", 2024-04-23T16:17:26.657 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_kill_journal_expire_at": "0", 2024-04-23T16:17:26.657 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_kill_journal_replay_at": "0", 2024-04-23T16:17:26.658 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_kill_link_at": "0", 2024-04-23T16:17:26.658 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_kill_mdstable_at": "0", 2024-04-23T16:17:26.658 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_kill_openc_at": "0", 2024-04-23T16:17:26.658 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_kill_rename_at": "0", 2024-04-23T16:17:26.658 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_kill_shutdown_at": "0", 2024-04-23T16:17:26.658 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-23T16:17:26.658 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log_event_large_threshold": "524288", 2024-04-23T16:17:26.658 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log_events_per_segment": "1024", 2024-04-23T16:17:26.658 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-23T16:17:26.658 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log_max_events": "-1", 2024-04-23T16:17:26.658 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log_max_segments": "128", 2024-04-23T16:17:26.658 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log_pause": "false", 2024-04-23T16:17:26.658 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log_segment_size": "0", 2024-04-23T16:17:26.658 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-23T16:17:26.659 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-23T16:17:26.659 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-23T16:17:26.659 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log_trim_threshold": "128", 2024-04-23T16:17:26.659 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-23T16:17:26.659 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log_warn_factor": "2.000000", 2024-04-23T16:17:26.659 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_max_caps_per_client": "1048576", 2024-04-23T16:17:26.659 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_max_completed_flushes": "100000", 2024-04-23T16:17:26.659 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_max_completed_requests": "100000", 2024-04-23T16:17:26.659 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_max_export_size": "20971520", 2024-04-23T16:17:26.659 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_max_file_recover": "32", 2024-04-23T16:17:26.659 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_max_purge_files": "64", 2024-04-23T16:17:26.659 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_max_purge_ops": "8192", 2024-04-23T16:17:26.659 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-23T16:17:26.660 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-23T16:17:26.660 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_max_snaps_per_dir": "100", 2024-04-23T16:17:26.660 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_metrics_update_interval": "2", 2024-04-23T16:17:26.660 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_min_caps_per_client": "100", 2024-04-23T16:17:26.660 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_min_caps_working_set": "10000", 2024-04-23T16:17:26.660 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-23T16:17:26.660 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_numa_node": "-1", 2024-04-23T16:17:26.660 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-23T16:17:26.660 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_op_complaint_time": "30.000000", 2024-04-23T16:17:26.660 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_op_history_duration": "600", 2024-04-23T16:17:26.660 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_op_history_size": "20", 2024-04-23T16:17:26.660 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_op_history_slow_op_size": "20", 2024-04-23T16:17:26.660 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-23T16:17:26.660 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_op_log_threshold": "5", 2024-04-23T16:17:26.660 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_ping_grace": "15", 2024-04-23T16:17:26.661 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_ping_interval": "5", 2024-04-23T16:17:26.661 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-23T16:17:26.661 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-23T16:17:26.661 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_recall_max_caps": "30000", 2024-04-23T16:17:26.661 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-23T16:17:26.661 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-23T16:17:26.661 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-23T16:17:26.661 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_recall_warning_threshold": "262144", 2024-04-23T16:17:26.661 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-23T16:17:26.661 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_replay_interval": "1.000000", 2024-04-23T16:17:26.661 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-23T16:17:26.661 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-23T16:17:26.661 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_root_ino_gid": "0", 2024-04-23T16:17:26.661 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_root_ino_uid": "0", 2024-04-23T16:17:26.662 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-23T16:17:26.662 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-23T16:17:26.662 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-23T16:17:26.662 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-23T16:17:26.662 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-23T16:17:26.662 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-23T16:17:26.662 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-23T16:17:26.662 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-23T16:17:26.662 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-23T16:17:26.662 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-23T16:17:26.662 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_shutdown_check": "0", 2024-04-23T16:17:26.662 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_skip_ino": "0", 2024-04-23T16:17:26.662 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-23T16:17:26.663 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_snap_max_uid": "4294967294", 2024-04-23T16:17:26.663 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_snap_min_uid": "0", 2024-04-23T16:17:26.663 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_snap_rstat": "false", 2024-04-23T16:17:26.663 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_standby_replay_damaged": "false", 2024-04-23T16:17:26.663 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_symlink_recovery": "true", 2024-04-23T16:17:26.663 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-23T16:17:26.663 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_thrash_exports": "0", 2024-04-23T16:17:26.663 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_thrash_fragments": "0", 2024-04-23T16:17:26.663 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_tick_interval": "5.000000", 2024-04-23T16:17:26.663 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_valgrind_exit": "false", 2024-04-23T16:17:26.663 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_verify_backtrace": "1", 2024-04-23T16:17:26.663 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_verify_scatter": "false", 2024-04-23T16:17:26.663 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-23T16:17:26.663 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_wipe_sessions": "false", 2024-04-23T16:17:26.664 INFO:teuthology.orchestra.run.smithi103.stdout: "mempool_debug": "false", 2024-04-23T16:17:26.664 INFO:teuthology.orchestra.run.smithi103.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-23T16:17:26.664 INFO:teuthology.orchestra.run.smithi103.stdout: "memstore_device_bytes": "1073741824", 2024-04-23T16:17:26.664 INFO:teuthology.orchestra.run.smithi103.stdout: "memstore_page_set": "false", 2024-04-23T16:17:26.664 INFO:teuthology.orchestra.run.smithi103.stdout: "memstore_page_size": "65536", 2024-04-23T16:17:26.664 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_client_bytes": "134217728", 2024-04-23T16:17:26.664 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_client_messages": "512", 2024-04-23T16:17:26.664 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-23T16:17:26.664 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-23T16:17:26.664 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-23T16:17:26.664 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-23T16:17:26.664 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_disabled_modules": "", 2024-04-23T16:17:26.664 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-23T16:17:26.664 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_max_pg_creating": "1024", 2024-04-23T16:17:26.664 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_max_pg_num_change": "128", 2024-04-23T16:17:26.665 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_mds_bytes": "134217728", 2024-04-23T16:17:26.665 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_mds_messages": "128", 2024-04-23T16:17:26.665 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-23T16:17:26.665 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_mon_bytes": "134217728", 2024-04-23T16:17:26.665 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_mon_messages": "128", 2024-04-23T16:17:26.665 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_osd_bytes": "536870912", 2024-04-23T16:17:26.665 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_osd_messages": "8192", 2024-04-23T16:17:26.665 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_pool": "true", 2024-04-23T16:17:26.665 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-23T16:17:26.665 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_standby_modules": "true", 2024-04-23T16:17:26.665 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_stats_period": "5", 2024-04-23T16:17:26.665 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_stats_threshold": "5", 2024-04-23T16:17:26.665 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_tick_period": "2", 2024-04-23T16:17:26.665 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-23T16:17:26.665 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-23T16:17:26.665 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_allow_pool_delete": "true", 2024-04-23T16:17:26.666 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_allow_pool_size_one": "false", 2024-04-23T16:17:26.666 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_auth_validate_all_caps": "true", 2024-04-23T16:17:26.666 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-23T16:17:26.666 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-23T16:17:26.666 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_client_bytes": "104857600", 2024-04-23T16:17:26.666 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_client_directed_command_retry": "5", 2024-04-23T16:17:26.666 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-23T16:17:26.666 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-23T16:17:26.666 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-23T16:17:26.666 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-23T16:17:26.666 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_client_hunt_parallel": "3", 2024-04-23T16:17:26.666 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_client_log_interval": "1.000000", 2024-04-23T16:17:26.666 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-23T16:17:26.666 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_client_ping_interval": "10.000000", 2024-04-23T16:17:26.667 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-23T16:17:26.667 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_client_target_rank": "-1", 2024-04-23T16:17:26.667 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-23T16:17:26.667 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-23T16:17:26.667 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-23T16:17:26.667 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_cluster_log_level": "debug", 2024-04-23T16:17:26.667 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_cluster_log_to_file": "true", 2024-04-23T16:17:26.667 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-23T16:17:26.667 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-23T16:17:26.667 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-23T16:17:26.667 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_cluster_log_to_journald": "true", 2024-04-23T16:17:26.667 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-23T16:17:26.667 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-23T16:17:26.667 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-23T16:17:26.667 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_compact_on_bootstrap": "false", 2024-04-23T16:17:26.668 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_compact_on_start": "false", 2024-04-23T16:17:26.668 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_compact_on_trim": "true", 2024-04-23T16:17:26.668 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-23T16:17:26.668 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-23T16:17:26.668 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-23T16:17:26.668 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_cpu_threads": "4", 2024-04-23T16:17:26.668 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_crush_min_required_version": "hammer", 2024-04-23T16:17:26.668 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_daemon_bytes": "419430400", 2024-04-23T16:17:26.668 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-23T16:17:26.668 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_data_avail_crit": "5", 2024-04-23T16:17:26.668 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_data_avail_warn": "5", 2024-04-23T16:17:26.668 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_data_size_warn": "16106127360", 2024-04-23T16:17:26.668 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-23T16:17:26.668 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-23T16:17:26.669 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_debug_dump_json": "false", 2024-04-23T16:17:26.669 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-23T16:17:26.669 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_debug_dump_transactions": "false", 2024-04-23T16:17:26.669 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_debug_extra_checks": "false", 2024-04-23T16:17:26.669 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-23T16:17:26.669 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-23T16:17:26.669 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_debug_no_require_reef": "false", 2024-04-23T16:17:26.669 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_debug_no_require_squid": "false", 2024-04-23T16:17:26.669 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-23T16:17:26.669 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-23T16:17:26.669 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-23T16:17:26.669 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_down_mkfs_grace": "60", 2024-04-23T16:17:26.669 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_down_uptime_grace": "60", 2024-04-23T16:17:26.670 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_election_default_strategy": "1", 2024-04-23T16:17:26.670 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_election_timeout": "5.000000", 2024-04-23T16:17:26.670 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-23T16:17:26.670 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_elector_ping_divisor": "2", 2024-04-23T16:17:26.670 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-23T16:17:26.670 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_enable_op_tracker": "true", 2024-04-23T16:17:26.670 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_fake_pool_delete": "false", 2024-04-23T16:17:26.670 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_force_quorum_join": "false", 2024-04-23T16:17:26.670 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_globalid_prealloc": "10000", 2024-04-23T16:17:26.670 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_health_detail_to_clog": "true", 2024-04-23T16:17:26.670 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_health_log_update_period": "5", 2024-04-23T16:17:26.670 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_health_max_detail": "50", 2024-04-23T16:17:26.670 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_health_to_clog": "true", 2024-04-23T16:17:26.670 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_health_to_clog_interval": "600", 2024-04-23T16:17:26.671 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-23T16:17:26.671 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_host": "172.21.15.103:6789/0", 2024-04-23T16:17:26.671 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_host_override": "", 2024-04-23T16:17:26.671 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_initial_members": "", 2024-04-23T16:17:26.671 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-23T16:17:26.671 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-23T16:17:26.671 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-23T16:17:26.671 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-23T16:17:26.671 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-23T16:17:26.671 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_lease": "5.000000", 2024-04-23T16:17:26.671 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-23T16:17:26.671 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-23T16:17:26.671 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_log_full_interval": "50", 2024-04-23T16:17:26.671 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_log_max": "10000", 2024-04-23T16:17:26.671 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_log_max_summary": "50", 2024-04-23T16:17:26.672 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-23T16:17:26.672 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_max_log_epochs": "500", 2024-04-23T16:17:26.672 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-23T16:17:26.672 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-23T16:17:26.672 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_max_osd": "10000", 2024-04-23T16:17:26.672 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_max_pg_per_osd": "10000", 2024-04-23T16:17:26.672 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_max_pool_pg_num": "65536", 2024-04-23T16:17:26.672 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-23T16:17:26.672 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-23T16:17:26.672 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_mds_force_trim_to": "0", 2024-04-23T16:17:26.672 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_mds_skip_sanity": "false", 2024-04-23T16:17:26.672 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_memory_autotune": "true", 2024-04-23T16:17:26.672 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_memory_target": "2147483648", 2024-04-23T16:17:26.672 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_mgr_beacon_grace": "30", 2024-04-23T16:17:26.672 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-23T16:17:26.673 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_mgr_digest_period": "5", 2024-04-23T16:17:26.673 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_mgr_inactive_grace": "60", 2024-04-23T16:17:26.673 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-23T16:17:26.673 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-23T16:17:26.673 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_min_osdmap_epochs": "50", 2024-04-23T16:17:26.673 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_op_complaint_time": "30", 2024-04-23T16:17:26.673 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_op_history_duration": "600", 2024-04-23T16:17:26.673 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_op_history_size": "20", 2024-04-23T16:17:26.673 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_op_history_slow_op_size": "20", 2024-04-23T16:17:26.673 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-23T16:17:26.673 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_op_log_threshold": "5", 2024-04-23T16:17:26.673 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-23T16:17:26.673 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-23T16:17:26.673 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-23T16:17:26.673 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_auto_mark_in": "false", 2024-04-23T16:17:26.674 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-23T16:17:26.674 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-23T16:17:26.674 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-23T16:17:26.674 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_cache_size": "500", 2024-04-23T16:17:26.674 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-23T16:17:26.674 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-23T16:17:26.674 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-23T16:17:26.674 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_down_out_interval": "600", 2024-04-23T16:17:26.674 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-23T16:17:26.674 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-23T16:17:26.674 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_force_trim_to": "0", 2024-04-23T16:17:26.674 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-23T16:17:26.674 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-23T16:17:26.674 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-23T16:17:26.675 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-23T16:17:26.675 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-23T16:17:26.675 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-23T16:17:26.675 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-23T16:17:26.675 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_min_down_reporters": "2", 2024-04-23T16:17:26.675 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-23T16:17:26.675 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-23T16:17:26.675 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-23T16:17:26.675 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-23T16:17:26.675 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-23T16:17:26.675 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-23T16:17:26.675 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_report_timeout": "900", 2024-04-23T16:17:26.675 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-23T16:17:26.675 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-23T16:17:26.675 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-23T16:17:26.675 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-23T16:17:26.676 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-23T16:17:26.676 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osdmap_full_prune_interval": "2", 2024-04-23T16:17:26.676 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osdmap_full_prune_min": "15", 2024-04-23T16:17:26.676 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-04-23T16:17:26.676 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-23T16:17:26.676 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_pg_stuck_threshold": "60", 2024-04-23T16:17:26.676 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-23T16:17:26.676 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-23T16:17:26.676 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-23T16:17:26.676 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-23T16:17:26.676 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-23T16:17:26.676 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-23T16:17:26.676 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_probe_timeout": "2.000000", 2024-04-23T16:17:26.676 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_reweight_max_change": "0.050000", 2024-04-23T16:17:26.676 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_reweight_max_osds": "4", 2024-04-23T16:17:26.676 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-23T16:17:26.677 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-23T16:17:26.677 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-23T16:17:26.677 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-23T16:17:26.677 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-23T16:17:26.677 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_scrub_interval": "300", 2024-04-23T16:17:26.677 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_scrub_max_keys": "100", 2024-04-23T16:17:26.677 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_scrub_timeout": "300", 2024-04-23T16:17:26.677 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_session_timeout": "300", 2024-04-23T16:17:26.677 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_smart_report_timeout": "5", 2024-04-23T16:17:26.677 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_stat_smooth_intervals": "6", 2024-04-23T16:17:26.677 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-23T16:17:26.677 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_stretch_pool_min_size": "2", 2024-04-23T16:17:26.677 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_stretch_pool_size": "4", 2024-04-23T16:17:26.678 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-23T16:17:26.678 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-23T16:17:26.678 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_sync_debug": "false", 2024-04-23T16:17:26.678 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-23T16:17:26.678 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-23T16:17:26.678 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_sync_provider_kill_at": "0", 2024-04-23T16:17:26.678 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_sync_requester_kill_at": "0", 2024-04-23T16:17:26.678 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_sync_timeout": "60.000000", 2024-04-23T16:17:26.678 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_target_pg_per_osd": "100", 2024-04-23T16:17:26.678 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_tick_interval": "5", 2024-04-23T16:17:26.678 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_timecheck_interval": "300.000000", 2024-04-23T16:17:26.678 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-23T16:17:26.678 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_use_min_delay_socket": "false", 2024-04-23T16:17:26.678 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_older_version_delay": "604800", 2024-04-23T16:17:26.679 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-23T16:17:26.679 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-23T16:17:26.679 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-23T16:17:26.679 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_on_filestore_osds": "true", 2024-04-23T16:17:26.679 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-23T16:17:26.679 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-23T16:17:26.679 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-23T16:17:26.679 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_on_misplaced": "false", 2024-04-23T16:17:26.679 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-23T16:17:26.679 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_on_older_version": "true", 2024-04-23T16:17:26.679 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-23T16:17:26.679 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-23T16:17:26.679 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-04-23T16:17:26.679 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-23T16:17:26.680 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-23T16:17:26.680 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-23T16:17:26.680 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-23T16:17:26.680 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-23T16:17:26.680 INFO:teuthology.orchestra.run.smithi103.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-23T16:17:26.680 INFO:teuthology.orchestra.run.smithi103.stdout: "monmap": "", 2024-04-23T16:17:26.680 INFO:teuthology.orchestra.run.smithi103.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-23T16:17:26.680 INFO:teuthology.orchestra.run.smithi103.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-23T16:17:26.680 INFO:teuthology.orchestra.run.smithi103.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-23T16:17:26.680 INFO:teuthology.orchestra.run.smithi103.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-23T16:17:26.680 INFO:teuthology.orchestra.run.smithi103.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-23T16:17:26.680 INFO:teuthology.orchestra.run.smithi103.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-23T16:17:26.680 INFO:teuthology.orchestra.run.smithi103.stdout: "motr_tracing_enabled": "false", 2024-04-23T16:17:26.680 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_async_op_threads": "3", 2024-04-23T16:17:26.680 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-23T16:17:26.680 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_async_rdma_cm": "false", 2024-04-23T16:17:26.680 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_async_rdma_device_name": "", 2024-04-23T16:17:26.681 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_async_rdma_dscp": "96", 2024-04-23T16:17:26.681 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-23T16:17:26.681 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-23T16:17:26.681 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_async_rdma_local_gid": "", 2024-04-23T16:17:26.681 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-23T16:17:26.681 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_async_rdma_port_num": "1", 2024-04-23T16:17:26.681 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-23T16:17:26.681 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-23T16:17:26.681 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-23T16:17:26.681 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-23T16:17:26.681 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_async_rdma_sl": "3", 2024-04-23T16:17:26.681 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_async_rdma_support_srq": "true", 2024-04-23T16:17:26.681 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_async_rdma_type": "ib", 2024-04-23T16:17:26.681 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_async_reap_threshold": "5", 2024-04-23T16:17:26.681 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_bind_before_connect": "false", 2024-04-23T16:17:26.681 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_bind_ipv4": "true", 2024-04-23T16:17:26.682 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_bind_ipv6": "false", 2024-04-23T16:17:26.682 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_bind_msgr1": "true", 2024-04-23T16:17:26.682 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_bind_msgr2": "false", 2024-04-23T16:17:26.682 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_bind_port_max": "7568", 2024-04-23T16:17:26.682 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_bind_port_min": "6800", 2024-04-23T16:17:26.682 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-23T16:17:26.682 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_bind_retry_count": "3", 2024-04-23T16:17:26.682 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_bind_retry_delay": "5", 2024-04-23T16:17:26.682 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_blackhole_client": "false", 2024-04-23T16:17:26.682 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_blackhole_mds": "false", 2024-04-23T16:17:26.682 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_blackhole_mgr": "false", 2024-04-23T16:17:26.682 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_blackhole_mon": "false", 2024-04-23T16:17:26.682 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_blackhole_osd": "false", 2024-04-23T16:17:26.682 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_client_mode": "crc secure", 2024-04-23T16:17:26.682 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_cluster_mode": "crc secure", 2024-04-23T16:17:26.683 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_cluster_type": "", 2024-04-23T16:17:26.683 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_compress_secure": "false", 2024-04-23T16:17:26.683 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_connection_idle_timeout": "900", 2024-04-23T16:17:26.683 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_connection_ready_timeout": "10", 2024-04-23T16:17:26.683 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_crc_data": "true", 2024-04-23T16:17:26.683 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_crc_header": "true", 2024-04-23T16:17:26.683 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_die_on_bad_msg": "false", 2024-04-23T16:17:26.683 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_die_on_bug": "true", 2024-04-23T16:17:26.683 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_die_on_old_message": "true", 2024-04-23T16:17:26.683 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_die_on_skipped_message": "false", 2024-04-23T16:17:26.683 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-23T16:17:26.683 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-23T16:17:26.683 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_dpdk_coremask": "0xF", 2024-04-23T16:17:26.683 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-23T16:17:26.683 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-23T16:17:26.683 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_dpdk_enable_tso": "true", 2024-04-23T16:17:26.684 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-23T16:17:26.684 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-23T16:17:26.684 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_dpdk_hugepages": "", 2024-04-23T16:17:26.684 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-23T16:17:26.684 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-23T16:17:26.684 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_dpdk_lro": "true", 2024-04-23T16:17:26.684 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_dpdk_memory_channel": "4", 2024-04-23T16:17:26.684 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-23T16:17:26.684 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_dpdk_pmd": "", 2024-04-23T16:17:26.684 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_dpdk_port_id": "0", 2024-04-23T16:17:26.684 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-23T16:17:26.684 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-23T16:17:26.684 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_dump_on_send": "false", 2024-04-23T16:17:26.685 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_initial_backoff": "0.200000", 2024-04-23T16:17:26.685 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_inject_delay_max": "1.000000", 2024-04-23T16:17:26.685 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-23T16:17:26.685 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_inject_delay_type": "", 2024-04-23T16:17:26.685 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-23T16:17:26.685 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_inject_network_congestion": "0", 2024-04-23T16:17:26.685 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_inject_socket_failures": "5000", 2024-04-23T16:17:26.685 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_learn_addr_from_peer": "true", 2024-04-23T16:17:26.685 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_max_accept_failures": "4", 2024-04-23T16:17:26.685 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_max_backoff": "15.000000", 2024-04-23T16:17:26.685 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_mon_client_mode": "secure crc", 2024-04-23T16:17:26.685 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-23T16:17:26.685 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_mon_service_mode": "secure crc", 2024-04-23T16:17:26.685 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_osd_compress_min_size": "1024", 2024-04-23T16:17:26.685 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_osd_compress_mode": "none", 2024-04-23T16:17:26.686 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-23T16:17:26.686 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-23T16:17:26.686 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_pq_min_cost": "65536", 2024-04-23T16:17:26.686 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_public_type": "", 2024-04-23T16:17:26.686 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_service_mode": "crc secure", 2024-04-23T16:17:26.686 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_tcp_listen_backlog": "512", 2024-04-23T16:17:26.686 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_tcp_nodelay": "true", 2024-04-23T16:17:26.686 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-23T16:17:26.686 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_tcp_rcvbuf": "0", 2024-04-23T16:17:26.686 INFO:teuthology.orchestra.run.smithi103.stdout: "ms_type": "async+posix", 2024-04-23T16:17:26.686 INFO:teuthology.orchestra.run.smithi103.stdout: "no_config_file": "false", 2024-04-23T16:17:26.686 INFO:teuthology.orchestra.run.smithi103.stdout: "objecter_completion_locks_per_session": "32", 2024-04-23T16:17:26.686 INFO:teuthology.orchestra.run.smithi103.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-23T16:17:26.686 INFO:teuthology.orchestra.run.smithi103.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-23T16:17:26.687 INFO:teuthology.orchestra.run.smithi103.stdout: "objecter_inflight_ops": "1024", 2024-04-23T16:17:26.687 INFO:teuthology.orchestra.run.smithi103.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-23T16:17:26.687 INFO:teuthology.orchestra.run.smithi103.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-23T16:17:26.687 INFO:teuthology.orchestra.run.smithi103.stdout: "objecter_tick_interval": "5.000000", 2024-04-23T16:17:26.687 INFO:teuthology.orchestra.run.smithi103.stdout: "objecter_timeout": "10.000000", 2024-04-23T16:17:26.687 INFO:teuthology.orchestra.run.smithi103.stdout: "objectstore_blackhole": "false", 2024-04-23T16:17:26.687 INFO:teuthology.orchestra.run.smithi103.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-23T16:17:26.687 INFO:teuthology.orchestra.run.smithi103.stdout: "openssl_engine_opts": "", 2024-04-23T16:17:26.687 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_agent_delay_time": "5.000000", 2024-04-23T16:17:26.687 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_agent_hist_halflife": "1000", 2024-04-23T16:17:26.687 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_agent_max_low_ops": "2", 2024-04-23T16:17:26.687 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_agent_max_ops": "4", 2024-04-23T16:17:26.687 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-23T16:17:26.687 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-23T16:17:26.687 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_agent_slop": "0.020000", 2024-04-23T16:17:26.687 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-23T16:17:26.688 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-23T16:17:26.688 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_asio_thread_count": "2", 2024-04-23T16:17:26.688 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_async_recovery_min_cost": "100", 2024-04-23T16:17:26.688 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-23T16:17:26.688 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-23T16:17:26.688 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_backfill_scan_max": "512", 2024-04-23T16:17:26.688 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_backfill_scan_min": "64", 2024-04-23T16:17:26.688 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_backoff_on_degraded": "false", 2024-04-23T16:17:26.688 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_backoff_on_peering": "false", 2024-04-23T16:17:26.688 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_backoff_on_unfound": "true", 2024-04-23T16:17:26.688 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_beacon_report_interval": "300", 2024-04-23T16:17:26.688 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bench_duration": "30", 2024-04-23T16:17:26.688 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-23T16:17:26.688 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bench_max_block_size": "67108864", 2024-04-23T16:17:26.689 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-23T16:17:26.689 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_blkin_trace_all": "false", 2024-04-23T16:17:26.689 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-23T16:17:26.689 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-23T16:17:26.689 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-23T16:17:26.689 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-23T16:17:26.689 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_check_for_log_corruption": "false", 2024-04-23T16:17:26.689 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-23T16:17:26.689 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-23T16:17:26.689 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-23T16:17:26.689 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-23T16:17:26.689 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_class_update_on_start": "true", 2024-04-23T16:17:26.689 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_client_message_cap": "256", 2024-04-23T16:17:26.689 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_client_message_size_cap": "524288000", 2024-04-23T16:17:26.689 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_client_op_priority": "63", 2024-04-23T16:17:26.689 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_client_watch_timeout": "30", 2024-04-23T16:17:26.690 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_command_max_records": "256", 2024-04-23T16:17:26.690 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_compact_on_start": "false", 2024-04-23T16:17:26.690 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-23T16:17:26.690 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_crush_chooseleaf_type": "1", 2024-04-23T16:17:26.690 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-23T16:17:26.690 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_crush_update_on_start": "true", 2024-04-23T16:17:26.690 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_crush_update_weight_set": "true", 2024-04-23T16:17:26.690 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-23T16:17:26.690 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-23T16:17:26.690 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-23T16:17:26.690 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-23T16:17:26.690 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-23T16:17:26.690 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_feed_pullee": "-1", 2024-04-23T16:17:26.690 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-23T16:17:26.690 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-23T16:17:26.691 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-23T16:17:26.691 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_misdirected_ops": "false", 2024-04-23T16:17:26.691 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_no_acting_change": "false", 2024-04-23T16:17:26.691 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_no_purge_strays": "false", 2024-04-23T16:17:26.691 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_op_order": "false", 2024-04-23T16:17:26.691 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-23T16:17:26.691 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-23T16:17:26.691 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-23T16:17:26.691 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-23T16:17:26.691 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_shutdown": "false", 2024-04-23T16:17:26.691 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-23T16:17:26.691 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-23T16:17:26.691 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_trim_objects": "false", 2024-04-23T16:17:26.692 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-23T16:17:26.692 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-23T16:17:26.692 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_verify_snaps": "false", 2024-04-23T16:17:26.692 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-23T16:17:26.692 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-23T16:17:26.692 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_deep_scrub_keys": "1024", 2024-04-23T16:17:26.692 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-23T16:17:26.692 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-23T16:17:26.692 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-23T16:17:26.692 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_deep_scrub_stride": "524288", 2024-04-23T16:17:26.692 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-23T16:17:26.692 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_default_data_pool_replay_window": "45", 2024-04-23T16:17:26.692 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_default_notify_timeout": "30", 2024-04-23T16:17:26.693 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_delete_sleep": "0.000000", 2024-04-23T16:17:26.693 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-23T16:17:26.693 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-23T16:17:26.693 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-23T16:17:26.693 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_discard_disconnected_ops": "true", 2024-04-23T16:17:26.693 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_enable_op_tracker": "true", 2024-04-23T16:17:26.693 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-23T16:17:26.693 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-23T16:17:26.693 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-23T16:17:26.693 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-23T16:17:26.693 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_fast_info": "true", 2024-04-23T16:17:26.693 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_fast_shutdown": "true", 2024-04-23T16:17:26.693 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-23T16:17:26.693 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-23T16:17:26.694 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-23T16:17:26.694 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-23T16:17:26.694 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-23T16:17:26.694 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_function_tracing": "false", 2024-04-23T16:17:26.694 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_heartbeat_grace": "20", 2024-04-23T16:17:26.694 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_heartbeat_interval": "6", 2024-04-23T16:17:26.694 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-23T16:17:26.694 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_heartbeat_min_peers": "10", 2024-04-23T16:17:26.694 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_heartbeat_min_size": "2000", 2024-04-23T16:17:26.694 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_heartbeat_stale": "600", 2024-04-23T16:17:26.694 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-23T16:17:26.694 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_hit_set_max_size": "100000", 2024-04-23T16:17:26.694 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_hit_set_min_size": "1000", 2024-04-23T16:17:26.695 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-23T16:17:26.695 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-23T16:17:26.695 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-23T16:17:26.695 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-23T16:17:26.695 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-23T16:17:26.695 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-23T16:17:26.695 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_journal_size": "5120", 2024-04-23T16:17:26.695 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_kill_backfill_at": "0", 2024-04-23T16:17:26.695 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-23T16:17:26.695 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_map_cache_size": "50", 2024-04-23T16:17:26.695 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_map_dedup": "true", 2024-04-23T16:17:26.695 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_map_message_max": "40", 2024-04-23T16:17:26.695 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-23T16:17:26.696 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_map_share_max_epochs": "40", 2024-04-23T16:17:26.696 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_attr_name_len": "100", 2024-04-23T16:17:26.696 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_attr_size": "0", 2024-04-23T16:17:26.696 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_backfills": "1", 2024-04-23T16:17:26.696 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_markdown_count": "5", 2024-04-23T16:17:26.696 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_markdown_period": "600", 2024-04-23T16:17:26.696 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_object_name_len": "2048", 2024-04-23T16:17:26.696 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_object_namespace_len": "256", 2024-04-23T16:17:26.696 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_object_size": "134217728", 2024-04-23T16:17:26.696 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-23T16:17:26.696 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-23T16:17:26.696 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_pg_blocked_by": "16", 2024-04-23T16:17:26.696 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_pg_log_entries": "10000", 2024-04-23T16:17:26.696 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-23T16:17:26.696 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_pgls": "1024", 2024-04-23T16:17:26.697 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_push_cost": "8388608", 2024-04-23T16:17:26.697 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_push_objects": "10", 2024-04-23T16:17:26.697 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_scrubs": "3", 2024-04-23T16:17:26.697 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-23T16:17:26.697 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_trimming_pgs": "2", 2024-04-23T16:17:26.697 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_write_op_reply_len": "64", 2024-04-23T16:17:26.697 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_max_write_size": "90", 2024-04-23T16:17:26.697 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-23T16:17:26.697 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-23T16:17:26.697 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-23T16:17:26.697 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-23T16:17:26.697 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-23T16:17:26.697 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-23T16:17:26.698 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-23T16:17:26.698 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-23T16:17:26.698 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_profile": "balanced", 2024-04-23T16:17:26.698 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-23T16:17:26.698 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-23T16:17:26.698 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-23T16:17:26.698 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-23T16:17:26.698 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-23T16:17:26.698 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-23T16:17:26.698 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-23T16:17:26.698 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-23T16:17:26.698 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-23T16:17:26.698 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-23T16:17:26.698 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-23T16:17:26.699 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_memory_base": "805306368", 2024-04-23T16:17:26.699 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_memory_cache_min": "134217728", 2024-04-23T16:17:26.699 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-23T16:17:26.699 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-23T16:17:26.699 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_memory_target": "4294967296", 2024-04-23T16:17:26.699 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_memory_target_autotune": "false", 2024-04-23T16:17:26.699 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-23T16:17:26.699 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_min_pg_log_entries": "250", 2024-04-23T16:17:26.699 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_min_recovery_priority": "0", 2024-04-23T16:17:26.699 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-23T16:17:26.699 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-23T16:17:26.699 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mon_report_interval": "5", 2024-04-23T16:17:26.699 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-23T16:17:26.699 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-23T16:17:26.700 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_num_cache_shards": "32", 2024-04-23T16:17:26.700 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_num_op_tracker_shard": "32", 2024-04-23T16:17:26.700 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_numa_auto_affinity": "true", 2024-04-23T16:17:26.700 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_numa_node": "-1", 2024-04-23T16:17:26.700 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_numa_prefer_iface": "true", 2024-04-23T16:17:26.700 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-23T16:17:26.700 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_objecter_finishers": "1", 2024-04-23T16:17:26.700 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_objectstore": "bluestore", 2024-04-23T16:17:26.700 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_objectstore_fuse": "false", 2024-04-23T16:17:26.700 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_objectstore_tracing": "false", 2024-04-23T16:17:26.700 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_op_complaint_time": "30.000000", 2024-04-23T16:17:26.700 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_op_history_duration": "600", 2024-04-23T16:17:26.700 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_op_history_size": "20", 2024-04-23T16:17:26.701 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_op_history_slow_op_size": "20", 2024-04-23T16:17:26.701 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-23T16:17:26.701 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_op_log_threshold": "5", 2024-04-23T16:17:26.701 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_op_num_shards": "0", 2024-04-23T16:17:26.701 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_op_num_shards_hdd": "5", 2024-04-23T16:17:26.701 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_op_num_shards_ssd": "8", 2024-04-23T16:17:26.701 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-23T16:17:26.701 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-23T16:17:26.701 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-23T16:17:26.701 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-23T16:17:26.701 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_op_pq_min_cost": "65536", 2024-04-23T16:17:26.701 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-23T16:17:26.701 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_op_queue_cut_off": "high", 2024-04-23T16:17:26.701 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-23T16:17:26.701 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_op_thread_timeout": "15", 2024-04-23T16:17:26.702 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_open_classes_on_start": "true", 2024-04-23T16:17:26.702 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_os_flags": "0", 2024-04-23T16:17:26.702 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_peering_op_priority": "255", 2024-04-23T16:17:26.702 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pg_delete_cost": "1048576", 2024-04-23T16:17:26.702 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pg_delete_priority": "5", 2024-04-23T16:17:26.702 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-23T16:17:26.702 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-23T16:17:26.702 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-23T16:17:26.702 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pg_log_trim_max": "10000", 2024-04-23T16:17:26.702 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pg_log_trim_min": "100", 2024-04-23T16:17:26.702 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-23T16:17:26.702 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-23T16:17:26.702 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-23T16:17:26.702 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-23T16:17:26.703 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-23T16:17:26.703 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-23T16:17:26.703 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-23T16:17:26.703 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-23T16:17:26.703 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-23T16:17:26.703 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-23T16:17:26.703 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_crimson": "false", 2024-04-23T16:17:26.703 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-23T16:17:26.703 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-23T16:17:26.703 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-23T16:17:26.703 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-23T16:17:26.703 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-23T16:17:26.703 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-23T16:17:26.703 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-23T16:17:26.703 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-23T16:17:26.703 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_flags": "0", 2024-04-23T16:17:26.704 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-23T16:17:26.704 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_min_size": "2", 2024-04-23T16:17:26.704 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-23T16:17:26.704 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_pg_num": "32", 2024-04-23T16:17:26.704 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_pgp_num": "0", 2024-04-23T16:17:26.704 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-23T16:17:26.704 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_read_ratio": "70", 2024-04-23T16:17:26.704 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_size": "2", 2024-04-23T16:17:26.704 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_default_type": "replicated", 2024-04-23T16:17:26.704 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-23T16:17:26.704 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-23T16:17:26.704 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_push_per_object_cost": "1000", 2024-04-23T16:17:26.704 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-23T16:17:26.704 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_recover_clone_overlap": "true", 2024-04-23T16:17:26.705 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-23T16:17:26.705 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_recovery_cost": "20971520", 2024-04-23T16:17:26.705 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-23T16:17:26.705 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_recovery_max_active": "0", 2024-04-23T16:17:26.705 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-23T16:17:26.705 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-23T16:17:26.705 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-23T16:17:26.705 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-23T16:17:26.705 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_recovery_max_single_start": "1", 2024-04-23T16:17:26.705 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_recovery_op_priority": "3", 2024-04-23T16:17:26.705 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-23T16:17:26.705 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_recovery_priority": "5", 2024-04-23T16:17:26.705 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-23T16:17:26.705 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_recovery_sleep": "0.000000", 2024-04-23T16:17:26.706 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-23T16:17:26.706 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-23T16:17:26.706 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-23T16:17:26.706 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_repair_during_recovery": "false", 2024-04-23T16:17:26.706 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_requested_scrub_priority": "120", 2024-04-23T16:17:26.706 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-23T16:17:26.706 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-23T16:17:26.706 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_auto_repair": "false", 2024-04-23T16:17:26.706 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-23T16:17:26.706 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-23T16:17:26.706 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_begin_hour": "0", 2024-04-23T16:17:26.706 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_begin_week_day": "0", 2024-04-23T16:17:26.706 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_chunk_max": "25", 2024-04-23T16:17:26.706 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_chunk_min": "5", 2024-04-23T16:17:26.707 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_cost": "52428800", 2024-04-23T16:17:26.707 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-23T16:17:26.707 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_during_recovery": "false", 2024-04-23T16:17:26.707 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_end_hour": "0", 2024-04-23T16:17:26.707 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_end_week_day": "0", 2024-04-23T16:17:26.707 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_event_cost": "4096", 2024-04-23T16:17:26.707 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-23T16:17:26.707 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-23T16:17:26.707 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_invalid_stats": "true", 2024-04-23T16:17:26.707 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-23T16:17:26.707 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-23T16:17:26.707 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_max_preemptions": "5", 2024-04-23T16:17:26.707 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-23T16:17:26.707 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_priority": "5", 2024-04-23T16:17:26.708 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-23T16:17:26.708 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_scrub_sleep": "0.000000", 2024-04-23T16:17:26.708 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-23T16:17:26.708 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-23T16:17:26.708 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-23T16:17:26.708 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-04-23T16:17:26.708 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_skip_data_digest": "false", 2024-04-23T16:17:26.708 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_smart_report_timeout": "5", 2024-04-23T16:17:26.708 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_snap_trim_cost": "1048576", 2024-04-23T16:17:26.708 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_snap_trim_priority": "5", 2024-04-23T16:17:26.708 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-23T16:17:26.708 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-23T16:17:26.708 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-23T16:17:26.709 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-23T16:17:26.709 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-23T16:17:26.709 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-23T16:17:26.709 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-23T16:17:26.709 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_target_transaction_size": "30", 2024-04-23T16:17:26.709 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-23T16:17:26.709 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-23T16:17:26.709 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-23T16:17:26.709 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-23T16:17:26.709 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-23T16:17:26.709 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-23T16:17:26.709 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-23T16:17:26.709 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-23T16:17:26.709 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-23T16:17:26.710 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-23T16:17:26.710 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_tracing": "false", 2024-04-23T16:17:26.710 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_use_stale_snap": "false", 2024-04-23T16:17:26.710 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-23T16:17:26.710 INFO:teuthology.orchestra.run.smithi103.stdout: "osdc_blkin_trace_all": "false", 2024-04-23T16:17:26.710 INFO:teuthology.orchestra.run.smithi103.stdout: "paxos_kill_at": "0", 2024-04-23T16:17:26.710 INFO:teuthology.orchestra.run.smithi103.stdout: "paxos_max_join_drift": "10", 2024-04-23T16:17:26.710 INFO:teuthology.orchestra.run.smithi103.stdout: "paxos_min": "500", 2024-04-23T16:17:26.710 INFO:teuthology.orchestra.run.smithi103.stdout: "paxos_min_wait": "0.050000", 2024-04-23T16:17:26.710 INFO:teuthology.orchestra.run.smithi103.stdout: "paxos_propose_interval": "1.000000", 2024-04-23T16:17:26.710 INFO:teuthology.orchestra.run.smithi103.stdout: "paxos_service_trim_max": "500", 2024-04-23T16:17:26.710 INFO:teuthology.orchestra.run.smithi103.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-23T16:17:26.710 INFO:teuthology.orchestra.run.smithi103.stdout: "paxos_service_trim_min": "10", 2024-04-23T16:17:26.711 INFO:teuthology.orchestra.run.smithi103.stdout: "paxos_stash_full_interval": "25", 2024-04-23T16:17:26.711 INFO:teuthology.orchestra.run.smithi103.stdout: "paxos_trim_max": "500", 2024-04-23T16:17:26.711 INFO:teuthology.orchestra.run.smithi103.stdout: "paxos_trim_min": "250", 2024-04-23T16:17:26.711 INFO:teuthology.orchestra.run.smithi103.stdout: "perf": "true", 2024-04-23T16:17:26.711 INFO:teuthology.orchestra.run.smithi103.stdout: "pid_file": "", 2024-04-23T16:17:26.711 INFO:teuthology.orchestra.run.smithi103.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-23T16:17:26.711 INFO:teuthology.orchestra.run.smithi103.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-23T16:17:26.711 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "-", 2024-04-23T16:17:26.711 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrv": "", 2024-04-23T16:17:26.711 INFO:teuthology.orchestra.run.smithi103.stdout: "public_bind_addr": "-", 2024-04-23T16:17:26.711 INFO:teuthology.orchestra.run.smithi103.stdout: "public_network": "", 2024-04-23T16:17:26.711 INFO:teuthology.orchestra.run.smithi103.stdout: "public_network_interface": "", 2024-04-23T16:17:26.711 INFO:teuthology.orchestra.run.smithi103.stdout: "qat_compressor_enabled": "false", 2024-04-23T16:17:26.711 INFO:teuthology.orchestra.run.smithi103.stdout: "qat_compressor_session_max_number": "256", 2024-04-23T16:17:26.711 INFO:teuthology.orchestra.run.smithi103.stdout: "rados_mon_op_timeout": "0", 2024-04-23T16:17:26.712 INFO:teuthology.orchestra.run.smithi103.stdout: "rados_osd_op_timeout": "0", 2024-04-23T16:17:26.712 INFO:teuthology.orchestra.run.smithi103.stdout: "rados_tracing": "false", 2024-04-23T16:17:26.712 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_atime_update_interval": "60", 2024-04-23T16:17:26.712 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-23T16:17:26.712 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_balance_parent_reads": "false", 2024-04-23T16:17:26.712 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_balance_snap_reads": "false", 2024-04-23T16:17:26.712 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_blkin_trace_all": "false", 2024-04-23T16:17:26.712 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-23T16:17:26.712 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-23T16:17:26.712 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_cache": "true", 2024-04-23T16:17:26.712 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-23T16:17:26.712 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-23T16:17:26.712 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-23T16:17:26.712 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-23T16:17:26.712 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_cache_policy": "writearound", 2024-04-23T16:17:26.713 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_cache_size": "33554432", 2024-04-23T16:17:26.713 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-23T16:17:26.713 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-23T16:17:26.713 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_clone_copy_on_read": "false", 2024-04-23T16:17:26.713 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_compression_hint": "none", 2024-04-23T16:17:26.713 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_concurrent_management_ops": "10", 2024-04-23T16:17:26.713 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-23T16:17:26.713 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_default_clone_format": "auto", 2024-04-23T16:17:26.713 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_default_data_pool": "", 2024-04-23T16:17:26.713 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_default_features": "61", 2024-04-23T16:17:26.713 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_default_format": "2", 2024-04-23T16:17:26.713 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_default_map_options": "", 2024-04-23T16:17:26.713 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_default_order": "22", 2024-04-23T16:17:26.713 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_default_pool": "rbd", 2024-04-23T16:17:26.714 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-23T16:17:26.714 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_default_stripe_count": "0", 2024-04-23T16:17:26.714 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_default_stripe_unit": "0", 2024-04-23T16:17:26.714 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-23T16:17:26.714 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-23T16:17:26.714 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-23T16:17:26.714 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_enable_alloc_hint": "true", 2024-04-23T16:17:26.714 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-23T16:17:26.714 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_io_scheduler": "simple", 2024-04-23T16:17:26.714 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-23T16:17:26.714 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-23T16:17:26.714 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-23T16:17:26.714 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-23T16:17:26.714 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-23T16:17:26.715 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-23T16:17:26.715 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-23T16:17:26.715 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-23T16:17:26.715 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-23T16:17:26.715 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_journal_order": "24", 2024-04-23T16:17:26.715 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_journal_pool": "", 2024-04-23T16:17:26.715 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_journal_splay_width": "4", 2024-04-23T16:17:26.715 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_localize_parent_reads": "false", 2024-04-23T16:17:26.715 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_localize_snap_reads": "false", 2024-04-23T16:17:26.715 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-23T16:17:26.715 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-23T16:17:26.715 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-23T16:17:26.715 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-23T16:17:26.715 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-23T16:17:26.715 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-23T16:17:26.716 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-23T16:17:26.716 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-23T16:17:26.716 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-23T16:17:26.716 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-23T16:17:26.716 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-23T16:17:26.716 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-23T16:17:26.716 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-23T16:17:26.716 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-23T16:17:26.716 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-23T16:17:26.716 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-23T16:17:26.716 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-23T16:17:26.716 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-23T16:17:26.716 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-23T16:17:26.716 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-23T16:17:26.716 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-23T16:17:26.717 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-23T16:17:26.717 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-23T16:17:26.717 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-23T16:17:26.717 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-23T16:17:26.717 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-23T16:17:26.717 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-23T16:17:26.717 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-23T16:17:26.717 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-23T16:17:26.717 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-23T16:17:26.717 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-23T16:17:26.717 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_mtime_update_interval": "60", 2024-04-23T16:17:26.717 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_non_blocking_aio": "true", 2024-04-23T16:17:26.717 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_op_thread_timeout": "60", 2024-04-23T16:17:26.717 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_op_threads": "1", 2024-04-23T16:17:26.718 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_parent_cache_enabled": "false", 2024-04-23T16:17:26.718 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-23T16:17:26.718 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-23T16:17:26.718 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-23T16:17:26.718 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_plugins": "", 2024-04-23T16:17:26.718 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_bps_burst": "0", 2024-04-23T16:17:26.718 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-23T16:17:26.718 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_bps_limit": "0", 2024-04-23T16:17:26.718 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_exclude_ops": "0", 2024-04-23T16:17:26.718 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_iops_burst": "0", 2024-04-23T16:17:26.718 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-23T16:17:26.718 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_iops_limit": "0", 2024-04-23T16:17:26.718 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-23T16:17:26.718 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-23T16:17:26.719 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-23T16:17:26.719 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-23T16:17:26.719 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-23T16:17:26.719 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-23T16:17:26.719 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-23T16:17:26.719 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-23T16:17:26.719 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-23T16:17:26.719 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-23T16:17:26.719 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-23T16:17:26.719 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-23T16:17:26.719 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-23T16:17:26.719 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-23T16:17:26.719 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_read_from_replica_policy": "default", 2024-04-23T16:17:26.719 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-23T16:17:26.719 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-23T16:17:26.720 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-23T16:17:26.720 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-23T16:17:26.720 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_skip_partial_discard": "true", 2024-04-23T16:17:26.720 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-23T16:17:26.720 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_tracing": "false", 2024-04-23T16:17:26.720 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_validate_names": "true", 2024-04-23T16:17:26.720 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_validate_pool": "true", 2024-04-23T16:17:26.720 INFO:teuthology.orchestra.run.smithi103.stdout: "restapi_base_url": "", 2024-04-23T16:17:26.720 INFO:teuthology.orchestra.run.smithi103.stdout: "restapi_log_level": "", 2024-04-23T16:17:26.720 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_acl_grants_max_num": "100", 2024-04-23T16:17:26.720 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_admin_entry": "admin", 2024-04-23T16:17:26.720 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-23T16:17:26.720 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_backend_store": "rados", 2024-04-23T16:17:26.720 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_barbican_url": "", 2024-04-23T16:17:26.720 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_beast_enable_async": "true", 2024-04-23T16:17:26.721 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_bucket_counters_cache": "false", 2024-04-23T16:17:26.721 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-23T16:17:26.721 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-23T16:17:26.721 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-23T16:17:26.721 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-23T16:17:26.721 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-23T16:17:26.721 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-23T16:17:26.721 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-23T16:17:26.721 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-23T16:17:26.721 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_cache_enabled": "true", 2024-04-23T16:17:26.721 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_cache_expiry_interval": "900", 2024-04-23T16:17:26.721 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_cache_lru_size": "10000", 2024-04-23T16:17:26.721 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_config_store": "rados", 2024-04-23T16:17:26.721 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_content_length_compat": "false", 2024-04-23T16:17:26.721 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_copy_obj_progress": "true", 2024-04-23T16:17:26.722 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-23T16:17:26.722 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_cors_rules_max_num": "100", 2024-04-23T16:17:26.722 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_cross_domain_policy": "", 2024-04-23T16:17:26.722 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-23T16:17:26.722 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_kmip_addr": "", 2024-04-23T16:17:26.722 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-23T16:17:26.722 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-23T16:17:26.722 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-23T16:17:26.722 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-23T16:17:26.722 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_kmip_password": "", 2024-04-23T16:17:26.722 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-23T16:17:26.722 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_kmip_username": "", 2024-04-23T16:17:26.722 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_require_ssl": "true", 2024-04-23T16:17:26.722 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-23T16:17:26.723 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-23T16:17:26.723 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-23T16:17:26.723 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-23T16:17:26.723 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-23T16:17:26.723 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-23T16:17:26.723 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-23T16:17:26.723 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-23T16:17:26.723 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-23T16:17:26.723 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-23T16:17:26.723 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-23T16:17:26.723 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-23T16:17:26.723 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-23T16:17:26.723 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-23T16:17:26.723 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-23T16:17:26.723 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_vault_addr": "", 2024-04-23T16:17:26.723 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_vault_auth": "token", 2024-04-23T16:17:26.724 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_vault_namespace": "", 2024-04-23T16:17:26.724 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_vault_prefix": "", 2024-04-23T16:17:26.724 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-23T16:17:26.724 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-23T16:17:26.724 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-23T16:17:26.724 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-23T16:17:26.724 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_vault_token_file": "", 2024-04-23T16:17:26.724 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-23T16:17:26.724 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_curl_buffersize": "524288", 2024-04-23T16:17:26.724 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-23T16:17:26.724 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_curl_low_speed_time": "300", 2024-04-23T16:17:26.724 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-23T16:17:26.724 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-23T16:17:26.724 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-23T16:17:26.724 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-23T16:17:26.725 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-23T16:17:26.725 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-23T16:17:26.725 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-23T16:17:26.725 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-23T16:17:26.725 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-23T16:17:26.725 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-23T16:17:26.725 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-23T16:17:26.725 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-04-23T16:17:26.725 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-04-23T16:17:26.725 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-04-23T16:17:26.725 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_d4n_l1_fadvise": "4", 2024-04-23T16:17:26.725 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-04-23T16:17:26.725 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-04-23T16:17:26.725 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_d4n_port": "6379", 2024-04-23T16:17:26.725 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-23T16:17:26.725 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_data_log_changes_size": "1000", 2024-04-23T16:17:26.726 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_data_log_num_shards": "128", 2024-04-23T16:17:26.726 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-23T16:17:26.726 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_data_log_window": "30", 2024-04-23T16:17:26.726 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-23T16:17:26.726 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-23T16:17:26.726 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-23T16:17:26.726 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-23T16:17:26.726 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-23T16:17:26.726 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-23T16:17:26.726 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-23T16:17:26.726 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-23T16:17:26.726 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-23T16:17:26.726 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-23T16:17:26.726 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-23T16:17:26.726 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-23T16:17:26.726 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-23T16:17:26.727 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-23T16:17:26.727 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-23T16:17:26.727 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-23T16:17:26.727 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-23T16:17:26.727 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-23T16:17:26.727 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-23T16:17:26.727 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-23T16:17:26.727 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-23T16:17:26.727 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-23T16:17:26.727 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-23T16:17:26.727 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-23T16:17:26.727 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-23T16:17:26.727 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_dns_name": "", 2024-04-23T16:17:26.727 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_dns_s3website_name": "", 2024-04-23T16:17:26.727 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_dynamic_resharding": "true", 2024-04-23T16:17:26.727 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-23T16:17:26.728 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_enable_gc_threads": "true", 2024-04-23T16:17:26.728 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_enable_lc_threads": "true", 2024-04-23T16:17:26.728 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_enable_ops_log": "false", 2024-04-23T16:17:26.728 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_enable_quota_threads": "true", 2024-04-23T16:17:26.728 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_enable_static_website": "false", 2024-04-23T16:17:26.728 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_enable_usage_log": "false", 2024-04-23T16:17:26.728 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_enforce_swift_acls": "true", 2024-04-23T16:17:26.728 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_exit_timeout_secs": "120", 2024-04-23T16:17:26.728 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_expose_bucket": "false", 2024-04-23T16:17:26.728 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_extended_http_attrs": "", 2024-04-23T16:17:26.728 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_filter": "none", 2024-04-23T16:17:26.728 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-23T16:17:26.728 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_frontends": "beast port=7480", 2024-04-23T16:17:26.728 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-23T16:17:26.729 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_gc_max_deferred": "50", 2024-04-23T16:17:26.729 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-23T16:17:26.729 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_gc_max_objs": "32", 2024-04-23T16:17:26.729 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-23T16:17:26.729 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-23T16:17:26.729 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-23T16:17:26.729 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-23T16:17:26.729 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_gc_processor_period": "3600", 2024-04-23T16:17:26.729 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-23T16:17:26.729 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-23T16:17:26.729 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-23T16:17:26.729 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-23T16:17:26.729 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_init_timeout": "300", 2024-04-23T16:17:26.729 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-23T16:17:26.730 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-23T16:17:26.730 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_kafka_connection_idle": "300", 2024-04-23T16:17:26.730 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-23T16:17:26.730 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-23T16:17:26.730 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-23T16:17:26.730 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-23T16:17:26.730 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-23T16:17:26.730 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_admin_domain": "", 2024-04-23T16:17:26.730 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_admin_password": "", 2024-04-23T16:17:26.730 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_admin_password_path": "", 2024-04-23T16:17:26.730 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_admin_project": "", 2024-04-23T16:17:26.730 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_admin_tenant": "", 2024-04-23T16:17:26.730 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_admin_token": "", 2024-04-23T16:17:26.730 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_admin_token_path": "", 2024-04-23T16:17:26.730 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_admin_user": "", 2024-04-23T16:17:26.731 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_api_version": "2", 2024-04-23T16:17:26.731 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_barbican_domain": "", 2024-04-23T16:17:26.731 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_barbican_password": "", 2024-04-23T16:17:26.731 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_barbican_project": "", 2024-04-23T16:17:26.731 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-23T16:17:26.731 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_barbican_user": "", 2024-04-23T16:17:26.731 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-23T16:17:26.731 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-23T16:17:26.731 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-23T16:17:26.731 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-23T16:17:26.731 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-23T16:17:26.731 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_url": "", 2024-04-23T16:17:26.731 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-23T16:17:26.731 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_lc_debug_interval": "-1", 2024-04-23T16:17:26.731 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_lc_lock_max_time": "90", 2024-04-23T16:17:26.732 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_lc_max_objs": "32", 2024-04-23T16:17:26.732 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_lc_max_rules": "1000", 2024-04-23T16:17:26.732 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_lc_max_worker": "3", 2024-04-23T16:17:26.732 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-23T16:17:26.732 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_lc_thread_delay": "0", 2024-04-23T16:17:26.732 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-23T16:17:26.732 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_ldap_dnattr": "uid", 2024-04-23T16:17:26.732 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-23T16:17:26.732 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_ldap_searchfilter": "", 2024-04-23T16:17:26.732 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-23T16:17:26.732 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_ldap_uri": "", 2024-04-23T16:17:26.732 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_lfuda_sync_frequency": "60", 2024-04-23T16:17:26.732 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-23T16:17:26.732 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-23T16:17:26.732 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-23T16:17:26.732 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_local_cache_address": "127.0.0.1:6379", 2024-04-23T16:17:26.733 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_log_http_headers": "", 2024-04-23T16:17:26.733 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-23T16:17:26.733 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-23T16:17:26.733 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_log_object_name_utc": "false", 2024-04-23T16:17:26.733 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-23T16:17:26.733 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-23T16:17:26.733 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_max_attr_name_len": "0", 2024-04-23T16:17:26.733 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_max_attr_size": "0", 2024-04-23T16:17:26.733 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-23T16:17:26.733 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_max_chunk_size": "4194304", 2024-04-23T16:17:26.733 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-23T16:17:26.733 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-23T16:17:26.733 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-23T16:17:26.733 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_max_listing_results": "1000", 2024-04-23T16:17:26.733 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_max_notify_retries": "10", 2024-04-23T16:17:26.733 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-23T16:17:26.734 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_max_put_param_size": "1048576", 2024-04-23T16:17:26.734 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_max_put_size": "5368709120", 2024-04-23T16:17:26.734 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_max_slo_entries": "1000", 2024-04-23T16:17:26.734 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_md_log_max_shards": "64", 2024-04-23T16:17:26.734 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-23T16:17:26.734 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-23T16:17:26.734 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-23T16:17:26.734 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-23T16:17:26.734 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_mp_lock_max_time": "600", 2024-04-23T16:17:26.734 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-23T16:17:26.734 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-23T16:17:26.734 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-23T16:17:26.734 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-23T16:17:26.734 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-23T16:17:26.734 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-23T16:17:26.735 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-23T16:17:26.735 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-23T16:17:26.735 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_nfs_max_gc": "300", 2024-04-23T16:17:26.735 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-23T16:17:26.735 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-23T16:17:26.735 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-23T16:17:26.735 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-23T16:17:26.735 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-23T16:17:26.735 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-23T16:17:26.735 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-23T16:17:26.735 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_num_async_rados_threads": "32", 2024-04-23T16:17:26.735 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_num_control_oids": "8", 2024-04-23T16:17:26.735 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_numa_node": "-1", 2024-04-23T16:17:26.735 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-23T16:17:26.736 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-23T16:17:26.736 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_objexp_chunk_size": "100", 2024-04-23T16:17:26.736 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_objexp_gc_interval": "600", 2024-04-23T16:17:26.736 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-23T16:17:26.736 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-23T16:17:26.736 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-23T16:17:26.736 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_op_thread_timeout": "600", 2024-04-23T16:17:26.736 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_op_tracing": "false", 2024-04-23T16:17:26.736 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_opa_token": "", 2024-04-23T16:17:26.736 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_opa_url": "", 2024-04-23T16:17:26.736 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_opa_verify_ssl": "true", 2024-04-23T16:17:26.736 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-23T16:17:26.736 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-23T16:17:26.736 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_ops_log_rados": "false", 2024-04-23T16:17:26.736 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_ops_log_socket_path": "", 2024-04-23T16:17:26.736 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-23T16:17:26.737 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-23T16:17:26.737 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-23T16:17:26.737 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_period_push_interval": "2.000000", 2024-04-23T16:17:26.737 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-23T16:17:26.737 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-23T16:17:26.737 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-23T16:17:26.737 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-23T16:17:26.737 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_posix_cache_lanes": "3", 2024-04-23T16:17:26.737 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-23T16:17:26.737 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-23T16:17:26.737 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_posix_cache_partitions": "3", 2024-04-23T16:17:26.737 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-23T16:17:26.737 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_print_continue": "true", 2024-04-23T16:17:26.737 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-23T16:17:26.738 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-23T16:17:26.738 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-23T16:17:26.738 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-23T16:17:26.738 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-23T16:17:26.738 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_rados_tracing": "false", 2024-04-23T16:17:26.738 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_realm": "", 2024-04-23T16:17:26.738 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_realm_id": "", 2024-04-23T16:17:26.738 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-23T16:17:26.738 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_region": "", 2024-04-23T16:17:26.738 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-23T16:17:26.738 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-23T16:17:26.738 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-23T16:17:26.738 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-23T16:17:26.739 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_request_uri": "", 2024-04-23T16:17:26.739 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_reshard_batch_size": "64", 2024-04-23T16:17:26.739 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-23T16:17:26.739 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_reshard_max_aio": "128", 2024-04-23T16:17:26.739 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_reshard_num_logs": "16", 2024-04-23T16:17:26.739 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_reshard_thread_interval": "600", 2024-04-23T16:17:26.739 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_resolve_cname": "false", 2024-04-23T16:17:26.739 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-23T16:17:26.739 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_run_sync_thread": "true", 2024-04-23T16:17:26.739 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-23T16:17:26.739 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-23T16:17:26.739 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-23T16:17:26.739 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-23T16:17:26.739 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-23T16:17:26.740 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-23T16:17:26.740 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-23T16:17:26.740 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-23T16:17:26.740 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-23T16:17:26.740 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_scheduler_type": "throttler", 2024-04-23T16:17:26.740 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_script_uri": "", 2024-04-23T16:17:26.740 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_service_provider_name": "", 2024-04-23T16:17:26.740 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-23T16:17:26.740 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_sts_client_id": "", 2024-04-23T16:17:26.740 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_sts_client_secret": "", 2024-04-23T16:17:26.740 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_sts_entry": "sts", 2024-04-23T16:17:26.740 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_sts_key": "sts", 2024-04-23T16:17:26.740 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-23T16:17:26.740 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_sts_min_session_duration": "900", 2024-04-23T16:17:26.740 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_sts_token_introspection_url": "", 2024-04-23T16:17:26.741 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_swift_account_in_url": "false", 2024-04-23T16:17:26.741 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_swift_auth_entry": "auth", 2024-04-23T16:17:26.741 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_swift_auth_url": "", 2024-04-23T16:17:26.741 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_swift_custom_header": "", 2024-04-23T16:17:26.741 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-23T16:17:26.741 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_swift_need_stats": "true", 2024-04-23T16:17:26.741 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_swift_tenant_name": "", 2024-04-23T16:17:26.741 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_swift_token_expiration": "86400", 2024-04-23T16:17:26.741 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_swift_url": "", 2024-04-23T16:17:26.741 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_swift_url_prefix": "swift", 2024-04-23T16:17:26.741 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-23T16:17:26.741 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-23T16:17:26.741 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-23T16:17:26.741 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_sync_lease_period": "120", 2024-04-23T16:17:26.741 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-23T16:17:26.741 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-23T16:17:26.742 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-23T16:17:26.742 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-23T16:17:26.742 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-23T16:17:26.742 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-23T16:17:26.742 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-23T16:17:26.742 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-23T16:17:26.742 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-23T16:17:26.742 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_thread_pool_size": "512", 2024-04-23T16:17:26.742 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-23T16:17:26.742 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-23T16:17:26.742 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-23T16:17:26.742 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-23T16:17:26.742 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_torrent_comment": "", 2024-04-23T16:17:26.743 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_torrent_createby": "", 2024-04-23T16:17:26.743 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_torrent_encoding": "", 2024-04-23T16:17:26.743 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_torrent_flag": "false", 2024-04-23T16:17:26.743 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-23T16:17:26.743 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_torrent_origin": "", 2024-04-23T16:17:26.743 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-23T16:17:26.743 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_torrent_tracker": "", 2024-04-23T16:17:26.743 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_trust_forwarded_https": "false", 2024-04-23T16:17:26.743 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-23T16:17:26.743 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-23T16:17:26.743 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_usage_max_shards": "32", 2024-04-23T16:17:26.743 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_usage_max_user_shards": "1", 2024-04-23T16:17:26.743 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_use_opa_authz": "false", 2024-04-23T16:17:26.743 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_user_counters_cache": "false", 2024-04-23T16:17:26.743 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-23T16:17:26.743 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-23T16:17:26.744 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-23T16:17:26.744 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_user_max_buckets": "1000", 2024-04-23T16:17:26.744 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_user_policies_max_num": "100", 2024-04-23T16:17:26.744 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-23T16:17:26.744 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-23T16:17:26.744 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-23T16:17:26.744 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-23T16:17:26.744 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_user_unique_email": "true", 2024-04-23T16:17:26.744 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_verify_ssl": "true", 2024-04-23T16:17:26.744 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-23T16:17:26.744 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_zone": "", 2024-04-23T16:17:26.744 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_zone_id": "", 2024-04-23T16:17:26.744 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-23T16:17:26.744 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_zonegroup": "", 2024-04-23T16:17:26.744 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_zonegroup_id": "", 2024-04-23T16:17:26.744 INFO:teuthology.orchestra.run.smithi103.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-23T16:17:26.744 INFO:teuthology.orchestra.run.smithi103.stdout: "rgwlc_auto_session_clear": "true", 2024-04-23T16:17:26.745 INFO:teuthology.orchestra.run.smithi103.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-23T16:17:26.745 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_block_size": "4096", 2024-04-23T16:17:26.745 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-23T16:17:26.745 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-23T16:17:26.745 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-23T16:17:26.745 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-23T16:17:26.745 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-23T16:17:26.745 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_cache_size": "536870912", 2024-04-23T16:17:26.745 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-23T16:17:26.745 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-23T16:17:26.745 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-23T16:17:26.745 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-23T16:17:26.745 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-23T16:17:26.745 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-23T16:17:26.745 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-23T16:17:26.745 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-23T16:17:26.746 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_index_type": "binary_search", 2024-04-23T16:17:26.746 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-23T16:17:26.746 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-23T16:17:26.746 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_partition_filters": "false", 2024-04-23T16:17:26.746 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_perf": "false", 2024-04-23T16:17:26.746 INFO:teuthology.orchestra.run.smithi103.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-23T16:17:26.746 INFO:teuthology.orchestra.run.smithi103.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-23T16:17:26.746 INFO:teuthology.orchestra.run.smithi103.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-23T16:17:26.746 INFO:teuthology.orchestra.run.smithi103.stdout: "run_dir": "/var/run/ceph", 2024-04-23T16:17:26.746 INFO:teuthology.orchestra.run.smithi103.stdout: "seastore_block_create": "true", 2024-04-23T16:17:26.746 INFO:teuthology.orchestra.run.smithi103.stdout: "seastore_cache_lru_size": "67108864", 2024-04-23T16:17:26.746 INFO:teuthology.orchestra.run.smithi103.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-23T16:17:26.746 INFO:teuthology.orchestra.run.smithi103.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-23T16:17:26.746 INFO:teuthology.orchestra.run.smithi103.stdout: "seastore_default_max_object_size": "16777216", 2024-04-23T16:17:26.746 INFO:teuthology.orchestra.run.smithi103.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-23T16:17:26.746 INFO:teuthology.orchestra.run.smithi103.stdout: "seastore_device_size": "53687091200", 2024-04-23T16:17:26.747 INFO:teuthology.orchestra.run.smithi103.stdout: "seastore_journal_batch_capacity": "16", 2024-04-23T16:17:26.747 INFO:teuthology.orchestra.run.smithi103.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-23T16:17:26.747 INFO:teuthology.orchestra.run.smithi103.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-23T16:17:26.747 INFO:teuthology.orchestra.run.smithi103.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-23T16:17:26.747 INFO:teuthology.orchestra.run.smithi103.stdout: "seastore_main_device_type": "SSD", 2024-04-23T16:17:26.747 INFO:teuthology.orchestra.run.smithi103.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-23T16:17:26.747 INFO:teuthology.orchestra.run.smithi103.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-23T16:17:26.747 INFO:teuthology.orchestra.run.smithi103.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-23T16:17:26.747 INFO:teuthology.orchestra.run.smithi103.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-23T16:17:26.747 INFO:teuthology.orchestra.run.smithi103.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-23T16:17:26.747 INFO:teuthology.orchestra.run.smithi103.stdout: "seastore_segment_size": "67108864", 2024-04-23T16:17:26.747 INFO:teuthology.orchestra.run.smithi103.stdout: "set_keepcaps": "false", 2024-04-23T16:17:26.747 INFO:teuthology.orchestra.run.smithi103.stdout: "setgroup": "ceph", 2024-04-23T16:17:26.747 INFO:teuthology.orchestra.run.smithi103.stdout: "setuser": "ceph", 2024-04-23T16:17:26.747 INFO:teuthology.orchestra.run.smithi103.stdout: "setuser_match_path": "", 2024-04-23T16:17:26.747 INFO:teuthology.orchestra.run.smithi103.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-23T16:17:26.748 INFO:teuthology.orchestra.run.smithi103.stdout: "thp": "false", 2024-04-23T16:17:26.748 INFO:teuthology.orchestra.run.smithi103.stdout: "threadpool_default_timeout": "60", 2024-04-23T16:17:26.748 INFO:teuthology.orchestra.run.smithi103.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-23T16:17:26.748 INFO:teuthology.orchestra.run.smithi103.stdout: "throttler_perf_counter": "true" 2024-04-23T16:17:26.748 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:17:27.372 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-23T16:17:27.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:27 smithi103 ceph-mon[39629]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:27.638 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:27.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:27 smithi148 ceph-mon[44520]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:27.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:27 smithi188 ceph-mon[44101]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:29.398 INFO:teuthology.orchestra.run.smithi103.stdout:mon.a: {} 2024-04-23T16:17:29.409 INFO:teuthology.orchestra.run.smithi103.stdout:mon.c: {} 2024-04-23T16:17:29.420 INFO:teuthology.orchestra.run.smithi103.stdout:mon.b: {} 2024-04-23T16:17:29.421 INFO:teuthology.orchestra.run.smithi103.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-23T16:17:29.421 INFO:teuthology.orchestra.run.smithi103.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-23T16:17:29.421 INFO:teuthology.orchestra.run.smithi103.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-23T16:17:29.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:29 smithi103 ceph-mon[39629]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:29 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3899590367' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T16:17:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:29 smithi148 ceph-mon[44520]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:29 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3899590367' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T16:17:29.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:29 smithi188 ceph-mon[44101]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:29 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3899590367' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T16:17:30.194 INFO:teuthology.run_tasks:Running task full_sequential... 2024-04-23T16:17:30.203 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-23T16:17:30.205 INFO:tasks.radosbench:Beginning radosbench... 2024-04-23T16:17:30.205 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-23T16:17:30.205 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-04-23T16:17:30.206 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-23T16:17:30.206 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-23T16:17:30.206 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:17:30.206 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-23T16:17:30.206 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-04-23T16:17:30.207 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-23T16:17:30.207 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:17:30.207 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-23T16:17:30.207 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:17:30.208 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-23T16:17:30.208 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-23T16:17:30.208 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-23T16:17:30.208 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:17:30.246 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:30.268 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:17:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:30.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:17:30.373 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:17:30.408 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:30.430 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:17:30.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:30.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:17:30.535 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:17:30.560 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:30.564 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:30.567 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:30.568 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:30.589 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:17:30.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:30.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:17:30.693 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:17:30.723 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:30.743 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:17:30.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:30.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:17:30.847 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:17:30.880 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:30.900 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:17:30.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:30.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:17:31.004 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:17:31.032 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:31.131 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:17:31.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:31.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:17:31.146 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:31.146 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:31.235 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:17:31.267 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:31.288 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:17:31.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:31.291 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:17:31.392 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:17:31.422 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:31.442 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:17:31.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:31.446 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:17:31.546 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:17:31.574 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:31.672 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:17:31.674 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:31 smithi103 ceph-mon[39629]: pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:31.674 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:31 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:17:31.674 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:31 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:17:31.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:31.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:17:31.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:31 smithi188 ceph-mon[44101]: pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:31.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:31 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:17:31.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:31 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:17:31.779 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:17:31.809 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:31 smithi148 ceph-mon[44520]: pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:31.809 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:31 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:17:31.809 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:31 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:17:31.811 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:31.833 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:17:31.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:31.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:17:31.936 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:17:31.968 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:31.988 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:17:31.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:31.992 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:17:32.093 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:17:32.123 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:32.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:32 smithi103 ceph-mon[39629]: osdmap e107: 12 total, 12 up, 12 in 2024-04-23T16:17:32.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:32 smithi148 ceph-mon[44520]: osdmap e107: 12 total, 12 up, 12 in 2024-04-23T16:17:32.971 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:17:32.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:32 smithi188 ceph-mon[44101]: osdmap e107: 12 total, 12 up, 12 in 2024-04-23T16:17:32.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:32.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:17:33.076 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:17:33.103 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:33.130 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:17:33.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:33.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:17:33.234 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:17:33.264 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:33 smithi103 ceph-mon[39629]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:33.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:33 smithi148 ceph-mon[44520]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:33.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:33 smithi188 ceph-mon[44101]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:34.113 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:17:34.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:34.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:17:34.218 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:17:34.249 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:34.271 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:17:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:34.274 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:17:34.375 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:17:34.407 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:34.428 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:17:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:34.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:17:34.532 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:17:34.560 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:34.711 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:17:34.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:34.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:17:34.814 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:17:34.843 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:34 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:17:34.843 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:34 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:17:34.843 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:34 smithi103 ceph-mon[39629]: osdmap e108: 12 total, 12 up, 12 in 2024-04-23T16:17:34.843 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:34 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:17:34.844 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:34 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:17:34.845 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:34.872 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:17:34.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:34 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:17:34.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:34 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:17:34.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:34 smithi148 ceph-mon[44520]: osdmap e108: 12 total, 12 up, 12 in 2024-04-23T16:17:34.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:34 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:17:34.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:34 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:17:34.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:34.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:17:34.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:34 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:17:34.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:34 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:17:34.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:34 smithi188 ceph-mon[44101]: osdmap e108: 12 total, 12 up, 12 in 2024-04-23T16:17:34.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:34 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:17:34.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:34 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:17:34.976 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:17:35.003 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:35.786 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'unique_pool_0' created 2024-04-23T16:17:35.807 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:35 smithi103 ceph-mon[39629]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:35.807 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:35 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:17:35.807 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:35 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:17:35.807 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:35 smithi103 ceph-mon[39629]: osdmap e109: 12 total, 12 up, 12 in 2024-04-23T16:17:35.807 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:35 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2485884446' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-23T16:17:35.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:35 smithi148 ceph-mon[44520]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:35.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:35 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:17:35.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:35 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:17:35.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:35 smithi148 ceph-mon[44520]: osdmap e109: 12 total, 12 up, 12 in 2024-04-23T16:17:35.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:35 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2485884446' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-23T16:17:35.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:35 smithi188 ceph-mon[44101]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:35.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:35 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:17:35.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:35 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:17:35.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:35 smithi188 ceph-mon[44101]: osdmap e109: 12 total, 12 up, 12 in 2024-04-23T16:17:35.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:35 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2485884446' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-23T16:17:36.789 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:17:37.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2485884446' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-04-23T16:17:37.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:36 smithi103 ceph-mon[39629]: osdmap e110: 12 total, 12 up, 12 in 2024-04-23T16:17:37.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2638753190' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:17:37.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:36 smithi103 ceph-mon[39629]: pgmap v351: 17 pgs: 3 creating+peering, 1 active+clean, 13 unknown; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:37.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2485884446' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-04-23T16:17:37.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:36 smithi148 ceph-mon[44520]: osdmap e110: 12 total, 12 up, 12 in 2024-04-23T16:17:37.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2638753190' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:17:37.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:36 smithi148 ceph-mon[44520]: pgmap v351: 17 pgs: 3 creating+peering, 1 active+clean, 13 unknown; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:37.153 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:17:37.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:37.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:17:37.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2485884446' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-04-23T16:17:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:36 smithi188 ceph-mon[44101]: osdmap e110: 12 total, 12 up, 12 in 2024-04-23T16:17:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2638753190' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:17:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:36 smithi188 ceph-mon[44101]: pgmap v351: 17 pgs: 3 creating+peering, 1 active+clean, 13 unknown; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:37.259 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:17:37.288 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:37.436 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:17:37.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:37.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:17:37.540 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:17:37.572 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:37.593 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:17:37.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:37.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:17:37.696 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:17:37.729 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:37.750 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:17:37.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:37.753 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:17:37.854 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:17:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:37 smithi148 ceph-mon[44520]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:17:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:37 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:17:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:37 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2638753190' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:17:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:37 smithi148 ceph-mon[44520]: osdmap e111: 12 total, 12 up, 12 in 2024-04-23T16:17:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:37 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:17:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:37 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:17:37.883 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:37.902 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:17:37.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:37.905 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:17:38.006 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:17:38.039 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:38.060 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:17:38.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:38.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:17:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:37 smithi103 ceph-mon[39629]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:17:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:37 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:17:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:37 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2638753190' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:17:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:37 smithi103 ceph-mon[39629]: osdmap e111: 12 total, 12 up, 12 in 2024-04-23T16:17:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:37 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:17:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:37 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:17:38.164 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:17:38.191 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:38.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:37 smithi188 ceph-mon[44101]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:17:38.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:37 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:17:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:37 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2638753190' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:17:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:37 smithi188 ceph-mon[44101]: osdmap e111: 12 total, 12 up, 12 in 2024-04-23T16:17:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:37 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:17:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:37 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:17:38.623 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-23T16:17:38.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:38 smithi103 ceph-mon[39629]: osdmap e112: 12 total, 12 up, 12 in 2024-04-23T16:17:38.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:38 smithi103 ceph-mon[39629]: pgmap v354: 17 pgs: 3 creating+peering, 6 active+clean, 8 unknown; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:39.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:38 smithi148 ceph-mon[44520]: osdmap e112: 12 total, 12 up, 12 in 2024-04-23T16:17:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:38 smithi148 ceph-mon[44520]: pgmap v354: 17 pgs: 3 creating+peering, 6 active+clean, 8 unknown; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:39.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:38 smithi188 ceph-mon[44101]: osdmap e112: 12 total, 12 up, 12 in 2024-04-23T16:17:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:38 smithi188 ceph-mon[44101]: pgmap v354: 17 pgs: 3 creating+peering, 6 active+clean, 8 unknown; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:39.350 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:17:39.350 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:17:39.350 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-23T16:17:39.350 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:17:39.373 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:17:39.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:39.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:17:39.458 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:17:39.477 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:17:39.528 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:39.529 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-04-23T16:17:39.529 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-04-23T16:17:39.529 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-04-23T16:17:39.529 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-04-23T16:17:39.529 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-04-23T16:17:39.529 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-04-23T16:17:39.529 INFO:teuthology.orchestra.run.smithi103.stderr:bench [] [] [] [] 2024-04-23T16:17:39.529 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs debug_inject_read_zeros 2024-04-23T16:17:39.530 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs files list 2024-04-23T16:17:39.530 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs stats 2024-04-23T16:17:39.530 INFO:teuthology.orchestra.run.smithi103.stderr:bluestore allocator dump block 2024-04-23T16:17:39.530 INFO:teuthology.orchestra.run.smithi103.stderr:Error EINVAL: invalid command 2024-04-23T16:17:39.678 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:17:39.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:39.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:17:39.783 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:17:39.815 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:39.836 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:17:39.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:39.839 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:17:39.940 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:17:39.968 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:40.021 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:17:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:40.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:17:40.124 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:17:40.151 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:40.399 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:17:40.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:40.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:17:40.432 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-23T16:17:40.503 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:17:40.534 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:40.555 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:17:40.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:40.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:17:40.659 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:17:40.691 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:40.714 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:17:40.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:40.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:17:40.816 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:17:40.846 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:40.866 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:17:40.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:40.869 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:17:40.885 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T16:17:40.886 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-23T16:17:40.886 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-04-23T16:17:40.969 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:17:40.998 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:41.018 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:17:41.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:41.021 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:17:41.122 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:17:41.154 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:41.174 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:17:41.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:41.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:17:41.278 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:17:41.310 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:41.317 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:41.331 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:17:41.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:41.334 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:17:41.334 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:17:41.358 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:41.359 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:17:41.380 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:17:41.399 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:17:41.419 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:17:41.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:17:41.439 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:17:41.462 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:41.463 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:17:41.484 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:17:41.504 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:17:41.526 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:17:41.545 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:17:41.566 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:17:41.567 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:17:41.567 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:17:41.567 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:17:41.567 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:17:41.567 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:17:41.567 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:17:41.567 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:17:41.567 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:17:41.567 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:17:41.567 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:17:41.567 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:17:41.568 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:17:41.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:41 smithi188 ceph-mon[44101]: pgmap v355: 17 pgs: 3 creating+peering, 12 active+clean, 2 unknown; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:41.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:41 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3509359107' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:17:41.843 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:17:41.845 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:41 smithi103 ceph-mon[39629]: pgmap v355: 17 pgs: 3 creating+peering, 12 active+clean, 2 unknown; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:41.845 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:41 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3509359107' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:17:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:41.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:17:41.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:41 smithi148 ceph-mon[44520]: pgmap v355: 17 pgs: 3 creating+peering, 12 active+clean, 2 unknown; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:41.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:41 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3509359107' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:17:41.947 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:17:41.979 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:41.993 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:42.000 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:17:42.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:42.003 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:17:42.104 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:17:42.134 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:42.155 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:17:42.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:42.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:17:42.258 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:17:42.287 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:42.307 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:17:42.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:42.310 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:17:42.411 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:17:42.442 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:42.540 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:17:42.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:42.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:17:42.652 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:17:42.680 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:42.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:42.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:17:43.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:43 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:17:43.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:43 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:17:43.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:43 smithi103 ceph-mon[39629]: pgmap v356: 17 pgs: 17 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:43.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:43 smithi103 ceph-mon[39629]: osdmap e113: 12 total, 12 up, 12 in 2024-04-23T16:17:43.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:43 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:17:43.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:43 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:17:43.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:43 smithi148 ceph-mon[44520]: pgmap v356: 17 pgs: 17 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:43.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:43 smithi148 ceph-mon[44520]: osdmap e113: 12 total, 12 up, 12 in 2024-04-23T16:17:43.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:43 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:17:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:43 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:17:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:43 smithi188 ceph-mon[44101]: pgmap v356: 17 pgs: 17 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:43 smithi188 ceph-mon[44101]: osdmap e113: 12 total, 12 up, 12 in 2024-04-23T16:17:44.495 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:17:44.813 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:44 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/70292943' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:17:44.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:44 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/70292943' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:17:44.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:44 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/70292943' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:17:45.129 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:17:45.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:45.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:17:45.233 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:17:45.265 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:45.286 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:17:45.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:45.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:17:45.390 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:17:45.421 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:45.620 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:17:45.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:45.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:17:45.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:45 smithi148 ceph-mon[44520]: pgmap v358: 17 pgs: 17 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:45.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:45 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:17:45.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:45 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/70292943' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:17:45.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:45 smithi148 ceph-mon[44520]: osdmap e114: 12 total, 12 up, 12 in 2024-04-23T16:17:45.724 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:17:45.730 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:45 smithi103 ceph-mon[39629]: pgmap v358: 17 pgs: 17 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:45.730 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:45 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:17:45.730 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:45 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/70292943' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:17:45.730 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:45 smithi103 ceph-mon[39629]: osdmap e114: 12 total, 12 up, 12 in 2024-04-23T16:17:45.739 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:17:45.756 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:45.778 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:17:45.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:45.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:17:45.882 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:17:45.912 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:45.931 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:17:45.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:45.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:17:45.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:45 smithi188 ceph-mon[44101]: pgmap v358: 17 pgs: 17 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:45.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:45 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:17:45.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:45 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/70292943' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:17:45.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:45 smithi188 ceph-mon[44101]: osdmap e114: 12 total, 12 up, 12 in 2024-04-23T16:17:46.036 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:17:46.063 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:46.083 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:17:46.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:46.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:17:46.187 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:17:46.219 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:46.240 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:17:46.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:46.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:17:46.312 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:46.344 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:17:46.374 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:46.395 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:17:46.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:46.398 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:17:46.499 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:17:46.526 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:47.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:47 smithi103 ceph-mon[39629]: pgmap v360: 17 pgs: 17 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:47.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:47 smithi148 ceph-mon[44520]: pgmap v360: 17 pgs: 17 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:47.943 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:17:47.943 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:17:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:47 smithi188 ceph-mon[44101]: pgmap v360: 17 pgs: 17 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:48.103 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:17:48.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:48.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:17:48.207 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:17:48.239 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:48.260 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:17:48.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:48.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:17:48.364 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:17:48.393 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:48.413 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:17:48.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:48.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:17:48.502 INFO:teuthology.orchestra.run.smithi103.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-23T16:17:48.516 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:17:48.548 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:48.568 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:17:48.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:48.571 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:17:48.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:48 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/780877491' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:17:48.672 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:17:48.701 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:48.721 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:17:48.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:48.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:17:48.765 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:17:48.824 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:17:48.852 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:48.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:48 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/780877491' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:17:48.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:48 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/780877491' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:17:49.502 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:17:49.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:49.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:17:49.606 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:17:49.638 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:49.659 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:17:49.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:49.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:17:49.762 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:17:49.791 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:49 smithi148 ceph-mon[44520]: pgmap v361: 17 pgs: 17 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:49.791 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:49 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/780877491' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:17:49.791 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:49 smithi148 ceph-mon[44520]: osdmap e115: 12 total, 12 up, 12 in 2024-04-23T16:17:49.793 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:49.813 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:17:49.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:49.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:17:49.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:49 smithi103 ceph-mon[39629]: pgmap v361: 17 pgs: 17 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:49.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:49 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/780877491' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:17:49.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:49 smithi103 ceph-mon[39629]: osdmap e115: 12 total, 12 up, 12 in 2024-04-23T16:17:49.916 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:17:49.945 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:49 smithi188 ceph-mon[44101]: pgmap v361: 17 pgs: 17 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:49.945 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:49 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/780877491' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:17:49.945 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:49 smithi188 ceph-mon[44101]: osdmap e115: 12 total, 12 up, 12 in 2024-04-23T16:17:49.947 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:49.968 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:17:49.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:49.972 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:17:50.017 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-23T16:17:50.072 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:17:50.101 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:50.121 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:17:50.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:50.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:17:50.225 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:17:50.257 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:50.278 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:17:50.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:50.281 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:17:50.382 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:17:50.408 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:50.530 DEBUG:teuthology.orchestra.run.smithi188:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-04-23T16:17:50.533 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-23T16:17:50.533 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-23T16:17:50.605 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:17:50.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:50.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:17:50.656 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:50.687 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:50.709 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:17:50.738 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:50.758 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:17:50.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:50.761 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:17:50.807 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:hints = 1 2024-04-23T16:17:50.807 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-23T16:17:50.807 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Object prefix: benchmark_data_smithi188_77627 2024-04-23T16:17:50.808 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:17:50.808 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 0 0 0 0 0 0 - 0 2024-04-23T16:17:50.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:50 smithi103 ceph-mon[39629]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:17:50.862 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:17:50.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:50 smithi148 ceph-mon[44520]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:17:50.899 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:50.926 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:17:50.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:50.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:17:50.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:50 smithi188 ceph-mon[44101]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:17:51.030 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:17:51.062 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:51.087 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:17:51.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:51.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:17:51.191 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:17:51.240 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:51.554 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:51 smithi103 ceph-mon[39629]: pgmap v363: 17 pgs: 17 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:51.687 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-04-23T16:17:51.687 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-04-23T16:17:51.687 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-04-23T16:17:51.687 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-04-23T16:17:51.687 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-04-23T16:17:51.687 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-04-23T16:17:51.687 INFO:teuthology.orchestra.run.smithi103.stderr:bench [] [] [] [] 2024-04-23T16:17:51.687 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs debug_inject_read_zeros 2024-04-23T16:17:51.687 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs files list 2024-04-23T16:17:51.687 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs stats 2024-04-23T16:17:51.687 INFO:teuthology.orchestra.run.smithi103.stderr:bluestore allocator dump block 2024-04-23T16:17:51.688 INFO:teuthology.orchestra.run.smithi103.stderr:Error EINVAL: invalid command 2024-04-23T16:17:51.808 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 1 16 5874 5858 366.028 366.125 0.00204593 0.00272469 2024-04-23T16:17:51.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:51 smithi148 ceph-mon[44520]: pgmap v363: 17 pgs: 17 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:51.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:51 smithi188 ceph-mon[44101]: pgmap v363: 17 pgs: 17 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:17:52.808 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 2 16 10584 10568 330.183 294.375 0.00245418 0.00302458 2024-04-23T16:17:53.004 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:17:53.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:53.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:17:53.112 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:17:53.158 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:53.627 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:17:53.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:53.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:17:53.731 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:17:53.766 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:53.769 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:53 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:17:53.769 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:53 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:17:53.769 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:53 smithi103 ceph-mon[39629]: pgmap v364: 17 pgs: 17 active+clean; 17 MiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s wr, 33 op/s 2024-04-23T16:17:53.769 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:53 smithi103 ceph-mon[39629]: osdmap e116: 12 total, 12 up, 12 in 2024-04-23T16:17:53.771 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:53.790 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:17:53.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:53.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:17:53.808 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 3 15 15652 15637 325.711 316.812 0.0055901 0.00306702 2024-04-23T16:17:53.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:53 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:17:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:53 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:17:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:53 smithi148 ceph-mon[44520]: pgmap v364: 17 pgs: 17 active+clean; 17 MiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s wr, 33 op/s 2024-04-23T16:17:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:53 smithi148 ceph-mon[44520]: osdmap e116: 12 total, 12 up, 12 in 2024-04-23T16:17:53.894 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:17:53.933 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:53.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:53 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:17:53.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:53 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:17:53.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:53 smithi188 ceph-mon[44101]: pgmap v364: 17 pgs: 17 active+clean; 17 MiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s wr, 33 op/s 2024-04-23T16:17:53.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:53 smithi188 ceph-mon[44101]: osdmap e116: 12 total, 12 up, 12 in 2024-04-23T16:17:54.493 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:17:54.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:54.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:17:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T16:17:54.513 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-23T16:17:54.513 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-04-23T16:17:54.596 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:17:54.628 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:54.653 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:17:54.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:54.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:17:54.757 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:17:54.797 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:54.808 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 4 16 20172 20156 314.887 282.438 0.00325514 0.00317178 2024-04-23T16:17:54.821 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:17:54.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:54.824 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:17:54.925 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:17:54.970 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:54.995 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:17:54.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:54.999 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:17:55.060 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:54 smithi103 ceph-mon[39629]: pgmap v366: 17 pgs: 17 active+clean; 443 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 884 op/s 2024-04-23T16:17:55.099 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:17:55.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:54 smithi148 ceph-mon[44520]: pgmap v366: 17 pgs: 17 active+clean; 443 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 884 op/s 2024-04-23T16:17:55.133 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:55.156 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:17:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:55.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:17:55.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:54 smithi188 ceph-mon[44101]: pgmap v366: 17 pgs: 17 active+clean; 443 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 884 op/s 2024-04-23T16:17:55.261 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:17:55.292 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:55.316 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:17:55.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:55.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:17:55.421 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:17:55.457 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:55.484 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:17:55.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:55.488 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:17:55.529 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:17:55.529 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:17:55.589 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:17:55.631 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:55.662 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:17:55.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:55.667 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:17:55.767 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:17:55.802 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:55.805 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:55 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/749393778' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:17:55.806 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:55 smithi103 ceph-mon[39629]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:17:55.808 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 5 16 24174 24158 301.931 250.125 0.0139377 0.0033092 2024-04-23T16:17:55.812 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:17:55.828 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:17:55.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:55.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:17:55.845 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:55.933 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:17:55.975 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:56.004 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:17:56.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:56.009 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:17:56.110 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:17:56.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:55 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/749393778' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:17:56.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:55 smithi148 ceph-mon[44520]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:17:56.153 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:56.188 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:17:56.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:56.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:17:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:55 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/749393778' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:17:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:55 smithi188 ceph-mon[44101]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:17:56.294 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:17:56.324 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:56.349 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:17:56.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:56.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:17:56.454 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:17:56.487 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:56.509 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:17:56.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:56.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:17:56.614 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:17:56.659 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:56.705 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:17:56.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:56.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:17:56.748 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-23T16:17:56.808 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 6 16 28128 28112 292.794 247.125 0.00525312 0.00341361 2024-04-23T16:17:56.814 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:17:56.847 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:56.870 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:17:56.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:56.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:17:56.976 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:17:56.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:56 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:17:56.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:56 smithi103 ceph-mon[39629]: from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:17:56.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:56 smithi103 ceph-mon[39629]: osdmap e117: 12 total, 12 up, 12 in 2024-04-23T16:17:56.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:56 smithi103 ceph-mon[39629]: pgmap v368: 17 pgs: 17 active+clean; 1.0 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s wr, 2.12k op/s 2024-04-23T16:17:57.019 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:57.046 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:17:57.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:57.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:17:57.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:56 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:17:57.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:56 smithi148 ceph-mon[44520]: from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:17:57.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:56 smithi148 ceph-mon[44520]: osdmap e117: 12 total, 12 up, 12 in 2024-04-23T16:17:57.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:56 smithi148 ceph-mon[44520]: pgmap v368: 17 pgs: 17 active+clean; 1.0 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s wr, 2.12k op/s 2024-04-23T16:17:57.150 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:17:57.177 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:57.195 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:17:57.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:57.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:17:57.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:56 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:17:57.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:56 smithi188 ceph-mon[44101]: from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:17:57.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:56 smithi188 ceph-mon[44101]: osdmap e117: 12 total, 12 up, 12 in 2024-04-23T16:17:57.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:56 smithi188 ceph-mon[44101]: pgmap v368: 17 pgs: 17 active+clean; 1.0 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s wr, 2.12k op/s 2024-04-23T16:17:57.299 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:17:57.331 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:57.357 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:17:57.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:57.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:17:57.461 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:17:57.493 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:57.517 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:17:57.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:57.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:17:57.621 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:17:57.663 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:57.808 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 7 15 31659 31644 282.5 220.75 0.00430088 0.00353855 2024-04-23T16:17:58.023 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:17:58.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:58.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:17:58.127 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:17:58.165 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:58.528 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:17:58.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:58.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:17:58.632 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:17:58.668 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:58.689 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:17:58.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:58.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:17:58.794 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:17:58.808 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 8 15 35305 35290 275.67 227.875 0.00729303 0.00362542 2024-04-23T16:17:58.838 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:59.043 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:17:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:59.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:17:59.122 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:17:59.148 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:17:59.178 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:59.204 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:17:59.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:17:59.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:17:59.311 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:17:59.357 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:17:59.491 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:17:59 smithi103 ceph-mon[39629]: pgmap v369: 17 pgs: 17 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 183 MiB/s wr, 2.93k op/s 2024-04-23T16:17:59.494 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:17:59.808 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 9 16 38688 38672 268.524 211.375 0.00486611 0.00372238 2024-04-23T16:17:59.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:17:59 smithi148 ceph-mon[44520]: pgmap v369: 17 pgs: 17 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 183 MiB/s wr, 2.93k op/s 2024-04-23T16:17:59.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:17:59 smithi188 ceph-mon[44101]: pgmap v369: 17 pgs: 17 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 183 MiB/s wr, 2.93k op/s 2024-04-23T16:18:00.809 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 10 16 41837 41821 261.35 196.812 0.00202733 0.00381171 2024-04-23T16:18:01.809 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 11 16 45074 45058 255.982 202.312 0.00267717 0.00390498 2024-04-23T16:18:01.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:01 smithi103 ceph-mon[39629]: pgmap v370: 17 pgs: 17 active+clean; 1.9 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 239 MiB/s wr, 3.82k op/s 2024-04-23T16:18:01.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:01 smithi148 ceph-mon[44520]: pgmap v370: 17 pgs: 17 active+clean; 1.9 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 239 MiB/s wr, 3.82k op/s 2024-04-23T16:18:01.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:01 smithi188 ceph-mon[44101]: pgmap v370: 17 pgs: 17 active+clean; 1.9 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 239 MiB/s wr, 3.82k op/s 2024-04-23T16:18:02.809 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 12 16 47456 47440 247.053 148.875 0.00438145 0.00404595 2024-04-23T16:18:03.809 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 13 16 49944 49928 240.009 155.5 0.00166392 0.00414911 2024-04-23T16:18:03.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:03 smithi103 ceph-mon[39629]: pgmap v371: 17 pgs: 17 active+clean; 2.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 221 MiB/s wr, 3.54k op/s 2024-04-23T16:18:03.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:03 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:18:03.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:03 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:18:03.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:03 smithi148 ceph-mon[44520]: pgmap v371: 17 pgs: 17 active+clean; 2.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 221 MiB/s wr, 3.54k op/s 2024-04-23T16:18:03.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:03 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:18:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:03 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:18:03.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:03 smithi188 ceph-mon[44101]: pgmap v371: 17 pgs: 17 active+clean; 2.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 221 MiB/s wr, 3.54k op/s 2024-04-23T16:18:03.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:03 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:18:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:03 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:18:04.025 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:18:04.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:04.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:18:04.132 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:18:04.171 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:04.429 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:18:04.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:04.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:18:04.534 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:18:04.566 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:04.586 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:18:04.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:04.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:18:04.691 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:18:04.730 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:04 smithi103 ceph-mon[39629]: osdmap e118: 12 total, 12 up, 12 in 2024-04-23T16:18:04.730 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:04 smithi103 ceph-mon[39629]: osdmap e119: 12 total, 12 up, 12 in 2024-04-23T16:18:04.733 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:04.744 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-04-23T16:18:04.744 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-04-23T16:18:04.744 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-04-23T16:18:04.744 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-04-23T16:18:04.745 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-04-23T16:18:04.745 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-04-23T16:18:04.745 INFO:teuthology.orchestra.run.smithi103.stderr:bench [] [] [] [] 2024-04-23T16:18:04.745 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs debug_inject_read_zeros 2024-04-23T16:18:04.745 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs files list 2024-04-23T16:18:04.745 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs stats 2024-04-23T16:18:04.745 INFO:teuthology.orchestra.run.smithi103.stderr:bluestore allocator dump block 2024-04-23T16:18:04.745 INFO:teuthology.orchestra.run.smithi103.stderr:Error EINVAL: invalid command 2024-04-23T16:18:04.774 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:18:04.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:04.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:18:04.809 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 14 16 52285 52269 233.316 146.312 0.0100795 0.00428416 2024-04-23T16:18:04.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:04 smithi148 ceph-mon[44520]: osdmap e118: 12 total, 12 up, 12 in 2024-04-23T16:18:04.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:04 smithi148 ceph-mon[44520]: osdmap e119: 12 total, 12 up, 12 in 2024-04-23T16:18:04.881 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:18:04.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:04 smithi188 ceph-mon[44101]: osdmap e118: 12 total, 12 up, 12 in 2024-04-23T16:18:04.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:04 smithi188 ceph-mon[44101]: osdmap e119: 12 total, 12 up, 12 in 2024-04-23T16:18:04.918 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:04.947 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:18:04.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:04.950 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:18:05.051 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:05.084 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:05.109 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:05.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:05.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:05.213 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:18:05.244 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:05.272 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:18:05.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:05.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:18:05.377 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:18:05.419 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:05.452 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:18:05.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:05.457 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:18:05.534 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:05 smithi103 ceph-mon[39629]: pgmap v374: 17 pgs: 17 active+clean; 2.8 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 221 MiB/s wr, 3.53k op/s 2024-04-23T16:18:05.534 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:05 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:18:05.534 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:05 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:18:05.534 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:05 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1066497368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:18:05.558 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:18:05.587 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:05.610 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:18:05.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:05.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:18:05.715 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:18:05.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:05 smithi188 ceph-mon[44101]: pgmap v374: 17 pgs: 17 active+clean; 2.8 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 221 MiB/s wr, 3.53k op/s 2024-04-23T16:18:05.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:05 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:18:05.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:05 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:18:05.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:05 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1066497368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:18:05.752 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:05.782 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:18:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:05.785 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:18:05.809 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 15 16 54940 54924 228.824 165.938 0.00386081 0.00436776 2024-04-23T16:18:05.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:05 smithi148 ceph-mon[44520]: pgmap v374: 17 pgs: 17 active+clean; 2.8 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 221 MiB/s wr, 3.53k op/s 2024-04-23T16:18:05.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:05 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:18:05.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:05 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:18:05.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:05 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1066497368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:18:05.886 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:18:05.935 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:05.961 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:18:05.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:05.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:18:06.066 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:06.095 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:06.120 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:06.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:06.225 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:18:06.269 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:06.299 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:18:06.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:06.303 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:18:06.390 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T16:18:06.390 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-23T16:18:06.390 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-04-23T16:18:06.404 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:18:06.442 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:06.469 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:18:06.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:06.472 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:18:06.489 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:18:06.489 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:18:06.489 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:18:06.489 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:18:06.490 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:18:06.490 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:18:06.490 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:18:06.490 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:18:06.490 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:18:06.490 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:18:06.490 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:18:06.490 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:18:06.490 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:18:06.490 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:18:06.490 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:18:06.491 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:18:06.491 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:18:06.491 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:18:06.491 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:18:06.491 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:18:06.491 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:18:06.491 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:18:06.491 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:18:06.491 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:18:06.573 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:18:06.615 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:06.643 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:18:06.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:06.647 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:18:06.690 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:18:06.748 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:18:06.788 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.788 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.788 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.788 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.788 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.788 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.788 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.788 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.788 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:06 smithi103 ceph-mon[39629]: osdmap e120: 12 total, 12 up, 12 in 2024-04-23T16:18:06.788 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.788 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.791 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:06.809 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 16 16 57493 57477 224.494 159.562 0.00470855 0.00445246 2024-04-23T16:18:06.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:06 smithi148 ceph-mon[44520]: osdmap e120: 12 total, 12 up, 12 in 2024-04-23T16:18:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:06 smithi188 ceph-mon[44101]: osdmap e120: 12 total, 12 up, 12 in 2024-04-23T16:18:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:07.586 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:18:07.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:07.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:18:07.699 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:07.733 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:07.757 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:07.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:07.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:07.809 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 17 16 60117 60101 220.935 164 0.00535631 0.0045245 2024-04-23T16:18:07.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:07.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:07.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:07 smithi103 ceph-mon[39629]: pgmap v376: 17 pgs: 17 active+clean; 3.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 208 MiB/s wr, 3.33k op/s 2024-04-23T16:18:07.862 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:18:07.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:07.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:07.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:07 smithi148 ceph-mon[44520]: pgmap v376: 17 pgs: 17 active+clean; 3.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 208 MiB/s wr, 3.33k op/s 2024-04-23T16:18:07.906 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:07.907 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:07.907 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:07 smithi188 ceph-mon[44101]: pgmap v376: 17 pgs: 17 active+clean; 3.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 208 MiB/s wr, 3.33k op/s 2024-04-23T16:18:07.911 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:07.937 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:18:07.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:07.941 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:18:08.042 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:18:08.082 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:08.114 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:18:08.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:08.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:18:08.223 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:18:08.266 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:08.295 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:18:08.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:08.298 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:18:08.399 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:08.431 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:08.455 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:08.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:08.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:08.560 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:18:08.598 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:08.665 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:18:08.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:08.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:18:08.776 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:18:08.809 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 18 16 62575 62559 217.195 153.625 0.00503302 0.00460279 2024-04-23T16:18:08.813 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:08.832 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:18:08.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:08.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:18:08.937 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:18:08.967 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:08.989 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:18:08.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:08.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:18:09.064 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:18:09.094 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:18:09.136 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:09.148 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:18:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:18:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:63367 2.4 2:28721e8e:::benchmark_data_smithi188_77627_object63366:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e121)", 2024-04-23T16:18:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:09.139351+0000", 2024-04-23T16:18:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.0032795739999999999, 2024-04-23T16:18:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0033135019999999999, 2024-04-23T16:18:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:18:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:18:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:18:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:18:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:18:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:18:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 63367 2024-04-23T16:18:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:18:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:18:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.139351+0000", 2024-04-23T16:18:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:18:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.139351+0000", 2024-04-23T16:18:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:18:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.139353+0000", 2024-04-23T16:18:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4959999999999999e-06 2024-04-23T16:18:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:18:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.139414+0000", 2024-04-23T16:18:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1842999999999997e-05 2024-04-23T16:18:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:18:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.139423+0000", 2024-04-23T16:18:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8720000000000008e-06 2024-04-23T16:18:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:18:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.139429+0000", 2024-04-23T16:18:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2769999999999996e-06 2024-04-23T16:18:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:18:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.139472+0000", 2024-04-23T16:18:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3925999999999999e-05 2024-04-23T16:18:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:18:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.139547+0000", 2024-04-23T16:18:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4548000000000005e-05 2024-04-23T16:18:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-23T16:18:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.139628+0000", 2024-04-23T16:18:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1395000000000003e-05 2024-04-23T16:18:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:18:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:63369 2.6 2:607afac1:::benchmark_data_smithi188_77627_object63368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e121)", 2024-04-23T16:18:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:09.140470+0000", 2024-04-23T16:18:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.0021606099999999999, 2024-04-23T16:18:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022810769999999998, 2024-04-23T16:18:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:18:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:18:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:18:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:18:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:18:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:18:09.154 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 63369 2024-04-23T16:18:09.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.154 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:18:09.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:18:09.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.140470+0000", 2024-04-23T16:18:09.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:09.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:18:09.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.140470+0000", 2024-04-23T16:18:09.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:09.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.155 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:18:09.155 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.140478+0000", 2024-04-23T16:18:09.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.242e-06 2024-04-23T16:18:09.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.155 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:18:09.155 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.140514+0000", 2024-04-23T16:18:09.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5249000000000001e-05 2024-04-23T16:18:09.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:18:09.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.140523+0000", 2024-04-23T16:18:09.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5750000000000007e-06 2024-04-23T16:18:09.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:18:09.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.140536+0000", 2024-04-23T16:18:09.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2998e-05 2024-04-23T16:18:09.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:18:09.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.140626+0000", 2024-04-23T16:18:09.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0093000000000006e-05 2024-04-23T16:18:09.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:18:09.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.140751+0000", 2024-04-23T16:18:09.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000124535 2024-04-23T16:18:09.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 8", 2024-04-23T16:18:09.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.140883+0000", 2024-04-23T16:18:09.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013235800000000001 2024-04-23T16:18:09.157 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:09.157 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:18:09.157 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:09.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.158 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:63370 2.6 2:6292ed9c:::benchmark_data_smithi188_77627_object63369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e121)", 2024-04-23T16:18:09.158 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:09.140554+0000", 2024-04-23T16:18:09.158 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.002076872, 2024-04-23T16:18:09.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044336599999999999, 2024-04-23T16:18:09.158 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:18:09.158 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:18:09.158 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:18:09.158 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:18:09.158 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:18:09.158 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:18:09.158 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 63370 2024-04-23T16:18:09.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.159 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:18:09.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:18:09.159 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.140554+0000", 2024-04-23T16:18:09.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:09.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:18:09.159 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.140554+0000", 2024-04-23T16:18:09.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:09.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:18:09.159 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.140555+0000", 2024-04-23T16:18:09.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8199999999999999e-07 2024-04-23T16:18:09.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:18:09.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.141496+0000", 2024-04-23T16:18:09.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00094174199999999997 2024-04-23T16:18:09.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:18:09.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.141506+0000", 2024-04-23T16:18:09.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7569999999999999e-06 2024-04-23T16:18:09.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:18:09.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.141513+0000", 2024-04-23T16:18:09.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2899999999999997e-06 2024-04-23T16:18:09.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:18:09.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.142576+0000", 2024-04-23T16:18:09.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001062592 2024-04-23T16:18:09.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:18:09.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.142656+0000", 2024-04-23T16:18:09.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0207e-05 2024-04-23T16:18:09.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 8", 2024-04-23T16:18:09.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.142734+0000", 2024-04-23T16:18:09.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8219000000000004e-05 2024-04-23T16:18:09.162 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:09.162 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:18:09.162 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:09.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.162 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:63371 2.6 2:68639202:::benchmark_data_smithi188_77627_object63370:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e121)", 2024-04-23T16:18:09.162 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:09.141525+0000", 2024-04-23T16:18:09.162 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.0011059310000000001, 2024-04-23T16:18:09.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035129010000000001, 2024-04-23T16:18:09.162 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:18:09.162 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:18:09.162 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:18:09.162 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:18:09.162 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:18:09.163 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:18:09.163 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 63371 2024-04-23T16:18:09.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.163 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:18:09.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:18:09.163 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.141525+0000", 2024-04-23T16:18:09.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:09.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:18:09.163 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.141525+0000", 2024-04-23T16:18:09.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:09.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:18:09.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.141526+0000", 2024-04-23T16:18:09.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0400000000000005e-07 2024-04-23T16:18:09.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:18:09.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.141558+0000", 2024-04-23T16:18:09.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1956999999999999e-05 2024-04-23T16:18:09.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:18:09.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.141563+0000", 2024-04-23T16:18:09.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.93e-06 2024-04-23T16:18:09.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:18:09.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.141569+0000", 2024-04-23T16:18:09.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2660000000000001e-06 2024-04-23T16:18:09.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:18:09.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.144187+0000", 2024-04-23T16:18:09.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002618399 2024-04-23T16:18:09.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:18:09.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.144283+0000", 2024-04-23T16:18:09.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5574000000000003e-05 2024-04-23T16:18:09.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.167 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 8", 2024-04-23T16:18:09.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.144366+0000", 2024-04-23T16:18:09.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2854000000000003e-05 2024-04-23T16:18:09.167 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:09.167 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:18:09.167 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:09.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.167 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:63372 2.6 2.d09cb2f6 (undecoded) ondisk+write+known_if_redirected e121)", 2024-04-23T16:18:09.167 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:09.141746+0000", 2024-04-23T16:18:09.167 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.00088414699999999995, 2024-04-23T16:18:09.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0033317519999999999, 2024-04-23T16:18:09.167 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:18:09.168 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:18:09.168 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "queued for pg", 2024-04-23T16:18:09.168 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:18:09.168 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:18:09.168 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:18:09.168 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 63372 2024-04-23T16:18:09.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.168 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:18:09.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:18:09.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.141746+0000", 2024-04-23T16:18:09.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:09.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:18:09.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.141746+0000", 2024-04-23T16:18:09.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:09.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:18:09.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.141748+0000", 2024-04-23T16:18:09.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2860000000000001e-06 2024-04-23T16:18:09.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:18:09.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.141780+0000", 2024-04-23T16:18:09.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2468999999999997e-05 2024-04-23T16:18:09.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:18:09.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.141788+0000", 2024-04-23T16:18:09.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.374e-06 2024-04-23T16:18:09.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:09.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:09.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:18:09.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:09.141794+0000", 2024-04-23T16:18:09.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3230000000000003e-06 2024-04-23T16:18:09.170 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:09.170 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:18:09.170 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:09.170 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:09.170 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:18:09.170 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 5 2024-04-23T16:18:09.171 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:18:09.171 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:18:09.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:09.172 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:18:09.173 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:18:09.273 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:18:09.337 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:09.413 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:18:09.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:09.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:18:09.517 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:18:09.558 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:09.587 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:18:09.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:09.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:18:09.693 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:18:09.747 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:09.810 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 19 16 64880 64864 213.346 144.062 0.00236533 0.00468541 2024-04-23T16:18:09.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:09 smithi103 ceph-mon[39629]: pgmap v377: 17 pgs: 17 active+clean; 3.3 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 193 MiB/s wr, 3.08k op/s 2024-04-23T16:18:09.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:09 smithi103 ceph-mon[39629]: osdmap e121: 12 total, 12 up, 12 in 2024-04-23T16:18:09.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:09 smithi148 ceph-mon[44520]: pgmap v377: 17 pgs: 17 active+clean; 3.3 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 193 MiB/s wr, 3.08k op/s 2024-04-23T16:18:09.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:09 smithi148 ceph-mon[44520]: osdmap e121: 12 total, 12 up, 12 in 2024-04-23T16:18:09.890 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:18:09.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:09.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:18:09.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:09 smithi188 ceph-mon[44101]: pgmap v377: 17 pgs: 17 active+clean; 3.3 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 193 MiB/s wr, 3.08k op/s 2024-04-23T16:18:09.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:09 smithi188 ceph-mon[44101]: osdmap e121: 12 total, 12 up, 12 in 2024-04-23T16:18:09.995 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:18:10.039 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:10.067 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:18:10.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:10.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:18:10.172 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:18:10.221 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:10.330 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:18:10.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:10.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:18:10.377 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-23T16:18:10.438 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:18:10.476 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:10.501 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:18:10.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:10.507 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:18:10.608 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:18:10.638 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:10.658 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:18:10.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:10.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:18:10.763 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:18:10.807 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:10.809 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:18:10.809915 min lat: 0.000949989 max lat: 0.207898 avg lat: 0.00476349 2024-04-23T16:18:10.809 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:18:10.809 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 20 15 67166 67151 209.825 142.938 0.0110458 0.00476349 2024-04-23T16:18:11.370 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:18:11.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:11.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:18:11.429 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:18:11.475 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:18:11.516 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:11.547 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:18:11.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:11.551 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:18:11.555 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:18:11.652 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:18:11.686 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:11 smithi103 ceph-mon[39629]: pgmap v379: 17 pgs: 17 active+clean; 3.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 144 MiB/s wr, 2.31k op/s 2024-04-23T16:18:11.687 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:11 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3173028932' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:18:11.687 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:11 smithi103 ceph-mon[39629]: from='client.24751 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:18:11.688 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:11.810 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 21 16 69332 69316 206.276 135.312 0.00333573 0.00484636 2024-04-23T16:18:11.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:11 smithi148 ceph-mon[44520]: pgmap v379: 17 pgs: 17 active+clean; 3.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 144 MiB/s wr, 2.31k op/s 2024-04-23T16:18:11.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:11 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3173028932' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:18:11.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:11 smithi148 ceph-mon[44520]: from='client.24751 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:18:11.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:11 smithi188 ceph-mon[44101]: pgmap v379: 17 pgs: 17 active+clean; 3.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 144 MiB/s wr, 2.31k op/s 2024-04-23T16:18:11.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:11 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3173028932' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:18:11.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:11 smithi188 ceph-mon[44101]: from='client.24751 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:18:12.254 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:18:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:12.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:18:12.361 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:12.393 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:12.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:12.419 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:12.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:12.495 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:18:12.520 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:18:12.549 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:12.558 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:12 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:18:12.558 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:12 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:18:12.558 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:12 smithi103 ceph-mon[39629]: from='client.24751 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:18:12.558 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:12 smithi103 ceph-mon[39629]: osdmap e122: 12 total, 12 up, 12 in 2024-04-23T16:18:12.572 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:18:12.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:12.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:18:12.678 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:18:12.711 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:12.742 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:18:12.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:12.747 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:18:12.810 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 22 15 71621 71606 203.405 143.125 0.00562544 0.00491485 2024-04-23T16:18:12.848 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:18:12.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:12 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:18:12.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:12 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:18:12.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:12 smithi148 ceph-mon[44520]: from='client.24751 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:18:12.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:12 smithi148 ceph-mon[44520]: osdmap e122: 12 total, 12 up, 12 in 2024-04-23T16:18:12.886 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:12 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:18:12.886 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:12 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:18:12.886 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:12 smithi188 ceph-mon[44101]: from='client.24751 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:18:12.886 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:12 smithi188 ceph-mon[44101]: osdmap e122: 12 total, 12 up, 12 in 2024-04-23T16:18:12.889 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:12.918 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:18:12.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:12.923 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:18:13.024 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:18:13.081 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:13.166 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:18:13.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:13.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:18:13.277 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:18:13.323 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:13.347 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:18:13.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:13.351 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:18:13.452 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:18:13.484 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:13.506 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:18:13.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:13.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:18:13.612 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:18:13.645 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:13 smithi188 ceph-mon[44101]: pgmap v381: 17 pgs: 17 active+clean; 3.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.25k op/s 2024-04-23T16:18:13.646 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:13.676 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:18:13.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:13.681 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:18:13.781 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:18:13.810 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 23 15 73758 73743 200.368 133.562 0.00777789 0.00498873 2024-04-23T16:18:13.817 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:13 smithi148 ceph-mon[44520]: pgmap v381: 17 pgs: 17 active+clean; 3.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.25k op/s 2024-04-23T16:18:13.818 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:13.844 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:18:13.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:13.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:18:13.848 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:13 smithi103 ceph-mon[39629]: pgmap v381: 17 pgs: 17 active+clean; 3.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.25k op/s 2024-04-23T16:18:13.948 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:13.980 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:14.012 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:14.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:14.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:14.116 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:18:14.161 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:14.472 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:18:14.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:14.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:18:14.580 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:18:14.605 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:18:14.605 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:14 smithi103 ceph-mon[39629]: osdmap e123: 12 total, 12 up, 12 in 2024-04-23T16:18:14.622 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:14.810 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 24 16 75739 75723 197.175 123.75 0.0047195 0.00507046 2024-04-23T16:18:14.832 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:18:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:14.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:18:14.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:18:14.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:14 smithi148 ceph-mon[44520]: osdmap e123: 12 total, 12 up, 12 in 2024-04-23T16:18:14.945 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:18:14.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:18:14.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:14 smithi188 ceph-mon[44101]: osdmap e123: 12 total, 12 up, 12 in 2024-04-23T16:18:14.981 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:15.060 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:18:15.545 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:18:15.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:15.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:18:15.650 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:18:15.700 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:15.729 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:18:15.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:15.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:18:15.810 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 25 16 77839 77823 194.538 131.25 0.00329532 0.00513883 2024-04-23T16:18:15.840 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:18:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:15 smithi103 ceph-mon[39629]: pgmap v383: 17 pgs: 17 active+clean; 4.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 156 MiB/s wr, 2.49k op/s 2024-04-23T16:18:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:18:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:18:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:18:15.868 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:15 smithi148 ceph-mon[44520]: pgmap v383: 17 pgs: 17 active+clean; 4.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 156 MiB/s wr, 2.49k op/s 2024-04-23T16:18:15.868 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:15 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:18:15.868 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:15 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:18:15.868 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:15 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:18:15.873 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:15.895 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:18:15.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:15.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:18:15.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:15 smithi188 ceph-mon[44101]: pgmap v383: 17 pgs: 17 active+clean; 4.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 156 MiB/s wr, 2.49k op/s 2024-04-23T16:18:15.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:18:15.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:18:15.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:18:15.999 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:18:16.036 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:16.069 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:18:16.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:16.072 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:18:16.173 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:18:16.265 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:16.302 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-04-23T16:18:16.302 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-04-23T16:18:16.302 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-04-23T16:18:16.302 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-04-23T16:18:16.302 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-04-23T16:18:16.302 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-04-23T16:18:16.302 INFO:teuthology.orchestra.run.smithi103.stderr:bench [] [] [] [] 2024-04-23T16:18:16.302 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs debug_inject_read_zeros 2024-04-23T16:18:16.302 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs files list 2024-04-23T16:18:16.303 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs stats 2024-04-23T16:18:16.303 INFO:teuthology.orchestra.run.smithi103.stderr:bluestore allocator dump block 2024-04-23T16:18:16.303 INFO:teuthology.orchestra.run.smithi103.stderr:Error EINVAL: invalid command 2024-04-23T16:18:16.810 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 26 16 79874 79858 191.947 127.188 0.00550277 0.0052086 2024-04-23T16:18:16.835 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:18:16.835 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:18:16.835 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-23T16:18:16.835 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-23T16:18:16.835 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:18:17.023 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:18:17.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:17.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:18:17.127 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:18:17.169 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:17.196 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:18:17.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:17.200 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:18:17.301 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:17.330 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:17.351 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:17.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:17.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:17.455 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:17.485 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:17.510 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:17.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:17.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:17.556 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T16:18:17.556 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-23T16:18:17.557 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-04-23T16:18:17.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:17 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:18:17.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:17 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:18:17.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:17 smithi103 ceph-mon[39629]: pgmap v384: 17 pgs: 17 active+clean; 4.5 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 142 MiB/s wr, 2.27k op/s 2024-04-23T16:18:17.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:17 smithi103 ceph-mon[39629]: osdmap e124: 12 total, 12 up, 12 in 2024-04-23T16:18:17.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:17 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/645006850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:18:17.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:17.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:17.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:17.614 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:18:17.648 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:17.673 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:18:17.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:17.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:18:17.779 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:18:17.810 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:17.810 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 27 16 81895 81879 189.516 126.312 0.00720449 0.00527545 2024-04-23T16:18:17.834 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:18:17.839 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:17 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:18:17.839 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:17 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:18:17.839 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:17 smithi148 ceph-mon[44520]: pgmap v384: 17 pgs: 17 active+clean; 4.5 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 142 MiB/s wr, 2.27k op/s 2024-04-23T16:18:17.839 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:17 smithi148 ceph-mon[44520]: osdmap e124: 12 total, 12 up, 12 in 2024-04-23T16:18:17.839 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:17 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/645006850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:18:17.839 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:17.839 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:17.839 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:17.839 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:17.839 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:17.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:17.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:18:17.941 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:18:17.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:17 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:18:17.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:17 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:18:17.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:17 smithi188 ceph-mon[44101]: pgmap v384: 17 pgs: 17 active+clean; 4.5 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 142 MiB/s wr, 2.27k op/s 2024-04-23T16:18:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:17 smithi188 ceph-mon[44101]: osdmap e124: 12 total, 12 up, 12 in 2024-04-23T16:18:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:17 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/645006850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:18:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:17.983 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:18.094 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:18:18.094 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:18:18.094 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:18:18.094 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:18:18.094 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:18:18.094 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:18:18.094 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:18:18.094 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:18:18.094 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:18:18.094 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:18:18.094 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:18:18.095 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:18:18.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:18:18.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:18:18.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:18:18.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:18:18.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:18:18.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:18:18.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:18:18.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:18:18.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:18:18.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:18:18.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:18:18.096 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:18:18.501 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-23T16:18:18.627 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:18:18.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:18.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:18:18.734 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:18.765 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:18.789 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:18.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:18.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:18.793 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:18.794 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:18.794 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:18.794 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:18.794 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:18.794 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:18.794 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:18 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:18:18.794 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:18 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:18:18.810 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 28 16 83961 83945 187.359 129.125 0.00351154 0.00533504 2024-04-23T16:18:18.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:18 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:18:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:18 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:18:18.893 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:18:18.896 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:18:18.922 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:18:18.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:18.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:18.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:18.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:18.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:18.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:18.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:18 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:18:18.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:18 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:18:18.930 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:18.956 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:18:18.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:18.959 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:18:19.061 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:18:19.099 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:19.219 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:18:19.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:19.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:18:19.326 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:18:19.372 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:19.431 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:18:19.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:19.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:18:19.539 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:19.570 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:19.590 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:19.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:19.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:19.630 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:19 smithi103 ceph-mon[39629]: pgmap v386: 17 pgs: 17 active+clean; 4.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s wr, 2.08k op/s 2024-04-23T16:18:19.630 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:19 smithi103 ceph-mon[39629]: osdmap e125: 12 total, 12 up, 12 in 2024-04-23T16:18:19.630 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:19 smithi103 ceph-mon[39629]: osdmap e126: 12 total, 12 up, 12 in 2024-04-23T16:18:19.695 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:19.727 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:19.753 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:19.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:19.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:19.810 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 29 16 85901 85885 185.079 121.25 0.00797295 0.00540064 2024-04-23T16:18:19.859 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:18:19.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:19 smithi148 ceph-mon[44520]: pgmap v386: 17 pgs: 17 active+clean; 4.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s wr, 2.08k op/s 2024-04-23T16:18:19.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:19 smithi148 ceph-mon[44520]: osdmap e125: 12 total, 12 up, 12 in 2024-04-23T16:18:19.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:19 smithi148 ceph-mon[44520]: osdmap e126: 12 total, 12 up, 12 in 2024-04-23T16:18:19.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:19 smithi188 ceph-mon[44101]: pgmap v386: 17 pgs: 17 active+clean; 4.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s wr, 2.08k op/s 2024-04-23T16:18:19.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:19 smithi188 ceph-mon[44101]: osdmap e125: 12 total, 12 up, 12 in 2024-04-23T16:18:19.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:19 smithi188 ceph-mon[44101]: osdmap e126: 12 total, 12 up, 12 in 2024-04-23T16:18:19.892 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:19.921 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:18:19.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:19.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:18:20.026 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:18:20.075 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:20.103 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:18:20.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:20.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:18:20.209 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:18:20.242 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:20.263 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:18:20.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:20.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:18:20.367 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:18:20.408 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:20.441 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:18:20.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:20.446 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:18:20.547 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:18:20.577 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:20.600 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:18:20.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:20.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:18:20.704 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:18:20.754 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:20.810 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 30 16 87874 87858 183.019 123.312 0.015376 0.00546206 2024-04-23T16:18:20.849 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:18:21.126 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:18:21.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:21.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:18:21.233 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:18:21.276 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:21.303 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:18:21.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:21.307 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:18:21.408 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:18:21.446 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:21.583 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:18:21.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:21.589 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:18:21.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:18:21.691 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:18:21.723 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:21.746 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:18:21.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:21.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:18:21.811 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 31 16 89779 89763 180.956 119.062 0.00900021 0.00552475 2024-04-23T16:18:21.849 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:18:21.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:21 smithi103 ceph-mon[39629]: pgmap v389: 17 pgs: 17 active+clean; 5.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s wr, 1.95k op/s 2024-04-23T16:18:21.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:21 smithi148 ceph-mon[44520]: pgmap v389: 17 pgs: 17 active+clean; 5.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s wr, 1.95k op/s 2024-04-23T16:18:21.898 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:21 smithi188 ceph-mon[44101]: pgmap v389: 17 pgs: 17 active+clean; 5.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s wr, 1.95k op/s 2024-04-23T16:18:21.906 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:21.933 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:18:21.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:21.937 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:18:22.038 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:18:22.077 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:22.811 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 32 16 91624 91608 178.904 115.312 0.00756545 0.00558835 2024-04-23T16:18:22.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:22 smithi188 ceph-mon[44101]: pgmap v390: 17 pgs: 17 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.69k op/s 2024-04-23T16:18:23.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:22 smithi103 ceph-mon[39629]: pgmap v390: 17 pgs: 17 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.69k op/s 2024-04-23T16:18:23.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:22 smithi148 ceph-mon[44520]: pgmap v390: 17 pgs: 17 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.69k op/s 2024-04-23T16:18:23.149 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:18:23.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:23.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:18:23.255 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:23.284 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:23.306 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:23.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:23.409 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:18:23.412 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:18:23.449 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:23.477 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:18:23.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:23.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:18:23.496 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-04-23T16:18:23.496 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-04-23T16:18:23.496 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-04-23T16:18:23.496 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-04-23T16:18:23.496 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-04-23T16:18:23.496 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-04-23T16:18:23.496 INFO:teuthology.orchestra.run.smithi103.stderr:bench [] [] [] [] 2024-04-23T16:18:23.496 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs debug_inject_read_zeros 2024-04-23T16:18:23.496 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs files list 2024-04-23T16:18:23.496 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs stats 2024-04-23T16:18:23.496 INFO:teuthology.orchestra.run.smithi103.stderr:bluestore allocator dump block 2024-04-23T16:18:23.496 INFO:teuthology.orchestra.run.smithi103.stderr:Error EINVAL: invalid command 2024-04-23T16:18:23.581 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:18:23.611 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:23.635 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:18:23.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:23.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:18:23.671 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:18:23.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:23 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1840129399' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:18:23.740 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:18:23.788 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:23 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1840129399' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:18:23.796 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:23.810 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 33 16 93464 93448 176.968 115 0.0104001 0.00564919 2024-04-23T16:18:23.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:23 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1840129399' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:18:23.930 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:18:23.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:23.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:18:24.041 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:18:24.081 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:24.113 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:18:24.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:24.116 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:18:24.217 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:18:24.256 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:24.288 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:18:24.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:24.294 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:18:24.396 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:18:24.425 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:24.448 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:18:24.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:24.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:18:24.554 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:18:24.600 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:24.629 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:18:24.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:24.633 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:18:24.733 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:18:24.775 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:24.811 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 34 15 95289 95274 175.119 114.125 0.0038945 0.00570891 2024-04-23T16:18:24.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:24 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:18:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:24 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1840129399' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:18:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:24 smithi148 ceph-mon[44520]: osdmap e127: 12 total, 12 up, 12 in 2024-04-23T16:18:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:24 smithi148 ceph-mon[44520]: osdmap e128: 12 total, 12 up, 12 in 2024-04-23T16:18:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:24 smithi148 ceph-mon[44520]: pgmap v393: 17 pgs: 17 active+clean; 5.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s wr, 2.13k op/s 2024-04-23T16:18:24.915 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:24 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:18:24.916 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1840129399' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:18:24.916 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:24 smithi103 ceph-mon[39629]: osdmap e127: 12 total, 12 up, 12 in 2024-04-23T16:18:24.916 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:24 smithi103 ceph-mon[39629]: osdmap e128: 12 total, 12 up, 12 in 2024-04-23T16:18:24.916 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:24 smithi103 ceph-mon[39629]: pgmap v393: 17 pgs: 17 active+clean; 5.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s wr, 2.13k op/s 2024-04-23T16:18:24.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:24 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:18:24.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1840129399' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:18:24.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:24 smithi188 ceph-mon[44101]: osdmap e127: 12 total, 12 up, 12 in 2024-04-23T16:18:24.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:24 smithi188 ceph-mon[44101]: osdmap e128: 12 total, 12 up, 12 in 2024-04-23T16:18:24.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:24 smithi188 ceph-mon[44101]: pgmap v393: 17 pgs: 17 active+clean; 5.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s wr, 2.13k op/s 2024-04-23T16:18:25.811 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 35 16 97173 97157 173.478 117.688 0.00309822 0.00576276 2024-04-23T16:18:25.817 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T16:18:25.817 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-23T16:18:25.818 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-04-23T16:18:26.087 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:18:26.088 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:18:26.543 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:18:26.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:26.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:18:26.651 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:18:26.677 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:18:26.691 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:26.714 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:18:26.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:26.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:18:26.811 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 36 16 98935 98919 171.718 110.125 0.011747 0.00582248 2024-04-23T16:18:26.818 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:18:26.863 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:27.025 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:18:27.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:27.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:18:27.059 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-23T16:18:27.130 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:18:27.160 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:27.182 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:18:27.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:27.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:18:27.286 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:18:27.316 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:27.337 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:18:27.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:27.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:18:27.442 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:18:27.490 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:27.499 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:27 smithi103 ceph-mon[39629]: pgmap v394: 17 pgs: 17 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 120 MiB/s wr, 1.91k op/s 2024-04-23T16:18:27.500 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:27 smithi148 ceph-mon[44520]: pgmap v394: 17 pgs: 17 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 120 MiB/s wr, 1.91k op/s 2024-04-23T16:18:27.517 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:18:27.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:27.520 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:18:27.621 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:18:27.665 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:27.695 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:18:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:27.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:18:27.806 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:18:27.811 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 37 16 100607 100591 169.901 104.5 0.00388851 0.00588431 2024-04-23T16:18:27.846 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:27.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:27 smithi188 ceph-mon[44101]: pgmap v394: 17 pgs: 17 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 120 MiB/s wr, 1.91k op/s 2024-04-23T16:18:28.029 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:18:28.035 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:18:28.059 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:18:28.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:28.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:18:28.170 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:28.199 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:28.222 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:28.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:28.328 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:18:28.398 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:28.811 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 38 16 102321 102305 168.249 107.125 0.00441636 0.0059417 2024-04-23T16:18:29.096 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:18:29.128 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:18:29.169 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:18:29.212 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:18:29.244 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:18:29.271 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:18:29.311 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:18:29.347 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:18:29.380 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:18:29.406 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:18:29.433 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:18:29.440 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:18:29.463 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:18:29.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:29.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:18:29.473 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:18:29.483 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:18:29.483 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:18:29.483 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:18:29.483 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:18:29.483 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:18:29.483 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:18:29.483 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:18:29.483 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:18:29.484 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:18:29.484 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:18:29.484 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:18:29.484 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:18:29.569 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:18:29.625 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:29.811 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 39 16 104034 104018 166.68 107.062 0.00512164 0.00599814 2024-04-23T16:18:29.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:29 smithi103 ceph-mon[39629]: pgmap v395: 17 pgs: 17 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.94k op/s 2024-04-23T16:18:29.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:29 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2985101571' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:18:29.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:29.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:29.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:29.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:29.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:29 smithi148 ceph-mon[44520]: pgmap v395: 17 pgs: 17 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.94k op/s 2024-04-23T16:18:29.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:29 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2985101571' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:18:29.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:29 smithi188 ceph-mon[44101]: pgmap v395: 17 pgs: 17 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.94k op/s 2024-04-23T16:18:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:29 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2985101571' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:18:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:29.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.811 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:18:30.811605 min lat: 0.000949989 max lat: 0.207898 avg lat: 0.00604878 2024-04-23T16:18:30.811 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:18:30.811 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 40 16 105810 105794 165.287 111 0.00331493 0.00604878 2024-04-23T16:18:30.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:30.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:31.811 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 41 16 107498 107482 163.829 105.5 0.00408959 0.00610247 2024-04-23T16:18:31.854 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:31 smithi103 ceph-mon[39629]: pgmap v396: 17 pgs: 17 active+clean; 6.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s wr, 1.96k op/s 2024-04-23T16:18:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:31 smithi148 ceph-mon[44520]: pgmap v396: 17 pgs: 17 active+clean; 6.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s wr, 1.96k op/s 2024-04-23T16:18:31.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:31 smithi188 ceph-mon[44101]: pgmap v396: 17 pgs: 17 active+clean; 6.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s wr, 1.96k op/s 2024-04-23T16:18:32.811 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 42 16 109282 109266 162.583 111.5 0.0181899 0.00614936 2024-04-23T16:18:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:32 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:18:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:32 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:18:32.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:32 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:18:32.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:32 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:18:32.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:32 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:18:32.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:32 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:18:33.812 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 43 16 110929 110913 161.195 102.938 0.00918313 0.00619927 2024-04-23T16:18:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:33 smithi103 ceph-mon[39629]: pgmap v397: 17 pgs: 17 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-04-23T16:18:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:33 smithi103 ceph-mon[39629]: osdmap e129: 12 total, 12 up, 12 in 2024-04-23T16:18:33.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:33 smithi148 ceph-mon[44520]: pgmap v397: 17 pgs: 17 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-04-23T16:18:33.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:33 smithi148 ceph-mon[44520]: osdmap e129: 12 total, 12 up, 12 in 2024-04-23T16:18:33.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:33 smithi188 ceph-mon[44101]: pgmap v397: 17 pgs: 17 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-04-23T16:18:33.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:33 smithi188 ceph-mon[44101]: osdmap e129: 12 total, 12 up, 12 in 2024-04-23T16:18:34.812 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 44 16 112538 112522 159.817 100.562 0.00554792 0.00625603 2024-04-23T16:18:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:34 smithi103 ceph-mon[39629]: osdmap e130: 12 total, 12 up, 12 in 2024-04-23T16:18:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:34 smithi103 ceph-mon[39629]: pgmap v400: 17 pgs: 17 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.72k op/s 2024-04-23T16:18:35.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:34 smithi148 ceph-mon[44520]: osdmap e130: 12 total, 12 up, 12 in 2024-04-23T16:18:35.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:34 smithi148 ceph-mon[44520]: pgmap v400: 17 pgs: 17 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.72k op/s 2024-04-23T16:18:35.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:34 smithi188 ceph-mon[44101]: osdmap e130: 12 total, 12 up, 12 in 2024-04-23T16:18:35.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:34 smithi188 ceph-mon[44101]: pgmap v400: 17 pgs: 17 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.72k op/s 2024-04-23T16:18:35.812 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 45 16 114189 114173 158.558 103.188 0.0178227 0.00630531 2024-04-23T16:18:35.817 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:35 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:18:35.817 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:35 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:18:35.818 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:35 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:18:35.818 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:35 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:18:36.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:35 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:18:36.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:35 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:18:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:35 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:18:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:35 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:18:36.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:35 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:18:36.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:35 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:18:36.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:35 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:18:36.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:35 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:18:36.812 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 46 16 115832 115816 157.343 102.688 0.00697273 0.00635386 2024-04-23T16:18:37.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:36 smithi103 ceph-mon[39629]: osdmap e131: 12 total, 12 up, 12 in 2024-04-23T16:18:37.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:36 smithi103 ceph-mon[39629]: pgmap v402: 17 pgs: 17 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 117 MiB/s wr, 1.88k op/s 2024-04-23T16:18:37.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:36 smithi148 ceph-mon[44520]: osdmap e131: 12 total, 12 up, 12 in 2024-04-23T16:18:37.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:36 smithi148 ceph-mon[44520]: pgmap v402: 17 pgs: 17 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 117 MiB/s wr, 1.88k op/s 2024-04-23T16:18:37.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:36 smithi188 ceph-mon[44101]: osdmap e131: 12 total, 12 up, 12 in 2024-04-23T16:18:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:36 smithi188 ceph-mon[44101]: pgmap v402: 17 pgs: 17 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 117 MiB/s wr, 1.88k op/s 2024-04-23T16:18:37.812 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 47 16 117331 117315 155.989 93.6875 0.0126848 0.00640962 2024-04-23T16:18:38.697 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:18:38.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:38.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:18:38.804 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:18:38.812 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 48 15 118923 118908 154.813 99.5625 0.00899974 0.00645817 2024-04-23T16:18:38.839 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:39.204 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:18:39.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:39.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:18:39.320 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:18:39.368 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:39.789 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:39 smithi103 ceph-mon[39629]: pgmap v403: 17 pgs: 17 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 1.83k op/s 2024-04-23T16:18:39.789 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:39 smithi103 ceph-mon[39629]: osdmap e132: 12 total, 12 up, 12 in 2024-04-23T16:18:39.812 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 49 16 120440 120424 153.587 94.75 0.00434618 0.00650915 2024-04-23T16:18:39.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:39 smithi148 ceph-mon[44520]: pgmap v403: 17 pgs: 17 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 1.83k op/s 2024-04-23T16:18:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:39 smithi148 ceph-mon[44520]: osdmap e132: 12 total, 12 up, 12 in 2024-04-23T16:18:39.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:39 smithi188 ceph-mon[44101]: pgmap v403: 17 pgs: 17 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 1.83k op/s 2024-04-23T16:18:39.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:39 smithi188 ceph-mon[44101]: osdmap e132: 12 total, 12 up, 12 in 2024-04-23T16:18:40.266 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-04-23T16:18:40.267 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-04-23T16:18:40.267 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-04-23T16:18:40.267 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-04-23T16:18:40.267 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-04-23T16:18:40.267 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-04-23T16:18:40.267 INFO:teuthology.orchestra.run.smithi103.stderr:bench [] [] [] [] 2024-04-23T16:18:40.267 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs debug_inject_read_zeros 2024-04-23T16:18:40.267 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs files list 2024-04-23T16:18:40.267 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs stats 2024-04-23T16:18:40.267 INFO:teuthology.orchestra.run.smithi103.stderr:bluestore allocator dump block 2024-04-23T16:18:40.267 INFO:teuthology.orchestra.run.smithi103.stderr:Error EINVAL: invalid command 2024-04-23T16:18:40.811 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:40 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:18:40.811 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:40 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:18:40.811 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:40 smithi103 ceph-mon[39629]: osdmap e133: 12 total, 12 up, 12 in 2024-04-23T16:18:40.811 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:40 smithi103 ceph-mon[39629]: pgmap v406: 17 pgs: 17 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-23T16:18:40.813 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 50 16 121959 121943 152.413 94.9375 0.0178806 0.00655939 2024-04-23T16:18:40.813 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:18:41.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:40 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2917247105' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:18:41.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:40 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:18:41.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:40 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:18:41.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:40 smithi148 ceph-mon[44520]: osdmap e133: 12 total, 12 up, 12 in 2024-04-23T16:18:41.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:40 smithi148 ceph-mon[44520]: pgmap v406: 17 pgs: 17 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-23T16:18:41.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:40 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2917247105' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:18:41.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:40 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:18:41.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:40 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:18:41.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:40 smithi188 ceph-mon[44101]: osdmap e133: 12 total, 12 up, 12 in 2024-04-23T16:18:41.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:40 smithi188 ceph-mon[44101]: pgmap v406: 17 pgs: 17 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-23T16:18:41.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:40 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2917247105' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:18:41.231 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:18:41.542 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:18:41.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:41.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:18:41.652 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:18:41.705 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:41.813 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 51 16 123449 123433 151.25 93.125 0.0102409 0.00661013 2024-04-23T16:18:41.813 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:41 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:18:41.813 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:41 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2917247105' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:18:41.813 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:41 smithi103 ceph-mon[39629]: osdmap e134: 12 total, 12 up, 12 in 2024-04-23T16:18:42.068 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:18:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:42.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:18:42.097 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T16:18:42.097 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-23T16:18:42.097 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-04-23T16:18:42.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:41 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:18:42.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:41 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2917247105' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:18:42.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:41 smithi148 ceph-mon[44520]: osdmap e134: 12 total, 12 up, 12 in 2024-04-23T16:18:42.179 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:18:42.207 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:41 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:18:42.207 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:41 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2917247105' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:18:42.207 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:41 smithi188 ceph-mon[44101]: osdmap e134: 12 total, 12 up, 12 in 2024-04-23T16:18:42.209 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:42.234 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:18:42.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:42.240 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:18:42.341 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:18:42.378 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:42.399 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:18:42.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:42.403 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:18:42.434 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:18:42.434 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:18:42.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.434 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:124289 2.e 2:79cc79bb:::benchmark_data_smithi188_77627_object124288:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e134)", 2024-04-23T16:18:42.434 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:42.401345+0000", 2024-04-23T16:18:42.434 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.029159485999999998, 2024-04-23T16:18:42.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.029171610000000001, 2024-04-23T16:18:42.435 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:18:42.435 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:18:42.435 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:18:42.435 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:18:42.435 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:18:42.435 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:18:42.435 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 124289 2024-04-23T16:18:42.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.435 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:18:42.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:18:42.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.401345+0000", 2024-04-23T16:18:42.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:18:42.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.401345+0000", 2024-04-23T16:18:42.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:18:42.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.401347+0000", 2024-04-23T16:18:42.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.883e-06 2024-04-23T16:18:42.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:18:42.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.401379+0000", 2024-04-23T16:18:42.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2005000000000003e-05 2024-04-23T16:18:42.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:18:42.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.401391+0000", 2024-04-23T16:18:42.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2841e-05 2024-04-23T16:18:42.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:18:42.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.401403+0000", 2024-04-23T16:18:42.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1379999999999999e-05 2024-04-23T16:18:42.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:18:42.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.401633+0000", 2024-04-23T16:18:42.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00023020500000000001 2024-04-23T16:18:42.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:18:42.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.401769+0000", 2024-04-23T16:18:42.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136069 2024-04-23T16:18:42.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:18:42.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.401933+0000", 2024-04-23T16:18:42.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016436900000000001 2024-04-23T16:18:42.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:18:42.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.407370+0000", 2024-04-23T16:18:42.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054368319999999999 2024-04-23T16:18:42.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:18:42.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.430449+0000", 2024-04-23T16:18:42.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.023078481000000001 2024-04-23T16:18:42.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:18:42.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.430485+0000", 2024-04-23T16:18:42.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6198999999999997e-05 2024-04-23T16:18:42.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:18:42.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.430516+0000", 2024-04-23T16:18:42.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1346000000000001e-05 2024-04-23T16:18:42.440 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.440 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:18:42.440 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.440 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:124296 2.e 2:7424192d:::benchmark_data_smithi188_77627_object124295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e134)", 2024-04-23T16:18:42.440 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:42.409600+0000", 2024-04-23T16:18:42.441 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.020903831000000001, 2024-04-23T16:18:42.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.021001393, 2024-04-23T16:18:42.441 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:18:42.441 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:18:42.441 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:18:42.441 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:18:42.441 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:18:42.441 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:18:42.441 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 124296 2024-04-23T16:18:42.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.441 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:18:42.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:18:42.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.409600+0000", 2024-04-23T16:18:42.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:18:42.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.409600+0000", 2024-04-23T16:18:42.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:18:42.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.409602+0000", 2024-04-23T16:18:42.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.606e-06 2024-04-23T16:18:42.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:18:42.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.409629+0000", 2024-04-23T16:18:42.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6910000000000002e-05 2024-04-23T16:18:42.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:18:42.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.409640+0000", 2024-04-23T16:18:42.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1590000000000001e-05 2024-04-23T16:18:42.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:18:42.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.409647+0000", 2024-04-23T16:18:42.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6320000000000002e-06 2024-04-23T16:18:42.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:18:42.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.409945+0000", 2024-04-23T16:18:42.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00029755200000000003 2024-04-23T16:18:42.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:18:42.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.410082+0000", 2024-04-23T16:18:42.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013753300000000001 2024-04-23T16:18:42.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:18:42.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.410218+0000", 2024-04-23T16:18:42.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013545699999999999 2024-04-23T16:18:42.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:18:42.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.418444+0000", 2024-04-23T16:18:42.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0082263630000000004 2024-04-23T16:18:42.445 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.445 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:18:42.445 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.445 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:124299 2.1 2:8f6388e0:::benchmark_data_smithi188_77627_object124298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e134)", 2024-04-23T16:18:42.445 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:42.413759+0000", 2024-04-23T16:18:42.445 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.016744892000000001, 2024-04-23T16:18:42.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016891795000000001, 2024-04-23T16:18:42.446 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:18:42.446 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:18:42.446 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:18:42.446 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:18:42.446 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:18:42.446 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:18:42.446 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 124299 2024-04-23T16:18:42.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.446 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:18:42.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:18:42.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.413759+0000", 2024-04-23T16:18:42.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:18:42.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.413759+0000", 2024-04-23T16:18:42.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:18:42.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.413761+0000", 2024-04-23T16:18:42.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6530000000000001e-06 2024-04-23T16:18:42.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:18:42.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.413793+0000", 2024-04-23T16:18:42.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1946000000000002e-05 2024-04-23T16:18:42.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:18:42.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.413804+0000", 2024-04-23T16:18:42.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0849999999999999e-05 2024-04-23T16:18:42.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:18:42.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.413812+0000", 2024-04-23T16:18:42.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4249999999999998e-06 2024-04-23T16:18:42.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:18:42.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.413865+0000", 2024-04-23T16:18:42.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3213e-05 2024-04-23T16:18:42.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:18:42.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414013+0000", 2024-04-23T16:18:42.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000147293 2024-04-23T16:18:42.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:18:42.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414168+0000", 2024-04-23T16:18:42.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000155406 2024-04-23T16:18:42.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:18:42.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.422319+0000", 2024-04-23T16:18:42.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.008151162 2024-04-23T16:18:42.450 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.450 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:18:42.450 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.450 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:124303 2.e 2:71af0785:::benchmark_data_smithi188_77627_object124302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e134)", 2024-04-23T16:18:42.450 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:42.414523+0000", 2024-04-23T16:18:42.450 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.01598134, 2024-04-23T16:18:42.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016176037000000001, 2024-04-23T16:18:42.450 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:18:42.450 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:18:42.451 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:18:42.451 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:18:42.451 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:18:42.451 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:18:42.451 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 124303 2024-04-23T16:18:42.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.451 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:18:42.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:18:42.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414523+0000", 2024-04-23T16:18:42.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:18:42.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414523+0000", 2024-04-23T16:18:42.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:18:42.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414524+0000", 2024-04-23T16:18:42.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1340000000000001e-06 2024-04-23T16:18:42.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:18:42.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414555+0000", 2024-04-23T16:18:42.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0818999999999997e-05 2024-04-23T16:18:42.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:18:42.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414566+0000", 2024-04-23T16:18:42.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0900999999999999e-05 2024-04-23T16:18:42.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:18:42.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414573+0000", 2024-04-23T16:18:42.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.844e-06 2024-04-23T16:18:42.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:18:42.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414810+0000", 2024-04-23T16:18:42.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000237338 2024-04-23T16:18:42.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:18:42.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414921+0000", 2024-04-23T16:18:42.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000111376 2024-04-23T16:18:42.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:18:42.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.415060+0000", 2024-04-23T16:18:42.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000138956 2024-04-23T16:18:42.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:18:42.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.418598+0000", 2024-04-23T16:18:42.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035380810000000002 2024-04-23T16:18:42.454 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.454 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:18:42.455 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.455 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:124304 2.1 2:80ca4fd4:::benchmark_data_smithi188_77627_object124303:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e134)", 2024-04-23T16:18:42.455 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:42.414582+0000", 2024-04-23T16:18:42.455 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.015922527999999998, 2024-04-23T16:18:42.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016163843000000001, 2024-04-23T16:18:42.455 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:18:42.455 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:18:42.455 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:18:42.455 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:18:42.455 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:18:42.455 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:18:42.455 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 124304 2024-04-23T16:18:42.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.456 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:18:42.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:18:42.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414582+0000", 2024-04-23T16:18:42.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:18:42.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414582+0000", 2024-04-23T16:18:42.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:18:42.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414582+0000", 2024-04-23T16:18:42.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7599999999999997e-07 2024-04-23T16:18:42.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:18:42.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414637+0000", 2024-04-23T16:18:42.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4212999999999997e-05 2024-04-23T16:18:42.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:18:42.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414648+0000", 2024-04-23T16:18:42.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1314999999999999e-05 2024-04-23T16:18:42.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:18:42.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414656+0000", 2024-04-23T16:18:42.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0260000000000007e-06 2024-04-23T16:18:42.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:18:42.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.418697+0000", 2024-04-23T16:18:42.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0040407380000000003 2024-04-23T16:18:42.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:18:42.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.418796+0000", 2024-04-23T16:18:42.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9358999999999995e-05 2024-04-23T16:18:42.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:18:42.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.418895+0000", 2024-04-23T16:18:42.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9141000000000005e-05 2024-04-23T16:18:42.459 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.459 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:18:42.459 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.459 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:124306 2.e 2:7cf4367f:::benchmark_data_smithi188_77627_object124305:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e134)", 2024-04-23T16:18:42.459 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:42.414685+0000", 2024-04-23T16:18:42.459 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.015819435, 2024-04-23T16:18:42.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01610321, 2024-04-23T16:18:42.459 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:18:42.459 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:18:42.459 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:18:42.459 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:18:42.459 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:18:42.460 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:18:42.460 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 124306 2024-04-23T16:18:42.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.460 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:18:42.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:18:42.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414685+0000", 2024-04-23T16:18:42.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:18:42.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414685+0000", 2024-04-23T16:18:42.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:18:42.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414686+0000", 2024-04-23T16:18:42.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.15e-06 2024-04-23T16:18:42.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:18:42.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414747+0000", 2024-04-23T16:18:42.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0939e-05 2024-04-23T16:18:42.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:18:42.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414754+0000", 2024-04-23T16:18:42.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.046e-06 2024-04-23T16:18:42.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:18:42.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.414760+0000", 2024-04-23T16:18:42.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.665e-06 2024-04-23T16:18:42.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:18:42.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.419501+0000", 2024-04-23T16:18:42.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047411789999999999 2024-04-23T16:18:42.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:18:42.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.419614+0000", 2024-04-23T16:18:42.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000113775 2024-04-23T16:18:42.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:18:42.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.419738+0000", 2024-04-23T16:18:42.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012323299999999999 2024-04-23T16:18:42.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:18:42.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.426862+0000", 2024-04-23T16:18:42.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071246800000000004 2024-04-23T16:18:42.463 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.463 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:18:42.463 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.463 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:124309 2.e 2:74f6829f:::benchmark_data_smithi188_77627_object124308:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e134)", 2024-04-23T16:18:42.463 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:42.422500+0000", 2024-04-23T16:18:42.463 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.0080037770000000001, 2024-04-23T16:18:42.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083358219999999997, 2024-04-23T16:18:42.464 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:18:42.464 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:18:42.464 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:18:42.464 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:18:42.464 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:18:42.464 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:18:42.464 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 124309 2024-04-23T16:18:42.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.464 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:18:42.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:18:42.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.422500+0000", 2024-04-23T16:18:42.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:18:42.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.422500+0000", 2024-04-23T16:18:42.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:18:42.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.422503+0000", 2024-04-23T16:18:42.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8650000000000001e-06 2024-04-23T16:18:42.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:18:42.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.422541+0000", 2024-04-23T16:18:42.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8143999999999999e-05 2024-04-23T16:18:42.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:18:42.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.422555+0000", 2024-04-23T16:18:42.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3584000000000001e-05 2024-04-23T16:18:42.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:18:42.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.422565+0000", 2024-04-23T16:18:42.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8320000000000001e-06 2024-04-23T16:18:42.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:18:42.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.423167+0000", 2024-04-23T16:18:42.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000602682 2024-04-23T16:18:42.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:18:42.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.423317+0000", 2024-04-23T16:18:42.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014995399999999999 2024-04-23T16:18:42.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:18:42.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.423494+0000", 2024-04-23T16:18:42.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000176078 2024-04-23T16:18:42.467 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.467 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:18:42.467 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.468 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:124312 2.e 2:70e7076c:::benchmark_data_smithi188_77627_object124311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e134)", 2024-04-23T16:18:42.468 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:42.422588+0000", 2024-04-23T16:18:42.468 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.0079165539999999993, 2024-04-23T16:18:42.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0082983129999999999, 2024-04-23T16:18:42.468 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:18:42.468 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:18:42.468 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:18:42.468 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:18:42.468 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:18:42.468 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:18:42.468 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 124312 2024-04-23T16:18:42.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.468 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:18:42.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:18:42.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.422588+0000", 2024-04-23T16:18:42.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:18:42.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.422588+0000", 2024-04-23T16:18:42.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:18:42.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.422588+0000", 2024-04-23T16:18:42.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7499999999999999e-07 2024-04-23T16:18:42.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:18:42.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.422619+0000", 2024-04-23T16:18:42.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0877000000000003e-05 2024-04-23T16:18:42.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:18:42.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.422628+0000", 2024-04-23T16:18:42.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7809999999999994e-06 2024-04-23T16:18:42.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:18:42.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.422634+0000", 2024-04-23T16:18:42.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2890000000000003e-06 2024-04-23T16:18:42.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:18:42.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.426904+0000", 2024-04-23T16:18:42.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042692219999999996 2024-04-23T16:18:42.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:18:42.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.427031+0000", 2024-04-23T16:18:42.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012758299999999999 2024-04-23T16:18:42.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:18:42.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.427169+0000", 2024-04-23T16:18:42.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013806100000000001 2024-04-23T16:18:42.472 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.472 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:18:42.472 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.472 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:124316 2.e 2.7214d2ce (undecoded) ondisk+write+known_if_redirected e134)", 2024-04-23T16:18:42.472 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:42.426973+0000", 2024-04-23T16:18:42.472 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.003530942, 2024-04-23T16:18:42.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039879779999999997, 2024-04-23T16:18:42.472 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:18:42.472 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:18:42.472 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "queued for pg", 2024-04-23T16:18:42.473 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:18:42.473 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:18:42.473 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:18:42.473 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 124316 2024-04-23T16:18:42.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.473 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:18:42.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:18:42.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.426973+0000", 2024-04-23T16:18:42.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:18:42.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.426973+0000", 2024-04-23T16:18:42.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:18:42.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.426975+0000", 2024-04-23T16:18:42.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.491e-06 2024-04-23T16:18:42.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:18:42.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.427015+0000", 2024-04-23T16:18:42.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0207000000000003e-05 2024-04-23T16:18:42.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:18:42.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.427023+0000", 2024-04-23T16:18:42.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9750000000000007e-06 2024-04-23T16:18:42.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:18:42.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.427030+0000", 2024-04-23T16:18:42.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9670000000000004e-06 2024-04-23T16:18:42.475 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.475 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:18:42.475 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.475 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:124320 2.1 2:8624e28d:::benchmark_data_smithi188_77627_object124319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e134)", 2024-04-23T16:18:42.475 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:42.427042+0000", 2024-04-23T16:18:42.476 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.0034624130000000001, 2024-04-23T16:18:42.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039659029999999998, 2024-04-23T16:18:42.476 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:18:42.476 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:18:42.476 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:18:42.476 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:18:42.476 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:18:42.476 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:18:42.476 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 124320 2024-04-23T16:18:42.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.476 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:18:42.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:18:42.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.427042+0000", 2024-04-23T16:18:42.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:18:42.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.427042+0000", 2024-04-23T16:18:42.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:18:42.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.427042+0000", 2024-04-23T16:18:42.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5499999999999998e-07 2024-04-23T16:18:42.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:18:42.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.427070+0000", 2024-04-23T16:18:42.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7898e-05 2024-04-23T16:18:42.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:18:42.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.427077+0000", 2024-04-23T16:18:42.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3380000000000001e-06 2024-04-23T16:18:42.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:18:42.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.427082+0000", 2024-04-23T16:18:42.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5199999999999997e-06 2024-04-23T16:18:42.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:18:42.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.430838+0000", 2024-04-23T16:18:42.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037556360000000001 2024-04-23T16:18:42.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:18:42.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.430955+0000", 2024-04-23T16:18:42.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117649 2024-04-23T16:18:42.479 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.479 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:18:42.479 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.479 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop_reply(client.14862.0:124296 2.e e134/110)", 2024-04-23T16:18:42.479 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:42.430896+0000", 2024-04-23T16:18:42.480 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4294967295.9996076, 2024-04-23T16:18:42.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000149631, 2024-04-23T16:18:42.480 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:18:42.480 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:18:42.480 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "queued for pg", 2024-04-23T16:18:42.480 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:18:42.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:18:42.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.430896+0000", 2024-04-23T16:18:42.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:18:42.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.430896+0000", 2024-04-23T16:18:42.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:18:42.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.430898+0000", 2024-04-23T16:18:42.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.606e-06 2024-04-23T16:18:42.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:18:42.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.430904+0000", 2024-04-23T16:18:42.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0070000000000003e-06 2024-04-23T16:18:42.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:18:42.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.430914+0000", 2024-04-23T16:18:42.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0382e-05 2024-04-23T16:18:42.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:18:42.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.430923+0000", 2024-04-23T16:18:42.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4009999999999997e-06 2024-04-23T16:18:42.482 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.482 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:18:42.482 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.482 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop_reply(client.14862.0:124299 2.1 e134/110)", 2024-04-23T16:18:42.482 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:42.430942+0000", 2024-04-23T16:18:42.482 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4294967295.9995627, 2024-04-23T16:18:42.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014649199999999999, 2024-04-23T16:18:42.483 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:18:42.483 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:18:42.483 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "queued for pg", 2024-04-23T16:18:42.483 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:18:42.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:18:42.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.430942+0000", 2024-04-23T16:18:42.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:18:42.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.430942+0000", 2024-04-23T16:18:42.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:18:42.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:18:42.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.430942+0000", 2024-04-23T16:18:42.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3000000000000001e-07 2024-04-23T16:18:42.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:18:42.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.430945+0000", 2024-04-23T16:18:42.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6819999999999999e-06 2024-04-23T16:18:42.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:18:42.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.430950+0000", 2024-04-23T16:18:42.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9799999999999998e-06 2024-04-23T16:18:42.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:18:42.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:18:42.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:18:42.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.430954+0000", 2024-04-23T16:18:42.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2300000000000002e-06 2024-04-23T16:18:42.485 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.485 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:18:42.485 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.485 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:18:42.485 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:18:42.485 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 12 2024-04-23T16:18:42.485 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:18:42.503 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:18:42.541 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:42.566 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:18:42.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:42.572 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:18:42.673 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:18:42.704 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:42.727 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:18:42.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:42.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:18:42.815 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 52 16 124911 124895 150.092 91.375 0.00762202 0.00666091 2024-04-23T16:18:42.823 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:42 smithi103 ceph-mon[39629]: pgmap v408: 17 pgs: 17 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-04-23T16:18:42.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:42 smithi188 ceph-mon[44101]: pgmap v408: 17 pgs: 17 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-04-23T16:18:42.831 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:18:42.870 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:42.894 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:18:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:42.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:18:42.999 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:18:43.030 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:42 smithi148 ceph-mon[44520]: pgmap v408: 17 pgs: 17 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-04-23T16:18:43.032 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:43.055 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:18:43.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:43.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:18:43.164 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:18:43.214 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:43.285 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:18:43.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:43.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:18:43.393 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:18:43.435 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:43.642 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:18:43.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:43.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:18:43.688 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-23T16:18:43.749 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:18:43.753 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:18:43.781 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:43.797 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:18:43.801 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:18:43.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:43.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:18:43.815 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 53 15 126396 126381 149.012 92.875 0.02001 0.0067086 2024-04-23T16:18:43.907 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:18:43.935 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:43.954 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:18:43.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:43.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:18:44.057 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:43 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:18:44.057 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:43 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:18:44.057 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:43 smithi103 ceph-mon[39629]: osdmap e135: 12 total, 12 up, 12 in 2024-04-23T16:18:44.057 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:43 smithi103 ceph-mon[39629]: osdmap e136: 12 total, 12 up, 12 in 2024-04-23T16:18:44.057 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:18:44.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:43 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:18:44.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:43 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:18:44.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:43 smithi148 ceph-mon[44520]: osdmap e135: 12 total, 12 up, 12 in 2024-04-23T16:18:44.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:43 smithi148 ceph-mon[44520]: osdmap e136: 12 total, 12 up, 12 in 2024-04-23T16:18:44.124 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:44.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:43 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:18:44.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:43 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:18:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:43 smithi188 ceph-mon[44101]: osdmap e135: 12 total, 12 up, 12 in 2024-04-23T16:18:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:43 smithi188 ceph-mon[44101]: osdmap e136: 12 total, 12 up, 12 in 2024-04-23T16:18:44.342 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:18:44.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:44.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:18:44.446 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:18:44.479 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:44.508 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:18:44.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:44.513 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:18:44.614 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:18:44.659 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:44.686 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:18:44.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:44.690 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:18:44.791 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:18:44.815 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 54 16 127840 127824 147.923 90.1875 0.0146117 0.00675896 2024-04-23T16:18:44.821 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:44.849 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:18:44.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:44.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:18:44.949 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:18:44.954 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:18:44.984 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:45.006 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:18:45.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:45.010 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:18:45.111 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:18:45.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:44 smithi148 ceph-mon[44520]: pgmap v411: 17 pgs: 17 active+clean; 7.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.43k op/s 2024-04-23T16:18:45.167 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:44 smithi103 ceph-mon[39629]: pgmap v411: 17 pgs: 17 active+clean; 7.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.43k op/s 2024-04-23T16:18:45.167 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:45.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:44 smithi188 ceph-mon[44101]: pgmap v411: 17 pgs: 17 active+clean; 7.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.43k op/s 2024-04-23T16:18:45.443 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:18:45.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:45.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:18:45.552 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:18:45.586 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:45.815 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 55 16 129303 129287 146.896 91.4375 0.00822253 0.00680592 2024-04-23T16:18:46.295 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:18:46.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:46.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:18:46.400 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:18:46.446 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:46.815 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 56 16 130788 130772 145.93 92.8125 0.0043245 0.00685105 2024-04-23T16:18:46.836 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:18:46.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:46.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:18:46.950 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:18:46.988 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:18:47.017 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:47.046 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:18:47.076 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:18:47.114 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:18:47.151 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:18:47.186 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:18:47.219 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:18:47.241 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:18:47.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:47.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:18:47.256 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:18:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:18:47.342 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:18:47.353 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:18:47.379 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:18:47.389 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:47.418 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:18:47.420 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:18:47.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:47.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:18:47.453 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:18:47.463 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:18:47.463 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:18:47.463 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:18:47.463 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:18:47.463 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:18:47.463 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:18:47.463 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:18:47.464 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:18:47.464 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:18:47.464 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:18:47.464 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:18:47.464 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:18:47.525 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:18:47.554 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:47 smithi103 ceph-mon[39629]: pgmap v412: 17 pgs: 17 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.65k op/s 2024-04-23T16:18:47.554 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:47 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2125249251' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:18:47.554 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:47.554 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:47.554 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:47.554 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:47.555 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:47.581 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:18:47.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:47.584 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:18:47.685 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:18:47.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:47 smithi188 ceph-mon[44101]: pgmap v412: 17 pgs: 17 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.65k op/s 2024-04-23T16:18:47.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:47 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2125249251' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:18:47.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:47.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:47.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:47.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:47.730 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:47.762 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:18:47.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:47.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:18:47.816 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 57 16 132118 132102 144.828 83.125 0.00480592 0.00690304 2024-04-23T16:18:47.867 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:18:47.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:47 smithi148 ceph-mon[44520]: pgmap v412: 17 pgs: 17 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.65k op/s 2024-04-23T16:18:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:47 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2125249251' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:18:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:47.900 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:47.919 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:18:47.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:47.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:18:48.023 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:18:48.068 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:48.235 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:18:48.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:48.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:18:48.346 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:18:48.375 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:48.400 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:18:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:48.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:18:48.507 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:48.516 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.517 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.517 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.517 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.517 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.517 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.517 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.517 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.537 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:48.559 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:48.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:48.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:48.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.664 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:48.694 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:48.721 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:48.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:48.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:48.816 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 58 16 133589 133573 143.916 91.9375 0.00517672 0.00694713 2024-04-23T16:18:48.825 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:48.854 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:48.879 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:48.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:48.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:48.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:18:48.983 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:18:49.036 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:49.110 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-04-23T16:18:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-04-23T16:18:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-04-23T16:18:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-04-23T16:18:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-04-23T16:18:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-04-23T16:18:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:bench [] [] [] [] 2024-04-23T16:18:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs debug_inject_read_zeros 2024-04-23T16:18:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs files list 2024-04-23T16:18:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs stats 2024-04-23T16:18:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:bluestore allocator dump block 2024-04-23T16:18:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:Error EINVAL: invalid command 2024-04-23T16:18:49.816 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 59 16 135047 135031 143.021 91.125 0.0128685 0.00699073 2024-04-23T16:18:49.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:49 smithi103 ceph-mon[39629]: pgmap v413: 17 pgs: 17 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-23T16:18:49.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:49 smithi148 ceph-mon[44520]: pgmap v413: 17 pgs: 17 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-23T16:18:49.937 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T16:18:49.938 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-23T16:18:49.938 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-04-23T16:18:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:49 smithi188 ceph-mon[44101]: pgmap v413: 17 pgs: 17 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-23T16:18:50.389 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:18:50.413 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:18:50.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:50.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:18:50.518 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:18:50.552 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:50.579 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:18:50.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:50.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:18:50.684 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:18:50.716 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:50.740 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:18:50.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:50.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:18:50.801 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:50 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:18:50.801 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:50 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:18:50.801 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:50 smithi103 ceph-mon[39629]: osdmap e137: 12 total, 12 up, 12 in 2024-04-23T16:18:50.816 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:18:50.816175 min lat: 0.000949989 max lat: 0.207898 avg lat: 0.00703615 2024-04-23T16:18:50.816 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:18:50.816 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 60 16 136446 136430 142.095 87.4375 0.00997872 0.00703615 2024-04-23T16:18:50.844 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:18:50.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:50 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:18:50.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:50 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:18:50.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:50 smithi148 ceph-mon[44520]: osdmap e137: 12 total, 12 up, 12 in 2024-04-23T16:18:50.879 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:50 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:18:50.879 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:50 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:18:50.879 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:50 smithi188 ceph-mon[44101]: osdmap e137: 12 total, 12 up, 12 in 2024-04-23T16:18:50.881 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:50.912 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:18:50.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:50.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:18:51.021 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:51.051 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:51.073 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:51.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:51.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:51.178 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:18:51.208 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:51.242 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:18:51.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:51.246 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:18:51.347 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:18:51.375 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:51.396 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:18:51.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:51.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:18:51.499 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:18:51.541 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:51.545 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:18:51.569 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:18:51.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:51.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:18:51.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:51 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:18:51.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:51 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:18:51.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:51 smithi148 ceph-mon[44520]: pgmap v415: 17 pgs: 17 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s 2024-04-23T16:18:51.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:51 smithi148 ceph-mon[44520]: osdmap e138: 12 total, 12 up, 12 in 2024-04-23T16:18:51.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:51 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2266771879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:18:51.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:51 smithi148 ceph-mon[44520]: from='client.34805 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:18:51.672 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:18:51.711 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:51.746 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:18:51.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:51.749 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:18:51.816 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 61 16 137793 137777 141.145 84.1875 0.00494879 0.00708385 2024-04-23T16:18:51.850 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:51.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:51 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:18:51.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:51 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:18:51.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:51 smithi103 ceph-mon[39629]: pgmap v415: 17 pgs: 17 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s 2024-04-23T16:18:51.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:51 smithi103 ceph-mon[39629]: osdmap e138: 12 total, 12 up, 12 in 2024-04-23T16:18:51.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:51 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2266771879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:18:51.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:51 smithi103 ceph-mon[39629]: from='client.34805 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:18:51.882 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:51.904 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:51.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:51.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:51.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:51 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:18:51.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:51 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:18:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:51 smithi188 ceph-mon[44101]: pgmap v415: 17 pgs: 17 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s 2024-04-23T16:18:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:51 smithi188 ceph-mon[44101]: osdmap e138: 12 total, 12 up, 12 in 2024-04-23T16:18:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:51 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2266771879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:18:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:51 smithi188 ceph-mon[44101]: from='client.34805 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:18:52.007 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:18:52.045 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:52.073 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:18:52.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:52.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:18:52.178 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:18:52.216 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:52.358 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:18:52.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:52.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:18:52.442 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:18:52.464 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:18:52.498 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:52.524 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:18:52.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:52.531 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:18:52.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:52 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:18:52.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:52 smithi103 ceph-mon[39629]: from='client.34805 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:18:52.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:52 smithi103 ceph-mon[39629]: osdmap e139: 12 total, 12 up, 12 in 2024-04-23T16:18:52.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:52 smithi103 ceph-mon[39629]: 2.d scrub starts 2024-04-23T16:18:52.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:52 smithi103 ceph-mon[39629]: 2.7 scrub starts 2024-04-23T16:18:52.632 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:18:52.686 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:52.714 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:18:52.715 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:18:52.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:52.722 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:18:52.816 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 62 16 138993 138977 140.078 75 0.00554854 0.00713738 2024-04-23T16:18:52.822 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:18:52.854 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:52 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:18:52.854 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:52 smithi148 ceph-mon[44520]: from='client.34805 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:18:52.854 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:52 smithi148 ceph-mon[44520]: osdmap e139: 12 total, 12 up, 12 in 2024-04-23T16:18:52.854 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:52 smithi148 ceph-mon[44520]: 2.d scrub starts 2024-04-23T16:18:52.854 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:52 smithi148 ceph-mon[44520]: 2.7 scrub starts 2024-04-23T16:18:52.856 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:52.878 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:18:52.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:52.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:18:52.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:52 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:18:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:52 smithi188 ceph-mon[44101]: from='client.34805 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:18:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:52 smithi188 ceph-mon[44101]: osdmap e139: 12 total, 12 up, 12 in 2024-04-23T16:18:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:52 smithi188 ceph-mon[44101]: 2.d scrub starts 2024-04-23T16:18:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:52 smithi188 ceph-mon[44101]: 2.7 scrub starts 2024-04-23T16:18:52.981 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:18:53.023 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:53.802 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:18:53.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:53.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:18:53.814 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:53 smithi103 ceph-mon[39629]: pgmap v418: 17 pgs: 17 active+clean; 8.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-04-23T16:18:53.815 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:53 smithi103 ceph-mon[39629]: 2.7 scrub ok 2024-04-23T16:18:53.816 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 63 16 140282 140266 139.134 80.5625 0.00939781 0.00718449 2024-04-23T16:18:53.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:53 smithi148 ceph-mon[44520]: pgmap v418: 17 pgs: 17 active+clean; 8.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-04-23T16:18:53.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:53 smithi148 ceph-mon[44520]: 2.7 scrub ok 2024-04-23T16:18:53.915 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:18:53.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:53 smithi188 ceph-mon[44101]: pgmap v418: 17 pgs: 17 active+clean; 8.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-04-23T16:18:53.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:53 smithi188 ceph-mon[44101]: 2.7 scrub ok 2024-04-23T16:18:53.963 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:53.988 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:18:53.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:53.991 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:18:54.091 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:54.119 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:54.141 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:54.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:54.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:54.246 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:54.274 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:54.293 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:54.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:54.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:54.392 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:18:54.396 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:18:54.424 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:54.446 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:18:54.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:54.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:18:54.551 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:18:54.559 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:54 smithi103 ceph-mon[39629]: 2.d scrub ok 2024-04-23T16:18:54.559 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:54 smithi103 ceph-mon[39629]: 2.1 scrub starts 2024-04-23T16:18:54.559 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:54 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:18:54.559 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:54 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:18:54.559 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:54 smithi103 ceph-mon[39629]: osdmap e140: 12 total, 12 up, 12 in 2024-04-23T16:18:54.579 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:54.595 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:18:54.599 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:18:54.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:54.603 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:18:54.643 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:18:54.643 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:18:54.704 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:18:54.730 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:54 smithi148 ceph-mon[44520]: 2.d scrub ok 2024-04-23T16:18:54.730 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:54 smithi148 ceph-mon[44520]: 2.1 scrub starts 2024-04-23T16:18:54.730 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:54 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:18:54.730 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:54 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:18:54.730 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:54 smithi148 ceph-mon[44520]: osdmap e140: 12 total, 12 up, 12 in 2024-04-23T16:18:54.732 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:54.753 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:18:54.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:54.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:18:54.816 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 64 16 141427 141411 138.078 71.5625 0.0120922 0.00724072 2024-04-23T16:18:54.857 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:18:54.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:54 smithi188 ceph-mon[44101]: 2.d scrub ok 2024-04-23T16:18:54.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:54 smithi188 ceph-mon[44101]: 2.1 scrub starts 2024-04-23T16:18:54.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:54 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:18:54.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:54 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:18:54.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:54 smithi188 ceph-mon[44101]: osdmap e140: 12 total, 12 up, 12 in 2024-04-23T16:18:54.889 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:54.912 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:18:54.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:54.918 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:18:55.019 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:18:55.087 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:55.162 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:18:55.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:55.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:18:55.274 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:18:55.370 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:55.408 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:18:55.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:55.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:18:55.520 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:18:55.527 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-04-23T16:18:55.527 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-04-23T16:18:55.527 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-04-23T16:18:55.527 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-04-23T16:18:55.527 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-04-23T16:18:55.527 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-04-23T16:18:55.527 INFO:teuthology.orchestra.run.smithi103.stderr:bench [] [] [] [] 2024-04-23T16:18:55.527 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs debug_inject_read_zeros 2024-04-23T16:18:55.528 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs files list 2024-04-23T16:18:55.528 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs stats 2024-04-23T16:18:55.528 INFO:teuthology.orchestra.run.smithi103.stderr:bluestore allocator dump block 2024-04-23T16:18:55.528 INFO:teuthology.orchestra.run.smithi103.stderr:Error EINVAL: invalid command 2024-04-23T16:18:55.563 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:55.595 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:18:55.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:55.599 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:18:55.700 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:18:55.729 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:55.752 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:18:55.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:55.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:18:55.816 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 65 16 142648 142632 137.128 76.3125 0.00653897 0.00729139 2024-04-23T16:18:55.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:55 smithi103 ceph-mon[39629]: 2.e scrub starts 2024-04-23T16:18:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:55 smithi103 ceph-mon[39629]: pgmap v420: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-23T16:18:55.856 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:18:55.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:55 smithi148 ceph-mon[44520]: 2.e scrub starts 2024-04-23T16:18:55.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:55 smithi148 ceph-mon[44520]: pgmap v420: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-23T16:18:55.921 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:55.941 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-23T16:18:55.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:55 smithi188 ceph-mon[44101]: 2.e scrub starts 2024-04-23T16:18:55.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:55 smithi188 ceph-mon[44101]: pgmap v420: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-23T16:18:56.131 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:18:56.411 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:18:56.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:56.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:18:56.519 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:18:56.552 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:56.584 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:18:56.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:56.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:18:56.659 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:56 smithi103 ceph-mon[39629]: 2.1 scrub ok 2024-04-23T16:18:56.687 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:56.716 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:56.741 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:56.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:56.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:56.816 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 66 16 143892 143876 136.228 77.75 0.00627378 0.00733858 2024-04-23T16:18:56.847 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:18:56.872 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:56 smithi148 ceph-mon[44520]: 2.1 scrub ok 2024-04-23T16:18:56.875 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:56.896 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:18:56.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:56.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:18:56.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:56 smithi188 ceph-mon[44101]: 2.1 scrub ok 2024-04-23T16:18:57.000 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:18:57.046 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:57.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:57 smithi103 ceph-mon[39629]: 2.e scrub ok 2024-04-23T16:18:57.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:57 smithi103 ceph-mon[39629]: pgmap v421: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-23T16:18:57.816 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 67 16 145230 145214 135.443 83.625 0.0043997 0.00738175 2024-04-23T16:18:57.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:57 smithi148 ceph-mon[44520]: 2.e scrub ok 2024-04-23T16:18:57.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:57 smithi148 ceph-mon[44520]: pgmap v421: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-23T16:18:57.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:57 smithi188 ceph-mon[44101]: 2.e scrub ok 2024-04-23T16:18:57.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:57 smithi188 ceph-mon[44101]: pgmap v421: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-23T16:18:58.416 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:18:58.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:58.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:18:58.438 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:18:58.527 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:18:58.553 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:58.574 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:18:58.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:58.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:18:58.678 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:18:58.713 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:58.739 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:18:58.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:58.744 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:18:58.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 68 16 146521 146505 134.637 80.6875 0.0140624 0.007426 2024-04-23T16:18:58.845 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:18:58.893 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:59.057 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:18:59.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:59.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:18:59.162 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:18:59.191 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:59.211 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:18:59.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:59.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:18:59.316 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:18:59.346 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:59.368 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:18:59.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:59.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:18:59.405 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:18:59.437 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:18:59.472 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:18:59.496 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:18:59.519 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:59.532 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:18:59.544 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:18:59.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:59.548 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:18:59.572 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:18:59.605 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:18:59.646 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:18:59.649 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:18:59.683 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:59.693 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:18:59.724 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:18:59.724 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:18:59.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:59.729 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:18:59.758 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:18:59.799 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:18:59.816 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 69 16 147584 147568 133.649 66.4375 0.0312388 0.00748036 2024-04-23T16:18:59.830 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:18:59.837 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:18:59.844 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:18:59.844 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:18:59.844 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:18:59.844 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:18:59.844 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:18:59.844 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:18:59.844 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:18:59.844 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:18:59.844 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:18:59.844 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:18:59.845 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:18:59.845 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:18:59.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:59 smithi103 ceph-mon[39629]: 2.4 scrub starts 2024-04-23T16:18:59.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:59 smithi103 ceph-mon[39629]: pgmap v422: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-23T16:18:59.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:59 smithi103 ceph-mon[39629]: 2.5 scrub starts 2024-04-23T16:18:59.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:18:59 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/229583339' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:18:59.858 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:59 smithi148 ceph-mon[44520]: 2.4 scrub starts 2024-04-23T16:18:59.858 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:59 smithi148 ceph-mon[44520]: pgmap v422: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-23T16:18:59.858 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:59 smithi148 ceph-mon[44520]: 2.5 scrub starts 2024-04-23T16:18:59.858 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:18:59 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/229583339' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:18:59.860 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:18:59.883 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:18:59.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:18:59.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:18:59.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:59 smithi188 ceph-mon[44101]: 2.4 scrub starts 2024-04-23T16:18:59.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:59 smithi188 ceph-mon[44101]: pgmap v422: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-23T16:18:59.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:59 smithi188 ceph-mon[44101]: 2.5 scrub starts 2024-04-23T16:18:59.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:18:59 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/229583339' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:18:59.987 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:19:00.025 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:00.175 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T16:19:00.175 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-23T16:19:00.175 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.8 'trim stale osdmaps' 2024-04-23T16:19:00.605 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:19:00.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 70 16 148574 148558 132.624 61.875 0.0235812 0.00753903 2024-04-23T16:19:00.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:00 smithi103 ceph-mon[39629]: 2.a scrub starts 2024-04-23T16:19:00.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:00 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2903087947' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:19:00.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:00 smithi103 ceph-mon[39629]: 2.f scrub starts 2024-04-23T16:19:00.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:00 smithi148 ceph-mon[44520]: 2.a scrub starts 2024-04-23T16:19:00.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:00 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2903087947' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:19:00.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:00 smithi148 ceph-mon[44520]: 2.f scrub starts 2024-04-23T16:19:00.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:00 smithi188 ceph-mon[44101]: 2.a scrub starts 2024-04-23T16:19:00.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:00.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:00 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2903087947' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:19:00.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:00 smithi188 ceph-mon[44101]: 2.f scrub starts 2024-04-23T16:19:00.988 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:19:00.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:00.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:19:01.098 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:19:01.130 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:01.157 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:19:01.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:01.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:19:01.265 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:19:01.294 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:01.319 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:19:01.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:01.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:19:01.426 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:19:01.477 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:01.653 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:01 smithi103 ceph-mon[39629]: 2.4 scrub ok 2024-04-23T16:19:01.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:01 smithi103 ceph-mon[39629]: 2.8 scrub starts 2024-04-23T16:19:01.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:01 smithi103 ceph-mon[39629]: pgmap v423: 17 pgs: 17 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-04-23T16:19:01.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:01 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:19:01.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:01 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:19:01.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:01 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2903087947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:19:01.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:01 smithi103 ceph-mon[39629]: osdmap e141: 12 total, 12 up, 12 in 2024-04-23T16:19:01.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:01 smithi103 ceph-mon[39629]: 2.5 scrub ok 2024-04-23T16:19:01.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:01 smithi103 ceph-mon[39629]: 2.a scrub ok 2024-04-23T16:19:01.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 71 16 149690 149674 131.738 69.75 0.00454851 0.00758874 2024-04-23T16:19:01.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:01 smithi148 ceph-mon[44520]: 2.4 scrub ok 2024-04-23T16:19:01.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:01 smithi148 ceph-mon[44520]: 2.8 scrub starts 2024-04-23T16:19:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:01 smithi148 ceph-mon[44520]: pgmap v423: 17 pgs: 17 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-04-23T16:19:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:01 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:19:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:01 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:19:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:01 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2903087947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:19:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:01 smithi148 ceph-mon[44520]: osdmap e141: 12 total, 12 up, 12 in 2024-04-23T16:19:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:01 smithi148 ceph-mon[44520]: 2.5 scrub ok 2024-04-23T16:19:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:01 smithi148 ceph-mon[44520]: 2.a scrub ok 2024-04-23T16:19:01.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:01 smithi188 ceph-mon[44101]: 2.4 scrub ok 2024-04-23T16:19:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:01 smithi188 ceph-mon[44101]: 2.8 scrub starts 2024-04-23T16:19:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:01 smithi188 ceph-mon[44101]: pgmap v423: 17 pgs: 17 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-04-23T16:19:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:01 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:19:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:01 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:19:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:01 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2903087947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:19:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:01 smithi188 ceph-mon[44101]: osdmap e141: 12 total, 12 up, 12 in 2024-04-23T16:19:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:01 smithi188 ceph-mon[44101]: 2.5 scrub ok 2024-04-23T16:19:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:01 smithi188 ceph-mon[44101]: 2.a scrub ok 2024-04-23T16:19:02.053 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:19:02.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:02.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:19:02.160 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:19:02.206 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:02.241 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:19:02.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:02.249 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:19:02.350 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:19:02.395 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:02.603 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:19:02.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:02.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:19:02.708 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:19:02.738 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:02.762 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:19:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:02.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:19:02.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 72 16 150791 150775 130.864 68.8125 0.0048858 0.00763992 2024-04-23T16:19:02.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:02 smithi103 ceph-mon[39629]: 2.2 deep-scrub starts 2024-04-23T16:19:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:02 smithi103 ceph-mon[39629]: 2.8 scrub ok 2024-04-23T16:19:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:02 smithi103 ceph-mon[39629]: 2.f scrub ok 2024-04-23T16:19:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:02 smithi103 ceph-mon[39629]: 2.b scrub starts 2024-04-23T16:19:02.865 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:19:02.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:02 smithi148 ceph-mon[44520]: 2.2 deep-scrub starts 2024-04-23T16:19:02.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:02 smithi148 ceph-mon[44520]: 2.8 scrub ok 2024-04-23T16:19:02.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:02 smithi148 ceph-mon[44520]: 2.f scrub ok 2024-04-23T16:19:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:02 smithi148 ceph-mon[44520]: 2.b scrub starts 2024-04-23T16:19:02.912 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:02.940 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:19:02.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:02.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:19:02.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:02 smithi188 ceph-mon[44101]: 2.2 deep-scrub starts 2024-04-23T16:19:02.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:02 smithi188 ceph-mon[44101]: 2.8 scrub ok 2024-04-23T16:19:02.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:02 smithi188 ceph-mon[44101]: 2.f scrub ok 2024-04-23T16:19:02.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:02 smithi188 ceph-mon[44101]: 2.b scrub starts 2024-04-23T16:19:03.011 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:19:03.052 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:19:03.093 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:03.122 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:19:03.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:03.128 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:19:03.229 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:19:03.266 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:03.317 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:19:03.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:03.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:19:03.399 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:19:03.426 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:19:03.461 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:03.492 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:19:03.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:03.497 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:19:03.598 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:19:03.631 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:03.655 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:19:03.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:03.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:19:03.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:03 smithi188 ceph-mon[44101]: pgmap v425: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-23T16:19:03.759 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:19:03.807 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:03.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 73 16 151818 151802 129.951 64.1875 0.0088044 0.00769374 2024-04-23T16:19:03.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:03 smithi148 ceph-mon[44520]: pgmap v425: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-23T16:19:04.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:03 smithi103 ceph-mon[39629]: pgmap v425: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-23T16:19:04.532 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:19:04.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 74 16 152959 152943 129.158 71.3125 0.00547675 0.00774025 2024-04-23T16:19:04.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:04 smithi103 ceph-mon[39629]: 2.b scrub ok 2024-04-23T16:19:04.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:04 smithi148 ceph-mon[44520]: 2.b scrub ok 2024-04-23T16:19:04.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:04 smithi188 ceph-mon[44101]: 2.b scrub ok 2024-04-23T16:19:05.469 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:19:05.469 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:19:05.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:19:05.469 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:19:05.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.470 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:110908 2.1 2:8d5e4d45:::benchmark_data_smithi188_77627_object110907:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e129)", 2024-04-23T16:19:05.470 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:33.782538+0000", 2024-04-23T16:19:05.470 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.205026862, 2024-04-23T16:19:05.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.05294343, 2024-04-23T16:19:05.470 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.470 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.470 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:05.470 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:19:05.470 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:19:05.470 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:19:05.470 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110908 2024-04-23T16:19:05.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.471 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:33.782538+0000", 2024-04-23T16:19:05.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:33.782538+0000", 2024-04-23T16:19:05.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:33.782539+0000", 2024-04-23T16:19:05.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.066e-06 2024-04-23T16:19:05.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:33.782602+0000", 2024-04-23T16:19:05.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2978999999999995e-05 2024-04-23T16:19:05.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:33.782609+0000", 2024-04-23T16:19:05.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7279999999999998e-06 2024-04-23T16:19:05.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:33.782613+0000", 2024-04-23T16:19:05.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6949999999999999e-06 2024-04-23T16:19:05.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:33.782641+0000", 2024-04-23T16:19:05.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7168000000000001e-05 2024-04-23T16:19:05.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:33.782712+0000", 2024-04-23T16:19:05.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1693999999999995e-05 2024-04-23T16:19:05.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:19:05.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:33.828735+0000", 2024-04-23T16:19:05.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046023080000000001 2024-04-23T16:19:05.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:19:05.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:33.833299+0000", 2024-04-23T16:19:05.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004564004 2024-04-23T16:19:05.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:19:05.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:33.835374+0000", 2024-04-23T16:19:05.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020743010000000002 2024-04-23T16:19:05.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:05.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:33.835431+0000", 2024-04-23T16:19:05.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7246e-05 2024-04-23T16:19:05.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:33.835481+0000", 2024-04-23T16:19:05.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0469000000000001e-05 2024-04-23T16:19:05.475 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.475 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.475 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.475 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:111287 2.3 e130/110 2:c1312921:::benchmark_data_smithi188_77627_object111286:head v 130'6933, mlcod=130'6930)", 2024-04-23T16:19:05.476 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:34.045153+0000", 2024-04-23T16:19:05.476 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 30.942411636999999, 2024-04-23T16:19:05.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.065976758999999996, 2024-04-23T16:19:05.476 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.476 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.476 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:05.476 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.045153+0000", 2024-04-23T16:19:05.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.045153+0000", 2024-04-23T16:19:05.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.045154+0000", 2024-04-23T16:19:05.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1400000000000001e-06 2024-04-23T16:19:05.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.045217+0000", 2024-04-23T16:19:05.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2706000000000006e-05 2024-04-23T16:19:05.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.045224+0000", 2024-04-23T16:19:05.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9210000000000003e-06 2024-04-23T16:19:05.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.045229+0000", 2024-04-23T16:19:05.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7210000000000002e-06 2024-04-23T16:19:05.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.045665+0000", 2024-04-23T16:19:05.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043620399999999999 2024-04-23T16:19:05.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.045681+0000", 2024-04-23T16:19:05.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6554000000000001e-05 2024-04-23T16:19:05.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:05.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.111091+0000", 2024-04-23T16:19:05.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.065409321000000006 2024-04-23T16:19:05.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.111130+0000", 2024-04-23T16:19:05.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9192e-05 2024-04-23T16:19:05.480 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.480 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.480 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.480 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:111299 2.3 e130/110 2:cf529538:::benchmark_data_smithi188_77627_object111298:head v 130'6934, mlcod=130'6931)", 2024-04-23T16:19:05.480 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:34.055321+0000", 2024-04-23T16:19:05.480 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 30.932243708000001, 2024-04-23T16:19:05.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.055843206999999999, 2024-04-23T16:19:05.480 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.480 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.480 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:05.480 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.055321+0000", 2024-04-23T16:19:05.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.055321+0000", 2024-04-23T16:19:05.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.055322+0000", 2024-04-23T16:19:05.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3650000000000001e-06 2024-04-23T16:19:05.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.055357+0000", 2024-04-23T16:19:05.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4952e-05 2024-04-23T16:19:05.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.055366+0000", 2024-04-23T16:19:05.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3340000000000002e-06 2024-04-23T16:19:05.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.055372+0000", 2024-04-23T16:19:05.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3389999999999998e-06 2024-04-23T16:19:05.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.056516+0000", 2024-04-23T16:19:05.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0011442100000000001 2024-04-23T16:19:05.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.056535+0000", 2024-04-23T16:19:05.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8406e-05 2024-04-23T16:19:05.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:05.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.111140+0000", 2024-04-23T16:19:05.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.054605057999999998 2024-04-23T16:19:05.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.111164+0000", 2024-04-23T16:19:05.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4542999999999999e-05 2024-04-23T16:19:05.484 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.484 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.484 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.484 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:111304 2.e 2:7d9ee220:::benchmark_data_smithi188_77627_object111303:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-04-23T16:19:05.484 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:34.057901+0000", 2024-04-23T16:19:05.484 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 30.929664284000001, 2024-04-23T16:19:05.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.053803544000000002, 2024-04-23T16:19:05.484 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.485 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.485 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:05.485 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:19:05.485 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:19:05.485 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:19:05.485 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111304 2024-04-23T16:19:05.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.485 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.057901+0000", 2024-04-23T16:19:05.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.057901+0000", 2024-04-23T16:19:05.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.057902+0000", 2024-04-23T16:19:05.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7880000000000001e-06 2024-04-23T16:19:05.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058107+0000", 2024-04-23T16:19:05.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020509 2024-04-23T16:19:05.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058118+0000", 2024-04-23T16:19:05.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0206e-05 2024-04-23T16:19:05.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058125+0000", 2024-04-23T16:19:05.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2599999999999999e-06 2024-04-23T16:19:05.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058277+0000", 2024-04-23T16:19:05.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000151688 2024-04-23T16:19:05.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058381+0000", 2024-04-23T16:19:05.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010408600000000001 2024-04-23T16:19:05.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:19:05.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.103797+0000", 2024-04-23T16:19:05.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045416758000000002 2024-04-23T16:19:05.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:19:05.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.110376+0000", 2024-04-23T16:19:05.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0065782269999999999 2024-04-23T16:19:05.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:19:05.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.111602+0000", 2024-04-23T16:19:05.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001226771 2024-04-23T16:19:05.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:05.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.111661+0000", 2024-04-23T16:19:05.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8192000000000001e-05 2024-04-23T16:19:05.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.111704+0000", 2024-04-23T16:19:05.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3477999999999999e-05 2024-04-23T16:19:05.490 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.490 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.490 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.490 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:111305 2.1 2:8c0ac4c2:::benchmark_data_smithi188_77627_object111304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-04-23T16:19:05.490 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:34.058150+0000", 2024-04-23T16:19:05.490 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 30.929415031000001, 2024-04-23T16:19:05.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.058512308999999998, 2024-04-23T16:19:05.490 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.490 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.490 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:05.491 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:19:05.491 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:19:05.491 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:19:05.491 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111305 2024-04-23T16:19:05.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.491 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058150+0000", 2024-04-23T16:19:05.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058150+0000", 2024-04-23T16:19:05.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058151+0000", 2024-04-23T16:19:05.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6799999999999999e-07 2024-04-23T16:19:05.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058361+0000", 2024-04-23T16:19:05.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021016200000000001 2024-04-23T16:19:05.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058369+0000", 2024-04-23T16:19:05.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7789999999999996e-06 2024-04-23T16:19:05.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058375+0000", 2024-04-23T16:19:05.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.438e-06 2024-04-23T16:19:05.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058417+0000", 2024-04-23T16:19:05.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2456000000000001e-05 2024-04-23T16:19:05.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.103743+0000", 2024-04-23T16:19:05.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045325229000000002 2024-04-23T16:19:05.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:19:05.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.103940+0000", 2024-04-23T16:19:05.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000197715 2024-04-23T16:19:05.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:19:05.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.110315+0000", 2024-04-23T16:19:05.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0063742800000000004 2024-04-23T16:19:05.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:19:05.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.116571+0000", 2024-04-23T16:19:05.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062562659999999999 2024-04-23T16:19:05.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:05.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.116605+0000", 2024-04-23T16:19:05.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4338999999999998e-05 2024-04-23T16:19:05.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.116662+0000", 2024-04-23T16:19:05.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6876999999999998e-05 2024-04-23T16:19:05.495 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.495 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.495 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.495 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:111308 2.1 2:8fec7474:::benchmark_data_smithi188_77627_object111307:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-04-23T16:19:05.495 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:34.058387+0000", 2024-04-23T16:19:05.496 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 30.929177787, 2024-04-23T16:19:05.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.058429353000000003, 2024-04-23T16:19:05.496 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.496 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.496 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:05.496 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:19:05.496 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:19:05.496 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:19:05.496 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111308 2024-04-23T16:19:05.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.496 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058387+0000", 2024-04-23T16:19:05.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058387+0000", 2024-04-23T16:19:05.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058388+0000", 2024-04-23T16:19:05.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.54e-07 2024-04-23T16:19:05.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058691+0000", 2024-04-23T16:19:05.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000303403 2024-04-23T16:19:05.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.103546+0000", 2024-04-23T16:19:05.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.044854484999999999 2024-04-23T16:19:05.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.103561+0000", 2024-04-23T16:19:05.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5118e-05 2024-04-23T16:19:05.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.106994+0000", 2024-04-23T16:19:05.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034329320000000001 2024-04-23T16:19:05.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.107108+0000", 2024-04-23T16:19:05.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011389500000000001 2024-04-23T16:19:05.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:19:05.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.107255+0000", 2024-04-23T16:19:05.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014715799999999999 2024-04-23T16:19:05.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:19:05.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.116678+0000", 2024-04-23T16:19:05.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094229199999999996 2024-04-23T16:19:05.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:19:05.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.116720+0000", 2024-04-23T16:19:05.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1953999999999998e-05 2024-04-23T16:19:05.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:05.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.116771+0000", 2024-04-23T16:19:05.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1246999999999999e-05 2024-04-23T16:19:05.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.116816+0000", 2024-04-23T16:19:05.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5487e-05 2024-04-23T16:19:05.501 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.501 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.501 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.501 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:111307 2.3 e130/110 2:ca7620eb:::benchmark_data_smithi188_77627_object111306:head v 130'6935, mlcod=130'6931)", 2024-04-23T16:19:05.501 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:34.058730+0000", 2024-04-23T16:19:05.501 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 30.928834508000001, 2024-04-23T16:19:05.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.052454589000000003, 2024-04-23T16:19:05.502 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.502 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.502 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:05.502 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058730+0000", 2024-04-23T16:19:05.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058730+0000", 2024-04-23T16:19:05.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058732+0000", 2024-04-23T16:19:05.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3209999999999999e-06 2024-04-23T16:19:05.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058785+0000", 2024-04-23T16:19:05.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3758999999999998e-05 2024-04-23T16:19:05.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.103556+0000", 2024-04-23T16:19:05.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.044770681 2024-04-23T16:19:05.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.103566+0000", 2024-04-23T16:19:05.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9010000000000006e-06 2024-04-23T16:19:05.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.103806+0000", 2024-04-23T16:19:05.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000239595 2024-04-23T16:19:05.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.103827+0000", 2024-04-23T16:19:05.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1382999999999998e-05 2024-04-23T16:19:05.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:05.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.111166+0000", 2024-04-23T16:19:05.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007339328 2024-04-23T16:19:05.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.111185+0000", 2024-04-23T16:19:05.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8621e-05 2024-04-23T16:19:05.505 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.505 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.505 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.506 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:114411 2.1 2:814709c2:::benchmark_data_smithi188_77627_object114410:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-23T16:19:05.506 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:35.939257+0000", 2024-04-23T16:19:05.506 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 29.048308143, 2024-04-23T16:19:05.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.058078545000000002, 2024-04-23T16:19:05.506 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.506 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.506 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:05.506 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:19:05.506 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:19:05.506 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:19:05.506 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114411 2024-04-23T16:19:05.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.507 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.939257+0000", 2024-04-23T16:19:05.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.939257+0000", 2024-04-23T16:19:05.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.939258+0000", 2024-04-23T16:19:05.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6750000000000001e-06 2024-04-23T16:19:05.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.939311+0000", 2024-04-23T16:19:05.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2608999999999997e-05 2024-04-23T16:19:05.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.939319+0000", 2024-04-23T16:19:05.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0949999999999996e-06 2024-04-23T16:19:05.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.939325+0000", 2024-04-23T16:19:05.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4879999999999998e-06 2024-04-23T16:19:05.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.939362+0000", 2024-04-23T16:19:05.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7787000000000003e-05 2024-04-23T16:19:05.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.939445+0000", 2024-04-23T16:19:05.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3014000000000002e-05 2024-04-23T16:19:05.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:19:05.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.939556+0000", 2024-04-23T16:19:05.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110722 2024-04-23T16:19:05.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:19:05.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.984646+0000", 2024-04-23T16:19:05.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045089865999999999 2024-04-23T16:19:05.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:19:05.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.997243+0000", 2024-04-23T16:19:05.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012597215 2024-04-23T16:19:05.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:05.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.997286+0000", 2024-04-23T16:19:05.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2883999999999998e-05 2024-04-23T16:19:05.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.997335+0000", 2024-04-23T16:19:05.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9190000000000002e-05 2024-04-23T16:19:05.511 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.511 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.511 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.511 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:114415 2.1 2:8b5e5c4a:::benchmark_data_smithi188_77627_object114414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-23T16:19:05.511 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:35.940072+0000", 2024-04-23T16:19:05.512 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 29.047492632000001, 2024-04-23T16:19:05.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.057339935000000002, 2024-04-23T16:19:05.512 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.512 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.512 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:05.512 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:19:05.512 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:19:05.512 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:19:05.512 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114415 2024-04-23T16:19:05.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.512 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.940072+0000", 2024-04-23T16:19:05.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.940072+0000", 2024-04-23T16:19:05.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.940073+0000", 2024-04-23T16:19:05.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1399999999999996e-07 2024-04-23T16:19:05.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.940139+0000", 2024-04-23T16:19:05.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5727000000000003e-05 2024-04-23T16:19:05.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.940143+0000", 2024-04-23T16:19:05.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3889999999999998e-06 2024-04-23T16:19:05.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.940147+0000", 2024-04-23T16:19:05.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.642e-06 2024-04-23T16:19:05.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.942603+0000", 2024-04-23T16:19:05.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024561269999999998 2024-04-23T16:19:05.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.942691+0000", 2024-04-23T16:19:05.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8257999999999994e-05 2024-04-23T16:19:05.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:19:05.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.942793+0000", 2024-04-23T16:19:05.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010204600000000001 2024-04-23T16:19:05.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:19:05.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.984689+0000", 2024-04-23T16:19:05.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.041895296999999998 2024-04-23T16:19:05.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:19:05.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.997345+0000", 2024-04-23T16:19:05.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012656598999999999 2024-04-23T16:19:05.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:05.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.997379+0000", 2024-04-23T16:19:05.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3908000000000001e-05 2024-04-23T16:19:05.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.997412+0000", 2024-04-23T16:19:05.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3127999999999999e-05 2024-04-23T16:19:05.517 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.517 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.517 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.517 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:114416 2.1 2:82210ae6:::benchmark_data_smithi188_77627_object114415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-23T16:19:05.517 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:35.941462+0000", 2024-04-23T16:19:05.517 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 29.046103083999999, 2024-04-23T16:19:05.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.056040627000000003, 2024-04-23T16:19:05.517 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.517 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.517 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:05.518 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:19:05.518 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:19:05.518 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:19:05.518 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114416 2024-04-23T16:19:05.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.518 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.941462+0000", 2024-04-23T16:19:05.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.941462+0000", 2024-04-23T16:19:05.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.941463+0000", 2024-04-23T16:19:05.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5990000000000001e-06 2024-04-23T16:19:05.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.941511+0000", 2024-04-23T16:19:05.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7491000000000003e-05 2024-04-23T16:19:05.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.941521+0000", 2024-04-23T16:19:05.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0499000000000001e-05 2024-04-23T16:19:05.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.941529+0000", 2024-04-23T16:19:05.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3529999999999997e-06 2024-04-23T16:19:05.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.984719+0000", 2024-04-23T16:19:05.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.043189981000000002 2024-04-23T16:19:05.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.984859+0000", 2024-04-23T16:19:05.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000140598 2024-04-23T16:19:05.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:19:05.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.985004+0000", 2024-04-23T16:19:05.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000144306 2024-04-23T16:19:05.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:19:05.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.997115+0000", 2024-04-23T16:19:05.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012111555 2024-04-23T16:19:05.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:19:05.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.997421+0000", 2024-04-23T16:19:05.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00030607799999999999 2024-04-23T16:19:05.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:05.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.997472+0000", 2024-04-23T16:19:05.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0467999999999999e-05 2024-04-23T16:19:05.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.997502+0000", 2024-04-23T16:19:05.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0698999999999999e-05 2024-04-23T16:19:05.522 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.523 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.523 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.523 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:116790 2.e 2:7e8b970c:::benchmark_data_smithi188_77627_object116789:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-23T16:19:05.523 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.402019+0000", 2024-04-23T16:19:05.523 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.585546218000001, 2024-04-23T16:19:05.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.100066103, 2024-04-23T16:19:05.523 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.523 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.523 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:05.523 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:19:05.523 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:19:05.524 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:19:05.524 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116790 2024-04-23T16:19:05.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.524 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402019+0000", 2024-04-23T16:19:05.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402019+0000", 2024-04-23T16:19:05.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402020+0000", 2024-04-23T16:19:05.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.885e-06 2024-04-23T16:19:05.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402047+0000", 2024-04-23T16:19:05.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6883e-05 2024-04-23T16:19:05.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402057+0000", 2024-04-23T16:19:05.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6690000000000001e-06 2024-04-23T16:19:05.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402063+0000", 2024-04-23T16:19:05.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0809999999999999e-06 2024-04-23T16:19:05.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402125+0000", 2024-04-23T16:19:05.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2100999999999994e-05 2024-04-23T16:19:05.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402250+0000", 2024-04-23T16:19:05.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000124552 2024-04-23T16:19:05.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:19:05.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402389+0000", 2024-04-23T16:19:05.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139049 2024-04-23T16:19:05.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:19:05.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494492+0000", 2024-04-23T16:19:05.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.092102772999999999 2024-04-23T16:19:05.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:19:05.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.501987+0000", 2024-04-23T16:19:05.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007495303 2024-04-23T16:19:05.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:05.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502040+0000", 2024-04-23T16:19:05.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3383e-05 2024-04-23T16:19:05.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502085+0000", 2024-04-23T16:19:05.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4424e-05 2024-04-23T16:19:05.528 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.528 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.529 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.529 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:116791 2.e 2:745fcdbe:::benchmark_data_smithi188_77627_object116790:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-23T16:19:05.529 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.402079+0000", 2024-04-23T16:19:05.529 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.585486223, 2024-04-23T16:19:05.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.10008534600000001, 2024-04-23T16:19:05.529 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.529 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.529 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:05.529 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:19:05.529 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:19:05.529 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:19:05.530 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116791 2024-04-23T16:19:05.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.530 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402079+0000", 2024-04-23T16:19:05.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402079+0000", 2024-04-23T16:19:05.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402079+0000", 2024-04-23T16:19:05.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.89e-07 2024-04-23T16:19:05.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402202+0000", 2024-04-23T16:19:05.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000122843 2024-04-23T16:19:05.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402208+0000", 2024-04-23T16:19:05.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5039999999999999e-06 2024-04-23T16:19:05.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402214+0000", 2024-04-23T16:19:05.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5559999999999998e-06 2024-04-23T16:19:05.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.404848+0000", 2024-04-23T16:19:05.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026339369999999998 2024-04-23T16:19:05.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.404935+0000", 2024-04-23T16:19:05.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6588999999999994e-05 2024-04-23T16:19:05.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:19:05.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.405030+0000", 2024-04-23T16:19:05.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5508999999999996e-05 2024-04-23T16:19:05.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:19:05.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494530+0000", 2024-04-23T16:19:05.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.089499971999999997 2024-04-23T16:19:05.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:19:05.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502094+0000", 2024-04-23T16:19:05.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075642790000000001 2024-04-23T16:19:05.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:05.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502134+0000", 2024-04-23T16:19:05.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9814000000000002e-05 2024-04-23T16:19:05.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502164+0000", 2024-04-23T16:19:05.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9853999999999999e-05 2024-04-23T16:19:05.534 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.535 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.535 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.535 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:116793 2.e 2:7bd7d2a5:::benchmark_data_smithi188_77627_object116792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-23T16:19:05.535 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.402222+0000", 2024-04-23T16:19:05.535 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.585343293000001, 2024-04-23T16:19:05.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.10001112299999999, 2024-04-23T16:19:05.535 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.535 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.535 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:05.535 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:19:05.535 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:19:05.535 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:19:05.535 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116793 2024-04-23T16:19:05.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.536 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402222+0000", 2024-04-23T16:19:05.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402222+0000", 2024-04-23T16:19:05.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402222+0000", 2024-04-23T16:19:05.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9900000000000001e-07 2024-04-23T16:19:05.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402249+0000", 2024-04-23T16:19:05.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7432000000000001e-05 2024-04-23T16:19:05.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402255+0000", 2024-04-23T16:19:05.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2240000000000003e-06 2024-04-23T16:19:05.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402258+0000", 2024-04-23T16:19:05.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3450000000000002e-06 2024-04-23T16:19:05.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407377+0000", 2024-04-23T16:19:05.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051189169999999997 2024-04-23T16:19:05.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407489+0000", 2024-04-23T16:19:05.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112452 2024-04-23T16:19:05.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:19:05.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407588+0000", 2024-04-23T16:19:05.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8595e-05 2024-04-23T16:19:05.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:19:05.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494565+0000", 2024-04-23T16:19:05.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.086977445 2024-04-23T16:19:05.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:19:05.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502170+0000", 2024-04-23T16:19:05.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076041119999999997 2024-04-23T16:19:05.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:05.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502205+0000", 2024-04-23T16:19:05.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5193e-05 2024-04-23T16:19:05.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502233+0000", 2024-04-23T16:19:05.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7909e-05 2024-04-23T16:19:05.540 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.540 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.540 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.540 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:116795 2.e 2:79edba58:::benchmark_data_smithi188_77627_object116794:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-23T16:19:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.402263+0000", 2024-04-23T16:19:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.585301705999999, 2024-04-23T16:19:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.10041691, 2024-04-23T16:19:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:19:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:19:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:19:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116795 2024-04-23T16:19:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402263+0000", 2024-04-23T16:19:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402263+0000", 2024-04-23T16:19:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402264+0000", 2024-04-23T16:19:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2199999999999999e-07 2024-04-23T16:19:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402356+0000", 2024-04-23T16:19:05.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2485000000000006e-05 2024-04-23T16:19:05.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402361+0000", 2024-04-23T16:19:05.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.434e-06 2024-04-23T16:19:05.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402367+0000", 2024-04-23T16:19:05.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9290000000000003e-06 2024-04-23T16:19:05.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494594+0000", 2024-04-23T16:19:05.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.092226630000000004 2024-04-23T16:19:05.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494699+0000", 2024-04-23T16:19:05.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010501399999999999 2024-04-23T16:19:05.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:19:05.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494851+0000", 2024-04-23T16:19:05.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015200399999999999 2024-04-23T16:19:05.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:19:05.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502238+0000", 2024-04-23T16:19:05.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0073872599999999997 2024-04-23T16:19:05.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:19:05.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502586+0000", 2024-04-23T16:19:05.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00034732900000000001 2024-04-23T16:19:05.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:05.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502631+0000", 2024-04-23T16:19:05.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5120999999999997e-05 2024-04-23T16:19:05.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502680+0000", 2024-04-23T16:19:05.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9282e-05 2024-04-23T16:19:05.546 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.546 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.546 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.546 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop_reply(client.14862.0:116790 2.e e131/110 ondisk, result = 0)", 2024-04-23T16:19:05.546 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.405073+0000", 2024-04-23T16:19:05.546 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.582492058, 2024-04-23T16:19:05.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.089426004000000003, 2024-04-23T16:19:05.547 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.547 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.547 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:19:05.547 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.405073+0000", 2024-04-23T16:19:05.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.405073+0000", 2024-04-23T16:19:05.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.405074+0000", 2024-04-23T16:19:05.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4169999999999999e-06 2024-04-23T16:19:05.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.405079+0000", 2024-04-23T16:19:05.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.198e-06 2024-04-23T16:19:05.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.405088+0000", 2024-04-23T16:19:05.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2619999999999998e-06 2024-04-23T16:19:05.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.405095+0000", 2024-04-23T16:19:05.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9439999999999999e-06 2024-04-23T16:19:05.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494479+0000", 2024-04-23T16:19:05.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.089384384999999997 2024-04-23T16:19:05.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494486+0000", 2024-04-23T16:19:05.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9970000000000001e-06 2024-04-23T16:19:05.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494499+0000", 2024-04-23T16:19:05.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2801e-05 2024-04-23T16:19:05.550 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.550 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.550 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.550 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:116799 2.1 2:839676f0:::benchmark_data_smithi188_77627_object116798:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-23T16:19:05.550 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.406915+0000", 2024-04-23T16:19:05.551 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.580649639000001, 2024-04-23T16:19:05.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.087679254999999998, 2024-04-23T16:19:05.551 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.551 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.551 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:05.551 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:19:05.551 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:19:05.551 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:19:05.551 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116799 2024-04-23T16:19:05.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.551 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.406915+0000", 2024-04-23T16:19:05.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.406915+0000", 2024-04-23T16:19:05.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.406917+0000", 2024-04-23T16:19:05.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0219999999999999e-06 2024-04-23T16:19:05.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.406975+0000", 2024-04-23T16:19:05.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7541999999999999e-05 2024-04-23T16:19:05.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.406984+0000", 2024-04-23T16:19:05.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1959999999999997e-06 2024-04-23T16:19:05.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.406995+0000", 2024-04-23T16:19:05.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0864e-05 2024-04-23T16:19:05.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407037+0000", 2024-04-23T16:19:05.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2375e-05 2024-04-23T16:19:05.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407120+0000", 2024-04-23T16:19:05.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2659999999999998e-05 2024-04-23T16:19:05.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:19:05.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407226+0000", 2024-04-23T16:19:05.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000106579 2024-04-23T16:19:05.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:19:05.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.490902+0000", 2024-04-23T16:19:05.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.083675362000000003 2024-04-23T16:19:05.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:19:05.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494487+0000", 2024-04-23T16:19:05.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035854189999999998 2024-04-23T16:19:05.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:05.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494547+0000", 2024-04-23T16:19:05.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9864999999999997e-05 2024-04-23T16:19:05.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494594+0000", 2024-04-23T16:19:05.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7370999999999998e-05 2024-04-23T16:19:05.556 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.556 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.556 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.556 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:116801 2.1 2:89be7249:::benchmark_data_smithi188_77627_object116800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-23T16:19:05.556 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.407018+0000", 2024-04-23T16:19:05.556 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.580546978000001, 2024-04-23T16:19:05.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.095095130999999999, 2024-04-23T16:19:05.557 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.557 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.557 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:05.557 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:19:05.557 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:19:05.557 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:19:05.557 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116801 2024-04-23T16:19:05.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.557 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407018+0000", 2024-04-23T16:19:05.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407018+0000", 2024-04-23T16:19:05.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407019+0000", 2024-04-23T16:19:05.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2799999999999995e-07 2024-04-23T16:19:05.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407049+0000", 2024-04-23T16:19:05.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0757999999999999e-05 2024-04-23T16:19:05.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407054+0000", 2024-04-23T16:19:05.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6140000000000002e-06 2024-04-23T16:19:05.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407065+0000", 2024-04-23T16:19:05.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0797e-05 2024-04-23T16:19:05.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.490930+0000", 2024-04-23T16:19:05.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.083864994999999998 2024-04-23T16:19:05.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.491057+0000", 2024-04-23T16:19:05.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012708399999999999 2024-04-23T16:19:05.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:19:05.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.491201+0000", 2024-04-23T16:19:05.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014382899999999999 2024-04-23T16:19:05.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:19:05.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.495451+0000", 2024-04-23T16:19:05.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042499419999999996 2024-04-23T16:19:05.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:19:05.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502029+0000", 2024-04-23T16:19:05.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0065782059999999996 2024-04-23T16:19:05.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:05.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502074+0000", 2024-04-23T16:19:05.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5132000000000001e-05 2024-04-23T16:19:05.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502113+0000", 2024-04-23T16:19:05.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8946000000000003e-05 2024-04-23T16:19:05.562 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.562 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.562 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.562 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:116802 2.e 2:7ce58b94:::benchmark_data_smithi188_77627_object116801:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-23T16:19:05.562 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.407074+0000", 2024-04-23T16:19:05.562 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.580490650000002, 2024-04-23T16:19:05.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.095874181000000003, 2024-04-23T16:19:05.563 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.563 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.563 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:05.563 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:19:05.563 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:19:05.563 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:19:05.563 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116802 2024-04-23T16:19:05.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.563 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407074+0000", 2024-04-23T16:19:05.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407074+0000", 2024-04-23T16:19:05.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407075+0000", 2024-04-23T16:19:05.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7999999999999996e-07 2024-04-23T16:19:05.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407098+0000", 2024-04-23T16:19:05.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3476999999999999e-05 2024-04-23T16:19:05.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407102+0000", 2024-04-23T16:19:05.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0820000000000001e-06 2024-04-23T16:19:05.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407106+0000", 2024-04-23T16:19:05.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9720000000000003e-06 2024-04-23T16:19:05.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.498268+0000", 2024-04-23T16:19:05.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.091162033000000003 2024-04-23T16:19:05.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.498414+0000", 2024-04-23T16:19:05.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000146062 2024-04-23T16:19:05.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:19:05.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.498616+0000", 2024-04-23T16:19:05.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020167899999999999 2024-04-23T16:19:05.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:19:05.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502791+0000", 2024-04-23T16:19:05.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041754560000000001 2024-04-23T16:19:05.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:19:05.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502835+0000", 2024-04-23T16:19:05.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3497e-05 2024-04-23T16:19:05.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:05.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502882+0000", 2024-04-23T16:19:05.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7104000000000002e-05 2024-04-23T16:19:05.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502948+0000", 2024-04-23T16:19:05.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6339000000000003e-05 2024-04-23T16:19:05.568 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.568 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.568 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.568 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop_reply(client.14862.0:116791 2.e e131/110 ondisk, result = 0)", 2024-04-23T16:19:05.568 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.407574+0000", 2024-04-23T16:19:05.568 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.57999088, 2024-04-23T16:19:05.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.086961826000000006, 2024-04-23T16:19:05.569 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.569 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.569 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:19:05.569 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407574+0000", 2024-04-23T16:19:05.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407574+0000", 2024-04-23T16:19:05.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407575+0000", 2024-04-23T16:19:05.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1739999999999999e-06 2024-04-23T16:19:05.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407579+0000", 2024-04-23T16:19:05.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.048e-06 2024-04-23T16:19:05.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407587+0000", 2024-04-23T16:19:05.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3660000000000003e-06 2024-04-23T16:19:05.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407591+0000", 2024-04-23T16:19:05.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9219999999999996e-06 2024-04-23T16:19:05.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494520+0000", 2024-04-23T16:19:05.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.086928064999999999 2024-04-23T16:19:05.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494525+0000", 2024-04-23T16:19:05.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7540000000000003e-06 2024-04-23T16:19:05.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494536+0000", 2024-04-23T16:19:05.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0497e-05 2024-04-23T16:19:05.572 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.572 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.572 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.572 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop_reply(client.14862.0:116799 2.1 e131/110 ondisk, result = 0)", 2024-04-23T16:19:05.572 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.409816+0000", 2024-04-23T16:19:05.572 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.577748866, 2024-04-23T16:19:05.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.084785458999999994, 2024-04-23T16:19:05.573 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:05.573 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:05.573 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:19:05.573 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:05.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:05.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.409816+0000", 2024-04-23T16:19:05.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:05.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.409816+0000", 2024-04-23T16:19:05.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:05.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:05.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.409817+0000", 2024-04-23T16:19:05.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.469e-06 2024-04-23T16:19:05.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:05.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.409822+0000", 2024-04-23T16:19:05.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3409999999999996e-06 2024-04-23T16:19:05.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:05.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.491187+0000", 2024-04-23T16:19:05.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.081364991999999997 2024-04-23T16:19:05.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:05.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.491196+0000", 2024-04-23T16:19:05.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0610000000000002e-06 2024-04-23T16:19:05.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:05.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494473+0000", 2024-04-23T16:19:05.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003276953 2024-04-23T16:19:05.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:05.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494481+0000", 2024-04-23T16:19:05.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4379999999999995e-06 2024-04-23T16:19:05.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:05.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:05.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:05.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494601+0000", 2024-04-23T16:19:05.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012020500000000001 2024-04-23T16:19:05.576 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.576 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.576 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.576 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:05.576 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:05.576 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:19:05.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 75 16 154083 154067 128.373 70.25 0.00784604 0.00778846 2024-04-23T16:19:05.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:05 smithi148 ceph-mon[44520]: pgmap v426: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 14 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-23T16:19:05.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:05 smithi148 ceph-mon[44520]: 2.3 scrub starts 2024-04-23T16:19:05.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:05 smithi188 ceph-mon[44101]: pgmap v426: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 14 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-23T16:19:05.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:05 smithi188 ceph-mon[44101]: 2.3 scrub starts 2024-04-23T16:19:06.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:05 smithi103 ceph-mon[39629]: pgmap v426: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 14 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-23T16:19:06.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:05 smithi103 ceph-mon[39629]: 2.3 scrub starts 2024-04-23T16:19:06.182 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:19:06.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:06.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:19:06.222 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-23T16:19:06.293 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:19:06.324 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:06.352 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:19:06.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:06.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:19:06.458 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:19:06.494 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:06.520 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:19:06.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:06.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:19:06.624 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:19:06.657 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:06.686 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:19:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:06.697 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:19:06.798 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:19:06.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 76 16 155188 155172 127.592 69.0625 0.00542539 0.00783568 2024-04-23T16:19:06.831 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:06.853 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:19:06.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:06.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:19:06.960 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:19:06.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:06 smithi188 ceph-mon[44101]: 2.0 scrub starts 2024-04-23T16:19:06.990 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:06 smithi148 ceph-mon[44520]: 2.0 scrub starts 2024-04-23T16:19:06.993 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:07.023 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:19:07.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:07.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:19:07.033 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:06 smithi103 ceph-mon[39629]: 2.0 scrub starts 2024-04-23T16:19:07.130 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:19:07.158 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:07.177 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:19:07.179 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:19:07.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:07.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:19:07.281 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:19:07.331 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:07.430 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:19:07.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:07.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:19:07.536 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:19:07.574 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:07.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 77 16 156217 156201 126.77 64.3125 0.00586324 0.00788694 2024-04-23T16:19:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:07 smithi188 ceph-mon[44101]: 2.3 scrub ok 2024-04-23T16:19:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:07 smithi188 ceph-mon[44101]: pgmap v427: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-04-23T16:19:07.979 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:19:08.081 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-04-23T16:19:08.081 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-04-23T16:19:08.081 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-04-23T16:19:08.081 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-04-23T16:19:08.081 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-04-23T16:19:08.081 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-04-23T16:19:08.081 INFO:teuthology.orchestra.run.smithi103.stderr:bench [] [] [] [] 2024-04-23T16:19:08.082 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs debug_inject_read_zeros 2024-04-23T16:19:08.082 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs files list 2024-04-23T16:19:08.082 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs stats 2024-04-23T16:19:08.082 INFO:teuthology.orchestra.run.smithi103.stderr:bluestore allocator dump block 2024-04-23T16:19:08.082 INFO:teuthology.orchestra.run.smithi103.stderr:Error EINVAL: invalid command 2024-04-23T16:19:08.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:07 smithi103 ceph-mon[39629]: 2.3 scrub ok 2024-04-23T16:19:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:07 smithi103 ceph-mon[39629]: pgmap v427: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-04-23T16:19:08.119 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:19:08.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:07 smithi148 ceph-mon[44520]: 2.3 scrub ok 2024-04-23T16:19:08.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:07 smithi148 ceph-mon[44520]: pgmap v427: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-04-23T16:19:08.452 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:19:08.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:08.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:19:08.556 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:19:08.594 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:08.682 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:19:08.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:08.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:19:08.788 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:19:08.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 78 16 157290 157274 126.005 67.0625 0.016704 0.00793429 2024-04-23T16:19:08.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:08 smithi103 ceph-mon[39629]: 2.0 scrub ok 2024-04-23T16:19:08.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:08 smithi103 ceph-mon[39629]: pgmap v428: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-23T16:19:08.836 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:08.966 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:19:08.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:08.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:19:08.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:08 smithi188 ceph-mon[44101]: 2.0 scrub ok 2024-04-23T16:19:08.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:08 smithi188 ceph-mon[44101]: pgmap v428: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-23T16:19:09.071 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:19:09.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:08 smithi148 ceph-mon[44520]: 2.0 scrub ok 2024-04-23T16:19:09.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:08 smithi148 ceph-mon[44520]: pgmap v428: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-23T16:19:09.126 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:09.157 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:19:09.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:09.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:19:09.262 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:19:09.289 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:09.313 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:19:09.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:09.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:19:09.417 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:19:09.455 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:09.486 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:19:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:09.491 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:19:09.591 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:19:09.640 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T16:19:09.640 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-23T16:19:09.641 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.9 'trim stale osdmaps' 2024-04-23T16:19:09.650 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:09 smithi188 ceph-mon[44101]: 2.c scrub starts 2024-04-23T16:19:09.773 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:19:09.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:09.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:19:09.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 79 16 158302 158286 125.211 63.25 0.00771227 0.0079848 2024-04-23T16:19:09.880 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:19:09.905 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:09 smithi148 ceph-mon[44520]: 2.c scrub starts 2024-04-23T16:19:09.907 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:09.930 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:19:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:09.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:19:10.036 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:19:10.084 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:10.084 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:09 smithi103 ceph-mon[39629]: 2.c scrub starts 2024-04-23T16:19:10.140 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:19:10.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:10.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:19:10.247 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:19:10.281 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:10.307 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:19:10.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:10.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:19:10.411 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:19:10.477 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:10.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:19:10.817841 min lat: 0.000949989 max lat: 0.207898 avg lat: 0.00803148 2024-04-23T16:19:10.818 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:19:10.818 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 80 16 159358 159342 124.47 66 0.0109453 0.00803148 2024-04-23T16:19:10.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:10 smithi103 ceph-mon[39629]: 2.c scrub ok 2024-04-23T16:19:10.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:10 smithi103 ceph-mon[39629]: pgmap v429: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-04-23T16:19:10.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:10 smithi188 ceph-mon[44101]: 2.c scrub ok 2024-04-23T16:19:10.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:10 smithi188 ceph-mon[44101]: pgmap v429: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-04-23T16:19:11.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:10 smithi148 ceph-mon[44520]: 2.c scrub ok 2024-04-23T16:19:11.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:10 smithi148 ceph-mon[44520]: pgmap v429: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-04-23T16:19:11.458 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:19:11.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:11.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:19:11.567 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:19:11.603 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:11.820 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 81 16 160422 160406 123.75 66.5 0.00571857 0.00807928 2024-04-23T16:19:12.821 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 82 16 161473 161457 123.042 65.6875 0.00627473 0.00812539 2024-04-23T16:19:13.821 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 83 16 162329 162313 122.204 53.5 0.0123293 0.00818109 2024-04-23T16:19:13.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:13 smithi103 ceph-mon[39629]: 2.9 scrub starts 2024-04-23T16:19:13.858 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:13 smithi103 ceph-mon[39629]: pgmap v430: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.08k op/s 2024-04-23T16:19:13.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:13 smithi148 ceph-mon[44520]: 2.9 scrub starts 2024-04-23T16:19:13.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:13 smithi148 ceph-mon[44520]: pgmap v430: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.08k op/s 2024-04-23T16:19:13.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:13 smithi188 ceph-mon[44101]: 2.9 scrub starts 2024-04-23T16:19:13.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:13 smithi188 ceph-mon[44101]: pgmap v430: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.08k op/s 2024-04-23T16:19:14.273 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:19:14.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:14.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:19:14.385 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:19:14.422 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:14.445 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:19:14.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:14.449 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:19:14.550 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:19:14.581 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:14.604 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:19:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:14.611 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:19:14.712 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:19:14.759 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:14.821 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 84 16 163262 163246 121.444 58.3125 0.00804957 0.00822871 2024-04-23T16:19:14.905 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:19:14.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:14.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:19:15.026 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:19:15.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:14 smithi103 ceph-mon[39629]: 2.6 scrub starts 2024-04-23T16:19:15.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:14 smithi103 ceph-mon[39629]: pgmap v431: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.14k op/s 2024-04-23T16:19:15.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:19:15.106 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:15.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:14 smithi148 ceph-mon[44520]: 2.6 scrub starts 2024-04-23T16:19:15.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:14 smithi148 ceph-mon[44520]: pgmap v431: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.14k op/s 2024-04-23T16:19:15.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:19:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:14 smithi188 ceph-mon[44101]: 2.6 scrub starts 2024-04-23T16:19:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:14 smithi188 ceph-mon[44101]: pgmap v431: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.14k op/s 2024-04-23T16:19:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:19:15.590 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:19:15.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:15.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:19:15.695 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:19:15.724 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:15.744 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:19:15.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:15.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:19:15.821 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 85 16 164178 164162 120.689 57.25 0.00910991 0.00828404 2024-04-23T16:19:15.848 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:19:15.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:15 smithi103 ceph-mon[39629]: 2.9 scrub ok 2024-04-23T16:19:15.876 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:15.897 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:19:15.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:15.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:19:16.001 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:19:16.038 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:16.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:15 smithi148 ceph-mon[44520]: 2.9 scrub ok 2024-04-23T16:19:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:15 smithi188 ceph-mon[44101]: 2.9 scrub ok 2024-04-23T16:19:16.423 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:19:16.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:16.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:19:16.427 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:19:16.528 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:19:16.554 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:16.577 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:19:16.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:16.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:19:16.682 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:19:16.710 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:16.730 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:19:16.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:16.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:19:16.765 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:19:16.766 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:19:16.766 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.766 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:163558 2.e e141/110 2:7cb1c3fc:::benchmark_data_smithi188_77627_object163557:head v 141'10203, mlcod=141'10201)", 2024-04-23T16:19:16.766 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:15.220542+0000", 2024-04-23T16:19:16.766 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.005642724, 2024-04-23T16:19:16.766 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.005667081, 2024-04-23T16:19:16.766 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:16.766 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:16.766 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:19:16.766 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:16.766 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.766 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:16.767 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.220542+0000", 2024-04-23T16:19:16.767 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:16.767 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.767 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.767 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:16.767 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.220542+0000", 2024-04-23T16:19:16.767 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:16.767 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.767 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.767 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:16.767 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.220544+0000", 2024-04-23T16:19:16.767 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9819999999999998e-06 2024-04-23T16:19:16.768 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.768 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.768 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:16.768 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.220568+0000", 2024-04-23T16:19:16.768 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4187999999999999e-05 2024-04-23T16:19:16.768 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.768 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.768 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:16.768 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.220584+0000", 2024-04-23T16:19:16.768 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6072000000000001e-05 2024-04-23T16:19:16.768 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.768 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.768 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:16.769 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.220593+0000", 2024-04-23T16:19:16.769 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9279999999999999e-06 2024-04-23T16:19:16.769 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.769 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.769 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:16.769 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.220693+0000", 2024-04-23T16:19:16.769 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9539999999999999e-05 2024-04-23T16:19:16.769 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.769 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.769 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:16.769 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.220712+0000", 2024-04-23T16:19:16.769 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9726999999999999e-05 2024-04-23T16:19:16.769 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:16.770 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:16.770 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:16.770 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.770 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.770 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:163567 2.e e141/110 2:75fcc7fa:::benchmark_data_smithi188_77627_object163566:head v 141'10204, mlcod=141'10201)", 2024-04-23T16:19:16.770 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:15.221540+0000", 2024-04-23T16:19:16.770 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.0046448449999999999, 2024-04-23T16:19:16.770 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047219250000000001, 2024-04-23T16:19:16.770 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:16.770 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:16.770 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:19:16.770 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:16.770 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.771 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:16.771 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.221540+0000", 2024-04-23T16:19:16.771 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:16.771 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.771 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.771 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:16.771 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.221540+0000", 2024-04-23T16:19:16.771 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:16.771 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.771 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.771 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:16.771 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.221541+0000", 2024-04-23T16:19:16.771 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2589999999999999e-06 2024-04-23T16:19:16.772 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.772 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.772 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:16.772 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.221562+0000", 2024-04-23T16:19:16.772 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0903999999999998e-05 2024-04-23T16:19:16.772 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.772 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.772 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:16.772 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.221571+0000", 2024-04-23T16:19:16.772 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3589999999999997e-06 2024-04-23T16:19:16.772 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.772 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.772 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:16.772 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.221578+0000", 2024-04-23T16:19:16.773 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8809999999999998e-06 2024-04-23T16:19:16.773 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.773 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.773 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:16.773 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.224246+0000", 2024-04-23T16:19:16.773 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026676360000000001 2024-04-23T16:19:16.773 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.773 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.773 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:16.773 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.224267+0000", 2024-04-23T16:19:16.773 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1047e-05 2024-04-23T16:19:16.773 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:16.773 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:16.773 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:16.774 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.774 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.774 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:163570 2.a e141/110 2:58193750:::benchmark_data_smithi188_77627_object163569:head v 141'10266, mlcod=141'10264)", 2024-04-23T16:19:16.774 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:15.221773+0000", 2024-04-23T16:19:16.774 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.0044116420000000003, 2024-04-23T16:19:16.774 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045289980000000002, 2024-04-23T16:19:16.774 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:16.774 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:16.774 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:19:16.774 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:16.774 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.775 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:16.775 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.221773+0000", 2024-04-23T16:19:16.775 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:16.775 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.775 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.775 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:16.775 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.221773+0000", 2024-04-23T16:19:16.775 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:16.775 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.775 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.775 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:16.775 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.221774+0000", 2024-04-23T16:19:16.775 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5600000000000004e-07 2024-04-23T16:19:16.775 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.776 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.776 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:16.776 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.221826+0000", 2024-04-23T16:19:16.776 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1830999999999999e-05 2024-04-23T16:19:16.776 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.776 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.776 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:16.776 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.221833+0000", 2024-04-23T16:19:16.776 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5700000000000004e-06 2024-04-23T16:19:16.776 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.776 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.776 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:16.776 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.221839+0000", 2024-04-23T16:19:16.777 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5679999999999999e-06 2024-04-23T16:19:16.777 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.777 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.777 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:16.777 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.221897+0000", 2024-04-23T16:19:16.777 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7873e-05 2024-04-23T16:19:16.777 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:16.777 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:16.777 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:16.777 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.221917+0000", 2024-04-23T16:19:16.777 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9964999999999999e-05 2024-04-23T16:19:16.777 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:16.778 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:16.778 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:16.778 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:16.778 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:19:16.778 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 3 2024-04-23T16:19:16.778 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:19:16.821 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 86 16 165184 165168 120.016 62.875 0.0055882 0.00833006 2024-04-23T16:19:16.834 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:16 smithi148 ceph-mon[44520]: 2.6 scrub ok 2024-04-23T16:19:16.834 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:19:16.834 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:19:16.834 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:16 smithi148 ceph-mon[44520]: pgmap v432: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-23T16:19:16.835 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:19:16.877 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:16.900 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:19:16.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:16.904 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:19:17.005 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:19:17.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:16 smithi103 ceph-mon[39629]: 2.6 scrub ok 2024-04-23T16:19:17.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:19:17.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:19:17.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:16 smithi103 ceph-mon[39629]: pgmap v432: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-23T16:19:17.072 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:17.135 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:19:17.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:17.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:19:17.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:16 smithi188 ceph-mon[44101]: 2.6 scrub ok 2024-04-23T16:19:17.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:19:17.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:19:17.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:16 smithi188 ceph-mon[44101]: pgmap v432: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-23T16:19:17.257 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:19:17.288 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:17.320 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:19:17.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:17.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:19:17.423 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:19:17.457 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:17.485 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:19:17.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:17.490 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:19:17.526 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-23T16:19:17.591 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:19:17.623 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:17.646 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:19:17.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:17.650 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:19:17.751 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:19:17.780 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:17.809 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:19:17.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:17.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:19:17.821 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 87 16 166201 166185 119.367 63.5625 0.00445597 0.00837547 2024-04-23T16:19:17.912 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:19:17.939 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:17 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:19:17.939 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:17 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:19:17.939 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:19:17.939 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:19:17.939 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:19:17.939 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:17 smithi188 ceph-mon[44101]: osdmap e142: 12 total, 12 up, 12 in 2024-04-23T16:19:17.939 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:17 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:19:17.939 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:17 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:19:17.941 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:17.978 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:19:17.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:17.982 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:19:18.069 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:19:18.082 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:19:18.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:17 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:19:18.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:17 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:19:18.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:19:18.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:19:18.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:19:18.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:17 smithi103 ceph-mon[39629]: osdmap e142: 12 total, 12 up, 12 in 2024-04-23T16:19:18.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:17 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:19:18.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:17 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:19:18.108 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:17 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:19:18.108 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:17 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:19:18.109 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:19:18.109 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:19:18.109 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:19:18.109 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:17 smithi148 ceph-mon[44520]: osdmap e142: 12 total, 12 up, 12 in 2024-04-23T16:19:18.109 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:17 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:19:18.109 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:17 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:19:18.110 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:18.111 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:19:18.133 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:19:18.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:18.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:19:18.155 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:19:18.194 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:19:18.229 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:19:18.238 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:19:18.266 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:19:18.288 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:18.305 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:19:18.335 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:19:18.366 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:19:18.394 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:19:18.421 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:19:18.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:18.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:19:18.427 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:19:18.460 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:19:18.463 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:19:18.463 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:19:18.464 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:19:18.464 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:19:18.464 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:19:18.464 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:19:18.464 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:19:18.464 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:19:18.464 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:19:18.464 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:19:18.464 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:19:18.464 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:19:18.528 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:19:18.555 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:18.580 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:19:18.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:18.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:19:18.684 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:19:18.741 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:18.821 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 88 16 167221 167205 118.735 63.75 0.00969783 0.0084184 2024-04-23T16:19:18.843 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:18 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/318659235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:19:18.843 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:18.843 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:18.843 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:18.843 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:18 smithi148 ceph-mon[44520]: osdmap e143: 12 total, 12 up, 12 in 2024-04-23T16:19:18.843 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:18.843 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:18.843 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:18.843 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:18 smithi148 ceph-mon[44520]: pgmap v435: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-04-23T16:19:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:18 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/318659235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:19:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:18 smithi103 ceph-mon[39629]: osdmap e143: 12 total, 12 up, 12 in 2024-04-23T16:19:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:18 smithi103 ceph-mon[39629]: pgmap v435: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-04-23T16:19:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:18 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/318659235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:19:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:18 smithi188 ceph-mon[44101]: osdmap e143: 12 total, 12 up, 12 in 2024-04-23T16:19:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:18 smithi188 ceph-mon[44101]: pgmap v435: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-04-23T16:19:19.821 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 89 16 168327 168311 118.178 69.125 0.01883 0.00846066 2024-04-23T16:19:19.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:19.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:19.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:19.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:19.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:19.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:20.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:20.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:20.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:20.795 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:19:20.821 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 90 16 169346 169330 117.572 63.6875 0.0124363 0.00850282 2024-04-23T16:19:20.841 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:19:20.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:20.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:19:20.878 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-04-23T16:19:20.878 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-04-23T16:19:20.878 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-04-23T16:19:20.878 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-04-23T16:19:20.878 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-04-23T16:19:20.878 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-04-23T16:19:20.878 INFO:teuthology.orchestra.run.smithi103.stderr:bench [] [] [] [] 2024-04-23T16:19:20.878 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs debug_inject_read_zeros 2024-04-23T16:19:20.878 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs files list 2024-04-23T16:19:20.878 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs stats 2024-04-23T16:19:20.878 INFO:teuthology.orchestra.run.smithi103.stderr:bluestore allocator dump block 2024-04-23T16:19:20.879 INFO:teuthology.orchestra.run.smithi103.stderr:Error EINVAL: invalid command 2024-04-23T16:19:20.944 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:19:20.975 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:21.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:20 smithi103 ceph-mon[39629]: pgmap v436: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-04-23T16:19:21.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:20 smithi148 ceph-mon[44520]: pgmap v436: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-04-23T16:19:21.124 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:19:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:21.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:19:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:20 smithi188 ceph-mon[44101]: pgmap v436: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-04-23T16:19:21.228 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:19:21.256 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:21.300 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:19:21.822 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Total time run: 90.032 2024-04-23T16:19:21.822 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Total writes made: 169346 2024-04-23T16:19:21.822 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Write size: 65536 2024-04-23T16:19:21.822 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Object size: 65536 2024-04-23T16:19:21.822 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Bandwidth (MB/sec): 117.56 2024-04-23T16:19:21.822 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev Bandwidth: 61.628 2024-04-23T16:19:21.822 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max bandwidth (MB/sec): 366.125 2024-04-23T16:19:21.822 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min bandwidth (MB/sec): 53.5 2024-04-23T16:19:21.822 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Average IOPS: 1880 2024-04-23T16:19:21.823 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev IOPS: 986.048 2024-04-23T16:19:21.823 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max IOPS: 5858 2024-04-23T16:19:21.823 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min IOPS: 856 2024-04-23T16:19:21.823 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Average Latency(s): 0.00850571 2024-04-23T16:19:21.823 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev Latency(s): 0.00862555 2024-04-23T16:19:21.823 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max latency(s): 0.207898 2024-04-23T16:19:21.823 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min latency(s): 0.000949989 2024-04-23T16:19:21.828 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Cleaning up (deleting benchmark objects) 2024-04-23T16:19:21.858 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:21 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:19:21.858 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:21 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:19:21.858 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:21 smithi103 ceph-mon[39629]: osdmap e144: 12 total, 12 up, 12 in 2024-04-23T16:19:21.858 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:21 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/321774133' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:19:21.884 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T16:19:21.885 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-23T16:19:21.885 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.10 'trim stale osdmaps' 2024-04-23T16:19:21.886 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:19:22.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:21 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:19:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:21 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:19:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:21 smithi148 ceph-mon[44520]: osdmap e144: 12 total, 12 up, 12 in 2024-04-23T16:19:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:21 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/321774133' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:19:22.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:21 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:19:22.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:21 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:19:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:21 smithi188 ceph-mon[44101]: osdmap e144: 12 total, 12 up, 12 in 2024-04-23T16:19:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:21 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/321774133' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:19:22.308 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:19:22.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:22.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:19:22.414 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:19:22.463 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:22.829 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:19:22.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:22.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:19:22.933 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:19:22.971 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:23.001 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:19:23.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:23.004 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:19:23.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:22 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:19:23.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:22 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/321774133' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:19:23.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:22 smithi103 ceph-mon[39629]: osdmap e145: 12 total, 12 up, 12 in 2024-04-23T16:19:23.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:22 smithi103 ceph-mon[39629]: pgmap v439: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 930 op/s 2024-04-23T16:19:23.105 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:19:23.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:22 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:19:23.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:22 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/321774133' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:19:23.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:22 smithi148 ceph-mon[44520]: osdmap e145: 12 total, 12 up, 12 in 2024-04-23T16:19:23.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:22 smithi148 ceph-mon[44520]: pgmap v439: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 930 op/s 2024-04-23T16:19:23.144 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:22 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:19:23.144 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:22 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/321774133' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:19:23.144 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:22 smithi188 ceph-mon[44101]: osdmap e145: 12 total, 12 up, 12 in 2024-04-23T16:19:23.144 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:22 smithi188 ceph-mon[44101]: pgmap v439: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 930 op/s 2024-04-23T16:19:23.144 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:23.173 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:19:23.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:23.181 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:19:23.282 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:19:23.322 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:23.351 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:19:23.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:23.354 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:19:23.455 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:19:23.503 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:23.671 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:19:23.676 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:19:23.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:23.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:19:23.709 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:19:23.779 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:19:23.806 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:23.833 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:19:23.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:23.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:19:23.937 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:19:23.977 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:24.005 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:19:24.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:24.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:19:24.110 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:19:24.140 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:24.160 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:19:24.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:24.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:19:24.267 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:19:24.295 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:24.322 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:19:24.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:24.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:19:24.426 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:19:24.468 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:24.865 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:19:25.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:25 smithi103 ceph-mon[39629]: pgmap v440: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 164 B/s rd, 50 MiB/s wr, 930 op/s 2024-04-23T16:19:25.822 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:19:25.822 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:19:25.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:25 smithi148 ceph-mon[44520]: pgmap v440: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 164 B/s rd, 50 MiB/s wr, 930 op/s 2024-04-23T16:19:25.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:25 smithi188 ceph-mon[44101]: pgmap v440: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 164 B/s rd, 50 MiB/s wr, 930 op/s 2024-04-23T16:19:26.557 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-23T16:19:26.655 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:19:26.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:26.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:19:26.768 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:19:26.818 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:27.482 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:19:27.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:27.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:19:27.589 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:19:27.619 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:27.640 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:19:27.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:27.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:19:27.738 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:27 smithi103 ceph-mon[39629]: pgmap v441: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 40 MiB/s wr, 972 op/s 2024-04-23T16:19:27.744 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:19:27.773 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:27.793 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:19:27.795 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:27 smithi148 ceph-mon[44520]: pgmap v441: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 40 MiB/s wr, 972 op/s 2024-04-23T16:19:27.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:27.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:19:27.897 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:19:27.937 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:27.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:27 smithi188 ceph-mon[44101]: pgmap v441: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 40 MiB/s wr, 972 op/s 2024-04-23T16:19:28.307 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:19:28.308 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:19:28.308 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:19:28.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:28.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:19:28.360 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:19:28.409 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:19:28.422 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:19:28.448 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:19:28.448 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:28.456 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-04-23T16:19:28.456 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-04-23T16:19:28.456 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-04-23T16:19:28.456 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-04-23T16:19:28.456 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-04-23T16:19:28.456 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-04-23T16:19:28.456 INFO:teuthology.orchestra.run.smithi103.stderr:bench [] [] [] [] 2024-04-23T16:19:28.456 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs debug_inject_read_zeros 2024-04-23T16:19:28.456 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs files list 2024-04-23T16:19:28.456 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs stats 2024-04-23T16:19:28.457 INFO:teuthology.orchestra.run.smithi103.stderr:bluestore allocator dump block 2024-04-23T16:19:28.457 INFO:teuthology.orchestra.run.smithi103.stderr:Error EINVAL: invalid command 2024-04-23T16:19:28.468 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:19:28.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:28.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:19:28.483 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:19:28.508 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:19:28.533 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:19:28.567 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:19:28.574 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:19:28.593 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:19:28.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:28 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/754257290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:19:28.621 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:28.628 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:19:28.645 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:19:28.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:28.653 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:19:28.659 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:19:28.702 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:19:28.713 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:19:28.713 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:19:28.713 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:19:28.713 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:19:28.713 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:19:28.713 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:19:28.714 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:19:28.714 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:19:28.714 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:19:28.714 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:19:28.714 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:19:28.714 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:19:28.723 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:19:28.753 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:19:28.779 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:28 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/754257290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:19:28.781 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:28.806 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:19:28.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:28.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:19:28.913 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:19:28.953 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:28.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:28 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/754257290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:19:29.163 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:19:29.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:29.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:19:29.267 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:19:29.302 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:29.324 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:19:29.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:29.329 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:19:29.430 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:19:29.460 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:29.481 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:19:29.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:29.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:19:29.550 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.550 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.550 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.550 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.551 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.551 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.551 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.551 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.551 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:29 smithi188 ceph-mon[44101]: pgmap v442: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 26 MiB/s wr, 962 op/s 2024-04-23T16:19:29.551 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.551 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.551 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.552 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.552 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.552 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.552 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.588 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:19:29.629 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:29 smithi103 ceph-mon[39629]: pgmap v442: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 26 MiB/s wr, 962 op/s 2024-04-23T16:19:29.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:29 smithi148 ceph-mon[44520]: pgmap v442: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 26 MiB/s wr, 962 op/s 2024-04-23T16:19:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:29.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:30.812 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:30.854 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T16:19:30.855 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-23T16:19:30.855 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-04-23T16:19:30.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:30.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:31.102 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:19:31.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:31.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:19:31.209 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:19:31.237 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:31.260 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:19:31.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:31.265 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:19:31.366 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:19:31.395 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:31.417 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:19:31.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:31.420 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:19:31.521 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:19:31.550 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:31.566 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:31 smithi188 ceph-mon[44101]: pgmap v443: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 106 B/s rd, 17 MiB/s wr, 945 op/s 2024-04-23T16:19:31.566 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:31 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:19:31.566 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:31 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:19:31.573 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:19:31.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:31.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:19:31.678 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:19:31.695 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:31 smithi103 ceph-mon[39629]: pgmap v443: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 106 B/s rd, 17 MiB/s wr, 945 op/s 2024-04-23T16:19:31.695 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:31 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:19:31.695 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:31 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:19:31.710 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:31.732 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:19:31.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:31.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:19:31.836 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:19:31.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:31 smithi148 ceph-mon[44520]: pgmap v443: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 106 B/s rd, 17 MiB/s wr, 945 op/s 2024-04-23T16:19:31.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:31 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:19:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:31 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:19:31.885 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:32.705 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:32 smithi103 ceph-mon[39629]: osdmap e146: 12 total, 12 up, 12 in 2024-04-23T16:19:32.820 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:19:32.821 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:19:32.821 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:19:32.821 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:19:32.821 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.821 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:118285 2.a e131/110 2:5a4e7f4f:::benchmark_data_smithi188_77627_object118284:head v 131'7446, mlcod=131'7443)", 2024-04-23T16:19:32.821 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:38.414694+0000", 2024-04-23T16:19:32.821 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 54.364242496000003, 2024-04-23T16:19:32.821 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024118554, 2024-04-23T16:19:32.821 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.821 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.821 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.821 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.822 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.822 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.822 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:38.414694+0000", 2024-04-23T16:19:32.822 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.822 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.822 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.822 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.822 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:38.414694+0000", 2024-04-23T16:19:32.822 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.822 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.822 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.822 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.822 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:38.414694+0000", 2024-04-23T16:19:32.822 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2e-07 2024-04-23T16:19:32.823 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.823 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.823 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.823 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:38.414721+0000", 2024-04-23T16:19:32.823 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6466e-05 2024-04-23T16:19:32.823 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.823 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.823 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.823 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:38.414726+0000", 2024-04-23T16:19:32.823 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1800000000000004e-06 2024-04-23T16:19:32.823 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.823 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.823 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.823 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:38.414731+0000", 2024-04-23T16:19:32.823 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6240000000000001e-06 2024-04-23T16:19:32.824 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.824 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.824 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.824 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:38.417715+0000", 2024-04-23T16:19:32.824 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0029847570000000002 2024-04-23T16:19:32.824 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.824 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.824 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.824 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:38.417735+0000", 2024-04-23T16:19:32.824 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9522999999999999e-05 2024-04-23T16:19:32.824 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.824 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.824 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.824 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:38.438762+0000", 2024-04-23T16:19:32.825 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.021027496 2024-04-23T16:19:32.825 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.825 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.825 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.825 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:38.438813+0000", 2024-04-23T16:19:32.825 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0087999999999999e-05 2024-04-23T16:19:32.825 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.825 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.825 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.825 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.825 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.825 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:124235 2.a e134/110 2:5637ce15:::benchmark_data_smithi188_77627_object124234:head v 134'7851, mlcod=134'7846)", 2024-04-23T16:19:32.825 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:42.355206+0000", 2024-04-23T16:19:32.825 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 50.423730968000001, 2024-04-23T16:19:32.825 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.026703952999999999, 2024-04-23T16:19:32.826 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.826 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.826 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.826 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.826 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.826 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.826 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.355206+0000", 2024-04-23T16:19:32.826 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.826 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.826 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.826 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.826 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.355206+0000", 2024-04-23T16:19:32.826 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.827 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.827 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.827 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.827 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.355208+0000", 2024-04-23T16:19:32.827 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.046e-06 2024-04-23T16:19:32.827 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.827 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.827 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.827 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.355243+0000", 2024-04-23T16:19:32.827 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5633000000000003e-05 2024-04-23T16:19:32.827 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.827 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.827 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.827 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.355256+0000", 2024-04-23T16:19:32.828 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2322e-05 2024-04-23T16:19:32.828 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.828 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.828 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.828 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.355265+0000", 2024-04-23T16:19:32.828 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2590000000000006e-06 2024-04-23T16:19:32.828 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.828 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.828 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.828 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.355380+0000", 2024-04-23T16:19:32.828 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000115158 2024-04-23T16:19:32.828 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.828 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.828 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.829 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.355402+0000", 2024-04-23T16:19:32.829 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1559000000000001e-05 2024-04-23T16:19:32.829 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.829 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.829 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.829 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.381859+0000", 2024-04-23T16:19:32.829 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.026457722999999999 2024-04-23T16:19:32.829 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.829 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.829 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.829 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:42.381909+0000", 2024-04-23T16:19:32.829 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0253000000000001e-05 2024-04-23T16:19:32.829 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.829 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.829 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.830 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.830 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.830 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:126401 2.a e136/110 2:5ffb5666:::benchmark_data_smithi188_77627_object126400:head v 136'7981, mlcod=135'7978)", 2024-04-23T16:19:32.830 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:43.819070+0000", 2024-04-23T16:19:32.830 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.959866495, 2024-04-23T16:19:32.830 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.023420601999999999, 2024-04-23T16:19:32.830 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.830 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.830 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.830 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.830 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.830 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.830 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:43.819070+0000", 2024-04-23T16:19:32.831 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.831 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.831 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.831 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.831 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:43.819070+0000", 2024-04-23T16:19:32.831 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.831 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.831 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.831 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.831 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:43.819072+0000", 2024-04-23T16:19:32.831 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6750000000000001e-06 2024-04-23T16:19:32.832 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.832 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.832 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:43.819098+0000", 2024-04-23T16:19:32.832 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6112999999999999e-05 2024-04-23T16:19:32.832 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.832 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.832 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:43.819110+0000", 2024-04-23T16:19:32.832 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2215000000000001e-05 2024-04-23T16:19:32.832 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.832 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.832 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:43.819117+0000", 2024-04-23T16:19:32.832 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.781e-06 2024-04-23T16:19:32.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.833 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.833 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.833 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:43.819170+0000", 2024-04-23T16:19:32.833 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3494e-05 2024-04-23T16:19:32.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.833 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.833 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.833 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:43.819185+0000", 2024-04-23T16:19:32.833 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5023e-05 2024-04-23T16:19:32.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.833 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.833 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.833 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:43.842440+0000", 2024-04-23T16:19:32.834 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.023254885999999999 2024-04-23T16:19:32.834 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.834 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.834 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.834 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:43.842491+0000", 2024-04-23T16:19:32.834 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0414999999999998e-05 2024-04-23T16:19:32.834 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.834 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.834 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.834 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.834 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.834 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:147383 2.a e140/110 2:5346537f:::benchmark_data_smithi188_77627_object147382:head v 140'9271, mlcod=140'9267)", 2024-04-23T16:19:32.834 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:59.601452+0000", 2024-04-23T16:19:32.834 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 33.177484603000003, 2024-04-23T16:19:32.834 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.026908653000000001, 2024-04-23T16:19:32.835 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.835 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.835 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.835 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.835 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.835 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:59.601452+0000", 2024-04-23T16:19:32.835 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.835 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.835 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:59.601452+0000", 2024-04-23T16:19:32.835 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:59.601462+0000", 2024-04-23T16:19:32.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6090000000000007e-06 2024-04-23T16:19:32.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:59.601522+0000", 2024-04-23T16:19:32.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0646999999999997e-05 2024-04-23T16:19:32.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:59.601527+0000", 2024-04-23T16:19:32.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8160000000000001e-06 2024-04-23T16:19:32.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:59.601530+0000", 2024-04-23T16:19:32.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3670000000000001e-06 2024-04-23T16:19:32.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:59.609537+0000", 2024-04-23T16:19:32.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080068980000000001 2024-04-23T16:19:32.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:59.609557+0000", 2024-04-23T16:19:32.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9888999999999999e-05 2024-04-23T16:19:32.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:59.628309+0000", 2024-04-23T16:19:32.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018751651000000001 2024-04-23T16:19:32.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:59.628361+0000", 2024-04-23T16:19:32.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1776e-05 2024-04-23T16:19:32.838 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.838 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.838 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.838 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:155541 2.1 e141/110 2:8736581c:::benchmark_data_smithi188_77627_object155540:head v 141'9674, mlcod=141'9668)", 2024-04-23T16:19:32.838 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:07.149922+0000", 2024-04-23T16:19:32.839 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 25.629014421000001, 2024-04-23T16:19:32.839 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.023147481000000001, 2024-04-23T16:19:32.839 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.839 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.839 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.839 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.839 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.839 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.839 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:07.149922+0000", 2024-04-23T16:19:32.839 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.839 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.839 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.839 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:07.149922+0000", 2024-04-23T16:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:07.149923+0000", 2024-04-23T16:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.06e-07 2024-04-23T16:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:07.149975+0000", 2024-04-23T16:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2546000000000001e-05 2024-04-23T16:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:07.149981+0000", 2024-04-23T16:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5999999999999997e-06 2024-04-23T16:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:07.149986+0000", 2024-04-23T16:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9760000000000003e-06 2024-04-23T16:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:07.166844+0000", 2024-04-23T16:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016858360999999999 2024-04-23T16:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:07.166863+0000", 2024-04-23T16:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9253999999999999e-05 2024-04-23T16:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:07.173044+0000", 2024-04-23T16:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0061809650000000001 2024-04-23T16:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:07.173070+0000", 2024-04-23T16:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5273e-05 2024-04-23T16:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:157257 2.e e141/110 2:790f7b29:::benchmark_data_smithi188_77627_object157256:head v 141'9831, mlcod=141'9827)", 2024-04-23T16:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:08.798471+0000", 2024-04-23T16:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.980465245000001, 2024-04-23T16:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.023131636000000001, 2024-04-23T16:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.798471+0000", 2024-04-23T16:19:32.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.844 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.798471+0000", 2024-04-23T16:19:32.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.844 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.798472+0000", 2024-04-23T16:19:32.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1230000000000001e-06 2024-04-23T16:19:32.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.798500+0000", 2024-04-23T16:19:32.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7807e-05 2024-04-23T16:19:32.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.798510+0000", 2024-04-23T16:19:32.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.978e-06 2024-04-23T16:19:32.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.798523+0000", 2024-04-23T16:19:32.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2652e-05 2024-04-23T16:19:32.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.846 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.846 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.846 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.798575+0000", 2024-04-23T16:19:32.846 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2256000000000002e-05 2024-04-23T16:19:32.846 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.846 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.846 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.846 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.798588+0000", 2024-04-23T16:19:32.846 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2719e-05 2024-04-23T16:19:32.846 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.846 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.846 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.846 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.821558+0000", 2024-04-23T16:19:32.846 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.022970021 2024-04-23T16:19:32.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.847 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.847 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.821603+0000", 2024-04-23T16:19:32.847 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5080000000000002e-05 2024-04-23T16:19:32.847 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.847 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.847 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.847 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:158317 2.a e141/110 2:5a20f1d5:::benchmark_data_smithi188_77627_object158316:head v 141'9938, mlcod=141'9935)", 2024-04-23T16:19:32.847 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:09.832561+0000", 2024-04-23T16:19:32.847 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.946375016000001, 2024-04-23T16:19:32.847 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.031207292000000001, 2024-04-23T16:19:32.847 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.848 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.848 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.848 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.848 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.848 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.832561+0000", 2024-04-23T16:19:32.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.848 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.848 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.832561+0000", 2024-04-23T16:19:32.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.849 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.849 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.832564+0000", 2024-04-23T16:19:32.849 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0829999999999998e-06 2024-04-23T16:19:32.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.849 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.849 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.832611+0000", 2024-04-23T16:19:32.849 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7330000000000003e-05 2024-04-23T16:19:32.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.849 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.849 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.832628+0000", 2024-04-23T16:19:32.849 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7146000000000002e-05 2024-04-23T16:19:32.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.850 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.832637+0000", 2024-04-23T16:19:32.850 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9789999999999999e-06 2024-04-23T16:19:32.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.850 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.832720+0000", 2024-04-23T16:19:32.850 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2614999999999994e-05 2024-04-23T16:19:32.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.851 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.832739+0000", 2024-04-23T16:19:32.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9613999999999999e-05 2024-04-23T16:19:32.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.851 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.851 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.863725+0000", 2024-04-23T16:19:32.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.030986198999999999 2024-04-23T16:19:32.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.851 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.851 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.863769+0000", 2024-04-23T16:19:32.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3325999999999998e-05 2024-04-23T16:19:32.851 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.851 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.852 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.852 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:158323 2.a e141/110 2:595f6f13:::benchmark_data_smithi188_77627_object158322:head v 141'9939, mlcod=141'9935)", 2024-04-23T16:19:32.852 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:09.837143+0000", 2024-04-23T16:19:32.852 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.941793057000002, 2024-04-23T16:19:32.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.026650956, 2024-04-23T16:19:32.852 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.852 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.852 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.852 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.852 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.837143+0000", 2024-04-23T16:19:32.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.853 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.853 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.837143+0000", 2024-04-23T16:19:32.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.853 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.853 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.837147+0000", 2024-04-23T16:19:32.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3629999999999998e-06 2024-04-23T16:19:32.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.853 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.837204+0000", 2024-04-23T16:19:32.854 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7562000000000002e-05 2024-04-23T16:19:32.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.837217+0000", 2024-04-23T16:19:32.854 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2337999999999999e-05 2024-04-23T16:19:32.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.837226+0000", 2024-04-23T16:19:32.854 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8689999999999993e-06 2024-04-23T16:19:32.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.855 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.855 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.837277+0000", 2024-04-23T16:19:32.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1209999999999999e-05 2024-04-23T16:19:32.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.855 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.855 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.837295+0000", 2024-04-23T16:19:32.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8709999999999999e-05 2024-04-23T16:19:32.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.855 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.855 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.863772+0000", 2024-04-23T16:19:32.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.026476992000000001 2024-04-23T16:19:32.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.856 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.856 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.856 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:09.863794+0000", 2024-04-23T16:19:32.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1912e-05 2024-04-23T16:19:32.856 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.856 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.856 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.856 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.856 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:159641 2.a e141/110 2:5833d93a:::benchmark_data_smithi188_77627_object159640:head v 141'10016, mlcod=141'10013)", 2024-04-23T16:19:32.856 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:11.084304+0000", 2024-04-23T16:19:32.856 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.694632064, 2024-04-23T16:19:32.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.023104053999999999, 2024-04-23T16:19:32.856 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.857 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.857 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.857 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.857 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.857 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.084304+0000", 2024-04-23T16:19:32.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.857 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.857 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.084304+0000", 2024-04-23T16:19:32.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.857 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.084306+0000", 2024-04-23T16:19:32.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.77e-06 2024-04-23T16:19:32.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.084384+0000", 2024-04-23T16:19:32.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7336999999999995e-05 2024-04-23T16:19:32.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.084395+0000", 2024-04-23T16:19:32.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1164e-05 2024-04-23T16:19:32.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.859 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.859 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.084401+0000", 2024-04-23T16:19:32.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3729999999999998e-06 2024-04-23T16:19:32.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.859 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.859 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.859 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.084452+0000", 2024-04-23T16:19:32.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0559999999999997e-05 2024-04-23T16:19:32.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.859 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.859 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.859 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.084482+0000", 2024-04-23T16:19:32.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0346e-05 2024-04-23T16:19:32.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.107369+0000", 2024-04-23T16:19:32.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.022886749000000001 2024-04-23T16:19:32.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.107408+0000", 2024-04-23T16:19:32.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9755000000000001e-05 2024-04-23T16:19:32.860 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.860 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.860 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.860 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:162234 2.a e141/110 2:5c96f9ef:::benchmark_data_smithi188_77627_object162233:head v 141'10180, mlcod=141'10177)", 2024-04-23T16:19:32.861 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:13.730665+0000", 2024-04-23T16:19:32.861 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.048271190000001, 2024-04-23T16:19:32.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.023062678, 2024-04-23T16:19:32.861 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.861 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.861 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.861 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.730665+0000", 2024-04-23T16:19:32.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.730665+0000", 2024-04-23T16:19:32.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.862 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.730666+0000", 2024-04-23T16:19:32.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2900000000000003e-07 2024-04-23T16:19:32.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.862 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.730722+0000", 2024-04-23T16:19:32.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5476999999999998e-05 2024-04-23T16:19:32.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.862 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.730729+0000", 2024-04-23T16:19:32.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8979999999999996e-06 2024-04-23T16:19:32.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.730735+0000", 2024-04-23T16:19:32.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4430000000000002e-06 2024-04-23T16:19:32.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.735822+0000", 2024-04-23T16:19:32.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050872909999999999 2024-04-23T16:19:32.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.735842+0000", 2024-04-23T16:19:32.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9978000000000001e-05 2024-04-23T16:19:32.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.864 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.864 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.753683+0000", 2024-04-23T16:19:32.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017841342 2024-04-23T16:19:32.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.864 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.864 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.753728+0000", 2024-04-23T16:19:32.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4520000000000001e-05 2024-04-23T16:19:32.864 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.864 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.864 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.864 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:162480 2.e e141/110 2:74a0a079:::benchmark_data_smithi188_77627_object162479:head v 141'10141, mlcod=141'10138)", 2024-04-23T16:19:32.865 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:13.968382+0000", 2024-04-23T16:19:32.865 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.810554004, 2024-04-23T16:19:32.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025446086, 2024-04-23T16:19:32.865 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.865 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.865 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.865 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.968382+0000", 2024-04-23T16:19:32.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.968382+0000", 2024-04-23T16:19:32.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.968384+0000", 2024-04-23T16:19:32.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1939999999999999e-06 2024-04-23T16:19:32.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.968405+0000", 2024-04-23T16:19:32.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1078e-05 2024-04-23T16:19:32.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.968415+0000", 2024-04-23T16:19:32.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0118e-05 2024-04-23T16:19:32.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.968421+0000", 2024-04-23T16:19:32.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5289999999999997e-06 2024-04-23T16:19:32.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.974711+0000", 2024-04-23T16:19:32.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062894520000000001 2024-04-23T16:19:32.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.868 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.868 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.974731+0000", 2024-04-23T16:19:32.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9871e-05 2024-04-23T16:19:32.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.868 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.868 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.993783+0000", 2024-04-23T16:19:32.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019052494999999999 2024-04-23T16:19:32.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.868 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.868 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.993829+0000", 2024-04-23T16:19:32.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5349000000000002e-05 2024-04-23T16:19:32.869 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.869 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.869 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.869 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:164523 2.a e141/110 2:55275484:::benchmark_data_smithi188_77627_object164522:head v 141'10333, mlcod=141'10331)", 2024-04-23T16:19:32.869 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:16.153501+0000", 2024-04-23T16:19:32.869 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.625435676999999, 2024-04-23T16:19:32.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.030638795999999999, 2024-04-23T16:19:32.869 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.869 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.869 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.869 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.153501+0000", 2024-04-23T16:19:32.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.153501+0000", 2024-04-23T16:19:32.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.153503+0000", 2024-04-23T16:19:32.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4739999999999999e-06 2024-04-23T16:19:32.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.153548+0000", 2024-04-23T16:19:32.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4703999999999998e-05 2024-04-23T16:19:32.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.153560+0000", 2024-04-23T16:19:32.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.223e-05 2024-04-23T16:19:32.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.153570+0000", 2024-04-23T16:19:32.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9790000000000005e-06 2024-04-23T16:19:32.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.153739+0000", 2024-04-23T16:19:32.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016832899999999999 2024-04-23T16:19:32.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.153762+0000", 2024-04-23T16:19:32.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3682000000000001e-05 2024-04-23T16:19:32.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.184088+0000", 2024-04-23T16:19:32.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.03032528 2024-04-23T16:19:32.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.184140+0000", 2024-04-23T16:19:32.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2117999999999998e-05 2024-04-23T16:19:32.873 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.873 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.873 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.873 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:164524 2.a e141/110 2:5017d41d:::benchmark_data_smithi188_77627_object164523:head v 141'10334, mlcod=141'10331)", 2024-04-23T16:19:32.873 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:16.154292+0000", 2024-04-23T16:19:32.873 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.624644003, 2024-04-23T16:19:32.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.029876934000000001, 2024-04-23T16:19:32.873 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.873 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.873 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.873 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.874 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.154292+0000", 2024-04-23T16:19:32.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.874 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.874 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.874 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.154292+0000", 2024-04-23T16:19:32.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.874 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.874 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.874 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.154293+0000", 2024-04-23T16:19:32.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9599999999999998e-07 2024-04-23T16:19:32.874 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.874 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.874 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.875 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.154344+0000", 2024-04-23T16:19:32.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0717999999999998e-05 2024-04-23T16:19:32.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.875 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.875 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.154352+0000", 2024-04-23T16:19:32.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5510000000000002e-06 2024-04-23T16:19:32.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.875 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.875 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.154357+0000", 2024-04-23T16:19:32.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5790000000000003e-06 2024-04-23T16:19:32.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.875 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.875 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.163040+0000", 2024-04-23T16:19:32.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086829669999999998 2024-04-23T16:19:32.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.163062+0000", 2024-04-23T16:19:32.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1798999999999999e-05 2024-04-23T16:19:32.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.184144+0000", 2024-04-23T16:19:32.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.021081613999999999 2024-04-23T16:19:32.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.184169+0000", 2024-04-23T16:19:32.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5809999999999999e-05 2024-04-23T16:19:32.877 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.877 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.877 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.877 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:176701 2.e e145/110 2:706fa2cc:::benchmark_data_smithi188_77627_object7352:head v 145'11030, mlcod=145'11028)", 2024-04-23T16:19:32.877 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:28.555849+0000", 2024-04-23T16:19:32.877 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.223087992, 2024-04-23T16:19:32.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.023700704, 2024-04-23T16:19:32.877 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.877 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.877 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.877 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.555849+0000", 2024-04-23T16:19:32.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.555849+0000", 2024-04-23T16:19:32.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.555850+0000", 2024-04-23T16:19:32.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4619999999999999e-06 2024-04-23T16:19:32.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.555859+0000", 2024-04-23T16:19:32.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.482e-06 2024-04-23T16:19:32.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.555874+0000", 2024-04-23T16:19:32.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4854e-05 2024-04-23T16:19:32.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.555883+0000", 2024-04-23T16:19:32.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8890000000000008e-06 2024-04-23T16:19:32.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.880 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.880 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.570130+0000", 2024-04-23T16:19:32.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014247176 2024-04-23T16:19:32.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.880 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.880 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.570153+0000", 2024-04-23T16:19:32.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3098000000000001e-05 2024-04-23T16:19:32.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.880 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.880 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.579499+0000", 2024-04-23T16:19:32.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.009345875 2024-04-23T16:19:32.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.881 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.881 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.579549+0000", 2024-04-23T16:19:32.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9867999999999998e-05 2024-04-23T16:19:32.881 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.881 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.881 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.881 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:176959 2.1 e145/110 2:87194917:::benchmark_data_smithi188_77627_object7610:head v 145'11000, mlcod=145'10998)", 2024-04-23T16:19:32.881 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:28.813467+0000", 2024-04-23T16:19:32.881 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.9654694450000001, 2024-04-23T16:19:32.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024829653, 2024-04-23T16:19:32.882 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.882 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.882 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.882 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.813467+0000", 2024-04-23T16:19:32.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.813467+0000", 2024-04-23T16:19:32.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.813469+0000", 2024-04-23T16:19:32.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.81e-06 2024-04-23T16:19:32.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.813479+0000", 2024-04-23T16:19:32.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6970000000000005e-06 2024-04-23T16:19:32.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.813490+0000", 2024-04-23T16:19:32.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1291000000000001e-05 2024-04-23T16:19:32.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.813496+0000", 2024-04-23T16:19:32.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2979999999999997e-06 2024-04-23T16:19:32.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.813578+0000", 2024-04-23T16:19:32.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1666e-05 2024-04-23T16:19:32.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.813600+0000", 2024-04-23T16:19:32.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2337e-05 2024-04-23T16:19:32.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.838234+0000", 2024-04-23T16:19:32.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024633966 2024-04-23T16:19:32.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:28.838297+0000", 2024-04-23T16:19:32.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2588000000000005e-05 2024-04-23T16:19:32.885 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.885 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.885 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.885 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:179660 2.e e145/110 2:757d3756:::benchmark_data_smithi188_77627_object10311:head v 145'11231, mlcod=145'11229)", 2024-04-23T16:19:32.885 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:31.267547+0000", 2024-04-23T16:19:32.886 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.511389849, 2024-04-23T16:19:32.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.031556358999999999, 2024-04-23T16:19:32.886 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.886 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.886 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.886 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.267547+0000", 2024-04-23T16:19:32.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.267547+0000", 2024-04-23T16:19:32.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.267549+0000", 2024-04-23T16:19:32.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.26e-06 2024-04-23T16:19:32.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.267560+0000", 2024-04-23T16:19:32.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1372999999999999e-05 2024-04-23T16:19:32.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.267578+0000", 2024-04-23T16:19:32.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8108e-05 2024-04-23T16:19:32.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.267588+0000", 2024-04-23T16:19:32.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8570000000000007e-06 2024-04-23T16:19:32.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.270648+0000", 2024-04-23T16:19:32.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030598330000000001 2024-04-23T16:19:32.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.270670+0000", 2024-04-23T16:19:32.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2325999999999999e-05 2024-04-23T16:19:32.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.299062+0000", 2024-04-23T16:19:32.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.028391939000000001 2024-04-23T16:19:32.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.299103+0000", 2024-04-23T16:19:32.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0663e-05 2024-04-23T16:19:32.890 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.890 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.890 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.890 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:179661 2.e e145/110 2:7e91f4c0:::benchmark_data_smithi188_77627_object10312:head v 145'11232, mlcod=145'11229)", 2024-04-23T16:19:32.891 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:31.271481+0000", 2024-04-23T16:19:32.891 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.5074555620000001, 2024-04-23T16:19:32.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.027650022, 2024-04-23T16:19:32.891 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.891 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.891 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.891 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.271481+0000", 2024-04-23T16:19:32.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.271481+0000", 2024-04-23T16:19:32.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.271483+0000", 2024-04-23T16:19:32.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2220000000000001e-06 2024-04-23T16:19:32.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.271495+0000", 2024-04-23T16:19:32.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1857e-05 2024-04-23T16:19:32.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.271507+0000", 2024-04-23T16:19:32.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1888999999999999e-05 2024-04-23T16:19:32.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.271515+0000", 2024-04-23T16:19:32.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4549999999999995e-06 2024-04-23T16:19:32.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.275077+0000", 2024-04-23T16:19:32.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035621020000000002 2024-04-23T16:19:32.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.275095+0000", 2024-04-23T16:19:32.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7621999999999998e-05 2024-04-23T16:19:32.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.299106+0000", 2024-04-23T16:19:32.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024010480000000001 2024-04-23T16:19:32.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.299131+0000", 2024-04-23T16:19:32.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5395e-05 2024-04-23T16:19:32.895 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.895 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.895 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.895 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:179678 2.e e145/110 2:7df21e4e:::benchmark_data_smithi188_77627_object10329:head v 145'11233, mlcod=145'11229)", 2024-04-23T16:19:32.895 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:31.273578+0000", 2024-04-23T16:19:32.895 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.505358494, 2024-04-23T16:19:32.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025578045000000001, 2024-04-23T16:19:32.895 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.895 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.895 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.895 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.273578+0000", 2024-04-23T16:19:32.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.273578+0000", 2024-04-23T16:19:32.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.273579+0000", 2024-04-23T16:19:32.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0399999999999996e-07 2024-04-23T16:19:32.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.273583+0000", 2024-04-23T16:19:32.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5990000000000004e-06 2024-04-23T16:19:32.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.273588+0000", 2024-04-23T16:19:32.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3800000000000002e-06 2024-04-23T16:19:32.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.273593+0000", 2024-04-23T16:19:32.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8760000000000004e-06 2024-04-23T16:19:32.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.898 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.898 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.279235+0000", 2024-04-23T16:19:32.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056415659999999998 2024-04-23T16:19:32.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.898 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.898 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.279255+0000", 2024-04-23T16:19:32.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9911e-05 2024-04-23T16:19:32.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.898 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.898 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.299133+0000", 2024-04-23T16:19:32.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019878271999999999 2024-04-23T16:19:32.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:31.299156+0000", 2024-04-23T16:19:32.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2937000000000001e-05 2024-04-23T16:19:32.899 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.899 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.899 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.899 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:180649 2.a e146/110 2:55b8f190:::benchmark_data_smithi188_77627_object11300:head v 146'11366, mlcod=146'11362)", 2024-04-23T16:19:32.899 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:32.186963+0000", 2024-04-23T16:19:32.899 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.59197379000000006, 2024-04-23T16:19:32.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024851120000000001, 2024-04-23T16:19:32.900 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.900 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.900 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.900 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.186963+0000", 2024-04-23T16:19:32.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.186963+0000", 2024-04-23T16:19:32.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.186964+0000", 2024-04-23T16:19:32.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.229e-06 2024-04-23T16:19:32.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.186971+0000", 2024-04-23T16:19:32.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5969999999999997e-06 2024-04-23T16:19:32.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.186979+0000", 2024-04-23T16:19:32.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1650000000000006e-06 2024-04-23T16:19:32.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.186986+0000", 2024-04-23T16:19:32.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7589999999999997e-06 2024-04-23T16:19:32.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.190549+0000", 2024-04-23T16:19:32.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003562122 2024-04-23T16:19:32.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.190570+0000", 2024-04-23T16:19:32.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1654999999999998e-05 2024-04-23T16:19:32.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.903 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.903 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.211766+0000", 2024-04-23T16:19:32.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.021195986999999999 2024-04-23T16:19:32.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.903 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.903 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.211814+0000", 2024-04-23T16:19:32.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7605999999999998e-05 2024-04-23T16:19:32.903 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.903 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.903 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.904 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:180937 2.e e146/110 2:780fa29e:::benchmark_data_smithi188_77627_object11588:head v 146'11319, mlcod=146'11317)", 2024-04-23T16:19:32.904 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:32.457325+0000", 2024-04-23T16:19:32.904 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.32161125499999998, 2024-04-23T16:19:32.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025967589999999999, 2024-04-23T16:19:32.904 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:19:32.904 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:19:32.904 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:19:32.904 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:19:32.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:19:32.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.457325+0000", 2024-04-23T16:19:32.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:19:32.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.457325+0000", 2024-04-23T16:19:32.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:19:32.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:19:32.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.457327+0000", 2024-04-23T16:19:32.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6929999999999999e-06 2024-04-23T16:19:32.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:19:32.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.457336+0000", 2024-04-23T16:19:32.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1689999999999998e-06 2024-04-23T16:19:32.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:19:32.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.457351+0000", 2024-04-23T16:19:32.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4409999999999999e-05 2024-04-23T16:19:32.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:19:32.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.457358+0000", 2024-04-23T16:19:32.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8809999999999996e-06 2024-04-23T16:19:32.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:19:32.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.462064+0000", 2024-04-23T16:19:32.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004705351 2024-04-23T16:19:32.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:19:32.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.462083+0000", 2024-04-23T16:19:32.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9743999999999999e-05 2024-04-23T16:19:32.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:19:32.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.483250+0000", 2024-04-23T16:19:32.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.021166767 2024-04-23T16:19:32.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:32.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:32.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:19:32.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:32.483293+0000", 2024-04-23T16:19:32.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2574999999999998e-05 2024-04-23T16:19:32.908 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.908 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.908 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.908 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:32.908 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:32.908 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:19:32.909 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:32 smithi148 ceph-mon[44520]: osdmap e146: 12 total, 12 up, 12 in 2024-04-23T16:19:32.909 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:19:32.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:32 smithi188 ceph-mon[44101]: osdmap e146: 12 total, 12 up, 12 in 2024-04-23T16:19:32.982 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:19:33.359 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:19:33.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:33.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:19:33.464 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:19:33.517 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:33.659 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:19:33.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:33.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:19:33.667 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:33 smithi103 ceph-mon[39629]: pgmap v445: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 11 MiB/s wr, 986 op/s 2024-04-23T16:19:33.765 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:19:33.815 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:33.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:33 smithi148 ceph-mon[44520]: pgmap v445: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 11 MiB/s wr, 986 op/s 2024-04-23T16:19:33.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:33 smithi188 ceph-mon[44101]: pgmap v445: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 11 MiB/s wr, 986 op/s 2024-04-23T16:19:34.318 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:19:34.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:34.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:19:34.352 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-23T16:19:34.423 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:19:34.457 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:34.482 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:19:34.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:34.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:19:34.588 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:19:34.634 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:34.696 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:19:34.817 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:19:34.830 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:34 smithi148 ceph-mon[44520]: pgmap v446: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s wr, 1.05k op/s 2024-04-23T16:19:34.841 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:19:34.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:34.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:19:34.948 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:19:34.984 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:35.007 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:19:35.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:35.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:19:35.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:34 smithi103 ceph-mon[39629]: pgmap v446: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s wr, 1.05k op/s 2024-04-23T16:19:35.113 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:19:35.148 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:35.151 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:34 smithi188 ceph-mon[44101]: pgmap v446: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s wr, 1.05k op/s 2024-04-23T16:19:35.175 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:19:35.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:35.181 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:19:35.282 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:19:35.311 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:37.166 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:19:37.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:37.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:19:37.271 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:19:37.299 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:37.323 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:19:37.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:37.326 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:19:37.427 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:19:37.461 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:37.500 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:19:37.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:37.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:19:37.536 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:37 smithi103 ceph-mon[39629]: pgmap v447: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:19:37.597 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-04-23T16:19:37.598 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-04-23T16:19:37.598 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-04-23T16:19:37.598 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-04-23T16:19:37.598 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-04-23T16:19:37.598 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-04-23T16:19:37.598 INFO:teuthology.orchestra.run.smithi103.stderr:bench [] [] [] [] 2024-04-23T16:19:37.598 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs debug_inject_read_zeros 2024-04-23T16:19:37.598 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs files list 2024-04-23T16:19:37.598 INFO:teuthology.orchestra.run.smithi103.stderr:bluefs stats 2024-04-23T16:19:37.598 INFO:teuthology.orchestra.run.smithi103.stderr:bluestore allocator dump block 2024-04-23T16:19:37.598 INFO:teuthology.orchestra.run.smithi103.stderr:Error EINVAL: invalid command 2024-04-23T16:19:37.606 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:19:37.643 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:37.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:37 smithi188 ceph-mon[44101]: pgmap v447: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:19:37.798 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:19:37.799 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:37 smithi148 ceph-mon[44520]: pgmap v447: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:19:37.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:37.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:19:37.902 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:19:37.937 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:37.966 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:19:37.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:37.971 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:19:38.072 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:19:38.100 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:38.122 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:19:38.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:38.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:19:38.227 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:19:38.269 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:38.337 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:19:38.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:38.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:19:38.442 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:19:38.480 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:38.507 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:19:38.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:38.512 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:19:38.613 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:19:38.644 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:38.676 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:19:38.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:38.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:19:38.782 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:19:38.811 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:38.837 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:19:38.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:38.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:19:38.941 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:19:38.969 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:38.971 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:19:38.972 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:19:38.972 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:19:38.972 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2024-04-23T16:19:38.972 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:19:38.988 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:19:38.988 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:19:38.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:38.994 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:19:39.027 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:19:39.059 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:19:39.093 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:19:39.095 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:19:39.132 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:19:39.164 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:19:39.179 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:39.209 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:19:39.243 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:19:39.272 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:19:39.301 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:19:39.337 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:19:39.350 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:19:39.350 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:19:39.350 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:19:39.350 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:19:39.350 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:19:39.350 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:19:39.351 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:19:39.351 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:19:39.351 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:19:39.351 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:19:39.351 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:19:39.351 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:19:39.548 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:39 smithi103 ceph-mon[39629]: pgmap v448: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:19:39.548 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:39 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2469741382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:19:39.548 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:39.565 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:19:39.678 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:39 smithi148 ceph-mon[44520]: pgmap v448: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:19:39.678 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:39 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2469741382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:19:39.678 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:39.678 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:39.678 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:39 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:19:39.678 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:39 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:19:39.678 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:39.678 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:39.678 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:39 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2856461115' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:19:39.678 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:39 smithi148 ceph-mon[44520]: from='client.24832 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:19:39.678 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:39.713 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-23T16:19:39.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:39 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:19:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:39 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:19:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:39 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2856461115' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:19:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:39 smithi103 ceph-mon[39629]: from='client.24832 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:19:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:39 smithi188 ceph-mon[44101]: pgmap v448: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:19:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:39 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2469741382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:19:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:39.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:39.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:39 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:19:39.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:39 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:19:39.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:39.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:39.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:39 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2856461115' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:19:39.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:39 smithi188 ceph-mon[44101]: from='client.24832 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:19:39.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:40.035 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:19:40.069 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:19:40.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:40.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:19:40.173 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:19:40.206 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:40.309 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:19:40.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:40.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:19:40.417 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:19:40.467 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:40.492 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:19:40.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:40.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:19:40.570 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T16:19:40.571 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-23T16:19:40.600 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:19:40.632 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:40.662 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:19:40.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:40.669 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:19:40.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:40.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:40.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:40.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:40.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:40.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:40.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:40 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:19:40.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:40 smithi188 ceph-mon[44101]: from='client.24832 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:19:40.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:40 smithi188 ceph-mon[44101]: osdmap e147: 12 total, 12 up, 12 in 2024-04-23T16:19:40.770 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:19:40.807 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:40 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:19:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:40 smithi103 ceph-mon[39629]: from='client.24832 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:19:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:40 smithi103 ceph-mon[39629]: osdmap e147: 12 total, 12 up, 12 in 2024-04-23T16:19:40.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:40.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:40.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:40.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:40.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:40.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:40.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:40 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:19:40.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:40 smithi148 ceph-mon[44520]: from='client.24832 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:19:40.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:40 smithi148 ceph-mon[44520]: osdmap e147: 12 total, 12 up, 12 in 2024-04-23T16:19:40.881 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:19:40.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:40.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:19:40.985 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:19:41.030 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:41.059 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:19:41.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:41.064 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:19:41.165 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:19:41.199 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:41.568 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:19:41.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:41.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:19:41.686 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:19:41.704 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:41 smithi148 ceph-mon[44520]: pgmap v450: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-04-23T16:19:41.722 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:41.752 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:19:41.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:41.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:19:41.856 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:19:41.857 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:41 smithi103 ceph-mon[39629]: pgmap v450: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-04-23T16:19:41.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:41 smithi188 ceph-mon[44101]: pgmap v450: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-04-23T16:19:41.893 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:41.924 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:19:41.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:41.929 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:19:41.995 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:19:41.996 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:19:42.030 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:19:42.089 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:42.126 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:19:42.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:42.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:19:42.230 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:19:42.264 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:42.295 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:19:42.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:42.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:19:42.407 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:19:42.450 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:42.487 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:19:42.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:42.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:19:42.562 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:19:42.595 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:19:42.624 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:42.722 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:19:42.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:42.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:19:42.739 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-23T16:19:42.827 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:19:42.865 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:42.889 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:19:42.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:42.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:19:42.994 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:19:42.996 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:19:43.031 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:43.063 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:19:43.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:43.066 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:19:43.106 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:19:43.167 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:19:43.211 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:43.238 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:19:43.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:43.243 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:19:43.344 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:19:43.389 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:43.423 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:19:43.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:43.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:19:43.529 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:19:43.562 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:43.589 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:19:43.856 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:43 smithi103 ceph-mon[39629]: pgmap v451: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:19:43.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:43 smithi148 ceph-mon[44520]: pgmap v451: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:19:43.926 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:19:43.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:43.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:19:43.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:43 smithi188 ceph-mon[44101]: pgmap v451: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:19:44.033 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:19:44.067 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:44.421 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:19:44.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:44.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:19:44.466 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:19:44.530 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:19:44.583 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:44.815 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:19:44.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:44.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:19:44.921 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:19:44.955 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:44.983 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:19:44.983 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:19:44.983 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:19:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:44.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:19:45.087 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:19:45.124 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:45.157 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:19:45.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:45.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:19:45.262 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:19:45.306 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:45.353 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:19:45.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:45.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:19:45.467 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:19:45.504 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:45.572 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-23T16:19:45.572 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:19:45.572 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-23T16:19:45.572 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-23T16:19:45.595 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:45 smithi103 ceph-mon[39629]: pgmap v452: 17 pgs: 17 active+clean; 9.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-04-23T16:19:45.600 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:19:45.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:45.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:19:45.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:45 smithi148 ceph-mon[44520]: pgmap v452: 17 pgs: 17 active+clean; 9.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-04-23T16:19:45.691 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-23T16:19:45.706 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:19:45.738 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:45.765 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:19:45.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:45.768 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:19:45.869 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:19:45.898 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:45 smithi188 ceph-mon[44101]: pgmap v452: 17 pgs: 17 active+clean; 9.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-04-23T16:19:45.901 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:45.923 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:19:45.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:45.929 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:19:46.020 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:19:46.031 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:19:46.083 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:46.208 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:19:46.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:46.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:19:46.313 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:19:46.360 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:46.820 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:19:46.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:46.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:19:46.925 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:19:46.961 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:46.993 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:19:46.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:46.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:19:47.097 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:19:47.138 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:47.619 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:47 smithi148 ceph-mon[44520]: pgmap v453: 17 pgs: 17 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:19:47.619 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:47 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1855226118' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:19:47.619 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:47 smithi148 ceph-mon[44520]: from='client.34862 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:19:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:47 smithi188 ceph-mon[44101]: pgmap v453: 17 pgs: 17 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:19:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:47 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1855226118' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:19:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:47 smithi188 ceph-mon[44101]: from='client.34862 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:19:47.985 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:19:47.986 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:19:48.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:47 smithi103 ceph-mon[39629]: pgmap v453: 17 pgs: 17 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:19:48.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:47 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1855226118' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:19:48.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:47 smithi103 ceph-mon[39629]: from='client.34862 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:19:48.494 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:19:48.691 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-23T16:19:48.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:48 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:19:48.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:48 smithi188 ceph-mon[44101]: from='client.34862 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:19:48.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:48 smithi188 ceph-mon[44101]: osdmap e148: 12 total, 12 up, 12 in 2024-04-23T16:19:49.026 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:19:49.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:48 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:19:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:48 smithi103 ceph-mon[39629]: from='client.34862 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:19:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:48 smithi103 ceph-mon[39629]: osdmap e148: 12 total, 12 up, 12 in 2024-04-23T16:19:49.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:48 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:19:49.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:48 smithi148 ceph-mon[44520]: from='client.34862 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:19:49.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:48 smithi148 ceph-mon[44520]: osdmap e148: 12 total, 12 up, 12 in 2024-04-23T16:19:49.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:49 smithi148 ceph-mon[44520]: pgmap v455: 17 pgs: 17 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:19:49.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:49 smithi188 ceph-mon[44101]: pgmap v455: 17 pgs: 17 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:19:50.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:49 smithi103 ceph-mon[39629]: pgmap v455: 17 pgs: 17 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:19:50.717 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:19:50.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:50.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:19:50.823 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:19:50.859 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:19:50.859 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:19:50.864 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:51.525 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-23T16:19:51.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:51 smithi148 ceph-mon[44520]: pgmap v456: 17 pgs: 17 active+clean; 8.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:19:51.878 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:19:51.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:51 smithi188 ceph-mon[44101]: pgmap v456: 17 pgs: 17 active+clean; 8.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:19:52.030 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:51 smithi103 ceph-mon[39629]: pgmap v456: 17 pgs: 17 active+clean; 8.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:19:52.664 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:52 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:19:52.664 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:52 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:19:52.664 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:52 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:19:52.952 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:52 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:19:52.952 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:52 smithi148 ceph-mon[44520]: osdmap e149: 12 total, 12 up, 12 in 2024-04-23T16:19:52.952 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:52 smithi148 ceph-mon[44520]: pgmap v458: 17 pgs: 17 active+clean; 8.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-04-23T16:19:52.967 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:52 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:19:52.967 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:52 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:19:52.967 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:52 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:19:52.967 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:52 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:19:52.967 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:52 smithi103 ceph-mon[39629]: osdmap e149: 12 total, 12 up, 12 in 2024-04-23T16:19:52.968 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:52 smithi103 ceph-mon[39629]: pgmap v458: 17 pgs: 17 active+clean; 8.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-04-23T16:19:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:52 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:19:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:52 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:19:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:52 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:19:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:52 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:19:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:52 smithi188 ceph-mon[44101]: osdmap e149: 12 total, 12 up, 12 in 2024-04-23T16:19:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:52 smithi188 ceph-mon[44101]: pgmap v458: 17 pgs: 17 active+clean; 8.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-04-23T16:19:53.245 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:19:53.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:53.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:19:53.251 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:19:53.353 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:19:53.386 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:53.588 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:19:53.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:53.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:19:53.699 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:19:53.728 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:53.747 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:19:53.760 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:19:53.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:53.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:19:53.776 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:19:53.810 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:19:53.852 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:19:53.861 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:19:53.861 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:19:53.873 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:19:53.893 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:19:53.919 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:53.927 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:19:53.961 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:19:53.993 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:19:54.031 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:19:54.068 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:19:54.075 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:19:54.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:54.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:19:54.101 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:19:54.142 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:19:54.144 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:19:54.144 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:19:54.144 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:19:54.144 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:19:54.144 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:19:54.144 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:19:54.145 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:19:54.145 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:19:54.145 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:19:54.145 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:19:54.145 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:19:54.145 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:19:54.179 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:19:54.212 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:54.560 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-23T16:19:54.568 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:19:54.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:54.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:19:54.673 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:19:54.723 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:54.749 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:19:54.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:54.754 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:19:54.855 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:19:54.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:54 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:19:54.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:54 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:19:54.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:54 smithi148 ceph-mon[44520]: osdmap e150: 12 total, 12 up, 12 in 2024-04-23T16:19:54.878 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:54 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4221818188' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:19:54.878 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.878 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.878 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:54 smithi148 ceph-mon[44520]: 2.2 scrub starts 2024-04-23T16:19:54.878 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.878 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.878 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.878 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.878 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.878 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.878 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.878 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.878 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:54 smithi148 ceph-mon[44520]: pgmap v460: 17 pgs: 17 active+clean; 8.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.44k op/s 2024-04-23T16:19:54.898 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:54.920 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:19:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:54 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:19:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:54 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:19:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:54 smithi188 ceph-mon[44101]: osdmap e150: 12 total, 12 up, 12 in 2024-04-23T16:19:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:54 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4221818188' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:19:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:54 smithi188 ceph-mon[44101]: 2.2 scrub starts 2024-04-23T16:19:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:54.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:54 smithi188 ceph-mon[44101]: pgmap v460: 17 pgs: 17 active+clean; 8.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.44k op/s 2024-04-23T16:19:55.005 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:19:55.053 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:54 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:19:55.053 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:54 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:19:55.053 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:54 smithi103 ceph-mon[39629]: osdmap e150: 12 total, 12 up, 12 in 2024-04-23T16:19:55.053 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:54 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4221818188' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:19:55.053 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:55.053 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:55.053 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:54 smithi103 ceph-mon[39629]: 2.2 scrub starts 2024-04-23T16:19:55.053 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:55.054 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:55.054 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:55.054 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:55.054 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:55.054 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:55.054 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:55.054 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:55.054 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:54 smithi103 ceph-mon[39629]: pgmap v460: 17 pgs: 17 active+clean; 8.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.44k op/s 2024-04-23T16:19:55.263 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:19:55.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:55.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:19:55.368 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:19:55.405 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:55.439 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:19:55.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:55.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:19:55.545 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:19:55.583 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:55.610 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:19:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:55.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:19:55.684 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:55.684 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:55.719 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:19:55.766 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:55.799 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:19:55.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:55.803 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:19:55.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:55.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:55.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:55.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:19:55.874 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:19:55.904 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:19:55.977 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:56.062 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:19:56.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:56.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:19:56.175 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:19:56.203 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:56.280 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:19:56.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:56.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:19:56.388 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:19:56.439 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:56.491 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:19:56.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:56.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:19:56.599 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:19:56.643 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:56.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:56 smithi103 ceph-mon[39629]: 2.2 scrub ok 2024-04-23T16:19:56.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:56 smithi103 ceph-mon[39629]: pgmap v461: 17 pgs: 17 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.34k op/s 2024-04-23T16:19:56.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:56 smithi148 ceph-mon[44520]: 2.2 scrub ok 2024-04-23T16:19:56.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:56 smithi148 ceph-mon[44520]: pgmap v461: 17 pgs: 17 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.34k op/s 2024-04-23T16:19:56.885 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:19:56.885 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:19:56.885 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2024-04-23T16:19:56.885 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:19:56.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:56 smithi188 ceph-mon[44101]: 2.2 scrub ok 2024-04-23T16:19:56.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:56 smithi188 ceph-mon[44101]: pgmap v461: 17 pgs: 17 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.34k op/s 2024-04-23T16:19:57.108 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:19:57.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:57.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:19:57.215 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:19:57.232 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:19:57.232 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:19:57.232 INFO:teuthology.orchestra.run.smithi103.stdout: "epoch": 150, 2024-04-23T16:19:57.233 INFO:teuthology.orchestra.run.smithi103.stdout: "fsid": "8e3594f2-018b-11ef-bc93-c7b262605968", 2024-04-23T16:19:57.233 INFO:teuthology.orchestra.run.smithi103.stdout: "created": "2024-04-23T16:08:13.477590+0000", 2024-04-23T16:19:57.233 INFO:teuthology.orchestra.run.smithi103.stdout: "modified": "2024-04-23T16:19:53.663018+0000", 2024-04-23T16:19:57.233 INFO:teuthology.orchestra.run.smithi103.stdout: "last_up_change": "2024-04-23T16:15:34.871303+0000", 2024-04-23T16:19:57.233 INFO:teuthology.orchestra.run.smithi103.stdout: "last_in_change": "2024-04-23T16:15:17.238203+0000", 2024-04-23T16:19:57.233 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-23T16:19:57.233 INFO:teuthology.orchestra.run.smithi103.stdout: "flags_num": 5804032, 2024-04-23T16:19:57.233 INFO:teuthology.orchestra.run.smithi103.stdout: "flags_set": [ 2024-04-23T16:19:57.233 INFO:teuthology.orchestra.run.smithi103.stdout: "nodeep-scrub", 2024-04-23T16:19:57.233 INFO:teuthology.orchestra.run.smithi103.stdout: "pglog_hardlimit", 2024-04-23T16:19:57.233 INFO:teuthology.orchestra.run.smithi103.stdout: "purged_snapdirs", 2024-04-23T16:19:57.233 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_deletes", 2024-04-23T16:19:57.233 INFO:teuthology.orchestra.run.smithi103.stdout: "sortbitwise" 2024-04-23T16:19:57.234 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:19:57.234 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_version": 27, 2024-04-23T16:19:57.234 INFO:teuthology.orchestra.run.smithi103.stdout: "full_ratio": 0.94999998807907104, 2024-04-23T16:19:57.234 INFO:teuthology.orchestra.run.smithi103.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-23T16:19:57.234 INFO:teuthology.orchestra.run.smithi103.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-23T16:19:57.234 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_snapshot": "", 2024-04-23T16:19:57.234 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_max": 2, 2024-04-23T16:19:57.234 INFO:teuthology.orchestra.run.smithi103.stdout: "max_osd": 12, 2024-04-23T16:19:57.234 INFO:teuthology.orchestra.run.smithi103.stdout: "require_min_compat_client": "luminous", 2024-04-23T16:19:57.234 INFO:teuthology.orchestra.run.smithi103.stdout: "min_compat_client": "jewel", 2024-04-23T16:19:57.234 INFO:teuthology.orchestra.run.smithi103.stdout: "require_osd_release": "squid", 2024-04-23T16:19:57.234 INFO:teuthology.orchestra.run.smithi103.stdout: "allow_crimson": false, 2024-04-23T16:19:57.234 INFO:teuthology.orchestra.run.smithi103.stdout: "pools": [ 2024-04-23T16:19:57.234 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.234 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-04-23T16:19:57.235 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_name": ".mgr", 2024-04-23T16:19:57.235 INFO:teuthology.orchestra.run.smithi103.stdout: "create_time": "2024-04-23T16:11:22.416777+0000", 2024-04-23T16:19:57.235 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-04-23T16:19:57.235 INFO:teuthology.orchestra.run.smithi103.stdout: "flags_names": "hashpspool", 2024-04-23T16:19:57.235 INFO:teuthology.orchestra.run.smithi103.stdout: "type": 1, 2024-04-23T16:19:57.235 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 2, 2024-04-23T16:19:57.235 INFO:teuthology.orchestra.run.smithi103.stdout: "min_size": 2, 2024-04-23T16:19:57.235 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_rule": 0, 2024-04-23T16:19:57.235 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_count": 0, 2024-04-23T16:19:57.235 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_target": 0, 2024-04-23T16:19:57.235 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T16:19:57.235 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T16:19:57.235 INFO:teuthology.orchestra.run.smithi103.stdout: "object_hash": 2, 2024-04-23T16:19:57.235 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_autoscale_mode": "off", 2024-04-23T16:19:57.235 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num": 1, 2024-04-23T16:19:57.236 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_placement_num": 1, 2024-04-23T16:19:57.236 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_placement_num_target": 1, 2024-04-23T16:19:57.236 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_target": 1, 2024-04-23T16:19:57.236 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_pending": 1, 2024-04-23T16:19:57.236 INFO:teuthology.orchestra.run.smithi103.stdout: "last_pg_merge_meta": { 2024-04-23T16:19:57.236 INFO:teuthology.orchestra.run.smithi103.stdout: "source_pgid": "0.0", 2024-04-23T16:19:57.236 INFO:teuthology.orchestra.run.smithi103.stdout: "ready_epoch": 0, 2024-04-23T16:19:57.236 INFO:teuthology.orchestra.run.smithi103.stdout: "last_epoch_started": 0, 2024-04-23T16:19:57.236 INFO:teuthology.orchestra.run.smithi103.stdout: "last_epoch_clean": 0, 2024-04-23T16:19:57.236 INFO:teuthology.orchestra.run.smithi103.stdout: "source_version": "0'0", 2024-04-23T16:19:57.236 INFO:teuthology.orchestra.run.smithi103.stdout: "target_version": "0'0" 2024-04-23T16:19:57.236 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.236 INFO:teuthology.orchestra.run.smithi103.stdout: "last_change": "17", 2024-04-23T16:19:57.236 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend": "0", 2024-04-23T16:19:57.236 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T16:19:57.237 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T16:19:57.237 INFO:teuthology.orchestra.run.smithi103.stdout: "auid": 0, 2024-04-23T16:19:57.237 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_mode": "selfmanaged", 2024-04-23T16:19:57.237 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_seq": 0, 2024-04-23T16:19:57.237 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_epoch": 0, 2024-04-23T16:19:57.237 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_snaps": [], 2024-04-23T16:19:57.237 INFO:teuthology.orchestra.run.smithi103.stdout: "removed_snaps": "[]", 2024-04-23T16:19:57.237 INFO:teuthology.orchestra.run.smithi103.stdout: "quota_max_bytes": 0, 2024-04-23T16:19:57.237 INFO:teuthology.orchestra.run.smithi103.stdout: "quota_max_objects": 0, 2024-04-23T16:19:57.237 INFO:teuthology.orchestra.run.smithi103.stdout: "tiers": [], 2024-04-23T16:19:57.237 INFO:teuthology.orchestra.run.smithi103.stdout: "tier_of": -1, 2024-04-23T16:19:57.237 INFO:teuthology.orchestra.run.smithi103.stdout: "read_tier": -1, 2024-04-23T16:19:57.237 INFO:teuthology.orchestra.run.smithi103.stdout: "write_tier": -1, 2024-04-23T16:19:57.237 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_mode": "none", 2024-04-23T16:19:57.237 INFO:teuthology.orchestra.run.smithi103.stdout: "target_max_bytes": 0, 2024-04-23T16:19:57.238 INFO:teuthology.orchestra.run.smithi103.stdout: "target_max_objects": 0, 2024-04-23T16:19:57.238 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T16:19:57.238 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T16:19:57.238 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T16:19:57.238 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_min_flush_age": 0, 2024-04-23T16:19:57.238 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_min_evict_age": 0, 2024-04-23T16:19:57.238 INFO:teuthology.orchestra.run.smithi103.stdout: "erasure_code_profile": "", 2024-04-23T16:19:57.238 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_params": { 2024-04-23T16:19:57.238 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "none" 2024-04-23T16:19:57.238 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.238 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_period": 0, 2024-04-23T16:19:57.238 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_count": 0, 2024-04-23T16:19:57.238 INFO:teuthology.orchestra.run.smithi103.stdout: "use_gmt_hitset": true, 2024-04-23T16:19:57.238 INFO:teuthology.orchestra.run.smithi103.stdout: "min_read_recency_for_promote": 0, 2024-04-23T16:19:57.239 INFO:teuthology.orchestra.run.smithi103.stdout: "min_write_recency_for_promote": 0, 2024-04-23T16:19:57.239 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T16:19:57.239 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_search_last_n": 0, 2024-04-23T16:19:57.239 INFO:teuthology.orchestra.run.smithi103.stdout: "grade_table": [], 2024-04-23T16:19:57.239 INFO:teuthology.orchestra.run.smithi103.stdout: "stripe_width": 0, 2024-04-23T16:19:57.239 INFO:teuthology.orchestra.run.smithi103.stdout: "expected_num_objects": 0, 2024-04-23T16:19:57.239 INFO:teuthology.orchestra.run.smithi103.stdout: "fast_read": false, 2024-04-23T16:19:57.239 INFO:teuthology.orchestra.run.smithi103.stdout: "options": { 2024-04-23T16:19:57.239 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_max": 32, 2024-04-23T16:19:57.239 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_min": 1 2024-04-23T16:19:57.239 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.239 INFO:teuthology.orchestra.run.smithi103.stdout: "application_metadata": { 2024-04-23T16:19:57.239 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": {} 2024-04-23T16:19:57.239 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.239 INFO:teuthology.orchestra.run.smithi103.stdout: "read_balance": { 2024-04-23T16:19:57.240 INFO:teuthology.orchestra.run.smithi103.stdout: "score_type": "Fair distribution", 2024-04-23T16:19:57.240 INFO:teuthology.orchestra.run.smithi103.stdout: "score_acting": 11.760000228881836, 2024-04-23T16:19:57.240 INFO:teuthology.orchestra.run.smithi103.stdout: "score_stable": 11.760000228881836, 2024-04-23T16:19:57.240 INFO:teuthology.orchestra.run.smithi103.stdout: "optimal_score": 0.17000000178813934, 2024-04-23T16:19:57.240 INFO:teuthology.orchestra.run.smithi103.stdout: "raw_score_acting": 2, 2024-04-23T16:19:57.240 INFO:teuthology.orchestra.run.smithi103.stdout: "raw_score_stable": 2, 2024-04-23T16:19:57.240 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-04-23T16:19:57.240 INFO:teuthology.orchestra.run.smithi103.stdout: "average_primary_affinity": 1, 2024-04-23T16:19:57.240 INFO:teuthology.orchestra.run.smithi103.stdout: "average_primary_affinity_weighted": 1 2024-04-23T16:19:57.240 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.240 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.240 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.240 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 2, 2024-04-23T16:19:57.240 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_name": "unique_pool_0", 2024-04-23T16:19:57.240 INFO:teuthology.orchestra.run.smithi103.stdout: "create_time": "2024-04-23T16:17:35.058379+0000", 2024-04-23T16:19:57.241 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-04-23T16:19:57.241 INFO:teuthology.orchestra.run.smithi103.stdout: "flags_names": "hashpspool", 2024-04-23T16:19:57.241 INFO:teuthology.orchestra.run.smithi103.stdout: "type": 1, 2024-04-23T16:19:57.241 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 2, 2024-04-23T16:19:57.241 INFO:teuthology.orchestra.run.smithi103.stdout: "min_size": 2, 2024-04-23T16:19:57.241 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_rule": 0, 2024-04-23T16:19:57.241 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_count": 0, 2024-04-23T16:19:57.241 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_target": 0, 2024-04-23T16:19:57.241 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T16:19:57.241 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T16:19:57.241 INFO:teuthology.orchestra.run.smithi103.stdout: "object_hash": 2, 2024-04-23T16:19:57.241 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_autoscale_mode": "off", 2024-04-23T16:19:57.241 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num": 16, 2024-04-23T16:19:57.241 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_placement_num": 16, 2024-04-23T16:19:57.242 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_placement_num_target": 16, 2024-04-23T16:19:57.242 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_target": 16, 2024-04-23T16:19:57.242 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_pending": 16, 2024-04-23T16:19:57.242 INFO:teuthology.orchestra.run.smithi103.stdout: "last_pg_merge_meta": { 2024-04-23T16:19:57.242 INFO:teuthology.orchestra.run.smithi103.stdout: "source_pgid": "0.0", 2024-04-23T16:19:57.242 INFO:teuthology.orchestra.run.smithi103.stdout: "ready_epoch": 0, 2024-04-23T16:19:57.242 INFO:teuthology.orchestra.run.smithi103.stdout: "last_epoch_started": 0, 2024-04-23T16:19:57.242 INFO:teuthology.orchestra.run.smithi103.stdout: "last_epoch_clean": 0, 2024-04-23T16:19:57.242 INFO:teuthology.orchestra.run.smithi103.stdout: "source_version": "0'0", 2024-04-23T16:19:57.242 INFO:teuthology.orchestra.run.smithi103.stdout: "target_version": "0'0" 2024-04-23T16:19:57.242 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.242 INFO:teuthology.orchestra.run.smithi103.stdout: "last_change": "115", 2024-04-23T16:19:57.242 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend": "0", 2024-04-23T16:19:57.242 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T16:19:57.242 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T16:19:57.243 INFO:teuthology.orchestra.run.smithi103.stdout: "auid": 0, 2024-04-23T16:19:57.243 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_mode": "selfmanaged", 2024-04-23T16:19:57.243 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_seq": 0, 2024-04-23T16:19:57.243 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_epoch": 0, 2024-04-23T16:19:57.243 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_snaps": [], 2024-04-23T16:19:57.243 INFO:teuthology.orchestra.run.smithi103.stdout: "removed_snaps": "[]", 2024-04-23T16:19:57.243 INFO:teuthology.orchestra.run.smithi103.stdout: "quota_max_bytes": 0, 2024-04-23T16:19:57.243 INFO:teuthology.orchestra.run.smithi103.stdout: "quota_max_objects": 0, 2024-04-23T16:19:57.243 INFO:teuthology.orchestra.run.smithi103.stdout: "tiers": [], 2024-04-23T16:19:57.243 INFO:teuthology.orchestra.run.smithi103.stdout: "tier_of": -1, 2024-04-23T16:19:57.243 INFO:teuthology.orchestra.run.smithi103.stdout: "read_tier": -1, 2024-04-23T16:19:57.243 INFO:teuthology.orchestra.run.smithi103.stdout: "write_tier": -1, 2024-04-23T16:19:57.243 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_mode": "none", 2024-04-23T16:19:57.243 INFO:teuthology.orchestra.run.smithi103.stdout: "target_max_bytes": 0, 2024-04-23T16:19:57.243 INFO:teuthology.orchestra.run.smithi103.stdout: "target_max_objects": 0, 2024-04-23T16:19:57.243 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T16:19:57.244 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T16:19:57.244 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T16:19:57.244 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_min_flush_age": 0, 2024-04-23T16:19:57.244 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_min_evict_age": 0, 2024-04-23T16:19:57.244 INFO:teuthology.orchestra.run.smithi103.stdout: "erasure_code_profile": "", 2024-04-23T16:19:57.244 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_params": { 2024-04-23T16:19:57.244 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "none" 2024-04-23T16:19:57.244 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.244 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_period": 0, 2024-04-23T16:19:57.244 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_count": 0, 2024-04-23T16:19:57.244 INFO:teuthology.orchestra.run.smithi103.stdout: "use_gmt_hitset": true, 2024-04-23T16:19:57.244 INFO:teuthology.orchestra.run.smithi103.stdout: "min_read_recency_for_promote": 0, 2024-04-23T16:19:57.244 INFO:teuthology.orchestra.run.smithi103.stdout: "min_write_recency_for_promote": 0, 2024-04-23T16:19:57.244 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T16:19:57.244 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_search_last_n": 0, 2024-04-23T16:19:57.245 INFO:teuthology.orchestra.run.smithi103.stdout: "grade_table": [], 2024-04-23T16:19:57.245 INFO:teuthology.orchestra.run.smithi103.stdout: "stripe_width": 0, 2024-04-23T16:19:57.245 INFO:teuthology.orchestra.run.smithi103.stdout: "expected_num_objects": 0, 2024-04-23T16:19:57.245 INFO:teuthology.orchestra.run.smithi103.stdout: "fast_read": false, 2024-04-23T16:19:57.245 INFO:teuthology.orchestra.run.smithi103.stdout: "options": {}, 2024-04-23T16:19:57.245 INFO:teuthology.orchestra.run.smithi103.stdout: "application_metadata": { 2024-04-23T16:19:57.245 INFO:teuthology.orchestra.run.smithi103.stdout: "rados": {} 2024-04-23T16:19:57.245 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.245 INFO:teuthology.orchestra.run.smithi103.stdout: "read_balance": { 2024-04-23T16:19:57.245 INFO:teuthology.orchestra.run.smithi103.stdout: "score_type": "Fair distribution", 2024-04-23T16:19:57.245 INFO:teuthology.orchestra.run.smithi103.stdout: "score_acting": 2.25, 2024-04-23T16:19:57.245 INFO:teuthology.orchestra.run.smithi103.stdout: "score_stable": 2.25, 2024-04-23T16:19:57.245 INFO:teuthology.orchestra.run.smithi103.stdout: "optimal_score": 1, 2024-04-23T16:19:57.245 INFO:teuthology.orchestra.run.smithi103.stdout: "raw_score_acting": 2.25, 2024-04-23T16:19:57.245 INFO:teuthology.orchestra.run.smithi103.stdout: "raw_score_stable": 2.25, 2024-04-23T16:19:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-04-23T16:19:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: "average_primary_affinity": 1, 2024-04-23T16:19:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: "average_primary_affinity_weighted": 1 2024-04-23T16:19:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:19:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: "osds": [ 2024-04-23T16:19:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 0, 2024-04-23T16:19:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "4c084047-8f01-4b86-a60c-5592735b5862", 2024-04-23T16:19:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:19:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:19:57.246 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:19:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:19:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:19:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:19:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 8, 2024-04-23T16:19:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 58, 2024-04-23T16:19:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:19:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:19:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:19:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.247 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6801", 2024-04-23T16:19:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:19:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:19:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6802", 2024-04-23T16:19:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:19:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:19:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.248 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.249 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.249 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6804", 2024-04-23T16:19:57.249 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:19:57.249 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.249 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.249 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:19:57.249 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.249 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.249 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.249 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6803", 2024-04-23T16:19:57.249 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:19:57.249 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.249 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.250 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6801/4274040097", 2024-04-23T16:19:57.250 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6802/4274040097", 2024-04-23T16:19:57.250 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6804/4274040097", 2024-04-23T16:19:57.250 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6803/4274040097", 2024-04-23T16:19:57.250 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:19:57.250 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:19:57.250 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:19:57.250 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.250 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.250 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.250 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 1, 2024-04-23T16:19:57.250 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "3438f2bd-5fb7-432c-b383-29e18c442f20", 2024-04-23T16:19:57.250 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:19:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:19:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:19:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:19:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:19:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:19:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 13, 2024-04-23T16:19:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 110, 2024-04-23T16:19:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:19:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:19:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:19:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6805", 2024-04-23T16:19:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3716921156 2024-04-23T16:19:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.252 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.252 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.252 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:19:57.252 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.252 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.252 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.252 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6806", 2024-04-23T16:19:57.252 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3716921156 2024-04-23T16:19:57.252 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.252 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.252 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.252 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:19:57.252 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.253 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.253 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.253 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6808", 2024-04-23T16:19:57.253 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3716921156 2024-04-23T16:19:57.253 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.253 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.253 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:19:57.253 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.253 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.253 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.253 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6807", 2024-04-23T16:19:57.253 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3716921156 2024-04-23T16:19:57.253 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.253 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.254 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6805/3716921156", 2024-04-23T16:19:57.254 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6806/3716921156", 2024-04-23T16:19:57.254 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6808/3716921156", 2024-04-23T16:19:57.254 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6807/3716921156", 2024-04-23T16:19:57.254 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:19:57.254 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:19:57.254 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:19:57.254 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.254 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.254 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.254 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 2, 2024-04-23T16:19:57.254 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "c17fffb4-8d17-4546-ba24-e5951a2a37fe", 2024-04-23T16:19:57.254 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:19:57.255 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:19:57.255 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:19:57.255 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:19:57.255 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:19:57.255 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:19:57.255 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 21, 2024-04-23T16:19:57.255 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 110, 2024-04-23T16:19:57.255 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:19:57.255 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:19:57.255 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:19:57.255 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.255 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.255 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.255 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6809", 2024-04-23T16:19:57.255 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 424240088 2024-04-23T16:19:57.256 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.256 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.256 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:19:57.256 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.256 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.256 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6810", 2024-04-23T16:19:57.256 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 424240088 2024-04-23T16:19:57.256 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.256 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.256 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:19:57.256 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.257 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.257 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6812", 2024-04-23T16:19:57.257 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 424240088 2024-04-23T16:19:57.257 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.257 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.257 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.257 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:19:57.257 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.257 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.257 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.257 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6811", 2024-04-23T16:19:57.257 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 424240088 2024-04-23T16:19:57.257 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.258 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.258 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.258 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6809/424240088", 2024-04-23T16:19:57.258 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6810/424240088", 2024-04-23T16:19:57.258 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6812/424240088", 2024-04-23T16:19:57.258 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6811/424240088", 2024-04-23T16:19:57.258 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:19:57.258 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:19:57.258 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:19:57.258 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.258 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.258 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.258 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 3, 2024-04-23T16:19:57.258 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "cee91a3d-55dc-4e89-b568-86821632bb1c", 2024-04-23T16:19:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:19:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:19:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:19:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:19:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:19:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:19:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 27, 2024-04-23T16:19:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 0, 2024-04-23T16:19:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:19:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:19:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:19:57.260 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.260 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.260 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.260 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6813", 2024-04-23T16:19:57.260 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1895183074 2024-04-23T16:19:57.260 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.260 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.260 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.260 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:19:57.260 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.260 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.260 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.261 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6814", 2024-04-23T16:19:57.261 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1895183074 2024-04-23T16:19:57.261 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.261 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.261 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.261 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:19:57.261 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.261 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.261 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.261 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6816", 2024-04-23T16:19:57.261 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1895183074 2024-04-23T16:19:57.261 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.262 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.262 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:19:57.262 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.262 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.262 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.262 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6815", 2024-04-23T16:19:57.262 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1895183074 2024-04-23T16:19:57.262 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.262 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.262 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6813/1895183074", 2024-04-23T16:19:57.262 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6814/1895183074", 2024-04-23T16:19:57.262 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6816/1895183074", 2024-04-23T16:19:57.263 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6815/1895183074", 2024-04-23T16:19:57.263 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:19:57.263 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:19:57.263 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:19:57.263 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.263 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.263 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.263 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 4, 2024-04-23T16:19:57.263 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "61d5385b-2c82-42f8-93c3-8f1baf71c65e", 2024-04-23T16:19:57.263 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:19:57.263 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:19:57.263 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:19:57.263 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:19:57.263 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:19:57.263 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:19:57.264 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 33, 2024-04-23T16:19:57.264 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 0, 2024-04-23T16:19:57.264 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:19:57.264 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:19:57.264 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:19:57.264 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.264 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.264 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6800", 2024-04-23T16:19:57.264 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 794945071 2024-04-23T16:19:57.264 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.264 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.264 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:19:57.264 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.265 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.265 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6801", 2024-04-23T16:19:57.265 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 794945071 2024-04-23T16:19:57.265 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.265 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.265 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:19:57.265 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.265 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.265 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6803", 2024-04-23T16:19:57.265 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 794945071 2024-04-23T16:19:57.265 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.265 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.266 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.266 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:19:57.266 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.266 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.266 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.266 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6802", 2024-04-23T16:19:57.266 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 794945071 2024-04-23T16:19:57.266 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.266 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.266 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.266 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6800/794945071", 2024-04-23T16:19:57.266 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6801/794945071", 2024-04-23T16:19:57.266 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6803/794945071", 2024-04-23T16:19:57.266 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6802/794945071", 2024-04-23T16:19:57.266 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:19:57.267 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:19:57.267 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:19:57.267 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.267 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.267 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.267 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 5, 2024-04-23T16:19:57.267 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "5d492189-c3cd-47ad-a12a-ed4d275bcfe3", 2024-04-23T16:19:57.267 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:19:57.267 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:19:57.267 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:19:57.267 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:19:57.267 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:19:57.267 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:19:57.267 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 38, 2024-04-23T16:19:57.267 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 110, 2024-04-23T16:19:57.267 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:19:57.268 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:19:57.268 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:19:57.268 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.268 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.268 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6804", 2024-04-23T16:19:57.268 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:19:57.268 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.268 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.268 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:19:57.268 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.269 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.269 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6805", 2024-04-23T16:19:57.269 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:19:57.269 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.269 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.269 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.269 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:19:57.269 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.269 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.269 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.269 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6807", 2024-04-23T16:19:57.269 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:19:57.269 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.269 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.269 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.270 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:19:57.270 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.270 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.270 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.270 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6806", 2024-04-23T16:19:57.270 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:19:57.270 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.270 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.270 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.270 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6804/3547200711", 2024-04-23T16:19:57.270 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6805/3547200711", 2024-04-23T16:19:57.270 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6807/3547200711", 2024-04-23T16:19:57.271 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6806/3547200711", 2024-04-23T16:19:57.271 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:19:57.271 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:19:57.271 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:19:57.271 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.271 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.271 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 6, 2024-04-23T16:19:57.271 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "89f2379e-cf14-4e69-8747-7535db81b70b", 2024-04-23T16:19:57.271 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:19:57.272 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:19:57.272 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:19:57.272 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:19:57.272 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:19:57.272 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:19:57.272 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 47, 2024-04-23T16:19:57.272 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 110, 2024-04-23T16:19:57.272 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:19:57.272 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:19:57.272 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:19:57.272 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.272 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.272 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.272 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6808", 2024-04-23T16:19:57.272 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:19:57.273 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.273 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.273 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.273 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:19:57.273 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.273 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.273 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.273 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6809", 2024-04-23T16:19:57.273 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:19:57.273 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.273 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.273 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.273 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:19:57.273 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.273 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.273 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.274 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6811", 2024-04-23T16:19:57.274 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:19:57.274 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.274 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.274 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.274 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:19:57.274 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.274 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.274 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.274 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6810", 2024-04-23T16:19:57.274 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:19:57.274 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.274 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.274 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.275 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6808/366919758", 2024-04-23T16:19:57.275 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6809/366919758", 2024-04-23T16:19:57.275 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6811/366919758", 2024-04-23T16:19:57.275 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6810/366919758", 2024-04-23T16:19:57.275 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:19:57.275 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:19:57.275 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:19:57.275 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.275 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 7, 2024-04-23T16:19:57.275 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "f27b035a-558f-4528-80ea-2a00d03f6539", 2024-04-23T16:19:57.275 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:19:57.275 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:19:57.276 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:19:57.276 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:19:57.276 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:19:57.276 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:19:57.276 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 56, 2024-04-23T16:19:57.276 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 110, 2024-04-23T16:19:57.276 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:19:57.276 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:19:57.276 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:19:57.276 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.276 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.276 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.276 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6812", 2024-04-23T16:19:57.276 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:19:57.276 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.276 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.277 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.277 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:19:57.277 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.277 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.277 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.277 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6813", 2024-04-23T16:19:57.277 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:19:57.277 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.277 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.277 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.277 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:19:57.277 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.277 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.277 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.277 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6815", 2024-04-23T16:19:57.278 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:19:57.278 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.278 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.278 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.278 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:19:57.278 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.278 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.278 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.278 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6814", 2024-04-23T16:19:57.278 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:19:57.278 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.278 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.278 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.278 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6812/3719550888", 2024-04-23T16:19:57.279 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6813/3719550888", 2024-04-23T16:19:57.279 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6815/3719550888", 2024-04-23T16:19:57.279 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6814/3719550888", 2024-04-23T16:19:57.279 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:19:57.279 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:19:57.279 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:19:57.279 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.279 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 8, 2024-04-23T16:19:57.279 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed", 2024-04-23T16:19:57.279 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:19:57.279 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:19:57.279 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:19:57.279 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:19:57.280 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:19:57.280 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:19:57.280 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 66, 2024-04-23T16:19:57.280 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 110, 2024-04-23T16:19:57.280 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:19:57.280 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:19:57.280 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:19:57.280 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.280 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.280 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.280 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6800", 2024-04-23T16:19:57.280 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:19:57.280 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.280 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.280 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.280 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:19:57.281 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.281 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.281 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.281 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6801", 2024-04-23T16:19:57.281 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:19:57.281 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.281 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.281 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:19:57.281 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.281 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.281 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.281 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6803", 2024-04-23T16:19:57.281 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:19:57.281 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.281 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.282 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:19:57.282 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.282 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.282 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.282 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6802", 2024-04-23T16:19:57.282 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:19:57.282 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.282 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.282 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6800/1151408595", 2024-04-23T16:19:57.283 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6801/1151408595", 2024-04-23T16:19:57.283 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6803/1151408595", 2024-04-23T16:19:57.283 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6802/1151408595", 2024-04-23T16:19:57.283 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:19:57.283 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:19:57.283 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:19:57.283 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.283 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.283 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.283 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 9, 2024-04-23T16:19:57.283 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "124d9b05-11eb-4622-b6a2-c4d5bd60b7e6", 2024-04-23T16:19:57.283 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:19:57.283 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:19:57.283 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:19:57.283 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:19:57.283 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:19:57.284 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:19:57.284 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 73, 2024-04-23T16:19:57.284 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 110, 2024-04-23T16:19:57.284 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:19:57.284 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:19:57.284 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:19:57.284 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.284 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.284 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.284 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6804", 2024-04-23T16:19:57.284 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:19:57.284 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.284 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.284 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.284 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:19:57.284 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.285 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.285 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6805", 2024-04-23T16:19:57.285 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:19:57.285 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.285 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.285 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:19:57.285 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.285 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.285 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6807", 2024-04-23T16:19:57.285 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:19:57.285 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.286 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.286 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:19:57.286 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.286 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.286 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.286 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6806", 2024-04-23T16:19:57.286 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:19:57.286 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.286 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.286 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6804/306501862", 2024-04-23T16:19:57.286 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6805/306501862", 2024-04-23T16:19:57.286 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6807/306501862", 2024-04-23T16:19:57.286 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6806/306501862", 2024-04-23T16:19:57.287 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:19:57.287 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:19:57.287 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:19:57.287 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.287 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.287 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.287 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 10, 2024-04-23T16:19:57.287 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "3dfb9070-e534-4ea4-a25e-ee1c51297447", 2024-04-23T16:19:57.287 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:19:57.287 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:19:57.287 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:19:57.287 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:19:57.287 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:19:57.287 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:19:57.287 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 81, 2024-04-23T16:19:57.288 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 0, 2024-04-23T16:19:57.288 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:19:57.288 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:19:57.288 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:19:57.288 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.288 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.288 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.288 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6808", 2024-04-23T16:19:57.288 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:19:57.288 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.288 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.288 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:19:57.289 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.289 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.289 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.289 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6809", 2024-04-23T16:19:57.289 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:19:57.289 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.289 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.289 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.289 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:19:57.289 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.289 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.289 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.289 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6811", 2024-04-23T16:19:57.289 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:19:57.290 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.290 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.290 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.290 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:19:57.290 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.290 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.290 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.290 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6810", 2024-04-23T16:19:57.290 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:19:57.290 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.290 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.290 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.290 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6808/1231263138", 2024-04-23T16:19:57.290 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6809/1231263138", 2024-04-23T16:19:57.290 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6811/1231263138", 2024-04-23T16:19:57.291 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6810/1231263138", 2024-04-23T16:19:57.291 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:19:57.291 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:19:57.291 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:19:57.291 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.291 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.291 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.291 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 11, 2024-04-23T16:19:57.291 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "bc163925-f623-4f63-ab89-35e2d511a730", 2024-04-23T16:19:57.291 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:19:57.291 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:19:57.291 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:19:57.291 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:19:57.291 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:19:57.292 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:19:57.292 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 90, 2024-04-23T16:19:57.292 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 110, 2024-04-23T16:19:57.292 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:19:57.292 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:19:57.292 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:19:57.292 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.292 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.292 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6812", 2024-04-23T16:19:57.292 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:19:57.292 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.292 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.293 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:19:57.293 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.293 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.293 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6813", 2024-04-23T16:19:57.293 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:19:57.293 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.293 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.293 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:19:57.293 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.293 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.293 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6815", 2024-04-23T16:19:57.294 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:19:57.294 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.294 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.294 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:19:57.294 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:19:57.294 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.294 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:19:57.294 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6814", 2024-04-23T16:19:57.294 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:19:57.294 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.294 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.294 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6812/457992661", 2024-04-23T16:19:57.294 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6813/457992661", 2024-04-23T16:19:57.295 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6815/457992661", 2024-04-23T16:19:57.295 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6814/457992661", 2024-04-23T16:19:57.295 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:19:57.295 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:19:57.295 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:19:57.295 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:19:57.295 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.295 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:19:57.295 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_xinfo": [ 2024-04-23T16:19:57.295 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.295 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 0, 2024-04-23T16:19:57.295 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:19:57.295 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:19:57.295 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:19:57.295 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:19:57.295 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:19:57.296 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:19:30.833614+0000", 2024-04-23T16:19:57.296 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:19:57.296 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.296 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.296 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 1, 2024-04-23T16:19:57.296 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:19:57.296 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:19:57.296 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:19:57.296 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:19:57.296 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:19:57.296 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:19:16.484041+0000", 2024-04-23T16:19:57.296 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:19:57.296 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.296 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.296 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 2, 2024-04-23T16:19:57.297 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:19:57.297 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:19:57.297 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:19:57.297 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:19:57.297 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:19:57.297 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:18:50.160987+0000", 2024-04-23T16:19:57.297 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:19:57.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.297 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.297 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 3, 2024-04-23T16:19:57.297 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:19:57.297 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:19:57.297 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:19:57.298 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:19:57.298 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:19:57.298 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:18:34.884478+0000", 2024-04-23T16:19:57.298 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:19:57.298 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.299 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 4, 2024-04-23T16:19:57.299 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:19:57.299 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:19:57.299 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:19:57.299 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:19:57.299 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:19:57.299 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:19:39.054565+0000", 2024-04-23T16:19:57.299 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:19:57.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.299 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.299 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 5, 2024-04-23T16:19:57.299 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:19:57.299 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:19:57.299 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:19:51.041136+0000", 2024-04-23T16:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 6, 2024-04-23T16:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:18:34.813518+0000", 2024-04-23T16:19:57.300 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 7, 2024-04-23T16:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:19:51.271456+0000", 2024-04-23T16:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 8, 2024-04-23T16:19:57.301 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:19:20.828977+0000", 2024-04-23T16:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 9, 2024-04-23T16:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:19:57.302 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:18:31.728192+0000", 2024-04-23T16:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 10, 2024-04-23T16:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:19:57.303 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:19:57.304 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:19:53.410408+0000", 2024-04-23T16:19:57.304 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:19:57.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:19:57.304 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 11, 2024-04-23T16:19:57.304 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:19:57.304 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:19:57.304 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:19:57.304 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:19:57.304 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:19:57.304 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:19:17.294653+0000", 2024-04-23T16:19:57.304 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:19:57.304 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.304 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:19:57.304 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_upmap": [], 2024-04-23T16:19:57.305 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_upmap_items": [], 2024-04-23T16:19:57.305 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_upmap_primaries": [], 2024-04-23T16:19:57.305 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_temp": [], 2024-04-23T16:19:57.305 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_temp": [], 2024-04-23T16:19:57.305 INFO:teuthology.orchestra.run.smithi103.stdout: "blocklist": { 2024-04-23T16:19:57.305 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:6800/3535470771": "2024-04-24T16:09:14.324448+0000", 2024-04-23T16:19:57.305 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/1188058303": "2024-04-24T16:09:14.324448+0000", 2024-04-23T16:19:57.305 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/1510553721": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:19:57.305 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/281182801": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:19:57.305 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:6800/2049124518": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:19:57.305 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/466241963": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:19:57.305 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/2726838698": "2024-04-24T16:09:14.324448+0000", 2024-04-23T16:19:57.305 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/596230573": "2024-04-24T16:09:14.324448+0000" 2024-04-23T16:19:57.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.305 INFO:teuthology.orchestra.run.smithi103.stdout: "range_blocklist": {}, 2024-04-23T16:19:57.306 INFO:teuthology.orchestra.run.smithi103.stdout: "erasure_code_profiles": { 2024-04-23T16:19:57.306 INFO:teuthology.orchestra.run.smithi103.stdout: "default": { 2024-04-23T16:19:57.306 INFO:teuthology.orchestra.run.smithi103.stdout: "crush-failure-domain": "osd", 2024-04-23T16:19:57.306 INFO:teuthology.orchestra.run.smithi103.stdout: "k": "2", 2024-04-23T16:19:57.306 INFO:teuthology.orchestra.run.smithi103.stdout: "m": "1", 2024-04-23T16:19:57.306 INFO:teuthology.orchestra.run.smithi103.stdout: "plugin": "jerasure", 2024-04-23T16:19:57.306 INFO:teuthology.orchestra.run.smithi103.stdout: "technique": "reed_sol_van" 2024-04-23T16:19:57.306 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.306 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:19:57.306 INFO:teuthology.orchestra.run.smithi103.stdout: "removed_snaps_queue": [], 2024-04-23T16:19:57.306 INFO:teuthology.orchestra.run.smithi103.stdout: "new_removed_snaps": [], 2024-04-23T16:19:57.306 INFO:teuthology.orchestra.run.smithi103.stdout: "new_purged_snaps": [], 2024-04-23T16:19:57.306 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_node_flags": {}, 2024-04-23T16:19:57.306 INFO:teuthology.orchestra.run.smithi103.stdout: "device_class_flags": {}, 2024-04-23T16:19:57.306 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_mode": { 2024-04-23T16:19:57.306 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_mode_enabled": false, 2024-04-23T16:19:57.307 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_bucket_count": 0, 2024-04-23T16:19:57.307 INFO:teuthology.orchestra.run.smithi103.stdout: "degraded_stretch_mode": 0, 2024-04-23T16:19:57.307 INFO:teuthology.orchestra.run.smithi103.stdout: "recovering_stretch_mode": 0, 2024-04-23T16:19:57.307 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_mode_bucket": 0 2024-04-23T16:19:57.307 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:19:57.307 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:19:57.307 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:19:57.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:57.308 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:57.308 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:19:57.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:19:57.409 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:19:57.439 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:57.473 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:19:57.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:57.477 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:19:57.578 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:19:57.597 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-23T16:19:57.642 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:57.705 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:57 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/116467085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-23T16:19:57.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:57 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/116467085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-23T16:19:57.932 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:19:58.040 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:57 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/116467085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-23T16:19:58.115 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:19:58.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:58.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:19:58.225 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:19:58.274 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:58.412 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:19:58.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:58.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:19:58.516 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:19:58.547 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:58.576 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:19:58.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:58.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:19:58.680 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:19:58.716 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:58 smithi103 ceph-mon[39629]: pgmap v462: 17 pgs: 17 active+clean; 7.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s 2024-04-23T16:19:58.743 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:58.787 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:19:58.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:58.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:19:58.896 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:19:58.925 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:58.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:58 smithi188 ceph-mon[44101]: pgmap v462: 17 pgs: 17 active+clean; 7.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s 2024-04-23T16:19:59.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:58 smithi148 ceph-mon[44520]: pgmap v462: 17 pgs: 17 active+clean; 7.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s 2024-04-23T16:19:59.177 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:19:59.226 INFO:tasks.thrashosds.thrasher:j is {'epoch': 150, 'fsid': '8e3594f2-018b-11ef-bc93-c7b262605968', 'created': '2024-04-23T16:08:13.477590+0000', 'modified': '2024-04-23T16:19:53.663018+0000', 'last_up_change': '2024-04-23T16:15:34.871303+0000', 'last_in_change': '2024-04-23T16:15:17.238203+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T16:11:22.416777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-04-23T16:17:35.058379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '115', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '4c084047-8f01-4b86-a60c-5592735b5862', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 58, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6801', 'nonce': 4274040097}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6802', 'nonce': 4274040097}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6804', 'nonce': 4274040097}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6803', 'nonce': 4274040097}]}, 'public_addr': '172.21.15.103:6801/4274040097', 'cluster_addr': '172.21.15.103:6802/4274040097', 'heartbeat_back_addr': '172.21.15.103:6804/4274040097', 'heartbeat_front_addr': '172.21.15.103:6803/4274040097', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '3438f2bd-5fb7-432c-b383-29e18c442f20', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6805', 'nonce': 3716921156}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6806', 'nonce': 3716921156}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6808', 'nonce': 3716921156}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6807', 'nonce': 3716921156}]}, 'public_addr': '172.21.15.103:6805/3716921156', 'cluster_addr': '172.21.15.103:6806/3716921156', 'heartbeat_back_addr': '172.21.15.103:6808/3716921156', 'heartbeat_front_addr': '172.21.15.103:6807/3716921156', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c17fffb4-8d17-4546-ba24-e5951a2a37fe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6809', 'nonce': 424240088}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6810', 'nonce': 424240088}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6812', 'nonce': 424240088}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6811', 'nonce': 424240088}]}, 'public_addr': '172.21.15.103:6809/424240088', 'cluster_addr': '172.21.15.103:6810/424240088', 'heartbeat_back_addr': '172.21.15.103:6812/424240088', 'heartbeat_front_addr': '172.21.15.103:6811/424240088', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'cee91a3d-55dc-4e89-b568-86821632bb1c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6813', 'nonce': 1895183074}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6814', 'nonce': 1895183074}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6816', 'nonce': 1895183074}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6815', 'nonce': 1895183074}]}, 'public_addr': '172.21.15.103:6813/1895183074', 'cluster_addr': '172.21.15.103:6814/1895183074', 'heartbeat_back_addr': '172.21.15.103:6816/1895183074', 'heartbeat_front_addr': '172.21.15.103:6815/1895183074', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '61d5385b-2c82-42f8-93c3-8f1baf71c65e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6800', 'nonce': 794945071}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6801', 'nonce': 794945071}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6803', 'nonce': 794945071}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6802', 'nonce': 794945071}]}, 'public_addr': '172.21.15.148:6800/794945071', 'cluster_addr': '172.21.15.148:6801/794945071', 'heartbeat_back_addr': '172.21.15.148:6803/794945071', 'heartbeat_front_addr': '172.21.15.148:6802/794945071', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '5d492189-c3cd-47ad-a12a-ed4d275bcfe3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6804', 'nonce': 3547200711}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6805', 'nonce': 3547200711}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6807', 'nonce': 3547200711}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6806', 'nonce': 3547200711}]}, 'public_addr': '172.21.15.148:6804/3547200711', 'cluster_addr': '172.21.15.148:6805/3547200711', 'heartbeat_back_addr': '172.21.15.148:6807/3547200711', 'heartbeat_front_addr': '172.21.15.148:6806/3547200711', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '89f2379e-cf14-4e69-8747-7535db81b70b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6808', 'nonce': 366919758}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6809', 'nonce': 366919758}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6811', 'nonce': 366919758}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6810', 'nonce': 366919758}]}, 'public_addr': '172.21.15.148:6808/366919758', 'cluster_addr': '172.21.15.148:6809/366919758', 'heartbeat_back_addr': '172.21.15.148:6811/366919758', 'heartbeat_front_addr': '172.21.15.148:6810/366919758', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f27b035a-558f-4528-80ea-2a00d03f6539', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6812', 'nonce': 3719550888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6813', 'nonce': 3719550888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6815', 'nonce': 3719550888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6814', 'nonce': 3719550888}]}, 'public_addr': '172.21.15.148:6812/3719550888', 'cluster_addr': '172.21.15.148:6813/3719550888', 'heartbeat_back_addr': '172.21.15.148:6815/3719550888', 'heartbeat_front_addr': '172.21.15.148:6814/3719550888', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6800', 'nonce': 1151408595}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6801', 'nonce': 1151408595}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6803', 'nonce': 1151408595}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6802', 'nonce': 1151408595}]}, 'public_addr': '172.21.15.188:6800/1151408595', 'cluster_addr': '172.21.15.188:6801/1151408595', 'heartbeat_back_addr': '172.21.15.188:6803/1151408595', 'heartbeat_front_addr': '172.21.15.188:6802/1151408595', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '124d9b05-11eb-4622-b6a2-c4d5bd60b7e6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6804', 'nonce': 306501862}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6805', 'nonce': 306501862}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6807', 'nonce': 306501862}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6806', 'nonce': 306501862}]}, 'public_addr': '172.21.15.188:6804/306501862', 'cluster_addr': '172.21.15.188:6805/306501862', 'heartbeat_back_addr': '172.21.15.188:6807/306501862', 'heartbeat_front_addr': '172.21.15.188:6806/306501862', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '3dfb9070-e534-4ea4-a25e-ee1c51297447', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6808', 'nonce': 1231263138}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6809', 'nonce': 1231263138}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6811', 'nonce': 1231263138}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6810', 'nonce': 1231263138}]}, 'public_addr': '172.21.15.188:6808/1231263138', 'cluster_addr': '172.21.15.188:6809/1231263138', 'heartbeat_back_addr': '172.21.15.188:6811/1231263138', 'heartbeat_front_addr': '172.21.15.188:6810/1231263138', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'bc163925-f623-4f63-ab89-35e2d511a730', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6812', 'nonce': 457992661}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6813', 'nonce': 457992661}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6815', 'nonce': 457992661}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6814', 'nonce': 457992661}]}, 'public_addr': '172.21.15.188:6812/457992661', 'cluster_addr': '172.21.15.188:6813/457992661', 'heartbeat_back_addr': '172.21.15.188:6815/457992661', 'heartbeat_front_addr': '172.21.15.188:6814/457992661', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:19:30.833614+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:19:16.484041+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:18:50.160987+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:18:34.884478+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:19:39.054565+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:19:51.041136+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:18:34.813518+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:19:51.271456+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:19:20.828977+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:18:31.728192+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:19:53.410408+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:19:17.294653+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.103:6800/3535470771': '2024-04-24T16:09:14.324448+0000', '172.21.15.103:0/1188058303': '2024-04-24T16:09:14.324448+0000', '172.21.15.103:0/1510553721': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:0/281182801': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:6800/2049124518': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:0/466241963': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:0/2726838698': '2024-04-24T16:09:14.324448+0000', '172.21.15.103:0/596230573': '2024-04-24T16:09:14.324448+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-23T16:19:59.226 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:19:59.275 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:19:59.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:59.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:19:59.379 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:19:59.442 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:59.556 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:19:59.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:59.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:19:59.662 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:19:59.706 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:59.745 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:19:59.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:59.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:19:59.761 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:19:59.777 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:19:59.777 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:19:59.777 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2024-04-23T16:19:59.777 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2024-04-23T16:19:59.777 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:19:59.797 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:59 smithi148 ceph-mon[44520]: 2.e scrub starts 2024-04-23T16:19:59.797 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:59 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3977890300' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:19:59.797 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:19:59 smithi148 ceph-mon[44520]: from='client.34877 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:19:59.853 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:19:59.889 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:19:59.917 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:19:59.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:19:59.959 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:20:00.060 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:20:00.093 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:59 smithi188 ceph-mon[44101]: 2.e scrub starts 2024-04-23T16:20:00.093 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:59 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3977890300' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:20:00.093 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:19:59 smithi188 ceph-mon[44101]: from='client.34877 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:20:00.096 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:00.098 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:59 smithi103 ceph-mon[39629]: 2.e scrub starts 2024-04-23T16:20:00.098 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:59 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3977890300' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:20:00.098 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:19:59 smithi103 ceph-mon[39629]: from='client.34877 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:20:00.124 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:20:00.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:00.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:20:00.228 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:20:00.260 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:00.290 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:20:00.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:00.294 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:20:00.395 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:20:00.432 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:00.516 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:20:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:00.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:20:00.551 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-23T16:20:00.621 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:20:00.663 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:00.688 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:20:00.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:00.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:20:00.792 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:20:00.829 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:00.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:00 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:20:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:00 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:20:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:00 smithi103 ceph-mon[39629]: from='client.34877 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:20:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:00 smithi103 ceph-mon[39629]: osdmap e151: 12 total, 12 up, 12 in 2024-04-23T16:20:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:00 smithi103 ceph-mon[39629]: overall HEALTH_OK 2024-04-23T16:20:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:00 smithi103 ceph-mon[39629]: pgmap v464: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s 2024-04-23T16:20:00.852 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:20:00.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:00.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:20:00.935 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:20:00.956 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:20:00.971 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:20:00.997 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:00 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:20:00.997 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:00 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:20:00.997 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:00 smithi188 ceph-mon[44101]: from='client.34877 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:20:00.997 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:00 smithi188 ceph-mon[44101]: osdmap e151: 12 total, 12 up, 12 in 2024-04-23T16:20:00.997 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:00 smithi188 ceph-mon[44101]: overall HEALTH_OK 2024-04-23T16:20:00.997 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:00 smithi188 ceph-mon[44101]: pgmap v464: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s 2024-04-23T16:20:00.997 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:01.022 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:20:01.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:01.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:20:01.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:00 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:20:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:00 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:20:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:00 smithi148 ceph-mon[44520]: from='client.34877 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:20:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:00 smithi148 ceph-mon[44520]: osdmap e151: 12 total, 12 up, 12 in 2024-04-23T16:20:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:00 smithi148 ceph-mon[44520]: overall HEALTH_OK 2024-04-23T16:20:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:00 smithi148 ceph-mon[44520]: pgmap v464: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s 2024-04-23T16:20:01.129 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:20:01.163 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:01.196 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:20:01.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:01.201 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:20:01.302 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:20:01.360 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:01.387 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:20:01.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:01.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:20:01.509 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:20:01.554 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:01.810 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:20:01.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:01.819 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:01 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:20:01.820 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:01 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:20:01.820 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:01 smithi103 ceph-mon[39629]: 2.e scrub ok 2024-04-23T16:20:01.820 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:01 smithi103 ceph-mon[39629]: osdmap e152: 12 total, 12 up, 12 in 2024-04-23T16:20:01.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:20:01.921 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:20:01.950 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:01.989 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:20:01.995 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:01 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:20:01.995 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:01 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:20:01.995 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:01 smithi188 ceph-mon[44101]: 2.e scrub ok 2024-04-23T16:20:01.995 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:01 smithi188 ceph-mon[44101]: osdmap e152: 12 total, 12 up, 12 in 2024-04-23T16:20:01.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:01.996 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:20:02.097 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:20:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:01 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:20:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:01 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:20:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:01 smithi148 ceph-mon[44520]: 2.e scrub ok 2024-04-23T16:20:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:01 smithi148 ceph-mon[44520]: osdmap e152: 12 total, 12 up, 12 in 2024-04-23T16:20:02.146 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:02.172 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:20:02.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:02.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:20:02.277 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:20:02.309 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:02.337 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:20:02.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:02.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:20:02.441 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:02.472 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:02.604 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:02.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:02.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:02.634 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:20:02.707 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:20:02.783 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:02.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:02 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:20:02.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:02 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:20:02.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:02 smithi103 ceph-mon[39629]: osdmap e153: 12 total, 12 up, 12 in 2024-04-23T16:20:02.854 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:02 smithi103 ceph-mon[39629]: pgmap v467: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 997 op/s 2024-04-23T16:20:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:02 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:20:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:02 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:20:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:02 smithi148 ceph-mon[44520]: osdmap e153: 12 total, 12 up, 12 in 2024-04-23T16:20:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:02 smithi148 ceph-mon[44520]: pgmap v467: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 997 op/s 2024-04-23T16:20:02.927 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:20:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2024-04-23T16:20:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2024-04-23T16:20:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:20:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:106720 2.7 2:e3855b91:::benchmark_data_smithi188_77627_object106719:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e128)", 2024-04-23T16:20:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:18:31.336294+0000", 2024-04-23T16:20:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 91.563033723999993, 2024-04-23T16:20:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021898656999999998, 2024-04-23T16:20:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 106720 2024-04-23T16:20:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:31.336294+0000", 2024-04-23T16:20:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:31.336294+0000", 2024-04-23T16:20:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:31.336295+0000", 2024-04-23T16:20:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4799999999999998e-07 2024-04-23T16:20:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:31.336364+0000", 2024-04-23T16:20:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8862999999999994e-05 2024-04-23T16:20:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:31.336370+0000", 2024-04-23T16:20:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1650000000000003e-06 2024-04-23T16:20:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:31.336374+0000", 2024-04-23T16:20:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4340000000000002e-06 2024-04-23T16:20:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:31.345029+0000", 2024-04-23T16:20:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0086543179999999994 2024-04-23T16:20:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:31.345169+0000", 2024-04-23T16:20:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000140307 2024-04-23T16:20:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:31.345313+0000", 2024-04-23T16:20:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000144451 2024-04-23T16:20:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:31.352707+0000", 2024-04-23T16:20:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0073941220000000004 2024-04-23T16:20:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:31.358096+0000", 2024-04-23T16:20:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0053889530000000001 2024-04-23T16:20:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:31.358152+0000", 2024-04-23T16:20:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5516000000000002e-05 2024-04-23T16:20:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:31.358193+0000", 2024-04-23T16:20:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0979999999999997e-05 2024-04-23T16:20:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:112745 2.7 2:e615d61e:::benchmark_data_smithi188_77627_object112744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-04-23T16:20:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:18:34.939368+0000", 2024-04-23T16:20:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 87.959959706999996, 2024-04-23T16:20:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022370468000000001, 2024-04-23T16:20:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 112745 2024-04-23T16:20:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:34.939368+0000", 2024-04-23T16:20:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:34.939368+0000", 2024-04-23T16:20:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:34.939370+0000", 2024-04-23T16:20:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.824e-06 2024-04-23T16:20:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:34.939477+0000", 2024-04-23T16:20:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000106504 2024-04-23T16:20:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:34.939486+0000", 2024-04-23T16:20:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8670000000000006e-06 2024-04-23T16:20:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:34.939493+0000", 2024-04-23T16:20:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0560000000000001e-06 2024-04-23T16:20:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:34.939545+0000", 2024-04-23T16:20:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2614000000000001e-05 2024-04-23T16:20:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:34.939652+0000", 2024-04-23T16:20:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107154 2024-04-23T16:20:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:34.939755+0000", 2024-04-23T16:20:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000102539 2024-04-23T16:20:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:34.944708+0000", 2024-04-23T16:20:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0049530349999999997 2024-04-23T16:20:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:34.961630+0000", 2024-04-23T16:20:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016922613 2024-04-23T16:20:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:34.961692+0000", 2024-04-23T16:20:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1521999999999998e-05 2024-04-23T16:20:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:18:34.961739+0000", 2024-04-23T16:20:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6740000000000003e-05 2024-04-23T16:20:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:150504 2.7 2:e69ff4b1:::benchmark_data_smithi188_77627_object150503:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-04-23T16:20:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:02.553873+0000", 2024-04-23T16:20:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 60.345455200000004, 2024-04-23T16:20:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023308675000000001, 2024-04-23T16:20:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 150504 2024-04-23T16:20:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.553873+0000", 2024-04-23T16:20:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.553873+0000", 2024-04-23T16:20:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.553873+0000", 2024-04-23T16:20:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7999999999999995e-07 2024-04-23T16:20:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.553995+0000", 2024-04-23T16:20:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012124400000000001 2024-04-23T16:20:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.554003+0000", 2024-04-23T16:20:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2539999999999992e-06 2024-04-23T16:20:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.554010+0000", 2024-04-23T16:20:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6440000000000003e-06 2024-04-23T16:20:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.557261+0000", 2024-04-23T16:20:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032509230000000002 2024-04-23T16:20:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.557372+0000", 2024-04-23T16:20:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011110700000000001 2024-04-23T16:20:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.557511+0000", 2024-04-23T16:20:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000139386 2024-04-23T16:20:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.569738+0000", 2024-04-23T16:20:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012226561 2024-04-23T16:20:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.577073+0000", 2024-04-23T16:20:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0073356810000000001 2024-04-23T16:20:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.577135+0000", 2024-04-23T16:20:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2217000000000004e-05 2024-04-23T16:20:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.577181+0000", 2024-04-23T16:20:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5977999999999999e-05 2024-04-23T16:20:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:150506 2.7 2:ec1593a3:::benchmark_data_smithi188_77627_object150505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-04-23T16:20:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:02.554070+0000", 2024-04-23T16:20:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 60.345257744999998, 2024-04-23T16:20:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.025834587999999999, 2024-04-23T16:20:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 150506 2024-04-23T16:20:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.554070+0000", 2024-04-23T16:20:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.554070+0000", 2024-04-23T16:20:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.554071+0000", 2024-04-23T16:20:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8999999999999995e-07 2024-04-23T16:20:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.554159+0000", 2024-04-23T16:20:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7686e-05 2024-04-23T16:20:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.554170+0000", 2024-04-23T16:20:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1004e-05 2024-04-23T16:20:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.554178+0000", 2024-04-23T16:20:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.9149999999999996e-06 2024-04-23T16:20:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.561374+0000", 2024-04-23T16:20:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0071960499999999998 2024-04-23T16:20:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.561507+0000", 2024-04-23T16:20:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000133645 2024-04-23T16:20:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.561647+0000", 2024-04-23T16:20:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013928899999999999 2024-04-23T16:20:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.577026+0000", 2024-04-23T16:20:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015378887000000001 2024-04-23T16:20:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.579794+0000", 2024-04-23T16:20:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0027681350000000001 2024-04-23T16:20:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.579858+0000", 2024-04-23T16:20:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4727999999999994e-05 2024-04-23T16:20:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.579905+0000", 2024-04-23T16:20:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6359000000000001e-05 2024-04-23T16:20:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:02.951 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.951 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.951 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.951 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:150586 2.7 2:efb18dfb:::benchmark_data_smithi188_77627_object150585:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-04-23T16:20:02.951 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:02.629241+0000", 2024-04-23T16:20:02.951 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 60.270087072000003, 2024-04-23T16:20:02.951 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022313324999999998, 2024-04-23T16:20:02.951 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:02.951 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 150586 2024-04-23T16:20:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.629241+0000", 2024-04-23T16:20:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.629241+0000", 2024-04-23T16:20:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.629241+0000", 2024-04-23T16:20:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2499999999999995e-07 2024-04-23T16:20:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.629306+0000", 2024-04-23T16:20:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4788000000000004e-05 2024-04-23T16:20:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.629314+0000", 2024-04-23T16:20:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1340000000000004e-06 2024-04-23T16:20:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.629320+0000", 2024-04-23T16:20:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0589999999999999e-06 2024-04-23T16:20:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.635413+0000", 2024-04-23T16:20:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0060921580000000003 2024-04-23T16:20:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.635499+0000", 2024-04-23T16:20:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6713999999999997e-05 2024-04-23T16:20:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.635580+0000", 2024-04-23T16:20:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0648000000000004e-05 2024-04-23T16:20:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.640852+0000", 2024-04-23T16:20:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0052717279999999998 2024-04-23T16:20:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.651483+0000", 2024-04-23T16:20:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010631277999999999 2024-04-23T16:20:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.651519+0000", 2024-04-23T16:20:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5673999999999998e-05 2024-04-23T16:20:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:02.651554+0000", 2024-04-23T16:20:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5618999999999999e-05 2024-04-23T16:20:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.957 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:156970 2.7 2:eac8dbe8:::benchmark_data_smithi188_77627_object156969:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-04-23T16:20:02.957 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:08.512362+0000", 2024-04-23T16:20:02.957 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 54.386965881000002, 2024-04-23T16:20:02.957 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021961305, 2024-04-23T16:20:02.957 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:02.957 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:02.957 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:02.957 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:02.957 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:02.957 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:02.957 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 156970 2024-04-23T16:20:02.957 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.957 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:02.958 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.958 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:02.958 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:08.512362+0000", 2024-04-23T16:20:02.958 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.958 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.958 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.958 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:02.958 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:08.512362+0000", 2024-04-23T16:20:02.958 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.958 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.958 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.958 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:02.958 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:08.512364+0000", 2024-04-23T16:20:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.838e-06 2024-04-23T16:20:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:08.512402+0000", 2024-04-23T16:20:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7879000000000001e-05 2024-04-23T16:20:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:08.512414+0000", 2024-04-23T16:20:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2649e-05 2024-04-23T16:20:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:08.512434+0000", 2024-04-23T16:20:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9660000000000002e-05 2024-04-23T16:20:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:08.512503+0000", 2024-04-23T16:20:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9128000000000006e-05 2024-04-23T16:20:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:08.512606+0000", 2024-04-23T16:20:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010292400000000001 2024-04-23T16:20:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:08.512781+0000", 2024-04-23T16:20:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000175207 2024-04-23T16:20:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:08.517369+0000", 2024-04-23T16:20:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045874540000000004 2024-04-23T16:20:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:08.534244+0000", 2024-04-23T16:20:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016874903 2024-04-23T16:20:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.962 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:02.962 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:08.534282+0000", 2024-04-23T16:20:02.962 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8192000000000003e-05 2024-04-23T16:20:02.962 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.962 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.962 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:02.962 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:08.534323+0000", 2024-04-23T16:20:02.962 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1471000000000003e-05 2024-04-23T16:20:02.962 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.962 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:02.962 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.962 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.963 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.963 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:163656 2.7 2:eed41374:::benchmark_data_smithi188_77627_object163655:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-04-23T16:20:02.963 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:15.329597+0000", 2024-04-23T16:20:02.963 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 47.569730927000002, 2024-04-23T16:20:02.963 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023701479000000001, 2024-04-23T16:20:02.963 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:02.963 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:02.963 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:02.963 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:02.963 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:02.963 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:02.963 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 163656 2024-04-23T16:20:02.963 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:15.329597+0000", 2024-04-23T16:20:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:15.329597+0000", 2024-04-23T16:20:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:15.329598+0000", 2024-04-23T16:20:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0900000000000001e-07 2024-04-23T16:20:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:15.329657+0000", 2024-04-23T16:20:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9277999999999997e-05 2024-04-23T16:20:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:15.329665+0000", 2024-04-23T16:20:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1810000000000001e-06 2024-04-23T16:20:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:15.329671+0000", 2024-04-23T16:20:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6919999999999999e-06 2024-04-23T16:20:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:15.334373+0000", 2024-04-23T16:20:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0047017630000000003 2024-04-23T16:20:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:15.334509+0000", 2024-04-23T16:20:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013598399999999999 2024-04-23T16:20:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:15.334648+0000", 2024-04-23T16:20:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000139584 2024-04-23T16:20:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:15.340887+0000", 2024-04-23T16:20:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0062387780000000004 2024-04-23T16:20:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:15.353222+0000", 2024-04-23T16:20:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012335400999999999 2024-04-23T16:20:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:15.353256+0000", 2024-04-23T16:20:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3319999999999999e-05 2024-04-23T16:20:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:15.353299+0000", 2024-04-23T16:20:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2789e-05 2024-04-23T16:20:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:165019 2.7 2:e3d243a8:::benchmark_data_smithi188_77627_object165018:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-04-23T16:20:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:16.639688+0000", 2024-04-23T16:20:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 46.259639868999997, 2024-04-23T16:20:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022923849999999999, 2024-04-23T16:20:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 165019 2024-04-23T16:20:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:16.639688+0000", 2024-04-23T16:20:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:02.970 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:16.639688+0000", 2024-04-23T16:20:02.970 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:02.970 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:16.639689+0000", 2024-04-23T16:20:02.970 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2600000000000002e-07 2024-04-23T16:20:02.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:02.970 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:16.639776+0000", 2024-04-23T16:20:02.970 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6786999999999994e-05 2024-04-23T16:20:02.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:02.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:16.639782+0000", 2024-04-23T16:20:02.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4389999999999997e-06 2024-04-23T16:20:02.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:02.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:16.639786+0000", 2024-04-23T16:20:02.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2220000000000004e-06 2024-04-23T16:20:02.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:02.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:16.650499+0000", 2024-04-23T16:20:02.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010713088000000001 2024-04-23T16:20:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:16.650589+0000", 2024-04-23T16:20:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9291999999999998e-05 2024-04-23T16:20:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:16.650684+0000", 2024-04-23T16:20:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.4994999999999994e-05 2024-04-23T16:20:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:16.654986+0000", 2024-04-23T16:20:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0043019399999999998 2024-04-23T16:20:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:16.662545+0000", 2024-04-23T16:20:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0075595669999999997 2024-04-23T16:20:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:16.662583+0000", 2024-04-23T16:20:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7780000000000001e-05 2024-04-23T16:20:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:16.662612+0000", 2024-04-23T16:20:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9014e-05 2024-04-23T16:20:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:166311 2.7 2:e836c75f:::benchmark_data_smithi188_77627_object166310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-23T16:20:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:17.947207+0000", 2024-04-23T16:20:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 44.952120698000002, 2024-04-23T16:20:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023304925000000001, 2024-04-23T16:20:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 166311 2024-04-23T16:20:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.947207+0000", 2024-04-23T16:20:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.947207+0000", 2024-04-23T16:20:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.947210+0000", 2024-04-23T16:20:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1089999999999998e-06 2024-04-23T16:20:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.947265+0000", 2024-04-23T16:20:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4596999999999999e-05 2024-04-23T16:20:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.947279+0000", 2024-04-23T16:20:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3934000000000001e-05 2024-04-23T16:20:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.947286+0000", 2024-04-23T16:20:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4379999999999998e-06 2024-04-23T16:20:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.947346+0000", 2024-04-23T16:20:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9530000000000001e-05 2024-04-23T16:20:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.947482+0000", 2024-04-23T16:20:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000135816 2024-04-23T16:20:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.947570+0000", 2024-04-23T16:20:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8558000000000002e-05 2024-04-23T16:20:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.954088+0000", 2024-04-23T16:20:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0065173510000000002 2024-04-23T16:20:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.970396+0000", 2024-04-23T16:20:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016307947999999999 2024-04-23T16:20:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:02.979 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.970465+0000", 2024-04-23T16:20:02.979 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9777000000000006e-05 2024-04-23T16:20:02.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.979 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.979 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:02.979 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.970512+0000", 2024-04-23T16:20:02.979 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6866999999999997e-05 2024-04-23T16:20:02.979 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.979 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:02.979 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.979 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.979 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:166313 2.7 2:e7e627db:::benchmark_data_smithi188_77627_object166312:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-23T16:20:02.980 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:17.947313+0000", 2024-04-23T16:20:02.980 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 44.952015283000001, 2024-04-23T16:20:02.980 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023310358, 2024-04-23T16:20:02.980 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:02.980 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:02.980 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:02.980 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:02.980 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:02.980 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:02.980 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 166313 2024-04-23T16:20:02.980 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.980 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:02.980 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.980 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:02.980 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.947313+0000", 2024-04-23T16:20:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.947313+0000", 2024-04-23T16:20:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.947313+0000", 2024-04-23T16:20:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8199999999999999e-07 2024-04-23T16:20:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.947388+0000", 2024-04-23T16:20:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4975000000000006e-05 2024-04-23T16:20:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.947395+0000", 2024-04-23T16:20:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.365e-06 2024-04-23T16:20:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.947401+0000", 2024-04-23T16:20:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1720000000000004e-06 2024-04-23T16:20:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.950788+0000", 2024-04-23T16:20:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0033870509999999999 2024-04-23T16:20:02.983 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.983 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.983 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:02.983 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.950930+0000", 2024-04-23T16:20:02.983 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014217599999999999 2024-04-23T16:20:02.983 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.983 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.983 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:02.983 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.951107+0000", 2024-04-23T16:20:02.983 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00017641600000000001 2024-04-23T16:20:02.983 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.983 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.983 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.954196+0000", 2024-04-23T16:20:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0030895269999999999 2024-04-23T16:20:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.970554+0000", 2024-04-23T16:20:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016357634999999999 2024-04-23T16:20:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.970587+0000", 2024-04-23T16:20:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3269999999999998e-05 2024-04-23T16:20:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:17.970623+0000", 2024-04-23T16:20:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5988999999999997e-05 2024-04-23T16:20:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:186696 2.7 2:ed292b5b:::benchmark_data_smithi188_77627_object17347:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-04-23T16:20:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:37.814035+0000", 2024-04-23T16:20:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 25.085292855999999, 2024-04-23T16:20:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023858356000000001, 2024-04-23T16:20:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 186696 2024-04-23T16:20:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.814035+0000", 2024-04-23T16:20:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.814035+0000", 2024-04-23T16:20:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.814036+0000", 2024-04-23T16:20:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0079999999999999e-06 2024-04-23T16:20:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.814039+0000", 2024-04-23T16:20:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.146e-06 2024-04-23T16:20:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.814045+0000", 2024-04-23T16:20:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7540000000000003e-06 2024-04-23T16:20:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.814050+0000", 2024-04-23T16:20:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4450000000000004e-06 2024-04-23T16:20:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.825334+0000", 2024-04-23T16:20:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011283891000000001 2024-04-23T16:20:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.825448+0000", 2024-04-23T16:20:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000113992 2024-04-23T16:20:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.825550+0000", 2024-04-23T16:20:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000101495 2024-04-23T16:20:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.833327+0000", 2024-04-23T16:20:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0077768810000000002 2024-04-23T16:20:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.837802+0000", 2024-04-23T16:20:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0044750099999999998 2024-04-23T16:20:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.837850+0000", 2024-04-23T16:20:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8680000000000001e-05 2024-04-23T16:20:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.837894+0000", 2024-04-23T16:20:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3053999999999998e-05 2024-04-23T16:20:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:186697 2.7 2:ef1f293c:::benchmark_data_smithi188_77627_object17348:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-04-23T16:20:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:37.817946+0000", 2024-04-23T16:20:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 25.081382439999999, 2024-04-23T16:20:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026903527999999999, 2024-04-23T16:20:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 186697 2024-04-23T16:20:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.817946+0000", 2024-04-23T16:20:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.817946+0000", 2024-04-23T16:20:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.817947+0000", 2024-04-23T16:20:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1930000000000001e-06 2024-04-23T16:20:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.817950+0000", 2024-04-23T16:20:02.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6160000000000002e-06 2024-04-23T16:20:02.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.993 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:02.993 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.817959+0000", 2024-04-23T16:20:02.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5779999999999999e-06 2024-04-23T16:20:02.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.993 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:02.993 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.817966+0000", 2024-04-23T16:20:02.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6549999999999998e-06 2024-04-23T16:20:02.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.993 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:02.994 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.829804+0000", 2024-04-23T16:20:02.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011838784 2024-04-23T16:20:02.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.994 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:02.994 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.829896+0000", 2024-04-23T16:20:02.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1561e-05 2024-04-23T16:20:02.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.994 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:02.994 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.829968+0000", 2024-04-23T16:20:02.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1647e-05 2024-04-23T16:20:02.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.994 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.837748+0000", 2024-04-23T16:20:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0077806280000000004 2024-04-23T16:20:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.844773+0000", 2024-04-23T16:20:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0070248510000000004 2024-04-23T16:20:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.844809+0000", 2024-04-23T16:20:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6022000000000002e-05 2024-04-23T16:20:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:37.844849+0000", 2024-04-23T16:20:02.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9993000000000001e-05 2024-04-23T16:20:02.996 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.996 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:02.996 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:02.996 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.996 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.996 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:207804 2.7 2:ebc864f0:::benchmark_data_smithi188_77627_object38455:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-04-23T16:20:02.996 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:54.872768+0000", 2024-04-23T16:20:02.996 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 8.0265605000000004, 2024-04-23T16:20:02.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.024568689000000001, 2024-04-23T16:20:02.996 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:02.996 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:02.996 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:02.996 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:02.996 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 207804 2024-04-23T16:20:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.872768+0000", 2024-04-23T16:20:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.872768+0000", 2024-04-23T16:20:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:02.998 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.872769+0000", 2024-04-23T16:20:02.998 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8339999999999999e-06 2024-04-23T16:20:02.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.998 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:02.998 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.872775+0000", 2024-04-23T16:20:02.998 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6180000000000003e-06 2024-04-23T16:20:02.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.998 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:02.998 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.872786+0000", 2024-04-23T16:20:02.998 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0688e-05 2024-04-23T16:20:02.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.872792+0000", 2024-04-23T16:20:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6429999999999998e-06 2024-04-23T16:20:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.877656+0000", 2024-04-23T16:20:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0048641939999999996 2024-04-23T16:20:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.877789+0000", 2024-04-23T16:20:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013209699999999999 2024-04-23T16:20:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.877925+0000", 2024-04-23T16:20:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013645199999999999 2024-04-23T16:20:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.886492+0000", 2024-04-23T16:20:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.008567313 2024-04-23T16:20:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.897240+0000", 2024-04-23T16:20:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010747873 2024-04-23T16:20:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.897294+0000", 2024-04-23T16:20:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3995000000000001e-05 2024-04-23T16:20:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.897336+0000", 2024-04-23T16:20:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1981999999999999e-05 2024-04-23T16:20:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:209923 2.7 2:e51af5d7:::benchmark_data_smithi188_77627_object40574:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-04-23T16:20:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:56.588159+0000", 2024-04-23T16:20:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 6.3111692020000003, 2024-04-23T16:20:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023343275, 2024-04-23T16:20:03.002 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:03.002 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:03.002 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:03.002 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:03.002 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:03.002 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:03.002 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 209923 2024-04-23T16:20:03.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.002 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:03.002 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.002 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:03.002 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:56.588159+0000", 2024-04-23T16:20:03.002 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:03.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:03.003 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:56.588159+0000", 2024-04-23T16:20:03.003 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:03.003 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:03.003 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:56.588159+0000", 2024-04-23T16:20:03.003 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4700000000000001e-07 2024-04-23T16:20:03.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.004 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:03.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:56.588163+0000", 2024-04-23T16:20:03.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.089e-06 2024-04-23T16:20:03.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.004 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:03.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:56.588172+0000", 2024-04-23T16:20:03.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5340000000000005e-06 2024-04-23T16:20:03.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.004 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:03.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:56.588178+0000", 2024-04-23T16:20:03.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8499999999999999e-06 2024-04-23T16:20:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:56.595352+0000", 2024-04-23T16:20:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0071737700000000003 2024-04-23T16:20:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:56.595434+0000", 2024-04-23T16:20:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2391000000000004e-05 2024-04-23T16:20:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:56.595517+0000", 2024-04-23T16:20:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2632000000000004e-05 2024-04-23T16:20:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.006 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.006 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:03.006 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:56.601269+0000", 2024-04-23T16:20:03.006 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0057520480000000001 2024-04-23T16:20:03.006 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.007 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:03.007 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:56.611416+0000", 2024-04-23T16:20:03.007 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010147611000000001 2024-04-23T16:20:03.007 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.007 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:03.007 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:56.611473+0000", 2024-04-23T16:20:03.007 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6734000000000003e-05 2024-04-23T16:20:03.007 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.008 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.008 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:03.008 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:56.611502+0000", 2024-04-23T16:20:03.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8969e-05 2024-04-23T16:20:03.008 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:03.008 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:03.008 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:03.008 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.008 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.008 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:211255 2.7 2:e8e7f9b8:::benchmark_data_smithi188_77627_object41906:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-04-23T16:20:03.008 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:57.689305+0000", 2024-04-23T16:20:03.008 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.2100229980000003, 2024-04-23T16:20:03.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.025555564999999999, 2024-04-23T16:20:03.008 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:03.009 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:03.009 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:03.009 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:03.009 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:03.009 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:03.009 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 211255 2024-04-23T16:20:03.009 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.009 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:03.009 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.009 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:03.009 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.689305+0000", 2024-04-23T16:20:03.009 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:03.009 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.009 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.009 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.689305+0000", 2024-04-23T16:20:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.689306+0000", 2024-04-23T16:20:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0100000000000003e-07 2024-04-23T16:20:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.689309+0000", 2024-04-23T16:20:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.551e-06 2024-04-23T16:20:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.689315+0000", 2024-04-23T16:20:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8950000000000003e-06 2024-04-23T16:20:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.689321+0000", 2024-04-23T16:20:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.699e-06 2024-04-23T16:20:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.697404+0000", 2024-04-23T16:20:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0080825970000000004 2024-04-23T16:20:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.697532+0000", 2024-04-23T16:20:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012834599999999999 2024-04-23T16:20:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.697647+0000", 2024-04-23T16:20:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000115425 2024-04-23T16:20:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.714676+0000", 2024-04-23T16:20:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.017028861999999999 2024-04-23T16:20:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.714747+0000", 2024-04-23T16:20:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0919000000000003e-05 2024-04-23T16:20:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.714808+0000", 2024-04-23T16:20:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1016e-05 2024-04-23T16:20:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.714861+0000", 2024-04-23T16:20:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2354000000000001e-05 2024-04-23T16:20:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:211258 2.7 2:e0e8539e:::benchmark_data_smithi188_77627_object41909:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-04-23T16:20:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:57.689331+0000", 2024-04-23T16:20:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.2099965900000003, 2024-04-23T16:20:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.025713305999999998, 2024-04-23T16:20:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 211258 2024-04-23T16:20:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.689331+0000", 2024-04-23T16:20:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.689331+0000", 2024-04-23T16:20:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.689332+0000", 2024-04-23T16:20:03.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8299999999999996e-07 2024-04-23T16:20:03.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.016 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:03.016 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.689335+0000", 2024-04-23T16:20:03.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7700000000000002e-06 2024-04-23T16:20:03.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.016 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:03.016 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.689341+0000", 2024-04-23T16:20:03.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9780000000000002e-06 2024-04-23T16:20:03.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.689346+0000", 2024-04-23T16:20:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2680000000000002e-06 2024-04-23T16:20:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.701850+0000", 2024-04-23T16:20:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012503525 2024-04-23T16:20:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.701926+0000", 2024-04-23T16:20:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5906000000000001e-05 2024-04-23T16:20:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.701991+0000", 2024-04-23T16:20:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5489999999999998e-05 2024-04-23T16:20:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.714925+0000", 2024-04-23T16:20:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012934421999999999 2024-04-23T16:20:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.714940+0000", 2024-04-23T16:20:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4287999999999999e-05 2024-04-23T16:20:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.714989+0000", 2024-04-23T16:20:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9459000000000002e-05 2024-04-23T16:20:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.715045+0000", 2024-04-23T16:20:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5516999999999997e-05 2024-04-23T16:20:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:211264 2.7 2:e491690a:::benchmark_data_smithi188_77627_object41915:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-04-23T16:20:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:57.695206+0000", 2024-04-23T16:20:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.2041215340000004, 2024-04-23T16:20:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022113628999999999, 2024-04-23T16:20:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 211264 2024-04-23T16:20:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:03.021 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.695206+0000", 2024-04-23T16:20:03.021 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:03.021 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.021 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.021 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:03.021 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.695206+0000", 2024-04-23T16:20:03.021 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:03.021 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.021 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.021 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:03.021 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.695208+0000", 2024-04-23T16:20:03.021 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7260000000000001e-06 2024-04-23T16:20:03.021 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.021 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.021 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:03.022 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.695214+0000", 2024-04-23T16:20:03.022 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9340000000000003e-06 2024-04-23T16:20:03.022 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.022 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.022 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:03.022 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.695225+0000", 2024-04-23T16:20:03.022 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0434999999999999e-05 2024-04-23T16:20:03.022 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.022 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.022 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:03.022 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.695234+0000", 2024-04-23T16:20:03.022 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3950000000000007e-06 2024-04-23T16:20:03.022 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.022 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.706520+0000", 2024-04-23T16:20:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011286122000000001 2024-04-23T16:20:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.706637+0000", 2024-04-23T16:20:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000116756 2024-04-23T16:20:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.706745+0000", 2024-04-23T16:20:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107978 2024-04-23T16:20:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.715051+0000", 2024-04-23T16:20:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0083059250000000005 2024-04-23T16:20:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.717218+0000", 2024-04-23T16:20:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021674480000000002 2024-04-23T16:20:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.717277+0000", 2024-04-23T16:20:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9123000000000003e-05 2024-04-23T16:20:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:57.717320+0000", 2024-04-23T16:20:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2787000000000002e-05 2024-04-23T16:20:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:215007 2.7 2:e9e7c6ce:::benchmark_data_smithi188_77627_object45658:head [delete] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-04-23T16:20:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:20:00.862354+0000", 2024-04-23T16:20:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 2.0369744820000002, 2024-04-23T16:20:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022131198000000001, 2024-04-23T16:20:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 215007 2024-04-23T16:20:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.862354+0000", 2024-04-23T16:20:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.862354+0000", 2024-04-23T16:20:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.862354+0000", 2024-04-23T16:20:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8599999999999997e-07 2024-04-23T16:20:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.862358+0000", 2024-04-23T16:20:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4979999999999998e-06 2024-04-23T16:20:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.862365+0000", 2024-04-23T16:20:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4710000000000001e-06 2024-04-23T16:20:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.862371+0000", 2024-04-23T16:20:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2089999999999996e-06 2024-04-23T16:20:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.866656+0000", 2024-04-23T16:20:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0042853129999999998 2024-04-23T16:20:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.866792+0000", 2024-04-23T16:20:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013641200000000001 2024-04-23T16:20:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.866925+0000", 2024-04-23T16:20:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013278500000000001 2024-04-23T16:20:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.879541+0000", 2024-04-23T16:20:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012615497 2024-04-23T16:20:03.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:03.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.884381+0000", 2024-04-23T16:20:03.030 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0048400689999999998 2024-04-23T16:20:03.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:03.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.884433+0000", 2024-04-23T16:20:03.030 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2028999999999999e-05 2024-04-23T16:20:03.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.884485+0000", 2024-04-23T16:20:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2028999999999999e-05 2024-04-23T16:20:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:216251 2.7 2:e1bd377b:::benchmark_data_smithi188_77627_object46902:head [delete] snapc 0=[] ondisk+write+known_if_redirected e153)", 2024-04-23T16:20:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:20:01.871186+0000", 2024-04-23T16:20:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 1.0281422710000001, 2024-04-23T16:20:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022197088, 2024-04-23T16:20:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 216251 2024-04-23T16:20:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.871186+0000", 2024-04-23T16:20:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.871186+0000", 2024-04-23T16:20:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.871186+0000", 2024-04-23T16:20:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1099999999999995e-07 2024-04-23T16:20:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.871189+0000", 2024-04-23T16:20:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8540000000000001e-06 2024-04-23T16:20:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.871195+0000", 2024-04-23T16:20:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0410000000000003e-06 2024-04-23T16:20:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.871201+0000", 2024-04-23T16:20:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2730000000000002e-06 2024-04-23T16:20:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.879324+0000", 2024-04-23T16:20:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0081237940000000002 2024-04-23T16:20:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.879434+0000", 2024-04-23T16:20:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000109425 2024-04-23T16:20:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.879553+0000", 2024-04-23T16:20:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011954499999999999 2024-04-23T16:20:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.888711+0000", 2024-04-23T16:20:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0091577550000000001 2024-04-23T16:20:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.893319+0000", 2024-04-23T16:20:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.004608015 2024-04-23T16:20:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.893354+0000", 2024-04-23T16:20:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4598999999999999e-05 2024-04-23T16:20:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.893383+0000", 2024-04-23T16:20:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9176e-05 2024-04-23T16:20:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:216255 2.7 2:e548a3a6:::benchmark_data_smithi188_77627_object46906:head [delete] snapc 0=[] ondisk+write+known_if_redirected e153)", 2024-04-23T16:20:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:20:01.874359+0000", 2024-04-23T16:20:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 1.0249686650000001, 2024-04-23T16:20:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022236411000000001, 2024-04-23T16:20:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 216255 2024-04-23T16:20:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.874359+0000", 2024-04-23T16:20:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.874359+0000", 2024-04-23T16:20:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.874361+0000", 2024-04-23T16:20:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.672e-06 2024-04-23T16:20:03.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:03.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.874366+0000", 2024-04-23T16:20:03.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.429e-06 2024-04-23T16:20:03.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:03.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.874378+0000", 2024-04-23T16:20:03.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1449e-05 2024-04-23T16:20:03.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.874386+0000", 2024-04-23T16:20:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2900000000000002e-06 2024-04-23T16:20:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.884328+0000", 2024-04-23T16:20:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0099420900000000006 2024-04-23T16:20:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.884438+0000", 2024-04-23T16:20:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010952500000000001 2024-04-23T16:20:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.884557+0000", 2024-04-23T16:20:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000119401 2024-04-23T16:20:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.896501+0000", 2024-04-23T16:20:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011943858 2024-04-23T16:20:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.896539+0000", 2024-04-23T16:20:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7605999999999999e-05 2024-04-23T16:20:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:03.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.896568+0000", 2024-04-23T16:20:03.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9496e-05 2024-04-23T16:20:03.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:03.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:03.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:03.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:01.896596+0000", 2024-04-23T16:20:03.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7594999999999999e-05 2024-04-23T16:20:03.042 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:03.042 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:03.042 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:03.042 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:03.042 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:03.042 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:20:03.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:02 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:20:03.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:02 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:20:03.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:02 smithi188 ceph-mon[44101]: osdmap e153: 12 total, 12 up, 12 in 2024-04-23T16:20:03.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:02 smithi188 ceph-mon[44101]: pgmap v467: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 997 op/s 2024-04-23T16:20:03.603 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:20:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:03.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:20:03.691 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-23T16:20:03.713 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:03.768 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:03.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:03.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:03.856 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:03 smithi103 ceph-mon[39629]: 2.1 scrub starts 2024-04-23T16:20:03.913 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:20:03.996 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:04.021 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:20:04.028 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:20:04.061 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:20:04.111 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:20:04.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:03 smithi148 ceph-mon[44520]: 2.1 scrub starts 2024-04-23T16:20:04.150 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:20:04.189 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:20:04.215 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:20:04.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:04.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:20:04.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:03 smithi188 ceph-mon[44101]: 2.1 scrub starts 2024-04-23T16:20:04.228 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:20:04.246 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:20:04.303 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:20:04.322 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:20:04.333 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:20:04.353 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:04.367 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:20:04.387 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:20:04.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:04.390 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:20:04.404 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:20:04.446 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:20:04.486 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:20:04.491 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:20:04.508 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:20:04.508 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:20:04.508 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:20:04.508 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:20:04.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:20:04.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:20:04.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:20:04.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:20:04.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:20:04.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:20:04.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:20:04.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:20:04.550 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:04.567 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:20:04.591 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:20:04.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:04 smithi103 ceph-mon[39629]: 2.1 scrub ok 2024-04-23T16:20:04.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:04 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:20:04.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:04 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:20:04.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:04 smithi103 ceph-mon[39629]: osdmap e154: 12 total, 12 up, 12 in 2024-04-23T16:20:04.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:04 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/429005459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:04.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:04.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:04.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:04.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:04.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:04.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:04 smithi103 ceph-mon[39629]: pgmap v469: 17 pgs: 17 active+clean; 7.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:20:04.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:04 smithi103 ceph-mon[39629]: from='client.34886 v1:172.21.15.103:0/543009452' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:20:04.918 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:20:04.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:04.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:20:05.028 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:20:05.057 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:04 smithi148 ceph-mon[44520]: 2.1 scrub ok 2024-04-23T16:20:05.057 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:04 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:20:05.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:04 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:20:05.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:04 smithi148 ceph-mon[44520]: osdmap e154: 12 total, 12 up, 12 in 2024-04-23T16:20:05.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:04 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/429005459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:05.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:04 smithi148 ceph-mon[44520]: pgmap v469: 17 pgs: 17 active+clean; 7.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:20:05.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:04 smithi148 ceph-mon[44520]: from='client.34886 v1:172.21.15.103:0/543009452' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:20:05.061 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:05.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:04 smithi188 ceph-mon[44101]: 2.1 scrub ok 2024-04-23T16:20:05.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:04 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:20:05.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:04 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:20:05.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:04 smithi188 ceph-mon[44101]: osdmap e154: 12 total, 12 up, 12 in 2024-04-23T16:20:05.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:04 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/429005459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:05.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:04 smithi188 ceph-mon[44101]: pgmap v469: 17 pgs: 17 active+clean; 7.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:20:05.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:04 smithi188 ceph-mon[44101]: from='client.34886 v1:172.21.15.103:0/543009452' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:20:05.413 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:20:05.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:05.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:20:05.518 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:20:05.562 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:05.626 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:20:05.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:05.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:20:05.739 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:20:05.791 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:05.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:05 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:20:05.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:05 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:20:05.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:05 smithi103 ceph-mon[39629]: 2.8 scrub starts 2024-04-23T16:20:05.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:05 smithi103 ceph-mon[39629]: osdmap e155: 12 total, 12 up, 12 in 2024-04-23T16:20:05.947 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:20:05.947 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:20:05.971 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.972 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.972 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:05 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:20:05.972 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:05 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:20:05.972 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.972 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.972 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.972 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.972 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:05 smithi148 ceph-mon[44520]: 2.8 scrub starts 2024-04-23T16:20:05.972 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:05.972 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:05 smithi148 ceph-mon[44520]: osdmap e155: 12 total, 12 up, 12 in 2024-04-23T16:20:06.002 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:20:06.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:06.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:20:06.030 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":469,"stamp":"2024-04-23T16:20:04.499291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8011776568,"num_objects":122244,"num_object_clones":0,"num_object_copies":244488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":122244,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":216633,"num_write_kb":10841083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160918,"ondisk_log_size":160918,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":16546736,"kb_used_data":15741572,"kb_used_omap":18,"kb_used_meta":804333,"kb_avail":1108395088,"statfs":{"total":1151940427776,"available":1134996570112,"internally_reserved":0,"allocated":16119369728,"data_stored":16115238184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":823637305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,11,10,8],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-337051648,"num_objects":-5143,"num_object_clones":0,"num_object_copies":-10286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5143,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.768426"},"pg_stats":[{"pgid":"2.f","version":"153'13448","reported_seq":13644,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-23T16:20:02.902688+0000","last_change":"2024-04-23T16:19:01.946211+0000","last_active":"2024-04-23T16:20:02.902688+0000","last_peered":"2024-04-23T16:20:02.902688+0000","last_clean":"2024-04-23T16:20:02.902688+0000","last_became_active":"2024-04-23T16:17:36.792080+0000","last_became_peered":"2024-04-23T16:17:36.792080+0000","last_unstale":"2024-04-23T16:20:02.902688+0000","last_undegraded":"2024-04-23T16:20:02.902688+0000","last_fullsized":"2024-04-23T16:20:02.902688+0000","mapping_epoch":110,"log_start":"120'3400","ondisk_log_start":"120'3400","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'9285","last_scrub_stamp":"2024-04-23T16:19:01.946056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:17:35.778609+0000","last_clean_scrub_stamp":"2024-04-23T16:19:01.946056+0000","objects_scrubbed":9222,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:20:30.886884+0000","scrub_duration":1845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":494141440,"num_objects":7540,"num_object_clones":0,"num_object_copies":15080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13448,"num_write_kb":671616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"152'13353","reported_seq":13546,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-04-23T16:20:01.234802+0000","last_change":"2024-04-23T16:19:00.384248+0000","last_active":"2024-04-23T16:20:01.234802+0000","last_peered":"2024-04-23T16:20:01.234802+0000","last_clean":"2024-04-23T16:20:01.234802+0000","last_became_active":"2024-04-23T16:17:36.794888+0000","last_became_peered":"2024-04-23T16:17:36.794888+0000","last_unstale":"2024-04-23T16:20:01.234802+0000","last_undegraded":"2024-04-23T16:20:01.234802+0000","last_fullsized":"2024-04-23T16:20:01.234802+0000","mapping_epoch":110,"log_start":"119'3300","ondisk_log_start":"119'3300","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'9240","last_scrub_stamp":"2024-04-23T16:19:00.384091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:17:35.778609+0000","last_clean_scrub_stamp":"2024-04-23T16:19:00.384091+0000","objects_scrubbed":9167,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:20:16.247811+0000","scrub_duration":2020,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":510853120,"num_objects":7795,"num_object_clones":0,"num_object_copies":15590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13353,"num_write_kb":676736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"153'18972","reported_seq":19438,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-23T16:20:02.543647+0000","last_change":"2024-04-23T16:19:55.200158+0000","last_active":"2024-04-23T16:20:02.543647+0000","last_peered":"2024-04-23T16:20:02.543647+0000","last_clean":"2024-04-23T16:20:02.543647+0000","last_became_active":"2024-04-23T16:17:36.793246+0000","last_became_peered":"2024-04-23T16:17:36.793246+0000","last_unstale":"2024-04-23T16:20:02.543647+0000","last_undegraded":"2024-04-23T16:20:02.543647+0000","last_fullsized":"2024-04-23T16:20:02.543647+0000","mapping_epoch":110,"log_start":"140'8901","ondisk_log_start":"140'8901","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'18418","last_scrub_stamp":"2024-04-23T16:19:55.199901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:17:35.778609+0000","last_clean_scrub_stamp":"2024-04-23T16:19:55.199901+0000","objects_scrubbed":8085,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:21:05.663727+0000","scrub_duration":1407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490864640,"num_objects":7490,"num_object_clones":0,"num_object_copies":14980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13442,"num_write_kb":669824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"154'13534","reported_seq":13812,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-23T16:20:03.800745+0000","last_change":"2024-04-23T16:20:03.579805+0000","last_active":"2024-04-23T16:20:03.800745+0000","last_peered":"2024-04-23T16:20:03.800745+0000","last_clean":"2024-04-23T16:20:03.800745+0000","last_became_active":"2024-04-23T16:17:36.880310+0000","last_became_peered":"2024-04-23T16:17:36.880310+0000","last_unstale":"2024-04-23T16:20:03.800745+0000","last_undegraded":"2024-04-23T16:20:03.800745+0000","last_fullsized":"2024-04-23T16:20:03.800745+0000","mapping_epoch":110,"log_start":"120'3500","ondisk_log_start":"120'3500","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'13517","last_scrub_stamp":"2024-04-23T16:20:03.579638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:17:35.778609+0000","last_clean_scrub_stamp":"2024-04-23T16:20:03.579638+0000","objects_scrubbed":7538,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:21:32.372401+0000","scrub_duration":1417,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490733568,"num_objects":7488,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13534,"num_write_kb":672704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"153'13476","reported_seq":13675,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-23T16:20:02.603628+0000","last_change":"2024-04-23T16:19:06.019465+0000","last_active":"2024-04-23T16:20:02.603628+0000","last_peered":"2024-04-23T16:20:02.603628+0000","last_clean":"2024-04-23T16:20:02.603628+0000","last_became_active":"2024-04-23T16:17:36.793267+0000","last_became_peered":"2024-04-23T16:17:36.793267+0000","last_unstale":"2024-04-23T16:20:02.603628+0000","last_undegraded":"2024-04-23T16:20:02.603628+0000","last_fullsized":"2024-04-23T16:20:02.603628+0000","mapping_epoch":110,"log_start":"119'3400","ondisk_log_start":"119'3400","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'9574","last_scrub_stamp":"2024-04-23T16:19:06.019366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:17:35.778609+0000","last_clean_scrub_stamp":"2024-04-23T16:19:06.019366+0000","objects_scrubbed":9527,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:20:15.505936+0000","scrub_duration":1474,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488767488,"num_objects":7458,"num_object_clones":0,"num_object_copies":14916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13476,"num_write_kb":669888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'137","reported_seq":456,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-23T16:19:53.679429+0000","last_change":"2024-04-23T16:18:53.355363+0000","last_active":"2024-04-23T16:19:53.679429+0000","last_peered":"2024-04-23T16:19:53.679429+0000","last_clean":"2024-04-23T16:19:53.679429+0000","last_became_active":"2024-04-23T16:15:36.918552+0000","last_became_peered":"2024-04-23T16:15:36.918552+0000","last_unstale":"2024-04-23T16:19:53.679429+0000","last_undegraded":"2024-04-23T16:19:53.679429+0000","last_fullsized":"2024-04-23T16:19:53.679429+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"16'137","last_scrub_stamp":"2024-04-23T16:18:53.355280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:11:23.401899+0000","last_clean_scrub_stamp":"2024-04-23T16:18:53.355280+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:20:19.494051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"150'13411","reported_seq":13605,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-23T16:19:59.550499+0000","last_change":"2024-04-23T16:19:07.154139+0000","last_active":"2024-04-23T16:19:59.550499+0000","last_peered":"2024-04-23T16:19:59.550499+0000","last_clean":"2024-04-23T16:19:59.550499+0000","last_became_active":"2024-04-23T16:17:36.880654+0000","last_became_peered":"2024-04-23T16:17:36.880654+0000","last_unstale":"2024-04-23T16:19:59.550499+0000","last_undegraded":"2024-04-23T16:19:59.550499+0000","last_fullsized":"2024-04-23T16:19:59.550499+0000","mapping_epoch":110,"log_start":"119'3400","ondisk_log_start":"119'3400","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'9739","last_scrub_stamp":"2024-04-23T16:19:07.154000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:17:35.778609+0000","last_clean_scrub_stamp":"2024-04-23T16:19:07.154000+0000","objects_scrubbed":9686,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:20:17.538827+0000","scrub_duration":1677,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":509673472,"num_objects":7777,"num_object_clones":0,"num_object_copies":15554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13411,"num_write_kb":678016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"153'13557","reported_seq":13753,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-23T16:20:02.854009+0000","last_change":"2024-04-23T16:19:00.954839+0000","last_active":"2024-04-23T16:20:02.854009+0000","last_peered":"2024-04-23T16:20:02.854009+0000","last_clean":"2024-04-23T16:20:02.854009+0000","last_became_active":"2024-04-23T16:17:36.788991+0000","last_became_peered":"2024-04-23T16:17:36.788991+0000","last_unstale":"2024-04-23T16:20:02.854009+0000","last_undegraded":"2024-04-23T16:20:02.854009+0000","last_fullsized":"2024-04-23T16:20:02.854009+0000","mapping_epoch":110,"log_start":"120'3500","ondisk_log_start":"120'3500","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'9276","last_scrub_stamp":"2024-04-23T16:19:00.954697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:17:35.778609+0000","last_clean_scrub_stamp":"2024-04-23T16:19:00.954697+0000","objects_scrubbed":9211,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:20:27.233526+0000","scrub_duration":1859,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":497352704,"num_objects":7589,"num_object_clones":0,"num_object_copies":15178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13557,"num_write_kb":676672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"152'13460","reported_seq":13665,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-04-23T16:20:01.258688+0000","last_change":"2024-04-23T16:19:15.628613+0000","last_active":"2024-04-23T16:20:01.258688+0000","last_peered":"2024-04-23T16:20:01.258688+0000","last_clean":"2024-04-23T16:20:01.258688+0000","last_became_active":"2024-04-23T16:17:36.793094+0000","last_became_peered":"2024-04-23T16:17:36.793094+0000","last_unstale":"2024-04-23T16:20:01.258688+0000","last_undegraded":"2024-04-23T16:20:01.258688+0000","last_fullsized":"2024-04-23T16:20:01.258688+0000","mapping_epoch":110,"log_start":"119'3400","ondisk_log_start":"119'3400","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10207","last_scrub_stamp":"2024-04-23T16:19:15.628518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:17:35.778609+0000","last_clean_scrub_stamp":"2024-04-23T16:19:15.628518+0000","objects_scrubbed":10160,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:20:42.121246+0000","scrub_duration":2181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":500170752,"num_objects":7632,"num_object_clones":0,"num_object_copies":15264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13460,"num_write_kb":674944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"154'13852","reported_seq":14045,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-23T16:20:03.972799+0000","last_change":"2024-04-23T16:18:53.212720+0000","last_active":"2024-04-23T16:20:03.972799+0000","last_peered":"2024-04-23T16:20:03.972799+0000","last_clean":"2024-04-23T16:20:03.972799+0000","last_became_active":"2024-04-23T16:17:36.880648+0000","last_became_peered":"2024-04-23T16:17:36.880648+0000","last_unstale":"2024-04-23T16:20:03.972799+0000","last_undegraded":"2024-04-23T16:20:03.972799+0000","last_fullsized":"2024-04-23T16:20:03.972799+0000","mapping_epoch":110,"log_start":"120'3800","ondisk_log_start":"120'3800","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'8815","last_scrub_stamp":"2024-04-23T16:18:53.212516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:17:35.778609+0000","last_clean_scrub_stamp":"2024-04-23T16:18:53.212516+0000","objects_scrubbed":8767,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:20:11.426034+0000","scrub_duration":1227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490471424,"num_objects":7484,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13852,"num_write_kb":682752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"150'13413","reported_seq":13604,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-23T16:19:59.549790+0000","last_change":"2024-04-23T16:19:01.891576+0000","last_active":"2024-04-23T16:19:59.549790+0000","last_peered":"2024-04-23T16:19:59.549790+0000","last_clean":"2024-04-23T16:19:59.549790+0000","last_became_active":"2024-04-23T16:17:36.793547+0000","last_became_peered":"2024-04-23T16:17:36.793547+0000","last_unstale":"2024-04-23T16:19:59.549790+0000","last_undegraded":"2024-04-23T16:19:59.549790+0000","last_fullsized":"2024-04-23T16:19:59.549790+0000","mapping_epoch":110,"log_start":"119'3400","ondisk_log_start":"119'3400","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'9352","last_scrub_stamp":"2024-04-23T16:19:01.891387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:17:35.778609+0000","last_clean_scrub_stamp":"2024-04-23T16:19:01.891387+0000","objects_scrubbed":9304,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:20:02.835669+0000","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514785280,"num_objects":7855,"num_object_clones":0,"num_object_copies":15710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13413,"num_write_kb":680576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"152'13642","reported_seq":13847,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-04-23T16:20:01.261769+0000","last_change":"2024-04-23T16:19:14.781844+0000","last_active":"2024-04-23T16:20:01.261769+0000","last_peered":"2024-04-23T16:20:01.261769+0000","last_clean":"2024-04-23T16:20:01.261769+0000","last_became_active":"2024-04-23T16:17:36.793071+0000","last_became_peered":"2024-04-23T16:17:36.793071+0000","last_unstale":"2024-04-23T16:20:01.261769+0000","last_undegraded":"2024-04-23T16:20:01.261769+0000","last_fullsized":"2024-04-23T16:20:01.261769+0000","mapping_epoch":110,"log_start":"120'3600","ondisk_log_start":"120'3600","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10402","last_scrub_stamp":"2024-04-23T16:19:14.781687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:17:35.778609+0000","last_clean_scrub_stamp":"2024-04-23T16:19:14.781687+0000","objects_scrubbed":10325,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:20:25.433242+0000","scrub_duration":2328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":519438336,"num_objects":7926,"num_object_clones":0,"num_object_copies":15852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13642,"num_write_kb":690176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"152'13417","reported_seq":13612,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-04-23T16:20:00.778615+0000","last_change":"2024-04-23T16:19:01.242459+0000","last_active":"2024-04-23T16:20:00.778615+0000","last_peered":"2024-04-23T16:20:00.778615+0000","last_clean":"2024-04-23T16:20:00.778615+0000","last_became_active":"2024-04-23T16:17:36.880183+0000","last_became_peered":"2024-04-23T16:17:36.880183+0000","last_unstale":"2024-04-23T16:20:00.778615+0000","last_undegraded":"2024-04-23T16:20:00.778615+0000","last_fullsized":"2024-04-23T16:20:00.778615+0000","mapping_epoch":110,"log_start":"120'3400","ondisk_log_start":"120'3400","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'9356","last_scrub_stamp":"2024-04-23T16:19:01.242227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:17:35.778609+0000","last_clean_scrub_stamp":"2024-04-23T16:19:01.242227+0000","objects_scrubbed":9319,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:20:04.255731+0000","scrub_duration":1842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":516227072,"num_objects":7877,"num_object_clones":0,"num_object_copies":15754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13417,"num_write_kb":681408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"153'13796","reported_seq":13994,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-23T16:20:02.891565+0000","last_change":"2024-04-23T16:19:03.589389+0000","last_active":"2024-04-23T16:20:02.891565+0000","last_peered":"2024-04-23T16:20:02.891565+0000","last_clean":"2024-04-23T16:20:02.891565+0000","last_became_active":"2024-04-23T16:17:36.792052+0000","last_became_peered":"2024-04-23T16:17:36.792052+0000","last_unstale":"2024-04-23T16:20:02.891565+0000","last_undegraded":"2024-04-23T16:20:02.891565+0000","last_fullsized":"2024-04-23T16:20:02.891565+0000","mapping_epoch":110,"log_start":"120'3700","ondisk_log_start":"120'3700","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'9601","last_scrub_stamp":"2024-04-23T16:19:03.589253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:17:35.778609+0000","last_clean_scrub_stamp":"2024-04-23T16:19:03.589253+0000","objects_scrubbed":9566,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:20:15.594626+0000","scrub_duration":1472,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505282560,"num_objects":7710,"num_object_clones":0,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13796,"num_write_kb":688192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"152'13362","reported_seq":13563,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-04-23T16:20:00.748104+0000","last_change":"2024-04-23T16:19:10.260105+0000","last_active":"2024-04-23T16:20:00.748104+0000","last_peered":"2024-04-23T16:20:00.748104+0000","last_clean":"2024-04-23T16:20:00.748104+0000","last_became_active":"2024-04-23T16:17:36.794930+0000","last_became_peered":"2024-04-23T16:17:36.794930+0000","last_unstale":"2024-04-23T16:20:00.748104+0000","last_undegraded":"2024-04-23T16:20:00.748104+0000","last_fullsized":"2024-04-23T16:20:00.748104+0000","mapping_epoch":110,"log_start":"119'3300","ondisk_log_start":"119'3300","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'9891","last_scrub_stamp":"2024-04-23T16:19:10.259984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:17:35.778609+0000","last_clean_scrub_stamp":"2024-04-23T16:19:10.259984+0000","objects_scrubbed":9826,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:20:29.626196+0000","scrub_duration":1870,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506462208,"num_objects":7728,"num_object_clones":0,"num_object_copies":15456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13362,"num_write_kb":674880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"153'13656","reported_seq":13847,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-23T16:20:03.669229+0000","last_change":"2024-04-23T16:18:52.985292+0000","last_active":"2024-04-23T16:20:03.669229+0000","last_peered":"2024-04-23T16:20:03.669229+0000","last_clean":"2024-04-23T16:20:03.669229+0000","last_became_active":"2024-04-23T16:17:36.792846+0000","last_became_peered":"2024-04-23T16:17:36.792846+0000","last_unstale":"2024-04-23T16:20:03.669229+0000","last_undegraded":"2024-04-23T16:20:03.669229+0000","last_fullsized":"2024-04-23T16:20:03.669229+0000","mapping_epoch":110,"log_start":"120'3600","ondisk_log_start":"120'3600","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'8677","last_scrub_stamp":"2024-04-23T16:18:52.985084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:17:35.778609+0000","last_clean_scrub_stamp":"2024-04-23T16:18:52.985084+0000","objects_scrubbed":8629,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:20:21.289897+0000","scrub_duration":1388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484507672,"num_objects":7394,"num_object_clones":0,"num_object_copies":14788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7394,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13656,"num_write_kb":673537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"153'13633","reported_seq":13916,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-23T16:20:03.781333+0000","last_change":"2024-04-23T16:19:59.878702+0000","last_active":"2024-04-23T16:20:03.781333+0000","last_peered":"2024-04-23T16:20:03.781333+0000","last_clean":"2024-04-23T16:20:03.781333+0000","last_became_active":"2024-04-23T16:17:36.880222+0000","last_became_peered":"2024-04-23T16:17:36.880222+0000","last_unstale":"2024-04-23T16:20:03.781333+0000","last_undegraded":"2024-04-23T16:20:03.781333+0000","last_fullsized":"2024-04-23T16:20:03.781333+0000","mapping_epoch":110,"log_start":"120'3600","ondisk_log_start":"120'3600","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'13336","last_scrub_stamp":"2024-04-23T16:19:59.878528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:17:35.778609+0000","last_clean_scrub_stamp":"2024-04-23T16:19:59.878528+0000","objects_scrubbed":7851,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:21:09.570878+0000","scrub_duration":1687,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491454464,"num_objects":7499,"num_object_clones":0,"num_object_copies":14998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13633,"num_write_kb":676224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8011186200,"num_objects":122242,"num_object_clones":0,"num_object_copies":244484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":122242,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":216452,"num_write_kb":10838145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16030834688,"data_stored":16030826544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160781,"ondisk_log_size":160781,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056696,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1576004,"kb_used_data":1508608,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92169148,"statfs":{"total":95995035648,"available":94381207552,"internally_reserved":0,"allocated":1544814592,"data_stored":1544485632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":68876740},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351038,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1049836,"kb_used_data":1000612,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92695316,"statfs":{"total":95995035648,"available":94920003584,"internally_reserved":0,"allocated":1024626688,"data_stored":1024240870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612677,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1509408,"kb_used_data":1437972,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92235744,"statfs":{"total":95995035648,"available":94449401856,"internally_reserved":0,"allocated":1472483328,"data_stored":1472141764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841608,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2042716,"kb_used_data":1961424,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":91702436,"statfs":{"total":95995035648,"available":93903294464,"internally_reserved":0,"allocated":2008498176,"data_stored":2008160684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168653,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1596212,"kb_used_data":1520620,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92148940,"statfs":{"total":95995035648,"available":94360514560,"internally_reserved":0,"allocated":1557114880,"data_stored":1556725530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462995,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":515372,"kb_used_data":480028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93229780,"statfs":{"total":95995035648,"available":95467294720,"internally_reserved":0,"allocated":491548672,"data_stored":491205128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2099452,"kb_used_data":1995832,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":91645700,"statfs":{"total":95995035648,"available":93845196800,"internally_reserved":0,"allocated":2043731968,"data_stored":2043406580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738481,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1526044,"kb_used_data":1458704,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92219108,"statfs":{"total":95995035648,"available":94432366592,"internally_reserved":0,"allocated":1493712896,"data_stored":1493375404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313318,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1512280,"kb_used_data":1440848,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92232872,"statfs":{"total":95995035648,"available":94446460928,"internally_reserved":0,"allocated":1475428352,"data_stored":1475090860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117089,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1573436,"kb_used_data":1497720,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92171716,"statfs":{"total":95995035648,"available":94383837184,"internally_reserved":0,"allocated":1533665280,"data_stored":1533339892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920860,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":505692,"kb_used_data":474516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93239460,"statfs":{"total":95995035648,"available":95477207040,"internally_reserved":0,"allocated":485904384,"data_stored":485562820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1040284,"kb_used_data":964688,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92704868,"statfs":{"total":95995035648,"available":94929784832,"internally_reserved":0,"allocated":987840512,"data_stored":987503020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1481900032,"data_stored":1481900032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2027421696,"data_stored":2027421696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1465647104,"data_stored":1465647104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1530200064,"data_stored":1530200064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":483004416,"data_stored":483000344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":979369984,"data_stored":979369984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":490471424,"data_stored":490471424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1553661952,"data_stored":1553661952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1989083136,"data_stored":1989083136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1470435328,"data_stored":1470431256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1020985344,"data_stored":1020985344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1538654208,"data_stored":1538654208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:20:06.031 INFO:tasks.thrashosds.thrasher:Setting 2.1 to [7, 0, 5, 10] 2024-04-23T16:20:06.031 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.1', '7', '0', '5', '10'] 2024-04-23T16:20:06.032 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pg-upmap-items 2.1 7 0 5 10 2024-04-23T16:20:06.111 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:20:06.148 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:06.186 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:20:06.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:06.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:20:06.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:06.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:06.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:05 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:20:06.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:05 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:20:06.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:06.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:06.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:06.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:06.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:05 smithi188 ceph-mon[44101]: 2.8 scrub starts 2024-04-23T16:20:06.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:06.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:05 smithi188 ceph-mon[44101]: osdmap e155: 12 total, 12 up, 12 in 2024-04-23T16:20:06.289 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:20:06.318 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:06.656 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-23T16:20:06.671 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:20:06.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:06.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:20:06.774 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:06.805 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:06.826 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:06.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:06.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:06.931 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:06.962 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:06 smithi148 ceph-mon[44520]: 2.8 scrub ok 2024-04-23T16:20:06.962 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:06 smithi148 ceph-mon[44520]: pgmap v471: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.35k op/s 2024-04-23T16:20:06.965 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:06.988 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:06.990 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:20:06.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:06.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:06.997 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:06 smithi103 ceph-mon[39629]: 2.8 scrub ok 2024-04-23T16:20:06.997 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:06 smithi103 ceph-mon[39629]: pgmap v471: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.35k op/s 2024-04-23T16:20:07.092 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:07.121 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:07.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:06 smithi188 ceph-mon[44101]: 2.8 scrub ok 2024-04-23T16:20:07.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:06 smithi188 ceph-mon[44101]: pgmap v471: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.35k op/s 2024-04-23T16:20:07.253 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:07.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:07.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:07.358 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:07.390 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:07.626 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:20:07.798 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:07 smithi103 ceph-mon[39629]: 2.a scrub starts 2024-04-23T16:20:08.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:07 smithi148 ceph-mon[44520]: 2.a scrub starts 2024-04-23T16:20:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:07 smithi188 ceph-mon[44101]: 2.a scrub starts 2024-04-23T16:20:08.847 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:20:09.032 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:20:09.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:08 smithi103 ceph-mon[39629]: 2.a scrub ok 2024-04-23T16:20:09.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:08 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2327108918' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:20:09.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:08 smithi103 ceph-mon[39629]: from='client.34892 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:20:09.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:08 smithi103 ceph-mon[39629]: pgmap v472: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:20:09.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:08 smithi148 ceph-mon[44520]: 2.a scrub ok 2024-04-23T16:20:09.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:08 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2327108918' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:20:09.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:08 smithi148 ceph-mon[44520]: from='client.34892 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:20:09.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:08 smithi148 ceph-mon[44520]: pgmap v472: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:20:09.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:08 smithi188 ceph-mon[44101]: 2.a scrub ok 2024-04-23T16:20:09.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:08 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2327108918' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:20:09.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:08 smithi188 ceph-mon[44101]: from='client.34892 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:20:09.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:08 smithi188 ceph-mon[44101]: pgmap v472: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:20:09.947 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:10.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:10.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:09 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:20:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:09 smithi103 ceph-mon[39629]: from='client.34892 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:20:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:09 smithi103 ceph-mon[39629]: osdmap e156: 12 total, 12 up, 12 in 2024-04-23T16:20:10.102 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:10.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:09 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:20:10.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:09 smithi148 ceph-mon[44520]: from='client.34892 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:20:10.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:09 smithi148 ceph-mon[44520]: osdmap e156: 12 total, 12 up, 12 in 2024-04-23T16:20:10.137 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:10.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:09 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:20:10.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:09 smithi188 ceph-mon[44101]: from='client.34892 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:20:10.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:09 smithi188 ceph-mon[44101]: osdmap e156: 12 total, 12 up, 12 in 2024-04-23T16:20:11.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:10 smithi103 ceph-mon[39629]: pgmap v474: 17 pgs: 17 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:20:11.121 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:20:11.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:10 smithi148 ceph-mon[44520]: pgmap v474: 17 pgs: 17 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:20:11.188 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:11.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:11.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:11.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:10 smithi188 ceph-mon[44101]: pgmap v474: 17 pgs: 17 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:20:11.294 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:20:11.345 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:11.452 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:20:11.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:11.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:20:11.556 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:20:11.595 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:11.857 INFO:teuthology.orchestra.run.smithi103.stderr:set 2.1 pg_upmap_items mapping to [7->0,5->10] 2024-04-23T16:20:12.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:11 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/598326964' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.1", "id": [7, 0, 5, 10]}]: dispatch 2024-04-23T16:20:12.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:11 smithi103 ceph-mon[39629]: from='client.34898 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.1", "id": [7, 0, 5, 10]}]: dispatch 2024-04-23T16:20:12.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:11 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/598326964' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.1", "id": [7, 0, 5, 10]}]: dispatch 2024-04-23T16:20:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:11 smithi148 ceph-mon[44520]: from='client.34898 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.1", "id": [7, 0, 5, 10]}]: dispatch 2024-04-23T16:20:12.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:11 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/598326964' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.1", "id": [7, 0, 5, 10]}]: dispatch 2024-04-23T16:20:12.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:11 smithi188 ceph-mon[44101]: from='client.34898 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.1", "id": [7, 0, 5, 10]}]: dispatch 2024-04-23T16:20:12.364 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:20:12.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:12.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:20:12.471 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:20:12.527 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:12.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:12 smithi103 ceph-mon[39629]: from='client.34898 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.1", "id": [7, 0, 5, 10]}]': finished 2024-04-23T16:20:12.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:12 smithi103 ceph-mon[39629]: osdmap e157: 12 total, 12 up, 12 in 2024-04-23T16:20:12.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:12 smithi103 ceph-mon[39629]: pgmap v476: 17 pgs: 17 active+clean; 6.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-04-23T16:20:13.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:12 smithi148 ceph-mon[44520]: from='client.34898 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.1", "id": [7, 0, 5, 10]}]': finished 2024-04-23T16:20:13.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:12 smithi148 ceph-mon[44520]: osdmap e157: 12 total, 12 up, 12 in 2024-04-23T16:20:13.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:12 smithi148 ceph-mon[44520]: pgmap v476: 17 pgs: 17 active+clean; 6.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-04-23T16:20:13.151 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:20:13.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:12 smithi188 ceph-mon[44101]: from='client.34898 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.1", "id": [7, 0, 5, 10]}]': finished 2024-04-23T16:20:13.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:12 smithi188 ceph-mon[44101]: osdmap e157: 12 total, 12 up, 12 in 2024-04-23T16:20:13.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:12 smithi188 ceph-mon[44101]: pgmap v476: 17 pgs: 17 active+clean; 6.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-04-23T16:20:13.391 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:20:13.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:13.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:20:13.506 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:20:13.541 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:13.692 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:20:13.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:13.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:20:13.798 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:20:13.828 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:13.930 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:20:13.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:13.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:20:14.035 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:20:14.067 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:14.088 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:20:14.088 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:20:14.091 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:20:14.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:14.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:20:14.107 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:20:14.119 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:20:14.161 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:20:14.197 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:20:14.207 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:20:14.237 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:14.243 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:20:14.268 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:20:14.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:14.274 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:20:14.280 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:20:14.312 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:20:14.345 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:20:14.375 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:20:14.389 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:20:14.439 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:14.440 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:20:14.468 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:20:14.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:14.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:20:14.487 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:20:14.528 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:20:14.568 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:20:14.575 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:20:14.577 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:20:14.577 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:20:14.577 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:20:14.577 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:20:14.577 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:20:14.578 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:20:14.578 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:20:14.578 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:20:14.578 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:20:14.578 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:20:14.578 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:20:14.578 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:20:14.608 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:14.759 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:20:14.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:14.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:20:14.788 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-23T16:20:14.864 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:20:14.877 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:14 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1190730513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:14.877 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:14.877 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:14.877 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:14.877 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:14.878 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:14.878 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:14.878 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:14 smithi103 ceph-mon[39629]: pgmap v477: 17 pgs: 17 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:20:14.903 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:14.931 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:20:14.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:14.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:20:15.036 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:20:15.082 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:14 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1190730513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:14 smithi148 ceph-mon[44520]: pgmap v477: 17 pgs: 17 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:20:15.127 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:20:15.130 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:14 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1190730513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:15.130 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:15.130 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:15.130 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:15.130 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:15.130 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:15.130 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:15.130 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:14 smithi188 ceph-mon[44101]: pgmap v477: 17 pgs: 17 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:20:15.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:15.131 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:20:15.185 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:20:15.231 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:20:15.285 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:15.757 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:20:15.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:15.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:20:15.867 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:20:15.930 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:16.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:16.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:16.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:16.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:16.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:16.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:16.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:16.693 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:20:16.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:16.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:20:16.798 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:20:16.830 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:16.862 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:20:16.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:16.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:20:16.970 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:20:17.007 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:16 smithi103 ceph-mon[39629]: pgmap v478: 17 pgs: 17 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:20:17.009 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:17.025 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:20:17.025 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2024-04-23T16:20:17.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2024-04-23T16:20:17.026 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:20:17.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.026 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:147947 2.8 2:140b8018:::benchmark_data_smithi188_77627_object147946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e140)", 2024-04-23T16:20:17.026 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:00.176326+0000", 2024-04-23T16:20:17.026 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 76.846363259, 2024-04-23T16:20:17.026 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033195334, 2024-04-23T16:20:17.026 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.026 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.026 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.026 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.026 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.026 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.026 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 147947 2024-04-23T16:20:17.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.027 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.176326+0000", 2024-04-23T16:20:17.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.176326+0000", 2024-04-23T16:20:17.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.176328+0000", 2024-04-23T16:20:17.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6470000000000001e-06 2024-04-23T16:20:17.028 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.028 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.176386+0000", 2024-04-23T16:20:17.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8310999999999998e-05 2024-04-23T16:20:17.028 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.028 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.176396+0000", 2024-04-23T16:20:17.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0164e-05 2024-04-23T16:20:17.028 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.029 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.029 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.176402+0000", 2024-04-23T16:20:17.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7409999999999998e-06 2024-04-23T16:20:17.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.029 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.029 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.179828+0000", 2024-04-23T16:20:17.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0034257710000000002 2024-04-23T16:20:17.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.029 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.029 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.179911+0000", 2024-04-23T16:20:17.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3312000000000005e-05 2024-04-23T16:20:17.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:20:17.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.180011+0000", 2024-04-23T16:20:17.030 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9877000000000006e-05 2024-04-23T16:20:17.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.182964+0000", 2024-04-23T16:20:17.030 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0029525279999999998 2024-04-23T16:20:17.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.031 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.209357+0000", 2024-04-23T16:20:17.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026393308000000001 2024-04-23T16:20:17.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.031 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.031 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.209476+0000", 2024-04-23T16:20:17.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000118652 2024-04-23T16:20:17.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.031 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.031 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.209522+0000", 2024-04-23T16:20:17.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6023000000000003e-05 2024-04-23T16:20:17.032 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.032 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.032 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.032 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.032 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:148471 2.8 2:1d2a53db:::benchmark_data_smithi188_77627_object148470:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-04-23T16:20:17.032 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:00.717309+0000", 2024-04-23T16:20:17.032 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 76.305380455999995, 2024-04-23T16:20:17.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.030129982999999999, 2024-04-23T16:20:17.032 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.032 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.032 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.032 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.032 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.033 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.033 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 148471 2024-04-23T16:20:17.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.033 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.717309+0000", 2024-04-23T16:20:17.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.717309+0000", 2024-04-23T16:20:17.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.034 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.717310+0000", 2024-04-23T16:20:17.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.95e-07 2024-04-23T16:20:17.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.034 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.717383+0000", 2024-04-23T16:20:17.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2917999999999995e-05 2024-04-23T16:20:17.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.034 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.717391+0000", 2024-04-23T16:20:17.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2209999999999997e-06 2024-04-23T16:20:17.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.717398+0000", 2024-04-23T16:20:17.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2010000000000002e-06 2024-04-23T16:20:17.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.724743+0000", 2024-04-23T16:20:17.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0073442500000000001 2024-04-23T16:20:17.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.724830+0000", 2024-04-23T16:20:17.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7293999999999995e-05 2024-04-23T16:20:17.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:20:17.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.724922+0000", 2024-04-23T16:20:17.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2003999999999995e-05 2024-04-23T16:20:17.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.728595+0000", 2024-04-23T16:20:17.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0036733970000000001 2024-04-23T16:20:17.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.747344+0000", 2024-04-23T16:20:17.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018748694 2024-04-23T16:20:17.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.037 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.747385+0000", 2024-04-23T16:20:17.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1090000000000001e-05 2024-04-23T16:20:17.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.037 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.747439+0000", 2024-04-23T16:20:17.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3919000000000003e-05 2024-04-23T16:20:17.037 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.037 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.037 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.037 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:148489 2.8 2:12abcfa7:::benchmark_data_smithi188_77627_object148488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-04-23T16:20:17.037 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:00.734932+0000", 2024-04-23T16:20:17.038 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 76.287757209999995, 2024-04-23T16:20:17.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033275978999999997, 2024-04-23T16:20:17.038 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.038 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.038 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.038 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.038 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.038 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.038 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 148489 2024-04-23T16:20:17.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.038 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.734932+0000", 2024-04-23T16:20:17.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.734932+0000", 2024-04-23T16:20:17.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.734933+0000", 2024-04-23T16:20:17.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.113e-06 2024-04-23T16:20:17.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.735018+0000", 2024-04-23T16:20:17.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4163e-05 2024-04-23T16:20:17.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.735029+0000", 2024-04-23T16:20:17.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0908e-05 2024-04-23T16:20:17.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.735037+0000", 2024-04-23T16:20:17.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2050000000000002e-06 2024-04-23T16:20:17.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.750710+0000", 2024-04-23T16:20:17.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015672878000000001 2024-04-23T16:20:17.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.750788+0000", 2024-04-23T16:20:17.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7953000000000004e-05 2024-04-23T16:20:17.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:20:17.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.750879+0000", 2024-04-23T16:20:17.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1658999999999998e-05 2024-04-23T16:20:17.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.758664+0000", 2024-04-23T16:20:17.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0077852390000000002 2024-04-23T16:20:17.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.768137+0000", 2024-04-23T16:20:17.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0094727460000000006 2024-04-23T16:20:17.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.768172+0000", 2024-04-23T16:20:17.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4879000000000003e-05 2024-04-23T16:20:17.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:00.768208+0000", 2024-04-23T16:20:17.043 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6235999999999997e-05 2024-04-23T16:20:17.043 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.043 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.043 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.043 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:148797 2.0 2:0dd08a2b:::benchmark_data_smithi188_77627_object148796:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-04-23T16:20:17.043 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:01.031553+0000", 2024-04-23T16:20:17.043 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 75.991136588000003, 2024-04-23T16:20:17.043 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.036261416999999997, 2024-04-23T16:20:17.043 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.043 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.043 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.044 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.044 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.044 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.044 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 148797 2024-04-23T16:20:17.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.044 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.044 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.031553+0000", 2024-04-23T16:20:17.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.044 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.031553+0000", 2024-04-23T16:20:17.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.045 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.031554+0000", 2024-04-23T16:20:17.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2780000000000001e-06 2024-04-23T16:20:17.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.045 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.031613+0000", 2024-04-23T16:20:17.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8328999999999997e-05 2024-04-23T16:20:17.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.045 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.031621+0000", 2024-04-23T16:20:17.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5900000000000008e-06 2024-04-23T16:20:17.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.046 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.031626+0000", 2024-04-23T16:20:17.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7520000000000004e-06 2024-04-23T16:20:17.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.046 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.036571+0000", 2024-04-23T16:20:17.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0049450340000000001 2024-04-23T16:20:17.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.036686+0000", 2024-04-23T16:20:17.047 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000115497 2024-04-23T16:20:17.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:17.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.036835+0000", 2024-04-23T16:20:17.047 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014898999999999999 2024-04-23T16:20:17.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.040120+0000", 2024-04-23T16:20:17.047 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003284824 2024-04-23T16:20:17.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.048 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.048 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.067712+0000", 2024-04-23T16:20:17.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.027591984 2024-04-23T16:20:17.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.048 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.048 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.048 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.067768+0000", 2024-04-23T16:20:17.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5546e-05 2024-04-23T16:20:17.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.048 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.048 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.048 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.067814+0000", 2024-04-23T16:20:17.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6592999999999999e-05 2024-04-23T16:20:17.048 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.049 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.049 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.049 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.049 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.049 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:149036 2.8 2:158f8c8b:::benchmark_data_smithi188_77627_object149035:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-04-23T16:20:17.049 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:01.243177+0000", 2024-04-23T16:20:17.049 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 75.779512076000003, 2024-04-23T16:20:17.049 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.031388639000000003, 2024-04-23T16:20:17.049 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.049 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.049 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.050 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.050 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.050 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.050 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 149036 2024-04-23T16:20:17.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.050 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.050 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.243177+0000", 2024-04-23T16:20:17.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.243177+0000", 2024-04-23T16:20:17.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.243179+0000", 2024-04-23T16:20:17.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.11e-06 2024-04-23T16:20:17.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.243238+0000", 2024-04-23T16:20:17.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9614000000000002e-05 2024-04-23T16:20:17.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.052 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.052 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.243246+0000", 2024-04-23T16:20:17.052 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2670000000000006e-06 2024-04-23T16:20:17.052 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.052 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.052 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.243251+0000", 2024-04-23T16:20:17.052 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0069999999999997e-06 2024-04-23T16:20:17.052 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.053 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.053 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.249433+0000", 2024-04-23T16:20:17.053 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0061812170000000001 2024-04-23T16:20:17.053 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.053 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.053 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.053 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.249560+0000", 2024-04-23T16:20:17.053 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012690800000000001 2024-04-23T16:20:17.053 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.053 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.053 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:20:17.053 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.249708+0000", 2024-04-23T16:20:17.053 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014802000000000001 2024-04-23T16:20:17.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.054 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.054 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.258601+0000", 2024-04-23T16:20:17.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0088931700000000006 2024-04-23T16:20:17.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.054 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.054 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.274416+0000", 2024-04-23T16:20:17.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015815218999999998 2024-04-23T16:20:17.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.054 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.274494+0000", 2024-04-23T16:20:17.055 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8213000000000005e-05 2024-04-23T16:20:17.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:01.274566+0000", 2024-04-23T16:20:17.055 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1894e-05 2024-04-23T16:20:17.055 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.055 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.055 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.055 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:154763 2.0 2:04ddd741:::benchmark_data_smithi188_77627_object154762:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-04-23T16:20:17.056 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:06.431718+0000", 2024-04-23T16:20:17.056 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 70.590971814, 2024-04-23T16:20:17.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.034493602999999998, 2024-04-23T16:20:17.056 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.056 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.056 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.056 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.056 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.056 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.056 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 154763 2024-04-23T16:20:17.056 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.056 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.056 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.056 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.056 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.431718+0000", 2024-04-23T16:20:17.057 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.057 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.057 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.057 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.057 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.431718+0000", 2024-04-23T16:20:17.057 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.057 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.057 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.057 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.057 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.431719+0000", 2024-04-23T16:20:17.057 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3099999999999996e-07 2024-04-23T16:20:17.057 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.057 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.057 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.058 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.431755+0000", 2024-04-23T16:20:17.058 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6709999999999999e-05 2024-04-23T16:20:17.058 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.058 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.058 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.058 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.431762+0000", 2024-04-23T16:20:17.058 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0419999999999997e-06 2024-04-23T16:20:17.058 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.058 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.058 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.058 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.431768+0000", 2024-04-23T16:20:17.058 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9889999999999997e-06 2024-04-23T16:20:17.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.059 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.059 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.059 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.442931+0000", 2024-04-23T16:20:17.059 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011162681000000001 2024-04-23T16:20:17.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.059 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.059 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.059 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.443017+0000", 2024-04-23T16:20:17.059 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6018000000000002e-05 2024-04-23T16:20:17.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.059 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.059 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:17.059 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.443149+0000", 2024-04-23T16:20:17.059 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013226000000000001 2024-04-23T16:20:17.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.060 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.060 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.446376+0000", 2024-04-23T16:20:17.060 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032272339999999998 2024-04-23T16:20:17.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.060 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.060 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.466091+0000", 2024-04-23T16:20:17.060 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019714211999999998 2024-04-23T16:20:17.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.060 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.060 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.466147+0000", 2024-04-23T16:20:17.060 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6677e-05 2024-04-23T16:20:17.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.061 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.061 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.061 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.466211+0000", 2024-04-23T16:20:17.061 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3948999999999994e-05 2024-04-23T16:20:17.061 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.061 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.061 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.061 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.061 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.061 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:154818 2.0 2:07784266:::benchmark_data_smithi188_77627_object154817:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-04-23T16:20:17.061 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:06.481542+0000", 2024-04-23T16:20:17.061 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 70.541147293999998, 2024-04-23T16:20:17.061 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.031017791999999999, 2024-04-23T16:20:17.061 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.062 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.062 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.062 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.062 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.062 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.062 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 154818 2024-04-23T16:20:17.062 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.062 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.062 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.062 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.062 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.481542+0000", 2024-04-23T16:20:17.062 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.062 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.062 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.063 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.063 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.481542+0000", 2024-04-23T16:20:17.063 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.063 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.063 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.063 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.063 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.481543+0000", 2024-04-23T16:20:17.063 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.540000000000001e-07 2024-04-23T16:20:17.063 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.063 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.063 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.063 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.481596+0000", 2024-04-23T16:20:17.064 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.325e-05 2024-04-23T16:20:17.064 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.064 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.064 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.064 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.481605+0000", 2024-04-23T16:20:17.064 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9539999999999993e-06 2024-04-23T16:20:17.064 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.064 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.064 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.064 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.481612+0000", 2024-04-23T16:20:17.064 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8850000000000001e-06 2024-04-23T16:20:17.064 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.064 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.064 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.064 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.498156+0000", 2024-04-23T16:20:17.064 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016543500999999999 2024-04-23T16:20:17.065 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.065 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.065 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.065 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.498298+0000", 2024-04-23T16:20:17.065 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000142168 2024-04-23T16:20:17.065 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.065 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.065 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:17.065 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.498420+0000", 2024-04-23T16:20:17.065 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012251300000000001 2024-04-23T16:20:17.065 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.065 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.065 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.065 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.502734+0000", 2024-04-23T16:20:17.065 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0043135409999999997 2024-04-23T16:20:17.065 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.065 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.066 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.066 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.512434+0000", 2024-04-23T16:20:17.066 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0096995699999999994 2024-04-23T16:20:17.066 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.066 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.066 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.066 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.512496+0000", 2024-04-23T16:20:17.066 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2476000000000003e-05 2024-04-23T16:20:17.066 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.066 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.066 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.066 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.512560+0000", 2024-04-23T16:20:17.066 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3979999999999994e-05 2024-04-23T16:20:17.067 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.067 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.067 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.067 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.067 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.067 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:154822 2.8 2:175592d2:::benchmark_data_smithi188_77627_object154821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-04-23T16:20:17.067 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:06.487512+0000", 2024-04-23T16:20:17.067 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 70.535177809999993, 2024-04-23T16:20:17.067 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.030046883999999999, 2024-04-23T16:20:17.067 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.067 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.067 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.067 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.067 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.068 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.068 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 154822 2024-04-23T16:20:17.068 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.068 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.068 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.068 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.068 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.487512+0000", 2024-04-23T16:20:17.068 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.068 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.068 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.068 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.068 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.487512+0000", 2024-04-23T16:20:17.068 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.069 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.069 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.069 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.487514+0000", 2024-04-23T16:20:17.069 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5620000000000002e-06 2024-04-23T16:20:17.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.069 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.069 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.069 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.487554+0000", 2024-04-23T16:20:17.069 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9882999999999997e-05 2024-04-23T16:20:17.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.069 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.069 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.070 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.487570+0000", 2024-04-23T16:20:17.070 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5647000000000001e-05 2024-04-23T16:20:17.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.070 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.070 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.070 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.487579+0000", 2024-04-23T16:20:17.070 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8580000000000006e-06 2024-04-23T16:20:17.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.070 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.070 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.070 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.502710+0000", 2024-04-23T16:20:17.070 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015131225999999999 2024-04-23T16:20:17.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.071 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.071 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.071 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.502838+0000", 2024-04-23T16:20:17.071 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012782200000000001 2024-04-23T16:20:17.071 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.071 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.071 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:20:17.071 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.502982+0000", 2024-04-23T16:20:17.071 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014418100000000001 2024-04-23T16:20:17.071 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.071 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.071 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.072 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.512523+0000", 2024-04-23T16:20:17.072 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0095414030000000004 2024-04-23T16:20:17.072 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.072 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.072 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.072 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.517438+0000", 2024-04-23T16:20:17.072 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0049148280000000004 2024-04-23T16:20:17.072 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.072 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.072 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.072 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.517506+0000", 2024-04-23T16:20:17.072 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7983000000000003e-05 2024-04-23T16:20:17.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.073 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.073 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.073 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.517559+0000", 2024-04-23T16:20:17.073 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2491000000000003e-05 2024-04-23T16:20:17.073 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.073 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.073 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.073 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.073 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:154824 2.8 2:17205abd:::benchmark_data_smithi188_77627_object154823:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-04-23T16:20:17.073 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:06.488725+0000", 2024-04-23T16:20:17.074 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 70.533964178000005, 2024-04-23T16:20:17.074 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033618821, 2024-04-23T16:20:17.074 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.074 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.074 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.074 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.074 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.074 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.074 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 154824 2024-04-23T16:20:17.074 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.074 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.074 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.488725+0000", 2024-04-23T16:20:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.488725+0000", 2024-04-23T16:20:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.488727+0000", 2024-04-23T16:20:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.452e-06 2024-04-23T16:20:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.076 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.076 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.488785+0000", 2024-04-23T16:20:17.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7899000000000002e-05 2024-04-23T16:20:17.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.076 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.076 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.076 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.488796+0000", 2024-04-23T16:20:17.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1547e-05 2024-04-23T16:20:17.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.076 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.076 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.076 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.488803+0000", 2024-04-23T16:20:17.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.002e-06 2024-04-23T16:20:17.077 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.507734+0000", 2024-04-23T16:20:17.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018930509000000002 2024-04-23T16:20:17.077 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.507861+0000", 2024-04-23T16:20:17.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000126919 2024-04-23T16:20:17.077 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.078 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.078 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:20:17.078 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.507991+0000", 2024-04-23T16:20:17.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012990199999999999 2024-04-23T16:20:17.078 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.078 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.078 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.078 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.517368+0000", 2024-04-23T16:20:17.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0093778200000000003 2024-04-23T16:20:17.078 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.078 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.079 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.079 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.522249+0000", 2024-04-23T16:20:17.079 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0048809630000000003 2024-04-23T16:20:17.079 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.079 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.079 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.079 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.522301+0000", 2024-04-23T16:20:17.079 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1703999999999998e-05 2024-04-23T16:20:17.079 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.079 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.079 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.079 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.522344+0000", 2024-04-23T16:20:17.079 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3103999999999999e-05 2024-04-23T16:20:17.079 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.080 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.080 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.080 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.080 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.080 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:154830 2.8 2:1aeb45fd:::benchmark_data_smithi188_77627_object154829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-04-23T16:20:17.080 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:06.494164+0000", 2024-04-23T16:20:17.080 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 70.528526002999996, 2024-04-23T16:20:17.080 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.032316875000000002, 2024-04-23T16:20:17.080 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.080 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.080 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.080 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.080 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.081 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.081 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 154830 2024-04-23T16:20:17.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.081 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.081 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.081 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.494164+0000", 2024-04-23T16:20:17.081 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.081 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.081 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.494164+0000", 2024-04-23T16:20:17.081 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.494165+0000", 2024-04-23T16:20:17.082 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9230000000000001e-06 2024-04-23T16:20:17.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.082 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.494196+0000", 2024-04-23T16:20:17.082 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0978000000000001e-05 2024-04-23T16:20:17.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.082 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.494205+0000", 2024-04-23T16:20:17.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3480000000000004e-06 2024-04-23T16:20:17.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.494213+0000", 2024-04-23T16:20:17.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7510000000000008e-06 2024-04-23T16:20:17.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.512558+0000", 2024-04-23T16:20:17.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018345441000000001 2024-04-23T16:20:17.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.512701+0000", 2024-04-23T16:20:17.084 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014315700000000001 2024-04-23T16:20:17.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.084 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:20:17.084 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.512858+0000", 2024-04-23T16:20:17.084 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015729000000000001 2024-04-23T16:20:17.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.084 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.084 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.517406+0000", 2024-04-23T16:20:17.084 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045473320000000003 2024-04-23T16:20:17.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.084 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.084 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.526377+0000", 2024-04-23T16:20:17.085 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0089717200000000007 2024-04-23T16:20:17.085 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.085 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.085 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.085 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.526429+0000", 2024-04-23T16:20:17.085 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1109000000000001e-05 2024-04-23T16:20:17.085 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.085 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.085 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.085 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.526480+0000", 2024-04-23T16:20:17.085 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1826000000000002e-05 2024-04-23T16:20:17.085 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.085 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.085 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.085 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.086 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.086 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:155394 2.8 2:18f78d2d:::benchmark_data_smithi188_77627_object155393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-04-23T16:20:17.086 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:06.991233+0000", 2024-04-23T16:20:17.086 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 70.031456698, 2024-04-23T16:20:17.086 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.03000456, 2024-04-23T16:20:17.086 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.086 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.086 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.086 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.086 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.086 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.086 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 155394 2024-04-23T16:20:17.087 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.087 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.087 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.087 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.087 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.991233+0000", 2024-04-23T16:20:17.087 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.087 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.087 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.087 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.087 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.991233+0000", 2024-04-23T16:20:17.087 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.087 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.088 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.088 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.088 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.991234+0000", 2024-04-23T16:20:17.088 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4300000000000001e-06 2024-04-23T16:20:17.088 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.088 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.088 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.088 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.991263+0000", 2024-04-23T16:20:17.088 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8612999999999999e-05 2024-04-23T16:20:17.088 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.088 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.088 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.088 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.991270+0000", 2024-04-23T16:20:17.088 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.956e-06 2024-04-23T16:20:17.088 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.089 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.089 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.089 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.991276+0000", 2024-04-23T16:20:17.089 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8869999999999997e-06 2024-04-23T16:20:17.089 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.089 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.089 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.089 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.998915+0000", 2024-04-23T16:20:17.089 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0076395869999999998 2024-04-23T16:20:17.089 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.089 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.089 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.089 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.999049+0000", 2024-04-23T16:20:17.090 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000133478 2024-04-23T16:20:17.090 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.090 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.090 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:20:17.090 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:06.999208+0000", 2024-04-23T16:20:17.090 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000159252 2024-04-23T16:20:17.090 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.090 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.090 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.090 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:07.004173+0000", 2024-04-23T16:20:17.090 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0049646450000000002 2024-04-23T16:20:17.090 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.090 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.091 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.091 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:07.021172+0000", 2024-04-23T16:20:17.091 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016999457999999999 2024-04-23T16:20:17.091 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.091 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.091 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.091 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:07.021208+0000", 2024-04-23T16:20:17.091 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5565999999999998e-05 2024-04-23T16:20:17.091 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.091 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.091 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.091 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:07.021237+0000", 2024-04-23T16:20:17.091 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9688000000000001e-05 2024-04-23T16:20:17.091 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.092 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.092 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.092 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.092 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.092 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:183778 2.8 2:1622770e:::benchmark_data_smithi188_77627_object14429:head [delete] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-04-23T16:20:17.092 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:35.103861+0000", 2024-04-23T16:20:17.092 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 41.918828122999997, 2024-04-23T16:20:17.092 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.030503832000000002, 2024-04-23T16:20:17.092 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.092 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.092 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.092 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.092 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.092 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.092 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 183778 2024-04-23T16:20:17.092 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.093 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.093 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.093 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.093 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:35.103861+0000", 2024-04-23T16:20:17.093 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.093 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.093 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.093 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.093 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:35.103861+0000", 2024-04-23T16:20:17.093 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.093 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.093 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.093 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.093 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:35.103863+0000", 2024-04-23T16:20:17.093 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.331e-06 2024-04-23T16:20:17.094 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.094 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.094 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.094 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:35.103867+0000", 2024-04-23T16:20:17.094 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0879999999999997e-06 2024-04-23T16:20:17.094 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.094 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.094 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.094 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:35.103876+0000", 2024-04-23T16:20:17.094 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5049999999999996e-06 2024-04-23T16:20:17.094 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.094 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.094 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.095 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:35.103882+0000", 2024-04-23T16:20:17.095 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3000000000000001e-06 2024-04-23T16:20:17.095 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.095 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.095 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.095 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:35.103930+0000", 2024-04-23T16:20:17.095 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8418999999999998e-05 2024-04-23T16:20:17.095 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.095 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.095 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.095 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:35.104056+0000", 2024-04-23T16:20:17.095 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012576800000000001 2024-04-23T16:20:17.095 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.095 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.096 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:20:17.096 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:35.104175+0000", 2024-04-23T16:20:17.096 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000119411 2024-04-23T16:20:17.096 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.096 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.096 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.096 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:35.107956+0000", 2024-04-23T16:20:17.096 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0037806570000000002 2024-04-23T16:20:17.096 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.096 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.096 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.096 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:35.134297+0000", 2024-04-23T16:20:17.096 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026341046 2024-04-23T16:20:17.096 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.097 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.097 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.097 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:35.134328+0000", 2024-04-23T16:20:17.097 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1257000000000002e-05 2024-04-23T16:20:17.097 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.097 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.097 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.097 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:35.134365+0000", 2024-04-23T16:20:17.097 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7049999999999999e-05 2024-04-23T16:20:17.097 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.097 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.097 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.097 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.098 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.098 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:207607 2.8 2:1a781d4e:::benchmark_data_smithi188_77627_object38258:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-04-23T16:20:17.098 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:19:54.705877+0000", 2024-04-23T16:20:17.098 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 22.316812305999999, 2024-04-23T16:20:17.098 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.035743946999999998, 2024-04-23T16:20:17.098 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.098 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.098 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.098 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.098 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.098 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.098 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 207607 2024-04-23T16:20:17.099 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.099 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.099 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.099 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.099 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.705877+0000", 2024-04-23T16:20:17.099 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.099 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.099 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.099 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.099 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.705877+0000", 2024-04-23T16:20:17.099 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.100 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.100 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.100 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.100 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.705878+0000", 2024-04-23T16:20:17.100 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9900000000000009e-07 2024-04-23T16:20:17.100 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.100 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.100 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.100 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.705882+0000", 2024-04-23T16:20:17.100 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8840000000000004e-06 2024-04-23T16:20:17.100 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.100 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.101 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.101 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.705889+0000", 2024-04-23T16:20:17.101 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9750000000000001e-06 2024-04-23T16:20:17.101 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.101 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.101 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.101 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.705895+0000", 2024-04-23T16:20:17.101 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7490000000000004e-06 2024-04-23T16:20:17.101 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.101 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.101 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.101 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.718091+0000", 2024-04-23T16:20:17.102 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012195691 2024-04-23T16:20:17.102 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.102 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.102 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.102 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.718210+0000", 2024-04-23T16:20:17.102 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011961200000000001 2024-04-23T16:20:17.102 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.102 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.102 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:20:17.102 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.718324+0000", 2024-04-23T16:20:17.102 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000113453 2024-04-23T16:20:17.103 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.103 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.103 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.103 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.732085+0000", 2024-04-23T16:20:17.103 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013761565 2024-04-23T16:20:17.103 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.103 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.103 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.103 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.741505+0000", 2024-04-23T16:20:17.103 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0094195010000000003 2024-04-23T16:20:17.103 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.103 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.103 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.103 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.741567+0000", 2024-04-23T16:20:17.103 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2463000000000001e-05 2024-04-23T16:20:17.104 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.104 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.104 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.104 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:19:54.741621+0000", 2024-04-23T16:20:17.104 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4055000000000003e-05 2024-04-23T16:20:17.104 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.104 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.104 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.104 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.104 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.104 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:214998 2.0 2:0caa206a:::benchmark_data_smithi188_77627_object45649:head [delete] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-04-23T16:20:17.104 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:20:00.855857+0000", 2024-04-23T16:20:17.105 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 16.166832914, 2024-04-23T16:20:17.105 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.036863092, 2024-04-23T16:20:17.105 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.105 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.105 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.105 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.105 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.105 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.105 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 214998 2024-04-23T16:20:17.105 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.105 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.105 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.105 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.105 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.855857+0000", 2024-04-23T16:20:17.106 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.106 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.106 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.106 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.106 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.855857+0000", 2024-04-23T16:20:17.106 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.106 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.106 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.106 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.106 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.855858+0000", 2024-04-23T16:20:17.106 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.282e-06 2024-04-23T16:20:17.106 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.106 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.106 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.107 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.855862+0000", 2024-04-23T16:20:17.107 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1949999999999996e-06 2024-04-23T16:20:17.107 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.107 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.107 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.107 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.855873+0000", 2024-04-23T16:20:17.107 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0995999999999999e-05 2024-04-23T16:20:17.107 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.107 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.107 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.107 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.855880+0000", 2024-04-23T16:20:17.107 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5780000000000004e-06 2024-04-23T16:20:17.107 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.107 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.108 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.108 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.874390+0000", 2024-04-23T16:20:17.108 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018510090999999999 2024-04-23T16:20:17.108 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.108 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.108 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.108 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.874510+0000", 2024-04-23T16:20:17.108 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000119886 2024-04-23T16:20:17.108 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.108 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.108 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:17.108 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.874625+0000", 2024-04-23T16:20:17.108 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000115149 2024-04-23T16:20:17.108 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.109 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.109 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.109 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.879575+0000", 2024-04-23T16:20:17.109 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0049503000000000004 2024-04-23T16:20:17.109 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.109 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.109 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.109 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.892622+0000", 2024-04-23T16:20:17.109 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013047034000000001 2024-04-23T16:20:17.109 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.109 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.109 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.109 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.892670+0000", 2024-04-23T16:20:17.109 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7375e-05 2024-04-23T16:20:17.110 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.110 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.110 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.110 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.892720+0000", 2024-04-23T16:20:17.110 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0206e-05 2024-04-23T16:20:17.110 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.110 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.110 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.110 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.110 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.110 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:215002 2.0 2:0d1e142c:::benchmark_data_smithi188_77627_object45653:head [delete] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-04-23T16:20:17.110 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:20:00.857382+0000", 2024-04-23T16:20:17.110 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 16.165307990999999, 2024-04-23T16:20:17.111 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.043320686999999997, 2024-04-23T16:20:17.111 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.111 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.111 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.111 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.111 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.111 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.111 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 215002 2024-04-23T16:20:17.111 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.111 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.111 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.111 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.112 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.857382+0000", 2024-04-23T16:20:17.112 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.112 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.112 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.112 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.112 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.857382+0000", 2024-04-23T16:20:17.112 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.112 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.112 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.112 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.112 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.857383+0000", 2024-04-23T16:20:17.112 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6390000000000001e-06 2024-04-23T16:20:17.112 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.112 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.113 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.113 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.857389+0000", 2024-04-23T16:20:17.113 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3519999999999997e-06 2024-04-23T16:20:17.113 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.113 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.113 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.113 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.857403+0000", 2024-04-23T16:20:17.113 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4137e-05 2024-04-23T16:20:17.113 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.113 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.113 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.113 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.857411+0000", 2024-04-23T16:20:17.113 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1159999999999999e-06 2024-04-23T16:20:17.113 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.114 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.114 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.114 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.879618+0000", 2024-04-23T16:20:17.114 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022206857999999999 2024-04-23T16:20:17.114 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.114 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.114 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.114 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.879705+0000", 2024-04-23T16:20:17.114 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7107000000000005e-05 2024-04-23T16:20:17.114 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.114 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.114 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:17.115 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.879779+0000", 2024-04-23T16:20:17.115 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4627000000000002e-05 2024-04-23T16:20:17.115 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.115 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.115 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.115 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.888291+0000", 2024-04-23T16:20:17.115 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0085114609999999997 2024-04-23T16:20:17.115 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.115 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.115 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.115 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.900640+0000", 2024-04-23T16:20:17.115 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012349549 2024-04-23T16:20:17.115 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.115 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.116 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.116 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.900671+0000", 2024-04-23T16:20:17.116 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0131e-05 2024-04-23T16:20:17.116 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.116 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.116 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.116 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.900702+0000", 2024-04-23T16:20:17.116 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.171e-05 2024-04-23T16:20:17.116 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.116 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.116 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.116 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.116 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.116 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:215006 2.8 2:183f4d0b:::benchmark_data_smithi188_77627_object45657:head [delete] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-04-23T16:20:17.117 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:20:00.862320+0000", 2024-04-23T16:20:17.117 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 16.160369975999998, 2024-04-23T16:20:17.117 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.038430978999999997, 2024-04-23T16:20:17.117 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.117 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.117 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.117 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.117 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.117 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.117 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 215006 2024-04-23T16:20:17.117 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.117 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.118 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.118 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.118 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.862320+0000", 2024-04-23T16:20:17.118 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.118 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.118 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.118 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.118 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.862320+0000", 2024-04-23T16:20:17.118 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.118 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.118 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.118 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.118 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.862322+0000", 2024-04-23T16:20:17.119 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3549999999999999e-06 2024-04-23T16:20:17.119 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.119 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.119 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.119 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.862328+0000", 2024-04-23T16:20:17.119 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7250000000000002e-06 2024-04-23T16:20:17.119 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.119 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.119 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.119 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.862342+0000", 2024-04-23T16:20:17.119 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4642e-05 2024-04-23T16:20:17.119 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.119 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.119 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.119 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.862351+0000", 2024-04-23T16:20:17.120 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6980000000000004e-06 2024-04-23T16:20:17.120 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.120 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.120 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.120 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.879610+0000", 2024-04-23T16:20:17.120 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.017259514 2024-04-23T16:20:17.120 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.120 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.120 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.120 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.879719+0000", 2024-04-23T16:20:17.120 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010868 2024-04-23T16:20:17.120 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.120 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.120 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:20:17.121 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.879826+0000", 2024-04-23T16:20:17.121 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107118 2024-04-23T16:20:17.121 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.121 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.121 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.121 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.884482+0000", 2024-04-23T16:20:17.121 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0046556519999999997 2024-04-23T16:20:17.121 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.121 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.121 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.121 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.900708+0000", 2024-04-23T16:20:17.121 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016225977999999999 2024-04-23T16:20:17.121 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.121 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.122 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.122 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.900731+0000", 2024-04-23T16:20:17.122 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3195999999999999e-05 2024-04-23T16:20:17.122 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.122 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.122 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.122 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.900751+0000", 2024-04-23T16:20:17.122 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9420999999999999e-05 2024-04-23T16:20:17.122 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.122 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.122 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.122 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.122 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.122 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:215010 2.8 2:11423aae:::benchmark_data_smithi188_77627_object45661:head [delete] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-04-23T16:20:17.123 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:20:00.864806+0000", 2024-04-23T16:20:17.123 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 16.157883092999999, 2024-04-23T16:20:17.123 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.035989650999999998, 2024-04-23T16:20:17.123 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.123 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.123 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.123 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.123 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.123 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.123 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 215010 2024-04-23T16:20:17.123 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.123 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.124 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.124 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.124 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.864806+0000", 2024-04-23T16:20:17.124 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.124 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.124 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.124 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.124 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.864806+0000", 2024-04-23T16:20:17.124 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.124 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.124 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.124 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.124 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.864808+0000", 2024-04-23T16:20:17.124 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9209999999999999e-06 2024-04-23T16:20:17.124 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.125 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.125 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.125 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.864814+0000", 2024-04-23T16:20:17.125 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5210000000000002e-06 2024-04-23T16:20:17.125 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.125 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.125 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.125 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.864826+0000", 2024-04-23T16:20:17.125 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1688000000000001e-05 2024-04-23T16:20:17.125 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.125 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.125 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.125 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.864834+0000", 2024-04-23T16:20:17.126 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0290000000000005e-06 2024-04-23T16:20:17.126 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.126 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.126 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.126 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.884516+0000", 2024-04-23T16:20:17.126 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019682049 2024-04-23T16:20:17.126 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.126 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.126 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.126 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.884648+0000", 2024-04-23T16:20:17.126 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013280599999999999 2024-04-23T16:20:17.126 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.126 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.126 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:20:17.126 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.884725+0000", 2024-04-23T16:20:17.127 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6286999999999997e-05 2024-04-23T16:20:17.127 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.127 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.127 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.127 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.900670+0000", 2024-04-23T16:20:17.127 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015945727999999999 2024-04-23T16:20:17.127 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.127 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.127 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.127 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.900755+0000", 2024-04-23T16:20:17.127 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4598000000000006e-05 2024-04-23T16:20:17.127 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.127 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.127 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.127 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.900777+0000", 2024-04-23T16:20:17.127 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1562000000000001e-05 2024-04-23T16:20:17.128 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.128 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.128 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.128 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.900796+0000", 2024-04-23T16:20:17.128 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9462000000000001e-05 2024-04-23T16:20:17.128 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.128 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.128 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.128 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.128 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.128 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:215020 2.8 2:15413584:::benchmark_data_smithi188_77627_object45671:head [delete] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-04-23T16:20:17.128 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:20:00.873846+0000", 2024-04-23T16:20:17.128 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 16.148843214999999, 2024-04-23T16:20:17.128 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.031275456, 2024-04-23T16:20:17.128 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.129 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.129 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.129 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.129 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.129 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.129 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 215020 2024-04-23T16:20:17.129 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.129 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.129 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.129 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.129 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.873846+0000", 2024-04-23T16:20:17.129 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.129 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.129 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.129 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.130 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.873846+0000", 2024-04-23T16:20:17.130 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.130 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.130 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.130 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.130 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.873849+0000", 2024-04-23T16:20:17.130 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4530000000000001e-06 2024-04-23T16:20:17.130 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.130 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.130 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.130 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.873857+0000", 2024-04-23T16:20:17.130 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7929999999999998e-06 2024-04-23T16:20:17.130 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.131 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.131 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.131 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.873873+0000", 2024-04-23T16:20:17.131 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6370000000000001e-05 2024-04-23T16:20:17.131 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.131 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.131 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.131 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.873884+0000", 2024-04-23T16:20:17.131 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0750999999999999e-05 2024-04-23T16:20:17.131 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.131 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.131 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.131 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.888287+0000", 2024-04-23T16:20:17.131 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01440309 2024-04-23T16:20:17.131 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.131 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.132 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.132 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.888390+0000", 2024-04-23T16:20:17.132 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010310900000000001 2024-04-23T16:20:17.132 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.132 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.132 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:20:17.132 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.888483+0000", 2024-04-23T16:20:17.132 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2964e-05 2024-04-23T16:20:17.132 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.132 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.132 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.132 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.900825+0000", 2024-04-23T16:20:17.132 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012342517000000001 2024-04-23T16:20:17.132 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.133 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.133 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.133 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.905005+0000", 2024-04-23T16:20:17.133 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0041792560000000001 2024-04-23T16:20:17.133 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.133 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.133 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.133 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.905067+0000", 2024-04-23T16:20:17.133 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2756999999999996e-05 2024-04-23T16:20:17.133 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.133 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.133 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.133 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:00.905122+0000", 2024-04-23T16:20:17.133 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4395999999999999e-05 2024-04-23T16:20:17.133 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.134 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.134 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.134 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.134 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.134 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:218458 2.8 2:1faabfbe:::benchmark_data_smithi188_77627_object49109:head [delete] snapc 0=[] ondisk+write+known_if_redirected e153)", 2024-04-23T16:20:17.134 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:20:03.760425+0000", 2024-04-23T16:20:17.134 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 13.262264349000001, 2024-04-23T16:20:17.134 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.032274275999999998, 2024-04-23T16:20:17.134 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.134 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.134 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.134 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.134 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.134 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.134 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 218458 2024-04-23T16:20:17.134 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.135 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.135 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.135 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.135 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:03.760425+0000", 2024-04-23T16:20:17.135 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.135 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.135 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.135 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.135 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:03.760425+0000", 2024-04-23T16:20:17.135 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.135 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.135 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.135 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.135 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:03.760426+0000", 2024-04-23T16:20:17.136 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.71e-07 2024-04-23T16:20:17.136 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.136 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.136 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.136 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:03.760429+0000", 2024-04-23T16:20:17.136 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.19e-06 2024-04-23T16:20:17.136 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.136 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.136 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.136 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:03.760438+0000", 2024-04-23T16:20:17.136 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3280000000000006e-06 2024-04-23T16:20:17.136 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.136 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.136 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.137 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:03.760449+0000", 2024-04-23T16:20:17.137 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1722000000000001e-05 2024-04-23T16:20:17.137 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.137 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.137 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.137 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:03.770200+0000", 2024-04-23T16:20:17.137 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0097511319999999992 2024-04-23T16:20:17.137 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.137 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.137 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.137 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:03.770317+0000", 2024-04-23T16:20:17.137 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000116932 2024-04-23T16:20:17.137 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.137 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.138 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:20:17.138 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:03.770400+0000", 2024-04-23T16:20:17.138 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2528999999999995e-05 2024-04-23T16:20:17.138 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.138 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.138 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.138 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:03.787185+0000", 2024-04-23T16:20:17.138 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016784757000000001 2024-04-23T16:20:17.138 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.138 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.138 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.138 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:03.792615+0000", 2024-04-23T16:20:17.138 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0054308270000000001 2024-04-23T16:20:17.139 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.139 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.139 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.139 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:03.792662+0000", 2024-04-23T16:20:17.139 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6217000000000002e-05 2024-04-23T16:20:17.139 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.139 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.139 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.139 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:03.792699+0000", 2024-04-23T16:20:17.139 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7771000000000001e-05 2024-04-23T16:20:17.139 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.140 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.140 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.140 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.140 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.140 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:231867 2.0 2:00cd3fbe:::benchmark_data_smithi188_77627_object62518:head [delete] snapc 0=[] ondisk+write+known_if_redirected e157)", 2024-04-23T16:20:17.140 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:20:14.883405+0000", 2024-04-23T16:20:17.140 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 2.1392841470000001, 2024-04-23T16:20:17.140 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.030957097999999999, 2024-04-23T16:20:17.140 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:20:17.140 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:20:17.140 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:17.140 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:20:17.140 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:20:17.140 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:17.141 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 231867 2024-04-23T16:20:17.141 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.141 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:20:17.141 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.141 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:20:17.141 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:14.883405+0000", 2024-04-23T16:20:17.141 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.141 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.141 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.141 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:20:17.141 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:14.883405+0000", 2024-04-23T16:20:17.141 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:20:17.141 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.141 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.141 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:20:17.142 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:14.883406+0000", 2024-04-23T16:20:17.142 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.4900000000000004e-07 2024-04-23T16:20:17.142 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.142 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.142 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:20:17.142 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:14.883410+0000", 2024-04-23T16:20:17.142 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.303e-06 2024-04-23T16:20:17.142 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.142 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.142 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:20:17.142 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:14.883417+0000", 2024-04-23T16:20:17.142 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6459999999999993e-06 2024-04-23T16:20:17.142 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.142 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.143 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:20:17.143 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:14.883423+0000", 2024-04-23T16:20:17.143 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6049999999999997e-06 2024-04-23T16:20:17.143 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.143 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.143 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:20:17.143 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:14.898139+0000", 2024-04-23T16:20:17.143 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01471627 2024-04-23T16:20:17.143 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.143 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.143 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:20:17.143 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:14.898275+0000", 2024-04-23T16:20:17.143 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013551700000000001 2024-04-23T16:20:17.143 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.144 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.144 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 10", 2024-04-23T16:20:17.144 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:14.898401+0000", 2024-04-23T16:20:17.144 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012633300000000001 2024-04-23T16:20:17.144 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.144 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.144 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:20:17.144 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:14.910068+0000", 2024-04-23T16:20:17.144 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011667023 2024-04-23T16:20:17.144 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.144 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.144 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:20:17.144 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:14.914303+0000", 2024-04-23T16:20:17.145 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0042345459999999996 2024-04-23T16:20:17.145 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.145 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.145 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:20:17.145 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:14.914332+0000", 2024-04-23T16:20:17.145 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.902e-05 2024-04-23T16:20:17.145 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:20:17.145 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:20:17.145 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:20:17.145 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:14.914362+0000", 2024-04-23T16:20:17.145 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0886000000000002e-05 2024-04-23T16:20:17.145 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.146 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.146 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.146 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:20:17.146 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:20:17.146 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:20:17.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:17.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:16 smithi148 ceph-mon[44520]: pgmap v478: 17 pgs: 17 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:20:17.148 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:20:17.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:20:17.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:16 smithi188 ceph-mon[44101]: pgmap v478: 17 pgs: 17 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:20:17.249 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:20:17.323 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:17.454 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:20:17.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:17.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:20:17.560 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:20:17.593 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:17.631 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:20:17.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:17.638 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:20:17.739 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:20:17.778 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-23T16:20:17.805 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:17.832 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:20:17.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:17.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:20:17.939 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:20:18.018 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:18.024 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-23T16:20:18.024 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:20:18.024 INFO:tasks.thrashosds.thrasher:inject_pause on osd.8 2024-04-23T16:20:18.024 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-23T16:20:18.024 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-23T16:20:18.024 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 config set filestore_inject_stall 3 2024-04-23T16:20:18.074 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:20:18.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:18.077 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:20:18.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:20:18.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:20:18.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:20:18.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:20:18.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:20:18.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:17 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/830172937' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:20:18.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:20:18.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:20:18.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:20:18.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:20:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:20:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:17 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/830172937' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:20:18.178 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:20:18.215 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:18.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:20:18.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:20:18.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:20:18.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:20:18.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:20:18.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:17 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/830172937' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:20:18.230 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:20:18.242 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:20:18.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:18.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:20:18.258 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:20:18.350 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:20:18.397 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:18.423 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:20:18.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:18.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:20:18.526 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:20:18.578 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:20:18.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:18.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:20:18.651 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:20:18.726 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:20:18.776 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:18.951 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:20:19.136 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:20:19.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=883941102,config=2679774077,health=3340005642,kv=2397043959}) 2024-04-23T16:20:19.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {kv=9,logm=91} crc {kv=3562496636,logm=3261809768}) 2024-04-23T16:20:19.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1064571433}) 2024-04-23T16:20:19.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=53580016}) 2024-04-23T16:20:19.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1489549420}) 2024-04-23T16:20:19.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2353168642}) 2024-04-23T16:20:19.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1862275500}) 2024-04-23T16:20:19.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1399964010}) 2024-04-23T16:20:19.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3191233381}) 2024-04-23T16:20:19.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2325942851}) 2024-04-23T16:20:19.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2407156120}) 2024-04-23T16:20:19.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=949787283}) 2024-04-23T16:20:19.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3866678659}) 2024-04-23T16:20:19.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2956384577}) 2024-04-23T16:20:19.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2926543305}) 2024-04-23T16:20:19.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4141165027}) 2024-04-23T16:20:19.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3641208035}) 2024-04-23T16:20:19.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3127309866}) 2024-04-23T16:20:19.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1198479113}) 2024-04-23T16:20:19.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1683111406}) 2024-04-23T16:20:19.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=888059469}) 2024-04-23T16:20:19.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1904025866}) 2024-04-23T16:20:19.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2651413820}) 2024-04-23T16:20:19.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2250408313}) 2024-04-23T16:20:19.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=504152615}) 2024-04-23T16:20:19.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3366854268}) 2024-04-23T16:20:19.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=5,mdsmap=3,mgr=19,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=60} crc {logm=148779271,mdsmap=2840952586,mgr=1765502803,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=2903517051,mon_config_key=588059249}) 2024-04-23T16:20:19.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=40692572}) 2024-04-23T16:20:19.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2893381452}) 2024-04-23T16:20:19.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=27,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=55} crc {mon_config_key=1777852892,monmap=2133572193,osd_metadata=1900671984,osd_pg_creating=241554252,osdmap=3675387637}) 2024-04-23T16:20:19.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=64} crc {osdmap=3568070313}) 2024-04-23T16:20:19.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:20:19.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:20:19.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:20:19.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: pgmap v479: 17 pgs: 17 active+clean; 6.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:20:19.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:20:19.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/830172937' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:20:19.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:18 smithi188 ceph-mon[44101]: osdmap e158: 12 total, 12 up, 12 in 2024-04-23T16:20:19.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:19.142 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:20:19.146 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:20:19.243 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:20:19.288 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=883941102,config=2679774077,health=3340005642,kv=2397043959}) 2024-04-23T16:20:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {kv=9,logm=91} crc {kv=3562496636,logm=3261809768}) 2024-04-23T16:20:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1064571433}) 2024-04-23T16:20:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=53580016}) 2024-04-23T16:20:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1489549420}) 2024-04-23T16:20:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2353168642}) 2024-04-23T16:20:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1862275500}) 2024-04-23T16:20:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1399964010}) 2024-04-23T16:20:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3191233381}) 2024-04-23T16:20:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2325942851}) 2024-04-23T16:20:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2407156120}) 2024-04-23T16:20:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=949787283}) 2024-04-23T16:20:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3866678659}) 2024-04-23T16:20:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2956384577}) 2024-04-23T16:20:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2926543305}) 2024-04-23T16:20:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4141165027}) 2024-04-23T16:20:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3641208035}) 2024-04-23T16:20:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3127309866}) 2024-04-23T16:20:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1198479113}) 2024-04-23T16:20:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1683111406}) 2024-04-23T16:20:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=888059469}) 2024-04-23T16:20:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1904025866}) 2024-04-23T16:20:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2651413820}) 2024-04-23T16:20:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2250408313}) 2024-04-23T16:20:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=504152615}) 2024-04-23T16:20:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3366854268}) 2024-04-23T16:20:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=5,mdsmap=3,mgr=19,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=60} crc {logm=148779271,mdsmap=2840952586,mgr=1765502803,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=2903517051,mon_config_key=588059249}) 2024-04-23T16:20:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=40692572}) 2024-04-23T16:20:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2893381452}) 2024-04-23T16:20:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=27,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=55} crc {mon_config_key=1777852892,monmap=2133572193,osd_metadata=1900671984,osd_pg_creating=241554252,osdmap=3675387637}) 2024-04-23T16:20:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=64} crc {osdmap=3568070313}) 2024-04-23T16:20:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:20:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:20:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:20:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: pgmap v479: 17 pgs: 17 active+clean; 6.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:20:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:20:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/830172937' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:20:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:18 smithi103 ceph-mon[39629]: osdmap e158: 12 total, 12 up, 12 in 2024-04-23T16:20:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=883941102,config=2679774077,health=3340005642,kv=2397043959}) 2024-04-23T16:20:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {kv=9,logm=91} crc {kv=3562496636,logm=3261809768}) 2024-04-23T16:20:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1064571433}) 2024-04-23T16:20:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=53580016}) 2024-04-23T16:20:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1489549420}) 2024-04-23T16:20:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2353168642}) 2024-04-23T16:20:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1862275500}) 2024-04-23T16:20:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1399964010}) 2024-04-23T16:20:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3191233381}) 2024-04-23T16:20:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2325942851}) 2024-04-23T16:20:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2407156120}) 2024-04-23T16:20:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=949787283}) 2024-04-23T16:20:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3866678659}) 2024-04-23T16:20:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2956384577}) 2024-04-23T16:20:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2926543305}) 2024-04-23T16:20:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4141165027}) 2024-04-23T16:20:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3641208035}) 2024-04-23T16:20:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3127309866}) 2024-04-23T16:20:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1198479113}) 2024-04-23T16:20:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1683111406}) 2024-04-23T16:20:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=888059469}) 2024-04-23T16:20:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1904025866}) 2024-04-23T16:20:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2651413820}) 2024-04-23T16:20:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2250408313}) 2024-04-23T16:20:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=504152615}) 2024-04-23T16:20:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3366854268}) 2024-04-23T16:20:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=5,mdsmap=3,mgr=19,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=60} crc {logm=148779271,mdsmap=2840952586,mgr=1765502803,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=2903517051,mon_config_key=588059249}) 2024-04-23T16:20:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=40692572}) 2024-04-23T16:20:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2893381452}) 2024-04-23T16:20:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=27,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=55} crc {mon_config_key=1777852892,monmap=2133572193,osd_metadata=1900671984,osd_pg_creating=241554252,osdmap=3675387637}) 2024-04-23T16:20:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=64} crc {osdmap=3568070313}) 2024-04-23T16:20:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:20:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:20:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:20:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: pgmap v479: 17 pgs: 17 active+clean; 6.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:20:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:20:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/830172937' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:20:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:18 smithi148 ceph-mon[44520]: osdmap e158: 12 total, 12 up, 12 in 2024-04-23T16:20:19.433 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:20:19.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:19.438 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:20:19.540 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:19.570 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:19.590 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:19.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:19.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:19.694 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:20:19.730 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:20.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:20 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:20:20.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:20 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:20:20.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:20 smithi103 ceph-mon[39629]: osdmap e159: 12 total, 12 up, 12 in 2024-04-23T16:20:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:20 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:20:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:20 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:20:20.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:20 smithi188 ceph-mon[44101]: osdmap e159: 12 total, 12 up, 12 in 2024-04-23T16:20:20.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:20 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:20:20.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:20 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:20:20.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:20 smithi148 ceph-mon[44520]: osdmap e159: 12 total, 12 up, 12 in 2024-04-23T16:20:20.698 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:20:20.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:20.706 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:20:20.807 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:20:20.835 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:20.858 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:20:20.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:20.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:20:20.964 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:20:21.028 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:21.275 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:21 smithi103 ceph-mon[39629]: pgmap v482: 17 pgs: 17 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s 2024-04-23T16:20:21.450 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:21 smithi188 ceph-mon[44101]: pgmap v482: 17 pgs: 17 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s 2024-04-23T16:20:21.513 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:20:21.597 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:20:21.598 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:20:21.598 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.598 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.14862.0:239176 2.5 2:a7051051:::benchmark_data_smithi188_77627_object69827:head [delete] snapc 0=[] ondisk+write+known_if_redirected e159)", 2024-04-23T16:20:21.598 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:21.110110+0000", 2024-04-23T16:20:21.598 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.012429188000000001, 2024-04-23T16:20:21.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012451668000000001, 2024-04-23T16:20:21.598 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:21.598 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:21.598 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:20:21.598 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:20:21.598 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.14862", 2024-04-23T16:20:21.599 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:21.599 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 239176 2024-04-23T16:20:21.599 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.599 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:21.599 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.599 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:21.599 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.110110+0000", 2024-04-23T16:20:21.599 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:21.599 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.599 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.599 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:21.599 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.110110+0000", 2024-04-23T16:20:21.599 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:21.599 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.599 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:21.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.110112+0000", 2024-04-23T16:20:21.600 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.813e-06 2024-04-23T16:20:21.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:21.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.110118+0000", 2024-04-23T16:20:21.600 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4620000000000004e-06 2024-04-23T16:20:21.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:21.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.110130+0000", 2024-04-23T16:20:21.600 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2295e-05 2024-04-23T16:20:21.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.601 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.601 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:21.601 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.110138+0000", 2024-04-23T16:20:21.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.712e-06 2024-04-23T16:20:21.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.601 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.601 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:21.601 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.111666+0000", 2024-04-23T16:20:21.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00152837 2024-04-23T16:20:21.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.601 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.601 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:21.601 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.111749+0000", 2024-04-23T16:20:21.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2983000000000002e-05 2024-04-23T16:20:21.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 0", 2024-04-23T16:20:21.602 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.111827+0000", 2024-04-23T16:20:21.602 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7948999999999995e-05 2024-04-23T16:20:21.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:20:21.602 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.121815+0000", 2024-04-23T16:20:21.602 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0099879879999999997 2024-04-23T16:20:21.602 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:21.602 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:21.602 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:21.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.603 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.14862.0:239182 2.5 2:abac228d:::benchmark_data_smithi188_77627_object69833:head [delete] snapc 0=[] ondisk+write+known_if_redirected e159)", 2024-04-23T16:20:21.603 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:21.116363+0000", 2024-04-23T16:20:21.603 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0061771509999999996, 2024-04-23T16:20:21.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006265095, 2024-04-23T16:20:21.603 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:21.603 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:21.603 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:20:21.603 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:20:21.603 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.14862", 2024-04-23T16:20:21.603 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:21.603 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 239182 2024-04-23T16:20:21.603 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.603 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:21.603 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.604 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:21.604 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.116363+0000", 2024-04-23T16:20:21.604 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:21.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.604 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.604 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:21.604 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.116363+0000", 2024-04-23T16:20:21.604 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:21.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.604 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.604 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:21.604 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.116364+0000", 2024-04-23T16:20:21.604 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4890000000000001e-06 2024-04-23T16:20:21.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.604 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.604 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:21.605 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.116369+0000", 2024-04-23T16:20:21.605 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.147e-06 2024-04-23T16:20:21.605 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.605 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.605 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:21.605 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.116381+0000", 2024-04-23T16:20:21.605 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1571e-05 2024-04-23T16:20:21.605 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.605 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.605 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:21.605 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.116388+0000", 2024-04-23T16:20:21.605 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.542e-06 2024-04-23T16:20:21.605 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.605 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.605 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:21.606 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.116442+0000", 2024-04-23T16:20:21.606 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3517999999999998e-05 2024-04-23T16:20:21.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.606 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.606 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:21.606 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.116580+0000", 2024-04-23T16:20:21.606 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013811800000000001 2024-04-23T16:20:21.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.606 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.606 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 0", 2024-04-23T16:20:21.606 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.116710+0000", 2024-04-23T16:20:21.606 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013055200000000001 2024-04-23T16:20:21.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:21.606 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:21.606 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:20:21.606 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:21.121865+0000", 2024-04-23T16:20:21.607 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005154166 2024-04-23T16:20:21.607 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:21.607 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:21.607 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:21.607 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:21.607 INFO:teuthology.orchestra.run.smithi188.stdout: ], 2024-04-23T16:20:21.607 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 2 2024-04-23T16:20:21.607 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:20:21.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:21 smithi148 ceph-mon[44520]: pgmap v482: 17 pgs: 17 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s 2024-04-23T16:20:22.260 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-23T16:20:22.797 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:20:22.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:22.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:20:22.805 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:20:22.904 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:22.916 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:20:22.928 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:20:22.929 INFO:teuthology.orchestra.run.smithi188.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-23T16:20:22.929 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:20:22.931 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:22.952 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:22.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:22.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:23.055 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:20:23.090 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:23.117 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:20:23.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:23.124 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:20:23.225 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:20:23.270 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:23.432 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:20:23.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:23.436 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:20:23.537 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:20:23.592 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:23.629 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:20:23.659 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:20:23.688 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:20:23.727 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:20:23.730 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:20:23.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:23.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:20:23.740 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:23 smithi103 ceph-mon[39629]: pgmap v483: 17 pgs: 17 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-04-23T16:20:23.759 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:20:23.786 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:20:23.823 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:20:23.840 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:20:23.862 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:20:23.867 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:23 smithi148 ceph-mon[44520]: pgmap v483: 17 pgs: 17 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-04-23T16:20:23.869 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:23.890 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:20:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:23.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:20:23.902 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:20:23.949 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:20:23.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:23 smithi188 ceph-mon[44101]: pgmap v483: 17 pgs: 17 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-04-23T16:20:23.983 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:20:23.993 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:20:24.008 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:20:24.012 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:20:24.012 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:20:24.012 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:20:24.012 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:20:24.012 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:20:24.012 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:20:24.012 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:20:24.012 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:20:24.012 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:20:24.012 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:20:24.012 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:20:24.012 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:20:24.035 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:24.076 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:20:24.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:24.093 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:20:24.193 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:24.221 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:24.248 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:24.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:24.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:24.352 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:20:24.400 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:24.426 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:20:24.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:24.429 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:20:24.530 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:20:24.557 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:24.579 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:20:24.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:24.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:20:24.687 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:20:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:24 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:20:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:24 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:20:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:24 smithi188 ceph-mon[44101]: osdmap e160: 12 total, 12 up, 12 in 2024-04-23T16:20:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/126761140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:24.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.735 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:24.814 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:24 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:20:24.814 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:24 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:20:24.814 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:24 smithi103 ceph-mon[39629]: osdmap e160: 12 total, 12 up, 12 in 2024-04-23T16:20:24.814 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/126761140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:24.814 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.814 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.814 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.814 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.814 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.814 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.814 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.814 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.814 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.815 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:24 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:20:24.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:24 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:20:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:24 smithi148 ceph-mon[44520]: osdmap e160: 12 total, 12 up, 12 in 2024-04-23T16:20:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:24 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/126761140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:25.097 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:20:25.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:25.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:20:25.205 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:20:25.237 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:25.258 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:20:25.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:25.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:20:25.362 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:20:25.389 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:25.415 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:20:25.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:25.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:20:25.519 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:25.547 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:25.566 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:25.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:25.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:25.670 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:20:25.719 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:25.719 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:25.719 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:25 smithi188 ceph-mon[44101]: pgmap v485: 17 pgs: 17 active+clean; 6.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-04-23T16:20:25.721 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:25.727 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:25.727 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:25.727 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:25 smithi103 ceph-mon[39629]: pgmap v485: 17 pgs: 17 active+clean; 6.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-04-23T16:20:25.750 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:20:25.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:25.751 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:20:25.788 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:20:25.789 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:20:25.851 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:20:25.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:25.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:25.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:25 smithi148 ceph-mon[44520]: pgmap v485: 17 pgs: 17 active+clean; 6.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-04-23T16:20:25.904 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:25.935 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:20:25.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:25.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:20:26.042 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:20:26.078 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:26.184 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:20:26.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:26.196 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:20:26.297 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:20:26.327 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:26.350 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:20:26.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:26.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:20:26.455 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:26.485 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:26.507 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:26.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:26.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:26.612 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:20:26.613 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:20:26.644 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-23T16:20:26.668 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:26.700 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:20:26.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:26.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:20:26.810 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:26.838 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:26.849 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:20:26.864 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:26.866 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:26 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4003629409' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:20:26.866 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:26 smithi148 ceph-mon[44520]: from='client.34916 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:20:26.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:26.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:26.896 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:26 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4003629409' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:20:26.896 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:26 smithi103 ceph-mon[39629]: from='client.34916 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:20:26.968 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:20:26.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:26 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4003629409' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:20:26.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:26 smithi188 ceph-mon[44101]: from='client.34916 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:20:27.000 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:27.067 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:20:27.467 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:20:27.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:27.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:20:27.571 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:20:27.572 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:27.599 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:27.621 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:27.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:27.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:27.652 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:27 smithi188 ceph-mon[44101]: pgmap v486: 17 pgs: 17 active+clean; 5.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-04-23T16:20:27.652 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:27 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:20:27.652 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:27 smithi188 ceph-mon[44101]: from='client.34916 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:20:27.652 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:27 smithi188 ceph-mon[44101]: osdmap e161: 12 total, 12 up, 12 in 2024-04-23T16:20:27.652 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:27 smithi188 ceph-mon[44101]: 2.b scrub starts 2024-04-23T16:20:27.726 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:20:27.774 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:27.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:27 smithi103 ceph-mon[39629]: pgmap v486: 17 pgs: 17 active+clean; 5.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-04-23T16:20:27.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:27 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:20:27.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:27 smithi103 ceph-mon[39629]: from='client.34916 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:20:27.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:27 smithi103 ceph-mon[39629]: osdmap e161: 12 total, 12 up, 12 in 2024-04-23T16:20:27.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:27 smithi103 ceph-mon[39629]: 2.b scrub starts 2024-04-23T16:20:27.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:27 smithi148 ceph-mon[44520]: pgmap v486: 17 pgs: 17 active+clean; 5.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-04-23T16:20:27.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:27 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:20:27.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:27 smithi148 ceph-mon[44520]: from='client.34916 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:20:27.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:27 smithi148 ceph-mon[44520]: osdmap e161: 12 total, 12 up, 12 in 2024-04-23T16:20:27.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:27 smithi148 ceph-mon[44520]: 2.b scrub starts 2024-04-23T16:20:28.636 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:20:28.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:28.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:20:28.743 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:28.771 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:28.793 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:28.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:28.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:28.897 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:28.923 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:28 smithi148 ceph-mon[44520]: 2.b scrub ok 2024-04-23T16:20:28.925 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:28.932 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:28 smithi103 ceph-mon[39629]: 2.b scrub ok 2024-04-23T16:20:28.948 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:28.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:28.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:28 smithi188 ceph-mon[44101]: 2.b scrub ok 2024-04-23T16:20:29.052 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:20:29.095 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:29.128 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-23T16:20:29.128 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:20:29.128 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-23T16:20:29.128 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.781826 2024-04-23T16:20:29.129 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 7 0.7818263313878266 2024-04-23T16:20:29.237 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:20:29.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:29.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:20:29.346 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:20:29.415 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:29.418 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:20:29.418 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:20:29.421 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:20:29.450 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:20:29.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:29.456 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:20:29.556 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:20:29.601 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:29.648 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:20:29.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:29.651 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:20:29.751 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:20:29.789 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:29.841 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:20:29.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:29.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:20:29.934 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:20:29.944 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:20:29.948 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:20:29.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:29 smithi188 ceph-mon[44101]: pgmap v488: 17 pgs: 17 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:20:29.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:29 smithi188 ceph-mon[44101]: 2.5 scrub starts 2024-04-23T16:20:29.991 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:29 smithi103 ceph-mon[39629]: pgmap v488: 17 pgs: 17 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:20:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:29 smithi103 ceph-mon[39629]: 2.5 scrub starts 2024-04-23T16:20:30.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:29 smithi148 ceph-mon[44520]: pgmap v488: 17 pgs: 17 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:20:30.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:29 smithi148 ceph-mon[44520]: 2.5 scrub starts 2024-04-23T16:20:30.164 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:20:30.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:30.168 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:20:30.190 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-23T16:20:30.269 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:20:30.334 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:30.367 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:20:30.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:30.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:20:30.470 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:20:30.510 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:30.622 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:20:30.725 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:20:30.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:30.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:20:30.808 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:20:30.829 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:20:30.846 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:20:30.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:30 smithi103 ceph-mon[39629]: 2.0 scrub starts 2024-04-23T16:20:30.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:30 smithi103 ceph-mon[39629]: 2.5 scrub ok 2024-04-23T16:20:30.871 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:30.874 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:20:30.908 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:20:30.909 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:20:30.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:30.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:20:30.940 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:20:30.942 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:30 smithi188 ceph-mon[44101]: 2.0 scrub starts 2024-04-23T16:20:30.942 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:30 smithi188 ceph-mon[44101]: 2.5 scrub ok 2024-04-23T16:20:30.982 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:20:31.016 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:31.018 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:20:31.041 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:30 smithi148 ceph-mon[44520]: 2.0 scrub starts 2024-04-23T16:20:31.041 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:30 smithi148 ceph-mon[44520]: 2.5 scrub ok 2024-04-23T16:20:31.043 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:31.046 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:20:31.064 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:31.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:31.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:31.089 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:20:31.121 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:20:31.154 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:20:31.169 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:20:31.191 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:20:31.196 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:31.198 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:20:31.198 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:20:31.198 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:20:31.198 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:20:31.198 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:20:31.198 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:20:31.198 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:20:31.199 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:20:31.199 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:20:31.199 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:20:31.199 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:20:31.199 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:20:31.223 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:20:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:31.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:20:31.328 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:20:31.375 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:31.466 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:20:31.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:31.473 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:20:31.574 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:20:31.621 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:31.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:31 smithi188 ceph-mon[44101]: 2.0 scrub ok 2024-04-23T16:20:31.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:31 smithi188 ceph-mon[44101]: 2.c scrub starts 2024-04-23T16:20:31.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:31 smithi188 ceph-mon[44101]: pgmap v489: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:20:31.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:31 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1060932860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:31.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:31.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:31.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:31.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:31.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:31.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:31.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:31.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:31.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:31 smithi103 ceph-mon[39629]: 2.0 scrub ok 2024-04-23T16:20:31.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:31 smithi103 ceph-mon[39629]: 2.c scrub starts 2024-04-23T16:20:31.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:31 smithi103 ceph-mon[39629]: pgmap v489: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:20:31.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:31 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1060932860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:31.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:31.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:31.940 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:31.940 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:31.940 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:31.940 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:31.940 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:31.940 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:32.082 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:20:32.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:32.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:20:32.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:31 smithi148 ceph-mon[44520]: 2.0 scrub ok 2024-04-23T16:20:32.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:31 smithi148 ceph-mon[44520]: 2.c scrub starts 2024-04-23T16:20:32.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:31 smithi148 ceph-mon[44520]: pgmap v489: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:20:32.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:31 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1060932860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:32.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:32.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:32.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:32.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:32.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:32.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:32.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:32.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:32.197 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:20:32.226 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:32.258 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:20:32.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:32.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:20:32.365 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:20:32.405 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:32.431 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:20:32.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:32.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:20:32.540 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:20:32.605 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:32.693 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:32.693 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:32.693 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:32.693 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:32.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:32 smithi188 ceph-mon[44101]: 2.3 scrub starts 2024-04-23T16:20:32.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:32 smithi188 ceph-mon[44101]: 2.c scrub ok 2024-04-23T16:20:32.724 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:20:32.724 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:20:32.724 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:32.724 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:252631 2.f e161/110 2:f4113c6c:::benchmark_data_smithi188_77627_object83282:head v 161'15665, mlcod=161'15663)", 2024-04-23T16:20:32.724 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:32.722169+0000", 2024-04-23T16:20:32.724 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.00087073399999999996, 2024-04-23T16:20:32.724 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00090090999999999999, 2024-04-23T16:20:32.724 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:32.724 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:32.724 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-04-23T16:20:32.725 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:32.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:32.725 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:32.725 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:32.722169+0000", 2024-04-23T16:20:32.725 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:32.725 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:32.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:32.725 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:32.725 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:32.722169+0000", 2024-04-23T16:20:32.725 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:32.725 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:32.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:32.725 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:32.725 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:32.722171+0000", 2024-04-23T16:20:32.725 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9130000000000001e-06 2024-04-23T16:20:32.726 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:32.726 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:32.726 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:32.726 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:32.722183+0000", 2024-04-23T16:20:32.726 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1433e-05 2024-04-23T16:20:32.726 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:32.726 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:32.726 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:32.726 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:32.722200+0000", 2024-04-23T16:20:32.726 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7668000000000001e-05 2024-04-23T16:20:32.726 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:32.726 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:32.726 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:32.726 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:32.722210+0000", 2024-04-23T16:20:32.727 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1959999999999997e-06 2024-04-23T16:20:32.727 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:32.727 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:32.727 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:32.727 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:32.722283+0000", 2024-04-23T16:20:32.727 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3543999999999999e-05 2024-04-23T16:20:32.727 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:32.727 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:32.727 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:32.727 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:32.722311+0000", 2024-04-23T16:20:32.727 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7464e-05 2024-04-23T16:20:32.727 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:32.727 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:32.727 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:32.727 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:32.728 INFO:teuthology.orchestra.run.smithi188.stdout: ], 2024-04-23T16:20:32.728 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 1 2024-04-23T16:20:32.728 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:20:32.766 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:20:32.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:32.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:20:32.882 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:20:32.934 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:32.976 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:20:32.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:32.979 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:20:33.080 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:20:33.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:33.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:33.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:33.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:33.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:32 smithi103 ceph-mon[39629]: 2.3 scrub starts 2024-04-23T16:20:33.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:32 smithi103 ceph-mon[39629]: 2.c scrub ok 2024-04-23T16:20:33.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:33.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:32 smithi148 ceph-mon[44520]: 2.3 scrub starts 2024-04-23T16:20:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:32 smithi148 ceph-mon[44520]: 2.c scrub ok 2024-04-23T16:20:33.138 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:33.584 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-23T16:20:33.693 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:20:33.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:33.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:20:33.798 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:20:33.833 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:33 smithi188 ceph-mon[44101]: pgmap v490: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-04-23T16:20:33.833 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:33 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:20:33.833 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:33 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:20:33.833 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:33 smithi188 ceph-mon[44101]: 2.3 scrub ok 2024-04-23T16:20:33.833 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:33 smithi188 ceph-mon[44101]: 2.4 scrub starts 2024-04-23T16:20:33.835 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:33.874 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:20:33.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:33.881 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:20:33.978 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:20:33.982 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:20:33.994 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:20:34.037 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:33 smithi103 ceph-mon[39629]: pgmap v490: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-04-23T16:20:34.037 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:33 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:20:34.037 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:33 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:20:34.037 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:33 smithi103 ceph-mon[39629]: 2.3 scrub ok 2024-04-23T16:20:34.037 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:33 smithi103 ceph-mon[39629]: 2.4 scrub starts 2024-04-23T16:20:34.072 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:34.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:33 smithi148 ceph-mon[44520]: pgmap v490: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-04-23T16:20:34.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:33 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:20:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:33 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:20:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:33 smithi148 ceph-mon[44520]: 2.3 scrub ok 2024-04-23T16:20:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:33 smithi148 ceph-mon[44520]: 2.4 scrub starts 2024-04-23T16:20:34.241 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:20:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:34.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:20:34.354 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:20:34.414 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:34.631 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:20:34.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:34.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:20:34.735 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:20:34.765 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:34.801 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:20:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:34.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:20:34.909 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:20:34.910 INFO:teuthology.orchestra.run.smithi103.stderr:set osd.7 primary-affinity to 0.781826 (8512372) 2024-04-23T16:20:34.937 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:20:34.966 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:35.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:34 smithi103 ceph-mon[39629]: 2.4 scrub ok 2024-04-23T16:20:35.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:34 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/33066211' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7818263313878266}]: dispatch 2024-04-23T16:20:35.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:34 smithi103 ceph-mon[39629]: pgmap v491: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:20:35.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:34 smithi103 ceph-mon[39629]: 2.d scrub starts 2024-04-23T16:20:35.194 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:20:35.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:35.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:20:35.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:34 smithi188 ceph-mon[44101]: 2.4 scrub ok 2024-04-23T16:20:35.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:34 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/33066211' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7818263313878266}]: dispatch 2024-04-23T16:20:35.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:34 smithi188 ceph-mon[44101]: pgmap v491: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:20:35.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:34 smithi188 ceph-mon[44101]: 2.d scrub starts 2024-04-23T16:20:35.306 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:20:35.354 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:35.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:34 smithi148 ceph-mon[44520]: 2.4 scrub ok 2024-04-23T16:20:35.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:34 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/33066211' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7818263313878266}]: dispatch 2024-04-23T16:20:35.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:34 smithi148 ceph-mon[44520]: pgmap v491: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:20:35.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:34 smithi148 ceph-mon[44520]: 2.d scrub starts 2024-04-23T16:20:35.496 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:20:35.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:35.501 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:20:35.601 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:20:35.645 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:35.853 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:20:35.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:35.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:20:35.932 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:20:35.961 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:20:36.015 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:36.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:35 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/33066211' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7818263313878266}]': finished 2024-04-23T16:20:36.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:35 smithi103 ceph-mon[39629]: osdmap e162: 12 total, 12 up, 12 in 2024-04-23T16:20:36.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:35 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1459802586' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:20:36.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:35 smithi103 ceph-mon[39629]: from='client.34925 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:20:36.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:35 smithi103 ceph-mon[39629]: 2.d scrub ok 2024-04-23T16:20:36.191 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:20:36.192 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:20:36.192 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:20:36.192 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:20:36.192 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:20:36.219 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:35 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/33066211' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7818263313878266}]': finished 2024-04-23T16:20:36.219 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:35 smithi188 ceph-mon[44101]: osdmap e162: 12 total, 12 up, 12 in 2024-04-23T16:20:36.219 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:35 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1459802586' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:20:36.219 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:35 smithi188 ceph-mon[44101]: from='client.34925 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:20:36.219 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:35 smithi188 ceph-mon[44101]: 2.d scrub ok 2024-04-23T16:20:36.233 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:20:36.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:36.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:20:36.340 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:20:36.366 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:35 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/33066211' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7818263313878266}]': finished 2024-04-23T16:20:36.366 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:35 smithi148 ceph-mon[44520]: osdmap e162: 12 total, 12 up, 12 in 2024-04-23T16:20:36.366 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:35 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1459802586' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:20:36.366 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:35 smithi148 ceph-mon[44520]: from='client.34925 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:20:36.366 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:35 smithi148 ceph-mon[44520]: 2.d scrub ok 2024-04-23T16:20:36.368 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:36.392 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:20:36.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:36.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:20:36.495 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:20:36.524 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:36.546 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:20:36.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:36.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:20:36.650 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:20:36.679 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:36.702 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:20:36.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:36.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:20:36.808 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:20:36.848 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:36.911 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:36 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:20:36.911 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:36 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:20:36.911 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:36 smithi103 ceph-mon[39629]: from='client.34925 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:20:36.911 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:36 smithi103 ceph-mon[39629]: osdmap e163: 12 total, 12 up, 12 in 2024-04-23T16:20:36.912 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:36 smithi103 ceph-mon[39629]: pgmap v494: 17 pgs: 17 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-04-23T16:20:37.059 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:20:37.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:37.062 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:20:37.079 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-23T16:20:37.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:36 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:20:37.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:36 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:20:37.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:36 smithi148 ceph-mon[44520]: from='client.34925 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:20:37.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:36 smithi148 ceph-mon[44520]: osdmap e163: 12 total, 12 up, 12 in 2024-04-23T16:20:37.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:36 smithi148 ceph-mon[44520]: pgmap v494: 17 pgs: 17 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-04-23T16:20:37.163 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:20:37.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:36 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:20:37.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:36 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:20:37.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:36 smithi188 ceph-mon[44101]: from='client.34925 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:20:37.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:36 smithi188 ceph-mon[44101]: osdmap e163: 12 total, 12 up, 12 in 2024-04-23T16:20:37.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:36 smithi188 ceph-mon[44101]: pgmap v494: 17 pgs: 17 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-04-23T16:20:37.227 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:37.260 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:20:37.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:37.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:20:37.368 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:20:37.412 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:37.549 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:20:37.633 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:20:37.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:37.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:20:37.740 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:20:37.772 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:37.793 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:20:37.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:37.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:20:37.899 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:20:37.934 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:37 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:20:37.934 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:37 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:20:37.934 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:37 smithi103 ceph-mon[39629]: 2.9 scrub starts 2024-04-23T16:20:37.934 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:37 smithi103 ceph-mon[39629]: 2.f scrub starts 2024-04-23T16:20:37.934 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:37 smithi103 ceph-mon[39629]: osdmap e164: 12 total, 12 up, 12 in 2024-04-23T16:20:37.934 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:37 smithi103 ceph-mon[39629]: 2.7 scrub starts 2024-04-23T16:20:37.947 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:37.984 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:20:37.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:37.986 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:20:38.087 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:20:38.117 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:37 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:20:38.117 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:37 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:20:38.117 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:37 smithi188 ceph-mon[44101]: 2.9 scrub starts 2024-04-23T16:20:38.117 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:37 smithi188 ceph-mon[44101]: 2.f scrub starts 2024-04-23T16:20:38.117 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:37 smithi188 ceph-mon[44101]: osdmap e164: 12 total, 12 up, 12 in 2024-04-23T16:20:38.117 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:37 smithi188 ceph-mon[44101]: 2.7 scrub starts 2024-04-23T16:20:38.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:37 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:20:38.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:37 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:20:38.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:37 smithi148 ceph-mon[44520]: 2.9 scrub starts 2024-04-23T16:20:38.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:37 smithi148 ceph-mon[44520]: 2.f scrub starts 2024-04-23T16:20:38.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:37 smithi148 ceph-mon[44520]: osdmap e164: 12 total, 12 up, 12 in 2024-04-23T16:20:38.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:37 smithi148 ceph-mon[44520]: 2.7 scrub starts 2024-04-23T16:20:38.156 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:38.185 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:20:38.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:38.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:20:38.289 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:20:38.327 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:38.408 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:20:38.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:38.413 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:20:38.513 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:20:38.541 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:38.564 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:20:38.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:38.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:20:38.667 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:20:38.720 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:38.891 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:20:38.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:38.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:20:38.966 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:20:38.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:38 smithi188 ceph-mon[44101]: 2.9 scrub ok 2024-04-23T16:20:38.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:38 smithi188 ceph-mon[44101]: 2.7 scrub ok 2024-04-23T16:20:38.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:38 smithi188 ceph-mon[44101]: 2.f scrub ok 2024-04-23T16:20:38.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:38 smithi188 ceph-mon[44101]: pgmap v496: 17 pgs: 17 active+clean; 5.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:20:38.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:38 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2538692965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:38.997 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:20:39.002 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:20:39.040 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:20:39.049 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:39.071 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:20:39.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:39.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:20:39.075 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:20:39.104 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:20:39.132 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:20:39.160 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:20:39.175 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:20:39.198 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:20:39.232 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:38 smithi103 ceph-mon[39629]: 2.9 scrub ok 2024-04-23T16:20:39.232 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:38 smithi103 ceph-mon[39629]: 2.7 scrub ok 2024-04-23T16:20:39.232 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:38 smithi103 ceph-mon[39629]: 2.f scrub ok 2024-04-23T16:20:39.232 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:38 smithi103 ceph-mon[39629]: pgmap v496: 17 pgs: 17 active+clean; 5.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:20:39.232 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:38 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2538692965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:39.233 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:20:39.238 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:20:39.239 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:39.266 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:20:39.280 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:20:39.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:39.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:20:39.310 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:20:39.343 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:20:39.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:20:39.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:20:39.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:20:39.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:20:39.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:20:39.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:20:39.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:20:39.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:20:39.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:20:39.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:20:39.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:20:39.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:20:39.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:38 smithi148 ceph-mon[44520]: 2.9 scrub ok 2024-04-23T16:20:39.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:38 smithi148 ceph-mon[44520]: 2.7 scrub ok 2024-04-23T16:20:39.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:38 smithi148 ceph-mon[44520]: 2.f scrub ok 2024-04-23T16:20:39.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:38 smithi148 ceph-mon[44520]: pgmap v496: 17 pgs: 17 active+clean; 5.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:20:39.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:38 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2538692965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:39.384 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:39.413 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:39.437 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:39.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:39.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:39.542 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:20:39.602 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:39.653 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:20:39.653 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:20:39.712 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:20:39.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:39.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:20:39.816 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:20:39.855 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:39.936 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:20:39.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:39.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:20:40.043 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:20:40.072 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:40.096 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:20:40.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:40.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:20:40.199 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:20:40.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.254 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:40.279 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.279 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.279 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.279 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.280 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.280 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.280 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.280 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.280 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.346 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:20:40.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.433 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:20:40.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:40.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:20:40.459 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-23T16:20:40.540 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:20:40.597 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:40.621 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:20:40.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:40.625 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:20:40.726 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:20:40.775 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:40.820 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-23T16:20:40.820 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:20:40.820 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-23T16:20:40.820 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd out 10 2024-04-23T16:20:40.887 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:20:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:40.904 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:20:40.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:20:40.969 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.969 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.969 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:40.969 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:40 smithi188 ceph-mon[44101]: pgmap v497: 17 pgs: 17 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-04-23T16:20:41.005 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:20:41.033 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:41.061 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:20:41.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:41.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:20:41.165 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:20:41.193 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:41.214 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:20:41.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:41.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:20:41.318 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:20:41.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:41.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:41.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:41.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:40 smithi103 ceph-mon[39629]: pgmap v497: 17 pgs: 17 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-04-23T16:20:41.365 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:41.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:41.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:41.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:41.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:40 smithi148 ceph-mon[44520]: pgmap v497: 17 pgs: 17 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-04-23T16:20:41.413 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:20:41.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:41.421 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:20:41.522 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:20:41.552 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:41.577 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:20:41.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:41.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:20:41.682 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:20:41.740 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:41.854 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:20:41.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:41.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:20:41.913 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:20:41.958 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:20:41.986 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:42.006 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:20:42.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:42.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:20:42.111 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:20:42.150 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:42.328 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:20:42.910 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:20:42.910 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:20:42.916 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:20:42.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:42.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:20:43.019 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:20:43.047 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:43.071 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:20:43.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:43.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:20:43.178 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:20:43.220 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:43.577 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:20:43.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:43.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:20:43.586 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:43 smithi188 ceph-mon[44101]: pgmap v498: 17 pgs: 17 active+clean; 4.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 974 op/s 2024-04-23T16:20:43.667 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-23T16:20:43.683 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:20:43.710 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:43.729 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:20:43.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:43.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:20:43.833 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:20:43.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:43 smithi103 ceph-mon[39629]: pgmap v498: 17 pgs: 17 active+clean; 4.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 974 op/s 2024-04-23T16:20:43.858 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:43 smithi148 ceph-mon[44520]: pgmap v498: 17 pgs: 17 active+clean; 4.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 974 op/s 2024-04-23T16:20:43.860 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:43.880 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:20:43.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:43.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:20:43.984 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:20:44.012 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:44.034 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:20:44.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:44.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:20:44.062 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:20:44.138 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:20:44.172 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:44.375 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:20:44.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:44.380 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:20:44.481 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:20:44.556 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:20:44.562 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:44.603 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:20:44.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:44.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:20:44.640 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:20:44.711 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:20:44.740 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:44.761 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:20:44.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:44.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:20:44.854 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:44 smithi103 ceph-mon[39629]: 2.6 scrub starts 2024-04-23T16:20:44.854 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:44 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/325040970' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:20:44.854 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:44 smithi103 ceph-mon[39629]: from='client.34937 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:20:44.865 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:20:44.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:44 smithi148 ceph-mon[44520]: 2.6 scrub starts 2024-04-23T16:20:44.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:44 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/325040970' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:20:44.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:44 smithi148 ceph-mon[44520]: from='client.34937 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:20:44.892 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:44.915 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:20:44.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:44.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:20:44.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:44 smithi188 ceph-mon[44101]: 2.6 scrub starts 2024-04-23T16:20:44.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:44 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/325040970' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:20:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:44 smithi188 ceph-mon[44101]: from='client.34937 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:20:45.018 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:20:45.062 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:45.192 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:44 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T16:20:44.879+0000 7f135b1c7700 -1 mon.a@0(leader).osd e165 definitely_dead 0 2024-04-23T16:20:45.422 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:20:45.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:45.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:20:45.529 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:20:45.586 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:45.641 INFO:teuthology.orchestra.run.smithi103.stderr:marked out osd.10. 2024-04-23T16:20:45.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:45 smithi188 ceph-mon[44101]: pgmap v499: 17 pgs: 17 active+clean; 4.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:20:45.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:45 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:20:45.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:45 smithi188 ceph-mon[44101]: from='client.34937 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:20:45.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:45 smithi188 ceph-mon[44101]: osdmap e165: 12 total, 12 up, 12 in 2024-04-23T16:20:45.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:45 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3664029392' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-04-23T16:20:45.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:45 smithi188 ceph-mon[44101]: from='client.34943 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-04-23T16:20:45.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:45 smithi188 ceph-mon[44101]: Client client.admin marked osd.10 out, while it was still marked up 2024-04-23T16:20:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:45 smithi103 ceph-mon[39629]: pgmap v499: 17 pgs: 17 active+clean; 4.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:20:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:45 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:20:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:45 smithi103 ceph-mon[39629]: from='client.34937 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:20:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:45 smithi103 ceph-mon[39629]: osdmap e165: 12 total, 12 up, 12 in 2024-04-23T16:20:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:45 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3664029392' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-04-23T16:20:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:45 smithi103 ceph-mon[39629]: from='client.34943 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-04-23T16:20:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:45 smithi103 ceph-mon[39629]: Client client.admin marked osd.10 out, while it was still marked up 2024-04-23T16:20:45.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:45 smithi148 ceph-mon[44520]: pgmap v499: 17 pgs: 17 active+clean; 4.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:20:45.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:45 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:20:45.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:45 smithi148 ceph-mon[44520]: from='client.34937 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:20:45.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:45 smithi148 ceph-mon[44520]: osdmap e165: 12 total, 12 up, 12 in 2024-04-23T16:20:45.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:45 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3664029392' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-04-23T16:20:45.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:45 smithi148 ceph-mon[44520]: from='client.34943 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-04-23T16:20:45.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:45 smithi148 ceph-mon[44520]: Client client.admin marked osd.10 out, while it was still marked up 2024-04-23T16:20:46.024 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:20:46.024 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:20:46.725 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-23T16:20:46.812 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:46 smithi103 ceph-mon[39629]: from='client.34943 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-04-23T16:20:46.813 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:46 smithi103 ceph-mon[39629]: osdmap e166: 12 total, 12 up, 11 in 2024-04-23T16:20:46.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:46 smithi188 ceph-mon[44101]: from='client.34943 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-04-23T16:20:46.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:46 smithi188 ceph-mon[44101]: osdmap e166: 12 total, 12 up, 11 in 2024-04-23T16:20:46.986 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:20:47.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:46 smithi148 ceph-mon[44520]: from='client.34943 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-04-23T16:20:47.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:46 smithi148 ceph-mon[44520]: osdmap e166: 12 total, 12 up, 11 in 2024-04-23T16:20:47.155 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:20:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:47.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:20:47.260 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:20:47.289 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:47.307 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:20:47.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:47.310 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:20:47.411 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:20:47.437 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:47.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:47 smithi188 ceph-mon[44101]: pgmap v502: 17 pgs: 17 active+clean; 4.5 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-23T16:20:47.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:47 smithi188 ceph-mon[44101]: osdmap e167: 12 total, 12 up, 11 in 2024-04-23T16:20:47.786 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:20:47.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:47.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:20:47.805 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:20:47.817 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:20:47.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:47 smithi103 ceph-mon[39629]: pgmap v502: 17 pgs: 17 active+clean; 4.5 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-23T16:20:47.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:47 smithi103 ceph-mon[39629]: osdmap e167: 12 total, 12 up, 11 in 2024-04-23T16:20:47.838 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:20:47.857 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:20:47.877 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:20:47.890 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:47.897 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:20:47.918 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:20:47.919 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:47 smithi148 ceph-mon[44520]: pgmap v502: 17 pgs: 17 active+clean; 4.5 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-23T16:20:47.920 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:47 smithi148 ceph-mon[44520]: osdmap e167: 12 total, 12 up, 11 in 2024-04-23T16:20:47.922 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:47.938 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:20:47.944 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:47.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:47.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:47.958 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:20:47.978 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:20:48.002 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:20:48.025 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:20:48.048 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:20:48.054 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:20:48.069 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:20:48.069 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:20:48.069 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:20:48.069 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:20:48.069 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:20:48.069 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:20:48.069 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:20:48.070 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:20:48.070 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:20:48.070 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:20:48.070 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:20:48.070 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:20:48.099 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:48.122 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:20:48.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:48.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:20:48.134 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:20:48.229 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:48.259 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:48.287 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:48.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:48.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:48.392 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:20:48.454 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:48.650 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:48 smithi188 ceph-mon[44101]: osdmap e168: 12 total, 12 up, 11 in 2024-04-23T16:20:48.651 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:48 smithi188 ceph-mon[44101]: 2.7 starting backfill to osd.2 from (0'0,0'0] MAX to 165'16884 2024-04-23T16:20:48.651 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:48 smithi188 ceph-mon[44101]: 2.0 starting backfill to osd.9 from (0'0,0'0] MAX to 165'16718 2024-04-23T16:20:48.651 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:48 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1111736945' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:48.651 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:48.651 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:48.651 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:48.651 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:48.651 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:48.651 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:48.651 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:48.651 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:48.651 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:48.651 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:48.671 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:20:48.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:48.675 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:20:48.776 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:20:48.810 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:48.831 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:20:48.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:48.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:20:48.936 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:20:49.001 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:49.013 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:20:49.013 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-04-23T16:20:49.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-04-23T16:20:49.013 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:20:49.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.014 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:148789 2.0 e141/110 2:0d8ae6e5:::benchmark_data_smithi188_77627_object148788:head v 141'9327, mlcod=141'9325)", 2024-04-23T16:20:49.014 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:19:01.033309+0000", 2024-04-23T16:20:49.014 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 107.976526268, 2024-04-23T16:20:49.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021918188000000002, 2024-04-23T16:20:49.014 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.014 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.014 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.014 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.033309+0000", 2024-04-23T16:20:49.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.033309+0000", 2024-04-23T16:20:49.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.033311+0000", 2024-04-23T16:20:49.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2900000000000001e-06 2024-04-23T16:20:49.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.037874+0000", 2024-04-23T16:20:49.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045624120000000001 2024-04-23T16:20:49.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.037888+0000", 2024-04-23T16:20:49.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4246e-05 2024-04-23T16:20:49.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.037898+0000", 2024-04-23T16:20:49.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8640000000000008e-06 2024-04-23T16:20:49.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.038036+0000", 2024-04-23T16:20:49.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013847499999999999 2024-04-23T16:20:49.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.038062+0000", 2024-04-23T16:20:49.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5831999999999999e-05 2024-04-23T16:20:49.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.055179+0000", 2024-04-23T16:20:49.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017116962999999999 2024-04-23T16:20:49.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.055227+0000", 2024-04-23T16:20:49.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8106000000000003e-05 2024-04-23T16:20:49.017 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.018 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.018 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.018 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:148797 2.0 e141/110 2:0dd08a2b:::benchmark_data_smithi188_77627_object148796:head v 141'9328, mlcod=141'9325)", 2024-04-23T16:20:49.018 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:19:01.037934+0000", 2024-04-23T16:20:49.018 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 107.971900885, 2024-04-23T16:20:49.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024960717, 2024-04-23T16:20:49.018 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.018 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.018 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.018 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.037934+0000", 2024-04-23T16:20:49.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.037934+0000", 2024-04-23T16:20:49.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.037936+0000", 2024-04-23T16:20:49.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1060000000000001e-06 2024-04-23T16:20:49.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.037968+0000", 2024-04-23T16:20:49.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2019e-05 2024-04-23T16:20:49.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.037973+0000", 2024-04-23T16:20:49.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8329999999999999e-06 2024-04-23T16:20:49.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.037979+0000", 2024-04-23T16:20:49.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3870000000000003e-06 2024-04-23T16:20:49.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.042144+0000", 2024-04-23T16:20:49.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0041647990000000003 2024-04-23T16:20:49.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.042166+0000", 2024-04-23T16:20:49.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2617e-05 2024-04-23T16:20:49.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.062849+0000", 2024-04-23T16:20:49.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020683132 2024-04-23T16:20:49.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.022 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:19:01.062895+0000", 2024-04-23T16:20:49.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5824e-05 2024-04-23T16:20:49.022 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.022 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.022 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.022 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:235546 2.0 e157/110 2:0abcde88:::benchmark_data_smithi188_77627_object66197:head v 157'14775, mlcod=157'14772)", 2024-04-23T16:20:49.022 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:18.012671+0000", 2024-04-23T16:20:49.022 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.997164373, 2024-04-23T16:20:49.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029306602000000001, 2024-04-23T16:20:49.022 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.022 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.023 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.023 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.012671+0000", 2024-04-23T16:20:49.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.012671+0000", 2024-04-23T16:20:49.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.012673+0000", 2024-04-23T16:20:49.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6160000000000001e-06 2024-04-23T16:20:49.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.012681+0000", 2024-04-23T16:20:49.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.456e-06 2024-04-23T16:20:49.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.012695+0000", 2024-04-23T16:20:49.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3825000000000001e-05 2024-04-23T16:20:49.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.012703+0000", 2024-04-23T16:20:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3699999999999995e-06 2024-04-23T16:20:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.012762+0000", 2024-04-23T16:20:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8888999999999998e-05 2024-04-23T16:20:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.012786+0000", 2024-04-23T16:20:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3620000000000001e-05 2024-04-23T16:20:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.041931+0000", 2024-04-23T16:20:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029145490999999999 2024-04-23T16:20:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.041978+0000", 2024-04-23T16:20:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6335000000000003e-05 2024-04-23T16:20:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:236016 2.7 e157/110 2:e9a54b98:::benchmark_data_smithi188_77627_object66667:head v 157'14899, mlcod=157'14895)", 2024-04-23T16:20:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:18.491346+0000", 2024-04-23T16:20:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.518489690999999, 2024-04-23T16:20:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020985021, 2024-04-23T16:20:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.491346+0000", 2024-04-23T16:20:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.491346+0000", 2024-04-23T16:20:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.491347+0000", 2024-04-23T16:20:49.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4410000000000001e-06 2024-04-23T16:20:49.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.491355+0000", 2024-04-23T16:20:49.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5530000000000004e-06 2024-04-23T16:20:49.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.491365+0000", 2024-04-23T16:20:49.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0684e-05 2024-04-23T16:20:49.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.491372+0000", 2024-04-23T16:20:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7399999999999998e-06 2024-04-23T16:20:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.496591+0000", 2024-04-23T16:20:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052193279999999996 2024-04-23T16:20:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.496614+0000", 2024-04-23T16:20:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2659000000000001e-05 2024-04-23T16:20:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.512298+0000", 2024-04-23T16:20:49.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015684364999999999 2024-04-23T16:20:49.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.512331+0000", 2024-04-23T16:20:49.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2251e-05 2024-04-23T16:20:49.030 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.030 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.030 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.030 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:236340 2.0 e158/110 2:011e7dc1:::benchmark_data_smithi188_77627_object66991:head v 158'14809, mlcod=158'14807)", 2024-04-23T16:20:49.030 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:18.774012+0000", 2024-04-23T16:20:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.235823063000002, 2024-04-23T16:20:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025339784000000001, 2024-04-23T16:20:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.774012+0000", 2024-04-23T16:20:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.774012+0000", 2024-04-23T16:20:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.774014+0000", 2024-04-23T16:20:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7150000000000001e-06 2024-04-23T16:20:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.774023+0000", 2024-04-23T16:20:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7940000000000008e-06 2024-04-23T16:20:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.774035+0000", 2024-04-23T16:20:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1815e-05 2024-04-23T16:20:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.774043+0000", 2024-04-23T16:20:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3950000000000001e-06 2024-04-23T16:20:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.778669+0000", 2024-04-23T16:20:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046258829999999999 2024-04-23T16:20:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.778686+0000", 2024-04-23T16:20:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6946e-05 2024-04-23T16:20:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.799307+0000", 2024-04-23T16:20:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020620808000000001 2024-04-23T16:20:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.799352+0000", 2024-04-23T16:20:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5427999999999999e-05 2024-04-23T16:20:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:238729 2.7 e159/110 2:eda5d94c:::benchmark_data_smithi188_77627_object69380:head v 159'15057, mlcod=159'15055)", 2024-04-23T16:20:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:20.721333+0000", 2024-04-23T16:20:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 28.288502745999999, 2024-04-23T16:20:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031982430999999999, 2024-04-23T16:20:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.721333+0000", 2024-04-23T16:20:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.721333+0000", 2024-04-23T16:20:49.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.721335+0000", 2024-04-23T16:20:49.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0269999999999998e-06 2024-04-23T16:20:49.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.721343+0000", 2024-04-23T16:20:49.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0479999999999999e-06 2024-04-23T16:20:49.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.721351+0000", 2024-04-23T16:20:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9480000000000008e-06 2024-04-23T16:20:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.721357+0000", 2024-04-23T16:20:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9129999999999997e-06 2024-04-23T16:20:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.721406+0000", 2024-04-23T16:20:49.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8748999999999997e-05 2024-04-23T16:20:49.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.721425+0000", 2024-04-23T16:20:49.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9188999999999999e-05 2024-04-23T16:20:49.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.753265+0000", 2024-04-23T16:20:49.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031839628000000002 2024-04-23T16:20:49.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.753315+0000", 2024-04-23T16:20:49.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9929000000000003e-05 2024-04-23T16:20:49.039 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.039 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.039 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.039 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:238731 2.7 e159/110 2:e11e3a7d:::benchmark_data_smithi188_77627_object69382:head v 159'15058, mlcod=159'15055)", 2024-04-23T16:20:49.039 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:20.726375+0000", 2024-04-23T16:20:49.039 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 28.283460464000001, 2024-04-23T16:20:49.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026970224000000001, 2024-04-23T16:20:49.039 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.040 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.040 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.040 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.726375+0000", 2024-04-23T16:20:49.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.726375+0000", 2024-04-23T16:20:49.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.726376+0000", 2024-04-23T16:20:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.13e-06 2024-04-23T16:20:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.726384+0000", 2024-04-23T16:20:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7810000000000005e-06 2024-04-23T16:20:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.726395+0000", 2024-04-23T16:20:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1525999999999999e-05 2024-04-23T16:20:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.726403+0000", 2024-04-23T16:20:49.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9160000000000001e-06 2024-04-23T16:20:49.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.730524+0000", 2024-04-23T16:20:49.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0041206869999999996 2024-04-23T16:20:49.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.730545+0000", 2024-04-23T16:20:49.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0925e-05 2024-04-23T16:20:49.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.753321+0000", 2024-04-23T16:20:49.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022776500000000002 2024-04-23T16:20:49.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.753345+0000", 2024-04-23T16:20:49.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3759000000000001e-05 2024-04-23T16:20:49.043 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.043 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.043 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.044 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:238733 2.7 e159/110 2:e254a516:::benchmark_data_smithi188_77627_object69384:head v 159'15059, mlcod=159'15055)", 2024-04-23T16:20:49.044 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:20.730415+0000", 2024-04-23T16:20:49.044 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 28.279420161000001, 2024-04-23T16:20:49.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022965640999999998, 2024-04-23T16:20:49.044 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.044 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.044 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.044 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.730415+0000", 2024-04-23T16:20:49.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.730415+0000", 2024-04-23T16:20:49.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.730417+0000", 2024-04-23T16:20:49.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3570000000000001e-06 2024-04-23T16:20:49.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.730425+0000", 2024-04-23T16:20:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4069999999999993e-06 2024-04-23T16:20:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.730441+0000", 2024-04-23T16:20:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5642999999999999e-05 2024-04-23T16:20:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.730449+0000", 2024-04-23T16:20:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4330000000000004e-06 2024-04-23T16:20:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.734965+0000", 2024-04-23T16:20:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045163479999999999 2024-04-23T16:20:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.734988+0000", 2024-04-23T16:20:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2580999999999999e-05 2024-04-23T16:20:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.753347+0000", 2024-04-23T16:20:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018358995999999999 2024-04-23T16:20:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:20.753381+0000", 2024-04-23T16:20:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3875999999999999e-05 2024-04-23T16:20:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:251380 2.7 e161/110 2:e62de693:::benchmark_data_smithi188_77627_object82031:head v 161'15873, mlcod=161'15871)", 2024-04-23T16:20:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:31.667376+0000", 2024-04-23T16:20:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.342459031000001, 2024-04-23T16:20:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026756411000000001, 2024-04-23T16:20:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:31.667376+0000", 2024-04-23T16:20:49.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:31.667376+0000", 2024-04-23T16:20:49.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:31.667378+0000", 2024-04-23T16:20:49.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.148e-06 2024-04-23T16:20:49.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:31.667389+0000", 2024-04-23T16:20:49.050 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0183000000000001e-05 2024-04-23T16:20:49.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:31.667406+0000", 2024-04-23T16:20:49.050 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.785e-05 2024-04-23T16:20:49.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:31.667416+0000", 2024-04-23T16:20:49.050 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4639999999999995e-06 2024-04-23T16:20:49.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:31.667486+0000", 2024-04-23T16:20:49.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0394000000000004e-05 2024-04-23T16:20:49.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:31.667508+0000", 2024-04-23T16:20:49.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1441e-05 2024-04-23T16:20:49.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:31.694075+0000", 2024-04-23T16:20:49.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026566836 2024-04-23T16:20:49.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:31.694133+0000", 2024-04-23T16:20:49.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8094999999999999e-05 2024-04-23T16:20:49.052 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.052 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.052 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.052 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:251981 2.0 e161/110 2:0b319a7f:::benchmark_data_smithi188_77627_object82632:head v 161'15788, mlcod=161'15785)", 2024-04-23T16:20:49.052 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:32.162545+0000", 2024-04-23T16:20:49.052 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.847290476000001, 2024-04-23T16:20:49.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022582656, 2024-04-23T16:20:49.053 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.053 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.053 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.053 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:32.162545+0000", 2024-04-23T16:20:49.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:32.162545+0000", 2024-04-23T16:20:49.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:32.162545+0000", 2024-04-23T16:20:49.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3499999999999996e-07 2024-04-23T16:20:49.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:32.162550+0000", 2024-04-23T16:20:49.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5140000000000003e-06 2024-04-23T16:20:49.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:32.162555+0000", 2024-04-23T16:20:49.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0860000000000001e-06 2024-04-23T16:20:49.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:32.162559+0000", 2024-04-23T16:20:49.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8539999999999999e-06 2024-04-23T16:20:49.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:32.168971+0000", 2024-04-23T16:20:49.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064124380000000003 2024-04-23T16:20:49.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:32.168996+0000", 2024-04-23T16:20:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4845999999999999e-05 2024-04-23T16:20:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:32.185083+0000", 2024-04-23T16:20:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016086763 2024-04-23T16:20:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:32.185127+0000", 2024-04-23T16:20:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4520000000000001e-05 2024-04-23T16:20:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.057 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:254688 2.0 e161/110 2:038ff647:::benchmark_data_smithi188_77627_object85339:head v 161'15982, mlcod=161'15980)", 2024-04-23T16:20:49.057 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:34.588281+0000", 2024-04-23T16:20:49.057 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.421554197000001, 2024-04-23T16:20:49.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023019359, 2024-04-23T16:20:49.057 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.057 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.057 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.057 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:34.588281+0000", 2024-04-23T16:20:49.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:34.588281+0000", 2024-04-23T16:20:49.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:34.588282+0000", 2024-04-23T16:20:49.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.331e-06 2024-04-23T16:20:49.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:34.588290+0000", 2024-04-23T16:20:49.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3780000000000003e-06 2024-04-23T16:20:49.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:34.588301+0000", 2024-04-23T16:20:49.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0747000000000001e-05 2024-04-23T16:20:49.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:34.588308+0000", 2024-04-23T16:20:49.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3309999999999998e-06 2024-04-23T16:20:49.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:34.588371+0000", 2024-04-23T16:20:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3091999999999998e-05 2024-04-23T16:20:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:34.588392+0000", 2024-04-23T16:20:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1475e-05 2024-04-23T16:20:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:34.611258+0000", 2024-04-23T16:20:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022865631000000001 2024-04-23T16:20:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:34.611300+0000", 2024-04-23T16:20:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2373999999999998e-05 2024-04-23T16:20:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:255298 2.7 e162/110 2:e2a95ba5:::benchmark_data_smithi188_77627_object85949:head v 162'16110, mlcod=162'16107)", 2024-04-23T16:20:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:35.167206+0000", 2024-04-23T16:20:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 13.842629633, 2024-04-23T16:20:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022175783000000001, 2024-04-23T16:20:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.062 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:35.167206+0000", 2024-04-23T16:20:49.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:35.167206+0000", 2024-04-23T16:20:49.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:35.167208+0000", 2024-04-23T16:20:49.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9350000000000001e-06 2024-04-23T16:20:49.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:35.167217+0000", 2024-04-23T16:20:49.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8449999999999998e-06 2024-04-23T16:20:49.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:35.167231+0000", 2024-04-23T16:20:49.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3348e-05 2024-04-23T16:20:49.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:35.167240+0000", 2024-04-23T16:20:49.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8470000000000002e-06 2024-04-23T16:20:49.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:35.167311+0000", 2024-04-23T16:20:49.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1845000000000001e-05 2024-04-23T16:20:49.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:35.167332+0000", 2024-04-23T16:20:49.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0157e-05 2024-04-23T16:20:49.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:35.189340+0000", 2024-04-23T16:20:49.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022008645 2024-04-23T16:20:49.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:35.189381+0000", 2024-04-23T16:20:49.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1161000000000001e-05 2024-04-23T16:20:49.065 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.065 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.065 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.066 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:257625 2.0 e164/110 2:0a460d67:::benchmark_data_smithi188_77627_object88276:head v 164'16167, mlcod=164'16165)", 2024-04-23T16:20:49.066 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:37.170711+0000", 2024-04-23T16:20:49.066 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.839123945000001, 2024-04-23T16:20:49.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023225539, 2024-04-23T16:20:49.066 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.066 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.066 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.066 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.170711+0000", 2024-04-23T16:20:49.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.170711+0000", 2024-04-23T16:20:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.170713+0000", 2024-04-23T16:20:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.73e-06 2024-04-23T16:20:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.170721+0000", 2024-04-23T16:20:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0290000000000005e-06 2024-04-23T16:20:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.170731+0000", 2024-04-23T16:20:49.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0139999999999999e-05 2024-04-23T16:20:49.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.170737+0000", 2024-04-23T16:20:49.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9220000000000002e-06 2024-04-23T16:20:49.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.170786+0000", 2024-04-23T16:20:49.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8736000000000002e-05 2024-04-23T16:20:49.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.170809+0000", 2024-04-23T16:20:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2659000000000001e-05 2024-04-23T16:20:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.193892+0000", 2024-04-23T16:20:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023083734000000002 2024-04-23T16:20:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.193937+0000", 2024-04-23T16:20:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4589000000000003e-05 2024-04-23T16:20:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:257956 2.7 e164/110 2:edcda3eb:::benchmark_data_smithi188_77627_object88607:head v 164'16281, mlcod=164'16278)", 2024-04-23T16:20:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:37.521127+0000", 2024-04-23T16:20:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.488707952, 2024-04-23T16:20:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023346341999999999, 2024-04-23T16:20:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.521127+0000", 2024-04-23T16:20:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.521127+0000", 2024-04-23T16:20:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.521129+0000", 2024-04-23T16:20:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9539999999999998e-06 2024-04-23T16:20:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.521140+0000", 2024-04-23T16:20:49.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.026e-05 2024-04-23T16:20:49.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.521153+0000", 2024-04-23T16:20:49.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3274e-05 2024-04-23T16:20:49.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.521162+0000", 2024-04-23T16:20:49.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9649999999999997e-06 2024-04-23T16:20:49.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.521255+0000", 2024-04-23T16:20:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3628000000000005e-05 2024-04-23T16:20:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.521279+0000", 2024-04-23T16:20:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3402999999999999e-05 2024-04-23T16:20:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.544434+0000", 2024-04-23T16:20:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023155149999999999 2024-04-23T16:20:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.544474+0000", 2024-04-23T16:20:49.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9708e-05 2024-04-23T16:20:49.074 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.074 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.074 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.074 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:257958 2.7 e164/110 2:ee0bd61c:::benchmark_data_smithi188_77627_object88609:head v 164'16282, mlcod=164'16279)", 2024-04-23T16:20:49.074 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:37.522598+0000", 2024-04-23T16:20:49.074 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.487237693000001, 2024-04-23T16:20:49.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022297454000000001, 2024-04-23T16:20:49.074 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.075 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.075 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.075 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.522598+0000", 2024-04-23T16:20:49.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.522598+0000", 2024-04-23T16:20:49.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.522599+0000", 2024-04-23T16:20:49.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3149999999999999e-06 2024-04-23T16:20:49.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.522606+0000", 2024-04-23T16:20:49.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1840000000000002e-06 2024-04-23T16:20:49.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.522619+0000", 2024-04-23T16:20:49.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2631e-05 2024-04-23T16:20:49.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.522626+0000", 2024-04-23T16:20:49.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9779999999999999e-06 2024-04-23T16:20:49.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.525900+0000", 2024-04-23T16:20:49.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0032742230000000002 2024-04-23T16:20:49.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.525920+0000", 2024-04-23T16:20:49.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9947000000000001e-05 2024-04-23T16:20:49.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.544869+0000", 2024-04-23T16:20:49.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018949500000000001 2024-04-23T16:20:49.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:37.544895+0000", 2024-04-23T16:20:49.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5675999999999999e-05 2024-04-23T16:20:49.078 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.078 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:267881 2.0 e168/167 2:0ef14e1d:::benchmark_data_smithi188_77627_object98532:head v 168'16750, mlcod=168'16748)", 2024-04-23T16:20:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:48.393303+0000", 2024-04-23T16:20:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.61653219400000003, 2024-04-23T16:20:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.039018575999999999, 2024-04-23T16:20:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393303+0000", 2024-04-23T16:20:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393303+0000", 2024-04-23T16:20:49.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393305+0000", 2024-04-23T16:20:49.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2800000000000002e-06 2024-04-23T16:20:49.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393318+0000", 2024-04-23T16:20:49.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2537999999999999e-05 2024-04-23T16:20:49.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393333+0000", 2024-04-23T16:20:49.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5281000000000002e-05 2024-04-23T16:20:49.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393343+0000", 2024-04-23T16:20:49.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5100000000000004e-06 2024-04-23T16:20:49.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393455+0000", 2024-04-23T16:20:49.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000112346 2024-04-23T16:20:49.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393484+0000", 2024-04-23T16:20:49.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8957000000000001e-05 2024-04-23T16:20:49.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.432261+0000", 2024-04-23T16:20:49.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.038777041999999998 2024-04-23T16:20:49.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.432322+0000", 2024-04-23T16:20:49.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0622000000000003e-05 2024-04-23T16:20:49.083 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.083 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.083 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.083 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:267888 2.7 e168/167 2:ebf38441:::benchmark_data_smithi188_77627_object98539:head v 168'16911, mlcod=168'16906)", 2024-04-23T16:20:49.083 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:48.393373+0000", 2024-04-23T16:20:49.083 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.61646265300000003, 2024-04-23T16:20:49.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025938233000000002, 2024-04-23T16:20:49.084 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.084 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.084 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.084 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393373+0000", 2024-04-23T16:20:49.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393373+0000", 2024-04-23T16:20:49.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393374+0000", 2024-04-23T16:20:49.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4900000000000005e-07 2024-04-23T16:20:49.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393380+0000", 2024-04-23T16:20:49.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0460000000000002e-06 2024-04-23T16:20:49.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393387+0000", 2024-04-23T16:20:49.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4449999999999999e-06 2024-04-23T16:20:49.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393392+0000", 2024-04-23T16:20:49.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4040000000000003e-06 2024-04-23T16:20:49.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.399076+0000", 2024-04-23T16:20:49.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0056833370000000001 2024-04-23T16:20:49.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.399104+0000", 2024-04-23T16:20:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8445999999999998e-05 2024-04-23T16:20:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.419251+0000", 2024-04-23T16:20:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020146592000000001 2024-04-23T16:20:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.419311+0000", 2024-04-23T16:20:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0114000000000001e-05 2024-04-23T16:20:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.088 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.088 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.088 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:267890 2.0 e168/167 2:0c836efb:::benchmark_data_smithi188_77627_object98541:head v 168'16751, mlcod=168'16748)", 2024-04-23T16:20:49.088 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:48.393758+0000", 2024-04-23T16:20:49.088 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.61607727199999995, 2024-04-23T16:20:49.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.038600184000000003, 2024-04-23T16:20:49.088 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.088 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.088 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.088 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393758+0000", 2024-04-23T16:20:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393758+0000", 2024-04-23T16:20:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393759+0000", 2024-04-23T16:20:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.269e-06 2024-04-23T16:20:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393766+0000", 2024-04-23T16:20:49.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9199999999999998e-06 2024-04-23T16:20:49.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393781+0000", 2024-04-23T16:20:49.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4588e-05 2024-04-23T16:20:49.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.393787+0000", 2024-04-23T16:20:49.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6810000000000001e-06 2024-04-23T16:20:49.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.398124+0000", 2024-04-23T16:20:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043362560000000001 2024-04-23T16:20:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.398150+0000", 2024-04-23T16:20:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6021999999999999e-05 2024-04-23T16:20:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.432327+0000", 2024-04-23T16:20:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.034177285000000002 2024-04-23T16:20:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.432358+0000", 2024-04-23T16:20:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1163e-05 2024-04-23T16:20:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:267893 2.0 e168/167 2:07aff785:::benchmark_data_smithi188_77627_object98544:head v 168'16752, mlcod=168'16748)", 2024-04-23T16:20:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:48.400054+0000", 2024-04-23T16:20:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.60978140199999997, 2024-04-23T16:20:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032340014, 2024-04-23T16:20:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.400054+0000", 2024-04-23T16:20:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.400054+0000", 2024-04-23T16:20:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.400056+0000", 2024-04-23T16:20:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.759e-06 2024-04-23T16:20:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.400066+0000", 2024-04-23T16:20:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0008e-05 2024-04-23T16:20:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.400076+0000", 2024-04-23T16:20:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0456e-05 2024-04-23T16:20:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.400084+0000", 2024-04-23T16:20:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1790000000000008e-06 2024-04-23T16:20:49.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.402751+0000", 2024-04-23T16:20:49.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002666383 2024-04-23T16:20:49.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.402772+0000", 2024-04-23T16:20:49.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1715000000000001e-05 2024-04-23T16:20:49.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.432361+0000", 2024-04-23T16:20:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029588392000000002 2024-04-23T16:20:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.432394+0000", 2024-04-23T16:20:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3121999999999999e-05 2024-04-23T16:20:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:267895 2.0 e168/167 2:092f95cd:::benchmark_data_smithi188_77627_object98546:head v 168'16753, mlcod=168'16748)", 2024-04-23T16:20:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:48.401049+0000", 2024-04-23T16:20:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.60878583200000003, 2024-04-23T16:20:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031372160000000003, 2024-04-23T16:20:49.097 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:49.097 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:49.097 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:20:49.097 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:49.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:49.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.401049+0000", 2024-04-23T16:20:49.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:49.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.401049+0000", 2024-04-23T16:20:49.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:49.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.401051+0000", 2024-04-23T16:20:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.032e-06 2024-04-23T16:20:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.401057+0000", 2024-04-23T16:20:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2140000000000001e-06 2024-04-23T16:20:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.401065+0000", 2024-04-23T16:20:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3890000000000005e-06 2024-04-23T16:20:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.401071+0000", 2024-04-23T16:20:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8989999999999998e-06 2024-04-23T16:20:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.408410+0000", 2024-04-23T16:20:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073391810000000002 2024-04-23T16:20:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.408431+0000", 2024-04-23T16:20:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1219000000000001e-05 2024-04-23T16:20:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:20:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.432396+0000", 2024-04-23T16:20:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023964787000000001 2024-04-23T16:20:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:20:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:48.432422+0000", 2024-04-23T16:20:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5439000000000002e-05 2024-04-23T16:20:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:49.100 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:20:49.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:49.102 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:20:49.102 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:20:49.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:48 smithi103 ceph-mon[39629]: osdmap e168: 12 total, 12 up, 11 in 2024-04-23T16:20:49.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:48 smithi103 ceph-mon[39629]: 2.7 starting backfill to osd.2 from (0'0,0'0] MAX to 165'16884 2024-04-23T16:20:49.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:48 smithi103 ceph-mon[39629]: 2.0 starting backfill to osd.9 from (0'0,0'0] MAX to 165'16718 2024-04-23T16:20:49.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:48 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1111736945' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:49.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:48 smithi148 ceph-mon[44520]: osdmap e168: 12 total, 12 up, 11 in 2024-04-23T16:20:49.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:48 smithi148 ceph-mon[44520]: 2.7 starting backfill to osd.2 from (0'0,0'0] MAX to 165'16884 2024-04-23T16:20:49.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:48 smithi148 ceph-mon[44520]: 2.0 starting backfill to osd.9 from (0'0,0'0] MAX to 165'16718 2024-04-23T16:20:49.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:48 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1111736945' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:49.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.202 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:20:49.275 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:49.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:49.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:49 smithi188 ceph-mon[44101]: pgmap v505: 17 pgs: 17 active+clean; 4.4 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 855 op/s 2024-04-23T16:20:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:49 smithi103 ceph-mon[39629]: pgmap v505: 17 pgs: 17 active+clean; 4.4 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 855 op/s 2024-04-23T16:20:50.119 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-23T16:20:50.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:50.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:50.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:49 smithi148 ceph-mon[44520]: pgmap v505: 17 pgs: 17 active+clean; 4.4 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 855 op/s 2024-04-23T16:20:50.503 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:20:50.846 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:20:50.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:50.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:20:50.949 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:20:50.979 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:51.007 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:20:51.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:51.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:20:51.111 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:20:51.162 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:51.181 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:20:51.372 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:20:51.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:51.377 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:20:51.455 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-23T16:20:51.455 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:20:51.455 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-04-23T16:20:51.455 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-23T16:20:51.455 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-23T16:20:51.455 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-04-23T16:20:51.478 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:20:51.540 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:51.579 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:20:51.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:51.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:20:51.643 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:20:51.687 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:20:51.729 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:51.774 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:20:51.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:51 smithi103 ceph-mon[39629]: pgmap v506: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 4.3 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 8270/141598 objects misplaced (5.840%); 5.2 MiB/s, 83 objects/s recovering 2024-04-23T16:20:51.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:51 smithi148 ceph-mon[44520]: pgmap v506: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 4.3 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 8270/141598 objects misplaced (5.840%); 5.2 MiB/s, 83 objects/s recovering 2024-04-23T16:20:51.955 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:20:51.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:51 smithi188 ceph-mon[44101]: pgmap v506: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 4.3 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 8270/141598 objects misplaced (5.840%); 5.2 MiB/s, 83 objects/s recovering 2024-04-23T16:20:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:51.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:20:52.062 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:20:52.107 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:52.154 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:20:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:52.161 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:20:52.262 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:20:52.322 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:52.452 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:20:52.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:52.456 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:20:52.557 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:20:52.603 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:52.626 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:20:52.626 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:20:52.626 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:52.626 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.14862.0:272502 2.c 2:3f4554b5:::benchmark_data_smithi188_77627_object103153:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-04-23T16:20:52.626 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:52.623471+0000", 2024-04-23T16:20:52.626 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0021464829999999998, 2024-04-23T16:20:52.627 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021701759999999998, 2024-04-23T16:20:52.627 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:20:52.627 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:20:52.627 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:20:52.627 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:20:52.627 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.14862", 2024-04-23T16:20:52.627 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:20:52.627 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 272502 2024-04-23T16:20:52.627 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:52.627 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:20:52.627 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:52.627 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:20:52.627 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:52.623471+0000", 2024-04-23T16:20:52.627 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:52.627 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:52.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:52.628 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:20:52.628 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:52.623471+0000", 2024-04-23T16:20:52.628 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:20:52.628 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:52.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:52.628 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:20:52.628 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:52.623473+0000", 2024-04-23T16:20:52.628 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3190000000000002e-06 2024-04-23T16:20:52.628 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:52.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:52.628 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:20:52.628 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:52.623481+0000", 2024-04-23T16:20:52.628 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6690000000000006e-06 2024-04-23T16:20:52.628 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:52.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:52.629 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:20:52.629 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:52.623498+0000", 2024-04-23T16:20:52.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7108e-05 2024-04-23T16:20:52.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:52.629 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:52.629 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:20:52.629 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:52.623509+0000", 2024-04-23T16:20:52.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0713999999999999e-05 2024-04-23T16:20:52.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:52.629 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:52.629 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:20:52.629 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:52.623574+0000", 2024-04-23T16:20:52.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4907999999999996e-05 2024-04-23T16:20:52.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:52.629 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:52.629 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:20:52.630 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:52.623700+0000", 2024-04-23T16:20:52.630 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012667000000000001 2024-04-23T16:20:52.630 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:20:52.630 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:20:52.630 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 0", 2024-04-23T16:20:52.630 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:52.623820+0000", 2024-04-23T16:20:52.630 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000120186 2024-04-23T16:20:52.630 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:52.630 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:20:52.630 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:52.630 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:20:52.630 INFO:teuthology.orchestra.run.smithi188.stdout: ], 2024-04-23T16:20:52.630 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 1 2024-04-23T16:20:52.630 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:20:52.681 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:20:52.972 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:20:52.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:52.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:20:52.976 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:52 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1913143000' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:20:52.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:52 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1913143000' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:20:53.077 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:20:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:52 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1913143000' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:20:53.126 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:53.477 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-23T16:20:53.485 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:20:53.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:53.490 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:20:53.590 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:20:53.627 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:53.649 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:20:53.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:53.651 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:20:53.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:53 smithi188 ceph-mon[44101]: pgmap v507: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 4.3 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 710 op/s; 8270/139452 objects misplaced (5.930%); 4.5 MiB/s, 71 objects/s recovering 2024-04-23T16:20:53.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:53 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:20:53.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:53 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1913143000' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:20:53.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:53 smithi188 ceph-mon[44101]: osdmap e169: 12 total, 12 up, 11 in 2024-04-23T16:20:53.752 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:20:53.792 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:53 smithi103 ceph-mon[39629]: pgmap v507: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 4.3 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 710 op/s; 8270/139452 objects misplaced (5.930%); 4.5 MiB/s, 71 objects/s recovering 2024-04-23T16:20:53.793 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:53 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:20:53.793 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:53 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1913143000' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:20:53.793 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:53 smithi103 ceph-mon[39629]: osdmap e169: 12 total, 12 up, 11 in 2024-04-23T16:20:53.795 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:53.868 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:20:53.871 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:20:53.871 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-23T16:20:53.871 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:20:53.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:53 smithi148 ceph-mon[44520]: pgmap v507: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 4.3 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 710 op/s; 8270/139452 objects misplaced (5.930%); 4.5 MiB/s, 71 objects/s recovering 2024-04-23T16:20:53.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:53 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:20:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:53 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1913143000' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:20:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:53 smithi148 ceph-mon[44520]: osdmap e169: 12 total, 12 up, 11 in 2024-04-23T16:20:53.953 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:20:53.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:53.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:20:54.058 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:20:54.095 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:54.215 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:20:54.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:54.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:20:54.323 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:54.357 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:54.562 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:54.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:54.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:54.622 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:20:54.660 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:20:54.670 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:20:54.709 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:20:54.731 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:54.747 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:20:54.760 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:20:54.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:54.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:20:54.781 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:20:54.820 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:20:54.854 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:20:54.867 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:20:54.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:54 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:20:54.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:54 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:20:54.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:54 smithi148 ceph-mon[44520]: osdmap e170: 12 total, 12 up, 11 in 2024-04-23T16:20:54.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:54 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/105582566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:54.891 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:20:54.902 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:54.927 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:20:54.938 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:20:54.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:54.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:20:54.963 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:20:54.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:54 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:20:54.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:54 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:20:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:54 smithi188 ceph-mon[44101]: osdmap e170: 12 total, 12 up, 11 in 2024-04-23T16:20:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:54 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/105582566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:54.996 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:20:55.040 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:20:55.041 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:20:55.059 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:20:55.059 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:20:55.059 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:20:55.059 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:20:55.059 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:20:55.059 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:20:55.059 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:20:55.059 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:20:55.059 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:20:55.059 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:20:55.059 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:20:55.059 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:20:55.091 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:54 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:20:55.091 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:54 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:20:55.091 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:54 smithi103 ceph-mon[39629]: osdmap e170: 12 total, 12 up, 11 in 2024-04-23T16:20:55.091 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:54 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/105582566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:20:55.103 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:55.139 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:20:55.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:55.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:20:55.243 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:20:55.297 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:55.457 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:20:55.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:55.461 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:20:55.562 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:20:55.593 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:55.615 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:20:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:55.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:20:55.719 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:20:55.753 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:55.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:55 smithi148 ceph-mon[44520]: pgmap v510: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 4.1 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 917 op/s; 6958/133306 objects misplaced (5.220%); 13 MiB/s, 215 objects/s recovering 2024-04-23T16:20:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:55 smithi188 ceph-mon[44101]: pgmap v510: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 4.1 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 917 op/s; 6958/133306 objects misplaced (5.220%); 13 MiB/s, 215 objects/s recovering 2024-04-23T16:20:55.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:55.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:56.005 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:20:56.050 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:55 smithi103 ceph-mon[39629]: pgmap v510: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 4.1 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 917 op/s; 6958/133306 objects misplaced (5.220%); 13 MiB/s, 215 objects/s recovering 2024-04-23T16:20:56.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:56.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:56.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:56.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:56.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:56.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:56.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:56.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:56.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:56.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:56.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:56.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:56.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:56 smithi188 ceph-mon[44101]: pgmap v511: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 3.9 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 956 op/s; 5811/128510 objects misplaced (4.522%); 19 MiB/s, 297 objects/s recovering 2024-04-23T16:20:57.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:57.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:56 smithi103 ceph-mon[39629]: pgmap v511: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 3.9 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 956 op/s; 5811/128510 objects misplaced (4.522%); 19 MiB/s, 297 objects/s recovering 2024-04-23T16:20:57.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:20:57.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:56 smithi148 ceph-mon[44520]: pgmap v511: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 3.9 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 956 op/s; 5811/128510 objects misplaced (4.522%); 19 MiB/s, 297 objects/s recovering 2024-04-23T16:20:57.256 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:20:58.218 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:20:58.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:58.225 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:20:58.326 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:20:58.384 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:58.622 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:20:58.622 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:20:58.862 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:20:58.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:58.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:20:58.972 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:20:59.013 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:59.149 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:20:59.378 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:20:59.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:59.385 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:20:59.397 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-23T16:20:59.486 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:20:59.518 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:59.540 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:20:59.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:20:59.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:20:59.575 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-23T16:20:59.575 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-23T16:20:59.644 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:20:59.724 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:20:59.740 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:20:59.788 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:20:59.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:59 smithi148 ceph-mon[44520]: pgmap v512: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 3.8 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 5811/125050 objects misplaced (4.647%); 15 MiB/s, 236 objects/s recovering 2024-04-23T16:20:59.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:59 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:20:59.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:59 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:20:59.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:59 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:20:59.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:20:59 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:20:59.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:59 smithi188 ceph-mon[44101]: pgmap v512: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 3.8 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 5811/125050 objects misplaced (4.647%); 15 MiB/s, 236 objects/s recovering 2024-04-23T16:20:59.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:59 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:20:59.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:59 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:20:59.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:59 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:20:59.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:20:59 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:21:00.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:59 smithi103 ceph-mon[39629]: pgmap v512: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 3.8 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 5811/125050 objects misplaced (4.647%); 15 MiB/s, 236 objects/s recovering 2024-04-23T16:21:00.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:59 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:21:00.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:59 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:21:00.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:59 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:21:00.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:20:59 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:21:00.040 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:21:00.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:00.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:21:00.146 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:21:00.199 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:00.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:00.294 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:21:00.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:21:00.395 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:21:00.434 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:00.636 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:00 smithi188 ceph-mon[44101]: osdmap e171: 12 total, 12 up, 11 in 2024-04-23T16:21:00.636 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:00 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/724713687' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:21:00.644 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:21:00.661 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:21:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:00.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:21:00.771 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:21:00.825 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:00 smithi103 ceph-mon[39629]: osdmap e171: 12 total, 12 up, 11 in 2024-04-23T16:21:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:00 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/724713687' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:21:01.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:00 smithi148 ceph-mon[44520]: osdmap e171: 12 total, 12 up, 11 in 2024-04-23T16:21:01.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:00 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/724713687' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:21:01.196 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:21:01.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:01.201 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:21:01.302 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:21:01.334 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:01.359 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:21:01.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:01.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:21:01.464 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:21:01.505 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:01.844 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:21:01.844 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:21:01.848 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:21:01.862 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:21:01.866 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:01 smithi188 ceph-mon[44101]: pgmap v514: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 3.7 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s; 3425/120868 objects misplaced (2.834%); 31 MiB/s, 493 objects/s recovering 2024-04-23T16:21:01.866 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:01 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:21:01.866 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:01 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/724713687' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:21:01.866 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:01 smithi188 ceph-mon[44101]: osdmap e172: 12 total, 12 up, 11 in 2024-04-23T16:21:01.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:01.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:21:01.974 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:21:02.004 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:01 smithi148 ceph-mon[44520]: pgmap v514: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 3.7 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s; 3425/120868 objects misplaced (2.834%); 31 MiB/s, 493 objects/s recovering 2024-04-23T16:21:02.004 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:01 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:21:02.004 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:01 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/724713687' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:21:02.004 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:01 smithi148 ceph-mon[44520]: osdmap e172: 12 total, 12 up, 11 in 2024-04-23T16:21:02.006 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:02.028 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:21:02.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:02.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:21:02.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:01 smithi103 ceph-mon[39629]: pgmap v514: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 3.7 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s; 3425/120868 objects misplaced (2.834%); 31 MiB/s, 493 objects/s recovering 2024-04-23T16:21:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:01 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:21:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:01 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/724713687' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:21:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:01 smithi103 ceph-mon[39629]: osdmap e172: 12 total, 12 up, 11 in 2024-04-23T16:21:02.132 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:21:02.173 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:02.593 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:21:02.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:02.605 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:21:02.632 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-23T16:21:02.707 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:02.754 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:02.779 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:02.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:02.788 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:02.889 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:21:02.920 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:02.947 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:21:02.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:02.952 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:21:03.053 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:21:03.110 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:03.133 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:21:03.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:03.141 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:21:03.242 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:21:03.286 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:03.318 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:21:03.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:03.324 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:21:03.425 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:21:03.456 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:03.481 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:21:03.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:03.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:21:03.584 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:21:03.623 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:03 smithi148 ceph-mon[44520]: pgmap v516: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 3.6 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 992 op/s; 3425/117418 objects misplaced (2.917%); 22 MiB/s, 359 objects/s recovering 2024-04-23T16:21:03.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:03 smithi188 ceph-mon[44101]: pgmap v516: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 3.6 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 992 op/s; 3425/117418 objects misplaced (2.917%); 22 MiB/s, 359 objects/s recovering 2024-04-23T16:21:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:03 smithi103 ceph-mon[39629]: pgmap v516: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 3.6 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 992 op/s; 3425/117418 objects misplaced (2.917%); 22 MiB/s, 359 objects/s recovering 2024-04-23T16:21:04.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:04 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:21:04.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:04 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:21:04.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:04 smithi188 ceph-mon[44101]: osdmap e173: 12 total, 12 up, 11 in 2024-04-23T16:21:04.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:04 smithi188 ceph-mon[44101]: 2.6 scrub starts 2024-04-23T16:21:05.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:04 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:21:05.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:04 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:21:05.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:04 smithi103 ceph-mon[39629]: osdmap e173: 12 total, 12 up, 11 in 2024-04-23T16:21:05.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:04 smithi103 ceph-mon[39629]: 2.6 scrub starts 2024-04-23T16:21:05.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:04 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:21:05.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:04 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:21:05.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:04 smithi148 ceph-mon[44520]: osdmap e173: 12 total, 12 up, 11 in 2024-04-23T16:21:05.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:04 smithi148 ceph-mon[44520]: 2.6 scrub starts 2024-04-23T16:21:05.968 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:05 smithi103 ceph-mon[39629]: pgmap v518: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 3.4 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s; 2358/111448 objects misplaced (2.116%); 31 MiB/s, 488 objects/s recovering 2024-04-23T16:21:05.968 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:05 smithi103 ceph-mon[39629]: 2.6 scrub ok 2024-04-23T16:21:05.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:05 smithi188 ceph-mon[44101]: pgmap v518: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 3.4 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s; 2358/111448 objects misplaced (2.116%); 31 MiB/s, 488 objects/s recovering 2024-04-23T16:21:05.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:05 smithi188 ceph-mon[44101]: 2.6 scrub ok 2024-04-23T16:21:05.986 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:21:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:06.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:21:06.103 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:21:06.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:05 smithi148 ceph-mon[44520]: pgmap v518: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 3.4 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s; 2358/111448 objects misplaced (2.116%); 31 MiB/s, 488 objects/s recovering 2024-04-23T16:21:06.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:05 smithi148 ceph-mon[44520]: 2.6 scrub ok 2024-04-23T16:21:06.133 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:06.154 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:21:06.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:06.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:21:06.258 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:06.287 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:06.309 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:06.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:06.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:06.418 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:21:06.470 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:06.727 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:21:06.732 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:21:06.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:06.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:21:06.842 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:21:06.895 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:06.929 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:21:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:06.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:21:07.034 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:21:07.090 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:07.744 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:21:07.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:07 smithi188 ceph-mon[44101]: pgmap v519: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 3.3 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 1187/106604 objects misplaced (1.113%); 18 MiB/s, 292 objects/s recovering 2024-04-23T16:21:07.982 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:07 smithi103 ceph-mon[39629]: pgmap v519: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 3.3 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 1187/106604 objects misplaced (1.113%); 18 MiB/s, 292 objects/s recovering 2024-04-23T16:21:08.098 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:21:08.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:08.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:21:08.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:07 smithi148 ceph-mon[44520]: pgmap v519: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 3.3 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 1187/106604 objects misplaced (1.113%); 18 MiB/s, 292 objects/s recovering 2024-04-23T16:21:08.202 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:08.229 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:08.248 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:08.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:08.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:08.352 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:21:08.379 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:08.396 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:21:08.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:08.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:21:08.501 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:21:08.519 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:21:08.527 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:08.538 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:21:08.545 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:21:08.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:08.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:21:08.556 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:21:08.576 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:21:08.596 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:21:08.615 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:21:08.634 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:21:08.649 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:21:08.654 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:21:08.673 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:21:08.674 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:08.682 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.0 to 1 (10000) 2024-04-23T16:21:08.696 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:21:08.697 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:21:08.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:08.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:21:08.717 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:21:08.742 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:21:08.745 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:21:08.745 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:21:08.745 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:21:08.746 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:21:08.746 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:21:08.746 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:21:08.746 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:21:08.746 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:21:08.746 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:21:08.746 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:21:08.746 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:21:08.746 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:21:08.801 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:08.831 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:08.831 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:08 smithi148 ceph-mon[44520]: 2.2 scrub starts 2024-04-23T16:21:08.831 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:08 smithi148 ceph-mon[44520]: osdmap e174: 12 total, 12 up, 11 in 2024-04-23T16:21:08.831 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:08 smithi148 ceph-mon[44520]: 2.2 scrub ok 2024-04-23T16:21:08.831 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:08 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4008952656' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:21:08.831 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:08 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3011949835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:21:08.850 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:08.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:08.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:08.955 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:21:08.956 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:08 smithi103 ceph-mon[39629]: 2.2 scrub starts 2024-04-23T16:21:08.956 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:08 smithi103 ceph-mon[39629]: osdmap e174: 12 total, 12 up, 11 in 2024-04-23T16:21:08.956 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:08 smithi103 ceph-mon[39629]: 2.2 scrub ok 2024-04-23T16:21:08.956 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:08 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4008952656' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:21:08.956 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:08 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3011949835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:21:08.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:08 smithi188 ceph-mon[44101]: 2.2 scrub starts 2024-04-23T16:21:08.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:08 smithi188 ceph-mon[44101]: osdmap e174: 12 total, 12 up, 11 in 2024-04-23T16:21:08.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:08 smithi188 ceph-mon[44101]: 2.2 scrub ok 2024-04-23T16:21:08.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:08 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4008952656' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:21:08.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:08 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3011949835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:21:08.991 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:09.019 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:21:09.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:09.020 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:21:09.121 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:21:09.149 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:09.173 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:21:09.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:09.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:21:09.277 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:21:09.308 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:09.331 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:21:09.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:09.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:21:09.434 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:09.463 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:09.491 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:09.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:09.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:09.598 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:21:09.644 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:09.697 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:09 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:21:09.697 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:09 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:21:09.697 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.697 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:09 smithi103 ceph-mon[39629]: pgmap v521: 17 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 14 active+clean; 3.2 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s; 1187/103236 objects misplaced (1.150%); 16 MiB/s, 256 objects/s recovering 2024-04-23T16:21:09.697 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.697 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.697 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.697 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.697 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.697 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.697 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.697 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.698 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:09 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4008952656' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:21:09.698 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:09 smithi103 ceph-mon[39629]: osdmap e175: 12 total, 12 up, 11 in 2024-04-23T16:21:09.698 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.698 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.698 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.721 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:21:09.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:09 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:21:09.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:09 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:21:09.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:09 smithi188 ceph-mon[44101]: pgmap v521: 17 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 14 active+clean; 3.2 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s; 1187/103236 objects misplaced (1.150%); 16 MiB/s, 256 objects/s recovering 2024-04-23T16:21:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:09 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4008952656' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:21:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:09 smithi188 ceph-mon[44101]: osdmap e175: 12 total, 12 up, 11 in 2024-04-23T16:21:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:09.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:09.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:21:09.837 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:09.865 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:09.885 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:09.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:09.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:09.989 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:21:10.046 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:09 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:21:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:09 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:21:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:09 smithi148 ceph-mon[44520]: pgmap v521: 17 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 14 active+clean; 3.2 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s; 1187/103236 objects misplaced (1.150%); 16 MiB/s, 256 objects/s recovering 2024-04-23T16:21:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:09 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4008952656' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:21:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:09 smithi148 ceph-mon[44520]: osdmap e175: 12 total, 12 up, 11 in 2024-04-23T16:21:10.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:10.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:10.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:11.234 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-23T16:21:11.266 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:21:11.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:11.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:21:11.372 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:21:11.414 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:11.446 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:21:11.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:11.452 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:21:11.553 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:11.588 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:11.632 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:21:11.632 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:21:11.632 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-23T16:21:11.632 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:21:11.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:11.633 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:11.633 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:11.653 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:21:11.734 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:11.772 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:11.797 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:11.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:11.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:11.905 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:21:11.909 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:21:11.932 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:11.951 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:21:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:11.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:21:11.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:11 smithi188 ceph-mon[44101]: pgmap v523: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.0 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s; 29 MiB/s, 459 objects/s recovering 2024-04-23T16:21:12.055 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:21:12.084 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:11 smithi148 ceph-mon[44520]: pgmap v523: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.0 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s; 29 MiB/s, 459 objects/s recovering 2024-04-23T16:21:12.086 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:12.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:11 smithi103 ceph-mon[39629]: pgmap v523: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.0 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s; 29 MiB/s, 459 objects/s recovering 2024-04-23T16:21:12.108 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:21:12.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:12.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:21:12.211 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:21:12.263 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:12.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:12 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:21:12.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:12 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:21:12.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:12 smithi103 ceph-mon[39629]: 2.e scrub starts 2024-04-23T16:21:12.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:12 smithi103 ceph-mon[39629]: osdmap e176: 12 total, 12 up, 11 in 2024-04-23T16:21:12.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:12 smithi103 ceph-mon[39629]: pgmap v525: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.0 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 819 op/s; 12 MiB/s, 190 objects/s recovering 2024-04-23T16:21:12.880 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:21:12.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:12 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:21:12.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:12 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:21:12.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:12 smithi188 ceph-mon[44101]: 2.e scrub starts 2024-04-23T16:21:12.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:12 smithi188 ceph-mon[44101]: osdmap e176: 12 total, 12 up, 11 in 2024-04-23T16:21:12.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:12 smithi188 ceph-mon[44101]: pgmap v525: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.0 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 819 op/s; 12 MiB/s, 190 objects/s recovering 2024-04-23T16:21:13.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:12 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:21:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:12 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:21:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:12 smithi148 ceph-mon[44520]: 2.e scrub starts 2024-04-23T16:21:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:12 smithi148 ceph-mon[44520]: osdmap e176: 12 total, 12 up, 11 in 2024-04-23T16:21:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:12 smithi148 ceph-mon[44520]: pgmap v525: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.0 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 819 op/s; 12 MiB/s, 190 objects/s recovering 2024-04-23T16:21:13.641 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:21:13.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:13.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:21:13.678 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-23T16:21:13.747 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:21:13.775 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:13.797 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:21:13.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:13.802 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:21:13.903 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:21:13.977 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:13 smithi103 ceph-mon[39629]: 2.e scrub ok 2024-04-23T16:21:13.988 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:14.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:13 smithi148 ceph-mon[44520]: 2.e scrub ok 2024-04-23T16:21:14.200 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:21:14.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:14.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:21:14.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:13 smithi188 ceph-mon[44101]: 2.e scrub ok 2024-04-23T16:21:14.310 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:21:14.337 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:14.362 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:21:14.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:14.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:21:14.470 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:21:14.504 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:14.525 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:21:14.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:14.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:21:14.629 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:21:14.673 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:14.833 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:21:14.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:14.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:21:14.940 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:21:14.972 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:14.975 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:14 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:21:14.976 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:14 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:21:14.976 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:14 smithi103 ceph-mon[39629]: osdmap e177: 12 total, 12 up, 11 in 2024-04-23T16:21:14.976 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:21:14.976 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:21:14.976 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:21:14.976 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:21:14.976 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:21:14.976 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:21:14.976 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:14 smithi103 ceph-mon[39629]: from='mgr.14154 ' entity='mgr.y' 2024-04-23T16:21:14.993 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:21:14.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:14.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:21:15.097 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:21:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:14 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:21:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:14 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:21:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:14 smithi148 ceph-mon[44520]: osdmap e177: 12 total, 12 up, 11 in 2024-04-23T16:21:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:21:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:21:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:21:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:21:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:21:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:21:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:14 smithi148 ceph-mon[44520]: from='mgr.14154 ' entity='mgr.y' 2024-04-23T16:21:15.179 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:15.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:14 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:21:15.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:14 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:21:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:14 smithi188 ceph-mon[44101]: osdmap e177: 12 total, 12 up, 11 in 2024-04-23T16:21:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:21:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:21:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:21:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:21:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:21:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:21:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:14 smithi188 ceph-mon[44101]: from='mgr.14154 ' entity='mgr.y' 2024-04-23T16:21:15.764 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:21:15.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:15.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:21:15.875 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:21:15.905 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:15.929 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:21:15.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:15.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:21:16.033 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:21:16.062 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:16.083 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:21:16.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:16.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:21:16.190 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:21:16.229 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:16.347 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:21:16.347 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:21:16.350 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:21:16.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:16.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:21:16.460 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:16.486 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:16.506 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:16.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:16.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:16.612 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:21:16.639 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:16.662 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:21:16.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:16.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:21:16.766 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:21:16.794 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:16.820 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:21:16.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:16.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:21:16.924 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:16.956 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:16.983 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:16.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:16.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:17.088 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:21:17.124 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:17.161 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:21:17.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:17.165 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:21:17.265 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:21:17.312 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:17.608 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:21:17.618 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.1 to 1 (10000) 2024-04-23T16:21:17.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:17 smithi148 ceph-mon[44520]: pgmap v528: 17 pgs: 17 active+clean; 2.6 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s; 9.1 MiB/s, 145 objects/s recovering 2024-04-23T16:21:17.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:17 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3140012670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:21:17.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:17 smithi148 ceph-mon[44520]: from='client.24934 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:21:17.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:17 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4109290507' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:21:17.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:17 smithi188 ceph-mon[44101]: pgmap v528: 17 pgs: 17 active+clean; 2.6 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s; 9.1 MiB/s, 145 objects/s recovering 2024-04-23T16:21:17.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:17 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3140012670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:21:17.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:17 smithi188 ceph-mon[44101]: from='client.24934 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:21:17.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:17 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4109290507' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:21:18.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:17 smithi103 ceph-mon[39629]: pgmap v528: 17 pgs: 17 active+clean; 2.6 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s; 9.1 MiB/s, 145 objects/s recovering 2024-04-23T16:21:18.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:17 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3140012670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:21:18.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:17 smithi103 ceph-mon[39629]: from='client.24934 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:21:18.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:17 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4109290507' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:21:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:18 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:21:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:18 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:21:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:18 smithi148 ceph-mon[44520]: from='client.24934 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:21:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:18 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4109290507' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-23T16:21:18.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:18 smithi148 ceph-mon[44520]: osdmap e178: 12 total, 12 up, 11 in 2024-04-23T16:21:18.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:18 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:21:18.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:18 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:21:18.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:18 smithi188 ceph-mon[44101]: from='client.24934 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:21:18.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:18 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4109290507' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-23T16:21:18.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:18 smithi188 ceph-mon[44101]: osdmap e178: 12 total, 12 up, 11 in 2024-04-23T16:21:19.024 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:18 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:21:19.024 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:18 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:21:19.024 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:18 smithi103 ceph-mon[39629]: from='client.24934 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:21:19.025 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:18 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4109290507' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-23T16:21:19.025 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:18 smithi103 ceph-mon[39629]: osdmap e178: 12 total, 12 up, 11 in 2024-04-23T16:21:19.290 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:21:19.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:19.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:21:19.405 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:21:19.434 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:19.453 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:21:19.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:19.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:21:19.559 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:21:19.590 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:19.610 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:21:19.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:19.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:21:19.713 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:21:19.740 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:19 smithi148 ceph-mon[44520]: pgmap v530: 17 pgs: 17 active+clean; 2.5 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:21:19.742 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:19.764 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:21:19.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:19.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:21:19.868 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:21:19.874 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-23T16:21:19.875 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:19 smithi103 ceph-mon[39629]: pgmap v530: 17 pgs: 17 active+clean; 2.5 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:21:19.944 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:19.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:19 smithi188 ceph-mon[44101]: pgmap v530: 17 pgs: 17 active+clean; 2.5 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:21:20.074 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:21:20.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:20.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:21:20.146 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:21:20.178 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:20.184 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:21:20.212 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:20.230 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:21:20.236 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:20.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:20.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:20.273 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:21:20.309 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:21:20.340 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:21:20.343 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:21:20.372 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:21:20.378 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:20.412 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:21:20.413 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:21:20.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:20.420 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:21:20.453 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:21:20.481 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:21:20.510 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:21:20.521 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:21:20.538 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:21:20.561 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:21:20.561 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:21:20.561 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:21:20.561 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:21:20.561 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:21:20.561 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:21:20.562 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:21:20.562 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:21:20.562 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:21:20.562 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:21:20.562 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:21:20.562 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:21:20.562 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:20.790 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:21:20.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:20.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:21:20.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:20 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1099122184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:21:20.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:20.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:20.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:20.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:20.896 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:21:20.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:20 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1099122184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:21:20.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:20.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:20.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:20.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:20.931 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:20.957 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:21:20.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:20.960 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:21:21.061 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:21:21.116 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:21.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:20 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1099122184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:21:21.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:21.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:21 smithi188 ceph-mon[44101]: pgmap v531: 17 pgs: 17 active+clean; 2.4 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:21:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:21 smithi188 ceph-mon[44101]: 2.8 scrub starts 2024-04-23T16:21:22.071 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:22.072 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:22.072 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:22.072 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:22.072 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:22.072 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:22.072 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:22.072 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:21 smithi103 ceph-mon[39629]: pgmap v531: 17 pgs: 17 active+clean; 2.4 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:21:22.072 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:22.072 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:21 smithi103 ceph-mon[39629]: 2.8 scrub starts 2024-04-23T16:21:22.097 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:21:22.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:22.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:21:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:21 smithi148 ceph-mon[44520]: pgmap v531: 17 pgs: 17 active+clean; 2.4 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:21:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:21 smithi148 ceph-mon[44520]: 2.8 scrub starts 2024-04-23T16:21:22.206 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:21:22.253 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:22.391 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:21:22.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:22.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:21:22.497 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:22.526 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:22.550 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:22.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:22.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:22.655 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:21:22.662 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:22 smithi103 ceph-mon[39629]: 2.8 scrub ok 2024-04-23T16:21:22.662 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:22 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:21:22.662 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:22 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:21:22.662 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:22 smithi103 ceph-mon[39629]: osdmap e179: 12 total, 12 up, 11 in 2024-04-23T16:21:22.712 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:22.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:22 smithi148 ceph-mon[44520]: 2.8 scrub ok 2024-04-23T16:21:22.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:22 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:21:22.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:22 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:21:22.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:22 smithi148 ceph-mon[44520]: osdmap e179: 12 total, 12 up, 11 in 2024-04-23T16:21:22.930 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:21:22.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:22.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:21:22.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:22 smithi188 ceph-mon[44101]: 2.8 scrub ok 2024-04-23T16:21:22.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:22 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:21:22.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:22 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:21:22.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:22 smithi188 ceph-mon[44101]: osdmap e179: 12 total, 12 up, 11 in 2024-04-23T16:21:23.034 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:21:23.099 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:23.134 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:21:23.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:23.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:21:23.242 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:21:23.256 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:21:23.305 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:23.350 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:21:23.350 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:21:23.888 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:21:23.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:23.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:21:23.940 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:23 smithi103 ceph-mon[39629]: pgmap v533: 17 pgs: 17 active+clean; 2.3 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-23T16:21:23.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:23 smithi188 ceph-mon[44101]: pgmap v533: 17 pgs: 17 active+clean; 2.3 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-23T16:21:24.006 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:21:24.031 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:23 smithi148 ceph-mon[44520]: pgmap v533: 17 pgs: 17 active+clean; 2.3 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-23T16:21:24.033 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:24.059 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:21:24.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:24.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:21:24.115 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:21:24.163 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:21:24.204 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:24.239 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:21:24.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:24.244 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:21:24.344 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:21:24.384 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:24.409 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:21:24.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:24.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:21:24.499 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-23T16:21:24.517 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:21:24.612 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:24.970 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:21:25.010 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:21:25.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:25.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:21:25.123 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:25.164 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:25.185 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:25.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:25.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:25.293 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:21:25.326 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:25.354 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:21:25.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:25.363 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:21:25.464 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:21:25.517 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:25.603 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:21:25.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:25.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:21:25.609 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:21:25.613 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:21:25.709 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:21:25.748 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:25.779 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:21:25.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:25.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:21:25.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:25 smithi103 ceph-mon[39629]: pgmap v534: 17 pgs: 17 active+clean; 2.0 GiB data, 4.9 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:21:25.886 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:21:25.918 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:25.948 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:21:25.949 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:25 smithi188 ceph-mon[44101]: pgmap v534: 17 pgs: 17 active+clean; 2.0 GiB data, 4.9 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:21:25.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:25.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:21:26.051 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:21:26.103 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:26.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:25 smithi148 ceph-mon[44520]: pgmap v534: 17 pgs: 17 active+clean; 2.0 GiB data, 4.9 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:21:26.191 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:21:26.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:26.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:21:26.299 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:26.326 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:26.350 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:26.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:26.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:26.455 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:21:26.485 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:26.507 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:21:26.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:26.510 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:21:26.611 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:26.651 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:26.683 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:26.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:26.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:26.786 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:26.821 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:26.845 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:26.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:26.848 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:26.949 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:21:27.066 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:27.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:27 smithi188 ceph-mon[44101]: pgmap v535: 17 pgs: 17 active+clean; 1.9 GiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.34k op/s 2024-04-23T16:21:28.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:27 smithi103 ceph-mon[39629]: pgmap v535: 17 pgs: 17 active+clean; 1.9 GiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.34k op/s 2024-04-23T16:21:28.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:27 smithi148 ceph-mon[44520]: pgmap v535: 17 pgs: 17 active+clean; 1.9 GiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.34k op/s 2024-04-23T16:21:28.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:28 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:21:28.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:28 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:21:28.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:28 smithi103 ceph-mon[39629]: osdmap e180: 12 total, 12 up, 11 in 2024-04-23T16:21:28.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:28 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:21:28.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:28 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:21:28.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:28 smithi188 ceph-mon[44101]: osdmap e180: 12 total, 12 up, 11 in 2024-04-23T16:21:29.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:28 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:21:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:28 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:21:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:28 smithi148 ceph-mon[44520]: osdmap e180: 12 total, 12 up, 11 in 2024-04-23T16:21:29.233 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:21:29.248 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:21:29.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:29.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:21:29.360 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:21:29.392 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:29.424 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:21:29.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:29.430 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:21:29.531 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:21:29.576 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:29 smithi103 ceph-mon[39629]: pgmap v537: 17 pgs: 17 active+clean; 1.8 GiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:21:29.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:29 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4044595582' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:21:29.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:29 smithi103 ceph-mon[39629]: from='client.34994 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:21:29.907 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.2 to 1 (10000) 2024-04-23T16:21:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:29 smithi188 ceph-mon[44101]: pgmap v537: 17 pgs: 17 active+clean; 1.8 GiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:21:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:29 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4044595582' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:21:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:29 smithi188 ceph-mon[44101]: from='client.34994 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:21:30.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:29 smithi148 ceph-mon[44520]: pgmap v537: 17 pgs: 17 active+clean; 1.8 GiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:21:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:29 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4044595582' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:21:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:29 smithi148 ceph-mon[44520]: from='client.34994 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:21:30.645 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:21:30.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:30.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:21:30.758 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:21:30.792 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:30.825 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:21:30.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:30.831 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:21:30.925 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:30 smithi103 ceph-mon[39629]: from='client.34994 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-23T16:21:30.925 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:30 smithi103 ceph-mon[39629]: osdmap e181: 12 total, 12 up, 11 in 2024-04-23T16:21:30.926 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:30 smithi103 ceph-mon[39629]: pgmap v539: 17 pgs: 17 active+clean; 1.6 GiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:21:30.932 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:21:30.963 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:30.999 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:21:31.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:31.001 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:21:31.102 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:31.146 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:30 smithi188 ceph-mon[44101]: from='client.34994 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-23T16:21:31.146 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:30 smithi188 ceph-mon[44101]: osdmap e181: 12 total, 12 up, 11 in 2024-04-23T16:21:31.147 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:30 smithi188 ceph-mon[44101]: pgmap v539: 17 pgs: 17 active+clean; 1.6 GiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:21:31.149 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:31.176 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:31.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:31.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:31.280 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:21:31.313 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:31.345 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:21:31.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:31.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:21:31.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:30 smithi148 ceph-mon[44520]: from='client.34994 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-23T16:21:31.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:30 smithi148 ceph-mon[44520]: osdmap e181: 12 total, 12 up, 11 in 2024-04-23T16:21:31.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:30 smithi148 ceph-mon[44520]: pgmap v539: 17 pgs: 17 active+clean; 1.6 GiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:21:31.450 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:31.479 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:31.510 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:31.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:31.526 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:31.627 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:21:31.653 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:31.676 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:21:31.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:31.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:21:31.780 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:21:31.820 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:31.841 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:21:31.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:31.844 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:21:31.919 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:21:31.919 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:21:31.919 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:21:31.919 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:21:31.919 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:21:31.919 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:21:31.919 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:21:31.919 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:21:31.919 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:21:31.919 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:21:31.919 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:21:31.945 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:21:31.948 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:21:31.962 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:21:31.962 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:21:31.962 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:21:31.962 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:21:31.962 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:21:31.962 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:21:31.963 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:21:31.963 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:21:31.963 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:21:31.963 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:21:31.963 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:21:31.963 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:21:32.000 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:32.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:31 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/192395913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:21:32.296 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:31 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/192395913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:21:32.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:31 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/192395913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:21:32.377 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:21:32.377 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:21:32.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:21:32.377 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:21:32.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.377 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:156618 2.4 e141/110 2:27cc5cce:::benchmark_data_smithi188_77627_object156617:head v 141'9766, mlcod=141'9763)", 2024-04-23T16:21:32.377 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:08.203486+0000", 2024-04-23T16:21:32.378 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 144.05709904099999, 2024-04-23T16:21:32.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.042624875999999999, 2024-04-23T16:21:32.378 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.378 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.378 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.378 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.203486+0000", 2024-04-23T16:21:32.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.203486+0000", 2024-04-23T16:21:32.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.203488+0000", 2024-04-23T16:21:32.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.607e-06 2024-04-23T16:21:32.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.203519+0000", 2024-04-23T16:21:32.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1420000000000001e-05 2024-04-23T16:21:32.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.203532+0000", 2024-04-23T16:21:32.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3123e-05 2024-04-23T16:21:32.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.203540+0000", 2024-04-23T16:21:32.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5839999999999997e-06 2024-04-23T16:21:32.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.209404+0000", 2024-04-23T16:21:32.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058641910000000004 2024-04-23T16:21:32.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.209450+0000", 2024-04-23T16:21:32.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6193999999999999e-05 2024-04-23T16:21:32.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.246059+0000", 2024-04-23T16:21:32.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.036608887999999999 2024-04-23T16:21:32.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.246111+0000", 2024-04-23T16:21:32.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1869000000000001e-05 2024-04-23T16:21:32.381 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.382 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.382 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.382 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:156619 2.4 e141/110 2:2cd1dde6:::benchmark_data_smithi188_77627_object156618:head v 141'9767, mlcod=141'9763)", 2024-04-23T16:21:32.382 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:08.204271+0000", 2024-04-23T16:21:32.382 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 144.056314501, 2024-04-23T16:21:32.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.041907252999999998, 2024-04-23T16:21:32.382 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.382 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.382 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.382 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.204271+0000", 2024-04-23T16:21:32.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.204271+0000", 2024-04-23T16:21:32.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.204272+0000", 2024-04-23T16:21:32.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.209e-06 2024-04-23T16:21:32.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.204296+0000", 2024-04-23T16:21:32.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4219e-05 2024-04-23T16:21:32.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.204308+0000", 2024-04-23T16:21:32.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1990999999999999e-05 2024-04-23T16:21:32.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.204315+0000", 2024-04-23T16:21:32.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4840000000000001e-06 2024-04-23T16:21:32.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.218684+0000", 2024-04-23T16:21:32.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014369036999999999 2024-04-23T16:21:32.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.218697+0000", 2024-04-23T16:21:32.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3392e-05 2024-04-23T16:21:32.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.246154+0000", 2024-04-23T16:21:32.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.027457286000000001 2024-04-23T16:21:32.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:08.246178+0000", 2024-04-23T16:21:32.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3635e-05 2024-04-23T16:21:32.386 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.386 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.386 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.386 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:160104 2.4 e141/110 2:243d968e:::benchmark_data_smithi188_77627_object160103:head v 141'9974, mlcod=141'9970)", 2024-04-23T16:21:32.386 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:11.511965+0000", 2024-04-23T16:21:32.387 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 140.74862060999999, 2024-04-23T16:21:32.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.040556559999999998, 2024-04-23T16:21:32.387 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.387 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.387 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.387 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.511965+0000", 2024-04-23T16:21:32.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.511965+0000", 2024-04-23T16:21:32.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.511967+0000", 2024-04-23T16:21:32.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0650000000000001e-06 2024-04-23T16:21:32.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.511999+0000", 2024-04-23T16:21:32.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1786999999999999e-05 2024-04-23T16:21:32.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.512012+0000", 2024-04-23T16:21:32.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3529e-05 2024-04-23T16:21:32.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.512021+0000", 2024-04-23T16:21:32.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1710000000000008e-06 2024-04-23T16:21:32.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.514073+0000", 2024-04-23T16:21:32.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020512249999999998 2024-04-23T16:21:32.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.514088+0000", 2024-04-23T16:21:32.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5384e-05 2024-04-23T16:21:32.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.552467+0000", 2024-04-23T16:21:32.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.03837898 2024-04-23T16:21:32.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.552521+0000", 2024-04-23T16:21:32.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4419000000000002e-05 2024-04-23T16:21:32.390 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.390 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.390 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.391 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:160122 2.5 e141/110 2:ac711dda:::benchmark_data_smithi188_77627_object160121:head v 141'9991, mlcod=141'9988)", 2024-04-23T16:21:32.391 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:11.515294+0000", 2024-04-23T16:21:32.391 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 140.745291546, 2024-04-23T16:21:32.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.041094482000000002, 2024-04-23T16:21:32.391 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.391 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.391 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.391 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.515294+0000", 2024-04-23T16:21:32.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.515294+0000", 2024-04-23T16:21:32.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.515294+0000", 2024-04-23T16:21:32.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9599999999999999e-07 2024-04-23T16:21:32.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.515397+0000", 2024-04-23T16:21:32.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000102569 2024-04-23T16:21:32.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.515403+0000", 2024-04-23T16:21:32.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1210000000000003e-06 2024-04-23T16:21:32.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.515408+0000", 2024-04-23T16:21:32.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.31e-06 2024-04-23T16:21:32.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.520631+0000", 2024-04-23T16:21:32.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052227690000000004 2024-04-23T16:21:32.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.520656+0000", 2024-04-23T16:21:32.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4678999999999999e-05 2024-04-23T16:21:32.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.556345+0000", 2024-04-23T16:21:32.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.035689027999999998 2024-04-23T16:21:32.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.556388+0000", 2024-04-23T16:21:32.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3309999999999997e-05 2024-04-23T16:21:32.394 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.394 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.395 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.395 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:295382 2.c e177/110 2:35c01739:::benchmark_data_smithi188_77627_object126033:head v 177'18367, mlcod=177'18365)", 2024-04-23T16:21:32.395 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:13.977698+0000", 2024-04-23T16:21:32.395 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.282887514999999, 2024-04-23T16:21:32.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.041153298999999997, 2024-04-23T16:21:32.395 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.395 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.395 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.395 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:13.977698+0000", 2024-04-23T16:21:32.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:13.977698+0000", 2024-04-23T16:21:32.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:13.977700+0000", 2024-04-23T16:21:32.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7630000000000001e-06 2024-04-23T16:21:32.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:13.977709+0000", 2024-04-23T16:21:32.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7790000000000007e-06 2024-04-23T16:21:32.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:13.977724+0000", 2024-04-23T16:21:32.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4773999999999999e-05 2024-04-23T16:21:32.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:13.977733+0000", 2024-04-23T16:21:32.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9449999999999999e-06 2024-04-23T16:21:32.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:13.977789+0000", 2024-04-23T16:21:32.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5952999999999998e-05 2024-04-23T16:21:32.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:13.977815+0000", 2024-04-23T16:21:32.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6347e-05 2024-04-23T16:21:32.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:14.018803+0000", 2024-04-23T16:21:32.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.040987827999999997 2024-04-23T16:21:32.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:14.018851+0000", 2024-04-23T16:21:32.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.791e-05 2024-04-23T16:21:32.399 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.399 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.399 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.399 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:300629 2.4 e178/110 2:28abfb87:::benchmark_data_smithi188_77627_object131280:head v 178'18783, mlcod=178'18778)", 2024-04-23T16:21:32.399 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:18.369607+0000", 2024-04-23T16:21:32.399 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.890978358, 2024-04-23T16:21:32.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.041442385999999998, 2024-04-23T16:21:32.400 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.400 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.400 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.400 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.369607+0000", 2024-04-23T16:21:32.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.369607+0000", 2024-04-23T16:21:32.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.369607+0000", 2024-04-23T16:21:32.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2599999999999998e-07 2024-04-23T16:21:32.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.369612+0000", 2024-04-23T16:21:32.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9350000000000002e-06 2024-04-23T16:21:32.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.369617+0000", 2024-04-23T16:21:32.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.826e-06 2024-04-23T16:21:32.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.369622+0000", 2024-04-23T16:21:32.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5519999999999998e-06 2024-04-23T16:21:32.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.373642+0000", 2024-04-23T16:21:32.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0040202989999999997 2024-04-23T16:21:32.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.373664+0000", 2024-04-23T16:21:32.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2342000000000001e-05 2024-04-23T16:21:32.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.410990+0000", 2024-04-23T16:21:32.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.037326047000000001 2024-04-23T16:21:32.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.411049+0000", 2024-04-23T16:21:32.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8959000000000003e-05 2024-04-23T16:21:32.403 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.403 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.403 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.404 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:300631 2.4 e178/110 2:293c394f:::benchmark_data_smithi188_77627_object131282:head v 178'18784, mlcod=178'18778)", 2024-04-23T16:21:32.404 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:18.370987+0000", 2024-04-23T16:21:32.404 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.889598024, 2024-04-23T16:21:32.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.040089161999999998, 2024-04-23T16:21:32.404 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.404 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.404 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.404 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.370987+0000", 2024-04-23T16:21:32.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.370987+0000", 2024-04-23T16:21:32.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.370989+0000", 2024-04-23T16:21:32.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1939999999999999e-06 2024-04-23T16:21:32.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.370995+0000", 2024-04-23T16:21:32.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5259999999999999e-06 2024-04-23T16:21:32.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.371005+0000", 2024-04-23T16:21:32.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0091e-05 2024-04-23T16:21:32.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.371011+0000", 2024-04-23T16:21:32.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6459999999999998e-06 2024-04-23T16:21:32.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.377829+0000", 2024-04-23T16:21:32.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068180330000000003 2024-04-23T16:21:32.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.377848+0000", 2024-04-23T16:21:32.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8955999999999999e-05 2024-04-23T16:21:32.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.411053+0000", 2024-04-23T16:21:32.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033205126000000001 2024-04-23T16:21:32.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.411076+0000", 2024-04-23T16:21:32.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3589999999999999e-05 2024-04-23T16:21:32.408 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.408 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.408 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.408 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:300646 2.c e178/110 2:389114a4:::benchmark_data_smithi188_77627_object131297:head v 178'18686, mlcod=178'18684)", 2024-04-23T16:21:32.408 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:18.371191+0000", 2024-04-23T16:21:32.408 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.889394686999999, 2024-04-23T16:21:32.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.044331044, 2024-04-23T16:21:32.408 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.409 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.409 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.409 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.371191+0000", 2024-04-23T16:21:32.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.371191+0000", 2024-04-23T16:21:32.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.371192+0000", 2024-04-23T16:21:32.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.638e-06 2024-04-23T16:21:32.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.371201+0000", 2024-04-23T16:21:32.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2619999999999998e-06 2024-04-23T16:21:32.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.371212+0000", 2024-04-23T16:21:32.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1576999999999999e-05 2024-04-23T16:21:32.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.371219+0000", 2024-04-23T16:21:32.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.72e-06 2024-04-23T16:21:32.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.377433+0000", 2024-04-23T16:21:32.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062137870000000001 2024-04-23T16:21:32.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.377458+0000", 2024-04-23T16:21:32.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5074000000000001e-05 2024-04-23T16:21:32.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.415484+0000", 2024-04-23T16:21:32.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.038025947999999997 2024-04-23T16:21:32.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.415522+0000", 2024-04-23T16:21:32.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8037999999999997e-05 2024-04-23T16:21:32.412 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.412 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.413 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.413 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:300647 2.c e178/110 2:30549918:::benchmark_data_smithi188_77627_object131298:head v 178'18687, mlcod=178'18684)", 2024-04-23T16:21:32.413 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:18.375300+0000", 2024-04-23T16:21:32.413 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.885285474, 2024-04-23T16:21:32.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.052515903000000003, 2024-04-23T16:21:32.413 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.413 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.413 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.413 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.375300+0000", 2024-04-23T16:21:32.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.375300+0000", 2024-04-23T16:21:32.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.375301+0000", 2024-04-23T16:21:32.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.42e-06 2024-04-23T16:21:32.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.375309+0000", 2024-04-23T16:21:32.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3919999999999997e-06 2024-04-23T16:21:32.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.375320+0000", 2024-04-23T16:21:32.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.114e-05 2024-04-23T16:21:32.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.375327+0000", 2024-04-23T16:21:32.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7209999999999997e-06 2024-04-23T16:21:32.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.390835+0000", 2024-04-23T16:21:32.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01550849 2024-04-23T16:21:32.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.390855+0000", 2024-04-23T16:21:32.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9661999999999999e-05 2024-04-23T16:21:32.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.427766+0000", 2024-04-23T16:21:32.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.036911117 2024-04-23T16:21:32.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.427816+0000", 2024-04-23T16:21:32.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9960999999999998e-05 2024-04-23T16:21:32.417 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.417 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.417 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.417 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:300632 2.4 e178/110 2:2b08d9d8:::benchmark_data_smithi188_77627_object131283:head v 178'18785, mlcod=178'18780)", 2024-04-23T16:21:32.417 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:18.376281+0000", 2024-04-23T16:21:32.417 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.884304718999999, 2024-04-23T16:21:32.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.042133480000000001, 2024-04-23T16:21:32.417 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.417 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.418 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.418 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.376281+0000", 2024-04-23T16:21:32.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.376281+0000", 2024-04-23T16:21:32.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.376282+0000", 2024-04-23T16:21:32.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3740000000000001e-06 2024-04-23T16:21:32.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.376289+0000", 2024-04-23T16:21:32.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3159999999999999e-06 2024-04-23T16:21:32.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.376302+0000", 2024-04-23T16:21:32.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3050999999999999e-05 2024-04-23T16:21:32.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.376309+0000", 2024-04-23T16:21:32.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1219999999999997e-06 2024-04-23T16:21:32.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.386504+0000", 2024-04-23T16:21:32.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010194561 2024-04-23T16:21:32.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.386524+0000", 2024-04-23T16:21:32.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9471999999999999e-05 2024-04-23T16:21:32.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.418378+0000", 2024-04-23T16:21:32.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.031854741999999998 2024-04-23T16:21:32.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.418414+0000", 2024-04-23T16:21:32.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5842e-05 2024-04-23T16:21:32.421 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.421 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.421 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.422 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:300659 2.c e178/110 2:31333e8c:::benchmark_data_smithi188_77627_object131310:head v 178'18688, mlcod=178'18684)", 2024-04-23T16:21:32.422 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:18.387327+0000", 2024-04-23T16:21:32.422 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.873258478, 2024-04-23T16:21:32.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.040535779000000001, 2024-04-23T16:21:32.422 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.422 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.422 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.422 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.387327+0000", 2024-04-23T16:21:32.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.387327+0000", 2024-04-23T16:21:32.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.387329+0000", 2024-04-23T16:21:32.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.688e-06 2024-04-23T16:21:32.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.387338+0000", 2024-04-23T16:21:32.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2210000000000003e-06 2024-04-23T16:21:32.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.387348+0000", 2024-04-23T16:21:32.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0071000000000001e-05 2024-04-23T16:21:32.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.387356+0000", 2024-04-23T16:21:32.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6639999999999998e-06 2024-04-23T16:21:32.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.395483+0000", 2024-04-23T16:21:32.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.008127502 2024-04-23T16:21:32.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.395499+0000", 2024-04-23T16:21:32.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6347000000000001e-05 2024-04-23T16:21:32.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.427848+0000", 2024-04-23T16:21:32.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.032348844000000002 2024-04-23T16:21:32.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.427863+0000", 2024-04-23T16:21:32.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4442e-05 2024-04-23T16:21:32.426 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.426 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.426 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.427 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:300658 2.4 e178/110 2:2a1c1c04:::benchmark_data_smithi188_77627_object131309:head v 178'18786, mlcod=178'18780)", 2024-04-23T16:21:32.427 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:18.387828+0000", 2024-04-23T16:21:32.427 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.872757345, 2024-04-23T16:21:32.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.040018277999999997, 2024-04-23T16:21:32.427 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.427 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.427 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.427 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.387828+0000", 2024-04-23T16:21:32.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.387828+0000", 2024-04-23T16:21:32.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.387830+0000", 2024-04-23T16:21:32.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.719e-06 2024-04-23T16:21:32.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.387838+0000", 2024-04-23T16:21:32.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3399999999999998e-06 2024-04-23T16:21:32.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.387849+0000", 2024-04-23T16:21:32.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1299e-05 2024-04-23T16:21:32.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.387857+0000", 2024-04-23T16:21:32.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2400000000000001e-06 2024-04-23T16:21:32.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.395112+0000", 2024-04-23T16:21:32.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007254941 2024-04-23T16:21:32.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.395134+0000", 2024-04-23T16:21:32.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2518e-05 2024-04-23T16:21:32.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.427823+0000", 2024-04-23T16:21:32.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.032689233999999998 2024-04-23T16:21:32.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.427846+0000", 2024-04-23T16:21:32.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2986999999999999e-05 2024-04-23T16:21:32.431 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.431 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.431 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.431 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:300660 2.4 e178/110 2:2ac5f260:::benchmark_data_smithi188_77627_object131311:head v 178'18787, mlcod=178'18780)", 2024-04-23T16:21:32.431 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:18.391958+0000", 2024-04-23T16:21:32.431 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.868627222000001, 2024-04-23T16:21:32.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046088456999999999, 2024-04-23T16:21:32.432 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.432 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.432 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.432 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.391958+0000", 2024-04-23T16:21:32.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.391958+0000", 2024-04-23T16:21:32.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.391960+0000", 2024-04-23T16:21:32.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6050000000000001e-06 2024-04-23T16:21:32.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.391968+0000", 2024-04-23T16:21:32.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8029999999999997e-06 2024-04-23T16:21:32.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.391976+0000", 2024-04-23T16:21:32.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3629999999999994e-06 2024-04-23T16:21:32.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.391982+0000", 2024-04-23T16:21:32.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.993e-06 2024-04-23T16:21:32.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.399812+0000", 2024-04-23T16:21:32.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078298269999999993 2024-04-23T16:21:32.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.399829+0000", 2024-04-23T16:21:32.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7076999999999999e-05 2024-04-23T16:21:32.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.438001+0000", 2024-04-23T16:21:32.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.038171990000000003 2024-04-23T16:21:32.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.438047+0000", 2024-04-23T16:21:32.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5799e-05 2024-04-23T16:21:32.435 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.435 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.435 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.436 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:300665 2.c e178/110 2:3ac07bbd:::benchmark_data_smithi188_77627_object131316:head v 178'18689, mlcod=178'18684)", 2024-04-23T16:21:32.436 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:18.392281+0000", 2024-04-23T16:21:32.436 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.868304069000001, 2024-04-23T16:21:32.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045797183999999998, 2024-04-23T16:21:32.436 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.436 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.436 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.436 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.392281+0000", 2024-04-23T16:21:32.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.392281+0000", 2024-04-23T16:21:32.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.392282+0000", 2024-04-23T16:21:32.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1310000000000001e-06 2024-04-23T16:21:32.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.392288+0000", 2024-04-23T16:21:32.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0079999999999999e-06 2024-04-23T16:21:32.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.392296+0000", 2024-04-23T16:21:32.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1369999999999996e-06 2024-04-23T16:21:32.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.392300+0000", 2024-04-23T16:21:32.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5900000000000001e-06 2024-04-23T16:21:32.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.403117+0000", 2024-04-23T16:21:32.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010816999000000001 2024-04-23T16:21:32.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.403137+0000", 2024-04-23T16:21:32.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9964999999999999e-05 2024-04-23T16:21:32.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.438049+0000", 2024-04-23T16:21:32.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.034912382999999998 2024-04-23T16:21:32.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.438078+0000", 2024-04-23T16:21:32.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8971000000000001e-05 2024-04-23T16:21:32.440 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.440 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.440 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.440 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:300666 2.c e178/110 2:30b365c3:::benchmark_data_smithi188_77627_object131317:head v 178'18690, mlcod=178'18684)", 2024-04-23T16:21:32.440 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:18.396147+0000", 2024-04-23T16:21:32.440 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.864438348, 2024-04-23T16:21:32.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.041953772, 2024-04-23T16:21:32.440 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.441 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.441 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.441 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.396147+0000", 2024-04-23T16:21:32.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.396147+0000", 2024-04-23T16:21:32.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.396149+0000", 2024-04-23T16:21:32.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.736e-06 2024-04-23T16:21:32.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.396157+0000", 2024-04-23T16:21:32.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2330000000000006e-06 2024-04-23T16:21:32.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.396166+0000", 2024-04-23T16:21:32.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5219999999999996e-06 2024-04-23T16:21:32.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.396174+0000", 2024-04-23T16:21:32.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7919999999999993e-06 2024-04-23T16:21:32.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.406494+0000", 2024-04-23T16:21:32.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010320050000000001 2024-04-23T16:21:32.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.406517+0000", 2024-04-23T16:21:32.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2356000000000001e-05 2024-04-23T16:21:32.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.438080+0000", 2024-04-23T16:21:32.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.031563539000000002 2024-04-23T16:21:32.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.438101+0000", 2024-04-23T16:21:32.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0543999999999998e-05 2024-04-23T16:21:32.444 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.444 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.444 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.444 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:300668 2.4 e178/110 2:249d12d0:::benchmark_data_smithi188_77627_object131319:head v 178'18788, mlcod=178'18780)", 2024-04-23T16:21:32.444 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:18.398015+0000", 2024-04-23T16:21:32.444 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.862569892, 2024-04-23T16:21:32.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.040108903000000001, 2024-04-23T16:21:32.445 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.445 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.445 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.445 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.398015+0000", 2024-04-23T16:21:32.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.398015+0000", 2024-04-23T16:21:32.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.398018+0000", 2024-04-23T16:21:32.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0619999999999999e-06 2024-04-23T16:21:32.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.398025+0000", 2024-04-23T16:21:32.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9799999999999998e-06 2024-04-23T16:21:32.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.398035+0000", 2024-04-23T16:21:32.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4220000000000006e-06 2024-04-23T16:21:32.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.398041+0000", 2024-04-23T16:21:32.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.959e-06 2024-04-23T16:21:32.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.406481+0000", 2024-04-23T16:21:32.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084406099999999994 2024-04-23T16:21:32.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.406500+0000", 2024-04-23T16:21:32.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8546000000000002e-05 2024-04-23T16:21:32.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.438103+0000", 2024-04-23T16:21:32.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.031602466000000003 2024-04-23T16:21:32.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:18.438124+0000", 2024-04-23T16:21:32.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1858e-05 2024-04-23T16:21:32.448 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.448 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.448 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.449 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:316426 2.c e181/110 2:33585b44:::benchmark_data_smithi188_77627_object147077:head v 181'19695, mlcod=181'19691)", 2024-04-23T16:21:32.449 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:31.840196+0000", 2024-04-23T16:21:32.449 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.42038924799999999, 2024-04-23T16:21:32.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.050486218999999999, 2024-04-23T16:21:32.449 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.449 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.449 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.449 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.840196+0000", 2024-04-23T16:21:32.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.840196+0000", 2024-04-23T16:21:32.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.840198+0000", 2024-04-23T16:21:32.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9759999999999998e-06 2024-04-23T16:21:32.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.840206+0000", 2024-04-23T16:21:32.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1000000000000004e-06 2024-04-23T16:21:32.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.840218+0000", 2024-04-23T16:21:32.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1654999999999999e-05 2024-04-23T16:21:32.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.840226+0000", 2024-04-23T16:21:32.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1110000000000008e-06 2024-04-23T16:21:32.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.848505+0000", 2024-04-23T16:21:32.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0082791640000000003 2024-04-23T16:21:32.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.848528+0000", 2024-04-23T16:21:32.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2464999999999999e-05 2024-04-23T16:21:32.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.890631+0000", 2024-04-23T16:21:32.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.042103373999999999 2024-04-23T16:21:32.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.890682+0000", 2024-04-23T16:21:32.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1374e-05 2024-04-23T16:21:32.453 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.453 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.453 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.453 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:316417 2.4 e181/110 2:2cded356:::benchmark_data_smithi188_77627_object147068:head v 181'19746, mlcod=181'19742)", 2024-04-23T16:21:32.453 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:31.843966+0000", 2024-04-23T16:21:32.453 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.41661943200000001, 2024-04-23T16:21:32.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.050839795, 2024-04-23T16:21:32.453 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.453 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.453 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.454 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.843966+0000", 2024-04-23T16:21:32.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.843966+0000", 2024-04-23T16:21:32.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.843968+0000", 2024-04-23T16:21:32.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8029999999999999e-06 2024-04-23T16:21:32.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.843976+0000", 2024-04-23T16:21:32.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7190000000000007e-06 2024-04-23T16:21:32.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.843990+0000", 2024-04-23T16:21:32.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3264999999999999e-05 2024-04-23T16:21:32.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.844000+0000", 2024-04-23T16:21:32.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8600000000000005e-06 2024-04-23T16:21:32.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.847815+0000", 2024-04-23T16:21:32.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003815887 2024-04-23T16:21:32.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.847837+0000", 2024-04-23T16:21:32.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1877000000000001e-05 2024-04-23T16:21:32.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.894750+0000", 2024-04-23T16:21:32.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046912715000000001 2024-04-23T16:21:32.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.894806+0000", 2024-04-23T16:21:32.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5668999999999998e-05 2024-04-23T16:21:32.457 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.457 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.457 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.458 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:316422 2.4 e181/110 2:26ebaa5f:::benchmark_data_smithi188_77627_object147073:head v 181'19747, mlcod=181'19742)", 2024-04-23T16:21:32.458 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:31.850884+0000", 2024-04-23T16:21:32.458 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.40970108599999999, 2024-04-23T16:21:32.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.043954877000000003, 2024-04-23T16:21:32.458 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.458 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.458 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.458 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.850884+0000", 2024-04-23T16:21:32.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.850884+0000", 2024-04-23T16:21:32.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.850886+0000", 2024-04-23T16:21:32.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5e-06 2024-04-23T16:21:32.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.850894+0000", 2024-04-23T16:21:32.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5569999999999996e-06 2024-04-23T16:21:32.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.850909+0000", 2024-04-23T16:21:32.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4831000000000001e-05 2024-04-23T16:21:32.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.850917+0000", 2024-04-23T16:21:32.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2220000000000002e-06 2024-04-23T16:21:32.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.854034+0000", 2024-04-23T16:21:32.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003116625 2024-04-23T16:21:32.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.854056+0000", 2024-04-23T16:21:32.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1583999999999999e-05 2024-04-23T16:21:32.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.894810+0000", 2024-04-23T16:21:32.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.040754093999999998 2024-04-23T16:21:32.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:31.894839+0000", 2024-04-23T16:21:32.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9464000000000001e-05 2024-04-23T16:21:32.462 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.462 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.462 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.462 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:316670 2.4 e181/110 2:28c49686:::benchmark_data_smithi188_77627_object147321:head v 181'19764, mlcod=181'19761)", 2024-04-23T16:21:32.462 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:32.087009+0000", 2024-04-23T16:21:32.462 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.17357622, 2024-04-23T16:21:32.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.040693162999999997, 2024-04-23T16:21:32.462 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:32.463 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:32.463 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:21:32.463 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:32.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:32.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:32.087009+0000", 2024-04-23T16:21:32.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:32.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:32.087009+0000", 2024-04-23T16:21:32.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:32.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:32.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:32.087011+0000", 2024-04-23T16:21:32.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.731e-06 2024-04-23T16:21:32.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:32.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:32.087019+0000", 2024-04-23T16:21:32.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6890000000000004e-06 2024-04-23T16:21:32.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:32.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:32.087034+0000", 2024-04-23T16:21:32.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5899000000000001e-05 2024-04-23T16:21:32.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:32.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:32.087043+0000", 2024-04-23T16:21:32.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5799999999999992e-06 2024-04-23T16:21:32.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:32.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:32.092579+0000", 2024-04-23T16:21:32.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055356329999999999 2024-04-23T16:21:32.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:32.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:32.092602+0000", 2024-04-23T16:21:32.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3348000000000001e-05 2024-04-23T16:21:32.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:21:32.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:32.127672+0000", 2024-04-23T16:21:32.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.035070194999999998 2024-04-23T16:21:32.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:32.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:32.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:21:32.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:32.127702+0000", 2024-04-23T16:21:32.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0088000000000001e-05 2024-04-23T16:21:32.466 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.466 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.466 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.466 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:32.466 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:32.467 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:21:32.696 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:21:32.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:32.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:21:32.816 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:32.850 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:32.876 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:32.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:32.881 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:32.983 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:33.010 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:33.034 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:33.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:33.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:33.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:32 smithi103 ceph-mon[39629]: pgmap v540: 17 pgs: 17 active+clean; 1.5 GiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-23T16:21:33.137 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:21:33.172 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:33.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:32 smithi188 ceph-mon[44101]: pgmap v540: 17 pgs: 17 active+clean; 1.5 GiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-23T16:21:33.293 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:21:33.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:33.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:21:33.328 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-23T16:21:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:33.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:32 smithi148 ceph-mon[44520]: pgmap v540: 17 pgs: 17 active+clean; 1.5 GiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-23T16:21:33.397 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:21:33.499 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:34.480 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:21:34.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:34.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:21:34.586 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:34.623 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:34.648 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:34.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:34.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:34.752 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:21:34.780 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:34.805 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:21:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:34.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:21:34.909 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:21:34.967 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:34.979 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-23T16:21:35.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:34 smithi188 ceph-mon[44101]: 2.1 scrub starts 2024-04-23T16:21:35.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:34 smithi188 ceph-mon[44101]: 2.1 scrub ok 2024-04-23T16:21:35.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:34 smithi188 ceph-mon[44101]: pgmap v541: 17 pgs: 17 active+clean; 1.3 GiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-23T16:21:35.324 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:34 smithi103 ceph-mon[39629]: 2.1 scrub starts 2024-04-23T16:21:35.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:34 smithi103 ceph-mon[39629]: 2.1 scrub ok 2024-04-23T16:21:35.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:34 smithi103 ceph-mon[39629]: pgmap v541: 17 pgs: 17 active+clean; 1.3 GiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-23T16:21:35.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:34 smithi148 ceph-mon[44520]: 2.1 scrub starts 2024-04-23T16:21:35.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:34 smithi148 ceph-mon[44520]: 2.1 scrub ok 2024-04-23T16:21:35.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:34 smithi148 ceph-mon[44520]: pgmap v541: 17 pgs: 17 active+clean; 1.3 GiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-23T16:21:35.395 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:21:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:35.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:21:35.500 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:21:35.540 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:35.565 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:21:35.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:35.573 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:21:35.674 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:35.691 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:21:35.701 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:35.721 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:35.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:35.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:35.741 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:21:35.824 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:21:35.897 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:36.038 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:21:36.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:36.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:21:36.092 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:21:36.143 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:21:36.172 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:36.195 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:21:36.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:36.198 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:21:36.299 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:36.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1214749783' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:21:36.350 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:36.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1214749783' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:21:36.376 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:36.378 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1214749783' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:21:36.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:36.379 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:36.480 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:21:36.500 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:21:36.540 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:36.771 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:21:36.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:36.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:21:36.880 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:36.915 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:36.948 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:36.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:36.955 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:37.056 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:21:37.096 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:37.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:37 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:21:37.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:37 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1214749783' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:21:37.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:37 smithi103 ceph-mon[39629]: osdmap e182: 12 total, 12 up, 11 in 2024-04-23T16:21:37.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:37 smithi103 ceph-mon[39629]: pgmap v543: 17 pgs: 17 active+clean; 1.2 GiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:21:37.123 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:21:37.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:37.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:21:37.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:37 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:21:37.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:37 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1214749783' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:21:37.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:37 smithi188 ceph-mon[44101]: osdmap e182: 12 total, 12 up, 11 in 2024-04-23T16:21:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:37 smithi188 ceph-mon[44101]: pgmap v543: 17 pgs: 17 active+clean; 1.2 GiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:21:37.229 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:37.270 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:37.306 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:37.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:37.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:37.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:37 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:21:37.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:37 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1214749783' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:21:37.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:37 smithi148 ceph-mon[44520]: osdmap e182: 12 total, 12 up, 11 in 2024-04-23T16:21:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:37 smithi148 ceph-mon[44520]: pgmap v543: 17 pgs: 17 active+clean; 1.2 GiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:21:37.413 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:37.440 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:37.461 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:37.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:37.564 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:21:37.591 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:37.610 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:21:37.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:37.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:21:37.713 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:21:37.744 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:37.774 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:21:37.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:37.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:21:37.878 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:21:37.916 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:39.269 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:21:39.596 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:39 smithi103 ceph-mon[39629]: pgmap v544: 17 pgs: 17 active+clean; 1.1 GiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-23T16:21:39.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:39 smithi148 ceph-mon[44520]: pgmap v544: 17 pgs: 17 active+clean; 1.1 GiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-23T16:21:39.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:39 smithi188 ceph-mon[44101]: pgmap v544: 17 pgs: 17 active+clean; 1.1 GiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-23T16:21:40.839 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:21:40.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:40.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:21:40.946 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:21:40.999 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:41.043 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:21:41.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:41.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:21:41.156 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:41.193 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:41.219 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:41.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:41.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:41.323 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:41.356 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:41.404 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:41.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:41.410 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:41.511 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:21:41.539 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:41.566 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:21:41.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:41.573 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:21:41.674 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:21:41.702 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:41.726 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:21:41.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:41.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:21:41.751 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:21:41.832 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:41.835 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:21:41.835 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:21:41.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.835 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:328175 2.2 e182/110 2:45991ddd:::benchmark_data_smithi188_77627_object158826:head v 182'25773, mlcod=182'25770)", 2024-04-23T16:21:41.835 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:41.831204+0000", 2024-04-23T16:21:41.835 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.003339631, 2024-04-23T16:21:41.835 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0033712659999999999, 2024-04-23T16:21:41.836 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:41.836 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:41.836 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:21:41.836 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:41.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:41.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.831204+0000", 2024-04-23T16:21:41.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:41.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:41.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.831204+0000", 2024-04-23T16:21:41.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:41.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:41.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.831206+0000", 2024-04-23T16:21:41.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.221e-06 2024-04-23T16:21:41.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:41.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.831212+0000", 2024-04-23T16:21:41.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9869999999999996e-06 2024-04-23T16:21:41.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:41.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.831219+0000", 2024-04-23T16:21:41.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5689999999999999e-06 2024-04-23T16:21:41.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:41.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.831224+0000", 2024-04-23T16:21:41.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0220000000000004e-06 2024-04-23T16:21:41.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:41.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.831275+0000", 2024-04-23T16:21:41.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0482000000000002e-05 2024-04-23T16:21:41.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:41.839 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.831296+0000", 2024-04-23T16:21:41.839 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1586e-05 2024-04-23T16:21:41.839 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:41.839 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:41.839 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:41.839 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.839 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.839 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:328171 2.4 2:25f4ed76:::benchmark_data_smithi188_77627_object158822:head [delete] snapc 0=[] ondisk+write+known_if_redirected e182)", 2024-04-23T16:21:41.839 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:41.834182+0000", 2024-04-23T16:21:41.839 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.00036191400000000002, 2024-04-23T16:21:41.839 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00046257299999999998, 2024-04-23T16:21:41.839 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:41.840 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:41.840 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:21:41.840 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:21:41.840 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:21:41.840 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:21:41.840 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 328171 2024-04-23T16:21:41.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.840 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:41.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:41.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.834182+0000", 2024-04-23T16:21:41.840 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:41.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:41.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.834182+0000", 2024-04-23T16:21:41.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:41.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:41.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.834184+0000", 2024-04-23T16:21:41.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1909999999999999e-06 2024-04-23T16:21:41.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:41.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.834192+0000", 2024-04-23T16:21:41.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4880000000000001e-06 2024-04-23T16:21:41.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:41.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.834206+0000", 2024-04-23T16:21:41.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4e-05 2024-04-23T16:21:41.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:41.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.834215+0000", 2024-04-23T16:21:41.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4369999999999996e-06 2024-04-23T16:21:41.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:21:41.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.834271+0000", 2024-04-23T16:21:41.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6119999999999998e-05 2024-04-23T16:21:41.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:21:41.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.834363+0000", 2024-04-23T16:21:41.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1809000000000001e-05 2024-04-23T16:21:41.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.843 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-23T16:21:41.843 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.834483+0000", 2024-04-23T16:21:41.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012002699999999999 2024-04-23T16:21:41.843 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:41.843 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:41.843 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:41.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.843 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:328174 2.4 2.6aece8e4 (undecoded) ondisk+write+known_if_redirected e182)", 2024-04-23T16:21:41.843 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:41.834241+0000", 2024-04-23T16:21:41.843 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.00030336399999999998, 2024-04-23T16:21:41.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047731800000000001, 2024-04-23T16:21:41.844 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:21:41.844 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:21:41.844 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "queued for pg", 2024-04-23T16:21:41.844 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:21:41.844 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:21:41.844 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:21:41.844 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 328174 2024-04-23T16:21:41.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.844 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:21:41.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.844 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:21:41.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.834241+0000", 2024-04-23T16:21:41.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:41.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.844 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:21:41.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.834241+0000", 2024-04-23T16:21:41.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:21:41.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:21:41.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.834242+0000", 2024-04-23T16:21:41.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7900000000000007e-07 2024-04-23T16:21:41.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:21:41.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.834245+0000", 2024-04-23T16:21:41.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1480000000000002e-06 2024-04-23T16:21:41.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:21:41.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.834252+0000", 2024-04-23T16:21:41.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1199999999999996e-06 2024-04-23T16:21:41.846 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:21:41.846 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:21:41.846 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:21:41.846 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:41.834257+0000", 2024-04-23T16:21:41.846 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2639999999999999e-06 2024-04-23T16:21:41.846 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:41.846 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:21:41.846 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:41.846 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:21:41.846 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:21:41.846 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 3 2024-04-23T16:21:41.846 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:21:41.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:41 smithi103 ceph-mon[39629]: pgmap v545: 17 pgs: 17 active+clean; 955 MiB data, 2.8 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:21:41.861 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:41 smithi188 ceph-mon[44101]: pgmap v545: 17 pgs: 17 active+clean; 955 MiB data, 2.8 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:21:41.863 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:41.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:41 smithi148 ceph-mon[44520]: pgmap v545: 17 pgs: 17 active+clean; 955 MiB data, 2.8 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:21:41.884 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:41.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:41.887 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:41.988 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:21:42.020 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:42.048 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:21:42.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:42.056 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:21:42.157 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:21:42.188 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:42.211 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:21:42.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:42.216 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:21:42.317 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:21:42.425 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:42.447 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:21:42.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:42.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:21:42.551 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:21:42.592 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:42.658 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:21:42.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:42.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:21:42.762 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:21:42.822 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:43.206 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:21:43.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-23T16:21:43.595 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:21:43.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:43.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:21:43.649 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.3 to 1 (10000) 2024-04-23T16:21:43.703 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:21:43.731 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:43.754 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:21:43.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:43.758 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:21:43.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:43 smithi103 ceph-mon[39629]: pgmap v546: 17 pgs: 17 active+clean; 838 MiB data, 2.6 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:21:43.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:43 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1420257200' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:21:43.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:43 smithi103 ceph-mon[39629]: from='client.35009 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:21:43.858 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:21:43.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:43 smithi148 ceph-mon[44520]: pgmap v546: 17 pgs: 17 active+clean; 838 MiB data, 2.6 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:21:43.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:43 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1420257200' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:21:43.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:43 smithi148 ceph-mon[44520]: from='client.35009 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:21:43.887 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:43.908 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:21:43.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:43.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:21:43.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:43 smithi188 ceph-mon[44101]: pgmap v546: 17 pgs: 17 active+clean; 838 MiB data, 2.6 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:21:43.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:43 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1420257200' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:21:43.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:43 smithi188 ceph-mon[44101]: from='client.35009 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:21:44.012 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:21:44.070 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:44.092 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:21:44.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:44.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:21:44.201 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:21:44.231 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:44.914 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:44 smithi103 ceph-mon[39629]: from='client.35009 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-23T16:21:44.914 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:44 smithi103 ceph-mon[39629]: osdmap e183: 12 total, 12 up, 11 in 2024-04-23T16:21:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:44 smithi188 ceph-mon[44101]: from='client.35009 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-23T16:21:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:44 smithi188 ceph-mon[44101]: osdmap e183: 12 total, 12 up, 11 in 2024-04-23T16:21:45.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:44 smithi148 ceph-mon[44520]: from='client.35009 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-23T16:21:45.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:44 smithi148 ceph-mon[44520]: osdmap e183: 12 total, 12 up, 11 in 2024-04-23T16:21:45.324 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:21:45.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:45.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:21:45.434 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:21:45.462 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:45.483 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:21:45.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:45.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:21:45.588 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:21:45.634 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:45.671 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:21:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:45.680 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:21:45.781 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:45.807 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:45 smithi148 ceph-mon[44520]: pgmap v548: 17 pgs: 17 active+clean; 635 MiB data, 2.2 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-23T16:21:45.810 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:45.830 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:45.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:45.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:45.886 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-23T16:21:45.934 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:45.937 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:45 smithi103 ceph-mon[39629]: pgmap v548: 17 pgs: 17 active+clean; 635 MiB data, 2.2 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-23T16:21:45.966 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:45 smithi188 ceph-mon[44101]: pgmap v548: 17 pgs: 17 active+clean; 635 MiB data, 2.2 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-23T16:21:45.969 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:45.992 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:45.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:46.096 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:21:46.165 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:46.457 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:21:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:46.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:21:46.562 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:46.591 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:46.613 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:46.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:46.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:46.718 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:21:46.768 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:47.005 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:21:47.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:47.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:21:47.113 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:47.140 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:47.161 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:47.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:47.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:47.269 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:21:47.299 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:47.318 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:21:47.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:47.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:21:47.423 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:21:47.458 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:47.482 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:21:47.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:47.487 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:21:47.588 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:21:47.616 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:47.635 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:21:47.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:47.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:21:47.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:47 smithi188 ceph-mon[44101]: pgmap v549: 17 pgs: 17 active+clean; 471 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:21:47.740 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:21:47.783 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:47.813 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:21:47.818 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:47 smithi103 ceph-mon[39629]: pgmap v549: 17 pgs: 17 active+clean; 471 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:21:47.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:47.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:21:47.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:47 smithi148 ceph-mon[44520]: pgmap v549: 17 pgs: 17 active+clean; 471 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:21:47.920 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:21:47.927 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:21:47.929 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:21:47.951 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:47.975 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:21:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:47.979 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:21:48.022 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:21:48.066 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:21:48.080 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:48.110 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:48.123 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:21:48.131 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:48.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:48.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:48.153 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:21:48.182 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:21:48.223 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:21:48.234 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:48.258 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:21:48.263 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:48.284 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:48.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:48.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:48.300 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:21:48.330 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:21:48.361 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:21:48.388 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:21:48.389 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:21:48.417 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:48.432 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:21:48.439 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:21:48.440 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:21:48.440 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:21:48.441 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:21:48.441 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:21:48.441 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:21:48.441 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:21:48.441 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:21:48.441 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:21:48.441 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:21:48.441 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:21:48.441 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:21:48.441 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:21:48.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:48.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:21:48.545 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:21:48.572 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:48.594 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:21:48.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:48.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:21:48.657 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:21:48.697 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:48.722 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:48 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2996416097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:21:48.722 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:48 smithi148 ceph-mon[44520]: from='client.24964 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:21:48.722 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:48 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2210384689' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:21:48.722 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:48.722 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:48.722 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:48.722 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:48.722 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:48.723 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:48.748 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:48.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:48.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:48.852 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:48.887 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:48.912 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:48 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2996416097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:21:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:48 smithi188 ceph-mon[44101]: from='client.24964 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:21:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:48 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2210384689' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:21:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:48.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:48.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:48.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:48.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:48.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:48.916 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:48.956 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:48 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2996416097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:21:48.957 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:48 smithi103 ceph-mon[39629]: from='client.24964 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:21:48.957 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:48 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2210384689' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:21:48.957 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:48.957 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:48.957 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:48.957 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:48.957 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:49.016 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:49.046 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:49.076 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:49.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:49.079 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:49.180 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:21:49.222 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:49.248 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:21:49.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:49.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:21:49.353 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:21:49.381 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:49.404 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:21:49.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:49.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:21:49.508 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:21:49.543 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:49.574 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:21:49.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:49.581 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:21:49.657 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:49.657 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:49.657 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:49.657 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:49.657 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:49.657 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:49.657 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:49.657 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:49 smithi148 ceph-mon[44520]: pgmap v550: 17 pgs: 17 active+clean; 355 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:21:49.657 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:49 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:21:49.657 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:49 smithi148 ceph-mon[44520]: from='client.24964 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:21:49.657 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:49 smithi148 ceph-mon[44520]: osdmap e184: 12 total, 12 up, 11 in 2024-04-23T16:21:49.657 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:49 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:21:49.657 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:49 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:21:49.682 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:21:49.712 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:49.741 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:21:49.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:49.749 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:21:49.850 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:21:49.877 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:49.897 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:21:49.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:49.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:21:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:49 smithi188 ceph-mon[44101]: pgmap v550: 17 pgs: 17 active+clean; 355 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:21:49.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:49 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:21:49.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:49 smithi188 ceph-mon[44101]: from='client.24964 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:21:49.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:49 smithi188 ceph-mon[44101]: osdmap e184: 12 total, 12 up, 11 in 2024-04-23T16:21:49.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:49 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:21:49.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:49 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:21:50.001 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:21:50.038 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:50.072 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:21:50.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:50.079 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:21:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:21:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:49 smithi103 ceph-mon[39629]: pgmap v550: 17 pgs: 17 active+clean; 355 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:21:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:49 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:21:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:49 smithi103 ceph-mon[39629]: from='client.24964 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:21:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:49 smithi103 ceph-mon[39629]: osdmap e184: 12 total, 12 up, 11 in 2024-04-23T16:21:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:49 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:21:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:49 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:21:50.180 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:50.209 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:50.233 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:50.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:50.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:50.338 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:21:50.375 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:50.403 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:21:50.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:50.408 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:21:50.508 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:21:50.585 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:50.796 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Removed 169346 objects 2024-04-23T16:21:50.796 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Clean up completed and total clean up time :148.967 2024-04-23T16:21:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:50 smithi103 ceph-mon[39629]: osdmap e185: 12 total, 12 up, 11 in 2024-04-23T16:21:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:50 smithi103 ceph-mon[39629]: pgmap v553: 17 pgs: 17 active+clean; 221 MiB data, 1.5 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:21:51.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:50 smithi188 ceph-mon[44101]: osdmap e185: 12 total, 12 up, 11 in 2024-04-23T16:21:51.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:50 smithi188 ceph-mon[44101]: pgmap v553: 17 pgs: 17 active+clean; 221 MiB data, 1.5 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:21:51.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:50 smithi148 ceph-mon[44520]: osdmap e185: 12 total, 12 up, 11 in 2024-04-23T16:21:51.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:50 smithi148 ceph-mon[44520]: pgmap v553: 17 pgs: 17 active+clean; 221 MiB data, 1.5 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:21:52.504 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:21:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:52 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:21:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:52 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:21:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:52 smithi188 ceph-mon[44101]: osdmap e186: 12 total, 12 up, 11 in 2024-04-23T16:21:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:52 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:21:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:52 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:21:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:52 smithi188 ceph-mon[44101]: pgmap v555: 17 pgs: 17 active+clean; 114 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 949 op/s 2024-04-23T16:21:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:52 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:21:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:52 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:21:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:52 smithi103 ceph-mon[39629]: osdmap e186: 12 total, 12 up, 11 in 2024-04-23T16:21:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:52 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:21:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:52 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:21:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:52 smithi103 ceph-mon[39629]: pgmap v555: 17 pgs: 17 active+clean; 114 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 949 op/s 2024-04-23T16:21:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:52 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:21:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:52 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:21:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:52 smithi148 ceph-mon[44520]: osdmap e186: 12 total, 12 up, 11 in 2024-04-23T16:21:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:52 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:21:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:52 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:21:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:52 smithi148 ceph-mon[44520]: pgmap v555: 17 pgs: 17 active+clean; 114 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 949 op/s 2024-04-23T16:21:53.869 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:21:53.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:53.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:21:53.973 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:21:53.998 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:54.016 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:21:54.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:54.020 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:21:54.121 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:21:54.145 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:53 smithi103 ceph-mon[39629]: osdmap e187: 12 total, 12 up, 11 in 2024-04-23T16:21:54.145 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:21:54.145 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:21:54.146 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:21:54.146 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:21:54.147 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:54.169 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:21:54.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:54.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:21:54.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:53 smithi188 ceph-mon[44101]: osdmap e187: 12 total, 12 up, 11 in 2024-04-23T16:21:54.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:53 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:21:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:53 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:21:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:53 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:21:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:53 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:21:54.273 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:21:54.301 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:54.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:53 smithi148 ceph-mon[44520]: osdmap e187: 12 total, 12 up, 11 in 2024-04-23T16:21:54.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:53 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:21:54.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:53 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:21:54.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:53 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:21:54.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:53 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:21:54.424 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:21:54.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:54.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:21:54.528 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:21:54.555 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:54.573 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:21:54.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:54.576 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:21:54.665 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-04-23T16:21:54.665 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-04-23T16:21:54.677 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:21:54.734 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:54.759 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:21:54.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:54.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:21:54.774 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:21:54.774 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:21:54.863 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:21:54.893 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:54.914 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.4 to 1 (10000) 2024-04-23T16:21:54.916 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:21:54.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:54.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:21:55.020 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:21:55.052 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:55.073 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:21:55.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:55.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:21:55.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:54 smithi103 ceph-mon[39629]: pgmap v557: 17 pgs: 17 active+clean; 11 MiB data, 1.0 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 933 op/s 2024-04-23T16:21:55.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:54 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1324427332' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-23T16:21:55.177 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:21:55.204 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:55.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:54 smithi188 ceph-mon[44101]: pgmap v557: 17 pgs: 17 active+clean; 11 MiB data, 1.0 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 933 op/s 2024-04-23T16:21:55.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:54 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1324427332' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-23T16:21:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:54 smithi148 ceph-mon[44520]: pgmap v557: 17 pgs: 17 active+clean; 11 MiB data, 1.0 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 933 op/s 2024-04-23T16:21:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:54 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1324427332' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-23T16:21:55.520 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:21:55.904 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-23T16:21:55.957 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:21:55.961 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:21:55.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:55.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:21:55.975 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:21:55.992 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:21:56.072 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:21:56.099 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:56.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:55 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1324427332' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-23T16:21:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:55 smithi188 ceph-mon[44101]: osdmap e188: 12 total, 12 up, 11 in 2024-04-23T16:21:56.235 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:55 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1324427332' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-23T16:21:56.235 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:55 smithi103 ceph-mon[39629]: osdmap e188: 12 total, 12 up, 11 in 2024-04-23T16:21:56.247 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:21:56.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:56.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:21:56.352 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:21:56.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:55 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1324427332' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-23T16:21:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:55 smithi148 ceph-mon[44520]: osdmap e188: 12 total, 12 up, 11 in 2024-04-23T16:21:56.378 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:57.127 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:21:57.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:57.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:21:57.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:56 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:21:57.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:56 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:21:57.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:56 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:21:57.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:56 smithi188 ceph-mon[44101]: pgmap v559: 17 pgs: 17 active+clean; 577 KiB data, 993 MiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 588 op/s 2024-04-23T16:21:57.236 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:21:57.267 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:56 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:21:57.267 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:56 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:21:57.267 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:56 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:21:57.267 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:56 smithi148 ceph-mon[44520]: pgmap v559: 17 pgs: 17 active+clean; 577 KiB data, 993 MiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 588 op/s 2024-04-23T16:21:57.269 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:57.290 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:56 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:21:57.290 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:56 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:21:57.290 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:56 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:21:57.290 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:56 smithi103 ceph-mon[39629]: pgmap v559: 17 pgs: 17 active+clean; 577 KiB data, 993 MiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 588 op/s 2024-04-23T16:21:57.291 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:21:57.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:57.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:21:57.395 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:21:57.427 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:57.448 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:21:57.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:57.451 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:21:57.552 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:21:57.580 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:57.702 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:21:57.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:57.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:21:57.806 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:21:57.833 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:58.030 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:21:58.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:58.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:21:58.134 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:21:58.166 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:58.186 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:21:58.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:58.190 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:21:58.290 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:21:58.317 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:58.543 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-23T16:21:58.743 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:21:58.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:58.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:21:58.847 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:21:58.875 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:59.525 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:21:59.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:59.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:21:59.629 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:21:59.656 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:59.675 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:21:59.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:59.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:21:59.779 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:21:59.811 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:59.832 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:21:59.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:21:59.835 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:21:59.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:21:59 smithi103 ceph-mon[39629]: pgmap v560: 17 pgs: 17 active+clean; 577 KiB data, 993 MiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 273 op/s 2024-04-23T16:21:59.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:21:59 smithi148 ceph-mon[44520]: pgmap v560: 17 pgs: 17 active+clean; 577 KiB data, 993 MiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 273 op/s 2024-04-23T16:21:59.936 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:21:59.965 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:21:59.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:21:59 smithi188 ceph-mon[44101]: pgmap v560: 17 pgs: 17 active+clean; 577 KiB data, 993 MiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 273 op/s 2024-04-23T16:22:00.316 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:22:00.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:00.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:22:00.419 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:00.447 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:00.484 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:00.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:00.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:00.593 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:22:00.609 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'unique_pool_0' removed 2024-04-23T16:22:00.627 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:00.648 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:22:00.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:00.652 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:22:00.752 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:22:00.779 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:00.798 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:22:00.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:00.801 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:22:00.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:00 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3346016253' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:22:00.902 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:22:00.927 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:00 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3346016253' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:22:00.932 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:00.952 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:22:00.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:00.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:22:00.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:00 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3346016253' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:22:01.056 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:22:01.083 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:01.101 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:22:01.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:01.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:22:01.186 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:22:01.205 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:22:01.205 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:22:01.225 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:22:01.233 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:01.247 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:22:01.268 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:22:01.288 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:22:01.308 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:22:01.328 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:22:01.347 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:22:01.383 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:22:01.412 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:22:01.434 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:22:01.436 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:22:01.436 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:22:01.436 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:22:01.436 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:22:01.436 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:22:01.436 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:22:01.436 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:22:01.436 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:22:01.436 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:22:01.436 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:22:01.437 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:22:01.437 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:22:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:01 smithi148 ceph-mon[44520]: pgmap v561: 17 pgs: 17 active+clean; 577 KiB data, 993 MiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 227 op/s 2024-04-23T16:22:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:01 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3346016253' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:22:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:01 smithi148 ceph-mon[44520]: osdmap e189: 12 total, 12 up, 11 in 2024-04-23T16:22:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:01 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1605900053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:22:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:01.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:01 smithi188 ceph-mon[44101]: pgmap v561: 17 pgs: 17 active+clean; 577 KiB data, 993 MiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 227 op/s 2024-04-23T16:22:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:01 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3346016253' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:22:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:01 smithi188 ceph-mon[44101]: osdmap e189: 12 total, 12 up, 11 in 2024-04-23T16:22:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:01 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1605900053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:22:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:01.981 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:22:01.983 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:01 smithi103 ceph-mon[39629]: pgmap v561: 17 pgs: 17 active+clean; 577 KiB data, 993 MiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 227 op/s 2024-04-23T16:22:01.983 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:01 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3346016253' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:22:01.983 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:01 smithi103 ceph-mon[39629]: osdmap e189: 12 total, 12 up, 11 in 2024-04-23T16:22:01.983 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:01 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1605900053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:22:01.983 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:01.983 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:01.983 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:01.984 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:01.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:01.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:22:02.085 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:22:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:02.451 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-23T16:22:02.452 INFO:tasks.radosbench:Beginning radosbench... 2024-04-23T16:22:02.452 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-04-23T16:22:02.452 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-04-23T16:22:02.617 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:22:02.645 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:22:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:02 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4009664570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:22:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:02 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4009664570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:22:02.961 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:22:02.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:02.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:22:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:02 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4009664570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:22:03.065 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:22:03.097 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:03.119 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:22:03.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:03.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:22:03.222 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:22:03.254 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:03.274 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:22:03.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:03.278 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:22:03.378 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:22:03.411 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:03 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:22:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:03 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:22:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:03 smithi103 ceph-mon[39629]: pgmap v563: 1 pgs: 1 active+clean; 577 KiB data, 993 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:03 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:22:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:03 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4009664570' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:22:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:03 smithi103 ceph-mon[39629]: osdmap e190: 12 total, 12 up, 11 in 2024-04-23T16:22:03.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:03 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:22:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:03 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:22:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:03 smithi148 ceph-mon[44520]: pgmap v563: 1 pgs: 1 active+clean; 577 KiB data, 993 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:03 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:22:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:03 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4009664570' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:22:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:03 smithi148 ceph-mon[44520]: osdmap e190: 12 total, 12 up, 11 in 2024-04-23T16:22:03.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:03 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:22:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:03 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:22:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:03 smithi188 ceph-mon[44101]: pgmap v563: 1 pgs: 1 active+clean; 577 KiB data, 993 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:03 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:22:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:03 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4009664570' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:22:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:03 smithi188 ceph-mon[44101]: osdmap e190: 12 total, 12 up, 11 in 2024-04-23T16:22:04.505 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:22:04.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:04.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:22:04.512 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:22:04.527 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:22:04.611 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:22:04.640 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:04.692 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:22:04.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:04.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:22:04.809 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:22:04.836 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:05.020 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:04 smithi103 ceph-mon[39629]: pgmap v565: 1 pgs: 1 active+clean; 577 KiB data, 1011 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:05.033 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:22:05.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:05.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:22:05.137 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:05.163 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:05.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:04 smithi188 ceph-mon[44101]: pgmap v565: 1 pgs: 1 active+clean; 577 KiB data, 1011 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:05.262 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:05.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:05.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:05.366 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:22:05.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:04 smithi148 ceph-mon[44520]: pgmap v565: 1 pgs: 1 active+clean; 577 KiB data, 1011 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:05.398 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:05.419 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:22:05.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:05.422 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:22:05.523 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:05.551 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:06.218 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:06.220 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:05 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:22:06.220 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:05 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:22:06.220 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:05 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:22:06.221 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:05 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:22:06.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:06.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:06.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:05 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:22:06.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:05 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:22:06.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:05 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:22:06.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:05 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:22:06.322 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:22:06.354 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:06.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:05 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:22:06.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:05 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:22:06.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:05 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:22:06.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:05 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:22:06.375 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:22:06.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:06.378 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:22:06.479 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:22:06.493 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:22:06.509 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:06.528 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:22:06.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:06.531 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:22:06.632 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:22:06.659 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:07.046 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:22:07.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:06 smithi188 ceph-mon[44101]: osdmap e191: 12 total, 12 up, 11 in 2024-04-23T16:22:07.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:06 smithi188 ceph-mon[44101]: pgmap v567: 1 pgs: 1 active+clean; 577 KiB data, 1015 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:07.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:06 smithi103 ceph-mon[39629]: osdmap e191: 12 total, 12 up, 11 in 2024-04-23T16:22:07.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:06 smithi103 ceph-mon[39629]: pgmap v567: 1 pgs: 1 active+clean; 577 KiB data, 1015 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:07.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:06 smithi148 ceph-mon[44520]: osdmap e191: 12 total, 12 up, 11 in 2024-04-23T16:22:07.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:06 smithi148 ceph-mon[44520]: pgmap v567: 1 pgs: 1 active+clean; 577 KiB data, 1015 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:08.216 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:22:08.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:08.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:22:08.320 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:22:08.352 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:08.374 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:22:08.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:08.378 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:22:08.479 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:22:08.510 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:08.531 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:22:08.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:08.535 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:22:08.636 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:22:08.665 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:08.685 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:22:08.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:08.688 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:22:08.789 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:22:08.817 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:08.836 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:22:08.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:08.839 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:22:08.940 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:08.967 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:09.066 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:09.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:09.170 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:22:09.202 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:09.223 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:22:09.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:09.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:22:09.328 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:09.358 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:09.506 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:09.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:09.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:09.609 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:09.635 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:09 smithi103 ceph-mon[39629]: pgmap v568: 1 pgs: 1 active+clean; 577 KiB data, 1019 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:09.637 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:09.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:09 smithi148 ceph-mon[44520]: pgmap v568: 1 pgs: 1 active+clean; 577 KiB data, 1019 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:09.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:09 smithi188 ceph-mon[44101]: pgmap v568: 1 pgs: 1 active+clean; 577 KiB data, 1019 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:10.188 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:10.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:10.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:10.291 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:22:10.291 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:22:10.291 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:22:10.291 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:22:10.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.292 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:160157 2.2 e141/110 2:4b70dc78:::benchmark_data_smithi188_77627_object160156:head v 141'11332, mlcod=141'11316)", 2024-04-23T16:22:10.292 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:11.557755+0000", 2024-04-23T16:22:10.292 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 178.339147435, 2024-04-23T16:22:10.292 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13620769899999999, 2024-04-23T16:22:10.292 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.292 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.292 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.292 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.292 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.557755+0000", 2024-04-23T16:22:10.292 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.557755+0000", 2024-04-23T16:22:10.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.293 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.557756+0000", 2024-04-23T16:22:10.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.376e-06 2024-04-23T16:22:10.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.293 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.557824+0000", 2024-04-23T16:22:10.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7821e-05 2024-04-23T16:22:10.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.293 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.557832+0000", 2024-04-23T16:22:10.294 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0129999999999993e-06 2024-04-23T16:22:10.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.294 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.294 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.294 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.557838+0000", 2024-04-23T16:22:10.294 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1510000000000001e-06 2024-04-23T16:22:10.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.294 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.294 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.294 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.651682+0000", 2024-04-23T16:22:10.294 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.093843831000000003 2024-04-23T16:22:10.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.294 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.295 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.295 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.651700+0000", 2024-04-23T16:22:10.295 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8456000000000001e-05 2024-04-23T16:22:10.295 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.295 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.295 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.295 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.693922+0000", 2024-04-23T16:22:10.295 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.042221375999999998 2024-04-23T16:22:10.295 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.295 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.295 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.295 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.693963+0000", 2024-04-23T16:22:10.295 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0674999999999999e-05 2024-04-23T16:22:10.295 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.295 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.296 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.296 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.296 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.296 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:160168 2.2 e141/110 2:4dd38e7d:::benchmark_data_smithi188_77627_object160167:head v 141'11333, mlcod=141'11318)", 2024-04-23T16:22:10.296 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:11.567253+0000", 2024-04-23T16:22:10.296 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 178.32964912599999, 2024-04-23T16:22:10.296 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13013219200000001, 2024-04-23T16:22:10.296 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.296 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.296 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.296 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.296 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.296 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.296 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.567253+0000", 2024-04-23T16:22:10.296 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.297 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.297 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.297 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.567253+0000", 2024-04-23T16:22:10.297 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.297 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.297 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.297 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.567260+0000", 2024-04-23T16:22:10.297 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4910000000000002e-06 2024-04-23T16:22:10.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.297 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.297 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.297 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.567325+0000", 2024-04-23T16:22:10.297 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5086999999999995e-05 2024-04-23T16:22:10.298 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.298 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.567334+0000", 2024-04-23T16:22:10.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4539999999999996e-06 2024-04-23T16:22:10.298 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.298 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.567341+0000", 2024-04-23T16:22:10.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1010000000000003e-06 2024-04-23T16:22:10.298 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.298 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.655762+0000", 2024-04-23T16:22:10.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.088420550000000001 2024-04-23T16:22:10.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.299 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.299 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.299 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.655776+0000", 2024-04-23T16:22:10.299 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4525999999999999e-05 2024-04-23T16:22:10.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.299 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.299 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.299 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.697338+0000", 2024-04-23T16:22:10.299 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.041562146000000001 2024-04-23T16:22:10.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.299 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.299 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.299 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:11.697385+0000", 2024-04-23T16:22:10.300 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6836999999999999e-05 2024-04-23T16:22:10.300 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.300 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.300 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.300 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.300 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.300 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:161689 2.2 e141/110 2:4278a834:::benchmark_data_smithi188_77627_object161688:head v 141'11624, mlcod=141'11599)", 2024-04-23T16:22:10.300 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:13.071501+0000", 2024-04-23T16:22:10.300 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 176.82540112999999, 2024-04-23T16:22:10.300 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.12783017599999999, 2024-04-23T16:22:10.300 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.300 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.300 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.300 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.300 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.300 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.301 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.071501+0000", 2024-04-23T16:22:10.301 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.301 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.301 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.301 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.301 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.071501+0000", 2024-04-23T16:22:10.301 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.301 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.301 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.301 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.301 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.071501+0000", 2024-04-23T16:22:10.301 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.65e-07 2024-04-23T16:22:10.301 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.301 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.301 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.301 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.071529+0000", 2024-04-23T16:22:10.301 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7830999999999998e-05 2024-04-23T16:22:10.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.302 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.071534+0000", 2024-04-23T16:22:10.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6450000000000004e-06 2024-04-23T16:22:10.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.302 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.071538+0000", 2024-04-23T16:22:10.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9509999999999999e-06 2024-04-23T16:22:10.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.302 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.177884+0000", 2024-04-23T16:22:10.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.106346231 2024-04-23T16:22:10.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.177901+0000", 2024-04-23T16:22:10.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6779e-05 2024-04-23T16:22:10.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.199294+0000", 2024-04-23T16:22:10.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.021392887999999999 2024-04-23T16:22:10.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:13.199331+0000", 2024-04-23T16:22:10.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7486e-05 2024-04-23T16:22:10.303 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.304 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.304 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.304 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:163500 2.2 e141/110 2:466f4e27:::benchmark_data_smithi188_77627_object163499:head v 141'11994, mlcod=141'11975)", 2024-04-23T16:22:10.304 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:15.135608+0000", 2024-04-23T16:22:10.304 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 174.761293833, 2024-04-23T16:22:10.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13519800700000001, 2024-04-23T16:22:10.304 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.304 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.304 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.304 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.135608+0000", 2024-04-23T16:22:10.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.305 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.135608+0000", 2024-04-23T16:22:10.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.305 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.135609+0000", 2024-04-23T16:22:10.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5200000000000002e-07 2024-04-23T16:22:10.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.305 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.135636+0000", 2024-04-23T16:22:10.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7032000000000001e-05 2024-04-23T16:22:10.306 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.306 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.306 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.306 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.135641+0000", 2024-04-23T16:22:10.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7199999999999997e-06 2024-04-23T16:22:10.306 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.306 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.306 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.306 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.135645+0000", 2024-04-23T16:22:10.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0559999999999998e-06 2024-04-23T16:22:10.306 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.306 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.306 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.306 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.193665+0000", 2024-04-23T16:22:10.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.058020229999999999 2024-04-23T16:22:10.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.193678+0000", 2024-04-23T16:22:10.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3387000000000001e-05 2024-04-23T16:22:10.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.270784+0000", 2024-04-23T16:22:10.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.077105514 2024-04-23T16:22:10.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.270806+0000", 2024-04-23T16:22:10.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2616000000000002e-05 2024-04-23T16:22:10.307 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.308 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.308 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.308 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:163502 2.2 e141/110 2:4cb38892:::benchmark_data_smithi188_77627_object163501:head v 141'11995, mlcod=141'11975)", 2024-04-23T16:22:10.308 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:15.135650+0000", 2024-04-23T16:22:10.308 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 174.761252015, 2024-04-23T16:22:10.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13517675500000001, 2024-04-23T16:22:10.308 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.308 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.308 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.308 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.135650+0000", 2024-04-23T16:22:10.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.309 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.135650+0000", 2024-04-23T16:22:10.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.309 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.135651+0000", 2024-04-23T16:22:10.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2599999999999998e-07 2024-04-23T16:22:10.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.309 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.135677+0000", 2024-04-23T16:22:10.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6129e-05 2024-04-23T16:22:10.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.310 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.135681+0000", 2024-04-23T16:22:10.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5879999999999999e-06 2024-04-23T16:22:10.310 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.310 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.310 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.135685+0000", 2024-04-23T16:22:10.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1899999999999997e-06 2024-04-23T16:22:10.310 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.310 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.310 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.197312+0000", 2024-04-23T16:22:10.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.061626945000000002 2024-04-23T16:22:10.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.311 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.311 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.197324+0000", 2024-04-23T16:22:10.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1728e-05 2024-04-23T16:22:10.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.311 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.311 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.270810+0000", 2024-04-23T16:22:10.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.073485889999999998 2024-04-23T16:22:10.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.311 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.311 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.270827+0000", 2024-04-23T16:22:10.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6959000000000001e-05 2024-04-23T16:22:10.311 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.312 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.312 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.312 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.312 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:163511 2.2 e141/110 2:492f17a2:::benchmark_data_smithi188_77627_object163510:head v 141'11996, mlcod=141'11978)", 2024-04-23T16:22:10.312 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:15.137552+0000", 2024-04-23T16:22:10.312 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 174.75935026799999, 2024-04-23T16:22:10.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13329332799999999, 2024-04-23T16:22:10.312 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.312 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.312 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.312 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.312 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.312 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.312 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.137552+0000", 2024-04-23T16:22:10.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.313 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.313 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.313 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.137552+0000", 2024-04-23T16:22:10.313 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.313 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.313 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.313 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.313 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.137553+0000", 2024-04-23T16:22:10.313 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2799999999999995e-07 2024-04-23T16:22:10.313 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.313 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.313 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.313 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.137584+0000", 2024-04-23T16:22:10.313 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1430000000000002e-05 2024-04-23T16:22:10.313 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.314 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.314 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.137590+0000", 2024-04-23T16:22:10.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8839999999999999e-06 2024-04-23T16:22:10.314 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.314 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.314 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.137595+0000", 2024-04-23T16:22:10.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1529999999999996e-06 2024-04-23T16:22:10.314 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.314 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.314 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.201992+0000", 2024-04-23T16:22:10.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.064397080999999995 2024-04-23T16:22:10.314 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.202019+0000", 2024-04-23T16:22:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6818999999999999e-05 2024-04-23T16:22:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.270829+0000", 2024-04-23T16:22:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.068809799000000005 2024-04-23T16:22:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:15.270845+0000", 2024-04-23T16:22:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6334e-05 2024-04-23T16:22:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.5.0:22443 2.2 e141/110 2:44bb8002:::benchmark_data_smithi188_77627_object33212:head v 141'12244, mlcod=141'12235)", 2024-04-23T16:22:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:16.438526+0000", 2024-04-23T16:22:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 173.458376668, 2024-04-23T16:22:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.137189283, 2024-04-23T16:22:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.438526+0000", 2024-04-23T16:22:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.438526+0000", 2024-04-23T16:22:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.438527+0000", 2024-04-23T16:22:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.497e-06 2024-04-23T16:22:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.438537+0000", 2024-04-23T16:22:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5000000000000005e-06 2024-04-23T16:22:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.318 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.318 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.438548+0000", 2024-04-23T16:22:10.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1875000000000001e-05 2024-04-23T16:22:10.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.318 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.318 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.438557+0000", 2024-04-23T16:22:10.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1990000000000006e-06 2024-04-23T16:22:10.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.318 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.318 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.461843+0000", 2024-04-23T16:22:10.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.023286107 2024-04-23T16:22:10.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.461862+0000", 2024-04-23T16:22:10.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9449e-05 2024-04-23T16:22:10.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.575662+0000", 2024-04-23T16:22:10.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.113799294 2024-04-23T16:22:10.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.575715+0000", 2024-04-23T16:22:10.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3362000000000002e-05 2024-04-23T16:22:10.319 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.320 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.320 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.320 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.5.0:22444 2.2 e141/110 2:44be3f9c:::benchmark_data_smithi188_77627_object32484:head v 141'12245, mlcod=141'12235)", 2024-04-23T16:22:10.320 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:16.438571+0000", 2024-04-23T16:22:10.320 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 173.45833117000001, 2024-04-23T16:22:10.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13717626399999999, 2024-04-23T16:22:10.320 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.320 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.320 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.320 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.438571+0000", 2024-04-23T16:22:10.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.438571+0000", 2024-04-23T16:22:10.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.438571+0000", 2024-04-23T16:22:10.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1300000000000001e-07 2024-04-23T16:22:10.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.438575+0000", 2024-04-23T16:22:10.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5700000000000001e-06 2024-04-23T16:22:10.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.438579+0000", 2024-04-23T16:22:10.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3699999999999997e-06 2024-04-23T16:22:10.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.438583+0000", 2024-04-23T16:22:10.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.275e-06 2024-04-23T16:22:10.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.323 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.323 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.466880+0000", 2024-04-23T16:22:10.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.028297157999999999 2024-04-23T16:22:10.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.323 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.323 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.466894+0000", 2024-04-23T16:22:10.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3631e-05 2024-04-23T16:22:10.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.323 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.323 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.575720+0000", 2024-04-23T16:22:10.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.1088262 2024-04-23T16:22:10.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.324 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.575747+0000", 2024-04-23T16:22:10.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7647000000000001e-05 2024-04-23T16:22:10.324 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.324 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.324 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.324 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.5.0:22445 2.2 e141/110 2:44be8a77:::benchmark_data_smithi188_77627_object44216:head v 141'12246, mlcod=141'12235)", 2024-04-23T16:22:10.324 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:16.440900+0000", 2024-04-23T16:22:10.324 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 173.456002213, 2024-04-23T16:22:10.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.134874139, 2024-04-23T16:22:10.324 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.325 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.325 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.325 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.440900+0000", 2024-04-23T16:22:10.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.440900+0000", 2024-04-23T16:22:10.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.440902+0000", 2024-04-23T16:22:10.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4649999999999999e-06 2024-04-23T16:22:10.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.440909+0000", 2024-04-23T16:22:10.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5360000000000004e-06 2024-04-23T16:22:10.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.440915+0000", 2024-04-23T16:22:10.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2899999999999999e-06 2024-04-23T16:22:10.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.440921+0000", 2024-04-23T16:22:10.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.728e-06 2024-04-23T16:22:10.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.470546+0000", 2024-04-23T16:22:10.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.029624885 2024-04-23T16:22:10.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.470566+0000", 2024-04-23T16:22:10.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9595000000000001e-05 2024-04-23T16:22:10.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.575749+0000", 2024-04-23T16:22:10.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.10518386 2024-04-23T16:22:10.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:16.575774+0000", 2024-04-23T16:22:10.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.478e-05 2024-04-23T16:22:10.328 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.328 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.328 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.328 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:322875 2.9 2:9397168b:::benchmark_data_smithi188_77627_object153526:head [delete] snapc 0=[] ondisk+write+known_if_redirected e182)", 2024-04-23T16:22:10.329 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:37.396815+0000", 2024-04-23T16:22:10.329 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.500087422999997, 2024-04-23T16:22:10.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.22352122699999999, 2024-04-23T16:22:10.329 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.329 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.329 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.329 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:10.329 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:10.329 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:10.329 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 322875 2024-04-23T16:22:10.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.329 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.329 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.396815+0000", 2024-04-23T16:22:10.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.396815+0000", 2024-04-23T16:22:10.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.396816+0000", 2024-04-23T16:22:10.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3379999999999999e-06 2024-04-23T16:22:10.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.396821+0000", 2024-04-23T16:22:10.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9910000000000002e-06 2024-04-23T16:22:10.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.396833+0000", 2024-04-23T16:22:10.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1425e-05 2024-04-23T16:22:10.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.396840+0000", 2024-04-23T16:22:10.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7260000000000002e-06 2024-04-23T16:22:10.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.400955+0000", 2024-04-23T16:22:10.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041142230000000002 2024-04-23T16:22:10.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.401071+0000", 2024-04-23T16:22:10.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116718 2024-04-23T16:22:10.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 7", 2024-04-23T16:22:10.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.401179+0000", 2024-04-23T16:22:10.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010777699999999999 2024-04-23T16:22:10.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:10.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.411631+0000", 2024-04-23T16:22:10.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010452095 2024-04-23T16:22:10.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:10.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.620248+0000", 2024-04-23T16:22:10.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.20861676400000001 2024-04-23T16:22:10.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.620281+0000", 2024-04-23T16:22:10.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3555e-05 2024-04-23T16:22:10.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.620336+0000", 2024-04-23T16:22:10.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4614999999999998e-05 2024-04-23T16:22:10.334 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.334 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.334 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.334 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:322883 2.9 2:9f9ee96e:::benchmark_data_smithi188_77627_object153534:head [delete] snapc 0=[] ondisk+write+known_if_redirected e182)", 2024-04-23T16:22:10.334 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:37.400850+0000", 2024-04-23T16:22:10.334 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.496052057, 2024-04-23T16:22:10.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.231656583, 2024-04-23T16:22:10.334 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.334 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.334 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.334 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:10.334 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:10.335 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:10.335 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 322883 2024-04-23T16:22:10.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.335 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.400850+0000", 2024-04-23T16:22:10.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.400850+0000", 2024-04-23T16:22:10.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.400852+0000", 2024-04-23T16:22:10.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1220000000000002e-06 2024-04-23T16:22:10.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.400862+0000", 2024-04-23T16:22:10.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2329999999999995e-06 2024-04-23T16:22:10.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.400874+0000", 2024-04-23T16:22:10.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2075e-05 2024-04-23T16:22:10.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.400882+0000", 2024-04-23T16:22:10.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1000000000000004e-06 2024-04-23T16:22:10.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.406575+0000", 2024-04-23T16:22:10.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056933080000000002 2024-04-23T16:22:10.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.406687+0000", 2024-04-23T16:22:10.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000111805 2024-04-23T16:22:10.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 7", 2024-04-23T16:22:10.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.406801+0000", 2024-04-23T16:22:10.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011424199999999999 2024-04-23T16:22:10.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:10.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.412595+0000", 2024-04-23T16:22:10.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0057941759999999998 2024-04-23T16:22:10.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:10.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.632422+0000", 2024-04-23T16:22:10.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.21982658399999999 2024-04-23T16:22:10.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.632450+0000", 2024-04-23T16:22:10.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8204e-05 2024-04-23T16:22:10.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.632507+0000", 2024-04-23T16:22:10.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6734000000000003e-05 2024-04-23T16:22:10.339 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.339 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.339 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:322899 2.9 2:9713c4ab:::benchmark_data_smithi188_77627_object153550:head [delete] snapc 0=[] ondisk+write+known_if_redirected e182)", 2024-04-23T16:22:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:37.416595+0000", 2024-04-23T16:22:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.480307580999998, 2024-04-23T16:22:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.21602859499999999, 2024-04-23T16:22:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 322899 2024-04-23T16:22:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.416595+0000", 2024-04-23T16:22:10.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.416595+0000", 2024-04-23T16:22:10.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.416597+0000", 2024-04-23T16:22:10.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.041e-06 2024-04-23T16:22:10.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.416602+0000", 2024-04-23T16:22:10.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5779999999999998e-06 2024-04-23T16:22:10.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.416617+0000", 2024-04-23T16:22:10.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4775e-05 2024-04-23T16:22:10.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.416625+0000", 2024-04-23T16:22:10.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9640000000000003e-06 2024-04-23T16:22:10.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.416677+0000", 2024-04-23T16:22:10.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2398000000000002e-05 2024-04-23T16:22:10.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.416803+0000", 2024-04-23T16:22:10.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012517 2024-04-23T16:22:10.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 7", 2024-04-23T16:22:10.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.416922+0000", 2024-04-23T16:22:10.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119127 2024-04-23T16:22:10.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:10.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.421882+0000", 2024-04-23T16:22:10.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049600160000000003 2024-04-23T16:22:10.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:10.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.632553+0000", 2024-04-23T16:22:10.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.210671678 2024-04-23T16:22:10.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.632584+0000", 2024-04-23T16:22:10.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0457999999999999e-05 2024-04-23T16:22:10.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.632623+0000", 2024-04-23T16:22:10.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9390000000000001e-05 2024-04-23T16:22:10.345 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.345 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.345 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.345 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:322907 2.9 2:93873cd0:::benchmark_data_smithi188_77627_object153558:head [delete] snapc 0=[] ondisk+write+known_if_redirected e182)", 2024-04-23T16:22:10.345 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:37.426934+0000", 2024-04-23T16:22:10.345 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.469967818000001, 2024-04-23T16:22:10.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.205775706, 2024-04-23T16:22:10.345 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.345 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.346 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.346 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:10.346 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:10.346 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:10.346 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 322907 2024-04-23T16:22:10.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.346 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.426934+0000", 2024-04-23T16:22:10.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.426934+0000", 2024-04-23T16:22:10.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.426936+0000", 2024-04-23T16:22:10.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.962e-06 2024-04-23T16:22:10.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.426943+0000", 2024-04-23T16:22:10.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7089999999999996e-06 2024-04-23T16:22:10.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.426956+0000", 2024-04-23T16:22:10.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2455e-05 2024-04-23T16:22:10.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.426964+0000", 2024-04-23T16:22:10.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6260000000000001e-06 2024-04-23T16:22:10.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.427023+0000", 2024-04-23T16:22:10.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8471000000000003e-05 2024-04-23T16:22:10.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.427153+0000", 2024-04-23T16:22:10.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013035400000000001 2024-04-23T16:22:10.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 7", 2024-04-23T16:22:10.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.427275+0000", 2024-04-23T16:22:10.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012149199999999999 2024-04-23T16:22:10.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:10.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.432170+0000", 2024-04-23T16:22:10.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0048951910000000001 2024-04-23T16:22:10.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:10.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.632655+0000", 2024-04-23T16:22:10.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.20048576800000001 2024-04-23T16:22:10.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.632683+0000", 2024-04-23T16:22:10.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7579000000000001e-05 2024-04-23T16:22:10.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.632710+0000", 2024-04-23T16:22:10.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7098999999999999e-05 2024-04-23T16:22:10.351 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.351 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.351 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.351 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:322928 2.9 2:92e119fe:::benchmark_data_smithi188_77627_object153579:head [delete] snapc 0=[] ondisk+write+known_if_redirected e182)", 2024-04-23T16:22:10.351 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:37.441842+0000", 2024-04-23T16:22:10.351 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.455060340000003, 2024-04-23T16:22:10.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.19099502099999999, 2024-04-23T16:22:10.351 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.351 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.351 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.351 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:10.351 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:10.351 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:10.352 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 322928 2024-04-23T16:22:10.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.352 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.441842+0000", 2024-04-23T16:22:10.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.441842+0000", 2024-04-23T16:22:10.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.441843+0000", 2024-04-23T16:22:10.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.04e-06 2024-04-23T16:22:10.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.441847+0000", 2024-04-23T16:22:10.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0280000000000002e-06 2024-04-23T16:22:10.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.441854+0000", 2024-04-23T16:22:10.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8379999999999996e-06 2024-04-23T16:22:10.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.441858+0000", 2024-04-23T16:22:10.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6530000000000002e-06 2024-04-23T16:22:10.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.441900+0000", 2024-04-23T16:22:10.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1906999999999997e-05 2024-04-23T16:22:10.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.442017+0000", 2024-04-23T16:22:10.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116329 2024-04-23T16:22:10.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 7", 2024-04-23T16:22:10.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.442122+0000", 2024-04-23T16:22:10.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010514999999999999 2024-04-23T16:22:10.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:10.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.451297+0000", 2024-04-23T16:22:10.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0091750749999999996 2024-04-23T16:22:10.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:10.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.632754+0000", 2024-04-23T16:22:10.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.18145734099999999 2024-04-23T16:22:10.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.632783+0000", 2024-04-23T16:22:10.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8674999999999999e-05 2024-04-23T16:22:10.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.632837+0000", 2024-04-23T16:22:10.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3984999999999999e-05 2024-04-23T16:22:10.356 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.356 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.356 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.356 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:322929 2.9 2:92fa7c25:::benchmark_data_smithi188_77627_object153580:head [delete] snapc 0=[] ondisk+write+known_if_redirected e182)", 2024-04-23T16:22:10.356 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:37.443515+0000", 2024-04-23T16:22:10.356 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.453386946000002, 2024-04-23T16:22:10.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.18941384999999999, 2024-04-23T16:22:10.357 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.357 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.357 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.357 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:10.357 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:10.357 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:10.357 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 322929 2024-04-23T16:22:10.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.357 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.443515+0000", 2024-04-23T16:22:10.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.443515+0000", 2024-04-23T16:22:10.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.443517+0000", 2024-04-23T16:22:10.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.604e-06 2024-04-23T16:22:10.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.443522+0000", 2024-04-23T16:22:10.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8960000000000002e-06 2024-04-23T16:22:10.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.443532+0000", 2024-04-23T16:22:10.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0146000000000001e-05 2024-04-23T16:22:10.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.443539+0000", 2024-04-23T16:22:10.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.126e-06 2024-04-23T16:22:10.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.446591+0000", 2024-04-23T16:22:10.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003051486 2024-04-23T16:22:10.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.446730+0000", 2024-04-23T16:22:10.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013946799999999999 2024-04-23T16:22:10.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 7", 2024-04-23T16:22:10.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.446851+0000", 2024-04-23T16:22:10.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000120709 2024-04-23T16:22:10.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:10.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.451410+0000", 2024-04-23T16:22:10.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045591410000000001 2024-04-23T16:22:10.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:10.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.632866+0000", 2024-04-23T16:22:10.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.181456538 2024-04-23T16:22:10.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.632901+0000", 2024-04-23T16:22:10.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4755000000000002e-05 2024-04-23T16:22:10.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.632929+0000", 2024-04-23T16:22:10.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7980999999999999e-05 2024-04-23T16:22:10.362 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.362 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.362 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.362 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:322940 2.9 2:9afdce9d:::benchmark_data_smithi188_77627_object153591:head [delete] snapc 0=[] ondisk+write+known_if_redirected e182)", 2024-04-23T16:22:10.362 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:37.452316+0000", 2024-04-23T16:22:10.362 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.444586207, 2024-04-23T16:22:10.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.18383755099999999, 2024-04-23T16:22:10.362 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.362 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.362 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.363 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:10.363 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:10.363 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:10.363 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 322940 2024-04-23T16:22:10.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.363 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.452316+0000", 2024-04-23T16:22:10.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.452316+0000", 2024-04-23T16:22:10.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.452318+0000", 2024-04-23T16:22:10.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.807e-06 2024-04-23T16:22:10.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.452323+0000", 2024-04-23T16:22:10.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2229999999999998e-06 2024-04-23T16:22:10.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.452337+0000", 2024-04-23T16:22:10.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3645000000000001e-05 2024-04-23T16:22:10.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.452345+0000", 2024-04-23T16:22:10.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7929999999999998e-06 2024-04-23T16:22:10.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.452395+0000", 2024-04-23T16:22:10.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0454999999999997e-05 2024-04-23T16:22:10.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.452504+0000", 2024-04-23T16:22:10.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010869700000000001 2024-04-23T16:22:10.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 7", 2024-04-23T16:22:10.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.452614+0000", 2024-04-23T16:22:10.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110431 2024-04-23T16:22:10.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:10.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.457411+0000", 2024-04-23T16:22:10.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047969470000000002 2024-04-23T16:22:10.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:10.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.636058+0000", 2024-04-23T16:22:10.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.178647009 2024-04-23T16:22:10.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.636103+0000", 2024-04-23T16:22:10.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5167999999999999e-05 2024-04-23T16:22:10.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.636154+0000", 2024-04-23T16:22:10.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0376e-05 2024-04-23T16:22:10.367 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.367 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.367 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.368 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:322966 2.9 2:9c918b5a:::benchmark_data_smithi188_77627_object153617:head [delete] snapc 0=[] ondisk+write+known_if_redirected e182)", 2024-04-23T16:22:10.368 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:37.481143+0000", 2024-04-23T16:22:10.368 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.415758795000002, 2024-04-23T16:22:10.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.15516331899999999, 2024-04-23T16:22:10.368 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.368 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.368 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.368 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:10.368 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:10.368 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:10.368 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 322966 2024-04-23T16:22:10.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.368 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.481143+0000", 2024-04-23T16:22:10.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.481143+0000", 2024-04-23T16:22:10.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.481145+0000", 2024-04-23T16:22:10.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.632e-06 2024-04-23T16:22:10.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.481151+0000", 2024-04-23T16:22:10.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6860000000000003e-06 2024-04-23T16:22:10.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.481163+0000", 2024-04-23T16:22:10.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1737e-05 2024-04-23T16:22:10.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.481170+0000", 2024-04-23T16:22:10.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5820000000000004e-06 2024-04-23T16:22:10.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.481214+0000", 2024-04-23T16:22:10.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3529000000000003e-05 2024-04-23T16:22:10.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.481299+0000", 2024-04-23T16:22:10.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5727000000000001e-05 2024-04-23T16:22:10.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 7", 2024-04-23T16:22:10.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.481371+0000", 2024-04-23T16:22:10.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1297999999999994e-05 2024-04-23T16:22:10.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:10.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.493943+0000", 2024-04-23T16:22:10.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012572679 2024-04-23T16:22:10.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:10.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.636200+0000", 2024-04-23T16:22:10.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.14225712400000001 2024-04-23T16:22:10.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.636253+0000", 2024-04-23T16:22:10.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2104000000000001e-05 2024-04-23T16:22:10.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.636307+0000", 2024-04-23T16:22:10.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4221000000000001e-05 2024-04-23T16:22:10.375 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.375 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.376 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.376 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:322971 2.9 2:92301ef6:::benchmark_data_smithi188_77627_object153622:head [delete] snapc 0=[] ondisk+write+known_if_redirected e182)", 2024-04-23T16:22:10.376 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:37.483838+0000", 2024-04-23T16:22:10.376 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.413064114000001, 2024-04-23T16:22:10.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.15922008600000001, 2024-04-23T16:22:10.376 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.376 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.376 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.376 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:10.376 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:10.376 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:10.376 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 322971 2024-04-23T16:22:10.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.377 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.483838+0000", 2024-04-23T16:22:10.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.483838+0000", 2024-04-23T16:22:10.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.483839+0000", 2024-04-23T16:22:10.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3319999999999999e-06 2024-04-23T16:22:10.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.483844+0000", 2024-04-23T16:22:10.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5379999999999996e-06 2024-04-23T16:22:10.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.483852+0000", 2024-04-23T16:22:10.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3389999999999993e-06 2024-04-23T16:22:10.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.483859+0000", 2024-04-23T16:22:10.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5230000000000001e-06 2024-04-23T16:22:10.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.485043+0000", 2024-04-23T16:22:10.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001184037 2024-04-23T16:22:10.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.485161+0000", 2024-04-23T16:22:10.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118305 2024-04-23T16:22:10.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 7", 2024-04-23T16:22:10.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.485271+0000", 2024-04-23T16:22:10.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010976699999999999 2024-04-23T16:22:10.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:10.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.493950+0000", 2024-04-23T16:22:10.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086794769999999997 2024-04-23T16:22:10.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:10.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.643001+0000", 2024-04-23T16:22:10.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.14905005399999999 2024-04-23T16:22:10.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.643026+0000", 2024-04-23T16:22:10.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5922999999999999e-05 2024-04-23T16:22:10.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.643058+0000", 2024-04-23T16:22:10.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1791000000000001e-05 2024-04-23T16:22:10.381 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.381 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.381 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.381 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:322978 2.9 2:95b0cc0d:::benchmark_data_smithi188_77627_object153629:head [delete] snapc 0=[] ondisk+write+known_if_redirected e182)", 2024-04-23T16:22:10.381 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:37.489722+0000", 2024-04-23T16:22:10.381 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.407180582999999, 2024-04-23T16:22:10.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.15344111199999999, 2024-04-23T16:22:10.382 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.382 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.382 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.382 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:10.382 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:10.382 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:10.382 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 322978 2024-04-23T16:22:10.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.382 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.489722+0000", 2024-04-23T16:22:10.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.489722+0000", 2024-04-23T16:22:10.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.489723+0000", 2024-04-23T16:22:10.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.333e-06 2024-04-23T16:22:10.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.489728+0000", 2024-04-23T16:22:10.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6609999999999999e-06 2024-04-23T16:22:10.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.489740+0000", 2024-04-23T16:22:10.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1898e-05 2024-04-23T16:22:10.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.489747+0000", 2024-04-23T16:22:10.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.571e-06 2024-04-23T16:22:10.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.489810+0000", 2024-04-23T16:22:10.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2708999999999999e-05 2024-04-23T16:22:10.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.489909+0000", 2024-04-23T16:22:10.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9518999999999994e-05 2024-04-23T16:22:10.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 7", 2024-04-23T16:22:10.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.490010+0000", 2024-04-23T16:22:10.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100244 2024-04-23T16:22:10.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:10.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.494059+0000", 2024-04-23T16:22:10.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0040493999999999999 2024-04-23T16:22:10.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:10.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.643103+0000", 2024-04-23T16:22:10.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.149044115 2024-04-23T16:22:10.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.643131+0000", 2024-04-23T16:22:10.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8356000000000001e-05 2024-04-23T16:22:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.643163+0000", 2024-04-23T16:22:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1306000000000001e-05 2024-04-23T16:22:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:322985 2.9 2:958bba1d:::benchmark_data_smithi188_77627_object153636:head [delete] snapc 0=[] ondisk+write+known_if_redirected e182)", 2024-04-23T16:22:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:21:37.498201+0000", 2024-04-23T16:22:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.398701449999997, 2024-04-23T16:22:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.15138764800000001, 2024-04-23T16:22:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:10.388 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:10.388 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:10.388 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:10.388 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:10.388 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:10.388 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 322985 2024-04-23T16:22:10.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.388 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:10.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:10.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.498201+0000", 2024-04-23T16:22:10.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:10.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.498201+0000", 2024-04-23T16:22:10.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:10.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:10.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.498203+0000", 2024-04-23T16:22:10.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9709999999999998e-06 2024-04-23T16:22:10.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:10.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.498209+0000", 2024-04-23T16:22:10.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2580000000000001e-06 2024-04-23T16:22:10.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:10.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.498222+0000", 2024-04-23T16:22:10.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2951e-05 2024-04-23T16:22:10.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:10.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.498229+0000", 2024-04-23T16:22:10.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2470000000000002e-06 2024-04-23T16:22:10.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:10.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.498289+0000", 2024-04-23T16:22:10.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9882e-05 2024-04-23T16:22:10.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:10.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.498418+0000", 2024-04-23T16:22:10.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000129357 2024-04-23T16:22:10.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 7", 2024-04-23T16:22:10.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.498551+0000", 2024-04-23T16:22:10.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013220300000000001 2024-04-23T16:22:10.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:10.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.502387+0000", 2024-04-23T16:22:10.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038367549999999999 2024-04-23T16:22:10.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:10.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.649514+0000", 2024-04-23T16:22:10.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.147126278 2024-04-23T16:22:10.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:10.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.649545+0000", 2024-04-23T16:22:10.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1603999999999998e-05 2024-04-23T16:22:10.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:10.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:10.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:10.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:21:37.649588+0000", 2024-04-23T16:22:10.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3142000000000001e-05 2024-04-23T16:22:10.393 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.393 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.393 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.393 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:10.393 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:10.393 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:22:10.393 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:22:10.421 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:10.473 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:22:10.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:10.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:22:10.577 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:22:10.609 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:10.614 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'unique_pool_1' created 2024-04-23T16:22:10.622 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.5 to 1 (10000) 2024-04-23T16:22:10.626 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:22:10.628 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:22:10.632 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:22:10.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:10.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:22:10.736 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:22:10.766 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:10.786 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:22:10.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:10.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:22:10.795 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:10 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1606815930' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-23T16:22:10.795 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:10 smithi103 ceph-mon[39629]: from='client.24994 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-23T16:22:10.795 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:10 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2403721823' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-23T16:22:10.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:10 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1606815930' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-23T16:22:10.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:10 smithi148 ceph-mon[44520]: from='client.24994 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-23T16:22:10.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:10 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2403721823' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-23T16:22:10.891 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:22:10.920 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:10.939 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:22:10.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:10.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:22:10.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:10 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1606815930' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-23T16:22:10.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:10 smithi188 ceph-mon[44101]: from='client.24994 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-23T16:22:10.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:10 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2403721823' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-23T16:22:11.043 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:22:11.075 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:11.096 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:22:11.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:11.100 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:22:11.200 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:22:11.229 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:11.248 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:22:11.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:11.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:22:11.352 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:22:11.379 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:11.391 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-23T16:22:11.398 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:22:11.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:11.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:22:11.502 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:22:11.531 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:11.618 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:11 smithi103 ceph-mon[39629]: pgmap v569: 1 pgs: 1 active+clean; 577 KiB data, 1019 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:11.618 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:11 smithi103 ceph-mon[39629]: from='client.24994 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-23T16:22:11.618 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:11 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2403721823' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-04-23T16:22:11.618 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:11 smithi103 ceph-mon[39629]: osdmap e192: 12 total, 12 up, 11 in 2024-04-23T16:22:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:11 smithi148 ceph-mon[44520]: pgmap v569: 1 pgs: 1 active+clean; 577 KiB data, 1019 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:11 smithi148 ceph-mon[44520]: from='client.24994 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-23T16:22:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:11 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2403721823' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-04-23T16:22:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:11 smithi148 ceph-mon[44520]: osdmap e192: 12 total, 12 up, 11 in 2024-04-23T16:22:11.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:11 smithi188 ceph-mon[44101]: pgmap v569: 1 pgs: 1 active+clean; 577 KiB data, 1019 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:11 smithi188 ceph-mon[44101]: from='client.24994 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-23T16:22:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:11 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2403721823' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-04-23T16:22:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:11 smithi188 ceph-mon[44101]: osdmap e192: 12 total, 12 up, 11 in 2024-04-23T16:22:12.381 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:22:12.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:12.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:22:12.487 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:22:12.520 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:12.541 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:22:12.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:12.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:22:12.645 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:22:12.677 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:12.697 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:22:12.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:12.700 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:22:12.801 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:22:12.830 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:12.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:12 smithi103 ceph-mon[39629]: osdmap e193: 12 total, 12 up, 11 in 2024-04-23T16:22:12.850 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:22:12.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:12.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:22:12.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:12 smithi148 ceph-mon[44520]: osdmap e193: 12 total, 12 up, 11 in 2024-04-23T16:22:12.953 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:22:12.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:12 smithi188 ceph-mon[44101]: osdmap e193: 12 total, 12 up, 11 in 2024-04-23T16:22:12.982 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:13.001 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:22:13.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:13.004 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:22:13.105 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:13.134 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:13.613 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-23T16:22:13.660 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:13.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:13.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:13.764 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:22:13.795 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:13.815 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:22:13.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:13.819 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:22:13.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:13 smithi103 ceph-mon[39629]: pgmap v572: 17 pgs: 7 creating+peering, 9 unknown, 1 active+clean; 577 KiB data, 999 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:13 smithi103 ceph-mon[39629]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:22:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:13 smithi103 ceph-mon[39629]: osdmap e194: 12 total, 12 up, 11 in 2024-04-23T16:22:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:13 smithi148 ceph-mon[44520]: pgmap v572: 17 pgs: 7 creating+peering, 9 unknown, 1 active+clean; 577 KiB data, 999 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:13 smithi148 ceph-mon[44520]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:22:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:13 smithi148 ceph-mon[44520]: osdmap e194: 12 total, 12 up, 11 in 2024-04-23T16:22:13.919 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:13.946 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:13.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:13 smithi188 ceph-mon[44101]: pgmap v572: 17 pgs: 7 creating+peering, 9 unknown, 1 active+clean; 577 KiB data, 999 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:13 smithi188 ceph-mon[44101]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:22:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:13 smithi188 ceph-mon[44101]: osdmap e194: 12 total, 12 up, 11 in 2024-04-23T16:22:15.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:14 smithi188 ceph-mon[44101]: pgmap v574: 17 pgs: 7 creating+peering, 1 unknown, 9 active+clean; 577 KiB data, 998 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:15.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:14 smithi103 ceph-mon[39629]: pgmap v574: 17 pgs: 7 creating+peering, 1 unknown, 9 active+clean; 577 KiB data, 998 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:15.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:14 smithi148 ceph-mon[44520]: pgmap v574: 17 pgs: 7 creating+peering, 1 unknown, 9 active+clean; 577 KiB data, 998 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:15.395 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:15.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:15.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:15.500 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:22:15.529 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:15.725 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:22:15.729 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-04-23T16:22:15.729 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:22:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:15.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:22:15.832 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:22:15.864 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:15.885 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:22:15.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:15.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:22:15.989 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:22:15.999 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:15 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:22:15.999 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:15 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:22:15.999 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:15 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:22:15.999 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:15 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:22:16.000 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:15 smithi103 ceph-mon[39629]: osdmap e195: 12 total, 12 up, 11 in 2024-04-23T16:22:16.017 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:16.037 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:22:16.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:16.039 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:22:16.140 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:22:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:15 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:22:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:15 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:22:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:15 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:22:16.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:15 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:22:16.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:15 smithi188 ceph-mon[44101]: osdmap e195: 12 total, 12 up, 11 in 2024-04-23T16:22:16.167 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:16.186 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:22:16.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:16.188 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:22:16.289 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:22:16.296 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:22:16.315 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:16.330 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:22:16.333 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:22:16.334 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:22:16.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:16.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:22:16.356 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:22:16.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:15 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:22:16.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:15 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:22:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:15 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:22:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:15 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:22:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:15 smithi148 ceph-mon[44520]: osdmap e195: 12 total, 12 up, 11 in 2024-04-23T16:22:16.375 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:22:16.394 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:22:16.414 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:22:16.433 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:22:16.438 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:22:16.455 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:22:16.467 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:16.476 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:22:16.496 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:22:16.516 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:22:16.535 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:22:16.554 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:22:16.556 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:22:16.556 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:22:16.556 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:22:16.556 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:22:16.556 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:22:16.556 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:22:16.556 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:22:16.556 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:22:16.556 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:22:16.556 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:22:16.556 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:22:16.556 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:22:17.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:16 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/331466983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:22:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:16 smithi188 ceph-mon[44101]: pgmap v576: 17 pgs: 17 active+clean; 577 KiB data, 998 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:16 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/331466983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:22:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:17.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:16 smithi103 ceph-mon[39629]: pgmap v576: 17 pgs: 17 active+clean; 577 KiB data, 998 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:17.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:16 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/331466983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:22:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:16 smithi148 ceph-mon[44520]: pgmap v576: 17 pgs: 17 active+clean; 577 KiB data, 998 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:17.424 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:22:17.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:17.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:22:17.532 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:22:17.564 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:17.585 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:22:17.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:17.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:22:17.689 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:17.717 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:18.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:19.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:18 smithi103 ceph-mon[39629]: pgmap v577: 17 pgs: 17 active+clean; 577 KiB data, 998 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:19.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:18 smithi188 ceph-mon[44101]: pgmap v577: 17 pgs: 17 active+clean; 577 KiB data, 998 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:18 smithi148 ceph-mon[44520]: pgmap v577: 17 pgs: 17 active+clean; 577 KiB data, 998 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:19.897 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:19.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:19.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:19.924 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:22:20.002 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:22:20.034 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:20.056 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:22:20.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:20.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:22:20.160 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:22:20.187 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:19 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1739098522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:22:20.189 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:20.210 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:22:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:20.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:22:20.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:19 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1739098522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:22:20.313 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:22:20.345 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:20.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:19 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1739098522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:22:20.366 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:22:20.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:20.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:22:20.470 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:22:20.502 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:20.523 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:22:20.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:20.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:22:20.628 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:22:20.659 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:20.681 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:22:20.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:20.684 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:22:20.785 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:22:20.817 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:20.838 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:22:20.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:20.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:22:20.943 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:22:20.974 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:20.995 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:22:20.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:20.998 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:22:21.099 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:22:21.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:20 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:22:21.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:20 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1739098522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:22:21.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:20 smithi103 ceph-mon[39629]: osdmap e196: 12 total, 12 up, 11 in 2024-04-23T16:22:21.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:20 smithi103 ceph-mon[39629]: pgmap v579: 17 pgs: 17 active+clean; 577 KiB data, 998 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:21.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:20 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:22:21.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:20 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1739098522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:22:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:20 smithi148 ceph-mon[44520]: osdmap e196: 12 total, 12 up, 11 in 2024-04-23T16:22:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:20 smithi148 ceph-mon[44520]: pgmap v579: 17 pgs: 17 active+clean; 577 KiB data, 998 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:21.131 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:21.152 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:22:21.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:21.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:22:21.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:20 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:22:21.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:20 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1739098522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:22:21.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:20 smithi188 ceph-mon[44101]: osdmap e196: 12 total, 12 up, 11 in 2024-04-23T16:22:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:20 smithi188 ceph-mon[44101]: pgmap v579: 17 pgs: 17 active+clean; 577 KiB data, 998 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:21.256 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:22:21.285 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:21.305 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:22:21.306 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:22:21.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:21.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:22:21.410 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:22:21.437 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:21.457 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:22:21.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:21.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:22:21.560 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:22:21.588 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:21.607 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:22:21.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:21.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:22:21.711 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:22:21.739 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:21.757 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:22:21.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:21.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:22:21.861 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:22:21.890 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:21.991 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:22:21.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:21.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:22:22.096 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:22.125 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:22.417 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:22:22.417 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:22:22.427 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:22.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:22.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:22.531 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:22:22.564 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:22.586 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:22:22.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:22.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:22:22.690 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:22:22.722 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:22.743 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:22:22.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:22.747 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:22:22.848 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:22:22.880 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:22.902 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:22:22.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:22.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:22:23.006 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:22:23.035 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:23.123 INFO:teuthology.orchestra.run.smithi103.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-04-23T16:22:23.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:23 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:22:23.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:23 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:22:23.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:23 smithi103 ceph-mon[39629]: osdmap e197: 12 total, 12 up, 11 in 2024-04-23T16:22:23.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:23 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2706107674' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:22:23.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:23 smithi103 ceph-mon[39629]: pgmap v581: 17 pgs: 17 active+clean; 577 KiB data, 998 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:23.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:23 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:22:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:23 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:22:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:23 smithi148 ceph-mon[44520]: osdmap e197: 12 total, 12 up, 11 in 2024-04-23T16:22:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:23 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2706107674' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:22:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:23 smithi148 ceph-mon[44520]: pgmap v581: 17 pgs: 17 active+clean; 577 KiB data, 998 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:23 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:22:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:23 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:22:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:23 smithi188 ceph-mon[44101]: osdmap e197: 12 total, 12 up, 11 in 2024-04-23T16:22:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:23 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2706107674' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:22:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:23 smithi188 ceph-mon[44101]: pgmap v581: 17 pgs: 17 active+clean; 577 KiB data, 998 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:23.655 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-23T16:22:24.125 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.6 to 1 (10000) 2024-04-23T16:22:24.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2706107674' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:22:24.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:24 smithi103 ceph-mon[39629]: osdmap e198: 12 total, 12 up, 11 in 2024-04-23T16:22:24.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4009298490' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:22:24.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2706107674' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:22:24.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:24 smithi188 ceph-mon[44101]: osdmap e198: 12 total, 12 up, 11 in 2024-04-23T16:22:24.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4009298490' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:22:24.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:24 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2706107674' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:22:24.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:24 smithi148 ceph-mon[44520]: osdmap e198: 12 total, 12 up, 11 in 2024-04-23T16:22:24.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:24 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4009298490' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:22:24.781 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-23T16:22:24.784 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:22:24.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:24.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:22:24.854 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:22:24.888 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:22:24.920 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:24.942 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:22:24.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:24.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:22:25.045 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:22:25.074 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:25.094 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:22:25.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:25.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:22:25.198 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:22:25.230 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:25.252 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:22:25.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:25.255 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:22:25.356 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:22:25.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:25 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4009298490' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:22:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:25 smithi148 ceph-mon[44520]: osdmap e199: 12 total, 12 up, 11 in 2024-04-23T16:22:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:25 smithi148 ceph-mon[44520]: pgmap v584: 17 pgs: 17 active+clean; 577 KiB data, 998 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:25.375 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:22:25.386 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:25.406 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:22:25.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:25.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:22:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:25 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4009298490' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:22:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:25 smithi188 ceph-mon[44101]: osdmap e199: 12 total, 12 up, 11 in 2024-04-23T16:22:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:25 smithi188 ceph-mon[44101]: pgmap v584: 17 pgs: 17 active+clean; 577 KiB data, 998 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:25.510 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:22:25.538 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:25.557 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:22:25.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:25.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:22:25.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:25 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4009298490' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:22:25.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:25 smithi103 ceph-mon[39629]: osdmap e199: 12 total, 12 up, 11 in 2024-04-23T16:22:25.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:25 smithi103 ceph-mon[39629]: pgmap v584: 17 pgs: 17 active+clean; 577 KiB data, 998 MiB used, 982 GiB / 983 GiB avail 2024-04-23T16:22:25.661 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:22:25.693 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:25.714 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:22:25.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:25.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:22:25.818 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:22:25.847 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:25.868 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:22:25.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:25.871 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:22:25.972 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:22:26.000 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:26.017 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:22:26.019 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:22:26.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:26.022 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:22:26.123 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:22:26.151 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:26.169 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:22:26.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:26.172 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:22:26.194 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:22:26.273 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:26.299 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:26.379 DEBUG:teuthology.orchestra.run.smithi188:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-04-23T16:22:26.381 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-23T16:22:26.381 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-23T16:22:26.437 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:26 smithi103 ceph-mon[39629]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:22:26.437 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:26 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:22:26.447 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:26.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:26.472 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:hints = 1 2024-04-23T16:22:26.472 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-23T16:22:26.472 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Object prefix: benchmark_data_smithi188_89276 2024-04-23T16:22:26.472 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:22:26.473 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 0 0 0 0 0 0 - 0 2024-04-23T16:22:26.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:26 smithi188 ceph-mon[44101]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:22:26.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:26 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:22:26.551 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:22:26.589 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:26.618 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:22:26.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:26.622 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:22:26.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:26 smithi148 ceph-mon[44520]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:22:26.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:26 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:22:26.722 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:22:26.762 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:26.789 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:22:26.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:26.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:22:26.893 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:22:26.934 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:26.958 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:22:26.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:26.961 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:22:27.062 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:22:27.100 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:27.127 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:22:27.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:27.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:22:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:27 smithi188 ceph-mon[44101]: pgmap v585: 17 pgs: 17 active+clean; 641 KiB data, 998 MiB used, 982 GiB / 983 GiB avail; 11 KiB/s wr, 0 op/s 2024-04-23T16:22:27.233 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:27.276 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:27.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:27 smithi148 ceph-mon[44520]: pgmap v585: 17 pgs: 17 active+clean; 641 KiB data, 998 MiB used, 982 GiB / 983 GiB avail; 11 KiB/s wr, 0 op/s 2024-04-23T16:22:27.472 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 1 15 6310 6295 393.416 393.438 0.00511447 0.00253419 2024-04-23T16:22:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:27 smithi103 ceph-mon[39629]: pgmap v585: 17 pgs: 17 active+clean; 641 KiB data, 998 MiB used, 982 GiB / 983 GiB avail; 11 KiB/s wr, 0 op/s 2024-04-23T16:22:28.472 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 2 16 12081 12065 377.006 360.625 0.00266254 0.00264773 2024-04-23T16:22:28.487 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:28.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:28.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:28.592 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:22:28.631 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:28.655 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:22:28.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:28.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:22:28.759 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:28.799 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:28.829 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:28.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:28.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:28.938 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:22:28.973 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:29.000 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:22:29.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:29.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:22:29.106 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:22:29.146 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:29.473 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 3 16 17220 17204 358.39 321.188 0.00158945 0.00278673 2024-04-23T16:22:29.735 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:22:29.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:29.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:22:29.739 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:29 smithi103 ceph-mon[39629]: pgmap v586: 17 pgs: 17 active+clean; 127 MiB data, 1.0 GiB used, 982 GiB / 983 GiB avail; 20 MiB/s wr, 313 op/s 2024-04-23T16:22:29.840 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:22:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:29 smithi148 ceph-mon[44520]: pgmap v586: 17 pgs: 17 active+clean; 127 MiB data, 1.0 GiB used, 982 GiB / 983 GiB avail; 20 MiB/s wr, 313 op/s 2024-04-23T16:22:29.878 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:29 smithi188 ceph-mon[44101]: pgmap v586: 17 pgs: 17 active+clean; 127 MiB data, 1.0 GiB used, 982 GiB / 983 GiB avail; 20 MiB/s wr, 313 op/s 2024-04-23T16:22:29.881 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:29.909 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:22:29.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:29.913 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:22:30.014 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:22:30.049 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:30.078 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:22:30.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:30.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:22:30.183 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:22:30.221 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:30.247 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:22:30.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:30.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:22:30.353 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:22:30.389 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:30.412 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:22:30.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:30.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:22:30.434 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:22:30.473 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 4 16 21550 21534 336.443 270.625 0.00393717 0.00296909 2024-04-23T16:22:30.516 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:22:30.548 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:30.549 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:22:30.549 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:22:30.574 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:22:30.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:30.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:22:30.631 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.7 to 1 (10000) 2024-04-23T16:22:30.679 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:22:30.708 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:30 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4254351018' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:22:30.710 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:30.737 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:22:30.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:30.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:22:30.843 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:22:30.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:30 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4254351018' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:22:30.892 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:30.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:30 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4254351018' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:22:31.152 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:22:31.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:31.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:22:31.257 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:22:31.294 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:31.321 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:22:31.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:31.325 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:22:31.425 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:22:31.460 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:31.473 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 5 16 25751 25735 321.663 262.562 0.00486354 0.00310625 2024-04-23T16:22:31.487 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:22:31.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:31.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:22:31.504 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:22:31.518 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:22:31.551 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:22:31.586 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:22:31.593 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:22:31.619 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:31 smithi188 ceph-mon[44101]: pgmap v587: 17 pgs: 17 active+clean; 425 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 53 MiB/s wr, 848 op/s 2024-04-23T16:22:31.619 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:31 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4254351018' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:22:31.619 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:31 smithi188 ceph-mon[44101]: osdmap e200: 12 total, 12 up, 11 in 2024-04-23T16:22:31.619 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:31 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2970005674' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:22:31.624 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:22:31.627 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:31.653 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:22:31.657 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:22:31.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:31.660 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:22:31.686 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:22:31.720 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:22:31.755 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:22:31.761 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:22:31.784 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:22:31.797 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:31.817 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:22:31.827 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:22:31.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:31.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:22:31.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:31 smithi103 ceph-mon[39629]: pgmap v587: 17 pgs: 17 active+clean; 425 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 53 MiB/s wr, 848 op/s 2024-04-23T16:22:31.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:31 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4254351018' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:22:31.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:31 smithi103 ceph-mon[39629]: osdmap e200: 12 total, 12 up, 11 in 2024-04-23T16:22:31.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:31 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2970005674' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:22:31.852 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:22:31.862 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:22:31.862 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:22:31.862 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:22:31.862 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:22:31.863 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:22:31.863 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:22:31.863 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:22:31.863 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:22:31.863 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:22:31.863 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:22:31.863 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:22:31.863 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:22:31.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:31 smithi148 ceph-mon[44520]: pgmap v587: 17 pgs: 17 active+clean; 425 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 53 MiB/s wr, 848 op/s 2024-04-23T16:22:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:31 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4254351018' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:22:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:31 smithi148 ceph-mon[44520]: osdmap e200: 12 total, 12 up, 11 in 2024-04-23T16:22:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:31 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2970005674' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:22:31.931 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:31.974 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:32.079 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:32.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:32.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:32.110 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-23T16:22:32.185 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:22:32.222 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:32.251 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:22:32.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:32.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:22:32.355 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:22:32.397 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:32.422 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:22:32.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:32.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:22:32.473 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 6 15 29574 29559 307.883 239 0.00692006 0.003246 2024-04-23T16:22:32.526 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:32.567 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:32.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:33.026 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:33.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:33.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:33.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:33.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:33.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:33.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:33.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:33.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:33.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:33.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:33.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:33.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:33.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:33.130 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:22:33.171 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:33.473 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 7 16 33524 33508 299.156 246.812 0.0070679 0.00334137 2024-04-23T16:22:33.560 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 8 1 2024-04-23T16:22:33.636 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:22:33.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:33.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:22:33.740 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:22:33.776 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:33.801 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:22:33.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:33.805 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:22:33.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:33 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:33.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:33 smithi103 ceph-mon[39629]: pgmap v589: 17 pgs: 17 active+clean; 1.0 GiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-04-23T16:22:33.906 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:22:33.946 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:33 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:33.946 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:33 smithi188 ceph-mon[44101]: pgmap v589: 17 pgs: 17 active+clean; 1.0 GiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-04-23T16:22:33.949 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:33.972 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:22:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:33.975 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:22:34.076 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:22:34.116 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:34.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:33 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:34.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:33 smithi148 ceph-mon[44520]: pgmap v589: 17 pgs: 17 active+clean; 1.0 GiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-04-23T16:22:34.323 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:22:34.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:34.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:22:34.343 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:22:34.360 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:22:34.427 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:22:34.465 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:34.473 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 8 16 36828 36812 287.572 206.5 0.00460996 0.00347559 2024-04-23T16:22:34.493 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:22:34.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:34.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:22:34.600 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:34.646 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:34.914 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:34 smithi103 ceph-mon[39629]: pgmap v590: 17 pgs: 17 active+clean; 1.7 GiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 173 MiB/s wr, 2.77k op/s 2024-04-23T16:22:35.071 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:22:35.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:34 smithi188 ceph-mon[44101]: pgmap v590: 17 pgs: 17 active+clean; 1.7 GiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 173 MiB/s wr, 2.77k op/s 2024-04-23T16:22:35.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:34 smithi148 ceph-mon[44520]: pgmap v590: 17 pgs: 17 active+clean; 1.7 GiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 173 MiB/s wr, 2.77k op/s 2024-04-23T16:22:35.473 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 9 16 40190 40174 278.965 210.125 0.00400125 0.00358299 2024-04-23T16:22:36.203 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:36.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:36.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:36.311 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:22:36.349 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:36.375 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:22:36.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:36.380 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:22:36.473 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 10 16 43451 43435 271.448 203.812 0.00525008 0.00368253 2024-04-23T16:22:36.481 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:22:36.534 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:37.040 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:22:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:37 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:22:37.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:37 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:22:37.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:37 smithi103 ceph-mon[39629]: osdmap e201: 12 total, 12 up, 11 in 2024-04-23T16:22:37.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:37 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:22:37.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:37 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:22:37.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:37 smithi103 ceph-mon[39629]: pgmap v592: 17 pgs: 17 active+clean; 2.3 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 273 MiB/s wr, 4.37k op/s 2024-04-23T16:22:37.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:37 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/559306705' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:22:37.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:37 smithi103 ceph-mon[39629]: from='client.25048 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:22:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:37 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:22:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:37 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:22:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:37 smithi148 ceph-mon[44520]: osdmap e201: 12 total, 12 up, 11 in 2024-04-23T16:22:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:37 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:22:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:37 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:22:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:37 smithi148 ceph-mon[44520]: pgmap v592: 17 pgs: 17 active+clean; 2.3 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 273 MiB/s wr, 4.37k op/s 2024-04-23T16:22:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:37 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/559306705' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:22:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:37 smithi148 ceph-mon[44520]: from='client.25048 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:22:37.473 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 11 15 46549 46534 264.378 193.688 0.00351784 0.00378115 2024-04-23T16:22:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:37 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:22:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:37 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:22:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:37 smithi188 ceph-mon[44101]: osdmap e201: 12 total, 12 up, 11 in 2024-04-23T16:22:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:37 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:22:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:37 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:22:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:37 smithi188 ceph-mon[44101]: pgmap v592: 17 pgs: 17 active+clean; 2.3 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 273 MiB/s wr, 4.37k op/s 2024-04-23T16:22:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:37 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/559306705' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:22:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:37 smithi188 ceph-mon[44101]: from='client.25048 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:22:37.504 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:22:37.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:37.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:22:37.611 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:22:37.648 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:37.680 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:22:37.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:37.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:22:37.785 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:22:37.823 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:37.851 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:22:37.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:37.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:22:37.955 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:22:38.003 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:38.033 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:22:38.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:38.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:22:38.140 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:38.186 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:38 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:22:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:38 smithi103 ceph-mon[39629]: from='client.25048 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:22:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:38 smithi103 ceph-mon[39629]: osdmap e202: 12 total, 12 up, 11 in 2024-04-23T16:22:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:38 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:22:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:38 smithi148 ceph-mon[44520]: from='client.25048 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:22:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:38 smithi148 ceph-mon[44520]: osdmap e202: 12 total, 12 up, 11 in 2024-04-23T16:22:38.473 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 12 16 49569 49553 258.069 188.688 0.00594516 0.00387348 2024-04-23T16:22:38.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:38 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:22:38.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:38 smithi188 ceph-mon[44101]: from='client.25048 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:22:38.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:38 smithi188 ceph-mon[44101]: osdmap e202: 12 total, 12 up, 11 in 2024-04-23T16:22:38.547 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:38.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:38.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:38.657 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:22:38.704 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:38.818 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:22:38.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:38.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:22:38.923 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:22:38.958 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:38.987 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:22:38.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:38.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:22:39.092 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:22:39.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:39 smithi103 ceph-mon[39629]: pgmap v594: 17 pgs: 17 active+clean; 2.5 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 271 MiB/s wr, 4.33k op/s 2024-04-23T16:22:39.129 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:39.154 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:22:39.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:39.158 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:22:39.259 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:22:39.290 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:39 smithi148 ceph-mon[44520]: pgmap v594: 17 pgs: 17 active+clean; 2.5 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 271 MiB/s wr, 4.33k op/s 2024-04-23T16:22:39.293 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:39.321 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:22:39.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:39.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:22:39.426 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:22:39.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:39 smithi188 ceph-mon[44101]: pgmap v594: 17 pgs: 17 active+clean; 2.5 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 271 MiB/s wr, 4.33k op/s 2024-04-23T16:22:39.465 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:39.473 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 13 16 52377 52361 251.716 175.5 0.0094133 0.00397076 2024-04-23T16:22:39.492 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:22:39.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:39.497 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:22:39.598 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:22:39.633 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:39.660 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:22:39.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:39.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:22:39.764 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:22:39.805 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:39.830 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:22:39.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:39.834 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:22:39.935 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:39.980 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:40.030 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:40.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:40.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:40.036 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:22:40.078 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.8 to 1 (10000) 2024-04-23T16:22:40.137 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:22:40.171 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:40.198 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:22:40.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:40.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:22:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:40 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1331513408' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:22:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:40 smithi188 ceph-mon[44101]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:22:40.306 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:22:40.340 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:40 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1331513408' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:22:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:40 smithi103 ceph-mon[39629]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:22:40.368 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:22:40.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:40.372 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:22:40.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:40 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1331513408' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:22:40.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:40 smithi148 ceph-mon[44520]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:22:40.472 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:22:40.473 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 14 16 55087 55071 245.834 169.375 0.00338604 0.00406588 2024-04-23T16:22:40.511 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:40.558 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:22:40.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:40.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:22:40.664 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:22:40.700 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:40.731 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:22:40.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:40.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:22:40.835 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:22:40.878 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:40.899 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:22:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:40.903 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:22:41.004 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:22:41.038 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:41.064 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:41 smithi103 ceph-mon[39629]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:22:41.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:41 smithi103 ceph-mon[39629]: osdmap e203: 12 total, 12 up, 11 in 2024-04-23T16:22:41.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:41 smithi103 ceph-mon[39629]: pgmap v596: 17 pgs: 17 active+clean; 2.8 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 191 MiB/s wr, 3.05k op/s 2024-04-23T16:22:41.065 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:22:41.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:41.071 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:22:41.172 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:22:41.206 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:41.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:41 smithi188 ceph-mon[44101]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:22:41.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:41 smithi188 ceph-mon[44101]: osdmap e203: 12 total, 12 up, 11 in 2024-04-23T16:22:41.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:41 smithi188 ceph-mon[44101]: pgmap v596: 17 pgs: 17 active+clean; 2.8 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 191 MiB/s wr, 3.05k op/s 2024-04-23T16:22:41.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:41 smithi148 ceph-mon[44520]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:22:41.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:41 smithi148 ceph-mon[44520]: osdmap e203: 12 total, 12 up, 11 in 2024-04-23T16:22:41.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:41 smithi148 ceph-mon[44520]: pgmap v596: 17 pgs: 17 active+clean; 2.8 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 191 MiB/s wr, 3.05k op/s 2024-04-23T16:22:41.473 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 15 16 57862 57846 241.006 173.438 0.00584218 0.00414759 2024-04-23T16:22:42.080 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:22:42.273 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:22:42.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:42.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:22:42.380 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:22:42.405 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:42.429 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:22:42.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:42.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:22:42.474 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 16 16 60293 60277 235.439 151.938 0.00723484 0.00424577 2024-04-23T16:22:42.535 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:42.576 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:43.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:43 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:22:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:43 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:22:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:43 smithi188 ceph-mon[44101]: osdmap e204: 12 total, 12 up, 11 in 2024-04-23T16:22:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:43 smithi188 ceph-mon[44101]: pgmap v598: 17 pgs: 17 active+clean; 3.2 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 162 MiB/s wr, 2.59k op/s 2024-04-23T16:22:43.474 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 17 16 62651 62635 230.258 147.375 0.00395938 0.00434131 2024-04-23T16:22:43.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:43 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:22:43.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:43 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:22:43.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:43 smithi103 ceph-mon[39629]: osdmap e204: 12 total, 12 up, 11 in 2024-04-23T16:22:43.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:43 smithi103 ceph-mon[39629]: pgmap v598: 17 pgs: 17 active+clean; 3.2 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 162 MiB/s wr, 2.59k op/s 2024-04-23T16:22:43.605 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:22:43.605 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:22:43.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:22:43.605 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:22:43.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.606 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:111287 2.3 e130/110 2:c1312921:::benchmark_data_smithi188_77627_object111286:head v 130'6933, mlcod=130'6930)", 2024-04-23T16:22:43.606 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:34.045153+0000", 2024-04-23T16:22:43.606 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 248.401198841, 2024-04-23T16:22:43.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.065976758999999996, 2024-04-23T16:22:43.606 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.606 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.606 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:43.606 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.045153+0000", 2024-04-23T16:22:43.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.045153+0000", 2024-04-23T16:22:43.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.045154+0000", 2024-04-23T16:22:43.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1400000000000001e-06 2024-04-23T16:22:43.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.045217+0000", 2024-04-23T16:22:43.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2706000000000006e-05 2024-04-23T16:22:43.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.045224+0000", 2024-04-23T16:22:43.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9210000000000003e-06 2024-04-23T16:22:43.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.045229+0000", 2024-04-23T16:22:43.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7210000000000002e-06 2024-04-23T16:22:43.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.045665+0000", 2024-04-23T16:22:43.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043620399999999999 2024-04-23T16:22:43.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.045681+0000", 2024-04-23T16:22:43.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6554000000000001e-05 2024-04-23T16:22:43.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:43.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.111091+0000", 2024-04-23T16:22:43.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.065409321000000006 2024-04-23T16:22:43.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.111130+0000", 2024-04-23T16:22:43.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9192e-05 2024-04-23T16:22:43.609 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.609 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.609 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.610 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:111305 2.1 2:8c0ac4c2:::benchmark_data_smithi188_77627_object111304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-04-23T16:22:43.610 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:34.058150+0000", 2024-04-23T16:22:43.610 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 248.38820223499999, 2024-04-23T16:22:43.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.058512308999999998, 2024-04-23T16:22:43.610 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.610 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.610 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:43.610 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:43.610 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:43.610 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:43.610 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111305 2024-04-23T16:22:43.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.611 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058150+0000", 2024-04-23T16:22:43.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058150+0000", 2024-04-23T16:22:43.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058151+0000", 2024-04-23T16:22:43.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6799999999999999e-07 2024-04-23T16:22:43.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058361+0000", 2024-04-23T16:22:43.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021016200000000001 2024-04-23T16:22:43.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058369+0000", 2024-04-23T16:22:43.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7789999999999996e-06 2024-04-23T16:22:43.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058375+0000", 2024-04-23T16:22:43.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.438e-06 2024-04-23T16:22:43.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058417+0000", 2024-04-23T16:22:43.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2456000000000001e-05 2024-04-23T16:22:43.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.103743+0000", 2024-04-23T16:22:43.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045325229000000002 2024-04-23T16:22:43.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:22:43.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.103940+0000", 2024-04-23T16:22:43.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000197715 2024-04-23T16:22:43.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:43.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.110315+0000", 2024-04-23T16:22:43.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0063742800000000004 2024-04-23T16:22:43.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:43.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.116571+0000", 2024-04-23T16:22:43.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062562659999999999 2024-04-23T16:22:43.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:43.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.116605+0000", 2024-04-23T16:22:43.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4338999999999998e-05 2024-04-23T16:22:43.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.116662+0000", 2024-04-23T16:22:43.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6876999999999998e-05 2024-04-23T16:22:43.615 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.615 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.615 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.616 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:111308 2.1 2:8fec7474:::benchmark_data_smithi188_77627_object111307:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-04-23T16:22:43.616 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:34.058387+0000", 2024-04-23T16:22:43.616 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 248.38796499099999, 2024-04-23T16:22:43.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.058429353000000003, 2024-04-23T16:22:43.616 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.616 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.616 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:43.616 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:43.616 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:43.616 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:43.616 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111308 2024-04-23T16:22:43.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.616 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058387+0000", 2024-04-23T16:22:43.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058387+0000", 2024-04-23T16:22:43.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058388+0000", 2024-04-23T16:22:43.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.54e-07 2024-04-23T16:22:43.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.058691+0000", 2024-04-23T16:22:43.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000303403 2024-04-23T16:22:43.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.103546+0000", 2024-04-23T16:22:43.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.044854484999999999 2024-04-23T16:22:43.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.103561+0000", 2024-04-23T16:22:43.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5118e-05 2024-04-23T16:22:43.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.106994+0000", 2024-04-23T16:22:43.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034329320000000001 2024-04-23T16:22:43.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.107108+0000", 2024-04-23T16:22:43.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011389500000000001 2024-04-23T16:22:43.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:22:43.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.107255+0000", 2024-04-23T16:22:43.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014715799999999999 2024-04-23T16:22:43.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:43.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.116678+0000", 2024-04-23T16:22:43.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094229199999999996 2024-04-23T16:22:43.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:43.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.116720+0000", 2024-04-23T16:22:43.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1953999999999998e-05 2024-04-23T16:22:43.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:43.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.116771+0000", 2024-04-23T16:22:43.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1246999999999999e-05 2024-04-23T16:22:43.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:34.116816+0000", 2024-04-23T16:22:43.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5487e-05 2024-04-23T16:22:43.621 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.621 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.621 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.621 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:114411 2.1 2:814709c2:::benchmark_data_smithi188_77627_object114410:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-23T16:22:43.621 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:35.939257+0000", 2024-04-23T16:22:43.621 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 246.50709534699999, 2024-04-23T16:22:43.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.058078545000000002, 2024-04-23T16:22:43.621 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.621 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.621 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:43.622 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:43.622 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:43.622 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:43.622 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114411 2024-04-23T16:22:43.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.622 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.939257+0000", 2024-04-23T16:22:43.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.939257+0000", 2024-04-23T16:22:43.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.939258+0000", 2024-04-23T16:22:43.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6750000000000001e-06 2024-04-23T16:22:43.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.939311+0000", 2024-04-23T16:22:43.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2608999999999997e-05 2024-04-23T16:22:43.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.939319+0000", 2024-04-23T16:22:43.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0949999999999996e-06 2024-04-23T16:22:43.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.939325+0000", 2024-04-23T16:22:43.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4879999999999998e-06 2024-04-23T16:22:43.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.939362+0000", 2024-04-23T16:22:43.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7787000000000003e-05 2024-04-23T16:22:43.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.939445+0000", 2024-04-23T16:22:43.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3014000000000002e-05 2024-04-23T16:22:43.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:22:43.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.939556+0000", 2024-04-23T16:22:43.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110722 2024-04-23T16:22:43.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:43.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.984646+0000", 2024-04-23T16:22:43.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045089865999999999 2024-04-23T16:22:43.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:43.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.997243+0000", 2024-04-23T16:22:43.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012597215 2024-04-23T16:22:43.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:43.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.997286+0000", 2024-04-23T16:22:43.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2883999999999998e-05 2024-04-23T16:22:43.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:35.997335+0000", 2024-04-23T16:22:43.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9190000000000002e-05 2024-04-23T16:22:43.627 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.627 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.627 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.627 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:116790 2.e 2:7e8b970c:::benchmark_data_smithi188_77627_object116789:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-23T16:22:43.627 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.402019+0000", 2024-04-23T16:22:43.627 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 245.04433342199999, 2024-04-23T16:22:43.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.100066103, 2024-04-23T16:22:43.627 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.627 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.627 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:43.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:43.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:43.628 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:43.628 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116790 2024-04-23T16:22:43.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.628 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402019+0000", 2024-04-23T16:22:43.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402019+0000", 2024-04-23T16:22:43.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402020+0000", 2024-04-23T16:22:43.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.885e-06 2024-04-23T16:22:43.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402047+0000", 2024-04-23T16:22:43.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6883e-05 2024-04-23T16:22:43.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402057+0000", 2024-04-23T16:22:43.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6690000000000001e-06 2024-04-23T16:22:43.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402063+0000", 2024-04-23T16:22:43.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0809999999999999e-06 2024-04-23T16:22:43.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402125+0000", 2024-04-23T16:22:43.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2100999999999994e-05 2024-04-23T16:22:43.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402250+0000", 2024-04-23T16:22:43.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000124552 2024-04-23T16:22:43.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.631 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:22:43.631 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402389+0000", 2024-04-23T16:22:43.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139049 2024-04-23T16:22:43.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.631 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:43.631 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494492+0000", 2024-04-23T16:22:43.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.092102772999999999 2024-04-23T16:22:43.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.631 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:43.631 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.501987+0000", 2024-04-23T16:22:43.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007495303 2024-04-23T16:22:43.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:43.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502040+0000", 2024-04-23T16:22:43.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3383e-05 2024-04-23T16:22:43.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502085+0000", 2024-04-23T16:22:43.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4424e-05 2024-04-23T16:22:43.632 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.632 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.632 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.633 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:116791 2.e 2:745fcdbe:::benchmark_data_smithi188_77627_object116790:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-23T16:22:43.633 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.402079+0000", 2024-04-23T16:22:43.633 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 245.04427342700001, 2024-04-23T16:22:43.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.10008534600000001, 2024-04-23T16:22:43.633 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.633 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.633 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:43.633 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:43.633 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:43.633 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:43.633 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116791 2024-04-23T16:22:43.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.634 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402079+0000", 2024-04-23T16:22:43.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402079+0000", 2024-04-23T16:22:43.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402079+0000", 2024-04-23T16:22:43.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.89e-07 2024-04-23T16:22:43.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402202+0000", 2024-04-23T16:22:43.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000122843 2024-04-23T16:22:43.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402208+0000", 2024-04-23T16:22:43.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5039999999999999e-06 2024-04-23T16:22:43.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402214+0000", 2024-04-23T16:22:43.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5559999999999998e-06 2024-04-23T16:22:43.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.636 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.636 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.404848+0000", 2024-04-23T16:22:43.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026339369999999998 2024-04-23T16:22:43.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.636 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.636 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.404935+0000", 2024-04-23T16:22:43.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6588999999999994e-05 2024-04-23T16:22:43.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.636 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:22:43.636 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.405030+0000", 2024-04-23T16:22:43.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5508999999999996e-05 2024-04-23T16:22:43.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:43.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494530+0000", 2024-04-23T16:22:43.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.089499971999999997 2024-04-23T16:22:43.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:43.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502094+0000", 2024-04-23T16:22:43.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075642790000000001 2024-04-23T16:22:43.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:43.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502134+0000", 2024-04-23T16:22:43.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9814000000000002e-05 2024-04-23T16:22:43.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502164+0000", 2024-04-23T16:22:43.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9853999999999999e-05 2024-04-23T16:22:43.638 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.638 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.638 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.638 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:116793 2.e 2:7bd7d2a5:::benchmark_data_smithi188_77627_object116792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-23T16:22:43.638 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.402222+0000", 2024-04-23T16:22:43.639 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 245.044130497, 2024-04-23T16:22:43.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.10001112299999999, 2024-04-23T16:22:43.639 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.639 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.639 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:43.639 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:43.639 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:43.639 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:43.639 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116793 2024-04-23T16:22:43.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.639 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402222+0000", 2024-04-23T16:22:43.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402222+0000", 2024-04-23T16:22:43.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402222+0000", 2024-04-23T16:22:43.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9900000000000001e-07 2024-04-23T16:22:43.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402249+0000", 2024-04-23T16:22:43.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7432000000000001e-05 2024-04-23T16:22:43.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402255+0000", 2024-04-23T16:22:43.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2240000000000003e-06 2024-04-23T16:22:43.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402258+0000", 2024-04-23T16:22:43.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3450000000000002e-06 2024-04-23T16:22:43.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407377+0000", 2024-04-23T16:22:43.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051189169999999997 2024-04-23T16:22:43.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407489+0000", 2024-04-23T16:22:43.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112452 2024-04-23T16:22:43.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:22:43.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407588+0000", 2024-04-23T16:22:43.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8595e-05 2024-04-23T16:22:43.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:43.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494565+0000", 2024-04-23T16:22:43.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.086977445 2024-04-23T16:22:43.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:43.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502170+0000", 2024-04-23T16:22:43.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076041119999999997 2024-04-23T16:22:43.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:43.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502205+0000", 2024-04-23T16:22:43.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5193e-05 2024-04-23T16:22:43.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502233+0000", 2024-04-23T16:22:43.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7909e-05 2024-04-23T16:22:43.644 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.644 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.644 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.644 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:116795 2.e 2:79edba58:::benchmark_data_smithi188_77627_object116794:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-23T16:22:43.644 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.402263+0000", 2024-04-23T16:22:43.644 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 245.04408891, 2024-04-23T16:22:43.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.10041691, 2024-04-23T16:22:43.644 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.644 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.644 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:43.644 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:43.645 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:43.645 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:43.645 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116795 2024-04-23T16:22:43.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.645 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402263+0000", 2024-04-23T16:22:43.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402263+0000", 2024-04-23T16:22:43.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402264+0000", 2024-04-23T16:22:43.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2199999999999999e-07 2024-04-23T16:22:43.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402356+0000", 2024-04-23T16:22:43.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2485000000000006e-05 2024-04-23T16:22:43.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402361+0000", 2024-04-23T16:22:43.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.434e-06 2024-04-23T16:22:43.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.402367+0000", 2024-04-23T16:22:43.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9290000000000003e-06 2024-04-23T16:22:43.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494594+0000", 2024-04-23T16:22:43.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.092226630000000004 2024-04-23T16:22:43.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494699+0000", 2024-04-23T16:22:43.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010501399999999999 2024-04-23T16:22:43.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:22:43.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494851+0000", 2024-04-23T16:22:43.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015200399999999999 2024-04-23T16:22:43.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:43.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502238+0000", 2024-04-23T16:22:43.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0073872599999999997 2024-04-23T16:22:43.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:43.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502586+0000", 2024-04-23T16:22:43.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00034732900000000001 2024-04-23T16:22:43.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:43.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502631+0000", 2024-04-23T16:22:43.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5120999999999997e-05 2024-04-23T16:22:43.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502680+0000", 2024-04-23T16:22:43.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9282e-05 2024-04-23T16:22:43.649 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.649 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.649 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.650 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop_reply(client.14862.0:116790 2.e e131/110 ondisk, result = 0)", 2024-04-23T16:22:43.650 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.405073+0000", 2024-04-23T16:22:43.650 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 245.04127926199999, 2024-04-23T16:22:43.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.089426004000000003, 2024-04-23T16:22:43.650 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.650 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.650 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:22:43.650 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.405073+0000", 2024-04-23T16:22:43.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.405073+0000", 2024-04-23T16:22:43.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.405074+0000", 2024-04-23T16:22:43.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4169999999999999e-06 2024-04-23T16:22:43.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.405079+0000", 2024-04-23T16:22:43.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.198e-06 2024-04-23T16:22:43.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.405088+0000", 2024-04-23T16:22:43.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2619999999999998e-06 2024-04-23T16:22:43.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.405095+0000", 2024-04-23T16:22:43.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9439999999999999e-06 2024-04-23T16:22:43.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494479+0000", 2024-04-23T16:22:43.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.089384384999999997 2024-04-23T16:22:43.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494486+0000", 2024-04-23T16:22:43.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9970000000000001e-06 2024-04-23T16:22:43.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494499+0000", 2024-04-23T16:22:43.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2801e-05 2024-04-23T16:22:43.653 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.654 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.654 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.654 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:116799 2.1 2:839676f0:::benchmark_data_smithi188_77627_object116798:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-23T16:22:43.654 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.406915+0000", 2024-04-23T16:22:43.654 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 245.039436843, 2024-04-23T16:22:43.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.087679254999999998, 2024-04-23T16:22:43.654 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.654 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.654 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:43.654 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:43.654 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:43.655 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:43.655 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116799 2024-04-23T16:22:43.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.655 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.406915+0000", 2024-04-23T16:22:43.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.406915+0000", 2024-04-23T16:22:43.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.406917+0000", 2024-04-23T16:22:43.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0219999999999999e-06 2024-04-23T16:22:43.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.406975+0000", 2024-04-23T16:22:43.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7541999999999999e-05 2024-04-23T16:22:43.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.406984+0000", 2024-04-23T16:22:43.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1959999999999997e-06 2024-04-23T16:22:43.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.406995+0000", 2024-04-23T16:22:43.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0864e-05 2024-04-23T16:22:43.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407037+0000", 2024-04-23T16:22:43.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2375e-05 2024-04-23T16:22:43.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407120+0000", 2024-04-23T16:22:43.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2659999999999998e-05 2024-04-23T16:22:43.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:22:43.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407226+0000", 2024-04-23T16:22:43.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000106579 2024-04-23T16:22:43.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:43.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.490902+0000", 2024-04-23T16:22:43.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.083675362000000003 2024-04-23T16:22:43.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:43.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494487+0000", 2024-04-23T16:22:43.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035854189999999998 2024-04-23T16:22:43.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:43.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494547+0000", 2024-04-23T16:22:43.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9864999999999997e-05 2024-04-23T16:22:43.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494594+0000", 2024-04-23T16:22:43.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7370999999999998e-05 2024-04-23T16:22:43.659 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.659 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.659 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.659 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:116801 2.1 2:89be7249:::benchmark_data_smithi188_77627_object116800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-23T16:22:43.660 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.407018+0000", 2024-04-23T16:22:43.660 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 245.039334182, 2024-04-23T16:22:43.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.095095130999999999, 2024-04-23T16:22:43.660 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.660 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.660 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:43.660 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:43.660 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:43.660 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:43.660 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116801 2024-04-23T16:22:43.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.660 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.660 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.660 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407018+0000", 2024-04-23T16:22:43.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407018+0000", 2024-04-23T16:22:43.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407019+0000", 2024-04-23T16:22:43.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2799999999999995e-07 2024-04-23T16:22:43.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407049+0000", 2024-04-23T16:22:43.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0757999999999999e-05 2024-04-23T16:22:43.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407054+0000", 2024-04-23T16:22:43.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6140000000000002e-06 2024-04-23T16:22:43.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407065+0000", 2024-04-23T16:22:43.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0797e-05 2024-04-23T16:22:43.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.490930+0000", 2024-04-23T16:22:43.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.083864994999999998 2024-04-23T16:22:43.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.491057+0000", 2024-04-23T16:22:43.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012708399999999999 2024-04-23T16:22:43.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:22:43.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.491201+0000", 2024-04-23T16:22:43.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014382899999999999 2024-04-23T16:22:43.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:43.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.495451+0000", 2024-04-23T16:22:43.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042499419999999996 2024-04-23T16:22:43.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:43.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502029+0000", 2024-04-23T16:22:43.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0065782059999999996 2024-04-23T16:22:43.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:43.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502074+0000", 2024-04-23T16:22:43.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5132000000000001e-05 2024-04-23T16:22:43.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.665 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.665 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502113+0000", 2024-04-23T16:22:43.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8946000000000003e-05 2024-04-23T16:22:43.665 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.665 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.665 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.665 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:116802 2.e 2:7ce58b94:::benchmark_data_smithi188_77627_object116801:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-23T16:22:43.665 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.407074+0000", 2024-04-23T16:22:43.665 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 245.03927785400001, 2024-04-23T16:22:43.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.095874181000000003, 2024-04-23T16:22:43.666 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.666 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.666 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:43.666 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:43.666 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:43.666 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:43.666 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116802 2024-04-23T16:22:43.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.666 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407074+0000", 2024-04-23T16:22:43.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407074+0000", 2024-04-23T16:22:43.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407075+0000", 2024-04-23T16:22:43.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7999999999999996e-07 2024-04-23T16:22:43.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407098+0000", 2024-04-23T16:22:43.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3476999999999999e-05 2024-04-23T16:22:43.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407102+0000", 2024-04-23T16:22:43.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0820000000000001e-06 2024-04-23T16:22:43.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407106+0000", 2024-04-23T16:22:43.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9720000000000003e-06 2024-04-23T16:22:43.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.498268+0000", 2024-04-23T16:22:43.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.091162033000000003 2024-04-23T16:22:43.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.498414+0000", 2024-04-23T16:22:43.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000146062 2024-04-23T16:22:43.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:22:43.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.498616+0000", 2024-04-23T16:22:43.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020167899999999999 2024-04-23T16:22:43.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:43.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502791+0000", 2024-04-23T16:22:43.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041754560000000001 2024-04-23T16:22:43.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.670 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:43.670 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502835+0000", 2024-04-23T16:22:43.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3497e-05 2024-04-23T16:22:43.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.670 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:43.670 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502882+0000", 2024-04-23T16:22:43.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7104000000000002e-05 2024-04-23T16:22:43.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.670 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.670 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.502948+0000", 2024-04-23T16:22:43.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6339000000000003e-05 2024-04-23T16:22:43.671 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.671 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.671 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.671 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop_reply(client.14862.0:116791 2.e e131/110 ondisk, result = 0)", 2024-04-23T16:22:43.671 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.407574+0000", 2024-04-23T16:22:43.671 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 245.038778084, 2024-04-23T16:22:43.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.086961826000000006, 2024-04-23T16:22:43.671 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.671 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.671 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:22:43.671 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.671 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407574+0000", 2024-04-23T16:22:43.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407574+0000", 2024-04-23T16:22:43.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407575+0000", 2024-04-23T16:22:43.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1739999999999999e-06 2024-04-23T16:22:43.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407579+0000", 2024-04-23T16:22:43.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.048e-06 2024-04-23T16:22:43.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407587+0000", 2024-04-23T16:22:43.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3660000000000003e-06 2024-04-23T16:22:43.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.407591+0000", 2024-04-23T16:22:43.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9219999999999996e-06 2024-04-23T16:22:43.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494520+0000", 2024-04-23T16:22:43.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.086928064999999999 2024-04-23T16:22:43.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494525+0000", 2024-04-23T16:22:43.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7540000000000003e-06 2024-04-23T16:22:43.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494536+0000", 2024-04-23T16:22:43.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0497e-05 2024-04-23T16:22:43.675 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.675 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.675 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.675 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop_reply(client.14862.0:116799 2.1 e131/110 ondisk, result = 0)", 2024-04-23T16:22:43.675 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:18:37.409816+0000", 2024-04-23T16:22:43.675 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 245.03653607000001, 2024-04-23T16:22:43.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.084785458999999994, 2024-04-23T16:22:43.675 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.675 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.675 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:22:43.676 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.409816+0000", 2024-04-23T16:22:43.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.409816+0000", 2024-04-23T16:22:43.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.409817+0000", 2024-04-23T16:22:43.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.469e-06 2024-04-23T16:22:43.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.409822+0000", 2024-04-23T16:22:43.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3409999999999996e-06 2024-04-23T16:22:43.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.491187+0000", 2024-04-23T16:22:43.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.081364991999999997 2024-04-23T16:22:43.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.491196+0000", 2024-04-23T16:22:43.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0610000000000002e-06 2024-04-23T16:22:43.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494473+0000", 2024-04-23T16:22:43.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003276953 2024-04-23T16:22:43.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494481+0000", 2024-04-23T16:22:43.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4379999999999995e-06 2024-04-23T16:22:43.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:18:37.494601+0000", 2024-04-23T16:22:43.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012020500000000001 2024-04-23T16:22:43.679 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.679 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.679 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.679 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:212472 2.e 2:7a958dd0:::benchmark_data_smithi188_77627_object43123:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-04-23T16:22:43.679 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:58.704328+0000", 2024-04-23T16:22:43.679 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 163.742023657, 2024-04-23T16:22:43.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.059670517999999999, 2024-04-23T16:22:43.679 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.679 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.680 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:43.680 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:43.680 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:43.680 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:43.680 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 212472 2024-04-23T16:22:43.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.680 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.704328+0000", 2024-04-23T16:22:43.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.704328+0000", 2024-04-23T16:22:43.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.704330+0000", 2024-04-23T16:22:43.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0660000000000002e-06 2024-04-23T16:22:43.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.704337+0000", 2024-04-23T16:22:43.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4989999999999999e-06 2024-04-23T16:22:43.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.704352+0000", 2024-04-23T16:22:43.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4953999999999999e-05 2024-04-23T16:22:43.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.704360+0000", 2024-04-23T16:22:43.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4200000000000007e-06 2024-04-23T16:22:43.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.711852+0000", 2024-04-23T16:22:43.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0074914020000000003 2024-04-23T16:22:43.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.711942+0000", 2024-04-23T16:22:43.683 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0062000000000006e-05 2024-04-23T16:22:43.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.683 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.683 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:22:43.683 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.712023+0000", 2024-04-23T16:22:43.683 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0877000000000005e-05 2024-04-23T16:22:43.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.683 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.683 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:43.683 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.739726+0000", 2024-04-23T16:22:43.683 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.027703425 2024-04-23T16:22:43.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.683 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.683 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:43.683 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.763821+0000", 2024-04-23T16:22:43.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024094477 2024-04-23T16:22:43.684 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:43.684 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.763868+0000", 2024-04-23T16:22:43.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7851e-05 2024-04-23T16:22:43.684 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.684 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.763999+0000", 2024-04-23T16:22:43.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013048500000000001 2024-04-23T16:22:43.684 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.684 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.684 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.685 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:212474 2.e 2:7374480d:::benchmark_data_smithi188_77627_object43125:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-04-23T16:22:43.685 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:58.704380+0000", 2024-04-23T16:22:43.685 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 163.741971842, 2024-04-23T16:22:43.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.059703842999999999, 2024-04-23T16:22:43.685 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.685 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.685 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:43.685 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:43.685 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:43.685 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:43.685 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 212474 2024-04-23T16:22:43.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.686 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.704380+0000", 2024-04-23T16:22:43.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.704380+0000", 2024-04-23T16:22:43.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.704381+0000", 2024-04-23T16:22:43.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6899999999999996e-07 2024-04-23T16:22:43.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.704385+0000", 2024-04-23T16:22:43.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6200000000000001e-06 2024-04-23T16:22:43.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.704390+0000", 2024-04-23T16:22:43.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4450000000000004e-06 2024-04-23T16:22:43.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.688 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.688 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.688 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.704395+0000", 2024-04-23T16:22:43.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9099999999999996e-06 2024-04-23T16:22:43.688 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.688 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.688 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.688 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.716177+0000", 2024-04-23T16:22:43.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011781979999999999 2024-04-23T16:22:43.688 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.688 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.688 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.688 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.716260+0000", 2024-04-23T16:22:43.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2945e-05 2024-04-23T16:22:43.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.689 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.689 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:22:43.689 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.716345+0000", 2024-04-23T16:22:43.689 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4815999999999996e-05 2024-04-23T16:22:43.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.689 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.689 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:43.689 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.739749+0000", 2024-04-23T16:22:43.689 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.023404541000000001 2024-04-23T16:22:43.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.689 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.689 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:43.689 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.764007+0000", 2024-04-23T16:22:43.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024258156999999999 2024-04-23T16:22:43.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:43.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.764043+0000", 2024-04-23T16:22:43.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6121000000000002e-05 2024-04-23T16:22:43.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.764084+0000", 2024-04-23T16:22:43.691 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0538999999999999e-05 2024-04-23T16:22:43.691 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.691 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.691 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.691 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:212478 2.e 2:71586d03:::benchmark_data_smithi188_77627_object43129:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-04-23T16:22:43.691 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:19:58.704431+0000", 2024-04-23T16:22:43.691 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 163.74192089900001, 2024-04-23T16:22:43.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.059732847999999998, 2024-04-23T16:22:43.692 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.692 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.692 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:43.692 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:43.692 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:43.692 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:43.692 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 212478 2024-04-23T16:22:43.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.692 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.692 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.704431+0000", 2024-04-23T16:22:43.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.704431+0000", 2024-04-23T16:22:43.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.704432+0000", 2024-04-23T16:22:43.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2399999999999998e-07 2024-04-23T16:22:43.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.694 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.704434+0000", 2024-04-23T16:22:43.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7209999999999999e-06 2024-04-23T16:22:43.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.694 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.694 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.704440+0000", 2024-04-23T16:22:43.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1080000000000001e-06 2024-04-23T16:22:43.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.694 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.694 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.704444+0000", 2024-04-23T16:22:43.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8749999999999999e-06 2024-04-23T16:22:43.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.695 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.695 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.695 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.720246+0000", 2024-04-23T16:22:43.695 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015801042000000001 2024-04-23T16:22:43.695 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.695 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.695 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.695 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.720371+0000", 2024-04-23T16:22:43.695 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012508899999999999 2024-04-23T16:22:43.695 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.695 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.695 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:22:43.695 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.720575+0000", 2024-04-23T16:22:43.695 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020476399999999999 2024-04-23T16:22:43.695 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:43.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.739772+0000", 2024-04-23T16:22:43.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019197064999999999 2024-04-23T16:22:43.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:43.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.764090+0000", 2024-04-23T16:22:43.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024317121000000001 2024-04-23T16:22:43.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:43.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.764123+0000", 2024-04-23T16:22:43.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3862999999999997e-05 2024-04-23T16:22:43.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.697 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:19:58.764164+0000", 2024-04-23T16:22:43.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0575999999999999e-05 2024-04-23T16:22:43.697 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.697 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.697 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.697 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.14862.0:217226 2.1 2:8029bfe0:::benchmark_data_smithi188_77627_object47877:head [delete] snapc 0=[] ondisk+write+known_if_redirected e153)", 2024-04-23T16:22:43.697 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:20:02.688546+0000", 2024-04-23T16:22:43.698 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 159.75780570800001, 2024-04-23T16:22:43.698 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.058182987999999998, 2024-04-23T16:22:43.698 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.698 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.698 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:43.698 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:22:43.698 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.14862", 2024-04-23T16:22:43.698 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:22:43.698 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 217226 2024-04-23T16:22:43.698 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.698 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.698 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.698 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.698 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:02.688546+0000", 2024-04-23T16:22:43.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:02.688546+0000", 2024-04-23T16:22:43.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:02.688547+0000", 2024-04-23T16:22:43.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3200000000000005e-07 2024-04-23T16:22:43.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:02.688549+0000", 2024-04-23T16:22:43.700 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1749999999999999e-06 2024-04-23T16:22:43.700 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.700 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.700 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.700 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:02.688554+0000", 2024-04-23T16:22:43.700 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5399999999999997e-06 2024-04-23T16:22:43.700 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.700 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.700 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.700 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:02.688558+0000", 2024-04-23T16:22:43.700 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7530000000000001e-06 2024-04-23T16:22:43.700 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.700 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.700 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.701 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:02.705887+0000", 2024-04-23T16:22:43.701 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017328393000000001 2024-04-23T16:22:43.701 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.701 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.701 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.701 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:02.706017+0000", 2024-04-23T16:22:43.701 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000130586 2024-04-23T16:22:43.701 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.701 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.701 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:22:43.701 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:02.706145+0000", 2024-04-23T16:22:43.701 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012736400000000001 2024-04-23T16:22:43.701 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.701 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.701 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:22:43.702 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:02.726961+0000", 2024-04-23T16:22:43.702 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020816395000000001 2024-04-23T16:22:43.702 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.702 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.702 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:22:43.702 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:02.746616+0000", 2024-04-23T16:22:43.702 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019654476000000001 2024-04-23T16:22:43.702 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.702 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.702 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:43.702 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:02.746672+0000", 2024-04-23T16:22:43.702 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6523000000000001e-05 2024-04-23T16:22:43.702 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.703 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.703 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:02.746729+0000", 2024-04-23T16:22:43.703 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7250999999999998e-05 2024-04-23T16:22:43.703 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.703 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.703 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.703 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.703 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:269181 2.3 e168/110 2:c3209f67:::benchmark_data_smithi188_77627_object99832:head v 168'16692, mlcod=168'16690)", 2024-04-23T16:22:43.703 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:20:49.556842+0000", 2024-04-23T16:22:43.703 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 112.88950978699999, 2024-04-23T16:22:43.703 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.070806754999999999, 2024-04-23T16:22:43.703 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.704 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.704 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:43.704 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.704 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.704 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.704 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.556842+0000", 2024-04-23T16:22:43.704 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.704 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.704 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.704 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.704 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.556842+0000", 2024-04-23T16:22:43.704 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.704 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.704 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.705 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.556843+0000", 2024-04-23T16:22:43.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2470000000000001e-06 2024-04-23T16:22:43.705 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.705 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.705 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.556850+0000", 2024-04-23T16:22:43.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.55e-06 2024-04-23T16:22:43.705 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.705 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.705 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.556859+0000", 2024-04-23T16:22:43.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3139999999999993e-06 2024-04-23T16:22:43.705 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.705 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.706 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.706 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.556866+0000", 2024-04-23T16:22:43.706 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.72e-06 2024-04-23T16:22:43.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.706 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.706 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.706 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.556910+0000", 2024-04-23T16:22:43.706 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4044e-05 2024-04-23T16:22:43.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.706 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.706 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.706 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.556929+0000", 2024-04-23T16:22:43.707 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9258000000000001e-05 2024-04-23T16:22:43.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.707 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:43.707 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.627603+0000", 2024-04-23T16:22:43.707 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.070673949999999999 2024-04-23T16:22:43.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.707 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.707 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.627649+0000", 2024-04-23T16:22:43.707 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5671999999999999e-05 2024-04-23T16:22:43.708 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.708 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.708 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.708 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.708 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:269184 2.3 e168/110 2:c03ebee3:::benchmark_data_smithi188_77627_object99835:head v 168'16693, mlcod=168'16690)", 2024-04-23T16:22:43.708 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:20:49.560770+0000", 2024-04-23T16:22:43.708 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 112.88558168500001, 2024-04-23T16:22:43.708 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.06690952, 2024-04-23T16:22:43.708 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:22:43.708 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:22:43.708 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:22:43.708 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:22:43.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:22:43.709 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.560770+0000", 2024-04-23T16:22:43.709 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:22:43.709 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.560770+0000", 2024-04-23T16:22:43.709 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:22:43.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:22:43.709 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.560772+0000", 2024-04-23T16:22:43.709 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2190000000000001e-06 2024-04-23T16:22:43.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:22:43.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.560779+0000", 2024-04-23T16:22:43.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1319999999999996e-06 2024-04-23T16:22:43.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:22:43.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.601661+0000", 2024-04-23T16:22:43.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.040882762000000003 2024-04-23T16:22:43.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:22:43.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.601668+0000", 2024-04-23T16:22:43.711 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0499999999999997e-06 2024-04-23T16:22:43.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.711 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:22:43.711 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.602327+0000", 2024-04-23T16:22:43.711 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00065981500000000001 2024-04-23T16:22:43.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.711 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:22:43.711 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.602729+0000", 2024-04-23T16:22:43.711 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040181599999999999 2024-04-23T16:22:43.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.711 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:22:43.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.627652+0000", 2024-04-23T16:22:43.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024922564000000001 2024-04-23T16:22:43.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:22:43.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:22:43.712 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:22:43.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.627680+0000", 2024-04-23T16:22:43.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8161999999999999e-05 2024-04-23T16:22:43.712 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.712 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.712 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.712 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:22:43.712 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:22:43.712 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:22:43.713 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:43 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:22:43.713 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:43 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:22:43.714 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:43 smithi148 ceph-mon[44520]: osdmap e204: 12 total, 12 up, 11 in 2024-04-23T16:22:43.714 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:43 smithi148 ceph-mon[44520]: pgmap v598: 17 pgs: 17 active+clean; 3.2 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 162 MiB/s wr, 2.59k op/s 2024-04-23T16:22:44.474 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 18 16 64985 64969 225.569 145.875 0.00658985 0.00443155 2024-04-23T16:22:45.474 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 19 15 67388 67373 221.605 150.25 0.00913274 0.0045113 2024-04-23T16:22:45.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:45 smithi148 ceph-mon[44520]: pgmap v599: 17 pgs: 17 active+clean; 3.6 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 183 MiB/s wr, 2.93k op/s 2024-04-23T16:22:45.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:45 smithi188 ceph-mon[44101]: pgmap v599: 17 pgs: 17 active+clean; 3.6 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 183 MiB/s wr, 2.93k op/s 2024-04-23T16:22:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:45 smithi103 ceph-mon[39629]: pgmap v599: 17 pgs: 17 active+clean; 3.6 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 183 MiB/s wr, 2.93k op/s 2024-04-23T16:22:46.474 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:22:46.474191 min lat: 0.00099496 max lat: 0.0996851 avg lat: 0.0045842 2024-04-23T16:22:46.474 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:22:46.474 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 20 16 69802 69786 218.064 150.812 0.00270759 0.0045842 2024-04-23T16:22:47.149 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:47.152 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 9 1 2024-04-23T16:22:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:47.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:47.401 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:47.464 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:47.474 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 21 16 71998 71982 214.215 137.25 0.0115777 0.00466629 2024-04-23T16:22:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:47 smithi103 ceph-mon[39629]: pgmap v600: 17 pgs: 17 active+clean; 4.0 GiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 188 MiB/s wr, 3.01k op/s 2024-04-23T16:22:47.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:47 smithi148 ceph-mon[44520]: pgmap v600: 17 pgs: 17 active+clean; 4.0 GiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 188 MiB/s wr, 3.01k op/s 2024-04-23T16:22:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:47 smithi188 ceph-mon[44101]: pgmap v600: 17 pgs: 17 active+clean; 4.0 GiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 188 MiB/s wr, 3.01k op/s 2024-04-23T16:22:48.474 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 22 16 74255 74239 210.89 141.062 0.0070643 0.00474012 2024-04-23T16:22:49.436 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:49.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:49.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:49.474 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 23 16 76471 76455 207.742 138.5 0.0180069 0.0048117 2024-04-23T16:22:49.542 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:22:49.578 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:49.604 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:22:49.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:49.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:22:49.701 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:49 smithi103 ceph-mon[39629]: pgmap v601: 17 pgs: 17 active+clean; 4.2 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 163 MiB/s wr, 2.60k op/s 2024-04-23T16:22:49.708 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:49.749 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:49.804 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:49.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:49.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:49.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:49 smithi148 ceph-mon[44520]: pgmap v601: 17 pgs: 17 active+clean; 4.2 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 163 MiB/s wr, 2.60k op/s 2024-04-23T16:22:49.909 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:49.951 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:49.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:49 smithi188 ceph-mon[44101]: pgmap v601: 17 pgs: 17 active+clean; 4.2 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 163 MiB/s wr, 2.60k op/s 2024-04-23T16:22:50.474 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 24 16 78705 78689 204.903 139.625 0.00495881 0.00487893 2024-04-23T16:22:50.614 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:50.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:50.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:50.719 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:22:50.766 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:50.790 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:22:50.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:50.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:22:50.897 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:22:50.937 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:50.969 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:22:50.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:50.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:22:50.987 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:22:51.011 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-23T16:22:51.026 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:22:51.079 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:22:51.111 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:51.134 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:22:51.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:51.139 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:22:51.240 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:22:51.272 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:51.294 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:22:51.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:51.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:22:51.399 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:22:51.426 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:51.449 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:22:51.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:51.453 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:22:51.474 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 25 15 80617 80602 201.489 119.562 0.00657531 0.00496145 2024-04-23T16:22:51.554 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:22:51.591 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:51.614 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:22:51.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:51.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:22:51.720 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:22:51.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:51 smithi188 ceph-mon[44101]: pgmap v602: 17 pgs: 17 active+clean; 4.4 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 161 MiB/s wr, 2.57k op/s 2024-04-23T16:22:51.750 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:51.776 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:22:51.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:51.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:22:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:51 smithi103 ceph-mon[39629]: pgmap v602: 17 pgs: 17 active+clean; 4.4 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 161 MiB/s wr, 2.57k op/s 2024-04-23T16:22:51.867 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:22:51.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:51 smithi148 ceph-mon[44520]: pgmap v602: 17 pgs: 17 active+clean; 4.4 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 161 MiB/s wr, 2.57k op/s 2024-04-23T16:22:51.882 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:22:51.894 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:22:51.927 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:22:51.986 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:22:52.015 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:22:52.042 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:22:52.067 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:22:52.091 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:22:52.112 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:22:52.135 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:22:52.154 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:22:52.175 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:22:52.176 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:22:52.176 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:22:52.176 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:22:52.176 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:22:52.176 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:22:52.176 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:22:52.176 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:22:52.177 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:22:52.177 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:22:52.177 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:22:52.177 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:22:52.177 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:22:52.303 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:22:52.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:52.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:22:52.408 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:22:52.455 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:52.474 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 26 16 82373 82357 197.958 109.688 0.00613693 0.00505001 2024-04-23T16:22:52.814 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:22:52.821 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:52 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4182749207' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:22:52.822 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.822 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.822 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.822 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.822 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.822 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.822 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.822 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:52.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:22:52.924 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:22:52.957 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:52 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4182749207' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:22:52.957 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.958 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.958 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.958 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.958 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.958 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.958 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.958 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.959 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:52.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:52 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4182749207' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:22:52.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:52.990 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:22:52.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:52.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:22:53.094 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:22:53.136 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:53.161 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:22:53.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:53.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:22:53.267 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:53.316 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:53.438 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:53.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:53.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:53.474 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 27 16 84450 84434 195.434 129.812 0.00860476 0.00511537 2024-04-23T16:22:53.545 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:22:53.576 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:53.599 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:22:53.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:53.603 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:22:53.704 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:22:53.737 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:53.761 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:22:53.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:53.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:22:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:53 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:53 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:53 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:53 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:53 smithi103 ceph-mon[39629]: pgmap v603: 17 pgs: 17 active+clean; 4.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-04-23T16:22:53.868 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:22:53.902 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:53 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:53.902 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:53 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:53.902 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:53 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:53.902 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:53 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:53.902 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:53 smithi188 ceph-mon[44101]: pgmap v603: 17 pgs: 17 active+clean; 4.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-04-23T16:22:53.916 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:53.939 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:22:53.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:53.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:22:54.044 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:54.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:53 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:54.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:53 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:54.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:53 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:54.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:53 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:22:54.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:53 smithi148 ceph-mon[44520]: pgmap v603: 17 pgs: 17 active+clean; 4.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-04-23T16:22:54.126 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:54.334 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:54.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:54.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:54.438 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:22:54.474 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 28 16 86489 86473 193.005 127.438 0.00642534 0.00517967 2024-04-23T16:22:54.476 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:54.836 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:22:54.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:54.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:22:54.940 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:22:54.992 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:55.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:54 smithi103 ceph-mon[39629]: pgmap v604: 17 pgs: 17 active+clean; 5.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 152 MiB/s wr, 2.43k op/s 2024-04-23T16:22:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:54 smithi188 ceph-mon[44101]: pgmap v604: 17 pgs: 17 active+clean; 5.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 152 MiB/s wr, 2.43k op/s 2024-04-23T16:22:55.289 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:22:55.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:55.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:22:55.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:54 smithi148 ceph-mon[44520]: pgmap v604: 17 pgs: 17 active+clean; 5.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 152 MiB/s wr, 2.43k op/s 2024-04-23T16:22:55.397 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:22:55.427 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:55.449 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:22:55.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:55.453 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:22:55.475 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 29 16 88448 88432 190.571 122.438 0.0140771 0.00524597 2024-04-23T16:22:55.553 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:22:55.584 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:55.612 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:22:55.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:55.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:22:55.717 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:22:55.748 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:55.772 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:22:55.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:55.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:22:55.876 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:22:55.938 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:56.020 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:22:56.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:56.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:22:56.047 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:22:56.124 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:22:56.158 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:56.184 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:22:56.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:56.188 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:22:56.276 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:56 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1886189127' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:22:56.276 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:56 smithi103 ceph-mon[39629]: from='client.25066 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:22:56.289 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:22:56.297 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.9 to 1 (10000) 2024-04-23T16:22:56.323 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:56.351 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:22:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:56.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:22:56.457 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:22:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:56 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1886189127' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:22:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:56 smithi188 ceph-mon[44101]: from='client.25066 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:22:56.475 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 30 16 90159 90143 187.783 106.938 0.0118488 0.00532334 2024-04-23T16:22:56.501 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:56.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:56 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1886189127' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:22:56.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:56 smithi148 ceph-mon[44520]: from='client.25066 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:22:56.652 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:22:56.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:56.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:22:56.760 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:22:56.808 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:56.837 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:22:56.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:56.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:22:56.946 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:22:56.982 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:57.013 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:22:57.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:57.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:22:57.059 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:22:57.121 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:22:57.165 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:57.191 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:22:57.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:57.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:22:57.299 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:22:57.332 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:22:57.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:57 smithi103 ceph-mon[39629]: from='client.25066 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:22:57.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:57 smithi103 ceph-mon[39629]: osdmap e205: 12 total, 12 up, 11 in 2024-04-23T16:22:57.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:22:57.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:57 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3391229878' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:22:57.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:57 smithi103 ceph-mon[39629]: from='client.25072 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:22:57.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:57 smithi103 ceph-mon[39629]: pgmap v606: 17 pgs: 17 active+clean; 5.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-04-23T16:22:57.355 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:57.475 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 31 16 91977 91961 185.391 113.625 0.00282392 0.00539256 2024-04-23T16:22:57.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:57 smithi148 ceph-mon[44520]: from='client.25066 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:22:57.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:57 smithi148 ceph-mon[44520]: osdmap e205: 12 total, 12 up, 11 in 2024-04-23T16:22:57.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:57 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:22:57.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:57 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3391229878' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:22:57.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:57 smithi148 ceph-mon[44520]: from='client.25072 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:22:57.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:57 smithi148 ceph-mon[44520]: pgmap v606: 17 pgs: 17 active+clean; 5.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-04-23T16:22:57.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:57 smithi188 ceph-mon[44101]: from='client.25066 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:22:57.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:57 smithi188 ceph-mon[44101]: osdmap e205: 12 total, 12 up, 11 in 2024-04-23T16:22:57.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:57 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:22:57.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:57 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3391229878' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:22:57.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:57 smithi188 ceph-mon[44101]: from='client.25072 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:22:57.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:57 smithi188 ceph-mon[44101]: pgmap v606: 17 pgs: 17 active+clean; 5.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-04-23T16:22:57.819 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:22:57.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:57.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:22:57.865 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 11 1 2024-04-23T16:22:57.926 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:22:57.958 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:57.985 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:22:57.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:57.990 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:22:58.091 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:22:58.131 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:58.473 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:58 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:22:58.473 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:58 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:22:58.473 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:58 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:22:58.473 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:58 smithi103 ceph-mon[39629]: from='client.25072 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:22:58.473 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:58 smithi103 ceph-mon[39629]: osdmap e206: 12 total, 12 up, 11 in 2024-04-23T16:22:58.475 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 32 16 93876 93860 183.306 118.688 0.00305209 0.00545395 2024-04-23T16:22:58.495 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:22:58.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:58.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:22:58.602 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:22:58.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:58 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:22:58.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:58 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:22:58.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:58 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:22:58.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:58 smithi148 ceph-mon[44520]: from='client.25072 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:22:58.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:58 smithi148 ceph-mon[44520]: osdmap e206: 12 total, 12 up, 11 in 2024-04-23T16:22:58.645 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:58.673 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:22:58.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:58.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:22:58.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:58 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:22:58.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:58 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:22:58.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:58 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:22:58.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:58 smithi188 ceph-mon[44101]: from='client.25072 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:22:58.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:58 smithi188 ceph-mon[44101]: osdmap e206: 12 total, 12 up, 11 in 2024-04-23T16:22:58.776 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:22:58.808 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:58.833 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:22:58.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:58.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:22:58.938 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:22:58.984 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:59.042 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:22:59.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:59.047 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:22:59.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:22:59.149 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:22:59.183 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:59.216 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:22:59.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:59.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:22:59.323 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:22:59.367 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:59.399 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:22:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:59.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:22:59.475 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 33 16 95623 95607 181.06 109.188 0.0153202 0.00552135 2024-04-23T16:22:59.507 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:22:59.538 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:59 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:22:59.538 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:59 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:22:59.538 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:59 smithi148 ceph-mon[44520]: osdmap e207: 12 total, 12 up, 11 in 2024-04-23T16:22:59.538 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:59 smithi148 ceph-mon[44520]: pgmap v609: 17 pgs: 17 active+clean; 5.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-04-23T16:22:59.539 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:22:59 smithi148 ceph-mon[44520]: osdmap e208: 12 total, 12 up, 11 in 2024-04-23T16:22:59.542 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:59.580 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:22:59.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:59.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:22:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:59 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:22:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:59 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:22:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:59 smithi103 ceph-mon[39629]: osdmap e207: 12 total, 12 up, 11 in 2024-04-23T16:22:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:59 smithi103 ceph-mon[39629]: pgmap v609: 17 pgs: 17 active+clean; 5.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-04-23T16:22:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:22:59 smithi103 ceph-mon[39629]: osdmap e208: 12 total, 12 up, 11 in 2024-04-23T16:22:59.685 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:22:59.716 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:22:59.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:59 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:22:59.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:59 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:22:59.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:59 smithi188 ceph-mon[44101]: osdmap e207: 12 total, 12 up, 11 in 2024-04-23T16:22:59.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:59 smithi188 ceph-mon[44101]: pgmap v609: 17 pgs: 17 active+clean; 5.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-04-23T16:22:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:22:59 smithi188 ceph-mon[44101]: osdmap e208: 12 total, 12 up, 11 in 2024-04-23T16:22:59.747 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:22:59.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:22:59.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:22:59.853 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:22:59.906 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:00.125 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:23:00.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:00.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:23:00.232 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:23:00.286 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:00.293 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:23:00.293 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:23:00.475 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 34 16 97422 97406 179.041 112.438 0.00386617 0.0055838 2024-04-23T16:23:00.493 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:23:00.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:00.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:23:00.600 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:23:00.631 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:00.660 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:23:00.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:00.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:23:00.764 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:23:00.799 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:00.837 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:23:00.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:00.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:23:00.942 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:00.977 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:01.008 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:01.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:01.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:01.030 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-23T16:23:01.095 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:23:01.111 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:23:01.230 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:01.441 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:23:01.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:01.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:23:01.475 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 35 16 99153 99137 177.017 108.188 0.00616526 0.00564787 2024-04-23T16:23:01.547 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:23:01.588 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:01.610 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:23:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:01.613 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:23:01.715 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:23:01.746 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:01.777 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:23:01.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:01.782 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:23:01.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:01 smithi103 ceph-mon[39629]: pgmap v611: 17 pgs: 17 active+clean; 5.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-04-23T16:23:01.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:01 smithi148 ceph-mon[44520]: pgmap v611: 17 pgs: 17 active+clean; 5.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-04-23T16:23:01.883 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:23:01.912 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:01.933 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:23:01.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:01.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:23:01.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:01 smithi188 ceph-mon[44101]: pgmap v611: 17 pgs: 17 active+clean; 5.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-04-23T16:23:02.037 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:23:02.071 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:02.094 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:23:02.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:02.099 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:23:02.200 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:23:02.247 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:02.475 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 36 16 100943 100927 175.207 111.875 0.0116414 0.00570598 2024-04-23T16:23:02.578 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:23:03.065 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:23:03.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:03.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:23:03.174 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:23:03.230 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:03.236 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:23:03.237 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:23:03.237 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:23:03.237 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:23:03.237 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:23:03.237 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:23:03.237 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:23:03.237 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:23:03.237 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:23:03.237 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:23:03.237 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:23:03.237 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:23:03.237 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:23:03.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:23:03.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:23:03.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:23:03.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:23:03.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:23:03.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:23:03.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:23:03.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:23:03.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:23:03.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:23:03.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:23:03.475 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 37 16 102607 102591 173.282 104 0.00792143 0.00576925 2024-04-23T16:23:03.601 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:23:03.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:03.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:23:03.704 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:03.736 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:03.773 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:03.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:03.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:03.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:03 smithi103 ceph-mon[39629]: pgmap v612: 17 pgs: 17 active+clean; 5.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-04-23T16:23:03.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:03 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2826201962' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:23:03.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:03 smithi148 ceph-mon[44520]: pgmap v612: 17 pgs: 17 active+clean; 5.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-04-23T16:23:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:03 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2826201962' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:23:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.879 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:03.909 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:03.939 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:03.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:03.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:03.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:03 smithi188 ceph-mon[44101]: pgmap v612: 17 pgs: 17 active+clean; 5.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-04-23T16:23:03.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:03 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2826201962' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:23:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:04.043 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:23:04.081 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:04.449 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:23:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:04.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:23:04.475 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 38 16 104367 104351 171.617 110 0.00656056 0.00582547 2024-04-23T16:23:04.554 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:04.594 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:04.622 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:04.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:04.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:04.728 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:23:04.761 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:04.788 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:23:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:04.792 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:23:04.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:04.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:04.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:04.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:04 smithi103 ceph-mon[39629]: osdmap e209: 12 total, 12 up, 11 in 2024-04-23T16:23:04.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:04.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:04.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:04.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:04 smithi148 ceph-mon[44520]: osdmap e209: 12 total, 12 up, 11 in 2024-04-23T16:23:04.893 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:23:04.935 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:04.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:04.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:04.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:04.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:04 smithi188 ceph-mon[44101]: osdmap e209: 12 total, 12 up, 11 in 2024-04-23T16:23:04.983 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:23:04.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:04.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:23:04.996 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:23:05.064 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:23:05.089 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:23:05.119 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:05.150 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:23:05.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:05.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:23:05.255 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:23:05.284 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:05.314 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:23:05.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:05.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:23:05.419 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:23:05.469 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:05.475 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 39 16 106005 105989 169.841 102.375 0.00398045 0.0058865 2024-04-23T16:23:05.937 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:05 smithi103 ceph-mon[39629]: pgmap v614: 17 pgs: 17 active+clean; 6.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 112 MiB/s wr, 1.78k op/s 2024-04-23T16:23:05.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:05 smithi188 ceph-mon[44101]: pgmap v614: 17 pgs: 17 active+clean; 6.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 112 MiB/s wr, 1.78k op/s 2024-04-23T16:23:06.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:05 smithi148 ceph-mon[44520]: pgmap v614: 17 pgs: 17 active+clean; 6.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 112 MiB/s wr, 1.78k op/s 2024-04-23T16:23:06.476 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:23:06.475739 min lat: 0.00099496 max lat: 0.206668 avg lat: 0.00594394 2024-04-23T16:23:06.476 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:23:06.476 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 40 16 107675 107659 168.204 104.375 0.00498818 0.00594394 2024-04-23T16:23:06.535 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:23:06.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:06.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:23:06.640 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:23:06.668 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.11 to 1 (10000) 2024-04-23T16:23:06.698 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:06.790 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:06 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/571176889' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:23:06.790 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:06 smithi103 ceph-mon[39629]: from='client.25084 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:23:06.791 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:06 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:23:06.791 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:06 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:23:06.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:06 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/571176889' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:23:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:06 smithi188 ceph-mon[44101]: from='client.25084 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:23:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:06 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:23:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:06 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:23:07.037 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:23:07.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:06 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/571176889' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:23:07.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:06 smithi148 ceph-mon[44520]: from='client.25084 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:23:07.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:06 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:23:07.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:06 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:23:07.476 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 41 16 109183 109167 166.4 94.25 0.0121699 0.00600766 2024-04-23T16:23:07.478 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:23:07.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:07.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:23:07.589 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:23:07.629 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:07.652 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:23:07.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:07.659 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:23:07.759 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:23:07.822 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:07 smithi103 ceph-mon[39629]: pgmap v615: 17 pgs: 17 active+clean; 6.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-04-23T16:23:07.822 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:07 smithi103 ceph-mon[39629]: from='client.25084 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:23:07.823 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:07 smithi103 ceph-mon[39629]: osdmap e210: 12 total, 12 up, 11 in 2024-04-23T16:23:07.823 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:23:07.823 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:23:07.823 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:23:07.825 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:07.936 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:23:07.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:07.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:23:07.973 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-04-23T16:23:07.973 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-04-23T16:23:07.973 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-23T16:23:07.974 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T16:23:07.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:07 smithi188 ceph-mon[44101]: pgmap v615: 17 pgs: 17 active+clean; 6.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-04-23T16:23:07.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:07 smithi188 ceph-mon[44101]: from='client.25084 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:23:07.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:07 smithi188 ceph-mon[44101]: osdmap e210: 12 total, 12 up, 11 in 2024-04-23T16:23:07.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:23:07.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:23:07.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:23:08.043 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:23:08.120 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:08.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:07 smithi148 ceph-mon[44520]: pgmap v615: 17 pgs: 17 active+clean; 6.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-04-23T16:23:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:07 smithi148 ceph-mon[44520]: from='client.25084 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:23:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:07 smithi148 ceph-mon[44520]: osdmap e210: 12 total, 12 up, 11 in 2024-04-23T16:23:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:07 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:23:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:07 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:23:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:07 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:23:08.476 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 42 16 110870 110854 164.948 105.438 0.00374525 0.00606117 2024-04-23T16:23:09.476 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 43 16 112515 112499 163.503 102.812 0.00702109 0.00611491 2024-04-23T16:23:09.673 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:09 smithi103 ceph-mon[39629]: pgmap v617: 17 pgs: 17 active+clean; 6.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-04-23T16:23:09.673 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:09 smithi103 ceph-mon[39629]: osdmap e211: 12 total, 12 up, 11 in 2024-04-23T16:23:09.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:09 smithi188 ceph-mon[44101]: pgmap v617: 17 pgs: 17 active+clean; 6.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-04-23T16:23:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:09 smithi188 ceph-mon[44101]: osdmap e211: 12 total, 12 up, 11 in 2024-04-23T16:23:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:09 smithi148 ceph-mon[44520]: pgmap v617: 17 pgs: 17 active+clean; 6.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-04-23T16:23:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:09 smithi148 ceph-mon[44520]: osdmap e211: 12 total, 12 up, 11 in 2024-04-23T16:23:10.476 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 44 16 114146 114130 162.104 101.938 0.00647428 0.00616779 2024-04-23T16:23:10.514 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:23:10.514 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:23:10.514 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-23T16:23:10.514 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-23T16:23:10.514 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:23:10.663 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:23:10.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:10 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/384881042' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:23:10.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:10 smithi188 ceph-mon[44101]: from='client.35141 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:23:11.000 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:23:11.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:11.008 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:10 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/384881042' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:23:11.008 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:10 smithi103 ceph-mon[39629]: from='client.35141 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:23:11.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:23:11.110 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:23:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:10 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/384881042' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:23:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:10 smithi148 ceph-mon[44520]: from='client.35141 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:23:11.145 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:11.169 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:23:11.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:11.172 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:23:11.273 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:23:11.320 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:11.476 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 45 16 115785 115769 160.778 102.438 0.00478247 0.00621844 2024-04-23T16:23:11.610 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:23:11.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:11.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:23:11.660 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:23:11.661 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:23:11.714 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:23:11.746 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:11.777 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:23:11.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:11.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:23:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:11 smithi103 ceph-mon[39629]: pgmap v619: 17 pgs: 17 active+clean; 6.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-04-23T16:23:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:11 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:23:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:11 smithi103 ceph-mon[39629]: from='client.35141 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:23:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:11 smithi103 ceph-mon[39629]: osdmap e212: 12 total, 12 up, 11 in 2024-04-23T16:23:11.881 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:23:11.910 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:11 smithi188 ceph-mon[44101]: pgmap v619: 17 pgs: 17 active+clean; 6.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-04-23T16:23:11.910 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:11 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:23:11.910 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:11 smithi188 ceph-mon[44101]: from='client.35141 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:23:11.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:11 smithi188 ceph-mon[44101]: osdmap e212: 12 total, 12 up, 11 in 2024-04-23T16:23:11.913 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:11.938 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:23:11.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:11.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:23:12.043 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:23:12.093 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:11 smithi148 ceph-mon[44520]: pgmap v619: 17 pgs: 17 active+clean; 6.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-04-23T16:23:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:11 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:23:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:11 smithi148 ceph-mon[44520]: from='client.35141 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:23:12.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:11 smithi148 ceph-mon[44520]: osdmap e212: 12 total, 12 up, 11 in 2024-04-23T16:23:12.147 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:23:12.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:12.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:23:12.193 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-23T16:23:12.256 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:23:12.289 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:12.329 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:23:12.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:12.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:23:12.434 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:12.470 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:12.476 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 46 16 117376 117360 159.444 99.4375 0.00941176 0.0062706 2024-04-23T16:23:12.501 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:12.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:12.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:12.604 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:23:12.662 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:12.826 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:23:12.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:12 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:23:12.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:12 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:23:12.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:12 smithi103 ceph-mon[39629]: osdmap e213: 12 total, 12 up, 11 in 2024-04-23T16:23:12.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:12.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:23:12.931 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:23:12.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:12 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:23:12.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:12 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:23:12.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:12 smithi188 ceph-mon[44101]: osdmap e213: 12 total, 12 up, 11 in 2024-04-23T16:23:12.988 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:12 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:23:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:12 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:23:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:12 smithi148 ceph-mon[44520]: osdmap e213: 12 total, 12 up, 11 in 2024-04-23T16:23:13.350 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:23:13.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:13.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:23:13.459 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:23:13.476 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 47 15 118882 118867 158.055 94.1875 0.00747332 0.00632545 2024-04-23T16:23:13.500 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:13.525 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:23:13.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:13.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:23:13.620 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:23:13.632 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:23:13.723 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:13.857 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:23:13.867 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:23:13.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:13.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:23:13.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:13 smithi188 ceph-mon[44101]: pgmap v622: 17 pgs: 17 active+clean; 6.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-04-23T16:23:13.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:13 smithi188 ceph-mon[44101]: 3.e scrub starts 2024-04-23T16:23:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:13 smithi188 ceph-mon[44101]: 3.9 scrub starts 2024-04-23T16:23:13.978 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:14.013 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:13 smithi148 ceph-mon[44520]: pgmap v622: 17 pgs: 17 active+clean; 6.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-04-23T16:23:14.013 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:13 smithi148 ceph-mon[44520]: 3.e scrub starts 2024-04-23T16:23:14.013 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:13 smithi148 ceph-mon[44520]: 3.9 scrub starts 2024-04-23T16:23:14.015 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:14.036 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:14.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:14.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:14.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:13 smithi103 ceph-mon[39629]: pgmap v622: 17 pgs: 17 active+clean; 6.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-04-23T16:23:14.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:13 smithi103 ceph-mon[39629]: 3.e scrub starts 2024-04-23T16:23:14.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:13 smithi103 ceph-mon[39629]: 3.9 scrub starts 2024-04-23T16:23:14.141 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:23:14.177 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:14.209 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:23:14.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:14.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:23:14.314 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:23:14.355 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:14.377 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:23:14.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:14.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:23:14.476 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 48 16 120175 120159 156.445 80.75 0.00614355 0.00639076 2024-04-23T16:23:14.482 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:23:14.516 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:14.543 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:23:14.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:14.550 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:23:14.651 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:23:14.696 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:14.721 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:23:14.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:14.725 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:23:14.827 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:23:14.887 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:14.963 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:23:14.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:14.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:23:15.071 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:23:15.106 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:15.107 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:14 smithi103 ceph-mon[39629]: pgmap v623: 17 pgs: 17 active+clean; 7.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-04-23T16:23:15.136 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:23:15.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:15.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:23:15.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:14 smithi188 ceph-mon[44101]: pgmap v623: 17 pgs: 17 active+clean; 7.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-04-23T16:23:15.239 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:23:15.266 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:14 smithi148 ceph-mon[44520]: pgmap v623: 17 pgs: 17 active+clean; 7.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-04-23T16:23:15.268 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:15.303 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:23:15.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:15.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:23:15.408 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:23:15.469 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:15.476 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 49 16 121579 121563 155.043 87.75 0.00641815 0.00644819 2024-04-23T16:23:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:15 smithi188 ceph-mon[44101]: 3.b scrub starts 2024-04-23T16:23:16.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:15 smithi188 ceph-mon[44101]: 3.e scrub ok 2024-04-23T16:23:16.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:15 smithi188 ceph-mon[44101]: 3.9 scrub ok 2024-04-23T16:23:16.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:15 smithi103 ceph-mon[39629]: 3.b scrub starts 2024-04-23T16:23:16.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:15 smithi103 ceph-mon[39629]: 3.e scrub ok 2024-04-23T16:23:16.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:15 smithi103 ceph-mon[39629]: 3.9 scrub ok 2024-04-23T16:23:16.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:15 smithi148 ceph-mon[44520]: 3.b scrub starts 2024-04-23T16:23:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:15 smithi148 ceph-mon[44520]: 3.e scrub ok 2024-04-23T16:23:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:15 smithi148 ceph-mon[44520]: 3.9 scrub ok 2024-04-23T16:23:16.476 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 50 16 122776 122760 153.438 74.8125 0.0189645 0.00651566 2024-04-23T16:23:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:16 smithi188 ceph-mon[44101]: 3.7 scrub starts 2024-04-23T16:23:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:16 smithi188 ceph-mon[44101]: 3.b scrub ok 2024-04-23T16:23:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:16 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:23:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:16 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:23:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:16 smithi188 ceph-mon[44101]: pgmap v624: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 112 MiB/s wr, 1.78k op/s 2024-04-23T16:23:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:16 smithi103 ceph-mon[39629]: 3.7 scrub starts 2024-04-23T16:23:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:16 smithi103 ceph-mon[39629]: 3.b scrub ok 2024-04-23T16:23:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:16 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:23:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:16 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:23:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:16 smithi103 ceph-mon[39629]: pgmap v624: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 112 MiB/s wr, 1.78k op/s 2024-04-23T16:23:17.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:16 smithi148 ceph-mon[44520]: 3.7 scrub starts 2024-04-23T16:23:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:16 smithi148 ceph-mon[44520]: 3.b scrub ok 2024-04-23T16:23:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:16 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:23:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:16 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:23:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:16 smithi148 ceph-mon[44520]: pgmap v624: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 112 MiB/s wr, 1.78k op/s 2024-04-23T16:23:17.476 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 51 16 124291 124275 152.286 94.6875 0.00631217 0.0065652 2024-04-23T16:23:18.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:17 smithi188 ceph-mon[44101]: 3.7 scrub ok 2024-04-23T16:23:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:17 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:23:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:17 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:23:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:17 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:23:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:17 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:23:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:17 smithi188 ceph-mon[44101]: osdmap e214: 12 total, 12 up, 11 in 2024-04-23T16:23:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:17 smithi103 ceph-mon[39629]: 3.7 scrub ok 2024-04-23T16:23:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:17 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:23:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:17 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:23:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:17 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:23:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:17 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:23:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:17 smithi103 ceph-mon[39629]: osdmap e214: 12 total, 12 up, 11 in 2024-04-23T16:23:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:17 smithi148 ceph-mon[44520]: 3.7 scrub ok 2024-04-23T16:23:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:17 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:23:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:17 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:23:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:17 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:23:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:17 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:23:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:17 smithi148 ceph-mon[44520]: osdmap e214: 12 total, 12 up, 11 in 2024-04-23T16:23:18.456 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:23:18.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:18.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:23:18.476 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 52 16 125733 125717 151.09 90.125 0.00747172 0.00661723 2024-04-23T16:23:18.562 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:23:18.588 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:18.609 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:23:18.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:18.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:23:18.715 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:23:18.752 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:18.778 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:23:18.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:18.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:23:18.884 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:23:18.925 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:18.971 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:18 smithi188 ceph-mon[44101]: 3.c scrub starts 2024-04-23T16:23:18.971 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:18 smithi188 ceph-mon[44101]: osdmap e215: 12 total, 12 up, 11 in 2024-04-23T16:23:18.972 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:18 smithi188 ceph-mon[44101]: pgmap v627: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-04-23T16:23:19.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:18 smithi148 ceph-mon[44520]: 3.c scrub starts 2024-04-23T16:23:19.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:18 smithi148 ceph-mon[44520]: osdmap e215: 12 total, 12 up, 11 in 2024-04-23T16:23:19.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:18 smithi148 ceph-mon[44520]: pgmap v627: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-04-23T16:23:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:18 smithi103 ceph-mon[39629]: 3.c scrub starts 2024-04-23T16:23:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:18 smithi103 ceph-mon[39629]: osdmap e215: 12 total, 12 up, 11 in 2024-04-23T16:23:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:18 smithi103 ceph-mon[39629]: pgmap v627: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-04-23T16:23:19.477 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 53 16 127192 127176 149.96 91.1875 0.00741454 0.00666692 2024-04-23T16:23:20.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:19 smithi103 ceph-mon[39629]: 3.c scrub ok 2024-04-23T16:23:20.115 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:23:20.143 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:23:20.172 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:23:20.208 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:23:20.237 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:23:20.268 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:23:20.288 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:23:20.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:20.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:23:20.303 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:23:20.334 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:23:20.372 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:23:20.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:19 smithi148 ceph-mon[44520]: 3.c scrub ok 2024-04-23T16:23:20.394 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:20.401 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:23:20.428 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:20.435 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:23:20.456 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:20.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:20.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:20.466 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:23:20.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:19 smithi188 ceph-mon[44101]: 3.c scrub ok 2024-04-23T16:23:20.477 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 54 16 128464 128448 148.655 79.5 0.0138833 0.0067257 2024-04-23T16:23:20.482 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:23:20.482 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:23:20.483 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:23:20.483 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:23:20.483 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:23:20.483 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:23:20.483 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:23:20.483 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:23:20.483 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:23:20.483 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:23:20.483 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:23:20.483 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:23:20.561 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:23:20.617 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:20.782 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:23:20.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:20.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:23:20.891 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:20.928 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:20.956 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:20.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:20.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:20.976 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:23:20.994 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:20 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2229307226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:23:20.994 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:20.994 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:20.994 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:20 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:23:20.994 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:20 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:23:20.994 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:20.994 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:20.995 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:20.995 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:20.995 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:20 smithi103 ceph-mon[39629]: pgmap v628: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-04-23T16:23:21.060 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:23:21.095 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:21.118 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:23:21.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:21.122 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:23:21.223 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:21.250 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:20 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2229307226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:23:21.250 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:21.250 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:21.251 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:20 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:23:21.251 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:20 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:23:21.251 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:21.251 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:21.251 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:21.251 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:21.251 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:20 smithi148 ceph-mon[44520]: pgmap v628: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-04-23T16:23:21.252 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:21.282 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:21.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:21.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:21.386 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:23:21.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:20 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2229307226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:23:21.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:21.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:21.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:20 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:23:21.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:20 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:23:21.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:21.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:21.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:21.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:21.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:20 smithi188 ceph-mon[44101]: pgmap v628: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-04-23T16:23:21.420 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:21.444 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:23:21.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:21.450 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:23:21.477 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 55 16 129900 129884 147.584 89.75 0.00673318 0.00677438 2024-04-23T16:23:21.545 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:23:21.545 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":216,"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","created":"2024-04-23T16:08:13.477590+0000","modified":"2024-04-23T16:23:20.974329+0000","last_up_change":"2024-04-23T16:15:34.871303+0000","last_in_change":"2024-04-23T16:20:45.624667+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T16:11:22.416777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.98181778192520142,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}},{"pool":3,"pool_name":"unique_pool_1","create_time":"2024-04-23T16:22:10.377567+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.2699999809265137,"score_stable":2.2699999809265137,"optimal_score":0.82999998331069946,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.98181778192520142,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}}],"osds":[{"osd":0,"uuid":"4c084047-8f01-4b86-a60c-5592735b5862","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6801","nonce":4274040097}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6802","nonce":4274040097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6804","nonce":4274040097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6803","nonce":4274040097}]},"public_addr":"172.21.15.103:6801/4274040097","cluster_addr":"172.21.15.103:6802/4274040097","heartbeat_back_addr":"172.21.15.103:6804/4274040097","heartbeat_front_addr":"172.21.15.103:6803/4274040097","state":["exists","up"]},{"osd":1,"uuid":"3438f2bd-5fb7-432c-b383-29e18c442f20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6805","nonce":3716921156}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6806","nonce":3716921156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6808","nonce":3716921156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6807","nonce":3716921156}]},"public_addr":"172.21.15.103:6805/3716921156","cluster_addr":"172.21.15.103:6806/3716921156","heartbeat_back_addr":"172.21.15.103:6808/3716921156","heartbeat_front_addr":"172.21.15.103:6807/3716921156","state":["exists","up"]},{"osd":2,"uuid":"c17fffb4-8d17-4546-ba24-e5951a2a37fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6809","nonce":424240088}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6810","nonce":424240088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6812","nonce":424240088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6811","nonce":424240088}]},"public_addr":"172.21.15.103:6809/424240088","cluster_addr":"172.21.15.103:6810/424240088","heartbeat_back_addr":"172.21.15.103:6812/424240088","heartbeat_front_addr":"172.21.15.103:6811/424240088","state":["exists","up"]},{"osd":3,"uuid":"cee91a3d-55dc-4e89-b568-86821632bb1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6813","nonce":1895183074}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6814","nonce":1895183074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6816","nonce":1895183074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6815","nonce":1895183074}]},"public_addr":"172.21.15.103:6813/1895183074","cluster_addr":"172.21.15.103:6814/1895183074","heartbeat_back_addr":"172.21.15.103:6816/1895183074","heartbeat_front_addr":"172.21.15.103:6815/1895183074","state":["exists","up"]},{"osd":4,"uuid":"61d5385b-2c82-42f8-93c3-8f1baf71c65e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6800","nonce":794945071}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6801","nonce":794945071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6803","nonce":794945071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6802","nonce":794945071}]},"public_addr":"172.21.15.148:6800/794945071","cluster_addr":"172.21.15.148:6801/794945071","heartbeat_back_addr":"172.21.15.148:6803/794945071","heartbeat_front_addr":"172.21.15.148:6802/794945071","state":["exists","up"]},{"osd":5,"uuid":"5d492189-c3cd-47ad-a12a-ed4d275bcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6804","nonce":3547200711}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6805","nonce":3547200711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6807","nonce":3547200711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6806","nonce":3547200711}]},"public_addr":"172.21.15.148:6804/3547200711","cluster_addr":"172.21.15.148:6805/3547200711","heartbeat_back_addr":"172.21.15.148:6807/3547200711","heartbeat_front_addr":"172.21.15.148:6806/3547200711","state":["exists","up"]},{"osd":6,"uuid":"89f2379e-cf14-4e69-8747-7535db81b70b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6808","nonce":366919758}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6809","nonce":366919758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6811","nonce":366919758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6810","nonce":366919758}]},"public_addr":"172.21.15.148:6808/366919758","cluster_addr":"172.21.15.148:6809/366919758","heartbeat_back_addr":"172.21.15.148:6811/366919758","heartbeat_front_addr":"172.21.15.148:6810/366919758","state":["exists","up"]},{"osd":7,"uuid":"f27b035a-558f-4528-80ea-2a00d03f6539","up":1,"in":1,"weight":1,"primary_affinity":0.7818145751953125,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6812","nonce":3719550888}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6813","nonce":3719550888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6815","nonce":3719550888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6814","nonce":3719550888}]},"public_addr":"172.21.15.148:6812/3719550888","cluster_addr":"172.21.15.148:6813/3719550888","heartbeat_back_addr":"172.21.15.148:6815/3719550888","heartbeat_front_addr":"172.21.15.148:6814/3719550888","state":["exists","up"]},{"osd":8,"uuid":"a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6800","nonce":1151408595}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6801","nonce":1151408595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6803","nonce":1151408595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6802","nonce":1151408595}]},"public_addr":"172.21.15.188:6800/1151408595","cluster_addr":"172.21.15.188:6801/1151408595","heartbeat_back_addr":"172.21.15.188:6803/1151408595","heartbeat_front_addr":"172.21.15.188:6802/1151408595","state":["exists","up"]},{"osd":9,"uuid":"124d9b05-11eb-4622-b6a2-c4d5bd60b7e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6804","nonce":306501862}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6805","nonce":306501862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6807","nonce":306501862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6806","nonce":306501862}]},"public_addr":"172.21.15.188:6804/306501862","cluster_addr":"172.21.15.188:6805/306501862","heartbeat_back_addr":"172.21.15.188:6807/306501862","heartbeat_front_addr":"172.21.15.188:6806/306501862","state":["exists","up"]},{"osd":10,"uuid":"3dfb9070-e534-4ea4-a25e-ee1c51297447","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6808","nonce":1231263138}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6809","nonce":1231263138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6811","nonce":1231263138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6810","nonce":1231263138}]},"public_addr":"172.21.15.188:6808/1231263138","cluster_addr":"172.21.15.188:6809/1231263138","heartbeat_back_addr":"172.21.15.188:6811/1231263138","heartbeat_front_addr":"172.21.15.188:6810/1231263138","state":["exists","up"]},{"osd":11,"uuid":"bc163925-f623-4f63-ab89-35e2d511a730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6812","nonce":457992661}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6813","nonce":457992661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6815","nonce":457992661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6814","nonce":457992661}]},"public_addr":"172.21.15.188:6812/457992661","cluster_addr":"172.21.15.188:6813/457992661","heartbeat_back_addr":"172.21.15.188:6815/457992661","heartbeat_front_addr":"172.21.15.188:6814/457992661","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:22:05.270238+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:22:56.499087+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:22:22.061930+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:22:36.036909+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:23:16.137378+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:23:11.406142+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:22:57.951086+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:22:42.086658+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:23:20.193725+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:22:35.947252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-23T16:22:14.677898+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:23:16.949271+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6800/3535470771":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1188058303":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1510553721":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/281182801":"2024-04-24T16:08:44.450632+0000","172.21.15.103:6800/2049124518":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/466241963":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/2726838698":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/596230573":"2024-04-24T16:09:14.324448+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T16:23:21.551 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:23:21.583 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:21.607 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:23:21.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:21.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:23:21.716 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:23:21.753 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:21.778 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:23:21.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:21.782 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:23:21.883 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:23:21.916 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:21.937 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:23:21.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:21.940 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:23:22.040 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:23:22.072 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:22.100 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:23:22.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:22.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:23:22.202 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:23:22.202 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:23:22.203 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:23:22.228 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:22.228 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:22.228 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:22.228 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:22.228 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:22.229 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:22.229 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:21 smithi103 ceph-mon[39629]: osdmap e216: 12 total, 12 up, 11 in 2024-04-23T16:23:22.229 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:21 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2112123998' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:23:22.240 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:22.265 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:23:22.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:22.271 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:23:22.271 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:22.271 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:22.271 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:22.271 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:22.272 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:22.272 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:22.272 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:21 smithi188 ceph-mon[44101]: osdmap e216: 12 total, 12 up, 11 in 2024-04-23T16:23:22.272 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:21 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2112123998' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:23:22.371 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:22.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:22.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:22.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:22.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:22.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:22.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:22.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:21 smithi148 ceph-mon[44520]: osdmap e216: 12 total, 12 up, 11 in 2024-04-23T16:23:22.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:21 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2112123998' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:23:22.404 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:22.432 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:22.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:22.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:22.477 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 56 16 131359 131343 146.577 91.1875 0.0101509 0.00682095 2024-04-23T16:23:22.538 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:23:22.567 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:22.593 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:23:22.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:22.598 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:23:22.699 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:23:22.729 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:22.755 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:23:22.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:22.762 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:23:22.863 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:23:22.895 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:22.917 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:23:22.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:22.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:23:22.946 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-23T16:23:23.021 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:23:23.055 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:23.080 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:23:23.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:23.087 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:23:23.188 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:23:23.223 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:23 smithi148 ceph-mon[44520]: pgmap v630: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-23T16:23:23.225 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:23.252 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:23:23.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:23.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:23:23.261 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:23 smithi103 ceph-mon[39629]: pgmap v630: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-23T16:23:23.358 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:23:23.398 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:23 smithi188 ceph-mon[44101]: pgmap v630: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-23T16:23:23.477 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 57 16 132830 132814 145.618 91.9375 0.0107022 0.006866 2024-04-23T16:23:23.737 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-23T16:23:23.911 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:23:24.075 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:23:24.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:24.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:23:24.114 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:23:24.187 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:23:24.250 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:24.477 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 58 16 134209 134193 144.593 86.1875 0.0232406 0.00691397 2024-04-23T16:23:24.617 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:23:24.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:24.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:23:24.721 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:23:24.768 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:24.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:24 smithi148 ceph-mon[44520]: osdmap e217: 12 total, 12 up, 11 in 2024-04-23T16:23:24.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:24 smithi148 ceph-mon[44520]: 3.8 scrub starts 2024-04-23T16:23:24.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:24 smithi148 ceph-mon[44520]: pgmap v632: 17 pgs: 17 active+clean; 8.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-23T16:23:25.208 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:24 smithi103 ceph-mon[39629]: osdmap e217: 12 total, 12 up, 11 in 2024-04-23T16:23:25.209 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:24 smithi103 ceph-mon[39629]: 3.8 scrub starts 2024-04-23T16:23:25.209 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:24 smithi103 ceph-mon[39629]: pgmap v632: 17 pgs: 17 active+clean; 8.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-23T16:23:25.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:24 smithi188 ceph-mon[44101]: osdmap e217: 12 total, 12 up, 11 in 2024-04-23T16:23:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:24 smithi188 ceph-mon[44101]: 3.8 scrub starts 2024-04-23T16:23:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:24 smithi188 ceph-mon[44101]: pgmap v632: 17 pgs: 17 active+clean; 8.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-23T16:23:25.234 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:23:25.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:25.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:23:25.339 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:23:25.371 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:25.477 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 59 16 135581 135565 143.596 85.75 0.00699247 0.006963 2024-04-23T16:23:25.583 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:23:25.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:25.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:23:25.660 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:23:25.688 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:23:25.735 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:25.768 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:23:25.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:25.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:23:25.874 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:23:25.908 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:25.933 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:23:25.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:25.938 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:23:26.039 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:23:26.068 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:26.106 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:23:26.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:26.111 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:23:26.168 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:23:26.168 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:23:26.212 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:23:26.249 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:26.273 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:23:26.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:26.276 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:23:26.332 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:23:26.339 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:26 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/281410669' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:23:26.339 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:26 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:23:26.377 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:23:26.413 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:26.444 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:23:26.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:26.449 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:23:26.477 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:23:26.477338 min lat: 0.00099496 max lat: 0.225871 avg lat: 0.00701401 2024-04-23T16:23:26.477 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:23:26.477 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 60 15 136864 136849 142.54 80.25 0.00969858 0.00701401 2024-04-23T16:23:26.549 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:23:26.601 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:26 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/281410669' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:23:26.602 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:26 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:23:26.606 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:26.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:26 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/281410669' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:23:26.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:26 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:23:26.977 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-23T16:23:26.977 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:23:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:26.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:23:27.083 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:23:27.132 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:27.155 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:23:27.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:27.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:23:27.264 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:23:27.317 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:27.356 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:27 smithi148 ceph-mon[44520]: 3.8 scrub ok 2024-04-23T16:23:27.356 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:27 smithi148 ceph-mon[44520]: 3.d scrub starts 2024-04-23T16:23:27.356 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:27 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:23:27.356 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:27 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/281410669' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:23:27.356 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:27 smithi148 ceph-mon[44520]: osdmap e218: 12 total, 12 up, 11 in 2024-04-23T16:23:27.356 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:27 smithi148 ceph-mon[44520]: pgmap v634: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.2 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-04-23T16:23:27.362 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:23:27.398 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:23:27.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:27.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:23:27.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:27 smithi188 ceph-mon[44101]: 3.8 scrub ok 2024-04-23T16:23:27.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:27 smithi188 ceph-mon[44101]: 3.d scrub starts 2024-04-23T16:23:27.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:27 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:23:27.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:27 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/281410669' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:23:27.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:27 smithi188 ceph-mon[44101]: osdmap e218: 12 total, 12 up, 11 in 2024-04-23T16:23:27.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:27 smithi188 ceph-mon[44101]: pgmap v634: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.2 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-04-23T16:23:27.477 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 61 16 138120 138104 141.489 78.4375 0.00734966 0.00706465 2024-04-23T16:23:27.509 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:23:27.551 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:27 smithi103 ceph-mon[39629]: 3.8 scrub ok 2024-04-23T16:23:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:27 smithi103 ceph-mon[39629]: 3.d scrub starts 2024-04-23T16:23:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:27 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:23:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:27 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/281410669' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:23:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:27 smithi103 ceph-mon[39629]: osdmap e218: 12 total, 12 up, 11 in 2024-04-23T16:23:27.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:27 smithi103 ceph-mon[39629]: pgmap v634: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.2 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-04-23T16:23:28.412 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:23:28.417 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:28 smithi103 ceph-mon[39629]: 3.0 deep-scrub starts 2024-04-23T16:23:28.417 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:28 smithi103 ceph-mon[39629]: 3.6 scrub starts 2024-04-23T16:23:28.417 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:28 smithi103 ceph-mon[39629]: 3.d scrub ok 2024-04-23T16:23:28.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:28.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:23:28.477 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 62 16 139217 139201 140.313 68.5625 0.0088345 0.00712396 2024-04-23T16:23:28.520 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:23:28.555 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:28 smithi148 ceph-mon[44520]: 3.0 deep-scrub starts 2024-04-23T16:23:28.555 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:28 smithi148 ceph-mon[44520]: 3.6 scrub starts 2024-04-23T16:23:28.555 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:28 smithi148 ceph-mon[44520]: 3.d scrub ok 2024-04-23T16:23:28.556 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:28.632 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:23:28.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:28.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:23:28.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:28 smithi188 ceph-mon[44101]: 3.0 deep-scrub starts 2024-04-23T16:23:28.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:28 smithi188 ceph-mon[44101]: 3.6 scrub starts 2024-04-23T16:23:28.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:28 smithi188 ceph-mon[44101]: 3.d scrub ok 2024-04-23T16:23:28.736 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:28.738 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:23:28.762 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:23:28.763 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:28.784 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:23:28.784 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:28.810 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:23:28.838 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:23:28.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:28.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:28.865 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:23:28.887 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:23:28.911 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:23:28.939 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:23:28.949 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:23:28.975 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:23:29.002 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:29.012 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:23:29.027 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:23:29.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:29.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:23:29.038 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:23:29.045 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:23:29.045 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:23:29.045 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:23:29.045 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:23:29.045 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:23:29.045 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:23:29.045 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:23:29.046 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:23:29.046 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:23:29.046 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:23:29.046 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:23:29.046 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:23:29.132 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:23:29.162 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:29.477 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 63 16 140251 140235 139.111 64.625 0.0078683 0.0071859 2024-04-23T16:23:29.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:29 smithi148 ceph-mon[44520]: pgmap v635: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-04-23T16:23:29.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:29 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1013189383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:23:29.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:29.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:29.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:29.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:29.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:29.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:29 smithi148 ceph-mon[44520]: osdmap e219: 12 total, 12 up, 11 in 2024-04-23T16:23:29.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:29.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:29 smithi148 ceph-mon[44520]: 3.5 scrub starts 2024-04-23T16:23:29.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:29 smithi188 ceph-mon[44101]: pgmap v635: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-04-23T16:23:29.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:29 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1013189383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:23:29.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:29.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:29.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:29.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:29.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:29.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:29 smithi188 ceph-mon[44101]: osdmap e219: 12 total, 12 up, 11 in 2024-04-23T16:23:29.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:29.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:29 smithi188 ceph-mon[44101]: 3.5 scrub starts 2024-04-23T16:23:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:29 smithi103 ceph-mon[39629]: pgmap v635: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-04-23T16:23:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:29 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1013189383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:23:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:29.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:29.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:29 smithi103 ceph-mon[39629]: osdmap e219: 12 total, 12 up, 11 in 2024-04-23T16:23:29.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:29.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:29 smithi103 ceph-mon[39629]: 3.5 scrub starts 2024-04-23T16:23:30.190 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:23:30.477 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 64 16 141286 141270 137.948 64.6875 0.00890475 0.00724525 2024-04-23T16:23:30.538 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:23:30.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:30 smithi103 ceph-mon[39629]: 3.6 scrub ok 2024-04-23T16:23:30.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:30.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:30.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:30.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:30.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:30.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:30.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:30 smithi103 ceph-mon[39629]: 3.3 scrub starts 2024-04-23T16:23:30.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:30 smithi148 ceph-mon[44520]: 3.6 scrub ok 2024-04-23T16:23:30.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:30.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:30.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:30.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:30.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:30.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:30.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:30 smithi148 ceph-mon[44520]: 3.3 scrub starts 2024-04-23T16:23:30.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:30 smithi188 ceph-mon[44101]: 3.6 scrub ok 2024-04-23T16:23:30.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:30.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:30.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:30.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:30.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:30.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:30.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:30 smithi188 ceph-mon[44101]: 3.3 scrub starts 2024-04-23T16:23:31.478 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 65 16 142205 142189 136.709 57.4375 0.00419773 0.00731025 2024-04-23T16:23:31.519 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:23:31.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:31.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:23:31.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:31 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:23:31.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:31 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:23:31.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:31 smithi103 ceph-mon[39629]: osdmap e220: 12 total, 12 up, 11 in 2024-04-23T16:23:31.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:31 smithi103 ceph-mon[39629]: pgmap v638: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-23T16:23:31.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:31 smithi103 ceph-mon[39629]: 3.5 scrub ok 2024-04-23T16:23:31.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:31 smithi103 ceph-mon[39629]: 3.3 scrub ok 2024-04-23T16:23:31.623 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:23:31.647 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:31 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:23:31.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:31 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:23:31.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:31 smithi148 ceph-mon[44520]: osdmap e220: 12 total, 12 up, 11 in 2024-04-23T16:23:31.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:31 smithi148 ceph-mon[44520]: pgmap v638: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-23T16:23:31.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:31 smithi148 ceph-mon[44520]: 3.5 scrub ok 2024-04-23T16:23:31.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:31 smithi148 ceph-mon[44520]: 3.3 scrub ok 2024-04-23T16:23:31.659 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:31 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:23:31.660 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:31 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:23:31.660 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:31 smithi188 ceph-mon[44101]: osdmap e220: 12 total, 12 up, 11 in 2024-04-23T16:23:31.660 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:31 smithi188 ceph-mon[44101]: pgmap v638: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-23T16:23:31.660 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:31 smithi188 ceph-mon[44101]: 3.5 scrub ok 2024-04-23T16:23:31.660 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:31 smithi188 ceph-mon[44101]: 3.3 scrub ok 2024-04-23T16:23:31.675 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:31.695 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:23:31.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:31.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:23:31.800 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:23:31.837 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:31.930 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:23:31.991 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:23:31.991 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:23:31.991 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2024-04-23T16:23:31.991 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2024-04-23T16:23:31.991 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:23:32.046 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:23:32.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:32.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:23:32.154 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:32.186 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:32.244 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:32.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:32.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:32.351 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:23:32.410 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:32.448 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:23:32.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:32.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:23:32.478 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 66 16 143069 143053 135.456 54 0.0289649 0.00737823 2024-04-23T16:23:32.556 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:23:32.586 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:32.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:32 smithi148 ceph-mon[44520]: 3.4 deep-scrub starts 2024-04-23T16:23:32.688 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:23:32.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:32.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:23:32.715 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-23T16:23:32.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:32 smithi188 ceph-mon[44101]: 3.4 deep-scrub starts 2024-04-23T16:23:32.793 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:23:32.839 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:23:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:32 smithi103 ceph-mon[39629]: 3.4 deep-scrub starts 2024-04-23T16:23:32.887 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:33.064 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:23:33.136 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:23:33.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:33.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:23:33.243 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:23:33.273 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:33.309 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:23:33.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:33.314 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:23:33.404 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:23:33.414 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:23:33.448 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:33.470 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:23:33.477 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 67 16 143905 143889 134.214 52.25 0.0069035 0.0074481 2024-04-23T16:23:33.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:33.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:23:33.581 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:33.612 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:33 smithi148 ceph-mon[44520]: 3.1 scrub starts 2024-04-23T16:23:33.612 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:33 smithi148 ceph-mon[44520]: pgmap v639: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-04-23T16:23:33.612 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:33 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3034413240' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:23:33.612 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:33 smithi148 ceph-mon[44520]: from='client.25108 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:23:33.613 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:33.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:33 smithi188 ceph-mon[44101]: 3.1 scrub starts 2024-04-23T16:23:33.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:33 smithi188 ceph-mon[44101]: pgmap v639: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-04-23T16:23:33.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:33 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3034413240' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:23:33.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:33 smithi188 ceph-mon[44101]: from='client.25108 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:23:33.814 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:33 smithi103 ceph-mon[39629]: 3.1 scrub starts 2024-04-23T16:23:33.814 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:33 smithi103 ceph-mon[39629]: pgmap v639: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-04-23T16:23:33.814 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:33 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3034413240' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:23:33.814 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:33 smithi103 ceph-mon[39629]: from='client.25108 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:23:34.363 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:34.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:34.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:34.467 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:34.478 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 68 16 144799 144783 133.062 55.875 0.00560808 0.00750854 2024-04-23T16:23:34.497 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:34.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:34 smithi103 ceph-mon[39629]: 3.a scrub starts 2024-04-23T16:23:34.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:34 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:23:34.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:34 smithi103 ceph-mon[39629]: from='client.25108 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:23:34.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:34 smithi103 ceph-mon[39629]: osdmap e221: 12 total, 12 up, 11 in 2024-04-23T16:23:34.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:34 smithi148 ceph-mon[44520]: 3.a scrub starts 2024-04-23T16:23:34.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:34 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:23:34.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:34 smithi148 ceph-mon[44520]: from='client.25108 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:23:34.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:34 smithi148 ceph-mon[44520]: osdmap e221: 12 total, 12 up, 11 in 2024-04-23T16:23:34.695 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:34.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:34.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:34.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:34 smithi188 ceph-mon[44101]: 3.a scrub starts 2024-04-23T16:23:34.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:34 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:23:34.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:34 smithi188 ceph-mon[44101]: from='client.25108 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:23:34.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:34 smithi188 ceph-mon[44101]: osdmap e221: 12 total, 12 up, 11 in 2024-04-23T16:23:34.799 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:23:34.840 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:34.866 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:23:34.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:34.871 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:23:34.944 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:23:34.944 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2024-04-23T16:23:34.945 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2024-04-23T16:23:34.945 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:23:34.945 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.945 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:60869 3.3 3:cee19e33:::benchmark_data_smithi188_89276_object60868:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:23:34.945 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:22:42.686807+0000", 2024-04-23T16:23:34.945 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 52.254061737999997, 2024-04-23T16:23:34.945 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.073128402999999995, 2024-04-23T16:23:34.945 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:34.945 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:34.945 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:34.945 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:34.945 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:34.945 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:34.945 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 60869 2024-04-23T16:23:34.945 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.946 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:34.946 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.946 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:34.946 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:42.686807+0000", 2024-04-23T16:23:34.946 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.946 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.946 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.946 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:34.946 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:42.686807+0000", 2024-04-23T16:23:34.946 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.946 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.946 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.946 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:34.947 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:42.686809+0000", 2024-04-23T16:23:34.947 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4389999999999999e-06 2024-04-23T16:23:34.947 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.947 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.947 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:34.947 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:42.686860+0000", 2024-04-23T16:23:34.947 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0519999999999997e-05 2024-04-23T16:23:34.947 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.947 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.947 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:34.947 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:42.686872+0000", 2024-04-23T16:23:34.947 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2336e-05 2024-04-23T16:23:34.947 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.948 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.948 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:34.948 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:42.686879+0000", 2024-04-23T16:23:34.948 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3289999999999996e-06 2024-04-23T16:23:34.948 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.948 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.948 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:34.948 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:42.686934+0000", 2024-04-23T16:23:34.948 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5022999999999998e-05 2024-04-23T16:23:34.948 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.948 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.948 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:34.948 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:42.687050+0000", 2024-04-23T16:23:34.948 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000115973 2024-04-23T16:23:34.949 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.949 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.949 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:23:34.949 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:42.687224+0000", 2024-04-23T16:23:34.949 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00017346500000000001 2024-04-23T16:23:34.949 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.949 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.949 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:34.949 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:42.688938+0000", 2024-04-23T16:23:34.949 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0017142959999999999 2024-04-23T16:23:34.949 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.949 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.949 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:34.949 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:42.759873+0000", 2024-04-23T16:23:34.950 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.070934340999999998 2024-04-23T16:23:34.950 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.950 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.950 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:34.950 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:42.759907+0000", 2024-04-23T16:23:34.950 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4022000000000001e-05 2024-04-23T16:23:34.950 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.950 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.950 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:34.950 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:42.759935+0000", 2024-04-23T16:23:34.950 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8659000000000001e-05 2024-04-23T16:23:34.950 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:34.950 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:34.951 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:34.951 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.951 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.951 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:63252 3.3 3:c2527cae:::benchmark_data_smithi188_89276_object63251:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:23:34.951 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:22:43.692052+0000", 2024-04-23T16:23:34.951 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 51.248816327, 2024-04-23T16:23:34.951 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.088123881000000001, 2024-04-23T16:23:34.951 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:34.951 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:34.951 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:34.951 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:34.951 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:34.951 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:34.951 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 63252 2024-04-23T16:23:34.952 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.952 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:34.952 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.952 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:34.952 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.692052+0000", 2024-04-23T16:23:34.952 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.952 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.952 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.952 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:34.952 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.692052+0000", 2024-04-23T16:23:34.952 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.952 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.952 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.953 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:34.953 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.692054+0000", 2024-04-23T16:23:34.953 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7889999999999999e-06 2024-04-23T16:23:34.953 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.953 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.953 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:34.953 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.692104+0000", 2024-04-23T16:23:34.953 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0235999999999998e-05 2024-04-23T16:23:34.953 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.953 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.953 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:34.953 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.692114+0000", 2024-04-23T16:23:34.954 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8660000000000001e-06 2024-04-23T16:23:34.954 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.954 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.954 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:34.954 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.692120+0000", 2024-04-23T16:23:34.954 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7479999999999999e-06 2024-04-23T16:23:34.954 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.954 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.954 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:34.954 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.692172+0000", 2024-04-23T16:23:34.954 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1801000000000001e-05 2024-04-23T16:23:34.954 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.954 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.954 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:34.955 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.692274+0000", 2024-04-23T16:23:34.955 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000102398 2024-04-23T16:23:34.955 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.955 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.955 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:23:34.955 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.692363+0000", 2024-04-23T16:23:34.955 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8979000000000003e-05 2024-04-23T16:23:34.955 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.955 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.955 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:34.955 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.694529+0000", 2024-04-23T16:23:34.955 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021654389999999999 2024-04-23T16:23:34.955 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.955 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.956 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:34.956 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.780076+0000", 2024-04-23T16:23:34.956 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.085547559999999995 2024-04-23T16:23:34.956 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.956 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.956 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:34.956 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.780134+0000", 2024-04-23T16:23:34.956 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8007999999999998e-05 2024-04-23T16:23:34.956 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.956 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.956 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:34.956 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.780176+0000", 2024-04-23T16:23:34.956 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2057000000000001e-05 2024-04-23T16:23:34.956 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:34.956 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:34.957 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:34.957 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.957 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.957 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:63604 3.3 3:c96b3782:::benchmark_data_smithi188_89276_object63603:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:23:34.957 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:22:43.905851+0000", 2024-04-23T16:23:34.957 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 51.035017228999997, 2024-04-23T16:23:34.957 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.050905763999999999, 2024-04-23T16:23:34.957 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:34.958 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:34.958 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:34.958 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:34.958 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:34.958 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:34.958 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 63604 2024-04-23T16:23:34.958 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.958 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:34.958 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.958 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:34.958 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.905851+0000", 2024-04-23T16:23:34.958 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.958 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.959 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.959 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:34.959 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.905851+0000", 2024-04-23T16:23:34.959 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.959 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.959 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.959 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:34.959 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.905853+0000", 2024-04-23T16:23:34.959 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8530000000000001e-06 2024-04-23T16:23:34.959 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.959 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.959 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:34.959 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.905892+0000", 2024-04-23T16:23:34.959 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8900999999999999e-05 2024-04-23T16:23:34.960 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.960 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.960 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:34.960 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.905901+0000", 2024-04-23T16:23:34.960 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0189999999999995e-06 2024-04-23T16:23:34.960 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.960 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.960 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:34.960 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.905907+0000", 2024-04-23T16:23:34.960 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7939999999999999e-06 2024-04-23T16:23:34.960 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.960 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.960 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:34.960 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.905951+0000", 2024-04-23T16:23:34.961 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4354999999999998e-05 2024-04-23T16:23:34.961 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.961 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.961 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:34.961 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.906044+0000", 2024-04-23T16:23:34.961 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2845999999999998e-05 2024-04-23T16:23:34.961 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.961 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.961 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:23:34.961 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.906127+0000", 2024-04-23T16:23:34.961 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3010000000000007e-05 2024-04-23T16:23:34.961 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.961 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.961 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:34.961 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.907525+0000", 2024-04-23T16:23:34.962 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001397744 2024-04-23T16:23:34.962 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.962 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.962 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:34.962 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.956662+0000", 2024-04-23T16:23:34.962 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.049136694000000002 2024-04-23T16:23:34.962 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.962 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.962 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:34.962 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.956715+0000", 2024-04-23T16:23:34.962 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3226000000000002e-05 2024-04-23T16:23:34.962 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.962 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.963 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:34.963 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.956757+0000", 2024-04-23T16:23:34.963 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2321999999999999e-05 2024-04-23T16:23:34.963 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:34.963 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:34.963 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:34.963 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.963 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.963 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:66736 3.3 3:c36fcd9b:::benchmark_data_smithi188_89276_object66735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:23:34.963 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:22:45.178169+0000", 2024-04-23T16:23:34.963 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 49.762699515999998, 2024-04-23T16:23:34.963 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052144429999999999, 2024-04-23T16:23:34.963 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:34.963 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:34.963 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:34.964 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:34.964 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:34.964 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:34.964 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 66736 2024-04-23T16:23:34.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.964 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:34.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.964 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:34.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.178169+0000", 2024-04-23T16:23:34.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.964 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:34.965 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.178169+0000", 2024-04-23T16:23:34.965 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.965 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:34.965 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.178171+0000", 2024-04-23T16:23:34.965 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9250000000000002e-06 2024-04-23T16:23:34.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.965 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:34.965 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.178219+0000", 2024-04-23T16:23:34.965 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8239999999999999e-05 2024-04-23T16:23:34.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.966 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:34.966 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.178230+0000", 2024-04-23T16:23:34.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0862999999999999e-05 2024-04-23T16:23:34.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.966 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:34.966 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.178238+0000", 2024-04-23T16:23:34.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8329999999999994e-06 2024-04-23T16:23:34.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.966 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:34.966 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.178297+0000", 2024-04-23T16:23:34.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9264e-05 2024-04-23T16:23:34.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.967 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.967 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:34.967 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.178422+0000", 2024-04-23T16:23:34.967 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000125017 2024-04-23T16:23:34.967 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.967 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.967 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:23:34.967 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.178569+0000", 2024-04-23T16:23:34.967 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014658800000000001 2024-04-23T16:23:34.967 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.967 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.967 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:34.967 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.180607+0000", 2024-04-23T16:23:34.967 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0020382479999999999 2024-04-23T16:23:34.968 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.968 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:34.968 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.230183+0000", 2024-04-23T16:23:34.968 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.049575793 2024-04-23T16:23:34.968 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.968 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:34.968 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.230283+0000", 2024-04-23T16:23:34.968 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9720000000000001e-05 2024-04-23T16:23:34.968 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.968 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:34.968 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.230314+0000", 2024-04-23T16:23:34.969 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0939000000000003e-05 2024-04-23T16:23:34.969 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:34.969 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:34.969 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:34.969 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.969 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.969 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:66750 3.3 3:cf57c817:::benchmark_data_smithi188_89276_object66749:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:23:34.969 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:22:45.184867+0000", 2024-04-23T16:23:34.969 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 49.756001245, 2024-04-23T16:23:34.969 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.050120623000000003, 2024-04-23T16:23:34.969 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:34.969 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:34.969 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:34.969 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:34.970 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:34.970 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:34.970 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 66750 2024-04-23T16:23:34.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.970 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:34.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:34.970 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.184867+0000", 2024-04-23T16:23:34.970 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:34.970 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.184867+0000", 2024-04-23T16:23:34.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:34.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.184869+0000", 2024-04-23T16:23:34.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9489999999999999e-06 2024-04-23T16:23:34.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:34.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.184912+0000", 2024-04-23T16:23:34.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2790999999999997e-05 2024-04-23T16:23:34.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:34.972 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.184923+0000", 2024-04-23T16:23:34.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0634e-05 2024-04-23T16:23:34.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.972 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:34.972 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.184929+0000", 2024-04-23T16:23:34.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3500000000000002e-06 2024-04-23T16:23:34.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.972 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:34.972 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.184970+0000", 2024-04-23T16:23:34.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1158000000000001e-05 2024-04-23T16:23:34.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.973 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:34.973 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.185055+0000", 2024-04-23T16:23:34.973 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5004000000000001e-05 2024-04-23T16:23:34.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.973 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:23:34.973 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.185152+0000", 2024-04-23T16:23:34.973 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6358000000000001e-05 2024-04-23T16:23:34.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.973 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:34.973 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.186660+0000", 2024-04-23T16:23:34.973 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001508199 2024-04-23T16:23:34.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:34.974 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.234889+0000", 2024-04-23T16:23:34.974 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048229460000000002 2024-04-23T16:23:34.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:34.974 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.234938+0000", 2024-04-23T16:23:34.974 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8625999999999998e-05 2024-04-23T16:23:34.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:34.975 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:45.234988+0000", 2024-04-23T16:23:34.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0093999999999998e-05 2024-04-23T16:23:34.975 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:34.975 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:34.975 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:34.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.975 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:71545 3.3 3:cfaa1cd7:::benchmark_data_smithi188_89276_object71544:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:23:34.975 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:22:47.220841+0000", 2024-04-23T16:23:34.975 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 47.720027758000001, 2024-04-23T16:23:34.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.062740046999999993, 2024-04-23T16:23:34.975 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:34.975 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:34.975 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:34.976 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:34.976 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:34.976 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:34.976 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 71545 2024-04-23T16:23:34.976 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.976 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:34.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:34.976 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:47.220841+0000", 2024-04-23T16:23:34.976 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.976 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:34.976 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:47.220841+0000", 2024-04-23T16:23:34.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.977 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:34.977 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:47.220843+0000", 2024-04-23T16:23:34.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.855e-06 2024-04-23T16:23:34.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.977 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:34.977 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:47.220952+0000", 2024-04-23T16:23:34.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000108835 2024-04-23T16:23:34.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.977 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:34.977 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:47.220963+0000", 2024-04-23T16:23:34.978 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1280999999999999e-05 2024-04-23T16:23:34.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.978 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.978 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:34.978 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:47.220969+0000", 2024-04-23T16:23:34.978 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3119999999999999e-06 2024-04-23T16:23:34.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.978 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.978 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:34.978 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:47.221018+0000", 2024-04-23T16:23:34.978 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9128000000000001e-05 2024-04-23T16:23:34.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.978 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.979 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:34.979 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:47.221125+0000", 2024-04-23T16:23:34.979 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107111 2024-04-23T16:23:34.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.979 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.979 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:23:34.979 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:47.221207+0000", 2024-04-23T16:23:34.979 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1773000000000005e-05 2024-04-23T16:23:34.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.979 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.979 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:34.979 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:47.223365+0000", 2024-04-23T16:23:34.979 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00215766 2024-04-23T16:23:34.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.979 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.980 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:34.980 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:47.283519+0000", 2024-04-23T16:23:34.980 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.060154249 2024-04-23T16:23:34.980 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.980 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.980 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:34.980 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:47.283553+0000", 2024-04-23T16:23:34.980 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3560999999999999e-05 2024-04-23T16:23:34.980 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.980 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.980 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:34.980 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:47.283581+0000", 2024-04-23T16:23:34.980 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8282000000000001e-05 2024-04-23T16:23:34.980 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:34.980 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:34.981 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:34.981 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.981 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.981 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:75229 3.3 3:ce7b8e43:::benchmark_data_smithi188_89276_object75228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:23:34.981 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:22:48.885135+0000", 2024-04-23T16:23:34.981 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 46.055733678999999, 2024-04-23T16:23:34.981 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.061926850999999998, 2024-04-23T16:23:34.981 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:34.981 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:34.981 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:34.981 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:34.981 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:34.981 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:34.981 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 75229 2024-04-23T16:23:34.981 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.982 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:34.982 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.982 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:34.982 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.885135+0000", 2024-04-23T16:23:34.982 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.982 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.982 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.982 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:34.982 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.885135+0000", 2024-04-23T16:23:34.982 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.982 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.982 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.982 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:34.982 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.885137+0000", 2024-04-23T16:23:34.982 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7770000000000001e-06 2024-04-23T16:23:34.983 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.983 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.983 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:34.983 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.885189+0000", 2024-04-23T16:23:34.983 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2672e-05 2024-04-23T16:23:34.983 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.983 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.983 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:34.983 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.885202+0000", 2024-04-23T16:23:34.983 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2948999999999999e-05 2024-04-23T16:23:34.983 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.983 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.983 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:34.983 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.885211+0000", 2024-04-23T16:23:34.983 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.49e-06 2024-04-23T16:23:34.983 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.984 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:34.984 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.885259+0000", 2024-04-23T16:23:34.984 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7821999999999997e-05 2024-04-23T16:23:34.984 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.984 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:34.984 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.885375+0000", 2024-04-23T16:23:34.984 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000115931 2024-04-23T16:23:34.984 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.984 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:23:34.984 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.885514+0000", 2024-04-23T16:23:34.984 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013971799999999999 2024-04-23T16:23:34.984 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.985 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.985 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:34.985 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.887510+0000", 2024-04-23T16:23:34.985 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0019955559999999999 2024-04-23T16:23:34.985 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.985 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.985 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:34.985 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.946975+0000", 2024-04-23T16:23:34.985 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.059465664000000001 2024-04-23T16:23:34.985 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.985 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.985 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:34.985 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.947018+0000", 2024-04-23T16:23:34.985 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2064000000000002e-05 2024-04-23T16:23:34.986 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.986 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.986 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:34.986 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.947062+0000", 2024-04-23T16:23:34.986 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4208000000000001e-05 2024-04-23T16:23:34.986 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:34.986 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:34.986 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:34.986 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.986 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.986 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:75238 3.3 3:cdcb0844:::benchmark_data_smithi188_89276_object75237:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:23:34.986 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:22:48.889864+0000", 2024-04-23T16:23:34.986 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 46.051004804000002, 2024-04-23T16:23:34.986 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.057350877000000001, 2024-04-23T16:23:34.987 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:34.987 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:34.987 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:34.987 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:34.987 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:34.987 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:34.987 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 75238 2024-04-23T16:23:34.987 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.987 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:34.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:34.987 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.889864+0000", 2024-04-23T16:23:34.987 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.987 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.988 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:34.988 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.889864+0000", 2024-04-23T16:23:34.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.988 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:34.988 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.889866+0000", 2024-04-23T16:23:34.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7940000000000001e-06 2024-04-23T16:23:34.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.988 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:34.988 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.889914+0000", 2024-04-23T16:23:34.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8823000000000003e-05 2024-04-23T16:23:34.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.989 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:34.989 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.889925+0000", 2024-04-23T16:23:34.989 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0149000000000001e-05 2024-04-23T16:23:34.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.989 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:34.989 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.889930+0000", 2024-04-23T16:23:34.989 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7389999999999996e-06 2024-04-23T16:23:34.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.989 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:34.989 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.889981+0000", 2024-04-23T16:23:34.989 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0377999999999998e-05 2024-04-23T16:23:34.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:34.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.890118+0000", 2024-04-23T16:23:34.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013694900000000001 2024-04-23T16:23:34.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:23:34.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.890263+0000", 2024-04-23T16:23:34.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014581500000000001 2024-04-23T16:23:34.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:34.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.893404+0000", 2024-04-23T16:23:34.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0031403080000000001 2024-04-23T16:23:34.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.991 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:34.991 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.947122+0000", 2024-04-23T16:23:34.991 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.053718464 2024-04-23T16:23:34.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.991 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:34.991 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.947181+0000", 2024-04-23T16:23:34.991 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9259000000000003e-05 2024-04-23T16:23:34.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.991 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:34.991 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.947215+0000", 2024-04-23T16:23:34.991 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3198999999999998e-05 2024-04-23T16:23:34.992 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:34.992 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:34.992 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:34.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.992 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:75240 3.3 3:c5586364:::benchmark_data_smithi188_89276_object75239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:23:34.992 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:22:48.889943+0000", 2024-04-23T16:23:34.992 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 46.050925231000001, 2024-04-23T16:23:34.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.057389961000000003, 2024-04-23T16:23:34.992 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:34.992 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:34.992 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:34.992 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:34.992 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:34.993 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:34.993 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 75240 2024-04-23T16:23:34.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.993 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:34.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.993 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:34.993 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.889943+0000", 2024-04-23T16:23:34.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.993 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:34.993 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.889943+0000", 2024-04-23T16:23:34.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.993 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:34.994 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.889944+0000", 2024-04-23T16:23:34.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3600000000000003e-07 2024-04-23T16:23:34.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.994 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:34.994 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.890030+0000", 2024-04-23T16:23:34.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5834000000000005e-05 2024-04-23T16:23:34.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.994 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:34.994 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.890035+0000", 2024-04-23T16:23:34.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6330000000000001e-06 2024-04-23T16:23:34.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:34.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.890040+0000", 2024-04-23T16:23:34.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5449999999999997e-06 2024-04-23T16:23:34.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:34.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.891888+0000", 2024-04-23T16:23:34.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00184826 2024-04-23T16:23:34.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:34.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.892012+0000", 2024-04-23T16:23:34.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012400700000000001 2024-04-23T16:23:34.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.996 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:23:34.996 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.892131+0000", 2024-04-23T16:23:34.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011835799999999999 2024-04-23T16:23:34.996 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.996 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.996 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:34.996 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.895100+0000", 2024-04-23T16:23:34.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0029696420000000002 2024-04-23T16:23:34.996 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.996 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.996 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:34.996 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.947258+0000", 2024-04-23T16:23:34.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052157235000000003 2024-04-23T16:23:34.996 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.996 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.997 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:34.997 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.947294+0000", 2024-04-23T16:23:34.997 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6195999999999997e-05 2024-04-23T16:23:34.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.997 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:34.997 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.947333+0000", 2024-04-23T16:23:34.997 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9614999999999999e-05 2024-04-23T16:23:34.997 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:34.997 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:34.997 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:34.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.997 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:75242 3.3 3:cc5c119b:::benchmark_data_smithi188_89276_object75241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:23:34.997 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:22:48.891885+0000", 2024-04-23T16:23:34.998 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 46.048984050999998, 2024-04-23T16:23:34.998 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.055608009999999999, 2024-04-23T16:23:34.998 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:34.998 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:34.998 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:34.998 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:34.998 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:34.998 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:34.998 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 75242 2024-04-23T16:23:34.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.998 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:34.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.998 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:34.999 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.891885+0000", 2024-04-23T16:23:34.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.999 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:34.999 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.891885+0000", 2024-04-23T16:23:34.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:34.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.999 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:34.999 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.891886+0000", 2024-04-23T16:23:34.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5200000000000001e-06 2024-04-23T16:23:34.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:34.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:34.999 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:35.000 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.891938+0000", 2024-04-23T16:23:35.000 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2128e-05 2024-04-23T16:23:35.000 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.000 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:35.000 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.891946+0000", 2024-04-23T16:23:35.000 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3119999999999994e-06 2024-04-23T16:23:35.000 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.000 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:35.000 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.891955+0000", 2024-04-23T16:23:35.000 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7399999999999993e-06 2024-04-23T16:23:35.000 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.001 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:35.001 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.893434+0000", 2024-04-23T16:23:35.001 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001478743 2024-04-23T16:23:35.001 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.001 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.001 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:35.001 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.893552+0000", 2024-04-23T16:23:35.001 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011831 2024-04-23T16:23:35.001 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.001 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.001 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:23:35.001 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.893688+0000", 2024-04-23T16:23:35.001 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013561500000000001 2024-04-23T16:23:35.001 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.001 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.002 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:35.002 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.895221+0000", 2024-04-23T16:23:35.002 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001533118 2024-04-23T16:23:35.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.002 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.002 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:35.002 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.947390+0000", 2024-04-23T16:23:35.002 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052169396 2024-04-23T16:23:35.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.002 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.002 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:35.002 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.947451+0000", 2024-04-23T16:23:35.002 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1067000000000003e-05 2024-04-23T16:23:35.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.002 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:35.003 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.947493+0000", 2024-04-23T16:23:35.003 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1060999999999999e-05 2024-04-23T16:23:35.003 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.003 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:35.003 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.003 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.003 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:75244 3.3 3:c85cdb43:::benchmark_data_smithi188_89276_object75243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:23:35.003 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:22:48.895318+0000", 2024-04-23T16:23:35.003 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 46.045550607999999, 2024-04-23T16:23:35.003 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052269285999999998, 2024-04-23T16:23:35.003 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:35.003 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:35.004 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:35.004 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:35.004 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:35.004 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:35.004 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 75244 2024-04-23T16:23:35.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.004 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:35.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.004 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:35.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.895318+0000", 2024-04-23T16:23:35.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.004 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:35.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.895318+0000", 2024-04-23T16:23:35.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:35.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.895320+0000", 2024-04-23T16:23:35.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6470000000000001e-06 2024-04-23T16:23:35.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:35.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.895370+0000", 2024-04-23T16:23:35.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9968e-05 2024-04-23T16:23:35.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:35.006 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.895380+0000", 2024-04-23T16:23:35.006 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0484e-05 2024-04-23T16:23:35.006 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.006 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.006 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:35.006 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.895388+0000", 2024-04-23T16:23:35.006 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0900000000000005e-06 2024-04-23T16:23:35.006 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.006 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.006 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:35.006 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.895422+0000", 2024-04-23T16:23:35.006 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4107999999999999e-05 2024-04-23T16:23:35.006 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.007 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:35.007 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.895547+0000", 2024-04-23T16:23:35.007 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000124313 2024-04-23T16:23:35.007 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.007 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:23:35.007 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.895639+0000", 2024-04-23T16:23:35.007 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2577000000000005e-05 2024-04-23T16:23:35.007 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.007 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:35.007 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.896704+0000", 2024-04-23T16:23:35.007 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00106499 2024-04-23T16:23:35.007 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.008 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:35.008 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.947530+0000", 2024-04-23T16:23:35.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.050825639999999998 2024-04-23T16:23:35.008 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.008 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.008 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:35.008 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.947564+0000", 2024-04-23T16:23:35.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3770999999999999e-05 2024-04-23T16:23:35.008 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.008 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.008 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:35.008 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:48.947587+0000", 2024-04-23T16:23:35.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3697999999999999e-05 2024-04-23T16:23:35.008 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.009 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:35.009 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.009 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.009 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.009 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.15150.0:80497 3.a e204/192 3:5e3f3a39:::benchmark_data_smithi188_89276_object80496:head v 204'4990, mlcod=204'4987)", 2024-04-23T16:23:35.009 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:22:51.363429+0000", 2024-04-23T16:23:35.009 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 43.577439423999998, 2024-04-23T16:23:35.009 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.063185993999999995, 2024-04-23T16:23:35.009 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:35.009 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:35.009 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:35.009 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:35.009 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.009 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:35.009 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363429+0000", 2024-04-23T16:23:35.009 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.010 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.010 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:35.010 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363429+0000", 2024-04-23T16:23:35.010 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.010 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.010 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:35.010 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363430+0000", 2024-04-23T16:23:35.010 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1149999999999999e-06 2024-04-23T16:23:35.010 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.010 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:35.010 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363456+0000", 2024-04-23T16:23:35.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5877e-05 2024-04-23T16:23:35.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.011 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.011 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:35.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363464+0000", 2024-04-23T16:23:35.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5109999999999997e-06 2024-04-23T16:23:35.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.011 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.011 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:35.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363468+0000", 2024-04-23T16:23:35.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6609999999999999e-06 2024-04-23T16:23:35.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.011 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.011 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:35.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.419257+0000", 2024-04-23T16:23:35.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.055788928000000002 2024-04-23T16:23:35.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:35.012 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.419284+0000", 2024-04-23T16:23:35.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6866e-05 2024-04-23T16:23:35.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:35.012 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.426565+0000", 2024-04-23T16:23:35.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0072812069999999996 2024-04-23T16:23:35.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:35.012 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.426615+0000", 2024-04-23T16:23:35.013 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9829e-05 2024-04-23T16:23:35.013 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.013 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:35.013 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.013 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.013 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.013 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.15150.0:80498 3.a e204/192 3:5a5eb108:::benchmark_data_smithi188_89276_object80497:head v 204'4991, mlcod=204'4987)", 2024-04-23T16:23:35.013 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:22:51.364784+0000", 2024-04-23T16:23:35.013 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 43.57608415, 2024-04-23T16:23:35.013 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.061860025999999999, 2024-04-23T16:23:35.013 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:35.013 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:35.013 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:35.013 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:35.014 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.014 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:35.014 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.364784+0000", 2024-04-23T16:23:35.014 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.014 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.014 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.014 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:35.014 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.364784+0000", 2024-04-23T16:23:35.014 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.014 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.014 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.014 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:35.014 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.364785+0000", 2024-04-23T16:23:35.014 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8700000000000004e-07 2024-04-23T16:23:35.015 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.015 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.015 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:35.015 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.364800+0000", 2024-04-23T16:23:35.015 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4972e-05 2024-04-23T16:23:35.015 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.015 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.015 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:35.015 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.419252+0000", 2024-04-23T16:23:35.015 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.054451635999999998 2024-04-23T16:23:35.015 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.015 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.015 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:35.015 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.419276+0000", 2024-04-23T16:23:35.015 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4144000000000001e-05 2024-04-23T16:23:35.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.016 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:35.016 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.421439+0000", 2024-04-23T16:23:35.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021627909999999998 2024-04-23T16:23:35.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.016 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:35.016 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.421474+0000", 2024-04-23T16:23:35.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4910000000000003e-05 2024-04-23T16:23:35.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.016 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:35.016 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.426621+0000", 2024-04-23T16:23:35.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0051468369999999996 2024-04-23T16:23:35.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.017 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:35.017 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.426644+0000", 2024-04-23T16:23:35.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3849000000000002e-05 2024-04-23T16:23:35.017 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.017 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:35.017 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.017 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:125828 3.3 3:c1b74123:::benchmark_data_smithi188_89276_object125827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e215)", 2024-04-23T16:23:35.017 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:18.535876+0000", 2024-04-23T16:23:35.017 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 16.404992704000001, 2024-04-23T16:23:35.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.062712880999999998, 2024-04-23T16:23:35.018 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:35.018 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:35.018 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:35.018 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:35.018 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:35.018 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:35.018 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 125828 2024-04-23T16:23:35.018 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.018 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:35.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.018 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:35.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.535876+0000", 2024-04-23T16:23:35.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.018 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.019 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:35.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.535876+0000", 2024-04-23T16:23:35.019 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.019 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.019 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:35.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.535878+0000", 2024-04-23T16:23:35.019 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.996e-06 2024-04-23T16:23:35.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.019 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.019 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:35.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.535924+0000", 2024-04-23T16:23:35.019 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.613e-05 2024-04-23T16:23:35.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.020 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.020 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:35.020 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.535933+0000", 2024-04-23T16:23:35.020 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.4579999999999999e-06 2024-04-23T16:23:35.020 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.020 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.020 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:35.020 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.535941+0000", 2024-04-23T16:23:35.020 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.661e-06 2024-04-23T16:23:35.020 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.020 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.020 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:35.020 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.536004+0000", 2024-04-23T16:23:35.020 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3002000000000004e-05 2024-04-23T16:23:35.020 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.021 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.021 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:35.021 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.536141+0000", 2024-04-23T16:23:35.021 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000136596 2024-04-23T16:23:35.021 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.021 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.021 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:23:35.021 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.536285+0000", 2024-04-23T16:23:35.021 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014390799999999999 2024-04-23T16:23:35.021 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.021 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.021 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:35.021 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.542829+0000", 2024-04-23T16:23:35.021 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0065442850000000004 2024-04-23T16:23:35.022 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.022 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.022 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:35.022 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.598500+0000", 2024-04-23T16:23:35.022 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.055671251999999997 2024-04-23T16:23:35.022 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.022 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.022 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:35.022 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.598548+0000", 2024-04-23T16:23:35.022 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8270999999999999e-05 2024-04-23T16:23:35.022 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.022 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.022 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:35.022 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.598589+0000", 2024-04-23T16:23:35.023 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0321999999999998e-05 2024-04-23T16:23:35.023 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.023 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:35.023 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.023 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:125831 3.3 3:cce3e511:::benchmark_data_smithi188_89276_object125830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e215)", 2024-04-23T16:23:35.023 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:18.537831+0000", 2024-04-23T16:23:35.023 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 16.40303767, 2024-04-23T16:23:35.023 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.060866301999999997, 2024-04-23T16:23:35.023 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:35.023 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:35.023 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:35.023 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:35.024 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:35.024 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:35.024 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 125831 2024-04-23T16:23:35.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.024 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:35.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.024 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:35.024 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.537831+0000", 2024-04-23T16:23:35.024 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.024 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:35.024 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.537831+0000", 2024-04-23T16:23:35.024 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.025 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:35.025 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.537832+0000", 2024-04-23T16:23:35.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.55e-06 2024-04-23T16:23:35.025 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.025 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:35.025 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.537889+0000", 2024-04-23T16:23:35.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6657000000000003e-05 2024-04-23T16:23:35.025 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.025 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:35.025 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.537898+0000", 2024-04-23T16:23:35.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6799999999999999e-06 2024-04-23T16:23:35.025 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.025 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:35.026 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.537904+0000", 2024-04-23T16:23:35.026 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3980000000000004e-06 2024-04-23T16:23:35.026 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.026 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:35.026 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.539583+0000", 2024-04-23T16:23:35.026 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001678398 2024-04-23T16:23:35.026 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.026 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:35.026 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.539704+0000", 2024-04-23T16:23:35.026 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000121551 2024-04-23T16:23:35.026 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:23:35.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.539830+0000", 2024-04-23T16:23:35.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012540000000000001 2024-04-23T16:23:35.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:35.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.545894+0000", 2024-04-23T16:23:35.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0060645849999999999 2024-04-23T16:23:35.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:35.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.598633+0000", 2024-04-23T16:23:35.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052739015 2024-04-23T16:23:35.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.028 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:35.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.598669+0000", 2024-04-23T16:23:35.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5559999999999998e-05 2024-04-23T16:23:35.028 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.028 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:35.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.598697+0000", 2024-04-23T16:23:35.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8507999999999999e-05 2024-04-23T16:23:35.028 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.028 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:35.028 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.028 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.028 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.029 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:125835 3.3 3:cfdbe151:::benchmark_data_smithi188_89276_object125834:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e215)", 2024-04-23T16:23:35.029 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:18.540025+0000", 2024-04-23T16:23:35.029 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 16.400843594000001, 2024-04-23T16:23:35.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.058772532000000002, 2024-04-23T16:23:35.029 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:35.029 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:35.029 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:35.029 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:35.029 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:35.029 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:35.029 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 125835 2024-04-23T16:23:35.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.029 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:35.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.029 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:35.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.540025+0000", 2024-04-23T16:23:35.030 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:35.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.540025+0000", 2024-04-23T16:23:35.030 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:35.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.540027+0000", 2024-04-23T16:23:35.030 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6440000000000001e-06 2024-04-23T16:23:35.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:35.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.540062+0000", 2024-04-23T16:23:35.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.561e-05 2024-04-23T16:23:35.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.031 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.031 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:35.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.540072+0000", 2024-04-23T16:23:35.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5950000000000005e-06 2024-04-23T16:23:35.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.031 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.031 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:35.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.540079+0000", 2024-04-23T16:23:35.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.267e-06 2024-04-23T16:23:35.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.031 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.031 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:35.032 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.542859+0000", 2024-04-23T16:23:35.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002780233 2024-04-23T16:23:35.032 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.032 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:35.032 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.542971+0000", 2024-04-23T16:23:35.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011206599999999999 2024-04-23T16:23:35.032 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.032 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:23:35.032 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.543086+0000", 2024-04-23T16:23:35.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000114728 2024-04-23T16:23:35.032 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:35.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.547638+0000", 2024-04-23T16:23:35.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045517869999999998 2024-04-23T16:23:35.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:35.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.598733+0000", 2024-04-23T16:23:35.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.051095094000000001 2024-04-23T16:23:35.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:35.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.598766+0000", 2024-04-23T16:23:35.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2644000000000001e-05 2024-04-23T16:23:35.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.034 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:35.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:18.598798+0000", 2024-04-23T16:23:35.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1863999999999998e-05 2024-04-23T16:23:35.034 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.034 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:35.034 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.034 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:141030 3.3 3:c1fd4ca9:::benchmark_data_smithi188_89276_object141029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-04-23T16:23:35.034 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:30.195998+0000", 2024-04-23T16:23:35.034 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.7448704230000001, 2024-04-23T16:23:35.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.064838245000000003, 2024-04-23T16:23:35.034 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:35.034 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:35.034 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:35.034 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:35.035 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:35.035 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:35.035 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 141030 2024-04-23T16:23:35.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.035 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:35.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:35.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.195998+0000", 2024-04-23T16:23:35.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:35.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.195998+0000", 2024-04-23T16:23:35.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:35.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.196004+0000", 2024-04-23T16:23:35.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5269999999999998e-06 2024-04-23T16:23:35.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:35.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.196060+0000", 2024-04-23T16:23:35.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.66e-05 2024-04-23T16:23:35.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:35.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.196072+0000", 2024-04-23T16:23:35.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1442e-05 2024-04-23T16:23:35.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:35.037 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.196080+0000", 2024-04-23T16:23:35.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8259999999999993e-06 2024-04-23T16:23:35.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:35.037 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.252434+0000", 2024-04-23T16:23:35.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.056354516 2024-04-23T16:23:35.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:35.037 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.252598+0000", 2024-04-23T16:23:35.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000163628 2024-04-23T16:23:35.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:23:35.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.252768+0000", 2024-04-23T16:23:35.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00016996700000000001 2024-04-23T16:23:35.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:35.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.260711+0000", 2024-04-23T16:23:35.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0079429770000000004 2024-04-23T16:23:35.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:35.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.260724+0000", 2024-04-23T16:23:35.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3190000000000001e-05 2024-04-23T16:23:35.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:35.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.260777+0000", 2024-04-23T16:23:35.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2879e-05 2024-04-23T16:23:35.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:35.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.260836+0000", 2024-04-23T16:23:35.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9692999999999999e-05 2024-04-23T16:23:35.039 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.039 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:35.039 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.039 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:141032 3.3 3:cca68cb9:::benchmark_data_smithi188_89276_object141031:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-04-23T16:23:35.040 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:30.196813+0000", 2024-04-23T16:23:35.040 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.7440560039999999, 2024-04-23T16:23:35.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.071604485999999995, 2024-04-23T16:23:35.040 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:35.040 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:35.040 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:35.040 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:35.040 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:35.040 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:35.040 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 141032 2024-04-23T16:23:35.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.040 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:35.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:35.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.196813+0000", 2024-04-23T16:23:35.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:35.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.196813+0000", 2024-04-23T16:23:35.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:35.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.196814+0000", 2024-04-23T16:23:35.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.31e-06 2024-04-23T16:23:35.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:35.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.196867+0000", 2024-04-23T16:23:35.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2729999999999998e-05 2024-04-23T16:23:35.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:35.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.196877+0000", 2024-04-23T16:23:35.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0557000000000001e-05 2024-04-23T16:23:35.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:35.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.196883+0000", 2024-04-23T16:23:35.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3229999999999997e-06 2024-04-23T16:23:35.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:35.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.256987+0000", 2024-04-23T16:23:35.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.060104989999999997 2024-04-23T16:23:35.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.043 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:35.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.257116+0000", 2024-04-23T16:23:35.043 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012839000000000001 2024-04-23T16:23:35.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.043 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:23:35.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.257264+0000", 2024-04-23T16:23:35.043 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014825 2024-04-23T16:23:35.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.043 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:35.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.264274+0000", 2024-04-23T16:23:35.043 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0070101410000000001 2024-04-23T16:23:35.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:35.044 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.268297+0000", 2024-04-23T16:23:35.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0040227689999999998 2024-04-23T16:23:35.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:35.044 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.268359+0000", 2024-04-23T16:23:35.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2361999999999997e-05 2024-04-23T16:23:35.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:35.044 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.268417+0000", 2024-04-23T16:23:35.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7664000000000002e-05 2024-04-23T16:23:35.045 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.045 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:35.045 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.045 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.15150.0:141033 3.e e219/192 3:734da6ac:::benchmark_data_smithi188_89276_object141032:head v 219'8819, mlcod=219'8817)", 2024-04-23T16:23:35.045 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:30.198716+0000", 2024-04-23T16:23:35.045 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.7421526170000003, 2024-04-23T16:23:35.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.059733215999999999, 2024-04-23T16:23:35.045 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:35.045 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:35.045 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:35.045 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:35.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:35.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.198716+0000", 2024-04-23T16:23:35.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.046 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:35.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.198716+0000", 2024-04-23T16:23:35.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.046 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:35.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.198718+0000", 2024-04-23T16:23:35.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.72e-06 2024-04-23T16:23:35.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:35.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.198772+0000", 2024-04-23T16:23:35.047 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4199000000000001e-05 2024-04-23T16:23:35.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:35.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.249974+0000", 2024-04-23T16:23:35.047 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.051201714000000002 2024-04-23T16:23:35.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:35.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.249986+0000", 2024-04-23T16:23:35.047 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2172e-05 2024-04-23T16:23:35.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.048 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.048 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:35.048 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.250136+0000", 2024-04-23T16:23:35.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015056600000000001 2024-04-23T16:23:35.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.048 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.048 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:35.048 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.250161+0000", 2024-04-23T16:23:35.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4556999999999999e-05 2024-04-23T16:23:35.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.048 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.048 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:35.048 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.258394+0000", 2024-04-23T16:23:35.049 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0082331820000000003 2024-04-23T16:23:35.049 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.049 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.049 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:35.049 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.258449+0000", 2024-04-23T16:23:35.049 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5105999999999997e-05 2024-04-23T16:23:35.049 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.049 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:35.049 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.049 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.049 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.049 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.15150.0:141369 3.1 e220/192 3:82854a72:::benchmark_data_smithi188_89276_object141368:head v 220'8763, mlcod=220'8761)", 2024-04-23T16:23:35.049 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:30.558176+0000", 2024-04-23T16:23:35.049 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.3826929549999996, 2024-04-23T16:23:35.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.070937597000000005, 2024-04-23T16:23:35.050 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:35.050 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:35.050 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:35.050 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:35.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:35.050 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.558176+0000", 2024-04-23T16:23:35.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:35.050 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.558176+0000", 2024-04-23T16:23:35.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:35.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:35.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.558177+0000", 2024-04-23T16:23:35.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8299999999999995e-07 2024-04-23T16:23:35.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:35.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.558245+0000", 2024-04-23T16:23:35.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8381999999999997e-05 2024-04-23T16:23:35.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:35.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.620698+0000", 2024-04-23T16:23:35.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.062452896000000001 2024-04-23T16:23:35.052 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.052 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:35.052 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.620712+0000", 2024-04-23T16:23:35.052 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4182000000000001e-05 2024-04-23T16:23:35.052 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.052 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:35.052 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.621280+0000", 2024-04-23T16:23:35.052 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00056819499999999996 2024-04-23T16:23:35.052 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.052 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:35.052 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.621297+0000", 2024-04-23T16:23:35.053 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7088999999999999e-05 2024-04-23T16:23:35.053 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.053 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.053 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:35.053 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.629059+0000", 2024-04-23T16:23:35.053 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0077618130000000002 2024-04-23T16:23:35.053 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:35.053 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:35.053 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:35.053 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:30.629113+0000", 2024-04-23T16:23:35.053 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4057000000000001e-05 2024-04-23T16:23:35.054 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.054 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:35.054 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.054 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:35.054 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:35.054 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:23:35.054 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:23:35.055 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:23:35.074 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:23:35.096 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:35.098 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:23:35.124 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:23:35.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:35.127 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:23:35.128 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:23:35.159 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:23:35.185 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:23:35.212 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:23:35.228 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:23:35.252 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:23:35.282 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:23:35.283 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:35.314 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:23:35.347 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:23:35.382 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:23:35.384 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:23:35.384 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:23:35.384 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:23:35.384 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:23:35.384 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:23:35.384 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:23:35.384 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:23:35.384 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:23:35.385 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:23:35.385 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:23:35.385 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:23:35.385 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:23:35.479 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 69 16 145606 145590 131.863 50.4375 0.0794525 0.00758029 2024-04-23T16:23:35.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:35 smithi103 ceph-mon[39629]: pgmap v641: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-04-23T16:23:35.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:35 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/326236761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:23:35.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:35.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:35.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:35.606 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:35 smithi148 ceph-mon[44520]: pgmap v641: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-04-23T16:23:35.606 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:35 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/326236761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:23:35.607 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:35.607 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:35.607 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:35.640 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-23T16:23:35.643 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:23:35.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:35.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:23:35.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:35 smithi188 ceph-mon[44101]: pgmap v641: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-04-23T16:23:35.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:35 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/326236761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:23:35.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:35.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:35.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:35.747 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:23:35.777 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:35.799 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:23:35.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:35.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:23:35.910 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:23:35.944 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:35.970 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:23:35.975 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:23:35.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:35.981 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:23:36.082 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:23:36.123 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:36.475 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:23:36.479 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 70 16 146491 146475 130.769 55.3125 0.0663884 0.00764345 2024-04-23T16:23:36.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:36.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:23:36.580 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:23:36.609 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.629 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:23:36.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:36.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:23:36.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.733 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:23:36.762 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.762 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.762 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.762 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.762 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.762 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.762 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.762 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.762 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:36.765 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:36.796 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:23:36.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:36.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:23:36.900 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:23:36.933 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:36.968 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:23:36.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:36.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:23:37.076 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:37.107 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:37.309 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:37.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:37.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:37.413 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:23:37.445 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:37.469 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:23:37.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:37.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:23:37.479 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 71 16 147373 147357 129.704 55.125 0.00680037 0.00770788 2024-04-23T16:23:37.574 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:37.605 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:37 smithi148 ceph-mon[44520]: pgmap v642: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-23T16:23:37.606 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:37 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:23:37.606 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:37 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:23:37.610 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:37.640 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:37.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:37.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:37.678 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:23:37.716 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:37 smithi103 ceph-mon[39629]: pgmap v642: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-23T16:23:37.717 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:37 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:23:37.717 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:37 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:23:37.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:37 smithi188 ceph-mon[44101]: pgmap v642: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-23T16:23:37.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:37 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:23:37.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:37 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:23:37.745 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:23:37.787 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:37.814 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:23:37.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:37.816 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:23:37.908 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:23:37.908 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:23:37.917 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:23:37.949 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:38.045 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:23:38.157 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:23:38.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:38.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:23:38.263 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:23:38.293 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:38.323 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:23:38.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:38.328 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:23:38.429 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:23:38.467 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:38.479 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 72 16 148222 148206 128.639 53.0625 0.00447038 0.00777039 2024-04-23T16:23:38.495 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:23:38.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:38.498 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:23:38.517 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:23:38.599 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:23:38.620 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:38 smithi148 ceph-mon[44520]: osdmap e222: 12 total, 12 up, 11 in 2024-04-23T16:23:38.645 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:38 smithi103 ceph-mon[39629]: osdmap e222: 12 total, 12 up, 11 in 2024-04-23T16:23:38.647 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:38.654 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-23T16:23:38.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:38 smithi188 ceph-mon[44101]: osdmap e222: 12 total, 12 up, 11 in 2024-04-23T16:23:38.981 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:23:39.414 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:39 smithi103 ceph-mon[39629]: pgmap v644: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-23T16:23:39.479 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 73 16 149067 149051 127.601 52.8125 0.00728091 0.00783055 2024-04-23T16:23:39.518 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:39 smithi148 ceph-mon[44520]: pgmap v644: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-23T16:23:39.607 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:23:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:39.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:23:39.611 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:23:39.711 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:23:39.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:39 smithi188 ceph-mon[44101]: pgmap v644: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-23T16:23:39.747 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:39.774 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:23:39.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:39.779 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:23:39.880 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:23:39.909 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:40.361 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:23:40.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:40.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:23:40.465 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:23:40.479 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 74 16 149862 149846 126.548 49.6875 0.00600475 0.00789627 2024-04-23T16:23:40.496 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:40.568 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:23:40.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:40.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:23:40.673 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:23:40.717 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:40.763 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:23:40.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:40.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:23:40.871 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:23:40.894 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:23:40.894 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:23:40.906 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:40.926 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:23:40.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:40.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:23:41.030 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:23:41.080 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:41.479 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 75 16 150690 150674 125.55 51.75 0.00466275 0.0079602 2024-04-23T16:23:41.686 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-23T16:23:41.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:41 smithi188 ceph-mon[44101]: pgmap v645: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 54 MiB/s wr, 866 op/s 2024-04-23T16:23:42.006 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:23:42.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:41 smithi103 ceph-mon[39629]: pgmap v645: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 54 MiB/s wr, 866 op/s 2024-04-23T16:23:42.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:41 smithi148 ceph-mon[44520]: pgmap v645: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 54 MiB/s wr, 866 op/s 2024-04-23T16:23:42.479 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 76 16 151541 151525 124.598 53.1875 0.00676422 0.00802415 2024-04-23T16:23:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:42 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1585234040' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:23:42.947 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-23T16:23:42.948 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:23:42.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:42 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1585234040' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:23:42.975 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:42 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1585234040' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:23:43.479 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 77 16 152689 152673 123.912 71.75 0.00681205 0.00806902 2024-04-23T16:23:43.510 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:23:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:43 smithi188 ceph-mon[44101]: pgmap v646: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 52 MiB/s wr, 834 op/s 2024-04-23T16:23:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:43 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:23:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:43 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1585234040' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:23:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:43 smithi188 ceph-mon[44101]: osdmap e223: 12 total, 12 up, 11 in 2024-04-23T16:23:44.034 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:23:44.035 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:23:44.065 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:43 smithi148 ceph-mon[44520]: pgmap v646: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 52 MiB/s wr, 834 op/s 2024-04-23T16:23:44.065 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:43 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:23:44.065 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:43 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1585234040' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:23:44.065 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:43 smithi148 ceph-mon[44520]: osdmap e223: 12 total, 12 up, 11 in 2024-04-23T16:23:44.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:43 smithi103 ceph-mon[39629]: pgmap v646: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 52 MiB/s wr, 834 op/s 2024-04-23T16:23:44.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:43 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:23:44.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:43 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1585234040' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:23:44.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:43 smithi103 ceph-mon[39629]: osdmap e223: 12 total, 12 up, 11 in 2024-04-23T16:23:44.479 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 78 16 153964 153948 123.345 79.6875 0.0189189 0.00810556 2024-04-23T16:23:44.739 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:44 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:23:44.740 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:44 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:23:44.775 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-23T16:23:44.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:44 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:23:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:44 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:23:45.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:44 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:23:45.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:44 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:23:45.151 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:23:45.479 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 79 16 155263 155247 122.811 81.1875 0.00722697 0.0081411 2024-04-23T16:23:45.744 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:23:45.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:45.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:23:45.848 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:23:45.899 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:45.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:45 smithi188 ceph-mon[44101]: pgmap v648: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 52 MiB/s wr, 836 op/s 2024-04-23T16:23:45.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:45 smithi188 ceph-mon[44101]: osdmap e224: 12 total, 12 up, 11 in 2024-04-23T16:23:46.011 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:23:46.012 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:45 smithi148 ceph-mon[44520]: pgmap v648: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 52 MiB/s wr, 836 op/s 2024-04-23T16:23:46.012 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:45 smithi148 ceph-mon[44520]: osdmap e224: 12 total, 12 up, 11 in 2024-04-23T16:23:46.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:46.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:23:46.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:45 smithi103 ceph-mon[39629]: pgmap v648: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 52 MiB/s wr, 836 op/s 2024-04-23T16:23:46.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:45 smithi103 ceph-mon[39629]: osdmap e224: 12 total, 12 up, 11 in 2024-04-23T16:23:46.114 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:23:46.154 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:46.480 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:23:46.479782 min lat: 0.00099496 max lat: 0.225871 avg lat: 0.00817709 2024-04-23T16:23:46.480 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:23:46.480 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 80 16 156537 156521 122.271 79.625 0.0234738 0.00817709 2024-04-23T16:23:47.161 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:23:47.161 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:23:47.480 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 81 16 157774 157758 121.716 77.3125 0.0162485 0.00821462 2024-04-23T16:23:47.865 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:47 smithi148 ceph-mon[44520]: pgmap v650: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 61 MiB/s wr, 980 op/s 2024-04-23T16:23:47.894 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-23T16:23:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:47 smithi188 ceph-mon[44101]: pgmap v650: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 61 MiB/s wr, 980 op/s 2024-04-23T16:23:48.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:47 smithi103 ceph-mon[39629]: pgmap v650: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 61 MiB/s wr, 980 op/s 2024-04-23T16:23:48.263 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:23:48.480 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 82 16 159009 158993 121.173 77.1875 0.00789548 0.00825099 2024-04-23T16:23:48.801 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:48 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:23:48.801 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:48 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:23:48.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:48 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:23:48.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:48 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:23:49.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:48 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:23:49.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:48 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:23:49.480 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 83 16 160285 160269 120.674 79.75 0.00421943 0.00828477 2024-04-23T16:23:49.819 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:49 smithi148 ceph-mon[44520]: pgmap v651: 17 pgs: 17 active+clean; 9.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-23T16:23:49.819 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:49 smithi148 ceph-mon[44520]: osdmap e225: 12 total, 12 up, 11 in 2024-04-23T16:23:49.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:49 smithi188 ceph-mon[44101]: pgmap v651: 17 pgs: 17 active+clean; 9.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-23T16:23:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:49 smithi188 ceph-mon[44101]: osdmap e225: 12 total, 12 up, 11 in 2024-04-23T16:23:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:49 smithi103 ceph-mon[39629]: pgmap v651: 17 pgs: 17 active+clean; 9.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-23T16:23:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:49 smithi103 ceph-mon[39629]: osdmap e225: 12 total, 12 up, 11 in 2024-04-23T16:23:50.184 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:23:50.184 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:23:50.337 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:23:50.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:50.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:23:50.440 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:50.480 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 84 16 161497 161481 120.139 75.75 0.0115299 0.00832222 2024-04-23T16:23:50.486 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:50.947 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:50.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:50.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:50.955 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-23T16:23:50.961 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:23:51.034 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:23:51.052 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:23:51.074 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:23:51.084 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:51.105 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:23:51.107 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:23:51.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:51.108 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:23:51.156 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:23:51.188 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:23:51.209 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:51.216 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:23:51.245 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:23:51.253 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:51.282 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:51.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:51.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:51.309 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:23:51.343 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:23:51.366 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:23:51.376 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:23:51.387 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:23:51.406 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:23:51.417 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:51.435 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:23:51.446 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:23:51.446 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:23:51.446 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:23:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:23:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:23:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:23:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:23:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:23:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:23:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:23:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:23:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:23:51.452 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:23:51.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:51.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:23:51.480 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 85 16 162676 162660 119.592 73.6875 0.00494768 0.00836045 2024-04-23T16:23:51.557 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:23:51.597 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:51.619 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:23:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:51.625 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:23:51.726 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:23:51.733 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:23:51.757 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:51.783 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:23:51.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:51.787 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:23:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:51 smithi103 ceph-mon[39629]: pgmap v653: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-23T16:23:51.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:51 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4240382303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:23:51.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:51.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:51.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:51.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:51.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:51.888 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:23:51.935 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:51.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:51 smithi188 ceph-mon[44101]: pgmap v653: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-23T16:23:51.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:51 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4240382303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:23:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.107 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:23:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:52.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:23:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:51 smithi148 ceph-mon[44520]: pgmap v653: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-23T16:23:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:51 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4240382303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:23:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.210 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:23:52.260 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:52.290 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:23:52.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:52.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:23:52.396 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:23:52.446 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:52.469 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:23:52.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:52.475 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:23:52.480 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 86 16 163868 163852 119.068 74.5 0.0154156 0.00839709 2024-04-23T16:23:52.576 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:52.613 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:52.693 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.866 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.866 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.866 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.866 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.866 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.866 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.866 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:52.870 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:52.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:52.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:52.978 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:23:53.009 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:53.041 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:23:53.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:53.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:23:53.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:53.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:53.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:53.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:53.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:53.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:53.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:23:53.145 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:23:53.177 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:53.210 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:23:53.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:53.215 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:23:53.315 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:53.364 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:53.480 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 87 16 165094 165078 118.58 76.625 0.0146002 0.00843177 2024-04-23T16:23:53.489 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:53.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:53.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:53.539 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:23:53.539 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2024-04-23T16:23:53.539 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2024-04-23T16:23:53.539 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:23:53.539 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.539 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:269177 2.7 2:e3b5e2b5:::benchmark_data_smithi188_77627_object99828:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-04-23T16:23:53.539 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:20:49.556474+0000", 2024-04-23T16:23:53.539 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 183.977639307, 2024-04-23T16:23:53.540 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.057656671, 2024-04-23T16:23:53.540 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.540 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.540 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:53.540 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.540 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:23:53.540 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:23:53.540 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 269177 2024-04-23T16:23:53.540 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.540 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.540 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.540 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.540 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.556474+0000", 2024-04-23T16:23:53.540 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.540 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.540 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.541 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.541 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.556474+0000", 2024-04-23T16:23:53.541 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.541 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.541 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.541 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.541 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.556476+0000", 2024-04-23T16:23:53.541 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1720000000000001e-06 2024-04-23T16:23:53.541 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.541 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.541 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.541 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.556482+0000", 2024-04-23T16:23:53.541 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5849999999999996e-06 2024-04-23T16:23:53.541 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.541 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.542 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.542 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.556493+0000", 2024-04-23T16:23:53.542 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1034e-05 2024-04-23T16:23:53.542 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.542 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.542 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.542 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.556501+0000", 2024-04-23T16:23:53.542 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8040000000000002e-06 2024-04-23T16:23:53.542 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.542 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.542 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.542 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.556692+0000", 2024-04-23T16:23:53.542 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000190979 2024-04-23T16:23:53.543 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.543 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.543 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:53.543 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.556818+0000", 2024-04-23T16:23:53.543 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012534499999999999 2024-04-23T16:23:53.543 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.543 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.543 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 2,10", 2024-04-23T16:23:53.543 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.556955+0000", 2024-04-23T16:23:53.543 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013779800000000001 2024-04-23T16:23:53.543 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.543 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.543 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:53.543 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.570247+0000", 2024-04-23T16:23:53.544 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013291347 2024-04-23T16:23:53.544 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.544 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.544 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.544 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.573501+0000", 2024-04-23T16:23:53.544 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032540659999999999 2024-04-23T16:23:53.544 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.544 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.544 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.544 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.613952+0000", 2024-04-23T16:23:53.544 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.040451503999999999 2024-04-23T16:23:53.544 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.544 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.544 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:53.544 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.614017+0000", 2024-04-23T16:23:53.545 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4276999999999995e-05 2024-04-23T16:23:53.545 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.545 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.545 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.545 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.614130+0000", 2024-04-23T16:23:53.545 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011376 2024-04-23T16:23:53.545 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.545 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.545 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.545 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.545 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.545 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:269180 2.7 2:e3fac87c:::benchmark_data_smithi188_77627_object99831:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-04-23T16:23:53.545 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:20:49.556520+0000", 2024-04-23T16:23:53.545 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 183.97759315299999, 2024-04-23T16:23:53.545 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.058126667999999999, 2024-04-23T16:23:53.546 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.546 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.546 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:53.546 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.546 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:23:53.546 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:23:53.546 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 269180 2024-04-23T16:23:53.546 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.546 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.546 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.546 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.546 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.556520+0000", 2024-04-23T16:23:53.546 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.546 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.547 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.547 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.547 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.556520+0000", 2024-04-23T16:23:53.547 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.547 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.547 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.547 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.547 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.556521+0000", 2024-04-23T16:23:53.547 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6199999999999997e-07 2024-04-23T16:23:53.547 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.547 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.547 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.547 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.556524+0000", 2024-04-23T16:23:53.547 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.422e-06 2024-04-23T16:23:53.547 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.548 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.548 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.548 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.556540+0000", 2024-04-23T16:23:53.548 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6302000000000001e-05 2024-04-23T16:23:53.548 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.548 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.548 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.548 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.556547+0000", 2024-04-23T16:23:53.548 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4169999999999997e-06 2024-04-23T16:23:53.548 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.548 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.548 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.548 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.561675+0000", 2024-04-23T16:23:53.548 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0051279339999999998 2024-04-23T16:23:53.548 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.548 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.549 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:53.549 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.561807+0000", 2024-04-23T16:23:53.549 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013258500000000001 2024-04-23T16:23:53.549 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.549 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.549 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 2,10", 2024-04-23T16:23:53.549 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.561961+0000", 2024-04-23T16:23:53.549 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015363900000000001 2024-04-23T16:23:53.549 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.549 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.549 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:53.549 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.573438+0000", 2024-04-23T16:23:53.549 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011477456 2024-04-23T16:23:53.549 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.549 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.549 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.550 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.577084+0000", 2024-04-23T16:23:53.550 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0036452559999999999 2024-04-23T16:23:53.550 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.550 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.550 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.550 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.614478+0000", 2024-04-23T16:23:53.550 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037394482999999999 2024-04-23T16:23:53.550 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.550 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.550 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:53.550 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.614539+0000", 2024-04-23T16:23:53.550 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0741e-05 2024-04-23T16:23:53.550 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.550 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.550 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.551 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.614646+0000", 2024-04-23T16:23:53.551 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010767100000000001 2024-04-23T16:23:53.551 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.551 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.551 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.551 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.551 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.551 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:269188 2.7 2:ebfaf46e:::benchmark_data_smithi188_77627_object99839:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-04-23T16:23:53.551 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:20:49.560170+0000", 2024-04-23T16:23:53.551 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 183.973942827, 2024-04-23T16:23:53.551 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.054674156000000002, 2024-04-23T16:23:53.551 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.551 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.551 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:53.551 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.551 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:23:53.552 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:23:53.552 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 269188 2024-04-23T16:23:53.552 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.552 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.552 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.552 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.552 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.560170+0000", 2024-04-23T16:23:53.552 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.552 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.552 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.552 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.552 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.560170+0000", 2024-04-23T16:23:53.552 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.552 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.552 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.553 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.553 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.560173+0000", 2024-04-23T16:23:53.553 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5749999999999999e-06 2024-04-23T16:23:53.553 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.553 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.553 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.553 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.560180+0000", 2024-04-23T16:23:53.553 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1500000000000002e-06 2024-04-23T16:23:53.553 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.553 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.553 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.553 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.560196+0000", 2024-04-23T16:23:53.553 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5934e-05 2024-04-23T16:23:53.553 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.553 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.553 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.553 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.560205+0000", 2024-04-23T16:23:53.554 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7539999999999995e-06 2024-04-23T16:23:53.554 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.554 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.554 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.554 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.565693+0000", 2024-04-23T16:23:53.554 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.005488021 2024-04-23T16:23:53.554 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.554 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.554 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:53.554 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.565821+0000", 2024-04-23T16:23:53.554 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012842999999999999 2024-04-23T16:23:53.554 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.554 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.554 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 2,10", 2024-04-23T16:23:53.555 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.565953+0000", 2024-04-23T16:23:53.555 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013178800000000001 2024-04-23T16:23:53.555 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.555 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.555 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:53.555 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.573448+0000", 2024-04-23T16:23:53.555 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0074953830000000004 2024-04-23T16:23:53.555 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.555 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.555 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.555 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.577133+0000", 2024-04-23T16:23:53.555 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0036845469999999998 2024-04-23T16:23:53.555 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.555 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.555 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.555 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.614716+0000", 2024-04-23T16:23:53.556 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037583146999999997 2024-04-23T16:23:53.556 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.556 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.556 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:53.556 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.614754+0000", 2024-04-23T16:23:53.556 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7904000000000002e-05 2024-04-23T16:23:53.556 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.556 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.556 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.556 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.614844+0000", 2024-04-23T16:23:53.556 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0523e-05 2024-04-23T16:23:53.556 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.556 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.556 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.556 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.556 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.557 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:269192 2.7 2:efb9bbd1:::benchmark_data_smithi188_77627_object99843:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-04-23T16:23:53.557 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:20:49.565027+0000", 2024-04-23T16:23:53.557 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 183.96908647199999, 2024-04-23T16:23:53.557 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.050058251999999998, 2024-04-23T16:23:53.557 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.557 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.557 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:53.557 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.557 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:23:53.557 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:23:53.557 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 269192 2024-04-23T16:23:53.557 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.557 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.557 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.557 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.558 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.565027+0000", 2024-04-23T16:23:53.558 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.558 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.558 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.558 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.558 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.565027+0000", 2024-04-23T16:23:53.558 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.558 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.558 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.558 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.558 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.565029+0000", 2024-04-23T16:23:53.558 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3879999999999998e-06 2024-04-23T16:23:53.558 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.558 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.559 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.559 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.565035+0000", 2024-04-23T16:23:53.559 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.319e-06 2024-04-23T16:23:53.559 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.559 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.559 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.559 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.565047+0000", 2024-04-23T16:23:53.559 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2096000000000001e-05 2024-04-23T16:23:53.559 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.559 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.559 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.559 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.565056+0000", 2024-04-23T16:23:53.559 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4630000000000001e-06 2024-04-23T16:23:53.559 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.559 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.559 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.560 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.570268+0000", 2024-04-23T16:23:53.560 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0052125799999999996 2024-04-23T16:23:53.560 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.560 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.560 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:53.560 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.570349+0000", 2024-04-23T16:23:53.560 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1116000000000001e-05 2024-04-23T16:23:53.560 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.560 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.560 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 2,10", 2024-04-23T16:23:53.560 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.570428+0000", 2024-04-23T16:23:53.560 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8718000000000001e-05 2024-04-23T16:23:53.560 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.560 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.560 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:53.560 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.573540+0000", 2024-04-23T16:23:53.561 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0031122419999999999 2024-04-23T16:23:53.561 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.561 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.561 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.561 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.577163+0000", 2024-04-23T16:23:53.561 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0036228739999999999 2024-04-23T16:23:53.561 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.561 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.561 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.561 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.614909+0000", 2024-04-23T16:23:53.561 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037745441999999997 2024-04-23T16:23:53.561 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.561 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.561 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:53.561 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.614971+0000", 2024-04-23T16:23:53.561 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1849999999999999e-05 2024-04-23T16:23:53.562 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.562 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.562 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.562 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:20:49.615085+0000", 2024-04-23T16:23:53.562 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000114164 2024-04-23T16:23:53.562 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.562 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.562 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.562 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.562 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.562 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:289101 2.7 2:e0478480:::benchmark_data_smithi188_77627_object119752:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e175)", 2024-04-23T16:23:53.562 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:21:08.686076+0000", 2024-04-23T16:23:53.562 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 164.84803691600001, 2024-04-23T16:23:53.562 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.036797318000000002, 2024-04-23T16:23:53.562 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.562 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.563 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2024-04-23T16:23:53.563 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.563 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:23:53.563 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:23:53.563 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 289101 2024-04-23T16:23:53.563 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.563 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.563 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.563 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.563 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686076+0000", 2024-04-23T16:23:53.563 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.563 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.563 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.563 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.563 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686076+0000", 2024-04-23T16:23:53.563 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.564 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.564 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.564 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.564 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686078+0000", 2024-04-23T16:23:53.564 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2900000000000001e-06 2024-04-23T16:23:53.564 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.564 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.564 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.564 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686088+0000", 2024-04-23T16:23:53.564 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1980000000000007e-06 2024-04-23T16:23:53.564 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.564 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.564 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.564 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686111+0000", 2024-04-23T16:23:53.564 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3872000000000001e-05 2024-04-23T16:23:53.565 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.565 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.565 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.565 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686124+0000", 2024-04-23T16:23:53.565 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3076e-05 2024-04-23T16:23:53.565 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.565 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.565 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.565 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686178+0000", 2024-04-23T16:23:53.565 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3661e-05 2024-04-23T16:23:53.565 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.565 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.565 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:23:53.565 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686187+0000", 2024-04-23T16:23:53.565 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1109999999999997e-06 2024-04-23T16:23:53.565 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.565 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.566 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.566 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.722821+0000", 2024-04-23T16:23:53.566 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.036633493000000003 2024-04-23T16:23:53.566 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.566 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.566 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.566 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.722873+0000", 2024-04-23T16:23:53.566 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2617000000000001e-05 2024-04-23T16:23:53.566 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.566 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.566 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.566 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.566 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.566 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:289108 2.7 2:e3660c62:::benchmark_data_smithi188_77627_object119759:head [delete] snapc 0=[] ondisk+write+known_if_redirected e175)", 2024-04-23T16:23:53.567 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:21:08.686144+0000", 2024-04-23T16:23:53.567 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 164.84796861999999, 2024-04-23T16:23:53.567 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048096913999999998, 2024-04-23T16:23:53.567 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.567 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.567 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:53.567 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.567 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:23:53.567 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:23:53.567 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 289108 2024-04-23T16:23:53.567 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.567 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.567 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.567 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.567 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686144+0000", 2024-04-23T16:23:53.567 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.568 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.568 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.568 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.568 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686144+0000", 2024-04-23T16:23:53.568 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.568 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.568 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.568 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.568 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686145+0000", 2024-04-23T16:23:53.568 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2699999999999999e-07 2024-04-23T16:23:53.568 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.568 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.568 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.568 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686149+0000", 2024-04-23T16:23:53.568 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4529999999999998e-06 2024-04-23T16:23:53.569 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.569 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.569 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.569 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686156+0000", 2024-04-23T16:23:53.569 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2189999999999998e-06 2024-04-23T16:23:53.569 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.569 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.569 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.569 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686161+0000", 2024-04-23T16:23:53.569 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.694e-06 2024-04-23T16:23:53.569 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.569 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.569 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.569 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686209+0000", 2024-04-23T16:23:53.569 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7718e-05 2024-04-23T16:23:53.569 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.570 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.570 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:23:53.570 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686216+0000", 2024-04-23T16:23:53.570 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0890000000000002e-06 2024-04-23T16:23:53.570 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.570 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.570 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.570 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.723128+0000", 2024-04-23T16:23:53.570 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.036911669000000001 2024-04-23T16:23:53.570 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.570 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.570 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:53.570 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.723190+0000", 2024-04-23T16:23:53.570 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1741000000000004e-05 2024-04-23T16:23:53.570 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.571 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.571 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 2", 2024-04-23T16:23:53.571 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.723266+0000", 2024-04-23T16:23:53.571 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6599000000000003e-05 2024-04-23T16:23:53.571 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.571 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.571 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.571 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.734097+0000", 2024-04-23T16:23:53.571 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010830296999999999 2024-04-23T16:23:53.571 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.571 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.571 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:53.571 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.734186+0000", 2024-04-23T16:23:53.571 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9905000000000001e-05 2024-04-23T16:23:53.571 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.571 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.572 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:53.572 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.734218+0000", 2024-04-23T16:23:53.572 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1294999999999998e-05 2024-04-23T16:23:53.572 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.572 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.572 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.572 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.734241+0000", 2024-04-23T16:23:53.572 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3507999999999999e-05 2024-04-23T16:23:53.572 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.572 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.572 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.572 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.572 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.572 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:289114 2.7 2:eeb7d9fc:::benchmark_data_smithi188_77627_object119765:head [delete] snapc 0=[] ondisk+write+known_if_redirected e175)", 2024-04-23T16:23:53.572 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:21:08.686171+0000", 2024-04-23T16:23:53.572 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 164.847941542, 2024-04-23T16:23:53.573 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048184306000000003, 2024-04-23T16:23:53.573 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.573 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.573 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:53.573 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.573 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:23:53.573 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:23:53.573 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 289114 2024-04-23T16:23:53.573 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.573 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.573 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.573 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.573 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686171+0000", 2024-04-23T16:23:53.574 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.574 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.574 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.574 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.574 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686171+0000", 2024-04-23T16:23:53.574 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.574 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.574 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.574 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.574 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686172+0000", 2024-04-23T16:23:53.574 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8199999999999999e-07 2024-04-23T16:23:53.574 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.574 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.574 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.574 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686175+0000", 2024-04-23T16:23:53.575 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9979999999999999e-06 2024-04-23T16:23:53.575 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.575 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.575 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.575 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686181+0000", 2024-04-23T16:23:53.575 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4709999999999998e-06 2024-04-23T16:23:53.575 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.575 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.575 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.575 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686186+0000", 2024-04-23T16:23:53.575 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.614e-06 2024-04-23T16:23:53.575 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.575 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.575 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.575 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686231+0000", 2024-04-23T16:23:53.576 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4808000000000002e-05 2024-04-23T16:23:53.576 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.576 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.576 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:23:53.576 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686235+0000", 2024-04-23T16:23:53.576 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1450000000000001e-06 2024-04-23T16:23:53.576 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.576 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.576 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.576 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.726635+0000", 2024-04-23T16:23:53.576 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.040399591999999998 2024-04-23T16:23:53.576 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.576 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.576 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:53.576 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.726704+0000", 2024-04-23T16:23:53.577 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9555999999999996e-05 2024-04-23T16:23:53.577 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.577 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.577 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 2", 2024-04-23T16:23:53.577 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.726767+0000", 2024-04-23T16:23:53.577 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2577999999999996e-05 2024-04-23T16:23:53.577 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.577 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.577 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.577 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.734279+0000", 2024-04-23T16:23:53.577 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0075126200000000002 2024-04-23T16:23:53.577 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.577 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.577 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:53.577 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.734311+0000", 2024-04-23T16:23:53.578 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1398e-05 2024-04-23T16:23:53.578 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.578 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.578 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:53.578 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.734334+0000", 2024-04-23T16:23:53.578 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3254000000000001e-05 2024-04-23T16:23:53.578 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.578 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.578 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.578 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.734356+0000", 2024-04-23T16:23:53.578 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1590000000000002e-05 2024-04-23T16:23:53.578 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.578 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.578 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.579 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.579 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.579 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:289145 2.7 2:eebe3be4:::benchmark_data_smithi188_77627_object119796:head [delete] snapc 0=[] ondisk+write+known_if_redirected e175)", 2024-04-23T16:23:53.579 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:21:08.686208+0000", 2024-04-23T16:23:53.579 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 164.84790478100001, 2024-04-23T16:23:53.579 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048190152, 2024-04-23T16:23:53.579 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.579 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.579 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:53.579 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.579 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:23:53.579 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:23:53.579 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 289145 2024-04-23T16:23:53.579 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.580 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.580 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.580 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.580 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686208+0000", 2024-04-23T16:23:53.580 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.580 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.580 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.580 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.580 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686208+0000", 2024-04-23T16:23:53.580 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.580 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.580 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.580 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.580 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686209+0000", 2024-04-23T16:23:53.580 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0200000000000002e-07 2024-04-23T16:23:53.580 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.581 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.581 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.581 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686212+0000", 2024-04-23T16:23:53.581 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1650000000000002e-06 2024-04-23T16:23:53.581 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.581 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.581 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.581 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686218+0000", 2024-04-23T16:23:53.581 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1449999999999996e-06 2024-04-23T16:23:53.581 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.581 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.581 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.581 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686227+0000", 2024-04-23T16:23:53.581 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4249999999999998e-06 2024-04-23T16:23:53.582 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.582 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.582 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.582 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686256+0000", 2024-04-23T16:23:53.582 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9133999999999999e-05 2024-04-23T16:23:53.582 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.582 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.582 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:23:53.582 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.686259+0000", 2024-04-23T16:23:53.582 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2289999999999999e-06 2024-04-23T16:23:53.582 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.582 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.582 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.582 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.729912+0000", 2024-04-23T16:23:53.583 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.043652814999999998 2024-04-23T16:23:53.583 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.583 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.583 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:53.583 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.729979+0000", 2024-04-23T16:23:53.583 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6985999999999999e-05 2024-04-23T16:23:53.583 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.583 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.583 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 2", 2024-04-23T16:23:53.583 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.730054+0000", 2024-04-23T16:23:53.583 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5349e-05 2024-04-23T16:23:53.583 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.583 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.583 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.583 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.734305+0000", 2024-04-23T16:23:53.584 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0042504700000000001 2024-04-23T16:23:53.584 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.584 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.584 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:53.584 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.734359+0000", 2024-04-23T16:23:53.584 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4731999999999997e-05 2024-04-23T16:23:53.584 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.584 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.584 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:53.584 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.734380+0000", 2024-04-23T16:23:53.584 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0556000000000001e-05 2024-04-23T16:23:53.584 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.584 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.584 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.584 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:08.734398+0000", 2024-04-23T16:23:53.585 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8544000000000001e-05 2024-04-23T16:23:53.585 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.585 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.585 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.585 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.585 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.585 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:292620 2.7 2:ecbba2a0:::benchmark_data_smithi188_77627_object123271:head [delete] snapc 0=[] ondisk+write+known_if_redirected e175)", 2024-04-23T16:23:53.585 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:21:11.592588+0000", 2024-04-23T16:23:53.585 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 161.94152500999999, 2024-04-23T16:23:53.585 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.038931512000000001, 2024-04-23T16:23:53.585 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.585 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.585 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:53.585 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.585 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:23:53.585 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:23:53.586 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 292620 2024-04-23T16:23:53.586 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.586 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.586 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.586 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.586 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.592588+0000", 2024-04-23T16:23:53.586 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.586 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.586 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.586 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.586 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.592588+0000", 2024-04-23T16:23:53.586 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.586 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.586 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.586 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.587 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.592590+0000", 2024-04-23T16:23:53.587 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0099999999999998e-06 2024-04-23T16:23:53.587 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.587 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.587 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.587 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.592595+0000", 2024-04-23T16:23:53.587 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2769999999999996e-06 2024-04-23T16:23:53.587 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.587 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.587 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.587 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.592608+0000", 2024-04-23T16:23:53.587 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2693e-05 2024-04-23T16:23:53.587 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.587 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.587 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.587 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.592616+0000", 2024-04-23T16:23:53.588 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7640000000000005e-06 2024-04-23T16:23:53.588 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.588 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.588 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.588 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.592660+0000", 2024-04-23T16:23:53.588 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4563e-05 2024-04-23T16:23:53.588 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.588 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.588 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:53.588 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.592739+0000", 2024-04-23T16:23:53.588 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8448999999999994e-05 2024-04-23T16:23:53.588 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.588 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.588 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 2", 2024-04-23T16:23:53.588 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.592807+0000", 2024-04-23T16:23:53.588 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8015999999999994e-05 2024-04-23T16:23:53.589 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.589 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.589 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:53.589 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.598582+0000", 2024-04-23T16:23:53.589 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0057757370000000004 2024-04-23T16:23:53.589 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.589 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.589 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.589 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.631399+0000", 2024-04-23T16:23:53.589 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.032816588000000001 2024-04-23T16:23:53.589 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.589 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.589 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:53.589 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.631485+0000", 2024-04-23T16:23:53.590 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5888000000000002e-05 2024-04-23T16:23:53.590 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.590 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.590 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.590 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.631519+0000", 2024-04-23T16:23:53.590 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4527000000000003e-05 2024-04-23T16:23:53.590 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.590 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.590 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.590 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.590 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.590 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:292622 2.7 2:e5dc666a:::benchmark_data_smithi188_77627_object123273:head [delete] snapc 0=[] ondisk+write+known_if_redirected e175)", 2024-04-23T16:23:53.590 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:21:11.595968+0000", 2024-04-23T16:23:53.590 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 161.938144752, 2024-04-23T16:23:53.591 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.035667163000000002, 2024-04-23T16:23:53.591 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.591 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.591 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:53.591 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.591 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:23:53.591 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:23:53.591 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 292622 2024-04-23T16:23:53.591 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.591 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.591 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.591 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.591 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.595968+0000", 2024-04-23T16:23:53.591 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.591 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.592 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.592 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.592 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.595968+0000", 2024-04-23T16:23:53.592 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.592 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.592 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.592 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.592 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.595970+0000", 2024-04-23T16:23:53.592 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3379999999999999e-06 2024-04-23T16:23:53.592 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.592 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.592 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.592 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.595975+0000", 2024-04-23T16:23:53.592 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2129999999999999e-06 2024-04-23T16:23:53.593 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.593 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.593 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.593 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.595985+0000", 2024-04-23T16:23:53.593 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0329000000000001e-05 2024-04-23T16:23:53.593 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.593 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.593 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.593 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.595992+0000", 2024-04-23T16:23:53.594 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6089999999999997e-06 2024-04-23T16:23:53.594 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.594 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.594 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.594 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.596035+0000", 2024-04-23T16:23:53.594 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3324000000000001e-05 2024-04-23T16:23:53.594 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.594 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.594 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:53.594 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.596102+0000", 2024-04-23T16:23:53.594 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7008999999999995e-05 2024-04-23T16:23:53.594 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.594 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.595 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 2", 2024-04-23T16:23:53.595 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.596176+0000", 2024-04-23T16:23:53.595 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4326999999999995e-05 2024-04-23T16:23:53.595 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.595 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.595 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:53.595 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.598744+0000", 2024-04-23T16:23:53.595 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025676359999999999 2024-04-23T16:23:53.595 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.595 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.595 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.595 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.631565+0000", 2024-04-23T16:23:53.595 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.032821438000000001 2024-04-23T16:23:53.595 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.595 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.596 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:53.596 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.631597+0000", 2024-04-23T16:23:53.596 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1149999999999998e-05 2024-04-23T16:23:53.596 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.596 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.596 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.596 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:11.631635+0000", 2024-04-23T16:23:53.596 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.879e-05 2024-04-23T16:23:53.596 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.596 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.596 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.596 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.596 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.596 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:298332 2.7 2:ebf95439:::benchmark_data_smithi188_77627_object128983:head [delete] snapc 0=[] ondisk+write+known_if_redirected e177)", 2024-04-23T16:23:53.596 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:21:16.456810+0000", 2024-04-23T16:23:53.597 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 157.07730295799999, 2024-04-23T16:23:53.597 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.038177127999999998, 2024-04-23T16:23:53.597 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.597 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.597 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:53.597 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.597 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:23:53.597 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:23:53.597 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 298332 2024-04-23T16:23:53.597 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.597 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.597 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.597 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.597 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:16.456810+0000", 2024-04-23T16:23:53.598 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.598 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.598 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.598 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.598 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:16.456810+0000", 2024-04-23T16:23:53.598 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.598 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.598 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.598 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.598 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:16.456812+0000", 2024-04-23T16:23:53.598 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.499e-06 2024-04-23T16:23:53.598 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.598 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.599 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.599 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:16.456817+0000", 2024-04-23T16:23:53.599 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1599999999999997e-06 2024-04-23T16:23:53.599 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.599 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.599 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.599 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:16.456831+0000", 2024-04-23T16:23:53.599 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4809e-05 2024-04-23T16:23:53.599 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.599 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.599 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.599 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:16.456839+0000", 2024-04-23T16:23:53.599 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.233e-06 2024-04-23T16:23:53.599 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.599 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.600 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.600 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:16.461162+0000", 2024-04-23T16:23:53.600 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0043233009999999999 2024-04-23T16:23:53.600 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.600 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.600 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:53.600 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:16.461245+0000", 2024-04-23T16:23:53.600 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3212000000000002e-05 2024-04-23T16:23:53.600 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.600 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.600 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 2", 2024-04-23T16:23:53.600 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:16.461317+0000", 2024-04-23T16:23:53.600 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1842999999999996e-05 2024-04-23T16:23:53.600 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.600 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.601 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:53.601 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:16.467257+0000", 2024-04-23T16:23:53.601 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0059398910000000001 2024-04-23T16:23:53.601 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.601 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.601 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.601 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:16.494874+0000", 2024-04-23T16:23:53.601 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.027616543 2024-04-23T16:23:53.601 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.601 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.601 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:53.601 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:16.494933+0000", 2024-04-23T16:23:53.601 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9528000000000003e-05 2024-04-23T16:23:53.601 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.602 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.602 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.602 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:16.494987+0000", 2024-04-23T16:23:53.602 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4109e-05 2024-04-23T16:23:53.602 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.602 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.602 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.602 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.602 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.602 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:302536 2.7 2:ef552cfe:::benchmark_data_smithi188_77627_object133187:head [delete] snapc 0=[] ondisk+write+known_if_redirected e178)", 2024-04-23T16:23:53.602 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:21:19.956137+0000", 2024-04-23T16:23:53.602 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 153.57797582000001, 2024-04-23T16:23:53.602 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.056594286000000001, 2024-04-23T16:23:53.602 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.602 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.603 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:53.603 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.603 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:23:53.603 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:23:53.603 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 302536 2024-04-23T16:23:53.603 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.603 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.603 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.603 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.603 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.956137+0000", 2024-04-23T16:23:53.603 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.603 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.603 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.603 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.604 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.956137+0000", 2024-04-23T16:23:53.604 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.604 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.604 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.604 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.604 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.956140+0000", 2024-04-23T16:23:53.604 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.802e-06 2024-04-23T16:23:53.604 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.604 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.604 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.604 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.956148+0000", 2024-04-23T16:23:53.604 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8779999999999998e-06 2024-04-23T16:23:53.604 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.604 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.604 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.605 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.956163+0000", 2024-04-23T16:23:53.605 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4820999999999999e-05 2024-04-23T16:23:53.605 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.605 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.605 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.605 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.956175+0000", 2024-04-23T16:23:53.605 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2312e-05 2024-04-23T16:23:53.605 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.605 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.605 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.605 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.956250+0000", 2024-04-23T16:23:53.605 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4698999999999997e-05 2024-04-23T16:23:53.605 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.605 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.605 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:53.606 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.956374+0000", 2024-04-23T16:23:53.606 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012412299999999999 2024-04-23T16:23:53.606 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.606 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.606 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 2", 2024-04-23T16:23:53.606 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.956505+0000", 2024-04-23T16:23:53.606 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000131357 2024-04-23T16:23:53.606 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.606 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.606 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:53.606 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.965565+0000", 2024-04-23T16:23:53.606 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0090593610000000001 2024-04-23T16:23:53.606 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.606 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.607 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.607 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:20.012671+0000", 2024-04-23T16:23:53.607 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.047106576999999997 2024-04-23T16:23:53.607 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.607 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.607 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:53.607 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:20.012703+0000", 2024-04-23T16:23:53.607 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2122000000000002e-05 2024-04-23T16:23:53.607 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.607 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.607 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.607 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:20.012731+0000", 2024-04-23T16:23:53.607 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8234000000000001e-05 2024-04-23T16:23:53.607 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.607 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.607 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.608 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.608 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.608 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:302538 2.7 2:e7f7936d:::benchmark_data_smithi188_77627_object133189:head [delete] snapc 0=[] ondisk+write+known_if_redirected e178)", 2024-04-23T16:23:53.608 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:21:19.956206+0000", 2024-04-23T16:23:53.608 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 153.57790660699999, 2024-04-23T16:23:53.608 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.056617776000000002, 2024-04-23T16:23:53.608 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.608 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.608 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:53.608 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.608 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:23:53.608 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:23:53.608 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 302538 2024-04-23T16:23:53.608 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.608 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.609 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.609 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.609 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.956206+0000", 2024-04-23T16:23:53.609 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.609 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.609 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.609 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.609 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.956206+0000", 2024-04-23T16:23:53.609 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.609 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.609 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.609 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.609 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.956207+0000", 2024-04-23T16:23:53.609 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6600000000000005e-07 2024-04-23T16:23:53.610 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.610 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.610 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.610 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.956211+0000", 2024-04-23T16:23:53.610 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6540000000000001e-06 2024-04-23T16:23:53.610 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.610 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.610 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.610 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.956218+0000", 2024-04-23T16:23:53.610 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0550000000000002e-06 2024-04-23T16:23:53.610 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.610 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.610 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.610 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.956224+0000", 2024-04-23T16:23:53.611 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3219999999999998e-06 2024-04-23T16:23:53.611 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.611 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.611 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.611 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.960265+0000", 2024-04-23T16:23:53.611 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0040402670000000002 2024-04-23T16:23:53.611 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.611 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.611 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:53.611 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.960381+0000", 2024-04-23T16:23:53.611 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000116134 2024-04-23T16:23:53.611 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.611 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.611 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 2", 2024-04-23T16:23:53.611 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.960469+0000", 2024-04-23T16:23:53.612 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7943999999999997e-05 2024-04-23T16:23:53.612 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.612 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.612 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:53.612 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.965571+0000", 2024-04-23T16:23:53.612 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0051023969999999998 2024-04-23T16:23:53.612 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.612 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.612 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.612 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:20.012769+0000", 2024-04-23T16:23:53.612 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.047197668999999998 2024-04-23T16:23:53.612 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.612 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.612 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:53.612 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:20.012797+0000", 2024-04-23T16:23:53.613 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8756999999999999e-05 2024-04-23T16:23:53.613 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.613 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.613 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.613 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:20.012824+0000", 2024-04-23T16:23:53.613 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6710999999999999e-05 2024-04-23T16:23:53.613 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.613 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.613 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.613 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.613 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.613 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:302545 2.7 2:ef953ca7:::benchmark_data_smithi188_77627_object133196:head [delete] snapc 0=[] ondisk+write+known_if_redirected e178)", 2024-04-23T16:23:53.613 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:21:19.962591+0000", 2024-04-23T16:23:53.614 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 153.57152196000001, 2024-04-23T16:23:53.614 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.05034893, 2024-04-23T16:23:53.614 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.614 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.614 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:53.614 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.614 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:23:53.614 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:23:53.614 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 302545 2024-04-23T16:23:53.614 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.614 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.614 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.614 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.614 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.962591+0000", 2024-04-23T16:23:53.615 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.615 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.615 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.615 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.615 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.962591+0000", 2024-04-23T16:23:53.615 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.615 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.615 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.615 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.615 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.962593+0000", 2024-04-23T16:23:53.615 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9700000000000002e-06 2024-04-23T16:23:53.615 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.615 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.615 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.615 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.962599+0000", 2024-04-23T16:23:53.616 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9540000000000001e-06 2024-04-23T16:23:53.616 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.616 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.616 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.616 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.962610+0000", 2024-04-23T16:23:53.616 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1131000000000001e-05 2024-04-23T16:23:53.616 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.616 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.616 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.616 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.962618+0000", 2024-04-23T16:23:53.616 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1769999999999998e-06 2024-04-23T16:23:53.616 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.616 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.616 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.616 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.963478+0000", 2024-04-23T16:23:53.616 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00086011699999999998 2024-04-23T16:23:53.617 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.617 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.617 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:53.617 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.963547+0000", 2024-04-23T16:23:53.617 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8775000000000005e-05 2024-04-23T16:23:53.617 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.617 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.617 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 2", 2024-04-23T16:23:53.617 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.963613+0000", 2024-04-23T16:23:53.617 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6231999999999999e-05 2024-04-23T16:23:53.617 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.617 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.617 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:53.617 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.965716+0000", 2024-04-23T16:23:53.617 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002102745 2024-04-23T16:23:53.618 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.618 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.618 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.618 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:20.012869+0000", 2024-04-23T16:23:53.618 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.047153384999999999 2024-04-23T16:23:53.618 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.618 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.618 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:53.618 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:20.012903+0000", 2024-04-23T16:23:53.618 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3798999999999999e-05 2024-04-23T16:23:53.618 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.618 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.618 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.618 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:20.012940+0000", 2024-04-23T16:23:53.619 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6644999999999999e-05 2024-04-23T16:23:53.619 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.619 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.619 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.619 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.619 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.619 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:302556 2.7 2:e0cdb99f:::benchmark_data_smithi188_77627_object133207:head [delete] snapc 0=[] ondisk+write+known_if_redirected e178)", 2024-04-23T16:23:53.619 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:21:19.974569+0000", 2024-04-23T16:23:53.619 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 153.55954388999999, 2024-04-23T16:23:53.619 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.038542318999999998, 2024-04-23T16:23:53.619 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.619 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.619 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:53.619 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.619 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:23:53.619 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:23:53.620 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 302556 2024-04-23T16:23:53.620 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.620 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.620 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.620 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.620 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.974569+0000", 2024-04-23T16:23:53.620 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.620 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.620 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.620 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.620 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.974569+0000", 2024-04-23T16:23:53.620 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.620 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.620 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.620 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.621 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.974571+0000", 2024-04-23T16:23:53.621 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7260000000000001e-06 2024-04-23T16:23:53.621 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.621 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.621 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.621 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.974577+0000", 2024-04-23T16:23:53.621 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8000000000000004e-06 2024-04-23T16:23:53.621 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.621 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.621 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.621 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.974587+0000", 2024-04-23T16:23:53.621 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0871e-05 2024-04-23T16:23:53.621 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.621 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.621 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.622 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.974594+0000", 2024-04-23T16:23:53.622 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5830000000000003e-06 2024-04-23T16:23:53.622 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.622 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.622 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.622 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.974639+0000", 2024-04-23T16:23:53.622 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4607000000000002e-05 2024-04-23T16:23:53.622 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.622 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.622 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:53.622 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.974721+0000", 2024-04-23T16:23:53.622 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2074000000000001e-05 2024-04-23T16:23:53.622 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.622 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.623 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 2", 2024-04-23T16:23:53.623 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.974789+0000", 2024-04-23T16:23:53.623 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8643e-05 2024-04-23T16:23:53.623 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.623 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.623 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:53.623 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:19.978033+0000", 2024-04-23T16:23:53.623 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032431750000000001 2024-04-23T16:23:53.623 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.623 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.623 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.623 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:20.013011+0000", 2024-04-23T16:23:53.623 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.034978324999999998 2024-04-23T16:23:53.623 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.623 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.624 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:53.624 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:20.013062+0000", 2024-04-23T16:23:53.624 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0763000000000002e-05 2024-04-23T16:23:53.624 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.624 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.624 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.624 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:20.013111+0000", 2024-04-23T16:23:53.624 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9752000000000001e-05 2024-04-23T16:23:53.624 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.624 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.624 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.624 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.624 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.624 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.14862.0:324544 2.7 2:edb92b64:::benchmark_data_smithi188_77627_object155195:head [delete] snapc 0=[] ondisk+write+known_if_redirected e182)", 2024-04-23T16:23:53.625 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:21:38.821414+0000", 2024-04-23T16:23:53.625 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 134.71269933299999, 2024-04-23T16:23:53.625 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.036630392999999997, 2024-04-23T16:23:53.625 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.625 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.625 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:53.625 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.625 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.14862", 2024-04-23T16:23:53.625 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/4075093561", 2024-04-23T16:23:53.625 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 324544 2024-04-23T16:23:53.625 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.625 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.625 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.625 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.625 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:38.821414+0000", 2024-04-23T16:23:53.626 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.626 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.626 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.626 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.626 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:38.821414+0000", 2024-04-23T16:23:53.626 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.626 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.626 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.626 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.626 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:38.821415+0000", 2024-04-23T16:23:53.626 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1799999999999999e-06 2024-04-23T16:23:53.626 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.626 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.626 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.627 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:38.821419+0000", 2024-04-23T16:23:53.627 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5870000000000003e-06 2024-04-23T16:23:53.627 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.627 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.627 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.627 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:38.821428+0000", 2024-04-23T16:23:53.627 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8810000000000002e-06 2024-04-23T16:23:53.627 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.627 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.627 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.627 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:38.821434+0000", 2024-04-23T16:23:53.627 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5350000000000004e-06 2024-04-23T16:23:53.627 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.627 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.627 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.628 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:38.832748+0000", 2024-04-23T16:23:53.628 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01131416 2024-04-23T16:23:53.628 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.628 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.628 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:53.628 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:38.832846+0000", 2024-04-23T16:23:53.628 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8277999999999997e-05 2024-04-23T16:23:53.628 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.628 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.628 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 2", 2024-04-23T16:23:53.628 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:38.832912+0000", 2024-04-23T16:23:53.628 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5728999999999994e-05 2024-04-23T16:23:53.628 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.628 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.628 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:53.628 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:38.835836+0000", 2024-04-23T16:23:53.629 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00292436 2024-04-23T16:23:53.629 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.629 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.629 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.629 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:38.857977+0000", 2024-04-23T16:23:53.629 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022140697000000001 2024-04-23T16:23:53.629 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.629 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.629 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:53.629 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:38.858006+0000", 2024-04-23T16:23:53.629 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8717999999999998e-05 2024-04-23T16:23:53.629 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.629 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.629 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.630 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:21:38.858044+0000", 2024-04-23T16:23:53.630 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8268e-05 2024-04-23T16:23:53.630 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.630 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.630 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.630 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.630 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.630 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:80497 3.a 3:5e3f3a39:::benchmark_data_smithi188_89276_object80496:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:23:53.630 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:22:51.362606+0000", 2024-04-23T16:23:53.630 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 62.171506692000001, 2024-04-23T16:23:53.630 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.064409358, 2024-04-23T16:23:53.630 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.630 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.630 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:53.631 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.631 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:53.631 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:53.631 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 80497 2024-04-23T16:23:53.631 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.631 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.631 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.631 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.631 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.362606+0000", 2024-04-23T16:23:53.631 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.631 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.631 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.631 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.631 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.362606+0000", 2024-04-23T16:23:53.631 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.632 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.632 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.632 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.632 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.362608+0000", 2024-04-23T16:23:53.632 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.618e-06 2024-04-23T16:23:53.632 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.632 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.632 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.632 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.362656+0000", 2024-04-23T16:23:53.632 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8266999999999997e-05 2024-04-23T16:23:53.632 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.632 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.632 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.632 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.362667+0000", 2024-04-23T16:23:53.632 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0437e-05 2024-04-23T16:23:53.633 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.633 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.633 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.633 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.362674+0000", 2024-04-23T16:23:53.633 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3350000000000001e-06 2024-04-23T16:23:53.633 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.633 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.633 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.633 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363179+0000", 2024-04-23T16:23:53.633 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00050552300000000002 2024-04-23T16:23:53.633 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.633 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.633 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:53.633 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363296+0000", 2024-04-23T16:23:53.634 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000116149 2024-04-23T16:23:53.634 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.634 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.634 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 4", 2024-04-23T16:23:53.634 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363377+0000", 2024-04-23T16:23:53.634 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0902000000000006e-05 2024-04-23T16:23:53.634 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.634 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.634 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:53.634 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.366756+0000", 2024-04-23T16:23:53.634 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003379609 2024-04-23T16:23:53.634 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.634 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.634 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.635 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.426903+0000", 2024-04-23T16:23:53.635 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.060147117 2024-04-23T16:23:53.635 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.635 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.635 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:53.635 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.426970+0000", 2024-04-23T16:23:53.635 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6859000000000005e-05 2024-04-23T16:23:53.635 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.635 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.635 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.635 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.427016+0000", 2024-04-23T16:23:53.635 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5541999999999999e-05 2024-04-23T16:23:53.635 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.635 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.636 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.636 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.636 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.636 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:80498 3.a 3:5a5eb108:::benchmark_data_smithi188_89276_object80497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:23:53.636 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:22:51.363023+0000", 2024-04-23T16:23:53.636 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 62.171090094999997, 2024-04-23T16:23:53.636 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.064145046999999997, 2024-04-23T16:23:53.636 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.636 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.636 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:53.636 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.636 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:53.636 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:53.636 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 80498 2024-04-23T16:23:53.636 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.637 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.637 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.637 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.637 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363023+0000", 2024-04-23T16:23:53.637 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.637 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.637 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.637 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.637 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363023+0000", 2024-04-23T16:23:53.637 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.637 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.637 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.637 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.637 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363024+0000", 2024-04-23T16:23:53.638 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0499999999999999e-06 2024-04-23T16:23:53.638 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.638 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.638 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.638 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363095+0000", 2024-04-23T16:23:53.638 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0840999999999995e-05 2024-04-23T16:23:53.638 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.638 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.638 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.638 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363102+0000", 2024-04-23T16:23:53.638 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6190000000000002e-06 2024-04-23T16:23:53.638 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.638 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.638 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.638 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363112+0000", 2024-04-23T16:23:53.639 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3899999999999999e-06 2024-04-23T16:23:53.639 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.639 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.639 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.639 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.364586+0000", 2024-04-23T16:23:53.639 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0014738830000000001 2024-04-23T16:23:53.639 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.639 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.639 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:53.639 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.364651+0000", 2024-04-23T16:23:53.639 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5362000000000002e-05 2024-04-23T16:23:53.639 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.639 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.639 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 4", 2024-04-23T16:23:53.639 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.364727+0000", 2024-04-23T16:23:53.639 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6289000000000001e-05 2024-04-23T16:23:53.640 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.640 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.640 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:53.640 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.366763+0000", 2024-04-23T16:23:53.640 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0020361289999999998 2024-04-23T16:23:53.640 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.640 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.640 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.640 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.427072+0000", 2024-04-23T16:23:53.640 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.060309019999999998 2024-04-23T16:23:53.640 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.640 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.640 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:53.640 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.427126+0000", 2024-04-23T16:23:53.640 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3275000000000001e-05 2024-04-23T16:23:53.640 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.641 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.641 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.641 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.427168+0000", 2024-04-23T16:23:53.641 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2188999999999999e-05 2024-04-23T16:23:53.641 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.641 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.641 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.641 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.641 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.641 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:80499 3.a 3:5e8abdc3:::benchmark_data_smithi188_89276_object80498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:23:53.641 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:22:51.363150+0000", 2024-04-23T16:23:53.641 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 62.170962901000003, 2024-04-23T16:23:53.641 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.065547096999999999, 2024-04-23T16:23:53.641 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.641 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.642 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:53.642 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.642 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:53.642 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:53.642 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 80499 2024-04-23T16:23:53.642 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.642 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.642 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.642 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.642 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363150+0000", 2024-04-23T16:23:53.642 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.642 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.642 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.642 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.643 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363150+0000", 2024-04-23T16:23:53.643 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.643 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.643 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.643 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.643 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363151+0000", 2024-04-23T16:23:53.643 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5999999999999991e-07 2024-04-23T16:23:53.643 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.643 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.643 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.643 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363185+0000", 2024-04-23T16:23:53.643 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3481e-05 2024-04-23T16:23:53.643 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.643 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.643 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.643 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363191+0000", 2024-04-23T16:23:53.644 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2419999999999997e-06 2024-04-23T16:23:53.644 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.644 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.644 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.644 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363197+0000", 2024-04-23T16:23:53.644 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1460000000000001e-06 2024-04-23T16:23:53.644 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.644 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.644 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.644 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.365745+0000", 2024-04-23T16:23:53.644 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025485849999999999 2024-04-23T16:23:53.644 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.644 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.644 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:53.644 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.365807+0000", 2024-04-23T16:23:53.644 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1884999999999994e-05 2024-04-23T16:23:53.645 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.645 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.645 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 4", 2024-04-23T16:23:53.645 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.365882+0000", 2024-04-23T16:23:53.645 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5100999999999998e-05 2024-04-23T16:23:53.645 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.645 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.645 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:53.645 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.368007+0000", 2024-04-23T16:23:53.645 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021242460000000002 2024-04-23T16:23:53.645 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.645 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.645 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.645 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.428631+0000", 2024-04-23T16:23:53.645 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.060624685999999997 2024-04-23T16:23:53.646 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.646 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.646 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:53.646 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.428671+0000", 2024-04-23T16:23:53.646 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9449000000000001e-05 2024-04-23T16:23:53.646 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.646 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.646 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.646 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.428697+0000", 2024-04-23T16:23:53.646 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6316e-05 2024-04-23T16:23:53.646 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.646 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.646 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.646 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.647 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.647 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:80500 3.a 3:5f44e714:::benchmark_data_smithi188_89276_object80499:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:23:53.647 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:22:51.363206+0000", 2024-04-23T16:23:53.647 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 62.170907305999997, 2024-04-23T16:23:53.647 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.065700248000000003, 2024-04-23T16:23:53.647 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:53.647 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:53.647 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:23:53.647 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:53.647 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:53.647 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:53.647 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 80500 2024-04-23T16:23:53.647 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.647 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:53.648 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.648 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:53.648 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363206+0000", 2024-04-23T16:23:53.648 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.648 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.648 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.648 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:53.648 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363206+0000", 2024-04-23T16:23:53.648 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:53.648 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.648 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.648 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:53.648 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363206+0000", 2024-04-23T16:23:53.648 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1800000000000005e-07 2024-04-23T16:23:53.648 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.648 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.649 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:53.649 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363266+0000", 2024-04-23T16:23:53.649 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9630999999999999e-05 2024-04-23T16:23:53.649 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.649 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.649 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:53.649 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363273+0000", 2024-04-23T16:23:53.649 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.759e-06 2024-04-23T16:23:53.649 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.649 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.649 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:53.649 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.363279+0000", 2024-04-23T16:23:53.649 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3559999999999998e-06 2024-04-23T16:23:53.649 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.649 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.650 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:53.650 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.366821+0000", 2024-04-23T16:23:53.650 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0035417420000000001 2024-04-23T16:23:53.650 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.650 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.650 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:53.650 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.366891+0000", 2024-04-23T16:23:53.650 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0438000000000006e-05 2024-04-23T16:23:53.650 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.650 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.650 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 4", 2024-04-23T16:23:53.650 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.366972+0000", 2024-04-23T16:23:53.650 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0953999999999998e-05 2024-04-23T16:23:53.650 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.650 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.651 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:23:53.651 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.368116+0000", 2024-04-23T16:23:53.651 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0011436269999999999 2024-04-23T16:23:53.651 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.651 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.651 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:23:53.651 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.428811+0000", 2024-04-23T16:23:53.651 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.060695297000000002 2024-04-23T16:23:53.651 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.651 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.651 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:23:53.651 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.428869+0000", 2024-04-23T16:23:53.651 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8199000000000003e-05 2024-04-23T16:23:53.651 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:53.651 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:53.652 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:23:53.652 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:51.428906+0000", 2024-04-23T16:23:53.652 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6526999999999998e-05 2024-04-23T16:23:53.652 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.652 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.652 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.652 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:53.652 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:53.652 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:23:53.652 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:23:53.689 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:53.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:53 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:23:53.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:53 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:23:53.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:53 smithi103 ceph-mon[39629]: pgmap v654: 17 pgs: 17 active+clean; 9.8 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-04-23T16:23:53.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:53 smithi103 ceph-mon[39629]: osdmap e226: 12 total, 12 up, 11 in 2024-04-23T16:23:53.852 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:23:53.856 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:23:53.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:53.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:23:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:53 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:23:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:53 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:23:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:53 smithi148 ceph-mon[44520]: pgmap v654: 17 pgs: 17 active+clean; 9.8 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-04-23T16:23:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:53 smithi148 ceph-mon[44520]: osdmap e226: 12 total, 12 up, 11 in 2024-04-23T16:23:53.963 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:23:53.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:53 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:23:53.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:53 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:23:53.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:53 smithi188 ceph-mon[44101]: pgmap v654: 17 pgs: 17 active+clean; 9.8 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-04-23T16:23:53.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:53 smithi188 ceph-mon[44101]: osdmap e226: 12 total, 12 up, 11 in 2024-04-23T16:23:53.993 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:54.017 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:23:54.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:54.022 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:23:54.123 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:23:54.154 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:54.177 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:23:54.180 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:23:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:54.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:23:54.186 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:23:54.282 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:23:54.302 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-23T16:23:54.312 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:54.343 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:23:54.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:54.346 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:23:54.447 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:54.480 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 88 16 166253 166237 118.056 72.4375 0.0147569 0.00846879 2024-04-23T16:23:54.500 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:54.531 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:54.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:54.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:54.635 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:23:54.664 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:54.679 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:23:54.688 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:23:54.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:54.696 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:23:54.796 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:23:54.826 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:54.847 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:23:54.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:54.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:23:54.954 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:23:54.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:54 smithi103 ceph-mon[39629]: from='client.15234 v1:172.21.15.103:0/942913026' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:23:54.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:54 smithi103 ceph-mon[39629]: pgmap v656: 17 pgs: 17 active+clean; 10 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-23T16:23:55.007 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:55.182 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:54 smithi148 ceph-mon[44520]: from='client.15234 v1:172.21.15.103:0/942913026' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:23:55.182 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:54 smithi148 ceph-mon[44520]: pgmap v656: 17 pgs: 17 active+clean; 10 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-23T16:23:55.262 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:23:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:54 smithi188 ceph-mon[44101]: from='client.15234 v1:172.21.15.103:0/942913026' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:23:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:54 smithi188 ceph-mon[44101]: pgmap v656: 17 pgs: 17 active+clean; 10 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-23T16:23:55.480 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 89 16 167475 167459 117.587 76.375 0.0102819 0.0085028 2024-04-23T16:23:55.576 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:23:55.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:55.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:23:55.683 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:23:55.738 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:56.122 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":655,"stamp":"2024-04-23T16:23:52.696843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10511319584,"num_objects":160383,"num_object_clones":0,"num_object_copies":320766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160383,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":160562,"num_write_kb":10267322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159635,"ondisk_log_size":159635,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":21979676,"kb_used_data":20488616,"kb_used_omap":17,"kb_used_meta":1490926,"kb_avail":1009216996,"statfs":{"total":1055945392128,"available":1033438203904,"internally_reserved":0,"allocated":20980342784,"data_stored":20976604217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17543,"internal_metadata":1526709113},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":591593472,"num_objects":9027,"num_object_clones":0,"num_object_copies":18054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9027,"num_write_kb":577728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.047764"},"pg_stats":[{"pgid":"3.f","version":"225'10113","reported_seq":10188,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-23T16:23:49.619648+0000","last_change":"2024-04-23T16:22:11.619510+0000","last_active":"2024-04-23T16:23:49.619648+0000","last_peered":"2024-04-23T16:23:49.619648+0000","last_clean":"2024-04-23T16:23:49.619648+0000","last_became_active":"2024-04-23T16:22:11.619226+0000","last_became_peered":"2024-04-23T16:22:11.619226+0000","last_unstale":"2024-04-23T16:23:49.619648+0000","last_undegraded":"2024-04-23T16:23:49.619648+0000","last_fullsized":"2024-04-23T16:23:49.619648+0000","mapping_epoch":192,"log_start":"199'100","ondisk_log_start":"199'100","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:22:10.601504+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":662765568,"num_objects":10113,"num_object_clones":0,"num_object_copies":20226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10113,"num_write_kb":647232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"225'10112","reported_seq":10271,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-23T16:23:51.085186+0000","last_change":"2024-04-23T16:23:14.444209+0000","last_active":"2024-04-23T16:23:51.085186+0000","last_peered":"2024-04-23T16:23:51.085186+0000","last_clean":"2024-04-23T16:23:51.085186+0000","last_became_active":"2024-04-23T16:22:11.618509+0000","last_became_peered":"2024-04-23T16:22:11.618509+0000","last_unstale":"2024-04-23T16:23:51.085186+0000","last_undegraded":"2024-04-23T16:23:51.085186+0000","last_fullsized":"2024-04-23T16:23:51.085186+0000","mapping_epoch":192,"log_start":"199'100","ondisk_log_start":"199'100","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7543","last_scrub_stamp":"2024-04-23T16:23:14.443997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:14.443997+0000","objects_scrubbed":7482,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:23.327257+0000","scrub_duration":1349,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":662700032,"num_objects":10112,"num_object_clones":0,"num_object_copies":20224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10112,"num_write_kb":647168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"224'9807","reported_seq":9973,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-23T16:23:47.691673+0000","last_change":"2024-04-23T16:23:27.617196+0000","last_active":"2024-04-23T16:23:47.691673+0000","last_peered":"2024-04-23T16:23:47.691673+0000","last_clean":"2024-04-23T16:23:47.691673+0000","last_became_active":"2024-04-23T16:22:11.616591+0000","last_became_peered":"2024-04-23T16:22:11.616591+0000","last_unstale":"2024-04-23T16:23:47.691673+0000","last_undegraded":"2024-04-23T16:23:47.691673+0000","last_fullsized":"2024-04-23T16:23:47.691673+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'8650","last_scrub_stamp":"2024-04-23T16:23:27.616985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:27.616985+0000","objects_scrubbed":8593,"log_size":9807,"log_dups_size":0,"ondisk_log_size":9807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:50.139450+0000","scrub_duration":1550,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":642711552,"num_objects":9807,"num_object_clones":0,"num_object_copies":19614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9807,"num_write_kb":627648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"224'9922","reported_seq":10082,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-23T16:23:47.708930+0000","last_change":"2024-04-23T16:23:18.692178+0000","last_active":"2024-04-23T16:23:47.708930+0000","last_peered":"2024-04-23T16:23:47.708930+0000","last_clean":"2024-04-23T16:23:47.708930+0000","last_became_active":"2024-04-23T16:22:11.616487+0000","last_became_peered":"2024-04-23T16:22:11.616487+0000","last_unstale":"2024-04-23T16:23:47.708930+0000","last_undegraded":"2024-04-23T16:23:47.708930+0000","last_fullsized":"2024-04-23T16:23:47.708930+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'7886","last_scrub_stamp":"2024-04-23T16:23:18.691923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:18.691923+0000","objects_scrubbed":7817,"log_size":9922,"log_dups_size":0,"ondisk_log_size":9922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:40.335633+0000","scrub_duration":1596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":650248192,"num_objects":9922,"num_object_clones":0,"num_object_copies":19844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9922,"num_write_kb":635008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"225'10181","reported_seq":10341,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-23T16:23:50.694474+0000","last_change":"2024-04-23T16:23:15.831556+0000","last_active":"2024-04-23T16:23:50.694474+0000","last_peered":"2024-04-23T16:23:50.694474+0000","last_clean":"2024-04-23T16:23:50.694474+0000","last_became_active":"2024-04-23T16:22:11.615324+0000","last_became_peered":"2024-04-23T16:22:11.615324+0000","last_unstale":"2024-04-23T16:23:50.694474+0000","last_undegraded":"2024-04-23T16:23:50.694474+0000","last_fullsized":"2024-04-23T16:23:50.694474+0000","mapping_epoch":192,"log_start":"199'100","ondisk_log_start":"199'100","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7648","last_scrub_stamp":"2024-04-23T16:23:15.831380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:15.831380+0000","objects_scrubbed":7565,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:16.038149+0000","scrub_duration":1721,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":667222016,"num_objects":10181,"num_object_clones":0,"num_object_copies":20362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10181,"num_write_kb":651584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"225'9932","reported_seq":10032,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-23T16:23:49.002280+0000","last_change":"2024-04-23T16:23:33.410487+0000","last_active":"2024-04-23T16:23:49.002280+0000","last_peered":"2024-04-23T16:23:49.002280+0000","last_clean":"2024-04-23T16:23:49.002280+0000","last_became_active":"2024-04-23T16:22:11.616590+0000","last_became_peered":"2024-04-23T16:22:11.616590+0000","last_unstale":"2024-04-23T16:23:49.002280+0000","last_undegraded":"2024-04-23T16:23:49.002280+0000","last_fullsized":"2024-04-23T16:23:49.002280+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:22:10.601504+0000","objects_scrubbed":2199,"log_size":9932,"log_dups_size":0,"ondisk_log_size":9932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:23:50.139057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":650903552,"num_objects":9932,"num_object_clones":0,"num_object_copies":19864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9932,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"224'9900","reported_seq":10055,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-23T16:23:48.098638+0000","last_change":"2024-04-23T16:23:14.734017+0000","last_active":"2024-04-23T16:23:48.098638+0000","last_peered":"2024-04-23T16:23:48.098638+0000","last_clean":"2024-04-23T16:23:48.098638+0000","last_became_active":"2024-04-23T16:22:11.618760+0000","last_became_peered":"2024-04-23T16:22:11.618760+0000","last_unstale":"2024-04-23T16:23:48.098638+0000","last_undegraded":"2024-04-23T16:23:48.098638+0000","last_fullsized":"2024-04-23T16:23:48.098638+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7491","last_scrub_stamp":"2024-04-23T16:23:14.733801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:14.733801+0000","objects_scrubbed":7430,"log_size":9900,"log_dups_size":0,"ondisk_log_size":9900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:38.345896+0000","scrub_duration":1267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":648806400,"num_objects":9900,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9900,"num_write_kb":633600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"225'10291","reported_seq":10461,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-23T16:23:51.097586+0000","last_change":"2024-04-23T16:23:26.020510+0000","last_active":"2024-04-23T16:23:51.097586+0000","last_peered":"2024-04-23T16:23:51.097586+0000","last_clean":"2024-04-23T16:23:51.097586+0000","last_became_active":"2024-04-23T16:22:11.622030+0000","last_became_peered":"2024-04-23T16:22:11.622030+0000","last_unstale":"2024-04-23T16:23:51.097586+0000","last_undegraded":"2024-04-23T16:23:51.097586+0000","last_fullsized":"2024-04-23T16:23:51.097586+0000","mapping_epoch":192,"log_start":"199'200","ondisk_log_start":"199'200","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'8646","last_scrub_stamp":"2024-04-23T16:23:26.020368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:26.020368+0000","objects_scrubbed":8556,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:52.131906+0000","scrub_duration":1878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":674430976,"num_objects":10291,"num_object_clones":0,"num_object_copies":20582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10291,"num_write_kb":658624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"225'10208","reported_seq":10369,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-23T16:23:50.708534+0000","last_change":"2024-04-23T16:23:16.404142+0000","last_active":"2024-04-23T16:23:50.708534+0000","last_peered":"2024-04-23T16:23:50.708534+0000","last_clean":"2024-04-23T16:23:50.708534+0000","last_became_active":"2024-04-23T16:22:11.619141+0000","last_became_peered":"2024-04-23T16:22:11.619141+0000","last_unstale":"2024-04-23T16:23:50.708534+0000","last_undegraded":"2024-04-23T16:23:50.708534+0000","last_fullsized":"2024-04-23T16:23:50.708534+0000","mapping_epoch":192,"log_start":"199'200","ondisk_log_start":"199'200","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7744","last_scrub_stamp":"2024-04-23T16:23:16.403981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:16.403981+0000","objects_scrubbed":7705,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:24.288145+0000","scrub_duration":1262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":668991488,"num_objects":10208,"num_object_clones":0,"num_object_copies":20416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10208,"num_write_kb":653312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"225'9865","reported_seq":10033,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-23T16:23:48.972024+0000","last_change":"2024-04-23T16:23:28.605101+0000","last_active":"2024-04-23T16:23:48.972024+0000","last_peered":"2024-04-23T16:23:48.972024+0000","last_clean":"2024-04-23T16:23:48.972024+0000","last_became_active":"2024-04-23T16:22:11.615706+0000","last_became_peered":"2024-04-23T16:22:11.615706+0000","last_unstale":"2024-04-23T16:23:48.972024+0000","last_undegraded":"2024-04-23T16:23:48.972024+0000","last_fullsized":"2024-04-23T16:23:48.972024+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'8608","last_scrub_stamp":"2024-04-23T16:23:28.604991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:28.604991+0000","objects_scrubbed":8566,"log_size":9865,"log_dups_size":0,"ondisk_log_size":9865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:42.709540+0000","scrub_duration":1382,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":646512640,"num_objects":9865,"num_object_clones":0,"num_object_copies":19730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9865,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"224'9817","reported_seq":9985,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-23T16:23:47.712804+0000","last_change":"2024-04-23T16:23:30.737830+0000","last_active":"2024-04-23T16:23:47.712804+0000","last_peered":"2024-04-23T16:23:47.712804+0000","last_clean":"2024-04-23T16:23:47.712804+0000","last_became_active":"2024-04-23T16:22:11.616683+0000","last_became_peered":"2024-04-23T16:22:11.616683+0000","last_unstale":"2024-04-23T16:23:47.712804+0000","last_undegraded":"2024-04-23T16:23:47.712804+0000","last_fullsized":"2024-04-23T16:23:47.712804+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'8829","last_scrub_stamp":"2024-04-23T16:23:30.737605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:30.737605+0000","objects_scrubbed":8773,"log_size":9817,"log_dups_size":0,"ondisk_log_size":9817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:43.515529+0000","scrub_duration":1732,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":643366912,"num_objects":9817,"num_object_clones":0,"num_object_copies":19634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9817,"num_write_kb":628288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"225'11849","reported_seq":12029,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-23T16:23:51.626071+0000","last_change":"2024-04-23T16:23:42.767527+0000","last_active":"2024-04-23T16:23:51.626071+0000","last_peered":"2024-04-23T16:23:51.626071+0000","last_clean":"2024-04-23T16:23:51.626071+0000","last_became_active":"2024-04-23T16:22:11.619030+0000","last_became_peered":"2024-04-23T16:22:11.619030+0000","last_unstale":"2024-04-23T16:23:51.626071+0000","last_undegraded":"2024-04-23T16:23:51.626071+0000","last_fullsized":"2024-04-23T16:23:51.626071+0000","mapping_epoch":192,"log_start":"200'1800","ondisk_log_start":"200'1800","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:22:10.601504+0000","objects_scrubbed":2536,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:23:56.162415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":653787136,"num_objects":9976,"num_object_clones":0,"num_object_copies":19952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9976,"num_write_kb":638464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"225'9965","reported_seq":10136,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-23T16:23:48.984960+0000","last_change":"2024-04-23T16:23:31.078016+0000","last_active":"2024-04-23T16:23:48.984960+0000","last_peered":"2024-04-23T16:23:48.984960+0000","last_clean":"2024-04-23T16:23:48.984960+0000","last_became_active":"2024-04-23T16:22:11.618441+0000","last_became_peered":"2024-04-23T16:22:11.618441+0000","last_unstale":"2024-04-23T16:23:48.984960+0000","last_undegraded":"2024-04-23T16:23:48.984960+0000","last_fullsized":"2024-04-23T16:23:48.984960+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'8843","last_scrub_stamp":"2024-04-23T16:23:31.077889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:31.077889+0000","objects_scrubbed":8806,"log_size":9965,"log_dups_size":0,"ondisk_log_size":9965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:40.425971+0000","scrub_duration":1453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":653066240,"num_objects":9965,"num_object_clones":0,"num_object_copies":19930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9965,"num_write_kb":637760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"225'9906","reported_seq":10057,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-23T16:23:49.013369+0000","last_change":"2024-04-23T16:23:33.420726+0000","last_active":"2024-04-23T16:23:49.013369+0000","last_peered":"2024-04-23T16:23:49.013369+0000","last_clean":"2024-04-23T16:23:49.013369+0000","last_became_active":"2024-04-23T16:22:11.616598+0000","last_became_peered":"2024-04-23T16:22:11.616598+0000","last_unstale":"2024-04-23T16:23:49.013369+0000","last_undegraded":"2024-04-23T16:23:49.013369+0000","last_fullsized":"2024-04-23T16:23:49.013369+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:22:10.601504+0000","objects_scrubbed":7349,"log_size":9906,"log_dups_size":0,"ondisk_log_size":9906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:23:53.177278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":649199616,"num_objects":9906,"num_object_clones":0,"num_object_copies":19812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9906,"num_write_kb":633984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"225'12182","reported_seq":12369,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-23T16:23:51.136214+0000","last_change":"2024-04-23T16:23:42.673174+0000","last_active":"2024-04-23T16:23:51.136214+0000","last_peered":"2024-04-23T16:23:51.136214+0000","last_clean":"2024-04-23T16:23:51.136214+0000","last_became_active":"2024-04-23T16:22:11.664655+0000","last_became_peered":"2024-04-23T16:22:11.664655+0000","last_unstale":"2024-04-23T16:23:51.136214+0000","last_undegraded":"2024-04-23T16:23:51.136214+0000","last_fullsized":"2024-04-23T16:23:51.136214+0000","mapping_epoch":192,"log_start":"200'2100","ondisk_log_start":"200'2100","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:22:10.601504+0000","objects_scrubbed":2686,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:23:54.257115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":670957568,"num_objects":10238,"num_object_clones":0,"num_object_copies":20476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10238,"num_write_kb":655232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"225'10148","reported_seq":10223,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-23T16:23:50.718704+0000","last_change":"2024-04-23T16:22:11.664979+0000","last_active":"2024-04-23T16:23:50.718704+0000","last_peered":"2024-04-23T16:23:50.718704+0000","last_clean":"2024-04-23T16:23:50.718704+0000","last_became_active":"2024-04-23T16:22:11.664765+0000","last_became_peered":"2024-04-23T16:22:11.664765+0000","last_unstale":"2024-04-23T16:23:50.718704+0000","last_undegraded":"2024-04-23T16:23:50.718704+0000","last_fullsized":"2024-04-23T16:23:50.718704+0000","mapping_epoch":192,"log_start":"199'100","ondisk_log_start":"199'100","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:22:10.601504+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:23:54.260140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":665059328,"num_objects":10148,"num_object_clones":0,"num_object_copies":20296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10148,"num_write_kb":649472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'137","reported_seq":631,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-23T16:23:48.672360+0000","last_change":"2024-04-23T16:23:16.763496+0000","last_active":"2024-04-23T16:23:48.672360+0000","last_peered":"2024-04-23T16:23:48.672360+0000","last_clean":"2024-04-23T16:23:48.672360+0000","last_became_active":"2024-04-23T16:15:36.918552+0000","last_became_peered":"2024-04-23T16:15:36.918552+0000","last_unstale":"2024-04-23T16:23:48.672360+0000","last_undegraded":"2024-04-23T16:23:48.672360+0000","last_fullsized":"2024-04-23T16:23:48.672360+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"16'137","last_scrub_stamp":"2024-04-23T16:23:16.763345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:11:23.401899+0000","last_clean_scrub_stamp":"2024-04-23T16:23:16.763345+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:41.299861+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":10510729216,"num_objects":160381,"num_object_clones":0,"num_object_copies":320762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":160381,"num_write_kb":10264384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21063860224,"data_stored":21063860224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159498,"ondisk_log_size":159498,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056742,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4764084,"kb_used_data":4530216,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":88981068,"statfs":{"total":95995035648,"available":91116613632,"internally_reserved":0,"allocated":4638941184,"data_stored":4638595115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":239466948},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351084,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612722,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1431196,"kb_used_data":1283412,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92313956,"statfs":{"total":95995035648,"available":94529490944,"internally_reserved":0,"allocated":1314213888,"data_stored":1313877395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":151321024},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841653,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":756828,"kb_used_data":631252,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":92988324,"statfs":{"total":95995035648,"available":95220043776,"internally_reserved":0,"allocated":646402048,"data_stored":646065555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168699,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1405360,"kb_used_data":1283944,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92339792,"statfs":{"total":95995035648,"available":94555947008,"internally_reserved":0,"allocated":1314758656,"data_stored":1314412607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":124320202},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463041,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":707484,"kb_used_data":631892,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93037668,"statfs":{"total":95995035648,"available":95270572032,"internally_reserved":0,"allocated":647057408,"data_stored":646720915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575002,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2092716,"kb_used_data":1939360,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":91652436,"statfs":{"total":95995035648,"available":93852094464,"internally_reserved":0,"allocated":1985904640,"data_stored":1985562143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738526,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2672412,"kb_used_data":2528788,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":91072740,"statfs":{"total":95995035648,"available":93258485760,"internally_reserved":0,"allocated":2589478912,"data_stored":2589142419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313363,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":130412,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93614740,"statfs":{"total":95995035648,"available":95861493760,"internally_reserved":0,"allocated":675840,"data_stored":339347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117135,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3371880,"kb_used_data":3236576,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":90373272,"statfs":{"total":95995035648,"available":92542230528,"internally_reserved":0,"allocated":3314253824,"data_stored":3313911327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":138541511},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920905,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2643628,"kb_used_data":2540256,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":91101524,"statfs":{"total":95995035648,"available":93287960576,"internally_reserved":0,"allocated":2601222144,"data_stored":2600879647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757380,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2003676,"kb_used_data":1882260,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":91741476,"statfs":{"total":95995035648,"available":93943271424,"internally_reserved":0,"allocated":1927434240,"data_stored":1927097747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2607677440,"data_stored":2607677440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1991639040,"data_stored":1991639040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3324968960,"data_stored":3324968960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2605645824,"data_stored":2605645824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1936392192,"data_stored":1936392192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":650903552,"data_stored":650903552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1314783232,"data_stored":1314783232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":648806400,"data_stored":648806400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1315831808,"data_stored":1315831808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4667211776,"data_stored":4667211776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:23:56.124 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:23:56.256 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:55 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:23:56.256 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:55 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:23:56.256 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:55 smithi148 ceph-mon[44520]: osdmap e227: 12 total, 12 up, 11 in 2024-04-23T16:23:56.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:55 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:23:56.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:55 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:23:56.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:55 smithi103 ceph-mon[39629]: osdmap e227: 12 total, 12 up, 11 in 2024-04-23T16:23:56.398 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:23:56.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:56.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:23:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:55 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:23:56.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:55 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:23:56.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:55 smithi188 ceph-mon[44101]: osdmap e227: 12 total, 12 up, 11 in 2024-04-23T16:23:56.481 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Total time run: 90.0061 2024-04-23T16:23:56.481 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Total writes made: 168641 2024-04-23T16:23:56.481 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Write size: 65536 2024-04-23T16:23:56.481 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Object size: 65536 2024-04-23T16:23:56.481 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Bandwidth (MB/sec): 117.104 2024-04-23T16:23:56.481 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev Bandwidth: 67.4936 2024-04-23T16:23:56.481 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max bandwidth (MB/sec): 393.438 2024-04-23T16:23:56.481 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min bandwidth (MB/sec): 49.6875 2024-04-23T16:23:56.481 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Average IOPS: 1873 2024-04-23T16:23:56.481 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev IOPS: 1079.9 2024-04-23T16:23:56.481 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max IOPS: 6295 2024-04-23T16:23:56.481 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min IOPS: 795 2024-04-23T16:23:56.481 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Average Latency(s): 0.00853881 2024-04-23T16:23:56.482 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev Latency(s): 0.00929861 2024-04-23T16:23:56.482 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max latency(s): 0.225871 2024-04-23T16:23:56.482 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min latency(s): 0.00099496 2024-04-23T16:23:56.486 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Cleaning up (deleting benchmark objects) 2024-04-23T16:23:56.505 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:23:56.579 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:56.606 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:23:56.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:56.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:23:56.620 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:23:56.620 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:23:56.620 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:56.620 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:168789 3.f 3:f760710c:::benchmark_data_smithi188_89276_object145:head [delete] snapc 0=[] ondisk+write+known_if_redirected e227)", 2024-04-23T16:23:56.620 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:56.612235+0000", 2024-04-23T16:23:56.620 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.00053260500000000001, 2024-04-23T16:23:56.621 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000566896, 2024-04-23T16:23:56.621 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:23:56.621 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:23:56.621 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:23:56.621 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:23:56.621 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:23:56.621 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:23:56.621 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 168789 2024-04-23T16:23:56.621 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:56.621 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:23:56.621 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:56.621 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:23:56.621 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:56.612235+0000", 2024-04-23T16:23:56.621 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:56.621 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:56.621 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:56.622 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:23:56.622 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:56.612235+0000", 2024-04-23T16:23:56.622 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:23:56.622 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:56.622 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:56.622 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:23:56.622 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:56.612238+0000", 2024-04-23T16:23:56.622 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.576e-06 2024-04-23T16:23:56.622 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:56.622 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:56.622 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:23:56.622 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:56.612247+0000", 2024-04-23T16:23:56.622 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8419999999999994e-06 2024-04-23T16:23:56.622 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:56.622 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:56.622 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:23:56.623 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:56.612264+0000", 2024-04-23T16:23:56.623 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7506000000000001e-05 2024-04-23T16:23:56.623 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:56.623 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:56.623 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:23:56.623 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:56.612274+0000", 2024-04-23T16:23:56.623 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0135e-05 2024-04-23T16:23:56.623 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:56.623 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:56.623 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:23:56.623 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:56.612339+0000", 2024-04-23T16:23:56.623 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4447000000000001e-05 2024-04-23T16:23:56.623 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:56.623 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:56.623 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:23:56.623 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:56.612473+0000", 2024-04-23T16:23:56.624 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000134809 2024-04-23T16:23:56.624 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:23:56.624 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:23:56.624 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:23:56.624 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:56.612600+0000", 2024-04-23T16:23:56.624 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012688399999999999 2024-04-23T16:23:56.624 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:56.624 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:23:56.624 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:56.624 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:23:56.624 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-04-23T16:23:56.624 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 1 2024-04-23T16:23:56.624 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:23:56.712 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:23:56.719 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:23:56.747 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:56.789 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:23:56.906 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:23:56.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:56.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:23:57.011 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:23:57.040 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:57.257 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:56 smithi103 ceph-mon[39629]: pgmap v658: 17 pgs: 17 active+clean; 10 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-23T16:23:57.321 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:56 smithi148 ceph-mon[44520]: pgmap v658: 17 pgs: 17 active+clean; 10 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-23T16:23:57.356 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-23T16:23:57.393 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:23:57.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:57.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:23:57.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:56 smithi188 ceph-mon[44101]: pgmap v658: 17 pgs: 17 active+clean; 10 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-23T16:23:57.497 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:23:57.542 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:57.749 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:23:58.022 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:23:58.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:57 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2278265091' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:23:58.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:57 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2278265091' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:23:58.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:57 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2278265091' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:23:59.110 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:23:59.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:59.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:23:59.219 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:23:59.253 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:58 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:23:59.253 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:58 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2278265091' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:23:59.253 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:58 smithi188 ceph-mon[44101]: osdmap e228: 12 total, 12 up, 11 in 2024-04-23T16:23:59.253 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:23:58 smithi188 ceph-mon[44101]: pgmap v660: 17 pgs: 17 active+clean; 10 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 170 B/s rd, 75 MiB/s wr, 1.26k op/s 2024-04-23T16:23:59.258 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:59.281 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:23:59.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:59.286 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:23:59.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:58 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:23:59.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:58 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2278265091' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:23:59.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:58 smithi103 ceph-mon[39629]: osdmap e228: 12 total, 12 up, 11 in 2024-04-23T16:23:59.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:23:58 smithi103 ceph-mon[39629]: pgmap v660: 17 pgs: 17 active+clean; 10 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 170 B/s rd, 75 MiB/s wr, 1.26k op/s 2024-04-23T16:23:59.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:58 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:23:59.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:58 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2278265091' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:23:59.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:58 smithi148 ceph-mon[44520]: osdmap e228: 12 total, 12 up, 11 in 2024-04-23T16:23:59.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:23:58 smithi148 ceph-mon[44520]: pgmap v660: 17 pgs: 17 active+clean; 10 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 170 B/s rd, 75 MiB/s wr, 1.26k op/s 2024-04-23T16:23:59.387 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:23:59.415 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:59.437 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:23:59.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:59.442 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:23:59.542 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:23:59.577 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:23:59.934 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:23:59.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:23:59.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:24:00.039 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:24:00.076 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:00.097 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:24:00.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:00.102 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:24:00.203 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:24:00.234 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:00.263 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:24:00.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:00.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:24:00.371 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:24:00.399 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:00.420 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:24:00.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:00.430 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:24:00.531 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:24:00.565 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:00.675 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:24:00.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:00.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:24:00.784 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:24:00.819 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:00.980 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:24:00.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:00.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:24:01.085 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:24:01.115 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:01.143 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:24:01.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:01.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:24:01.203 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:24:01.233 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:24:01.249 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:01.257 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:24:01.257 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:24:01.283 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:01.301 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:24:01.311 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:01.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:01.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:01.351 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:24:01.400 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:24:01.418 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:24:01.454 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:24:01.478 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:01.497 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:24:01.543 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:24:01.572 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:24:01.572 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:24:01.573 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:24:01.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:01.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:24:01.597 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:24:01.612 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:24:01.632 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:24:01.673 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:24:01.686 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:24:01.713 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:24:01.721 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:01.744 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:24:01.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:01.747 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:24:01.764 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:24:01.764 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:24:01.764 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:24:01.764 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:24:01.764 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:24:01.764 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:24:01.764 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:24:01.765 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:24:01.765 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:24:01.765 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:24:01.765 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:24:01.765 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:24:01.847 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:24:01.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:01 smithi103 ceph-mon[39629]: pgmap v661: 17 pgs: 17 active+clean; 10 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 129 B/s rd, 64 MiB/s wr, 1.21k op/s 2024-04-23T16:24:01.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:01 smithi103 ceph-mon[39629]: 3.1 scrub starts 2024-04-23T16:24:01.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:01 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2513504051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:01.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:01.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:01.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:01 smithi103 ceph-mon[39629]: 3.4 scrub starts 2024-04-23T16:24:01.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:01.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:01 smithi148 ceph-mon[44520]: pgmap v661: 17 pgs: 17 active+clean; 10 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 129 B/s rd, 64 MiB/s wr, 1.21k op/s 2024-04-23T16:24:01.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:01 smithi148 ceph-mon[44520]: 3.1 scrub starts 2024-04-23T16:24:01.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:01 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2513504051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:01.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:01.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:01.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:01 smithi148 ceph-mon[44520]: 3.4 scrub starts 2024-04-23T16:24:01.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:01.877 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:01.899 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:24:01.904 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:01 smithi188 ceph-mon[44101]: pgmap v661: 17 pgs: 17 active+clean; 10 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 129 B/s rd, 64 MiB/s wr, 1.21k op/s 2024-04-23T16:24:01.904 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:01 smithi188 ceph-mon[44101]: 3.1 scrub starts 2024-04-23T16:24:01.904 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:01 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2513504051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:01.904 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:01.904 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:01.904 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:01 smithi188 ceph-mon[44101]: 3.4 scrub starts 2024-04-23T16:24:01.904 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:01.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:01.905 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:24:02.006 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:24:02.037 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:02.289 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:24:02.290 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-23T16:24:02.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:02.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:24:02.311 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":661,"stamp":"2024-04-23T16:24:00.580734+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10940121656,"num_objects":166927,"num_object_clones":0,"num_object_copies":333854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166927,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":170086,"num_write_kb":10781435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161159,"ondisk_log_size":161159,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":22877320,"kb_used_data":21340296,"kb_used_omap":17,"kb_used_meta":1536750,"kb_avail":1008319352,"statfs":{"total":1055945392128,"available":1032519016448,"internally_reserved":0,"allocated":21852463104,"data_stored":21848511957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17543,"internal_metadata":1573632889},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,5,13,7,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":428802072,"num_objects":6544,"num_object_clones":0,"num_object_copies":13088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6544,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9524,"num_write_kb":514113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.883890"},"pg_stats":[{"pgid":"3.f","version":"228'10894","reported_seq":10975,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-23T16:23:59.647918+0000","last_change":"2024-04-23T16:22:11.619510+0000","last_active":"2024-04-23T16:23:59.647918+0000","last_peered":"2024-04-23T16:23:59.647918+0000","last_clean":"2024-04-23T16:23:59.647918+0000","last_became_active":"2024-04-23T16:22:11.619226+0000","last_became_peered":"2024-04-23T16:22:11.619226+0000","last_unstale":"2024-04-23T16:23:59.647918+0000","last_undegraded":"2024-04-23T16:23:59.647918+0000","last_fullsized":"2024-04-23T16:23:59.647918+0000","mapping_epoch":192,"log_start":"199'800","ondisk_log_start":"199'800","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:22:10.601504+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:01.569501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":678952960,"num_objects":10360,"num_object_clones":0,"num_object_copies":20720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10894,"num_write_kb":680128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"227'10488","reported_seq":10651,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-23T16:23:56.102450+0000","last_change":"2024-04-23T16:23:14.444209+0000","last_active":"2024-04-23T16:23:56.102450+0000","last_peered":"2024-04-23T16:23:56.102450+0000","last_clean":"2024-04-23T16:23:56.102450+0000","last_became_active":"2024-04-23T16:22:11.618509+0000","last_became_peered":"2024-04-23T16:22:11.618509+0000","last_unstale":"2024-04-23T16:23:56.102450+0000","last_undegraded":"2024-04-23T16:23:56.102450+0000","last_fullsized":"2024-04-23T16:23:56.102450+0000","mapping_epoch":192,"log_start":"199'400","ondisk_log_start":"199'400","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7543","last_scrub_stamp":"2024-04-23T16:23:14.443997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:14.443997+0000","objects_scrubbed":7482,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:23.327257+0000","scrub_duration":1349,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":687341568,"num_objects":10488,"num_object_clones":0,"num_object_copies":20976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10488,"num_write_kb":671232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"227'10562","reported_seq":10735,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-23T16:23:57.689141+0000","last_change":"2024-04-23T16:23:27.617196+0000","last_active":"2024-04-23T16:23:57.689141+0000","last_peered":"2024-04-23T16:23:57.689141+0000","last_clean":"2024-04-23T16:23:57.689141+0000","last_became_active":"2024-04-23T16:22:11.616591+0000","last_became_peered":"2024-04-23T16:22:11.616591+0000","last_unstale":"2024-04-23T16:23:57.689141+0000","last_undegraded":"2024-04-23T16:23:57.689141+0000","last_fullsized":"2024-04-23T16:23:57.689141+0000","mapping_epoch":192,"log_start":"199'500","ondisk_log_start":"199'500","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'8650","last_scrub_stamp":"2024-04-23T16:23:27.616985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:27.616985+0000","objects_scrubbed":8593,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:50.139450+0000","scrub_duration":1550,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":681377816,"num_objects":10398,"num_object_clones":0,"num_object_copies":20796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10398,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10562,"num_write_kb":670657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"227'10696","reported_seq":10862,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-23T16:23:57.690695+0000","last_change":"2024-04-23T16:23:18.692178+0000","last_active":"2024-04-23T16:23:57.690695+0000","last_peered":"2024-04-23T16:23:57.690695+0000","last_clean":"2024-04-23T16:23:57.690695+0000","last_became_active":"2024-04-23T16:22:11.616487+0000","last_became_peered":"2024-04-23T16:22:11.616487+0000","last_unstale":"2024-04-23T16:23:57.690695+0000","last_undegraded":"2024-04-23T16:23:57.690695+0000","last_fullsized":"2024-04-23T16:23:57.690695+0000","mapping_epoch":192,"log_start":"199'600","ondisk_log_start":"199'600","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'7886","last_scrub_stamp":"2024-04-23T16:23:18.691923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:18.691923+0000","objects_scrubbed":7817,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:40.335633+0000","scrub_duration":1596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":690749440,"num_objects":10540,"num_object_clones":0,"num_object_copies":21080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10696,"num_write_kb":679552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"227'10548","reported_seq":10712,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-23T16:23:55.731836+0000","last_change":"2024-04-23T16:23:15.831556+0000","last_active":"2024-04-23T16:23:55.731836+0000","last_peered":"2024-04-23T16:23:55.731836+0000","last_clean":"2024-04-23T16:23:55.731836+0000","last_became_active":"2024-04-23T16:22:11.615324+0000","last_became_peered":"2024-04-23T16:22:11.615324+0000","last_unstale":"2024-04-23T16:23:55.731836+0000","last_undegraded":"2024-04-23T16:23:55.731836+0000","last_fullsized":"2024-04-23T16:23:55.731836+0000","mapping_epoch":192,"log_start":"199'500","ondisk_log_start":"199'500","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7648","last_scrub_stamp":"2024-04-23T16:23:15.831380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:15.831380+0000","objects_scrubbed":7565,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:16.038149+0000","scrub_duration":1721,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":691273728,"num_objects":10548,"num_object_clones":0,"num_object_copies":21096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10548,"num_write_kb":675072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"228'10689","reported_seq":10795,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-23T16:23:58.983404+0000","last_change":"2024-04-23T16:23:33.410487+0000","last_active":"2024-04-23T16:23:58.983404+0000","last_peered":"2024-04-23T16:23:58.983404+0000","last_clean":"2024-04-23T16:23:58.983404+0000","last_became_active":"2024-04-23T16:22:11.616590+0000","last_became_peered":"2024-04-23T16:22:11.616590+0000","last_unstale":"2024-04-23T16:23:58.983404+0000","last_undegraded":"2024-04-23T16:23:58.983404+0000","last_fullsized":"2024-04-23T16:23:58.983404+0000","mapping_epoch":192,"log_start":"199'600","ondisk_log_start":"199'600","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:22:10.601504+0000","objects_scrubbed":2199,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:02.138402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":671547392,"num_objects":10247,"num_object_clones":0,"num_object_copies":20494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10689,"num_write_kb":669952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"228'10669","reported_seq":10831,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-23T16:23:58.126156+0000","last_change":"2024-04-23T16:23:14.734017+0000","last_active":"2024-04-23T16:23:58.126156+0000","last_peered":"2024-04-23T16:23:58.126156+0000","last_clean":"2024-04-23T16:23:58.126156+0000","last_became_active":"2024-04-23T16:22:11.618760+0000","last_became_peered":"2024-04-23T16:22:11.618760+0000","last_unstale":"2024-04-23T16:23:58.126156+0000","last_undegraded":"2024-04-23T16:23:58.126156+0000","last_fullsized":"2024-04-23T16:23:58.126156+0000","mapping_epoch":192,"log_start":"199'600","ondisk_log_start":"199'600","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7491","last_scrub_stamp":"2024-04-23T16:23:14.733801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:14.733801+0000","objects_scrubbed":7430,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:38.345896+0000","scrub_duration":1267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":682819584,"num_objects":10419,"num_object_clones":0,"num_object_copies":20838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10669,"num_write_kb":674816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"227'10666","reported_seq":10839,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-23T16:23:56.148048+0000","last_change":"2024-04-23T16:23:26.020510+0000","last_active":"2024-04-23T16:23:56.148048+0000","last_peered":"2024-04-23T16:23:56.148048+0000","last_clean":"2024-04-23T16:23:56.148048+0000","last_became_active":"2024-04-23T16:22:11.622030+0000","last_became_peered":"2024-04-23T16:22:11.622030+0000","last_unstale":"2024-04-23T16:23:56.148048+0000","last_undegraded":"2024-04-23T16:23:56.148048+0000","last_fullsized":"2024-04-23T16:23:56.148048+0000","mapping_epoch":192,"log_start":"199'600","ondisk_log_start":"199'600","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'8646","last_scrub_stamp":"2024-04-23T16:23:26.020368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:26.020368+0000","objects_scrubbed":8556,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:52.131906+0000","scrub_duration":1878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":699006976,"num_objects":10666,"num_object_clones":0,"num_object_copies":21332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10666,"num_write_kb":682624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"227'10606","reported_seq":10770,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-23T16:23:55.735680+0000","last_change":"2024-04-23T16:23:16.404142+0000","last_active":"2024-04-23T16:23:55.735680+0000","last_peered":"2024-04-23T16:23:55.735680+0000","last_clean":"2024-04-23T16:23:55.735680+0000","last_became_active":"2024-04-23T16:22:11.619141+0000","last_became_peered":"2024-04-23T16:22:11.619141+0000","last_unstale":"2024-04-23T16:23:55.735680+0000","last_undegraded":"2024-04-23T16:23:55.735680+0000","last_fullsized":"2024-04-23T16:23:55.735680+0000","mapping_epoch":192,"log_start":"199'600","ondisk_log_start":"199'600","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7744","last_scrub_stamp":"2024-04-23T16:23:16.403981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:16.403981+0000","objects_scrubbed":7705,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:24.288145+0000","scrub_duration":1262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":695074816,"num_objects":10606,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10606,"num_write_kb":678784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"228'10666","reported_seq":10840,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-23T16:23:59.031061+0000","last_change":"2024-04-23T16:23:28.605101+0000","last_active":"2024-04-23T16:23:59.031061+0000","last_peered":"2024-04-23T16:23:59.031061+0000","last_clean":"2024-04-23T16:23:59.031061+0000","last_became_active":"2024-04-23T16:22:11.615706+0000","last_became_peered":"2024-04-23T16:22:11.615706+0000","last_unstale":"2024-04-23T16:23:59.031061+0000","last_undegraded":"2024-04-23T16:23:59.031061+0000","last_fullsized":"2024-04-23T16:23:59.031061+0000","mapping_epoch":192,"log_start":"199'600","ondisk_log_start":"199'600","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'8608","last_scrub_stamp":"2024-04-23T16:23:28.604991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:28.604991+0000","objects_scrubbed":8566,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:42.709540+0000","scrub_duration":1382,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":671612928,"num_objects":10248,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10666,"num_write_kb":669248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"227'10566","reported_seq":10739,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-23T16:23:57.698321+0000","last_change":"2024-04-23T16:23:30.737830+0000","last_active":"2024-04-23T16:23:57.698321+0000","last_peered":"2024-04-23T16:23:57.698321+0000","last_clean":"2024-04-23T16:23:57.698321+0000","last_became_active":"2024-04-23T16:22:11.616683+0000","last_became_peered":"2024-04-23T16:22:11.616683+0000","last_unstale":"2024-04-23T16:23:57.698321+0000","last_undegraded":"2024-04-23T16:23:57.698321+0000","last_fullsized":"2024-04-23T16:23:57.698321+0000","mapping_epoch":192,"log_start":"199'500","ondisk_log_start":"199'500","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'8829","last_scrub_stamp":"2024-04-23T16:23:30.737605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:30.737605+0000","objects_scrubbed":8773,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:43.515529+0000","scrub_duration":1732,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":679477248,"num_objects":10368,"num_object_clones":0,"num_object_copies":20736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10566,"num_write_kb":669888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"227'12241","reported_seq":12426,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-23T16:23:56.643513+0000","last_change":"2024-04-23T16:23:42.767527+0000","last_active":"2024-04-23T16:23:56.643513+0000","last_peered":"2024-04-23T16:23:56.643513+0000","last_clean":"2024-04-23T16:23:56.643513+0000","last_became_active":"2024-04-23T16:22:11.619030+0000","last_became_peered":"2024-04-23T16:22:11.619030+0000","last_unstale":"2024-04-23T16:23:56.643513+0000","last_undegraded":"2024-04-23T16:23:56.643513+0000","last_fullsized":"2024-04-23T16:23:56.643513+0000","mapping_epoch":192,"log_start":"200'2200","ondisk_log_start":"200'2200","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:22:10.601504+0000","objects_scrubbed":2536,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:01.197649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":677904384,"num_objects":10344,"num_object_clones":0,"num_object_copies":20688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10368,"num_write_kb":662784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"228'10690","reported_seq":10867,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-23T16:23:59.017696+0000","last_change":"2024-04-23T16:23:31.078016+0000","last_active":"2024-04-23T16:23:59.017696+0000","last_peered":"2024-04-23T16:23:59.017696+0000","last_clean":"2024-04-23T16:23:59.017696+0000","last_became_active":"2024-04-23T16:22:11.618441+0000","last_became_peered":"2024-04-23T16:22:11.618441+0000","last_unstale":"2024-04-23T16:23:59.017696+0000","last_undegraded":"2024-04-23T16:23:59.017696+0000","last_fullsized":"2024-04-23T16:23:59.017696+0000","mapping_epoch":192,"log_start":"199'600","ondisk_log_start":"199'600","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'8843","last_scrub_stamp":"2024-04-23T16:23:31.077889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:23:31.077889+0000","objects_scrubbed":8806,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:40.425971+0000","scrub_duration":1453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":675151872,"num_objects":10302,"num_object_clones":0,"num_object_copies":20604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10690,"num_write_kb":671744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"228'10674","reported_seq":10830,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-23T16:23:59.016007+0000","last_change":"2024-04-23T16:23:33.420726+0000","last_active":"2024-04-23T16:23:59.016007+0000","last_peered":"2024-04-23T16:23:59.016007+0000","last_clean":"2024-04-23T16:23:59.016007+0000","last_became_active":"2024-04-23T16:22:11.616598+0000","last_became_peered":"2024-04-23T16:22:11.616598+0000","last_unstale":"2024-04-23T16:23:59.016007+0000","last_undegraded":"2024-04-23T16:23:59.016007+0000","last_fullsized":"2024-04-23T16:23:59.016007+0000","mapping_epoch":192,"log_start":"199'600","ondisk_log_start":"199'600","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:22:10.601504+0000","objects_scrubbed":7349,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:23:59.142569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672923648,"num_objects":10268,"num_object_clones":0,"num_object_copies":20536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10674,"num_write_kb":670144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"227'12529","reported_seq":12720,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-23T16:23:56.129124+0000","last_change":"2024-04-23T16:23:42.673174+0000","last_active":"2024-04-23T16:23:56.129124+0000","last_peered":"2024-04-23T16:23:56.129124+0000","last_clean":"2024-04-23T16:23:56.129124+0000","last_became_active":"2024-04-23T16:22:11.664655+0000","last_became_peered":"2024-04-23T16:22:11.664655+0000","last_unstale":"2024-04-23T16:23:56.129124+0000","last_undegraded":"2024-04-23T16:23:56.129124+0000","last_fullsized":"2024-04-23T16:23:56.129124+0000","mapping_epoch":192,"log_start":"200'2500","ondisk_log_start":"200'2500","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:22:10.601504+0000","objects_scrubbed":2686,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:00.205654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":693698560,"num_objects":10585,"num_object_clones":0,"num_object_copies":21170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10585,"num_write_kb":677440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"227'10538","reported_seq":10617,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-23T16:23:55.690748+0000","last_change":"2024-04-23T16:22:11.664979+0000","last_active":"2024-04-23T16:23:55.690748+0000","last_peered":"2024-04-23T16:23:55.690748+0000","last_clean":"2024-04-23T16:23:55.690748+0000","last_became_active":"2024-04-23T16:22:11.664765+0000","last_became_peered":"2024-04-23T16:22:11.664765+0000","last_unstale":"2024-04-23T16:23:55.690748+0000","last_undegraded":"2024-04-23T16:23:55.690748+0000","last_fullsized":"2024-04-23T16:23:55.690748+0000","mapping_epoch":192,"log_start":"199'500","ondisk_log_start":"199'500","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:22:10.601504+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":690618368,"num_objects":10538,"num_object_clones":0,"num_object_copies":21076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10538,"num_write_kb":674432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'137","reported_seq":638,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-23T16:23:58.019827+0000","last_change":"2024-04-23T16:23:16.763496+0000","last_active":"2024-04-23T16:23:58.019827+0000","last_peered":"2024-04-23T16:23:58.019827+0000","last_clean":"2024-04-23T16:23:58.019827+0000","last_became_active":"2024-04-23T16:15:36.918552+0000","last_became_peered":"2024-04-23T16:15:36.918552+0000","last_unstale":"2024-04-23T16:23:58.019827+0000","last_undegraded":"2024-04-23T16:23:58.019827+0000","last_fullsized":"2024-04-23T16:23:58.019827+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"16'137","last_scrub_stamp":"2024-04-23T16:23:16.763345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:11:23.401899+0000","last_clean_scrub_stamp":"2024-04-23T16:23:16.763345+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:41.299861+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":10939531288,"num_objects":166925,"num_object_clones":0,"num_object_copies":333850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166925,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169905,"num_write_kb":10778497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21869699072,"data_stored":21869690928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161022,"ondisk_log_size":161022,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056743,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4918860,"kb_used_data":4693312,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":88826292,"statfs":{"total":95995035648,"available":90958123008,"internally_reserved":0,"allocated":4805951488,"data_stored":4805593411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":230947268},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351086,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612724,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1490448,"kb_used_data":1337092,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":92254704,"statfs":{"total":95995035648,"available":94468816896,"internally_reserved":0,"allocated":1369182208,"data_stored":1368821649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":157022656},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":800836,"kb_used_data":671096,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92944316,"statfs":{"total":95995035648,"available":95174979584,"internally_reserved":0,"allocated":687202304,"data_stored":686847799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168701,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1448344,"kb_used_data":1318608,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92296808,"statfs":{"total":95995035648,"available":94511931392,"internally_reserved":0,"allocated":1350254592,"data_stored":1349886409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":132839882},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463043,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":725136,"kb_used_data":662020,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93020016,"statfs":{"total":95995035648,"available":95252496384,"internally_reserved":0,"allocated":677908480,"data_stored":677547921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575003,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2187588,"kb_used_data":2012024,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91557564,"statfs":{"total":95995035648,"available":93754945536,"internally_reserved":0,"allocated":2060312576,"data_stored":2059958071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738528,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2777232,"kb_used_data":2637636,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":90967920,"statfs":{"total":95995035648,"available":93151150080,"internally_reserved":0,"allocated":2700939264,"data_stored":2700578705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313365,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":130452,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93614700,"statfs":{"total":95995035648,"available":95861452800,"internally_reserved":0,"allocated":724992,"data_stored":364433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117136,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3521860,"kb_used_data":3360184,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":90223292,"statfs":{"total":95995035648,"available":92388651008,"internally_reserved":0,"allocated":3440828416,"data_stored":3440473911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":165542343},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920907,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2736272,"kb_used_data":2637060,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91008880,"statfs":{"total":95995035648,"available":93193093120,"internally_reserved":0,"allocated":2700349440,"data_stored":2699988881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757382,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2140292,"kb_used_data":2010556,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":91604860,"statfs":{"total":95995035648,"available":93803376640,"internally_reserved":0,"allocated":2058809344,"data_stored":2058450767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2700017664,"data_stored":2700017664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2062548992,"data_stored":2062548992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3452305408,"data_stored":3452305408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2695495680,"data_stored":2695495680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2051280896,"data_stored":2051276824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":671547392,"data_stored":671547392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1348866048,"data_stored":1348861976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":682819584,"data_stored":682819584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1366097920,"data_stored":1366097920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4838719488,"data_stored":4838719488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:24:02.313 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-23T16:24:02.394 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:24:02.425 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:02.449 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:24:02.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:02.452 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:24:02.553 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:24:02.591 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:02.630 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:24:02.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:02.637 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:24:02.689 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:24:02.738 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:24:02.774 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:02.959 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:02 smithi148 ceph-mon[44520]: from='client.15243 v1:172.21.15.103:0/808856726' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:24:02.959 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:02 smithi148 ceph-mon[44520]: 3.2 scrub starts 2024-04-23T16:24:02.959 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.959 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.959 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.959 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.959 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.959 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.959 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.959 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.960 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:02 smithi188 ceph-mon[44101]: from='client.15243 v1:172.21.15.103:0/808856726' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:24:02.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:02 smithi188 ceph-mon[44101]: 3.2 scrub starts 2024-04-23T16:24:02.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.976 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:24:02.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:02.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:24:02.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:02 smithi103 ceph-mon[39629]: from='client.15243 v1:172.21.15.103:0/808856726' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:24:02.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:02 smithi103 ceph-mon[39629]: 3.2 scrub starts 2024-04-23T16:24:02.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:02.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:03.081 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:24:03.120 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:03.129 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:24:03.195 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:24:03.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:03.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:24:03.299 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:24:03.359 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:03.447 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:24:03.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:03.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:24:03.551 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:24:03.610 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:03.770 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:24:03.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:03.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:24:03.873 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:24:03.903 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:03 smithi103 ceph-mon[39629]: pgmap v662: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 10 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 127 B/s rd, 40 MiB/s wr, 1.15k op/s 2024-04-23T16:24:03.903 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:03 smithi103 ceph-mon[39629]: 3.0 scrub starts 2024-04-23T16:24:03.903 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:03 smithi103 ceph-mon[39629]: 3.4 scrub ok 2024-04-23T16:24:03.904 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:03 smithi103 ceph-mon[39629]: 3.1 scrub ok 2024-04-23T16:24:03.906 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:03.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:03 smithi188 ceph-mon[44101]: pgmap v662: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 10 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 127 B/s rd, 40 MiB/s wr, 1.15k op/s 2024-04-23T16:24:03.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:03 smithi188 ceph-mon[44101]: 3.0 scrub starts 2024-04-23T16:24:03.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:03 smithi188 ceph-mon[44101]: 3.4 scrub ok 2024-04-23T16:24:03.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:03 smithi188 ceph-mon[44101]: 3.1 scrub ok 2024-04-23T16:24:04.065 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:03 smithi148 ceph-mon[44520]: pgmap v662: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 10 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 127 B/s rd, 40 MiB/s wr, 1.15k op/s 2024-04-23T16:24:04.065 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:03 smithi148 ceph-mon[44520]: 3.0 scrub starts 2024-04-23T16:24:04.065 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:03 smithi148 ceph-mon[44520]: 3.4 scrub ok 2024-04-23T16:24:04.065 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:03 smithi148 ceph-mon[44520]: 3.1 scrub ok 2024-04-23T16:24:04.165 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:24:04.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:04.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:24:04.269 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:24:04.301 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:04.328 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:24:04.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:04.332 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:24:04.433 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:24:04.473 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:04.504 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:24:04.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:04.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:24:04.612 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:24:04.651 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:04.675 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:24:04.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:04.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:24:04.693 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:04 smithi148 ceph-mon[44520]: 3.2 scrub ok 2024-04-23T16:24:04.743 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:24:04.743 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:24:04.779 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:24:04.829 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:04.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:04 smithi188 ceph-mon[44101]: 3.2 scrub ok 2024-04-23T16:24:05.014 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:24:05.047 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:04 smithi103 ceph-mon[39629]: 3.2 scrub ok 2024-04-23T16:24:05.479 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-23T16:24:05.485 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:24:05.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:05.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:24:05.535 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:24:05.589 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:24:05.631 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:05.661 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:24:05.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:05.664 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:24:05.709 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:05 smithi148 ceph-mon[44520]: pgmap v663: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 106 B/s rd, 31 MiB/s wr, 1.21k op/s 2024-04-23T16:24:05.709 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:05 smithi148 ceph-mon[44520]: 3.0 scrub ok 2024-04-23T16:24:05.765 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:24:05.806 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:05.834 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:24:05.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:05.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:24:05.851 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:24:05.949 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:24:05.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:05 smithi188 ceph-mon[44101]: pgmap v663: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 106 B/s rd, 31 MiB/s wr, 1.21k op/s 2024-04-23T16:24:05.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:05 smithi188 ceph-mon[44101]: 3.0 scrub ok 2024-04-23T16:24:05.996 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:06.017 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:24:06.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:06.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:24:06.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:05 smithi103 ceph-mon[39629]: pgmap v663: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 106 B/s rd, 31 MiB/s wr, 1.21k op/s 2024-04-23T16:24:06.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:05 smithi103 ceph-mon[39629]: 3.0 scrub ok 2024-04-23T16:24:06.121 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:24:06.153 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:06.213 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:24:06.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:06.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:24:06.319 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:06.347 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:06.372 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:06.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:06.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:06.480 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:24:06.522 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:06.734 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:24:06.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:06 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1599799986' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:24:06.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:06 smithi103 ceph-mon[39629]: from='client.35201 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:24:06.875 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:24:06.877 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:06 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1599799986' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:24:06.877 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:06 smithi188 ceph-mon[44101]: from='client.35201 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:24:06.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:06.878 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:24:06.979 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:24:07.021 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:07.047 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:24:07.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:07.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:24:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:06 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1599799986' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:24:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:06 smithi148 ceph-mon[44520]: from='client.35201 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:24:07.151 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:24:07.202 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:07.305 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:24:07.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:07.322 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:24:07.423 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:07.472 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:07.505 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:07.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:07.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:07.609 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:24:07.644 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:07.724 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:07 smithi103 ceph-mon[39629]: pgmap v664: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 102 B/s rd, 16 MiB/s wr, 1.18k op/s 2024-04-23T16:24:07.724 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:07 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:24:07.724 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:07 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:24:07.724 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:07 smithi103 ceph-mon[39629]: from='client.35201 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:24:07.724 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:07 smithi103 ceph-mon[39629]: osdmap e229: 12 total, 12 up, 11 in 2024-04-23T16:24:07.724 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:07 smithi103 ceph-mon[39629]: 3.a scrub starts 2024-04-23T16:24:07.724 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:24:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:07 smithi148 ceph-mon[44520]: pgmap v664: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 102 B/s rd, 16 MiB/s wr, 1.18k op/s 2024-04-23T16:24:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:07 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:24:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:07 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:24:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:07 smithi148 ceph-mon[44520]: from='client.35201 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:24:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:07 smithi148 ceph-mon[44520]: osdmap e229: 12 total, 12 up, 11 in 2024-04-23T16:24:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:07 smithi148 ceph-mon[44520]: 3.a scrub starts 2024-04-23T16:24:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:07 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:24:07.898 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:24:07.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:07.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:24:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:07 smithi188 ceph-mon[44101]: pgmap v664: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 102 B/s rd, 16 MiB/s wr, 1.18k op/s 2024-04-23T16:24:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:07 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:24:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:07 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:24:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:07 smithi188 ceph-mon[44101]: from='client.35201 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:24:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:07 smithi188 ceph-mon[44101]: osdmap e229: 12 total, 12 up, 11 in 2024-04-23T16:24:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:07 smithi188 ceph-mon[44101]: 3.a scrub starts 2024-04-23T16:24:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:24:07.990 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:24:07.990 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:24:08.013 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:24:08.093 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:08.120 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:24:08.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:08.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:24:08.226 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:24:08.254 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:08.536 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:24:08.559 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:24:08.581 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:24:08.610 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:24:08.639 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:24:08.667 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:24:08.701 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:24:08.735 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:24:08.763 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:24:08.801 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:24:08.832 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:24:08.861 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:24:08.869 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:24:08.869 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:24:08.869 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:24:08.869 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:24:08.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:24:08.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:24:08.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:24:08.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:24:08.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:24:08.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:24:08.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:24:08.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:24:08.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:08 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2718496963' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:09.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:08 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2718496963' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:09.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:08 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2718496963' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:09.703 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:24:09.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:09.708 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:24:09.809 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:24:09.837 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:09.862 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:24:09.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:09.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:24:09.909 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:24:09.968 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:24:09.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:09 smithi188 ceph-mon[44101]: pgmap v666: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 6.7 MiB/s wr, 1.18k op/s 2024-04-23T16:24:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:09 smithi188 ceph-mon[44101]: 3.a scrub ok 2024-04-23T16:24:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:09 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:24:09.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:09 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:24:09.995 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.995 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.995 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.995 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:09 smithi148 ceph-mon[44520]: pgmap v666: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 6.7 MiB/s wr, 1.18k op/s 2024-04-23T16:24:09.995 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.995 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:09 smithi148 ceph-mon[44520]: 3.a scrub ok 2024-04-23T16:24:09.995 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.995 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.996 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.996 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.996 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.996 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.996 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.996 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:09.996 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:09 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:24:09.996 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:09 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:24:09.997 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:10.017 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:24:10.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:10.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:24:10.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:10.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:10.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:10.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:09 smithi103 ceph-mon[39629]: pgmap v666: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 6.7 MiB/s wr, 1.18k op/s 2024-04-23T16:24:10.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:10.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:09 smithi103 ceph-mon[39629]: 3.a scrub ok 2024-04-23T16:24:10.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:10.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:10.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:10.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:10.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:10.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:10.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:10.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:10.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:09 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:24:10.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:09 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:24:10.121 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:24:10.148 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:10.306 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:24:10.912 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:10 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:24:10.912 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:10 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:24:10.912 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:10 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:24:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:10 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:24:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:10 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:24:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:10 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:24:11.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:10 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:24:11.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:10 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:24:11.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:10 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:24:11.828 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:24:12.057 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:11 smithi103 ceph-mon[39629]: pgmap v667: 17 pgs: 17 active+clean; 9.5 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 1.4 MiB/s wr, 1.17k op/s 2024-04-23T16:24:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:11 smithi148 ceph-mon[44520]: pgmap v667: 17 pgs: 17 active+clean; 9.5 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 1.4 MiB/s wr, 1.17k op/s 2024-04-23T16:24:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:11 smithi188 ceph-mon[44101]: pgmap v667: 17 pgs: 17 active+clean; 9.5 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 1.4 MiB/s wr, 1.17k op/s 2024-04-23T16:24:12.409 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:24:12.605 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:24:12.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:12.609 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:24:12.624 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-23T16:24:12.710 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:24:12.757 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:12.782 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:24:12.786 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:24:12.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:12.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:24:12.888 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:24:12.934 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:12 smithi103 ceph-mon[39629]: 3.f scrub starts 2024-04-23T16:24:12.934 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:12 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:24:12.934 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:12 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:24:12.934 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:12 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1219521961' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:24:12.934 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:12 smithi103 ceph-mon[39629]: from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:24:12.935 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:12.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:12 smithi188 ceph-mon[44101]: 3.f scrub starts 2024-04-23T16:24:12.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:12 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:24:12.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:12 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:24:12.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:12 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1219521961' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:24:12.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:12 smithi188 ceph-mon[44101]: from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:24:13.029 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:24:13.054 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:24:13.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:13.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:24:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:12 smithi148 ceph-mon[44520]: 3.f scrub starts 2024-04-23T16:24:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:12 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:24:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:12 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:24:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:12 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1219521961' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:24:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:12 smithi148 ceph-mon[44520]: from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:24:13.162 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:24:13.203 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:13.676 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:24:13.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:13.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:24:13.780 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:24:13.823 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:13.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:13 smithi103 ceph-mon[39629]: pgmap v668: 17 pgs: 17 active+clean; 9.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:24:13.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:13 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:24:13.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:13 smithi103 ceph-mon[39629]: from='client.25153 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:24:13.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:13 smithi103 ceph-mon[39629]: osdmap e230: 12 total, 12 up, 11 in 2024-04-23T16:24:13.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:13 smithi188 ceph-mon[44101]: pgmap v668: 17 pgs: 17 active+clean; 9.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:24:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:13 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:24:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:13 smithi188 ceph-mon[44101]: from='client.25153 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:24:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:13 smithi188 ceph-mon[44101]: osdmap e230: 12 total, 12 up, 11 in 2024-04-23T16:24:14.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:13 smithi148 ceph-mon[44520]: pgmap v668: 17 pgs: 17 active+clean; 9.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:24:14.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:13 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:24:14.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:13 smithi148 ceph-mon[44520]: from='client.25153 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:24:14.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:13 smithi148 ceph-mon[44520]: osdmap e230: 12 total, 12 up, 11 in 2024-04-23T16:24:14.183 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:24:14.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:14.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:24:14.288 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:24:14.318 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:14.358 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:24:14.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:14.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:24:14.462 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:14.504 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:14.530 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:14.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:14.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:14.637 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:24:14.670 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:14.704 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:24:14.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:14.708 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:24:14.809 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:24:14.847 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:14.956 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:24:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:14.962 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:24:14.966 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:14 smithi103 ceph-mon[39629]: pgmap v670: 17 pgs: 17 active+clean; 9.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:24:15.063 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:24:15.112 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:15.125 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:24:15.126 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:24:15.126 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:24:15.126 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:24:15.126 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:24:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:14 smithi188 ceph-mon[44101]: pgmap v670: 17 pgs: 17 active+clean; 9.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:24:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:14 smithi148 ceph-mon[44520]: pgmap v670: 17 pgs: 17 active+clean; 9.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:24:15.628 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:24:15.680 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:24:15.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:15.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:24:15.710 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:24:15.750 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:24:15.787 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:15.791 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:24:15.817 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:15.851 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:24:15.858 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-23T16:24:15.859 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:15.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:15.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:15.883 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:24:15.919 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:24:15.950 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:24:15.962 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:15.974 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:15 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1404264805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:15.979 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:24:15.997 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:16.015 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:24:16.025 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:16.029 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:24:16.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:16.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:16.067 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:24:16.104 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:24:16.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:15 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1404264805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:16.129 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:24:16.132 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:24:16.132 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:24:16.133 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:24:16.133 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:24:16.133 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:24:16.133 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:24:16.133 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:24:16.133 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:24:16.133 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:24:16.133 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:24:16.133 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:24:16.133 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:24:16.133 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:24:16.166 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:16.194 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:24:16.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:16.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:24:16.215 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:24:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:15 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1404264805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:16.298 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:24:16.337 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:16.499 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:24:16.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:16.502 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:24:16.603 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:24:16.631 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:16.651 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:24:16.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:16.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:24:16.755 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:24:16.793 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:17.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:16 smithi103 ceph-mon[39629]: pgmap v671: 17 pgs: 17 active+clean; 9.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-23T16:24:17.145 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:24:17.147 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.148 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.148 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.148 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.148 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.148 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.148 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.149 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.149 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.149 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.149 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.149 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:16 smithi188 ceph-mon[44101]: pgmap v671: 17 pgs: 17 active+clean; 9.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-23T16:24:17.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:17.151 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:24:17.252 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:24:17.295 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:17.337 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:24:17.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:17.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:24:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:17.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:16 smithi148 ceph-mon[44520]: pgmap v671: 17 pgs: 17 active+clean; 9.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-23T16:24:17.441 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:24:17.474 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:17.604 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:24:17.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:17.608 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:24:17.709 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:24:17.744 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:17.770 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:24:17.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:17.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:24:17.879 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:24:17.914 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:17.943 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:24:17.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:17.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:24:18.049 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:24:18.095 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:18.096 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:17 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:24:18.096 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:17 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:24:18.096 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:18.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:18.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:17 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:24:18.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:17 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:24:18.174 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:24:18.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:18.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:24:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:17 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:24:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:17 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:24:18.277 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:24:18.277 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:24:18.282 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:24:18.336 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:18.597 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:24:18.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:18.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:24:18.702 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:24:18.733 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:18.981 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:24:18.984 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-23T16:24:18.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:18.988 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:24:19.089 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:24:19.124 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:19.149 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:24:19.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:19.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:24:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:19 smithi188 ceph-mon[44101]: osdmap e231: 12 total, 12 up, 11 in 2024-04-23T16:24:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:19 smithi188 ceph-mon[44101]: pgmap v673: 17 pgs: 17 active+clean; 8.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:24:19.253 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:19.280 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:19 smithi148 ceph-mon[44520]: osdmap e231: 12 total, 12 up, 11 in 2024-04-23T16:24:19.280 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:19 smithi148 ceph-mon[44520]: pgmap v673: 17 pgs: 17 active+clean; 8.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:24:19.281 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:19.307 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:19.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:19.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:19.339 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:24:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:19 smithi103 ceph-mon[39629]: osdmap e231: 12 total, 12 up, 11 in 2024-04-23T16:24:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:19 smithi103 ceph-mon[39629]: pgmap v673: 17 pgs: 17 active+clean; 8.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:24:19.364 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:24:19.417 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:19.449 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:19.477 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:19.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:19.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:19.582 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:24:19.631 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:19.665 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:24:19.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:19.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:24:19.735 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:24:19.769 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:24:19.805 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:19.907 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:24:19.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:19.912 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:24:20.013 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:20.032 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:24:20.055 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:20.080 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:20.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:20.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:20.183 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:24:20.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:20 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3708405581' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:24:20.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:20 smithi188 ceph-mon[44101]: from='client.35216 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:24:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:20 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:24:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:20 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:24:20.237 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:20.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:20 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3708405581' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:24:20.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:20 smithi103 ceph-mon[39629]: from='client.35216 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:24:20.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:20 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:24:20.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:20 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:24:20.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:20 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3708405581' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:24:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:20 smithi148 ceph-mon[44520]: from='client.35216 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:24:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:20 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:24:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:20 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:24:21.111 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:21 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:24:21.111 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:21 smithi188 ceph-mon[44101]: from='client.35216 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:24:21.112 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:21 smithi188 ceph-mon[44101]: osdmap e232: 12 total, 12 up, 11 in 2024-04-23T16:24:21.112 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:21 smithi188 ceph-mon[44101]: pgmap v675: 17 pgs: 17 active+clean; 8.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-23T16:24:21.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:21 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:24:21.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:21 smithi103 ceph-mon[39629]: from='client.35216 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:24:21.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:21 smithi103 ceph-mon[39629]: osdmap e232: 12 total, 12 up, 11 in 2024-04-23T16:24:21.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:21 smithi103 ceph-mon[39629]: pgmap v675: 17 pgs: 17 active+clean; 8.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-23T16:24:21.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:21 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:24:21.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:21 smithi148 ceph-mon[44520]: from='client.35216 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:24:21.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:21 smithi148 ceph-mon[44520]: osdmap e232: 12 total, 12 up, 11 in 2024-04-23T16:24:21.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:21 smithi148 ceph-mon[44520]: pgmap v675: 17 pgs: 17 active+clean; 8.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-23T16:24:21.439 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:24:21.439 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:24:22.174 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-23T16:24:22.314 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-04-23T16:24:22.314 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-23T16:24:22.314 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-23T16:24:22.314 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.1 2024-04-23T16:24:22.518 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:24:22.850 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:24:22 smithi103 systemd[1]: Stopping Ceph osd.1 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T16:24:23.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:23 smithi188 ceph-mon[44101]: pgmap v676: 17 pgs: 17 active+clean; 8.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-23T16:24:24.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:23 smithi103 ceph-mon[39629]: pgmap v676: 17 pgs: 17 active+clean; 8.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-23T16:24:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:23 smithi148 ceph-mon[44520]: pgmap v676: 17 pgs: 17 active+clean; 8.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-23T16:24:24.588 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:24:24.589 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:24:24.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:24 smithi188 ceph-mon[44101]: pgmap v677: 17 pgs: 17 active+clean; 8.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-23T16:24:25.339 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-23T16:24:25.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:24 smithi103 ceph-mon[39629]: pgmap v677: 17 pgs: 17 active+clean; 8.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-23T16:24:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:24 smithi148 ceph-mon[44520]: pgmap v677: 17 pgs: 17 active+clean; 8.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-23T16:24:25.693 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:24:27.360 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:24:27.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:27.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:24:27.464 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:24:27.517 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:27.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:27 smithi188 ceph-mon[44101]: pgmap v678: 17 pgs: 17 active+clean; 8.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-23T16:24:27.771 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:24:27.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:27.775 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:24:27.776 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:24:27.776 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:24:27.876 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:27.916 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:27.918 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:27 smithi148 ceph-mon[44520]: pgmap v678: 17 pgs: 17 active+clean; 8.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-23T16:24:27.944 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:27.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:27.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:28.050 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:24:28.085 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:28.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:27 smithi103 ceph-mon[39629]: pgmap v678: 17 pgs: 17 active+clean; 8.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-23T16:24:28.116 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:24:28.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:28.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:24:28.220 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:24:28.264 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:28.517 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-23T16:24:28.897 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:24:28.934 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:24:28.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:28.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:24:29.042 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:24:29.074 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:29.098 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:24:29.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:29.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:24:29.205 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:24:29.234 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:24:29.235 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:29.261 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:24:29.262 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:24:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:29.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:24:29.287 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:24:29.313 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:24:29.348 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:24:29.369 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:29.373 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:24:29.400 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:24:29.402 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:29.425 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:29.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:29.429 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:24:29.463 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:24:29.497 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:24:29.530 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:24:29.543 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:24:29.585 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:24:29.589 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:24:29 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-1[65562]: 2024-04-23T16:24:29.281+0000 7fa722e5c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T16:24:29.589 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:24:29 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-1[65562]: 2024-04-23T16:24:29.281+0000 7fa722e5c700 -1 osd.1 232 *** Got signal Terminated *** 2024-04-23T16:24:29.589 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:24:29 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-1[65562]: 2024-04-23T16:24:29.281+0000 7fa722e5c700 -1 osd.1 232 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T16:24:29.591 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:29.593 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:24:29.593 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:24:29.593 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:24:29.593 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:24:29.593 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:24:29.593 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:24:29.593 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:24:29.593 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:24:29.593 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:24:29.593 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:24:29.593 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:24:29.593 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:24:29.627 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:24:29.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:29.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:24:29.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:29 smithi188 ceph-mon[44101]: pgmap v679: 17 pgs: 17 active+clean; 8.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:24:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:29 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/736865331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:29 smithi188 ceph-mon[44101]: osd.1 marked itself down and dead 2024-04-23T16:24:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:29.733 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:24:29.759 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:29 smithi103 ceph-mon[39629]: pgmap v679: 17 pgs: 17 active+clean; 8.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:24:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:29 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/736865331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:29 smithi103 ceph-mon[39629]: osd.1 marked itself down and dead 2024-04-23T16:24:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:29.856 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:24:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:29.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:24:29.961 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:24:29.989 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:29 smithi148 ceph-mon[44520]: pgmap v679: 17 pgs: 17 active+clean; 8.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:24:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:29 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/736865331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:29 smithi148 ceph-mon[44520]: osd.1 marked itself down and dead 2024-04-23T16:24:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:30.238 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:24:30.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:30.241 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:24:30.342 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:24:30.370 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:30.588 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:24:30.719 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:24:30.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:30.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:24:30.773 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:24:30.773 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:24:30.773 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-04-23T16:24:30.773 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:24:30.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:30.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:30.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:30.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:30.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:30.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:30.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:30.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:30.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:30.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:30 smithi188 ceph-mon[44101]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T16:24:30.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:30 smithi188 ceph-mon[44101]: osdmap e233: 12 total, 11 up, 11 in 2024-04-23T16:24:30.836 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:24:30.862 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:30.881 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:24:30.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:30.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:24:30.950 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:24:30.985 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:24:31.015 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:31.015 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:31.015 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:31.015 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:31.015 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:31.015 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:31.015 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:31.015 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:31.015 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:31.015 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:30 smithi148 ceph-mon[44520]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T16:24:31.015 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:30 smithi148 ceph-mon[44520]: osdmap e233: 12 total, 11 up, 11 in 2024-04-23T16:24:31.017 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:31.038 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:24:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:31.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:24:31.100 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:24:30 smithi103 bash[134545]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-1 2024-04-23T16:24:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:30 smithi103 ceph-mon[39629]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T16:24:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:30 smithi103 ceph-mon[39629]: osdmap e233: 12 total, 11 up, 11 in 2024-04-23T16:24:31.142 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:31.174 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:31.192 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:31.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:31.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:31.296 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:24:31.324 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:31.422 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:24:31.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:31.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:24:31.466 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-23T16:24:31.525 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:24:31.552 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:31.572 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:24:31.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:31.575 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:24:31.676 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:24:31.702 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:31 smithi188 ceph-mon[44101]: pgmap v681: 17 pgs: 1 stale+active+clean, 16 active+clean; 8.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:24:31.703 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:31 smithi188 ceph-mon[44101]: osdmap e234: 12 total, 11 up, 11 in 2024-04-23T16:24:31.705 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:31.725 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:24:31.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:31.728 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:24:31.737 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:24:31.830 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:31.862 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:31.883 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:31.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:31.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:31.986 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:24:32.014 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:31 smithi148 ceph-mon[44520]: pgmap v681: 17 pgs: 1 stale+active+clean, 16 active+clean; 8.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:24:32.014 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:31 smithi148 ceph-mon[44520]: osdmap e234: 12 total, 11 up, 11 in 2024-04-23T16:24:32.016 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:32.035 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:24:32.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:32.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:24:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:31 smithi103 ceph-mon[39629]: pgmap v681: 17 pgs: 1 stale+active+clean, 16 active+clean; 8.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:24:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:31 smithi103 ceph-mon[39629]: osdmap e234: 12 total, 11 up, 11 in 2024-04-23T16:24:32.139 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:32.168 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:32.187 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:32.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:32.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:32.291 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:24:32.319 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:32.476 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:24:32.518 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:24:32.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:32.520 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:24:32.621 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:24:32.649 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:32 smithi103 ceph-mon[39629]: Health check failed: Degraded data redundancy: 16047/258174 objects degraded (6.216%), 2 pgs degraded (PG_DEGRADED) 2024-04-23T16:24:32.910 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:24:32.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:32.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:24:32.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:32 smithi188 ceph-mon[44101]: Health check failed: Degraded data redundancy: 16047/258174 objects degraded (6.216%), 2 pgs degraded (PG_DEGRADED) 2024-04-23T16:24:33.013 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:24:33.043 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:32 smithi148 ceph-mon[44520]: Health check failed: Degraded data redundancy: 16047/258174 objects degraded (6.216%), 2 pgs degraded (PG_DEGRADED) 2024-04-23T16:24:33.045 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:33.067 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:24:33.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:33.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:24:33.170 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:24:33.188 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:24:33.201 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:33.554 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:24:33.555 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:24:33.555 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:24:33.555 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:24:33.555 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:24:33.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:33 smithi188 ceph-mon[44101]: pgmap v683: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s; 16047/258174 objects degraded (6.216%) 2024-04-23T16:24:33.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:33 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:24:33.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:33 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:24:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:33 smithi103 ceph-mon[39629]: pgmap v683: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s; 16047/258174 objects degraded (6.216%) 2024-04-23T16:24:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:33 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:24:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:33 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:24:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:33 smithi148 ceph-mon[44520]: pgmap v683: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s; 16047/258174 objects degraded (6.216%) 2024-04-23T16:24:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:33 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:24:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:33 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:24:34.245 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-23T16:24:34.251 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.1.service' 2024-04-23T16:24:34.388 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T16:24:34.389 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-23T16:24:34.389 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd down 1 2024-04-23T16:24:34.506 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:24:34.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:34 smithi103 ceph-mon[39629]: osdmap e235: 12 total, 11 up, 11 in 2024-04-23T16:24:34.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:34 smithi188 ceph-mon[44101]: osdmap e235: 12 total, 11 up, 11 in 2024-04-23T16:24:35.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:34 smithi148 ceph-mon[44520]: osdmap e235: 12 total, 11 up, 11 in 2024-04-23T16:24:35.187 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:24:35.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:35.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:24:35.294 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:24:35.323 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:35.420 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:24:35.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:35.431 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:24:35.532 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:24:35.561 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:35.581 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:24:35.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:35.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:24:35.684 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:35.716 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:35.737 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:35.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:35.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:35.819 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:35 smithi103 ceph-mon[39629]: pgmap v685: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 770 op/s; 24054/256634 objects degraded (9.373%) 2024-04-23T16:24:35.841 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:24:35.868 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:36.091 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:35 smithi188 ceph-mon[44101]: pgmap v685: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 770 op/s; 24054/256634 objects degraded (9.373%) 2024-04-23T16:24:36.117 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:24:36.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:36.121 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:24:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:35 smithi148 ceph-mon[44520]: pgmap v685: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 770 op/s; 24054/256634 objects degraded (9.373%) 2024-04-23T16:24:36.221 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:24:36.249 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:36.252 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:24:36.290 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:24:36.306 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:24:36.325 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:24:36.345 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:24:36.365 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:24:36.376 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:24:36.378 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:24:36.378 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-04-23T16:24:36.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-04-23T16:24:36.378 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:24:36.378 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.378 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:142532 3.8 3:19efe971:::benchmark_data_smithi188_89276_object142531:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e220)", 2024-04-23T16:24:36.379 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:31.810586+0000", 2024-04-23T16:24:36.379 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 64.564362942000002, 2024-04-23T16:24:36.379 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.11111224, 2024-04-23T16:24:36.379 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.379 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.379 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.379 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.379 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.379 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.380 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 142532 2024-04-23T16:24:36.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.380 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.810586+0000", 2024-04-23T16:24:36.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.381 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.810586+0000", 2024-04-23T16:24:36.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.381 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.381 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.381 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.810588+0000", 2024-04-23T16:24:36.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7940000000000001e-06 2024-04-23T16:24:36.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.381 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.381 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.381 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.810606+0000", 2024-04-23T16:24:36.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7756000000000001e-05 2024-04-23T16:24:36.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.382 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.810615+0000", 2024-04-23T16:24:36.382 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8780000000000004e-06 2024-04-23T16:24:36.382 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.382 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.810620+0000", 2024-04-23T16:24:36.382 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.215e-06 2024-04-23T16:24:36.382 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.383 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.830546+0000", 2024-04-23T16:24:36.383 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019926238999999998 2024-04-23T16:24:36.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.383 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.383 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.830670+0000", 2024-04-23T16:24:36.383 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012426000000000001 2024-04-23T16:24:36.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.384 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.467 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 1", 2024-04-23T16:24:36.467 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.830802+0000", 2024-04-23T16:24:36.467 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000131268 2024-04-23T16:24:36.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.468 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.468 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.468 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.844021+0000", 2024-04-23T16:24:36.468 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013219831 2024-04-23T16:24:36.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.468 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.468 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.468 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.921606+0000", 2024-04-23T16:24:36.468 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.077584901999999997 2024-04-23T16:24:36.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.468 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.469 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.469 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.921656+0000", 2024-04-23T16:24:36.469 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9246999999999998e-05 2024-04-23T16:24:36.469 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.469 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.469 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.469 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.921698+0000", 2024-04-23T16:24:36.469 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2849999999999998e-05 2024-04-23T16:24:36.469 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.469 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.469 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.470 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.470 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.470 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:142537 3.0 3:0a69a972:::benchmark_data_smithi188_89276_object142536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e220)", 2024-04-23T16:24:36.470 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:31.815412+0000", 2024-04-23T16:24:36.470 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 64.559536965999996, 2024-04-23T16:24:36.471 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.106437872, 2024-04-23T16:24:36.471 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.471 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.471 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.471 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.471 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.471 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.471 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 142537 2024-04-23T16:24:36.471 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.471 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.471 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.471 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.471 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.815412+0000", 2024-04-23T16:24:36.472 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.472 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.472 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.472 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.472 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.815412+0000", 2024-04-23T16:24:36.472 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.472 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.472 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.472 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.472 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.815413+0000", 2024-04-23T16:24:36.472 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8199999999999999e-07 2024-04-23T16:24:36.472 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.472 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.473 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.815431+0000", 2024-04-23T16:24:36.473 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8502e-05 2024-04-23T16:24:36.473 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.473 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.473 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.815438+0000", 2024-04-23T16:24:36.473 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6819999999999997e-06 2024-04-23T16:24:36.473 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.473 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.473 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.815445+0000", 2024-04-23T16:24:36.473 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.64e-06 2024-04-23T16:24:36.473 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.474 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.474 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.474 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.830666+0000", 2024-04-23T16:24:36.474 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015220971999999999 2024-04-23T16:24:36.474 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.474 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.474 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.474 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.830781+0000", 2024-04-23T16:24:36.474 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000115525 2024-04-23T16:24:36.474 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.474 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.474 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:24:36.475 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.830914+0000", 2024-04-23T16:24:36.475 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013270600000000001 2024-04-23T16:24:36.475 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.475 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.475 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.475 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.843921+0000", 2024-04-23T16:24:36.475 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013006877 2024-04-23T16:24:36.475 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.475 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.475 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.475 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.921739+0000", 2024-04-23T16:24:36.475 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.07781826 2024-04-23T16:24:36.475 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.475 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.476 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.476 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.921791+0000", 2024-04-23T16:24:36.476 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1910999999999998e-05 2024-04-23T16:24:36.476 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.476 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.476 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.476 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.921850+0000", 2024-04-23T16:24:36.476 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9015000000000003e-05 2024-04-23T16:24:36.476 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.476 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.476 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.476 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.476 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.476 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:142539 3.0 3:0982b120:::benchmark_data_smithi188_89276_object142538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e220)", 2024-04-23T16:24:36.476 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:31.817681+0000", 2024-04-23T16:24:36.476 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 64.557268147000002, 2024-04-23T16:24:36.477 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.104267177, 2024-04-23T16:24:36.477 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.477 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.477 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.477 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.477 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.477 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.477 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 142539 2024-04-23T16:24:36.477 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.477 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.477 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.477 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.477 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.817681+0000", 2024-04-23T16:24:36.477 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.477 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.478 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.478 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.478 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.817681+0000", 2024-04-23T16:24:36.478 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.478 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.478 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.478 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.478 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.817683+0000", 2024-04-23T16:24:36.478 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.863e-06 2024-04-23T16:24:36.478 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.479 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.479 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.479 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.817707+0000", 2024-04-23T16:24:36.479 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3986999999999999e-05 2024-04-23T16:24:36.479 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.479 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.479 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.479 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.817719+0000", 2024-04-23T16:24:36.479 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.257e-05 2024-04-23T16:24:36.479 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.479 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.479 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.479 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.817727+0000", 2024-04-23T16:24:36.480 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6990000000000003e-06 2024-04-23T16:24:36.480 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.480 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.480 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.480 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.835390+0000", 2024-04-23T16:24:36.480 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017662501000000001 2024-04-23T16:24:36.480 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.480 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.480 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.480 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.835523+0000", 2024-04-23T16:24:36.481 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000133373 2024-04-23T16:24:36.481 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.481 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.481 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:24:36.481 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.835663+0000", 2024-04-23T16:24:36.481 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014024 2024-04-23T16:24:36.481 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.481 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.482 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.482 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.921642+0000", 2024-04-23T16:24:36.482 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.085978635999999997 2024-04-23T16:24:36.482 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.482 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.482 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.482 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.921861+0000", 2024-04-23T16:24:36.482 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021872499999999999 2024-04-23T16:24:36.482 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.482 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.482 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.482 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.921904+0000", 2024-04-23T16:24:36.482 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3541000000000002e-05 2024-04-23T16:24:36.482 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.483 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.483 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.483 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:31.921948+0000", 2024-04-23T16:24:36.483 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4042000000000003e-05 2024-04-23T16:24:36.483 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.483 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.483 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.483 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.483 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.483 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:142893 3.0 3:0f6693e1:::benchmark_data_smithi188_89276_object142892:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e220)", 2024-04-23T16:24:36.483 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:32.255897+0000", 2024-04-23T16:24:36.483 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 64.119052449999998, 2024-04-23T16:24:36.483 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.103727954, 2024-04-23T16:24:36.483 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.483 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.484 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.484 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.484 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.484 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.484 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 142893 2024-04-23T16:24:36.484 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.484 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.484 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.484 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.484 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:32.255897+0000", 2024-04-23T16:24:36.484 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.484 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.485 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.485 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.485 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:32.255897+0000", 2024-04-23T16:24:36.485 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.485 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.485 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.485 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.485 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:32.255898+0000", 2024-04-23T16:24:36.485 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.119e-06 2024-04-23T16:24:36.485 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.485 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.485 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.485 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:32.255915+0000", 2024-04-23T16:24:36.485 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7595e-05 2024-04-23T16:24:36.485 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.486 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.486 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.486 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:32.255924+0000", 2024-04-23T16:24:36.486 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2810000000000008e-06 2024-04-23T16:24:36.486 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.486 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.486 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.486 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:32.255929+0000", 2024-04-23T16:24:36.486 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2129999999999999e-06 2024-04-23T16:24:36.486 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.486 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.487 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.487 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:32.273172+0000", 2024-04-23T16:24:36.487 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01724299 2024-04-23T16:24:36.487 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.487 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.487 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.487 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:32.273308+0000", 2024-04-23T16:24:36.487 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000136083 2024-04-23T16:24:36.487 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.487 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.487 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:24:36.487 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:32.273454+0000", 2024-04-23T16:24:36.487 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014593200000000001 2024-04-23T16:24:36.488 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.488 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.488 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.488 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:32.281843+0000", 2024-04-23T16:24:36.488 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0083887609999999998 2024-04-23T16:24:36.488 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.488 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.488 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.488 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:32.359513+0000", 2024-04-23T16:24:36.488 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.077670559 2024-04-23T16:24:36.488 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.488 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.488 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.489 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:32.359569+0000", 2024-04-23T16:24:36.489 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6172999999999999e-05 2024-04-23T16:24:36.489 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.489 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.489 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.489 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:32.359625+0000", 2024-04-23T16:24:36.489 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5247999999999997e-05 2024-04-23T16:24:36.489 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.489 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.489 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.489 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.490 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.490 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:144263 3.0 3:063b74ac:::benchmark_data_smithi188_89276_object144262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-23T16:24:36.490 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:33.863595+0000", 2024-04-23T16:24:36.490 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 62.511354576999999, 2024-04-23T16:24:36.490 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.10686381, 2024-04-23T16:24:36.490 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.490 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.490 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.490 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.490 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.490 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.490 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 144263 2024-04-23T16:24:36.491 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.491 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.491 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.491 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.491 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.863595+0000", 2024-04-23T16:24:36.491 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.491 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.491 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.491 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.491 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.863595+0000", 2024-04-23T16:24:36.491 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.491 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.492 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.492 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.492 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.863596+0000", 2024-04-23T16:24:36.492 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.561e-06 2024-04-23T16:24:36.492 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.492 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.492 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.492 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.863616+0000", 2024-04-23T16:24:36.492 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0234999999999999e-05 2024-04-23T16:24:36.492 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.492 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.492 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.492 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.863626+0000", 2024-04-23T16:24:36.492 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3910000000000004e-06 2024-04-23T16:24:36.493 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.493 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.493 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.493 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.863632+0000", 2024-04-23T16:24:36.493 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3690000000000003e-06 2024-04-23T16:24:36.493 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.493 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.493 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.493 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.877972+0000", 2024-04-23T16:24:36.493 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014339670000000001 2024-04-23T16:24:36.493 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.494 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.494 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.878120+0000", 2024-04-23T16:24:36.494 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014862999999999999 2024-04-23T16:24:36.494 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.494 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:24:36.494 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.878249+0000", 2024-04-23T16:24:36.494 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000128913 2024-04-23T16:24:36.494 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.494 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.494 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.883230+0000", 2024-04-23T16:24:36.494 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049810719999999996 2024-04-23T16:24:36.495 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.495 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.495 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.495 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.970355+0000", 2024-04-23T16:24:36.495 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.087124249000000001 2024-04-23T16:24:36.495 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.495 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.495 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.495 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.970399+0000", 2024-04-23T16:24:36.498 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4110999999999998e-05 2024-04-23T16:24:36.498 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.499 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.499 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.499 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.970458+0000", 2024-04-23T16:24:36.499 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9608999999999998e-05 2024-04-23T16:24:36.499 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.499 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.499 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.499 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.499 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.499 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:144286 3.0 3:04324bea:::benchmark_data_smithi188_89276_object144285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-23T16:24:36.499 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:33.891208+0000", 2024-04-23T16:24:36.499 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 62.483741643999998, 2024-04-23T16:24:36.499 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.104300183, 2024-04-23T16:24:36.499 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.500 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.500 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.500 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.500 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.500 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.500 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 144286 2024-04-23T16:24:36.500 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.500 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.500 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.500 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.500 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.891208+0000", 2024-04-23T16:24:36.500 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.500 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.501 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.501 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.501 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.891208+0000", 2024-04-23T16:24:36.501 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.501 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.501 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.501 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.501 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.891208+0000", 2024-04-23T16:24:36.501 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7700000000000004e-07 2024-04-23T16:24:36.501 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.501 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.501 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.501 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.891227+0000", 2024-04-23T16:24:36.502 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8717999999999999e-05 2024-04-23T16:24:36.502 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.502 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.502 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.502 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.891232+0000", 2024-04-23T16:24:36.502 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.181e-06 2024-04-23T16:24:36.502 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.502 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.502 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.502 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.891236+0000", 2024-04-23T16:24:36.502 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9029999999999997e-06 2024-04-23T16:24:36.502 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.502 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.503 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.503 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.971790+0000", 2024-04-23T16:24:36.503 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.080554076000000002 2024-04-23T16:24:36.503 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.503 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.503 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.503 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.971891+0000", 2024-04-23T16:24:36.503 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010105199999999999 2024-04-23T16:24:36.503 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.503 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.503 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:24:36.503 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.972006+0000", 2024-04-23T16:24:36.503 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000115246 2024-04-23T16:24:36.504 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.504 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.504 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.504 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.980886+0000", 2024-04-23T16:24:36.504 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0088790580000000004 2024-04-23T16:24:36.504 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.504 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.504 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.504 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.995394+0000", 2024-04-23T16:24:36.504 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014508734000000001 2024-04-23T16:24:36.504 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.504 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.504 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.505 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.995443+0000", 2024-04-23T16:24:36.505 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8826999999999999e-05 2024-04-23T16:24:36.505 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.505 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.505 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.505 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.995508+0000", 2024-04-23T16:24:36.505 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4610999999999995e-05 2024-04-23T16:24:36.505 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.505 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.505 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.505 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.505 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.505 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:144291 3.0 3:09dd8888:::benchmark_data_smithi188_89276_object144290:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-23T16:24:36.505 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:33.892269+0000", 2024-04-23T16:24:36.506 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 62.482680090999999, 2024-04-23T16:24:36.506 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.10346717, 2024-04-23T16:24:36.506 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.506 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.506 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.506 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.506 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.506 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.506 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 144291 2024-04-23T16:24:36.506 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.506 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.506 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.507 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.507 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.892269+0000", 2024-04-23T16:24:36.507 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.507 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.507 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.507 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.507 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.892269+0000", 2024-04-23T16:24:36.507 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.507 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.507 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.507 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.507 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.892270+0000", 2024-04-23T16:24:36.507 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3340000000000001e-06 2024-04-23T16:24:36.507 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.508 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.508 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.508 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.892299+0000", 2024-04-23T16:24:36.508 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8632999999999998e-05 2024-04-23T16:24:36.508 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.508 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.508 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.508 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.892310+0000", 2024-04-23T16:24:36.508 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0709e-05 2024-04-23T16:24:36.508 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.508 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.508 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.508 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.892317+0000", 2024-04-23T16:24:36.508 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.092e-06 2024-04-23T16:24:36.509 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.509 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.509 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.509 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.975822+0000", 2024-04-23T16:24:36.509 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.083505403000000006 2024-04-23T16:24:36.509 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.509 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.509 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.509 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.975949+0000", 2024-04-23T16:24:36.509 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000126711 2024-04-23T16:24:36.509 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.509 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.509 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:24:36.510 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.976116+0000", 2024-04-23T16:24:36.510 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016708000000000001 2024-04-23T16:24:36.510 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.510 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.510 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.510 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.995599+0000", 2024-04-23T16:24:36.510 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019483277 2024-04-23T16:24:36.510 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.510 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.510 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.510 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.995642+0000", 2024-04-23T16:24:36.510 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2314999999999997e-05 2024-04-23T16:24:36.510 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.511 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.511 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.511 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.995692+0000", 2024-04-23T16:24:36.511 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9931e-05 2024-04-23T16:24:36.511 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.511 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.511 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.511 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.995736+0000", 2024-04-23T16:24:36.511 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4685000000000003e-05 2024-04-23T16:24:36.511 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.511 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.511 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.511 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.511 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.512 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:144298 3.8 3:12d45068:::benchmark_data_smithi188_89276_object144297:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-23T16:24:36.512 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:33.896390+0000", 2024-04-23T16:24:36.512 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 62.478559203000003, 2024-04-23T16:24:36.512 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.099201626000000001, 2024-04-23T16:24:36.512 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.512 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.512 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.512 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.512 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.512 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.512 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 144298 2024-04-23T16:24:36.512 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.512 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.513 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.513 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.513 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.896390+0000", 2024-04-23T16:24:36.513 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.513 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.513 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.513 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.513 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.896390+0000", 2024-04-23T16:24:36.513 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.513 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.513 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.513 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.513 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.896395+0000", 2024-04-23T16:24:36.513 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1900000000000003e-06 2024-04-23T16:24:36.513 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.514 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.514 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.514 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.896420+0000", 2024-04-23T16:24:36.514 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4938000000000001e-05 2024-04-23T16:24:36.514 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.514 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.514 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.514 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.896433+0000", 2024-04-23T16:24:36.514 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2459999999999999e-05 2024-04-23T16:24:36.514 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.514 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.514 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.514 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.896441+0000", 2024-04-23T16:24:36.515 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1510000000000004e-06 2024-04-23T16:24:36.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.515 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.515 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.515 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.975813+0000", 2024-04-23T16:24:36.515 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.079372125000000002 2024-04-23T16:24:36.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.515 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.515 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.515 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.975949+0000", 2024-04-23T16:24:36.515 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013600099999999999 2024-04-23T16:24:36.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.516 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 1", 2024-04-23T16:24:36.516 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.976257+0000", 2024-04-23T16:24:36.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000307707 2024-04-23T16:24:36.516 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.516 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.516 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.980902+0000", 2024-04-23T16:24:36.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004645265 2024-04-23T16:24:36.516 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.516 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.516 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.995518+0000", 2024-04-23T16:24:36.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01461638 2024-04-23T16:24:36.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.517 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.517 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.995560+0000", 2024-04-23T16:24:36.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1900000000000002e-05 2024-04-23T16:24:36.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.517 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.517 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:33.995592+0000", 2024-04-23T16:24:36.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1508999999999999e-05 2024-04-23T16:24:36.517 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.517 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.517 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.518 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.518 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:144742 3.0 3:0f55c46b:::benchmark_data_smithi188_89276_object144741:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-23T16:24:36.518 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:34.381304+0000", 2024-04-23T16:24:36.518 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 61.993645274999999, 2024-04-23T16:24:36.518 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.101502201, 2024-04-23T16:24:36.518 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.518 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.518 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.518 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.518 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.518 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.518 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 144742 2024-04-23T16:24:36.519 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.519 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.519 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.519 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.519 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.381304+0000", 2024-04-23T16:24:36.519 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.519 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.519 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.519 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.519 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.381304+0000", 2024-04-23T16:24:36.519 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.519 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.519 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.519 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.520 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.381306+0000", 2024-04-23T16:24:36.520 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.159e-06 2024-04-23T16:24:36.520 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.520 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.520 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.520 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.381329+0000", 2024-04-23T16:24:36.520 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3298e-05 2024-04-23T16:24:36.520 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.520 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.520 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.520 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.381339+0000", 2024-04-23T16:24:36.520 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3009999999999995e-06 2024-04-23T16:24:36.520 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.521 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.521 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.521 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.381346+0000", 2024-04-23T16:24:36.521 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2099999999999996e-06 2024-04-23T16:24:36.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.521 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.521 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.521 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.393519+0000", 2024-04-23T16:24:36.521 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012173543 2024-04-23T16:24:36.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.521 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.521 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.521 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.393628+0000", 2024-04-23T16:24:36.521 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010840699999999999 2024-04-23T16:24:36.522 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.522 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:24:36.522 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.393725+0000", 2024-04-23T16:24:36.522 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6927000000000002e-05 2024-04-23T16:24:36.522 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.522 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.522 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.482636+0000", 2024-04-23T16:24:36.522 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.088910950000000002 2024-04-23T16:24:36.522 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.522 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.522 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.482715+0000", 2024-04-23T16:24:36.523 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8862000000000005e-05 2024-04-23T16:24:36.523 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.523 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.523 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.523 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.482756+0000", 2024-04-23T16:24:36.523 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.197e-05 2024-04-23T16:24:36.523 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.523 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.523 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.523 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.482806+0000", 2024-04-23T16:24:36.523 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9573999999999997e-05 2024-04-23T16:24:36.523 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.523 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.524 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.524 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.524 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.524 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:144759 3.8 3:1a263054:::benchmark_data_smithi188_89276_object144758:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-23T16:24:36.524 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:34.400008+0000", 2024-04-23T16:24:36.524 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 61.974941295000001, 2024-04-23T16:24:36.524 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.101758552, 2024-04-23T16:24:36.524 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.524 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.524 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.524 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.524 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.524 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.524 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 144759 2024-04-23T16:24:36.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.525 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.525 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.525 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.400008+0000", 2024-04-23T16:24:36.525 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.525 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.525 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.400008+0000", 2024-04-23T16:24:36.525 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.400009+0000", 2024-04-23T16:24:36.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4479999999999999e-06 2024-04-23T16:24:36.526 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.400031+0000", 2024-04-23T16:24:36.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1770999999999999e-05 2024-04-23T16:24:36.526 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.400042+0000", 2024-04-23T16:24:36.527 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1124999999999999e-05 2024-04-23T16:24:36.527 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.527 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.527 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.527 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.400050+0000", 2024-04-23T16:24:36.527 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6760000000000007e-06 2024-04-23T16:24:36.527 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.527 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.527 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.527 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.488528+0000", 2024-04-23T16:24:36.527 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.088478039999999994 2024-04-23T16:24:36.527 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.527 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.527 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.528 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.488622+0000", 2024-04-23T16:24:36.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4141000000000005e-05 2024-04-23T16:24:36.528 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.528 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 1", 2024-04-23T16:24:36.528 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.488720+0000", 2024-04-23T16:24:36.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7909000000000001e-05 2024-04-23T16:24:36.528 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.528 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.528 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.496975+0000", 2024-04-23T16:24:36.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0082551989999999995 2024-04-23T16:24:36.528 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.529 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.529 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.501668+0000", 2024-04-23T16:24:36.529 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046925140000000001 2024-04-23T16:24:36.529 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.529 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.529 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.501717+0000", 2024-04-23T16:24:36.529 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.977e-05 2024-04-23T16:24:36.529 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.529 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.529 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.501766+0000", 2024-04-23T16:24:36.529 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8959000000000003e-05 2024-04-23T16:24:36.530 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.530 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.530 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.530 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.530 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.530 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:144932 3.0 3:0b624f3e:::benchmark_data_smithi188_89276_object144931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-23T16:24:36.530 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:34.613825+0000", 2024-04-23T16:24:36.530 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 61.761124453000001, 2024-04-23T16:24:36.530 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.10535787200000001, 2024-04-23T16:24:36.530 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.530 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.530 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.530 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.531 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.531 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.531 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 144932 2024-04-23T16:24:36.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.531 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.531 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.531 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.613825+0000", 2024-04-23T16:24:36.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.531 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.531 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.613825+0000", 2024-04-23T16:24:36.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.532 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.613826+0000", 2024-04-23T16:24:36.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1629999999999999e-06 2024-04-23T16:24:36.532 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.613861+0000", 2024-04-23T16:24:36.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5466000000000002e-05 2024-04-23T16:24:36.532 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.613868+0000", 2024-04-23T16:24:36.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9090000000000003e-06 2024-04-23T16:24:36.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.613873+0000", 2024-04-23T16:24:36.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2490000000000001e-06 2024-04-23T16:24:36.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.625228+0000", 2024-04-23T16:24:36.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011354348 2024-04-23T16:24:36.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.534 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.625307+0000", 2024-04-23T16:24:36.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9507999999999998e-05 2024-04-23T16:24:36.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.534 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:24:36.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.625404+0000", 2024-04-23T16:24:36.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6461999999999999e-05 2024-04-23T16:24:36.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.534 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.719064+0000", 2024-04-23T16:24:36.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.093660210999999993 2024-04-23T16:24:36.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.719098+0000", 2024-04-23T16:24:36.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3729000000000002e-05 2024-04-23T16:24:36.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.719139+0000", 2024-04-23T16:24:36.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0822999999999999e-05 2024-04-23T16:24:36.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.536 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.719183+0000", 2024-04-23T16:24:36.536 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4004000000000001e-05 2024-04-23T16:24:36.536 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.536 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.536 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.536 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.536 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.536 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:144936 3.8 3:1faa9dab:::benchmark_data_smithi188_89276_object144935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-23T16:24:36.536 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:34.613934+0000", 2024-04-23T16:24:36.536 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 61.761015467, 2024-04-23T16:24:36.536 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.105317388, 2024-04-23T16:24:36.536 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.536 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.537 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.537 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.537 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.537 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.537 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 144936 2024-04-23T16:24:36.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.537 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.613934+0000", 2024-04-23T16:24:36.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.613934+0000", 2024-04-23T16:24:36.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.613934+0000", 2024-04-23T16:24:36.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7599999999999997e-07 2024-04-23T16:24:36.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.613952+0000", 2024-04-23T16:24:36.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7155000000000001e-05 2024-04-23T16:24:36.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.539 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.539 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.539 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.613956+0000", 2024-04-23T16:24:36.539 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1119999999999998e-06 2024-04-23T16:24:36.539 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.539 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.539 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.539 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.613959+0000", 2024-04-23T16:24:36.539 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6509999999999999e-06 2024-04-23T16:24:36.539 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.539 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.539 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.540 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.629324+0000", 2024-04-23T16:24:36.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015364296 2024-04-23T16:24:36.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.540 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.540 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.629458+0000", 2024-04-23T16:24:36.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000134614 2024-04-23T16:24:36.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.540 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 1", 2024-04-23T16:24:36.540 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.629554+0000", 2024-04-23T16:24:36.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5390999999999995e-05 2024-04-23T16:24:36.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.719077+0000", 2024-04-23T16:24:36.541 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.089523302999999999 2024-04-23T16:24:36.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.719196+0000", 2024-04-23T16:24:36.541 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000119059 2024-04-23T16:24:36.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.719224+0000", 2024-04-23T16:24:36.541 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7614999999999999e-05 2024-04-23T16:24:36.542 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.542 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.542 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.542 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.719251+0000", 2024-04-23T16:24:36.542 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7515999999999999e-05 2024-04-23T16:24:36.542 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.542 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.542 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.542 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.542 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.542 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:145057 3.0 3:0910186e:::benchmark_data_smithi188_89276_object145056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-23T16:24:36.542 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:34.780455+0000", 2024-04-23T16:24:36.543 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 61.594494292, 2024-04-23T16:24:36.543 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.104814117, 2024-04-23T16:24:36.543 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.543 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.543 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.543 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.543 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.543 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.543 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 145057 2024-04-23T16:24:36.543 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.543 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.543 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.543 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.544 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.780455+0000", 2024-04-23T16:24:36.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.544 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.544 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.544 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.780455+0000", 2024-04-23T16:24:36.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.544 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.544 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.544 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.780457+0000", 2024-04-23T16:24:36.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.685e-06 2024-04-23T16:24:36.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.544 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.545 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.545 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.780480+0000", 2024-04-23T16:24:36.545 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3465e-05 2024-04-23T16:24:36.545 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.545 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.545 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.545 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.780491+0000", 2024-04-23T16:24:36.545 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.111e-05 2024-04-23T16:24:36.545 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.545 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.545 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.546 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.780500+0000", 2024-04-23T16:24:36.546 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7760000000000003e-06 2024-04-23T16:24:36.546 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.546 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.546 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.546 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.782632+0000", 2024-04-23T16:24:36.546 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021322440000000002 2024-04-23T16:24:36.546 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.546 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.546 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.546 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.782730+0000", 2024-04-23T16:24:36.546 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8237000000000002e-05 2024-04-23T16:24:36.546 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.546 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:24:36.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.782829+0000", 2024-04-23T16:24:36.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8638e-05 2024-04-23T16:24:36.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.885058+0000", 2024-04-23T16:24:36.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.10222914900000001 2024-04-23T16:24:36.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.885122+0000", 2024-04-23T16:24:36.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4183999999999994e-05 2024-04-23T16:24:36.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.885187+0000", 2024-04-23T16:24:36.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4380000000000004e-05 2024-04-23T16:24:36.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.885269+0000", 2024-04-23T16:24:36.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2249000000000005e-05 2024-04-23T16:24:36.548 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.548 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.548 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.549 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:145076 3.8 3:1378ec84:::benchmark_data_smithi188_89276_object145075:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-23T16:24:36.549 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:34.798043+0000", 2024-04-23T16:24:36.549 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 61.576906319999999, 2024-04-23T16:24:36.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.100680086, 2024-04-23T16:24:36.549 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.549 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.549 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.549 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.549 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.549 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.549 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 145076 2024-04-23T16:24:36.549 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.549 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.798043+0000", 2024-04-23T16:24:36.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.798043+0000", 2024-04-23T16:24:36.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.798045+0000", 2024-04-23T16:24:36.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6690000000000002e-06 2024-04-23T16:24:36.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.551 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.551 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.551 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.798069+0000", 2024-04-23T16:24:36.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3793000000000001e-05 2024-04-23T16:24:36.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.551 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.551 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.551 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.798082+0000", 2024-04-23T16:24:36.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2999999999999999e-05 2024-04-23T16:24:36.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.551 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.551 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.551 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.798091+0000", 2024-04-23T16:24:36.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2379999999999997e-06 2024-04-23T16:24:36.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.552 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.552 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.887455+0000", 2024-04-23T16:24:36.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.089364631 2024-04-23T16:24:36.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.552 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.552 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.887563+0000", 2024-04-23T16:24:36.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000107978 2024-04-23T16:24:36.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.552 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 1", 2024-04-23T16:24:36.552 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.887657+0000", 2024-04-23T16:24:36.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3924000000000004e-05 2024-04-23T16:24:36.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.898399+0000", 2024-04-23T16:24:36.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010741741000000001 2024-04-23T16:24:36.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.898660+0000", 2024-04-23T16:24:36.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026145299999999998 2024-04-23T16:24:36.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.898691+0000", 2024-04-23T16:24:36.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0911000000000003e-05 2024-04-23T16:24:36.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.898723+0000", 2024-04-23T16:24:36.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1748000000000002e-05 2024-04-23T16:24:36.554 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.554 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.554 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.554 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:145077 3.8 3:13b6e2ed:::benchmark_data_smithi188_89276_object145076:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-23T16:24:36.555 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:34.798924+0000", 2024-04-23T16:24:36.555 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 61.576024767, 2024-04-23T16:24:36.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.108224034, 2024-04-23T16:24:36.555 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.555 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.555 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.555 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.555 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.556 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.556 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 145077 2024-04-23T16:24:36.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.556 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.798924+0000", 2024-04-23T16:24:36.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.798924+0000", 2024-04-23T16:24:36.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.557 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.557 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.798926+0000", 2024-04-23T16:24:36.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.254e-06 2024-04-23T16:24:36.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.557 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.557 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.798947+0000", 2024-04-23T16:24:36.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1429000000000001e-05 2024-04-23T16:24:36.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.557 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.798957+0000", 2024-04-23T16:24:36.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7480000000000005e-06 2024-04-23T16:24:36.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.558 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.798964+0000", 2024-04-23T16:24:36.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9870000000000002e-06 2024-04-23T16:24:36.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.558 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.891232+0000", 2024-04-23T16:24:36.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0922684 2024-04-23T16:24:36.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.891330+0000", 2024-04-23T16:24:36.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7608000000000005e-05 2024-04-23T16:24:36.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 1", 2024-04-23T16:24:36.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.891424+0000", 2024-04-23T16:24:36.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4561000000000005e-05 2024-04-23T16:24:36.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.902488+0000", 2024-04-23T16:24:36.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011064048 2024-04-23T16:24:36.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.560 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.907049+0000", 2024-04-23T16:24:36.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045603930000000003 2024-04-23T16:24:36.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.560 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.907099+0000", 2024-04-23T16:24:36.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0522999999999997e-05 2024-04-23T16:24:36.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.560 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.907148+0000", 2024-04-23T16:24:36.561 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9082999999999997e-05 2024-04-23T16:24:36.561 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.561 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.561 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.561 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.561 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.561 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:145079 3.8 3:15ba4229:::benchmark_data_smithi188_89276_object145078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-23T16:24:36.561 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:34.799427+0000", 2024-04-23T16:24:36.561 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 61.575521926999997, 2024-04-23T16:24:36.561 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.107809895, 2024-04-23T16:24:36.561 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.561 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.561 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.562 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.562 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.562 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.562 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 145079 2024-04-23T16:24:36.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.562 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.562 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.562 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.799427+0000", 2024-04-23T16:24:36.562 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.562 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.562 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.799427+0000", 2024-04-23T16:24:36.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.563 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.563 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.799428+0000", 2024-04-23T16:24:36.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.08e-06 2024-04-23T16:24:36.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.563 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.563 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.799449+0000", 2024-04-23T16:24:36.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1004000000000001e-05 2024-04-23T16:24:36.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.563 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.563 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.564 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.799460+0000", 2024-04-23T16:24:36.564 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0259e-05 2024-04-23T16:24:36.564 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.564 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.564 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.799466+0000", 2024-04-23T16:24:36.564 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5300000000000002e-06 2024-04-23T16:24:36.564 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.564 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.564 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.894659+0000", 2024-04-23T16:24:36.564 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.095192918000000001 2024-04-23T16:24:36.564 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.565 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.565 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.565 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.894745+0000", 2024-04-23T16:24:36.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6399999999999999e-05 2024-04-23T16:24:36.565 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.565 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.565 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 1", 2024-04-23T16:24:36.565 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.894844+0000", 2024-04-23T16:24:36.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8476999999999999e-05 2024-04-23T16:24:36.565 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.565 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.565 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.565 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.902510+0000", 2024-04-23T16:24:36.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076665880000000002 2024-04-23T16:24:36.565 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.907159+0000", 2024-04-23T16:24:36.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046489690000000002 2024-04-23T16:24:36.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.907204+0000", 2024-04-23T16:24:36.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5040999999999998e-05 2024-04-23T16:24:36.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.907237+0000", 2024-04-23T16:24:36.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2629000000000002e-05 2024-04-23T16:24:36.567 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.567 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.567 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.567 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:145087 3.0 3:051558f0:::benchmark_data_smithi188_89276_object145086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-23T16:24:36.567 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:34.804037+0000", 2024-04-23T16:24:36.567 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 61.570911762000001, 2024-04-23T16:24:36.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.11118993100000001, 2024-04-23T16:24:36.567 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.567 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.567 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.568 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.568 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.568 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.568 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 145087 2024-04-23T16:24:36.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.568 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.804037+0000", 2024-04-23T16:24:36.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.804037+0000", 2024-04-23T16:24:36.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.569 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.569 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.569 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.569 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.804039+0000", 2024-04-23T16:24:36.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6819999999999999e-06 2024-04-23T16:24:36.569 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.569 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.569 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.569 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.804062+0000", 2024-04-23T16:24:36.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3104000000000001e-05 2024-04-23T16:24:36.569 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.569 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.569 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.570 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.804070+0000", 2024-04-23T16:24:36.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7719999999999994e-06 2024-04-23T16:24:36.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.570 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.570 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.804076+0000", 2024-04-23T16:24:36.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4099999999999996e-06 2024-04-23T16:24:36.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.570 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.570 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.894648+0000", 2024-04-23T16:24:36.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.090571862000000003 2024-04-23T16:24:36.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.894755+0000", 2024-04-23T16:24:36.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000107045 2024-04-23T16:24:36.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:24:36.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.894854+0000", 2024-04-23T16:24:36.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8250000000000003e-05 2024-04-23T16:24:36.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.907080+0000", 2024-04-23T16:24:36.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012226812 2024-04-23T16:24:36.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.915124+0000", 2024-04-23T16:24:36.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080437789999999992 2024-04-23T16:24:36.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.915175+0000", 2024-04-23T16:24:36.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0529999999999999e-05 2024-04-23T16:24:36.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.915227+0000", 2024-04-23T16:24:36.573 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2685000000000001e-05 2024-04-23T16:24:36.573 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.573 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.573 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.573 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:145094 3.0 3:008d1f23:::benchmark_data_smithi188_89276_object145093:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-23T16:24:36.573 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:34.809869+0000", 2024-04-23T16:24:36.573 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 61.565079676000003, 2024-04-23T16:24:36.573 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.105502363, 2024-04-23T16:24:36.573 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.573 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.573 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.573 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.574 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.574 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.574 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 145094 2024-04-23T16:24:36.574 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.574 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.574 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.574 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.574 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.809869+0000", 2024-04-23T16:24:36.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.574 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.574 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.574 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.574 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.809869+0000", 2024-04-23T16:24:36.575 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.575 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.575 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.575 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.575 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.809871+0000", 2024-04-23T16:24:36.575 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.745e-06 2024-04-23T16:24:36.575 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.575 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.575 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.575 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.809892+0000", 2024-04-23T16:24:36.575 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1267000000000002e-05 2024-04-23T16:24:36.575 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.575 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.575 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.576 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.809902+0000", 2024-04-23T16:24:36.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1530000000000003e-06 2024-04-23T16:24:36.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.576 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.576 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.576 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.809909+0000", 2024-04-23T16:24:36.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2010000000000002e-06 2024-04-23T16:24:36.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.576 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.576 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.576 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.898674+0000", 2024-04-23T16:24:36.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.088764843999999996 2024-04-23T16:24:36.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.576 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.577 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.898790+0000", 2024-04-23T16:24:36.577 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011616999999999999 2024-04-23T16:24:36.577 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.577 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:24:36.577 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.898917+0000", 2024-04-23T16:24:36.577 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012719900000000001 2024-04-23T16:24:36.577 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.577 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.577 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.915240+0000", 2024-04-23T16:24:36.577 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016323160999999999 2024-04-23T16:24:36.578 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.578 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.578 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.578 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.915284+0000", 2024-04-23T16:24:36.578 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3912999999999998e-05 2024-04-23T16:24:36.578 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.578 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.578 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.578 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.915337+0000", 2024-04-23T16:24:36.578 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2689999999999999e-05 2024-04-23T16:24:36.578 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.578 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.578 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.578 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.915372+0000", 2024-04-23T16:24:36.579 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.502e-05 2024-04-23T16:24:36.579 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.579 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.579 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.579 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.579 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.579 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:145099 3.0 3:0c8fcb3d:::benchmark_data_smithi188_89276_object145098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-23T16:24:36.579 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:34.814577+0000", 2024-04-23T16:24:36.579 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 61.560372487000002, 2024-04-23T16:24:36.579 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.104915838, 2024-04-23T16:24:36.579 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.579 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.579 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.579 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.580 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.580 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.580 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 145099 2024-04-23T16:24:36.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.580 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.580 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.580 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.580 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.814577+0000", 2024-04-23T16:24:36.580 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.580 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.580 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.580 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.814577+0000", 2024-04-23T16:24:36.580 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.581 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.814577+0000", 2024-04-23T16:24:36.581 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1999999999999999e-07 2024-04-23T16:24:36.581 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.581 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.814597+0000", 2024-04-23T16:24:36.581 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9666000000000001e-05 2024-04-23T16:24:36.581 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.581 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.814607+0000", 2024-04-23T16:24:36.581 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0006999999999999e-05 2024-04-23T16:24:36.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.582 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.814613+0000", 2024-04-23T16:24:36.582 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8100000000000003e-06 2024-04-23T16:24:36.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.582 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.902496+0000", 2024-04-23T16:24:36.582 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.087883177000000007 2024-04-23T16:24:36.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.583 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.902632+0000", 2024-04-23T16:24:36.583 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013629500000000001 2024-04-23T16:24:36.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.583 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.583 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:24:36.583 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.902786+0000", 2024-04-23T16:24:36.583 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015319500000000001 2024-04-23T16:24:36.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.583 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.583 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.583 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.915430+0000", 2024-04-23T16:24:36.583 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012644446 2024-04-23T16:24:36.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.583 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.584 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.584 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.919371+0000", 2024-04-23T16:24:36.584 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039414940000000002 2024-04-23T16:24:36.584 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.584 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.584 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.584 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.919423+0000", 2024-04-23T16:24:36.584 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1288000000000001e-05 2024-04-23T16:24:36.584 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.584 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.584 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.584 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:34.919492+0000", 2024-04-23T16:24:36.584 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9740000000000007e-05 2024-04-23T16:24:36.584 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.585 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.585 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.585 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.585 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.585 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:145420 3.8 3:1d258184:::benchmark_data_smithi188_89276_object145419:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-23T16:24:36.585 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:35.224628+0000", 2024-04-23T16:24:36.585 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 61.15032128, 2024-04-23T16:24:36.585 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.100558893, 2024-04-23T16:24:36.585 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:24:36.585 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:24:36.585 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:24:36.585 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:24:36.585 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:24:36.585 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:24:36.586 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 145420 2024-04-23T16:24:36.586 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.586 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:24:36.586 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.586 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:24:36.586 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:35.224628+0000", 2024-04-23T16:24:36.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.586 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.586 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.586 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:24:36.586 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:35.224628+0000", 2024-04-23T16:24:36.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:24:36.586 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.586 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.587 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:24:36.587 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:35.224630+0000", 2024-04-23T16:24:36.587 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9599999999999999e-06 2024-04-23T16:24:36.587 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.587 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.587 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:24:36.587 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:35.224664+0000", 2024-04-23T16:24:36.587 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4156000000000003e-05 2024-04-23T16:24:36.587 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.587 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.587 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:24:36.587 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:35.224678+0000", 2024-04-23T16:24:36.587 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3859999999999999e-05 2024-04-23T16:24:36.587 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.588 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.588 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:24:36.588 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:35.224687+0000", 2024-04-23T16:24:36.588 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9129999999999992e-06 2024-04-23T16:24:36.588 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.588 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.588 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:24:36.588 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:35.226877+0000", 2024-04-23T16:24:36.588 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021903830000000002 2024-04-23T16:24:36.588 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.588 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.588 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:24:36.588 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:35.227414+0000", 2024-04-23T16:24:36.588 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000536612 2024-04-23T16:24:36.589 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.589 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.589 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 1", 2024-04-23T16:24:36.589 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:35.227562+0000", 2024-04-23T16:24:36.589 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014781399999999999 2024-04-23T16:24:36.589 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.589 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.589 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:24:36.589 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:35.236573+0000", 2024-04-23T16:24:36.589 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0090110680000000005 2024-04-23T16:24:36.589 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.589 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.589 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:24:36.589 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:35.325122+0000", 2024-04-23T16:24:36.590 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.088549701999999994 2024-04-23T16:24:36.590 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.590 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.590 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:24:36.590 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:35.325157+0000", 2024-04-23T16:24:36.590 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5120999999999998e-05 2024-04-23T16:24:36.590 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:24:36.590 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:24:36.590 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:24:36.590 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:35.325187+0000", 2024-04-23T16:24:36.590 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9303999999999999e-05 2024-04-23T16:24:36.590 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.590 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.591 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.591 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:24:36.591 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:24:36.591 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:24:36.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:36.592 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:24:36.593 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:24:36.593 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:24:36.593 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:24:36.593 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:24:36.593 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:24:36.593 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:24:36.594 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:24:36.594 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T16:24:36.594 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:24:36.594 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:24:36.594 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:24:36.594 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:24:36.594 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:24:36.594 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:24:36.594 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:24:36.594 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:24:36.594 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:24:36.594 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:24:36.693 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:24:36.724 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:36.736 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:24:36.759 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2599343363' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:36.759 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2886196431' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:24:36.759 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:36.759 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:36.760 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:36.760 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:37.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2599343363' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:37.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2886196431' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:24:37.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:37.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:37.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:37.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:37.075 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-23T16:24:37.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2599343363' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:37.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2886196431' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:24:37.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:37.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:37.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:37.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:37.675 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:24:37.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:37.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:24:37.777 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:24:37.804 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:37 smithi103 ceph-mon[39629]: pgmap v686: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 444 op/s; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:37 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:24:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:37 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2886196431' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:24:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:37 smithi103 ceph-mon[39629]: osdmap e236: 12 total, 11 up, 11 in 2024-04-23T16:24:38.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:37 smithi148 ceph-mon[44520]: pgmap v686: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 444 op/s; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:38.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:37 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:24:38.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:37 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2886196431' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:24:38.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:37 smithi148 ceph-mon[44520]: osdmap e236: 12 total, 11 up, 11 in 2024-04-23T16:24:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:37 smithi188 ceph-mon[44101]: pgmap v686: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 444 op/s; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:37 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:24:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:37 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2886196431' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:24:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:37 smithi188 ceph-mon[44101]: osdmap e236: 12 total, 11 up, 11 in 2024-04-23T16:24:38.652 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:24:38.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:38.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:24:38.760 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:24:38.790 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:38.811 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:24:38.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:38.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:24:38.915 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:24:38.942 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:38.962 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:24:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:38.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:24:39.007 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:38 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:24:39.008 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:38 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:24:39.008 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:38 smithi103 ceph-mon[39629]: osdmap e237: 12 total, 11 up, 11 in 2024-04-23T16:24:39.008 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:38 smithi103 ceph-mon[39629]: 3.e scrub starts 2024-04-23T16:24:39.008 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:38 smithi103 ceph-mon[39629]: 3.9 scrub starts 2024-04-23T16:24:39.032 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:24:39.033 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:24:39.066 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:24:39.096 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:39.111 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:24:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:38 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:24:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:38 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:24:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:38 smithi148 ceph-mon[44520]: osdmap e237: 12 total, 11 up, 11 in 2024-04-23T16:24:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:38 smithi148 ceph-mon[44520]: 3.e scrub starts 2024-04-23T16:24:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:38 smithi148 ceph-mon[44520]: 3.9 scrub starts 2024-04-23T16:24:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:38 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:24:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:38 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:24:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:38 smithi188 ceph-mon[44101]: osdmap e237: 12 total, 11 up, 11 in 2024-04-23T16:24:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:38 smithi188 ceph-mon[44101]: 3.e scrub starts 2024-04-23T16:24:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:38 smithi188 ceph-mon[44101]: 3.9 scrub starts 2024-04-23T16:24:39.676 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:24:39.835 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:39 smithi103 ceph-mon[39629]: pgmap v689: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 128 op/s; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:39.835 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:39 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:24:39.836 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:39 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:24:39.836 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:39 smithi103 ceph-mon[39629]: osdmap e238: 12 total, 11 up, 11 in 2024-04-23T16:24:39.836 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:39 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 24054/256626 objects degraded (9.373%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:24:39.836 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:39 smithi103 ceph-mon[39629]: 3.e scrub ok 2024-04-23T16:24:39.836 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:39 smithi103 ceph-mon[39629]: 3.9 scrub ok 2024-04-23T16:24:40.120 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1 is already down. 2024-04-23T16:24:40.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:39 smithi148 ceph-mon[44520]: pgmap v689: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 128 op/s; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:40.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:39 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:24:40.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:39 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:24:40.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:39 smithi148 ceph-mon[44520]: osdmap e238: 12 total, 11 up, 11 in 2024-04-23T16:24:40.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:39 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 24054/256626 objects degraded (9.373%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:24:40.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:39 smithi148 ceph-mon[44520]: 3.e scrub ok 2024-04-23T16:24:40.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:39 smithi148 ceph-mon[44520]: 3.9 scrub ok 2024-04-23T16:24:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:39 smithi188 ceph-mon[44101]: pgmap v689: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 128 op/s; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:39 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:24:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:39 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:24:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:39 smithi188 ceph-mon[44101]: osdmap e238: 12 total, 11 up, 11 in 2024-04-23T16:24:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:39 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 24054/256626 objects degraded (9.373%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:24:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:39 smithi188 ceph-mon[44101]: 3.e scrub ok 2024-04-23T16:24:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:39 smithi188 ceph-mon[44101]: 3.9 scrub ok 2024-04-23T16:24:40.521 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:40 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T16:24:40.115+0000 7f135b1c7700 -1 mon.a@0(leader).osd e238 definitely_dead 0 2024-04-23T16:24:40.746 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:24:40.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:40.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:24:40.839 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:40 smithi103 ceph-mon[39629]: 3.b scrub starts 2024-04-23T16:24:40.839 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:40 smithi103 ceph-mon[39629]: 3.b scrub ok 2024-04-23T16:24:40.839 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:40 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2689912965' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["1"]}]: dispatch 2024-04-23T16:24:40.852 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:24:40.880 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:40.900 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:24:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:40.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:24:41.006 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:24:41.034 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:24:41.036 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:40 smithi188 ceph-mon[44101]: 3.b scrub starts 2024-04-23T16:24:41.037 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:40 smithi188 ceph-mon[44101]: 3.b scrub ok 2024-04-23T16:24:41.037 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:40 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2689912965' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["1"]}]: dispatch 2024-04-23T16:24:41.041 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:41.062 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:24:41.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:41.065 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:24:41.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:40 smithi148 ceph-mon[44520]: 3.b scrub starts 2024-04-23T16:24:41.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:40 smithi148 ceph-mon[44520]: 3.b scrub ok 2024-04-23T16:24:41.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:40 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2689912965' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["1"]}]: dispatch 2024-04-23T16:24:41.166 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:24:41.194 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:41.219 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:24:41.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:41.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:24:41.319 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] 2024-04-23T16:24:41.320 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd out 1 2024-04-23T16:24:41.322 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:24:41.350 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:41.373 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:24:41.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:41.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:24:41.481 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:41.513 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:41.536 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:41.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:41.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:41.641 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:24:41.669 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:41.693 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:24:41.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:41.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:24:41.765 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:24:41.767 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:24:41.796 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:24:41.823 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:41 smithi148 ceph-mon[44520]: pgmap v691: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 0 op/s; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:41.823 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:41 smithi148 ceph-mon[44520]: 3.3 scrub starts 2024-04-23T16:24:41.825 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:41.843 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:24:41.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:41.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:24:41.947 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:24:41.978 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:41.999 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:24:42.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:42.002 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:24:42.030 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:41 smithi103 ceph-mon[39629]: pgmap v691: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 0 op/s; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:42.030 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:41 smithi103 ceph-mon[39629]: 3.3 scrub starts 2024-04-23T16:24:42.042 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:24:42.103 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:24:42.128 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:41 smithi188 ceph-mon[44101]: pgmap v691: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 0 op/s; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:42.128 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:41 smithi188 ceph-mon[44101]: 3.3 scrub starts 2024-04-23T16:24:42.130 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:42.149 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:24:42.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:42.152 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:24:42.253 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:24:42.282 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:42.600 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:24:42.600 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:24:43.026 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:42 smithi103 ceph-mon[39629]: 3.c scrub starts 2024-04-23T16:24:43.027 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:42 smithi103 ceph-mon[39629]: 3.f scrub starts 2024-04-23T16:24:43.027 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:42 smithi103 ceph-mon[39629]: 3.3 scrub ok 2024-04-23T16:24:43.027 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:42 smithi103 ceph-mon[39629]: 3.c scrub ok 2024-04-23T16:24:43.030 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:24:43.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:43.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:24:43.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:42 smithi148 ceph-mon[44520]: 3.c scrub starts 2024-04-23T16:24:43.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:42 smithi148 ceph-mon[44520]: 3.f scrub starts 2024-04-23T16:24:43.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:42 smithi148 ceph-mon[44520]: 3.3 scrub ok 2024-04-23T16:24:43.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:42 smithi148 ceph-mon[44520]: 3.c scrub ok 2024-04-23T16:24:43.135 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:24:43.166 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:43.188 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:24:43.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:43.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:24:43.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:42 smithi188 ceph-mon[44101]: 3.c scrub starts 2024-04-23T16:24:43.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:42 smithi188 ceph-mon[44101]: 3.f scrub starts 2024-04-23T16:24:43.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:42 smithi188 ceph-mon[44101]: 3.3 scrub ok 2024-04-23T16:24:43.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:42 smithi188 ceph-mon[44101]: 3.c scrub ok 2024-04-23T16:24:43.292 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:43.320 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:43.340 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:43.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:43.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:43.444 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:24:43.474 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:44.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:43 smithi103 ceph-mon[39629]: 3.f scrub ok 2024-04-23T16:24:44.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:43 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:24:44.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:43 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:24:44.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:43 smithi103 ceph-mon[39629]: pgmap v692: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:44.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:43 smithi148 ceph-mon[44520]: 3.f scrub ok 2024-04-23T16:24:44.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:43 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:24:44.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:43 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:24:44.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:43 smithi148 ceph-mon[44520]: pgmap v692: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:43 smithi188 ceph-mon[44101]: 3.f scrub ok 2024-04-23T16:24:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:43 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:24:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:43 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:24:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:43 smithi188 ceph-mon[44101]: pgmap v692: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:44.751 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:24:44.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:44.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:24:44.858 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:24:44.888 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:45.089 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:24:45.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:45.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:24:45.195 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:24:45.227 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:45.248 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:24:45.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:45.251 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:24:45.352 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:24:45.378 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:45.400 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:24:45.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:45.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:24:45.505 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:24:45.536 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:45.635 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:24:45.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:45.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:24:45.650 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:24:45.671 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:24:45.692 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:24:45.713 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:24:45.732 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:24:45.741 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:45.751 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:24:45.767 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:45.770 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:24:45.785 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:45.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:45.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:45.792 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:24:45.812 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:24:45.840 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:24:45.861 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:24:45.861 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:24:45.861 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T16:24:45.861 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:24:45.861 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:24:45.861 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:24:45.861 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:24:45.861 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:24:45.861 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:24:45.862 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:24:45.862 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:24:45.862 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:24:45.862 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:24:45.888 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:24:45.921 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:45.942 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:24:45.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:45.945 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:24:46.046 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:46.072 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:45 smithi148 ceph-mon[44520]: pgmap v693: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:46.072 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:45 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2149362091' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:46.074 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:46.093 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:46.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:46.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:46.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:45 smithi103 ceph-mon[39629]: pgmap v693: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:45 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2149362091' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:46.197 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:24:46.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:45 smithi188 ceph-mon[44101]: pgmap v693: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:46.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:45 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2149362091' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:46.224 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:46.242 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:24:46.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:46.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:24:46.346 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:24:46.374 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:46.922 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:24:46.923 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:24:47.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:46 smithi103 ceph-mon[39629]: 3.5 scrub starts 2024-04-23T16:24:47.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:46 smithi103 ceph-mon[39629]: 3.5 scrub ok 2024-04-23T16:24:47.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:46 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:24:47.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:46 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:24:47.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:46 smithi148 ceph-mon[44520]: 3.5 scrub starts 2024-04-23T16:24:47.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:46 smithi148 ceph-mon[44520]: 3.5 scrub ok 2024-04-23T16:24:47.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:46 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:24:47.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:46 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:24:47.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:47.128 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:24:47.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:24:47.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:46 smithi188 ceph-mon[44101]: 3.5 scrub starts 2024-04-23T16:24:47.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:46 smithi188 ceph-mon[44101]: 3.5 scrub ok 2024-04-23T16:24:47.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:47.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:46 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:24:47.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:46 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:24:47.230 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:24:47.258 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:47.281 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:24:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:47.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:24:47.384 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:24:47.417 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:47.439 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:24:47.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:47.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:24:47.543 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:24:47.573 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:47.593 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:24:47.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:47.596 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:24:47.697 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:24:47.725 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:47.744 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:24:47.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:47.747 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:24:47.848 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:24:47.866 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-23T16:24:47.880 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:47.901 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:24:47.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:47.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:24:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:47 smithi188 ceph-mon[44101]: pgmap v694: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:47 smithi188 ceph-mon[44101]: osdmap e239: 12 total, 11 up, 11 in 2024-04-23T16:24:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:47 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2646326650' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:24:48.004 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:24:48.030 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:48.048 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:24:48.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:48.051 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:24:48.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:47 smithi103 ceph-mon[39629]: pgmap v694: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:48.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:47 smithi103 ceph-mon[39629]: osdmap e239: 12 total, 11 up, 11 in 2024-04-23T16:24:48.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:47 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2646326650' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:24:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:47 smithi148 ceph-mon[44520]: pgmap v694: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:47 smithi148 ceph-mon[44520]: osdmap e239: 12 total, 11 up, 11 in 2024-04-23T16:24:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:47 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2646326650' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:24:48.152 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:24:48.179 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:48.232 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:24:48.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:48.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:24:48.336 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:24:48.364 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:48.812 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:24:48.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:48.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:24:48.912 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:24:48.916 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:24:48.924 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:24:48.948 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:48.969 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:24:48.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:48.972 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:24:49.073 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:24:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:48 smithi103 ceph-mon[39629]: 3.7 scrub starts 2024-04-23T16:24:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:48 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:24:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:48 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2646326650' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:24:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:48 smithi103 ceph-mon[39629]: osdmap e240: 12 total, 11 up, 11 in 2024-04-23T16:24:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:48 smithi103 ceph-mon[39629]: 3.7 scrub ok 2024-04-23T16:24:49.100 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:48 smithi188 ceph-mon[44101]: 3.7 scrub starts 2024-04-23T16:24:49.101 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:48 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:24:49.101 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:48 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2646326650' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:24:49.101 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:48 smithi188 ceph-mon[44101]: osdmap e240: 12 total, 11 up, 11 in 2024-04-23T16:24:49.101 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:48 smithi188 ceph-mon[44101]: 3.7 scrub ok 2024-04-23T16:24:49.103 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:49.123 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:24:49.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:48 smithi148 ceph-mon[44520]: 3.7 scrub starts 2024-04-23T16:24:49.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:48 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:24:49.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:48 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2646326650' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:24:49.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:48 smithi148 ceph-mon[44520]: osdmap e240: 12 total, 11 up, 11 in 2024-04-23T16:24:49.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:48 smithi148 ceph-mon[44520]: 3.7 scrub ok 2024-04-23T16:24:49.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:49.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:24:49.227 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:24:49.259 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:49.280 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:24:49.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:49.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:24:49.384 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:24:49.415 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:49.434 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:24:49.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:49.437 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:24:49.538 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:24:49.566 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:49.663 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:24:49.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:49.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:24:49.768 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:24:49.800 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:49.821 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:24:49.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:49.825 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:24:49.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:49 smithi103 ceph-mon[39629]: pgmap v697: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:49.926 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:49.927 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:24:49.958 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:49.959 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:24:49.979 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:49.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:49.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:50.083 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:24:50.112 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:49 smithi148 ceph-mon[44520]: pgmap v697: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:50.150 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:24:50.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:50.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:24:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:49 smithi188 ceph-mon[44101]: pgmap v697: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:50.253 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:24:50.280 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:50.430 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:24:50.433 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:50 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T16:24:50.114+0000 7f135b1c7700 -1 mon.a@0(leader).osd e240 definitely_dead 0 2024-04-23T16:24:50.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:50.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:24:50.537 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:24:50.565 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:50.792 INFO:teuthology.orchestra.run.smithi103.stderr:marked out osd.1. 2024-04-23T16:24:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:50 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:24:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:50 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:24:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:50 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2660020149' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-23T16:24:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:50 smithi103 ceph-mon[39629]: Client client.admin marked osd.1 out, after it was down for 20 seconds 2024-04-23T16:24:51.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:50 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:24:51.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:50 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:24:51.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:50 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2660020149' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-23T16:24:51.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:50 smithi148 ceph-mon[44520]: Client client.admin marked osd.1 out, after it was down for 20 seconds 2024-04-23T16:24:51.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:50 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:24:51.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:50 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:24:51.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:50 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2660020149' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-23T16:24:51.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:50 smithi188 ceph-mon[44101]: Client client.admin marked osd.1 out, after it was down for 20 seconds 2024-04-23T16:24:51.691 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:24:51.714 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:24:51.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:51.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:24:51.738 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.1 2024-04-23T16:24:51.738 INFO:tasks.thrashosds.thrasher:remote for osd 1 is ubuntu@smithi103.front.sepia.ceph.com 2024-04-23T16:24:51.738 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.1 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-04-23T16:24:51.819 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:24:51.852 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:51.872 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:24:51.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:51.875 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:24:51.976 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:24:52.006 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:52.025 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:24:52.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:52.028 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:24:52.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:51 smithi103 ceph-mon[39629]: pgmap v698: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:52.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:51 smithi103 ceph-mon[39629]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T16:24:52.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:51 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2660020149' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-04-23T16:24:52.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:51 smithi103 ceph-mon[39629]: osdmap e241: 12 total, 11 up, 10 in 2024-04-23T16:24:52.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:51 smithi148 ceph-mon[44520]: pgmap v698: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:51 smithi148 ceph-mon[44520]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T16:24:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:51 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2660020149' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-04-23T16:24:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:51 smithi148 ceph-mon[44520]: osdmap e241: 12 total, 11 up, 10 in 2024-04-23T16:24:52.129 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:24:52.155 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:51 smithi188 ceph-mon[44101]: pgmap v698: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 24054/256626 objects degraded (9.373%) 2024-04-23T16:24:52.155 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:51 smithi188 ceph-mon[44101]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T16:24:52.155 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:51 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2660020149' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-04-23T16:24:52.155 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:51 smithi188 ceph-mon[44101]: osdmap e241: 12 total, 11 up, 10 in 2024-04-23T16:24:52.157 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:52.175 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:24:52.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:52.178 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:24:52.223 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.1/config 2024-04-23T16:24:52.279 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:24:52.306 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:52.379 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:24:52.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:52.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:24:52.484 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:24:52.512 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:52.796 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:52 smithi103 ceph-mon[39629]: 3.d deep-scrub starts 2024-04-23T16:24:52.796 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:52 smithi103 ceph-mon[39629]: osdmap e242: 12 total, 11 up, 10 in 2024-04-23T16:24:52.796 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:52 smithi103 ceph-mon[39629]: osdmap e243: 12 total, 11 up, 10 in 2024-04-23T16:24:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:52 smithi148 ceph-mon[44520]: 3.d deep-scrub starts 2024-04-23T16:24:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:52 smithi148 ceph-mon[44520]: osdmap e242: 12 total, 11 up, 10 in 2024-04-23T16:24:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:52 smithi148 ceph-mon[44520]: osdmap e243: 12 total, 11 up, 10 in 2024-04-23T16:24:53.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:52 smithi188 ceph-mon[44101]: 3.d deep-scrub starts 2024-04-23T16:24:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:52 smithi188 ceph-mon[44101]: osdmap e242: 12 total, 11 up, 10 in 2024-04-23T16:24:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:52 smithi188 ceph-mon[44101]: osdmap e243: 12 total, 11 up, 10 in 2024-04-23T16:24:53.589 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:24:53.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:53.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:24:53.694 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:24:53.721 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:53.743 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:24:53.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:53.746 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:24:53.847 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:24:53.877 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:53.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:53 smithi188 ceph-mon[44101]: pgmap v701: 17 pgs: 2 peering, 1 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 8007/256626 objects degraded (3.120%) 2024-04-23T16:24:53.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:53 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 8007/256626 objects degraded (3.120%), 1 pg degraded (PG_DEGRADED) 2024-04-23T16:24:53.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:53 smithi188 ceph-mon[44101]: 3.6 starting backfill to osd.4 from (0'0,0'0] MAX to 232'12907 2024-04-23T16:24:53.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:53 smithi188 ceph-mon[44101]: 3.8 starting backfill to osd.4 from (0'0,0'0] MAX to 232'13250 2024-04-23T16:24:53.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:53 smithi188 ceph-mon[44101]: 3.4 starting backfill to osd.5 from (0'0,0'0] MAX to 232'14672 2024-04-23T16:24:54.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:53 smithi103 ceph-mon[39629]: pgmap v701: 17 pgs: 2 peering, 1 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 8007/256626 objects degraded (3.120%) 2024-04-23T16:24:54.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:53 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 8007/256626 objects degraded (3.120%), 1 pg degraded (PG_DEGRADED) 2024-04-23T16:24:54.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:53 smithi103 ceph-mon[39629]: 3.6 starting backfill to osd.4 from (0'0,0'0] MAX to 232'12907 2024-04-23T16:24:54.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:53 smithi103 ceph-mon[39629]: 3.8 starting backfill to osd.4 from (0'0,0'0] MAX to 232'13250 2024-04-23T16:24:54.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:53 smithi103 ceph-mon[39629]: 3.4 starting backfill to osd.5 from (0'0,0'0] MAX to 232'14672 2024-04-23T16:24:54.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:53 smithi148 ceph-mon[44520]: pgmap v701: 17 pgs: 2 peering, 1 undersized+degraded+peered, 14 active+clean; 7.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 8007/256626 objects degraded (3.120%) 2024-04-23T16:24:54.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:53 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 8007/256626 objects degraded (3.120%), 1 pg degraded (PG_DEGRADED) 2024-04-23T16:24:54.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:53 smithi148 ceph-mon[44520]: 3.6 starting backfill to osd.4 from (0'0,0'0] MAX to 232'12907 2024-04-23T16:24:54.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:53 smithi148 ceph-mon[44520]: 3.8 starting backfill to osd.4 from (0'0,0'0] MAX to 232'13250 2024-04-23T16:24:54.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:53 smithi148 ceph-mon[44520]: 3.4 starting backfill to osd.5 from (0'0,0'0] MAX to 232'14672 2024-04-23T16:24:54.428 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:24:54.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:54.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:24:54.532 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:24:54.564 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:54.589 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:24:54.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:54.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:24:54.694 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:24:54.724 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:54.746 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:24:54.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:54.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:24:54.851 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:24:54.882 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:54.969 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:24:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:24:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:24:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:24:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:24:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:24:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:24:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:24:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:24:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:24:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:24:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:24:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T16:24:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:24:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:24:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:24:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:24:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:24:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:24:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:24:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:24:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:24:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:24:55.234 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:24:55.235 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:54 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3472749530' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:55.235 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:55.235 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:55.235 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:55.235 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:55.235 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:55.235 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:55.235 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:54 smithi103 ceph-mon[39629]: pgmap v703: 17 pgs: 1 active+clean+scrubbing+deep, 2 peering, 1 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 7574/256626 objects degraded (2.951%); 4.5 MiB/s, 72 objects/s recovering 2024-04-23T16:24:55.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:55.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:24:55.337 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:55.367 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:54 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3472749530' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:55.368 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:55.368 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:55.368 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:55.368 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:55.368 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:55.368 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:55.368 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:54 smithi148 ceph-mon[44520]: pgmap v703: 17 pgs: 1 active+clean+scrubbing+deep, 2 peering, 1 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 7574/256626 objects degraded (2.951%); 4.5 MiB/s, 72 objects/s recovering 2024-04-23T16:24:55.370 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:55.395 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:55.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:55.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:54 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3472749530' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:24:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:54 smithi188 ceph-mon[44101]: pgmap v703: 17 pgs: 1 active+clean+scrubbing+deep, 2 peering, 1 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 7574/256626 objects degraded (2.951%); 4.5 MiB/s, 72 objects/s recovering 2024-04-23T16:24:55.499 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:24:55.526 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:55.547 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:24:55.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:55.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:24:55.652 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:24:55.683 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:55.696 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:24:55.696 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:24:56.034 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:24:56.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:56.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:24:56.084 INFO:teuthology.orchestra.run.smithi103.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:24:56.139 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:24:56.168 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:56.188 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:24:56.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:56.191 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:24:56.292 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:24:56.295 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:24:56.295 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:24:56.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:56.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:56.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:56.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:56.320 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:56.322 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:56.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:56.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:56.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:56.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:56.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:24:56.509 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:24:56.520 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:24:56.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:56.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:24:56.625 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:24:56.653 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:56.673 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:24:56.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:56.677 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:24:56.778 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:24:56.810 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:56.831 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:24:56.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:56.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:24:56.934 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:24:57.031 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:57.054 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:24:57.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:57.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:24:57.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:56 smithi103 ceph-mon[39629]: pgmap v704: 17 pgs: 1 active+clean+scrubbing+deep, 3 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 20033/256626 objects degraded (7.806%); 42 MiB/s, 670 objects/s recovering 2024-04-23T16:24:57.158 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:24:57.186 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:57.207 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:24:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:57.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:24:57.311 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:24:57.341 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:57.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:56 smithi148 ceph-mon[44520]: pgmap v704: 17 pgs: 1 active+clean+scrubbing+deep, 3 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 20033/256626 objects degraded (7.806%); 42 MiB/s, 670 objects/s recovering 2024-04-23T16:24:57.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:56 smithi188 ceph-mon[44101]: pgmap v704: 17 pgs: 1 active+clean+scrubbing+deep, 3 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 20033/256626 objects degraded (7.806%); 42 MiB/s, 670 objects/s recovering 2024-04-23T16:24:57.697 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:24:58.291 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:24:58.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:58.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:24:58.313 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-23T16:24:58.397 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:24:58.427 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:58.448 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:24:58.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:58.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:24:58.552 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:24:58.584 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:58.607 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:24:58.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:58.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:24:58.711 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:24:58.738 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:58.760 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:24:58.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:58.763 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:24:58.778 INFO:teuthology.orchestra.run.smithi103.stdout:3.8 2024-04-23T16:24:58.779 INFO:teuthology.orchestra.run.smithi103.stdout:3.4 2024-04-23T16:24:58.779 INFO:teuthology.orchestra.run.smithi103.stdout:3.6 2024-04-23T16:24:58.863 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:24:58.891 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:59.015 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:24:59.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:24:59.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:24:59.056 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:24:59.121 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:24:59.151 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:24:59.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:59 smithi188 ceph-mon[44101]: pgmap v705: 17 pgs: 1 active+clean+scrubbing+deep, 3 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 20033/256626 objects degraded (7.806%); 32 MiB/s, 515 objects/s recovering 2024-04-23T16:24:59.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:24:59 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 20033/256626 objects degraded (7.806%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:25:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:59 smithi103 ceph-mon[39629]: pgmap v705: 17 pgs: 1 active+clean+scrubbing+deep, 3 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 20033/256626 objects degraded (7.806%); 32 MiB/s, 515 objects/s recovering 2024-04-23T16:25:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:24:59 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 20033/256626 objects degraded (7.806%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:25:00.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:59 smithi148 ceph-mon[44520]: pgmap v705: 17 pgs: 1 active+clean+scrubbing+deep, 3 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 20033/256626 objects degraded (7.806%); 32 MiB/s, 515 objects/s recovering 2024-04-23T16:25:00.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:24:59 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 20033/256626 objects degraded (7.806%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:25:00.912 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:00.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:00.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:01.021 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:25:01.052 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:01.076 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:25:01.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:01.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:25:01.180 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:25:01.208 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:01.229 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:25:01.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:01.232 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:25:01.333 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:25:01.361 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:01.382 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:25:01.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:01.385 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:25:01.486 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:25:01.514 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:01.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:01 smithi103 ceph-mon[39629]: pgmap v706: 17 pgs: 1 active+clean+scrubbing+deep, 3 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 16983/256626 objects degraded (6.618%); 50 MiB/s, 803 objects/s recovering 2024-04-23T16:25:01.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:01 smithi188 ceph-mon[44101]: pgmap v706: 17 pgs: 1 active+clean+scrubbing+deep, 3 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 16983/256626 objects degraded (6.618%); 50 MiB/s, 803 objects/s recovering 2024-04-23T16:25:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:01 smithi148 ceph-mon[44520]: pgmap v706: 17 pgs: 1 active+clean+scrubbing+deep, 3 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 16983/256626 objects degraded (6.618%); 50 MiB/s, 803 objects/s recovering 2024-04-23T16:25:03.464 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:25:03.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:03.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:25:03.569 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:25:03.597 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:03.616 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:25:03.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:03.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:25:03.721 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:25:03.749 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:03.770 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:25:03.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:03.773 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:25:03.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:03 smithi103 ceph-mon[39629]: pgmap v707: 17 pgs: 1 active+clean+scrubbing+deep, 3 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 10797/256626 objects degraded (4.207%); 83 MiB/s, 1.32k objects/s recovering 2024-04-23T16:25:03.874 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:03.903 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:03 smithi188 ceph-mon[44101]: pgmap v707: 17 pgs: 1 active+clean+scrubbing+deep, 3 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 10797/256626 objects degraded (4.207%); 83 MiB/s, 1.32k objects/s recovering 2024-04-23T16:25:04.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:03 smithi148 ceph-mon[44520]: pgmap v707: 17 pgs: 1 active+clean+scrubbing+deep, 3 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 10797/256626 objects degraded (4.207%); 83 MiB/s, 1.32k objects/s recovering 2024-04-23T16:25:04.290 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.1 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op export --pgid 3.4 --file /var/log/ceph/exp.3.4.1 2024-04-23T16:25:04.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:04 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 10797/256626 objects degraded (4.207%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:25:05.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:04 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 10797/256626 objects degraded (4.207%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:25:05.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:04 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 10797/256626 objects degraded (4.207%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:25:05.252 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:05.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:05.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:05.358 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:25:05.386 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:05.406 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:25:05.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:05.409 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:25:05.510 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:25:05.544 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:05.566 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:25:05.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:05.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:25:05.671 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:25:05.703 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:05.727 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:25:05.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:05.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:25:05.832 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:25:05.836 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:05 smithi103 ceph-mon[39629]: pgmap v708: 17 pgs: 1 active+clean+scrubbing+deep, 3 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 7923/256626 objects degraded (3.087%); 85 MiB/s, 1.37k objects/s recovering 2024-04-23T16:25:05.862 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:05 smithi148 ceph-mon[44520]: pgmap v708: 17 pgs: 1 active+clean+scrubbing+deep, 3 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 7923/256626 objects degraded (3.087%); 85 MiB/s, 1.37k objects/s recovering 2024-04-23T16:25:05.864 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:05.880 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.1/config 2024-04-23T16:25:05.887 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:25:05.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:05.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:25:05.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:05 smithi188 ceph-mon[44101]: pgmap v708: 17 pgs: 1 active+clean+scrubbing+deep, 3 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 7923/256626 objects degraded (3.087%); 85 MiB/s, 1.37k objects/s recovering 2024-04-23T16:25:05.990 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:25:06.023 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:06.048 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:25:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:06.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:25:06.152 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:25:06.179 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:06.200 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:25:06.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:06.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:25:06.304 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:06.335 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:06.421 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:06.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:06.526 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:25:06.557 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:06.579 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:25:06.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:06.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:25:06.639 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:25:06.639 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:25:06.639 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:25:06.640 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:25:06.640 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:25:06.640 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:25:06.640 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:25:06.640 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:25:06.640 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:25:06.640 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:25:06.640 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:25:06.640 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:25:06.640 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T16:25:06.640 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:25:06.641 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:25:06.641 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:25:06.641 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:25:06.641 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:25:06.641 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:25:06.641 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:25:06.641 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:25:06.641 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:25:06.641 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:25:06.684 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:25:06.709 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:06.727 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:25:06.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:06.730 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:25:06.831 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:25:06.858 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:06 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:25:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:06 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:25:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:06 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3066844588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:25:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:06.876 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:25:06.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:06.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:25:06.964 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:06 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:25:06.964 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:06 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:25:06.964 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:06 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3066844588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:25:06.964 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:06.964 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:06.964 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:06.964 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:06.964 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:06.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:06 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:25:06.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:06 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:25:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:06 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3066844588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:25:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:06.979 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:25:07.006 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:07.026 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:25:07.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:07.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:25:07.130 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:25:07.157 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:07.439 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:25:07.440 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:25:07.606 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:25:07.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:07.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:25:07.677 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:07 smithi103 ceph-mon[39629]: pgmap v709: 17 pgs: 1 undersized+remapped+peered, 1 active+clean+scrubbing+deep, 2 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 1901/256626 objects degraded (0.741%); 113 MiB/s, 1.81k objects/s recovering 2024-04-23T16:25:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:07 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:25:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:07 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:25:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:07 smithi103 ceph-mon[39629]: osdmap e244: 12 total, 11 up, 10 in 2024-04-23T16:25:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:07 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/409837049' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:25:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:07 smithi103 ceph-mon[39629]: from='client.25216 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:25:07.680 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:25:07.709 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:25:07.741 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:07.768 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:25:07.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:07.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:25:07.874 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:25:07.908 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:07.928 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:25:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:07 smithi148 ceph-mon[44520]: pgmap v709: 17 pgs: 1 undersized+remapped+peered, 1 active+clean+scrubbing+deep, 2 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 1901/256626 objects degraded (0.741%); 113 MiB/s, 1.81k objects/s recovering 2024-04-23T16:25:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:07 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:25:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:07 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:25:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:07 smithi148 ceph-mon[44520]: osdmap e244: 12 total, 11 up, 10 in 2024-04-23T16:25:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:07 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/409837049' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:25:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:07 smithi148 ceph-mon[44520]: from='client.25216 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:25:07.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:07.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:25:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:07 smithi188 ceph-mon[44101]: pgmap v709: 17 pgs: 1 undersized+remapped+peered, 1 active+clean+scrubbing+deep, 2 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 1901/256626 objects degraded (0.741%); 113 MiB/s, 1.81k objects/s recovering 2024-04-23T16:25:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:07 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:25:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:07 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:25:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:07 smithi188 ceph-mon[44101]: osdmap e244: 12 total, 11 up, 10 in 2024-04-23T16:25:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:07 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/409837049' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:25:07.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:07 smithi188 ceph-mon[44101]: from='client.25216 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:25:08.032 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:25:08.058 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:08.077 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:25:08.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:08.079 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:25:08.180 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:25:08.202 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:25:08.209 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:08.233 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:25:08.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:08.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:25:08.337 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:25:08.364 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:08.713 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:25:08.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:08.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:25:08.820 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:25:08.861 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:08.891 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:25:08.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:08.894 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:25:08.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:08 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:25:08.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:08 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:25:08.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:08 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:25:08.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:08 smithi188 ceph-mon[44101]: from='client.25216 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:25:08.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:08 smithi188 ceph-mon[44101]: osdmap e245: 12 total, 11 up, 10 in 2024-04-23T16:25:08.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:08 smithi188 ceph-mon[44101]: osdmap e246: 12 total, 11 up, 10 in 2024-04-23T16:25:08.996 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:25:09.030 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:09.067 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:25:09.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:09.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:25:09.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:08 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:25:09.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:08 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:25:09.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:08 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:25:09.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:08 smithi103 ceph-mon[39629]: from='client.25216 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:25:09.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:08 smithi103 ceph-mon[39629]: osdmap e245: 12 total, 11 up, 10 in 2024-04-23T16:25:09.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:08 smithi103 ceph-mon[39629]: osdmap e246: 12 total, 11 up, 10 in 2024-04-23T16:25:09.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:08 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:25:09.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:08 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:25:09.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:08 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:25:09.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:08 smithi148 ceph-mon[44520]: from='client.25216 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:25:09.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:08 smithi148 ceph-mon[44520]: osdmap e245: 12 total, 11 up, 10 in 2024-04-23T16:25:09.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:08 smithi148 ceph-mon[44520]: osdmap e246: 12 total, 11 up, 10 in 2024-04-23T16:25:09.171 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:25:09.203 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:09.226 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:25:09.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:09.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:25:09.243 INFO:teuthology.orchestra.run.smithi103.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:25:09.333 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:25:09.368 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:09.389 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:25:09.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:09.392 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:25:09.441 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:25:09.492 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:25:09.523 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:09.549 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:25:09.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:09.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:25:09.654 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:25:09.687 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:09 smithi188 ceph-mon[44101]: pgmap v712: 17 pgs: 1 active+clean+scrubbing+deep, 1 undersized+remapped+peered, 2 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 2 op/s; 1901/256582 objects degraded (0.741%); 118 MiB/s, 1.88k objects/s recovering 2024-04-23T16:25:09.688 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:09 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 1901/256582 objects degraded (0.741%), 2 pgs degraded (PG_DEGRADED) 2024-04-23T16:25:09.692 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:09.726 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:25:09.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:09.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:25:09.831 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:25:09.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:09 smithi103 ceph-mon[39629]: pgmap v712: 17 pgs: 1 active+clean+scrubbing+deep, 1 undersized+remapped+peered, 2 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 2 op/s; 1901/256582 objects degraded (0.741%); 118 MiB/s, 1.88k objects/s recovering 2024-04-23T16:25:09.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:09 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 1901/256582 objects degraded (0.741%), 2 pgs degraded (PG_DEGRADED) 2024-04-23T16:25:09.877 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:09.925 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:25:09.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:09.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:25:10.027 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:10.065 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:10.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:09 smithi148 ceph-mon[44520]: pgmap v712: 17 pgs: 1 active+clean+scrubbing+deep, 1 undersized+remapped+peered, 2 undersized+degraded+remapped+backfilling+peered, 13 active+clean; 7.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 2 op/s; 1901/256582 objects degraded (0.741%); 118 MiB/s, 1.88k objects/s recovering 2024-04-23T16:25:10.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:09 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 1901/256582 objects degraded (0.741%), 2 pgs degraded (PG_DEGRADED) 2024-04-23T16:25:10.432 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:10.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:10.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:10.539 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:10.568 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:10.613 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:10.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:10.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:10.698 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:10 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:25:10.725 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:25:10.752 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:25:10.754 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:10.775 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:25:10.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:10.778 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:25:10.879 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:25:10.923 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:10.966 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:25:10.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:10.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:25:10.971 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:10 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:25:10.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:10 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:25:11.071 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:25:11.105 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:11.132 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:25:11.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:11.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:25:11.241 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:25:11.272 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:11.299 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:25:11.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:11.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:25:11.407 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:25:11.434 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:11.456 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:25:11.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:11.459 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:25:11.560 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:25:11.600 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:11 smithi103 ceph-mon[39629]: pgmap v714: 17 pgs: 1 active+clean+scrubbing+deep, 1 undersized+remapped+peered, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 42 op/s; 251/256114 objects degraded (0.098%); 80 MiB/s, 1.28k objects/s recovering 2024-04-23T16:25:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:11 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:25:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:11 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:25:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:11 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:25:11.955 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:25:11.973 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:25:11.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:11 smithi188 ceph-mon[44101]: pgmap v714: 17 pgs: 1 active+clean+scrubbing+deep, 1 undersized+remapped+peered, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 42 op/s; 251/256114 objects degraded (0.098%); 80 MiB/s, 1.28k objects/s recovering 2024-04-23T16:25:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:11 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:25:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:11 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:25:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:11 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:25:11.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:11.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:25:11.984 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:25:11.985 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:25:11.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:25:11.985 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:25:11.985 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.985 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:63237 3.6 3:65ed91f2:::benchmark_data_smithi188_89276_object63236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:25:11.985 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:22:43.686482+0000", 2024-04-23T16:25:11.985 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 148.28499952000001, 2024-04-23T16:25:11.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.090854089999999998, 2024-04-23T16:25:11.985 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:11.985 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:11.985 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:25:11.985 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:25:11.985 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:25:11.985 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:25:11.986 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 63237 2024-04-23T16:25:11.986 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.986 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:11.986 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.986 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:11.986 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.686482+0000", 2024-04-23T16:25:11.986 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:11.986 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.986 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.986 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:11.986 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.686482+0000", 2024-04-23T16:25:11.986 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:11.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.987 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:11.987 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.686484+0000", 2024-04-23T16:25:11.987 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6619999999999999e-06 2024-04-23T16:25:11.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.987 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:11.987 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.686630+0000", 2024-04-23T16:25:11.987 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145957 2024-04-23T16:25:11.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:11.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.686638+0000", 2024-04-23T16:25:11.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0330000000000008e-06 2024-04-23T16:25:11.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:11.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.686645+0000", 2024-04-23T16:25:11.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2570000000000001e-06 2024-04-23T16:25:11.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:11.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.686705+0000", 2024-04-23T16:25:11.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9985999999999998e-05 2024-04-23T16:25:11.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:25:11.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.686810+0000", 2024-04-23T16:25:11.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000104392 2024-04-23T16:25:11.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 1", 2024-04-23T16:25:11.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.686909+0000", 2024-04-23T16:25:11.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9334999999999997e-05 2024-04-23T16:25:11.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:25:11.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.688528+0000", 2024-04-23T16:25:11.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016195809999999999 2024-04-23T16:25:11.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:25:11.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777219+0000", 2024-04-23T16:25:11.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.088691014999999998 2024-04-23T16:25:11.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:25:11.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777290+0000", 2024-04-23T16:25:11.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0197999999999994e-05 2024-04-23T16:25:11.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.991 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:11.991 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777336+0000", 2024-04-23T16:25:11.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6674000000000001e-05 2024-04-23T16:25:11.991 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:11.991 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:11.991 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:11.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.991 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:63240 3.6 3:6a85c7b3:::benchmark_data_smithi188_89276_object63239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:25:11.991 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:22:43.688662+0000", 2024-04-23T16:25:11.991 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 148.282819749, 2024-04-23T16:25:11.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.088815627999999994, 2024-04-23T16:25:11.991 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:11.991 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:11.992 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:25:11.992 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:25:11.992 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:25:11.992 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:25:11.992 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 63240 2024-04-23T16:25:11.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.992 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:11.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:11.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.688662+0000", 2024-04-23T16:25:11.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:11.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:11.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.688662+0000", 2024-04-23T16:25:11.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:11.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:11.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.688664+0000", 2024-04-23T16:25:11.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.934e-06 2024-04-23T16:25:11.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:11.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.688721+0000", 2024-04-23T16:25:11.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6787999999999999e-05 2024-04-23T16:25:11.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:11.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.688730+0000", 2024-04-23T16:25:11.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0329999999999997e-06 2024-04-23T16:25:11.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:11.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.688737+0000", 2024-04-23T16:25:11.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3939999999999998e-06 2024-04-23T16:25:11.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:11.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.688788+0000", 2024-04-23T16:25:11.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0742999999999999e-05 2024-04-23T16:25:11.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.995 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:25:11.995 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.688901+0000", 2024-04-23T16:25:11.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000113417 2024-04-23T16:25:11.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.995 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 1", 2024-04-23T16:25:11.995 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.689021+0000", 2024-04-23T16:25:11.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119895 2024-04-23T16:25:11.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.995 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:25:11.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777096+0000", 2024-04-23T16:25:11.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.088074316999999999 2024-04-23T16:25:11.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:25:11.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777376+0000", 2024-04-23T16:25:11.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000280604 2024-04-23T16:25:11.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:25:11.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777434+0000", 2024-04-23T16:25:11.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7995000000000003e-05 2024-04-23T16:25:11.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:11.997 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777478+0000", 2024-04-23T16:25:11.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3507999999999997e-05 2024-04-23T16:25:11.997 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:11.997 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:11.997 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:11.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.997 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.997 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:63245 3.1 3:835a1b04:::benchmark_data_smithi188_89276_object63244:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:25:11.997 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:22:43.689827+0000", 2024-04-23T16:25:11.997 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 148.281654651, 2024-04-23T16:25:11.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.094048810999999996, 2024-04-23T16:25:11.997 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:11.997 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:11.997 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:25:11.998 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:25:11.998 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:25:11.998 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:25:11.998 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 63245 2024-04-23T16:25:11.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.998 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:11.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:11.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.689827+0000", 2024-04-23T16:25:11.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:11.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:11.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.689827+0000", 2024-04-23T16:25:11.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:11.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:11.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.689829+0000", 2024-04-23T16:25:11.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6819999999999999e-06 2024-04-23T16:25:11.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:11.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:11.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.689897+0000", 2024-04-23T16:25:11.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8207000000000006e-05 2024-04-23T16:25:11.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:11.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:12.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.689907+0000", 2024-04-23T16:25:12.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8530000000000004e-06 2024-04-23T16:25:12.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:12.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.689914+0000", 2024-04-23T16:25:12.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2089999999999999e-06 2024-04-23T16:25:12.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.689955+0000", 2024-04-23T16:25:12.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0852999999999997e-05 2024-04-23T16:25:12.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:25:12.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.690027+0000", 2024-04-23T16:25:12.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2235000000000002e-05 2024-04-23T16:25:12.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:25:12.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.690115+0000", 2024-04-23T16:25:12.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7941000000000004e-05 2024-04-23T16:25:12.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:25:12.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.779426+0000", 2024-04-23T16:25:12.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.089310794999999998 2024-04-23T16:25:12.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:25:12.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.783793+0000", 2024-04-23T16:25:12.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043673660000000001 2024-04-23T16:25:12.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:25:12.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.783834+0000", 2024-04-23T16:25:12.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1099000000000001e-05 2024-04-23T16:25:12.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:12.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.783876+0000", 2024-04-23T16:25:12.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1570999999999999e-05 2024-04-23T16:25:12.003 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.003 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:12.003 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.003 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:63246 3.1 3:84a49624:::benchmark_data_smithi188_89276_object63245:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:25:12.003 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:22:43.689934+0000", 2024-04-23T16:25:12.003 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 148.28154806000001, 2024-04-23T16:25:12.003 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.094007146, 2024-04-23T16:25:12.003 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:12.003 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:12.003 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:25:12.003 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:25:12.003 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:25:12.003 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:25:12.004 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 63246 2024-04-23T16:25:12.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.004 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:12.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:12.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.689934+0000", 2024-04-23T16:25:12.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:12.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.689934+0000", 2024-04-23T16:25:12.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:12.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.689934+0000", 2024-04-23T16:25:12.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9999999999999996e-07 2024-04-23T16:25:12.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:12.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.689967+0000", 2024-04-23T16:25:12.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2509000000000003e-05 2024-04-23T16:25:12.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:12.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.689976+0000", 2024-04-23T16:25:12.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6249999999999996e-06 2024-04-23T16:25:12.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:12.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.689981+0000", 2024-04-23T16:25:12.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5149999999999997e-06 2024-04-23T16:25:12.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.692051+0000", 2024-04-23T16:25:12.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020698740000000002 2024-04-23T16:25:12.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:25:12.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.692148+0000", 2024-04-23T16:25:12.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7387999999999997e-05 2024-04-23T16:25:12.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.007 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:25:12.007 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777268+0000", 2024-04-23T16:25:12.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.085119426999999998 2024-04-23T16:25:12.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.007 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:25:12.007 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.783705+0000", 2024-04-23T16:25:12.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0064375300000000003 2024-04-23T16:25:12.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.007 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:25:12.007 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.783884+0000", 2024-04-23T16:25:12.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017915300000000001 2024-04-23T16:25:12.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.008 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:25:12.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.783910+0000", 2024-04-23T16:25:12.008 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5542999999999999e-05 2024-04-23T16:25:12.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.008 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:12.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.783941+0000", 2024-04-23T16:25:12.008 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0781999999999998e-05 2024-04-23T16:25:12.008 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.008 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:12.008 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.009 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:63249 3.1 3:8192bfc8:::benchmark_data_smithi188_89276_object63248:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:25:12.009 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:22:43.690749+0000", 2024-04-23T16:25:12.009 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 148.28073312399999, 2024-04-23T16:25:12.009 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.093258340999999995, 2024-04-23T16:25:12.009 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:12.009 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:12.009 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:25:12.009 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:25:12.009 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:25:12.009 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:25:12.009 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 63249 2024-04-23T16:25:12.009 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.009 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:12.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.010 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:12.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.690749+0000", 2024-04-23T16:25:12.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.010 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:12.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.690749+0000", 2024-04-23T16:25:12.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.010 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:12.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.690750+0000", 2024-04-23T16:25:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4500000000000001e-06 2024-04-23T16:25:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.690821+0000", 2024-04-23T16:25:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1286999999999997e-05 2024-04-23T16:25:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.690831+0000", 2024-04-23T16:25:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6139999999999998e-06 2024-04-23T16:25:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.690839+0000", 2024-04-23T16:25:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6899999999999992e-06 2024-04-23T16:25:12.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.012 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.012 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.779488+0000", 2024-04-23T16:25:12.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.088649251999999998 2024-04-23T16:25:12.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.012 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:25:12.012 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.779633+0000", 2024-04-23T16:25:12.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014541099999999999 2024-04-23T16:25:12.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.012 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:25:12.012 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.779767+0000", 2024-04-23T16:25:12.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013331000000000001 2024-04-23T16:25:12.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.013 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:25:12.013 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.783780+0000", 2024-04-23T16:25:12.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0040136319999999996 2024-04-23T16:25:12.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.013 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:25:12.013 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.783948+0000", 2024-04-23T16:25:12.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016757400000000001 2024-04-23T16:25:12.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.013 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:25:12.013 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.783977+0000", 2024-04-23T16:25:12.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9059000000000001e-05 2024-04-23T16:25:12.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.014 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.014 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:12.014 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.784007+0000", 2024-04-23T16:25:12.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0062000000000001e-05 2024-04-23T16:25:12.014 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.014 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:12.014 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.014 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.014 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:63250 3.1 3:8378cf8c:::benchmark_data_smithi188_89276_object63249:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:25:12.014 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:22:43.690877+0000", 2024-04-23T16:25:12.015 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 148.280604334, 2024-04-23T16:25:12.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.095298190000000005, 2024-04-23T16:25:12.015 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:12.015 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:12.015 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:25:12.015 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:25:12.015 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:25:12.015 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:25:12.015 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 63250 2024-04-23T16:25:12.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.015 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:12.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.015 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:12.015 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.690877+0000", 2024-04-23T16:25:12.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:12.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.690877+0000", 2024-04-23T16:25:12.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:12.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.690879+0000", 2024-04-23T16:25:12.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.17e-06 2024-04-23T16:25:12.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:12.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.690968+0000", 2024-04-23T16:25:12.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9308999999999995e-05 2024-04-23T16:25:12.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:12.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.690977+0000", 2024-04-23T16:25:12.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9579999999999996e-06 2024-04-23T16:25:12.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:12.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.690984+0000", 2024-04-23T16:25:12.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1659999999999997e-06 2024-04-23T16:25:12.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.018 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.781859+0000", 2024-04-23T16:25:12.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.090874808000000001 2024-04-23T16:25:12.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:25:12.018 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.782003+0000", 2024-04-23T16:25:12.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014406600000000001 2024-04-23T16:25:12.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:25:12.018 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.782149+0000", 2024-04-23T16:25:12.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014619399999999999 2024-04-23T16:25:12.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.019 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:25:12.019 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.786064+0000", 2024-04-23T16:25:12.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039147940000000001 2024-04-23T16:25:12.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.019 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:25:12.019 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.786077+0000", 2024-04-23T16:25:12.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3267e-05 2024-04-23T16:25:12.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:25:12.020 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.786131+0000", 2024-04-23T16:25:12.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3940000000000002e-05 2024-04-23T16:25:12.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:12.020 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.786176+0000", 2024-04-23T16:25:12.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4518000000000003e-05 2024-04-23T16:25:12.020 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.020 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:12.020 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.021 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:63251 3.6 3:60823b57:::benchmark_data_smithi188_89276_object63250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:25:12.021 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:22:43.690994+0000", 2024-04-23T16:25:12.021 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 148.28048746900001, 2024-04-23T16:25:12.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.086605707000000004, 2024-04-23T16:25:12.021 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:12.021 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:12.021 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:25:12.021 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:25:12.021 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:25:12.021 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:25:12.021 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 63251 2024-04-23T16:25:12.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.021 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:12.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:12.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.690994+0000", 2024-04-23T16:25:12.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:12.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.690994+0000", 2024-04-23T16:25:12.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:12.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.690995+0000", 2024-04-23T16:25:12.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9299999999999996e-07 2024-04-23T16:25:12.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:12.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691043+0000", 2024-04-23T16:25:12.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8090999999999997e-05 2024-04-23T16:25:12.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:12.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691051+0000", 2024-04-23T16:25:12.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2559999999999996e-06 2024-04-23T16:25:12.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:12.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691056+0000", 2024-04-23T16:25:12.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8710000000000002e-06 2024-04-23T16:25:12.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.024 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691134+0000", 2024-04-23T16:25:12.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7459000000000005e-05 2024-04-23T16:25:12.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.024 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:25:12.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691245+0000", 2024-04-23T16:25:12.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110665 2024-04-23T16:25:12.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.024 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 1", 2024-04-23T16:25:12.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691347+0000", 2024-04-23T16:25:12.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000102346 2024-04-23T16:25:12.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:25:12.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777126+0000", 2024-04-23T16:25:12.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.085779223000000002 2024-04-23T16:25:12.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:25:12.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777522+0000", 2024-04-23T16:25:12.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00039632399999999998 2024-04-23T16:25:12.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:25:12.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777566+0000", 2024-04-23T16:25:12.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3816999999999997e-05 2024-04-23T16:25:12.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:12.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777600+0000", 2024-04-23T16:25:12.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3762e-05 2024-04-23T16:25:12.026 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.026 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:12.026 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.026 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.15150.0:63247 3.b e204/192 3:da287136:::benchmark_data_smithi188_89276_object63246:head v 204'3984, mlcod=204'3981)", 2024-04-23T16:25:12.026 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:22:43.691493+0000", 2024-04-23T16:25:12.027 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 148.27998847800001, 2024-04-23T16:25:12.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.085684885000000002, 2024-04-23T16:25:12.027 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:12.027 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:12.027 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:25:12.027 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:12.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:12.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691493+0000", 2024-04-23T16:25:12.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:12.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691493+0000", 2024-04-23T16:25:12.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:12.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691495+0000", 2024-04-23T16:25:12.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4309999999999999e-06 2024-04-23T16:25:12.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:12.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691516+0000", 2024-04-23T16:25:12.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1016e-05 2024-04-23T16:25:12.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:12.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691524+0000", 2024-04-23T16:25:12.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4570000000000005e-06 2024-04-23T16:25:12.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:12.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691530+0000", 2024-04-23T16:25:12.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2389999999999999e-06 2024-04-23T16:25:12.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691577+0000", 2024-04-23T16:25:12.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6419999999999999e-05 2024-04-23T16:25:12.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:25:12.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691594+0000", 2024-04-23T16:25:12.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7547e-05 2024-04-23T16:25:12.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:25:12.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777101+0000", 2024-04-23T16:25:12.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.085507043000000005 2024-04-23T16:25:12.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:12.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777178+0000", 2024-04-23T16:25:12.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6731999999999996e-05 2024-04-23T16:25:12.031 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.031 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:12.031 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.031 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop_reply(client.15150.0:63245 3.1 e204/192 ondisk, result = 0)", 2024-04-23T16:25:12.031 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:22:43.691686+0000", 2024-04-23T16:25:12.031 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 148.27979567700001, 2024-04-23T16:25:12.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.087747364999999994, 2024-04-23T16:25:12.031 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:12.031 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:12.031 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:25:12.032 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:12.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:12.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691686+0000", 2024-04-23T16:25:12.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:12.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691686+0000", 2024-04-23T16:25:12.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:12.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691687+0000", 2024-04-23T16:25:12.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5900000000000005e-07 2024-04-23T16:25:12.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:12.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691691+0000", 2024-04-23T16:25:12.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.659e-06 2024-04-23T16:25:12.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:12.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691698+0000", 2024-04-23T16:25:12.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7619999999999995e-06 2024-04-23T16:25:12.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:12.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.691704+0000", 2024-04-23T16:25:12.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5400000000000003e-06 2024-04-23T16:25:12.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.779412+0000", 2024-04-23T16:25:12.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.087708101999999996 2024-04-23T16:25:12.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:25:12.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.779420+0000", 2024-04-23T16:25:12.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.123e-06 2024-04-23T16:25:12.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:12.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.779433+0000", 2024-04-23T16:25:12.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.322e-05 2024-04-23T16:25:12.035 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.035 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:12.035 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.035 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop_reply(client.15150.0:63237 3.6 e204/192 ondisk, result = 0)", 2024-04-23T16:25:12.035 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:22:43.692141+0000", 2024-04-23T16:25:12.035 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 148.27934033099999, 2024-04-23T16:25:12.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.085202281000000005, 2024-04-23T16:25:12.036 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:12.036 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:12.036 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:25:12.036 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:12.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.036 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:12.036 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.692141+0000", 2024-04-23T16:25:12.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.036 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:12.036 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.692141+0000", 2024-04-23T16:25:12.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:12.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.692142+0000", 2024-04-23T16:25:12.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0219999999999999e-06 2024-04-23T16:25:12.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:12.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.692146+0000", 2024-04-23T16:25:12.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4910000000000001e-06 2024-04-23T16:25:12.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:12.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.692153+0000", 2024-04-23T16:25:12.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2559999999999996e-06 2024-04-23T16:25:12.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:12.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.692158+0000", 2024-04-23T16:25:12.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2310000000000004e-06 2024-04-23T16:25:12.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777206+0000", 2024-04-23T16:25:12.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.085047858000000004 2024-04-23T16:25:12.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:25:12.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777214+0000", 2024-04-23T16:25:12.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4460000000000004e-06 2024-04-23T16:25:12.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:12.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777344+0000", 2024-04-23T16:25:12.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012997700000000001 2024-04-23T16:25:12.039 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.039 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:12.039 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.039 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.15150.0:63252 3.3 e204/192 3:c2527cae:::benchmark_data_smithi188_89276_object63251:head v 204'3938, mlcod=204'3936)", 2024-04-23T16:25:12.039 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:22:43.692466+0000", 2024-04-23T16:25:12.039 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 148.27901534599999, 2024-04-23T16:25:12.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.087498736999999993, 2024-04-23T16:25:12.040 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:12.040 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:12.040 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:25:12.040 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:12.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:12.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.692466+0000", 2024-04-23T16:25:12.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:12.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.692466+0000", 2024-04-23T16:25:12.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:12.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.692467+0000", 2024-04-23T16:25:12.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2800000000000005e-07 2024-04-23T16:25:12.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:12.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.692521+0000", 2024-04-23T16:25:12.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3384999999999998e-05 2024-04-23T16:25:12.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:12.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777234+0000", 2024-04-23T16:25:12.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.084713089000000005 2024-04-23T16:25:12.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.042 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:12.042 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777244+0000", 2024-04-23T16:25:12.042 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.747e-06 2024-04-23T16:25:12.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.042 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.042 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777446+0000", 2024-04-23T16:25:12.042 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000202129 2024-04-23T16:25:12.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:25:12.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777492+0000", 2024-04-23T16:25:12.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5989000000000003e-05 2024-04-23T16:25:12.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:25:12.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.779908+0000", 2024-04-23T16:25:12.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024159839999999999 2024-04-23T16:25:12.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:12.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.779965+0000", 2024-04-23T16:25:12.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7485999999999998e-05 2024-04-23T16:25:12.043 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.044 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:12.044 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.044 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:63260 3.1 3:84131747:::benchmark_data_smithi188_89276_object63259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:25:12.044 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:22:43.695485+0000", 2024-04-23T16:25:12.044 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 148.275996695, 2024-04-23T16:25:12.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.094919697999999997, 2024-04-23T16:25:12.044 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:12.044 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:12.044 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:25:12.044 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:25:12.044 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:25:12.044 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:25:12.044 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 63260 2024-04-23T16:25:12.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.045 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:12.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:12.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.695485+0000", 2024-04-23T16:25:12.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:12.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.695485+0000", 2024-04-23T16:25:12.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:12.046 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.695487+0000", 2024-04-23T16:25:12.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7570000000000001e-06 2024-04-23T16:25:12.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:12.046 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.695547+0000", 2024-04-23T16:25:12.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0189999999999998e-05 2024-04-23T16:25:12.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:12.046 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777375+0000", 2024-04-23T16:25:12.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.081828333000000003 2024-04-23T16:25:12.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.047 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:12.047 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.777387+0000", 2024-04-23T16:25:12.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1501000000000001e-05 2024-04-23T16:25:12.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.047 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.047 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.784043+0000", 2024-04-23T16:25:12.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006656159 2024-04-23T16:25:12.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.047 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:25:12.047 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.784186+0000", 2024-04-23T16:25:12.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014288300000000001 2024-04-23T16:25:12.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:25:12.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.784350+0000", 2024-04-23T16:25:12.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016411099999999999 2024-04-23T16:25:12.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:25:12.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.790253+0000", 2024-04-23T16:25:12.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0059035210000000001 2024-04-23T16:25:12.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:25:12.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.790321+0000", 2024-04-23T16:25:12.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7322000000000004e-05 2024-04-23T16:25:12.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:25:12.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.790363+0000", 2024-04-23T16:25:12.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2088000000000001e-05 2024-04-23T16:25:12.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:12.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:22:43.790405+0000", 2024-04-23T16:25:12.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1832999999999997e-05 2024-04-23T16:25:12.049 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.049 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:12.050 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.050 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:208911 3.6 3.aa423426 (undecoded) ondisk+retry+write+known_if_redirected e242)", 2024-04-23T16:25:12.050 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:24:52.809143+0000", 2024-04-23T16:25:12.050 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.162338802000001, 2024-04-23T16:25:12.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.56618363100000002, 2024-04-23T16:25:12.050 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:12.050 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:12.050 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-23T16:25:12.050 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:25:12.050 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:25:12.050 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:25:12.050 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 208911 2024-04-23T16:25:12.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.051 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:12.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:12.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809143+0000", 2024-04-23T16:25:12.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:12.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809143+0000", 2024-04-23T16:25:12.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:12.052 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809145+0000", 2024-04-23T16:25:12.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4430000000000002e-06 2024-04-23T16:25:12.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.052 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:12.052 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809153+0000", 2024-04-23T16:25:12.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1810000000000004e-06 2024-04-23T16:25:12.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.052 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:12.052 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809172+0000", 2024-04-23T16:25:12.053 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9245e-05 2024-04-23T16:25:12.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.053 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:12.053 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809186+0000", 2024-04-23T16:25:12.053 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.454e-05 2024-04-23T16:25:12.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.053 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.053 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809291+0000", 2024-04-23T16:25:12.053 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000104857 2024-04-23T16:25:12.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.053 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:25:12.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809300+0000", 2024-04-23T16:25:12.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2420000000000006e-06 2024-04-23T16:25:12.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:53.375294+0000", 2024-04-23T16:25:12.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.56599307600000004 2024-04-23T16:25:12.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:12.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:53.375327+0000", 2024-04-23T16:25:12.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3046999999999997e-05 2024-04-23T16:25:12.054 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.055 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:12.055 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.055 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:208934 3.6 3.9d16ad26 (undecoded) ondisk+retry+write+known_if_redirected e242)", 2024-04-23T16:25:12.055 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:24:52.809228+0000", 2024-04-23T16:25:12.055 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.162254201, 2024-04-23T16:25:12.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.56626001400000003, 2024-04-23T16:25:12.055 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:12.055 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:12.055 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-23T16:25:12.055 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:25:12.055 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:25:12.055 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:25:12.056 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 208934 2024-04-23T16:25:12.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.056 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:12.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:12.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809228+0000", 2024-04-23T16:25:12.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:12.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809228+0000", 2024-04-23T16:25:12.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:12.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809229+0000", 2024-04-23T16:25:12.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0070000000000001e-06 2024-04-23T16:25:12.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:12.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809232+0000", 2024-04-23T16:25:12.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8720000000000004e-06 2024-04-23T16:25:12.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:12.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809240+0000", 2024-04-23T16:25:12.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7069999999999992e-06 2024-04-23T16:25:12.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:12.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809247+0000", 2024-04-23T16:25:12.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7229999999999998e-06 2024-04-23T16:25:12.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809331+0000", 2024-04-23T16:25:12.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3857999999999996e-05 2024-04-23T16:25:12.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:25:12.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809335+0000", 2024-04-23T16:25:12.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4390000000000002e-06 2024-04-23T16:25:12.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:53.375477+0000", 2024-04-23T16:25:12.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.566141581 2024-04-23T16:25:12.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:12.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:53.375488+0000", 2024-04-23T16:25:12.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0827e-05 2024-04-23T16:25:12.059 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.059 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:12.060 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.060 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:208953 3.6 3.ec5fdbb6 (undecoded) ondisk+retry+write+known_if_redirected e242)", 2024-04-23T16:25:12.060 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:24:52.809257+0000", 2024-04-23T16:25:12.060 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.162224460000001, 2024-04-23T16:25:12.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.566273264, 2024-04-23T16:25:12.060 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:12.060 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:12.060 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-23T16:25:12.060 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:25:12.061 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:25:12.061 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:25:12.061 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 208953 2024-04-23T16:25:12.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.061 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:12.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:12.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809257+0000", 2024-04-23T16:25:12.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:12.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809257+0000", 2024-04-23T16:25:12.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:12.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809258+0000", 2024-04-23T16:25:12.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0200000000000002e-07 2024-04-23T16:25:12.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:12.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809261+0000", 2024-04-23T16:25:12.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7630000000000001e-06 2024-04-23T16:25:12.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:12.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809278+0000", 2024-04-23T16:25:12.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7320999999999999e-05 2024-04-23T16:25:12.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:12.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809284+0000", 2024-04-23T16:25:12.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.122e-06 2024-04-23T16:25:12.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809359+0000", 2024-04-23T16:25:12.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5150999999999999e-05 2024-04-23T16:25:12.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:25:12.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809363+0000", 2024-04-23T16:25:12.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8600000000000003e-06 2024-04-23T16:25:12.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:53.375523+0000", 2024-04-23T16:25:12.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.56616038599999996 2024-04-23T16:25:12.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:12.065 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:53.375531+0000", 2024-04-23T16:25:12.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0589999999999997e-06 2024-04-23T16:25:12.065 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.065 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:12.065 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.065 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:208959 3.6 3.2ae49496 (undecoded) ondisk+retry+write+known_if_redirected e242)", 2024-04-23T16:25:12.065 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:24:52.809293+0000", 2024-04-23T16:25:12.065 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.162188603000001, 2024-04-23T16:25:12.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.56626500700000004, 2024-04-23T16:25:12.065 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:12.065 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:12.065 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-23T16:25:12.066 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:25:12.066 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:25:12.066 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:25:12.066 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 208959 2024-04-23T16:25:12.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.066 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:12.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:12.066 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809293+0000", 2024-04-23T16:25:12.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:12.066 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809293+0000", 2024-04-23T16:25:12.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:12.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809294+0000", 2024-04-23T16:25:12.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0999999999999998e-07 2024-04-23T16:25:12.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:12.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809297+0000", 2024-04-23T16:25:12.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7690000000000001e-06 2024-04-23T16:25:12.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:12.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809303+0000", 2024-04-23T16:25:12.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2439999999999998e-06 2024-04-23T16:25:12.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:12.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809322+0000", 2024-04-23T16:25:12.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8879999999999999e-05 2024-04-23T16:25:12.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809385+0000", 2024-04-23T16:25:12.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3087000000000001e-05 2024-04-23T16:25:12.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:25:12.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809388+0000", 2024-04-23T16:25:12.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4750000000000002e-06 2024-04-23T16:25:12.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:53.375553+0000", 2024-04-23T16:25:12.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.56616464099999997 2024-04-23T16:25:12.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:12.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:53.375558+0000", 2024-04-23T16:25:12.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3009999999999997e-06 2024-04-23T16:25:12.069 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.070 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:12.070 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.070 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:208978 3.6 3.ac8ca7f6 (undecoded) ondisk+retry+write+known_if_redirected e242)", 2024-04-23T16:25:12.070 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:24:52.809330+0000", 2024-04-23T16:25:12.070 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.162152077000002, 2024-04-23T16:25:12.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.56633739599999999, 2024-04-23T16:25:12.070 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:12.070 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:12.070 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-23T16:25:12.070 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:25:12.070 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:25:12.071 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:25:12.071 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 208978 2024-04-23T16:25:12.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.071 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:12.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:12.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809330+0000", 2024-04-23T16:25:12.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:12.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809330+0000", 2024-04-23T16:25:12.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:12.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809330+0000", 2024-04-23T16:25:12.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0299999999999999e-07 2024-04-23T16:25:12.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:12.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809333+0000", 2024-04-23T16:25:12.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8629999999999999e-06 2024-04-23T16:25:12.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:12.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809339+0000", 2024-04-23T16:25:12.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9200000000000001e-06 2024-04-23T16:25:12.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:12.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809345+0000", 2024-04-23T16:25:12.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7570000000000001e-06 2024-04-23T16:25:12.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809403+0000", 2024-04-23T16:25:12.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8669999999999999e-05 2024-04-23T16:25:12.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:25:12.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:52.809407+0000", 2024-04-23T16:25:12.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.456e-06 2024-04-23T16:25:12.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:53.375663+0000", 2024-04-23T16:25:12.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.56625571100000005 2024-04-23T16:25:12.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:12.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:24:53.375667+0000", 2024-04-23T16:25:12.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4159999999999997e-06 2024-04-23T16:25:12.074 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.074 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:12.074 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.075 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:209142 3.6 3:6c600c23:::benchmark_data_smithi188_89276_object40498:head [delete] snapc 0=[] ondisk+write+known_if_redirected e245)", 2024-04-23T16:25:12.075 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:25:08.677305+0000", 2024-04-23T16:25:12.075 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.2941770460000002, 2024-04-23T16:25:12.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.080074410999999998, 2024-04-23T16:25:12.075 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:12.075 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:12.075 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:25:12.075 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:25:12.075 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:25:12.075 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:25:12.075 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 209142 2024-04-23T16:25:12.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.076 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:12.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:12.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677305+0000", 2024-04-23T16:25:12.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:12.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677305+0000", 2024-04-23T16:25:12.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:12.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677305+0000", 2024-04-23T16:25:12.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8999999999999997e-07 2024-04-23T16:25:12.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:12.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677308+0000", 2024-04-23T16:25:12.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5890000000000001e-06 2024-04-23T16:25:12.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:12.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677313+0000", 2024-04-23T16:25:12.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6210000000000001e-06 2024-04-23T16:25:12.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:12.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677318+0000", 2024-04-23T16:25:12.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4070000000000003e-06 2024-04-23T16:25:12.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677466+0000", 2024-04-23T16:25:12.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000147966 2024-04-23T16:25:12.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:25:12.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677469+0000", 2024-04-23T16:25:12.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.591e-06 2024-04-23T16:25:12.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.736254+0000", 2024-04-23T16:25:12.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.058784267000000001 2024-04-23T16:25:12.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:25:12.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.736349+0000", 2024-04-23T16:25:12.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5480000000000001e-05 2024-04-23T16:25:12.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:25:12.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.736449+0000", 2024-04-23T16:25:12.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9907000000000004e-05 2024-04-23T16:25:12.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:25:12.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.757208+0000", 2024-04-23T16:25:12.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020759197 2024-04-23T16:25:12.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:25:12.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.757288+0000", 2024-04-23T16:25:12.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9827000000000007e-05 2024-04-23T16:25:12.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:25:12.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.757335+0000", 2024-04-23T16:25:12.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6727999999999997e-05 2024-04-23T16:25:12.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:12.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.757379+0000", 2024-04-23T16:25:12.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4341000000000001e-05 2024-04-23T16:25:12.081 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.081 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:12.081 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.082 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:209146 3.6 3:6da7c165:::benchmark_data_smithi188_89276_object40502:head [delete] snapc 0=[] ondisk+write+known_if_redirected e245)", 2024-04-23T16:25:12.082 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:25:08.677336+0000", 2024-04-23T16:25:12.082 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.294146129, 2024-04-23T16:25:12.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.080155730999999994, 2024-04-23T16:25:12.082 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:12.082 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:12.082 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:25:12.082 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:25:12.082 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:25:12.082 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:25:12.082 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 209146 2024-04-23T16:25:12.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.082 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:12.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:12.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677336+0000", 2024-04-23T16:25:12.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:12.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677336+0000", 2024-04-23T16:25:12.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:12.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677336+0000", 2024-04-23T16:25:12.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3299999999999998e-07 2024-04-23T16:25:12.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:12.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677338+0000", 2024-04-23T16:25:12.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9269999999999999e-06 2024-04-23T16:25:12.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:12.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677341+0000", 2024-04-23T16:25:12.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5279999999999999e-06 2024-04-23T16:25:12.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:12.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677345+0000", 2024-04-23T16:25:12.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.19e-06 2024-04-23T16:25:12.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677491+0000", 2024-04-23T16:25:12.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014666700000000001 2024-04-23T16:25:12.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:25:12.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677494+0000", 2024-04-23T16:25:12.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1319999999999998e-06 2024-04-23T16:25:12.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.740197+0000", 2024-04-23T16:25:12.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.062702298000000004 2024-04-23T16:25:12.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:25:12.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.740315+0000", 2024-04-23T16:25:12.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117936 2024-04-23T16:25:12.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:25:12.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.740428+0000", 2024-04-23T16:25:12.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011309799999999999 2024-04-23T16:25:12.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:25:12.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.757271+0000", 2024-04-23T16:25:12.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016843336 2024-04-23T16:25:12.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:25:12.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.757388+0000", 2024-04-23T16:25:12.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116766 2024-04-23T16:25:12.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:25:12.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.757427+0000", 2024-04-23T16:25:12.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9023000000000002e-05 2024-04-23T16:25:12.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:12.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.757491+0000", 2024-04-23T16:25:12.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4497000000000003e-05 2024-04-23T16:25:12.088 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.088 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:12.088 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.088 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:209191 3.6 3:6052f746:::benchmark_data_smithi188_89276_object40547:head [delete] snapc 0=[] ondisk+write+known_if_redirected e245)", 2024-04-23T16:25:12.089 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:25:08.677360+0000", 2024-04-23T16:25:12.089 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.2941220900000001, 2024-04-23T16:25:12.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.084454387000000006, 2024-04-23T16:25:12.089 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:25:12.089 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:25:12.089 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:25:12.089 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:25:12.089 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:25:12.089 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:25:12.089 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 209191 2024-04-23T16:25:12.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.089 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:25:12.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:25:12.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677360+0000", 2024-04-23T16:25:12.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:25:12.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677360+0000", 2024-04-23T16:25:12.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:25:12.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:25:12.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677360+0000", 2024-04-23T16:25:12.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4200000000000002e-07 2024-04-23T16:25:12.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:25:12.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677362+0000", 2024-04-23T16:25:12.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.905e-06 2024-04-23T16:25:12.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:25:12.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677366+0000", 2024-04-23T16:25:12.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7249999999999999e-06 2024-04-23T16:25:12.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:25:12.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677374+0000", 2024-04-23T16:25:12.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3580000000000003e-06 2024-04-23T16:25:12.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677509+0000", 2024-04-23T16:25:12.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013490099999999999 2024-04-23T16:25:12.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:25:12.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.677512+0000", 2024-04-23T16:25:12.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2049999999999998e-06 2024-04-23T16:25:12.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:25:12.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.744344+0000", 2024-04-23T16:25:12.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.066832254999999993 2024-04-23T16:25:12.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:25:12.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.744477+0000", 2024-04-23T16:25:12.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000133036 2024-04-23T16:25:12.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:25:12.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.744601+0000", 2024-04-23T16:25:12.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012381299999999999 2024-04-23T16:25:12.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:25:12.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.757496+0000", 2024-04-23T16:25:12.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012894882 2024-04-23T16:25:12.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:25:12.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.761750+0000", 2024-04-23T16:25:12.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042538469999999998 2024-04-23T16:25:12.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:25:12.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.761782+0000", 2024-04-23T16:25:12.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1766e-05 2024-04-23T16:25:12.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:25:12.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:25:12.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:25:12.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:25:08.761814+0000", 2024-04-23T16:25:12.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2351999999999998e-05 2024-04-23T16:25:12.095 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.095 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:12.095 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.095 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:25:12.095 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:25:12.095 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:25:12.095 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:12.126 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:12.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:11 smithi148 ceph-mon[44520]: pgmap v714: 17 pgs: 1 active+clean+scrubbing+deep, 1 undersized+remapped+peered, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 7.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 42 op/s; 251/256114 objects degraded (0.098%); 80 MiB/s, 1.28k objects/s recovering 2024-04-23T16:25:12.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:11 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:25:12.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:11 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:25:12.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:11 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:25:12.169 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:12.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:12.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:12.220 INFO:teuthology.orchestra.run.smithi103.stderr:Exporting 3.4 info 3.4( v 232'14672 (204'4600,232'14672] local-lis/les=192/193 n=7913 ec=192/192 lis/c=192/192 les/c/f=193/193/0 sis=192) 2024-04-23T16:25:12.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:200026e8:::benchmark_data_smithi188_89276_object110474:head# 2024-04-23T16:25:12.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2000f5dd:::benchmark_data_smithi188_89276_object78808:head# 2024-04-23T16:25:12.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20014054:::benchmark_data_smithi188_89276_object78008:head# 2024-04-23T16:25:12.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2001e0d9:::benchmark_data_smithi188_89276_object142273:head# 2024-04-23T16:25:12.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20023ee6:::benchmark_data_smithi188_89276_object69071:head# 2024-04-23T16:25:12.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20028e3c:::benchmark_data_smithi188_89276_object149088:head# 2024-04-23T16:25:12.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2002cb3c:::benchmark_data_smithi188_89276_object54086:head# 2024-04-23T16:25:12.273 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:25:12.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20031387:::benchmark_data_smithi188_89276_object93275:head# 2024-04-23T16:25:12.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2005b236:::benchmark_data_smithi188_89276_object147534:head# 2024-04-23T16:25:12.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20061582:::benchmark_data_smithi188_89276_object97962:head# 2024-04-23T16:25:12.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:200644b3:::benchmark_data_smithi188_89276_object72676:head# 2024-04-23T16:25:12.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20065104:::benchmark_data_smithi188_89276_object95126:head# 2024-04-23T16:25:12.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20073164:::benchmark_data_smithi188_89276_object93259:head# 2024-04-23T16:25:12.277 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2007a527:::benchmark_data_smithi188_89276_object41416:head# 2024-04-23T16:25:12.277 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2007b429:::benchmark_data_smithi188_89276_object118514:head# 2024-04-23T16:25:12.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2007d92f:::benchmark_data_smithi188_89276_object43767:head# 2024-04-23T16:25:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20083883:::benchmark_data_smithi188_89276_object160356:head# 2024-04-23T16:25:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20084de0:::benchmark_data_smithi188_89276_object98513:head# 2024-04-23T16:25:12.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20093d4e:::benchmark_data_smithi188_89276_object136812:head# 2024-04-23T16:25:12.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2009bcb7:::benchmark_data_smithi188_89276_object98740:head# 2024-04-23T16:25:12.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:200a1976:::benchmark_data_smithi188_89276_object47250:head# 2024-04-23T16:25:12.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:200a7783:::benchmark_data_smithi188_89276_object56969:head# 2024-04-23T16:25:12.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:200b3122:::benchmark_data_smithi188_89276_object149545:head# 2024-04-23T16:25:12.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:200b5764:::benchmark_data_smithi188_89276_object147577:head# 2024-04-23T16:25:12.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:200be0b6:::benchmark_data_smithi188_89276_object112363:head# 2024-04-23T16:25:12.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:200ca6ba:::benchmark_data_smithi188_89276_object70690:head# 2024-04-23T16:25:12.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:200d763b:::benchmark_data_smithi188_89276_object163144:head# 2024-04-23T16:25:12.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:200d815f:::benchmark_data_smithi188_89276_object42404:head# 2024-04-23T16:25:12.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:200dc711:::benchmark_data_smithi188_89276_object159854:head# 2024-04-23T16:25:12.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:200eb5c0:::benchmark_data_smithi188_89276_object142151:head# 2024-04-23T16:25:12.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:200ed284:::benchmark_data_smithi188_89276_object106706:head# 2024-04-23T16:25:12.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:200fed30:::benchmark_data_smithi188_89276_object56574:head# 2024-04-23T16:25:12.287 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:201091a5:::benchmark_data_smithi188_89276_object121737:head# 2024-04-23T16:25:12.287 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20115e15:::benchmark_data_smithi188_89276_object86448:head# 2024-04-23T16:25:12.287 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20129071:::benchmark_data_smithi188_89276_object59619:head# 2024-04-23T16:25:12.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2012bb75:::benchmark_data_smithi188_89276_object98761:head# 2024-04-23T16:25:12.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2012c83f:::benchmark_data_smithi188_89276_object44124:head# 2024-04-23T16:25:12.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20142c56:::benchmark_data_smithi188_89276_object70844:head# 2024-04-23T16:25:12.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20156ced:::benchmark_data_smithi188_89276_object166454:head# 2024-04-23T16:25:12.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2015ba28:::benchmark_data_smithi188_89276_object111393:head# 2024-04-23T16:25:12.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2015bf70:::benchmark_data_smithi188_89276_object69293:head# 2024-04-23T16:25:12.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:201613c2:::benchmark_data_smithi188_89276_object45556:head# 2024-04-23T16:25:12.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2016d72d:::benchmark_data_smithi188_89276_object128580:head# 2024-04-23T16:25:12.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2016fc3e:::benchmark_data_smithi188_89276_object137187:head# 2024-04-23T16:25:12.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20176e94:::benchmark_data_smithi188_89276_object151654:head# 2024-04-23T16:25:12.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20185070:::benchmark_data_smithi188_89276_object47264:head# 2024-04-23T16:25:12.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20185590:::benchmark_data_smithi188_89276_object142096:head# 2024-04-23T16:25:12.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20192450:::benchmark_data_smithi188_89276_object89878:head# 2024-04-23T16:25:12.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:201bf592:::benchmark_data_smithi188_89276_object98037:head# 2024-04-23T16:25:12.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:201d4625:::benchmark_data_smithi188_89276_object140977:head# 2024-04-23T16:25:12.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:201d7f7a:::benchmark_data_smithi188_89276_object46781:head# 2024-04-23T16:25:12.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:201dce51:::benchmark_data_smithi188_89276_object82357:head# 2024-04-23T16:25:12.295 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:201dee1d:::benchmark_data_smithi188_89276_object144816:head# 2024-04-23T16:25:12.295 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:201f44c5:::benchmark_data_smithi188_89276_object108351:head# 2024-04-23T16:25:12.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:201f7d22:::benchmark_data_smithi188_89276_object109760:head# 2024-04-23T16:25:12.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:201fa8ce:::benchmark_data_smithi188_89276_object95518:head# 2024-04-23T16:25:12.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2020c452:::benchmark_data_smithi188_89276_object111596:head# 2024-04-23T16:25:12.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2020da4a:::benchmark_data_smithi188_89276_object107127:head# 2024-04-23T16:25:12.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20220ce8:::benchmark_data_smithi188_89276_object125616:head# 2024-04-23T16:25:12.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:202256a4:::benchmark_data_smithi188_89276_object91553:head# 2024-04-23T16:25:12.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:202275d9:::benchmark_data_smithi188_89276_object72134:head# 2024-04-23T16:25:12.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2022da58:::benchmark_data_smithi188_89276_object125113:head# 2024-04-23T16:25:12.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20236537:::benchmark_data_smithi188_89276_object121594:head# 2024-04-23T16:25:12.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2023e6be:::benchmark_data_smithi188_89276_object149598:head# 2024-04-23T16:25:12.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2024829b:::benchmark_data_smithi188_89276_object71062:head# 2024-04-23T16:25:12.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20249b4b:::benchmark_data_smithi188_89276_object61513:head# 2024-04-23T16:25:12.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2024c96e:::benchmark_data_smithi188_89276_object48293:head# 2024-04-23T16:25:12.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20251c80:::benchmark_data_smithi188_89276_object92867:head# 2024-04-23T16:25:12.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20253689:::benchmark_data_smithi188_89276_object117130:head# 2024-04-23T16:25:12.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20255acf:::benchmark_data_smithi188_89276_object69357:head# 2024-04-23T16:25:12.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:202721cd:::benchmark_data_smithi188_89276_object99155:head# 2024-04-23T16:25:12.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2028a971:::benchmark_data_smithi188_89276_object126238:head# 2024-04-23T16:25:12.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:202938ed:::benchmark_data_smithi188_89276_object136303:head# 2024-04-23T16:25:12.304 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:12.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20296a45:::benchmark_data_smithi188_89276_object110561:head# 2024-04-23T16:25:12.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2029bf05:::benchmark_data_smithi188_89276_object100372:head# 2024-04-23T16:25:12.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:202a4dc2:::benchmark_data_smithi188_89276_object96846:head# 2024-04-23T16:25:12.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:202aee2f:::benchmark_data_smithi188_89276_object44574:head# 2024-04-23T16:25:12.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:202bde49:::benchmark_data_smithi188_89276_object62369:head# 2024-04-23T16:25:12.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:202c1bc9:::benchmark_data_smithi188_89276_object89049:head# 2024-04-23T16:25:12.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:202cb0ba:::benchmark_data_smithi188_89276_object144137:head# 2024-04-23T16:25:12.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:202cc908:::benchmark_data_smithi188_89276_object163998:head# 2024-04-23T16:25:12.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:202d5615:::benchmark_data_smithi188_89276_object159202:head# 2024-04-23T16:25:12.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:202d8c5b:::benchmark_data_smithi188_89276_object45867:head# 2024-04-23T16:25:12.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20310dc1:::benchmark_data_smithi188_89276_object90901:head# 2024-04-23T16:25:12.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2031d274:::benchmark_data_smithi188_89276_object104815:head# 2024-04-23T16:25:12.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2032d748:::benchmark_data_smithi188_89276_object115105:head# 2024-04-23T16:25:12.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2033c3ed:::benchmark_data_smithi188_89276_object96800:head# 2024-04-23T16:25:12.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2034396b:::benchmark_data_smithi188_89276_object123670:head# 2024-04-23T16:25:12.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20343a92:::benchmark_data_smithi188_89276_object132623:head# 2024-04-23T16:25:12.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2034d2fd:::benchmark_data_smithi188_89276_object91487:head# 2024-04-23T16:25:12.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2034e2f6:::benchmark_data_smithi188_89276_object153364:head# 2024-04-23T16:25:12.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2034f317:::benchmark_data_smithi188_89276_object89571:head# 2024-04-23T16:25:12.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2034f803:::benchmark_data_smithi188_89276_object50065:head# 2024-04-23T16:25:12.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20367792:::benchmark_data_smithi188_89276_object65109:head# 2024-04-23T16:25:12.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2036c55f:::benchmark_data_smithi188_89276_object65976:head# 2024-04-23T16:25:12.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20377400:::benchmark_data_smithi188_89276_object158221:head# 2024-04-23T16:25:12.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:203880ab:::benchmark_data_smithi188_89276_object129101:head# 2024-04-23T16:25:12.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20395b9f:::benchmark_data_smithi188_89276_object163905:head# 2024-04-23T16:25:12.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20398c96:::benchmark_data_smithi188_89276_object101004:head# 2024-04-23T16:25:12.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:203a6812:::benchmark_data_smithi188_89276_object127991:head# 2024-04-23T16:25:12.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:203aa6b4:::benchmark_data_smithi188_89276_object96303:head# 2024-04-23T16:25:12.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:203bbab8:::benchmark_data_smithi188_89276_object95311:head# 2024-04-23T16:25:12.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:203bf7db:::benchmark_data_smithi188_89276_object64774:head# 2024-04-23T16:25:12.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:203c1d25:::benchmark_data_smithi188_89276_object76686:head# 2024-04-23T16:25:12.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:203d02fc:::benchmark_data_smithi188_89276_object43912:head# 2024-04-23T16:25:12.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:203d2ccf:::benchmark_data_smithi188_89276_object136533:head# 2024-04-23T16:25:12.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:203d9e81:::benchmark_data_smithi188_89276_object136778:head# 2024-04-23T16:25:12.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:203e340f:::benchmark_data_smithi188_89276_object154318:head# 2024-04-23T16:25:12.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:203ed244:::benchmark_data_smithi188_89276_object158897:head# 2024-04-23T16:25:12.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:203efe0f:::benchmark_data_smithi188_89276_object136565:head# 2024-04-23T16:25:12.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:203f6945:::benchmark_data_smithi188_89276_object144058:head# 2024-04-23T16:25:12.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:203f94d0:::benchmark_data_smithi188_89276_object133348:head# 2024-04-23T16:25:12.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:203fd7e8:::benchmark_data_smithi188_89276_object143224:head# 2024-04-23T16:25:12.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20401769:::benchmark_data_smithi188_89276_object90946:head# 2024-04-23T16:25:12.326 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:25:12.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:204019fc:::benchmark_data_smithi188_89276_object67298:head# 2024-04-23T16:25:12.328 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2042be71:::benchmark_data_smithi188_89276_object140326:head# 2024-04-23T16:25:12.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:12.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:25:12.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2042c1ec:::benchmark_data_smithi188_89276_object105993:head# 2024-04-23T16:25:12.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2042ffdb:::benchmark_data_smithi188_89276_object105649:head# 2024-04-23T16:25:12.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20430017:::benchmark_data_smithi188_89276_object79840:head# 2024-04-23T16:25:12.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2043b0fe:::benchmark_data_smithi188_89276_object132113:head# 2024-04-23T16:25:12.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20440807:::benchmark_data_smithi188_89276_object116543:head# 2024-04-23T16:25:12.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20444a96:::benchmark_data_smithi188_89276_object125037:head# 2024-04-23T16:25:12.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20449b12:::benchmark_data_smithi188_89276_object165623:head# 2024-04-23T16:25:12.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2045a2e3:::benchmark_data_smithi188_89276_object116344:head# 2024-04-23T16:25:12.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2045eaec:::benchmark_data_smithi188_89276_object63478:head# 2024-04-23T16:25:12.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20465bbb:::benchmark_data_smithi188_89276_object112833:head# 2024-04-23T16:25:12.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2046ba9f:::benchmark_data_smithi188_89276_object43329:head# 2024-04-23T16:25:12.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2046e477:::benchmark_data_smithi188_89276_object133855:head# 2024-04-23T16:25:12.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:204837c1:::benchmark_data_smithi188_89276_object65587:head# 2024-04-23T16:25:12.334 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20488fcf:::benchmark_data_smithi188_89276_object50412:head# 2024-04-23T16:25:12.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:204892b0:::benchmark_data_smithi188_89276_object162839:head# 2024-04-23T16:25:12.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20490f84:::benchmark_data_smithi188_89276_object55799:head# 2024-04-23T16:25:12.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20492552:::benchmark_data_smithi188_89276_object64555:head# 2024-04-23T16:25:12.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:204c0ed7:::benchmark_data_smithi188_89276_object82345:head# 2024-04-23T16:25:12.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:204c1abd:::benchmark_data_smithi188_89276_object131971:head# 2024-04-23T16:25:12.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:204c382f:::benchmark_data_smithi188_89276_object49152:head# 2024-04-23T16:25:12.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:204d27cf:::benchmark_data_smithi188_89276_object55866:head# 2024-04-23T16:25:12.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:204e236a:::benchmark_data_smithi188_89276_object70809:head# 2024-04-23T16:25:12.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:204e706b:::benchmark_data_smithi188_89276_object153400:head# 2024-04-23T16:25:12.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:204e868c:::benchmark_data_smithi188_89276_object123607:head# 2024-04-23T16:25:12.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:204f8215:::benchmark_data_smithi188_89276_object98450:head# 2024-04-23T16:25:12.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:204f9b78:::benchmark_data_smithi188_89276_object110247:head# 2024-04-23T16:25:12.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:204fa8c6:::benchmark_data_smithi188_89276_object101048:head# 2024-04-23T16:25:12.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:205080cb:::benchmark_data_smithi188_89276_object51753:head# 2024-04-23T16:25:12.341 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2050ba96:::benchmark_data_smithi188_89276_object146726:head# 2024-04-23T16:25:12.341 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2051d98a:::benchmark_data_smithi188_89276_object124779:head# 2024-04-23T16:25:12.341 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2051e294:::benchmark_data_smithi188_89276_object118156:head# 2024-04-23T16:25:12.342 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20521175:::benchmark_data_smithi188_89276_object102939:head# 2024-04-23T16:25:12.342 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2052739d:::benchmark_data_smithi188_89276_object142208:head# 2024-04-23T16:25:12.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:205284b6:::benchmark_data_smithi188_89276_object166878:head# 2024-04-23T16:25:12.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2052c637:::benchmark_data_smithi188_89276_object79725:head# 2024-04-23T16:25:12.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2052f05a:::benchmark_data_smithi188_89276_object87850:head# 2024-04-23T16:25:12.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20531402:::benchmark_data_smithi188_89276_object85374:head# 2024-04-23T16:25:12.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20539538:::benchmark_data_smithi188_89276_object123509:head# 2024-04-23T16:25:12.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2053aa95:::benchmark_data_smithi188_89276_object130013:head# 2024-04-23T16:25:12.346 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2053e56b:::benchmark_data_smithi188_89276_object146183:head# 2024-04-23T16:25:12.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20544816:::benchmark_data_smithi188_89276_object87229:head# 2024-04-23T16:25:12.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:205497b4:::benchmark_data_smithi188_89276_object102570:head# 2024-04-23T16:25:12.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2054de37:::benchmark_data_smithi188_89276_object71835:head# 2024-04-23T16:25:12.348 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2054f6fb:::benchmark_data_smithi188_89276_object50142:head# 2024-04-23T16:25:12.348 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:205542d0:::benchmark_data_smithi188_89276_object167993:head# 2024-04-23T16:25:12.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2055987b:::benchmark_data_smithi188_89276_object109946:head# 2024-04-23T16:25:12.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2055a27e:::benchmark_data_smithi188_89276_object145910:head# 2024-04-23T16:25:12.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2055ee5e:::benchmark_data_smithi188_89276_object49225:head# 2024-04-23T16:25:12.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20563cd2:::benchmark_data_smithi188_89276_object111421:head# 2024-04-23T16:25:12.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2057502d:::benchmark_data_smithi188_89276_object162438:head# 2024-04-23T16:25:12.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20579485:::benchmark_data_smithi188_89276_object72455:head# 2024-04-23T16:25:12.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2058e11e:::benchmark_data_smithi188_89276_object46559:head# 2024-04-23T16:25:12.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:205aa197:::benchmark_data_smithi188_89276_object100662:head# 2024-04-23T16:25:12.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:205aa1c2:::benchmark_data_smithi188_89276_object69902:head# 2024-04-23T16:25:12.353 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:205aac8a:::benchmark_data_smithi188_89276_object96493:head# 2024-04-23T16:25:12.353 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:205ad0a3:::benchmark_data_smithi188_89276_object42998:head# 2024-04-23T16:25:12.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:205bb2a7:::benchmark_data_smithi188_89276_object88057:head# 2024-04-23T16:25:12.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:205ccf33:::benchmark_data_smithi188_89276_object121399:head# 2024-04-23T16:25:12.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:205d3bc4:::benchmark_data_smithi188_89276_object97540:head# 2024-04-23T16:25:12.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:205d6815:::benchmark_data_smithi188_89276_object144096:head# 2024-04-23T16:25:12.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:205d967c:::benchmark_data_smithi188_89276_object139786:head# 2024-04-23T16:25:12.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:205dd73d:::benchmark_data_smithi188_89276_object136640:head# 2024-04-23T16:25:12.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:205eca1a:::benchmark_data_smithi188_89276_object55622:head# 2024-04-23T16:25:12.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:205ecedd:::benchmark_data_smithi188_89276_object164588:head# 2024-04-23T16:25:12.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:205ef0c8:::benchmark_data_smithi188_89276_object93847:head# 2024-04-23T16:25:12.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:205ef5c9:::benchmark_data_smithi188_89276_object85352:head# 2024-04-23T16:25:12.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:205f31d1:::benchmark_data_smithi188_89276_object162315:head# 2024-04-23T16:25:12.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20604d1f:::benchmark_data_smithi188_89276_object114493:head# 2024-04-23T16:25:12.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20608922:::benchmark_data_smithi188_89276_object168292:head# 2024-04-23T16:25:12.361 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2060d6f6:::benchmark_data_smithi188_89276_object162184:head# 2024-04-23T16:25:12.361 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20612186:::benchmark_data_smithi188_89276_object86508:head# 2024-04-23T16:25:12.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:206172b8:::benchmark_data_smithi188_89276_object57158:head# 2024-04-23T16:25:12.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2061dfef:::benchmark_data_smithi188_89276_object103644:head# 2024-04-23T16:25:12.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2061f79e:::benchmark_data_smithi188_89276_object89424:head# 2024-04-23T16:25:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20626b01:::benchmark_data_smithi188_89276_object82448:head# 2024-04-23T16:25:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2062900c:::benchmark_data_smithi188_89276_object159925:head# 2024-04-23T16:25:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2063cf29:::benchmark_data_smithi188_89276_object78595:head# 2024-04-23T16:25:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2063f6b2:::benchmark_data_smithi188_89276_object89550:head# 2024-04-23T16:25:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2064085b:::benchmark_data_smithi188_89276_object58529:head# 2024-04-23T16:25:12.366 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:206411de:::benchmark_data_smithi188_89276_object63261:head# 2024-04-23T16:25:12.367 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20641e6a:::benchmark_data_smithi188_89276_object52086:head# 2024-04-23T16:25:12.367 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2064cc79:::benchmark_data_smithi188_89276_object69408:head# 2024-04-23T16:25:12.368 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:206626ef:::benchmark_data_smithi188_89276_object120804:head# 2024-04-23T16:25:12.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2066a3cb:::benchmark_data_smithi188_89276_object52348:head# 2024-04-23T16:25:12.371 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20674846:::benchmark_data_smithi188_89276_object134035:head# 2024-04-23T16:25:12.372 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2068441c:::benchmark_data_smithi188_89276_object74869:head# 2024-04-23T16:25:12.372 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2069b528:::benchmark_data_smithi188_89276_object71258:head# 2024-04-23T16:25:12.373 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2069bada:::benchmark_data_smithi188_89276_object84064:head# 2024-04-23T16:25:12.373 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2069bb89:::benchmark_data_smithi188_89276_object133211:head# 2024-04-23T16:25:12.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:206ac46f:::benchmark_data_smithi188_89276_object74101:head# 2024-04-23T16:25:12.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:206af9fc:::benchmark_data_smithi188_89276_object87321:head# 2024-04-23T16:25:12.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:206dc280:::benchmark_data_smithi188_89276_object153649:head# 2024-04-23T16:25:12.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:206dce15:::benchmark_data_smithi188_89276_object77560:head# 2024-04-23T16:25:12.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:206e2d1f:::benchmark_data_smithi188_89276_object162033:head# 2024-04-23T16:25:12.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:206f2d00:::benchmark_data_smithi188_89276_object162875:head# 2024-04-23T16:25:12.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:206fdcf7:::benchmark_data_smithi188_89276_object119804:head# 2024-04-23T16:25:12.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:207085b2:::benchmark_data_smithi188_89276_object130826:head# 2024-04-23T16:25:12.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2070a0cd:::benchmark_data_smithi188_89276_object149712:head# 2024-04-23T16:25:12.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2070c009:::benchmark_data_smithi188_89276_object98453:head# 2024-04-23T16:25:12.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20739866:::benchmark_data_smithi188_89276_object114754:head# 2024-04-23T16:25:12.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:207438bc:::benchmark_data_smithi188_89276_object41211:head# 2024-04-23T16:25:12.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2074e3c3:::benchmark_data_smithi188_89276_object86797:head# 2024-04-23T16:25:12.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2074f1b3:::benchmark_data_smithi188_89276_object152142:head# 2024-04-23T16:25:12.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20750d89:::benchmark_data_smithi188_89276_object151118:head# 2024-04-23T16:25:12.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:207595b9:::benchmark_data_smithi188_89276_object164097:head# 2024-04-23T16:25:12.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2075ceac:::benchmark_data_smithi188_89276_object57611:head# 2024-04-23T16:25:12.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2075f5f5:::benchmark_data_smithi188_89276_object51970:head# 2024-04-23T16:25:12.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2076638a:::benchmark_data_smithi188_89276_object150061:head# 2024-04-23T16:25:12.389 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20767820:::benchmark_data_smithi188_89276_object160171:head# 2024-04-23T16:25:12.389 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20768281:::benchmark_data_smithi188_89276_object52716:head# 2024-04-23T16:25:12.389 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20774f7f:::benchmark_data_smithi188_89276_object89503:head# 2024-04-23T16:25:12.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20780a63:::benchmark_data_smithi188_89276_object66011:head# 2024-04-23T16:25:12.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20785606:::benchmark_data_smithi188_89276_object105485:head# 2024-04-23T16:25:12.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20785b58:::benchmark_data_smithi188_89276_object151996:head# 2024-04-23T16:25:12.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2078cc49:::benchmark_data_smithi188_89276_object138916:head# 2024-04-23T16:25:12.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2078d7df:::benchmark_data_smithi188_89276_object166669:head# 2024-04-23T16:25:12.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2078eaea:::benchmark_data_smithi188_89276_object161741:head# 2024-04-23T16:25:12.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:207a37ea:::benchmark_data_smithi188_89276_object132724:head# 2024-04-23T16:25:12.391 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:207a4f38:::benchmark_data_smithi188_89276_object106455:head# 2024-04-23T16:25:12.391 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:207af32a:::benchmark_data_smithi188_89276_object93680:head# 2024-04-23T16:25:12.391 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:207b116f:::benchmark_data_smithi188_89276_object68196:head# 2024-04-23T16:25:12.391 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:207ba60f:::benchmark_data_smithi188_89276_object74171:head# 2024-04-23T16:25:12.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:207bff91:::benchmark_data_smithi188_89276_object146530:head# 2024-04-23T16:25:12.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:207c0177:::benchmark_data_smithi188_89276_object149431:head# 2024-04-23T16:25:12.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:207c6d8f:::benchmark_data_smithi188_89276_object51998:head# 2024-04-23T16:25:12.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:207d1624:::benchmark_data_smithi188_89276_object153646:head# 2024-04-23T16:25:12.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:207d4dde:::benchmark_data_smithi188_89276_object120074:head# 2024-04-23T16:25:12.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:207d5dfa:::benchmark_data_smithi188_89276_object166528:head# 2024-04-23T16:25:12.395 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:207d9862:::benchmark_data_smithi188_89276_object53330:head# 2024-04-23T16:25:12.395 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:207df6c2:::benchmark_data_smithi188_89276_object133642:head# 2024-04-23T16:25:12.395 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:207e8bbf:::benchmark_data_smithi188_89276_object153930:head# 2024-04-23T16:25:12.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:207ee6c4:::benchmark_data_smithi188_89276_object166394:head# 2024-04-23T16:25:12.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:207fcfdc:::benchmark_data_smithi188_89276_object69873:head# 2024-04-23T16:25:12.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20807dee:::benchmark_data_smithi188_89276_object130004:head# 2024-04-23T16:25:12.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2080e360:::benchmark_data_smithi188_89276_object113680:head# 2024-04-23T16:25:12.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20818809:::benchmark_data_smithi188_89276_object65752:head# 2024-04-23T16:25:12.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20819f79:::benchmark_data_smithi188_89276_object144789:head# 2024-04-23T16:25:12.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20828a17:::benchmark_data_smithi188_89276_object72038:head# 2024-04-23T16:25:12.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20830b52:::benchmark_data_smithi188_89276_object89013:head# 2024-04-23T16:25:12.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20837ed8:::benchmark_data_smithi188_89276_object78648:head# 2024-04-23T16:25:12.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2083c3a3:::benchmark_data_smithi188_89276_object60392:head# 2024-04-23T16:25:12.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2084aade:::benchmark_data_smithi188_89276_object113627:head# 2024-04-23T16:25:12.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2084c117:::benchmark_data_smithi188_89276_object165362:head# 2024-04-23T16:25:12.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2084fccb:::benchmark_data_smithi188_89276_object77166:head# 2024-04-23T16:25:12.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20853edd:::benchmark_data_smithi188_89276_object136727:head# 2024-04-23T16:25:12.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20855dc4:::benchmark_data_smithi188_89276_object41666:head# 2024-04-23T16:25:12.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20856796:::benchmark_data_smithi188_89276_object99245:head# 2024-04-23T16:25:12.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2086dc80:::benchmark_data_smithi188_89276_object147142:head# 2024-04-23T16:25:12.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2087ac8d:::benchmark_data_smithi188_89276_object75266:head# 2024-04-23T16:25:12.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2087f919:::benchmark_data_smithi188_89276_object45251:head# 2024-04-23T16:25:12.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20884915:::benchmark_data_smithi188_89276_object105111:head# 2024-04-23T16:25:12.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2088ffc6:::benchmark_data_smithi188_89276_object91679:head# 2024-04-23T16:25:12.412 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:208aa8d1:::benchmark_data_smithi188_89276_object147471:head# 2024-04-23T16:25:12.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:208aea33:::benchmark_data_smithi188_89276_object69874:head# 2024-04-23T16:25:12.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:208af4ef:::benchmark_data_smithi188_89276_object92171:head# 2024-04-23T16:25:12.414 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:208afac1:::benchmark_data_smithi188_89276_object95967:head# 2024-04-23T16:25:12.414 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:208b72ff:::benchmark_data_smithi188_89276_object149877:head# 2024-04-23T16:25:12.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:208ccfae:::benchmark_data_smithi188_89276_object40768:head# 2024-04-23T16:25:12.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:208ce45b:::benchmark_data_smithi188_89276_object133832:head# 2024-04-23T16:25:12.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:208cfa2a:::benchmark_data_smithi188_89276_object54310:head# 2024-04-23T16:25:12.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:208db075:::benchmark_data_smithi188_89276_object155753:head# 2024-04-23T16:25:12.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:208e7c68:::benchmark_data_smithi188_89276_object96944:head# 2024-04-23T16:25:12.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:208f6c2e:::benchmark_data_smithi188_89276_object145171:head# 2024-04-23T16:25:12.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2090f4d9:::benchmark_data_smithi188_89276_object130926:head# 2024-04-23T16:25:12.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209149bb:::benchmark_data_smithi188_89276_object67374:head# 2024-04-23T16:25:12.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209152a5:::benchmark_data_smithi188_89276_object103024:head# 2024-04-23T16:25:12.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209184a3:::benchmark_data_smithi188_89276_object121945:head# 2024-04-23T16:25:12.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2091a96c:::benchmark_data_smithi188_89276_object96879:head# 2024-04-23T16:25:12.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20928766:::benchmark_data_smithi188_89276_object100194:head# 2024-04-23T16:25:12.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2092f6b4:::benchmark_data_smithi188_89276_object136086:head# 2024-04-23T16:25:12.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20932b32:::benchmark_data_smithi188_89276_object54478:head# 2024-04-23T16:25:12.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20934b28:::benchmark_data_smithi188_89276_object156365:head# 2024-04-23T16:25:12.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2093764e:::benchmark_data_smithi188_89276_object146205:head# 2024-04-23T16:25:12.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20939feb:::benchmark_data_smithi188_89276_object79867:head# 2024-04-23T16:25:12.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20945256:::benchmark_data_smithi188_89276_object103121:head# 2024-04-23T16:25:12.426 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2094cd70:::benchmark_data_smithi188_89276_object156212:head# 2024-04-23T16:25:12.426 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20951808:::benchmark_data_smithi188_89276_object149859:head# 2024-04-23T16:25:12.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20955680:::benchmark_data_smithi188_89276_object61753:head# 2024-04-23T16:25:12.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209559d3:::benchmark_data_smithi188_89276_object161351:head# 2024-04-23T16:25:12.429 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:25:12.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209564a9:::benchmark_data_smithi188_89276_object74357:head# 2024-04-23T16:25:12.431 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209571c6:::benchmark_data_smithi188_89276_object115810:head# 2024-04-23T16:25:12.431 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2095cd91:::benchmark_data_smithi188_89276_object154265:head# 2024-04-23T16:25:12.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20960bd8:::benchmark_data_smithi188_89276_object168299:head# 2024-04-23T16:25:12.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20961037:::benchmark_data_smithi188_89276_object94883:head# 2024-04-23T16:25:12.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209779ef:::benchmark_data_smithi188_89276_object137093:head# 2024-04-23T16:25:12.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2097a2e6:::benchmark_data_smithi188_89276_object136366:head# 2024-04-23T16:25:12.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2097efc3:::benchmark_data_smithi188_89276_object168597:head# 2024-04-23T16:25:12.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20987dfc:::benchmark_data_smithi188_89276_object133959:head# 2024-04-23T16:25:12.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209966a0:::benchmark_data_smithi188_89276_object119477:head# 2024-04-23T16:25:12.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2099c89d:::benchmark_data_smithi188_89276_object45481:head# 2024-04-23T16:25:12.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2099e999:::benchmark_data_smithi188_89276_object152876:head# 2024-04-23T16:25:12.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209a17b8:::benchmark_data_smithi188_89276_object87191:head# 2024-04-23T16:25:12.438 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209a20a7:::benchmark_data_smithi188_89276_object63984:head# 2024-04-23T16:25:12.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209a46b4:::benchmark_data_smithi188_89276_object57168:head# 2024-04-23T16:25:12.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209ad30a:::benchmark_data_smithi188_89276_object43296:head# 2024-04-23T16:25:12.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209bf04a:::benchmark_data_smithi188_89276_object94854:head# 2024-04-23T16:25:12.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209d118c:::benchmark_data_smithi188_89276_object160533:head# 2024-04-23T16:25:12.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209d72d5:::benchmark_data_smithi188_89276_object121356:head# 2024-04-23T16:25:12.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209d7953:::benchmark_data_smithi188_89276_object101072:head# 2024-04-23T16:25:12.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209d9848:::benchmark_data_smithi188_89276_object123100:head# 2024-04-23T16:25:12.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209db7fb:::benchmark_data_smithi188_89276_object100830:head# 2024-04-23T16:25:12.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209e6103:::benchmark_data_smithi188_89276_object89764:head# 2024-04-23T16:25:12.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209ebde4:::benchmark_data_smithi188_89276_object90496:head# 2024-04-23T16:25:12.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:209f2613:::benchmark_data_smithi188_89276_object152176:head# 2024-04-23T16:25:12.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20a0242d:::benchmark_data_smithi188_89276_object152412:head# 2024-04-23T16:25:12.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20a157a1:::benchmark_data_smithi188_89276_object166508:head# 2024-04-23T16:25:12.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20a20298:::benchmark_data_smithi188_89276_object149793:head# 2024-04-23T16:25:12.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20a26073:::benchmark_data_smithi188_89276_object129489:head# 2024-04-23T16:25:12.446 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20a2ca2a:::benchmark_data_smithi188_89276_object90462:head# 2024-04-23T16:25:12.446 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20a3632f:::benchmark_data_smithi188_89276_object100612:head# 2024-04-23T16:25:12.449 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20a50dce:::benchmark_data_smithi188_89276_object92401:head# 2024-04-23T16:25:12.449 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20a58cbe:::benchmark_data_smithi188_89276_object143933:head# 2024-04-23T16:25:12.449 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20a60bbd:::benchmark_data_smithi188_89276_object90363:head# 2024-04-23T16:25:12.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20a64627:::benchmark_data_smithi188_89276_object115795:head# 2024-04-23T16:25:12.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20a66205:::benchmark_data_smithi188_89276_object156225:head# 2024-04-23T16:25:12.451 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20a67b1c:::benchmark_data_smithi188_89276_object79728:head# 2024-04-23T16:25:12.451 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20a764e0:::benchmark_data_smithi188_89276_object132896:head# 2024-04-23T16:25:12.452 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20a77900:::benchmark_data_smithi188_89276_object116863:head# 2024-04-23T16:25:12.452 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20a81dfb:::benchmark_data_smithi188_89276_object50788:head# 2024-04-23T16:25:12.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20a922dc:::benchmark_data_smithi188_89276_object161405:head# 2024-04-23T16:25:12.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20a94851:::benchmark_data_smithi188_89276_object134926:head# 2024-04-23T16:25:12.454 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20a9e980:::benchmark_data_smithi188_89276_object98907:head# 2024-04-23T16:25:12.454 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ab6bc7:::benchmark_data_smithi188_89276_object133210:head# 2024-04-23T16:25:12.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ab8d77:::benchmark_data_smithi188_89276_object160212:head# 2024-04-23T16:25:12.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20abc2fd:::benchmark_data_smithi188_89276_object142346:head# 2024-04-23T16:25:12.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ac2255:::benchmark_data_smithi188_89276_object128446:head# 2024-04-23T16:25:12.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ad1cdc:::benchmark_data_smithi188_89276_object82729:head# 2024-04-23T16:25:12.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20adc052:::benchmark_data_smithi188_89276_object92307:head# 2024-04-23T16:25:12.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20aee543:::benchmark_data_smithi188_89276_object66823:head# 2024-04-23T16:25:12.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20b04c9b:::benchmark_data_smithi188_89276_object126801:head# 2024-04-23T16:25:12.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20b19f62:::benchmark_data_smithi188_89276_object118502:head# 2024-04-23T16:25:12.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20b2632f:::benchmark_data_smithi188_89276_object102850:head# 2024-04-23T16:25:12.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20b366ca:::benchmark_data_smithi188_89276_object129732:head# 2024-04-23T16:25:12.462 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20b60981:::benchmark_data_smithi188_89276_object81526:head# 2024-04-23T16:25:12.462 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20b6eb07:::benchmark_data_smithi188_89276_object164162:head# 2024-04-23T16:25:12.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20b6f822:::benchmark_data_smithi188_89276_object88823:head# 2024-04-23T16:25:12.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20b742a9:::benchmark_data_smithi188_89276_object134565:head# 2024-04-23T16:25:12.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20b7c849:::benchmark_data_smithi188_89276_object114759:head# 2024-04-23T16:25:12.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20b7e620:::benchmark_data_smithi188_89276_object98041:head# 2024-04-23T16:25:12.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20b84085:::benchmark_data_smithi188_89276_object58175:head# 2024-04-23T16:25:12.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20b97dfc:::benchmark_data_smithi188_89276_object144828:head# 2024-04-23T16:25:12.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ba614c:::benchmark_data_smithi188_89276_object138442:head# 2024-04-23T16:25:12.470 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:12.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ba900e:::benchmark_data_smithi188_89276_object102029:head# 2024-04-23T16:25:12.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20bab020:::benchmark_data_smithi188_89276_object145578:head# 2024-04-23T16:25:12.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20baefb4:::benchmark_data_smithi188_89276_object152649:head# 2024-04-23T16:25:12.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20bc9752:::benchmark_data_smithi188_89276_object163354:head# 2024-04-23T16:25:12.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20bcf7bd:::benchmark_data_smithi188_89276_object95046:head# 2024-04-23T16:25:12.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20bdfea3:::benchmark_data_smithi188_89276_object145113:head# 2024-04-23T16:25:12.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20be0175:::benchmark_data_smithi188_89276_object42450:head# 2024-04-23T16:25:12.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20be4076:::benchmark_data_smithi188_89276_object167671:head# 2024-04-23T16:25:12.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20be58ce:::benchmark_data_smithi188_89276_object56965:head# 2024-04-23T16:25:12.474 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20be788e:::benchmark_data_smithi188_89276_object148199:head# 2024-04-23T16:25:12.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20bef2c2:::benchmark_data_smithi188_89276_object129622:head# 2024-04-23T16:25:12.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20befc1b:::benchmark_data_smithi188_89276_object59290:head# 2024-04-23T16:25:12.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20bf8dba:::benchmark_data_smithi188_89276_object131110:head# 2024-04-23T16:25:12.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20bfe72a:::benchmark_data_smithi188_89276_object94445:head# 2024-04-23T16:25:12.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c079ef:::benchmark_data_smithi188_89276_object124577:head# 2024-04-23T16:25:12.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c0cada:::benchmark_data_smithi188_89276_object98536:head# 2024-04-23T16:25:12.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c0cde5:::benchmark_data_smithi188_89276_object84407:head# 2024-04-23T16:25:12.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c1d434:::benchmark_data_smithi188_89276_object64515:head# 2024-04-23T16:25:12.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c2087d:::benchmark_data_smithi188_89276_object164245:head# 2024-04-23T16:25:12.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c2288c:::benchmark_data_smithi188_89276_object64759:head# 2024-04-23T16:25:12.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c28792:::benchmark_data_smithi188_89276_object125138:head# 2024-04-23T16:25:12.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c2e978:::benchmark_data_smithi188_89276_object91141:head# 2024-04-23T16:25:12.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c34b41:::benchmark_data_smithi188_89276_object133387:head# 2024-04-23T16:25:12.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c37685:::benchmark_data_smithi188_89276_object56492:head# 2024-04-23T16:25:12.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c37a57:::benchmark_data_smithi188_89276_object167620:head# 2024-04-23T16:25:12.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c381cf:::benchmark_data_smithi188_89276_object152315:head# 2024-04-23T16:25:12.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c3e9a2:::benchmark_data_smithi188_89276_object79722:head# 2024-04-23T16:25:12.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c467c6:::benchmark_data_smithi188_89276_object108735:head# 2024-04-23T16:25:12.486 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c467e7:::benchmark_data_smithi188_89276_object74429:head# 2024-04-23T16:25:12.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c4d231:::benchmark_data_smithi188_89276_object113218:head# 2024-04-23T16:25:12.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c561eb:::benchmark_data_smithi188_89276_object103247:head# 2024-04-23T16:25:12.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c5f68c:::benchmark_data_smithi188_89276_object103458:head# 2024-04-23T16:25:12.491 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c5fa23:::benchmark_data_smithi188_89276_object134850:head# 2024-04-23T16:25:12.491 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c66877:::benchmark_data_smithi188_89276_object53911:head# 2024-04-23T16:25:12.492 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c710fd:::benchmark_data_smithi188_89276_object42899:head# 2024-04-23T16:25:12.494 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c74041:::benchmark_data_smithi188_89276_object155797:head# 2024-04-23T16:25:12.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c78d63:::benchmark_data_smithi188_89276_object75571:head# 2024-04-23T16:25:12.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c7bfcc:::benchmark_data_smithi188_89276_object156866:head# 2024-04-23T16:25:12.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c85191:::benchmark_data_smithi188_89276_object92425:head# 2024-04-23T16:25:12.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c8827f:::benchmark_data_smithi188_89276_object154179:head# 2024-04-23T16:25:12.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c89138:::benchmark_data_smithi188_89276_object79008:head# 2024-04-23T16:25:12.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c94c00:::benchmark_data_smithi188_89276_object136077:head# 2024-04-23T16:25:12.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20c9e251:::benchmark_data_smithi188_89276_object132889:head# 2024-04-23T16:25:12.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ca8aac:::benchmark_data_smithi188_89276_object48479:head# 2024-04-23T16:25:12.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ca9caa:::benchmark_data_smithi188_89276_object88973:head# 2024-04-23T16:25:12.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20cb131a:::benchmark_data_smithi188_89276_object83651:head# 2024-04-23T16:25:12.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20cb3874:::benchmark_data_smithi188_89276_object92492:head# 2024-04-23T16:25:12.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20cbacde:::benchmark_data_smithi188_89276_object46747:head# 2024-04-23T16:25:12.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20cbc775:::benchmark_data_smithi188_89276_object113105:head# 2024-04-23T16:25:12.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20cbcbc4:::benchmark_data_smithi188_89276_object162612:head# 2024-04-23T16:25:12.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20cc9a35:::benchmark_data_smithi188_89276_object60535:head# 2024-04-23T16:25:12.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20cca821:::benchmark_data_smithi188_89276_object69846:head# 2024-04-23T16:25:12.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20cd0753:::benchmark_data_smithi188_89276_object168039:head# 2024-04-23T16:25:12.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20cd8150:::benchmark_data_smithi188_89276_object111003:head# 2024-04-23T16:25:12.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20cdec29:::benchmark_data_smithi188_89276_object167592:head# 2024-04-23T16:25:12.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20cf55cf:::benchmark_data_smithi188_89276_object151384:head# 2024-04-23T16:25:12.504 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20cf70bc:::benchmark_data_smithi188_89276_object161382:head# 2024-04-23T16:25:12.504 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20d0be5f:::benchmark_data_smithi188_89276_object117201:head# 2024-04-23T16:25:12.504 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20d1667e:::benchmark_data_smithi188_89276_object156788:head# 2024-04-23T16:25:12.505 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:25:12.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20d1d7a5:::benchmark_data_smithi188_89276_object89639:head# 2024-04-23T16:25:12.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20d23465:::benchmark_data_smithi188_89276_object161320:head# 2024-04-23T16:25:12.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20d2b3e2:::benchmark_data_smithi188_89276_object97706:head# 2024-04-23T16:25:12.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20d347b2:::benchmark_data_smithi188_89276_object112499:head# 2024-04-23T16:25:12.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20d516ba:::benchmark_data_smithi188_89276_object88430:head# 2024-04-23T16:25:12.507 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20d546cf:::benchmark_data_smithi188_89276_object42946:head# 2024-04-23T16:25:12.507 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20d613a5:::benchmark_data_smithi188_89276_object60879:head# 2024-04-23T16:25:12.508 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20d62699:::benchmark_data_smithi188_89276_object50516:head# 2024-04-23T16:25:12.508 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20d631a2:::benchmark_data_smithi188_89276_object59702:head# 2024-04-23T16:25:12.509 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20d7165e:::benchmark_data_smithi188_89276_object167886:head# 2024-04-23T16:25:12.509 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20d7456c:::benchmark_data_smithi188_89276_object68190:head# 2024-04-23T16:25:12.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20d850cf:::benchmark_data_smithi188_89276_object157017:head# 2024-04-23T16:25:12.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:12.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:25:12.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20d86bf3:::benchmark_data_smithi188_89276_object156240:head# 2024-04-23T16:25:12.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20d9f848:::benchmark_data_smithi188_89276_object121961:head# 2024-04-23T16:25:12.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20da2016:::benchmark_data_smithi188_89276_object151500:head# 2024-04-23T16:25:12.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20db3056:::benchmark_data_smithi188_89276_object68603:head# 2024-04-23T16:25:12.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20dbfb39:::benchmark_data_smithi188_89276_object112789:head# 2024-04-23T16:25:12.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20dc5399:::benchmark_data_smithi188_89276_object45751:head# 2024-04-23T16:25:12.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20dce1c2:::benchmark_data_smithi188_89276_object112626:head# 2024-04-23T16:25:12.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ddee33:::benchmark_data_smithi188_89276_object119158:head# 2024-04-23T16:25:12.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ddfd02:::benchmark_data_smithi188_89276_object47635:head# 2024-04-23T16:25:12.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20de53df:::benchmark_data_smithi188_89276_object74816:head# 2024-04-23T16:25:12.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20deb9ef:::benchmark_data_smithi188_89276_object101973:head# 2024-04-23T16:25:12.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20deecc4:::benchmark_data_smithi188_89276_object57710:head# 2024-04-23T16:25:12.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20df29ac:::benchmark_data_smithi188_89276_object119445:head# 2024-04-23T16:25:12.516 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20df4ca0:::benchmark_data_smithi188_89276_object114254:head# 2024-04-23T16:25:12.516 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20df6148:::benchmark_data_smithi188_89276_object98624:head# 2024-04-23T16:25:12.516 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20dfe304:::benchmark_data_smithi188_89276_object79246:head# 2024-04-23T16:25:12.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e0b9cf:::benchmark_data_smithi188_89276_object64860:head# 2024-04-23T16:25:12.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e0c10b:::benchmark_data_smithi188_89276_object129632:head# 2024-04-23T16:25:12.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e0c748:::benchmark_data_smithi188_89276_object141437:head# 2024-04-23T16:25:12.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e0fec5:::benchmark_data_smithi188_89276_object50439:head# 2024-04-23T16:25:12.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e12611:::benchmark_data_smithi188_89276_object134207:head# 2024-04-23T16:25:12.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e1c586:::benchmark_data_smithi188_89276_object117754:head# 2024-04-23T16:25:12.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e24c06:::benchmark_data_smithi188_89276_object131452:head# 2024-04-23T16:25:12.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e29442:::benchmark_data_smithi188_89276_object142925:head# 2024-04-23T16:25:12.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e2a98e:::benchmark_data_smithi188_89276_object158518:head# 2024-04-23T16:25:12.521 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e2d294:::benchmark_data_smithi188_89276_object158209:head# 2024-04-23T16:25:12.521 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e3d8dc:::benchmark_data_smithi188_89276_object60323:head# 2024-04-23T16:25:12.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e3dfa3:::benchmark_data_smithi188_89276_object46294:head# 2024-04-23T16:25:12.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e483d0:::benchmark_data_smithi188_89276_object151144:head# 2024-04-23T16:25:12.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e4d03d:::benchmark_data_smithi188_89276_object109776:head# 2024-04-23T16:25:12.523 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e560f4:::benchmark_data_smithi188_89276_object49159:head# 2024-04-23T16:25:12.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e6b5ac:::benchmark_data_smithi188_89276_object96350:head# 2024-04-23T16:25:12.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e6d9ae:::benchmark_data_smithi188_89276_object167215:head# 2024-04-23T16:25:12.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e757b0:::benchmark_data_smithi188_89276_object50588:head# 2024-04-23T16:25:12.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e780f9:::benchmark_data_smithi188_89276_object84748:head# 2024-04-23T16:25:12.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e7929b:::benchmark_data_smithi188_89276_object61784:head# 2024-04-23T16:25:12.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e84ecd:::benchmark_data_smithi188_89276_object77609:head# 2024-04-23T16:25:12.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20e913c9:::benchmark_data_smithi188_89276_object68939:head# 2024-04-23T16:25:12.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ea8a07:::benchmark_data_smithi188_89276_object62305:head# 2024-04-23T16:25:12.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ec109f:::benchmark_data_smithi188_89276_object166213:head# 2024-04-23T16:25:12.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ed2cce:::benchmark_data_smithi188_89276_object51797:head# 2024-04-23T16:25:12.531 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ed45a9:::benchmark_data_smithi188_89276_object114339:head# 2024-04-23T16:25:12.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20eec79a:::benchmark_data_smithi188_89276_object68304:head# 2024-04-23T16:25:12.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ef3cbb:::benchmark_data_smithi188_89276_object62532:head# 2024-04-23T16:25:12.533 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20efdfeb:::benchmark_data_smithi188_89276_object142120:head# 2024-04-23T16:25:12.533 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20f04fd5:::benchmark_data_smithi188_89276_object69345:head# 2024-04-23T16:25:12.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20f131c3:::benchmark_data_smithi188_89276_object78242:head# 2024-04-23T16:25:12.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20f1471e:::benchmark_data_smithi188_89276_object104299:head# 2024-04-23T16:25:12.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20f2017e:::benchmark_data_smithi188_89276_object148001:head# 2024-04-23T16:25:12.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20f225ca:::benchmark_data_smithi188_89276_object116252:head# 2024-04-23T16:25:12.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20f26719:::benchmark_data_smithi188_89276_object69126:head# 2024-04-23T16:25:12.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20f31d6b:::benchmark_data_smithi188_89276_object121025:head# 2024-04-23T16:25:12.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20f419d2:::benchmark_data_smithi188_89276_object150661:head# 2024-04-23T16:25:12.536 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20f60fbc:::benchmark_data_smithi188_89276_object114800:head# 2024-04-23T16:25:12.536 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20f652d6:::benchmark_data_smithi188_89276_object137455:head# 2024-04-23T16:25:12.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20f6ca5f:::benchmark_data_smithi188_89276_object145508:head# 2024-04-23T16:25:12.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20f6ec63:::benchmark_data_smithi188_89276_object132751:head# 2024-04-23T16:25:12.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20f71b8e:::benchmark_data_smithi188_89276_object115224:head# 2024-04-23T16:25:12.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20f78f9f:::benchmark_data_smithi188_89276_object58322:head# 2024-04-23T16:25:12.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20f851ed:::benchmark_data_smithi188_89276_object94244:head# 2024-04-23T16:25:12.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20f899bc:::benchmark_data_smithi188_89276_object47996:head# 2024-04-23T16:25:12.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20f9854f:::benchmark_data_smithi188_89276_object61972:head# 2024-04-23T16:25:12.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20fb7192:::benchmark_data_smithi188_89276_object152790:head# 2024-04-23T16:25:12.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20fbae4f:::benchmark_data_smithi188_89276_object94983:head# 2024-04-23T16:25:12.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20fc0b7e:::benchmark_data_smithi188_89276_object137205:head# 2024-04-23T16:25:12.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20fe1191:::benchmark_data_smithi188_89276_object125618:head# 2024-04-23T16:25:12.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20fe32a3:::benchmark_data_smithi188_89276_object64573:head# 2024-04-23T16:25:12.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20fe7741:::benchmark_data_smithi188_89276_object83849:head# 2024-04-23T16:25:12.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20feb2ce:::benchmark_data_smithi188_89276_object117670:head# 2024-04-23T16:25:12.544 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20fef6a7:::benchmark_data_smithi188_89276_object159937:head# 2024-04-23T16:25:12.544 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ff3ff2:::benchmark_data_smithi188_89276_object51080:head# 2024-04-23T16:25:12.544 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ffce9b:::benchmark_data_smithi188_89276_object65807:head# 2024-04-23T16:25:12.544 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ffd362:::benchmark_data_smithi188_89276_object120242:head# 2024-04-23T16:25:12.545 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ffd60f:::benchmark_data_smithi188_89276_object98627:head# 2024-04-23T16:25:12.545 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:20ffdd33:::benchmark_data_smithi188_89276_object143445:head# 2024-04-23T16:25:12.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2101aefa:::benchmark_data_smithi188_89276_object96990:head# 2024-04-23T16:25:12.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2102086e:::benchmark_data_smithi188_89276_object87358:head# 2024-04-23T16:25:12.547 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2102451e:::benchmark_data_smithi188_89276_object100471:head# 2024-04-23T16:25:12.548 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2102524b:::benchmark_data_smithi188_89276_object167178:head# 2024-04-23T16:25:12.548 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21026180:::benchmark_data_smithi188_89276_object164368:head# 2024-04-23T16:25:12.548 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:210280d6:::benchmark_data_smithi188_89276_object91107:head# 2024-04-23T16:25:12.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2102b47a:::benchmark_data_smithi188_89276_object136601:head# 2024-04-23T16:25:12.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21041e2f:::benchmark_data_smithi188_89276_object49112:head# 2024-04-23T16:25:12.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21045516:::benchmark_data_smithi188_89276_object153013:head# 2024-04-23T16:25:12.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21049bd5:::benchmark_data_smithi188_89276_object163538:head# 2024-04-23T16:25:12.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21058204:::benchmark_data_smithi188_89276_object124625:head# 2024-04-23T16:25:12.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21071ddf:::benchmark_data_smithi188_89276_object107369:head# 2024-04-23T16:25:12.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:210829a4:::benchmark_data_smithi188_89276_object123661:head# 2024-04-23T16:25:12.552 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:210a884d:::benchmark_data_smithi188_89276_object168562:head# 2024-04-23T16:25:12.553 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:210b2b65:::benchmark_data_smithi188_89276_object151836:head# 2024-04-23T16:25:12.553 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:210ba3d7:::benchmark_data_smithi188_89276_object100543:head# 2024-04-23T16:25:12.554 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:210bb71c:::benchmark_data_smithi188_89276_object48502:head# 2024-04-23T16:25:12.554 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:210bf7e1:::benchmark_data_smithi188_89276_object95969:head# 2024-04-23T16:25:12.555 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:210c19eb:::benchmark_data_smithi188_89276_object120532:head# 2024-04-23T16:25:12.555 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:210c1d29:::benchmark_data_smithi188_89276_object117994:head# 2024-04-23T16:25:12.555 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:210c6d98:::benchmark_data_smithi188_89276_object144926:head# 2024-04-23T16:25:12.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:210ca96e:::benchmark_data_smithi188_89276_object166223:head# 2024-04-23T16:25:12.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:210d5827:::benchmark_data_smithi188_89276_object69551:head# 2024-04-23T16:25:12.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:210ec945:::benchmark_data_smithi188_89276_object155263:head# 2024-04-23T16:25:12.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:210fa7d1:::benchmark_data_smithi188_89276_object160123:head# 2024-04-23T16:25:12.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:210fe7b0:::benchmark_data_smithi188_89276_object127439:head# 2024-04-23T16:25:12.558 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211016f3:::benchmark_data_smithi188_89276_object107673:head# 2024-04-23T16:25:12.558 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21105f1c:::benchmark_data_smithi188_89276_object109121:head# 2024-04-23T16:25:12.559 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2110f25a:::benchmark_data_smithi188_89276_object92363:head# 2024-04-23T16:25:12.559 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21119abb:::benchmark_data_smithi188_89276_object128432:head# 2024-04-23T16:25:12.559 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21119ec5:::benchmark_data_smithi188_89276_object136130:head# 2024-04-23T16:25:12.560 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2111bfaa:::benchmark_data_smithi188_89276_object60997:head# 2024-04-23T16:25:12.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2113c48f:::benchmark_data_smithi188_89276_object152262:head# 2024-04-23T16:25:12.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21140cec:::benchmark_data_smithi188_89276_object65916:head# 2024-04-23T16:25:12.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211488d4:::benchmark_data_smithi188_89276_object59982:head# 2024-04-23T16:25:12.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2114e461:::benchmark_data_smithi188_89276_object117839:head# 2024-04-23T16:25:12.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21167921:::benchmark_data_smithi188_89276_object73921:head# 2024-04-23T16:25:12.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2116bac6:::benchmark_data_smithi188_89276_object87861:head# 2024-04-23T16:25:12.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2116f831:::benchmark_data_smithi188_89276_object55019:head# 2024-04-23T16:25:12.564 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2117eea0:::benchmark_data_smithi188_89276_object108779:head# 2024-04-23T16:25:12.564 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2118e7c1:::benchmark_data_smithi188_89276_object120838:head# 2024-04-23T16:25:12.565 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2118f84d:::benchmark_data_smithi188_89276_object111535:head# 2024-04-23T16:25:12.566 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211901f0:::benchmark_data_smithi188_89276_object93214:head# 2024-04-23T16:25:12.566 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21192fb4:::benchmark_data_smithi188_89276_object119483:head# 2024-04-23T16:25:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21198f0d:::benchmark_data_smithi188_89276_object67590:head# 2024-04-23T16:25:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2119af83:::benchmark_data_smithi188_89276_object135003:head# 2024-04-23T16:25:12.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211a4c32:::benchmark_data_smithi188_89276_object134686:head# 2024-04-23T16:25:12.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211a8489:::benchmark_data_smithi188_89276_object155281:head# 2024-04-23T16:25:12.570 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211aa69c:::benchmark_data_smithi188_89276_object76889:head# 2024-04-23T16:25:12.570 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211acdee:::benchmark_data_smithi188_89276_object100417:head# 2024-04-23T16:25:12.571 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211adcde:::benchmark_data_smithi188_89276_object96650:head# 2024-04-23T16:25:12.571 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211b06bd:::benchmark_data_smithi188_89276_object65575:head# 2024-04-23T16:25:12.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211b983f:::benchmark_data_smithi188_89276_object109960:head# 2024-04-23T16:25:12.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211c7c20:::benchmark_data_smithi188_89276_object53806:head# 2024-04-23T16:25:12.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211c81cc:::benchmark_data_smithi188_89276_object47363:head# 2024-04-23T16:25:12.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211cb2cd:::benchmark_data_smithi188_89276_object103557:head# 2024-04-23T16:25:12.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211d0c19:::benchmark_data_smithi188_89276_object114664:head# 2024-04-23T16:25:12.574 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211da91e:::benchmark_data_smithi188_89276_object152510:head# 2024-04-23T16:25:12.574 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211ef7af:::benchmark_data_smithi188_89276_object109160:head# 2024-04-23T16:25:12.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211f4929:::benchmark_data_smithi188_89276_object160569:head# 2024-04-23T16:25:12.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211f55f2:::benchmark_data_smithi188_89276_object135846:head# 2024-04-23T16:25:12.576 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211f5e55:::benchmark_data_smithi188_89276_object69648:head# 2024-04-23T16:25:12.576 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:211f6d72:::benchmark_data_smithi188_89276_object111482:head# 2024-04-23T16:25:12.576 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21211b0a:::benchmark_data_smithi188_89276_object136649:head# 2024-04-23T16:25:12.577 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2122aaa6:::benchmark_data_smithi188_89276_object40699:head# 2024-04-23T16:25:12.577 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2122afaf:::benchmark_data_smithi188_89276_object82108:head# 2024-04-23T16:25:12.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:212444d5:::benchmark_data_smithi188_89276_object58264:head# 2024-04-23T16:25:12.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2124935d:::benchmark_data_smithi188_89276_object107497:head# 2024-04-23T16:25:12.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2124dfce:::benchmark_data_smithi188_89276_object48202:head# 2024-04-23T16:25:12.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:212612e0:::benchmark_data_smithi188_89276_object54151:head# 2024-04-23T16:25:12.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21265009:::benchmark_data_smithi188_89276_object110461:head# 2024-04-23T16:25:12.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2126ff35:::benchmark_data_smithi188_89276_object88803:head# 2024-04-23T16:25:12.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:212881e5:::benchmark_data_smithi188_89276_object66841:head# 2024-04-23T16:25:12.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:212a2aea:::benchmark_data_smithi188_89276_object79009:head# 2024-04-23T16:25:12.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:212aa881:::benchmark_data_smithi188_89276_object160627:head# 2024-04-23T16:25:12.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:212afdb0:::benchmark_data_smithi188_89276_object44030:head# 2024-04-23T16:25:12.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:212bd85a:::benchmark_data_smithi188_89276_object156017:head# 2024-04-23T16:25:12.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:212e263e:::benchmark_data_smithi188_89276_object89097:head# 2024-04-23T16:25:12.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:212eaf5c:::benchmark_data_smithi188_89276_object70334:head# 2024-04-23T16:25:12.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:212fdb63:::benchmark_data_smithi188_89276_object145184:head# 2024-04-23T16:25:12.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:212ffcac:::benchmark_data_smithi188_89276_object101466:head# 2024-04-23T16:25:12.588 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21304437:::benchmark_data_smithi188_89276_object138204:head# 2024-04-23T16:25:12.588 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2130e7ee:::benchmark_data_smithi188_89276_object51388:head# 2024-04-23T16:25:12.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:213122a1:::benchmark_data_smithi188_89276_object51761:head# 2024-04-23T16:25:12.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21322365:::benchmark_data_smithi188_89276_object61074:head# 2024-04-23T16:25:12.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21324586:::benchmark_data_smithi188_89276_object168128:head# 2024-04-23T16:25:12.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2132d1cd:::benchmark_data_smithi188_89276_object75951:head# 2024-04-23T16:25:12.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2132d9da:::benchmark_data_smithi188_89276_object163028:head# 2024-04-23T16:25:12.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21336d8f:::benchmark_data_smithi188_89276_object137100:head# 2024-04-23T16:25:12.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21339d38:::benchmark_data_smithi188_89276_object165692:head# 2024-04-23T16:25:12.594 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2133ab6a:::benchmark_data_smithi188_89276_object85311:head# 2024-04-23T16:25:12.594 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2134c85d:::benchmark_data_smithi188_89276_object53004:head# 2024-04-23T16:25:12.594 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2135922b:::benchmark_data_smithi188_89276_object124659:head# 2024-04-23T16:25:12.594 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2135ad8d:::benchmark_data_smithi188_89276_object151021:head# 2024-04-23T16:25:12.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2136c5a4:::benchmark_data_smithi188_89276_object140189:head# 2024-04-23T16:25:12.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2136f55c:::benchmark_data_smithi188_89276_object87336:head# 2024-04-23T16:25:12.596 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2137b053:::benchmark_data_smithi188_89276_object46120:head# 2024-04-23T16:25:12.598 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:213895b1:::benchmark_data_smithi188_89276_object56261:head# 2024-04-23T16:25:12.598 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21396424:::benchmark_data_smithi188_89276_object66379:head# 2024-04-23T16:25:12.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:213b3859:::benchmark_data_smithi188_89276_object141455:head# 2024-04-23T16:25:12.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:213bce3a:::benchmark_data_smithi188_89276_object87034:head# 2024-04-23T16:25:12.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:213bd1a8:::benchmark_data_smithi188_89276_object80353:head# 2024-04-23T16:25:12.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:213c1082:::benchmark_data_smithi188_89276_object142164:head# 2024-04-23T16:25:12.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:213c1f97:::benchmark_data_smithi188_89276_object159405:head# 2024-04-23T16:25:12.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:213c7d40:::benchmark_data_smithi188_89276_object154145:head# 2024-04-23T16:25:12.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:213c9eed:::benchmark_data_smithi188_89276_object60691:head# 2024-04-23T16:25:12.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:213d1596:::benchmark_data_smithi188_89276_object61936:head# 2024-04-23T16:25:12.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:213d6390:::benchmark_data_smithi188_89276_object149943:head# 2024-04-23T16:25:12.602 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:213dbd8e:::benchmark_data_smithi188_89276_object160524:head# 2024-04-23T16:25:12.602 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:213dd507:::benchmark_data_smithi188_89276_object103583:head# 2024-04-23T16:25:12.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:213f0969:::benchmark_data_smithi188_89276_object141527:head# 2024-04-23T16:25:12.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:213f2029:::benchmark_data_smithi188_89276_object76688:head# 2024-04-23T16:25:12.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:213f4e6e:::benchmark_data_smithi188_89276_object87287:head# 2024-04-23T16:25:12.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:213fe350:::benchmark_data_smithi188_89276_object93278:head# 2024-04-23T16:25:12.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:214088e4:::benchmark_data_smithi188_89276_object69138:head# 2024-04-23T16:25:12.605 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:214094a1:::benchmark_data_smithi188_89276_object80402:head# 2024-04-23T16:25:12.605 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2140d5a9:::benchmark_data_smithi188_89276_object137017:head# 2024-04-23T16:25:12.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2142d3a7:::benchmark_data_smithi188_89276_object164160:head# 2024-04-23T16:25:12.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2142eb09:::benchmark_data_smithi188_89276_object90444:head# 2024-04-23T16:25:12.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2142eb8b:::benchmark_data_smithi188_89276_object120889:head# 2024-04-23T16:25:12.607 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2143aa67:::benchmark_data_smithi188_89276_object80441:head# 2024-04-23T16:25:12.607 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2143b2be:::benchmark_data_smithi188_89276_object119970:head# 2024-04-23T16:25:12.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21445b0f:::benchmark_data_smithi188_89276_object76382:head# 2024-04-23T16:25:12.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2144f12f:::benchmark_data_smithi188_89276_object131719:head# 2024-04-23T16:25:12.609 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2145e8a5:::benchmark_data_smithi188_89276_object149259:head# 2024-04-23T16:25:12.609 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2146b44c:::benchmark_data_smithi188_89276_object75178:head# 2024-04-23T16:25:12.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2146d43c:::benchmark_data_smithi188_89276_object61738:head# 2024-04-23T16:25:12.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2147097b:::benchmark_data_smithi188_89276_object70144:head# 2024-04-23T16:25:12.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21475096:::benchmark_data_smithi188_89276_object51686:head# 2024-04-23T16:25:12.611 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:25:12.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21475603:::benchmark_data_smithi188_89276_object49286:head# 2024-04-23T16:25:12.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:214790bc:::benchmark_data_smithi188_89276_object158627:head# 2024-04-23T16:25:12.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2147dfe1:::benchmark_data_smithi188_89276_object91586:head# 2024-04-23T16:25:12.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21480acf:::benchmark_data_smithi188_89276_object116596:head# 2024-04-23T16:25:12.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21490578:::benchmark_data_smithi188_89276_object124220:head# 2024-04-23T16:25:12.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:214985fe:::benchmark_data_smithi188_89276_object66106:head# 2024-04-23T16:25:12.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21499f6b:::benchmark_data_smithi188_89276_object109586:head# 2024-04-23T16:25:12.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:214a1620:::benchmark_data_smithi188_89276_object98514:head# 2024-04-23T16:25:12.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:214bf624:::benchmark_data_smithi188_89276_object48205:head# 2024-04-23T16:25:12.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:214c1ee6:::benchmark_data_smithi188_89276_object98646:head# 2024-04-23T16:25:12.616 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:214c8954:::benchmark_data_smithi188_89276_object62057:head# 2024-04-23T16:25:12.616 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:214d0925:::benchmark_data_smithi188_89276_object69513:head# 2024-04-23T16:25:12.617 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:214ea62e:::benchmark_data_smithi188_89276_object85399:head# 2024-04-23T16:25:12.617 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:215025a9:::benchmark_data_smithi188_89276_object74761:head# 2024-04-23T16:25:12.618 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21517b6e:::benchmark_data_smithi188_89276_object47953:head# 2024-04-23T16:25:12.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2151a1d0:::benchmark_data_smithi188_89276_object99131:head# 2024-04-23T16:25:12.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2152dec4:::benchmark_data_smithi188_89276_object137282:head# 2024-04-23T16:25:12.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:215323d5:::benchmark_data_smithi188_89276_object149901:head# 2024-04-23T16:25:12.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2153ca5a:::benchmark_data_smithi188_89276_object110220:head# 2024-04-23T16:25:12.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2153eb68:::benchmark_data_smithi188_89276_object65438:head# 2024-04-23T16:25:12.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2153f232:::benchmark_data_smithi188_89276_object137973:head# 2024-04-23T16:25:12.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21543fa7:::benchmark_data_smithi188_89276_object60786:head# 2024-04-23T16:25:12.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2154a894:::benchmark_data_smithi188_89276_object77917:head# 2024-04-23T16:25:12.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2155b780:::benchmark_data_smithi188_89276_object131880:head# 2024-04-23T16:25:12.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:215692ca:::benchmark_data_smithi188_89276_object126203:head# 2024-04-23T16:25:12.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21572d24:::benchmark_data_smithi188_89276_object147428:head# 2024-04-23T16:25:12.624 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2157e056:::benchmark_data_smithi188_89276_object153733:head# 2024-04-23T16:25:12.624 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21582235:::benchmark_data_smithi188_89276_object54893:head# 2024-04-23T16:25:12.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:215882a6:::benchmark_data_smithi188_89276_object104912:head# 2024-04-23T16:25:12.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2158bf2e:::benchmark_data_smithi188_89276_object120268:head# 2024-04-23T16:25:12.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2158de1e:::benchmark_data_smithi188_89276_object122883:head# 2024-04-23T16:25:12.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21598cbf:::benchmark_data_smithi188_89276_object50376:head# 2024-04-23T16:25:12.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:215a1de9:::benchmark_data_smithi188_89276_object76159:head# 2024-04-23T16:25:12.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:215ab23e:::benchmark_data_smithi188_89276_object137747:head# 2024-04-23T16:25:12.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:215ad4b8:::benchmark_data_smithi188_89276_object135606:head# 2024-04-23T16:25:12.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:215db7c7:::benchmark_data_smithi188_89276_object48172:head# 2024-04-23T16:25:12.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:215dfc07:::benchmark_data_smithi188_89276_object94870:head# 2024-04-23T16:25:12.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:215ea03f:::benchmark_data_smithi188_89276_object119821:head# 2024-04-23T16:25:12.635 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:215ea05f:::benchmark_data_smithi188_89276_object155520:head# 2024-04-23T16:25:12.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:215eef75:::benchmark_data_smithi188_89276_object40941:head# 2024-04-23T16:25:12.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:215f3373:::benchmark_data_smithi188_89276_object146438:head# 2024-04-23T16:25:12.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:215fec12:::benchmark_data_smithi188_89276_object45803:head# 2024-04-23T16:25:12.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:216244de:::benchmark_data_smithi188_89276_object94271:head# 2024-04-23T16:25:12.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21636151:::benchmark_data_smithi188_89276_object128369:head# 2024-04-23T16:25:12.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21640eef:::benchmark_data_smithi188_89276_object69819:head# 2024-04-23T16:25:12.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21649fee:::benchmark_data_smithi188_89276_object98958:head# 2024-04-23T16:25:12.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2164a1f7:::benchmark_data_smithi188_89276_object80063:head# 2024-04-23T16:25:12.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2164f367:::benchmark_data_smithi188_89276_object164740:head# 2024-04-23T16:25:12.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2164f8af:::benchmark_data_smithi188_89276_object60516:head# 2024-04-23T16:25:12.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2167a5c1:::benchmark_data_smithi188_89276_object74193:head# 2024-04-23T16:25:12.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:216884a2:::benchmark_data_smithi188_89276_object168375:head# 2024-04-23T16:25:12.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2168d63c:::benchmark_data_smithi188_89276_object151480:head# 2024-04-23T16:25:12.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2169c55c:::benchmark_data_smithi188_89276_object108441:head# 2024-04-23T16:25:12.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2169dc08:::benchmark_data_smithi188_89276_object106364:head# 2024-04-23T16:25:12.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2169f12d:::benchmark_data_smithi188_89276_object107544:head# 2024-04-23T16:25:12.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:216ae7ec:::benchmark_data_smithi188_89276_object101899:head# 2024-04-23T16:25:12.643 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:12.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:216cf3ab:::benchmark_data_smithi188_89276_object105254:head# 2024-04-23T16:25:12.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:216cf43e:::benchmark_data_smithi188_89276_object75732:head# 2024-04-23T16:25:12.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:216d9480:::benchmark_data_smithi188_89276_object139982:head# 2024-04-23T16:25:12.645 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:216dc32c:::benchmark_data_smithi188_89276_object89659:head# 2024-04-23T16:25:12.645 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:216e5488:::benchmark_data_smithi188_89276_object64016:head# 2024-04-23T16:25:12.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:216f0ae1:::benchmark_data_smithi188_89276_object71991:head# 2024-04-23T16:25:12.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:216f2320:::benchmark_data_smithi188_89276_object119651:head# 2024-04-23T16:25:12.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21709f30:::benchmark_data_smithi188_89276_object123880:head# 2024-04-23T16:25:12.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21717013:::benchmark_data_smithi188_89276_object141877:head# 2024-04-23T16:25:12.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2171852a:::benchmark_data_smithi188_89276_object111222:head# 2024-04-23T16:25:12.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21718fbd:::benchmark_data_smithi188_89276_object137558:head# 2024-04-23T16:25:12.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2171fb25:::benchmark_data_smithi188_89276_object156188:head# 2024-04-23T16:25:12.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21744de1:::benchmark_data_smithi188_89276_object79483:head# 2024-04-23T16:25:12.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21748d32:::benchmark_data_smithi188_89276_object157118:head# 2024-04-23T16:25:12.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21759571:::benchmark_data_smithi188_89276_object60830:head# 2024-04-23T16:25:12.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21767f41:::benchmark_data_smithi188_89276_object164631:head# 2024-04-23T16:25:12.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2176bbc1:::benchmark_data_smithi188_89276_object100862:head# 2024-04-23T16:25:12.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2177c3a6:::benchmark_data_smithi188_89276_object101696:head# 2024-04-23T16:25:12.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21788859:::benchmark_data_smithi188_89276_object85951:head# 2024-04-23T16:25:12.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:217b1de0:::benchmark_data_smithi188_89276_object73184:head# 2024-04-23T16:25:12.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:217bde4a:::benchmark_data_smithi188_89276_object158773:head# 2024-04-23T16:25:12.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:217be446:::benchmark_data_smithi188_89276_object126806:head# 2024-04-23T16:25:12.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:217c8a98:::benchmark_data_smithi188_89276_object141361:head# 2024-04-23T16:25:12.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:217f5f11:::benchmark_data_smithi188_89276_object105202:head# 2024-04-23T16:25:12.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:217f6426:::benchmark_data_smithi188_89276_object97445:head# 2024-04-23T16:25:12.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2180b4d3:::benchmark_data_smithi188_89276_object71342:head# 2024-04-23T16:25:12.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2181447c:::benchmark_data_smithi188_89276_object146009:head# 2024-04-23T16:25:12.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21820259:::benchmark_data_smithi188_89276_object42439:head# 2024-04-23T16:25:12.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:218204d5:::benchmark_data_smithi188_89276_object47441:head# 2024-04-23T16:25:12.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2182fe80:::benchmark_data_smithi188_89276_object56995:head# 2024-04-23T16:25:12.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:218305c0:::benchmark_data_smithi188_89276_object143351:head# 2024-04-23T16:25:12.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21834ec0:::benchmark_data_smithi188_89276_object77792:head# 2024-04-23T16:25:12.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21835003:::benchmark_data_smithi188_89276_object161146:head# 2024-04-23T16:25:12.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:218367f0:::benchmark_data_smithi188_89276_object152573:head# 2024-04-23T16:25:12.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21836bdb:::benchmark_data_smithi188_89276_object71696:head# 2024-04-23T16:25:12.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:218377a0:::benchmark_data_smithi188_89276_object119246:head# 2024-04-23T16:25:12.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2183b539:::benchmark_data_smithi188_89276_object100762:head# 2024-04-23T16:25:12.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:218401c4:::benchmark_data_smithi188_89276_object112240:head# 2024-04-23T16:25:12.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:218468fc:::benchmark_data_smithi188_89276_object41360:head# 2024-04-23T16:25:12.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21854e7f:::benchmark_data_smithi188_89276_object167208:head# 2024-04-23T16:25:12.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21857bd6:::benchmark_data_smithi188_89276_object133417:head# 2024-04-23T16:25:12.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:218585c2:::benchmark_data_smithi188_89276_object62966:head# 2024-04-23T16:25:12.661 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2187fc9a:::benchmark_data_smithi188_89276_object107404:head# 2024-04-23T16:25:12.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:218902b2:::benchmark_data_smithi188_89276_object48554:head# 2024-04-23T16:25:12.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:218929e5:::benchmark_data_smithi188_89276_object57323:head# 2024-04-23T16:25:12.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21894923:::benchmark_data_smithi188_89276_object53553:head# 2024-04-23T16:25:12.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:218a6a5d:::benchmark_data_smithi188_89276_object95695:head# 2024-04-23T16:25:12.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:218b81c6:::benchmark_data_smithi188_89276_object83171:head# 2024-04-23T16:25:12.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:218bb7eb:::benchmark_data_smithi188_89276_object137846:head# 2024-04-23T16:25:12.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:218da731:::benchmark_data_smithi188_89276_object86092:head# 2024-04-23T16:25:12.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:218e9e9d:::benchmark_data_smithi188_89276_object87350:head# 2024-04-23T16:25:12.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:218f7ce5:::benchmark_data_smithi188_89276_object64830:head# 2024-04-23T16:25:12.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:218f9f0f:::benchmark_data_smithi188_89276_object49899:head# 2024-04-23T16:25:12.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:218fec61:::benchmark_data_smithi188_89276_object76929:head# 2024-04-23T16:25:12.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21909fb7:::benchmark_data_smithi188_89276_object124678:head# 2024-04-23T16:25:12.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21912f63:::benchmark_data_smithi188_89276_object50921:head# 2024-04-23T16:25:12.668 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2191fe86:::benchmark_data_smithi188_89276_object109346:head# 2024-04-23T16:25:12.668 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21925072:::benchmark_data_smithi188_89276_object139597:head# 2024-04-23T16:25:12.669 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21925c97:::benchmark_data_smithi188_89276_object112560:head# 2024-04-23T16:25:12.669 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21934574:::benchmark_data_smithi188_89276_object84692:head# 2024-04-23T16:25:12.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:219407ed:::benchmark_data_smithi188_89276_object68776:head# 2024-04-23T16:25:12.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21942fda:::benchmark_data_smithi188_89276_object158511:head# 2024-04-23T16:25:12.671 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:25:12.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21948712:::benchmark_data_smithi188_89276_object161060:head# 2024-04-23T16:25:12.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21959a33:::benchmark_data_smithi188_89276_object159814:head# 2024-04-23T16:25:12.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2195aead:::benchmark_data_smithi188_89276_object131973:head# 2024-04-23T16:25:12.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2196fc9f:::benchmark_data_smithi188_89276_object164055:head# 2024-04-23T16:25:12.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2197328e:::benchmark_data_smithi188_89276_object61526:head# 2024-04-23T16:25:12.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2197aa0b:::benchmark_data_smithi188_89276_object154204:head# 2024-04-23T16:25:12.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:219837b3:::benchmark_data_smithi188_89276_object109609:head# 2024-04-23T16:25:12.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:12.675 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:25:12.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2198714f:::benchmark_data_smithi188_89276_object102742:head# 2024-04-23T16:25:12.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2198ac1d:::benchmark_data_smithi188_89276_object124077:head# 2024-04-23T16:25:12.678 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2199df06:::benchmark_data_smithi188_89276_object68245:head# 2024-04-23T16:25:12.678 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2199f656:::benchmark_data_smithi188_89276_object41112:head# 2024-04-23T16:25:12.678 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:219a37e2:::benchmark_data_smithi188_89276_object161080:head# 2024-04-23T16:25:12.679 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:219a3f12:::benchmark_data_smithi188_89276_object128163:head# 2024-04-23T16:25:12.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:219a5f37:::benchmark_data_smithi188_89276_object76727:head# 2024-04-23T16:25:12.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:219b1a3c:::benchmark_data_smithi188_89276_object99018:head# 2024-04-23T16:25:12.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:219b85f6:::benchmark_data_smithi188_89276_object157552:head# 2024-04-23T16:25:12.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:219bb400:::benchmark_data_smithi188_89276_object128526:head# 2024-04-23T16:25:12.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:219bc015:::benchmark_data_smithi188_89276_object98677:head# 2024-04-23T16:25:12.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:219c2460:::benchmark_data_smithi188_89276_object84812:head# 2024-04-23T16:25:12.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:219c7b14:::benchmark_data_smithi188_89276_object138987:head# 2024-04-23T16:25:12.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:219cfd28:::benchmark_data_smithi188_89276_object135503:head# 2024-04-23T16:25:12.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:219d0d09:::benchmark_data_smithi188_89276_object82298:head# 2024-04-23T16:25:12.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:219d3a84:::benchmark_data_smithi188_89276_object64129:head# 2024-04-23T16:25:12.689 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:219e3827:::benchmark_data_smithi188_89276_object149104:head# 2024-04-23T16:25:12.689 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:219e8b3f:::benchmark_data_smithi188_89276_object127930:head# 2024-04-23T16:25:12.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a05989:::benchmark_data_smithi188_89276_object114700:head# 2024-04-23T16:25:12.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a068f3:::benchmark_data_smithi188_89276_object98943:head# 2024-04-23T16:25:12.691 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a08e74:::benchmark_data_smithi188_89276_object119814:head# 2024-04-23T16:25:12.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a12f81:::benchmark_data_smithi188_89276_object126144:head# 2024-04-23T16:25:12.694 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a13408:::benchmark_data_smithi188_89276_object120777:head# 2024-04-23T16:25:12.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a142a1:::benchmark_data_smithi188_89276_object92201:head# 2024-04-23T16:25:12.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a15771:::benchmark_data_smithi188_89276_object48661:head# 2024-04-23T16:25:12.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a20af7:::benchmark_data_smithi188_89276_object45502:head# 2024-04-23T16:25:12.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a3712b:::benchmark_data_smithi188_89276_object120068:head# 2024-04-23T16:25:12.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a371ce:::benchmark_data_smithi188_89276_object129899:head# 2024-04-23T16:25:12.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a3a37a:::benchmark_data_smithi188_89276_object90100:head# 2024-04-23T16:25:12.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a522bd:::benchmark_data_smithi188_89276_object150847:head# 2024-04-23T16:25:12.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a608d6:::benchmark_data_smithi188_89276_object78634:head# 2024-04-23T16:25:12.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a643a6:::benchmark_data_smithi188_89276_object139501:head# 2024-04-23T16:25:12.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a7184d:::benchmark_data_smithi188_89276_object162963:head# 2024-04-23T16:25:12.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a805ad:::benchmark_data_smithi188_89276_object143463:head# 2024-04-23T16:25:12.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a84149:::benchmark_data_smithi188_89276_object82644:head# 2024-04-23T16:25:12.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a9286d:::benchmark_data_smithi188_89276_object94988:head# 2024-04-23T16:25:12.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a95528:::benchmark_data_smithi188_89276_object104204:head# 2024-04-23T16:25:12.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a9a3fb:::benchmark_data_smithi188_89276_object111515:head# 2024-04-23T16:25:12.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21a9c3b1:::benchmark_data_smithi188_89276_object117242:head# 2024-04-23T16:25:12.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21aa2371:::benchmark_data_smithi188_89276_object109771:head# 2024-04-23T16:25:12.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21aaea37:::benchmark_data_smithi188_89276_object149695:head# 2024-04-23T16:25:12.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21ab314d:::benchmark_data_smithi188_89276_object52908:head# 2024-04-23T16:25:12.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21ab6eff:::benchmark_data_smithi188_89276_object68587:head# 2024-04-23T16:25:12.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21ab7a82:::benchmark_data_smithi188_89276_object156785:head# 2024-04-23T16:25:12.709 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21ac4ef9:::benchmark_data_smithi188_89276_object144844:head# 2024-04-23T16:25:12.709 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21ac5961:::benchmark_data_smithi188_89276_object42708:head# 2024-04-23T16:25:12.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21ac881d:::benchmark_data_smithi188_89276_object116446:head# 2024-04-23T16:25:12.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21acc077:::benchmark_data_smithi188_89276_object102188:head# 2024-04-23T16:25:12.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21acd607:::benchmark_data_smithi188_89276_object76605:head# 2024-04-23T16:25:12.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21ad09ad:::benchmark_data_smithi188_89276_object113867:head# 2024-04-23T16:25:12.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21ad54af:::benchmark_data_smithi188_89276_object167543:head# 2024-04-23T16:25:12.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21ae0494:::benchmark_data_smithi188_89276_object160927:head# 2024-04-23T16:25:12.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21aeb096:::benchmark_data_smithi188_89276_object135375:head# 2024-04-23T16:25:12.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21aecaa6:::benchmark_data_smithi188_89276_object88667:head# 2024-04-23T16:25:12.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21af31b7:::benchmark_data_smithi188_89276_object162243:head# 2024-04-23T16:25:12.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21afe09d:::benchmark_data_smithi188_89276_object159670:head# 2024-04-23T16:25:12.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b074ff:::benchmark_data_smithi188_89276_object90365:head# 2024-04-23T16:25:12.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b0ac67:::benchmark_data_smithi188_89276_object85298:head# 2024-04-23T16:25:12.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b0ba20:::benchmark_data_smithi188_89276_object157284:head# 2024-04-23T16:25:12.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b1b02e:::benchmark_data_smithi188_89276_object57693:head# 2024-04-23T16:25:12.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b35029:::benchmark_data_smithi188_89276_object153774:head# 2024-04-23T16:25:12.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b35899:::benchmark_data_smithi188_89276_object103853:head# 2024-04-23T16:25:12.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b3f30c:::benchmark_data_smithi188_89276_object121487:head# 2024-04-23T16:25:12.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b41b6f:::benchmark_data_smithi188_89276_object110325:head# 2024-04-23T16:25:12.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b4a6dc:::benchmark_data_smithi188_89276_object149668:head# 2024-04-23T16:25:12.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b4a7b7:::benchmark_data_smithi188_89276_object131552:head# 2024-04-23T16:25:12.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b4f080:::benchmark_data_smithi188_89276_object61552:head# 2024-04-23T16:25:12.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b55a17:::benchmark_data_smithi188_89276_object132829:head# 2024-04-23T16:25:12.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b614d1:::benchmark_data_smithi188_89276_object140037:head# 2024-04-23T16:25:12.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b624cc:::benchmark_data_smithi188_89276_object148457:head# 2024-04-23T16:25:12.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b634ff:::benchmark_data_smithi188_89276_object88282:head# 2024-04-23T16:25:12.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b68660:::benchmark_data_smithi188_89276_object102442:head# 2024-04-23T16:25:12.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b6e8b4:::benchmark_data_smithi188_89276_object41726:head# 2024-04-23T16:25:12.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b76440:::benchmark_data_smithi188_89276_object163239:head# 2024-04-23T16:25:12.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b90e32:::benchmark_data_smithi188_89276_object80235:head# 2024-04-23T16:25:12.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21b99f6e:::benchmark_data_smithi188_89276_object135970:head# 2024-04-23T16:25:12.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21ba5059:::benchmark_data_smithi188_89276_object87737:head# 2024-04-23T16:25:12.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21baf5ee:::benchmark_data_smithi188_89276_object80558:head# 2024-04-23T16:25:12.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21baf8f6:::benchmark_data_smithi188_89276_object84911:head# 2024-04-23T16:25:12.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21bb15a8:::benchmark_data_smithi188_89276_object129878:head# 2024-04-23T16:25:12.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21bb3eb7:::benchmark_data_smithi188_89276_object110330:head# 2024-04-23T16:25:12.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21bbcbfa:::benchmark_data_smithi188_89276_object166747:head# 2024-04-23T16:25:12.735 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21bd0729:::benchmark_data_smithi188_89276_object60178:head# 2024-04-23T16:25:12.735 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21bd246a:::benchmark_data_smithi188_89276_object50504:head# 2024-04-23T16:25:12.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21be2625:::benchmark_data_smithi188_89276_object75974:head# 2024-04-23T16:25:12.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21bffdc3:::benchmark_data_smithi188_89276_object85143:head# 2024-04-23T16:25:12.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c03af2:::benchmark_data_smithi188_89276_object141034:head# 2024-04-23T16:25:12.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c05ade:::benchmark_data_smithi188_89276_object61372:head# 2024-04-23T16:25:12.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c0e336:::benchmark_data_smithi188_89276_object63127:head# 2024-04-23T16:25:12.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c0fe99:::benchmark_data_smithi188_89276_object102830:head# 2024-04-23T16:25:12.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c10b58:::benchmark_data_smithi188_89276_object116360:head# 2024-04-23T16:25:12.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c18eba:::benchmark_data_smithi188_89276_object74083:head# 2024-04-23T16:25:12.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c2c8b8:::benchmark_data_smithi188_89276_object133871:head# 2024-04-23T16:25:12.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c2e710:::benchmark_data_smithi188_89276_object71600:head# 2024-04-23T16:25:12.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c36b21:::benchmark_data_smithi188_89276_object122606:head# 2024-04-23T16:25:12.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c3e76e:::benchmark_data_smithi188_89276_object61029:head# 2024-04-23T16:25:12.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c43fda:::benchmark_data_smithi188_89276_object95053:head# 2024-04-23T16:25:12.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c5289a:::benchmark_data_smithi188_89276_object154443:head# 2024-04-23T16:25:12.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c541ae:::benchmark_data_smithi188_89276_object111685:head# 2024-04-23T16:25:12.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c5bf44:::benchmark_data_smithi188_89276_object108954:head# 2024-04-23T16:25:12.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c76d9d:::benchmark_data_smithi188_89276_object67996:head# 2024-04-23T16:25:12.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c7ca3b:::benchmark_data_smithi188_89276_object124204:head# 2024-04-23T16:25:12.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c86848:::benchmark_data_smithi188_89276_object155903:head# 2024-04-23T16:25:12.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c8a9b6:::benchmark_data_smithi188_89276_object133675:head# 2024-04-23T16:25:12.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c92d08:::benchmark_data_smithi188_89276_object156023:head# 2024-04-23T16:25:12.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c9487a:::benchmark_data_smithi188_89276_object103722:head# 2024-04-23T16:25:12.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c9af42:::benchmark_data_smithi188_89276_object93795:head# 2024-04-23T16:25:12.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21c9be9e:::benchmark_data_smithi188_89276_object64560:head# 2024-04-23T16:25:12.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21cad55a:::benchmark_data_smithi188_89276_object115570:head# 2024-04-23T16:25:12.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21cb073a:::benchmark_data_smithi188_89276_object66122:head# 2024-04-23T16:25:12.750 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21cb7e09:::benchmark_data_smithi188_89276_object89437:head# 2024-04-23T16:25:12.750 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21cce2ba:::benchmark_data_smithi188_89276_object119064:head# 2024-04-23T16:25:12.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21cd92b5:::benchmark_data_smithi188_89276_object69465:head# 2024-04-23T16:25:12.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21ce54bb:::benchmark_data_smithi188_89276_object74366:head# 2024-04-23T16:25:12.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21cedd80:::benchmark_data_smithi188_89276_object88492:head# 2024-04-23T16:25:12.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21cfb72f:::benchmark_data_smithi188_89276_object66711:head# 2024-04-23T16:25:12.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21d2d6d3:::benchmark_data_smithi188_89276_object82533:head# 2024-04-23T16:25:12.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21d2e3ce:::benchmark_data_smithi188_89276_object74309:head# 2024-04-23T16:25:12.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21d3ae82:::benchmark_data_smithi188_89276_object64945:head# 2024-04-23T16:25:12.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21d3ebde:::benchmark_data_smithi188_89276_object140300:head# 2024-04-23T16:25:12.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21d427e6:::benchmark_data_smithi188_89276_object166139:head# 2024-04-23T16:25:12.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21d579f1:::benchmark_data_smithi188_89276_object65091:head# 2024-04-23T16:25:12.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21d58586:::benchmark_data_smithi188_89276_object78399:head# 2024-04-23T16:25:12.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21d58de5:::benchmark_data_smithi188_89276_object42216:head# 2024-04-23T16:25:12.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21d5b083:::benchmark_data_smithi188_89276_object107314:head# 2024-04-23T16:25:12.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21d5c0e4:::benchmark_data_smithi188_89276_object135097:head# 2024-04-23T16:25:12.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21d6c5df:::benchmark_data_smithi188_89276_object156176:head# 2024-04-23T16:25:12.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21d79991:::benchmark_data_smithi188_89276_object101782:head# 2024-04-23T16:25:12.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21d7bb30:::benchmark_data_smithi188_89276_object159682:head# 2024-04-23T16:25:12.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21d82c6e:::benchmark_data_smithi188_89276_object88323:head# 2024-04-23T16:25:12.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21d8bb70:::benchmark_data_smithi188_89276_object98775:head# 2024-04-23T16:25:12.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21d8e80c:::benchmark_data_smithi188_89276_object134423:head# 2024-04-23T16:25:12.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21d90d63:::benchmark_data_smithi188_89276_object85562:head# 2024-04-23T16:25:12.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21d9d762:::benchmark_data_smithi188_89276_object149713:head# 2024-04-23T16:25:12.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21da7470:::benchmark_data_smithi188_89276_object141483:head# 2024-04-23T16:25:12.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21daaa54:::benchmark_data_smithi188_89276_object57422:head# 2024-04-23T16:25:12.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21db200c:::benchmark_data_smithi188_89276_object114386:head# 2024-04-23T16:25:12.764 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21db7382:::benchmark_data_smithi188_89276_object156395:head# 2024-04-23T16:25:12.764 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21db9921:::benchmark_data_smithi188_89276_object144946:head# 2024-04-23T16:25:12.765 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21dc12e6:::benchmark_data_smithi188_89276_object77284:head# 2024-04-23T16:25:12.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21dc66d9:::benchmark_data_smithi188_89276_object80124:head# 2024-04-23T16:25:12.767 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21dc711b:::benchmark_data_smithi188_89276_object62818:head# 2024-04-23T16:25:12.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21dd1ec0:::benchmark_data_smithi188_89276_object144981:head# 2024-04-23T16:25:12.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21ddc35b:::benchmark_data_smithi188_89276_object78301:head# 2024-04-23T16:25:12.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21de17b7:::benchmark_data_smithi188_89276_object143809:head# 2024-04-23T16:25:12.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21deb847:::benchmark_data_smithi188_89276_object96287:head# 2024-04-23T16:25:12.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21decba2:::benchmark_data_smithi188_89276_object97344:head# 2024-04-23T16:25:12.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21df81a3:::benchmark_data_smithi188_89276_object54749:head# 2024-04-23T16:25:12.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e00d0f:::benchmark_data_smithi188_89276_object51200:head# 2024-04-23T16:25:12.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e084cb:::benchmark_data_smithi188_89276_object91696:head# 2024-04-23T16:25:12.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e0a692:::benchmark_data_smithi188_89276_object41714:head# 2024-04-23T16:25:12.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e2aa43:::benchmark_data_smithi188_89276_object121200:head# 2024-04-23T16:25:12.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e30673:::benchmark_data_smithi188_89276_object79351:head# 2024-04-23T16:25:12.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e370af:::benchmark_data_smithi188_89276_object119615:head# 2024-04-23T16:25:12.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e391ee:::benchmark_data_smithi188_89276_object84977:head# 2024-04-23T16:25:12.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e3bdff:::benchmark_data_smithi188_89276_object149593:head# 2024-04-23T16:25:12.775 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:12.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e3db35:::benchmark_data_smithi188_89276_object145373:head# 2024-04-23T16:25:12.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e3f8a5:::benchmark_data_smithi188_89276_object155302:head# 2024-04-23T16:25:12.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e4674e:::benchmark_data_smithi188_89276_object71346:head# 2024-04-23T16:25:12.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e47dcc:::benchmark_data_smithi188_89276_object150802:head# 2024-04-23T16:25:12.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e557a4:::benchmark_data_smithi188_89276_object104626:head# 2024-04-23T16:25:12.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e67a18:::benchmark_data_smithi188_89276_object93972:head# 2024-04-23T16:25:12.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e72dd9:::benchmark_data_smithi188_89276_object157384:head# 2024-04-23T16:25:12.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e7ed8d:::benchmark_data_smithi188_89276_object142631:head# 2024-04-23T16:25:12.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e816e4:::benchmark_data_smithi188_89276_object78148:head# 2024-04-23T16:25:12.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e82b37:::benchmark_data_smithi188_89276_object53927:head# 2024-04-23T16:25:12.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e833fb:::benchmark_data_smithi188_89276_object94860:head# 2024-04-23T16:25:12.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e834a3:::benchmark_data_smithi188_89276_object53807:head# 2024-04-23T16:25:12.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e84b15:::benchmark_data_smithi188_89276_object86650:head# 2024-04-23T16:25:12.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e900c1:::benchmark_data_smithi188_89276_object64968:head# 2024-04-23T16:25:12.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21e9d727:::benchmark_data_smithi188_89276_object92132:head# 2024-04-23T16:25:12.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21ea2cea:::benchmark_data_smithi188_89276_object147746:head# 2024-04-23T16:25:12.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21eab547:::benchmark_data_smithi188_89276_object129390:head# 2024-04-23T16:25:12.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21eac169:::benchmark_data_smithi188_89276_object61584:head# 2024-04-23T16:25:12.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21eb363b:::benchmark_data_smithi188_89276_object99685:head# 2024-04-23T16:25:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21eb4abc:::benchmark_data_smithi188_89276_object88299:head# 2024-04-23T16:25:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21eb5b6a:::benchmark_data_smithi188_89276_object124515:head# 2024-04-23T16:25:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21eb6f97:::benchmark_data_smithi188_89276_object118198:head# 2024-04-23T16:25:12.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21eca269:::benchmark_data_smithi188_89276_object47006:head# 2024-04-23T16:25:12.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21ed4d46:::benchmark_data_smithi188_89276_object117702:head# 2024-04-23T16:25:12.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21ed6bd7:::benchmark_data_smithi188_89276_object125397:head# 2024-04-23T16:25:12.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21eefd8b:::benchmark_data_smithi188_89276_object114130:head# 2024-04-23T16:25:12.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21efc615:::benchmark_data_smithi188_89276_object140428:head# 2024-04-23T16:25:12.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21f0fdb6:::benchmark_data_smithi188_89276_object65763:head# 2024-04-23T16:25:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21f16bda:::benchmark_data_smithi188_89276_object46520:head# 2024-04-23T16:25:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21f16de8:::benchmark_data_smithi188_89276_object95792:head# 2024-04-23T16:25:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21f1adfd:::benchmark_data_smithi188_89276_object130063:head# 2024-04-23T16:25:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21f1d13b:::benchmark_data_smithi188_89276_object59791:head# 2024-04-23T16:25:12.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21f1ea8d:::benchmark_data_smithi188_89276_object64982:head# 2024-04-23T16:25:12.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21f43f8a:::benchmark_data_smithi188_89276_object102593:head# 2024-04-23T16:25:12.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21f47c18:::benchmark_data_smithi188_89276_object99568:head# 2024-04-23T16:25:12.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21f64797:::benchmark_data_smithi188_89276_object60783:head# 2024-04-23T16:25:12.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21f7e8d1:::benchmark_data_smithi188_89276_object142109:head# 2024-04-23T16:25:12.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21f8beb2:::benchmark_data_smithi188_89276_object148080:head# 2024-04-23T16:25:12.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21f99d86:::benchmark_data_smithi188_89276_object143349:head# 2024-04-23T16:25:12.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21fa0c1f:::benchmark_data_smithi188_89276_object138480:head# 2024-04-23T16:25:12.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21fad6a7:::benchmark_data_smithi188_89276_object150172:head# 2024-04-23T16:25:12.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21fbeda5:::benchmark_data_smithi188_89276_object119456:head# 2024-04-23T16:25:12.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21fbf118:::benchmark_data_smithi188_89276_object41753:head# 2024-04-23T16:25:12.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21fbf75a:::benchmark_data_smithi188_89276_object53610:head# 2024-04-23T16:25:12.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21fc5529:::benchmark_data_smithi188_89276_object49074:head# 2024-04-23T16:25:12.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21fc7f96:::benchmark_data_smithi188_89276_object108726:head# 2024-04-23T16:25:12.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21fd84d3:::benchmark_data_smithi188_89276_object74577:head# 2024-04-23T16:25:12.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21fe2c22:::benchmark_data_smithi188_89276_object59820:head# 2024-04-23T16:25:12.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21fe7715:::benchmark_data_smithi188_89276_object42210:head# 2024-04-23T16:25:12.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:21ff87f5:::benchmark_data_smithi188_89276_object82508:head# 2024-04-23T16:25:12.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2200c0ae:::benchmark_data_smithi188_89276_object62390:head# 2024-04-23T16:25:12.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22011389:::benchmark_data_smithi188_89276_object166275:head# 2024-04-23T16:25:12.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22020f58:::benchmark_data_smithi188_89276_object166430:head# 2024-04-23T16:25:12.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22021374:::benchmark_data_smithi188_89276_object80045:head# 2024-04-23T16:25:12.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22023cf0:::benchmark_data_smithi188_89276_object152263:head# 2024-04-23T16:25:12.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2202e22c:::benchmark_data_smithi188_89276_object164174:head# 2024-04-23T16:25:12.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22033c12:::benchmark_data_smithi188_89276_object95395:head# 2024-04-23T16:25:12.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22033c2c:::benchmark_data_smithi188_89276_object148906:head# 2024-04-23T16:25:12.815 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:12 smithi103 ceph-mon[39629]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 251/256114 objects degraded (0.098%), 1 pg degraded) 2024-04-23T16:25:12.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:220344b8:::benchmark_data_smithi188_89276_object151601:head# 2024-04-23T16:25:12.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2203be4f:::benchmark_data_smithi188_89276_object92717:head# 2024-04-23T16:25:12.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2203ebda:::benchmark_data_smithi188_89276_object124733:head# 2024-04-23T16:25:12.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22041cba:::benchmark_data_smithi188_89276_object41294:head# 2024-04-23T16:25:12.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2204dc54:::benchmark_data_smithi188_89276_object83996:head# 2024-04-23T16:25:12.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2204f3cc:::benchmark_data_smithi188_89276_object115357:head# 2024-04-23T16:25:12.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2205a870:::benchmark_data_smithi188_89276_object113821:head# 2024-04-23T16:25:12.816 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:12.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:220601ad:::benchmark_data_smithi188_89276_object128085:head# 2024-04-23T16:25:12.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22060aba:::benchmark_data_smithi188_89276_object129407:head# 2024-04-23T16:25:12.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22065137:::benchmark_data_smithi188_89276_object91795:head# 2024-04-23T16:25:12.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2206a7dd:::benchmark_data_smithi188_89276_object116383:head# 2024-04-23T16:25:12.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2206a84a:::benchmark_data_smithi188_89276_object113460:head# 2024-04-23T16:25:12.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2206bc57:::benchmark_data_smithi188_89276_object128026:head# 2024-04-23T16:25:12.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22072825:::benchmark_data_smithi188_89276_object110507:head# 2024-04-23T16:25:12.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22074cd9:::benchmark_data_smithi188_89276_object96091:head# 2024-04-23T16:25:12.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22079556:::benchmark_data_smithi188_89276_object95575:head# 2024-04-23T16:25:12.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:220866ce:::benchmark_data_smithi188_89276_object72695:head# 2024-04-23T16:25:12.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22096fc5:::benchmark_data_smithi188_89276_object112937:head# 2024-04-23T16:25:12.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2209c62f:::benchmark_data_smithi188_89276_object83589:head# 2024-04-23T16:25:12.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:220a872f:::benchmark_data_smithi188_89276_object79811:head# 2024-04-23T16:25:12.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:220b3f14:::benchmark_data_smithi188_89276_object58324:head# 2024-04-23T16:25:12.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:220bba38:::benchmark_data_smithi188_89276_object59194:head# 2024-04-23T16:25:12.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:220c4eae:::benchmark_data_smithi188_89276_object79452:head# 2024-04-23T16:25:12.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:220cbe69:::benchmark_data_smithi188_89276_object65377:head# 2024-04-23T16:25:12.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:220d570a:::benchmark_data_smithi188_89276_object42896:head# 2024-04-23T16:25:12.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:220e2c56:::benchmark_data_smithi188_89276_object143073:head# 2024-04-23T16:25:12.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:220e74f0:::benchmark_data_smithi188_89276_object55270:head# 2024-04-23T16:25:12.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:220eb015:::benchmark_data_smithi188_89276_object92840:head# 2024-04-23T16:25:12.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:220f9282:::benchmark_data_smithi188_89276_object135748:head# 2024-04-23T16:25:12.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:220f9b24:::benchmark_data_smithi188_89276_object99949:head# 2024-04-23T16:25:12.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:220fb141:::benchmark_data_smithi188_89276_object90556:head# 2024-04-23T16:25:12.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22108c39:::benchmark_data_smithi188_89276_object61486:head# 2024-04-23T16:25:12.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22108f82:::benchmark_data_smithi188_89276_object101615:head# 2024-04-23T16:25:12.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2210a126:::benchmark_data_smithi188_89276_object83185:head# 2024-04-23T16:25:12.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2210f1cd:::benchmark_data_smithi188_89276_object130904:head# 2024-04-23T16:25:12.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2211d308:::benchmark_data_smithi188_89276_object72775:head# 2024-04-23T16:25:12.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2212c715:::benchmark_data_smithi188_89276_object88738:head# 2024-04-23T16:25:12.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22148316:::benchmark_data_smithi188_89276_object121891:head# 2024-04-23T16:25:12.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2214c9a3:::benchmark_data_smithi188_89276_object157307:head# 2024-04-23T16:25:12.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2214fde5:::benchmark_data_smithi188_89276_object143592:head# 2024-04-23T16:25:12.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22153cd2:::benchmark_data_smithi188_89276_object116682:head# 2024-04-23T16:25:12.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22154d83:::benchmark_data_smithi188_89276_object151181:head# 2024-04-23T16:25:12.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2215b33d:::benchmark_data_smithi188_89276_object55011:head# 2024-04-23T16:25:12.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22161c9e:::benchmark_data_smithi188_89276_object110273:head# 2024-04-23T16:25:12.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:221622ac:::benchmark_data_smithi188_89276_object83865:head# 2024-04-23T16:25:12.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2216d55d:::benchmark_data_smithi188_89276_object108375:head# 2024-04-23T16:25:12.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:221706ee:::benchmark_data_smithi188_89276_object107936:head# 2024-04-23T16:25:12.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22173fff:::benchmark_data_smithi188_89276_object85938:head# 2024-04-23T16:25:12.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22180bc6:::benchmark_data_smithi188_89276_object41349:head# 2024-04-23T16:25:12.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:221866af:::benchmark_data_smithi188_89276_object107255:head# 2024-04-23T16:25:12.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22188313:::benchmark_data_smithi188_89276_object69830:head# 2024-04-23T16:25:12.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22194e2c:::benchmark_data_smithi188_89276_object166320:head# 2024-04-23T16:25:12.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22198a64:::benchmark_data_smithi188_89276_object69858:head# 2024-04-23T16:25:12.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22198f1f:::benchmark_data_smithi188_89276_object66285:head# 2024-04-23T16:25:12.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2219a358:::benchmark_data_smithi188_89276_object106976:head# 2024-04-23T16:25:12.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:221a8d85:::benchmark_data_smithi188_89276_object92848:head# 2024-04-23T16:25:12.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:221ae4bf:::benchmark_data_smithi188_89276_object60550:head# 2024-04-23T16:25:12.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:221b534d:::benchmark_data_smithi188_89276_object67546:head# 2024-04-23T16:25:12.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:221b5abf:::benchmark_data_smithi188_89276_object136385:head# 2024-04-23T16:25:12.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:221ba996:::benchmark_data_smithi188_89276_object69720:head# 2024-04-23T16:25:12.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:221dd396:::benchmark_data_smithi188_89276_object64720:head# 2024-04-23T16:25:12.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:221dede0:::benchmark_data_smithi188_89276_object149777:head# 2024-04-23T16:25:12.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:221dfcba:::benchmark_data_smithi188_89276_object163928:head# 2024-04-23T16:25:12.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:221f1bd8:::benchmark_data_smithi188_89276_object152304:head# 2024-04-23T16:25:12.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:221f5410:::benchmark_data_smithi188_89276_object48082:head# 2024-04-23T16:25:12.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22212831:::benchmark_data_smithi188_89276_object131773:head# 2024-04-23T16:25:12.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22214014:::benchmark_data_smithi188_89276_object69938:head# 2024-04-23T16:25:12.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22221905:::benchmark_data_smithi188_89276_object156838:head# 2024-04-23T16:25:12.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22224b01:::benchmark_data_smithi188_89276_object53106:head# 2024-04-23T16:25:12.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22227beb:::benchmark_data_smithi188_89276_object88083:head# 2024-04-23T16:25:12.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2222f545:::benchmark_data_smithi188_89276_object154805:head# 2024-04-23T16:25:12.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2223d58b:::benchmark_data_smithi188_89276_object57318:head# 2024-04-23T16:25:12.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2224335d:::benchmark_data_smithi188_89276_object113255:head# 2024-04-23T16:25:12.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2224c8ef:::benchmark_data_smithi188_89276_object113325:head# 2024-04-23T16:25:12.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22255715:::benchmark_data_smithi188_89276_object167953:head# 2024-04-23T16:25:12.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22256cb3:::benchmark_data_smithi188_89276_object120793:head# 2024-04-23T16:25:12.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22257bac:::benchmark_data_smithi188_89276_object153444:head# 2024-04-23T16:25:12.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:222580c7:::benchmark_data_smithi188_89276_object142587:head# 2024-04-23T16:25:12.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2226df04:::benchmark_data_smithi188_89276_object72740:head# 2024-04-23T16:25:12.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:222760e1:::benchmark_data_smithi188_89276_object134505:head# 2024-04-23T16:25:12.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22276866:::benchmark_data_smithi188_89276_object159572:head# 2024-04-23T16:25:12.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2227ae1c:::benchmark_data_smithi188_89276_object149043:head# 2024-04-23T16:25:12.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2227efd8:::benchmark_data_smithi188_89276_object85602:head# 2024-04-23T16:25:12.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2227f8e0:::benchmark_data_smithi188_89276_object160111:head# 2024-04-23T16:25:12.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22284e4f:::benchmark_data_smithi188_89276_object63627:head# 2024-04-23T16:25:12.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22289be6:::benchmark_data_smithi188_89276_object82518:head# 2024-04-23T16:25:12.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2228ce78:::benchmark_data_smithi188_89276_object132852:head# 2024-04-23T16:25:12.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:222a2138:::benchmark_data_smithi188_89276_object164121:head# 2024-04-23T16:25:12.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:222a9d9b:::benchmark_data_smithi188_89276_object119023:head# 2024-04-23T16:25:12.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:222b0fff:::benchmark_data_smithi188_89276_object68273:head# 2024-04-23T16:25:12.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:222b248f:::benchmark_data_smithi188_89276_object159503:head# 2024-04-23T16:25:12.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:222b5884:::benchmark_data_smithi188_89276_object122246:head# 2024-04-23T16:25:12.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:222baf12:::benchmark_data_smithi188_89276_object84711:head# 2024-04-23T16:25:12.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:222bba5a:::benchmark_data_smithi188_89276_object116974:head# 2024-04-23T16:25:12.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:222c064d:::benchmark_data_smithi188_89276_object74722:head# 2024-04-23T16:25:12.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:222c5831:::benchmark_data_smithi188_89276_object102483:head# 2024-04-23T16:25:12.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:222c6908:::benchmark_data_smithi188_89276_object122601:head# 2024-04-23T16:25:12.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:222e48c8:::benchmark_data_smithi188_89276_object53333:head# 2024-04-23T16:25:12.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:222e766f:::benchmark_data_smithi188_89276_object161872:head# 2024-04-23T16:25:12.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:222ebc32:::benchmark_data_smithi188_89276_object144376:head# 2024-04-23T16:25:12.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:222f0b1c:::benchmark_data_smithi188_89276_object148439:head# 2024-04-23T16:25:12.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:222f13a1:::benchmark_data_smithi188_89276_object81072:head# 2024-04-23T16:25:12.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:222f4568:::benchmark_data_smithi188_89276_object122523:head# 2024-04-23T16:25:12.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:222f7bd3:::benchmark_data_smithi188_89276_object127657:head# 2024-04-23T16:25:12.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22301e77:::benchmark_data_smithi188_89276_object145709:head# 2024-04-23T16:25:12.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2230add8:::benchmark_data_smithi188_89276_object149872:head# 2024-04-23T16:25:12.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22310f33:::benchmark_data_smithi188_89276_object82511:head# 2024-04-23T16:25:12.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22311cda:::benchmark_data_smithi188_89276_object159517:head# 2024-04-23T16:25:12.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223202fd:::benchmark_data_smithi188_89276_object104191:head# 2024-04-23T16:25:12.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22333591:::benchmark_data_smithi188_89276_object77633:head# 2024-04-23T16:25:12.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2234668a:::benchmark_data_smithi188_89276_object80266:head# 2024-04-23T16:25:12.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22346e41:::benchmark_data_smithi188_89276_object141905:head# 2024-04-23T16:25:12.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2234b147:::benchmark_data_smithi188_89276_object134603:head# 2024-04-23T16:25:12.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2234e950:::benchmark_data_smithi188_89276_object150832:head# 2024-04-23T16:25:12.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22350e19:::benchmark_data_smithi188_89276_object159207:head# 2024-04-23T16:25:12.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22360d35:::benchmark_data_smithi188_89276_object120158:head# 2024-04-23T16:25:12.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2236422a:::benchmark_data_smithi188_89276_object85309:head# 2024-04-23T16:25:12.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22368356:::benchmark_data_smithi188_89276_object45030:head# 2024-04-23T16:25:12.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22369064:::benchmark_data_smithi188_89276_object125772:head# 2024-04-23T16:25:12.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2236c27a:::benchmark_data_smithi188_89276_object46032:head# 2024-04-23T16:25:12.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2236feb1:::benchmark_data_smithi188_89276_object107061:head# 2024-04-23T16:25:12.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223724e2:::benchmark_data_smithi188_89276_object85171:head# 2024-04-23T16:25:12.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2237a950:::benchmark_data_smithi188_89276_object56856:head# 2024-04-23T16:25:12.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2237f650:::benchmark_data_smithi188_89276_object52062:head# 2024-04-23T16:25:12.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223830c1:::benchmark_data_smithi188_89276_object164805:head# 2024-04-23T16:25:12.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223866f2:::benchmark_data_smithi188_89276_object90534:head# 2024-04-23T16:25:12.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223960d9:::benchmark_data_smithi188_89276_object163822:head# 2024-04-23T16:25:12.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223975d2:::benchmark_data_smithi188_89276_object89643:head# 2024-04-23T16:25:12.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22397f43:::benchmark_data_smithi188_89276_object131859:head# 2024-04-23T16:25:12.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223a4e38:::benchmark_data_smithi188_89276_object53449:head# 2024-04-23T16:25:12.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223ac9ef:::benchmark_data_smithi188_89276_object99821:head# 2024-04-23T16:25:12.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223b8e02:::benchmark_data_smithi188_89276_object68352:head# 2024-04-23T16:25:12.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223b9be1:::benchmark_data_smithi188_89276_object57203:head# 2024-04-23T16:25:12.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223ba5c0:::benchmark_data_smithi188_89276_object74765:head# 2024-04-23T16:25:12.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223bfbf9:::benchmark_data_smithi188_89276_object48188:head# 2024-04-23T16:25:12.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223ce8ea:::benchmark_data_smithi188_89276_object71011:head# 2024-04-23T16:25:12.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223e5e1b:::benchmark_data_smithi188_89276_object76519:head# 2024-04-23T16:25:12.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223e8195:::benchmark_data_smithi188_89276_object120361:head# 2024-04-23T16:25:12.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223e8f41:::benchmark_data_smithi188_89276_object144504:head# 2024-04-23T16:25:12.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223e9eb5:::benchmark_data_smithi188_89276_object56547:head# 2024-04-23T16:25:12.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223f3a9a:::benchmark_data_smithi188_89276_object156132:head# 2024-04-23T16:25:12.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223f4d61:::benchmark_data_smithi188_89276_object95076:head# 2024-04-23T16:25:12.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223f727e:::benchmark_data_smithi188_89276_object113079:head# 2024-04-23T16:25:12.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223fd8d7:::benchmark_data_smithi188_89276_object67388:head# 2024-04-23T16:25:12.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:223ffdb7:::benchmark_data_smithi188_89276_object110946:head# 2024-04-23T16:25:12.896 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22408547:::benchmark_data_smithi188_89276_object133066:head# 2024-04-23T16:25:12.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2240e1f6:::benchmark_data_smithi188_89276_object94521:head# 2024-04-23T16:25:12.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22421674:::benchmark_data_smithi188_89276_object134621:head# 2024-04-23T16:25:12.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22423245:::benchmark_data_smithi188_89276_object106690:head# 2024-04-23T16:25:12.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2242438d:::benchmark_data_smithi188_89276_object131759:head# 2024-04-23T16:25:12.899 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2243563e:::benchmark_data_smithi188_89276_object66305:head# 2024-04-23T16:25:12.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2243dba4:::benchmark_data_smithi188_89276_object154506:head# 2024-04-23T16:25:12.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2243fc83:::benchmark_data_smithi188_89276_object62311:head# 2024-04-23T16:25:12.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22441a94:::benchmark_data_smithi188_89276_object95040:head# 2024-04-23T16:25:12.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:224425cc:::benchmark_data_smithi188_89276_object93440:head# 2024-04-23T16:25:12.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:224580a7:::benchmark_data_smithi188_89276_object132875:head# 2024-04-23T16:25:12.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2245b3c6:::benchmark_data_smithi188_89276_object87028:head# 2024-04-23T16:25:12.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2247747c:::benchmark_data_smithi188_89276_object54470:head# 2024-04-23T16:25:12.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:224886a7:::benchmark_data_smithi188_89276_object98738:head# 2024-04-23T16:25:12.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2248e14d:::benchmark_data_smithi188_89276_object126311:head# 2024-04-23T16:25:12.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22495705:::benchmark_data_smithi188_89276_object80669:head# 2024-04-23T16:25:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22499f62:::benchmark_data_smithi188_89276_object107372:head# 2024-04-23T16:25:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2249a181:::benchmark_data_smithi188_89276_object74967:head# 2024-04-23T16:25:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:224a2692:::benchmark_data_smithi188_89276_object125666:head# 2024-04-23T16:25:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:224afb72:::benchmark_data_smithi188_89276_object167328:head# 2024-04-23T16:25:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:224bb249:::benchmark_data_smithi188_89276_object90297:head# 2024-04-23T16:25:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:224bb4e3:::benchmark_data_smithi188_89276_object77620:head# 2024-04-23T16:25:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:224c01df:::benchmark_data_smithi188_89276_object66350:head# 2024-04-23T16:25:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:224c1fc7:::benchmark_data_smithi188_89276_object120945:head# 2024-04-23T16:25:12.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:224d1439:::benchmark_data_smithi188_89276_object146886:head# 2024-04-23T16:25:12.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:224eb84c:::benchmark_data_smithi188_89276_object165743:head# 2024-04-23T16:25:12.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:224edadc:::benchmark_data_smithi188_89276_object56285:head# 2024-04-23T16:25:12.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22508f17:::benchmark_data_smithi188_89276_object118939:head# 2024-04-23T16:25:12.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2250c80d:::benchmark_data_smithi188_89276_object52655:head# 2024-04-23T16:25:12.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2251304f:::benchmark_data_smithi188_89276_object129748:head# 2024-04-23T16:25:12.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2252f7eb:::benchmark_data_smithi188_89276_object58625:head# 2024-04-23T16:25:12.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2253028e:::benchmark_data_smithi188_89276_object166913:head# 2024-04-23T16:25:12.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22536c62:::benchmark_data_smithi188_89276_object40306:head# 2024-04-23T16:25:12.914 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2253a0d3:::benchmark_data_smithi188_89276_object112500:head# 2024-04-23T16:25:12.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22543644:::benchmark_data_smithi188_89276_object86718:head# 2024-04-23T16:25:12.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:225441f0:::benchmark_data_smithi188_89276_object116276:head# 2024-04-23T16:25:12.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22548894:::benchmark_data_smithi188_89276_object167763:head# 2024-04-23T16:25:12.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22559cd6:::benchmark_data_smithi188_89276_object94801:head# 2024-04-23T16:25:12.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2255a08a:::benchmark_data_smithi188_89276_object44771:head# 2024-04-23T16:25:12.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2257f756:::benchmark_data_smithi188_89276_object144545:head# 2024-04-23T16:25:12.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2259ada3:::benchmark_data_smithi188_89276_object47887:head# 2024-04-23T16:25:12.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:225b418e:::benchmark_data_smithi188_89276_object92174:head# 2024-04-23T16:25:12.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:225bcc93:::benchmark_data_smithi188_89276_object112241:head# 2024-04-23T16:25:12.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:225bd28f:::benchmark_data_smithi188_89276_object70758:head# 2024-04-23T16:25:12.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:225be88a:::benchmark_data_smithi188_89276_object100942:head# 2024-04-23T16:25:12.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:225ce2e4:::benchmark_data_smithi188_89276_object164253:head# 2024-04-23T16:25:12.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:225d3cb5:::benchmark_data_smithi188_89276_object128213:head# 2024-04-23T16:25:12.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:225d7411:::benchmark_data_smithi188_89276_object133379:head# 2024-04-23T16:25:12.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:225db702:::benchmark_data_smithi188_89276_object63308:head# 2024-04-23T16:25:12.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:225f111e:::benchmark_data_smithi188_89276_object152440:head# 2024-04-23T16:25:12.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:225ff0f3:::benchmark_data_smithi188_89276_object45117:head# 2024-04-23T16:25:12.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:226018ee:::benchmark_data_smithi188_89276_object96785:head# 2024-04-23T16:25:12.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22610cab:::benchmark_data_smithi188_89276_object157327:head# 2024-04-23T16:25:12.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22618af1:::benchmark_data_smithi188_89276_object85805:head# 2024-04-23T16:25:12.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2261a4a7:::benchmark_data_smithi188_89276_object104139:head# 2024-04-23T16:25:12.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2261d647:::benchmark_data_smithi188_89276_object55714:head# 2024-04-23T16:25:12.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2262cecb:::benchmark_data_smithi188_89276_object129916:head# 2024-04-23T16:25:12.925 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2262ef69:::benchmark_data_smithi188_89276_object146144:head# 2024-04-23T16:25:12.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2264cc10:::benchmark_data_smithi188_89276_object87859:head# 2024-04-23T16:25:12.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:226521e5:::benchmark_data_smithi188_89276_object61338:head# 2024-04-23T16:25:12.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:226534c0:::benchmark_data_smithi188_89276_object71916:head# 2024-04-23T16:25:12.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2265bb73:::benchmark_data_smithi188_89276_object158058:head# 2024-04-23T16:25:12.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2265c0c7:::benchmark_data_smithi188_89276_object166753:head# 2024-04-23T16:25:12.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2265e0cc:::benchmark_data_smithi188_89276_object119197:head# 2024-04-23T16:25:12.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2268b1d5:::benchmark_data_smithi188_89276_object151532:head# 2024-04-23T16:25:12.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:226a2111:::benchmark_data_smithi188_89276_object78352:head# 2024-04-23T16:25:12.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:226b457f:::benchmark_data_smithi188_89276_object43385:head# 2024-04-23T16:25:12.930 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:226b46dd:::benchmark_data_smithi188_89276_object105271:head# 2024-04-23T16:25:12.930 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:226b785a:::benchmark_data_smithi188_89276_object134613:head# 2024-04-23T16:25:12.930 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:226cad03:::benchmark_data_smithi188_89276_object142280:head# 2024-04-23T16:25:12.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:226d026c:::benchmark_data_smithi188_89276_object135303:head# 2024-04-23T16:25:12.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:226d2283:::benchmark_data_smithi188_89276_object129195:head# 2024-04-23T16:25:12.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:226dbdf4:::benchmark_data_smithi188_89276_object74048:head# 2024-04-23T16:25:12.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:226ebcac:::benchmark_data_smithi188_89276_object147779:head# 2024-04-23T16:25:12.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:226f07c8:::benchmark_data_smithi188_89276_object61075:head# 2024-04-23T16:25:12.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:226f9d32:::benchmark_data_smithi188_89276_object55061:head# 2024-04-23T16:25:12.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:227105ee:::benchmark_data_smithi188_89276_object42131:head# 2024-04-23T16:25:12.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:227114b1:::benchmark_data_smithi188_89276_object53298:head# 2024-04-23T16:25:12.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2271f075:::benchmark_data_smithi188_89276_object144255:head# 2024-04-23T16:25:12.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22722aa9:::benchmark_data_smithi188_89276_object149795:head# 2024-04-23T16:25:12.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22724644:::benchmark_data_smithi188_89276_object51723:head# 2024-04-23T16:25:12.943 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2272d8e7:::benchmark_data_smithi188_89276_object158556:head# 2024-04-23T16:25:12.943 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2274035c:::benchmark_data_smithi188_89276_object56536:head# 2024-04-23T16:25:12.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22743da1:::benchmark_data_smithi188_89276_object79770:head# 2024-04-23T16:25:12.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22746324:::benchmark_data_smithi188_89276_object147248:head# 2024-04-23T16:25:12.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2274775c:::benchmark_data_smithi188_89276_object47150:head# 2024-04-23T16:25:12.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2274fb22:::benchmark_data_smithi188_89276_object100503:head# 2024-04-23T16:25:12.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22752487:::benchmark_data_smithi188_89276_object118792:head# 2024-04-23T16:25:12.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2275752e:::benchmark_data_smithi188_89276_object125133:head# 2024-04-23T16:25:12.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2276301f:::benchmark_data_smithi188_89276_object153509:head# 2024-04-23T16:25:12.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22763aa5:::benchmark_data_smithi188_89276_object93153:head# 2024-04-23T16:25:12.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2277bc60:::benchmark_data_smithi188_89276_object99678:head# 2024-04-23T16:25:12.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2279b473:::benchmark_data_smithi188_89276_object84299:head# 2024-04-23T16:25:12.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2279dfbc:::benchmark_data_smithi188_89276_object50184:head# 2024-04-23T16:25:12.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:227ab73d:::benchmark_data_smithi188_89276_object49674:head# 2024-04-23T16:25:12.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:227abe52:::benchmark_data_smithi188_89276_object46044:head# 2024-04-23T16:25:12.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:227af199:::benchmark_data_smithi188_89276_object147258:head# 2024-04-23T16:25:12.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:227b5f0f:::benchmark_data_smithi188_89276_object153081:head# 2024-04-23T16:25:12.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:227c0ed8:::benchmark_data_smithi188_89276_object57378:head# 2024-04-23T16:25:12.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:227c7af2:::benchmark_data_smithi188_89276_object156135:head# 2024-04-23T16:25:12.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:227d39cc:::benchmark_data_smithi188_89276_object79090:head# 2024-04-23T16:25:12.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:227dabc7:::benchmark_data_smithi188_89276_object114062:head# 2024-04-23T16:25:12.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:227e1f84:::benchmark_data_smithi188_89276_object119117:head# 2024-04-23T16:25:12.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:227f0099:::benchmark_data_smithi188_89276_object118721:head# 2024-04-23T16:25:12.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:227f1b21:::benchmark_data_smithi188_89276_object133797:head# 2024-04-23T16:25:12.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:227fd079:::benchmark_data_smithi188_89276_object138393:head# 2024-04-23T16:25:12.959 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2281459f:::benchmark_data_smithi188_89276_object71494:head# 2024-04-23T16:25:12.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2281f46c:::benchmark_data_smithi188_89276_object65306:head# 2024-04-23T16:25:12.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22821c70:::benchmark_data_smithi188_89276_object44459:head# 2024-04-23T16:25:12.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22827dbe:::benchmark_data_smithi188_89276_object134584:head# 2024-04-23T16:25:12.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2282ee2b:::benchmark_data_smithi188_89276_object156471:head# 2024-04-23T16:25:12.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2282f3b8:::benchmark_data_smithi188_89276_object71824:head# 2024-04-23T16:25:12.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22831ea5:::benchmark_data_smithi188_89276_object49805:head# 2024-04-23T16:25:12.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22839129:::benchmark_data_smithi188_89276_object67962:head# 2024-04-23T16:25:12.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22847949:::benchmark_data_smithi188_89276_object84331:head# 2024-04-23T16:25:12.966 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2284d945:::benchmark_data_smithi188_89276_object68803:head# 2024-04-23T16:25:12.967 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22850009:::benchmark_data_smithi188_89276_object115341:head# 2024-04-23T16:25:12.967 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22850581:::benchmark_data_smithi188_89276_object151860:head# 2024-04-23T16:25:12.969 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2285beae:::benchmark_data_smithi188_89276_object111384:head# 2024-04-23T16:25:12.969 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2287204b:::benchmark_data_smithi188_89276_object124745:head# 2024-04-23T16:25:12.969 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2287377e:::benchmark_data_smithi188_89276_object53450:head# 2024-04-23T16:25:12.973 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22885a5b:::benchmark_data_smithi188_89276_object165925:head# 2024-04-23T16:25:12.973 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:12.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:12 smithi188 ceph-mon[44101]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 251/256114 objects degraded (0.098%), 1 pg degraded) 2024-04-23T16:25:12.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:12.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2289b159:::benchmark_data_smithi188_89276_object60841:head# 2024-04-23T16:25:12.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2289d1c7:::benchmark_data_smithi188_89276_object148869:head# 2024-04-23T16:25:12.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2289eea1:::benchmark_data_smithi188_89276_object65994:head# 2024-04-23T16:25:12.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:228a4876:::benchmark_data_smithi188_89276_object45140:head# 2024-04-23T16:25:12.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:12.977 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:228ab737:::benchmark_data_smithi188_89276_object132346:head# 2024-04-23T16:25:12.977 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:228afb99:::benchmark_data_smithi188_89276_object104792:head# 2024-04-23T16:25:12.977 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:228c0f17:::benchmark_data_smithi188_89276_object116218:head# 2024-04-23T16:25:12.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:228c7683:::benchmark_data_smithi188_89276_object162081:head# 2024-04-23T16:25:12.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:228c8fd5:::benchmark_data_smithi188_89276_object97638:head# 2024-04-23T16:25:12.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:228cd8d0:::benchmark_data_smithi188_89276_object141500:head# 2024-04-23T16:25:12.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:228d48ea:::benchmark_data_smithi188_89276_object168460:head# 2024-04-23T16:25:12.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:228de651:::benchmark_data_smithi188_89276_object57327:head# 2024-04-23T16:25:12.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:228e1072:::benchmark_data_smithi188_89276_object77050:head# 2024-04-23T16:25:12.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:228ed1c8:::benchmark_data_smithi188_89276_object71668:head# 2024-04-23T16:25:12.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:228f1536:::benchmark_data_smithi188_89276_object63171:head# 2024-04-23T16:25:12.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:228f1e0a:::benchmark_data_smithi188_89276_object140494:head# 2024-04-23T16:25:12.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:228f1f75:::benchmark_data_smithi188_89276_object162820:head# 2024-04-23T16:25:12.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:228f2b39:::benchmark_data_smithi188_89276_object141897:head# 2024-04-23T16:25:12.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229005ea:::benchmark_data_smithi188_89276_object167331:head# 2024-04-23T16:25:12.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229058e5:::benchmark_data_smithi188_89276_object149334:head# 2024-04-23T16:25:12.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22910a66:::benchmark_data_smithi188_89276_object68485:head# 2024-04-23T16:25:12.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22912d93:::benchmark_data_smithi188_89276_object51859:head# 2024-04-23T16:25:12.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229139de:::benchmark_data_smithi188_89276_object127390:head# 2024-04-23T16:25:12.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22928ca1:::benchmark_data_smithi188_89276_object129822:head# 2024-04-23T16:25:12.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2292c82c:::benchmark_data_smithi188_89276_object144307:head# 2024-04-23T16:25:12.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22935b6a:::benchmark_data_smithi188_89276_object130476:head# 2024-04-23T16:25:12.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22940feb:::benchmark_data_smithi188_89276_object140445:head# 2024-04-23T16:25:12.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2294110c:::benchmark_data_smithi188_89276_object41350:head# 2024-04-23T16:25:12.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2294cb01:::benchmark_data_smithi188_89276_object75096:head# 2024-04-23T16:25:12.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22961522:::benchmark_data_smithi188_89276_object90162:head# 2024-04-23T16:25:12.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2296cfcd:::benchmark_data_smithi188_89276_object57114:head# 2024-04-23T16:25:12.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229763f4:::benchmark_data_smithi188_89276_object40714:head# 2024-04-23T16:25:12.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2297cfc1:::benchmark_data_smithi188_89276_object101053:head# 2024-04-23T16:25:12.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2297d53a:::benchmark_data_smithi188_89276_object157570:head# 2024-04-23T16:25:12.993 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-23T16:25:12.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229837be:::benchmark_data_smithi188_89276_object148255:head# 2024-04-23T16:25:12.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229841b2:::benchmark_data_smithi188_89276_object91173:head# 2024-04-23T16:25:12.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22993b73:::benchmark_data_smithi188_89276_object141149:head# 2024-04-23T16:25:12.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2299d91b:::benchmark_data_smithi188_89276_object143608:head# 2024-04-23T16:25:12.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229a539b:::benchmark_data_smithi188_89276_object137944:head# 2024-04-23T16:25:12.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229ad9f8:::benchmark_data_smithi188_89276_object46405:head# 2024-04-23T16:25:12.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229b2d2a:::benchmark_data_smithi188_89276_object85218:head# 2024-04-23T16:25:12.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229b5870:::benchmark_data_smithi188_89276_object84701:head# 2024-04-23T16:25:12.997 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229b801b:::benchmark_data_smithi188_89276_object90314:head# 2024-04-23T16:25:12.998 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229c036e:::benchmark_data_smithi188_89276_object54525:head# 2024-04-23T16:25:12.998 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229c1d23:::benchmark_data_smithi188_89276_object70380:head# 2024-04-23T16:25:12.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229c277d:::benchmark_data_smithi188_89276_object92678:head# 2024-04-23T16:25:12.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229caf83:::benchmark_data_smithi188_89276_object156083:head# 2024-04-23T16:25:13.002 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229d7663:::benchmark_data_smithi188_89276_object139018:head# 2024-04-23T16:25:13.003 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229db68f:::benchmark_data_smithi188_89276_object65136:head# 2024-04-23T16:25:13.003 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229e0147:::benchmark_data_smithi188_89276_object47658:head# 2024-04-23T16:25:13.003 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229e6c5d:::benchmark_data_smithi188_89276_object117956:head# 2024-04-23T16:25:13.003 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:229fe7e2:::benchmark_data_smithi188_89276_object126574:head# 2024-04-23T16:25:13.004 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22a17d26:::benchmark_data_smithi188_89276_object80187:head# 2024-04-23T16:25:13.004 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22a18647:::benchmark_data_smithi188_89276_object93267:head# 2024-04-23T16:25:13.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22a220c2:::benchmark_data_smithi188_89276_object82742:head# 2024-04-23T16:25:13.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22a23630:::benchmark_data_smithi188_89276_object148049:head# 2024-04-23T16:25:13.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22a2707b:::benchmark_data_smithi188_89276_object58119:head# 2024-04-23T16:25:13.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22a3ad35:::benchmark_data_smithi188_89276_object111796:head# 2024-04-23T16:25:13.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22a46c25:::benchmark_data_smithi188_89276_object89897:head# 2024-04-23T16:25:13.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22a5ed34:::benchmark_data_smithi188_89276_object85384:head# 2024-04-23T16:25:13.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22a61cfb:::benchmark_data_smithi188_89276_object64043:head# 2024-04-23T16:25:13.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22a6c2ec:::benchmark_data_smithi188_89276_object91166:head# 2024-04-23T16:25:13.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22a6ecda:::benchmark_data_smithi188_89276_object85413:head# 2024-04-23T16:25:13.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22a6ffb1:::benchmark_data_smithi188_89276_object116757:head# 2024-04-23T16:25:13.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22a756a8:::benchmark_data_smithi188_89276_object143423:head# 2024-04-23T16:25:13.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22a8a5cd:::benchmark_data_smithi188_89276_object122473:head# 2024-04-23T16:25:13.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22a96545:::benchmark_data_smithi188_89276_object40995:head# 2024-04-23T16:25:13.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22a97abe:::benchmark_data_smithi188_89276_object133323:head# 2024-04-23T16:25:13.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22a97c96:::benchmark_data_smithi188_89276_object52693:head# 2024-04-23T16:25:13.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22a9abca:::benchmark_data_smithi188_89276_object99513:head# 2024-04-23T16:25:13.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22aa265e:::benchmark_data_smithi188_89276_object155167:head# 2024-04-23T16:25:13.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22abd993:::benchmark_data_smithi188_89276_object93689:head# 2024-04-23T16:25:13.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22ac2bbf:::benchmark_data_smithi188_89276_object95635:head# 2024-04-23T16:25:13.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22ac88e4:::benchmark_data_smithi188_89276_object168029:head# 2024-04-23T16:25:13.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22ad0399:::benchmark_data_smithi188_89276_object151906:head# 2024-04-23T16:25:13.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22ad3e8f:::benchmark_data_smithi188_89276_object57660:head# 2024-04-23T16:25:13.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22ad4021:::benchmark_data_smithi188_89276_object152505:head# 2024-04-23T16:25:13.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22ad63ab:::benchmark_data_smithi188_89276_object74525:head# 2024-04-23T16:25:13.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22ad6d01:::benchmark_data_smithi188_89276_object41153:head# 2024-04-23T16:25:13.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22adc482:::benchmark_data_smithi188_89276_object130027:head# 2024-04-23T16:25:13.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22ae800c:::benchmark_data_smithi188_89276_object105342:head# 2024-04-23T16:25:13.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22af178e:::benchmark_data_smithi188_89276_object66067:head# 2024-04-23T16:25:13.018 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22af9892:::benchmark_data_smithi188_89276_object57308:head# 2024-04-23T16:25:13.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22afc793:::benchmark_data_smithi188_89276_object87534:head# 2024-04-23T16:25:13.022 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22afd08e:::benchmark_data_smithi188_89276_object155473:head# 2024-04-23T16:25:13.022 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b0053d:::benchmark_data_smithi188_89276_object69654:head# 2024-04-23T16:25:13.023 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b0e882:::benchmark_data_smithi188_89276_object77528:head# 2024-04-23T16:25:13.023 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b187f2:::benchmark_data_smithi188_89276_object136947:head# 2024-04-23T16:25:13.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b19700:::benchmark_data_smithi188_89276_object89127:head# 2024-04-23T16:25:13.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b1fe6a:::benchmark_data_smithi188_89276_object162094:head# 2024-04-23T16:25:13.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b29005:::benchmark_data_smithi188_89276_object57832:head# 2024-04-23T16:25:13.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b29ef2:::benchmark_data_smithi188_89276_object76013:head# 2024-04-23T16:25:13.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b3338e:::benchmark_data_smithi188_89276_object49437:head# 2024-04-23T16:25:13.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b46f79:::benchmark_data_smithi188_89276_object159366:head# 2024-04-23T16:25:13.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b490f2:::benchmark_data_smithi188_89276_object58296:head# 2024-04-23T16:25:13.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b4bcd3:::benchmark_data_smithi188_89276_object45079:head# 2024-04-23T16:25:13.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b4c3c6:::benchmark_data_smithi188_89276_object51674:head# 2024-04-23T16:25:13.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b4e872:::benchmark_data_smithi188_89276_object66492:head# 2024-04-23T16:25:13.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b580a7:::benchmark_data_smithi188_89276_object140339:head# 2024-04-23T16:25:13.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b74407:::benchmark_data_smithi188_89276_object109657:head# 2024-04-23T16:25:13.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b759ca:::benchmark_data_smithi188_89276_object113427:head# 2024-04-23T16:25:13.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b77d45:::benchmark_data_smithi188_89276_object41204:head# 2024-04-23T16:25:13.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b85a2e:::benchmark_data_smithi188_89276_object167457:head# 2024-04-23T16:25:13.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b8cd1d:::benchmark_data_smithi188_89276_object45435:head# 2024-04-23T16:25:13.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b91df2:::benchmark_data_smithi188_89276_object140225:head# 2024-04-23T16:25:13.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22b98de9:::benchmark_data_smithi188_89276_object136783:head# 2024-04-23T16:25:13.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22bab1de:::benchmark_data_smithi188_89276_object94279:head# 2024-04-23T16:25:13.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22bb0dd0:::benchmark_data_smithi188_89276_object165003:head# 2024-04-23T16:25:13.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22bb5466:::benchmark_data_smithi188_89276_object116537:head# 2024-04-23T16:25:13.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22bb5623:::benchmark_data_smithi188_89276_object99760:head# 2024-04-23T16:25:13.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22bb7062:::benchmark_data_smithi188_89276_object71966:head# 2024-04-23T16:25:13.034 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22bbe4dd:::benchmark_data_smithi188_89276_object43450:head# 2024-04-23T16:25:13.034 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22bcc234:::benchmark_data_smithi188_89276_object135122:head# 2024-04-23T16:25:13.034 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22bce655:::benchmark_data_smithi188_89276_object134885:head# 2024-04-23T16:25:13.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22bdd06a:::benchmark_data_smithi188_89276_object121751:head# 2024-04-23T16:25:13.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22bf19d1:::benchmark_data_smithi188_89276_object123009:head# 2024-04-23T16:25:13.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c02441:::benchmark_data_smithi188_89276_object118701:head# 2024-04-23T16:25:13.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c0474b:::benchmark_data_smithi188_89276_object72389:head# 2024-04-23T16:25:13.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c14639:::benchmark_data_smithi188_89276_object77205:head# 2024-04-23T16:25:13.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c1666c:::benchmark_data_smithi188_89276_object84083:head# 2024-04-23T16:25:13.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c1942f:::benchmark_data_smithi188_89276_object86580:head# 2024-04-23T16:25:13.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c1cc76:::benchmark_data_smithi188_89276_object135040:head# 2024-04-23T16:25:13.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c1eacb:::benchmark_data_smithi188_89276_object53432:head# 2024-04-23T16:25:13.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c21000:::benchmark_data_smithi188_89276_object136197:head# 2024-04-23T16:25:13.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c294da:::benchmark_data_smithi188_89276_object64383:head# 2024-04-23T16:25:13.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c296e0:::benchmark_data_smithi188_89276_object97399:head# 2024-04-23T16:25:13.040 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c2add4:::benchmark_data_smithi188_89276_object130815:head# 2024-04-23T16:25:13.040 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c2bf05:::benchmark_data_smithi188_89276_object128996:head# 2024-04-23T16:25:13.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c397fa:::benchmark_data_smithi188_89276_object68313:head# 2024-04-23T16:25:13.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c6206b:::benchmark_data_smithi188_89276_object48007:head# 2024-04-23T16:25:13.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c64117:::benchmark_data_smithi188_89276_object56296:head# 2024-04-23T16:25:13.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c64715:::benchmark_data_smithi188_89276_object123013:head# 2024-04-23T16:25:13.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c651a5:::benchmark_data_smithi188_89276_object62578:head# 2024-04-23T16:25:13.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c738be:::benchmark_data_smithi188_89276_object100228:head# 2024-04-23T16:25:13.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c74623:::benchmark_data_smithi188_89276_object58560:head# 2024-04-23T16:25:13.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c75a32:::benchmark_data_smithi188_89276_object57481:head# 2024-04-23T16:25:13.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c78cbc:::benchmark_data_smithi188_89276_object165843:head# 2024-04-23T16:25:13.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c7a146:::benchmark_data_smithi188_89276_object117838:head# 2024-04-23T16:25:13.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c8df72:::benchmark_data_smithi188_89276_object93908:head# 2024-04-23T16:25:13.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c958b8:::benchmark_data_smithi188_89276_object74980:head# 2024-04-23T16:25:13.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c9cd67:::benchmark_data_smithi188_89276_object57073:head# 2024-04-23T16:25:13.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22c9d7d8:::benchmark_data_smithi188_89276_object149562:head# 2024-04-23T16:25:13.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22caec31:::benchmark_data_smithi188_89276_object96724:head# 2024-04-23T16:25:13.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22cb0d18:::benchmark_data_smithi188_89276_object139104:head# 2024-04-23T16:25:13.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22cc2988:::benchmark_data_smithi188_89276_object92198:head# 2024-04-23T16:25:13.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22cc29e0:::benchmark_data_smithi188_89276_object130699:head# 2024-04-23T16:25:13.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22ce273d:::benchmark_data_smithi188_89276_object105727:head# 2024-04-23T16:25:13.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22cf09b5:::benchmark_data_smithi188_89276_object87126:head# 2024-04-23T16:25:13.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22cf30d7:::benchmark_data_smithi188_89276_object149731:head# 2024-04-23T16:25:13.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22cf90d2:::benchmark_data_smithi188_89276_object44314:head# 2024-04-23T16:25:13.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d01c95:::benchmark_data_smithi188_89276_object86863:head# 2024-04-23T16:25:13.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d0404b:::benchmark_data_smithi188_89276_object166671:head# 2024-04-23T16:25:13.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d094c0:::benchmark_data_smithi188_89276_object163967:head# 2024-04-23T16:25:13.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d10dd9:::benchmark_data_smithi188_89276_object75735:head# 2024-04-23T16:25:13.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d172d6:::benchmark_data_smithi188_89276_object160173:head# 2024-04-23T16:25:13.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d24ccb:::benchmark_data_smithi188_89276_object148533:head# 2024-04-23T16:25:13.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d278ea:::benchmark_data_smithi188_89276_object135002:head# 2024-04-23T16:25:13.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d2bace:::benchmark_data_smithi188_89276_object145204:head# 2024-04-23T16:25:13.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d37981:::benchmark_data_smithi188_89276_object110604:head# 2024-04-23T16:25:13.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d40fe1:::benchmark_data_smithi188_89276_object45951:head# 2024-04-23T16:25:13.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d423fe:::benchmark_data_smithi188_89276_object151135:head# 2024-04-23T16:25:13.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d493ff:::benchmark_data_smithi188_89276_object85581:head# 2024-04-23T16:25:13.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d4acb7:::benchmark_data_smithi188_89276_object123619:head# 2024-04-23T16:25:13.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d4e41d:::benchmark_data_smithi188_89276_object120850:head# 2024-04-23T16:25:13.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d4f4a0:::benchmark_data_smithi188_89276_object143954:head# 2024-04-23T16:25:13.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d54464:::benchmark_data_smithi188_89276_object73509:head# 2024-04-23T16:25:13.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d57609:::benchmark_data_smithi188_89276_object43896:head# 2024-04-23T16:25:13.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d796f4:::benchmark_data_smithi188_89276_object79972:head# 2024-04-23T16:25:13.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d7ddb9:::benchmark_data_smithi188_89276_object64611:head# 2024-04-23T16:25:13.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22d9a135:::benchmark_data_smithi188_89276_object61647:head# 2024-04-23T16:25:13.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22da81d6:::benchmark_data_smithi188_89276_object163362:head# 2024-04-23T16:25:13.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22daa729:::benchmark_data_smithi188_89276_object104900:head# 2024-04-23T16:25:13.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22dbb892:::benchmark_data_smithi188_89276_object125267:head# 2024-04-23T16:25:13.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22dc114b:::benchmark_data_smithi188_89276_object55234:head# 2024-04-23T16:25:13.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22dc4c9e:::benchmark_data_smithi188_89276_object161189:head# 2024-04-23T16:25:13.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22de29c3:::benchmark_data_smithi188_89276_object100232:head# 2024-04-23T16:25:13.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22dedef1:::benchmark_data_smithi188_89276_object43065:head# 2024-04-23T16:25:13.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22dfa878:::benchmark_data_smithi188_89276_object168498:head# 2024-04-23T16:25:13.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22dfd179:::benchmark_data_smithi188_89276_object84837:head# 2024-04-23T16:25:13.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22e0dec1:::benchmark_data_smithi188_89276_object107051:head# 2024-04-23T16:25:13.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22e0e45c:::benchmark_data_smithi188_89276_object138262:head# 2024-04-23T16:25:13.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22e1178a:::benchmark_data_smithi188_89276_object162166:head# 2024-04-23T16:25:13.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22e13288:::benchmark_data_smithi188_89276_object134573:head# 2024-04-23T16:25:13.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22e19f5d:::benchmark_data_smithi188_89276_object83239:head# 2024-04-23T16:25:13.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22e24e73:::benchmark_data_smithi188_89276_object75621:head# 2024-04-23T16:25:13.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22e55dbd:::benchmark_data_smithi188_89276_object156014:head# 2024-04-23T16:25:13.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22e58f60:::benchmark_data_smithi188_89276_object138756:head# 2024-04-23T16:25:13.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22e5c829:::benchmark_data_smithi188_89276_object78405:head# 2024-04-23T16:25:13.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22e5f75d:::benchmark_data_smithi188_89276_object102240:head# 2024-04-23T16:25:13.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22e69311:::benchmark_data_smithi188_89276_object109277:head# 2024-04-23T16:25:13.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22e6b707:::benchmark_data_smithi188_89276_object159948:head# 2024-04-23T16:25:13.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22e6d610:::benchmark_data_smithi188_89276_object112705:head# 2024-04-23T16:25:13.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22e7c90b:::benchmark_data_smithi188_89276_object81946:head# 2024-04-23T16:25:13.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22e7ccec:::benchmark_data_smithi188_89276_object115659:head# 2024-04-23T16:25:13.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22e8df44:::benchmark_data_smithi188_89276_object124823:head# 2024-04-23T16:25:13.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22e9b1d1:::benchmark_data_smithi188_89276_object45168:head# 2024-04-23T16:25:13.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22ea5b7a:::benchmark_data_smithi188_89276_object101522:head# 2024-04-23T16:25:13.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22eada34:::benchmark_data_smithi188_89276_object139264:head# 2024-04-23T16:25:13.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22eb4a39:::benchmark_data_smithi188_89276_object154735:head# 2024-04-23T16:25:13.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22eba2f2:::benchmark_data_smithi188_89276_object66721:head# 2024-04-23T16:25:13.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22ebecdd:::benchmark_data_smithi188_89276_object131545:head# 2024-04-23T16:25:13.077 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:25:13.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22ec01e6:::benchmark_data_smithi188_89276_object146667:head# 2024-04-23T16:25:13.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22ec0836:::benchmark_data_smithi188_89276_object59445:head# 2024-04-23T16:25:13.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22ec1fd1:::benchmark_data_smithi188_89276_object126059:head# 2024-04-23T16:25:13.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22ec98d9:::benchmark_data_smithi188_89276_object115595:head# 2024-04-23T16:25:13.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22edfed8:::benchmark_data_smithi188_89276_object74488:head# 2024-04-23T16:25:13.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22efac7f:::benchmark_data_smithi188_89276_object113179:head# 2024-04-23T16:25:13.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f07768:::benchmark_data_smithi188_89276_object150358:head# 2024-04-23T16:25:13.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f1beda:::benchmark_data_smithi188_89276_object139440:head# 2024-04-23T16:25:13.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f1ebd1:::benchmark_data_smithi188_89276_object98002:head# 2024-04-23T16:25:13.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f248c7:::benchmark_data_smithi188_89276_object95019:head# 2024-04-23T16:25:13.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f2b29b:::benchmark_data_smithi188_89276_object150711:head# 2024-04-23T16:25:13.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f328d9:::benchmark_data_smithi188_89276_object135224:head# 2024-04-23T16:25:13.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f3b253:::benchmark_data_smithi188_89276_object135258:head# 2024-04-23T16:25:13.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f5a28e:::benchmark_data_smithi188_89276_object75223:head# 2024-04-23T16:25:13.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f68ef0:::benchmark_data_smithi188_89276_object94071:head# 2024-04-23T16:25:13.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f76059:::benchmark_data_smithi188_89276_object133784:head# 2024-04-23T16:25:13.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f85f2e:::benchmark_data_smithi188_89276_object109858:head# 2024-04-23T16:25:13.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f86a22:::benchmark_data_smithi188_89276_object106591:head# 2024-04-23T16:25:13.087 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f8acd6:::benchmark_data_smithi188_89276_object120295:head# 2024-04-23T16:25:13.087 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f8ecdf:::benchmark_data_smithi188_89276_object52276:head# 2024-04-23T16:25:13.088 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f90ba1:::benchmark_data_smithi188_89276_object83642:head# 2024-04-23T16:25:13.088 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f922ed:::benchmark_data_smithi188_89276_object134614:head# 2024-04-23T16:25:13.088 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f937c2:::benchmark_data_smithi188_89276_object165134:head# 2024-04-23T16:25:13.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f9c402:::benchmark_data_smithi188_89276_object112742:head# 2024-04-23T16:25:13.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f9d426:::benchmark_data_smithi188_89276_object88377:head# 2024-04-23T16:25:13.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22f9e515:::benchmark_data_smithi188_89276_object159141:head# 2024-04-23T16:25:13.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22fa98de:::benchmark_data_smithi188_89276_object134767:head# 2024-04-23T16:25:13.091 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22fc6671:::benchmark_data_smithi188_89276_object77797:head# 2024-04-23T16:25:13.091 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22fca6f8:::benchmark_data_smithi188_89276_object88162:head# 2024-04-23T16:25:13.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22fd124d:::benchmark_data_smithi188_89276_object87745:head# 2024-04-23T16:25:13.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22fd2740:::benchmark_data_smithi188_89276_object40734:head# 2024-04-23T16:25:13.093 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22fd3445:::benchmark_data_smithi188_89276_object41590:head# 2024-04-23T16:25:13.093 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22fd4da6:::benchmark_data_smithi188_89276_object65634:head# 2024-04-23T16:25:13.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22fd6baf:::benchmark_data_smithi188_89276_object149612:head# 2024-04-23T16:25:13.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22fd933d:::benchmark_data_smithi188_89276_object97971:head# 2024-04-23T16:25:13.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22fdabd0:::benchmark_data_smithi188_89276_object132032:head# 2024-04-23T16:25:13.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22fdada7:::benchmark_data_smithi188_89276_object160785:head# 2024-04-23T16:25:13.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22fddf8c:::benchmark_data_smithi188_89276_object124675:head# 2024-04-23T16:25:13.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22fe0f57:::benchmark_data_smithi188_89276_object110731:head# 2024-04-23T16:25:13.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22febc00:::benchmark_data_smithi188_89276_object167222:head# 2024-04-23T16:25:13.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22ff49b0:::benchmark_data_smithi188_89276_object153721:head# 2024-04-23T16:25:13.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22ff68c4:::benchmark_data_smithi188_89276_object145281:head# 2024-04-23T16:25:13.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:22fffbcf:::benchmark_data_smithi188_89276_object152361:head# 2024-04-23T16:25:13.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23001c63:::benchmark_data_smithi188_89276_object51684:head# 2024-04-23T16:25:13.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2300aa64:::benchmark_data_smithi188_89276_object49430:head# 2024-04-23T16:25:13.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2301d852:::benchmark_data_smithi188_89276_object64818:head# 2024-04-23T16:25:13.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2302dfec:::benchmark_data_smithi188_89276_object156833:head# 2024-04-23T16:25:13.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2302e49a:::benchmark_data_smithi188_89276_object85757:head# 2024-04-23T16:25:13.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2303c4bd:::benchmark_data_smithi188_89276_object42314:head# 2024-04-23T16:25:13.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23040e9f:::benchmark_data_smithi188_89276_object124412:head# 2024-04-23T16:25:13.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23043fd2:::benchmark_data_smithi188_89276_object106162:head# 2024-04-23T16:25:13.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23044b7a:::benchmark_data_smithi188_89276_object118369:head# 2024-04-23T16:25:13.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:230586f8:::benchmark_data_smithi188_89276_object132663:head# 2024-04-23T16:25:13.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2305a054:::benchmark_data_smithi188_89276_object77387:head# 2024-04-23T16:25:13.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:230733b7:::benchmark_data_smithi188_89276_object69859:head# 2024-04-23T16:25:13.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23078854:::benchmark_data_smithi188_89276_object143450:head# 2024-04-23T16:25:13.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23079c15:::benchmark_data_smithi188_89276_object67231:head# 2024-04-23T16:25:13.107 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2307c194:::benchmark_data_smithi188_89276_object60265:head# 2024-04-23T16:25:13.107 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2308677d:::benchmark_data_smithi188_89276_object113962:head# 2024-04-23T16:25:13.108 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:13.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2309292d:::benchmark_data_smithi188_89276_object86462:head# 2024-04-23T16:25:13.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2309293a:::benchmark_data_smithi188_89276_object116337:head# 2024-04-23T16:25:13.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:230935f2:::benchmark_data_smithi188_89276_object159129:head# 2024-04-23T16:25:13.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2309c1d9:::benchmark_data_smithi188_89276_object68322:head# 2024-04-23T16:25:13.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:230a1ba1:::benchmark_data_smithi188_89276_object51670:head# 2024-04-23T16:25:13.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:230a9d79:::benchmark_data_smithi188_89276_object158237:head# 2024-04-23T16:25:13.111 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:230adbd4:::benchmark_data_smithi188_89276_object140860:head# 2024-04-23T16:25:13.111 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:230b3489:::benchmark_data_smithi188_89276_object105040:head# 2024-04-23T16:25:13.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:230cb600:::benchmark_data_smithi188_89276_object93036:head# 2024-04-23T16:25:13.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:230ce0a3:::benchmark_data_smithi188_89276_object139662:head# 2024-04-23T16:25:13.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:230da41b:::benchmark_data_smithi188_89276_object124380:head# 2024-04-23T16:25:13.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:230db8a3:::benchmark_data_smithi188_89276_object119825:head# 2024-04-23T16:25:13.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:230dc7f7:::benchmark_data_smithi188_89276_object80335:head# 2024-04-23T16:25:13.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:230e4b6e:::benchmark_data_smithi188_89276_object72450:head# 2024-04-23T16:25:13.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:230e7c51:::benchmark_data_smithi188_89276_object109570:head# 2024-04-23T16:25:13.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:230ea00e:::benchmark_data_smithi188_89276_object149211:head# 2024-04-23T16:25:13.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:230f5dcc:::benchmark_data_smithi188_89276_object71930:head# 2024-04-23T16:25:13.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:12 smithi148 ceph-mon[44520]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 251/256114 objects degraded (0.098%), 1 pg degraded) 2024-04-23T16:25:13.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:230f93b1:::benchmark_data_smithi188_89276_object61799:head# 2024-04-23T16:25:13.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:230fa28e:::benchmark_data_smithi188_89276_object134572:head# 2024-04-23T16:25:13.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:230fe071:::benchmark_data_smithi188_89276_object49310:head# 2024-04-23T16:25:13.131 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:25:13.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23103fcb:::benchmark_data_smithi188_89276_object60713:head# 2024-04-23T16:25:13.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23105080:::benchmark_data_smithi188_89276_object47597:head# 2024-04-23T16:25:13.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23106abd:::benchmark_data_smithi188_89276_object52504:head# 2024-04-23T16:25:13.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2310913c:::benchmark_data_smithi188_89276_object118682:head# 2024-04-23T16:25:13.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2310f040:::benchmark_data_smithi188_89276_object87207:head# 2024-04-23T16:25:13.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2312650c:::benchmark_data_smithi188_89276_object62834:head# 2024-04-23T16:25:13.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:231355a8:::benchmark_data_smithi188_89276_object117765:head# 2024-04-23T16:25:13.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:231395fd:::benchmark_data_smithi188_89276_object55739:head# 2024-04-23T16:25:13.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:13.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:25:13.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23151563:::benchmark_data_smithi188_89276_object142182:head# 2024-04-23T16:25:13.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:231555cb:::benchmark_data_smithi188_89276_object121548:head# 2024-04-23T16:25:13.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23161116:::benchmark_data_smithi188_89276_object119011:head# 2024-04-23T16:25:13.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23167e5c:::benchmark_data_smithi188_89276_object131432:head# 2024-04-23T16:25:13.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2316e2f0:::benchmark_data_smithi188_89276_object54231:head# 2024-04-23T16:25:13.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23172fcc:::benchmark_data_smithi188_89276_object68748:head# 2024-04-23T16:25:13.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2318538a:::benchmark_data_smithi188_89276_object156777:head# 2024-04-23T16:25:13.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2318ac38:::benchmark_data_smithi188_89276_object97334:head# 2024-04-23T16:25:13.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23197372:::benchmark_data_smithi188_89276_object126637:head# 2024-04-23T16:25:13.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23199bd2:::benchmark_data_smithi188_89276_object150892:head# 2024-04-23T16:25:13.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2319b7d1:::benchmark_data_smithi188_89276_object73341:head# 2024-04-23T16:25:13.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2319d250:::benchmark_data_smithi188_89276_object154726:head# 2024-04-23T16:25:13.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:231a827d:::benchmark_data_smithi188_89276_object126204:head# 2024-04-23T16:25:13.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:231ad0f5:::benchmark_data_smithi188_89276_object162487:head# 2024-04-23T16:25:13.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:231b7e93:::benchmark_data_smithi188_89276_object66304:head# 2024-04-23T16:25:13.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:231ca1f6:::benchmark_data_smithi188_89276_object62933:head# 2024-04-23T16:25:13.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:231cddb0:::benchmark_data_smithi188_89276_object76770:head# 2024-04-23T16:25:13.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:231cf9a7:::benchmark_data_smithi188_89276_object93245:head# 2024-04-23T16:25:13.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:231e2f14:::benchmark_data_smithi188_89276_object125811:head# 2024-04-23T16:25:13.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:231e86ed:::benchmark_data_smithi188_89276_object51981:head# 2024-04-23T16:25:13.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:231f75cd:::benchmark_data_smithi188_89276_object163032:head# 2024-04-23T16:25:13.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:231f9d46:::benchmark_data_smithi188_89276_object53181:head# 2024-04-23T16:25:13.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:232021a3:::benchmark_data_smithi188_89276_object128171:head# 2024-04-23T16:25:13.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2320fa94:::benchmark_data_smithi188_89276_object134654:head# 2024-04-23T16:25:13.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23211213:::benchmark_data_smithi188_89276_object143553:head# 2024-04-23T16:25:13.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2321d086:::benchmark_data_smithi188_89276_object159578:head# 2024-04-23T16:25:13.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23220860:::benchmark_data_smithi188_89276_object127047:head# 2024-04-23T16:25:13.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23220d41:::benchmark_data_smithi188_89276_object135147:head# 2024-04-23T16:25:13.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23223404:::benchmark_data_smithi188_89276_object65242:head# 2024-04-23T16:25:13.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2322a602:::benchmark_data_smithi188_89276_object64395:head# 2024-04-23T16:25:13.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2324207a:::benchmark_data_smithi188_89276_object139371:head# 2024-04-23T16:25:13.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2324f3c9:::benchmark_data_smithi188_89276_object112477:head# 2024-04-23T16:25:13.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23253717:::benchmark_data_smithi188_89276_object114442:head# 2024-04-23T16:25:13.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23259747:::benchmark_data_smithi188_89276_object134922:head# 2024-04-23T16:25:13.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2326d2e7:::benchmark_data_smithi188_89276_object77146:head# 2024-04-23T16:25:13.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2326fab9:::benchmark_data_smithi188_89276_object49104:head# 2024-04-23T16:25:13.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2327115c:::benchmark_data_smithi188_89276_object57239:head# 2024-04-23T16:25:13.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2327354d:::benchmark_data_smithi188_89276_object118578:head# 2024-04-23T16:25:13.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2327f23f:::benchmark_data_smithi188_89276_object76752:head# 2024-04-23T16:25:13.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2327fb45:::benchmark_data_smithi188_89276_object85456:head# 2024-04-23T16:25:13.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:232860a7:::benchmark_data_smithi188_89276_object70688:head# 2024-04-23T16:25:13.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23297465:::benchmark_data_smithi188_89276_object85019:head# 2024-04-23T16:25:13.167 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2329adad:::benchmark_data_smithi188_89276_object119830:head# 2024-04-23T16:25:13.167 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2329ff0e:::benchmark_data_smithi188_89276_object65378:head# 2024-04-23T16:25:13.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:232ab007:::benchmark_data_smithi188_89276_object51659:head# 2024-04-23T16:25:13.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:232ab873:::benchmark_data_smithi188_89276_object91815:head# 2024-04-23T16:25:13.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:232ab8d7:::benchmark_data_smithi188_89276_object118394:head# 2024-04-23T16:25:13.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:232b03cc:::benchmark_data_smithi188_89276_object146154:head# 2024-04-23T16:25:13.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:232bd39f:::benchmark_data_smithi188_89276_object67624:head# 2024-04-23T16:25:13.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:232bece8:::benchmark_data_smithi188_89276_object77759:head# 2024-04-23T16:25:13.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:232c09c7:::benchmark_data_smithi188_89276_object87943:head# 2024-04-23T16:25:13.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:232d1413:::benchmark_data_smithi188_89276_object156815:head# 2024-04-23T16:25:13.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:232fae2d:::benchmark_data_smithi188_89276_object73967:head# 2024-04-23T16:25:13.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:233069d9:::benchmark_data_smithi188_89276_object101606:head# 2024-04-23T16:25:13.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2330f7f0:::benchmark_data_smithi188_89276_object40630:head# 2024-04-23T16:25:13.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:233208f9:::benchmark_data_smithi188_89276_object81678:head# 2024-04-23T16:25:13.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23322ebf:::benchmark_data_smithi188_89276_object167742:head# 2024-04-23T16:25:13.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2332910d:::benchmark_data_smithi188_89276_object85184:head# 2024-04-23T16:25:13.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2332a018:::benchmark_data_smithi188_89276_object44786:head# 2024-04-23T16:25:13.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2332cc35:::benchmark_data_smithi188_89276_object129956:head# 2024-04-23T16:25:13.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:233323c0:::benchmark_data_smithi188_89276_object129028:head# 2024-04-23T16:25:13.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23338905:::benchmark_data_smithi188_89276_object43619:head# 2024-04-23T16:25:13.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2333a5b6:::benchmark_data_smithi188_89276_object42940:head# 2024-04-23T16:25:13.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2333b080:::benchmark_data_smithi188_89276_object143751:head# 2024-04-23T16:25:13.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:233409ef:::benchmark_data_smithi188_89276_object93932:head# 2024-04-23T16:25:13.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:233546f7:::benchmark_data_smithi188_89276_object121520:head# 2024-04-23T16:25:13.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2335cd93:::benchmark_data_smithi188_89276_object48517:head# 2024-04-23T16:25:13.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2335f634:::benchmark_data_smithi188_89276_object89075:head# 2024-04-23T16:25:13.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23360d6b:::benchmark_data_smithi188_89276_object48229:head# 2024-04-23T16:25:13.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23363ca7:::benchmark_data_smithi188_89276_object95473:head# 2024-04-23T16:25:13.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2336a5fe:::benchmark_data_smithi188_89276_object69721:head# 2024-04-23T16:25:13.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2336d81a:::benchmark_data_smithi188_89276_object160886:head# 2024-04-23T16:25:13.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23388b4b:::benchmark_data_smithi188_89276_object60109:head# 2024-04-23T16:25:13.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2338b80e:::benchmark_data_smithi188_89276_object92022:head# 2024-04-23T16:25:13.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:233975f2:::benchmark_data_smithi188_89276_object82953:head# 2024-04-23T16:25:13.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2339790a:::benchmark_data_smithi188_89276_object75246:head# 2024-04-23T16:25:13.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2339d0d4:::benchmark_data_smithi188_89276_object119446:head# 2024-04-23T16:25:13.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:233a71f4:::benchmark_data_smithi188_89276_object65507:head# 2024-04-23T16:25:13.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:233a8a0a:::benchmark_data_smithi188_89276_object163589:head# 2024-04-23T16:25:13.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:233b4ba8:::benchmark_data_smithi188_89276_object61561:head# 2024-04-23T16:25:13.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:233b522e:::benchmark_data_smithi188_89276_object112820:head# 2024-04-23T16:25:13.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:233c6586:::benchmark_data_smithi188_89276_object74515:head# 2024-04-23T16:25:13.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:233ce950:::benchmark_data_smithi188_89276_object68812:head# 2024-04-23T16:25:13.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:233cf10c:::benchmark_data_smithi188_89276_object163758:head# 2024-04-23T16:25:13.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:233ea6b2:::benchmark_data_smithi188_89276_object117539:head# 2024-04-23T16:25:13.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:233ecbc6:::benchmark_data_smithi188_89276_object135276:head# 2024-04-23T16:25:13.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:233f0153:::benchmark_data_smithi188_89276_object109405:head# 2024-04-23T16:25:13.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:233f0d9f:::benchmark_data_smithi188_89276_object73445:head# 2024-04-23T16:25:13.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23402da2:::benchmark_data_smithi188_89276_object72634:head# 2024-04-23T16:25:13.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2340a199:::benchmark_data_smithi188_89276_object164206:head# 2024-04-23T16:25:13.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2340e9ef:::benchmark_data_smithi188_89276_object162777:head# 2024-04-23T16:25:13.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23410144:::benchmark_data_smithi188_89276_object71414:head# 2024-04-23T16:25:13.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:234157f3:::benchmark_data_smithi188_89276_object134261:head# 2024-04-23T16:25:13.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2341e72d:::benchmark_data_smithi188_89276_object99622:head# 2024-04-23T16:25:13.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2342498c:::benchmark_data_smithi188_89276_object107948:head# 2024-04-23T16:25:13.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2342cbe9:::benchmark_data_smithi188_89276_object64370:head# 2024-04-23T16:25:13.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23431f7b:::benchmark_data_smithi188_89276_object78722:head# 2024-04-23T16:25:13.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23436bc5:::benchmark_data_smithi188_89276_object74508:head# 2024-04-23T16:25:13.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2344657a:::benchmark_data_smithi188_89276_object151645:head# 2024-04-23T16:25:13.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2344bd97:::benchmark_data_smithi188_89276_object56941:head# 2024-04-23T16:25:13.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23455c84:::benchmark_data_smithi188_89276_object163660:head# 2024-04-23T16:25:13.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23457735:::benchmark_data_smithi188_89276_object81143:head# 2024-04-23T16:25:13.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23471ddf:::benchmark_data_smithi188_89276_object154603:head# 2024-04-23T16:25:13.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23473e09:::benchmark_data_smithi188_89276_object68306:head# 2024-04-23T16:25:13.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2347498f:::benchmark_data_smithi188_89276_object137910:head# 2024-04-23T16:25:13.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23484004:::benchmark_data_smithi188_89276_object42121:head# 2024-04-23T16:25:13.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2349162e:::benchmark_data_smithi188_89276_object119807:head# 2024-04-23T16:25:13.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2349744e:::benchmark_data_smithi188_89276_object144604:head# 2024-04-23T16:25:13.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23498e7d:::benchmark_data_smithi188_89276_object91658:head# 2024-04-23T16:25:13.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23499678:::benchmark_data_smithi188_89276_object106068:head# 2024-04-23T16:25:13.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:234a011a:::benchmark_data_smithi188_89276_object137916:head# 2024-04-23T16:25:13.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:234ab452:::benchmark_data_smithi188_89276_object118898:head# 2024-04-23T16:25:13.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:234b7361:::benchmark_data_smithi188_89276_object117903:head# 2024-04-23T16:25:13.212 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:234c064b:::benchmark_data_smithi188_89276_object45398:head# 2024-04-23T16:25:13.212 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:234c0d04:::benchmark_data_smithi188_89276_object129056:head# 2024-04-23T16:25:13.213 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:234e427b:::benchmark_data_smithi188_89276_object150604:head# 2024-04-23T16:25:13.213 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:234ea750:::benchmark_data_smithi188_89276_object159071:head# 2024-04-23T16:25:13.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2350ae2b:::benchmark_data_smithi188_89276_object125244:head# 2024-04-23T16:25:13.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2351639a:::benchmark_data_smithi188_89276_object128045:head# 2024-04-23T16:25:13.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2352ae2b:::benchmark_data_smithi188_89276_object43918:head# 2024-04-23T16:25:13.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2353371c:::benchmark_data_smithi188_89276_object41228:head# 2024-04-23T16:25:13.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23545336:::benchmark_data_smithi188_89276_object41500:head# 2024-04-23T16:25:13.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2355260f:::benchmark_data_smithi188_89276_object79658:head# 2024-04-23T16:25:13.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23554523:::benchmark_data_smithi188_89276_object154624:head# 2024-04-23T16:25:13.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2355cd15:::benchmark_data_smithi188_89276_object154545:head# 2024-04-23T16:25:13.221 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2355f348:::benchmark_data_smithi188_89276_object111211:head# 2024-04-23T16:25:13.221 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23570a1d:::benchmark_data_smithi188_89276_object47714:head# 2024-04-23T16:25:13.221 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23577831:::benchmark_data_smithi188_89276_object162225:head# 2024-04-23T16:25:13.231 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23579496:::benchmark_data_smithi188_89276_object164312:head# 2024-04-23T16:25:13.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23585999:::benchmark_data_smithi188_89276_object74127:head# 2024-04-23T16:25:13.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2358af56:::benchmark_data_smithi188_89276_object146322:head# 2024-04-23T16:25:13.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2359fbed:::benchmark_data_smithi188_89276_object157693:head# 2024-04-23T16:25:13.238 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:25:13.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:235a650b:::benchmark_data_smithi188_89276_object79328:head# 2024-04-23T16:25:13.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:235a69ad:::benchmark_data_smithi188_89276_object61113:head# 2024-04-23T16:25:13.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:235ac1c5:::benchmark_data_smithi188_89276_object48775:head# 2024-04-23T16:25:13.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:235bb01e:::benchmark_data_smithi188_89276_object85773:head# 2024-04-23T16:25:13.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:235cebdf:::benchmark_data_smithi188_89276_object121220:head# 2024-04-23T16:25:13.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:235d1447:::benchmark_data_smithi188_89276_object110011:head# 2024-04-23T16:25:13.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:235d6653:::benchmark_data_smithi188_89276_object153326:head# 2024-04-23T16:25:13.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:235ee41d:::benchmark_data_smithi188_89276_object132624:head# 2024-04-23T16:25:13.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:235f342e:::benchmark_data_smithi188_89276_object93564:head# 2024-04-23T16:25:13.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236047cb:::benchmark_data_smithi188_89276_object101304:head# 2024-04-23T16:25:13.240 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236070f0:::benchmark_data_smithi188_89276_object57866:head# 2024-04-23T16:25:13.240 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236074de:::benchmark_data_smithi188_89276_object45715:head# 2024-04-23T16:25:13.241 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2360eb33:::benchmark_data_smithi188_89276_object123531:head# 2024-04-23T16:25:13.241 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236142c9:::benchmark_data_smithi188_89276_object54839:head# 2024-04-23T16:25:13.241 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23616faa:::benchmark_data_smithi188_89276_object102286:head# 2024-04-23T16:25:13.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236267c2:::benchmark_data_smithi188_89276_object94532:head# 2024-04-23T16:25:13.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23626db1:::benchmark_data_smithi188_89276_object64151:head# 2024-04-23T16:25:13.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23629eab:::benchmark_data_smithi188_89276_object55109:head# 2024-04-23T16:25:13.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2362b424:::benchmark_data_smithi188_89276_object136890:head# 2024-04-23T16:25:13.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23642815:::benchmark_data_smithi188_89276_object146583:head# 2024-04-23T16:25:13.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2364439d:::benchmark_data_smithi188_89276_object86303:head# 2024-04-23T16:25:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236526b6:::benchmark_data_smithi188_89276_object106828:head# 2024-04-23T16:25:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23658008:::benchmark_data_smithi188_89276_object88860:head# 2024-04-23T16:25:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2366327c:::benchmark_data_smithi188_89276_object64286:head# 2024-04-23T16:25:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23663db6:::benchmark_data_smithi188_89276_object68586:head# 2024-04-23T16:25:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2366ece4:::benchmark_data_smithi188_89276_object124885:head# 2024-04-23T16:25:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2366ee82:::benchmark_data_smithi188_89276_object73932:head# 2024-04-23T16:25:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23670844:::benchmark_data_smithi188_89276_object104134:head# 2024-04-23T16:25:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236753d8:::benchmark_data_smithi188_89276_object157328:head# 2024-04-23T16:25:13.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236786bd:::benchmark_data_smithi188_89276_object107941:head# 2024-04-23T16:25:13.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23681d4d:::benchmark_data_smithi188_89276_object61622:head# 2024-04-23T16:25:13.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23689994:::benchmark_data_smithi188_89276_object117464:head# 2024-04-23T16:25:13.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2368f762:::benchmark_data_smithi188_89276_object50500:head# 2024-04-23T16:25:13.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23692d2b:::benchmark_data_smithi188_89276_object88871:head# 2024-04-23T16:25:13.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23693a69:::benchmark_data_smithi188_89276_object108137:head# 2024-04-23T16:25:13.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236969de:::benchmark_data_smithi188_89276_object90661:head# 2024-04-23T16:25:13.258 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23698995:::benchmark_data_smithi188_89276_object156340:head# 2024-04-23T16:25:13.258 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2369afd2:::benchmark_data_smithi188_89276_object71949:head# 2024-04-23T16:25:13.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2369f756:::benchmark_data_smithi188_89276_object161781:head# 2024-04-23T16:25:13.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236a00b2:::benchmark_data_smithi188_89276_object115382:head# 2024-04-23T16:25:13.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236a069d:::benchmark_data_smithi188_89276_object109060:head# 2024-04-23T16:25:13.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236a74bf:::benchmark_data_smithi188_89276_object118973:head# 2024-04-23T16:25:13.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236a941c:::benchmark_data_smithi188_89276_object129834:head# 2024-04-23T16:25:13.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236ad1b6:::benchmark_data_smithi188_89276_object46412:head# 2024-04-23T16:25:13.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236ba3f3:::benchmark_data_smithi188_89276_object73149:head# 2024-04-23T16:25:13.262 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236cf53d:::benchmark_data_smithi188_89276_object114572:head# 2024-04-23T16:25:13.262 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236d75e8:::benchmark_data_smithi188_89276_object41900:head# 2024-04-23T16:25:13.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236d7a53:::benchmark_data_smithi188_89276_object148036:head# 2024-04-23T16:25:13.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236d8fab:::benchmark_data_smithi188_89276_object69904:head# 2024-04-23T16:25:13.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236dc628:::benchmark_data_smithi188_89276_object147128:head# 2024-04-23T16:25:13.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236dc6f6:::benchmark_data_smithi188_89276_object51748:head# 2024-04-23T16:25:13.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236df840:::benchmark_data_smithi188_89276_object142740:head# 2024-04-23T16:25:13.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236ea7b4:::benchmark_data_smithi188_89276_object168181:head# 2024-04-23T16:25:13.265 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:13.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236ee3af:::benchmark_data_smithi188_89276_object145830:head# 2024-04-23T16:25:13.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:236fb70a:::benchmark_data_smithi188_89276_object110779:head# 2024-04-23T16:25:13.266 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23700f4c:::benchmark_data_smithi188_89276_object136767:head# 2024-04-23T16:25:13.266 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:237069bd:::benchmark_data_smithi188_89276_object47473:head# 2024-04-23T16:25:13.267 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2370cfc6:::benchmark_data_smithi188_89276_object109380:head# 2024-04-23T16:25:13.267 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2372e6b5:::benchmark_data_smithi188_89276_object55287:head# 2024-04-23T16:25:13.268 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2372e7fb:::benchmark_data_smithi188_89276_object152265:head# 2024-04-23T16:25:13.268 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23738442:::benchmark_data_smithi188_89276_object132008:head# 2024-04-23T16:25:13.269 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23756e28:::benchmark_data_smithi188_89276_object127704:head# 2024-04-23T16:25:13.269 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23760221:::benchmark_data_smithi188_89276_object141135:head# 2024-04-23T16:25:13.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:237703c0:::benchmark_data_smithi188_89276_object133649:head# 2024-04-23T16:25:13.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2377f4e3:::benchmark_data_smithi188_89276_object84989:head# 2024-04-23T16:25:13.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23790d22:::benchmark_data_smithi188_89276_object76588:head# 2024-04-23T16:25:13.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23793354:::benchmark_data_smithi188_89276_object129268:head# 2024-04-23T16:25:13.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23796ac7:::benchmark_data_smithi188_89276_object155141:head# 2024-04-23T16:25:13.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:237af94f:::benchmark_data_smithi188_89276_object64437:head# 2024-04-23T16:25:13.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:237b944e:::benchmark_data_smithi188_89276_object71791:head# 2024-04-23T16:25:13.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:237bc800:::benchmark_data_smithi188_89276_object117684:head# 2024-04-23T16:25:13.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:237bfec3:::benchmark_data_smithi188_89276_object90028:head# 2024-04-23T16:25:13.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:237cc739:::benchmark_data_smithi188_89276_object95023:head# 2024-04-23T16:25:13.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:237ee65c:::benchmark_data_smithi188_89276_object167268:head# 2024-04-23T16:25:13.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:237f1550:::benchmark_data_smithi188_89276_object40357:head# 2024-04-23T16:25:13.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:237f80ec:::benchmark_data_smithi188_89276_object125078:head# 2024-04-23T16:25:13.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:237ff61f:::benchmark_data_smithi188_89276_object65435:head# 2024-04-23T16:25:13.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2381857f:::benchmark_data_smithi188_89276_object77021:head# 2024-04-23T16:25:13.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2381883e:::benchmark_data_smithi188_89276_object113490:head# 2024-04-23T16:25:13.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23825d64:::benchmark_data_smithi188_89276_object83334:head# 2024-04-23T16:25:13.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:238263fd:::benchmark_data_smithi188_89276_object92941:head# 2024-04-23T16:25:13.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2382693c:::benchmark_data_smithi188_89276_object46706:head# 2024-04-23T16:25:13.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23827b2a:::benchmark_data_smithi188_89276_object120505:head# 2024-04-23T16:25:13.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2382c1c2:::benchmark_data_smithi188_89276_object122608:head# 2024-04-23T16:25:13.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2383cb43:::benchmark_data_smithi188_89276_object85106:head# 2024-04-23T16:25:13.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2385d79d:::benchmark_data_smithi188_89276_object52604:head# 2024-04-23T16:25:13.284 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:25:13.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2386a3ce:::benchmark_data_smithi188_89276_object165593:head# 2024-04-23T16:25:13.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2386c4d7:::benchmark_data_smithi188_89276_object126870:head# 2024-04-23T16:25:13.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2387940f:::benchmark_data_smithi188_89276_object87551:head# 2024-04-23T16:25:13.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2389ca34:::benchmark_data_smithi188_89276_object115391:head# 2024-04-23T16:25:13.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2389d46e:::benchmark_data_smithi188_89276_object60093:head# 2024-04-23T16:25:13.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2389dde6:::benchmark_data_smithi188_89276_object150555:head# 2024-04-23T16:25:13.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:238b1c5f:::benchmark_data_smithi188_89276_object164535:head# 2024-04-23T16:25:13.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:238bac26:::benchmark_data_smithi188_89276_object162206:head# 2024-04-23T16:25:13.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:238c6d7f:::benchmark_data_smithi188_89276_object72493:head# 2024-04-23T16:25:13.287 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:238cd1c5:::benchmark_data_smithi188_89276_object134267:head# 2024-04-23T16:25:13.287 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:238d396f:::benchmark_data_smithi188_89276_object142978:head# 2024-04-23T16:25:13.287 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:238d5614:::benchmark_data_smithi188_89276_object68179:head# 2024-04-23T16:25:13.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:238eca20:::benchmark_data_smithi188_89276_object168342:head# 2024-04-23T16:25:13.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:239038e9:::benchmark_data_smithi188_89276_object162215:head# 2024-04-23T16:25:13.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23904a85:::benchmark_data_smithi188_89276_object110346:head# 2024-04-23T16:25:13.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23931fea:::benchmark_data_smithi188_89276_object123620:head# 2024-04-23T16:25:13.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:13.291 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:25:13.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23943389:::benchmark_data_smithi188_89276_object162761:head# 2024-04-23T16:25:13.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2394555f:::benchmark_data_smithi188_89276_object66709:head# 2024-04-23T16:25:13.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2394b2d6:::benchmark_data_smithi188_89276_object52583:head# 2024-04-23T16:25:13.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2395c927:::benchmark_data_smithi188_89276_object105760:head# 2024-04-23T16:25:13.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:239685a6:::benchmark_data_smithi188_89276_object143633:head# 2024-04-23T16:25:13.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23976b99:::benchmark_data_smithi188_89276_object143237:head# 2024-04-23T16:25:13.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:239809e2:::benchmark_data_smithi188_89276_object55636:head# 2024-04-23T16:25:13.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2398b006:::benchmark_data_smithi188_89276_object158207:head# 2024-04-23T16:25:13.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2398d92a:::benchmark_data_smithi188_89276_object150648:head# 2024-04-23T16:25:13.295 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:239979b7:::benchmark_data_smithi188_89276_object86665:head# 2024-04-23T16:25:13.295 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:239a2cb2:::benchmark_data_smithi188_89276_object110353:head# 2024-04-23T16:25:13.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:239a440b:::benchmark_data_smithi188_89276_object65706:head# 2024-04-23T16:25:13.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:239b5b10:::benchmark_data_smithi188_89276_object138234:head# 2024-04-23T16:25:13.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:239bcac5:::benchmark_data_smithi188_89276_object126517:head# 2024-04-23T16:25:13.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:239c5fa7:::benchmark_data_smithi188_89276_object109321:head# 2024-04-23T16:25:13.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:239d23e5:::benchmark_data_smithi188_89276_object164112:head# 2024-04-23T16:25:13.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:239d501d:::benchmark_data_smithi188_89276_object139271:head# 2024-04-23T16:25:13.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:239d5960:::benchmark_data_smithi188_89276_object136627:head# 2024-04-23T16:25:13.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:239f8e74:::benchmark_data_smithi188_89276_object155941:head# 2024-04-23T16:25:13.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a07598:::benchmark_data_smithi188_89276_object142145:head# 2024-04-23T16:25:13.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a109cb:::benchmark_data_smithi188_89276_object164547:head# 2024-04-23T16:25:13.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a34b59:::benchmark_data_smithi188_89276_object97663:head# 2024-04-23T16:25:13.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a4815e:::benchmark_data_smithi188_89276_object64051:head# 2024-04-23T16:25:13.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a4facf:::benchmark_data_smithi188_89276_object46829:head# 2024-04-23T16:25:13.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a51ab6:::benchmark_data_smithi188_89276_object130032:head# 2024-04-23T16:25:13.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a534b1:::benchmark_data_smithi188_89276_object96469:head# 2024-04-23T16:25:13.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a5a7e8:::benchmark_data_smithi188_89276_object125652:head# 2024-04-23T16:25:13.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a6630a:::benchmark_data_smithi188_89276_object99058:head# 2024-04-23T16:25:13.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a67658:::benchmark_data_smithi188_89276_object125528:head# 2024-04-23T16:25:13.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a67f2d:::benchmark_data_smithi188_89276_object47701:head# 2024-04-23T16:25:13.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a68384:::benchmark_data_smithi188_89276_object53498:head# 2024-04-23T16:25:13.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a6a1f1:::benchmark_data_smithi188_89276_object51422:head# 2024-04-23T16:25:13.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a710f6:::benchmark_data_smithi188_89276_object115685:head# 2024-04-23T16:25:13.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a72803:::benchmark_data_smithi188_89276_object46271:head# 2024-04-23T16:25:13.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a74bac:::benchmark_data_smithi188_89276_object106049:head# 2024-04-23T16:25:13.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a7a472:::benchmark_data_smithi188_89276_object94128:head# 2024-04-23T16:25:13.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a7dc3a:::benchmark_data_smithi188_89276_object139336:head# 2024-04-23T16:25:13.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a82d38:::benchmark_data_smithi188_89276_object167187:head# 2024-04-23T16:25:13.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a8bbf0:::benchmark_data_smithi188_89276_object135385:head# 2024-04-23T16:25:13.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a924c1:::benchmark_data_smithi188_89276_object152714:head# 2024-04-23T16:25:13.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23a978d7:::benchmark_data_smithi188_89276_object96292:head# 2024-04-23T16:25:13.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23aa1da0:::benchmark_data_smithi188_89276_object163514:head# 2024-04-23T16:25:13.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23aaa825:::benchmark_data_smithi188_89276_object110602:head# 2024-04-23T16:25:13.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23aad8ed:::benchmark_data_smithi188_89276_object168511:head# 2024-04-23T16:25:13.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ab1536:::benchmark_data_smithi188_89276_object82866:head# 2024-04-23T16:25:13.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ac0b90:::benchmark_data_smithi188_89276_object100123:head# 2024-04-23T16:25:13.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ac2e6a:::benchmark_data_smithi188_89276_object91900:head# 2024-04-23T16:25:13.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ac8190:::benchmark_data_smithi188_89276_object113550:head# 2024-04-23T16:25:13.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ad0c52:::benchmark_data_smithi188_89276_object158767:head# 2024-04-23T16:25:13.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ad1d77:::benchmark_data_smithi188_89276_object76580:head# 2024-04-23T16:25:13.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ad5466:::benchmark_data_smithi188_89276_object127186:head# 2024-04-23T16:25:13.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ad7b0d:::benchmark_data_smithi188_89276_object141139:head# 2024-04-23T16:25:13.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23af4d72:::benchmark_data_smithi188_89276_object93708:head# 2024-04-23T16:25:13.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23b011e5:::benchmark_data_smithi188_89276_object40696:head# 2024-04-23T16:25:13.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23b01556:::benchmark_data_smithi188_89276_object165329:head# 2024-04-23T16:25:13.326 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23b045fe:::benchmark_data_smithi188_89276_object62425:head# 2024-04-23T16:25:13.326 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23b0f2b3:::benchmark_data_smithi188_89276_object144780:head# 2024-04-23T16:25:13.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23b18d91:::benchmark_data_smithi188_89276_object81530:head# 2024-04-23T16:25:13.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23b26e71:::benchmark_data_smithi188_89276_object49176:head# 2024-04-23T16:25:13.328 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23b37e76:::benchmark_data_smithi188_89276_object82773:head# 2024-04-23T16:25:13.328 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23b4abf2:::benchmark_data_smithi188_89276_object127021:head# 2024-04-23T16:25:13.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23b523b1:::benchmark_data_smithi188_89276_object158157:head# 2024-04-23T16:25:13.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23b57b4f:::benchmark_data_smithi188_89276_object104913:head# 2024-04-23T16:25:13.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23b5f1a2:::benchmark_data_smithi188_89276_object41310:head# 2024-04-23T16:25:13.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23b60641:::benchmark_data_smithi188_89276_object48020:head# 2024-04-23T16:25:13.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23b792b4:::benchmark_data_smithi188_89276_object111489:head# 2024-04-23T16:25:13.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23b7f2c6:::benchmark_data_smithi188_89276_object41553:head# 2024-04-23T16:25:13.337 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23b802e6:::benchmark_data_smithi188_89276_object120288:head# 2024-04-23T16:25:13.337 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23b82f0c:::benchmark_data_smithi188_89276_object139653:head# 2024-04-23T16:25:13.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23b9e595:::benchmark_data_smithi188_89276_object148525:head# 2024-04-23T16:25:13.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ba3036:::benchmark_data_smithi188_89276_object74801:head# 2024-04-23T16:25:13.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ba822b:::benchmark_data_smithi188_89276_object75491:head# 2024-04-23T16:25:13.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23bac2ef:::benchmark_data_smithi188_89276_object73565:head# 2024-04-23T16:25:13.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23bf597a:::benchmark_data_smithi188_89276_object60702:head# 2024-04-23T16:25:13.341 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23bfa69f:::benchmark_data_smithi188_89276_object72256:head# 2024-04-23T16:25:13.341 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23c066d3:::benchmark_data_smithi188_89276_object106931:head# 2024-04-23T16:25:13.341 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23c0cf70:::benchmark_data_smithi188_89276_object64480:head# 2024-04-23T16:25:13.342 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23c192e3:::benchmark_data_smithi188_89276_object84744:head# 2024-04-23T16:25:13.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23c1fcf4:::benchmark_data_smithi188_89276_object109032:head# 2024-04-23T16:25:13.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23c358a3:::benchmark_data_smithi188_89276_object55078:head# 2024-04-23T16:25:13.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23c4b87e:::benchmark_data_smithi188_89276_object104083:head# 2024-04-23T16:25:13.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23c5c586:::benchmark_data_smithi188_89276_object83069:head# 2024-04-23T16:25:13.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23c68fcd:::benchmark_data_smithi188_89276_object96782:head# 2024-04-23T16:25:13.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23c7e226:::benchmark_data_smithi188_89276_object62155:head# 2024-04-23T16:25:13.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23c85eb5:::benchmark_data_smithi188_89276_object74970:head# 2024-04-23T16:25:13.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23c8856a:::benchmark_data_smithi188_89276_object144343:head# 2024-04-23T16:25:13.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23c89c6d:::benchmark_data_smithi188_89276_object46039:head# 2024-04-23T16:25:13.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23c8a640:::benchmark_data_smithi188_89276_object104441:head# 2024-04-23T16:25:13.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23c9659e:::benchmark_data_smithi188_89276_object150102:head# 2024-04-23T16:25:13.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23c981ba:::benchmark_data_smithi188_89276_object121176:head# 2024-04-23T16:25:13.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ca4288:::benchmark_data_smithi188_89276_object86451:head# 2024-04-23T16:25:13.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23cbc987:::benchmark_data_smithi188_89276_object161491:head# 2024-04-23T16:25:13.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23cc2760:::benchmark_data_smithi188_89276_object93545:head# 2024-04-23T16:25:13.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23cc7679:::benchmark_data_smithi188_89276_object138750:head# 2024-04-23T16:25:13.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23cc8be8:::benchmark_data_smithi188_89276_object88146:head# 2024-04-23T16:25:13.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23cde766:::benchmark_data_smithi188_89276_object105808:head# 2024-04-23T16:25:13.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ce2305:::benchmark_data_smithi188_89276_object104991:head# 2024-04-23T16:25:13.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ce6216:::benchmark_data_smithi188_89276_object74037:head# 2024-04-23T16:25:13.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ce94e3:::benchmark_data_smithi188_89276_object108067:head# 2024-04-23T16:25:13.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23cea183:::benchmark_data_smithi188_89276_object133428:head# 2024-04-23T16:25:13.353 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23cefe72:::benchmark_data_smithi188_89276_object101608:head# 2024-04-23T16:25:13.353 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23cfb0f2:::benchmark_data_smithi188_89276_object49506:head# 2024-04-23T16:25:13.353 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23cfbfff:::benchmark_data_smithi188_89276_object114362:head# 2024-04-23T16:25:13.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23d0cb2f:::benchmark_data_smithi188_89276_object129960:head# 2024-04-23T16:25:13.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23d18757:::benchmark_data_smithi188_89276_object105891:head# 2024-04-23T16:25:13.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23d1b06d:::benchmark_data_smithi188_89276_object165799:head# 2024-04-23T16:25:13.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23d1e60f:::benchmark_data_smithi188_89276_object104971:head# 2024-04-23T16:25:13.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23d27c8b:::benchmark_data_smithi188_89276_object100303:head# 2024-04-23T16:25:13.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23d3387e:::benchmark_data_smithi188_89276_object158340:head# 2024-04-23T16:25:13.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23d378fe:::benchmark_data_smithi188_89276_object142810:head# 2024-04-23T16:25:13.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23d41573:::benchmark_data_smithi188_89276_object92120:head# 2024-04-23T16:25:13.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23d4c9a0:::benchmark_data_smithi188_89276_object41000:head# 2024-04-23T16:25:13.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23d56731:::benchmark_data_smithi188_89276_object50730:head# 2024-04-23T16:25:13.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23d5bb0b:::benchmark_data_smithi188_89276_object100794:head# 2024-04-23T16:25:13.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23d5c2a2:::benchmark_data_smithi188_89276_object133144:head# 2024-04-23T16:25:13.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23d61c65:::benchmark_data_smithi188_89276_object50369:head# 2024-04-23T16:25:13.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23d7e17b:::benchmark_data_smithi188_89276_object164067:head# 2024-04-23T16:25:13.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23d8ad01:::benchmark_data_smithi188_89276_object68048:head# 2024-04-23T16:25:13.361 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23d912e8:::benchmark_data_smithi188_89276_object49723:head# 2024-04-23T16:25:13.361 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23d9a196:::benchmark_data_smithi188_89276_object57625:head# 2024-04-23T16:25:13.361 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23da7f86:::benchmark_data_smithi188_89276_object67528:head# 2024-04-23T16:25:13.362 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23daa3f3:::benchmark_data_smithi188_89276_object42007:head# 2024-04-23T16:25:13.362 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23dae8ae:::benchmark_data_smithi188_89276_object126855:head# 2024-04-23T16:25:13.362 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23dc3e7c:::benchmark_data_smithi188_89276_object156811:head# 2024-04-23T16:25:13.363 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23dd0952:::benchmark_data_smithi188_89276_object63841:head# 2024-04-23T16:25:13.363 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23dd61cc:::benchmark_data_smithi188_89276_object66805:head# 2024-04-23T16:25:13.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23de174d:::benchmark_data_smithi188_89276_object141792:head# 2024-04-23T16:25:13.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23de2179:::benchmark_data_smithi188_89276_object103556:head# 2024-04-23T16:25:13.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23de499e:::benchmark_data_smithi188_89276_object81479:head# 2024-04-23T16:25:13.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23de53d4:::benchmark_data_smithi188_89276_object56874:head# 2024-04-23T16:25:13.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23def0b4:::benchmark_data_smithi188_89276_object150941:head# 2024-04-23T16:25:13.366 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23df2c2f:::benchmark_data_smithi188_89276_object100277:head# 2024-04-23T16:25:13.366 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23dfd172:::benchmark_data_smithi188_89276_object137983:head# 2024-04-23T16:25:13.367 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23dfe2b6:::benchmark_data_smithi188_89276_object59753:head# 2024-04-23T16:25:13.367 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e1f93f:::benchmark_data_smithi188_89276_object44754:head# 2024-04-23T16:25:13.367 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e23ced:::benchmark_data_smithi188_89276_object101808:head# 2024-04-23T16:25:13.368 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e2fe35:::benchmark_data_smithi188_89276_object149991:head# 2024-04-23T16:25:13.368 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e31ad5:::benchmark_data_smithi188_89276_object137766:head# 2024-04-23T16:25:13.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e382d5:::benchmark_data_smithi188_89276_object116235:head# 2024-04-23T16:25:13.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e3c996:::benchmark_data_smithi188_89276_object156899:head# 2024-04-23T16:25:13.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e48ba8:::benchmark_data_smithi188_89276_object44920:head# 2024-04-23T16:25:13.371 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e4be28:::benchmark_data_smithi188_89276_object107533:head# 2024-04-23T16:25:13.371 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e4d0f3:::benchmark_data_smithi188_89276_object46500:head# 2024-04-23T16:25:13.375 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e52afa:::benchmark_data_smithi188_89276_object58051:head# 2024-04-23T16:25:13.375 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e58e2a:::benchmark_data_smithi188_89276_object76343:head# 2024-04-23T16:25:13.375 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e59dbd:::benchmark_data_smithi188_89276_object137682:head# 2024-04-23T16:25:13.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e5d57e:::benchmark_data_smithi188_89276_object111245:head# 2024-04-23T16:25:13.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e5f915:::benchmark_data_smithi188_89276_object88311:head# 2024-04-23T16:25:13.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e64c3a:::benchmark_data_smithi188_89276_object45512:head# 2024-04-23T16:25:13.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e65217:::benchmark_data_smithi188_89276_object102642:head# 2024-04-23T16:25:13.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e7a09a:::benchmark_data_smithi188_89276_object120387:head# 2024-04-23T16:25:13.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e7aaa4:::benchmark_data_smithi188_89276_object99953:head# 2024-04-23T16:25:13.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e879cb:::benchmark_data_smithi188_89276_object125876:head# 2024-04-23T16:25:13.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e88e5c:::benchmark_data_smithi188_89276_object100049:head# 2024-04-23T16:25:13.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e8ff12:::benchmark_data_smithi188_89276_object102582:head# 2024-04-23T16:25:13.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e97e32:::benchmark_data_smithi188_89276_object155531:head# 2024-04-23T16:25:13.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23e99caa:::benchmark_data_smithi188_89276_object111746:head# 2024-04-23T16:25:13.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23eafeb1:::benchmark_data_smithi188_89276_object53691:head# 2024-04-23T16:25:13.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23eb166c:::benchmark_data_smithi188_89276_object60442:head# 2024-04-23T16:25:13.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ec033e:::benchmark_data_smithi188_89276_object101495:head# 2024-04-23T16:25:13.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ec7761:::benchmark_data_smithi188_89276_object106999:head# 2024-04-23T16:25:13.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ed7c92:::benchmark_data_smithi188_89276_object40594:head# 2024-04-23T16:25:13.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23eda01e:::benchmark_data_smithi188_89276_object42012:head# 2024-04-23T16:25:13.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23eded29:::benchmark_data_smithi188_89276_object99258:head# 2024-04-23T16:25:13.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23eeb5c8:::benchmark_data_smithi188_89276_object154011:head# 2024-04-23T16:25:13.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23eeb6c0:::benchmark_data_smithi188_89276_object132915:head# 2024-04-23T16:25:13.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23efc474:::benchmark_data_smithi188_89276_object138651:head# 2024-04-23T16:25:13.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23eff8f9:::benchmark_data_smithi188_89276_object118936:head# 2024-04-23T16:25:13.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f01fdc:::benchmark_data_smithi188_89276_object123662:head# 2024-04-23T16:25:13.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f02cc1:::benchmark_data_smithi188_89276_object79190:head# 2024-04-23T16:25:13.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f05a28:::benchmark_data_smithi188_89276_object42338:head# 2024-04-23T16:25:13.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f06c69:::benchmark_data_smithi188_89276_object153428:head# 2024-04-23T16:25:13.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f12e79:::benchmark_data_smithi188_89276_object80689:head# 2024-04-23T16:25:13.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f1364b:::benchmark_data_smithi188_89276_object54555:head# 2024-04-23T16:25:13.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f28b46:::benchmark_data_smithi188_89276_object111037:head# 2024-04-23T16:25:13.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f2e9ff:::benchmark_data_smithi188_89276_object103044:head# 2024-04-23T16:25:13.387 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f3342e:::benchmark_data_smithi188_89276_object148074:head# 2024-04-23T16:25:13.387 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f3aa37:::benchmark_data_smithi188_89276_object53378:head# 2024-04-23T16:25:13.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f3eb14:::benchmark_data_smithi188_89276_object78943:head# 2024-04-23T16:25:13.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f41689:::benchmark_data_smithi188_89276_object64712:head# 2024-04-23T16:25:13.389 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f4752a:::benchmark_data_smithi188_89276_object114960:head# 2024-04-23T16:25:13.389 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f50d6c:::benchmark_data_smithi188_89276_object156500:head# 2024-04-23T16:25:13.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f54550:::benchmark_data_smithi188_89276_object150837:head# 2024-04-23T16:25:13.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f56697:::benchmark_data_smithi188_89276_object69297:head# 2024-04-23T16:25:13.391 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f59634:::benchmark_data_smithi188_89276_object45164:head# 2024-04-23T16:25:13.391 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:25:13.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f5b6b0:::benchmark_data_smithi188_89276_object43878:head# 2024-04-23T16:25:13.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f622e3:::benchmark_data_smithi188_89276_object138093:head# 2024-04-23T16:25:13.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f69646:::benchmark_data_smithi188_89276_object58180:head# 2024-04-23T16:25:13.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f6966d:::benchmark_data_smithi188_89276_object84998:head# 2024-04-23T16:25:13.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f6fdd4:::benchmark_data_smithi188_89276_object127414:head# 2024-04-23T16:25:13.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f7b399:::benchmark_data_smithi188_89276_object74195:head# 2024-04-23T16:25:13.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f8774f:::benchmark_data_smithi188_89276_object46250:head# 2024-04-23T16:25:13.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f8bcc7:::benchmark_data_smithi188_89276_object80312:head# 2024-04-23T16:25:13.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f8e6b5:::benchmark_data_smithi188_89276_object93068:head# 2024-04-23T16:25:13.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f8f85e:::benchmark_data_smithi188_89276_object61323:head# 2024-04-23T16:25:13.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f90a7e:::benchmark_data_smithi188_89276_object126790:head# 2024-04-23T16:25:13.397 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23f95d65:::benchmark_data_smithi188_89276_object93423:head# 2024-04-23T16:25:13.397 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23fa3145:::benchmark_data_smithi188_89276_object92165:head# 2024-04-23T16:25:13.397 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23fa74b4:::benchmark_data_smithi188_89276_object149302:head# 2024-04-23T16:25:13.397 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23fb0c6f:::benchmark_data_smithi188_89276_object78226:head# 2024-04-23T16:25:13.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23fbacb3:::benchmark_data_smithi188_89276_object83839:head# 2024-04-23T16:25:13.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23fd1ac7:::benchmark_data_smithi188_89276_object83108:head# 2024-04-23T16:25:13.399 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23fd5d53:::benchmark_data_smithi188_89276_object123386:head# 2024-04-23T16:25:13.399 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23fe2e89:::benchmark_data_smithi188_89276_object125330:head# 2024-04-23T16:25:13.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23fe81e7:::benchmark_data_smithi188_89276_object137958:head# 2024-04-23T16:25:13.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ff0201:::benchmark_data_smithi188_89276_object56954:head# 2024-04-23T16:25:13.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ff0ad6:::benchmark_data_smithi188_89276_object157044:head# 2024-04-23T16:25:13.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ff44fa:::benchmark_data_smithi188_89276_object118026:head# 2024-04-23T16:25:13.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ff5cf4:::benchmark_data_smithi188_89276_object89509:head# 2024-04-23T16:25:13.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:23ffea68:::benchmark_data_smithi188_89276_object62364:head# 2024-04-23T16:25:13.403 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24011dd1:::benchmark_data_smithi188_89276_object45624:head# 2024-04-23T16:25:13.403 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24012343:::benchmark_data_smithi188_89276_object55113:head# 2024-04-23T16:25:13.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2401326e:::benchmark_data_smithi188_89276_object107458:head# 2024-04-23T16:25:13.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:240132d1:::benchmark_data_smithi188_89276_object54473:head# 2024-04-23T16:25:13.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2401636d:::benchmark_data_smithi188_89276_object140705:head# 2024-04-23T16:25:13.405 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24017103:::benchmark_data_smithi188_89276_object125302:head# 2024-04-23T16:25:13.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2401877a:::benchmark_data_smithi188_89276_object114676:head# 2024-04-23T16:25:13.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2401d35b:::benchmark_data_smithi188_89276_object81362:head# 2024-04-23T16:25:13.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2401f02a:::benchmark_data_smithi188_89276_object143617:head# 2024-04-23T16:25:13.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24036d68:::benchmark_data_smithi188_89276_object167134:head# 2024-04-23T16:25:13.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2404b786:::benchmark_data_smithi188_89276_object64181:head# 2024-04-23T16:25:13.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2404f261:::benchmark_data_smithi188_89276_object56077:head# 2024-04-23T16:25:13.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2405834c:::benchmark_data_smithi188_89276_object87400:head# 2024-04-23T16:25:13.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24060ec4:::benchmark_data_smithi188_89276_object146148:head# 2024-04-23T16:25:13.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24077fc6:::benchmark_data_smithi188_89276_object110211:head# 2024-04-23T16:25:13.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24086db2:::benchmark_data_smithi188_89276_object47855:head# 2024-04-23T16:25:13.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2408b32c:::benchmark_data_smithi188_89276_object138397:head# 2024-04-23T16:25:13.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24095f77:::benchmark_data_smithi188_89276_object48098:head# 2024-04-23T16:25:13.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24097b41:::benchmark_data_smithi188_89276_object82682:head# 2024-04-23T16:25:13.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2409c51f:::benchmark_data_smithi188_89276_object144210:head# 2024-04-23T16:25:13.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:240a7d92:::benchmark_data_smithi188_89276_object161482:head# 2024-04-23T16:25:13.412 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:240a995d:::benchmark_data_smithi188_89276_object86674:head# 2024-04-23T16:25:13.412 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:240af54d:::benchmark_data_smithi188_89276_object117842:head# 2024-04-23T16:25:13.412 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:240b8127:::benchmark_data_smithi188_89276_object71822:head# 2024-04-23T16:25:13.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:240bb2d1:::benchmark_data_smithi188_89276_object145028:head# 2024-04-23T16:25:13.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:240bdd2e:::benchmark_data_smithi188_89276_object42313:head# 2024-04-23T16:25:13.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:240c3fec:::benchmark_data_smithi188_89276_object58358:head# 2024-04-23T16:25:13.414 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:240e5f28:::benchmark_data_smithi188_89276_object81525:head# 2024-04-23T16:25:13.414 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:240ed26a:::benchmark_data_smithi188_89276_object60257:head# 2024-04-23T16:25:13.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:240f470e:::benchmark_data_smithi188_89276_object142357:head# 2024-04-23T16:25:13.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:240faf25:::benchmark_data_smithi188_89276_object49252:head# 2024-04-23T16:25:13.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:241026e9:::benchmark_data_smithi188_89276_object86556:head# 2024-04-23T16:25:13.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24104dab:::benchmark_data_smithi188_89276_object122940:head# 2024-04-23T16:25:13.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2410cdf7:::benchmark_data_smithi188_89276_object86875:head# 2024-04-23T16:25:13.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24116c32:::benchmark_data_smithi188_89276_object136345:head# 2024-04-23T16:25:13.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24116d8a:::benchmark_data_smithi188_89276_object89278:head# 2024-04-23T16:25:13.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:241242bf:::benchmark_data_smithi188_89276_object117362:head# 2024-04-23T16:25:13.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:241284d3:::benchmark_data_smithi188_89276_object89355:head# 2024-04-23T16:25:13.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2412d887:::benchmark_data_smithi188_89276_object62592:head# 2024-04-23T16:25:13.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:241559f1:::benchmark_data_smithi188_89276_object73424:head# 2024-04-23T16:25:13.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24158ace:::benchmark_data_smithi188_89276_object121569:head# 2024-04-23T16:25:13.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:241681ee:::benchmark_data_smithi188_89276_object80397:head# 2024-04-23T16:25:13.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24171ee6:::benchmark_data_smithi188_89276_object166243:head# 2024-04-23T16:25:13.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:241761a9:::benchmark_data_smithi188_89276_object137767:head# 2024-04-23T16:25:13.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2418d279:::benchmark_data_smithi188_89276_object130915:head# 2024-04-23T16:25:13.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2418e30a:::benchmark_data_smithi188_89276_object69824:head# 2024-04-23T16:25:13.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2418e9aa:::benchmark_data_smithi188_89276_object94914:head# 2024-04-23T16:25:13.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2419451b:::benchmark_data_smithi188_89276_object144075:head# 2024-04-23T16:25:13.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24196e84:::benchmark_data_smithi188_89276_object65859:head# 2024-04-23T16:25:13.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2419a49e:::benchmark_data_smithi188_89276_object80730:head# 2024-04-23T16:25:13.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2419f076:::benchmark_data_smithi188_89276_object69252:head# 2024-04-23T16:25:13.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:241a105e:::benchmark_data_smithi188_89276_object155587:head# 2024-04-23T16:25:13.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:241a4504:::benchmark_data_smithi188_89276_object108596:head# 2024-04-23T16:25:13.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:241a58eb:::benchmark_data_smithi188_89276_object74263:head# 2024-04-23T16:25:13.426 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:241b390a:::benchmark_data_smithi188_89276_object135245:head# 2024-04-23T16:25:13.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:241b8a0d:::benchmark_data_smithi188_89276_object136933:head# 2024-04-23T16:25:13.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:241babf4:::benchmark_data_smithi188_89276_object83445:head# 2024-04-23T16:25:13.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:241d8543:::benchmark_data_smithi188_89276_object153720:head# 2024-04-23T16:25:13.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:241dee10:::benchmark_data_smithi188_89276_object132300:head# 2024-04-23T16:25:13.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:241e068d:::benchmark_data_smithi188_89276_object119829:head# 2024-04-23T16:25:13.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:241f0731:::benchmark_data_smithi188_89276_object78091:head# 2024-04-23T16:25:13.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:241fbc07:::benchmark_data_smithi188_89276_object82824:head# 2024-04-23T16:25:13.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24204665:::benchmark_data_smithi188_89276_object69559:head# 2024-04-23T16:25:13.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24208f88:::benchmark_data_smithi188_89276_object80111:head# 2024-04-23T16:25:13.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242093e8:::benchmark_data_smithi188_89276_object122090:head# 2024-04-23T16:25:13.431 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2420991c:::benchmark_data_smithi188_89276_object149814:head# 2024-04-23T16:25:13.431 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242171d7:::benchmark_data_smithi188_89276_object40654:head# 2024-04-23T16:25:13.432 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2421c6ae:::benchmark_data_smithi188_89276_object67444:head# 2024-04-23T16:25:13.432 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24226ab1:::benchmark_data_smithi188_89276_object42649:head# 2024-04-23T16:25:13.432 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2422a5e8:::benchmark_data_smithi188_89276_object82655:head# 2024-04-23T16:25:13.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2422ff05:::benchmark_data_smithi188_89276_object101418:head# 2024-04-23T16:25:13.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242301db:::benchmark_data_smithi188_89276_object43221:head# 2024-04-23T16:25:13.434 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2423a149:::benchmark_data_smithi188_89276_object152189:head# 2024-04-23T16:25:13.434 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2423dd9d:::benchmark_data_smithi188_89276_object80756:head# 2024-04-23T16:25:13.435 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2423f746:::benchmark_data_smithi188_89276_object57505:head# 2024-04-23T16:25:13.435 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24244826:::benchmark_data_smithi188_89276_object159879:head# 2024-04-23T16:25:13.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24245a8b:::benchmark_data_smithi188_89276_object68481:head# 2024-04-23T16:25:13.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2424c110:::benchmark_data_smithi188_89276_object140526:head# 2024-04-23T16:25:13.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2425515f:::benchmark_data_smithi188_89276_object74576:head# 2024-04-23T16:25:13.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2425b6cd:::benchmark_data_smithi188_89276_object164873:head# 2024-04-23T16:25:13.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2425d419:::benchmark_data_smithi188_89276_object53965:head# 2024-04-23T16:25:13.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2425eb60:::benchmark_data_smithi188_89276_object95435:head# 2024-04-23T16:25:13.438 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24279682:::benchmark_data_smithi188_89276_object90811:head# 2024-04-23T16:25:13.438 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:13.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2427add2:::benchmark_data_smithi188_89276_object162265:head# 2024-04-23T16:25:13.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242928e1:::benchmark_data_smithi188_89276_object50726:head# 2024-04-23T16:25:13.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24293cf1:::benchmark_data_smithi188_89276_object81475:head# 2024-04-23T16:25:13.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242adb40:::benchmark_data_smithi188_89276_object125254:head# 2024-04-23T16:25:13.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242b26e0:::benchmark_data_smithi188_89276_object148632:head# 2024-04-23T16:25:13.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242b5376:::benchmark_data_smithi188_89276_object45675:head# 2024-04-23T16:25:13.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242b8d25:::benchmark_data_smithi188_89276_object148028:head# 2024-04-23T16:25:13.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242ba5c6:::benchmark_data_smithi188_89276_object71010:head# 2024-04-23T16:25:13.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242bb4e7:::benchmark_data_smithi188_89276_object85440:head# 2024-04-23T16:25:13.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242bc1d9:::benchmark_data_smithi188_89276_object125122:head# 2024-04-23T16:25:13.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242bc99b:::benchmark_data_smithi188_89276_object94880:head# 2024-04-23T16:25:13.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242bea01:::benchmark_data_smithi188_89276_object163910:head# 2024-04-23T16:25:13.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242c40a7:::benchmark_data_smithi188_89276_object111452:head# 2024-04-23T16:25:13.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242c9e3a:::benchmark_data_smithi188_89276_object79686:head# 2024-04-23T16:25:13.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242cf77a:::benchmark_data_smithi188_89276_object126862:head# 2024-04-23T16:25:13.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242d3440:::benchmark_data_smithi188_89276_object119228:head# 2024-04-23T16:25:13.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242d6ac5:::benchmark_data_smithi188_89276_object103304:head# 2024-04-23T16:25:13.446 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242d7191:::benchmark_data_smithi188_89276_object99740:head# 2024-04-23T16:25:13.446 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242e5bd6:::benchmark_data_smithi188_89276_object157814:head# 2024-04-23T16:25:13.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242ec586:::benchmark_data_smithi188_89276_object59770:head# 2024-04-23T16:25:13.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242f5a2c:::benchmark_data_smithi188_89276_object147503:head# 2024-04-23T16:25:13.448 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:242f8678:::benchmark_data_smithi188_89276_object137214:head# 2024-04-23T16:25:13.448 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24307a56:::benchmark_data_smithi188_89276_object122978:head# 2024-04-23T16:25:13.449 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2431807d:::benchmark_data_smithi188_89276_object106094:head# 2024-04-23T16:25:13.449 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24319340:::benchmark_data_smithi188_89276_object140038:head# 2024-04-23T16:25:13.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2431c33f:::benchmark_data_smithi188_89276_object54261:head# 2024-04-23T16:25:13.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243310fa:::benchmark_data_smithi188_89276_object78597:head# 2024-04-23T16:25:13.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243424b4:::benchmark_data_smithi188_89276_object121301:head# 2024-04-23T16:25:13.451 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24349a9f:::benchmark_data_smithi188_89276_object49276:head# 2024-04-23T16:25:13.451 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243522bb:::benchmark_data_smithi188_89276_object150317:head# 2024-04-23T16:25:13.452 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2435e4de:::benchmark_data_smithi188_89276_object120362:head# 2024-04-23T16:25:13.452 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2436abaf:::benchmark_data_smithi188_89276_object66357:head# 2024-04-23T16:25:13.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24382565:::benchmark_data_smithi188_89276_object161729:head# 2024-04-23T16:25:13.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24382b4c:::benchmark_data_smithi188_89276_object163757:head# 2024-04-23T16:25:13.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243873eb:::benchmark_data_smithi188_89276_object118929:head# 2024-04-23T16:25:13.454 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2439e289:::benchmark_data_smithi188_89276_object125372:head# 2024-04-23T16:25:13.454 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243a31fc:::benchmark_data_smithi188_89276_object153281:head# 2024-04-23T16:25:13.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243a6438:::benchmark_data_smithi188_89276_object104445:head# 2024-04-23T16:25:13.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243a6481:::benchmark_data_smithi188_89276_object116152:head# 2024-04-23T16:25:13.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243a6982:::benchmark_data_smithi188_89276_object72752:head# 2024-04-23T16:25:13.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243a7f21:::benchmark_data_smithi188_89276_object124841:head# 2024-04-23T16:25:13.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243b61a3:::benchmark_data_smithi188_89276_object62908:head# 2024-04-23T16:25:13.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243b8314:::benchmark_data_smithi188_89276_object67559:head# 2024-04-23T16:25:13.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243baad6:::benchmark_data_smithi188_89276_object129893:head# 2024-04-23T16:25:13.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243c0d09:::benchmark_data_smithi188_89276_object114651:head# 2024-04-23T16:25:13.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243c78b0:::benchmark_data_smithi188_89276_object159986:head# 2024-04-23T16:25:13.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243d104d:::benchmark_data_smithi188_89276_object76221:head# 2024-04-23T16:25:13.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243d65ce:::benchmark_data_smithi188_89276_object43595:head# 2024-04-23T16:25:13.460 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243e3010:::benchmark_data_smithi188_89276_object137270:head# 2024-04-23T16:25:13.460 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243f2466:::benchmark_data_smithi188_89276_object60652:head# 2024-04-23T16:25:13.461 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243f86d3:::benchmark_data_smithi188_89276_object92084:head# 2024-04-23T16:25:13.461 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:243fc182:::benchmark_data_smithi188_89276_object57863:head# 2024-04-23T16:25:13.462 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2440c65f:::benchmark_data_smithi188_89276_object154995:head# 2024-04-23T16:25:13.462 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2441096f:::benchmark_data_smithi188_89276_object78018:head# 2024-04-23T16:25:13.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24412d74:::benchmark_data_smithi188_89276_object73479:head# 2024-04-23T16:25:13.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:244193f6:::benchmark_data_smithi188_89276_object91498:head# 2024-04-23T16:25:13.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2441b021:::benchmark_data_smithi188_89276_object114677:head# 2024-04-23T16:25:13.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24422bca:::benchmark_data_smithi188_89276_object45741:head# 2024-04-23T16:25:13.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2442b7f6:::benchmark_data_smithi188_89276_object154714:head# 2024-04-23T16:25:13.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24431e03:::benchmark_data_smithi188_89276_object83534:head# 2024-04-23T16:25:13.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2443ced6:::benchmark_data_smithi188_89276_object81609:head# 2024-04-23T16:25:13.466 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:25:13.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:244415c0:::benchmark_data_smithi188_89276_object148114:head# 2024-04-23T16:25:13.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2444295b:::benchmark_data_smithi188_89276_object116878:head# 2024-04-23T16:25:13.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2444518d:::benchmark_data_smithi188_89276_object66019:head# 2024-04-23T16:25:13.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2444dee3:::benchmark_data_smithi188_89276_object123568:head# 2024-04-23T16:25:13.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2445b568:::benchmark_data_smithi188_89276_object119101:head# 2024-04-23T16:25:13.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2445d3ac:::benchmark_data_smithi188_89276_object151052:head# 2024-04-23T16:25:13.469 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24462c73:::benchmark_data_smithi188_89276_object143458:head# 2024-04-23T16:25:13.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:13.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:25:13.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24470edc:::benchmark_data_smithi188_89276_object57014:head# 2024-04-23T16:25:13.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2447683a:::benchmark_data_smithi188_89276_object113686:head# 2024-04-23T16:25:13.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:244787bd:::benchmark_data_smithi188_89276_object79723:head# 2024-04-23T16:25:13.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2447ed36:::benchmark_data_smithi188_89276_object127192:head# 2024-04-23T16:25:13.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24483115:::benchmark_data_smithi188_89276_object114264:head# 2024-04-23T16:25:13.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24484c93:::benchmark_data_smithi188_89276_object146284:head# 2024-04-23T16:25:13.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24485f20:::benchmark_data_smithi188_89276_object132874:head# 2024-04-23T16:25:13.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2448ca8b:::benchmark_data_smithi188_89276_object155817:head# 2024-04-23T16:25:13.473 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:244a457b:::benchmark_data_smithi188_89276_object158903:head# 2024-04-23T16:25:13.473 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:244a9193:::benchmark_data_smithi188_89276_object146744:head# 2024-04-23T16:25:13.473 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:244b74ff:::benchmark_data_smithi188_89276_object77362:head# 2024-04-23T16:25:13.474 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:244b7e8e:::benchmark_data_smithi188_89276_object157518:head# 2024-04-23T16:25:13.474 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:244c0ff8:::benchmark_data_smithi188_89276_object103203:head# 2024-04-23T16:25:13.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:244c95cd:::benchmark_data_smithi188_89276_object142186:head# 2024-04-23T16:25:13.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:244cf726:::benchmark_data_smithi188_89276_object161680:head# 2024-04-23T16:25:13.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:244d2deb:::benchmark_data_smithi188_89276_object64241:head# 2024-04-23T16:25:13.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:244f0ffc:::benchmark_data_smithi188_89276_object86678:head# 2024-04-23T16:25:13.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:244f9531:::benchmark_data_smithi188_89276_object150113:head# 2024-04-23T16:25:13.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:244fb72e:::benchmark_data_smithi188_89276_object130329:head# 2024-04-23T16:25:13.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:244fb9f1:::benchmark_data_smithi188_89276_object147279:head# 2024-04-23T16:25:13.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:245008ac:::benchmark_data_smithi188_89276_object130180:head# 2024-04-23T16:25:13.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24500f88:::benchmark_data_smithi188_89276_object148355:head# 2024-04-23T16:25:13.481 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24508e81:::benchmark_data_smithi188_89276_object71113:head# 2024-04-23T16:25:13.486 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2450c60f:::benchmark_data_smithi188_89276_object105247:head# 2024-04-23T16:25:13.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24521a2d:::benchmark_data_smithi188_89276_object152228:head# 2024-04-23T16:25:13.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24523460:::benchmark_data_smithi188_89276_object134876:head# 2024-04-23T16:25:13.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:245263af:::benchmark_data_smithi188_89276_object62344:head# 2024-04-23T16:25:13.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2452e154:::benchmark_data_smithi188_89276_object164515:head# 2024-04-23T16:25:13.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2452ee37:::benchmark_data_smithi188_89276_object45787:head# 2024-04-23T16:25:13.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24532272:::benchmark_data_smithi188_89276_object130890:head# 2024-04-23T16:25:13.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24532382:::benchmark_data_smithi188_89276_object143353:head# 2024-04-23T16:25:13.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2454e2e5:::benchmark_data_smithi188_89276_object159566:head# 2024-04-23T16:25:13.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2454f4d2:::benchmark_data_smithi188_89276_object80317:head# 2024-04-23T16:25:13.488 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24553952:::benchmark_data_smithi188_89276_object132907:head# 2024-04-23T16:25:13.488 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24557557:::benchmark_data_smithi188_89276_object101144:head# 2024-04-23T16:25:13.488 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24560ebb:::benchmark_data_smithi188_89276_object106459:head# 2024-04-23T16:25:13.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2456298a:::benchmark_data_smithi188_89276_object94283:head# 2024-04-23T16:25:13.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2456f42f:::benchmark_data_smithi188_89276_object41947:head# 2024-04-23T16:25:13.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2456f5bb:::benchmark_data_smithi188_89276_object111075:head# 2024-04-23T16:25:13.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2457654b:::benchmark_data_smithi188_89276_object147071:head# 2024-04-23T16:25:13.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2459ea07:::benchmark_data_smithi188_89276_object166260:head# 2024-04-23T16:25:13.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:245baca2:::benchmark_data_smithi188_89276_object135192:head# 2024-04-23T16:25:13.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:245c333e:::benchmark_data_smithi188_89276_object47847:head# 2024-04-23T16:25:13.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:245c829b:::benchmark_data_smithi188_89276_object58854:head# 2024-04-23T16:25:13.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:245cb2d8:::benchmark_data_smithi188_89276_object139147:head# 2024-04-23T16:25:13.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:245e881d:::benchmark_data_smithi188_89276_object118234:head# 2024-04-23T16:25:13.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:245f45b7:::benchmark_data_smithi188_89276_object166276:head# 2024-04-23T16:25:13.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:246008ee:::benchmark_data_smithi188_89276_object136822:head# 2024-04-23T16:25:13.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24601567:::benchmark_data_smithi188_89276_object145635:head# 2024-04-23T16:25:13.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24612939:::benchmark_data_smithi188_89276_object79786:head# 2024-04-23T16:25:13.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:246135c2:::benchmark_data_smithi188_89276_object164649:head# 2024-04-23T16:25:13.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:246154d8:::benchmark_data_smithi188_89276_object156897:head# 2024-04-23T16:25:13.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2462ecca:::benchmark_data_smithi188_89276_object43718:head# 2024-04-23T16:25:13.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24631a22:::benchmark_data_smithi188_89276_object103171:head# 2024-04-23T16:25:13.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2463c780:::benchmark_data_smithi188_89276_object90395:head# 2024-04-23T16:25:13.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2464f56c:::benchmark_data_smithi188_89276_object71775:head# 2024-04-23T16:25:13.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:246509a8:::benchmark_data_smithi188_89276_object116647:head# 2024-04-23T16:25:13.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24663db3:::benchmark_data_smithi188_89276_object46241:head# 2024-04-23T16:25:13.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24666ad5:::benchmark_data_smithi188_89276_object44264:head# 2024-04-23T16:25:13.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24675a1b:::benchmark_data_smithi188_89276_object163921:head# 2024-04-23T16:25:13.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2467c126:::benchmark_data_smithi188_89276_object160203:head# 2024-04-23T16:25:13.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2468bdfc:::benchmark_data_smithi188_89276_object51053:head# 2024-04-23T16:25:13.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2468d024:::benchmark_data_smithi188_89276_object136916:head# 2024-04-23T16:25:13.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2468fc62:::benchmark_data_smithi188_89276_object124761:head# 2024-04-23T16:25:13.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24690a2c:::benchmark_data_smithi188_89276_object64951:head# 2024-04-23T16:25:13.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2469aeaa:::benchmark_data_smithi188_89276_object133118:head# 2024-04-23T16:25:13.516 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:246aca71:::benchmark_data_smithi188_89276_object167098:head# 2024-04-23T16:25:13.516 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:246b236a:::benchmark_data_smithi188_89276_object150477:head# 2024-04-23T16:25:13.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:246b6c5c:::benchmark_data_smithi188_89276_object154490:head# 2024-04-23T16:25:13.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:246b8c6f:::benchmark_data_smithi188_89276_object81611:head# 2024-04-23T16:25:13.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:246d22db:::benchmark_data_smithi188_89276_object101892:head# 2024-04-23T16:25:13.521 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:246d501f:::benchmark_data_smithi188_89276_object137197:head# 2024-04-23T16:25:13.521 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:246d7dc0:::benchmark_data_smithi188_89276_object58686:head# 2024-04-23T16:25:13.521 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:246d9137:::benchmark_data_smithi188_89276_object59848:head# 2024-04-23T16:25:13.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:246dce8d:::benchmark_data_smithi188_89276_object151370:head# 2024-04-23T16:25:13.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:246e0af9:::benchmark_data_smithi188_89276_object66183:head# 2024-04-23T16:25:13.523 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:246e45b0:::benchmark_data_smithi188_89276_object132123:head# 2024-04-23T16:25:13.523 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:246ec0df:::benchmark_data_smithi188_89276_object116895:head# 2024-04-23T16:25:13.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:246f557b:::benchmark_data_smithi188_89276_object75777:head# 2024-04-23T16:25:13.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:246f8c9f:::benchmark_data_smithi188_89276_object62584:head# 2024-04-23T16:25:13.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2470818b:::benchmark_data_smithi188_89276_object125524:head# 2024-04-23T16:25:13.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2470a0a1:::benchmark_data_smithi188_89276_object128024:head# 2024-04-23T16:25:13.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2470caa5:::benchmark_data_smithi188_89276_object78913:head# 2024-04-23T16:25:13.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24711c14:::benchmark_data_smithi188_89276_object41410:head# 2024-04-23T16:25:13.531 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24722f3e:::benchmark_data_smithi188_89276_object76949:head# 2024-04-23T16:25:13.531 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24739b35:::benchmark_data_smithi188_89276_object127487:head# 2024-04-23T16:25:13.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2473ced4:::benchmark_data_smithi188_89276_object69077:head# 2024-04-23T16:25:13.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2474f0f7:::benchmark_data_smithi188_89276_object149876:head# 2024-04-23T16:25:13.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24753833:::benchmark_data_smithi188_89276_object60533:head# 2024-04-23T16:25:13.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2475423f:::benchmark_data_smithi188_89276_object65917:head# 2024-04-23T16:25:13.536 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24764e4e:::benchmark_data_smithi188_89276_object117363:head# 2024-04-23T16:25:13.536 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:247808a3:::benchmark_data_smithi188_89276_object42532:head# 2024-04-23T16:25:13.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24797ad5:::benchmark_data_smithi188_89276_object151027:head# 2024-04-23T16:25:13.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:247a44fa:::benchmark_data_smithi188_89276_object85154:head# 2024-04-23T16:25:13.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:247a6756:::benchmark_data_smithi188_89276_object149080:head# 2024-04-23T16:25:13.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:247b650f:::benchmark_data_smithi188_89276_object97953:head# 2024-04-23T16:25:13.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:247cef35:::benchmark_data_smithi188_89276_object143848:head# 2024-04-23T16:25:13.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:247d1052:::benchmark_data_smithi188_89276_object63641:head# 2024-04-23T16:25:13.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:247d851a:::benchmark_data_smithi188_89276_object100014:head# 2024-04-23T16:25:13.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:247d8775:::benchmark_data_smithi188_89276_object116114:head# 2024-04-23T16:25:13.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:247e69b2:::benchmark_data_smithi188_89276_object55106:head# 2024-04-23T16:25:13.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:247ec81e:::benchmark_data_smithi188_89276_object80183:head# 2024-04-23T16:25:13.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:247f41b7:::benchmark_data_smithi188_89276_object58786:head# 2024-04-23T16:25:13.544 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:247f463b:::benchmark_data_smithi188_89276_object68005:head# 2024-04-23T16:25:13.544 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2480a080:::benchmark_data_smithi188_89276_object150948:head# 2024-04-23T16:25:13.545 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2480a4b8:::benchmark_data_smithi188_89276_object98348:head# 2024-04-23T16:25:13.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24811405:::benchmark_data_smithi188_89276_object47025:head# 2024-04-23T16:25:13.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2481338c:::benchmark_data_smithi188_89276_object105121:head# 2024-04-23T16:25:13.547 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2481a77a:::benchmark_data_smithi188_89276_object165875:head# 2024-04-23T16:25:13.547 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24824aee:::benchmark_data_smithi188_89276_object40515:head# 2024-04-23T16:25:13.547 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24835f5c:::benchmark_data_smithi188_89276_object103901:head# 2024-04-23T16:25:13.548 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:248391b8:::benchmark_data_smithi188_89276_object124638:head# 2024-04-23T16:25:13.548 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:248397bc:::benchmark_data_smithi188_89276_object123479:head# 2024-04-23T16:25:13.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24848f20:::benchmark_data_smithi188_89276_object87185:head# 2024-04-23T16:25:13.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2484c59f:::benchmark_data_smithi188_89276_object137459:head# 2024-04-23T16:25:13.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:248604e8:::benchmark_data_smithi188_89276_object139518:head# 2024-04-23T16:25:13.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24869453:::benchmark_data_smithi188_89276_object133935:head# 2024-04-23T16:25:13.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24876ffe:::benchmark_data_smithi188_89276_object103579:head# 2024-04-23T16:25:13.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:248a142f:::benchmark_data_smithi188_89276_object165684:head# 2024-04-23T16:25:13.553 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:248aa3cf:::benchmark_data_smithi188_89276_object42585:head# 2024-04-23T16:25:13.553 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:248aa733:::benchmark_data_smithi188_89276_object118003:head# 2024-04-23T16:25:13.554 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:248ad04b:::benchmark_data_smithi188_89276_object108604:head# 2024-04-23T16:25:13.554 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:248adf91:::benchmark_data_smithi188_89276_object120762:head# 2024-04-23T16:25:13.555 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:248b43c1:::benchmark_data_smithi188_89276_object161389:head# 2024-04-23T16:25:13.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:248ccb9a:::benchmark_data_smithi188_89276_object153200:head# 2024-04-23T16:25:13.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:248fe693:::benchmark_data_smithi188_89276_object131730:head# 2024-04-23T16:25:13.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24909734:::benchmark_data_smithi188_89276_object159560:head# 2024-04-23T16:25:13.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24909e5f:::benchmark_data_smithi188_89276_object66668:head# 2024-04-23T16:25:13.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2490a2ca:::benchmark_data_smithi188_89276_object131414:head# 2024-04-23T16:25:13.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2491340b:::benchmark_data_smithi188_89276_object66564:head# 2024-04-23T16:25:13.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2492b145:::benchmark_data_smithi188_89276_object80193:head# 2024-04-23T16:25:13.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24931c44:::benchmark_data_smithi188_89276_object56670:head# 2024-04-23T16:25:13.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2493b53c:::benchmark_data_smithi188_89276_object168099:head# 2024-04-23T16:25:13.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2494150b:::benchmark_data_smithi188_89276_object136512:head# 2024-04-23T16:25:13.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24949bff:::benchmark_data_smithi188_89276_object61211:head# 2024-04-23T16:25:13.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2494f865:::benchmark_data_smithi188_89276_object133315:head# 2024-04-23T16:25:13.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24965be8:::benchmark_data_smithi188_89276_object68378:head# 2024-04-23T16:25:13.564 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24981d3a:::benchmark_data_smithi188_89276_object72576:head# 2024-04-23T16:25:13.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2498287a:::benchmark_data_smithi188_89276_object166639:head# 2024-04-23T16:25:13.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24983ef0:::benchmark_data_smithi188_89276_object115299:head# 2024-04-23T16:25:13.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2498b2df:::benchmark_data_smithi188_89276_object122392:head# 2024-04-23T16:25:13.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24991a9e:::benchmark_data_smithi188_89276_object128770:head# 2024-04-23T16:25:13.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24993dac:::benchmark_data_smithi188_89276_object81587:head# 2024-04-23T16:25:13.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:249959f1:::benchmark_data_smithi188_89276_object115379:head# 2024-04-23T16:25:13.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:249a5c11:::benchmark_data_smithi188_89276_object52641:head# 2024-04-23T16:25:13.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:249c3177:::benchmark_data_smithi188_89276_object98455:head# 2024-04-23T16:25:13.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:249c3f38:::benchmark_data_smithi188_89276_object53648:head# 2024-04-23T16:25:13.569 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:249cff30:::benchmark_data_smithi188_89276_object154710:head# 2024-04-23T16:25:13.569 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:249e0d28:::benchmark_data_smithi188_89276_object69945:head# 2024-04-23T16:25:13.570 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:249e6c69:::benchmark_data_smithi188_89276_object104458:head# 2024-04-23T16:25:13.570 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:13.571 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:249ebe0b:::benchmark_data_smithi188_89276_object116084:head# 2024-04-23T16:25:13.571 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:249ed835:::benchmark_data_smithi188_89276_object46332:head# 2024-04-23T16:25:13.571 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24a1da5d:::benchmark_data_smithi188_89276_object105551:head# 2024-04-23T16:25:13.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24a24305:::benchmark_data_smithi188_89276_object67132:head# 2024-04-23T16:25:13.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24a2c15f:::benchmark_data_smithi188_89276_object59366:head# 2024-04-23T16:25:13.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24a35dce:::benchmark_data_smithi188_89276_object121029:head# 2024-04-23T16:25:13.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24a38f49:::benchmark_data_smithi188_89276_object85495:head# 2024-04-23T16:25:13.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24a3a91d:::benchmark_data_smithi188_89276_object67576:head# 2024-04-23T16:25:13.574 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24a41f32:::benchmark_data_smithi188_89276_object58574:head# 2024-04-23T16:25:13.574 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24a43e4b:::benchmark_data_smithi188_89276_object94765:head# 2024-04-23T16:25:13.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24a44439:::benchmark_data_smithi188_89276_object46914:head# 2024-04-23T16:25:13.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24a4a26b:::benchmark_data_smithi188_89276_object109785:head# 2024-04-23T16:25:13.576 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24a56476:::benchmark_data_smithi188_89276_object68134:head# 2024-04-23T16:25:13.577 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24a5cf7a:::benchmark_data_smithi188_89276_object139407:head# 2024-04-23T16:25:13.577 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24a72c9e:::benchmark_data_smithi188_89276_object75778:head# 2024-04-23T16:25:13.577 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24a7b51b:::benchmark_data_smithi188_89276_object136591:head# 2024-04-23T16:25:13.577 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24a9a1b2:::benchmark_data_smithi188_89276_object92191:head# 2024-04-23T16:25:13.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24ab3978:::benchmark_data_smithi188_89276_object125898:head# 2024-04-23T16:25:13.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24ac4d2f:::benchmark_data_smithi188_89276_object77809:head# 2024-04-23T16:25:13.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24aded27:::benchmark_data_smithi188_89276_object93842:head# 2024-04-23T16:25:13.579 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24adfe27:::benchmark_data_smithi188_89276_object54111:head# 2024-04-23T16:25:13.579 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24ae9a0b:::benchmark_data_smithi188_89276_object61675:head# 2024-04-23T16:25:13.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24aeef10:::benchmark_data_smithi188_89276_object140055:head# 2024-04-23T16:25:13.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24affd6a:::benchmark_data_smithi188_89276_object59829:head# 2024-04-23T16:25:13.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24b0306e:::benchmark_data_smithi188_89276_object132525:head# 2024-04-23T16:25:13.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24b0580d:::benchmark_data_smithi188_89276_object116865:head# 2024-04-23T16:25:13.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24b0ce94:::benchmark_data_smithi188_89276_object48558:head# 2024-04-23T16:25:13.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24b18c1f:::benchmark_data_smithi188_89276_object166598:head# 2024-04-23T16:25:13.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24b26cc7:::benchmark_data_smithi188_89276_object40964:head# 2024-04-23T16:25:13.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24b41e96:::benchmark_data_smithi188_89276_object122051:head# 2024-04-23T16:25:13.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24b5100e:::benchmark_data_smithi188_89276_object48224:head# 2024-04-23T16:25:13.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24b587ba:::benchmark_data_smithi188_89276_object141158:head# 2024-04-23T16:25:13.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24b5a11a:::benchmark_data_smithi188_89276_object101079:head# 2024-04-23T16:25:13.585 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24b5d2e4:::benchmark_data_smithi188_89276_object65920:head# 2024-04-23T16:25:13.585 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24b6294a:::benchmark_data_smithi188_89276_object93410:head# 2024-04-23T16:25:13.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24b62c5b:::benchmark_data_smithi188_89276_object144377:head# 2024-04-23T16:25:13.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24b6a2f3:::benchmark_data_smithi188_89276_object106551:head# 2024-04-23T16:25:13.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24b74528:::benchmark_data_smithi188_89276_object71879:head# 2024-04-23T16:25:13.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24b7c91d:::benchmark_data_smithi188_89276_object112514:head# 2024-04-23T16:25:13.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24b8cbf7:::benchmark_data_smithi188_89276_object123358:head# 2024-04-23T16:25:13.588 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24ba3b93:::benchmark_data_smithi188_89276_object48544:head# 2024-04-23T16:25:13.588 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24ba6744:::benchmark_data_smithi188_89276_object117557:head# 2024-04-23T16:25:13.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24ba9ea4:::benchmark_data_smithi188_89276_object44240:head# 2024-04-23T16:25:13.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24bb124b:::benchmark_data_smithi188_89276_object119171:head# 2024-04-23T16:25:13.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24bb16fc:::benchmark_data_smithi188_89276_object95116:head# 2024-04-23T16:25:13.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24bb6889:::benchmark_data_smithi188_89276_object77407:head# 2024-04-23T16:25:13.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24bbfc10:::benchmark_data_smithi188_89276_object147991:head# 2024-04-23T16:25:13.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24bc0da9:::benchmark_data_smithi188_89276_object143910:head# 2024-04-23T16:25:13.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24bc7869:::benchmark_data_smithi188_89276_object129237:head# 2024-04-23T16:25:13.592 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24bcc709:::benchmark_data_smithi188_89276_object138252:head# 2024-04-23T16:25:13.592 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24bd5ab4:::benchmark_data_smithi188_89276_object68178:head# 2024-04-23T16:25:13.593 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24bd7cfc:::benchmark_data_smithi188_89276_object130057:head# 2024-04-23T16:25:13.593 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24be674a:::benchmark_data_smithi188_89276_object76351:head# 2024-04-23T16:25:13.594 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24bf205e:::benchmark_data_smithi188_89276_object42513:head# 2024-04-23T16:25:13.594 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24c08d81:::benchmark_data_smithi188_89276_object67008:head# 2024-04-23T16:25:13.598 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24c1144d:::benchmark_data_smithi188_89276_object110415:head# 2024-04-23T16:25:13.598 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24c1380c:::benchmark_data_smithi188_89276_object57303:head# 2024-04-23T16:25:13.598 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24c1bfd7:::benchmark_data_smithi188_89276_object40384:head# 2024-04-23T16:25:13.598 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24c330b6:::benchmark_data_smithi188_89276_object121940:head# 2024-04-23T16:25:13.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24c37e24:::benchmark_data_smithi188_89276_object127915:head# 2024-04-23T16:25:13.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24c38714:::benchmark_data_smithi188_89276_object149328:head# 2024-04-23T16:25:13.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24c3a0ba:::benchmark_data_smithi188_89276_object155615:head# 2024-04-23T16:25:13.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24c3cb3d:::benchmark_data_smithi188_89276_object118559:head# 2024-04-23T16:25:13.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24c435c5:::benchmark_data_smithi188_89276_object71014:head# 2024-04-23T16:25:13.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24c4450f:::benchmark_data_smithi188_89276_object56018:head# 2024-04-23T16:25:13.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24c482e8:::benchmark_data_smithi188_89276_object59178:head# 2024-04-23T16:25:13.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24c5395b:::benchmark_data_smithi188_89276_object116809:head# 2024-04-23T16:25:13.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24c5652d:::benchmark_data_smithi188_89276_object103909:head# 2024-04-23T16:25:13.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24c5bf95:::benchmark_data_smithi188_89276_object58488:head# 2024-04-23T16:25:13.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24c745a6:::benchmark_data_smithi188_89276_object123312:head# 2024-04-23T16:25:13.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24c8410b:::benchmark_data_smithi188_89276_object81327:head# 2024-04-23T16:25:13.602 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:13.602 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24c98951:::benchmark_data_smithi188_89276_object100313:head# 2024-04-23T16:25:13.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24c9b321:::benchmark_data_smithi188_89276_object64741:head# 2024-04-23T16:25:13.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24caa369:::benchmark_data_smithi188_89276_object46374:head# 2024-04-23T16:25:13.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24cbe7c2:::benchmark_data_smithi188_89276_object168613:head# 2024-04-23T16:25:13.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24cc5aaf:::benchmark_data_smithi188_89276_object166692:head# 2024-04-23T16:25:13.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24cc9e24:::benchmark_data_smithi188_89276_object163376:head# 2024-04-23T16:25:13.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24cd2a66:::benchmark_data_smithi188_89276_object166378:head# 2024-04-23T16:25:13.605 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24cd3ade:::benchmark_data_smithi188_89276_object164877:head# 2024-04-23T16:25:13.605 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24ce5adf:::benchmark_data_smithi188_89276_object151976:head# 2024-04-23T16:25:13.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24cfc0b5:::benchmark_data_smithi188_89276_object51198:head# 2024-04-23T16:25:13.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24d02fbb:::benchmark_data_smithi188_89276_object51560:head# 2024-04-23T16:25:13.607 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24d1a0e5:::benchmark_data_smithi188_89276_object121191:head# 2024-04-23T16:25:13.607 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24d2d5cf:::benchmark_data_smithi188_89276_object136183:head# 2024-04-23T16:25:13.607 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24d2dc7f:::benchmark_data_smithi188_89276_object85258:head# 2024-04-23T16:25:13.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24d2e04c:::benchmark_data_smithi188_89276_object137201:head# 2024-04-23T16:25:13.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24d36803:::benchmark_data_smithi188_89276_object129986:head# 2024-04-23T16:25:13.609 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24d576d6:::benchmark_data_smithi188_89276_object79696:head# 2024-04-23T16:25:13.609 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24d64d0d:::benchmark_data_smithi188_89276_object103570:head# 2024-04-23T16:25:13.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24d66799:::benchmark_data_smithi188_89276_object124402:head# 2024-04-23T16:25:13.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24d6932e:::benchmark_data_smithi188_89276_object76947:head# 2024-04-23T16:25:13.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24d77142:::benchmark_data_smithi188_89276_object146186:head# 2024-04-23T16:25:13.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24d9b583:::benchmark_data_smithi188_89276_object47892:head# 2024-04-23T16:25:13.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24dbc905:::benchmark_data_smithi188_89276_object162045:head# 2024-04-23T16:25:13.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24dbe1c8:::benchmark_data_smithi188_89276_object67057:head# 2024-04-23T16:25:13.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24dc4103:::benchmark_data_smithi188_89276_object103463:head# 2024-04-23T16:25:13.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24dcef84:::benchmark_data_smithi188_89276_object52663:head# 2024-04-23T16:25:13.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24dd7fbb:::benchmark_data_smithi188_89276_object76680:head# 2024-04-23T16:25:13.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24dd882f:::benchmark_data_smithi188_89276_object85548:head# 2024-04-23T16:25:13.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24de0b56:::benchmark_data_smithi188_89276_object87105:head# 2024-04-23T16:25:13.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24de3e7e:::benchmark_data_smithi188_89276_object157678:head# 2024-04-23T16:25:13.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24de6b7a:::benchmark_data_smithi188_89276_object127934:head# 2024-04-23T16:25:13.616 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24de6f5c:::benchmark_data_smithi188_89276_object64322:head# 2024-04-23T16:25:13.616 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24deadee:::benchmark_data_smithi188_89276_object74663:head# 2024-04-23T16:25:13.617 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24dfc910:::benchmark_data_smithi188_89276_object87071:head# 2024-04-23T16:25:13.617 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e173a7:::benchmark_data_smithi188_89276_object158295:head# 2024-04-23T16:25:13.618 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e18e94:::benchmark_data_smithi188_89276_object73391:head# 2024-04-23T16:25:13.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e1ef01:::benchmark_data_smithi188_89276_object71001:head# 2024-04-23T16:25:13.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e27b3f:::benchmark_data_smithi188_89276_object136772:head# 2024-04-23T16:25:13.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e2afeb:::benchmark_data_smithi188_89276_object164545:head# 2024-04-23T16:25:13.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e2cc61:::benchmark_data_smithi188_89276_object147496:head# 2024-04-23T16:25:13.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e3b1a5:::benchmark_data_smithi188_89276_object88190:head# 2024-04-23T16:25:13.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e4315f:::benchmark_data_smithi188_89276_object100912:head# 2024-04-23T16:25:13.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e44b9c:::benchmark_data_smithi188_89276_object142070:head# 2024-04-23T16:25:13.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e45805:::benchmark_data_smithi188_89276_object102444:head# 2024-04-23T16:25:13.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e47f24:::benchmark_data_smithi188_89276_object84420:head# 2024-04-23T16:25:13.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e4b6d1:::benchmark_data_smithi188_89276_object76416:head# 2024-04-23T16:25:13.624 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e51022:::benchmark_data_smithi188_89276_object96049:head# 2024-04-23T16:25:13.624 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e59f73:::benchmark_data_smithi188_89276_object92064:head# 2024-04-23T16:25:13.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e5c7ab:::benchmark_data_smithi188_89276_object64908:head# 2024-04-23T16:25:13.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e5fa56:::benchmark_data_smithi188_89276_object76329:head# 2024-04-23T16:25:13.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e609fd:::benchmark_data_smithi188_89276_object104888:head# 2024-04-23T16:25:13.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e68cd9:::benchmark_data_smithi188_89276_object156308:head# 2024-04-23T16:25:13.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e6b591:::benchmark_data_smithi188_89276_object77919:head# 2024-04-23T16:25:13.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e78760:::benchmark_data_smithi188_89276_object135538:head# 2024-04-23T16:25:13.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e7f31d:::benchmark_data_smithi188_89276_object95720:head# 2024-04-23T16:25:13.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e80035:::benchmark_data_smithi188_89276_object144940:head# 2024-04-23T16:25:13.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e81cc5:::benchmark_data_smithi188_89276_object160398:head# 2024-04-23T16:25:13.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e81e3f:::benchmark_data_smithi188_89276_object117491:head# 2024-04-23T16:25:13.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e87f6e:::benchmark_data_smithi188_89276_object107135:head# 2024-04-23T16:25:13.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e898eb:::benchmark_data_smithi188_89276_object134524:head# 2024-04-23T16:25:13.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e8c0e9:::benchmark_data_smithi188_89276_object118292:head# 2024-04-23T16:25:13.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e8f402:::benchmark_data_smithi188_89276_object102762:head# 2024-04-23T16:25:13.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e95422:::benchmark_data_smithi188_89276_object56679:head# 2024-04-23T16:25:13.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24e96d09:::benchmark_data_smithi188_89276_object77598:head# 2024-04-23T16:25:13.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24ea8d79:::benchmark_data_smithi188_89276_object58413:head# 2024-04-23T16:25:13.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24eb31df:::benchmark_data_smithi188_89276_object45157:head# 2024-04-23T16:25:13.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24ebf336:::benchmark_data_smithi188_89276_object118836:head# 2024-04-23T16:25:13.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24ed7225:::benchmark_data_smithi188_89276_object151778:head# 2024-04-23T16:25:13.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24edbbbc:::benchmark_data_smithi188_89276_object168412:head# 2024-04-23T16:25:13.635 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24edf9f6:::benchmark_data_smithi188_89276_object110723:head# 2024-04-23T16:25:13.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24eeb55e:::benchmark_data_smithi188_89276_object94400:head# 2024-04-23T16:25:13.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24efd722:::benchmark_data_smithi188_89276_object150970:head# 2024-04-23T16:25:13.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24f176c9:::benchmark_data_smithi188_89276_object168607:head# 2024-04-23T16:25:13.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24f1b897:::benchmark_data_smithi188_89276_object90288:head# 2024-04-23T16:25:13.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24f24a34:::benchmark_data_smithi188_89276_object131098:head# 2024-04-23T16:25:13.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24f26db1:::benchmark_data_smithi188_89276_object87155:head# 2024-04-23T16:25:13.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24f39460:::benchmark_data_smithi188_89276_object134970:head# 2024-04-23T16:25:13.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24f4a739:::benchmark_data_smithi188_89276_object159927:head# 2024-04-23T16:25:13.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24f4bdc3:::benchmark_data_smithi188_89276_object65576:head# 2024-04-23T16:25:13.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24f52c92:::benchmark_data_smithi188_89276_object111736:head# 2024-04-23T16:25:13.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24f623d2:::benchmark_data_smithi188_89276_object61284:head# 2024-04-23T16:25:13.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24f6bfe3:::benchmark_data_smithi188_89276_object142760:head# 2024-04-23T16:25:13.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24f71fc2:::benchmark_data_smithi188_89276_object118343:head# 2024-04-23T16:25:13.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24f74e23:::benchmark_data_smithi188_89276_object61152:head# 2024-04-23T16:25:13.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24f7efd2:::benchmark_data_smithi188_89276_object144808:head# 2024-04-23T16:25:13.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24f804b6:::benchmark_data_smithi188_89276_object116631:head# 2024-04-23T16:25:13.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24f86458:::benchmark_data_smithi188_89276_object116511:head# 2024-04-23T16:25:13.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24f8aef5:::benchmark_data_smithi188_89276_object99548:head# 2024-04-23T16:25:13.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24f8db11:::benchmark_data_smithi188_89276_object117004:head# 2024-04-23T16:25:13.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24f914df:::benchmark_data_smithi188_89276_object78849:head# 2024-04-23T16:25:13.645 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24f9c9b9:::benchmark_data_smithi188_89276_object52404:head# 2024-04-23T16:25:13.645 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24faaff1:::benchmark_data_smithi188_89276_object71285:head# 2024-04-23T16:25:13.645 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24fb277c:::benchmark_data_smithi188_89276_object94797:head# 2024-04-23T16:25:13.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24fc1f65:::benchmark_data_smithi188_89276_object149537:head# 2024-04-23T16:25:13.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24fc94e9:::benchmark_data_smithi188_89276_object107583:head# 2024-04-23T16:25:13.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24fcad0a:::benchmark_data_smithi188_89276_object56253:head# 2024-04-23T16:25:13.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24fd0cf4:::benchmark_data_smithi188_89276_object53917:head# 2024-04-23T16:25:13.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24ff3294:::benchmark_data_smithi188_89276_object44686:head# 2024-04-23T16:25:13.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24ff87e9:::benchmark_data_smithi188_89276_object139789:head# 2024-04-23T16:25:13.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:24ffcd5b:::benchmark_data_smithi188_89276_object129802:head# 2024-04-23T16:25:13.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25001534:::benchmark_data_smithi188_89276_object60757:head# 2024-04-23T16:25:13.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25013b28:::benchmark_data_smithi188_89276_object72444:head# 2024-04-23T16:25:13.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25017ac0:::benchmark_data_smithi188_89276_object71046:head# 2024-04-23T16:25:13.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25031795:::benchmark_data_smithi188_89276_object93491:head# 2024-04-23T16:25:13.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2503b7ed:::benchmark_data_smithi188_89276_object85010:head# 2024-04-23T16:25:13.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2503e11a:::benchmark_data_smithi188_89276_object100045:head# 2024-04-23T16:25:13.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2503ea25:::benchmark_data_smithi188_89276_object41065:head# 2024-04-23T16:25:13.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:250462cd:::benchmark_data_smithi188_89276_object59998:head# 2024-04-23T16:25:13.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2504a5a5:::benchmark_data_smithi188_89276_object95047:head# 2024-04-23T16:25:13.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2504b7c8:::benchmark_data_smithi188_89276_object136040:head# 2024-04-23T16:25:13.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25057a10:::benchmark_data_smithi188_89276_object128686:head# 2024-04-23T16:25:13.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2505c578:::benchmark_data_smithi188_89276_object158581:head# 2024-04-23T16:25:13.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2506a2f7:::benchmark_data_smithi188_89276_object136568:head# 2024-04-23T16:25:13.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2506c38e:::benchmark_data_smithi188_89276_object66257:head# 2024-04-23T16:25:13.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:250792f0:::benchmark_data_smithi188_89276_object75332:head# 2024-04-23T16:25:13.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2507cec0:::benchmark_data_smithi188_89276_object167384:head# 2024-04-23T16:25:13.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25089be2:::benchmark_data_smithi188_89276_object131030:head# 2024-04-23T16:25:13.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2508ded2:::benchmark_data_smithi188_89276_object45874:head# 2024-04-23T16:25:13.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2509556c:::benchmark_data_smithi188_89276_object47822:head# 2024-04-23T16:25:13.661 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:250a6a1a:::benchmark_data_smithi188_89276_object46731:head# 2024-04-23T16:25:13.661 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:250a7765:::benchmark_data_smithi188_89276_object89068:head# 2024-04-23T16:25:13.661 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:250afce9:::benchmark_data_smithi188_89276_object136308:head# 2024-04-23T16:25:13.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:250afde2:::benchmark_data_smithi188_89276_object133213:head# 2024-04-23T16:25:13.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:250b31a4:::benchmark_data_smithi188_89276_object83736:head# 2024-04-23T16:25:13.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:250ba811:::benchmark_data_smithi188_89276_object119961:head# 2024-04-23T16:25:13.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:250bb905:::benchmark_data_smithi188_89276_object135204:head# 2024-04-23T16:25:13.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:250d167e:::benchmark_data_smithi188_89276_object68705:head# 2024-04-23T16:25:13.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:250e8617:::benchmark_data_smithi188_89276_object118449:head# 2024-04-23T16:25:13.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:250ec1c4:::benchmark_data_smithi188_89276_object92868:head# 2024-04-23T16:25:13.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:250ef08d:::benchmark_data_smithi188_89276_object64124:head# 2024-04-23T16:25:13.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:250f6ccc:::benchmark_data_smithi188_89276_object156802:head# 2024-04-23T16:25:13.669 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25101ac1:::benchmark_data_smithi188_89276_object136358:head# 2024-04-23T16:25:13.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25113d32:::benchmark_data_smithi188_89276_object45550:head# 2024-04-23T16:25:13.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2511a8fd:::benchmark_data_smithi188_89276_object142487:head# 2024-04-23T16:25:13.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2511e71c:::benchmark_data_smithi188_89276_object64636:head# 2024-04-23T16:25:13.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25123ce3:::benchmark_data_smithi188_89276_object100419:head# 2024-04-23T16:25:13.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2512d4a8:::benchmark_data_smithi188_89276_object59746:head# 2024-04-23T16:25:13.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2512d9e6:::benchmark_data_smithi188_89276_object126925:head# 2024-04-23T16:25:13.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25142d82:::benchmark_data_smithi188_89276_object44762:head# 2024-04-23T16:25:13.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2514c0b7:::benchmark_data_smithi188_89276_object72072:head# 2024-04-23T16:25:13.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2515359d:::benchmark_data_smithi188_89276_object66208:head# 2024-04-23T16:25:13.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25158354:::benchmark_data_smithi188_89276_object112489:head# 2024-04-23T16:25:13.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25164af4:::benchmark_data_smithi188_89276_object85199:head# 2024-04-23T16:25:13.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2516a7fd:::benchmark_data_smithi188_89276_object59658:head# 2024-04-23T16:25:13.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25170cbb:::benchmark_data_smithi188_89276_object97216:head# 2024-04-23T16:25:13.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:251738e8:::benchmark_data_smithi188_89276_object75711:head# 2024-04-23T16:25:13.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:251769ae:::benchmark_data_smithi188_89276_object99757:head# 2024-04-23T16:25:13.679 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2517a3e0:::benchmark_data_smithi188_89276_object138941:head# 2024-04-23T16:25:13.680 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2517dc3e:::benchmark_data_smithi188_89276_object100776:head# 2024-04-23T16:25:13.680 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2518378e:::benchmark_data_smithi188_89276_object78190:head# 2024-04-23T16:25:13.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:251860aa:::benchmark_data_smithi188_89276_object152319:head# 2024-04-23T16:25:13.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2518886a:::benchmark_data_smithi188_89276_object134266:head# 2024-04-23T16:25:13.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2518c192:::benchmark_data_smithi188_89276_object119619:head# 2024-04-23T16:25:13.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:251933e9:::benchmark_data_smithi188_89276_object58295:head# 2024-04-23T16:25:13.683 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2519575e:::benchmark_data_smithi188_89276_object165987:head# 2024-04-23T16:25:13.683 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2519bb44:::benchmark_data_smithi188_89276_object122065:head# 2024-04-23T16:25:13.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2519ea93:::benchmark_data_smithi188_89276_object92593:head# 2024-04-23T16:25:13.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:251a4ad7:::benchmark_data_smithi188_89276_object101282:head# 2024-04-23T16:25:13.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:251aaed1:::benchmark_data_smithi188_89276_object120136:head# 2024-04-23T16:25:13.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:251b0b81:::benchmark_data_smithi188_89276_object90312:head# 2024-04-23T16:25:13.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:251b2e30:::benchmark_data_smithi188_89276_object64672:head# 2024-04-23T16:25:13.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:251cd562:::benchmark_data_smithi188_89276_object110268:head# 2024-04-23T16:25:13.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:251d2d15:::benchmark_data_smithi188_89276_object128296:head# 2024-04-23T16:25:13.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:251d7f75:::benchmark_data_smithi188_89276_object87855:head# 2024-04-23T16:25:13.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:251ddad4:::benchmark_data_smithi188_89276_object115079:head# 2024-04-23T16:25:13.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:251dde47:::benchmark_data_smithi188_89276_object71714:head# 2024-04-23T16:25:13.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:251ec74d:::benchmark_data_smithi188_89276_object160372:head# 2024-04-23T16:25:13.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:252143c8:::benchmark_data_smithi188_89276_object117178:head# 2024-04-23T16:25:13.689 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2521f012:::benchmark_data_smithi188_89276_object103407:head# 2024-04-23T16:25:13.689 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:252235f6:::benchmark_data_smithi188_89276_object92459:head# 2024-04-23T16:25:13.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:252265c8:::benchmark_data_smithi188_89276_object168516:head# 2024-04-23T16:25:13.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2522dba3:::benchmark_data_smithi188_89276_object127230:head# 2024-04-23T16:25:13.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25231004:::benchmark_data_smithi188_89276_object155707:head# 2024-04-23T16:25:13.691 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2523ea82:::benchmark_data_smithi188_89276_object72716:head# 2024-04-23T16:25:13.691 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25240d07:::benchmark_data_smithi188_89276_object52327:head# 2024-04-23T16:25:13.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2524239c:::benchmark_data_smithi188_89276_object84363:head# 2024-04-23T16:25:13.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2525a06b:::benchmark_data_smithi188_89276_object155718:head# 2024-04-23T16:25:13.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2525d56d:::benchmark_data_smithi188_89276_object99399:head# 2024-04-23T16:25:13.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2525fb8a:::benchmark_data_smithi188_89276_object148982:head# 2024-04-23T16:25:13.694 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25262730:::benchmark_data_smithi188_89276_object83412:head# 2024-04-23T16:25:13.694 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2527bca6:::benchmark_data_smithi188_89276_object90351:head# 2024-04-23T16:25:13.694 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2527c44b:::benchmark_data_smithi188_89276_object79140:head# 2024-04-23T16:25:13.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2528e002:::benchmark_data_smithi188_89276_object53124:head# 2024-04-23T16:25:13.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2529f1ad:::benchmark_data_smithi188_89276_object61257:head# 2024-04-23T16:25:13.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:252ad86e:::benchmark_data_smithi188_89276_object46490:head# 2024-04-23T16:25:13.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:252b5c41:::benchmark_data_smithi188_89276_object57850:head# 2024-04-23T16:25:13.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:252bac14:::benchmark_data_smithi188_89276_object162823:head# 2024-04-23T16:25:13.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:252bf8b9:::benchmark_data_smithi188_89276_object72528:head# 2024-04-23T16:25:13.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:252c1c8d:::benchmark_data_smithi188_89276_object74751:head# 2024-04-23T16:25:13.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:252c2fc7:::benchmark_data_smithi188_89276_object151742:head# 2024-04-23T16:25:13.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:252cc598:::benchmark_data_smithi188_89276_object73656:head# 2024-04-23T16:25:13.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:252ccad8:::benchmark_data_smithi188_89276_object57414:head# 2024-04-23T16:25:13.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:252eae33:::benchmark_data_smithi188_89276_object45941:head# 2024-04-23T16:25:13.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:252ecbc5:::benchmark_data_smithi188_89276_object58759:head# 2024-04-23T16:25:13.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:252eeef8:::benchmark_data_smithi188_89276_object51619:head# 2024-04-23T16:25:13.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:252f5be3:::benchmark_data_smithi188_89276_object126469:head# 2024-04-23T16:25:13.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:253012b2:::benchmark_data_smithi188_89276_object128507:head# 2024-04-23T16:25:13.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25302059:::benchmark_data_smithi188_89276_object140537:head# 2024-04-23T16:25:13.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25307cf2:::benchmark_data_smithi188_89276_object62992:head# 2024-04-23T16:25:13.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2531e11f:::benchmark_data_smithi188_89276_object113928:head# 2024-04-23T16:25:13.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25341ff5:::benchmark_data_smithi188_89276_object102749:head# 2024-04-23T16:25:13.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2534657e:::benchmark_data_smithi188_89276_object52335:head# 2024-04-23T16:25:13.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2534c619:::benchmark_data_smithi188_89276_object139907:head# 2024-04-23T16:25:13.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2535ef4a:::benchmark_data_smithi188_89276_object83257:head# 2024-04-23T16:25:13.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2535fb69:::benchmark_data_smithi188_89276_object73963:head# 2024-04-23T16:25:13.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:253621b9:::benchmark_data_smithi188_89276_object99419:head# 2024-04-23T16:25:13.709 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2536b2ed:::benchmark_data_smithi188_89276_object165004:head# 2024-04-23T16:25:13.709 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2536dc3d:::benchmark_data_smithi188_89276_object110553:head# 2024-04-23T16:25:13.709 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25374eb9:::benchmark_data_smithi188_89276_object151586:head# 2024-04-23T16:25:13.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2537b8c4:::benchmark_data_smithi188_89276_object110941:head# 2024-04-23T16:25:13.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:253818d1:::benchmark_data_smithi188_89276_object134996:head# 2024-04-23T16:25:13.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:253953ce:::benchmark_data_smithi188_89276_object58213:head# 2024-04-23T16:25:13.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25397511:::benchmark_data_smithi188_89276_object113007:head# 2024-04-23T16:25:13.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:253a777b:::benchmark_data_smithi188_89276_object161963:head# 2024-04-23T16:25:13.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:253b010a:::benchmark_data_smithi188_89276_object166545:head# 2024-04-23T16:25:13.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:253ba130:::benchmark_data_smithi188_89276_object143169:head# 2024-04-23T16:25:13.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:253c1e5c:::benchmark_data_smithi188_89276_object125882:head# 2024-04-23T16:25:13.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:253c532b:::benchmark_data_smithi188_89276_object103297:head# 2024-04-23T16:25:13.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:253dec01:::benchmark_data_smithi188_89276_object45076:head# 2024-04-23T16:25:13.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:253e0623:::benchmark_data_smithi188_89276_object57127:head# 2024-04-23T16:25:13.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:253e2d54:::benchmark_data_smithi188_89276_object146218:head# 2024-04-23T16:25:13.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:253eae59:::benchmark_data_smithi188_89276_object163063:head# 2024-04-23T16:25:13.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:253f8787:::benchmark_data_smithi188_89276_object80731:head# 2024-04-23T16:25:13.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:254059ff:::benchmark_data_smithi188_89276_object146474:head# 2024-04-23T16:25:13.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25418241:::benchmark_data_smithi188_89276_object96103:head# 2024-04-23T16:25:13.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25421ea7:::benchmark_data_smithi188_89276_object54614:head# 2024-04-23T16:25:13.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2542736a:::benchmark_data_smithi188_89276_object76289:head# 2024-04-23T16:25:13.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2543ebb2:::benchmark_data_smithi188_89276_object161864:head# 2024-04-23T16:25:13.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25443191:::benchmark_data_smithi188_89276_object71122:head# 2024-04-23T16:25:13.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2544b1d3:::benchmark_data_smithi188_89276_object135618:head# 2024-04-23T16:25:13.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2545f81d:::benchmark_data_smithi188_89276_object51337:head# 2024-04-23T16:25:13.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:254688eb:::benchmark_data_smithi188_89276_object144437:head# 2024-04-23T16:25:13.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2546d648:::benchmark_data_smithi188_89276_object127397:head# 2024-04-23T16:25:13.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25477944:::benchmark_data_smithi188_89276_object58564:head# 2024-04-23T16:25:13.724 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:25:13.725 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25480c7d:::benchmark_data_smithi188_89276_object143581:head# 2024-04-23T16:25:13.725 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25485a3f:::benchmark_data_smithi188_89276_object150601:head# 2024-04-23T16:25:13.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2549021e:::benchmark_data_smithi188_89276_object66238:head# 2024-04-23T16:25:13.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2549d280:::benchmark_data_smithi188_89276_object64017:head# 2024-04-23T16:25:13.726 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:13.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:254b6e8d:::benchmark_data_smithi188_89276_object82903:head# 2024-04-23T16:25:13.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:254be962:::benchmark_data_smithi188_89276_object61718:head# 2024-04-23T16:25:13.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:254bebe2:::benchmark_data_smithi188_89276_object92463:head# 2024-04-23T16:25:13.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:13.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:13.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:254cf3fd:::benchmark_data_smithi188_89276_object78813:head# 2024-04-23T16:25:13.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:254d9c59:::benchmark_data_smithi188_89276_object140558:head# 2024-04-23T16:25:13.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:254e3666:::benchmark_data_smithi188_89276_object165214:head# 2024-04-23T16:25:13.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:254e512a:::benchmark_data_smithi188_89276_object162729:head# 2024-04-23T16:25:13.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:254e536a:::benchmark_data_smithi188_89276_object80308:head# 2024-04-23T16:25:13.735 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:254e6620:::benchmark_data_smithi188_89276_object47160:head# 2024-04-23T16:25:13.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:254f04f9:::benchmark_data_smithi188_89276_object70574:head# 2024-04-23T16:25:13.737 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:254f80eb:::benchmark_data_smithi188_89276_object92828:head# 2024-04-23T16:25:13.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25500062:::benchmark_data_smithi188_89276_object70679:head# 2024-04-23T16:25:13.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2550d4f8:::benchmark_data_smithi188_89276_object106987:head# 2024-04-23T16:25:13.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2550d728:::benchmark_data_smithi188_89276_object88347:head# 2024-04-23T16:25:13.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2550feb6:::benchmark_data_smithi188_89276_object126827:head# 2024-04-23T16:25:13.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2551a5b7:::benchmark_data_smithi188_89276_object71347:head# 2024-04-23T16:25:13.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2551e319:::benchmark_data_smithi188_89276_object132948:head# 2024-04-23T16:25:13.744 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:255343f6:::benchmark_data_smithi188_89276_object100479:head# 2024-04-23T16:25:13.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25539b14:::benchmark_data_smithi188_89276_object89745:head# 2024-04-23T16:25:13.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25541f6d:::benchmark_data_smithi188_89276_object128753:head# 2024-04-23T16:25:13.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25549a43:::benchmark_data_smithi188_89276_object41006:head# 2024-04-23T16:25:13.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:255583e5:::benchmark_data_smithi188_89276_object63905:head# 2024-04-23T16:25:13.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2555cd15:::benchmark_data_smithi188_89276_object149602:head# 2024-04-23T16:25:13.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25562485:::benchmark_data_smithi188_89276_object166384:head# 2024-04-23T16:25:13.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25569072:::benchmark_data_smithi188_89276_object92049:head# 2024-04-23T16:25:13.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25580289:::benchmark_data_smithi188_89276_object110613:head# 2024-04-23T16:25:13.750 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25582d04:::benchmark_data_smithi188_89276_object98333:head# 2024-04-23T16:25:13.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2559061e:::benchmark_data_smithi188_89276_object127768:head# 2024-04-23T16:25:13.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:255913cf:::benchmark_data_smithi188_89276_object144901:head# 2024-04-23T16:25:13.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25595861:::benchmark_data_smithi188_89276_object92981:head# 2024-04-23T16:25:13.752 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:13 smithi148 ceph-mon[44520]: pgmap v715: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 228 op/s; 20 MiB/s, 316 objects/s recovering 2024-04-23T16:25:13.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2559c92c:::benchmark_data_smithi188_89276_object65624:head# 2024-04-23T16:25:13.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:255a1228:::benchmark_data_smithi188_89276_object129787:head# 2024-04-23T16:25:13.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:255a3735:::benchmark_data_smithi188_89276_object76606:head# 2024-04-23T16:25:13.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:255aced2:::benchmark_data_smithi188_89276_object91300:head# 2024-04-23T16:25:13.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:255ad189:::benchmark_data_smithi188_89276_object166711:head# 2024-04-23T16:25:13.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:255c23e6:::benchmark_data_smithi188_89276_object150545:head# 2024-04-23T16:25:13.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:255ce3c5:::benchmark_data_smithi188_89276_object95180:head# 2024-04-23T16:25:13.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:255cf6da:::benchmark_data_smithi188_89276_object159587:head# 2024-04-23T16:25:13.756 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:255d2ff1:::benchmark_data_smithi188_89276_object151331:head# 2024-04-23T16:25:13.756 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:255d4de6:::benchmark_data_smithi188_89276_object82876:head# 2024-04-23T16:25:13.757 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:255db744:::benchmark_data_smithi188_89276_object113546:head# 2024-04-23T16:25:13.757 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:255df74d:::benchmark_data_smithi188_89276_object59075:head# 2024-04-23T16:25:13.757 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:255e3c63:::benchmark_data_smithi188_89276_object85966:head# 2024-04-23T16:25:13.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:255ecbda:::benchmark_data_smithi188_89276_object153541:head# 2024-04-23T16:25:13.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:255f151c:::benchmark_data_smithi188_89276_object123583:head# 2024-04-23T16:25:13.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:255fe4a5:::benchmark_data_smithi188_89276_object157387:head# 2024-04-23T16:25:13.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:256070e4:::benchmark_data_smithi188_89276_object98804:head# 2024-04-23T16:25:13.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2560ca62:::benchmark_data_smithi188_89276_object163633:head# 2024-04-23T16:25:13.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2560e64a:::benchmark_data_smithi188_89276_object60455:head# 2024-04-23T16:25:13.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:256103f4:::benchmark_data_smithi188_89276_object105491:head# 2024-04-23T16:25:13.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25621df8:::benchmark_data_smithi188_89276_object104729:head# 2024-04-23T16:25:13.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25623c2a:::benchmark_data_smithi188_89276_object157863:head# 2024-04-23T16:25:13.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25627960:::benchmark_data_smithi188_89276_object105132:head# 2024-04-23T16:25:13.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2562be34:::benchmark_data_smithi188_89276_object43994:head# 2024-04-23T16:25:13.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2562dedf:::benchmark_data_smithi188_89276_object122264:head# 2024-04-23T16:25:13.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:256324cd:::benchmark_data_smithi188_89276_object84870:head# 2024-04-23T16:25:13.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2563c37c:::benchmark_data_smithi188_89276_object88080:head# 2024-04-23T16:25:13.767 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25647b2c:::benchmark_data_smithi188_89276_object53729:head# 2024-04-23T16:25:13.767 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2565383c:::benchmark_data_smithi188_89276_object68895:head# 2024-04-23T16:25:13.767 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25653e20:::benchmark_data_smithi188_89276_object124485:head# 2024-04-23T16:25:13.767 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:256564f1:::benchmark_data_smithi188_89276_object101550:head# 2024-04-23T16:25:13.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:256588ec:::benchmark_data_smithi188_89276_object125872:head# 2024-04-23T16:25:13.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2565df57:::benchmark_data_smithi188_89276_object97740:head# 2024-04-23T16:25:13.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25666e06:::benchmark_data_smithi188_89276_object138103:head# 2024-04-23T16:25:13.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25676efb:::benchmark_data_smithi188_89276_object54597:head# 2024-04-23T16:25:13.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:256795ee:::benchmark_data_smithi188_89276_object49036:head# 2024-04-23T16:25:13.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2569429a:::benchmark_data_smithi188_89276_object107326:head# 2024-04-23T16:25:13.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:256b588c:::benchmark_data_smithi188_89276_object157736:head# 2024-04-23T16:25:13.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:256b8666:::benchmark_data_smithi188_89276_object66998:head# 2024-04-23T16:25:13.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:256baf42:::benchmark_data_smithi188_89276_object96192:head# 2024-04-23T16:25:13.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:256c9446:::benchmark_data_smithi188_89276_object79003:head# 2024-04-23T16:25:13.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:256d0a69:::benchmark_data_smithi188_89276_object60377:head# 2024-04-23T16:25:13.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:256d859a:::benchmark_data_smithi188_89276_object154327:head# 2024-04-23T16:25:13.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:256da100:::benchmark_data_smithi188_89276_object44378:head# 2024-04-23T16:25:13.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:256df710:::benchmark_data_smithi188_89276_object94414:head# 2024-04-23T16:25:13.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:256e1cdd:::benchmark_data_smithi188_89276_object94506:head# 2024-04-23T16:25:13.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:256e6e8b:::benchmark_data_smithi188_89276_object52916:head# 2024-04-23T16:25:13.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:256ed354:::benchmark_data_smithi188_89276_object153120:head# 2024-04-23T16:25:13.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:256f5ebf:::benchmark_data_smithi188_89276_object88793:head# 2024-04-23T16:25:13.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:256fb51c:::benchmark_data_smithi188_89276_object154678:head# 2024-04-23T16:25:13.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2570d208:::benchmark_data_smithi188_89276_object134454:head# 2024-04-23T16:25:13.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2570e35b:::benchmark_data_smithi188_89276_object57733:head# 2024-04-23T16:25:13.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2570f0cc:::benchmark_data_smithi188_89276_object64013:head# 2024-04-23T16:25:13.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:257185f4:::benchmark_data_smithi188_89276_object81306:head# 2024-04-23T16:25:13.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25718824:::benchmark_data_smithi188_89276_object141790:head# 2024-04-23T16:25:13.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2571c71d:::benchmark_data_smithi188_89276_object88370:head# 2024-04-23T16:25:13.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25722961:::benchmark_data_smithi188_89276_object65635:head# 2024-04-23T16:25:13.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2572707d:::benchmark_data_smithi188_89276_object76537:head# 2024-04-23T16:25:13.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25732324:::benchmark_data_smithi188_89276_object45133:head# 2024-04-23T16:25:13.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25734e4e:::benchmark_data_smithi188_89276_object54185:head# 2024-04-23T16:25:13.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25738214:::benchmark_data_smithi188_89276_object57794:head# 2024-04-23T16:25:13.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2573c161:::benchmark_data_smithi188_89276_object108911:head# 2024-04-23T16:25:13.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2574f9b4:::benchmark_data_smithi188_89276_object57934:head# 2024-04-23T16:25:13.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2575b6e0:::benchmark_data_smithi188_89276_object83105:head# 2024-04-23T16:25:13.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2575d3e6:::benchmark_data_smithi188_89276_object75450:head# 2024-04-23T16:25:13.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2575f875:::benchmark_data_smithi188_89276_object75566:head# 2024-04-23T16:25:13.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2576a728:::benchmark_data_smithi188_89276_object167644:head# 2024-04-23T16:25:13.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2576d2e7:::benchmark_data_smithi188_89276_object154065:head# 2024-04-23T16:25:13.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25774fe1:::benchmark_data_smithi188_89276_object51655:head# 2024-04-23T16:25:13.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25784d60:::benchmark_data_smithi188_89276_object75624:head# 2024-04-23T16:25:13.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2578c440:::benchmark_data_smithi188_89276_object50706:head# 2024-04-23T16:25:13.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25797c4d:::benchmark_data_smithi188_89276_object46704:head# 2024-04-23T16:25:13.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2579c9d0:::benchmark_data_smithi188_89276_object164891:head# 2024-04-23T16:25:13.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2579f6ee:::benchmark_data_smithi188_89276_object142559:head# 2024-04-23T16:25:13.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:257a790e:::benchmark_data_smithi188_89276_object145660:head# 2024-04-23T16:25:13.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:257aa38a:::benchmark_data_smithi188_89276_object147697:head# 2024-04-23T16:25:13.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:257b34ec:::benchmark_data_smithi188_89276_object61899:head# 2024-04-23T16:25:13.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:257b5cc0:::benchmark_data_smithi188_89276_object110224:head# 2024-04-23T16:25:13.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:257d318c:::benchmark_data_smithi188_89276_object155491:head# 2024-04-23T16:25:13.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:257ea230:::benchmark_data_smithi188_89276_object63707:head# 2024-04-23T16:25:13.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:257ee02b:::benchmark_data_smithi188_89276_object139655:head# 2024-04-23T16:25:13.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:257f9720:::benchmark_data_smithi188_89276_object113370:head# 2024-04-23T16:25:13.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25802eac:::benchmark_data_smithi188_89276_object147238:head# 2024-04-23T16:25:13.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2580cba3:::benchmark_data_smithi188_89276_object149441:head# 2024-04-23T16:25:13.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25816813:::benchmark_data_smithi188_89276_object96014:head# 2024-04-23T16:25:13.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258198a2:::benchmark_data_smithi188_89276_object67602:head# 2024-04-23T16:25:13.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25821483:::benchmark_data_smithi188_89276_object82102:head# 2024-04-23T16:25:13.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25821912:::benchmark_data_smithi188_89276_object66513:head# 2024-04-23T16:25:13.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25821bdf:::benchmark_data_smithi188_89276_object166211:head# 2024-04-23T16:25:13.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25822d20:::benchmark_data_smithi188_89276_object142185:head# 2024-04-23T16:25:13.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258258e4:::benchmark_data_smithi188_89276_object115459:head# 2024-04-23T16:25:13.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2584fe32:::benchmark_data_smithi188_89276_object147611:head# 2024-04-23T16:25:13.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258531c8:::benchmark_data_smithi188_89276_object101098:head# 2024-04-23T16:25:13.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2585a0de:::benchmark_data_smithi188_89276_object139355:head# 2024-04-23T16:25:13.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2585a6aa:::benchmark_data_smithi188_89276_object60934:head# 2024-04-23T16:25:13.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258726af:::benchmark_data_smithi188_89276_object127691:head# 2024-04-23T16:25:13.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25872fe8:::benchmark_data_smithi188_89276_object55961:head# 2024-04-23T16:25:13.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25875be3:::benchmark_data_smithi188_89276_object50580:head# 2024-04-23T16:25:13.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2587a7b3:::benchmark_data_smithi188_89276_object43091:head# 2024-04-23T16:25:13.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2587e178:::benchmark_data_smithi188_89276_object130746:head# 2024-04-23T16:25:13.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25888cf6:::benchmark_data_smithi188_89276_object68607:head# 2024-04-23T16:25:13.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25890544:::benchmark_data_smithi188_89276_object119157:head# 2024-04-23T16:25:13.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2589c7e2:::benchmark_data_smithi188_89276_object141472:head# 2024-04-23T16:25:13.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2589e18c:::benchmark_data_smithi188_89276_object53426:head# 2024-04-23T16:25:13.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258aaeeb:::benchmark_data_smithi188_89276_object141711:head# 2024-04-23T16:25:13.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258b02cb:::benchmark_data_smithi188_89276_object85956:head# 2024-04-23T16:25:13.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258bebe0:::benchmark_data_smithi188_89276_object127710:head# 2024-04-23T16:25:13.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258bed35:::benchmark_data_smithi188_89276_object79928:head# 2024-04-23T16:25:13.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258bed74:::benchmark_data_smithi188_89276_object119173:head# 2024-04-23T16:25:13.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258c0bf7:::benchmark_data_smithi188_89276_object130772:head# 2024-04-23T16:25:13.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258c70d9:::benchmark_data_smithi188_89276_object168567:head# 2024-04-23T16:25:13.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258cf5db:::benchmark_data_smithi188_89276_object103274:head# 2024-04-23T16:25:13.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258cfc2b:::benchmark_data_smithi188_89276_object161794:head# 2024-04-23T16:25:13.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258d5348:::benchmark_data_smithi188_89276_object96455:head# 2024-04-23T16:25:13.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258d9a38:::benchmark_data_smithi188_89276_object47581:head# 2024-04-23T16:25:13.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258db96d:::benchmark_data_smithi188_89276_object133492:head# 2024-04-23T16:25:13.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258e1c0f:::benchmark_data_smithi188_89276_object168308:head# 2024-04-23T16:25:13.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258ec6d4:::benchmark_data_smithi188_89276_object142255:head# 2024-04-23T16:25:13.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258f88da:::benchmark_data_smithi188_89276_object107733:head# 2024-04-23T16:25:13.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:258fa77d:::benchmark_data_smithi188_89276_object67139:head# 2024-04-23T16:25:13.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25903d18:::benchmark_data_smithi188_89276_object147600:head# 2024-04-23T16:25:13.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2590a5c9:::benchmark_data_smithi188_89276_object73894:head# 2024-04-23T16:25:13.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25914cdd:::benchmark_data_smithi188_89276_object145063:head# 2024-04-23T16:25:13.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2592eeba:::benchmark_data_smithi188_89276_object68686:head# 2024-04-23T16:25:13.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25930b3b:::benchmark_data_smithi188_89276_object41031:head# 2024-04-23T16:25:13.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25933edc:::benchmark_data_smithi188_89276_object98213:head# 2024-04-23T16:25:13.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25934f54:::benchmark_data_smithi188_89276_object149449:head# 2024-04-23T16:25:13.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25935bb2:::benchmark_data_smithi188_89276_object115370:head# 2024-04-23T16:25:13.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2593e525:::benchmark_data_smithi188_89276_object42584:head# 2024-04-23T16:25:13.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25943c8c:::benchmark_data_smithi188_89276_object82689:head# 2024-04-23T16:25:13.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25945563:::benchmark_data_smithi188_89276_object85122:head# 2024-04-23T16:25:13.814 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2594d253:::benchmark_data_smithi188_89276_object73657:head# 2024-04-23T16:25:13.814 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2594f103:::benchmark_data_smithi188_89276_object164548:head# 2024-04-23T16:25:13.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25962013:::benchmark_data_smithi188_89276_object124379:head# 2024-04-23T16:25:13.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25964764:::benchmark_data_smithi188_89276_object152761:head# 2024-04-23T16:25:13.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25967a00:::benchmark_data_smithi188_89276_object78502:head# 2024-04-23T16:25:13.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2596c76d:::benchmark_data_smithi188_89276_object66850:head# 2024-04-23T16:25:13.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2597d221:::benchmark_data_smithi188_89276_object98898:head# 2024-04-23T16:25:13.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25980204:::benchmark_data_smithi188_89276_object147716:head# 2024-04-23T16:25:13.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25991183:::benchmark_data_smithi188_89276_object167435:head# 2024-04-23T16:25:13.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:259bb8e4:::benchmark_data_smithi188_89276_object162347:head# 2024-04-23T16:25:13.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:259bcf32:::benchmark_data_smithi188_89276_object53167:head# 2024-04-23T16:25:13.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:259db4c2:::benchmark_data_smithi188_89276_object65491:head# 2024-04-23T16:25:13.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:259db9e0:::benchmark_data_smithi188_89276_object81134:head# 2024-04-23T16:25:13.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:259dd2b2:::benchmark_data_smithi188_89276_object158519:head# 2024-04-23T16:25:13.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:259ddc1d:::benchmark_data_smithi188_89276_object40294:head# 2024-04-23T16:25:13.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:259e717f:::benchmark_data_smithi188_89276_object154589:head# 2024-04-23T16:25:13.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:259e7c92:::benchmark_data_smithi188_89276_object152403:head# 2024-04-23T16:25:13.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:259e81e2:::benchmark_data_smithi188_89276_object158594:head# 2024-04-23T16:25:13.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:259eb8c9:::benchmark_data_smithi188_89276_object85880:head# 2024-04-23T16:25:13.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:259f79b8:::benchmark_data_smithi188_89276_object41566:head# 2024-04-23T16:25:13.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a052be:::benchmark_data_smithi188_89276_object89300:head# 2024-04-23T16:25:13.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a070af:::benchmark_data_smithi188_89276_object112467:head# 2024-04-23T16:25:13.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a0e92b:::benchmark_data_smithi188_89276_object137528:head# 2024-04-23T16:25:13.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a17624:::benchmark_data_smithi188_89276_object55124:head# 2024-04-23T16:25:13.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a19076:::benchmark_data_smithi188_89276_object107929:head# 2024-04-23T16:25:13.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a1c0ee:::benchmark_data_smithi188_89276_object78657:head# 2024-04-23T16:25:13.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a1c656:::benchmark_data_smithi188_89276_object125090:head# 2024-04-23T16:25:13.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a35182:::benchmark_data_smithi188_89276_object109193:head# 2024-04-23T16:25:13.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a3db41:::benchmark_data_smithi188_89276_object74169:head# 2024-04-23T16:25:13.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a3ea05:::benchmark_data_smithi188_89276_object158418:head# 2024-04-23T16:25:13.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a504c8:::benchmark_data_smithi188_89276_object76942:head# 2024-04-23T16:25:13.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a518ab:::benchmark_data_smithi188_89276_object136699:head# 2024-04-23T16:25:13.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a5600b:::benchmark_data_smithi188_89276_object148745:head# 2024-04-23T16:25:13.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a65150:::benchmark_data_smithi188_89276_object79692:head# 2024-04-23T16:25:13.829 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:25:13.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a6c5fb:::benchmark_data_smithi188_89276_object149652:head# 2024-04-23T16:25:13.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a6eb94:::benchmark_data_smithi188_89276_object121726:head# 2024-04-23T16:25:13.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a71ac5:::benchmark_data_smithi188_89276_object77953:head# 2024-04-23T16:25:13.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a7b072:::benchmark_data_smithi188_89276_object162563:head# 2024-04-23T16:25:13.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a7e06c:::benchmark_data_smithi188_89276_object120794:head# 2024-04-23T16:25:13.832 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a7ef65:::benchmark_data_smithi188_89276_object122128:head# 2024-04-23T16:25:13.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25a8bdf8:::benchmark_data_smithi188_89276_object139177:head# 2024-04-23T16:25:13.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25aa5041:::benchmark_data_smithi188_89276_object92675:head# 2024-04-23T16:25:13.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ab20a0:::benchmark_data_smithi188_89276_object68073:head# 2024-04-23T16:25:13.834 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ab3bcd:::benchmark_data_smithi188_89276_object87327:head# 2024-04-23T16:25:13.834 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ab8cd4:::benchmark_data_smithi188_89276_object136766:head# 2024-04-23T16:25:13.834 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ac2074:::benchmark_data_smithi188_89276_object103077:head# 2024-04-23T16:25:13.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25acf4d1:::benchmark_data_smithi188_89276_object141265:head# 2024-04-23T16:25:13.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ae5456:::benchmark_data_smithi188_89276_object83207:head# 2024-04-23T16:25:13.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25aef333:::benchmark_data_smithi188_89276_object154196:head# 2024-04-23T16:25:13.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25aef95a:::benchmark_data_smithi188_89276_object89496:head# 2024-04-23T16:25:13.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25afaa96:::benchmark_data_smithi188_89276_object105127:head# 2024-04-23T16:25:13.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25aff449:::benchmark_data_smithi188_89276_object161053:head# 2024-04-23T16:25:13.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25b102b5:::benchmark_data_smithi188_89276_object61221:head# 2024-04-23T16:25:13.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25b1ad62:::benchmark_data_smithi188_89276_object84204:head# 2024-04-23T16:25:13.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25b24c4d:::benchmark_data_smithi188_89276_object93076:head# 2024-04-23T16:25:13.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25b48f28:::benchmark_data_smithi188_89276_object147413:head# 2024-04-23T16:25:13.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25b516ea:::benchmark_data_smithi188_89276_object48651:head# 2024-04-23T16:25:13.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25b5512e:::benchmark_data_smithi188_89276_object125541:head# 2024-04-23T16:25:13.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25b59de1:::benchmark_data_smithi188_89276_object130976:head# 2024-04-23T16:25:13.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25b630c1:::benchmark_data_smithi188_89276_object45669:head# 2024-04-23T16:25:13.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25b6bda5:::benchmark_data_smithi188_89276_object119823:head# 2024-04-23T16:25:13.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25b6fc02:::benchmark_data_smithi188_89276_object43359:head# 2024-04-23T16:25:13.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25b78d84:::benchmark_data_smithi188_89276_object80646:head# 2024-04-23T16:25:13.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25b7b048:::benchmark_data_smithi188_89276_object54744:head# 2024-04-23T16:25:13.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25b7ca33:::benchmark_data_smithi188_89276_object77585:head# 2024-04-23T16:25:13.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25b86cc9:::benchmark_data_smithi188_89276_object102266:head# 2024-04-23T16:25:13.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25b9cd69:::benchmark_data_smithi188_89276_object145274:head# 2024-04-23T16:25:13.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ba9bd0:::benchmark_data_smithi188_89276_object121416:head# 2024-04-23T16:25:13.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25bac225:::benchmark_data_smithi188_89276_object135839:head# 2024-04-23T16:25:13.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25bd4d86:::benchmark_data_smithi188_89276_object164131:head# 2024-04-23T16:25:13.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25be0518:::benchmark_data_smithi188_89276_object77847:head# 2024-04-23T16:25:13.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25be55ff:::benchmark_data_smithi188_89276_object153708:head# 2024-04-23T16:25:13.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25bf21e2:::benchmark_data_smithi188_89276_object94575:head# 2024-04-23T16:25:13.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25bf2999:::benchmark_data_smithi188_89276_object65791:head# 2024-04-23T16:25:13.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c066fd:::benchmark_data_smithi188_89276_object70016:head# 2024-04-23T16:25:13.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c0a336:::benchmark_data_smithi188_89276_object149333:head# 2024-04-23T16:25:13.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c0aac5:::benchmark_data_smithi188_89276_object124441:head# 2024-04-23T16:25:13.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c0c10d:::benchmark_data_smithi188_89276_object67282:head# 2024-04-23T16:25:13.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c0f4fc:::benchmark_data_smithi188_89276_object159823:head# 2024-04-23T16:25:13.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c189a1:::benchmark_data_smithi188_89276_object100067:head# 2024-04-23T16:25:13.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c19069:::benchmark_data_smithi188_89276_object119056:head# 2024-04-23T16:25:13.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c22cce:::benchmark_data_smithi188_89276_object118833:head# 2024-04-23T16:25:13.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c25388:::benchmark_data_smithi188_89276_object136513:head# 2024-04-23T16:25:13.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c262f5:::benchmark_data_smithi188_89276_object62869:head# 2024-04-23T16:25:13.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c2f00d:::benchmark_data_smithi188_89276_object104205:head# 2024-04-23T16:25:13.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c40136:::benchmark_data_smithi188_89276_object50875:head# 2024-04-23T16:25:13.858 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:13.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c438b8:::benchmark_data_smithi188_89276_object84991:head# 2024-04-23T16:25:13.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c44e94:::benchmark_data_smithi188_89276_object117779:head# 2024-04-23T16:25:13.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c4533f:::benchmark_data_smithi188_89276_object40917:head# 2024-04-23T16:25:13.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c51825:::benchmark_data_smithi188_89276_object84603:head# 2024-04-23T16:25:13.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c5c187:::benchmark_data_smithi188_89276_object103478:head# 2024-04-23T16:25:13.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c65068:::benchmark_data_smithi188_89276_object116746:head# 2024-04-23T16:25:13.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c6d063:::benchmark_data_smithi188_89276_object130780:head# 2024-04-23T16:25:13.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c6f532:::benchmark_data_smithi188_89276_object54810:head# 2024-04-23T16:25:13.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c72029:::benchmark_data_smithi188_89276_object101303:head# 2024-04-23T16:25:13.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c74ba0:::benchmark_data_smithi188_89276_object150930:head# 2024-04-23T16:25:13.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c84f5e:::benchmark_data_smithi188_89276_object116781:head# 2024-04-23T16:25:13.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25c9838e:::benchmark_data_smithi188_89276_object94022:head# 2024-04-23T16:25:13.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ca044f:::benchmark_data_smithi188_89276_object114824:head# 2024-04-23T16:25:13.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ca812e:::benchmark_data_smithi188_89276_object125152:head# 2024-04-23T16:25:13.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25cb7c68:::benchmark_data_smithi188_89276_object85217:head# 2024-04-23T16:25:13.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25cc3be2:::benchmark_data_smithi188_89276_object46954:head# 2024-04-23T16:25:13.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25cc622e:::benchmark_data_smithi188_89276_object135729:head# 2024-04-23T16:25:13.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ccd402:::benchmark_data_smithi188_89276_object123499:head# 2024-04-23T16:25:13.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25cd10e7:::benchmark_data_smithi188_89276_object81508:head# 2024-04-23T16:25:13.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25cdeccd:::benchmark_data_smithi188_89276_object41840:head# 2024-04-23T16:25:13.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25cf5344:::benchmark_data_smithi188_89276_object83447:head# 2024-04-23T16:25:13.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25cffe56:::benchmark_data_smithi188_89276_object62594:head# 2024-04-23T16:25:13.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25d137e0:::benchmark_data_smithi188_89276_object116648:head# 2024-04-23T16:25:13.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25d24bf0:::benchmark_data_smithi188_89276_object93236:head# 2024-04-23T16:25:13.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25d32e90:::benchmark_data_smithi188_89276_object121843:head# 2024-04-23T16:25:13.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25d3c3f2:::benchmark_data_smithi188_89276_object127705:head# 2024-04-23T16:25:13.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25d41d71:::benchmark_data_smithi188_89276_object84282:head# 2024-04-23T16:25:13.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25d4c730:::benchmark_data_smithi188_89276_object145699:head# 2024-04-23T16:25:13.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25d68ceb:::benchmark_data_smithi188_89276_object64679:head# 2024-04-23T16:25:13.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25d6cfe8:::benchmark_data_smithi188_89276_object79139:head# 2024-04-23T16:25:13.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25d79694:::benchmark_data_smithi188_89276_object64967:head# 2024-04-23T16:25:13.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25d7a6fd:::benchmark_data_smithi188_89276_object87315:head# 2024-04-23T16:25:13.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25d7d44b:::benchmark_data_smithi188_89276_object66797:head# 2024-04-23T16:25:13.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25d89feb:::benchmark_data_smithi188_89276_object59285:head# 2024-04-23T16:25:13.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25d8cf3a:::benchmark_data_smithi188_89276_object105788:head# 2024-04-23T16:25:13.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25d93377:::benchmark_data_smithi188_89276_object152788:head# 2024-04-23T16:25:13.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25d96690:::benchmark_data_smithi188_89276_object98935:head# 2024-04-23T16:25:13.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25d9af57:::benchmark_data_smithi188_89276_object108382:head# 2024-04-23T16:25:13.875 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25db1927:::benchmark_data_smithi188_89276_object156081:head# 2024-04-23T16:25:13.875 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25dbb5da:::benchmark_data_smithi188_89276_object136922:head# 2024-04-23T16:25:13.876 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25dcd518:::benchmark_data_smithi188_89276_object143928:head# 2024-04-23T16:25:13.876 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25dcdde9:::benchmark_data_smithi188_89276_object68994:head# 2024-04-23T16:25:13.876 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25dcedd2:::benchmark_data_smithi188_89276_object59650:head# 2024-04-23T16:25:13.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25dd8b3e:::benchmark_data_smithi188_89276_object50747:head# 2024-04-23T16:25:13.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25de307e:::benchmark_data_smithi188_89276_object128273:head# 2024-04-23T16:25:13.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25dfcbe7:::benchmark_data_smithi188_89276_object92504:head# 2024-04-23T16:25:13.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25e0d59d:::benchmark_data_smithi188_89276_object57532:head# 2024-04-23T16:25:13.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25e0fdd1:::benchmark_data_smithi188_89276_object55635:head# 2024-04-23T16:25:13.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25e23333:::benchmark_data_smithi188_89276_object149662:head# 2024-04-23T16:25:13.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25e26ae2:::benchmark_data_smithi188_89276_object104194:head# 2024-04-23T16:25:13.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25e27c3c:::benchmark_data_smithi188_89276_object150322:head# 2024-04-23T16:25:13.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25e2ef1d:::benchmark_data_smithi188_89276_object145839:head# 2024-04-23T16:25:13.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25e314b5:::benchmark_data_smithi188_89276_object165669:head# 2024-04-23T16:25:13.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25e49e4a:::benchmark_data_smithi188_89276_object76408:head# 2024-04-23T16:25:13.882 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:25:13.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25e4dd90:::benchmark_data_smithi188_89276_object71567:head# 2024-04-23T16:25:13.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25e4f6de:::benchmark_data_smithi188_89276_object123710:head# 2024-04-23T16:25:13.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25e5c500:::benchmark_data_smithi188_89276_object113708:head# 2024-04-23T16:25:13.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25e6a812:::benchmark_data_smithi188_89276_object66215:head# 2024-04-23T16:25:13.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25e6b79a:::benchmark_data_smithi188_89276_object62010:head# 2024-04-23T16:25:13.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25e87f88:::benchmark_data_smithi188_89276_object129661:head# 2024-04-23T16:25:13.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:13.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:25:13.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25e8e278:::benchmark_data_smithi188_89276_object143992:head# 2024-04-23T16:25:13.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25e95760:::benchmark_data_smithi188_89276_object58936:head# 2024-04-23T16:25:13.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25e9f721:::benchmark_data_smithi188_89276_object45994:head# 2024-04-23T16:25:13.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ea0a98:::benchmark_data_smithi188_89276_object74451:head# 2024-04-23T16:25:13.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ebf104:::benchmark_data_smithi188_89276_object85428:head# 2024-04-23T16:25:13.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ec597f:::benchmark_data_smithi188_89276_object163071:head# 2024-04-23T16:25:13.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ec5ee4:::benchmark_data_smithi188_89276_object126434:head# 2024-04-23T16:25:13.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ecd18d:::benchmark_data_smithi188_89276_object69356:head# 2024-04-23T16:25:13.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ecd4a5:::benchmark_data_smithi188_89276_object94543:head# 2024-04-23T16:25:13.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ed16b2:::benchmark_data_smithi188_89276_object101383:head# 2024-04-23T16:25:13.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ede61e:::benchmark_data_smithi188_89276_object95354:head# 2024-04-23T16:25:13.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ee1440:::benchmark_data_smithi188_89276_object99793:head# 2024-04-23T16:25:13.891 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25eed285:::benchmark_data_smithi188_89276_object140727:head# 2024-04-23T16:25:13.891 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ef98ec:::benchmark_data_smithi188_89276_object132407:head# 2024-04-23T16:25:13.892 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f28c56:::benchmark_data_smithi188_89276_object161844:head# 2024-04-23T16:25:13.892 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f2c8b2:::benchmark_data_smithi188_89276_object137858:head# 2024-04-23T16:25:13.893 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f3b704:::benchmark_data_smithi188_89276_object77955:head# 2024-04-23T16:25:13.893 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f3d086:::benchmark_data_smithi188_89276_object134149:head# 2024-04-23T16:25:13.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f3e871:::benchmark_data_smithi188_89276_object119672:head# 2024-04-23T16:25:13.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f3eca7:::benchmark_data_smithi188_89276_object93542:head# 2024-04-23T16:25:13.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f41842:::benchmark_data_smithi188_89276_object87907:head# 2024-04-23T16:25:13.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f45c85:::benchmark_data_smithi188_89276_object73314:head# 2024-04-23T16:25:13.896 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f51c8a:::benchmark_data_smithi188_89276_object123790:head# 2024-04-23T16:25:13.896 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f62470:::benchmark_data_smithi188_89276_object113637:head# 2024-04-23T16:25:13.896 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f6726d:::benchmark_data_smithi188_89276_object130908:head# 2024-04-23T16:25:13.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f6a747:::benchmark_data_smithi188_89276_object68525:head# 2024-04-23T16:25:13.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f70c78:::benchmark_data_smithi188_89276_object62064:head# 2024-04-23T16:25:13.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f70dd0:::benchmark_data_smithi188_89276_object69635:head# 2024-04-23T16:25:13.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f72d79:::benchmark_data_smithi188_89276_object85041:head# 2024-04-23T16:25:13.899 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f7afcf:::benchmark_data_smithi188_89276_object87721:head# 2024-04-23T16:25:13.899 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f87034:::benchmark_data_smithi188_89276_object103116:head# 2024-04-23T16:25:13.900 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f87c85:::benchmark_data_smithi188_89276_object62408:head# 2024-04-23T16:25:13.900 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f87eee:::benchmark_data_smithi188_89276_object145035:head# 2024-04-23T16:25:13.900 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f8d984:::benchmark_data_smithi188_89276_object63129:head# 2024-04-23T16:25:13.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25f98541:::benchmark_data_smithi188_89276_object130737:head# 2024-04-23T16:25:13.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25fad1e5:::benchmark_data_smithi188_89276_object82397:head# 2024-04-23T16:25:13.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25fadf98:::benchmark_data_smithi188_89276_object66101:head# 2024-04-23T16:25:13.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25fc0cad:::benchmark_data_smithi188_89276_object134176:head# 2024-04-23T16:25:13.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25fc10ea:::benchmark_data_smithi188_89276_object125750:head# 2024-04-23T16:25:13.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25fcd989:::benchmark_data_smithi188_89276_object162155:head# 2024-04-23T16:25:13.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25fe00a0:::benchmark_data_smithi188_89276_object72351:head# 2024-04-23T16:25:13.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25fe606c:::benchmark_data_smithi188_89276_object41087:head# 2024-04-23T16:25:13.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25febc5d:::benchmark_data_smithi188_89276_object45958:head# 2024-04-23T16:25:13.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25fee714:::benchmark_data_smithi188_89276_object99228:head# 2024-04-23T16:25:13.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:25ff93d2:::benchmark_data_smithi188_89276_object118364:head# 2024-04-23T16:25:13.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2600046f:::benchmark_data_smithi188_89276_object140632:head# 2024-04-23T16:25:13.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:260030e6:::benchmark_data_smithi188_89276_object137728:head# 2024-04-23T16:25:13.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2600638d:::benchmark_data_smithi188_89276_object132995:head# 2024-04-23T16:25:13.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26007054:::benchmark_data_smithi188_89276_object67890:head# 2024-04-23T16:25:13.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2600ef09:::benchmark_data_smithi188_89276_object148842:head# 2024-04-23T16:25:13.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:260133e7:::benchmark_data_smithi188_89276_object71192:head# 2024-04-23T16:25:13.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26025a02:::benchmark_data_smithi188_89276_object103332:head# 2024-04-23T16:25:13.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26029e20:::benchmark_data_smithi188_89276_object57960:head# 2024-04-23T16:25:13.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2603119f:::benchmark_data_smithi188_89276_object91676:head# 2024-04-23T16:25:13.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26032c4e:::benchmark_data_smithi188_89276_object108961:head# 2024-04-23T16:25:13.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2603745c:::benchmark_data_smithi188_89276_object117879:head# 2024-04-23T16:25:13.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:260468e5:::benchmark_data_smithi188_89276_object114143:head# 2024-04-23T16:25:13.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2606095b:::benchmark_data_smithi188_89276_object157419:head# 2024-04-23T16:25:13.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26067c83:::benchmark_data_smithi188_89276_object128125:head# 2024-04-23T16:25:13.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:260735c0:::benchmark_data_smithi188_89276_object145842:head# 2024-04-23T16:25:13.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26073eb9:::benchmark_data_smithi188_89276_object153373:head# 2024-04-23T16:25:13.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:260747f5:::benchmark_data_smithi188_89276_object132181:head# 2024-04-23T16:25:13.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26078cba:::benchmark_data_smithi188_89276_object106773:head# 2024-04-23T16:25:13.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2607eb0e:::benchmark_data_smithi188_89276_object92755:head# 2024-04-23T16:25:13.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26093a77:::benchmark_data_smithi188_89276_object72701:head# 2024-04-23T16:25:13.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:260965c0:::benchmark_data_smithi188_89276_object50461:head# 2024-04-23T16:25:13.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:260984c6:::benchmark_data_smithi188_89276_object96788:head# 2024-04-23T16:25:13.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:260a48df:::benchmark_data_smithi188_89276_object85368:head# 2024-04-23T16:25:13.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:260b19c2:::benchmark_data_smithi188_89276_object116937:head# 2024-04-23T16:25:13.925 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:260b449c:::benchmark_data_smithi188_89276_object106694:head# 2024-04-23T16:25:13.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:260bcbc3:::benchmark_data_smithi188_89276_object40754:head# 2024-04-23T16:25:13.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:260c1da3:::benchmark_data_smithi188_89276_object140232:head# 2024-04-23T16:25:13.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:260c9bc4:::benchmark_data_smithi188_89276_object158081:head# 2024-04-23T16:25:13.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:260ceb14:::benchmark_data_smithi188_89276_object161126:head# 2024-04-23T16:25:13.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:260e7cb4:::benchmark_data_smithi188_89276_object103061:head# 2024-04-23T16:25:13.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:260e915b:::benchmark_data_smithi188_89276_object127112:head# 2024-04-23T16:25:13.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:260f2af4:::benchmark_data_smithi188_89276_object136787:head# 2024-04-23T16:25:13.930 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:260f7a93:::benchmark_data_smithi188_89276_object102230:head# 2024-04-23T16:25:13.930 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:260fd26f:::benchmark_data_smithi188_89276_object64680:head# 2024-04-23T16:25:13.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2610284f:::benchmark_data_smithi188_89276_object88653:head# 2024-04-23T16:25:13.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2610c3ef:::benchmark_data_smithi188_89276_object119678:head# 2024-04-23T16:25:13.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26127b61:::benchmark_data_smithi188_89276_object107361:head# 2024-04-23T16:25:13.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2612f3f4:::benchmark_data_smithi188_89276_object139644:head# 2024-04-23T16:25:13.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2613c672:::benchmark_data_smithi188_89276_object119809:head# 2024-04-23T16:25:13.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26142eb9:::benchmark_data_smithi188_89276_object149829:head# 2024-04-23T16:25:13.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2614417f:::benchmark_data_smithi188_89276_object83713:head# 2024-04-23T16:25:13.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26145c69:::benchmark_data_smithi188_89276_object99589:head# 2024-04-23T16:25:13.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26147fb0:::benchmark_data_smithi188_89276_object95666:head# 2024-04-23T16:25:13.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2615af66:::benchmark_data_smithi188_89276_object137746:head# 2024-04-23T16:25:13.939 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2615b98b:::benchmark_data_smithi188_89276_object122642:head# 2024-04-23T16:25:13.939 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26168910:::benchmark_data_smithi188_89276_object127914:head# 2024-04-23T16:25:13.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:261694a7:::benchmark_data_smithi188_89276_object94174:head# 2024-04-23T16:25:13.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2616aa95:::benchmark_data_smithi188_89276_object84158:head# 2024-04-23T16:25:13.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2616b8ce:::benchmark_data_smithi188_89276_object49171:head# 2024-04-23T16:25:13.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2616cfab:::benchmark_data_smithi188_89276_object65222:head# 2024-04-23T16:25:13.943 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:261728db:::benchmark_data_smithi188_89276_object158172:head# 2024-04-23T16:25:13.943 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26176998:::benchmark_data_smithi188_89276_object91035:head# 2024-04-23T16:25:13.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26177fe9:::benchmark_data_smithi188_89276_object134006:head# 2024-04-23T16:25:13.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:261793dc:::benchmark_data_smithi188_89276_object49122:head# 2024-04-23T16:25:13.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2617cc7d:::benchmark_data_smithi188_89276_object56727:head# 2024-04-23T16:25:13.947 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2618ef5c:::benchmark_data_smithi188_89276_object60210:head# 2024-04-23T16:25:13.947 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2619b244:::benchmark_data_smithi188_89276_object123850:head# 2024-04-23T16:25:13.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:261a4f60:::benchmark_data_smithi188_89276_object132155:head# 2024-04-23T16:25:13.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:261aa50f:::benchmark_data_smithi188_89276_object95337:head# 2024-04-23T16:25:13.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:261aa940:::benchmark_data_smithi188_89276_object119690:head# 2024-04-23T16:25:13.949 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:261adcfc:::benchmark_data_smithi188_89276_object120375:head# 2024-04-23T16:25:13.950 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:261b5021:::benchmark_data_smithi188_89276_object125212:head# 2024-04-23T16:25:13.950 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:261b541a:::benchmark_data_smithi188_89276_object126539:head# 2024-04-23T16:25:13.950 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:261b70e8:::benchmark_data_smithi188_89276_object163808:head# 2024-04-23T16:25:13.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:261bda87:::benchmark_data_smithi188_89276_object155891:head# 2024-04-23T16:25:13.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:261bfeb4:::benchmark_data_smithi188_89276_object118658:head# 2024-04-23T16:25:13.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:261bfef8:::benchmark_data_smithi188_89276_object153360:head# 2024-04-23T16:25:13.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:261cb73b:::benchmark_data_smithi188_89276_object72880:head# 2024-04-23T16:25:13.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:261f763f:::benchmark_data_smithi188_89276_object110965:head# 2024-04-23T16:25:13.953 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:261f873c:::benchmark_data_smithi188_89276_object97158:head# 2024-04-23T16:25:13.953 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:261fbd05:::benchmark_data_smithi188_89276_object62224:head# 2024-04-23T16:25:13.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26205f7e:::benchmark_data_smithi188_89276_object112081:head# 2024-04-23T16:25:13.955 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2621d2b5:::benchmark_data_smithi188_89276_object57862:head# 2024-04-23T16:25:13.956 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2621f9c3:::benchmark_data_smithi188_89276_object50682:head# 2024-04-23T16:25:13.956 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:262264db:::benchmark_data_smithi188_89276_object109709:head# 2024-04-23T16:25:13.956 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2622d0e6:::benchmark_data_smithi188_89276_object108933:head# 2024-04-23T16:25:13.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26242761:::benchmark_data_smithi188_89276_object156862:head# 2024-04-23T16:25:13.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26243e9f:::benchmark_data_smithi188_89276_object67061:head# 2024-04-23T16:25:13.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26247ae8:::benchmark_data_smithi188_89276_object56135:head# 2024-04-23T16:25:13.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:262502fb:::benchmark_data_smithi188_89276_object165870:head# 2024-04-23T16:25:13.959 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26250f43:::benchmark_data_smithi188_89276_object154089:head# 2024-04-23T16:25:13.959 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2626ce5b:::benchmark_data_smithi188_89276_object106509:head# 2024-04-23T16:25:13.960 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2628dd97:::benchmark_data_smithi188_89276_object121096:head# 2024-04-23T16:25:13.960 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26298eca:::benchmark_data_smithi188_89276_object136629:head# 2024-04-23T16:25:13.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26299f07:::benchmark_data_smithi188_89276_object159681:head# 2024-04-23T16:25:13.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2629bf20:::benchmark_data_smithi188_89276_object52448:head# 2024-04-23T16:25:13.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:262a7655:::benchmark_data_smithi188_89276_object147781:head# 2024-04-23T16:25:13.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:262acfe2:::benchmark_data_smithi188_89276_object128624:head# 2024-04-23T16:25:13.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:262b336d:::benchmark_data_smithi188_89276_object99191:head# 2024-04-23T16:25:13.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:262b3bc2:::benchmark_data_smithi188_89276_object80715:head# 2024-04-23T16:25:13.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:262c744d:::benchmark_data_smithi188_89276_object89204:head# 2024-04-23T16:25:13.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:262d3c6e:::benchmark_data_smithi188_89276_object86051:head# 2024-04-23T16:25:13.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:262d8dad:::benchmark_data_smithi188_89276_object92096:head# 2024-04-23T16:25:13.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:262e6f79:::benchmark_data_smithi188_89276_object44490:head# 2024-04-23T16:25:13.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:262ebb47:::benchmark_data_smithi188_89276_object61020:head# 2024-04-23T16:25:13.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:262ec2aa:::benchmark_data_smithi188_89276_object41093:head# 2024-04-23T16:25:13.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:262f16a4:::benchmark_data_smithi188_89276_object94114:head# 2024-04-23T16:25:13.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:262f6767:::benchmark_data_smithi188_89276_object140987:head# 2024-04-23T16:25:13.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:262fbed1:::benchmark_data_smithi188_89276_object149153:head# 2024-04-23T16:25:13.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26310760:::benchmark_data_smithi188_89276_object93649:head# 2024-04-23T16:25:13.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263149fc:::benchmark_data_smithi188_89276_object51944:head# 2024-04-23T16:25:13.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263189ab:::benchmark_data_smithi188_89276_object151709:head# 2024-04-23T16:25:13.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2631eaa8:::benchmark_data_smithi188_89276_object125175:head# 2024-04-23T16:25:13.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26324757:::benchmark_data_smithi188_89276_object99635:head# 2024-04-23T16:25:13.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2632fc0a:::benchmark_data_smithi188_89276_object85092:head# 2024-04-23T16:25:13.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26333345:::benchmark_data_smithi188_89276_object162073:head# 2024-04-23T16:25:13.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2633cfc2:::benchmark_data_smithi188_89276_object95228:head# 2024-04-23T16:25:13.973 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263456ed:::benchmark_data_smithi188_89276_object62477:head# 2024-04-23T16:25:13.973 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263483e4:::benchmark_data_smithi188_89276_object71427:head# 2024-04-23T16:25:13.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2634f565:::benchmark_data_smithi188_89276_object147981:head# 2024-04-23T16:25:13.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2634fb92:::benchmark_data_smithi188_89276_object163299:head# 2024-04-23T16:25:13.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263524f6:::benchmark_data_smithi188_89276_object54258:head# 2024-04-23T16:25:13.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2635e79c:::benchmark_data_smithi188_89276_object125431:head# 2024-04-23T16:25:13.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26376607:::benchmark_data_smithi188_89276_object54109:head# 2024-04-23T16:25:13.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263785a2:::benchmark_data_smithi188_89276_object43671:head# 2024-04-23T16:25:13.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26381e15:::benchmark_data_smithi188_89276_object67151:head# 2024-04-23T16:25:13.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263902e3:::benchmark_data_smithi188_89276_object88512:head# 2024-04-23T16:25:13.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26399ea9:::benchmark_data_smithi188_89276_object101730:head# 2024-04-23T16:25:13.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263ad656:::benchmark_data_smithi188_89276_object151252:head# 2024-04-23T16:25:13.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263b0d8b:::benchmark_data_smithi188_89276_object113827:head# 2024-04-23T16:25:13.986 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:25:13.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263c0217:::benchmark_data_smithi188_89276_object63419:head# 2024-04-23T16:25:13.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263c3e69:::benchmark_data_smithi188_89276_object80916:head# 2024-04-23T16:25:13.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263cc994:::benchmark_data_smithi188_89276_object43642:head# 2024-04-23T16:25:13.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263d1c4b:::benchmark_data_smithi188_89276_object70030:head# 2024-04-23T16:25:13.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263d1dc2:::benchmark_data_smithi188_89276_object164385:head# 2024-04-23T16:25:13.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263d4527:::benchmark_data_smithi188_89276_object154168:head# 2024-04-23T16:25:13.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263d93dc:::benchmark_data_smithi188_89276_object118185:head# 2024-04-23T16:25:13.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263e1fe9:::benchmark_data_smithi188_89276_object67047:head# 2024-04-23T16:25:13.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263e2247:::benchmark_data_smithi188_89276_object167172:head# 2024-04-23T16:25:13.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263f1a7e:::benchmark_data_smithi188_89276_object129940:head# 2024-04-23T16:25:13.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263fac1a:::benchmark_data_smithi188_89276_object107166:head# 2024-04-23T16:25:13.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:263fcc87:::benchmark_data_smithi188_89276_object76255:head# 2024-04-23T16:25:13.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26409551:::benchmark_data_smithi188_89276_object51821:head# 2024-04-23T16:25:13.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2640ed0e:::benchmark_data_smithi188_89276_object124833:head# 2024-04-23T16:25:13.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2640f1bb:::benchmark_data_smithi188_89276_object120142:head# 2024-04-23T16:25:13.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26422524:::benchmark_data_smithi188_89276_object101717:head# 2024-04-23T16:25:13.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2642c963:::benchmark_data_smithi188_89276_object109595:head# 2024-04-23T16:25:13.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26440633:::benchmark_data_smithi188_89276_object161922:head# 2024-04-23T16:25:13.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:264673c0:::benchmark_data_smithi188_89276_object156542:head# 2024-04-23T16:25:13.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:264684c6:::benchmark_data_smithi188_89276_object110101:head# 2024-04-23T16:25:13.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2646a89f:::benchmark_data_smithi188_89276_object137886:head# 2024-04-23T16:25:13.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26476962:::benchmark_data_smithi188_89276_object142167:head# 2024-04-23T16:25:13.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:13 smithi103 ceph-mon[39629]: pgmap v715: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 228 op/s; 20 MiB/s, 316 objects/s recovering 2024-04-23T16:25:13.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2647c95b:::benchmark_data_smithi188_89276_object90474:head# 2024-04-23T16:25:13.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26482ea4:::benchmark_data_smithi188_89276_object81716:head# 2024-04-23T16:25:13.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26491682:::benchmark_data_smithi188_89276_object57904:head# 2024-04-23T16:25:13.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26491dae:::benchmark_data_smithi188_89276_object95511:head# 2024-04-23T16:25:13.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2649a914:::benchmark_data_smithi188_89276_object45629:head# 2024-04-23T16:25:13.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:264a5dca:::benchmark_data_smithi188_89276_object152799:head# 2024-04-23T16:25:13.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:264a8042:::benchmark_data_smithi188_89276_object156416:head# 2024-04-23T16:25:13.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:264ac030:::benchmark_data_smithi188_89276_object76016:head# 2024-04-23T16:25:13.997 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:264b2d54:::benchmark_data_smithi188_89276_object122273:head# 2024-04-23T16:25:13.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:264b6df1:::benchmark_data_smithi188_89276_object151560:head# 2024-04-23T16:25:13.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:264b7e51:::benchmark_data_smithi188_89276_object100824:head# 2024-04-23T16:25:14.001 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:264c531d:::benchmark_data_smithi188_89276_object53265:head# 2024-04-23T16:25:14.001 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:264c6016:::benchmark_data_smithi188_89276_object117092:head# 2024-04-23T16:25:14.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:264cee9d:::benchmark_data_smithi188_89276_object70964:head# 2024-04-23T16:25:14.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:264de0ba:::benchmark_data_smithi188_89276_object92776:head# 2024-04-23T16:25:14.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:264ee125:::benchmark_data_smithi188_89276_object84037:head# 2024-04-23T16:25:14.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:264f27cf:::benchmark_data_smithi188_89276_object55834:head# 2024-04-23T16:25:14.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:264fe364:::benchmark_data_smithi188_89276_object79233:head# 2024-04-23T16:25:14.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2650b7b3:::benchmark_data_smithi188_89276_object111896:head# 2024-04-23T16:25:14.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26523591:::benchmark_data_smithi188_89276_object100338:head# 2024-04-23T16:25:14.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2652aa9f:::benchmark_data_smithi188_89276_object59173:head# 2024-04-23T16:25:14.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2652b8bf:::benchmark_data_smithi188_89276_object57731:head# 2024-04-23T16:25:14.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26538616:::benchmark_data_smithi188_89276_object62514:head# 2024-04-23T16:25:14.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26542ab4:::benchmark_data_smithi188_89276_object163911:head# 2024-04-23T16:25:14.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2655c51b:::benchmark_data_smithi188_89276_object153279:head# 2024-04-23T16:25:14.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:265625b5:::benchmark_data_smithi188_89276_object103796:head# 2024-04-23T16:25:14.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26565db0:::benchmark_data_smithi188_89276_object67505:head# 2024-04-23T16:25:14.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26567d75:::benchmark_data_smithi188_89276_object97945:head# 2024-04-23T16:25:14.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:265690bf:::benchmark_data_smithi188_89276_object76303:head# 2024-04-23T16:25:14.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2656e4e1:::benchmark_data_smithi188_89276_object45121:head# 2024-04-23T16:25:14.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2656fe5a:::benchmark_data_smithi188_89276_object42241:head# 2024-04-23T16:25:14.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26572ebd:::benchmark_data_smithi188_89276_object162249:head# 2024-04-23T16:25:14.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26578be2:::benchmark_data_smithi188_89276_object97931:head# 2024-04-23T16:25:14.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2657b8d6:::benchmark_data_smithi188_89276_object159785:head# 2024-04-23T16:25:14.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2658774c:::benchmark_data_smithi188_89276_object73664:head# 2024-04-23T16:25:14.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26587b4e:::benchmark_data_smithi188_89276_object137505:head# 2024-04-23T16:25:14.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26589df0:::benchmark_data_smithi188_89276_object141253:head# 2024-04-23T16:25:14.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2658c8f8:::benchmark_data_smithi188_89276_object115146:head# 2024-04-23T16:25:14.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2659895c:::benchmark_data_smithi188_89276_object64080:head# 2024-04-23T16:25:14.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2659dccc:::benchmark_data_smithi188_89276_object66521:head# 2024-04-23T16:25:14.018 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:265a5a08:::benchmark_data_smithi188_89276_object89340:head# 2024-04-23T16:25:14.018 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:265afba3:::benchmark_data_smithi188_89276_object94751:head# 2024-04-23T16:25:14.019 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:265b6e00:::benchmark_data_smithi188_89276_object168528:head# 2024-04-23T16:25:14.019 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:265bc41a:::benchmark_data_smithi188_89276_object55710:head# 2024-04-23T16:25:14.019 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:265bcdad:::benchmark_data_smithi188_89276_object81252:head# 2024-04-23T16:25:14.020 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:265bd3c8:::benchmark_data_smithi188_89276_object168012:head# 2024-04-23T16:25:14.020 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:265bd538:::benchmark_data_smithi188_89276_object69017:head# 2024-04-23T16:25:14.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:265d856c:::benchmark_data_smithi188_89276_object65958:head# 2024-04-23T16:25:14.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:265d8cb0:::benchmark_data_smithi188_89276_object106329:head# 2024-04-23T16:25:14.022 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:14.022 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:265e0834:::benchmark_data_smithi188_89276_object144786:head# 2024-04-23T16:25:14.022 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:265f882b:::benchmark_data_smithi188_89276_object66969:head# 2024-04-23T16:25:14.022 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:265fa84a:::benchmark_data_smithi188_89276_object103720:head# 2024-04-23T16:25:14.023 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:265fc1fa:::benchmark_data_smithi188_89276_object120259:head# 2024-04-23T16:25:14.023 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26611b47:::benchmark_data_smithi188_89276_object144513:head# 2024-04-23T16:25:14.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:266129a4:::benchmark_data_smithi188_89276_object133009:head# 2024-04-23T16:25:14.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:266138af:::benchmark_data_smithi188_89276_object100151:head# 2024-04-23T16:25:14.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26614b69:::benchmark_data_smithi188_89276_object95489:head# 2024-04-23T16:25:14.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:266183bf:::benchmark_data_smithi188_89276_object72546:head# 2024-04-23T16:25:14.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2662950a:::benchmark_data_smithi188_89276_object101386:head# 2024-04-23T16:25:14.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26632e68:::benchmark_data_smithi188_89276_object77621:head# 2024-04-23T16:25:14.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26634964:::benchmark_data_smithi188_89276_object98368:head# 2024-04-23T16:25:14.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:266374bc:::benchmark_data_smithi188_89276_object69035:head# 2024-04-23T16:25:14.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2663bdf5:::benchmark_data_smithi188_89276_object56980:head# 2024-04-23T16:25:14.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2664e66d:::benchmark_data_smithi188_89276_object133163:head# 2024-04-23T16:25:14.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:266538c9:::benchmark_data_smithi188_89276_object67365:head# 2024-04-23T16:25:14.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26660e0f:::benchmark_data_smithi188_89276_object133785:head# 2024-04-23T16:25:14.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26665d3f:::benchmark_data_smithi188_89276_object88067:head# 2024-04-23T16:25:14.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2667cc5d:::benchmark_data_smithi188_89276_object116212:head# 2024-04-23T16:25:14.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2668193f:::benchmark_data_smithi188_89276_object62899:head# 2024-04-23T16:25:14.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26698c5b:::benchmark_data_smithi188_89276_object135146:head# 2024-04-23T16:25:14.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2669d581:::benchmark_data_smithi188_89276_object135583:head# 2024-04-23T16:25:14.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:266a70f1:::benchmark_data_smithi188_89276_object84633:head# 2024-04-23T16:25:14.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:266a710d:::benchmark_data_smithi188_89276_object124828:head# 2024-04-23T16:25:14.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:266b0054:::benchmark_data_smithi188_89276_object167803:head# 2024-04-23T16:25:14.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:266b7316:::benchmark_data_smithi188_89276_object74010:head# 2024-04-23T16:25:14.034 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:266bd7c6:::benchmark_data_smithi188_89276_object128042:head# 2024-04-23T16:25:14.034 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:266c02ed:::benchmark_data_smithi188_89276_object64579:head# 2024-04-23T16:25:14.034 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:266c1283:::benchmark_data_smithi188_89276_object43088:head# 2024-04-23T16:25:14.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:266d460c:::benchmark_data_smithi188_89276_object138082:head# 2024-04-23T16:25:14.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:266db932:::benchmark_data_smithi188_89276_object155968:head# 2024-04-23T16:25:14.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:266ea968:::benchmark_data_smithi188_89276_object67939:head# 2024-04-23T16:25:14.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:266f22f5:::benchmark_data_smithi188_89276_object168372:head# 2024-04-23T16:25:14.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:266fc15e:::benchmark_data_smithi188_89276_object69732:head# 2024-04-23T16:25:14.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2670a9dd:::benchmark_data_smithi188_89276_object81744:head# 2024-04-23T16:25:14.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26714a38:::benchmark_data_smithi188_89276_object163178:head# 2024-04-23T16:25:14.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:267192c3:::benchmark_data_smithi188_89276_object134142:head# 2024-04-23T16:25:14.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2671be05:::benchmark_data_smithi188_89276_object59712:head# 2024-04-23T16:25:14.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:267255fd:::benchmark_data_smithi188_89276_object109701:head# 2024-04-23T16:25:14.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2672d300:::benchmark_data_smithi188_89276_object137978:head# 2024-04-23T16:25:14.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2672fc09:::benchmark_data_smithi188_89276_object57467:head# 2024-04-23T16:25:14.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2673ccbe:::benchmark_data_smithi188_89276_object71584:head# 2024-04-23T16:25:14.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2673ce46:::benchmark_data_smithi188_89276_object88193:head# 2024-04-23T16:25:14.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:267402ff:::benchmark_data_smithi188_89276_object156113:head# 2024-04-23T16:25:14.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:267522e3:::benchmark_data_smithi188_89276_object161039:head# 2024-04-23T16:25:14.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2675800f:::benchmark_data_smithi188_89276_object149183:head# 2024-04-23T16:25:14.043 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:25:14.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2676b4de:::benchmark_data_smithi188_89276_object73141:head# 2024-04-23T16:25:14.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2676c18e:::benchmark_data_smithi188_89276_object72380:head# 2024-04-23T16:25:14.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:267713b4:::benchmark_data_smithi188_89276_object162364:head# 2024-04-23T16:25:14.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2677260a:::benchmark_data_smithi188_89276_object142636:head# 2024-04-23T16:25:14.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26773947:::benchmark_data_smithi188_89276_object122341:head# 2024-04-23T16:25:14.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26777f2f:::benchmark_data_smithi188_89276_object94179:head# 2024-04-23T16:25:14.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:14.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:25:14.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2677b6e9:::benchmark_data_smithi188_89276_object76269:head# 2024-04-23T16:25:14.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26789bed:::benchmark_data_smithi188_89276_object56137:head# 2024-04-23T16:25:14.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26791992:::benchmark_data_smithi188_89276_object79510:head# 2024-04-23T16:25:14.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:267965f0:::benchmark_data_smithi188_89276_object43549:head# 2024-04-23T16:25:14.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2679d553:::benchmark_data_smithi188_89276_object84846:head# 2024-04-23T16:25:14.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:267a4537:::benchmark_data_smithi188_89276_object100765:head# 2024-04-23T16:25:14.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:267a4c54:::benchmark_data_smithi188_89276_object153399:head# 2024-04-23T16:25:14.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:267c53ac:::benchmark_data_smithi188_89276_object107463:head# 2024-04-23T16:25:14.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:267cb3da:::benchmark_data_smithi188_89276_object155596:head# 2024-04-23T16:25:14.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:267d0ff0:::benchmark_data_smithi188_89276_object63201:head# 2024-04-23T16:25:14.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:267ece41:::benchmark_data_smithi188_89276_object131532:head# 2024-04-23T16:25:14.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:267efba8:::benchmark_data_smithi188_89276_object82728:head# 2024-04-23T16:25:14.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:267f8bd8:::benchmark_data_smithi188_89276_object143862:head# 2024-04-23T16:25:14.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:267fd118:::benchmark_data_smithi188_89276_object133265:head# 2024-04-23T16:25:14.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2680282b:::benchmark_data_smithi188_89276_object128947:head# 2024-04-23T16:25:14.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26816eb4:::benchmark_data_smithi188_89276_object66819:head# 2024-04-23T16:25:14.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26817600:::benchmark_data_smithi188_89276_object112093:head# 2024-04-23T16:25:14.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2682facf:::benchmark_data_smithi188_89276_object70309:head# 2024-04-23T16:25:14.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2684ac5e:::benchmark_data_smithi188_89276_object118680:head# 2024-04-23T16:25:14.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2684efd5:::benchmark_data_smithi188_89276_object68340:head# 2024-04-23T16:25:14.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26853018:::benchmark_data_smithi188_89276_object107213:head# 2024-04-23T16:25:14.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26854a19:::benchmark_data_smithi188_89276_object90112:head# 2024-04-23T16:25:14.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2685a3ad:::benchmark_data_smithi188_89276_object61245:head# 2024-04-23T16:25:14.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:268623fc:::benchmark_data_smithi188_89276_object99434:head# 2024-04-23T16:25:14.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26872a8d:::benchmark_data_smithi188_89276_object129213:head# 2024-04-23T16:25:14.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26882a48:::benchmark_data_smithi188_89276_object146738:head# 2024-04-23T16:25:14.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26890f5d:::benchmark_data_smithi188_89276_object108193:head# 2024-04-23T16:25:14.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2689b47e:::benchmark_data_smithi188_89276_object115455:head# 2024-04-23T16:25:14.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:268a9184:::benchmark_data_smithi188_89276_object59787:head# 2024-04-23T16:25:14.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:268ae579:::benchmark_data_smithi188_89276_object109541:head# 2024-04-23T16:25:14.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:268af870:::benchmark_data_smithi188_89276_object90748:head# 2024-04-23T16:25:14.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:268b50c3:::benchmark_data_smithi188_89276_object44101:head# 2024-04-23T16:25:14.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:268c57e0:::benchmark_data_smithi188_89276_object47593:head# 2024-04-23T16:25:14.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:268dd3b7:::benchmark_data_smithi188_89276_object168071:head# 2024-04-23T16:25:14.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:268edc28:::benchmark_data_smithi188_89276_object143548:head# 2024-04-23T16:25:14.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:268edce9:::benchmark_data_smithi188_89276_object77397:head# 2024-04-23T16:25:14.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26923f65:::benchmark_data_smithi188_89276_object77097:head# 2024-04-23T16:25:14.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2693cdcd:::benchmark_data_smithi188_89276_object151417:head# 2024-04-23T16:25:14.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2694122b:::benchmark_data_smithi188_89276_object94708:head# 2024-04-23T16:25:14.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26948b01:::benchmark_data_smithi188_89276_object78305:head# 2024-04-23T16:25:14.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2695045d:::benchmark_data_smithi188_89276_object102380:head# 2024-04-23T16:25:14.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26959c2d:::benchmark_data_smithi188_89276_object60389:head# 2024-04-23T16:25:14.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2696230f:::benchmark_data_smithi188_89276_object102527:head# 2024-04-23T16:25:14.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:269623e8:::benchmark_data_smithi188_89276_object72085:head# 2024-04-23T16:25:14.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26963755:::benchmark_data_smithi188_89276_object119541:head# 2024-04-23T16:25:14.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2696f50f:::benchmark_data_smithi188_89276_object113590:head# 2024-04-23T16:25:14.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26980744:::benchmark_data_smithi188_89276_object161805:head# 2024-04-23T16:25:14.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:269973bb:::benchmark_data_smithi188_89276_object148304:head# 2024-04-23T16:25:14.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2699af6b:::benchmark_data_smithi188_89276_object134466:head# 2024-04-23T16:25:14.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2699f56e:::benchmark_data_smithi188_89276_object123447:head# 2024-04-23T16:25:14.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:269a4457:::benchmark_data_smithi188_89276_object121592:head# 2024-04-23T16:25:14.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:269a7df8:::benchmark_data_smithi188_89276_object64353:head# 2024-04-23T16:25:14.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:269a8bd2:::benchmark_data_smithi188_89276_object83231:head# 2024-04-23T16:25:14.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:269b44ff:::benchmark_data_smithi188_89276_object136666:head# 2024-04-23T16:25:14.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:269ca5b3:::benchmark_data_smithi188_89276_object103834:head# 2024-04-23T16:25:14.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:269cffa4:::benchmark_data_smithi188_89276_object83125:head# 2024-04-23T16:25:14.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:269d954f:::benchmark_data_smithi188_89276_object146571:head# 2024-04-23T16:25:14.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:269e7bd7:::benchmark_data_smithi188_89276_object155057:head# 2024-04-23T16:25:14.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:269f6774:::benchmark_data_smithi188_89276_object116200:head# 2024-04-23T16:25:14.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:269fc628:::benchmark_data_smithi188_89276_object157664:head# 2024-04-23T16:25:14.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:269fe7ad:::benchmark_data_smithi188_89276_object144390:head# 2024-04-23T16:25:14.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26a044c9:::benchmark_data_smithi188_89276_object89053:head# 2024-04-23T16:25:14.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26a0b60e:::benchmark_data_smithi188_89276_object52756:head# 2024-04-23T16:25:14.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26a0e6af:::benchmark_data_smithi188_89276_object152214:head# 2024-04-23T16:25:14.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26a0f480:::benchmark_data_smithi188_89276_object132717:head# 2024-04-23T16:25:14.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26a17bc1:::benchmark_data_smithi188_89276_object97157:head# 2024-04-23T16:25:14.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26a1f922:::benchmark_data_smithi188_89276_object165865:head# 2024-04-23T16:25:14.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26a21aa7:::benchmark_data_smithi188_89276_object126864:head# 2024-04-23T16:25:14.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26a23d86:::benchmark_data_smithi188_89276_object78025:head# 2024-04-23T16:25:14.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26a37992:::benchmark_data_smithi188_89276_object130929:head# 2024-04-23T16:25:14.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26a3b9ab:::benchmark_data_smithi188_89276_object43971:head# 2024-04-23T16:25:14.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26a4c0d0:::benchmark_data_smithi188_89276_object106682:head# 2024-04-23T16:25:14.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26a4c275:::benchmark_data_smithi188_89276_object66258:head# 2024-04-23T16:25:14.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26a57fb1:::benchmark_data_smithi188_89276_object122124:head# 2024-04-23T16:25:14.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26a5c9eb:::benchmark_data_smithi188_89276_object45433:head# 2024-04-23T16:25:14.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26a6c765:::benchmark_data_smithi188_89276_object95827:head# 2024-04-23T16:25:14.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26a6dc31:::benchmark_data_smithi188_89276_object142316:head# 2024-04-23T16:25:14.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26a6fac1:::benchmark_data_smithi188_89276_object116130:head# 2024-04-23T16:25:14.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26a9704f:::benchmark_data_smithi188_89276_object142253:head# 2024-04-23T16:25:14.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26aa670b:::benchmark_data_smithi188_89276_object140772:head# 2024-04-23T16:25:14.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ab4806:::benchmark_data_smithi188_89276_object128290:head# 2024-04-23T16:25:14.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ab571c:::benchmark_data_smithi188_89276_object161037:head# 2024-04-23T16:25:14.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ac03cf:::benchmark_data_smithi188_89276_object44309:head# 2024-04-23T16:25:14.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ac467d:::benchmark_data_smithi188_89276_object96512:head# 2024-04-23T16:25:14.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ad5ebd:::benchmark_data_smithi188_89276_object69887:head# 2024-04-23T16:25:14.088 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26adc7ed:::benchmark_data_smithi188_89276_object61708:head# 2024-04-23T16:25:14.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ae673d:::benchmark_data_smithi188_89276_object150417:head# 2024-04-23T16:25:14.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26aec943:::benchmark_data_smithi188_89276_object109544:head# 2024-04-23T16:25:14.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26afd7d6:::benchmark_data_smithi188_89276_object93913:head# 2024-04-23T16:25:14.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26aff748:::benchmark_data_smithi188_89276_object129011:head# 2024-04-23T16:25:14.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b0cc6b:::benchmark_data_smithi188_89276_object112885:head# 2024-04-23T16:25:14.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b160c5:::benchmark_data_smithi188_89276_object104642:head# 2024-04-23T16:25:14.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b171d1:::benchmark_data_smithi188_89276_object138242:head# 2024-04-23T16:25:14.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b17dc3:::benchmark_data_smithi188_89276_object145238:head# 2024-04-23T16:25:14.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b1acf7:::benchmark_data_smithi188_89276_object40291:head# 2024-04-23T16:25:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b205f5:::benchmark_data_smithi188_89276_object94211:head# 2024-04-23T16:25:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b241d1:::benchmark_data_smithi188_89276_object167712:head# 2024-04-23T16:25:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b2885d:::benchmark_data_smithi188_89276_object83165:head# 2024-04-23T16:25:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b32d1e:::benchmark_data_smithi188_89276_object105534:head# 2024-04-23T16:25:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b330f5:::benchmark_data_smithi188_89276_object159282:head# 2024-04-23T16:25:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b3c736:::benchmark_data_smithi188_89276_object127096:head# 2024-04-23T16:25:14.096 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b3f548:::benchmark_data_smithi188_89276_object127850:head# 2024-04-23T16:25:14.102 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b4dd02:::benchmark_data_smithi188_89276_object165236:head# 2024-04-23T16:25:14.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b5de14:::benchmark_data_smithi188_89276_object77490:head# 2024-04-23T16:25:14.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b61fb2:::benchmark_data_smithi188_89276_object106705:head# 2024-04-23T16:25:14.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b71016:::benchmark_data_smithi188_89276_object138261:head# 2024-04-23T16:25:14.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b72944:::benchmark_data_smithi188_89276_object48594:head# 2024-04-23T16:25:14.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b7842b:::benchmark_data_smithi188_89276_object84827:head# 2024-04-23T16:25:14.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b8561b:::benchmark_data_smithi188_89276_object84187:head# 2024-04-23T16:25:14.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b8f9c3:::benchmark_data_smithi188_89276_object163550:head# 2024-04-23T16:25:14.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26b994e4:::benchmark_data_smithi188_89276_object147299:head# 2024-04-23T16:25:14.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ba5986:::benchmark_data_smithi188_89276_object40448:head# 2024-04-23T16:25:14.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ba6acd:::benchmark_data_smithi188_89276_object126610:head# 2024-04-23T16:25:14.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26baab07:::benchmark_data_smithi188_89276_object81394:head# 2024-04-23T16:25:14.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26bb24ca:::benchmark_data_smithi188_89276_object85975:head# 2024-04-23T16:25:14.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26bba62e:::benchmark_data_smithi188_89276_object85381:head# 2024-04-23T16:25:14.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26bbc461:::benchmark_data_smithi188_89276_object153564:head# 2024-04-23T16:25:14.111 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26bc3561:::benchmark_data_smithi188_89276_object118360:head# 2024-04-23T16:25:14.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26bc6ae7:::benchmark_data_smithi188_89276_object159687:head# 2024-04-23T16:25:14.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26bd0f9f:::benchmark_data_smithi188_89276_object156816:head# 2024-04-23T16:25:14.113 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26bd52a9:::benchmark_data_smithi188_89276_object150673:head# 2024-04-23T16:25:14.113 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26bdff0e:::benchmark_data_smithi188_89276_object96773:head# 2024-04-23T16:25:14.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26be006f:::benchmark_data_smithi188_89276_object74521:head# 2024-04-23T16:25:14.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26becace:::benchmark_data_smithi188_89276_object143299:head# 2024-04-23T16:25:14.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26bf35b6:::benchmark_data_smithi188_89276_object49230:head# 2024-04-23T16:25:14.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26bfc9e2:::benchmark_data_smithi188_89276_object112203:head# 2024-04-23T16:25:14.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26c08b1b:::benchmark_data_smithi188_89276_object159205:head# 2024-04-23T16:25:14.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26c18e9d:::benchmark_data_smithi188_89276_object129925:head# 2024-04-23T16:25:14.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26c2514a:::benchmark_data_smithi188_89276_object74387:head# 2024-04-23T16:25:14.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26c258f6:::benchmark_data_smithi188_89276_object142878:head# 2024-04-23T16:25:14.118 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26c2f464:::benchmark_data_smithi188_89276_object109310:head# 2024-04-23T16:25:14.118 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26c35a7f:::benchmark_data_smithi188_89276_object44729:head# 2024-04-23T16:25:14.118 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26c4d1ce:::benchmark_data_smithi188_89276_object165872:head# 2024-04-23T16:25:14.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26c514d8:::benchmark_data_smithi188_89276_object145527:head# 2024-04-23T16:25:14.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26c5d7a1:::benchmark_data_smithi188_89276_object116786:head# 2024-04-23T16:25:14.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26c5ef2d:::benchmark_data_smithi188_89276_object55412:head# 2024-04-23T16:25:14.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26c6f6f4:::benchmark_data_smithi188_89276_object129909:head# 2024-04-23T16:25:14.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26c85e77:::benchmark_data_smithi188_89276_object121090:head# 2024-04-23T16:25:14.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26c95fc8:::benchmark_data_smithi188_89276_object149171:head# 2024-04-23T16:25:14.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26c99891:::benchmark_data_smithi188_89276_object68588:head# 2024-04-23T16:25:14.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26c9a75f:::benchmark_data_smithi188_89276_object108679:head# 2024-04-23T16:25:14.122 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26c9d964:::benchmark_data_smithi188_89276_object50601:head# 2024-04-23T16:25:14.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ca3089:::benchmark_data_smithi188_89276_object70448:head# 2024-04-23T16:25:14.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ca6800:::benchmark_data_smithi188_89276_object144277:head# 2024-04-23T16:25:14.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26cba3eb:::benchmark_data_smithi188_89276_object76783:head# 2024-04-23T16:25:14.124 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26cbae6c:::benchmark_data_smithi188_89276_object106906:head# 2024-04-23T16:25:14.124 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26cc6496:::benchmark_data_smithi188_89276_object134916:head# 2024-04-23T16:25:14.124 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26cccf0a:::benchmark_data_smithi188_89276_object163041:head# 2024-04-23T16:25:14.125 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26cce75e:::benchmark_data_smithi188_89276_object56514:head# 2024-04-23T16:25:14.125 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26cddf6f:::benchmark_data_smithi188_89276_object104618:head# 2024-04-23T16:25:14.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26cde28a:::benchmark_data_smithi188_89276_object112704:head# 2024-04-23T16:25:14.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ce27a4:::benchmark_data_smithi188_89276_object94407:head# 2024-04-23T16:25:14.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ce8ab7:::benchmark_data_smithi188_89276_object81378:head# 2024-04-23T16:25:14.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26cf7347:::benchmark_data_smithi188_89276_object57460:head# 2024-04-23T16:25:14.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26cfa531:::benchmark_data_smithi188_89276_object154905:head# 2024-04-23T16:25:14.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26cfaed7:::benchmark_data_smithi188_89276_object158862:head# 2024-04-23T16:25:14.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26d02397:::benchmark_data_smithi188_89276_object53230:head# 2024-04-23T16:25:14.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26d06ea6:::benchmark_data_smithi188_89276_object41532:head# 2024-04-23T16:25:14.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26d097eb:::benchmark_data_smithi188_89276_object108944:head# 2024-04-23T16:25:14.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26d1e582:::benchmark_data_smithi188_89276_object41110:head# 2024-04-23T16:25:14.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26d1e8da:::benchmark_data_smithi188_89276_object82141:head# 2024-04-23T16:25:14.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26d1f2cb:::benchmark_data_smithi188_89276_object130152:head# 2024-04-23T16:25:14.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26d3a6a2:::benchmark_data_smithi188_89276_object128522:head# 2024-04-23T16:25:14.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26d519dc:::benchmark_data_smithi188_89276_object113521:head# 2024-04-23T16:25:14.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26d56e26:::benchmark_data_smithi188_89276_object156921:head# 2024-04-23T16:25:14.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26d6bb9d:::benchmark_data_smithi188_89276_object102532:head# 2024-04-23T16:25:14.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26d71366:::benchmark_data_smithi188_89276_object165088:head# 2024-04-23T16:25:14.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26d77564:::benchmark_data_smithi188_89276_object153300:head# 2024-04-23T16:25:14.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26d83216:::benchmark_data_smithi188_89276_object42631:head# 2024-04-23T16:25:14.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26d89ebb:::benchmark_data_smithi188_89276_object103254:head# 2024-04-23T16:25:14.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26d8cc5b:::benchmark_data_smithi188_89276_object61533:head# 2024-04-23T16:25:14.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26d96fc2:::benchmark_data_smithi188_89276_object69193:head# 2024-04-23T16:25:14.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26d97f8a:::benchmark_data_smithi188_89276_object74503:head# 2024-04-23T16:25:14.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26d9870c:::benchmark_data_smithi188_89276_object153345:head# 2024-04-23T16:25:14.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26da1f03:::benchmark_data_smithi188_89276_object143850:head# 2024-04-23T16:25:14.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26da6de5:::benchmark_data_smithi188_89276_object54414:head# 2024-04-23T16:25:14.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26da9cf0:::benchmark_data_smithi188_89276_object92153:head# 2024-04-23T16:25:14.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26db6c84:::benchmark_data_smithi188_89276_object127907:head# 2024-04-23T16:25:14.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26db9c19:::benchmark_data_smithi188_89276_object100250:head# 2024-04-23T16:25:14.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26dbed61:::benchmark_data_smithi188_89276_object78090:head# 2024-04-23T16:25:14.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26dc7a03:::benchmark_data_smithi188_89276_object132098:head# 2024-04-23T16:25:14.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26dc986e:::benchmark_data_smithi188_89276_object100618:head# 2024-04-23T16:25:14.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26dcd694:::benchmark_data_smithi188_89276_object163620:head# 2024-04-23T16:25:14.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26dda503:::benchmark_data_smithi188_89276_object107070:head# 2024-04-23T16:25:14.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ddb7bd:::benchmark_data_smithi188_89276_object152442:head# 2024-04-23T16:25:14.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26de0c84:::benchmark_data_smithi188_89276_object75567:head# 2024-04-23T16:25:14.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26de484d:::benchmark_data_smithi188_89276_object137479:head# 2024-04-23T16:25:14.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26de7933:::benchmark_data_smithi188_89276_object84789:head# 2024-04-23T16:25:14.142 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26df0fce:::benchmark_data_smithi188_89276_object92856:head# 2024-04-23T16:25:14.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26e00735:::benchmark_data_smithi188_89276_object57802:head# 2024-04-23T16:25:14.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26e1a2ff:::benchmark_data_smithi188_89276_object163056:head# 2024-04-23T16:25:14.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26e245f5:::benchmark_data_smithi188_89276_object91684:head# 2024-04-23T16:25:14.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26e2bba7:::benchmark_data_smithi188_89276_object113339:head# 2024-04-23T16:25:14.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26e2de7c:::benchmark_data_smithi188_89276_object75595:head# 2024-04-23T16:25:14.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26e332bf:::benchmark_data_smithi188_89276_object166203:head# 2024-04-23T16:25:14.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26e36e3d:::benchmark_data_smithi188_89276_object142479:head# 2024-04-23T16:25:14.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26e3a598:::benchmark_data_smithi188_89276_object109574:head# 2024-04-23T16:25:14.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26e41421:::benchmark_data_smithi188_89276_object140698:head# 2024-04-23T16:25:14.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26e5050e:::benchmark_data_smithi188_89276_object43119:head# 2024-04-23T16:25:14.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26e63726:::benchmark_data_smithi188_89276_object106243:head# 2024-04-23T16:25:14.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26e6bc5c:::benchmark_data_smithi188_89276_object59105:head# 2024-04-23T16:25:14.148 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:14.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26e6d8df:::benchmark_data_smithi188_89276_object55462:head# 2024-04-23T16:25:14.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26e83f48:::benchmark_data_smithi188_89276_object132808:head# 2024-04-23T16:25:14.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26e919a0:::benchmark_data_smithi188_89276_object109845:head# 2024-04-23T16:25:14.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ea0e66:::benchmark_data_smithi188_89276_object66833:head# 2024-04-23T16:25:14.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ea367a:::benchmark_data_smithi188_89276_object70176:head# 2024-04-23T16:25:14.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ea4ed8:::benchmark_data_smithi188_89276_object114431:head# 2024-04-23T16:25:14.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26eaa818:::benchmark_data_smithi188_89276_object128518:head# 2024-04-23T16:25:14.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26eadfdd:::benchmark_data_smithi188_89276_object47227:head# 2024-04-23T16:25:14.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ebd5f4:::benchmark_data_smithi188_89276_object150465:head# 2024-04-23T16:25:14.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ec37ac:::benchmark_data_smithi188_89276_object55814:head# 2024-04-23T16:25:14.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ecca35:::benchmark_data_smithi188_89276_object55716:head# 2024-04-23T16:25:14.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ed0ff5:::benchmark_data_smithi188_89276_object78530:head# 2024-04-23T16:25:14.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ed13a3:::benchmark_data_smithi188_89276_object91446:head# 2024-04-23T16:25:14.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ee19a9:::benchmark_data_smithi188_89276_object46243:head# 2024-04-23T16:25:14.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26eecdc3:::benchmark_data_smithi188_89276_object52674:head# 2024-04-23T16:25:14.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ef2a22:::benchmark_data_smithi188_89276_object87733:head# 2024-04-23T16:25:14.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f01ad8:::benchmark_data_smithi188_89276_object139396:head# 2024-04-23T16:25:14.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f0241f:::benchmark_data_smithi188_89276_object89105:head# 2024-04-23T16:25:14.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f07443:::benchmark_data_smithi188_89276_object101106:head# 2024-04-23T16:25:14.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f0cfaa:::benchmark_data_smithi188_89276_object51073:head# 2024-04-23T16:25:14.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f0d37e:::benchmark_data_smithi188_89276_object158762:head# 2024-04-23T16:25:14.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f177d0:::benchmark_data_smithi188_89276_object42073:head# 2024-04-23T16:25:14.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f2e674:::benchmark_data_smithi188_89276_object72698:head# 2024-04-23T16:25:14.158 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f36b28:::benchmark_data_smithi188_89276_object82066:head# 2024-04-23T16:25:14.158 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f3745e:::benchmark_data_smithi188_89276_object135356:head# 2024-04-23T16:25:14.158 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f3a23b:::benchmark_data_smithi188_89276_object85739:head# 2024-04-23T16:25:14.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f3ccdc:::benchmark_data_smithi188_89276_object63214:head# 2024-04-23T16:25:14.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f499fd:::benchmark_data_smithi188_89276_object98528:head# 2024-04-23T16:25:14.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f4cc3d:::benchmark_data_smithi188_89276_object64584:head# 2024-04-23T16:25:14.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f4ea6d:::benchmark_data_smithi188_89276_object50043:head# 2024-04-23T16:25:14.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f59f00:::benchmark_data_smithi188_89276_object59711:head# 2024-04-23T16:25:14.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f63584:::benchmark_data_smithi188_89276_object127081:head# 2024-04-23T16:25:14.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f68fd1:::benchmark_data_smithi188_89276_object81620:head# 2024-04-23T16:25:14.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f6a3c6:::benchmark_data_smithi188_89276_object74365:head# 2024-04-23T16:25:14.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f7d51f:::benchmark_data_smithi188_89276_object96533:head# 2024-04-23T16:25:14.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f7db0c:::benchmark_data_smithi188_89276_object163981:head# 2024-04-23T16:25:14.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f7eb20:::benchmark_data_smithi188_89276_object144887:head# 2024-04-23T16:25:14.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f8420d:::benchmark_data_smithi188_89276_object96405:head# 2024-04-23T16:25:14.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f84dd6:::benchmark_data_smithi188_89276_object133831:head# 2024-04-23T16:25:14.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f8ab3f:::benchmark_data_smithi188_89276_object130429:head# 2024-04-23T16:25:14.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f8c8c6:::benchmark_data_smithi188_89276_object91137:head# 2024-04-23T16:25:14.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f91033:::benchmark_data_smithi188_89276_object150031:head# 2024-04-23T16:25:14.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f94bad:::benchmark_data_smithi188_89276_object83682:head# 2024-04-23T16:25:14.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f98d6d:::benchmark_data_smithi188_89276_object47118:head# 2024-04-23T16:25:14.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26f9dd00:::benchmark_data_smithi188_89276_object144521:head# 2024-04-23T16:25:14.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26fa9e9d:::benchmark_data_smithi188_89276_object120530:head# 2024-04-23T16:25:14.176 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26fafdd3:::benchmark_data_smithi188_89276_object107059:head# 2024-04-23T16:25:14.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26fc14f0:::benchmark_data_smithi188_89276_object126841:head# 2024-04-23T16:25:14.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26fc5ffc:::benchmark_data_smithi188_89276_object126784:head# 2024-04-23T16:25:14.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26fcf13e:::benchmark_data_smithi188_89276_object138582:head# 2024-04-23T16:25:14.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26fe7afc:::benchmark_data_smithi188_89276_object117022:head# 2024-04-23T16:25:14.181 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26fec601:::benchmark_data_smithi188_89276_object83324:head# 2024-04-23T16:25:14.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ffac54:::benchmark_data_smithi188_89276_object56776:head# 2024-04-23T16:25:14.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:26ffc0c0:::benchmark_data_smithi188_89276_object97580:head# 2024-04-23T16:25:14.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2701657e:::benchmark_data_smithi188_89276_object79665:head# 2024-04-23T16:25:14.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27017806:::benchmark_data_smithi188_89276_object83955:head# 2024-04-23T16:25:14.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2701b29b:::benchmark_data_smithi188_89276_object44927:head# 2024-04-23T16:25:14.185 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:14.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27024600:::benchmark_data_smithi188_89276_object100528:head# 2024-04-23T16:25:14.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27025dde:::benchmark_data_smithi188_89276_object151864:head# 2024-04-23T16:25:14.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2702f262:::benchmark_data_smithi188_89276_object144743:head# 2024-04-23T16:25:14.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2703243f:::benchmark_data_smithi188_89276_object98837:head# 2024-04-23T16:25:14.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2703bb37:::benchmark_data_smithi188_89276_object40905:head# 2024-04-23T16:25:14.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2703bf61:::benchmark_data_smithi188_89276_object152029:head# 2024-04-23T16:25:14.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2703cf6f:::benchmark_data_smithi188_89276_object43508:head# 2024-04-23T16:25:14.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:270490cb:::benchmark_data_smithi188_89276_object112196:head# 2024-04-23T16:25:14.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2704dbfc:::benchmark_data_smithi188_89276_object109453:head# 2024-04-23T16:25:14.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2704f87c:::benchmark_data_smithi188_89276_object101233:head# 2024-04-23T16:25:14.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:270514a3:::benchmark_data_smithi188_89276_object96444:head# 2024-04-23T16:25:14.192 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:270571de:::benchmark_data_smithi188_89276_object123764:head# 2024-04-23T16:25:14.192 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27058a0c:::benchmark_data_smithi188_89276_object127752:head# 2024-04-23T16:25:14.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2705d278:::benchmark_data_smithi188_89276_object133120:head# 2024-04-23T16:25:14.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27067804:::benchmark_data_smithi188_89276_object62056:head# 2024-04-23T16:25:14.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2707fd0a:::benchmark_data_smithi188_89276_object66880:head# 2024-04-23T16:25:14.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2708203d:::benchmark_data_smithi188_89276_object45636:head# 2024-04-23T16:25:14.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2708ca47:::benchmark_data_smithi188_89276_object76782:head# 2024-04-23T16:25:14.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27090a91:::benchmark_data_smithi188_89276_object69224:head# 2024-04-23T16:25:14.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2709faf3:::benchmark_data_smithi188_89276_object144458:head# 2024-04-23T16:25:14.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:270ae8e6:::benchmark_data_smithi188_89276_object134181:head# 2024-04-23T16:25:14.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:270b7984:::benchmark_data_smithi188_89276_object121227:head# 2024-04-23T16:25:14.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:270ba80a:::benchmark_data_smithi188_89276_object54284:head# 2024-04-23T16:25:14.197 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:270c98dd:::benchmark_data_smithi188_89276_object121773:head# 2024-04-23T16:25:14.197 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:270d8680:::benchmark_data_smithi188_89276_object79468:head# 2024-04-23T16:25:14.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:270d9d9b:::benchmark_data_smithi188_89276_object128071:head# 2024-04-23T16:25:14.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:270da8ec:::benchmark_data_smithi188_89276_object118739:head# 2024-04-23T16:25:14.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:270e831b:::benchmark_data_smithi188_89276_object94947:head# 2024-04-23T16:25:14.199 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:270f0344:::benchmark_data_smithi188_89276_object142580:head# 2024-04-23T16:25:14.199 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:270f1504:::benchmark_data_smithi188_89276_object122082:head# 2024-04-23T16:25:14.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:270f9eb5:::benchmark_data_smithi188_89276_object96626:head# 2024-04-23T16:25:14.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:270fdd00:::benchmark_data_smithi188_89276_object60171:head# 2024-04-23T16:25:14.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2711cd30:::benchmark_data_smithi188_89276_object70235:head# 2024-04-23T16:25:14.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2712723d:::benchmark_data_smithi188_89276_object59685:head# 2024-04-23T16:25:14.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27128129:::benchmark_data_smithi188_89276_object78105:head# 2024-04-23T16:25:14.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2713e93f:::benchmark_data_smithi188_89276_object146994:head# 2024-04-23T16:25:14.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:271486fc:::benchmark_data_smithi188_89276_object68949:head# 2024-04-23T16:25:14.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2715bab4:::benchmark_data_smithi188_89276_object122451:head# 2024-04-23T16:25:14.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2716c243:::benchmark_data_smithi188_89276_object47497:head# 2024-04-23T16:25:14.206 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2716cb25:::benchmark_data_smithi188_89276_object129643:head# 2024-04-23T16:25:14.207 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27172926:::benchmark_data_smithi188_89276_object88857:head# 2024-04-23T16:25:14.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27173e3e:::benchmark_data_smithi188_89276_object112708:head# 2024-04-23T16:25:14.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27180709:::benchmark_data_smithi188_89276_object44060:head# 2024-04-23T16:25:14.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2718a363:::benchmark_data_smithi188_89276_object145568:head# 2024-04-23T16:25:14.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2718aed1:::benchmark_data_smithi188_89276_object158178:head# 2024-04-23T16:25:14.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2718eac8:::benchmark_data_smithi188_89276_object75668:head# 2024-04-23T16:25:14.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27190318:::benchmark_data_smithi188_89276_object95140:head# 2024-04-23T16:25:14.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:271a505d:::benchmark_data_smithi188_89276_object123445:head# 2024-04-23T16:25:14.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:271aaafb:::benchmark_data_smithi188_89276_object135256:head# 2024-04-23T16:25:14.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:271ae22f:::benchmark_data_smithi188_89276_object91344:head# 2024-04-23T16:25:14.212 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:271afa2d:::benchmark_data_smithi188_89276_object144715:head# 2024-04-23T16:25:14.213 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:271b13de:::benchmark_data_smithi188_89276_object76342:head# 2024-04-23T16:25:14.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:271c1364:::benchmark_data_smithi188_89276_object74419:head# 2024-04-23T16:25:14.216 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:271d1b08:::benchmark_data_smithi188_89276_object68752:head# 2024-04-23T16:25:14.216 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:271d4912:::benchmark_data_smithi188_89276_object61477:head# 2024-04-23T16:25:14.216 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:271dce72:::benchmark_data_smithi188_89276_object126563:head# 2024-04-23T16:25:14.216 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:271e160a:::benchmark_data_smithi188_89276_object168519:head# 2024-04-23T16:25:14.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:271e7e1a:::benchmark_data_smithi188_89276_object51217:head# 2024-04-23T16:25:14.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:271ed90c:::benchmark_data_smithi188_89276_object79283:head# 2024-04-23T16:25:14.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:271f0f35:::benchmark_data_smithi188_89276_object123494:head# 2024-04-23T16:25:14.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:271f5d97:::benchmark_data_smithi188_89276_object96384:head# 2024-04-23T16:25:14.221 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:271f8d43:::benchmark_data_smithi188_89276_object73597:head# 2024-04-23T16:25:14.221 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:271fa816:::benchmark_data_smithi188_89276_object81062:head# 2024-04-23T16:25:14.221 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:271fed3a:::benchmark_data_smithi188_89276_object101441:head# 2024-04-23T16:25:14.222 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2720344e:::benchmark_data_smithi188_89276_object109540:head# 2024-04-23T16:25:14.222 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2720f1ae:::benchmark_data_smithi188_89276_object104823:head# 2024-04-23T16:25:14.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:13 smithi188 ceph-mon[44101]: pgmap v715: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 228 op/s; 20 MiB/s, 316 objects/s recovering 2024-04-23T16:25:14.224 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:272328f3:::benchmark_data_smithi188_89276_object116853:head# 2024-04-23T16:25:14.224 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:272349a0:::benchmark_data_smithi188_89276_object94148:head# 2024-04-23T16:25:14.225 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2723c6ed:::benchmark_data_smithi188_89276_object61749:head# 2024-04-23T16:25:14.225 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2723e227:::benchmark_data_smithi188_89276_object88495:head# 2024-04-23T16:25:14.225 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2723f93e:::benchmark_data_smithi188_89276_object87713:head# 2024-04-23T16:25:14.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2724a407:::benchmark_data_smithi188_89276_object106382:head# 2024-04-23T16:25:14.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:272564ed:::benchmark_data_smithi188_89276_object97212:head# 2024-04-23T16:25:14.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2726e208:::benchmark_data_smithi188_89276_object147211:head# 2024-04-23T16:25:14.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27270bec:::benchmark_data_smithi188_89276_object159647:head# 2024-04-23T16:25:14.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27278002:::benchmark_data_smithi188_89276_object56986:head# 2024-04-23T16:25:14.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2727fb88:::benchmark_data_smithi188_89276_object147714:head# 2024-04-23T16:25:14.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27281293:::benchmark_data_smithi188_89276_object125438:head# 2024-04-23T16:25:14.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2728db94:::benchmark_data_smithi188_89276_object95330:head# 2024-04-23T16:25:14.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2728f215:::benchmark_data_smithi188_89276_object153687:head# 2024-04-23T16:25:14.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2729d068:::benchmark_data_smithi188_89276_object108720:head# 2024-04-23T16:25:14.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:272ad8cb:::benchmark_data_smithi188_89276_object90730:head# 2024-04-23T16:25:14.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:272b0dbd:::benchmark_data_smithi188_89276_object82390:head# 2024-04-23T16:25:14.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:272c2749:::benchmark_data_smithi188_89276_object132788:head# 2024-04-23T16:25:14.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:272d530e:::benchmark_data_smithi188_89276_object47954:head# 2024-04-23T16:25:14.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:272daa19:::benchmark_data_smithi188_89276_object166498:head# 2024-04-23T16:25:14.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:272e4b46:::benchmark_data_smithi188_89276_object101464:head# 2024-04-23T16:25:14.234 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:272f3567:::benchmark_data_smithi188_89276_object64691:head# 2024-04-23T16:25:14.234 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:272f5cb0:::benchmark_data_smithi188_89276_object135460:head# 2024-04-23T16:25:14.234 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27310200:::benchmark_data_smithi188_89276_object63347:head# 2024-04-23T16:25:14.235 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2731121b:::benchmark_data_smithi188_89276_object50122:head# 2024-04-23T16:25:14.235 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2731b1ae:::benchmark_data_smithi188_89276_object41683:head# 2024-04-23T16:25:14.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2731c4b7:::benchmark_data_smithi188_89276_object103563:head# 2024-04-23T16:25:14.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2731ec1a:::benchmark_data_smithi188_89276_object75262:head# 2024-04-23T16:25:14.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27329cac:::benchmark_data_smithi188_89276_object122483:head# 2024-04-23T16:25:14.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2732a3dc:::benchmark_data_smithi188_89276_object52279:head# 2024-04-23T16:25:14.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2732f60f:::benchmark_data_smithi188_89276_object87615:head# 2024-04-23T16:25:14.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2733a5cc:::benchmark_data_smithi188_89276_object44875:head# 2024-04-23T16:25:14.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:273508fa:::benchmark_data_smithi188_89276_object73147:head# 2024-04-23T16:25:14.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27353d16:::benchmark_data_smithi188_89276_object109017:head# 2024-04-23T16:25:14.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:273561aa:::benchmark_data_smithi188_89276_object134994:head# 2024-04-23T16:25:14.240 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:273569fc:::benchmark_data_smithi188_89276_object42892:head# 2024-04-23T16:25:14.240 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:273672d3:::benchmark_data_smithi188_89276_object60304:head# 2024-04-23T16:25:14.241 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2736f476:::benchmark_data_smithi188_89276_object43042:head# 2024-04-23T16:25:14.241 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2736f7a1:::benchmark_data_smithi188_89276_object67071:head# 2024-04-23T16:25:14.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27376a65:::benchmark_data_smithi188_89276_object88535:head# 2024-04-23T16:25:14.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27378e75:::benchmark_data_smithi188_89276_object75603:head# 2024-04-23T16:25:14.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2737e6c8:::benchmark_data_smithi188_89276_object102134:head# 2024-04-23T16:25:14.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:273828ad:::benchmark_data_smithi188_89276_object132985:head# 2024-04-23T16:25:14.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2738716a:::benchmark_data_smithi188_89276_object161271:head# 2024-04-23T16:25:14.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2738b383:::benchmark_data_smithi188_89276_object105368:head# 2024-04-23T16:25:14.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27390d8b:::benchmark_data_smithi188_89276_object49716:head# 2024-04-23T16:25:14.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:273912d4:::benchmark_data_smithi188_89276_object67271:head# 2024-04-23T16:25:14.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:273925b5:::benchmark_data_smithi188_89276_object104563:head# 2024-04-23T16:25:14.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27392f32:::benchmark_data_smithi188_89276_object90398:head# 2024-04-23T16:25:14.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27397aaa:::benchmark_data_smithi188_89276_object111026:head# 2024-04-23T16:25:14.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2739804d:::benchmark_data_smithi188_89276_object99000:head# 2024-04-23T16:25:14.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2739ba05:::benchmark_data_smithi188_89276_object102007:head# 2024-04-23T16:25:14.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2739da46:::benchmark_data_smithi188_89276_object90120:head# 2024-04-23T16:25:14.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2739ea9e:::benchmark_data_smithi188_89276_object79641:head# 2024-04-23T16:25:14.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:273a340c:::benchmark_data_smithi188_89276_object75058:head# 2024-04-23T16:25:14.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:273b0e97:::benchmark_data_smithi188_89276_object98005:head# 2024-04-23T16:25:14.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:273ba19e:::benchmark_data_smithi188_89276_object88792:head# 2024-04-23T16:25:14.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:273c59f1:::benchmark_data_smithi188_89276_object143062:head# 2024-04-23T16:25:14.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:273d5ed8:::benchmark_data_smithi188_89276_object51432:head# 2024-04-23T16:25:14.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:273dad8a:::benchmark_data_smithi188_89276_object97498:head# 2024-04-23T16:25:14.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:273de6ba:::benchmark_data_smithi188_89276_object148840:head# 2024-04-23T16:25:14.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:273e8e4b:::benchmark_data_smithi188_89276_object107971:head# 2024-04-23T16:25:14.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:273ee04f:::benchmark_data_smithi188_89276_object134299:head# 2024-04-23T16:25:14.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:273f07b5:::benchmark_data_smithi188_89276_object92935:head# 2024-04-23T16:25:14.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:273f1893:::benchmark_data_smithi188_89276_object53812:head# 2024-04-23T16:25:14.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:273f3557:::benchmark_data_smithi188_89276_object142599:head# 2024-04-23T16:25:14.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:274081cc:::benchmark_data_smithi188_89276_object77245:head# 2024-04-23T16:25:14.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27408f79:::benchmark_data_smithi188_89276_object81890:head# 2024-04-23T16:25:14.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:274299d3:::benchmark_data_smithi188_89276_object152248:head# 2024-04-23T16:25:14.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2742e378:::benchmark_data_smithi188_89276_object156530:head# 2024-04-23T16:25:14.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2743023c:::benchmark_data_smithi188_89276_object78923:head# 2024-04-23T16:25:14.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27430acf:::benchmark_data_smithi188_89276_object153863:head# 2024-04-23T16:25:14.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2743a0e4:::benchmark_data_smithi188_89276_object120787:head# 2024-04-23T16:25:14.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:274410b1:::benchmark_data_smithi188_89276_object108611:head# 2024-04-23T16:25:14.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:274431b4:::benchmark_data_smithi188_89276_object123549:head# 2024-04-23T16:25:14.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27447292:::benchmark_data_smithi188_89276_object161265:head# 2024-04-23T16:25:14.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27459462:::benchmark_data_smithi188_89276_object166298:head# 2024-04-23T16:25:14.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2745d028:::benchmark_data_smithi188_89276_object128155:head# 2024-04-23T16:25:14.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2745ec92:::benchmark_data_smithi188_89276_object119544:head# 2024-04-23T16:25:14.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27461deb:::benchmark_data_smithi188_89276_object136936:head# 2024-04-23T16:25:14.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2746be57:::benchmark_data_smithi188_89276_object113562:head# 2024-04-23T16:25:14.261 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:14.262 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2746d480:::benchmark_data_smithi188_89276_object100459:head# 2024-04-23T16:25:14.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27470edd:::benchmark_data_smithi188_89276_object61067:head# 2024-04-23T16:25:14.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27474237:::benchmark_data_smithi188_89276_object124332:head# 2024-04-23T16:25:14.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2747eb51:::benchmark_data_smithi188_89276_object102698:head# 2024-04-23T16:25:14.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:274991a7:::benchmark_data_smithi188_89276_object44358:head# 2024-04-23T16:25:14.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2749b0ff:::benchmark_data_smithi188_89276_object121204:head# 2024-04-23T16:25:14.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:14.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:14.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:274a2af1:::benchmark_data_smithi188_89276_object98980:head# 2024-04-23T16:25:14.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:274a60df:::benchmark_data_smithi188_89276_object60131:head# 2024-04-23T16:25:14.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:274b10d6:::benchmark_data_smithi188_89276_object82429:head# 2024-04-23T16:25:14.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:274b41df:::benchmark_data_smithi188_89276_object168469:head# 2024-04-23T16:25:14.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:274b9e4e:::benchmark_data_smithi188_89276_object41267:head# 2024-04-23T16:25:14.266 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:274bc9b8:::benchmark_data_smithi188_89276_object98588:head# 2024-04-23T16:25:14.266 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:274d61fa:::benchmark_data_smithi188_89276_object60991:head# 2024-04-23T16:25:14.266 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:274d7434:::benchmark_data_smithi188_89276_object150116:head# 2024-04-23T16:25:14.267 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:274da6ad:::benchmark_data_smithi188_89276_object137457:head# 2024-04-23T16:25:14.267 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:274efd3e:::benchmark_data_smithi188_89276_object123833:head# 2024-04-23T16:25:14.267 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27505944:::benchmark_data_smithi188_89276_object147351:head# 2024-04-23T16:25:14.268 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27505d0f:::benchmark_data_smithi188_89276_object61286:head# 2024-04-23T16:25:14.268 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:275148f8:::benchmark_data_smithi188_89276_object124059:head# 2024-04-23T16:25:14.269 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:275238cc:::benchmark_data_smithi188_89276_object128567:head# 2024-04-23T16:25:14.269 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27526407:::benchmark_data_smithi188_89276_object71125:head# 2024-04-23T16:25:14.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2752d158:::benchmark_data_smithi188_89276_object65798:head# 2024-04-23T16:25:14.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2752df97:::benchmark_data_smithi188_89276_object83741:head# 2024-04-23T16:25:14.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27530d92:::benchmark_data_smithi188_89276_object156524:head# 2024-04-23T16:25:14.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27531638:::benchmark_data_smithi188_89276_object132738:head# 2024-04-23T16:25:14.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27532eaa:::benchmark_data_smithi188_89276_object124452:head# 2024-04-23T16:25:14.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27537e00:::benchmark_data_smithi188_89276_object41810:head# 2024-04-23T16:25:14.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2753c7d3:::benchmark_data_smithi188_89276_object100322:head# 2024-04-23T16:25:14.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:275460b0:::benchmark_data_smithi188_89276_object67055:head# 2024-04-23T16:25:14.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2754679d:::benchmark_data_smithi188_89276_object82491:head# 2024-04-23T16:25:14.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2754c012:::benchmark_data_smithi188_89276_object105499:head# 2024-04-23T16:25:14.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27567bd9:::benchmark_data_smithi188_89276_object79497:head# 2024-04-23T16:25:14.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27573351:::benchmark_data_smithi188_89276_object87354:head# 2024-04-23T16:25:14.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2758436d:::benchmark_data_smithi188_89276_object66710:head# 2024-04-23T16:25:14.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:275a23cf:::benchmark_data_smithi188_89276_object55310:head# 2024-04-23T16:25:14.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:275bb89d:::benchmark_data_smithi188_89276_object125415:head# 2024-04-23T16:25:14.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:275bd294:::benchmark_data_smithi188_89276_object165487:head# 2024-04-23T16:25:14.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:275c31a3:::benchmark_data_smithi188_89276_object136517:head# 2024-04-23T16:25:14.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:275cb8fe:::benchmark_data_smithi188_89276_object97248:head# 2024-04-23T16:25:14.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:275d87ea:::benchmark_data_smithi188_89276_object161948:head# 2024-04-23T16:25:14.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:275d8bb6:::benchmark_data_smithi188_89276_object111066:head# 2024-04-23T16:25:14.277 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:275daf26:::benchmark_data_smithi188_89276_object147601:head# 2024-04-23T16:25:14.277 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:275df54c:::benchmark_data_smithi188_89276_object148996:head# 2024-04-23T16:25:14.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:275faa55:::benchmark_data_smithi188_89276_object156487:head# 2024-04-23T16:25:14.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2760a433:::benchmark_data_smithi188_89276_object50522:head# 2024-04-23T16:25:14.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:276125d4:::benchmark_data_smithi188_89276_object43777:head# 2024-04-23T16:25:14.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2762a795:::benchmark_data_smithi188_89276_object149227:head# 2024-04-23T16:25:14.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2762b880:::benchmark_data_smithi188_89276_object61456:head# 2024-04-23T16:25:14.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2762f919:::benchmark_data_smithi188_89276_object63258:head# 2024-04-23T16:25:14.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27632816:::benchmark_data_smithi188_89276_object110747:head# 2024-04-23T16:25:14.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27636af8:::benchmark_data_smithi188_89276_object49939:head# 2024-04-23T16:25:14.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27639ab6:::benchmark_data_smithi188_89276_object153135:head# 2024-04-23T16:25:14.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2763b774:::benchmark_data_smithi188_89276_object77018:head# 2024-04-23T16:25:14.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2764cf68:::benchmark_data_smithi188_89276_object151020:head# 2024-04-23T16:25:14.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2764fed1:::benchmark_data_smithi188_89276_object111576:head# 2024-04-23T16:25:14.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2765a491:::benchmark_data_smithi188_89276_object47139:head# 2024-04-23T16:25:14.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2765c04a:::benchmark_data_smithi188_89276_object79226:head# 2024-04-23T16:25:14.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2765f14c:::benchmark_data_smithi188_89276_object77169:head# 2024-04-23T16:25:14.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2765f940:::benchmark_data_smithi188_89276_object64783:head# 2024-04-23T16:25:14.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27662794:::benchmark_data_smithi188_89276_object126750:head# 2024-04-23T16:25:14.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:276751cb:::benchmark_data_smithi188_89276_object81029:head# 2024-04-23T16:25:14.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27682fba:::benchmark_data_smithi188_89276_object158231:head# 2024-04-23T16:25:14.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27688b70:::benchmark_data_smithi188_89276_object159173:head# 2024-04-23T16:25:14.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2768bc06:::benchmark_data_smithi188_89276_object67941:head# 2024-04-23T16:25:14.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27690ce6:::benchmark_data_smithi188_89276_object72573:head# 2024-04-23T16:25:14.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27691d4c:::benchmark_data_smithi188_89276_object65529:head# 2024-04-23T16:25:14.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27693aaa:::benchmark_data_smithi188_89276_object152435:head# 2024-04-23T16:25:14.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:276a7efc:::benchmark_data_smithi188_89276_object49433:head# 2024-04-23T16:25:14.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:276b0b56:::benchmark_data_smithi188_89276_object154931:head# 2024-04-23T16:25:14.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:276c140c:::benchmark_data_smithi188_89276_object41946:head# 2024-04-23T16:25:14.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:276d2184:::benchmark_data_smithi188_89276_object62476:head# 2024-04-23T16:25:14.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:276dc543:::benchmark_data_smithi188_89276_object54736:head# 2024-04-23T16:25:14.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:276e431f:::benchmark_data_smithi188_89276_object58292:head# 2024-04-23T16:25:14.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:276e66a6:::benchmark_data_smithi188_89276_object82440:head# 2024-04-23T16:25:14.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:276f8325:::benchmark_data_smithi188_89276_object122853:head# 2024-04-23T16:25:14.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:276fad3b:::benchmark_data_smithi188_89276_object138123:head# 2024-04-23T16:25:14.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27703fd7:::benchmark_data_smithi188_89276_object53067:head# 2024-04-23T16:25:14.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27708a81:::benchmark_data_smithi188_89276_object147028:head# 2024-04-23T16:25:14.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2770ac18:::benchmark_data_smithi188_89276_object116083:head# 2024-04-23T16:25:14.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2771327f:::benchmark_data_smithi188_89276_object161900:head# 2024-04-23T16:25:14.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2771a80e:::benchmark_data_smithi188_89276_object107252:head# 2024-04-23T16:25:14.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2772450f:::benchmark_data_smithi188_89276_object126030:head# 2024-04-23T16:25:14.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2774fe24:::benchmark_data_smithi188_89276_object82714:head# 2024-04-23T16:25:14.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27750df9:::benchmark_data_smithi188_89276_object117462:head# 2024-04-23T16:25:14.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27750e80:::benchmark_data_smithi188_89276_object151383:head# 2024-04-23T16:25:14.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27753de4:::benchmark_data_smithi188_89276_object100505:head# 2024-04-23T16:25:14.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2778b267:::benchmark_data_smithi188_89276_object91630:head# 2024-04-23T16:25:14.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27795e60:::benchmark_data_smithi188_89276_object83710:head# 2024-04-23T16:25:14.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2779c868:::benchmark_data_smithi188_89276_object123209:head# 2024-04-23T16:25:14.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2779d3b4:::benchmark_data_smithi188_89276_object106460:head# 2024-04-23T16:25:14.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:277a0f04:::benchmark_data_smithi188_89276_object113288:head# 2024-04-23T16:25:14.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:277a553d:::benchmark_data_smithi188_89276_object106667:head# 2024-04-23T16:25:14.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:277b4804:::benchmark_data_smithi188_89276_object132935:head# 2024-04-23T16:25:14.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:277b524b:::benchmark_data_smithi188_89276_object80247:head# 2024-04-23T16:25:14.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:277bb201:::benchmark_data_smithi188_89276_object66074:head# 2024-04-23T16:25:14.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:277c1560:::benchmark_data_smithi188_89276_object43566:head# 2024-04-23T16:25:14.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:277c3843:::benchmark_data_smithi188_89276_object66476:head# 2024-04-23T16:25:14.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:277cfa94:::benchmark_data_smithi188_89276_object142735:head# 2024-04-23T16:25:14.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:277d17ec:::benchmark_data_smithi188_89276_object125711:head# 2024-04-23T16:25:14.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:277d508b:::benchmark_data_smithi188_89276_object109691:head# 2024-04-23T16:25:14.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:277d79db:::benchmark_data_smithi188_89276_object104435:head# 2024-04-23T16:25:14.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:277dc765:::benchmark_data_smithi188_89276_object143461:head# 2024-04-23T16:25:14.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:277dcfd9:::benchmark_data_smithi188_89276_object135008:head# 2024-04-23T16:25:14.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:277e0db0:::benchmark_data_smithi188_89276_object154553:head# 2024-04-23T16:25:14.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:277e5652:::benchmark_data_smithi188_89276_object156520:head# 2024-04-23T16:25:14.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:277ed9d7:::benchmark_data_smithi188_89276_object101000:head# 2024-04-23T16:25:14.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:277ee475:::benchmark_data_smithi188_89276_object120264:head# 2024-04-23T16:25:14.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:277f0555:::benchmark_data_smithi188_89276_object78918:head# 2024-04-23T16:25:14.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:277fe1c9:::benchmark_data_smithi188_89276_object108686:head# 2024-04-23T16:25:14.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27813b61:::benchmark_data_smithi188_89276_object123851:head# 2024-04-23T16:25:14.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27823c43:::benchmark_data_smithi188_89276_object56598:head# 2024-04-23T16:25:14.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27831e6c:::benchmark_data_smithi188_89276_object42272:head# 2024-04-23T16:25:14.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:278341a3:::benchmark_data_smithi188_89276_object162496:head# 2024-04-23T16:25:14.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2783c558:::benchmark_data_smithi188_89276_object147049:head# 2024-04-23T16:25:14.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2783c9fa:::benchmark_data_smithi188_89276_object139307:head# 2024-04-23T16:25:14.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27840561:::benchmark_data_smithi188_89276_object67663:head# 2024-04-23T16:25:14.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27856a3e:::benchmark_data_smithi188_89276_object106386:head# 2024-04-23T16:25:14.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:278613d7:::benchmark_data_smithi188_89276_object163806:head# 2024-04-23T16:25:14.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2786c36f:::benchmark_data_smithi188_89276_object143252:head# 2024-04-23T16:25:14.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27875741:::benchmark_data_smithi188_89276_object156238:head# 2024-04-23T16:25:14.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2787ba2b:::benchmark_data_smithi188_89276_object56477:head# 2024-04-23T16:25:14.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2787fe45:::benchmark_data_smithi188_89276_object70101:head# 2024-04-23T16:25:14.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27884e3c:::benchmark_data_smithi188_89276_object121037:head# 2024-04-23T16:25:14.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2789be8a:::benchmark_data_smithi188_89276_object60548:head# 2024-04-23T16:25:14.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:278b84e9:::benchmark_data_smithi188_89276_object69556:head# 2024-04-23T16:25:14.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:278bc41b:::benchmark_data_smithi188_89276_object166866:head# 2024-04-23T16:25:14.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:278bd7b6:::benchmark_data_smithi188_89276_object110252:head# 2024-04-23T16:25:14.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:278be778:::benchmark_data_smithi188_89276_object72745:head# 2024-04-23T16:25:14.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:278c005d:::benchmark_data_smithi188_89276_object76830:head# 2024-04-23T16:25:14.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:278ce92c:::benchmark_data_smithi188_89276_object137460:head# 2024-04-23T16:25:14.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:278d5966:::benchmark_data_smithi188_89276_object59240:head# 2024-04-23T16:25:14.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:278d6dc3:::benchmark_data_smithi188_89276_object79251:head# 2024-04-23T16:25:14.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:278e85af:::benchmark_data_smithi188_89276_object84952:head# 2024-04-23T16:25:14.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:278f2948:::benchmark_data_smithi188_89276_object67825:head# 2024-04-23T16:25:14.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:278fd850:::benchmark_data_smithi188_89276_object99401:head# 2024-04-23T16:25:14.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27902576:::benchmark_data_smithi188_89276_object118867:head# 2024-04-23T16:25:14.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2791069e:::benchmark_data_smithi188_89276_object165041:head# 2024-04-23T16:25:14.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:279113e6:::benchmark_data_smithi188_89276_object120547:head# 2024-04-23T16:25:14.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:279129a5:::benchmark_data_smithi188_89276_object161653:head# 2024-04-23T16:25:14.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2791ae34:::benchmark_data_smithi188_89276_object48281:head# 2024-04-23T16:25:14.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2791e209:::benchmark_data_smithi188_89276_object109592:head# 2024-04-23T16:25:14.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27922e2f:::benchmark_data_smithi188_89276_object41913:head# 2024-04-23T16:25:14.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2794415d:::benchmark_data_smithi188_89276_object124033:head# 2024-04-23T16:25:14.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2794a75a:::benchmark_data_smithi188_89276_object88882:head# 2024-04-23T16:25:14.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2794d3e9:::benchmark_data_smithi188_89276_object76535:head# 2024-04-23T16:25:14.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2795a6f9:::benchmark_data_smithi188_89276_object54484:head# 2024-04-23T16:25:14.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27960d74:::benchmark_data_smithi188_89276_object44690:head# 2024-04-23T16:25:14.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27964d86:::benchmark_data_smithi188_89276_object44933:head# 2024-04-23T16:25:14.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27967f97:::benchmark_data_smithi188_89276_object89233:head# 2024-04-23T16:25:14.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2796efb0:::benchmark_data_smithi188_89276_object97280:head# 2024-04-23T16:25:14.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:279703a5:::benchmark_data_smithi188_89276_object58476:head# 2024-04-23T16:25:14.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27977cf2:::benchmark_data_smithi188_89276_object153772:head# 2024-04-23T16:25:14.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2797aa2d:::benchmark_data_smithi188_89276_object60413:head# 2024-04-23T16:25:14.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27981f8e:::benchmark_data_smithi188_89276_object102090:head# 2024-04-23T16:25:14.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:279827f2:::benchmark_data_smithi188_89276_object126630:head# 2024-04-23T16:25:14.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2799410d:::benchmark_data_smithi188_89276_object102626:head# 2024-04-23T16:25:14.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:279aaef4:::benchmark_data_smithi188_89276_object48643:head# 2024-04-23T16:25:14.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:279ade89:::benchmark_data_smithi188_89276_object72401:head# 2024-04-23T16:25:14.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:279d5cdb:::benchmark_data_smithi188_89276_object116056:head# 2024-04-23T16:25:14.326 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:279e858a:::benchmark_data_smithi188_89276_object49495:head# 2024-04-23T16:25:14.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:279ea117:::benchmark_data_smithi188_89276_object151548:head# 2024-04-23T16:25:14.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:279ec7c7:::benchmark_data_smithi188_89276_object79866:head# 2024-04-23T16:25:14.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:279ee4f2:::benchmark_data_smithi188_89276_object125901:head# 2024-04-23T16:25:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:279f4b6a:::benchmark_data_smithi188_89276_object143615:head# 2024-04-23T16:25:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27a017b4:::benchmark_data_smithi188_89276_object134194:head# 2024-04-23T16:25:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27a0bee1:::benchmark_data_smithi188_89276_object66586:head# 2024-04-23T16:25:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27a1476b:::benchmark_data_smithi188_89276_object106060:head# 2024-04-23T16:25:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27a2ba0e:::benchmark_data_smithi188_89276_object105681:head# 2024-04-23T16:25:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27a2c523:::benchmark_data_smithi188_89276_object103366:head# 2024-04-23T16:25:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27a3b7ab:::benchmark_data_smithi188_89276_object63082:head# 2024-04-23T16:25:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27a412fa:::benchmark_data_smithi188_89276_object151326:head# 2024-04-23T16:25:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27a4d7b1:::benchmark_data_smithi188_89276_object51310:head# 2024-04-23T16:25:14.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27a51680:::benchmark_data_smithi188_89276_object145800:head# 2024-04-23T16:25:14.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27a5311b:::benchmark_data_smithi188_89276_object118990:head# 2024-04-23T16:25:14.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27a57a0a:::benchmark_data_smithi188_89276_object167281:head# 2024-04-23T16:25:14.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27a6cb09:::benchmark_data_smithi188_89276_object94919:head# 2024-04-23T16:25:14.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27a8726c:::benchmark_data_smithi188_89276_object85794:head# 2024-04-23T16:25:14.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27a90fa5:::benchmark_data_smithi188_89276_object61010:head# 2024-04-23T16:25:14.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27a94063:::benchmark_data_smithi188_89276_object131119:head# 2024-04-23T16:25:14.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27a9dfe8:::benchmark_data_smithi188_89276_object107626:head# 2024-04-23T16:25:14.337 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ab19fc:::benchmark_data_smithi188_89276_object72221:head# 2024-04-23T16:25:14.337 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ab6c3a:::benchmark_data_smithi188_89276_object63573:head# 2024-04-23T16:25:14.337 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ab97ec:::benchmark_data_smithi188_89276_object111795:head# 2024-04-23T16:25:14.338 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ac4b38:::benchmark_data_smithi188_89276_object98817:head# 2024-04-23T16:25:14.338 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ac6e46:::benchmark_data_smithi188_89276_object61589:head# 2024-04-23T16:25:14.338 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ad0632:::benchmark_data_smithi188_89276_object102574:head# 2024-04-23T16:25:14.338 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27addc1f:::benchmark_data_smithi188_89276_object49101:head# 2024-04-23T16:25:14.338 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27adfcdf:::benchmark_data_smithi188_89276_object42849:head# 2024-04-23T16:25:14.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ae0cbc:::benchmark_data_smithi188_89276_object48559:head# 2024-04-23T16:25:14.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27afe982:::benchmark_data_smithi188_89276_object160059:head# 2024-04-23T16:25:14.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27affa06:::benchmark_data_smithi188_89276_object135099:head# 2024-04-23T16:25:14.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b00d1d:::benchmark_data_smithi188_89276_object82369:head# 2024-04-23T16:25:14.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b03625:::benchmark_data_smithi188_89276_object160181:head# 2024-04-23T16:25:14.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b040e1:::benchmark_data_smithi188_89276_object93722:head# 2024-04-23T16:25:14.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b08a76:::benchmark_data_smithi188_89276_object163583:head# 2024-04-23T16:25:14.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b09131:::benchmark_data_smithi188_89276_object168047:head# 2024-04-23T16:25:14.348 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b09ddb:::benchmark_data_smithi188_89276_object77243:head# 2024-04-23T16:25:14.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b14296:::benchmark_data_smithi188_89276_object42225:head# 2024-04-23T16:25:14.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b16829:::benchmark_data_smithi188_89276_object59566:head# 2024-04-23T16:25:14.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b27a86:::benchmark_data_smithi188_89276_object103920:head# 2024-04-23T16:25:14.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b2bcc2:::benchmark_data_smithi188_89276_object77984:head# 2024-04-23T16:25:14.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b2e846:::benchmark_data_smithi188_89276_object82904:head# 2024-04-23T16:25:14.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b2f80c:::benchmark_data_smithi188_89276_object103525:head# 2024-04-23T16:25:14.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b329a3:::benchmark_data_smithi188_89276_object156358:head# 2024-04-23T16:25:14.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b46676:::benchmark_data_smithi188_89276_object66874:head# 2024-04-23T16:25:14.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b49386:::benchmark_data_smithi188_89276_object67701:head# 2024-04-23T16:25:14.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b4a9c2:::benchmark_data_smithi188_89276_object112041:head# 2024-04-23T16:25:14.353 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b4fe0c:::benchmark_data_smithi188_89276_object163404:head# 2024-04-23T16:25:14.353 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b53498:::benchmark_data_smithi188_89276_object81387:head# 2024-04-23T16:25:14.353 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b6335c:::benchmark_data_smithi188_89276_object143390:head# 2024-04-23T16:25:14.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b6bd82:::benchmark_data_smithi188_89276_object161847:head# 2024-04-23T16:25:14.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b75bdc:::benchmark_data_smithi188_89276_object105076:head# 2024-04-23T16:25:14.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b78817:::benchmark_data_smithi188_89276_object103694:head# 2024-04-23T16:25:14.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b93115:::benchmark_data_smithi188_89276_object123306:head# 2024-04-23T16:25:14.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27b9372d:::benchmark_data_smithi188_89276_object150726:head# 2024-04-23T16:25:14.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ba3082:::benchmark_data_smithi188_89276_object44744:head# 2024-04-23T16:25:14.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ba7930:::benchmark_data_smithi188_89276_object134184:head# 2024-04-23T16:25:14.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ba8b19:::benchmark_data_smithi188_89276_object49597:head# 2024-04-23T16:25:14.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27bade21:::benchmark_data_smithi188_89276_object64852:head# 2024-04-23T16:25:14.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27bb26f5:::benchmark_data_smithi188_89276_object113818:head# 2024-04-23T16:25:14.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27bb3b45:::benchmark_data_smithi188_89276_object167429:head# 2024-04-23T16:25:14.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27bb5c46:::benchmark_data_smithi188_89276_object114548:head# 2024-04-23T16:25:14.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27bb77ff:::benchmark_data_smithi188_89276_object149847:head# 2024-04-23T16:25:14.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27bbaa03:::benchmark_data_smithi188_89276_object86179:head# 2024-04-23T16:25:14.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27bbb00b:::benchmark_data_smithi188_89276_object122127:head# 2024-04-23T16:25:14.361 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27bbc78b:::benchmark_data_smithi188_89276_object61990:head# 2024-04-23T16:25:14.361 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27bcedc8:::benchmark_data_smithi188_89276_object45802:head# 2024-04-23T16:25:14.361 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27bd75ae:::benchmark_data_smithi188_89276_object75947:head# 2024-04-23T16:25:14.362 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27bd993d:::benchmark_data_smithi188_89276_object159333:head# 2024-04-23T16:25:14.362 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27be1957:::benchmark_data_smithi188_89276_object106562:head# 2024-04-23T16:25:14.363 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27bfe168:::benchmark_data_smithi188_89276_object149264:head# 2024-04-23T16:25:14.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c010f5:::benchmark_data_smithi188_89276_object145891:head# 2024-04-23T16:25:14.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c04cd2:::benchmark_data_smithi188_89276_object129381:head# 2024-04-23T16:25:14.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c1b6d8:::benchmark_data_smithi188_89276_object54007:head# 2024-04-23T16:25:14.365 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:14.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c2248e:::benchmark_data_smithi188_89276_object145609:head# 2024-04-23T16:25:14.366 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c30a5b:::benchmark_data_smithi188_89276_object71142:head# 2024-04-23T16:25:14.367 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c3e1c3:::benchmark_data_smithi188_89276_object139308:head# 2024-04-23T16:25:14.367 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c3eec4:::benchmark_data_smithi188_89276_object89733:head# 2024-04-23T16:25:14.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c437d8:::benchmark_data_smithi188_89276_object80129:head# 2024-04-23T16:25:14.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c44f32:::benchmark_data_smithi188_89276_object124459:head# 2024-04-23T16:25:14.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c47573:::benchmark_data_smithi188_89276_object100263:head# 2024-04-23T16:25:14.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c4b95a:::benchmark_data_smithi188_89276_object89254:head# 2024-04-23T16:25:14.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c700fd:::benchmark_data_smithi188_89276_object67341:head# 2024-04-23T16:25:14.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c7544c:::benchmark_data_smithi188_89276_object101376:head# 2024-04-23T16:25:14.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c7ce5c:::benchmark_data_smithi188_89276_object43287:head# 2024-04-23T16:25:14.371 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c7d33d:::benchmark_data_smithi188_89276_object101673:head# 2024-04-23T16:25:14.371 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c831e5:::benchmark_data_smithi188_89276_object119486:head# 2024-04-23T16:25:14.373 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c8c9a4:::benchmark_data_smithi188_89276_object48068:head# 2024-04-23T16:25:14.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c91cf4:::benchmark_data_smithi188_89276_object148964:head# 2024-04-23T16:25:14.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c95af5:::benchmark_data_smithi188_89276_object77737:head# 2024-04-23T16:25:14.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c9859c:::benchmark_data_smithi188_89276_object88048:head# 2024-04-23T16:25:14.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27c995fe:::benchmark_data_smithi188_89276_object115193:head# 2024-04-23T16:25:14.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ca1fd9:::benchmark_data_smithi188_89276_object50251:head# 2024-04-23T16:25:14.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ca6040:::benchmark_data_smithi188_89276_object161162:head# 2024-04-23T16:25:14.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27cb1cf1:::benchmark_data_smithi188_89276_object168482:head# 2024-04-23T16:25:14.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27cdbc68:::benchmark_data_smithi188_89276_object51636:head# 2024-04-23T16:25:14.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27cdd3fd:::benchmark_data_smithi188_89276_object104170:head# 2024-04-23T16:25:14.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27cf19a7:::benchmark_data_smithi188_89276_object63037:head# 2024-04-23T16:25:14.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27cf9e93:::benchmark_data_smithi188_89276_object102174:head# 2024-04-23T16:25:14.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27cfb19e:::benchmark_data_smithi188_89276_object140923:head# 2024-04-23T16:25:14.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27d0b347:::benchmark_data_smithi188_89276_object50186:head# 2024-04-23T16:25:14.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27d0bee6:::benchmark_data_smithi188_89276_object94757:head# 2024-04-23T16:25:14.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27d1db69:::benchmark_data_smithi188_89276_object48918:head# 2024-04-23T16:25:14.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27d1f676:::benchmark_data_smithi188_89276_object137241:head# 2024-04-23T16:25:14.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27d294d6:::benchmark_data_smithi188_89276_object92268:head# 2024-04-23T16:25:14.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27d3ed2f:::benchmark_data_smithi188_89276_object105224:head# 2024-04-23T16:25:14.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27d406bc:::benchmark_data_smithi188_89276_object110241:head# 2024-04-23T16:25:14.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27d4eba6:::benchmark_data_smithi188_89276_object72348:head# 2024-04-23T16:25:14.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27d59d86:::benchmark_data_smithi188_89276_object123367:head# 2024-04-23T16:25:14.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27d5d49e:::benchmark_data_smithi188_89276_object146054:head# 2024-04-23T16:25:14.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27d63d0f:::benchmark_data_smithi188_89276_object114598:head# 2024-04-23T16:25:14.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27d67ff6:::benchmark_data_smithi188_89276_object133655:head# 2024-04-23T16:25:14.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27d70e03:::benchmark_data_smithi188_89276_object69065:head# 2024-04-23T16:25:14.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27d78de4:::benchmark_data_smithi188_89276_object117887:head# 2024-04-23T16:25:14.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27d8b257:::benchmark_data_smithi188_89276_object139631:head# 2024-04-23T16:25:14.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27d94c97:::benchmark_data_smithi188_89276_object107374:head# 2024-04-23T16:25:14.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27d97dc3:::benchmark_data_smithi188_89276_object103789:head# 2024-04-23T16:25:14.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27d9ceaa:::benchmark_data_smithi188_89276_object134815:head# 2024-04-23T16:25:14.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27d9ddc8:::benchmark_data_smithi188_89276_object91406:head# 2024-04-23T16:25:14.387 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27da496a:::benchmark_data_smithi188_89276_object82323:head# 2024-04-23T16:25:14.387 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27db7f9e:::benchmark_data_smithi188_89276_object154337:head# 2024-04-23T16:25:14.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27dbd1c5:::benchmark_data_smithi188_89276_object79272:head# 2024-04-23T16:25:14.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27dc1dcc:::benchmark_data_smithi188_89276_object76971:head# 2024-04-23T16:25:14.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27dcc1fd:::benchmark_data_smithi188_89276_object153632:head# 2024-04-23T16:25:14.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27dd4091:::benchmark_data_smithi188_89276_object109777:head# 2024-04-23T16:25:14.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27dda83e:::benchmark_data_smithi188_89276_object168083:head# 2024-04-23T16:25:14.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27dec108:::benchmark_data_smithi188_89276_object66666:head# 2024-04-23T16:25:14.395 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27df0bbd:::benchmark_data_smithi188_89276_object107559:head# 2024-04-23T16:25:14.395 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27df3769:::benchmark_data_smithi188_89276_object55512:head# 2024-04-23T16:25:14.395 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27df5817:::benchmark_data_smithi188_89276_object45630:head# 2024-04-23T16:25:14.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27df66f8:::benchmark_data_smithi188_89276_object135945:head# 2024-04-23T16:25:14.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27dfd196:::benchmark_data_smithi188_89276_object139559:head# 2024-04-23T16:25:14.397 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27e0097a:::benchmark_data_smithi188_89276_object41593:head# 2024-04-23T16:25:14.397 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27e038f8:::benchmark_data_smithi188_89276_object43040:head# 2024-04-23T16:25:14.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27e09775:::benchmark_data_smithi188_89276_object104741:head# 2024-04-23T16:25:14.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27e0e159:::benchmark_data_smithi188_89276_object163115:head# 2024-04-23T16:25:14.399 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27e0eb36:::benchmark_data_smithi188_89276_object55405:head# 2024-04-23T16:25:14.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27e25b46:::benchmark_data_smithi188_89276_object56479:head# 2024-04-23T16:25:14.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27e2c2de:::benchmark_data_smithi188_89276_object55151:head# 2024-04-23T16:25:14.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27e39a07:::benchmark_data_smithi188_89276_object142364:head# 2024-04-23T16:25:14.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27e3ce2a:::benchmark_data_smithi188_89276_object120369:head# 2024-04-23T16:25:14.403 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:14.403 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27e424b4:::benchmark_data_smithi188_89276_object74305:head# 2024-04-23T16:25:14.403 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27e5fe20:::benchmark_data_smithi188_89276_object159706:head# 2024-04-23T16:25:14.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27e622b5:::benchmark_data_smithi188_89276_object150314:head# 2024-04-23T16:25:14.405 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27e6b152:::benchmark_data_smithi188_89276_object140960:head# 2024-04-23T16:25:14.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27e75c30:::benchmark_data_smithi188_89276_object147106:head# 2024-04-23T16:25:14.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27e7b52c:::benchmark_data_smithi188_89276_object137453:head# 2024-04-23T16:25:14.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27e94d11:::benchmark_data_smithi188_89276_object82901:head# 2024-04-23T16:25:14.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ea9758:::benchmark_data_smithi188_89276_object116199:head# 2024-04-23T16:25:14.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27eb1bd1:::benchmark_data_smithi188_89276_object155767:head# 2024-04-23T16:25:14.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27eb419e:::benchmark_data_smithi188_89276_object93827:head# 2024-04-23T16:25:14.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ec1e57:::benchmark_data_smithi188_89276_object140917:head# 2024-04-23T16:25:14.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ec2a84:::benchmark_data_smithi188_89276_object124986:head# 2024-04-23T16:25:14.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ecdd8c:::benchmark_data_smithi188_89276_object49502:head# 2024-04-23T16:25:14.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ed1069:::benchmark_data_smithi188_89276_object94511:head# 2024-04-23T16:25:14.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ed4007:::benchmark_data_smithi188_89276_object117855:head# 2024-04-23T16:25:14.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ed6448:::benchmark_data_smithi188_89276_object46265:head# 2024-04-23T16:25:14.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ed66cc:::benchmark_data_smithi188_89276_object66683:head# 2024-04-23T16:25:14.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ed7cf7:::benchmark_data_smithi188_89276_object156965:head# 2024-04-23T16:25:14.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27eeb19f:::benchmark_data_smithi188_89276_object136664:head# 2024-04-23T16:25:14.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ef620d:::benchmark_data_smithi188_89276_object156835:head# 2024-04-23T16:25:14.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ef6e2b:::benchmark_data_smithi188_89276_object141764:head# 2024-04-23T16:25:14.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27efa2cd:::benchmark_data_smithi188_89276_object118901:head# 2024-04-23T16:25:14.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f06157:::benchmark_data_smithi188_89276_object145600:head# 2024-04-23T16:25:14.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f19b40:::benchmark_data_smithi188_89276_object72472:head# 2024-04-23T16:25:14.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f1d3de:::benchmark_data_smithi188_89276_object119599:head# 2024-04-23T16:25:14.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f21643:::benchmark_data_smithi188_89276_object41163:head# 2024-04-23T16:25:14.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f270ef:::benchmark_data_smithi188_89276_object138233:head# 2024-04-23T16:25:14.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f2a71c:::benchmark_data_smithi188_89276_object64745:head# 2024-04-23T16:25:14.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f2df37:::benchmark_data_smithi188_89276_object48530:head# 2024-04-23T16:25:14.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f42f4d:::benchmark_data_smithi188_89276_object101268:head# 2024-04-23T16:25:14.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f44439:::benchmark_data_smithi188_89276_object56559:head# 2024-04-23T16:25:14.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f4b104:::benchmark_data_smithi188_89276_object63839:head# 2024-04-23T16:25:14.431 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f599a6:::benchmark_data_smithi188_89276_object73734:head# 2024-04-23T16:25:14.431 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f634ed:::benchmark_data_smithi188_89276_object144827:head# 2024-04-23T16:25:14.431 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f635e5:::benchmark_data_smithi188_89276_object110555:head# 2024-04-23T16:25:14.432 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f63c8d:::benchmark_data_smithi188_89276_object71726:head# 2024-04-23T16:25:14.432 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f762c0:::benchmark_data_smithi188_89276_object47685:head# 2024-04-23T16:25:14.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f78472:::benchmark_data_smithi188_89276_object72451:head# 2024-04-23T16:25:14.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f811e4:::benchmark_data_smithi188_89276_object65715:head# 2024-04-23T16:25:14.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f85dfc:::benchmark_data_smithi188_89276_object122307:head# 2024-04-23T16:25:14.434 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f8af40:::benchmark_data_smithi188_89276_object75952:head# 2024-04-23T16:25:14.434 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f96b9d:::benchmark_data_smithi188_89276_object162702:head# 2024-04-23T16:25:14.435 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f9c1dc:::benchmark_data_smithi188_89276_object70939:head# 2024-04-23T16:25:14.435 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27f9f1ab:::benchmark_data_smithi188_89276_object59041:head# 2024-04-23T16:25:14.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27fa1ca8:::benchmark_data_smithi188_89276_object142992:head# 2024-04-23T16:25:14.438 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27fa5df9:::benchmark_data_smithi188_89276_object157405:head# 2024-04-23T16:25:14.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27fa81b1:::benchmark_data_smithi188_89276_object63682:head# 2024-04-23T16:25:14.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27fae407:::benchmark_data_smithi188_89276_object129828:head# 2024-04-23T16:25:14.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27fc373b:::benchmark_data_smithi188_89276_object116000:head# 2024-04-23T16:25:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27fe5188:::benchmark_data_smithi188_89276_object105530:head# 2024-04-23T16:25:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27fe8303:::benchmark_data_smithi188_89276_object121672:head# 2024-04-23T16:25:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27feb363:::benchmark_data_smithi188_89276_object118764:head# 2024-04-23T16:25:14.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27feddb1:::benchmark_data_smithi188_89276_object79583:head# 2024-04-23T16:25:14.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:27ff24e3:::benchmark_data_smithi188_89276_object133963:head# 2024-04-23T16:25:14.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:280015a6:::benchmark_data_smithi188_89276_object72304:head# 2024-04-23T16:25:14.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2800a90c:::benchmark_data_smithi188_89276_object97905:head# 2024-04-23T16:25:14.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2801906e:::benchmark_data_smithi188_89276_object161582:head# 2024-04-23T16:25:14.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28019363:::benchmark_data_smithi188_89276_object119093:head# 2024-04-23T16:25:14.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2801f196:::benchmark_data_smithi188_89276_object117513:head# 2024-04-23T16:25:14.446 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28054dab:::benchmark_data_smithi188_89276_object129437:head# 2024-04-23T16:25:14.446 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2805713a:::benchmark_data_smithi188_89276_object81918:head# 2024-04-23T16:25:14.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2805e180:::benchmark_data_smithi188_89276_object96081:head# 2024-04-23T16:25:14.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2805e9cc:::benchmark_data_smithi188_89276_object163247:head# 2024-04-23T16:25:14.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2805ff84:::benchmark_data_smithi188_89276_object115467:head# 2024-04-23T16:25:14.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28066bbf:::benchmark_data_smithi188_89276_object140469:head# 2024-04-23T16:25:14.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2806953a:::benchmark_data_smithi188_89276_object65486:head# 2024-04-23T16:25:14.451 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2806980b:::benchmark_data_smithi188_89276_object109162:head# 2024-04-23T16:25:14.451 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28074aa4:::benchmark_data_smithi188_89276_object104612:head# 2024-04-23T16:25:14.454 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:280763a1:::benchmark_data_smithi188_89276_object146158:head# 2024-04-23T16:25:14.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:280764b7:::benchmark_data_smithi188_89276_object45898:head# 2024-04-23T16:25:14.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:280774f7:::benchmark_data_smithi188_89276_object72288:head# 2024-04-23T16:25:14.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28077586:::benchmark_data_smithi188_89276_object86669:head# 2024-04-23T16:25:14.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28088ac5:::benchmark_data_smithi188_89276_object137183:head# 2024-04-23T16:25:14.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28090522:::benchmark_data_smithi188_89276_object137328:head# 2024-04-23T16:25:14.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2809d180:::benchmark_data_smithi188_89276_object117239:head# 2024-04-23T16:25:14.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2809ea1f:::benchmark_data_smithi188_89276_object161679:head# 2024-04-23T16:25:14.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:280a6786:::benchmark_data_smithi188_89276_object88530:head# 2024-04-23T16:25:14.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:280a8ae3:::benchmark_data_smithi188_89276_object128235:head# 2024-04-23T16:25:14.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:280b374d:::benchmark_data_smithi188_89276_object80779:head# 2024-04-23T16:25:14.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:280b9040:::benchmark_data_smithi188_89276_object132836:head# 2024-04-23T16:25:14.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:280bd725:::benchmark_data_smithi188_89276_object166082:head# 2024-04-23T16:25:14.460 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:280cac3b:::benchmark_data_smithi188_89276_object132622:head# 2024-04-23T16:25:14.460 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:280cc4f1:::benchmark_data_smithi188_89276_object107470:head# 2024-04-23T16:25:14.461 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:280d3fc6:::benchmark_data_smithi188_89276_object101867:head# 2024-04-23T16:25:14.461 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:280d5d0b:::benchmark_data_smithi188_89276_object62921:head# 2024-04-23T16:25:14.462 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:280dc3f2:::benchmark_data_smithi188_89276_object85508:head# 2024-04-23T16:25:14.462 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:280e6ee9:::benchmark_data_smithi188_89276_object124531:head# 2024-04-23T16:25:14.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:280eace5:::benchmark_data_smithi188_89276_object162258:head# 2024-04-23T16:25:14.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:280ed85c:::benchmark_data_smithi188_89276_object104829:head# 2024-04-23T16:25:14.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:280effd1:::benchmark_data_smithi188_89276_object58795:head# 2024-04-23T16:25:14.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:280f57c6:::benchmark_data_smithi188_89276_object54009:head# 2024-04-23T16:25:14.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28102c08:::benchmark_data_smithi188_89276_object59974:head# 2024-04-23T16:25:14.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2810348a:::benchmark_data_smithi188_89276_object120631:head# 2024-04-23T16:25:14.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28103941:::benchmark_data_smithi188_89276_object103163:head# 2024-04-23T16:25:14.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2810452c:::benchmark_data_smithi188_89276_object133058:head# 2024-04-23T16:25:14.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28110f58:::benchmark_data_smithi188_89276_object137399:head# 2024-04-23T16:25:14.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28118d61:::benchmark_data_smithi188_89276_object130094:head# 2024-04-23T16:25:14.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28125d3a:::benchmark_data_smithi188_89276_object111557:head# 2024-04-23T16:25:14.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2812f8d2:::benchmark_data_smithi188_89276_object119703:head# 2024-04-23T16:25:14.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28151d71:::benchmark_data_smithi188_89276_object124131:head# 2024-04-23T16:25:14.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28165c56:::benchmark_data_smithi188_89276_object165069:head# 2024-04-23T16:25:14.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2816efa6:::benchmark_data_smithi188_89276_object160777:head# 2024-04-23T16:25:14.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2817c75a:::benchmark_data_smithi188_89276_object87023:head# 2024-04-23T16:25:14.470 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2818b63b:::benchmark_data_smithi188_89276_object126276:head# 2024-04-23T16:25:14.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28194d43:::benchmark_data_smithi188_89276_object135332:head# 2024-04-23T16:25:14.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2819970a:::benchmark_data_smithi188_89276_object157849:head# 2024-04-23T16:25:14.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:281a4467:::benchmark_data_smithi188_89276_object78490:head# 2024-04-23T16:25:14.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:281a621d:::benchmark_data_smithi188_89276_object112765:head# 2024-04-23T16:25:14.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:281a81f1:::benchmark_data_smithi188_89276_object141299:head# 2024-04-23T16:25:14.473 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:281c1768:::benchmark_data_smithi188_89276_object93158:head# 2024-04-23T16:25:14.473 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:281c65d1:::benchmark_data_smithi188_89276_object151477:head# 2024-04-23T16:25:14.474 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:281c79fd:::benchmark_data_smithi188_89276_object49504:head# 2024-04-23T16:25:14.474 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:281c9481:::benchmark_data_smithi188_89276_object64613:head# 2024-04-23T16:25:14.474 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:281ca78b:::benchmark_data_smithi188_89276_object133837:head# 2024-04-23T16:25:14.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:281cb04b:::benchmark_data_smithi188_89276_object153528:head# 2024-04-23T16:25:14.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:281d6bf8:::benchmark_data_smithi188_89276_object79955:head# 2024-04-23T16:25:14.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:281dcc1d:::benchmark_data_smithi188_89276_object97527:head# 2024-04-23T16:25:14.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:281f9749:::benchmark_data_smithi188_89276_object43701:head# 2024-04-23T16:25:14.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:281fb7a1:::benchmark_data_smithi188_89276_object86433:head# 2024-04-23T16:25:14.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28204422:::benchmark_data_smithi188_89276_object75536:head# 2024-04-23T16:25:14.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2820448c:::benchmark_data_smithi188_89276_object168246:head# 2024-04-23T16:25:14.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:282055a3:::benchmark_data_smithi188_89276_object92050:head# 2024-04-23T16:25:14.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:282069c4:::benchmark_data_smithi188_89276_object46658:head# 2024-04-23T16:25:14.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2820883e:::benchmark_data_smithi188_89276_object134084:head# 2024-04-23T16:25:14.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2821a783:::benchmark_data_smithi188_89276_object114310:head# 2024-04-23T16:25:14.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2821ed60:::benchmark_data_smithi188_89276_object105529:head# 2024-04-23T16:25:14.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2822161c:::benchmark_data_smithi188_89276_object139991:head# 2024-04-23T16:25:14.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28227868:::benchmark_data_smithi188_89276_object138981:head# 2024-04-23T16:25:14.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2823ac7a:::benchmark_data_smithi188_89276_object82011:head# 2024-04-23T16:25:14.481 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2823c428:::benchmark_data_smithi188_89276_object57018:head# 2024-04-23T16:25:14.481 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28241251:::benchmark_data_smithi188_89276_object129051:head# 2024-04-23T16:25:14.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28250f5a:::benchmark_data_smithi188_89276_object96284:head# 2024-04-23T16:25:14.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2825399d:::benchmark_data_smithi188_89276_object50394:head# 2024-04-23T16:25:14.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2825aee3:::benchmark_data_smithi188_89276_object116361:head# 2024-04-23T16:25:14.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2827aedc:::benchmark_data_smithi188_89276_object167076:head# 2024-04-23T16:25:14.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28283f7c:::benchmark_data_smithi188_89276_object50912:head# 2024-04-23T16:25:14.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2828938f:::benchmark_data_smithi188_89276_object71832:head# 2024-04-23T16:25:14.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2828a472:::benchmark_data_smithi188_89276_object69258:head# 2024-04-23T16:25:14.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2828c75c:::benchmark_data_smithi188_89276_object155939:head# 2024-04-23T16:25:14.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:282a941d:::benchmark_data_smithi188_89276_object150859:head# 2024-04-23T16:25:14.486 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:282a9b24:::benchmark_data_smithi188_89276_object145988:head# 2024-04-23T16:25:14.486 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:282b2801:::benchmark_data_smithi188_89276_object104479:head# 2024-04-23T16:25:14.488 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:282ba886:::benchmark_data_smithi188_89276_object44879:head# 2024-04-23T16:25:14.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:282c0168:::benchmark_data_smithi188_89276_object148362:head# 2024-04-23T16:25:14.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:282c9ada:::benchmark_data_smithi188_89276_object149701:head# 2024-04-23T16:25:14.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:282d86a4:::benchmark_data_smithi188_89276_object49793:head# 2024-04-23T16:25:14.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:282e0dc4:::benchmark_data_smithi188_89276_object66026:head# 2024-04-23T16:25:14.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:282e1291:::benchmark_data_smithi188_89276_object167885:head# 2024-04-23T16:25:14.491 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:282ee89f:::benchmark_data_smithi188_89276_object110971:head# 2024-04-23T16:25:14.491 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28306afa:::benchmark_data_smithi188_89276_object140397:head# 2024-04-23T16:25:14.492 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2830e14e:::benchmark_data_smithi188_89276_object51681:head# 2024-04-23T16:25:14.493 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28312f09:::benchmark_data_smithi188_89276_object75658:head# 2024-04-23T16:25:14.493 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2831be28:::benchmark_data_smithi188_89276_object50807:head# 2024-04-23T16:25:14.494 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28333113:::benchmark_data_smithi188_89276_object147380:head# 2024-04-23T16:25:14.494 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:283347a0:::benchmark_data_smithi188_89276_object78938:head# 2024-04-23T16:25:14.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28337d28:::benchmark_data_smithi188_89276_object127198:head# 2024-04-23T16:25:14.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28356760:::benchmark_data_smithi188_89276_object143267:head# 2024-04-23T16:25:14.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28368996:::benchmark_data_smithi188_89276_object45677:head# 2024-04-23T16:25:14.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28379e2c:::benchmark_data_smithi188_89276_object42681:head# 2024-04-23T16:25:14.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2837f3a3:::benchmark_data_smithi188_89276_object110263:head# 2024-04-23T16:25:14.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28380437:::benchmark_data_smithi188_89276_object82607:head# 2024-04-23T16:25:14.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28385935:::benchmark_data_smithi188_89276_object74850:head# 2024-04-23T16:25:14.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2838d4ea:::benchmark_data_smithi188_89276_object141279:head# 2024-04-23T16:25:14.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2838e39e:::benchmark_data_smithi188_89276_object144428:head# 2024-04-23T16:25:14.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:283973c3:::benchmark_data_smithi188_89276_object107674:head# 2024-04-23T16:25:14.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2839cc89:::benchmark_data_smithi188_89276_object111409:head# 2024-04-23T16:25:14.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:283b3b9d:::benchmark_data_smithi188_89276_object101372:head# 2024-04-23T16:25:14.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:283b6436:::benchmark_data_smithi188_89276_object125174:head# 2024-04-23T16:25:14.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:283c6c26:::benchmark_data_smithi188_89276_object113211:head# 2024-04-23T16:25:14.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:283cb358:::benchmark_data_smithi188_89276_object123996:head# 2024-04-23T16:25:14.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:283ce86f:::benchmark_data_smithi188_89276_object150447:head# 2024-04-23T16:25:14.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:283ce9ef:::benchmark_data_smithi188_89276_object100168:head# 2024-04-23T16:25:14.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:283d9a67:::benchmark_data_smithi188_89276_object123867:head# 2024-04-23T16:25:14.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:283da2f3:::benchmark_data_smithi188_89276_object85142:head# 2024-04-23T16:25:14.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:283df143:::benchmark_data_smithi188_89276_object97595:head# 2024-04-23T16:25:14.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:283e60ad:::benchmark_data_smithi188_89276_object135619:head# 2024-04-23T16:25:14.504 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:283ec6ff:::benchmark_data_smithi188_89276_object79255:head# 2024-04-23T16:25:14.504 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:283fb9d5:::benchmark_data_smithi188_89276_object134812:head# 2024-04-23T16:25:14.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28401aab:::benchmark_data_smithi188_89276_object74125:head# 2024-04-23T16:25:14.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2841234b:::benchmark_data_smithi188_89276_object95549:head# 2024-04-23T16:25:14.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2841d577:::benchmark_data_smithi188_89276_object125932:head# 2024-04-23T16:25:14.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28443f7e:::benchmark_data_smithi188_89276_object134198:head# 2024-04-23T16:25:14.507 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:284440a3:::benchmark_data_smithi188_89276_object45151:head# 2024-04-23T16:25:14.508 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:284504a1:::benchmark_data_smithi188_89276_object117336:head# 2024-04-23T16:25:14.508 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28453712:::benchmark_data_smithi188_89276_object118350:head# 2024-04-23T16:25:14.509 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28455281:::benchmark_data_smithi188_89276_object150861:head# 2024-04-23T16:25:14.509 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2845c129:::benchmark_data_smithi188_89276_object54042:head# 2024-04-23T16:25:14.509 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28460aec:::benchmark_data_smithi188_89276_object114341:head# 2024-04-23T16:25:14.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28460d30:::benchmark_data_smithi188_89276_object67263:head# 2024-04-23T16:25:14.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28464a8a:::benchmark_data_smithi188_89276_object161076:head# 2024-04-23T16:25:14.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2846d32a:::benchmark_data_smithi188_89276_object154836:head# 2024-04-23T16:25:14.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2846f2bc:::benchmark_data_smithi188_89276_object84010:head# 2024-04-23T16:25:14.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28477a61:::benchmark_data_smithi188_89276_object160904:head# 2024-04-23T16:25:14.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2848920f:::benchmark_data_smithi188_89276_object57316:head# 2024-04-23T16:25:14.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2848d75d:::benchmark_data_smithi188_89276_object115138:head# 2024-04-23T16:25:14.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28494da4:::benchmark_data_smithi188_89276_object102360:head# 2024-04-23T16:25:14.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:284969cf:::benchmark_data_smithi188_89276_object94321:head# 2024-04-23T16:25:14.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28496d01:::benchmark_data_smithi188_89276_object135296:head# 2024-04-23T16:25:14.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28498a06:::benchmark_data_smithi188_89276_object75355:head# 2024-04-23T16:25:14.516 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:284a08b0:::benchmark_data_smithi188_89276_object158101:head# 2024-04-23T16:25:14.516 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:284a47f8:::benchmark_data_smithi188_89276_object128543:head# 2024-04-23T16:25:14.516 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:284b25a2:::benchmark_data_smithi188_89276_object46354:head# 2024-04-23T16:25:14.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:284beb4c:::benchmark_data_smithi188_89276_object140594:head# 2024-04-23T16:25:14.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:284c48a9:::benchmark_data_smithi188_89276_object66023:head# 2024-04-23T16:25:14.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:284cfdf8:::benchmark_data_smithi188_89276_object54101:head# 2024-04-23T16:25:14.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:284d9b24:::benchmark_data_smithi188_89276_object109026:head# 2024-04-23T16:25:14.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:284fbe96:::benchmark_data_smithi188_89276_object164717:head# 2024-04-23T16:25:14.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:284fc4ac:::benchmark_data_smithi188_89276_object47317:head# 2024-04-23T16:25:14.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285042d5:::benchmark_data_smithi188_89276_object85899:head# 2024-04-23T16:25:14.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28504c7b:::benchmark_data_smithi188_89276_object94620:head# 2024-04-23T16:25:14.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285055d3:::benchmark_data_smithi188_89276_object121044:head# 2024-04-23T16:25:14.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2850c671:::benchmark_data_smithi188_89276_object130738:head# 2024-04-23T16:25:14.521 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285118b0:::benchmark_data_smithi188_89276_object122111:head# 2024-04-23T16:25:14.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28511f0a:::benchmark_data_smithi188_89276_object90429:head# 2024-04-23T16:25:14.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285378ef:::benchmark_data_smithi188_89276_object80125:head# 2024-04-23T16:25:14.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28542357:::benchmark_data_smithi188_89276_object40680:head# 2024-04-23T16:25:14.523 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285427d4:::benchmark_data_smithi188_89276_object112923:head# 2024-04-23T16:25:14.523 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2854b0ff:::benchmark_data_smithi188_89276_object43561:head# 2024-04-23T16:25:14.523 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2854ccc2:::benchmark_data_smithi188_89276_object124216:head# 2024-04-23T16:25:14.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2854e0d4:::benchmark_data_smithi188_89276_object149690:head# 2024-04-23T16:25:14.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28558b58:::benchmark_data_smithi188_89276_object145814:head# 2024-04-23T16:25:14.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2857dc75:::benchmark_data_smithi188_89276_object140183:head# 2024-04-23T16:25:14.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285873d6:::benchmark_data_smithi188_89276_object163423:head# 2024-04-23T16:25:14.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2859b4ba:::benchmark_data_smithi188_89276_object98736:head# 2024-04-23T16:25:14.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285a9eae:::benchmark_data_smithi188_89276_object85098:head# 2024-04-23T16:25:14.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285accf2:::benchmark_data_smithi188_89276_object103348:head# 2024-04-23T16:25:14.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285ae490:::benchmark_data_smithi188_89276_object109418:head# 2024-04-23T16:25:14.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285ba24c:::benchmark_data_smithi188_89276_object134632:head# 2024-04-23T16:25:14.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285bb69b:::benchmark_data_smithi188_89276_object122989:head# 2024-04-23T16:25:14.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285c42c6:::benchmark_data_smithi188_89276_object125162:head# 2024-04-23T16:25:14.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285c7295:::benchmark_data_smithi188_89276_object167402:head# 2024-04-23T16:25:14.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285cea37:::benchmark_data_smithi188_89276_object91389:head# 2024-04-23T16:25:14.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285d03f5:::benchmark_data_smithi188_89276_object127194:head# 2024-04-23T16:25:14.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285d610c:::benchmark_data_smithi188_89276_object70362:head# 2024-04-23T16:25:14.531 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285e9702:::benchmark_data_smithi188_89276_object117331:head# 2024-04-23T16:25:14.531 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285ea20b:::benchmark_data_smithi188_89276_object164398:head# 2024-04-23T16:25:14.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285eda66:::benchmark_data_smithi188_89276_object48401:head# 2024-04-23T16:25:14.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285f51d2:::benchmark_data_smithi188_89276_object157464:head# 2024-04-23T16:25:14.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:285f725e:::benchmark_data_smithi188_89276_object131417:head# 2024-04-23T16:25:14.533 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:286012cd:::benchmark_data_smithi188_89276_object43224:head# 2024-04-23T16:25:14.533 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28605763:::benchmark_data_smithi188_89276_object113029:head# 2024-04-23T16:25:14.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2860f655:::benchmark_data_smithi188_89276_object130930:head# 2024-04-23T16:25:14.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28618d9e:::benchmark_data_smithi188_89276_object90759:head# 2024-04-23T16:25:14.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2861d6b3:::benchmark_data_smithi188_89276_object79698:head# 2024-04-23T16:25:14.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2861ea40:::benchmark_data_smithi188_89276_object56764:head# 2024-04-23T16:25:14.536 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28625152:::benchmark_data_smithi188_89276_object47468:head# 2024-04-23T16:25:14.536 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28631569:::benchmark_data_smithi188_89276_object127731:head# 2024-04-23T16:25:14.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28645618:::benchmark_data_smithi188_89276_object56828:head# 2024-04-23T16:25:14.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2865fca3:::benchmark_data_smithi188_89276_object41043:head# 2024-04-23T16:25:14.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28666f6b:::benchmark_data_smithi188_89276_object146619:head# 2024-04-23T16:25:14.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2866e557:::benchmark_data_smithi188_89276_object132272:head# 2024-04-23T16:25:14.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2866ed84:::benchmark_data_smithi188_89276_object94619:head# 2024-04-23T16:25:14.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28676359:::benchmark_data_smithi188_89276_object146493:head# 2024-04-23T16:25:14.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28681144:::benchmark_data_smithi188_89276_object103222:head# 2024-04-23T16:25:14.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28684e58:::benchmark_data_smithi188_89276_object162655:head# 2024-04-23T16:25:14.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28685861:::benchmark_data_smithi188_89276_object75626:head# 2024-04-23T16:25:14.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28689aae:::benchmark_data_smithi188_89276_object101936:head# 2024-04-23T16:25:14.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2868f249:::benchmark_data_smithi188_89276_object152481:head# 2024-04-23T16:25:14.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28693a7e:::benchmark_data_smithi188_89276_object59775:head# 2024-04-23T16:25:14.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2869e270:::benchmark_data_smithi188_89276_object129749:head# 2024-04-23T16:25:14.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:286a4a0f:::benchmark_data_smithi188_89276_object125247:head# 2024-04-23T16:25:14.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:286b6480:::benchmark_data_smithi188_89276_object148140:head# 2024-04-23T16:25:14.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:286b9159:::benchmark_data_smithi188_89276_object57701:head# 2024-04-23T16:25:14.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:286bd43a:::benchmark_data_smithi188_89276_object166060:head# 2024-04-23T16:25:14.545 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:286bdca8:::benchmark_data_smithi188_89276_object86084:head# 2024-04-23T16:25:14.545 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:286c9601:::benchmark_data_smithi188_89276_object112427:head# 2024-04-23T16:25:14.545 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:286cb124:::benchmark_data_smithi188_89276_object80429:head# 2024-04-23T16:25:14.545 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:286cecc0:::benchmark_data_smithi188_89276_object61905:head# 2024-04-23T16:25:14.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:286d5d92:::benchmark_data_smithi188_89276_object129412:head# 2024-04-23T16:25:14.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:286de5f2:::benchmark_data_smithi188_89276_object47590:head# 2024-04-23T16:25:14.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:286e127b:::benchmark_data_smithi188_89276_object66896:head# 2024-04-23T16:25:14.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:286ea6f0:::benchmark_data_smithi188_89276_object134986:head# 2024-04-23T16:25:14.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:286ed7d5:::benchmark_data_smithi188_89276_object160628:head# 2024-04-23T16:25:14.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:286eda3e:::benchmark_data_smithi188_89276_object80951:head# 2024-04-23T16:25:14.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:286fb67f:::benchmark_data_smithi188_89276_object51689:head# 2024-04-23T16:25:14.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:286fbca2:::benchmark_data_smithi188_89276_object43507:head# 2024-04-23T16:25:14.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:286fbd91:::benchmark_data_smithi188_89276_object105008:head# 2024-04-23T16:25:14.552 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:287024dd:::benchmark_data_smithi188_89276_object58933:head# 2024-04-23T16:25:14.552 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28703610:::benchmark_data_smithi188_89276_object165453:head# 2024-04-23T16:25:14.553 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28720784:::benchmark_data_smithi188_89276_object157621:head# 2024-04-23T16:25:14.553 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28739567:::benchmark_data_smithi188_89276_object45395:head# 2024-04-23T16:25:14.554 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28746279:::benchmark_data_smithi188_89276_object164442:head# 2024-04-23T16:25:14.554 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:287462bf:::benchmark_data_smithi188_89276_object81847:head# 2024-04-23T16:25:14.555 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28749fd8:::benchmark_data_smithi188_89276_object163940:head# 2024-04-23T16:25:14.555 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28758d79:::benchmark_data_smithi188_89276_object149961:head# 2024-04-23T16:25:14.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:287627de:::benchmark_data_smithi188_89276_object165885:head# 2024-04-23T16:25:14.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28767354:::benchmark_data_smithi188_89276_object158798:head# 2024-04-23T16:25:14.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:287673a7:::benchmark_data_smithi188_89276_object113084:head# 2024-04-23T16:25:14.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28771f42:::benchmark_data_smithi188_89276_object93345:head# 2024-04-23T16:25:14.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2877a3c1:::benchmark_data_smithi188_89276_object99286:head# 2024-04-23T16:25:14.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2877b6e7:::benchmark_data_smithi188_89276_object77015:head# 2024-04-23T16:25:14.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2878363e:::benchmark_data_smithi188_89276_object160521:head# 2024-04-23T16:25:14.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:287840ec:::benchmark_data_smithi188_89276_object135030:head# 2024-04-23T16:25:14.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28793c77:::benchmark_data_smithi188_89276_object73507:head# 2024-04-23T16:25:14.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28794e6e:::benchmark_data_smithi188_89276_object166819:head# 2024-04-23T16:25:14.564 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:287a1bf3:::benchmark_data_smithi188_89276_object73373:head# 2024-04-23T16:25:14.564 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:287abb47:::benchmark_data_smithi188_89276_object162537:head# 2024-04-23T16:25:14.565 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:287bb395:::benchmark_data_smithi188_89276_object51307:head# 2024-04-23T16:25:14.565 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:287bdfbd:::benchmark_data_smithi188_89276_object111181:head# 2024-04-23T16:25:14.565 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:287e12ef:::benchmark_data_smithi188_89276_object58171:head# 2024-04-23T16:25:14.566 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:287e747f:::benchmark_data_smithi188_89276_object83104:head# 2024-04-23T16:25:14.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:287f8d69:::benchmark_data_smithi188_89276_object66368:head# 2024-04-23T16:25:14.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:287fe621:::benchmark_data_smithi188_89276_object143841:head# 2024-04-23T16:25:14.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28814108:::benchmark_data_smithi188_89276_object94939:head# 2024-04-23T16:25:14.569 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28814abe:::benchmark_data_smithi188_89276_object104615:head# 2024-04-23T16:25:14.569 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2881c27c:::benchmark_data_smithi188_89276_object55968:head# 2024-04-23T16:25:14.569 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2882654d:::benchmark_data_smithi188_89276_object104523:head# 2024-04-23T16:25:14.569 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28826bd6:::benchmark_data_smithi188_89276_object145745:head# 2024-04-23T16:25:14.570 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28836ab0:::benchmark_data_smithi188_89276_object65883:head# 2024-04-23T16:25:14.570 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28840e64:::benchmark_data_smithi188_89276_object128943:head# 2024-04-23T16:25:14.571 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2884507e:::benchmark_data_smithi188_89276_object72215:head# 2024-04-23T16:25:14.571 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28846c6e:::benchmark_data_smithi188_89276_object155515:head# 2024-04-23T16:25:14.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28850e0d:::benchmark_data_smithi188_89276_object63013:head# 2024-04-23T16:25:14.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2885caac:::benchmark_data_smithi188_89276_object166685:head# 2024-04-23T16:25:14.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:288672b4:::benchmark_data_smithi188_89276_object74183:head# 2024-04-23T16:25:14.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2886ac2f:::benchmark_data_smithi188_89276_object160393:head# 2024-04-23T16:25:14.574 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2886cbd8:::benchmark_data_smithi188_89276_object142448:head# 2024-04-23T16:25:14.574 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2886dbd1:::benchmark_data_smithi188_89276_object163530:head# 2024-04-23T16:25:14.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28870432:::benchmark_data_smithi188_89276_object52789:head# 2024-04-23T16:25:14.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28875511:::benchmark_data_smithi188_89276_object139986:head# 2024-04-23T16:25:14.576 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28878516:::benchmark_data_smithi188_89276_object164950:head# 2024-04-23T16:25:14.576 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2887e7ee:::benchmark_data_smithi188_89276_object133830:head# 2024-04-23T16:25:14.577 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2887f051:::benchmark_data_smithi188_89276_object50436:head# 2024-04-23T16:25:14.577 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:288848ae:::benchmark_data_smithi188_89276_object140640:head# 2024-04-23T16:25:14.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2888a2ce:::benchmark_data_smithi188_89276_object48839:head# 2024-04-23T16:25:14.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2888baed:::benchmark_data_smithi188_89276_object99074:head# 2024-04-23T16:25:14.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2888c813:::benchmark_data_smithi188_89276_object110391:head# 2024-04-23T16:25:14.579 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2888dd5b:::benchmark_data_smithi188_89276_object156269:head# 2024-04-23T16:25:14.579 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2888fd47:::benchmark_data_smithi188_89276_object115709:head# 2024-04-23T16:25:14.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:288950e3:::benchmark_data_smithi188_89276_object124928:head# 2024-04-23T16:25:14.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:288b0dda:::benchmark_data_smithi188_89276_object150122:head# 2024-04-23T16:25:14.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:288b72cc:::benchmark_data_smithi188_89276_object139238:head# 2024-04-23T16:25:14.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:288d91f2:::benchmark_data_smithi188_89276_object145007:head# 2024-04-23T16:25:14.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:288d9ac7:::benchmark_data_smithi188_89276_object119255:head# 2024-04-23T16:25:14.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:288e536a:::benchmark_data_smithi188_89276_object40794:head# 2024-04-23T16:25:14.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:288e6070:::benchmark_data_smithi188_89276_object129355:head# 2024-04-23T16:25:14.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28902f5f:::benchmark_data_smithi188_89276_object167311:head# 2024-04-23T16:25:14.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28905142:::benchmark_data_smithi188_89276_object126971:head# 2024-04-23T16:25:14.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28914f3a:::benchmark_data_smithi188_89276_object157134:head# 2024-04-23T16:25:14.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2891a73a:::benchmark_data_smithi188_89276_object63681:head# 2024-04-23T16:25:14.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:289201eb:::benchmark_data_smithi188_89276_object125672:head# 2024-04-23T16:25:14.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28927e80:::benchmark_data_smithi188_89276_object158745:head# 2024-04-23T16:25:14.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2892ec2f:::benchmark_data_smithi188_89276_object140094:head# 2024-04-23T16:25:14.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2894193d:::benchmark_data_smithi188_89276_object92470:head# 2024-04-23T16:25:14.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28951322:::benchmark_data_smithi188_89276_object124063:head# 2024-04-23T16:25:14.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28956b79:::benchmark_data_smithi188_89276_object80278:head# 2024-04-23T16:25:14.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2895a876:::benchmark_data_smithi188_89276_object53418:head# 2024-04-23T16:25:14.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2896ef35:::benchmark_data_smithi188_89276_object78350:head# 2024-04-23T16:25:14.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2897c6e3:::benchmark_data_smithi188_89276_object94973:head# 2024-04-23T16:25:14.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2898eccc:::benchmark_data_smithi188_89276_object166217:head# 2024-04-23T16:25:14.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2899ecad:::benchmark_data_smithi188_89276_object118110:head# 2024-04-23T16:25:14.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:289a21e6:::benchmark_data_smithi188_89276_object65628:head# 2024-04-23T16:25:14.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:289a3849:::benchmark_data_smithi188_89276_object156010:head# 2024-04-23T16:25:14.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:289a869a:::benchmark_data_smithi188_89276_object155033:head# 2024-04-23T16:25:14.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:289b3c0a:::benchmark_data_smithi188_89276_object153339:head# 2024-04-23T16:25:14.592 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:289b7ef5:::benchmark_data_smithi188_89276_object160786:head# 2024-04-23T16:25:14.592 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:289b9c42:::benchmark_data_smithi188_89276_object113592:head# 2024-04-23T16:25:14.592 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:289c7695:::benchmark_data_smithi188_89276_object109838:head# 2024-04-23T16:25:14.593 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:289ca4ea:::benchmark_data_smithi188_89276_object68205:head# 2024-04-23T16:25:14.593 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:289d687e:::benchmark_data_smithi188_89276_object122182:head# 2024-04-23T16:25:14.594 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:289d9918:::benchmark_data_smithi188_89276_object101337:head# 2024-04-23T16:25:14.594 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:289d9ddb:::benchmark_data_smithi188_89276_object158341:head# 2024-04-23T16:25:14.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:289eb362:::benchmark_data_smithi188_89276_object45521:head# 2024-04-23T16:25:14.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:289f1027:::benchmark_data_smithi188_89276_object66701:head# 2024-04-23T16:25:14.596 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:289fef03:::benchmark_data_smithi188_89276_object110887:head# 2024-04-23T16:25:14.596 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28a159f2:::benchmark_data_smithi188_89276_object151004:head# 2024-04-23T16:25:14.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28a18660:::benchmark_data_smithi188_89276_object106878:head# 2024-04-23T16:25:14.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28a20489:::benchmark_data_smithi188_89276_object63584:head# 2024-04-23T16:25:14.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28a34033:::benchmark_data_smithi188_89276_object45961:head# 2024-04-23T16:25:14.598 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28a51d3f:::benchmark_data_smithi188_89276_object142306:head# 2024-04-23T16:25:14.598 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28a53d21:::benchmark_data_smithi188_89276_object136892:head# 2024-04-23T16:25:14.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28a70777:::benchmark_data_smithi188_89276_object141967:head# 2024-04-23T16:25:14.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28a75265:::benchmark_data_smithi188_89276_object119910:head# 2024-04-23T16:25:14.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28a85252:::benchmark_data_smithi188_89276_object75372:head# 2024-04-23T16:25:14.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28a8f75a:::benchmark_data_smithi188_89276_object100062:head# 2024-04-23T16:25:14.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28a8fe1d:::benchmark_data_smithi188_89276_object113083:head# 2024-04-23T16:25:14.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28a91c8d:::benchmark_data_smithi188_89276_object162982:head# 2024-04-23T16:25:14.602 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28aa7890:::benchmark_data_smithi188_89276_object129855:head# 2024-04-23T16:25:14.602 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28aa83fb:::benchmark_data_smithi188_89276_object43483:head# 2024-04-23T16:25:14.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28aad94c:::benchmark_data_smithi188_89276_object165078:head# 2024-04-23T16:25:14.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28aadf46:::benchmark_data_smithi188_89276_object151738:head# 2024-04-23T16:25:14.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ab1b08:::benchmark_data_smithi188_89276_object106768:head# 2024-04-23T16:25:14.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ab29d9:::benchmark_data_smithi188_89276_object127076:head# 2024-04-23T16:25:14.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ab385c:::benchmark_data_smithi188_89276_object163872:head# 2024-04-23T16:25:14.605 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ab7915:::benchmark_data_smithi188_89276_object83581:head# 2024-04-23T16:25:14.605 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28abb6ed:::benchmark_data_smithi188_89276_object144082:head# 2024-04-23T16:25:14.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ac20de:::benchmark_data_smithi188_89276_object60984:head# 2024-04-23T16:25:14.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ace5f4:::benchmark_data_smithi188_89276_object43466:head# 2024-04-23T16:25:14.607 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ad8375:::benchmark_data_smithi188_89276_object62374:head# 2024-04-23T16:25:14.607 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28adc084:::benchmark_data_smithi188_89276_object66363:head# 2024-04-23T16:25:14.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ae66b2:::benchmark_data_smithi188_89276_object107483:head# 2024-04-23T16:25:14.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28af6479:::benchmark_data_smithi188_89276_object73849:head# 2024-04-23T16:25:14.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28b1973c:::benchmark_data_smithi188_89276_object81166:head# 2024-04-23T16:25:14.609 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28b1c7af:::benchmark_data_smithi188_89276_object61853:head# 2024-04-23T16:25:14.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28b1dc37:::benchmark_data_smithi188_89276_object143937:head# 2024-04-23T16:25:14.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28b2ac4e:::benchmark_data_smithi188_89276_object119848:head# 2024-04-23T16:25:14.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28b42fda:::benchmark_data_smithi188_89276_object120215:head# 2024-04-23T16:25:14.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28b641b4:::benchmark_data_smithi188_89276_object114282:head# 2024-04-23T16:25:14.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28b6481c:::benchmark_data_smithi188_89276_object165478:head# 2024-04-23T16:25:14.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28b69071:::benchmark_data_smithi188_89276_object104983:head# 2024-04-23T16:25:14.618 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28b6b99f:::benchmark_data_smithi188_89276_object75356:head# 2024-04-23T16:25:14.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28b6f933:::benchmark_data_smithi188_89276_object127037:head# 2024-04-23T16:25:14.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28b7d28c:::benchmark_data_smithi188_89276_object134716:head# 2024-04-23T16:25:14.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28b85623:::benchmark_data_smithi188_89276_object73555:head# 2024-04-23T16:25:14.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28b8b0cd:::benchmark_data_smithi188_89276_object79350:head# 2024-04-23T16:25:14.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28b929d1:::benchmark_data_smithi188_89276_object58553:head# 2024-04-23T16:25:14.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28b93f99:::benchmark_data_smithi188_89276_object58460:head# 2024-04-23T16:25:14.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28b95a33:::benchmark_data_smithi188_89276_object165555:head# 2024-04-23T16:25:14.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28b9cbdf:::benchmark_data_smithi188_89276_object113318:head# 2024-04-23T16:25:14.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28b9db67:::benchmark_data_smithi188_89276_object130185:head# 2024-04-23T16:25:14.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28b9ef78:::benchmark_data_smithi188_89276_object132436:head# 2024-04-23T16:25:14.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ba0910:::benchmark_data_smithi188_89276_object70124:head# 2024-04-23T16:25:14.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ba3faf:::benchmark_data_smithi188_89276_object117772:head# 2024-04-23T16:25:14.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ba705b:::benchmark_data_smithi188_89276_object77978:head# 2024-04-23T16:25:14.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28bae613:::benchmark_data_smithi188_89276_object84548:head# 2024-04-23T16:25:14.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28bb8dba:::benchmark_data_smithi188_89276_object88158:head# 2024-04-23T16:25:14.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28bba405:::benchmark_data_smithi188_89276_object84683:head# 2024-04-23T16:25:14.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28bd4124:::benchmark_data_smithi188_89276_object151677:head# 2024-04-23T16:25:14.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28bd43fd:::benchmark_data_smithi188_89276_object131209:head# 2024-04-23T16:25:14.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28bd5bdb:::benchmark_data_smithi188_89276_object65284:head# 2024-04-23T16:25:14.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28be238a:::benchmark_data_smithi188_89276_object80776:head# 2024-04-23T16:25:14.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28be3a27:::benchmark_data_smithi188_89276_object41094:head# 2024-04-23T16:25:14.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28beb762:::benchmark_data_smithi188_89276_object68387:head# 2024-04-23T16:25:14.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c01b21:::benchmark_data_smithi188_89276_object164439:head# 2024-04-23T16:25:14.624 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c0479c:::benchmark_data_smithi188_89276_object160034:head# 2024-04-23T16:25:14.624 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c0e750:::benchmark_data_smithi188_89276_object54021:head# 2024-04-23T16:25:14.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c2b058:::benchmark_data_smithi188_89276_object155659:head# 2024-04-23T16:25:14.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c312a6:::benchmark_data_smithi188_89276_object77663:head# 2024-04-23T16:25:14.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c3638b:::benchmark_data_smithi188_89276_object149676:head# 2024-04-23T16:25:14.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c386ca:::benchmark_data_smithi188_89276_object93892:head# 2024-04-23T16:25:14.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c44f32:::benchmark_data_smithi188_89276_object151650:head# 2024-04-23T16:25:14.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c47276:::benchmark_data_smithi188_89276_object125664:head# 2024-04-23T16:25:14.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c4733a:::benchmark_data_smithi188_89276_object110622:head# 2024-04-23T16:25:14.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c4f5c3:::benchmark_data_smithi188_89276_object58255:head# 2024-04-23T16:25:14.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c593d8:::benchmark_data_smithi188_89276_object118207:head# 2024-04-23T16:25:14.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c5968a:::benchmark_data_smithi188_89276_object64309:head# 2024-04-23T16:25:14.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c6a7c5:::benchmark_data_smithi188_89276_object43993:head# 2024-04-23T16:25:14.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c70be1:::benchmark_data_smithi188_89276_object114000:head# 2024-04-23T16:25:14.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c7b27c:::benchmark_data_smithi188_89276_object56607:head# 2024-04-23T16:25:14.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c7c0ec:::benchmark_data_smithi188_89276_object81509:head# 2024-04-23T16:25:14.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c7c1cc:::benchmark_data_smithi188_89276_object99533:head# 2024-04-23T16:25:14.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c8ef51:::benchmark_data_smithi188_89276_object71874:head# 2024-04-23T16:25:14.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c8ef9c:::benchmark_data_smithi188_89276_object64152:head# 2024-04-23T16:25:14.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c94434:::benchmark_data_smithi188_89276_object135265:head# 2024-04-23T16:25:14.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c9b96b:::benchmark_data_smithi188_89276_object78658:head# 2024-04-23T16:25:14.635 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28c9e6f5:::benchmark_data_smithi188_89276_object100193:head# 2024-04-23T16:25:14.635 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28cac72a:::benchmark_data_smithi188_89276_object56569:head# 2024-04-23T16:25:14.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28caf9a3:::benchmark_data_smithi188_89276_object56526:head# 2024-04-23T16:25:14.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28cb1473:::benchmark_data_smithi188_89276_object118412:head# 2024-04-23T16:25:14.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28cbdf82:::benchmark_data_smithi188_89276_object104754:head# 2024-04-23T16:25:14.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28cd5dfc:::benchmark_data_smithi188_89276_object126125:head# 2024-04-23T16:25:14.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ce538b:::benchmark_data_smithi188_89276_object77788:head# 2024-04-23T16:25:14.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28cec366:::benchmark_data_smithi188_89276_object166882:head# 2024-04-23T16:25:14.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28cf6285:::benchmark_data_smithi188_89276_object116163:head# 2024-04-23T16:25:14.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28cf72d9:::benchmark_data_smithi188_89276_object46378:head# 2024-04-23T16:25:14.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28cfd124:::benchmark_data_smithi188_89276_object157523:head# 2024-04-23T16:25:14.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d00138:::benchmark_data_smithi188_89276_object141360:head# 2024-04-23T16:25:14.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d147ad:::benchmark_data_smithi188_89276_object72157:head# 2024-04-23T16:25:14.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d1c9b8:::benchmark_data_smithi188_89276_object60579:head# 2024-04-23T16:25:14.645 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d1cb39:::benchmark_data_smithi188_89276_object41296:head# 2024-04-23T16:25:14.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d24184:::benchmark_data_smithi188_89276_object47079:head# 2024-04-23T16:25:14.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d247b7:::benchmark_data_smithi188_89276_object47056:head# 2024-04-23T16:25:14.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d286e7:::benchmark_data_smithi188_89276_object67786:head# 2024-04-23T16:25:14.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d2bb48:::benchmark_data_smithi188_89276_object139758:head# 2024-04-23T16:25:14.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d2bda3:::benchmark_data_smithi188_89276_object125346:head# 2024-04-23T16:25:14.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d3d78e:::benchmark_data_smithi188_89276_object93474:head# 2024-04-23T16:25:14.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d4ffc4:::benchmark_data_smithi188_89276_object60684:head# 2024-04-23T16:25:14.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d50683:::benchmark_data_smithi188_89276_object129131:head# 2024-04-23T16:25:14.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d523f5:::benchmark_data_smithi188_89276_object77844:head# 2024-04-23T16:25:14.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d58a35:::benchmark_data_smithi188_89276_object91541:head# 2024-04-23T16:25:14.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d6198a:::benchmark_data_smithi188_89276_object113836:head# 2024-04-23T16:25:14.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d61c29:::benchmark_data_smithi188_89276_object109741:head# 2024-04-23T16:25:14.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d667c2:::benchmark_data_smithi188_89276_object149680:head# 2024-04-23T16:25:14.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d6bf28:::benchmark_data_smithi188_89276_object135295:head# 2024-04-23T16:25:14.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d75465:::benchmark_data_smithi188_89276_object86486:head# 2024-04-23T16:25:14.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d79a88:::benchmark_data_smithi188_89276_object58080:head# 2024-04-23T16:25:14.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d7cd3c:::benchmark_data_smithi188_89276_object89022:head# 2024-04-23T16:25:14.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d913dd:::benchmark_data_smithi188_89276_object52706:head# 2024-04-23T16:25:14.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28d9264e:::benchmark_data_smithi188_89276_object110392:head# 2024-04-23T16:25:14.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28da03d0:::benchmark_data_smithi188_89276_object83019:head# 2024-04-23T16:25:14.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28da902b:::benchmark_data_smithi188_89276_object45338:head# 2024-04-23T16:25:14.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28dab43d:::benchmark_data_smithi188_89276_object129780:head# 2024-04-23T16:25:14.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28dbc59e:::benchmark_data_smithi188_89276_object60073:head# 2024-04-23T16:25:14.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28dbf9fd:::benchmark_data_smithi188_89276_object81348:head# 2024-04-23T16:25:14.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28dbfade:::benchmark_data_smithi188_89276_object106415:head# 2024-04-23T16:25:14.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28dd3dd1:::benchmark_data_smithi188_89276_object126481:head# 2024-04-23T16:25:14.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28dd95dd:::benchmark_data_smithi188_89276_object97504:head# 2024-04-23T16:25:14.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28de5c70:::benchmark_data_smithi188_89276_object113237:head# 2024-04-23T16:25:14.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28def57f:::benchmark_data_smithi188_89276_object50267:head# 2024-04-23T16:25:14.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28df60c5:::benchmark_data_smithi188_89276_object77875:head# 2024-04-23T16:25:14.661 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28df7624:::benchmark_data_smithi188_89276_object168130:head# 2024-04-23T16:25:14.661 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28df8cd4:::benchmark_data_smithi188_89276_object102631:head# 2024-04-23T16:25:14.661 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28df9dec:::benchmark_data_smithi188_89276_object97960:head# 2024-04-23T16:25:14.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28dfc84a:::benchmark_data_smithi188_89276_object42757:head# 2024-04-23T16:25:14.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28dfe28d:::benchmark_data_smithi188_89276_object161984:head# 2024-04-23T16:25:14.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28e01255:::benchmark_data_smithi188_89276_object148815:head# 2024-04-23T16:25:14.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28e07105:::benchmark_data_smithi188_89276_object95976:head# 2024-04-23T16:25:14.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28e0965e:::benchmark_data_smithi188_89276_object119159:head# 2024-04-23T16:25:14.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28e0dd16:::benchmark_data_smithi188_89276_object83925:head# 2024-04-23T16:25:14.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28e18048:::benchmark_data_smithi188_89276_object149285:head# 2024-04-23T16:25:14.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28e19c83:::benchmark_data_smithi188_89276_object47585:head# 2024-04-23T16:25:14.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28e1a165:::benchmark_data_smithi188_89276_object132308:head# 2024-04-23T16:25:14.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28e35d13:::benchmark_data_smithi188_89276_object146973:head# 2024-04-23T16:25:14.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28e3d87c:::benchmark_data_smithi188_89276_object74271:head# 2024-04-23T16:25:14.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28e489e1:::benchmark_data_smithi188_89276_object149040:head# 2024-04-23T16:25:14.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28e4a7ac:::benchmark_data_smithi188_89276_object93806:head# 2024-04-23T16:25:14.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28e55f08:::benchmark_data_smithi188_89276_object43430:head# 2024-04-23T16:25:14.668 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28e5bf62:::benchmark_data_smithi188_89276_object112030:head# 2024-04-23T16:25:14.668 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28e786d5:::benchmark_data_smithi188_89276_object65083:head# 2024-04-23T16:25:14.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28e7d07b:::benchmark_data_smithi188_89276_object51845:head# 2024-04-23T16:25:14.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28e8246f:::benchmark_data_smithi188_89276_object81196:head# 2024-04-23T16:25:14.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28e86229:::benchmark_data_smithi188_89276_object73526:head# 2024-04-23T16:25:14.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28e8ca3f:::benchmark_data_smithi188_89276_object45544:head# 2024-04-23T16:25:14.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28e8cc8c:::benchmark_data_smithi188_89276_object97026:head# 2024-04-23T16:25:14.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ea3143:::benchmark_data_smithi188_89276_object151583:head# 2024-04-23T16:25:14.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ea53c7:::benchmark_data_smithi188_89276_object148311:head# 2024-04-23T16:25:14.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ea7c41:::benchmark_data_smithi188_89276_object144459:head# 2024-04-23T16:25:14.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28eaa305:::benchmark_data_smithi188_89276_object82580:head# 2024-04-23T16:25:14.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28eacf03:::benchmark_data_smithi188_89276_object130344:head# 2024-04-23T16:25:14.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28eadd73:::benchmark_data_smithi188_89276_object69340:head# 2024-04-23T16:25:14.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28eaf648:::benchmark_data_smithi188_89276_object80886:head# 2024-04-23T16:25:14.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28eb71cb:::benchmark_data_smithi188_89276_object111278:head# 2024-04-23T16:25:14.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28eb8f81:::benchmark_data_smithi188_89276_object58906:head# 2024-04-23T16:25:14.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28eb8fa9:::benchmark_data_smithi188_89276_object120974:head# 2024-04-23T16:25:14.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28eb9360:::benchmark_data_smithi188_89276_object85339:head# 2024-04-23T16:25:14.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ebe0d2:::benchmark_data_smithi188_89276_object150662:head# 2024-04-23T16:25:14.678 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ec1f3d:::benchmark_data_smithi188_89276_object120799:head# 2024-04-23T16:25:14.678 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ecc82d:::benchmark_data_smithi188_89276_object46971:head# 2024-04-23T16:25:14.680 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ee8885:::benchmark_data_smithi188_89276_object144261:head# 2024-04-23T16:25:14.680 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28eeaea8:::benchmark_data_smithi188_89276_object97709:head# 2024-04-23T16:25:14.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28eee27c:::benchmark_data_smithi188_89276_object161859:head# 2024-04-23T16:25:14.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28eee41b:::benchmark_data_smithi188_89276_object164621:head# 2024-04-23T16:25:14.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ef02bb:::benchmark_data_smithi188_89276_object63980:head# 2024-04-23T16:25:14.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ef1dcc:::benchmark_data_smithi188_89276_object110442:head# 2024-04-23T16:25:14.683 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28efaaae:::benchmark_data_smithi188_89276_object168473:head# 2024-04-23T16:25:14.683 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28efc422:::benchmark_data_smithi188_89276_object83749:head# 2024-04-23T16:25:14.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f00453:::benchmark_data_smithi188_89276_object150576:head# 2024-04-23T16:25:14.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f0505c:::benchmark_data_smithi188_89276_object75588:head# 2024-04-23T16:25:14.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f0983e:::benchmark_data_smithi188_89276_object57179:head# 2024-04-23T16:25:14.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f18645:::benchmark_data_smithi188_89276_object60950:head# 2024-04-23T16:25:14.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f270f0:::benchmark_data_smithi188_89276_object98955:head# 2024-04-23T16:25:14.689 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f2e986:::benchmark_data_smithi188_89276_object160214:head# 2024-04-23T16:25:14.689 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f4462c:::benchmark_data_smithi188_89276_object124133:head# 2024-04-23T16:25:14.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f46ce5:::benchmark_data_smithi188_89276_object168376:head# 2024-04-23T16:25:14.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f4dee5:::benchmark_data_smithi188_89276_object79697:head# 2024-04-23T16:25:14.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f4fe10:::benchmark_data_smithi188_89276_object144012:head# 2024-04-23T16:25:14.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f60173:::benchmark_data_smithi188_89276_object123070:head# 2024-04-23T16:25:14.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f62824:::benchmark_data_smithi188_89276_object163507:head# 2024-04-23T16:25:14.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f6fb66:::benchmark_data_smithi188_89276_object125164:head# 2024-04-23T16:25:14.694 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f730d7:::benchmark_data_smithi188_89276_object80329:head# 2024-04-23T16:25:14.694 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f77616:::benchmark_data_smithi188_89276_object88965:head# 2024-04-23T16:25:14.694 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f7968f:::benchmark_data_smithi188_89276_object114949:head# 2024-04-23T16:25:14.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f7af5f:::benchmark_data_smithi188_89276_object48321:head# 2024-04-23T16:25:14.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f85a2e:::benchmark_data_smithi188_89276_object54269:head# 2024-04-23T16:25:14.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f8a4f8:::benchmark_data_smithi188_89276_object139014:head# 2024-04-23T16:25:14.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f8c422:::benchmark_data_smithi188_89276_object101468:head# 2024-04-23T16:25:14.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28f8f51e:::benchmark_data_smithi188_89276_object168633:head# 2024-04-23T16:25:14.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28fa9128:::benchmark_data_smithi188_89276_object109294:head# 2024-04-23T16:25:14.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28fbe3e5:::benchmark_data_smithi188_89276_object105404:head# 2024-04-23T16:25:14.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28fc5c8a:::benchmark_data_smithi188_89276_object87467:head# 2024-04-23T16:25:14.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28fcb8ed:::benchmark_data_smithi188_89276_object40346:head# 2024-04-23T16:25:14.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ff2f80:::benchmark_data_smithi188_89276_object70475:head# 2024-04-23T16:25:14.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:28ff6971:::benchmark_data_smithi188_89276_object165815:head# 2024-04-23T16:25:14.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29000606:::benchmark_data_smithi188_89276_object88815:head# 2024-04-23T16:25:14.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:290025df:::benchmark_data_smithi188_89276_object103143:head# 2024-04-23T16:25:14.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29006507:::benchmark_data_smithi188_89276_object168148:head# 2024-04-23T16:25:14.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29011d45:::benchmark_data_smithi188_89276_object60165:head# 2024-04-23T16:25:14.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29014f33:::benchmark_data_smithi188_89276_object103276:head# 2024-04-23T16:25:14.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29019827:::benchmark_data_smithi188_89276_object71764:head# 2024-04-23T16:25:14.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2901bab6:::benchmark_data_smithi188_89276_object118842:head# 2024-04-23T16:25:14.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29028b50:::benchmark_data_smithi188_89276_object112738:head# 2024-04-23T16:25:14.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2902a7a7:::benchmark_data_smithi188_89276_object140317:head# 2024-04-23T16:25:14.709 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2902f1d0:::benchmark_data_smithi188_89276_object68147:head# 2024-04-23T16:25:14.709 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2903332c:::benchmark_data_smithi188_89276_object130917:head# 2024-04-23T16:25:14.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29042e4e:::benchmark_data_smithi188_89276_object75030:head# 2024-04-23T16:25:14.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29045eff:::benchmark_data_smithi188_89276_object58858:head# 2024-04-23T16:25:14.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2904e4ac:::benchmark_data_smithi188_89276_object65777:head# 2024-04-23T16:25:14.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29063372:::benchmark_data_smithi188_89276_object105855:head# 2024-04-23T16:25:14.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2906f85c:::benchmark_data_smithi188_89276_object163597:head# 2024-04-23T16:25:14.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29074145:::benchmark_data_smithi188_89276_object86496:head# 2024-04-23T16:25:14.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29075408:::benchmark_data_smithi188_89276_object93400:head# 2024-04-23T16:25:14.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2907681c:::benchmark_data_smithi188_89276_object83526:head# 2024-04-23T16:25:14.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29085164:::benchmark_data_smithi188_89276_object132312:head# 2024-04-23T16:25:14.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:290913a0:::benchmark_data_smithi188_89276_object107156:head# 2024-04-23T16:25:14.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:290951b3:::benchmark_data_smithi188_89276_object141146:head# 2024-04-23T16:25:14.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:290a26eb:::benchmark_data_smithi188_89276_object104367:head# 2024-04-23T16:25:14.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:290a6ae2:::benchmark_data_smithi188_89276_object121412:head# 2024-04-23T16:25:14.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:290b6970:::benchmark_data_smithi188_89276_object106306:head# 2024-04-23T16:25:14.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:290c3b89:::benchmark_data_smithi188_89276_object60563:head# 2024-04-23T16:25:14.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:290d4422:::benchmark_data_smithi188_89276_object71428:head# 2024-04-23T16:25:14.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:290e36b5:::benchmark_data_smithi188_89276_object94419:head# 2024-04-23T16:25:14.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:290e3a5d:::benchmark_data_smithi188_89276_object51091:head# 2024-04-23T16:25:14.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:290e988d:::benchmark_data_smithi188_89276_object104617:head# 2024-04-23T16:25:14.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:290f50a6:::benchmark_data_smithi188_89276_object156989:head# 2024-04-23T16:25:14.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29106dd1:::benchmark_data_smithi188_89276_object84171:head# 2024-04-23T16:25:14.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291154fd:::benchmark_data_smithi188_89276_object62685:head# 2024-04-23T16:25:14.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291169aa:::benchmark_data_smithi188_89276_object96059:head# 2024-04-23T16:25:14.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291258e7:::benchmark_data_smithi188_89276_object165484:head# 2024-04-23T16:25:14.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2912b437:::benchmark_data_smithi188_89276_object128973:head# 2024-04-23T16:25:14.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2912c429:::benchmark_data_smithi188_89276_object168566:head# 2024-04-23T16:25:14.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2913d97f:::benchmark_data_smithi188_89276_object77305:head# 2024-04-23T16:25:14.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2913e9f4:::benchmark_data_smithi188_89276_object131670:head# 2024-04-23T16:25:14.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29142298:::benchmark_data_smithi188_89276_object145920:head# 2024-04-23T16:25:14.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291427cc:::benchmark_data_smithi188_89276_object117527:head# 2024-04-23T16:25:14.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2914b11f:::benchmark_data_smithi188_89276_object41537:head# 2024-04-23T16:25:14.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2914b3da:::benchmark_data_smithi188_89276_object129110:head# 2024-04-23T16:25:14.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29152cf6:::benchmark_data_smithi188_89276_object52423:head# 2024-04-23T16:25:14.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291548aa:::benchmark_data_smithi188_89276_object109177:head# 2024-04-23T16:25:14.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29160412:::benchmark_data_smithi188_89276_object89885:head# 2024-04-23T16:25:14.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29174c5c:::benchmark_data_smithi188_89276_object164763:head# 2024-04-23T16:25:14.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29175f84:::benchmark_data_smithi188_89276_object68336:head# 2024-04-23T16:25:14.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291901a6:::benchmark_data_smithi188_89276_object71549:head# 2024-04-23T16:25:14.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29194bc0:::benchmark_data_smithi188_89276_object154292:head# 2024-04-23T16:25:14.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291988ad:::benchmark_data_smithi188_89276_object166391:head# 2024-04-23T16:25:14.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291a0103:::benchmark_data_smithi188_89276_object147350:head# 2024-04-23T16:25:14.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291a0f82:::benchmark_data_smithi188_89276_object67476:head# 2024-04-23T16:25:14.735 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291a1bf2:::benchmark_data_smithi188_89276_object144004:head# 2024-04-23T16:25:14.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291a6c21:::benchmark_data_smithi188_89276_object145142:head# 2024-04-23T16:25:14.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291a9b3d:::benchmark_data_smithi188_89276_object49010:head# 2024-04-23T16:25:14.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291b0062:::benchmark_data_smithi188_89276_object69424:head# 2024-04-23T16:25:14.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291b09eb:::benchmark_data_smithi188_89276_object126288:head# 2024-04-23T16:25:14.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291b8d17:::benchmark_data_smithi188_89276_object76778:head# 2024-04-23T16:25:14.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291bf484:::benchmark_data_smithi188_89276_object57089:head# 2024-04-23T16:25:14.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291d66ec:::benchmark_data_smithi188_89276_object64114:head# 2024-04-23T16:25:14.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291d99ac:::benchmark_data_smithi188_89276_object68096:head# 2024-04-23T16:25:14.744 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291e5d22:::benchmark_data_smithi188_89276_object47620:head# 2024-04-23T16:25:14.744 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:291fb447:::benchmark_data_smithi188_89276_object164181:head# 2024-04-23T16:25:14.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29200d3d:::benchmark_data_smithi188_89276_object104768:head# 2024-04-23T16:25:14.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29215728:::benchmark_data_smithi188_89276_object50340:head# 2024-04-23T16:25:14.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2921f23f:::benchmark_data_smithi188_89276_object94671:head# 2024-04-23T16:25:14.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29223537:::benchmark_data_smithi188_89276_object53889:head# 2024-04-23T16:25:14.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29224a4d:::benchmark_data_smithi188_89276_object50728:head# 2024-04-23T16:25:14.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2922d2d0:::benchmark_data_smithi188_89276_object65921:head# 2024-04-23T16:25:14.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29233e30:::benchmark_data_smithi188_89276_object54232:head# 2024-04-23T16:25:14.750 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2923c8a5:::benchmark_data_smithi188_89276_object78358:head# 2024-04-23T16:25:14.750 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:292410d0:::benchmark_data_smithi188_89276_object161988:head# 2024-04-23T16:25:14.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2924e19f:::benchmark_data_smithi188_89276_object41248:head# 2024-04-23T16:25:14.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2925f325:::benchmark_data_smithi188_89276_object64547:head# 2024-04-23T16:25:14.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2926219e:::benchmark_data_smithi188_89276_object78781:head# 2024-04-23T16:25:14.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2926adb6:::benchmark_data_smithi188_89276_object124049:head# 2024-04-23T16:25:14.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29270b9f:::benchmark_data_smithi188_89276_object57633:head# 2024-04-23T16:25:14.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2927944c:::benchmark_data_smithi188_89276_object148949:head# 2024-04-23T16:25:14.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2927d05d:::benchmark_data_smithi188_89276_object159115:head# 2024-04-23T16:25:14.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:292831bc:::benchmark_data_smithi188_89276_object56750:head# 2024-04-23T16:25:14.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2929a87c:::benchmark_data_smithi188_89276_object50349:head# 2024-04-23T16:25:14.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2929c8a5:::benchmark_data_smithi188_89276_object89563:head# 2024-04-23T16:25:14.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2929d21e:::benchmark_data_smithi188_89276_object121235:head# 2024-04-23T16:25:14.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:292a1739:::benchmark_data_smithi188_89276_object137292:head# 2024-04-23T16:25:14.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:292b389d:::benchmark_data_smithi188_89276_object166358:head# 2024-04-23T16:25:14.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:292c5c47:::benchmark_data_smithi188_89276_object154713:head# 2024-04-23T16:25:14.760 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:14.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:292d3f07:::benchmark_data_smithi188_89276_object81100:head# 2024-04-23T16:25:14.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:14.765 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:292d4857:::benchmark_data_smithi188_89276_object71634:head# 2024-04-23T16:25:14.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:292d7d0d:::benchmark_data_smithi188_89276_object145012:head# 2024-04-23T16:25:14.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:292dccc2:::benchmark_data_smithi188_89276_object112131:head# 2024-04-23T16:25:14.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:292e3061:::benchmark_data_smithi188_89276_object90649:head# 2024-04-23T16:25:14.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:292e86ce:::benchmark_data_smithi188_89276_object144279:head# 2024-04-23T16:25:14.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:292ec19e:::benchmark_data_smithi188_89276_object149473:head# 2024-04-23T16:25:14.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:292eecfd:::benchmark_data_smithi188_89276_object156967:head# 2024-04-23T16:25:14.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:292f1399:::benchmark_data_smithi188_89276_object129789:head# 2024-04-23T16:25:14.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:14.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293012f3:::benchmark_data_smithi188_89276_object125758:head# 2024-04-23T16:25:14.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29302693:::benchmark_data_smithi188_89276_object89671:head# 2024-04-23T16:25:14.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29306a75:::benchmark_data_smithi188_89276_object85538:head# 2024-04-23T16:25:14.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2931fd02:::benchmark_data_smithi188_89276_object75305:head# 2024-04-23T16:25:14.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29324bcd:::benchmark_data_smithi188_89276_object157904:head# 2024-04-23T16:25:14.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2933159d:::benchmark_data_smithi188_89276_object167238:head# 2024-04-23T16:25:14.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293356ce:::benchmark_data_smithi188_89276_object115351:head# 2024-04-23T16:25:14.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29335c2c:::benchmark_data_smithi188_89276_object54866:head# 2024-04-23T16:25:14.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293376b4:::benchmark_data_smithi188_89276_object94481:head# 2024-04-23T16:25:14.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2934008b:::benchmark_data_smithi188_89276_object44664:head# 2024-04-23T16:25:14.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2934d35b:::benchmark_data_smithi188_89276_object115131:head# 2024-04-23T16:25:14.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29354abd:::benchmark_data_smithi188_89276_object82880:head# 2024-04-23T16:25:14.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29355c01:::benchmark_data_smithi188_89276_object161791:head# 2024-04-23T16:25:14.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2935c953:::benchmark_data_smithi188_89276_object102522:head# 2024-04-23T16:25:14.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29367ae3:::benchmark_data_smithi188_89276_object160540:head# 2024-04-23T16:25:14.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29368773:::benchmark_data_smithi188_89276_object138980:head# 2024-04-23T16:25:14.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2936e295:::benchmark_data_smithi188_89276_object58154:head# 2024-04-23T16:25:14.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293753a5:::benchmark_data_smithi188_89276_object107141:head# 2024-04-23T16:25:14.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29378fd2:::benchmark_data_smithi188_89276_object152092:head# 2024-04-23T16:25:14.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2938949a:::benchmark_data_smithi188_89276_object56118:head# 2024-04-23T16:25:14.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2939b656:::benchmark_data_smithi188_89276_object152572:head# 2024-04-23T16:25:14.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293a2bd0:::benchmark_data_smithi188_89276_object89010:head# 2024-04-23T16:25:14.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293a3c5f:::benchmark_data_smithi188_89276_object108366:head# 2024-04-23T16:25:14.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293a6bca:::benchmark_data_smithi188_89276_object64557:head# 2024-04-23T16:25:14.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293ab4a1:::benchmark_data_smithi188_89276_object92782:head# 2024-04-23T16:25:14.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293af16b:::benchmark_data_smithi188_89276_object115032:head# 2024-04-23T16:25:14.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293b281d:::benchmark_data_smithi188_89276_object62270:head# 2024-04-23T16:25:14.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293b454c:::benchmark_data_smithi188_89276_object64724:head# 2024-04-23T16:25:14.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293b7607:::benchmark_data_smithi188_89276_object46191:head# 2024-04-23T16:25:14.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293b8b8a:::benchmark_data_smithi188_89276_object70346:head# 2024-04-23T16:25:14.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293bfeb6:::benchmark_data_smithi188_89276_object147969:head# 2024-04-23T16:25:14.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293c60fc:::benchmark_data_smithi188_89276_object54397:head# 2024-04-23T16:25:14.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293cdda9:::benchmark_data_smithi188_89276_object48768:head# 2024-04-23T16:25:14.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293d2704:::benchmark_data_smithi188_89276_object113610:head# 2024-04-23T16:25:14.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293dbd49:::benchmark_data_smithi188_89276_object135900:head# 2024-04-23T16:25:14.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293dc08b:::benchmark_data_smithi188_89276_object141187:head# 2024-04-23T16:25:14.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293e5346:::benchmark_data_smithi188_89276_object48966:head# 2024-04-23T16:25:14.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293e6286:::benchmark_data_smithi188_89276_object136760:head# 2024-04-23T16:25:14.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293e9980:::benchmark_data_smithi188_89276_object44236:head# 2024-04-23T16:25:14.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293ecc87:::benchmark_data_smithi188_89276_object60833:head# 2024-04-23T16:25:14.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293eef4d:::benchmark_data_smithi188_89276_object90904:head# 2024-04-23T16:25:14.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:293f88bb:::benchmark_data_smithi188_89276_object79023:head# 2024-04-23T16:25:14.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29409f57:::benchmark_data_smithi188_89276_object113327:head# 2024-04-23T16:25:14.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2941a620:::benchmark_data_smithi188_89276_object91391:head# 2024-04-23T16:25:14.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2941faed:::benchmark_data_smithi188_89276_object56127:head# 2024-04-23T16:25:14.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:294260d3:::benchmark_data_smithi188_89276_object103156:head# 2024-04-23T16:25:14.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2942ae8d:::benchmark_data_smithi188_89276_object140434:head# 2024-04-23T16:25:14.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2942d717:::benchmark_data_smithi188_89276_object105726:head# 2024-04-23T16:25:14.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2944f686:::benchmark_data_smithi188_89276_object82879:head# 2024-04-23T16:25:14.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2945486e:::benchmark_data_smithi188_89276_object131393:head# 2024-04-23T16:25:14.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2945aa6a:::benchmark_data_smithi188_89276_object71719:head# 2024-04-23T16:25:14.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:294657d4:::benchmark_data_smithi188_89276_object102489:head# 2024-04-23T16:25:14.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2947354d:::benchmark_data_smithi188_89276_object125866:head# 2024-04-23T16:25:14.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29478ef1:::benchmark_data_smithi188_89276_object161724:head# 2024-04-23T16:25:14.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:294afe82:::benchmark_data_smithi188_89276_object134315:head# 2024-04-23T16:25:14.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:294b6f19:::benchmark_data_smithi188_89276_object155759:head# 2024-04-23T16:25:14.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:294bdd83:::benchmark_data_smithi188_89276_object80396:head# 2024-04-23T16:25:14.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:294c0130:::benchmark_data_smithi188_89276_object77877:head# 2024-04-23T16:25:14.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:294c084f:::benchmark_data_smithi188_89276_object137915:head# 2024-04-23T16:25:14.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:294c4792:::benchmark_data_smithi188_89276_object127816:head# 2024-04-23T16:25:14.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:294c77a9:::benchmark_data_smithi188_89276_object126423:head# 2024-04-23T16:25:14.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:294ce8bb:::benchmark_data_smithi188_89276_object114939:head# 2024-04-23T16:25:14.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:294d2379:::benchmark_data_smithi188_89276_object91954:head# 2024-04-23T16:25:14.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:294d648a:::benchmark_data_smithi188_89276_object151000:head# 2024-04-23T16:25:14.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:294d910d:::benchmark_data_smithi188_89276_object113793:head# 2024-04-23T16:25:14.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:294de50a:::benchmark_data_smithi188_89276_object164458:head# 2024-04-23T16:25:14.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:294e210b:::benchmark_data_smithi188_89276_object86261:head# 2024-04-23T16:25:14.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:294e3cb6:::benchmark_data_smithi188_89276_object85463:head# 2024-04-23T16:25:14.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:294f8a4d:::benchmark_data_smithi188_89276_object93424:head# 2024-04-23T16:25:14.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:294f8f6d:::benchmark_data_smithi188_89276_object132009:head# 2024-04-23T16:25:14.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:294fa06e:::benchmark_data_smithi188_89276_object69683:head# 2024-04-23T16:25:14.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2950ae13:::benchmark_data_smithi188_89276_object63465:head# 2024-04-23T16:25:14.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2950db24:::benchmark_data_smithi188_89276_object83198:head# 2024-04-23T16:25:14.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29522a9f:::benchmark_data_smithi188_89276_object41912:head# 2024-04-23T16:25:14.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29528c8c:::benchmark_data_smithi188_89276_object126851:head# 2024-04-23T16:25:14.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2952dff0:::benchmark_data_smithi188_89276_object98871:head# 2024-04-23T16:25:14.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29572f3c:::benchmark_data_smithi188_89276_object97792:head# 2024-04-23T16:25:14.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29578e9a:::benchmark_data_smithi188_89276_object86537:head# 2024-04-23T16:25:14.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2957f085:::benchmark_data_smithi188_89276_object115061:head# 2024-04-23T16:25:14.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2957f097:::benchmark_data_smithi188_89276_object48467:head# 2024-04-23T16:25:14.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29585711:::benchmark_data_smithi188_89276_object107210:head# 2024-04-23T16:25:14.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2958c4dd:::benchmark_data_smithi188_89276_object126480:head# 2024-04-23T16:25:14.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:295a6ea0:::benchmark_data_smithi188_89276_object74774:head# 2024-04-23T16:25:14.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:295bf883:::benchmark_data_smithi188_89276_object133984:head# 2024-04-23T16:25:14.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:295d0219:::benchmark_data_smithi188_89276_object145003:head# 2024-04-23T16:25:14.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:295d1fc8:::benchmark_data_smithi188_89276_object160821:head# 2024-04-23T16:25:14.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:295d7d57:::benchmark_data_smithi188_89276_object95465:head# 2024-04-23T16:25:14.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:295ddc3a:::benchmark_data_smithi188_89276_object158383:head# 2024-04-23T16:25:14.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:295e7508:::benchmark_data_smithi188_89276_object103141:head# 2024-04-23T16:25:14.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:295efb02:::benchmark_data_smithi188_89276_object146842:head# 2024-04-23T16:25:14.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:295f2f4f:::benchmark_data_smithi188_89276_object97276:head# 2024-04-23T16:25:14.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:295f85d7:::benchmark_data_smithi188_89276_object112911:head# 2024-04-23T16:25:14.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:295f9072:::benchmark_data_smithi188_89276_object142110:head# 2024-04-23T16:25:14.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29601d2c:::benchmark_data_smithi188_89276_object154073:head# 2024-04-23T16:25:14.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29602acf:::benchmark_data_smithi188_89276_object166876:head# 2024-04-23T16:25:14.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2961a008:::benchmark_data_smithi188_89276_object163673:head# 2024-04-23T16:25:14.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2961dd3d:::benchmark_data_smithi188_89276_object55936:head# 2024-04-23T16:25:14.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:296359a0:::benchmark_data_smithi188_89276_object159211:head# 2024-04-23T16:25:14.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2963a63c:::benchmark_data_smithi188_89276_object159753:head# 2024-04-23T16:25:14.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2963e61e:::benchmark_data_smithi188_89276_object127156:head# 2024-04-23T16:25:14.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2963f038:::benchmark_data_smithi188_89276_object121019:head# 2024-04-23T16:25:14.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29641c22:::benchmark_data_smithi188_89276_object157465:head# 2024-04-23T16:25:14.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2964777f:::benchmark_data_smithi188_89276_object98730:head# 2024-04-23T16:25:14.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:296572ec:::benchmark_data_smithi188_89276_object163893:head# 2024-04-23T16:25:14.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29670825:::benchmark_data_smithi188_89276_object64601:head# 2024-04-23T16:25:14.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:296715fc:::benchmark_data_smithi188_89276_object149079:head# 2024-04-23T16:25:14.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29679515:::benchmark_data_smithi188_89276_object45749:head# 2024-04-23T16:25:14.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29682644:::benchmark_data_smithi188_89276_object116117:head# 2024-04-23T16:25:14.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:296985d6:::benchmark_data_smithi188_89276_object114034:head# 2024-04-23T16:25:14.814 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:296ab5eb:::benchmark_data_smithi188_89276_object146038:head# 2024-04-23T16:25:14.814 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:296af43a:::benchmark_data_smithi188_89276_object141119:head# 2024-04-23T16:25:14.814 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:296b8423:::benchmark_data_smithi188_89276_object40567:head# 2024-04-23T16:25:14.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:296b9b55:::benchmark_data_smithi188_89276_object130469:head# 2024-04-23T16:25:14.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:296bd2d3:::benchmark_data_smithi188_89276_object133569:head# 2024-04-23T16:25:14.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:296cfc4f:::benchmark_data_smithi188_89276_object101458:head# 2024-04-23T16:25:14.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:296e324b:::benchmark_data_smithi188_89276_object81263:head# 2024-04-23T16:25:14.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:296ed164:::benchmark_data_smithi188_89276_object131602:head# 2024-04-23T16:25:14.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:296f246a:::benchmark_data_smithi188_89276_object112889:head# 2024-04-23T16:25:14.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29706cdd:::benchmark_data_smithi188_89276_object148729:head# 2024-04-23T16:25:14.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2971142d:::benchmark_data_smithi188_89276_object83953:head# 2024-04-23T16:25:14.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29713d48:::benchmark_data_smithi188_89276_object116475:head# 2024-04-23T16:25:14.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2972b19f:::benchmark_data_smithi188_89276_object94642:head# 2024-04-23T16:25:14.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2972d602:::benchmark_data_smithi188_89276_object139129:head# 2024-04-23T16:25:14.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2972def7:::benchmark_data_smithi188_89276_object146819:head# 2024-04-23T16:25:14.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:297328de:::benchmark_data_smithi188_89276_object63150:head# 2024-04-23T16:25:14.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2973979a:::benchmark_data_smithi188_89276_object55338:head# 2024-04-23T16:25:14.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29742e8e:::benchmark_data_smithi188_89276_object123784:head# 2024-04-23T16:25:14.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2975134c:::benchmark_data_smithi188_89276_object85727:head# 2024-04-23T16:25:14.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29755132:::benchmark_data_smithi188_89276_object156975:head# 2024-04-23T16:25:14.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29755c14:::benchmark_data_smithi188_89276_object153854:head# 2024-04-23T16:25:14.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:297589bc:::benchmark_data_smithi188_89276_object109978:head# 2024-04-23T16:25:14.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2975d525:::benchmark_data_smithi188_89276_object87865:head# 2024-04-23T16:25:14.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:297603c1:::benchmark_data_smithi188_89276_object57455:head# 2024-04-23T16:25:14.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2976c89f:::benchmark_data_smithi188_89276_object111216:head# 2024-04-23T16:25:14.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2976f11c:::benchmark_data_smithi188_89276_object138641:head# 2024-04-23T16:25:14.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2976fe84:::benchmark_data_smithi188_89276_object41314:head# 2024-04-23T16:25:14.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29773a78:::benchmark_data_smithi188_89276_object129500:head# 2024-04-23T16:25:14.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29795066:::benchmark_data_smithi188_89276_object135785:head# 2024-04-23T16:25:14.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:297a303c:::benchmark_data_smithi188_89276_object60074:head# 2024-04-23T16:25:14.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:297a4c51:::benchmark_data_smithi188_89276_object150568:head# 2024-04-23T16:25:14.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:297b2dd9:::benchmark_data_smithi188_89276_object104126:head# 2024-04-23T16:25:14.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:297bb0a6:::benchmark_data_smithi188_89276_object65455:head# 2024-04-23T16:25:14.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:297cab05:::benchmark_data_smithi188_89276_object142372:head# 2024-04-23T16:25:14.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:297db7a5:::benchmark_data_smithi188_89276_object122384:head# 2024-04-23T16:25:14.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:297dfbb4:::benchmark_data_smithi188_89276_object150840:head# 2024-04-23T16:25:14.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:297e0f96:::benchmark_data_smithi188_89276_object133375:head# 2024-04-23T16:25:14.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:297e7a36:::benchmark_data_smithi188_89276_object81714:head# 2024-04-23T16:25:14.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:297e82d3:::benchmark_data_smithi188_89276_object77303:head# 2024-04-23T16:25:14.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29801910:::benchmark_data_smithi188_89276_object145552:head# 2024-04-23T16:25:14.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29803657:::benchmark_data_smithi188_89276_object83005:head# 2024-04-23T16:25:14.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:298124bf:::benchmark_data_smithi188_89276_object127333:head# 2024-04-23T16:25:14.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2981bd6c:::benchmark_data_smithi188_89276_object143058:head# 2024-04-23T16:25:14.832 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29822497:::benchmark_data_smithi188_89276_object97981:head# 2024-04-23T16:25:14.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:298243d2:::benchmark_data_smithi188_89276_object85083:head# 2024-04-23T16:25:14.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29827808:::benchmark_data_smithi188_89276_object87527:head# 2024-04-23T16:25:14.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:298291e9:::benchmark_data_smithi188_89276_object57000:head# 2024-04-23T16:25:14.834 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2982ba7a:::benchmark_data_smithi188_89276_object164344:head# 2024-04-23T16:25:14.834 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2982e656:::benchmark_data_smithi188_89276_object136016:head# 2024-04-23T16:25:14.834 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:298336d6:::benchmark_data_smithi188_89276_object64667:head# 2024-04-23T16:25:14.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:298390f8:::benchmark_data_smithi188_89276_object81301:head# 2024-04-23T16:25:14.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2983bc72:::benchmark_data_smithi188_89276_object90947:head# 2024-04-23T16:25:14.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29849b12:::benchmark_data_smithi188_89276_object167503:head# 2024-04-23T16:25:14.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2984d7e9:::benchmark_data_smithi188_89276_object167263:head# 2024-04-23T16:25:14.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29850413:::benchmark_data_smithi188_89276_object78293:head# 2024-04-23T16:25:14.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:298591fe:::benchmark_data_smithi188_89276_object168201:head# 2024-04-23T16:25:14.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29865beb:::benchmark_data_smithi188_89276_object145883:head# 2024-04-23T16:25:14.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29866c46:::benchmark_data_smithi188_89276_object81232:head# 2024-04-23T16:25:14.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2986bd92:::benchmark_data_smithi188_89276_object125891:head# 2024-04-23T16:25:14.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2987477c:::benchmark_data_smithi188_89276_object72125:head# 2024-04-23T16:25:14.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29876642:::benchmark_data_smithi188_89276_object159119:head# 2024-04-23T16:25:14.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2987e480:::benchmark_data_smithi188_89276_object166788:head# 2024-04-23T16:25:14.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2987f207:::benchmark_data_smithi188_89276_object94937:head# 2024-04-23T16:25:14.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29885b11:::benchmark_data_smithi188_89276_object140152:head# 2024-04-23T16:25:14.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2989039d:::benchmark_data_smithi188_89276_object117433:head# 2024-04-23T16:25:14.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:298ad740:::benchmark_data_smithi188_89276_object106063:head# 2024-04-23T16:25:14.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:298ae5ed:::benchmark_data_smithi188_89276_object65462:head# 2024-04-23T16:25:14.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:298bffa7:::benchmark_data_smithi188_89276_object84220:head# 2024-04-23T16:25:14.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:298ce9eb:::benchmark_data_smithi188_89276_object100411:head# 2024-04-23T16:25:14.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:298de073:::benchmark_data_smithi188_89276_object71748:head# 2024-04-23T16:25:14.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:298e0573:::benchmark_data_smithi188_89276_object123683:head# 2024-04-23T16:25:14.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:298f9d77:::benchmark_data_smithi188_89276_object139938:head# 2024-04-23T16:25:14.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:298fb535:::benchmark_data_smithi188_89276_object45173:head# 2024-04-23T16:25:14.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2990690a:::benchmark_data_smithi188_89276_object144908:head# 2024-04-23T16:25:14.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29909b90:::benchmark_data_smithi188_89276_object75617:head# 2024-04-23T16:25:14.845 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2991023f:::benchmark_data_smithi188_89276_object95977:head# 2024-04-23T16:25:14.845 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2991109f:::benchmark_data_smithi188_89276_object62060:head# 2024-04-23T16:25:14.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2991e628:::benchmark_data_smithi188_89276_object70169:head# 2024-04-23T16:25:14.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29925b1a:::benchmark_data_smithi188_89276_object115763:head# 2024-04-23T16:25:14.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2992ff84:::benchmark_data_smithi188_89276_object91853:head# 2024-04-23T16:25:14.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29932fe1:::benchmark_data_smithi188_89276_object40364:head# 2024-04-23T16:25:14.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2993c320:::benchmark_data_smithi188_89276_object88303:head# 2024-04-23T16:25:14.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29944013:::benchmark_data_smithi188_89276_object109324:head# 2024-04-23T16:25:14.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29948781:::benchmark_data_smithi188_89276_object141864:head# 2024-04-23T16:25:14.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2995e0f5:::benchmark_data_smithi188_89276_object121340:head# 2024-04-23T16:25:14.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29963d59:::benchmark_data_smithi188_89276_object42602:head# 2024-04-23T16:25:14.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2996722e:::benchmark_data_smithi188_89276_object112154:head# 2024-04-23T16:25:14.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:299679f5:::benchmark_data_smithi188_89276_object131705:head# 2024-04-23T16:25:14.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2998b1d5:::benchmark_data_smithi188_89276_object106177:head# 2024-04-23T16:25:14.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2998d778:::benchmark_data_smithi188_89276_object57020:head# 2024-04-23T16:25:14.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:299a2c27:::benchmark_data_smithi188_89276_object106261:head# 2024-04-23T16:25:14.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:299a6f5b:::benchmark_data_smithi188_89276_object51817:head# 2024-04-23T16:25:14.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:299ad5f7:::benchmark_data_smithi188_89276_object156218:head# 2024-04-23T16:25:14.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:299b4b23:::benchmark_data_smithi188_89276_object46686:head# 2024-04-23T16:25:14.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:299c011c:::benchmark_data_smithi188_89276_object138866:head# 2024-04-23T16:25:14.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:299c6215:::benchmark_data_smithi188_89276_object153336:head# 2024-04-23T16:25:14.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:299d19af:::benchmark_data_smithi188_89276_object88747:head# 2024-04-23T16:25:14.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:299d54f2:::benchmark_data_smithi188_89276_object168057:head# 2024-04-23T16:25:14.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:299d69e5:::benchmark_data_smithi188_89276_object151418:head# 2024-04-23T16:25:14.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:299de20a:::benchmark_data_smithi188_89276_object70846:head# 2024-04-23T16:25:14.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:299e6dc8:::benchmark_data_smithi188_89276_object116013:head# 2024-04-23T16:25:14.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:299e98b4:::benchmark_data_smithi188_89276_object141423:head# 2024-04-23T16:25:14.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:299eb110:::benchmark_data_smithi188_89276_object66292:head# 2024-04-23T16:25:14.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:299ffc21:::benchmark_data_smithi188_89276_object114131:head# 2024-04-23T16:25:14.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29a0ccaf:::benchmark_data_smithi188_89276_object54013:head# 2024-04-23T16:25:14.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29a14956:::benchmark_data_smithi188_89276_object55201:head# 2024-04-23T16:25:14.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29a1527a:::benchmark_data_smithi188_89276_object147668:head# 2024-04-23T16:25:14.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29a16fc7:::benchmark_data_smithi188_89276_object127520:head# 2024-04-23T16:25:14.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29a28d57:::benchmark_data_smithi188_89276_object43309:head# 2024-04-23T16:25:14.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29a2d798:::benchmark_data_smithi188_89276_object145153:head# 2024-04-23T16:25:14.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29a2f766:::benchmark_data_smithi188_89276_object108922:head# 2024-04-23T16:25:14.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29a3cd96:::benchmark_data_smithi188_89276_object158227:head# 2024-04-23T16:25:14.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29a64029:::benchmark_data_smithi188_89276_object158424:head# 2024-04-23T16:25:14.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29a64e60:::benchmark_data_smithi188_89276_object147068:head# 2024-04-23T16:25:14.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29a69bca:::benchmark_data_smithi188_89276_object67518:head# 2024-04-23T16:25:14.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29a718c1:::benchmark_data_smithi188_89276_object156805:head# 2024-04-23T16:25:14.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29a78fea:::benchmark_data_smithi188_89276_object77491:head# 2024-04-23T16:25:14.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29a7932d:::benchmark_data_smithi188_89276_object126164:head# 2024-04-23T16:25:14.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29a7b096:::benchmark_data_smithi188_89276_object71894:head# 2024-04-23T16:25:14.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29a8ba1e:::benchmark_data_smithi188_89276_object152890:head# 2024-04-23T16:25:14.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29a904bd:::benchmark_data_smithi188_89276_object51657:head# 2024-04-23T16:25:14.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29a9c9be:::benchmark_data_smithi188_89276_object61668:head# 2024-04-23T16:25:14.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29ac23c7:::benchmark_data_smithi188_89276_object165917:head# 2024-04-23T16:25:14.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29ad42cc:::benchmark_data_smithi188_89276_object56028:head# 2024-04-23T16:25:14.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29ad9c1c:::benchmark_data_smithi188_89276_object118202:head# 2024-04-23T16:25:14.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29adc657:::benchmark_data_smithi188_89276_object84579:head# 2024-04-23T16:25:14.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29ae0489:::benchmark_data_smithi188_89276_object82498:head# 2024-04-23T16:25:14.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29ae10c2:::benchmark_data_smithi188_89276_object148209:head# 2024-04-23T16:25:14.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29ae6763:::benchmark_data_smithi188_89276_object158320:head# 2024-04-23T16:25:14.867 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:25:14.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29aef720:::benchmark_data_smithi188_89276_object59529:head# 2024-04-23T16:25:14.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29af186d:::benchmark_data_smithi188_89276_object108014:head# 2024-04-23T16:25:14.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29af4201:::benchmark_data_smithi188_89276_object113122:head# 2024-04-23T16:25:14.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29aff920:::benchmark_data_smithi188_89276_object80290:head# 2024-04-23T16:25:14.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29b04d5f:::benchmark_data_smithi188_89276_object113442:head# 2024-04-23T16:25:14.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29b076b5:::benchmark_data_smithi188_89276_object107938:head# 2024-04-23T16:25:14.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29b0c8fc:::benchmark_data_smithi188_89276_object166627:head# 2024-04-23T16:25:14.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29b14607:::benchmark_data_smithi188_89276_object45959:head# 2024-04-23T16:25:14.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29b1d498:::benchmark_data_smithi188_89276_object90342:head# 2024-04-23T16:25:14.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29b2494e:::benchmark_data_smithi188_89276_object148342:head# 2024-04-23T16:25:14.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29b2629f:::benchmark_data_smithi188_89276_object93875:head# 2024-04-23T16:25:14.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29b267f0:::benchmark_data_smithi188_89276_object45878:head# 2024-04-23T16:25:14.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29b26d3c:::benchmark_data_smithi188_89276_object112228:head# 2024-04-23T16:25:14.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29b294d1:::benchmark_data_smithi188_89276_object93585:head# 2024-04-23T16:25:14.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29b456d6:::benchmark_data_smithi188_89276_object146729:head# 2024-04-23T16:25:14.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29b4d99e:::benchmark_data_smithi188_89276_object79382:head# 2024-04-23T16:25:14.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29b4e093:::benchmark_data_smithi188_89276_object55839:head# 2024-04-23T16:25:14.875 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29b68450:::benchmark_data_smithi188_89276_object121802:head# 2024-04-23T16:25:14.876 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29b6c954:::benchmark_data_smithi188_89276_object153659:head# 2024-04-23T16:25:14.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29b717d7:::benchmark_data_smithi188_89276_object62052:head# 2024-04-23T16:25:14.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29b77497:::benchmark_data_smithi188_89276_object159782:head# 2024-04-23T16:25:14.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29b7cf22:::benchmark_data_smithi188_89276_object97707:head# 2024-04-23T16:25:14.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29b86b14:::benchmark_data_smithi188_89276_object148617:head# 2024-04-23T16:25:14.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29bb8ebe:::benchmark_data_smithi188_89276_object51331:head# 2024-04-23T16:25:14.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29bba290:::benchmark_data_smithi188_89276_object138055:head# 2024-04-23T16:25:14.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29bbc07e:::benchmark_data_smithi188_89276_object40730:head# 2024-04-23T16:25:14.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29bc00dd:::benchmark_data_smithi188_89276_object63237:head# 2024-04-23T16:25:14.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29bd8ceb:::benchmark_data_smithi188_89276_object73738:head# 2024-04-23T16:25:14.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29bdcab3:::benchmark_data_smithi188_89276_object86766:head# 2024-04-23T16:25:14.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29bf88ae:::benchmark_data_smithi188_89276_object58951:head# 2024-04-23T16:25:14.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29bfa320:::benchmark_data_smithi188_89276_object144000:head# 2024-04-23T16:25:14.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29c0f4a9:::benchmark_data_smithi188_89276_object156616:head# 2024-04-23T16:25:14.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29c18170:::benchmark_data_smithi188_89276_object110053:head# 2024-04-23T16:25:14.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29c2d123:::benchmark_data_smithi188_89276_object160690:head# 2024-04-23T16:25:14.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29c39562:::benchmark_data_smithi188_89276_object140204:head# 2024-04-23T16:25:14.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29c560a4:::benchmark_data_smithi188_89276_object117815:head# 2024-04-23T16:25:14.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29c5f6fe:::benchmark_data_smithi188_89276_object116575:head# 2024-04-23T16:25:14.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29c7916b:::benchmark_data_smithi188_89276_object83611:head# 2024-04-23T16:25:14.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29c84f01:::benchmark_data_smithi188_89276_object150857:head# 2024-04-23T16:25:14.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29c89ca6:::benchmark_data_smithi188_89276_object63530:head# 2024-04-23T16:25:14.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29c8b0e0:::benchmark_data_smithi188_89276_object91622:head# 2024-04-23T16:25:14.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29c9437c:::benchmark_data_smithi188_89276_object125512:head# 2024-04-23T16:25:14.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29c98598:::benchmark_data_smithi188_89276_object47434:head# 2024-04-23T16:25:14.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29c9ad1d:::benchmark_data_smithi188_89276_object144240:head# 2024-04-23T16:25:14.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29c9aef5:::benchmark_data_smithi188_89276_object160221:head# 2024-04-23T16:25:14.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29c9f978:::benchmark_data_smithi188_89276_object81140:head# 2024-04-23T16:25:14.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29ca7d7d:::benchmark_data_smithi188_89276_object47475:head# 2024-04-23T16:25:14.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29cabae7:::benchmark_data_smithi188_89276_object56824:head# 2024-04-23T16:25:14.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29caf406:::benchmark_data_smithi188_89276_object98306:head# 2024-04-23T16:25:14.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29cb45eb:::benchmark_data_smithi188_89276_object156468:head# 2024-04-23T16:25:14.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29cb5ba2:::benchmark_data_smithi188_89276_object61833:head# 2024-04-23T16:25:14.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29cc09f2:::benchmark_data_smithi188_89276_object51225:head# 2024-04-23T16:25:14.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29cc22f0:::benchmark_data_smithi188_89276_object53053:head# 2024-04-23T16:25:14.891 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29cd6a17:::benchmark_data_smithi188_89276_object150416:head# 2024-04-23T16:25:14.891 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29cdca95:::benchmark_data_smithi188_89276_object72699:head# 2024-04-23T16:25:14.891 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29cdf311:::benchmark_data_smithi188_89276_object81449:head# 2024-04-23T16:25:14.892 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29cea70d:::benchmark_data_smithi188_89276_object159500:head# 2024-04-23T16:25:14.892 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29d0086c:::benchmark_data_smithi188_89276_object149191:head# 2024-04-23T16:25:14.892 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29d02988:::benchmark_data_smithi188_89276_object90906:head# 2024-04-23T16:25:14.893 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29d0fec3:::benchmark_data_smithi188_89276_object138078:head# 2024-04-23T16:25:14.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29d1f316:::benchmark_data_smithi188_89276_object44535:head# 2024-04-23T16:25:14.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29d3bd56:::benchmark_data_smithi188_89276_object138022:head# 2024-04-23T16:25:14.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29d4816b:::benchmark_data_smithi188_89276_object52234:head# 2024-04-23T16:25:14.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29d4dc78:::benchmark_data_smithi188_89276_object161026:head# 2024-04-23T16:25:14.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29d6c8ca:::benchmark_data_smithi188_89276_object96156:head# 2024-04-23T16:25:14.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29d775aa:::benchmark_data_smithi188_89276_object128863:head# 2024-04-23T16:25:14.896 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29d78763:::benchmark_data_smithi188_89276_object168236:head# 2024-04-23T16:25:14.896 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29d7c852:::benchmark_data_smithi188_89276_object64772:head# 2024-04-23T16:25:14.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29d8484c:::benchmark_data_smithi188_89276_object134731:head# 2024-04-23T16:25:14.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29d94399:::benchmark_data_smithi188_89276_object157230:head# 2024-04-23T16:25:14.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29d998b6:::benchmark_data_smithi188_89276_object112130:head# 2024-04-23T16:25:14.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29d9d7da:::benchmark_data_smithi188_89276_object129264:head# 2024-04-23T16:25:14.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29db1d2e:::benchmark_data_smithi188_89276_object45202:head# 2024-04-23T16:25:14.900 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:14.900 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29db569c:::benchmark_data_smithi188_89276_object156553:head# 2024-04-23T16:25:14.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29dc24cc:::benchmark_data_smithi188_89276_object91702:head# 2024-04-23T16:25:14.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29dc7d54:::benchmark_data_smithi188_89276_object110577:head# 2024-04-23T16:25:14.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29dc8f70:::benchmark_data_smithi188_89276_object150277:head# 2024-04-23T16:25:14.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29de08ca:::benchmark_data_smithi188_89276_object48703:head# 2024-04-23T16:25:14.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29de5d98:::benchmark_data_smithi188_89276_object105141:head# 2024-04-23T16:25:14.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29e01b55:::benchmark_data_smithi188_89276_object73941:head# 2024-04-23T16:25:14.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29e0ac12:::benchmark_data_smithi188_89276_object163496:head# 2024-04-23T16:25:14.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29e16f0b:::benchmark_data_smithi188_89276_object159092:head# 2024-04-23T16:25:14.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29e1866a:::benchmark_data_smithi188_89276_object163817:head# 2024-04-23T16:25:14.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29e21cef:::benchmark_data_smithi188_89276_object87385:head# 2024-04-23T16:25:14.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29e30e45:::benchmark_data_smithi188_89276_object90943:head# 2024-04-23T16:25:14.904 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29e33f8d:::benchmark_data_smithi188_89276_object73214:head# 2024-04-23T16:25:14.904 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29e364ca:::benchmark_data_smithi188_89276_object122237:head# 2024-04-23T16:25:14.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29e521ad:::benchmark_data_smithi188_89276_object47989:head# 2024-04-23T16:25:14.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29e55ba0:::benchmark_data_smithi188_89276_object113071:head# 2024-04-23T16:25:14.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29e60ee2:::benchmark_data_smithi188_89276_object142509:head# 2024-04-23T16:25:14.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29e759e4:::benchmark_data_smithi188_89276_object69237:head# 2024-04-23T16:25:14.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29e78b03:::benchmark_data_smithi188_89276_object64317:head# 2024-04-23T16:25:14.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29e78cc6:::benchmark_data_smithi188_89276_object143657:head# 2024-04-23T16:25:14.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29e8516c:::benchmark_data_smithi188_89276_object158925:head# 2024-04-23T16:25:14.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29e861c1:::benchmark_data_smithi188_89276_object140677:head# 2024-04-23T16:25:14.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29e8ede3:::benchmark_data_smithi188_89276_object116329:head# 2024-04-23T16:25:14.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29e92b60:::benchmark_data_smithi188_89276_object94987:head# 2024-04-23T16:25:14.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29ea0d79:::benchmark_data_smithi188_89276_object122703:head# 2024-04-23T16:25:14.909 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29ea43f5:::benchmark_data_smithi188_89276_object73017:head# 2024-04-23T16:25:14.909 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29ea4741:::benchmark_data_smithi188_89276_object88024:head# 2024-04-23T16:25:14.910 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29ea5c77:::benchmark_data_smithi188_89276_object125463:head# 2024-04-23T16:25:14.910 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29eb198f:::benchmark_data_smithi188_89276_object71910:head# 2024-04-23T16:25:14.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29eba8a7:::benchmark_data_smithi188_89276_object101394:head# 2024-04-23T16:25:14.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29ec1963:::benchmark_data_smithi188_89276_object103003:head# 2024-04-23T16:25:14.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29ec55e7:::benchmark_data_smithi188_89276_object152963:head# 2024-04-23T16:25:14.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29ec71c3:::benchmark_data_smithi188_89276_object130264:head# 2024-04-23T16:25:14.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29ed0331:::benchmark_data_smithi188_89276_object124586:head# 2024-04-23T16:25:14.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29ed5250:::benchmark_data_smithi188_89276_object122873:head# 2024-04-23T16:25:14.914 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29edcc51:::benchmark_data_smithi188_89276_object114701:head# 2024-04-23T16:25:14.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29ee7259:::benchmark_data_smithi188_89276_object47610:head# 2024-04-23T16:25:14.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29eeb24d:::benchmark_data_smithi188_89276_object95838:head# 2024-04-23T16:25:14.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29ef2e07:::benchmark_data_smithi188_89276_object100672:head# 2024-04-23T16:25:14.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f0714e:::benchmark_data_smithi188_89276_object82234:head# 2024-04-23T16:25:14.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f114a7:::benchmark_data_smithi188_89276_object91542:head# 2024-04-23T16:25:14.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f15182:::benchmark_data_smithi188_89276_object53137:head# 2024-04-23T16:25:14.917 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f1c781:::benchmark_data_smithi188_89276_object157297:head# 2024-04-23T16:25:14.917 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f1f117:::benchmark_data_smithi188_89276_object103176:head# 2024-04-23T16:25:14.917 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f337a4:::benchmark_data_smithi188_89276_object93754:head# 2024-04-23T16:25:14.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f4762e:::benchmark_data_smithi188_89276_object116733:head# 2024-04-23T16:25:14.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f4cb03:::benchmark_data_smithi188_89276_object54306:head# 2024-04-23T16:25:14.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f4e1cc:::benchmark_data_smithi188_89276_object103261:head# 2024-04-23T16:25:14.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f50dd6:::benchmark_data_smithi188_89276_object53436:head# 2024-04-23T16:25:14.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f51a3b:::benchmark_data_smithi188_89276_object88980:head# 2024-04-23T16:25:14.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f52a99:::benchmark_data_smithi188_89276_object60115:head# 2024-04-23T16:25:14.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f59088:::benchmark_data_smithi188_89276_object157703:head# 2024-04-23T16:25:14.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f65a48:::benchmark_data_smithi188_89276_object99224:head# 2024-04-23T16:25:14.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f70be0:::benchmark_data_smithi188_89276_object102913:head# 2024-04-23T16:25:14.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f72554:::benchmark_data_smithi188_89276_object58961:head# 2024-04-23T16:25:14.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f7466f:::benchmark_data_smithi188_89276_object49072:head# 2024-04-23T16:25:14.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f77616:::benchmark_data_smithi188_89276_object98544:head# 2024-04-23T16:25:14.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f7e911:::benchmark_data_smithi188_89276_object80900:head# 2024-04-23T16:25:14.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f8923a:::benchmark_data_smithi188_89276_object114966:head# 2024-04-23T16:25:14.925 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:25:14.925 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f8d836:::benchmark_data_smithi188_89276_object122695:head# 2024-04-23T16:25:14.925 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29f9fdd6:::benchmark_data_smithi188_89276_object114497:head# 2024-04-23T16:25:14.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29fa6681:::benchmark_data_smithi188_89276_object96420:head# 2024-04-23T16:25:14.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29fa8fa6:::benchmark_data_smithi188_89276_object57328:head# 2024-04-23T16:25:14.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29fadfae:::benchmark_data_smithi188_89276_object167108:head# 2024-04-23T16:25:14.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29fb42e5:::benchmark_data_smithi188_89276_object83362:head# 2024-04-23T16:25:14.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29fc2486:::benchmark_data_smithi188_89276_object68011:head# 2024-04-23T16:25:14.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29fc3e02:::benchmark_data_smithi188_89276_object60262:head# 2024-04-23T16:25:14.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29fd5035:::benchmark_data_smithi188_89276_object42973:head# 2024-04-23T16:25:14.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:14.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:25:14.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29fd8a5d:::benchmark_data_smithi188_89276_object82327:head# 2024-04-23T16:25:14.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29fdbbc1:::benchmark_data_smithi188_89276_object149152:head# 2024-04-23T16:25:14.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29fe1c42:::benchmark_data_smithi188_89276_object61370:head# 2024-04-23T16:25:14.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29fe553e:::benchmark_data_smithi188_89276_object120485:head# 2024-04-23T16:25:14.930 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29fee977:::benchmark_data_smithi188_89276_object74039:head# 2024-04-23T16:25:14.930 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:29ff6b74:::benchmark_data_smithi188_89276_object81520:head# 2024-04-23T16:25:14.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a0006ed:::benchmark_data_smithi188_89276_object154491:head# 2024-04-23T16:25:14.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a006d5a:::benchmark_data_smithi188_89276_object84340:head# 2024-04-23T16:25:14.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a00c3bd:::benchmark_data_smithi188_89276_object73824:head# 2024-04-23T16:25:14.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a0121bd:::benchmark_data_smithi188_89276_object77739:head# 2024-04-23T16:25:14.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a01d23d:::benchmark_data_smithi188_89276_object96336:head# 2024-04-23T16:25:14.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a028348:::benchmark_data_smithi188_89276_object58037:head# 2024-04-23T16:25:14.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a0302ee:::benchmark_data_smithi188_89276_object94079:head# 2024-04-23T16:25:14.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a031342:::benchmark_data_smithi188_89276_object81608:head# 2024-04-23T16:25:14.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a036959:::benchmark_data_smithi188_89276_object54296:head# 2024-04-23T16:25:14.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a0373e6:::benchmark_data_smithi188_89276_object97893:head# 2024-04-23T16:25:14.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a0395e3:::benchmark_data_smithi188_89276_object99248:head# 2024-04-23T16:25:14.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a044d2d:::benchmark_data_smithi188_89276_object114472:head# 2024-04-23T16:25:14.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a04549d:::benchmark_data_smithi188_89276_object63382:head# 2024-04-23T16:25:14.939 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a048773:::benchmark_data_smithi188_89276_object65132:head# 2024-04-23T16:25:14.939 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a04a042:::benchmark_data_smithi188_89276_object138501:head# 2024-04-23T16:25:14.940 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a05124b:::benchmark_data_smithi188_89276_object44325:head# 2024-04-23T16:25:14.940 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a0519d1:::benchmark_data_smithi188_89276_object136802:head# 2024-04-23T16:25:14.940 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a082799:::benchmark_data_smithi188_89276_object50012:head# 2024-04-23T16:25:14.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a0aaace:::benchmark_data_smithi188_89276_object114580:head# 2024-04-23T16:25:14.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a0afb69:::benchmark_data_smithi188_89276_object160278:head# 2024-04-23T16:25:14.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a0b8066:::benchmark_data_smithi188_89276_object110344:head# 2024-04-23T16:25:14.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a0bd82e:::benchmark_data_smithi188_89276_object85206:head# 2024-04-23T16:25:14.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a0c456b:::benchmark_data_smithi188_89276_object92138:head# 2024-04-23T16:25:14.943 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a0d60e6:::benchmark_data_smithi188_89276_object136416:head# 2024-04-23T16:25:14.943 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a0dd649:::benchmark_data_smithi188_89276_object153322:head# 2024-04-23T16:25:14.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a0e7104:::benchmark_data_smithi188_89276_object147323:head# 2024-04-23T16:25:14.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a0f9db6:::benchmark_data_smithi188_89276_object46852:head# 2024-04-23T16:25:14.945 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a0fdae1:::benchmark_data_smithi188_89276_object107771:head# 2024-04-23T16:25:14.946 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a1086eb:::benchmark_data_smithi188_89276_object117040:head# 2024-04-23T16:25:14.946 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a126432:::benchmark_data_smithi188_89276_object64766:head# 2024-04-23T16:25:14.947 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a139476:::benchmark_data_smithi188_89276_object141799:head# 2024-04-23T16:25:14.947 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a13f708:::benchmark_data_smithi188_89276_object108806:head# 2024-04-23T16:25:14.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a14378d:::benchmark_data_smithi188_89276_object81473:head# 2024-04-23T16:25:14.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a149da2:::benchmark_data_smithi188_89276_object44159:head# 2024-04-23T16:25:14.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a156190:::benchmark_data_smithi188_89276_object52607:head# 2024-04-23T16:25:14.949 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a15db99:::benchmark_data_smithi188_89276_object42521:head# 2024-04-23T16:25:14.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a16a7d1:::benchmark_data_smithi188_89276_object166026:head# 2024-04-23T16:25:14.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a175a89:::benchmark_data_smithi188_89276_object124566:head# 2024-04-23T16:25:14.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a17e2cb:::benchmark_data_smithi188_89276_object111480:head# 2024-04-23T16:25:14.953 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a187557:::benchmark_data_smithi188_89276_object143049:head# 2024-04-23T16:25:14.953 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a18b85c:::benchmark_data_smithi188_89276_object57229:head# 2024-04-23T16:25:14.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a19061d:::benchmark_data_smithi188_89276_object71320:head# 2024-04-23T16:25:14.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a190d13:::benchmark_data_smithi188_89276_object80968:head# 2024-04-23T16:25:14.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a19faee:::benchmark_data_smithi188_89276_object134345:head# 2024-04-23T16:25:14.955 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a1a9c97:::benchmark_data_smithi188_89276_object126579:head# 2024-04-23T16:25:14.956 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a1ac38b:::benchmark_data_smithi188_89276_object47546:head# 2024-04-23T16:25:14.956 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a1ac560:::benchmark_data_smithi188_89276_object59442:head# 2024-04-23T16:25:14.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a1ae65b:::benchmark_data_smithi188_89276_object165876:head# 2024-04-23T16:25:14.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a1b1671:::benchmark_data_smithi188_89276_object55349:head# 2024-04-23T16:25:14.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a1b3fc5:::benchmark_data_smithi188_89276_object144602:head# 2024-04-23T16:25:14.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a1b41c3:::benchmark_data_smithi188_89276_object42071:head# 2024-04-23T16:25:14.960 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a1b7f27:::benchmark_data_smithi188_89276_object127901:head# 2024-04-23T16:25:14.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a1bde30:::benchmark_data_smithi188_89276_object88500:head# 2024-04-23T16:25:14.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a1be85d:::benchmark_data_smithi188_89276_object165529:head# 2024-04-23T16:25:14.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a1cafa8:::benchmark_data_smithi188_89276_object125315:head# 2024-04-23T16:25:14.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a1d70fc:::benchmark_data_smithi188_89276_object158111:head# 2024-04-23T16:25:14.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a1dd6bc:::benchmark_data_smithi188_89276_object102844:head# 2024-04-23T16:25:14.965 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a1e27c6:::benchmark_data_smithi188_89276_object103206:head# 2024-04-23T16:25:14.965 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a1e5be5:::benchmark_data_smithi188_89276_object140226:head# 2024-04-23T16:25:14.965 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a1f03c4:::benchmark_data_smithi188_89276_object127550:head# 2024-04-23T16:25:14.965 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a1f8dbe:::benchmark_data_smithi188_89276_object79416:head# 2024-04-23T16:25:14.965 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a2002cf:::benchmark_data_smithi188_89276_object99762:head# 2024-04-23T16:25:14.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a207d65:::benchmark_data_smithi188_89276_object121956:head# 2024-04-23T16:25:14.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a20c5a7:::benchmark_data_smithi188_89276_object127171:head# 2024-04-23T16:25:14.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a220c53:::benchmark_data_smithi188_89276_object53414:head# 2024-04-23T16:25:14.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a225209:::benchmark_data_smithi188_89276_object117427:head# 2024-04-23T16:25:14.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a2299f2:::benchmark_data_smithi188_89276_object138473:head# 2024-04-23T16:25:14.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a230c91:::benchmark_data_smithi188_89276_object154383:head# 2024-04-23T16:25:14.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a231baf:::benchmark_data_smithi188_89276_object136429:head# 2024-04-23T16:25:14.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a24962e:::benchmark_data_smithi188_89276_object115306:head# 2024-04-23T16:25:14.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a25a385:::benchmark_data_smithi188_89276_object162880:head# 2024-04-23T16:25:14.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a25c6b2:::benchmark_data_smithi188_89276_object116732:head# 2024-04-23T16:25:14.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a2666c5:::benchmark_data_smithi188_89276_object61348:head# 2024-04-23T16:25:14.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a26a942:::benchmark_data_smithi188_89276_object130310:head# 2024-04-23T16:25:14.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a26e871:::benchmark_data_smithi188_89276_object139046:head# 2024-04-23T16:25:14.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a27026e:::benchmark_data_smithi188_89276_object164536:head# 2024-04-23T16:25:14.977 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a273a4c:::benchmark_data_smithi188_89276_object70543:head# 2024-04-23T16:25:14.977 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a28a086:::benchmark_data_smithi188_89276_object122188:head# 2024-04-23T16:25:14.977 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a2a7af0:::benchmark_data_smithi188_89276_object87032:head# 2024-04-23T16:25:14.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a2abc3a:::benchmark_data_smithi188_89276_object118641:head# 2024-04-23T16:25:14.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a2b4119:::benchmark_data_smithi188_89276_object105260:head# 2024-04-23T16:25:14.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a2b4bb2:::benchmark_data_smithi188_89276_object129143:head# 2024-04-23T16:25:14.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a2b954e:::benchmark_data_smithi188_89276_object81026:head# 2024-04-23T16:25:14.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a2bee7e:::benchmark_data_smithi188_89276_object49390:head# 2024-04-23T16:25:14.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a2c01ad:::benchmark_data_smithi188_89276_object108249:head# 2024-04-23T16:25:14.980 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a2cdb76:::benchmark_data_smithi188_89276_object71911:head# 2024-04-23T16:25:14.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a2e4ede:::benchmark_data_smithi188_89276_object49258:head# 2024-04-23T16:25:14.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a2e6ef9:::benchmark_data_smithi188_89276_object141546:head# 2024-04-23T16:25:14.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a2eb7c7:::benchmark_data_smithi188_89276_object78858:head# 2024-04-23T16:25:14.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a2ef378:::benchmark_data_smithi188_89276_object53775:head# 2024-04-23T16:25:14.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a309151:::benchmark_data_smithi188_89276_object69829:head# 2024-04-23T16:25:14.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a3136e1:::benchmark_data_smithi188_89276_object72152:head# 2024-04-23T16:25:14.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a3366df:::benchmark_data_smithi188_89276_object87152:head# 2024-04-23T16:25:14.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a33c79a:::benchmark_data_smithi188_89276_object128842:head# 2024-04-23T16:25:14.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a34afe4:::benchmark_data_smithi188_89276_object108412:head# 2024-04-23T16:25:14.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a3599f5:::benchmark_data_smithi188_89276_object116739:head# 2024-04-23T16:25:14.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a365788:::benchmark_data_smithi188_89276_object129091:head# 2024-04-23T16:25:14.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a3673cb:::benchmark_data_smithi188_89276_object46542:head# 2024-04-23T16:25:14.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a3735b0:::benchmark_data_smithi188_89276_object63031:head# 2024-04-23T16:25:14.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a374140:::benchmark_data_smithi188_89276_object63762:head# 2024-04-23T16:25:14.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a3749a9:::benchmark_data_smithi188_89276_object79469:head# 2024-04-23T16:25:14.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a3a0767:::benchmark_data_smithi188_89276_object43447:head# 2024-04-23T16:25:14.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a3b0e78:::benchmark_data_smithi188_89276_object85676:head# 2024-04-23T16:25:14.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a3b5090:::benchmark_data_smithi188_89276_object109566:head# 2024-04-23T16:25:14.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a3b56ad:::benchmark_data_smithi188_89276_object136164:head# 2024-04-23T16:25:14.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:14 smithi103 ceph-mon[39629]: pgmap v716: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 445 op/s; 15 MiB/s, 239 objects/s recovering 2024-04-23T16:25:14.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a3da6f2:::benchmark_data_smithi188_89276_object90899:head# 2024-04-23T16:25:14.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a3e898c:::benchmark_data_smithi188_89276_object78101:head# 2024-04-23T16:25:14.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a3ecb1f:::benchmark_data_smithi188_89276_object162395:head# 2024-04-23T16:25:14.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a3f345c:::benchmark_data_smithi188_89276_object51239:head# 2024-04-23T16:25:14.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a3f80d1:::benchmark_data_smithi188_89276_object130097:head# 2024-04-23T16:25:14.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a3f8db9:::benchmark_data_smithi188_89276_object46209:head# 2024-04-23T16:25:14.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a400e12:::benchmark_data_smithi188_89276_object84503:head# 2024-04-23T16:25:14.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4113cc:::benchmark_data_smithi188_89276_object134463:head# 2024-04-23T16:25:14.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a41e523:::benchmark_data_smithi188_89276_object166099:head# 2024-04-23T16:25:14.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4269fb:::benchmark_data_smithi188_89276_object146261:head# 2024-04-23T16:25:14.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a429ae5:::benchmark_data_smithi188_89276_object86893:head# 2024-04-23T16:25:14.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a432c3c:::benchmark_data_smithi188_89276_object99686:head# 2024-04-23T16:25:14.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a435955:::benchmark_data_smithi188_89276_object112321:head# 2024-04-23T16:25:14.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4371e6:::benchmark_data_smithi188_89276_object77431:head# 2024-04-23T16:25:14.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a43b96e:::benchmark_data_smithi188_89276_object165175:head# 2024-04-23T16:25:14.998 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a442aee:::benchmark_data_smithi188_89276_object104881:head# 2024-04-23T16:25:14.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4479f3:::benchmark_data_smithi188_89276_object134434:head# 2024-04-23T16:25:14.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a44866c:::benchmark_data_smithi188_89276_object118639:head# 2024-04-23T16:25:14.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a44bdd1:::benchmark_data_smithi188_89276_object128502:head# 2024-04-23T16:25:14.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4635cc:::benchmark_data_smithi188_89276_object111984:head# 2024-04-23T16:25:14.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a47abb9:::benchmark_data_smithi188_89276_object148296:head# 2024-04-23T16:25:14.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a47b6d6:::benchmark_data_smithi188_89276_object149913:head# 2024-04-23T16:25:14.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a47bc44:::benchmark_data_smithi188_89276_object125817:head# 2024-04-23T16:25:14.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a47e0af:::benchmark_data_smithi188_89276_object59112:head# 2024-04-23T16:25:15.003 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4863aa:::benchmark_data_smithi188_89276_object41059:head# 2024-04-23T16:25:15.003 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4864d3:::benchmark_data_smithi188_89276_object86323:head# 2024-04-23T16:25:15.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a490f50:::benchmark_data_smithi188_89276_object102151:head# 2024-04-23T16:25:15.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4926ad:::benchmark_data_smithi188_89276_object148988:head# 2024-04-23T16:25:15.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a496a49:::benchmark_data_smithi188_89276_object67219:head# 2024-04-23T16:25:15.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4a048b:::benchmark_data_smithi188_89276_object59228:head# 2024-04-23T16:25:15.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4a2eeb:::benchmark_data_smithi188_89276_object46900:head# 2024-04-23T16:25:15.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4a6b61:::benchmark_data_smithi188_89276_object72796:head# 2024-04-23T16:25:15.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4b84cb:::benchmark_data_smithi188_89276_object125960:head# 2024-04-23T16:25:15.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4b93c5:::benchmark_data_smithi188_89276_object115896:head# 2024-04-23T16:25:15.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4d7f83:::benchmark_data_smithi188_89276_object69287:head# 2024-04-23T16:25:15.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4dd4fe:::benchmark_data_smithi188_89276_object146963:head# 2024-04-23T16:25:15.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4e0436:::benchmark_data_smithi188_89276_object75749:head# 2024-04-23T16:25:15.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4e0579:::benchmark_data_smithi188_89276_object129752:head# 2024-04-23T16:25:15.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4e13b3:::benchmark_data_smithi188_89276_object54286:head# 2024-04-23T16:25:15.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4e5a84:::benchmark_data_smithi188_89276_object62090:head# 2024-04-23T16:25:15.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a4ead62:::benchmark_data_smithi188_89276_object101118:head# 2024-04-23T16:25:15.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a5060f0:::benchmark_data_smithi188_89276_object85847:head# 2024-04-23T16:25:15.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a50e324:::benchmark_data_smithi188_89276_object108392:head# 2024-04-23T16:25:15.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a50ec09:::benchmark_data_smithi188_89276_object88556:head# 2024-04-23T16:25:15.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a50eee7:::benchmark_data_smithi188_89276_object166376:head# 2024-04-23T16:25:15.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a512c08:::benchmark_data_smithi188_89276_object100134:head# 2024-04-23T16:25:15.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a5149d0:::benchmark_data_smithi188_89276_object144149:head# 2024-04-23T16:25:15.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a517956:::benchmark_data_smithi188_89276_object43341:head# 2024-04-23T16:25:15.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a519992:::benchmark_data_smithi188_89276_object132876:head# 2024-04-23T16:25:15.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a5237bd:::benchmark_data_smithi188_89276_object144955:head# 2024-04-23T16:25:15.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a52cc75:::benchmark_data_smithi188_89276_object112895:head# 2024-04-23T16:25:15.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a53057c:::benchmark_data_smithi188_89276_object162658:head# 2024-04-23T16:25:15.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a53647f:::benchmark_data_smithi188_89276_object87963:head# 2024-04-23T16:25:15.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a536ae5:::benchmark_data_smithi188_89276_object75475:head# 2024-04-23T16:25:15.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a549367:::benchmark_data_smithi188_89276_object86929:head# 2024-04-23T16:25:15.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a54c953:::benchmark_data_smithi188_89276_object120004:head# 2024-04-23T16:25:15.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a5508e6:::benchmark_data_smithi188_89276_object76105:head# 2024-04-23T16:25:15.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a560874:::benchmark_data_smithi188_89276_object63745:head# 2024-04-23T16:25:15.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a56b493:::benchmark_data_smithi188_89276_object132496:head# 2024-04-23T16:25:15.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a576fb6:::benchmark_data_smithi188_89276_object61629:head# 2024-04-23T16:25:15.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a582062:::benchmark_data_smithi188_89276_object97717:head# 2024-04-23T16:25:15.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a596769:::benchmark_data_smithi188_89276_object107192:head# 2024-04-23T16:25:15.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a59f7d5:::benchmark_data_smithi188_89276_object88342:head# 2024-04-23T16:25:15.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a5a1c61:::benchmark_data_smithi188_89276_object54363:head# 2024-04-23T16:25:15.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a5b7793:::benchmark_data_smithi188_89276_object149472:head# 2024-04-23T16:25:15.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a5b7f0d:::benchmark_data_smithi188_89276_object52808:head# 2024-04-23T16:25:15.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a5b8a03:::benchmark_data_smithi188_89276_object121521:head# 2024-04-23T16:25:15.019 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a5b8dee:::benchmark_data_smithi188_89276_object77087:head# 2024-04-23T16:25:15.020 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a5be0fb:::benchmark_data_smithi188_89276_object88568:head# 2024-04-23T16:25:15.020 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a5cb60f:::benchmark_data_smithi188_89276_object139649:head# 2024-04-23T16:25:15.020 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a5cc119:::benchmark_data_smithi188_89276_object168596:head# 2024-04-23T16:25:15.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a5cdc13:::benchmark_data_smithi188_89276_object115498:head# 2024-04-23T16:25:15.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a5d4058:::benchmark_data_smithi188_89276_object104274:head# 2024-04-23T16:25:15.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a5d81ff:::benchmark_data_smithi188_89276_object99639:head# 2024-04-23T16:25:15.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a5dddbf:::benchmark_data_smithi188_89276_object74500:head# 2024-04-23T16:25:15.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a5f54bb:::benchmark_data_smithi188_89276_object51563:head# 2024-04-23T16:25:15.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a6197aa:::benchmark_data_smithi188_89276_object128311:head# 2024-04-23T16:25:15.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a620450:::benchmark_data_smithi188_89276_object66277:head# 2024-04-23T16:25:15.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a62ffb1:::benchmark_data_smithi188_89276_object41792:head# 2024-04-23T16:25:15.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a63abee:::benchmark_data_smithi188_89276_object48960:head# 2024-04-23T16:25:15.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a63b1e1:::benchmark_data_smithi188_89276_object99410:head# 2024-04-23T16:25:15.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a649b73:::benchmark_data_smithi188_89276_object101199:head# 2024-04-23T16:25:15.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a663798:::benchmark_data_smithi188_89276_object93588:head# 2024-04-23T16:25:15.029 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:25:15.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a67b9b7:::benchmark_data_smithi188_89276_object73495:head# 2024-04-23T16:25:15.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a67d854:::benchmark_data_smithi188_89276_object81941:head# 2024-04-23T16:25:15.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a67e36d:::benchmark_data_smithi188_89276_object48447:head# 2024-04-23T16:25:15.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a68d693:::benchmark_data_smithi188_89276_object81468:head# 2024-04-23T16:25:15.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a698073:::benchmark_data_smithi188_89276_object167201:head# 2024-04-23T16:25:15.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a69b570:::benchmark_data_smithi188_89276_object55916:head# 2024-04-23T16:25:15.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a6a15c8:::benchmark_data_smithi188_89276_object167852:head# 2024-04-23T16:25:15.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a6a5533:::benchmark_data_smithi188_89276_object46859:head# 2024-04-23T16:25:15.034 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a6abce4:::benchmark_data_smithi188_89276_object56668:head# 2024-04-23T16:25:15.034 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a6b4085:::benchmark_data_smithi188_89276_object91414:head# 2024-04-23T16:25:15.034 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a6c5219:::benchmark_data_smithi188_89276_object153156:head# 2024-04-23T16:25:15.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a6d59d4:::benchmark_data_smithi188_89276_object147565:head# 2024-04-23T16:25:15.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a6de541:::benchmark_data_smithi188_89276_object101781:head# 2024-04-23T16:25:15.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a6e84ff:::benchmark_data_smithi188_89276_object140511:head# 2024-04-23T16:25:15.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a6eab4f:::benchmark_data_smithi188_89276_object133853:head# 2024-04-23T16:25:15.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a6f1b29:::benchmark_data_smithi188_89276_object76924:head# 2024-04-23T16:25:15.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a6f6344:::benchmark_data_smithi188_89276_object164818:head# 2024-04-23T16:25:15.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a6fbcc4:::benchmark_data_smithi188_89276_object89753:head# 2024-04-23T16:25:15.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a708e3f:::benchmark_data_smithi188_89276_object158629:head# 2024-04-23T16:25:15.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a71a604:::benchmark_data_smithi188_89276_object43837:head# 2024-04-23T16:25:15.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a71a856:::benchmark_data_smithi188_89276_object137935:head# 2024-04-23T16:25:15.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a71e931:::benchmark_data_smithi188_89276_object54334:head# 2024-04-23T16:25:15.040 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a72e3e8:::benchmark_data_smithi188_89276_object43324:head# 2024-04-23T16:25:15.040 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a7358a5:::benchmark_data_smithi188_89276_object119858:head# 2024-04-23T16:25:15.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a746b52:::benchmark_data_smithi188_89276_object113524:head# 2024-04-23T16:25:15.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a74e604:::benchmark_data_smithi188_89276_object106214:head# 2024-04-23T16:25:15.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a75d799:::benchmark_data_smithi188_89276_object47781:head# 2024-04-23T16:25:15.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a7601cc:::benchmark_data_smithi188_89276_object76813:head# 2024-04-23T16:25:15.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a764e08:::benchmark_data_smithi188_89276_object122512:head# 2024-04-23T16:25:15.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a76a0db:::benchmark_data_smithi188_89276_object106894:head# 2024-04-23T16:25:15.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a76a334:::benchmark_data_smithi188_89276_object95496:head# 2024-04-23T16:25:15.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a76cc80:::benchmark_data_smithi188_89276_object118455:head# 2024-04-23T16:25:15.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a76d552:::benchmark_data_smithi188_89276_object65057:head# 2024-04-23T16:25:15.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a77d234:::benchmark_data_smithi188_89276_object100160:head# 2024-04-23T16:25:15.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a78217c:::benchmark_data_smithi188_89276_object54613:head# 2024-04-23T16:25:15.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a78da30:::benchmark_data_smithi188_89276_object97868:head# 2024-04-23T16:25:15.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a799a43:::benchmark_data_smithi188_89276_object99570:head# 2024-04-23T16:25:15.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a7a5d04:::benchmark_data_smithi188_89276_object71756:head# 2024-04-23T16:25:15.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a7acee6:::benchmark_data_smithi188_89276_object143534:head# 2024-04-23T16:25:15.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a7b2735:::benchmark_data_smithi188_89276_object95020:head# 2024-04-23T16:25:15.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a7bec0e:::benchmark_data_smithi188_89276_object48392:head# 2024-04-23T16:25:15.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a7c0632:::benchmark_data_smithi188_89276_object125376:head# 2024-04-23T16:25:15.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a7c16aa:::benchmark_data_smithi188_89276_object123381:head# 2024-04-23T16:25:15.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a7c2ee7:::benchmark_data_smithi188_89276_object146675:head# 2024-04-23T16:25:15.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a7c8659:::benchmark_data_smithi188_89276_object137361:head# 2024-04-23T16:25:15.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a7e11a1:::benchmark_data_smithi188_89276_object157799:head# 2024-04-23T16:25:15.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a7e2f59:::benchmark_data_smithi188_89276_object100561:head# 2024-04-23T16:25:15.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a7f1706:::benchmark_data_smithi188_89276_object54832:head# 2024-04-23T16:25:15.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a7f9d65:::benchmark_data_smithi188_89276_object108666:head# 2024-04-23T16:25:15.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a80b587:::benchmark_data_smithi188_89276_object87202:head# 2024-04-23T16:25:15.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a80fa6a:::benchmark_data_smithi188_89276_object104410:head# 2024-04-23T16:25:15.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a82a6a7:::benchmark_data_smithi188_89276_object71087:head# 2024-04-23T16:25:15.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a842f38:::benchmark_data_smithi188_89276_object119840:head# 2024-04-23T16:25:15.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a8439ac:::benchmark_data_smithi188_89276_object91468:head# 2024-04-23T16:25:15.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a84cdc5:::benchmark_data_smithi188_89276_object74447:head# 2024-04-23T16:25:15.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a853d83:::benchmark_data_smithi188_89276_object144052:head# 2024-04-23T16:25:15.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a860306:::benchmark_data_smithi188_89276_object123424:head# 2024-04-23T16:25:15.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a869f28:::benchmark_data_smithi188_89276_object109971:head# 2024-04-23T16:25:15.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a86aa28:::benchmark_data_smithi188_89276_object73850:head# 2024-04-23T16:25:15.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a876d8d:::benchmark_data_smithi188_89276_object110490:head# 2024-04-23T16:25:15.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a87c513:::benchmark_data_smithi188_89276_object62866:head# 2024-04-23T16:25:15.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a882fa8:::benchmark_data_smithi188_89276_object76954:head# 2024-04-23T16:25:15.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a886d00:::benchmark_data_smithi188_89276_object90111:head# 2024-04-23T16:25:15.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a887c2d:::benchmark_data_smithi188_89276_object70404:head# 2024-04-23T16:25:15.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a888de1:::benchmark_data_smithi188_89276_object147108:head# 2024-04-23T16:25:15.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a88aa60:::benchmark_data_smithi188_89276_object97755:head# 2024-04-23T16:25:15.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a8905ff:::benchmark_data_smithi188_89276_object104658:head# 2024-04-23T16:25:15.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a893895:::benchmark_data_smithi188_89276_object104805:head# 2024-04-23T16:25:15.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a898fac:::benchmark_data_smithi188_89276_object158090:head# 2024-04-23T16:25:15.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a8a4f40:::benchmark_data_smithi188_89276_object119879:head# 2024-04-23T16:25:15.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a8a5ee2:::benchmark_data_smithi188_89276_object121125:head# 2024-04-23T16:25:15.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a8bf98e:::benchmark_data_smithi188_89276_object84111:head# 2024-04-23T16:25:15.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a8c13d7:::benchmark_data_smithi188_89276_object92056:head# 2024-04-23T16:25:15.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a8c4c36:::benchmark_data_smithi188_89276_object138064:head# 2024-04-23T16:25:15.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a8c847b:::benchmark_data_smithi188_89276_object166242:head# 2024-04-23T16:25:15.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a8d70e8:::benchmark_data_smithi188_89276_object115694:head# 2024-04-23T16:25:15.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a8e121f:::benchmark_data_smithi188_89276_object45347:head# 2024-04-23T16:25:15.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a8e3084:::benchmark_data_smithi188_89276_object88270:head# 2024-04-23T16:25:15.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a8ec84b:::benchmark_data_smithi188_89276_object98167:head# 2024-04-23T16:25:15.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a8eec30:::benchmark_data_smithi188_89276_object111980:head# 2024-04-23T16:25:15.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a903c66:::benchmark_data_smithi188_89276_object92333:head# 2024-04-23T16:25:15.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a91d0db:::benchmark_data_smithi188_89276_object141868:head# 2024-04-23T16:25:15.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a924290:::benchmark_data_smithi188_89276_object80147:head# 2024-04-23T16:25:15.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a950fdd:::benchmark_data_smithi188_89276_object151542:head# 2024-04-23T16:25:15.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a97532c:::benchmark_data_smithi188_89276_object66971:head# 2024-04-23T16:25:15.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a9787c0:::benchmark_data_smithi188_89276_object134979:head# 2024-04-23T16:25:15.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a9929d9:::benchmark_data_smithi188_89276_object87447:head# 2024-04-23T16:25:15.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a9a369f:::benchmark_data_smithi188_89276_object117034:head# 2024-04-23T16:25:15.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a9aad92:::benchmark_data_smithi188_89276_object71889:head# 2024-04-23T16:25:15.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a9ab5d5:::benchmark_data_smithi188_89276_object103953:head# 2024-04-23T16:25:15.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a9ace43:::benchmark_data_smithi188_89276_object47448:head# 2024-04-23T16:25:15.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a9b6847:::benchmark_data_smithi188_89276_object51427:head# 2024-04-23T16:25:15.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a9bb691:::benchmark_data_smithi188_89276_object158466:head# 2024-04-23T16:25:15.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a9bd39b:::benchmark_data_smithi188_89276_object120589:head# 2024-04-23T16:25:15.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a9c8a69:::benchmark_data_smithi188_89276_object130176:head# 2024-04-23T16:25:15.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a9cb7a9:::benchmark_data_smithi188_89276_object101789:head# 2024-04-23T16:25:15.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a9d3b2d:::benchmark_data_smithi188_89276_object74729:head# 2024-04-23T16:25:15.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a9db427:::benchmark_data_smithi188_89276_object83572:head# 2024-04-23T16:25:15.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a9e0420:::benchmark_data_smithi188_89276_object113192:head# 2024-04-23T16:25:15.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a9e4c4c:::benchmark_data_smithi188_89276_object158287:head# 2024-04-23T16:25:15.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a9ebaaf:::benchmark_data_smithi188_89276_object64776:head# 2024-04-23T16:25:15.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a9faecf:::benchmark_data_smithi188_89276_object50586:head# 2024-04-23T16:25:15.076 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:15.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2a9fb92e:::benchmark_data_smithi188_89276_object62975:head# 2024-04-23T16:25:15.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aa00dc8:::benchmark_data_smithi188_89276_object146937:head# 2024-04-23T16:25:15.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aa2318f:::benchmark_data_smithi188_89276_object142513:head# 2024-04-23T16:25:15.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aa269a9:::benchmark_data_smithi188_89276_object164921:head# 2024-04-23T16:25:15.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aa3bc95:::benchmark_data_smithi188_89276_object72878:head# 2024-04-23T16:25:15.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aa3d8c5:::benchmark_data_smithi188_89276_object133077:head# 2024-04-23T16:25:15.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aa57337:::benchmark_data_smithi188_89276_object63938:head# 2024-04-23T16:25:15.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aa5739b:::benchmark_data_smithi188_89276_object146891:head# 2024-04-23T16:25:15.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aa5a367:::benchmark_data_smithi188_89276_object97665:head# 2024-04-23T16:25:15.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aa66ed0:::benchmark_data_smithi188_89276_object109333:head# 2024-04-23T16:25:15.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aa71ef9:::benchmark_data_smithi188_89276_object48901:head# 2024-04-23T16:25:15.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aa73c1f:::benchmark_data_smithi188_89276_object97395:head# 2024-04-23T16:25:15.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aa748b1:::benchmark_data_smithi188_89276_object98960:head# 2024-04-23T16:25:15.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aa76341:::benchmark_data_smithi188_89276_object68915:head# 2024-04-23T16:25:15.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aa9d776:::benchmark_data_smithi188_89276_object117728:head# 2024-04-23T16:25:15.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aab3550:::benchmark_data_smithi188_89276_object166851:head# 2024-04-23T16:25:15.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aac1844:::benchmark_data_smithi188_89276_object49837:head# 2024-04-23T16:25:15.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aacf749:::benchmark_data_smithi188_89276_object59044:head# 2024-04-23T16:25:15.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aad0ddd:::benchmark_data_smithi188_89276_object131286:head# 2024-04-23T16:25:15.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aad3cc2:::benchmark_data_smithi188_89276_object161987:head# 2024-04-23T16:25:15.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aad83fc:::benchmark_data_smithi188_89276_object155722:head# 2024-04-23T16:25:15.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aae2326:::benchmark_data_smithi188_89276_object129144:head# 2024-04-23T16:25:15.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aae40ff:::benchmark_data_smithi188_89276_object73423:head# 2024-04-23T16:25:15.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aae9cc7:::benchmark_data_smithi188_89276_object144613:head# 2024-04-23T16:25:15.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aaeab4d:::benchmark_data_smithi188_89276_object42186:head# 2024-04-23T16:25:15.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aaecdba:::benchmark_data_smithi188_89276_object131707:head# 2024-04-23T16:25:15.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aaf69c8:::benchmark_data_smithi188_89276_object79418:head# 2024-04-23T16:25:15.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aaf7a3f:::benchmark_data_smithi188_89276_object104222:head# 2024-04-23T16:25:15.091 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab03d4e:::benchmark_data_smithi188_89276_object98101:head# 2024-04-23T16:25:15.091 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab19a4f:::benchmark_data_smithi188_89276_object69203:head# 2024-04-23T16:25:15.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab1ad0c:::benchmark_data_smithi188_89276_object141658:head# 2024-04-23T16:25:15.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab1d990:::benchmark_data_smithi188_89276_object115297:head# 2024-04-23T16:25:15.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab26302:::benchmark_data_smithi188_89276_object129485:head# 2024-04-23T16:25:15.093 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab2ca8b:::benchmark_data_smithi188_89276_object129197:head# 2024-04-23T16:25:15.093 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab3c9e3:::benchmark_data_smithi188_89276_object107179:head# 2024-04-23T16:25:15.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab3ee8d:::benchmark_data_smithi188_89276_object55674:head# 2024-04-23T16:25:15.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab4da05:::benchmark_data_smithi188_89276_object104955:head# 2024-04-23T16:25:15.095 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab5264c:::benchmark_data_smithi188_89276_object147695:head# 2024-04-23T16:25:15.095 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab52e24:::benchmark_data_smithi188_89276_object90784:head# 2024-04-23T16:25:15.095 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab5ba6d:::benchmark_data_smithi188_89276_object125344:head# 2024-04-23T16:25:15.096 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab6adee:::benchmark_data_smithi188_89276_object59393:head# 2024-04-23T16:25:15.096 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab6d167:::benchmark_data_smithi188_89276_object55476:head# 2024-04-23T16:25:15.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab70d7b:::benchmark_data_smithi188_89276_object62511:head# 2024-04-23T16:25:15.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab79258:::benchmark_data_smithi188_89276_object153139:head# 2024-04-23T16:25:15.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab7e5ca:::benchmark_data_smithi188_89276_object131468:head# 2024-04-23T16:25:15.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab885e5:::benchmark_data_smithi188_89276_object50226:head# 2024-04-23T16:25:15.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab8b32a:::benchmark_data_smithi188_89276_object123001:head# 2024-04-23T16:25:15.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab8c333:::benchmark_data_smithi188_89276_object115405:head# 2024-04-23T16:25:15.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab8fa5f:::benchmark_data_smithi188_89276_object97548:head# 2024-04-23T16:25:15.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab90c89:::benchmark_data_smithi188_89276_object70832:head# 2024-04-23T16:25:15.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab98aaf:::benchmark_data_smithi188_89276_object133288:head# 2024-04-23T16:25:15.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ab9c354:::benchmark_data_smithi188_89276_object129363:head# 2024-04-23T16:25:15.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aba5af4:::benchmark_data_smithi188_89276_object160476:head# 2024-04-23T16:25:15.102 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aba697c:::benchmark_data_smithi188_89276_object67836:head# 2024-04-23T16:25:15.102 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2abb7953:::benchmark_data_smithi188_89276_object145584:head# 2024-04-23T16:25:15.102 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2abbafa6:::benchmark_data_smithi188_89276_object114428:head# 2024-04-23T16:25:15.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2abbc85b:::benchmark_data_smithi188_89276_object139515:head# 2024-04-23T16:25:15.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2abcdf41:::benchmark_data_smithi188_89276_object148383:head# 2024-04-23T16:25:15.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2abda3b3:::benchmark_data_smithi188_89276_object82582:head# 2024-04-23T16:25:15.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2abda7eb:::benchmark_data_smithi188_89276_object111620:head# 2024-04-23T16:25:15.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2abde831:::benchmark_data_smithi188_89276_object91723:head# 2024-04-23T16:25:15.105 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2abe541e:::benchmark_data_smithi188_89276_object47355:head# 2024-04-23T16:25:15.105 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2abe6095:::benchmark_data_smithi188_89276_object146821:head# 2024-04-23T16:25:15.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2abec759:::benchmark_data_smithi188_89276_object108885:head# 2024-04-23T16:25:15.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2abf551c:::benchmark_data_smithi188_89276_object44743:head# 2024-04-23T16:25:15.107 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2abf8249:::benchmark_data_smithi188_89276_object113406:head# 2024-04-23T16:25:15.107 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:25:15.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2abfb377:::benchmark_data_smithi188_89276_object66547:head# 2024-04-23T16:25:15.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac0115e:::benchmark_data_smithi188_89276_object142286:head# 2024-04-23T16:25:15.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac05915:::benchmark_data_smithi188_89276_object88467:head# 2024-04-23T16:25:15.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac06a50:::benchmark_data_smithi188_89276_object133840:head# 2024-04-23T16:25:15.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac180ad:::benchmark_data_smithi188_89276_object149976:head# 2024-04-23T16:25:15.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac197e3:::benchmark_data_smithi188_89276_object111240:head# 2024-04-23T16:25:15.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac21d03:::benchmark_data_smithi188_89276_object139139:head# 2024-04-23T16:25:15.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac28690:::benchmark_data_smithi188_89276_object140785:head# 2024-04-23T16:25:15.111 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac32365:::benchmark_data_smithi188_89276_object165956:head# 2024-04-23T16:25:15.111 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac3ed7f:::benchmark_data_smithi188_89276_object61804:head# 2024-04-23T16:25:15.111 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac46c80:::benchmark_data_smithi188_89276_object151136:head# 2024-04-23T16:25:15.111 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac4c3a1:::benchmark_data_smithi188_89276_object47738:head# 2024-04-23T16:25:15.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:15.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:25:15.113 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac4d906:::benchmark_data_smithi188_89276_object101956:head# 2024-04-23T16:25:15.113 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac50ccb:::benchmark_data_smithi188_89276_object159367:head# 2024-04-23T16:25:15.113 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac5512a:::benchmark_data_smithi188_89276_object135340:head# 2024-04-23T16:25:15.113 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac6de80:::benchmark_data_smithi188_89276_object151598:head# 2024-04-23T16:25:15.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac7181a:::benchmark_data_smithi188_89276_object168335:head# 2024-04-23T16:25:15.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac762b7:::benchmark_data_smithi188_89276_object148792:head# 2024-04-23T16:25:15.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac77ce7:::benchmark_data_smithi188_89276_object136774:head# 2024-04-23T16:25:15.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac7a027:::benchmark_data_smithi188_89276_object51143:head# 2024-04-23T16:25:15.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac81182:::benchmark_data_smithi188_89276_object102601:head# 2024-04-23T16:25:15.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac848be:::benchmark_data_smithi188_89276_object167975:head# 2024-04-23T16:25:15.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ac9f927:::benchmark_data_smithi188_89276_object130296:head# 2024-04-23T16:25:15.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2acac4f9:::benchmark_data_smithi188_89276_object59793:head# 2024-04-23T16:25:15.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2acd4e39:::benchmark_data_smithi188_89276_object93155:head# 2024-04-23T16:25:15.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2acda7fc:::benchmark_data_smithi188_89276_object73609:head# 2024-04-23T16:25:15.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2acedb4f:::benchmark_data_smithi188_89276_object50650:head# 2024-04-23T16:25:15.118 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2acf0f75:::benchmark_data_smithi188_89276_object97772:head# 2024-04-23T16:25:15.118 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2acf7cd7:::benchmark_data_smithi188_89276_object93146:head# 2024-04-23T16:25:15.118 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2acfcc80:::benchmark_data_smithi188_89276_object97388:head# 2024-04-23T16:25:15.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2acfea90:::benchmark_data_smithi188_89276_object167515:head# 2024-04-23T16:25:15.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ad0a63a:::benchmark_data_smithi188_89276_object146585:head# 2024-04-23T16:25:15.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ad187b2:::benchmark_data_smithi188_89276_object156658:head# 2024-04-23T16:25:15.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ad1b784:::benchmark_data_smithi188_89276_object118081:head# 2024-04-23T16:25:15.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ad224b4:::benchmark_data_smithi188_89276_object163123:head# 2024-04-23T16:25:15.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ad351d3:::benchmark_data_smithi188_89276_object126821:head# 2024-04-23T16:25:15.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ad48089:::benchmark_data_smithi188_89276_object107541:head# 2024-04-23T16:25:15.122 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ad4fdb9:::benchmark_data_smithi188_89276_object118180:head# 2024-04-23T16:25:15.122 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ad533d9:::benchmark_data_smithi188_89276_object157684:head# 2024-04-23T16:25:15.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ad5d0ac:::benchmark_data_smithi188_89276_object124880:head# 2024-04-23T16:25:15.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ad5dc50:::benchmark_data_smithi188_89276_object120099:head# 2024-04-23T16:25:15.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ad6eb64:::benchmark_data_smithi188_89276_object73167:head# 2024-04-23T16:25:15.124 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ad714ee:::benchmark_data_smithi188_89276_object42238:head# 2024-04-23T16:25:15.124 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ad732fe:::benchmark_data_smithi188_89276_object46767:head# 2024-04-23T16:25:15.125 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ad81f95:::benchmark_data_smithi188_89276_object93690:head# 2024-04-23T16:25:15.125 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ad8ebd7:::benchmark_data_smithi188_89276_object69430:head# 2024-04-23T16:25:15.125 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ad9ffc3:::benchmark_data_smithi188_89276_object105523:head# 2024-04-23T16:25:15.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ada7138:::benchmark_data_smithi188_89276_object62982:head# 2024-04-23T16:25:15.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2adaa44b:::benchmark_data_smithi188_89276_object138202:head# 2024-04-23T16:25:15.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2adab43f:::benchmark_data_smithi188_89276_object66657:head# 2024-04-23T16:25:15.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2adc3a5c:::benchmark_data_smithi188_89276_object41995:head# 2024-04-23T16:25:15.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2adc4d13:::benchmark_data_smithi188_89276_object65279:head# 2024-04-23T16:25:15.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2adc81d6:::benchmark_data_smithi188_89276_object147950:head# 2024-04-23T16:25:15.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2adcde53:::benchmark_data_smithi188_89276_object75360:head# 2024-04-23T16:25:15.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2add284c:::benchmark_data_smithi188_89276_object110729:head# 2024-04-23T16:25:15.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2add532c:::benchmark_data_smithi188_89276_object129976:head# 2024-04-23T16:25:15.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2addb1e1:::benchmark_data_smithi188_89276_object166886:head# 2024-04-23T16:25:15.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2adeb129:::benchmark_data_smithi188_89276_object132204:head# 2024-04-23T16:25:15.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2adef023:::benchmark_data_smithi188_89276_object149528:head# 2024-04-23T16:25:15.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ae1019b:::benchmark_data_smithi188_89276_object84335:head# 2024-04-23T16:25:15.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ae1e807:::benchmark_data_smithi188_89276_object119635:head# 2024-04-23T16:25:15.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ae2070c:::benchmark_data_smithi188_89276_object101365:head# 2024-04-23T16:25:15.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ae2255b:::benchmark_data_smithi188_89276_object128768:head# 2024-04-23T16:25:15.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ae29898:::benchmark_data_smithi188_89276_object113286:head# 2024-04-23T16:25:15.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ae2d743:::benchmark_data_smithi188_89276_object141038:head# 2024-04-23T16:25:15.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ae3271b:::benchmark_data_smithi188_89276_object78565:head# 2024-04-23T16:25:15.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ae3c90a:::benchmark_data_smithi188_89276_object98796:head# 2024-04-23T16:25:15.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ae4017e:::benchmark_data_smithi188_89276_object116473:head# 2024-04-23T16:25:15.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ae57b1b:::benchmark_data_smithi188_89276_object131581:head# 2024-04-23T16:25:15.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ae6c732:::benchmark_data_smithi188_89276_object63284:head# 2024-04-23T16:25:15.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ae721a2:::benchmark_data_smithi188_89276_object96844:head# 2024-04-23T16:25:15.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ae7bbd9:::benchmark_data_smithi188_89276_object62123:head# 2024-04-23T16:25:15.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ae80b78:::benchmark_data_smithi188_89276_object54889:head# 2024-04-23T16:25:15.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ae992dd:::benchmark_data_smithi188_89276_object125940:head# 2024-04-23T16:25:15.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ae9b4c6:::benchmark_data_smithi188_89276_object51559:head# 2024-04-23T16:25:15.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aea7e2a:::benchmark_data_smithi188_89276_object166590:head# 2024-04-23T16:25:15.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aead698:::benchmark_data_smithi188_89276_object54278:head# 2024-04-23T16:25:15.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aeb80ee:::benchmark_data_smithi188_89276_object54728:head# 2024-04-23T16:25:15.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aebe4a8:::benchmark_data_smithi188_89276_object90639:head# 2024-04-23T16:25:15.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aecba26:::benchmark_data_smithi188_89276_object164083:head# 2024-04-23T16:25:15.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aed062a:::benchmark_data_smithi188_89276_object132534:head# 2024-04-23T16:25:15.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aed41bf:::benchmark_data_smithi188_89276_object135535:head# 2024-04-23T16:25:15.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aed58af:::benchmark_data_smithi188_89276_object161555:head# 2024-04-23T16:25:15.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aeddf76:::benchmark_data_smithi188_89276_object88508:head# 2024-04-23T16:25:15.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aee3bcc:::benchmark_data_smithi188_89276_object117088:head# 2024-04-23T16:25:15.142 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aee7380:::benchmark_data_smithi188_89276_object70131:head# 2024-04-23T16:25:15.142 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aef1f1f:::benchmark_data_smithi188_89276_object150901:head# 2024-04-23T16:25:15.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2aef66e9:::benchmark_data_smithi188_89276_object102986:head# 2024-04-23T16:25:15.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2af050d4:::benchmark_data_smithi188_89276_object137913:head# 2024-04-23T16:25:15.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2af055c4:::benchmark_data_smithi188_89276_object63361:head# 2024-04-23T16:25:15.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2af0cae2:::benchmark_data_smithi188_89276_object75047:head# 2024-04-23T16:25:15.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2af51c02:::benchmark_data_smithi188_89276_object62371:head# 2024-04-23T16:25:15.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2af5223c:::benchmark_data_smithi188_89276_object71230:head# 2024-04-23T16:25:15.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2af54de4:::benchmark_data_smithi188_89276_object108435:head# 2024-04-23T16:25:15.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2af5a5eb:::benchmark_data_smithi188_89276_object85660:head# 2024-04-23T16:25:15.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2af60824:::benchmark_data_smithi188_89276_object127702:head# 2024-04-23T16:25:15.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2af70b30:::benchmark_data_smithi188_89276_object50417:head# 2024-04-23T16:25:15.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2af7df66:::benchmark_data_smithi188_89276_object160355:head# 2024-04-23T16:25:15.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2af925ca:::benchmark_data_smithi188_89276_object98974:head# 2024-04-23T16:25:15.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2af9593d:::benchmark_data_smithi188_89276_object84249:head# 2024-04-23T16:25:15.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2af9a9e2:::benchmark_data_smithi188_89276_object157385:head# 2024-04-23T16:25:15.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2af9e212:::benchmark_data_smithi188_89276_object109084:head# 2024-04-23T16:25:15.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2afa441f:::benchmark_data_smithi188_89276_object79114:head# 2024-04-23T16:25:15.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2afa6af8:::benchmark_data_smithi188_89276_object91590:head# 2024-04-23T16:25:15.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2afa6eae:::benchmark_data_smithi188_89276_object85304:head# 2024-04-23T16:25:15.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2afa9821:::benchmark_data_smithi188_89276_object118303:head# 2024-04-23T16:25:15.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2afa9b31:::benchmark_data_smithi188_89276_object40634:head# 2024-04-23T16:25:15.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2afbad66:::benchmark_data_smithi188_89276_object47656:head# 2024-04-23T16:25:15.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2afbca93:::benchmark_data_smithi188_89276_object161133:head# 2024-04-23T16:25:15.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2afc3782:::benchmark_data_smithi188_89276_object70638:head# 2024-04-23T16:25:15.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2afe041b:::benchmark_data_smithi188_89276_object126617:head# 2024-04-23T16:25:15.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2afe74e4:::benchmark_data_smithi188_89276_object141148:head# 2024-04-23T16:25:15.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2afe8177:::benchmark_data_smithi188_89276_object122869:head# 2024-04-23T16:25:15.158 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2afeeb96:::benchmark_data_smithi188_89276_object74493:head# 2024-04-23T16:25:15.158 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2affbd1b:::benchmark_data_smithi188_89276_object141835:head# 2024-04-23T16:25:15.158 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2affe1c3:::benchmark_data_smithi188_89276_object121502:head# 2024-04-23T16:25:15.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b01f08b:::benchmark_data_smithi188_89276_object87258:head# 2024-04-23T16:25:15.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b0227ba:::benchmark_data_smithi188_89276_object64591:head# 2024-04-23T16:25:15.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b023fe8:::benchmark_data_smithi188_89276_object75412:head# 2024-04-23T16:25:15.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b02e13e:::benchmark_data_smithi188_89276_object97970:head# 2024-04-23T16:25:15.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b030214:::benchmark_data_smithi188_89276_object111162:head# 2024-04-23T16:25:15.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b0405ec:::benchmark_data_smithi188_89276_object148352:head# 2024-04-23T16:25:15.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b04d8cc:::benchmark_data_smithi188_89276_object147202:head# 2024-04-23T16:25:15.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b04d8ce:::benchmark_data_smithi188_89276_object45736:head# 2024-04-23T16:25:15.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b0534fd:::benchmark_data_smithi188_89276_object93824:head# 2024-04-23T16:25:15.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b05e2f0:::benchmark_data_smithi188_89276_object64755:head# 2024-04-23T16:25:15.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b066b40:::benchmark_data_smithi188_89276_object160205:head# 2024-04-23T16:25:15.167 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b0673b8:::benchmark_data_smithi188_89276_object73709:head# 2024-04-23T16:25:15.167 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b070833:::benchmark_data_smithi188_89276_object157488:head# 2024-04-23T16:25:15.167 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b072445:::benchmark_data_smithi188_89276_object43014:head# 2024-04-23T16:25:15.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b07613f:::benchmark_data_smithi188_89276_object58342:head# 2024-04-23T16:25:15.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b077ac1:::benchmark_data_smithi188_89276_object66411:head# 2024-04-23T16:25:15.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b081dec:::benchmark_data_smithi188_89276_object87874:head# 2024-04-23T16:25:15.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b0840b5:::benchmark_data_smithi188_89276_object115652:head# 2024-04-23T16:25:15.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b0905e5:::benchmark_data_smithi188_89276_object83228:head# 2024-04-23T16:25:15.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b09ab32:::benchmark_data_smithi188_89276_object107068:head# 2024-04-23T16:25:15.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b0a2637:::benchmark_data_smithi188_89276_object49196:head# 2024-04-23T16:25:15.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b0a5816:::benchmark_data_smithi188_89276_object150607:head# 2024-04-23T16:25:15.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b0aa69f:::benchmark_data_smithi188_89276_object96271:head# 2024-04-23T16:25:15.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b0e3ff7:::benchmark_data_smithi188_89276_object46602:head# 2024-04-23T16:25:15.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b0e4efc:::benchmark_data_smithi188_89276_object153475:head# 2024-04-23T16:25:15.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b0ee9f0:::benchmark_data_smithi188_89276_object145533:head# 2024-04-23T16:25:15.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b0f781f:::benchmark_data_smithi188_89276_object120649:head# 2024-04-23T16:25:15.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b0f9b1a:::benchmark_data_smithi188_89276_object87747:head# 2024-04-23T16:25:15.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b0fe87a:::benchmark_data_smithi188_89276_object156203:head# 2024-04-23T16:25:15.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b1031c8:::benchmark_data_smithi188_89276_object73817:head# 2024-04-23T16:25:15.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b10e0d5:::benchmark_data_smithi188_89276_object70908:head# 2024-04-23T16:25:15.176 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b130acc:::benchmark_data_smithi188_89276_object105143:head# 2024-04-23T16:25:15.176 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b137e52:::benchmark_data_smithi188_89276_object118453:head# 2024-04-23T16:25:15.177 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b140f2a:::benchmark_data_smithi188_89276_object109314:head# 2024-04-23T16:25:15.177 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b150d3b:::benchmark_data_smithi188_89276_object49780:head# 2024-04-23T16:25:15.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b15f095:::benchmark_data_smithi188_89276_object84073:head# 2024-04-23T16:25:15.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b16d426:::benchmark_data_smithi188_89276_object143033:head# 2024-04-23T16:25:15.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b16e880:::benchmark_data_smithi188_89276_object141697:head# 2024-04-23T16:25:15.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b17165f:::benchmark_data_smithi188_89276_object82191:head# 2024-04-23T16:25:15.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b17ac33:::benchmark_data_smithi188_89276_object89604:head# 2024-04-23T16:25:15.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b18f75b:::benchmark_data_smithi188_89276_object94767:head# 2024-04-23T16:25:15.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b19ee2b:::benchmark_data_smithi188_89276_object107577:head# 2024-04-23T16:25:15.183 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b1ae24a:::benchmark_data_smithi188_89276_object144970:head# 2024-04-23T16:25:15.183 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b1bdd5d:::benchmark_data_smithi188_89276_object79306:head# 2024-04-23T16:25:15.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b1be904:::benchmark_data_smithi188_89276_object120323:head# 2024-04-23T16:25:15.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b1c026b:::benchmark_data_smithi188_89276_object161998:head# 2024-04-23T16:25:15.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b1cdf94:::benchmark_data_smithi188_89276_object80617:head# 2024-04-23T16:25:15.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b1cefe5:::benchmark_data_smithi188_89276_object92507:head# 2024-04-23T16:25:15.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b1f66b4:::benchmark_data_smithi188_89276_object115335:head# 2024-04-23T16:25:15.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b229ebd:::benchmark_data_smithi188_89276_object131546:head# 2024-04-23T16:25:15.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b22c1ca:::benchmark_data_smithi188_89276_object47717:head# 2024-04-23T16:25:15.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b23b467:::benchmark_data_smithi188_89276_object132505:head# 2024-04-23T16:25:15.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b23be4f:::benchmark_data_smithi188_89276_object166497:head# 2024-04-23T16:25:15.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b245199:::benchmark_data_smithi188_89276_object138348:head# 2024-04-23T16:25:15.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b2480c5:::benchmark_data_smithi188_89276_object87501:head# 2024-04-23T16:25:15.192 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b24cb29:::benchmark_data_smithi188_89276_object68534:head# 2024-04-23T16:25:15.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b254799:::benchmark_data_smithi188_89276_object56459:head# 2024-04-23T16:25:15.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b25d74f:::benchmark_data_smithi188_89276_object63339:head# 2024-04-23T16:25:15.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b25e2b5:::benchmark_data_smithi188_89276_object61764:head# 2024-04-23T16:25:15.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b25fa8f:::benchmark_data_smithi188_89276_object79645:head# 2024-04-23T16:25:15.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b26534d:::benchmark_data_smithi188_89276_object85030:head# 2024-04-23T16:25:15.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b267350:::benchmark_data_smithi188_89276_object139047:head# 2024-04-23T16:25:15.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b267bdb:::benchmark_data_smithi188_89276_object71012:head# 2024-04-23T16:25:15.197 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b2771c3:::benchmark_data_smithi188_89276_object55231:head# 2024-04-23T16:25:15.197 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b286437:::benchmark_data_smithi188_89276_object126879:head# 2024-04-23T16:25:15.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b291f3e:::benchmark_data_smithi188_89276_object83283:head# 2024-04-23T16:25:15.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b2931a3:::benchmark_data_smithi188_89276_object54588:head# 2024-04-23T16:25:15.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b2a75e4:::benchmark_data_smithi188_89276_object46805:head# 2024-04-23T16:25:15.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b2b4e32:::benchmark_data_smithi188_89276_object84697:head# 2024-04-23T16:25:15.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b2b5b45:::benchmark_data_smithi188_89276_object73144:head# 2024-04-23T16:25:15.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b2b734b:::benchmark_data_smithi188_89276_object145668:head# 2024-04-23T16:25:15.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b2bec42:::benchmark_data_smithi188_89276_object150862:head# 2024-04-23T16:25:15.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b2e64ee:::benchmark_data_smithi188_89276_object42407:head# 2024-04-23T16:25:15.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b2e697d:::benchmark_data_smithi188_89276_object151442:head# 2024-04-23T16:25:15.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b2ea048:::benchmark_data_smithi188_89276_object101267:head# 2024-04-23T16:25:15.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b2ee665:::benchmark_data_smithi188_89276_object152639:head# 2024-04-23T16:25:15.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b2ee843:::benchmark_data_smithi188_89276_object57974:head# 2024-04-23T16:25:15.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b2f1eb0:::benchmark_data_smithi188_89276_object142867:head# 2024-04-23T16:25:15.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b2f30b3:::benchmark_data_smithi188_89276_object111262:head# 2024-04-23T16:25:15.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b30b109:::benchmark_data_smithi188_89276_object131246:head# 2024-04-23T16:25:15.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b30fc48:::benchmark_data_smithi188_89276_object92466:head# 2024-04-23T16:25:15.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b3222cf:::benchmark_data_smithi188_89276_object89156:head# 2024-04-23T16:25:15.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b32b46f:::benchmark_data_smithi188_89276_object143830:head# 2024-04-23T16:25:15.206 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b32c215:::benchmark_data_smithi188_89276_object130919:head# 2024-04-23T16:25:15.206 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b32d331:::benchmark_data_smithi188_89276_object119315:head# 2024-04-23T16:25:15.206 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b32e138:::benchmark_data_smithi188_89276_object87427:head# 2024-04-23T16:25:15.207 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b334ec2:::benchmark_data_smithi188_89276_object155729:head# 2024-04-23T16:25:15.207 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b346a29:::benchmark_data_smithi188_89276_object54990:head# 2024-04-23T16:25:15.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b353f1f:::benchmark_data_smithi188_89276_object116477:head# 2024-04-23T16:25:15.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b360a6c:::benchmark_data_smithi188_89276_object107835:head# 2024-04-23T16:25:15.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b361188:::benchmark_data_smithi188_89276_object163301:head# 2024-04-23T16:25:15.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b36d0c6:::benchmark_data_smithi188_89276_object89815:head# 2024-04-23T16:25:15.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b36fd96:::benchmark_data_smithi188_89276_object62110:head# 2024-04-23T16:25:15.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b375475:::benchmark_data_smithi188_89276_object99408:head# 2024-04-23T16:25:15.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b375774:::benchmark_data_smithi188_89276_object43858:head# 2024-04-23T16:25:15.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b3784f5:::benchmark_data_smithi188_89276_object83714:head# 2024-04-23T16:25:15.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b37c776:::benchmark_data_smithi188_89276_object52193:head# 2024-04-23T16:25:15.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b37dcb9:::benchmark_data_smithi188_89276_object86716:head# 2024-04-23T16:25:15.212 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b38afac:::benchmark_data_smithi188_89276_object144874:head# 2024-04-23T16:25:15.212 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b38f5d5:::benchmark_data_smithi188_89276_object165025:head# 2024-04-23T16:25:15.213 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b39162d:::benchmark_data_smithi188_89276_object128022:head# 2024-04-23T16:25:15.213 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:25:15.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b3955e5:::benchmark_data_smithi188_89276_object62582:head# 2024-04-23T16:25:15.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b3ab9c2:::benchmark_data_smithi188_89276_object119929:head# 2024-04-23T16:25:15.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b3b0299:::benchmark_data_smithi188_89276_object101945:head# 2024-04-23T16:25:15.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b3b8990:::benchmark_data_smithi188_89276_object61901:head# 2024-04-23T16:25:15.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b3cf3b6:::benchmark_data_smithi188_89276_object107933:head# 2024-04-23T16:25:15.216 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b3d04cd:::benchmark_data_smithi188_89276_object123142:head# 2024-04-23T16:25:15.216 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b3d184f:::benchmark_data_smithi188_89276_object141641:head# 2024-04-23T16:25:15.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b3deae1:::benchmark_data_smithi188_89276_object93977:head# 2024-04-23T16:25:15.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b3e150d:::benchmark_data_smithi188_89276_object66645:head# 2024-04-23T16:25:15.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b3e63c1:::benchmark_data_smithi188_89276_object152541:head# 2024-04-23T16:25:15.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b3e68c3:::benchmark_data_smithi188_89276_object117514:head# 2024-04-23T16:25:15.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b3f2bac:::benchmark_data_smithi188_89276_object110821:head# 2024-04-23T16:25:15.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b3f9b83:::benchmark_data_smithi188_89276_object60910:head# 2024-04-23T16:25:15.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b3fc22b:::benchmark_data_smithi188_89276_object97268:head# 2024-04-23T16:25:15.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b3fc7c0:::benchmark_data_smithi188_89276_object115468:head# 2024-04-23T16:25:15.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b40149e:::benchmark_data_smithi188_89276_object40429:head# 2024-04-23T16:25:15.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b402e53:::benchmark_data_smithi188_89276_object58998:head# 2024-04-23T16:25:15.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b405de0:::benchmark_data_smithi188_89276_object125239:head# 2024-04-23T16:25:15.221 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b41a248:::benchmark_data_smithi188_89276_object90199:head# 2024-04-23T16:25:15.221 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b43306c:::benchmark_data_smithi188_89276_object108932:head# 2024-04-23T16:25:15.221 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b4385a3:::benchmark_data_smithi188_89276_object132057:head# 2024-04-23T16:25:15.225 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b43f96c:::benchmark_data_smithi188_89276_object57106:head# 2024-04-23T16:25:15.225 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b45a538:::benchmark_data_smithi188_89276_object151596:head# 2024-04-23T16:25:15.225 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b463291:::benchmark_data_smithi188_89276_object47784:head# 2024-04-23T16:25:15.225 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b479e3f:::benchmark_data_smithi188_89276_object55192:head# 2024-04-23T16:25:15.225 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b48307c:::benchmark_data_smithi188_89276_object61164:head# 2024-04-23T16:25:15.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b492fa2:::benchmark_data_smithi188_89276_object117902:head# 2024-04-23T16:25:15.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b49576c:::benchmark_data_smithi188_89276_object156733:head# 2024-04-23T16:25:15.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b4a769a:::benchmark_data_smithi188_89276_object76526:head# 2024-04-23T16:25:15.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b4ad1eb:::benchmark_data_smithi188_89276_object100220:head# 2024-04-23T16:25:15.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b4b13a9:::benchmark_data_smithi188_89276_object156455:head# 2024-04-23T16:25:15.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b4df908:::benchmark_data_smithi188_89276_object72077:head# 2024-04-23T16:25:15.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b4e2d7e:::benchmark_data_smithi188_89276_object163268:head# 2024-04-23T16:25:15.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b4e9021:::benchmark_data_smithi188_89276_object88359:head# 2024-04-23T16:25:15.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b4f3a8f:::benchmark_data_smithi188_89276_object94115:head# 2024-04-23T16:25:15.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b4f60ff:::benchmark_data_smithi188_89276_object90612:head# 2024-04-23T16:25:15.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b505ef4:::benchmark_data_smithi188_89276_object165736:head# 2024-04-23T16:25:15.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b50b1db:::benchmark_data_smithi188_89276_object103181:head# 2024-04-23T16:25:15.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b542480:::benchmark_data_smithi188_89276_object123834:head# 2024-04-23T16:25:15.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b5500f0:::benchmark_data_smithi188_89276_object130948:head# 2024-04-23T16:25:15.231 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b554981:::benchmark_data_smithi188_89276_object165147:head# 2024-04-23T16:25:15.231 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b560103:::benchmark_data_smithi188_89276_object143294:head# 2024-04-23T16:25:15.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b564a42:::benchmark_data_smithi188_89276_object75772:head# 2024-04-23T16:25:15.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b56901e:::benchmark_data_smithi188_89276_object126372:head# 2024-04-23T16:25:15.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b569479:::benchmark_data_smithi188_89276_object77828:head# 2024-04-23T16:25:15.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b56b12c:::benchmark_data_smithi188_89276_object60240:head# 2024-04-23T16:25:15.234 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b56d3b4:::benchmark_data_smithi188_89276_object66058:head# 2024-04-23T16:25:15.234 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b576e42:::benchmark_data_smithi188_89276_object62533:head# 2024-04-23T16:25:15.235 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b57785f:::benchmark_data_smithi188_89276_object146999:head# 2024-04-23T16:25:15.235 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b577eeb:::benchmark_data_smithi188_89276_object42395:head# 2024-04-23T16:25:15.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b584e4f:::benchmark_data_smithi188_89276_object55759:head# 2024-04-23T16:25:15.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b597fe9:::benchmark_data_smithi188_89276_object147749:head# 2024-04-23T16:25:15.237 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b5a1500:::benchmark_data_smithi188_89276_object97674:head# 2024-04-23T16:25:15.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b5a5412:::benchmark_data_smithi188_89276_object166957:head# 2024-04-23T16:25:15.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b5a7e2e:::benchmark_data_smithi188_89276_object60211:head# 2024-04-23T16:25:15.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b5ab57b:::benchmark_data_smithi188_89276_object96739:head# 2024-04-23T16:25:15.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b5abe55:::benchmark_data_smithi188_89276_object115990:head# 2024-04-23T16:25:15.240 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b5b04f6:::benchmark_data_smithi188_89276_object89621:head# 2024-04-23T16:25:15.240 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b5b596f:::benchmark_data_smithi188_89276_object120929:head# 2024-04-23T16:25:15.241 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b5b6f95:::benchmark_data_smithi188_89276_object106881:head# 2024-04-23T16:25:15.241 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b5bb1a9:::benchmark_data_smithi188_89276_object95846:head# 2024-04-23T16:25:15.241 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b5c15b2:::benchmark_data_smithi188_89276_object123762:head# 2024-04-23T16:25:15.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b5c2c3e:::benchmark_data_smithi188_89276_object92104:head# 2024-04-23T16:25:15.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b5cf8e6:::benchmark_data_smithi188_89276_object84941:head# 2024-04-23T16:25:15.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b5d2106:::benchmark_data_smithi188_89276_object141833:head# 2024-04-23T16:25:15.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b5e1411:::benchmark_data_smithi188_89276_object130355:head# 2024-04-23T16:25:15.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b5f0eea:::benchmark_data_smithi188_89276_object48175:head# 2024-04-23T16:25:15.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b5f6301:::benchmark_data_smithi188_89276_object63885:head# 2024-04-23T16:25:15.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b5fba2a:::benchmark_data_smithi188_89276_object46935:head# 2024-04-23T16:25:15.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b603c41:::benchmark_data_smithi188_89276_object137291:head# 2024-04-23T16:25:15.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b6044e0:::benchmark_data_smithi188_89276_object156584:head# 2024-04-23T16:25:15.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b606df0:::benchmark_data_smithi188_89276_object85615:head# 2024-04-23T16:25:15.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b6193e8:::benchmark_data_smithi188_89276_object66624:head# 2024-04-23T16:25:15.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b61afb6:::benchmark_data_smithi188_89276_object57818:head# 2024-04-23T16:25:15.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b61c898:::benchmark_data_smithi188_89276_object76736:head# 2024-04-23T16:25:15.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b6342b4:::benchmark_data_smithi188_89276_object164670:head# 2024-04-23T16:25:15.248 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b642027:::benchmark_data_smithi188_89276_object164266:head# 2024-04-23T16:25:15.248 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b647c63:::benchmark_data_smithi188_89276_object144723:head# 2024-04-23T16:25:15.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b651bc4:::benchmark_data_smithi188_89276_object102464:head# 2024-04-23T16:25:15.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b65ceed:::benchmark_data_smithi188_89276_object137423:head# 2024-04-23T16:25:15.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b66a95e:::benchmark_data_smithi188_89276_object129895:head# 2024-04-23T16:25:15.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b66b6b8:::benchmark_data_smithi188_89276_object80231:head# 2024-04-23T16:25:15.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b67760a:::benchmark_data_smithi188_89276_object57530:head# 2024-04-23T16:25:15.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b67f5bc:::benchmark_data_smithi188_89276_object70761:head# 2024-04-23T16:25:15.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b684dc7:::benchmark_data_smithi188_89276_object110533:head# 2024-04-23T16:25:15.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b69982a:::benchmark_data_smithi188_89276_object50930:head# 2024-04-23T16:25:15.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b6a070a:::benchmark_data_smithi188_89276_object67811:head# 2024-04-23T16:25:15.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b6b1c14:::benchmark_data_smithi188_89276_object79179:head# 2024-04-23T16:25:15.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b6bba6a:::benchmark_data_smithi188_89276_object94582:head# 2024-04-23T16:25:15.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b6c52ac:::benchmark_data_smithi188_89276_object164050:head# 2024-04-23T16:25:15.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b6c961f:::benchmark_data_smithi188_89276_object139424:head# 2024-04-23T16:25:15.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b6d7a42:::benchmark_data_smithi188_89276_object45153:head# 2024-04-23T16:25:15.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b6ea453:::benchmark_data_smithi188_89276_object135622:head# 2024-04-23T16:25:15.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b6ef1ca:::benchmark_data_smithi188_89276_object86061:head# 2024-04-23T16:25:15.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b6f69e9:::benchmark_data_smithi188_89276_object51456:head# 2024-04-23T16:25:15.257 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:15.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b6fed36:::benchmark_data_smithi188_89276_object144094:head# 2024-04-23T16:25:15.258 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b706805:::benchmark_data_smithi188_89276_object82830:head# 2024-04-23T16:25:15.258 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b706c0f:::benchmark_data_smithi188_89276_object114717:head# 2024-04-23T16:25:15.259 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:14 smithi148 ceph-mon[44520]: pgmap v716: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 445 op/s; 15 MiB/s, 239 objects/s recovering 2024-04-23T16:25:15.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b70b7ff:::benchmark_data_smithi188_89276_object134288:head# 2024-04-23T16:25:15.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b70f6ea:::benchmark_data_smithi188_89276_object143412:head# 2024-04-23T16:25:15.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b71149c:::benchmark_data_smithi188_89276_object111388:head# 2024-04-23T16:25:15.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b71e97d:::benchmark_data_smithi188_89276_object123274:head# 2024-04-23T16:25:15.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b72523b:::benchmark_data_smithi188_89276_object54730:head# 2024-04-23T16:25:15.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b72c37b:::benchmark_data_smithi188_89276_object52841:head# 2024-04-23T16:25:15.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b7326e1:::benchmark_data_smithi188_89276_object55935:head# 2024-04-23T16:25:15.262 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b744ba1:::benchmark_data_smithi188_89276_object117474:head# 2024-04-23T16:25:15.262 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b74582b:::benchmark_data_smithi188_89276_object144061:head# 2024-04-23T16:25:15.262 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b75347f:::benchmark_data_smithi188_89276_object164861:head# 2024-04-23T16:25:15.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b75771d:::benchmark_data_smithi188_89276_object122938:head# 2024-04-23T16:25:15.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b7627d9:::benchmark_data_smithi188_89276_object144983:head# 2024-04-23T16:25:15.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b763819:::benchmark_data_smithi188_89276_object151597:head# 2024-04-23T16:25:15.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b76ab9e:::benchmark_data_smithi188_89276_object102357:head# 2024-04-23T16:25:15.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b779848:::benchmark_data_smithi188_89276_object73150:head# 2024-04-23T16:25:15.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b77ba4c:::benchmark_data_smithi188_89276_object145114:head# 2024-04-23T16:25:15.266 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b77c9aa:::benchmark_data_smithi188_89276_object150345:head# 2024-04-23T16:25:15.266 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b79fae9:::benchmark_data_smithi188_89276_object134199:head# 2024-04-23T16:25:15.266 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b7a8b6d:::benchmark_data_smithi188_89276_object153220:head# 2024-04-23T16:25:15.267 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b7a9a96:::benchmark_data_smithi188_89276_object58992:head# 2024-04-23T16:25:15.267 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b7b1fed:::benchmark_data_smithi188_89276_object110014:head# 2024-04-23T16:25:15.267 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b7b474e:::benchmark_data_smithi188_89276_object62523:head# 2024-04-23T16:25:15.268 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b7bbeca:::benchmark_data_smithi188_89276_object121114:head# 2024-04-23T16:25:15.268 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b7be449:::benchmark_data_smithi188_89276_object42369:head# 2024-04-23T16:25:15.268 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b7c0e26:::benchmark_data_smithi188_89276_object82474:head# 2024-04-23T16:25:15.269 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b7c2089:::benchmark_data_smithi188_89276_object99306:head# 2024-04-23T16:25:15.269 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b7c4ef9:::benchmark_data_smithi188_89276_object56677:head# 2024-04-23T16:25:15.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b7c5cbf:::benchmark_data_smithi188_89276_object163719:head# 2024-04-23T16:25:15.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b7cddc3:::benchmark_data_smithi188_89276_object100617:head# 2024-04-23T16:25:15.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b7e0bf5:::benchmark_data_smithi188_89276_object132489:head# 2024-04-23T16:25:15.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b7e2dad:::benchmark_data_smithi188_89276_object149318:head# 2024-04-23T16:25:15.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b7f66bb:::benchmark_data_smithi188_89276_object52834:head# 2024-04-23T16:25:15.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b7fe187:::benchmark_data_smithi188_89276_object168383:head# 2024-04-23T16:25:15.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b805208:::benchmark_data_smithi188_89276_object100462:head# 2024-04-23T16:25:15.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8093c6:::benchmark_data_smithi188_89276_object132921:head# 2024-04-23T16:25:15.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b80cf8c:::benchmark_data_smithi188_89276_object59435:head# 2024-04-23T16:25:15.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b81736b:::benchmark_data_smithi188_89276_object131447:head# 2024-04-23T16:25:15.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8322cf:::benchmark_data_smithi188_89276_object144625:head# 2024-04-23T16:25:15.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b838bca:::benchmark_data_smithi188_89276_object156521:head# 2024-04-23T16:25:15.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b83ae51:::benchmark_data_smithi188_89276_object64971:head# 2024-04-23T16:25:15.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b845683:::benchmark_data_smithi188_89276_object93525:head# 2024-04-23T16:25:15.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b847bba:::benchmark_data_smithi188_89276_object88507:head# 2024-04-23T16:25:15.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b85dd89:::benchmark_data_smithi188_89276_object156124:head# 2024-04-23T16:25:15.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b87893c:::benchmark_data_smithi188_89276_object142596:head# 2024-04-23T16:25:15.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8790d7:::benchmark_data_smithi188_89276_object128865:head# 2024-04-23T16:25:15.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b87bc53:::benchmark_data_smithi188_89276_object148257:head# 2024-04-23T16:25:15.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b884dc6:::benchmark_data_smithi188_89276_object146490:head# 2024-04-23T16:25:15.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b885821:::benchmark_data_smithi188_89276_object118656:head# 2024-04-23T16:25:15.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b88de16:::benchmark_data_smithi188_89276_object118753:head# 2024-04-23T16:25:15.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8a8c7d:::benchmark_data_smithi188_89276_object149984:head# 2024-04-23T16:25:15.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8a93f6:::benchmark_data_smithi188_89276_object47572:head# 2024-04-23T16:25:15.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8a9833:::benchmark_data_smithi188_89276_object42841:head# 2024-04-23T16:25:15.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8ae009:::benchmark_data_smithi188_89276_object149567:head# 2024-04-23T16:25:15.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8af23a:::benchmark_data_smithi188_89276_object73596:head# 2024-04-23T16:25:15.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8c1c3c:::benchmark_data_smithi188_89276_object134476:head# 2024-04-23T16:25:15.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8c1cd1:::benchmark_data_smithi188_89276_object148770:head# 2024-04-23T16:25:15.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8c91cc:::benchmark_data_smithi188_89276_object63855:head# 2024-04-23T16:25:15.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8cca4a:::benchmark_data_smithi188_89276_object63778:head# 2024-04-23T16:25:15.287 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8d03ef:::benchmark_data_smithi188_89276_object145665:head# 2024-04-23T16:25:15.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8d17f1:::benchmark_data_smithi188_89276_object66480:head# 2024-04-23T16:25:15.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8d385b:::benchmark_data_smithi188_89276_object87981:head# 2024-04-23T16:25:15.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8d555b:::benchmark_data_smithi188_89276_object118177:head# 2024-04-23T16:25:15.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8dcf68:::benchmark_data_smithi188_89276_object49646:head# 2024-04-23T16:25:15.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8f4e62:::benchmark_data_smithi188_89276_object152450:head# 2024-04-23T16:25:15.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8f5e0f:::benchmark_data_smithi188_89276_object105065:head# 2024-04-23T16:25:15.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b8fa37f:::benchmark_data_smithi188_89276_object60219:head# 2024-04-23T16:25:15.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b909ab7:::benchmark_data_smithi188_89276_object76331:head# 2024-04-23T16:25:15.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b913169:::benchmark_data_smithi188_89276_object60827:head# 2024-04-23T16:25:15.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b91b168:::benchmark_data_smithi188_89276_object166107:head# 2024-04-23T16:25:15.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9207e0:::benchmark_data_smithi188_89276_object101609:head# 2024-04-23T16:25:15.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b92532c:::benchmark_data_smithi188_89276_object127729:head# 2024-04-23T16:25:15.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9298dc:::benchmark_data_smithi188_89276_object89196:head# 2024-04-23T16:25:15.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9308b5:::benchmark_data_smithi188_89276_object55461:head# 2024-04-23T16:25:15.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b933d7d:::benchmark_data_smithi188_89276_object63785:head# 2024-04-23T16:25:15.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b93cf24:::benchmark_data_smithi188_89276_object60989:head# 2024-04-23T16:25:15.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b948e48:::benchmark_data_smithi188_89276_object64911:head# 2024-04-23T16:25:15.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b957562:::benchmark_data_smithi188_89276_object148916:head# 2024-04-23T16:25:15.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b958d45:::benchmark_data_smithi188_89276_object98816:head# 2024-04-23T16:25:15.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9734af:::benchmark_data_smithi188_89276_object110070:head# 2024-04-23T16:25:15.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b978ac4:::benchmark_data_smithi188_89276_object88711:head# 2024-04-23T16:25:15.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b97bfdf:::benchmark_data_smithi188_89276_object139433:head# 2024-04-23T16:25:15.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b97d1b3:::benchmark_data_smithi188_89276_object139309:head# 2024-04-23T16:25:15.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b98433b:::benchmark_data_smithi188_89276_object164082:head# 2024-04-23T16:25:15.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b988030:::benchmark_data_smithi188_89276_object76380:head# 2024-04-23T16:25:15.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b98e30f:::benchmark_data_smithi188_89276_object85935:head# 2024-04-23T16:25:15.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9984d5:::benchmark_data_smithi188_89276_object132504:head# 2024-04-23T16:25:15.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9a0cc9:::benchmark_data_smithi188_89276_object69388:head# 2024-04-23T16:25:15.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9a3189:::benchmark_data_smithi188_89276_object123982:head# 2024-04-23T16:25:15.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9a600e:::benchmark_data_smithi188_89276_object129884:head# 2024-04-23T16:25:15.303 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:25:15.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9bfaa9:::benchmark_data_smithi188_89276_object158609:head# 2024-04-23T16:25:15.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9c4726:::benchmark_data_smithi188_89276_object101838:head# 2024-04-23T16:25:15.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9c5c26:::benchmark_data_smithi188_89276_object115742:head# 2024-04-23T16:25:15.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9cc839:::benchmark_data_smithi188_89276_object51069:head# 2024-04-23T16:25:15.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9cca7d:::benchmark_data_smithi188_89276_object139979:head# 2024-04-23T16:25:15.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9d635f:::benchmark_data_smithi188_89276_object98411:head# 2024-04-23T16:25:15.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9dc8b1:::benchmark_data_smithi188_89276_object93702:head# 2024-04-23T16:25:15.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9e06bc:::benchmark_data_smithi188_89276_object161302:head# 2024-04-23T16:25:15.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9e0f5f:::benchmark_data_smithi188_89276_object129230:head# 2024-04-23T16:25:15.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9ec760:::benchmark_data_smithi188_89276_object68300:head# 2024-04-23T16:25:15.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9ed40a:::benchmark_data_smithi188_89276_object84590:head# 2024-04-23T16:25:15.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9ed96d:::benchmark_data_smithi188_89276_object64002:head# 2024-04-23T16:25:15.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9f27af:::benchmark_data_smithi188_89276_object156364:head# 2024-04-23T16:25:15.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9f9488:::benchmark_data_smithi188_89276_object143004:head# 2024-04-23T16:25:15.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2b9fa4f0:::benchmark_data_smithi188_89276_object147650:head# 2024-04-23T16:25:15.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:15.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:25:15.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba0b866:::benchmark_data_smithi188_89276_object49952:head# 2024-04-23T16:25:15.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba0cb89:::benchmark_data_smithi188_89276_object160352:head# 2024-04-23T16:25:15.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba0f2b2:::benchmark_data_smithi188_89276_object72577:head# 2024-04-23T16:25:15.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba147b2:::benchmark_data_smithi188_89276_object92918:head# 2024-04-23T16:25:15.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba1a321:::benchmark_data_smithi188_89276_object110514:head# 2024-04-23T16:25:15.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba1b5f8:::benchmark_data_smithi188_89276_object108260:head# 2024-04-23T16:25:15.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba265d6:::benchmark_data_smithi188_89276_object125343:head# 2024-04-23T16:25:15.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba293bf:::benchmark_data_smithi188_89276_object100059:head# 2024-04-23T16:25:15.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba2e266:::benchmark_data_smithi188_89276_object123890:head# 2024-04-23T16:25:15.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba2e39d:::benchmark_data_smithi188_89276_object67869:head# 2024-04-23T16:25:15.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba32562:::benchmark_data_smithi188_89276_object70443:head# 2024-04-23T16:25:15.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba3625b:::benchmark_data_smithi188_89276_object58314:head# 2024-04-23T16:25:15.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba42bc5:::benchmark_data_smithi188_89276_object75978:head# 2024-04-23T16:25:15.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba448a9:::benchmark_data_smithi188_89276_object122586:head# 2024-04-23T16:25:15.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba46cc8:::benchmark_data_smithi188_89276_object155928:head# 2024-04-23T16:25:15.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba4a1f6:::benchmark_data_smithi188_89276_object90968:head# 2024-04-23T16:25:15.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba4bd77:::benchmark_data_smithi188_89276_object113967:head# 2024-04-23T16:25:15.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba4e074:::benchmark_data_smithi188_89276_object129994:head# 2024-04-23T16:25:15.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba5ccf3:::benchmark_data_smithi188_89276_object153255:head# 2024-04-23T16:25:15.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba63083:::benchmark_data_smithi188_89276_object51823:head# 2024-04-23T16:25:15.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba65cce:::benchmark_data_smithi188_89276_object54287:head# 2024-04-23T16:25:15.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba6b76a:::benchmark_data_smithi188_89276_object100720:head# 2024-04-23T16:25:15.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba6c00f:::benchmark_data_smithi188_89276_object159673:head# 2024-04-23T16:25:15.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ba73c3e:::benchmark_data_smithi188_89276_object148196:head# 2024-04-23T16:25:15.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2baa580d:::benchmark_data_smithi188_89276_object90114:head# 2024-04-23T16:25:15.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2baa6d5f:::benchmark_data_smithi188_89276_object114699:head# 2024-04-23T16:25:15.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2baa9c32:::benchmark_data_smithi188_89276_object142918:head# 2024-04-23T16:25:15.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2baae79e:::benchmark_data_smithi188_89276_object124808:head# 2024-04-23T16:25:15.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2babcb5e:::benchmark_data_smithi188_89276_object114725:head# 2024-04-23T16:25:15.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bac26e1:::benchmark_data_smithi188_89276_object122394:head# 2024-04-23T16:25:15.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bac4b57:::benchmark_data_smithi188_89276_object51632:head# 2024-04-23T16:25:15.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bac7c39:::benchmark_data_smithi188_89276_object95363:head# 2024-04-23T16:25:15.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bacd0a9:::benchmark_data_smithi188_89276_object98466:head# 2024-04-23T16:25:15.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bad2e30:::benchmark_data_smithi188_89276_object138591:head# 2024-04-23T16:25:15.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bad9eff:::benchmark_data_smithi188_89276_object78264:head# 2024-04-23T16:25:15.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2baddf8e:::benchmark_data_smithi188_89276_object90539:head# 2024-04-23T16:25:15.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bae1e5b:::benchmark_data_smithi188_89276_object76022:head# 2024-04-23T16:25:15.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2baf0010:::benchmark_data_smithi188_89276_object125056:head# 2024-04-23T16:25:15.334 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2baf4813:::benchmark_data_smithi188_89276_object66900:head# 2024-04-23T16:25:15.334 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2baf5314:::benchmark_data_smithi188_89276_object115693:head# 2024-04-23T16:25:15.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2baf64e1:::benchmark_data_smithi188_89276_object70300:head# 2024-04-23T16:25:15.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bafede9:::benchmark_data_smithi188_89276_object46340:head# 2024-04-23T16:25:15.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bb1f23c:::benchmark_data_smithi188_89276_object101619:head# 2024-04-23T16:25:15.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bb275d3:::benchmark_data_smithi188_89276_object149158:head# 2024-04-23T16:25:15.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bb31e3c:::benchmark_data_smithi188_89276_object116404:head# 2024-04-23T16:25:15.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bb320cb:::benchmark_data_smithi188_89276_object76368:head# 2024-04-23T16:25:15.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bb36e86:::benchmark_data_smithi188_89276_object124798:head# 2024-04-23T16:25:15.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bb4001b:::benchmark_data_smithi188_89276_object92614:head# 2024-04-23T16:25:15.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bb4e9eb:::benchmark_data_smithi188_89276_object72867:head# 2024-04-23T16:25:15.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bb4f757:::benchmark_data_smithi188_89276_object71274:head# 2024-04-23T16:25:15.341 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bb6a4b4:::benchmark_data_smithi188_89276_object75494:head# 2024-04-23T16:25:15.341 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bb6bb99:::benchmark_data_smithi188_89276_object131356:head# 2024-04-23T16:25:15.341 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bb750bb:::benchmark_data_smithi188_89276_object98206:head# 2024-04-23T16:25:15.342 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bb7be60:::benchmark_data_smithi188_89276_object83854:head# 2024-04-23T16:25:15.342 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bb87a77:::benchmark_data_smithi188_89276_object64283:head# 2024-04-23T16:25:15.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bb90d18:::benchmark_data_smithi188_89276_object94585:head# 2024-04-23T16:25:15.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bb9c0cf:::benchmark_data_smithi188_89276_object87998:head# 2024-04-23T16:25:15.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bba4cfa:::benchmark_data_smithi188_89276_object93341:head# 2024-04-23T16:25:15.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bbb5725:::benchmark_data_smithi188_89276_object159944:head# 2024-04-23T16:25:15.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bbba08e:::benchmark_data_smithi188_89276_object75970:head# 2024-04-23T16:25:15.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bbcf3fd:::benchmark_data_smithi188_89276_object42858:head# 2024-04-23T16:25:15.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bbe10cd:::benchmark_data_smithi188_89276_object147237:head# 2024-04-23T16:25:15.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bbe73a8:::benchmark_data_smithi188_89276_object77432:head# 2024-04-23T16:25:15.346 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bbea7b9:::benchmark_data_smithi188_89276_object50632:head# 2024-04-23T16:25:15.346 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bbeed1d:::benchmark_data_smithi188_89276_object51390:head# 2024-04-23T16:25:15.346 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bbefefe:::benchmark_data_smithi188_89276_object52104:head# 2024-04-23T16:25:15.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bbf05dc:::benchmark_data_smithi188_89276_object85210:head# 2024-04-23T16:25:15.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bbfe940:::benchmark_data_smithi188_89276_object41550:head# 2024-04-23T16:25:15.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bc0e66d:::benchmark_data_smithi188_89276_object56470:head# 2024-04-23T16:25:15.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bc13ede:::benchmark_data_smithi188_89276_object149166:head# 2024-04-23T16:25:15.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bc2705b:::benchmark_data_smithi188_89276_object139287:head# 2024-04-23T16:25:15.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bc2d094:::benchmark_data_smithi188_89276_object128302:head# 2024-04-23T16:25:15.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bc37fb0:::benchmark_data_smithi188_89276_object121665:head# 2024-04-23T16:25:15.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bc40646:::benchmark_data_smithi188_89276_object127938:head# 2024-04-23T16:25:15.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bc41ef8:::benchmark_data_smithi188_89276_object113067:head# 2024-04-23T16:25:15.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bc4a4a4:::benchmark_data_smithi188_89276_object75899:head# 2024-04-23T16:25:15.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bc4c7a4:::benchmark_data_smithi188_89276_object77838:head# 2024-04-23T16:25:15.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bc57ee4:::benchmark_data_smithi188_89276_object70546:head# 2024-04-23T16:25:15.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bc70280:::benchmark_data_smithi188_89276_object142950:head# 2024-04-23T16:25:15.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bc730e5:::benchmark_data_smithi188_89276_object68844:head# 2024-04-23T16:25:15.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bc798d3:::benchmark_data_smithi188_89276_object133486:head# 2024-04-23T16:25:15.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bc86c38:::benchmark_data_smithi188_89276_object66423:head# 2024-04-23T16:25:15.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bc8b779:::benchmark_data_smithi188_89276_object82329:head# 2024-04-23T16:25:15.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bc8c01f:::benchmark_data_smithi188_89276_object69457:head# 2024-04-23T16:25:15.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bc9a37e:::benchmark_data_smithi188_89276_object59381:head# 2024-04-23T16:25:15.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bca373b:::benchmark_data_smithi188_89276_object51613:head# 2024-04-23T16:25:15.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bca73b4:::benchmark_data_smithi188_89276_object137398:head# 2024-04-23T16:25:15.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bca745e:::benchmark_data_smithi188_89276_object167736:head# 2024-04-23T16:25:15.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bcaa61c:::benchmark_data_smithi188_89276_object135370:head# 2024-04-23T16:25:15.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bcae42a:::benchmark_data_smithi188_89276_object51862:head# 2024-04-23T16:25:15.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bcc1b7a:::benchmark_data_smithi188_89276_object121254:head# 2024-04-23T16:25:15.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bcc1b9d:::benchmark_data_smithi188_89276_object71718:head# 2024-04-23T16:25:15.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bccdf9e:::benchmark_data_smithi188_89276_object126674:head# 2024-04-23T16:25:15.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bccebc8:::benchmark_data_smithi188_89276_object70670:head# 2024-04-23T16:25:15.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bcd827b:::benchmark_data_smithi188_89276_object70899:head# 2024-04-23T16:25:15.361 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bcda607:::benchmark_data_smithi188_89276_object67364:head# 2024-04-23T16:25:15.361 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bcdbd43:::benchmark_data_smithi188_89276_object86253:head# 2024-04-23T16:25:15.362 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bcf4c44:::benchmark_data_smithi188_89276_object128525:head# 2024-04-23T16:25:15.362 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bcf9e4f:::benchmark_data_smithi188_89276_object149623:head# 2024-04-23T16:25:15.363 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bcfa18d:::benchmark_data_smithi188_89276_object46563:head# 2024-04-23T16:25:15.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bd0bf0c:::benchmark_data_smithi188_89276_object109447:head# 2024-04-23T16:25:15.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bd17c18:::benchmark_data_smithi188_89276_object75943:head# 2024-04-23T16:25:15.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bd1bb75:::benchmark_data_smithi188_89276_object99161:head# 2024-04-23T16:25:15.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bd2dbff:::benchmark_data_smithi188_89276_object152268:head# 2024-04-23T16:25:15.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bd43606:::benchmark_data_smithi188_89276_object110111:head# 2024-04-23T16:25:15.366 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bd4dc6d:::benchmark_data_smithi188_89276_object113080:head# 2024-04-23T16:25:15.366 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bd53e8a:::benchmark_data_smithi188_89276_object89888:head# 2024-04-23T16:25:15.368 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bd62a32:::benchmark_data_smithi188_89276_object60979:head# 2024-04-23T16:25:15.368 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bd6da37:::benchmark_data_smithi188_89276_object132884:head# 2024-04-23T16:25:15.368 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bd72926:::benchmark_data_smithi188_89276_object143350:head# 2024-04-23T16:25:15.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bd73df3:::benchmark_data_smithi188_89276_object137020:head# 2024-04-23T16:25:15.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bd78758:::benchmark_data_smithi188_89276_object132324:head# 2024-04-23T16:25:15.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bd7979c:::benchmark_data_smithi188_89276_object150153:head# 2024-04-23T16:25:15.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bd814ff:::benchmark_data_smithi188_89276_object96407:head# 2024-04-23T16:25:15.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bd8e719:::benchmark_data_smithi188_89276_object104195:head# 2024-04-23T16:25:15.371 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bd953a4:::benchmark_data_smithi188_89276_object83314:head# 2024-04-23T16:25:15.373 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bda08ec:::benchmark_data_smithi188_89276_object161029:head# 2024-04-23T16:25:15.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bda5258:::benchmark_data_smithi188_89276_object101871:head# 2024-04-23T16:25:15.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bdab346:::benchmark_data_smithi188_89276_object154402:head# 2024-04-23T16:25:15.375 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bdbd9ba:::benchmark_data_smithi188_89276_object75977:head# 2024-04-23T16:25:15.375 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bdceac2:::benchmark_data_smithi188_89276_object94388:head# 2024-04-23T16:25:15.375 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bdd5fd4:::benchmark_data_smithi188_89276_object143646:head# 2024-04-23T16:25:15.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bdd63fb:::benchmark_data_smithi188_89276_object66153:head# 2024-04-23T16:25:15.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bddb75c:::benchmark_data_smithi188_89276_object40451:head# 2024-04-23T16:25:15.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bde5f8f:::benchmark_data_smithi188_89276_object74857:head# 2024-04-23T16:25:15.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bde6b2e:::benchmark_data_smithi188_89276_object113308:head# 2024-04-23T16:25:15.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bdea80d:::benchmark_data_smithi188_89276_object168450:head# 2024-04-23T16:25:15.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bdeddaf:::benchmark_data_smithi188_89276_object48576:head# 2024-04-23T16:25:15.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be155c1:::benchmark_data_smithi188_89276_object134637:head# 2024-04-23T16:25:15.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be1a034:::benchmark_data_smithi188_89276_object162169:head# 2024-04-23T16:25:15.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be1b4f4:::benchmark_data_smithi188_89276_object48130:head# 2024-04-23T16:25:15.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be21b24:::benchmark_data_smithi188_89276_object121192:head# 2024-04-23T16:25:15.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be3a902:::benchmark_data_smithi188_89276_object58204:head# 2024-04-23T16:25:15.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be3d51a:::benchmark_data_smithi188_89276_object82564:head# 2024-04-23T16:25:15.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be5d6ef:::benchmark_data_smithi188_89276_object87748:head# 2024-04-23T16:25:15.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be60404:::benchmark_data_smithi188_89276_object81494:head# 2024-04-23T16:25:15.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be6320e:::benchmark_data_smithi188_89276_object137176:head# 2024-04-23T16:25:15.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be6a6fe:::benchmark_data_smithi188_89276_object149685:head# 2024-04-23T16:25:15.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be7063e:::benchmark_data_smithi188_89276_object88394:head# 2024-04-23T16:25:15.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be7b193:::benchmark_data_smithi188_89276_object78765:head# 2024-04-23T16:25:15.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be7c600:::benchmark_data_smithi188_89276_object69798:head# 2024-04-23T16:25:15.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be7ed18:::benchmark_data_smithi188_89276_object141102:head# 2024-04-23T16:25:15.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be864be:::benchmark_data_smithi188_89276_object167571:head# 2024-04-23T16:25:15.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be88828:::benchmark_data_smithi188_89276_object57755:head# 2024-04-23T16:25:15.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be904df:::benchmark_data_smithi188_89276_object62314:head# 2024-04-23T16:25:15.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be94c83:::benchmark_data_smithi188_89276_object147089:head# 2024-04-23T16:25:15.389 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be98cce:::benchmark_data_smithi188_89276_object165874:head# 2024-04-23T16:25:15.389 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2be9d1ef:::benchmark_data_smithi188_89276_object43531:head# 2024-04-23T16:25:15.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bea2c78:::benchmark_data_smithi188_89276_object135110:head# 2024-04-23T16:25:15.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2beb2eb0:::benchmark_data_smithi188_89276_object148915:head# 2024-04-23T16:25:15.391 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bebba20:::benchmark_data_smithi188_89276_object49378:head# 2024-04-23T16:25:15.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bec10b1:::benchmark_data_smithi188_89276_object92360:head# 2024-04-23T16:25:15.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bee2d3f:::benchmark_data_smithi188_89276_object120057:head# 2024-04-23T16:25:15.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bef5710:::benchmark_data_smithi188_89276_object150198:head# 2024-04-23T16:25:15.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bf096a6:::benchmark_data_smithi188_89276_object62421:head# 2024-04-23T16:25:15.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bf147a0:::benchmark_data_smithi188_89276_object126395:head# 2024-04-23T16:25:15.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bf2187b:::benchmark_data_smithi188_89276_object148562:head# 2024-04-23T16:25:15.395 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bf24934:::benchmark_data_smithi188_89276_object138327:head# 2024-04-23T16:25:15.395 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bf38b94:::benchmark_data_smithi188_89276_object166902:head# 2024-04-23T16:25:15.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bf49f48:::benchmark_data_smithi188_89276_object61265:head# 2024-04-23T16:25:15.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bf576c2:::benchmark_data_smithi188_89276_object45769:head# 2024-04-23T16:25:15.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bf5f491:::benchmark_data_smithi188_89276_object79961:head# 2024-04-23T16:25:15.397 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bf61374:::benchmark_data_smithi188_89276_object50261:head# 2024-04-23T16:25:15.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bf71a63:::benchmark_data_smithi188_89276_object136987:head# 2024-04-23T16:25:15.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bf77eb5:::benchmark_data_smithi188_89276_object108450:head# 2024-04-23T16:25:15.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bf81f92:::benchmark_data_smithi188_89276_object140635:head# 2024-04-23T16:25:15.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bf9bc93:::benchmark_data_smithi188_89276_object129826:head# 2024-04-23T16:25:15.399 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bf9dbe2:::benchmark_data_smithi188_89276_object141860:head# 2024-04-23T16:25:15.399 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bfa4453:::benchmark_data_smithi188_89276_object129879:head# 2024-04-23T16:25:15.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bfb31d2:::benchmark_data_smithi188_89276_object149137:head# 2024-04-23T16:25:15.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bfb4e2c:::benchmark_data_smithi188_89276_object104514:head# 2024-04-23T16:25:15.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bfc0a54:::benchmark_data_smithi188_89276_object47322:head# 2024-04-23T16:25:15.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bfc1cf6:::benchmark_data_smithi188_89276_object161288:head# 2024-04-23T16:25:15.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bfcd006:::benchmark_data_smithi188_89276_object82461:head# 2024-04-23T16:25:15.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bfd1394:::benchmark_data_smithi188_89276_object163989:head# 2024-04-23T16:25:15.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bfda138:::benchmark_data_smithi188_89276_object139822:head# 2024-04-23T16:25:15.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bfdd1cc:::benchmark_data_smithi188_89276_object58700:head# 2024-04-23T16:25:15.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bfde9fe:::benchmark_data_smithi188_89276_object90582:head# 2024-04-23T16:25:15.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bfdefe9:::benchmark_data_smithi188_89276_object109770:head# 2024-04-23T16:25:15.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bfe0605:::benchmark_data_smithi188_89276_object52628:head# 2024-04-23T16:25:15.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bfe254e:::benchmark_data_smithi188_89276_object157355:head# 2024-04-23T16:25:15.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bfea192:::benchmark_data_smithi188_89276_object158226:head# 2024-04-23T16:25:15.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bfea8b1:::benchmark_data_smithi188_89276_object75689:head# 2024-04-23T16:25:15.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2bfecb05:::benchmark_data_smithi188_89276_object74197:head# 2024-04-23T16:25:15.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c014759:::benchmark_data_smithi188_89276_object151624:head# 2024-04-23T16:25:15.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c02db28:::benchmark_data_smithi188_89276_object52851:head# 2024-04-23T16:25:15.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c030a30:::benchmark_data_smithi188_89276_object82116:head# 2024-04-23T16:25:15.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c032123:::benchmark_data_smithi188_89276_object96905:head# 2024-04-23T16:25:15.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c042433:::benchmark_data_smithi188_89276_object117397:head# 2024-04-23T16:25:15.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c0446d0:::benchmark_data_smithi188_89276_object126833:head# 2024-04-23T16:25:15.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c045ae5:::benchmark_data_smithi188_89276_object98440:head# 2024-04-23T16:25:15.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c04b58f:::benchmark_data_smithi188_89276_object94467:head# 2024-04-23T16:25:15.411 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:25:15.412 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c04bb2c:::benchmark_data_smithi188_89276_object101861:head# 2024-04-23T16:25:15.412 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c06e24d:::benchmark_data_smithi188_89276_object89165:head# 2024-04-23T16:25:15.412 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c06f4e1:::benchmark_data_smithi188_89276_object49552:head# 2024-04-23T16:25:15.412 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c0703e6:::benchmark_data_smithi188_89276_object90254:head# 2024-04-23T16:25:15.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c077b2a:::benchmark_data_smithi188_89276_object99084:head# 2024-04-23T16:25:15.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c07bd22:::benchmark_data_smithi188_89276_object101831:head# 2024-04-23T16:25:15.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c07cc2a:::benchmark_data_smithi188_89276_object44313:head# 2024-04-23T16:25:15.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c08cc21:::benchmark_data_smithi188_89276_object64213:head# 2024-04-23T16:25:15.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c0a0a56:::benchmark_data_smithi188_89276_object69271:head# 2024-04-23T16:25:15.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c0a3f06:::benchmark_data_smithi188_89276_object95256:head# 2024-04-23T16:25:15.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c0a9b9d:::benchmark_data_smithi188_89276_object42083:head# 2024-04-23T16:25:15.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c0b1c68:::benchmark_data_smithi188_89276_object161306:head# 2024-04-23T16:25:15.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c0bba65:::benchmark_data_smithi188_89276_object95130:head# 2024-04-23T16:25:15.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c0be58e:::benchmark_data_smithi188_89276_object70553:head# 2024-04-23T16:25:15.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c0db7d8:::benchmark_data_smithi188_89276_object46838:head# 2024-04-23T16:25:15.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c0e8bab:::benchmark_data_smithi188_89276_object129597:head# 2024-04-23T16:25:15.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c0f37ed:::benchmark_data_smithi188_89276_object85530:head# 2024-04-23T16:25:15.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c0fcd00:::benchmark_data_smithi188_89276_object51916:head# 2024-04-23T16:25:15.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c0fcfe9:::benchmark_data_smithi188_89276_object94576:head# 2024-04-23T16:25:15.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c1057a7:::benchmark_data_smithi188_89276_object61219:head# 2024-04-23T16:25:15.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c10a830:::benchmark_data_smithi188_89276_object47935:head# 2024-04-23T16:25:15.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c10c20a:::benchmark_data_smithi188_89276_object119206:head# 2024-04-23T16:25:15.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c10cce0:::benchmark_data_smithi188_89276_object83578:head# 2024-04-23T16:25:15.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c120c53:::benchmark_data_smithi188_89276_object99679:head# 2024-04-23T16:25:15.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c127875:::benchmark_data_smithi188_89276_object134293:head# 2024-04-23T16:25:15.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c12eee3:::benchmark_data_smithi188_89276_object116135:head# 2024-04-23T16:25:15.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c1357d4:::benchmark_data_smithi188_89276_object149902:head# 2024-04-23T16:25:15.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c147d49:::benchmark_data_smithi188_89276_object120582:head# 2024-04-23T16:25:15.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c155e06:::benchmark_data_smithi188_89276_object150896:head# 2024-04-23T16:25:15.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c16b8c8:::benchmark_data_smithi188_89276_object73820:head# 2024-04-23T16:25:15.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c172865:::benchmark_data_smithi188_89276_object168090:head# 2024-04-23T16:25:15.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c17e8ba:::benchmark_data_smithi188_89276_object126992:head# 2024-04-23T16:25:15.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c183b5b:::benchmark_data_smithi188_89276_object115320:head# 2024-04-23T16:25:15.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c18781d:::benchmark_data_smithi188_89276_object57159:head# 2024-04-23T16:25:15.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c187993:::benchmark_data_smithi188_89276_object152904:head# 2024-04-23T16:25:15.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c18957c:::benchmark_data_smithi188_89276_object145821:head# 2024-04-23T16:25:15.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c19b0ee:::benchmark_data_smithi188_89276_object41152:head# 2024-04-23T16:25:15.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c19c47a:::benchmark_data_smithi188_89276_object64188:head# 2024-04-23T16:25:15.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c1a5cf8:::benchmark_data_smithi188_89276_object89950:head# 2024-04-23T16:25:15.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c1ab3db:::benchmark_data_smithi188_89276_object41806:head# 2024-04-23T16:25:15.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c1ade82:::benchmark_data_smithi188_89276_object160588:head# 2024-04-23T16:25:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c1bf6fd:::benchmark_data_smithi188_89276_object112774:head# 2024-04-23T16:25:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c1c1f3f:::benchmark_data_smithi188_89276_object134450:head# 2024-04-23T16:25:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c1cfd5c:::benchmark_data_smithi188_89276_object102398:head# 2024-04-23T16:25:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c1d39cd:::benchmark_data_smithi188_89276_object165513:head# 2024-04-23T16:25:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c1e40bb:::benchmark_data_smithi188_89276_object129740:head# 2024-04-23T16:25:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c1e6e4b:::benchmark_data_smithi188_89276_object155957:head# 2024-04-23T16:25:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c1f998f:::benchmark_data_smithi188_89276_object86464:head# 2024-04-23T16:25:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c1fbdce:::benchmark_data_smithi188_89276_object109525:head# 2024-04-23T16:25:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c202e7c:::benchmark_data_smithi188_89276_object164037:head# 2024-04-23T16:25:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c204198:::benchmark_data_smithi188_89276_object144720:head# 2024-04-23T16:25:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c2072d1:::benchmark_data_smithi188_89276_object164364:head# 2024-04-23T16:25:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c20b49c:::benchmark_data_smithi188_89276_object96200:head# 2024-04-23T16:25:15.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c20b5f9:::benchmark_data_smithi188_89276_object66961:head# 2024-04-23T16:25:15.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c20da2c:::benchmark_data_smithi188_89276_object142881:head# 2024-04-23T16:25:15.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c20ed78:::benchmark_data_smithi188_89276_object146678:head# 2024-04-23T16:25:15.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c216318:::benchmark_data_smithi188_89276_object80728:head# 2024-04-23T16:25:15.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c21cd7f:::benchmark_data_smithi188_89276_object140484:head# 2024-04-23T16:25:15.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c21f33b:::benchmark_data_smithi188_89276_object113227:head# 2024-04-23T16:25:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c229cca:::benchmark_data_smithi188_89276_object131413:head# 2024-04-23T16:25:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c235612:::benchmark_data_smithi188_89276_object49447:head# 2024-04-23T16:25:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c243220:::benchmark_data_smithi188_89276_object130507:head# 2024-04-23T16:25:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c2450ef:::benchmark_data_smithi188_89276_object87744:head# 2024-04-23T16:25:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c2464b2:::benchmark_data_smithi188_89276_object100056:head# 2024-04-23T16:25:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c2568a6:::benchmark_data_smithi188_89276_object82749:head# 2024-04-23T16:25:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c25b861:::benchmark_data_smithi188_89276_object47270:head# 2024-04-23T16:25:15.444 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:15.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c25fecc:::benchmark_data_smithi188_89276_object76803:head# 2024-04-23T16:25:15.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c26244a:::benchmark_data_smithi188_89276_object105498:head# 2024-04-23T16:25:15.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c26630f:::benchmark_data_smithi188_89276_object165413:head# 2024-04-23T16:25:15.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c26cda2:::benchmark_data_smithi188_89276_object48347:head# 2024-04-23T16:25:15.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c26e23c:::benchmark_data_smithi188_89276_object123752:head# 2024-04-23T16:25:15.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c2783c6:::benchmark_data_smithi188_89276_object55984:head# 2024-04-23T16:25:15.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c27e526:::benchmark_data_smithi188_89276_object84639:head# 2024-04-23T16:25:15.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c28149b:::benchmark_data_smithi188_89276_object61220:head# 2024-04-23T16:25:15.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c298b41:::benchmark_data_smithi188_89276_object89770:head# 2024-04-23T16:25:15.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c299d2a:::benchmark_data_smithi188_89276_object88124:head# 2024-04-23T16:25:15.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c29e242:::benchmark_data_smithi188_89276_object99653:head# 2024-04-23T16:25:15.448 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c2aa9b3:::benchmark_data_smithi188_89276_object78961:head# 2024-04-23T16:25:15.449 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c2b0e51:::benchmark_data_smithi188_89276_object81358:head# 2024-04-23T16:25:15.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c2c7241:::benchmark_data_smithi188_89276_object80614:head# 2024-04-23T16:25:15.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c2c748f:::benchmark_data_smithi188_89276_object109874:head# 2024-04-23T16:25:15.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c2d5075:::benchmark_data_smithi188_89276_object96352:head# 2024-04-23T16:25:15.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c2d65d4:::benchmark_data_smithi188_89276_object135859:head# 2024-04-23T16:25:15.461 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c2d8a57:::benchmark_data_smithi188_89276_object141227:head# 2024-04-23T16:25:15.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c2da2b4:::benchmark_data_smithi188_89276_object51737:head# 2024-04-23T16:25:15.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c2dae56:::benchmark_data_smithi188_89276_object134183:head# 2024-04-23T16:25:15.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c2dd5e1:::benchmark_data_smithi188_89276_object49687:head# 2024-04-23T16:25:15.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c2f1356:::benchmark_data_smithi188_89276_object90490:head# 2024-04-23T16:25:15.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c308a8d:::benchmark_data_smithi188_89276_object120742:head# 2024-04-23T16:25:15.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c3097a3:::benchmark_data_smithi188_89276_object97731:head# 2024-04-23T16:25:15.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c3140b3:::benchmark_data_smithi188_89276_object115354:head# 2024-04-23T16:25:15.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c31d003:::benchmark_data_smithi188_89276_object58317:head# 2024-04-23T16:25:15.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c323a7e:::benchmark_data_smithi188_89276_object43977:head# 2024-04-23T16:25:15.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c33740e:::benchmark_data_smithi188_89276_object75581:head# 2024-04-23T16:25:15.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c338ee0:::benchmark_data_smithi188_89276_object69390:head# 2024-04-23T16:25:15.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c33a239:::benchmark_data_smithi188_89276_object154562:head# 2024-04-23T16:25:15.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c343403:::benchmark_data_smithi188_89276_object135966:head# 2024-04-23T16:25:15.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c34b80e:::benchmark_data_smithi188_89276_object79962:head# 2024-04-23T16:25:15.469 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c34bc0c:::benchmark_data_smithi188_89276_object148733:head# 2024-04-23T16:25:15.469 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c356aaa:::benchmark_data_smithi188_89276_object129785:head# 2024-04-23T16:25:15.470 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c3571e4:::benchmark_data_smithi188_89276_object74741:head# 2024-04-23T16:25:15.470 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c35b503:::benchmark_data_smithi188_89276_object57848:head# 2024-04-23T16:25:15.470 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c35c1aa:::benchmark_data_smithi188_89276_object59876:head# 2024-04-23T16:25:15.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c3622e9:::benchmark_data_smithi188_89276_object154615:head# 2024-04-23T16:25:15.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c36df91:::benchmark_data_smithi188_89276_object128681:head# 2024-04-23T16:25:15.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c3716d0:::benchmark_data_smithi188_89276_object122819:head# 2024-04-23T16:25:15.473 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c372c8c:::benchmark_data_smithi188_89276_object127546:head# 2024-04-23T16:25:15.473 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c38162d:::benchmark_data_smithi188_89276_object142098:head# 2024-04-23T16:25:15.474 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c38a4fb:::benchmark_data_smithi188_89276_object95517:head# 2024-04-23T16:25:15.474 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c39c133:::benchmark_data_smithi188_89276_object47445:head# 2024-04-23T16:25:15.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c39df36:::benchmark_data_smithi188_89276_object83199:head# 2024-04-23T16:25:15.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c3a5004:::benchmark_data_smithi188_89276_object118756:head# 2024-04-23T16:25:15.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c3b8b4f:::benchmark_data_smithi188_89276_object80643:head# 2024-04-23T16:25:15.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c3c368f:::benchmark_data_smithi188_89276_object62004:head# 2024-04-23T16:25:15.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c3c910e:::benchmark_data_smithi188_89276_object106406:head# 2024-04-23T16:25:15.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:14 smithi188 ceph-mon[44101]: pgmap v716: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 445 op/s; 15 MiB/s, 239 objects/s recovering 2024-04-23T16:25:15.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c3d8987:::benchmark_data_smithi188_89276_object122345:head# 2024-04-23T16:25:15.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c3df742:::benchmark_data_smithi188_89276_object91105:head# 2024-04-23T16:25:15.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c3e6882:::benchmark_data_smithi188_89276_object155997:head# 2024-04-23T16:25:15.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c3f8a41:::benchmark_data_smithi188_89276_object119946:head# 2024-04-23T16:25:15.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c3fb0f9:::benchmark_data_smithi188_89276_object128392:head# 2024-04-23T16:25:15.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c401511:::benchmark_data_smithi188_89276_object100214:head# 2024-04-23T16:25:15.481 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c408bcc:::benchmark_data_smithi188_89276_object153825:head# 2024-04-23T16:25:15.481 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c4093db:::benchmark_data_smithi188_89276_object66939:head# 2024-04-23T16:25:15.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c40eceb:::benchmark_data_smithi188_89276_object125452:head# 2024-04-23T16:25:15.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c4144d4:::benchmark_data_smithi188_89276_object73710:head# 2024-04-23T16:25:15.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c419e0d:::benchmark_data_smithi188_89276_object153433:head# 2024-04-23T16:25:15.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c41e78e:::benchmark_data_smithi188_89276_object115589:head# 2024-04-23T16:25:15.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c41fcb6:::benchmark_data_smithi188_89276_object73056:head# 2024-04-23T16:25:15.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c4299e3:::benchmark_data_smithi188_89276_object157944:head# 2024-04-23T16:25:15.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c42efcb:::benchmark_data_smithi188_89276_object133017:head# 2024-04-23T16:25:15.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c43226b:::benchmark_data_smithi188_89276_object44808:head# 2024-04-23T16:25:15.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c43514b:::benchmark_data_smithi188_89276_object58575:head# 2024-04-23T16:25:15.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c43d578:::benchmark_data_smithi188_89276_object47274:head# 2024-04-23T16:25:15.486 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c474550:::benchmark_data_smithi188_89276_object162508:head# 2024-04-23T16:25:15.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c47a428:::benchmark_data_smithi188_89276_object83000:head# 2024-04-23T16:25:15.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c481e03:::benchmark_data_smithi188_89276_object86160:head# 2024-04-23T16:25:15.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c4873ff:::benchmark_data_smithi188_89276_object133016:head# 2024-04-23T16:25:15.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c492e8a:::benchmark_data_smithi188_89276_object137145:head# 2024-04-23T16:25:15.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c499cda:::benchmark_data_smithi188_89276_object120164:head# 2024-04-23T16:25:15.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c49c068:::benchmark_data_smithi188_89276_object47926:head# 2024-04-23T16:25:15.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c4a0034:::benchmark_data_smithi188_89276_object68538:head# 2024-04-23T16:25:15.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c4b9cb4:::benchmark_data_smithi188_89276_object142504:head# 2024-04-23T16:25:15.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c4c8568:::benchmark_data_smithi188_89276_object116616:head# 2024-04-23T16:25:15.491 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c4c8cd0:::benchmark_data_smithi188_89276_object78269:head# 2024-04-23T16:25:15.491 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c4d06df:::benchmark_data_smithi188_89276_object101967:head# 2024-04-23T16:25:15.491 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c4d672a:::benchmark_data_smithi188_89276_object134586:head# 2024-04-23T16:25:15.491 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c4d833a:::benchmark_data_smithi188_89276_object125294:head# 2024-04-23T16:25:15.492 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c4dad8c:::benchmark_data_smithi188_89276_object117299:head# 2024-04-23T16:25:15.492 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c4dec1f:::benchmark_data_smithi188_89276_object112251:head# 2024-04-23T16:25:15.493 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c4efac0:::benchmark_data_smithi188_89276_object81688:head# 2024-04-23T16:25:15.493 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c4f117d:::benchmark_data_smithi188_89276_object52709:head# 2024-04-23T16:25:15.494 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c4fd1cd:::benchmark_data_smithi188_89276_object88031:head# 2024-04-23T16:25:15.494 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c501bb0:::benchmark_data_smithi188_89276_object158083:head# 2024-04-23T16:25:15.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c522864:::benchmark_data_smithi188_89276_object64609:head# 2024-04-23T16:25:15.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c524dfc:::benchmark_data_smithi188_89276_object136758:head# 2024-04-23T16:25:15.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c526783:::benchmark_data_smithi188_89276_object146522:head# 2024-04-23T16:25:15.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c52af31:::benchmark_data_smithi188_89276_object150946:head# 2024-04-23T16:25:15.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c52da78:::benchmark_data_smithi188_89276_object138857:head# 2024-04-23T16:25:15.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c53356d:::benchmark_data_smithi188_89276_object69622:head# 2024-04-23T16:25:15.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c535449:::benchmark_data_smithi188_89276_object84002:head# 2024-04-23T16:25:15.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c53e0ac:::benchmark_data_smithi188_89276_object127857:head# 2024-04-23T16:25:15.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c53e884:::benchmark_data_smithi188_89276_object44073:head# 2024-04-23T16:25:15.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c5472c8:::benchmark_data_smithi188_89276_object119913:head# 2024-04-23T16:25:15.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c556d97:::benchmark_data_smithi188_89276_object73612:head# 2024-04-23T16:25:15.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c55d414:::benchmark_data_smithi188_89276_object139213:head# 2024-04-23T16:25:15.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c55e3c0:::benchmark_data_smithi188_89276_object156196:head# 2024-04-23T16:25:15.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c565965:::benchmark_data_smithi188_89276_object68578:head# 2024-04-23T16:25:15.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c56ea1f:::benchmark_data_smithi188_89276_object102790:head# 2024-04-23T16:25:15.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c572551:::benchmark_data_smithi188_89276_object113618:head# 2024-04-23T16:25:15.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c576a62:::benchmark_data_smithi188_89276_object107891:head# 2024-04-23T16:25:15.507 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c57826a:::benchmark_data_smithi188_89276_object83936:head# 2024-04-23T16:25:15.507 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c5830a9:::benchmark_data_smithi188_89276_object112855:head# 2024-04-23T16:25:15.507 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c58808d:::benchmark_data_smithi188_89276_object62819:head# 2024-04-23T16:25:15.508 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c58ae2d:::benchmark_data_smithi188_89276_object81886:head# 2024-04-23T16:25:15.508 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c597148:::benchmark_data_smithi188_89276_object65062:head# 2024-04-23T16:25:15.509 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c59a09b:::benchmark_data_smithi188_89276_object166081:head# 2024-04-23T16:25:15.509 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c59d326:::benchmark_data_smithi188_89276_object65257:head# 2024-04-23T16:25:15.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c5a9992:::benchmark_data_smithi188_89276_object93284:head# 2024-04-23T16:25:15.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c5aeb23:::benchmark_data_smithi188_89276_object133616:head# 2024-04-23T16:25:15.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c5c4844:::benchmark_data_smithi188_89276_object87052:head# 2024-04-23T16:25:15.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c5c72d1:::benchmark_data_smithi188_89276_object102235:head# 2024-04-23T16:25:15.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c5ca33b:::benchmark_data_smithi188_89276_object107558:head# 2024-04-23T16:25:15.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c5d00a4:::benchmark_data_smithi188_89276_object52879:head# 2024-04-23T16:25:15.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c5d69f5:::benchmark_data_smithi188_89276_object161504:head# 2024-04-23T16:25:15.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c5daddb:::benchmark_data_smithi188_89276_object114963:head# 2024-04-23T16:25:15.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c5dc578:::benchmark_data_smithi188_89276_object69270:head# 2024-04-23T16:25:15.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c5e6136:::benchmark_data_smithi188_89276_object114242:head# 2024-04-23T16:25:15.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c5ea459:::benchmark_data_smithi188_89276_object154459:head# 2024-04-23T16:25:15.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c5fe615:::benchmark_data_smithi188_89276_object121087:head# 2024-04-23T16:25:15.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c60f17f:::benchmark_data_smithi188_89276_object56775:head# 2024-04-23T16:25:15.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c611f63:::benchmark_data_smithi188_89276_object135484:head# 2024-04-23T16:25:15.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c619977:::benchmark_data_smithi188_89276_object124242:head# 2024-04-23T16:25:15.516 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c61ce78:::benchmark_data_smithi188_89276_object58958:head# 2024-04-23T16:25:15.516 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c62095a:::benchmark_data_smithi188_89276_object91875:head# 2024-04-23T16:25:15.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c623453:::benchmark_data_smithi188_89276_object150923:head# 2024-04-23T16:25:15.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c630dbf:::benchmark_data_smithi188_89276_object167632:head# 2024-04-23T16:25:15.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c641af7:::benchmark_data_smithi188_89276_object125226:head# 2024-04-23T16:25:15.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c6433f3:::benchmark_data_smithi188_89276_object114434:head# 2024-04-23T16:25:15.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c643fba:::benchmark_data_smithi188_89276_object98973:head# 2024-04-23T16:25:15.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c65009c:::benchmark_data_smithi188_89276_object88386:head# 2024-04-23T16:25:15.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c65544d:::benchmark_data_smithi188_89276_object58497:head# 2024-04-23T16:25:15.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c655a76:::benchmark_data_smithi188_89276_object119905:head# 2024-04-23T16:25:15.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c66a66b:::benchmark_data_smithi188_89276_object146406:head# 2024-04-23T16:25:15.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c671ca5:::benchmark_data_smithi188_89276_object76301:head# 2024-04-23T16:25:15.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c672a1e:::benchmark_data_smithi188_89276_object50824:head# 2024-04-23T16:25:15.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c67583a:::benchmark_data_smithi188_89276_object166565:head# 2024-04-23T16:25:15.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c678b10:::benchmark_data_smithi188_89276_object41433:head# 2024-04-23T16:25:15.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c67952e:::benchmark_data_smithi188_89276_object62291:head# 2024-04-23T16:25:15.523 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c67c8b6:::benchmark_data_smithi188_89276_object153000:head# 2024-04-23T16:25:15.523 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c68079e:::benchmark_data_smithi188_89276_object94059:head# 2024-04-23T16:25:15.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c6835d8:::benchmark_data_smithi188_89276_object55921:head# 2024-04-23T16:25:15.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c6873e4:::benchmark_data_smithi188_89276_object152278:head# 2024-04-23T16:25:15.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c691c18:::benchmark_data_smithi188_89276_object57062:head# 2024-04-23T16:25:15.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c6985e2:::benchmark_data_smithi188_89276_object112815:head# 2024-04-23T16:25:15.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c6a0c67:::benchmark_data_smithi188_89276_object161787:head# 2024-04-23T16:25:15.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c6a5032:::benchmark_data_smithi188_89276_object139241:head# 2024-04-23T16:25:15.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c6a8e40:::benchmark_data_smithi188_89276_object72622:head# 2024-04-23T16:25:15.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c6a92f4:::benchmark_data_smithi188_89276_object116043:head# 2024-04-23T16:25:15.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c6c7e36:::benchmark_data_smithi188_89276_object146880:head# 2024-04-23T16:25:15.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c6ccefe:::benchmark_data_smithi188_89276_object45352:head# 2024-04-23T16:25:15.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c6d1309:::benchmark_data_smithi188_89276_object70291:head# 2024-04-23T16:25:15.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c6d54b9:::benchmark_data_smithi188_89276_object44348:head# 2024-04-23T16:25:15.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c6ee705:::benchmark_data_smithi188_89276_object145811:head# 2024-04-23T16:25:15.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c6f9dc3:::benchmark_data_smithi188_89276_object53595:head# 2024-04-23T16:25:15.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c710696:::benchmark_data_smithi188_89276_object131993:head# 2024-04-23T16:25:15.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c712b13:::benchmark_data_smithi188_89276_object63503:head# 2024-04-23T16:25:15.531 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c717f12:::benchmark_data_smithi188_89276_object97296:head# 2024-04-23T16:25:15.531 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c71f352:::benchmark_data_smithi188_89276_object42422:head# 2024-04-23T16:25:15.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c735a8f:::benchmark_data_smithi188_89276_object66643:head# 2024-04-23T16:25:15.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c73ea02:::benchmark_data_smithi188_89276_object160603:head# 2024-04-23T16:25:15.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c744921:::benchmark_data_smithi188_89276_object132567:head# 2024-04-23T16:25:15.533 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c747575:::benchmark_data_smithi188_89276_object151188:head# 2024-04-23T16:25:15.533 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c74b5a4:::benchmark_data_smithi188_89276_object123405:head# 2024-04-23T16:25:15.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c74e931:::benchmark_data_smithi188_89276_object85702:head# 2024-04-23T16:25:15.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c74e933:::benchmark_data_smithi188_89276_object41462:head# 2024-04-23T16:25:15.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c750c30:::benchmark_data_smithi188_89276_object63017:head# 2024-04-23T16:25:15.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c75e456:::benchmark_data_smithi188_89276_object111915:head# 2024-04-23T16:25:15.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c7645f5:::benchmark_data_smithi188_89276_object156126:head# 2024-04-23T16:25:15.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c76b174:::benchmark_data_smithi188_89276_object128224:head# 2024-04-23T16:25:15.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c795e12:::benchmark_data_smithi188_89276_object54213:head# 2024-04-23T16:25:15.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c79b7de:::benchmark_data_smithi188_89276_object82717:head# 2024-04-23T16:25:15.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c7b33fa:::benchmark_data_smithi188_89276_object68818:head# 2024-04-23T16:25:15.541 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c7b9d91:::benchmark_data_smithi188_89276_object86248:head# 2024-04-23T16:25:15.541 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c7bc5e4:::benchmark_data_smithi188_89276_object81366:head# 2024-04-23T16:25:15.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c7c1f56:::benchmark_data_smithi188_89276_object114519:head# 2024-04-23T16:25:15.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c7c4442:::benchmark_data_smithi188_89276_object127202:head# 2024-04-23T16:25:15.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c7c7b42:::benchmark_data_smithi188_89276_object71690:head# 2024-04-23T16:25:15.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c7c9db9:::benchmark_data_smithi188_89276_object135754:head# 2024-04-23T16:25:15.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c7ddad8:::benchmark_data_smithi188_89276_object43800:head# 2024-04-23T16:25:15.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c7ec33e:::benchmark_data_smithi188_89276_object150381:head# 2024-04-23T16:25:15.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c80cb22:::benchmark_data_smithi188_89276_object165463:head# 2024-04-23T16:25:15.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c80e070:::benchmark_data_smithi188_89276_object43974:head# 2024-04-23T16:25:15.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c80e690:::benchmark_data_smithi188_89276_object96611:head# 2024-04-23T16:25:15.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c8117c9:::benchmark_data_smithi188_89276_object106938:head# 2024-04-23T16:25:15.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c829f61:::benchmark_data_smithi188_89276_object74947:head# 2024-04-23T16:25:15.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c8470da:::benchmark_data_smithi188_89276_object55027:head# 2024-04-23T16:25:15.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c847986:::benchmark_data_smithi188_89276_object85748:head# 2024-04-23T16:25:15.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c848101:::benchmark_data_smithi188_89276_object63322:head# 2024-04-23T16:25:15.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c84a7ee:::benchmark_data_smithi188_89276_object51018:head# 2024-04-23T16:25:15.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c84d431:::benchmark_data_smithi188_89276_object76921:head# 2024-04-23T16:25:15.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c85163d:::benchmark_data_smithi188_89276_object62222:head# 2024-04-23T16:25:15.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c857535:::benchmark_data_smithi188_89276_object103568:head# 2024-04-23T16:25:15.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c85946e:::benchmark_data_smithi188_89276_object46480:head# 2024-04-23T16:25:15.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c860b09:::benchmark_data_smithi188_89276_object75714:head# 2024-04-23T16:25:15.552 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c86b97a:::benchmark_data_smithi188_89276_object151739:head# 2024-04-23T16:25:15.552 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c8707d9:::benchmark_data_smithi188_89276_object110923:head# 2024-04-23T16:25:15.555 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c874f30:::benchmark_data_smithi188_89276_object164233:head# 2024-04-23T16:25:15.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c87ddff:::benchmark_data_smithi188_89276_object87282:head# 2024-04-23T16:25:15.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c885820:::benchmark_data_smithi188_89276_object65244:head# 2024-04-23T16:25:15.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c88c807:::benchmark_data_smithi188_89276_object53361:head# 2024-04-23T16:25:15.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c88e270:::benchmark_data_smithi188_89276_object154951:head# 2024-04-23T16:25:15.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c8973c5:::benchmark_data_smithi188_89276_object64144:head# 2024-04-23T16:25:15.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c89b982:::benchmark_data_smithi188_89276_object44943:head# 2024-04-23T16:25:15.558 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c8ab04c:::benchmark_data_smithi188_89276_object81798:head# 2024-04-23T16:25:15.558 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c8ad193:::benchmark_data_smithi188_89276_object149279:head# 2024-04-23T16:25:15.558 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c8b862e:::benchmark_data_smithi188_89276_object129224:head# 2024-04-23T16:25:15.559 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c8cb076:::benchmark_data_smithi188_89276_object51482:head# 2024-04-23T16:25:15.559 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c8d24ca:::benchmark_data_smithi188_89276_object119291:head# 2024-04-23T16:25:15.560 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c8da1c8:::benchmark_data_smithi188_89276_object65565:head# 2024-04-23T16:25:15.560 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c8fc585:::benchmark_data_smithi188_89276_object149134:head# 2024-04-23T16:25:15.560 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c9017b6:::benchmark_data_smithi188_89276_object89005:head# 2024-04-23T16:25:15.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c90430d:::benchmark_data_smithi188_89276_object72159:head# 2024-04-23T16:25:15.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c90f357:::benchmark_data_smithi188_89276_object40273:head# 2024-04-23T16:25:15.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c9103f4:::benchmark_data_smithi188_89276_object146291:head# 2024-04-23T16:25:15.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c91de8f:::benchmark_data_smithi188_89276_object92257:head# 2024-04-23T16:25:15.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c933a24:::benchmark_data_smithi188_89276_object137690:head# 2024-04-23T16:25:15.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c93f122:::benchmark_data_smithi188_89276_object157494:head# 2024-04-23T16:25:15.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c946314:::benchmark_data_smithi188_89276_object83833:head# 2024-04-23T16:25:15.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c957616:::benchmark_data_smithi188_89276_object84600:head# 2024-04-23T16:25:15.564 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c962b59:::benchmark_data_smithi188_89276_object86163:head# 2024-04-23T16:25:15.565 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c96cd2d:::benchmark_data_smithi188_89276_object80624:head# 2024-04-23T16:25:15.565 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c971f86:::benchmark_data_smithi188_89276_object94836:head# 2024-04-23T16:25:15.565 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c978464:::benchmark_data_smithi188_89276_object120356:head# 2024-04-23T16:25:15.565 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c97ee95:::benchmark_data_smithi188_89276_object164212:head# 2024-04-23T16:25:15.566 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c9863ba:::benchmark_data_smithi188_89276_object50642:head# 2024-04-23T16:25:15.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c98fd50:::benchmark_data_smithi188_89276_object46587:head# 2024-04-23T16:25:15.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c995b91:::benchmark_data_smithi188_89276_object168614:head# 2024-04-23T16:25:15.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c9961a1:::benchmark_data_smithi188_89276_object113420:head# 2024-04-23T16:25:15.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c996891:::benchmark_data_smithi188_89276_object139086:head# 2024-04-23T16:25:15.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c9d3a70:::benchmark_data_smithi188_89276_object115814:head# 2024-04-23T16:25:15.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c9db14a:::benchmark_data_smithi188_89276_object75549:head# 2024-04-23T16:25:15.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c9dfa74:::benchmark_data_smithi188_89276_object99332:head# 2024-04-23T16:25:15.571 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c9ed4c0:::benchmark_data_smithi188_89276_object42113:head# 2024-04-23T16:25:15.571 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c9f03b2:::benchmark_data_smithi188_89276_object59664:head# 2024-04-23T16:25:15.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c9f3bb0:::benchmark_data_smithi188_89276_object72972:head# 2024-04-23T16:25:15.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c9f6d2d:::benchmark_data_smithi188_89276_object167992:head# 2024-04-23T16:25:15.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c9f7ba8:::benchmark_data_smithi188_89276_object87139:head# 2024-04-23T16:25:15.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c9f80b5:::benchmark_data_smithi188_89276_object164176:head# 2024-04-23T16:25:15.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c9fdc24:::benchmark_data_smithi188_89276_object80385:head# 2024-04-23T16:25:15.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2c9ffed4:::benchmark_data_smithi188_89276_object140629:head# 2024-04-23T16:25:15.574 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ca10d55:::benchmark_data_smithi188_89276_object151518:head# 2024-04-23T16:25:15.574 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ca1a8d8:::benchmark_data_smithi188_89276_object167879:head# 2024-04-23T16:25:15.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ca239c7:::benchmark_data_smithi188_89276_object141876:head# 2024-04-23T16:25:15.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ca25e79:::benchmark_data_smithi188_89276_object162829:head# 2024-04-23T16:25:15.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ca4c764:::benchmark_data_smithi188_89276_object116259:head# 2024-04-23T16:25:15.577 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ca4d6b8:::benchmark_data_smithi188_89276_object88234:head# 2024-04-23T16:25:15.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ca54e88:::benchmark_data_smithi188_89276_object41055:head# 2024-04-23T16:25:15.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ca5b06b:::benchmark_data_smithi188_89276_object99190:head# 2024-04-23T16:25:15.579 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ca61a26:::benchmark_data_smithi188_89276_object55955:head# 2024-04-23T16:25:15.579 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ca6a4c3:::benchmark_data_smithi188_89276_object75122:head# 2024-04-23T16:25:15.579 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ca8e1ad:::benchmark_data_smithi188_89276_object113329:head# 2024-04-23T16:25:15.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ca92984:::benchmark_data_smithi188_89276_object83770:head# 2024-04-23T16:25:15.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ca9dae8:::benchmark_data_smithi188_89276_object154329:head# 2024-04-23T16:25:15.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ca9ea7a:::benchmark_data_smithi188_89276_object118583:head# 2024-04-23T16:25:15.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ca9ec1a:::benchmark_data_smithi188_89276_object66134:head# 2024-04-23T16:25:15.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2caa2fc0:::benchmark_data_smithi188_89276_object135737:head# 2024-04-23T16:25:15.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2caa840c:::benchmark_data_smithi188_89276_object93532:head# 2024-04-23T16:25:15.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2caaf367:::benchmark_data_smithi188_89276_object129647:head# 2024-04-23T16:25:15.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cabe337:::benchmark_data_smithi188_89276_object121453:head# 2024-04-23T16:25:15.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cad1682:::benchmark_data_smithi188_89276_object46986:head# 2024-04-23T16:25:15.585 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cad9b99:::benchmark_data_smithi188_89276_object74223:head# 2024-04-23T16:25:15.585 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cadc5f1:::benchmark_data_smithi188_89276_object167413:head# 2024-04-23T16:25:15.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2caddc1c:::benchmark_data_smithi188_89276_object155130:head# 2024-04-23T16:25:15.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cae6048:::benchmark_data_smithi188_89276_object138533:head# 2024-04-23T16:25:15.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2caf02d9:::benchmark_data_smithi188_89276_object41632:head# 2024-04-23T16:25:15.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cb0f71d:::benchmark_data_smithi188_89276_object147811:head# 2024-04-23T16:25:15.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cb117b6:::benchmark_data_smithi188_89276_object95388:head# 2024-04-23T16:25:15.588 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cb22017:::benchmark_data_smithi188_89276_object76417:head# 2024-04-23T16:25:15.588 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cb27bd4:::benchmark_data_smithi188_89276_object120036:head# 2024-04-23T16:25:15.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cb2e0cb:::benchmark_data_smithi188_89276_object109512:head# 2024-04-23T16:25:15.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cb32abe:::benchmark_data_smithi188_89276_object62954:head# 2024-04-23T16:25:15.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cb3b0ba:::benchmark_data_smithi188_89276_object88888:head# 2024-04-23T16:25:15.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cb41611:::benchmark_data_smithi188_89276_object118846:head# 2024-04-23T16:25:15.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cb44604:::benchmark_data_smithi188_89276_object63983:head# 2024-04-23T16:25:15.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cb57edf:::benchmark_data_smithi188_89276_object98224:head# 2024-04-23T16:25:15.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cb5830f:::benchmark_data_smithi188_89276_object90271:head# 2024-04-23T16:25:15.592 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cb6ade8:::benchmark_data_smithi188_89276_object74946:head# 2024-04-23T16:25:15.593 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cb6d52e:::benchmark_data_smithi188_89276_object149138:head# 2024-04-23T16:25:15.596 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cb7fa60:::benchmark_data_smithi188_89276_object132859:head# 2024-04-23T16:25:15.596 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cb82c3c:::benchmark_data_smithi188_89276_object117295:head# 2024-04-23T16:25:15.596 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cb9e6a1:::benchmark_data_smithi188_89276_object149252:head# 2024-04-23T16:25:15.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cbaae89:::benchmark_data_smithi188_89276_object135628:head# 2024-04-23T16:25:15.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cbb02f1:::benchmark_data_smithi188_89276_object67236:head# 2024-04-23T16:25:15.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cbb093d:::benchmark_data_smithi188_89276_object72138:head# 2024-04-23T16:25:15.598 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cbb4f88:::benchmark_data_smithi188_89276_object120860:head# 2024-04-23T16:25:15.598 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cbb8561:::benchmark_data_smithi188_89276_object92173:head# 2024-04-23T16:25:15.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cbb8cef:::benchmark_data_smithi188_89276_object118128:head# 2024-04-23T16:25:15.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cbbad5e:::benchmark_data_smithi188_89276_object48568:head# 2024-04-23T16:25:15.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cbc3423:::benchmark_data_smithi188_89276_object123166:head# 2024-04-23T16:25:15.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cbc3e76:::benchmark_data_smithi188_89276_object72762:head# 2024-04-23T16:25:15.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cbc45a1:::benchmark_data_smithi188_89276_object109081:head# 2024-04-23T16:25:15.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cbc893f:::benchmark_data_smithi188_89276_object75975:head# 2024-04-23T16:25:15.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cbdca1f:::benchmark_data_smithi188_89276_object164996:head# 2024-04-23T16:25:15.602 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cbed521:::benchmark_data_smithi188_89276_object55219:head# 2024-04-23T16:25:15.602 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cbfa628:::benchmark_data_smithi188_89276_object141082:head# 2024-04-23T16:25:15.602 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cbfdaad:::benchmark_data_smithi188_89276_object59055:head# 2024-04-23T16:25:15.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cbfff27:::benchmark_data_smithi188_89276_object138996:head# 2024-04-23T16:25:15.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cc081a7:::benchmark_data_smithi188_89276_object137443:head# 2024-04-23T16:25:15.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cc0825d:::benchmark_data_smithi188_89276_object96360:head# 2024-04-23T16:25:15.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cc1068d:::benchmark_data_smithi188_89276_object136281:head# 2024-04-23T16:25:15.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cc2047f:::benchmark_data_smithi188_89276_object56848:head# 2024-04-23T16:25:15.605 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cc29377:::benchmark_data_smithi188_89276_object101783:head# 2024-04-23T16:25:15.605 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cc2be56:::benchmark_data_smithi188_89276_object43722:head# 2024-04-23T16:25:15.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cc394b8:::benchmark_data_smithi188_89276_object90323:head# 2024-04-23T16:25:15.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cc3a9ce:::benchmark_data_smithi188_89276_object111558:head# 2024-04-23T16:25:15.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cc3fb03:::benchmark_data_smithi188_89276_object52111:head# 2024-04-23T16:25:15.609 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cc44879:::benchmark_data_smithi188_89276_object117439:head# 2024-04-23T16:25:15.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cc4a9cf:::benchmark_data_smithi188_89276_object69881:head# 2024-04-23T16:25:15.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cc57997:::benchmark_data_smithi188_89276_object68469:head# 2024-04-23T16:25:15.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cc60e7d:::benchmark_data_smithi188_89276_object145504:head# 2024-04-23T16:25:15.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cc67a58:::benchmark_data_smithi188_89276_object48580:head# 2024-04-23T16:25:15.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cc7cee8:::benchmark_data_smithi188_89276_object142945:head# 2024-04-23T16:25:15.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cc7f5f7:::benchmark_data_smithi188_89276_object97363:head# 2024-04-23T16:25:15.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cc9c46a:::benchmark_data_smithi188_89276_object138673:head# 2024-04-23T16:25:15.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cca2617:::benchmark_data_smithi188_89276_object135166:head# 2024-04-23T16:25:15.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cca37af:::benchmark_data_smithi188_89276_object110447:head# 2024-04-23T16:25:15.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ccb7753:::benchmark_data_smithi188_89276_object157120:head# 2024-04-23T16:25:15.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ccbc99f:::benchmark_data_smithi188_89276_object76008:head# 2024-04-23T16:25:15.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ccbdb19:::benchmark_data_smithi188_89276_object75431:head# 2024-04-23T16:25:15.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ccbfd88:::benchmark_data_smithi188_89276_object66532:head# 2024-04-23T16:25:15.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ccc407e:::benchmark_data_smithi188_89276_object143519:head# 2024-04-23T16:25:15.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ccc804f:::benchmark_data_smithi188_89276_object149266:head# 2024-04-23T16:25:15.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cccc318:::benchmark_data_smithi188_89276_object83602:head# 2024-04-23T16:25:15.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ccd27fc:::benchmark_data_smithi188_89276_object116856:head# 2024-04-23T16:25:15.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cce10f0:::benchmark_data_smithi188_89276_object152552:head# 2024-04-23T16:25:15.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cce90cc:::benchmark_data_smithi188_89276_object55652:head# 2024-04-23T16:25:15.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cce928a:::benchmark_data_smithi188_89276_object150757:head# 2024-04-23T16:25:15.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ccf00d0:::benchmark_data_smithi188_89276_object95876:head# 2024-04-23T16:25:15.616 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ccf0b8b:::benchmark_data_smithi188_89276_object152022:head# 2024-04-23T16:25:15.616 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cd03792:::benchmark_data_smithi188_89276_object96272:head# 2024-04-23T16:25:15.617 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cd0e216:::benchmark_data_smithi188_89276_object158541:head# 2024-04-23T16:25:15.617 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cd16280:::benchmark_data_smithi188_89276_object135978:head# 2024-04-23T16:25:15.618 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cd1e62f:::benchmark_data_smithi188_89276_object75499:head# 2024-04-23T16:25:15.618 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cd2ca75:::benchmark_data_smithi188_89276_object85170:head# 2024-04-23T16:25:15.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cd3daa6:::benchmark_data_smithi188_89276_object92451:head# 2024-04-23T16:25:15.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cd4aab9:::benchmark_data_smithi188_89276_object42674:head# 2024-04-23T16:25:15.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cd5a769:::benchmark_data_smithi188_89276_object79245:head# 2024-04-23T16:25:15.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cd6167d:::benchmark_data_smithi188_89276_object108889:head# 2024-04-23T16:25:15.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cd69c3e:::benchmark_data_smithi188_89276_object158995:head# 2024-04-23T16:25:15.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cd794e5:::benchmark_data_smithi188_89276_object141983:head# 2024-04-23T16:25:15.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cd872e5:::benchmark_data_smithi188_89276_object95266:head# 2024-04-23T16:25:15.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cd9462d:::benchmark_data_smithi188_89276_object152069:head# 2024-04-23T16:25:15.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cdb1d1f:::benchmark_data_smithi188_89276_object57550:head# 2024-04-23T16:25:15.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cdb48d6:::benchmark_data_smithi188_89276_object73182:head# 2024-04-23T16:25:15.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cdba610:::benchmark_data_smithi188_89276_object73567:head# 2024-04-23T16:25:15.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cdba72b:::benchmark_data_smithi188_89276_object103320:head# 2024-04-23T16:25:15.624 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cdc76af:::benchmark_data_smithi188_89276_object166931:head# 2024-04-23T16:25:15.624 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cdc78e9:::benchmark_data_smithi188_89276_object89917:head# 2024-04-23T16:25:15.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cdcc314:::benchmark_data_smithi188_89276_object153021:head# 2024-04-23T16:25:15.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cdcc7fd:::benchmark_data_smithi188_89276_object85328:head# 2024-04-23T16:25:15.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cdd2393:::benchmark_data_smithi188_89276_object75928:head# 2024-04-23T16:25:15.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cdd308d:::benchmark_data_smithi188_89276_object62277:head# 2024-04-23T16:25:15.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cddd871:::benchmark_data_smithi188_89276_object51243:head# 2024-04-23T16:25:15.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cddf7fd:::benchmark_data_smithi188_89276_object152731:head# 2024-04-23T16:25:15.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cde5b18:::benchmark_data_smithi188_89276_object133747:head# 2024-04-23T16:25:15.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cdfa0de:::benchmark_data_smithi188_89276_object55923:head# 2024-04-23T16:25:15.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cdfcdac:::benchmark_data_smithi188_89276_object82946:head# 2024-04-23T16:25:15.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce014f4:::benchmark_data_smithi188_89276_object66630:head# 2024-04-23T16:25:15.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce0c1ae:::benchmark_data_smithi188_89276_object77665:head# 2024-04-23T16:25:15.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce11960:::benchmark_data_smithi188_89276_object77441:head# 2024-04-23T16:25:15.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce15c1d:::benchmark_data_smithi188_89276_object129937:head# 2024-04-23T16:25:15.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce1a870:::benchmark_data_smithi188_89276_object78639:head# 2024-04-23T16:25:15.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce25e98:::benchmark_data_smithi188_89276_object168569:head# 2024-04-23T16:25:15.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce443fa:::benchmark_data_smithi188_89276_object151228:head# 2024-04-23T16:25:15.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce4922d:::benchmark_data_smithi188_89276_object145422:head# 2024-04-23T16:25:15.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce5250e:::benchmark_data_smithi188_89276_object147891:head# 2024-04-23T16:25:15.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce6200a:::benchmark_data_smithi188_89276_object54837:head# 2024-04-23T16:25:15.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce63343:::benchmark_data_smithi188_89276_object80672:head# 2024-04-23T16:25:15.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce6b1b8:::benchmark_data_smithi188_89276_object104106:head# 2024-04-23T16:25:15.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce70b13:::benchmark_data_smithi188_89276_object127594:head# 2024-04-23T16:25:15.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce768a3:::benchmark_data_smithi188_89276_object162292:head# 2024-04-23T16:25:15.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce7a858:::benchmark_data_smithi188_89276_object95306:head# 2024-04-23T16:25:15.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce8b7a7:::benchmark_data_smithi188_89276_object104868:head# 2024-04-23T16:25:15.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce8c954:::benchmark_data_smithi188_89276_object44336:head# 2024-04-23T16:25:15.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce91571:::benchmark_data_smithi188_89276_object76134:head# 2024-04-23T16:25:15.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce97068:::benchmark_data_smithi188_89276_object148111:head# 2024-04-23T16:25:15.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce9734d:::benchmark_data_smithi188_89276_object102294:head# 2024-04-23T16:25:15.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ce98440:::benchmark_data_smithi188_89276_object58349:head# 2024-04-23T16:25:15.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cea714d:::benchmark_data_smithi188_89276_object95750:head# 2024-04-23T16:25:15.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ceaaf41:::benchmark_data_smithi188_89276_object43369:head# 2024-04-23T16:25:15.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ceb4204:::benchmark_data_smithi188_89276_object149475:head# 2024-04-23T16:25:15.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ceb4db7:::benchmark_data_smithi188_89276_object157105:head# 2024-04-23T16:25:15.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cebeec7:::benchmark_data_smithi188_89276_object91903:head# 2024-04-23T16:25:15.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cec3982:::benchmark_data_smithi188_89276_object124876:head# 2024-04-23T16:25:15.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ced3c7d:::benchmark_data_smithi188_89276_object41312:head# 2024-04-23T16:25:15.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cedabb3:::benchmark_data_smithi188_89276_object127549:head# 2024-04-23T16:25:15.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cee45aa:::benchmark_data_smithi188_89276_object43525:head# 2024-04-23T16:25:15.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ceebc24:::benchmark_data_smithi188_89276_object43999:head# 2024-04-23T16:25:15.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ceefd37:::benchmark_data_smithi188_89276_object144843:head# 2024-04-23T16:25:15.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cef2323:::benchmark_data_smithi188_89276_object52730:head# 2024-04-23T16:25:15.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cef53f5:::benchmark_data_smithi188_89276_object87637:head# 2024-04-23T16:25:15.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cef852c:::benchmark_data_smithi188_89276_object78337:head# 2024-04-23T16:25:15.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf094d1:::benchmark_data_smithi188_89276_object168280:head# 2024-04-23T16:25:15.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf0bd0c:::benchmark_data_smithi188_89276_object83158:head# 2024-04-23T16:25:15.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf11249:::benchmark_data_smithi188_89276_object46908:head# 2024-04-23T16:25:15.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf1b2f6:::benchmark_data_smithi188_89276_object82216:head# 2024-04-23T16:25:15.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf1c5be:::benchmark_data_smithi188_89276_object81595:head# 2024-04-23T16:25:15.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf2125e:::benchmark_data_smithi188_89276_object138893:head# 2024-04-23T16:25:15.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf2ce3d:::benchmark_data_smithi188_89276_object138197:head# 2024-04-23T16:25:15.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf2e05f:::benchmark_data_smithi188_89276_object97904:head# 2024-04-23T16:25:15.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf38f9f:::benchmark_data_smithi188_89276_object130223:head# 2024-04-23T16:25:15.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf3b4bc:::benchmark_data_smithi188_89276_object159140:head# 2024-04-23T16:25:15.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf43fca:::benchmark_data_smithi188_89276_object125434:head# 2024-04-23T16:25:15.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf4af96:::benchmark_data_smithi188_89276_object131821:head# 2024-04-23T16:25:15.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf4b384:::benchmark_data_smithi188_89276_object132510:head# 2024-04-23T16:25:15.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf53abb:::benchmark_data_smithi188_89276_object153309:head# 2024-04-23T16:25:15.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf53df7:::benchmark_data_smithi188_89276_object75405:head# 2024-04-23T16:25:15.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf5f022:::benchmark_data_smithi188_89276_object63204:head# 2024-04-23T16:25:15.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf651b3:::benchmark_data_smithi188_89276_object142557:head# 2024-04-23T16:25:15.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf68282:::benchmark_data_smithi188_89276_object117814:head# 2024-04-23T16:25:15.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf704bc:::benchmark_data_smithi188_89276_object102265:head# 2024-04-23T16:25:15.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf7252b:::benchmark_data_smithi188_89276_object62891:head# 2024-04-23T16:25:15.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf7da4e:::benchmark_data_smithi188_89276_object59295:head# 2024-04-23T16:25:15.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf93c65:::benchmark_data_smithi188_89276_object89414:head# 2024-04-23T16:25:15.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf95a16:::benchmark_data_smithi188_89276_object149322:head# 2024-04-23T16:25:15.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cf9e7f8:::benchmark_data_smithi188_89276_object165475:head# 2024-04-23T16:25:15.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cfa01bf:::benchmark_data_smithi188_89276_object62957:head# 2024-04-23T16:25:15.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cfa05ff:::benchmark_data_smithi188_89276_object98145:head# 2024-04-23T16:25:15.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cfa1c30:::benchmark_data_smithi188_89276_object61908:head# 2024-04-23T16:25:15.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cfae6b0:::benchmark_data_smithi188_89276_object54963:head# 2024-04-23T16:25:15.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cfb074c:::benchmark_data_smithi188_89276_object143957:head# 2024-04-23T16:25:15.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cfb36c6:::benchmark_data_smithi188_89276_object68900:head# 2024-04-23T16:25:15.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cfb75ac:::benchmark_data_smithi188_89276_object147989:head# 2024-04-23T16:25:15.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cfbe543:::benchmark_data_smithi188_89276_object151547:head# 2024-04-23T16:25:15.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cfc000e:::benchmark_data_smithi188_89276_object47986:head# 2024-04-23T16:25:15.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cfc7d79:::benchmark_data_smithi188_89276_object157460:head# 2024-04-23T16:25:15.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cfca13a:::benchmark_data_smithi188_89276_object85811:head# 2024-04-23T16:25:15.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cfca241:::benchmark_data_smithi188_89276_object42951:head# 2024-04-23T16:25:15.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cfcb5ce:::benchmark_data_smithi188_89276_object76007:head# 2024-04-23T16:25:15.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cfcc938:::benchmark_data_smithi188_89276_object131167:head# 2024-04-23T16:25:15.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cfe63c0:::benchmark_data_smithi188_89276_object73760:head# 2024-04-23T16:25:15.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cfeb0b2:::benchmark_data_smithi188_89276_object142807:head# 2024-04-23T16:25:15.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cffafd2:::benchmark_data_smithi188_89276_object137788:head# 2024-04-23T16:25:15.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2cffbc33:::benchmark_data_smithi188_89276_object83206:head# 2024-04-23T16:25:15.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d00fe60:::benchmark_data_smithi188_89276_object106883:head# 2024-04-23T16:25:15.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d01e70e:::benchmark_data_smithi188_89276_object118781:head# 2024-04-23T16:25:15.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d0217dc:::benchmark_data_smithi188_89276_object87997:head# 2024-04-23T16:25:15.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d022364:::benchmark_data_smithi188_89276_object149660:head# 2024-04-23T16:25:15.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d026a5c:::benchmark_data_smithi188_89276_object104600:head# 2024-04-23T16:25:15.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d027599:::benchmark_data_smithi188_89276_object57237:head# 2024-04-23T16:25:15.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d028ece:::benchmark_data_smithi188_89276_object120214:head# 2024-04-23T16:25:15.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d03f251:::benchmark_data_smithi188_89276_object130665:head# 2024-04-23T16:25:15.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d04a744:::benchmark_data_smithi188_89276_object90764:head# 2024-04-23T16:25:15.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d04b482:::benchmark_data_smithi188_89276_object155099:head# 2024-04-23T16:25:15.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d056271:::benchmark_data_smithi188_89276_object98456:head# 2024-04-23T16:25:15.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d05972e:::benchmark_data_smithi188_89276_object159314:head# 2024-04-23T16:25:15.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d063269:::benchmark_data_smithi188_89276_object58516:head# 2024-04-23T16:25:15.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d071b8c:::benchmark_data_smithi188_89276_object98701:head# 2024-04-23T16:25:15.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d0791c7:::benchmark_data_smithi188_89276_object57267:head# 2024-04-23T16:25:15.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d0828cc:::benchmark_data_smithi188_89276_object84902:head# 2024-04-23T16:25:15.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d084043:::benchmark_data_smithi188_89276_object70876:head# 2024-04-23T16:25:15.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d08da6a:::benchmark_data_smithi188_89276_object165429:head# 2024-04-23T16:25:15.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d0940bb:::benchmark_data_smithi188_89276_object132343:head# 2024-04-23T16:25:15.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d09798f:::benchmark_data_smithi188_89276_object88038:head# 2024-04-23T16:25:15.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d0aafcc:::benchmark_data_smithi188_89276_object127680:head# 2024-04-23T16:25:15.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d0aca37:::benchmark_data_smithi188_89276_object111042:head# 2024-04-23T16:25:15.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d0b0527:::benchmark_data_smithi188_89276_object143322:head# 2024-04-23T16:25:15.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d0c58f9:::benchmark_data_smithi188_89276_object130704:head# 2024-04-23T16:25:15.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d0c8d57:::benchmark_data_smithi188_89276_object87215:head# 2024-04-23T16:25:15.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d0cc810:::benchmark_data_smithi188_89276_object82943:head# 2024-04-23T16:25:15.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d0cf040:::benchmark_data_smithi188_89276_object76621:head# 2024-04-23T16:25:15.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d0d1149:::benchmark_data_smithi188_89276_object113405:head# 2024-04-23T16:25:15.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d0d9242:::benchmark_data_smithi188_89276_object68426:head# 2024-04-23T16:25:15.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d0dcc04:::benchmark_data_smithi188_89276_object88764:head# 2024-04-23T16:25:15.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d0dce2e:::benchmark_data_smithi188_89276_object100511:head# 2024-04-23T16:25:15.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d0e7d7a:::benchmark_data_smithi188_89276_object75364:head# 2024-04-23T16:25:15.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d0eaa61:::benchmark_data_smithi188_89276_object144144:head# 2024-04-23T16:25:15.678 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d0f250e:::benchmark_data_smithi188_89276_object66214:head# 2024-04-23T16:25:15.678 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d0f9c6c:::benchmark_data_smithi188_89276_object79684:head# 2024-04-23T16:25:15.679 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d11ff50:::benchmark_data_smithi188_89276_object78127:head# 2024-04-23T16:25:15.679 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d12a3f3:::benchmark_data_smithi188_89276_object117052:head# 2024-04-23T16:25:15.679 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d13c0b2:::benchmark_data_smithi188_89276_object150080:head# 2024-04-23T16:25:15.680 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d1544b9:::benchmark_data_smithi188_89276_object143727:head# 2024-04-23T16:25:15.680 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d156259:::benchmark_data_smithi188_89276_object75219:head# 2024-04-23T16:25:15.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d158268:::benchmark_data_smithi188_89276_object90667:head# 2024-04-23T16:25:15.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d16ef96:::benchmark_data_smithi188_89276_object142926:head# 2024-04-23T16:25:15.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d175643:::benchmark_data_smithi188_89276_object102634:head# 2024-04-23T16:25:15.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d177784:::benchmark_data_smithi188_89276_object125635:head# 2024-04-23T16:25:15.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d1792a4:::benchmark_data_smithi188_89276_object70675:head# 2024-04-23T16:25:15.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d184eae:::benchmark_data_smithi188_89276_object77976:head# 2024-04-23T16:25:15.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d187139:::benchmark_data_smithi188_89276_object46368:head# 2024-04-23T16:25:15.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d18dc88:::benchmark_data_smithi188_89276_object81297:head# 2024-04-23T16:25:15.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d19bacf:::benchmark_data_smithi188_89276_object101415:head# 2024-04-23T16:25:15.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d19d6e1:::benchmark_data_smithi188_89276_object154464:head# 2024-04-23T16:25:15.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d1a9d09:::benchmark_data_smithi188_89276_object118387:head# 2024-04-23T16:25:15.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d1b745e:::benchmark_data_smithi188_89276_object58644:head# 2024-04-23T16:25:15.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d1c0afa:::benchmark_data_smithi188_89276_object136757:head# 2024-04-23T16:25:15.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d1c4d03:::benchmark_data_smithi188_89276_object83673:head# 2024-04-23T16:25:15.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d1c8088:::benchmark_data_smithi188_89276_object98273:head# 2024-04-23T16:25:15.689 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d1d4a26:::benchmark_data_smithi188_89276_object82326:head# 2024-04-23T16:25:15.689 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d1d550f:::benchmark_data_smithi188_89276_object142537:head# 2024-04-23T16:25:15.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d1dd2e6:::benchmark_data_smithi188_89276_object164886:head# 2024-04-23T16:25:15.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d1de97c:::benchmark_data_smithi188_89276_object57417:head# 2024-04-23T16:25:15.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d1e40f9:::benchmark_data_smithi188_89276_object69191:head# 2024-04-23T16:25:15.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d1edc64:::benchmark_data_smithi188_89276_object123522:head# 2024-04-23T16:25:15.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d1fa974:::benchmark_data_smithi188_89276_object52290:head# 2024-04-23T16:25:15.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d1fb5c6:::benchmark_data_smithi188_89276_object106468:head# 2024-04-23T16:25:15.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d21b4fa:::benchmark_data_smithi188_89276_object131364:head# 2024-04-23T16:25:15.694 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d22d4ec:::benchmark_data_smithi188_89276_object128385:head# 2024-04-23T16:25:15.694 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d22e897:::benchmark_data_smithi188_89276_object106785:head# 2024-04-23T16:25:15.694 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d239b91:::benchmark_data_smithi188_89276_object135575:head# 2024-04-23T16:25:15.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d239c8c:::benchmark_data_smithi188_89276_object106728:head# 2024-04-23T16:25:15.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d2440a4:::benchmark_data_smithi188_89276_object85268:head# 2024-04-23T16:25:15.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d2461c0:::benchmark_data_smithi188_89276_object67977:head# 2024-04-23T16:25:15.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d24856c:::benchmark_data_smithi188_89276_object153600:head# 2024-04-23T16:25:15.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d250ff1:::benchmark_data_smithi188_89276_object126586:head# 2024-04-23T16:25:15.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d27543a:::benchmark_data_smithi188_89276_object54162:head# 2024-04-23T16:25:15.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d276ac5:::benchmark_data_smithi188_89276_object148502:head# 2024-04-23T16:25:15.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d278d22:::benchmark_data_smithi188_89276_object47857:head# 2024-04-23T16:25:15.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d27a9af:::benchmark_data_smithi188_89276_object148574:head# 2024-04-23T16:25:15.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d284b78:::benchmark_data_smithi188_89276_object105525:head# 2024-04-23T16:25:15.699 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d28ecf0:::benchmark_data_smithi188_89276_object138061:head# 2024-04-23T16:25:15.699 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d296435:::benchmark_data_smithi188_89276_object79624:head# 2024-04-23T16:25:15.699 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d2b583b:::benchmark_data_smithi188_89276_object64836:head# 2024-04-23T16:25:15.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d2b9323:::benchmark_data_smithi188_89276_object54535:head# 2024-04-23T16:25:15.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d2b9e74:::benchmark_data_smithi188_89276_object108258:head# 2024-04-23T16:25:15.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d2cee9f:::benchmark_data_smithi188_89276_object151653:head# 2024-04-23T16:25:15.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d2d7c06:::benchmark_data_smithi188_89276_object139054:head# 2024-04-23T16:25:15.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d2ddd39:::benchmark_data_smithi188_89276_object162821:head# 2024-04-23T16:25:15.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d2de75e:::benchmark_data_smithi188_89276_object68814:head# 2024-04-23T16:25:15.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d2e0671:::benchmark_data_smithi188_89276_object69914:head# 2024-04-23T16:25:15.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d2fa4de:::benchmark_data_smithi188_89276_object72733:head# 2024-04-23T16:25:15.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d2ff847:::benchmark_data_smithi188_89276_object67410:head# 2024-04-23T16:25:15.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d30d397:::benchmark_data_smithi188_89276_object90874:head# 2024-04-23T16:25:15.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d30e18a:::benchmark_data_smithi188_89276_object60700:head# 2024-04-23T16:25:15.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d31c6ed:::benchmark_data_smithi188_89276_object58473:head# 2024-04-23T16:25:15.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d31f81b:::benchmark_data_smithi188_89276_object107118:head# 2024-04-23T16:25:15.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d321cdc:::benchmark_data_smithi188_89276_object114685:head# 2024-04-23T16:25:15.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d32673e:::benchmark_data_smithi188_89276_object81078:head# 2024-04-23T16:25:15.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d3295a5:::benchmark_data_smithi188_89276_object45919:head# 2024-04-23T16:25:15.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d32fe8c:::benchmark_data_smithi188_89276_object166509:head# 2024-04-23T16:25:15.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d332ea7:::benchmark_data_smithi188_89276_object152640:head# 2024-04-23T16:25:15.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d3581a3:::benchmark_data_smithi188_89276_object89184:head# 2024-04-23T16:25:15.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d364664:::benchmark_data_smithi188_89276_object72273:head# 2024-04-23T16:25:15.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d366451:::benchmark_data_smithi188_89276_object129502:head# 2024-04-23T16:25:15.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d36ca2d:::benchmark_data_smithi188_89276_object81571:head# 2024-04-23T16:25:15.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d36ed4e:::benchmark_data_smithi188_89276_object161784:head# 2024-04-23T16:25:15.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d3748e8:::benchmark_data_smithi188_89276_object136105:head# 2024-04-23T16:25:15.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d374c38:::benchmark_data_smithi188_89276_object96298:head# 2024-04-23T16:25:15.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d378840:::benchmark_data_smithi188_89276_object41121:head# 2024-04-23T16:25:15.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d393f52:::benchmark_data_smithi188_89276_object70756:head# 2024-04-23T16:25:15.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d39e8de:::benchmark_data_smithi188_89276_object151327:head# 2024-04-23T16:25:15.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d3abf0f:::benchmark_data_smithi188_89276_object61989:head# 2024-04-23T16:25:15.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d3b356b:::benchmark_data_smithi188_89276_object167990:head# 2024-04-23T16:25:15.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d3c1bbe:::benchmark_data_smithi188_89276_object141529:head# 2024-04-23T16:25:15.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d3c3962:::benchmark_data_smithi188_89276_object72387:head# 2024-04-23T16:25:15.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d3c3ae3:::benchmark_data_smithi188_89276_object153086:head# 2024-04-23T16:25:15.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d3c5e17:::benchmark_data_smithi188_89276_object75378:head# 2024-04-23T16:25:15.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d3d8176:::benchmark_data_smithi188_89276_object97561:head# 2024-04-23T16:25:15.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d3d8253:::benchmark_data_smithi188_89276_object73968:head# 2024-04-23T16:25:15.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d3ef1d6:::benchmark_data_smithi188_89276_object153618:head# 2024-04-23T16:25:15.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d3f97ac:::benchmark_data_smithi188_89276_object115837:head# 2024-04-23T16:25:15.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d3fee7d:::benchmark_data_smithi188_89276_object143256:head# 2024-04-23T16:25:15.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d40ec75:::benchmark_data_smithi188_89276_object127019:head# 2024-04-23T16:25:15.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d40fdd8:::benchmark_data_smithi188_89276_object45977:head# 2024-04-23T16:25:15.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d41038f:::benchmark_data_smithi188_89276_object120031:head# 2024-04-23T16:25:15.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d411473:::benchmark_data_smithi188_89276_object160392:head# 2024-04-23T16:25:15.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d4367c9:::benchmark_data_smithi188_89276_object101247:head# 2024-04-23T16:25:15.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d43a33b:::benchmark_data_smithi188_89276_object41107:head# 2024-04-23T16:25:15.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d43c63a:::benchmark_data_smithi188_89276_object115918:head# 2024-04-23T16:25:15.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d43e604:::benchmark_data_smithi188_89276_object80738:head# 2024-04-23T16:25:15.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d447874:::benchmark_data_smithi188_89276_object47651:head# 2024-04-23T16:25:15.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d44cb4b:::benchmark_data_smithi188_89276_object101435:head# 2024-04-23T16:25:15.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d44d212:::benchmark_data_smithi188_89276_object108894:head# 2024-04-23T16:25:15.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d4520b1:::benchmark_data_smithi188_89276_object52281:head# 2024-04-23T16:25:15.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d4546b5:::benchmark_data_smithi188_89276_object66137:head# 2024-04-23T16:25:15.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d456c92:::benchmark_data_smithi188_89276_object66788:head# 2024-04-23T16:25:15.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d462964:::benchmark_data_smithi188_89276_object55473:head# 2024-04-23T16:25:15.725 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d4691e8:::benchmark_data_smithi188_89276_object147853:head# 2024-04-23T16:25:15.725 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d46e453:::benchmark_data_smithi188_89276_object132042:head# 2024-04-23T16:25:15.725 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d472c96:::benchmark_data_smithi188_89276_object137552:head# 2024-04-23T16:25:15.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d4772ba:::benchmark_data_smithi188_89276_object80316:head# 2024-04-23T16:25:15.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d4869dd:::benchmark_data_smithi188_89276_object127617:head# 2024-04-23T16:25:15.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d48a73c:::benchmark_data_smithi188_89276_object102049:head# 2024-04-23T16:25:15.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d48b178:::benchmark_data_smithi188_89276_object127070:head# 2024-04-23T16:25:15.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d490f64:::benchmark_data_smithi188_89276_object117597:head# 2024-04-23T16:25:15.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d499c4f:::benchmark_data_smithi188_89276_object154018:head# 2024-04-23T16:25:15.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d49fbb9:::benchmark_data_smithi188_89276_object110759:head# 2024-04-23T16:25:15.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d4a349f:::benchmark_data_smithi188_89276_object166513:head# 2024-04-23T16:25:15.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d4b1866:::benchmark_data_smithi188_89276_object149823:head# 2024-04-23T16:25:15.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d4b407d:::benchmark_data_smithi188_89276_object67831:head# 2024-04-23T16:25:15.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d4bc118:::benchmark_data_smithi188_89276_object61551:head# 2024-04-23T16:25:15.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d4d4a37:::benchmark_data_smithi188_89276_object78710:head# 2024-04-23T16:25:15.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d4d8b64:::benchmark_data_smithi188_89276_object143015:head# 2024-04-23T16:25:15.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d4dcbc7:::benchmark_data_smithi188_89276_object114109:head# 2024-04-23T16:25:15.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d4dcdd8:::benchmark_data_smithi188_89276_object115874:head# 2024-04-23T16:25:15.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d4e2b39:::benchmark_data_smithi188_89276_object88932:head# 2024-04-23T16:25:15.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d4e9a4a:::benchmark_data_smithi188_89276_object82915:head# 2024-04-23T16:25:15.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d4eadc0:::benchmark_data_smithi188_89276_object157441:head# 2024-04-23T16:25:15.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d4ede40:::benchmark_data_smithi188_89276_object72960:head# 2024-04-23T16:25:15.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d4fdb9f:::benchmark_data_smithi188_89276_object58191:head# 2024-04-23T16:25:15.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d5007ac:::benchmark_data_smithi188_89276_object116438:head# 2024-04-23T16:25:15.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d502a49:::benchmark_data_smithi188_89276_object71217:head# 2024-04-23T16:25:15.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d5069c4:::benchmark_data_smithi188_89276_object137946:head# 2024-04-23T16:25:15.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d507916:::benchmark_data_smithi188_89276_object141889:head# 2024-04-23T16:25:15.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d50df12:::benchmark_data_smithi188_89276_object109518:head# 2024-04-23T16:25:15.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d513c35:::benchmark_data_smithi188_89276_object137195:head# 2024-04-23T16:25:15.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d514227:::benchmark_data_smithi188_89276_object50166:head# 2024-04-23T16:25:15.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d517a40:::benchmark_data_smithi188_89276_object132198:head# 2024-04-23T16:25:15.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d51a6e6:::benchmark_data_smithi188_89276_object109019:head# 2024-04-23T16:25:15.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d52781b:::benchmark_data_smithi188_89276_object149540:head# 2024-04-23T16:25:15.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d52c0fc:::benchmark_data_smithi188_89276_object54997:head# 2024-04-23T16:25:15.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d533715:::benchmark_data_smithi188_89276_object165718:head# 2024-04-23T16:25:15.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d53899a:::benchmark_data_smithi188_89276_object100487:head# 2024-04-23T16:25:15.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d53edf0:::benchmark_data_smithi188_89276_object167959:head# 2024-04-23T16:25:15.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d545acd:::benchmark_data_smithi188_89276_object55209:head# 2024-04-23T16:25:15.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d56ca83:::benchmark_data_smithi188_89276_object74742:head# 2024-04-23T16:25:15.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d56fd8f:::benchmark_data_smithi188_89276_object66758:head# 2024-04-23T16:25:15.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d57360f:::benchmark_data_smithi188_89276_object88197:head# 2024-04-23T16:25:15.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d57fa06:::benchmark_data_smithi188_89276_object132038:head# 2024-04-23T16:25:15.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d590a77:::benchmark_data_smithi188_89276_object152794:head# 2024-04-23T16:25:15.744 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d599997:::benchmark_data_smithi188_89276_object114132:head# 2024-04-23T16:25:15.744 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d5b5a13:::benchmark_data_smithi188_89276_object160773:head# 2024-04-23T16:25:15.745 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d5bf5db:::benchmark_data_smithi188_89276_object46927:head# 2024-04-23T16:25:15.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d5c8469:::benchmark_data_smithi188_89276_object145341:head# 2024-04-23T16:25:15.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d5cb1c0:::benchmark_data_smithi188_89276_object79410:head# 2024-04-23T16:25:15.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d5d66b1:::benchmark_data_smithi188_89276_object110035:head# 2024-04-23T16:25:15.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d5d9ae4:::benchmark_data_smithi188_89276_object115843:head# 2024-04-23T16:25:15.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d5f21da:::benchmark_data_smithi188_89276_object51537:head# 2024-04-23T16:25:15.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d60846d:::benchmark_data_smithi188_89276_object110875:head# 2024-04-23T16:25:15.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d60a336:::benchmark_data_smithi188_89276_object57964:head# 2024-04-23T16:25:15.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d61b659:::benchmark_data_smithi188_89276_object63202:head# 2024-04-23T16:25:15.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d61e7fe:::benchmark_data_smithi188_89276_object162686:head# 2024-04-23T16:25:15.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d622d6e:::benchmark_data_smithi188_89276_object111981:head# 2024-04-23T16:25:15.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d631a84:::benchmark_data_smithi188_89276_object128820:head# 2024-04-23T16:25:15.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d642235:::benchmark_data_smithi188_89276_object54735:head# 2024-04-23T16:25:15.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d65c0ea:::benchmark_data_smithi188_89276_object67355:head# 2024-04-23T16:25:15.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d663619:::benchmark_data_smithi188_89276_object158354:head# 2024-04-23T16:25:15.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d668ddb:::benchmark_data_smithi188_89276_object87244:head# 2024-04-23T16:25:15.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d66efab:::benchmark_data_smithi188_89276_object44616:head# 2024-04-23T16:25:15.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d66fb4a:::benchmark_data_smithi188_89276_object131182:head# 2024-04-23T16:25:15.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d671a66:::benchmark_data_smithi188_89276_object79764:head# 2024-04-23T16:25:15.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d673670:::benchmark_data_smithi188_89276_object60125:head# 2024-04-23T16:25:15.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d682bc0:::benchmark_data_smithi188_89276_object161532:head# 2024-04-23T16:25:15.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d68abd5:::benchmark_data_smithi188_89276_object91806:head# 2024-04-23T16:25:15.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d68c6a8:::benchmark_data_smithi188_89276_object65347:head# 2024-04-23T16:25:15.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d68c8ca:::benchmark_data_smithi188_89276_object98825:head# 2024-04-23T16:25:15.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d68d795:::benchmark_data_smithi188_89276_object129196:head# 2024-04-23T16:25:15.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d69de52:::benchmark_data_smithi188_89276_object153793:head# 2024-04-23T16:25:15.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d6a718e:::benchmark_data_smithi188_89276_object51043:head# 2024-04-23T16:25:15.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d6ae519:::benchmark_data_smithi188_89276_object107464:head# 2024-04-23T16:25:15.764 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d6afbbe:::benchmark_data_smithi188_89276_object64789:head# 2024-04-23T16:25:15.764 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d6b2c8d:::benchmark_data_smithi188_89276_object51950:head# 2024-04-23T16:25:15.765 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d6d0d3b:::benchmark_data_smithi188_89276_object70386:head# 2024-04-23T16:25:15.765 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d6d23f2:::benchmark_data_smithi188_89276_object102580:head# 2024-04-23T16:25:15.765 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d6d608b:::benchmark_data_smithi188_89276_object122875:head# 2024-04-23T16:25:15.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d6e0803:::benchmark_data_smithi188_89276_object62209:head# 2024-04-23T16:25:15.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d6e51a2:::benchmark_data_smithi188_89276_object128019:head# 2024-04-23T16:25:15.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d6e9025:::benchmark_data_smithi188_89276_object121895:head# 2024-04-23T16:25:15.767 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d6fae03:::benchmark_data_smithi188_89276_object129955:head# 2024-04-23T16:25:15.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d704f7e:::benchmark_data_smithi188_89276_object130864:head# 2024-04-23T16:25:15.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d70ab72:::benchmark_data_smithi188_89276_object108833:head# 2024-04-23T16:25:15.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d70c6c7:::benchmark_data_smithi188_89276_object138846:head# 2024-04-23T16:25:15.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d710665:::benchmark_data_smithi188_89276_object123934:head# 2024-04-23T16:25:15.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7118cc:::benchmark_data_smithi188_89276_object111573:head# 2024-04-23T16:25:15.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d712e1e:::benchmark_data_smithi188_89276_object145126:head# 2024-04-23T16:25:15.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d722a51:::benchmark_data_smithi188_89276_object87645:head# 2024-04-23T16:25:15.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d730a56:::benchmark_data_smithi188_89276_object72920:head# 2024-04-23T16:25:15.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7323ec:::benchmark_data_smithi188_89276_object113376:head# 2024-04-23T16:25:15.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d74c4cb:::benchmark_data_smithi188_89276_object47607:head# 2024-04-23T16:25:15.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d74cc40:::benchmark_data_smithi188_89276_object95121:head# 2024-04-23T16:25:15.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d750f7f:::benchmark_data_smithi188_89276_object116367:head# 2024-04-23T16:25:15.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d75c4f3:::benchmark_data_smithi188_89276_object48690:head# 2024-04-23T16:25:15.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7641ab:::benchmark_data_smithi188_89276_object138737:head# 2024-04-23T16:25:15.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d769abe:::benchmark_data_smithi188_89276_object56904:head# 2024-04-23T16:25:15.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d76d921:::benchmark_data_smithi188_89276_object76431:head# 2024-04-23T16:25:15.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d76fd78:::benchmark_data_smithi188_89276_object88321:head# 2024-04-23T16:25:15.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d77f2ae:::benchmark_data_smithi188_89276_object104870:head# 2024-04-23T16:25:15.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d789810:::benchmark_data_smithi188_89276_object81617:head# 2024-04-23T16:25:15.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d78ad41:::benchmark_data_smithi188_89276_object78256:head# 2024-04-23T16:25:15.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7a1ff9:::benchmark_data_smithi188_89276_object160864:head# 2024-04-23T16:25:15.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7a8c40:::benchmark_data_smithi188_89276_object81755:head# 2024-04-23T16:25:15.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7aa2fa:::benchmark_data_smithi188_89276_object147424:head# 2024-04-23T16:25:15.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7aae54:::benchmark_data_smithi188_89276_object124674:head# 2024-04-23T16:25:15.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7b0b33:::benchmark_data_smithi188_89276_object100423:head# 2024-04-23T16:25:15.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7b6fa9:::benchmark_data_smithi188_89276_object105394:head# 2024-04-23T16:25:15.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7be5ea:::benchmark_data_smithi188_89276_object105705:head# 2024-04-23T16:25:15.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7bf74a:::benchmark_data_smithi188_89276_object88666:head# 2024-04-23T16:25:15.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7c3dd1:::benchmark_data_smithi188_89276_object68960:head# 2024-04-23T16:25:15.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7c726c:::benchmark_data_smithi188_89276_object74059:head# 2024-04-23T16:25:15.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7cc83c:::benchmark_data_smithi188_89276_object60239:head# 2024-04-23T16:25:15.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7d2a13:::benchmark_data_smithi188_89276_object114933:head# 2024-04-23T16:25:15.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7d819a:::benchmark_data_smithi188_89276_object129897:head# 2024-04-23T16:25:15.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7dd9d1:::benchmark_data_smithi188_89276_object44252:head# 2024-04-23T16:25:15.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7e0380:::benchmark_data_smithi188_89276_object146802:head# 2024-04-23T16:25:15.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7e2be6:::benchmark_data_smithi188_89276_object45626:head# 2024-04-23T16:25:15.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7e41f7:::benchmark_data_smithi188_89276_object103673:head# 2024-04-23T16:25:15.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7e773c:::benchmark_data_smithi188_89276_object127214:head# 2024-04-23T16:25:15.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7ea3ec:::benchmark_data_smithi188_89276_object104850:head# 2024-04-23T16:25:15.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7ee347:::benchmark_data_smithi188_89276_object72190:head# 2024-04-23T16:25:15.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7efadd:::benchmark_data_smithi188_89276_object49703:head# 2024-04-23T16:25:15.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d7f7081:::benchmark_data_smithi188_89276_object128111:head# 2024-04-23T16:25:15.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d803584:::benchmark_data_smithi188_89276_object79768:head# 2024-04-23T16:25:15.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d803da0:::benchmark_data_smithi188_89276_object81122:head# 2024-04-23T16:25:15.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d80a128:::benchmark_data_smithi188_89276_object141301:head# 2024-04-23T16:25:15.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d81d02b:::benchmark_data_smithi188_89276_object90706:head# 2024-04-23T16:25:15.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d81f280:::benchmark_data_smithi188_89276_object138315:head# 2024-04-23T16:25:15.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d824cdd:::benchmark_data_smithi188_89276_object118612:head# 2024-04-23T16:25:15.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d82c66d:::benchmark_data_smithi188_89276_object163305:head# 2024-04-23T16:25:15.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d82f001:::benchmark_data_smithi188_89276_object142003:head# 2024-04-23T16:25:15.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d8351e4:::benchmark_data_smithi188_89276_object119420:head# 2024-04-23T16:25:15.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d837711:::benchmark_data_smithi188_89276_object81331:head# 2024-04-23T16:25:15.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d838046:::benchmark_data_smithi188_89276_object159829:head# 2024-04-23T16:25:15.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d8381da:::benchmark_data_smithi188_89276_object162457:head# 2024-04-23T16:25:15.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d83b51a:::benchmark_data_smithi188_89276_object114731:head# 2024-04-23T16:25:15.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d83e8e4:::benchmark_data_smithi188_89276_object150915:head# 2024-04-23T16:25:15.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d840ad8:::benchmark_data_smithi188_89276_object102584:head# 2024-04-23T16:25:15.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d844568:::benchmark_data_smithi188_89276_object43276:head# 2024-04-23T16:25:15.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d844f74:::benchmark_data_smithi188_89276_object146848:head# 2024-04-23T16:25:15.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d85053f:::benchmark_data_smithi188_89276_object83970:head# 2024-04-23T16:25:15.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d850799:::benchmark_data_smithi188_89276_object86282:head# 2024-04-23T16:25:15.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d852621:::benchmark_data_smithi188_89276_object71821:head# 2024-04-23T16:25:15.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d85fb73:::benchmark_data_smithi188_89276_object139153:head# 2024-04-23T16:25:15.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d86691d:::benchmark_data_smithi188_89276_object71969:head# 2024-04-23T16:25:15.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d86b5d6:::benchmark_data_smithi188_89276_object61779:head# 2024-04-23T16:25:15.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d888b4c:::benchmark_data_smithi188_89276_object48956:head# 2024-04-23T16:25:15.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d88e866:::benchmark_data_smithi188_89276_object61355:head# 2024-04-23T16:25:15.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d89d93f:::benchmark_data_smithi188_89276_object80337:head# 2024-04-23T16:25:15.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d8a3783:::benchmark_data_smithi188_89276_object60877:head# 2024-04-23T16:25:15.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d8a6bc7:::benchmark_data_smithi188_89276_object158479:head# 2024-04-23T16:25:15.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d8bbc6e:::benchmark_data_smithi188_89276_object50187:head# 2024-04-23T16:25:15.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d8bc849:::benchmark_data_smithi188_89276_object64211:head# 2024-04-23T16:25:15.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d8c8fb5:::benchmark_data_smithi188_89276_object124552:head# 2024-04-23T16:25:15.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d8d6004:::benchmark_data_smithi188_89276_object65605:head# 2024-04-23T16:25:15.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d8daa6a:::benchmark_data_smithi188_89276_object113258:head# 2024-04-23T16:25:15.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d8dbbcf:::benchmark_data_smithi188_89276_object84072:head# 2024-04-23T16:25:15.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d8def7d:::benchmark_data_smithi188_89276_object70208:head# 2024-04-23T16:25:15.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d9055dc:::benchmark_data_smithi188_89276_object47506:head# 2024-04-23T16:25:15.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d90eca2:::benchmark_data_smithi188_89276_object65842:head# 2024-04-23T16:25:15.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d90fff8:::benchmark_data_smithi188_89276_object102543:head# 2024-04-23T16:25:15.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d924d85:::benchmark_data_smithi188_89276_object45983:head# 2024-04-23T16:25:15.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d92ed54:::benchmark_data_smithi188_89276_object134748:head# 2024-04-23T16:25:15.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d93b5f5:::benchmark_data_smithi188_89276_object56196:head# 2024-04-23T16:25:15.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d93f884:::benchmark_data_smithi188_89276_object142486:head# 2024-04-23T16:25:15.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d961c1d:::benchmark_data_smithi188_89276_object43638:head# 2024-04-23T16:25:15.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d966910:::benchmark_data_smithi188_89276_object152080:head# 2024-04-23T16:25:15.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d970fbd:::benchmark_data_smithi188_89276_object114204:head# 2024-04-23T16:25:15.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d97a17e:::benchmark_data_smithi188_89276_object99805:head# 2024-04-23T16:25:15.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d97abbe:::benchmark_data_smithi188_89276_object74636:head# 2024-04-23T16:25:15.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d9853ba:::benchmark_data_smithi188_89276_object137122:head# 2024-04-23T16:25:15.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d9880b8:::benchmark_data_smithi188_89276_object130504:head# 2024-04-23T16:25:15.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d9a4385:::benchmark_data_smithi188_89276_object100096:head# 2024-04-23T16:25:15.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d9ac497:::benchmark_data_smithi188_89276_object63610:head# 2024-04-23T16:25:15.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d9ae761:::benchmark_data_smithi188_89276_object74689:head# 2024-04-23T16:25:15.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d9b0d98:::benchmark_data_smithi188_89276_object77279:head# 2024-04-23T16:25:15.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d9b28af:::benchmark_data_smithi188_89276_object163780:head# 2024-04-23T16:25:15.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d9b2976:::benchmark_data_smithi188_89276_object153799:head# 2024-04-23T16:25:15.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d9dda70:::benchmark_data_smithi188_89276_object62920:head# 2024-04-23T16:25:15.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d9ddff6:::benchmark_data_smithi188_89276_object165043:head# 2024-04-23T16:25:15.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d9e4363:::benchmark_data_smithi188_89276_object65776:head# 2024-04-23T16:25:15.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d9e53de:::benchmark_data_smithi188_89276_object80556:head# 2024-04-23T16:25:15.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d9eb4a3:::benchmark_data_smithi188_89276_object112163:head# 2024-04-23T16:25:15.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d9f1e70:::benchmark_data_smithi188_89276_object86146:head# 2024-04-23T16:25:15.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d9f29b5:::benchmark_data_smithi188_89276_object100678:head# 2024-04-23T16:25:15.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2d9fdd12:::benchmark_data_smithi188_89276_object98623:head# 2024-04-23T16:25:15.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2da01c31:::benchmark_data_smithi188_89276_object160087:head# 2024-04-23T16:25:15.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2da0844a:::benchmark_data_smithi188_89276_object148358:head# 2024-04-23T16:25:15.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2da0f6e2:::benchmark_data_smithi188_89276_object55181:head# 2024-04-23T16:25:15.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2da22b0c:::benchmark_data_smithi188_89276_object134831:head# 2024-04-23T16:25:15.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2da3743c:::benchmark_data_smithi188_89276_object158218:head# 2024-04-23T16:25:15.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2da37df9:::benchmark_data_smithi188_89276_object109624:head# 2024-04-23T16:25:15.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2da39394:::benchmark_data_smithi188_89276_object131916:head# 2024-04-23T16:25:15.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2da3e179:::benchmark_data_smithi188_89276_object72052:head# 2024-04-23T16:25:15.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2da3fa71:::benchmark_data_smithi188_89276_object123823:head# 2024-04-23T16:25:15.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2da4044b:::benchmark_data_smithi188_89276_object62952:head# 2024-04-23T16:25:15.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2da4587a:::benchmark_data_smithi188_89276_object104200:head# 2024-04-23T16:25:15.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2da5151a:::benchmark_data_smithi188_89276_object105041:head# 2024-04-23T16:25:15.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2da67d5c:::benchmark_data_smithi188_89276_object133680:head# 2024-04-23T16:25:15.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2da6bda7:::benchmark_data_smithi188_89276_object50843:head# 2024-04-23T16:25:15.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2da6c148:::benchmark_data_smithi188_89276_object129289:head# 2024-04-23T16:25:15.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2da6db6e:::benchmark_data_smithi188_89276_object109148:head# 2024-04-23T16:25:15.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2da73b4f:::benchmark_data_smithi188_89276_object44131:head# 2024-04-23T16:25:15.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2da9197c:::benchmark_data_smithi188_89276_object47440:head# 2024-04-23T16:25:15.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2da9a7a6:::benchmark_data_smithi188_89276_object68034:head# 2024-04-23T16:25:15.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2daa27de:::benchmark_data_smithi188_89276_object52692:head# 2024-04-23T16:25:15.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2daa433f:::benchmark_data_smithi188_89276_object93961:head# 2024-04-23T16:25:15.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2daaaa44:::benchmark_data_smithi188_89276_object110184:head# 2024-04-23T16:25:15.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dab17ea:::benchmark_data_smithi188_89276_object84727:head# 2024-04-23T16:25:15.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dab3e22:::benchmark_data_smithi188_89276_object58187:head# 2024-04-23T16:25:15.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dab6036:::benchmark_data_smithi188_89276_object45577:head# 2024-04-23T16:25:15.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dab6213:::benchmark_data_smithi188_89276_object70375:head# 2024-04-23T16:25:15.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dab893f:::benchmark_data_smithi188_89276_object136693:head# 2024-04-23T16:25:15.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dabb898:::benchmark_data_smithi188_89276_object141848:head# 2024-04-23T16:25:15.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dac17de:::benchmark_data_smithi188_89276_object86430:head# 2024-04-23T16:25:15.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dad09e8:::benchmark_data_smithi188_89276_object134528:head# 2024-04-23T16:25:15.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dae95cb:::benchmark_data_smithi188_89276_object94682:head# 2024-04-23T16:25:15.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2daece81:::benchmark_data_smithi188_89276_object87589:head# 2024-04-23T16:25:15.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2daf3b2b:::benchmark_data_smithi188_89276_object94257:head# 2024-04-23T16:25:15.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2daf8c34:::benchmark_data_smithi188_89276_object86575:head# 2024-04-23T16:25:15.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dafb5c4:::benchmark_data_smithi188_89276_object109215:head# 2024-04-23T16:25:15.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db08ed5:::benchmark_data_smithi188_89276_object133479:head# 2024-04-23T16:25:15.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db0b77e:::benchmark_data_smithi188_89276_object82987:head# 2024-04-23T16:25:15.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db0e2f4:::benchmark_data_smithi188_89276_object41084:head# 2024-04-23T16:25:15.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db1c6a1:::benchmark_data_smithi188_89276_object45533:head# 2024-04-23T16:25:15.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db230dc:::benchmark_data_smithi188_89276_object87410:head# 2024-04-23T16:25:15.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db24396:::benchmark_data_smithi188_89276_object151475:head# 2024-04-23T16:25:15.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db27564:::benchmark_data_smithi188_89276_object45226:head# 2024-04-23T16:25:15.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db42a40:::benchmark_data_smithi188_89276_object101493:head# 2024-04-23T16:25:15.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db4bd9b:::benchmark_data_smithi188_89276_object159672:head# 2024-04-23T16:25:15.832 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db4ca2f:::benchmark_data_smithi188_89276_object64993:head# 2024-04-23T16:25:15.832 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db4edd1:::benchmark_data_smithi188_89276_object133318:head# 2024-04-23T16:25:15.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db58a93:::benchmark_data_smithi188_89276_object167927:head# 2024-04-23T16:25:15.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db66049:::benchmark_data_smithi188_89276_object161483:head# 2024-04-23T16:25:15.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db6abcf:::benchmark_data_smithi188_89276_object127127:head# 2024-04-23T16:25:15.834 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db6abf3:::benchmark_data_smithi188_89276_object119565:head# 2024-04-23T16:25:15.834 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db73b41:::benchmark_data_smithi188_89276_object127181:head# 2024-04-23T16:25:15.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db73c01:::benchmark_data_smithi188_89276_object47474:head# 2024-04-23T16:25:15.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db7c9e5:::benchmark_data_smithi188_89276_object57042:head# 2024-04-23T16:25:15.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db7fed4:::benchmark_data_smithi188_89276_object57708:head# 2024-04-23T16:25:15.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db805c7:::benchmark_data_smithi188_89276_object116669:head# 2024-04-23T16:25:15.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2db86704:::benchmark_data_smithi188_89276_object138259:head# 2024-04-23T16:25:15.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dba02b2:::benchmark_data_smithi188_89276_object129453:head# 2024-04-23T16:25:15.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dba49b2:::benchmark_data_smithi188_89276_object86042:head# 2024-04-23T16:25:15.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dbd0aa0:::benchmark_data_smithi188_89276_object138363:head# 2024-04-23T16:25:15.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dbd9713:::benchmark_data_smithi188_89276_object144814:head# 2024-04-23T16:25:15.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dbe4caf:::benchmark_data_smithi188_89276_object139181:head# 2024-04-23T16:25:15.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dbede28:::benchmark_data_smithi188_89276_object158046:head# 2024-04-23T16:25:15.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dbee0de:::benchmark_data_smithi188_89276_object59317:head# 2024-04-23T16:25:15.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc036b7:::benchmark_data_smithi188_89276_object77082:head# 2024-04-23T16:25:15.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc061ca:::benchmark_data_smithi188_89276_object89275:head# 2024-04-23T16:25:15.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc0a37b:::benchmark_data_smithi188_89276_object132743:head# 2024-04-23T16:25:15.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc0c097:::benchmark_data_smithi188_89276_object118763:head# 2024-04-23T16:25:15.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc0d63b:::benchmark_data_smithi188_89276_object101085:head# 2024-04-23T16:25:15.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc0e4b3:::benchmark_data_smithi188_89276_object60063:head# 2024-04-23T16:25:15.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc15a11:::benchmark_data_smithi188_89276_object168281:head# 2024-04-23T16:25:15.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc19f4d:::benchmark_data_smithi188_89276_object40777:head# 2024-04-23T16:25:15.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc1a14f:::benchmark_data_smithi188_89276_object102662:head# 2024-04-23T16:25:15.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc1f60c:::benchmark_data_smithi188_89276_object154075:head# 2024-04-23T16:25:15.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc28991:::benchmark_data_smithi188_89276_object163409:head# 2024-04-23T16:25:15.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc2e092:::benchmark_data_smithi188_89276_object93770:head# 2024-04-23T16:25:15.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc2fe85:::benchmark_data_smithi188_89276_object153414:head# 2024-04-23T16:25:15.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc3323a:::benchmark_data_smithi188_89276_object66284:head# 2024-04-23T16:25:15.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc45694:::benchmark_data_smithi188_89276_object50946:head# 2024-04-23T16:25:15.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc497fc:::benchmark_data_smithi188_89276_object129130:head# 2024-04-23T16:25:15.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc4b159:::benchmark_data_smithi188_89276_object120676:head# 2024-04-23T16:25:15.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc56052:::benchmark_data_smithi188_89276_object121514:head# 2024-04-23T16:25:15.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc61562:::benchmark_data_smithi188_89276_object76213:head# 2024-04-23T16:25:15.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc615ef:::benchmark_data_smithi188_89276_object159024:head# 2024-04-23T16:25:15.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc647a0:::benchmark_data_smithi188_89276_object159239:head# 2024-04-23T16:25:15.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc64c5e:::benchmark_data_smithi188_89276_object42041:head# 2024-04-23T16:25:15.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc663ab:::benchmark_data_smithi188_89276_object90642:head# 2024-04-23T16:25:15.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc6ffd4:::benchmark_data_smithi188_89276_object69849:head# 2024-04-23T16:25:15.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc71632:::benchmark_data_smithi188_89276_object74103:head# 2024-04-23T16:25:15.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc73daa:::benchmark_data_smithi188_89276_object155357:head# 2024-04-23T16:25:15.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc7709d:::benchmark_data_smithi188_89276_object137466:head# 2024-04-23T16:25:15.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc8284d:::benchmark_data_smithi188_89276_object145548:head# 2024-04-23T16:25:15.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc83b01:::benchmark_data_smithi188_89276_object74596:head# 2024-04-23T16:25:15.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc8f5d7:::benchmark_data_smithi188_89276_object72354:head# 2024-04-23T16:25:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dc990b6:::benchmark_data_smithi188_89276_object105114:head# 2024-04-23T16:25:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dca0335:::benchmark_data_smithi188_89276_object140757:head# 2024-04-23T16:25:15.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dca13af:::benchmark_data_smithi188_89276_object74076:head# 2024-04-23T16:25:15.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dcaee11:::benchmark_data_smithi188_89276_object154161:head# 2024-04-23T16:25:15.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dcba58c:::benchmark_data_smithi188_89276_object102787:head# 2024-04-23T16:25:15.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dcc24a8:::benchmark_data_smithi188_89276_object92379:head# 2024-04-23T16:25:15.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dcc473a:::benchmark_data_smithi188_89276_object150735:head# 2024-04-23T16:25:15.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dcd2361:::benchmark_data_smithi188_89276_object124067:head# 2024-04-23T16:25:15.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dcd56c6:::benchmark_data_smithi188_89276_object49835:head# 2024-04-23T16:25:15.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dcd9c18:::benchmark_data_smithi188_89276_object58571:head# 2024-04-23T16:25:15.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dcdbe4d:::benchmark_data_smithi188_89276_object119086:head# 2024-04-23T16:25:15.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dce39bd:::benchmark_data_smithi188_89276_object156837:head# 2024-04-23T16:25:15.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dcf717b:::benchmark_data_smithi188_89276_object122957:head# 2024-04-23T16:25:15.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dcfbeb6:::benchmark_data_smithi188_89276_object141322:head# 2024-04-23T16:25:15.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dd05082:::benchmark_data_smithi188_89276_object113169:head# 2024-04-23T16:25:15.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dd0cd5d:::benchmark_data_smithi188_89276_object50146:head# 2024-04-23T16:25:15.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dd20114:::benchmark_data_smithi188_89276_object102251:head# 2024-04-23T16:25:15.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dd23d1e:::benchmark_data_smithi188_89276_object163448:head# 2024-04-23T16:25:15.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dd456cf:::benchmark_data_smithi188_89276_object62444:head# 2024-04-23T16:25:15.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dd46368:::benchmark_data_smithi188_89276_object73557:head# 2024-04-23T16:25:15.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dd46412:::benchmark_data_smithi188_89276_object99864:head# 2024-04-23T16:25:15.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dd4e4d7:::benchmark_data_smithi188_89276_object107779:head# 2024-04-23T16:25:15.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dd55bf4:::benchmark_data_smithi188_89276_object117897:head# 2024-04-23T16:25:15.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dd66bc1:::benchmark_data_smithi188_89276_object72741:head# 2024-04-23T16:25:15.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dd6e15b:::benchmark_data_smithi188_89276_object70191:head# 2024-04-23T16:25:15.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dd86fe2:::benchmark_data_smithi188_89276_object63386:head# 2024-04-23T16:25:15.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dd8b023:::benchmark_data_smithi188_89276_object97857:head# 2024-04-23T16:25:15.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dd949de:::benchmark_data_smithi188_89276_object64109:head# 2024-04-23T16:25:15.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dd9ebc9:::benchmark_data_smithi188_89276_object90024:head# 2024-04-23T16:25:15.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dda1d8f:::benchmark_data_smithi188_89276_object70616:head# 2024-04-23T16:25:15.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ddad46c:::benchmark_data_smithi188_89276_object112507:head# 2024-04-23T16:25:15.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ddb1adf:::benchmark_data_smithi188_89276_object68114:head# 2024-04-23T16:25:15.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ddb739a:::benchmark_data_smithi188_89276_object148866:head# 2024-04-23T16:25:15.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ddba1f9:::benchmark_data_smithi188_89276_object79714:head# 2024-04-23T16:25:15.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ddbf465:::benchmark_data_smithi188_89276_object162452:head# 2024-04-23T16:25:15.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dde2b14:::benchmark_data_smithi188_89276_object89634:head# 2024-04-23T16:25:15.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ddeacf3:::benchmark_data_smithi188_89276_object74797:head# 2024-04-23T16:25:15.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ddfd9fd:::benchmark_data_smithi188_89276_object158729:head# 2024-04-23T16:25:15.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de0518d:::benchmark_data_smithi188_89276_object117559:head# 2024-04-23T16:25:15.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de05df9:::benchmark_data_smithi188_89276_object55060:head# 2024-04-23T16:25:15.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de0c75e:::benchmark_data_smithi188_89276_object41050:head# 2024-04-23T16:25:15.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de16be7:::benchmark_data_smithi188_89276_object166673:head# 2024-04-23T16:25:15.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de21175:::benchmark_data_smithi188_89276_object102249:head# 2024-04-23T16:25:15.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de298c2:::benchmark_data_smithi188_89276_object161668:head# 2024-04-23T16:25:15.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de2de09:::benchmark_data_smithi188_89276_object93152:head# 2024-04-23T16:25:15.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de32e5e:::benchmark_data_smithi188_89276_object87762:head# 2024-04-23T16:25:15.875 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de3a55c:::benchmark_data_smithi188_89276_object74102:head# 2024-04-23T16:25:15.875 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de4273c:::benchmark_data_smithi188_89276_object134988:head# 2024-04-23T16:25:15.876 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de4b128:::benchmark_data_smithi188_89276_object63879:head# 2024-04-23T16:25:15.876 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de552b6:::benchmark_data_smithi188_89276_object76466:head# 2024-04-23T16:25:15.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de58bdc:::benchmark_data_smithi188_89276_object107685:head# 2024-04-23T16:25:15.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de683f1:::benchmark_data_smithi188_89276_object149100:head# 2024-04-23T16:25:15.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de69acd:::benchmark_data_smithi188_89276_object51137:head# 2024-04-23T16:25:15.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de69afb:::benchmark_data_smithi188_89276_object167109:head# 2024-04-23T16:25:15.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de6b4e9:::benchmark_data_smithi188_89276_object135014:head# 2024-04-23T16:25:15.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de6f6ac:::benchmark_data_smithi188_89276_object81001:head# 2024-04-23T16:25:15.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de7978d:::benchmark_data_smithi188_89276_object119392:head# 2024-04-23T16:25:15.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de84775:::benchmark_data_smithi188_89276_object99171:head# 2024-04-23T16:25:15.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2de88e92:::benchmark_data_smithi188_89276_object77964:head# 2024-04-23T16:25:15.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dea0290:::benchmark_data_smithi188_89276_object166661:head# 2024-04-23T16:25:15.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dea1f75:::benchmark_data_smithi188_89276_object102054:head# 2024-04-23T16:25:15.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dea7d43:::benchmark_data_smithi188_89276_object99782:head# 2024-04-23T16:25:15.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dea9b32:::benchmark_data_smithi188_89276_object116096:head# 2024-04-23T16:25:15.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2deaa545:::benchmark_data_smithi188_89276_object164822:head# 2024-04-23T16:25:15.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2deb0a8d:::benchmark_data_smithi188_89276_object122684:head# 2024-04-23T16:25:15.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2deb0f5f:::benchmark_data_smithi188_89276_object78022:head# 2024-04-23T16:25:15.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2deb40c3:::benchmark_data_smithi188_89276_object107466:head# 2024-04-23T16:25:15.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2deb82b9:::benchmark_data_smithi188_89276_object121780:head# 2024-04-23T16:25:15.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2debbb27:::benchmark_data_smithi188_89276_object45444:head# 2024-04-23T16:25:15.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2debd707:::benchmark_data_smithi188_89276_object77271:head# 2024-04-23T16:25:15.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dec0804:::benchmark_data_smithi188_89276_object42804:head# 2024-04-23T16:25:15.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dec1c47:::benchmark_data_smithi188_89276_object122948:head# 2024-04-23T16:25:15.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ded1882:::benchmark_data_smithi188_89276_object161292:head# 2024-04-23T16:25:15.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dedd326:::benchmark_data_smithi188_89276_object102326:head# 2024-04-23T16:25:15.891 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dedda1e:::benchmark_data_smithi188_89276_object119284:head# 2024-04-23T16:25:15.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dede815:::benchmark_data_smithi188_89276_object80318:head# 2024-04-23T16:25:15.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dee1311:::benchmark_data_smithi188_89276_object155547:head# 2024-04-23T16:25:15.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dee291d:::benchmark_data_smithi188_89276_object155795:head# 2024-04-23T16:25:15.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2deeea6f:::benchmark_data_smithi188_89276_object129136:head# 2024-04-23T16:25:15.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2def1156:::benchmark_data_smithi188_89276_object40632:head# 2024-04-23T16:25:15.896 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2defd3fd:::benchmark_data_smithi188_89276_object42663:head# 2024-04-23T16:25:15.896 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df0887b:::benchmark_data_smithi188_89276_object150491:head# 2024-04-23T16:25:15.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df0aa3a:::benchmark_data_smithi188_89276_object135550:head# 2024-04-23T16:25:15.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df0c506:::benchmark_data_smithi188_89276_object131399:head# 2024-04-23T16:25:15.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df0fa28:::benchmark_data_smithi188_89276_object114571:head# 2024-04-23T16:25:15.899 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df10d36:::benchmark_data_smithi188_89276_object60056:head# 2024-04-23T16:25:15.900 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df16183:::benchmark_data_smithi188_89276_object99355:head# 2024-04-23T16:25:15.900 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df16c4d:::benchmark_data_smithi188_89276_object51149:head# 2024-04-23T16:25:15.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df19afe:::benchmark_data_smithi188_89276_object99179:head# 2024-04-23T16:25:15.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df21c89:::benchmark_data_smithi188_89276_object127575:head# 2024-04-23T16:25:15.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df29f62:::benchmark_data_smithi188_89276_object109981:head# 2024-04-23T16:25:15.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df35076:::benchmark_data_smithi188_89276_object48805:head# 2024-04-23T16:25:15.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df374fd:::benchmark_data_smithi188_89276_object131873:head# 2024-04-23T16:25:15.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df3e760:::benchmark_data_smithi188_89276_object166945:head# 2024-04-23T16:25:15.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df4f6f3:::benchmark_data_smithi188_89276_object96972:head# 2024-04-23T16:25:15.904 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df6062a:::benchmark_data_smithi188_89276_object138475:head# 2024-04-23T16:25:15.904 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df65705:::benchmark_data_smithi188_89276_object90423:head# 2024-04-23T16:25:15.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df679f2:::benchmark_data_smithi188_89276_object67116:head# 2024-04-23T16:25:15.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df689a8:::benchmark_data_smithi188_89276_object71887:head# 2024-04-23T16:25:15.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df69748:::benchmark_data_smithi188_89276_object82583:head# 2024-04-23T16:25:15.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df71426:::benchmark_data_smithi188_89276_object89107:head# 2024-04-23T16:25:15.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df79ed8:::benchmark_data_smithi188_89276_object166898:head# 2024-04-23T16:25:15.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df85839:::benchmark_data_smithi188_89276_object70099:head# 2024-04-23T16:25:15.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df86138:::benchmark_data_smithi188_89276_object138511:head# 2024-04-23T16:25:15.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df945c8:::benchmark_data_smithi188_89276_object114324:head# 2024-04-23T16:25:15.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df954a1:::benchmark_data_smithi188_89276_object136644:head# 2024-04-23T16:25:15.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2df97e62:::benchmark_data_smithi188_89276_object126496:head# 2024-04-23T16:25:15.909 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dfa104d:::benchmark_data_smithi188_89276_object72521:head# 2024-04-23T16:25:15.909 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dfa93c2:::benchmark_data_smithi188_89276_object82402:head# 2024-04-23T16:25:15.909 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dfa9429:::benchmark_data_smithi188_89276_object104062:head# 2024-04-23T16:25:15.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dfc49b1:::benchmark_data_smithi188_89276_object99907:head# 2024-04-23T16:25:15.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dfc9528:::benchmark_data_smithi188_89276_object117974:head# 2024-04-23T16:25:15.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dfd6ddd:::benchmark_data_smithi188_89276_object48465:head# 2024-04-23T16:25:15.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dfd8c24:::benchmark_data_smithi188_89276_object56420:head# 2024-04-23T16:25:15.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dfdaaab:::benchmark_data_smithi188_89276_object100964:head# 2024-04-23T16:25:15.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dfec6a6:::benchmark_data_smithi188_89276_object63626:head# 2024-04-23T16:25:15.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dff0993:::benchmark_data_smithi188_89276_object51446:head# 2024-04-23T16:25:15.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2dff4585:::benchmark_data_smithi188_89276_object83163:head# 2024-04-23T16:25:15.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e0008ff:::benchmark_data_smithi188_89276_object131605:head# 2024-04-23T16:25:15.914 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e00d9a3:::benchmark_data_smithi188_89276_object89452:head# 2024-04-23T16:25:15.914 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e012af3:::benchmark_data_smithi188_89276_object79898:head# 2024-04-23T16:25:15.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e0245fc:::benchmark_data_smithi188_89276_object141427:head# 2024-04-23T16:25:15.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e02809d:::benchmark_data_smithi188_89276_object151894:head# 2024-04-23T16:25:15.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e02d2f3:::benchmark_data_smithi188_89276_object100446:head# 2024-04-23T16:25:15.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e04b641:::benchmark_data_smithi188_89276_object144044:head# 2024-04-23T16:25:15.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e057cd3:::benchmark_data_smithi188_89276_object163119:head# 2024-04-23T16:25:15.917 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e058098:::benchmark_data_smithi188_89276_object109284:head# 2024-04-23T16:25:15.917 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e059fad:::benchmark_data_smithi188_89276_object165437:head# 2024-04-23T16:25:15.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e05ea19:::benchmark_data_smithi188_89276_object141129:head# 2024-04-23T16:25:15.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e062a8c:::benchmark_data_smithi188_89276_object154401:head# 2024-04-23T16:25:15.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e0724b8:::benchmark_data_smithi188_89276_object120138:head# 2024-04-23T16:25:15.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e0781a2:::benchmark_data_smithi188_89276_object48604:head# 2024-04-23T16:25:15.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e07858b:::benchmark_data_smithi188_89276_object156846:head# 2024-04-23T16:25:15.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e07a260:::benchmark_data_smithi188_89276_object124816:head# 2024-04-23T16:25:15.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e07bd81:::benchmark_data_smithi188_89276_object73642:head# 2024-04-23T16:25:15.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e092906:::benchmark_data_smithi188_89276_object90741:head# 2024-04-23T16:25:15.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e09374c:::benchmark_data_smithi188_89276_object165062:head# 2024-04-23T16:25:15.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e09e053:::benchmark_data_smithi188_89276_object48070:head# 2024-04-23T16:25:15.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e0a0b86:::benchmark_data_smithi188_89276_object142872:head# 2024-04-23T16:25:15.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e0a1d40:::benchmark_data_smithi188_89276_object93825:head# 2024-04-23T16:25:15.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e0a1f42:::benchmark_data_smithi188_89276_object96162:head# 2024-04-23T16:25:15.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e0aa734:::benchmark_data_smithi188_89276_object53100:head# 2024-04-23T16:25:15.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e0b654c:::benchmark_data_smithi188_89276_object137056:head# 2024-04-23T16:25:15.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e0bc961:::benchmark_data_smithi188_89276_object69061:head# 2024-04-23T16:25:15.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e0eae2f:::benchmark_data_smithi188_89276_object47542:head# 2024-04-23T16:25:15.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e0f667a:::benchmark_data_smithi188_89276_object136144:head# 2024-04-23T16:25:15.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e0fcfa1:::benchmark_data_smithi188_89276_object146290:head# 2024-04-23T16:25:15.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e0fdf93:::benchmark_data_smithi188_89276_object152276:head# 2024-04-23T16:25:15.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e101d31:::benchmark_data_smithi188_89276_object162483:head# 2024-04-23T16:25:15.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e108ca6:::benchmark_data_smithi188_89276_object42781:head# 2024-04-23T16:25:15.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e116339:::benchmark_data_smithi188_89276_object123496:head# 2024-04-23T16:25:15.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e127adc:::benchmark_data_smithi188_89276_object121051:head# 2024-04-23T16:25:15.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e12a39d:::benchmark_data_smithi188_89276_object116017:head# 2024-04-23T16:25:15.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e12ee91:::benchmark_data_smithi188_89276_object46670:head# 2024-04-23T16:25:15.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e1428df:::benchmark_data_smithi188_89276_object74827:head# 2024-04-23T16:25:15.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e15111f:::benchmark_data_smithi188_89276_object103312:head# 2024-04-23T16:25:15.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e155d3b:::benchmark_data_smithi188_89276_object140771:head# 2024-04-23T16:25:15.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e15c623:::benchmark_data_smithi188_89276_object50482:head# 2024-04-23T16:25:15.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e15f4b0:::benchmark_data_smithi188_89276_object109968:head# 2024-04-23T16:25:15.943 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e160044:::benchmark_data_smithi188_89276_object73805:head# 2024-04-23T16:25:15.945 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e162fe1:::benchmark_data_smithi188_89276_object79777:head# 2024-04-23T16:25:15.945 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e16c954:::benchmark_data_smithi188_89276_object133677:head# 2024-04-23T16:25:15.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e16df18:::benchmark_data_smithi188_89276_object99721:head# 2024-04-23T16:25:15.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e18198c:::benchmark_data_smithi188_89276_object64583:head# 2024-04-23T16:25:15.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e186c39:::benchmark_data_smithi188_89276_object154376:head# 2024-04-23T16:25:15.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e199177:::benchmark_data_smithi188_89276_object130627:head# 2024-04-23T16:25:15.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e1995fa:::benchmark_data_smithi188_89276_object158812:head# 2024-04-23T16:25:15.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e1a8261:::benchmark_data_smithi188_89276_object166709:head# 2024-04-23T16:25:15.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e1abe4c:::benchmark_data_smithi188_89276_object122089:head# 2024-04-23T16:25:15.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e1b02d9:::benchmark_data_smithi188_89276_object69541:head# 2024-04-23T16:25:15.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e1bd22f:::benchmark_data_smithi188_89276_object139894:head# 2024-04-23T16:25:15.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e1bfb32:::benchmark_data_smithi188_89276_object77072:head# 2024-04-23T16:25:15.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e1d1a9e:::benchmark_data_smithi188_89276_object108854:head# 2024-04-23T16:25:15.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e1d313f:::benchmark_data_smithi188_89276_object41044:head# 2024-04-23T16:25:15.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e1de052:::benchmark_data_smithi188_89276_object158297:head# 2024-04-23T16:25:15.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e1ecd4a:::benchmark_data_smithi188_89276_object96492:head# 2024-04-23T16:25:15.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e1ed247:::benchmark_data_smithi188_89276_object128318:head# 2024-04-23T16:25:15.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e1ee7b3:::benchmark_data_smithi188_89276_object68466:head# 2024-04-23T16:25:15.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e1fe866:::benchmark_data_smithi188_89276_object165305:head# 2024-04-23T16:25:15.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e204a83:::benchmark_data_smithi188_89276_object82225:head# 2024-04-23T16:25:15.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2120dc:::benchmark_data_smithi188_89276_object70911:head# 2024-04-23T16:25:15.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e212e00:::benchmark_data_smithi188_89276_object142016:head# 2024-04-23T16:25:15.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e21b828:::benchmark_data_smithi188_89276_object90821:head# 2024-04-23T16:25:15.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e21c962:::benchmark_data_smithi188_89276_object122855:head# 2024-04-23T16:25:15.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e221ab7:::benchmark_data_smithi188_89276_object77607:head# 2024-04-23T16:25:15.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e22ca1c:::benchmark_data_smithi188_89276_object89387:head# 2024-04-23T16:25:15.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e22f142:::benchmark_data_smithi188_89276_object104813:head# 2024-04-23T16:25:15.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e23d1dd:::benchmark_data_smithi188_89276_object149029:head# 2024-04-23T16:25:15.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2408c2:::benchmark_data_smithi188_89276_object118010:head# 2024-04-23T16:25:15.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e24c7bc:::benchmark_data_smithi188_89276_object95597:head# 2024-04-23T16:25:15.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2542e8:::benchmark_data_smithi188_89276_object125271:head# 2024-04-23T16:25:15.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e259f28:::benchmark_data_smithi188_89276_object152645:head# 2024-04-23T16:25:15.969 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e25f3d5:::benchmark_data_smithi188_89276_object82981:head# 2024-04-23T16:25:15.969 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e26177a:::benchmark_data_smithi188_89276_object122233:head# 2024-04-23T16:25:15.969 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2682ee:::benchmark_data_smithi188_89276_object139037:head# 2024-04-23T16:25:15.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e26978c:::benchmark_data_smithi188_89276_object148725:head# 2024-04-23T16:25:15.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e26a9f2:::benchmark_data_smithi188_89276_object87593:head# 2024-04-23T16:25:15.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e293c62:::benchmark_data_smithi188_89276_object111433:head# 2024-04-23T16:25:15.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e29d936:::benchmark_data_smithi188_89276_object73417:head# 2024-04-23T16:25:15.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2a0d5f:::benchmark_data_smithi188_89276_object94565:head# 2024-04-23T16:25:15.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2a8f71:::benchmark_data_smithi188_89276_object136862:head# 2024-04-23T16:25:15.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2ab942:::benchmark_data_smithi188_89276_object44247:head# 2024-04-23T16:25:15.973 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2addf2:::benchmark_data_smithi188_89276_object139214:head# 2024-04-23T16:25:15.973 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2b27ec:::benchmark_data_smithi188_89276_object162339:head# 2024-04-23T16:25:15.973 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2b41ef:::benchmark_data_smithi188_89276_object89980:head# 2024-04-23T16:25:15.974 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2ba9a1:::benchmark_data_smithi188_89276_object151086:head# 2024-04-23T16:25:15.974 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2bab2a:::benchmark_data_smithi188_89276_object65851:head# 2024-04-23T16:25:15.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2bcd21:::benchmark_data_smithi188_89276_object61182:head# 2024-04-23T16:25:15.977 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2bed59:::benchmark_data_smithi188_89276_object126933:head# 2024-04-23T16:25:15.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2c970a:::benchmark_data_smithi188_89276_object77521:head# 2024-04-23T16:25:15.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2d1a6f:::benchmark_data_smithi188_89276_object50418:head# 2024-04-23T16:25:15.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2e5dd4:::benchmark_data_smithi188_89276_object119190:head# 2024-04-23T16:25:15.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2eaba9:::benchmark_data_smithi188_89276_object137023:head# 2024-04-23T16:25:15.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2eeecf:::benchmark_data_smithi188_89276_object99121:head# 2024-04-23T16:25:15.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2f44c2:::benchmark_data_smithi188_89276_object113436:head# 2024-04-23T16:25:15.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e2f55a5:::benchmark_data_smithi188_89276_object148596:head# 2024-04-23T16:25:15.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e301177:::benchmark_data_smithi188_89276_object52959:head# 2024-04-23T16:25:15.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e304070:::benchmark_data_smithi188_89276_object69005:head# 2024-04-23T16:25:15.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e30777a:::benchmark_data_smithi188_89276_object68851:head# 2024-04-23T16:25:15.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e308816:::benchmark_data_smithi188_89276_object85111:head# 2024-04-23T16:25:15.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e30f27b:::benchmark_data_smithi188_89276_object89590:head# 2024-04-23T16:25:15.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e3112e3:::benchmark_data_smithi188_89276_object138291:head# 2024-04-23T16:25:15.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e311913:::benchmark_data_smithi188_89276_object97438:head# 2024-04-23T16:25:15.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e31977a:::benchmark_data_smithi188_89276_object142008:head# 2024-04-23T16:25:15.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e31dbee:::benchmark_data_smithi188_89276_object67806:head# 2024-04-23T16:25:15.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e324b19:::benchmark_data_smithi188_89276_object143684:head# 2024-04-23T16:25:15.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e3269f3:::benchmark_data_smithi188_89276_object74708:head# 2024-04-23T16:25:15.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e32c25a:::benchmark_data_smithi188_89276_object130317:head# 2024-04-23T16:25:15.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e3318a7:::benchmark_data_smithi188_89276_object95075:head# 2024-04-23T16:25:15.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e3406e7:::benchmark_data_smithi188_89276_object149343:head# 2024-04-23T16:25:15.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e350879:::benchmark_data_smithi188_89276_object54380:head# 2024-04-23T16:25:15.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e357e21:::benchmark_data_smithi188_89276_object157106:head# 2024-04-23T16:25:15.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e35e287:::benchmark_data_smithi188_89276_object165418:head# 2024-04-23T16:25:15.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e35fcc3:::benchmark_data_smithi188_89276_object51930:head# 2024-04-23T16:25:15.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e36a28b:::benchmark_data_smithi188_89276_object69709:head# 2024-04-23T16:25:15.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e36acc1:::benchmark_data_smithi188_89276_object50552:head# 2024-04-23T16:25:15.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e37b9d1:::benchmark_data_smithi188_89276_object94116:head# 2024-04-23T16:25:15.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e37f4ea:::benchmark_data_smithi188_89276_object101753:head# 2024-04-23T16:25:15.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e381448:::benchmark_data_smithi188_89276_object78583:head# 2024-04-23T16:25:15.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e383527:::benchmark_data_smithi188_89276_object47436:head# 2024-04-23T16:25:15.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e391135:::benchmark_data_smithi188_89276_object50684:head# 2024-04-23T16:25:15.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e392457:::benchmark_data_smithi188_89276_object159029:head# 2024-04-23T16:25:15.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e393665:::benchmark_data_smithi188_89276_object74411:head# 2024-04-23T16:25:15.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e39627e:::benchmark_data_smithi188_89276_object126066:head# 2024-04-23T16:25:15.997 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e39951f:::benchmark_data_smithi188_89276_object103285:head# 2024-04-23T16:25:15.997 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e39ecc4:::benchmark_data_smithi188_89276_object136060:head# 2024-04-23T16:25:15.998 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e3ac9e4:::benchmark_data_smithi188_89276_object48578:head# 2024-04-23T16:25:15.998 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e3c0930:::benchmark_data_smithi188_89276_object116851:head# 2024-04-23T16:25:15.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e3c2669:::benchmark_data_smithi188_89276_object80557:head# 2024-04-23T16:25:15.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e3cac1a:::benchmark_data_smithi188_89276_object149839:head# 2024-04-23T16:25:15.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e3cba5c:::benchmark_data_smithi188_89276_object128301:head# 2024-04-23T16:25:16.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e3cf166:::benchmark_data_smithi188_89276_object125153:head# 2024-04-23T16:25:16.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e3d1a2c:::benchmark_data_smithi188_89276_object158625:head# 2024-04-23T16:25:16.001 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e3d4b08:::benchmark_data_smithi188_89276_object155789:head# 2024-04-23T16:25:16.001 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e3d5f59:::benchmark_data_smithi188_89276_object88709:head# 2024-04-23T16:25:16.001 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e3df7b4:::benchmark_data_smithi188_89276_object113111:head# 2024-04-23T16:25:16.002 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e3df874:::benchmark_data_smithi188_89276_object61440:head# 2024-04-23T16:25:16.002 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e3e0a60:::benchmark_data_smithi188_89276_object74696:head# 2024-04-23T16:25:16.003 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e41a5ba:::benchmark_data_smithi188_89276_object125494:head# 2024-04-23T16:25:16.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e41cef6:::benchmark_data_smithi188_89276_object149490:head# 2024-04-23T16:25:16.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e41ec36:::benchmark_data_smithi188_89276_object79292:head# 2024-04-23T16:25:16.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e423cf3:::benchmark_data_smithi188_89276_object164192:head# 2024-04-23T16:25:16.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e429db0:::benchmark_data_smithi188_89276_object53335:head# 2024-04-23T16:25:16.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e42a9a0:::benchmark_data_smithi188_89276_object127353:head# 2024-04-23T16:25:16.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e42d61e:::benchmark_data_smithi188_89276_object112247:head# 2024-04-23T16:25:16.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e432a9e:::benchmark_data_smithi188_89276_object157669:head# 2024-04-23T16:25:16.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e438f87:::benchmark_data_smithi188_89276_object79348:head# 2024-04-23T16:25:16.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e44cdf9:::benchmark_data_smithi188_89276_object106781:head# 2024-04-23T16:25:16.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e468941:::benchmark_data_smithi188_89276_object43264:head# 2024-04-23T16:25:16.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e47360f:::benchmark_data_smithi188_89276_object82484:head# 2024-04-23T16:25:16.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e475b6f:::benchmark_data_smithi188_89276_object155633:head# 2024-04-23T16:25:16.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e47d352:::benchmark_data_smithi188_89276_object78721:head# 2024-04-23T16:25:16.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e49d1b1:::benchmark_data_smithi188_89276_object83861:head# 2024-04-23T16:25:16.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e49e872:::benchmark_data_smithi188_89276_object69205:head# 2024-04-23T16:25:16.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e4a32e1:::benchmark_data_smithi188_89276_object161553:head# 2024-04-23T16:25:16.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e4a998b:::benchmark_data_smithi188_89276_object142709:head# 2024-04-23T16:25:16.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e4af81d:::benchmark_data_smithi188_89276_object66179:head# 2024-04-23T16:25:16.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e4af9df:::benchmark_data_smithi188_89276_object89356:head# 2024-04-23T16:25:16.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e4c1ca8:::benchmark_data_smithi188_89276_object137674:head# 2024-04-23T16:25:16.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e4c9ae2:::benchmark_data_smithi188_89276_object123249:head# 2024-04-23T16:25:16.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e4ce021:::benchmark_data_smithi188_89276_object143076:head# 2024-04-23T16:25:16.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e4fc793:::benchmark_data_smithi188_89276_object147945:head# 2024-04-23T16:25:16.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5067be:::benchmark_data_smithi188_89276_object74199:head# 2024-04-23T16:25:16.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e506a84:::benchmark_data_smithi188_89276_object145407:head# 2024-04-23T16:25:16.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e50ae9f:::benchmark_data_smithi188_89276_object84344:head# 2024-04-23T16:25:16.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e515875:::benchmark_data_smithi188_89276_object68180:head# 2024-04-23T16:25:16.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e528679:::benchmark_data_smithi188_89276_object62688:head# 2024-04-23T16:25:16.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5294be:::benchmark_data_smithi188_89276_object162878:head# 2024-04-23T16:25:16.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e52d06d:::benchmark_data_smithi188_89276_object52757:head# 2024-04-23T16:25:16.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e53e0ce:::benchmark_data_smithi188_89276_object92487:head# 2024-04-23T16:25:16.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e540bcb:::benchmark_data_smithi188_89276_object128088:head# 2024-04-23T16:25:16.018 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e546aa0:::benchmark_data_smithi188_89276_object166114:head# 2024-04-23T16:25:16.018 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5488d4:::benchmark_data_smithi188_89276_object126721:head# 2024-04-23T16:25:16.018 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e553619:::benchmark_data_smithi188_89276_object135991:head# 2024-04-23T16:25:16.019 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e56e678:::benchmark_data_smithi188_89276_object87369:head# 2024-04-23T16:25:16.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e57bbd1:::benchmark_data_smithi188_89276_object78397:head# 2024-04-23T16:25:16.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e57d98a:::benchmark_data_smithi188_89276_object43868:head# 2024-04-23T16:25:16.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5924e1:::benchmark_data_smithi188_89276_object136655:head# 2024-04-23T16:25:16.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e59a917:::benchmark_data_smithi188_89276_object61139:head# 2024-04-23T16:25:16.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e59fb8f:::benchmark_data_smithi188_89276_object103845:head# 2024-04-23T16:25:16.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5a078e:::benchmark_data_smithi188_89276_object163405:head# 2024-04-23T16:25:16.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5a6472:::benchmark_data_smithi188_89276_object140160:head# 2024-04-23T16:25:16.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5a73ca:::benchmark_data_smithi188_89276_object141994:head# 2024-04-23T16:25:16.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5aff84:::benchmark_data_smithi188_89276_object58650:head# 2024-04-23T16:25:16.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5affc4:::benchmark_data_smithi188_89276_object157197:head# 2024-04-23T16:25:16.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5b75ca:::benchmark_data_smithi188_89276_object93013:head# 2024-04-23T16:25:16.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5bc12f:::benchmark_data_smithi188_89276_object130378:head# 2024-04-23T16:25:16.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5c4987:::benchmark_data_smithi188_89276_object133759:head# 2024-04-23T16:25:16.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5c6a20:::benchmark_data_smithi188_89276_object56476:head# 2024-04-23T16:25:16.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5e783e:::benchmark_data_smithi188_89276_object109329:head# 2024-04-23T16:25:16.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5e8a49:::benchmark_data_smithi188_89276_object61919:head# 2024-04-23T16:25:16.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5e93d7:::benchmark_data_smithi188_89276_object116164:head# 2024-04-23T16:25:16.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5eaa7d:::benchmark_data_smithi188_89276_object88004:head# 2024-04-23T16:25:16.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5edc85:::benchmark_data_smithi188_89276_object135052:head# 2024-04-23T16:25:16.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5f029c:::benchmark_data_smithi188_89276_object141425:head# 2024-04-23T16:25:16.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e5f76a6:::benchmark_data_smithi188_89276_object168202:head# 2024-04-23T16:25:16.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e60987f:::benchmark_data_smithi188_89276_object52544:head# 2024-04-23T16:25:16.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e609b46:::benchmark_data_smithi188_89276_object110766:head# 2024-04-23T16:25:16.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e614baf:::benchmark_data_smithi188_89276_object159319:head# 2024-04-23T16:25:16.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e616d5e:::benchmark_data_smithi188_89276_object94369:head# 2024-04-23T16:25:16.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e61efcd:::benchmark_data_smithi188_89276_object165507:head# 2024-04-23T16:25:16.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e659cec:::benchmark_data_smithi188_89276_object77475:head# 2024-04-23T16:25:16.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e66de20:::benchmark_data_smithi188_89276_object87103:head# 2024-04-23T16:25:16.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e66eb83:::benchmark_data_smithi188_89276_object152721:head# 2024-04-23T16:25:16.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e683dc7:::benchmark_data_smithi188_89276_object61148:head# 2024-04-23T16:25:16.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e69048e:::benchmark_data_smithi188_89276_object120172:head# 2024-04-23T16:25:16.034 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e6a235e:::benchmark_data_smithi188_89276_object70622:head# 2024-04-23T16:25:16.034 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e6a2440:::benchmark_data_smithi188_89276_object63963:head# 2024-04-23T16:25:16.034 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e6a627b:::benchmark_data_smithi188_89276_object44417:head# 2024-04-23T16:25:16.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e6bab34:::benchmark_data_smithi188_89276_object90031:head# 2024-04-23T16:25:16.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e6bb180:::benchmark_data_smithi188_89276_object79621:head# 2024-04-23T16:25:16.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e6bcc43:::benchmark_data_smithi188_89276_object79210:head# 2024-04-23T16:25:16.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e6c38e5:::benchmark_data_smithi188_89276_object168595:head# 2024-04-23T16:25:16.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e6cab2b:::benchmark_data_smithi188_89276_object127003:head# 2024-04-23T16:25:16.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e6cd6f6:::benchmark_data_smithi188_89276_object116027:head# 2024-04-23T16:25:16.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e6cf6c1:::benchmark_data_smithi188_89276_object70381:head# 2024-04-23T16:25:16.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e6dc8e1:::benchmark_data_smithi188_89276_object164850:head# 2024-04-23T16:25:16.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e6e701a:::benchmark_data_smithi188_89276_object64174:head# 2024-04-23T16:25:16.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e71a29f:::benchmark_data_smithi188_89276_object65964:head# 2024-04-23T16:25:16.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e71c8f6:::benchmark_data_smithi188_89276_object71934:head# 2024-04-23T16:25:16.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e727670:::benchmark_data_smithi188_89276_object50134:head# 2024-04-23T16:25:16.040 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e7395fb:::benchmark_data_smithi188_89276_object123882:head# 2024-04-23T16:25:16.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e73d0e6:::benchmark_data_smithi188_89276_object46763:head# 2024-04-23T16:25:16.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e74890c:::benchmark_data_smithi188_89276_object81163:head# 2024-04-23T16:25:16.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e74a7cb:::benchmark_data_smithi188_89276_object53639:head# 2024-04-23T16:25:16.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e75516b:::benchmark_data_smithi188_89276_object55621:head# 2024-04-23T16:25:16.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e76964e:::benchmark_data_smithi188_89276_object74187:head# 2024-04-23T16:25:16.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e76efe4:::benchmark_data_smithi188_89276_object57651:head# 2024-04-23T16:25:16.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e771192:::benchmark_data_smithi188_89276_object87334:head# 2024-04-23T16:25:16.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e77b842:::benchmark_data_smithi188_89276_object57432:head# 2024-04-23T16:25:16.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e789985:::benchmark_data_smithi188_89276_object109005:head# 2024-04-23T16:25:16.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e78a1a0:::benchmark_data_smithi188_89276_object159713:head# 2024-04-23T16:25:16.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e79709f:::benchmark_data_smithi188_89276_object71931:head# 2024-04-23T16:25:16.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e79939b:::benchmark_data_smithi188_89276_object164533:head# 2024-04-23T16:25:16.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e79c3c9:::benchmark_data_smithi188_89276_object74634:head# 2024-04-23T16:25:16.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e79e28d:::benchmark_data_smithi188_89276_object55682:head# 2024-04-23T16:25:16.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e7a9bb1:::benchmark_data_smithi188_89276_object109201:head# 2024-04-23T16:25:16.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e7b2fb7:::benchmark_data_smithi188_89276_object45466:head# 2024-04-23T16:25:16.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e7b3834:::benchmark_data_smithi188_89276_object89475:head# 2024-04-23T16:25:16.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e7ba7b5:::benchmark_data_smithi188_89276_object70993:head# 2024-04-23T16:25:16.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e7be0ad:::benchmark_data_smithi188_89276_object103800:head# 2024-04-23T16:25:16.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e7c8956:::benchmark_data_smithi188_89276_object101140:head# 2024-04-23T16:25:16.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e7d6c0e:::benchmark_data_smithi188_89276_object115368:head# 2024-04-23T16:25:16.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e7e3f11:::benchmark_data_smithi188_89276_object146437:head# 2024-04-23T16:25:16.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e7fa208:::benchmark_data_smithi188_89276_object99351:head# 2024-04-23T16:25:16.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e802cf4:::benchmark_data_smithi188_89276_object46469:head# 2024-04-23T16:25:16.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8088dd:::benchmark_data_smithi188_89276_object88780:head# 2024-04-23T16:25:16.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e80e7b3:::benchmark_data_smithi188_89276_object167863:head# 2024-04-23T16:25:16.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e812e83:::benchmark_data_smithi188_89276_object153748:head# 2024-04-23T16:25:16.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8224fe:::benchmark_data_smithi188_89276_object165084:head# 2024-04-23T16:25:16.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e83521d:::benchmark_data_smithi188_89276_object53058:head# 2024-04-23T16:25:16.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e83aef7:::benchmark_data_smithi188_89276_object53556:head# 2024-04-23T16:25:16.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e84028a:::benchmark_data_smithi188_89276_object95685:head# 2024-04-23T16:25:16.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8407e4:::benchmark_data_smithi188_89276_object59739:head# 2024-04-23T16:25:16.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e847922:::benchmark_data_smithi188_89276_object111924:head# 2024-04-23T16:25:16.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e84cdb1:::benchmark_data_smithi188_89276_object76058:head# 2024-04-23T16:25:16.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8561fd:::benchmark_data_smithi188_89276_object40421:head# 2024-04-23T16:25:16.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e85c7e9:::benchmark_data_smithi188_89276_object44824:head# 2024-04-23T16:25:16.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e85defa:::benchmark_data_smithi188_89276_object61573:head# 2024-04-23T16:25:16.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8602f8:::benchmark_data_smithi188_89276_object148022:head# 2024-04-23T16:25:16.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e869fac:::benchmark_data_smithi188_89276_object49865:head# 2024-04-23T16:25:16.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e86da57:::benchmark_data_smithi188_89276_object68897:head# 2024-04-23T16:25:16.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e875929:::benchmark_data_smithi188_89276_object96398:head# 2024-04-23T16:25:16.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8764c4:::benchmark_data_smithi188_89276_object117947:head# 2024-04-23T16:25:16.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e885689:::benchmark_data_smithi188_89276_object125161:head# 2024-04-23T16:25:16.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8857fe:::benchmark_data_smithi188_89276_object144089:head# 2024-04-23T16:25:16.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e89bc50:::benchmark_data_smithi188_89276_object66985:head# 2024-04-23T16:25:16.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e89f9f3:::benchmark_data_smithi188_89276_object147689:head# 2024-04-23T16:25:16.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8a4b93:::benchmark_data_smithi188_89276_object136634:head# 2024-04-23T16:25:16.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8b3db2:::benchmark_data_smithi188_89276_object127136:head# 2024-04-23T16:25:16.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8bb07b:::benchmark_data_smithi188_89276_object112882:head# 2024-04-23T16:25:16.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8bc18e:::benchmark_data_smithi188_89276_object89607:head# 2024-04-23T16:25:16.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8bcf22:::benchmark_data_smithi188_89276_object123899:head# 2024-04-23T16:25:16.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8c06db:::benchmark_data_smithi188_89276_object51901:head# 2024-04-23T16:25:16.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8c3260:::benchmark_data_smithi188_89276_object65364:head# 2024-04-23T16:25:16.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8c9b93:::benchmark_data_smithi188_89276_object107859:head# 2024-04-23T16:25:16.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8cae79:::benchmark_data_smithi188_89276_object117895:head# 2024-04-23T16:25:16.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8df98b:::benchmark_data_smithi188_89276_object118942:head# 2024-04-23T16:25:16.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8e2584:::benchmark_data_smithi188_89276_object93771:head# 2024-04-23T16:25:16.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8e329d:::benchmark_data_smithi188_89276_object96266:head# 2024-04-23T16:25:16.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8e53fa:::benchmark_data_smithi188_89276_object164586:head# 2024-04-23T16:25:16.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8eff44:::benchmark_data_smithi188_89276_object168192:head# 2024-04-23T16:25:16.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8f5db4:::benchmark_data_smithi188_89276_object166605:head# 2024-04-23T16:25:16.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e8ffb6f:::benchmark_data_smithi188_89276_object119727:head# 2024-04-23T16:25:16.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e92053f:::benchmark_data_smithi188_89276_object40560:head# 2024-04-23T16:25:16.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e921cc7:::benchmark_data_smithi188_89276_object164844:head# 2024-04-23T16:25:16.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e929411:::benchmark_data_smithi188_89276_object97355:head# 2024-04-23T16:25:16.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e92ff52:::benchmark_data_smithi188_89276_object167004:head# 2024-04-23T16:25:16.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e952039:::benchmark_data_smithi188_89276_object149658:head# 2024-04-23T16:25:16.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e95480d:::benchmark_data_smithi188_89276_object148198:head# 2024-04-23T16:25:16.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e95ec0a:::benchmark_data_smithi188_89276_object127748:head# 2024-04-23T16:25:16.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e962f36:::benchmark_data_smithi188_89276_object67619:head# 2024-04-23T16:25:16.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e971a53:::benchmark_data_smithi188_89276_object69014:head# 2024-04-23T16:25:16.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e980a9f:::benchmark_data_smithi188_89276_object132085:head# 2024-04-23T16:25:16.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e99df12:::benchmark_data_smithi188_89276_object164862:head# 2024-04-23T16:25:16.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e9a6560:::benchmark_data_smithi188_89276_object122280:head# 2024-04-23T16:25:16.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e9aed41:::benchmark_data_smithi188_89276_object130445:head# 2024-04-23T16:25:16.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e9b89b6:::benchmark_data_smithi188_89276_object74256:head# 2024-04-23T16:25:16.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e9db5b8:::benchmark_data_smithi188_89276_object85555:head# 2024-04-23T16:25:16.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2e9f2e33:::benchmark_data_smithi188_89276_object152173:head# 2024-04-23T16:25:16.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ea0c724:::benchmark_data_smithi188_89276_object48761:head# 2024-04-23T16:25:16.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ea1778e:::benchmark_data_smithi188_89276_object54333:head# 2024-04-23T16:25:16.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ea2710a:::benchmark_data_smithi188_89276_object91331:head# 2024-04-23T16:25:16.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ea298b0:::benchmark_data_smithi188_89276_object87245:head# 2024-04-23T16:25:16.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ea2b1a9:::benchmark_data_smithi188_89276_object102068:head# 2024-04-23T16:25:16.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ea32964:::benchmark_data_smithi188_89276_object63296:head# 2024-04-23T16:25:16.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ea40ad9:::benchmark_data_smithi188_89276_object94889:head# 2024-04-23T16:25:16.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ea41a6c:::benchmark_data_smithi188_89276_object101793:head# 2024-04-23T16:25:16.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ea4c8b1:::benchmark_data_smithi188_89276_object129535:head# 2024-04-23T16:25:16.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ea53c2a:::benchmark_data_smithi188_89276_object102659:head# 2024-04-23T16:25:16.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ea5f4cc:::benchmark_data_smithi188_89276_object59378:head# 2024-04-23T16:25:16.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ea70200:::benchmark_data_smithi188_89276_object77463:head# 2024-04-23T16:25:16.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ea76e96:::benchmark_data_smithi188_89276_object147807:head# 2024-04-23T16:25:16.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ea79a60:::benchmark_data_smithi188_89276_object49782:head# 2024-04-23T16:25:16.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ea8ed1d:::benchmark_data_smithi188_89276_object95908:head# 2024-04-23T16:25:16.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ea957df:::benchmark_data_smithi188_89276_object49412:head# 2024-04-23T16:25:16.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ea98986:::benchmark_data_smithi188_89276_object91056:head# 2024-04-23T16:25:16.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eaade55:::benchmark_data_smithi188_89276_object97405:head# 2024-04-23T16:25:16.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eab17c2:::benchmark_data_smithi188_89276_object112768:head# 2024-04-23T16:25:16.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eab75cb:::benchmark_data_smithi188_89276_object124965:head# 2024-04-23T16:25:16.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eab7f9a:::benchmark_data_smithi188_89276_object83419:head# 2024-04-23T16:25:16.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eac3b07:::benchmark_data_smithi188_89276_object166496:head# 2024-04-23T16:25:16.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eac81c9:::benchmark_data_smithi188_89276_object68224:head# 2024-04-23T16:25:16.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eaced21:::benchmark_data_smithi188_89276_object50338:head# 2024-04-23T16:25:16.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ead9346:::benchmark_data_smithi188_89276_object105549:head# 2024-04-23T16:25:16.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eadf70d:::benchmark_data_smithi188_89276_object77425:head# 2024-04-23T16:25:16.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eae28ae:::benchmark_data_smithi188_89276_object43184:head# 2024-04-23T16:25:16.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eaec45f:::benchmark_data_smithi188_89276_object43632:head# 2024-04-23T16:25:16.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eaf433d:::benchmark_data_smithi188_89276_object47776:head# 2024-04-23T16:25:16.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eafda6b:::benchmark_data_smithi188_89276_object107253:head# 2024-04-23T16:25:16.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb00fc4:::benchmark_data_smithi188_89276_object148151:head# 2024-04-23T16:25:16.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb01bb4:::benchmark_data_smithi188_89276_object84665:head# 2024-04-23T16:25:16.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb182fc:::benchmark_data_smithi188_89276_object122141:head# 2024-04-23T16:25:16.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb2bf61:::benchmark_data_smithi188_89276_object128435:head# 2024-04-23T16:25:16.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb3ca39:::benchmark_data_smithi188_89276_object156436:head# 2024-04-23T16:25:16.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb4ffd0:::benchmark_data_smithi188_89276_object149883:head# 2024-04-23T16:25:16.091 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb52ebf:::benchmark_data_smithi188_89276_object125905:head# 2024-04-23T16:25:16.091 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb54f2b:::benchmark_data_smithi188_89276_object117519:head# 2024-04-23T16:25:16.091 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb5a46e:::benchmark_data_smithi188_89276_object168384:head# 2024-04-23T16:25:16.091 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb5a6ca:::benchmark_data_smithi188_89276_object126920:head# 2024-04-23T16:25:16.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb5d8c8:::benchmark_data_smithi188_89276_object61654:head# 2024-04-23T16:25:16.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb5fd66:::benchmark_data_smithi188_89276_object47980:head# 2024-04-23T16:25:16.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb68894:::benchmark_data_smithi188_89276_object41428:head# 2024-04-23T16:25:16.093 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb78f99:::benchmark_data_smithi188_89276_object89572:head# 2024-04-23T16:25:16.093 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb7922b:::benchmark_data_smithi188_89276_object64212:head# 2024-04-23T16:25:16.093 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb79de2:::benchmark_data_smithi188_89276_object156248:head# 2024-04-23T16:25:16.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb8d63e:::benchmark_data_smithi188_89276_object82103:head# 2024-04-23T16:25:16.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb8e370:::benchmark_data_smithi188_89276_object59388:head# 2024-04-23T16:25:16.095 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb8f00c:::benchmark_data_smithi188_89276_object65986:head# 2024-04-23T16:25:16.095 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb93fc6:::benchmark_data_smithi188_89276_object71337:head# 2024-04-23T16:25:16.096 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb965bc:::benchmark_data_smithi188_89276_object75802:head# 2024-04-23T16:25:16.096 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eb9dddd:::benchmark_data_smithi188_89276_object142341:head# 2024-04-23T16:25:16.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eba52c9:::benchmark_data_smithi188_89276_object105609:head# 2024-04-23T16:25:16.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ebc1c73:::benchmark_data_smithi188_89276_object148892:head# 2024-04-23T16:25:16.099 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:25:16.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ebc3062:::benchmark_data_smithi188_89276_object146638:head# 2024-04-23T16:25:16.102 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ebc7665:::benchmark_data_smithi188_89276_object165489:head# 2024-04-23T16:25:16.102 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ebcaf74:::benchmark_data_smithi188_89276_object151444:head# 2024-04-23T16:25:16.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ebcdeae:::benchmark_data_smithi188_89276_object102644:head# 2024-04-23T16:25:16.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ebe2aec:::benchmark_data_smithi188_89276_object104413:head# 2024-04-23T16:25:16.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:16.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:25:16.105 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ebe4750:::benchmark_data_smithi188_89276_object49063:head# 2024-04-23T16:25:16.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ebed9cc:::benchmark_data_smithi188_89276_object125501:head# 2024-04-23T16:25:16.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ebf2b69:::benchmark_data_smithi188_89276_object119742:head# 2024-04-23T16:25:16.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ebfc367:::benchmark_data_smithi188_89276_object157256:head# 2024-04-23T16:25:16.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ebfd291:::benchmark_data_smithi188_89276_object157042:head# 2024-04-23T16:25:16.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ebfecd8:::benchmark_data_smithi188_89276_object52258:head# 2024-04-23T16:25:16.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ec05425:::benchmark_data_smithi188_89276_object79162:head# 2024-04-23T16:25:16.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ec0b511:::benchmark_data_smithi188_89276_object150633:head# 2024-04-23T16:25:16.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ec0e3bb:::benchmark_data_smithi188_89276_object82196:head# 2024-04-23T16:25:16.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ec174d9:::benchmark_data_smithi188_89276_object58498:head# 2024-04-23T16:25:16.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ec18027:::benchmark_data_smithi188_89276_object102639:head# 2024-04-23T16:25:16.113 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ec321cf:::benchmark_data_smithi188_89276_object61706:head# 2024-04-23T16:25:16.113 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ec3b351:::benchmark_data_smithi188_89276_object114423:head# 2024-04-23T16:25:16.113 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ec3d2f4:::benchmark_data_smithi188_89276_object42473:head# 2024-04-23T16:25:16.113 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ec43b34:::benchmark_data_smithi188_89276_object114225:head# 2024-04-23T16:25:16.113 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ec4832d:::benchmark_data_smithi188_89276_object54253:head# 2024-04-23T16:25:16.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ec4bf5e:::benchmark_data_smithi188_89276_object68223:head# 2024-04-23T16:25:16.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ec6a19a:::benchmark_data_smithi188_89276_object103752:head# 2024-04-23T16:25:16.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ec73526:::benchmark_data_smithi188_89276_object142497:head# 2024-04-23T16:25:16.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ec7ed10:::benchmark_data_smithi188_89276_object151998:head# 2024-04-23T16:25:16.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ec81a16:::benchmark_data_smithi188_89276_object135033:head# 2024-04-23T16:25:16.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ec95efe:::benchmark_data_smithi188_89276_object92289:head# 2024-04-23T16:25:16.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eca5df6:::benchmark_data_smithi188_89276_object59138:head# 2024-04-23T16:25:16.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ecb4dce:::benchmark_data_smithi188_89276_object52047:head# 2024-04-23T16:25:16.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ecc4517:::benchmark_data_smithi188_89276_object85991:head# 2024-04-23T16:25:16.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ecc588f:::benchmark_data_smithi188_89276_object136159:head# 2024-04-23T16:25:16.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ecc846c:::benchmark_data_smithi188_89276_object110713:head# 2024-04-23T16:25:16.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eccd158:::benchmark_data_smithi188_89276_object139455:head# 2024-04-23T16:25:16.122 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ecd111e:::benchmark_data_smithi188_89276_object121574:head# 2024-04-23T16:25:16.122 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eceb131:::benchmark_data_smithi188_89276_object167398:head# 2024-04-23T16:25:16.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eceebe2:::benchmark_data_smithi188_89276_object144039:head# 2024-04-23T16:25:16.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ecf843f:::benchmark_data_smithi188_89276_object135750:head# 2024-04-23T16:25:16.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ecfd966:::benchmark_data_smithi188_89276_object132776:head# 2024-04-23T16:25:16.124 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ed033ef:::benchmark_data_smithi188_89276_object94434:head# 2024-04-23T16:25:16.124 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ed09844:::benchmark_data_smithi188_89276_object71261:head# 2024-04-23T16:25:16.125 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ed221bf:::benchmark_data_smithi188_89276_object50540:head# 2024-04-23T16:25:16.125 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ed3e8a5:::benchmark_data_smithi188_89276_object142009:head# 2024-04-23T16:25:16.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ed49609:::benchmark_data_smithi188_89276_object48405:head# 2024-04-23T16:25:16.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ed4e758:::benchmark_data_smithi188_89276_object132079:head# 2024-04-23T16:25:16.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ed4f08c:::benchmark_data_smithi188_89276_object150684:head# 2024-04-23T16:25:16.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ed73cf2:::benchmark_data_smithi188_89276_object166716:head# 2024-04-23T16:25:16.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ed7507f:::benchmark_data_smithi188_89276_object58605:head# 2024-04-23T16:25:16.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ed8355d:::benchmark_data_smithi188_89276_object160408:head# 2024-04-23T16:25:16.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ed8a6b5:::benchmark_data_smithi188_89276_object87633:head# 2024-04-23T16:25:16.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ed8dfe7:::benchmark_data_smithi188_89276_object72898:head# 2024-04-23T16:25:16.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ed912af:::benchmark_data_smithi188_89276_object41922:head# 2024-04-23T16:25:16.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ed92f06:::benchmark_data_smithi188_89276_object130779:head# 2024-04-23T16:25:16.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2edb13ad:::benchmark_data_smithi188_89276_object137824:head# 2024-04-23T16:25:16.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2edb48fa:::benchmark_data_smithi188_89276_object40700:head# 2024-04-23T16:25:16.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2edbe776:::benchmark_data_smithi188_89276_object157263:head# 2024-04-23T16:25:16.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2edc579a:::benchmark_data_smithi188_89276_object163174:head# 2024-04-23T16:25:16.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2edca302:::benchmark_data_smithi188_89276_object151192:head# 2024-04-23T16:25:16.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2edd0a14:::benchmark_data_smithi188_89276_object75325:head# 2024-04-23T16:25:16.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2edd283d:::benchmark_data_smithi188_89276_object58769:head# 2024-04-23T16:25:16.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2edd3923:::benchmark_data_smithi188_89276_object130581:head# 2024-04-23T16:25:16.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2edd64f6:::benchmark_data_smithi188_89276_object80036:head# 2024-04-23T16:25:16.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2edd822d:::benchmark_data_smithi188_89276_object110213:head# 2024-04-23T16:25:16.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eddb679:::benchmark_data_smithi188_89276_object94488:head# 2024-04-23T16:25:16.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eddcf5e:::benchmark_data_smithi188_89276_object58347:head# 2024-04-23T16:25:16.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ede0685:::benchmark_data_smithi188_89276_object90750:head# 2024-04-23T16:25:16.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ede09f8:::benchmark_data_smithi188_89276_object128027:head# 2024-04-23T16:25:16.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ede4a19:::benchmark_data_smithi188_89276_object122591:head# 2024-04-23T16:25:16.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2edec543:::benchmark_data_smithi188_89276_object120872:head# 2024-04-23T16:25:16.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2edf21d1:::benchmark_data_smithi188_89276_object126431:head# 2024-04-23T16:25:16.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee05cbe:::benchmark_data_smithi188_89276_object46774:head# 2024-04-23T16:25:16.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee192f2:::benchmark_data_smithi188_89276_object100699:head# 2024-04-23T16:25:16.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee1fbf7:::benchmark_data_smithi188_89276_object90653:head# 2024-04-23T16:25:16.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee22af9:::benchmark_data_smithi188_89276_object159126:head# 2024-04-23T16:25:16.142 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee29a24:::benchmark_data_smithi188_89276_object92705:head# 2024-04-23T16:25:16.142 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee2c9b4:::benchmark_data_smithi188_89276_object128035:head# 2024-04-23T16:25:16.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee2e9a7:::benchmark_data_smithi188_89276_object45690:head# 2024-04-23T16:25:16.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee2ec08:::benchmark_data_smithi188_89276_object89214:head# 2024-04-23T16:25:16.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee2f779:::benchmark_data_smithi188_89276_object163016:head# 2024-04-23T16:25:16.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee2fdab:::benchmark_data_smithi188_89276_object78365:head# 2024-04-23T16:25:16.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee2fe24:::benchmark_data_smithi188_89276_object81878:head# 2024-04-23T16:25:16.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee3d564:::benchmark_data_smithi188_89276_object140961:head# 2024-04-23T16:25:16.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee48f62:::benchmark_data_smithi188_89276_object98533:head# 2024-04-23T16:25:16.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee4ccb9:::benchmark_data_smithi188_89276_object91150:head# 2024-04-23T16:25:16.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee4e0b8:::benchmark_data_smithi188_89276_object56275:head# 2024-04-23T16:25:16.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee4f7ed:::benchmark_data_smithi188_89276_object157434:head# 2024-04-23T16:25:16.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee53e73:::benchmark_data_smithi188_89276_object69373:head# 2024-04-23T16:25:16.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee685e1:::benchmark_data_smithi188_89276_object46660:head# 2024-04-23T16:25:16.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee6c1af:::benchmark_data_smithi188_89276_object159640:head# 2024-04-23T16:25:16.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee6d5a4:::benchmark_data_smithi188_89276_object73355:head# 2024-04-23T16:25:16.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee7d95f:::benchmark_data_smithi188_89276_object76668:head# 2024-04-23T16:25:16.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee82c54:::benchmark_data_smithi188_89276_object93393:head# 2024-04-23T16:25:16.155 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee8d2ca:::benchmark_data_smithi188_89276_object161716:head# 2024-04-23T16:25:16.155 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee8fd7a:::benchmark_data_smithi188_89276_object147845:head# 2024-04-23T16:25:16.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee9235d:::benchmark_data_smithi188_89276_object149549:head# 2024-04-23T16:25:16.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ee948c3:::benchmark_data_smithi188_89276_object154352:head# 2024-04-23T16:25:16.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eea068f:::benchmark_data_smithi188_89276_object99712:head# 2024-04-23T16:25:16.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eeb124d:::benchmark_data_smithi188_89276_object99291:head# 2024-04-23T16:25:16.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eeb2bf5:::benchmark_data_smithi188_89276_object156272:head# 2024-04-23T16:25:16.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eeb4580:::benchmark_data_smithi188_89276_object121459:head# 2024-04-23T16:25:16.158 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eebb674:::benchmark_data_smithi188_89276_object69338:head# 2024-04-23T16:25:16.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eec22db:::benchmark_data_smithi188_89276_object130937:head# 2024-04-23T16:25:16.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eecd805:::benchmark_data_smithi188_89276_object132213:head# 2024-04-23T16:25:16.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eede7d4:::benchmark_data_smithi188_89276_object85485:head# 2024-04-23T16:25:16.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2eefe2d6:::benchmark_data_smithi188_89276_object135824:head# 2024-04-23T16:25:16.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ef046b7:::benchmark_data_smithi188_89276_object61866:head# 2024-04-23T16:25:16.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ef0960a:::benchmark_data_smithi188_89276_object60021:head# 2024-04-23T16:25:16.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ef0c223:::benchmark_data_smithi188_89276_object105590:head# 2024-04-23T16:25:16.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ef106bc:::benchmark_data_smithi188_89276_object42915:head# 2024-04-23T16:25:16.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ef1ca16:::benchmark_data_smithi188_89276_object140296:head# 2024-04-23T16:25:16.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ef267d8:::benchmark_data_smithi188_89276_object94293:head# 2024-04-23T16:25:16.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ef36573:::benchmark_data_smithi188_89276_object85632:head# 2024-04-23T16:25:16.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ef38205:::benchmark_data_smithi188_89276_object147388:head# 2024-04-23T16:25:16.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ef44fcc:::benchmark_data_smithi188_89276_object91174:head# 2024-04-23T16:25:16.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ef4e13f:::benchmark_data_smithi188_89276_object159736:head# 2024-04-23T16:25:16.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ef52cc2:::benchmark_data_smithi188_89276_object153275:head# 2024-04-23T16:25:16.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ef5a15a:::benchmark_data_smithi188_89276_object104122:head# 2024-04-23T16:25:16.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ef5df3e:::benchmark_data_smithi188_89276_object80947:head# 2024-04-23T16:25:16.167 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ef672d5:::benchmark_data_smithi188_89276_object68907:head# 2024-04-23T16:25:16.167 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ef682ef:::benchmark_data_smithi188_89276_object76687:head# 2024-04-23T16:25:16.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ef7b6c9:::benchmark_data_smithi188_89276_object96759:head# 2024-04-23T16:25:16.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ef97c79:::benchmark_data_smithi188_89276_object125425:head# 2024-04-23T16:25:16.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ef9d9cd:::benchmark_data_smithi188_89276_object89499:head# 2024-04-23T16:25:16.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2efa0201:::benchmark_data_smithi188_89276_object156223:head# 2024-04-23T16:25:16.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2efb4d6b:::benchmark_data_smithi188_89276_object96117:head# 2024-04-23T16:25:16.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2efd3f26:::benchmark_data_smithi188_89276_object44984:head# 2024-04-23T16:25:16.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2efd43ae:::benchmark_data_smithi188_89276_object87329:head# 2024-04-23T16:25:16.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2efdf18d:::benchmark_data_smithi188_89276_object97885:head# 2024-04-23T16:25:16.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2efe43b3:::benchmark_data_smithi188_89276_object81168:head# 2024-04-23T16:25:16.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2efe9f01:::benchmark_data_smithi188_89276_object67273:head# 2024-04-23T16:25:16.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f00115f:::benchmark_data_smithi188_89276_object137312:head# 2024-04-23T16:25:16.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f00b251:::benchmark_data_smithi188_89276_object50962:head# 2024-04-23T16:25:16.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f00ce8d:::benchmark_data_smithi188_89276_object86472:head# 2024-04-23T16:25:16.176 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f013a7b:::benchmark_data_smithi188_89276_object113377:head# 2024-04-23T16:25:16.176 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f01550c:::benchmark_data_smithi188_89276_object123831:head# 2024-04-23T16:25:16.176 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f0158ce:::benchmark_data_smithi188_89276_object47375:head# 2024-04-23T16:25:16.177 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f01be0d:::benchmark_data_smithi188_89276_object118324:head# 2024-04-23T16:25:16.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f039ef1:::benchmark_data_smithi188_89276_object146467:head# 2024-04-23T16:25:16.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f050270:::benchmark_data_smithi188_89276_object153899:head# 2024-04-23T16:25:16.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f050bfa:::benchmark_data_smithi188_89276_object83514:head# 2024-04-23T16:25:16.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f054f3c:::benchmark_data_smithi188_89276_object107184:head# 2024-04-23T16:25:16.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f05bdeb:::benchmark_data_smithi188_89276_object43060:head# 2024-04-23T16:25:16.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f05ea05:::benchmark_data_smithi188_89276_object67852:head# 2024-04-23T16:25:16.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f05edc6:::benchmark_data_smithi188_89276_object44479:head# 2024-04-23T16:25:16.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f063a4b:::benchmark_data_smithi188_89276_object85324:head# 2024-04-23T16:25:16.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f06cdf3:::benchmark_data_smithi188_89276_object113569:head# 2024-04-23T16:25:16.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f074a54:::benchmark_data_smithi188_89276_object140899:head# 2024-04-23T16:25:16.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f07c10f:::benchmark_data_smithi188_89276_object165905:head# 2024-04-23T16:25:16.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f085640:::benchmark_data_smithi188_89276_object156656:head# 2024-04-23T16:25:16.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f08cb26:::benchmark_data_smithi188_89276_object131292:head# 2024-04-23T16:25:16.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f08cceb:::benchmark_data_smithi188_89276_object77273:head# 2024-04-23T16:25:16.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f09a0f6:::benchmark_data_smithi188_89276_object148801:head# 2024-04-23T16:25:16.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f0ac3b5:::benchmark_data_smithi188_89276_object56037:head# 2024-04-23T16:25:16.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f0ac90f:::benchmark_data_smithi188_89276_object117143:head# 2024-04-23T16:25:16.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f0af30c:::benchmark_data_smithi188_89276_object162902:head# 2024-04-23T16:25:16.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f0c1d1b:::benchmark_data_smithi188_89276_object88534:head# 2024-04-23T16:25:16.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f0e63e1:::benchmark_data_smithi188_89276_object163101:head# 2024-04-23T16:25:16.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f0f0768:::benchmark_data_smithi188_89276_object79439:head# 2024-04-23T16:25:16.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f0f3434:::benchmark_data_smithi188_89276_object54504:head# 2024-04-23T16:25:16.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f0f6468:::benchmark_data_smithi188_89276_object113160:head# 2024-04-23T16:25:16.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f10511b:::benchmark_data_smithi188_89276_object89430:head# 2024-04-23T16:25:16.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f11768d:::benchmark_data_smithi188_89276_object129620:head# 2024-04-23T16:25:16.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f11bafc:::benchmark_data_smithi188_89276_object116692:head# 2024-04-23T16:25:16.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f11d64a:::benchmark_data_smithi188_89276_object66471:head# 2024-04-23T16:25:16.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1308f2:::benchmark_data_smithi188_89276_object70697:head# 2024-04-23T16:25:16.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f13f846:::benchmark_data_smithi188_89276_object124333:head# 2024-04-23T16:25:16.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f141cd2:::benchmark_data_smithi188_89276_object116740:head# 2024-04-23T16:25:16.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f147cae:::benchmark_data_smithi188_89276_object151669:head# 2024-04-23T16:25:16.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f14f9a8:::benchmark_data_smithi188_89276_object156691:head# 2024-04-23T16:25:16.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f151dba:::benchmark_data_smithi188_89276_object90409:head# 2024-04-23T16:25:16.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f16538a:::benchmark_data_smithi188_89276_object56446:head# 2024-04-23T16:25:16.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f17c9be:::benchmark_data_smithi188_89276_object49956:head# 2024-04-23T16:25:16.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1899d9:::benchmark_data_smithi188_89276_object168197:head# 2024-04-23T16:25:16.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f18ca26:::benchmark_data_smithi188_89276_object62324:head# 2024-04-23T16:25:16.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f190aee:::benchmark_data_smithi188_89276_object89797:head# 2024-04-23T16:25:16.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f191d80:::benchmark_data_smithi188_89276_object136124:head# 2024-04-23T16:25:16.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1981fd:::benchmark_data_smithi188_89276_object46175:head# 2024-04-23T16:25:16.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f19df19:::benchmark_data_smithi188_89276_object167169:head# 2024-04-23T16:25:16.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1a1c5e:::benchmark_data_smithi188_89276_object56497:head# 2024-04-23T16:25:16.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1a6236:::benchmark_data_smithi188_89276_object128951:head# 2024-04-23T16:25:16.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1ac694:::benchmark_data_smithi188_89276_object84154:head# 2024-04-23T16:25:16.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1b5fcf:::benchmark_data_smithi188_89276_object154728:head# 2024-04-23T16:25:16.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1bb9af:::benchmark_data_smithi188_89276_object59425:head# 2024-04-23T16:25:16.197 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1bbff9:::benchmark_data_smithi188_89276_object139687:head# 2024-04-23T16:25:16.197 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1c054d:::benchmark_data_smithi188_89276_object74803:head# 2024-04-23T16:25:16.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1cadf6:::benchmark_data_smithi188_89276_object81257:head# 2024-04-23T16:25:16.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1cff1d:::benchmark_data_smithi188_89276_object44923:head# 2024-04-23T16:25:16.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1dbdc7:::benchmark_data_smithi188_89276_object52939:head# 2024-04-23T16:25:16.199 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1e0799:::benchmark_data_smithi188_89276_object113032:head# 2024-04-23T16:25:16.199 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1e6876:::benchmark_data_smithi188_89276_object154326:head# 2024-04-23T16:25:16.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1e9d2a:::benchmark_data_smithi188_89276_object62202:head# 2024-04-23T16:25:16.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1ed0d1:::benchmark_data_smithi188_89276_object55086:head# 2024-04-23T16:25:16.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1eed16:::benchmark_data_smithi188_89276_object133301:head# 2024-04-23T16:25:16.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1f1936:::benchmark_data_smithi188_89276_object120188:head# 2024-04-23T16:25:16.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f1f3229:::benchmark_data_smithi188_89276_object139672:head# 2024-04-23T16:25:16.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f20089e:::benchmark_data_smithi188_89276_object82690:head# 2024-04-23T16:25:16.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f203104:::benchmark_data_smithi188_89276_object98271:head# 2024-04-23T16:25:16.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f21548b:::benchmark_data_smithi188_89276_object104811:head# 2024-04-23T16:25:16.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f220972:::benchmark_data_smithi188_89276_object42717:head# 2024-04-23T16:25:16.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f229033:::benchmark_data_smithi188_89276_object82837:head# 2024-04-23T16:25:16.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f2413e4:::benchmark_data_smithi188_89276_object78322:head# 2024-04-23T16:25:16.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f2440d2:::benchmark_data_smithi188_89276_object52149:head# 2024-04-23T16:25:16.205 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:25:16.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f2443b3:::benchmark_data_smithi188_89276_object133122:head# 2024-04-23T16:25:16.206 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f244ada:::benchmark_data_smithi188_89276_object66865:head# 2024-04-23T16:25:16.206 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f248cf0:::benchmark_data_smithi188_89276_object127730:head# 2024-04-23T16:25:16.206 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f24ca31:::benchmark_data_smithi188_89276_object106749:head# 2024-04-23T16:25:16.207 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f24dcfc:::benchmark_data_smithi188_89276_object158047:head# 2024-04-23T16:25:16.207 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f2554fe:::benchmark_data_smithi188_89276_object47770:head# 2024-04-23T16:25:16.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f267247:::benchmark_data_smithi188_89276_object90979:head# 2024-04-23T16:25:16.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f278959:::benchmark_data_smithi188_89276_object90685:head# 2024-04-23T16:25:16.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f27bfcd:::benchmark_data_smithi188_89276_object59680:head# 2024-04-23T16:25:16.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f27d16f:::benchmark_data_smithi188_89276_object60177:head# 2024-04-23T16:25:16.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f281abb:::benchmark_data_smithi188_89276_object167601:head# 2024-04-23T16:25:16.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f2860f7:::benchmark_data_smithi188_89276_object60098:head# 2024-04-23T16:25:16.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f287e4c:::benchmark_data_smithi188_89276_object137712:head# 2024-04-23T16:25:16.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f299cc5:::benchmark_data_smithi188_89276_object60107:head# 2024-04-23T16:25:16.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f2a12df:::benchmark_data_smithi188_89276_object101391:head# 2024-04-23T16:25:16.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f2b1589:::benchmark_data_smithi188_89276_object43637:head# 2024-04-23T16:25:16.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f2b3733:::benchmark_data_smithi188_89276_object104842:head# 2024-04-23T16:25:16.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f2c664a:::benchmark_data_smithi188_89276_object59324:head# 2024-04-23T16:25:16.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f2dc3b6:::benchmark_data_smithi188_89276_object161831:head# 2024-04-23T16:25:16.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f2dc40b:::benchmark_data_smithi188_89276_object56368:head# 2024-04-23T16:25:16.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f2e18e7:::benchmark_data_smithi188_89276_object149098:head# 2024-04-23T16:25:16.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f2f0cfe:::benchmark_data_smithi188_89276_object40307:head# 2024-04-23T16:25:16.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f300dea:::benchmark_data_smithi188_89276_object57012:head# 2024-04-23T16:25:16.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f31b36d:::benchmark_data_smithi188_89276_object76640:head# 2024-04-23T16:25:16.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f31da87:::benchmark_data_smithi188_89276_object51006:head# 2024-04-23T16:25:16.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f32a27c:::benchmark_data_smithi188_89276_object67085:head# 2024-04-23T16:25:16.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f33dbc8:::benchmark_data_smithi188_89276_object70657:head# 2024-04-23T16:25:16.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f349ab4:::benchmark_data_smithi188_89276_object117269:head# 2024-04-23T16:25:16.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f351dfa:::benchmark_data_smithi188_89276_object103367:head# 2024-04-23T16:25:16.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f3564b8:::benchmark_data_smithi188_89276_object148373:head# 2024-04-23T16:25:16.224 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f357428:::benchmark_data_smithi188_89276_object139017:head# 2024-04-23T16:25:16.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f38de3b:::benchmark_data_smithi188_89276_object119624:head# 2024-04-23T16:25:16.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f392818:::benchmark_data_smithi188_89276_object67550:head# 2024-04-23T16:25:16.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f3b205d:::benchmark_data_smithi188_89276_object158849:head# 2024-04-23T16:25:16.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f3b4132:::benchmark_data_smithi188_89276_object163909:head# 2024-04-23T16:25:16.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f3c02b6:::benchmark_data_smithi188_89276_object74316:head# 2024-04-23T16:25:16.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f3c4bab:::benchmark_data_smithi188_89276_object64681:head# 2024-04-23T16:25:16.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f3c6519:::benchmark_data_smithi188_89276_object153877:head# 2024-04-23T16:25:16.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f3c843b:::benchmark_data_smithi188_89276_object167071:head# 2024-04-23T16:25:16.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f3caa08:::benchmark_data_smithi188_89276_object99205:head# 2024-04-23T16:25:16.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f3d98fc:::benchmark_data_smithi188_89276_object168312:head# 2024-04-23T16:25:16.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f3de9cb:::benchmark_data_smithi188_89276_object52072:head# 2024-04-23T16:25:16.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f3e5746:::benchmark_data_smithi188_89276_object101461:head# 2024-04-23T16:25:16.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f3fef96:::benchmark_data_smithi188_89276_object143072:head# 2024-04-23T16:25:16.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f402a25:::benchmark_data_smithi188_89276_object127152:head# 2024-04-23T16:25:16.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f406f75:::benchmark_data_smithi188_89276_object92502:head# 2024-04-23T16:25:16.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f40e0e8:::benchmark_data_smithi188_89276_object51401:head# 2024-04-23T16:25:16.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f40f8b5:::benchmark_data_smithi188_89276_object112012:head# 2024-04-23T16:25:16.231 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f422964:::benchmark_data_smithi188_89276_object115394:head# 2024-04-23T16:25:16.231 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f42b35f:::benchmark_data_smithi188_89276_object85570:head# 2024-04-23T16:25:16.231 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f43017c:::benchmark_data_smithi188_89276_object84707:head# 2024-04-23T16:25:16.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f434cc3:::benchmark_data_smithi188_89276_object82338:head# 2024-04-23T16:25:16.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f4488fb:::benchmark_data_smithi188_89276_object79356:head# 2024-04-23T16:25:16.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f44cd8f:::benchmark_data_smithi188_89276_object151199:head# 2024-04-23T16:25:16.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f450573:::benchmark_data_smithi188_89276_object159860:head# 2024-04-23T16:25:16.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f459573:::benchmark_data_smithi188_89276_object140874:head# 2024-04-23T16:25:16.234 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f462e1b:::benchmark_data_smithi188_89276_object41889:head# 2024-04-23T16:25:16.234 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:16 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:25:16.234 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:16 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:25:16.235 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f46b61f:::benchmark_data_smithi188_89276_object168199:head# 2024-04-23T16:25:16.235 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f472f1b:::benchmark_data_smithi188_89276_object65619:head# 2024-04-23T16:25:16.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f482791:::benchmark_data_smithi188_89276_object94470:head# 2024-04-23T16:25:16.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f48a5c0:::benchmark_data_smithi188_89276_object166397:head# 2024-04-23T16:25:16.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f48c7ee:::benchmark_data_smithi188_89276_object116405:head# 2024-04-23T16:25:16.237 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f49ed13:::benchmark_data_smithi188_89276_object155445:head# 2024-04-23T16:25:16.237 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f4b0257:::benchmark_data_smithi188_89276_object118329:head# 2024-04-23T16:25:16.240 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:16.241 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f4c02b2:::benchmark_data_smithi188_89276_object74179:head# 2024-04-23T16:25:16.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f4cb84a:::benchmark_data_smithi188_89276_object71516:head# 2024-04-23T16:25:16.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f4d1a88:::benchmark_data_smithi188_89276_object106876:head# 2024-04-23T16:25:16.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f4ddfd5:::benchmark_data_smithi188_89276_object100520:head# 2024-04-23T16:25:16.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f4eb129:::benchmark_data_smithi188_89276_object54847:head# 2024-04-23T16:25:16.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f4f9c27:::benchmark_data_smithi188_89276_object130514:head# 2024-04-23T16:25:16.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f50c20f:::benchmark_data_smithi188_89276_object126704:head# 2024-04-23T16:25:16.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f51f265:::benchmark_data_smithi188_89276_object52904:head# 2024-04-23T16:25:16.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f526135:::benchmark_data_smithi188_89276_object50401:head# 2024-04-23T16:25:16.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f52a406:::benchmark_data_smithi188_89276_object123998:head# 2024-04-23T16:25:16.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f52c09f:::benchmark_data_smithi188_89276_object41546:head# 2024-04-23T16:25:16.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f5303d8:::benchmark_data_smithi188_89276_object55877:head# 2024-04-23T16:25:16.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f534886:::benchmark_data_smithi188_89276_object120650:head# 2024-04-23T16:25:16.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f53bfc3:::benchmark_data_smithi188_89276_object140841:head# 2024-04-23T16:25:16.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f53c37f:::benchmark_data_smithi188_89276_object104480:head# 2024-04-23T16:25:16.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f541798:::benchmark_data_smithi188_89276_object73376:head# 2024-04-23T16:25:16.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f543082:::benchmark_data_smithi188_89276_object125846:head# 2024-04-23T16:25:16.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f54d573:::benchmark_data_smithi188_89276_object133666:head# 2024-04-23T16:25:16.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f555b6b:::benchmark_data_smithi188_89276_object71439:head# 2024-04-23T16:25:16.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f55ee0d:::benchmark_data_smithi188_89276_object150953:head# 2024-04-23T16:25:16.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f56f212:::benchmark_data_smithi188_89276_object167301:head# 2024-04-23T16:25:16.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f57083b:::benchmark_data_smithi188_89276_object61439:head# 2024-04-23T16:25:16.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f57e465:::benchmark_data_smithi188_89276_object69617:head# 2024-04-23T16:25:16.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f581a9d:::benchmark_data_smithi188_89276_object98114:head# 2024-04-23T16:25:16.248 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f5869a9:::benchmark_data_smithi188_89276_object161255:head# 2024-04-23T16:25:16.248 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f587012:::benchmark_data_smithi188_89276_object77006:head# 2024-04-23T16:25:16.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f589ea5:::benchmark_data_smithi188_89276_object46392:head# 2024-04-23T16:25:16.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f59b6b2:::benchmark_data_smithi188_89276_object65436:head# 2024-04-23T16:25:16.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f59f0fc:::benchmark_data_smithi188_89276_object120384:head# 2024-04-23T16:25:16.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f5a0df5:::benchmark_data_smithi188_89276_object111588:head# 2024-04-23T16:25:16.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f5a7ae2:::benchmark_data_smithi188_89276_object147449:head# 2024-04-23T16:25:16.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f5ac5f4:::benchmark_data_smithi188_89276_object53937:head# 2024-04-23T16:25:16.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f5af96b:::benchmark_data_smithi188_89276_object119454:head# 2024-04-23T16:25:16.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f5b0951:::benchmark_data_smithi188_89276_object99045:head# 2024-04-23T16:25:16.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f5b7557:::benchmark_data_smithi188_89276_object74350:head# 2024-04-23T16:25:16.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f5c2e44:::benchmark_data_smithi188_89276_object53509:head# 2024-04-23T16:25:16.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f5dd69e:::benchmark_data_smithi188_89276_object57015:head# 2024-04-23T16:25:16.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f5dd786:::benchmark_data_smithi188_89276_object160086:head# 2024-04-23T16:25:16.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f5e8b76:::benchmark_data_smithi188_89276_object55553:head# 2024-04-23T16:25:16.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f5eadc9:::benchmark_data_smithi188_89276_object89882:head# 2024-04-23T16:25:16.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f5ee045:::benchmark_data_smithi188_89276_object122511:head# 2024-04-23T16:25:16.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f5f25a6:::benchmark_data_smithi188_89276_object94012:head# 2024-04-23T16:25:16.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f5f608a:::benchmark_data_smithi188_89276_object75553:head# 2024-04-23T16:25:16.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f5fbf98:::benchmark_data_smithi188_89276_object85606:head# 2024-04-23T16:25:16.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f5fe25f:::benchmark_data_smithi188_89276_object100843:head# 2024-04-23T16:25:16.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f602e94:::benchmark_data_smithi188_89276_object99852:head# 2024-04-23T16:25:16.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f606347:::benchmark_data_smithi188_89276_object61171:head# 2024-04-23T16:25:16.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f606d97:::benchmark_data_smithi188_89276_object129553:head# 2024-04-23T16:25:16.258 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f60e119:::benchmark_data_smithi188_89276_object95409:head# 2024-04-23T16:25:16.258 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f612550:::benchmark_data_smithi188_89276_object101801:head# 2024-04-23T16:25:16.258 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f62fa36:::benchmark_data_smithi188_89276_object137645:head# 2024-04-23T16:25:16.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f633c0a:::benchmark_data_smithi188_89276_object131026:head# 2024-04-23T16:25:16.259 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:25:16.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f63957f:::benchmark_data_smithi188_89276_object91504:head# 2024-04-23T16:25:16.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f63a1f3:::benchmark_data_smithi188_89276_object168558:head# 2024-04-23T16:25:16.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f6500cf:::benchmark_data_smithi188_89276_object167573:head# 2024-04-23T16:25:16.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f650aad:::benchmark_data_smithi188_89276_object65163:head# 2024-04-23T16:25:16.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f65c580:::benchmark_data_smithi188_89276_object139244:head# 2024-04-23T16:25:16.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f660bf6:::benchmark_data_smithi188_89276_object109768:head# 2024-04-23T16:25:16.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:16.262 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:25:16.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f670989:::benchmark_data_smithi188_89276_object41667:head# 2024-04-23T16:25:16.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f6765c4:::benchmark_data_smithi188_89276_object113846:head# 2024-04-23T16:25:16.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f689fc7:::benchmark_data_smithi188_89276_object109856:head# 2024-04-23T16:25:16.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f68f058:::benchmark_data_smithi188_89276_object114346:head# 2024-04-23T16:25:16.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f697868:::benchmark_data_smithi188_89276_object145397:head# 2024-04-23T16:25:16.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f6a0aab:::benchmark_data_smithi188_89276_object43947:head# 2024-04-23T16:25:16.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f6a6e31:::benchmark_data_smithi188_89276_object118069:head# 2024-04-23T16:25:16.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f6b3e9b:::benchmark_data_smithi188_89276_object51791:head# 2024-04-23T16:25:16.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f6b578a:::benchmark_data_smithi188_89276_object48646:head# 2024-04-23T16:25:16.266 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f6bce1c:::benchmark_data_smithi188_89276_object133088:head# 2024-04-23T16:25:16.266 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f6bf4a0:::benchmark_data_smithi188_89276_object60839:head# 2024-04-23T16:25:16.267 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f6c30bf:::benchmark_data_smithi188_89276_object67665:head# 2024-04-23T16:25:16.267 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f6cebf0:::benchmark_data_smithi188_89276_object57206:head# 2024-04-23T16:25:16.268 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f6d56d5:::benchmark_data_smithi188_89276_object128066:head# 2024-04-23T16:25:16.268 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f6dbc05:::benchmark_data_smithi188_89276_object45526:head# 2024-04-23T16:25:16.269 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f6e61c9:::benchmark_data_smithi188_89276_object81111:head# 2024-04-23T16:25:16.269 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f6ee989:::benchmark_data_smithi188_89276_object163925:head# 2024-04-23T16:25:16.269 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f6f068c:::benchmark_data_smithi188_89276_object59786:head# 2024-04-23T16:25:16.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f6f0cda:::benchmark_data_smithi188_89276_object74639:head# 2024-04-23T16:25:16.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f70c000:::benchmark_data_smithi188_89276_object118474:head# 2024-04-23T16:25:16.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f70ce86:::benchmark_data_smithi188_89276_object58000:head# 2024-04-23T16:25:16.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f7115ec:::benchmark_data_smithi188_89276_object58206:head# 2024-04-23T16:25:16.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f718d77:::benchmark_data_smithi188_89276_object79329:head# 2024-04-23T16:25:16.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f71bbee:::benchmark_data_smithi188_89276_object76314:head# 2024-04-23T16:25:16.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f722381:::benchmark_data_smithi188_89276_object110859:head# 2024-04-23T16:25:16.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f725605:::benchmark_data_smithi188_89276_object44536:head# 2024-04-23T16:25:16.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f731b4b:::benchmark_data_smithi188_89276_object147415:head# 2024-04-23T16:25:16.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f732178:::benchmark_data_smithi188_89276_object86873:head# 2024-04-23T16:25:16.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f73660b:::benchmark_data_smithi188_89276_object41283:head# 2024-04-23T16:25:16.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f7369b2:::benchmark_data_smithi188_89276_object144731:head# 2024-04-23T16:25:16.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f73b8d3:::benchmark_data_smithi188_89276_object156165:head# 2024-04-23T16:25:16.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f742019:::benchmark_data_smithi188_89276_object118558:head# 2024-04-23T16:25:16.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f747b59:::benchmark_data_smithi188_89276_object124364:head# 2024-04-23T16:25:16.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f750324:::benchmark_data_smithi188_89276_object142473:head# 2024-04-23T16:25:16.277 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f75cb53:::benchmark_data_smithi188_89276_object150816:head# 2024-04-23T16:25:16.277 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f76011c:::benchmark_data_smithi188_89276_object133498:head# 2024-04-23T16:25:16.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f7665d1:::benchmark_data_smithi188_89276_object76534:head# 2024-04-23T16:25:16.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f773dd6:::benchmark_data_smithi188_89276_object75531:head# 2024-04-23T16:25:16.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f777fbd:::benchmark_data_smithi188_89276_object75192:head# 2024-04-23T16:25:16.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f781095:::benchmark_data_smithi188_89276_object107055:head# 2024-04-23T16:25:16.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f782069:::benchmark_data_smithi188_89276_object162860:head# 2024-04-23T16:25:16.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f7865f9:::benchmark_data_smithi188_89276_object56449:head# 2024-04-23T16:25:16.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f794e04:::benchmark_data_smithi188_89276_object95065:head# 2024-04-23T16:25:16.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f79621e:::benchmark_data_smithi188_89276_object72302:head# 2024-04-23T16:25:16.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f7a5ffd:::benchmark_data_smithi188_89276_object146121:head# 2024-04-23T16:25:16.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f7aac48:::benchmark_data_smithi188_89276_object140191:head# 2024-04-23T16:25:16.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f7b1707:::benchmark_data_smithi188_89276_object61166:head# 2024-04-23T16:25:16.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f7c0b43:::benchmark_data_smithi188_89276_object138901:head# 2024-04-23T16:25:16.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f7ca2a2:::benchmark_data_smithi188_89276_object133553:head# 2024-04-23T16:25:16.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f7e4368:::benchmark_data_smithi188_89276_object123691:head# 2024-04-23T16:25:16.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f7e828a:::benchmark_data_smithi188_89276_object140515:head# 2024-04-23T16:25:16.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f7e86cb:::benchmark_data_smithi188_89276_object62252:head# 2024-04-23T16:25:16.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f7fc557:::benchmark_data_smithi188_89276_object147014:head# 2024-04-23T16:25:16.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f806ff7:::benchmark_data_smithi188_89276_object119067:head# 2024-04-23T16:25:16.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f8162bc:::benchmark_data_smithi188_89276_object134853:head# 2024-04-23T16:25:16.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f81faa3:::benchmark_data_smithi188_89276_object135494:head# 2024-04-23T16:25:16.287 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f821044:::benchmark_data_smithi188_89276_object135700:head# 2024-04-23T16:25:16.287 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f8276d2:::benchmark_data_smithi188_89276_object148875:head# 2024-04-23T16:25:16.287 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f8282b7:::benchmark_data_smithi188_89276_object52806:head# 2024-04-23T16:25:16.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f830b37:::benchmark_data_smithi188_89276_object92484:head# 2024-04-23T16:25:16.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f84683e:::benchmark_data_smithi188_89276_object123837:head# 2024-04-23T16:25:16.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f84f95c:::benchmark_data_smithi188_89276_object152914:head# 2024-04-23T16:25:16.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f856f7a:::benchmark_data_smithi188_89276_object65613:head# 2024-04-23T16:25:16.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f861134:::benchmark_data_smithi188_89276_object69965:head# 2024-04-23T16:25:16.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f8687b4:::benchmark_data_smithi188_89276_object60536:head# 2024-04-23T16:25:16.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f868dc3:::benchmark_data_smithi188_89276_object69240:head# 2024-04-23T16:25:16.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f86a96d:::benchmark_data_smithi188_89276_object80866:head# 2024-04-23T16:25:16.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f86cbbd:::benchmark_data_smithi188_89276_object115828:head# 2024-04-23T16:25:16.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f86e21a:::benchmark_data_smithi188_89276_object133794:head# 2024-04-23T16:25:16.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f8701b1:::benchmark_data_smithi188_89276_object85537:head# 2024-04-23T16:25:16.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f8729ce:::benchmark_data_smithi188_89276_object96288:head# 2024-04-23T16:25:16.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f877fb9:::benchmark_data_smithi188_89276_object76302:head# 2024-04-23T16:25:16.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f88b128:::benchmark_data_smithi188_89276_object134339:head# 2024-04-23T16:25:16.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f88bbf0:::benchmark_data_smithi188_89276_object93861:head# 2024-04-23T16:25:16.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f89b51c:::benchmark_data_smithi188_89276_object106646:head# 2024-04-23T16:25:16.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f8b04d5:::benchmark_data_smithi188_89276_object98654:head# 2024-04-23T16:25:16.295 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f8c4ebd:::benchmark_data_smithi188_89276_object81187:head# 2024-04-23T16:25:16.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f8c9387:::benchmark_data_smithi188_89276_object143611:head# 2024-04-23T16:25:16.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f8c9e15:::benchmark_data_smithi188_89276_object94099:head# 2024-04-23T16:25:16.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f8cae1b:::benchmark_data_smithi188_89276_object121644:head# 2024-04-23T16:25:16.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f8cdeac:::benchmark_data_smithi188_89276_object56753:head# 2024-04-23T16:25:16.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f8e1426:::benchmark_data_smithi188_89276_object151177:head# 2024-04-23T16:25:16.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f8f53cc:::benchmark_data_smithi188_89276_object84488:head# 2024-04-23T16:25:16.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f8f7a7a:::benchmark_data_smithi188_89276_object81370:head# 2024-04-23T16:25:16.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f8f8dcc:::benchmark_data_smithi188_89276_object77406:head# 2024-04-23T16:25:16.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f8fa454:::benchmark_data_smithi188_89276_object80923:head# 2024-04-23T16:25:16.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f91016a:::benchmark_data_smithi188_89276_object67690:head# 2024-04-23T16:25:16.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f91aa32:::benchmark_data_smithi188_89276_object135207:head# 2024-04-23T16:25:16.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f91d64e:::benchmark_data_smithi188_89276_object92567:head# 2024-04-23T16:25:16.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f934ac2:::benchmark_data_smithi188_89276_object126344:head# 2024-04-23T16:25:16.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f93afbb:::benchmark_data_smithi188_89276_object73125:head# 2024-04-23T16:25:16.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f946cc9:::benchmark_data_smithi188_89276_object48853:head# 2024-04-23T16:25:16.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f9478b1:::benchmark_data_smithi188_89276_object63199:head# 2024-04-23T16:25:16.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f947db3:::benchmark_data_smithi188_89276_object165112:head# 2024-04-23T16:25:16.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f948962:::benchmark_data_smithi188_89276_object157922:head# 2024-04-23T16:25:16.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f953754:::benchmark_data_smithi188_89276_object90359:head# 2024-04-23T16:25:16.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f9580d1:::benchmark_data_smithi188_89276_object64187:head# 2024-04-23T16:25:16.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f95cdf7:::benchmark_data_smithi188_89276_object168258:head# 2024-04-23T16:25:16.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f963543:::benchmark_data_smithi188_89276_object118408:head# 2024-04-23T16:25:16.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f9770de:::benchmark_data_smithi188_89276_object158613:head# 2024-04-23T16:25:16.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f999c1f:::benchmark_data_smithi188_89276_object158410:head# 2024-04-23T16:25:16.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f99d3ee:::benchmark_data_smithi188_89276_object57805:head# 2024-04-23T16:25:16.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f9a5eaa:::benchmark_data_smithi188_89276_object87742:head# 2024-04-23T16:25:16.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f9bb958:::benchmark_data_smithi188_89276_object136315:head# 2024-04-23T16:25:16.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f9bf3cc:::benchmark_data_smithi188_89276_object104976:head# 2024-04-23T16:25:16.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f9c18d9:::benchmark_data_smithi188_89276_object45497:head# 2024-04-23T16:25:16.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f9d19e3:::benchmark_data_smithi188_89276_object164276:head# 2024-04-23T16:25:16.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f9d2ba2:::benchmark_data_smithi188_89276_object97120:head# 2024-04-23T16:25:16.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f9d2fe2:::benchmark_data_smithi188_89276_object139812:head# 2024-04-23T16:25:16.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f9da850:::benchmark_data_smithi188_89276_object114063:head# 2024-04-23T16:25:16.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f9e39bf:::benchmark_data_smithi188_89276_object112622:head# 2024-04-23T16:25:16.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f9e66c6:::benchmark_data_smithi188_89276_object105713:head# 2024-04-23T16:25:16.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f9f0508:::benchmark_data_smithi188_89276_object113429:head# 2024-04-23T16:25:16.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f9f76ee:::benchmark_data_smithi188_89276_object120996:head# 2024-04-23T16:25:16.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2f9f7d7b:::benchmark_data_smithi188_89276_object69082:head# 2024-04-23T16:25:16.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fa08571:::benchmark_data_smithi188_89276_object96126:head# 2024-04-23T16:25:16.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fa166cd:::benchmark_data_smithi188_89276_object153126:head# 2024-04-23T16:25:16.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fa1911b:::benchmark_data_smithi188_89276_object64630:head# 2024-04-23T16:25:16.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fa1a001:::benchmark_data_smithi188_89276_object65290:head# 2024-04-23T16:25:16.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fa1d922:::benchmark_data_smithi188_89276_object81785:head# 2024-04-23T16:25:16.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fa36e43:::benchmark_data_smithi188_89276_object62121:head# 2024-04-23T16:25:16.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fa40375:::benchmark_data_smithi188_89276_object146028:head# 2024-04-23T16:25:16.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fa425d6:::benchmark_data_smithi188_89276_object61449:head# 2024-04-23T16:25:16.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fa43230:::benchmark_data_smithi188_89276_object43997:head# 2024-04-23T16:25:16.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fa4fc5e:::benchmark_data_smithi188_89276_object125216:head# 2024-04-23T16:25:16.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fa56350:::benchmark_data_smithi188_89276_object63927:head# 2024-04-23T16:25:16.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fa5958a:::benchmark_data_smithi188_89276_object151424:head# 2024-04-23T16:25:16.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fa66c00:::benchmark_data_smithi188_89276_object62380:head# 2024-04-23T16:25:16.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fa6afa6:::benchmark_data_smithi188_89276_object123076:head# 2024-04-23T16:25:16.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fa7df79:::benchmark_data_smithi188_89276_object125574:head# 2024-04-23T16:25:16.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fa7f681:::benchmark_data_smithi188_89276_object63283:head# 2024-04-23T16:25:16.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fa99d90:::benchmark_data_smithi188_89276_object41959:head# 2024-04-23T16:25:16.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fa9fff8:::benchmark_data_smithi188_89276_object146900:head# 2024-04-23T16:25:16.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2faac681:::benchmark_data_smithi188_89276_object57586:head# 2024-04-23T16:25:16.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fabc75b:::benchmark_data_smithi188_89276_object105828:head# 2024-04-23T16:25:16.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fac2a66:::benchmark_data_smithi188_89276_object155972:head# 2024-04-23T16:25:16.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fac4a62:::benchmark_data_smithi188_89276_object54482:head# 2024-04-23T16:25:16.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2faca227:::benchmark_data_smithi188_89276_object121053:head# 2024-04-23T16:25:16.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2faca974:::benchmark_data_smithi188_89276_object135490:head# 2024-04-23T16:25:16.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fadd7f7:::benchmark_data_smithi188_89276_object154038:head# 2024-04-23T16:25:16.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2faddda7:::benchmark_data_smithi188_89276_object120599:head# 2024-04-23T16:25:16.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fae16aa:::benchmark_data_smithi188_89276_object119450:head# 2024-04-23T16:25:16.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fae672b:::benchmark_data_smithi188_89276_object85305:head# 2024-04-23T16:25:16.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fae8bdf:::benchmark_data_smithi188_89276_object127568:head# 2024-04-23T16:25:16.326 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2faeb6b6:::benchmark_data_smithi188_89276_object116484:head# 2024-04-23T16:25:16.326 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2faed123:::benchmark_data_smithi188_89276_object142832:head# 2024-04-23T16:25:16.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fb0d9b8:::benchmark_data_smithi188_89276_object90737:head# 2024-04-23T16:25:16.329 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fb12903:::benchmark_data_smithi188_89276_object160474:head# 2024-04-23T16:25:16.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fb4ce44:::benchmark_data_smithi188_89276_object64564:head# 2024-04-23T16:25:16.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fb4da0d:::benchmark_data_smithi188_89276_object63001:head# 2024-04-23T16:25:16.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fb59e71:::benchmark_data_smithi188_89276_object100302:head# 2024-04-23T16:25:16.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fb5c6b3:::benchmark_data_smithi188_89276_object82013:head# 2024-04-23T16:25:16.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fb60abc:::benchmark_data_smithi188_89276_object99087:head# 2024-04-23T16:25:16.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fb6c854:::benchmark_data_smithi188_89276_object119266:head# 2024-04-23T16:25:16.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fb790c7:::benchmark_data_smithi188_89276_object164828:head# 2024-04-23T16:25:16.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fb79cdb:::benchmark_data_smithi188_89276_object165065:head# 2024-04-23T16:25:16.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fb815c1:::benchmark_data_smithi188_89276_object74327:head# 2024-04-23T16:25:16.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fb834ca:::benchmark_data_smithi188_89276_object49818:head# 2024-04-23T16:25:16.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fb83524:::benchmark_data_smithi188_89276_object102146:head# 2024-04-23T16:25:16.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fb91c15:::benchmark_data_smithi188_89276_object152598:head# 2024-04-23T16:25:16.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fba0191:::benchmark_data_smithi188_89276_object41644:head# 2024-04-23T16:25:16.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fbacd08:::benchmark_data_smithi188_89276_object98565:head# 2024-04-23T16:25:16.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fbc3d8a:::benchmark_data_smithi188_89276_object143873:head# 2024-04-23T16:25:16.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fbc57fc:::benchmark_data_smithi188_89276_object44258:head# 2024-04-23T16:25:16.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fbc597e:::benchmark_data_smithi188_89276_object40707:head# 2024-04-23T16:25:16.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fbc92a4:::benchmark_data_smithi188_89276_object132736:head# 2024-04-23T16:25:16.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fbd70df:::benchmark_data_smithi188_89276_object163099:head# 2024-04-23T16:25:16.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fbdc3de:::benchmark_data_smithi188_89276_object164322:head# 2024-04-23T16:25:16.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fbe6ee2:::benchmark_data_smithi188_89276_object69242:head# 2024-04-23T16:25:16.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fbea400:::benchmark_data_smithi188_89276_object86488:head# 2024-04-23T16:25:16.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fbee1f8:::benchmark_data_smithi188_89276_object139749:head# 2024-04-23T16:25:16.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fbee8bc:::benchmark_data_smithi188_89276_object149011:head# 2024-04-23T16:25:16.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fbf154d:::benchmark_data_smithi188_89276_object88417:head# 2024-04-23T16:25:16.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fbfa652:::benchmark_data_smithi188_89276_object99954:head# 2024-04-23T16:25:16.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fbfc426:::benchmark_data_smithi188_89276_object55353:head# 2024-04-23T16:25:16.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc06dbf:::benchmark_data_smithi188_89276_object153478:head# 2024-04-23T16:25:16.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc07a6b:::benchmark_data_smithi188_89276_object80475:head# 2024-04-23T16:25:16.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc0cdf2:::benchmark_data_smithi188_89276_object159450:head# 2024-04-23T16:25:16.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc13a86:::benchmark_data_smithi188_89276_object136349:head# 2024-04-23T16:25:16.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc1804d:::benchmark_data_smithi188_89276_object98452:head# 2024-04-23T16:25:16.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc1bbb3:::benchmark_data_smithi188_89276_object104198:head# 2024-04-23T16:25:16.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc25f93:::benchmark_data_smithi188_89276_object134365:head# 2024-04-23T16:25:16.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc2cf61:::benchmark_data_smithi188_89276_object134800:head# 2024-04-23T16:25:16.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc32ec3:::benchmark_data_smithi188_89276_object150047:head# 2024-04-23T16:25:16.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc35348:::benchmark_data_smithi188_89276_object116910:head# 2024-04-23T16:25:16.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc36cf1:::benchmark_data_smithi188_89276_object124952:head# 2024-04-23T16:25:16.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc3bde7:::benchmark_data_smithi188_89276_object129055:head# 2024-04-23T16:25:16.361 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc4098a:::benchmark_data_smithi188_89276_object98883:head# 2024-04-23T16:25:16.361 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc4133b:::benchmark_data_smithi188_89276_object51454:head# 2024-04-23T16:25:16.362 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc46018:::benchmark_data_smithi188_89276_object74071:head# 2024-04-23T16:25:16.362 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc47da3:::benchmark_data_smithi188_89276_object103503:head# 2024-04-23T16:25:16.363 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:25:16.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc50b5b:::benchmark_data_smithi188_89276_object70287:head# 2024-04-23T16:25:16.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc560d8:::benchmark_data_smithi188_89276_object153206:head# 2024-04-23T16:25:16.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc64269:::benchmark_data_smithi188_89276_object106000:head# 2024-04-23T16:25:16.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc6626d:::benchmark_data_smithi188_89276_object147605:head# 2024-04-23T16:25:16.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc7267e:::benchmark_data_smithi188_89276_object45027:head# 2024-04-23T16:25:16.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc7304e:::benchmark_data_smithi188_89276_object95238:head# 2024-04-23T16:25:16.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc73acb:::benchmark_data_smithi188_89276_object81450:head# 2024-04-23T16:25:16.371 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc7b0de:::benchmark_data_smithi188_89276_object120675:head# 2024-04-23T16:25:16.372 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc84b39:::benchmark_data_smithi188_89276_object136439:head# 2024-04-23T16:25:16.373 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc8b6b6:::benchmark_data_smithi188_89276_object148534:head# 2024-04-23T16:25:16.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc95510:::benchmark_data_smithi188_89276_object50204:head# 2024-04-23T16:25:16.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fc97c88:::benchmark_data_smithi188_89276_object152984:head# 2024-04-23T16:25:16.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fcac7db:::benchmark_data_smithi188_89276_object59038:head# 2024-04-23T16:25:16.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fcb1299:::benchmark_data_smithi188_89276_object92611:head# 2024-04-23T16:25:16.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fcbe9ee:::benchmark_data_smithi188_89276_object111747:head# 2024-04-23T16:25:16.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fcbf212:::benchmark_data_smithi188_89276_object158242:head# 2024-04-23T16:25:16.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fcc2758:::benchmark_data_smithi188_89276_object113651:head# 2024-04-23T16:25:16.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fccff71:::benchmark_data_smithi188_89276_object127443:head# 2024-04-23T16:25:16.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fcdf4b5:::benchmark_data_smithi188_89276_object76818:head# 2024-04-23T16:25:16.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fce3fb2:::benchmark_data_smithi188_89276_object97316:head# 2024-04-23T16:25:16.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fce9e6b:::benchmark_data_smithi188_89276_object132070:head# 2024-04-23T16:25:16.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fcea5ba:::benchmark_data_smithi188_89276_object109385:head# 2024-04-23T16:25:16.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fcf3cb4:::benchmark_data_smithi188_89276_object134809:head# 2024-04-23T16:25:16.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fcfca87:::benchmark_data_smithi188_89276_object108750:head# 2024-04-23T16:25:16.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fd0f66b:::benchmark_data_smithi188_89276_object83657:head# 2024-04-23T16:25:16.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fd25a7d:::benchmark_data_smithi188_89276_object98634:head# 2024-04-23T16:25:16.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fd2f380:::benchmark_data_smithi188_89276_object53832:head# 2024-04-23T16:25:16.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fd44afb:::benchmark_data_smithi188_89276_object49852:head# 2024-04-23T16:25:16.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fd46ef5:::benchmark_data_smithi188_89276_object124014:head# 2024-04-23T16:25:16.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fd4f648:::benchmark_data_smithi188_89276_object92909:head# 2024-04-23T16:25:16.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fd5eb3e:::benchmark_data_smithi188_89276_object75345:head# 2024-04-23T16:25:16.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fd6c8f6:::benchmark_data_smithi188_89276_object80724:head# 2024-04-23T16:25:16.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fd76c6c:::benchmark_data_smithi188_89276_object127345:head# 2024-04-23T16:25:16.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fd7e334:::benchmark_data_smithi188_89276_object117176:head# 2024-04-23T16:25:16.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fd7f846:::benchmark_data_smithi188_89276_object124365:head# 2024-04-23T16:25:16.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fd81cf9:::benchmark_data_smithi188_89276_object162747:head# 2024-04-23T16:25:16.387 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fd951fd:::benchmark_data_smithi188_89276_object92846:head# 2024-04-23T16:25:16.387 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fd9ac00:::benchmark_data_smithi188_89276_object165817:head# 2024-04-23T16:25:16.387 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fda2886:::benchmark_data_smithi188_89276_object54546:head# 2024-04-23T16:25:16.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fda8348:::benchmark_data_smithi188_89276_object78769:head# 2024-04-23T16:25:16.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fdb6291:::benchmark_data_smithi188_89276_object150582:head# 2024-04-23T16:25:16.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:16 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:25:16.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:16 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:25:16.389 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fdd7de9:::benchmark_data_smithi188_89276_object121055:head# 2024-04-23T16:25:16.389 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fdeadef:::benchmark_data_smithi188_89276_object164002:head# 2024-04-23T16:25:16.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fdf59d5:::benchmark_data_smithi188_89276_object163258:head# 2024-04-23T16:25:16.391 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:16.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fdfff7d:::benchmark_data_smithi188_89276_object92951:head# 2024-04-23T16:25:16.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe00103:::benchmark_data_smithi188_89276_object96153:head# 2024-04-23T16:25:16.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe03c44:::benchmark_data_smithi188_89276_object129167:head# 2024-04-23T16:25:16.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe04393:::benchmark_data_smithi188_89276_object127753:head# 2024-04-23T16:25:16.395 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe0920e:::benchmark_data_smithi188_89276_object75369:head# 2024-04-23T16:25:16.395 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe142c7:::benchmark_data_smithi188_89276_object83286:head# 2024-04-23T16:25:16.395 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe2128e:::benchmark_data_smithi188_89276_object44136:head# 2024-04-23T16:25:16.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe349f1:::benchmark_data_smithi188_89276_object139268:head# 2024-04-23T16:25:16.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe47225:::benchmark_data_smithi188_89276_object151040:head# 2024-04-23T16:25:16.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe49525:::benchmark_data_smithi188_89276_object150195:head# 2024-04-23T16:25:16.397 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe50322:::benchmark_data_smithi188_89276_object65442:head# 2024-04-23T16:25:16.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe52778:::benchmark_data_smithi188_89276_object74670:head# 2024-04-23T16:25:16.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe54e85:::benchmark_data_smithi188_89276_object129903:head# 2024-04-23T16:25:16.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe5976d:::benchmark_data_smithi188_89276_object44404:head# 2024-04-23T16:25:16.399 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe6e90c:::benchmark_data_smithi188_89276_object51170:head# 2024-04-23T16:25:16.399 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe72412:::benchmark_data_smithi188_89276_object102220:head# 2024-04-23T16:25:16.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe84634:::benchmark_data_smithi188_89276_object61648:head# 2024-04-23T16:25:16.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe84eac:::benchmark_data_smithi188_89276_object53749:head# 2024-04-23T16:25:16.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe8ae61:::benchmark_data_smithi188_89276_object47835:head# 2024-04-23T16:25:16.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe8b85a:::benchmark_data_smithi188_89276_object135726:head# 2024-04-23T16:25:16.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe904bd:::benchmark_data_smithi188_89276_object144515:head# 2024-04-23T16:25:16.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe973a3:::benchmark_data_smithi188_89276_object96037:head# 2024-04-23T16:25:16.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fe9d5c1:::benchmark_data_smithi188_89276_object63468:head# 2024-04-23T16:25:16.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fea3951:::benchmark_data_smithi188_89276_object83663:head# 2024-04-23T16:25:16.403 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2feb67d6:::benchmark_data_smithi188_89276_object116309:head# 2024-04-23T16:25:16.403 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fec4b4c:::benchmark_data_smithi188_89276_object68575:head# 2024-04-23T16:25:16.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2feca18a:::benchmark_data_smithi188_89276_object58002:head# 2024-04-23T16:25:16.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fecb237:::benchmark_data_smithi188_89276_object107808:head# 2024-04-23T16:25:16.405 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fed1866:::benchmark_data_smithi188_89276_object51133:head# 2024-04-23T16:25:16.405 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2feda75a:::benchmark_data_smithi188_89276_object104233:head# 2024-04-23T16:25:16.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fedadcb:::benchmark_data_smithi188_89276_object144295:head# 2024-04-23T16:25:16.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fef286e:::benchmark_data_smithi188_89276_object65671:head# 2024-04-23T16:25:16.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fef3495:::benchmark_data_smithi188_89276_object41092:head# 2024-04-23T16:25:16.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2fef9e96:::benchmark_data_smithi188_89276_object155762:head# 2024-04-23T16:25:16.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff01528:::benchmark_data_smithi188_89276_object59142:head# 2024-04-23T16:25:16.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff03377:::benchmark_data_smithi188_89276_object50481:head# 2024-04-23T16:25:16.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff0718f:::benchmark_data_smithi188_89276_object54272:head# 2024-04-23T16:25:16.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff0d3ff:::benchmark_data_smithi188_89276_object158216:head# 2024-04-23T16:25:16.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff15bc2:::benchmark_data_smithi188_89276_object59700:head# 2024-04-23T16:25:16.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff18b9e:::benchmark_data_smithi188_89276_object43030:head# 2024-04-23T16:25:16.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff20e6f:::benchmark_data_smithi188_89276_object150724:head# 2024-04-23T16:25:16.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff3613a:::benchmark_data_smithi188_89276_object161095:head# 2024-04-23T16:25:16.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff38772:::benchmark_data_smithi188_89276_object156242:head# 2024-04-23T16:25:16.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff3a247:::benchmark_data_smithi188_89276_object48635:head# 2024-04-23T16:25:16.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff428dc:::benchmark_data_smithi188_89276_object92992:head# 2024-04-23T16:25:16.412 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff464f1:::benchmark_data_smithi188_89276_object84835:head# 2024-04-23T16:25:16.412 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff4ab04:::benchmark_data_smithi188_89276_object126930:head# 2024-04-23T16:25:16.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff4eeee:::benchmark_data_smithi188_89276_object111958:head# 2024-04-23T16:25:16.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff569e4:::benchmark_data_smithi188_89276_object41760:head# 2024-04-23T16:25:16.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff588b2:::benchmark_data_smithi188_89276_object140900:head# 2024-04-23T16:25:16.414 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff650cd:::benchmark_data_smithi188_89276_object85446:head# 2024-04-23T16:25:16.414 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff690b7:::benchmark_data_smithi188_89276_object111499:head# 2024-04-23T16:25:16.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff6d178:::benchmark_data_smithi188_89276_object101587:head# 2024-04-23T16:25:16.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff784b9:::benchmark_data_smithi188_89276_object140989:head# 2024-04-23T16:25:16.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff7f3b1:::benchmark_data_smithi188_89276_object73405:head# 2024-04-23T16:25:16.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff891de:::benchmark_data_smithi188_89276_object157415:head# 2024-04-23T16:25:16.416 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:25:16.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff8dbcb:::benchmark_data_smithi188_89276_object125864:head# 2024-04-23T16:25:16.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff9820b:::benchmark_data_smithi188_89276_object112537:head# 2024-04-23T16:25:16.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff990ba:::benchmark_data_smithi188_89276_object98984:head# 2024-04-23T16:25:16.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ff9bf77:::benchmark_data_smithi188_89276_object46905:head# 2024-04-23T16:25:16.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ffa97ed:::benchmark_data_smithi188_89276_object65102:head# 2024-04-23T16:25:16.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ffb018c:::benchmark_data_smithi188_89276_object165222:head# 2024-04-23T16:25:16.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:16.420 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:25:16.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ffb15a3:::benchmark_data_smithi188_89276_object64661:head# 2024-04-23T16:25:16.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ffb60c5:::benchmark_data_smithi188_89276_object117699:head# 2024-04-23T16:25:16.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ffc0965:::benchmark_data_smithi188_89276_object90825:head# 2024-04-23T16:25:16.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ffc5dd8:::benchmark_data_smithi188_89276_object121015:head# 2024-04-23T16:25:16.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ffc670f:::benchmark_data_smithi188_89276_object156604:head# 2024-04-23T16:25:16.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ffe0625:::benchmark_data_smithi188_89276_object156855:head# 2024-04-23T16:25:16.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ffe9e81:::benchmark_data_smithi188_89276_object81891:head# 2024-04-23T16:25:16.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #3:2ffee255:::benchmark_data_smithi188_89276_object92386:head# 2024-04-23T16:25:16.430 INFO:teuthology.orchestra.run.smithi103.stderr:Export successful 2024-04-23T16:25:16.520 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:25:16.574 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:25:16.574 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:25:16.586 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:16.604 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:25:16.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:16 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:25:16.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:16 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:25:16.628 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:25:16.655 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:25:16.683 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:25:16.711 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:25:16.717 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:25:16.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:16.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:25:16.728 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:25:16.750 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:25:16.788 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:25:16.821 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:25:16.829 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:25:16.852 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:25:16.853 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:25:16.853 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T16:25:16.853 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:25:16.853 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:25:16.853 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:25:16.853 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:25:16.853 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:25:16.853 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:25:16.853 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:25:16.853 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:25:16.854 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:25:16.854 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:25:16.856 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:16.879 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:25:16.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:16.883 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:25:16.983 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:25:17.021 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:17.049 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:25:17.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:17.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:25:17.151 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:25:17.157 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:25:17.184 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:17.208 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:25:17.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:17.216 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:25:17.317 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:25:17.359 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:17.454 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:17 smithi148 ceph-mon[44520]: osdmap e247: 12 total, 11 up, 10 in 2024-04-23T16:25:17.454 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:17 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3055038125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:25:17.454 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:17.454 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:17 smithi148 ceph-mon[44520]: pgmap v718: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 752 op/s; 15 MiB/s, 237 objects/s recovering 2024-04-23T16:25:17.454 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:17.454 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:17.454 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:17.454 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:17.454 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:17.454 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:17 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/660760928' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:25:17.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:17 smithi188 ceph-mon[44101]: osdmap e247: 12 total, 11 up, 10 in 2024-04-23T16:25:17.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:17 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3055038125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:25:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:17 smithi188 ceph-mon[44101]: pgmap v718: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 752 op/s; 15 MiB/s, 237 objects/s recovering 2024-04-23T16:25:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:17 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/660760928' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:25:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:17 smithi103 ceph-mon[39629]: osdmap e247: 12 total, 11 up, 10 in 2024-04-23T16:25:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:17 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3055038125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:25:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:17 smithi103 ceph-mon[39629]: pgmap v718: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 752 op/s; 15 MiB/s, 237 objects/s recovering 2024-04-23T16:25:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:17.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:17.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:17 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/660760928' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:25:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:18.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:18.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:25:18.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/660760928' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:25:18.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: osdmap e248: 12 total, 11 up, 10 in 2024-04-23T16:25:18.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=3888107154,config=2679774077,health=179246142,kv=757428853}) 2024-04-23T16:25:18.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {kv=6,logm=94} crc {kv=3900185971,logm=765008258}) 2024-04-23T16:25:18.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1697731184}) 2024-04-23T16:25:18.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1640573080}) 2024-04-23T16:25:18.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1894658491}) 2024-04-23T16:25:18.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3515976580}) 2024-04-23T16:25:18.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3718378316}) 2024-04-23T16:25:18.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=163409682}) 2024-04-23T16:25:18.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3031676889}) 2024-04-23T16:25:18.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131987484}) 2024-04-23T16:25:18.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339945755}) 2024-04-23T16:25:18.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2548970789}) 2024-04-23T16:25:18.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3751315787}) 2024-04-23T16:25:18.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2171580365}) 2024-04-23T16:25:18.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4110102592}) 2024-04-23T16:25:18.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3575208581}) 2024-04-23T16:25:18.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2696675427}) 2024-04-23T16:25:18.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4087077356}) 2024-04-23T16:25:18.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1708629641}) 2024-04-23T16:25:18.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4277549604}) 2024-04-23T16:25:18.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3894017690}) 2024-04-23T16:25:18.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1964382640}) 2024-04-23T16:25:18.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3604205921}) 2024-04-23T16:25:18.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=424419643}) 2024-04-23T16:25:18.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2050266225}) 2024-04-23T16:25:18.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4110267170}) 2024-04-23T16:25:18.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2083218364}) 2024-04-23T16:25:18.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1088742049}) 2024-04-23T16:25:18.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2976777457}) 2024-04-23T16:25:18.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3333445736}) 2024-04-23T16:25:18.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3096402485}) 2024-04-23T16:25:18.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3989572212}) 2024-04-23T16:25:18.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231338298}) 2024-04-23T16:25:18.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=959351190}) 2024-04-23T16:25:18.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2132669668}) 2024-04-23T16:25:18.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=80,mdsmap=3,mgr=17} crc {logm=2784554569,mdsmap=2840952586,mgr=4139503336}) 2024-04-23T16:25:18.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=85} crc {mgr=1987707978,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=1813455958,mon_config_key=2940537053}) 2024-04-23T16:25:18.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4149138437}) 2024-04-23T16:25:18.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1969955899}) 2024-04-23T16:25:18.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=2,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=80} crc {mon_config_key=786301893,monmap=2133572193,osd_metadata=1900671984,osd_pg_creating=1663572640,osdmap=2452254852}) 2024-04-23T16:25:18.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=35} crc {osdmap=2944503689}) 2024-04-23T16:25:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:18.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:18.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:18.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:25:18.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/660760928' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:25:18.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: osdmap e248: 12 total, 11 up, 10 in 2024-04-23T16:25:18.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=3888107154,config=2679774077,health=179246142,kv=757428853}) 2024-04-23T16:25:18.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {kv=6,logm=94} crc {kv=3900185971,logm=765008258}) 2024-04-23T16:25:18.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1697731184}) 2024-04-23T16:25:18.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1640573080}) 2024-04-23T16:25:18.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1894658491}) 2024-04-23T16:25:18.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3515976580}) 2024-04-23T16:25:18.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3718378316}) 2024-04-23T16:25:18.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=163409682}) 2024-04-23T16:25:18.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3031676889}) 2024-04-23T16:25:18.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131987484}) 2024-04-23T16:25:18.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339945755}) 2024-04-23T16:25:18.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2548970789}) 2024-04-23T16:25:18.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3751315787}) 2024-04-23T16:25:18.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2171580365}) 2024-04-23T16:25:18.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4110102592}) 2024-04-23T16:25:18.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3575208581}) 2024-04-23T16:25:18.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2696675427}) 2024-04-23T16:25:18.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4087077356}) 2024-04-23T16:25:18.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1708629641}) 2024-04-23T16:25:18.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4277549604}) 2024-04-23T16:25:18.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3894017690}) 2024-04-23T16:25:18.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1964382640}) 2024-04-23T16:25:18.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3604205921}) 2024-04-23T16:25:18.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=424419643}) 2024-04-23T16:25:18.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2050266225}) 2024-04-23T16:25:18.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4110267170}) 2024-04-23T16:25:18.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2083218364}) 2024-04-23T16:25:18.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1088742049}) 2024-04-23T16:25:18.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2976777457}) 2024-04-23T16:25:18.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3333445736}) 2024-04-23T16:25:18.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3096402485}) 2024-04-23T16:25:18.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3989572212}) 2024-04-23T16:25:18.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231338298}) 2024-04-23T16:25:18.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=959351190}) 2024-04-23T16:25:18.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2132669668}) 2024-04-23T16:25:18.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=80,mdsmap=3,mgr=17} crc {logm=2784554569,mdsmap=2840952586,mgr=4139503336}) 2024-04-23T16:25:18.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=85} crc {mgr=1987707978,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=1813455958,mon_config_key=2940537053}) 2024-04-23T16:25:18.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4149138437}) 2024-04-23T16:25:18.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1969955899}) 2024-04-23T16:25:18.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=2,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=80} crc {mon_config_key=786301893,monmap=2133572193,osd_metadata=1900671984,osd_pg_creating=1663572640,osdmap=2452254852}) 2024-04-23T16:25:18.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=35} crc {osdmap=2944503689}) 2024-04-23T16:25:18.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:25:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/660760928' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:25:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: osdmap e248: 12 total, 11 up, 10 in 2024-04-23T16:25:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=3888107154,config=2679774077,health=179246142,kv=757428853}) 2024-04-23T16:25:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {kv=6,logm=94} crc {kv=3900185971,logm=765008258}) 2024-04-23T16:25:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1697731184}) 2024-04-23T16:25:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1640573080}) 2024-04-23T16:25:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1894658491}) 2024-04-23T16:25:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3515976580}) 2024-04-23T16:25:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3718378316}) 2024-04-23T16:25:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=163409682}) 2024-04-23T16:25:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3031676889}) 2024-04-23T16:25:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131987484}) 2024-04-23T16:25:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339945755}) 2024-04-23T16:25:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2548970789}) 2024-04-23T16:25:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3751315787}) 2024-04-23T16:25:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2171580365}) 2024-04-23T16:25:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4110102592}) 2024-04-23T16:25:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3575208581}) 2024-04-23T16:25:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2696675427}) 2024-04-23T16:25:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4087077356}) 2024-04-23T16:25:18.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1708629641}) 2024-04-23T16:25:18.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4277549604}) 2024-04-23T16:25:18.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3894017690}) 2024-04-23T16:25:18.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1964382640}) 2024-04-23T16:25:18.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3604205921}) 2024-04-23T16:25:18.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=424419643}) 2024-04-23T16:25:18.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2050266225}) 2024-04-23T16:25:18.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4110267170}) 2024-04-23T16:25:18.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2083218364}) 2024-04-23T16:25:18.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1088742049}) 2024-04-23T16:25:18.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2976777457}) 2024-04-23T16:25:18.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3333445736}) 2024-04-23T16:25:18.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3096402485}) 2024-04-23T16:25:18.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3989572212}) 2024-04-23T16:25:18.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231338298}) 2024-04-23T16:25:18.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=959351190}) 2024-04-23T16:25:18.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2132669668}) 2024-04-23T16:25:18.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=80,mdsmap=3,mgr=17} crc {logm=2784554569,mdsmap=2840952586,mgr=4139503336}) 2024-04-23T16:25:18.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=85} crc {mgr=1987707978,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=1813455958,mon_config_key=2940537053}) 2024-04-23T16:25:18.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4149138437}) 2024-04-23T16:25:18.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1969955899}) 2024-04-23T16:25:18.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=2,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=80} crc {mon_config_key=786301893,monmap=2133572193,osd_metadata=1900671984,osd_pg_creating=1663572640,osdmap=2452254852}) 2024-04-23T16:25:18.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=35} crc {osdmap=2944503689}) 2024-04-23T16:25:19.433 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:19 smithi103 ceph-mon[39629]: pgmap v720: 17 pgs: 17 active+clean; 7.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 883 op/s; 2.0 MiB/s, 31 objects/s recovering 2024-04-23T16:25:19.434 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:25:19.434 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:25:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:19 smithi188 ceph-mon[44101]: pgmap v720: 17 pgs: 17 active+clean; 7.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 883 op/s; 2.0 MiB/s, 31 objects/s recovering 2024-04-23T16:25:19.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:19 smithi148 ceph-mon[44520]: pgmap v720: 17 pgs: 17 active+clean; 7.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 883 op/s; 2.0 MiB/s, 31 objects/s recovering 2024-04-23T16:25:19.878 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.1 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --force --op remove --pgid 3.4 2024-04-23T16:25:20.107 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:25:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:20 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:25:20.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:20 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:25:20.497 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:20 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:25:20.497 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:20 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:25:20.524 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:25:20.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:20.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:25:20.582 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.1/config 2024-04-23T16:25:20.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:20 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:25:20.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:20 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:25:20.630 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:25:20.673 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:20.695 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:25:20.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:20.701 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:25:20.802 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:25:20.845 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:20.872 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:25:20.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:20.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:25:20.979 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:25:21.016 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:21.042 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:25:21.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:21.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:25:21.149 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:25:21.188 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:21.213 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:25:21.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:21.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:25:21.240 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:25:21.240 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:25:21.317 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:25:21.361 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:21.389 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:25:21.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:21.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:25:21.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:25:21.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:21 smithi103 ceph-mon[39629]: osdmap e249: 12 total, 11 up, 10 in 2024-04-23T16:25:21.464 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:21 smithi103 ceph-mon[39629]: pgmap v722: 17 pgs: 17 active+clean; 7.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 930 op/s 2024-04-23T16:25:21.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:21 smithi188 ceph-mon[44101]: osdmap e249: 12 total, 11 up, 10 in 2024-04-23T16:25:21.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:21 smithi188 ceph-mon[44101]: pgmap v722: 17 pgs: 17 active+clean; 7.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 930 op/s 2024-04-23T16:25:21.496 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:25:21.538 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:21 smithi148 ceph-mon[44520]: osdmap e249: 12 total, 11 up, 10 in 2024-04-23T16:25:21.539 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:21 smithi148 ceph-mon[44520]: pgmap v722: 17 pgs: 17 active+clean; 7.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 930 op/s 2024-04-23T16:25:21.541 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:21.563 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:25:21.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:21.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:25:21.671 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:25:21.700 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:21.724 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:25:21.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:21.727 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:25:21.828 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:25:21.866 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:22.191 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:22 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:25:22.191 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:22 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:25:22.233 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:25:22.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:22.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:25:22.337 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:25:22.378 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:22.398 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:25:22.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:22.405 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:25:22.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:22 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:25:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:22 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:25:22.506 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:25:22.536 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:22.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:22 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:25:22.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:22 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:25:23.094 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:25:23.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:23.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:25:23.098 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-23T16:25:23.199 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:23.257 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:23.288 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:23.292 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:25:23.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:23.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:23.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:23 smithi103 ceph-mon[39629]: osdmap e250: 12 total, 11 up, 10 in 2024-04-23T16:25:23.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:23 smithi103 ceph-mon[39629]: pgmap v724: 17 pgs: 17 active+clean; 7.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 964 op/s 2024-04-23T16:25:23.379 INFO:teuthology.orchestra.run.smithi103.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:25:23.399 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:25:23.441 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:23.472 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:25:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:23 smithi188 ceph-mon[44101]: osdmap e250: 12 total, 11 up, 10 in 2024-04-23T16:25:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:23 smithi188 ceph-mon[44101]: pgmap v724: 17 pgs: 17 active+clean; 7.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 964 op/s 2024-04-23T16:25:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:23.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:25:23.484 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:23 smithi148 ceph-mon[44520]: osdmap e250: 12 total, 11 up, 10 in 2024-04-23T16:25:23.484 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:23 smithi148 ceph-mon[44520]: pgmap v724: 17 pgs: 17 active+clean; 7.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 964 op/s 2024-04-23T16:25:23.559 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:25:23.582 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:25:23.634 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:23.853 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:25:23.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:23.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:25:23.958 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:25:23.993 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:24.015 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:25:24.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:24.021 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:25:24.122 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:25:24.152 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:24.179 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:25:24.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:24.190 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:25:24.291 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:25:24.334 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:24.360 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:25:24.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:24.362 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:25:24.463 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:25:24.497 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:24.532 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:25:24.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:24.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:25:24.638 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:24.673 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:24.711 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:24.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:24.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:24.819 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:25:24.849 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:24.873 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:25:24.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:24.876 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:25:24.977 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:25:25.013 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:25.043 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:25:25.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:25.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:25:25.150 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:25:25.184 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:24 smithi188 ceph-mon[44101]: pgmap v725: 17 pgs: 17 active+clean; 6.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-23T16:25:25.186 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:25.213 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:25:25.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:25.216 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:25:25.317 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:25:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:24 smithi103 ceph-mon[39629]: pgmap v725: 17 pgs: 17 active+clean; 6.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-23T16:25:25.354 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:24 smithi148 ceph-mon[44520]: pgmap v725: 17 pgs: 17 active+clean; 6.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-23T16:25:25.356 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:25.381 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:25:25.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:25.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:25:25.494 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:25:25.534 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:25.564 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:25:25.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:25.568 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:25:25.668 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:25:25.715 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:25.762 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:25:25.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:25.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:25:25.866 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:25:25.915 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:25.959 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:25:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:25.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:25:26.067 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:25:26.107 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:26.135 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:25:26.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:26.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:25:26.240 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:26.295 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:26.433 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:26.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:26.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:26.548 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:25:26.589 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:26.623 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:25:26.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:26.627 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:25:26.728 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:25:26.767 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:26.800 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:25:26.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:26.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:25:26.908 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:25:26.916 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:25:26.962 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:27.107 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:25:27.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:27.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:25:27.213 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:25:27.254 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:27.833 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:25:27.833 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:25:27.833 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:25:27.833 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:25:27.833 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:25:27.833 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:25:27.833 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:25:27.863 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:25:27.894 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:25:27.914 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:25:27.949 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:25:27.949 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:25:27.950 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T16:25:27.950 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:25:27.950 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:25:27.950 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:25:27.950 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:25:27.950 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:25:27.950 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:25:27.950 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:25:27.950 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:25:27.950 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:25:27.950 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:25:27.973 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:27 smithi103 ceph-mon[39629]: pgmap v726: 17 pgs: 17 active+clean; 6.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.37k op/s 2024-04-23T16:25:27.973 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:27 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2387356198' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:25:27.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:27 smithi188 ceph-mon[44101]: pgmap v726: 17 pgs: 17 active+clean; 6.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.37k op/s 2024-04-23T16:25:27.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:27 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2387356198' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:25:28.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:27 smithi148 ceph-mon[44520]: pgmap v726: 17 pgs: 17 active+clean; 6.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.37k op/s 2024-04-23T16:25:28.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:27 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2387356198' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:25:28.141 INFO:teuthology.orchestra.run.smithi103.stdout: marking collection for removal 2024-04-23T16:25:28.141 INFO:teuthology.orchestra.run.smithi103.stdout:setting '_remove' omap key 2024-04-23T16:25:28.146 INFO:teuthology.orchestra.run.smithi103.stdout:finish_remove_pgs 3.4_head removing 3.4 2024-04-23T16:25:28.302 INFO:teuthology.orchestra.run.smithi103.stdout:Remove successful 2024-04-23T16:25:28.648 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:25:28.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:28.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:25:28.754 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:25:28.792 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:28.826 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:25:28.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:28.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:25:28.930 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:28.969 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:28.969 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:28.969 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:28.969 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:28.970 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:28.970 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:28.970 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:28.970 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:28.970 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:28.970 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:28.970 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:28.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:28.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:28.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:29.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:29.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:29.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:29.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:29.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:29.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:29.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:29.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:29.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:29.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:29.706 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:25:29.732 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:29.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:29.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:29.763 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:25:29.764 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:25:29.838 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:25:29.868 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:29.898 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:25:29.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:29.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:25:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:29 smithi188 ceph-mon[44101]: pgmap v727: 17 pgs: 17 active+clean; 6.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.27k op/s 2024-04-23T16:25:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:29 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3456387375' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:25:30.004 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:25:30.041 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:30.070 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:25:30.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:30.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:29 smithi103 ceph-mon[39629]: pgmap v727: 17 pgs: 17 active+clean; 6.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.27k op/s 2024-04-23T16:25:30.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:29 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3456387375' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:25:30.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:30.077 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:25:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:29 smithi148 ceph-mon[44520]: pgmap v727: 17 pgs: 17 active+clean; 6.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.27k op/s 2024-04-23T16:25:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:29 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3456387375' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:25:30.178 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:25:30.205 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:30.233 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:25:30.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:30.242 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:25:30.343 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:30.382 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:30.692 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:30 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:25:30.693 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:30 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3456387375' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:25:30.693 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:30 smithi103 ceph-mon[39629]: osdmap e251: 12 total, 11 up, 10 in 2024-04-23T16:25:30.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:30 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:25:30.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:30 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3456387375' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:25:30.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:30 smithi188 ceph-mon[44101]: osdmap e251: 12 total, 11 up, 10 in 2024-04-23T16:25:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:30 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:25:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:30 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3456387375' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:25:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:30 smithi148 ceph-mon[44520]: osdmap e251: 12 total, 11 up, 10 in 2024-04-23T16:25:31.686 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:31.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:31.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:31.765 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:25:31.792 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:25:31.843 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:31.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:31 smithi188 ceph-mon[44101]: pgmap v729: 17 pgs: 17 active+clean; 6.5 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-23T16:25:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:31 smithi103 ceph-mon[39629]: pgmap v729: 17 pgs: 17 active+clean; 6.5 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-23T16:25:32.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:31 smithi148 ceph-mon[44520]: pgmap v729: 17 pgs: 17 active+clean; 6.5 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-23T16:25:32.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:32 smithi188 ceph-mon[44101]: 3.6 scrub starts 2024-04-23T16:25:33.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:32 smithi103 ceph-mon[39629]: 3.6 scrub starts 2024-04-23T16:25:33.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:32 smithi148 ceph-mon[44520]: 3.6 scrub starts 2024-04-23T16:25:33.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:33 smithi188 ceph-mon[44101]: pgmap v730: 17 pgs: 17 active+clean; 6.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.29k op/s 2024-04-23T16:25:33.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:33 smithi188 ceph-mon[44101]: 3.4 scrub starts 2024-04-23T16:25:33.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:33 smithi188 ceph-mon[44101]: 3.6 scrub ok 2024-04-23T16:25:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:33 smithi103 ceph-mon[39629]: pgmap v730: 17 pgs: 17 active+clean; 6.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.29k op/s 2024-04-23T16:25:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:33 smithi103 ceph-mon[39629]: 3.4 scrub starts 2024-04-23T16:25:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:33 smithi103 ceph-mon[39629]: 3.6 scrub ok 2024-04-23T16:25:34.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:33 smithi148 ceph-mon[44520]: pgmap v730: 17 pgs: 17 active+clean; 6.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.29k op/s 2024-04-23T16:25:34.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:33 smithi148 ceph-mon[44520]: 3.4 scrub starts 2024-04-23T16:25:34.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:33 smithi148 ceph-mon[44520]: 3.6 scrub ok 2024-04-23T16:25:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:34 smithi103 ceph-mon[39629]: 3.2 scrub starts 2024-04-23T16:25:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:34 smithi103 ceph-mon[39629]: 3.4 scrub ok 2024-04-23T16:25:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:34 smithi103 ceph-mon[39629]: pgmap v731: 17 pgs: 17 active+clean; 6.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.27k op/s 2024-04-23T16:25:35.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:34 smithi148 ceph-mon[44520]: 3.2 scrub starts 2024-04-23T16:25:35.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:34 smithi148 ceph-mon[44520]: 3.4 scrub ok 2024-04-23T16:25:35.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:34 smithi148 ceph-mon[44520]: pgmap v731: 17 pgs: 17 active+clean; 6.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.27k op/s 2024-04-23T16:25:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:34 smithi188 ceph-mon[44101]: 3.2 scrub starts 2024-04-23T16:25:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:34 smithi188 ceph-mon[44101]: 3.4 scrub ok 2024-04-23T16:25:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:34 smithi188 ceph-mon[44101]: pgmap v731: 17 pgs: 17 active+clean; 6.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.27k op/s 2024-04-23T16:25:36.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:35 smithi103 ceph-mon[39629]: 3.2 scrub ok 2024-04-23T16:25:36.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:35 smithi103 ceph-mon[39629]: 3.1 scrub starts 2024-04-23T16:25:36.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:35 smithi103 ceph-mon[39629]: 3.a scrub starts 2024-04-23T16:25:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:35 smithi148 ceph-mon[44520]: 3.2 scrub ok 2024-04-23T16:25:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:35 smithi148 ceph-mon[44520]: 3.1 scrub starts 2024-04-23T16:25:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:35 smithi148 ceph-mon[44520]: 3.a scrub starts 2024-04-23T16:25:36.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:35 smithi188 ceph-mon[44101]: 3.2 scrub ok 2024-04-23T16:25:36.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:35 smithi188 ceph-mon[44101]: 3.1 scrub starts 2024-04-23T16:25:36.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:35 smithi188 ceph-mon[44101]: 3.a scrub starts 2024-04-23T16:25:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:36 smithi103 ceph-mon[39629]: 3.1 scrub ok 2024-04-23T16:25:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:36 smithi103 ceph-mon[39629]: pgmap v732: 17 pgs: 17 active+clean; 6.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:25:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:36 smithi148 ceph-mon[44520]: 3.1 scrub ok 2024-04-23T16:25:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:36 smithi148 ceph-mon[44520]: pgmap v732: 17 pgs: 17 active+clean; 6.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:25:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:36 smithi188 ceph-mon[44101]: 3.1 scrub ok 2024-04-23T16:25:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:36 smithi188 ceph-mon[44101]: pgmap v732: 17 pgs: 17 active+clean; 6.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:25:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:37 smithi103 ceph-mon[39629]: 3.d scrub starts 2024-04-23T16:25:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:37 smithi103 ceph-mon[39629]: 3.a scrub ok 2024-04-23T16:25:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:37 smithi148 ceph-mon[44520]: 3.d scrub starts 2024-04-23T16:25:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:37 smithi148 ceph-mon[44520]: 3.a scrub ok 2024-04-23T16:25:38.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:37 smithi188 ceph-mon[44101]: 3.d scrub starts 2024-04-23T16:25:38.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:37 smithi188 ceph-mon[44101]: 3.a scrub ok 2024-04-23T16:25:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:39 smithi103 ceph-mon[39629]: 3.d scrub ok 2024-04-23T16:25:39.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:39 smithi103 ceph-mon[39629]: pgmap v733: 17 pgs: 17 active+clean; 5.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-23T16:25:39.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:39 smithi148 ceph-mon[44520]: 3.d scrub ok 2024-04-23T16:25:39.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:39 smithi148 ceph-mon[44520]: pgmap v733: 17 pgs: 17 active+clean; 5.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-23T16:25:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:39 smithi188 ceph-mon[44101]: 3.d scrub ok 2024-04-23T16:25:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:39 smithi188 ceph-mon[44101]: pgmap v733: 17 pgs: 17 active+clean; 5.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-23T16:25:40.786 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:40 smithi103 ceph-mon[39629]: 3.8 scrub starts 2024-04-23T16:25:40.812 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.1 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.4.1 2024-04-23T16:25:40.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:40 smithi148 ceph-mon[44520]: 3.8 scrub starts 2024-04-23T16:25:40.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:40 smithi188 ceph-mon[44101]: 3.8 scrub starts 2024-04-23T16:25:41.013 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:25:41.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:41.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:25:41.074 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:25:41.082 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:25:41.082 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:25:41.082 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-23T16:25:41.082 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-23T16:25:41.082 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:25:41.119 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:25:41.121 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.1/config 2024-04-23T16:25:41.154 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:41.184 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:25:41.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:41.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:25:41.292 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:41.328 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:41.481 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:41.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:41.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:41.513 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:41 smithi148 ceph-mon[44520]: pgmap v734: 17 pgs: 17 active+clean; 5.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-23T16:25:41.513 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:41 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:25:41.513 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:41 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:25:41.513 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:41 smithi148 ceph-mon[44520]: 3.8 scrub ok 2024-04-23T16:25:41.590 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:25:41.620 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:41.643 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:25:41.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:41.646 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:25:41.740 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:41 smithi103 ceph-mon[39629]: pgmap v734: 17 pgs: 17 active+clean; 5.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-23T16:25:41.740 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:41 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:25:41.740 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:41 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:25:41.741 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:41 smithi103 ceph-mon[39629]: 3.8 scrub ok 2024-04-23T16:25:41.746 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:25:41.787 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:41.821 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:25:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:41.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:25:41.929 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:41.961 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:41.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:41 smithi188 ceph-mon[44101]: pgmap v734: 17 pgs: 17 active+clean; 5.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-23T16:25:41.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:41 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:25:41.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:41 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:25:41.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:41 smithi188 ceph-mon[44101]: 3.8 scrub ok 2024-04-23T16:25:42.290 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:25:42.370 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-23T16:25:42.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:42 smithi103 ceph-mon[39629]: osdmap e252: 12 total, 11 up, 10 in 2024-04-23T16:25:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:42 smithi148 ceph-mon[44520]: osdmap e252: 12 total, 11 up, 10 in 2024-04-23T16:25:42.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:42 smithi188 ceph-mon[44101]: osdmap e252: 12 total, 11 up, 10 in 2024-04-23T16:25:43.024 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:43.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:43.127 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:25:43.162 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:43.186 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:25:43.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:43.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:25:43.294 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:25:43.335 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:43.363 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:25:43.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:43.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:25:43.468 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:25:43.508 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:43.528 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:43 smithi103 ceph-mon[39629]: pgmap v736: 17 pgs: 17 active+clean; 5.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:25:43.528 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:43 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:25:43.528 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:43 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:25:43.533 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:25:43.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:43.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:25:43.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:43 smithi148 ceph-mon[44520]: pgmap v736: 17 pgs: 17 active+clean; 5.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:25:43.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:43 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:25:43.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:43 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:25:43.638 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:25:43.671 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:43.699 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:25:43.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:43.705 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:25:43.805 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:25:43.855 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:43.881 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:25:43.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:43.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:25:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:43 smithi188 ceph-mon[44101]: pgmap v736: 17 pgs: 17 active+clean; 5.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:25:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:43 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:25:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:43 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:25:43.991 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:25:44.033 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:44.058 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:25:44.067 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:25:44.067 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:25:44.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:44.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:25:44.098 INFO:teuthology.orchestra.run.smithi103.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:25:44.171 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:25:44.207 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:44.234 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:25:44.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:44.237 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:25:44.338 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:25:44.369 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:44.392 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:25:44.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:44.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:25:44.497 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:25:44.536 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:44.567 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:25:44.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:44.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:25:44.674 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:25:44.694 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:44 smithi103 ceph-mon[39629]: osdmap e253: 12 total, 11 up, 10 in 2024-04-23T16:25:44.694 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:44 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:25:44.694 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:44 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:25:44.705 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:44.730 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:25:44.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:44.743 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:25:44.844 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:25:44.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:44 smithi148 ceph-mon[44520]: osdmap e253: 12 total, 11 up, 10 in 2024-04-23T16:25:44.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:44 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:25:44.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:44 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:25:44.884 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:44 smithi188 ceph-mon[44101]: osdmap e253: 12 total, 11 up, 10 in 2024-04-23T16:25:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:44 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:25:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:44 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:25:45.270 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:25:45.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:45.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:25:45.374 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:25:45.422 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:45.453 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:25:45.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:45.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:25:45.563 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:25:45.601 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:45.631 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:25:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:45.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:25:45.735 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:25:45.767 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:45 smithi148 ceph-mon[44520]: osdmap e254: 12 total, 11 up, 10 in 2024-04-23T16:25:45.767 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:45 smithi148 ceph-mon[44520]: pgmap v739: 17 pgs: 17 active+clean; 5.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:25:45.770 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:45.795 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:25:45.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:45.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:25:45.854 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:45 smithi103 ceph-mon[39629]: osdmap e254: 12 total, 11 up, 10 in 2024-04-23T16:25:45.855 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:45 smithi103 ceph-mon[39629]: pgmap v739: 17 pgs: 17 active+clean; 5.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:25:45.900 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:25:45.933 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:45 smithi188 ceph-mon[44101]: osdmap e254: 12 total, 11 up, 10 in 2024-04-23T16:25:45.933 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:45 smithi188 ceph-mon[44101]: pgmap v739: 17 pgs: 17 active+clean; 5.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:25:45.934 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:45.956 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:25:45.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:45.959 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:25:46.061 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:25:46.098 INFO:teuthology.orchestra.run.smithi103.stdout:get_pg_num_history pg_num_history pg_num_history(e232 pg_nums {1={15=1},2={110=16},3={192=16}} deleted_pools 189,2) 2024-04-23T16:25:46.098 INFO:teuthology.orchestra.run.smithi103.stdout:Importing pgid 3.4 2024-04-23T16:25:46.102 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:46.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:200026e8:::benchmark_data_smithi188_89276_object110474:head# 2024-04-23T16:25:46.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2000f5dd:::benchmark_data_smithi188_89276_object78808:head# 2024-04-23T16:25:46.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20014054:::benchmark_data_smithi188_89276_object78008:head# 2024-04-23T16:25:46.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2001e0d9:::benchmark_data_smithi188_89276_object142273:head# 2024-04-23T16:25:46.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20023ee6:::benchmark_data_smithi188_89276_object69071:head# 2024-04-23T16:25:46.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20028e3c:::benchmark_data_smithi188_89276_object149088:head# 2024-04-23T16:25:46.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2002cb3c:::benchmark_data_smithi188_89276_object54086:head# 2024-04-23T16:25:46.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20031387:::benchmark_data_smithi188_89276_object93275:head# 2024-04-23T16:25:46.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2005b236:::benchmark_data_smithi188_89276_object147534:head# 2024-04-23T16:25:46.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20061582:::benchmark_data_smithi188_89276_object97962:head# 2024-04-23T16:25:46.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:200644b3:::benchmark_data_smithi188_89276_object72676:head# 2024-04-23T16:25:46.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20065104:::benchmark_data_smithi188_89276_object95126:head# 2024-04-23T16:25:46.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20073164:::benchmark_data_smithi188_89276_object93259:head# 2024-04-23T16:25:46.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2007a527:::benchmark_data_smithi188_89276_object41416:head# 2024-04-23T16:25:46.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2007b429:::benchmark_data_smithi188_89276_object118514:head# 2024-04-23T16:25:46.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2007d92f:::benchmark_data_smithi188_89276_object43767:head# 2024-04-23T16:25:46.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20083883:::benchmark_data_smithi188_89276_object160356:head# 2024-04-23T16:25:46.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20084de0:::benchmark_data_smithi188_89276_object98513:head# 2024-04-23T16:25:46.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20093d4e:::benchmark_data_smithi188_89276_object136812:head# 2024-04-23T16:25:46.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2009bcb7:::benchmark_data_smithi188_89276_object98740:head# 2024-04-23T16:25:46.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.120 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:200a1976:::benchmark_data_smithi188_89276_object47250:head# 2024-04-23T16:25:46.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.120 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:200a7783:::benchmark_data_smithi188_89276_object56969:head# 2024-04-23T16:25:46.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.120 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:200b3122:::benchmark_data_smithi188_89276_object149545:head# 2024-04-23T16:25:46.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.121 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:200b5764:::benchmark_data_smithi188_89276_object147577:head# 2024-04-23T16:25:46.121 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.121 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:200be0b6:::benchmark_data_smithi188_89276_object112363:head# 2024-04-23T16:25:46.121 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.121 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:200ca6ba:::benchmark_data_smithi188_89276_object70690:head# 2024-04-23T16:25:46.121 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.121 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:200d763b:::benchmark_data_smithi188_89276_object163144:head# 2024-04-23T16:25:46.121 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.121 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:200d815f:::benchmark_data_smithi188_89276_object42404:head# 2024-04-23T16:25:46.121 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.122 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:200dc711:::benchmark_data_smithi188_89276_object159854:head# 2024-04-23T16:25:46.122 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.123 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:200eb5c0:::benchmark_data_smithi188_89276_object142151:head# 2024-04-23T16:25:46.123 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.123 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:200ed284:::benchmark_data_smithi188_89276_object106706:head# 2024-04-23T16:25:46.123 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.123 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:200fed30:::benchmark_data_smithi188_89276_object56574:head# 2024-04-23T16:25:46.123 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.123 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:201091a5:::benchmark_data_smithi188_89276_object121737:head# 2024-04-23T16:25:46.123 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20115e15:::benchmark_data_smithi188_89276_object86448:head# 2024-04-23T16:25:46.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20129071:::benchmark_data_smithi188_89276_object59619:head# 2024-04-23T16:25:46.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2012bb75:::benchmark_data_smithi188_89276_object98761:head# 2024-04-23T16:25:46.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2012c83f:::benchmark_data_smithi188_89276_object44124:head# 2024-04-23T16:25:46.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20142c56:::benchmark_data_smithi188_89276_object70844:head# 2024-04-23T16:25:46.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20156ced:::benchmark_data_smithi188_89276_object166454:head# 2024-04-23T16:25:46.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2015ba28:::benchmark_data_smithi188_89276_object111393:head# 2024-04-23T16:25:46.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2015bf70:::benchmark_data_smithi188_89276_object69293:head# 2024-04-23T16:25:46.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:201613c2:::benchmark_data_smithi188_89276_object45556:head# 2024-04-23T16:25:46.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2016d72d:::benchmark_data_smithi188_89276_object128580:head# 2024-04-23T16:25:46.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2016fc3e:::benchmark_data_smithi188_89276_object137187:head# 2024-04-23T16:25:46.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20176e94:::benchmark_data_smithi188_89276_object151654:head# 2024-04-23T16:25:46.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.134 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20185070:::benchmark_data_smithi188_89276_object47264:head# 2024-04-23T16:25:46.134 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.134 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20185590:::benchmark_data_smithi188_89276_object142096:head# 2024-04-23T16:25:46.134 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.134 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20192450:::benchmark_data_smithi188_89276_object89878:head# 2024-04-23T16:25:46.134 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:201bf592:::benchmark_data_smithi188_89276_object98037:head# 2024-04-23T16:25:46.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:201d4625:::benchmark_data_smithi188_89276_object140977:head# 2024-04-23T16:25:46.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:201d7f7a:::benchmark_data_smithi188_89276_object46781:head# 2024-04-23T16:25:46.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:201dce51:::benchmark_data_smithi188_89276_object82357:head# 2024-04-23T16:25:46.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:201dee1d:::benchmark_data_smithi188_89276_object144816:head# 2024-04-23T16:25:46.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:201f44c5:::benchmark_data_smithi188_89276_object108351:head# 2024-04-23T16:25:46.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:201f7d22:::benchmark_data_smithi188_89276_object109760:head# 2024-04-23T16:25:46.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.137 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:201fa8ce:::benchmark_data_smithi188_89276_object95518:head# 2024-04-23T16:25:46.137 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.137 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2020c452:::benchmark_data_smithi188_89276_object111596:head# 2024-04-23T16:25:46.137 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.137 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2020da4a:::benchmark_data_smithi188_89276_object107127:head# 2024-04-23T16:25:46.137 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.137 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20220ce8:::benchmark_data_smithi188_89276_object125616:head# 2024-04-23T16:25:46.137 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:202256a4:::benchmark_data_smithi188_89276_object91553:head# 2024-04-23T16:25:46.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:202275d9:::benchmark_data_smithi188_89276_object72134:head# 2024-04-23T16:25:46.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2022da58:::benchmark_data_smithi188_89276_object125113:head# 2024-04-23T16:25:46.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20236537:::benchmark_data_smithi188_89276_object121594:head# 2024-04-23T16:25:46.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2023e6be:::benchmark_data_smithi188_89276_object149598:head# 2024-04-23T16:25:46.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2024829b:::benchmark_data_smithi188_89276_object71062:head# 2024-04-23T16:25:46.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20249b4b:::benchmark_data_smithi188_89276_object61513:head# 2024-04-23T16:25:46.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2024c96e:::benchmark_data_smithi188_89276_object48293:head# 2024-04-23T16:25:46.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20251c80:::benchmark_data_smithi188_89276_object92867:head# 2024-04-23T16:25:46.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20253689:::benchmark_data_smithi188_89276_object117130:head# 2024-04-23T16:25:46.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20255acf:::benchmark_data_smithi188_89276_object69357:head# 2024-04-23T16:25:46.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:202721cd:::benchmark_data_smithi188_89276_object99155:head# 2024-04-23T16:25:46.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2028a971:::benchmark_data_smithi188_89276_object126238:head# 2024-04-23T16:25:46.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:202938ed:::benchmark_data_smithi188_89276_object136303:head# 2024-04-23T16:25:46.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20296a45:::benchmark_data_smithi188_89276_object110561:head# 2024-04-23T16:25:46.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2029bf05:::benchmark_data_smithi188_89276_object100372:head# 2024-04-23T16:25:46.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:202a4dc2:::benchmark_data_smithi188_89276_object96846:head# 2024-04-23T16:25:46.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:202aee2f:::benchmark_data_smithi188_89276_object44574:head# 2024-04-23T16:25:46.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:202bde49:::benchmark_data_smithi188_89276_object62369:head# 2024-04-23T16:25:46.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:202c1bc9:::benchmark_data_smithi188_89276_object89049:head# 2024-04-23T16:25:46.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:202cb0ba:::benchmark_data_smithi188_89276_object144137:head# 2024-04-23T16:25:46.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:202cc908:::benchmark_data_smithi188_89276_object163998:head# 2024-04-23T16:25:46.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:202d5615:::benchmark_data_smithi188_89276_object159202:head# 2024-04-23T16:25:46.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:202d8c5b:::benchmark_data_smithi188_89276_object45867:head# 2024-04-23T16:25:46.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20310dc1:::benchmark_data_smithi188_89276_object90901:head# 2024-04-23T16:25:46.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2031d274:::benchmark_data_smithi188_89276_object104815:head# 2024-04-23T16:25:46.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2032d748:::benchmark_data_smithi188_89276_object115105:head# 2024-04-23T16:25:46.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2033c3ed:::benchmark_data_smithi188_89276_object96800:head# 2024-04-23T16:25:46.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2034396b:::benchmark_data_smithi188_89276_object123670:head# 2024-04-23T16:25:46.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20343a92:::benchmark_data_smithi188_89276_object132623:head# 2024-04-23T16:25:46.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2034d2fd:::benchmark_data_smithi188_89276_object91487:head# 2024-04-23T16:25:46.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2034e2f6:::benchmark_data_smithi188_89276_object153364:head# 2024-04-23T16:25:46.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2034f317:::benchmark_data_smithi188_89276_object89571:head# 2024-04-23T16:25:46.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2034f803:::benchmark_data_smithi188_89276_object50065:head# 2024-04-23T16:25:46.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20367792:::benchmark_data_smithi188_89276_object65109:head# 2024-04-23T16:25:46.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2036c55f:::benchmark_data_smithi188_89276_object65976:head# 2024-04-23T16:25:46.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20377400:::benchmark_data_smithi188_89276_object158221:head# 2024-04-23T16:25:46.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:203880ab:::benchmark_data_smithi188_89276_object129101:head# 2024-04-23T16:25:46.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20395b9f:::benchmark_data_smithi188_89276_object163905:head# 2024-04-23T16:25:46.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20398c96:::benchmark_data_smithi188_89276_object101004:head# 2024-04-23T16:25:46.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:203a6812:::benchmark_data_smithi188_89276_object127991:head# 2024-04-23T16:25:46.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:203aa6b4:::benchmark_data_smithi188_89276_object96303:head# 2024-04-23T16:25:46.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:203bbab8:::benchmark_data_smithi188_89276_object95311:head# 2024-04-23T16:25:46.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:203bf7db:::benchmark_data_smithi188_89276_object64774:head# 2024-04-23T16:25:46.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:203c1d25:::benchmark_data_smithi188_89276_object76686:head# 2024-04-23T16:25:46.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:203d02fc:::benchmark_data_smithi188_89276_object43912:head# 2024-04-23T16:25:46.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:203d2ccf:::benchmark_data_smithi188_89276_object136533:head# 2024-04-23T16:25:46.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:203d9e81:::benchmark_data_smithi188_89276_object136778:head# 2024-04-23T16:25:46.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:203e340f:::benchmark_data_smithi188_89276_object154318:head# 2024-04-23T16:25:46.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:203ed244:::benchmark_data_smithi188_89276_object158897:head# 2024-04-23T16:25:46.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:203efe0f:::benchmark_data_smithi188_89276_object136565:head# 2024-04-23T16:25:46.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:203f6945:::benchmark_data_smithi188_89276_object144058:head# 2024-04-23T16:25:46.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:203f94d0:::benchmark_data_smithi188_89276_object133348:head# 2024-04-23T16:25:46.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:203fd7e8:::benchmark_data_smithi188_89276_object143224:head# 2024-04-23T16:25:46.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20401769:::benchmark_data_smithi188_89276_object90946:head# 2024-04-23T16:25:46.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:204019fc:::benchmark_data_smithi188_89276_object67298:head# 2024-04-23T16:25:46.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2042be71:::benchmark_data_smithi188_89276_object140326:head# 2024-04-23T16:25:46.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2042c1ec:::benchmark_data_smithi188_89276_object105993:head# 2024-04-23T16:25:46.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2042ffdb:::benchmark_data_smithi188_89276_object105649:head# 2024-04-23T16:25:46.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20430017:::benchmark_data_smithi188_89276_object79840:head# 2024-04-23T16:25:46.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2043b0fe:::benchmark_data_smithi188_89276_object132113:head# 2024-04-23T16:25:46.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20440807:::benchmark_data_smithi188_89276_object116543:head# 2024-04-23T16:25:46.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20444a96:::benchmark_data_smithi188_89276_object125037:head# 2024-04-23T16:25:46.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20449b12:::benchmark_data_smithi188_89276_object165623:head# 2024-04-23T16:25:46.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2045a2e3:::benchmark_data_smithi188_89276_object116344:head# 2024-04-23T16:25:46.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2045eaec:::benchmark_data_smithi188_89276_object63478:head# 2024-04-23T16:25:46.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20465bbb:::benchmark_data_smithi188_89276_object112833:head# 2024-04-23T16:25:46.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2046ba9f:::benchmark_data_smithi188_89276_object43329:head# 2024-04-23T16:25:46.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2046e477:::benchmark_data_smithi188_89276_object133855:head# 2024-04-23T16:25:46.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:204837c1:::benchmark_data_smithi188_89276_object65587:head# 2024-04-23T16:25:46.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20488fcf:::benchmark_data_smithi188_89276_object50412:head# 2024-04-23T16:25:46.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:204892b0:::benchmark_data_smithi188_89276_object162839:head# 2024-04-23T16:25:46.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20490f84:::benchmark_data_smithi188_89276_object55799:head# 2024-04-23T16:25:46.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20492552:::benchmark_data_smithi188_89276_object64555:head# 2024-04-23T16:25:46.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:204c0ed7:::benchmark_data_smithi188_89276_object82345:head# 2024-04-23T16:25:46.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:204c1abd:::benchmark_data_smithi188_89276_object131971:head# 2024-04-23T16:25:46.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:204c382f:::benchmark_data_smithi188_89276_object49152:head# 2024-04-23T16:25:46.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:204d27cf:::benchmark_data_smithi188_89276_object55866:head# 2024-04-23T16:25:46.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:204e236a:::benchmark_data_smithi188_89276_object70809:head# 2024-04-23T16:25:46.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:204e706b:::benchmark_data_smithi188_89276_object153400:head# 2024-04-23T16:25:46.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:204e868c:::benchmark_data_smithi188_89276_object123607:head# 2024-04-23T16:25:46.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:204f8215:::benchmark_data_smithi188_89276_object98450:head# 2024-04-23T16:25:46.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:204f9b78:::benchmark_data_smithi188_89276_object110247:head# 2024-04-23T16:25:46.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:204fa8c6:::benchmark_data_smithi188_89276_object101048:head# 2024-04-23T16:25:46.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:205080cb:::benchmark_data_smithi188_89276_object51753:head# 2024-04-23T16:25:46.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2050ba96:::benchmark_data_smithi188_89276_object146726:head# 2024-04-23T16:25:46.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2051d98a:::benchmark_data_smithi188_89276_object124779:head# 2024-04-23T16:25:46.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2051e294:::benchmark_data_smithi188_89276_object118156:head# 2024-04-23T16:25:46.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20521175:::benchmark_data_smithi188_89276_object102939:head# 2024-04-23T16:25:46.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2052739d:::benchmark_data_smithi188_89276_object142208:head# 2024-04-23T16:25:46.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:205284b6:::benchmark_data_smithi188_89276_object166878:head# 2024-04-23T16:25:46.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2052c637:::benchmark_data_smithi188_89276_object79725:head# 2024-04-23T16:25:46.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2052f05a:::benchmark_data_smithi188_89276_object87850:head# 2024-04-23T16:25:46.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20531402:::benchmark_data_smithi188_89276_object85374:head# 2024-04-23T16:25:46.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20539538:::benchmark_data_smithi188_89276_object123509:head# 2024-04-23T16:25:46.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2053aa95:::benchmark_data_smithi188_89276_object130013:head# 2024-04-23T16:25:46.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2053e56b:::benchmark_data_smithi188_89276_object146183:head# 2024-04-23T16:25:46.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20544816:::benchmark_data_smithi188_89276_object87229:head# 2024-04-23T16:25:46.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:205497b4:::benchmark_data_smithi188_89276_object102570:head# 2024-04-23T16:25:46.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2054de37:::benchmark_data_smithi188_89276_object71835:head# 2024-04-23T16:25:46.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2054f6fb:::benchmark_data_smithi188_89276_object50142:head# 2024-04-23T16:25:46.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:205542d0:::benchmark_data_smithi188_89276_object167993:head# 2024-04-23T16:25:46.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2055987b:::benchmark_data_smithi188_89276_object109946:head# 2024-04-23T16:25:46.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2055a27e:::benchmark_data_smithi188_89276_object145910:head# 2024-04-23T16:25:46.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2055ee5e:::benchmark_data_smithi188_89276_object49225:head# 2024-04-23T16:25:46.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20563cd2:::benchmark_data_smithi188_89276_object111421:head# 2024-04-23T16:25:46.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2057502d:::benchmark_data_smithi188_89276_object162438:head# 2024-04-23T16:25:46.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20579485:::benchmark_data_smithi188_89276_object72455:head# 2024-04-23T16:25:46.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2058e11e:::benchmark_data_smithi188_89276_object46559:head# 2024-04-23T16:25:46.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:205aa197:::benchmark_data_smithi188_89276_object100662:head# 2024-04-23T16:25:46.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:205aa1c2:::benchmark_data_smithi188_89276_object69902:head# 2024-04-23T16:25:46.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:205aac8a:::benchmark_data_smithi188_89276_object96493:head# 2024-04-23T16:25:46.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:205ad0a3:::benchmark_data_smithi188_89276_object42998:head# 2024-04-23T16:25:46.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:205bb2a7:::benchmark_data_smithi188_89276_object88057:head# 2024-04-23T16:25:46.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:205ccf33:::benchmark_data_smithi188_89276_object121399:head# 2024-04-23T16:25:46.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:205d3bc4:::benchmark_data_smithi188_89276_object97540:head# 2024-04-23T16:25:46.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:205d6815:::benchmark_data_smithi188_89276_object144096:head# 2024-04-23T16:25:46.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:205d967c:::benchmark_data_smithi188_89276_object139786:head# 2024-04-23T16:25:46.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:205dd73d:::benchmark_data_smithi188_89276_object136640:head# 2024-04-23T16:25:46.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:205eca1a:::benchmark_data_smithi188_89276_object55622:head# 2024-04-23T16:25:46.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:205ecedd:::benchmark_data_smithi188_89276_object164588:head# 2024-04-23T16:25:46.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:205ef0c8:::benchmark_data_smithi188_89276_object93847:head# 2024-04-23T16:25:46.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:205ef5c9:::benchmark_data_smithi188_89276_object85352:head# 2024-04-23T16:25:46.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:205f31d1:::benchmark_data_smithi188_89276_object162315:head# 2024-04-23T16:25:46.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20604d1f:::benchmark_data_smithi188_89276_object114493:head# 2024-04-23T16:25:46.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20608922:::benchmark_data_smithi188_89276_object168292:head# 2024-04-23T16:25:46.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2060d6f6:::benchmark_data_smithi188_89276_object162184:head# 2024-04-23T16:25:46.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20612186:::benchmark_data_smithi188_89276_object86508:head# 2024-04-23T16:25:46.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:206172b8:::benchmark_data_smithi188_89276_object57158:head# 2024-04-23T16:25:46.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2061dfef:::benchmark_data_smithi188_89276_object103644:head# 2024-04-23T16:25:46.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2061f79e:::benchmark_data_smithi188_89276_object89424:head# 2024-04-23T16:25:46.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20626b01:::benchmark_data_smithi188_89276_object82448:head# 2024-04-23T16:25:46.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2062900c:::benchmark_data_smithi188_89276_object159925:head# 2024-04-23T16:25:46.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2063cf29:::benchmark_data_smithi188_89276_object78595:head# 2024-04-23T16:25:46.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2063f6b2:::benchmark_data_smithi188_89276_object89550:head# 2024-04-23T16:25:46.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2064085b:::benchmark_data_smithi188_89276_object58529:head# 2024-04-23T16:25:46.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:206411de:::benchmark_data_smithi188_89276_object63261:head# 2024-04-23T16:25:46.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20641e6a:::benchmark_data_smithi188_89276_object52086:head# 2024-04-23T16:25:46.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2064cc79:::benchmark_data_smithi188_89276_object69408:head# 2024-04-23T16:25:46.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:206626ef:::benchmark_data_smithi188_89276_object120804:head# 2024-04-23T16:25:46.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2066a3cb:::benchmark_data_smithi188_89276_object52348:head# 2024-04-23T16:25:46.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20674846:::benchmark_data_smithi188_89276_object134035:head# 2024-04-23T16:25:46.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2068441c:::benchmark_data_smithi188_89276_object74869:head# 2024-04-23T16:25:46.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2069b528:::benchmark_data_smithi188_89276_object71258:head# 2024-04-23T16:25:46.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2069bada:::benchmark_data_smithi188_89276_object84064:head# 2024-04-23T16:25:46.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2069bb89:::benchmark_data_smithi188_89276_object133211:head# 2024-04-23T16:25:46.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:206ac46f:::benchmark_data_smithi188_89276_object74101:head# 2024-04-23T16:25:46.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:206af9fc:::benchmark_data_smithi188_89276_object87321:head# 2024-04-23T16:25:46.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:206dc280:::benchmark_data_smithi188_89276_object153649:head# 2024-04-23T16:25:46.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:206dce15:::benchmark_data_smithi188_89276_object77560:head# 2024-04-23T16:25:46.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:206e2d1f:::benchmark_data_smithi188_89276_object162033:head# 2024-04-23T16:25:46.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:206f2d00:::benchmark_data_smithi188_89276_object162875:head# 2024-04-23T16:25:46.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:206fdcf7:::benchmark_data_smithi188_89276_object119804:head# 2024-04-23T16:25:46.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:207085b2:::benchmark_data_smithi188_89276_object130826:head# 2024-04-23T16:25:46.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2070a0cd:::benchmark_data_smithi188_89276_object149712:head# 2024-04-23T16:25:46.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2070c009:::benchmark_data_smithi188_89276_object98453:head# 2024-04-23T16:25:46.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20739866:::benchmark_data_smithi188_89276_object114754:head# 2024-04-23T16:25:46.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:207438bc:::benchmark_data_smithi188_89276_object41211:head# 2024-04-23T16:25:46.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2074e3c3:::benchmark_data_smithi188_89276_object86797:head# 2024-04-23T16:25:46.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2074f1b3:::benchmark_data_smithi188_89276_object152142:head# 2024-04-23T16:25:46.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20750d89:::benchmark_data_smithi188_89276_object151118:head# 2024-04-23T16:25:46.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:207595b9:::benchmark_data_smithi188_89276_object164097:head# 2024-04-23T16:25:46.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2075ceac:::benchmark_data_smithi188_89276_object57611:head# 2024-04-23T16:25:46.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2075f5f5:::benchmark_data_smithi188_89276_object51970:head# 2024-04-23T16:25:46.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2076638a:::benchmark_data_smithi188_89276_object150061:head# 2024-04-23T16:25:46.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20767820:::benchmark_data_smithi188_89276_object160171:head# 2024-04-23T16:25:46.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20768281:::benchmark_data_smithi188_89276_object52716:head# 2024-04-23T16:25:46.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20774f7f:::benchmark_data_smithi188_89276_object89503:head# 2024-04-23T16:25:46.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20780a63:::benchmark_data_smithi188_89276_object66011:head# 2024-04-23T16:25:46.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20785606:::benchmark_data_smithi188_89276_object105485:head# 2024-04-23T16:25:46.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20785b58:::benchmark_data_smithi188_89276_object151996:head# 2024-04-23T16:25:46.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2078cc49:::benchmark_data_smithi188_89276_object138916:head# 2024-04-23T16:25:46.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2078d7df:::benchmark_data_smithi188_89276_object166669:head# 2024-04-23T16:25:46.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2078eaea:::benchmark_data_smithi188_89276_object161741:head# 2024-04-23T16:25:46.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:207a37ea:::benchmark_data_smithi188_89276_object132724:head# 2024-04-23T16:25:46.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:207a4f38:::benchmark_data_smithi188_89276_object106455:head# 2024-04-23T16:25:46.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:207af32a:::benchmark_data_smithi188_89276_object93680:head# 2024-04-23T16:25:46.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:207b116f:::benchmark_data_smithi188_89276_object68196:head# 2024-04-23T16:25:46.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:207ba60f:::benchmark_data_smithi188_89276_object74171:head# 2024-04-23T16:25:46.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:207bff91:::benchmark_data_smithi188_89276_object146530:head# 2024-04-23T16:25:46.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:207c0177:::benchmark_data_smithi188_89276_object149431:head# 2024-04-23T16:25:46.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:207c6d8f:::benchmark_data_smithi188_89276_object51998:head# 2024-04-23T16:25:46.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:207d1624:::benchmark_data_smithi188_89276_object153646:head# 2024-04-23T16:25:46.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:207d4dde:::benchmark_data_smithi188_89276_object120074:head# 2024-04-23T16:25:46.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:207d5dfa:::benchmark_data_smithi188_89276_object166528:head# 2024-04-23T16:25:46.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:207d9862:::benchmark_data_smithi188_89276_object53330:head# 2024-04-23T16:25:46.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:207df6c2:::benchmark_data_smithi188_89276_object133642:head# 2024-04-23T16:25:46.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:207e8bbf:::benchmark_data_smithi188_89276_object153930:head# 2024-04-23T16:25:46.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:207ee6c4:::benchmark_data_smithi188_89276_object166394:head# 2024-04-23T16:25:46.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:207fcfdc:::benchmark_data_smithi188_89276_object69873:head# 2024-04-23T16:25:46.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20807dee:::benchmark_data_smithi188_89276_object130004:head# 2024-04-23T16:25:46.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2080e360:::benchmark_data_smithi188_89276_object113680:head# 2024-04-23T16:25:46.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20818809:::benchmark_data_smithi188_89276_object65752:head# 2024-04-23T16:25:46.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20819f79:::benchmark_data_smithi188_89276_object144789:head# 2024-04-23T16:25:46.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20828a17:::benchmark_data_smithi188_89276_object72038:head# 2024-04-23T16:25:46.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20830b52:::benchmark_data_smithi188_89276_object89013:head# 2024-04-23T16:25:46.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20837ed8:::benchmark_data_smithi188_89276_object78648:head# 2024-04-23T16:25:46.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2083c3a3:::benchmark_data_smithi188_89276_object60392:head# 2024-04-23T16:25:46.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2084aade:::benchmark_data_smithi188_89276_object113627:head# 2024-04-23T16:25:46.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2084c117:::benchmark_data_smithi188_89276_object165362:head# 2024-04-23T16:25:46.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2084fccb:::benchmark_data_smithi188_89276_object77166:head# 2024-04-23T16:25:46.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20853edd:::benchmark_data_smithi188_89276_object136727:head# 2024-04-23T16:25:46.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20855dc4:::benchmark_data_smithi188_89276_object41666:head# 2024-04-23T16:25:46.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20856796:::benchmark_data_smithi188_89276_object99245:head# 2024-04-23T16:25:46.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2086dc80:::benchmark_data_smithi188_89276_object147142:head# 2024-04-23T16:25:46.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2087ac8d:::benchmark_data_smithi188_89276_object75266:head# 2024-04-23T16:25:46.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2087f919:::benchmark_data_smithi188_89276_object45251:head# 2024-04-23T16:25:46.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20884915:::benchmark_data_smithi188_89276_object105111:head# 2024-04-23T16:25:46.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2088ffc6:::benchmark_data_smithi188_89276_object91679:head# 2024-04-23T16:25:46.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:208aa8d1:::benchmark_data_smithi188_89276_object147471:head# 2024-04-23T16:25:46.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:208aea33:::benchmark_data_smithi188_89276_object69874:head# 2024-04-23T16:25:46.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:208af4ef:::benchmark_data_smithi188_89276_object92171:head# 2024-04-23T16:25:46.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:208afac1:::benchmark_data_smithi188_89276_object95967:head# 2024-04-23T16:25:46.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:208b72ff:::benchmark_data_smithi188_89276_object149877:head# 2024-04-23T16:25:46.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:208ccfae:::benchmark_data_smithi188_89276_object40768:head# 2024-04-23T16:25:46.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:208ce45b:::benchmark_data_smithi188_89276_object133832:head# 2024-04-23T16:25:46.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:208cfa2a:::benchmark_data_smithi188_89276_object54310:head# 2024-04-23T16:25:46.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:208db075:::benchmark_data_smithi188_89276_object155753:head# 2024-04-23T16:25:46.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:208e7c68:::benchmark_data_smithi188_89276_object96944:head# 2024-04-23T16:25:46.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:208f6c2e:::benchmark_data_smithi188_89276_object145171:head# 2024-04-23T16:25:46.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2090f4d9:::benchmark_data_smithi188_89276_object130926:head# 2024-04-23T16:25:46.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209149bb:::benchmark_data_smithi188_89276_object67374:head# 2024-04-23T16:25:46.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209152a5:::benchmark_data_smithi188_89276_object103024:head# 2024-04-23T16:25:46.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209184a3:::benchmark_data_smithi188_89276_object121945:head# 2024-04-23T16:25:46.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2091a96c:::benchmark_data_smithi188_89276_object96879:head# 2024-04-23T16:25:46.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20928766:::benchmark_data_smithi188_89276_object100194:head# 2024-04-23T16:25:46.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2092f6b4:::benchmark_data_smithi188_89276_object136086:head# 2024-04-23T16:25:46.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20932b32:::benchmark_data_smithi188_89276_object54478:head# 2024-04-23T16:25:46.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20934b28:::benchmark_data_smithi188_89276_object156365:head# 2024-04-23T16:25:46.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2093764e:::benchmark_data_smithi188_89276_object146205:head# 2024-04-23T16:25:46.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20939feb:::benchmark_data_smithi188_89276_object79867:head# 2024-04-23T16:25:46.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20945256:::benchmark_data_smithi188_89276_object103121:head# 2024-04-23T16:25:46.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2094cd70:::benchmark_data_smithi188_89276_object156212:head# 2024-04-23T16:25:46.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20951808:::benchmark_data_smithi188_89276_object149859:head# 2024-04-23T16:25:46.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20955680:::benchmark_data_smithi188_89276_object61753:head# 2024-04-23T16:25:46.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209559d3:::benchmark_data_smithi188_89276_object161351:head# 2024-04-23T16:25:46.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209564a9:::benchmark_data_smithi188_89276_object74357:head# 2024-04-23T16:25:46.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209571c6:::benchmark_data_smithi188_89276_object115810:head# 2024-04-23T16:25:46.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2095cd91:::benchmark_data_smithi188_89276_object154265:head# 2024-04-23T16:25:46.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20960bd8:::benchmark_data_smithi188_89276_object168299:head# 2024-04-23T16:25:46.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20961037:::benchmark_data_smithi188_89276_object94883:head# 2024-04-23T16:25:46.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209779ef:::benchmark_data_smithi188_89276_object137093:head# 2024-04-23T16:25:46.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2097a2e6:::benchmark_data_smithi188_89276_object136366:head# 2024-04-23T16:25:46.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2097efc3:::benchmark_data_smithi188_89276_object168597:head# 2024-04-23T16:25:46.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20987dfc:::benchmark_data_smithi188_89276_object133959:head# 2024-04-23T16:25:46.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209966a0:::benchmark_data_smithi188_89276_object119477:head# 2024-04-23T16:25:46.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2099c89d:::benchmark_data_smithi188_89276_object45481:head# 2024-04-23T16:25:46.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2099e999:::benchmark_data_smithi188_89276_object152876:head# 2024-04-23T16:25:46.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209a17b8:::benchmark_data_smithi188_89276_object87191:head# 2024-04-23T16:25:46.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209a20a7:::benchmark_data_smithi188_89276_object63984:head# 2024-04-23T16:25:46.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209a46b4:::benchmark_data_smithi188_89276_object57168:head# 2024-04-23T16:25:46.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209ad30a:::benchmark_data_smithi188_89276_object43296:head# 2024-04-23T16:25:46.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209bf04a:::benchmark_data_smithi188_89276_object94854:head# 2024-04-23T16:25:46.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209d118c:::benchmark_data_smithi188_89276_object160533:head# 2024-04-23T16:25:46.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209d72d5:::benchmark_data_smithi188_89276_object121356:head# 2024-04-23T16:25:46.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209d7953:::benchmark_data_smithi188_89276_object101072:head# 2024-04-23T16:25:46.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209d9848:::benchmark_data_smithi188_89276_object123100:head# 2024-04-23T16:25:46.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209db7fb:::benchmark_data_smithi188_89276_object100830:head# 2024-04-23T16:25:46.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209e6103:::benchmark_data_smithi188_89276_object89764:head# 2024-04-23T16:25:46.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209ebde4:::benchmark_data_smithi188_89276_object90496:head# 2024-04-23T16:25:46.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:209f2613:::benchmark_data_smithi188_89276_object152176:head# 2024-04-23T16:25:46.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20a0242d:::benchmark_data_smithi188_89276_object152412:head# 2024-04-23T16:25:46.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20a157a1:::benchmark_data_smithi188_89276_object166508:head# 2024-04-23T16:25:46.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20a20298:::benchmark_data_smithi188_89276_object149793:head# 2024-04-23T16:25:46.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20a26073:::benchmark_data_smithi188_89276_object129489:head# 2024-04-23T16:25:46.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20a2ca2a:::benchmark_data_smithi188_89276_object90462:head# 2024-04-23T16:25:46.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20a3632f:::benchmark_data_smithi188_89276_object100612:head# 2024-04-23T16:25:46.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20a50dce:::benchmark_data_smithi188_89276_object92401:head# 2024-04-23T16:25:46.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20a58cbe:::benchmark_data_smithi188_89276_object143933:head# 2024-04-23T16:25:46.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20a60bbd:::benchmark_data_smithi188_89276_object90363:head# 2024-04-23T16:25:46.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20a64627:::benchmark_data_smithi188_89276_object115795:head# 2024-04-23T16:25:46.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20a66205:::benchmark_data_smithi188_89276_object156225:head# 2024-04-23T16:25:46.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20a67b1c:::benchmark_data_smithi188_89276_object79728:head# 2024-04-23T16:25:46.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20a764e0:::benchmark_data_smithi188_89276_object132896:head# 2024-04-23T16:25:46.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20a77900:::benchmark_data_smithi188_89276_object116863:head# 2024-04-23T16:25:46.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20a81dfb:::benchmark_data_smithi188_89276_object50788:head# 2024-04-23T16:25:46.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20a922dc:::benchmark_data_smithi188_89276_object161405:head# 2024-04-23T16:25:46.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20a94851:::benchmark_data_smithi188_89276_object134926:head# 2024-04-23T16:25:46.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20a9e980:::benchmark_data_smithi188_89276_object98907:head# 2024-04-23T16:25:46.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ab6bc7:::benchmark_data_smithi188_89276_object133210:head# 2024-04-23T16:25:46.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ab8d77:::benchmark_data_smithi188_89276_object160212:head# 2024-04-23T16:25:46.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20abc2fd:::benchmark_data_smithi188_89276_object142346:head# 2024-04-23T16:25:46.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ac2255:::benchmark_data_smithi188_89276_object128446:head# 2024-04-23T16:25:46.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ad1cdc:::benchmark_data_smithi188_89276_object82729:head# 2024-04-23T16:25:46.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20adc052:::benchmark_data_smithi188_89276_object92307:head# 2024-04-23T16:25:46.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20aee543:::benchmark_data_smithi188_89276_object66823:head# 2024-04-23T16:25:46.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20b04c9b:::benchmark_data_smithi188_89276_object126801:head# 2024-04-23T16:25:46.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20b19f62:::benchmark_data_smithi188_89276_object118502:head# 2024-04-23T16:25:46.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20b2632f:::benchmark_data_smithi188_89276_object102850:head# 2024-04-23T16:25:46.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20b366ca:::benchmark_data_smithi188_89276_object129732:head# 2024-04-23T16:25:46.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20b60981:::benchmark_data_smithi188_89276_object81526:head# 2024-04-23T16:25:46.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20b6eb07:::benchmark_data_smithi188_89276_object164162:head# 2024-04-23T16:25:46.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20b6f822:::benchmark_data_smithi188_89276_object88823:head# 2024-04-23T16:25:46.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20b742a9:::benchmark_data_smithi188_89276_object134565:head# 2024-04-23T16:25:46.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20b7c849:::benchmark_data_smithi188_89276_object114759:head# 2024-04-23T16:25:46.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20b7e620:::benchmark_data_smithi188_89276_object98041:head# 2024-04-23T16:25:46.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20b84085:::benchmark_data_smithi188_89276_object58175:head# 2024-04-23T16:25:46.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20b97dfc:::benchmark_data_smithi188_89276_object144828:head# 2024-04-23T16:25:46.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ba614c:::benchmark_data_smithi188_89276_object138442:head# 2024-04-23T16:25:46.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ba900e:::benchmark_data_smithi188_89276_object102029:head# 2024-04-23T16:25:46.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20bab020:::benchmark_data_smithi188_89276_object145578:head# 2024-04-23T16:25:46.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20baefb4:::benchmark_data_smithi188_89276_object152649:head# 2024-04-23T16:25:46.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20bc9752:::benchmark_data_smithi188_89276_object163354:head# 2024-04-23T16:25:46.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20bcf7bd:::benchmark_data_smithi188_89276_object95046:head# 2024-04-23T16:25:46.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20bdfea3:::benchmark_data_smithi188_89276_object145113:head# 2024-04-23T16:25:46.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20be0175:::benchmark_data_smithi188_89276_object42450:head# 2024-04-23T16:25:46.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20be4076:::benchmark_data_smithi188_89276_object167671:head# 2024-04-23T16:25:46.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20be58ce:::benchmark_data_smithi188_89276_object56965:head# 2024-04-23T16:25:46.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20be788e:::benchmark_data_smithi188_89276_object148199:head# 2024-04-23T16:25:46.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20bef2c2:::benchmark_data_smithi188_89276_object129622:head# 2024-04-23T16:25:46.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20befc1b:::benchmark_data_smithi188_89276_object59290:head# 2024-04-23T16:25:46.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20bf8dba:::benchmark_data_smithi188_89276_object131110:head# 2024-04-23T16:25:46.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20bfe72a:::benchmark_data_smithi188_89276_object94445:head# 2024-04-23T16:25:46.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c079ef:::benchmark_data_smithi188_89276_object124577:head# 2024-04-23T16:25:46.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c0cada:::benchmark_data_smithi188_89276_object98536:head# 2024-04-23T16:25:46.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c0cde5:::benchmark_data_smithi188_89276_object84407:head# 2024-04-23T16:25:46.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c1d434:::benchmark_data_smithi188_89276_object64515:head# 2024-04-23T16:25:46.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c2087d:::benchmark_data_smithi188_89276_object164245:head# 2024-04-23T16:25:46.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c2288c:::benchmark_data_smithi188_89276_object64759:head# 2024-04-23T16:25:46.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c28792:::benchmark_data_smithi188_89276_object125138:head# 2024-04-23T16:25:46.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c2e978:::benchmark_data_smithi188_89276_object91141:head# 2024-04-23T16:25:46.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c34b41:::benchmark_data_smithi188_89276_object133387:head# 2024-04-23T16:25:46.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c37685:::benchmark_data_smithi188_89276_object56492:head# 2024-04-23T16:25:46.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c37a57:::benchmark_data_smithi188_89276_object167620:head# 2024-04-23T16:25:46.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c381cf:::benchmark_data_smithi188_89276_object152315:head# 2024-04-23T16:25:46.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c3e9a2:::benchmark_data_smithi188_89276_object79722:head# 2024-04-23T16:25:46.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c467c6:::benchmark_data_smithi188_89276_object108735:head# 2024-04-23T16:25:46.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c467e7:::benchmark_data_smithi188_89276_object74429:head# 2024-04-23T16:25:46.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c4d231:::benchmark_data_smithi188_89276_object113218:head# 2024-04-23T16:25:46.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c561eb:::benchmark_data_smithi188_89276_object103247:head# 2024-04-23T16:25:46.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c5f68c:::benchmark_data_smithi188_89276_object103458:head# 2024-04-23T16:25:46.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c5fa23:::benchmark_data_smithi188_89276_object134850:head# 2024-04-23T16:25:46.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c66877:::benchmark_data_smithi188_89276_object53911:head# 2024-04-23T16:25:46.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c710fd:::benchmark_data_smithi188_89276_object42899:head# 2024-04-23T16:25:46.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c74041:::benchmark_data_smithi188_89276_object155797:head# 2024-04-23T16:25:46.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c78d63:::benchmark_data_smithi188_89276_object75571:head# 2024-04-23T16:25:46.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c7bfcc:::benchmark_data_smithi188_89276_object156866:head# 2024-04-23T16:25:46.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c85191:::benchmark_data_smithi188_89276_object92425:head# 2024-04-23T16:25:46.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c8827f:::benchmark_data_smithi188_89276_object154179:head# 2024-04-23T16:25:46.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c89138:::benchmark_data_smithi188_89276_object79008:head# 2024-04-23T16:25:46.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c94c00:::benchmark_data_smithi188_89276_object136077:head# 2024-04-23T16:25:46.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20c9e251:::benchmark_data_smithi188_89276_object132889:head# 2024-04-23T16:25:46.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ca8aac:::benchmark_data_smithi188_89276_object48479:head# 2024-04-23T16:25:46.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ca9caa:::benchmark_data_smithi188_89276_object88973:head# 2024-04-23T16:25:46.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20cb131a:::benchmark_data_smithi188_89276_object83651:head# 2024-04-23T16:25:46.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20cb3874:::benchmark_data_smithi188_89276_object92492:head# 2024-04-23T16:25:46.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20cbacde:::benchmark_data_smithi188_89276_object46747:head# 2024-04-23T16:25:46.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20cbc775:::benchmark_data_smithi188_89276_object113105:head# 2024-04-23T16:25:46.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20cbcbc4:::benchmark_data_smithi188_89276_object162612:head# 2024-04-23T16:25:46.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20cc9a35:::benchmark_data_smithi188_89276_object60535:head# 2024-04-23T16:25:46.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20cca821:::benchmark_data_smithi188_89276_object69846:head# 2024-04-23T16:25:46.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20cd0753:::benchmark_data_smithi188_89276_object168039:head# 2024-04-23T16:25:46.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20cd8150:::benchmark_data_smithi188_89276_object111003:head# 2024-04-23T16:25:46.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20cdec29:::benchmark_data_smithi188_89276_object167592:head# 2024-04-23T16:25:46.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20cf55cf:::benchmark_data_smithi188_89276_object151384:head# 2024-04-23T16:25:46.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20cf70bc:::benchmark_data_smithi188_89276_object161382:head# 2024-04-23T16:25:46.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20d0be5f:::benchmark_data_smithi188_89276_object117201:head# 2024-04-23T16:25:46.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20d1667e:::benchmark_data_smithi188_89276_object156788:head# 2024-04-23T16:25:46.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20d1d7a5:::benchmark_data_smithi188_89276_object89639:head# 2024-04-23T16:25:46.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20d23465:::benchmark_data_smithi188_89276_object161320:head# 2024-04-23T16:25:46.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20d2b3e2:::benchmark_data_smithi188_89276_object97706:head# 2024-04-23T16:25:46.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20d347b2:::benchmark_data_smithi188_89276_object112499:head# 2024-04-23T16:25:46.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20d516ba:::benchmark_data_smithi188_89276_object88430:head# 2024-04-23T16:25:46.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20d546cf:::benchmark_data_smithi188_89276_object42946:head# 2024-04-23T16:25:46.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20d613a5:::benchmark_data_smithi188_89276_object60879:head# 2024-04-23T16:25:46.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20d62699:::benchmark_data_smithi188_89276_object50516:head# 2024-04-23T16:25:46.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20d631a2:::benchmark_data_smithi188_89276_object59702:head# 2024-04-23T16:25:46.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20d7165e:::benchmark_data_smithi188_89276_object167886:head# 2024-04-23T16:25:46.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20d7456c:::benchmark_data_smithi188_89276_object68190:head# 2024-04-23T16:25:46.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20d850cf:::benchmark_data_smithi188_89276_object157017:head# 2024-04-23T16:25:46.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20d86bf3:::benchmark_data_smithi188_89276_object156240:head# 2024-04-23T16:25:46.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20d9f848:::benchmark_data_smithi188_89276_object121961:head# 2024-04-23T16:25:46.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20da2016:::benchmark_data_smithi188_89276_object151500:head# 2024-04-23T16:25:46.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20db3056:::benchmark_data_smithi188_89276_object68603:head# 2024-04-23T16:25:46.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20dbfb39:::benchmark_data_smithi188_89276_object112789:head# 2024-04-23T16:25:46.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20dc5399:::benchmark_data_smithi188_89276_object45751:head# 2024-04-23T16:25:46.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20dce1c2:::benchmark_data_smithi188_89276_object112626:head# 2024-04-23T16:25:46.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ddee33:::benchmark_data_smithi188_89276_object119158:head# 2024-04-23T16:25:46.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ddfd02:::benchmark_data_smithi188_89276_object47635:head# 2024-04-23T16:25:46.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20de53df:::benchmark_data_smithi188_89276_object74816:head# 2024-04-23T16:25:46.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20deb9ef:::benchmark_data_smithi188_89276_object101973:head# 2024-04-23T16:25:46.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20deecc4:::benchmark_data_smithi188_89276_object57710:head# 2024-04-23T16:25:46.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20df29ac:::benchmark_data_smithi188_89276_object119445:head# 2024-04-23T16:25:46.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20df4ca0:::benchmark_data_smithi188_89276_object114254:head# 2024-04-23T16:25:46.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20df6148:::benchmark_data_smithi188_89276_object98624:head# 2024-04-23T16:25:46.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20dfe304:::benchmark_data_smithi188_89276_object79246:head# 2024-04-23T16:25:46.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e0b9cf:::benchmark_data_smithi188_89276_object64860:head# 2024-04-23T16:25:46.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e0c10b:::benchmark_data_smithi188_89276_object129632:head# 2024-04-23T16:25:46.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e0c748:::benchmark_data_smithi188_89276_object141437:head# 2024-04-23T16:25:46.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e0fec5:::benchmark_data_smithi188_89276_object50439:head# 2024-04-23T16:25:46.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e12611:::benchmark_data_smithi188_89276_object134207:head# 2024-04-23T16:25:46.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e1c586:::benchmark_data_smithi188_89276_object117754:head# 2024-04-23T16:25:46.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e24c06:::benchmark_data_smithi188_89276_object131452:head# 2024-04-23T16:25:46.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e29442:::benchmark_data_smithi188_89276_object142925:head# 2024-04-23T16:25:46.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e2a98e:::benchmark_data_smithi188_89276_object158518:head# 2024-04-23T16:25:46.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e2d294:::benchmark_data_smithi188_89276_object158209:head# 2024-04-23T16:25:46.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e3d8dc:::benchmark_data_smithi188_89276_object60323:head# 2024-04-23T16:25:46.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e3dfa3:::benchmark_data_smithi188_89276_object46294:head# 2024-04-23T16:25:46.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e483d0:::benchmark_data_smithi188_89276_object151144:head# 2024-04-23T16:25:46.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e4d03d:::benchmark_data_smithi188_89276_object109776:head# 2024-04-23T16:25:46.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e560f4:::benchmark_data_smithi188_89276_object49159:head# 2024-04-23T16:25:46.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e6b5ac:::benchmark_data_smithi188_89276_object96350:head# 2024-04-23T16:25:46.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e6d9ae:::benchmark_data_smithi188_89276_object167215:head# 2024-04-23T16:25:46.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e757b0:::benchmark_data_smithi188_89276_object50588:head# 2024-04-23T16:25:46.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e780f9:::benchmark_data_smithi188_89276_object84748:head# 2024-04-23T16:25:46.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e7929b:::benchmark_data_smithi188_89276_object61784:head# 2024-04-23T16:25:46.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e84ecd:::benchmark_data_smithi188_89276_object77609:head# 2024-04-23T16:25:46.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20e913c9:::benchmark_data_smithi188_89276_object68939:head# 2024-04-23T16:25:46.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ea8a07:::benchmark_data_smithi188_89276_object62305:head# 2024-04-23T16:25:46.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ec109f:::benchmark_data_smithi188_89276_object166213:head# 2024-04-23T16:25:46.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ed2cce:::benchmark_data_smithi188_89276_object51797:head# 2024-04-23T16:25:46.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ed45a9:::benchmark_data_smithi188_89276_object114339:head# 2024-04-23T16:25:46.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20eec79a:::benchmark_data_smithi188_89276_object68304:head# 2024-04-23T16:25:46.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ef3cbb:::benchmark_data_smithi188_89276_object62532:head# 2024-04-23T16:25:46.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20efdfeb:::benchmark_data_smithi188_89276_object142120:head# 2024-04-23T16:25:46.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20f04fd5:::benchmark_data_smithi188_89276_object69345:head# 2024-04-23T16:25:46.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20f131c3:::benchmark_data_smithi188_89276_object78242:head# 2024-04-23T16:25:46.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20f1471e:::benchmark_data_smithi188_89276_object104299:head# 2024-04-23T16:25:46.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20f2017e:::benchmark_data_smithi188_89276_object148001:head# 2024-04-23T16:25:46.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20f225ca:::benchmark_data_smithi188_89276_object116252:head# 2024-04-23T16:25:46.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20f26719:::benchmark_data_smithi188_89276_object69126:head# 2024-04-23T16:25:46.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20f31d6b:::benchmark_data_smithi188_89276_object121025:head# 2024-04-23T16:25:46.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20f419d2:::benchmark_data_smithi188_89276_object150661:head# 2024-04-23T16:25:46.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20f60fbc:::benchmark_data_smithi188_89276_object114800:head# 2024-04-23T16:25:46.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20f652d6:::benchmark_data_smithi188_89276_object137455:head# 2024-04-23T16:25:46.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20f6ca5f:::benchmark_data_smithi188_89276_object145508:head# 2024-04-23T16:25:46.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20f6ec63:::benchmark_data_smithi188_89276_object132751:head# 2024-04-23T16:25:46.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20f71b8e:::benchmark_data_smithi188_89276_object115224:head# 2024-04-23T16:25:46.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20f78f9f:::benchmark_data_smithi188_89276_object58322:head# 2024-04-23T16:25:46.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20f851ed:::benchmark_data_smithi188_89276_object94244:head# 2024-04-23T16:25:46.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20f899bc:::benchmark_data_smithi188_89276_object47996:head# 2024-04-23T16:25:46.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20f9854f:::benchmark_data_smithi188_89276_object61972:head# 2024-04-23T16:25:46.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20fb7192:::benchmark_data_smithi188_89276_object152790:head# 2024-04-23T16:25:46.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20fbae4f:::benchmark_data_smithi188_89276_object94983:head# 2024-04-23T16:25:46.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20fc0b7e:::benchmark_data_smithi188_89276_object137205:head# 2024-04-23T16:25:46.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20fe1191:::benchmark_data_smithi188_89276_object125618:head# 2024-04-23T16:25:46.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20fe32a3:::benchmark_data_smithi188_89276_object64573:head# 2024-04-23T16:25:46.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20fe7741:::benchmark_data_smithi188_89276_object83849:head# 2024-04-23T16:25:46.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20feb2ce:::benchmark_data_smithi188_89276_object117670:head# 2024-04-23T16:25:46.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20fef6a7:::benchmark_data_smithi188_89276_object159937:head# 2024-04-23T16:25:46.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ff3ff2:::benchmark_data_smithi188_89276_object51080:head# 2024-04-23T16:25:46.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ffce9b:::benchmark_data_smithi188_89276_object65807:head# 2024-04-23T16:25:46.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ffd362:::benchmark_data_smithi188_89276_object120242:head# 2024-04-23T16:25:46.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ffd60f:::benchmark_data_smithi188_89276_object98627:head# 2024-04-23T16:25:46.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:20ffdd33:::benchmark_data_smithi188_89276_object143445:head# 2024-04-23T16:25:46.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2101aefa:::benchmark_data_smithi188_89276_object96990:head# 2024-04-23T16:25:46.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2102086e:::benchmark_data_smithi188_89276_object87358:head# 2024-04-23T16:25:46.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2102451e:::benchmark_data_smithi188_89276_object100471:head# 2024-04-23T16:25:46.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2102524b:::benchmark_data_smithi188_89276_object167178:head# 2024-04-23T16:25:46.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21026180:::benchmark_data_smithi188_89276_object164368:head# 2024-04-23T16:25:46.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:210280d6:::benchmark_data_smithi188_89276_object91107:head# 2024-04-23T16:25:46.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2102b47a:::benchmark_data_smithi188_89276_object136601:head# 2024-04-23T16:25:46.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21041e2f:::benchmark_data_smithi188_89276_object49112:head# 2024-04-23T16:25:46.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21045516:::benchmark_data_smithi188_89276_object153013:head# 2024-04-23T16:25:46.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21049bd5:::benchmark_data_smithi188_89276_object163538:head# 2024-04-23T16:25:46.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21058204:::benchmark_data_smithi188_89276_object124625:head# 2024-04-23T16:25:46.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21071ddf:::benchmark_data_smithi188_89276_object107369:head# 2024-04-23T16:25:46.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:210829a4:::benchmark_data_smithi188_89276_object123661:head# 2024-04-23T16:25:46.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:210a884d:::benchmark_data_smithi188_89276_object168562:head# 2024-04-23T16:25:46.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:210b2b65:::benchmark_data_smithi188_89276_object151836:head# 2024-04-23T16:25:46.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:210ba3d7:::benchmark_data_smithi188_89276_object100543:head# 2024-04-23T16:25:46.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:210bb71c:::benchmark_data_smithi188_89276_object48502:head# 2024-04-23T16:25:46.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:210bf7e1:::benchmark_data_smithi188_89276_object95969:head# 2024-04-23T16:25:46.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:210c19eb:::benchmark_data_smithi188_89276_object120532:head# 2024-04-23T16:25:46.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:210c1d29:::benchmark_data_smithi188_89276_object117994:head# 2024-04-23T16:25:46.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:210c6d98:::benchmark_data_smithi188_89276_object144926:head# 2024-04-23T16:25:46.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:210ca96e:::benchmark_data_smithi188_89276_object166223:head# 2024-04-23T16:25:46.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:210d5827:::benchmark_data_smithi188_89276_object69551:head# 2024-04-23T16:25:46.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:210ec945:::benchmark_data_smithi188_89276_object155263:head# 2024-04-23T16:25:46.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:210fa7d1:::benchmark_data_smithi188_89276_object160123:head# 2024-04-23T16:25:46.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:210fe7b0:::benchmark_data_smithi188_89276_object127439:head# 2024-04-23T16:25:46.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211016f3:::benchmark_data_smithi188_89276_object107673:head# 2024-04-23T16:25:46.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21105f1c:::benchmark_data_smithi188_89276_object109121:head# 2024-04-23T16:25:46.322 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2110f25a:::benchmark_data_smithi188_89276_object92363:head# 2024-04-23T16:25:46.322 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21119abb:::benchmark_data_smithi188_89276_object128432:head# 2024-04-23T16:25:46.322 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21119ec5:::benchmark_data_smithi188_89276_object136130:head# 2024-04-23T16:25:46.322 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2111bfaa:::benchmark_data_smithi188_89276_object60997:head# 2024-04-23T16:25:46.322 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2113c48f:::benchmark_data_smithi188_89276_object152262:head# 2024-04-23T16:25:46.322 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21140cec:::benchmark_data_smithi188_89276_object65916:head# 2024-04-23T16:25:46.322 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211488d4:::benchmark_data_smithi188_89276_object59982:head# 2024-04-23T16:25:46.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2114e461:::benchmark_data_smithi188_89276_object117839:head# 2024-04-23T16:25:46.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21167921:::benchmark_data_smithi188_89276_object73921:head# 2024-04-23T16:25:46.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2116bac6:::benchmark_data_smithi188_89276_object87861:head# 2024-04-23T16:25:46.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2116f831:::benchmark_data_smithi188_89276_object55019:head# 2024-04-23T16:25:46.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2117eea0:::benchmark_data_smithi188_89276_object108779:head# 2024-04-23T16:25:46.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2118e7c1:::benchmark_data_smithi188_89276_object120838:head# 2024-04-23T16:25:46.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.325 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2118f84d:::benchmark_data_smithi188_89276_object111535:head# 2024-04-23T16:25:46.325 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.325 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211901f0:::benchmark_data_smithi188_89276_object93214:head# 2024-04-23T16:25:46.325 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.325 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21192fb4:::benchmark_data_smithi188_89276_object119483:head# 2024-04-23T16:25:46.325 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.325 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21198f0d:::benchmark_data_smithi188_89276_object67590:head# 2024-04-23T16:25:46.325 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.325 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2119af83:::benchmark_data_smithi188_89276_object135003:head# 2024-04-23T16:25:46.325 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.325 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211a4c32:::benchmark_data_smithi188_89276_object134686:head# 2024-04-23T16:25:46.326 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.326 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211a8489:::benchmark_data_smithi188_89276_object155281:head# 2024-04-23T16:25:46.326 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.326 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211aa69c:::benchmark_data_smithi188_89276_object76889:head# 2024-04-23T16:25:46.326 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.326 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211acdee:::benchmark_data_smithi188_89276_object100417:head# 2024-04-23T16:25:46.326 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.326 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211adcde:::benchmark_data_smithi188_89276_object96650:head# 2024-04-23T16:25:46.326 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.326 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211b06bd:::benchmark_data_smithi188_89276_object65575:head# 2024-04-23T16:25:46.326 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.326 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211b983f:::benchmark_data_smithi188_89276_object109960:head# 2024-04-23T16:25:46.326 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.326 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211c7c20:::benchmark_data_smithi188_89276_object53806:head# 2024-04-23T16:25:46.327 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.327 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211c81cc:::benchmark_data_smithi188_89276_object47363:head# 2024-04-23T16:25:46.327 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.327 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211cb2cd:::benchmark_data_smithi188_89276_object103557:head# 2024-04-23T16:25:46.327 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.327 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211d0c19:::benchmark_data_smithi188_89276_object114664:head# 2024-04-23T16:25:46.327 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.327 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211da91e:::benchmark_data_smithi188_89276_object152510:head# 2024-04-23T16:25:46.327 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.327 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211ef7af:::benchmark_data_smithi188_89276_object109160:head# 2024-04-23T16:25:46.327 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.327 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211f4929:::benchmark_data_smithi188_89276_object160569:head# 2024-04-23T16:25:46.327 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.327 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211f55f2:::benchmark_data_smithi188_89276_object135846:head# 2024-04-23T16:25:46.328 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.328 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211f5e55:::benchmark_data_smithi188_89276_object69648:head# 2024-04-23T16:25:46.328 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.328 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:211f6d72:::benchmark_data_smithi188_89276_object111482:head# 2024-04-23T16:25:46.328 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.328 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21211b0a:::benchmark_data_smithi188_89276_object136649:head# 2024-04-23T16:25:46.328 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.328 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2122aaa6:::benchmark_data_smithi188_89276_object40699:head# 2024-04-23T16:25:46.328 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.328 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2122afaf:::benchmark_data_smithi188_89276_object82108:head# 2024-04-23T16:25:46.328 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.328 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:212444d5:::benchmark_data_smithi188_89276_object58264:head# 2024-04-23T16:25:46.328 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.328 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2124935d:::benchmark_data_smithi188_89276_object107497:head# 2024-04-23T16:25:46.328 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2124dfce:::benchmark_data_smithi188_89276_object48202:head# 2024-04-23T16:25:46.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:212612e0:::benchmark_data_smithi188_89276_object54151:head# 2024-04-23T16:25:46.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21265009:::benchmark_data_smithi188_89276_object110461:head# 2024-04-23T16:25:46.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2126ff35:::benchmark_data_smithi188_89276_object88803:head# 2024-04-23T16:25:46.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:212881e5:::benchmark_data_smithi188_89276_object66841:head# 2024-04-23T16:25:46.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:212a2aea:::benchmark_data_smithi188_89276_object79009:head# 2024-04-23T16:25:46.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:212aa881:::benchmark_data_smithi188_89276_object160627:head# 2024-04-23T16:25:46.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:212afdb0:::benchmark_data_smithi188_89276_object44030:head# 2024-04-23T16:25:46.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:212bd85a:::benchmark_data_smithi188_89276_object156017:head# 2024-04-23T16:25:46.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:212e263e:::benchmark_data_smithi188_89276_object89097:head# 2024-04-23T16:25:46.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:212eaf5c:::benchmark_data_smithi188_89276_object70334:head# 2024-04-23T16:25:46.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:212fdb63:::benchmark_data_smithi188_89276_object145184:head# 2024-04-23T16:25:46.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:212ffcac:::benchmark_data_smithi188_89276_object101466:head# 2024-04-23T16:25:46.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21304437:::benchmark_data_smithi188_89276_object138204:head# 2024-04-23T16:25:46.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2130e7ee:::benchmark_data_smithi188_89276_object51388:head# 2024-04-23T16:25:46.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:213122a1:::benchmark_data_smithi188_89276_object51761:head# 2024-04-23T16:25:46.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21322365:::benchmark_data_smithi188_89276_object61074:head# 2024-04-23T16:25:46.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21324586:::benchmark_data_smithi188_89276_object168128:head# 2024-04-23T16:25:46.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2132d1cd:::benchmark_data_smithi188_89276_object75951:head# 2024-04-23T16:25:46.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2132d9da:::benchmark_data_smithi188_89276_object163028:head# 2024-04-23T16:25:46.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21336d8f:::benchmark_data_smithi188_89276_object137100:head# 2024-04-23T16:25:46.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21339d38:::benchmark_data_smithi188_89276_object165692:head# 2024-04-23T16:25:46.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2133ab6a:::benchmark_data_smithi188_89276_object85311:head# 2024-04-23T16:25:46.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2134c85d:::benchmark_data_smithi188_89276_object53004:head# 2024-04-23T16:25:46.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2135922b:::benchmark_data_smithi188_89276_object124659:head# 2024-04-23T16:25:46.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2135ad8d:::benchmark_data_smithi188_89276_object151021:head# 2024-04-23T16:25:46.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2136c5a4:::benchmark_data_smithi188_89276_object140189:head# 2024-04-23T16:25:46.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2136f55c:::benchmark_data_smithi188_89276_object87336:head# 2024-04-23T16:25:46.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2137b053:::benchmark_data_smithi188_89276_object46120:head# 2024-04-23T16:25:46.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:213895b1:::benchmark_data_smithi188_89276_object56261:head# 2024-04-23T16:25:46.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21396424:::benchmark_data_smithi188_89276_object66379:head# 2024-04-23T16:25:46.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:213b3859:::benchmark_data_smithi188_89276_object141455:head# 2024-04-23T16:25:46.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:213bce3a:::benchmark_data_smithi188_89276_object87034:head# 2024-04-23T16:25:46.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:213bd1a8:::benchmark_data_smithi188_89276_object80353:head# 2024-04-23T16:25:46.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:213c1082:::benchmark_data_smithi188_89276_object142164:head# 2024-04-23T16:25:46.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:213c1f97:::benchmark_data_smithi188_89276_object159405:head# 2024-04-23T16:25:46.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:213c7d40:::benchmark_data_smithi188_89276_object154145:head# 2024-04-23T16:25:46.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:213c9eed:::benchmark_data_smithi188_89276_object60691:head# 2024-04-23T16:25:46.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:213d1596:::benchmark_data_smithi188_89276_object61936:head# 2024-04-23T16:25:46.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:213d6390:::benchmark_data_smithi188_89276_object149943:head# 2024-04-23T16:25:46.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:213dbd8e:::benchmark_data_smithi188_89276_object160524:head# 2024-04-23T16:25:46.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:213dd507:::benchmark_data_smithi188_89276_object103583:head# 2024-04-23T16:25:46.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:213f0969:::benchmark_data_smithi188_89276_object141527:head# 2024-04-23T16:25:46.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:213f2029:::benchmark_data_smithi188_89276_object76688:head# 2024-04-23T16:25:46.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:213f4e6e:::benchmark_data_smithi188_89276_object87287:head# 2024-04-23T16:25:46.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:213fe350:::benchmark_data_smithi188_89276_object93278:head# 2024-04-23T16:25:46.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:214088e4:::benchmark_data_smithi188_89276_object69138:head# 2024-04-23T16:25:46.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:214094a1:::benchmark_data_smithi188_89276_object80402:head# 2024-04-23T16:25:46.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2140d5a9:::benchmark_data_smithi188_89276_object137017:head# 2024-04-23T16:25:46.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2142d3a7:::benchmark_data_smithi188_89276_object164160:head# 2024-04-23T16:25:46.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2142eb09:::benchmark_data_smithi188_89276_object90444:head# 2024-04-23T16:25:46.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2142eb8b:::benchmark_data_smithi188_89276_object120889:head# 2024-04-23T16:25:46.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2143aa67:::benchmark_data_smithi188_89276_object80441:head# 2024-04-23T16:25:46.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2143b2be:::benchmark_data_smithi188_89276_object119970:head# 2024-04-23T16:25:46.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21445b0f:::benchmark_data_smithi188_89276_object76382:head# 2024-04-23T16:25:46.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2144f12f:::benchmark_data_smithi188_89276_object131719:head# 2024-04-23T16:25:46.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2145e8a5:::benchmark_data_smithi188_89276_object149259:head# 2024-04-23T16:25:46.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2146b44c:::benchmark_data_smithi188_89276_object75178:head# 2024-04-23T16:25:46.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2146d43c:::benchmark_data_smithi188_89276_object61738:head# 2024-04-23T16:25:46.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2147097b:::benchmark_data_smithi188_89276_object70144:head# 2024-04-23T16:25:46.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21475096:::benchmark_data_smithi188_89276_object51686:head# 2024-04-23T16:25:46.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21475603:::benchmark_data_smithi188_89276_object49286:head# 2024-04-23T16:25:46.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:214790bc:::benchmark_data_smithi188_89276_object158627:head# 2024-04-23T16:25:46.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2147dfe1:::benchmark_data_smithi188_89276_object91586:head# 2024-04-23T16:25:46.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21480acf:::benchmark_data_smithi188_89276_object116596:head# 2024-04-23T16:25:46.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21490578:::benchmark_data_smithi188_89276_object124220:head# 2024-04-23T16:25:46.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:214985fe:::benchmark_data_smithi188_89276_object66106:head# 2024-04-23T16:25:46.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21499f6b:::benchmark_data_smithi188_89276_object109586:head# 2024-04-23T16:25:46.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:214a1620:::benchmark_data_smithi188_89276_object98514:head# 2024-04-23T16:25:46.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:214bf624:::benchmark_data_smithi188_89276_object48205:head# 2024-04-23T16:25:46.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:214c1ee6:::benchmark_data_smithi188_89276_object98646:head# 2024-04-23T16:25:46.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:214c8954:::benchmark_data_smithi188_89276_object62057:head# 2024-04-23T16:25:46.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:214d0925:::benchmark_data_smithi188_89276_object69513:head# 2024-04-23T16:25:46.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:214ea62e:::benchmark_data_smithi188_89276_object85399:head# 2024-04-23T16:25:46.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:215025a9:::benchmark_data_smithi188_89276_object74761:head# 2024-04-23T16:25:46.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21517b6e:::benchmark_data_smithi188_89276_object47953:head# 2024-04-23T16:25:46.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2151a1d0:::benchmark_data_smithi188_89276_object99131:head# 2024-04-23T16:25:46.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2152dec4:::benchmark_data_smithi188_89276_object137282:head# 2024-04-23T16:25:46.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:215323d5:::benchmark_data_smithi188_89276_object149901:head# 2024-04-23T16:25:46.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2153ca5a:::benchmark_data_smithi188_89276_object110220:head# 2024-04-23T16:25:46.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2153eb68:::benchmark_data_smithi188_89276_object65438:head# 2024-04-23T16:25:46.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2153f232:::benchmark_data_smithi188_89276_object137973:head# 2024-04-23T16:25:46.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21543fa7:::benchmark_data_smithi188_89276_object60786:head# 2024-04-23T16:25:46.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2154a894:::benchmark_data_smithi188_89276_object77917:head# 2024-04-23T16:25:46.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2155b780:::benchmark_data_smithi188_89276_object131880:head# 2024-04-23T16:25:46.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:215692ca:::benchmark_data_smithi188_89276_object126203:head# 2024-04-23T16:25:46.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21572d24:::benchmark_data_smithi188_89276_object147428:head# 2024-04-23T16:25:46.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2157e056:::benchmark_data_smithi188_89276_object153733:head# 2024-04-23T16:25:46.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21582235:::benchmark_data_smithi188_89276_object54893:head# 2024-04-23T16:25:46.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:215882a6:::benchmark_data_smithi188_89276_object104912:head# 2024-04-23T16:25:46.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2158bf2e:::benchmark_data_smithi188_89276_object120268:head# 2024-04-23T16:25:46.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2158de1e:::benchmark_data_smithi188_89276_object122883:head# 2024-04-23T16:25:46.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21598cbf:::benchmark_data_smithi188_89276_object50376:head# 2024-04-23T16:25:46.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:215a1de9:::benchmark_data_smithi188_89276_object76159:head# 2024-04-23T16:25:46.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:215ab23e:::benchmark_data_smithi188_89276_object137747:head# 2024-04-23T16:25:46.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:215ad4b8:::benchmark_data_smithi188_89276_object135606:head# 2024-04-23T16:25:46.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:215db7c7:::benchmark_data_smithi188_89276_object48172:head# 2024-04-23T16:25:46.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:215dfc07:::benchmark_data_smithi188_89276_object94870:head# 2024-04-23T16:25:46.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:215ea03f:::benchmark_data_smithi188_89276_object119821:head# 2024-04-23T16:25:46.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:215ea05f:::benchmark_data_smithi188_89276_object155520:head# 2024-04-23T16:25:46.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:215eef75:::benchmark_data_smithi188_89276_object40941:head# 2024-04-23T16:25:46.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:215f3373:::benchmark_data_smithi188_89276_object146438:head# 2024-04-23T16:25:46.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:215fec12:::benchmark_data_smithi188_89276_object45803:head# 2024-04-23T16:25:46.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:216244de:::benchmark_data_smithi188_89276_object94271:head# 2024-04-23T16:25:46.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21636151:::benchmark_data_smithi188_89276_object128369:head# 2024-04-23T16:25:46.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21640eef:::benchmark_data_smithi188_89276_object69819:head# 2024-04-23T16:25:46.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21649fee:::benchmark_data_smithi188_89276_object98958:head# 2024-04-23T16:25:46.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2164a1f7:::benchmark_data_smithi188_89276_object80063:head# 2024-04-23T16:25:46.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2164f367:::benchmark_data_smithi188_89276_object164740:head# 2024-04-23T16:25:46.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2164f8af:::benchmark_data_smithi188_89276_object60516:head# 2024-04-23T16:25:46.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2167a5c1:::benchmark_data_smithi188_89276_object74193:head# 2024-04-23T16:25:46.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:216884a2:::benchmark_data_smithi188_89276_object168375:head# 2024-04-23T16:25:46.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2168d63c:::benchmark_data_smithi188_89276_object151480:head# 2024-04-23T16:25:46.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2169c55c:::benchmark_data_smithi188_89276_object108441:head# 2024-04-23T16:25:46.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2169dc08:::benchmark_data_smithi188_89276_object106364:head# 2024-04-23T16:25:46.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2169f12d:::benchmark_data_smithi188_89276_object107544:head# 2024-04-23T16:25:46.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:216ae7ec:::benchmark_data_smithi188_89276_object101899:head# 2024-04-23T16:25:46.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:216cf3ab:::benchmark_data_smithi188_89276_object105254:head# 2024-04-23T16:25:46.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:216cf43e:::benchmark_data_smithi188_89276_object75732:head# 2024-04-23T16:25:46.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:216d9480:::benchmark_data_smithi188_89276_object139982:head# 2024-04-23T16:25:46.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:216dc32c:::benchmark_data_smithi188_89276_object89659:head# 2024-04-23T16:25:46.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:216e5488:::benchmark_data_smithi188_89276_object64016:head# 2024-04-23T16:25:46.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:216f0ae1:::benchmark_data_smithi188_89276_object71991:head# 2024-04-23T16:25:46.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:216f2320:::benchmark_data_smithi188_89276_object119651:head# 2024-04-23T16:25:46.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21709f30:::benchmark_data_smithi188_89276_object123880:head# 2024-04-23T16:25:46.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21717013:::benchmark_data_smithi188_89276_object141877:head# 2024-04-23T16:25:46.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2171852a:::benchmark_data_smithi188_89276_object111222:head# 2024-04-23T16:25:46.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21718fbd:::benchmark_data_smithi188_89276_object137558:head# 2024-04-23T16:25:46.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2171fb25:::benchmark_data_smithi188_89276_object156188:head# 2024-04-23T16:25:46.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21744de1:::benchmark_data_smithi188_89276_object79483:head# 2024-04-23T16:25:46.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21748d32:::benchmark_data_smithi188_89276_object157118:head# 2024-04-23T16:25:46.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21759571:::benchmark_data_smithi188_89276_object60830:head# 2024-04-23T16:25:46.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21767f41:::benchmark_data_smithi188_89276_object164631:head# 2024-04-23T16:25:46.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2176bbc1:::benchmark_data_smithi188_89276_object100862:head# 2024-04-23T16:25:46.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2177c3a6:::benchmark_data_smithi188_89276_object101696:head# 2024-04-23T16:25:46.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21788859:::benchmark_data_smithi188_89276_object85951:head# 2024-04-23T16:25:46.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:217b1de0:::benchmark_data_smithi188_89276_object73184:head# 2024-04-23T16:25:46.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:217bde4a:::benchmark_data_smithi188_89276_object158773:head# 2024-04-23T16:25:46.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:217be446:::benchmark_data_smithi188_89276_object126806:head# 2024-04-23T16:25:46.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:217c8a98:::benchmark_data_smithi188_89276_object141361:head# 2024-04-23T16:25:46.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:217f5f11:::benchmark_data_smithi188_89276_object105202:head# 2024-04-23T16:25:46.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:217f6426:::benchmark_data_smithi188_89276_object97445:head# 2024-04-23T16:25:46.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2180b4d3:::benchmark_data_smithi188_89276_object71342:head# 2024-04-23T16:25:46.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2181447c:::benchmark_data_smithi188_89276_object146009:head# 2024-04-23T16:25:46.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21820259:::benchmark_data_smithi188_89276_object42439:head# 2024-04-23T16:25:46.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:218204d5:::benchmark_data_smithi188_89276_object47441:head# 2024-04-23T16:25:46.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2182fe80:::benchmark_data_smithi188_89276_object56995:head# 2024-04-23T16:25:46.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:218305c0:::benchmark_data_smithi188_89276_object143351:head# 2024-04-23T16:25:46.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21834ec0:::benchmark_data_smithi188_89276_object77792:head# 2024-04-23T16:25:46.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21835003:::benchmark_data_smithi188_89276_object161146:head# 2024-04-23T16:25:46.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:218367f0:::benchmark_data_smithi188_89276_object152573:head# 2024-04-23T16:25:46.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21836bdb:::benchmark_data_smithi188_89276_object71696:head# 2024-04-23T16:25:46.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:218377a0:::benchmark_data_smithi188_89276_object119246:head# 2024-04-23T16:25:46.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2183b539:::benchmark_data_smithi188_89276_object100762:head# 2024-04-23T16:25:46.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:218401c4:::benchmark_data_smithi188_89276_object112240:head# 2024-04-23T16:25:46.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:218468fc:::benchmark_data_smithi188_89276_object41360:head# 2024-04-23T16:25:46.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21854e7f:::benchmark_data_smithi188_89276_object167208:head# 2024-04-23T16:25:46.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21857bd6:::benchmark_data_smithi188_89276_object133417:head# 2024-04-23T16:25:46.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:218585c2:::benchmark_data_smithi188_89276_object62966:head# 2024-04-23T16:25:46.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2187fc9a:::benchmark_data_smithi188_89276_object107404:head# 2024-04-23T16:25:46.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:218902b2:::benchmark_data_smithi188_89276_object48554:head# 2024-04-23T16:25:46.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:218929e5:::benchmark_data_smithi188_89276_object57323:head# 2024-04-23T16:25:46.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21894923:::benchmark_data_smithi188_89276_object53553:head# 2024-04-23T16:25:46.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:218a6a5d:::benchmark_data_smithi188_89276_object95695:head# 2024-04-23T16:25:46.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:218b81c6:::benchmark_data_smithi188_89276_object83171:head# 2024-04-23T16:25:46.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:218bb7eb:::benchmark_data_smithi188_89276_object137846:head# 2024-04-23T16:25:46.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:218da731:::benchmark_data_smithi188_89276_object86092:head# 2024-04-23T16:25:46.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:218e9e9d:::benchmark_data_smithi188_89276_object87350:head# 2024-04-23T16:25:46.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:218f7ce5:::benchmark_data_smithi188_89276_object64830:head# 2024-04-23T16:25:46.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:218f9f0f:::benchmark_data_smithi188_89276_object49899:head# 2024-04-23T16:25:46.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:218fec61:::benchmark_data_smithi188_89276_object76929:head# 2024-04-23T16:25:46.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21909fb7:::benchmark_data_smithi188_89276_object124678:head# 2024-04-23T16:25:46.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21912f63:::benchmark_data_smithi188_89276_object50921:head# 2024-04-23T16:25:46.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2191fe86:::benchmark_data_smithi188_89276_object109346:head# 2024-04-23T16:25:46.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21925072:::benchmark_data_smithi188_89276_object139597:head# 2024-04-23T16:25:46.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21925c97:::benchmark_data_smithi188_89276_object112560:head# 2024-04-23T16:25:46.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21934574:::benchmark_data_smithi188_89276_object84692:head# 2024-04-23T16:25:46.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:219407ed:::benchmark_data_smithi188_89276_object68776:head# 2024-04-23T16:25:46.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21942fda:::benchmark_data_smithi188_89276_object158511:head# 2024-04-23T16:25:46.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21948712:::benchmark_data_smithi188_89276_object161060:head# 2024-04-23T16:25:46.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21959a33:::benchmark_data_smithi188_89276_object159814:head# 2024-04-23T16:25:46.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2195aead:::benchmark_data_smithi188_89276_object131973:head# 2024-04-23T16:25:46.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2196fc9f:::benchmark_data_smithi188_89276_object164055:head# 2024-04-23T16:25:46.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2197328e:::benchmark_data_smithi188_89276_object61526:head# 2024-04-23T16:25:46.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2197aa0b:::benchmark_data_smithi188_89276_object154204:head# 2024-04-23T16:25:46.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:219837b3:::benchmark_data_smithi188_89276_object109609:head# 2024-04-23T16:25:46.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2198714f:::benchmark_data_smithi188_89276_object102742:head# 2024-04-23T16:25:46.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2198ac1d:::benchmark_data_smithi188_89276_object124077:head# 2024-04-23T16:25:46.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2199df06:::benchmark_data_smithi188_89276_object68245:head# 2024-04-23T16:25:46.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2199f656:::benchmark_data_smithi188_89276_object41112:head# 2024-04-23T16:25:46.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:219a37e2:::benchmark_data_smithi188_89276_object161080:head# 2024-04-23T16:25:46.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:219a3f12:::benchmark_data_smithi188_89276_object128163:head# 2024-04-23T16:25:46.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:219a5f37:::benchmark_data_smithi188_89276_object76727:head# 2024-04-23T16:25:46.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:219b1a3c:::benchmark_data_smithi188_89276_object99018:head# 2024-04-23T16:25:46.378 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.378 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:219b85f6:::benchmark_data_smithi188_89276_object157552:head# 2024-04-23T16:25:46.378 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.378 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:219bb400:::benchmark_data_smithi188_89276_object128526:head# 2024-04-23T16:25:46.378 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.378 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:219bc015:::benchmark_data_smithi188_89276_object98677:head# 2024-04-23T16:25:46.378 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.378 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:219c2460:::benchmark_data_smithi188_89276_object84812:head# 2024-04-23T16:25:46.378 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.378 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:219c7b14:::benchmark_data_smithi188_89276_object138987:head# 2024-04-23T16:25:46.378 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.378 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:219cfd28:::benchmark_data_smithi188_89276_object135503:head# 2024-04-23T16:25:46.378 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:219d0d09:::benchmark_data_smithi188_89276_object82298:head# 2024-04-23T16:25:46.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:219d3a84:::benchmark_data_smithi188_89276_object64129:head# 2024-04-23T16:25:46.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:219e3827:::benchmark_data_smithi188_89276_object149104:head# 2024-04-23T16:25:46.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:219e8b3f:::benchmark_data_smithi188_89276_object127930:head# 2024-04-23T16:25:46.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a05989:::benchmark_data_smithi188_89276_object114700:head# 2024-04-23T16:25:46.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a068f3:::benchmark_data_smithi188_89276_object98943:head# 2024-04-23T16:25:46.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a08e74:::benchmark_data_smithi188_89276_object119814:head# 2024-04-23T16:25:46.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a12f81:::benchmark_data_smithi188_89276_object126144:head# 2024-04-23T16:25:46.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a13408:::benchmark_data_smithi188_89276_object120777:head# 2024-04-23T16:25:46.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a142a1:::benchmark_data_smithi188_89276_object92201:head# 2024-04-23T16:25:46.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a15771:::benchmark_data_smithi188_89276_object48661:head# 2024-04-23T16:25:46.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a20af7:::benchmark_data_smithi188_89276_object45502:head# 2024-04-23T16:25:46.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a3712b:::benchmark_data_smithi188_89276_object120068:head# 2024-04-23T16:25:46.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a371ce:::benchmark_data_smithi188_89276_object129899:head# 2024-04-23T16:25:46.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a3a37a:::benchmark_data_smithi188_89276_object90100:head# 2024-04-23T16:25:46.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a522bd:::benchmark_data_smithi188_89276_object150847:head# 2024-04-23T16:25:46.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a608d6:::benchmark_data_smithi188_89276_object78634:head# 2024-04-23T16:25:46.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a643a6:::benchmark_data_smithi188_89276_object139501:head# 2024-04-23T16:25:46.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a7184d:::benchmark_data_smithi188_89276_object162963:head# 2024-04-23T16:25:46.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a805ad:::benchmark_data_smithi188_89276_object143463:head# 2024-04-23T16:25:46.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a84149:::benchmark_data_smithi188_89276_object82644:head# 2024-04-23T16:25:46.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a9286d:::benchmark_data_smithi188_89276_object94988:head# 2024-04-23T16:25:46.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a95528:::benchmark_data_smithi188_89276_object104204:head# 2024-04-23T16:25:46.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a9a3fb:::benchmark_data_smithi188_89276_object111515:head# 2024-04-23T16:25:46.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21a9c3b1:::benchmark_data_smithi188_89276_object117242:head# 2024-04-23T16:25:46.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21aa2371:::benchmark_data_smithi188_89276_object109771:head# 2024-04-23T16:25:46.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21aaea37:::benchmark_data_smithi188_89276_object149695:head# 2024-04-23T16:25:46.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21ab314d:::benchmark_data_smithi188_89276_object52908:head# 2024-04-23T16:25:46.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21ab6eff:::benchmark_data_smithi188_89276_object68587:head# 2024-04-23T16:25:46.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21ab7a82:::benchmark_data_smithi188_89276_object156785:head# 2024-04-23T16:25:46.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21ac4ef9:::benchmark_data_smithi188_89276_object144844:head# 2024-04-23T16:25:46.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21ac5961:::benchmark_data_smithi188_89276_object42708:head# 2024-04-23T16:25:46.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21ac881d:::benchmark_data_smithi188_89276_object116446:head# 2024-04-23T16:25:46.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21acc077:::benchmark_data_smithi188_89276_object102188:head# 2024-04-23T16:25:46.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21acd607:::benchmark_data_smithi188_89276_object76605:head# 2024-04-23T16:25:46.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21ad09ad:::benchmark_data_smithi188_89276_object113867:head# 2024-04-23T16:25:46.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21ad54af:::benchmark_data_smithi188_89276_object167543:head# 2024-04-23T16:25:46.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21ae0494:::benchmark_data_smithi188_89276_object160927:head# 2024-04-23T16:25:46.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21aeb096:::benchmark_data_smithi188_89276_object135375:head# 2024-04-23T16:25:46.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21aecaa6:::benchmark_data_smithi188_89276_object88667:head# 2024-04-23T16:25:46.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21af31b7:::benchmark_data_smithi188_89276_object162243:head# 2024-04-23T16:25:46.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21afe09d:::benchmark_data_smithi188_89276_object159670:head# 2024-04-23T16:25:46.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b074ff:::benchmark_data_smithi188_89276_object90365:head# 2024-04-23T16:25:46.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b0ac67:::benchmark_data_smithi188_89276_object85298:head# 2024-04-23T16:25:46.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b0ba20:::benchmark_data_smithi188_89276_object157284:head# 2024-04-23T16:25:46.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b1b02e:::benchmark_data_smithi188_89276_object57693:head# 2024-04-23T16:25:46.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b35029:::benchmark_data_smithi188_89276_object153774:head# 2024-04-23T16:25:46.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b35899:::benchmark_data_smithi188_89276_object103853:head# 2024-04-23T16:25:46.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b3f30c:::benchmark_data_smithi188_89276_object121487:head# 2024-04-23T16:25:46.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b41b6f:::benchmark_data_smithi188_89276_object110325:head# 2024-04-23T16:25:46.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b4a6dc:::benchmark_data_smithi188_89276_object149668:head# 2024-04-23T16:25:46.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b4a7b7:::benchmark_data_smithi188_89276_object131552:head# 2024-04-23T16:25:46.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b4f080:::benchmark_data_smithi188_89276_object61552:head# 2024-04-23T16:25:46.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b55a17:::benchmark_data_smithi188_89276_object132829:head# 2024-04-23T16:25:46.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b614d1:::benchmark_data_smithi188_89276_object140037:head# 2024-04-23T16:25:46.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b624cc:::benchmark_data_smithi188_89276_object148457:head# 2024-04-23T16:25:46.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b634ff:::benchmark_data_smithi188_89276_object88282:head# 2024-04-23T16:25:46.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b68660:::benchmark_data_smithi188_89276_object102442:head# 2024-04-23T16:25:46.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b6e8b4:::benchmark_data_smithi188_89276_object41726:head# 2024-04-23T16:25:46.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b76440:::benchmark_data_smithi188_89276_object163239:head# 2024-04-23T16:25:46.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b90e32:::benchmark_data_smithi188_89276_object80235:head# 2024-04-23T16:25:46.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21b99f6e:::benchmark_data_smithi188_89276_object135970:head# 2024-04-23T16:25:46.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21ba5059:::benchmark_data_smithi188_89276_object87737:head# 2024-04-23T16:25:46.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21baf5ee:::benchmark_data_smithi188_89276_object80558:head# 2024-04-23T16:25:46.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21baf8f6:::benchmark_data_smithi188_89276_object84911:head# 2024-04-23T16:25:46.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21bb15a8:::benchmark_data_smithi188_89276_object129878:head# 2024-04-23T16:25:46.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21bb3eb7:::benchmark_data_smithi188_89276_object110330:head# 2024-04-23T16:25:46.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21bbcbfa:::benchmark_data_smithi188_89276_object166747:head# 2024-04-23T16:25:46.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21bd0729:::benchmark_data_smithi188_89276_object60178:head# 2024-04-23T16:25:46.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21bd246a:::benchmark_data_smithi188_89276_object50504:head# 2024-04-23T16:25:46.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21be2625:::benchmark_data_smithi188_89276_object75974:head# 2024-04-23T16:25:46.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21bffdc3:::benchmark_data_smithi188_89276_object85143:head# 2024-04-23T16:25:46.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c03af2:::benchmark_data_smithi188_89276_object141034:head# 2024-04-23T16:25:46.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c05ade:::benchmark_data_smithi188_89276_object61372:head# 2024-04-23T16:25:46.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c0e336:::benchmark_data_smithi188_89276_object63127:head# 2024-04-23T16:25:46.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c0fe99:::benchmark_data_smithi188_89276_object102830:head# 2024-04-23T16:25:46.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c10b58:::benchmark_data_smithi188_89276_object116360:head# 2024-04-23T16:25:46.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c18eba:::benchmark_data_smithi188_89276_object74083:head# 2024-04-23T16:25:46.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c2c8b8:::benchmark_data_smithi188_89276_object133871:head# 2024-04-23T16:25:46.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c2e710:::benchmark_data_smithi188_89276_object71600:head# 2024-04-23T16:25:46.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c36b21:::benchmark_data_smithi188_89276_object122606:head# 2024-04-23T16:25:46.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c3e76e:::benchmark_data_smithi188_89276_object61029:head# 2024-04-23T16:25:46.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c43fda:::benchmark_data_smithi188_89276_object95053:head# 2024-04-23T16:25:46.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c5289a:::benchmark_data_smithi188_89276_object154443:head# 2024-04-23T16:25:46.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c541ae:::benchmark_data_smithi188_89276_object111685:head# 2024-04-23T16:25:46.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c5bf44:::benchmark_data_smithi188_89276_object108954:head# 2024-04-23T16:25:46.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c76d9d:::benchmark_data_smithi188_89276_object67996:head# 2024-04-23T16:25:46.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c7ca3b:::benchmark_data_smithi188_89276_object124204:head# 2024-04-23T16:25:46.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c86848:::benchmark_data_smithi188_89276_object155903:head# 2024-04-23T16:25:46.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c8a9b6:::benchmark_data_smithi188_89276_object133675:head# 2024-04-23T16:25:46.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c92d08:::benchmark_data_smithi188_89276_object156023:head# 2024-04-23T16:25:46.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c9487a:::benchmark_data_smithi188_89276_object103722:head# 2024-04-23T16:25:46.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c9af42:::benchmark_data_smithi188_89276_object93795:head# 2024-04-23T16:25:46.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21c9be9e:::benchmark_data_smithi188_89276_object64560:head# 2024-04-23T16:25:46.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21cad55a:::benchmark_data_smithi188_89276_object115570:head# 2024-04-23T16:25:46.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21cb073a:::benchmark_data_smithi188_89276_object66122:head# 2024-04-23T16:25:46.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21cb7e09:::benchmark_data_smithi188_89276_object89437:head# 2024-04-23T16:25:46.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21cce2ba:::benchmark_data_smithi188_89276_object119064:head# 2024-04-23T16:25:46.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21cd92b5:::benchmark_data_smithi188_89276_object69465:head# 2024-04-23T16:25:46.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21ce54bb:::benchmark_data_smithi188_89276_object74366:head# 2024-04-23T16:25:46.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21cedd80:::benchmark_data_smithi188_89276_object88492:head# 2024-04-23T16:25:46.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21cfb72f:::benchmark_data_smithi188_89276_object66711:head# 2024-04-23T16:25:46.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21d2d6d3:::benchmark_data_smithi188_89276_object82533:head# 2024-04-23T16:25:46.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21d2e3ce:::benchmark_data_smithi188_89276_object74309:head# 2024-04-23T16:25:46.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21d3ae82:::benchmark_data_smithi188_89276_object64945:head# 2024-04-23T16:25:46.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21d3ebde:::benchmark_data_smithi188_89276_object140300:head# 2024-04-23T16:25:46.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21d427e6:::benchmark_data_smithi188_89276_object166139:head# 2024-04-23T16:25:46.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21d579f1:::benchmark_data_smithi188_89276_object65091:head# 2024-04-23T16:25:46.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21d58586:::benchmark_data_smithi188_89276_object78399:head# 2024-04-23T16:25:46.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21d58de5:::benchmark_data_smithi188_89276_object42216:head# 2024-04-23T16:25:46.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21d5b083:::benchmark_data_smithi188_89276_object107314:head# 2024-04-23T16:25:46.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21d5c0e4:::benchmark_data_smithi188_89276_object135097:head# 2024-04-23T16:25:46.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21d6c5df:::benchmark_data_smithi188_89276_object156176:head# 2024-04-23T16:25:46.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21d79991:::benchmark_data_smithi188_89276_object101782:head# 2024-04-23T16:25:46.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21d7bb30:::benchmark_data_smithi188_89276_object159682:head# 2024-04-23T16:25:46.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21d82c6e:::benchmark_data_smithi188_89276_object88323:head# 2024-04-23T16:25:46.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21d8bb70:::benchmark_data_smithi188_89276_object98775:head# 2024-04-23T16:25:46.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21d8e80c:::benchmark_data_smithi188_89276_object134423:head# 2024-04-23T16:25:46.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21d90d63:::benchmark_data_smithi188_89276_object85562:head# 2024-04-23T16:25:46.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21d9d762:::benchmark_data_smithi188_89276_object149713:head# 2024-04-23T16:25:46.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21da7470:::benchmark_data_smithi188_89276_object141483:head# 2024-04-23T16:25:46.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21daaa54:::benchmark_data_smithi188_89276_object57422:head# 2024-04-23T16:25:46.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21db200c:::benchmark_data_smithi188_89276_object114386:head# 2024-04-23T16:25:46.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21db7382:::benchmark_data_smithi188_89276_object156395:head# 2024-04-23T16:25:46.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21db9921:::benchmark_data_smithi188_89276_object144946:head# 2024-04-23T16:25:46.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21dc12e6:::benchmark_data_smithi188_89276_object77284:head# 2024-04-23T16:25:46.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21dc66d9:::benchmark_data_smithi188_89276_object80124:head# 2024-04-23T16:25:46.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21dc711b:::benchmark_data_smithi188_89276_object62818:head# 2024-04-23T16:25:46.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21dd1ec0:::benchmark_data_smithi188_89276_object144981:head# 2024-04-23T16:25:46.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21ddc35b:::benchmark_data_smithi188_89276_object78301:head# 2024-04-23T16:25:46.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21de17b7:::benchmark_data_smithi188_89276_object143809:head# 2024-04-23T16:25:46.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21deb847:::benchmark_data_smithi188_89276_object96287:head# 2024-04-23T16:25:46.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21decba2:::benchmark_data_smithi188_89276_object97344:head# 2024-04-23T16:25:46.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21df81a3:::benchmark_data_smithi188_89276_object54749:head# 2024-04-23T16:25:46.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e00d0f:::benchmark_data_smithi188_89276_object51200:head# 2024-04-23T16:25:46.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e084cb:::benchmark_data_smithi188_89276_object91696:head# 2024-04-23T16:25:46.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e0a692:::benchmark_data_smithi188_89276_object41714:head# 2024-04-23T16:25:46.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e2aa43:::benchmark_data_smithi188_89276_object121200:head# 2024-04-23T16:25:46.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e30673:::benchmark_data_smithi188_89276_object79351:head# 2024-04-23T16:25:46.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e370af:::benchmark_data_smithi188_89276_object119615:head# 2024-04-23T16:25:46.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e391ee:::benchmark_data_smithi188_89276_object84977:head# 2024-04-23T16:25:46.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e3bdff:::benchmark_data_smithi188_89276_object149593:head# 2024-04-23T16:25:46.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e3db35:::benchmark_data_smithi188_89276_object145373:head# 2024-04-23T16:25:46.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e3f8a5:::benchmark_data_smithi188_89276_object155302:head# 2024-04-23T16:25:46.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e4674e:::benchmark_data_smithi188_89276_object71346:head# 2024-04-23T16:25:46.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e47dcc:::benchmark_data_smithi188_89276_object150802:head# 2024-04-23T16:25:46.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e557a4:::benchmark_data_smithi188_89276_object104626:head# 2024-04-23T16:25:46.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e67a18:::benchmark_data_smithi188_89276_object93972:head# 2024-04-23T16:25:46.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e72dd9:::benchmark_data_smithi188_89276_object157384:head# 2024-04-23T16:25:46.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e7ed8d:::benchmark_data_smithi188_89276_object142631:head# 2024-04-23T16:25:46.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e816e4:::benchmark_data_smithi188_89276_object78148:head# 2024-04-23T16:25:46.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e82b37:::benchmark_data_smithi188_89276_object53927:head# 2024-04-23T16:25:46.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e833fb:::benchmark_data_smithi188_89276_object94860:head# 2024-04-23T16:25:46.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e834a3:::benchmark_data_smithi188_89276_object53807:head# 2024-04-23T16:25:46.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e84b15:::benchmark_data_smithi188_89276_object86650:head# 2024-04-23T16:25:46.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e900c1:::benchmark_data_smithi188_89276_object64968:head# 2024-04-23T16:25:46.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21e9d727:::benchmark_data_smithi188_89276_object92132:head# 2024-04-23T16:25:46.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21ea2cea:::benchmark_data_smithi188_89276_object147746:head# 2024-04-23T16:25:46.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21eab547:::benchmark_data_smithi188_89276_object129390:head# 2024-04-23T16:25:46.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21eac169:::benchmark_data_smithi188_89276_object61584:head# 2024-04-23T16:25:46.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21eb363b:::benchmark_data_smithi188_89276_object99685:head# 2024-04-23T16:25:46.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21eb4abc:::benchmark_data_smithi188_89276_object88299:head# 2024-04-23T16:25:46.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21eb5b6a:::benchmark_data_smithi188_89276_object124515:head# 2024-04-23T16:25:46.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21eb6f97:::benchmark_data_smithi188_89276_object118198:head# 2024-04-23T16:25:46.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21eca269:::benchmark_data_smithi188_89276_object47006:head# 2024-04-23T16:25:46.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21ed4d46:::benchmark_data_smithi188_89276_object117702:head# 2024-04-23T16:25:46.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21ed6bd7:::benchmark_data_smithi188_89276_object125397:head# 2024-04-23T16:25:46.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21eefd8b:::benchmark_data_smithi188_89276_object114130:head# 2024-04-23T16:25:46.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21efc615:::benchmark_data_smithi188_89276_object140428:head# 2024-04-23T16:25:46.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21f0fdb6:::benchmark_data_smithi188_89276_object65763:head# 2024-04-23T16:25:46.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21f16bda:::benchmark_data_smithi188_89276_object46520:head# 2024-04-23T16:25:46.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21f16de8:::benchmark_data_smithi188_89276_object95792:head# 2024-04-23T16:25:46.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21f1adfd:::benchmark_data_smithi188_89276_object130063:head# 2024-04-23T16:25:46.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21f1d13b:::benchmark_data_smithi188_89276_object59791:head# 2024-04-23T16:25:46.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21f1ea8d:::benchmark_data_smithi188_89276_object64982:head# 2024-04-23T16:25:46.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21f43f8a:::benchmark_data_smithi188_89276_object102593:head# 2024-04-23T16:25:46.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21f47c18:::benchmark_data_smithi188_89276_object99568:head# 2024-04-23T16:25:46.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21f64797:::benchmark_data_smithi188_89276_object60783:head# 2024-04-23T16:25:46.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21f7e8d1:::benchmark_data_smithi188_89276_object142109:head# 2024-04-23T16:25:46.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21f8beb2:::benchmark_data_smithi188_89276_object148080:head# 2024-04-23T16:25:46.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21f99d86:::benchmark_data_smithi188_89276_object143349:head# 2024-04-23T16:25:46.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21fa0c1f:::benchmark_data_smithi188_89276_object138480:head# 2024-04-23T16:25:46.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21fad6a7:::benchmark_data_smithi188_89276_object150172:head# 2024-04-23T16:25:46.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21fbeda5:::benchmark_data_smithi188_89276_object119456:head# 2024-04-23T16:25:46.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21fbf118:::benchmark_data_smithi188_89276_object41753:head# 2024-04-23T16:25:46.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21fbf75a:::benchmark_data_smithi188_89276_object53610:head# 2024-04-23T16:25:46.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21fc5529:::benchmark_data_smithi188_89276_object49074:head# 2024-04-23T16:25:46.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21fc7f96:::benchmark_data_smithi188_89276_object108726:head# 2024-04-23T16:25:46.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21fd84d3:::benchmark_data_smithi188_89276_object74577:head# 2024-04-23T16:25:46.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21fe2c22:::benchmark_data_smithi188_89276_object59820:head# 2024-04-23T16:25:46.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21fe7715:::benchmark_data_smithi188_89276_object42210:head# 2024-04-23T16:25:46.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:21ff87f5:::benchmark_data_smithi188_89276_object82508:head# 2024-04-23T16:25:46.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2200c0ae:::benchmark_data_smithi188_89276_object62390:head# 2024-04-23T16:25:46.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22011389:::benchmark_data_smithi188_89276_object166275:head# 2024-04-23T16:25:46.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22020f58:::benchmark_data_smithi188_89276_object166430:head# 2024-04-23T16:25:46.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22021374:::benchmark_data_smithi188_89276_object80045:head# 2024-04-23T16:25:46.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22023cf0:::benchmark_data_smithi188_89276_object152263:head# 2024-04-23T16:25:46.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2202e22c:::benchmark_data_smithi188_89276_object164174:head# 2024-04-23T16:25:46.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22033c12:::benchmark_data_smithi188_89276_object95395:head# 2024-04-23T16:25:46.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22033c2c:::benchmark_data_smithi188_89276_object148906:head# 2024-04-23T16:25:46.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:220344b8:::benchmark_data_smithi188_89276_object151601:head# 2024-04-23T16:25:46.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2203be4f:::benchmark_data_smithi188_89276_object92717:head# 2024-04-23T16:25:46.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2203ebda:::benchmark_data_smithi188_89276_object124733:head# 2024-04-23T16:25:46.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22041cba:::benchmark_data_smithi188_89276_object41294:head# 2024-04-23T16:25:46.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2204dc54:::benchmark_data_smithi188_89276_object83996:head# 2024-04-23T16:25:46.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2204f3cc:::benchmark_data_smithi188_89276_object115357:head# 2024-04-23T16:25:46.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2205a870:::benchmark_data_smithi188_89276_object113821:head# 2024-04-23T16:25:46.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:220601ad:::benchmark_data_smithi188_89276_object128085:head# 2024-04-23T16:25:46.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22060aba:::benchmark_data_smithi188_89276_object129407:head# 2024-04-23T16:25:46.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22065137:::benchmark_data_smithi188_89276_object91795:head# 2024-04-23T16:25:46.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2206a7dd:::benchmark_data_smithi188_89276_object116383:head# 2024-04-23T16:25:46.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2206a84a:::benchmark_data_smithi188_89276_object113460:head# 2024-04-23T16:25:46.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2206bc57:::benchmark_data_smithi188_89276_object128026:head# 2024-04-23T16:25:46.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22072825:::benchmark_data_smithi188_89276_object110507:head# 2024-04-23T16:25:46.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22074cd9:::benchmark_data_smithi188_89276_object96091:head# 2024-04-23T16:25:46.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22079556:::benchmark_data_smithi188_89276_object95575:head# 2024-04-23T16:25:46.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:220866ce:::benchmark_data_smithi188_89276_object72695:head# 2024-04-23T16:25:46.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22096fc5:::benchmark_data_smithi188_89276_object112937:head# 2024-04-23T16:25:46.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2209c62f:::benchmark_data_smithi188_89276_object83589:head# 2024-04-23T16:25:46.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:220a872f:::benchmark_data_smithi188_89276_object79811:head# 2024-04-23T16:25:46.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:220b3f14:::benchmark_data_smithi188_89276_object58324:head# 2024-04-23T16:25:46.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:220bba38:::benchmark_data_smithi188_89276_object59194:head# 2024-04-23T16:25:46.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:220c4eae:::benchmark_data_smithi188_89276_object79452:head# 2024-04-23T16:25:46.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:220cbe69:::benchmark_data_smithi188_89276_object65377:head# 2024-04-23T16:25:46.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:220d570a:::benchmark_data_smithi188_89276_object42896:head# 2024-04-23T16:25:46.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:220e2c56:::benchmark_data_smithi188_89276_object143073:head# 2024-04-23T16:25:46.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:220e74f0:::benchmark_data_smithi188_89276_object55270:head# 2024-04-23T16:25:46.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:220eb015:::benchmark_data_smithi188_89276_object92840:head# 2024-04-23T16:25:46.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:220f9282:::benchmark_data_smithi188_89276_object135748:head# 2024-04-23T16:25:46.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:220f9b24:::benchmark_data_smithi188_89276_object99949:head# 2024-04-23T16:25:46.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:220fb141:::benchmark_data_smithi188_89276_object90556:head# 2024-04-23T16:25:46.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22108c39:::benchmark_data_smithi188_89276_object61486:head# 2024-04-23T16:25:46.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22108f82:::benchmark_data_smithi188_89276_object101615:head# 2024-04-23T16:25:46.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2210a126:::benchmark_data_smithi188_89276_object83185:head# 2024-04-23T16:25:46.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2210f1cd:::benchmark_data_smithi188_89276_object130904:head# 2024-04-23T16:25:46.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2211d308:::benchmark_data_smithi188_89276_object72775:head# 2024-04-23T16:25:46.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2212c715:::benchmark_data_smithi188_89276_object88738:head# 2024-04-23T16:25:46.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22148316:::benchmark_data_smithi188_89276_object121891:head# 2024-04-23T16:25:46.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2214c9a3:::benchmark_data_smithi188_89276_object157307:head# 2024-04-23T16:25:46.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2214fde5:::benchmark_data_smithi188_89276_object143592:head# 2024-04-23T16:25:46.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22153cd2:::benchmark_data_smithi188_89276_object116682:head# 2024-04-23T16:25:46.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22154d83:::benchmark_data_smithi188_89276_object151181:head# 2024-04-23T16:25:46.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2215b33d:::benchmark_data_smithi188_89276_object55011:head# 2024-04-23T16:25:46.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22161c9e:::benchmark_data_smithi188_89276_object110273:head# 2024-04-23T16:25:46.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:221622ac:::benchmark_data_smithi188_89276_object83865:head# 2024-04-23T16:25:46.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2216d55d:::benchmark_data_smithi188_89276_object108375:head# 2024-04-23T16:25:46.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:221706ee:::benchmark_data_smithi188_89276_object107936:head# 2024-04-23T16:25:46.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22173fff:::benchmark_data_smithi188_89276_object85938:head# 2024-04-23T16:25:46.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22180bc6:::benchmark_data_smithi188_89276_object41349:head# 2024-04-23T16:25:46.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:221866af:::benchmark_data_smithi188_89276_object107255:head# 2024-04-23T16:25:46.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22188313:::benchmark_data_smithi188_89276_object69830:head# 2024-04-23T16:25:46.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22194e2c:::benchmark_data_smithi188_89276_object166320:head# 2024-04-23T16:25:46.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22198a64:::benchmark_data_smithi188_89276_object69858:head# 2024-04-23T16:25:46.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22198f1f:::benchmark_data_smithi188_89276_object66285:head# 2024-04-23T16:25:46.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2219a358:::benchmark_data_smithi188_89276_object106976:head# 2024-04-23T16:25:46.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:221a8d85:::benchmark_data_smithi188_89276_object92848:head# 2024-04-23T16:25:46.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:221ae4bf:::benchmark_data_smithi188_89276_object60550:head# 2024-04-23T16:25:46.432 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:221b534d:::benchmark_data_smithi188_89276_object67546:head# 2024-04-23T16:25:46.433 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.434 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:221b5abf:::benchmark_data_smithi188_89276_object136385:head# 2024-04-23T16:25:46.434 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.434 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:221ba996:::benchmark_data_smithi188_89276_object69720:head# 2024-04-23T16:25:46.434 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.435 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:221dd396:::benchmark_data_smithi188_89276_object64720:head# 2024-04-23T16:25:46.435 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.435 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:221dede0:::benchmark_data_smithi188_89276_object149777:head# 2024-04-23T16:25:46.435 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.435 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:221dfcba:::benchmark_data_smithi188_89276_object163928:head# 2024-04-23T16:25:46.435 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.435 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:221f1bd8:::benchmark_data_smithi188_89276_object152304:head# 2024-04-23T16:25:46.435 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:221f5410:::benchmark_data_smithi188_89276_object48082:head# 2024-04-23T16:25:46.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22212831:::benchmark_data_smithi188_89276_object131773:head# 2024-04-23T16:25:46.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22214014:::benchmark_data_smithi188_89276_object69938:head# 2024-04-23T16:25:46.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22221905:::benchmark_data_smithi188_89276_object156838:head# 2024-04-23T16:25:46.439 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22224b01:::benchmark_data_smithi188_89276_object53106:head# 2024-04-23T16:25:46.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22227beb:::benchmark_data_smithi188_89276_object88083:head# 2024-04-23T16:25:46.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2222f545:::benchmark_data_smithi188_89276_object154805:head# 2024-04-23T16:25:46.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2223d58b:::benchmark_data_smithi188_89276_object57318:head# 2024-04-23T16:25:46.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2224335d:::benchmark_data_smithi188_89276_object113255:head# 2024-04-23T16:25:46.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2224c8ef:::benchmark_data_smithi188_89276_object113325:head# 2024-04-23T16:25:46.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22255715:::benchmark_data_smithi188_89276_object167953:head# 2024-04-23T16:25:46.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22256cb3:::benchmark_data_smithi188_89276_object120793:head# 2024-04-23T16:25:46.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22257bac:::benchmark_data_smithi188_89276_object153444:head# 2024-04-23T16:25:46.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:222580c7:::benchmark_data_smithi188_89276_object142587:head# 2024-04-23T16:25:46.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.445 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2226df04:::benchmark_data_smithi188_89276_object72740:head# 2024-04-23T16:25:46.445 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.452 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:222760e1:::benchmark_data_smithi188_89276_object134505:head# 2024-04-23T16:25:46.452 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22276866:::benchmark_data_smithi188_89276_object159572:head# 2024-04-23T16:25:46.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2227ae1c:::benchmark_data_smithi188_89276_object149043:head# 2024-04-23T16:25:46.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.454 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2227efd8:::benchmark_data_smithi188_89276_object85602:head# 2024-04-23T16:25:46.454 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.454 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2227f8e0:::benchmark_data_smithi188_89276_object160111:head# 2024-04-23T16:25:46.454 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.454 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22284e4f:::benchmark_data_smithi188_89276_object63627:head# 2024-04-23T16:25:46.454 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22289be6:::benchmark_data_smithi188_89276_object82518:head# 2024-04-23T16:25:46.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2228ce78:::benchmark_data_smithi188_89276_object132852:head# 2024-04-23T16:25:46.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:222a2138:::benchmark_data_smithi188_89276_object164121:head# 2024-04-23T16:25:46.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:222a9d9b:::benchmark_data_smithi188_89276_object119023:head# 2024-04-23T16:25:46.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:222b0fff:::benchmark_data_smithi188_89276_object68273:head# 2024-04-23T16:25:46.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:222b248f:::benchmark_data_smithi188_89276_object159503:head# 2024-04-23T16:25:46.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:222b5884:::benchmark_data_smithi188_89276_object122246:head# 2024-04-23T16:25:46.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:222baf12:::benchmark_data_smithi188_89276_object84711:head# 2024-04-23T16:25:46.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:222bba5a:::benchmark_data_smithi188_89276_object116974:head# 2024-04-23T16:25:46.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:222c064d:::benchmark_data_smithi188_89276_object74722:head# 2024-04-23T16:25:46.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:222c5831:::benchmark_data_smithi188_89276_object102483:head# 2024-04-23T16:25:46.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:222c6908:::benchmark_data_smithi188_89276_object122601:head# 2024-04-23T16:25:46.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:222e48c8:::benchmark_data_smithi188_89276_object53333:head# 2024-04-23T16:25:46.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:222e766f:::benchmark_data_smithi188_89276_object161872:head# 2024-04-23T16:25:46.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:222ebc32:::benchmark_data_smithi188_89276_object144376:head# 2024-04-23T16:25:46.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:222f0b1c:::benchmark_data_smithi188_89276_object148439:head# 2024-04-23T16:25:46.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:222f13a1:::benchmark_data_smithi188_89276_object81072:head# 2024-04-23T16:25:46.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:222f4568:::benchmark_data_smithi188_89276_object122523:head# 2024-04-23T16:25:46.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:222f7bd3:::benchmark_data_smithi188_89276_object127657:head# 2024-04-23T16:25:46.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22301e77:::benchmark_data_smithi188_89276_object145709:head# 2024-04-23T16:25:46.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2230add8:::benchmark_data_smithi188_89276_object149872:head# 2024-04-23T16:25:46.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22310f33:::benchmark_data_smithi188_89276_object82511:head# 2024-04-23T16:25:46.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22311cda:::benchmark_data_smithi188_89276_object159517:head# 2024-04-23T16:25:46.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223202fd:::benchmark_data_smithi188_89276_object104191:head# 2024-04-23T16:25:46.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22333591:::benchmark_data_smithi188_89276_object77633:head# 2024-04-23T16:25:46.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2234668a:::benchmark_data_smithi188_89276_object80266:head# 2024-04-23T16:25:46.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22346e41:::benchmark_data_smithi188_89276_object141905:head# 2024-04-23T16:25:46.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2234b147:::benchmark_data_smithi188_89276_object134603:head# 2024-04-23T16:25:46.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2234e950:::benchmark_data_smithi188_89276_object150832:head# 2024-04-23T16:25:46.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22350e19:::benchmark_data_smithi188_89276_object159207:head# 2024-04-23T16:25:46.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22360d35:::benchmark_data_smithi188_89276_object120158:head# 2024-04-23T16:25:46.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2236422a:::benchmark_data_smithi188_89276_object85309:head# 2024-04-23T16:25:46.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22368356:::benchmark_data_smithi188_89276_object45030:head# 2024-04-23T16:25:46.464 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.464 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22369064:::benchmark_data_smithi188_89276_object125772:head# 2024-04-23T16:25:46.464 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.464 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2236c27a:::benchmark_data_smithi188_89276_object46032:head# 2024-04-23T16:25:46.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2236feb1:::benchmark_data_smithi188_89276_object107061:head# 2024-04-23T16:25:46.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223724e2:::benchmark_data_smithi188_89276_object85171:head# 2024-04-23T16:25:46.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2237a950:::benchmark_data_smithi188_89276_object56856:head# 2024-04-23T16:25:46.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2237f650:::benchmark_data_smithi188_89276_object52062:head# 2024-04-23T16:25:46.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223830c1:::benchmark_data_smithi188_89276_object164805:head# 2024-04-23T16:25:46.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.467 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223866f2:::benchmark_data_smithi188_89276_object90534:head# 2024-04-23T16:25:46.467 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.467 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223960d9:::benchmark_data_smithi188_89276_object163822:head# 2024-04-23T16:25:46.468 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.468 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223975d2:::benchmark_data_smithi188_89276_object89643:head# 2024-04-23T16:25:46.468 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.470 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22397f43:::benchmark_data_smithi188_89276_object131859:head# 2024-04-23T16:25:46.470 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.470 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223a4e38:::benchmark_data_smithi188_89276_object53449:head# 2024-04-23T16:25:46.470 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.470 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223ac9ef:::benchmark_data_smithi188_89276_object99821:head# 2024-04-23T16:25:46.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223b8e02:::benchmark_data_smithi188_89276_object68352:head# 2024-04-23T16:25:46.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223b9be1:::benchmark_data_smithi188_89276_object57203:head# 2024-04-23T16:25:46.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223ba5c0:::benchmark_data_smithi188_89276_object74765:head# 2024-04-23T16:25:46.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223bfbf9:::benchmark_data_smithi188_89276_object48188:head# 2024-04-23T16:25:46.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223ce8ea:::benchmark_data_smithi188_89276_object71011:head# 2024-04-23T16:25:46.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223e5e1b:::benchmark_data_smithi188_89276_object76519:head# 2024-04-23T16:25:46.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223e8195:::benchmark_data_smithi188_89276_object120361:head# 2024-04-23T16:25:46.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223e8f41:::benchmark_data_smithi188_89276_object144504:head# 2024-04-23T16:25:46.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223e9eb5:::benchmark_data_smithi188_89276_object56547:head# 2024-04-23T16:25:46.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223f3a9a:::benchmark_data_smithi188_89276_object156132:head# 2024-04-23T16:25:46.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223f4d61:::benchmark_data_smithi188_89276_object95076:head# 2024-04-23T16:25:46.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223f727e:::benchmark_data_smithi188_89276_object113079:head# 2024-04-23T16:25:46.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223fd8d7:::benchmark_data_smithi188_89276_object67388:head# 2024-04-23T16:25:46.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:223ffdb7:::benchmark_data_smithi188_89276_object110946:head# 2024-04-23T16:25:46.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22408547:::benchmark_data_smithi188_89276_object133066:head# 2024-04-23T16:25:46.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2240e1f6:::benchmark_data_smithi188_89276_object94521:head# 2024-04-23T16:25:46.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22421674:::benchmark_data_smithi188_89276_object134621:head# 2024-04-23T16:25:46.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22423245:::benchmark_data_smithi188_89276_object106690:head# 2024-04-23T16:25:46.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2242438d:::benchmark_data_smithi188_89276_object131759:head# 2024-04-23T16:25:46.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2243563e:::benchmark_data_smithi188_89276_object66305:head# 2024-04-23T16:25:46.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2243dba4:::benchmark_data_smithi188_89276_object154506:head# 2024-04-23T16:25:46.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2243fc83:::benchmark_data_smithi188_89276_object62311:head# 2024-04-23T16:25:46.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22441a94:::benchmark_data_smithi188_89276_object95040:head# 2024-04-23T16:25:46.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:224425cc:::benchmark_data_smithi188_89276_object93440:head# 2024-04-23T16:25:46.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:224580a7:::benchmark_data_smithi188_89276_object132875:head# 2024-04-23T16:25:46.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2245b3c6:::benchmark_data_smithi188_89276_object87028:head# 2024-04-23T16:25:46.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2247747c:::benchmark_data_smithi188_89276_object54470:head# 2024-04-23T16:25:46.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:224886a7:::benchmark_data_smithi188_89276_object98738:head# 2024-04-23T16:25:46.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2248e14d:::benchmark_data_smithi188_89276_object126311:head# 2024-04-23T16:25:46.478 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22495705:::benchmark_data_smithi188_89276_object80669:head# 2024-04-23T16:25:46.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22499f62:::benchmark_data_smithi188_89276_object107372:head# 2024-04-23T16:25:46.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2249a181:::benchmark_data_smithi188_89276_object74967:head# 2024-04-23T16:25:46.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:224a2692:::benchmark_data_smithi188_89276_object125666:head# 2024-04-23T16:25:46.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:224afb72:::benchmark_data_smithi188_89276_object167328:head# 2024-04-23T16:25:46.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:224bb249:::benchmark_data_smithi188_89276_object90297:head# 2024-04-23T16:25:46.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:224bb4e3:::benchmark_data_smithi188_89276_object77620:head# 2024-04-23T16:25:46.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:224c01df:::benchmark_data_smithi188_89276_object66350:head# 2024-04-23T16:25:46.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:224c1fc7:::benchmark_data_smithi188_89276_object120945:head# 2024-04-23T16:25:46.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:224d1439:::benchmark_data_smithi188_89276_object146886:head# 2024-04-23T16:25:46.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:224eb84c:::benchmark_data_smithi188_89276_object165743:head# 2024-04-23T16:25:46.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:224edadc:::benchmark_data_smithi188_89276_object56285:head# 2024-04-23T16:25:46.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22508f17:::benchmark_data_smithi188_89276_object118939:head# 2024-04-23T16:25:46.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2250c80d:::benchmark_data_smithi188_89276_object52655:head# 2024-04-23T16:25:46.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2251304f:::benchmark_data_smithi188_89276_object129748:head# 2024-04-23T16:25:46.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2252f7eb:::benchmark_data_smithi188_89276_object58625:head# 2024-04-23T16:25:46.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2253028e:::benchmark_data_smithi188_89276_object166913:head# 2024-04-23T16:25:46.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22536c62:::benchmark_data_smithi188_89276_object40306:head# 2024-04-23T16:25:46.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2253a0d3:::benchmark_data_smithi188_89276_object112500:head# 2024-04-23T16:25:46.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22543644:::benchmark_data_smithi188_89276_object86718:head# 2024-04-23T16:25:46.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:225441f0:::benchmark_data_smithi188_89276_object116276:head# 2024-04-23T16:25:46.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22548894:::benchmark_data_smithi188_89276_object167763:head# 2024-04-23T16:25:46.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22559cd6:::benchmark_data_smithi188_89276_object94801:head# 2024-04-23T16:25:46.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2255a08a:::benchmark_data_smithi188_89276_object44771:head# 2024-04-23T16:25:46.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2257f756:::benchmark_data_smithi188_89276_object144545:head# 2024-04-23T16:25:46.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2259ada3:::benchmark_data_smithi188_89276_object47887:head# 2024-04-23T16:25:46.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:225b418e:::benchmark_data_smithi188_89276_object92174:head# 2024-04-23T16:25:46.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:225bcc93:::benchmark_data_smithi188_89276_object112241:head# 2024-04-23T16:25:46.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.486 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:225bd28f:::benchmark_data_smithi188_89276_object70758:head# 2024-04-23T16:25:46.486 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.486 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:225be88a:::benchmark_data_smithi188_89276_object100942:head# 2024-04-23T16:25:46.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:225ce2e4:::benchmark_data_smithi188_89276_object164253:head# 2024-04-23T16:25:46.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:225d3cb5:::benchmark_data_smithi188_89276_object128213:head# 2024-04-23T16:25:46.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:225d7411:::benchmark_data_smithi188_89276_object133379:head# 2024-04-23T16:25:46.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.488 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:225db702:::benchmark_data_smithi188_89276_object63308:head# 2024-04-23T16:25:46.488 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.488 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:225f111e:::benchmark_data_smithi188_89276_object152440:head# 2024-04-23T16:25:46.488 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.489 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:225ff0f3:::benchmark_data_smithi188_89276_object45117:head# 2024-04-23T16:25:46.489 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.489 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:226018ee:::benchmark_data_smithi188_89276_object96785:head# 2024-04-23T16:25:46.489 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.489 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22610cab:::benchmark_data_smithi188_89276_object157327:head# 2024-04-23T16:25:46.489 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.489 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22618af1:::benchmark_data_smithi188_89276_object85805:head# 2024-04-23T16:25:46.490 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.490 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2261a4a7:::benchmark_data_smithi188_89276_object104139:head# 2024-04-23T16:25:46.490 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.490 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2261d647:::benchmark_data_smithi188_89276_object55714:head# 2024-04-23T16:25:46.490 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.490 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2262cecb:::benchmark_data_smithi188_89276_object129916:head# 2024-04-23T16:25:46.491 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.491 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2262ef69:::benchmark_data_smithi188_89276_object146144:head# 2024-04-23T16:25:46.491 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.496 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2264cc10:::benchmark_data_smithi188_89276_object87859:head# 2024-04-23T16:25:46.496 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.496 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:226521e5:::benchmark_data_smithi188_89276_object61338:head# 2024-04-23T16:25:46.496 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.496 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:226534c0:::benchmark_data_smithi188_89276_object71916:head# 2024-04-23T16:25:46.496 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2265bb73:::benchmark_data_smithi188_89276_object158058:head# 2024-04-23T16:25:46.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2265c0c7:::benchmark_data_smithi188_89276_object166753:head# 2024-04-23T16:25:46.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2265e0cc:::benchmark_data_smithi188_89276_object119197:head# 2024-04-23T16:25:46.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2268b1d5:::benchmark_data_smithi188_89276_object151532:head# 2024-04-23T16:25:46.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:226a2111:::benchmark_data_smithi188_89276_object78352:head# 2024-04-23T16:25:46.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:226b457f:::benchmark_data_smithi188_89276_object43385:head# 2024-04-23T16:25:46.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:226b46dd:::benchmark_data_smithi188_89276_object105271:head# 2024-04-23T16:25:46.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:226b785a:::benchmark_data_smithi188_89276_object134613:head# 2024-04-23T16:25:46.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:226cad03:::benchmark_data_smithi188_89276_object142280:head# 2024-04-23T16:25:46.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:226d026c:::benchmark_data_smithi188_89276_object135303:head# 2024-04-23T16:25:46.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.518 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:226d2283:::benchmark_data_smithi188_89276_object129195:head# 2024-04-23T16:25:46.518 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.518 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:226dbdf4:::benchmark_data_smithi188_89276_object74048:head# 2024-04-23T16:25:46.518 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:226ebcac:::benchmark_data_smithi188_89276_object147779:head# 2024-04-23T16:25:46.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:226f07c8:::benchmark_data_smithi188_89276_object61075:head# 2024-04-23T16:25:46.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:226f9d32:::benchmark_data_smithi188_89276_object55061:head# 2024-04-23T16:25:46.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:227105ee:::benchmark_data_smithi188_89276_object42131:head# 2024-04-23T16:25:46.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:227114b1:::benchmark_data_smithi188_89276_object53298:head# 2024-04-23T16:25:46.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.522 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2271f075:::benchmark_data_smithi188_89276_object144255:head# 2024-04-23T16:25:46.522 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.522 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22722aa9:::benchmark_data_smithi188_89276_object149795:head# 2024-04-23T16:25:46.522 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22724644:::benchmark_data_smithi188_89276_object51723:head# 2024-04-23T16:25:46.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2272d8e7:::benchmark_data_smithi188_89276_object158556:head# 2024-04-23T16:25:46.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2274035c:::benchmark_data_smithi188_89276_object56536:head# 2024-04-23T16:25:46.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22743da1:::benchmark_data_smithi188_89276_object79770:head# 2024-04-23T16:25:46.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.524 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22746324:::benchmark_data_smithi188_89276_object147248:head# 2024-04-23T16:25:46.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.525 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2274775c:::benchmark_data_smithi188_89276_object47150:head# 2024-04-23T16:25:46.525 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.525 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2274fb22:::benchmark_data_smithi188_89276_object100503:head# 2024-04-23T16:25:46.525 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.525 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22752487:::benchmark_data_smithi188_89276_object118792:head# 2024-04-23T16:25:46.525 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.525 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2275752e:::benchmark_data_smithi188_89276_object125133:head# 2024-04-23T16:25:46.525 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.525 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2276301f:::benchmark_data_smithi188_89276_object153509:head# 2024-04-23T16:25:46.525 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.529 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22763aa5:::benchmark_data_smithi188_89276_object93153:head# 2024-04-23T16:25:46.529 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2277bc60:::benchmark_data_smithi188_89276_object99678:head# 2024-04-23T16:25:46.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2279b473:::benchmark_data_smithi188_89276_object84299:head# 2024-04-23T16:25:46.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2279dfbc:::benchmark_data_smithi188_89276_object50184:head# 2024-04-23T16:25:46.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:227ab73d:::benchmark_data_smithi188_89276_object49674:head# 2024-04-23T16:25:46.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:227abe52:::benchmark_data_smithi188_89276_object46044:head# 2024-04-23T16:25:46.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:227af199:::benchmark_data_smithi188_89276_object147258:head# 2024-04-23T16:25:46.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:227b5f0f:::benchmark_data_smithi188_89276_object153081:head# 2024-04-23T16:25:46.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:227c0ed8:::benchmark_data_smithi188_89276_object57378:head# 2024-04-23T16:25:46.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:227c7af2:::benchmark_data_smithi188_89276_object156135:head# 2024-04-23T16:25:46.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:227d39cc:::benchmark_data_smithi188_89276_object79090:head# 2024-04-23T16:25:46.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:227dabc7:::benchmark_data_smithi188_89276_object114062:head# 2024-04-23T16:25:46.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:227e1f84:::benchmark_data_smithi188_89276_object119117:head# 2024-04-23T16:25:46.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:227f0099:::benchmark_data_smithi188_89276_object118721:head# 2024-04-23T16:25:46.533 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.533 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:227f1b21:::benchmark_data_smithi188_89276_object133797:head# 2024-04-23T16:25:46.533 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.533 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:227fd079:::benchmark_data_smithi188_89276_object138393:head# 2024-04-23T16:25:46.533 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.533 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2281459f:::benchmark_data_smithi188_89276_object71494:head# 2024-04-23T16:25:46.533 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.533 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2281f46c:::benchmark_data_smithi188_89276_object65306:head# 2024-04-23T16:25:46.534 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.546 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22821c70:::benchmark_data_smithi188_89276_object44459:head# 2024-04-23T16:25:46.546 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.546 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22827dbe:::benchmark_data_smithi188_89276_object134584:head# 2024-04-23T16:25:46.546 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.546 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2282ee2b:::benchmark_data_smithi188_89276_object156471:head# 2024-04-23T16:25:46.546 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.546 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2282f3b8:::benchmark_data_smithi188_89276_object71824:head# 2024-04-23T16:25:46.546 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.546 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22831ea5:::benchmark_data_smithi188_89276_object49805:head# 2024-04-23T16:25:46.546 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.546 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22839129:::benchmark_data_smithi188_89276_object67962:head# 2024-04-23T16:25:46.547 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.547 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22847949:::benchmark_data_smithi188_89276_object84331:head# 2024-04-23T16:25:46.547 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.547 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2284d945:::benchmark_data_smithi188_89276_object68803:head# 2024-04-23T16:25:46.547 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.547 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22850009:::benchmark_data_smithi188_89276_object115341:head# 2024-04-23T16:25:46.547 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.547 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22850581:::benchmark_data_smithi188_89276_object151860:head# 2024-04-23T16:25:46.547 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.547 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2285beae:::benchmark_data_smithi188_89276_object111384:head# 2024-04-23T16:25:46.547 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.547 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2287204b:::benchmark_data_smithi188_89276_object124745:head# 2024-04-23T16:25:46.547 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.547 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2287377e:::benchmark_data_smithi188_89276_object53450:head# 2024-04-23T16:25:46.548 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.548 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22885a5b:::benchmark_data_smithi188_89276_object165925:head# 2024-04-23T16:25:46.548 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.548 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2289b159:::benchmark_data_smithi188_89276_object60841:head# 2024-04-23T16:25:46.548 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.548 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2289d1c7:::benchmark_data_smithi188_89276_object148869:head# 2024-04-23T16:25:46.548 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.548 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2289eea1:::benchmark_data_smithi188_89276_object65994:head# 2024-04-23T16:25:46.548 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.548 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:228a4876:::benchmark_data_smithi188_89276_object45140:head# 2024-04-23T16:25:46.548 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.548 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:228ab737:::benchmark_data_smithi188_89276_object132346:head# 2024-04-23T16:25:46.548 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.549 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:228afb99:::benchmark_data_smithi188_89276_object104792:head# 2024-04-23T16:25:46.549 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.549 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:228c0f17:::benchmark_data_smithi188_89276_object116218:head# 2024-04-23T16:25:46.549 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.549 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:228c7683:::benchmark_data_smithi188_89276_object162081:head# 2024-04-23T16:25:46.549 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.549 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:228c8fd5:::benchmark_data_smithi188_89276_object97638:head# 2024-04-23T16:25:46.549 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.549 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:228cd8d0:::benchmark_data_smithi188_89276_object141500:head# 2024-04-23T16:25:46.549 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.549 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:228d48ea:::benchmark_data_smithi188_89276_object168460:head# 2024-04-23T16:25:46.549 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.549 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:228de651:::benchmark_data_smithi188_89276_object57327:head# 2024-04-23T16:25:46.549 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.550 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:228e1072:::benchmark_data_smithi188_89276_object77050:head# 2024-04-23T16:25:46.550 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.550 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:228ed1c8:::benchmark_data_smithi188_89276_object71668:head# 2024-04-23T16:25:46.550 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.550 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:228f1536:::benchmark_data_smithi188_89276_object63171:head# 2024-04-23T16:25:46.550 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.550 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:228f1e0a:::benchmark_data_smithi188_89276_object140494:head# 2024-04-23T16:25:46.550 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.550 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:228f1f75:::benchmark_data_smithi188_89276_object162820:head# 2024-04-23T16:25:46.550 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.550 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:228f2b39:::benchmark_data_smithi188_89276_object141897:head# 2024-04-23T16:25:46.550 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.550 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229005ea:::benchmark_data_smithi188_89276_object167331:head# 2024-04-23T16:25:46.550 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.551 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229058e5:::benchmark_data_smithi188_89276_object149334:head# 2024-04-23T16:25:46.551 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.551 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22910a66:::benchmark_data_smithi188_89276_object68485:head# 2024-04-23T16:25:46.551 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.551 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22912d93:::benchmark_data_smithi188_89276_object51859:head# 2024-04-23T16:25:46.551 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.551 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229139de:::benchmark_data_smithi188_89276_object127390:head# 2024-04-23T16:25:46.551 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.551 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22928ca1:::benchmark_data_smithi188_89276_object129822:head# 2024-04-23T16:25:46.551 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.551 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2292c82c:::benchmark_data_smithi188_89276_object144307:head# 2024-04-23T16:25:46.551 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.551 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22935b6a:::benchmark_data_smithi188_89276_object130476:head# 2024-04-23T16:25:46.551 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.552 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22940feb:::benchmark_data_smithi188_89276_object140445:head# 2024-04-23T16:25:46.552 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.552 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2294110c:::benchmark_data_smithi188_89276_object41350:head# 2024-04-23T16:25:46.552 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.552 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2294cb01:::benchmark_data_smithi188_89276_object75096:head# 2024-04-23T16:25:46.552 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.552 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22961522:::benchmark_data_smithi188_89276_object90162:head# 2024-04-23T16:25:46.552 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.552 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2296cfcd:::benchmark_data_smithi188_89276_object57114:head# 2024-04-23T16:25:46.553 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.553 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229763f4:::benchmark_data_smithi188_89276_object40714:head# 2024-04-23T16:25:46.553 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.553 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2297cfc1:::benchmark_data_smithi188_89276_object101053:head# 2024-04-23T16:25:46.553 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.553 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2297d53a:::benchmark_data_smithi188_89276_object157570:head# 2024-04-23T16:25:46.553 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.553 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229837be:::benchmark_data_smithi188_89276_object148255:head# 2024-04-23T16:25:46.553 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.553 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229841b2:::benchmark_data_smithi188_89276_object91173:head# 2024-04-23T16:25:46.553 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.553 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22993b73:::benchmark_data_smithi188_89276_object141149:head# 2024-04-23T16:25:46.553 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.553 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2299d91b:::benchmark_data_smithi188_89276_object143608:head# 2024-04-23T16:25:46.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229a539b:::benchmark_data_smithi188_89276_object137944:head# 2024-04-23T16:25:46.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229ad9f8:::benchmark_data_smithi188_89276_object46405:head# 2024-04-23T16:25:46.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229b2d2a:::benchmark_data_smithi188_89276_object85218:head# 2024-04-23T16:25:46.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229b5870:::benchmark_data_smithi188_89276_object84701:head# 2024-04-23T16:25:46.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229b801b:::benchmark_data_smithi188_89276_object90314:head# 2024-04-23T16:25:46.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229c036e:::benchmark_data_smithi188_89276_object54525:head# 2024-04-23T16:25:46.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229c1d23:::benchmark_data_smithi188_89276_object70380:head# 2024-04-23T16:25:46.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229c277d:::benchmark_data_smithi188_89276_object92678:head# 2024-04-23T16:25:46.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229caf83:::benchmark_data_smithi188_89276_object156083:head# 2024-04-23T16:25:46.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229d7663:::benchmark_data_smithi188_89276_object139018:head# 2024-04-23T16:25:46.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229db68f:::benchmark_data_smithi188_89276_object65136:head# 2024-04-23T16:25:46.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229e0147:::benchmark_data_smithi188_89276_object47658:head# 2024-04-23T16:25:46.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229e6c5d:::benchmark_data_smithi188_89276_object117956:head# 2024-04-23T16:25:46.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:229fe7e2:::benchmark_data_smithi188_89276_object126574:head# 2024-04-23T16:25:46.556 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.558 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22a17d26:::benchmark_data_smithi188_89276_object80187:head# 2024-04-23T16:25:46.558 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.558 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22a18647:::benchmark_data_smithi188_89276_object93267:head# 2024-04-23T16:25:46.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22a220c2:::benchmark_data_smithi188_89276_object82742:head# 2024-04-23T16:25:46.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22a23630:::benchmark_data_smithi188_89276_object148049:head# 2024-04-23T16:25:46.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22a2707b:::benchmark_data_smithi188_89276_object58119:head# 2024-04-23T16:25:46.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22a3ad35:::benchmark_data_smithi188_89276_object111796:head# 2024-04-23T16:25:46.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22a46c25:::benchmark_data_smithi188_89276_object89897:head# 2024-04-23T16:25:46.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22a5ed34:::benchmark_data_smithi188_89276_object85384:head# 2024-04-23T16:25:46.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22a61cfb:::benchmark_data_smithi188_89276_object64043:head# 2024-04-23T16:25:46.560 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.560 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22a6c2ec:::benchmark_data_smithi188_89276_object91166:head# 2024-04-23T16:25:46.560 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.560 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22a6ecda:::benchmark_data_smithi188_89276_object85413:head# 2024-04-23T16:25:46.560 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.560 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22a6ffb1:::benchmark_data_smithi188_89276_object116757:head# 2024-04-23T16:25:46.560 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.560 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22a756a8:::benchmark_data_smithi188_89276_object143423:head# 2024-04-23T16:25:46.560 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.560 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22a8a5cd:::benchmark_data_smithi188_89276_object122473:head# 2024-04-23T16:25:46.560 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.560 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22a96545:::benchmark_data_smithi188_89276_object40995:head# 2024-04-23T16:25:46.560 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.560 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22a97abe:::benchmark_data_smithi188_89276_object133323:head# 2024-04-23T16:25:46.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22a97c96:::benchmark_data_smithi188_89276_object52693:head# 2024-04-23T16:25:46.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22a9abca:::benchmark_data_smithi188_89276_object99513:head# 2024-04-23T16:25:46.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22aa265e:::benchmark_data_smithi188_89276_object155167:head# 2024-04-23T16:25:46.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22abd993:::benchmark_data_smithi188_89276_object93689:head# 2024-04-23T16:25:46.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22ac2bbf:::benchmark_data_smithi188_89276_object95635:head# 2024-04-23T16:25:46.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22ac88e4:::benchmark_data_smithi188_89276_object168029:head# 2024-04-23T16:25:46.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22ad0399:::benchmark_data_smithi188_89276_object151906:head# 2024-04-23T16:25:46.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22ad3e8f:::benchmark_data_smithi188_89276_object57660:head# 2024-04-23T16:25:46.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22ad4021:::benchmark_data_smithi188_89276_object152505:head# 2024-04-23T16:25:46.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22ad63ab:::benchmark_data_smithi188_89276_object74525:head# 2024-04-23T16:25:46.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22ad6d01:::benchmark_data_smithi188_89276_object41153:head# 2024-04-23T16:25:46.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22adc482:::benchmark_data_smithi188_89276_object130027:head# 2024-04-23T16:25:46.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22ae800c:::benchmark_data_smithi188_89276_object105342:head# 2024-04-23T16:25:46.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.563 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22af178e:::benchmark_data_smithi188_89276_object66067:head# 2024-04-23T16:25:46.563 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.563 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22af9892:::benchmark_data_smithi188_89276_object57308:head# 2024-04-23T16:25:46.563 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.563 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22afc793:::benchmark_data_smithi188_89276_object87534:head# 2024-04-23T16:25:46.563 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.563 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22afd08e:::benchmark_data_smithi188_89276_object155473:head# 2024-04-23T16:25:46.563 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.563 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b0053d:::benchmark_data_smithi188_89276_object69654:head# 2024-04-23T16:25:46.563 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.569 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b0e882:::benchmark_data_smithi188_89276_object77528:head# 2024-04-23T16:25:46.570 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.570 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b187f2:::benchmark_data_smithi188_89276_object136947:head# 2024-04-23T16:25:46.570 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.570 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b19700:::benchmark_data_smithi188_89276_object89127:head# 2024-04-23T16:25:46.570 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.570 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b1fe6a:::benchmark_data_smithi188_89276_object162094:head# 2024-04-23T16:25:46.570 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.570 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b29005:::benchmark_data_smithi188_89276_object57832:head# 2024-04-23T16:25:46.570 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.570 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b29ef2:::benchmark_data_smithi188_89276_object76013:head# 2024-04-23T16:25:46.570 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.570 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b3338e:::benchmark_data_smithi188_89276_object49437:head# 2024-04-23T16:25:46.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b46f79:::benchmark_data_smithi188_89276_object159366:head# 2024-04-23T16:25:46.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b490f2:::benchmark_data_smithi188_89276_object58296:head# 2024-04-23T16:25:46.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b4bcd3:::benchmark_data_smithi188_89276_object45079:head# 2024-04-23T16:25:46.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b4c3c6:::benchmark_data_smithi188_89276_object51674:head# 2024-04-23T16:25:46.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b4e872:::benchmark_data_smithi188_89276_object66492:head# 2024-04-23T16:25:46.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b580a7:::benchmark_data_smithi188_89276_object140339:head# 2024-04-23T16:25:46.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b74407:::benchmark_data_smithi188_89276_object109657:head# 2024-04-23T16:25:46.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b759ca:::benchmark_data_smithi188_89276_object113427:head# 2024-04-23T16:25:46.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b77d45:::benchmark_data_smithi188_89276_object41204:head# 2024-04-23T16:25:46.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b85a2e:::benchmark_data_smithi188_89276_object167457:head# 2024-04-23T16:25:46.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b8cd1d:::benchmark_data_smithi188_89276_object45435:head# 2024-04-23T16:25:46.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b91df2:::benchmark_data_smithi188_89276_object140225:head# 2024-04-23T16:25:46.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22b98de9:::benchmark_data_smithi188_89276_object136783:head# 2024-04-23T16:25:46.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22bab1de:::benchmark_data_smithi188_89276_object94279:head# 2024-04-23T16:25:46.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22bb0dd0:::benchmark_data_smithi188_89276_object165003:head# 2024-04-23T16:25:46.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22bb5466:::benchmark_data_smithi188_89276_object116537:head# 2024-04-23T16:25:46.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22bb5623:::benchmark_data_smithi188_89276_object99760:head# 2024-04-23T16:25:46.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22bb7062:::benchmark_data_smithi188_89276_object71966:head# 2024-04-23T16:25:46.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22bbe4dd:::benchmark_data_smithi188_89276_object43450:head# 2024-04-23T16:25:46.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22bcc234:::benchmark_data_smithi188_89276_object135122:head# 2024-04-23T16:25:46.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22bce655:::benchmark_data_smithi188_89276_object134885:head# 2024-04-23T16:25:46.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22bdd06a:::benchmark_data_smithi188_89276_object121751:head# 2024-04-23T16:25:46.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22bf19d1:::benchmark_data_smithi188_89276_object123009:head# 2024-04-23T16:25:46.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c02441:::benchmark_data_smithi188_89276_object118701:head# 2024-04-23T16:25:46.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c0474b:::benchmark_data_smithi188_89276_object72389:head# 2024-04-23T16:25:46.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c14639:::benchmark_data_smithi188_89276_object77205:head# 2024-04-23T16:25:46.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c1666c:::benchmark_data_smithi188_89276_object84083:head# 2024-04-23T16:25:46.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c1942f:::benchmark_data_smithi188_89276_object86580:head# 2024-04-23T16:25:46.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c1cc76:::benchmark_data_smithi188_89276_object135040:head# 2024-04-23T16:25:46.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c1eacb:::benchmark_data_smithi188_89276_object53432:head# 2024-04-23T16:25:46.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c21000:::benchmark_data_smithi188_89276_object136197:head# 2024-04-23T16:25:46.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c294da:::benchmark_data_smithi188_89276_object64383:head# 2024-04-23T16:25:46.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c296e0:::benchmark_data_smithi188_89276_object97399:head# 2024-04-23T16:25:46.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c2add4:::benchmark_data_smithi188_89276_object130815:head# 2024-04-23T16:25:46.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c2bf05:::benchmark_data_smithi188_89276_object128996:head# 2024-04-23T16:25:46.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.576 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c397fa:::benchmark_data_smithi188_89276_object68313:head# 2024-04-23T16:25:46.576 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.576 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c6206b:::benchmark_data_smithi188_89276_object48007:head# 2024-04-23T16:25:46.576 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.581 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c64117:::benchmark_data_smithi188_89276_object56296:head# 2024-04-23T16:25:46.581 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.581 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c64715:::benchmark_data_smithi188_89276_object123013:head# 2024-04-23T16:25:46.581 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.581 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c651a5:::benchmark_data_smithi188_89276_object62578:head# 2024-04-23T16:25:46.581 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.581 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c738be:::benchmark_data_smithi188_89276_object100228:head# 2024-04-23T16:25:46.581 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c74623:::benchmark_data_smithi188_89276_object58560:head# 2024-04-23T16:25:46.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c75a32:::benchmark_data_smithi188_89276_object57481:head# 2024-04-23T16:25:46.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c78cbc:::benchmark_data_smithi188_89276_object165843:head# 2024-04-23T16:25:46.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c7a146:::benchmark_data_smithi188_89276_object117838:head# 2024-04-23T16:25:46.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c8df72:::benchmark_data_smithi188_89276_object93908:head# 2024-04-23T16:25:46.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c958b8:::benchmark_data_smithi188_89276_object74980:head# 2024-04-23T16:25:46.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c9cd67:::benchmark_data_smithi188_89276_object57073:head# 2024-04-23T16:25:46.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22c9d7d8:::benchmark_data_smithi188_89276_object149562:head# 2024-04-23T16:25:46.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22caec31:::benchmark_data_smithi188_89276_object96724:head# 2024-04-23T16:25:46.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22cb0d18:::benchmark_data_smithi188_89276_object139104:head# 2024-04-23T16:25:46.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22cc2988:::benchmark_data_smithi188_89276_object92198:head# 2024-04-23T16:25:46.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22cc29e0:::benchmark_data_smithi188_89276_object130699:head# 2024-04-23T16:25:46.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22ce273d:::benchmark_data_smithi188_89276_object105727:head# 2024-04-23T16:25:46.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22cf09b5:::benchmark_data_smithi188_89276_object87126:head# 2024-04-23T16:25:46.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.584 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22cf30d7:::benchmark_data_smithi188_89276_object149731:head# 2024-04-23T16:25:46.584 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.584 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22cf90d2:::benchmark_data_smithi188_89276_object44314:head# 2024-04-23T16:25:46.584 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.584 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d01c95:::benchmark_data_smithi188_89276_object86863:head# 2024-04-23T16:25:46.584 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.584 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d0404b:::benchmark_data_smithi188_89276_object166671:head# 2024-04-23T16:25:46.584 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.584 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d094c0:::benchmark_data_smithi188_89276_object163967:head# 2024-04-23T16:25:46.584 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.584 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d10dd9:::benchmark_data_smithi188_89276_object75735:head# 2024-04-23T16:25:46.584 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.584 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d172d6:::benchmark_data_smithi188_89276_object160173:head# 2024-04-23T16:25:46.584 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.585 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d24ccb:::benchmark_data_smithi188_89276_object148533:head# 2024-04-23T16:25:46.585 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.585 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d278ea:::benchmark_data_smithi188_89276_object135002:head# 2024-04-23T16:25:46.585 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.585 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d2bace:::benchmark_data_smithi188_89276_object145204:head# 2024-04-23T16:25:46.585 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.585 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d37981:::benchmark_data_smithi188_89276_object110604:head# 2024-04-23T16:25:46.585 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.585 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d40fe1:::benchmark_data_smithi188_89276_object45951:head# 2024-04-23T16:25:46.585 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.585 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d423fe:::benchmark_data_smithi188_89276_object151135:head# 2024-04-23T16:25:46.585 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.585 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d493ff:::benchmark_data_smithi188_89276_object85581:head# 2024-04-23T16:25:46.585 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.585 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d4acb7:::benchmark_data_smithi188_89276_object123619:head# 2024-04-23T16:25:46.586 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.586 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d4e41d:::benchmark_data_smithi188_89276_object120850:head# 2024-04-23T16:25:46.586 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.586 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d4f4a0:::benchmark_data_smithi188_89276_object143954:head# 2024-04-23T16:25:46.586 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.586 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d54464:::benchmark_data_smithi188_89276_object73509:head# 2024-04-23T16:25:46.586 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.586 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d57609:::benchmark_data_smithi188_89276_object43896:head# 2024-04-23T16:25:46.586 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.586 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d796f4:::benchmark_data_smithi188_89276_object79972:head# 2024-04-23T16:25:46.586 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.586 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d7ddb9:::benchmark_data_smithi188_89276_object64611:head# 2024-04-23T16:25:46.586 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.587 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22d9a135:::benchmark_data_smithi188_89276_object61647:head# 2024-04-23T16:25:46.587 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.587 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22da81d6:::benchmark_data_smithi188_89276_object163362:head# 2024-04-23T16:25:46.587 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.587 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22daa729:::benchmark_data_smithi188_89276_object104900:head# 2024-04-23T16:25:46.587 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.591 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22dbb892:::benchmark_data_smithi188_89276_object125267:head# 2024-04-23T16:25:46.591 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.591 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22dc114b:::benchmark_data_smithi188_89276_object55234:head# 2024-04-23T16:25:46.592 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.592 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22dc4c9e:::benchmark_data_smithi188_89276_object161189:head# 2024-04-23T16:25:46.592 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.592 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22de29c3:::benchmark_data_smithi188_89276_object100232:head# 2024-04-23T16:25:46.592 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.592 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22dedef1:::benchmark_data_smithi188_89276_object43065:head# 2024-04-23T16:25:46.592 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.592 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22dfa878:::benchmark_data_smithi188_89276_object168498:head# 2024-04-23T16:25:46.592 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.592 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22dfd179:::benchmark_data_smithi188_89276_object84837:head# 2024-04-23T16:25:46.592 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.592 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22e0dec1:::benchmark_data_smithi188_89276_object107051:head# 2024-04-23T16:25:46.592 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.593 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22e0e45c:::benchmark_data_smithi188_89276_object138262:head# 2024-04-23T16:25:46.593 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.593 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22e1178a:::benchmark_data_smithi188_89276_object162166:head# 2024-04-23T16:25:46.593 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.593 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22e13288:::benchmark_data_smithi188_89276_object134573:head# 2024-04-23T16:25:46.593 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.593 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22e19f5d:::benchmark_data_smithi188_89276_object83239:head# 2024-04-23T16:25:46.593 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.593 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22e24e73:::benchmark_data_smithi188_89276_object75621:head# 2024-04-23T16:25:46.593 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.593 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22e55dbd:::benchmark_data_smithi188_89276_object156014:head# 2024-04-23T16:25:46.593 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.593 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22e58f60:::benchmark_data_smithi188_89276_object138756:head# 2024-04-23T16:25:46.593 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.593 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22e5c829:::benchmark_data_smithi188_89276_object78405:head# 2024-04-23T16:25:46.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.594 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22e5f75d:::benchmark_data_smithi188_89276_object102240:head# 2024-04-23T16:25:46.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.594 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22e69311:::benchmark_data_smithi188_89276_object109277:head# 2024-04-23T16:25:46.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.594 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22e6b707:::benchmark_data_smithi188_89276_object159948:head# 2024-04-23T16:25:46.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.594 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22e6d610:::benchmark_data_smithi188_89276_object112705:head# 2024-04-23T16:25:46.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.594 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22e7c90b:::benchmark_data_smithi188_89276_object81946:head# 2024-04-23T16:25:46.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22e7ccec:::benchmark_data_smithi188_89276_object115659:head# 2024-04-23T16:25:46.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22e8df44:::benchmark_data_smithi188_89276_object124823:head# 2024-04-23T16:25:46.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22e9b1d1:::benchmark_data_smithi188_89276_object45168:head# 2024-04-23T16:25:46.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22ea5b7a:::benchmark_data_smithi188_89276_object101522:head# 2024-04-23T16:25:46.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22eada34:::benchmark_data_smithi188_89276_object139264:head# 2024-04-23T16:25:46.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22eb4a39:::benchmark_data_smithi188_89276_object154735:head# 2024-04-23T16:25:46.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22eba2f2:::benchmark_data_smithi188_89276_object66721:head# 2024-04-23T16:25:46.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22ebecdd:::benchmark_data_smithi188_89276_object131545:head# 2024-04-23T16:25:46.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22ec01e6:::benchmark_data_smithi188_89276_object146667:head# 2024-04-23T16:25:46.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22ec0836:::benchmark_data_smithi188_89276_object59445:head# 2024-04-23T16:25:46.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22ec1fd1:::benchmark_data_smithi188_89276_object126059:head# 2024-04-23T16:25:46.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22ec98d9:::benchmark_data_smithi188_89276_object115595:head# 2024-04-23T16:25:46.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22edfed8:::benchmark_data_smithi188_89276_object74488:head# 2024-04-23T16:25:46.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22efac7f:::benchmark_data_smithi188_89276_object113179:head# 2024-04-23T16:25:46.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.600 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f07768:::benchmark_data_smithi188_89276_object150358:head# 2024-04-23T16:25:46.600 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.600 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f1beda:::benchmark_data_smithi188_89276_object139440:head# 2024-04-23T16:25:46.600 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.600 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f1ebd1:::benchmark_data_smithi188_89276_object98002:head# 2024-04-23T16:25:46.600 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.600 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f248c7:::benchmark_data_smithi188_89276_object95019:head# 2024-04-23T16:25:46.600 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.600 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f2b29b:::benchmark_data_smithi188_89276_object150711:head# 2024-04-23T16:25:46.600 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f328d9:::benchmark_data_smithi188_89276_object135224:head# 2024-04-23T16:25:46.601 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f3b253:::benchmark_data_smithi188_89276_object135258:head# 2024-04-23T16:25:46.601 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f5a28e:::benchmark_data_smithi188_89276_object75223:head# 2024-04-23T16:25:46.601 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f68ef0:::benchmark_data_smithi188_89276_object94071:head# 2024-04-23T16:25:46.601 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f76059:::benchmark_data_smithi188_89276_object133784:head# 2024-04-23T16:25:46.601 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f85f2e:::benchmark_data_smithi188_89276_object109858:head# 2024-04-23T16:25:46.601 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f86a22:::benchmark_data_smithi188_89276_object106591:head# 2024-04-23T16:25:46.601 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f8acd6:::benchmark_data_smithi188_89276_object120295:head# 2024-04-23T16:25:46.602 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.602 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f8ecdf:::benchmark_data_smithi188_89276_object52276:head# 2024-04-23T16:25:46.602 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.602 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f90ba1:::benchmark_data_smithi188_89276_object83642:head# 2024-04-23T16:25:46.602 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.602 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f922ed:::benchmark_data_smithi188_89276_object134614:head# 2024-04-23T16:25:46.602 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.602 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f937c2:::benchmark_data_smithi188_89276_object165134:head# 2024-04-23T16:25:46.602 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.602 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f9c402:::benchmark_data_smithi188_89276_object112742:head# 2024-04-23T16:25:46.602 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.602 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f9d426:::benchmark_data_smithi188_89276_object88377:head# 2024-04-23T16:25:46.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22f9e515:::benchmark_data_smithi188_89276_object159141:head# 2024-04-23T16:25:46.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22fa98de:::benchmark_data_smithi188_89276_object134767:head# 2024-04-23T16:25:46.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22fc6671:::benchmark_data_smithi188_89276_object77797:head# 2024-04-23T16:25:46.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22fca6f8:::benchmark_data_smithi188_89276_object88162:head# 2024-04-23T16:25:46.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22fd124d:::benchmark_data_smithi188_89276_object87745:head# 2024-04-23T16:25:46.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22fd2740:::benchmark_data_smithi188_89276_object40734:head# 2024-04-23T16:25:46.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22fd3445:::benchmark_data_smithi188_89276_object41590:head# 2024-04-23T16:25:46.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22fd4da6:::benchmark_data_smithi188_89276_object65634:head# 2024-04-23T16:25:46.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.607 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22fd6baf:::benchmark_data_smithi188_89276_object149612:head# 2024-04-23T16:25:46.607 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.607 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22fd933d:::benchmark_data_smithi188_89276_object97971:head# 2024-04-23T16:25:46.607 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22fdabd0:::benchmark_data_smithi188_89276_object132032:head# 2024-04-23T16:25:46.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22fdada7:::benchmark_data_smithi188_89276_object160785:head# 2024-04-23T16:25:46.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22fddf8c:::benchmark_data_smithi188_89276_object124675:head# 2024-04-23T16:25:46.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22fe0f57:::benchmark_data_smithi188_89276_object110731:head# 2024-04-23T16:25:46.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22febc00:::benchmark_data_smithi188_89276_object167222:head# 2024-04-23T16:25:46.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22ff49b0:::benchmark_data_smithi188_89276_object153721:head# 2024-04-23T16:25:46.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22ff68c4:::benchmark_data_smithi188_89276_object145281:head# 2024-04-23T16:25:46.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:22fffbcf:::benchmark_data_smithi188_89276_object152361:head# 2024-04-23T16:25:46.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23001c63:::benchmark_data_smithi188_89276_object51684:head# 2024-04-23T16:25:46.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2300aa64:::benchmark_data_smithi188_89276_object49430:head# 2024-04-23T16:25:46.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2301d852:::benchmark_data_smithi188_89276_object64818:head# 2024-04-23T16:25:46.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2302dfec:::benchmark_data_smithi188_89276_object156833:head# 2024-04-23T16:25:46.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2302e49a:::benchmark_data_smithi188_89276_object85757:head# 2024-04-23T16:25:46.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2303c4bd:::benchmark_data_smithi188_89276_object42314:head# 2024-04-23T16:25:46.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23040e9f:::benchmark_data_smithi188_89276_object124412:head# 2024-04-23T16:25:46.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23043fd2:::benchmark_data_smithi188_89276_object106162:head# 2024-04-23T16:25:46.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23044b7a:::benchmark_data_smithi188_89276_object118369:head# 2024-04-23T16:25:46.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:230586f8:::benchmark_data_smithi188_89276_object132663:head# 2024-04-23T16:25:46.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2305a054:::benchmark_data_smithi188_89276_object77387:head# 2024-04-23T16:25:46.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:230733b7:::benchmark_data_smithi188_89276_object69859:head# 2024-04-23T16:25:46.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23078854:::benchmark_data_smithi188_89276_object143450:head# 2024-04-23T16:25:46.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23079c15:::benchmark_data_smithi188_89276_object67231:head# 2024-04-23T16:25:46.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2307c194:::benchmark_data_smithi188_89276_object60265:head# 2024-04-23T16:25:46.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2308677d:::benchmark_data_smithi188_89276_object113962:head# 2024-04-23T16:25:46.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2309292d:::benchmark_data_smithi188_89276_object86462:head# 2024-04-23T16:25:46.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2309293a:::benchmark_data_smithi188_89276_object116337:head# 2024-04-23T16:25:46.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:230935f2:::benchmark_data_smithi188_89276_object159129:head# 2024-04-23T16:25:46.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2309c1d9:::benchmark_data_smithi188_89276_object68322:head# 2024-04-23T16:25:46.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:230a1ba1:::benchmark_data_smithi188_89276_object51670:head# 2024-04-23T16:25:46.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:230a9d79:::benchmark_data_smithi188_89276_object158237:head# 2024-04-23T16:25:46.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:230adbd4:::benchmark_data_smithi188_89276_object140860:head# 2024-04-23T16:25:46.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:230b3489:::benchmark_data_smithi188_89276_object105040:head# 2024-04-23T16:25:46.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:230cb600:::benchmark_data_smithi188_89276_object93036:head# 2024-04-23T16:25:46.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:230ce0a3:::benchmark_data_smithi188_89276_object139662:head# 2024-04-23T16:25:46.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:230da41b:::benchmark_data_smithi188_89276_object124380:head# 2024-04-23T16:25:46.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:230db8a3:::benchmark_data_smithi188_89276_object119825:head# 2024-04-23T16:25:46.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:230dc7f7:::benchmark_data_smithi188_89276_object80335:head# 2024-04-23T16:25:46.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:230e4b6e:::benchmark_data_smithi188_89276_object72450:head# 2024-04-23T16:25:46.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:230e7c51:::benchmark_data_smithi188_89276_object109570:head# 2024-04-23T16:25:46.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:230ea00e:::benchmark_data_smithi188_89276_object149211:head# 2024-04-23T16:25:46.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:230f5dcc:::benchmark_data_smithi188_89276_object71930:head# 2024-04-23T16:25:46.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:230f93b1:::benchmark_data_smithi188_89276_object61799:head# 2024-04-23T16:25:46.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:230fa28e:::benchmark_data_smithi188_89276_object134572:head# 2024-04-23T16:25:46.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:230fe071:::benchmark_data_smithi188_89276_object49310:head# 2024-04-23T16:25:46.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23103fcb:::benchmark_data_smithi188_89276_object60713:head# 2024-04-23T16:25:46.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23105080:::benchmark_data_smithi188_89276_object47597:head# 2024-04-23T16:25:46.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23106abd:::benchmark_data_smithi188_89276_object52504:head# 2024-04-23T16:25:46.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2310913c:::benchmark_data_smithi188_89276_object118682:head# 2024-04-23T16:25:46.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2310f040:::benchmark_data_smithi188_89276_object87207:head# 2024-04-23T16:25:46.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2312650c:::benchmark_data_smithi188_89276_object62834:head# 2024-04-23T16:25:46.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:231355a8:::benchmark_data_smithi188_89276_object117765:head# 2024-04-23T16:25:46.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.621 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:231395fd:::benchmark_data_smithi188_89276_object55739:head# 2024-04-23T16:25:46.621 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.621 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23151563:::benchmark_data_smithi188_89276_object142182:head# 2024-04-23T16:25:46.621 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.621 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:231555cb:::benchmark_data_smithi188_89276_object121548:head# 2024-04-23T16:25:46.621 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.621 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23161116:::benchmark_data_smithi188_89276_object119011:head# 2024-04-23T16:25:46.621 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23167e5c:::benchmark_data_smithi188_89276_object131432:head# 2024-04-23T16:25:46.622 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2316e2f0:::benchmark_data_smithi188_89276_object54231:head# 2024-04-23T16:25:46.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23172fcc:::benchmark_data_smithi188_89276_object68748:head# 2024-04-23T16:25:46.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2318538a:::benchmark_data_smithi188_89276_object156777:head# 2024-04-23T16:25:46.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2318ac38:::benchmark_data_smithi188_89276_object97334:head# 2024-04-23T16:25:46.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.624 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23197372:::benchmark_data_smithi188_89276_object126637:head# 2024-04-23T16:25:46.624 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.624 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23199bd2:::benchmark_data_smithi188_89276_object150892:head# 2024-04-23T16:25:46.624 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.624 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2319b7d1:::benchmark_data_smithi188_89276_object73341:head# 2024-04-23T16:25:46.624 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.624 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2319d250:::benchmark_data_smithi188_89276_object154726:head# 2024-04-23T16:25:46.624 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.624 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:231a827d:::benchmark_data_smithi188_89276_object126204:head# 2024-04-23T16:25:46.625 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.625 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:231ad0f5:::benchmark_data_smithi188_89276_object162487:head# 2024-04-23T16:25:46.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:231b7e93:::benchmark_data_smithi188_89276_object66304:head# 2024-04-23T16:25:46.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:231ca1f6:::benchmark_data_smithi188_89276_object62933:head# 2024-04-23T16:25:46.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:231cddb0:::benchmark_data_smithi188_89276_object76770:head# 2024-04-23T16:25:46.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:231cf9a7:::benchmark_data_smithi188_89276_object93245:head# 2024-04-23T16:25:46.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:231e2f14:::benchmark_data_smithi188_89276_object125811:head# 2024-04-23T16:25:46.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:231e86ed:::benchmark_data_smithi188_89276_object51981:head# 2024-04-23T16:25:46.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:231f75cd:::benchmark_data_smithi188_89276_object163032:head# 2024-04-23T16:25:46.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.628 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:231f9d46:::benchmark_data_smithi188_89276_object53181:head# 2024-04-23T16:25:46.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.628 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:232021a3:::benchmark_data_smithi188_89276_object128171:head# 2024-04-23T16:25:46.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.628 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2320fa94:::benchmark_data_smithi188_89276_object134654:head# 2024-04-23T16:25:46.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.628 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23211213:::benchmark_data_smithi188_89276_object143553:head# 2024-04-23T16:25:46.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2321d086:::benchmark_data_smithi188_89276_object159578:head# 2024-04-23T16:25:46.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23220860:::benchmark_data_smithi188_89276_object127047:head# 2024-04-23T16:25:46.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23220d41:::benchmark_data_smithi188_89276_object135147:head# 2024-04-23T16:25:46.630 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.630 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23223404:::benchmark_data_smithi188_89276_object65242:head# 2024-04-23T16:25:46.630 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.630 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2322a602:::benchmark_data_smithi188_89276_object64395:head# 2024-04-23T16:25:46.630 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.630 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2324207a:::benchmark_data_smithi188_89276_object139371:head# 2024-04-23T16:25:46.630 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.630 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2324f3c9:::benchmark_data_smithi188_89276_object112477:head# 2024-04-23T16:25:46.630 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.630 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23253717:::benchmark_data_smithi188_89276_object114442:head# 2024-04-23T16:25:46.630 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.631 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23259747:::benchmark_data_smithi188_89276_object134922:head# 2024-04-23T16:25:46.631 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.631 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2326d2e7:::benchmark_data_smithi188_89276_object77146:head# 2024-04-23T16:25:46.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2326fab9:::benchmark_data_smithi188_89276_object49104:head# 2024-04-23T16:25:46.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2327115c:::benchmark_data_smithi188_89276_object57239:head# 2024-04-23T16:25:46.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2327354d:::benchmark_data_smithi188_89276_object118578:head# 2024-04-23T16:25:46.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2327f23f:::benchmark_data_smithi188_89276_object76752:head# 2024-04-23T16:25:46.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2327fb45:::benchmark_data_smithi188_89276_object85456:head# 2024-04-23T16:25:46.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:232860a7:::benchmark_data_smithi188_89276_object70688:head# 2024-04-23T16:25:46.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.633 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23297465:::benchmark_data_smithi188_89276_object85019:head# 2024-04-23T16:25:46.633 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2329adad:::benchmark_data_smithi188_89276_object119830:head# 2024-04-23T16:25:46.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2329ff0e:::benchmark_data_smithi188_89276_object65378:head# 2024-04-23T16:25:46.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:232ab007:::benchmark_data_smithi188_89276_object51659:head# 2024-04-23T16:25:46.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:232ab873:::benchmark_data_smithi188_89276_object91815:head# 2024-04-23T16:25:46.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:232ab8d7:::benchmark_data_smithi188_89276_object118394:head# 2024-04-23T16:25:46.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:232b03cc:::benchmark_data_smithi188_89276_object146154:head# 2024-04-23T16:25:46.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.636 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:232bd39f:::benchmark_data_smithi188_89276_object67624:head# 2024-04-23T16:25:46.636 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.637 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:232bece8:::benchmark_data_smithi188_89276_object77759:head# 2024-04-23T16:25:46.637 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:232c09c7:::benchmark_data_smithi188_89276_object87943:head# 2024-04-23T16:25:46.638 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:232d1413:::benchmark_data_smithi188_89276_object156815:head# 2024-04-23T16:25:46.638 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:232fae2d:::benchmark_data_smithi188_89276_object73967:head# 2024-04-23T16:25:46.638 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:233069d9:::benchmark_data_smithi188_89276_object101606:head# 2024-04-23T16:25:46.638 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2330f7f0:::benchmark_data_smithi188_89276_object40630:head# 2024-04-23T16:25:46.638 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:233208f9:::benchmark_data_smithi188_89276_object81678:head# 2024-04-23T16:25:46.639 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.640 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23322ebf:::benchmark_data_smithi188_89276_object167742:head# 2024-04-23T16:25:46.640 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.640 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2332910d:::benchmark_data_smithi188_89276_object85184:head# 2024-04-23T16:25:46.640 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.640 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2332a018:::benchmark_data_smithi188_89276_object44786:head# 2024-04-23T16:25:46.640 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.640 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2332cc35:::benchmark_data_smithi188_89276_object129956:head# 2024-04-23T16:25:46.640 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.640 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:233323c0:::benchmark_data_smithi188_89276_object129028:head# 2024-04-23T16:25:46.641 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.641 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23338905:::benchmark_data_smithi188_89276_object43619:head# 2024-04-23T16:25:46.641 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.641 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2333a5b6:::benchmark_data_smithi188_89276_object42940:head# 2024-04-23T16:25:46.641 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.641 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2333b080:::benchmark_data_smithi188_89276_object143751:head# 2024-04-23T16:25:46.641 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.642 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:233409ef:::benchmark_data_smithi188_89276_object93932:head# 2024-04-23T16:25:46.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:233546f7:::benchmark_data_smithi188_89276_object121520:head# 2024-04-23T16:25:46.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2335cd93:::benchmark_data_smithi188_89276_object48517:head# 2024-04-23T16:25:46.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2335f634:::benchmark_data_smithi188_89276_object89075:head# 2024-04-23T16:25:46.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23360d6b:::benchmark_data_smithi188_89276_object48229:head# 2024-04-23T16:25:46.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23363ca7:::benchmark_data_smithi188_89276_object95473:head# 2024-04-23T16:25:46.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2336a5fe:::benchmark_data_smithi188_89276_object69721:head# 2024-04-23T16:25:46.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2336d81a:::benchmark_data_smithi188_89276_object160886:head# 2024-04-23T16:25:46.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.644 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23388b4b:::benchmark_data_smithi188_89276_object60109:head# 2024-04-23T16:25:46.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.644 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2338b80e:::benchmark_data_smithi188_89276_object92022:head# 2024-04-23T16:25:46.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.646 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:233975f2:::benchmark_data_smithi188_89276_object82953:head# 2024-04-23T16:25:46.646 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.646 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2339790a:::benchmark_data_smithi188_89276_object75246:head# 2024-04-23T16:25:46.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2339d0d4:::benchmark_data_smithi188_89276_object119446:head# 2024-04-23T16:25:46.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:233a71f4:::benchmark_data_smithi188_89276_object65507:head# 2024-04-23T16:25:46.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:233a8a0a:::benchmark_data_smithi188_89276_object163589:head# 2024-04-23T16:25:46.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:233b4ba8:::benchmark_data_smithi188_89276_object61561:head# 2024-04-23T16:25:46.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:233b522e:::benchmark_data_smithi188_89276_object112820:head# 2024-04-23T16:25:46.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:233c6586:::benchmark_data_smithi188_89276_object74515:head# 2024-04-23T16:25:46.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:233ce950:::benchmark_data_smithi188_89276_object68812:head# 2024-04-23T16:25:46.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:233cf10c:::benchmark_data_smithi188_89276_object163758:head# 2024-04-23T16:25:46.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:233ea6b2:::benchmark_data_smithi188_89276_object117539:head# 2024-04-23T16:25:46.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:233ecbc6:::benchmark_data_smithi188_89276_object135276:head# 2024-04-23T16:25:46.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:233f0153:::benchmark_data_smithi188_89276_object109405:head# 2024-04-23T16:25:46.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:233f0d9f:::benchmark_data_smithi188_89276_object73445:head# 2024-04-23T16:25:46.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23402da2:::benchmark_data_smithi188_89276_object72634:head# 2024-04-23T16:25:46.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.654 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2340a199:::benchmark_data_smithi188_89276_object164206:head# 2024-04-23T16:25:46.654 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.654 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2340e9ef:::benchmark_data_smithi188_89276_object162777:head# 2024-04-23T16:25:46.654 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.654 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23410144:::benchmark_data_smithi188_89276_object71414:head# 2024-04-23T16:25:46.654 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.654 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:234157f3:::benchmark_data_smithi188_89276_object134261:head# 2024-04-23T16:25:46.654 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2341e72d:::benchmark_data_smithi188_89276_object99622:head# 2024-04-23T16:25:46.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2342498c:::benchmark_data_smithi188_89276_object107948:head# 2024-04-23T16:25:46.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2342cbe9:::benchmark_data_smithi188_89276_object64370:head# 2024-04-23T16:25:46.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23431f7b:::benchmark_data_smithi188_89276_object78722:head# 2024-04-23T16:25:46.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23436bc5:::benchmark_data_smithi188_89276_object74508:head# 2024-04-23T16:25:46.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2344657a:::benchmark_data_smithi188_89276_object151645:head# 2024-04-23T16:25:46.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2344bd97:::benchmark_data_smithi188_89276_object56941:head# 2024-04-23T16:25:46.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23455c84:::benchmark_data_smithi188_89276_object163660:head# 2024-04-23T16:25:46.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23457735:::benchmark_data_smithi188_89276_object81143:head# 2024-04-23T16:25:46.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23471ddf:::benchmark_data_smithi188_89276_object154603:head# 2024-04-23T16:25:46.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23473e09:::benchmark_data_smithi188_89276_object68306:head# 2024-04-23T16:25:46.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2347498f:::benchmark_data_smithi188_89276_object137910:head# 2024-04-23T16:25:46.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23484004:::benchmark_data_smithi188_89276_object42121:head# 2024-04-23T16:25:46.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2349162e:::benchmark_data_smithi188_89276_object119807:head# 2024-04-23T16:25:46.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2349744e:::benchmark_data_smithi188_89276_object144604:head# 2024-04-23T16:25:46.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23498e7d:::benchmark_data_smithi188_89276_object91658:head# 2024-04-23T16:25:46.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23499678:::benchmark_data_smithi188_89276_object106068:head# 2024-04-23T16:25:46.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:234a011a:::benchmark_data_smithi188_89276_object137916:head# 2024-04-23T16:25:46.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:234ab452:::benchmark_data_smithi188_89276_object118898:head# 2024-04-23T16:25:46.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:234b7361:::benchmark_data_smithi188_89276_object117903:head# 2024-04-23T16:25:46.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:234c064b:::benchmark_data_smithi188_89276_object45398:head# 2024-04-23T16:25:46.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:234c0d04:::benchmark_data_smithi188_89276_object129056:head# 2024-04-23T16:25:46.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.658 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:234e427b:::benchmark_data_smithi188_89276_object150604:head# 2024-04-23T16:25:46.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.658 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:234ea750:::benchmark_data_smithi188_89276_object159071:head# 2024-04-23T16:25:46.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.658 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2350ae2b:::benchmark_data_smithi188_89276_object125244:head# 2024-04-23T16:25:46.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.658 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2351639a:::benchmark_data_smithi188_89276_object128045:head# 2024-04-23T16:25:46.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.658 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2352ae2b:::benchmark_data_smithi188_89276_object43918:head# 2024-04-23T16:25:46.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.658 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2353371c:::benchmark_data_smithi188_89276_object41228:head# 2024-04-23T16:25:46.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23545336:::benchmark_data_smithi188_89276_object41500:head# 2024-04-23T16:25:46.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2355260f:::benchmark_data_smithi188_89276_object79658:head# 2024-04-23T16:25:46.663 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23554523:::benchmark_data_smithi188_89276_object154624:head# 2024-04-23T16:25:46.663 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2355cd15:::benchmark_data_smithi188_89276_object154545:head# 2024-04-23T16:25:46.663 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2355f348:::benchmark_data_smithi188_89276_object111211:head# 2024-04-23T16:25:46.663 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23570a1d:::benchmark_data_smithi188_89276_object47714:head# 2024-04-23T16:25:46.663 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23577831:::benchmark_data_smithi188_89276_object162225:head# 2024-04-23T16:25:46.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23579496:::benchmark_data_smithi188_89276_object164312:head# 2024-04-23T16:25:46.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23585999:::benchmark_data_smithi188_89276_object74127:head# 2024-04-23T16:25:46.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2358af56:::benchmark_data_smithi188_89276_object146322:head# 2024-04-23T16:25:46.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2359fbed:::benchmark_data_smithi188_89276_object157693:head# 2024-04-23T16:25:46.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:235a650b:::benchmark_data_smithi188_89276_object79328:head# 2024-04-23T16:25:46.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:235a69ad:::benchmark_data_smithi188_89276_object61113:head# 2024-04-23T16:25:46.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:235ac1c5:::benchmark_data_smithi188_89276_object48775:head# 2024-04-23T16:25:46.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:235bb01e:::benchmark_data_smithi188_89276_object85773:head# 2024-04-23T16:25:46.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:235cebdf:::benchmark_data_smithi188_89276_object121220:head# 2024-04-23T16:25:46.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:235d1447:::benchmark_data_smithi188_89276_object110011:head# 2024-04-23T16:25:46.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:235d6653:::benchmark_data_smithi188_89276_object153326:head# 2024-04-23T16:25:46.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:235ee41d:::benchmark_data_smithi188_89276_object132624:head# 2024-04-23T16:25:46.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:235f342e:::benchmark_data_smithi188_89276_object93564:head# 2024-04-23T16:25:46.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236047cb:::benchmark_data_smithi188_89276_object101304:head# 2024-04-23T16:25:46.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236070f0:::benchmark_data_smithi188_89276_object57866:head# 2024-04-23T16:25:46.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236074de:::benchmark_data_smithi188_89276_object45715:head# 2024-04-23T16:25:46.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2360eb33:::benchmark_data_smithi188_89276_object123531:head# 2024-04-23T16:25:46.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236142c9:::benchmark_data_smithi188_89276_object54839:head# 2024-04-23T16:25:46.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23616faa:::benchmark_data_smithi188_89276_object102286:head# 2024-04-23T16:25:46.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236267c2:::benchmark_data_smithi188_89276_object94532:head# 2024-04-23T16:25:46.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23626db1:::benchmark_data_smithi188_89276_object64151:head# 2024-04-23T16:25:46.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23629eab:::benchmark_data_smithi188_89276_object55109:head# 2024-04-23T16:25:46.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2362b424:::benchmark_data_smithi188_89276_object136890:head# 2024-04-23T16:25:46.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23642815:::benchmark_data_smithi188_89276_object146583:head# 2024-04-23T16:25:46.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2364439d:::benchmark_data_smithi188_89276_object86303:head# 2024-04-23T16:25:46.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236526b6:::benchmark_data_smithi188_89276_object106828:head# 2024-04-23T16:25:46.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23658008:::benchmark_data_smithi188_89276_object88860:head# 2024-04-23T16:25:46.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2366327c:::benchmark_data_smithi188_89276_object64286:head# 2024-04-23T16:25:46.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23663db6:::benchmark_data_smithi188_89276_object68586:head# 2024-04-23T16:25:46.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2366ece4:::benchmark_data_smithi188_89276_object124885:head# 2024-04-23T16:25:46.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2366ee82:::benchmark_data_smithi188_89276_object73932:head# 2024-04-23T16:25:46.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23670844:::benchmark_data_smithi188_89276_object104134:head# 2024-04-23T16:25:46.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236753d8:::benchmark_data_smithi188_89276_object157328:head# 2024-04-23T16:25:46.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236786bd:::benchmark_data_smithi188_89276_object107941:head# 2024-04-23T16:25:46.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23681d4d:::benchmark_data_smithi188_89276_object61622:head# 2024-04-23T16:25:46.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23689994:::benchmark_data_smithi188_89276_object117464:head# 2024-04-23T16:25:46.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2368f762:::benchmark_data_smithi188_89276_object50500:head# 2024-04-23T16:25:46.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23692d2b:::benchmark_data_smithi188_89276_object88871:head# 2024-04-23T16:25:46.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23693a69:::benchmark_data_smithi188_89276_object108137:head# 2024-04-23T16:25:46.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236969de:::benchmark_data_smithi188_89276_object90661:head# 2024-04-23T16:25:46.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23698995:::benchmark_data_smithi188_89276_object156340:head# 2024-04-23T16:25:46.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2369afd2:::benchmark_data_smithi188_89276_object71949:head# 2024-04-23T16:25:46.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2369f756:::benchmark_data_smithi188_89276_object161781:head# 2024-04-23T16:25:46.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236a00b2:::benchmark_data_smithi188_89276_object115382:head# 2024-04-23T16:25:46.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236a069d:::benchmark_data_smithi188_89276_object109060:head# 2024-04-23T16:25:46.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.674 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236a74bf:::benchmark_data_smithi188_89276_object118973:head# 2024-04-23T16:25:46.674 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.674 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236a941c:::benchmark_data_smithi188_89276_object129834:head# 2024-04-23T16:25:46.674 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236ad1b6:::benchmark_data_smithi188_89276_object46412:head# 2024-04-23T16:25:46.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236ba3f3:::benchmark_data_smithi188_89276_object73149:head# 2024-04-23T16:25:46.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236cf53d:::benchmark_data_smithi188_89276_object114572:head# 2024-04-23T16:25:46.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236d75e8:::benchmark_data_smithi188_89276_object41900:head# 2024-04-23T16:25:46.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236d7a53:::benchmark_data_smithi188_89276_object148036:head# 2024-04-23T16:25:46.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236d8fab:::benchmark_data_smithi188_89276_object69904:head# 2024-04-23T16:25:46.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236dc628:::benchmark_data_smithi188_89276_object147128:head# 2024-04-23T16:25:46.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236dc6f6:::benchmark_data_smithi188_89276_object51748:head# 2024-04-23T16:25:46.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.677 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236df840:::benchmark_data_smithi188_89276_object142740:head# 2024-04-23T16:25:46.677 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.677 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236ea7b4:::benchmark_data_smithi188_89276_object168181:head# 2024-04-23T16:25:46.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236ee3af:::benchmark_data_smithi188_89276_object145830:head# 2024-04-23T16:25:46.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:236fb70a:::benchmark_data_smithi188_89276_object110779:head# 2024-04-23T16:25:46.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23700f4c:::benchmark_data_smithi188_89276_object136767:head# 2024-04-23T16:25:46.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:237069bd:::benchmark_data_smithi188_89276_object47473:head# 2024-04-23T16:25:46.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2370cfc6:::benchmark_data_smithi188_89276_object109380:head# 2024-04-23T16:25:46.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2372e6b5:::benchmark_data_smithi188_89276_object55287:head# 2024-04-23T16:25:46.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2372e7fb:::benchmark_data_smithi188_89276_object152265:head# 2024-04-23T16:25:46.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23738442:::benchmark_data_smithi188_89276_object132008:head# 2024-04-23T16:25:46.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23756e28:::benchmark_data_smithi188_89276_object127704:head# 2024-04-23T16:25:46.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23760221:::benchmark_data_smithi188_89276_object141135:head# 2024-04-23T16:25:46.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:237703c0:::benchmark_data_smithi188_89276_object133649:head# 2024-04-23T16:25:46.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2377f4e3:::benchmark_data_smithi188_89276_object84989:head# 2024-04-23T16:25:46.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23790d22:::benchmark_data_smithi188_89276_object76588:head# 2024-04-23T16:25:46.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23793354:::benchmark_data_smithi188_89276_object129268:head# 2024-04-23T16:25:46.684 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23796ac7:::benchmark_data_smithi188_89276_object155141:head# 2024-04-23T16:25:46.684 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:237af94f:::benchmark_data_smithi188_89276_object64437:head# 2024-04-23T16:25:46.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:237b944e:::benchmark_data_smithi188_89276_object71791:head# 2024-04-23T16:25:46.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:237bc800:::benchmark_data_smithi188_89276_object117684:head# 2024-04-23T16:25:46.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:237bfec3:::benchmark_data_smithi188_89276_object90028:head# 2024-04-23T16:25:46.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:237cc739:::benchmark_data_smithi188_89276_object95023:head# 2024-04-23T16:25:46.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:237ee65c:::benchmark_data_smithi188_89276_object167268:head# 2024-04-23T16:25:46.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:237f1550:::benchmark_data_smithi188_89276_object40357:head# 2024-04-23T16:25:46.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:237f80ec:::benchmark_data_smithi188_89276_object125078:head# 2024-04-23T16:25:46.686 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:237ff61f:::benchmark_data_smithi188_89276_object65435:head# 2024-04-23T16:25:46.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2381857f:::benchmark_data_smithi188_89276_object77021:head# 2024-04-23T16:25:46.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2381883e:::benchmark_data_smithi188_89276_object113490:head# 2024-04-23T16:25:46.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23825d64:::benchmark_data_smithi188_89276_object83334:head# 2024-04-23T16:25:46.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:238263fd:::benchmark_data_smithi188_89276_object92941:head# 2024-04-23T16:25:46.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2382693c:::benchmark_data_smithi188_89276_object46706:head# 2024-04-23T16:25:46.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23827b2a:::benchmark_data_smithi188_89276_object120505:head# 2024-04-23T16:25:46.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2382c1c2:::benchmark_data_smithi188_89276_object122608:head# 2024-04-23T16:25:46.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2383cb43:::benchmark_data_smithi188_89276_object85106:head# 2024-04-23T16:25:46.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2385d79d:::benchmark_data_smithi188_89276_object52604:head# 2024-04-23T16:25:46.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2386a3ce:::benchmark_data_smithi188_89276_object165593:head# 2024-04-23T16:25:46.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2386c4d7:::benchmark_data_smithi188_89276_object126870:head# 2024-04-23T16:25:46.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2387940f:::benchmark_data_smithi188_89276_object87551:head# 2024-04-23T16:25:46.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2389ca34:::benchmark_data_smithi188_89276_object115391:head# 2024-04-23T16:25:46.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2389d46e:::benchmark_data_smithi188_89276_object60093:head# 2024-04-23T16:25:46.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2389dde6:::benchmark_data_smithi188_89276_object150555:head# 2024-04-23T16:25:46.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:238b1c5f:::benchmark_data_smithi188_89276_object164535:head# 2024-04-23T16:25:46.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:238bac26:::benchmark_data_smithi188_89276_object162206:head# 2024-04-23T16:25:46.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:238c6d7f:::benchmark_data_smithi188_89276_object72493:head# 2024-04-23T16:25:46.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:238cd1c5:::benchmark_data_smithi188_89276_object134267:head# 2024-04-23T16:25:46.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.694 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:238d396f:::benchmark_data_smithi188_89276_object142978:head# 2024-04-23T16:25:46.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.694 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:238d5614:::benchmark_data_smithi188_89276_object68179:head# 2024-04-23T16:25:46.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:238eca20:::benchmark_data_smithi188_89276_object168342:head# 2024-04-23T16:25:46.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:239038e9:::benchmark_data_smithi188_89276_object162215:head# 2024-04-23T16:25:46.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23904a85:::benchmark_data_smithi188_89276_object110346:head# 2024-04-23T16:25:46.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.696 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23931fea:::benchmark_data_smithi188_89276_object123620:head# 2024-04-23T16:25:46.696 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.696 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23943389:::benchmark_data_smithi188_89276_object162761:head# 2024-04-23T16:25:46.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2394555f:::benchmark_data_smithi188_89276_object66709:head# 2024-04-23T16:25:46.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2394b2d6:::benchmark_data_smithi188_89276_object52583:head# 2024-04-23T16:25:46.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2395c927:::benchmark_data_smithi188_89276_object105760:head# 2024-04-23T16:25:46.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:239685a6:::benchmark_data_smithi188_89276_object143633:head# 2024-04-23T16:25:46.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23976b99:::benchmark_data_smithi188_89276_object143237:head# 2024-04-23T16:25:46.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:239809e2:::benchmark_data_smithi188_89276_object55636:head# 2024-04-23T16:25:46.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2398b006:::benchmark_data_smithi188_89276_object158207:head# 2024-04-23T16:25:46.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.699 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2398d92a:::benchmark_data_smithi188_89276_object150648:head# 2024-04-23T16:25:46.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:239979b7:::benchmark_data_smithi188_89276_object86665:head# 2024-04-23T16:25:46.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:239a2cb2:::benchmark_data_smithi188_89276_object110353:head# 2024-04-23T16:25:46.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:239a440b:::benchmark_data_smithi188_89276_object65706:head# 2024-04-23T16:25:46.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:239b5b10:::benchmark_data_smithi188_89276_object138234:head# 2024-04-23T16:25:46.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:239bcac5:::benchmark_data_smithi188_89276_object126517:head# 2024-04-23T16:25:46.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:239c5fa7:::benchmark_data_smithi188_89276_object109321:head# 2024-04-23T16:25:46.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:239d23e5:::benchmark_data_smithi188_89276_object164112:head# 2024-04-23T16:25:46.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:239d501d:::benchmark_data_smithi188_89276_object139271:head# 2024-04-23T16:25:46.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:239d5960:::benchmark_data_smithi188_89276_object136627:head# 2024-04-23T16:25:46.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:239f8e74:::benchmark_data_smithi188_89276_object155941:head# 2024-04-23T16:25:46.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a07598:::benchmark_data_smithi188_89276_object142145:head# 2024-04-23T16:25:46.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a109cb:::benchmark_data_smithi188_89276_object164547:head# 2024-04-23T16:25:46.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a34b59:::benchmark_data_smithi188_89276_object97663:head# 2024-04-23T16:25:46.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a4815e:::benchmark_data_smithi188_89276_object64051:head# 2024-04-23T16:25:46.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a4facf:::benchmark_data_smithi188_89276_object46829:head# 2024-04-23T16:25:46.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a51ab6:::benchmark_data_smithi188_89276_object130032:head# 2024-04-23T16:25:46.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a534b1:::benchmark_data_smithi188_89276_object96469:head# 2024-04-23T16:25:46.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a5a7e8:::benchmark_data_smithi188_89276_object125652:head# 2024-04-23T16:25:46.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a6630a:::benchmark_data_smithi188_89276_object99058:head# 2024-04-23T16:25:46.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a67658:::benchmark_data_smithi188_89276_object125528:head# 2024-04-23T16:25:46.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a67f2d:::benchmark_data_smithi188_89276_object47701:head# 2024-04-23T16:25:46.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a68384:::benchmark_data_smithi188_89276_object53498:head# 2024-04-23T16:25:46.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a6a1f1:::benchmark_data_smithi188_89276_object51422:head# 2024-04-23T16:25:46.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a710f6:::benchmark_data_smithi188_89276_object115685:head# 2024-04-23T16:25:46.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.708 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a72803:::benchmark_data_smithi188_89276_object46271:head# 2024-04-23T16:25:46.708 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.710 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a74bac:::benchmark_data_smithi188_89276_object106049:head# 2024-04-23T16:25:46.710 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.710 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a7a472:::benchmark_data_smithi188_89276_object94128:head# 2024-04-23T16:25:46.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a7dc3a:::benchmark_data_smithi188_89276_object139336:head# 2024-04-23T16:25:46.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a82d38:::benchmark_data_smithi188_89276_object167187:head# 2024-04-23T16:25:46.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a8bbf0:::benchmark_data_smithi188_89276_object135385:head# 2024-04-23T16:25:46.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a924c1:::benchmark_data_smithi188_89276_object152714:head# 2024-04-23T16:25:46.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23a978d7:::benchmark_data_smithi188_89276_object96292:head# 2024-04-23T16:25:46.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23aa1da0:::benchmark_data_smithi188_89276_object163514:head# 2024-04-23T16:25:46.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23aaa825:::benchmark_data_smithi188_89276_object110602:head# 2024-04-23T16:25:46.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23aad8ed:::benchmark_data_smithi188_89276_object168511:head# 2024-04-23T16:25:46.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ab1536:::benchmark_data_smithi188_89276_object82866:head# 2024-04-23T16:25:46.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ac0b90:::benchmark_data_smithi188_89276_object100123:head# 2024-04-23T16:25:46.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ac2e6a:::benchmark_data_smithi188_89276_object91900:head# 2024-04-23T16:25:46.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ac8190:::benchmark_data_smithi188_89276_object113550:head# 2024-04-23T16:25:46.714 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ad0c52:::benchmark_data_smithi188_89276_object158767:head# 2024-04-23T16:25:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ad1d77:::benchmark_data_smithi188_89276_object76580:head# 2024-04-23T16:25:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ad5466:::benchmark_data_smithi188_89276_object127186:head# 2024-04-23T16:25:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ad7b0d:::benchmark_data_smithi188_89276_object141139:head# 2024-04-23T16:25:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23af4d72:::benchmark_data_smithi188_89276_object93708:head# 2024-04-23T16:25:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23b011e5:::benchmark_data_smithi188_89276_object40696:head# 2024-04-23T16:25:46.716 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.716 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23b01556:::benchmark_data_smithi188_89276_object165329:head# 2024-04-23T16:25:46.716 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23b045fe:::benchmark_data_smithi188_89276_object62425:head# 2024-04-23T16:25:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23b0f2b3:::benchmark_data_smithi188_89276_object144780:head# 2024-04-23T16:25:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23b18d91:::benchmark_data_smithi188_89276_object81530:head# 2024-04-23T16:25:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23b26e71:::benchmark_data_smithi188_89276_object49176:head# 2024-04-23T16:25:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23b37e76:::benchmark_data_smithi188_89276_object82773:head# 2024-04-23T16:25:46.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23b4abf2:::benchmark_data_smithi188_89276_object127021:head# 2024-04-23T16:25:46.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23b523b1:::benchmark_data_smithi188_89276_object158157:head# 2024-04-23T16:25:46.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23b57b4f:::benchmark_data_smithi188_89276_object104913:head# 2024-04-23T16:25:46.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23b5f1a2:::benchmark_data_smithi188_89276_object41310:head# 2024-04-23T16:25:46.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23b60641:::benchmark_data_smithi188_89276_object48020:head# 2024-04-23T16:25:46.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23b792b4:::benchmark_data_smithi188_89276_object111489:head# 2024-04-23T16:25:46.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23b7f2c6:::benchmark_data_smithi188_89276_object41553:head# 2024-04-23T16:25:46.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23b802e6:::benchmark_data_smithi188_89276_object120288:head# 2024-04-23T16:25:46.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23b82f0c:::benchmark_data_smithi188_89276_object139653:head# 2024-04-23T16:25:46.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23b9e595:::benchmark_data_smithi188_89276_object148525:head# 2024-04-23T16:25:46.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ba3036:::benchmark_data_smithi188_89276_object74801:head# 2024-04-23T16:25:46.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ba822b:::benchmark_data_smithi188_89276_object75491:head# 2024-04-23T16:25:46.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23bac2ef:::benchmark_data_smithi188_89276_object73565:head# 2024-04-23T16:25:46.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23bf597a:::benchmark_data_smithi188_89276_object60702:head# 2024-04-23T16:25:46.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23bfa69f:::benchmark_data_smithi188_89276_object72256:head# 2024-04-23T16:25:46.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23c066d3:::benchmark_data_smithi188_89276_object106931:head# 2024-04-23T16:25:46.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23c0cf70:::benchmark_data_smithi188_89276_object64480:head# 2024-04-23T16:25:46.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23c192e3:::benchmark_data_smithi188_89276_object84744:head# 2024-04-23T16:25:46.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23c1fcf4:::benchmark_data_smithi188_89276_object109032:head# 2024-04-23T16:25:46.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23c358a3:::benchmark_data_smithi188_89276_object55078:head# 2024-04-23T16:25:46.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23c4b87e:::benchmark_data_smithi188_89276_object104083:head# 2024-04-23T16:25:46.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23c5c586:::benchmark_data_smithi188_89276_object83069:head# 2024-04-23T16:25:46.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23c68fcd:::benchmark_data_smithi188_89276_object96782:head# 2024-04-23T16:25:46.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23c7e226:::benchmark_data_smithi188_89276_object62155:head# 2024-04-23T16:25:46.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23c85eb5:::benchmark_data_smithi188_89276_object74970:head# 2024-04-23T16:25:46.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23c8856a:::benchmark_data_smithi188_89276_object144343:head# 2024-04-23T16:25:46.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23c89c6d:::benchmark_data_smithi188_89276_object46039:head# 2024-04-23T16:25:46.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23c8a640:::benchmark_data_smithi188_89276_object104441:head# 2024-04-23T16:25:46.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23c9659e:::benchmark_data_smithi188_89276_object150102:head# 2024-04-23T16:25:46.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.729 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23c981ba:::benchmark_data_smithi188_89276_object121176:head# 2024-04-23T16:25:46.729 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ca4288:::benchmark_data_smithi188_89276_object86451:head# 2024-04-23T16:25:46.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23cbc987:::benchmark_data_smithi188_89276_object161491:head# 2024-04-23T16:25:46.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23cc2760:::benchmark_data_smithi188_89276_object93545:head# 2024-04-23T16:25:46.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23cc7679:::benchmark_data_smithi188_89276_object138750:head# 2024-04-23T16:25:46.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23cc8be8:::benchmark_data_smithi188_89276_object88146:head# 2024-04-23T16:25:46.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23cde766:::benchmark_data_smithi188_89276_object105808:head# 2024-04-23T16:25:46.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ce2305:::benchmark_data_smithi188_89276_object104991:head# 2024-04-23T16:25:46.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ce6216:::benchmark_data_smithi188_89276_object74037:head# 2024-04-23T16:25:46.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ce94e3:::benchmark_data_smithi188_89276_object108067:head# 2024-04-23T16:25:46.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23cea183:::benchmark_data_smithi188_89276_object133428:head# 2024-04-23T16:25:46.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23cefe72:::benchmark_data_smithi188_89276_object101608:head# 2024-04-23T16:25:46.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23cfb0f2:::benchmark_data_smithi188_89276_object49506:head# 2024-04-23T16:25:46.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23cfbfff:::benchmark_data_smithi188_89276_object114362:head# 2024-04-23T16:25:46.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23d0cb2f:::benchmark_data_smithi188_89276_object129960:head# 2024-04-23T16:25:46.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23d18757:::benchmark_data_smithi188_89276_object105891:head# 2024-04-23T16:25:46.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23d1b06d:::benchmark_data_smithi188_89276_object165799:head# 2024-04-23T16:25:46.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23d1e60f:::benchmark_data_smithi188_89276_object104971:head# 2024-04-23T16:25:46.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23d27c8b:::benchmark_data_smithi188_89276_object100303:head# 2024-04-23T16:25:46.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.737 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23d3387e:::benchmark_data_smithi188_89276_object158340:head# 2024-04-23T16:25:46.737 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.737 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23d378fe:::benchmark_data_smithi188_89276_object142810:head# 2024-04-23T16:25:46.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23d41573:::benchmark_data_smithi188_89276_object92120:head# 2024-04-23T16:25:46.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23d4c9a0:::benchmark_data_smithi188_89276_object41000:head# 2024-04-23T16:25:46.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23d56731:::benchmark_data_smithi188_89276_object50730:head# 2024-04-23T16:25:46.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23d5bb0b:::benchmark_data_smithi188_89276_object100794:head# 2024-04-23T16:25:46.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23d5c2a2:::benchmark_data_smithi188_89276_object133144:head# 2024-04-23T16:25:46.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23d61c65:::benchmark_data_smithi188_89276_object50369:head# 2024-04-23T16:25:46.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.743 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23d7e17b:::benchmark_data_smithi188_89276_object164067:head# 2024-04-23T16:25:46.744 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.744 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23d8ad01:::benchmark_data_smithi188_89276_object68048:head# 2024-04-23T16:25:46.744 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23d912e8:::benchmark_data_smithi188_89276_object49723:head# 2024-04-23T16:25:46.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.746 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23d9a196:::benchmark_data_smithi188_89276_object57625:head# 2024-04-23T16:25:46.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.746 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23da7f86:::benchmark_data_smithi188_89276_object67528:head# 2024-04-23T16:25:46.747 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.747 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23daa3f3:::benchmark_data_smithi188_89276_object42007:head# 2024-04-23T16:25:46.747 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.747 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23dae8ae:::benchmark_data_smithi188_89276_object126855:head# 2024-04-23T16:25:46.747 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.748 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23dc3e7c:::benchmark_data_smithi188_89276_object156811:head# 2024-04-23T16:25:46.748 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.748 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23dd0952:::benchmark_data_smithi188_89276_object63841:head# 2024-04-23T16:25:46.748 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.748 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23dd61cc:::benchmark_data_smithi188_89276_object66805:head# 2024-04-23T16:25:46.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23de174d:::benchmark_data_smithi188_89276_object141792:head# 2024-04-23T16:25:46.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23de2179:::benchmark_data_smithi188_89276_object103556:head# 2024-04-23T16:25:46.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23de499e:::benchmark_data_smithi188_89276_object81479:head# 2024-04-23T16:25:46.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23de53d4:::benchmark_data_smithi188_89276_object56874:head# 2024-04-23T16:25:46.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23def0b4:::benchmark_data_smithi188_89276_object150941:head# 2024-04-23T16:25:46.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23df2c2f:::benchmark_data_smithi188_89276_object100277:head# 2024-04-23T16:25:46.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23dfd172:::benchmark_data_smithi188_89276_object137983:head# 2024-04-23T16:25:46.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23dfe2b6:::benchmark_data_smithi188_89276_object59753:head# 2024-04-23T16:25:46.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e1f93f:::benchmark_data_smithi188_89276_object44754:head# 2024-04-23T16:25:46.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e23ced:::benchmark_data_smithi188_89276_object101808:head# 2024-04-23T16:25:46.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e2fe35:::benchmark_data_smithi188_89276_object149991:head# 2024-04-23T16:25:46.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e31ad5:::benchmark_data_smithi188_89276_object137766:head# 2024-04-23T16:25:46.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e382d5:::benchmark_data_smithi188_89276_object116235:head# 2024-04-23T16:25:46.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e3c996:::benchmark_data_smithi188_89276_object156899:head# 2024-04-23T16:25:46.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.755 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e48ba8:::benchmark_data_smithi188_89276_object44920:head# 2024-04-23T16:25:46.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.756 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e4be28:::benchmark_data_smithi188_89276_object107533:head# 2024-04-23T16:25:46.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.756 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e4d0f3:::benchmark_data_smithi188_89276_object46500:head# 2024-04-23T16:25:46.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.757 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e52afa:::benchmark_data_smithi188_89276_object58051:head# 2024-04-23T16:25:46.757 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.757 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e58e2a:::benchmark_data_smithi188_89276_object76343:head# 2024-04-23T16:25:46.757 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.758 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e59dbd:::benchmark_data_smithi188_89276_object137682:head# 2024-04-23T16:25:46.758 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.758 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e5d57e:::benchmark_data_smithi188_89276_object111245:head# 2024-04-23T16:25:46.758 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.758 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e5f915:::benchmark_data_smithi188_89276_object88311:head# 2024-04-23T16:25:46.758 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.758 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e64c3a:::benchmark_data_smithi188_89276_object45512:head# 2024-04-23T16:25:46.758 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.759 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e65217:::benchmark_data_smithi188_89276_object102642:head# 2024-04-23T16:25:46.759 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.759 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e7a09a:::benchmark_data_smithi188_89276_object120387:head# 2024-04-23T16:25:46.759 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e7aaa4:::benchmark_data_smithi188_89276_object99953:head# 2024-04-23T16:25:46.760 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e879cb:::benchmark_data_smithi188_89276_object125876:head# 2024-04-23T16:25:46.760 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e88e5c:::benchmark_data_smithi188_89276_object100049:head# 2024-04-23T16:25:46.760 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e8ff12:::benchmark_data_smithi188_89276_object102582:head# 2024-04-23T16:25:46.760 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e97e32:::benchmark_data_smithi188_89276_object155531:head# 2024-04-23T16:25:46.760 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23e99caa:::benchmark_data_smithi188_89276_object111746:head# 2024-04-23T16:25:46.761 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.762 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23eafeb1:::benchmark_data_smithi188_89276_object53691:head# 2024-04-23T16:25:46.762 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.762 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23eb166c:::benchmark_data_smithi188_89276_object60442:head# 2024-04-23T16:25:46.762 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.762 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ec033e:::benchmark_data_smithi188_89276_object101495:head# 2024-04-23T16:25:46.762 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.762 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ec7761:::benchmark_data_smithi188_89276_object106999:head# 2024-04-23T16:25:46.762 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.762 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ed7c92:::benchmark_data_smithi188_89276_object40594:head# 2024-04-23T16:25:46.762 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.763 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23eda01e:::benchmark_data_smithi188_89276_object42012:head# 2024-04-23T16:25:46.763 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.763 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23eded29:::benchmark_data_smithi188_89276_object99258:head# 2024-04-23T16:25:46.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23eeb5c8:::benchmark_data_smithi188_89276_object154011:head# 2024-04-23T16:25:46.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23eeb6c0:::benchmark_data_smithi188_89276_object132915:head# 2024-04-23T16:25:46.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23efc474:::benchmark_data_smithi188_89276_object138651:head# 2024-04-23T16:25:46.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23eff8f9:::benchmark_data_smithi188_89276_object118936:head# 2024-04-23T16:25:46.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f01fdc:::benchmark_data_smithi188_89276_object123662:head# 2024-04-23T16:25:46.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f02cc1:::benchmark_data_smithi188_89276_object79190:head# 2024-04-23T16:25:46.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.765 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f05a28:::benchmark_data_smithi188_89276_object42338:head# 2024-04-23T16:25:46.765 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f06c69:::benchmark_data_smithi188_89276_object153428:head# 2024-04-23T16:25:46.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f12e79:::benchmark_data_smithi188_89276_object80689:head# 2024-04-23T16:25:46.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f1364b:::benchmark_data_smithi188_89276_object54555:head# 2024-04-23T16:25:46.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f28b46:::benchmark_data_smithi188_89276_object111037:head# 2024-04-23T16:25:46.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f2e9ff:::benchmark_data_smithi188_89276_object103044:head# 2024-04-23T16:25:46.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f3342e:::benchmark_data_smithi188_89276_object148074:head# 2024-04-23T16:25:46.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f3aa37:::benchmark_data_smithi188_89276_object53378:head# 2024-04-23T16:25:46.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.768 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f3eb14:::benchmark_data_smithi188_89276_object78943:head# 2024-04-23T16:25:46.768 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.768 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f41689:::benchmark_data_smithi188_89276_object64712:head# 2024-04-23T16:25:46.768 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.768 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f4752a:::benchmark_data_smithi188_89276_object114960:head# 2024-04-23T16:25:46.768 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.768 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f50d6c:::benchmark_data_smithi188_89276_object156500:head# 2024-04-23T16:25:46.768 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.768 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f54550:::benchmark_data_smithi188_89276_object150837:head# 2024-04-23T16:25:46.768 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.768 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f56697:::benchmark_data_smithi188_89276_object69297:head# 2024-04-23T16:25:46.769 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.769 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f59634:::benchmark_data_smithi188_89276_object45164:head# 2024-04-23T16:25:46.769 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.769 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f5b6b0:::benchmark_data_smithi188_89276_object43878:head# 2024-04-23T16:25:46.769 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.769 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f622e3:::benchmark_data_smithi188_89276_object138093:head# 2024-04-23T16:25:46.769 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.769 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f69646:::benchmark_data_smithi188_89276_object58180:head# 2024-04-23T16:25:46.769 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.770 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f6966d:::benchmark_data_smithi188_89276_object84998:head# 2024-04-23T16:25:46.770 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.771 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f6fdd4:::benchmark_data_smithi188_89276_object127414:head# 2024-04-23T16:25:46.771 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.771 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f7b399:::benchmark_data_smithi188_89276_object74195:head# 2024-04-23T16:25:46.771 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.771 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f8774f:::benchmark_data_smithi188_89276_object46250:head# 2024-04-23T16:25:46.771 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.771 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f8bcc7:::benchmark_data_smithi188_89276_object80312:head# 2024-04-23T16:25:46.771 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.771 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f8e6b5:::benchmark_data_smithi188_89276_object93068:head# 2024-04-23T16:25:46.771 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.771 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f8f85e:::benchmark_data_smithi188_89276_object61323:head# 2024-04-23T16:25:46.771 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.771 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f90a7e:::benchmark_data_smithi188_89276_object126790:head# 2024-04-23T16:25:46.771 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.772 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23f95d65:::benchmark_data_smithi188_89276_object93423:head# 2024-04-23T16:25:46.772 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23fa3145:::benchmark_data_smithi188_89276_object92165:head# 2024-04-23T16:25:46.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23fa74b4:::benchmark_data_smithi188_89276_object149302:head# 2024-04-23T16:25:46.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23fb0c6f:::benchmark_data_smithi188_89276_object78226:head# 2024-04-23T16:25:46.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23fbacb3:::benchmark_data_smithi188_89276_object83839:head# 2024-04-23T16:25:46.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23fd1ac7:::benchmark_data_smithi188_89276_object83108:head# 2024-04-23T16:25:46.774 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.774 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23fd5d53:::benchmark_data_smithi188_89276_object123386:head# 2024-04-23T16:25:46.774 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.774 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23fe2e89:::benchmark_data_smithi188_89276_object125330:head# 2024-04-23T16:25:46.774 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.774 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23fe81e7:::benchmark_data_smithi188_89276_object137958:head# 2024-04-23T16:25:46.774 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.774 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ff0201:::benchmark_data_smithi188_89276_object56954:head# 2024-04-23T16:25:46.774 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.776 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ff0ad6:::benchmark_data_smithi188_89276_object157044:head# 2024-04-23T16:25:46.776 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.776 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ff44fa:::benchmark_data_smithi188_89276_object118026:head# 2024-04-23T16:25:46.776 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.776 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ff5cf4:::benchmark_data_smithi188_89276_object89509:head# 2024-04-23T16:25:46.776 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.776 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:23ffea68:::benchmark_data_smithi188_89276_object62364:head# 2024-04-23T16:25:46.776 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.776 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24011dd1:::benchmark_data_smithi188_89276_object45624:head# 2024-04-23T16:25:46.776 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.776 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24012343:::benchmark_data_smithi188_89276_object55113:head# 2024-04-23T16:25:46.776 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.776 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2401326e:::benchmark_data_smithi188_89276_object107458:head# 2024-04-23T16:25:46.777 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.777 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:240132d1:::benchmark_data_smithi188_89276_object54473:head# 2024-04-23T16:25:46.777 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.777 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2401636d:::benchmark_data_smithi188_89276_object140705:head# 2024-04-23T16:25:46.777 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.777 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24017103:::benchmark_data_smithi188_89276_object125302:head# 2024-04-23T16:25:46.777 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.778 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2401877a:::benchmark_data_smithi188_89276_object114676:head# 2024-04-23T16:25:46.778 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.778 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2401d35b:::benchmark_data_smithi188_89276_object81362:head# 2024-04-23T16:25:46.778 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.778 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2401f02a:::benchmark_data_smithi188_89276_object143617:head# 2024-04-23T16:25:46.778 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24036d68:::benchmark_data_smithi188_89276_object167134:head# 2024-04-23T16:25:46.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2404b786:::benchmark_data_smithi188_89276_object64181:head# 2024-04-23T16:25:46.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2404f261:::benchmark_data_smithi188_89276_object56077:head# 2024-04-23T16:25:46.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2405834c:::benchmark_data_smithi188_89276_object87400:head# 2024-04-23T16:25:46.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24060ec4:::benchmark_data_smithi188_89276_object146148:head# 2024-04-23T16:25:46.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24077fc6:::benchmark_data_smithi188_89276_object110211:head# 2024-04-23T16:25:46.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.783 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24086db2:::benchmark_data_smithi188_89276_object47855:head# 2024-04-23T16:25:46.783 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.783 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2408b32c:::benchmark_data_smithi188_89276_object138397:head# 2024-04-23T16:25:46.783 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24095f77:::benchmark_data_smithi188_89276_object48098:head# 2024-04-23T16:25:46.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24097b41:::benchmark_data_smithi188_89276_object82682:head# 2024-04-23T16:25:46.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2409c51f:::benchmark_data_smithi188_89276_object144210:head# 2024-04-23T16:25:46.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:240a7d92:::benchmark_data_smithi188_89276_object161482:head# 2024-04-23T16:25:46.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:240a995d:::benchmark_data_smithi188_89276_object86674:head# 2024-04-23T16:25:46.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:240af54d:::benchmark_data_smithi188_89276_object117842:head# 2024-04-23T16:25:46.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:240b8127:::benchmark_data_smithi188_89276_object71822:head# 2024-04-23T16:25:46.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:240bb2d1:::benchmark_data_smithi188_89276_object145028:head# 2024-04-23T16:25:46.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:240bdd2e:::benchmark_data_smithi188_89276_object42313:head# 2024-04-23T16:25:46.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:240c3fec:::benchmark_data_smithi188_89276_object58358:head# 2024-04-23T16:25:46.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:240e5f28:::benchmark_data_smithi188_89276_object81525:head# 2024-04-23T16:25:46.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:240ed26a:::benchmark_data_smithi188_89276_object60257:head# 2024-04-23T16:25:46.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:240f470e:::benchmark_data_smithi188_89276_object142357:head# 2024-04-23T16:25:46.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:240faf25:::benchmark_data_smithi188_89276_object49252:head# 2024-04-23T16:25:46.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:241026e9:::benchmark_data_smithi188_89276_object86556:head# 2024-04-23T16:25:46.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24104dab:::benchmark_data_smithi188_89276_object122940:head# 2024-04-23T16:25:46.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2410cdf7:::benchmark_data_smithi188_89276_object86875:head# 2024-04-23T16:25:46.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24116c32:::benchmark_data_smithi188_89276_object136345:head# 2024-04-23T16:25:46.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24116d8a:::benchmark_data_smithi188_89276_object89278:head# 2024-04-23T16:25:46.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:241242bf:::benchmark_data_smithi188_89276_object117362:head# 2024-04-23T16:25:46.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:241284d3:::benchmark_data_smithi188_89276_object89355:head# 2024-04-23T16:25:46.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2412d887:::benchmark_data_smithi188_89276_object62592:head# 2024-04-23T16:25:46.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:241559f1:::benchmark_data_smithi188_89276_object73424:head# 2024-04-23T16:25:46.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24158ace:::benchmark_data_smithi188_89276_object121569:head# 2024-04-23T16:25:46.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:241681ee:::benchmark_data_smithi188_89276_object80397:head# 2024-04-23T16:25:46.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24171ee6:::benchmark_data_smithi188_89276_object166243:head# 2024-04-23T16:25:46.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:241761a9:::benchmark_data_smithi188_89276_object137767:head# 2024-04-23T16:25:46.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2418d279:::benchmark_data_smithi188_89276_object130915:head# 2024-04-23T16:25:46.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2418e30a:::benchmark_data_smithi188_89276_object69824:head# 2024-04-23T16:25:46.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2418e9aa:::benchmark_data_smithi188_89276_object94914:head# 2024-04-23T16:25:46.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2419451b:::benchmark_data_smithi188_89276_object144075:head# 2024-04-23T16:25:46.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24196e84:::benchmark_data_smithi188_89276_object65859:head# 2024-04-23T16:25:46.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2419a49e:::benchmark_data_smithi188_89276_object80730:head# 2024-04-23T16:25:46.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2419f076:::benchmark_data_smithi188_89276_object69252:head# 2024-04-23T16:25:46.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:241a105e:::benchmark_data_smithi188_89276_object155587:head# 2024-04-23T16:25:46.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:241a4504:::benchmark_data_smithi188_89276_object108596:head# 2024-04-23T16:25:46.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:241a58eb:::benchmark_data_smithi188_89276_object74263:head# 2024-04-23T16:25:46.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:241b390a:::benchmark_data_smithi188_89276_object135245:head# 2024-04-23T16:25:46.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:241b8a0d:::benchmark_data_smithi188_89276_object136933:head# 2024-04-23T16:25:46.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:241babf4:::benchmark_data_smithi188_89276_object83445:head# 2024-04-23T16:25:46.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:241d8543:::benchmark_data_smithi188_89276_object153720:head# 2024-04-23T16:25:46.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:241dee10:::benchmark_data_smithi188_89276_object132300:head# 2024-04-23T16:25:46.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:241e068d:::benchmark_data_smithi188_89276_object119829:head# 2024-04-23T16:25:46.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:241f0731:::benchmark_data_smithi188_89276_object78091:head# 2024-04-23T16:25:46.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:241fbc07:::benchmark_data_smithi188_89276_object82824:head# 2024-04-23T16:25:46.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24204665:::benchmark_data_smithi188_89276_object69559:head# 2024-04-23T16:25:46.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24208f88:::benchmark_data_smithi188_89276_object80111:head# 2024-04-23T16:25:46.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.796 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242093e8:::benchmark_data_smithi188_89276_object122090:head# 2024-04-23T16:25:46.796 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.796 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2420991c:::benchmark_data_smithi188_89276_object149814:head# 2024-04-23T16:25:46.796 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.796 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242171d7:::benchmark_data_smithi188_89276_object40654:head# 2024-04-23T16:25:46.796 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.796 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2421c6ae:::benchmark_data_smithi188_89276_object67444:head# 2024-04-23T16:25:46.796 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.796 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24226ab1:::benchmark_data_smithi188_89276_object42649:head# 2024-04-23T16:25:46.796 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2422a5e8:::benchmark_data_smithi188_89276_object82655:head# 2024-04-23T16:25:46.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2422ff05:::benchmark_data_smithi188_89276_object101418:head# 2024-04-23T16:25:46.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242301db:::benchmark_data_smithi188_89276_object43221:head# 2024-04-23T16:25:46.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2423a149:::benchmark_data_smithi188_89276_object152189:head# 2024-04-23T16:25:46.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2423dd9d:::benchmark_data_smithi188_89276_object80756:head# 2024-04-23T16:25:46.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2423f746:::benchmark_data_smithi188_89276_object57505:head# 2024-04-23T16:25:46.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24244826:::benchmark_data_smithi188_89276_object159879:head# 2024-04-23T16:25:46.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24245a8b:::benchmark_data_smithi188_89276_object68481:head# 2024-04-23T16:25:46.798 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.798 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2424c110:::benchmark_data_smithi188_89276_object140526:head# 2024-04-23T16:25:46.798 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.798 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2425515f:::benchmark_data_smithi188_89276_object74576:head# 2024-04-23T16:25:46.798 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.798 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2425b6cd:::benchmark_data_smithi188_89276_object164873:head# 2024-04-23T16:25:46.798 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2425d419:::benchmark_data_smithi188_89276_object53965:head# 2024-04-23T16:25:46.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2425eb60:::benchmark_data_smithi188_89276_object95435:head# 2024-04-23T16:25:46.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24279682:::benchmark_data_smithi188_89276_object90811:head# 2024-04-23T16:25:46.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2427add2:::benchmark_data_smithi188_89276_object162265:head# 2024-04-23T16:25:46.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242928e1:::benchmark_data_smithi188_89276_object50726:head# 2024-04-23T16:25:46.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24293cf1:::benchmark_data_smithi188_89276_object81475:head# 2024-04-23T16:25:46.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242adb40:::benchmark_data_smithi188_89276_object125254:head# 2024-04-23T16:25:46.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.803 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242b26e0:::benchmark_data_smithi188_89276_object148632:head# 2024-04-23T16:25:46.803 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.803 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242b5376:::benchmark_data_smithi188_89276_object45675:head# 2024-04-23T16:25:46.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242b8d25:::benchmark_data_smithi188_89276_object148028:head# 2024-04-23T16:25:46.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242ba5c6:::benchmark_data_smithi188_89276_object71010:head# 2024-04-23T16:25:46.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242bb4e7:::benchmark_data_smithi188_89276_object85440:head# 2024-04-23T16:25:46.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242bc1d9:::benchmark_data_smithi188_89276_object125122:head# 2024-04-23T16:25:46.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242bc99b:::benchmark_data_smithi188_89276_object94880:head# 2024-04-23T16:25:46.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242bea01:::benchmark_data_smithi188_89276_object163910:head# 2024-04-23T16:25:46.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242c40a7:::benchmark_data_smithi188_89276_object111452:head# 2024-04-23T16:25:46.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242c9e3a:::benchmark_data_smithi188_89276_object79686:head# 2024-04-23T16:25:46.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242cf77a:::benchmark_data_smithi188_89276_object126862:head# 2024-04-23T16:25:46.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242d3440:::benchmark_data_smithi188_89276_object119228:head# 2024-04-23T16:25:46.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242d6ac5:::benchmark_data_smithi188_89276_object103304:head# 2024-04-23T16:25:46.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242d7191:::benchmark_data_smithi188_89276_object99740:head# 2024-04-23T16:25:46.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242e5bd6:::benchmark_data_smithi188_89276_object157814:head# 2024-04-23T16:25:46.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242ec586:::benchmark_data_smithi188_89276_object59770:head# 2024-04-23T16:25:46.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242f5a2c:::benchmark_data_smithi188_89276_object147503:head# 2024-04-23T16:25:46.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:242f8678:::benchmark_data_smithi188_89276_object137214:head# 2024-04-23T16:25:46.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24307a56:::benchmark_data_smithi188_89276_object122978:head# 2024-04-23T16:25:46.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2431807d:::benchmark_data_smithi188_89276_object106094:head# 2024-04-23T16:25:46.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24319340:::benchmark_data_smithi188_89276_object140038:head# 2024-04-23T16:25:46.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2431c33f:::benchmark_data_smithi188_89276_object54261:head# 2024-04-23T16:25:46.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243310fa:::benchmark_data_smithi188_89276_object78597:head# 2024-04-23T16:25:46.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243424b4:::benchmark_data_smithi188_89276_object121301:head# 2024-04-23T16:25:46.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24349a9f:::benchmark_data_smithi188_89276_object49276:head# 2024-04-23T16:25:46.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243522bb:::benchmark_data_smithi188_89276_object150317:head# 2024-04-23T16:25:46.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2435e4de:::benchmark_data_smithi188_89276_object120362:head# 2024-04-23T16:25:46.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2436abaf:::benchmark_data_smithi188_89276_object66357:head# 2024-04-23T16:25:46.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24382565:::benchmark_data_smithi188_89276_object161729:head# 2024-04-23T16:25:46.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24382b4c:::benchmark_data_smithi188_89276_object163757:head# 2024-04-23T16:25:46.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243873eb:::benchmark_data_smithi188_89276_object118929:head# 2024-04-23T16:25:46.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2439e289:::benchmark_data_smithi188_89276_object125372:head# 2024-04-23T16:25:46.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243a31fc:::benchmark_data_smithi188_89276_object153281:head# 2024-04-23T16:25:46.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.816 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243a6438:::benchmark_data_smithi188_89276_object104445:head# 2024-04-23T16:25:46.816 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243a6481:::benchmark_data_smithi188_89276_object116152:head# 2024-04-23T16:25:46.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243a6982:::benchmark_data_smithi188_89276_object72752:head# 2024-04-23T16:25:46.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243a7f21:::benchmark_data_smithi188_89276_object124841:head# 2024-04-23T16:25:46.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243b61a3:::benchmark_data_smithi188_89276_object62908:head# 2024-04-23T16:25:46.824 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243b8314:::benchmark_data_smithi188_89276_object67559:head# 2024-04-23T16:25:46.824 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.825 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243baad6:::benchmark_data_smithi188_89276_object129893:head# 2024-04-23T16:25:46.825 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.825 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243c0d09:::benchmark_data_smithi188_89276_object114651:head# 2024-04-23T16:25:46.825 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243c78b0:::benchmark_data_smithi188_89276_object159986:head# 2024-04-23T16:25:46.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243d104d:::benchmark_data_smithi188_89276_object76221:head# 2024-04-23T16:25:46.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243d65ce:::benchmark_data_smithi188_89276_object43595:head# 2024-04-23T16:25:46.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243e3010:::benchmark_data_smithi188_89276_object137270:head# 2024-04-23T16:25:46.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243f2466:::benchmark_data_smithi188_89276_object60652:head# 2024-04-23T16:25:46.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243f86d3:::benchmark_data_smithi188_89276_object92084:head# 2024-04-23T16:25:46.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:243fc182:::benchmark_data_smithi188_89276_object57863:head# 2024-04-23T16:25:46.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2440c65f:::benchmark_data_smithi188_89276_object154995:head# 2024-04-23T16:25:46.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2441096f:::benchmark_data_smithi188_89276_object78018:head# 2024-04-23T16:25:46.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24412d74:::benchmark_data_smithi188_89276_object73479:head# 2024-04-23T16:25:46.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:244193f6:::benchmark_data_smithi188_89276_object91498:head# 2024-04-23T16:25:46.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2441b021:::benchmark_data_smithi188_89276_object114677:head# 2024-04-23T16:25:46.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24422bca:::benchmark_data_smithi188_89276_object45741:head# 2024-04-23T16:25:46.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.833 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2442b7f6:::benchmark_data_smithi188_89276_object154714:head# 2024-04-23T16:25:46.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.834 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24431e03:::benchmark_data_smithi188_89276_object83534:head# 2024-04-23T16:25:46.834 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.839 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2443ced6:::benchmark_data_smithi188_89276_object81609:head# 2024-04-23T16:25:46.839 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.841 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:244415c0:::benchmark_data_smithi188_89276_object148114:head# 2024-04-23T16:25:46.841 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.841 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2444295b:::benchmark_data_smithi188_89276_object116878:head# 2024-04-23T16:25:46.841 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.841 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2444518d:::benchmark_data_smithi188_89276_object66019:head# 2024-04-23T16:25:46.842 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.842 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2444dee3:::benchmark_data_smithi188_89276_object123568:head# 2024-04-23T16:25:46.842 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.842 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2445b568:::benchmark_data_smithi188_89276_object119101:head# 2024-04-23T16:25:46.842 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.842 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2445d3ac:::benchmark_data_smithi188_89276_object151052:head# 2024-04-23T16:25:46.842 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24462c73:::benchmark_data_smithi188_89276_object143458:head# 2024-04-23T16:25:46.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24470edc:::benchmark_data_smithi188_89276_object57014:head# 2024-04-23T16:25:46.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2447683a:::benchmark_data_smithi188_89276_object113686:head# 2024-04-23T16:25:46.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:244787bd:::benchmark_data_smithi188_89276_object79723:head# 2024-04-23T16:25:46.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2447ed36:::benchmark_data_smithi188_89276_object127192:head# 2024-04-23T16:25:46.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24483115:::benchmark_data_smithi188_89276_object114264:head# 2024-04-23T16:25:46.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24484c93:::benchmark_data_smithi188_89276_object146284:head# 2024-04-23T16:25:46.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24485f20:::benchmark_data_smithi188_89276_object132874:head# 2024-04-23T16:25:46.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2448ca8b:::benchmark_data_smithi188_89276_object155817:head# 2024-04-23T16:25:46.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:244a457b:::benchmark_data_smithi188_89276_object158903:head# 2024-04-23T16:25:46.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:244a9193:::benchmark_data_smithi188_89276_object146744:head# 2024-04-23T16:25:46.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:244b74ff:::benchmark_data_smithi188_89276_object77362:head# 2024-04-23T16:25:46.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:244b7e8e:::benchmark_data_smithi188_89276_object157518:head# 2024-04-23T16:25:46.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:244c0ff8:::benchmark_data_smithi188_89276_object103203:head# 2024-04-23T16:25:46.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:244c95cd:::benchmark_data_smithi188_89276_object142186:head# 2024-04-23T16:25:46.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.853 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:244cf726:::benchmark_data_smithi188_89276_object161680:head# 2024-04-23T16:25:46.853 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.853 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:244d2deb:::benchmark_data_smithi188_89276_object64241:head# 2024-04-23T16:25:46.853 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.853 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:244f0ffc:::benchmark_data_smithi188_89276_object86678:head# 2024-04-23T16:25:46.853 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:244f9531:::benchmark_data_smithi188_89276_object150113:head# 2024-04-23T16:25:46.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:244fb72e:::benchmark_data_smithi188_89276_object130329:head# 2024-04-23T16:25:46.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:244fb9f1:::benchmark_data_smithi188_89276_object147279:head# 2024-04-23T16:25:46.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:245008ac:::benchmark_data_smithi188_89276_object130180:head# 2024-04-23T16:25:46.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24500f88:::benchmark_data_smithi188_89276_object148355:head# 2024-04-23T16:25:46.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24508e81:::benchmark_data_smithi188_89276_object71113:head# 2024-04-23T16:25:46.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2450c60f:::benchmark_data_smithi188_89276_object105247:head# 2024-04-23T16:25:46.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.856 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24521a2d:::benchmark_data_smithi188_89276_object152228:head# 2024-04-23T16:25:46.856 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.857 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24523460:::benchmark_data_smithi188_89276_object134876:head# 2024-04-23T16:25:46.857 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.857 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:245263af:::benchmark_data_smithi188_89276_object62344:head# 2024-04-23T16:25:46.857 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.858 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2452e154:::benchmark_data_smithi188_89276_object164515:head# 2024-04-23T16:25:46.858 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.858 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2452ee37:::benchmark_data_smithi188_89276_object45787:head# 2024-04-23T16:25:46.858 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.858 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24532272:::benchmark_data_smithi188_89276_object130890:head# 2024-04-23T16:25:46.858 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.859 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24532382:::benchmark_data_smithi188_89276_object143353:head# 2024-04-23T16:25:46.859 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.859 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2454e2e5:::benchmark_data_smithi188_89276_object159566:head# 2024-04-23T16:25:46.859 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.859 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2454f4d2:::benchmark_data_smithi188_89276_object80317:head# 2024-04-23T16:25:46.859 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.859 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24553952:::benchmark_data_smithi188_89276_object132907:head# 2024-04-23T16:25:46.859 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.860 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24557557:::benchmark_data_smithi188_89276_object101144:head# 2024-04-23T16:25:46.860 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.860 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24560ebb:::benchmark_data_smithi188_89276_object106459:head# 2024-04-23T16:25:46.861 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.861 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2456298a:::benchmark_data_smithi188_89276_object94283:head# 2024-04-23T16:25:46.861 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.861 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2456f42f:::benchmark_data_smithi188_89276_object41947:head# 2024-04-23T16:25:46.861 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.861 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2456f5bb:::benchmark_data_smithi188_89276_object111075:head# 2024-04-23T16:25:46.861 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.861 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2457654b:::benchmark_data_smithi188_89276_object147071:head# 2024-04-23T16:25:46.861 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2459ea07:::benchmark_data_smithi188_89276_object166260:head# 2024-04-23T16:25:46.862 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:245baca2:::benchmark_data_smithi188_89276_object135192:head# 2024-04-23T16:25:46.862 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:245c333e:::benchmark_data_smithi188_89276_object47847:head# 2024-04-23T16:25:46.862 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:245c829b:::benchmark_data_smithi188_89276_object58854:head# 2024-04-23T16:25:46.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:245cb2d8:::benchmark_data_smithi188_89276_object139147:head# 2024-04-23T16:25:46.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:245e881d:::benchmark_data_smithi188_89276_object118234:head# 2024-04-23T16:25:46.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:245f45b7:::benchmark_data_smithi188_89276_object166276:head# 2024-04-23T16:25:46.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:246008ee:::benchmark_data_smithi188_89276_object136822:head# 2024-04-23T16:25:46.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24601567:::benchmark_data_smithi188_89276_object145635:head# 2024-04-23T16:25:46.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24612939:::benchmark_data_smithi188_89276_object79786:head# 2024-04-23T16:25:46.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:246135c2:::benchmark_data_smithi188_89276_object164649:head# 2024-04-23T16:25:46.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:246154d8:::benchmark_data_smithi188_89276_object156897:head# 2024-04-23T16:25:46.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2462ecca:::benchmark_data_smithi188_89276_object43718:head# 2024-04-23T16:25:46.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24631a22:::benchmark_data_smithi188_89276_object103171:head# 2024-04-23T16:25:46.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2463c780:::benchmark_data_smithi188_89276_object90395:head# 2024-04-23T16:25:46.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2464f56c:::benchmark_data_smithi188_89276_object71775:head# 2024-04-23T16:25:46.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:246509a8:::benchmark_data_smithi188_89276_object116647:head# 2024-04-23T16:25:46.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24663db3:::benchmark_data_smithi188_89276_object46241:head# 2024-04-23T16:25:46.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24666ad5:::benchmark_data_smithi188_89276_object44264:head# 2024-04-23T16:25:46.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24675a1b:::benchmark_data_smithi188_89276_object163921:head# 2024-04-23T16:25:46.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2467c126:::benchmark_data_smithi188_89276_object160203:head# 2024-04-23T16:25:46.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2468bdfc:::benchmark_data_smithi188_89276_object51053:head# 2024-04-23T16:25:46.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2468d024:::benchmark_data_smithi188_89276_object136916:head# 2024-04-23T16:25:46.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2468fc62:::benchmark_data_smithi188_89276_object124761:head# 2024-04-23T16:25:46.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24690a2c:::benchmark_data_smithi188_89276_object64951:head# 2024-04-23T16:25:46.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2469aeaa:::benchmark_data_smithi188_89276_object133118:head# 2024-04-23T16:25:46.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:246aca71:::benchmark_data_smithi188_89276_object167098:head# 2024-04-23T16:25:46.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:246b236a:::benchmark_data_smithi188_89276_object150477:head# 2024-04-23T16:25:46.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:246b6c5c:::benchmark_data_smithi188_89276_object154490:head# 2024-04-23T16:25:46.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:246b8c6f:::benchmark_data_smithi188_89276_object81611:head# 2024-04-23T16:25:46.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:246d22db:::benchmark_data_smithi188_89276_object101892:head# 2024-04-23T16:25:46.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:246d501f:::benchmark_data_smithi188_89276_object137197:head# 2024-04-23T16:25:46.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:246d7dc0:::benchmark_data_smithi188_89276_object58686:head# 2024-04-23T16:25:46.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:246d9137:::benchmark_data_smithi188_89276_object59848:head# 2024-04-23T16:25:46.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:246dce8d:::benchmark_data_smithi188_89276_object151370:head# 2024-04-23T16:25:46.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:246e0af9:::benchmark_data_smithi188_89276_object66183:head# 2024-04-23T16:25:46.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:246e45b0:::benchmark_data_smithi188_89276_object132123:head# 2024-04-23T16:25:46.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:246ec0df:::benchmark_data_smithi188_89276_object116895:head# 2024-04-23T16:25:46.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:246f557b:::benchmark_data_smithi188_89276_object75777:head# 2024-04-23T16:25:46.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:246f8c9f:::benchmark_data_smithi188_89276_object62584:head# 2024-04-23T16:25:46.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2470818b:::benchmark_data_smithi188_89276_object125524:head# 2024-04-23T16:25:46.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2470a0a1:::benchmark_data_smithi188_89276_object128024:head# 2024-04-23T16:25:46.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2470caa5:::benchmark_data_smithi188_89276_object78913:head# 2024-04-23T16:25:46.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24711c14:::benchmark_data_smithi188_89276_object41410:head# 2024-04-23T16:25:46.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24722f3e:::benchmark_data_smithi188_89276_object76949:head# 2024-04-23T16:25:46.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24739b35:::benchmark_data_smithi188_89276_object127487:head# 2024-04-23T16:25:46.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2473ced4:::benchmark_data_smithi188_89276_object69077:head# 2024-04-23T16:25:46.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2474f0f7:::benchmark_data_smithi188_89276_object149876:head# 2024-04-23T16:25:46.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24753833:::benchmark_data_smithi188_89276_object60533:head# 2024-04-23T16:25:46.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2475423f:::benchmark_data_smithi188_89276_object65917:head# 2024-04-23T16:25:46.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24764e4e:::benchmark_data_smithi188_89276_object117363:head# 2024-04-23T16:25:46.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:247808a3:::benchmark_data_smithi188_89276_object42532:head# 2024-04-23T16:25:46.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24797ad5:::benchmark_data_smithi188_89276_object151027:head# 2024-04-23T16:25:46.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:247a44fa:::benchmark_data_smithi188_89276_object85154:head# 2024-04-23T16:25:46.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:247a6756:::benchmark_data_smithi188_89276_object149080:head# 2024-04-23T16:25:46.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:247b650f:::benchmark_data_smithi188_89276_object97953:head# 2024-04-23T16:25:46.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:247cef35:::benchmark_data_smithi188_89276_object143848:head# 2024-04-23T16:25:46.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:247d1052:::benchmark_data_smithi188_89276_object63641:head# 2024-04-23T16:25:46.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:247d851a:::benchmark_data_smithi188_89276_object100014:head# 2024-04-23T16:25:46.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:247d8775:::benchmark_data_smithi188_89276_object116114:head# 2024-04-23T16:25:46.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:247e69b2:::benchmark_data_smithi188_89276_object55106:head# 2024-04-23T16:25:46.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:247ec81e:::benchmark_data_smithi188_89276_object80183:head# 2024-04-23T16:25:46.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:247f41b7:::benchmark_data_smithi188_89276_object58786:head# 2024-04-23T16:25:46.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:247f463b:::benchmark_data_smithi188_89276_object68005:head# 2024-04-23T16:25:46.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2480a080:::benchmark_data_smithi188_89276_object150948:head# 2024-04-23T16:25:46.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2480a4b8:::benchmark_data_smithi188_89276_object98348:head# 2024-04-23T16:25:46.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24811405:::benchmark_data_smithi188_89276_object47025:head# 2024-04-23T16:25:46.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2481338c:::benchmark_data_smithi188_89276_object105121:head# 2024-04-23T16:25:46.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2481a77a:::benchmark_data_smithi188_89276_object165875:head# 2024-04-23T16:25:46.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24824aee:::benchmark_data_smithi188_89276_object40515:head# 2024-04-23T16:25:46.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24835f5c:::benchmark_data_smithi188_89276_object103901:head# 2024-04-23T16:25:46.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:248391b8:::benchmark_data_smithi188_89276_object124638:head# 2024-04-23T16:25:46.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:248397bc:::benchmark_data_smithi188_89276_object123479:head# 2024-04-23T16:25:46.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24848f20:::benchmark_data_smithi188_89276_object87185:head# 2024-04-23T16:25:46.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2484c59f:::benchmark_data_smithi188_89276_object137459:head# 2024-04-23T16:25:46.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:248604e8:::benchmark_data_smithi188_89276_object139518:head# 2024-04-23T16:25:46.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24869453:::benchmark_data_smithi188_89276_object133935:head# 2024-04-23T16:25:46.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24876ffe:::benchmark_data_smithi188_89276_object103579:head# 2024-04-23T16:25:46.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:248a142f:::benchmark_data_smithi188_89276_object165684:head# 2024-04-23T16:25:46.884 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:248aa3cf:::benchmark_data_smithi188_89276_object42585:head# 2024-04-23T16:25:46.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:248aa733:::benchmark_data_smithi188_89276_object118003:head# 2024-04-23T16:25:46.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:248ad04b:::benchmark_data_smithi188_89276_object108604:head# 2024-04-23T16:25:46.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:248adf91:::benchmark_data_smithi188_89276_object120762:head# 2024-04-23T16:25:46.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:248b43c1:::benchmark_data_smithi188_89276_object161389:head# 2024-04-23T16:25:46.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:248ccb9a:::benchmark_data_smithi188_89276_object153200:head# 2024-04-23T16:25:46.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:248fe693:::benchmark_data_smithi188_89276_object131730:head# 2024-04-23T16:25:46.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24909734:::benchmark_data_smithi188_89276_object159560:head# 2024-04-23T16:25:46.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24909e5f:::benchmark_data_smithi188_89276_object66668:head# 2024-04-23T16:25:46.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2490a2ca:::benchmark_data_smithi188_89276_object131414:head# 2024-04-23T16:25:46.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2491340b:::benchmark_data_smithi188_89276_object66564:head# 2024-04-23T16:25:46.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2492b145:::benchmark_data_smithi188_89276_object80193:head# 2024-04-23T16:25:46.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24931c44:::benchmark_data_smithi188_89276_object56670:head# 2024-04-23T16:25:46.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.893 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2493b53c:::benchmark_data_smithi188_89276_object168099:head# 2024-04-23T16:25:46.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2494150b:::benchmark_data_smithi188_89276_object136512:head# 2024-04-23T16:25:46.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24949bff:::benchmark_data_smithi188_89276_object61211:head# 2024-04-23T16:25:46.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2494f865:::benchmark_data_smithi188_89276_object133315:head# 2024-04-23T16:25:46.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24965be8:::benchmark_data_smithi188_89276_object68378:head# 2024-04-23T16:25:46.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24981d3a:::benchmark_data_smithi188_89276_object72576:head# 2024-04-23T16:25:46.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2498287a:::benchmark_data_smithi188_89276_object166639:head# 2024-04-23T16:25:46.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24983ef0:::benchmark_data_smithi188_89276_object115299:head# 2024-04-23T16:25:46.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2498b2df:::benchmark_data_smithi188_89276_object122392:head# 2024-04-23T16:25:46.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24991a9e:::benchmark_data_smithi188_89276_object128770:head# 2024-04-23T16:25:46.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24993dac:::benchmark_data_smithi188_89276_object81587:head# 2024-04-23T16:25:46.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:249959f1:::benchmark_data_smithi188_89276_object115379:head# 2024-04-23T16:25:46.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:249a5c11:::benchmark_data_smithi188_89276_object52641:head# 2024-04-23T16:25:46.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:249c3177:::benchmark_data_smithi188_89276_object98455:head# 2024-04-23T16:25:46.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:249c3f38:::benchmark_data_smithi188_89276_object53648:head# 2024-04-23T16:25:46.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:249cff30:::benchmark_data_smithi188_89276_object154710:head# 2024-04-23T16:25:46.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:249e0d28:::benchmark_data_smithi188_89276_object69945:head# 2024-04-23T16:25:46.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:249e6c69:::benchmark_data_smithi188_89276_object104458:head# 2024-04-23T16:25:46.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:249ebe0b:::benchmark_data_smithi188_89276_object116084:head# 2024-04-23T16:25:46.914 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.914 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:249ed835:::benchmark_data_smithi188_89276_object46332:head# 2024-04-23T16:25:46.914 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.914 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24a1da5d:::benchmark_data_smithi188_89276_object105551:head# 2024-04-23T16:25:46.914 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.914 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24a24305:::benchmark_data_smithi188_89276_object67132:head# 2024-04-23T16:25:46.914 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.916 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24a2c15f:::benchmark_data_smithi188_89276_object59366:head# 2024-04-23T16:25:46.916 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.916 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24a35dce:::benchmark_data_smithi188_89276_object121029:head# 2024-04-23T16:25:46.916 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.916 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24a38f49:::benchmark_data_smithi188_89276_object85495:head# 2024-04-23T16:25:46.916 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24a3a91d:::benchmark_data_smithi188_89276_object67576:head# 2024-04-23T16:25:46.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24a41f32:::benchmark_data_smithi188_89276_object58574:head# 2024-04-23T16:25:46.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24a43e4b:::benchmark_data_smithi188_89276_object94765:head# 2024-04-23T16:25:46.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24a44439:::benchmark_data_smithi188_89276_object46914:head# 2024-04-23T16:25:46.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24a4a26b:::benchmark_data_smithi188_89276_object109785:head# 2024-04-23T16:25:46.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24a56476:::benchmark_data_smithi188_89276_object68134:head# 2024-04-23T16:25:46.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24a5cf7a:::benchmark_data_smithi188_89276_object139407:head# 2024-04-23T16:25:46.918 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.918 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24a72c9e:::benchmark_data_smithi188_89276_object75778:head# 2024-04-23T16:25:46.918 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24a7b51b:::benchmark_data_smithi188_89276_object136591:head# 2024-04-23T16:25:46.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.929 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24a9a1b2:::benchmark_data_smithi188_89276_object92191:head# 2024-04-23T16:25:46.929 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24ab3978:::benchmark_data_smithi188_89276_object125898:head# 2024-04-23T16:25:46.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24ac4d2f:::benchmark_data_smithi188_89276_object77809:head# 2024-04-23T16:25:46.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24aded27:::benchmark_data_smithi188_89276_object93842:head# 2024-04-23T16:25:46.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24adfe27:::benchmark_data_smithi188_89276_object54111:head# 2024-04-23T16:25:46.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24ae9a0b:::benchmark_data_smithi188_89276_object61675:head# 2024-04-23T16:25:46.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24aeef10:::benchmark_data_smithi188_89276_object140055:head# 2024-04-23T16:25:46.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24affd6a:::benchmark_data_smithi188_89276_object59829:head# 2024-04-23T16:25:46.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24b0306e:::benchmark_data_smithi188_89276_object132525:head# 2024-04-23T16:25:46.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24b0580d:::benchmark_data_smithi188_89276_object116865:head# 2024-04-23T16:25:46.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24b0ce94:::benchmark_data_smithi188_89276_object48558:head# 2024-04-23T16:25:46.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24b18c1f:::benchmark_data_smithi188_89276_object166598:head# 2024-04-23T16:25:46.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24b26cc7:::benchmark_data_smithi188_89276_object40964:head# 2024-04-23T16:25:46.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24b41e96:::benchmark_data_smithi188_89276_object122051:head# 2024-04-23T16:25:46.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24b5100e:::benchmark_data_smithi188_89276_object48224:head# 2024-04-23T16:25:46.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24b587ba:::benchmark_data_smithi188_89276_object141158:head# 2024-04-23T16:25:46.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24b5a11a:::benchmark_data_smithi188_89276_object101079:head# 2024-04-23T16:25:46.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24b5d2e4:::benchmark_data_smithi188_89276_object65920:head# 2024-04-23T16:25:46.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24b6294a:::benchmark_data_smithi188_89276_object93410:head# 2024-04-23T16:25:46.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24b62c5b:::benchmark_data_smithi188_89276_object144377:head# 2024-04-23T16:25:46.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24b6a2f3:::benchmark_data_smithi188_89276_object106551:head# 2024-04-23T16:25:46.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24b74528:::benchmark_data_smithi188_89276_object71879:head# 2024-04-23T16:25:46.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24b7c91d:::benchmark_data_smithi188_89276_object112514:head# 2024-04-23T16:25:46.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24b8cbf7:::benchmark_data_smithi188_89276_object123358:head# 2024-04-23T16:25:46.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24ba3b93:::benchmark_data_smithi188_89276_object48544:head# 2024-04-23T16:25:46.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24ba6744:::benchmark_data_smithi188_89276_object117557:head# 2024-04-23T16:25:46.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24ba9ea4:::benchmark_data_smithi188_89276_object44240:head# 2024-04-23T16:25:46.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24bb124b:::benchmark_data_smithi188_89276_object119171:head# 2024-04-23T16:25:46.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24bb16fc:::benchmark_data_smithi188_89276_object95116:head# 2024-04-23T16:25:46.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24bb6889:::benchmark_data_smithi188_89276_object77407:head# 2024-04-23T16:25:46.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24bbfc10:::benchmark_data_smithi188_89276_object147991:head# 2024-04-23T16:25:46.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24bc0da9:::benchmark_data_smithi188_89276_object143910:head# 2024-04-23T16:25:46.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24bc7869:::benchmark_data_smithi188_89276_object129237:head# 2024-04-23T16:25:46.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24bcc709:::benchmark_data_smithi188_89276_object138252:head# 2024-04-23T16:25:46.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24bd5ab4:::benchmark_data_smithi188_89276_object68178:head# 2024-04-23T16:25:46.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24bd7cfc:::benchmark_data_smithi188_89276_object130057:head# 2024-04-23T16:25:46.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24be674a:::benchmark_data_smithi188_89276_object76351:head# 2024-04-23T16:25:46.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24bf205e:::benchmark_data_smithi188_89276_object42513:head# 2024-04-23T16:25:46.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24c08d81:::benchmark_data_smithi188_89276_object67008:head# 2024-04-23T16:25:46.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24c1144d:::benchmark_data_smithi188_89276_object110415:head# 2024-04-23T16:25:46.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24c1380c:::benchmark_data_smithi188_89276_object57303:head# 2024-04-23T16:25:46.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24c1bfd7:::benchmark_data_smithi188_89276_object40384:head# 2024-04-23T16:25:46.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24c330b6:::benchmark_data_smithi188_89276_object121940:head# 2024-04-23T16:25:46.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24c37e24:::benchmark_data_smithi188_89276_object127915:head# 2024-04-23T16:25:46.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24c38714:::benchmark_data_smithi188_89276_object149328:head# 2024-04-23T16:25:46.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24c3a0ba:::benchmark_data_smithi188_89276_object155615:head# 2024-04-23T16:25:46.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24c3cb3d:::benchmark_data_smithi188_89276_object118559:head# 2024-04-23T16:25:46.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24c435c5:::benchmark_data_smithi188_89276_object71014:head# 2024-04-23T16:25:46.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24c4450f:::benchmark_data_smithi188_89276_object56018:head# 2024-04-23T16:25:46.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24c482e8:::benchmark_data_smithi188_89276_object59178:head# 2024-04-23T16:25:46.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24c5395b:::benchmark_data_smithi188_89276_object116809:head# 2024-04-23T16:25:46.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24c5652d:::benchmark_data_smithi188_89276_object103909:head# 2024-04-23T16:25:46.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24c5bf95:::benchmark_data_smithi188_89276_object58488:head# 2024-04-23T16:25:46.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24c745a6:::benchmark_data_smithi188_89276_object123312:head# 2024-04-23T16:25:46.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24c8410b:::benchmark_data_smithi188_89276_object81327:head# 2024-04-23T16:25:46.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24c98951:::benchmark_data_smithi188_89276_object100313:head# 2024-04-23T16:25:46.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24c9b321:::benchmark_data_smithi188_89276_object64741:head# 2024-04-23T16:25:46.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24caa369:::benchmark_data_smithi188_89276_object46374:head# 2024-04-23T16:25:46.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24cbe7c2:::benchmark_data_smithi188_89276_object168613:head# 2024-04-23T16:25:46.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24cc5aaf:::benchmark_data_smithi188_89276_object166692:head# 2024-04-23T16:25:46.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24cc9e24:::benchmark_data_smithi188_89276_object163376:head# 2024-04-23T16:25:46.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24cd2a66:::benchmark_data_smithi188_89276_object166378:head# 2024-04-23T16:25:46.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24cd3ade:::benchmark_data_smithi188_89276_object164877:head# 2024-04-23T16:25:46.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24ce5adf:::benchmark_data_smithi188_89276_object151976:head# 2024-04-23T16:25:46.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24cfc0b5:::benchmark_data_smithi188_89276_object51198:head# 2024-04-23T16:25:46.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24d02fbb:::benchmark_data_smithi188_89276_object51560:head# 2024-04-23T16:25:46.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24d1a0e5:::benchmark_data_smithi188_89276_object121191:head# 2024-04-23T16:25:46.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24d2d5cf:::benchmark_data_smithi188_89276_object136183:head# 2024-04-23T16:25:46.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24d2dc7f:::benchmark_data_smithi188_89276_object85258:head# 2024-04-23T16:25:46.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24d2e04c:::benchmark_data_smithi188_89276_object137201:head# 2024-04-23T16:25:46.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.962 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24d36803:::benchmark_data_smithi188_89276_object129986:head# 2024-04-23T16:25:46.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.962 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24d576d6:::benchmark_data_smithi188_89276_object79696:head# 2024-04-23T16:25:46.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.962 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24d64d0d:::benchmark_data_smithi188_89276_object103570:head# 2024-04-23T16:25:46.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.962 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24d66799:::benchmark_data_smithi188_89276_object124402:head# 2024-04-23T16:25:46.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.962 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24d6932e:::benchmark_data_smithi188_89276_object76947:head# 2024-04-23T16:25:46.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.962 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24d77142:::benchmark_data_smithi188_89276_object146186:head# 2024-04-23T16:25:46.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24d9b583:::benchmark_data_smithi188_89276_object47892:head# 2024-04-23T16:25:46.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24dbc905:::benchmark_data_smithi188_89276_object162045:head# 2024-04-23T16:25:46.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.964 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24dbe1c8:::benchmark_data_smithi188_89276_object67057:head# 2024-04-23T16:25:46.965 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.965 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24dc4103:::benchmark_data_smithi188_89276_object103463:head# 2024-04-23T16:25:46.965 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.965 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24dcef84:::benchmark_data_smithi188_89276_object52663:head# 2024-04-23T16:25:46.965 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.965 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24dd7fbb:::benchmark_data_smithi188_89276_object76680:head# 2024-04-23T16:25:46.965 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24dd882f:::benchmark_data_smithi188_89276_object85548:head# 2024-04-23T16:25:46.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24de0b56:::benchmark_data_smithi188_89276_object87105:head# 2024-04-23T16:25:46.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24de3e7e:::benchmark_data_smithi188_89276_object157678:head# 2024-04-23T16:25:46.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24de6b7a:::benchmark_data_smithi188_89276_object127934:head# 2024-04-23T16:25:46.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24de6f5c:::benchmark_data_smithi188_89276_object64322:head# 2024-04-23T16:25:46.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24deadee:::benchmark_data_smithi188_89276_object74663:head# 2024-04-23T16:25:46.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24dfc910:::benchmark_data_smithi188_89276_object87071:head# 2024-04-23T16:25:46.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e173a7:::benchmark_data_smithi188_89276_object158295:head# 2024-04-23T16:25:46.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e18e94:::benchmark_data_smithi188_89276_object73391:head# 2024-04-23T16:25:46.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e1ef01:::benchmark_data_smithi188_89276_object71001:head# 2024-04-23T16:25:46.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e27b3f:::benchmark_data_smithi188_89276_object136772:head# 2024-04-23T16:25:46.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e2afeb:::benchmark_data_smithi188_89276_object164545:head# 2024-04-23T16:25:46.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e2cc61:::benchmark_data_smithi188_89276_object147496:head# 2024-04-23T16:25:46.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e3b1a5:::benchmark_data_smithi188_89276_object88190:head# 2024-04-23T16:25:46.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e4315f:::benchmark_data_smithi188_89276_object100912:head# 2024-04-23T16:25:46.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e44b9c:::benchmark_data_smithi188_89276_object142070:head# 2024-04-23T16:25:46.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e45805:::benchmark_data_smithi188_89276_object102444:head# 2024-04-23T16:25:46.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e47f24:::benchmark_data_smithi188_89276_object84420:head# 2024-04-23T16:25:46.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e4b6d1:::benchmark_data_smithi188_89276_object76416:head# 2024-04-23T16:25:46.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e51022:::benchmark_data_smithi188_89276_object96049:head# 2024-04-23T16:25:46.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e59f73:::benchmark_data_smithi188_89276_object92064:head# 2024-04-23T16:25:46.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e5c7ab:::benchmark_data_smithi188_89276_object64908:head# 2024-04-23T16:25:46.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e5fa56:::benchmark_data_smithi188_89276_object76329:head# 2024-04-23T16:25:46.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e609fd:::benchmark_data_smithi188_89276_object104888:head# 2024-04-23T16:25:46.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e68cd9:::benchmark_data_smithi188_89276_object156308:head# 2024-04-23T16:25:46.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e6b591:::benchmark_data_smithi188_89276_object77919:head# 2024-04-23T16:25:46.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e78760:::benchmark_data_smithi188_89276_object135538:head# 2024-04-23T16:25:46.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e7f31d:::benchmark_data_smithi188_89276_object95720:head# 2024-04-23T16:25:46.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e80035:::benchmark_data_smithi188_89276_object144940:head# 2024-04-23T16:25:46.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e81cc5:::benchmark_data_smithi188_89276_object160398:head# 2024-04-23T16:25:46.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e81e3f:::benchmark_data_smithi188_89276_object117491:head# 2024-04-23T16:25:46.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e87f6e:::benchmark_data_smithi188_89276_object107135:head# 2024-04-23T16:25:46.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e898eb:::benchmark_data_smithi188_89276_object134524:head# 2024-04-23T16:25:46.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e8c0e9:::benchmark_data_smithi188_89276_object118292:head# 2024-04-23T16:25:46.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e8f402:::benchmark_data_smithi188_89276_object102762:head# 2024-04-23T16:25:46.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e95422:::benchmark_data_smithi188_89276_object56679:head# 2024-04-23T16:25:46.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24e96d09:::benchmark_data_smithi188_89276_object77598:head# 2024-04-23T16:25:46.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.981 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24ea8d79:::benchmark_data_smithi188_89276_object58413:head# 2024-04-23T16:25:46.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.981 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24eb31df:::benchmark_data_smithi188_89276_object45157:head# 2024-04-23T16:25:46.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.981 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24ebf336:::benchmark_data_smithi188_89276_object118836:head# 2024-04-23T16:25:46.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.981 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24ed7225:::benchmark_data_smithi188_89276_object151778:head# 2024-04-23T16:25:46.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.982 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24edbbbc:::benchmark_data_smithi188_89276_object168412:head# 2024-04-23T16:25:46.982 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.982 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24edf9f6:::benchmark_data_smithi188_89276_object110723:head# 2024-04-23T16:25:46.982 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.982 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24eeb55e:::benchmark_data_smithi188_89276_object94400:head# 2024-04-23T16:25:46.982 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.982 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24efd722:::benchmark_data_smithi188_89276_object150970:head# 2024-04-23T16:25:46.982 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.982 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24f176c9:::benchmark_data_smithi188_89276_object168607:head# 2024-04-23T16:25:46.983 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.983 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24f1b897:::benchmark_data_smithi188_89276_object90288:head# 2024-04-23T16:25:46.983 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.983 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24f24a34:::benchmark_data_smithi188_89276_object131098:head# 2024-04-23T16:25:46.983 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.983 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24f26db1:::benchmark_data_smithi188_89276_object87155:head# 2024-04-23T16:25:46.984 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.984 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24f39460:::benchmark_data_smithi188_89276_object134970:head# 2024-04-23T16:25:46.984 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.984 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24f4a739:::benchmark_data_smithi188_89276_object159927:head# 2024-04-23T16:25:46.984 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.984 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24f4bdc3:::benchmark_data_smithi188_89276_object65576:head# 2024-04-23T16:25:46.984 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.984 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24f52c92:::benchmark_data_smithi188_89276_object111736:head# 2024-04-23T16:25:46.984 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.984 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24f623d2:::benchmark_data_smithi188_89276_object61284:head# 2024-04-23T16:25:46.985 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.985 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24f6bfe3:::benchmark_data_smithi188_89276_object142760:head# 2024-04-23T16:25:46.985 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.985 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24f71fc2:::benchmark_data_smithi188_89276_object118343:head# 2024-04-23T16:25:46.985 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.985 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24f74e23:::benchmark_data_smithi188_89276_object61152:head# 2024-04-23T16:25:46.985 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.985 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24f7efd2:::benchmark_data_smithi188_89276_object144808:head# 2024-04-23T16:25:46.985 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.991 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24f804b6:::benchmark_data_smithi188_89276_object116631:head# 2024-04-23T16:25:46.991 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.991 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24f86458:::benchmark_data_smithi188_89276_object116511:head# 2024-04-23T16:25:46.991 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.991 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24f8aef5:::benchmark_data_smithi188_89276_object99548:head# 2024-04-23T16:25:46.991 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.991 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24f8db11:::benchmark_data_smithi188_89276_object117004:head# 2024-04-23T16:25:46.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24f914df:::benchmark_data_smithi188_89276_object78849:head# 2024-04-23T16:25:46.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24f9c9b9:::benchmark_data_smithi188_89276_object52404:head# 2024-04-23T16:25:46.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24faaff1:::benchmark_data_smithi188_89276_object71285:head# 2024-04-23T16:25:46.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24fb277c:::benchmark_data_smithi188_89276_object94797:head# 2024-04-23T16:25:46.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24fc1f65:::benchmark_data_smithi188_89276_object149537:head# 2024-04-23T16:25:46.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24fc94e9:::benchmark_data_smithi188_89276_object107583:head# 2024-04-23T16:25:46.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24fcad0a:::benchmark_data_smithi188_89276_object56253:head# 2024-04-23T16:25:46.993 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.993 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24fd0cf4:::benchmark_data_smithi188_89276_object53917:head# 2024-04-23T16:25:46.993 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.993 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24ff3294:::benchmark_data_smithi188_89276_object44686:head# 2024-04-23T16:25:46.993 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.993 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24ff87e9:::benchmark_data_smithi188_89276_object139789:head# 2024-04-23T16:25:46.993 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.993 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:24ffcd5b:::benchmark_data_smithi188_89276_object129802:head# 2024-04-23T16:25:46.993 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.993 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25001534:::benchmark_data_smithi188_89276_object60757:head# 2024-04-23T16:25:46.993 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.993 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25013b28:::benchmark_data_smithi188_89276_object72444:head# 2024-04-23T16:25:46.993 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.994 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25017ac0:::benchmark_data_smithi188_89276_object71046:head# 2024-04-23T16:25:46.994 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.997 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25031795:::benchmark_data_smithi188_89276_object93491:head# 2024-04-23T16:25:46.997 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2503b7ed:::benchmark_data_smithi188_89276_object85010:head# 2024-04-23T16:25:46.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2503e11a:::benchmark_data_smithi188_89276_object100045:head# 2024-04-23T16:25:46.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2503ea25:::benchmark_data_smithi188_89276_object41065:head# 2024-04-23T16:25:46.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:250462cd:::benchmark_data_smithi188_89276_object59998:head# 2024-04-23T16:25:46.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2504a5a5:::benchmark_data_smithi188_89276_object95047:head# 2024-04-23T16:25:46.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2504b7c8:::benchmark_data_smithi188_89276_object136040:head# 2024-04-23T16:25:46.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25057a10:::benchmark_data_smithi188_89276_object128686:head# 2024-04-23T16:25:46.999 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.999 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2505c578:::benchmark_data_smithi188_89276_object158581:head# 2024-04-23T16:25:46.999 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.999 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2506a2f7:::benchmark_data_smithi188_89276_object136568:head# 2024-04-23T16:25:46.999 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.999 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2506c38e:::benchmark_data_smithi188_89276_object66257:head# 2024-04-23T16:25:46.999 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:46.999 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:250792f0:::benchmark_data_smithi188_89276_object75332:head# 2024-04-23T16:25:46.999 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.000 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2507cec0:::benchmark_data_smithi188_89276_object167384:head# 2024-04-23T16:25:47.000 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.002 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25089be2:::benchmark_data_smithi188_89276_object131030:head# 2024-04-23T16:25:47.002 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.003 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2508ded2:::benchmark_data_smithi188_89276_object45874:head# 2024-04-23T16:25:47.003 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.003 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2509556c:::benchmark_data_smithi188_89276_object47822:head# 2024-04-23T16:25:47.003 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.003 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:250a6a1a:::benchmark_data_smithi188_89276_object46731:head# 2024-04-23T16:25:47.003 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.003 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:250a7765:::benchmark_data_smithi188_89276_object89068:head# 2024-04-23T16:25:47.004 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.004 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:250afce9:::benchmark_data_smithi188_89276_object136308:head# 2024-04-23T16:25:47.004 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.004 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:250afde2:::benchmark_data_smithi188_89276_object133213:head# 2024-04-23T16:25:47.004 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.004 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:250b31a4:::benchmark_data_smithi188_89276_object83736:head# 2024-04-23T16:25:47.004 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.004 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:250ba811:::benchmark_data_smithi188_89276_object119961:head# 2024-04-23T16:25:47.004 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.005 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:250bb905:::benchmark_data_smithi188_89276_object135204:head# 2024-04-23T16:25:47.005 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.005 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:250d167e:::benchmark_data_smithi188_89276_object68705:head# 2024-04-23T16:25:47.005 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.005 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:250e8617:::benchmark_data_smithi188_89276_object118449:head# 2024-04-23T16:25:47.005 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.005 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:250ec1c4:::benchmark_data_smithi188_89276_object92868:head# 2024-04-23T16:25:47.005 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.006 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:250ef08d:::benchmark_data_smithi188_89276_object64124:head# 2024-04-23T16:25:47.006 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.006 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:250f6ccc:::benchmark_data_smithi188_89276_object156802:head# 2024-04-23T16:25:47.006 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.010 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25101ac1:::benchmark_data_smithi188_89276_object136358:head# 2024-04-23T16:25:47.010 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.010 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25113d32:::benchmark_data_smithi188_89276_object45550:head# 2024-04-23T16:25:47.010 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.010 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2511a8fd:::benchmark_data_smithi188_89276_object142487:head# 2024-04-23T16:25:47.011 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.011 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2511e71c:::benchmark_data_smithi188_89276_object64636:head# 2024-04-23T16:25:47.011 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.011 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25123ce3:::benchmark_data_smithi188_89276_object100419:head# 2024-04-23T16:25:47.011 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.011 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2512d4a8:::benchmark_data_smithi188_89276_object59746:head# 2024-04-23T16:25:47.011 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.011 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2512d9e6:::benchmark_data_smithi188_89276_object126925:head# 2024-04-23T16:25:47.011 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.011 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25142d82:::benchmark_data_smithi188_89276_object44762:head# 2024-04-23T16:25:47.011 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.011 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2514c0b7:::benchmark_data_smithi188_89276_object72072:head# 2024-04-23T16:25:47.011 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.012 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2515359d:::benchmark_data_smithi188_89276_object66208:head# 2024-04-23T16:25:47.013 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.013 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25158354:::benchmark_data_smithi188_89276_object112489:head# 2024-04-23T16:25:47.013 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.013 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25164af4:::benchmark_data_smithi188_89276_object85199:head# 2024-04-23T16:25:47.013 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.014 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2516a7fd:::benchmark_data_smithi188_89276_object59658:head# 2024-04-23T16:25:47.014 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.014 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25170cbb:::benchmark_data_smithi188_89276_object97216:head# 2024-04-23T16:25:47.014 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.014 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:251738e8:::benchmark_data_smithi188_89276_object75711:head# 2024-04-23T16:25:47.014 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.014 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:251769ae:::benchmark_data_smithi188_89276_object99757:head# 2024-04-23T16:25:47.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2517a3e0:::benchmark_data_smithi188_89276_object138941:head# 2024-04-23T16:25:47.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2517dc3e:::benchmark_data_smithi188_89276_object100776:head# 2024-04-23T16:25:47.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2518378e:::benchmark_data_smithi188_89276_object78190:head# 2024-04-23T16:25:47.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:251860aa:::benchmark_data_smithi188_89276_object152319:head# 2024-04-23T16:25:47.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2518886a:::benchmark_data_smithi188_89276_object134266:head# 2024-04-23T16:25:47.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2518c192:::benchmark_data_smithi188_89276_object119619:head# 2024-04-23T16:25:47.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:251933e9:::benchmark_data_smithi188_89276_object58295:head# 2024-04-23T16:25:47.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2519575e:::benchmark_data_smithi188_89276_object165987:head# 2024-04-23T16:25:47.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2519bb44:::benchmark_data_smithi188_89276_object122065:head# 2024-04-23T16:25:47.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2519ea93:::benchmark_data_smithi188_89276_object92593:head# 2024-04-23T16:25:47.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:251a4ad7:::benchmark_data_smithi188_89276_object101282:head# 2024-04-23T16:25:47.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:251aaed1:::benchmark_data_smithi188_89276_object120136:head# 2024-04-23T16:25:47.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:251b0b81:::benchmark_data_smithi188_89276_object90312:head# 2024-04-23T16:25:47.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:251b2e30:::benchmark_data_smithi188_89276_object64672:head# 2024-04-23T16:25:47.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:251cd562:::benchmark_data_smithi188_89276_object110268:head# 2024-04-23T16:25:47.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:251d2d15:::benchmark_data_smithi188_89276_object128296:head# 2024-04-23T16:25:47.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:251d7f75:::benchmark_data_smithi188_89276_object87855:head# 2024-04-23T16:25:47.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:251ddad4:::benchmark_data_smithi188_89276_object115079:head# 2024-04-23T16:25:47.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:251dde47:::benchmark_data_smithi188_89276_object71714:head# 2024-04-23T16:25:47.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.032 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:251ec74d:::benchmark_data_smithi188_89276_object160372:head# 2024-04-23T16:25:47.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.032 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:252143c8:::benchmark_data_smithi188_89276_object117178:head# 2024-04-23T16:25:47.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.032 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2521f012:::benchmark_data_smithi188_89276_object103407:head# 2024-04-23T16:25:47.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.032 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:252235f6:::benchmark_data_smithi188_89276_object92459:head# 2024-04-23T16:25:47.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.033 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:252265c8:::benchmark_data_smithi188_89276_object168516:head# 2024-04-23T16:25:47.033 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.033 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2522dba3:::benchmark_data_smithi188_89276_object127230:head# 2024-04-23T16:25:47.033 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.033 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25231004:::benchmark_data_smithi188_89276_object155707:head# 2024-04-23T16:25:47.033 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.033 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2523ea82:::benchmark_data_smithi188_89276_object72716:head# 2024-04-23T16:25:47.033 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.034 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25240d07:::benchmark_data_smithi188_89276_object52327:head# 2024-04-23T16:25:47.034 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.034 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2524239c:::benchmark_data_smithi188_89276_object84363:head# 2024-04-23T16:25:47.034 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.034 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2525a06b:::benchmark_data_smithi188_89276_object155718:head# 2024-04-23T16:25:47.034 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.034 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2525d56d:::benchmark_data_smithi188_89276_object99399:head# 2024-04-23T16:25:47.034 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2525fb8a:::benchmark_data_smithi188_89276_object148982:head# 2024-04-23T16:25:47.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25262730:::benchmark_data_smithi188_89276_object83412:head# 2024-04-23T16:25:47.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2527bca6:::benchmark_data_smithi188_89276_object90351:head# 2024-04-23T16:25:47.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2527c44b:::benchmark_data_smithi188_89276_object79140:head# 2024-04-23T16:25:47.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2528e002:::benchmark_data_smithi188_89276_object53124:head# 2024-04-23T16:25:47.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.036 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2529f1ad:::benchmark_data_smithi188_89276_object61257:head# 2024-04-23T16:25:47.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.036 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:252ad86e:::benchmark_data_smithi188_89276_object46490:head# 2024-04-23T16:25:47.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.036 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:252b5c41:::benchmark_data_smithi188_89276_object57850:head# 2024-04-23T16:25:47.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.037 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:252bac14:::benchmark_data_smithi188_89276_object162823:head# 2024-04-23T16:25:47.037 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.037 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:252bf8b9:::benchmark_data_smithi188_89276_object72528:head# 2024-04-23T16:25:47.037 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.037 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:252c1c8d:::benchmark_data_smithi188_89276_object74751:head# 2024-04-23T16:25:47.037 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.037 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:252c2fc7:::benchmark_data_smithi188_89276_object151742:head# 2024-04-23T16:25:47.038 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.042 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:252cc598:::benchmark_data_smithi188_89276_object73656:head# 2024-04-23T16:25:47.042 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.042 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:252ccad8:::benchmark_data_smithi188_89276_object57414:head# 2024-04-23T16:25:47.043 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.043 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:252eae33:::benchmark_data_smithi188_89276_object45941:head# 2024-04-23T16:25:47.043 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.043 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:252ecbc5:::benchmark_data_smithi188_89276_object58759:head# 2024-04-23T16:25:47.043 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.043 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:252eeef8:::benchmark_data_smithi188_89276_object51619:head# 2024-04-23T16:25:47.043 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.043 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:252f5be3:::benchmark_data_smithi188_89276_object126469:head# 2024-04-23T16:25:47.043 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.043 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:253012b2:::benchmark_data_smithi188_89276_object128507:head# 2024-04-23T16:25:47.043 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.043 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25302059:::benchmark_data_smithi188_89276_object140537:head# 2024-04-23T16:25:47.044 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.044 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25307cf2:::benchmark_data_smithi188_89276_object62992:head# 2024-04-23T16:25:47.044 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.044 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2531e11f:::benchmark_data_smithi188_89276_object113928:head# 2024-04-23T16:25:47.044 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.044 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25341ff5:::benchmark_data_smithi188_89276_object102749:head# 2024-04-23T16:25:47.044 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.044 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2534657e:::benchmark_data_smithi188_89276_object52335:head# 2024-04-23T16:25:47.044 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.044 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2534c619:::benchmark_data_smithi188_89276_object139907:head# 2024-04-23T16:25:47.044 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.045 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2535ef4a:::benchmark_data_smithi188_89276_object83257:head# 2024-04-23T16:25:47.045 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.045 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2535fb69:::benchmark_data_smithi188_89276_object73963:head# 2024-04-23T16:25:47.045 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.045 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:253621b9:::benchmark_data_smithi188_89276_object99419:head# 2024-04-23T16:25:47.045 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.045 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2536b2ed:::benchmark_data_smithi188_89276_object165004:head# 2024-04-23T16:25:47.045 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.045 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2536dc3d:::benchmark_data_smithi188_89276_object110553:head# 2024-04-23T16:25:47.045 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.045 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25374eb9:::benchmark_data_smithi188_89276_object151586:head# 2024-04-23T16:25:47.045 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.046 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2537b8c4:::benchmark_data_smithi188_89276_object110941:head# 2024-04-23T16:25:47.046 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.046 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:253818d1:::benchmark_data_smithi188_89276_object134996:head# 2024-04-23T16:25:47.046 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.046 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:253953ce:::benchmark_data_smithi188_89276_object58213:head# 2024-04-23T16:25:47.046 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.046 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25397511:::benchmark_data_smithi188_89276_object113007:head# 2024-04-23T16:25:47.047 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.047 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:253a777b:::benchmark_data_smithi188_89276_object161963:head# 2024-04-23T16:25:47.047 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.047 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:253b010a:::benchmark_data_smithi188_89276_object166545:head# 2024-04-23T16:25:47.047 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.047 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:253ba130:::benchmark_data_smithi188_89276_object143169:head# 2024-04-23T16:25:47.047 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.047 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:253c1e5c:::benchmark_data_smithi188_89276_object125882:head# 2024-04-23T16:25:47.047 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.047 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:253c532b:::benchmark_data_smithi188_89276_object103297:head# 2024-04-23T16:25:47.048 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.048 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:253dec01:::benchmark_data_smithi188_89276_object45076:head# 2024-04-23T16:25:47.048 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.049 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:253e0623:::benchmark_data_smithi188_89276_object57127:head# 2024-04-23T16:25:47.049 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.050 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:253e2d54:::benchmark_data_smithi188_89276_object146218:head# 2024-04-23T16:25:47.050 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.050 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:253eae59:::benchmark_data_smithi188_89276_object163063:head# 2024-04-23T16:25:47.050 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.050 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:253f8787:::benchmark_data_smithi188_89276_object80731:head# 2024-04-23T16:25:47.050 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.050 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:254059ff:::benchmark_data_smithi188_89276_object146474:head# 2024-04-23T16:25:47.051 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.051 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25418241:::benchmark_data_smithi188_89276_object96103:head# 2024-04-23T16:25:47.051 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.051 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25421ea7:::benchmark_data_smithi188_89276_object54614:head# 2024-04-23T16:25:47.051 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.051 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2542736a:::benchmark_data_smithi188_89276_object76289:head# 2024-04-23T16:25:47.051 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.052 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2543ebb2:::benchmark_data_smithi188_89276_object161864:head# 2024-04-23T16:25:47.052 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.052 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25443191:::benchmark_data_smithi188_89276_object71122:head# 2024-04-23T16:25:47.052 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.052 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2544b1d3:::benchmark_data_smithi188_89276_object135618:head# 2024-04-23T16:25:47.052 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.052 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2545f81d:::benchmark_data_smithi188_89276_object51337:head# 2024-04-23T16:25:47.053 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.053 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:254688eb:::benchmark_data_smithi188_89276_object144437:head# 2024-04-23T16:25:47.053 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.053 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2546d648:::benchmark_data_smithi188_89276_object127397:head# 2024-04-23T16:25:47.053 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.053 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25477944:::benchmark_data_smithi188_89276_object58564:head# 2024-04-23T16:25:47.054 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.055 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25480c7d:::benchmark_data_smithi188_89276_object143581:head# 2024-04-23T16:25:47.055 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.055 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25485a3f:::benchmark_data_smithi188_89276_object150601:head# 2024-04-23T16:25:47.055 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.055 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2549021e:::benchmark_data_smithi188_89276_object66238:head# 2024-04-23T16:25:47.056 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.056 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2549d280:::benchmark_data_smithi188_89276_object64017:head# 2024-04-23T16:25:47.056 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.056 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:254b6e8d:::benchmark_data_smithi188_89276_object82903:head# 2024-04-23T16:25:47.056 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.056 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:254be962:::benchmark_data_smithi188_89276_object61718:head# 2024-04-23T16:25:47.056 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.056 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:254bebe2:::benchmark_data_smithi188_89276_object92463:head# 2024-04-23T16:25:47.057 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.057 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:254cf3fd:::benchmark_data_smithi188_89276_object78813:head# 2024-04-23T16:25:47.057 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.057 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:254d9c59:::benchmark_data_smithi188_89276_object140558:head# 2024-04-23T16:25:47.057 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:254e3666:::benchmark_data_smithi188_89276_object165214:head# 2024-04-23T16:25:47.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:254e512a:::benchmark_data_smithi188_89276_object162729:head# 2024-04-23T16:25:47.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:254e536a:::benchmark_data_smithi188_89276_object80308:head# 2024-04-23T16:25:47.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:254e6620:::benchmark_data_smithi188_89276_object47160:head# 2024-04-23T16:25:47.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.063 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:254f04f9:::benchmark_data_smithi188_89276_object70574:head# 2024-04-23T16:25:47.063 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.063 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:254f80eb:::benchmark_data_smithi188_89276_object92828:head# 2024-04-23T16:25:47.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25500062:::benchmark_data_smithi188_89276_object70679:head# 2024-04-23T16:25:47.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2550d4f8:::benchmark_data_smithi188_89276_object106987:head# 2024-04-23T16:25:47.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2550d728:::benchmark_data_smithi188_89276_object88347:head# 2024-04-23T16:25:47.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2550feb6:::benchmark_data_smithi188_89276_object126827:head# 2024-04-23T16:25:47.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2551a5b7:::benchmark_data_smithi188_89276_object71347:head# 2024-04-23T16:25:47.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2551e319:::benchmark_data_smithi188_89276_object132948:head# 2024-04-23T16:25:47.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:255343f6:::benchmark_data_smithi188_89276_object100479:head# 2024-04-23T16:25:47.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25539b14:::benchmark_data_smithi188_89276_object89745:head# 2024-04-23T16:25:47.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25541f6d:::benchmark_data_smithi188_89276_object128753:head# 2024-04-23T16:25:47.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25549a43:::benchmark_data_smithi188_89276_object41006:head# 2024-04-23T16:25:47.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:255583e5:::benchmark_data_smithi188_89276_object63905:head# 2024-04-23T16:25:47.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2555cd15:::benchmark_data_smithi188_89276_object149602:head# 2024-04-23T16:25:47.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25562485:::benchmark_data_smithi188_89276_object166384:head# 2024-04-23T16:25:47.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25569072:::benchmark_data_smithi188_89276_object92049:head# 2024-04-23T16:25:47.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25580289:::benchmark_data_smithi188_89276_object110613:head# 2024-04-23T16:25:47.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25582d04:::benchmark_data_smithi188_89276_object98333:head# 2024-04-23T16:25:47.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2559061e:::benchmark_data_smithi188_89276_object127768:head# 2024-04-23T16:25:47.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:255913cf:::benchmark_data_smithi188_89276_object144901:head# 2024-04-23T16:25:47.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25595861:::benchmark_data_smithi188_89276_object92981:head# 2024-04-23T16:25:47.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2559c92c:::benchmark_data_smithi188_89276_object65624:head# 2024-04-23T16:25:47.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.071 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:255a1228:::benchmark_data_smithi188_89276_object129787:head# 2024-04-23T16:25:47.071 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.072 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:255a3735:::benchmark_data_smithi188_89276_object76606:head# 2024-04-23T16:25:47.072 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.072 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:255aced2:::benchmark_data_smithi188_89276_object91300:head# 2024-04-23T16:25:47.072 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.072 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:255ad189:::benchmark_data_smithi188_89276_object166711:head# 2024-04-23T16:25:47.072 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.072 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:255c23e6:::benchmark_data_smithi188_89276_object150545:head# 2024-04-23T16:25:47.072 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.072 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:255ce3c5:::benchmark_data_smithi188_89276_object95180:head# 2024-04-23T16:25:47.072 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.072 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:255cf6da:::benchmark_data_smithi188_89276_object159587:head# 2024-04-23T16:25:47.073 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.073 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:255d2ff1:::benchmark_data_smithi188_89276_object151331:head# 2024-04-23T16:25:47.073 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.073 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:255d4de6:::benchmark_data_smithi188_89276_object82876:head# 2024-04-23T16:25:47.073 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.073 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:255db744:::benchmark_data_smithi188_89276_object113546:head# 2024-04-23T16:25:47.073 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.073 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:255df74d:::benchmark_data_smithi188_89276_object59075:head# 2024-04-23T16:25:47.073 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.073 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:255e3c63:::benchmark_data_smithi188_89276_object85966:head# 2024-04-23T16:25:47.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:255ecbda:::benchmark_data_smithi188_89276_object153541:head# 2024-04-23T16:25:47.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:255f151c:::benchmark_data_smithi188_89276_object123583:head# 2024-04-23T16:25:47.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:255fe4a5:::benchmark_data_smithi188_89276_object157387:head# 2024-04-23T16:25:47.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:256070e4:::benchmark_data_smithi188_89276_object98804:head# 2024-04-23T16:25:47.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2560ca62:::benchmark_data_smithi188_89276_object163633:head# 2024-04-23T16:25:47.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2560e64a:::benchmark_data_smithi188_89276_object60455:head# 2024-04-23T16:25:47.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:256103f4:::benchmark_data_smithi188_89276_object105491:head# 2024-04-23T16:25:47.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25621df8:::benchmark_data_smithi188_89276_object104729:head# 2024-04-23T16:25:47.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25623c2a:::benchmark_data_smithi188_89276_object157863:head# 2024-04-23T16:25:47.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25627960:::benchmark_data_smithi188_89276_object105132:head# 2024-04-23T16:25:47.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2562be34:::benchmark_data_smithi188_89276_object43994:head# 2024-04-23T16:25:47.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2562dedf:::benchmark_data_smithi188_89276_object122264:head# 2024-04-23T16:25:47.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:256324cd:::benchmark_data_smithi188_89276_object84870:head# 2024-04-23T16:25:47.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2563c37c:::benchmark_data_smithi188_89276_object88080:head# 2024-04-23T16:25:47.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25647b2c:::benchmark_data_smithi188_89276_object53729:head# 2024-04-23T16:25:47.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2565383c:::benchmark_data_smithi188_89276_object68895:head# 2024-04-23T16:25:47.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25653e20:::benchmark_data_smithi188_89276_object124485:head# 2024-04-23T16:25:47.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:256564f1:::benchmark_data_smithi188_89276_object101550:head# 2024-04-23T16:25:47.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:256588ec:::benchmark_data_smithi188_89276_object125872:head# 2024-04-23T16:25:47.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2565df57:::benchmark_data_smithi188_89276_object97740:head# 2024-04-23T16:25:47.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25666e06:::benchmark_data_smithi188_89276_object138103:head# 2024-04-23T16:25:47.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.078 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25676efb:::benchmark_data_smithi188_89276_object54597:head# 2024-04-23T16:25:47.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.078 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:256795ee:::benchmark_data_smithi188_89276_object49036:head# 2024-04-23T16:25:47.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.078 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2569429a:::benchmark_data_smithi188_89276_object107326:head# 2024-04-23T16:25:47.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.078 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:256b588c:::benchmark_data_smithi188_89276_object157736:head# 2024-04-23T16:25:47.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.089 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:256b8666:::benchmark_data_smithi188_89276_object66998:head# 2024-04-23T16:25:47.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:256baf42:::benchmark_data_smithi188_89276_object96192:head# 2024-04-23T16:25:47.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:256c9446:::benchmark_data_smithi188_89276_object79003:head# 2024-04-23T16:25:47.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:256d0a69:::benchmark_data_smithi188_89276_object60377:head# 2024-04-23T16:25:47.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:256d859a:::benchmark_data_smithi188_89276_object154327:head# 2024-04-23T16:25:47.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:256da100:::benchmark_data_smithi188_89276_object44378:head# 2024-04-23T16:25:47.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:256df710:::benchmark_data_smithi188_89276_object94414:head# 2024-04-23T16:25:47.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:256e1cdd:::benchmark_data_smithi188_89276_object94506:head# 2024-04-23T16:25:47.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:256e6e8b:::benchmark_data_smithi188_89276_object52916:head# 2024-04-23T16:25:47.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:256ed354:::benchmark_data_smithi188_89276_object153120:head# 2024-04-23T16:25:47.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:256f5ebf:::benchmark_data_smithi188_89276_object88793:head# 2024-04-23T16:25:47.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:256fb51c:::benchmark_data_smithi188_89276_object154678:head# 2024-04-23T16:25:47.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2570d208:::benchmark_data_smithi188_89276_object134454:head# 2024-04-23T16:25:47.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2570e35b:::benchmark_data_smithi188_89276_object57733:head# 2024-04-23T16:25:47.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2570f0cc:::benchmark_data_smithi188_89276_object64013:head# 2024-04-23T16:25:47.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:257185f4:::benchmark_data_smithi188_89276_object81306:head# 2024-04-23T16:25:47.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25718824:::benchmark_data_smithi188_89276_object141790:head# 2024-04-23T16:25:47.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2571c71d:::benchmark_data_smithi188_89276_object88370:head# 2024-04-23T16:25:47.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25722961:::benchmark_data_smithi188_89276_object65635:head# 2024-04-23T16:25:47.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2572707d:::benchmark_data_smithi188_89276_object76537:head# 2024-04-23T16:25:47.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25732324:::benchmark_data_smithi188_89276_object45133:head# 2024-04-23T16:25:47.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25734e4e:::benchmark_data_smithi188_89276_object54185:head# 2024-04-23T16:25:47.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25738214:::benchmark_data_smithi188_89276_object57794:head# 2024-04-23T16:25:47.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2573c161:::benchmark_data_smithi188_89276_object108911:head# 2024-04-23T16:25:47.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2574f9b4:::benchmark_data_smithi188_89276_object57934:head# 2024-04-23T16:25:47.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2575b6e0:::benchmark_data_smithi188_89276_object83105:head# 2024-04-23T16:25:47.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2575d3e6:::benchmark_data_smithi188_89276_object75450:head# 2024-04-23T16:25:47.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2575f875:::benchmark_data_smithi188_89276_object75566:head# 2024-04-23T16:25:47.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2576a728:::benchmark_data_smithi188_89276_object167644:head# 2024-04-23T16:25:47.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2576d2e7:::benchmark_data_smithi188_89276_object154065:head# 2024-04-23T16:25:47.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25774fe1:::benchmark_data_smithi188_89276_object51655:head# 2024-04-23T16:25:47.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25784d60:::benchmark_data_smithi188_89276_object75624:head# 2024-04-23T16:25:47.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2578c440:::benchmark_data_smithi188_89276_object50706:head# 2024-04-23T16:25:47.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25797c4d:::benchmark_data_smithi188_89276_object46704:head# 2024-04-23T16:25:47.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2579c9d0:::benchmark_data_smithi188_89276_object164891:head# 2024-04-23T16:25:47.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2579f6ee:::benchmark_data_smithi188_89276_object142559:head# 2024-04-23T16:25:47.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:257a790e:::benchmark_data_smithi188_89276_object145660:head# 2024-04-23T16:25:47.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:257aa38a:::benchmark_data_smithi188_89276_object147697:head# 2024-04-23T16:25:47.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:257b34ec:::benchmark_data_smithi188_89276_object61899:head# 2024-04-23T16:25:47.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:257b5cc0:::benchmark_data_smithi188_89276_object110224:head# 2024-04-23T16:25:47.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:257d318c:::benchmark_data_smithi188_89276_object155491:head# 2024-04-23T16:25:47.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:257ea230:::benchmark_data_smithi188_89276_object63707:head# 2024-04-23T16:25:47.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:257ee02b:::benchmark_data_smithi188_89276_object139655:head# 2024-04-23T16:25:47.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:257f9720:::benchmark_data_smithi188_89276_object113370:head# 2024-04-23T16:25:47.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25802eac:::benchmark_data_smithi188_89276_object147238:head# 2024-04-23T16:25:47.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2580cba3:::benchmark_data_smithi188_89276_object149441:head# 2024-04-23T16:25:47.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25816813:::benchmark_data_smithi188_89276_object96014:head# 2024-04-23T16:25:47.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258198a2:::benchmark_data_smithi188_89276_object67602:head# 2024-04-23T16:25:47.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25821483:::benchmark_data_smithi188_89276_object82102:head# 2024-04-23T16:25:47.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25821912:::benchmark_data_smithi188_89276_object66513:head# 2024-04-23T16:25:47.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25821bdf:::benchmark_data_smithi188_89276_object166211:head# 2024-04-23T16:25:47.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25822d20:::benchmark_data_smithi188_89276_object142185:head# 2024-04-23T16:25:47.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258258e4:::benchmark_data_smithi188_89276_object115459:head# 2024-04-23T16:25:47.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2584fe32:::benchmark_data_smithi188_89276_object147611:head# 2024-04-23T16:25:47.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258531c8:::benchmark_data_smithi188_89276_object101098:head# 2024-04-23T16:25:47.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2585a0de:::benchmark_data_smithi188_89276_object139355:head# 2024-04-23T16:25:47.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2585a6aa:::benchmark_data_smithi188_89276_object60934:head# 2024-04-23T16:25:47.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258726af:::benchmark_data_smithi188_89276_object127691:head# 2024-04-23T16:25:47.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25872fe8:::benchmark_data_smithi188_89276_object55961:head# 2024-04-23T16:25:47.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25875be3:::benchmark_data_smithi188_89276_object50580:head# 2024-04-23T16:25:47.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2587a7b3:::benchmark_data_smithi188_89276_object43091:head# 2024-04-23T16:25:47.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2587e178:::benchmark_data_smithi188_89276_object130746:head# 2024-04-23T16:25:47.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25888cf6:::benchmark_data_smithi188_89276_object68607:head# 2024-04-23T16:25:47.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25890544:::benchmark_data_smithi188_89276_object119157:head# 2024-04-23T16:25:47.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2589c7e2:::benchmark_data_smithi188_89276_object141472:head# 2024-04-23T16:25:47.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2589e18c:::benchmark_data_smithi188_89276_object53426:head# 2024-04-23T16:25:47.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258aaeeb:::benchmark_data_smithi188_89276_object141711:head# 2024-04-23T16:25:47.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258b02cb:::benchmark_data_smithi188_89276_object85956:head# 2024-04-23T16:25:47.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258bebe0:::benchmark_data_smithi188_89276_object127710:head# 2024-04-23T16:25:47.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258bed35:::benchmark_data_smithi188_89276_object79928:head# 2024-04-23T16:25:47.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258bed74:::benchmark_data_smithi188_89276_object119173:head# 2024-04-23T16:25:47.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258c0bf7:::benchmark_data_smithi188_89276_object130772:head# 2024-04-23T16:25:47.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258c70d9:::benchmark_data_smithi188_89276_object168567:head# 2024-04-23T16:25:47.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258cf5db:::benchmark_data_smithi188_89276_object103274:head# 2024-04-23T16:25:47.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258cfc2b:::benchmark_data_smithi188_89276_object161794:head# 2024-04-23T16:25:47.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258d5348:::benchmark_data_smithi188_89276_object96455:head# 2024-04-23T16:25:47.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258d9a38:::benchmark_data_smithi188_89276_object47581:head# 2024-04-23T16:25:47.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258db96d:::benchmark_data_smithi188_89276_object133492:head# 2024-04-23T16:25:47.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258e1c0f:::benchmark_data_smithi188_89276_object168308:head# 2024-04-23T16:25:47.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258ec6d4:::benchmark_data_smithi188_89276_object142255:head# 2024-04-23T16:25:47.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258f88da:::benchmark_data_smithi188_89276_object107733:head# 2024-04-23T16:25:47.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:258fa77d:::benchmark_data_smithi188_89276_object67139:head# 2024-04-23T16:25:47.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25903d18:::benchmark_data_smithi188_89276_object147600:head# 2024-04-23T16:25:47.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2590a5c9:::benchmark_data_smithi188_89276_object73894:head# 2024-04-23T16:25:47.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25914cdd:::benchmark_data_smithi188_89276_object145063:head# 2024-04-23T16:25:47.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2592eeba:::benchmark_data_smithi188_89276_object68686:head# 2024-04-23T16:25:47.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25930b3b:::benchmark_data_smithi188_89276_object41031:head# 2024-04-23T16:25:47.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25933edc:::benchmark_data_smithi188_89276_object98213:head# 2024-04-23T16:25:47.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25934f54:::benchmark_data_smithi188_89276_object149449:head# 2024-04-23T16:25:47.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25935bb2:::benchmark_data_smithi188_89276_object115370:head# 2024-04-23T16:25:47.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2593e525:::benchmark_data_smithi188_89276_object42584:head# 2024-04-23T16:25:47.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25943c8c:::benchmark_data_smithi188_89276_object82689:head# 2024-04-23T16:25:47.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25945563:::benchmark_data_smithi188_89276_object85122:head# 2024-04-23T16:25:47.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2594d253:::benchmark_data_smithi188_89276_object73657:head# 2024-04-23T16:25:47.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2594f103:::benchmark_data_smithi188_89276_object164548:head# 2024-04-23T16:25:47.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25962013:::benchmark_data_smithi188_89276_object124379:head# 2024-04-23T16:25:47.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25964764:::benchmark_data_smithi188_89276_object152761:head# 2024-04-23T16:25:47.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25967a00:::benchmark_data_smithi188_89276_object78502:head# 2024-04-23T16:25:47.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2596c76d:::benchmark_data_smithi188_89276_object66850:head# 2024-04-23T16:25:47.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2597d221:::benchmark_data_smithi188_89276_object98898:head# 2024-04-23T16:25:47.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25980204:::benchmark_data_smithi188_89276_object147716:head# 2024-04-23T16:25:47.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25991183:::benchmark_data_smithi188_89276_object167435:head# 2024-04-23T16:25:47.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:259bb8e4:::benchmark_data_smithi188_89276_object162347:head# 2024-04-23T16:25:47.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:259bcf32:::benchmark_data_smithi188_89276_object53167:head# 2024-04-23T16:25:47.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:259db4c2:::benchmark_data_smithi188_89276_object65491:head# 2024-04-23T16:25:47.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:259db9e0:::benchmark_data_smithi188_89276_object81134:head# 2024-04-23T16:25:47.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:259dd2b2:::benchmark_data_smithi188_89276_object158519:head# 2024-04-23T16:25:47.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:259ddc1d:::benchmark_data_smithi188_89276_object40294:head# 2024-04-23T16:25:47.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:259e717f:::benchmark_data_smithi188_89276_object154589:head# 2024-04-23T16:25:47.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:259e7c92:::benchmark_data_smithi188_89276_object152403:head# 2024-04-23T16:25:47.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:259e81e2:::benchmark_data_smithi188_89276_object158594:head# 2024-04-23T16:25:47.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:259eb8c9:::benchmark_data_smithi188_89276_object85880:head# 2024-04-23T16:25:47.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:259f79b8:::benchmark_data_smithi188_89276_object41566:head# 2024-04-23T16:25:47.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a052be:::benchmark_data_smithi188_89276_object89300:head# 2024-04-23T16:25:47.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a070af:::benchmark_data_smithi188_89276_object112467:head# 2024-04-23T16:25:47.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a0e92b:::benchmark_data_smithi188_89276_object137528:head# 2024-04-23T16:25:47.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a17624:::benchmark_data_smithi188_89276_object55124:head# 2024-04-23T16:25:47.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a19076:::benchmark_data_smithi188_89276_object107929:head# 2024-04-23T16:25:47.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a1c0ee:::benchmark_data_smithi188_89276_object78657:head# 2024-04-23T16:25:47.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a1c656:::benchmark_data_smithi188_89276_object125090:head# 2024-04-23T16:25:47.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a35182:::benchmark_data_smithi188_89276_object109193:head# 2024-04-23T16:25:47.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a3db41:::benchmark_data_smithi188_89276_object74169:head# 2024-04-23T16:25:47.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a3ea05:::benchmark_data_smithi188_89276_object158418:head# 2024-04-23T16:25:47.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a504c8:::benchmark_data_smithi188_89276_object76942:head# 2024-04-23T16:25:47.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a518ab:::benchmark_data_smithi188_89276_object136699:head# 2024-04-23T16:25:47.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a5600b:::benchmark_data_smithi188_89276_object148745:head# 2024-04-23T16:25:47.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a65150:::benchmark_data_smithi188_89276_object79692:head# 2024-04-23T16:25:47.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a6c5fb:::benchmark_data_smithi188_89276_object149652:head# 2024-04-23T16:25:47.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a6eb94:::benchmark_data_smithi188_89276_object121726:head# 2024-04-23T16:25:47.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a71ac5:::benchmark_data_smithi188_89276_object77953:head# 2024-04-23T16:25:47.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a7b072:::benchmark_data_smithi188_89276_object162563:head# 2024-04-23T16:25:47.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a7e06c:::benchmark_data_smithi188_89276_object120794:head# 2024-04-23T16:25:47.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a7ef65:::benchmark_data_smithi188_89276_object122128:head# 2024-04-23T16:25:47.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25a8bdf8:::benchmark_data_smithi188_89276_object139177:head# 2024-04-23T16:25:47.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25aa5041:::benchmark_data_smithi188_89276_object92675:head# 2024-04-23T16:25:47.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ab20a0:::benchmark_data_smithi188_89276_object68073:head# 2024-04-23T16:25:47.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ab3bcd:::benchmark_data_smithi188_89276_object87327:head# 2024-04-23T16:25:47.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ab8cd4:::benchmark_data_smithi188_89276_object136766:head# 2024-04-23T16:25:47.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ac2074:::benchmark_data_smithi188_89276_object103077:head# 2024-04-23T16:25:47.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25acf4d1:::benchmark_data_smithi188_89276_object141265:head# 2024-04-23T16:25:47.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ae5456:::benchmark_data_smithi188_89276_object83207:head# 2024-04-23T16:25:47.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25aef333:::benchmark_data_smithi188_89276_object154196:head# 2024-04-23T16:25:47.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25aef95a:::benchmark_data_smithi188_89276_object89496:head# 2024-04-23T16:25:47.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25afaa96:::benchmark_data_smithi188_89276_object105127:head# 2024-04-23T16:25:47.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25aff449:::benchmark_data_smithi188_89276_object161053:head# 2024-04-23T16:25:47.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25b102b5:::benchmark_data_smithi188_89276_object61221:head# 2024-04-23T16:25:47.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25b1ad62:::benchmark_data_smithi188_89276_object84204:head# 2024-04-23T16:25:47.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25b24c4d:::benchmark_data_smithi188_89276_object93076:head# 2024-04-23T16:25:47.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25b48f28:::benchmark_data_smithi188_89276_object147413:head# 2024-04-23T16:25:47.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25b516ea:::benchmark_data_smithi188_89276_object48651:head# 2024-04-23T16:25:47.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25b5512e:::benchmark_data_smithi188_89276_object125541:head# 2024-04-23T16:25:47.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25b59de1:::benchmark_data_smithi188_89276_object130976:head# 2024-04-23T16:25:47.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25b630c1:::benchmark_data_smithi188_89276_object45669:head# 2024-04-23T16:25:47.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25b6bda5:::benchmark_data_smithi188_89276_object119823:head# 2024-04-23T16:25:47.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25b6fc02:::benchmark_data_smithi188_89276_object43359:head# 2024-04-23T16:25:47.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25b78d84:::benchmark_data_smithi188_89276_object80646:head# 2024-04-23T16:25:47.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25b7b048:::benchmark_data_smithi188_89276_object54744:head# 2024-04-23T16:25:47.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25b7ca33:::benchmark_data_smithi188_89276_object77585:head# 2024-04-23T16:25:47.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25b86cc9:::benchmark_data_smithi188_89276_object102266:head# 2024-04-23T16:25:47.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25b9cd69:::benchmark_data_smithi188_89276_object145274:head# 2024-04-23T16:25:47.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ba9bd0:::benchmark_data_smithi188_89276_object121416:head# 2024-04-23T16:25:47.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25bac225:::benchmark_data_smithi188_89276_object135839:head# 2024-04-23T16:25:47.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25bd4d86:::benchmark_data_smithi188_89276_object164131:head# 2024-04-23T16:25:47.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25be0518:::benchmark_data_smithi188_89276_object77847:head# 2024-04-23T16:25:47.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25be55ff:::benchmark_data_smithi188_89276_object153708:head# 2024-04-23T16:25:47.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25bf21e2:::benchmark_data_smithi188_89276_object94575:head# 2024-04-23T16:25:47.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25bf2999:::benchmark_data_smithi188_89276_object65791:head# 2024-04-23T16:25:47.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c066fd:::benchmark_data_smithi188_89276_object70016:head# 2024-04-23T16:25:47.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c0a336:::benchmark_data_smithi188_89276_object149333:head# 2024-04-23T16:25:47.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c0aac5:::benchmark_data_smithi188_89276_object124441:head# 2024-04-23T16:25:47.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c0c10d:::benchmark_data_smithi188_89276_object67282:head# 2024-04-23T16:25:47.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c0f4fc:::benchmark_data_smithi188_89276_object159823:head# 2024-04-23T16:25:47.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c189a1:::benchmark_data_smithi188_89276_object100067:head# 2024-04-23T16:25:47.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c19069:::benchmark_data_smithi188_89276_object119056:head# 2024-04-23T16:25:47.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c22cce:::benchmark_data_smithi188_89276_object118833:head# 2024-04-23T16:25:47.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c25388:::benchmark_data_smithi188_89276_object136513:head# 2024-04-23T16:25:47.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c262f5:::benchmark_data_smithi188_89276_object62869:head# 2024-04-23T16:25:47.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c2f00d:::benchmark_data_smithi188_89276_object104205:head# 2024-04-23T16:25:47.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c40136:::benchmark_data_smithi188_89276_object50875:head# 2024-04-23T16:25:47.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c438b8:::benchmark_data_smithi188_89276_object84991:head# 2024-04-23T16:25:47.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c44e94:::benchmark_data_smithi188_89276_object117779:head# 2024-04-23T16:25:47.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c4533f:::benchmark_data_smithi188_89276_object40917:head# 2024-04-23T16:25:47.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c51825:::benchmark_data_smithi188_89276_object84603:head# 2024-04-23T16:25:47.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c5c187:::benchmark_data_smithi188_89276_object103478:head# 2024-04-23T16:25:47.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c65068:::benchmark_data_smithi188_89276_object116746:head# 2024-04-23T16:25:47.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c6d063:::benchmark_data_smithi188_89276_object130780:head# 2024-04-23T16:25:47.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c6f532:::benchmark_data_smithi188_89276_object54810:head# 2024-04-23T16:25:47.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c72029:::benchmark_data_smithi188_89276_object101303:head# 2024-04-23T16:25:47.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c74ba0:::benchmark_data_smithi188_89276_object150930:head# 2024-04-23T16:25:47.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c84f5e:::benchmark_data_smithi188_89276_object116781:head# 2024-04-23T16:25:47.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.138 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25c9838e:::benchmark_data_smithi188_89276_object94022:head# 2024-04-23T16:25:47.138 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.138 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ca044f:::benchmark_data_smithi188_89276_object114824:head# 2024-04-23T16:25:47.138 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.138 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ca812e:::benchmark_data_smithi188_89276_object125152:head# 2024-04-23T16:25:47.138 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.138 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25cb7c68:::benchmark_data_smithi188_89276_object85217:head# 2024-04-23T16:25:47.138 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.138 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25cc3be2:::benchmark_data_smithi188_89276_object46954:head# 2024-04-23T16:25:47.138 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.138 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25cc622e:::benchmark_data_smithi188_89276_object135729:head# 2024-04-23T16:25:47.138 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ccd402:::benchmark_data_smithi188_89276_object123499:head# 2024-04-23T16:25:47.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25cd10e7:::benchmark_data_smithi188_89276_object81508:head# 2024-04-23T16:25:47.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25cdeccd:::benchmark_data_smithi188_89276_object41840:head# 2024-04-23T16:25:47.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25cf5344:::benchmark_data_smithi188_89276_object83447:head# 2024-04-23T16:25:47.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25cffe56:::benchmark_data_smithi188_89276_object62594:head# 2024-04-23T16:25:47.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25d137e0:::benchmark_data_smithi188_89276_object116648:head# 2024-04-23T16:25:47.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25d24bf0:::benchmark_data_smithi188_89276_object93236:head# 2024-04-23T16:25:47.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25d32e90:::benchmark_data_smithi188_89276_object121843:head# 2024-04-23T16:25:47.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25d3c3f2:::benchmark_data_smithi188_89276_object127705:head# 2024-04-23T16:25:47.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25d41d71:::benchmark_data_smithi188_89276_object84282:head# 2024-04-23T16:25:47.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25d4c730:::benchmark_data_smithi188_89276_object145699:head# 2024-04-23T16:25:47.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25d68ceb:::benchmark_data_smithi188_89276_object64679:head# 2024-04-23T16:25:47.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25d6cfe8:::benchmark_data_smithi188_89276_object79139:head# 2024-04-23T16:25:47.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25d79694:::benchmark_data_smithi188_89276_object64967:head# 2024-04-23T16:25:47.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25d7a6fd:::benchmark_data_smithi188_89276_object87315:head# 2024-04-23T16:25:47.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25d7d44b:::benchmark_data_smithi188_89276_object66797:head# 2024-04-23T16:25:47.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25d89feb:::benchmark_data_smithi188_89276_object59285:head# 2024-04-23T16:25:47.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25d8cf3a:::benchmark_data_smithi188_89276_object105788:head# 2024-04-23T16:25:47.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25d93377:::benchmark_data_smithi188_89276_object152788:head# 2024-04-23T16:25:47.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25d96690:::benchmark_data_smithi188_89276_object98935:head# 2024-04-23T16:25:47.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25d9af57:::benchmark_data_smithi188_89276_object108382:head# 2024-04-23T16:25:47.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25db1927:::benchmark_data_smithi188_89276_object156081:head# 2024-04-23T16:25:47.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25dbb5da:::benchmark_data_smithi188_89276_object136922:head# 2024-04-23T16:25:47.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25dcd518:::benchmark_data_smithi188_89276_object143928:head# 2024-04-23T16:25:47.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25dcdde9:::benchmark_data_smithi188_89276_object68994:head# 2024-04-23T16:25:47.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25dcedd2:::benchmark_data_smithi188_89276_object59650:head# 2024-04-23T16:25:47.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25dd8b3e:::benchmark_data_smithi188_89276_object50747:head# 2024-04-23T16:25:47.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25de307e:::benchmark_data_smithi188_89276_object128273:head# 2024-04-23T16:25:47.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25dfcbe7:::benchmark_data_smithi188_89276_object92504:head# 2024-04-23T16:25:47.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25e0d59d:::benchmark_data_smithi188_89276_object57532:head# 2024-04-23T16:25:47.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25e0fdd1:::benchmark_data_smithi188_89276_object55635:head# 2024-04-23T16:25:47.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25e23333:::benchmark_data_smithi188_89276_object149662:head# 2024-04-23T16:25:47.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25e26ae2:::benchmark_data_smithi188_89276_object104194:head# 2024-04-23T16:25:47.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25e27c3c:::benchmark_data_smithi188_89276_object150322:head# 2024-04-23T16:25:47.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25e2ef1d:::benchmark_data_smithi188_89276_object145839:head# 2024-04-23T16:25:47.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25e314b5:::benchmark_data_smithi188_89276_object165669:head# 2024-04-23T16:25:47.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25e49e4a:::benchmark_data_smithi188_89276_object76408:head# 2024-04-23T16:25:47.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25e4dd90:::benchmark_data_smithi188_89276_object71567:head# 2024-04-23T16:25:47.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25e4f6de:::benchmark_data_smithi188_89276_object123710:head# 2024-04-23T16:25:47.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25e5c500:::benchmark_data_smithi188_89276_object113708:head# 2024-04-23T16:25:47.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25e6a812:::benchmark_data_smithi188_89276_object66215:head# 2024-04-23T16:25:47.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25e6b79a:::benchmark_data_smithi188_89276_object62010:head# 2024-04-23T16:25:47.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25e87f88:::benchmark_data_smithi188_89276_object129661:head# 2024-04-23T16:25:47.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25e8e278:::benchmark_data_smithi188_89276_object143992:head# 2024-04-23T16:25:47.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25e95760:::benchmark_data_smithi188_89276_object58936:head# 2024-04-23T16:25:47.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25e9f721:::benchmark_data_smithi188_89276_object45994:head# 2024-04-23T16:25:47.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ea0a98:::benchmark_data_smithi188_89276_object74451:head# 2024-04-23T16:25:47.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ebf104:::benchmark_data_smithi188_89276_object85428:head# 2024-04-23T16:25:47.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ec597f:::benchmark_data_smithi188_89276_object163071:head# 2024-04-23T16:25:47.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ec5ee4:::benchmark_data_smithi188_89276_object126434:head# 2024-04-23T16:25:47.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ecd18d:::benchmark_data_smithi188_89276_object69356:head# 2024-04-23T16:25:47.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ecd4a5:::benchmark_data_smithi188_89276_object94543:head# 2024-04-23T16:25:47.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ed16b2:::benchmark_data_smithi188_89276_object101383:head# 2024-04-23T16:25:47.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ede61e:::benchmark_data_smithi188_89276_object95354:head# 2024-04-23T16:25:47.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ee1440:::benchmark_data_smithi188_89276_object99793:head# 2024-04-23T16:25:47.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25eed285:::benchmark_data_smithi188_89276_object140727:head# 2024-04-23T16:25:47.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ef98ec:::benchmark_data_smithi188_89276_object132407:head# 2024-04-23T16:25:47.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f28c56:::benchmark_data_smithi188_89276_object161844:head# 2024-04-23T16:25:47.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f2c8b2:::benchmark_data_smithi188_89276_object137858:head# 2024-04-23T16:25:47.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f3b704:::benchmark_data_smithi188_89276_object77955:head# 2024-04-23T16:25:47.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f3d086:::benchmark_data_smithi188_89276_object134149:head# 2024-04-23T16:25:47.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f3e871:::benchmark_data_smithi188_89276_object119672:head# 2024-04-23T16:25:47.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.155 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f3eca7:::benchmark_data_smithi188_89276_object93542:head# 2024-04-23T16:25:47.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.155 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f41842:::benchmark_data_smithi188_89276_object87907:head# 2024-04-23T16:25:47.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.156 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f45c85:::benchmark_data_smithi188_89276_object73314:head# 2024-04-23T16:25:47.156 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.156 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f51c8a:::benchmark_data_smithi188_89276_object123790:head# 2024-04-23T16:25:47.156 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.156 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f62470:::benchmark_data_smithi188_89276_object113637:head# 2024-04-23T16:25:47.156 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.157 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f6726d:::benchmark_data_smithi188_89276_object130908:head# 2024-04-23T16:25:47.157 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.157 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f6a747:::benchmark_data_smithi188_89276_object68525:head# 2024-04-23T16:25:47.158 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.158 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f70c78:::benchmark_data_smithi188_89276_object62064:head# 2024-04-23T16:25:47.158 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.158 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f70dd0:::benchmark_data_smithi188_89276_object69635:head# 2024-04-23T16:25:47.158 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.158 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f72d79:::benchmark_data_smithi188_89276_object85041:head# 2024-04-23T16:25:47.158 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f7afcf:::benchmark_data_smithi188_89276_object87721:head# 2024-04-23T16:25:47.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f87034:::benchmark_data_smithi188_89276_object103116:head# 2024-04-23T16:25:47.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f87c85:::benchmark_data_smithi188_89276_object62408:head# 2024-04-23T16:25:47.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f87eee:::benchmark_data_smithi188_89276_object145035:head# 2024-04-23T16:25:47.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f8d984:::benchmark_data_smithi188_89276_object63129:head# 2024-04-23T16:25:47.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25f98541:::benchmark_data_smithi188_89276_object130737:head# 2024-04-23T16:25:47.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25fad1e5:::benchmark_data_smithi188_89276_object82397:head# 2024-04-23T16:25:47.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25fadf98:::benchmark_data_smithi188_89276_object66101:head# 2024-04-23T16:25:47.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25fc0cad:::benchmark_data_smithi188_89276_object134176:head# 2024-04-23T16:25:47.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25fc10ea:::benchmark_data_smithi188_89276_object125750:head# 2024-04-23T16:25:47.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25fcd989:::benchmark_data_smithi188_89276_object162155:head# 2024-04-23T16:25:47.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25fe00a0:::benchmark_data_smithi188_89276_object72351:head# 2024-04-23T16:25:47.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25fe606c:::benchmark_data_smithi188_89276_object41087:head# 2024-04-23T16:25:47.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25febc5d:::benchmark_data_smithi188_89276_object45958:head# 2024-04-23T16:25:47.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25fee714:::benchmark_data_smithi188_89276_object99228:head# 2024-04-23T16:25:47.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:25ff93d2:::benchmark_data_smithi188_89276_object118364:head# 2024-04-23T16:25:47.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2600046f:::benchmark_data_smithi188_89276_object140632:head# 2024-04-23T16:25:47.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:260030e6:::benchmark_data_smithi188_89276_object137728:head# 2024-04-23T16:25:47.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2600638d:::benchmark_data_smithi188_89276_object132995:head# 2024-04-23T16:25:47.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26007054:::benchmark_data_smithi188_89276_object67890:head# 2024-04-23T16:25:47.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2600ef09:::benchmark_data_smithi188_89276_object148842:head# 2024-04-23T16:25:47.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:260133e7:::benchmark_data_smithi188_89276_object71192:head# 2024-04-23T16:25:47.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26025a02:::benchmark_data_smithi188_89276_object103332:head# 2024-04-23T16:25:47.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26029e20:::benchmark_data_smithi188_89276_object57960:head# 2024-04-23T16:25:47.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2603119f:::benchmark_data_smithi188_89276_object91676:head# 2024-04-23T16:25:47.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26032c4e:::benchmark_data_smithi188_89276_object108961:head# 2024-04-23T16:25:47.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2603745c:::benchmark_data_smithi188_89276_object117879:head# 2024-04-23T16:25:47.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:260468e5:::benchmark_data_smithi188_89276_object114143:head# 2024-04-23T16:25:47.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2606095b:::benchmark_data_smithi188_89276_object157419:head# 2024-04-23T16:25:47.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26067c83:::benchmark_data_smithi188_89276_object128125:head# 2024-04-23T16:25:47.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:260735c0:::benchmark_data_smithi188_89276_object145842:head# 2024-04-23T16:25:47.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26073eb9:::benchmark_data_smithi188_89276_object153373:head# 2024-04-23T16:25:47.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:260747f5:::benchmark_data_smithi188_89276_object132181:head# 2024-04-23T16:25:47.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26078cba:::benchmark_data_smithi188_89276_object106773:head# 2024-04-23T16:25:47.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2607eb0e:::benchmark_data_smithi188_89276_object92755:head# 2024-04-23T16:25:47.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26093a77:::benchmark_data_smithi188_89276_object72701:head# 2024-04-23T16:25:47.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:260965c0:::benchmark_data_smithi188_89276_object50461:head# 2024-04-23T16:25:47.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:260984c6:::benchmark_data_smithi188_89276_object96788:head# 2024-04-23T16:25:47.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:260a48df:::benchmark_data_smithi188_89276_object85368:head# 2024-04-23T16:25:47.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:260b19c2:::benchmark_data_smithi188_89276_object116937:head# 2024-04-23T16:25:47.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:260b449c:::benchmark_data_smithi188_89276_object106694:head# 2024-04-23T16:25:47.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:260bcbc3:::benchmark_data_smithi188_89276_object40754:head# 2024-04-23T16:25:47.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:260c1da3:::benchmark_data_smithi188_89276_object140232:head# 2024-04-23T16:25:47.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:260c9bc4:::benchmark_data_smithi188_89276_object158081:head# 2024-04-23T16:25:47.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:260ceb14:::benchmark_data_smithi188_89276_object161126:head# 2024-04-23T16:25:47.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:260e7cb4:::benchmark_data_smithi188_89276_object103061:head# 2024-04-23T16:25:47.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:260e915b:::benchmark_data_smithi188_89276_object127112:head# 2024-04-23T16:25:47.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:260f2af4:::benchmark_data_smithi188_89276_object136787:head# 2024-04-23T16:25:47.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:260f7a93:::benchmark_data_smithi188_89276_object102230:head# 2024-04-23T16:25:47.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:260fd26f:::benchmark_data_smithi188_89276_object64680:head# 2024-04-23T16:25:47.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2610284f:::benchmark_data_smithi188_89276_object88653:head# 2024-04-23T16:25:47.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2610c3ef:::benchmark_data_smithi188_89276_object119678:head# 2024-04-23T16:25:47.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26127b61:::benchmark_data_smithi188_89276_object107361:head# 2024-04-23T16:25:47.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2612f3f4:::benchmark_data_smithi188_89276_object139644:head# 2024-04-23T16:25:47.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2613c672:::benchmark_data_smithi188_89276_object119809:head# 2024-04-23T16:25:47.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26142eb9:::benchmark_data_smithi188_89276_object149829:head# 2024-04-23T16:25:47.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2614417f:::benchmark_data_smithi188_89276_object83713:head# 2024-04-23T16:25:47.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26145c69:::benchmark_data_smithi188_89276_object99589:head# 2024-04-23T16:25:47.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26147fb0:::benchmark_data_smithi188_89276_object95666:head# 2024-04-23T16:25:47.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2615af66:::benchmark_data_smithi188_89276_object137746:head# 2024-04-23T16:25:47.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2615b98b:::benchmark_data_smithi188_89276_object122642:head# 2024-04-23T16:25:47.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26168910:::benchmark_data_smithi188_89276_object127914:head# 2024-04-23T16:25:47.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:261694a7:::benchmark_data_smithi188_89276_object94174:head# 2024-04-23T16:25:47.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2616aa95:::benchmark_data_smithi188_89276_object84158:head# 2024-04-23T16:25:47.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2616b8ce:::benchmark_data_smithi188_89276_object49171:head# 2024-04-23T16:25:47.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2616cfab:::benchmark_data_smithi188_89276_object65222:head# 2024-04-23T16:25:47.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:261728db:::benchmark_data_smithi188_89276_object158172:head# 2024-04-23T16:25:47.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26176998:::benchmark_data_smithi188_89276_object91035:head# 2024-04-23T16:25:47.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26177fe9:::benchmark_data_smithi188_89276_object134006:head# 2024-04-23T16:25:47.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:261793dc:::benchmark_data_smithi188_89276_object49122:head# 2024-04-23T16:25:47.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2617cc7d:::benchmark_data_smithi188_89276_object56727:head# 2024-04-23T16:25:47.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2618ef5c:::benchmark_data_smithi188_89276_object60210:head# 2024-04-23T16:25:47.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2619b244:::benchmark_data_smithi188_89276_object123850:head# 2024-04-23T16:25:47.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:261a4f60:::benchmark_data_smithi188_89276_object132155:head# 2024-04-23T16:25:47.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:261aa50f:::benchmark_data_smithi188_89276_object95337:head# 2024-04-23T16:25:47.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:261aa940:::benchmark_data_smithi188_89276_object119690:head# 2024-04-23T16:25:47.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:261adcfc:::benchmark_data_smithi188_89276_object120375:head# 2024-04-23T16:25:47.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:261b5021:::benchmark_data_smithi188_89276_object125212:head# 2024-04-23T16:25:47.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:261b541a:::benchmark_data_smithi188_89276_object126539:head# 2024-04-23T16:25:47.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:261b70e8:::benchmark_data_smithi188_89276_object163808:head# 2024-04-23T16:25:47.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:261bda87:::benchmark_data_smithi188_89276_object155891:head# 2024-04-23T16:25:47.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:261bfeb4:::benchmark_data_smithi188_89276_object118658:head# 2024-04-23T16:25:47.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:261bfef8:::benchmark_data_smithi188_89276_object153360:head# 2024-04-23T16:25:47.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:261cb73b:::benchmark_data_smithi188_89276_object72880:head# 2024-04-23T16:25:47.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:261f763f:::benchmark_data_smithi188_89276_object110965:head# 2024-04-23T16:25:47.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:261f873c:::benchmark_data_smithi188_89276_object97158:head# 2024-04-23T16:25:47.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:261fbd05:::benchmark_data_smithi188_89276_object62224:head# 2024-04-23T16:25:47.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26205f7e:::benchmark_data_smithi188_89276_object112081:head# 2024-04-23T16:25:47.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2621d2b5:::benchmark_data_smithi188_89276_object57862:head# 2024-04-23T16:25:47.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2621f9c3:::benchmark_data_smithi188_89276_object50682:head# 2024-04-23T16:25:47.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:262264db:::benchmark_data_smithi188_89276_object109709:head# 2024-04-23T16:25:47.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2622d0e6:::benchmark_data_smithi188_89276_object108933:head# 2024-04-23T16:25:47.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26242761:::benchmark_data_smithi188_89276_object156862:head# 2024-04-23T16:25:47.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26243e9f:::benchmark_data_smithi188_89276_object67061:head# 2024-04-23T16:25:47.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26247ae8:::benchmark_data_smithi188_89276_object56135:head# 2024-04-23T16:25:47.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:262502fb:::benchmark_data_smithi188_89276_object165870:head# 2024-04-23T16:25:47.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26250f43:::benchmark_data_smithi188_89276_object154089:head# 2024-04-23T16:25:47.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2626ce5b:::benchmark_data_smithi188_89276_object106509:head# 2024-04-23T16:25:47.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2628dd97:::benchmark_data_smithi188_89276_object121096:head# 2024-04-23T16:25:47.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26298eca:::benchmark_data_smithi188_89276_object136629:head# 2024-04-23T16:25:47.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26299f07:::benchmark_data_smithi188_89276_object159681:head# 2024-04-23T16:25:47.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2629bf20:::benchmark_data_smithi188_89276_object52448:head# 2024-04-23T16:25:47.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:262a7655:::benchmark_data_smithi188_89276_object147781:head# 2024-04-23T16:25:47.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:262acfe2:::benchmark_data_smithi188_89276_object128624:head# 2024-04-23T16:25:47.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:262b336d:::benchmark_data_smithi188_89276_object99191:head# 2024-04-23T16:25:47.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:262b3bc2:::benchmark_data_smithi188_89276_object80715:head# 2024-04-23T16:25:47.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:262c744d:::benchmark_data_smithi188_89276_object89204:head# 2024-04-23T16:25:47.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:262d3c6e:::benchmark_data_smithi188_89276_object86051:head# 2024-04-23T16:25:47.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:262d8dad:::benchmark_data_smithi188_89276_object92096:head# 2024-04-23T16:25:47.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:262e6f79:::benchmark_data_smithi188_89276_object44490:head# 2024-04-23T16:25:47.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:262ebb47:::benchmark_data_smithi188_89276_object61020:head# 2024-04-23T16:25:47.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:262ec2aa:::benchmark_data_smithi188_89276_object41093:head# 2024-04-23T16:25:47.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:262f16a4:::benchmark_data_smithi188_89276_object94114:head# 2024-04-23T16:25:47.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:262f6767:::benchmark_data_smithi188_89276_object140987:head# 2024-04-23T16:25:47.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:262fbed1:::benchmark_data_smithi188_89276_object149153:head# 2024-04-23T16:25:47.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26310760:::benchmark_data_smithi188_89276_object93649:head# 2024-04-23T16:25:47.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263149fc:::benchmark_data_smithi188_89276_object51944:head# 2024-04-23T16:25:47.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263189ab:::benchmark_data_smithi188_89276_object151709:head# 2024-04-23T16:25:47.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2631eaa8:::benchmark_data_smithi188_89276_object125175:head# 2024-04-23T16:25:47.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26324757:::benchmark_data_smithi188_89276_object99635:head# 2024-04-23T16:25:47.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2632fc0a:::benchmark_data_smithi188_89276_object85092:head# 2024-04-23T16:25:47.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26333345:::benchmark_data_smithi188_89276_object162073:head# 2024-04-23T16:25:47.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2633cfc2:::benchmark_data_smithi188_89276_object95228:head# 2024-04-23T16:25:47.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263456ed:::benchmark_data_smithi188_89276_object62477:head# 2024-04-23T16:25:47.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263483e4:::benchmark_data_smithi188_89276_object71427:head# 2024-04-23T16:25:47.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2634f565:::benchmark_data_smithi188_89276_object147981:head# 2024-04-23T16:25:47.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2634fb92:::benchmark_data_smithi188_89276_object163299:head# 2024-04-23T16:25:47.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263524f6:::benchmark_data_smithi188_89276_object54258:head# 2024-04-23T16:25:47.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2635e79c:::benchmark_data_smithi188_89276_object125431:head# 2024-04-23T16:25:47.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26376607:::benchmark_data_smithi188_89276_object54109:head# 2024-04-23T16:25:47.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263785a2:::benchmark_data_smithi188_89276_object43671:head# 2024-04-23T16:25:47.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26381e15:::benchmark_data_smithi188_89276_object67151:head# 2024-04-23T16:25:47.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263902e3:::benchmark_data_smithi188_89276_object88512:head# 2024-04-23T16:25:47.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26399ea9:::benchmark_data_smithi188_89276_object101730:head# 2024-04-23T16:25:47.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263ad656:::benchmark_data_smithi188_89276_object151252:head# 2024-04-23T16:25:47.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263b0d8b:::benchmark_data_smithi188_89276_object113827:head# 2024-04-23T16:25:47.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263c0217:::benchmark_data_smithi188_89276_object63419:head# 2024-04-23T16:25:47.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263c3e69:::benchmark_data_smithi188_89276_object80916:head# 2024-04-23T16:25:47.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263cc994:::benchmark_data_smithi188_89276_object43642:head# 2024-04-23T16:25:47.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263d1c4b:::benchmark_data_smithi188_89276_object70030:head# 2024-04-23T16:25:47.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263d1dc2:::benchmark_data_smithi188_89276_object164385:head# 2024-04-23T16:25:47.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263d4527:::benchmark_data_smithi188_89276_object154168:head# 2024-04-23T16:25:47.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263d93dc:::benchmark_data_smithi188_89276_object118185:head# 2024-04-23T16:25:47.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263e1fe9:::benchmark_data_smithi188_89276_object67047:head# 2024-04-23T16:25:47.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263e2247:::benchmark_data_smithi188_89276_object167172:head# 2024-04-23T16:25:47.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263f1a7e:::benchmark_data_smithi188_89276_object129940:head# 2024-04-23T16:25:47.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263fac1a:::benchmark_data_smithi188_89276_object107166:head# 2024-04-23T16:25:47.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:263fcc87:::benchmark_data_smithi188_89276_object76255:head# 2024-04-23T16:25:47.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26409551:::benchmark_data_smithi188_89276_object51821:head# 2024-04-23T16:25:47.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2640ed0e:::benchmark_data_smithi188_89276_object124833:head# 2024-04-23T16:25:47.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2640f1bb:::benchmark_data_smithi188_89276_object120142:head# 2024-04-23T16:25:47.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26422524:::benchmark_data_smithi188_89276_object101717:head# 2024-04-23T16:25:47.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2642c963:::benchmark_data_smithi188_89276_object109595:head# 2024-04-23T16:25:47.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26440633:::benchmark_data_smithi188_89276_object161922:head# 2024-04-23T16:25:47.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:264673c0:::benchmark_data_smithi188_89276_object156542:head# 2024-04-23T16:25:47.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:264684c6:::benchmark_data_smithi188_89276_object110101:head# 2024-04-23T16:25:47.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2646a89f:::benchmark_data_smithi188_89276_object137886:head# 2024-04-23T16:25:47.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26476962:::benchmark_data_smithi188_89276_object142167:head# 2024-04-23T16:25:47.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2647c95b:::benchmark_data_smithi188_89276_object90474:head# 2024-04-23T16:25:47.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26482ea4:::benchmark_data_smithi188_89276_object81716:head# 2024-04-23T16:25:47.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26491682:::benchmark_data_smithi188_89276_object57904:head# 2024-04-23T16:25:47.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26491dae:::benchmark_data_smithi188_89276_object95511:head# 2024-04-23T16:25:47.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2649a914:::benchmark_data_smithi188_89276_object45629:head# 2024-04-23T16:25:47.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:264a5dca:::benchmark_data_smithi188_89276_object152799:head# 2024-04-23T16:25:47.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:264a8042:::benchmark_data_smithi188_89276_object156416:head# 2024-04-23T16:25:47.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:264ac030:::benchmark_data_smithi188_89276_object76016:head# 2024-04-23T16:25:47.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:264b2d54:::benchmark_data_smithi188_89276_object122273:head# 2024-04-23T16:25:47.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:264b6df1:::benchmark_data_smithi188_89276_object151560:head# 2024-04-23T16:25:47.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:264b7e51:::benchmark_data_smithi188_89276_object100824:head# 2024-04-23T16:25:47.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:264c531d:::benchmark_data_smithi188_89276_object53265:head# 2024-04-23T16:25:47.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:264c6016:::benchmark_data_smithi188_89276_object117092:head# 2024-04-23T16:25:47.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:264cee9d:::benchmark_data_smithi188_89276_object70964:head# 2024-04-23T16:25:47.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:264de0ba:::benchmark_data_smithi188_89276_object92776:head# 2024-04-23T16:25:47.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:264ee125:::benchmark_data_smithi188_89276_object84037:head# 2024-04-23T16:25:47.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:264f27cf:::benchmark_data_smithi188_89276_object55834:head# 2024-04-23T16:25:47.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:264fe364:::benchmark_data_smithi188_89276_object79233:head# 2024-04-23T16:25:47.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2650b7b3:::benchmark_data_smithi188_89276_object111896:head# 2024-04-23T16:25:47.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26523591:::benchmark_data_smithi188_89276_object100338:head# 2024-04-23T16:25:47.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2652aa9f:::benchmark_data_smithi188_89276_object59173:head# 2024-04-23T16:25:47.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2652b8bf:::benchmark_data_smithi188_89276_object57731:head# 2024-04-23T16:25:47.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26538616:::benchmark_data_smithi188_89276_object62514:head# 2024-04-23T16:25:47.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26542ab4:::benchmark_data_smithi188_89276_object163911:head# 2024-04-23T16:25:47.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2655c51b:::benchmark_data_smithi188_89276_object153279:head# 2024-04-23T16:25:47.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:265625b5:::benchmark_data_smithi188_89276_object103796:head# 2024-04-23T16:25:47.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26565db0:::benchmark_data_smithi188_89276_object67505:head# 2024-04-23T16:25:47.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26567d75:::benchmark_data_smithi188_89276_object97945:head# 2024-04-23T16:25:47.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:265690bf:::benchmark_data_smithi188_89276_object76303:head# 2024-04-23T16:25:47.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2656e4e1:::benchmark_data_smithi188_89276_object45121:head# 2024-04-23T16:25:47.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2656fe5a:::benchmark_data_smithi188_89276_object42241:head# 2024-04-23T16:25:47.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26572ebd:::benchmark_data_smithi188_89276_object162249:head# 2024-04-23T16:25:47.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26578be2:::benchmark_data_smithi188_89276_object97931:head# 2024-04-23T16:25:47.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2657b8d6:::benchmark_data_smithi188_89276_object159785:head# 2024-04-23T16:25:47.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2658774c:::benchmark_data_smithi188_89276_object73664:head# 2024-04-23T16:25:47.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26587b4e:::benchmark_data_smithi188_89276_object137505:head# 2024-04-23T16:25:47.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26589df0:::benchmark_data_smithi188_89276_object141253:head# 2024-04-23T16:25:47.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2658c8f8:::benchmark_data_smithi188_89276_object115146:head# 2024-04-23T16:25:47.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2659895c:::benchmark_data_smithi188_89276_object64080:head# 2024-04-23T16:25:47.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2659dccc:::benchmark_data_smithi188_89276_object66521:head# 2024-04-23T16:25:47.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:265a5a08:::benchmark_data_smithi188_89276_object89340:head# 2024-04-23T16:25:47.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:265afba3:::benchmark_data_smithi188_89276_object94751:head# 2024-04-23T16:25:47.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:265b6e00:::benchmark_data_smithi188_89276_object168528:head# 2024-04-23T16:25:47.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:265bc41a:::benchmark_data_smithi188_89276_object55710:head# 2024-04-23T16:25:47.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:265bcdad:::benchmark_data_smithi188_89276_object81252:head# 2024-04-23T16:25:47.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:265bd3c8:::benchmark_data_smithi188_89276_object168012:head# 2024-04-23T16:25:47.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:265bd538:::benchmark_data_smithi188_89276_object69017:head# 2024-04-23T16:25:47.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:265d856c:::benchmark_data_smithi188_89276_object65958:head# 2024-04-23T16:25:47.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:265d8cb0:::benchmark_data_smithi188_89276_object106329:head# 2024-04-23T16:25:47.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:265e0834:::benchmark_data_smithi188_89276_object144786:head# 2024-04-23T16:25:47.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:265f882b:::benchmark_data_smithi188_89276_object66969:head# 2024-04-23T16:25:47.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:265fa84a:::benchmark_data_smithi188_89276_object103720:head# 2024-04-23T16:25:47.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:265fc1fa:::benchmark_data_smithi188_89276_object120259:head# 2024-04-23T16:25:47.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26611b47:::benchmark_data_smithi188_89276_object144513:head# 2024-04-23T16:25:47.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:266129a4:::benchmark_data_smithi188_89276_object133009:head# 2024-04-23T16:25:47.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:266138af:::benchmark_data_smithi188_89276_object100151:head# 2024-04-23T16:25:47.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26614b69:::benchmark_data_smithi188_89276_object95489:head# 2024-04-23T16:25:47.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:266183bf:::benchmark_data_smithi188_89276_object72546:head# 2024-04-23T16:25:47.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2662950a:::benchmark_data_smithi188_89276_object101386:head# 2024-04-23T16:25:47.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26632e68:::benchmark_data_smithi188_89276_object77621:head# 2024-04-23T16:25:47.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26634964:::benchmark_data_smithi188_89276_object98368:head# 2024-04-23T16:25:47.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:266374bc:::benchmark_data_smithi188_89276_object69035:head# 2024-04-23T16:25:47.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2663bdf5:::benchmark_data_smithi188_89276_object56980:head# 2024-04-23T16:25:47.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2664e66d:::benchmark_data_smithi188_89276_object133163:head# 2024-04-23T16:25:47.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:266538c9:::benchmark_data_smithi188_89276_object67365:head# 2024-04-23T16:25:47.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26660e0f:::benchmark_data_smithi188_89276_object133785:head# 2024-04-23T16:25:47.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26665d3f:::benchmark_data_smithi188_89276_object88067:head# 2024-04-23T16:25:47.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2667cc5d:::benchmark_data_smithi188_89276_object116212:head# 2024-04-23T16:25:47.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2668193f:::benchmark_data_smithi188_89276_object62899:head# 2024-04-23T16:25:47.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26698c5b:::benchmark_data_smithi188_89276_object135146:head# 2024-04-23T16:25:47.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2669d581:::benchmark_data_smithi188_89276_object135583:head# 2024-04-23T16:25:47.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:266a70f1:::benchmark_data_smithi188_89276_object84633:head# 2024-04-23T16:25:47.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:266a710d:::benchmark_data_smithi188_89276_object124828:head# 2024-04-23T16:25:47.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:266b0054:::benchmark_data_smithi188_89276_object167803:head# 2024-04-23T16:25:47.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:266b7316:::benchmark_data_smithi188_89276_object74010:head# 2024-04-23T16:25:47.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:266bd7c6:::benchmark_data_smithi188_89276_object128042:head# 2024-04-23T16:25:47.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:266c02ed:::benchmark_data_smithi188_89276_object64579:head# 2024-04-23T16:25:47.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:266c1283:::benchmark_data_smithi188_89276_object43088:head# 2024-04-23T16:25:47.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:266d460c:::benchmark_data_smithi188_89276_object138082:head# 2024-04-23T16:25:47.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:266db932:::benchmark_data_smithi188_89276_object155968:head# 2024-04-23T16:25:47.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:266ea968:::benchmark_data_smithi188_89276_object67939:head# 2024-04-23T16:25:47.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:266f22f5:::benchmark_data_smithi188_89276_object168372:head# 2024-04-23T16:25:47.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:266fc15e:::benchmark_data_smithi188_89276_object69732:head# 2024-04-23T16:25:47.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2670a9dd:::benchmark_data_smithi188_89276_object81744:head# 2024-04-23T16:25:47.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26714a38:::benchmark_data_smithi188_89276_object163178:head# 2024-04-23T16:25:47.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:267192c3:::benchmark_data_smithi188_89276_object134142:head# 2024-04-23T16:25:47.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2671be05:::benchmark_data_smithi188_89276_object59712:head# 2024-04-23T16:25:47.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:267255fd:::benchmark_data_smithi188_89276_object109701:head# 2024-04-23T16:25:47.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2672d300:::benchmark_data_smithi188_89276_object137978:head# 2024-04-23T16:25:47.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2672fc09:::benchmark_data_smithi188_89276_object57467:head# 2024-04-23T16:25:47.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2673ccbe:::benchmark_data_smithi188_89276_object71584:head# 2024-04-23T16:25:47.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2673ce46:::benchmark_data_smithi188_89276_object88193:head# 2024-04-23T16:25:47.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:267402ff:::benchmark_data_smithi188_89276_object156113:head# 2024-04-23T16:25:47.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:267522e3:::benchmark_data_smithi188_89276_object161039:head# 2024-04-23T16:25:47.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2675800f:::benchmark_data_smithi188_89276_object149183:head# 2024-04-23T16:25:47.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2676b4de:::benchmark_data_smithi188_89276_object73141:head# 2024-04-23T16:25:47.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2676c18e:::benchmark_data_smithi188_89276_object72380:head# 2024-04-23T16:25:47.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:267713b4:::benchmark_data_smithi188_89276_object162364:head# 2024-04-23T16:25:47.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2677260a:::benchmark_data_smithi188_89276_object142636:head# 2024-04-23T16:25:47.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26773947:::benchmark_data_smithi188_89276_object122341:head# 2024-04-23T16:25:47.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26777f2f:::benchmark_data_smithi188_89276_object94179:head# 2024-04-23T16:25:47.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2677b6e9:::benchmark_data_smithi188_89276_object76269:head# 2024-04-23T16:25:47.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26789bed:::benchmark_data_smithi188_89276_object56137:head# 2024-04-23T16:25:47.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26791992:::benchmark_data_smithi188_89276_object79510:head# 2024-04-23T16:25:47.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:267965f0:::benchmark_data_smithi188_89276_object43549:head# 2024-04-23T16:25:47.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2679d553:::benchmark_data_smithi188_89276_object84846:head# 2024-04-23T16:25:47.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:267a4537:::benchmark_data_smithi188_89276_object100765:head# 2024-04-23T16:25:47.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:267a4c54:::benchmark_data_smithi188_89276_object153399:head# 2024-04-23T16:25:47.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:267c53ac:::benchmark_data_smithi188_89276_object107463:head# 2024-04-23T16:25:47.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:267cb3da:::benchmark_data_smithi188_89276_object155596:head# 2024-04-23T16:25:47.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:267d0ff0:::benchmark_data_smithi188_89276_object63201:head# 2024-04-23T16:25:47.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:267ece41:::benchmark_data_smithi188_89276_object131532:head# 2024-04-23T16:25:47.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:267efba8:::benchmark_data_smithi188_89276_object82728:head# 2024-04-23T16:25:47.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:267f8bd8:::benchmark_data_smithi188_89276_object143862:head# 2024-04-23T16:25:47.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:267fd118:::benchmark_data_smithi188_89276_object133265:head# 2024-04-23T16:25:47.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2680282b:::benchmark_data_smithi188_89276_object128947:head# 2024-04-23T16:25:47.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26816eb4:::benchmark_data_smithi188_89276_object66819:head# 2024-04-23T16:25:47.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26817600:::benchmark_data_smithi188_89276_object112093:head# 2024-04-23T16:25:47.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2682facf:::benchmark_data_smithi188_89276_object70309:head# 2024-04-23T16:25:47.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2684ac5e:::benchmark_data_smithi188_89276_object118680:head# 2024-04-23T16:25:47.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2684efd5:::benchmark_data_smithi188_89276_object68340:head# 2024-04-23T16:25:47.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26853018:::benchmark_data_smithi188_89276_object107213:head# 2024-04-23T16:25:47.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26854a19:::benchmark_data_smithi188_89276_object90112:head# 2024-04-23T16:25:47.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2685a3ad:::benchmark_data_smithi188_89276_object61245:head# 2024-04-23T16:25:47.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:268623fc:::benchmark_data_smithi188_89276_object99434:head# 2024-04-23T16:25:47.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26872a8d:::benchmark_data_smithi188_89276_object129213:head# 2024-04-23T16:25:47.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26882a48:::benchmark_data_smithi188_89276_object146738:head# 2024-04-23T16:25:47.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26890f5d:::benchmark_data_smithi188_89276_object108193:head# 2024-04-23T16:25:47.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2689b47e:::benchmark_data_smithi188_89276_object115455:head# 2024-04-23T16:25:47.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:268a9184:::benchmark_data_smithi188_89276_object59787:head# 2024-04-23T16:25:47.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:268ae579:::benchmark_data_smithi188_89276_object109541:head# 2024-04-23T16:25:47.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:268af870:::benchmark_data_smithi188_89276_object90748:head# 2024-04-23T16:25:47.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:268b50c3:::benchmark_data_smithi188_89276_object44101:head# 2024-04-23T16:25:47.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:268c57e0:::benchmark_data_smithi188_89276_object47593:head# 2024-04-23T16:25:47.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:268dd3b7:::benchmark_data_smithi188_89276_object168071:head# 2024-04-23T16:25:47.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:268edc28:::benchmark_data_smithi188_89276_object143548:head# 2024-04-23T16:25:47.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:268edce9:::benchmark_data_smithi188_89276_object77397:head# 2024-04-23T16:25:47.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26923f65:::benchmark_data_smithi188_89276_object77097:head# 2024-04-23T16:25:47.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2693cdcd:::benchmark_data_smithi188_89276_object151417:head# 2024-04-23T16:25:47.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2694122b:::benchmark_data_smithi188_89276_object94708:head# 2024-04-23T16:25:47.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26948b01:::benchmark_data_smithi188_89276_object78305:head# 2024-04-23T16:25:47.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2695045d:::benchmark_data_smithi188_89276_object102380:head# 2024-04-23T16:25:47.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26959c2d:::benchmark_data_smithi188_89276_object60389:head# 2024-04-23T16:25:47.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2696230f:::benchmark_data_smithi188_89276_object102527:head# 2024-04-23T16:25:47.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:269623e8:::benchmark_data_smithi188_89276_object72085:head# 2024-04-23T16:25:47.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26963755:::benchmark_data_smithi188_89276_object119541:head# 2024-04-23T16:25:47.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2696f50f:::benchmark_data_smithi188_89276_object113590:head# 2024-04-23T16:25:47.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26980744:::benchmark_data_smithi188_89276_object161805:head# 2024-04-23T16:25:47.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:269973bb:::benchmark_data_smithi188_89276_object148304:head# 2024-04-23T16:25:47.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2699af6b:::benchmark_data_smithi188_89276_object134466:head# 2024-04-23T16:25:47.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2699f56e:::benchmark_data_smithi188_89276_object123447:head# 2024-04-23T16:25:47.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:269a4457:::benchmark_data_smithi188_89276_object121592:head# 2024-04-23T16:25:47.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:269a7df8:::benchmark_data_smithi188_89276_object64353:head# 2024-04-23T16:25:47.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:269a8bd2:::benchmark_data_smithi188_89276_object83231:head# 2024-04-23T16:25:47.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:269b44ff:::benchmark_data_smithi188_89276_object136666:head# 2024-04-23T16:25:47.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:269ca5b3:::benchmark_data_smithi188_89276_object103834:head# 2024-04-23T16:25:47.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:269cffa4:::benchmark_data_smithi188_89276_object83125:head# 2024-04-23T16:25:47.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:269d954f:::benchmark_data_smithi188_89276_object146571:head# 2024-04-23T16:25:47.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:269e7bd7:::benchmark_data_smithi188_89276_object155057:head# 2024-04-23T16:25:47.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:269f6774:::benchmark_data_smithi188_89276_object116200:head# 2024-04-23T16:25:47.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:269fc628:::benchmark_data_smithi188_89276_object157664:head# 2024-04-23T16:25:47.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:269fe7ad:::benchmark_data_smithi188_89276_object144390:head# 2024-04-23T16:25:47.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26a044c9:::benchmark_data_smithi188_89276_object89053:head# 2024-04-23T16:25:47.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26a0b60e:::benchmark_data_smithi188_89276_object52756:head# 2024-04-23T16:25:47.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26a0e6af:::benchmark_data_smithi188_89276_object152214:head# 2024-04-23T16:25:47.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26a0f480:::benchmark_data_smithi188_89276_object132717:head# 2024-04-23T16:25:47.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26a17bc1:::benchmark_data_smithi188_89276_object97157:head# 2024-04-23T16:25:47.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26a1f922:::benchmark_data_smithi188_89276_object165865:head# 2024-04-23T16:25:47.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26a21aa7:::benchmark_data_smithi188_89276_object126864:head# 2024-04-23T16:25:47.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26a23d86:::benchmark_data_smithi188_89276_object78025:head# 2024-04-23T16:25:47.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26a37992:::benchmark_data_smithi188_89276_object130929:head# 2024-04-23T16:25:47.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26a3b9ab:::benchmark_data_smithi188_89276_object43971:head# 2024-04-23T16:25:47.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26a4c0d0:::benchmark_data_smithi188_89276_object106682:head# 2024-04-23T16:25:47.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26a4c275:::benchmark_data_smithi188_89276_object66258:head# 2024-04-23T16:25:47.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26a57fb1:::benchmark_data_smithi188_89276_object122124:head# 2024-04-23T16:25:47.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26a5c9eb:::benchmark_data_smithi188_89276_object45433:head# 2024-04-23T16:25:47.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26a6c765:::benchmark_data_smithi188_89276_object95827:head# 2024-04-23T16:25:47.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26a6dc31:::benchmark_data_smithi188_89276_object142316:head# 2024-04-23T16:25:47.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26a6fac1:::benchmark_data_smithi188_89276_object116130:head# 2024-04-23T16:25:47.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26a9704f:::benchmark_data_smithi188_89276_object142253:head# 2024-04-23T16:25:47.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26aa670b:::benchmark_data_smithi188_89276_object140772:head# 2024-04-23T16:25:47.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ab4806:::benchmark_data_smithi188_89276_object128290:head# 2024-04-23T16:25:47.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ab571c:::benchmark_data_smithi188_89276_object161037:head# 2024-04-23T16:25:47.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ac03cf:::benchmark_data_smithi188_89276_object44309:head# 2024-04-23T16:25:47.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ac467d:::benchmark_data_smithi188_89276_object96512:head# 2024-04-23T16:25:47.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ad5ebd:::benchmark_data_smithi188_89276_object69887:head# 2024-04-23T16:25:47.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26adc7ed:::benchmark_data_smithi188_89276_object61708:head# 2024-04-23T16:25:47.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ae673d:::benchmark_data_smithi188_89276_object150417:head# 2024-04-23T16:25:47.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26aec943:::benchmark_data_smithi188_89276_object109544:head# 2024-04-23T16:25:47.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26afd7d6:::benchmark_data_smithi188_89276_object93913:head# 2024-04-23T16:25:47.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26aff748:::benchmark_data_smithi188_89276_object129011:head# 2024-04-23T16:25:47.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b0cc6b:::benchmark_data_smithi188_89276_object112885:head# 2024-04-23T16:25:47.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b160c5:::benchmark_data_smithi188_89276_object104642:head# 2024-04-23T16:25:47.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b171d1:::benchmark_data_smithi188_89276_object138242:head# 2024-04-23T16:25:47.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b17dc3:::benchmark_data_smithi188_89276_object145238:head# 2024-04-23T16:25:47.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b1acf7:::benchmark_data_smithi188_89276_object40291:head# 2024-04-23T16:25:47.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b205f5:::benchmark_data_smithi188_89276_object94211:head# 2024-04-23T16:25:47.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b241d1:::benchmark_data_smithi188_89276_object167712:head# 2024-04-23T16:25:47.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b2885d:::benchmark_data_smithi188_89276_object83165:head# 2024-04-23T16:25:47.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b32d1e:::benchmark_data_smithi188_89276_object105534:head# 2024-04-23T16:25:47.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b330f5:::benchmark_data_smithi188_89276_object159282:head# 2024-04-23T16:25:47.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b3c736:::benchmark_data_smithi188_89276_object127096:head# 2024-04-23T16:25:47.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b3f548:::benchmark_data_smithi188_89276_object127850:head# 2024-04-23T16:25:47.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b4dd02:::benchmark_data_smithi188_89276_object165236:head# 2024-04-23T16:25:47.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b5de14:::benchmark_data_smithi188_89276_object77490:head# 2024-04-23T16:25:47.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b61fb2:::benchmark_data_smithi188_89276_object106705:head# 2024-04-23T16:25:47.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b71016:::benchmark_data_smithi188_89276_object138261:head# 2024-04-23T16:25:47.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b72944:::benchmark_data_smithi188_89276_object48594:head# 2024-04-23T16:25:47.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b7842b:::benchmark_data_smithi188_89276_object84827:head# 2024-04-23T16:25:47.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b8561b:::benchmark_data_smithi188_89276_object84187:head# 2024-04-23T16:25:47.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b8f9c3:::benchmark_data_smithi188_89276_object163550:head# 2024-04-23T16:25:47.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26b994e4:::benchmark_data_smithi188_89276_object147299:head# 2024-04-23T16:25:47.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ba5986:::benchmark_data_smithi188_89276_object40448:head# 2024-04-23T16:25:47.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ba6acd:::benchmark_data_smithi188_89276_object126610:head# 2024-04-23T16:25:47.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26baab07:::benchmark_data_smithi188_89276_object81394:head# 2024-04-23T16:25:47.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26bb24ca:::benchmark_data_smithi188_89276_object85975:head# 2024-04-23T16:25:47.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26bba62e:::benchmark_data_smithi188_89276_object85381:head# 2024-04-23T16:25:47.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26bbc461:::benchmark_data_smithi188_89276_object153564:head# 2024-04-23T16:25:47.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26bc3561:::benchmark_data_smithi188_89276_object118360:head# 2024-04-23T16:25:47.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26bc6ae7:::benchmark_data_smithi188_89276_object159687:head# 2024-04-23T16:25:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26bd0f9f:::benchmark_data_smithi188_89276_object156816:head# 2024-04-23T16:25:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26bd52a9:::benchmark_data_smithi188_89276_object150673:head# 2024-04-23T16:25:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26bdff0e:::benchmark_data_smithi188_89276_object96773:head# 2024-04-23T16:25:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26be006f:::benchmark_data_smithi188_89276_object74521:head# 2024-04-23T16:25:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26becace:::benchmark_data_smithi188_89276_object143299:head# 2024-04-23T16:25:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26bf35b6:::benchmark_data_smithi188_89276_object49230:head# 2024-04-23T16:25:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26bfc9e2:::benchmark_data_smithi188_89276_object112203:head# 2024-04-23T16:25:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26c08b1b:::benchmark_data_smithi188_89276_object159205:head# 2024-04-23T16:25:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26c18e9d:::benchmark_data_smithi188_89276_object129925:head# 2024-04-23T16:25:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26c2514a:::benchmark_data_smithi188_89276_object74387:head# 2024-04-23T16:25:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26c258f6:::benchmark_data_smithi188_89276_object142878:head# 2024-04-23T16:25:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26c2f464:::benchmark_data_smithi188_89276_object109310:head# 2024-04-23T16:25:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26c35a7f:::benchmark_data_smithi188_89276_object44729:head# 2024-04-23T16:25:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26c4d1ce:::benchmark_data_smithi188_89276_object165872:head# 2024-04-23T16:25:47.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26c514d8:::benchmark_data_smithi188_89276_object145527:head# 2024-04-23T16:25:47.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26c5d7a1:::benchmark_data_smithi188_89276_object116786:head# 2024-04-23T16:25:47.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26c5ef2d:::benchmark_data_smithi188_89276_object55412:head# 2024-04-23T16:25:47.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26c6f6f4:::benchmark_data_smithi188_89276_object129909:head# 2024-04-23T16:25:47.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26c85e77:::benchmark_data_smithi188_89276_object121090:head# 2024-04-23T16:25:47.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26c95fc8:::benchmark_data_smithi188_89276_object149171:head# 2024-04-23T16:25:47.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26c99891:::benchmark_data_smithi188_89276_object68588:head# 2024-04-23T16:25:47.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26c9a75f:::benchmark_data_smithi188_89276_object108679:head# 2024-04-23T16:25:47.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26c9d964:::benchmark_data_smithi188_89276_object50601:head# 2024-04-23T16:25:47.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ca3089:::benchmark_data_smithi188_89276_object70448:head# 2024-04-23T16:25:47.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ca6800:::benchmark_data_smithi188_89276_object144277:head# 2024-04-23T16:25:47.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26cba3eb:::benchmark_data_smithi188_89276_object76783:head# 2024-04-23T16:25:47.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26cbae6c:::benchmark_data_smithi188_89276_object106906:head# 2024-04-23T16:25:47.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26cc6496:::benchmark_data_smithi188_89276_object134916:head# 2024-04-23T16:25:47.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26cccf0a:::benchmark_data_smithi188_89276_object163041:head# 2024-04-23T16:25:47.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26cce75e:::benchmark_data_smithi188_89276_object56514:head# 2024-04-23T16:25:47.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26cddf6f:::benchmark_data_smithi188_89276_object104618:head# 2024-04-23T16:25:47.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26cde28a:::benchmark_data_smithi188_89276_object112704:head# 2024-04-23T16:25:47.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ce27a4:::benchmark_data_smithi188_89276_object94407:head# 2024-04-23T16:25:47.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ce8ab7:::benchmark_data_smithi188_89276_object81378:head# 2024-04-23T16:25:47.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26cf7347:::benchmark_data_smithi188_89276_object57460:head# 2024-04-23T16:25:47.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26cfa531:::benchmark_data_smithi188_89276_object154905:head# 2024-04-23T16:25:47.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26cfaed7:::benchmark_data_smithi188_89276_object158862:head# 2024-04-23T16:25:47.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26d02397:::benchmark_data_smithi188_89276_object53230:head# 2024-04-23T16:25:47.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26d06ea6:::benchmark_data_smithi188_89276_object41532:head# 2024-04-23T16:25:47.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26d097eb:::benchmark_data_smithi188_89276_object108944:head# 2024-04-23T16:25:47.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26d1e582:::benchmark_data_smithi188_89276_object41110:head# 2024-04-23T16:25:47.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26d1e8da:::benchmark_data_smithi188_89276_object82141:head# 2024-04-23T16:25:47.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26d1f2cb:::benchmark_data_smithi188_89276_object130152:head# 2024-04-23T16:25:47.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26d3a6a2:::benchmark_data_smithi188_89276_object128522:head# 2024-04-23T16:25:47.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26d519dc:::benchmark_data_smithi188_89276_object113521:head# 2024-04-23T16:25:47.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26d56e26:::benchmark_data_smithi188_89276_object156921:head# 2024-04-23T16:25:47.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26d6bb9d:::benchmark_data_smithi188_89276_object102532:head# 2024-04-23T16:25:47.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26d71366:::benchmark_data_smithi188_89276_object165088:head# 2024-04-23T16:25:47.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26d77564:::benchmark_data_smithi188_89276_object153300:head# 2024-04-23T16:25:47.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26d83216:::benchmark_data_smithi188_89276_object42631:head# 2024-04-23T16:25:47.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26d89ebb:::benchmark_data_smithi188_89276_object103254:head# 2024-04-23T16:25:47.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26d8cc5b:::benchmark_data_smithi188_89276_object61533:head# 2024-04-23T16:25:47.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26d96fc2:::benchmark_data_smithi188_89276_object69193:head# 2024-04-23T16:25:47.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26d97f8a:::benchmark_data_smithi188_89276_object74503:head# 2024-04-23T16:25:47.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26d9870c:::benchmark_data_smithi188_89276_object153345:head# 2024-04-23T16:25:47.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26da1f03:::benchmark_data_smithi188_89276_object143850:head# 2024-04-23T16:25:47.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26da6de5:::benchmark_data_smithi188_89276_object54414:head# 2024-04-23T16:25:47.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26da9cf0:::benchmark_data_smithi188_89276_object92153:head# 2024-04-23T16:25:47.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26db6c84:::benchmark_data_smithi188_89276_object127907:head# 2024-04-23T16:25:47.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26db9c19:::benchmark_data_smithi188_89276_object100250:head# 2024-04-23T16:25:47.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26dbed61:::benchmark_data_smithi188_89276_object78090:head# 2024-04-23T16:25:47.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26dc7a03:::benchmark_data_smithi188_89276_object132098:head# 2024-04-23T16:25:47.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26dc986e:::benchmark_data_smithi188_89276_object100618:head# 2024-04-23T16:25:47.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26dcd694:::benchmark_data_smithi188_89276_object163620:head# 2024-04-23T16:25:47.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26dda503:::benchmark_data_smithi188_89276_object107070:head# 2024-04-23T16:25:47.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ddb7bd:::benchmark_data_smithi188_89276_object152442:head# 2024-04-23T16:25:47.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26de0c84:::benchmark_data_smithi188_89276_object75567:head# 2024-04-23T16:25:47.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26de484d:::benchmark_data_smithi188_89276_object137479:head# 2024-04-23T16:25:47.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26de7933:::benchmark_data_smithi188_89276_object84789:head# 2024-04-23T16:25:47.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26df0fce:::benchmark_data_smithi188_89276_object92856:head# 2024-04-23T16:25:47.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26e00735:::benchmark_data_smithi188_89276_object57802:head# 2024-04-23T16:25:47.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26e1a2ff:::benchmark_data_smithi188_89276_object163056:head# 2024-04-23T16:25:47.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26e245f5:::benchmark_data_smithi188_89276_object91684:head# 2024-04-23T16:25:47.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26e2bba7:::benchmark_data_smithi188_89276_object113339:head# 2024-04-23T16:25:47.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26e2de7c:::benchmark_data_smithi188_89276_object75595:head# 2024-04-23T16:25:47.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26e332bf:::benchmark_data_smithi188_89276_object166203:head# 2024-04-23T16:25:47.322 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26e36e3d:::benchmark_data_smithi188_89276_object142479:head# 2024-04-23T16:25:47.322 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26e3a598:::benchmark_data_smithi188_89276_object109574:head# 2024-04-23T16:25:47.322 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26e41421:::benchmark_data_smithi188_89276_object140698:head# 2024-04-23T16:25:47.322 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26e5050e:::benchmark_data_smithi188_89276_object43119:head# 2024-04-23T16:25:47.322 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26e63726:::benchmark_data_smithi188_89276_object106243:head# 2024-04-23T16:25:47.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26e6bc5c:::benchmark_data_smithi188_89276_object59105:head# 2024-04-23T16:25:47.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26e6d8df:::benchmark_data_smithi188_89276_object55462:head# 2024-04-23T16:25:47.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26e83f48:::benchmark_data_smithi188_89276_object132808:head# 2024-04-23T16:25:47.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26e919a0:::benchmark_data_smithi188_89276_object109845:head# 2024-04-23T16:25:47.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ea0e66:::benchmark_data_smithi188_89276_object66833:head# 2024-04-23T16:25:47.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ea367a:::benchmark_data_smithi188_89276_object70176:head# 2024-04-23T16:25:47.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ea4ed8:::benchmark_data_smithi188_89276_object114431:head# 2024-04-23T16:25:47.324 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26eaa818:::benchmark_data_smithi188_89276_object128518:head# 2024-04-23T16:25:47.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26eadfdd:::benchmark_data_smithi188_89276_object47227:head# 2024-04-23T16:25:47.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ebd5f4:::benchmark_data_smithi188_89276_object150465:head# 2024-04-23T16:25:47.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ec37ac:::benchmark_data_smithi188_89276_object55814:head# 2024-04-23T16:25:47.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ecca35:::benchmark_data_smithi188_89276_object55716:head# 2024-04-23T16:25:47.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ed0ff5:::benchmark_data_smithi188_89276_object78530:head# 2024-04-23T16:25:47.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ed13a3:::benchmark_data_smithi188_89276_object91446:head# 2024-04-23T16:25:47.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ee19a9:::benchmark_data_smithi188_89276_object46243:head# 2024-04-23T16:25:47.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26eecdc3:::benchmark_data_smithi188_89276_object52674:head# 2024-04-23T16:25:47.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ef2a22:::benchmark_data_smithi188_89276_object87733:head# 2024-04-23T16:25:47.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f01ad8:::benchmark_data_smithi188_89276_object139396:head# 2024-04-23T16:25:47.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f0241f:::benchmark_data_smithi188_89276_object89105:head# 2024-04-23T16:25:47.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f07443:::benchmark_data_smithi188_89276_object101106:head# 2024-04-23T16:25:47.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f0cfaa:::benchmark_data_smithi188_89276_object51073:head# 2024-04-23T16:25:47.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f0d37e:::benchmark_data_smithi188_89276_object158762:head# 2024-04-23T16:25:47.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f177d0:::benchmark_data_smithi188_89276_object42073:head# 2024-04-23T16:25:47.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f2e674:::benchmark_data_smithi188_89276_object72698:head# 2024-04-23T16:25:47.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f36b28:::benchmark_data_smithi188_89276_object82066:head# 2024-04-23T16:25:47.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f3745e:::benchmark_data_smithi188_89276_object135356:head# 2024-04-23T16:25:47.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f3a23b:::benchmark_data_smithi188_89276_object85739:head# 2024-04-23T16:25:47.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f3ccdc:::benchmark_data_smithi188_89276_object63214:head# 2024-04-23T16:25:47.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f499fd:::benchmark_data_smithi188_89276_object98528:head# 2024-04-23T16:25:47.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f4cc3d:::benchmark_data_smithi188_89276_object64584:head# 2024-04-23T16:25:47.334 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.334 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f4ea6d:::benchmark_data_smithi188_89276_object50043:head# 2024-04-23T16:25:47.334 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.334 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f59f00:::benchmark_data_smithi188_89276_object59711:head# 2024-04-23T16:25:47.334 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.334 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f63584:::benchmark_data_smithi188_89276_object127081:head# 2024-04-23T16:25:47.334 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.334 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f68fd1:::benchmark_data_smithi188_89276_object81620:head# 2024-04-23T16:25:47.334 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.334 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f6a3c6:::benchmark_data_smithi188_89276_object74365:head# 2024-04-23T16:25:47.334 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.334 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f7d51f:::benchmark_data_smithi188_89276_object96533:head# 2024-04-23T16:25:47.334 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.334 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f7db0c:::benchmark_data_smithi188_89276_object163981:head# 2024-04-23T16:25:47.335 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.335 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f7eb20:::benchmark_data_smithi188_89276_object144887:head# 2024-04-23T16:25:47.335 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.335 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f8420d:::benchmark_data_smithi188_89276_object96405:head# 2024-04-23T16:25:47.335 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.335 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f84dd6:::benchmark_data_smithi188_89276_object133831:head# 2024-04-23T16:25:47.335 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.335 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f8ab3f:::benchmark_data_smithi188_89276_object130429:head# 2024-04-23T16:25:47.335 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.335 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f8c8c6:::benchmark_data_smithi188_89276_object91137:head# 2024-04-23T16:25:47.336 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.336 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f91033:::benchmark_data_smithi188_89276_object150031:head# 2024-04-23T16:25:47.336 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.336 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f94bad:::benchmark_data_smithi188_89276_object83682:head# 2024-04-23T16:25:47.336 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.336 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f98d6d:::benchmark_data_smithi188_89276_object47118:head# 2024-04-23T16:25:47.336 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.336 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26f9dd00:::benchmark_data_smithi188_89276_object144521:head# 2024-04-23T16:25:47.336 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.336 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26fa9e9d:::benchmark_data_smithi188_89276_object120530:head# 2024-04-23T16:25:47.336 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.336 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26fafdd3:::benchmark_data_smithi188_89276_object107059:head# 2024-04-23T16:25:47.336 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.336 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26fc14f0:::benchmark_data_smithi188_89276_object126841:head# 2024-04-23T16:25:47.336 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.337 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26fc5ffc:::benchmark_data_smithi188_89276_object126784:head# 2024-04-23T16:25:47.337 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.337 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26fcf13e:::benchmark_data_smithi188_89276_object138582:head# 2024-04-23T16:25:47.337 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.337 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26fe7afc:::benchmark_data_smithi188_89276_object117022:head# 2024-04-23T16:25:47.337 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.337 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26fec601:::benchmark_data_smithi188_89276_object83324:head# 2024-04-23T16:25:47.337 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.337 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ffac54:::benchmark_data_smithi188_89276_object56776:head# 2024-04-23T16:25:47.337 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.337 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:26ffc0c0:::benchmark_data_smithi188_89276_object97580:head# 2024-04-23T16:25:47.338 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.339 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2701657e:::benchmark_data_smithi188_89276_object79665:head# 2024-04-23T16:25:47.339 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27017806:::benchmark_data_smithi188_89276_object83955:head# 2024-04-23T16:25:47.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2701b29b:::benchmark_data_smithi188_89276_object44927:head# 2024-04-23T16:25:47.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27024600:::benchmark_data_smithi188_89276_object100528:head# 2024-04-23T16:25:47.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27025dde:::benchmark_data_smithi188_89276_object151864:head# 2024-04-23T16:25:47.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2702f262:::benchmark_data_smithi188_89276_object144743:head# 2024-04-23T16:25:47.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2703243f:::benchmark_data_smithi188_89276_object98837:head# 2024-04-23T16:25:47.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.341 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2703bb37:::benchmark_data_smithi188_89276_object40905:head# 2024-04-23T16:25:47.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.341 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2703bf61:::benchmark_data_smithi188_89276_object152029:head# 2024-04-23T16:25:47.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.341 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2703cf6f:::benchmark_data_smithi188_89276_object43508:head# 2024-04-23T16:25:47.342 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.342 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:270490cb:::benchmark_data_smithi188_89276_object112196:head# 2024-04-23T16:25:47.342 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2704dbfc:::benchmark_data_smithi188_89276_object109453:head# 2024-04-23T16:25:47.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2704f87c:::benchmark_data_smithi188_89276_object101233:head# 2024-04-23T16:25:47.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:270514a3:::benchmark_data_smithi188_89276_object96444:head# 2024-04-23T16:25:47.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:270571de:::benchmark_data_smithi188_89276_object123764:head# 2024-04-23T16:25:47.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27058a0c:::benchmark_data_smithi188_89276_object127752:head# 2024-04-23T16:25:47.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2705d278:::benchmark_data_smithi188_89276_object133120:head# 2024-04-23T16:25:47.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27067804:::benchmark_data_smithi188_89276_object62056:head# 2024-04-23T16:25:47.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2707fd0a:::benchmark_data_smithi188_89276_object66880:head# 2024-04-23T16:25:47.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2708203d:::benchmark_data_smithi188_89276_object45636:head# 2024-04-23T16:25:47.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2708ca47:::benchmark_data_smithi188_89276_object76782:head# 2024-04-23T16:25:47.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27090a91:::benchmark_data_smithi188_89276_object69224:head# 2024-04-23T16:25:47.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2709faf3:::benchmark_data_smithi188_89276_object144458:head# 2024-04-23T16:25:47.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:270ae8e6:::benchmark_data_smithi188_89276_object134181:head# 2024-04-23T16:25:47.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:270b7984:::benchmark_data_smithi188_89276_object121227:head# 2024-04-23T16:25:47.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:270ba80a:::benchmark_data_smithi188_89276_object54284:head# 2024-04-23T16:25:47.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:270c98dd:::benchmark_data_smithi188_89276_object121773:head# 2024-04-23T16:25:47.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:270d8680:::benchmark_data_smithi188_89276_object79468:head# 2024-04-23T16:25:47.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:270d9d9b:::benchmark_data_smithi188_89276_object128071:head# 2024-04-23T16:25:47.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:270da8ec:::benchmark_data_smithi188_89276_object118739:head# 2024-04-23T16:25:47.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:270e831b:::benchmark_data_smithi188_89276_object94947:head# 2024-04-23T16:25:47.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:270f0344:::benchmark_data_smithi188_89276_object142580:head# 2024-04-23T16:25:47.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:270f1504:::benchmark_data_smithi188_89276_object122082:head# 2024-04-23T16:25:47.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:270f9eb5:::benchmark_data_smithi188_89276_object96626:head# 2024-04-23T16:25:47.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:270fdd00:::benchmark_data_smithi188_89276_object60171:head# 2024-04-23T16:25:47.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2711cd30:::benchmark_data_smithi188_89276_object70235:head# 2024-04-23T16:25:47.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2712723d:::benchmark_data_smithi188_89276_object59685:head# 2024-04-23T16:25:47.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27128129:::benchmark_data_smithi188_89276_object78105:head# 2024-04-23T16:25:47.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2713e93f:::benchmark_data_smithi188_89276_object146994:head# 2024-04-23T16:25:47.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:271486fc:::benchmark_data_smithi188_89276_object68949:head# 2024-04-23T16:25:47.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2715bab4:::benchmark_data_smithi188_89276_object122451:head# 2024-04-23T16:25:47.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2716c243:::benchmark_data_smithi188_89276_object47497:head# 2024-04-23T16:25:47.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2716cb25:::benchmark_data_smithi188_89276_object129643:head# 2024-04-23T16:25:47.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27172926:::benchmark_data_smithi188_89276_object88857:head# 2024-04-23T16:25:47.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27173e3e:::benchmark_data_smithi188_89276_object112708:head# 2024-04-23T16:25:47.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27180709:::benchmark_data_smithi188_89276_object44060:head# 2024-04-23T16:25:47.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2718a363:::benchmark_data_smithi188_89276_object145568:head# 2024-04-23T16:25:47.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2718aed1:::benchmark_data_smithi188_89276_object158178:head# 2024-04-23T16:25:47.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2718eac8:::benchmark_data_smithi188_89276_object75668:head# 2024-04-23T16:25:47.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27190318:::benchmark_data_smithi188_89276_object95140:head# 2024-04-23T16:25:47.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:271a505d:::benchmark_data_smithi188_89276_object123445:head# 2024-04-23T16:25:47.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:271aaafb:::benchmark_data_smithi188_89276_object135256:head# 2024-04-23T16:25:47.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:271ae22f:::benchmark_data_smithi188_89276_object91344:head# 2024-04-23T16:25:47.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:271afa2d:::benchmark_data_smithi188_89276_object144715:head# 2024-04-23T16:25:47.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:271b13de:::benchmark_data_smithi188_89276_object76342:head# 2024-04-23T16:25:47.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:271c1364:::benchmark_data_smithi188_89276_object74419:head# 2024-04-23T16:25:47.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:271d1b08:::benchmark_data_smithi188_89276_object68752:head# 2024-04-23T16:25:47.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:271d4912:::benchmark_data_smithi188_89276_object61477:head# 2024-04-23T16:25:47.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:271dce72:::benchmark_data_smithi188_89276_object126563:head# 2024-04-23T16:25:47.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:271e160a:::benchmark_data_smithi188_89276_object168519:head# 2024-04-23T16:25:47.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.372 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:271e7e1a:::benchmark_data_smithi188_89276_object51217:head# 2024-04-23T16:25:47.372 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.372 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:271ed90c:::benchmark_data_smithi188_89276_object79283:head# 2024-04-23T16:25:47.372 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.372 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:271f0f35:::benchmark_data_smithi188_89276_object123494:head# 2024-04-23T16:25:47.372 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.372 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:271f5d97:::benchmark_data_smithi188_89276_object96384:head# 2024-04-23T16:25:47.372 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.373 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:271f8d43:::benchmark_data_smithi188_89276_object73597:head# 2024-04-23T16:25:47.373 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.373 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:271fa816:::benchmark_data_smithi188_89276_object81062:head# 2024-04-23T16:25:47.373 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.373 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:271fed3a:::benchmark_data_smithi188_89276_object101441:head# 2024-04-23T16:25:47.373 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.373 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2720344e:::benchmark_data_smithi188_89276_object109540:head# 2024-04-23T16:25:47.373 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.374 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2720f1ae:::benchmark_data_smithi188_89276_object104823:head# 2024-04-23T16:25:47.374 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.374 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:272328f3:::benchmark_data_smithi188_89276_object116853:head# 2024-04-23T16:25:47.374 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.374 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:272349a0:::benchmark_data_smithi188_89276_object94148:head# 2024-04-23T16:25:47.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2723c6ed:::benchmark_data_smithi188_89276_object61749:head# 2024-04-23T16:25:47.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2723e227:::benchmark_data_smithi188_89276_object88495:head# 2024-04-23T16:25:47.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2723f93e:::benchmark_data_smithi188_89276_object87713:head# 2024-04-23T16:25:47.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2724a407:::benchmark_data_smithi188_89276_object106382:head# 2024-04-23T16:25:47.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:272564ed:::benchmark_data_smithi188_89276_object97212:head# 2024-04-23T16:25:47.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2726e208:::benchmark_data_smithi188_89276_object147211:head# 2024-04-23T16:25:47.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27270bec:::benchmark_data_smithi188_89276_object159647:head# 2024-04-23T16:25:47.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27278002:::benchmark_data_smithi188_89276_object56986:head# 2024-04-23T16:25:47.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2727fb88:::benchmark_data_smithi188_89276_object147714:head# 2024-04-23T16:25:47.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27281293:::benchmark_data_smithi188_89276_object125438:head# 2024-04-23T16:25:47.378 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.378 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2728db94:::benchmark_data_smithi188_89276_object95330:head# 2024-04-23T16:25:47.378 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.378 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2728f215:::benchmark_data_smithi188_89276_object153687:head# 2024-04-23T16:25:47.378 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.378 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2729d068:::benchmark_data_smithi188_89276_object108720:head# 2024-04-23T16:25:47.378 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.378 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:272ad8cb:::benchmark_data_smithi188_89276_object90730:head# 2024-04-23T16:25:47.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:272b0dbd:::benchmark_data_smithi188_89276_object82390:head# 2024-04-23T16:25:47.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:272c2749:::benchmark_data_smithi188_89276_object132788:head# 2024-04-23T16:25:47.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:272d530e:::benchmark_data_smithi188_89276_object47954:head# 2024-04-23T16:25:47.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:272daa19:::benchmark_data_smithi188_89276_object166498:head# 2024-04-23T16:25:47.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:272e4b46:::benchmark_data_smithi188_89276_object101464:head# 2024-04-23T16:25:47.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:272f3567:::benchmark_data_smithi188_89276_object64691:head# 2024-04-23T16:25:47.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:272f5cb0:::benchmark_data_smithi188_89276_object135460:head# 2024-04-23T16:25:47.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27310200:::benchmark_data_smithi188_89276_object63347:head# 2024-04-23T16:25:47.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2731121b:::benchmark_data_smithi188_89276_object50122:head# 2024-04-23T16:25:47.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2731b1ae:::benchmark_data_smithi188_89276_object41683:head# 2024-04-23T16:25:47.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2731c4b7:::benchmark_data_smithi188_89276_object103563:head# 2024-04-23T16:25:47.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.387 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2731ec1a:::benchmark_data_smithi188_89276_object75262:head# 2024-04-23T16:25:47.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27329cac:::benchmark_data_smithi188_89276_object122483:head# 2024-04-23T16:25:47.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2732a3dc:::benchmark_data_smithi188_89276_object52279:head# 2024-04-23T16:25:47.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2732f60f:::benchmark_data_smithi188_89276_object87615:head# 2024-04-23T16:25:47.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2733a5cc:::benchmark_data_smithi188_89276_object44875:head# 2024-04-23T16:25:47.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:273508fa:::benchmark_data_smithi188_89276_object73147:head# 2024-04-23T16:25:47.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27353d16:::benchmark_data_smithi188_89276_object109017:head# 2024-04-23T16:25:47.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:273561aa:::benchmark_data_smithi188_89276_object134994:head# 2024-04-23T16:25:47.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:273569fc:::benchmark_data_smithi188_89276_object42892:head# 2024-04-23T16:25:47.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:273672d3:::benchmark_data_smithi188_89276_object60304:head# 2024-04-23T16:25:47.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2736f476:::benchmark_data_smithi188_89276_object43042:head# 2024-04-23T16:25:47.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2736f7a1:::benchmark_data_smithi188_89276_object67071:head# 2024-04-23T16:25:47.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27376a65:::benchmark_data_smithi188_89276_object88535:head# 2024-04-23T16:25:47.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27378e75:::benchmark_data_smithi188_89276_object75603:head# 2024-04-23T16:25:47.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2737e6c8:::benchmark_data_smithi188_89276_object102134:head# 2024-04-23T16:25:47.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:273828ad:::benchmark_data_smithi188_89276_object132985:head# 2024-04-23T16:25:47.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2738716a:::benchmark_data_smithi188_89276_object161271:head# 2024-04-23T16:25:47.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2738b383:::benchmark_data_smithi188_89276_object105368:head# 2024-04-23T16:25:47.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27390d8b:::benchmark_data_smithi188_89276_object49716:head# 2024-04-23T16:25:47.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:273912d4:::benchmark_data_smithi188_89276_object67271:head# 2024-04-23T16:25:47.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:273925b5:::benchmark_data_smithi188_89276_object104563:head# 2024-04-23T16:25:47.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27392f32:::benchmark_data_smithi188_89276_object90398:head# 2024-04-23T16:25:47.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27397aaa:::benchmark_data_smithi188_89276_object111026:head# 2024-04-23T16:25:47.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2739804d:::benchmark_data_smithi188_89276_object99000:head# 2024-04-23T16:25:47.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2739ba05:::benchmark_data_smithi188_89276_object102007:head# 2024-04-23T16:25:47.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2739da46:::benchmark_data_smithi188_89276_object90120:head# 2024-04-23T16:25:47.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2739ea9e:::benchmark_data_smithi188_89276_object79641:head# 2024-04-23T16:25:47.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:273a340c:::benchmark_data_smithi188_89276_object75058:head# 2024-04-23T16:25:47.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:273b0e97:::benchmark_data_smithi188_89276_object98005:head# 2024-04-23T16:25:47.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:273ba19e:::benchmark_data_smithi188_89276_object88792:head# 2024-04-23T16:25:47.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:273c59f1:::benchmark_data_smithi188_89276_object143062:head# 2024-04-23T16:25:47.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:273d5ed8:::benchmark_data_smithi188_89276_object51432:head# 2024-04-23T16:25:47.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:273dad8a:::benchmark_data_smithi188_89276_object97498:head# 2024-04-23T16:25:47.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:273de6ba:::benchmark_data_smithi188_89276_object148840:head# 2024-04-23T16:25:47.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:273e8e4b:::benchmark_data_smithi188_89276_object107971:head# 2024-04-23T16:25:47.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:273ee04f:::benchmark_data_smithi188_89276_object134299:head# 2024-04-23T16:25:47.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:273f07b5:::benchmark_data_smithi188_89276_object92935:head# 2024-04-23T16:25:47.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:273f1893:::benchmark_data_smithi188_89276_object53812:head# 2024-04-23T16:25:47.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:273f3557:::benchmark_data_smithi188_89276_object142599:head# 2024-04-23T16:25:47.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:274081cc:::benchmark_data_smithi188_89276_object77245:head# 2024-04-23T16:25:47.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27408f79:::benchmark_data_smithi188_89276_object81890:head# 2024-04-23T16:25:47.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:274299d3:::benchmark_data_smithi188_89276_object152248:head# 2024-04-23T16:25:47.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2742e378:::benchmark_data_smithi188_89276_object156530:head# 2024-04-23T16:25:47.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2743023c:::benchmark_data_smithi188_89276_object78923:head# 2024-04-23T16:25:47.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27430acf:::benchmark_data_smithi188_89276_object153863:head# 2024-04-23T16:25:47.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2743a0e4:::benchmark_data_smithi188_89276_object120787:head# 2024-04-23T16:25:47.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:274410b1:::benchmark_data_smithi188_89276_object108611:head# 2024-04-23T16:25:47.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:274431b4:::benchmark_data_smithi188_89276_object123549:head# 2024-04-23T16:25:47.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27447292:::benchmark_data_smithi188_89276_object161265:head# 2024-04-23T16:25:47.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27459462:::benchmark_data_smithi188_89276_object166298:head# 2024-04-23T16:25:47.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2745d028:::benchmark_data_smithi188_89276_object128155:head# 2024-04-23T16:25:47.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2745ec92:::benchmark_data_smithi188_89276_object119544:head# 2024-04-23T16:25:47.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27461deb:::benchmark_data_smithi188_89276_object136936:head# 2024-04-23T16:25:47.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2746be57:::benchmark_data_smithi188_89276_object113562:head# 2024-04-23T16:25:47.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2746d480:::benchmark_data_smithi188_89276_object100459:head# 2024-04-23T16:25:47.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27470edd:::benchmark_data_smithi188_89276_object61067:head# 2024-04-23T16:25:47.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27474237:::benchmark_data_smithi188_89276_object124332:head# 2024-04-23T16:25:47.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2747eb51:::benchmark_data_smithi188_89276_object102698:head# 2024-04-23T16:25:47.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:274991a7:::benchmark_data_smithi188_89276_object44358:head# 2024-04-23T16:25:47.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2749b0ff:::benchmark_data_smithi188_89276_object121204:head# 2024-04-23T16:25:47.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:274a2af1:::benchmark_data_smithi188_89276_object98980:head# 2024-04-23T16:25:47.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:274a60df:::benchmark_data_smithi188_89276_object60131:head# 2024-04-23T16:25:47.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:274b10d6:::benchmark_data_smithi188_89276_object82429:head# 2024-04-23T16:25:47.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:274b41df:::benchmark_data_smithi188_89276_object168469:head# 2024-04-23T16:25:47.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:274b9e4e:::benchmark_data_smithi188_89276_object41267:head# 2024-04-23T16:25:47.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:274bc9b8:::benchmark_data_smithi188_89276_object98588:head# 2024-04-23T16:25:47.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:274d61fa:::benchmark_data_smithi188_89276_object60991:head# 2024-04-23T16:25:47.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:274d7434:::benchmark_data_smithi188_89276_object150116:head# 2024-04-23T16:25:47.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:274da6ad:::benchmark_data_smithi188_89276_object137457:head# 2024-04-23T16:25:47.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:274efd3e:::benchmark_data_smithi188_89276_object123833:head# 2024-04-23T16:25:47.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27505944:::benchmark_data_smithi188_89276_object147351:head# 2024-04-23T16:25:47.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27505d0f:::benchmark_data_smithi188_89276_object61286:head# 2024-04-23T16:25:47.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:275148f8:::benchmark_data_smithi188_89276_object124059:head# 2024-04-23T16:25:47.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:275238cc:::benchmark_data_smithi188_89276_object128567:head# 2024-04-23T16:25:47.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27526407:::benchmark_data_smithi188_89276_object71125:head# 2024-04-23T16:25:47.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2752d158:::benchmark_data_smithi188_89276_object65798:head# 2024-04-23T16:25:47.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2752df97:::benchmark_data_smithi188_89276_object83741:head# 2024-04-23T16:25:47.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27530d92:::benchmark_data_smithi188_89276_object156524:head# 2024-04-23T16:25:47.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27531638:::benchmark_data_smithi188_89276_object132738:head# 2024-04-23T16:25:47.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27532eaa:::benchmark_data_smithi188_89276_object124452:head# 2024-04-23T16:25:47.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27537e00:::benchmark_data_smithi188_89276_object41810:head# 2024-04-23T16:25:47.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2753c7d3:::benchmark_data_smithi188_89276_object100322:head# 2024-04-23T16:25:47.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:275460b0:::benchmark_data_smithi188_89276_object67055:head# 2024-04-23T16:25:47.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2754679d:::benchmark_data_smithi188_89276_object82491:head# 2024-04-23T16:25:47.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2754c012:::benchmark_data_smithi188_89276_object105499:head# 2024-04-23T16:25:47.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27567bd9:::benchmark_data_smithi188_89276_object79497:head# 2024-04-23T16:25:47.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27573351:::benchmark_data_smithi188_89276_object87354:head# 2024-04-23T16:25:47.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2758436d:::benchmark_data_smithi188_89276_object66710:head# 2024-04-23T16:25:47.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:275a23cf:::benchmark_data_smithi188_89276_object55310:head# 2024-04-23T16:25:47.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:275bb89d:::benchmark_data_smithi188_89276_object125415:head# 2024-04-23T16:25:47.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:275bd294:::benchmark_data_smithi188_89276_object165487:head# 2024-04-23T16:25:47.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:275c31a3:::benchmark_data_smithi188_89276_object136517:head# 2024-04-23T16:25:47.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:275cb8fe:::benchmark_data_smithi188_89276_object97248:head# 2024-04-23T16:25:47.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:275d87ea:::benchmark_data_smithi188_89276_object161948:head# 2024-04-23T16:25:47.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:275d8bb6:::benchmark_data_smithi188_89276_object111066:head# 2024-04-23T16:25:47.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:275daf26:::benchmark_data_smithi188_89276_object147601:head# 2024-04-23T16:25:47.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:275df54c:::benchmark_data_smithi188_89276_object148996:head# 2024-04-23T16:25:47.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:275faa55:::benchmark_data_smithi188_89276_object156487:head# 2024-04-23T16:25:47.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2760a433:::benchmark_data_smithi188_89276_object50522:head# 2024-04-23T16:25:47.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:276125d4:::benchmark_data_smithi188_89276_object43777:head# 2024-04-23T16:25:47.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2762a795:::benchmark_data_smithi188_89276_object149227:head# 2024-04-23T16:25:47.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2762b880:::benchmark_data_smithi188_89276_object61456:head# 2024-04-23T16:25:47.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2762f919:::benchmark_data_smithi188_89276_object63258:head# 2024-04-23T16:25:47.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27632816:::benchmark_data_smithi188_89276_object110747:head# 2024-04-23T16:25:47.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27636af8:::benchmark_data_smithi188_89276_object49939:head# 2024-04-23T16:25:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27639ab6:::benchmark_data_smithi188_89276_object153135:head# 2024-04-23T16:25:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2763b774:::benchmark_data_smithi188_89276_object77018:head# 2024-04-23T16:25:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2764cf68:::benchmark_data_smithi188_89276_object151020:head# 2024-04-23T16:25:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2764fed1:::benchmark_data_smithi188_89276_object111576:head# 2024-04-23T16:25:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2765a491:::benchmark_data_smithi188_89276_object47139:head# 2024-04-23T16:25:47.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2765c04a:::benchmark_data_smithi188_89276_object79226:head# 2024-04-23T16:25:47.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2765f14c:::benchmark_data_smithi188_89276_object77169:head# 2024-04-23T16:25:47.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2765f940:::benchmark_data_smithi188_89276_object64783:head# 2024-04-23T16:25:47.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27662794:::benchmark_data_smithi188_89276_object126750:head# 2024-04-23T16:25:47.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:276751cb:::benchmark_data_smithi188_89276_object81029:head# 2024-04-23T16:25:47.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27682fba:::benchmark_data_smithi188_89276_object158231:head# 2024-04-23T16:25:47.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27688b70:::benchmark_data_smithi188_89276_object159173:head# 2024-04-23T16:25:47.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2768bc06:::benchmark_data_smithi188_89276_object67941:head# 2024-04-23T16:25:47.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27690ce6:::benchmark_data_smithi188_89276_object72573:head# 2024-04-23T16:25:47.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27691d4c:::benchmark_data_smithi188_89276_object65529:head# 2024-04-23T16:25:47.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27693aaa:::benchmark_data_smithi188_89276_object152435:head# 2024-04-23T16:25:47.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:276a7efc:::benchmark_data_smithi188_89276_object49433:head# 2024-04-23T16:25:47.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:276b0b56:::benchmark_data_smithi188_89276_object154931:head# 2024-04-23T16:25:47.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:276c140c:::benchmark_data_smithi188_89276_object41946:head# 2024-04-23T16:25:47.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:276d2184:::benchmark_data_smithi188_89276_object62476:head# 2024-04-23T16:25:47.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:276dc543:::benchmark_data_smithi188_89276_object54736:head# 2024-04-23T16:25:47.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:276e431f:::benchmark_data_smithi188_89276_object58292:head# 2024-04-23T16:25:47.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:276e66a6:::benchmark_data_smithi188_89276_object82440:head# 2024-04-23T16:25:47.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:276f8325:::benchmark_data_smithi188_89276_object122853:head# 2024-04-23T16:25:47.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:276fad3b:::benchmark_data_smithi188_89276_object138123:head# 2024-04-23T16:25:47.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27703fd7:::benchmark_data_smithi188_89276_object53067:head# 2024-04-23T16:25:47.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27708a81:::benchmark_data_smithi188_89276_object147028:head# 2024-04-23T16:25:47.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2770ac18:::benchmark_data_smithi188_89276_object116083:head# 2024-04-23T16:25:47.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2771327f:::benchmark_data_smithi188_89276_object161900:head# 2024-04-23T16:25:47.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2771a80e:::benchmark_data_smithi188_89276_object107252:head# 2024-04-23T16:25:47.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2772450f:::benchmark_data_smithi188_89276_object126030:head# 2024-04-23T16:25:47.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2774fe24:::benchmark_data_smithi188_89276_object82714:head# 2024-04-23T16:25:47.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27750df9:::benchmark_data_smithi188_89276_object117462:head# 2024-04-23T16:25:47.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27750e80:::benchmark_data_smithi188_89276_object151383:head# 2024-04-23T16:25:47.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27753de4:::benchmark_data_smithi188_89276_object100505:head# 2024-04-23T16:25:47.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2778b267:::benchmark_data_smithi188_89276_object91630:head# 2024-04-23T16:25:47.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27795e60:::benchmark_data_smithi188_89276_object83710:head# 2024-04-23T16:25:47.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2779c868:::benchmark_data_smithi188_89276_object123209:head# 2024-04-23T16:25:47.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2779d3b4:::benchmark_data_smithi188_89276_object106460:head# 2024-04-23T16:25:47.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:277a0f04:::benchmark_data_smithi188_89276_object113288:head# 2024-04-23T16:25:47.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:277a553d:::benchmark_data_smithi188_89276_object106667:head# 2024-04-23T16:25:47.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:277b4804:::benchmark_data_smithi188_89276_object132935:head# 2024-04-23T16:25:47.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:277b524b:::benchmark_data_smithi188_89276_object80247:head# 2024-04-23T16:25:47.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:277bb201:::benchmark_data_smithi188_89276_object66074:head# 2024-04-23T16:25:47.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:277c1560:::benchmark_data_smithi188_89276_object43566:head# 2024-04-23T16:25:47.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:277c3843:::benchmark_data_smithi188_89276_object66476:head# 2024-04-23T16:25:47.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.438 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:277cfa94:::benchmark_data_smithi188_89276_object142735:head# 2024-04-23T16:25:47.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.438 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:277d17ec:::benchmark_data_smithi188_89276_object125711:head# 2024-04-23T16:25:47.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.438 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:277d508b:::benchmark_data_smithi188_89276_object109691:head# 2024-04-23T16:25:47.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.438 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:277d79db:::benchmark_data_smithi188_89276_object104435:head# 2024-04-23T16:25:47.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:277dc765:::benchmark_data_smithi188_89276_object143461:head# 2024-04-23T16:25:47.439 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:277dcfd9:::benchmark_data_smithi188_89276_object135008:head# 2024-04-23T16:25:47.439 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:277e0db0:::benchmark_data_smithi188_89276_object154553:head# 2024-04-23T16:25:47.439 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:277e5652:::benchmark_data_smithi188_89276_object156520:head# 2024-04-23T16:25:47.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:277ed9d7:::benchmark_data_smithi188_89276_object101000:head# 2024-04-23T16:25:47.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:277ee475:::benchmark_data_smithi188_89276_object120264:head# 2024-04-23T16:25:47.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:277f0555:::benchmark_data_smithi188_89276_object78918:head# 2024-04-23T16:25:47.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:277fe1c9:::benchmark_data_smithi188_89276_object108686:head# 2024-04-23T16:25:47.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27813b61:::benchmark_data_smithi188_89276_object123851:head# 2024-04-23T16:25:47.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27823c43:::benchmark_data_smithi188_89276_object56598:head# 2024-04-23T16:25:47.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27831e6c:::benchmark_data_smithi188_89276_object42272:head# 2024-04-23T16:25:47.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:278341a3:::benchmark_data_smithi188_89276_object162496:head# 2024-04-23T16:25:47.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2783c558:::benchmark_data_smithi188_89276_object147049:head# 2024-04-23T16:25:47.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2783c9fa:::benchmark_data_smithi188_89276_object139307:head# 2024-04-23T16:25:47.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27840561:::benchmark_data_smithi188_89276_object67663:head# 2024-04-23T16:25:47.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27856a3e:::benchmark_data_smithi188_89276_object106386:head# 2024-04-23T16:25:47.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:278613d7:::benchmark_data_smithi188_89276_object163806:head# 2024-04-23T16:25:47.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2786c36f:::benchmark_data_smithi188_89276_object143252:head# 2024-04-23T16:25:47.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27875741:::benchmark_data_smithi188_89276_object156238:head# 2024-04-23T16:25:47.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2787ba2b:::benchmark_data_smithi188_89276_object56477:head# 2024-04-23T16:25:47.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2787fe45:::benchmark_data_smithi188_89276_object70101:head# 2024-04-23T16:25:47.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27884e3c:::benchmark_data_smithi188_89276_object121037:head# 2024-04-23T16:25:47.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2789be8a:::benchmark_data_smithi188_89276_object60548:head# 2024-04-23T16:25:47.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:278b84e9:::benchmark_data_smithi188_89276_object69556:head# 2024-04-23T16:25:47.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:278bc41b:::benchmark_data_smithi188_89276_object166866:head# 2024-04-23T16:25:47.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:278bd7b6:::benchmark_data_smithi188_89276_object110252:head# 2024-04-23T16:25:47.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:278be778:::benchmark_data_smithi188_89276_object72745:head# 2024-04-23T16:25:47.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:278c005d:::benchmark_data_smithi188_89276_object76830:head# 2024-04-23T16:25:47.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:278ce92c:::benchmark_data_smithi188_89276_object137460:head# 2024-04-23T16:25:47.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:278d5966:::benchmark_data_smithi188_89276_object59240:head# 2024-04-23T16:25:47.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:278d6dc3:::benchmark_data_smithi188_89276_object79251:head# 2024-04-23T16:25:47.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:278e85af:::benchmark_data_smithi188_89276_object84952:head# 2024-04-23T16:25:47.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:278f2948:::benchmark_data_smithi188_89276_object67825:head# 2024-04-23T16:25:47.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:278fd850:::benchmark_data_smithi188_89276_object99401:head# 2024-04-23T16:25:47.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27902576:::benchmark_data_smithi188_89276_object118867:head# 2024-04-23T16:25:47.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.447 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2791069e:::benchmark_data_smithi188_89276_object165041:head# 2024-04-23T16:25:47.447 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.447 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:279113e6:::benchmark_data_smithi188_89276_object120547:head# 2024-04-23T16:25:47.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:279129a5:::benchmark_data_smithi188_89276_object161653:head# 2024-04-23T16:25:47.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2791ae34:::benchmark_data_smithi188_89276_object48281:head# 2024-04-23T16:25:47.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2791e209:::benchmark_data_smithi188_89276_object109592:head# 2024-04-23T16:25:47.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27922e2f:::benchmark_data_smithi188_89276_object41913:head# 2024-04-23T16:25:47.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2794415d:::benchmark_data_smithi188_89276_object124033:head# 2024-04-23T16:25:47.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2794a75a:::benchmark_data_smithi188_89276_object88882:head# 2024-04-23T16:25:47.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2794d3e9:::benchmark_data_smithi188_89276_object76535:head# 2024-04-23T16:25:47.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2795a6f9:::benchmark_data_smithi188_89276_object54484:head# 2024-04-23T16:25:47.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27960d74:::benchmark_data_smithi188_89276_object44690:head# 2024-04-23T16:25:47.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27964d86:::benchmark_data_smithi188_89276_object44933:head# 2024-04-23T16:25:47.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27967f97:::benchmark_data_smithi188_89276_object89233:head# 2024-04-23T16:25:47.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2796efb0:::benchmark_data_smithi188_89276_object97280:head# 2024-04-23T16:25:47.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:279703a5:::benchmark_data_smithi188_89276_object58476:head# 2024-04-23T16:25:47.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27977cf2:::benchmark_data_smithi188_89276_object153772:head# 2024-04-23T16:25:47.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2797aa2d:::benchmark_data_smithi188_89276_object60413:head# 2024-04-23T16:25:47.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27981f8e:::benchmark_data_smithi188_89276_object102090:head# 2024-04-23T16:25:47.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:279827f2:::benchmark_data_smithi188_89276_object126630:head# 2024-04-23T16:25:47.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2799410d:::benchmark_data_smithi188_89276_object102626:head# 2024-04-23T16:25:47.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:279aaef4:::benchmark_data_smithi188_89276_object48643:head# 2024-04-23T16:25:47.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:279ade89:::benchmark_data_smithi188_89276_object72401:head# 2024-04-23T16:25:47.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:279d5cdb:::benchmark_data_smithi188_89276_object116056:head# 2024-04-23T16:25:47.451 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.451 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:279e858a:::benchmark_data_smithi188_89276_object49495:head# 2024-04-23T16:25:47.451 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.451 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:279ea117:::benchmark_data_smithi188_89276_object151548:head# 2024-04-23T16:25:47.451 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.452 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:279ec7c7:::benchmark_data_smithi188_89276_object79866:head# 2024-04-23T16:25:47.452 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.452 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:279ee4f2:::benchmark_data_smithi188_89276_object125901:head# 2024-04-23T16:25:47.452 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.452 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:279f4b6a:::benchmark_data_smithi188_89276_object143615:head# 2024-04-23T16:25:47.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27a017b4:::benchmark_data_smithi188_89276_object134194:head# 2024-04-23T16:25:47.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27a0bee1:::benchmark_data_smithi188_89276_object66586:head# 2024-04-23T16:25:47.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27a1476b:::benchmark_data_smithi188_89276_object106060:head# 2024-04-23T16:25:47.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27a2ba0e:::benchmark_data_smithi188_89276_object105681:head# 2024-04-23T16:25:47.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27a2c523:::benchmark_data_smithi188_89276_object103366:head# 2024-04-23T16:25:47.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27a3b7ab:::benchmark_data_smithi188_89276_object63082:head# 2024-04-23T16:25:47.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.454 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27a412fa:::benchmark_data_smithi188_89276_object151326:head# 2024-04-23T16:25:47.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27a4d7b1:::benchmark_data_smithi188_89276_object51310:head# 2024-04-23T16:25:47.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27a51680:::benchmark_data_smithi188_89276_object145800:head# 2024-04-23T16:25:47.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27a5311b:::benchmark_data_smithi188_89276_object118990:head# 2024-04-23T16:25:47.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27a57a0a:::benchmark_data_smithi188_89276_object167281:head# 2024-04-23T16:25:47.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27a6cb09:::benchmark_data_smithi188_89276_object94919:head# 2024-04-23T16:25:47.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27a8726c:::benchmark_data_smithi188_89276_object85794:head# 2024-04-23T16:25:47.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27a90fa5:::benchmark_data_smithi188_89276_object61010:head# 2024-04-23T16:25:47.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.457 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27a94063:::benchmark_data_smithi188_89276_object131119:head# 2024-04-23T16:25:47.457 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27a9dfe8:::benchmark_data_smithi188_89276_object107626:head# 2024-04-23T16:25:47.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ab19fc:::benchmark_data_smithi188_89276_object72221:head# 2024-04-23T16:25:47.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ab6c3a:::benchmark_data_smithi188_89276_object63573:head# 2024-04-23T16:25:47.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ab97ec:::benchmark_data_smithi188_89276_object111795:head# 2024-04-23T16:25:47.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ac4b38:::benchmark_data_smithi188_89276_object98817:head# 2024-04-23T16:25:47.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ac6e46:::benchmark_data_smithi188_89276_object61589:head# 2024-04-23T16:25:47.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ad0632:::benchmark_data_smithi188_89276_object102574:head# 2024-04-23T16:25:47.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27addc1f:::benchmark_data_smithi188_89276_object49101:head# 2024-04-23T16:25:47.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.460 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27adfcdf:::benchmark_data_smithi188_89276_object42849:head# 2024-04-23T16:25:47.460 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.460 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ae0cbc:::benchmark_data_smithi188_89276_object48559:head# 2024-04-23T16:25:47.460 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.460 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27afe982:::benchmark_data_smithi188_89276_object160059:head# 2024-04-23T16:25:47.461 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.461 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27affa06:::benchmark_data_smithi188_89276_object135099:head# 2024-04-23T16:25:47.461 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.461 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b00d1d:::benchmark_data_smithi188_89276_object82369:head# 2024-04-23T16:25:47.461 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.461 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b03625:::benchmark_data_smithi188_89276_object160181:head# 2024-04-23T16:25:47.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.464 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b040e1:::benchmark_data_smithi188_89276_object93722:head# 2024-04-23T16:25:47.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b08a76:::benchmark_data_smithi188_89276_object163583:head# 2024-04-23T16:25:47.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b09131:::benchmark_data_smithi188_89276_object168047:head# 2024-04-23T16:25:47.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b09ddb:::benchmark_data_smithi188_89276_object77243:head# 2024-04-23T16:25:47.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b14296:::benchmark_data_smithi188_89276_object42225:head# 2024-04-23T16:25:47.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b16829:::benchmark_data_smithi188_89276_object59566:head# 2024-04-23T16:25:47.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b27a86:::benchmark_data_smithi188_89276_object103920:head# 2024-04-23T16:25:47.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b2bcc2:::benchmark_data_smithi188_89276_object77984:head# 2024-04-23T16:25:47.466 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.466 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b2e846:::benchmark_data_smithi188_89276_object82904:head# 2024-04-23T16:25:47.466 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.466 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b2f80c:::benchmark_data_smithi188_89276_object103525:head# 2024-04-23T16:25:47.466 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.466 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b329a3:::benchmark_data_smithi188_89276_object156358:head# 2024-04-23T16:25:47.466 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.467 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b46676:::benchmark_data_smithi188_89276_object66874:head# 2024-04-23T16:25:47.467 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.470 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b49386:::benchmark_data_smithi188_89276_object67701:head# 2024-04-23T16:25:47.470 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b4a9c2:::benchmark_data_smithi188_89276_object112041:head# 2024-04-23T16:25:47.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b4fe0c:::benchmark_data_smithi188_89276_object163404:head# 2024-04-23T16:25:47.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b53498:::benchmark_data_smithi188_89276_object81387:head# 2024-04-23T16:25:47.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b6335c:::benchmark_data_smithi188_89276_object143390:head# 2024-04-23T16:25:47.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b6bd82:::benchmark_data_smithi188_89276_object161847:head# 2024-04-23T16:25:47.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b75bdc:::benchmark_data_smithi188_89276_object105076:head# 2024-04-23T16:25:47.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b78817:::benchmark_data_smithi188_89276_object103694:head# 2024-04-23T16:25:47.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b93115:::benchmark_data_smithi188_89276_object123306:head# 2024-04-23T16:25:47.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27b9372d:::benchmark_data_smithi188_89276_object150726:head# 2024-04-23T16:25:47.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ba3082:::benchmark_data_smithi188_89276_object44744:head# 2024-04-23T16:25:47.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ba7930:::benchmark_data_smithi188_89276_object134184:head# 2024-04-23T16:25:47.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ba8b19:::benchmark_data_smithi188_89276_object49597:head# 2024-04-23T16:25:47.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27bade21:::benchmark_data_smithi188_89276_object64852:head# 2024-04-23T16:25:47.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27bb26f5:::benchmark_data_smithi188_89276_object113818:head# 2024-04-23T16:25:47.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27bb3b45:::benchmark_data_smithi188_89276_object167429:head# 2024-04-23T16:25:47.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27bb5c46:::benchmark_data_smithi188_89276_object114548:head# 2024-04-23T16:25:47.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27bb77ff:::benchmark_data_smithi188_89276_object149847:head# 2024-04-23T16:25:47.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27bbaa03:::benchmark_data_smithi188_89276_object86179:head# 2024-04-23T16:25:47.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27bbb00b:::benchmark_data_smithi188_89276_object122127:head# 2024-04-23T16:25:47.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27bbc78b:::benchmark_data_smithi188_89276_object61990:head# 2024-04-23T16:25:47.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27bcedc8:::benchmark_data_smithi188_89276_object45802:head# 2024-04-23T16:25:47.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27bd75ae:::benchmark_data_smithi188_89276_object75947:head# 2024-04-23T16:25:47.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27bd993d:::benchmark_data_smithi188_89276_object159333:head# 2024-04-23T16:25:47.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.478 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27be1957:::benchmark_data_smithi188_89276_object106562:head# 2024-04-23T16:25:47.478 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.478 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27bfe168:::benchmark_data_smithi188_89276_object149264:head# 2024-04-23T16:25:47.479 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c010f5:::benchmark_data_smithi188_89276_object145891:head# 2024-04-23T16:25:47.479 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c04cd2:::benchmark_data_smithi188_89276_object129381:head# 2024-04-23T16:25:47.479 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c1b6d8:::benchmark_data_smithi188_89276_object54007:head# 2024-04-23T16:25:47.479 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c2248e:::benchmark_data_smithi188_89276_object145609:head# 2024-04-23T16:25:47.479 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c30a5b:::benchmark_data_smithi188_89276_object71142:head# 2024-04-23T16:25:47.479 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c3e1c3:::benchmark_data_smithi188_89276_object139308:head# 2024-04-23T16:25:47.479 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c3eec4:::benchmark_data_smithi188_89276_object89733:head# 2024-04-23T16:25:47.480 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.491 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c437d8:::benchmark_data_smithi188_89276_object80129:head# 2024-04-23T16:25:47.491 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.491 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c44f32:::benchmark_data_smithi188_89276_object124459:head# 2024-04-23T16:25:47.491 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.491 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c47573:::benchmark_data_smithi188_89276_object100263:head# 2024-04-23T16:25:47.491 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.491 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c4b95a:::benchmark_data_smithi188_89276_object89254:head# 2024-04-23T16:25:47.491 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c700fd:::benchmark_data_smithi188_89276_object67341:head# 2024-04-23T16:25:47.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c7544c:::benchmark_data_smithi188_89276_object101376:head# 2024-04-23T16:25:47.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c7ce5c:::benchmark_data_smithi188_89276_object43287:head# 2024-04-23T16:25:47.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c7d33d:::benchmark_data_smithi188_89276_object101673:head# 2024-04-23T16:25:47.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c831e5:::benchmark_data_smithi188_89276_object119486:head# 2024-04-23T16:25:47.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c8c9a4:::benchmark_data_smithi188_89276_object48068:head# 2024-04-23T16:25:47.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c91cf4:::benchmark_data_smithi188_89276_object148964:head# 2024-04-23T16:25:47.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c95af5:::benchmark_data_smithi188_89276_object77737:head# 2024-04-23T16:25:47.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c9859c:::benchmark_data_smithi188_89276_object88048:head# 2024-04-23T16:25:47.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27c995fe:::benchmark_data_smithi188_89276_object115193:head# 2024-04-23T16:25:47.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ca1fd9:::benchmark_data_smithi188_89276_object50251:head# 2024-04-23T16:25:47.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ca6040:::benchmark_data_smithi188_89276_object161162:head# 2024-04-23T16:25:47.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27cb1cf1:::benchmark_data_smithi188_89276_object168482:head# 2024-04-23T16:25:47.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27cdbc68:::benchmark_data_smithi188_89276_object51636:head# 2024-04-23T16:25:47.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27cdd3fd:::benchmark_data_smithi188_89276_object104170:head# 2024-04-23T16:25:47.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27cf19a7:::benchmark_data_smithi188_89276_object63037:head# 2024-04-23T16:25:47.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27cf9e93:::benchmark_data_smithi188_89276_object102174:head# 2024-04-23T16:25:47.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27cfb19e:::benchmark_data_smithi188_89276_object140923:head# 2024-04-23T16:25:47.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27d0b347:::benchmark_data_smithi188_89276_object50186:head# 2024-04-23T16:25:47.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27d0bee6:::benchmark_data_smithi188_89276_object94757:head# 2024-04-23T16:25:47.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27d1db69:::benchmark_data_smithi188_89276_object48918:head# 2024-04-23T16:25:47.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27d1f676:::benchmark_data_smithi188_89276_object137241:head# 2024-04-23T16:25:47.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27d294d6:::benchmark_data_smithi188_89276_object92268:head# 2024-04-23T16:25:47.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27d3ed2f:::benchmark_data_smithi188_89276_object105224:head# 2024-04-23T16:25:47.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27d406bc:::benchmark_data_smithi188_89276_object110241:head# 2024-04-23T16:25:47.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27d4eba6:::benchmark_data_smithi188_89276_object72348:head# 2024-04-23T16:25:47.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27d59d86:::benchmark_data_smithi188_89276_object123367:head# 2024-04-23T16:25:47.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27d5d49e:::benchmark_data_smithi188_89276_object146054:head# 2024-04-23T16:25:47.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27d63d0f:::benchmark_data_smithi188_89276_object114598:head# 2024-04-23T16:25:47.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27d67ff6:::benchmark_data_smithi188_89276_object133655:head# 2024-04-23T16:25:47.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.500 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27d70e03:::benchmark_data_smithi188_89276_object69065:head# 2024-04-23T16:25:47.500 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.500 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27d78de4:::benchmark_data_smithi188_89276_object117887:head# 2024-04-23T16:25:47.500 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.500 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27d8b257:::benchmark_data_smithi188_89276_object139631:head# 2024-04-23T16:25:47.500 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.500 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27d94c97:::benchmark_data_smithi188_89276_object107374:head# 2024-04-23T16:25:47.500 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.500 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27d97dc3:::benchmark_data_smithi188_89276_object103789:head# 2024-04-23T16:25:47.500 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.500 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27d9ceaa:::benchmark_data_smithi188_89276_object134815:head# 2024-04-23T16:25:47.500 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.501 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27d9ddc8:::benchmark_data_smithi188_89276_object91406:head# 2024-04-23T16:25:47.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.502 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27da496a:::benchmark_data_smithi188_89276_object82323:head# 2024-04-23T16:25:47.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.502 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27db7f9e:::benchmark_data_smithi188_89276_object154337:head# 2024-04-23T16:25:47.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.502 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27dbd1c5:::benchmark_data_smithi188_89276_object79272:head# 2024-04-23T16:25:47.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.502 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27dc1dcc:::benchmark_data_smithi188_89276_object76971:head# 2024-04-23T16:25:47.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.502 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27dcc1fd:::benchmark_data_smithi188_89276_object153632:head# 2024-04-23T16:25:47.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.502 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27dd4091:::benchmark_data_smithi188_89276_object109777:head# 2024-04-23T16:25:47.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27dda83e:::benchmark_data_smithi188_89276_object168083:head# 2024-04-23T16:25:47.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27dec108:::benchmark_data_smithi188_89276_object66666:head# 2024-04-23T16:25:47.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27df0bbd:::benchmark_data_smithi188_89276_object107559:head# 2024-04-23T16:25:47.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27df3769:::benchmark_data_smithi188_89276_object55512:head# 2024-04-23T16:25:47.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27df5817:::benchmark_data_smithi188_89276_object45630:head# 2024-04-23T16:25:47.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27df66f8:::benchmark_data_smithi188_89276_object135945:head# 2024-04-23T16:25:47.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27dfd196:::benchmark_data_smithi188_89276_object139559:head# 2024-04-23T16:25:47.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27e0097a:::benchmark_data_smithi188_89276_object41593:head# 2024-04-23T16:25:47.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27e038f8:::benchmark_data_smithi188_89276_object43040:head# 2024-04-23T16:25:47.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27e09775:::benchmark_data_smithi188_89276_object104741:head# 2024-04-23T16:25:47.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27e0e159:::benchmark_data_smithi188_89276_object163115:head# 2024-04-23T16:25:47.507 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27e0eb36:::benchmark_data_smithi188_89276_object55405:head# 2024-04-23T16:25:47.507 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27e25b46:::benchmark_data_smithi188_89276_object56479:head# 2024-04-23T16:25:47.507 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27e2c2de:::benchmark_data_smithi188_89276_object55151:head# 2024-04-23T16:25:47.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27e39a07:::benchmark_data_smithi188_89276_object142364:head# 2024-04-23T16:25:47.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27e3ce2a:::benchmark_data_smithi188_89276_object120369:head# 2024-04-23T16:25:47.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27e424b4:::benchmark_data_smithi188_89276_object74305:head# 2024-04-23T16:25:47.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27e5fe20:::benchmark_data_smithi188_89276_object159706:head# 2024-04-23T16:25:47.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27e622b5:::benchmark_data_smithi188_89276_object150314:head# 2024-04-23T16:25:47.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27e6b152:::benchmark_data_smithi188_89276_object140960:head# 2024-04-23T16:25:47.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27e75c30:::benchmark_data_smithi188_89276_object147106:head# 2024-04-23T16:25:47.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27e7b52c:::benchmark_data_smithi188_89276_object137453:head# 2024-04-23T16:25:47.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27e94d11:::benchmark_data_smithi188_89276_object82901:head# 2024-04-23T16:25:47.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ea9758:::benchmark_data_smithi188_89276_object116199:head# 2024-04-23T16:25:47.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27eb1bd1:::benchmark_data_smithi188_89276_object155767:head# 2024-04-23T16:25:47.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27eb419e:::benchmark_data_smithi188_89276_object93827:head# 2024-04-23T16:25:47.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ec1e57:::benchmark_data_smithi188_89276_object140917:head# 2024-04-23T16:25:47.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ec2a84:::benchmark_data_smithi188_89276_object124986:head# 2024-04-23T16:25:47.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.510 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ecdd8c:::benchmark_data_smithi188_89276_object49502:head# 2024-04-23T16:25:47.510 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.510 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ed1069:::benchmark_data_smithi188_89276_object94511:head# 2024-04-23T16:25:47.510 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.511 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ed4007:::benchmark_data_smithi188_89276_object117855:head# 2024-04-23T16:25:47.511 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.511 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ed6448:::benchmark_data_smithi188_89276_object46265:head# 2024-04-23T16:25:47.511 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ed66cc:::benchmark_data_smithi188_89276_object66683:head# 2024-04-23T16:25:47.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ed7cf7:::benchmark_data_smithi188_89276_object156965:head# 2024-04-23T16:25:47.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27eeb19f:::benchmark_data_smithi188_89276_object136664:head# 2024-04-23T16:25:47.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ef620d:::benchmark_data_smithi188_89276_object156835:head# 2024-04-23T16:25:47.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ef6e2b:::benchmark_data_smithi188_89276_object141764:head# 2024-04-23T16:25:47.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27efa2cd:::benchmark_data_smithi188_89276_object118901:head# 2024-04-23T16:25:47.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f06157:::benchmark_data_smithi188_89276_object145600:head# 2024-04-23T16:25:47.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f19b40:::benchmark_data_smithi188_89276_object72472:head# 2024-04-23T16:25:47.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f1d3de:::benchmark_data_smithi188_89276_object119599:head# 2024-04-23T16:25:47.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f21643:::benchmark_data_smithi188_89276_object41163:head# 2024-04-23T16:25:47.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f270ef:::benchmark_data_smithi188_89276_object138233:head# 2024-04-23T16:25:47.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f2a71c:::benchmark_data_smithi188_89276_object64745:head# 2024-04-23T16:25:47.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f2df37:::benchmark_data_smithi188_89276_object48530:head# 2024-04-23T16:25:47.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f42f4d:::benchmark_data_smithi188_89276_object101268:head# 2024-04-23T16:25:47.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f44439:::benchmark_data_smithi188_89276_object56559:head# 2024-04-23T16:25:47.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f4b104:::benchmark_data_smithi188_89276_object63839:head# 2024-04-23T16:25:47.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f599a6:::benchmark_data_smithi188_89276_object73734:head# 2024-04-23T16:25:47.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f634ed:::benchmark_data_smithi188_89276_object144827:head# 2024-04-23T16:25:47.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f635e5:::benchmark_data_smithi188_89276_object110555:head# 2024-04-23T16:25:47.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f63c8d:::benchmark_data_smithi188_89276_object71726:head# 2024-04-23T16:25:47.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f762c0:::benchmark_data_smithi188_89276_object47685:head# 2024-04-23T16:25:47.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f78472:::benchmark_data_smithi188_89276_object72451:head# 2024-04-23T16:25:47.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f811e4:::benchmark_data_smithi188_89276_object65715:head# 2024-04-23T16:25:47.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f85dfc:::benchmark_data_smithi188_89276_object122307:head# 2024-04-23T16:25:47.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f8af40:::benchmark_data_smithi188_89276_object75952:head# 2024-04-23T16:25:47.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f96b9d:::benchmark_data_smithi188_89276_object162702:head# 2024-04-23T16:25:47.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f9c1dc:::benchmark_data_smithi188_89276_object70939:head# 2024-04-23T16:25:47.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27f9f1ab:::benchmark_data_smithi188_89276_object59041:head# 2024-04-23T16:25:47.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27fa1ca8:::benchmark_data_smithi188_89276_object142992:head# 2024-04-23T16:25:47.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27fa5df9:::benchmark_data_smithi188_89276_object157405:head# 2024-04-23T16:25:47.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27fa81b1:::benchmark_data_smithi188_89276_object63682:head# 2024-04-23T16:25:47.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27fae407:::benchmark_data_smithi188_89276_object129828:head# 2024-04-23T16:25:47.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27fc373b:::benchmark_data_smithi188_89276_object116000:head# 2024-04-23T16:25:47.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.524 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27fe5188:::benchmark_data_smithi188_89276_object105530:head# 2024-04-23T16:25:47.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.524 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27fe8303:::benchmark_data_smithi188_89276_object121672:head# 2024-04-23T16:25:47.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.524 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27feb363:::benchmark_data_smithi188_89276_object118764:head# 2024-04-23T16:25:47.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.524 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27feddb1:::benchmark_data_smithi188_89276_object79583:head# 2024-04-23T16:25:47.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.524 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:27ff24e3:::benchmark_data_smithi188_89276_object133963:head# 2024-04-23T16:25:47.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.524 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:280015a6:::benchmark_data_smithi188_89276_object72304:head# 2024-04-23T16:25:47.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.524 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2800a90c:::benchmark_data_smithi188_89276_object97905:head# 2024-04-23T16:25:47.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.526 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2801906e:::benchmark_data_smithi188_89276_object161582:head# 2024-04-23T16:25:47.526 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.526 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28019363:::benchmark_data_smithi188_89276_object119093:head# 2024-04-23T16:25:47.527 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.527 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2801f196:::benchmark_data_smithi188_89276_object117513:head# 2024-04-23T16:25:47.527 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.527 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28054dab:::benchmark_data_smithi188_89276_object129437:head# 2024-04-23T16:25:47.527 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.527 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2805713a:::benchmark_data_smithi188_89276_object81918:head# 2024-04-23T16:25:47.527 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.527 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2805e180:::benchmark_data_smithi188_89276_object96081:head# 2024-04-23T16:25:47.527 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.527 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2805e9cc:::benchmark_data_smithi188_89276_object163247:head# 2024-04-23T16:25:47.527 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.527 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2805ff84:::benchmark_data_smithi188_89276_object115467:head# 2024-04-23T16:25:47.527 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.527 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28066bbf:::benchmark_data_smithi188_89276_object140469:head# 2024-04-23T16:25:47.527 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.528 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2806953a:::benchmark_data_smithi188_89276_object65486:head# 2024-04-23T16:25:47.528 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.528 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2806980b:::benchmark_data_smithi188_89276_object109162:head# 2024-04-23T16:25:47.528 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.528 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28074aa4:::benchmark_data_smithi188_89276_object104612:head# 2024-04-23T16:25:47.528 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.528 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:280763a1:::benchmark_data_smithi188_89276_object146158:head# 2024-04-23T16:25:47.528 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.529 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:280764b7:::benchmark_data_smithi188_89276_object45898:head# 2024-04-23T16:25:47.529 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.529 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:280774f7:::benchmark_data_smithi188_89276_object72288:head# 2024-04-23T16:25:47.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28077586:::benchmark_data_smithi188_89276_object86669:head# 2024-04-23T16:25:47.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28088ac5:::benchmark_data_smithi188_89276_object137183:head# 2024-04-23T16:25:47.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28090522:::benchmark_data_smithi188_89276_object137328:head# 2024-04-23T16:25:47.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2809d180:::benchmark_data_smithi188_89276_object117239:head# 2024-04-23T16:25:47.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2809ea1f:::benchmark_data_smithi188_89276_object161679:head# 2024-04-23T16:25:47.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:280a6786:::benchmark_data_smithi188_89276_object88530:head# 2024-04-23T16:25:47.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:280a8ae3:::benchmark_data_smithi188_89276_object128235:head# 2024-04-23T16:25:47.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:280b374d:::benchmark_data_smithi188_89276_object80779:head# 2024-04-23T16:25:47.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:280b9040:::benchmark_data_smithi188_89276_object132836:head# 2024-04-23T16:25:47.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.532 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:280bd725:::benchmark_data_smithi188_89276_object166082:head# 2024-04-23T16:25:47.532 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.532 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:280cac3b:::benchmark_data_smithi188_89276_object132622:head# 2024-04-23T16:25:47.532 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.533 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:280cc4f1:::benchmark_data_smithi188_89276_object107470:head# 2024-04-23T16:25:47.533 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.533 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:280d3fc6:::benchmark_data_smithi188_89276_object101867:head# 2024-04-23T16:25:47.533 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.533 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:280d5d0b:::benchmark_data_smithi188_89276_object62921:head# 2024-04-23T16:25:47.533 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.533 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:280dc3f2:::benchmark_data_smithi188_89276_object85508:head# 2024-04-23T16:25:47.533 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.533 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:280e6ee9:::benchmark_data_smithi188_89276_object124531:head# 2024-04-23T16:25:47.533 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.533 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:280eace5:::benchmark_data_smithi188_89276_object162258:head# 2024-04-23T16:25:47.533 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.533 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:280ed85c:::benchmark_data_smithi188_89276_object104829:head# 2024-04-23T16:25:47.533 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.534 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:280effd1:::benchmark_data_smithi188_89276_object58795:head# 2024-04-23T16:25:47.534 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.534 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:280f57c6:::benchmark_data_smithi188_89276_object54009:head# 2024-04-23T16:25:47.534 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.535 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28102c08:::benchmark_data_smithi188_89276_object59974:head# 2024-04-23T16:25:47.535 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.535 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2810348a:::benchmark_data_smithi188_89276_object120631:head# 2024-04-23T16:25:47.535 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.535 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28103941:::benchmark_data_smithi188_89276_object103163:head# 2024-04-23T16:25:47.535 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.535 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2810452c:::benchmark_data_smithi188_89276_object133058:head# 2024-04-23T16:25:47.535 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.535 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28110f58:::benchmark_data_smithi188_89276_object137399:head# 2024-04-23T16:25:47.536 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.536 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28118d61:::benchmark_data_smithi188_89276_object130094:head# 2024-04-23T16:25:47.536 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.536 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28125d3a:::benchmark_data_smithi188_89276_object111557:head# 2024-04-23T16:25:47.536 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.536 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2812f8d2:::benchmark_data_smithi188_89276_object119703:head# 2024-04-23T16:25:47.536 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.536 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28151d71:::benchmark_data_smithi188_89276_object124131:head# 2024-04-23T16:25:47.536 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.537 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28165c56:::benchmark_data_smithi188_89276_object165069:head# 2024-04-23T16:25:47.537 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.537 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2816efa6:::benchmark_data_smithi188_89276_object160777:head# 2024-04-23T16:25:47.537 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.537 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2817c75a:::benchmark_data_smithi188_89276_object87023:head# 2024-04-23T16:25:47.537 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.537 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2818b63b:::benchmark_data_smithi188_89276_object126276:head# 2024-04-23T16:25:47.537 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.538 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28194d43:::benchmark_data_smithi188_89276_object135332:head# 2024-04-23T16:25:47.538 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.538 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2819970a:::benchmark_data_smithi188_89276_object157849:head# 2024-04-23T16:25:47.538 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.538 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:281a4467:::benchmark_data_smithi188_89276_object78490:head# 2024-04-23T16:25:47.538 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.541 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:281a621d:::benchmark_data_smithi188_89276_object112765:head# 2024-04-23T16:25:47.541 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.541 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:281a81f1:::benchmark_data_smithi188_89276_object141299:head# 2024-04-23T16:25:47.541 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.542 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:281c1768:::benchmark_data_smithi188_89276_object93158:head# 2024-04-23T16:25:47.542 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.543 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:281c65d1:::benchmark_data_smithi188_89276_object151477:head# 2024-04-23T16:25:47.543 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.543 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:281c79fd:::benchmark_data_smithi188_89276_object49504:head# 2024-04-23T16:25:47.543 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.545 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:281c9481:::benchmark_data_smithi188_89276_object64613:head# 2024-04-23T16:25:47.545 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.546 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:281ca78b:::benchmark_data_smithi188_89276_object133837:head# 2024-04-23T16:25:47.546 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.550 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:281cb04b:::benchmark_data_smithi188_89276_object153528:head# 2024-04-23T16:25:47.550 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.550 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:281d6bf8:::benchmark_data_smithi188_89276_object79955:head# 2024-04-23T16:25:47.550 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:281dcc1d:::benchmark_data_smithi188_89276_object97527:head# 2024-04-23T16:25:47.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:281f9749:::benchmark_data_smithi188_89276_object43701:head# 2024-04-23T16:25:47.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:281fb7a1:::benchmark_data_smithi188_89276_object86433:head# 2024-04-23T16:25:47.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28204422:::benchmark_data_smithi188_89276_object75536:head# 2024-04-23T16:25:47.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2820448c:::benchmark_data_smithi188_89276_object168246:head# 2024-04-23T16:25:47.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:282055a3:::benchmark_data_smithi188_89276_object92050:head# 2024-04-23T16:25:47.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:282069c4:::benchmark_data_smithi188_89276_object46658:head# 2024-04-23T16:25:47.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2820883e:::benchmark_data_smithi188_89276_object134084:head# 2024-04-23T16:25:47.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.556 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2821a783:::benchmark_data_smithi188_89276_object114310:head# 2024-04-23T16:25:47.556 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.556 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2821ed60:::benchmark_data_smithi188_89276_object105529:head# 2024-04-23T16:25:47.556 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.556 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2822161c:::benchmark_data_smithi188_89276_object139991:head# 2024-04-23T16:25:47.557 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.557 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28227868:::benchmark_data_smithi188_89276_object138981:head# 2024-04-23T16:25:47.557 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.557 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2823ac7a:::benchmark_data_smithi188_89276_object82011:head# 2024-04-23T16:25:47.557 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.557 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2823c428:::benchmark_data_smithi188_89276_object57018:head# 2024-04-23T16:25:47.557 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.557 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28241251:::benchmark_data_smithi188_89276_object129051:head# 2024-04-23T16:25:47.557 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28250f5a:::benchmark_data_smithi188_89276_object96284:head# 2024-04-23T16:25:47.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2825399d:::benchmark_data_smithi188_89276_object50394:head# 2024-04-23T16:25:47.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2825aee3:::benchmark_data_smithi188_89276_object116361:head# 2024-04-23T16:25:47.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2827aedc:::benchmark_data_smithi188_89276_object167076:head# 2024-04-23T16:25:47.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28283f7c:::benchmark_data_smithi188_89276_object50912:head# 2024-04-23T16:25:47.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2828938f:::benchmark_data_smithi188_89276_object71832:head# 2024-04-23T16:25:47.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.560 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2828a472:::benchmark_data_smithi188_89276_object69258:head# 2024-04-23T16:25:47.560 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.560 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2828c75c:::benchmark_data_smithi188_89276_object155939:head# 2024-04-23T16:25:47.560 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.560 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:282a941d:::benchmark_data_smithi188_89276_object150859:head# 2024-04-23T16:25:47.560 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.560 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:282a9b24:::benchmark_data_smithi188_89276_object145988:head# 2024-04-23T16:25:47.560 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:282b2801:::benchmark_data_smithi188_89276_object104479:head# 2024-04-23T16:25:47.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:282ba886:::benchmark_data_smithi188_89276_object44879:head# 2024-04-23T16:25:47.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:282c0168:::benchmark_data_smithi188_89276_object148362:head# 2024-04-23T16:25:47.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:282c9ada:::benchmark_data_smithi188_89276_object149701:head# 2024-04-23T16:25:47.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:282d86a4:::benchmark_data_smithi188_89276_object49793:head# 2024-04-23T16:25:47.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.566 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:282e0dc4:::benchmark_data_smithi188_89276_object66026:head# 2024-04-23T16:25:47.566 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.566 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:282e1291:::benchmark_data_smithi188_89276_object167885:head# 2024-04-23T16:25:47.566 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.566 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:282ee89f:::benchmark_data_smithi188_89276_object110971:head# 2024-04-23T16:25:47.566 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28306afa:::benchmark_data_smithi188_89276_object140397:head# 2024-04-23T16:25:47.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2830e14e:::benchmark_data_smithi188_89276_object51681:head# 2024-04-23T16:25:47.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28312f09:::benchmark_data_smithi188_89276_object75658:head# 2024-04-23T16:25:47.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2831be28:::benchmark_data_smithi188_89276_object50807:head# 2024-04-23T16:25:47.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28333113:::benchmark_data_smithi188_89276_object147380:head# 2024-04-23T16:25:47.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:283347a0:::benchmark_data_smithi188_89276_object78938:head# 2024-04-23T16:25:47.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28337d28:::benchmark_data_smithi188_89276_object127198:head# 2024-04-23T16:25:47.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28356760:::benchmark_data_smithi188_89276_object143267:head# 2024-04-23T16:25:47.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.568 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28368996:::benchmark_data_smithi188_89276_object45677:head# 2024-04-23T16:25:47.568 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.568 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28379e2c:::benchmark_data_smithi188_89276_object42681:head# 2024-04-23T16:25:47.568 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.568 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2837f3a3:::benchmark_data_smithi188_89276_object110263:head# 2024-04-23T16:25:47.568 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.568 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28380437:::benchmark_data_smithi188_89276_object82607:head# 2024-04-23T16:25:47.568 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.568 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28385935:::benchmark_data_smithi188_89276_object74850:head# 2024-04-23T16:25:47.568 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.568 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2838d4ea:::benchmark_data_smithi188_89276_object141279:head# 2024-04-23T16:25:47.568 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.568 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2838e39e:::benchmark_data_smithi188_89276_object144428:head# 2024-04-23T16:25:47.568 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:283973c3:::benchmark_data_smithi188_89276_object107674:head# 2024-04-23T16:25:47.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2839cc89:::benchmark_data_smithi188_89276_object111409:head# 2024-04-23T16:25:47.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:283b3b9d:::benchmark_data_smithi188_89276_object101372:head# 2024-04-23T16:25:47.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:283b6436:::benchmark_data_smithi188_89276_object125174:head# 2024-04-23T16:25:47.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:283c6c26:::benchmark_data_smithi188_89276_object113211:head# 2024-04-23T16:25:47.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:283cb358:::benchmark_data_smithi188_89276_object123996:head# 2024-04-23T16:25:47.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:283ce86f:::benchmark_data_smithi188_89276_object150447:head# 2024-04-23T16:25:47.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:283ce9ef:::benchmark_data_smithi188_89276_object100168:head# 2024-04-23T16:25:47.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:283d9a67:::benchmark_data_smithi188_89276_object123867:head# 2024-04-23T16:25:47.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:283da2f3:::benchmark_data_smithi188_89276_object85142:head# 2024-04-23T16:25:47.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:283df143:::benchmark_data_smithi188_89276_object97595:head# 2024-04-23T16:25:47.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:283e60ad:::benchmark_data_smithi188_89276_object135619:head# 2024-04-23T16:25:47.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:283ec6ff:::benchmark_data_smithi188_89276_object79255:head# 2024-04-23T16:25:47.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:283fb9d5:::benchmark_data_smithi188_89276_object134812:head# 2024-04-23T16:25:47.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28401aab:::benchmark_data_smithi188_89276_object74125:head# 2024-04-23T16:25:47.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2841234b:::benchmark_data_smithi188_89276_object95549:head# 2024-04-23T16:25:47.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2841d577:::benchmark_data_smithi188_89276_object125932:head# 2024-04-23T16:25:47.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28443f7e:::benchmark_data_smithi188_89276_object134198:head# 2024-04-23T16:25:47.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:284440a3:::benchmark_data_smithi188_89276_object45151:head# 2024-04-23T16:25:47.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:284504a1:::benchmark_data_smithi188_89276_object117336:head# 2024-04-23T16:25:47.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28453712:::benchmark_data_smithi188_89276_object118350:head# 2024-04-23T16:25:47.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28455281:::benchmark_data_smithi188_89276_object150861:head# 2024-04-23T16:25:47.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2845c129:::benchmark_data_smithi188_89276_object54042:head# 2024-04-23T16:25:47.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28460aec:::benchmark_data_smithi188_89276_object114341:head# 2024-04-23T16:25:47.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28460d30:::benchmark_data_smithi188_89276_object67263:head# 2024-04-23T16:25:47.578 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.578 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28464a8a:::benchmark_data_smithi188_89276_object161076:head# 2024-04-23T16:25:47.578 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.578 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2846d32a:::benchmark_data_smithi188_89276_object154836:head# 2024-04-23T16:25:47.578 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.578 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2846f2bc:::benchmark_data_smithi188_89276_object84010:head# 2024-04-23T16:25:47.578 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.578 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28477a61:::benchmark_data_smithi188_89276_object160904:head# 2024-04-23T16:25:47.578 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.578 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2848920f:::benchmark_data_smithi188_89276_object57316:head# 2024-04-23T16:25:47.578 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.578 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2848d75d:::benchmark_data_smithi188_89276_object115138:head# 2024-04-23T16:25:47.578 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.578 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28494da4:::benchmark_data_smithi188_89276_object102360:head# 2024-04-23T16:25:47.578 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:284969cf:::benchmark_data_smithi188_89276_object94321:head# 2024-04-23T16:25:47.579 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28496d01:::benchmark_data_smithi188_89276_object135296:head# 2024-04-23T16:25:47.579 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28498a06:::benchmark_data_smithi188_89276_object75355:head# 2024-04-23T16:25:47.579 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:284a08b0:::benchmark_data_smithi188_89276_object158101:head# 2024-04-23T16:25:47.579 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:284a47f8:::benchmark_data_smithi188_89276_object128543:head# 2024-04-23T16:25:47.579 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:284b25a2:::benchmark_data_smithi188_89276_object46354:head# 2024-04-23T16:25:47.579 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:284beb4c:::benchmark_data_smithi188_89276_object140594:head# 2024-04-23T16:25:47.579 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:284c48a9:::benchmark_data_smithi188_89276_object66023:head# 2024-04-23T16:25:47.580 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.580 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:284cfdf8:::benchmark_data_smithi188_89276_object54101:head# 2024-04-23T16:25:47.580 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:284d9b24:::benchmark_data_smithi188_89276_object109026:head# 2024-04-23T16:25:47.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:284fbe96:::benchmark_data_smithi188_89276_object164717:head# 2024-04-23T16:25:47.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:284fc4ac:::benchmark_data_smithi188_89276_object47317:head# 2024-04-23T16:25:47.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285042d5:::benchmark_data_smithi188_89276_object85899:head# 2024-04-23T16:25:47.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28504c7b:::benchmark_data_smithi188_89276_object94620:head# 2024-04-23T16:25:47.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285055d3:::benchmark_data_smithi188_89276_object121044:head# 2024-04-23T16:25:47.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2850c671:::benchmark_data_smithi188_89276_object130738:head# 2024-04-23T16:25:47.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285118b0:::benchmark_data_smithi188_89276_object122111:head# 2024-04-23T16:25:47.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28511f0a:::benchmark_data_smithi188_89276_object90429:head# 2024-04-23T16:25:47.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285378ef:::benchmark_data_smithi188_89276_object80125:head# 2024-04-23T16:25:47.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28542357:::benchmark_data_smithi188_89276_object40680:head# 2024-04-23T16:25:47.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285427d4:::benchmark_data_smithi188_89276_object112923:head# 2024-04-23T16:25:47.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.589 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2854b0ff:::benchmark_data_smithi188_89276_object43561:head# 2024-04-23T16:25:47.589 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.589 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2854ccc2:::benchmark_data_smithi188_89276_object124216:head# 2024-04-23T16:25:47.589 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.589 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2854e0d4:::benchmark_data_smithi188_89276_object149690:head# 2024-04-23T16:25:47.589 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.589 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28558b58:::benchmark_data_smithi188_89276_object145814:head# 2024-04-23T16:25:47.589 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.589 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2857dc75:::benchmark_data_smithi188_89276_object140183:head# 2024-04-23T16:25:47.589 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.589 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285873d6:::benchmark_data_smithi188_89276_object163423:head# 2024-04-23T16:25:47.589 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.589 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2859b4ba:::benchmark_data_smithi188_89276_object98736:head# 2024-04-23T16:25:47.589 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.590 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285a9eae:::benchmark_data_smithi188_89276_object85098:head# 2024-04-23T16:25:47.590 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.590 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285accf2:::benchmark_data_smithi188_89276_object103348:head# 2024-04-23T16:25:47.590 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.590 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285ae490:::benchmark_data_smithi188_89276_object109418:head# 2024-04-23T16:25:47.590 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.590 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285ba24c:::benchmark_data_smithi188_89276_object134632:head# 2024-04-23T16:25:47.590 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.590 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285bb69b:::benchmark_data_smithi188_89276_object122989:head# 2024-04-23T16:25:47.590 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.590 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285c42c6:::benchmark_data_smithi188_89276_object125162:head# 2024-04-23T16:25:47.590 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.590 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285c7295:::benchmark_data_smithi188_89276_object167402:head# 2024-04-23T16:25:47.590 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.591 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285cea37:::benchmark_data_smithi188_89276_object91389:head# 2024-04-23T16:25:47.591 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.593 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285d03f5:::benchmark_data_smithi188_89276_object127194:head# 2024-04-23T16:25:47.593 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.593 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285d610c:::benchmark_data_smithi188_89276_object70362:head# 2024-04-23T16:25:47.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.594 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285e9702:::benchmark_data_smithi188_89276_object117331:head# 2024-04-23T16:25:47.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.594 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285ea20b:::benchmark_data_smithi188_89276_object164398:head# 2024-04-23T16:25:47.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.594 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285eda66:::benchmark_data_smithi188_89276_object48401:head# 2024-04-23T16:25:47.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.594 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285f51d2:::benchmark_data_smithi188_89276_object157464:head# 2024-04-23T16:25:47.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.594 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:285f725e:::benchmark_data_smithi188_89276_object131417:head# 2024-04-23T16:25:47.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.594 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:286012cd:::benchmark_data_smithi188_89276_object43224:head# 2024-04-23T16:25:47.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.594 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28605763:::benchmark_data_smithi188_89276_object113029:head# 2024-04-23T16:25:47.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2860f655:::benchmark_data_smithi188_89276_object130930:head# 2024-04-23T16:25:47.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28618d9e:::benchmark_data_smithi188_89276_object90759:head# 2024-04-23T16:25:47.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2861d6b3:::benchmark_data_smithi188_89276_object79698:head# 2024-04-23T16:25:47.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2861ea40:::benchmark_data_smithi188_89276_object56764:head# 2024-04-23T16:25:47.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28625152:::benchmark_data_smithi188_89276_object47468:head# 2024-04-23T16:25:47.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28631569:::benchmark_data_smithi188_89276_object127731:head# 2024-04-23T16:25:47.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28645618:::benchmark_data_smithi188_89276_object56828:head# 2024-04-23T16:25:47.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2865fca3:::benchmark_data_smithi188_89276_object41043:head# 2024-04-23T16:25:47.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28666f6b:::benchmark_data_smithi188_89276_object146619:head# 2024-04-23T16:25:47.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2866e557:::benchmark_data_smithi188_89276_object132272:head# 2024-04-23T16:25:47.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2866ed84:::benchmark_data_smithi188_89276_object94619:head# 2024-04-23T16:25:47.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.597 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28676359:::benchmark_data_smithi188_89276_object146493:head# 2024-04-23T16:25:47.597 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.597 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28681144:::benchmark_data_smithi188_89276_object103222:head# 2024-04-23T16:25:47.597 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.597 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28684e58:::benchmark_data_smithi188_89276_object162655:head# 2024-04-23T16:25:47.597 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.597 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28685861:::benchmark_data_smithi188_89276_object75626:head# 2024-04-23T16:25:47.597 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.597 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28689aae:::benchmark_data_smithi188_89276_object101936:head# 2024-04-23T16:25:47.598 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.598 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2868f249:::benchmark_data_smithi188_89276_object152481:head# 2024-04-23T16:25:47.598 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.600 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28693a7e:::benchmark_data_smithi188_89276_object59775:head# 2024-04-23T16:25:47.600 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.600 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2869e270:::benchmark_data_smithi188_89276_object129749:head# 2024-04-23T16:25:47.600 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.600 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:286a4a0f:::benchmark_data_smithi188_89276_object125247:head# 2024-04-23T16:25:47.600 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:286b6480:::benchmark_data_smithi188_89276_object148140:head# 2024-04-23T16:25:47.601 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:286b9159:::benchmark_data_smithi188_89276_object57701:head# 2024-04-23T16:25:47.601 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:286bd43a:::benchmark_data_smithi188_89276_object166060:head# 2024-04-23T16:25:47.601 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.602 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:286bdca8:::benchmark_data_smithi188_89276_object86084:head# 2024-04-23T16:25:47.602 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:286c9601:::benchmark_data_smithi188_89276_object112427:head# 2024-04-23T16:25:47.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:286cb124:::benchmark_data_smithi188_89276_object80429:head# 2024-04-23T16:25:47.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:286cecc0:::benchmark_data_smithi188_89276_object61905:head# 2024-04-23T16:25:47.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:286d5d92:::benchmark_data_smithi188_89276_object129412:head# 2024-04-23T16:25:47.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:286de5f2:::benchmark_data_smithi188_89276_object47590:head# 2024-04-23T16:25:47.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:286e127b:::benchmark_data_smithi188_89276_object66896:head# 2024-04-23T16:25:47.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:286ea6f0:::benchmark_data_smithi188_89276_object134986:head# 2024-04-23T16:25:47.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.605 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:286ed7d5:::benchmark_data_smithi188_89276_object160628:head# 2024-04-23T16:25:47.605 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.606 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:286eda3e:::benchmark_data_smithi188_89276_object80951:head# 2024-04-23T16:25:47.606 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.606 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:286fb67f:::benchmark_data_smithi188_89276_object51689:head# 2024-04-23T16:25:47.606 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.606 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:286fbca2:::benchmark_data_smithi188_89276_object43507:head# 2024-04-23T16:25:47.606 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.606 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:286fbd91:::benchmark_data_smithi188_89276_object105008:head# 2024-04-23T16:25:47.606 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.606 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:287024dd:::benchmark_data_smithi188_89276_object58933:head# 2024-04-23T16:25:47.606 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.606 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28703610:::benchmark_data_smithi188_89276_object165453:head# 2024-04-23T16:25:47.606 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28720784:::benchmark_data_smithi188_89276_object157621:head# 2024-04-23T16:25:47.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28739567:::benchmark_data_smithi188_89276_object45395:head# 2024-04-23T16:25:47.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28746279:::benchmark_data_smithi188_89276_object164442:head# 2024-04-23T16:25:47.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:287462bf:::benchmark_data_smithi188_89276_object81847:head# 2024-04-23T16:25:47.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28749fd8:::benchmark_data_smithi188_89276_object163940:head# 2024-04-23T16:25:47.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28758d79:::benchmark_data_smithi188_89276_object149961:head# 2024-04-23T16:25:47.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:287627de:::benchmark_data_smithi188_89276_object165885:head# 2024-04-23T16:25:47.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28767354:::benchmark_data_smithi188_89276_object158798:head# 2024-04-23T16:25:47.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:287673a7:::benchmark_data_smithi188_89276_object113084:head# 2024-04-23T16:25:47.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28771f42:::benchmark_data_smithi188_89276_object93345:head# 2024-04-23T16:25:47.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2877a3c1:::benchmark_data_smithi188_89276_object99286:head# 2024-04-23T16:25:47.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2877b6e7:::benchmark_data_smithi188_89276_object77015:head# 2024-04-23T16:25:47.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2878363e:::benchmark_data_smithi188_89276_object160521:head# 2024-04-23T16:25:47.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:287840ec:::benchmark_data_smithi188_89276_object135030:head# 2024-04-23T16:25:47.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28793c77:::benchmark_data_smithi188_89276_object73507:head# 2024-04-23T16:25:47.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28794e6e:::benchmark_data_smithi188_89276_object166819:head# 2024-04-23T16:25:47.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:287a1bf3:::benchmark_data_smithi188_89276_object73373:head# 2024-04-23T16:25:47.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:287abb47:::benchmark_data_smithi188_89276_object162537:head# 2024-04-23T16:25:47.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:287bb395:::benchmark_data_smithi188_89276_object51307:head# 2024-04-23T16:25:47.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:287bdfbd:::benchmark_data_smithi188_89276_object111181:head# 2024-04-23T16:25:47.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:287e12ef:::benchmark_data_smithi188_89276_object58171:head# 2024-04-23T16:25:47.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:287e747f:::benchmark_data_smithi188_89276_object83104:head# 2024-04-23T16:25:47.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:287f8d69:::benchmark_data_smithi188_89276_object66368:head# 2024-04-23T16:25:47.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:287fe621:::benchmark_data_smithi188_89276_object143841:head# 2024-04-23T16:25:47.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28814108:::benchmark_data_smithi188_89276_object94939:head# 2024-04-23T16:25:47.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28814abe:::benchmark_data_smithi188_89276_object104615:head# 2024-04-23T16:25:47.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2881c27c:::benchmark_data_smithi188_89276_object55968:head# 2024-04-23T16:25:47.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2882654d:::benchmark_data_smithi188_89276_object104523:head# 2024-04-23T16:25:47.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28826bd6:::benchmark_data_smithi188_89276_object145745:head# 2024-04-23T16:25:47.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28836ab0:::benchmark_data_smithi188_89276_object65883:head# 2024-04-23T16:25:47.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28840e64:::benchmark_data_smithi188_89276_object128943:head# 2024-04-23T16:25:47.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2884507e:::benchmark_data_smithi188_89276_object72215:head# 2024-04-23T16:25:47.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28846c6e:::benchmark_data_smithi188_89276_object155515:head# 2024-04-23T16:25:47.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28850e0d:::benchmark_data_smithi188_89276_object63013:head# 2024-04-23T16:25:47.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2885caac:::benchmark_data_smithi188_89276_object166685:head# 2024-04-23T16:25:47.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:288672b4:::benchmark_data_smithi188_89276_object74183:head# 2024-04-23T16:25:47.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2886ac2f:::benchmark_data_smithi188_89276_object160393:head# 2024-04-23T16:25:47.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2886cbd8:::benchmark_data_smithi188_89276_object142448:head# 2024-04-23T16:25:47.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2886dbd1:::benchmark_data_smithi188_89276_object163530:head# 2024-04-23T16:25:47.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28870432:::benchmark_data_smithi188_89276_object52789:head# 2024-04-23T16:25:47.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28875511:::benchmark_data_smithi188_89276_object139986:head# 2024-04-23T16:25:47.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28878516:::benchmark_data_smithi188_89276_object164950:head# 2024-04-23T16:25:47.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2887e7ee:::benchmark_data_smithi188_89276_object133830:head# 2024-04-23T16:25:47.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2887f051:::benchmark_data_smithi188_89276_object50436:head# 2024-04-23T16:25:47.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:288848ae:::benchmark_data_smithi188_89276_object140640:head# 2024-04-23T16:25:47.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2888a2ce:::benchmark_data_smithi188_89276_object48839:head# 2024-04-23T16:25:47.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2888baed:::benchmark_data_smithi188_89276_object99074:head# 2024-04-23T16:25:47.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2888c813:::benchmark_data_smithi188_89276_object110391:head# 2024-04-23T16:25:47.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2888dd5b:::benchmark_data_smithi188_89276_object156269:head# 2024-04-23T16:25:47.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2888fd47:::benchmark_data_smithi188_89276_object115709:head# 2024-04-23T16:25:47.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:288950e3:::benchmark_data_smithi188_89276_object124928:head# 2024-04-23T16:25:47.622 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:288b0dda:::benchmark_data_smithi188_89276_object150122:head# 2024-04-23T16:25:47.622 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:288b72cc:::benchmark_data_smithi188_89276_object139238:head# 2024-04-23T16:25:47.622 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:288d91f2:::benchmark_data_smithi188_89276_object145007:head# 2024-04-23T16:25:47.622 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:288d9ac7:::benchmark_data_smithi188_89276_object119255:head# 2024-04-23T16:25:47.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:288e536a:::benchmark_data_smithi188_89276_object40794:head# 2024-04-23T16:25:47.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:288e6070:::benchmark_data_smithi188_89276_object129355:head# 2024-04-23T16:25:47.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28902f5f:::benchmark_data_smithi188_89276_object167311:head# 2024-04-23T16:25:47.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.625 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28905142:::benchmark_data_smithi188_89276_object126971:head# 2024-04-23T16:25:47.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28914f3a:::benchmark_data_smithi188_89276_object157134:head# 2024-04-23T16:25:47.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2891a73a:::benchmark_data_smithi188_89276_object63681:head# 2024-04-23T16:25:47.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:289201eb:::benchmark_data_smithi188_89276_object125672:head# 2024-04-23T16:25:47.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28927e80:::benchmark_data_smithi188_89276_object158745:head# 2024-04-23T16:25:47.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2892ec2f:::benchmark_data_smithi188_89276_object140094:head# 2024-04-23T16:25:47.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2894193d:::benchmark_data_smithi188_89276_object92470:head# 2024-04-23T16:25:47.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28951322:::benchmark_data_smithi188_89276_object124063:head# 2024-04-23T16:25:47.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28956b79:::benchmark_data_smithi188_89276_object80278:head# 2024-04-23T16:25:47.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2895a876:::benchmark_data_smithi188_89276_object53418:head# 2024-04-23T16:25:47.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2896ef35:::benchmark_data_smithi188_89276_object78350:head# 2024-04-23T16:25:47.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2897c6e3:::benchmark_data_smithi188_89276_object94973:head# 2024-04-23T16:25:47.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2898eccc:::benchmark_data_smithi188_89276_object166217:head# 2024-04-23T16:25:47.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2899ecad:::benchmark_data_smithi188_89276_object118110:head# 2024-04-23T16:25:47.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:289a21e6:::benchmark_data_smithi188_89276_object65628:head# 2024-04-23T16:25:47.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:289a3849:::benchmark_data_smithi188_89276_object156010:head# 2024-04-23T16:25:47.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:289a869a:::benchmark_data_smithi188_89276_object155033:head# 2024-04-23T16:25:47.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:289b3c0a:::benchmark_data_smithi188_89276_object153339:head# 2024-04-23T16:25:47.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:289b7ef5:::benchmark_data_smithi188_89276_object160786:head# 2024-04-23T16:25:47.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:289b9c42:::benchmark_data_smithi188_89276_object113592:head# 2024-04-23T16:25:47.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:289c7695:::benchmark_data_smithi188_89276_object109838:head# 2024-04-23T16:25:47.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.631 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:289ca4ea:::benchmark_data_smithi188_89276_object68205:head# 2024-04-23T16:25:47.631 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:289d687e:::benchmark_data_smithi188_89276_object122182:head# 2024-04-23T16:25:47.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.633 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:289d9918:::benchmark_data_smithi188_89276_object101337:head# 2024-04-23T16:25:47.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:289d9ddb:::benchmark_data_smithi188_89276_object158341:head# 2024-04-23T16:25:47.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:289eb362:::benchmark_data_smithi188_89276_object45521:head# 2024-04-23T16:25:47.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:289f1027:::benchmark_data_smithi188_89276_object66701:head# 2024-04-23T16:25:47.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:289fef03:::benchmark_data_smithi188_89276_object110887:head# 2024-04-23T16:25:47.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28a159f2:::benchmark_data_smithi188_89276_object151004:head# 2024-04-23T16:25:47.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28a18660:::benchmark_data_smithi188_89276_object106878:head# 2024-04-23T16:25:47.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.635 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28a20489:::benchmark_data_smithi188_89276_object63584:head# 2024-04-23T16:25:47.635 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.635 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28a34033:::benchmark_data_smithi188_89276_object45961:head# 2024-04-23T16:25:47.635 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.635 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28a51d3f:::benchmark_data_smithi188_89276_object142306:head# 2024-04-23T16:25:47.635 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.636 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28a53d21:::benchmark_data_smithi188_89276_object136892:head# 2024-04-23T16:25:47.636 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.636 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28a70777:::benchmark_data_smithi188_89276_object141967:head# 2024-04-23T16:25:47.636 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.636 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28a75265:::benchmark_data_smithi188_89276_object119910:head# 2024-04-23T16:25:47.636 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.639 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28a85252:::benchmark_data_smithi188_89276_object75372:head# 2024-04-23T16:25:47.639 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.641 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28a8f75a:::benchmark_data_smithi188_89276_object100062:head# 2024-04-23T16:25:47.641 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.642 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28a8fe1d:::benchmark_data_smithi188_89276_object113083:head# 2024-04-23T16:25:47.642 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.642 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28a91c8d:::benchmark_data_smithi188_89276_object162982:head# 2024-04-23T16:25:47.642 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28aa7890:::benchmark_data_smithi188_89276_object129855:head# 2024-04-23T16:25:47.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28aa83fb:::benchmark_data_smithi188_89276_object43483:head# 2024-04-23T16:25:47.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28aad94c:::benchmark_data_smithi188_89276_object165078:head# 2024-04-23T16:25:47.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28aadf46:::benchmark_data_smithi188_89276_object151738:head# 2024-04-23T16:25:47.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.644 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ab1b08:::benchmark_data_smithi188_89276_object106768:head# 2024-04-23T16:25:47.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ab29d9:::benchmark_data_smithi188_89276_object127076:head# 2024-04-23T16:25:47.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ab385c:::benchmark_data_smithi188_89276_object163872:head# 2024-04-23T16:25:47.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ab7915:::benchmark_data_smithi188_89276_object83581:head# 2024-04-23T16:25:47.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28abb6ed:::benchmark_data_smithi188_89276_object144082:head# 2024-04-23T16:25:47.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ac20de:::benchmark_data_smithi188_89276_object60984:head# 2024-04-23T16:25:47.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ace5f4:::benchmark_data_smithi188_89276_object43466:head# 2024-04-23T16:25:47.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ad8375:::benchmark_data_smithi188_89276_object62374:head# 2024-04-23T16:25:47.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28adc084:::benchmark_data_smithi188_89276_object66363:head# 2024-04-23T16:25:47.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ae66b2:::benchmark_data_smithi188_89276_object107483:head# 2024-04-23T16:25:47.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28af6479:::benchmark_data_smithi188_89276_object73849:head# 2024-04-23T16:25:47.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28b1973c:::benchmark_data_smithi188_89276_object81166:head# 2024-04-23T16:25:47.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28b1c7af:::benchmark_data_smithi188_89276_object61853:head# 2024-04-23T16:25:47.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28b1dc37:::benchmark_data_smithi188_89276_object143937:head# 2024-04-23T16:25:47.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28b2ac4e:::benchmark_data_smithi188_89276_object119848:head# 2024-04-23T16:25:47.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28b42fda:::benchmark_data_smithi188_89276_object120215:head# 2024-04-23T16:25:47.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28b641b4:::benchmark_data_smithi188_89276_object114282:head# 2024-04-23T16:25:47.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28b6481c:::benchmark_data_smithi188_89276_object165478:head# 2024-04-23T16:25:47.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28b69071:::benchmark_data_smithi188_89276_object104983:head# 2024-04-23T16:25:47.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28b6b99f:::benchmark_data_smithi188_89276_object75356:head# 2024-04-23T16:25:47.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28b6f933:::benchmark_data_smithi188_89276_object127037:head# 2024-04-23T16:25:47.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28b7d28c:::benchmark_data_smithi188_89276_object134716:head# 2024-04-23T16:25:47.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28b85623:::benchmark_data_smithi188_89276_object73555:head# 2024-04-23T16:25:47.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28b8b0cd:::benchmark_data_smithi188_89276_object79350:head# 2024-04-23T16:25:47.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28b929d1:::benchmark_data_smithi188_89276_object58553:head# 2024-04-23T16:25:47.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28b93f99:::benchmark_data_smithi188_89276_object58460:head# 2024-04-23T16:25:47.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28b95a33:::benchmark_data_smithi188_89276_object165555:head# 2024-04-23T16:25:47.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28b9cbdf:::benchmark_data_smithi188_89276_object113318:head# 2024-04-23T16:25:47.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28b9db67:::benchmark_data_smithi188_89276_object130185:head# 2024-04-23T16:25:47.668 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.668 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28b9ef78:::benchmark_data_smithi188_89276_object132436:head# 2024-04-23T16:25:47.668 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ba0910:::benchmark_data_smithi188_89276_object70124:head# 2024-04-23T16:25:47.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ba3faf:::benchmark_data_smithi188_89276_object117772:head# 2024-04-23T16:25:47.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ba705b:::benchmark_data_smithi188_89276_object77978:head# 2024-04-23T16:25:47.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28bae613:::benchmark_data_smithi188_89276_object84548:head# 2024-04-23T16:25:47.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28bb8dba:::benchmark_data_smithi188_89276_object88158:head# 2024-04-23T16:25:47.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28bba405:::benchmark_data_smithi188_89276_object84683:head# 2024-04-23T16:25:47.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28bd4124:::benchmark_data_smithi188_89276_object151677:head# 2024-04-23T16:25:47.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28bd43fd:::benchmark_data_smithi188_89276_object131209:head# 2024-04-23T16:25:47.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28bd5bdb:::benchmark_data_smithi188_89276_object65284:head# 2024-04-23T16:25:47.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28be238a:::benchmark_data_smithi188_89276_object80776:head# 2024-04-23T16:25:47.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28be3a27:::benchmark_data_smithi188_89276_object41094:head# 2024-04-23T16:25:47.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28beb762:::benchmark_data_smithi188_89276_object68387:head# 2024-04-23T16:25:47.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c01b21:::benchmark_data_smithi188_89276_object164439:head# 2024-04-23T16:25:47.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c0479c:::benchmark_data_smithi188_89276_object160034:head# 2024-04-23T16:25:47.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c0e750:::benchmark_data_smithi188_89276_object54021:head# 2024-04-23T16:25:47.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c2b058:::benchmark_data_smithi188_89276_object155659:head# 2024-04-23T16:25:47.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c312a6:::benchmark_data_smithi188_89276_object77663:head# 2024-04-23T16:25:47.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c3638b:::benchmark_data_smithi188_89276_object149676:head# 2024-04-23T16:25:47.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c386ca:::benchmark_data_smithi188_89276_object93892:head# 2024-04-23T16:25:47.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c44f32:::benchmark_data_smithi188_89276_object151650:head# 2024-04-23T16:25:47.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c47276:::benchmark_data_smithi188_89276_object125664:head# 2024-04-23T16:25:47.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c4733a:::benchmark_data_smithi188_89276_object110622:head# 2024-04-23T16:25:47.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c4f5c3:::benchmark_data_smithi188_89276_object58255:head# 2024-04-23T16:25:47.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c593d8:::benchmark_data_smithi188_89276_object118207:head# 2024-04-23T16:25:47.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c5968a:::benchmark_data_smithi188_89276_object64309:head# 2024-04-23T16:25:47.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c6a7c5:::benchmark_data_smithi188_89276_object43993:head# 2024-04-23T16:25:47.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c70be1:::benchmark_data_smithi188_89276_object114000:head# 2024-04-23T16:25:47.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c7b27c:::benchmark_data_smithi188_89276_object56607:head# 2024-04-23T16:25:47.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c7c0ec:::benchmark_data_smithi188_89276_object81509:head# 2024-04-23T16:25:47.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c7c1cc:::benchmark_data_smithi188_89276_object99533:head# 2024-04-23T16:25:47.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c8ef51:::benchmark_data_smithi188_89276_object71874:head# 2024-04-23T16:25:47.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c8ef9c:::benchmark_data_smithi188_89276_object64152:head# 2024-04-23T16:25:47.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c94434:::benchmark_data_smithi188_89276_object135265:head# 2024-04-23T16:25:47.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c9b96b:::benchmark_data_smithi188_89276_object78658:head# 2024-04-23T16:25:47.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28c9e6f5:::benchmark_data_smithi188_89276_object100193:head# 2024-04-23T16:25:47.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28cac72a:::benchmark_data_smithi188_89276_object56569:head# 2024-04-23T16:25:47.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28caf9a3:::benchmark_data_smithi188_89276_object56526:head# 2024-04-23T16:25:47.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28cb1473:::benchmark_data_smithi188_89276_object118412:head# 2024-04-23T16:25:47.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28cbdf82:::benchmark_data_smithi188_89276_object104754:head# 2024-04-23T16:25:47.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28cd5dfc:::benchmark_data_smithi188_89276_object126125:head# 2024-04-23T16:25:47.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ce538b:::benchmark_data_smithi188_89276_object77788:head# 2024-04-23T16:25:47.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28cec366:::benchmark_data_smithi188_89276_object166882:head# 2024-04-23T16:25:47.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28cf6285:::benchmark_data_smithi188_89276_object116163:head# 2024-04-23T16:25:47.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28cf72d9:::benchmark_data_smithi188_89276_object46378:head# 2024-04-23T16:25:47.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28cfd124:::benchmark_data_smithi188_89276_object157523:head# 2024-04-23T16:25:47.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d00138:::benchmark_data_smithi188_89276_object141360:head# 2024-04-23T16:25:47.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d147ad:::benchmark_data_smithi188_89276_object72157:head# 2024-04-23T16:25:47.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d1c9b8:::benchmark_data_smithi188_89276_object60579:head# 2024-04-23T16:25:47.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d1cb39:::benchmark_data_smithi188_89276_object41296:head# 2024-04-23T16:25:47.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d24184:::benchmark_data_smithi188_89276_object47079:head# 2024-04-23T16:25:47.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d247b7:::benchmark_data_smithi188_89276_object47056:head# 2024-04-23T16:25:47.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d286e7:::benchmark_data_smithi188_89276_object67786:head# 2024-04-23T16:25:47.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d2bb48:::benchmark_data_smithi188_89276_object139758:head# 2024-04-23T16:25:47.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d2bda3:::benchmark_data_smithi188_89276_object125346:head# 2024-04-23T16:25:47.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d3d78e:::benchmark_data_smithi188_89276_object93474:head# 2024-04-23T16:25:47.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d4ffc4:::benchmark_data_smithi188_89276_object60684:head# 2024-04-23T16:25:47.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d50683:::benchmark_data_smithi188_89276_object129131:head# 2024-04-23T16:25:47.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d523f5:::benchmark_data_smithi188_89276_object77844:head# 2024-04-23T16:25:47.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d58a35:::benchmark_data_smithi188_89276_object91541:head# 2024-04-23T16:25:47.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d6198a:::benchmark_data_smithi188_89276_object113836:head# 2024-04-23T16:25:47.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d61c29:::benchmark_data_smithi188_89276_object109741:head# 2024-04-23T16:25:47.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d667c2:::benchmark_data_smithi188_89276_object149680:head# 2024-04-23T16:25:47.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.683 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:47 smithi103 ceph-mon[39629]: pgmap v740: 17 pgs: 17 active+clean; 5.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.50k op/s 2024-04-23T16:25:47.686 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d6bf28:::benchmark_data_smithi188_89276_object135295:head# 2024-04-23T16:25:47.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d75465:::benchmark_data_smithi188_89276_object86486:head# 2024-04-23T16:25:47.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d79a88:::benchmark_data_smithi188_89276_object58080:head# 2024-04-23T16:25:47.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d7cd3c:::benchmark_data_smithi188_89276_object89022:head# 2024-04-23T16:25:47.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d913dd:::benchmark_data_smithi188_89276_object52706:head# 2024-04-23T16:25:47.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28d9264e:::benchmark_data_smithi188_89276_object110392:head# 2024-04-23T16:25:47.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28da03d0:::benchmark_data_smithi188_89276_object83019:head# 2024-04-23T16:25:47.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28da902b:::benchmark_data_smithi188_89276_object45338:head# 2024-04-23T16:25:47.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28dab43d:::benchmark_data_smithi188_89276_object129780:head# 2024-04-23T16:25:47.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28dbc59e:::benchmark_data_smithi188_89276_object60073:head# 2024-04-23T16:25:47.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28dbf9fd:::benchmark_data_smithi188_89276_object81348:head# 2024-04-23T16:25:47.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28dbfade:::benchmark_data_smithi188_89276_object106415:head# 2024-04-23T16:25:47.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28dd3dd1:::benchmark_data_smithi188_89276_object126481:head# 2024-04-23T16:25:47.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28dd95dd:::benchmark_data_smithi188_89276_object97504:head# 2024-04-23T16:25:47.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28de5c70:::benchmark_data_smithi188_89276_object113237:head# 2024-04-23T16:25:47.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28def57f:::benchmark_data_smithi188_89276_object50267:head# 2024-04-23T16:25:47.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28df60c5:::benchmark_data_smithi188_89276_object77875:head# 2024-04-23T16:25:47.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28df7624:::benchmark_data_smithi188_89276_object168130:head# 2024-04-23T16:25:47.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28df8cd4:::benchmark_data_smithi188_89276_object102631:head# 2024-04-23T16:25:47.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28df9dec:::benchmark_data_smithi188_89276_object97960:head# 2024-04-23T16:25:47.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28dfc84a:::benchmark_data_smithi188_89276_object42757:head# 2024-04-23T16:25:47.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28dfe28d:::benchmark_data_smithi188_89276_object161984:head# 2024-04-23T16:25:47.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28e01255:::benchmark_data_smithi188_89276_object148815:head# 2024-04-23T16:25:47.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28e07105:::benchmark_data_smithi188_89276_object95976:head# 2024-04-23T16:25:47.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28e0965e:::benchmark_data_smithi188_89276_object119159:head# 2024-04-23T16:25:47.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28e0dd16:::benchmark_data_smithi188_89276_object83925:head# 2024-04-23T16:25:47.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28e18048:::benchmark_data_smithi188_89276_object149285:head# 2024-04-23T16:25:47.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28e19c83:::benchmark_data_smithi188_89276_object47585:head# 2024-04-23T16:25:47.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28e1a165:::benchmark_data_smithi188_89276_object132308:head# 2024-04-23T16:25:47.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28e35d13:::benchmark_data_smithi188_89276_object146973:head# 2024-04-23T16:25:47.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28e3d87c:::benchmark_data_smithi188_89276_object74271:head# 2024-04-23T16:25:47.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28e489e1:::benchmark_data_smithi188_89276_object149040:head# 2024-04-23T16:25:47.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28e4a7ac:::benchmark_data_smithi188_89276_object93806:head# 2024-04-23T16:25:47.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28e55f08:::benchmark_data_smithi188_89276_object43430:head# 2024-04-23T16:25:47.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28e5bf62:::benchmark_data_smithi188_89276_object112030:head# 2024-04-23T16:25:47.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28e786d5:::benchmark_data_smithi188_89276_object65083:head# 2024-04-23T16:25:47.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28e7d07b:::benchmark_data_smithi188_89276_object51845:head# 2024-04-23T16:25:47.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28e8246f:::benchmark_data_smithi188_89276_object81196:head# 2024-04-23T16:25:47.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28e86229:::benchmark_data_smithi188_89276_object73526:head# 2024-04-23T16:25:47.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28e8ca3f:::benchmark_data_smithi188_89276_object45544:head# 2024-04-23T16:25:47.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28e8cc8c:::benchmark_data_smithi188_89276_object97026:head# 2024-04-23T16:25:47.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ea3143:::benchmark_data_smithi188_89276_object151583:head# 2024-04-23T16:25:47.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ea53c7:::benchmark_data_smithi188_89276_object148311:head# 2024-04-23T16:25:47.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ea7c41:::benchmark_data_smithi188_89276_object144459:head# 2024-04-23T16:25:47.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28eaa305:::benchmark_data_smithi188_89276_object82580:head# 2024-04-23T16:25:47.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28eacf03:::benchmark_data_smithi188_89276_object130344:head# 2024-04-23T16:25:47.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28eadd73:::benchmark_data_smithi188_89276_object69340:head# 2024-04-23T16:25:47.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28eaf648:::benchmark_data_smithi188_89276_object80886:head# 2024-04-23T16:25:47.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28eb71cb:::benchmark_data_smithi188_89276_object111278:head# 2024-04-23T16:25:47.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28eb8f81:::benchmark_data_smithi188_89276_object58906:head# 2024-04-23T16:25:47.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.694 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28eb8fa9:::benchmark_data_smithi188_89276_object120974:head# 2024-04-23T16:25:47.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.694 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28eb9360:::benchmark_data_smithi188_89276_object85339:head# 2024-04-23T16:25:47.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ebe0d2:::benchmark_data_smithi188_89276_object150662:head# 2024-04-23T16:25:47.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ec1f3d:::benchmark_data_smithi188_89276_object120799:head# 2024-04-23T16:25:47.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ecc82d:::benchmark_data_smithi188_89276_object46971:head# 2024-04-23T16:25:47.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ee8885:::benchmark_data_smithi188_89276_object144261:head# 2024-04-23T16:25:47.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28eeaea8:::benchmark_data_smithi188_89276_object97709:head# 2024-04-23T16:25:47.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28eee27c:::benchmark_data_smithi188_89276_object161859:head# 2024-04-23T16:25:47.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28eee41b:::benchmark_data_smithi188_89276_object164621:head# 2024-04-23T16:25:47.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ef02bb:::benchmark_data_smithi188_89276_object63980:head# 2024-04-23T16:25:47.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ef1dcc:::benchmark_data_smithi188_89276_object110442:head# 2024-04-23T16:25:47.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28efaaae:::benchmark_data_smithi188_89276_object168473:head# 2024-04-23T16:25:47.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28efc422:::benchmark_data_smithi188_89276_object83749:head# 2024-04-23T16:25:47.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f00453:::benchmark_data_smithi188_89276_object150576:head# 2024-04-23T16:25:47.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f0505c:::benchmark_data_smithi188_89276_object75588:head# 2024-04-23T16:25:47.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f0983e:::benchmark_data_smithi188_89276_object57179:head# 2024-04-23T16:25:47.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f18645:::benchmark_data_smithi188_89276_object60950:head# 2024-04-23T16:25:47.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f270f0:::benchmark_data_smithi188_89276_object98955:head# 2024-04-23T16:25:47.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f2e986:::benchmark_data_smithi188_89276_object160214:head# 2024-04-23T16:25:47.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f4462c:::benchmark_data_smithi188_89276_object124133:head# 2024-04-23T16:25:47.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f46ce5:::benchmark_data_smithi188_89276_object168376:head# 2024-04-23T16:25:47.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f4dee5:::benchmark_data_smithi188_89276_object79697:head# 2024-04-23T16:25:47.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f4fe10:::benchmark_data_smithi188_89276_object144012:head# 2024-04-23T16:25:47.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f60173:::benchmark_data_smithi188_89276_object123070:head# 2024-04-23T16:25:47.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f62824:::benchmark_data_smithi188_89276_object163507:head# 2024-04-23T16:25:47.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f6fb66:::benchmark_data_smithi188_89276_object125164:head# 2024-04-23T16:25:47.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f730d7:::benchmark_data_smithi188_89276_object80329:head# 2024-04-23T16:25:47.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f77616:::benchmark_data_smithi188_89276_object88965:head# 2024-04-23T16:25:47.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f7968f:::benchmark_data_smithi188_89276_object114949:head# 2024-04-23T16:25:47.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f7af5f:::benchmark_data_smithi188_89276_object48321:head# 2024-04-23T16:25:47.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f85a2e:::benchmark_data_smithi188_89276_object54269:head# 2024-04-23T16:25:47.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f8a4f8:::benchmark_data_smithi188_89276_object139014:head# 2024-04-23T16:25:47.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f8c422:::benchmark_data_smithi188_89276_object101468:head# 2024-04-23T16:25:47.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28f8f51e:::benchmark_data_smithi188_89276_object168633:head# 2024-04-23T16:25:47.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28fa9128:::benchmark_data_smithi188_89276_object109294:head# 2024-04-23T16:25:47.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28fbe3e5:::benchmark_data_smithi188_89276_object105404:head# 2024-04-23T16:25:47.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28fc5c8a:::benchmark_data_smithi188_89276_object87467:head# 2024-04-23T16:25:47.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28fcb8ed:::benchmark_data_smithi188_89276_object40346:head# 2024-04-23T16:25:47.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ff2f80:::benchmark_data_smithi188_89276_object70475:head# 2024-04-23T16:25:47.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:28ff6971:::benchmark_data_smithi188_89276_object165815:head# 2024-04-23T16:25:47.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29000606:::benchmark_data_smithi188_89276_object88815:head# 2024-04-23T16:25:47.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:290025df:::benchmark_data_smithi188_89276_object103143:head# 2024-04-23T16:25:47.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29006507:::benchmark_data_smithi188_89276_object168148:head# 2024-04-23T16:25:47.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29011d45:::benchmark_data_smithi188_89276_object60165:head# 2024-04-23T16:25:47.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29014f33:::benchmark_data_smithi188_89276_object103276:head# 2024-04-23T16:25:47.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.707 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29019827:::benchmark_data_smithi188_89276_object71764:head# 2024-04-23T16:25:47.707 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.707 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2901bab6:::benchmark_data_smithi188_89276_object118842:head# 2024-04-23T16:25:47.707 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.707 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29028b50:::benchmark_data_smithi188_89276_object112738:head# 2024-04-23T16:25:47.707 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.707 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2902a7a7:::benchmark_data_smithi188_89276_object140317:head# 2024-04-23T16:25:47.707 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.708 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2902f1d0:::benchmark_data_smithi188_89276_object68147:head# 2024-04-23T16:25:47.708 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.708 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2903332c:::benchmark_data_smithi188_89276_object130917:head# 2024-04-23T16:25:47.708 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.708 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29042e4e:::benchmark_data_smithi188_89276_object75030:head# 2024-04-23T16:25:47.708 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.708 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29045eff:::benchmark_data_smithi188_89276_object58858:head# 2024-04-23T16:25:47.708 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2904e4ac:::benchmark_data_smithi188_89276_object65777:head# 2024-04-23T16:25:47.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29063372:::benchmark_data_smithi188_89276_object105855:head# 2024-04-23T16:25:47.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2906f85c:::benchmark_data_smithi188_89276_object163597:head# 2024-04-23T16:25:47.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29074145:::benchmark_data_smithi188_89276_object86496:head# 2024-04-23T16:25:47.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29075408:::benchmark_data_smithi188_89276_object93400:head# 2024-04-23T16:25:47.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2907681c:::benchmark_data_smithi188_89276_object83526:head# 2024-04-23T16:25:47.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.716 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29085164:::benchmark_data_smithi188_89276_object132312:head# 2024-04-23T16:25:47.716 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.716 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:290913a0:::benchmark_data_smithi188_89276_object107156:head# 2024-04-23T16:25:47.716 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.716 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:290951b3:::benchmark_data_smithi188_89276_object141146:head# 2024-04-23T16:25:47.716 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.716 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:290a26eb:::benchmark_data_smithi188_89276_object104367:head# 2024-04-23T16:25:47.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:290a6ae2:::benchmark_data_smithi188_89276_object121412:head# 2024-04-23T16:25:47.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:290b6970:::benchmark_data_smithi188_89276_object106306:head# 2024-04-23T16:25:47.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:290c3b89:::benchmark_data_smithi188_89276_object60563:head# 2024-04-23T16:25:47.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:290d4422:::benchmark_data_smithi188_89276_object71428:head# 2024-04-23T16:25:47.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:290e36b5:::benchmark_data_smithi188_89276_object94419:head# 2024-04-23T16:25:47.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:290e3a5d:::benchmark_data_smithi188_89276_object51091:head# 2024-04-23T16:25:47.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:290e988d:::benchmark_data_smithi188_89276_object104617:head# 2024-04-23T16:25:47.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:290f50a6:::benchmark_data_smithi188_89276_object156989:head# 2024-04-23T16:25:47.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29106dd1:::benchmark_data_smithi188_89276_object84171:head# 2024-04-23T16:25:47.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291154fd:::benchmark_data_smithi188_89276_object62685:head# 2024-04-23T16:25:47.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291169aa:::benchmark_data_smithi188_89276_object96059:head# 2024-04-23T16:25:47.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291258e7:::benchmark_data_smithi188_89276_object165484:head# 2024-04-23T16:25:47.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2912b437:::benchmark_data_smithi188_89276_object128973:head# 2024-04-23T16:25:47.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2912c429:::benchmark_data_smithi188_89276_object168566:head# 2024-04-23T16:25:47.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2913d97f:::benchmark_data_smithi188_89276_object77305:head# 2024-04-23T16:25:47.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2913e9f4:::benchmark_data_smithi188_89276_object131670:head# 2024-04-23T16:25:47.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29142298:::benchmark_data_smithi188_89276_object145920:head# 2024-04-23T16:25:47.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291427cc:::benchmark_data_smithi188_89276_object117527:head# 2024-04-23T16:25:47.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2914b11f:::benchmark_data_smithi188_89276_object41537:head# 2024-04-23T16:25:47.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2914b3da:::benchmark_data_smithi188_89276_object129110:head# 2024-04-23T16:25:47.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29152cf6:::benchmark_data_smithi188_89276_object52423:head# 2024-04-23T16:25:47.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291548aa:::benchmark_data_smithi188_89276_object109177:head# 2024-04-23T16:25:47.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29160412:::benchmark_data_smithi188_89276_object89885:head# 2024-04-23T16:25:47.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29174c5c:::benchmark_data_smithi188_89276_object164763:head# 2024-04-23T16:25:47.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29175f84:::benchmark_data_smithi188_89276_object68336:head# 2024-04-23T16:25:47.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291901a6:::benchmark_data_smithi188_89276_object71549:head# 2024-04-23T16:25:47.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29194bc0:::benchmark_data_smithi188_89276_object154292:head# 2024-04-23T16:25:47.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291988ad:::benchmark_data_smithi188_89276_object166391:head# 2024-04-23T16:25:47.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291a0103:::benchmark_data_smithi188_89276_object147350:head# 2024-04-23T16:25:47.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291a0f82:::benchmark_data_smithi188_89276_object67476:head# 2024-04-23T16:25:47.727 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291a1bf2:::benchmark_data_smithi188_89276_object144004:head# 2024-04-23T16:25:47.727 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291a6c21:::benchmark_data_smithi188_89276_object145142:head# 2024-04-23T16:25:47.727 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291a9b3d:::benchmark_data_smithi188_89276_object49010:head# 2024-04-23T16:25:47.728 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.728 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291b0062:::benchmark_data_smithi188_89276_object69424:head# 2024-04-23T16:25:47.728 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.728 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291b09eb:::benchmark_data_smithi188_89276_object126288:head# 2024-04-23T16:25:47.728 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291b8d17:::benchmark_data_smithi188_89276_object76778:head# 2024-04-23T16:25:47.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291bf484:::benchmark_data_smithi188_89276_object57089:head# 2024-04-23T16:25:47.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291d66ec:::benchmark_data_smithi188_89276_object64114:head# 2024-04-23T16:25:47.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291d99ac:::benchmark_data_smithi188_89276_object68096:head# 2024-04-23T16:25:47.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291e5d22:::benchmark_data_smithi188_89276_object47620:head# 2024-04-23T16:25:47.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:291fb447:::benchmark_data_smithi188_89276_object164181:head# 2024-04-23T16:25:47.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29200d3d:::benchmark_data_smithi188_89276_object104768:head# 2024-04-23T16:25:47.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29215728:::benchmark_data_smithi188_89276_object50340:head# 2024-04-23T16:25:47.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2921f23f:::benchmark_data_smithi188_89276_object94671:head# 2024-04-23T16:25:47.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29223537:::benchmark_data_smithi188_89276_object53889:head# 2024-04-23T16:25:47.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29224a4d:::benchmark_data_smithi188_89276_object50728:head# 2024-04-23T16:25:47.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2922d2d0:::benchmark_data_smithi188_89276_object65921:head# 2024-04-23T16:25:47.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29233e30:::benchmark_data_smithi188_89276_object54232:head# 2024-04-23T16:25:47.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2923c8a5:::benchmark_data_smithi188_89276_object78358:head# 2024-04-23T16:25:47.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:292410d0:::benchmark_data_smithi188_89276_object161988:head# 2024-04-23T16:25:47.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2924e19f:::benchmark_data_smithi188_89276_object41248:head# 2024-04-23T16:25:47.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2925f325:::benchmark_data_smithi188_89276_object64547:head# 2024-04-23T16:25:47.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2926219e:::benchmark_data_smithi188_89276_object78781:head# 2024-04-23T16:25:47.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2926adb6:::benchmark_data_smithi188_89276_object124049:head# 2024-04-23T16:25:47.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.737 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29270b9f:::benchmark_data_smithi188_89276_object57633:head# 2024-04-23T16:25:47.737 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.737 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2927944c:::benchmark_data_smithi188_89276_object148949:head# 2024-04-23T16:25:47.737 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2927d05d:::benchmark_data_smithi188_89276_object159115:head# 2024-04-23T16:25:47.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:292831bc:::benchmark_data_smithi188_89276_object56750:head# 2024-04-23T16:25:47.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2929a87c:::benchmark_data_smithi188_89276_object50349:head# 2024-04-23T16:25:47.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2929c8a5:::benchmark_data_smithi188_89276_object89563:head# 2024-04-23T16:25:47.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2929d21e:::benchmark_data_smithi188_89276_object121235:head# 2024-04-23T16:25:47.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:292a1739:::benchmark_data_smithi188_89276_object137292:head# 2024-04-23T16:25:47.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:292b389d:::benchmark_data_smithi188_89276_object166358:head# 2024-04-23T16:25:47.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:292c5c47:::benchmark_data_smithi188_89276_object154713:head# 2024-04-23T16:25:47.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:292d3f07:::benchmark_data_smithi188_89276_object81100:head# 2024-04-23T16:25:47.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:292d4857:::benchmark_data_smithi188_89276_object71634:head# 2024-04-23T16:25:47.740 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.742 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:292d7d0d:::benchmark_data_smithi188_89276_object145012:head# 2024-04-23T16:25:47.742 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.742 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:292dccc2:::benchmark_data_smithi188_89276_object112131:head# 2024-04-23T16:25:47.742 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.742 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:292e3061:::benchmark_data_smithi188_89276_object90649:head# 2024-04-23T16:25:47.742 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.742 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:292e86ce:::benchmark_data_smithi188_89276_object144279:head# 2024-04-23T16:25:47.742 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.742 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:292ec19e:::benchmark_data_smithi188_89276_object149473:head# 2024-04-23T16:25:47.742 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.742 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:292eecfd:::benchmark_data_smithi188_89276_object156967:head# 2024-04-23T16:25:47.742 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.743 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:292f1399:::benchmark_data_smithi188_89276_object129789:head# 2024-04-23T16:25:47.743 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.743 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293012f3:::benchmark_data_smithi188_89276_object125758:head# 2024-04-23T16:25:47.743 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.743 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29302693:::benchmark_data_smithi188_89276_object89671:head# 2024-04-23T16:25:47.743 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.743 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29306a75:::benchmark_data_smithi188_89276_object85538:head# 2024-04-23T16:25:47.743 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.743 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2931fd02:::benchmark_data_smithi188_89276_object75305:head# 2024-04-23T16:25:47.743 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.743 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29324bcd:::benchmark_data_smithi188_89276_object157904:head# 2024-04-23T16:25:47.743 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.743 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2933159d:::benchmark_data_smithi188_89276_object167238:head# 2024-04-23T16:25:47.743 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293356ce:::benchmark_data_smithi188_89276_object115351:head# 2024-04-23T16:25:47.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29335c2c:::benchmark_data_smithi188_89276_object54866:head# 2024-04-23T16:25:47.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293376b4:::benchmark_data_smithi188_89276_object94481:head# 2024-04-23T16:25:47.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2934008b:::benchmark_data_smithi188_89276_object44664:head# 2024-04-23T16:25:47.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.746 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2934d35b:::benchmark_data_smithi188_89276_object115131:head# 2024-04-23T16:25:47.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.746 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29354abd:::benchmark_data_smithi188_89276_object82880:head# 2024-04-23T16:25:47.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.746 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29355c01:::benchmark_data_smithi188_89276_object161791:head# 2024-04-23T16:25:47.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.746 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2935c953:::benchmark_data_smithi188_89276_object102522:head# 2024-04-23T16:25:47.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.746 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29367ae3:::benchmark_data_smithi188_89276_object160540:head# 2024-04-23T16:25:47.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.746 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29368773:::benchmark_data_smithi188_89276_object138980:head# 2024-04-23T16:25:47.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.746 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2936e295:::benchmark_data_smithi188_89276_object58154:head# 2024-04-23T16:25:47.747 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.748 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293753a5:::benchmark_data_smithi188_89276_object107141:head# 2024-04-23T16:25:47.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29378fd2:::benchmark_data_smithi188_89276_object152092:head# 2024-04-23T16:25:47.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2938949a:::benchmark_data_smithi188_89276_object56118:head# 2024-04-23T16:25:47.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2939b656:::benchmark_data_smithi188_89276_object152572:head# 2024-04-23T16:25:47.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293a2bd0:::benchmark_data_smithi188_89276_object89010:head# 2024-04-23T16:25:47.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293a3c5f:::benchmark_data_smithi188_89276_object108366:head# 2024-04-23T16:25:47.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293a6bca:::benchmark_data_smithi188_89276_object64557:head# 2024-04-23T16:25:47.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293ab4a1:::benchmark_data_smithi188_89276_object92782:head# 2024-04-23T16:25:47.750 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.750 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293af16b:::benchmark_data_smithi188_89276_object115032:head# 2024-04-23T16:25:47.750 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.750 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293b281d:::benchmark_data_smithi188_89276_object62270:head# 2024-04-23T16:25:47.750 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.750 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293b454c:::benchmark_data_smithi188_89276_object64724:head# 2024-04-23T16:25:47.750 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.750 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293b7607:::benchmark_data_smithi188_89276_object46191:head# 2024-04-23T16:25:47.750 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.750 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293b8b8a:::benchmark_data_smithi188_89276_object70346:head# 2024-04-23T16:25:47.750 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293bfeb6:::benchmark_data_smithi188_89276_object147969:head# 2024-04-23T16:25:47.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293c60fc:::benchmark_data_smithi188_89276_object54397:head# 2024-04-23T16:25:47.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293cdda9:::benchmark_data_smithi188_89276_object48768:head# 2024-04-23T16:25:47.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293d2704:::benchmark_data_smithi188_89276_object113610:head# 2024-04-23T16:25:47.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293dbd49:::benchmark_data_smithi188_89276_object135900:head# 2024-04-23T16:25:47.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293dc08b:::benchmark_data_smithi188_89276_object141187:head# 2024-04-23T16:25:47.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293e5346:::benchmark_data_smithi188_89276_object48966:head# 2024-04-23T16:25:47.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293e6286:::benchmark_data_smithi188_89276_object136760:head# 2024-04-23T16:25:47.753 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.755 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293e9980:::benchmark_data_smithi188_89276_object44236:head# 2024-04-23T16:25:47.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.756 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293ecc87:::benchmark_data_smithi188_89276_object60833:head# 2024-04-23T16:25:47.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.756 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293eef4d:::benchmark_data_smithi188_89276_object90904:head# 2024-04-23T16:25:47.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.756 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:293f88bb:::benchmark_data_smithi188_89276_object79023:head# 2024-04-23T16:25:47.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.756 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29409f57:::benchmark_data_smithi188_89276_object113327:head# 2024-04-23T16:25:47.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.756 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2941a620:::benchmark_data_smithi188_89276_object91391:head# 2024-04-23T16:25:47.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.756 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2941faed:::benchmark_data_smithi188_89276_object56127:head# 2024-04-23T16:25:47.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.757 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:294260d3:::benchmark_data_smithi188_89276_object103156:head# 2024-04-23T16:25:47.757 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.757 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2942ae8d:::benchmark_data_smithi188_89276_object140434:head# 2024-04-23T16:25:47.757 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.757 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2942d717:::benchmark_data_smithi188_89276_object105726:head# 2024-04-23T16:25:47.757 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.757 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2944f686:::benchmark_data_smithi188_89276_object82879:head# 2024-04-23T16:25:47.757 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.757 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2945486e:::benchmark_data_smithi188_89276_object131393:head# 2024-04-23T16:25:47.757 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.757 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2945aa6a:::benchmark_data_smithi188_89276_object71719:head# 2024-04-23T16:25:47.757 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.757 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:294657d4:::benchmark_data_smithi188_89276_object102489:head# 2024-04-23T16:25:47.757 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.757 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2947354d:::benchmark_data_smithi188_89276_object125866:head# 2024-04-23T16:25:47.758 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.758 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29478ef1:::benchmark_data_smithi188_89276_object161724:head# 2024-04-23T16:25:47.758 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.758 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:294afe82:::benchmark_data_smithi188_89276_object134315:head# 2024-04-23T16:25:47.758 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.759 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:294b6f19:::benchmark_data_smithi188_89276_object155759:head# 2024-04-23T16:25:47.759 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.759 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:294bdd83:::benchmark_data_smithi188_89276_object80396:head# 2024-04-23T16:25:47.759 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.762 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:294c0130:::benchmark_data_smithi188_89276_object77877:head# 2024-04-23T16:25:47.762 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.762 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:294c084f:::benchmark_data_smithi188_89276_object137915:head# 2024-04-23T16:25:47.762 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.762 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:294c4792:::benchmark_data_smithi188_89276_object127816:head# 2024-04-23T16:25:47.762 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.762 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:294c77a9:::benchmark_data_smithi188_89276_object126423:head# 2024-04-23T16:25:47.762 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.763 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:294ce8bb:::benchmark_data_smithi188_89276_object114939:head# 2024-04-23T16:25:47.763 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.763 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:294d2379:::benchmark_data_smithi188_89276_object91954:head# 2024-04-23T16:25:47.763 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:294d648a:::benchmark_data_smithi188_89276_object151000:head# 2024-04-23T16:25:47.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:294d910d:::benchmark_data_smithi188_89276_object113793:head# 2024-04-23T16:25:47.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:294de50a:::benchmark_data_smithi188_89276_object164458:head# 2024-04-23T16:25:47.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:294e210b:::benchmark_data_smithi188_89276_object86261:head# 2024-04-23T16:25:47.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:294e3cb6:::benchmark_data_smithi188_89276_object85463:head# 2024-04-23T16:25:47.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.770 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:294f8a4d:::benchmark_data_smithi188_89276_object93424:head# 2024-04-23T16:25:47.770 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.771 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:294f8f6d:::benchmark_data_smithi188_89276_object132009:head# 2024-04-23T16:25:47.771 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.771 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:25:47.772 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:294fa06e:::benchmark_data_smithi188_89276_object69683:head# 2024-04-23T16:25:47.772 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2950ae13:::benchmark_data_smithi188_89276_object63465:head# 2024-04-23T16:25:47.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2950db24:::benchmark_data_smithi188_89276_object83198:head# 2024-04-23T16:25:47.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29522a9f:::benchmark_data_smithi188_89276_object41912:head# 2024-04-23T16:25:47.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:47.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:25:47.778 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29528c8c:::benchmark_data_smithi188_89276_object126851:head# 2024-04-23T16:25:47.778 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.778 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2952dff0:::benchmark_data_smithi188_89276_object98871:head# 2024-04-23T16:25:47.778 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.778 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29572f3c:::benchmark_data_smithi188_89276_object97792:head# 2024-04-23T16:25:47.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29578e9a:::benchmark_data_smithi188_89276_object86537:head# 2024-04-23T16:25:47.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2957f085:::benchmark_data_smithi188_89276_object115061:head# 2024-04-23T16:25:47.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2957f097:::benchmark_data_smithi188_89276_object48467:head# 2024-04-23T16:25:47.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29585711:::benchmark_data_smithi188_89276_object107210:head# 2024-04-23T16:25:47.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2958c4dd:::benchmark_data_smithi188_89276_object126480:head# 2024-04-23T16:25:47.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:295a6ea0:::benchmark_data_smithi188_89276_object74774:head# 2024-04-23T16:25:47.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:295bf883:::benchmark_data_smithi188_89276_object133984:head# 2024-04-23T16:25:47.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:295d0219:::benchmark_data_smithi188_89276_object145003:head# 2024-04-23T16:25:47.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:295d1fc8:::benchmark_data_smithi188_89276_object160821:head# 2024-04-23T16:25:47.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:295d7d57:::benchmark_data_smithi188_89276_object95465:head# 2024-04-23T16:25:47.782 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.782 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:295ddc3a:::benchmark_data_smithi188_89276_object158383:head# 2024-04-23T16:25:47.782 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.782 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:295e7508:::benchmark_data_smithi188_89276_object103141:head# 2024-04-23T16:25:47.782 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.782 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:295efb02:::benchmark_data_smithi188_89276_object146842:head# 2024-04-23T16:25:47.783 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.783 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:295f2f4f:::benchmark_data_smithi188_89276_object97276:head# 2024-04-23T16:25:47.783 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:295f85d7:::benchmark_data_smithi188_89276_object112911:head# 2024-04-23T16:25:47.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:295f9072:::benchmark_data_smithi188_89276_object142110:head# 2024-04-23T16:25:47.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29601d2c:::benchmark_data_smithi188_89276_object154073:head# 2024-04-23T16:25:47.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29602acf:::benchmark_data_smithi188_89276_object166876:head# 2024-04-23T16:25:47.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2961a008:::benchmark_data_smithi188_89276_object163673:head# 2024-04-23T16:25:47.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2961dd3d:::benchmark_data_smithi188_89276_object55936:head# 2024-04-23T16:25:47.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:296359a0:::benchmark_data_smithi188_89276_object159211:head# 2024-04-23T16:25:47.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2963a63c:::benchmark_data_smithi188_89276_object159753:head# 2024-04-23T16:25:47.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2963e61e:::benchmark_data_smithi188_89276_object127156:head# 2024-04-23T16:25:47.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2963f038:::benchmark_data_smithi188_89276_object121019:head# 2024-04-23T16:25:47.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29641c22:::benchmark_data_smithi188_89276_object157465:head# 2024-04-23T16:25:47.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2964777f:::benchmark_data_smithi188_89276_object98730:head# 2024-04-23T16:25:47.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:296572ec:::benchmark_data_smithi188_89276_object163893:head# 2024-04-23T16:25:47.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29670825:::benchmark_data_smithi188_89276_object64601:head# 2024-04-23T16:25:47.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:296715fc:::benchmark_data_smithi188_89276_object149079:head# 2024-04-23T16:25:47.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29679515:::benchmark_data_smithi188_89276_object45749:head# 2024-04-23T16:25:47.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29682644:::benchmark_data_smithi188_89276_object116117:head# 2024-04-23T16:25:47.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:296985d6:::benchmark_data_smithi188_89276_object114034:head# 2024-04-23T16:25:47.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:296ab5eb:::benchmark_data_smithi188_89276_object146038:head# 2024-04-23T16:25:47.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:296af43a:::benchmark_data_smithi188_89276_object141119:head# 2024-04-23T16:25:47.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.791 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:296b8423:::benchmark_data_smithi188_89276_object40567:head# 2024-04-23T16:25:47.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:296b9b55:::benchmark_data_smithi188_89276_object130469:head# 2024-04-23T16:25:47.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:296bd2d3:::benchmark_data_smithi188_89276_object133569:head# 2024-04-23T16:25:47.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:296cfc4f:::benchmark_data_smithi188_89276_object101458:head# 2024-04-23T16:25:47.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:296e324b:::benchmark_data_smithi188_89276_object81263:head# 2024-04-23T16:25:47.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:296ed164:::benchmark_data_smithi188_89276_object131602:head# 2024-04-23T16:25:47.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:296f246a:::benchmark_data_smithi188_89276_object112889:head# 2024-04-23T16:25:47.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29706cdd:::benchmark_data_smithi188_89276_object148729:head# 2024-04-23T16:25:47.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2971142d:::benchmark_data_smithi188_89276_object83953:head# 2024-04-23T16:25:47.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29713d48:::benchmark_data_smithi188_89276_object116475:head# 2024-04-23T16:25:47.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2972b19f:::benchmark_data_smithi188_89276_object94642:head# 2024-04-23T16:25:47.796 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.796 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2972d602:::benchmark_data_smithi188_89276_object139129:head# 2024-04-23T16:25:47.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2972def7:::benchmark_data_smithi188_89276_object146819:head# 2024-04-23T16:25:47.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:297328de:::benchmark_data_smithi188_89276_object63150:head# 2024-04-23T16:25:47.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2973979a:::benchmark_data_smithi188_89276_object55338:head# 2024-04-23T16:25:47.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29742e8e:::benchmark_data_smithi188_89276_object123784:head# 2024-04-23T16:25:47.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2975134c:::benchmark_data_smithi188_89276_object85727:head# 2024-04-23T16:25:47.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29755132:::benchmark_data_smithi188_89276_object156975:head# 2024-04-23T16:25:47.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.798 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29755c14:::benchmark_data_smithi188_89276_object153854:head# 2024-04-23T16:25:47.798 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.798 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:297589bc:::benchmark_data_smithi188_89276_object109978:head# 2024-04-23T16:25:47.798 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.799 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2975d525:::benchmark_data_smithi188_89276_object87865:head# 2024-04-23T16:25:47.799 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.799 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:297603c1:::benchmark_data_smithi188_89276_object57455:head# 2024-04-23T16:25:47.799 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.799 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2976c89f:::benchmark_data_smithi188_89276_object111216:head# 2024-04-23T16:25:47.799 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.799 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2976f11c:::benchmark_data_smithi188_89276_object138641:head# 2024-04-23T16:25:47.799 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2976fe84:::benchmark_data_smithi188_89276_object41314:head# 2024-04-23T16:25:47.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29773a78:::benchmark_data_smithi188_89276_object129500:head# 2024-04-23T16:25:47.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29795066:::benchmark_data_smithi188_89276_object135785:head# 2024-04-23T16:25:47.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:297a303c:::benchmark_data_smithi188_89276_object60074:head# 2024-04-23T16:25:47.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:297a4c51:::benchmark_data_smithi188_89276_object150568:head# 2024-04-23T16:25:47.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:297b2dd9:::benchmark_data_smithi188_89276_object104126:head# 2024-04-23T16:25:47.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:297bb0a6:::benchmark_data_smithi188_89276_object65455:head# 2024-04-23T16:25:47.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:297cab05:::benchmark_data_smithi188_89276_object142372:head# 2024-04-23T16:25:47.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:297db7a5:::benchmark_data_smithi188_89276_object122384:head# 2024-04-23T16:25:47.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:297dfbb4:::benchmark_data_smithi188_89276_object150840:head# 2024-04-23T16:25:47.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:297e0f96:::benchmark_data_smithi188_89276_object133375:head# 2024-04-23T16:25:47.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:297e7a36:::benchmark_data_smithi188_89276_object81714:head# 2024-04-23T16:25:47.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:297e82d3:::benchmark_data_smithi188_89276_object77303:head# 2024-04-23T16:25:47.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29801910:::benchmark_data_smithi188_89276_object145552:head# 2024-04-23T16:25:47.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29803657:::benchmark_data_smithi188_89276_object83005:head# 2024-04-23T16:25:47.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:298124bf:::benchmark_data_smithi188_89276_object127333:head# 2024-04-23T16:25:47.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2981bd6c:::benchmark_data_smithi188_89276_object143058:head# 2024-04-23T16:25:47.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29822497:::benchmark_data_smithi188_89276_object97981:head# 2024-04-23T16:25:47.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:298243d2:::benchmark_data_smithi188_89276_object85083:head# 2024-04-23T16:25:47.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29827808:::benchmark_data_smithi188_89276_object87527:head# 2024-04-23T16:25:47.820 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:298291e9:::benchmark_data_smithi188_89276_object57000:head# 2024-04-23T16:25:47.820 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2982ba7a:::benchmark_data_smithi188_89276_object164344:head# 2024-04-23T16:25:47.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2982e656:::benchmark_data_smithi188_89276_object136016:head# 2024-04-23T16:25:47.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:298336d6:::benchmark_data_smithi188_89276_object64667:head# 2024-04-23T16:25:47.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:298390f8:::benchmark_data_smithi188_89276_object81301:head# 2024-04-23T16:25:47.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2983bc72:::benchmark_data_smithi188_89276_object90947:head# 2024-04-23T16:25:47.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29849b12:::benchmark_data_smithi188_89276_object167503:head# 2024-04-23T16:25:47.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2984d7e9:::benchmark_data_smithi188_89276_object167263:head# 2024-04-23T16:25:47.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29850413:::benchmark_data_smithi188_89276_object78293:head# 2024-04-23T16:25:47.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:298591fe:::benchmark_data_smithi188_89276_object168201:head# 2024-04-23T16:25:47.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29865beb:::benchmark_data_smithi188_89276_object145883:head# 2024-04-23T16:25:47.824 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29866c46:::benchmark_data_smithi188_89276_object81232:head# 2024-04-23T16:25:47.824 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2986bd92:::benchmark_data_smithi188_89276_object125891:head# 2024-04-23T16:25:47.824 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2987477c:::benchmark_data_smithi188_89276_object72125:head# 2024-04-23T16:25:47.825 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.825 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29876642:::benchmark_data_smithi188_89276_object159119:head# 2024-04-23T16:25:47.825 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.825 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2987e480:::benchmark_data_smithi188_89276_object166788:head# 2024-04-23T16:25:47.825 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.825 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2987f207:::benchmark_data_smithi188_89276_object94937:head# 2024-04-23T16:25:47.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29885b11:::benchmark_data_smithi188_89276_object140152:head# 2024-04-23T16:25:47.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2989039d:::benchmark_data_smithi188_89276_object117433:head# 2024-04-23T16:25:47.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:298ad740:::benchmark_data_smithi188_89276_object106063:head# 2024-04-23T16:25:47.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:298ae5ed:::benchmark_data_smithi188_89276_object65462:head# 2024-04-23T16:25:47.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:298bffa7:::benchmark_data_smithi188_89276_object84220:head# 2024-04-23T16:25:47.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:298ce9eb:::benchmark_data_smithi188_89276_object100411:head# 2024-04-23T16:25:47.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.828 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:298de073:::benchmark_data_smithi188_89276_object71748:head# 2024-04-23T16:25:47.828 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.828 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:298e0573:::benchmark_data_smithi188_89276_object123683:head# 2024-04-23T16:25:47.828 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.828 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:298f9d77:::benchmark_data_smithi188_89276_object139938:head# 2024-04-23T16:25:47.828 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:298fb535:::benchmark_data_smithi188_89276_object45173:head# 2024-04-23T16:25:47.829 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2990690a:::benchmark_data_smithi188_89276_object144908:head# 2024-04-23T16:25:47.829 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29909b90:::benchmark_data_smithi188_89276_object75617:head# 2024-04-23T16:25:47.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.830 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2991023f:::benchmark_data_smithi188_89276_object95977:head# 2024-04-23T16:25:47.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.830 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2991109f:::benchmark_data_smithi188_89276_object62060:head# 2024-04-23T16:25:47.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.830 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2991e628:::benchmark_data_smithi188_89276_object70169:head# 2024-04-23T16:25:47.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29925b1a:::benchmark_data_smithi188_89276_object115763:head# 2024-04-23T16:25:47.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2992ff84:::benchmark_data_smithi188_89276_object91853:head# 2024-04-23T16:25:47.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29932fe1:::benchmark_data_smithi188_89276_object40364:head# 2024-04-23T16:25:47.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.842 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2993c320:::benchmark_data_smithi188_89276_object88303:head# 2024-04-23T16:25:47.842 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.843 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29944013:::benchmark_data_smithi188_89276_object109324:head# 2024-04-23T16:25:47.843 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.843 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29948781:::benchmark_data_smithi188_89276_object141864:head# 2024-04-23T16:25:47.843 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.843 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2995e0f5:::benchmark_data_smithi188_89276_object121340:head# 2024-04-23T16:25:47.843 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.843 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29963d59:::benchmark_data_smithi188_89276_object42602:head# 2024-04-23T16:25:47.843 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.843 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2996722e:::benchmark_data_smithi188_89276_object112154:head# 2024-04-23T16:25:47.843 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.843 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:299679f5:::benchmark_data_smithi188_89276_object131705:head# 2024-04-23T16:25:47.843 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.843 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2998b1d5:::benchmark_data_smithi188_89276_object106177:head# 2024-04-23T16:25:47.843 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2998d778:::benchmark_data_smithi188_89276_object57020:head# 2024-04-23T16:25:47.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:299a2c27:::benchmark_data_smithi188_89276_object106261:head# 2024-04-23T16:25:47.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:299a6f5b:::benchmark_data_smithi188_89276_object51817:head# 2024-04-23T16:25:47.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:299ad5f7:::benchmark_data_smithi188_89276_object156218:head# 2024-04-23T16:25:47.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:299b4b23:::benchmark_data_smithi188_89276_object46686:head# 2024-04-23T16:25:47.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:299c011c:::benchmark_data_smithi188_89276_object138866:head# 2024-04-23T16:25:47.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:299c6215:::benchmark_data_smithi188_89276_object153336:head# 2024-04-23T16:25:47.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:299d19af:::benchmark_data_smithi188_89276_object88747:head# 2024-04-23T16:25:47.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:299d54f2:::benchmark_data_smithi188_89276_object168057:head# 2024-04-23T16:25:47.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:299d69e5:::benchmark_data_smithi188_89276_object151418:head# 2024-04-23T16:25:47.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:299de20a:::benchmark_data_smithi188_89276_object70846:head# 2024-04-23T16:25:47.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:299e6dc8:::benchmark_data_smithi188_89276_object116013:head# 2024-04-23T16:25:47.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:299e98b4:::benchmark_data_smithi188_89276_object141423:head# 2024-04-23T16:25:47.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:299eb110:::benchmark_data_smithi188_89276_object66292:head# 2024-04-23T16:25:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:299ffc21:::benchmark_data_smithi188_89276_object114131:head# 2024-04-23T16:25:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29a0ccaf:::benchmark_data_smithi188_89276_object54013:head# 2024-04-23T16:25:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29a14956:::benchmark_data_smithi188_89276_object55201:head# 2024-04-23T16:25:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29a1527a:::benchmark_data_smithi188_89276_object147668:head# 2024-04-23T16:25:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29a16fc7:::benchmark_data_smithi188_89276_object127520:head# 2024-04-23T16:25:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29a28d57:::benchmark_data_smithi188_89276_object43309:head# 2024-04-23T16:25:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29a2d798:::benchmark_data_smithi188_89276_object145153:head# 2024-04-23T16:25:47.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29a2f766:::benchmark_data_smithi188_89276_object108922:head# 2024-04-23T16:25:47.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29a3cd96:::benchmark_data_smithi188_89276_object158227:head# 2024-04-23T16:25:47.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29a64029:::benchmark_data_smithi188_89276_object158424:head# 2024-04-23T16:25:47.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29a64e60:::benchmark_data_smithi188_89276_object147068:head# 2024-04-23T16:25:47.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29a69bca:::benchmark_data_smithi188_89276_object67518:head# 2024-04-23T16:25:47.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29a718c1:::benchmark_data_smithi188_89276_object156805:head# 2024-04-23T16:25:47.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29a78fea:::benchmark_data_smithi188_89276_object77491:head# 2024-04-23T16:25:47.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29a7932d:::benchmark_data_smithi188_89276_object126164:head# 2024-04-23T16:25:47.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29a7b096:::benchmark_data_smithi188_89276_object71894:head# 2024-04-23T16:25:47.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29a8ba1e:::benchmark_data_smithi188_89276_object152890:head# 2024-04-23T16:25:47.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29a904bd:::benchmark_data_smithi188_89276_object51657:head# 2024-04-23T16:25:47.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29a9c9be:::benchmark_data_smithi188_89276_object61668:head# 2024-04-23T16:25:47.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29ac23c7:::benchmark_data_smithi188_89276_object165917:head# 2024-04-23T16:25:47.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29ad42cc:::benchmark_data_smithi188_89276_object56028:head# 2024-04-23T16:25:47.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29ad9c1c:::benchmark_data_smithi188_89276_object118202:head# 2024-04-23T16:25:47.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29adc657:::benchmark_data_smithi188_89276_object84579:head# 2024-04-23T16:25:47.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29ae0489:::benchmark_data_smithi188_89276_object82498:head# 2024-04-23T16:25:47.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29ae10c2:::benchmark_data_smithi188_89276_object148209:head# 2024-04-23T16:25:47.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29ae6763:::benchmark_data_smithi188_89276_object158320:head# 2024-04-23T16:25:47.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29aef720:::benchmark_data_smithi188_89276_object59529:head# 2024-04-23T16:25:47.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29af186d:::benchmark_data_smithi188_89276_object108014:head# 2024-04-23T16:25:47.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29af4201:::benchmark_data_smithi188_89276_object113122:head# 2024-04-23T16:25:47.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29aff920:::benchmark_data_smithi188_89276_object80290:head# 2024-04-23T16:25:47.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29b04d5f:::benchmark_data_smithi188_89276_object113442:head# 2024-04-23T16:25:47.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29b076b5:::benchmark_data_smithi188_89276_object107938:head# 2024-04-23T16:25:47.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29b0c8fc:::benchmark_data_smithi188_89276_object166627:head# 2024-04-23T16:25:47.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29b14607:::benchmark_data_smithi188_89276_object45959:head# 2024-04-23T16:25:47.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29b1d498:::benchmark_data_smithi188_89276_object90342:head# 2024-04-23T16:25:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29b2494e:::benchmark_data_smithi188_89276_object148342:head# 2024-04-23T16:25:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29b2629f:::benchmark_data_smithi188_89276_object93875:head# 2024-04-23T16:25:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29b267f0:::benchmark_data_smithi188_89276_object45878:head# 2024-04-23T16:25:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29b26d3c:::benchmark_data_smithi188_89276_object112228:head# 2024-04-23T16:25:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29b294d1:::benchmark_data_smithi188_89276_object93585:head# 2024-04-23T16:25:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29b456d6:::benchmark_data_smithi188_89276_object146729:head# 2024-04-23T16:25:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29b4d99e:::benchmark_data_smithi188_89276_object79382:head# 2024-04-23T16:25:47.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29b4e093:::benchmark_data_smithi188_89276_object55839:head# 2024-04-23T16:25:47.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29b68450:::benchmark_data_smithi188_89276_object121802:head# 2024-04-23T16:25:47.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29b6c954:::benchmark_data_smithi188_89276_object153659:head# 2024-04-23T16:25:47.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29b717d7:::benchmark_data_smithi188_89276_object62052:head# 2024-04-23T16:25:47.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29b77497:::benchmark_data_smithi188_89276_object159782:head# 2024-04-23T16:25:47.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29b7cf22:::benchmark_data_smithi188_89276_object97707:head# 2024-04-23T16:25:47.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29b86b14:::benchmark_data_smithi188_89276_object148617:head# 2024-04-23T16:25:47.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.853 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29bb8ebe:::benchmark_data_smithi188_89276_object51331:head# 2024-04-23T16:25:47.853 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.853 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29bba290:::benchmark_data_smithi188_89276_object138055:head# 2024-04-23T16:25:47.853 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.853 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29bbc07e:::benchmark_data_smithi188_89276_object40730:head# 2024-04-23T16:25:47.853 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.853 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29bc00dd:::benchmark_data_smithi188_89276_object63237:head# 2024-04-23T16:25:47.853 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.853 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29bd8ceb:::benchmark_data_smithi188_89276_object73738:head# 2024-04-23T16:25:47.853 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.853 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29bdcab3:::benchmark_data_smithi188_89276_object86766:head# 2024-04-23T16:25:47.853 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.853 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29bf88ae:::benchmark_data_smithi188_89276_object58951:head# 2024-04-23T16:25:47.853 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.853 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29bfa320:::benchmark_data_smithi188_89276_object144000:head# 2024-04-23T16:25:47.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29c0f4a9:::benchmark_data_smithi188_89276_object156616:head# 2024-04-23T16:25:47.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29c18170:::benchmark_data_smithi188_89276_object110053:head# 2024-04-23T16:25:47.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29c2d123:::benchmark_data_smithi188_89276_object160690:head# 2024-04-23T16:25:47.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29c39562:::benchmark_data_smithi188_89276_object140204:head# 2024-04-23T16:25:47.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29c560a4:::benchmark_data_smithi188_89276_object117815:head# 2024-04-23T16:25:47.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29c5f6fe:::benchmark_data_smithi188_89276_object116575:head# 2024-04-23T16:25:47.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29c7916b:::benchmark_data_smithi188_89276_object83611:head# 2024-04-23T16:25:47.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29c84f01:::benchmark_data_smithi188_89276_object150857:head# 2024-04-23T16:25:47.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29c89ca6:::benchmark_data_smithi188_89276_object63530:head# 2024-04-23T16:25:47.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29c8b0e0:::benchmark_data_smithi188_89276_object91622:head# 2024-04-23T16:25:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29c9437c:::benchmark_data_smithi188_89276_object125512:head# 2024-04-23T16:25:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29c98598:::benchmark_data_smithi188_89276_object47434:head# 2024-04-23T16:25:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29c9ad1d:::benchmark_data_smithi188_89276_object144240:head# 2024-04-23T16:25:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29c9aef5:::benchmark_data_smithi188_89276_object160221:head# 2024-04-23T16:25:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29c9f978:::benchmark_data_smithi188_89276_object81140:head# 2024-04-23T16:25:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29ca7d7d:::benchmark_data_smithi188_89276_object47475:head# 2024-04-23T16:25:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29cabae7:::benchmark_data_smithi188_89276_object56824:head# 2024-04-23T16:25:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29caf406:::benchmark_data_smithi188_89276_object98306:head# 2024-04-23T16:25:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29cb45eb:::benchmark_data_smithi188_89276_object156468:head# 2024-04-23T16:25:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29cb5ba2:::benchmark_data_smithi188_89276_object61833:head# 2024-04-23T16:25:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29cc09f2:::benchmark_data_smithi188_89276_object51225:head# 2024-04-23T16:25:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29cc22f0:::benchmark_data_smithi188_89276_object53053:head# 2024-04-23T16:25:47.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29cd6a17:::benchmark_data_smithi188_89276_object150416:head# 2024-04-23T16:25:47.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29cdca95:::benchmark_data_smithi188_89276_object72699:head# 2024-04-23T16:25:47.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29cdf311:::benchmark_data_smithi188_89276_object81449:head# 2024-04-23T16:25:47.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29cea70d:::benchmark_data_smithi188_89276_object159500:head# 2024-04-23T16:25:47.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29d0086c:::benchmark_data_smithi188_89276_object149191:head# 2024-04-23T16:25:47.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29d02988:::benchmark_data_smithi188_89276_object90906:head# 2024-04-23T16:25:47.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29d0fec3:::benchmark_data_smithi188_89276_object138078:head# 2024-04-23T16:25:47.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29d1f316:::benchmark_data_smithi188_89276_object44535:head# 2024-04-23T16:25:47.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29d3bd56:::benchmark_data_smithi188_89276_object138022:head# 2024-04-23T16:25:47.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29d4816b:::benchmark_data_smithi188_89276_object52234:head# 2024-04-23T16:25:47.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29d4dc78:::benchmark_data_smithi188_89276_object161026:head# 2024-04-23T16:25:47.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29d6c8ca:::benchmark_data_smithi188_89276_object96156:head# 2024-04-23T16:25:47.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29d775aa:::benchmark_data_smithi188_89276_object128863:head# 2024-04-23T16:25:47.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29d78763:::benchmark_data_smithi188_89276_object168236:head# 2024-04-23T16:25:47.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29d7c852:::benchmark_data_smithi188_89276_object64772:head# 2024-04-23T16:25:47.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29d8484c:::benchmark_data_smithi188_89276_object134731:head# 2024-04-23T16:25:47.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29d94399:::benchmark_data_smithi188_89276_object157230:head# 2024-04-23T16:25:47.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29d998b6:::benchmark_data_smithi188_89276_object112130:head# 2024-04-23T16:25:47.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29d9d7da:::benchmark_data_smithi188_89276_object129264:head# 2024-04-23T16:25:47.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29db1d2e:::benchmark_data_smithi188_89276_object45202:head# 2024-04-23T16:25:47.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29db569c:::benchmark_data_smithi188_89276_object156553:head# 2024-04-23T16:25:47.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29dc24cc:::benchmark_data_smithi188_89276_object91702:head# 2024-04-23T16:25:47.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29dc7d54:::benchmark_data_smithi188_89276_object110577:head# 2024-04-23T16:25:47.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29dc8f70:::benchmark_data_smithi188_89276_object150277:head# 2024-04-23T16:25:47.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29de08ca:::benchmark_data_smithi188_89276_object48703:head# 2024-04-23T16:25:47.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29de5d98:::benchmark_data_smithi188_89276_object105141:head# 2024-04-23T16:25:47.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29e01b55:::benchmark_data_smithi188_89276_object73941:head# 2024-04-23T16:25:47.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29e0ac12:::benchmark_data_smithi188_89276_object163496:head# 2024-04-23T16:25:47.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29e16f0b:::benchmark_data_smithi188_89276_object159092:head# 2024-04-23T16:25:47.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29e1866a:::benchmark_data_smithi188_89276_object163817:head# 2024-04-23T16:25:47.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29e21cef:::benchmark_data_smithi188_89276_object87385:head# 2024-04-23T16:25:47.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29e30e45:::benchmark_data_smithi188_89276_object90943:head# 2024-04-23T16:25:47.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29e33f8d:::benchmark_data_smithi188_89276_object73214:head# 2024-04-23T16:25:47.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29e364ca:::benchmark_data_smithi188_89276_object122237:head# 2024-04-23T16:25:47.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29e521ad:::benchmark_data_smithi188_89276_object47989:head# 2024-04-23T16:25:47.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29e55ba0:::benchmark_data_smithi188_89276_object113071:head# 2024-04-23T16:25:47.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29e60ee2:::benchmark_data_smithi188_89276_object142509:head# 2024-04-23T16:25:47.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29e759e4:::benchmark_data_smithi188_89276_object69237:head# 2024-04-23T16:25:47.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29e78b03:::benchmark_data_smithi188_89276_object64317:head# 2024-04-23T16:25:47.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29e78cc6:::benchmark_data_smithi188_89276_object143657:head# 2024-04-23T16:25:47.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29e8516c:::benchmark_data_smithi188_89276_object158925:head# 2024-04-23T16:25:47.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29e861c1:::benchmark_data_smithi188_89276_object140677:head# 2024-04-23T16:25:47.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29e8ede3:::benchmark_data_smithi188_89276_object116329:head# 2024-04-23T16:25:47.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29e92b60:::benchmark_data_smithi188_89276_object94987:head# 2024-04-23T16:25:47.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29ea0d79:::benchmark_data_smithi188_89276_object122703:head# 2024-04-23T16:25:47.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29ea43f5:::benchmark_data_smithi188_89276_object73017:head# 2024-04-23T16:25:47.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29ea4741:::benchmark_data_smithi188_89276_object88024:head# 2024-04-23T16:25:47.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29ea5c77:::benchmark_data_smithi188_89276_object125463:head# 2024-04-23T16:25:47.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29eb198f:::benchmark_data_smithi188_89276_object71910:head# 2024-04-23T16:25:47.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29eba8a7:::benchmark_data_smithi188_89276_object101394:head# 2024-04-23T16:25:47.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29ec1963:::benchmark_data_smithi188_89276_object103003:head# 2024-04-23T16:25:47.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29ec55e7:::benchmark_data_smithi188_89276_object152963:head# 2024-04-23T16:25:47.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29ec71c3:::benchmark_data_smithi188_89276_object130264:head# 2024-04-23T16:25:47.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29ed0331:::benchmark_data_smithi188_89276_object124586:head# 2024-04-23T16:25:47.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29ed5250:::benchmark_data_smithi188_89276_object122873:head# 2024-04-23T16:25:47.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29edcc51:::benchmark_data_smithi188_89276_object114701:head# 2024-04-23T16:25:47.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29ee7259:::benchmark_data_smithi188_89276_object47610:head# 2024-04-23T16:25:47.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29eeb24d:::benchmark_data_smithi188_89276_object95838:head# 2024-04-23T16:25:47.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29ef2e07:::benchmark_data_smithi188_89276_object100672:head# 2024-04-23T16:25:47.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f0714e:::benchmark_data_smithi188_89276_object82234:head# 2024-04-23T16:25:47.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f114a7:::benchmark_data_smithi188_89276_object91542:head# 2024-04-23T16:25:47.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f15182:::benchmark_data_smithi188_89276_object53137:head# 2024-04-23T16:25:47.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f1c781:::benchmark_data_smithi188_89276_object157297:head# 2024-04-23T16:25:47.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f1f117:::benchmark_data_smithi188_89276_object103176:head# 2024-04-23T16:25:47.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f337a4:::benchmark_data_smithi188_89276_object93754:head# 2024-04-23T16:25:47.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f4762e:::benchmark_data_smithi188_89276_object116733:head# 2024-04-23T16:25:47.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f4cb03:::benchmark_data_smithi188_89276_object54306:head# 2024-04-23T16:25:47.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f4e1cc:::benchmark_data_smithi188_89276_object103261:head# 2024-04-23T16:25:47.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f50dd6:::benchmark_data_smithi188_89276_object53436:head# 2024-04-23T16:25:47.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f51a3b:::benchmark_data_smithi188_89276_object88980:head# 2024-04-23T16:25:47.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f52a99:::benchmark_data_smithi188_89276_object60115:head# 2024-04-23T16:25:47.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f59088:::benchmark_data_smithi188_89276_object157703:head# 2024-04-23T16:25:47.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f65a48:::benchmark_data_smithi188_89276_object99224:head# 2024-04-23T16:25:47.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f70be0:::benchmark_data_smithi188_89276_object102913:head# 2024-04-23T16:25:47.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f72554:::benchmark_data_smithi188_89276_object58961:head# 2024-04-23T16:25:47.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f7466f:::benchmark_data_smithi188_89276_object49072:head# 2024-04-23T16:25:47.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f77616:::benchmark_data_smithi188_89276_object98544:head# 2024-04-23T16:25:47.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f7e911:::benchmark_data_smithi188_89276_object80900:head# 2024-04-23T16:25:47.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f8923a:::benchmark_data_smithi188_89276_object114966:head# 2024-04-23T16:25:47.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f8d836:::benchmark_data_smithi188_89276_object122695:head# 2024-04-23T16:25:47.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29f9fdd6:::benchmark_data_smithi188_89276_object114497:head# 2024-04-23T16:25:47.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29fa6681:::benchmark_data_smithi188_89276_object96420:head# 2024-04-23T16:25:47.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29fa8fa6:::benchmark_data_smithi188_89276_object57328:head# 2024-04-23T16:25:47.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29fadfae:::benchmark_data_smithi188_89276_object167108:head# 2024-04-23T16:25:47.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29fb42e5:::benchmark_data_smithi188_89276_object83362:head# 2024-04-23T16:25:47.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29fc2486:::benchmark_data_smithi188_89276_object68011:head# 2024-04-23T16:25:47.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29fc3e02:::benchmark_data_smithi188_89276_object60262:head# 2024-04-23T16:25:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29fd5035:::benchmark_data_smithi188_89276_object42973:head# 2024-04-23T16:25:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29fd8a5d:::benchmark_data_smithi188_89276_object82327:head# 2024-04-23T16:25:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29fdbbc1:::benchmark_data_smithi188_89276_object149152:head# 2024-04-23T16:25:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29fe1c42:::benchmark_data_smithi188_89276_object61370:head# 2024-04-23T16:25:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29fe553e:::benchmark_data_smithi188_89276_object120485:head# 2024-04-23T16:25:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29fee977:::benchmark_data_smithi188_89276_object74039:head# 2024-04-23T16:25:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:29ff6b74:::benchmark_data_smithi188_89276_object81520:head# 2024-04-23T16:25:47.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a0006ed:::benchmark_data_smithi188_89276_object154491:head# 2024-04-23T16:25:47.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a006d5a:::benchmark_data_smithi188_89276_object84340:head# 2024-04-23T16:25:47.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a00c3bd:::benchmark_data_smithi188_89276_object73824:head# 2024-04-23T16:25:47.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a0121bd:::benchmark_data_smithi188_89276_object77739:head# 2024-04-23T16:25:47.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.882 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:25:47.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a01d23d:::benchmark_data_smithi188_89276_object96336:head# 2024-04-23T16:25:47.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a028348:::benchmark_data_smithi188_89276_object58037:head# 2024-04-23T16:25:47.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a0302ee:::benchmark_data_smithi188_89276_object94079:head# 2024-04-23T16:25:47.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a031342:::benchmark_data_smithi188_89276_object81608:head# 2024-04-23T16:25:47.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a036959:::benchmark_data_smithi188_89276_object54296:head# 2024-04-23T16:25:47.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a0373e6:::benchmark_data_smithi188_89276_object97893:head# 2024-04-23T16:25:47.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a0395e3:::benchmark_data_smithi188_89276_object99248:head# 2024-04-23T16:25:47.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a044d2d:::benchmark_data_smithi188_89276_object114472:head# 2024-04-23T16:25:47.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a04549d:::benchmark_data_smithi188_89276_object63382:head# 2024-04-23T16:25:47.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a048773:::benchmark_data_smithi188_89276_object65132:head# 2024-04-23T16:25:47.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a04a042:::benchmark_data_smithi188_89276_object138501:head# 2024-04-23T16:25:47.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a05124b:::benchmark_data_smithi188_89276_object44325:head# 2024-04-23T16:25:47.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a0519d1:::benchmark_data_smithi188_89276_object136802:head# 2024-04-23T16:25:47.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a082799:::benchmark_data_smithi188_89276_object50012:head# 2024-04-23T16:25:47.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a0aaace:::benchmark_data_smithi188_89276_object114580:head# 2024-04-23T16:25:47.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a0afb69:::benchmark_data_smithi188_89276_object160278:head# 2024-04-23T16:25:47.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a0b8066:::benchmark_data_smithi188_89276_object110344:head# 2024-04-23T16:25:47.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a0bd82e:::benchmark_data_smithi188_89276_object85206:head# 2024-04-23T16:25:47.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a0c456b:::benchmark_data_smithi188_89276_object92138:head# 2024-04-23T16:25:47.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a0d60e6:::benchmark_data_smithi188_89276_object136416:head# 2024-04-23T16:25:47.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a0dd649:::benchmark_data_smithi188_89276_object153322:head# 2024-04-23T16:25:47.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a0e7104:::benchmark_data_smithi188_89276_object147323:head# 2024-04-23T16:25:47.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a0f9db6:::benchmark_data_smithi188_89276_object46852:head# 2024-04-23T16:25:47.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a0fdae1:::benchmark_data_smithi188_89276_object107771:head# 2024-04-23T16:25:47.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a1086eb:::benchmark_data_smithi188_89276_object117040:head# 2024-04-23T16:25:47.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.893 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a126432:::benchmark_data_smithi188_89276_object64766:head# 2024-04-23T16:25:47.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.893 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a139476:::benchmark_data_smithi188_89276_object141799:head# 2024-04-23T16:25:47.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.893 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a13f708:::benchmark_data_smithi188_89276_object108806:head# 2024-04-23T16:25:47.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.893 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a14378d:::benchmark_data_smithi188_89276_object81473:head# 2024-04-23T16:25:47.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a149da2:::benchmark_data_smithi188_89276_object44159:head# 2024-04-23T16:25:47.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a156190:::benchmark_data_smithi188_89276_object52607:head# 2024-04-23T16:25:47.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a15db99:::benchmark_data_smithi188_89276_object42521:head# 2024-04-23T16:25:47.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a16a7d1:::benchmark_data_smithi188_89276_object166026:head# 2024-04-23T16:25:47.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a175a89:::benchmark_data_smithi188_89276_object124566:head# 2024-04-23T16:25:47.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a17e2cb:::benchmark_data_smithi188_89276_object111480:head# 2024-04-23T16:25:47.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a187557:::benchmark_data_smithi188_89276_object143049:head# 2024-04-23T16:25:47.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a18b85c:::benchmark_data_smithi188_89276_object57229:head# 2024-04-23T16:25:47.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a19061d:::benchmark_data_smithi188_89276_object71320:head# 2024-04-23T16:25:47.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a190d13:::benchmark_data_smithi188_89276_object80968:head# 2024-04-23T16:25:47.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a19faee:::benchmark_data_smithi188_89276_object134345:head# 2024-04-23T16:25:47.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a1a9c97:::benchmark_data_smithi188_89276_object126579:head# 2024-04-23T16:25:47.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a1ac38b:::benchmark_data_smithi188_89276_object47546:head# 2024-04-23T16:25:47.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a1ac560:::benchmark_data_smithi188_89276_object59442:head# 2024-04-23T16:25:47.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a1ae65b:::benchmark_data_smithi188_89276_object165876:head# 2024-04-23T16:25:47.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a1b1671:::benchmark_data_smithi188_89276_object55349:head# 2024-04-23T16:25:47.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a1b3fc5:::benchmark_data_smithi188_89276_object144602:head# 2024-04-23T16:25:47.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a1b41c3:::benchmark_data_smithi188_89276_object42071:head# 2024-04-23T16:25:47.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a1b7f27:::benchmark_data_smithi188_89276_object127901:head# 2024-04-23T16:25:47.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a1bde30:::benchmark_data_smithi188_89276_object88500:head# 2024-04-23T16:25:47.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a1be85d:::benchmark_data_smithi188_89276_object165529:head# 2024-04-23T16:25:47.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.899 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a1cafa8:::benchmark_data_smithi188_89276_object125315:head# 2024-04-23T16:25:47.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.899 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a1d70fc:::benchmark_data_smithi188_89276_object158111:head# 2024-04-23T16:25:47.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.899 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a1dd6bc:::benchmark_data_smithi188_89276_object102844:head# 2024-04-23T16:25:47.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a1e27c6:::benchmark_data_smithi188_89276_object103206:head# 2024-04-23T16:25:47.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a1e5be5:::benchmark_data_smithi188_89276_object140226:head# 2024-04-23T16:25:47.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a1f03c4:::benchmark_data_smithi188_89276_object127550:head# 2024-04-23T16:25:47.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a1f8dbe:::benchmark_data_smithi188_89276_object79416:head# 2024-04-23T16:25:47.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a2002cf:::benchmark_data_smithi188_89276_object99762:head# 2024-04-23T16:25:47.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a207d65:::benchmark_data_smithi188_89276_object121956:head# 2024-04-23T16:25:47.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a20c5a7:::benchmark_data_smithi188_89276_object127171:head# 2024-04-23T16:25:47.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.901 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a220c53:::benchmark_data_smithi188_89276_object53414:head# 2024-04-23T16:25:47.901 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.901 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a225209:::benchmark_data_smithi188_89276_object117427:head# 2024-04-23T16:25:47.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a2299f2:::benchmark_data_smithi188_89276_object138473:head# 2024-04-23T16:25:47.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a230c91:::benchmark_data_smithi188_89276_object154383:head# 2024-04-23T16:25:47.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a231baf:::benchmark_data_smithi188_89276_object136429:head# 2024-04-23T16:25:47.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a24962e:::benchmark_data_smithi188_89276_object115306:head# 2024-04-23T16:25:47.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a25a385:::benchmark_data_smithi188_89276_object162880:head# 2024-04-23T16:25:47.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a25c6b2:::benchmark_data_smithi188_89276_object116732:head# 2024-04-23T16:25:47.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a2666c5:::benchmark_data_smithi188_89276_object61348:head# 2024-04-23T16:25:47.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a26a942:::benchmark_data_smithi188_89276_object130310:head# 2024-04-23T16:25:47.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a26e871:::benchmark_data_smithi188_89276_object139046:head# 2024-04-23T16:25:47.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a27026e:::benchmark_data_smithi188_89276_object164536:head# 2024-04-23T16:25:47.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a273a4c:::benchmark_data_smithi188_89276_object70543:head# 2024-04-23T16:25:47.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a28a086:::benchmark_data_smithi188_89276_object122188:head# 2024-04-23T16:25:47.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a2a7af0:::benchmark_data_smithi188_89276_object87032:head# 2024-04-23T16:25:47.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a2abc3a:::benchmark_data_smithi188_89276_object118641:head# 2024-04-23T16:25:47.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a2b4119:::benchmark_data_smithi188_89276_object105260:head# 2024-04-23T16:25:47.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a2b4bb2:::benchmark_data_smithi188_89276_object129143:head# 2024-04-23T16:25:47.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a2b954e:::benchmark_data_smithi188_89276_object81026:head# 2024-04-23T16:25:47.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a2bee7e:::benchmark_data_smithi188_89276_object49390:head# 2024-04-23T16:25:47.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a2c01ad:::benchmark_data_smithi188_89276_object108249:head# 2024-04-23T16:25:47.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a2cdb76:::benchmark_data_smithi188_89276_object71911:head# 2024-04-23T16:25:47.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a2e4ede:::benchmark_data_smithi188_89276_object49258:head# 2024-04-23T16:25:47.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a2e6ef9:::benchmark_data_smithi188_89276_object141546:head# 2024-04-23T16:25:47.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a2eb7c7:::benchmark_data_smithi188_89276_object78858:head# 2024-04-23T16:25:47.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a2ef378:::benchmark_data_smithi188_89276_object53775:head# 2024-04-23T16:25:47.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a309151:::benchmark_data_smithi188_89276_object69829:head# 2024-04-23T16:25:47.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a3136e1:::benchmark_data_smithi188_89276_object72152:head# 2024-04-23T16:25:47.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a3366df:::benchmark_data_smithi188_89276_object87152:head# 2024-04-23T16:25:47.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a33c79a:::benchmark_data_smithi188_89276_object128842:head# 2024-04-23T16:25:47.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a34afe4:::benchmark_data_smithi188_89276_object108412:head# 2024-04-23T16:25:47.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.909 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a3599f5:::benchmark_data_smithi188_89276_object116739:head# 2024-04-23T16:25:47.909 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.909 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a365788:::benchmark_data_smithi188_89276_object129091:head# 2024-04-23T16:25:47.909 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.909 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a3673cb:::benchmark_data_smithi188_89276_object46542:head# 2024-04-23T16:25:47.909 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a3735b0:::benchmark_data_smithi188_89276_object63031:head# 2024-04-23T16:25:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a374140:::benchmark_data_smithi188_89276_object63762:head# 2024-04-23T16:25:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a3749a9:::benchmark_data_smithi188_89276_object79469:head# 2024-04-23T16:25:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a3a0767:::benchmark_data_smithi188_89276_object43447:head# 2024-04-23T16:25:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a3b0e78:::benchmark_data_smithi188_89276_object85676:head# 2024-04-23T16:25:47.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.916 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a3b5090:::benchmark_data_smithi188_89276_object109566:head# 2024-04-23T16:25:47.916 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a3b56ad:::benchmark_data_smithi188_89276_object136164:head# 2024-04-23T16:25:47.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a3da6f2:::benchmark_data_smithi188_89276_object90899:head# 2024-04-23T16:25:47.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.918 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:47.918 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a3e898c:::benchmark_data_smithi188_89276_object78101:head# 2024-04-23T16:25:47.918 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.918 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a3ecb1f:::benchmark_data_smithi188_89276_object162395:head# 2024-04-23T16:25:47.918 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a3f345c:::benchmark_data_smithi188_89276_object51239:head# 2024-04-23T16:25:47.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a3f80d1:::benchmark_data_smithi188_89276_object130097:head# 2024-04-23T16:25:47.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a3f8db9:::benchmark_data_smithi188_89276_object46209:head# 2024-04-23T16:25:47.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a400e12:::benchmark_data_smithi188_89276_object84503:head# 2024-04-23T16:25:47.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4113cc:::benchmark_data_smithi188_89276_object134463:head# 2024-04-23T16:25:47.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a41e523:::benchmark_data_smithi188_89276_object166099:head# 2024-04-23T16:25:47.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4269fb:::benchmark_data_smithi188_89276_object146261:head# 2024-04-23T16:25:47.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a429ae5:::benchmark_data_smithi188_89276_object86893:head# 2024-04-23T16:25:47.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a432c3c:::benchmark_data_smithi188_89276_object99686:head# 2024-04-23T16:25:47.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a435955:::benchmark_data_smithi188_89276_object112321:head# 2024-04-23T16:25:47.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4371e6:::benchmark_data_smithi188_89276_object77431:head# 2024-04-23T16:25:47.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a43b96e:::benchmark_data_smithi188_89276_object165175:head# 2024-04-23T16:25:47.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a442aee:::benchmark_data_smithi188_89276_object104881:head# 2024-04-23T16:25:47.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4479f3:::benchmark_data_smithi188_89276_object134434:head# 2024-04-23T16:25:47.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a44866c:::benchmark_data_smithi188_89276_object118639:head# 2024-04-23T16:25:47.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a44bdd1:::benchmark_data_smithi188_89276_object128502:head# 2024-04-23T16:25:47.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4635cc:::benchmark_data_smithi188_89276_object111984:head# 2024-04-23T16:25:47.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a47abb9:::benchmark_data_smithi188_89276_object148296:head# 2024-04-23T16:25:47.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a47b6d6:::benchmark_data_smithi188_89276_object149913:head# 2024-04-23T16:25:47.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a47bc44:::benchmark_data_smithi188_89276_object125817:head# 2024-04-23T16:25:47.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a47e0af:::benchmark_data_smithi188_89276_object59112:head# 2024-04-23T16:25:47.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4863aa:::benchmark_data_smithi188_89276_object41059:head# 2024-04-23T16:25:47.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4864d3:::benchmark_data_smithi188_89276_object86323:head# 2024-04-23T16:25:47.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a490f50:::benchmark_data_smithi188_89276_object102151:head# 2024-04-23T16:25:47.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4926ad:::benchmark_data_smithi188_89276_object148988:head# 2024-04-23T16:25:47.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a496a49:::benchmark_data_smithi188_89276_object67219:head# 2024-04-23T16:25:47.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4a048b:::benchmark_data_smithi188_89276_object59228:head# 2024-04-23T16:25:47.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4a2eeb:::benchmark_data_smithi188_89276_object46900:head# 2024-04-23T16:25:47.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4a6b61:::benchmark_data_smithi188_89276_object72796:head# 2024-04-23T16:25:47.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4b84cb:::benchmark_data_smithi188_89276_object125960:head# 2024-04-23T16:25:47.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4b93c5:::benchmark_data_smithi188_89276_object115896:head# 2024-04-23T16:25:47.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4d7f83:::benchmark_data_smithi188_89276_object69287:head# 2024-04-23T16:25:47.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4dd4fe:::benchmark_data_smithi188_89276_object146963:head# 2024-04-23T16:25:47.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4e0436:::benchmark_data_smithi188_89276_object75749:head# 2024-04-23T16:25:47.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4e0579:::benchmark_data_smithi188_89276_object129752:head# 2024-04-23T16:25:47.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4e13b3:::benchmark_data_smithi188_89276_object54286:head# 2024-04-23T16:25:47.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4e5a84:::benchmark_data_smithi188_89276_object62090:head# 2024-04-23T16:25:47.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a4ead62:::benchmark_data_smithi188_89276_object101118:head# 2024-04-23T16:25:47.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a5060f0:::benchmark_data_smithi188_89276_object85847:head# 2024-04-23T16:25:47.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a50e324:::benchmark_data_smithi188_89276_object108392:head# 2024-04-23T16:25:47.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a50ec09:::benchmark_data_smithi188_89276_object88556:head# 2024-04-23T16:25:47.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a50eee7:::benchmark_data_smithi188_89276_object166376:head# 2024-04-23T16:25:47.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a512c08:::benchmark_data_smithi188_89276_object100134:head# 2024-04-23T16:25:47.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a5149d0:::benchmark_data_smithi188_89276_object144149:head# 2024-04-23T16:25:47.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a517956:::benchmark_data_smithi188_89276_object43341:head# 2024-04-23T16:25:47.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a519992:::benchmark_data_smithi188_89276_object132876:head# 2024-04-23T16:25:47.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a5237bd:::benchmark_data_smithi188_89276_object144955:head# 2024-04-23T16:25:47.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a52cc75:::benchmark_data_smithi188_89276_object112895:head# 2024-04-23T16:25:47.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a53057c:::benchmark_data_smithi188_89276_object162658:head# 2024-04-23T16:25:47.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a53647f:::benchmark_data_smithi188_89276_object87963:head# 2024-04-23T16:25:47.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a536ae5:::benchmark_data_smithi188_89276_object75475:head# 2024-04-23T16:25:47.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a549367:::benchmark_data_smithi188_89276_object86929:head# 2024-04-23T16:25:47.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a54c953:::benchmark_data_smithi188_89276_object120004:head# 2024-04-23T16:25:47.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a5508e6:::benchmark_data_smithi188_89276_object76105:head# 2024-04-23T16:25:47.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a560874:::benchmark_data_smithi188_89276_object63745:head# 2024-04-23T16:25:47.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a56b493:::benchmark_data_smithi188_89276_object132496:head# 2024-04-23T16:25:47.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a576fb6:::benchmark_data_smithi188_89276_object61629:head# 2024-04-23T16:25:47.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a582062:::benchmark_data_smithi188_89276_object97717:head# 2024-04-23T16:25:47.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a596769:::benchmark_data_smithi188_89276_object107192:head# 2024-04-23T16:25:47.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a59f7d5:::benchmark_data_smithi188_89276_object88342:head# 2024-04-23T16:25:47.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a5a1c61:::benchmark_data_smithi188_89276_object54363:head# 2024-04-23T16:25:47.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a5b7793:::benchmark_data_smithi188_89276_object149472:head# 2024-04-23T16:25:47.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a5b7f0d:::benchmark_data_smithi188_89276_object52808:head# 2024-04-23T16:25:47.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.946 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:25:47.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a5b8a03:::benchmark_data_smithi188_89276_object121521:head# 2024-04-23T16:25:47.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a5b8dee:::benchmark_data_smithi188_89276_object77087:head# 2024-04-23T16:25:47.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a5be0fb:::benchmark_data_smithi188_89276_object88568:head# 2024-04-23T16:25:47.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a5cb60f:::benchmark_data_smithi188_89276_object139649:head# 2024-04-23T16:25:47.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a5cc119:::benchmark_data_smithi188_89276_object168596:head# 2024-04-23T16:25:47.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a5cdc13:::benchmark_data_smithi188_89276_object115498:head# 2024-04-23T16:25:47.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a5d4058:::benchmark_data_smithi188_89276_object104274:head# 2024-04-23T16:25:47.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.948 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:47 smithi188 ceph-mon[44101]: pgmap v740: 17 pgs: 17 active+clean; 5.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.50k op/s 2024-04-23T16:25:47.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:47.949 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:25:47.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a5d81ff:::benchmark_data_smithi188_89276_object99639:head# 2024-04-23T16:25:47.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a5dddbf:::benchmark_data_smithi188_89276_object74500:head# 2024-04-23T16:25:47.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a5f54bb:::benchmark_data_smithi188_89276_object51563:head# 2024-04-23T16:25:47.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a6197aa:::benchmark_data_smithi188_89276_object128311:head# 2024-04-23T16:25:47.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a620450:::benchmark_data_smithi188_89276_object66277:head# 2024-04-23T16:25:47.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a62ffb1:::benchmark_data_smithi188_89276_object41792:head# 2024-04-23T16:25:47.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a63abee:::benchmark_data_smithi188_89276_object48960:head# 2024-04-23T16:25:47.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a63b1e1:::benchmark_data_smithi188_89276_object99410:head# 2024-04-23T16:25:47.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a649b73:::benchmark_data_smithi188_89276_object101199:head# 2024-04-23T16:25:47.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a663798:::benchmark_data_smithi188_89276_object93588:head# 2024-04-23T16:25:47.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a67b9b7:::benchmark_data_smithi188_89276_object73495:head# 2024-04-23T16:25:47.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a67d854:::benchmark_data_smithi188_89276_object81941:head# 2024-04-23T16:25:47.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a67e36d:::benchmark_data_smithi188_89276_object48447:head# 2024-04-23T16:25:47.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a68d693:::benchmark_data_smithi188_89276_object81468:head# 2024-04-23T16:25:47.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a698073:::benchmark_data_smithi188_89276_object167201:head# 2024-04-23T16:25:47.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a69b570:::benchmark_data_smithi188_89276_object55916:head# 2024-04-23T16:25:47.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a6a15c8:::benchmark_data_smithi188_89276_object167852:head# 2024-04-23T16:25:47.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a6a5533:::benchmark_data_smithi188_89276_object46859:head# 2024-04-23T16:25:47.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a6abce4:::benchmark_data_smithi188_89276_object56668:head# 2024-04-23T16:25:47.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a6b4085:::benchmark_data_smithi188_89276_object91414:head# 2024-04-23T16:25:47.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a6c5219:::benchmark_data_smithi188_89276_object153156:head# 2024-04-23T16:25:47.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a6d59d4:::benchmark_data_smithi188_89276_object147565:head# 2024-04-23T16:25:47.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a6de541:::benchmark_data_smithi188_89276_object101781:head# 2024-04-23T16:25:47.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a6e84ff:::benchmark_data_smithi188_89276_object140511:head# 2024-04-23T16:25:47.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a6eab4f:::benchmark_data_smithi188_89276_object133853:head# 2024-04-23T16:25:47.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a6f1b29:::benchmark_data_smithi188_89276_object76924:head# 2024-04-23T16:25:47.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a6f6344:::benchmark_data_smithi188_89276_object164818:head# 2024-04-23T16:25:47.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a6fbcc4:::benchmark_data_smithi188_89276_object89753:head# 2024-04-23T16:25:47.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a708e3f:::benchmark_data_smithi188_89276_object158629:head# 2024-04-23T16:25:47.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a71a604:::benchmark_data_smithi188_89276_object43837:head# 2024-04-23T16:25:47.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a71a856:::benchmark_data_smithi188_89276_object137935:head# 2024-04-23T16:25:47.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a71e931:::benchmark_data_smithi188_89276_object54334:head# 2024-04-23T16:25:47.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a72e3e8:::benchmark_data_smithi188_89276_object43324:head# 2024-04-23T16:25:47.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a7358a5:::benchmark_data_smithi188_89276_object119858:head# 2024-04-23T16:25:47.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a746b52:::benchmark_data_smithi188_89276_object113524:head# 2024-04-23T16:25:47.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a74e604:::benchmark_data_smithi188_89276_object106214:head# 2024-04-23T16:25:47.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a75d799:::benchmark_data_smithi188_89276_object47781:head# 2024-04-23T16:25:47.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a7601cc:::benchmark_data_smithi188_89276_object76813:head# 2024-04-23T16:25:47.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a764e08:::benchmark_data_smithi188_89276_object122512:head# 2024-04-23T16:25:47.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a76a0db:::benchmark_data_smithi188_89276_object106894:head# 2024-04-23T16:25:47.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a76a334:::benchmark_data_smithi188_89276_object95496:head# 2024-04-23T16:25:47.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.978 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a76cc80:::benchmark_data_smithi188_89276_object118455:head# 2024-04-23T16:25:47.978 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.978 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a76d552:::benchmark_data_smithi188_89276_object65057:head# 2024-04-23T16:25:47.978 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.979 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a77d234:::benchmark_data_smithi188_89276_object100160:head# 2024-04-23T16:25:47.979 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.979 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a78217c:::benchmark_data_smithi188_89276_object54613:head# 2024-04-23T16:25:47.979 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.979 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a78da30:::benchmark_data_smithi188_89276_object97868:head# 2024-04-23T16:25:47.979 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.979 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a799a43:::benchmark_data_smithi188_89276_object99570:head# 2024-04-23T16:25:47.979 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.980 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a7a5d04:::benchmark_data_smithi188_89276_object71756:head# 2024-04-23T16:25:47.980 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.980 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a7acee6:::benchmark_data_smithi188_89276_object143534:head# 2024-04-23T16:25:47.980 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.980 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a7b2735:::benchmark_data_smithi188_89276_object95020:head# 2024-04-23T16:25:47.980 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.980 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a7bec0e:::benchmark_data_smithi188_89276_object48392:head# 2024-04-23T16:25:47.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.982 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a7c0632:::benchmark_data_smithi188_89276_object125376:head# 2024-04-23T16:25:47.982 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.982 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a7c16aa:::benchmark_data_smithi188_89276_object123381:head# 2024-04-23T16:25:47.982 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.982 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a7c2ee7:::benchmark_data_smithi188_89276_object146675:head# 2024-04-23T16:25:47.983 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.983 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a7c8659:::benchmark_data_smithi188_89276_object137361:head# 2024-04-23T16:25:47.983 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.983 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a7e11a1:::benchmark_data_smithi188_89276_object157799:head# 2024-04-23T16:25:47.983 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.983 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a7e2f59:::benchmark_data_smithi188_89276_object100561:head# 2024-04-23T16:25:47.983 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.983 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a7f1706:::benchmark_data_smithi188_89276_object54832:head# 2024-04-23T16:25:47.983 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.983 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a7f9d65:::benchmark_data_smithi188_89276_object108666:head# 2024-04-23T16:25:47.983 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.983 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a80b587:::benchmark_data_smithi188_89276_object87202:head# 2024-04-23T16:25:47.983 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.985 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a80fa6a:::benchmark_data_smithi188_89276_object104410:head# 2024-04-23T16:25:47.985 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.985 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a82a6a7:::benchmark_data_smithi188_89276_object71087:head# 2024-04-23T16:25:47.985 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.985 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a842f38:::benchmark_data_smithi188_89276_object119840:head# 2024-04-23T16:25:47.985 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.986 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a8439ac:::benchmark_data_smithi188_89276_object91468:head# 2024-04-23T16:25:47.986 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.986 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a84cdc5:::benchmark_data_smithi188_89276_object74447:head# 2024-04-23T16:25:47.986 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.986 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a853d83:::benchmark_data_smithi188_89276_object144052:head# 2024-04-23T16:25:47.986 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.986 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a860306:::benchmark_data_smithi188_89276_object123424:head# 2024-04-23T16:25:47.986 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.986 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a869f28:::benchmark_data_smithi188_89276_object109971:head# 2024-04-23T16:25:47.986 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.986 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a86aa28:::benchmark_data_smithi188_89276_object73850:head# 2024-04-23T16:25:47.986 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.986 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a876d8d:::benchmark_data_smithi188_89276_object110490:head# 2024-04-23T16:25:47.987 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.987 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a87c513:::benchmark_data_smithi188_89276_object62866:head# 2024-04-23T16:25:47.987 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.989 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a882fa8:::benchmark_data_smithi188_89276_object76954:head# 2024-04-23T16:25:47.989 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.989 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a886d00:::benchmark_data_smithi188_89276_object90111:head# 2024-04-23T16:25:47.989 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.989 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a887c2d:::benchmark_data_smithi188_89276_object70404:head# 2024-04-23T16:25:47.989 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.989 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a888de1:::benchmark_data_smithi188_89276_object147108:head# 2024-04-23T16:25:47.989 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.990 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a88aa60:::benchmark_data_smithi188_89276_object97755:head# 2024-04-23T16:25:47.990 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.990 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a8905ff:::benchmark_data_smithi188_89276_object104658:head# 2024-04-23T16:25:47.990 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.990 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a893895:::benchmark_data_smithi188_89276_object104805:head# 2024-04-23T16:25:47.990 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.990 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a898fac:::benchmark_data_smithi188_89276_object158090:head# 2024-04-23T16:25:47.990 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.990 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a8a4f40:::benchmark_data_smithi188_89276_object119879:head# 2024-04-23T16:25:47.990 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.990 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a8a5ee2:::benchmark_data_smithi188_89276_object121125:head# 2024-04-23T16:25:47.990 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.990 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a8bf98e:::benchmark_data_smithi188_89276_object84111:head# 2024-04-23T16:25:47.990 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.991 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a8c13d7:::benchmark_data_smithi188_89276_object92056:head# 2024-04-23T16:25:47.991 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.991 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a8c4c36:::benchmark_data_smithi188_89276_object138064:head# 2024-04-23T16:25:47.991 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.991 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a8c847b:::benchmark_data_smithi188_89276_object166242:head# 2024-04-23T16:25:47.991 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.991 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a8d70e8:::benchmark_data_smithi188_89276_object115694:head# 2024-04-23T16:25:47.991 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.993 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a8e121f:::benchmark_data_smithi188_89276_object45347:head# 2024-04-23T16:25:47.994 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.994 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a8e3084:::benchmark_data_smithi188_89276_object88270:head# 2024-04-23T16:25:47.994 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.994 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a8ec84b:::benchmark_data_smithi188_89276_object98167:head# 2024-04-23T16:25:47.994 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.994 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a8eec30:::benchmark_data_smithi188_89276_object111980:head# 2024-04-23T16:25:47.994 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.994 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a903c66:::benchmark_data_smithi188_89276_object92333:head# 2024-04-23T16:25:47.994 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.994 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a91d0db:::benchmark_data_smithi188_89276_object141868:head# 2024-04-23T16:25:47.994 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.994 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a924290:::benchmark_data_smithi188_89276_object80147:head# 2024-04-23T16:25:47.994 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.995 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a950fdd:::benchmark_data_smithi188_89276_object151542:head# 2024-04-23T16:25:47.995 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.995 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a97532c:::benchmark_data_smithi188_89276_object66971:head# 2024-04-23T16:25:47.995 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.995 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a9787c0:::benchmark_data_smithi188_89276_object134979:head# 2024-04-23T16:25:47.995 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.995 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a9929d9:::benchmark_data_smithi188_89276_object87447:head# 2024-04-23T16:25:47.995 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.995 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a9a369f:::benchmark_data_smithi188_89276_object117034:head# 2024-04-23T16:25:47.995 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.995 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a9aad92:::benchmark_data_smithi188_89276_object71889:head# 2024-04-23T16:25:47.995 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.995 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a9ab5d5:::benchmark_data_smithi188_89276_object103953:head# 2024-04-23T16:25:47.995 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.995 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a9ace43:::benchmark_data_smithi188_89276_object47448:head# 2024-04-23T16:25:47.996 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.996 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a9b6847:::benchmark_data_smithi188_89276_object51427:head# 2024-04-23T16:25:47.996 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.997 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a9bb691:::benchmark_data_smithi188_89276_object158466:head# 2024-04-23T16:25:47.997 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a9bd39b:::benchmark_data_smithi188_89276_object120589:head# 2024-04-23T16:25:47.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a9c8a69:::benchmark_data_smithi188_89276_object130176:head# 2024-04-23T16:25:47.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a9cb7a9:::benchmark_data_smithi188_89276_object101789:head# 2024-04-23T16:25:47.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a9d3b2d:::benchmark_data_smithi188_89276_object74729:head# 2024-04-23T16:25:47.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a9db427:::benchmark_data_smithi188_89276_object83572:head# 2024-04-23T16:25:47.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a9e0420:::benchmark_data_smithi188_89276_object113192:head# 2024-04-23T16:25:47.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a9e4c4c:::benchmark_data_smithi188_89276_object158287:head# 2024-04-23T16:25:47.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.999 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a9ebaaf:::benchmark_data_smithi188_89276_object64776:head# 2024-04-23T16:25:47.999 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.999 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a9faecf:::benchmark_data_smithi188_89276_object50586:head# 2024-04-23T16:25:47.999 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.999 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2a9fb92e:::benchmark_data_smithi188_89276_object62975:head# 2024-04-23T16:25:47.999 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.999 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aa00dc8:::benchmark_data_smithi188_89276_object146937:head# 2024-04-23T16:25:47.999 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:47.999 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aa2318f:::benchmark_data_smithi188_89276_object142513:head# 2024-04-23T16:25:47.999 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.002 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aa269a9:::benchmark_data_smithi188_89276_object164921:head# 2024-04-23T16:25:48.002 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.002 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aa3bc95:::benchmark_data_smithi188_89276_object72878:head# 2024-04-23T16:25:48.002 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.002 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aa3d8c5:::benchmark_data_smithi188_89276_object133077:head# 2024-04-23T16:25:48.002 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.002 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aa57337:::benchmark_data_smithi188_89276_object63938:head# 2024-04-23T16:25:48.003 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.003 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aa5739b:::benchmark_data_smithi188_89276_object146891:head# 2024-04-23T16:25:48.003 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.003 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aa5a367:::benchmark_data_smithi188_89276_object97665:head# 2024-04-23T16:25:48.003 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.003 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aa66ed0:::benchmark_data_smithi188_89276_object109333:head# 2024-04-23T16:25:48.003 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.003 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aa71ef9:::benchmark_data_smithi188_89276_object48901:head# 2024-04-23T16:25:48.003 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.003 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aa73c1f:::benchmark_data_smithi188_89276_object97395:head# 2024-04-23T16:25:48.003 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.003 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aa748b1:::benchmark_data_smithi188_89276_object98960:head# 2024-04-23T16:25:48.003 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.003 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aa76341:::benchmark_data_smithi188_89276_object68915:head# 2024-04-23T16:25:48.004 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.004 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aa9d776:::benchmark_data_smithi188_89276_object117728:head# 2024-04-23T16:25:48.004 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.004 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aab3550:::benchmark_data_smithi188_89276_object166851:head# 2024-04-23T16:25:48.004 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.004 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aac1844:::benchmark_data_smithi188_89276_object49837:head# 2024-04-23T16:25:48.004 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.004 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aacf749:::benchmark_data_smithi188_89276_object59044:head# 2024-04-23T16:25:48.004 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.004 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aad0ddd:::benchmark_data_smithi188_89276_object131286:head# 2024-04-23T16:25:48.004 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.004 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aad3cc2:::benchmark_data_smithi188_89276_object161987:head# 2024-04-23T16:25:48.004 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.005 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aad83fc:::benchmark_data_smithi188_89276_object155722:head# 2024-04-23T16:25:48.005 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.006 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aae2326:::benchmark_data_smithi188_89276_object129144:head# 2024-04-23T16:25:48.006 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.006 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aae40ff:::benchmark_data_smithi188_89276_object73423:head# 2024-04-23T16:25:48.006 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.006 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aae9cc7:::benchmark_data_smithi188_89276_object144613:head# 2024-04-23T16:25:48.006 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.006 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aaeab4d:::benchmark_data_smithi188_89276_object42186:head# 2024-04-23T16:25:48.006 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.007 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aaecdba:::benchmark_data_smithi188_89276_object131707:head# 2024-04-23T16:25:48.007 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.007 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aaf69c8:::benchmark_data_smithi188_89276_object79418:head# 2024-04-23T16:25:48.007 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.007 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aaf7a3f:::benchmark_data_smithi188_89276_object104222:head# 2024-04-23T16:25:48.008 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.008 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab03d4e:::benchmark_data_smithi188_89276_object98101:head# 2024-04-23T16:25:48.008 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.008 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab19a4f:::benchmark_data_smithi188_89276_object69203:head# 2024-04-23T16:25:48.008 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.008 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab1ad0c:::benchmark_data_smithi188_89276_object141658:head# 2024-04-23T16:25:48.008 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.009 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab1d990:::benchmark_data_smithi188_89276_object115297:head# 2024-04-23T16:25:48.009 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.009 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab26302:::benchmark_data_smithi188_89276_object129485:head# 2024-04-23T16:25:48.009 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.009 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab2ca8b:::benchmark_data_smithi188_89276_object129197:head# 2024-04-23T16:25:48.009 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.009 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab3c9e3:::benchmark_data_smithi188_89276_object107179:head# 2024-04-23T16:25:48.009 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.010 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab3ee8d:::benchmark_data_smithi188_89276_object55674:head# 2024-04-23T16:25:48.010 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.010 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab4da05:::benchmark_data_smithi188_89276_object104955:head# 2024-04-23T16:25:48.010 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.010 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab5264c:::benchmark_data_smithi188_89276_object147695:head# 2024-04-23T16:25:48.010 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.014 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab52e24:::benchmark_data_smithi188_89276_object90784:head# 2024-04-23T16:25:48.014 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab5ba6d:::benchmark_data_smithi188_89276_object125344:head# 2024-04-23T16:25:48.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab6adee:::benchmark_data_smithi188_89276_object59393:head# 2024-04-23T16:25:48.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab6d167:::benchmark_data_smithi188_89276_object55476:head# 2024-04-23T16:25:48.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.018 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab70d7b:::benchmark_data_smithi188_89276_object62511:head# 2024-04-23T16:25:48.018 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.018 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab79258:::benchmark_data_smithi188_89276_object153139:head# 2024-04-23T16:25:48.019 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.019 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab7e5ca:::benchmark_data_smithi188_89276_object131468:head# 2024-04-23T16:25:48.019 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab885e5:::benchmark_data_smithi188_89276_object50226:head# 2024-04-23T16:25:48.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab8b32a:::benchmark_data_smithi188_89276_object123001:head# 2024-04-23T16:25:48.021 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab8c333:::benchmark_data_smithi188_89276_object115405:head# 2024-04-23T16:25:48.021 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab8fa5f:::benchmark_data_smithi188_89276_object97548:head# 2024-04-23T16:25:48.021 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab90c89:::benchmark_data_smithi188_89276_object70832:head# 2024-04-23T16:25:48.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.032 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab98aaf:::benchmark_data_smithi188_89276_object133288:head# 2024-04-23T16:25:48.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.032 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ab9c354:::benchmark_data_smithi188_89276_object129363:head# 2024-04-23T16:25:48.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.032 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aba5af4:::benchmark_data_smithi188_89276_object160476:head# 2024-04-23T16:25:48.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.033 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aba697c:::benchmark_data_smithi188_89276_object67836:head# 2024-04-23T16:25:48.033 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.033 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2abb7953:::benchmark_data_smithi188_89276_object145584:head# 2024-04-23T16:25:48.033 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.033 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2abbafa6:::benchmark_data_smithi188_89276_object114428:head# 2024-04-23T16:25:48.033 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.034 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2abbc85b:::benchmark_data_smithi188_89276_object139515:head# 2024-04-23T16:25:48.034 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.034 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2abcdf41:::benchmark_data_smithi188_89276_object148383:head# 2024-04-23T16:25:48.034 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.034 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2abda3b3:::benchmark_data_smithi188_89276_object82582:head# 2024-04-23T16:25:48.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2abda7eb:::benchmark_data_smithi188_89276_object111620:head# 2024-04-23T16:25:48.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2abde831:::benchmark_data_smithi188_89276_object91723:head# 2024-04-23T16:25:48.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.036 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2abe541e:::benchmark_data_smithi188_89276_object47355:head# 2024-04-23T16:25:48.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.036 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2abe6095:::benchmark_data_smithi188_89276_object146821:head# 2024-04-23T16:25:48.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.036 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2abec759:::benchmark_data_smithi188_89276_object108885:head# 2024-04-23T16:25:48.037 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.037 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2abf551c:::benchmark_data_smithi188_89276_object44743:head# 2024-04-23T16:25:48.037 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.037 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2abf8249:::benchmark_data_smithi188_89276_object113406:head# 2024-04-23T16:25:48.037 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.037 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2abfb377:::benchmark_data_smithi188_89276_object66547:head# 2024-04-23T16:25:48.037 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.038 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac0115e:::benchmark_data_smithi188_89276_object142286:head# 2024-04-23T16:25:48.038 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.038 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac05915:::benchmark_data_smithi188_89276_object88467:head# 2024-04-23T16:25:48.038 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.038 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac06a50:::benchmark_data_smithi188_89276_object133840:head# 2024-04-23T16:25:48.038 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.039 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac180ad:::benchmark_data_smithi188_89276_object149976:head# 2024-04-23T16:25:48.039 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.039 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac197e3:::benchmark_data_smithi188_89276_object111240:head# 2024-04-23T16:25:48.039 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.039 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac21d03:::benchmark_data_smithi188_89276_object139139:head# 2024-04-23T16:25:48.040 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.040 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac28690:::benchmark_data_smithi188_89276_object140785:head# 2024-04-23T16:25:48.040 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.040 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac32365:::benchmark_data_smithi188_89276_object165956:head# 2024-04-23T16:25:48.040 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.040 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac3ed7f:::benchmark_data_smithi188_89276_object61804:head# 2024-04-23T16:25:48.041 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.041 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac46c80:::benchmark_data_smithi188_89276_object151136:head# 2024-04-23T16:25:48.041 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.041 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac4c3a1:::benchmark_data_smithi188_89276_object47738:head# 2024-04-23T16:25:48.041 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.042 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac4d906:::benchmark_data_smithi188_89276_object101956:head# 2024-04-23T16:25:48.042 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.042 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac50ccb:::benchmark_data_smithi188_89276_object159367:head# 2024-04-23T16:25:48.042 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.042 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac5512a:::benchmark_data_smithi188_89276_object135340:head# 2024-04-23T16:25:48.042 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.042 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac6de80:::benchmark_data_smithi188_89276_object151598:head# 2024-04-23T16:25:48.043 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.043 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac7181a:::benchmark_data_smithi188_89276_object168335:head# 2024-04-23T16:25:48.043 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.043 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac762b7:::benchmark_data_smithi188_89276_object148792:head# 2024-04-23T16:25:48.043 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.043 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac77ce7:::benchmark_data_smithi188_89276_object136774:head# 2024-04-23T16:25:48.044 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.044 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac7a027:::benchmark_data_smithi188_89276_object51143:head# 2024-04-23T16:25:48.044 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.044 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac81182:::benchmark_data_smithi188_89276_object102601:head# 2024-04-23T16:25:48.044 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.044 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac848be:::benchmark_data_smithi188_89276_object167975:head# 2024-04-23T16:25:48.045 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.052 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:25:48.071 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ac9f927:::benchmark_data_smithi188_89276_object130296:head# 2024-04-23T16:25:48.071 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.071 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2acac4f9:::benchmark_data_smithi188_89276_object59793:head# 2024-04-23T16:25:48.072 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.072 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2acd4e39:::benchmark_data_smithi188_89276_object93155:head# 2024-04-23T16:25:48.072 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.072 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2acda7fc:::benchmark_data_smithi188_89276_object73609:head# 2024-04-23T16:25:48.072 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.072 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2acedb4f:::benchmark_data_smithi188_89276_object50650:head# 2024-04-23T16:25:48.072 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.073 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2acf0f75:::benchmark_data_smithi188_89276_object97772:head# 2024-04-23T16:25:48.073 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.073 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2acf7cd7:::benchmark_data_smithi188_89276_object93146:head# 2024-04-23T16:25:48.073 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.073 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2acfcc80:::benchmark_data_smithi188_89276_object97388:head# 2024-04-23T16:25:48.073 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2acfea90:::benchmark_data_smithi188_89276_object167515:head# 2024-04-23T16:25:48.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ad0a63a:::benchmark_data_smithi188_89276_object146585:head# 2024-04-23T16:25:48.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ad187b2:::benchmark_data_smithi188_89276_object156658:head# 2024-04-23T16:25:48.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ad1b784:::benchmark_data_smithi188_89276_object118081:head# 2024-04-23T16:25:48.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ad224b4:::benchmark_data_smithi188_89276_object163123:head# 2024-04-23T16:25:48.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ad351d3:::benchmark_data_smithi188_89276_object126821:head# 2024-04-23T16:25:48.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ad48089:::benchmark_data_smithi188_89276_object107541:head# 2024-04-23T16:25:48.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ad4fdb9:::benchmark_data_smithi188_89276_object118180:head# 2024-04-23T16:25:48.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ad533d9:::benchmark_data_smithi188_89276_object157684:head# 2024-04-23T16:25:48.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ad5d0ac:::benchmark_data_smithi188_89276_object124880:head# 2024-04-23T16:25:48.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ad5dc50:::benchmark_data_smithi188_89276_object120099:head# 2024-04-23T16:25:48.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ad6eb64:::benchmark_data_smithi188_89276_object73167:head# 2024-04-23T16:25:48.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.078 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ad714ee:::benchmark_data_smithi188_89276_object42238:head# 2024-04-23T16:25:48.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.078 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ad732fe:::benchmark_data_smithi188_89276_object46767:head# 2024-04-23T16:25:48.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.079 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ad81f95:::benchmark_data_smithi188_89276_object93690:head# 2024-04-23T16:25:48.079 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.079 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ad8ebd7:::benchmark_data_smithi188_89276_object69430:head# 2024-04-23T16:25:48.079 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.079 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ad9ffc3:::benchmark_data_smithi188_89276_object105523:head# 2024-04-23T16:25:48.079 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.079 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ada7138:::benchmark_data_smithi188_89276_object62982:head# 2024-04-23T16:25:48.080 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.080 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2adaa44b:::benchmark_data_smithi188_89276_object138202:head# 2024-04-23T16:25:48.080 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.080 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2adab43f:::benchmark_data_smithi188_89276_object66657:head# 2024-04-23T16:25:48.081 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.081 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2adc3a5c:::benchmark_data_smithi188_89276_object41995:head# 2024-04-23T16:25:48.081 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.081 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2adc4d13:::benchmark_data_smithi188_89276_object65279:head# 2024-04-23T16:25:48.081 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.081 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2adc81d6:::benchmark_data_smithi188_89276_object147950:head# 2024-04-23T16:25:48.082 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.082 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2adcde53:::benchmark_data_smithi188_89276_object75360:head# 2024-04-23T16:25:48.082 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.082 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2add284c:::benchmark_data_smithi188_89276_object110729:head# 2024-04-23T16:25:48.082 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.083 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2add532c:::benchmark_data_smithi188_89276_object129976:head# 2024-04-23T16:25:48.083 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.083 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2addb1e1:::benchmark_data_smithi188_89276_object166886:head# 2024-04-23T16:25:48.083 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.083 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2adeb129:::benchmark_data_smithi188_89276_object132204:head# 2024-04-23T16:25:48.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2adef023:::benchmark_data_smithi188_89276_object149528:head# 2024-04-23T16:25:48.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ae1019b:::benchmark_data_smithi188_89276_object84335:head# 2024-04-23T16:25:48.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ae1e807:::benchmark_data_smithi188_89276_object119635:head# 2024-04-23T16:25:48.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ae2070c:::benchmark_data_smithi188_89276_object101365:head# 2024-04-23T16:25:48.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ae2255b:::benchmark_data_smithi188_89276_object128768:head# 2024-04-23T16:25:48.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ae29898:::benchmark_data_smithi188_89276_object113286:head# 2024-04-23T16:25:48.086 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.086 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ae2d743:::benchmark_data_smithi188_89276_object141038:head# 2024-04-23T16:25:48.086 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.086 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ae3271b:::benchmark_data_smithi188_89276_object78565:head# 2024-04-23T16:25:48.086 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.086 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ae3c90a:::benchmark_data_smithi188_89276_object98796:head# 2024-04-23T16:25:48.086 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.087 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ae4017e:::benchmark_data_smithi188_89276_object116473:head# 2024-04-23T16:25:48.087 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.087 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ae57b1b:::benchmark_data_smithi188_89276_object131581:head# 2024-04-23T16:25:48.087 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.087 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ae6c732:::benchmark_data_smithi188_89276_object63284:head# 2024-04-23T16:25:48.087 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.088 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ae721a2:::benchmark_data_smithi188_89276_object96844:head# 2024-04-23T16:25:48.088 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.088 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ae7bbd9:::benchmark_data_smithi188_89276_object62123:head# 2024-04-23T16:25:48.088 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.088 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ae80b78:::benchmark_data_smithi188_89276_object54889:head# 2024-04-23T16:25:48.088 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.089 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ae992dd:::benchmark_data_smithi188_89276_object125940:head# 2024-04-23T16:25:48.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.089 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ae9b4c6:::benchmark_data_smithi188_89276_object51559:head# 2024-04-23T16:25:48.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.089 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aea7e2a:::benchmark_data_smithi188_89276_object166590:head# 2024-04-23T16:25:48.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aead698:::benchmark_data_smithi188_89276_object54278:head# 2024-04-23T16:25:48.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aeb80ee:::benchmark_data_smithi188_89276_object54728:head# 2024-04-23T16:25:48.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aebe4a8:::benchmark_data_smithi188_89276_object90639:head# 2024-04-23T16:25:48.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aecba26:::benchmark_data_smithi188_89276_object164083:head# 2024-04-23T16:25:48.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aed062a:::benchmark_data_smithi188_89276_object132534:head# 2024-04-23T16:25:48.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aed41bf:::benchmark_data_smithi188_89276_object135535:head# 2024-04-23T16:25:48.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aed58af:::benchmark_data_smithi188_89276_object161555:head# 2024-04-23T16:25:48.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aeddf76:::benchmark_data_smithi188_89276_object88508:head# 2024-04-23T16:25:48.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aee3bcc:::benchmark_data_smithi188_89276_object117088:head# 2024-04-23T16:25:48.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aee7380:::benchmark_data_smithi188_89276_object70131:head# 2024-04-23T16:25:48.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aef1f1f:::benchmark_data_smithi188_89276_object150901:head# 2024-04-23T16:25:48.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2aef66e9:::benchmark_data_smithi188_89276_object102986:head# 2024-04-23T16:25:48.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2af050d4:::benchmark_data_smithi188_89276_object137913:head# 2024-04-23T16:25:48.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2af055c4:::benchmark_data_smithi188_89276_object63361:head# 2024-04-23T16:25:48.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2af0cae2:::benchmark_data_smithi188_89276_object75047:head# 2024-04-23T16:25:48.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2af51c02:::benchmark_data_smithi188_89276_object62371:head# 2024-04-23T16:25:48.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2af5223c:::benchmark_data_smithi188_89276_object71230:head# 2024-04-23T16:25:48.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2af54de4:::benchmark_data_smithi188_89276_object108435:head# 2024-04-23T16:25:48.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2af5a5eb:::benchmark_data_smithi188_89276_object85660:head# 2024-04-23T16:25:48.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2af60824:::benchmark_data_smithi188_89276_object127702:head# 2024-04-23T16:25:48.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2af70b30:::benchmark_data_smithi188_89276_object50417:head# 2024-04-23T16:25:48.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2af7df66:::benchmark_data_smithi188_89276_object160355:head# 2024-04-23T16:25:48.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2af925ca:::benchmark_data_smithi188_89276_object98974:head# 2024-04-23T16:25:48.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2af9593d:::benchmark_data_smithi188_89276_object84249:head# 2024-04-23T16:25:48.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2af9a9e2:::benchmark_data_smithi188_89276_object157385:head# 2024-04-23T16:25:48.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2af9e212:::benchmark_data_smithi188_89276_object109084:head# 2024-04-23T16:25:48.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2afa441f:::benchmark_data_smithi188_89276_object79114:head# 2024-04-23T16:25:48.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2afa6af8:::benchmark_data_smithi188_89276_object91590:head# 2024-04-23T16:25:48.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2afa6eae:::benchmark_data_smithi188_89276_object85304:head# 2024-04-23T16:25:48.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2afa9821:::benchmark_data_smithi188_89276_object118303:head# 2024-04-23T16:25:48.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2afa9b31:::benchmark_data_smithi188_89276_object40634:head# 2024-04-23T16:25:48.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2afbad66:::benchmark_data_smithi188_89276_object47656:head# 2024-04-23T16:25:48.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2afbca93:::benchmark_data_smithi188_89276_object161133:head# 2024-04-23T16:25:48.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2afc3782:::benchmark_data_smithi188_89276_object70638:head# 2024-04-23T16:25:48.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2afe041b:::benchmark_data_smithi188_89276_object126617:head# 2024-04-23T16:25:48.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2afe74e4:::benchmark_data_smithi188_89276_object141148:head# 2024-04-23T16:25:48.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2afe8177:::benchmark_data_smithi188_89276_object122869:head# 2024-04-23T16:25:48.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2afeeb96:::benchmark_data_smithi188_89276_object74493:head# 2024-04-23T16:25:48.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2affbd1b:::benchmark_data_smithi188_89276_object141835:head# 2024-04-23T16:25:48.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2affe1c3:::benchmark_data_smithi188_89276_object121502:head# 2024-04-23T16:25:48.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b01f08b:::benchmark_data_smithi188_89276_object87258:head# 2024-04-23T16:25:48.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b0227ba:::benchmark_data_smithi188_89276_object64591:head# 2024-04-23T16:25:48.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b023fe8:::benchmark_data_smithi188_89276_object75412:head# 2024-04-23T16:25:48.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b02e13e:::benchmark_data_smithi188_89276_object97970:head# 2024-04-23T16:25:48.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b030214:::benchmark_data_smithi188_89276_object111162:head# 2024-04-23T16:25:48.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b0405ec:::benchmark_data_smithi188_89276_object148352:head# 2024-04-23T16:25:48.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b04d8cc:::benchmark_data_smithi188_89276_object147202:head# 2024-04-23T16:25:48.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b04d8ce:::benchmark_data_smithi188_89276_object45736:head# 2024-04-23T16:25:48.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b0534fd:::benchmark_data_smithi188_89276_object93824:head# 2024-04-23T16:25:48.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b05e2f0:::benchmark_data_smithi188_89276_object64755:head# 2024-04-23T16:25:48.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b066b40:::benchmark_data_smithi188_89276_object160205:head# 2024-04-23T16:25:48.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b0673b8:::benchmark_data_smithi188_89276_object73709:head# 2024-04-23T16:25:48.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b070833:::benchmark_data_smithi188_89276_object157488:head# 2024-04-23T16:25:48.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b072445:::benchmark_data_smithi188_89276_object43014:head# 2024-04-23T16:25:48.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b07613f:::benchmark_data_smithi188_89276_object58342:head# 2024-04-23T16:25:48.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b077ac1:::benchmark_data_smithi188_89276_object66411:head# 2024-04-23T16:25:48.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b081dec:::benchmark_data_smithi188_89276_object87874:head# 2024-04-23T16:25:48.102 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b0840b5:::benchmark_data_smithi188_89276_object115652:head# 2024-04-23T16:25:48.102 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b0905e5:::benchmark_data_smithi188_89276_object83228:head# 2024-04-23T16:25:48.102 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b09ab32:::benchmark_data_smithi188_89276_object107068:head# 2024-04-23T16:25:48.102 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b0a2637:::benchmark_data_smithi188_89276_object49196:head# 2024-04-23T16:25:48.102 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b0a5816:::benchmark_data_smithi188_89276_object150607:head# 2024-04-23T16:25:48.102 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b0aa69f:::benchmark_data_smithi188_89276_object96271:head# 2024-04-23T16:25:48.103 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.104 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:48.118 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:25:48.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:48.128 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:25:48.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:47 smithi148 ceph-mon[44520]: pgmap v740: 17 pgs: 17 active+clean; 5.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.50k op/s 2024-04-23T16:25:48.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b0e3ff7:::benchmark_data_smithi188_89276_object46602:head# 2024-04-23T16:25:48.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b0e4efc:::benchmark_data_smithi188_89276_object153475:head# 2024-04-23T16:25:48.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b0ee9f0:::benchmark_data_smithi188_89276_object145533:head# 2024-04-23T16:25:48.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b0f781f:::benchmark_data_smithi188_89276_object120649:head# 2024-04-23T16:25:48.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b0f9b1a:::benchmark_data_smithi188_89276_object87747:head# 2024-04-23T16:25:48.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b0fe87a:::benchmark_data_smithi188_89276_object156203:head# 2024-04-23T16:25:48.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b1031c8:::benchmark_data_smithi188_89276_object73817:head# 2024-04-23T16:25:48.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b10e0d5:::benchmark_data_smithi188_89276_object70908:head# 2024-04-23T16:25:48.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b130acc:::benchmark_data_smithi188_89276_object105143:head# 2024-04-23T16:25:48.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b137e52:::benchmark_data_smithi188_89276_object118453:head# 2024-04-23T16:25:48.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b140f2a:::benchmark_data_smithi188_89276_object109314:head# 2024-04-23T16:25:48.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b150d3b:::benchmark_data_smithi188_89276_object49780:head# 2024-04-23T16:25:48.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b15f095:::benchmark_data_smithi188_89276_object84073:head# 2024-04-23T16:25:48.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b16d426:::benchmark_data_smithi188_89276_object143033:head# 2024-04-23T16:25:48.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b16e880:::benchmark_data_smithi188_89276_object141697:head# 2024-04-23T16:25:48.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b17165f:::benchmark_data_smithi188_89276_object82191:head# 2024-04-23T16:25:48.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b17ac33:::benchmark_data_smithi188_89276_object89604:head# 2024-04-23T16:25:48.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b18f75b:::benchmark_data_smithi188_89276_object94767:head# 2024-04-23T16:25:48.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b19ee2b:::benchmark_data_smithi188_89276_object107577:head# 2024-04-23T16:25:48.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b1ae24a:::benchmark_data_smithi188_89276_object144970:head# 2024-04-23T16:25:48.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b1bdd5d:::benchmark_data_smithi188_89276_object79306:head# 2024-04-23T16:25:48.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b1be904:::benchmark_data_smithi188_89276_object120323:head# 2024-04-23T16:25:48.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b1c026b:::benchmark_data_smithi188_89276_object161998:head# 2024-04-23T16:25:48.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b1cdf94:::benchmark_data_smithi188_89276_object80617:head# 2024-04-23T16:25:48.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b1cefe5:::benchmark_data_smithi188_89276_object92507:head# 2024-04-23T16:25:48.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b1f66b4:::benchmark_data_smithi188_89276_object115335:head# 2024-04-23T16:25:48.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b229ebd:::benchmark_data_smithi188_89276_object131546:head# 2024-04-23T16:25:48.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b22c1ca:::benchmark_data_smithi188_89276_object47717:head# 2024-04-23T16:25:48.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b23b467:::benchmark_data_smithi188_89276_object132505:head# 2024-04-23T16:25:48.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b23be4f:::benchmark_data_smithi188_89276_object166497:head# 2024-04-23T16:25:48.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b245199:::benchmark_data_smithi188_89276_object138348:head# 2024-04-23T16:25:48.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b2480c5:::benchmark_data_smithi188_89276_object87501:head# 2024-04-23T16:25:48.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b24cb29:::benchmark_data_smithi188_89276_object68534:head# 2024-04-23T16:25:48.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b254799:::benchmark_data_smithi188_89276_object56459:head# 2024-04-23T16:25:48.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b25d74f:::benchmark_data_smithi188_89276_object63339:head# 2024-04-23T16:25:48.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b25e2b5:::benchmark_data_smithi188_89276_object61764:head# 2024-04-23T16:25:48.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b25fa8f:::benchmark_data_smithi188_89276_object79645:head# 2024-04-23T16:25:48.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b26534d:::benchmark_data_smithi188_89276_object85030:head# 2024-04-23T16:25:48.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b267350:::benchmark_data_smithi188_89276_object139047:head# 2024-04-23T16:25:48.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b267bdb:::benchmark_data_smithi188_89276_object71012:head# 2024-04-23T16:25:48.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b2771c3:::benchmark_data_smithi188_89276_object55231:head# 2024-04-23T16:25:48.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b286437:::benchmark_data_smithi188_89276_object126879:head# 2024-04-23T16:25:48.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b291f3e:::benchmark_data_smithi188_89276_object83283:head# 2024-04-23T16:25:48.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b2931a3:::benchmark_data_smithi188_89276_object54588:head# 2024-04-23T16:25:48.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b2a75e4:::benchmark_data_smithi188_89276_object46805:head# 2024-04-23T16:25:48.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b2b4e32:::benchmark_data_smithi188_89276_object84697:head# 2024-04-23T16:25:48.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b2b5b45:::benchmark_data_smithi188_89276_object73144:head# 2024-04-23T16:25:48.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b2b734b:::benchmark_data_smithi188_89276_object145668:head# 2024-04-23T16:25:48.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b2bec42:::benchmark_data_smithi188_89276_object150862:head# 2024-04-23T16:25:48.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b2e64ee:::benchmark_data_smithi188_89276_object42407:head# 2024-04-23T16:25:48.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b2e697d:::benchmark_data_smithi188_89276_object151442:head# 2024-04-23T16:25:48.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b2ea048:::benchmark_data_smithi188_89276_object101267:head# 2024-04-23T16:25:48.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b2ee665:::benchmark_data_smithi188_89276_object152639:head# 2024-04-23T16:25:48.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b2ee843:::benchmark_data_smithi188_89276_object57974:head# 2024-04-23T16:25:48.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b2f1eb0:::benchmark_data_smithi188_89276_object142867:head# 2024-04-23T16:25:48.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b2f30b3:::benchmark_data_smithi188_89276_object111262:head# 2024-04-23T16:25:48.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b30b109:::benchmark_data_smithi188_89276_object131246:head# 2024-04-23T16:25:48.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b30fc48:::benchmark_data_smithi188_89276_object92466:head# 2024-04-23T16:25:48.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b3222cf:::benchmark_data_smithi188_89276_object89156:head# 2024-04-23T16:25:48.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b32b46f:::benchmark_data_smithi188_89276_object143830:head# 2024-04-23T16:25:48.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b32c215:::benchmark_data_smithi188_89276_object130919:head# 2024-04-23T16:25:48.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b32d331:::benchmark_data_smithi188_89276_object119315:head# 2024-04-23T16:25:48.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b32e138:::benchmark_data_smithi188_89276_object87427:head# 2024-04-23T16:25:48.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b334ec2:::benchmark_data_smithi188_89276_object155729:head# 2024-04-23T16:25:48.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b346a29:::benchmark_data_smithi188_89276_object54990:head# 2024-04-23T16:25:48.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b353f1f:::benchmark_data_smithi188_89276_object116477:head# 2024-04-23T16:25:48.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b360a6c:::benchmark_data_smithi188_89276_object107835:head# 2024-04-23T16:25:48.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b361188:::benchmark_data_smithi188_89276_object163301:head# 2024-04-23T16:25:48.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b36d0c6:::benchmark_data_smithi188_89276_object89815:head# 2024-04-23T16:25:48.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b36fd96:::benchmark_data_smithi188_89276_object62110:head# 2024-04-23T16:25:48.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b375475:::benchmark_data_smithi188_89276_object99408:head# 2024-04-23T16:25:48.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b375774:::benchmark_data_smithi188_89276_object43858:head# 2024-04-23T16:25:48.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.155 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b3784f5:::benchmark_data_smithi188_89276_object83714:head# 2024-04-23T16:25:48.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.155 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b37c776:::benchmark_data_smithi188_89276_object52193:head# 2024-04-23T16:25:48.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.155 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b37dcb9:::benchmark_data_smithi188_89276_object86716:head# 2024-04-23T16:25:48.156 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.156 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b38afac:::benchmark_data_smithi188_89276_object144874:head# 2024-04-23T16:25:48.156 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.156 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b38f5d5:::benchmark_data_smithi188_89276_object165025:head# 2024-04-23T16:25:48.156 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.156 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b39162d:::benchmark_data_smithi188_89276_object128022:head# 2024-04-23T16:25:48.156 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.157 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b3955e5:::benchmark_data_smithi188_89276_object62582:head# 2024-04-23T16:25:48.157 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.157 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b3ab9c2:::benchmark_data_smithi188_89276_object119929:head# 2024-04-23T16:25:48.157 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.157 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b3b0299:::benchmark_data_smithi188_89276_object101945:head# 2024-04-23T16:25:48.157 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.158 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b3b8990:::benchmark_data_smithi188_89276_object61901:head# 2024-04-23T16:25:48.158 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.158 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b3cf3b6:::benchmark_data_smithi188_89276_object107933:head# 2024-04-23T16:25:48.158 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.158 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b3d04cd:::benchmark_data_smithi188_89276_object123142:head# 2024-04-23T16:25:48.158 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.159 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b3d184f:::benchmark_data_smithi188_89276_object141641:head# 2024-04-23T16:25:48.159 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.159 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b3deae1:::benchmark_data_smithi188_89276_object93977:head# 2024-04-23T16:25:48.159 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.159 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b3e150d:::benchmark_data_smithi188_89276_object66645:head# 2024-04-23T16:25:48.159 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.159 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b3e63c1:::benchmark_data_smithi188_89276_object152541:head# 2024-04-23T16:25:48.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b3e68c3:::benchmark_data_smithi188_89276_object117514:head# 2024-04-23T16:25:48.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b3f2bac:::benchmark_data_smithi188_89276_object110821:head# 2024-04-23T16:25:48.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b3f9b83:::benchmark_data_smithi188_89276_object60910:head# 2024-04-23T16:25:48.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b3fc22b:::benchmark_data_smithi188_89276_object97268:head# 2024-04-23T16:25:48.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b3fc7c0:::benchmark_data_smithi188_89276_object115468:head# 2024-04-23T16:25:48.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b40149e:::benchmark_data_smithi188_89276_object40429:head# 2024-04-23T16:25:48.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b402e53:::benchmark_data_smithi188_89276_object58998:head# 2024-04-23T16:25:48.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b405de0:::benchmark_data_smithi188_89276_object125239:head# 2024-04-23T16:25:48.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b41a248:::benchmark_data_smithi188_89276_object90199:head# 2024-04-23T16:25:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b43306c:::benchmark_data_smithi188_89276_object108932:head# 2024-04-23T16:25:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b4385a3:::benchmark_data_smithi188_89276_object132057:head# 2024-04-23T16:25:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b43f96c:::benchmark_data_smithi188_89276_object57106:head# 2024-04-23T16:25:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b45a538:::benchmark_data_smithi188_89276_object151596:head# 2024-04-23T16:25:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b463291:::benchmark_data_smithi188_89276_object47784:head# 2024-04-23T16:25:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b479e3f:::benchmark_data_smithi188_89276_object55192:head# 2024-04-23T16:25:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b48307c:::benchmark_data_smithi188_89276_object61164:head# 2024-04-23T16:25:48.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b492fa2:::benchmark_data_smithi188_89276_object117902:head# 2024-04-23T16:25:48.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b49576c:::benchmark_data_smithi188_89276_object156733:head# 2024-04-23T16:25:48.166 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.166 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b4a769a:::benchmark_data_smithi188_89276_object76526:head# 2024-04-23T16:25:48.166 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.166 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b4ad1eb:::benchmark_data_smithi188_89276_object100220:head# 2024-04-23T16:25:48.166 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.166 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b4b13a9:::benchmark_data_smithi188_89276_object156455:head# 2024-04-23T16:25:48.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b4df908:::benchmark_data_smithi188_89276_object72077:head# 2024-04-23T16:25:48.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b4e2d7e:::benchmark_data_smithi188_89276_object163268:head# 2024-04-23T16:25:48.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b4e9021:::benchmark_data_smithi188_89276_object88359:head# 2024-04-23T16:25:48.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b4f3a8f:::benchmark_data_smithi188_89276_object94115:head# 2024-04-23T16:25:48.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b4f60ff:::benchmark_data_smithi188_89276_object90612:head# 2024-04-23T16:25:48.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b505ef4:::benchmark_data_smithi188_89276_object165736:head# 2024-04-23T16:25:48.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b50b1db:::benchmark_data_smithi188_89276_object103181:head# 2024-04-23T16:25:48.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b542480:::benchmark_data_smithi188_89276_object123834:head# 2024-04-23T16:25:48.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b5500f0:::benchmark_data_smithi188_89276_object130948:head# 2024-04-23T16:25:48.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b554981:::benchmark_data_smithi188_89276_object165147:head# 2024-04-23T16:25:48.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b560103:::benchmark_data_smithi188_89276_object143294:head# 2024-04-23T16:25:48.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b564a42:::benchmark_data_smithi188_89276_object75772:head# 2024-04-23T16:25:48.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b56901e:::benchmark_data_smithi188_89276_object126372:head# 2024-04-23T16:25:48.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b569479:::benchmark_data_smithi188_89276_object77828:head# 2024-04-23T16:25:48.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b56b12c:::benchmark_data_smithi188_89276_object60240:head# 2024-04-23T16:25:48.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b56d3b4:::benchmark_data_smithi188_89276_object66058:head# 2024-04-23T16:25:48.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b576e42:::benchmark_data_smithi188_89276_object62533:head# 2024-04-23T16:25:48.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b57785f:::benchmark_data_smithi188_89276_object146999:head# 2024-04-23T16:25:48.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b577eeb:::benchmark_data_smithi188_89276_object42395:head# 2024-04-23T16:25:48.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b584e4f:::benchmark_data_smithi188_89276_object55759:head# 2024-04-23T16:25:48.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b597fe9:::benchmark_data_smithi188_89276_object147749:head# 2024-04-23T16:25:48.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b5a1500:::benchmark_data_smithi188_89276_object97674:head# 2024-04-23T16:25:48.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b5a5412:::benchmark_data_smithi188_89276_object166957:head# 2024-04-23T16:25:48.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b5a7e2e:::benchmark_data_smithi188_89276_object60211:head# 2024-04-23T16:25:48.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b5ab57b:::benchmark_data_smithi188_89276_object96739:head# 2024-04-23T16:25:48.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b5abe55:::benchmark_data_smithi188_89276_object115990:head# 2024-04-23T16:25:48.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b5b04f6:::benchmark_data_smithi188_89276_object89621:head# 2024-04-23T16:25:48.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b5b596f:::benchmark_data_smithi188_89276_object120929:head# 2024-04-23T16:25:48.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b5b6f95:::benchmark_data_smithi188_89276_object106881:head# 2024-04-23T16:25:48.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b5bb1a9:::benchmark_data_smithi188_89276_object95846:head# 2024-04-23T16:25:48.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b5c15b2:::benchmark_data_smithi188_89276_object123762:head# 2024-04-23T16:25:48.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b5c2c3e:::benchmark_data_smithi188_89276_object92104:head# 2024-04-23T16:25:48.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b5cf8e6:::benchmark_data_smithi188_89276_object84941:head# 2024-04-23T16:25:48.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b5d2106:::benchmark_data_smithi188_89276_object141833:head# 2024-04-23T16:25:48.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b5e1411:::benchmark_data_smithi188_89276_object130355:head# 2024-04-23T16:25:48.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b5f0eea:::benchmark_data_smithi188_89276_object48175:head# 2024-04-23T16:25:48.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b5f6301:::benchmark_data_smithi188_89276_object63885:head# 2024-04-23T16:25:48.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b5fba2a:::benchmark_data_smithi188_89276_object46935:head# 2024-04-23T16:25:48.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b603c41:::benchmark_data_smithi188_89276_object137291:head# 2024-04-23T16:25:48.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b6044e0:::benchmark_data_smithi188_89276_object156584:head# 2024-04-23T16:25:48.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b606df0:::benchmark_data_smithi188_89276_object85615:head# 2024-04-23T16:25:48.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b6193e8:::benchmark_data_smithi188_89276_object66624:head# 2024-04-23T16:25:48.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b61afb6:::benchmark_data_smithi188_89276_object57818:head# 2024-04-23T16:25:48.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b61c898:::benchmark_data_smithi188_89276_object76736:head# 2024-04-23T16:25:48.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b6342b4:::benchmark_data_smithi188_89276_object164670:head# 2024-04-23T16:25:48.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b642027:::benchmark_data_smithi188_89276_object164266:head# 2024-04-23T16:25:48.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b647c63:::benchmark_data_smithi188_89276_object144723:head# 2024-04-23T16:25:48.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b651bc4:::benchmark_data_smithi188_89276_object102464:head# 2024-04-23T16:25:48.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b65ceed:::benchmark_data_smithi188_89276_object137423:head# 2024-04-23T16:25:48.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b66a95e:::benchmark_data_smithi188_89276_object129895:head# 2024-04-23T16:25:48.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b66b6b8:::benchmark_data_smithi188_89276_object80231:head# 2024-04-23T16:25:48.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b67760a:::benchmark_data_smithi188_89276_object57530:head# 2024-04-23T16:25:48.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b67f5bc:::benchmark_data_smithi188_89276_object70761:head# 2024-04-23T16:25:48.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b684dc7:::benchmark_data_smithi188_89276_object110533:head# 2024-04-23T16:25:48.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b69982a:::benchmark_data_smithi188_89276_object50930:head# 2024-04-23T16:25:48.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b6a070a:::benchmark_data_smithi188_89276_object67811:head# 2024-04-23T16:25:48.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b6b1c14:::benchmark_data_smithi188_89276_object79179:head# 2024-04-23T16:25:48.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b6bba6a:::benchmark_data_smithi188_89276_object94582:head# 2024-04-23T16:25:48.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b6c52ac:::benchmark_data_smithi188_89276_object164050:head# 2024-04-23T16:25:48.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b6c961f:::benchmark_data_smithi188_89276_object139424:head# 2024-04-23T16:25:48.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b6d7a42:::benchmark_data_smithi188_89276_object45153:head# 2024-04-23T16:25:48.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b6ea453:::benchmark_data_smithi188_89276_object135622:head# 2024-04-23T16:25:48.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b6ef1ca:::benchmark_data_smithi188_89276_object86061:head# 2024-04-23T16:25:48.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b6f69e9:::benchmark_data_smithi188_89276_object51456:head# 2024-04-23T16:25:48.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b6fed36:::benchmark_data_smithi188_89276_object144094:head# 2024-04-23T16:25:48.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b706805:::benchmark_data_smithi188_89276_object82830:head# 2024-04-23T16:25:48.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b706c0f:::benchmark_data_smithi188_89276_object114717:head# 2024-04-23T16:25:48.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b70b7ff:::benchmark_data_smithi188_89276_object134288:head# 2024-04-23T16:25:48.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b70f6ea:::benchmark_data_smithi188_89276_object143412:head# 2024-04-23T16:25:48.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b71149c:::benchmark_data_smithi188_89276_object111388:head# 2024-04-23T16:25:48.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b71e97d:::benchmark_data_smithi188_89276_object123274:head# 2024-04-23T16:25:48.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b72523b:::benchmark_data_smithi188_89276_object54730:head# 2024-04-23T16:25:48.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b72c37b:::benchmark_data_smithi188_89276_object52841:head# 2024-04-23T16:25:48.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b7326e1:::benchmark_data_smithi188_89276_object55935:head# 2024-04-23T16:25:48.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b744ba1:::benchmark_data_smithi188_89276_object117474:head# 2024-04-23T16:25:48.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b74582b:::benchmark_data_smithi188_89276_object144061:head# 2024-04-23T16:25:48.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b75347f:::benchmark_data_smithi188_89276_object164861:head# 2024-04-23T16:25:48.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b75771d:::benchmark_data_smithi188_89276_object122938:head# 2024-04-23T16:25:48.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b7627d9:::benchmark_data_smithi188_89276_object144983:head# 2024-04-23T16:25:48.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b763819:::benchmark_data_smithi188_89276_object151597:head# 2024-04-23T16:25:48.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b76ab9e:::benchmark_data_smithi188_89276_object102357:head# 2024-04-23T16:25:48.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b779848:::benchmark_data_smithi188_89276_object73150:head# 2024-04-23T16:25:48.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b77ba4c:::benchmark_data_smithi188_89276_object145114:head# 2024-04-23T16:25:48.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b77c9aa:::benchmark_data_smithi188_89276_object150345:head# 2024-04-23T16:25:48.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b79fae9:::benchmark_data_smithi188_89276_object134199:head# 2024-04-23T16:25:48.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b7a8b6d:::benchmark_data_smithi188_89276_object153220:head# 2024-04-23T16:25:48.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b7a9a96:::benchmark_data_smithi188_89276_object58992:head# 2024-04-23T16:25:48.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b7b1fed:::benchmark_data_smithi188_89276_object110014:head# 2024-04-23T16:25:48.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b7b474e:::benchmark_data_smithi188_89276_object62523:head# 2024-04-23T16:25:48.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b7bbeca:::benchmark_data_smithi188_89276_object121114:head# 2024-04-23T16:25:48.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b7be449:::benchmark_data_smithi188_89276_object42369:head# 2024-04-23T16:25:48.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b7c0e26:::benchmark_data_smithi188_89276_object82474:head# 2024-04-23T16:25:48.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b7c2089:::benchmark_data_smithi188_89276_object99306:head# 2024-04-23T16:25:48.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b7c4ef9:::benchmark_data_smithi188_89276_object56677:head# 2024-04-23T16:25:48.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b7c5cbf:::benchmark_data_smithi188_89276_object163719:head# 2024-04-23T16:25:48.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b7cddc3:::benchmark_data_smithi188_89276_object100617:head# 2024-04-23T16:25:48.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b7e0bf5:::benchmark_data_smithi188_89276_object132489:head# 2024-04-23T16:25:48.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b7e2dad:::benchmark_data_smithi188_89276_object149318:head# 2024-04-23T16:25:48.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b7f66bb:::benchmark_data_smithi188_89276_object52834:head# 2024-04-23T16:25:48.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b7fe187:::benchmark_data_smithi188_89276_object168383:head# 2024-04-23T16:25:48.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b805208:::benchmark_data_smithi188_89276_object100462:head# 2024-04-23T16:25:48.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8093c6:::benchmark_data_smithi188_89276_object132921:head# 2024-04-23T16:25:48.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b80cf8c:::benchmark_data_smithi188_89276_object59435:head# 2024-04-23T16:25:48.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b81736b:::benchmark_data_smithi188_89276_object131447:head# 2024-04-23T16:25:48.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8322cf:::benchmark_data_smithi188_89276_object144625:head# 2024-04-23T16:25:48.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b838bca:::benchmark_data_smithi188_89276_object156521:head# 2024-04-23T16:25:48.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b83ae51:::benchmark_data_smithi188_89276_object64971:head# 2024-04-23T16:25:48.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b845683:::benchmark_data_smithi188_89276_object93525:head# 2024-04-23T16:25:48.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b847bba:::benchmark_data_smithi188_89276_object88507:head# 2024-04-23T16:25:48.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b85dd89:::benchmark_data_smithi188_89276_object156124:head# 2024-04-23T16:25:48.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b87893c:::benchmark_data_smithi188_89276_object142596:head# 2024-04-23T16:25:48.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8790d7:::benchmark_data_smithi188_89276_object128865:head# 2024-04-23T16:25:48.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b87bc53:::benchmark_data_smithi188_89276_object148257:head# 2024-04-23T16:25:48.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b884dc6:::benchmark_data_smithi188_89276_object146490:head# 2024-04-23T16:25:48.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b885821:::benchmark_data_smithi188_89276_object118656:head# 2024-04-23T16:25:48.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b88de16:::benchmark_data_smithi188_89276_object118753:head# 2024-04-23T16:25:48.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8a8c7d:::benchmark_data_smithi188_89276_object149984:head# 2024-04-23T16:25:48.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8a93f6:::benchmark_data_smithi188_89276_object47572:head# 2024-04-23T16:25:48.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8a9833:::benchmark_data_smithi188_89276_object42841:head# 2024-04-23T16:25:48.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8ae009:::benchmark_data_smithi188_89276_object149567:head# 2024-04-23T16:25:48.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8af23a:::benchmark_data_smithi188_89276_object73596:head# 2024-04-23T16:25:48.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8c1c3c:::benchmark_data_smithi188_89276_object134476:head# 2024-04-23T16:25:48.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8c1cd1:::benchmark_data_smithi188_89276_object148770:head# 2024-04-23T16:25:48.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8c91cc:::benchmark_data_smithi188_89276_object63855:head# 2024-04-23T16:25:48.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8cca4a:::benchmark_data_smithi188_89276_object63778:head# 2024-04-23T16:25:48.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8d03ef:::benchmark_data_smithi188_89276_object145665:head# 2024-04-23T16:25:48.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8d17f1:::benchmark_data_smithi188_89276_object66480:head# 2024-04-23T16:25:48.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8d385b:::benchmark_data_smithi188_89276_object87981:head# 2024-04-23T16:25:48.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8d555b:::benchmark_data_smithi188_89276_object118177:head# 2024-04-23T16:25:48.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8dcf68:::benchmark_data_smithi188_89276_object49646:head# 2024-04-23T16:25:48.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8f4e62:::benchmark_data_smithi188_89276_object152450:head# 2024-04-23T16:25:48.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8f5e0f:::benchmark_data_smithi188_89276_object105065:head# 2024-04-23T16:25:48.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b8fa37f:::benchmark_data_smithi188_89276_object60219:head# 2024-04-23T16:25:48.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b909ab7:::benchmark_data_smithi188_89276_object76331:head# 2024-04-23T16:25:48.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b913169:::benchmark_data_smithi188_89276_object60827:head# 2024-04-23T16:25:48.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b91b168:::benchmark_data_smithi188_89276_object166107:head# 2024-04-23T16:25:48.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9207e0:::benchmark_data_smithi188_89276_object101609:head# 2024-04-23T16:25:48.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b92532c:::benchmark_data_smithi188_89276_object127729:head# 2024-04-23T16:25:48.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9298dc:::benchmark_data_smithi188_89276_object89196:head# 2024-04-23T16:25:48.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9308b5:::benchmark_data_smithi188_89276_object55461:head# 2024-04-23T16:25:48.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b933d7d:::benchmark_data_smithi188_89276_object63785:head# 2024-04-23T16:25:48.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b93cf24:::benchmark_data_smithi188_89276_object60989:head# 2024-04-23T16:25:48.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b948e48:::benchmark_data_smithi188_89276_object64911:head# 2024-04-23T16:25:48.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b957562:::benchmark_data_smithi188_89276_object148916:head# 2024-04-23T16:25:48.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b958d45:::benchmark_data_smithi188_89276_object98816:head# 2024-04-23T16:25:48.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9734af:::benchmark_data_smithi188_89276_object110070:head# 2024-04-23T16:25:48.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b978ac4:::benchmark_data_smithi188_89276_object88711:head# 2024-04-23T16:25:48.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b97bfdf:::benchmark_data_smithi188_89276_object139433:head# 2024-04-23T16:25:48.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b97d1b3:::benchmark_data_smithi188_89276_object139309:head# 2024-04-23T16:25:48.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b98433b:::benchmark_data_smithi188_89276_object164082:head# 2024-04-23T16:25:48.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b988030:::benchmark_data_smithi188_89276_object76380:head# 2024-04-23T16:25:48.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b98e30f:::benchmark_data_smithi188_89276_object85935:head# 2024-04-23T16:25:48.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9984d5:::benchmark_data_smithi188_89276_object132504:head# 2024-04-23T16:25:48.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9a0cc9:::benchmark_data_smithi188_89276_object69388:head# 2024-04-23T16:25:48.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9a3189:::benchmark_data_smithi188_89276_object123982:head# 2024-04-23T16:25:48.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9a600e:::benchmark_data_smithi188_89276_object129884:head# 2024-04-23T16:25:48.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9bfaa9:::benchmark_data_smithi188_89276_object158609:head# 2024-04-23T16:25:48.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9c4726:::benchmark_data_smithi188_89276_object101838:head# 2024-04-23T16:25:48.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9c5c26:::benchmark_data_smithi188_89276_object115742:head# 2024-04-23T16:25:48.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9cc839:::benchmark_data_smithi188_89276_object51069:head# 2024-04-23T16:25:48.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9cca7d:::benchmark_data_smithi188_89276_object139979:head# 2024-04-23T16:25:48.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9d635f:::benchmark_data_smithi188_89276_object98411:head# 2024-04-23T16:25:48.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9dc8b1:::benchmark_data_smithi188_89276_object93702:head# 2024-04-23T16:25:48.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9e06bc:::benchmark_data_smithi188_89276_object161302:head# 2024-04-23T16:25:48.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9e0f5f:::benchmark_data_smithi188_89276_object129230:head# 2024-04-23T16:25:48.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9ec760:::benchmark_data_smithi188_89276_object68300:head# 2024-04-23T16:25:48.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9ed40a:::benchmark_data_smithi188_89276_object84590:head# 2024-04-23T16:25:48.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9ed96d:::benchmark_data_smithi188_89276_object64002:head# 2024-04-23T16:25:48.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9f27af:::benchmark_data_smithi188_89276_object156364:head# 2024-04-23T16:25:48.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9f9488:::benchmark_data_smithi188_89276_object143004:head# 2024-04-23T16:25:48.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2b9fa4f0:::benchmark_data_smithi188_89276_object147650:head# 2024-04-23T16:25:48.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba0b866:::benchmark_data_smithi188_89276_object49952:head# 2024-04-23T16:25:48.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba0cb89:::benchmark_data_smithi188_89276_object160352:head# 2024-04-23T16:25:48.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba0f2b2:::benchmark_data_smithi188_89276_object72577:head# 2024-04-23T16:25:48.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba147b2:::benchmark_data_smithi188_89276_object92918:head# 2024-04-23T16:25:48.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba1a321:::benchmark_data_smithi188_89276_object110514:head# 2024-04-23T16:25:48.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba1b5f8:::benchmark_data_smithi188_89276_object108260:head# 2024-04-23T16:25:48.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba265d6:::benchmark_data_smithi188_89276_object125343:head# 2024-04-23T16:25:48.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba293bf:::benchmark_data_smithi188_89276_object100059:head# 2024-04-23T16:25:48.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba2e266:::benchmark_data_smithi188_89276_object123890:head# 2024-04-23T16:25:48.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba2e39d:::benchmark_data_smithi188_89276_object67869:head# 2024-04-23T16:25:48.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba32562:::benchmark_data_smithi188_89276_object70443:head# 2024-04-23T16:25:48.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba3625b:::benchmark_data_smithi188_89276_object58314:head# 2024-04-23T16:25:48.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba42bc5:::benchmark_data_smithi188_89276_object75978:head# 2024-04-23T16:25:48.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba448a9:::benchmark_data_smithi188_89276_object122586:head# 2024-04-23T16:25:48.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba46cc8:::benchmark_data_smithi188_89276_object155928:head# 2024-04-23T16:25:48.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba4a1f6:::benchmark_data_smithi188_89276_object90968:head# 2024-04-23T16:25:48.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba4bd77:::benchmark_data_smithi188_89276_object113967:head# 2024-04-23T16:25:48.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba4e074:::benchmark_data_smithi188_89276_object129994:head# 2024-04-23T16:25:48.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba5ccf3:::benchmark_data_smithi188_89276_object153255:head# 2024-04-23T16:25:48.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba63083:::benchmark_data_smithi188_89276_object51823:head# 2024-04-23T16:25:48.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba65cce:::benchmark_data_smithi188_89276_object54287:head# 2024-04-23T16:25:48.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba6b76a:::benchmark_data_smithi188_89276_object100720:head# 2024-04-23T16:25:48.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba6c00f:::benchmark_data_smithi188_89276_object159673:head# 2024-04-23T16:25:48.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ba73c3e:::benchmark_data_smithi188_89276_object148196:head# 2024-04-23T16:25:48.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2baa580d:::benchmark_data_smithi188_89276_object90114:head# 2024-04-23T16:25:48.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2baa6d5f:::benchmark_data_smithi188_89276_object114699:head# 2024-04-23T16:25:48.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2baa9c32:::benchmark_data_smithi188_89276_object142918:head# 2024-04-23T16:25:48.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2baae79e:::benchmark_data_smithi188_89276_object124808:head# 2024-04-23T16:25:48.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2babcb5e:::benchmark_data_smithi188_89276_object114725:head# 2024-04-23T16:25:48.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bac26e1:::benchmark_data_smithi188_89276_object122394:head# 2024-04-23T16:25:48.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bac4b57:::benchmark_data_smithi188_89276_object51632:head# 2024-04-23T16:25:48.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bac7c39:::benchmark_data_smithi188_89276_object95363:head# 2024-04-23T16:25:48.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bacd0a9:::benchmark_data_smithi188_89276_object98466:head# 2024-04-23T16:25:48.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bad2e30:::benchmark_data_smithi188_89276_object138591:head# 2024-04-23T16:25:48.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bad9eff:::benchmark_data_smithi188_89276_object78264:head# 2024-04-23T16:25:48.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2baddf8e:::benchmark_data_smithi188_89276_object90539:head# 2024-04-23T16:25:48.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bae1e5b:::benchmark_data_smithi188_89276_object76022:head# 2024-04-23T16:25:48.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2baf0010:::benchmark_data_smithi188_89276_object125056:head# 2024-04-23T16:25:48.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2baf4813:::benchmark_data_smithi188_89276_object66900:head# 2024-04-23T16:25:48.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2baf5314:::benchmark_data_smithi188_89276_object115693:head# 2024-04-23T16:25:48.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2baf64e1:::benchmark_data_smithi188_89276_object70300:head# 2024-04-23T16:25:48.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bafede9:::benchmark_data_smithi188_89276_object46340:head# 2024-04-23T16:25:48.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bb1f23c:::benchmark_data_smithi188_89276_object101619:head# 2024-04-23T16:25:48.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bb275d3:::benchmark_data_smithi188_89276_object149158:head# 2024-04-23T16:25:48.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bb31e3c:::benchmark_data_smithi188_89276_object116404:head# 2024-04-23T16:25:48.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bb320cb:::benchmark_data_smithi188_89276_object76368:head# 2024-04-23T16:25:48.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bb36e86:::benchmark_data_smithi188_89276_object124798:head# 2024-04-23T16:25:48.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bb4001b:::benchmark_data_smithi188_89276_object92614:head# 2024-04-23T16:25:48.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bb4e9eb:::benchmark_data_smithi188_89276_object72867:head# 2024-04-23T16:25:48.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bb4f757:::benchmark_data_smithi188_89276_object71274:head# 2024-04-23T16:25:48.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bb6a4b4:::benchmark_data_smithi188_89276_object75494:head# 2024-04-23T16:25:48.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bb6bb99:::benchmark_data_smithi188_89276_object131356:head# 2024-04-23T16:25:48.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bb750bb:::benchmark_data_smithi188_89276_object98206:head# 2024-04-23T16:25:48.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bb7be60:::benchmark_data_smithi188_89276_object83854:head# 2024-04-23T16:25:48.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bb87a77:::benchmark_data_smithi188_89276_object64283:head# 2024-04-23T16:25:48.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.229 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:25:48.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bb90d18:::benchmark_data_smithi188_89276_object94585:head# 2024-04-23T16:25:48.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bb9c0cf:::benchmark_data_smithi188_89276_object87998:head# 2024-04-23T16:25:48.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bba4cfa:::benchmark_data_smithi188_89276_object93341:head# 2024-04-23T16:25:48.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bbb5725:::benchmark_data_smithi188_89276_object159944:head# 2024-04-23T16:25:48.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bbba08e:::benchmark_data_smithi188_89276_object75970:head# 2024-04-23T16:25:48.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bbcf3fd:::benchmark_data_smithi188_89276_object42858:head# 2024-04-23T16:25:48.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bbe10cd:::benchmark_data_smithi188_89276_object147237:head# 2024-04-23T16:25:48.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bbe73a8:::benchmark_data_smithi188_89276_object77432:head# 2024-04-23T16:25:48.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bbea7b9:::benchmark_data_smithi188_89276_object50632:head# 2024-04-23T16:25:48.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bbeed1d:::benchmark_data_smithi188_89276_object51390:head# 2024-04-23T16:25:48.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bbefefe:::benchmark_data_smithi188_89276_object52104:head# 2024-04-23T16:25:48.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bbf05dc:::benchmark_data_smithi188_89276_object85210:head# 2024-04-23T16:25:48.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bbfe940:::benchmark_data_smithi188_89276_object41550:head# 2024-04-23T16:25:48.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bc0e66d:::benchmark_data_smithi188_89276_object56470:head# 2024-04-23T16:25:48.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bc13ede:::benchmark_data_smithi188_89276_object149166:head# 2024-04-23T16:25:48.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bc2705b:::benchmark_data_smithi188_89276_object139287:head# 2024-04-23T16:25:48.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bc2d094:::benchmark_data_smithi188_89276_object128302:head# 2024-04-23T16:25:48.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bc37fb0:::benchmark_data_smithi188_89276_object121665:head# 2024-04-23T16:25:48.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bc40646:::benchmark_data_smithi188_89276_object127938:head# 2024-04-23T16:25:48.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bc41ef8:::benchmark_data_smithi188_89276_object113067:head# 2024-04-23T16:25:48.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bc4a4a4:::benchmark_data_smithi188_89276_object75899:head# 2024-04-23T16:25:48.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bc4c7a4:::benchmark_data_smithi188_89276_object77838:head# 2024-04-23T16:25:48.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bc57ee4:::benchmark_data_smithi188_89276_object70546:head# 2024-04-23T16:25:48.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bc70280:::benchmark_data_smithi188_89276_object142950:head# 2024-04-23T16:25:48.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bc730e5:::benchmark_data_smithi188_89276_object68844:head# 2024-04-23T16:25:48.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bc798d3:::benchmark_data_smithi188_89276_object133486:head# 2024-04-23T16:25:48.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bc86c38:::benchmark_data_smithi188_89276_object66423:head# 2024-04-23T16:25:48.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bc8b779:::benchmark_data_smithi188_89276_object82329:head# 2024-04-23T16:25:48.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bc8c01f:::benchmark_data_smithi188_89276_object69457:head# 2024-04-23T16:25:48.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bc9a37e:::benchmark_data_smithi188_89276_object59381:head# 2024-04-23T16:25:48.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bca373b:::benchmark_data_smithi188_89276_object51613:head# 2024-04-23T16:25:48.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bca73b4:::benchmark_data_smithi188_89276_object137398:head# 2024-04-23T16:25:48.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bca745e:::benchmark_data_smithi188_89276_object167736:head# 2024-04-23T16:25:48.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bcaa61c:::benchmark_data_smithi188_89276_object135370:head# 2024-04-23T16:25:48.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bcae42a:::benchmark_data_smithi188_89276_object51862:head# 2024-04-23T16:25:48.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bcc1b7a:::benchmark_data_smithi188_89276_object121254:head# 2024-04-23T16:25:48.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bcc1b9d:::benchmark_data_smithi188_89276_object71718:head# 2024-04-23T16:25:48.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bccdf9e:::benchmark_data_smithi188_89276_object126674:head# 2024-04-23T16:25:48.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bccebc8:::benchmark_data_smithi188_89276_object70670:head# 2024-04-23T16:25:48.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bcd827b:::benchmark_data_smithi188_89276_object70899:head# 2024-04-23T16:25:48.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bcda607:::benchmark_data_smithi188_89276_object67364:head# 2024-04-23T16:25:48.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bcdbd43:::benchmark_data_smithi188_89276_object86253:head# 2024-04-23T16:25:48.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bcf4c44:::benchmark_data_smithi188_89276_object128525:head# 2024-04-23T16:25:48.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bcf9e4f:::benchmark_data_smithi188_89276_object149623:head# 2024-04-23T16:25:48.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.271 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:48.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bcfa18d:::benchmark_data_smithi188_89276_object46563:head# 2024-04-23T16:25:48.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bd0bf0c:::benchmark_data_smithi188_89276_object109447:head# 2024-04-23T16:25:48.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bd17c18:::benchmark_data_smithi188_89276_object75943:head# 2024-04-23T16:25:48.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bd1bb75:::benchmark_data_smithi188_89276_object99161:head# 2024-04-23T16:25:48.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bd2dbff:::benchmark_data_smithi188_89276_object152268:head# 2024-04-23T16:25:48.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bd43606:::benchmark_data_smithi188_89276_object110111:head# 2024-04-23T16:25:48.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bd4dc6d:::benchmark_data_smithi188_89276_object113080:head# 2024-04-23T16:25:48.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bd53e8a:::benchmark_data_smithi188_89276_object89888:head# 2024-04-23T16:25:48.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bd62a32:::benchmark_data_smithi188_89276_object60979:head# 2024-04-23T16:25:48.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bd6da37:::benchmark_data_smithi188_89276_object132884:head# 2024-04-23T16:25:48.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bd72926:::benchmark_data_smithi188_89276_object143350:head# 2024-04-23T16:25:48.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bd73df3:::benchmark_data_smithi188_89276_object137020:head# 2024-04-23T16:25:48.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bd78758:::benchmark_data_smithi188_89276_object132324:head# 2024-04-23T16:25:48.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bd7979c:::benchmark_data_smithi188_89276_object150153:head# 2024-04-23T16:25:48.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bd814ff:::benchmark_data_smithi188_89276_object96407:head# 2024-04-23T16:25:48.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bd8e719:::benchmark_data_smithi188_89276_object104195:head# 2024-04-23T16:25:48.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bd953a4:::benchmark_data_smithi188_89276_object83314:head# 2024-04-23T16:25:48.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bda08ec:::benchmark_data_smithi188_89276_object161029:head# 2024-04-23T16:25:48.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bda5258:::benchmark_data_smithi188_89276_object101871:head# 2024-04-23T16:25:48.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bdab346:::benchmark_data_smithi188_89276_object154402:head# 2024-04-23T16:25:48.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bdbd9ba:::benchmark_data_smithi188_89276_object75977:head# 2024-04-23T16:25:48.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bdceac2:::benchmark_data_smithi188_89276_object94388:head# 2024-04-23T16:25:48.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bdd5fd4:::benchmark_data_smithi188_89276_object143646:head# 2024-04-23T16:25:48.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bdd63fb:::benchmark_data_smithi188_89276_object66153:head# 2024-04-23T16:25:48.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bddb75c:::benchmark_data_smithi188_89276_object40451:head# 2024-04-23T16:25:48.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bde5f8f:::benchmark_data_smithi188_89276_object74857:head# 2024-04-23T16:25:48.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bde6b2e:::benchmark_data_smithi188_89276_object113308:head# 2024-04-23T16:25:48.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bdea80d:::benchmark_data_smithi188_89276_object168450:head# 2024-04-23T16:25:48.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bdeddaf:::benchmark_data_smithi188_89276_object48576:head# 2024-04-23T16:25:48.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be155c1:::benchmark_data_smithi188_89276_object134637:head# 2024-04-23T16:25:48.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be1a034:::benchmark_data_smithi188_89276_object162169:head# 2024-04-23T16:25:48.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be1b4f4:::benchmark_data_smithi188_89276_object48130:head# 2024-04-23T16:25:48.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be21b24:::benchmark_data_smithi188_89276_object121192:head# 2024-04-23T16:25:48.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be3a902:::benchmark_data_smithi188_89276_object58204:head# 2024-04-23T16:25:48.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be3d51a:::benchmark_data_smithi188_89276_object82564:head# 2024-04-23T16:25:48.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be5d6ef:::benchmark_data_smithi188_89276_object87748:head# 2024-04-23T16:25:48.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be60404:::benchmark_data_smithi188_89276_object81494:head# 2024-04-23T16:25:48.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be6320e:::benchmark_data_smithi188_89276_object137176:head# 2024-04-23T16:25:48.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be6a6fe:::benchmark_data_smithi188_89276_object149685:head# 2024-04-23T16:25:48.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be7063e:::benchmark_data_smithi188_89276_object88394:head# 2024-04-23T16:25:48.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be7b193:::benchmark_data_smithi188_89276_object78765:head# 2024-04-23T16:25:48.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be7c600:::benchmark_data_smithi188_89276_object69798:head# 2024-04-23T16:25:48.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be7ed18:::benchmark_data_smithi188_89276_object141102:head# 2024-04-23T16:25:48.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be864be:::benchmark_data_smithi188_89276_object167571:head# 2024-04-23T16:25:48.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be88828:::benchmark_data_smithi188_89276_object57755:head# 2024-04-23T16:25:48.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be904df:::benchmark_data_smithi188_89276_object62314:head# 2024-04-23T16:25:48.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be94c83:::benchmark_data_smithi188_89276_object147089:head# 2024-04-23T16:25:48.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be98cce:::benchmark_data_smithi188_89276_object165874:head# 2024-04-23T16:25:48.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2be9d1ef:::benchmark_data_smithi188_89276_object43531:head# 2024-04-23T16:25:48.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bea2c78:::benchmark_data_smithi188_89276_object135110:head# 2024-04-23T16:25:48.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2beb2eb0:::benchmark_data_smithi188_89276_object148915:head# 2024-04-23T16:25:48.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bebba20:::benchmark_data_smithi188_89276_object49378:head# 2024-04-23T16:25:48.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bec10b1:::benchmark_data_smithi188_89276_object92360:head# 2024-04-23T16:25:48.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bee2d3f:::benchmark_data_smithi188_89276_object120057:head# 2024-04-23T16:25:48.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bef5710:::benchmark_data_smithi188_89276_object150198:head# 2024-04-23T16:25:48.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bf096a6:::benchmark_data_smithi188_89276_object62421:head# 2024-04-23T16:25:48.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bf147a0:::benchmark_data_smithi188_89276_object126395:head# 2024-04-23T16:25:48.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bf2187b:::benchmark_data_smithi188_89276_object148562:head# 2024-04-23T16:25:48.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bf24934:::benchmark_data_smithi188_89276_object138327:head# 2024-04-23T16:25:48.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bf38b94:::benchmark_data_smithi188_89276_object166902:head# 2024-04-23T16:25:48.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bf49f48:::benchmark_data_smithi188_89276_object61265:head# 2024-04-23T16:25:48.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bf576c2:::benchmark_data_smithi188_89276_object45769:head# 2024-04-23T16:25:48.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bf5f491:::benchmark_data_smithi188_89276_object79961:head# 2024-04-23T16:25:48.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bf61374:::benchmark_data_smithi188_89276_object50261:head# 2024-04-23T16:25:48.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bf71a63:::benchmark_data_smithi188_89276_object136987:head# 2024-04-23T16:25:48.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bf77eb5:::benchmark_data_smithi188_89276_object108450:head# 2024-04-23T16:25:48.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bf81f92:::benchmark_data_smithi188_89276_object140635:head# 2024-04-23T16:25:48.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bf9bc93:::benchmark_data_smithi188_89276_object129826:head# 2024-04-23T16:25:48.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bf9dbe2:::benchmark_data_smithi188_89276_object141860:head# 2024-04-23T16:25:48.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bfa4453:::benchmark_data_smithi188_89276_object129879:head# 2024-04-23T16:25:48.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bfb31d2:::benchmark_data_smithi188_89276_object149137:head# 2024-04-23T16:25:48.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bfb4e2c:::benchmark_data_smithi188_89276_object104514:head# 2024-04-23T16:25:48.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bfc0a54:::benchmark_data_smithi188_89276_object47322:head# 2024-04-23T16:25:48.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bfc1cf6:::benchmark_data_smithi188_89276_object161288:head# 2024-04-23T16:25:48.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bfcd006:::benchmark_data_smithi188_89276_object82461:head# 2024-04-23T16:25:48.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bfd1394:::benchmark_data_smithi188_89276_object163989:head# 2024-04-23T16:25:48.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bfda138:::benchmark_data_smithi188_89276_object139822:head# 2024-04-23T16:25:48.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bfdd1cc:::benchmark_data_smithi188_89276_object58700:head# 2024-04-23T16:25:48.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bfde9fe:::benchmark_data_smithi188_89276_object90582:head# 2024-04-23T16:25:48.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bfdefe9:::benchmark_data_smithi188_89276_object109770:head# 2024-04-23T16:25:48.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bfe0605:::benchmark_data_smithi188_89276_object52628:head# 2024-04-23T16:25:48.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bfe254e:::benchmark_data_smithi188_89276_object157355:head# 2024-04-23T16:25:48.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bfea192:::benchmark_data_smithi188_89276_object158226:head# 2024-04-23T16:25:48.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.324 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bfea8b1:::benchmark_data_smithi188_89276_object75689:head# 2024-04-23T16:25:48.324 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.324 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2bfecb05:::benchmark_data_smithi188_89276_object74197:head# 2024-04-23T16:25:48.324 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.324 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c014759:::benchmark_data_smithi188_89276_object151624:head# 2024-04-23T16:25:48.324 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.325 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c02db28:::benchmark_data_smithi188_89276_object52851:head# 2024-04-23T16:25:48.325 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.325 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c030a30:::benchmark_data_smithi188_89276_object82116:head# 2024-04-23T16:25:48.325 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.325 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c032123:::benchmark_data_smithi188_89276_object96905:head# 2024-04-23T16:25:48.325 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.325 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c042433:::benchmark_data_smithi188_89276_object117397:head# 2024-04-23T16:25:48.325 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.325 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c0446d0:::benchmark_data_smithi188_89276_object126833:head# 2024-04-23T16:25:48.325 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.325 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c045ae5:::benchmark_data_smithi188_89276_object98440:head# 2024-04-23T16:25:48.325 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.325 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c04b58f:::benchmark_data_smithi188_89276_object94467:head# 2024-04-23T16:25:48.325 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.326 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c04bb2c:::benchmark_data_smithi188_89276_object101861:head# 2024-04-23T16:25:48.326 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.326 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c06e24d:::benchmark_data_smithi188_89276_object89165:head# 2024-04-23T16:25:48.326 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.326 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c06f4e1:::benchmark_data_smithi188_89276_object49552:head# 2024-04-23T16:25:48.326 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.326 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c0703e6:::benchmark_data_smithi188_89276_object90254:head# 2024-04-23T16:25:48.326 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.327 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c077b2a:::benchmark_data_smithi188_89276_object99084:head# 2024-04-23T16:25:48.327 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.327 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c07bd22:::benchmark_data_smithi188_89276_object101831:head# 2024-04-23T16:25:48.327 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.327 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c07cc2a:::benchmark_data_smithi188_89276_object44313:head# 2024-04-23T16:25:48.327 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.327 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c08cc21:::benchmark_data_smithi188_89276_object64213:head# 2024-04-23T16:25:48.327 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.328 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c0a0a56:::benchmark_data_smithi188_89276_object69271:head# 2024-04-23T16:25:48.328 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c0a3f06:::benchmark_data_smithi188_89276_object95256:head# 2024-04-23T16:25:48.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c0a9b9d:::benchmark_data_smithi188_89276_object42083:head# 2024-04-23T16:25:48.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c0b1c68:::benchmark_data_smithi188_89276_object161306:head# 2024-04-23T16:25:48.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c0bba65:::benchmark_data_smithi188_89276_object95130:head# 2024-04-23T16:25:48.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c0be58e:::benchmark_data_smithi188_89276_object70553:head# 2024-04-23T16:25:48.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c0db7d8:::benchmark_data_smithi188_89276_object46838:head# 2024-04-23T16:25:48.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c0e8bab:::benchmark_data_smithi188_89276_object129597:head# 2024-04-23T16:25:48.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c0f37ed:::benchmark_data_smithi188_89276_object85530:head# 2024-04-23T16:25:48.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c0fcd00:::benchmark_data_smithi188_89276_object51916:head# 2024-04-23T16:25:48.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c0fcfe9:::benchmark_data_smithi188_89276_object94576:head# 2024-04-23T16:25:48.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c1057a7:::benchmark_data_smithi188_89276_object61219:head# 2024-04-23T16:25:48.334 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.334 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c10a830:::benchmark_data_smithi188_89276_object47935:head# 2024-04-23T16:25:48.334 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.335 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c10c20a:::benchmark_data_smithi188_89276_object119206:head# 2024-04-23T16:25:48.335 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.335 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c10cce0:::benchmark_data_smithi188_89276_object83578:head# 2024-04-23T16:25:48.335 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.335 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c120c53:::benchmark_data_smithi188_89276_object99679:head# 2024-04-23T16:25:48.335 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.335 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c127875:::benchmark_data_smithi188_89276_object134293:head# 2024-04-23T16:25:48.335 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.336 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c12eee3:::benchmark_data_smithi188_89276_object116135:head# 2024-04-23T16:25:48.336 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.336 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c1357d4:::benchmark_data_smithi188_89276_object149902:head# 2024-04-23T16:25:48.336 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.336 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c147d49:::benchmark_data_smithi188_89276_object120582:head# 2024-04-23T16:25:48.337 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.337 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c155e06:::benchmark_data_smithi188_89276_object150896:head# 2024-04-23T16:25:48.337 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.337 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c16b8c8:::benchmark_data_smithi188_89276_object73820:head# 2024-04-23T16:25:48.337 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.337 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c172865:::benchmark_data_smithi188_89276_object168090:head# 2024-04-23T16:25:48.337 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.337 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c17e8ba:::benchmark_data_smithi188_89276_object126992:head# 2024-04-23T16:25:48.337 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.338 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c183b5b:::benchmark_data_smithi188_89276_object115320:head# 2024-04-23T16:25:48.338 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.338 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c18781d:::benchmark_data_smithi188_89276_object57159:head# 2024-04-23T16:25:48.338 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.338 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c187993:::benchmark_data_smithi188_89276_object152904:head# 2024-04-23T16:25:48.338 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.338 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c18957c:::benchmark_data_smithi188_89276_object145821:head# 2024-04-23T16:25:48.339 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.339 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c19b0ee:::benchmark_data_smithi188_89276_object41152:head# 2024-04-23T16:25:48.339 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.339 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c19c47a:::benchmark_data_smithi188_89276_object64188:head# 2024-04-23T16:25:48.339 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.339 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c1a5cf8:::benchmark_data_smithi188_89276_object89950:head# 2024-04-23T16:25:48.339 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c1ab3db:::benchmark_data_smithi188_89276_object41806:head# 2024-04-23T16:25:48.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c1ade82:::benchmark_data_smithi188_89276_object160588:head# 2024-04-23T16:25:48.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c1bf6fd:::benchmark_data_smithi188_89276_object112774:head# 2024-04-23T16:25:48.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c1c1f3f:::benchmark_data_smithi188_89276_object134450:head# 2024-04-23T16:25:48.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c1cfd5c:::benchmark_data_smithi188_89276_object102398:head# 2024-04-23T16:25:48.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.341 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c1d39cd:::benchmark_data_smithi188_89276_object165513:head# 2024-04-23T16:25:48.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.341 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c1e40bb:::benchmark_data_smithi188_89276_object129740:head# 2024-04-23T16:25:48.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.341 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c1e6e4b:::benchmark_data_smithi188_89276_object155957:head# 2024-04-23T16:25:48.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c1f998f:::benchmark_data_smithi188_89276_object86464:head# 2024-04-23T16:25:48.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c1fbdce:::benchmark_data_smithi188_89276_object109525:head# 2024-04-23T16:25:48.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c202e7c:::benchmark_data_smithi188_89276_object164037:head# 2024-04-23T16:25:48.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c204198:::benchmark_data_smithi188_89276_object144720:head# 2024-04-23T16:25:48.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c2072d1:::benchmark_data_smithi188_89276_object164364:head# 2024-04-23T16:25:48.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c20b49c:::benchmark_data_smithi188_89276_object96200:head# 2024-04-23T16:25:48.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c20b5f9:::benchmark_data_smithi188_89276_object66961:head# 2024-04-23T16:25:48.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c20da2c:::benchmark_data_smithi188_89276_object142881:head# 2024-04-23T16:25:48.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c20ed78:::benchmark_data_smithi188_89276_object146678:head# 2024-04-23T16:25:48.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c216318:::benchmark_data_smithi188_89276_object80728:head# 2024-04-23T16:25:48.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c21cd7f:::benchmark_data_smithi188_89276_object140484:head# 2024-04-23T16:25:48.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c21f33b:::benchmark_data_smithi188_89276_object113227:head# 2024-04-23T16:25:48.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c229cca:::benchmark_data_smithi188_89276_object131413:head# 2024-04-23T16:25:48.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c235612:::benchmark_data_smithi188_89276_object49447:head# 2024-04-23T16:25:48.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c243220:::benchmark_data_smithi188_89276_object130507:head# 2024-04-23T16:25:48.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c2450ef:::benchmark_data_smithi188_89276_object87744:head# 2024-04-23T16:25:48.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c2464b2:::benchmark_data_smithi188_89276_object100056:head# 2024-04-23T16:25:48.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c2568a6:::benchmark_data_smithi188_89276_object82749:head# 2024-04-23T16:25:48.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c25b861:::benchmark_data_smithi188_89276_object47270:head# 2024-04-23T16:25:48.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c25fecc:::benchmark_data_smithi188_89276_object76803:head# 2024-04-23T16:25:48.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c26244a:::benchmark_data_smithi188_89276_object105498:head# 2024-04-23T16:25:48.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c26630f:::benchmark_data_smithi188_89276_object165413:head# 2024-04-23T16:25:48.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c26cda2:::benchmark_data_smithi188_89276_object48347:head# 2024-04-23T16:25:48.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c26e23c:::benchmark_data_smithi188_89276_object123752:head# 2024-04-23T16:25:48.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c2783c6:::benchmark_data_smithi188_89276_object55984:head# 2024-04-23T16:25:48.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c27e526:::benchmark_data_smithi188_89276_object84639:head# 2024-04-23T16:25:48.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c28149b:::benchmark_data_smithi188_89276_object61220:head# 2024-04-23T16:25:48.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c298b41:::benchmark_data_smithi188_89276_object89770:head# 2024-04-23T16:25:48.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c299d2a:::benchmark_data_smithi188_89276_object88124:head# 2024-04-23T16:25:48.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c29e242:::benchmark_data_smithi188_89276_object99653:head# 2024-04-23T16:25:48.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c2aa9b3:::benchmark_data_smithi188_89276_object78961:head# 2024-04-23T16:25:48.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c2b0e51:::benchmark_data_smithi188_89276_object81358:head# 2024-04-23T16:25:48.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c2c7241:::benchmark_data_smithi188_89276_object80614:head# 2024-04-23T16:25:48.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c2c748f:::benchmark_data_smithi188_89276_object109874:head# 2024-04-23T16:25:48.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c2d5075:::benchmark_data_smithi188_89276_object96352:head# 2024-04-23T16:25:48.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c2d65d4:::benchmark_data_smithi188_89276_object135859:head# 2024-04-23T16:25:48.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c2d8a57:::benchmark_data_smithi188_89276_object141227:head# 2024-04-23T16:25:48.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c2da2b4:::benchmark_data_smithi188_89276_object51737:head# 2024-04-23T16:25:48.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c2dae56:::benchmark_data_smithi188_89276_object134183:head# 2024-04-23T16:25:48.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c2dd5e1:::benchmark_data_smithi188_89276_object49687:head# 2024-04-23T16:25:48.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c2f1356:::benchmark_data_smithi188_89276_object90490:head# 2024-04-23T16:25:48.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c308a8d:::benchmark_data_smithi188_89276_object120742:head# 2024-04-23T16:25:48.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c3097a3:::benchmark_data_smithi188_89276_object97731:head# 2024-04-23T16:25:48.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.374 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c3140b3:::benchmark_data_smithi188_89276_object115354:head# 2024-04-23T16:25:48.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c31d003:::benchmark_data_smithi188_89276_object58317:head# 2024-04-23T16:25:48.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c323a7e:::benchmark_data_smithi188_89276_object43977:head# 2024-04-23T16:25:48.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c33740e:::benchmark_data_smithi188_89276_object75581:head# 2024-04-23T16:25:48.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c338ee0:::benchmark_data_smithi188_89276_object69390:head# 2024-04-23T16:25:48.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c33a239:::benchmark_data_smithi188_89276_object154562:head# 2024-04-23T16:25:48.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c343403:::benchmark_data_smithi188_89276_object135966:head# 2024-04-23T16:25:48.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c34b80e:::benchmark_data_smithi188_89276_object79962:head# 2024-04-23T16:25:48.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c34bc0c:::benchmark_data_smithi188_89276_object148733:head# 2024-04-23T16:25:48.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c356aaa:::benchmark_data_smithi188_89276_object129785:head# 2024-04-23T16:25:48.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c3571e4:::benchmark_data_smithi188_89276_object74741:head# 2024-04-23T16:25:48.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c35b503:::benchmark_data_smithi188_89276_object57848:head# 2024-04-23T16:25:48.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c35c1aa:::benchmark_data_smithi188_89276_object59876:head# 2024-04-23T16:25:48.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c3622e9:::benchmark_data_smithi188_89276_object154615:head# 2024-04-23T16:25:48.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c36df91:::benchmark_data_smithi188_89276_object128681:head# 2024-04-23T16:25:48.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c3716d0:::benchmark_data_smithi188_89276_object122819:head# 2024-04-23T16:25:48.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c372c8c:::benchmark_data_smithi188_89276_object127546:head# 2024-04-23T16:25:48.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c38162d:::benchmark_data_smithi188_89276_object142098:head# 2024-04-23T16:25:48.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c38a4fb:::benchmark_data_smithi188_89276_object95517:head# 2024-04-23T16:25:48.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c39c133:::benchmark_data_smithi188_89276_object47445:head# 2024-04-23T16:25:48.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c39df36:::benchmark_data_smithi188_89276_object83199:head# 2024-04-23T16:25:48.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c3a5004:::benchmark_data_smithi188_89276_object118756:head# 2024-04-23T16:25:48.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c3b8b4f:::benchmark_data_smithi188_89276_object80643:head# 2024-04-23T16:25:48.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c3c368f:::benchmark_data_smithi188_89276_object62004:head# 2024-04-23T16:25:48.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c3c910e:::benchmark_data_smithi188_89276_object106406:head# 2024-04-23T16:25:48.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c3d8987:::benchmark_data_smithi188_89276_object122345:head# 2024-04-23T16:25:48.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c3df742:::benchmark_data_smithi188_89276_object91105:head# 2024-04-23T16:25:48.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c3e6882:::benchmark_data_smithi188_89276_object155997:head# 2024-04-23T16:25:48.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c3f8a41:::benchmark_data_smithi188_89276_object119946:head# 2024-04-23T16:25:48.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c3fb0f9:::benchmark_data_smithi188_89276_object128392:head# 2024-04-23T16:25:48.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c401511:::benchmark_data_smithi188_89276_object100214:head# 2024-04-23T16:25:48.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c408bcc:::benchmark_data_smithi188_89276_object153825:head# 2024-04-23T16:25:48.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c4093db:::benchmark_data_smithi188_89276_object66939:head# 2024-04-23T16:25:48.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c40eceb:::benchmark_data_smithi188_89276_object125452:head# 2024-04-23T16:25:48.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c4144d4:::benchmark_data_smithi188_89276_object73710:head# 2024-04-23T16:25:48.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c419e0d:::benchmark_data_smithi188_89276_object153433:head# 2024-04-23T16:25:48.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c41e78e:::benchmark_data_smithi188_89276_object115589:head# 2024-04-23T16:25:48.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c41fcb6:::benchmark_data_smithi188_89276_object73056:head# 2024-04-23T16:25:48.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c4299e3:::benchmark_data_smithi188_89276_object157944:head# 2024-04-23T16:25:48.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c42efcb:::benchmark_data_smithi188_89276_object133017:head# 2024-04-23T16:25:48.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c43226b:::benchmark_data_smithi188_89276_object44808:head# 2024-04-23T16:25:48.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c43514b:::benchmark_data_smithi188_89276_object58575:head# 2024-04-23T16:25:48.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c43d578:::benchmark_data_smithi188_89276_object47274:head# 2024-04-23T16:25:48.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c474550:::benchmark_data_smithi188_89276_object162508:head# 2024-04-23T16:25:48.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c47a428:::benchmark_data_smithi188_89276_object83000:head# 2024-04-23T16:25:48.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c481e03:::benchmark_data_smithi188_89276_object86160:head# 2024-04-23T16:25:48.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c4873ff:::benchmark_data_smithi188_89276_object133016:head# 2024-04-23T16:25:48.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c492e8a:::benchmark_data_smithi188_89276_object137145:head# 2024-04-23T16:25:48.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c499cda:::benchmark_data_smithi188_89276_object120164:head# 2024-04-23T16:25:48.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c49c068:::benchmark_data_smithi188_89276_object47926:head# 2024-04-23T16:25:48.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c4a0034:::benchmark_data_smithi188_89276_object68538:head# 2024-04-23T16:25:48.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c4b9cb4:::benchmark_data_smithi188_89276_object142504:head# 2024-04-23T16:25:48.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c4c8568:::benchmark_data_smithi188_89276_object116616:head# 2024-04-23T16:25:48.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c4c8cd0:::benchmark_data_smithi188_89276_object78269:head# 2024-04-23T16:25:48.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c4d06df:::benchmark_data_smithi188_89276_object101967:head# 2024-04-23T16:25:48.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c4d672a:::benchmark_data_smithi188_89276_object134586:head# 2024-04-23T16:25:48.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c4d833a:::benchmark_data_smithi188_89276_object125294:head# 2024-04-23T16:25:48.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c4dad8c:::benchmark_data_smithi188_89276_object117299:head# 2024-04-23T16:25:48.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c4dec1f:::benchmark_data_smithi188_89276_object112251:head# 2024-04-23T16:25:48.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c4efac0:::benchmark_data_smithi188_89276_object81688:head# 2024-04-23T16:25:48.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c4f117d:::benchmark_data_smithi188_89276_object52709:head# 2024-04-23T16:25:48.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c4fd1cd:::benchmark_data_smithi188_89276_object88031:head# 2024-04-23T16:25:48.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c501bb0:::benchmark_data_smithi188_89276_object158083:head# 2024-04-23T16:25:48.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c522864:::benchmark_data_smithi188_89276_object64609:head# 2024-04-23T16:25:48.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c524dfc:::benchmark_data_smithi188_89276_object136758:head# 2024-04-23T16:25:48.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c526783:::benchmark_data_smithi188_89276_object146522:head# 2024-04-23T16:25:48.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c52af31:::benchmark_data_smithi188_89276_object150946:head# 2024-04-23T16:25:48.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c52da78:::benchmark_data_smithi188_89276_object138857:head# 2024-04-23T16:25:48.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c53356d:::benchmark_data_smithi188_89276_object69622:head# 2024-04-23T16:25:48.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c535449:::benchmark_data_smithi188_89276_object84002:head# 2024-04-23T16:25:48.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c53e0ac:::benchmark_data_smithi188_89276_object127857:head# 2024-04-23T16:25:48.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c53e884:::benchmark_data_smithi188_89276_object44073:head# 2024-04-23T16:25:48.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c5472c8:::benchmark_data_smithi188_89276_object119913:head# 2024-04-23T16:25:48.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c556d97:::benchmark_data_smithi188_89276_object73612:head# 2024-04-23T16:25:48.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c55d414:::benchmark_data_smithi188_89276_object139213:head# 2024-04-23T16:25:48.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c55e3c0:::benchmark_data_smithi188_89276_object156196:head# 2024-04-23T16:25:48.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c565965:::benchmark_data_smithi188_89276_object68578:head# 2024-04-23T16:25:48.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c56ea1f:::benchmark_data_smithi188_89276_object102790:head# 2024-04-23T16:25:48.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c572551:::benchmark_data_smithi188_89276_object113618:head# 2024-04-23T16:25:48.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c576a62:::benchmark_data_smithi188_89276_object107891:head# 2024-04-23T16:25:48.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c57826a:::benchmark_data_smithi188_89276_object83936:head# 2024-04-23T16:25:48.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c5830a9:::benchmark_data_smithi188_89276_object112855:head# 2024-04-23T16:25:48.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c58808d:::benchmark_data_smithi188_89276_object62819:head# 2024-04-23T16:25:48.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c58ae2d:::benchmark_data_smithi188_89276_object81886:head# 2024-04-23T16:25:48.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c597148:::benchmark_data_smithi188_89276_object65062:head# 2024-04-23T16:25:48.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c59a09b:::benchmark_data_smithi188_89276_object166081:head# 2024-04-23T16:25:48.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c59d326:::benchmark_data_smithi188_89276_object65257:head# 2024-04-23T16:25:48.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c5a9992:::benchmark_data_smithi188_89276_object93284:head# 2024-04-23T16:25:48.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c5aeb23:::benchmark_data_smithi188_89276_object133616:head# 2024-04-23T16:25:48.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c5c4844:::benchmark_data_smithi188_89276_object87052:head# 2024-04-23T16:25:48.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c5c72d1:::benchmark_data_smithi188_89276_object102235:head# 2024-04-23T16:25:48.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c5ca33b:::benchmark_data_smithi188_89276_object107558:head# 2024-04-23T16:25:48.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c5d00a4:::benchmark_data_smithi188_89276_object52879:head# 2024-04-23T16:25:48.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c5d69f5:::benchmark_data_smithi188_89276_object161504:head# 2024-04-23T16:25:48.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c5daddb:::benchmark_data_smithi188_89276_object114963:head# 2024-04-23T16:25:48.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c5dc578:::benchmark_data_smithi188_89276_object69270:head# 2024-04-23T16:25:48.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c5e6136:::benchmark_data_smithi188_89276_object114242:head# 2024-04-23T16:25:48.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c5ea459:::benchmark_data_smithi188_89276_object154459:head# 2024-04-23T16:25:48.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c5fe615:::benchmark_data_smithi188_89276_object121087:head# 2024-04-23T16:25:48.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c60f17f:::benchmark_data_smithi188_89276_object56775:head# 2024-04-23T16:25:48.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c611f63:::benchmark_data_smithi188_89276_object135484:head# 2024-04-23T16:25:48.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c619977:::benchmark_data_smithi188_89276_object124242:head# 2024-04-23T16:25:48.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c61ce78:::benchmark_data_smithi188_89276_object58958:head# 2024-04-23T16:25:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c62095a:::benchmark_data_smithi188_89276_object91875:head# 2024-04-23T16:25:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c623453:::benchmark_data_smithi188_89276_object150923:head# 2024-04-23T16:25:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c630dbf:::benchmark_data_smithi188_89276_object167632:head# 2024-04-23T16:25:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c641af7:::benchmark_data_smithi188_89276_object125226:head# 2024-04-23T16:25:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c6433f3:::benchmark_data_smithi188_89276_object114434:head# 2024-04-23T16:25:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c643fba:::benchmark_data_smithi188_89276_object98973:head# 2024-04-23T16:25:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c65009c:::benchmark_data_smithi188_89276_object88386:head# 2024-04-23T16:25:48.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c65544d:::benchmark_data_smithi188_89276_object58497:head# 2024-04-23T16:25:48.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c655a76:::benchmark_data_smithi188_89276_object119905:head# 2024-04-23T16:25:48.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c66a66b:::benchmark_data_smithi188_89276_object146406:head# 2024-04-23T16:25:48.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c671ca5:::benchmark_data_smithi188_89276_object76301:head# 2024-04-23T16:25:48.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c672a1e:::benchmark_data_smithi188_89276_object50824:head# 2024-04-23T16:25:48.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.432 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c67583a:::benchmark_data_smithi188_89276_object166565:head# 2024-04-23T16:25:48.432 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c678b10:::benchmark_data_smithi188_89276_object41433:head# 2024-04-23T16:25:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c67952e:::benchmark_data_smithi188_89276_object62291:head# 2024-04-23T16:25:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c67c8b6:::benchmark_data_smithi188_89276_object153000:head# 2024-04-23T16:25:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c68079e:::benchmark_data_smithi188_89276_object94059:head# 2024-04-23T16:25:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c6835d8:::benchmark_data_smithi188_89276_object55921:head# 2024-04-23T16:25:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.434 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c6873e4:::benchmark_data_smithi188_89276_object152278:head# 2024-04-23T16:25:48.434 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c691c18:::benchmark_data_smithi188_89276_object57062:head# 2024-04-23T16:25:48.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c6985e2:::benchmark_data_smithi188_89276_object112815:head# 2024-04-23T16:25:48.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c6a0c67:::benchmark_data_smithi188_89276_object161787:head# 2024-04-23T16:25:48.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c6a5032:::benchmark_data_smithi188_89276_object139241:head# 2024-04-23T16:25:48.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c6a8e40:::benchmark_data_smithi188_89276_object72622:head# 2024-04-23T16:25:48.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c6a92f4:::benchmark_data_smithi188_89276_object116043:head# 2024-04-23T16:25:48.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c6c7e36:::benchmark_data_smithi188_89276_object146880:head# 2024-04-23T16:25:48.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c6ccefe:::benchmark_data_smithi188_89276_object45352:head# 2024-04-23T16:25:48.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c6d1309:::benchmark_data_smithi188_89276_object70291:head# 2024-04-23T16:25:48.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c6d54b9:::benchmark_data_smithi188_89276_object44348:head# 2024-04-23T16:25:48.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c6ee705:::benchmark_data_smithi188_89276_object145811:head# 2024-04-23T16:25:48.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c6f9dc3:::benchmark_data_smithi188_89276_object53595:head# 2024-04-23T16:25:48.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.438 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c710696:::benchmark_data_smithi188_89276_object131993:head# 2024-04-23T16:25:48.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c712b13:::benchmark_data_smithi188_89276_object63503:head# 2024-04-23T16:25:48.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c717f12:::benchmark_data_smithi188_89276_object97296:head# 2024-04-23T16:25:48.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c71f352:::benchmark_data_smithi188_89276_object42422:head# 2024-04-23T16:25:48.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c735a8f:::benchmark_data_smithi188_89276_object66643:head# 2024-04-23T16:25:48.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c73ea02:::benchmark_data_smithi188_89276_object160603:head# 2024-04-23T16:25:48.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c744921:::benchmark_data_smithi188_89276_object132567:head# 2024-04-23T16:25:48.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c747575:::benchmark_data_smithi188_89276_object151188:head# 2024-04-23T16:25:48.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c74b5a4:::benchmark_data_smithi188_89276_object123405:head# 2024-04-23T16:25:48.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c74e931:::benchmark_data_smithi188_89276_object85702:head# 2024-04-23T16:25:48.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c74e933:::benchmark_data_smithi188_89276_object41462:head# 2024-04-23T16:25:48.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c750c30:::benchmark_data_smithi188_89276_object63017:head# 2024-04-23T16:25:48.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c75e456:::benchmark_data_smithi188_89276_object111915:head# 2024-04-23T16:25:48.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c7645f5:::benchmark_data_smithi188_89276_object156126:head# 2024-04-23T16:25:48.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c76b174:::benchmark_data_smithi188_89276_object128224:head# 2024-04-23T16:25:48.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c795e12:::benchmark_data_smithi188_89276_object54213:head# 2024-04-23T16:25:48.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c79b7de:::benchmark_data_smithi188_89276_object82717:head# 2024-04-23T16:25:48.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c7b33fa:::benchmark_data_smithi188_89276_object68818:head# 2024-04-23T16:25:48.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c7b9d91:::benchmark_data_smithi188_89276_object86248:head# 2024-04-23T16:25:48.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.454 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c7bc5e4:::benchmark_data_smithi188_89276_object81366:head# 2024-04-23T16:25:48.454 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c7c1f56:::benchmark_data_smithi188_89276_object114519:head# 2024-04-23T16:25:48.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.457 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c7c4442:::benchmark_data_smithi188_89276_object127202:head# 2024-04-23T16:25:48.457 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c7c7b42:::benchmark_data_smithi188_89276_object71690:head# 2024-04-23T16:25:48.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c7c9db9:::benchmark_data_smithi188_89276_object135754:head# 2024-04-23T16:25:48.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c7ddad8:::benchmark_data_smithi188_89276_object43800:head# 2024-04-23T16:25:48.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c7ec33e:::benchmark_data_smithi188_89276_object150381:head# 2024-04-23T16:25:48.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c80cb22:::benchmark_data_smithi188_89276_object165463:head# 2024-04-23T16:25:48.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c80e070:::benchmark_data_smithi188_89276_object43974:head# 2024-04-23T16:25:48.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.460 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c80e690:::benchmark_data_smithi188_89276_object96611:head# 2024-04-23T16:25:48.460 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.466 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c8117c9:::benchmark_data_smithi188_89276_object106938:head# 2024-04-23T16:25:48.466 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.466 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c829f61:::benchmark_data_smithi188_89276_object74947:head# 2024-04-23T16:25:48.466 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.467 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c8470da:::benchmark_data_smithi188_89276_object55027:head# 2024-04-23T16:25:48.467 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.467 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c847986:::benchmark_data_smithi188_89276_object85748:head# 2024-04-23T16:25:48.467 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.470 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c848101:::benchmark_data_smithi188_89276_object63322:head# 2024-04-23T16:25:48.470 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.470 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c84a7ee:::benchmark_data_smithi188_89276_object51018:head# 2024-04-23T16:25:48.470 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c84d431:::benchmark_data_smithi188_89276_object76921:head# 2024-04-23T16:25:48.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c85163d:::benchmark_data_smithi188_89276_object62222:head# 2024-04-23T16:25:48.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c857535:::benchmark_data_smithi188_89276_object103568:head# 2024-04-23T16:25:48.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c85946e:::benchmark_data_smithi188_89276_object46480:head# 2024-04-23T16:25:48.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c860b09:::benchmark_data_smithi188_89276_object75714:head# 2024-04-23T16:25:48.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c86b97a:::benchmark_data_smithi188_89276_object151739:head# 2024-04-23T16:25:48.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c8707d9:::benchmark_data_smithi188_89276_object110923:head# 2024-04-23T16:25:48.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c874f30:::benchmark_data_smithi188_89276_object164233:head# 2024-04-23T16:25:48.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c87ddff:::benchmark_data_smithi188_89276_object87282:head# 2024-04-23T16:25:48.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c885820:::benchmark_data_smithi188_89276_object65244:head# 2024-04-23T16:25:48.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c88c807:::benchmark_data_smithi188_89276_object53361:head# 2024-04-23T16:25:48.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c88e270:::benchmark_data_smithi188_89276_object154951:head# 2024-04-23T16:25:48.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.481 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c8973c5:::benchmark_data_smithi188_89276_object64144:head# 2024-04-23T16:25:48.481 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.481 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c89b982:::benchmark_data_smithi188_89276_object44943:head# 2024-04-23T16:25:48.481 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c8ab04c:::benchmark_data_smithi188_89276_object81798:head# 2024-04-23T16:25:48.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c8ad193:::benchmark_data_smithi188_89276_object149279:head# 2024-04-23T16:25:48.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c8b862e:::benchmark_data_smithi188_89276_object129224:head# 2024-04-23T16:25:48.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c8cb076:::benchmark_data_smithi188_89276_object51482:head# 2024-04-23T16:25:48.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c8d24ca:::benchmark_data_smithi188_89276_object119291:head# 2024-04-23T16:25:48.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.486 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c8da1c8:::benchmark_data_smithi188_89276_object65565:head# 2024-04-23T16:25:48.486 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.486 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c8fc585:::benchmark_data_smithi188_89276_object149134:head# 2024-04-23T16:25:48.486 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.486 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c9017b6:::benchmark_data_smithi188_89276_object89005:head# 2024-04-23T16:25:48.486 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c90430d:::benchmark_data_smithi188_89276_object72159:head# 2024-04-23T16:25:48.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c90f357:::benchmark_data_smithi188_89276_object40273:head# 2024-04-23T16:25:48.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c9103f4:::benchmark_data_smithi188_89276_object146291:head# 2024-04-23T16:25:48.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c91de8f:::benchmark_data_smithi188_89276_object92257:head# 2024-04-23T16:25:48.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c933a24:::benchmark_data_smithi188_89276_object137690:head# 2024-04-23T16:25:48.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c93f122:::benchmark_data_smithi188_89276_object157494:head# 2024-04-23T16:25:48.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c946314:::benchmark_data_smithi188_89276_object83833:head# 2024-04-23T16:25:48.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c957616:::benchmark_data_smithi188_89276_object84600:head# 2024-04-23T16:25:48.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c962b59:::benchmark_data_smithi188_89276_object86163:head# 2024-04-23T16:25:48.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c96cd2d:::benchmark_data_smithi188_89276_object80624:head# 2024-04-23T16:25:48.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c971f86:::benchmark_data_smithi188_89276_object94836:head# 2024-04-23T16:25:48.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c978464:::benchmark_data_smithi188_89276_object120356:head# 2024-04-23T16:25:48.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c97ee95:::benchmark_data_smithi188_89276_object164212:head# 2024-04-23T16:25:48.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c9863ba:::benchmark_data_smithi188_89276_object50642:head# 2024-04-23T16:25:48.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c98fd50:::benchmark_data_smithi188_89276_object46587:head# 2024-04-23T16:25:48.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c995b91:::benchmark_data_smithi188_89276_object168614:head# 2024-04-23T16:25:48.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c9961a1:::benchmark_data_smithi188_89276_object113420:head# 2024-04-23T16:25:48.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c996891:::benchmark_data_smithi188_89276_object139086:head# 2024-04-23T16:25:48.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c9d3a70:::benchmark_data_smithi188_89276_object115814:head# 2024-04-23T16:25:48.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c9db14a:::benchmark_data_smithi188_89276_object75549:head# 2024-04-23T16:25:48.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c9dfa74:::benchmark_data_smithi188_89276_object99332:head# 2024-04-23T16:25:48.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c9ed4c0:::benchmark_data_smithi188_89276_object42113:head# 2024-04-23T16:25:48.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c9f03b2:::benchmark_data_smithi188_89276_object59664:head# 2024-04-23T16:25:48.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c9f3bb0:::benchmark_data_smithi188_89276_object72972:head# 2024-04-23T16:25:48.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c9f6d2d:::benchmark_data_smithi188_89276_object167992:head# 2024-04-23T16:25:48.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c9f7ba8:::benchmark_data_smithi188_89276_object87139:head# 2024-04-23T16:25:48.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c9f80b5:::benchmark_data_smithi188_89276_object164176:head# 2024-04-23T16:25:48.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.505 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c9fdc24:::benchmark_data_smithi188_89276_object80385:head# 2024-04-23T16:25:48.505 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.506 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2c9ffed4:::benchmark_data_smithi188_89276_object140629:head# 2024-04-23T16:25:48.506 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.506 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ca10d55:::benchmark_data_smithi188_89276_object151518:head# 2024-04-23T16:25:48.507 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ca1a8d8:::benchmark_data_smithi188_89276_object167879:head# 2024-04-23T16:25:48.507 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ca239c7:::benchmark_data_smithi188_89276_object141876:head# 2024-04-23T16:25:48.507 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ca25e79:::benchmark_data_smithi188_89276_object162829:head# 2024-04-23T16:25:48.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ca4c764:::benchmark_data_smithi188_89276_object116259:head# 2024-04-23T16:25:48.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ca4d6b8:::benchmark_data_smithi188_89276_object88234:head# 2024-04-23T16:25:48.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ca54e88:::benchmark_data_smithi188_89276_object41055:head# 2024-04-23T16:25:48.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ca5b06b:::benchmark_data_smithi188_89276_object99190:head# 2024-04-23T16:25:48.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ca61a26:::benchmark_data_smithi188_89276_object55955:head# 2024-04-23T16:25:48.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ca6a4c3:::benchmark_data_smithi188_89276_object75122:head# 2024-04-23T16:25:48.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ca8e1ad:::benchmark_data_smithi188_89276_object113329:head# 2024-04-23T16:25:48.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ca92984:::benchmark_data_smithi188_89276_object83770:head# 2024-04-23T16:25:48.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.510 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ca9dae8:::benchmark_data_smithi188_89276_object154329:head# 2024-04-23T16:25:48.510 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ca9ea7a:::benchmark_data_smithi188_89276_object118583:head# 2024-04-23T16:25:48.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ca9ec1a:::benchmark_data_smithi188_89276_object66134:head# 2024-04-23T16:25:48.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2caa2fc0:::benchmark_data_smithi188_89276_object135737:head# 2024-04-23T16:25:48.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2caa840c:::benchmark_data_smithi188_89276_object93532:head# 2024-04-23T16:25:48.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2caaf367:::benchmark_data_smithi188_89276_object129647:head# 2024-04-23T16:25:48.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cabe337:::benchmark_data_smithi188_89276_object121453:head# 2024-04-23T16:25:48.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cad1682:::benchmark_data_smithi188_89276_object46986:head# 2024-04-23T16:25:48.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cad9b99:::benchmark_data_smithi188_89276_object74223:head# 2024-04-23T16:25:48.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cadc5f1:::benchmark_data_smithi188_89276_object167413:head# 2024-04-23T16:25:48.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2caddc1c:::benchmark_data_smithi188_89276_object155130:head# 2024-04-23T16:25:48.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cae6048:::benchmark_data_smithi188_89276_object138533:head# 2024-04-23T16:25:48.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2caf02d9:::benchmark_data_smithi188_89276_object41632:head# 2024-04-23T16:25:48.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cb0f71d:::benchmark_data_smithi188_89276_object147811:head# 2024-04-23T16:25:48.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.515 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cb117b6:::benchmark_data_smithi188_89276_object95388:head# 2024-04-23T16:25:48.515 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.515 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cb22017:::benchmark_data_smithi188_89276_object76417:head# 2024-04-23T16:25:48.515 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.515 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cb27bd4:::benchmark_data_smithi188_89276_object120036:head# 2024-04-23T16:25:48.515 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.515 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cb2e0cb:::benchmark_data_smithi188_89276_object109512:head# 2024-04-23T16:25:48.515 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cb32abe:::benchmark_data_smithi188_89276_object62954:head# 2024-04-23T16:25:48.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cb3b0ba:::benchmark_data_smithi188_89276_object88888:head# 2024-04-23T16:25:48.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cb41611:::benchmark_data_smithi188_89276_object118846:head# 2024-04-23T16:25:48.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cb44604:::benchmark_data_smithi188_89276_object63983:head# 2024-04-23T16:25:48.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cb57edf:::benchmark_data_smithi188_89276_object98224:head# 2024-04-23T16:25:48.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cb5830f:::benchmark_data_smithi188_89276_object90271:head# 2024-04-23T16:25:48.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cb6ade8:::benchmark_data_smithi188_89276_object74946:head# 2024-04-23T16:25:48.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.522 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cb6d52e:::benchmark_data_smithi188_89276_object149138:head# 2024-04-23T16:25:48.522 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.522 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cb7fa60:::benchmark_data_smithi188_89276_object132859:head# 2024-04-23T16:25:48.522 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.522 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cb82c3c:::benchmark_data_smithi188_89276_object117295:head# 2024-04-23T16:25:48.522 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.522 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cb9e6a1:::benchmark_data_smithi188_89276_object149252:head# 2024-04-23T16:25:48.522 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.522 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cbaae89:::benchmark_data_smithi188_89276_object135628:head# 2024-04-23T16:25:48.522 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.522 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cbb02f1:::benchmark_data_smithi188_89276_object67236:head# 2024-04-23T16:25:48.522 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.522 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cbb093d:::benchmark_data_smithi188_89276_object72138:head# 2024-04-23T16:25:48.522 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cbb4f88:::benchmark_data_smithi188_89276_object120860:head# 2024-04-23T16:25:48.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cbb8561:::benchmark_data_smithi188_89276_object92173:head# 2024-04-23T16:25:48.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cbb8cef:::benchmark_data_smithi188_89276_object118128:head# 2024-04-23T16:25:48.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cbbad5e:::benchmark_data_smithi188_89276_object48568:head# 2024-04-23T16:25:48.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cbc3423:::benchmark_data_smithi188_89276_object123166:head# 2024-04-23T16:25:48.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cbc3e76:::benchmark_data_smithi188_89276_object72762:head# 2024-04-23T16:25:48.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cbc45a1:::benchmark_data_smithi188_89276_object109081:head# 2024-04-23T16:25:48.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.524 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cbc893f:::benchmark_data_smithi188_89276_object75975:head# 2024-04-23T16:25:48.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.524 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cbdca1f:::benchmark_data_smithi188_89276_object164996:head# 2024-04-23T16:25:48.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.524 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cbed521:::benchmark_data_smithi188_89276_object55219:head# 2024-04-23T16:25:48.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.525 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cbfa628:::benchmark_data_smithi188_89276_object141082:head# 2024-04-23T16:25:48.525 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.525 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cbfdaad:::benchmark_data_smithi188_89276_object59055:head# 2024-04-23T16:25:48.525 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.525 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cbfff27:::benchmark_data_smithi188_89276_object138996:head# 2024-04-23T16:25:48.525 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.525 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cc081a7:::benchmark_data_smithi188_89276_object137443:head# 2024-04-23T16:25:48.525 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.525 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cc0825d:::benchmark_data_smithi188_89276_object96360:head# 2024-04-23T16:25:48.525 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.525 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cc1068d:::benchmark_data_smithi188_89276_object136281:head# 2024-04-23T16:25:48.525 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.525 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cc2047f:::benchmark_data_smithi188_89276_object56848:head# 2024-04-23T16:25:48.525 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.526 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cc29377:::benchmark_data_smithi188_89276_object101783:head# 2024-04-23T16:25:48.526 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.526 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cc2be56:::benchmark_data_smithi188_89276_object43722:head# 2024-04-23T16:25:48.526 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.526 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cc394b8:::benchmark_data_smithi188_89276_object90323:head# 2024-04-23T16:25:48.526 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.527 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cc3a9ce:::benchmark_data_smithi188_89276_object111558:head# 2024-04-23T16:25:48.527 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.528 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cc3fb03:::benchmark_data_smithi188_89276_object52111:head# 2024-04-23T16:25:48.528 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.528 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cc44879:::benchmark_data_smithi188_89276_object117439:head# 2024-04-23T16:25:48.528 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.528 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cc4a9cf:::benchmark_data_smithi188_89276_object69881:head# 2024-04-23T16:25:48.528 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.528 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cc57997:::benchmark_data_smithi188_89276_object68469:head# 2024-04-23T16:25:48.528 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.528 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cc60e7d:::benchmark_data_smithi188_89276_object145504:head# 2024-04-23T16:25:48.528 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.528 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cc67a58:::benchmark_data_smithi188_89276_object48580:head# 2024-04-23T16:25:48.528 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.528 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cc7cee8:::benchmark_data_smithi188_89276_object142945:head# 2024-04-23T16:25:48.528 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.528 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cc7f5f7:::benchmark_data_smithi188_89276_object97363:head# 2024-04-23T16:25:48.529 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cc9c46a:::benchmark_data_smithi188_89276_object138673:head# 2024-04-23T16:25:48.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cca2617:::benchmark_data_smithi188_89276_object135166:head# 2024-04-23T16:25:48.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cca37af:::benchmark_data_smithi188_89276_object110447:head# 2024-04-23T16:25:48.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ccb7753:::benchmark_data_smithi188_89276_object157120:head# 2024-04-23T16:25:48.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ccbc99f:::benchmark_data_smithi188_89276_object76008:head# 2024-04-23T16:25:48.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ccbdb19:::benchmark_data_smithi188_89276_object75431:head# 2024-04-23T16:25:48.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.538 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ccbfd88:::benchmark_data_smithi188_89276_object66532:head# 2024-04-23T16:25:48.538 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.538 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ccc407e:::benchmark_data_smithi188_89276_object143519:head# 2024-04-23T16:25:48.538 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.538 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ccc804f:::benchmark_data_smithi188_89276_object149266:head# 2024-04-23T16:25:48.538 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.538 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cccc318:::benchmark_data_smithi188_89276_object83602:head# 2024-04-23T16:25:48.538 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.538 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ccd27fc:::benchmark_data_smithi188_89276_object116856:head# 2024-04-23T16:25:48.538 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.539 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cce10f0:::benchmark_data_smithi188_89276_object152552:head# 2024-04-23T16:25:48.539 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.539 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cce90cc:::benchmark_data_smithi188_89276_object55652:head# 2024-04-23T16:25:48.540 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.540 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cce928a:::benchmark_data_smithi188_89276_object150757:head# 2024-04-23T16:25:48.540 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.540 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ccf00d0:::benchmark_data_smithi188_89276_object95876:head# 2024-04-23T16:25:48.540 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.540 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ccf0b8b:::benchmark_data_smithi188_89276_object152022:head# 2024-04-23T16:25:48.540 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.541 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cd03792:::benchmark_data_smithi188_89276_object96272:head# 2024-04-23T16:25:48.541 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.541 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cd0e216:::benchmark_data_smithi188_89276_object158541:head# 2024-04-23T16:25:48.541 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.541 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cd16280:::benchmark_data_smithi188_89276_object135978:head# 2024-04-23T16:25:48.541 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.541 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cd1e62f:::benchmark_data_smithi188_89276_object75499:head# 2024-04-23T16:25:48.541 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.542 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cd2ca75:::benchmark_data_smithi188_89276_object85170:head# 2024-04-23T16:25:48.542 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.542 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cd3daa6:::benchmark_data_smithi188_89276_object92451:head# 2024-04-23T16:25:48.542 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.542 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cd4aab9:::benchmark_data_smithi188_89276_object42674:head# 2024-04-23T16:25:48.542 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.543 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cd5a769:::benchmark_data_smithi188_89276_object79245:head# 2024-04-23T16:25:48.543 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.543 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cd6167d:::benchmark_data_smithi188_89276_object108889:head# 2024-04-23T16:25:48.543 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.543 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cd69c3e:::benchmark_data_smithi188_89276_object158995:head# 2024-04-23T16:25:48.543 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.544 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cd794e5:::benchmark_data_smithi188_89276_object141983:head# 2024-04-23T16:25:48.544 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.544 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cd872e5:::benchmark_data_smithi188_89276_object95266:head# 2024-04-23T16:25:48.544 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.544 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cd9462d:::benchmark_data_smithi188_89276_object152069:head# 2024-04-23T16:25:48.544 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.545 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cdb1d1f:::benchmark_data_smithi188_89276_object57550:head# 2024-04-23T16:25:48.545 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.545 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cdb48d6:::benchmark_data_smithi188_89276_object73182:head# 2024-04-23T16:25:48.545 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.545 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cdba610:::benchmark_data_smithi188_89276_object73567:head# 2024-04-23T16:25:48.545 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.547 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cdba72b:::benchmark_data_smithi188_89276_object103320:head# 2024-04-23T16:25:48.547 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.547 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cdc76af:::benchmark_data_smithi188_89276_object166931:head# 2024-04-23T16:25:48.547 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.547 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cdc78e9:::benchmark_data_smithi188_89276_object89917:head# 2024-04-23T16:25:48.547 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.547 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cdcc314:::benchmark_data_smithi188_89276_object153021:head# 2024-04-23T16:25:48.547 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.547 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cdcc7fd:::benchmark_data_smithi188_89276_object85328:head# 2024-04-23T16:25:48.547 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.547 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cdd2393:::benchmark_data_smithi188_89276_object75928:head# 2024-04-23T16:25:48.547 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.548 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cdd308d:::benchmark_data_smithi188_89276_object62277:head# 2024-04-23T16:25:48.548 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.548 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cddd871:::benchmark_data_smithi188_89276_object51243:head# 2024-04-23T16:25:48.548 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.548 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cddf7fd:::benchmark_data_smithi188_89276_object152731:head# 2024-04-23T16:25:48.548 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.549 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cde5b18:::benchmark_data_smithi188_89276_object133747:head# 2024-04-23T16:25:48.549 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.549 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cdfa0de:::benchmark_data_smithi188_89276_object55923:head# 2024-04-23T16:25:48.549 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.549 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cdfcdac:::benchmark_data_smithi188_89276_object82946:head# 2024-04-23T16:25:48.550 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.550 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce014f4:::benchmark_data_smithi188_89276_object66630:head# 2024-04-23T16:25:48.550 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.550 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce0c1ae:::benchmark_data_smithi188_89276_object77665:head# 2024-04-23T16:25:48.550 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.550 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce11960:::benchmark_data_smithi188_89276_object77441:head# 2024-04-23T16:25:48.550 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.552 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce15c1d:::benchmark_data_smithi188_89276_object129937:head# 2024-04-23T16:25:48.552 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.552 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce1a870:::benchmark_data_smithi188_89276_object78639:head# 2024-04-23T16:25:48.552 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.552 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce25e98:::benchmark_data_smithi188_89276_object168569:head# 2024-04-23T16:25:48.552 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.552 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce443fa:::benchmark_data_smithi188_89276_object151228:head# 2024-04-23T16:25:48.552 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.552 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce4922d:::benchmark_data_smithi188_89276_object145422:head# 2024-04-23T16:25:48.553 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.553 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce5250e:::benchmark_data_smithi188_89276_object147891:head# 2024-04-23T16:25:48.553 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce6200a:::benchmark_data_smithi188_89276_object54837:head# 2024-04-23T16:25:48.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce63343:::benchmark_data_smithi188_89276_object80672:head# 2024-04-23T16:25:48.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce6b1b8:::benchmark_data_smithi188_89276_object104106:head# 2024-04-23T16:25:48.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce70b13:::benchmark_data_smithi188_89276_object127594:head# 2024-04-23T16:25:48.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce768a3:::benchmark_data_smithi188_89276_object162292:head# 2024-04-23T16:25:48.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce7a858:::benchmark_data_smithi188_89276_object95306:head# 2024-04-23T16:25:48.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce8b7a7:::benchmark_data_smithi188_89276_object104868:head# 2024-04-23T16:25:48.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce8c954:::benchmark_data_smithi188_89276_object44336:head# 2024-04-23T16:25:48.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce91571:::benchmark_data_smithi188_89276_object76134:head# 2024-04-23T16:25:48.556 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.556 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce97068:::benchmark_data_smithi188_89276_object148111:head# 2024-04-23T16:25:48.556 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.556 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce9734d:::benchmark_data_smithi188_89276_object102294:head# 2024-04-23T16:25:48.556 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ce98440:::benchmark_data_smithi188_89276_object58349:head# 2024-04-23T16:25:48.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cea714d:::benchmark_data_smithi188_89276_object95750:head# 2024-04-23T16:25:48.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ceaaf41:::benchmark_data_smithi188_89276_object43369:head# 2024-04-23T16:25:48.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ceb4204:::benchmark_data_smithi188_89276_object149475:head# 2024-04-23T16:25:48.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ceb4db7:::benchmark_data_smithi188_89276_object157105:head# 2024-04-23T16:25:48.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cebeec7:::benchmark_data_smithi188_89276_object91903:head# 2024-04-23T16:25:48.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cec3982:::benchmark_data_smithi188_89276_object124876:head# 2024-04-23T16:25:48.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ced3c7d:::benchmark_data_smithi188_89276_object41312:head# 2024-04-23T16:25:48.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cedabb3:::benchmark_data_smithi188_89276_object127549:head# 2024-04-23T16:25:48.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cee45aa:::benchmark_data_smithi188_89276_object43525:head# 2024-04-23T16:25:48.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ceebc24:::benchmark_data_smithi188_89276_object43999:head# 2024-04-23T16:25:48.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ceefd37:::benchmark_data_smithi188_89276_object144843:head# 2024-04-23T16:25:48.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cef2323:::benchmark_data_smithi188_89276_object52730:head# 2024-04-23T16:25:48.563 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.563 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cef53f5:::benchmark_data_smithi188_89276_object87637:head# 2024-04-23T16:25:48.563 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.563 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cef852c:::benchmark_data_smithi188_89276_object78337:head# 2024-04-23T16:25:48.563 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.563 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf094d1:::benchmark_data_smithi188_89276_object168280:head# 2024-04-23T16:25:48.563 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.564 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf0bd0c:::benchmark_data_smithi188_89276_object83158:head# 2024-04-23T16:25:48.564 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.564 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf11249:::benchmark_data_smithi188_89276_object46908:head# 2024-04-23T16:25:48.564 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.564 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf1b2f6:::benchmark_data_smithi188_89276_object82216:head# 2024-04-23T16:25:48.565 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.565 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf1c5be:::benchmark_data_smithi188_89276_object81595:head# 2024-04-23T16:25:48.565 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.565 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf2125e:::benchmark_data_smithi188_89276_object138893:head# 2024-04-23T16:25:48.565 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.565 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf2ce3d:::benchmark_data_smithi188_89276_object138197:head# 2024-04-23T16:25:48.565 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.565 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf2e05f:::benchmark_data_smithi188_89276_object97904:head# 2024-04-23T16:25:48.565 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.565 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf38f9f:::benchmark_data_smithi188_89276_object130223:head# 2024-04-23T16:25:48.565 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.565 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf3b4bc:::benchmark_data_smithi188_89276_object159140:head# 2024-04-23T16:25:48.566 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.566 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf43fca:::benchmark_data_smithi188_89276_object125434:head# 2024-04-23T16:25:48.566 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.566 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf4af96:::benchmark_data_smithi188_89276_object131821:head# 2024-04-23T16:25:48.566 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf4b384:::benchmark_data_smithi188_89276_object132510:head# 2024-04-23T16:25:48.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf53abb:::benchmark_data_smithi188_89276_object153309:head# 2024-04-23T16:25:48.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf53df7:::benchmark_data_smithi188_89276_object75405:head# 2024-04-23T16:25:48.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf5f022:::benchmark_data_smithi188_89276_object63204:head# 2024-04-23T16:25:48.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf651b3:::benchmark_data_smithi188_89276_object142557:head# 2024-04-23T16:25:48.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf68282:::benchmark_data_smithi188_89276_object117814:head# 2024-04-23T16:25:48.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.570 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf704bc:::benchmark_data_smithi188_89276_object102265:head# 2024-04-23T16:25:48.570 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.570 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf7252b:::benchmark_data_smithi188_89276_object62891:head# 2024-04-23T16:25:48.570 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.570 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf7da4e:::benchmark_data_smithi188_89276_object59295:head# 2024-04-23T16:25:48.570 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf93c65:::benchmark_data_smithi188_89276_object89414:head# 2024-04-23T16:25:48.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf95a16:::benchmark_data_smithi188_89276_object149322:head# 2024-04-23T16:25:48.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cf9e7f8:::benchmark_data_smithi188_89276_object165475:head# 2024-04-23T16:25:48.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cfa01bf:::benchmark_data_smithi188_89276_object62957:head# 2024-04-23T16:25:48.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cfa05ff:::benchmark_data_smithi188_89276_object98145:head# 2024-04-23T16:25:48.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cfa1c30:::benchmark_data_smithi188_89276_object61908:head# 2024-04-23T16:25:48.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cfae6b0:::benchmark_data_smithi188_89276_object54963:head# 2024-04-23T16:25:48.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cfb074c:::benchmark_data_smithi188_89276_object143957:head# 2024-04-23T16:25:48.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cfb36c6:::benchmark_data_smithi188_89276_object68900:head# 2024-04-23T16:25:48.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cfb75ac:::benchmark_data_smithi188_89276_object147989:head# 2024-04-23T16:25:48.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cfbe543:::benchmark_data_smithi188_89276_object151547:head# 2024-04-23T16:25:48.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cfc000e:::benchmark_data_smithi188_89276_object47986:head# 2024-04-23T16:25:48.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cfc7d79:::benchmark_data_smithi188_89276_object157460:head# 2024-04-23T16:25:48.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cfca13a:::benchmark_data_smithi188_89276_object85811:head# 2024-04-23T16:25:48.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cfca241:::benchmark_data_smithi188_89276_object42951:head# 2024-04-23T16:25:48.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cfcb5ce:::benchmark_data_smithi188_89276_object76007:head# 2024-04-23T16:25:48.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cfcc938:::benchmark_data_smithi188_89276_object131167:head# 2024-04-23T16:25:48.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cfe63c0:::benchmark_data_smithi188_89276_object73760:head# 2024-04-23T16:25:48.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cfeb0b2:::benchmark_data_smithi188_89276_object142807:head# 2024-04-23T16:25:48.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cffafd2:::benchmark_data_smithi188_89276_object137788:head# 2024-04-23T16:25:48.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2cffbc33:::benchmark_data_smithi188_89276_object83206:head# 2024-04-23T16:25:48.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d00fe60:::benchmark_data_smithi188_89276_object106883:head# 2024-04-23T16:25:48.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d01e70e:::benchmark_data_smithi188_89276_object118781:head# 2024-04-23T16:25:48.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d0217dc:::benchmark_data_smithi188_89276_object87997:head# 2024-04-23T16:25:48.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d022364:::benchmark_data_smithi188_89276_object149660:head# 2024-04-23T16:25:48.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.578 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d026a5c:::benchmark_data_smithi188_89276_object104600:head# 2024-04-23T16:25:48.578 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.578 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d027599:::benchmark_data_smithi188_89276_object57237:head# 2024-04-23T16:25:48.578 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.578 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d028ece:::benchmark_data_smithi188_89276_object120214:head# 2024-04-23T16:25:48.578 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d03f251:::benchmark_data_smithi188_89276_object130665:head# 2024-04-23T16:25:48.579 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d04a744:::benchmark_data_smithi188_89276_object90764:head# 2024-04-23T16:25:48.579 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d04b482:::benchmark_data_smithi188_89276_object155099:head# 2024-04-23T16:25:48.579 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d056271:::benchmark_data_smithi188_89276_object98456:head# 2024-04-23T16:25:48.579 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d05972e:::benchmark_data_smithi188_89276_object159314:head# 2024-04-23T16:25:48.579 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.580 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d063269:::benchmark_data_smithi188_89276_object58516:head# 2024-04-23T16:25:48.580 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.580 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d071b8c:::benchmark_data_smithi188_89276_object98701:head# 2024-04-23T16:25:48.580 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.581 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d0791c7:::benchmark_data_smithi188_89276_object57267:head# 2024-04-23T16:25:48.581 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.581 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d0828cc:::benchmark_data_smithi188_89276_object84902:head# 2024-04-23T16:25:48.581 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.581 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d084043:::benchmark_data_smithi188_89276_object70876:head# 2024-04-23T16:25:48.581 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.581 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d08da6a:::benchmark_data_smithi188_89276_object165429:head# 2024-04-23T16:25:48.581 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d0940bb:::benchmark_data_smithi188_89276_object132343:head# 2024-04-23T16:25:48.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d09798f:::benchmark_data_smithi188_89276_object88038:head# 2024-04-23T16:25:48.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d0aafcc:::benchmark_data_smithi188_89276_object127680:head# 2024-04-23T16:25:48.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d0aca37:::benchmark_data_smithi188_89276_object111042:head# 2024-04-23T16:25:48.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d0b0527:::benchmark_data_smithi188_89276_object143322:head# 2024-04-23T16:25:48.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d0c58f9:::benchmark_data_smithi188_89276_object130704:head# 2024-04-23T16:25:48.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d0c8d57:::benchmark_data_smithi188_89276_object87215:head# 2024-04-23T16:25:48.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.584 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d0cc810:::benchmark_data_smithi188_89276_object82943:head# 2024-04-23T16:25:48.584 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.584 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d0cf040:::benchmark_data_smithi188_89276_object76621:head# 2024-04-23T16:25:48.584 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.584 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d0d1149:::benchmark_data_smithi188_89276_object113405:head# 2024-04-23T16:25:48.584 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.584 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d0d9242:::benchmark_data_smithi188_89276_object68426:head# 2024-04-23T16:25:48.585 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.585 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d0dcc04:::benchmark_data_smithi188_89276_object88764:head# 2024-04-23T16:25:48.585 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.585 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d0dce2e:::benchmark_data_smithi188_89276_object100511:head# 2024-04-23T16:25:48.586 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.586 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d0e7d7a:::benchmark_data_smithi188_89276_object75364:head# 2024-04-23T16:25:48.586 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.586 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d0eaa61:::benchmark_data_smithi188_89276_object144144:head# 2024-04-23T16:25:48.586 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.588 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d0f250e:::benchmark_data_smithi188_89276_object66214:head# 2024-04-23T16:25:48.588 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.588 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d0f9c6c:::benchmark_data_smithi188_89276_object79684:head# 2024-04-23T16:25:48.588 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.588 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d11ff50:::benchmark_data_smithi188_89276_object78127:head# 2024-04-23T16:25:48.589 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.589 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d12a3f3:::benchmark_data_smithi188_89276_object117052:head# 2024-04-23T16:25:48.589 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.589 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d13c0b2:::benchmark_data_smithi188_89276_object150080:head# 2024-04-23T16:25:48.589 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.589 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d1544b9:::benchmark_data_smithi188_89276_object143727:head# 2024-04-23T16:25:48.589 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.589 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d156259:::benchmark_data_smithi188_89276_object75219:head# 2024-04-23T16:25:48.589 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.590 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d158268:::benchmark_data_smithi188_89276_object90667:head# 2024-04-23T16:25:48.590 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.590 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d16ef96:::benchmark_data_smithi188_89276_object142926:head# 2024-04-23T16:25:48.590 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.590 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d175643:::benchmark_data_smithi188_89276_object102634:head# 2024-04-23T16:25:48.590 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.590 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d177784:::benchmark_data_smithi188_89276_object125635:head# 2024-04-23T16:25:48.590 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.591 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d1792a4:::benchmark_data_smithi188_89276_object70675:head# 2024-04-23T16:25:48.591 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.591 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d184eae:::benchmark_data_smithi188_89276_object77976:head# 2024-04-23T16:25:48.591 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.592 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d187139:::benchmark_data_smithi188_89276_object46368:head# 2024-04-23T16:25:48.592 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.592 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d18dc88:::benchmark_data_smithi188_89276_object81297:head# 2024-04-23T16:25:48.592 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d19bacf:::benchmark_data_smithi188_89276_object101415:head# 2024-04-23T16:25:48.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d19d6e1:::benchmark_data_smithi188_89276_object154464:head# 2024-04-23T16:25:48.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d1a9d09:::benchmark_data_smithi188_89276_object118387:head# 2024-04-23T16:25:48.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d1b745e:::benchmark_data_smithi188_89276_object58644:head# 2024-04-23T16:25:48.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d1c0afa:::benchmark_data_smithi188_89276_object136757:head# 2024-04-23T16:25:48.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d1c4d03:::benchmark_data_smithi188_89276_object83673:head# 2024-04-23T16:25:48.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d1c8088:::benchmark_data_smithi188_89276_object98273:head# 2024-04-23T16:25:48.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d1d4a26:::benchmark_data_smithi188_89276_object82326:head# 2024-04-23T16:25:48.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d1d550f:::benchmark_data_smithi188_89276_object142537:head# 2024-04-23T16:25:48.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.597 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d1dd2e6:::benchmark_data_smithi188_89276_object164886:head# 2024-04-23T16:25:48.597 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.597 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d1de97c:::benchmark_data_smithi188_89276_object57417:head# 2024-04-23T16:25:48.597 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.598 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d1e40f9:::benchmark_data_smithi188_89276_object69191:head# 2024-04-23T16:25:48.598 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.598 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d1edc64:::benchmark_data_smithi188_89276_object123522:head# 2024-04-23T16:25:48.598 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.598 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d1fa974:::benchmark_data_smithi188_89276_object52290:head# 2024-04-23T16:25:48.598 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.598 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d1fb5c6:::benchmark_data_smithi188_89276_object106468:head# 2024-04-23T16:25:48.598 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.598 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d21b4fa:::benchmark_data_smithi188_89276_object131364:head# 2024-04-23T16:25:48.598 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.598 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d22d4ec:::benchmark_data_smithi188_89276_object128385:head# 2024-04-23T16:25:48.599 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.599 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d22e897:::benchmark_data_smithi188_89276_object106785:head# 2024-04-23T16:25:48.599 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.599 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d239b91:::benchmark_data_smithi188_89276_object135575:head# 2024-04-23T16:25:48.599 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.599 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d239c8c:::benchmark_data_smithi188_89276_object106728:head# 2024-04-23T16:25:48.599 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.600 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d2440a4:::benchmark_data_smithi188_89276_object85268:head# 2024-04-23T16:25:48.600 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.600 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d2461c0:::benchmark_data_smithi188_89276_object67977:head# 2024-04-23T16:25:48.600 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.600 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d24856c:::benchmark_data_smithi188_89276_object153600:head# 2024-04-23T16:25:48.600 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d250ff1:::benchmark_data_smithi188_89276_object126586:head# 2024-04-23T16:25:48.601 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d27543a:::benchmark_data_smithi188_89276_object54162:head# 2024-04-23T16:25:48.601 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d276ac5:::benchmark_data_smithi188_89276_object148502:head# 2024-04-23T16:25:48.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d278d22:::benchmark_data_smithi188_89276_object47857:head# 2024-04-23T16:25:48.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d27a9af:::benchmark_data_smithi188_89276_object148574:head# 2024-04-23T16:25:48.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d284b78:::benchmark_data_smithi188_89276_object105525:head# 2024-04-23T16:25:48.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d28ecf0:::benchmark_data_smithi188_89276_object138061:head# 2024-04-23T16:25:48.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d296435:::benchmark_data_smithi188_89276_object79624:head# 2024-04-23T16:25:48.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d2b583b:::benchmark_data_smithi188_89276_object64836:head# 2024-04-23T16:25:48.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.631 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d2b9323:::benchmark_data_smithi188_89276_object54535:head# 2024-04-23T16:25:48.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.633 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d2b9e74:::benchmark_data_smithi188_89276_object108258:head# 2024-04-23T16:25:48.633 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.633 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d2cee9f:::benchmark_data_smithi188_89276_object151653:head# 2024-04-23T16:25:48.633 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d2d7c06:::benchmark_data_smithi188_89276_object139054:head# 2024-04-23T16:25:48.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d2ddd39:::benchmark_data_smithi188_89276_object162821:head# 2024-04-23T16:25:48.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d2de75e:::benchmark_data_smithi188_89276_object68814:head# 2024-04-23T16:25:48.635 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.645 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d2e0671:::benchmark_data_smithi188_89276_object69914:head# 2024-04-23T16:25:48.645 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.645 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d2fa4de:::benchmark_data_smithi188_89276_object72733:head# 2024-04-23T16:25:48.645 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.646 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d2ff847:::benchmark_data_smithi188_89276_object67410:head# 2024-04-23T16:25:48.646 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.646 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d30d397:::benchmark_data_smithi188_89276_object90874:head# 2024-04-23T16:25:48.646 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.646 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d30e18a:::benchmark_data_smithi188_89276_object60700:head# 2024-04-23T16:25:48.646 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d31c6ed:::benchmark_data_smithi188_89276_object58473:head# 2024-04-23T16:25:48.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d31f81b:::benchmark_data_smithi188_89276_object107118:head# 2024-04-23T16:25:48.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d321cdc:::benchmark_data_smithi188_89276_object114685:head# 2024-04-23T16:25:48.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d32673e:::benchmark_data_smithi188_89276_object81078:head# 2024-04-23T16:25:48.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d3295a5:::benchmark_data_smithi188_89276_object45919:head# 2024-04-23T16:25:48.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d32fe8c:::benchmark_data_smithi188_89276_object166509:head# 2024-04-23T16:25:48.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d332ea7:::benchmark_data_smithi188_89276_object152640:head# 2024-04-23T16:25:48.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d3581a3:::benchmark_data_smithi188_89276_object89184:head# 2024-04-23T16:25:48.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d364664:::benchmark_data_smithi188_89276_object72273:head# 2024-04-23T16:25:48.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d366451:::benchmark_data_smithi188_89276_object129502:head# 2024-04-23T16:25:48.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d36ca2d:::benchmark_data_smithi188_89276_object81571:head# 2024-04-23T16:25:48.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d36ed4e:::benchmark_data_smithi188_89276_object161784:head# 2024-04-23T16:25:48.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d3748e8:::benchmark_data_smithi188_89276_object136105:head# 2024-04-23T16:25:48.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d374c38:::benchmark_data_smithi188_89276_object96298:head# 2024-04-23T16:25:48.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.652 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d378840:::benchmark_data_smithi188_89276_object41121:head# 2024-04-23T16:25:48.652 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.652 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d393f52:::benchmark_data_smithi188_89276_object70756:head# 2024-04-23T16:25:48.652 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.652 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d39e8de:::benchmark_data_smithi188_89276_object151327:head# 2024-04-23T16:25:48.652 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.653 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d3abf0f:::benchmark_data_smithi188_89276_object61989:head# 2024-04-23T16:25:48.653 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d3b356b:::benchmark_data_smithi188_89276_object167990:head# 2024-04-23T16:25:48.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d3c1bbe:::benchmark_data_smithi188_89276_object141529:head# 2024-04-23T16:25:48.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d3c3962:::benchmark_data_smithi188_89276_object72387:head# 2024-04-23T16:25:48.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d3c3ae3:::benchmark_data_smithi188_89276_object153086:head# 2024-04-23T16:25:48.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d3c5e17:::benchmark_data_smithi188_89276_object75378:head# 2024-04-23T16:25:48.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d3d8176:::benchmark_data_smithi188_89276_object97561:head# 2024-04-23T16:25:48.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d3d8253:::benchmark_data_smithi188_89276_object73968:head# 2024-04-23T16:25:48.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d3ef1d6:::benchmark_data_smithi188_89276_object153618:head# 2024-04-23T16:25:48.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d3f97ac:::benchmark_data_smithi188_89276_object115837:head# 2024-04-23T16:25:48.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d3fee7d:::benchmark_data_smithi188_89276_object143256:head# 2024-04-23T16:25:48.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d40ec75:::benchmark_data_smithi188_89276_object127019:head# 2024-04-23T16:25:48.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d40fdd8:::benchmark_data_smithi188_89276_object45977:head# 2024-04-23T16:25:48.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d41038f:::benchmark_data_smithi188_89276_object120031:head# 2024-04-23T16:25:48.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d411473:::benchmark_data_smithi188_89276_object160392:head# 2024-04-23T16:25:48.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d4367c9:::benchmark_data_smithi188_89276_object101247:head# 2024-04-23T16:25:48.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d43a33b:::benchmark_data_smithi188_89276_object41107:head# 2024-04-23T16:25:48.663 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d43c63a:::benchmark_data_smithi188_89276_object115918:head# 2024-04-23T16:25:48.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d43e604:::benchmark_data_smithi188_89276_object80738:head# 2024-04-23T16:25:48.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d447874:::benchmark_data_smithi188_89276_object47651:head# 2024-04-23T16:25:48.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.668 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d44cb4b:::benchmark_data_smithi188_89276_object101435:head# 2024-04-23T16:25:48.668 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d44d212:::benchmark_data_smithi188_89276_object108894:head# 2024-04-23T16:25:48.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d4520b1:::benchmark_data_smithi188_89276_object52281:head# 2024-04-23T16:25:48.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d4546b5:::benchmark_data_smithi188_89276_object66137:head# 2024-04-23T16:25:48.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d456c92:::benchmark_data_smithi188_89276_object66788:head# 2024-04-23T16:25:48.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d462964:::benchmark_data_smithi188_89276_object55473:head# 2024-04-23T16:25:48.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d4691e8:::benchmark_data_smithi188_89276_object147853:head# 2024-04-23T16:25:48.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d46e453:::benchmark_data_smithi188_89276_object132042:head# 2024-04-23T16:25:48.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d472c96:::benchmark_data_smithi188_89276_object137552:head# 2024-04-23T16:25:48.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d4772ba:::benchmark_data_smithi188_89276_object80316:head# 2024-04-23T16:25:48.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d4869dd:::benchmark_data_smithi188_89276_object127617:head# 2024-04-23T16:25:48.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d48a73c:::benchmark_data_smithi188_89276_object102049:head# 2024-04-23T16:25:48.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d48b178:::benchmark_data_smithi188_89276_object127070:head# 2024-04-23T16:25:48.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d490f64:::benchmark_data_smithi188_89276_object117597:head# 2024-04-23T16:25:48.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d499c4f:::benchmark_data_smithi188_89276_object154018:head# 2024-04-23T16:25:48.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.677 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d49fbb9:::benchmark_data_smithi188_89276_object110759:head# 2024-04-23T16:25:48.677 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.677 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d4a349f:::benchmark_data_smithi188_89276_object166513:head# 2024-04-23T16:25:48.677 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d4b1866:::benchmark_data_smithi188_89276_object149823:head# 2024-04-23T16:25:48.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d4b407d:::benchmark_data_smithi188_89276_object67831:head# 2024-04-23T16:25:48.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d4bc118:::benchmark_data_smithi188_89276_object61551:head# 2024-04-23T16:25:48.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d4d4a37:::benchmark_data_smithi188_89276_object78710:head# 2024-04-23T16:25:48.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d4d8b64:::benchmark_data_smithi188_89276_object143015:head# 2024-04-23T16:25:48.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d4dcbc7:::benchmark_data_smithi188_89276_object114109:head# 2024-04-23T16:25:48.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d4dcdd8:::benchmark_data_smithi188_89276_object115874:head# 2024-04-23T16:25:48.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d4e2b39:::benchmark_data_smithi188_89276_object88932:head# 2024-04-23T16:25:48.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d4e9a4a:::benchmark_data_smithi188_89276_object82915:head# 2024-04-23T16:25:48.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d4eadc0:::benchmark_data_smithi188_89276_object157441:head# 2024-04-23T16:25:48.683 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.683 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d4ede40:::benchmark_data_smithi188_89276_object72960:head# 2024-04-23T16:25:48.683 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.683 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d4fdb9f:::benchmark_data_smithi188_89276_object58191:head# 2024-04-23T16:25:48.683 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.683 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d5007ac:::benchmark_data_smithi188_89276_object116438:head# 2024-04-23T16:25:48.683 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d502a49:::benchmark_data_smithi188_89276_object71217:head# 2024-04-23T16:25:48.684 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d5069c4:::benchmark_data_smithi188_89276_object137946:head# 2024-04-23T16:25:48.684 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d507916:::benchmark_data_smithi188_89276_object141889:head# 2024-04-23T16:25:48.684 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.686 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d50df12:::benchmark_data_smithi188_89276_object109518:head# 2024-04-23T16:25:48.686 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d513c35:::benchmark_data_smithi188_89276_object137195:head# 2024-04-23T16:25:48.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d514227:::benchmark_data_smithi188_89276_object50166:head# 2024-04-23T16:25:48.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d517a40:::benchmark_data_smithi188_89276_object132198:head# 2024-04-23T16:25:48.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d51a6e6:::benchmark_data_smithi188_89276_object109019:head# 2024-04-23T16:25:48.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d52781b:::benchmark_data_smithi188_89276_object149540:head# 2024-04-23T16:25:48.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d52c0fc:::benchmark_data_smithi188_89276_object54997:head# 2024-04-23T16:25:48.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d533715:::benchmark_data_smithi188_89276_object165718:head# 2024-04-23T16:25:48.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d53899a:::benchmark_data_smithi188_89276_object100487:head# 2024-04-23T16:25:48.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d53edf0:::benchmark_data_smithi188_89276_object167959:head# 2024-04-23T16:25:48.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d545acd:::benchmark_data_smithi188_89276_object55209:head# 2024-04-23T16:25:48.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d56ca83:::benchmark_data_smithi188_89276_object74742:head# 2024-04-23T16:25:48.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d56fd8f:::benchmark_data_smithi188_89276_object66758:head# 2024-04-23T16:25:48.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d57360f:::benchmark_data_smithi188_89276_object88197:head# 2024-04-23T16:25:48.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d57fa06:::benchmark_data_smithi188_89276_object132038:head# 2024-04-23T16:25:48.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d590a77:::benchmark_data_smithi188_89276_object152794:head# 2024-04-23T16:25:48.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d599997:::benchmark_data_smithi188_89276_object114132:head# 2024-04-23T16:25:48.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d5b5a13:::benchmark_data_smithi188_89276_object160773:head# 2024-04-23T16:25:48.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d5bf5db:::benchmark_data_smithi188_89276_object46927:head# 2024-04-23T16:25:48.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d5c8469:::benchmark_data_smithi188_89276_object145341:head# 2024-04-23T16:25:48.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d5cb1c0:::benchmark_data_smithi188_89276_object79410:head# 2024-04-23T16:25:48.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d5d66b1:::benchmark_data_smithi188_89276_object110035:head# 2024-04-23T16:25:48.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d5d9ae4:::benchmark_data_smithi188_89276_object115843:head# 2024-04-23T16:25:48.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d5f21da:::benchmark_data_smithi188_89276_object51537:head# 2024-04-23T16:25:48.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d60846d:::benchmark_data_smithi188_89276_object110875:head# 2024-04-23T16:25:48.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d60a336:::benchmark_data_smithi188_89276_object57964:head# 2024-04-23T16:25:48.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d61b659:::benchmark_data_smithi188_89276_object63202:head# 2024-04-23T16:25:48.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d61e7fe:::benchmark_data_smithi188_89276_object162686:head# 2024-04-23T16:25:48.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d622d6e:::benchmark_data_smithi188_89276_object111981:head# 2024-04-23T16:25:48.696 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d631a84:::benchmark_data_smithi188_89276_object128820:head# 2024-04-23T16:25:48.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d642235:::benchmark_data_smithi188_89276_object54735:head# 2024-04-23T16:25:48.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d65c0ea:::benchmark_data_smithi188_89276_object67355:head# 2024-04-23T16:25:48.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d663619:::benchmark_data_smithi188_89276_object158354:head# 2024-04-23T16:25:48.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d668ddb:::benchmark_data_smithi188_89276_object87244:head# 2024-04-23T16:25:48.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d66efab:::benchmark_data_smithi188_89276_object44616:head# 2024-04-23T16:25:48.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d66fb4a:::benchmark_data_smithi188_89276_object131182:head# 2024-04-23T16:25:48.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d671a66:::benchmark_data_smithi188_89276_object79764:head# 2024-04-23T16:25:48.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d673670:::benchmark_data_smithi188_89276_object60125:head# 2024-04-23T16:25:48.699 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.699 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d682bc0:::benchmark_data_smithi188_89276_object161532:head# 2024-04-23T16:25:48.699 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.699 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d68abd5:::benchmark_data_smithi188_89276_object91806:head# 2024-04-23T16:25:48.699 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d68c6a8:::benchmark_data_smithi188_89276_object65347:head# 2024-04-23T16:25:48.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d68c8ca:::benchmark_data_smithi188_89276_object98825:head# 2024-04-23T16:25:48.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d68d795:::benchmark_data_smithi188_89276_object129196:head# 2024-04-23T16:25:48.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d69de52:::benchmark_data_smithi188_89276_object153793:head# 2024-04-23T16:25:48.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d6a718e:::benchmark_data_smithi188_89276_object51043:head# 2024-04-23T16:25:48.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d6ae519:::benchmark_data_smithi188_89276_object107464:head# 2024-04-23T16:25:48.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d6afbbe:::benchmark_data_smithi188_89276_object64789:head# 2024-04-23T16:25:48.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d6b2c8d:::benchmark_data_smithi188_89276_object51950:head# 2024-04-23T16:25:48.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d6d0d3b:::benchmark_data_smithi188_89276_object70386:head# 2024-04-23T16:25:48.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d6d23f2:::benchmark_data_smithi188_89276_object102580:head# 2024-04-23T16:25:48.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d6d608b:::benchmark_data_smithi188_89276_object122875:head# 2024-04-23T16:25:48.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d6e0803:::benchmark_data_smithi188_89276_object62209:head# 2024-04-23T16:25:48.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d6e51a2:::benchmark_data_smithi188_89276_object128019:head# 2024-04-23T16:25:48.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d6e9025:::benchmark_data_smithi188_89276_object121895:head# 2024-04-23T16:25:48.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d6fae03:::benchmark_data_smithi188_89276_object129955:head# 2024-04-23T16:25:48.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d704f7e:::benchmark_data_smithi188_89276_object130864:head# 2024-04-23T16:25:48.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d70ab72:::benchmark_data_smithi188_89276_object108833:head# 2024-04-23T16:25:48.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d70c6c7:::benchmark_data_smithi188_89276_object138846:head# 2024-04-23T16:25:48.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d710665:::benchmark_data_smithi188_89276_object123934:head# 2024-04-23T16:25:48.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.710 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7118cc:::benchmark_data_smithi188_89276_object111573:head# 2024-04-23T16:25:48.710 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d712e1e:::benchmark_data_smithi188_89276_object145126:head# 2024-04-23T16:25:48.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d722a51:::benchmark_data_smithi188_89276_object87645:head# 2024-04-23T16:25:48.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d730a56:::benchmark_data_smithi188_89276_object72920:head# 2024-04-23T16:25:48.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7323ec:::benchmark_data_smithi188_89276_object113376:head# 2024-04-23T16:25:48.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d74c4cb:::benchmark_data_smithi188_89276_object47607:head# 2024-04-23T16:25:48.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d74cc40:::benchmark_data_smithi188_89276_object95121:head# 2024-04-23T16:25:48.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d750f7f:::benchmark_data_smithi188_89276_object116367:head# 2024-04-23T16:25:48.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d75c4f3:::benchmark_data_smithi188_89276_object48690:head# 2024-04-23T16:25:48.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7641ab:::benchmark_data_smithi188_89276_object138737:head# 2024-04-23T16:25:48.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.714 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d769abe:::benchmark_data_smithi188_89276_object56904:head# 2024-04-23T16:25:48.714 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.714 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d76d921:::benchmark_data_smithi188_89276_object76431:head# 2024-04-23T16:25:48.714 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.714 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d76fd78:::benchmark_data_smithi188_89276_object88321:head# 2024-04-23T16:25:48.714 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d77f2ae:::benchmark_data_smithi188_89276_object104870:head# 2024-04-23T16:25:48.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d789810:::benchmark_data_smithi188_89276_object81617:head# 2024-04-23T16:25:48.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d78ad41:::benchmark_data_smithi188_89276_object78256:head# 2024-04-23T16:25:48.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7a1ff9:::benchmark_data_smithi188_89276_object160864:head# 2024-04-23T16:25:48.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7a8c40:::benchmark_data_smithi188_89276_object81755:head# 2024-04-23T16:25:48.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7aa2fa:::benchmark_data_smithi188_89276_object147424:head# 2024-04-23T16:25:48.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7aae54:::benchmark_data_smithi188_89276_object124674:head# 2024-04-23T16:25:48.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7b0b33:::benchmark_data_smithi188_89276_object100423:head# 2024-04-23T16:25:48.727 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7b6fa9:::benchmark_data_smithi188_89276_object105394:head# 2024-04-23T16:25:48.727 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7be5ea:::benchmark_data_smithi188_89276_object105705:head# 2024-04-23T16:25:48.727 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.728 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7bf74a:::benchmark_data_smithi188_89276_object88666:head# 2024-04-23T16:25:48.728 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.728 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7c3dd1:::benchmark_data_smithi188_89276_object68960:head# 2024-04-23T16:25:48.728 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.729 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7c726c:::benchmark_data_smithi188_89276_object74059:head# 2024-04-23T16:25:48.729 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.729 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7cc83c:::benchmark_data_smithi188_89276_object60239:head# 2024-04-23T16:25:48.729 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.729 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7d2a13:::benchmark_data_smithi188_89276_object114933:head# 2024-04-23T16:25:48.729 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.729 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7d819a:::benchmark_data_smithi188_89276_object129897:head# 2024-04-23T16:25:48.729 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7dd9d1:::benchmark_data_smithi188_89276_object44252:head# 2024-04-23T16:25:48.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7e0380:::benchmark_data_smithi188_89276_object146802:head# 2024-04-23T16:25:48.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7e2be6:::benchmark_data_smithi188_89276_object45626:head# 2024-04-23T16:25:48.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7e41f7:::benchmark_data_smithi188_89276_object103673:head# 2024-04-23T16:25:48.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7e773c:::benchmark_data_smithi188_89276_object127214:head# 2024-04-23T16:25:48.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7ea3ec:::benchmark_data_smithi188_89276_object104850:head# 2024-04-23T16:25:48.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7ee347:::benchmark_data_smithi188_89276_object72190:head# 2024-04-23T16:25:48.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7efadd:::benchmark_data_smithi188_89276_object49703:head# 2024-04-23T16:25:48.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d7f7081:::benchmark_data_smithi188_89276_object128111:head# 2024-04-23T16:25:48.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d803584:::benchmark_data_smithi188_89276_object79768:head# 2024-04-23T16:25:48.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d803da0:::benchmark_data_smithi188_89276_object81122:head# 2024-04-23T16:25:48.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d80a128:::benchmark_data_smithi188_89276_object141301:head# 2024-04-23T16:25:48.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d81d02b:::benchmark_data_smithi188_89276_object90706:head# 2024-04-23T16:25:48.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d81f280:::benchmark_data_smithi188_89276_object138315:head# 2024-04-23T16:25:48.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d824cdd:::benchmark_data_smithi188_89276_object118612:head# 2024-04-23T16:25:48.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d82c66d:::benchmark_data_smithi188_89276_object163305:head# 2024-04-23T16:25:48.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d82f001:::benchmark_data_smithi188_89276_object142003:head# 2024-04-23T16:25:48.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d8351e4:::benchmark_data_smithi188_89276_object119420:head# 2024-04-23T16:25:48.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d837711:::benchmark_data_smithi188_89276_object81331:head# 2024-04-23T16:25:48.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d838046:::benchmark_data_smithi188_89276_object159829:head# 2024-04-23T16:25:48.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d8381da:::benchmark_data_smithi188_89276_object162457:head# 2024-04-23T16:25:48.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.737 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d83b51a:::benchmark_data_smithi188_89276_object114731:head# 2024-04-23T16:25:48.737 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d83e8e4:::benchmark_data_smithi188_89276_object150915:head# 2024-04-23T16:25:48.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d840ad8:::benchmark_data_smithi188_89276_object102584:head# 2024-04-23T16:25:48.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d844568:::benchmark_data_smithi188_89276_object43276:head# 2024-04-23T16:25:48.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d844f74:::benchmark_data_smithi188_89276_object146848:head# 2024-04-23T16:25:48.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.742 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d85053f:::benchmark_data_smithi188_89276_object83970:head# 2024-04-23T16:25:48.742 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.744 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d850799:::benchmark_data_smithi188_89276_object86282:head# 2024-04-23T16:25:48.744 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.744 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d852621:::benchmark_data_smithi188_89276_object71821:head# 2024-04-23T16:25:48.744 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.744 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d85fb73:::benchmark_data_smithi188_89276_object139153:head# 2024-04-23T16:25:48.744 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d86691d:::benchmark_data_smithi188_89276_object71969:head# 2024-04-23T16:25:48.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d86b5d6:::benchmark_data_smithi188_89276_object61779:head# 2024-04-23T16:25:48.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d888b4c:::benchmark_data_smithi188_89276_object48956:head# 2024-04-23T16:25:48.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d88e866:::benchmark_data_smithi188_89276_object61355:head# 2024-04-23T16:25:48.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d89d93f:::benchmark_data_smithi188_89276_object80337:head# 2024-04-23T16:25:48.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d8a3783:::benchmark_data_smithi188_89276_object60877:head# 2024-04-23T16:25:48.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.753 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d8a6bc7:::benchmark_data_smithi188_89276_object158479:head# 2024-04-23T16:25:48.753 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.753 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d8bbc6e:::benchmark_data_smithi188_89276_object50187:head# 2024-04-23T16:25:48.753 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.753 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d8bc849:::benchmark_data_smithi188_89276_object64211:head# 2024-04-23T16:25:48.753 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.753 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d8c8fb5:::benchmark_data_smithi188_89276_object124552:head# 2024-04-23T16:25:48.753 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.753 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d8d6004:::benchmark_data_smithi188_89276_object65605:head# 2024-04-23T16:25:48.753 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.754 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d8daa6a:::benchmark_data_smithi188_89276_object113258:head# 2024-04-23T16:25:48.754 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.754 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d8dbbcf:::benchmark_data_smithi188_89276_object84072:head# 2024-04-23T16:25:48.754 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.754 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d8def7d:::benchmark_data_smithi188_89276_object70208:head# 2024-04-23T16:25:48.754 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.758 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d9055dc:::benchmark_data_smithi188_89276_object47506:head# 2024-04-23T16:25:48.758 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d90eca2:::benchmark_data_smithi188_89276_object65842:head# 2024-04-23T16:25:48.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.771 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d90fff8:::benchmark_data_smithi188_89276_object102543:head# 2024-04-23T16:25:48.771 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.771 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d924d85:::benchmark_data_smithi188_89276_object45983:head# 2024-04-23T16:25:48.771 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.772 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d92ed54:::benchmark_data_smithi188_89276_object134748:head# 2024-04-23T16:25:48.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d93b5f5:::benchmark_data_smithi188_89276_object56196:head# 2024-04-23T16:25:48.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d93f884:::benchmark_data_smithi188_89276_object142486:head# 2024-04-23T16:25:48.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d961c1d:::benchmark_data_smithi188_89276_object43638:head# 2024-04-23T16:25:48.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d966910:::benchmark_data_smithi188_89276_object152080:head# 2024-04-23T16:25:48.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d970fbd:::benchmark_data_smithi188_89276_object114204:head# 2024-04-23T16:25:48.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.774 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d97a17e:::benchmark_data_smithi188_89276_object99805:head# 2024-04-23T16:25:48.774 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.777 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d97abbe:::benchmark_data_smithi188_89276_object74636:head# 2024-04-23T16:25:48.777 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.777 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d9853ba:::benchmark_data_smithi188_89276_object137122:head# 2024-04-23T16:25:48.778 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d9880b8:::benchmark_data_smithi188_89276_object130504:head# 2024-04-23T16:25:48.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d9a4385:::benchmark_data_smithi188_89276_object100096:head# 2024-04-23T16:25:48.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d9ac497:::benchmark_data_smithi188_89276_object63610:head# 2024-04-23T16:25:48.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.791 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d9ae761:::benchmark_data_smithi188_89276_object74689:head# 2024-04-23T16:25:48.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.791 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d9b0d98:::benchmark_data_smithi188_89276_object77279:head# 2024-04-23T16:25:48.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d9b28af:::benchmark_data_smithi188_89276_object163780:head# 2024-04-23T16:25:48.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d9b2976:::benchmark_data_smithi188_89276_object153799:head# 2024-04-23T16:25:48.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d9dda70:::benchmark_data_smithi188_89276_object62920:head# 2024-04-23T16:25:48.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d9ddff6:::benchmark_data_smithi188_89276_object165043:head# 2024-04-23T16:25:48.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d9e4363:::benchmark_data_smithi188_89276_object65776:head# 2024-04-23T16:25:48.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d9e53de:::benchmark_data_smithi188_89276_object80556:head# 2024-04-23T16:25:48.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d9eb4a3:::benchmark_data_smithi188_89276_object112163:head# 2024-04-23T16:25:48.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d9f1e70:::benchmark_data_smithi188_89276_object86146:head# 2024-04-23T16:25:48.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d9f29b5:::benchmark_data_smithi188_89276_object100678:head# 2024-04-23T16:25:48.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2d9fdd12:::benchmark_data_smithi188_89276_object98623:head# 2024-04-23T16:25:48.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2da01c31:::benchmark_data_smithi188_89276_object160087:head# 2024-04-23T16:25:48.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2da0844a:::benchmark_data_smithi188_89276_object148358:head# 2024-04-23T16:25:48.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2da0f6e2:::benchmark_data_smithi188_89276_object55181:head# 2024-04-23T16:25:48.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2da22b0c:::benchmark_data_smithi188_89276_object134831:head# 2024-04-23T16:25:48.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2da3743c:::benchmark_data_smithi188_89276_object158218:head# 2024-04-23T16:25:48.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2da37df9:::benchmark_data_smithi188_89276_object109624:head# 2024-04-23T16:25:48.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2da39394:::benchmark_data_smithi188_89276_object131916:head# 2024-04-23T16:25:48.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2da3e179:::benchmark_data_smithi188_89276_object72052:head# 2024-04-23T16:25:48.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2da3fa71:::benchmark_data_smithi188_89276_object123823:head# 2024-04-23T16:25:48.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2da4044b:::benchmark_data_smithi188_89276_object62952:head# 2024-04-23T16:25:48.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2da4587a:::benchmark_data_smithi188_89276_object104200:head# 2024-04-23T16:25:48.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2da5151a:::benchmark_data_smithi188_89276_object105041:head# 2024-04-23T16:25:48.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2da67d5c:::benchmark_data_smithi188_89276_object133680:head# 2024-04-23T16:25:48.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2da6bda7:::benchmark_data_smithi188_89276_object50843:head# 2024-04-23T16:25:48.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2da6c148:::benchmark_data_smithi188_89276_object129289:head# 2024-04-23T16:25:48.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2da6db6e:::benchmark_data_smithi188_89276_object109148:head# 2024-04-23T16:25:48.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2da73b4f:::benchmark_data_smithi188_89276_object44131:head# 2024-04-23T16:25:48.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2da9197c:::benchmark_data_smithi188_89276_object47440:head# 2024-04-23T16:25:48.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2da9a7a6:::benchmark_data_smithi188_89276_object68034:head# 2024-04-23T16:25:48.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.816 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2daa27de:::benchmark_data_smithi188_89276_object52692:head# 2024-04-23T16:25:48.816 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.816 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2daa433f:::benchmark_data_smithi188_89276_object93961:head# 2024-04-23T16:25:48.816 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.816 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2daaaa44:::benchmark_data_smithi188_89276_object110184:head# 2024-04-23T16:25:48.816 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.817 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dab17ea:::benchmark_data_smithi188_89276_object84727:head# 2024-04-23T16:25:48.817 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.817 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dab3e22:::benchmark_data_smithi188_89276_object58187:head# 2024-04-23T16:25:48.817 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dab6036:::benchmark_data_smithi188_89276_object45577:head# 2024-04-23T16:25:48.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dab6213:::benchmark_data_smithi188_89276_object70375:head# 2024-04-23T16:25:48.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dab893f:::benchmark_data_smithi188_89276_object136693:head# 2024-04-23T16:25:48.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dabb898:::benchmark_data_smithi188_89276_object141848:head# 2024-04-23T16:25:48.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dac17de:::benchmark_data_smithi188_89276_object86430:head# 2024-04-23T16:25:48.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dad09e8:::benchmark_data_smithi188_89276_object134528:head# 2024-04-23T16:25:48.819 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.819 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dae95cb:::benchmark_data_smithi188_89276_object94682:head# 2024-04-23T16:25:48.819 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2daece81:::benchmark_data_smithi188_89276_object87589:head# 2024-04-23T16:25:48.820 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2daf3b2b:::benchmark_data_smithi188_89276_object94257:head# 2024-04-23T16:25:48.820 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2daf8c34:::benchmark_data_smithi188_89276_object86575:head# 2024-04-23T16:25:48.820 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dafb5c4:::benchmark_data_smithi188_89276_object109215:head# 2024-04-23T16:25:48.820 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db08ed5:::benchmark_data_smithi188_89276_object133479:head# 2024-04-23T16:25:48.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db0b77e:::benchmark_data_smithi188_89276_object82987:head# 2024-04-23T16:25:48.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db0e2f4:::benchmark_data_smithi188_89276_object41084:head# 2024-04-23T16:25:48.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db1c6a1:::benchmark_data_smithi188_89276_object45533:head# 2024-04-23T16:25:48.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db230dc:::benchmark_data_smithi188_89276_object87410:head# 2024-04-23T16:25:48.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db24396:::benchmark_data_smithi188_89276_object151475:head# 2024-04-23T16:25:48.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db27564:::benchmark_data_smithi188_89276_object45226:head# 2024-04-23T16:25:48.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db42a40:::benchmark_data_smithi188_89276_object101493:head# 2024-04-23T16:25:48.824 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.825 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db4bd9b:::benchmark_data_smithi188_89276_object159672:head# 2024-04-23T16:25:48.825 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.825 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db4ca2f:::benchmark_data_smithi188_89276_object64993:head# 2024-04-23T16:25:48.825 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.825 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db4edd1:::benchmark_data_smithi188_89276_object133318:head# 2024-04-23T16:25:48.825 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.825 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db58a93:::benchmark_data_smithi188_89276_object167927:head# 2024-04-23T16:25:48.825 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.825 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db66049:::benchmark_data_smithi188_89276_object161483:head# 2024-04-23T16:25:48.825 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db6abcf:::benchmark_data_smithi188_89276_object127127:head# 2024-04-23T16:25:48.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db6abf3:::benchmark_data_smithi188_89276_object119565:head# 2024-04-23T16:25:48.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db73b41:::benchmark_data_smithi188_89276_object127181:head# 2024-04-23T16:25:48.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db73c01:::benchmark_data_smithi188_89276_object47474:head# 2024-04-23T16:25:48.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db7c9e5:::benchmark_data_smithi188_89276_object57042:head# 2024-04-23T16:25:48.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db7fed4:::benchmark_data_smithi188_89276_object57708:head# 2024-04-23T16:25:48.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db805c7:::benchmark_data_smithi188_89276_object116669:head# 2024-04-23T16:25:48.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2db86704:::benchmark_data_smithi188_89276_object138259:head# 2024-04-23T16:25:48.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dba02b2:::benchmark_data_smithi188_89276_object129453:head# 2024-04-23T16:25:48.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.830 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dba49b2:::benchmark_data_smithi188_89276_object86042:head# 2024-04-23T16:25:48.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.830 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dbd0aa0:::benchmark_data_smithi188_89276_object138363:head# 2024-04-23T16:25:48.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.830 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dbd9713:::benchmark_data_smithi188_89276_object144814:head# 2024-04-23T16:25:48.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.830 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dbe4caf:::benchmark_data_smithi188_89276_object139181:head# 2024-04-23T16:25:48.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.830 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dbede28:::benchmark_data_smithi188_89276_object158046:head# 2024-04-23T16:25:48.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.830 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dbee0de:::benchmark_data_smithi188_89276_object59317:head# 2024-04-23T16:25:48.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.830 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc036b7:::benchmark_data_smithi188_89276_object77082:head# 2024-04-23T16:25:48.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc061ca:::benchmark_data_smithi188_89276_object89275:head# 2024-04-23T16:25:48.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc0a37b:::benchmark_data_smithi188_89276_object132743:head# 2024-04-23T16:25:48.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc0c097:::benchmark_data_smithi188_89276_object118763:head# 2024-04-23T16:25:48.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc0d63b:::benchmark_data_smithi188_89276_object101085:head# 2024-04-23T16:25:48.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc0e4b3:::benchmark_data_smithi188_89276_object60063:head# 2024-04-23T16:25:48.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.834 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc15a11:::benchmark_data_smithi188_89276_object168281:head# 2024-04-23T16:25:48.834 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.834 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc19f4d:::benchmark_data_smithi188_89276_object40777:head# 2024-04-23T16:25:48.834 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.834 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc1a14f:::benchmark_data_smithi188_89276_object102662:head# 2024-04-23T16:25:48.834 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.834 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc1f60c:::benchmark_data_smithi188_89276_object154075:head# 2024-04-23T16:25:48.834 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.834 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc28991:::benchmark_data_smithi188_89276_object163409:head# 2024-04-23T16:25:48.834 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.835 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc2e092:::benchmark_data_smithi188_89276_object93770:head# 2024-04-23T16:25:48.835 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.835 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc2fe85:::benchmark_data_smithi188_89276_object153414:head# 2024-04-23T16:25:48.835 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.835 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc3323a:::benchmark_data_smithi188_89276_object66284:head# 2024-04-23T16:25:48.835 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.835 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc45694:::benchmark_data_smithi188_89276_object50946:head# 2024-04-23T16:25:48.835 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.835 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc497fc:::benchmark_data_smithi188_89276_object129130:head# 2024-04-23T16:25:48.835 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.835 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc4b159:::benchmark_data_smithi188_89276_object120676:head# 2024-04-23T16:25:48.835 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.835 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc56052:::benchmark_data_smithi188_89276_object121514:head# 2024-04-23T16:25:48.835 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.836 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc61562:::benchmark_data_smithi188_89276_object76213:head# 2024-04-23T16:25:48.836 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.838 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc615ef:::benchmark_data_smithi188_89276_object159024:head# 2024-04-23T16:25:48.838 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.838 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc647a0:::benchmark_data_smithi188_89276_object159239:head# 2024-04-23T16:25:48.839 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.839 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc64c5e:::benchmark_data_smithi188_89276_object42041:head# 2024-04-23T16:25:48.839 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.839 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc663ab:::benchmark_data_smithi188_89276_object90642:head# 2024-04-23T16:25:48.839 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.839 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc6ffd4:::benchmark_data_smithi188_89276_object69849:head# 2024-04-23T16:25:48.839 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.839 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc71632:::benchmark_data_smithi188_89276_object74103:head# 2024-04-23T16:25:48.839 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.839 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc73daa:::benchmark_data_smithi188_89276_object155357:head# 2024-04-23T16:25:48.839 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.839 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc7709d:::benchmark_data_smithi188_89276_object137466:head# 2024-04-23T16:25:48.839 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.840 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc8284d:::benchmark_data_smithi188_89276_object145548:head# 2024-04-23T16:25:48.840 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.840 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc83b01:::benchmark_data_smithi188_89276_object74596:head# 2024-04-23T16:25:48.840 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.840 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc8f5d7:::benchmark_data_smithi188_89276_object72354:head# 2024-04-23T16:25:48.840 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.840 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dc990b6:::benchmark_data_smithi188_89276_object105114:head# 2024-04-23T16:25:48.840 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.840 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dca0335:::benchmark_data_smithi188_89276_object140757:head# 2024-04-23T16:25:48.840 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.840 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dca13af:::benchmark_data_smithi188_89276_object74076:head# 2024-04-23T16:25:48.840 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.841 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dcaee11:::benchmark_data_smithi188_89276_object154161:head# 2024-04-23T16:25:48.841 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.841 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dcba58c:::benchmark_data_smithi188_89276_object102787:head# 2024-04-23T16:25:48.841 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.841 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dcc24a8:::benchmark_data_smithi188_89276_object92379:head# 2024-04-23T16:25:48.841 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dcc473a:::benchmark_data_smithi188_89276_object150735:head# 2024-04-23T16:25:48.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dcd2361:::benchmark_data_smithi188_89276_object124067:head# 2024-04-23T16:25:48.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dcd56c6:::benchmark_data_smithi188_89276_object49835:head# 2024-04-23T16:25:48.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dcd9c18:::benchmark_data_smithi188_89276_object58571:head# 2024-04-23T16:25:48.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dcdbe4d:::benchmark_data_smithi188_89276_object119086:head# 2024-04-23T16:25:48.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dce39bd:::benchmark_data_smithi188_89276_object156837:head# 2024-04-23T16:25:48.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dcf717b:::benchmark_data_smithi188_89276_object122957:head# 2024-04-23T16:25:48.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dcfbeb6:::benchmark_data_smithi188_89276_object141322:head# 2024-04-23T16:25:48.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dd05082:::benchmark_data_smithi188_89276_object113169:head# 2024-04-23T16:25:48.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dd0cd5d:::benchmark_data_smithi188_89276_object50146:head# 2024-04-23T16:25:48.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dd20114:::benchmark_data_smithi188_89276_object102251:head# 2024-04-23T16:25:48.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dd23d1e:::benchmark_data_smithi188_89276_object163448:head# 2024-04-23T16:25:48.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dd456cf:::benchmark_data_smithi188_89276_object62444:head# 2024-04-23T16:25:48.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dd46368:::benchmark_data_smithi188_89276_object73557:head# 2024-04-23T16:25:48.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dd46412:::benchmark_data_smithi188_89276_object99864:head# 2024-04-23T16:25:48.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dd4e4d7:::benchmark_data_smithi188_89276_object107779:head# 2024-04-23T16:25:48.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dd55bf4:::benchmark_data_smithi188_89276_object117897:head# 2024-04-23T16:25:48.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dd66bc1:::benchmark_data_smithi188_89276_object72741:head# 2024-04-23T16:25:48.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dd6e15b:::benchmark_data_smithi188_89276_object70191:head# 2024-04-23T16:25:48.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dd86fe2:::benchmark_data_smithi188_89276_object63386:head# 2024-04-23T16:25:48.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dd8b023:::benchmark_data_smithi188_89276_object97857:head# 2024-04-23T16:25:48.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dd949de:::benchmark_data_smithi188_89276_object64109:head# 2024-04-23T16:25:48.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dd9ebc9:::benchmark_data_smithi188_89276_object90024:head# 2024-04-23T16:25:48.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dda1d8f:::benchmark_data_smithi188_89276_object70616:head# 2024-04-23T16:25:48.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ddad46c:::benchmark_data_smithi188_89276_object112507:head# 2024-04-23T16:25:48.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ddb1adf:::benchmark_data_smithi188_89276_object68114:head# 2024-04-23T16:25:48.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ddb739a:::benchmark_data_smithi188_89276_object148866:head# 2024-04-23T16:25:48.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ddba1f9:::benchmark_data_smithi188_89276_object79714:head# 2024-04-23T16:25:48.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ddbf465:::benchmark_data_smithi188_89276_object162452:head# 2024-04-23T16:25:48.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dde2b14:::benchmark_data_smithi188_89276_object89634:head# 2024-04-23T16:25:48.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ddeacf3:::benchmark_data_smithi188_89276_object74797:head# 2024-04-23T16:25:48.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ddfd9fd:::benchmark_data_smithi188_89276_object158729:head# 2024-04-23T16:25:48.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de0518d:::benchmark_data_smithi188_89276_object117559:head# 2024-04-23T16:25:48.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de05df9:::benchmark_data_smithi188_89276_object55060:head# 2024-04-23T16:25:48.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de0c75e:::benchmark_data_smithi188_89276_object41050:head# 2024-04-23T16:25:48.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de16be7:::benchmark_data_smithi188_89276_object166673:head# 2024-04-23T16:25:48.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de21175:::benchmark_data_smithi188_89276_object102249:head# 2024-04-23T16:25:48.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de298c2:::benchmark_data_smithi188_89276_object161668:head# 2024-04-23T16:25:48.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de2de09:::benchmark_data_smithi188_89276_object93152:head# 2024-04-23T16:25:48.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.853 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de32e5e:::benchmark_data_smithi188_89276_object87762:head# 2024-04-23T16:25:48.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de3a55c:::benchmark_data_smithi188_89276_object74102:head# 2024-04-23T16:25:48.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de4273c:::benchmark_data_smithi188_89276_object134988:head# 2024-04-23T16:25:48.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de4b128:::benchmark_data_smithi188_89276_object63879:head# 2024-04-23T16:25:48.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de552b6:::benchmark_data_smithi188_89276_object76466:head# 2024-04-23T16:25:48.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de58bdc:::benchmark_data_smithi188_89276_object107685:head# 2024-04-23T16:25:48.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de683f1:::benchmark_data_smithi188_89276_object149100:head# 2024-04-23T16:25:48.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de69acd:::benchmark_data_smithi188_89276_object51137:head# 2024-04-23T16:25:48.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de69afb:::benchmark_data_smithi188_89276_object167109:head# 2024-04-23T16:25:48.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de6b4e9:::benchmark_data_smithi188_89276_object135014:head# 2024-04-23T16:25:48.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de6f6ac:::benchmark_data_smithi188_89276_object81001:head# 2024-04-23T16:25:48.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de7978d:::benchmark_data_smithi188_89276_object119392:head# 2024-04-23T16:25:48.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de84775:::benchmark_data_smithi188_89276_object99171:head# 2024-04-23T16:25:48.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2de88e92:::benchmark_data_smithi188_89276_object77964:head# 2024-04-23T16:25:48.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.856 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dea0290:::benchmark_data_smithi188_89276_object166661:head# 2024-04-23T16:25:48.856 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.856 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dea1f75:::benchmark_data_smithi188_89276_object102054:head# 2024-04-23T16:25:48.856 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.856 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dea7d43:::benchmark_data_smithi188_89276_object99782:head# 2024-04-23T16:25:48.857 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.857 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dea9b32:::benchmark_data_smithi188_89276_object116096:head# 2024-04-23T16:25:48.857 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.857 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2deaa545:::benchmark_data_smithi188_89276_object164822:head# 2024-04-23T16:25:48.857 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.857 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2deb0a8d:::benchmark_data_smithi188_89276_object122684:head# 2024-04-23T16:25:48.857 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.857 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2deb0f5f:::benchmark_data_smithi188_89276_object78022:head# 2024-04-23T16:25:48.857 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2deb40c3:::benchmark_data_smithi188_89276_object107466:head# 2024-04-23T16:25:48.862 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2deb82b9:::benchmark_data_smithi188_89276_object121780:head# 2024-04-23T16:25:48.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2debbb27:::benchmark_data_smithi188_89276_object45444:head# 2024-04-23T16:25:48.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2debd707:::benchmark_data_smithi188_89276_object77271:head# 2024-04-23T16:25:48.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dec0804:::benchmark_data_smithi188_89276_object42804:head# 2024-04-23T16:25:48.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dec1c47:::benchmark_data_smithi188_89276_object122948:head# 2024-04-23T16:25:48.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ded1882:::benchmark_data_smithi188_89276_object161292:head# 2024-04-23T16:25:48.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dedd326:::benchmark_data_smithi188_89276_object102326:head# 2024-04-23T16:25:48.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dedda1e:::benchmark_data_smithi188_89276_object119284:head# 2024-04-23T16:25:48.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dede815:::benchmark_data_smithi188_89276_object80318:head# 2024-04-23T16:25:48.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dee1311:::benchmark_data_smithi188_89276_object155547:head# 2024-04-23T16:25:48.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dee291d:::benchmark_data_smithi188_89276_object155795:head# 2024-04-23T16:25:48.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2deeea6f:::benchmark_data_smithi188_89276_object129136:head# 2024-04-23T16:25:48.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2def1156:::benchmark_data_smithi188_89276_object40632:head# 2024-04-23T16:25:48.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2defd3fd:::benchmark_data_smithi188_89276_object42663:head# 2024-04-23T16:25:48.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df0887b:::benchmark_data_smithi188_89276_object150491:head# 2024-04-23T16:25:48.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df0aa3a:::benchmark_data_smithi188_89276_object135550:head# 2024-04-23T16:25:48.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df0c506:::benchmark_data_smithi188_89276_object131399:head# 2024-04-23T16:25:48.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df0fa28:::benchmark_data_smithi188_89276_object114571:head# 2024-04-23T16:25:48.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df10d36:::benchmark_data_smithi188_89276_object60056:head# 2024-04-23T16:25:48.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df16183:::benchmark_data_smithi188_89276_object99355:head# 2024-04-23T16:25:48.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df16c4d:::benchmark_data_smithi188_89276_object51149:head# 2024-04-23T16:25:48.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df19afe:::benchmark_data_smithi188_89276_object99179:head# 2024-04-23T16:25:48.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df21c89:::benchmark_data_smithi188_89276_object127575:head# 2024-04-23T16:25:48.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df29f62:::benchmark_data_smithi188_89276_object109981:head# 2024-04-23T16:25:48.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df35076:::benchmark_data_smithi188_89276_object48805:head# 2024-04-23T16:25:48.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df374fd:::benchmark_data_smithi188_89276_object131873:head# 2024-04-23T16:25:48.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df3e760:::benchmark_data_smithi188_89276_object166945:head# 2024-04-23T16:25:48.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df4f6f3:::benchmark_data_smithi188_89276_object96972:head# 2024-04-23T16:25:48.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df6062a:::benchmark_data_smithi188_89276_object138475:head# 2024-04-23T16:25:48.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df65705:::benchmark_data_smithi188_89276_object90423:head# 2024-04-23T16:25:48.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df679f2:::benchmark_data_smithi188_89276_object67116:head# 2024-04-23T16:25:48.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df689a8:::benchmark_data_smithi188_89276_object71887:head# 2024-04-23T16:25:48.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df69748:::benchmark_data_smithi188_89276_object82583:head# 2024-04-23T16:25:48.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df71426:::benchmark_data_smithi188_89276_object89107:head# 2024-04-23T16:25:48.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df79ed8:::benchmark_data_smithi188_89276_object166898:head# 2024-04-23T16:25:48.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df85839:::benchmark_data_smithi188_89276_object70099:head# 2024-04-23T16:25:48.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df86138:::benchmark_data_smithi188_89276_object138511:head# 2024-04-23T16:25:48.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df945c8:::benchmark_data_smithi188_89276_object114324:head# 2024-04-23T16:25:48.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df954a1:::benchmark_data_smithi188_89276_object136644:head# 2024-04-23T16:25:48.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2df97e62:::benchmark_data_smithi188_89276_object126496:head# 2024-04-23T16:25:48.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dfa104d:::benchmark_data_smithi188_89276_object72521:head# 2024-04-23T16:25:48.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dfa93c2:::benchmark_data_smithi188_89276_object82402:head# 2024-04-23T16:25:48.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dfa9429:::benchmark_data_smithi188_89276_object104062:head# 2024-04-23T16:25:48.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dfc49b1:::benchmark_data_smithi188_89276_object99907:head# 2024-04-23T16:25:48.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dfc9528:::benchmark_data_smithi188_89276_object117974:head# 2024-04-23T16:25:48.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dfd6ddd:::benchmark_data_smithi188_89276_object48465:head# 2024-04-23T16:25:48.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dfd8c24:::benchmark_data_smithi188_89276_object56420:head# 2024-04-23T16:25:48.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dfdaaab:::benchmark_data_smithi188_89276_object100964:head# 2024-04-23T16:25:48.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dfec6a6:::benchmark_data_smithi188_89276_object63626:head# 2024-04-23T16:25:48.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dff0993:::benchmark_data_smithi188_89276_object51446:head# 2024-04-23T16:25:48.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2dff4585:::benchmark_data_smithi188_89276_object83163:head# 2024-04-23T16:25:48.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e0008ff:::benchmark_data_smithi188_89276_object131605:head# 2024-04-23T16:25:48.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e00d9a3:::benchmark_data_smithi188_89276_object89452:head# 2024-04-23T16:25:48.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e012af3:::benchmark_data_smithi188_89276_object79898:head# 2024-04-23T16:25:48.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e0245fc:::benchmark_data_smithi188_89276_object141427:head# 2024-04-23T16:25:48.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e02809d:::benchmark_data_smithi188_89276_object151894:head# 2024-04-23T16:25:48.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e02d2f3:::benchmark_data_smithi188_89276_object100446:head# 2024-04-23T16:25:48.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e04b641:::benchmark_data_smithi188_89276_object144044:head# 2024-04-23T16:25:48.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e057cd3:::benchmark_data_smithi188_89276_object163119:head# 2024-04-23T16:25:48.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e058098:::benchmark_data_smithi188_89276_object109284:head# 2024-04-23T16:25:48.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e059fad:::benchmark_data_smithi188_89276_object165437:head# 2024-04-23T16:25:48.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e05ea19:::benchmark_data_smithi188_89276_object141129:head# 2024-04-23T16:25:48.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e062a8c:::benchmark_data_smithi188_89276_object154401:head# 2024-04-23T16:25:48.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e0724b8:::benchmark_data_smithi188_89276_object120138:head# 2024-04-23T16:25:48.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e0781a2:::benchmark_data_smithi188_89276_object48604:head# 2024-04-23T16:25:48.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e07858b:::benchmark_data_smithi188_89276_object156846:head# 2024-04-23T16:25:48.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e07a260:::benchmark_data_smithi188_89276_object124816:head# 2024-04-23T16:25:48.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e07bd81:::benchmark_data_smithi188_89276_object73642:head# 2024-04-23T16:25:48.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e092906:::benchmark_data_smithi188_89276_object90741:head# 2024-04-23T16:25:48.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e09374c:::benchmark_data_smithi188_89276_object165062:head# 2024-04-23T16:25:48.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e09e053:::benchmark_data_smithi188_89276_object48070:head# 2024-04-23T16:25:48.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e0a0b86:::benchmark_data_smithi188_89276_object142872:head# 2024-04-23T16:25:48.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e0a1d40:::benchmark_data_smithi188_89276_object93825:head# 2024-04-23T16:25:48.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e0a1f42:::benchmark_data_smithi188_89276_object96162:head# 2024-04-23T16:25:48.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e0aa734:::benchmark_data_smithi188_89276_object53100:head# 2024-04-23T16:25:48.884 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e0b654c:::benchmark_data_smithi188_89276_object137056:head# 2024-04-23T16:25:48.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e0bc961:::benchmark_data_smithi188_89276_object69061:head# 2024-04-23T16:25:48.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e0eae2f:::benchmark_data_smithi188_89276_object47542:head# 2024-04-23T16:25:48.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e0f667a:::benchmark_data_smithi188_89276_object136144:head# 2024-04-23T16:25:48.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e0fcfa1:::benchmark_data_smithi188_89276_object146290:head# 2024-04-23T16:25:48.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e0fdf93:::benchmark_data_smithi188_89276_object152276:head# 2024-04-23T16:25:48.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e101d31:::benchmark_data_smithi188_89276_object162483:head# 2024-04-23T16:25:48.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e108ca6:::benchmark_data_smithi188_89276_object42781:head# 2024-04-23T16:25:48.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e116339:::benchmark_data_smithi188_89276_object123496:head# 2024-04-23T16:25:48.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e127adc:::benchmark_data_smithi188_89276_object121051:head# 2024-04-23T16:25:48.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e12a39d:::benchmark_data_smithi188_89276_object116017:head# 2024-04-23T16:25:48.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e12ee91:::benchmark_data_smithi188_89276_object46670:head# 2024-04-23T16:25:48.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e1428df:::benchmark_data_smithi188_89276_object74827:head# 2024-04-23T16:25:48.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e15111f:::benchmark_data_smithi188_89276_object103312:head# 2024-04-23T16:25:48.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e155d3b:::benchmark_data_smithi188_89276_object140771:head# 2024-04-23T16:25:48.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e15c623:::benchmark_data_smithi188_89276_object50482:head# 2024-04-23T16:25:48.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e15f4b0:::benchmark_data_smithi188_89276_object109968:head# 2024-04-23T16:25:48.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e160044:::benchmark_data_smithi188_89276_object73805:head# 2024-04-23T16:25:48.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e162fe1:::benchmark_data_smithi188_89276_object79777:head# 2024-04-23T16:25:48.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e16c954:::benchmark_data_smithi188_89276_object133677:head# 2024-04-23T16:25:48.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e16df18:::benchmark_data_smithi188_89276_object99721:head# 2024-04-23T16:25:48.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e18198c:::benchmark_data_smithi188_89276_object64583:head# 2024-04-23T16:25:48.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e186c39:::benchmark_data_smithi188_89276_object154376:head# 2024-04-23T16:25:48.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e199177:::benchmark_data_smithi188_89276_object130627:head# 2024-04-23T16:25:48.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e1995fa:::benchmark_data_smithi188_89276_object158812:head# 2024-04-23T16:25:48.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e1a8261:::benchmark_data_smithi188_89276_object166709:head# 2024-04-23T16:25:48.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e1abe4c:::benchmark_data_smithi188_89276_object122089:head# 2024-04-23T16:25:48.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e1b02d9:::benchmark_data_smithi188_89276_object69541:head# 2024-04-23T16:25:48.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e1bd22f:::benchmark_data_smithi188_89276_object139894:head# 2024-04-23T16:25:48.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e1bfb32:::benchmark_data_smithi188_89276_object77072:head# 2024-04-23T16:25:48.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e1d1a9e:::benchmark_data_smithi188_89276_object108854:head# 2024-04-23T16:25:48.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.899 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e1d313f:::benchmark_data_smithi188_89276_object41044:head# 2024-04-23T16:25:48.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.899 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e1de052:::benchmark_data_smithi188_89276_object158297:head# 2024-04-23T16:25:48.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.899 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e1ecd4a:::benchmark_data_smithi188_89276_object96492:head# 2024-04-23T16:25:48.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e1ed247:::benchmark_data_smithi188_89276_object128318:head# 2024-04-23T16:25:48.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e1ee7b3:::benchmark_data_smithi188_89276_object68466:head# 2024-04-23T16:25:48.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e1fe866:::benchmark_data_smithi188_89276_object165305:head# 2024-04-23T16:25:48.901 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.901 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e204a83:::benchmark_data_smithi188_89276_object82225:head# 2024-04-23T16:25:48.901 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.901 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2120dc:::benchmark_data_smithi188_89276_object70911:head# 2024-04-23T16:25:48.901 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e212e00:::benchmark_data_smithi188_89276_object142016:head# 2024-04-23T16:25:48.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e21b828:::benchmark_data_smithi188_89276_object90821:head# 2024-04-23T16:25:48.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e21c962:::benchmark_data_smithi188_89276_object122855:head# 2024-04-23T16:25:48.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e221ab7:::benchmark_data_smithi188_89276_object77607:head# 2024-04-23T16:25:48.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e22ca1c:::benchmark_data_smithi188_89276_object89387:head# 2024-04-23T16:25:48.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e22f142:::benchmark_data_smithi188_89276_object104813:head# 2024-04-23T16:25:48.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e23d1dd:::benchmark_data_smithi188_89276_object149029:head# 2024-04-23T16:25:48.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.909 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2408c2:::benchmark_data_smithi188_89276_object118010:head# 2024-04-23T16:25:48.909 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e24c7bc:::benchmark_data_smithi188_89276_object95597:head# 2024-04-23T16:25:48.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2542e8:::benchmark_data_smithi188_89276_object125271:head# 2024-04-23T16:25:48.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e259f28:::benchmark_data_smithi188_89276_object152645:head# 2024-04-23T16:25:48.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e25f3d5:::benchmark_data_smithi188_89276_object82981:head# 2024-04-23T16:25:48.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e26177a:::benchmark_data_smithi188_89276_object122233:head# 2024-04-23T16:25:48.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2682ee:::benchmark_data_smithi188_89276_object139037:head# 2024-04-23T16:25:48.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e26978c:::benchmark_data_smithi188_89276_object148725:head# 2024-04-23T16:25:48.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e26a9f2:::benchmark_data_smithi188_89276_object87593:head# 2024-04-23T16:25:48.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e293c62:::benchmark_data_smithi188_89276_object111433:head# 2024-04-23T16:25:48.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e29d936:::benchmark_data_smithi188_89276_object73417:head# 2024-04-23T16:25:48.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2a0d5f:::benchmark_data_smithi188_89276_object94565:head# 2024-04-23T16:25:48.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2a8f71:::benchmark_data_smithi188_89276_object136862:head# 2024-04-23T16:25:48.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2ab942:::benchmark_data_smithi188_89276_object44247:head# 2024-04-23T16:25:48.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.915 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2addf2:::benchmark_data_smithi188_89276_object139214:head# 2024-04-23T16:25:48.915 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.915 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2b27ec:::benchmark_data_smithi188_89276_object162339:head# 2024-04-23T16:25:48.915 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2b41ef:::benchmark_data_smithi188_89276_object89980:head# 2024-04-23T16:25:48.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2ba9a1:::benchmark_data_smithi188_89276_object151086:head# 2024-04-23T16:25:48.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2bab2a:::benchmark_data_smithi188_89276_object65851:head# 2024-04-23T16:25:48.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2bcd21:::benchmark_data_smithi188_89276_object61182:head# 2024-04-23T16:25:48.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2bed59:::benchmark_data_smithi188_89276_object126933:head# 2024-04-23T16:25:48.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2c970a:::benchmark_data_smithi188_89276_object77521:head# 2024-04-23T16:25:48.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2d1a6f:::benchmark_data_smithi188_89276_object50418:head# 2024-04-23T16:25:48.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2e5dd4:::benchmark_data_smithi188_89276_object119190:head# 2024-04-23T16:25:48.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2eaba9:::benchmark_data_smithi188_89276_object137023:head# 2024-04-23T16:25:48.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2eeecf:::benchmark_data_smithi188_89276_object99121:head# 2024-04-23T16:25:48.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2f44c2:::benchmark_data_smithi188_89276_object113436:head# 2024-04-23T16:25:48.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e2f55a5:::benchmark_data_smithi188_89276_object148596:head# 2024-04-23T16:25:48.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e301177:::benchmark_data_smithi188_89276_object52959:head# 2024-04-23T16:25:48.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e304070:::benchmark_data_smithi188_89276_object69005:head# 2024-04-23T16:25:48.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e30777a:::benchmark_data_smithi188_89276_object68851:head# 2024-04-23T16:25:48.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e308816:::benchmark_data_smithi188_89276_object85111:head# 2024-04-23T16:25:48.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e30f27b:::benchmark_data_smithi188_89276_object89590:head# 2024-04-23T16:25:48.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e3112e3:::benchmark_data_smithi188_89276_object138291:head# 2024-04-23T16:25:48.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.928 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e311913:::benchmark_data_smithi188_89276_object97438:head# 2024-04-23T16:25:48.928 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.928 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e31977a:::benchmark_data_smithi188_89276_object142008:head# 2024-04-23T16:25:48.928 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.929 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e31dbee:::benchmark_data_smithi188_89276_object67806:head# 2024-04-23T16:25:48.929 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.929 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e324b19:::benchmark_data_smithi188_89276_object143684:head# 2024-04-23T16:25:48.929 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.929 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e3269f3:::benchmark_data_smithi188_89276_object74708:head# 2024-04-23T16:25:48.929 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.929 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e32c25a:::benchmark_data_smithi188_89276_object130317:head# 2024-04-23T16:25:48.929 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e3318a7:::benchmark_data_smithi188_89276_object95075:head# 2024-04-23T16:25:48.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e3406e7:::benchmark_data_smithi188_89276_object149343:head# 2024-04-23T16:25:48.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e350879:::benchmark_data_smithi188_89276_object54380:head# 2024-04-23T16:25:48.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e357e21:::benchmark_data_smithi188_89276_object157106:head# 2024-04-23T16:25:48.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e35e287:::benchmark_data_smithi188_89276_object165418:head# 2024-04-23T16:25:48.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e35fcc3:::benchmark_data_smithi188_89276_object51930:head# 2024-04-23T16:25:48.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e36a28b:::benchmark_data_smithi188_89276_object69709:head# 2024-04-23T16:25:48.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e36acc1:::benchmark_data_smithi188_89276_object50552:head# 2024-04-23T16:25:48.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e37b9d1:::benchmark_data_smithi188_89276_object94116:head# 2024-04-23T16:25:48.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e37f4ea:::benchmark_data_smithi188_89276_object101753:head# 2024-04-23T16:25:48.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e381448:::benchmark_data_smithi188_89276_object78583:head# 2024-04-23T16:25:48.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e383527:::benchmark_data_smithi188_89276_object47436:head# 2024-04-23T16:25:48.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e391135:::benchmark_data_smithi188_89276_object50684:head# 2024-04-23T16:25:48.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e392457:::benchmark_data_smithi188_89276_object159029:head# 2024-04-23T16:25:48.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e393665:::benchmark_data_smithi188_89276_object74411:head# 2024-04-23T16:25:48.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e39627e:::benchmark_data_smithi188_89276_object126066:head# 2024-04-23T16:25:48.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e39951f:::benchmark_data_smithi188_89276_object103285:head# 2024-04-23T16:25:48.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e39ecc4:::benchmark_data_smithi188_89276_object136060:head# 2024-04-23T16:25:48.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e3ac9e4:::benchmark_data_smithi188_89276_object48578:head# 2024-04-23T16:25:48.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e3c0930:::benchmark_data_smithi188_89276_object116851:head# 2024-04-23T16:25:48.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e3c2669:::benchmark_data_smithi188_89276_object80557:head# 2024-04-23T16:25:48.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e3cac1a:::benchmark_data_smithi188_89276_object149839:head# 2024-04-23T16:25:48.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e3cba5c:::benchmark_data_smithi188_89276_object128301:head# 2024-04-23T16:25:48.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e3cf166:::benchmark_data_smithi188_89276_object125153:head# 2024-04-23T16:25:48.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e3d1a2c:::benchmark_data_smithi188_89276_object158625:head# 2024-04-23T16:25:48.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e3d4b08:::benchmark_data_smithi188_89276_object155789:head# 2024-04-23T16:25:48.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e3d5f59:::benchmark_data_smithi188_89276_object88709:head# 2024-04-23T16:25:48.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e3df7b4:::benchmark_data_smithi188_89276_object113111:head# 2024-04-23T16:25:48.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e3df874:::benchmark_data_smithi188_89276_object61440:head# 2024-04-23T16:25:48.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e3e0a60:::benchmark_data_smithi188_89276_object74696:head# 2024-04-23T16:25:48.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e41a5ba:::benchmark_data_smithi188_89276_object125494:head# 2024-04-23T16:25:48.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e41cef6:::benchmark_data_smithi188_89276_object149490:head# 2024-04-23T16:25:48.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e41ec36:::benchmark_data_smithi188_89276_object79292:head# 2024-04-23T16:25:48.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e423cf3:::benchmark_data_smithi188_89276_object164192:head# 2024-04-23T16:25:48.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e429db0:::benchmark_data_smithi188_89276_object53335:head# 2024-04-23T16:25:48.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e42a9a0:::benchmark_data_smithi188_89276_object127353:head# 2024-04-23T16:25:48.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e42d61e:::benchmark_data_smithi188_89276_object112247:head# 2024-04-23T16:25:48.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e432a9e:::benchmark_data_smithi188_89276_object157669:head# 2024-04-23T16:25:48.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e438f87:::benchmark_data_smithi188_89276_object79348:head# 2024-04-23T16:25:48.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e44cdf9:::benchmark_data_smithi188_89276_object106781:head# 2024-04-23T16:25:48.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e468941:::benchmark_data_smithi188_89276_object43264:head# 2024-04-23T16:25:48.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e47360f:::benchmark_data_smithi188_89276_object82484:head# 2024-04-23T16:25:48.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e475b6f:::benchmark_data_smithi188_89276_object155633:head# 2024-04-23T16:25:48.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e47d352:::benchmark_data_smithi188_89276_object78721:head# 2024-04-23T16:25:48.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e49d1b1:::benchmark_data_smithi188_89276_object83861:head# 2024-04-23T16:25:48.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e49e872:::benchmark_data_smithi188_89276_object69205:head# 2024-04-23T16:25:48.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e4a32e1:::benchmark_data_smithi188_89276_object161553:head# 2024-04-23T16:25:48.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e4a998b:::benchmark_data_smithi188_89276_object142709:head# 2024-04-23T16:25:48.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e4af81d:::benchmark_data_smithi188_89276_object66179:head# 2024-04-23T16:25:48.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e4af9df:::benchmark_data_smithi188_89276_object89356:head# 2024-04-23T16:25:48.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e4c1ca8:::benchmark_data_smithi188_89276_object137674:head# 2024-04-23T16:25:48.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e4c9ae2:::benchmark_data_smithi188_89276_object123249:head# 2024-04-23T16:25:48.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e4ce021:::benchmark_data_smithi188_89276_object143076:head# 2024-04-23T16:25:48.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e4fc793:::benchmark_data_smithi188_89276_object147945:head# 2024-04-23T16:25:48.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5067be:::benchmark_data_smithi188_89276_object74199:head# 2024-04-23T16:25:48.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e506a84:::benchmark_data_smithi188_89276_object145407:head# 2024-04-23T16:25:48.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e50ae9f:::benchmark_data_smithi188_89276_object84344:head# 2024-04-23T16:25:48.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e515875:::benchmark_data_smithi188_89276_object68180:head# 2024-04-23T16:25:48.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e528679:::benchmark_data_smithi188_89276_object62688:head# 2024-04-23T16:25:48.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5294be:::benchmark_data_smithi188_89276_object162878:head# 2024-04-23T16:25:48.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e52d06d:::benchmark_data_smithi188_89276_object52757:head# 2024-04-23T16:25:48.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e53e0ce:::benchmark_data_smithi188_89276_object92487:head# 2024-04-23T16:25:48.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e540bcb:::benchmark_data_smithi188_89276_object128088:head# 2024-04-23T16:25:48.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e546aa0:::benchmark_data_smithi188_89276_object166114:head# 2024-04-23T16:25:48.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5488d4:::benchmark_data_smithi188_89276_object126721:head# 2024-04-23T16:25:48.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e553619:::benchmark_data_smithi188_89276_object135991:head# 2024-04-23T16:25:48.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e56e678:::benchmark_data_smithi188_89276_object87369:head# 2024-04-23T16:25:48.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e57bbd1:::benchmark_data_smithi188_89276_object78397:head# 2024-04-23T16:25:48.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e57d98a:::benchmark_data_smithi188_89276_object43868:head# 2024-04-23T16:25:48.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5924e1:::benchmark_data_smithi188_89276_object136655:head# 2024-04-23T16:25:48.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e59a917:::benchmark_data_smithi188_89276_object61139:head# 2024-04-23T16:25:48.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e59fb8f:::benchmark_data_smithi188_89276_object103845:head# 2024-04-23T16:25:48.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5a078e:::benchmark_data_smithi188_89276_object163405:head# 2024-04-23T16:25:48.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5a6472:::benchmark_data_smithi188_89276_object140160:head# 2024-04-23T16:25:48.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5a73ca:::benchmark_data_smithi188_89276_object141994:head# 2024-04-23T16:25:48.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5aff84:::benchmark_data_smithi188_89276_object58650:head# 2024-04-23T16:25:48.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5affc4:::benchmark_data_smithi188_89276_object157197:head# 2024-04-23T16:25:48.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5b75ca:::benchmark_data_smithi188_89276_object93013:head# 2024-04-23T16:25:48.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5bc12f:::benchmark_data_smithi188_89276_object130378:head# 2024-04-23T16:25:48.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5c4987:::benchmark_data_smithi188_89276_object133759:head# 2024-04-23T16:25:48.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5c6a20:::benchmark_data_smithi188_89276_object56476:head# 2024-04-23T16:25:48.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.964 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5e783e:::benchmark_data_smithi188_89276_object109329:head# 2024-04-23T16:25:48.964 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5e8a49:::benchmark_data_smithi188_89276_object61919:head# 2024-04-23T16:25:48.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5e93d7:::benchmark_data_smithi188_89276_object116164:head# 2024-04-23T16:25:48.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5eaa7d:::benchmark_data_smithi188_89276_object88004:head# 2024-04-23T16:25:48.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5edc85:::benchmark_data_smithi188_89276_object135052:head# 2024-04-23T16:25:48.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5f029c:::benchmark_data_smithi188_89276_object141425:head# 2024-04-23T16:25:48.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e5f76a6:::benchmark_data_smithi188_89276_object168202:head# 2024-04-23T16:25:48.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e60987f:::benchmark_data_smithi188_89276_object52544:head# 2024-04-23T16:25:48.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e609b46:::benchmark_data_smithi188_89276_object110766:head# 2024-04-23T16:25:48.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e614baf:::benchmark_data_smithi188_89276_object159319:head# 2024-04-23T16:25:48.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e616d5e:::benchmark_data_smithi188_89276_object94369:head# 2024-04-23T16:25:48.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e61efcd:::benchmark_data_smithi188_89276_object165507:head# 2024-04-23T16:25:48.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e659cec:::benchmark_data_smithi188_89276_object77475:head# 2024-04-23T16:25:48.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e66de20:::benchmark_data_smithi188_89276_object87103:head# 2024-04-23T16:25:48.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e66eb83:::benchmark_data_smithi188_89276_object152721:head# 2024-04-23T16:25:48.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e683dc7:::benchmark_data_smithi188_89276_object61148:head# 2024-04-23T16:25:48.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e69048e:::benchmark_data_smithi188_89276_object120172:head# 2024-04-23T16:25:48.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e6a235e:::benchmark_data_smithi188_89276_object70622:head# 2024-04-23T16:25:48.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e6a2440:::benchmark_data_smithi188_89276_object63963:head# 2024-04-23T16:25:48.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e6a627b:::benchmark_data_smithi188_89276_object44417:head# 2024-04-23T16:25:48.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e6bab34:::benchmark_data_smithi188_89276_object90031:head# 2024-04-23T16:25:48.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e6bb180:::benchmark_data_smithi188_89276_object79621:head# 2024-04-23T16:25:48.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e6bcc43:::benchmark_data_smithi188_89276_object79210:head# 2024-04-23T16:25:48.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e6c38e5:::benchmark_data_smithi188_89276_object168595:head# 2024-04-23T16:25:48.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e6cab2b:::benchmark_data_smithi188_89276_object127003:head# 2024-04-23T16:25:48.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e6cd6f6:::benchmark_data_smithi188_89276_object116027:head# 2024-04-23T16:25:48.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e6cf6c1:::benchmark_data_smithi188_89276_object70381:head# 2024-04-23T16:25:48.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e6dc8e1:::benchmark_data_smithi188_89276_object164850:head# 2024-04-23T16:25:48.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e6e701a:::benchmark_data_smithi188_89276_object64174:head# 2024-04-23T16:25:48.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e71a29f:::benchmark_data_smithi188_89276_object65964:head# 2024-04-23T16:25:48.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e71c8f6:::benchmark_data_smithi188_89276_object71934:head# 2024-04-23T16:25:48.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e727670:::benchmark_data_smithi188_89276_object50134:head# 2024-04-23T16:25:48.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e7395fb:::benchmark_data_smithi188_89276_object123882:head# 2024-04-23T16:25:48.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e73d0e6:::benchmark_data_smithi188_89276_object46763:head# 2024-04-23T16:25:48.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e74890c:::benchmark_data_smithi188_89276_object81163:head# 2024-04-23T16:25:48.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e74a7cb:::benchmark_data_smithi188_89276_object53639:head# 2024-04-23T16:25:48.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e75516b:::benchmark_data_smithi188_89276_object55621:head# 2024-04-23T16:25:48.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e76964e:::benchmark_data_smithi188_89276_object74187:head# 2024-04-23T16:25:48.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e76efe4:::benchmark_data_smithi188_89276_object57651:head# 2024-04-23T16:25:48.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e771192:::benchmark_data_smithi188_89276_object87334:head# 2024-04-23T16:25:48.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e77b842:::benchmark_data_smithi188_89276_object57432:head# 2024-04-23T16:25:48.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.978 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e789985:::benchmark_data_smithi188_89276_object109005:head# 2024-04-23T16:25:48.978 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.978 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e78a1a0:::benchmark_data_smithi188_89276_object159713:head# 2024-04-23T16:25:48.978 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.978 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e79709f:::benchmark_data_smithi188_89276_object71931:head# 2024-04-23T16:25:48.982 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.982 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e79939b:::benchmark_data_smithi188_89276_object164533:head# 2024-04-23T16:25:48.982 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.982 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e79c3c9:::benchmark_data_smithi188_89276_object74634:head# 2024-04-23T16:25:48.982 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.983 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e79e28d:::benchmark_data_smithi188_89276_object55682:head# 2024-04-23T16:25:48.983 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.983 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e7a9bb1:::benchmark_data_smithi188_89276_object109201:head# 2024-04-23T16:25:48.983 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.986 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e7b2fb7:::benchmark_data_smithi188_89276_object45466:head# 2024-04-23T16:25:48.986 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.986 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e7b3834:::benchmark_data_smithi188_89276_object89475:head# 2024-04-23T16:25:48.986 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.986 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e7ba7b5:::benchmark_data_smithi188_89276_object70993:head# 2024-04-23T16:25:48.986 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.986 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e7be0ad:::benchmark_data_smithi188_89276_object103800:head# 2024-04-23T16:25:48.987 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.987 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e7c8956:::benchmark_data_smithi188_89276_object101140:head# 2024-04-23T16:25:48.987 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.987 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e7d6c0e:::benchmark_data_smithi188_89276_object115368:head# 2024-04-23T16:25:48.987 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.987 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e7e3f11:::benchmark_data_smithi188_89276_object146437:head# 2024-04-23T16:25:48.987 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.987 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e7fa208:::benchmark_data_smithi188_89276_object99351:head# 2024-04-23T16:25:48.987 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.987 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e802cf4:::benchmark_data_smithi188_89276_object46469:head# 2024-04-23T16:25:48.987 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.987 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8088dd:::benchmark_data_smithi188_89276_object88780:head# 2024-04-23T16:25:48.987 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.988 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e80e7b3:::benchmark_data_smithi188_89276_object167863:head# 2024-04-23T16:25:48.988 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.988 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e812e83:::benchmark_data_smithi188_89276_object153748:head# 2024-04-23T16:25:48.988 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.988 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8224fe:::benchmark_data_smithi188_89276_object165084:head# 2024-04-23T16:25:48.988 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.988 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e83521d:::benchmark_data_smithi188_89276_object53058:head# 2024-04-23T16:25:48.988 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.988 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e83aef7:::benchmark_data_smithi188_89276_object53556:head# 2024-04-23T16:25:48.988 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.988 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e84028a:::benchmark_data_smithi188_89276_object95685:head# 2024-04-23T16:25:48.989 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.989 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8407e4:::benchmark_data_smithi188_89276_object59739:head# 2024-04-23T16:25:48.989 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.989 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e847922:::benchmark_data_smithi188_89276_object111924:head# 2024-04-23T16:25:48.989 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.989 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e84cdb1:::benchmark_data_smithi188_89276_object76058:head# 2024-04-23T16:25:48.989 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.991 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8561fd:::benchmark_data_smithi188_89276_object40421:head# 2024-04-23T16:25:48.991 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.991 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e85c7e9:::benchmark_data_smithi188_89276_object44824:head# 2024-04-23T16:25:48.991 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e85defa:::benchmark_data_smithi188_89276_object61573:head# 2024-04-23T16:25:48.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8602f8:::benchmark_data_smithi188_89276_object148022:head# 2024-04-23T16:25:48.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e869fac:::benchmark_data_smithi188_89276_object49865:head# 2024-04-23T16:25:48.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e86da57:::benchmark_data_smithi188_89276_object68897:head# 2024-04-23T16:25:48.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e875929:::benchmark_data_smithi188_89276_object96398:head# 2024-04-23T16:25:48.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8764c4:::benchmark_data_smithi188_89276_object117947:head# 2024-04-23T16:25:48.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e885689:::benchmark_data_smithi188_89276_object125161:head# 2024-04-23T16:25:48.993 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.993 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8857fe:::benchmark_data_smithi188_89276_object144089:head# 2024-04-23T16:25:48.993 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.993 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e89bc50:::benchmark_data_smithi188_89276_object66985:head# 2024-04-23T16:25:48.993 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.993 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e89f9f3:::benchmark_data_smithi188_89276_object147689:head# 2024-04-23T16:25:48.993 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.994 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8a4b93:::benchmark_data_smithi188_89276_object136634:head# 2024-04-23T16:25:48.994 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.997 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8b3db2:::benchmark_data_smithi188_89276_object127136:head# 2024-04-23T16:25:48.997 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8bb07b:::benchmark_data_smithi188_89276_object112882:head# 2024-04-23T16:25:48.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8bc18e:::benchmark_data_smithi188_89276_object89607:head# 2024-04-23T16:25:48.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8bcf22:::benchmark_data_smithi188_89276_object123899:head# 2024-04-23T16:25:48.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8c06db:::benchmark_data_smithi188_89276_object51901:head# 2024-04-23T16:25:48.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.999 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8c3260:::benchmark_data_smithi188_89276_object65364:head# 2024-04-23T16:25:48.999 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.999 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8c9b93:::benchmark_data_smithi188_89276_object107859:head# 2024-04-23T16:25:48.999 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:48.999 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8cae79:::benchmark_data_smithi188_89276_object117895:head# 2024-04-23T16:25:48.999 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.003 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8df98b:::benchmark_data_smithi188_89276_object118942:head# 2024-04-23T16:25:49.003 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.004 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8e2584:::benchmark_data_smithi188_89276_object93771:head# 2024-04-23T16:25:49.004 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.004 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8e329d:::benchmark_data_smithi188_89276_object96266:head# 2024-04-23T16:25:49.004 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.005 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8e53fa:::benchmark_data_smithi188_89276_object164586:head# 2024-04-23T16:25:49.005 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.005 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8eff44:::benchmark_data_smithi188_89276_object168192:head# 2024-04-23T16:25:49.005 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.005 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8f5db4:::benchmark_data_smithi188_89276_object166605:head# 2024-04-23T16:25:49.005 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.006 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e8ffb6f:::benchmark_data_smithi188_89276_object119727:head# 2024-04-23T16:25:49.006 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.006 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e92053f:::benchmark_data_smithi188_89276_object40560:head# 2024-04-23T16:25:49.007 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.007 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e921cc7:::benchmark_data_smithi188_89276_object164844:head# 2024-04-23T16:25:49.007 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.007 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e929411:::benchmark_data_smithi188_89276_object97355:head# 2024-04-23T16:25:49.007 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.007 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e92ff52:::benchmark_data_smithi188_89276_object167004:head# 2024-04-23T16:25:49.007 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.009 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e952039:::benchmark_data_smithi188_89276_object149658:head# 2024-04-23T16:25:49.009 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.009 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e95480d:::benchmark_data_smithi188_89276_object148198:head# 2024-04-23T16:25:49.009 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.009 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e95ec0a:::benchmark_data_smithi188_89276_object127748:head# 2024-04-23T16:25:49.009 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.009 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e962f36:::benchmark_data_smithi188_89276_object67619:head# 2024-04-23T16:25:49.009 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.009 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e971a53:::benchmark_data_smithi188_89276_object69014:head# 2024-04-23T16:25:49.010 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.010 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e980a9f:::benchmark_data_smithi188_89276_object132085:head# 2024-04-23T16:25:49.010 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.010 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e99df12:::benchmark_data_smithi188_89276_object164862:head# 2024-04-23T16:25:49.010 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.014 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e9a6560:::benchmark_data_smithi188_89276_object122280:head# 2024-04-23T16:25:49.014 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.014 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e9aed41:::benchmark_data_smithi188_89276_object130445:head# 2024-04-23T16:25:49.014 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.014 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e9b89b6:::benchmark_data_smithi188_89276_object74256:head# 2024-04-23T16:25:49.014 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.014 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e9db5b8:::benchmark_data_smithi188_89276_object85555:head# 2024-04-23T16:25:49.014 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.014 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2e9f2e33:::benchmark_data_smithi188_89276_object152173:head# 2024-04-23T16:25:49.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ea0c724:::benchmark_data_smithi188_89276_object48761:head# 2024-04-23T16:25:49.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ea1778e:::benchmark_data_smithi188_89276_object54333:head# 2024-04-23T16:25:49.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ea2710a:::benchmark_data_smithi188_89276_object91331:head# 2024-04-23T16:25:49.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ea298b0:::benchmark_data_smithi188_89276_object87245:head# 2024-04-23T16:25:49.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ea2b1a9:::benchmark_data_smithi188_89276_object102068:head# 2024-04-23T16:25:49.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ea32964:::benchmark_data_smithi188_89276_object63296:head# 2024-04-23T16:25:49.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ea40ad9:::benchmark_data_smithi188_89276_object94889:head# 2024-04-23T16:25:49.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ea41a6c:::benchmark_data_smithi188_89276_object101793:head# 2024-04-23T16:25:49.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ea4c8b1:::benchmark_data_smithi188_89276_object129535:head# 2024-04-23T16:25:49.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ea53c2a:::benchmark_data_smithi188_89276_object102659:head# 2024-04-23T16:25:49.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ea5f4cc:::benchmark_data_smithi188_89276_object59378:head# 2024-04-23T16:25:49.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ea70200:::benchmark_data_smithi188_89276_object77463:head# 2024-04-23T16:25:49.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ea76e96:::benchmark_data_smithi188_89276_object147807:head# 2024-04-23T16:25:49.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.017 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ea79a60:::benchmark_data_smithi188_89276_object49782:head# 2024-04-23T16:25:49.017 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.017 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ea8ed1d:::benchmark_data_smithi188_89276_object95908:head# 2024-04-23T16:25:49.017 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.017 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ea957df:::benchmark_data_smithi188_89276_object49412:head# 2024-04-23T16:25:49.017 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.017 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ea98986:::benchmark_data_smithi188_89276_object91056:head# 2024-04-23T16:25:49.017 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.018 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eaade55:::benchmark_data_smithi188_89276_object97405:head# 2024-04-23T16:25:49.018 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.018 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eab17c2:::benchmark_data_smithi188_89276_object112768:head# 2024-04-23T16:25:49.018 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.018 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eab75cb:::benchmark_data_smithi188_89276_object124965:head# 2024-04-23T16:25:49.019 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.019 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eab7f9a:::benchmark_data_smithi188_89276_object83419:head# 2024-04-23T16:25:49.019 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eac3b07:::benchmark_data_smithi188_89276_object166496:head# 2024-04-23T16:25:49.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eac81c9:::benchmark_data_smithi188_89276_object68224:head# 2024-04-23T16:25:49.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eaced21:::benchmark_data_smithi188_89276_object50338:head# 2024-04-23T16:25:49.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ead9346:::benchmark_data_smithi188_89276_object105549:head# 2024-04-23T16:25:49.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eadf70d:::benchmark_data_smithi188_89276_object77425:head# 2024-04-23T16:25:49.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eae28ae:::benchmark_data_smithi188_89276_object43184:head# 2024-04-23T16:25:49.021 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eaec45f:::benchmark_data_smithi188_89276_object43632:head# 2024-04-23T16:25:49.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eaf433d:::benchmark_data_smithi188_89276_object47776:head# 2024-04-23T16:25:49.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eafda6b:::benchmark_data_smithi188_89276_object107253:head# 2024-04-23T16:25:49.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb00fc4:::benchmark_data_smithi188_89276_object148151:head# 2024-04-23T16:25:49.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb01bb4:::benchmark_data_smithi188_89276_object84665:head# 2024-04-23T16:25:49.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb182fc:::benchmark_data_smithi188_89276_object122141:head# 2024-04-23T16:25:49.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb2bf61:::benchmark_data_smithi188_89276_object128435:head# 2024-04-23T16:25:49.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb3ca39:::benchmark_data_smithi188_89276_object156436:head# 2024-04-23T16:25:49.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb4ffd0:::benchmark_data_smithi188_89276_object149883:head# 2024-04-23T16:25:49.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb52ebf:::benchmark_data_smithi188_89276_object125905:head# 2024-04-23T16:25:49.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb54f2b:::benchmark_data_smithi188_89276_object117519:head# 2024-04-23T16:25:49.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb5a46e:::benchmark_data_smithi188_89276_object168384:head# 2024-04-23T16:25:49.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb5a6ca:::benchmark_data_smithi188_89276_object126920:head# 2024-04-23T16:25:49.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb5d8c8:::benchmark_data_smithi188_89276_object61654:head# 2024-04-23T16:25:49.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb5fd66:::benchmark_data_smithi188_89276_object47980:head# 2024-04-23T16:25:49.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb68894:::benchmark_data_smithi188_89276_object41428:head# 2024-04-23T16:25:49.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb78f99:::benchmark_data_smithi188_89276_object89572:head# 2024-04-23T16:25:49.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb7922b:::benchmark_data_smithi188_89276_object64212:head# 2024-04-23T16:25:49.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb79de2:::benchmark_data_smithi188_89276_object156248:head# 2024-04-23T16:25:49.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb8d63e:::benchmark_data_smithi188_89276_object82103:head# 2024-04-23T16:25:49.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb8e370:::benchmark_data_smithi188_89276_object59388:head# 2024-04-23T16:25:49.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb8f00c:::benchmark_data_smithi188_89276_object65986:head# 2024-04-23T16:25:49.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb93fc6:::benchmark_data_smithi188_89276_object71337:head# 2024-04-23T16:25:49.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb965bc:::benchmark_data_smithi188_89276_object75802:head# 2024-04-23T16:25:49.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eb9dddd:::benchmark_data_smithi188_89276_object142341:head# 2024-04-23T16:25:49.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eba52c9:::benchmark_data_smithi188_89276_object105609:head# 2024-04-23T16:25:49.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ebc1c73:::benchmark_data_smithi188_89276_object148892:head# 2024-04-23T16:25:49.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ebc3062:::benchmark_data_smithi188_89276_object146638:head# 2024-04-23T16:25:49.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ebc7665:::benchmark_data_smithi188_89276_object165489:head# 2024-04-23T16:25:49.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ebcaf74:::benchmark_data_smithi188_89276_object151444:head# 2024-04-23T16:25:49.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ebcdeae:::benchmark_data_smithi188_89276_object102644:head# 2024-04-23T16:25:49.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.032 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ebe2aec:::benchmark_data_smithi188_89276_object104413:head# 2024-04-23T16:25:49.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.032 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ebe4750:::benchmark_data_smithi188_89276_object49063:head# 2024-04-23T16:25:49.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.032 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ebed9cc:::benchmark_data_smithi188_89276_object125501:head# 2024-04-23T16:25:49.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.032 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ebf2b69:::benchmark_data_smithi188_89276_object119742:head# 2024-04-23T16:25:49.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.034 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ebfc367:::benchmark_data_smithi188_89276_object157256:head# 2024-04-23T16:25:49.034 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.034 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ebfd291:::benchmark_data_smithi188_89276_object157042:head# 2024-04-23T16:25:49.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ebfecd8:::benchmark_data_smithi188_89276_object52258:head# 2024-04-23T16:25:49.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ec05425:::benchmark_data_smithi188_89276_object79162:head# 2024-04-23T16:25:49.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ec0b511:::benchmark_data_smithi188_89276_object150633:head# 2024-04-23T16:25:49.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ec0e3bb:::benchmark_data_smithi188_89276_object82196:head# 2024-04-23T16:25:49.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ec174d9:::benchmark_data_smithi188_89276_object58498:head# 2024-04-23T16:25:49.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ec18027:::benchmark_data_smithi188_89276_object102639:head# 2024-04-23T16:25:49.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.036 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ec321cf:::benchmark_data_smithi188_89276_object61706:head# 2024-04-23T16:25:49.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.036 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ec3b351:::benchmark_data_smithi188_89276_object114423:head# 2024-04-23T16:25:49.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.036 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ec3d2f4:::benchmark_data_smithi188_89276_object42473:head# 2024-04-23T16:25:49.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.036 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ec43b34:::benchmark_data_smithi188_89276_object114225:head# 2024-04-23T16:25:49.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.036 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ec4832d:::benchmark_data_smithi188_89276_object54253:head# 2024-04-23T16:25:49.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.036 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ec4bf5e:::benchmark_data_smithi188_89276_object68223:head# 2024-04-23T16:25:49.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.037 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ec6a19a:::benchmark_data_smithi188_89276_object103752:head# 2024-04-23T16:25:49.037 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.039 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ec73526:::benchmark_data_smithi188_89276_object142497:head# 2024-04-23T16:25:49.039 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.039 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ec7ed10:::benchmark_data_smithi188_89276_object151998:head# 2024-04-23T16:25:49.039 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.039 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ec81a16:::benchmark_data_smithi188_89276_object135033:head# 2024-04-23T16:25:49.039 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.039 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ec95efe:::benchmark_data_smithi188_89276_object92289:head# 2024-04-23T16:25:49.039 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.039 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eca5df6:::benchmark_data_smithi188_89276_object59138:head# 2024-04-23T16:25:49.040 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.040 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ecb4dce:::benchmark_data_smithi188_89276_object52047:head# 2024-04-23T16:25:49.040 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.040 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ecc4517:::benchmark_data_smithi188_89276_object85991:head# 2024-04-23T16:25:49.040 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.040 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ecc588f:::benchmark_data_smithi188_89276_object136159:head# 2024-04-23T16:25:49.040 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.040 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ecc846c:::benchmark_data_smithi188_89276_object110713:head# 2024-04-23T16:25:49.040 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.040 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eccd158:::benchmark_data_smithi188_89276_object139455:head# 2024-04-23T16:25:49.040 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.040 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ecd111e:::benchmark_data_smithi188_89276_object121574:head# 2024-04-23T16:25:49.041 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.041 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eceb131:::benchmark_data_smithi188_89276_object167398:head# 2024-04-23T16:25:49.041 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.041 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eceebe2:::benchmark_data_smithi188_89276_object144039:head# 2024-04-23T16:25:49.041 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.041 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ecf843f:::benchmark_data_smithi188_89276_object135750:head# 2024-04-23T16:25:49.041 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.043 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ecfd966:::benchmark_data_smithi188_89276_object132776:head# 2024-04-23T16:25:49.043 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.044 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ed033ef:::benchmark_data_smithi188_89276_object94434:head# 2024-04-23T16:25:49.044 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.047 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:25:49.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:49.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:25:49.052 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ed09844:::benchmark_data_smithi188_89276_object71261:head# 2024-04-23T16:25:49.052 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ed221bf:::benchmark_data_smithi188_89276_object50540:head# 2024-04-23T16:25:49.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ed3e8a5:::benchmark_data_smithi188_89276_object142009:head# 2024-04-23T16:25:49.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ed49609:::benchmark_data_smithi188_89276_object48405:head# 2024-04-23T16:25:49.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ed4e758:::benchmark_data_smithi188_89276_object132079:head# 2024-04-23T16:25:49.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ed4f08c:::benchmark_data_smithi188_89276_object150684:head# 2024-04-23T16:25:49.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ed73cf2:::benchmark_data_smithi188_89276_object166716:head# 2024-04-23T16:25:49.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ed7507f:::benchmark_data_smithi188_89276_object58605:head# 2024-04-23T16:25:49.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ed8355d:::benchmark_data_smithi188_89276_object160408:head# 2024-04-23T16:25:49.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ed8a6b5:::benchmark_data_smithi188_89276_object87633:head# 2024-04-23T16:25:49.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ed8dfe7:::benchmark_data_smithi188_89276_object72898:head# 2024-04-23T16:25:49.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.061 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ed912af:::benchmark_data_smithi188_89276_object41922:head# 2024-04-23T16:25:49.061 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.061 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ed92f06:::benchmark_data_smithi188_89276_object130779:head# 2024-04-23T16:25:49.061 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.061 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2edb13ad:::benchmark_data_smithi188_89276_object137824:head# 2024-04-23T16:25:49.061 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.061 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2edb48fa:::benchmark_data_smithi188_89276_object40700:head# 2024-04-23T16:25:49.061 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.061 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2edbe776:::benchmark_data_smithi188_89276_object157263:head# 2024-04-23T16:25:49.061 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.062 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2edc579a:::benchmark_data_smithi188_89276_object163174:head# 2024-04-23T16:25:49.062 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.062 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2edca302:::benchmark_data_smithi188_89276_object151192:head# 2024-04-23T16:25:49.062 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.062 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2edd0a14:::benchmark_data_smithi188_89276_object75325:head# 2024-04-23T16:25:49.062 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2edd283d:::benchmark_data_smithi188_89276_object58769:head# 2024-04-23T16:25:49.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2edd3923:::benchmark_data_smithi188_89276_object130581:head# 2024-04-23T16:25:49.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2edd64f6:::benchmark_data_smithi188_89276_object80036:head# 2024-04-23T16:25:49.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.070 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2edd822d:::benchmark_data_smithi188_89276_object110213:head# 2024-04-23T16:25:49.071 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.072 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:25:49.072 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eddb679:::benchmark_data_smithi188_89276_object94488:head# 2024-04-23T16:25:49.072 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.072 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eddcf5e:::benchmark_data_smithi188_89276_object58347:head# 2024-04-23T16:25:49.072 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ede0685:::benchmark_data_smithi188_89276_object90750:head# 2024-04-23T16:25:49.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ede09f8:::benchmark_data_smithi188_89276_object128027:head# 2024-04-23T16:25:49.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ede4a19:::benchmark_data_smithi188_89276_object122591:head# 2024-04-23T16:25:49.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2edec543:::benchmark_data_smithi188_89276_object120872:head# 2024-04-23T16:25:49.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2edf21d1:::benchmark_data_smithi188_89276_object126431:head# 2024-04-23T16:25:49.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee05cbe:::benchmark_data_smithi188_89276_object46774:head# 2024-04-23T16:25:49.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee192f2:::benchmark_data_smithi188_89276_object100699:head# 2024-04-23T16:25:49.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.086 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee1fbf7:::benchmark_data_smithi188_89276_object90653:head# 2024-04-23T16:25:49.086 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.086 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee22af9:::benchmark_data_smithi188_89276_object159126:head# 2024-04-23T16:25:49.087 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.087 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee29a24:::benchmark_data_smithi188_89276_object92705:head# 2024-04-23T16:25:49.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.089 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee2c9b4:::benchmark_data_smithi188_89276_object128035:head# 2024-04-23T16:25:49.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee2e9a7:::benchmark_data_smithi188_89276_object45690:head# 2024-04-23T16:25:49.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee2ec08:::benchmark_data_smithi188_89276_object89214:head# 2024-04-23T16:25:49.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee2f779:::benchmark_data_smithi188_89276_object163016:head# 2024-04-23T16:25:49.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee2fdab:::benchmark_data_smithi188_89276_object78365:head# 2024-04-23T16:25:49.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee2fe24:::benchmark_data_smithi188_89276_object81878:head# 2024-04-23T16:25:49.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee3d564:::benchmark_data_smithi188_89276_object140961:head# 2024-04-23T16:25:49.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee48f62:::benchmark_data_smithi188_89276_object98533:head# 2024-04-23T16:25:49.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee4ccb9:::benchmark_data_smithi188_89276_object91150:head# 2024-04-23T16:25:49.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee4e0b8:::benchmark_data_smithi188_89276_object56275:head# 2024-04-23T16:25:49.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee4f7ed:::benchmark_data_smithi188_89276_object157434:head# 2024-04-23T16:25:49.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee53e73:::benchmark_data_smithi188_89276_object69373:head# 2024-04-23T16:25:49.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee685e1:::benchmark_data_smithi188_89276_object46660:head# 2024-04-23T16:25:49.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee6c1af:::benchmark_data_smithi188_89276_object159640:head# 2024-04-23T16:25:49.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee6d5a4:::benchmark_data_smithi188_89276_object73355:head# 2024-04-23T16:25:49.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee7d95f:::benchmark_data_smithi188_89276_object76668:head# 2024-04-23T16:25:49.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee82c54:::benchmark_data_smithi188_89276_object93393:head# 2024-04-23T16:25:49.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee8d2ca:::benchmark_data_smithi188_89276_object161716:head# 2024-04-23T16:25:49.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee8fd7a:::benchmark_data_smithi188_89276_object147845:head# 2024-04-23T16:25:49.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee9235d:::benchmark_data_smithi188_89276_object149549:head# 2024-04-23T16:25:49.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ee948c3:::benchmark_data_smithi188_89276_object154352:head# 2024-04-23T16:25:49.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eea068f:::benchmark_data_smithi188_89276_object99712:head# 2024-04-23T16:25:49.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eeb124d:::benchmark_data_smithi188_89276_object99291:head# 2024-04-23T16:25:49.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eeb2bf5:::benchmark_data_smithi188_89276_object156272:head# 2024-04-23T16:25:49.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eeb4580:::benchmark_data_smithi188_89276_object121459:head# 2024-04-23T16:25:49.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eebb674:::benchmark_data_smithi188_89276_object69338:head# 2024-04-23T16:25:49.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eec22db:::benchmark_data_smithi188_89276_object130937:head# 2024-04-23T16:25:49.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eecd805:::benchmark_data_smithi188_89276_object132213:head# 2024-04-23T16:25:49.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eede7d4:::benchmark_data_smithi188_89276_object85485:head# 2024-04-23T16:25:49.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2eefe2d6:::benchmark_data_smithi188_89276_object135824:head# 2024-04-23T16:25:49.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ef046b7:::benchmark_data_smithi188_89276_object61866:head# 2024-04-23T16:25:49.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ef0960a:::benchmark_data_smithi188_89276_object60021:head# 2024-04-23T16:25:49.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ef0c223:::benchmark_data_smithi188_89276_object105590:head# 2024-04-23T16:25:49.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ef106bc:::benchmark_data_smithi188_89276_object42915:head# 2024-04-23T16:25:49.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ef1ca16:::benchmark_data_smithi188_89276_object140296:head# 2024-04-23T16:25:49.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ef267d8:::benchmark_data_smithi188_89276_object94293:head# 2024-04-23T16:25:49.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ef36573:::benchmark_data_smithi188_89276_object85632:head# 2024-04-23T16:25:49.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ef38205:::benchmark_data_smithi188_89276_object147388:head# 2024-04-23T16:25:49.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ef44fcc:::benchmark_data_smithi188_89276_object91174:head# 2024-04-23T16:25:49.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ef4e13f:::benchmark_data_smithi188_89276_object159736:head# 2024-04-23T16:25:49.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ef52cc2:::benchmark_data_smithi188_89276_object153275:head# 2024-04-23T16:25:49.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ef5a15a:::benchmark_data_smithi188_89276_object104122:head# 2024-04-23T16:25:49.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ef5df3e:::benchmark_data_smithi188_89276_object80947:head# 2024-04-23T16:25:49.102 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ef672d5:::benchmark_data_smithi188_89276_object68907:head# 2024-04-23T16:25:49.102 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.102 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:25:49.104 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ef682ef:::benchmark_data_smithi188_89276_object76687:head# 2024-04-23T16:25:49.104 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.104 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ef7b6c9:::benchmark_data_smithi188_89276_object96759:head# 2024-04-23T16:25:49.104 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.104 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ef97c79:::benchmark_data_smithi188_89276_object125425:head# 2024-04-23T16:25:49.104 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.105 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ef9d9cd:::benchmark_data_smithi188_89276_object89499:head# 2024-04-23T16:25:49.105 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.105 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2efa0201:::benchmark_data_smithi188_89276_object156223:head# 2024-04-23T16:25:49.105 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.105 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2efb4d6b:::benchmark_data_smithi188_89276_object96117:head# 2024-04-23T16:25:49.105 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.105 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2efd3f26:::benchmark_data_smithi188_89276_object44984:head# 2024-04-23T16:25:49.105 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.105 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2efd43ae:::benchmark_data_smithi188_89276_object87329:head# 2024-04-23T16:25:49.105 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.105 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2efdf18d:::benchmark_data_smithi188_89276_object97885:head# 2024-04-23T16:25:49.105 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.105 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2efe43b3:::benchmark_data_smithi188_89276_object81168:head# 2024-04-23T16:25:49.105 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.106 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2efe9f01:::benchmark_data_smithi188_89276_object67273:head# 2024-04-23T16:25:49.106 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.106 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f00115f:::benchmark_data_smithi188_89276_object137312:head# 2024-04-23T16:25:49.106 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.107 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f00b251:::benchmark_data_smithi188_89276_object50962:head# 2024-04-23T16:25:49.107 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.107 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f00ce8d:::benchmark_data_smithi188_89276_object86472:head# 2024-04-23T16:25:49.107 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f013a7b:::benchmark_data_smithi188_89276_object113377:head# 2024-04-23T16:25:49.108 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f01550c:::benchmark_data_smithi188_89276_object123831:head# 2024-04-23T16:25:49.108 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f0158ce:::benchmark_data_smithi188_89276_object47375:head# 2024-04-23T16:25:49.108 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f01be0d:::benchmark_data_smithi188_89276_object118324:head# 2024-04-23T16:25:49.108 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f039ef1:::benchmark_data_smithi188_89276_object146467:head# 2024-04-23T16:25:49.108 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f050270:::benchmark_data_smithi188_89276_object153899:head# 2024-04-23T16:25:49.108 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f050bfa:::benchmark_data_smithi188_89276_object83514:head# 2024-04-23T16:25:49.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f054f3c:::benchmark_data_smithi188_89276_object107184:head# 2024-04-23T16:25:49.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f05bdeb:::benchmark_data_smithi188_89276_object43060:head# 2024-04-23T16:25:49.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f05ea05:::benchmark_data_smithi188_89276_object67852:head# 2024-04-23T16:25:49.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f05edc6:::benchmark_data_smithi188_89276_object44479:head# 2024-04-23T16:25:49.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f063a4b:::benchmark_data_smithi188_89276_object85324:head# 2024-04-23T16:25:49.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f06cdf3:::benchmark_data_smithi188_89276_object113569:head# 2024-04-23T16:25:49.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f074a54:::benchmark_data_smithi188_89276_object140899:head# 2024-04-23T16:25:49.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f07c10f:::benchmark_data_smithi188_89276_object165905:head# 2024-04-23T16:25:49.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f085640:::benchmark_data_smithi188_89276_object156656:head# 2024-04-23T16:25:49.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f08cb26:::benchmark_data_smithi188_89276_object131292:head# 2024-04-23T16:25:49.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f08cceb:::benchmark_data_smithi188_89276_object77273:head# 2024-04-23T16:25:49.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f09a0f6:::benchmark_data_smithi188_89276_object148801:head# 2024-04-23T16:25:49.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f0ac3b5:::benchmark_data_smithi188_89276_object56037:head# 2024-04-23T16:25:49.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f0ac90f:::benchmark_data_smithi188_89276_object117143:head# 2024-04-23T16:25:49.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f0af30c:::benchmark_data_smithi188_89276_object162902:head# 2024-04-23T16:25:49.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f0c1d1b:::benchmark_data_smithi188_89276_object88534:head# 2024-04-23T16:25:49.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f0e63e1:::benchmark_data_smithi188_89276_object163101:head# 2024-04-23T16:25:49.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f0f0768:::benchmark_data_smithi188_89276_object79439:head# 2024-04-23T16:25:49.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f0f3434:::benchmark_data_smithi188_89276_object54504:head# 2024-04-23T16:25:49.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f0f6468:::benchmark_data_smithi188_89276_object113160:head# 2024-04-23T16:25:49.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f10511b:::benchmark_data_smithi188_89276_object89430:head# 2024-04-23T16:25:49.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f11768d:::benchmark_data_smithi188_89276_object129620:head# 2024-04-23T16:25:49.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f11bafc:::benchmark_data_smithi188_89276_object116692:head# 2024-04-23T16:25:49.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f11d64a:::benchmark_data_smithi188_89276_object66471:head# 2024-04-23T16:25:49.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1308f2:::benchmark_data_smithi188_89276_object70697:head# 2024-04-23T16:25:49.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f13f846:::benchmark_data_smithi188_89276_object124333:head# 2024-04-23T16:25:49.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f141cd2:::benchmark_data_smithi188_89276_object116740:head# 2024-04-23T16:25:49.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f147cae:::benchmark_data_smithi188_89276_object151669:head# 2024-04-23T16:25:49.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f14f9a8:::benchmark_data_smithi188_89276_object156691:head# 2024-04-23T16:25:49.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f151dba:::benchmark_data_smithi188_89276_object90409:head# 2024-04-23T16:25:49.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f16538a:::benchmark_data_smithi188_89276_object56446:head# 2024-04-23T16:25:49.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.120 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f17c9be:::benchmark_data_smithi188_89276_object49956:head# 2024-04-23T16:25:49.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1899d9:::benchmark_data_smithi188_89276_object168197:head# 2024-04-23T16:25:49.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f18ca26:::benchmark_data_smithi188_89276_object62324:head# 2024-04-23T16:25:49.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.129 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:25:49.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f190aee:::benchmark_data_smithi188_89276_object89797:head# 2024-04-23T16:25:49.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f191d80:::benchmark_data_smithi188_89276_object136124:head# 2024-04-23T16:25:49.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1981fd:::benchmark_data_smithi188_89276_object46175:head# 2024-04-23T16:25:49.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f19df19:::benchmark_data_smithi188_89276_object167169:head# 2024-04-23T16:25:49.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1a1c5e:::benchmark_data_smithi188_89276_object56497:head# 2024-04-23T16:25:49.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1a6236:::benchmark_data_smithi188_89276_object128951:head# 2024-04-23T16:25:49.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1ac694:::benchmark_data_smithi188_89276_object84154:head# 2024-04-23T16:25:49.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1b5fcf:::benchmark_data_smithi188_89276_object154728:head# 2024-04-23T16:25:49.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1bb9af:::benchmark_data_smithi188_89276_object59425:head# 2024-04-23T16:25:49.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1bbff9:::benchmark_data_smithi188_89276_object139687:head# 2024-04-23T16:25:49.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1c054d:::benchmark_data_smithi188_89276_object74803:head# 2024-04-23T16:25:49.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1cadf6:::benchmark_data_smithi188_89276_object81257:head# 2024-04-23T16:25:49.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1cff1d:::benchmark_data_smithi188_89276_object44923:head# 2024-04-23T16:25:49.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1dbdc7:::benchmark_data_smithi188_89276_object52939:head# 2024-04-23T16:25:49.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1e0799:::benchmark_data_smithi188_89276_object113032:head# 2024-04-23T16:25:49.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1e6876:::benchmark_data_smithi188_89276_object154326:head# 2024-04-23T16:25:49.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.135 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1e9d2a:::benchmark_data_smithi188_89276_object62202:head# 2024-04-23T16:25:49.135 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.135 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1ed0d1:::benchmark_data_smithi188_89276_object55086:head# 2024-04-23T16:25:49.135 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.135 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1eed16:::benchmark_data_smithi188_89276_object133301:head# 2024-04-23T16:25:49.135 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1f1936:::benchmark_data_smithi188_89276_object120188:head# 2024-04-23T16:25:49.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f1f3229:::benchmark_data_smithi188_89276_object139672:head# 2024-04-23T16:25:49.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f20089e:::benchmark_data_smithi188_89276_object82690:head# 2024-04-23T16:25:49.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f203104:::benchmark_data_smithi188_89276_object98271:head# 2024-04-23T16:25:49.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f21548b:::benchmark_data_smithi188_89276_object104811:head# 2024-04-23T16:25:49.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f220972:::benchmark_data_smithi188_89276_object42717:head# 2024-04-23T16:25:49.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f229033:::benchmark_data_smithi188_89276_object82837:head# 2024-04-23T16:25:49.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f2413e4:::benchmark_data_smithi188_89276_object78322:head# 2024-04-23T16:25:49.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f2440d2:::benchmark_data_smithi188_89276_object52149:head# 2024-04-23T16:25:49.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f2443b3:::benchmark_data_smithi188_89276_object133122:head# 2024-04-23T16:25:49.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f244ada:::benchmark_data_smithi188_89276_object66865:head# 2024-04-23T16:25:49.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f248cf0:::benchmark_data_smithi188_89276_object127730:head# 2024-04-23T16:25:49.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f24ca31:::benchmark_data_smithi188_89276_object106749:head# 2024-04-23T16:25:49.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f24dcfc:::benchmark_data_smithi188_89276_object158047:head# 2024-04-23T16:25:49.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f2554fe:::benchmark_data_smithi188_89276_object47770:head# 2024-04-23T16:25:49.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f267247:::benchmark_data_smithi188_89276_object90979:head# 2024-04-23T16:25:49.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f278959:::benchmark_data_smithi188_89276_object90685:head# 2024-04-23T16:25:49.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f27bfcd:::benchmark_data_smithi188_89276_object59680:head# 2024-04-23T16:25:49.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f27d16f:::benchmark_data_smithi188_89276_object60177:head# 2024-04-23T16:25:49.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f281abb:::benchmark_data_smithi188_89276_object167601:head# 2024-04-23T16:25:49.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f2860f7:::benchmark_data_smithi188_89276_object60098:head# 2024-04-23T16:25:49.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f287e4c:::benchmark_data_smithi188_89276_object137712:head# 2024-04-23T16:25:49.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f299cc5:::benchmark_data_smithi188_89276_object60107:head# 2024-04-23T16:25:49.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f2a12df:::benchmark_data_smithi188_89276_object101391:head# 2024-04-23T16:25:49.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f2b1589:::benchmark_data_smithi188_89276_object43637:head# 2024-04-23T16:25:49.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f2b3733:::benchmark_data_smithi188_89276_object104842:head# 2024-04-23T16:25:49.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f2c664a:::benchmark_data_smithi188_89276_object59324:head# 2024-04-23T16:25:49.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f2dc3b6:::benchmark_data_smithi188_89276_object161831:head# 2024-04-23T16:25:49.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f2dc40b:::benchmark_data_smithi188_89276_object56368:head# 2024-04-23T16:25:49.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f2e18e7:::benchmark_data_smithi188_89276_object149098:head# 2024-04-23T16:25:49.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f2f0cfe:::benchmark_data_smithi188_89276_object40307:head# 2024-04-23T16:25:49.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f300dea:::benchmark_data_smithi188_89276_object57012:head# 2024-04-23T16:25:49.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f31b36d:::benchmark_data_smithi188_89276_object76640:head# 2024-04-23T16:25:49.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f31da87:::benchmark_data_smithi188_89276_object51006:head# 2024-04-23T16:25:49.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f32a27c:::benchmark_data_smithi188_89276_object67085:head# 2024-04-23T16:25:49.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f33dbc8:::benchmark_data_smithi188_89276_object70657:head# 2024-04-23T16:25:49.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f349ab4:::benchmark_data_smithi188_89276_object117269:head# 2024-04-23T16:25:49.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f351dfa:::benchmark_data_smithi188_89276_object103367:head# 2024-04-23T16:25:49.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f3564b8:::benchmark_data_smithi188_89276_object148373:head# 2024-04-23T16:25:49.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f357428:::benchmark_data_smithi188_89276_object139017:head# 2024-04-23T16:25:49.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f38de3b:::benchmark_data_smithi188_89276_object119624:head# 2024-04-23T16:25:49.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f392818:::benchmark_data_smithi188_89276_object67550:head# 2024-04-23T16:25:49.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f3b205d:::benchmark_data_smithi188_89276_object158849:head# 2024-04-23T16:25:49.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f3b4132:::benchmark_data_smithi188_89276_object163909:head# 2024-04-23T16:25:49.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f3c02b6:::benchmark_data_smithi188_89276_object74316:head# 2024-04-23T16:25:49.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f3c4bab:::benchmark_data_smithi188_89276_object64681:head# 2024-04-23T16:25:49.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f3c6519:::benchmark_data_smithi188_89276_object153877:head# 2024-04-23T16:25:49.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f3c843b:::benchmark_data_smithi188_89276_object167071:head# 2024-04-23T16:25:49.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f3caa08:::benchmark_data_smithi188_89276_object99205:head# 2024-04-23T16:25:49.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f3d98fc:::benchmark_data_smithi188_89276_object168312:head# 2024-04-23T16:25:49.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f3de9cb:::benchmark_data_smithi188_89276_object52072:head# 2024-04-23T16:25:49.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f3e5746:::benchmark_data_smithi188_89276_object101461:head# 2024-04-23T16:25:49.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f3fef96:::benchmark_data_smithi188_89276_object143072:head# 2024-04-23T16:25:49.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f402a25:::benchmark_data_smithi188_89276_object127152:head# 2024-04-23T16:25:49.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f406f75:::benchmark_data_smithi188_89276_object92502:head# 2024-04-23T16:25:49.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f40e0e8:::benchmark_data_smithi188_89276_object51401:head# 2024-04-23T16:25:49.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f40f8b5:::benchmark_data_smithi188_89276_object112012:head# 2024-04-23T16:25:49.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f422964:::benchmark_data_smithi188_89276_object115394:head# 2024-04-23T16:25:49.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f42b35f:::benchmark_data_smithi188_89276_object85570:head# 2024-04-23T16:25:49.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f43017c:::benchmark_data_smithi188_89276_object84707:head# 2024-04-23T16:25:49.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.153 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:25:49.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f434cc3:::benchmark_data_smithi188_89276_object82338:head# 2024-04-23T16:25:49.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f4488fb:::benchmark_data_smithi188_89276_object79356:head# 2024-04-23T16:25:49.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f44cd8f:::benchmark_data_smithi188_89276_object151199:head# 2024-04-23T16:25:49.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.168 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:25:49.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f450573:::benchmark_data_smithi188_89276_object159860:head# 2024-04-23T16:25:49.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f459573:::benchmark_data_smithi188_89276_object140874:head# 2024-04-23T16:25:49.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f462e1b:::benchmark_data_smithi188_89276_object41889:head# 2024-04-23T16:25:49.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f46b61f:::benchmark_data_smithi188_89276_object168199:head# 2024-04-23T16:25:49.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f472f1b:::benchmark_data_smithi188_89276_object65619:head# 2024-04-23T16:25:49.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f482791:::benchmark_data_smithi188_89276_object94470:head# 2024-04-23T16:25:49.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f48a5c0:::benchmark_data_smithi188_89276_object166397:head# 2024-04-23T16:25:49.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f48c7ee:::benchmark_data_smithi188_89276_object116405:head# 2024-04-23T16:25:49.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f49ed13:::benchmark_data_smithi188_89276_object155445:head# 2024-04-23T16:25:49.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f4b0257:::benchmark_data_smithi188_89276_object118329:head# 2024-04-23T16:25:49.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f4c02b2:::benchmark_data_smithi188_89276_object74179:head# 2024-04-23T16:25:49.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f4cb84a:::benchmark_data_smithi188_89276_object71516:head# 2024-04-23T16:25:49.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f4d1a88:::benchmark_data_smithi188_89276_object106876:head# 2024-04-23T16:25:49.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f4ddfd5:::benchmark_data_smithi188_89276_object100520:head# 2024-04-23T16:25:49.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f4eb129:::benchmark_data_smithi188_89276_object54847:head# 2024-04-23T16:25:49.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f4f9c27:::benchmark_data_smithi188_89276_object130514:head# 2024-04-23T16:25:49.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f50c20f:::benchmark_data_smithi188_89276_object126704:head# 2024-04-23T16:25:49.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f51f265:::benchmark_data_smithi188_89276_object52904:head# 2024-04-23T16:25:49.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f526135:::benchmark_data_smithi188_89276_object50401:head# 2024-04-23T16:25:49.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f52a406:::benchmark_data_smithi188_89276_object123998:head# 2024-04-23T16:25:49.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f52c09f:::benchmark_data_smithi188_89276_object41546:head# 2024-04-23T16:25:49.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f5303d8:::benchmark_data_smithi188_89276_object55877:head# 2024-04-23T16:25:49.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.185 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:49.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f534886:::benchmark_data_smithi188_89276_object120650:head# 2024-04-23T16:25:49.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f53bfc3:::benchmark_data_smithi188_89276_object140841:head# 2024-04-23T16:25:49.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f53c37f:::benchmark_data_smithi188_89276_object104480:head# 2024-04-23T16:25:49.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f541798:::benchmark_data_smithi188_89276_object73376:head# 2024-04-23T16:25:49.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f543082:::benchmark_data_smithi188_89276_object125846:head# 2024-04-23T16:25:49.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f54d573:::benchmark_data_smithi188_89276_object133666:head# 2024-04-23T16:25:49.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f555b6b:::benchmark_data_smithi188_89276_object71439:head# 2024-04-23T16:25:49.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f55ee0d:::benchmark_data_smithi188_89276_object150953:head# 2024-04-23T16:25:49.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f56f212:::benchmark_data_smithi188_89276_object167301:head# 2024-04-23T16:25:49.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f57083b:::benchmark_data_smithi188_89276_object61439:head# 2024-04-23T16:25:49.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f57e465:::benchmark_data_smithi188_89276_object69617:head# 2024-04-23T16:25:49.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f581a9d:::benchmark_data_smithi188_89276_object98114:head# 2024-04-23T16:25:49.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f5869a9:::benchmark_data_smithi188_89276_object161255:head# 2024-04-23T16:25:49.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f587012:::benchmark_data_smithi188_89276_object77006:head# 2024-04-23T16:25:49.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f589ea5:::benchmark_data_smithi188_89276_object46392:head# 2024-04-23T16:25:49.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f59b6b2:::benchmark_data_smithi188_89276_object65436:head# 2024-04-23T16:25:49.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f59f0fc:::benchmark_data_smithi188_89276_object120384:head# 2024-04-23T16:25:49.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f5a0df5:::benchmark_data_smithi188_89276_object111588:head# 2024-04-23T16:25:49.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f5a7ae2:::benchmark_data_smithi188_89276_object147449:head# 2024-04-23T16:25:49.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f5ac5f4:::benchmark_data_smithi188_89276_object53937:head# 2024-04-23T16:25:49.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f5af96b:::benchmark_data_smithi188_89276_object119454:head# 2024-04-23T16:25:49.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f5b0951:::benchmark_data_smithi188_89276_object99045:head# 2024-04-23T16:25:49.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f5b7557:::benchmark_data_smithi188_89276_object74350:head# 2024-04-23T16:25:49.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f5c2e44:::benchmark_data_smithi188_89276_object53509:head# 2024-04-23T16:25:49.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f5dd69e:::benchmark_data_smithi188_89276_object57015:head# 2024-04-23T16:25:49.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f5dd786:::benchmark_data_smithi188_89276_object160086:head# 2024-04-23T16:25:49.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.198 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:25:49.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f5e8b76:::benchmark_data_smithi188_89276_object55553:head# 2024-04-23T16:25:49.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f5eadc9:::benchmark_data_smithi188_89276_object89882:head# 2024-04-23T16:25:49.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f5ee045:::benchmark_data_smithi188_89276_object122511:head# 2024-04-23T16:25:49.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f5f25a6:::benchmark_data_smithi188_89276_object94012:head# 2024-04-23T16:25:49.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f5f608a:::benchmark_data_smithi188_89276_object75553:head# 2024-04-23T16:25:49.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f5fbf98:::benchmark_data_smithi188_89276_object85606:head# 2024-04-23T16:25:49.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f5fe25f:::benchmark_data_smithi188_89276_object100843:head# 2024-04-23T16:25:49.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f602e94:::benchmark_data_smithi188_89276_object99852:head# 2024-04-23T16:25:49.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f606347:::benchmark_data_smithi188_89276_object61171:head# 2024-04-23T16:25:49.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f606d97:::benchmark_data_smithi188_89276_object129553:head# 2024-04-23T16:25:49.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f60e119:::benchmark_data_smithi188_89276_object95409:head# 2024-04-23T16:25:49.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f612550:::benchmark_data_smithi188_89276_object101801:head# 2024-04-23T16:25:49.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f62fa36:::benchmark_data_smithi188_89276_object137645:head# 2024-04-23T16:25:49.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f633c0a:::benchmark_data_smithi188_89276_object131026:head# 2024-04-23T16:25:49.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f63957f:::benchmark_data_smithi188_89276_object91504:head# 2024-04-23T16:25:49.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f63a1f3:::benchmark_data_smithi188_89276_object168558:head# 2024-04-23T16:25:49.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f6500cf:::benchmark_data_smithi188_89276_object167573:head# 2024-04-23T16:25:49.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f650aad:::benchmark_data_smithi188_89276_object65163:head# 2024-04-23T16:25:49.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f65c580:::benchmark_data_smithi188_89276_object139244:head# 2024-04-23T16:25:49.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f660bf6:::benchmark_data_smithi188_89276_object109768:head# 2024-04-23T16:25:49.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f670989:::benchmark_data_smithi188_89276_object41667:head# 2024-04-23T16:25:49.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f6765c4:::benchmark_data_smithi188_89276_object113846:head# 2024-04-23T16:25:49.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f689fc7:::benchmark_data_smithi188_89276_object109856:head# 2024-04-23T16:25:49.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f68f058:::benchmark_data_smithi188_89276_object114346:head# 2024-04-23T16:25:49.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f697868:::benchmark_data_smithi188_89276_object145397:head# 2024-04-23T16:25:49.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f6a0aab:::benchmark_data_smithi188_89276_object43947:head# 2024-04-23T16:25:49.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f6a6e31:::benchmark_data_smithi188_89276_object118069:head# 2024-04-23T16:25:49.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f6b3e9b:::benchmark_data_smithi188_89276_object51791:head# 2024-04-23T16:25:49.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f6b578a:::benchmark_data_smithi188_89276_object48646:head# 2024-04-23T16:25:49.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f6bce1c:::benchmark_data_smithi188_89276_object133088:head# 2024-04-23T16:25:49.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f6bf4a0:::benchmark_data_smithi188_89276_object60839:head# 2024-04-23T16:25:49.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f6c30bf:::benchmark_data_smithi188_89276_object67665:head# 2024-04-23T16:25:49.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f6cebf0:::benchmark_data_smithi188_89276_object57206:head# 2024-04-23T16:25:49.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f6d56d5:::benchmark_data_smithi188_89276_object128066:head# 2024-04-23T16:25:49.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f6dbc05:::benchmark_data_smithi188_89276_object45526:head# 2024-04-23T16:25:49.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f6e61c9:::benchmark_data_smithi188_89276_object81111:head# 2024-04-23T16:25:49.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f6ee989:::benchmark_data_smithi188_89276_object163925:head# 2024-04-23T16:25:49.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f6f068c:::benchmark_data_smithi188_89276_object59786:head# 2024-04-23T16:25:49.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f6f0cda:::benchmark_data_smithi188_89276_object74639:head# 2024-04-23T16:25:49.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f70c000:::benchmark_data_smithi188_89276_object118474:head# 2024-04-23T16:25:49.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f70ce86:::benchmark_data_smithi188_89276_object58000:head# 2024-04-23T16:25:49.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.217 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:25:49.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:49.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:25:49.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f7115ec:::benchmark_data_smithi188_89276_object58206:head# 2024-04-23T16:25:49.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f718d77:::benchmark_data_smithi188_89276_object79329:head# 2024-04-23T16:25:49.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f71bbee:::benchmark_data_smithi188_89276_object76314:head# 2024-04-23T16:25:49.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f722381:::benchmark_data_smithi188_89276_object110859:head# 2024-04-23T16:25:49.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f725605:::benchmark_data_smithi188_89276_object44536:head# 2024-04-23T16:25:49.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f731b4b:::benchmark_data_smithi188_89276_object147415:head# 2024-04-23T16:25:49.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f732178:::benchmark_data_smithi188_89276_object86873:head# 2024-04-23T16:25:49.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.235 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:25:49.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f73660b:::benchmark_data_smithi188_89276_object41283:head# 2024-04-23T16:25:49.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f7369b2:::benchmark_data_smithi188_89276_object144731:head# 2024-04-23T16:25:49.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f73b8d3:::benchmark_data_smithi188_89276_object156165:head# 2024-04-23T16:25:49.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f742019:::benchmark_data_smithi188_89276_object118558:head# 2024-04-23T16:25:49.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f747b59:::benchmark_data_smithi188_89276_object124364:head# 2024-04-23T16:25:49.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f750324:::benchmark_data_smithi188_89276_object142473:head# 2024-04-23T16:25:49.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f75cb53:::benchmark_data_smithi188_89276_object150816:head# 2024-04-23T16:25:49.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f76011c:::benchmark_data_smithi188_89276_object133498:head# 2024-04-23T16:25:49.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f7665d1:::benchmark_data_smithi188_89276_object76534:head# 2024-04-23T16:25:49.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f773dd6:::benchmark_data_smithi188_89276_object75531:head# 2024-04-23T16:25:49.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f777fbd:::benchmark_data_smithi188_89276_object75192:head# 2024-04-23T16:25:49.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f781095:::benchmark_data_smithi188_89276_object107055:head# 2024-04-23T16:25:49.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f782069:::benchmark_data_smithi188_89276_object162860:head# 2024-04-23T16:25:49.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f7865f9:::benchmark_data_smithi188_89276_object56449:head# 2024-04-23T16:25:49.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f794e04:::benchmark_data_smithi188_89276_object95065:head# 2024-04-23T16:25:49.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f79621e:::benchmark_data_smithi188_89276_object72302:head# 2024-04-23T16:25:49.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f7a5ffd:::benchmark_data_smithi188_89276_object146121:head# 2024-04-23T16:25:49.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f7aac48:::benchmark_data_smithi188_89276_object140191:head# 2024-04-23T16:25:49.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f7b1707:::benchmark_data_smithi188_89276_object61166:head# 2024-04-23T16:25:49.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f7c0b43:::benchmark_data_smithi188_89276_object138901:head# 2024-04-23T16:25:49.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f7ca2a2:::benchmark_data_smithi188_89276_object133553:head# 2024-04-23T16:25:49.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f7e4368:::benchmark_data_smithi188_89276_object123691:head# 2024-04-23T16:25:49.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f7e828a:::benchmark_data_smithi188_89276_object140515:head# 2024-04-23T16:25:49.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f7e86cb:::benchmark_data_smithi188_89276_object62252:head# 2024-04-23T16:25:49.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f7fc557:::benchmark_data_smithi188_89276_object147014:head# 2024-04-23T16:25:49.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f806ff7:::benchmark_data_smithi188_89276_object119067:head# 2024-04-23T16:25:49.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f8162bc:::benchmark_data_smithi188_89276_object134853:head# 2024-04-23T16:25:49.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f81faa3:::benchmark_data_smithi188_89276_object135494:head# 2024-04-23T16:25:49.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f821044:::benchmark_data_smithi188_89276_object135700:head# 2024-04-23T16:25:49.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f8276d2:::benchmark_data_smithi188_89276_object148875:head# 2024-04-23T16:25:49.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f8282b7:::benchmark_data_smithi188_89276_object52806:head# 2024-04-23T16:25:49.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f830b37:::benchmark_data_smithi188_89276_object92484:head# 2024-04-23T16:25:49.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f84683e:::benchmark_data_smithi188_89276_object123837:head# 2024-04-23T16:25:49.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f84f95c:::benchmark_data_smithi188_89276_object152914:head# 2024-04-23T16:25:49.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f856f7a:::benchmark_data_smithi188_89276_object65613:head# 2024-04-23T16:25:49.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f861134:::benchmark_data_smithi188_89276_object69965:head# 2024-04-23T16:25:49.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f8687b4:::benchmark_data_smithi188_89276_object60536:head# 2024-04-23T16:25:49.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f868dc3:::benchmark_data_smithi188_89276_object69240:head# 2024-04-23T16:25:49.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f86a96d:::benchmark_data_smithi188_89276_object80866:head# 2024-04-23T16:25:49.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f86cbbd:::benchmark_data_smithi188_89276_object115828:head# 2024-04-23T16:25:49.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f86e21a:::benchmark_data_smithi188_89276_object133794:head# 2024-04-23T16:25:49.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f8701b1:::benchmark_data_smithi188_89276_object85537:head# 2024-04-23T16:25:49.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f8729ce:::benchmark_data_smithi188_89276_object96288:head# 2024-04-23T16:25:49.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f877fb9:::benchmark_data_smithi188_89276_object76302:head# 2024-04-23T16:25:49.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f88b128:::benchmark_data_smithi188_89276_object134339:head# 2024-04-23T16:25:49.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f88bbf0:::benchmark_data_smithi188_89276_object93861:head# 2024-04-23T16:25:49.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f89b51c:::benchmark_data_smithi188_89276_object106646:head# 2024-04-23T16:25:49.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f8b04d5:::benchmark_data_smithi188_89276_object98654:head# 2024-04-23T16:25:49.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f8c4ebd:::benchmark_data_smithi188_89276_object81187:head# 2024-04-23T16:25:49.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f8c9387:::benchmark_data_smithi188_89276_object143611:head# 2024-04-23T16:25:49.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f8c9e15:::benchmark_data_smithi188_89276_object94099:head# 2024-04-23T16:25:49.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f8cae1b:::benchmark_data_smithi188_89276_object121644:head# 2024-04-23T16:25:49.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f8cdeac:::benchmark_data_smithi188_89276_object56753:head# 2024-04-23T16:25:49.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f8e1426:::benchmark_data_smithi188_89276_object151177:head# 2024-04-23T16:25:49.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f8f53cc:::benchmark_data_smithi188_89276_object84488:head# 2024-04-23T16:25:49.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f8f7a7a:::benchmark_data_smithi188_89276_object81370:head# 2024-04-23T16:25:49.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f8f8dcc:::benchmark_data_smithi188_89276_object77406:head# 2024-04-23T16:25:49.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f8fa454:::benchmark_data_smithi188_89276_object80923:head# 2024-04-23T16:25:49.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f91016a:::benchmark_data_smithi188_89276_object67690:head# 2024-04-23T16:25:49.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f91aa32:::benchmark_data_smithi188_89276_object135207:head# 2024-04-23T16:25:49.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f91d64e:::benchmark_data_smithi188_89276_object92567:head# 2024-04-23T16:25:49.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f934ac2:::benchmark_data_smithi188_89276_object126344:head# 2024-04-23T16:25:49.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f93afbb:::benchmark_data_smithi188_89276_object73125:head# 2024-04-23T16:25:49.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f946cc9:::benchmark_data_smithi188_89276_object48853:head# 2024-04-23T16:25:49.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f9478b1:::benchmark_data_smithi188_89276_object63199:head# 2024-04-23T16:25:49.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f947db3:::benchmark_data_smithi188_89276_object165112:head# 2024-04-23T16:25:49.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f948962:::benchmark_data_smithi188_89276_object157922:head# 2024-04-23T16:25:49.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f953754:::benchmark_data_smithi188_89276_object90359:head# 2024-04-23T16:25:49.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f9580d1:::benchmark_data_smithi188_89276_object64187:head# 2024-04-23T16:25:49.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f95cdf7:::benchmark_data_smithi188_89276_object168258:head# 2024-04-23T16:25:49.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f963543:::benchmark_data_smithi188_89276_object118408:head# 2024-04-23T16:25:49.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f9770de:::benchmark_data_smithi188_89276_object158613:head# 2024-04-23T16:25:49.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f999c1f:::benchmark_data_smithi188_89276_object158410:head# 2024-04-23T16:25:49.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f99d3ee:::benchmark_data_smithi188_89276_object57805:head# 2024-04-23T16:25:49.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f9a5eaa:::benchmark_data_smithi188_89276_object87742:head# 2024-04-23T16:25:49.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f9bb958:::benchmark_data_smithi188_89276_object136315:head# 2024-04-23T16:25:49.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f9bf3cc:::benchmark_data_smithi188_89276_object104976:head# 2024-04-23T16:25:49.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f9c18d9:::benchmark_data_smithi188_89276_object45497:head# 2024-04-23T16:25:49.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f9d19e3:::benchmark_data_smithi188_89276_object164276:head# 2024-04-23T16:25:49.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f9d2ba2:::benchmark_data_smithi188_89276_object97120:head# 2024-04-23T16:25:49.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f9d2fe2:::benchmark_data_smithi188_89276_object139812:head# 2024-04-23T16:25:49.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f9da850:::benchmark_data_smithi188_89276_object114063:head# 2024-04-23T16:25:49.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f9e39bf:::benchmark_data_smithi188_89276_object112622:head# 2024-04-23T16:25:49.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f9e66c6:::benchmark_data_smithi188_89276_object105713:head# 2024-04-23T16:25:49.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f9f0508:::benchmark_data_smithi188_89276_object113429:head# 2024-04-23T16:25:49.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f9f76ee:::benchmark_data_smithi188_89276_object120996:head# 2024-04-23T16:25:49.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2f9f7d7b:::benchmark_data_smithi188_89276_object69082:head# 2024-04-23T16:25:49.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fa08571:::benchmark_data_smithi188_89276_object96126:head# 2024-04-23T16:25:49.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.268 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:25:49.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fa166cd:::benchmark_data_smithi188_89276_object153126:head# 2024-04-23T16:25:49.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fa1911b:::benchmark_data_smithi188_89276_object64630:head# 2024-04-23T16:25:49.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fa1a001:::benchmark_data_smithi188_89276_object65290:head# 2024-04-23T16:25:49.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fa1d922:::benchmark_data_smithi188_89276_object81785:head# 2024-04-23T16:25:49.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fa36e43:::benchmark_data_smithi188_89276_object62121:head# 2024-04-23T16:25:49.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fa40375:::benchmark_data_smithi188_89276_object146028:head# 2024-04-23T16:25:49.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fa425d6:::benchmark_data_smithi188_89276_object61449:head# 2024-04-23T16:25:49.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fa43230:::benchmark_data_smithi188_89276_object43997:head# 2024-04-23T16:25:49.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fa4fc5e:::benchmark_data_smithi188_89276_object125216:head# 2024-04-23T16:25:49.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fa56350:::benchmark_data_smithi188_89276_object63927:head# 2024-04-23T16:25:49.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fa5958a:::benchmark_data_smithi188_89276_object151424:head# 2024-04-23T16:25:49.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fa66c00:::benchmark_data_smithi188_89276_object62380:head# 2024-04-23T16:25:49.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fa6afa6:::benchmark_data_smithi188_89276_object123076:head# 2024-04-23T16:25:49.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fa7df79:::benchmark_data_smithi188_89276_object125574:head# 2024-04-23T16:25:49.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fa7f681:::benchmark_data_smithi188_89276_object63283:head# 2024-04-23T16:25:49.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fa99d90:::benchmark_data_smithi188_89276_object41959:head# 2024-04-23T16:25:49.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fa9fff8:::benchmark_data_smithi188_89276_object146900:head# 2024-04-23T16:25:49.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2faac681:::benchmark_data_smithi188_89276_object57586:head# 2024-04-23T16:25:49.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fabc75b:::benchmark_data_smithi188_89276_object105828:head# 2024-04-23T16:25:49.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fac2a66:::benchmark_data_smithi188_89276_object155972:head# 2024-04-23T16:25:49.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fac4a62:::benchmark_data_smithi188_89276_object54482:head# 2024-04-23T16:25:49.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2faca227:::benchmark_data_smithi188_89276_object121053:head# 2024-04-23T16:25:49.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2faca974:::benchmark_data_smithi188_89276_object135490:head# 2024-04-23T16:25:49.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fadd7f7:::benchmark_data_smithi188_89276_object154038:head# 2024-04-23T16:25:49.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2faddda7:::benchmark_data_smithi188_89276_object120599:head# 2024-04-23T16:25:49.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fae16aa:::benchmark_data_smithi188_89276_object119450:head# 2024-04-23T16:25:49.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fae672b:::benchmark_data_smithi188_89276_object85305:head# 2024-04-23T16:25:49.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fae8bdf:::benchmark_data_smithi188_89276_object127568:head# 2024-04-23T16:25:49.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2faeb6b6:::benchmark_data_smithi188_89276_object116484:head# 2024-04-23T16:25:49.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2faed123:::benchmark_data_smithi188_89276_object142832:head# 2024-04-23T16:25:49.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fb0d9b8:::benchmark_data_smithi188_89276_object90737:head# 2024-04-23T16:25:49.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.298 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:25:49.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fb12903:::benchmark_data_smithi188_89276_object160474:head# 2024-04-23T16:25:49.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fb4ce44:::benchmark_data_smithi188_89276_object64564:head# 2024-04-23T16:25:49.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fb4da0d:::benchmark_data_smithi188_89276_object63001:head# 2024-04-23T16:25:49.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fb59e71:::benchmark_data_smithi188_89276_object100302:head# 2024-04-23T16:25:49.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fb5c6b3:::benchmark_data_smithi188_89276_object82013:head# 2024-04-23T16:25:49.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fb60abc:::benchmark_data_smithi188_89276_object99087:head# 2024-04-23T16:25:49.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fb6c854:::benchmark_data_smithi188_89276_object119266:head# 2024-04-23T16:25:49.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fb790c7:::benchmark_data_smithi188_89276_object164828:head# 2024-04-23T16:25:49.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fb79cdb:::benchmark_data_smithi188_89276_object165065:head# 2024-04-23T16:25:49.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fb815c1:::benchmark_data_smithi188_89276_object74327:head# 2024-04-23T16:25:49.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fb834ca:::benchmark_data_smithi188_89276_object49818:head# 2024-04-23T16:25:49.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fb83524:::benchmark_data_smithi188_89276_object102146:head# 2024-04-23T16:25:49.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fb91c15:::benchmark_data_smithi188_89276_object152598:head# 2024-04-23T16:25:49.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fba0191:::benchmark_data_smithi188_89276_object41644:head# 2024-04-23T16:25:49.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fbacd08:::benchmark_data_smithi188_89276_object98565:head# 2024-04-23T16:25:49.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fbc3d8a:::benchmark_data_smithi188_89276_object143873:head# 2024-04-23T16:25:49.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fbc57fc:::benchmark_data_smithi188_89276_object44258:head# 2024-04-23T16:25:49.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fbc597e:::benchmark_data_smithi188_89276_object40707:head# 2024-04-23T16:25:49.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fbc92a4:::benchmark_data_smithi188_89276_object132736:head# 2024-04-23T16:25:49.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fbd70df:::benchmark_data_smithi188_89276_object163099:head# 2024-04-23T16:25:49.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fbdc3de:::benchmark_data_smithi188_89276_object164322:head# 2024-04-23T16:25:49.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fbe6ee2:::benchmark_data_smithi188_89276_object69242:head# 2024-04-23T16:25:49.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fbea400:::benchmark_data_smithi188_89276_object86488:head# 2024-04-23T16:25:49.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fbee1f8:::benchmark_data_smithi188_89276_object139749:head# 2024-04-23T16:25:49.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fbee8bc:::benchmark_data_smithi188_89276_object149011:head# 2024-04-23T16:25:49.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fbf154d:::benchmark_data_smithi188_89276_object88417:head# 2024-04-23T16:25:49.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fbfa652:::benchmark_data_smithi188_89276_object99954:head# 2024-04-23T16:25:49.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fbfc426:::benchmark_data_smithi188_89276_object55353:head# 2024-04-23T16:25:49.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc06dbf:::benchmark_data_smithi188_89276_object153478:head# 2024-04-23T16:25:49.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc07a6b:::benchmark_data_smithi188_89276_object80475:head# 2024-04-23T16:25:49.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc0cdf2:::benchmark_data_smithi188_89276_object159450:head# 2024-04-23T16:25:49.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc13a86:::benchmark_data_smithi188_89276_object136349:head# 2024-04-23T16:25:49.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc1804d:::benchmark_data_smithi188_89276_object98452:head# 2024-04-23T16:25:49.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc1bbb3:::benchmark_data_smithi188_89276_object104198:head# 2024-04-23T16:25:49.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc25f93:::benchmark_data_smithi188_89276_object134365:head# 2024-04-23T16:25:49.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc2cf61:::benchmark_data_smithi188_89276_object134800:head# 2024-04-23T16:25:49.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc32ec3:::benchmark_data_smithi188_89276_object150047:head# 2024-04-23T16:25:49.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc35348:::benchmark_data_smithi188_89276_object116910:head# 2024-04-23T16:25:49.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc36cf1:::benchmark_data_smithi188_89276_object124952:head# 2024-04-23T16:25:49.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc3bde7:::benchmark_data_smithi188_89276_object129055:head# 2024-04-23T16:25:49.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc4098a:::benchmark_data_smithi188_89276_object98883:head# 2024-04-23T16:25:49.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc4133b:::benchmark_data_smithi188_89276_object51454:head# 2024-04-23T16:25:49.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.320 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:25:49.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc46018:::benchmark_data_smithi188_89276_object74071:head# 2024-04-23T16:25:49.322 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc47da3:::benchmark_data_smithi188_89276_object103503:head# 2024-04-23T16:25:49.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc50b5b:::benchmark_data_smithi188_89276_object70287:head# 2024-04-23T16:25:49.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc560d8:::benchmark_data_smithi188_89276_object153206:head# 2024-04-23T16:25:49.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc64269:::benchmark_data_smithi188_89276_object106000:head# 2024-04-23T16:25:49.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc6626d:::benchmark_data_smithi188_89276_object147605:head# 2024-04-23T16:25:49.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc7267e:::benchmark_data_smithi188_89276_object45027:head# 2024-04-23T16:25:49.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.324 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc7304e:::benchmark_data_smithi188_89276_object95238:head# 2024-04-23T16:25:49.324 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.324 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc73acb:::benchmark_data_smithi188_89276_object81450:head# 2024-04-23T16:25:49.324 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.324 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc7b0de:::benchmark_data_smithi188_89276_object120675:head# 2024-04-23T16:25:49.324 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.328 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc84b39:::benchmark_data_smithi188_89276_object136439:head# 2024-04-23T16:25:49.328 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc8b6b6:::benchmark_data_smithi188_89276_object148534:head# 2024-04-23T16:25:49.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc95510:::benchmark_data_smithi188_89276_object50204:head# 2024-04-23T16:25:49.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fc97c88:::benchmark_data_smithi188_89276_object152984:head# 2024-04-23T16:25:49.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fcac7db:::benchmark_data_smithi188_89276_object59038:head# 2024-04-23T16:25:49.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fcb1299:::benchmark_data_smithi188_89276_object92611:head# 2024-04-23T16:25:49.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fcbe9ee:::benchmark_data_smithi188_89276_object111747:head# 2024-04-23T16:25:49.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fcbf212:::benchmark_data_smithi188_89276_object158242:head# 2024-04-23T16:25:49.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fcc2758:::benchmark_data_smithi188_89276_object113651:head# 2024-04-23T16:25:49.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fccff71:::benchmark_data_smithi188_89276_object127443:head# 2024-04-23T16:25:49.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fcdf4b5:::benchmark_data_smithi188_89276_object76818:head# 2024-04-23T16:25:49.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fce3fb2:::benchmark_data_smithi188_89276_object97316:head# 2024-04-23T16:25:49.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fce9e6b:::benchmark_data_smithi188_89276_object132070:head# 2024-04-23T16:25:49.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fcea5ba:::benchmark_data_smithi188_89276_object109385:head# 2024-04-23T16:25:49.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fcf3cb4:::benchmark_data_smithi188_89276_object134809:head# 2024-04-23T16:25:49.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fcfca87:::benchmark_data_smithi188_89276_object108750:head# 2024-04-23T16:25:49.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fd0f66b:::benchmark_data_smithi188_89276_object83657:head# 2024-04-23T16:25:49.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.331 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:25:49.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fd25a7d:::benchmark_data_smithi188_89276_object98634:head# 2024-04-23T16:25:49.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.332 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:25:49.332 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:25:49.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fd2f380:::benchmark_data_smithi188_89276_object53832:head# 2024-04-23T16:25:49.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fd44afb:::benchmark_data_smithi188_89276_object49852:head# 2024-04-23T16:25:49.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fd46ef5:::benchmark_data_smithi188_89276_object124014:head# 2024-04-23T16:25:49.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fd4f648:::benchmark_data_smithi188_89276_object92909:head# 2024-04-23T16:25:49.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.337 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fd5eb3e:::benchmark_data_smithi188_89276_object75345:head# 2024-04-23T16:25:49.337 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.344 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fd6c8f6:::benchmark_data_smithi188_89276_object80724:head# 2024-04-23T16:25:49.344 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fd76c6c:::benchmark_data_smithi188_89276_object127345:head# 2024-04-23T16:25:49.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fd7e334:::benchmark_data_smithi188_89276_object117176:head# 2024-04-23T16:25:49.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fd7f846:::benchmark_data_smithi188_89276_object124365:head# 2024-04-23T16:25:49.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fd81cf9:::benchmark_data_smithi188_89276_object162747:head# 2024-04-23T16:25:49.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fd951fd:::benchmark_data_smithi188_89276_object92846:head# 2024-04-23T16:25:49.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fd9ac00:::benchmark_data_smithi188_89276_object165817:head# 2024-04-23T16:25:49.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fda2886:::benchmark_data_smithi188_89276_object54546:head# 2024-04-23T16:25:49.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fda8348:::benchmark_data_smithi188_89276_object78769:head# 2024-04-23T16:25:49.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fdb6291:::benchmark_data_smithi188_89276_object150582:head# 2024-04-23T16:25:49.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fdd7de9:::benchmark_data_smithi188_89276_object121055:head# 2024-04-23T16:25:49.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fdeadef:::benchmark_data_smithi188_89276_object164002:head# 2024-04-23T16:25:49.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fdf59d5:::benchmark_data_smithi188_89276_object163258:head# 2024-04-23T16:25:49.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fdfff7d:::benchmark_data_smithi188_89276_object92951:head# 2024-04-23T16:25:49.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe00103:::benchmark_data_smithi188_89276_object96153:head# 2024-04-23T16:25:49.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe03c44:::benchmark_data_smithi188_89276_object129167:head# 2024-04-23T16:25:49.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe04393:::benchmark_data_smithi188_89276_object127753:head# 2024-04-23T16:25:49.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe0920e:::benchmark_data_smithi188_89276_object75369:head# 2024-04-23T16:25:49.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe142c7:::benchmark_data_smithi188_89276_object83286:head# 2024-04-23T16:25:49.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe2128e:::benchmark_data_smithi188_89276_object44136:head# 2024-04-23T16:25:49.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe349f1:::benchmark_data_smithi188_89276_object139268:head# 2024-04-23T16:25:49.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.358 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:25:49.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe47225:::benchmark_data_smithi188_89276_object151040:head# 2024-04-23T16:25:49.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe49525:::benchmark_data_smithi188_89276_object150195:head# 2024-04-23T16:25:49.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe50322:::benchmark_data_smithi188_89276_object65442:head# 2024-04-23T16:25:49.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe52778:::benchmark_data_smithi188_89276_object74670:head# 2024-04-23T16:25:49.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe54e85:::benchmark_data_smithi188_89276_object129903:head# 2024-04-23T16:25:49.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe5976d:::benchmark_data_smithi188_89276_object44404:head# 2024-04-23T16:25:49.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe6e90c:::benchmark_data_smithi188_89276_object51170:head# 2024-04-23T16:25:49.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe72412:::benchmark_data_smithi188_89276_object102220:head# 2024-04-23T16:25:49.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe84634:::benchmark_data_smithi188_89276_object61648:head# 2024-04-23T16:25:49.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe84eac:::benchmark_data_smithi188_89276_object53749:head# 2024-04-23T16:25:49.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe8ae61:::benchmark_data_smithi188_89276_object47835:head# 2024-04-23T16:25:49.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe8b85a:::benchmark_data_smithi188_89276_object135726:head# 2024-04-23T16:25:49.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe904bd:::benchmark_data_smithi188_89276_object144515:head# 2024-04-23T16:25:49.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe973a3:::benchmark_data_smithi188_89276_object96037:head# 2024-04-23T16:25:49.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.365 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:49.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fe9d5c1:::benchmark_data_smithi188_89276_object63468:head# 2024-04-23T16:25:49.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fea3951:::benchmark_data_smithi188_89276_object83663:head# 2024-04-23T16:25:49.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2feb67d6:::benchmark_data_smithi188_89276_object116309:head# 2024-04-23T16:25:49.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fec4b4c:::benchmark_data_smithi188_89276_object68575:head# 2024-04-23T16:25:49.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2feca18a:::benchmark_data_smithi188_89276_object58002:head# 2024-04-23T16:25:49.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fecb237:::benchmark_data_smithi188_89276_object107808:head# 2024-04-23T16:25:49.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fed1866:::benchmark_data_smithi188_89276_object51133:head# 2024-04-23T16:25:49.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2feda75a:::benchmark_data_smithi188_89276_object104233:head# 2024-04-23T16:25:49.387 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.387 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fedadcb:::benchmark_data_smithi188_89276_object144295:head# 2024-04-23T16:25:49.387 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.387 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fef286e:::benchmark_data_smithi188_89276_object65671:head# 2024-04-23T16:25:49.387 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fef3495:::benchmark_data_smithi188_89276_object41092:head# 2024-04-23T16:25:49.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2fef9e96:::benchmark_data_smithi188_89276_object155762:head# 2024-04-23T16:25:49.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff01528:::benchmark_data_smithi188_89276_object59142:head# 2024-04-23T16:25:49.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff03377:::benchmark_data_smithi188_89276_object50481:head# 2024-04-23T16:25:49.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff0718f:::benchmark_data_smithi188_89276_object54272:head# 2024-04-23T16:25:49.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff0d3ff:::benchmark_data_smithi188_89276_object158216:head# 2024-04-23T16:25:49.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff15bc2:::benchmark_data_smithi188_89276_object59700:head# 2024-04-23T16:25:49.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff18b9e:::benchmark_data_smithi188_89276_object43030:head# 2024-04-23T16:25:49.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff20e6f:::benchmark_data_smithi188_89276_object150724:head# 2024-04-23T16:25:49.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff3613a:::benchmark_data_smithi188_89276_object161095:head# 2024-04-23T16:25:49.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff38772:::benchmark_data_smithi188_89276_object156242:head# 2024-04-23T16:25:49.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff3a247:::benchmark_data_smithi188_89276_object48635:head# 2024-04-23T16:25:49.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff428dc:::benchmark_data_smithi188_89276_object92992:head# 2024-04-23T16:25:49.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff464f1:::benchmark_data_smithi188_89276_object84835:head# 2024-04-23T16:25:49.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff4ab04:::benchmark_data_smithi188_89276_object126930:head# 2024-04-23T16:25:49.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff4eeee:::benchmark_data_smithi188_89276_object111958:head# 2024-04-23T16:25:49.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.393 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:25:49.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff569e4:::benchmark_data_smithi188_89276_object41760:head# 2024-04-23T16:25:49.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.394 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:25:49.394 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T16:25:49.394 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:25:49.394 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:25:49.394 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:25:49.394 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:25:49.395 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:25:49.395 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:25:49.395 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:25:49.395 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:25:49.395 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:25:49.395 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:25:49.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff588b2:::benchmark_data_smithi188_89276_object140900:head# 2024-04-23T16:25:49.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff650cd:::benchmark_data_smithi188_89276_object85446:head# 2024-04-23T16:25:49.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff690b7:::benchmark_data_smithi188_89276_object111499:head# 2024-04-23T16:25:49.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff6d178:::benchmark_data_smithi188_89276_object101587:head# 2024-04-23T16:25:49.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff784b9:::benchmark_data_smithi188_89276_object140989:head# 2024-04-23T16:25:49.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff7f3b1:::benchmark_data_smithi188_89276_object73405:head# 2024-04-23T16:25:49.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff891de:::benchmark_data_smithi188_89276_object157415:head# 2024-04-23T16:25:49.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff8dbcb:::benchmark_data_smithi188_89276_object125864:head# 2024-04-23T16:25:49.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff9820b:::benchmark_data_smithi188_89276_object112537:head# 2024-04-23T16:25:49.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff990ba:::benchmark_data_smithi188_89276_object98984:head# 2024-04-23T16:25:49.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ff9bf77:::benchmark_data_smithi188_89276_object46905:head# 2024-04-23T16:25:49.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ffa97ed:::benchmark_data_smithi188_89276_object65102:head# 2024-04-23T16:25:49.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ffb018c:::benchmark_data_smithi188_89276_object165222:head# 2024-04-23T16:25:49.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ffb15a3:::benchmark_data_smithi188_89276_object64661:head# 2024-04-23T16:25:49.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ffb60c5:::benchmark_data_smithi188_89276_object117699:head# 2024-04-23T16:25:49.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ffc0965:::benchmark_data_smithi188_89276_object90825:head# 2024-04-23T16:25:49.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ffc5dd8:::benchmark_data_smithi188_89276_object121015:head# 2024-04-23T16:25:49.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ffc670f:::benchmark_data_smithi188_89276_object156604:head# 2024-04-23T16:25:49.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ffe0625:::benchmark_data_smithi188_89276_object156855:head# 2024-04-23T16:25:49.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ffe9e81:::benchmark_data_smithi188_89276_object81891:head# 2024-04-23T16:25:49.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.438 INFO:teuthology.orchestra.run.smithi103.stdout:Write #3:2ffee255:::benchmark_data_smithi188_89276_object92386:head# 2024-04-23T16:25:49.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:25:49.450 INFO:teuthology.orchestra.run.smithi103.stdout:write_pg epoch 232 info 3.4( v 232'14672 (204'4600,232'14672] local-lis/les=192/193 n=7913 ec=192/192 lis/c=192/192 les/c/f=193/193/0 sis=192) 2024-04-23T16:25:49.519 INFO:teuthology.orchestra.run.smithi103.stdout:Import successful 2024-04-23T16:25:49.683 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:25:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:49 smithi188 ceph-mon[44101]: 3.5 scrub starts 2024-04-23T16:25:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:49 smithi188 ceph-mon[44101]: pgmap v741: 17 pgs: 17 active+clean; 5.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-23T16:25:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:49 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4287301657' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:25:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:49 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1522334881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:25:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:49 smithi188 ceph-mon[44101]: 3.0 scrub starts 2024-04-23T16:25:49.977 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:49 smithi103 ceph-mon[39629]: 3.5 scrub starts 2024-04-23T16:25:49.977 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:49 smithi103 ceph-mon[39629]: pgmap v741: 17 pgs: 17 active+clean; 5.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-23T16:25:49.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:49 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4287301657' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:25:49.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:49 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1522334881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:25:49.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:49.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:49.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:49.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:49.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:49.978 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:49 smithi103 ceph-mon[39629]: 3.0 scrub starts 2024-04-23T16:25:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:49 smithi148 ceph-mon[44520]: 3.5 scrub starts 2024-04-23T16:25:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:49 smithi148 ceph-mon[44520]: pgmap v741: 17 pgs: 17 active+clean; 5.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-23T16:25:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:49 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4287301657' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:25:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:49 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1522334881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:25:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:50.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:50.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:50.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:50.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:49 smithi148 ceph-mon[44520]: 3.0 scrub starts 2024-04-23T16:25:50.676 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:50.676 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:50.676 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:50.676 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:50.676 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:50.676 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:50 smithi148 ceph-mon[44520]: 3.5 scrub ok 2024-04-23T16:25:50.676 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:50 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:25:50.677 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:50 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:25:50.677 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:50 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4287301657' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:25:50.677 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:50 smithi148 ceph-mon[44520]: osdmap e255: 12 total, 11 up, 10 in 2024-04-23T16:25:50.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:50.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:50.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:50.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:50.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:50.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:50 smithi188 ceph-mon[44101]: 3.5 scrub ok 2024-04-23T16:25:50.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:50 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:25:50.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:50 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:25:50.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:50 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4287301657' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:25:50.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:50 smithi188 ceph-mon[44101]: osdmap e255: 12 total, 11 up, 10 in 2024-04-23T16:25:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:25:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:50 smithi103 ceph-mon[39629]: 3.5 scrub ok 2024-04-23T16:25:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:50 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:25:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:50 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:25:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:50 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4287301657' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:25:51.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:50 smithi103 ceph-mon[39629]: osdmap e255: 12 total, 11 up, 10 in 2024-04-23T16:25:51.263 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:25:51.264 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:25:51.471 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:25:51.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:51.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:25:51.575 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:25:51.615 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:51.878 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:25:51.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:51.886 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:51 smithi103 ceph-mon[39629]: 3.0 scrub ok 2024-04-23T16:25:51.886 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:51 smithi103 ceph-mon[39629]: pgmap v743: 17 pgs: 17 active+clean; 5.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:25:51.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:25:51.899 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-23T16:25:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:51 smithi188 ceph-mon[44101]: 3.0 scrub ok 2024-04-23T16:25:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:51 smithi188 ceph-mon[44101]: pgmap v743: 17 pgs: 17 active+clean; 5.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:25:51.988 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:25:52.021 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:51 smithi148 ceph-mon[44520]: 3.0 scrub ok 2024-04-23T16:25:52.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:51 smithi148 ceph-mon[44520]: pgmap v743: 17 pgs: 17 active+clean; 5.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:25:52.468 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/exp.3.4.1 2024-04-23T16:25:52.492 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:25:52.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:52.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:25:52.596 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:25:52.599 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-04-23T16:25:52.599 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-23T16:25:52.599 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:25:52.622 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:25:52.652 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:52.709 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:52 smithi103 ceph-mon[39629]: 3.f scrub starts 2024-04-23T16:25:52.737 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:25:52.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:52 smithi188 ceph-mon[44101]: 3.f scrub starts 2024-04-23T16:25:53.028 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:25:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:53.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:25:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:52 smithi148 ceph-mon[44520]: 3.f scrub starts 2024-04-23T16:25:53.140 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:25:53.186 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:53.239 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:25:53.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:53.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:25:53.265 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:25:53.346 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:25:53.379 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:53.407 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:25:53.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:53.410 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:25:53.511 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:53.550 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:53.592 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:53.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:53.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:53.706 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:25:53.707 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:53 smithi188 ceph-mon[44101]: 3.f scrub ok 2024-04-23T16:25:53.707 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:53 smithi188 ceph-mon[44101]: 1.0 deep-scrub starts 2024-04-23T16:25:53.707 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:53 smithi188 ceph-mon[44101]: 1.0 deep-scrub ok 2024-04-23T16:25:53.707 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:53 smithi188 ceph-mon[44101]: pgmap v744: 17 pgs: 17 active+clean; 4.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:25:53.767 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:53.806 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:25:53.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:53.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:25:53.910 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:25:53.949 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:53 smithi103 ceph-mon[39629]: 3.f scrub ok 2024-04-23T16:25:53.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:53 smithi103 ceph-mon[39629]: 1.0 deep-scrub starts 2024-04-23T16:25:53.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:53 smithi103 ceph-mon[39629]: 1.0 deep-scrub ok 2024-04-23T16:25:53.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:53 smithi103 ceph-mon[39629]: pgmap v744: 17 pgs: 17 active+clean; 4.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:25:53.959 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:54.114 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:25:54.120 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:25:54.121 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:25:54.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:53 smithi148 ceph-mon[44520]: 3.f scrub ok 2024-04-23T16:25:54.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:53 smithi148 ceph-mon[44520]: 1.0 deep-scrub starts 2024-04-23T16:25:54.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:53 smithi148 ceph-mon[44520]: 1.0 deep-scrub ok 2024-04-23T16:25:54.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:53 smithi148 ceph-mon[44520]: pgmap v744: 17 pgs: 17 active+clean; 4.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:25:54.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:54.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:25:54.239 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:25:54.294 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:54.335 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:25:54.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:54.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:25:54.442 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:25:54.479 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:54.516 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:25:54.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:54.519 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:25:54.620 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:25:54.656 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:54.682 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:25:54.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:54.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:25:54.786 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:25:54.828 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:54.853 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:25:54.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:54.856 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:25:54.957 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:25:54.992 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:55.025 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:25:55.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:55.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:25:55.136 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:55.152 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:25:55.178 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:55 smithi188 ceph-mon[44101]: pgmap v745: 17 pgs: 17 active+clean; 4.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:25:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:55 smithi103 ceph-mon[39629]: pgmap v745: 17 pgs: 17 active+clean; 4.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:25:55.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:55 smithi148 ceph-mon[44520]: pgmap v745: 17 pgs: 17 active+clean; 4.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:25:55.419 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:55.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:55.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:55.533 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:25:55.572 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:57.428 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:25:57.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:57.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:25:57.532 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:57.563 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:57 smithi103 ceph-mon[39629]: pgmap v746: 17 pgs: 17 active+clean; 4.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:25:57.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:57 smithi103 ceph-mon[39629]: 3.9 scrub starts 2024-04-23T16:25:57.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:57 smithi188 ceph-mon[44101]: pgmap v746: 17 pgs: 17 active+clean; 4.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:25:57.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:57 smithi188 ceph-mon[44101]: 3.9 scrub starts 2024-04-23T16:25:58.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:57 smithi148 ceph-mon[44520]: pgmap v746: 17 pgs: 17 active+clean; 4.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:25:58.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:57 smithi148 ceph-mon[44520]: 3.9 scrub starts 2024-04-23T16:25:58.415 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:58.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:58.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:58.520 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:25:58.551 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:58.717 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:25:58.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:58.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:25:58.821 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:25:58.857 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:58.908 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:25:58.908 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:25:58.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:58 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:25:58.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:58 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:25:58.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:58 smithi188 ceph-mon[44101]: 3.9 scrub ok 2024-04-23T16:25:59.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:58 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:25:59.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:58 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:25:59.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:58 smithi103 ceph-mon[39629]: 3.9 scrub ok 2024-04-23T16:25:59.114 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:25:59.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:59.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:25:59.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:58 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:25:59.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:58 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:25:59.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:58 smithi148 ceph-mon[44520]: 3.9 scrub ok 2024-04-23T16:25:59.218 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:25:59.256 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:59.394 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:25:59.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:59.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:25:59.502 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:25:59.537 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:59.560 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:25:59.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:59.566 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:25:59.667 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:25:59.718 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:59.742 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:25:59.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:59.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:25:59.847 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:25:59.884 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:25:59.915 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:25:59.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:25:59.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:25:59.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:59 smithi188 ceph-mon[44101]: pgmap v747: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 11 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:25:59.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:25:59 smithi188 ceph-mon[44101]: osdmap e256: 12 total, 11 up, 10 in 2024-04-23T16:25:59.979 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:25:59.990 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:26:00.022 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:26:00.026 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:59 smithi103 ceph-mon[39629]: pgmap v747: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 11 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:26:00.026 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:25:59 smithi103 ceph-mon[39629]: osdmap e256: 12 total, 11 up, 10 in 2024-04-23T16:26:00.064 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:00.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:59 smithi148 ceph-mon[44520]: pgmap v747: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 11 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:26:00.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:25:59 smithi148 ceph-mon[44520]: osdmap e256: 12 total, 11 up, 10 in 2024-04-23T16:26:00.243 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-23T16:26:00.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:00 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3620508510' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:26:00.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:00 smithi188 ceph-mon[44101]: from='client.25255 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:26:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:00 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3620508510' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:26:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:00 smithi103 ceph-mon[39629]: from='client.25255 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:26:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:00 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3620508510' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:26:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:00 smithi148 ceph-mon[44520]: from='client.25255 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:26:01.142 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:26:01.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:01.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:26:01.248 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:26:01.279 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:01.322 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":748,"stamp":"2024-04-23T16:25:58.686803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4672455224,"num_objects":71290,"num_object_clones":0,"num_object_copies":142580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71290,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":266177,"num_write_kb":10795963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24062,"num_bytes_recovered":1578518656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160800,"ondisk_log_size":160800,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":937451520,"kb_used":11027772,"kb_used_data":9320744,"kb_used_omap":15,"kb_used_meta":1706864,"kb_avail":926423748,"statfs":{"total":959950356480,"available":948657917952,"internally_reserved":0,"allocated":9544441856,"data_stored":9541126218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15948,"internal_metadata":1747829172},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-706543616,"num_objects":-10781,"num_object_clones":0,"num_object_copies":-21562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10781,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.008388"},"pg_stats":[{"pgid":"3.f","version":"255'16691","reported_seq":17041,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-23T16:25:54.671073+0000","last_change":"2024-04-23T16:25:52.488019+0000","last_active":"2024-04-23T16:25:54.671073+0000","last_peered":"2024-04-23T16:25:54.671073+0000","last_clean":"2024-04-23T16:25:54.671073+0000","last_became_active":"2024-04-23T16:22:11.619226+0000","last_became_peered":"2024-04-23T16:22:11.619226+0000","last_unstale":"2024-04-23T16:25:54.671073+0000","last_undegraded":"2024-04-23T16:25:54.671073+0000","last_fullsized":"2024-04-23T16:25:54.671073+0000","mapping_epoch":192,"log_start":"209'6600","ondisk_log_start":"209'6600","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'16533","last_scrub_stamp":"2024-04-23T16:25:52.487870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:52.487870+0000","objects_scrubbed":4765,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:19.498435+0000","scrub_duration":999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299040768,"num_objects":4563,"num_object_clones":0,"num_object_copies":9126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16691,"num_write_kb":680128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"255'16693","reported_seq":16996,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-23T16:25:56.333417+0000","last_change":"2024-04-23T16:24:39.023998+0000","last_active":"2024-04-23T16:25:56.333417+0000","last_peered":"2024-04-23T16:25:56.333417+0000","last_clean":"2024-04-23T16:25:56.333417+0000","last_became_active":"2024-04-23T16:22:11.618509+0000","last_became_peered":"2024-04-23T16:22:11.618509+0000","last_unstale":"2024-04-23T16:25:56.333417+0000","last_undegraded":"2024-04-23T16:25:56.333417+0000","last_fullsized":"2024-04-23T16:25:56.333417+0000","mapping_epoch":192,"log_start":"209'6600","ondisk_log_start":"209'6600","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'13046","last_scrub_stamp":"2024-04-23T16:24:39.023867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:24:39.023867+0000","objects_scrubbed":7982,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:25:59.930741+0000","scrub_duration":733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284098560,"num_objects":4335,"num_object_clones":0,"num_object_copies":8670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16693,"num_write_kb":672896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"255'21507","reported_seq":21999,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-23T16:25:58.105792+0000","last_change":"2024-04-23T16:25:37.975879+0000","last_active":"2024-04-23T16:25:58.105792+0000","last_peered":"2024-04-23T16:25:58.105792+0000","last_clean":"2024-04-23T16:25:58.105792+0000","last_became_active":"2024-04-23T16:22:11.616591+0000","last_became_peered":"2024-04-23T16:22:11.616591+0000","last_unstale":"2024-04-23T16:25:58.105792+0000","last_undegraded":"2024-04-23T16:25:58.105792+0000","last_fullsized":"2024-04-23T16:25:58.105792+0000","mapping_epoch":192,"log_start":"229'11501","ondisk_log_start":"229'11501","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'19917","last_scrub_stamp":"2024-04-23T16:25:37.975750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:37.975750+0000","objects_scrubbed":5860,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:26:41.837100+0000","scrub_duration":1618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275316760,"num_objects":4202,"num_object_clones":0,"num_object_copies":8404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4202,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16758,"num_write_kb":670657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"255'16884","reported_seq":17195,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-23T16:25:58.109116+0000","last_change":"2024-04-23T16:24:41.812462+0000","last_active":"2024-04-23T16:25:58.109116+0000","last_peered":"2024-04-23T16:25:58.109116+0000","last_clean":"2024-04-23T16:25:58.109116+0000","last_became_active":"2024-04-23T16:22:11.616487+0000","last_became_peered":"2024-04-23T16:22:11.616487+0000","last_unstale":"2024-04-23T16:25:58.109116+0000","last_undegraded":"2024-04-23T16:25:58.109116+0000","last_fullsized":"2024-04-23T16:25:58.109116+0000","mapping_epoch":192,"log_start":"210'6800","ondisk_log_start":"210'6800","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'13202","last_scrub_stamp":"2024-04-23T16:24:41.812321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:24:41.812321+0000","objects_scrubbed":8034,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:26:05.997906+0000","scrub_duration":755,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285212672,"num_objects":4352,"num_object_clones":0,"num_object_copies":8704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16884,"num_write_kb":679552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"255'16626","reported_seq":16937,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-23T16:25:55.818269+0000","last_change":"2024-04-23T16:24:40.077638+0000","last_active":"2024-04-23T16:25:55.818269+0000","last_peered":"2024-04-23T16:25:55.818269+0000","last_clean":"2024-04-23T16:25:55.818269+0000","last_became_active":"2024-04-23T16:22:11.615324+0000","last_became_peered":"2024-04-23T16:22:11.615324+0000","last_unstale":"2024-04-23T16:25:55.818269+0000","last_undegraded":"2024-04-23T16:25:55.818269+0000","last_fullsized":"2024-04-23T16:25:55.818269+0000","mapping_epoch":192,"log_start":"209'6600","ondisk_log_start":"209'6600","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'13079","last_scrub_stamp":"2024-04-23T16:24:40.077504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:24:40.077504+0000","objects_scrubbed":8107,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:26:00.452549+0000","scrub_duration":678,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298844160,"num_objects":4560,"num_object_clones":0,"num_object_copies":9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16626,"num_write_kb":677952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"255'16376","reported_seq":16709,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-23T16:25:54.015354+0000","last_change":"2024-04-23T16:25:36.443258+0000","last_active":"2024-04-23T16:25:54.015354+0000","last_peered":"2024-04-23T16:25:54.015354+0000","last_clean":"2024-04-23T16:25:54.015354+0000","last_became_active":"2024-04-23T16:22:11.616590+0000","last_became_peered":"2024-04-23T16:22:11.616590+0000","last_unstale":"2024-04-23T16:25:54.015354+0000","last_undegraded":"2024-04-23T16:25:54.015354+0000","last_fullsized":"2024-04-23T16:25:54.015354+0000","mapping_epoch":192,"log_start":"208'6300","ondisk_log_start":"208'6300","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14959","last_scrub_stamp":"2024-04-23T16:25:36.442995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:36.442995+0000","objects_scrubbed":6023,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:26:55.901259+0000","scrub_duration":1101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298844160,"num_objects":4560,"num_object_clones":0,"num_object_copies":9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16376,"num_write_kb":669952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"255'16706","reported_seq":17052,"reported_epoch":255,"state":"active+clean+scrubbing","last_fresh":"2024-04-23T16:25:58.184743+0000","last_change":"2024-04-23T16:25:57.499860+0000","last_active":"2024-04-23T16:25:58.184743+0000","last_peered":"2024-04-23T16:25:58.184743+0000","last_clean":"2024-04-23T16:25:58.184743+0000","last_became_active":"2024-04-23T16:22:11.618760+0000","last_became_peered":"2024-04-23T16:22:11.618760+0000","last_unstale":"2024-04-23T16:25:58.184743+0000","last_undegraded":"2024-04-23T16:25:58.184743+0000","last_fullsized":"2024-04-23T16:25:58.184743+0000","mapping_epoch":192,"log_start":"210'6700","ondisk_log_start":"210'6700","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'13051","last_scrub_stamp":"2024-04-23T16:24:39.272154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:24:39.272154+0000","objects_scrubbed":3799,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":644,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287178752,"num_objects":4382,"num_object_clones":0,"num_object_copies":8764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16706,"num_write_kb":674816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"255'16834","reported_seq":33410,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-23T16:25:56.335857+0000","last_change":"2024-04-23T16:25:40.784964+0000","last_active":"2024-04-23T16:25:56.335857+0000","last_peered":"2024-04-23T16:25:56.335857+0000","last_clean":"2024-04-23T16:25:56.335857+0000","last_became_active":"2024-04-23T16:25:07.676206+0000","last_became_peered":"2024-04-23T16:25:07.676206+0000","last_unstale":"2024-04-23T16:25:56.335857+0000","last_undegraded":"2024-04-23T16:25:56.335857+0000","last_fullsized":"2024-04-23T16:25:56.335857+0000","mapping_epoch":244,"log_start":"209'6800","ondisk_log_start":"209'6800","created":192,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'15635","last_scrub_stamp":"2024-04-23T16:25:40.784747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:40.784747+0000","objects_scrubbed":5792,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:26:56.977250+0000","scrub_duration":1304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298188800,"num_objects":4550,"num_object_clones":0,"num_object_copies":9100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16834,"num_write_kb":684288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8134,"num_bytes_recovered":533069824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"255'16843","reported_seq":17153,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-23T16:25:55.818808+0000","last_change":"2024-04-23T16:24:47.952146+0000","last_active":"2024-04-23T16:25:55.818808+0000","last_peered":"2024-04-23T16:25:55.818808+0000","last_clean":"2024-04-23T16:25:55.818808+0000","last_became_active":"2024-04-23T16:22:11.619141+0000","last_became_peered":"2024-04-23T16:22:11.619141+0000","last_unstale":"2024-04-23T16:25:55.818808+0000","last_undegraded":"2024-04-23T16:25:55.818808+0000","last_fullsized":"2024-04-23T16:25:55.818808+0000","mapping_epoch":192,"log_start":"209'6800","ondisk_log_start":"209'6800","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'13282","last_scrub_stamp":"2024-04-23T16:24:47.952029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:24:47.952029+0000","objects_scrubbed":8030,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:26:04.775833+0000","scrub_duration":656,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292880384,"num_objects":4469,"num_object_clones":0,"num_object_copies":8938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16843,"num_write_kb":681984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"255'16283","reported_seq":32608,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-23T16:25:53.691940+0000","last_change":"2024-04-23T16:25:33.363231+0000","last_active":"2024-04-23T16:25:53.691940+0000","last_peered":"2024-04-23T16:25:53.691940+0000","last_clean":"2024-04-23T16:25:53.691940+0000","last_became_active":"2024-04-23T16:25:08.678140+0000","last_became_peered":"2024-04-23T16:25:08.678140+0000","last_unstale":"2024-04-23T16:25:53.691940+0000","last_undegraded":"2024-04-23T16:25:53.691940+0000","last_fullsized":"2024-04-23T16:25:53.691940+0000","mapping_epoch":245,"log_start":"208'6200","ondisk_log_start":"208'6200","created":192,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14713","last_scrub_stamp":"2024-04-23T16:25:33.363009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:33.363009+0000","objects_scrubbed":6244,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:26:58.427475+0000","scrub_duration":1212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303497216,"num_objects":4631,"num_object_clones":0,"num_object_copies":9262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16283,"num_write_kb":669248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8007,"num_bytes_recovered":524746752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"255'16701","reported_seq":17075,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-23T16:25:58.085596+0000","last_change":"2024-04-23T16:25:49.363034+0000","last_active":"2024-04-23T16:25:58.085596+0000","last_peered":"2024-04-23T16:25:58.085596+0000","last_clean":"2024-04-23T16:25:58.085596+0000","last_became_active":"2024-04-23T16:22:11.616683+0000","last_became_peered":"2024-04-23T16:22:11.616683+0000","last_unstale":"2024-04-23T16:25:58.085596+0000","last_undegraded":"2024-04-23T16:25:58.085596+0000","last_fullsized":"2024-04-23T16:25:58.085596+0000","mapping_epoch":192,"log_start":"209'6700","ondisk_log_start":"209'6700","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'16005","last_scrub_stamp":"2024-04-23T16:25:49.362861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:49.362861+0000","objects_scrubbed":4958,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:13.160535+0000","scrub_duration":988,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277413888,"num_objects":4233,"num_object_clones":0,"num_object_copies":8466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16701,"num_write_kb":669888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"255'18305","reported_seq":34561,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-23T16:25:56.318792+0000","last_change":"2024-04-23T16:25:33.698766+0000","last_active":"2024-04-23T16:25:56.318792+0000","last_peered":"2024-04-23T16:25:56.318792+0000","last_clean":"2024-04-23T16:25:56.318792+0000","last_became_active":"2024-04-23T16:25:07.676317+0000","last_became_peered":"2024-04-23T16:25:07.676317+0000","last_unstale":"2024-04-23T16:25:56.318792+0000","last_undegraded":"2024-04-23T16:25:56.318792+0000","last_fullsized":"2024-04-23T16:25:56.318792+0000","mapping_epoch":244,"log_start":"217'8300","ondisk_log_start":"217'8300","created":192,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'16525","last_scrub_stamp":"2024-04-23T16:25:33.698520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:33.698520+0000","objects_scrubbed":6094,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:01.572295+0000","scrub_duration":1066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280494080,"num_objects":4280,"num_object_clones":0,"num_object_copies":8560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16432,"num_write_kb":662784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7913,"num_bytes_recovered":518586368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3","version":"255'16505","reported_seq":16824,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-23T16:25:54.091230+0000","last_change":"2024-04-23T16:24:41.533231+0000","last_active":"2024-04-23T16:25:54.091230+0000","last_peered":"2024-04-23T16:25:54.091230+0000","last_clean":"2024-04-23T16:25:54.091230+0000","last_became_active":"2024-04-23T16:22:11.618441+0000","last_became_peered":"2024-04-23T16:22:11.618441+0000","last_unstale":"2024-04-23T16:25:54.091230+0000","last_undegraded":"2024-04-23T16:25:54.091230+0000","last_fullsized":"2024-04-23T16:25:54.091230+0000","mapping_epoch":192,"log_start":"208'6500","ondisk_log_start":"208'6500","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'12982","last_scrub_stamp":"2024-04-23T16:24:41.533078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:24:41.533078+0000","objects_scrubbed":8010,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:26:02.825815+0000","scrub_duration":734,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294060032,"num_objects":4487,"num_object_clones":0,"num_object_copies":8974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16505,"num_write_kb":671744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"255'16245","reported_seq":16634,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-23T16:25:53.683481+0000","last_change":"2024-04-23T16:25:36.467689+0000","last_active":"2024-04-23T16:25:53.683481+0000","last_peered":"2024-04-23T16:25:53.683481+0000","last_clean":"2024-04-23T16:25:53.683481+0000","last_became_active":"2024-04-23T16:22:11.616598+0000","last_became_peered":"2024-04-23T16:22:11.616598+0000","last_unstale":"2024-04-23T16:25:53.683481+0000","last_undegraded":"2024-04-23T16:25:53.683481+0000","last_fullsized":"2024-04-23T16:25:53.683481+0000","mapping_epoch":192,"log_start":"208'6200","ondisk_log_start":"208'6200","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14966","last_scrub_stamp":"2024-04-23T16:25:36.467397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:36.467397+0000","objects_scrubbed":6018,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:03.230886+0000","scrub_duration":1341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307822592,"num_objects":4697,"num_object_clones":0,"num_object_copies":9394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16245,"num_write_kb":670144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"255'18718","reported_seq":19132,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-23T16:25:56.329138+0000","last_change":"2024-04-23T16:25:50.467241+0000","last_active":"2024-04-23T16:25:56.329138+0000","last_peered":"2024-04-23T16:25:56.329138+0000","last_clean":"2024-04-23T16:25:56.329138+0000","last_became_active":"2024-04-23T16:22:11.664655+0000","last_became_peered":"2024-04-23T16:22:11.664655+0000","last_unstale":"2024-04-23T16:25:56.329138+0000","last_undegraded":"2024-04-23T16:25:56.329138+0000","last_fullsized":"2024-04-23T16:25:56.329138+0000","mapping_epoch":192,"log_start":"218'8700","ondisk_log_start":"218'8700","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'18246","last_scrub_stamp":"2024-04-23T16:25:50.467059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:50.467059+0000","objects_scrubbed":4956,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:26:50.497653+0000","scrub_duration":1052,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291504128,"num_objects":4448,"num_object_clones":0,"num_object_copies":8896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16774,"num_write_kb":679104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"255'16645","reported_seq":16960,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-23T16:25:55.842943+0000","last_change":"2024-04-23T16:25:34.355976+0000","last_active":"2024-04-23T16:25:55.842943+0000","last_peered":"2024-04-23T16:25:55.842943+0000","last_clean":"2024-04-23T16:25:55.842943+0000","last_became_active":"2024-04-23T16:22:11.664765+0000","last_became_peered":"2024-04-23T16:22:11.664765+0000","last_unstale":"2024-04-23T16:25:55.842943+0000","last_undegraded":"2024-04-23T16:25:55.842943+0000","last_fullsized":"2024-04-23T16:25:55.842943+0000","mapping_epoch":192,"log_start":"209'6600","ondisk_log_start":"209'6600","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'15004","last_scrub_stamp":"2024-04-23T16:25:34.355861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:34.355861+0000","objects_scrubbed":6218,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:26:54.699437+0000","scrub_duration":1137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297467904,"num_objects":4539,"num_object_clones":0,"num_object_copies":9078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16645,"num_write_kb":677888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":710,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-23T16:25:52.560133+0000","last_change":"2024-04-23T16:25:52.560187+0000","last_active":"2024-04-23T16:25:52.560133+0000","last_peered":"2024-04-23T16:25:52.560133+0000","last_clean":"2024-04-23T16:25:52.560133+0000","last_became_active":"2024-04-23T16:15:36.918552+0000","last_became_peered":"2024-04-23T16:15:36.918552+0000","last_unstale":"2024-04-23T16:25:52.560133+0000","last_undegraded":"2024-04-23T16:25:52.560133+0000","last_fullsized":"2024-04-23T16:25:52.560133+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:25:52.560053+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:03.175518+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":4671864856,"num_objects":71288,"num_object_clones":0,"num_object_copies":142576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71288,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":265996,"num_write_kb":10793025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24054,"num_bytes_recovered":1576402944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11047673856,"data_stored":11047665712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160661,"ondisk_log_size":160661,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056767,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2284440,"kb_used_data":2018576,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":91460712,"statfs":{"total":95995035648,"available":93655769088,"internally_reserved":0,"allocated":2067021824,"data_stored":2066688215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":272169412},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351109,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612747,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":770320,"kb_used_data":604488,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92974832,"statfs":{"total":95995035648,"available":95206227968,"internally_reserved":0,"allocated":618995712,"data_stored":618665675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":169802176},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841679,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":405904,"kb_used_data":284488,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93339248,"statfs":{"total":95995035648,"available":95579389952,"internally_reserved":0,"allocated":291315712,"data_stored":290985675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168724,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":706588,"kb_used_data":581012,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93038564,"statfs":{"total":95995035648,"available":95271489536,"internally_reserved":0,"allocated":594956288,"data_stored":594618627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":128580042},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463066,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":367888,"kb_used_data":296456,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93377264,"statfs":{"total":95995035648,"available":95618318336,"internally_reserved":0,"allocated":303570944,"data_stored":303240907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738551,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1357072,"kb_used_data":1199560,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92388080,"statfs":{"total":95995035648,"available":94605393920,"internally_reserved":0,"allocated":1228349440,"data_stored":1228019403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313388,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":130388,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93614764,"statfs":{"total":95995035648,"available":95861518336,"internally_reserved":0,"allocated":663552,"data_stored":333515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117160,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1693456,"kb_used_data":1463752,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":92051696,"statfs":{"total":95995035648,"available":94260936704,"internally_reserved":0,"allocated":1498882048,"data_stored":1498552011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":235207111},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920930,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1995792,"kb_used_data":1775816,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":91749360,"statfs":{"total":95995035648,"available":93951344640,"internally_reserved":0,"allocated":1818435584,"data_stored":1818105547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757406,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1315924,"kb_used_data":1095948,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92429228,"statfs":{"total":95995035648,"available":94647529472,"internally_reserved":0,"allocated":1122250752,"data_stored":1121916643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1217069056,"data_stored":1217069056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1619918848,"data_stored":1619918848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1474297856,"data_stored":1474297856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1805778944,"data_stored":1805778944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1108545536,"data_stored":1108541464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":298844160,"data_stored":298844160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":592777216,"data_stored":592773144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":287178752,"data_stored":287178752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":612302848,"data_stored":612302848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2030960640,"data_stored":2030960640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:26:01.323 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:26:01.393 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:26:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:01 smithi188 ceph-mon[44101]: from='client.15369 v1:172.21.15.103:0/2688253929' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:26:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:01 smithi188 ceph-mon[44101]: pgmap v749: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:26:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:01 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:26:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:01 smithi188 ceph-mon[44101]: from='client.25255 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:26:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:01 smithi188 ceph-mon[44101]: osdmap e257: 12 total, 11 up, 10 in 2024-04-23T16:26:02.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:01 smithi103 ceph-mon[39629]: from='client.15369 v1:172.21.15.103:0/2688253929' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:26:02.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:01 smithi103 ceph-mon[39629]: pgmap v749: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:26:02.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:01 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:26:02.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:01 smithi103 ceph-mon[39629]: from='client.25255 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:26:02.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:01 smithi103 ceph-mon[39629]: osdmap e257: 12 total, 11 up, 10 in 2024-04-23T16:26:02.074 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:26:02.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:02.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:26:02.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:01 smithi148 ceph-mon[44520]: from='client.15369 v1:172.21.15.103:0/2688253929' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:26:02.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:01 smithi148 ceph-mon[44520]: pgmap v749: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:26:02.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:01 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:26:02.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:01 smithi148 ceph-mon[44520]: from='client.25255 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:26:02.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:01 smithi148 ceph-mon[44520]: osdmap e257: 12 total, 11 up, 10 in 2024-04-23T16:26:02.171 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:26:02.171 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:26:02.172 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:26:02.172 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:26:02.172 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:26:02.172 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:26:02.199 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:26:02.200 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:26:02.228 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:26:02.237 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:02.265 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:26:02.271 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:26:02.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:02.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:26:02.293 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:26:02.323 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:26:02.323 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:26:02.323 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T16:26:02.324 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:26:02.324 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:26:02.324 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:26:02.324 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:26:02.324 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:26:02.324 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:26:02.325 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:26:02.325 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:26:02.325 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:26:02.325 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:26:02.383 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:26:02.420 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:02.456 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:26:02.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:02.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:26:02.563 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:26:02.610 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:02.635 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:26:02.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:02.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:26:02.741 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:26:02.782 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:02.817 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:26:02.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:02.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:26:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:02 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4249057627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:26:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:02.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:02.926 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:26:02.962 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:02 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4249057627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:26:02.962 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:02.963 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:02.963 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:02.963 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:02.963 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:02.963 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:02.965 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:02.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:02 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4249057627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:26:02.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:03.091 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:26:03.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:03.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:26:03.200 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:03.231 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:03.476 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:26:03.477 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:26:03.486 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:03.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:03.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:03.535 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:26:03.541 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:26:03.590 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:26:03.621 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:03.646 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:26:03.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:03.652 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:26:03.703 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:03.703 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:03.703 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:03.703 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:03 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:26:03.703 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:03 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:26:03.703 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:03.703 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:03.703 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:03 smithi188 ceph-mon[44101]: pgmap v751: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.1 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-23T16:26:03.703 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:03 smithi188 ceph-mon[44101]: osdmap e258: 12 total, 11 up, 10 in 2024-04-23T16:26:03.753 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:26:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:03.823 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:26:03.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:03.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:26:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:03 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:26:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:03 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:26:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:03.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:03.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:03 smithi103 ceph-mon[39629]: pgmap v751: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.1 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-23T16:26:03.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:03 smithi103 ceph-mon[39629]: osdmap e258: 12 total, 11 up, 10 in 2024-04-23T16:26:03.933 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:26:03.961 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:03.961 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:03.962 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:03.962 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:03 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:26:03.962 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:03 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:26:03.962 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:03.962 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:03.962 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:03 smithi148 ceph-mon[44520]: pgmap v751: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.1 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-23T16:26:03.962 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:03 smithi148 ceph-mon[44520]: osdmap e258: 12 total, 11 up, 10 in 2024-04-23T16:26:03.963 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:03.996 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:26:04.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:04.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:26:04.103 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:26:04.134 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:04.156 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:26:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:04.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:26:04.260 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:26:04.291 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:04.328 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:26:04.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:04.331 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:26:04.432 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:26:04.490 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:04.691 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:26:04.748 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:26:04.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:04.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:26:04.854 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:26:04.890 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:04.925 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:26:04.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:04.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:26:05.033 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:26:05.064 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:05.089 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:26:05.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:05.095 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:26:05.196 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:26:05.240 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:05.284 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:26:05.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:05.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:26:05.391 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:26:05.424 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:05.447 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:26:05.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:05.451 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:26:05.477 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:26:05.552 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:26:05.615 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:05.639 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:26:05.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:05.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:26:05.709 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:05 smithi188 ceph-mon[44101]: pgmap v753: 17 pgs: 17 active+clean; 3.9 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.33k op/s 2024-04-23T16:26:05.743 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:26:05.803 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:05.840 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:26:05.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:05.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:26:05.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:05 smithi103 ceph-mon[39629]: pgmap v753: 17 pgs: 17 active+clean; 3.9 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.33k op/s 2024-04-23T16:26:05.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:05 smithi148 ceph-mon[44520]: pgmap v753: 17 pgs: 17 active+clean; 3.9 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.33k op/s 2024-04-23T16:26:05.947 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:26:05.990 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:06.023 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:26:06.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:06.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:26:06.131 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:26:06.167 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:06.200 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:26:06.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:06.205 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:26:06.306 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:26:06.340 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:06.370 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:26:06.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:06.374 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:26:06.475 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:26:06.521 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:06.886 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:26:06.886 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:26:06.886 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-23T16:26:06.886 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-23T16:26:06.886 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:26:07.879 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-23T16:26:07.880 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:26:07.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:07.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:26:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:07 smithi188 ceph-mon[44101]: pgmap v754: 17 pgs: 17 active+clean; 3.7 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.41k op/s 2024-04-23T16:26:07.983 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:26:08.040 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:07 smithi103 ceph-mon[39629]: pgmap v754: 17 pgs: 17 active+clean; 3.7 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.41k op/s 2024-04-23T16:26:08.043 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:08.064 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:26:08.064 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:26:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:07 smithi148 ceph-mon[44520]: pgmap v754: 17 pgs: 17 active+clean; 3.7 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.41k op/s 2024-04-23T16:26:08.243 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:26:08.252 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:26:08.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:08.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:26:08.359 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:26:08.387 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:08.724 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:08 smithi103 ceph-mon[39629]: from='client.15378 v1:172.21.15.103:0/279213850' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:26:08.891 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":754,"stamp":"2024-04-23T16:26:06.614057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3942449720,"num_objects":60151,"num_object_clones":0,"num_object_copies":120302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60151,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":277316,"num_write_kb":10795963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24062,"num_bytes_recovered":1578518656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161139,"ondisk_log_size":161139,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":937451520,"kb_used":9493284,"kb_used_data":7811256,"kb_used_omap":15,"kb_used_meta":1681904,"kb_avail":927958236,"statfs":{"total":959950356480,"available":950229233664,"internally_reserved":0,"allocated":7998726144,"data_stored":7995241894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15948,"internal_metadata":1722270132},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,4,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-730005504,"num_objects":-11139,"num_object_clones":0,"num_object_copies":-22278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11139,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.927254"},"pg_stats":[{"pgid":"3.f","version":"258'17487","reported_seq":17843,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-23T16:26:04.681296+0000","last_change":"2024-04-23T16:25:52.488019+0000","last_active":"2024-04-23T16:26:04.681296+0000","last_peered":"2024-04-23T16:26:04.681296+0000","last_clean":"2024-04-23T16:26:04.681296+0000","last_became_active":"2024-04-23T16:22:11.619226+0000","last_became_peered":"2024-04-23T16:22:11.619226+0000","last_unstale":"2024-04-23T16:26:04.681296+0000","last_undegraded":"2024-04-23T16:26:04.681296+0000","last_fullsized":"2024-04-23T16:26:04.681296+0000","mapping_epoch":192,"log_start":"213'7400","ondisk_log_start":"213'7400","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'16533","last_scrub_stamp":"2024-04-23T16:25:52.487870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:52.487870+0000","objects_scrubbed":4765,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:19.498435+0000","scrub_duration":999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246874112,"num_objects":3767,"num_object_clones":0,"num_object_copies":7534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17487,"num_write_kb":680128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"258'17477","reported_seq":17789,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-23T16:26:06.339916+0000","last_change":"2024-04-23T16:24:39.023998+0000","last_active":"2024-04-23T16:26:06.339916+0000","last_peered":"2024-04-23T16:26:06.339916+0000","last_clean":"2024-04-23T16:26:06.339916+0000","last_became_active":"2024-04-23T16:22:11.618509+0000","last_became_peered":"2024-04-23T16:22:11.618509+0000","last_unstale":"2024-04-23T16:26:06.339916+0000","last_undegraded":"2024-04-23T16:26:06.339916+0000","last_fullsized":"2024-04-23T16:26:06.339916+0000","mapping_epoch":192,"log_start":"213'7400","ondisk_log_start":"213'7400","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'13046","last_scrub_stamp":"2024-04-23T16:24:39.023867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:24:39.023867+0000","objects_scrubbed":7982,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232718336,"num_objects":3551,"num_object_clones":0,"num_object_copies":7102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17477,"num_write_kb":672896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"258'21929","reported_seq":22427,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-23T16:26:03.078208+0000","last_change":"2024-04-23T16:25:37.975879+0000","last_active":"2024-04-23T16:26:03.078208+0000","last_peered":"2024-04-23T16:26:03.078208+0000","last_clean":"2024-04-23T16:26:03.078208+0000","last_became_active":"2024-04-23T16:22:11.616591+0000","last_became_peered":"2024-04-23T16:22:11.616591+0000","last_unstale":"2024-04-23T16:26:03.078208+0000","last_undegraded":"2024-04-23T16:26:03.078208+0000","last_fullsized":"2024-04-23T16:26:03.078208+0000","mapping_epoch":192,"log_start":"230'11901","ondisk_log_start":"230'11901","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'19917","last_scrub_stamp":"2024-04-23T16:25:37.975750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:37.975750+0000","objects_scrubbed":5860,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:26:41.837100+0000","scrub_duration":1618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247660568,"num_objects":3780,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3780,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17180,"num_write_kb":670657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"258'17267","reported_seq":17584,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-23T16:26:03.120595+0000","last_change":"2024-04-23T16:24:41.812462+0000","last_active":"2024-04-23T16:26:03.120595+0000","last_peered":"2024-04-23T16:26:03.120595+0000","last_clean":"2024-04-23T16:26:03.120595+0000","last_became_active":"2024-04-23T16:22:11.616487+0000","last_became_peered":"2024-04-23T16:22:11.616487+0000","last_unstale":"2024-04-23T16:26:03.120595+0000","last_undegraded":"2024-04-23T16:26:03.120595+0000","last_fullsized":"2024-04-23T16:26:03.120595+0000","mapping_epoch":192,"log_start":"212'7200","ondisk_log_start":"212'7200","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'13202","last_scrub_stamp":"2024-04-23T16:24:41.812321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:24:41.812321+0000","objects_scrubbed":8034,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:26:05.997906+0000","scrub_duration":755,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260112384,"num_objects":3969,"num_object_clones":0,"num_object_copies":7938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17267,"num_write_kb":679552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"258'17451","reported_seq":17768,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-23T16:26:05.814359+0000","last_change":"2024-04-23T16:24:40.077638+0000","last_active":"2024-04-23T16:26:05.814359+0000","last_peered":"2024-04-23T16:26:05.814359+0000","last_clean":"2024-04-23T16:26:05.814359+0000","last_became_active":"2024-04-23T16:22:11.615324+0000","last_became_peered":"2024-04-23T16:22:11.615324+0000","last_unstale":"2024-04-23T16:26:05.814359+0000","last_undegraded":"2024-04-23T16:26:05.814359+0000","last_fullsized":"2024-04-23T16:26:05.814359+0000","mapping_epoch":192,"log_start":"213'7400","ondisk_log_start":"213'7400","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'13079","last_scrub_stamp":"2024-04-23T16:24:40.077504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:24:40.077504+0000","objects_scrubbed":8107,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:26:07.922201+0000","scrub_duration":678,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244776960,"num_objects":3735,"num_object_clones":0,"num_object_copies":7470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17451,"num_write_kb":677952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"258'17146","reported_seq":17485,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-23T16:26:04.032833+0000","last_change":"2024-04-23T16:25:36.443258+0000","last_active":"2024-04-23T16:26:04.032833+0000","last_peered":"2024-04-23T16:26:04.032833+0000","last_clean":"2024-04-23T16:26:04.032833+0000","last_became_active":"2024-04-23T16:22:11.616590+0000","last_became_peered":"2024-04-23T16:22:11.616590+0000","last_unstale":"2024-04-23T16:26:04.032833+0000","last_undegraded":"2024-04-23T16:26:04.032833+0000","last_fullsized":"2024-04-23T16:26:04.032833+0000","mapping_epoch":192,"log_start":"211'7100","ondisk_log_start":"211'7100","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14959","last_scrub_stamp":"2024-04-23T16:25:36.442995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:36.442995+0000","objects_scrubbed":6023,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:26:55.901259+0000","scrub_duration":1101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248381440,"num_objects":3790,"num_object_clones":0,"num_object_copies":7580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17146,"num_write_kb":669952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"258'17078","reported_seq":17443,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-23T16:26:03.171680+0000","last_change":"2024-04-23T16:25:58.270591+0000","last_active":"2024-04-23T16:26:03.171680+0000","last_peered":"2024-04-23T16:26:03.171680+0000","last_clean":"2024-04-23T16:26:03.171680+0000","last_became_active":"2024-04-23T16:22:11.618760+0000","last_became_peered":"2024-04-23T16:22:11.618760+0000","last_unstale":"2024-04-23T16:26:03.171680+0000","last_undegraded":"2024-04-23T16:26:03.171680+0000","last_fullsized":"2024-04-23T16:26:03.171680+0000","mapping_epoch":192,"log_start":"211'7000","ondisk_log_start":"211'7000","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'16709","last_scrub_stamp":"2024-04-23T16:25:58.270497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:58.270497+0000","objects_scrubbed":4411,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:22.643002+0000","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262799360,"num_objects":4010,"num_object_clones":0,"num_object_copies":8020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17078,"num_write_kb":674816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"258'17674","reported_seq":34256,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-23T16:26:06.337426+0000","last_change":"2024-04-23T16:25:40.784964+0000","last_active":"2024-04-23T16:26:06.337426+0000","last_peered":"2024-04-23T16:26:06.337426+0000","last_clean":"2024-04-23T16:26:06.337426+0000","last_became_active":"2024-04-23T16:25:07.676206+0000","last_became_peered":"2024-04-23T16:25:07.676206+0000","last_unstale":"2024-04-23T16:26:06.337426+0000","last_undegraded":"2024-04-23T16:26:06.337426+0000","last_fullsized":"2024-04-23T16:26:06.337426+0000","mapping_epoch":244,"log_start":"213'7600","ondisk_log_start":"213'7600","created":192,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'15635","last_scrub_stamp":"2024-04-23T16:25:40.784747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:40.784747+0000","objects_scrubbed":5792,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:26:56.977250+0000","scrub_duration":1304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243138560,"num_objects":3710,"num_object_clones":0,"num_object_copies":7420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17674,"num_write_kb":684288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8134,"num_bytes_recovered":533069824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"258'17665","reported_seq":17981,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-23T16:26:05.835149+0000","last_change":"2024-04-23T16:24:47.952146+0000","last_active":"2024-04-23T16:26:05.835149+0000","last_peered":"2024-04-23T16:26:05.835149+0000","last_clean":"2024-04-23T16:26:05.835149+0000","last_became_active":"2024-04-23T16:22:11.619141+0000","last_became_peered":"2024-04-23T16:22:11.619141+0000","last_unstale":"2024-04-23T16:26:05.835149+0000","last_undegraded":"2024-04-23T16:26:05.835149+0000","last_fullsized":"2024-04-23T16:26:05.835149+0000","mapping_epoch":192,"log_start":"213'7600","ondisk_log_start":"213'7600","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'13282","last_scrub_stamp":"2024-04-23T16:24:47.952029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:24:47.952029+0000","objects_scrubbed":8030,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":656,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239009792,"num_objects":3647,"num_object_clones":0,"num_object_copies":7294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17665,"num_write_kb":681984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"258'17070","reported_seq":33401,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-23T16:26:03.699399+0000","last_change":"2024-04-23T16:25:33.363231+0000","last_active":"2024-04-23T16:26:03.699399+0000","last_peered":"2024-04-23T16:26:03.699399+0000","last_clean":"2024-04-23T16:26:03.699399+0000","last_became_active":"2024-04-23T16:25:08.678140+0000","last_became_peered":"2024-04-23T16:25:08.678140+0000","last_unstale":"2024-04-23T16:26:03.699399+0000","last_undegraded":"2024-04-23T16:26:03.699399+0000","last_fullsized":"2024-04-23T16:26:03.699399+0000","mapping_epoch":245,"log_start":"211'7000","ondisk_log_start":"211'7000","created":192,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14713","last_scrub_stamp":"2024-04-23T16:25:33.363009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:33.363009+0000","objects_scrubbed":6244,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:26:58.427475+0000","scrub_duration":1212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251920384,"num_objects":3844,"num_object_clones":0,"num_object_copies":7688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17070,"num_write_kb":669248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8007,"num_bytes_recovered":524746752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"258'17109","reported_seq":17490,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-23T16:26:03.117248+0000","last_change":"2024-04-23T16:25:49.363034+0000","last_active":"2024-04-23T16:26:03.117248+0000","last_peered":"2024-04-23T16:26:03.117248+0000","last_clean":"2024-04-23T16:26:03.117248+0000","last_became_active":"2024-04-23T16:22:11.616683+0000","last_became_peered":"2024-04-23T16:22:11.616683+0000","last_unstale":"2024-04-23T16:26:03.117248+0000","last_undegraded":"2024-04-23T16:26:03.117248+0000","last_fullsized":"2024-04-23T16:26:03.117248+0000","mapping_epoch":192,"log_start":"211'7100","ondisk_log_start":"211'7100","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'16005","last_scrub_stamp":"2024-04-23T16:25:49.362861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:49.362861+0000","objects_scrubbed":4958,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:13.160535+0000","scrub_duration":988,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250675200,"num_objects":3825,"num_object_clones":0,"num_object_copies":7650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17109,"num_write_kb":669888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"258'19070","reported_seq":35332,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-23T16:26:06.332452+0000","last_change":"2024-04-23T16:25:33.698766+0000","last_active":"2024-04-23T16:26:06.332452+0000","last_peered":"2024-04-23T16:26:06.332452+0000","last_clean":"2024-04-23T16:26:06.332452+0000","last_became_active":"2024-04-23T16:25:07.676317+0000","last_became_peered":"2024-04-23T16:25:07.676317+0000","last_unstale":"2024-04-23T16:26:06.332452+0000","last_undegraded":"2024-04-23T16:26:06.332452+0000","last_fullsized":"2024-04-23T16:26:06.332452+0000","mapping_epoch":244,"log_start":"220'9000","ondisk_log_start":"220'9000","created":192,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'16525","last_scrub_stamp":"2024-04-23T16:25:33.698520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:33.698520+0000","objects_scrubbed":6094,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:01.572295+0000","scrub_duration":1066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230359040,"num_objects":3515,"num_object_clones":0,"num_object_copies":7030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17197,"num_write_kb":662784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7913,"num_bytes_recovered":518586368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3","version":"258'17288","reported_seq":17613,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-23T16:26:04.121072+0000","last_change":"2024-04-23T16:24:41.533231+0000","last_active":"2024-04-23T16:26:04.121072+0000","last_peered":"2024-04-23T16:26:04.121072+0000","last_clean":"2024-04-23T16:26:04.121072+0000","last_became_active":"2024-04-23T16:22:11.618441+0000","last_became_peered":"2024-04-23T16:22:11.618441+0000","last_unstale":"2024-04-23T16:26:04.121072+0000","last_undegraded":"2024-04-23T16:26:04.121072+0000","last_fullsized":"2024-04-23T16:26:04.121072+0000","mapping_epoch":192,"log_start":"212'7200","ondisk_log_start":"212'7200","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'12982","last_scrub_stamp":"2024-04-23T16:24:41.533078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:24:41.533078+0000","objects_scrubbed":8010,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:26:07.894592+0000","scrub_duration":734,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242745344,"num_objects":3704,"num_object_clones":0,"num_object_copies":7408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":671744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"258'17030","reported_seq":17425,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-23T16:26:03.628930+0000","last_change":"2024-04-23T16:25:36.467689+0000","last_active":"2024-04-23T16:26:03.628930+0000","last_peered":"2024-04-23T16:26:03.628930+0000","last_clean":"2024-04-23T16:26:03.628930+0000","last_became_active":"2024-04-23T16:22:11.616598+0000","last_became_peered":"2024-04-23T16:22:11.616598+0000","last_unstale":"2024-04-23T16:26:03.628930+0000","last_undegraded":"2024-04-23T16:26:03.628930+0000","last_fullsized":"2024-04-23T16:26:03.628930+0000","mapping_epoch":192,"log_start":"211'7000","ondisk_log_start":"211'7000","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14966","last_scrub_stamp":"2024-04-23T16:25:36.467397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:36.467397+0000","objects_scrubbed":6018,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:03.230886+0000","scrub_duration":1341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256376832,"num_objects":3912,"num_object_clones":0,"num_object_copies":7824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17030,"num_write_kb":670144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"258'19489","reported_seq":19909,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-23T16:26:06.264190+0000","last_change":"2024-04-23T16:25:50.467241+0000","last_active":"2024-04-23T16:26:06.264190+0000","last_peered":"2024-04-23T16:26:06.264190+0000","last_clean":"2024-04-23T16:26:06.264190+0000","last_became_active":"2024-04-23T16:22:11.664655+0000","last_became_peered":"2024-04-23T16:22:11.664655+0000","last_unstale":"2024-04-23T16:26:06.264190+0000","last_undegraded":"2024-04-23T16:26:06.264190+0000","last_fullsized":"2024-04-23T16:26:06.264190+0000","mapping_epoch":192,"log_start":"220'9400","ondisk_log_start":"220'9400","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'18246","last_scrub_stamp":"2024-04-23T16:25:50.467059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:50.467059+0000","objects_scrubbed":4956,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:26:50.497653+0000","scrub_duration":1052,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240975872,"num_objects":3677,"num_object_clones":0,"num_object_copies":7354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17545,"num_write_kb":679104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"258'17471","reported_seq":17792,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-23T16:26:05.840442+0000","last_change":"2024-04-23T16:25:34.355976+0000","last_active":"2024-04-23T16:26:05.840442+0000","last_peered":"2024-04-23T16:26:05.840442+0000","last_clean":"2024-04-23T16:26:05.840442+0000","last_became_active":"2024-04-23T16:22:11.664765+0000","last_became_peered":"2024-04-23T16:22:11.664765+0000","last_unstale":"2024-04-23T16:26:05.840442+0000","last_undegraded":"2024-04-23T16:26:05.840442+0000","last_fullsized":"2024-04-23T16:26:05.840442+0000","mapping_epoch":192,"log_start":"213'7400","ondisk_log_start":"213'7400","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'15004","last_scrub_stamp":"2024-04-23T16:25:34.355861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:22:10.601504+0000","last_clean_scrub_stamp":"2024-04-23T16:25:34.355861+0000","objects_scrubbed":6218,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:26:54.699437+0000","scrub_duration":1137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243335168,"num_objects":3713,"num_object_clones":0,"num_object_copies":7426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17471,"num_write_kb":677888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":717,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-23T16:26:02.704723+0000","last_change":"2024-04-23T16:25:52.560187+0000","last_active":"2024-04-23T16:26:02.704723+0000","last_peered":"2024-04-23T16:26:02.704723+0000","last_clean":"2024-04-23T16:26:02.704723+0000","last_became_active":"2024-04-23T16:15:36.918552+0000","last_became_peered":"2024-04-23T16:15:36.918552+0000","last_unstale":"2024-04-23T16:26:02.704723+0000","last_undegraded":"2024-04-23T16:26:02.704723+0000","last_fullsized":"2024-04-23T16:26:02.704723+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:25:52.560053+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:03.175518+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":3941859352,"num_objects":60149,"num_object_clones":0,"num_object_copies":120298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60149,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":277135,"num_write_kb":10793025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24054,"num_bytes_recovered":1576402944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9507446784,"data_stored":9507438640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161000,"ondisk_log_size":161000,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056769,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1932796,"kb_used_data":1662836,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":91812356,"statfs":{"total":95995035648,"available":94015852544,"internally_reserved":0,"allocated":1702744064,"data_stored":1702392393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":276429252},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351111,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612749,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":658872,"kb_used_data":501356,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93086280,"statfs":{"total":95995035648,"available":95320350720,"internally_reserved":0,"allocated":513388544,"data_stored":513040445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":161282496},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841680,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":382440,"kb_used_data":261024,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93362712,"statfs":{"total":95995035648,"available":95603417088,"internally_reserved":0,"allocated":267288576,"data_stored":266946481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168726,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":593220,"kb_used_data":475960,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93151932,"statfs":{"total":95995035648,"available":95387578368,"internally_reserved":0,"allocated":487383040,"data_stored":487027317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":120060362},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463068,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319864,"kb_used_data":248428,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93425288,"statfs":{"total":95995035648,"available":95667494912,"internally_reserved":0,"allocated":254390272,"data_stored":254042173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738553,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1147320,"kb_used_data":993964,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":92597832,"statfs":{"total":95995035648,"available":94820179968,"internally_reserved":0,"allocated":1017819136,"data_stored":1017471037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313390,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":130424,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93614728,"statfs":{"total":95995035648,"available":95861481472,"internally_reserved":0,"allocated":700416,"data_stored":352317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117162,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1433528,"kb_used_data":1203820,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":92311624,"statfs":{"total":95995035648,"available":94527102976,"internally_reserved":0,"allocated":1232711680,"data_stored":1232363581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":235207111},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920932,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1689208,"kb_used_data":1469228,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92055944,"statfs":{"total":95995035648,"available":94265286656,"internally_reserved":0,"allocated":1504489472,"data_stored":1504141373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757407,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1205612,"kb_used_data":993956,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92539540,"statfs":{"total":95995035648,"available":94760488960,"internally_reserved":0,"allocated":1017810944,"data_stored":1017464777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1009451008,"data_stored":1009451008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1619918848,"data_stored":1619918848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1208549376,"data_stored":1208549376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1495859200,"data_stored":1495859200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1004015616,"data_stored":1004011544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":248184832,"data_stored":248184832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":485429248,"data_stored":485425176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":262799360,"data_stored":262799360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":505348096,"data_stored":505348096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1667891200,"data_stored":1667891200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:26:08.892 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-23T16:26:08.892 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-04-23T16:26:08.919 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:26:08.945 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:26:08.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:08.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:26:09.057 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:26:09.086 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:09.090 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:08 smithi188 ceph-mon[44101]: from='client.15378 v1:172.21.15.103:0/279213850' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:26:09.114 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:26:09.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:09.117 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:26:09.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:08 smithi148 ceph-mon[44520]: from='client.15378 v1:172.21.15.103:0/279213850' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:26:09.218 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:09.250 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:09.280 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:09.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:09.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:09.383 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:26:09.413 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:09.434 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:26:09.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:09.437 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:26:09.538 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:26:09.564 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:09.583 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:26:09.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:09.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:26:09.686 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:26:09.738 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:09.769 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:26:09.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:09.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:26:09.876 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:09.918 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:09.920 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:09 smithi103 ceph-mon[39629]: pgmap v755: 17 pgs: 17 active+clean; 3.6 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.34k op/s 2024-04-23T16:26:10.076 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:10.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:10.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:09 smithi148 ceph-mon[44520]: pgmap v755: 17 pgs: 17 active+clean; 3.6 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.34k op/s 2024-04-23T16:26:10.181 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:26:10.221 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:10.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:09 smithi188 ceph-mon[44101]: pgmap v755: 17 pgs: 17 active+clean; 3.6 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.34k op/s 2024-04-23T16:26:10.252 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:26:10.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:10.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:26:10.356 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:26:10.407 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:10.430 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:26:10.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:10.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:26:10.541 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:26:10.589 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:10.620 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:26:10.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:10.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:26:10.729 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:26:10.767 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:11.044 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:26:12.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:11 smithi103 ceph-mon[39629]: pgmap v756: 17 pgs: 17 active+clean; 3.5 GiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:26:12.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:11 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:26:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:11 smithi148 ceph-mon[44520]: pgmap v756: 17 pgs: 17 active+clean; 3.5 GiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:26:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:11 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:26:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:11 smithi188 ceph-mon[44101]: pgmap v756: 17 pgs: 17 active+clean; 3.5 GiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:26:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:11 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:26:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:26:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:26:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:13 smithi148 ceph-mon[44520]: pgmap v757: 17 pgs: 17 active+clean; 3.3 GiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:26:13.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:26:13.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:26:13.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:13 smithi188 ceph-mon[44101]: pgmap v757: 17 pgs: 17 active+clean; 3.3 GiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:26:14.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:26:14.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:26:14.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:13 smithi103 ceph-mon[39629]: pgmap v757: 17 pgs: 17 active+clean; 3.3 GiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:26:14.826 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:26:14.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:14.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:26:14.935 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:26:14.964 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:14 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:26:14.964 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:14 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:26:14.964 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:14 smithi103 ceph-mon[39629]: osdmap e259: 12 total, 11 up, 10 in 2024-04-23T16:26:14.964 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:26:14.964 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:26:14.964 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:26:14.968 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:14.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:14 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:26:14.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:14 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:26:14.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:14 smithi188 ceph-mon[44101]: osdmap e259: 12 total, 11 up, 10 in 2024-04-23T16:26:14.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:26:14.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:26:14.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:26:15.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:14 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:26:15.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:14 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:26:15.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:14 smithi148 ceph-mon[44520]: osdmap e259: 12 total, 11 up, 10 in 2024-04-23T16:26:15.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:26:15.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:26:15.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:26:15.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:15 smithi188 ceph-mon[44101]: pgmap v759: 17 pgs: 17 active+clean; 3.1 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:26:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:15 smithi103 ceph-mon[39629]: pgmap v759: 17 pgs: 17 active+clean; 3.1 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:26:16.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:15 smithi148 ceph-mon[44520]: pgmap v759: 17 pgs: 17 active+clean; 3.1 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:26:16.124 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:26:16.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:16.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:26:16.229 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:26:16.277 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:16.312 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:26:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:16.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:26:16.418 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:26:16.454 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:16.479 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:26:16.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:16.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:26:16.587 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:26:16.625 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:16.643 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:26:16.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:16.650 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:26:16.751 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:26:16.787 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:16.811 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:26:16.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:16.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:26:16.920 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:26:16.955 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:16.983 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:26:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:16.987 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:26:17.088 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:17.125 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:17.374 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:17.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:17.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:17.478 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:26:17.510 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:17.531 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:26:17.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:17.534 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:26:17.632 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:17 smithi103 ceph-mon[39629]: pgmap v760: 17 pgs: 17 active+clean; 2.9 GiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-23T16:26:17.635 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:26:17.669 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:17.691 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:26:17.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:17.694 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:26:17.767 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:26:17.789 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:26:17.795 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:26:17.812 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:26:17.826 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:17 smithi188 ceph-mon[44101]: pgmap v760: 17 pgs: 17 active+clean; 2.9 GiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-23T16:26:17.828 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:17.841 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:26:17.851 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:26:17.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:17.856 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:26:17.866 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:26:17.896 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:26:17.924 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:26:17.948 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:26:17.956 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:26:17.973 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:26:17.999 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:26:18.013 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:17 smithi148 ceph-mon[44520]: pgmap v760: 17 pgs: 17 active+clean; 2.9 GiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-23T16:26:18.017 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:18.026 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:26:18.026 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:26:18.027 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T16:26:18.027 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:26:18.027 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:26:18.027 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:26:18.027 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:26:18.027 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:26:18.027 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:26:18.027 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:26:18.027 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:26:18.027 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:26:18.027 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:26:18.057 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:26:18.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:18.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:26:18.162 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:26:18.200 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:18.229 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:26:18.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:18.235 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:26:18.336 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:26:18.379 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:18.417 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:26:18.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:18.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:26:18.524 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:26:18.553 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:18.575 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:26:18.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:18.578 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:26:18.641 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:18 smithi148 ceph-mon[44520]: from='client.15384 v1:172.21.15.103:0/3025024103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:26:18.641 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:18 smithi148 ceph-mon[44520]: from='client.15384 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:26:18.642 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:18 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1239911188' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:26:18.642 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.642 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.642 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.642 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.642 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.642 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.642 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.642 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.642 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.656 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:26:18.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:18 smithi103 ceph-mon[39629]: from='client.15384 v1:172.21.15.103:0/3025024103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:26:18.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:18 smithi103 ceph-mon[39629]: from='client.15384 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:26:18.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:18 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1239911188' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:26:18.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.679 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.679 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.679 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.679 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.679 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.679 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.679 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.679 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.679 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:26:18.709 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:18.927 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:26:18.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:18.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:26:18.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:18 smithi188 ceph-mon[44101]: from='client.15384 v1:172.21.15.103:0/3025024103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:26:18.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:18 smithi188 ceph-mon[44101]: from='client.15384 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:26:18.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:18 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1239911188' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:26:18.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:18.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:19.033 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:26:19.070 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:19.173 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:26:19.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:19.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:26:19.277 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:26:19.332 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:19.362 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:26:19.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:19.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:26:19.465 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:26:19.506 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:19.535 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:26:19.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:19.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:26:19.648 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:26:19.680 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:19.714 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:26:19.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:19.717 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:26:19.818 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:26:19.856 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:19.880 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:26:19.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:19.885 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:26:19.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:19.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:19.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:19 smithi188 ceph-mon[44101]: pgmap v761: 17 pgs: 17 active+clean; 2.8 GiB data, 7.5 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-23T16:26:19.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:19 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:26:19.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:19 smithi188 ceph-mon[44101]: from='client.15384 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:26:19.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:19 smithi188 ceph-mon[44101]: osdmap e260: 12 total, 11 up, 10 in 2024-04-23T16:26:19.986 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:26:20.023 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:20.059 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:26:20.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:20.065 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:26:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:19 smithi103 ceph-mon[39629]: pgmap v761: 17 pgs: 17 active+clean; 2.8 GiB data, 7.5 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-23T16:26:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:19 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:26:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:19 smithi103 ceph-mon[39629]: from='client.15384 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:26:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:19 smithi103 ceph-mon[39629]: osdmap e260: 12 total, 11 up, 10 in 2024-04-23T16:26:20.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:20.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:20.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:19 smithi148 ceph-mon[44520]: pgmap v761: 17 pgs: 17 active+clean; 2.8 GiB data, 7.5 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-23T16:26:20.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:19 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:26:20.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:19 smithi148 ceph-mon[44520]: from='client.15384 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:26:20.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:19 smithi148 ceph-mon[44520]: osdmap e260: 12 total, 11 up, 10 in 2024-04-23T16:26:20.154 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:26:20.155 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:26:20.166 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:26:20.205 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:20.239 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:26:20.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:20.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:26:20.345 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:20.382 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:20.586 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:20.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:20.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:20.600 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:26:20.601 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:26:20.692 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:26:20.726 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:20.750 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:26:20.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:20.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:26:20.857 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:26:20.892 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:21.349 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-23T16:26:21.354 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:26:21.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:21.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:26:21.459 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:26:21.486 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:21.518 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:26:21.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:21.522 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:26:21.622 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:26:21.647 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:21 smithi148 ceph-mon[44520]: pgmap v763: 17 pgs: 17 active+clean; 2.7 GiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-23T16:26:21.652 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:21.677 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:26:21.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:21.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:26:21.728 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:26:21.783 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:26:21.816 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:21 smithi188 ceph-mon[44101]: pgmap v763: 17 pgs: 17 active+clean; 2.7 GiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-23T16:26:21.817 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:21.843 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:26:21.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:21.848 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:26:21.856 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:26:21.948 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:26:21.978 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:21.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:21 smithi103 ceph-mon[39629]: pgmap v763: 17 pgs: 17 active+clean; 2.7 GiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-23T16:26:22.002 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:26:22.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:22.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:26:22.106 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:26:22.134 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:22.154 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:26:22.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:22.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:26:22.180 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:26:22.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:22.298 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:22.497 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:22.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:22.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:22.601 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:26:22.602 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:26:22.653 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:23.190 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:26:23.214 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:26:23.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:23.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:26:23.320 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:26:23.350 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:23.383 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:26:23.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:23.386 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:26:23.487 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:26:23.516 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:23.542 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:26:23.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:23.549 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:26:23.650 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:26:23.687 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:23.715 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:26:23.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:23.719 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:26:23.820 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:23.844 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:26:23.844 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:26:23.860 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:23.869 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:23 smithi148 ceph-mon[44520]: pgmap v764: 17 pgs: 17 active+clean; 2.5 GiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:26:23.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:23 smithi188 ceph-mon[44101]: pgmap v764: 17 pgs: 17 active+clean; 2.5 GiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:26:24.089 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:23 smithi103 ceph-mon[39629]: pgmap v764: 17 pgs: 17 active+clean; 2.5 GiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:26:24.119 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:24.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:24.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:24.226 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:26:24.264 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:24.373 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:26:24.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:24.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:26:24.477 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:26:24.538 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:24.570 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:26:24.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:24.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:26:24.616 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-23T16:26:24.684 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:24.736 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:24.760 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:24.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:24.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:24.863 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:26:24.919 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:24.951 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:26:24.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:24.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:26:25.003 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:26:25.058 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:26:25.073 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:26:25.092 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:24 smithi103 ceph-mon[39629]: pgmap v765: 17 pgs: 17 active+clean; 2.3 GiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-23T16:26:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2423184448' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:26:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:24 smithi103 ceph-mon[39629]: from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:26:25.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:24 smithi148 ceph-mon[44520]: pgmap v765: 17 pgs: 17 active+clean; 2.3 GiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-23T16:26:25.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:24 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2423184448' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:26:25.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:24 smithi148 ceph-mon[44520]: from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:26:25.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:24 smithi188 ceph-mon[44101]: pgmap v765: 17 pgs: 17 active+clean; 2.3 GiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-23T16:26:25.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2423184448' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:26:25.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:24 smithi188 ceph-mon[44101]: from='client.25276 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:26:25.558 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:26:25.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:25.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:26:25.663 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:26:25.694 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:26.004 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:25 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:26:26.004 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:25 smithi148 ceph-mon[44520]: from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:26:26.004 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:25 smithi148 ceph-mon[44520]: osdmap e261: 12 total, 11 up, 10 in 2024-04-23T16:26:26.147 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:26:26.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:26.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:26:26.253 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:26:26.298 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:26.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:25 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:26:26.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:25 smithi103 ceph-mon[39629]: from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:26:26.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:25 smithi103 ceph-mon[39629]: osdmap e261: 12 total, 11 up, 10 in 2024-04-23T16:26:26.365 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:26:26.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:26.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:26:26.470 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:26:26.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:25 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:26:26.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:25 smithi188 ceph-mon[44101]: from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:26:26.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:25 smithi188 ceph-mon[44101]: osdmap e261: 12 total, 11 up, 10 in 2024-04-23T16:26:26.501 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:26.958 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:26:26.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:26.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:26:27.064 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:26:27.073 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:26:27.073 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:26:27.100 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:27.128 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:26:27.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:27.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:26:27.235 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:27.253 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:27 smithi103 ceph-mon[39629]: pgmap v767: 17 pgs: 17 active+clean; 2.2 GiB data, 6.1 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.36k op/s 2024-04-23T16:26:27.275 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:27.303 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:27.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:27.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:27.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:27 smithi148 ceph-mon[44520]: pgmap v767: 17 pgs: 17 active+clean; 2.2 GiB data, 6.1 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.36k op/s 2024-04-23T16:26:27.414 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:26:27.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:27 smithi188 ceph-mon[44101]: pgmap v767: 17 pgs: 17 active+clean; 2.2 GiB data, 6.1 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.36k op/s 2024-04-23T16:26:27.448 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:27.473 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:26:27.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:27.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:26:27.580 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:26:27.612 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:26:27.617 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:27.634 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:26:27.640 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:26:27.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:27.644 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:26:27.658 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:26:27.683 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:26:27.705 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:26:27.733 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:26:27.745 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:26:27.761 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:26:27.790 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:26:27.797 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:27.817 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:26:27.835 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-23T16:26:27.841 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:26:27.845 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:26:27.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:27.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:26:27.863 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:26:27.864 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:26:27.864 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T16:26:27.864 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:26:27.864 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:26:27.864 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:26:27.864 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:26:27.864 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:26:27.864 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:26:27.864 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:26:27.864 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:26:27.865 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:26:27.865 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:26:27.948 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:26:27.987 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:28.020 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:26:28.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:28.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:26:28.126 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:26:28.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:28 smithi148 ceph-mon[44520]: 3.b scrub starts 2024-04-23T16:26:28.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:28 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:26:28.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:28 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:26:28.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:28 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2174625492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:26:28.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:28.154 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:28.185 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:26:28.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:28.195 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:26:28.250 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:28 smithi103 ceph-mon[39629]: 3.b scrub starts 2024-04-23T16:26:28.250 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:28 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:26:28.251 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:28 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:26:28.251 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:28 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2174625492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:26:28.251 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:28.251 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:26:28.295 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:26:28.342 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:28.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:28 smithi188 ceph-mon[44101]: 3.b scrub starts 2024-04-23T16:26:28.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:28 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:26:28.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:28 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:26:28.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:28 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2174625492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:26:28.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:28.555 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:26:28.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:28.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:26:28.666 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:28.700 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:28.727 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:28.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:28.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:28.772 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:26:28.772 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:26:28.837 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:28.870 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:28.895 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:28.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:28.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:28.972 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:26:29.000 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:26:29.035 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:29.073 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:26:29.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:29.077 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:26:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:29 smithi148 ceph-mon[44520]: 3.b scrub ok 2024-04-23T16:26:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:29 smithi148 ceph-mon[44520]: osdmap e262: 12 total, 11 up, 10 in 2024-04-23T16:26:29.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:29 smithi148 ceph-mon[44520]: pgmap v769: 17 pgs: 17 active+clean; 2.1 GiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-23T16:26:29.178 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:26:29.223 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:29.254 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:26:29.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:29.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:26:29.316 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:26:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:29 smithi103 ceph-mon[39629]: 3.b scrub ok 2024-04-23T16:26:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:29 smithi103 ceph-mon[39629]: osdmap e262: 12 total, 11 up, 10 in 2024-04-23T16:26:29.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:29 smithi103 ceph-mon[39629]: pgmap v769: 17 pgs: 17 active+clean; 2.1 GiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-23T16:26:29.366 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:29.397 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:29.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:29 smithi188 ceph-mon[44101]: 3.b scrub ok 2024-04-23T16:26:29.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:29.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:29 smithi188 ceph-mon[44101]: osdmap e262: 12 total, 11 up, 10 in 2024-04-23T16:26:29.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:29 smithi188 ceph-mon[44101]: pgmap v769: 17 pgs: 17 active+clean; 2.1 GiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-23T16:26:29.652 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:29.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:29.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:29.759 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:26:29.796 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:29.904 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:26:29.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:29.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:26:30.009 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:26:30.041 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:30.065 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:26:30.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:30.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:26:30.170 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:26:30.203 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:30.224 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:26:30.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:30.227 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:26:30.329 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:26:30.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:30 smithi103 ceph-mon[39629]: 3.3 scrub starts 2024-04-23T16:26:30.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:30 smithi103 ceph-mon[39629]: 3.3 scrub ok 2024-04-23T16:26:30.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:30 smithi148 ceph-mon[44520]: 3.3 scrub starts 2024-04-23T16:26:30.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:30 smithi148 ceph-mon[44520]: 3.3 scrub ok 2024-04-23T16:26:30.387 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:30.421 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:26:30.424 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:26:30.424 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2024-04-23T16:26:30.424 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2024-04-23T16:26:30.424 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:26:30.424 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.424 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:63252 3.3 3:c2527cae:::benchmark_data_smithi188_89276_object63251:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-23T16:26:30.424 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:22:43.692052+0000", 2024-04-23T16:26:30.424 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 226.72533832799999, 2024-04-23T16:26:30.424 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.088123881000000001, 2024-04-23T16:26:30.424 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.424 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.425 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.425 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.425 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.425 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.425 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 63252 2024-04-23T16:26:30.425 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.425 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.425 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.425 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.425 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.692052+0000", 2024-04-23T16:26:30.425 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.425 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.425 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.425 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.426 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.692052+0000", 2024-04-23T16:26:30.426 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.426 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.426 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.426 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.426 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.692054+0000", 2024-04-23T16:26:30.426 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7889999999999999e-06 2024-04-23T16:26:30.426 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.426 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.426 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.426 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.692104+0000", 2024-04-23T16:26:30.426 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0235999999999998e-05 2024-04-23T16:26:30.426 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.426 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.427 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.427 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.692114+0000", 2024-04-23T16:26:30.427 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8660000000000001e-06 2024-04-23T16:26:30.427 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.427 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.427 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.427 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.692120+0000", 2024-04-23T16:26:30.427 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7479999999999999e-06 2024-04-23T16:26:30.427 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.427 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.427 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.427 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.692172+0000", 2024-04-23T16:26:30.427 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1801000000000001e-05 2024-04-23T16:26:30.427 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.428 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.428 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.428 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.692274+0000", 2024-04-23T16:26:30.428 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000102398 2024-04-23T16:26:30.428 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.428 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.428 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.428 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.692363+0000", 2024-04-23T16:26:30.428 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8979000000000003e-05 2024-04-23T16:26:30.428 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.428 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.428 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.428 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.694529+0000", 2024-04-23T16:26:30.428 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021654389999999999 2024-04-23T16:26:30.429 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.429 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.429 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.429 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.780076+0000", 2024-04-23T16:26:30.429 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.085547559999999995 2024-04-23T16:26:30.429 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.429 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.429 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.429 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.780134+0000", 2024-04-23T16:26:30.429 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8007999999999998e-05 2024-04-23T16:26:30.429 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.429 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.429 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.429 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:22:43.780176+0000", 2024-04-23T16:26:30.430 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2057000000000001e-05 2024-04-23T16:26:30.430 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.430 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.430 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.430 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.430 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.430 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:237645 3.3 3:cd9a8c2e:::benchmark_data_smithi188_89276_object69001:head [delete] snapc 0=[] ondisk+write+known_if_redirected e251)", 2024-04-23T16:26:30.430 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:25:32.903651+0000", 2024-04-23T16:26:30.430 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 57.513739364999999, 2024-04-23T16:26:30.430 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.086462013000000004, 2024-04-23T16:26:30.430 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.430 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.430 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.431 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.431 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.431 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.431 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 237645 2024-04-23T16:26:30.431 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.431 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.431 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.431 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.431 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.903651+0000", 2024-04-23T16:26:30.431 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.432 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.432 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.432 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.432 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.903651+0000", 2024-04-23T16:26:30.432 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.432 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.432 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.432 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.432 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.903653+0000", 2024-04-23T16:26:30.432 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.984e-06 2024-04-23T16:26:30.432 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.432 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.432 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.432 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.903659+0000", 2024-04-23T16:26:30.433 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3489999999999999e-06 2024-04-23T16:26:30.433 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.433 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.433 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.433 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.903670+0000", 2024-04-23T16:26:30.433 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1739999999999999e-05 2024-04-23T16:26:30.433 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.433 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.433 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.433 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.903676+0000", 2024-04-23T16:26:30.433 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8069999999999996e-06 2024-04-23T16:26:30.433 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.433 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.433 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.434 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.903740+0000", 2024-04-23T16:26:30.434 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3837000000000006e-05 2024-04-23T16:26:30.434 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.434 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.434 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.434 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.903861+0000", 2024-04-23T16:26:30.434 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000121218 2024-04-23T16:26:30.434 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.434 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.434 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.434 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.903945+0000", 2024-04-23T16:26:30.434 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3763000000000004e-05 2024-04-23T16:26:30.434 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.435 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.435 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.435 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.912765+0000", 2024-04-23T16:26:30.435 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0088197689999999999 2024-04-23T16:26:30.435 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.435 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.435 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.435 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.990031+0000", 2024-04-23T16:26:30.435 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.077266218999999997 2024-04-23T16:26:30.435 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.435 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.435 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.435 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.990067+0000", 2024-04-23T16:26:30.435 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6384000000000003e-05 2024-04-23T16:26:30.435 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.436 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.436 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.436 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.990113+0000", 2024-04-23T16:26:30.436 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5942999999999997e-05 2024-04-23T16:26:30.436 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.436 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.436 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.436 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.436 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.436 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:237650 3.3 3:c52d960a:::benchmark_data_smithi188_89276_object69006:head [delete] snapc 0=[] ondisk+write+known_if_redirected e251)", 2024-04-23T16:26:30.436 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:25:32.903689+0000", 2024-04-23T16:26:30.436 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 57.513701932000004, 2024-04-23T16:26:30.436 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.10695505499999999, 2024-04-23T16:26:30.436 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.437 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.437 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.437 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.437 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.437 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.437 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 237650 2024-04-23T16:26:30.437 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.437 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.437 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.437 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.437 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.903689+0000", 2024-04-23T16:26:30.437 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.437 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.438 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.438 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.438 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.903689+0000", 2024-04-23T16:26:30.438 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.438 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.438 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.438 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.438 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.903689+0000", 2024-04-23T16:26:30.438 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9599999999999999e-07 2024-04-23T16:26:30.438 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.438 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.438 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.439 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.903692+0000", 2024-04-23T16:26:30.439 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9550000000000001e-06 2024-04-23T16:26:30.439 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.439 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.439 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.439 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.903700+0000", 2024-04-23T16:26:30.439 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1629999999999996e-06 2024-04-23T16:26:30.439 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.439 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.439 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.439 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.903705+0000", 2024-04-23T16:26:30.439 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1359999999999999e-06 2024-04-23T16:26:30.439 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.439 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.439 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.440 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.907938+0000", 2024-04-23T16:26:30.440 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0042336229999999997 2024-04-23T16:26:30.440 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.440 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.440 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.440 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.908063+0000", 2024-04-23T16:26:30.440 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000125024 2024-04-23T16:26:30.440 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.440 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.440 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.440 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.908184+0000", 2024-04-23T16:26:30.440 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000120918 2024-04-23T16:26:30.440 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.440 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.441 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.441 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.912865+0000", 2024-04-23T16:26:30.441 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0046806679999999998 2024-04-23T16:26:30.441 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.441 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.441 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.441 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:33.010545+0000", 2024-04-23T16:26:30.441 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.097680120999999995 2024-04-23T16:26:30.441 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.441 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.441 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.441 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:33.010600+0000", 2024-04-23T16:26:30.441 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5316999999999999e-05 2024-04-23T16:26:30.441 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.442 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.442 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.442 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:33.010644+0000", 2024-04-23T16:26:30.442 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3433999999999998e-05 2024-04-23T16:26:30.442 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.442 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.442 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.442 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.442 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.442 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:237662 3.3 3:cf9d6208:::benchmark_data_smithi188_89276_object69018:head [delete] snapc 0=[] ondisk+write+known_if_redirected e251)", 2024-04-23T16:26:30.442 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:25:32.915721+0000", 2024-04-23T16:26:30.442 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 57.501669122999999, 2024-04-23T16:26:30.442 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.095043907999999996, 2024-04-23T16:26:30.442 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.443 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.443 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.443 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.443 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.443 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.443 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 237662 2024-04-23T16:26:30.443 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.443 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.443 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.443 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.443 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.915721+0000", 2024-04-23T16:26:30.443 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.443 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.443 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.444 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.444 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.915721+0000", 2024-04-23T16:26:30.444 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.444 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.444 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.444 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.444 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.915724+0000", 2024-04-23T16:26:30.444 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4660000000000002e-06 2024-04-23T16:26:30.444 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.444 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.444 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.444 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.915730+0000", 2024-04-23T16:26:30.444 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.297e-06 2024-04-23T16:26:30.445 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.445 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.445 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.445 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.915745+0000", 2024-04-23T16:26:30.445 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4919000000000001e-05 2024-04-23T16:26:30.445 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.445 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.445 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.445 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.915753+0000", 2024-04-23T16:26:30.445 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5780000000000001e-06 2024-04-23T16:26:30.445 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.445 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.445 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.445 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.915804+0000", 2024-04-23T16:26:30.445 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1208000000000002e-05 2024-04-23T16:26:30.446 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.446 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.446 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.446 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.915911+0000", 2024-04-23T16:26:30.446 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107229 2024-04-23T16:26:30.446 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.446 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.446 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.446 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.916013+0000", 2024-04-23T16:26:30.446 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000101683 2024-04-23T16:26:30.446 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.446 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.446 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.446 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:32.923706+0000", 2024-04-23T16:26:30.447 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0076930210000000004 2024-04-23T16:26:30.447 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.447 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.447 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.447 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:33.010686+0000", 2024-04-23T16:26:30.447 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.086979719999999996 2024-04-23T16:26:30.447 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.447 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.447 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.447 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:33.010726+0000", 2024-04-23T16:26:30.447 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0824000000000001e-05 2024-04-23T16:26:30.447 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.447 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.447 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.447 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:33.010765+0000", 2024-04-23T16:26:30.448 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8962999999999999e-05 2024-04-23T16:26:30.448 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.448 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.448 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.448 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.448 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.448 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:240260 3.3 3:c6cc028d:::benchmark_data_smithi188_89276_object71616:head [delete] snapc 0=[] ondisk+write+known_if_redirected e251)", 2024-04-23T16:26:30.448 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:25:35.149292+0000", 2024-04-23T16:26:30.448 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 55.268099065999998, 2024-04-23T16:26:30.448 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.083697234999999995, 2024-04-23T16:26:30.448 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.448 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.448 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.448 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.449 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.449 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.449 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 240260 2024-04-23T16:26:30.449 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.449 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.449 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.449 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.449 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.149292+0000", 2024-04-23T16:26:30.449 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.449 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.449 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.449 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.449 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.149292+0000", 2024-04-23T16:26:30.449 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.449 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.450 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.450 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.450 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.149294+0000", 2024-04-23T16:26:30.450 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7580000000000001e-06 2024-04-23T16:26:30.450 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.450 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.450 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.450 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.149300+0000", 2024-04-23T16:26:30.450 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8309999999999997e-06 2024-04-23T16:26:30.450 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.450 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.450 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.450 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.149311+0000", 2024-04-23T16:26:30.450 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0729e-05 2024-04-23T16:26:30.451 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.451 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.451 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.451 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.149317+0000", 2024-04-23T16:26:30.451 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9240000000000004e-06 2024-04-23T16:26:30.451 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.451 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.451 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.451 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.149367+0000", 2024-04-23T16:26:30.451 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9910999999999997e-05 2024-04-23T16:26:30.451 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.451 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.451 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.451 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.149455+0000", 2024-04-23T16:26:30.452 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8251999999999995e-05 2024-04-23T16:26:30.452 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.452 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.452 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.452 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.149523+0000", 2024-04-23T16:26:30.452 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8034999999999995e-05 2024-04-23T16:26:30.452 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.452 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.452 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.452 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.153087+0000", 2024-04-23T16:26:30.452 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0035641100000000001 2024-04-23T16:26:30.452 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.452 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.452 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.452 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.232920+0000", 2024-04-23T16:26:30.453 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.079832944000000003 2024-04-23T16:26:30.453 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.453 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.453 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.453 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.232954+0000", 2024-04-23T16:26:30.453 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4301999999999998e-05 2024-04-23T16:26:30.453 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.453 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.453 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.453 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.232989+0000", 2024-04-23T16:26:30.453 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4439e-05 2024-04-23T16:26:30.453 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.453 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.453 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.454 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.454 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.454 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:240271 3.3 3:cf16f9f0:::benchmark_data_smithi188_89276_object71627:head [delete] snapc 0=[] ondisk+write+known_if_redirected e251)", 2024-04-23T16:26:30.454 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:25:35.159457+0000", 2024-04-23T16:26:30.454 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 55.257934065999997, 2024-04-23T16:26:30.454 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.085425784000000005, 2024-04-23T16:26:30.454 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.454 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.454 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.454 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.454 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.454 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.454 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 240271 2024-04-23T16:26:30.454 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.455 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.455 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.455 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.455 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.159457+0000", 2024-04-23T16:26:30.455 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.455 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.455 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.455 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.455 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.159457+0000", 2024-04-23T16:26:30.455 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.455 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.455 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.455 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.455 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.159458+0000", 2024-04-23T16:26:30.456 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.649e-06 2024-04-23T16:26:30.456 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.456 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.456 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.456 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.159463+0000", 2024-04-23T16:26:30.456 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5560000000000001e-06 2024-04-23T16:26:30.456 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.456 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.456 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.456 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.159471+0000", 2024-04-23T16:26:30.456 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2630000000000003e-06 2024-04-23T16:26:30.456 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.456 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.457 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.457 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.159476+0000", 2024-04-23T16:26:30.457 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4419999999999997e-06 2024-04-23T16:26:30.457 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.457 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.457 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.457 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.163855+0000", 2024-04-23T16:26:30.457 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0043786110000000001 2024-04-23T16:26:30.457 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.457 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.457 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.457 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.163972+0000", 2024-04-23T16:26:30.457 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000117296 2024-04-23T16:26:30.457 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.458 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.458 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.458 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.164091+0000", 2024-04-23T16:26:30.458 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000118237 2024-04-23T16:26:30.458 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.458 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.458 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.458 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.175842+0000", 2024-04-23T16:26:30.458 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011750941000000001 2024-04-23T16:26:30.458 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.458 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.458 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.458 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.244778+0000", 2024-04-23T16:26:30.458 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.068936171000000004 2024-04-23T16:26:30.459 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.459 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.459 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.459 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.244839+0000", 2024-04-23T16:26:30.459 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0970000000000001e-05 2024-04-23T16:26:30.459 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.459 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.459 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.459 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.244882+0000", 2024-04-23T16:26:30.460 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3648e-05 2024-04-23T16:26:30.460 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.460 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.460 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.460 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.460 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.460 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:240274 3.3 3:c0dabf4c:::benchmark_data_smithi188_89276_object71630:head [delete] snapc 0=[] ondisk+write+known_if_redirected e251)", 2024-04-23T16:26:30.460 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:25:35.161025+0000", 2024-04-23T16:26:30.460 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 55.256365559999999, 2024-04-23T16:26:30.460 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.084001892999999994, 2024-04-23T16:26:30.460 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.460 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.460 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.461 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.461 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.461 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.461 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 240274 2024-04-23T16:26:30.461 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.461 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.461 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.461 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.461 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.161025+0000", 2024-04-23T16:26:30.461 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.461 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.461 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.461 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.461 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.161025+0000", 2024-04-23T16:26:30.462 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.462 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.462 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.462 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.462 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.161027+0000", 2024-04-23T16:26:30.462 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6810000000000001e-06 2024-04-23T16:26:30.462 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.462 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.462 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.462 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.161032+0000", 2024-04-23T16:26:30.462 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0459999999999996e-06 2024-04-23T16:26:30.462 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.462 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.463 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.463 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.161042+0000", 2024-04-23T16:26:30.463 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0071000000000001e-05 2024-04-23T16:26:30.463 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.463 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.463 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.463 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.161054+0000", 2024-04-23T16:26:30.463 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2359e-05 2024-04-23T16:26:30.463 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.463 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.463 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.463 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.168502+0000", 2024-04-23T16:26:30.463 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.007448156 2024-04-23T16:26:30.463 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.464 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.464 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.464 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.168613+0000", 2024-04-23T16:26:30.464 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011067500000000001 2024-04-23T16:26:30.464 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.464 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.464 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.464 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.168721+0000", 2024-04-23T16:26:30.464 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000108125 2024-04-23T16:26:30.464 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.464 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.464 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.464 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.175847+0000", 2024-04-23T16:26:30.464 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.007125363 2024-04-23T16:26:30.465 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.465 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.465 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.465 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.244934+0000", 2024-04-23T16:26:30.465 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.069087503999999994 2024-04-23T16:26:30.465 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.465 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.465 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.465 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.244987+0000", 2024-04-23T16:26:30.465 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3029000000000003e-05 2024-04-23T16:26:30.465 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.465 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.465 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.465 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.245027+0000", 2024-04-23T16:26:30.466 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9884e-05 2024-04-23T16:26:30.466 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.466 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.466 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.466 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.466 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.466 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:240275 3.3 3:c8389f62:::benchmark_data_smithi188_89276_object71631:head [delete] snapc 0=[] ondisk+write+known_if_redirected e251)", 2024-04-23T16:26:30.466 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:25:35.161070+0000", 2024-04-23T16:26:30.466 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 55.256320412000001, 2024-04-23T16:26:30.466 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.084078508999999996, 2024-04-23T16:26:30.466 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.466 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.467 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.467 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.467 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.467 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.467 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 240275 2024-04-23T16:26:30.467 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.467 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.467 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.467 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.467 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.161070+0000", 2024-04-23T16:26:30.467 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.467 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.467 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.467 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.467 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.161070+0000", 2024-04-23T16:26:30.468 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.468 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.468 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.468 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.468 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.161071+0000", 2024-04-23T16:26:30.468 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5799999999999998e-07 2024-04-23T16:26:30.468 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.468 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.468 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.468 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.161074+0000", 2024-04-23T16:26:30.468 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5130000000000001e-06 2024-04-23T16:26:30.468 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.468 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.469 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.469 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.161080+0000", 2024-04-23T16:26:30.469 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7150000000000003e-06 2024-04-23T16:26:30.469 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.469 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.469 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.469 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.161085+0000", 2024-04-23T16:26:30.469 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.186e-06 2024-04-23T16:26:30.469 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.469 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.469 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.469 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.172274+0000", 2024-04-23T16:26:30.469 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011188758 2024-04-23T16:26:30.469 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.469 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.470 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.470 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.172395+0000", 2024-04-23T16:26:30.470 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012112099999999999 2024-04-23T16:26:30.470 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.470 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.470 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.470 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.172526+0000", 2024-04-23T16:26:30.470 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000130662 2024-04-23T16:26:30.470 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.470 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.470 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.470 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.175983+0000", 2024-04-23T16:26:30.470 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0034568250000000002 2024-04-23T16:26:30.471 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.471 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.471 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.471 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.245070+0000", 2024-04-23T16:26:30.471 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.069087470999999998 2024-04-23T16:26:30.471 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.471 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.471 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.471 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.245114+0000", 2024-04-23T16:26:30.471 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3927000000000001e-05 2024-04-23T16:26:30.471 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.471 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.471 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.471 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.245149+0000", 2024-04-23T16:26:30.472 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4573000000000003e-05 2024-04-23T16:26:30.472 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.472 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.472 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.472 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.472 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.472 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:240549 3.3 3:ccc106cf:::benchmark_data_smithi188_89276_object71905:head [delete] snapc 0=[] ondisk+write+known_if_redirected e251)", 2024-04-23T16:26:30.472 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:25:35.410799+0000", 2024-04-23T16:26:30.472 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 55.006591594, 2024-04-23T16:26:30.472 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.086864980999999994, 2024-04-23T16:26:30.472 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.472 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.472 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.472 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.473 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.473 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.473 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 240549 2024-04-23T16:26:30.473 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.473 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.473 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.473 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.473 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.410799+0000", 2024-04-23T16:26:30.473 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.473 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.473 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.473 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.473 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.410799+0000", 2024-04-23T16:26:30.473 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.474 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.474 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.474 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.474 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.410802+0000", 2024-04-23T16:26:30.474 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.57e-06 2024-04-23T16:26:30.474 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.474 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.474 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.474 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.410807+0000", 2024-04-23T16:26:30.474 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7780000000000004e-06 2024-04-23T16:26:30.474 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.474 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.474 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.474 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.410819+0000", 2024-04-23T16:26:30.475 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1668000000000001e-05 2024-04-23T16:26:30.475 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.475 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.475 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.475 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.410827+0000", 2024-04-23T16:26:30.475 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8800000000000008e-06 2024-04-23T16:26:30.475 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.475 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.475 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.475 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.410877+0000", 2024-04-23T16:26:30.475 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0553000000000002e-05 2024-04-23T16:26:30.475 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.475 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.475 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.476 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.410974+0000", 2024-04-23T16:26:30.476 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6448999999999998e-05 2024-04-23T16:26:30.476 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.476 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.476 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.476 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.411061+0000", 2024-04-23T16:26:30.476 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6705000000000004e-05 2024-04-23T16:26:30.476 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.476 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.476 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.476 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.419531+0000", 2024-04-23T16:26:30.476 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0084707500000000008 2024-04-23T16:26:30.476 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.476 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.477 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.477 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.497558+0000", 2024-04-23T16:26:30.477 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.078027014000000006 2024-04-23T16:26:30.477 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.477 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.477 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.477 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.497613+0000", 2024-04-23T16:26:30.477 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4979000000000003e-05 2024-04-23T16:26:30.477 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.477 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.477 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.477 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.497664+0000", 2024-04-23T16:26:30.477 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0634999999999999e-05 2024-04-23T16:26:30.477 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.478 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.478 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.478 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.478 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.478 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:240552 3.3 3:c8d15742:::benchmark_data_smithi188_89276_object71908:head [delete] snapc 0=[] ondisk+write+known_if_redirected e251)", 2024-04-23T16:26:30.478 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:25:35.413896+0000", 2024-04-23T16:26:30.478 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 55.003494494999998, 2024-04-23T16:26:30.478 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.091366769, 2024-04-23T16:26:30.478 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.478 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.478 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.478 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.478 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.478 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.479 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 240552 2024-04-23T16:26:30.479 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.479 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.479 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.479 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.479 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.413896+0000", 2024-04-23T16:26:30.479 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.479 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.479 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.479 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.479 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.413896+0000", 2024-04-23T16:26:30.479 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.479 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.479 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.480 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.480 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.413898+0000", 2024-04-23T16:26:30.480 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.894e-06 2024-04-23T16:26:30.480 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.480 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.480 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.480 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.413903+0000", 2024-04-23T16:26:30.480 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3550000000000004e-06 2024-04-23T16:26:30.480 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.480 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.480 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.480 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.413912+0000", 2024-04-23T16:26:30.480 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0899999999999994e-06 2024-04-23T16:26:30.480 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.481 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.481 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.481 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.413918+0000", 2024-04-23T16:26:30.481 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0249999999999999e-06 2024-04-23T16:26:30.481 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.481 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.481 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.481 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.415699+0000", 2024-04-23T16:26:30.481 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001780958 2024-04-23T16:26:30.481 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.481 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.481 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.481 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.415781+0000", 2024-04-23T16:26:30.481 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1687999999999994e-05 2024-04-23T16:26:30.481 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.482 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.482 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.482 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.415886+0000", 2024-04-23T16:26:30.482 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000104603 2024-04-23T16:26:30.482 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.482 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.482 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.482 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.419621+0000", 2024-04-23T16:26:30.482 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003735019 2024-04-23T16:26:30.482 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.482 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.482 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.482 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.505159+0000", 2024-04-23T16:26:30.483 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.085538397000000002 2024-04-23T16:26:30.483 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.483 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.483 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.483 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.505216+0000", 2024-04-23T16:26:30.483 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6761000000000001e-05 2024-04-23T16:26:30.483 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.483 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.483 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.483 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:35.505263+0000", 2024-04-23T16:26:30.483 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6978999999999999e-05 2024-04-23T16:26:30.483 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.483 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.484 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.484 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.484 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.484 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:246017 3.3 3:ceccb78b:::benchmark_data_smithi188_89276_object77373:head [delete] snapc 0=[] ondisk+write+known_if_redirected e251)", 2024-04-23T16:26:30.484 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:25:39.978946+0000", 2024-04-23T16:26:30.484 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 50.438444795000002, 2024-04-23T16:26:30.484 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.088423976000000001, 2024-04-23T16:26:30.484 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.484 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.484 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.484 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.484 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.484 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.484 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 246017 2024-04-23T16:26:30.484 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.485 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.485 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.485 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.485 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:39.978946+0000", 2024-04-23T16:26:30.485 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.485 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.485 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.485 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.485 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:39.978946+0000", 2024-04-23T16:26:30.485 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.485 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.485 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.485 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.485 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:39.978947+0000", 2024-04-23T16:26:30.486 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6300000000000001e-06 2024-04-23T16:26:30.486 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.486 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.486 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.486 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:39.978952+0000", 2024-04-23T16:26:30.486 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6659999999999999e-06 2024-04-23T16:26:30.486 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.486 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.486 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.486 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:39.978962+0000", 2024-04-23T16:26:30.486 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6399999999999992e-06 2024-04-23T16:26:30.486 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.486 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.487 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.487 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:39.978968+0000", 2024-04-23T16:26:30.487 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.037e-06 2024-04-23T16:26:30.487 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.487 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.487 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.487 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:39.979010+0000", 2024-04-23T16:26:30.487 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2623000000000002e-05 2024-04-23T16:26:30.487 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.487 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.487 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.487 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:40.055181+0000", 2024-04-23T16:26:30.487 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.076170337000000005 2024-04-23T16:26:30.487 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.487 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.488 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.488 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:40.055300+0000", 2024-04-23T16:26:30.488 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000119695 2024-04-23T16:26:30.488 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.488 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.488 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.488 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:40.067260+0000", 2024-04-23T16:26:30.488 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011959141 2024-04-23T16:26:30.488 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.488 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.488 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.488 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:40.067274+0000", 2024-04-23T16:26:30.488 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4462e-05 2024-04-23T16:26:30.488 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:40.067323+0000", 2024-04-23T16:26:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9187999999999997e-05 2024-04-23T16:26:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:40.067370+0000", 2024-04-23T16:26:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6557000000000002e-05 2024-04-23T16:26:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.489 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:283055 3.3 3:c80795a4:::benchmark_data_smithi188_89276_object114411:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:26:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:09.381356+0000", 2024-04-23T16:26:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.036034402999999, 2024-04-23T16:26:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.22286794500000001, 2024-04-23T16:26:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 283055 2024-04-23T16:26:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.490 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.491 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.491 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381356+0000", 2024-04-23T16:26:30.491 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.491 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.491 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.491 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.491 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381356+0000", 2024-04-23T16:26:30.491 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.491 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.491 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.491 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.491 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381358+0000", 2024-04-23T16:26:30.491 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9300000000000002e-06 2024-04-23T16:26:30.491 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.491 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.492 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.492 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381364+0000", 2024-04-23T16:26:30.492 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2099999999999998e-06 2024-04-23T16:26:30.492 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.492 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.492 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.492 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381375+0000", 2024-04-23T16:26:30.492 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1060000000000001e-05 2024-04-23T16:26:30.492 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.492 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.492 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.492 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381382+0000", 2024-04-23T16:26:30.492 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5860000000000001e-06 2024-04-23T16:26:30.493 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.493 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.493 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.493 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381435+0000", 2024-04-23T16:26:30.493 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2787000000000001e-05 2024-04-23T16:26:30.493 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.493 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.493 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.493 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381576+0000", 2024-04-23T16:26:30.493 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000141256 2024-04-23T16:26:30.493 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.493 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.493 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.493 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381695+0000", 2024-04-23T16:26:30.493 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000119413 2024-04-23T16:26:30.494 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.494 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.494 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.494 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.396183+0000", 2024-04-23T16:26:30.494 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014487451 2024-04-23T16:26:30.494 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.494 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.494 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.494 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.604137+0000", 2024-04-23T16:26:30.494 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20795429000000001 2024-04-23T16:26:30.494 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.494 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.494 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.494 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.604181+0000", 2024-04-23T16:26:30.495 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3563000000000003e-05 2024-04-23T16:26:30.495 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.495 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.495 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.495 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.604224+0000", 2024-04-23T16:26:30.495 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3399000000000002e-05 2024-04-23T16:26:30.495 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.495 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.495 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.495 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.495 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.495 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:283057 3.3 3:cdfb5812:::benchmark_data_smithi188_89276_object114413:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:26:30.495 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:09.381405+0000", 2024-04-23T16:26:30.495 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.035985944, 2024-04-23T16:26:30.496 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.22294652200000001, 2024-04-23T16:26:30.496 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.496 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.496 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.496 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.496 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.496 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.496 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 283057 2024-04-23T16:26:30.496 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.496 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.496 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.496 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.496 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381405+0000", 2024-04-23T16:26:30.496 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.496 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.497 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.497 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.497 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381405+0000", 2024-04-23T16:26:30.497 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.497 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.497 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.497 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.497 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381405+0000", 2024-04-23T16:26:30.497 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9100000000000004e-07 2024-04-23T16:26:30.497 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.497 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.497 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.497 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381408+0000", 2024-04-23T16:26:30.497 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.695e-06 2024-04-23T16:26:30.498 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.498 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.498 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.498 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381414+0000", 2024-04-23T16:26:30.498 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8189999999999997e-06 2024-04-23T16:26:30.498 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.498 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.498 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.498 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381418+0000", 2024-04-23T16:26:30.498 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7680000000000002e-06 2024-04-23T16:26:30.498 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.498 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.498 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.499 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.386409+0000", 2024-04-23T16:26:30.499 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0049914310000000002 2024-04-23T16:26:30.499 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.499 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.499 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.499 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.386516+0000", 2024-04-23T16:26:30.499 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010747300000000001 2024-04-23T16:26:30.499 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.499 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.499 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.499 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.386620+0000", 2024-04-23T16:26:30.499 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010332600000000001 2024-04-23T16:26:30.499 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.499 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.500 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.500 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.396194+0000", 2024-04-23T16:26:30.500 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.009573781 2024-04-23T16:26:30.500 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.500 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.500 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.500 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.604283+0000", 2024-04-23T16:26:30.500 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20808908300000001 2024-04-23T16:26:30.500 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.500 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.500 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.500 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.604314+0000", 2024-04-23T16:26:30.500 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0997000000000002e-05 2024-04-23T16:26:30.500 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.501 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.501 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.501 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.604351+0000", 2024-04-23T16:26:30.501 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7558000000000002e-05 2024-04-23T16:26:30.501 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.501 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.501 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.501 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.501 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.501 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:283059 3.3 3:cd8d2e79:::benchmark_data_smithi188_89276_object114415:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:26:30.501 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:09.381492+0000", 2024-04-23T16:26:30.501 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.035899061999999, 2024-04-23T16:26:30.501 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.22299651700000001, 2024-04-23T16:26:30.501 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.501 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.502 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.502 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.502 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.502 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.502 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 283059 2024-04-23T16:26:30.502 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.502 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.502 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.502 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.502 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381492+0000", 2024-04-23T16:26:30.502 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.502 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.502 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.502 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.503 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381492+0000", 2024-04-23T16:26:30.503 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.503 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.503 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.503 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.503 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381493+0000", 2024-04-23T16:26:30.503 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.136e-06 2024-04-23T16:26:30.503 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.503 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.503 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.503 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381497+0000", 2024-04-23T16:26:30.503 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9199999999999997e-06 2024-04-23T16:26:30.503 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.504 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.504 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.504 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381505+0000", 2024-04-23T16:26:30.504 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.011e-06 2024-04-23T16:26:30.504 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.504 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.504 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.504 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381513+0000", 2024-04-23T16:26:30.504 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4810000000000006e-06 2024-04-23T16:26:30.504 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.504 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.504 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.504 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.391290+0000", 2024-04-23T16:26:30.504 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0097767040000000006 2024-04-23T16:26:30.504 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.505 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.505 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.505 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.391406+0000", 2024-04-23T16:26:30.505 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000116353 2024-04-23T16:26:30.505 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.505 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.505 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.505 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.391528+0000", 2024-04-23T16:26:30.505 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012218400000000001 2024-04-23T16:26:30.505 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.505 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.505 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.505 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.396347+0000", 2024-04-23T16:26:30.505 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0048190960000000001 2024-04-23T16:26:30.506 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.506 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.506 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.506 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.604392+0000", 2024-04-23T16:26:30.506 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20804426300000001 2024-04-23T16:26:30.506 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.506 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.506 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.506 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.604441+0000", 2024-04-23T16:26:30.506 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8931000000000003e-05 2024-04-23T16:26:30.506 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.506 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.506 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.506 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.604488+0000", 2024-04-23T16:26:30.507 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7438000000000002e-05 2024-04-23T16:26:30.507 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.507 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.507 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.507 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.507 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.507 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:283081 3.3 3:ca861c37:::benchmark_data_smithi188_89276_object114437:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:26:30.507 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:09.405748+0000", 2024-04-23T16:26:30.507 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.011642621, 2024-04-23T16:26:30.507 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20958642199999999, 2024-04-23T16:26:30.507 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.507 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.507 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.508 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.508 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.508 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.508 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 283081 2024-04-23T16:26:30.508 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.508 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.508 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.508 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.508 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405748+0000", 2024-04-23T16:26:30.508 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.508 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.508 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.508 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.508 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405748+0000", 2024-04-23T16:26:30.509 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.509 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.509 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.509 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.509 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405750+0000", 2024-04-23T16:26:30.509 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2259999999999999e-06 2024-04-23T16:26:30.509 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.509 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.509 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.509 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405756+0000", 2024-04-23T16:26:30.509 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0519999999999998e-06 2024-04-23T16:26:30.509 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.509 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.509 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.510 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405768+0000", 2024-04-23T16:26:30.510 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.188e-05 2024-04-23T16:26:30.510 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.510 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.510 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.510 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405776+0000", 2024-04-23T16:26:30.510 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0250000000000002e-06 2024-04-23T16:26:30.510 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.510 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.510 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.510 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405837+0000", 2024-04-23T16:26:30.510 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1134000000000001e-05 2024-04-23T16:26:30.510 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.510 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.511 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.511 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405938+0000", 2024-04-23T16:26:30.511 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000100917 2024-04-23T16:26:30.511 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.511 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.511 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.511 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.406015+0000", 2024-04-23T16:26:30.511 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6728000000000001e-05 2024-04-23T16:26:30.511 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.511 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.511 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.511 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.413731+0000", 2024-04-23T16:26:30.511 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0077160579999999996 2024-04-23T16:26:30.511 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.511 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.512 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.512 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.615248+0000", 2024-04-23T16:26:30.512 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.201516636 2024-04-23T16:26:30.512 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.512 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.512 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.512 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.615286+0000", 2024-04-23T16:26:30.512 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8609999999999998e-05 2024-04-23T16:26:30.512 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.512 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.512 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.512 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.615334+0000", 2024-04-23T16:26:30.512 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8155999999999998e-05 2024-04-23T16:26:30.512 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.512 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.513 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.513 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.513 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.513 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:283082 3.3 3:c2076386:::benchmark_data_smithi188_89276_object114438:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:26:30.513 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:09.405795+0000", 2024-04-23T16:26:30.513 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.011595674999999, 2024-04-23T16:26:30.513 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.214633718, 2024-04-23T16:26:30.513 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.513 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.513 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.513 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.513 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.513 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.513 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 283082 2024-04-23T16:26:30.513 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.514 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.514 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.514 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.514 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405795+0000", 2024-04-23T16:26:30.514 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.514 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.514 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.514 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.514 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405795+0000", 2024-04-23T16:26:30.514 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.514 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.514 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.514 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.514 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405796+0000", 2024-04-23T16:26:30.515 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6899999999999996e-07 2024-04-23T16:26:30.515 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.515 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.515 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.515 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405798+0000", 2024-04-23T16:26:30.515 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7099999999999999e-06 2024-04-23T16:26:30.515 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.515 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.515 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.515 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405811+0000", 2024-04-23T16:26:30.515 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2969000000000001e-05 2024-04-23T16:26:30.515 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.515 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.515 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.515 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405817+0000", 2024-04-23T16:26:30.516 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6230000000000002e-06 2024-04-23T16:26:30.516 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.516 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.516 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.516 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.409560+0000", 2024-04-23T16:26:30.516 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0037428079999999998 2024-04-23T16:26:30.516 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.516 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.516 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.516 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.409651+0000", 2024-04-23T16:26:30.516 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1077999999999998e-05 2024-04-23T16:26:30.516 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.516 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.516 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.517 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.409728+0000", 2024-04-23T16:26:30.517 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6837999999999999e-05 2024-04-23T16:26:30.517 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.517 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.517 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.517 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.421862+0000", 2024-04-23T16:26:30.517 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01213412 2024-04-23T16:26:30.517 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.517 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.517 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.517 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.620338+0000", 2024-04-23T16:26:30.517 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19847571999999999 2024-04-23T16:26:30.517 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.517 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.517 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.517 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.620388+0000", 2024-04-23T16:26:30.518 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0874000000000001e-05 2024-04-23T16:26:30.518 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.518 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.518 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.518 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.620429+0000", 2024-04-23T16:26:30.518 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0209000000000001e-05 2024-04-23T16:26:30.518 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.518 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.518 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.518 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.518 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.518 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:283085 3.3 3:c5fb6a0c:::benchmark_data_smithi188_89276_object114441:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:26:30.518 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:09.409895+0000", 2024-04-23T16:26:30.518 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.00749549, 2024-04-23T16:26:30.519 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.21995653700000001, 2024-04-23T16:26:30.519 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.519 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.519 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.519 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.519 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.519 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.519 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 283085 2024-04-23T16:26:30.519 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.519 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.519 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.519 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.519 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.409895+0000", 2024-04-23T16:26:30.519 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.520 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.520 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.520 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.520 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.409895+0000", 2024-04-23T16:26:30.520 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.520 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.520 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.520 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.520 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.409897+0000", 2024-04-23T16:26:30.520 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.666e-06 2024-04-23T16:26:30.520 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.520 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.520 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.520 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.409902+0000", 2024-04-23T16:26:30.521 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4909999999999996e-06 2024-04-23T16:26:30.521 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.521 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.521 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.521 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.409913+0000", 2024-04-23T16:26:30.521 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0317e-05 2024-04-23T16:26:30.521 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.521 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.521 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.521 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.409918+0000", 2024-04-23T16:26:30.521 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5369999999999997e-06 2024-04-23T16:26:30.521 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.521 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.521 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.522 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.413761+0000", 2024-04-23T16:26:30.522 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0038424539999999999 2024-04-23T16:26:30.522 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.522 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.522 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.522 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.413880+0000", 2024-04-23T16:26:30.522 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011968099999999999 2024-04-23T16:26:30.522 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.522 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.522 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.522 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.413993+0000", 2024-04-23T16:26:30.522 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000112883 2024-04-23T16:26:30.522 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.522 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.523 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.523 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.421867+0000", 2024-04-23T16:26:30.523 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0078734059999999995 2024-04-23T16:26:30.523 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.523 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.523 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.523 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.629780+0000", 2024-04-23T16:26:30.523 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20791299999999999 2024-04-23T16:26:30.523 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.523 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.523 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.523 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.629811+0000", 2024-04-23T16:26:30.523 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1387999999999998e-05 2024-04-23T16:26:30.523 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.524 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.524 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.524 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.629852+0000", 2024-04-23T16:26:30.524 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0713999999999997e-05 2024-04-23T16:26:30.524 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.524 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.524 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.524 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.524 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.524 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:283092 3.3 3:c6d6d9fb:::benchmark_data_smithi188_89276_object114448:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:26:30.524 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:09.416225+0000", 2024-04-23T16:26:30.524 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.00116585, 2024-04-23T16:26:30.524 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.21378214000000001, 2024-04-23T16:26:30.524 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.525 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.525 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.525 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.525 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.525 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.525 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 283092 2024-04-23T16:26:30.525 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.525 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.525 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.525 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.525 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.416225+0000", 2024-04-23T16:26:30.525 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.525 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.525 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.525 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.526 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.416225+0000", 2024-04-23T16:26:30.526 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.526 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.526 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.526 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.526 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.416227+0000", 2024-04-23T16:26:30.526 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4540000000000001e-06 2024-04-23T16:26:30.526 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.526 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.526 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.526 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.416233+0000", 2024-04-23T16:26:30.526 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7250000000000002e-06 2024-04-23T16:26:30.526 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.527 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.527 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.527 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.416243+0000", 2024-04-23T16:26:30.527 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0395e-05 2024-04-23T16:26:30.527 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.527 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.527 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.527 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.416249+0000", 2024-04-23T16:26:30.527 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7470000000000002e-06 2024-04-23T16:26:30.527 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.527 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.527 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.527 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.418349+0000", 2024-04-23T16:26:30.527 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0020998269999999999 2024-04-23T16:26:30.527 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.528 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.528 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.528 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.418427+0000", 2024-04-23T16:26:30.528 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7626999999999994e-05 2024-04-23T16:26:30.528 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.528 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.528 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.528 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.418503+0000", 2024-04-23T16:26:30.528 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6625999999999994e-05 2024-04-23T16:26:30.528 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.528 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.528 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.528 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.422039+0000", 2024-04-23T16:26:30.528 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0035362990000000001 2024-04-23T16:26:30.528 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.529 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.529 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.529 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.629907+0000", 2024-04-23T16:26:30.529 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20786766100000001 2024-04-23T16:26:30.529 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.529 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.529 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.529 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.629954+0000", 2024-04-23T16:26:30.529 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7329e-05 2024-04-23T16:26:30.529 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.529 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.529 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.529 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.630007+0000", 2024-04-23T16:26:30.529 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2450000000000001e-05 2024-04-23T16:26:30.529 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.530 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.530 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.530 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.530 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.530 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:283111 3.3 3:ccaa2632:::benchmark_data_smithi188_89276_object114467:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:26:30.530 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:09.445340+0000", 2024-04-23T16:26:30.530 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 20.972050533000001, 2024-04-23T16:26:30.530 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.18482868599999999, 2024-04-23T16:26:30.530 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.530 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.530 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.530 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.530 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.530 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.531 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 283111 2024-04-23T16:26:30.531 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.531 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.531 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.531 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.531 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.445340+0000", 2024-04-23T16:26:30.531 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.531 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.531 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.531 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.531 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.445340+0000", 2024-04-23T16:26:30.531 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.531 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.531 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.532 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.532 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.445344+0000", 2024-04-23T16:26:30.532 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7730000000000001e-06 2024-04-23T16:26:30.532 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.532 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.532 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.532 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.445356+0000", 2024-04-23T16:26:30.532 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1943000000000001e-05 2024-04-23T16:26:30.532 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.532 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.532 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.532 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.445382+0000", 2024-04-23T16:26:30.532 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6063000000000001e-05 2024-04-23T16:26:30.532 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.532 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.533 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.533 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.445397+0000", 2024-04-23T16:26:30.533 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.486e-05 2024-04-23T16:26:30.533 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.533 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.533 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.533 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.445514+0000", 2024-04-23T16:26:30.533 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011703 2024-04-23T16:26:30.533 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.533 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.533 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.533 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.445600+0000", 2024-04-23T16:26:30.533 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6274999999999996e-05 2024-04-23T16:26:30.533 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.533 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.534 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.534 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.445677+0000", 2024-04-23T16:26:30.534 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6563999999999994e-05 2024-04-23T16:26:30.534 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.534 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.534 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.534 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.449211+0000", 2024-04-23T16:26:30.534 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0035343829999999999 2024-04-23T16:26:30.534 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.534 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.534 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.534 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.630075+0000", 2024-04-23T16:26:30.534 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.18086406799999999 2024-04-23T16:26:30.535 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.535 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.535 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.535 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.630125+0000", 2024-04-23T16:26:30.535 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0098e-05 2024-04-23T16:26:30.535 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.535 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.535 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.535 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.630169+0000", 2024-04-23T16:26:30.535 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3628999999999999e-05 2024-04-23T16:26:30.535 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.535 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.535 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.535 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.535 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.536 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:283115 3.3 3:c783fede:::benchmark_data_smithi188_89276_object114471:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:26:30.536 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:09.462085+0000", 2024-04-23T16:26:30.536 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 20.955305954, 2024-04-23T16:26:30.536 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.175558731, 2024-04-23T16:26:30.536 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:30.536 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:30.536 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:26:30.536 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:30.536 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:30.536 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:30.536 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 283115 2024-04-23T16:26:30.536 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.536 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:30.536 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.537 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:30.537 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.462085+0000", 2024-04-23T16:26:30.537 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.537 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.537 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.537 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:30.537 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.462085+0000", 2024-04-23T16:26:30.537 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:30.537 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.537 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.537 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:30.537 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.462086+0000", 2024-04-23T16:26:30.537 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.463e-06 2024-04-23T16:26:30.537 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.537 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.538 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:30.538 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.462091+0000", 2024-04-23T16:26:30.538 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5549999999999996e-06 2024-04-23T16:26:30.538 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.538 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.538 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:30.538 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.462100+0000", 2024-04-23T16:26:30.538 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.329e-06 2024-04-23T16:26:30.538 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.538 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.538 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:30.538 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.462122+0000", 2024-04-23T16:26:30.538 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2436999999999999e-05 2024-04-23T16:26:30.538 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.538 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.539 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:30.539 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.462196+0000", 2024-04-23T16:26:30.539 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3334e-05 2024-04-23T16:26:30.539 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.539 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.539 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:30.539 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.462273+0000", 2024-04-23T16:26:30.539 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7674000000000002e-05 2024-04-23T16:26:30.539 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.539 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.539 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:26:30.539 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.462343+0000", 2024-04-23T16:26:30.539 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9408999999999999e-05 2024-04-23T16:26:30.539 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.539 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.540 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:30.540 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.464641+0000", 2024-04-23T16:26:30.540 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002298394 2024-04-23T16:26:30.540 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.540 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.540 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:26:30.540 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.637545+0000", 2024-04-23T16:26:30.540 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.172903328 2024-04-23T16:26:30.540 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.540 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.540 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:26:30.540 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.637598+0000", 2024-04-23T16:26:30.540 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3278e-05 2024-04-23T16:26:30.540 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:30.540 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:30.541 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:26:30.541 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.637643+0000", 2024-04-23T16:26:30.541 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5529999999999999e-05 2024-04-23T16:26:30.541 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.541 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.541 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.541 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:30.541 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:30.541 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:26:30.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:30 smithi188 ceph-mon[44101]: 3.3 scrub starts 2024-04-23T16:26:30.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:30 smithi188 ceph-mon[44101]: 3.3 scrub ok 2024-04-23T16:26:30.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:30.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:26:30.644 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:26:30.683 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:30.773 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:26:31.061 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:31 smithi103 ceph-mon[39629]: pgmap v770: 17 pgs: 17 active+clean; 2.0 GiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:26:31.062 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:31 smithi148 ceph-mon[44520]: pgmap v770: 17 pgs: 17 active+clean; 2.0 GiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:26:31.122 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-23T16:26:31.153 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:26:31.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:31.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:26:31.257 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:31.305 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:31.456 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:31.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:31.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:31.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:31 smithi188 ceph-mon[44101]: pgmap v770: 17 pgs: 17 active+clean; 2.0 GiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:26:31.531 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:26:31.561 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:26:31.611 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:31.627 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:26:32.080 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:26:32.277 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:26:32.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:32.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:26:32.319 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:32 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:26:32.319 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:32 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:26:32.319 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:32 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:26:32.319 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:32 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:26:32.319 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:32 smithi148 ceph-mon[44520]: 3.7 scrub starts 2024-04-23T16:26:32.319 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:32 smithi148 ceph-mon[44520]: osdmap e263: 12 total, 11 up, 10 in 2024-04-23T16:26:32.319 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:32 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3582209471' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:26:32.319 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:32 smithi148 ceph-mon[44520]: from='client.25285 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:26:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:32 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:26:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:32 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:26:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:32 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:26:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:32 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:26:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:32 smithi103 ceph-mon[39629]: 3.7 scrub starts 2024-04-23T16:26:32.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:32 smithi103 ceph-mon[39629]: osdmap e263: 12 total, 11 up, 10 in 2024-04-23T16:26:32.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:32 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3582209471' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:26:32.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:32 smithi103 ceph-mon[39629]: from='client.25285 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:26:32.380 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:26:32.425 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:32.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:32 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:26:32.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:32 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:26:32.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:32 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:26:32.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:32 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:26:32.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:32 smithi188 ceph-mon[44101]: 3.7 scrub starts 2024-04-23T16:26:32.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:32 smithi188 ceph-mon[44101]: osdmap e263: 12 total, 11 up, 10 in 2024-04-23T16:26:32.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:32 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3582209471' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:26:32.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:32 smithi188 ceph-mon[44101]: from='client.25285 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:26:32.511 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:26:32.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:32.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:26:32.625 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:26:32.658 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:32.689 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:26:32.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:32.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:26:32.798 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:32.833 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:33.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:33 smithi103 ceph-mon[39629]: 3.7 scrub ok 2024-04-23T16:26:33.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:33 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:26:33.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:33 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:26:33.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:33 smithi103 ceph-mon[39629]: from='client.25285 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:26:33.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:33 smithi103 ceph-mon[39629]: osdmap e264: 12 total, 11 up, 10 in 2024-04-23T16:26:33.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:33 smithi103 ceph-mon[39629]: pgmap v773: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.8 GiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-23T16:26:33.354 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:33 smithi148 ceph-mon[44520]: 3.7 scrub ok 2024-04-23T16:26:33.354 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:33 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:26:33.354 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:33 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:26:33.354 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:33 smithi148 ceph-mon[44520]: from='client.25285 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:26:33.354 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:33 smithi148 ceph-mon[44520]: osdmap e264: 12 total, 11 up, 10 in 2024-04-23T16:26:33.354 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:33 smithi148 ceph-mon[44520]: pgmap v773: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.8 GiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-23T16:26:33.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:33 smithi188 ceph-mon[44101]: 3.7 scrub ok 2024-04-23T16:26:33.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:33 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:26:33.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:33 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:26:33.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:33 smithi188 ceph-mon[44101]: from='client.25285 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:26:33.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:33 smithi188 ceph-mon[44101]: osdmap e264: 12 total, 11 up, 10 in 2024-04-23T16:26:33.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:33 smithi188 ceph-mon[44101]: pgmap v773: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.8 GiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-23T16:26:33.688 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:33.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:33.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:33.793 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:33.837 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:33.939 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:26:33.946 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:33.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:33.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:33.971 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:26:33.997 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:26:34.031 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:26:34.031 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:26:34.031 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.031 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:313051 3.d 3:bef51a02:::benchmark_data_smithi188_89276_object144407:head [delete] snapc 0=[] ondisk+write+known_if_redirected e264)", 2024-04-23T16:26:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:34.013441+0000", 2024-04-23T16:26:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.016681215999999999, 2024-04-23T16:26:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016705200999999999, 2024-04-23T16:26:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:26:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 313051 2024-04-23T16:26:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.013441+0000", 2024-04-23T16:26:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.013441+0000", 2024-04-23T16:26:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.013471+0000", 2024-04-23T16:26:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9485e-05 2024-04-23T16:26:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.013476+0000", 2024-04-23T16:26:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5099999999999998e-06 2024-04-23T16:26:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.013486+0000", 2024-04-23T16:26:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9089999999999995e-06 2024-04-23T16:26:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.013494+0000", 2024-04-23T16:26:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2690000000000002e-06 2024-04-23T16:26:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.020233+0000", 2024-04-23T16:26:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0067397919999999997 2024-04-23T16:26:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.020345+0000", 2024-04-23T16:26:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000111799 2024-04-23T16:26:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:26:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.020468+0000", 2024-04-23T16:26:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000122454 2024-04-23T16:26:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:313054 3.d 3:bf90e3d6:::benchmark_data_smithi188_89276_object144410:head [delete] snapc 0=[] ondisk+write+known_if_redirected e264)", 2024-04-23T16:26:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:34.013911+0000", 2024-04-23T16:26:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.016211373000000001, 2024-04-23T16:26:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016289140000000001, 2024-04-23T16:26:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:26:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 313054 2024-04-23T16:26:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.013911+0000", 2024-04-23T16:26:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.013911+0000", 2024-04-23T16:26:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.013913+0000", 2024-04-23T16:26:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2890000000000001e-06 2024-04-23T16:26:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.013917+0000", 2024-04-23T16:26:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2250000000000002e-06 2024-04-23T16:26:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.013927+0000", 2024-04-23T16:26:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0679999999999999e-05 2024-04-23T16:26:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.013935+0000", 2024-04-23T16:26:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4270000000000002e-06 2024-04-23T16:26:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.024596+0000", 2024-04-23T16:26:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010660882 2024-04-23T16:26:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.024706+0000", 2024-04-23T16:26:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000110616 2024-04-23T16:26:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:26:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.024820+0000", 2024-04-23T16:26:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011369100000000001 2024-04-23T16:26:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:313060 3.5 3:abf4de79:::benchmark_data_smithi188_89276_object144416:head [delete] snapc 0=[] ondisk+write+known_if_redirected e264)", 2024-04-23T16:26:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:34.021752+0000", 2024-04-23T16:26:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.0083705719999999997, 2024-04-23T16:26:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0084950909999999998, 2024-04-23T16:26:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:26:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 313060 2024-04-23T16:26:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.021752+0000", 2024-04-23T16:26:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.021752+0000", 2024-04-23T16:26:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.021754+0000", 2024-04-23T16:26:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9069999999999999e-06 2024-04-23T16:26:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.021759+0000", 2024-04-23T16:26:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4020000000000001e-06 2024-04-23T16:26:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.021770+0000", 2024-04-23T16:26:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0757999999999999e-05 2024-04-23T16:26:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.021778+0000", 2024-04-23T16:26:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5660000000000001e-06 2024-04-23T16:26:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.024776+0000", 2024-04-23T16:26:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0029984859999999999 2024-04-23T16:26:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.026124+0000", 2024-04-23T16:26:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001347319 2024-04-23T16:26:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:26:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.026223+0000", 2024-04-23T16:26:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9429e-05 2024-04-23T16:26:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:313064 3.d 3:b7a5afe5:::benchmark_data_smithi188_89276_object144420:head [delete] snapc 0=[] ondisk+write+known_if_redirected e264)", 2024-04-23T16:26:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:34.025113+0000", 2024-04-23T16:26:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.0050093469999999999, 2024-04-23T16:26:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0051795189999999996, 2024-04-23T16:26:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:26:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 313064 2024-04-23T16:26:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.025113+0000", 2024-04-23T16:26:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.025113+0000", 2024-04-23T16:26:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.025115+0000", 2024-04-23T16:26:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9089999999999998e-06 2024-04-23T16:26:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.025120+0000", 2024-04-23T16:26:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9810000000000003e-06 2024-04-23T16:26:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.025131+0000", 2024-04-23T16:26:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0457e-05 2024-04-23T16:26:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.025138+0000", 2024-04-23T16:26:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8390000000000007e-06 2024-04-23T16:26:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.029731+0000", 2024-04-23T16:26:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045921010000000003 2024-04-23T16:26:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.029833+0000", 2024-04-23T16:26:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000102196 2024-04-23T16:26:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:26:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.029935+0000", 2024-04-23T16:26:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010212700000000001 2024-04-23T16:26:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:313067 3.5 3.696a12f5 (undecoded) ondisk+write+known_if_redirected e264)", 2024-04-23T16:26:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:34.025485+0000", 2024-04-23T16:26:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.0046373580000000003, 2024-04-23T16:26:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0048504389999999998, 2024-04-23T16:26:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "queued for pg", 2024-04-23T16:26:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 313067 2024-04-23T16:26:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.025485+0000", 2024-04-23T16:26:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.025485+0000", 2024-04-23T16:26:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.025486+0000", 2024-04-23T16:26:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.136e-06 2024-04-23T16:26:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.025490+0000", 2024-04-23T16:26:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6210000000000001e-06 2024-04-23T16:26:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.025499+0000", 2024-04-23T16:26:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3850000000000008e-06 2024-04-23T16:26:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.025506+0000", 2024-04-23T16:26:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1659999999999999e-06 2024-04-23T16:26:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:313044 3.d 3:b7ae24cb:::benchmark_data_smithi188_89276_object144400:head [delete] snapc 0=[] ondisk+write+known_if_redirected e264)", 2024-04-23T16:26:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:34.007507+0000", 2024-04-23T16:26:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.022615485000000001, 2024-04-23T16:26:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022863426999999999, 2024-04-23T16:26:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:26:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 313044 2024-04-23T16:26:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.007507+0000", 2024-04-23T16:26:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.007507+0000", 2024-04-23T16:26:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.007510+0000", 2024-04-23T16:26:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5450000000000002e-06 2024-04-23T16:26:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.007518+0000", 2024-04-23T16:26:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4300000000000006e-06 2024-04-23T16:26:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.007536+0000", 2024-04-23T16:26:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7524999999999999e-05 2024-04-23T16:26:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.007545+0000", 2024-04-23T16:26:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2029999999999998e-06 2024-04-23T16:26:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.007661+0000", 2024-04-23T16:26:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011568799999999999 2024-04-23T16:26:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.007791+0000", 2024-04-23T16:26:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013032300000000001 2024-04-23T16:26:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:26:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.007907+0000", 2024-04-23T16:26:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000115648 2024-04-23T16:26:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:34.024549+0000", 2024-04-23T16:26:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016642724000000001 2024-04-23T16:26:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:34.057 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:34.057 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:34.057 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:34.057 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-04-23T16:26:34.057 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 6 2024-04-23T16:26:34.057 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:26:34.057 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:34.058 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:26:34.060 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:26:34.092 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:26:34.092 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:34.127 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:26:34.156 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:26:34.183 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:26:34.206 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:26:34.230 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:26:34.230 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:26:34.230 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T16:26:34.231 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:26:34.231 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:26:34.231 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:26:34.231 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:26:34.231 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:26:34.231 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:26:34.231 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:26:34.231 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:26:34.231 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:26:34.231 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:26:34.246 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:34 smithi103 ceph-mon[39629]: 3.e scrub starts 2024-04-23T16:26:34.246 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:34 smithi103 ceph-mon[39629]: 3.c scrub starts 2024-04-23T16:26:34.246 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:34 smithi103 ceph-mon[39629]: 3.e scrub ok 2024-04-23T16:26:34.246 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:34 smithi103 ceph-mon[39629]: 3.c scrub ok 2024-04-23T16:26:34.246 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:34 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2605946094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:26:34.246 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:34 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:34 smithi148 ceph-mon[44520]: 3.e scrub starts 2024-04-23T16:26:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:34 smithi148 ceph-mon[44520]: 3.c scrub starts 2024-04-23T16:26:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:34 smithi148 ceph-mon[44520]: 3.e scrub ok 2024-04-23T16:26:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:34 smithi148 ceph-mon[44520]: 3.c scrub ok 2024-04-23T16:26:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:34 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2605946094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:26:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:34 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:34 smithi188 ceph-mon[44101]: 3.e scrub starts 2024-04-23T16:26:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:34 smithi188 ceph-mon[44101]: 3.c scrub starts 2024-04-23T16:26:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:34 smithi188 ceph-mon[44101]: 3.e scrub ok 2024-04-23T16:26:34.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:34 smithi188 ceph-mon[44101]: 3.c scrub ok 2024-04-23T16:26:34.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:34 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2605946094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:26:34.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:34 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:34.813 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-23T16:26:34.858 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:34.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:34.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:34.962 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:26:35.011 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:35.034 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:26:35.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:35.037 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:26:35.070 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.070 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.070 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.071 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.071 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.071 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.071 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.071 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.071 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:35 smithi148 ceph-mon[44520]: pgmap v774: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.6 GiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.37k op/s 2024-04-23T16:26:35.137 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:26:35.197 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:35.227 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:26:35.233 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:26:35.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:35.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:26:35.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:35 smithi103 ceph-mon[39629]: pgmap v774: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.6 GiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.37k op/s 2024-04-23T16:26:35.336 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:26:35.337 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:26:35.340 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:26:35.377 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:35 smithi188 ceph-mon[44101]: pgmap v774: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.6 GiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.37k op/s 2024-04-23T16:26:35.588 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:26:35.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:35.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:26:35.695 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:26:35.754 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:35.791 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:26:35.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:35.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:26:35.897 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:26:35.935 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:35.963 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:26:35.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:35.965 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:26:36.066 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:26:36.091 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:36.098 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:36.127 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:26:36.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:36.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:26:36.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:36.231 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:26:36.264 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:36.288 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:26:36.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:36.294 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:26:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:36.395 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:26:36.428 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:36.558 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:26:36.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:36.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:26:36.561 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:26:36.661 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:36.698 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:36.727 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:36.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:36.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:36.831 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:26:36.873 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:36.903 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:26:36.909 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:26:36.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:36.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:26:37.012 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:26:37.044 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:37.069 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:26:37.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:37.075 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:26:37.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:37 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:26:37.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:37 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:26:37.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:37 smithi148 ceph-mon[44520]: osdmap e265: 12 total, 11 up, 10 in 2024-04-23T16:26:37.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:37 smithi148 ceph-mon[44520]: pgmap v776: 17 pgs: 17 active+clean; 1.4 GiB data, 4.6 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.44k op/s 2024-04-23T16:26:37.176 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:26:37.204 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:37.229 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:26:37.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:37.232 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:26:37.293 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:26:37.293 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:26:37.333 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:26:37.337 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:26:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:37 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:26:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:37 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:26:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:37 smithi103 ceph-mon[39629]: osdmap e265: 12 total, 11 up, 10 in 2024-04-23T16:26:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:37 smithi103 ceph-mon[39629]: pgmap v776: 17 pgs: 17 active+clean; 1.4 GiB data, 4.6 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.44k op/s 2024-04-23T16:26:37.384 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:37.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:37 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:26:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:37 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:26:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:37 smithi188 ceph-mon[44101]: osdmap e265: 12 total, 11 up, 10 in 2024-04-23T16:26:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:37 smithi188 ceph-mon[44101]: pgmap v776: 17 pgs: 17 active+clean; 1.4 GiB data, 4.6 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.44k op/s 2024-04-23T16:26:37.549 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:26:37.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:37.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:26:37.655 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:26:37.690 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:38.065 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:26:38.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:38.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:26:38.079 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-23T16:26:38.170 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:26:38.198 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:38.221 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:26:38.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:38.225 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:26:38.326 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:26:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:38 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:26:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:38 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:26:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:38 smithi103 ceph-mon[39629]: osdmap e266: 12 total, 11 up, 10 in 2024-04-23T16:26:38.370 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:38 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:26:38.370 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:38 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:26:38.371 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:38 smithi188 ceph-mon[44101]: osdmap e266: 12 total, 11 up, 10 in 2024-04-23T16:26:38.372 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:38.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:38 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:26:38.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:38 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:26:38.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:38 smithi148 ceph-mon[44520]: osdmap e266: 12 total, 11 up, 10 in 2024-04-23T16:26:38.402 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:26:38.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:38.406 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:26:38.507 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:26:38.527 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:26:38.539 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:26:38.549 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:38.580 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:26:38.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:38.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:26:38.685 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:26:38.727 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:38.813 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:26:38.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:38.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:26:38.919 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:26:38.970 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:39.100 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:26:39.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:39.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:26:39.204 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:39.254 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:39.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:39 smithi148 ceph-mon[44520]: pgmap v778: 17 pgs: 17 active+clean; 1.3 GiB data, 4.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-23T16:26:39.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:39 smithi188 ceph-mon[44101]: pgmap v778: 17 pgs: 17 active+clean; 1.3 GiB data, 4.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-23T16:26:39.486 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:39 smithi103 ceph-mon[39629]: pgmap v778: 17 pgs: 17 active+clean; 1.3 GiB data, 4.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-23T16:26:39.517 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:39.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:39.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:39.621 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:26:39.653 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:39.886 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:26:39.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:39.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:26:39.993 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:26:40.025 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:40.071 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:26:40.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:40.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:26:40.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:40 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2846959558' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:26:40.124 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:26:40.175 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:26:40.243 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:40.276 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:26:40.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:40.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:26:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:40 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2846959558' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:26:40.384 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:26:40.415 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:40.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:40 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2846959558' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:26:40.476 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:26:40.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:40.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:26:40.578 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:26:40.578 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:26:40.580 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:26:40.632 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:40.848 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:26:40.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:40.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:26:40.954 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:26:40.988 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:41.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:41 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:26:41.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:41 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2846959558' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:26:41.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:41 smithi103 ceph-mon[39629]: osdmap e267: 12 total, 11 up, 10 in 2024-04-23T16:26:41.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:41 smithi103 ceph-mon[39629]: pgmap v780: 17 pgs: 17 active+clean; 1.2 GiB data, 4.1 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-23T16:26:41.318 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:41 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:26:41.318 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:41 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2846959558' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:26:41.318 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:41 smithi148 ceph-mon[44520]: osdmap e267: 12 total, 11 up, 10 in 2024-04-23T16:26:41.318 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:41 smithi148 ceph-mon[44520]: pgmap v780: 17 pgs: 17 active+clean; 1.2 GiB data, 4.1 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-23T16:26:41.347 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:26:41.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:41.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:26:41.355 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-23T16:26:41.451 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:26:41.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:41 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:26:41.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:41 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2846959558' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:26:41.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:41 smithi188 ceph-mon[44101]: osdmap e267: 12 total, 11 up, 10 in 2024-04-23T16:26:41.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:41 smithi188 ceph-mon[44101]: pgmap v780: 17 pgs: 17 active+clean; 1.2 GiB data, 4.1 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-23T16:26:41.490 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:41.754 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:26:41.948 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:26:41.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:41.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:26:42.054 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:26:42.112 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:42.142 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:26:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:42.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:26:42.246 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:26:42.249 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:26:42.271 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:26:42.292 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:26:42.295 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:42.315 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:26:42.320 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:26:42.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:42.324 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:26:42.341 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:26:42.372 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:26:42.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:42 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1256500577' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:26:42.397 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:26:42.423 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:26:42.425 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:26:42.447 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:26:42.455 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:42.476 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:26:42.480 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:26:42.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:42.484 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:26:42.508 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:26:42.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:26:42.509 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T16:26:42.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:26:42.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:26:42.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:26:42.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:26:42.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:26:42.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:26:42.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:26:42.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:26:42.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:26:42.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:26:42.530 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:42 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1256500577' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:26:42.585 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:26:42.619 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:42.660 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:26:42.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:42.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:26:42.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:42 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1256500577' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:26:42.763 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:42.804 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:42.827 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:42.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:42.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:42.931 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:26:42.966 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:42.971 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:26:43.474 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:26:43.474 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:26:43.493 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.494 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.494 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.494 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.494 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.494 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.494 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.494 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.494 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.494 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.494 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:43 smithi148 ceph-mon[44520]: pgmap v781: 17 pgs: 17 active+clean; 1.1 GiB data, 3.9 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-23T16:26:43.522 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:26:43.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:43.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:26:43.558 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:26:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:43 smithi103 ceph-mon[39629]: pgmap v781: 17 pgs: 17 active+clean; 1.1 GiB data, 3.9 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-23T16:26:43.632 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:26:43.662 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:43 smithi188 ceph-mon[44101]: pgmap v781: 17 pgs: 17 active+clean; 1.1 GiB data, 3.9 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-23T16:26:43.683 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:26:43.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:43.686 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:26:43.787 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:26:43.854 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:43.900 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:26:43.901 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:26:43.901 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:43.901 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:324907 3.a 3:5fee7a0f:::benchmark_data_smithi188_89276_object156263:head [delete] snapc 0=[] ondisk+write+known_if_redirected e267)", 2024-04-23T16:26:43.901 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:43.887630+0000", 2024-04-23T16:26:43.901 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.0071366060000000002, 2024-04-23T16:26:43.901 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0071622099999999996, 2024-04-23T16:26:43.901 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:26:43.901 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:26:43.901 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:26:43.901 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:26:43.901 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:26:43.902 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:26:43.902 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 324907 2024-04-23T16:26:43.902 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:43.902 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:26:43.902 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:43.902 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:26:43.902 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:43.887630+0000", 2024-04-23T16:26:43.902 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:43.902 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:43.902 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:43.902 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:26:43.902 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:43.887630+0000", 2024-04-23T16:26:43.902 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:26:43.902 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:43.903 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:43.903 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:26:43.903 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:43.887632+0000", 2024-04-23T16:26:43.903 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7039999999999999e-06 2024-04-23T16:26:43.903 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:43.903 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:43.903 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:26:43.903 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:43.887640+0000", 2024-04-23T16:26:43.903 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8099999999999998e-06 2024-04-23T16:26:43.903 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:43.903 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:43.903 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:26:43.903 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:43.887658+0000", 2024-04-23T16:26:43.903 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7895999999999999e-05 2024-04-23T16:26:43.903 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:43.904 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:43.904 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:26:43.904 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:43.887670+0000", 2024-04-23T16:26:43.904 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2031000000000001e-05 2024-04-23T16:26:43.904 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:43.904 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:43.904 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:26:43.904 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:43.887756+0000", 2024-04-23T16:26:43.904 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5958999999999995e-05 2024-04-23T16:26:43.904 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:43.904 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:43.905 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:26:43.905 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:43.887873+0000", 2024-04-23T16:26:43.905 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000116662 2024-04-23T16:26:43.905 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:43.905 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:43.905 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 4", 2024-04-23T16:26:43.905 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:43.888586+0000", 2024-04-23T16:26:43.905 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00071361500000000002 2024-04-23T16:26:43.905 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:26:43.905 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:26:43.905 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:26:43.905 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:43.894067+0000", 2024-04-23T16:26:43.905 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0054807639999999999 2024-04-23T16:26:43.906 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:43.906 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:26:43.906 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:43.906 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:26:43.906 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-04-23T16:26:43.906 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 1 2024-04-23T16:26:43.906 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:26:43.906 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:26:43.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:43.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:26:44.015 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:26:44.051 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:44.075 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:26:44.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:44.077 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:26:44.178 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:26:44.217 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:44.261 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:26:44.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:44.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:26:44.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:44.375 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:26:44.413 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:44.596 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:44.624 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:26:44.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:44.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:26:44.634 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-23T16:26:44.730 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:26:44.760 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:44.789 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:26:44.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:44.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:26:44.901 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:26:44.968 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:45.007 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:26:45.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:45.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:26:45.096 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:26:45.114 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:26:45.148 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:45.176 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:26:45.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:45.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:26:45.282 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:26:45.345 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:45.420 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:26:45.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:45.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:26:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:45 smithi188 ceph-mon[44101]: pgmap v782: 17 pgs: 17 active+clean; 894 MiB data, 3.5 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-23T16:26:45.475 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:26:45.529 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:26:45.578 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:45 smithi148 ceph-mon[44520]: pgmap v782: 17 pgs: 17 active+clean; 894 MiB data, 3.5 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-23T16:26:45.580 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:45.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:45 smithi103 ceph-mon[39629]: pgmap v782: 17 pgs: 17 active+clean; 894 MiB data, 3.5 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-23T16:26:45.664 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:26:45.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:45.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:26:45.773 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:26:45.802 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:45.824 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:26:45.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:45.831 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:26:45.843 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:26:45.933 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:26:45.984 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:46.309 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:46 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2695728529' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:26:46.310 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:46 smithi148 ceph-mon[44520]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:26:46.316 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:26:46.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:46 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2695728529' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:26:46.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:46 smithi103 ceph-mon[39629]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:26:46.442 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:26:46.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:46.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:26:46.547 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:26:46.585 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:46.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:46 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2695728529' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:26:46.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:46 smithi188 ceph-mon[44101]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:26:46.730 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:26:46.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:46.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:26:46.835 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:26:46.897 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:46.925 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:26:46.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:46.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:26:47.029 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:26:47.113 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:26:47.113 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:26:47.113 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2024-04-23T16:26:47.114 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2024-04-23T16:26:47.114 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:26:47.118 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:47.145 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:26:47.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:47.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:26:47.257 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:26:47.290 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:47.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:47 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:26:47.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:47 smithi103 ceph-mon[39629]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:26:47.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:47 smithi103 ceph-mon[39629]: osdmap e268: 12 total, 11 up, 10 in 2024-04-23T16:26:47.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:47 smithi103 ceph-mon[39629]: pgmap v784: 17 pgs: 17 active+clean; 705 MiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-23T16:26:47.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:47 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:26:47.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:47 smithi148 ceph-mon[44520]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:26:47.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:47 smithi148 ceph-mon[44520]: osdmap e268: 12 total, 11 up, 10 in 2024-04-23T16:26:47.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:47 smithi148 ceph-mon[44520]: pgmap v784: 17 pgs: 17 active+clean; 705 MiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-23T16:26:47.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:47 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:26:47.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:47 smithi188 ceph-mon[44101]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:26:47.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:47 smithi188 ceph-mon[44101]: osdmap e268: 12 total, 11 up, 10 in 2024-04-23T16:26:47.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:47 smithi188 ceph-mon[44101]: pgmap v784: 17 pgs: 17 active+clean; 705 MiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-23T16:26:48.893 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-04-23T16:26:48.893 INFO:tasks.cephadm.osd.1:Restarting osd.1 (starting--it wasn't running)... 2024-04-23T16:26:48.893 DEBUG:teuthology.orchestra.run.smithi103:osd.1> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.1.service 2024-04-23T16:26:48.896 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl start ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.1 2024-04-23T16:26:48.930 INFO:journalctl@ceph.osd.1.smithi103.stdout:-- Logs begin at Tue 2024-04-23 15:57:24 UTC. -- 2024-04-23T16:26:49.350 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:26:48 smithi103 systemd[1]: Starting Ceph osd.1 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T16:26:49.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:49 smithi188 ceph-mon[44101]: pgmap v785: 17 pgs: 17 active+clean; 613 MiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:26:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:49 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:26:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:49 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:26:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:49 smithi188 ceph-mon[44101]: osdmap e269: 12 total, 11 up, 10 in 2024-04-23T16:26:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:49 smithi103 ceph-mon[39629]: pgmap v785: 17 pgs: 17 active+clean; 613 MiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:26:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:49 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:26:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:49 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:26:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:49 smithi103 ceph-mon[39629]: osdmap e269: 12 total, 11 up, 10 in 2024-04-23T16:26:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:49 smithi148 ceph-mon[44520]: pgmap v785: 17 pgs: 17 active+clean; 613 MiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:26:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:49 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:26:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:49 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:26:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:49 smithi148 ceph-mon[44520]: osdmap e269: 12 total, 11 up, 10 in 2024-04-23T16:26:51.545 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:26:51.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:51.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:26:51.649 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:26:51.687 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:51.727 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:26:51.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:51.732 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:26:51.833 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:26:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:51 smithi103 ceph-mon[39629]: pgmap v787: 17 pgs: 17 active+clean; 496 MiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:26:51.881 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:51.967 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:51 smithi148 ceph-mon[44520]: pgmap v787: 17 pgs: 17 active+clean; 496 MiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:26:51.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:51 smithi188 ceph-mon[44101]: pgmap v787: 17 pgs: 17 active+clean; 496 MiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:26:52.011 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-23T16:26:52.041 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:26:52.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:52.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:26:52.147 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:26:52.178 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:52.206 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:26:52.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:52.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:26:52.314 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:26:52.356 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:52.395 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:26:52.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:52.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:26:52.435 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:26:52.507 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:26:52.553 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:52.679 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:26:52.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:52.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:26:52.787 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:26:52.843 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:52.960 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:26:52.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:52.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:26:53.065 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:26:53.096 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:53.126 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:26:53.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:53.133 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:26:53.179 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:26:53.179 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:26:53.179 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:26:53.179 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:26:53.179 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:26:53.179 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:26:53.179 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:26:53.185 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:26:53.213 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:26:53.234 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:26:53.250 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:26:53.264 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:53.286 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:26:53.286 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:26:53.287 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T16:26:53.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:26:53.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:26:53.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:26:53.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:26:53.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:26:53.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:26:53.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:26:53.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:26:53.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:26:53.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:26:53.304 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:26:53.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:53.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:26:53.410 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:26:53.463 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:53.545 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:26:53.590 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:26:53 smithi103 bash[149286]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T16:26:53.590 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:26:53 smithi103 bash[149286]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-23T16:26:53.590 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:26:53 smithi103 bash[149286]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-23T16:26:53.591 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:26:53 smithi103 bash[149286]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-23T16:26:53.591 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:26:53 smithi103 bash[149286]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-23T16:26:53.591 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:26:53 smithi103 bash[149286]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-23T16:26:53.591 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:26:53 smithi103 bash[149286]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-23T16:26:53.591 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:26:53 smithi103 bash[149286]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-23T16:26:53.714 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:53 smithi148 ceph-mon[44520]: pgmap v788: 17 pgs: 17 active+clean; 339 MiB data, 2.4 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-23T16:26:53.715 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:53 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2197688644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:26:53.715 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:53 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:53.715 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:53 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:53.732 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:26:53.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:53.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:26:53.843 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:26:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:53 smithi103 ceph-mon[39629]: pgmap v788: 17 pgs: 17 active+clean; 339 MiB data, 2.4 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-23T16:26:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:53 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2197688644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:26:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:53 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:53 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:53 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:53 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:53 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:53.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:53 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:53.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:53 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:53.873 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:53.898 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:26:53.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:53.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:26:53.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:53 smithi188 ceph-mon[44101]: pgmap v788: 17 pgs: 17 active+clean; 339 MiB data, 2.4 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-23T16:26:53.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:53 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2197688644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:26:53.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:53 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:53.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:53 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:53.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:53 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:53.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:53 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:53.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:53 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:53.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:53 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:53.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:53 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:54.002 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:26:54.036 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:54.061 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:26:54.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:54.066 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:26:54.075 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:53 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:54.076 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:53 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:54.076 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:53 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:54.076 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:53 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:54.076 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:53 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:54.096 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:26:54.097 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:26:54.167 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:26:54.218 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:54.361 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:26:54.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:54.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:26:54.395 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Removed 168641 objects 2024-04-23T16:26:54.396 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Clean up completed and total clean up time :177.909 2024-04-23T16:26:54.409 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:26:54.409 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:26:54.469 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:26:54.497 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:54.645 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:26:54.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:54.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:26:54.749 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:26:54.775 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:54.794 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:26:54.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:54.797 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:26:54.898 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:26:54.924 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:54.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:54.946 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:26:54.947 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:26:54.947 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-04-23T16:26:54.947 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-04-23T16:26:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:54.974 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:54.975 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:54.975 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:54.975 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:55.047 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:26:55.073 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:55.085 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:55.086 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:55.086 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:55.086 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:26:55.111 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-23T16:26:55.271 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:26:55.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:55.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:26:55.370 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:26:55.374 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:26:55.402 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:55.529 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-23T16:26:55.551 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:26:55.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:55.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:26:55.654 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:55.658 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:26:55.667 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:26:55.688 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:55 smithi103 ceph-mon[39629]: pgmap v789: 17 pgs: 17 active+clean; 155 MiB data, 2.1 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:26:55.850 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:26:55 smithi103 bash[149676]: 0236630712e315f4b3bf924c7ce64cb593f24f64e2125792c458bcd707c02229 2024-04-23T16:26:55.850 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:26:55 smithi103 systemd[1]: Started Ceph osd.1 for 8e3594f2-018b-11ef-bc93-c7b262605968. 2024-04-23T16:26:55.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:55 smithi148 ceph-mon[44520]: pgmap v789: 17 pgs: 17 active+clean; 155 MiB data, 2.1 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:26:55.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:55 smithi188 ceph-mon[44101]: pgmap v789: 17 pgs: 17 active+clean; 155 MiB data, 2.1 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:26:56.041 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:56.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:56.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:56.098 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:26:56.146 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:26:56.173 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:56.296 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:26:55 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-1[149775]: 2024-04-23T16:26:55.936+0000 7fd073a257c0 -1 Falling back to public interface 2024-04-23T16:26:56.296 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:26:56.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:56.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:26:56.403 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:26:56.433 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:56.455 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:26:56.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:56.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:26:56.559 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:26:56.586 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:56.734 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:26:56.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:56.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:26:56.840 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:56.869 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:56.982 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:26:56.982 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:26:57.685 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-23T16:26:57.717 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:57.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:57.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:57.730 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:26:57.822 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:26:57.854 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:57.875 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:26:57.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:57.878 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:26:57.950 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:26:57.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:57 smithi188 ceph-mon[44101]: pgmap v790: 17 pgs: 17 active+clean; 30 MiB data, 1.8 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:26:57.979 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:26:58.006 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:57 smithi103 ceph-mon[39629]: pgmap v790: 17 pgs: 17 active+clean; 30 MiB data, 1.8 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:26:58.010 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:58.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:57 smithi148 ceph-mon[44520]: pgmap v790: 17 pgs: 17 active+clean; 30 MiB data, 1.8 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:26:58.186 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:26:58.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:58.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:26:58.196 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:26:58.292 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:26:58.319 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:59.068 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:26:59.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:59.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:26:59.172 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:26:59.199 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:59.316 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:26:59.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:59.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:26:59.421 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:26:59.453 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:59.474 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:26:59.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:59.477 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:26:59.573 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:26:59.574 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:26:59.577 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:26:59.600 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:26:59 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-1[149775]: 2024-04-23T16:26:59.286+0000 7fd073a257c0 -1 osd.1 232 log_to_monitors true 2024-04-23T16:26:59.606 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:59.627 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:26:59.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:59.630 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:26:59.730 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:26:59.731 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:26:59.765 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:26:59.858 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:59 smithi148 ceph-mon[44520]: pgmap v791: 17 pgs: 17 active+clean; 8.4 MiB data, 1.8 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 967 op/s 2024-04-23T16:26:59.858 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:59 smithi148 ceph-mon[44520]: from='osd.1 v1:172.21.15.103:6805/859282187' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T16:26:59.858 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:59 smithi148 ceph-mon[44520]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T16:26:59.858 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:59 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/401835071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:26:59.858 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:26:59 smithi148 ceph-mon[44520]: from='client.35369 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:26:59.889 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:26:59.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:26:59.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:26:59.992 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:27:00.017 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 16:26:59 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-1[149775]: 2024-04-23T16:26:59.736+0000 7fd060e75700 -1 osd.1 232 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:27:00.017 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:59 smithi103 ceph-mon[39629]: pgmap v791: 17 pgs: 17 active+clean; 8.4 MiB data, 1.8 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 967 op/s 2024-04-23T16:27:00.017 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:59 smithi103 ceph-mon[39629]: from='osd.1 v1:172.21.15.103:6805/859282187' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T16:27:00.017 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:59 smithi103 ceph-mon[39629]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T16:27:00.018 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:59 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/401835071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:27:00.018 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:26:59 smithi103 ceph-mon[39629]: from='client.35369 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:27:00.020 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:00.092 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:27:00.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:00.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:27:00.163 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-23T16:27:00.195 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:27:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:59 smithi188 ceph-mon[44101]: pgmap v791: 17 pgs: 17 active+clean; 8.4 MiB data, 1.8 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 967 op/s 2024-04-23T16:27:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:59 smithi188 ceph-mon[44101]: from='osd.1 v1:172.21.15.103:6805/859282187' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T16:27:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:59 smithi188 ceph-mon[44101]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T16:27:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:59 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/401835071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:27:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:26:59 smithi188 ceph-mon[44101]: from='client.35369 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:27:00.227 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:00.248 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:27:00.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:00.251 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:27:00.352 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:27:00.380 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:00.400 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:27:00.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:00.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:27:00.428 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:27:00.504 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:27:00.531 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:00 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:27:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:00 smithi103 ceph-mon[39629]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T16:27:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:00 smithi103 ceph-mon[39629]: from='client.35369 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:27:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:00 smithi103 ceph-mon[39629]: osdmap e270: 12 total, 11 up, 10 in 2024-04-23T16:27:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:00 smithi103 ceph-mon[39629]: from='osd.1 v1:172.21.15.103:6805/859282187' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:27:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:00 smithi103 ceph-mon[39629]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:27:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:00 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:27:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:00 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:27:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:00 smithi148 ceph-mon[44520]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T16:27:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:00 smithi148 ceph-mon[44520]: from='client.35369 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:27:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:00 smithi148 ceph-mon[44520]: osdmap e270: 12 total, 11 up, 10 in 2024-04-23T16:27:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:00 smithi148 ceph-mon[44520]: from='osd.1 v1:172.21.15.103:6805/859282187' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:27:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:00 smithi148 ceph-mon[44520]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:27:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:00 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:27:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:00 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:27:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:00 smithi188 ceph-mon[44101]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T16:27:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:00 smithi188 ceph-mon[44101]: from='client.35369 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:27:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:00 smithi188 ceph-mon[44101]: osdmap e270: 12 total, 11 up, 10 in 2024-04-23T16:27:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:00 smithi188 ceph-mon[44101]: from='osd.1 v1:172.21.15.103:6805/859282187' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:27:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:00 smithi188 ceph-mon[44101]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:27:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:00 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:27:01.740 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'unique_pool_1' removed 2024-04-23T16:27:01.818 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:01 smithi148 ceph-mon[44520]: pgmap v793: 17 pgs: 17 active+clean; 577 KiB data, 1.7 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 792 op/s 2024-04-23T16:27:01.818 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:01 smithi148 ceph-mon[44520]: osd.1 v1:172.21.15.103:6805/859282187 boot 2024-04-23T16:27:01.818 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:01 smithi148 ceph-mon[44520]: osdmap e271: 12 total, 12 up, 10 in 2024-04-23T16:27:01.818 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:01 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:27:01.818 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:01 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3267931035' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:27:01.980 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:27:01.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:01.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:27:02.084 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:27:02.086 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:27:02.086 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:27:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:01 smithi103 ceph-mon[39629]: pgmap v793: 17 pgs: 17 active+clean; 577 KiB data, 1.7 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 792 op/s 2024-04-23T16:27:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:01 smithi103 ceph-mon[39629]: osd.1 v1:172.21.15.103:6805/859282187 boot 2024-04-23T16:27:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:01 smithi103 ceph-mon[39629]: osdmap e271: 12 total, 12 up, 10 in 2024-04-23T16:27:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:01 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:27:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:01 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3267931035' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:27:02.114 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:01 smithi188 ceph-mon[44101]: pgmap v793: 17 pgs: 17 active+clean; 577 KiB data, 1.7 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 792 op/s 2024-04-23T16:27:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:01 smithi188 ceph-mon[44101]: osd.1 v1:172.21.15.103:6805/859282187 boot 2024-04-23T16:27:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:01 smithi188 ceph-mon[44101]: osdmap e271: 12 total, 12 up, 10 in 2024-04-23T16:27:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:01 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T16:27:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:01 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3267931035' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:27:02.663 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:27:02.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:02.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:27:02.672 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:27:02.672 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:27:02.672 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-23T16:27:02.672 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:27:02.766 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:27:02.796 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:03.013 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:03.013 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:02 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:27:03.013 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:02 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:27:03.013 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:02 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3267931035' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:27:03.013 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:02 smithi148 ceph-mon[44520]: osdmap e272: 12 total, 12 up, 10 in 2024-04-23T16:27:03.013 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:02 smithi148 ceph-mon[44520]: 3.0 scrub starts 2024-04-23T16:27:03.014 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:02 smithi148 ceph-mon[44520]: 3.0 scrub ok 2024-04-23T16:27:03.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:03.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:02 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:27:03.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:02 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:27:03.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:02 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3267931035' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:27:03.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:02 smithi103 ceph-mon[39629]: osdmap e272: 12 total, 12 up, 10 in 2024-04-23T16:27:03.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:02 smithi103 ceph-mon[39629]: 3.0 scrub starts 2024-04-23T16:27:03.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:02 smithi103 ceph-mon[39629]: 3.0 scrub ok 2024-04-23T16:27:03.104 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:27:03.104 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:27:03.104 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:27:03.104 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:27:03.104 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:27:03.105 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:27:03.105 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:27:03.105 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:27:03.105 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:27:03.105 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:27:03.105 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:27:03.125 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:27:03.126 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:27:03.126 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-04-23T16:27:03.126 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:27:03.126 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:27:03.126 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:27:03.126 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:27:03.126 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:27:03.126 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:27:03.127 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:27:03.127 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:27:03.127 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:27:03.127 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:27:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:02 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:27:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:02 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:27:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:02 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3267931035' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:27:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:02 smithi188 ceph-mon[44101]: osdmap e272: 12 total, 12 up, 10 in 2024-04-23T16:27:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:02 smithi188 ceph-mon[44101]: 3.0 scrub starts 2024-04-23T16:27:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:02 smithi188 ceph-mon[44101]: 3.0 scrub ok 2024-04-23T16:27:03.405 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-23T16:27:03.676 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:27:03.831 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-23T16:27:03.832 INFO:tasks.radosbench:Beginning radosbench... 2024-04-23T16:27:03.832 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-04-23T16:27:03.832 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-04-23T16:27:03.849 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:27:03.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:03.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:27:03.954 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:27:03.980 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:03 smithi103 ceph-mon[39629]: pgmap v796: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:03 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1017780765' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:27:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:03 smithi148 ceph-mon[44520]: pgmap v796: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:04.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:03 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1017780765' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:27:04.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:03 smithi188 ceph-mon[44101]: pgmap v796: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:03 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1017780765' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:27:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.315 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:27:04.315 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-04-23T16:27:04.316 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-23T16:27:04.316 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-23T16:27:04.316 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-04-23T16:27:04.529 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:27:04.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:04.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:27:04.639 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:27:04.666 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:04.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:04.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:05.033 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:05.033 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:05.114 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:27:05.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:05.118 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:27:05.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:05.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:05.218 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:27:05.247 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:05.266 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:27:05.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:05.268 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:27:05.369 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:27:05.397 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:27:05.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:05.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:27:05.530 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:27:05.600 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:27:05.627 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:05.783 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:27:05.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:05.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:27:05.887 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:27:05.915 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:05.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:05 smithi188 ceph-mon[44101]: pgmap v797: 1 pgs: 1 active+clean; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:05.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:05 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:27:05.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:05 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:27:06.075 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:27:06.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:05 smithi103 ceph-mon[39629]: pgmap v797: 1 pgs: 1 active+clean; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:06.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:05 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:27:06.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:05 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:27:06.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:05 smithi148 ceph-mon[44520]: pgmap v797: 1 pgs: 1 active+clean; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:06.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:05 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:27:06.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:05 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:27:06.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:06 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:27:06.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:06 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:27:06.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:06 smithi103 ceph-mon[39629]: osdmap e273: 12 total, 12 up, 10 in 2024-04-23T16:27:06.879 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:27:06.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:06 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:27:06.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:06 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:27:06.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:06 smithi188 ceph-mon[44101]: osdmap e273: 12 total, 12 up, 10 in 2024-04-23T16:27:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:06 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:27:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:06 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:27:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:06 smithi148 ceph-mon[44520]: osdmap e273: 12 total, 12 up, 10 in 2024-04-23T16:27:07.532 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:27:07.559 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:27:07.664 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:27:07.664 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:27:07.664 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-04-23T16:27:07.664 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:27:07.787 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'unique_pool_2' created 2024-04-23T16:27:07.817 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:07 smithi103 ceph-mon[39629]: pgmap v799: 1 pgs: 1 active+clean; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:07.818 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:07 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:27:07.818 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:07 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:27:07.844 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:27:07.846 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:07 smithi188 ceph-mon[44101]: pgmap v799: 1 pgs: 1 active+clean; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:07.846 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:07 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:27:07.846 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:07 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:27:07.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:07.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:27:07.951 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:27:07.980 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:08.033 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:27:08.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:08.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:27:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:07 smithi148 ceph-mon[44520]: pgmap v799: 1 pgs: 1 active+clean; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:07 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:27:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:07 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:27:08.136 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:27:08.174 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:08.195 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:27:08.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:08.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:27:08.239 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-23T16:27:08.300 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:27:08.333 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:08.354 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:27:08.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:08.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:27:08.458 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:27:08.488 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:08.508 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:27:08.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:08.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:27:08.612 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:27:08.642 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:08.679 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:27:08.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:08.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:27:08.706 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:27:08.716 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:27:08.716 INFO:teuthology.orchestra.run.smithi188.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-23T16:27:08.716 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:27:08.783 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:27:08.814 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:08.852 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:27:08.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:08.855 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:27:08.956 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:27:08.983 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:09.084 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:08 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3731240019' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-23T16:27:09.084 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:08 smithi103 ceph-mon[39629]: from='client.25330 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-23T16:27:09.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:08 smithi103 ceph-mon[39629]: from='client.25330 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-04-23T16:27:09.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:08 smithi103 ceph-mon[39629]: osdmap e274: 12 total, 12 up, 10 in 2024-04-23T16:27:09.112 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-04-23T16:27:09.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:08 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3731240019' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-23T16:27:09.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:08 smithi148 ceph-mon[44520]: from='client.25330 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-23T16:27:09.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:08 smithi148 ceph-mon[44520]: from='client.25330 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-04-23T16:27:09.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:08 smithi148 ceph-mon[44520]: osdmap e274: 12 total, 12 up, 10 in 2024-04-23T16:27:09.174 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:27:09.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:08 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3731240019' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-23T16:27:09.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:08 smithi188 ceph-mon[44101]: from='client.25330 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-23T16:27:09.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:08 smithi188 ceph-mon[44101]: from='client.25330 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-04-23T16:27:09.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:08 smithi188 ceph-mon[44101]: osdmap e274: 12 total, 12 up, 10 in 2024-04-23T16:27:09.873 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:09 smithi103 ceph-mon[39629]: pgmap v801: 17 pgs: 1 creating+peering, 15 unknown, 1 active+clean; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:09.874 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:09 smithi103 ceph-mon[39629]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:27:09.874 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:09 smithi103 ceph-mon[39629]: osdmap e275: 12 total, 12 up, 10 in 2024-04-23T16:27:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:09 smithi188 ceph-mon[44101]: pgmap v801: 17 pgs: 1 creating+peering, 15 unknown, 1 active+clean; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:09 smithi188 ceph-mon[44101]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:27:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:09 smithi188 ceph-mon[44101]: osdmap e275: 12 total, 12 up, 10 in 2024-04-23T16:27:10.031 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:27:10.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:10.035 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:27:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:09 smithi148 ceph-mon[44520]: pgmap v801: 17 pgs: 1 creating+peering, 15 unknown, 1 active+clean; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:09 smithi148 ceph-mon[44520]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:27:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:09 smithi148 ceph-mon[44520]: osdmap e275: 12 total, 12 up, 10 in 2024-04-23T16:27:10.136 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:27:10.164 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:10.513 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:27:10.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:10.516 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:27:10.617 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:27:10.650 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:10.671 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:27:10.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:10.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:27:10.775 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:27:10.805 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:10.825 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:27:10.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:10.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:27:10.929 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:27:10.959 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:11.018 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:27:11.018 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:27:11.040 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:10 smithi188 ceph-mon[44101]: osdmap e276: 12 total, 12 up, 10 in 2024-04-23T16:27:11.040 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:10 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:27:11.040 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:10 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:27:11.040 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:10 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1299128731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:27:11.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:10 smithi103 ceph-mon[39629]: osdmap e276: 12 total, 12 up, 10 in 2024-04-23T16:27:11.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:10 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:27:11.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:10 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:27:11.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:10 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1299128731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:27:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:10 smithi148 ceph-mon[44520]: osdmap e276: 12 total, 12 up, 10 in 2024-04-23T16:27:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:10 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:27:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:10 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:27:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:10 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1299128731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:27:11.359 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:27:11.359 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:27:11.359 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:27:11.359 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:27:11.359 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:27:11.359 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:27:11.359 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:27:11.359 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:27:11.360 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:27:11.360 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:27:11.360 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:27:11.360 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:27:11.360 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:27:11.360 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:27:11.360 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:27:11.360 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:27:11.360 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:27:11.360 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:27:11.360 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:27:11.361 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:27:11.361 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:27:11.361 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:27:11.361 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:27:11.361 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:27:11.748 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-23T16:27:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:11.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:11.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:11.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:11 smithi188 ceph-mon[44101]: pgmap v804: 17 pgs: 7 active+clean, 1 creating+activating, 3 creating+peering, 6 unknown; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:11.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:11.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:11.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:11.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:11.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:11.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:11.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:11.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:11 smithi188 ceph-mon[44101]: osdmap e277: 12 total, 12 up, 10 in 2024-04-23T16:27:12.010 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:27:12.044 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.044 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.044 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.044 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.044 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.044 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:11 smithi103 ceph-mon[39629]: pgmap v804: 17 pgs: 7 active+clean, 1 creating+activating, 3 creating+peering, 6 unknown; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:12.044 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.044 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:11 smithi103 ceph-mon[39629]: osdmap e277: 12 total, 12 up, 10 in 2024-04-23T16:27:12.101 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:27:12.108 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:27:12.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:12.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:27:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:11 smithi148 ceph-mon[44520]: pgmap v804: 17 pgs: 7 active+clean, 1 creating+activating, 3 creating+peering, 6 unknown; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:11 smithi148 ceph-mon[44520]: osdmap e277: 12 total, 12 up, 10 in 2024-04-23T16:27:12.211 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:27:12.242 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:13.090 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:27:13.093 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:12 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:27:13.093 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:12 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:27:13.093 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:12 smithi103 ceph-mon[39629]: osdmap e278: 12 total, 12 up, 10 in 2024-04-23T16:27:13.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:13.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:27:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:12 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:27:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:12 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:27:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:12 smithi148 ceph-mon[44520]: osdmap e278: 12 total, 12 up, 10 in 2024-04-23T16:27:13.160 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:12 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:27:13.160 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:12 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:27:13.160 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:12 smithi188 ceph-mon[44101]: osdmap e278: 12 total, 12 up, 10 in 2024-04-23T16:27:13.194 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:27:13.224 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:13.246 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:27:13.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:13.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:27:13.350 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:27:13.380 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:13.478 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:27:13.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:13.481 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:27:13.582 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:27:13.614 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:13.635 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:27:13.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:13.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:27:13.740 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:27:13.768 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:13.771 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:27:13.771 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:27:13.786 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:27:13.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:13.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:27:13.814 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:27:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:13 smithi148 ceph-mon[44520]: pgmap v807: 17 pgs: 13 active+clean, 1 creating+activating, 3 creating+peering; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:13 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1770394606' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:27:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:13 smithi148 ceph-mon[44520]: from='client.35402 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:27:13.890 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:27:13.923 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:13.945 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:27:13.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:13.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:27:14.016 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:13 smithi103 ceph-mon[39629]: pgmap v807: 17 pgs: 13 active+clean, 1 creating+activating, 3 creating+peering; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:14.017 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:13 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1770394606' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:27:14.017 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:13 smithi103 ceph-mon[39629]: from='client.35402 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:27:14.048 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:27:14.074 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:27:14.075 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:14.093 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:27:14.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:14.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:27:14.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:13 smithi188 ceph-mon[44101]: pgmap v807: 17 pgs: 13 active+clean, 1 creating+activating, 3 creating+peering; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:14.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:13 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1770394606' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:27:14.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:13 smithi188 ceph-mon[44101]: from='client.35402 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:27:14.197 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:27:14.225 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:14.473 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:27:14.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:14.477 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:27:14.494 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-23T16:27:14.578 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:27:14.608 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:14.792 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:27:14.859 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:27:14.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:14.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:27:14.963 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:27:14.990 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:15.049 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:27:15.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:14 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:27:15.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:14 smithi188 ceph-mon[44101]: from='client.35402 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:27:15.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:14 smithi188 ceph-mon[44101]: osdmap e279: 12 total, 12 up, 10 in 2024-04-23T16:27:15.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:27:15.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:14 smithi188 ceph-mon[44101]: osdmap e280: 12 total, 12 up, 10 in 2024-04-23T16:27:15.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:27:15.073 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:27:15.073 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:27:15.073 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:14 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2418325320' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:27:15.088 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:27:15.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:15.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:27:15.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:14 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:27:15.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:14 smithi103 ceph-mon[39629]: from='client.35402 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:27:15.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:14 smithi103 ceph-mon[39629]: osdmap e279: 12 total, 12 up, 10 in 2024-04-23T16:27:15.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:27:15.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:14 smithi103 ceph-mon[39629]: osdmap e280: 12 total, 12 up, 10 in 2024-04-23T16:27:15.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:27:15.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:27:15.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:27:15.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:14 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2418325320' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:27:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:14 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:27:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:14 smithi148 ceph-mon[44520]: from='client.35402 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:27:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:14 smithi148 ceph-mon[44520]: osdmap e279: 12 total, 12 up, 10 in 2024-04-23T16:27:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:27:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:14 smithi148 ceph-mon[44520]: osdmap e280: 12 total, 12 up, 10 in 2024-04-23T16:27:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:27:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:27:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:27:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:14 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2418325320' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:27:15.192 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:27:15.220 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:15.242 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:27:15.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:15.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:27:15.344 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:27:15.371 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:15.389 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:27:15.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:15.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:27:15.481 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10, 1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1] 2024-04-23T16:27:15.481 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:27:15.481 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-23T16:27:15.481 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 1.000000 2024-04-23T16:27:15.481 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 3 1 2024-04-23T16:27:15.494 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:27:15.520 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:15.538 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:27:15.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:15.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:27:15.642 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:27:15.669 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:15.686 INFO:teuthology.orchestra.run.smithi103.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-04-23T16:27:15.924 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:27:15.925 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:15 smithi103 ceph-mon[39629]: pgmap v810: 17 pgs: 17 active+clean; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:15.926 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:15 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2418325320' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:27:15.926 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:15 smithi103 ceph-mon[39629]: osdmap e281: 12 total, 12 up, 10 in 2024-04-23T16:27:15.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:15.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:27:15.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:15 smithi188 ceph-mon[44101]: pgmap v810: 17 pgs: 17 active+clean; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:15.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:15 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2418325320' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:27:15.966 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:15 smithi188 ceph-mon[44101]: osdmap e281: 12 total, 12 up, 10 in 2024-04-23T16:27:16.027 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:27:16.056 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:15 smithi148 ceph-mon[44520]: pgmap v810: 17 pgs: 17 active+clean; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:16.056 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:15 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2418325320' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:27:16.056 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:15 smithi148 ceph-mon[44520]: osdmap e281: 12 total, 12 up, 10 in 2024-04-23T16:27:16.057 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:16.077 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:27:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:16.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:27:16.182 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:27:16.209 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:16.229 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:27:16.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:16.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:27:16.333 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:27:16.360 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:16.513 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:27:16.514 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:27:16.708 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:27:16.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:16.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:27:16.729 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:27:16.812 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:27:16.838 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:16.875 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:27:17.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:16 smithi103 ceph-mon[39629]: pgmap v812: 17 pgs: 17 active+clean; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:17.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:16 smithi103 ceph-mon[39629]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:27:17.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:16 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:27:17.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:16 smithi148 ceph-mon[44520]: pgmap v812: 17 pgs: 17 active+clean; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:17.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:16 smithi148 ceph-mon[44520]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:27:17.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:16 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:27:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:16 smithi188 ceph-mon[44101]: pgmap v812: 17 pgs: 17 active+clean; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail 2024-04-23T16:27:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:16 smithi188 ceph-mon[44101]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:27:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:16 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:27:17.191 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-23T16:27:17.452 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:27:17.587 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:27:17.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:17.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:27:17.692 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:27:17.720 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:17.723 DEBUG:teuthology.orchestra.run.smithi188:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-04-23T16:27:17.725 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-23T16:27:17.725 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-23T16:27:17.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:hints = 1 2024-04-23T16:27:17.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-23T16:27:17.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Object prefix: benchmark_data_smithi188_101370 2024-04-23T16:27:17.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:27:17.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 0 0 0 0 0 0 - 0 2024-04-23T16:27:17.919 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:27:17.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:17.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:27:18.027 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:27:18.064 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:18.269 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:27:18.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:18.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:27:18.377 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:27:18.420 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:18.699 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:18 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2127484448' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:27:18.775 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:27:18.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:18.779 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:27:18.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 1 16 6101 6085 380.292 380.312 0.00226024 0.00262315 2024-04-23T16:27:18.881 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:27:18.926 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:18.959 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:27:18.960 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:18 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2127484448' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:27:18.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:18.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:27:19.033 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:18 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2127484448' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:27:19.064 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:27:19.110 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:19.132 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:27:19.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:19.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:27:19.241 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:27:19.281 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:19.305 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:27:19.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:19.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:27:19.410 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:27:19.456 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:27:19.456 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:27:19.456 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:27:19.456 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:27:19.456 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:27:19.464 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:19.492 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:27:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:19.497 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:27:19.538 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:27:19.538 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:27:19.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:27:19.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:27:19.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:27:19.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:27:19.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:27:19.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:27:19.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:27:19.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:27:19.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:27:19.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:27:19.539 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:27:19.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:27:19.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:27:19.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:27:19.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:27:19.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:27:19.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:27:19.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:27:19.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:27:19.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:27:19.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:27:19.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:27:19.598 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:27:19.628 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:19.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:19 smithi188 ceph-mon[44101]: pgmap v813: 17 pgs: 17 active+clean; 20 MiB data, 1.8 GiB used, 892 GiB / 894 GiB avail; 3.2 MiB/s wr, 50 op/s 2024-04-23T16:27:19.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:19.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:19.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:19.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:19.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:19.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:19.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:19.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:19.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:19.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:19.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:19.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:19 smithi188 ceph-mon[44101]: osdmap e282: 12 total, 12 up, 10 in 2024-04-23T16:27:19.751 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:27:19.752 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:27:19.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:19.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:27:19.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 2 16 10455 10439 326.193 272.125 0.00178713 0.0030621 2024-04-23T16:27:19.859 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:27:19.904 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:19 smithi103 ceph-mon[39629]: pgmap v813: 17 pgs: 17 active+clean; 20 MiB data, 1.8 GiB used, 892 GiB / 894 GiB avail; 3.2 MiB/s wr, 50 op/s 2024-04-23T16:27:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:19 smithi103 ceph-mon[39629]: osdmap e282: 12 total, 12 up, 10 in 2024-04-23T16:27:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:19 smithi148 ceph-mon[44520]: pgmap v813: 17 pgs: 17 active+clean; 20 MiB data, 1.8 GiB used, 892 GiB / 894 GiB avail; 3.2 MiB/s wr, 50 op/s 2024-04-23T16:27:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:19 smithi148 ceph-mon[44520]: osdmap e282: 12 total, 12 up, 10 in 2024-04-23T16:27:20.248 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-23T16:27:20.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:20.325 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:27:20.325 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:27:20.426 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:27:20.464 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:20.497 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:27:20.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:20.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:27:20.603 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:27:20.656 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:20.682 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:27:20.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:20.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:27:20.761 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:27:20.786 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:27:20.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 3 16 14783 14767 307.619 270.5 0.00477159 0.00324727 2024-04-23T16:27:20.824 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:20.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:20.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:21.081 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:27:21.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:21.085 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:27:21.186 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:27:21.232 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:21.630 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:27:21.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:21.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:27:21.715 INFO:teuthology.orchestra.run.smithi103.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-23T16:27:21.734 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:27:21.775 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:21.801 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:27:21.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:21.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:27:21.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 4 16 19452 19436 303.662 291.812 0.00283266 0.0032916 2024-04-23T16:27:21.907 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:27:21.946 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:21 smithi103 ceph-mon[39629]: pgmap v815: 17 pgs: 17 active+clean; 373 MiB data, 2.4 GiB used, 892 GiB / 894 GiB avail; 56 MiB/s wr, 898 op/s 2024-04-23T16:27:21.946 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:21 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:27:21.946 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:21 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:27:21.946 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:21 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/507065735' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:27:21.949 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:21 smithi188 ceph-mon[44101]: pgmap v815: 17 pgs: 17 active+clean; 373 MiB data, 2.4 GiB used, 892 GiB / 894 GiB avail; 56 MiB/s wr, 898 op/s 2024-04-23T16:27:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:21 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:27:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:21 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:27:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:21 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/507065735' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:27:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:21 smithi148 ceph-mon[44520]: pgmap v815: 17 pgs: 17 active+clean; 373 MiB data, 2.4 GiB used, 892 GiB / 894 GiB avail; 56 MiB/s wr, 898 op/s 2024-04-23T16:27:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:21 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:27:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:21 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:27:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:21 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/507065735' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:27:22.326 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:27:22.505 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:27:22.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:22.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:27:22.611 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:27:22.649 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:22.677 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:27:22.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:22.682 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:27:22.783 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:27:22.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 5 16 23745 23729 296.585 268.312 0.0030631 0.00336966 2024-04-23T16:27:22.826 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:22 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/507065735' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-23T16:27:22.826 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:22 smithi188 ceph-mon[44101]: osdmap e283: 12 total, 12 up, 10 in 2024-04-23T16:27:22.829 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:22.859 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:27:22.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:22.863 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:27:22.874 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:27:22.875 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-04-23T16:27:22.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-04-23T16:27:22.875 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:27:22.875 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.875 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.14862.0:235583 2.8 e157/110 2:191024a3:::benchmark_data_smithi188_77627_object66234:head v 157'14825, mlcod=157'14823)", 2024-04-23T16:27:22.875 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:20:18.047697+0000", 2024-04-23T16:27:22.875 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 424.822600335, 2024-04-23T16:27:22.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.043764773, 2024-04-23T16:27:22.875 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.875 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.875 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.875 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.876 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.047697+0000", 2024-04-23T16:27:22.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.876 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.876 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.047697+0000", 2024-04-23T16:27:22.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.876 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.876 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.047700+0000", 2024-04-23T16:27:22.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8420000000000001e-06 2024-04-23T16:27:22.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.877 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.877 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.047717+0000", 2024-04-23T16:27:22.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6829000000000001e-05 2024-04-23T16:27:22.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.877 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.877 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.047736+0000", 2024-04-23T16:27:22.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9134999999999999e-05 2024-04-23T16:27:22.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.877 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.047748+0000", 2024-04-23T16:27:22.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1958e-05 2024-04-23T16:27:22.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.049550+0000", 2024-04-23T16:27:22.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018021090000000001 2024-04-23T16:27:22.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.049576+0000", 2024-04-23T16:27:22.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5596999999999999e-05 2024-04-23T16:27:22.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.879 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.091419+0000", 2024-04-23T16:27:22.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.041842611000000002 2024-04-23T16:27:22.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.879 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:20:18.091462+0000", 2024-04-23T16:27:22.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3692000000000001e-05 2024-04-23T16:27:22.879 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.879 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.879 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.880 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.15150.0:127632 3.2 e215/192 3:4a47de80:::benchmark_data_smithi188_89276_object127631:head v 215'7988, mlcod=215'7974)", 2024-04-23T16:27:22.880 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:19.876146+0000", 2024-04-23T16:27:22.880 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 242.994151332, 2024-04-23T16:27:22.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.037958249999999999, 2024-04-23T16:27:22.880 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.880 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.880 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.880 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.876146+0000", 2024-04-23T16:27:22.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.880 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.881 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.876146+0000", 2024-04-23T16:27:22.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.881 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.881 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.876147+0000", 2024-04-23T16:27:22.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9799999999999999e-07 2024-04-23T16:27:22.881 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.876189+0000", 2024-04-23T16:27:22.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2605999999999998e-05 2024-04-23T16:27:22.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.882 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.876194+0000", 2024-04-23T16:27:22.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3549999999999998e-06 2024-04-23T16:27:22.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.882 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.876197+0000", 2024-04-23T16:27:22.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9799999999999998e-06 2024-04-23T16:27:22.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.883 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.899849+0000", 2024-04-23T16:27:22.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023652672999999999 2024-04-23T16:27:22.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.883 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.899871+0000", 2024-04-23T16:27:22.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1792000000000001e-05 2024-04-23T16:27:22.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.883 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.914066+0000", 2024-04-23T16:27:22.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014194574 2024-04-23T16:27:22.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.914105+0000", 2024-04-23T16:27:22.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8971999999999999e-05 2024-04-23T16:27:22.884 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.884 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.884 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.884 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.15150.0:127638 3.2 e215/192 3:42b2e3a9:::benchmark_data_smithi188_89276_object127637:head v 215'7989, mlcod=215'7974)", 2024-04-23T16:27:22.884 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:19.876201+0000", 2024-04-23T16:27:22.884 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 242.994096869, 2024-04-23T16:27:22.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.037932050000000002, 2024-04-23T16:27:22.884 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.885 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.885 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.885 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.885 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.885 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.885 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.876201+0000", 2024-04-23T16:27:22.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.885 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.885 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.885 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.876201+0000", 2024-04-23T16:27:22.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.886 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.876201+0000", 2024-04-23T16:27:22.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3000000000000002e-07 2024-04-23T16:27:22.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.886 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.876276+0000", 2024-04-23T16:27:22.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4263000000000004e-05 2024-04-23T16:27:22.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.887 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.887 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.876281+0000", 2024-04-23T16:27:22.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4960000000000004e-06 2024-04-23T16:27:22.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.887 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.887 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.876285+0000", 2024-04-23T16:27:22.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6480000000000001e-06 2024-04-23T16:27:22.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.887 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.887 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.902508+0000", 2024-04-23T16:27:22.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026223311999999999 2024-04-23T16:27:22.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.902526+0000", 2024-04-23T16:27:22.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8003999999999999e-05 2024-04-23T16:27:22.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.914107+0000", 2024-04-23T16:27:22.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011581315999999999 2024-04-23T16:27:22.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.914133+0000", 2024-04-23T16:27:22.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5681e-05 2024-04-23T16:27:22.889 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.889 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.889 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.889 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.15150.0:127643 3.2 e215/192 3:4d1239c4:::benchmark_data_smithi188_89276_object127642:head v 215'7990, mlcod=215'7974)", 2024-04-23T16:27:22.889 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:23:19.876289+0000", 2024-04-23T16:27:22.889 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 242.99400871, 2024-04-23T16:27:22.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.037873549999999999, 2024-04-23T16:27:22.889 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.889 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.889 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.889 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.876289+0000", 2024-04-23T16:27:22.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.890 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.876289+0000", 2024-04-23T16:27:22.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.890 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.876289+0000", 2024-04-23T16:27:22.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6100000000000002e-07 2024-04-23T16:27:22.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.891 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.876333+0000", 2024-04-23T16:27:22.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3495999999999998e-05 2024-04-23T16:27:22.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.891 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.876337+0000", 2024-04-23T16:27:22.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5390000000000001e-06 2024-04-23T16:27:22.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.891 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.876340+0000", 2024-04-23T16:27:22.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8370000000000001e-06 2024-04-23T16:27:22.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.892 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.892 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.905661+0000", 2024-04-23T16:27:22.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029321047999999999 2024-04-23T16:27:22.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.892 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.892 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.905684+0000", 2024-04-23T16:27:22.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.262e-05 2024-04-23T16:27:22.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.892 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.892 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.914140+0000", 2024-04-23T16:27:22.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0084557409999999993 2024-04-23T16:27:22.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.893 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.893 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:23:19.914163+0000", 2024-04-23T16:27:22.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3008000000000001e-05 2024-04-23T16:27:22.893 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.893 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.893 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.893 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9614 4.2 4:4f13b2dc:::benchmark_data_smithi188_101370_object9613:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:27:22.893 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.494209+0000", 2024-04-23T16:27:22.893 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.3760888790000001, 2024-04-23T16:27:22.894 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.20416315199999999, 2024-04-23T16:27:22.894 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.894 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.894 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.894 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:22.894 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:27:22.894 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:27:22.894 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9614 2024-04-23T16:27:22.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.894 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.894 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.494209+0000", 2024-04-23T16:27:22.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.494209+0000", 2024-04-23T16:27:22.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.494216+0000", 2024-04-23T16:27:22.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6649999999999997e-06 2024-04-23T16:27:22.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.494245+0000", 2024-04-23T16:27:22.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9430000000000001e-05 2024-04-23T16:27:22.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.494262+0000", 2024-04-23T16:27:22.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6810999999999999e-05 2024-04-23T16:27:22.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.494275+0000", 2024-04-23T16:27:22.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3341e-05 2024-04-23T16:27:22.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.494337+0000", 2024-04-23T16:27:22.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2297000000000003e-05 2024-04-23T16:27:22.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.494486+0000", 2024-04-23T16:27:22.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000148887 2024-04-23T16:27:22.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:27:22.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.494645+0000", 2024-04-23T16:27:22.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015817899999999999 2024-04-23T16:27:22.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:27:22.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.495299+0000", 2024-04-23T16:27:22.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00065442099999999997 2024-04-23T16:27:22.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:27:22.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.698303+0000", 2024-04-23T16:27:22.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.20300422500000001 2024-04-23T16:27:22.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.698340+0000", 2024-04-23T16:27:22.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6795999999999998e-05 2024-04-23T16:27:22.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.899 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.899 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.698372+0000", 2024-04-23T16:27:22.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2100000000000001e-05 2024-04-23T16:27:22.899 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.899 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.899 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.899 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9636 4.2 4:420d7b21:::benchmark_data_smithi188_101370_object9635:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:27:22.899 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.498108+0000", 2024-04-23T16:27:22.899 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.3721896139999998, 2024-04-23T16:27:22.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.202126424, 2024-04-23T16:27:22.899 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.899 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.900 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.900 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:22.900 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:27:22.900 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:27:22.900 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9636 2024-04-23T16:27:22.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.900 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.498108+0000", 2024-04-23T16:27:22.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.498108+0000", 2024-04-23T16:27:22.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.498109+0000", 2024-04-23T16:27:22.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7899999999999997e-07 2024-04-23T16:27:22.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.498139+0000", 2024-04-23T16:27:22.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9977000000000001e-05 2024-04-23T16:27:22.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.498151+0000", 2024-04-23T16:27:22.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2261e-05 2024-04-23T16:27:22.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.498158+0000", 2024-04-23T16:27:22.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7619999999999998e-06 2024-04-23T16:27:22.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.498221+0000", 2024-04-23T16:27:22.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3248000000000002e-05 2024-04-23T16:27:22.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.498340+0000", 2024-04-23T16:27:22.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011917900000000001 2024-04-23T16:27:22.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:27:22.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.498485+0000", 2024-04-23T16:27:22.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000144563 2024-04-23T16:27:22.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:27:22.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.500232+0000", 2024-04-23T16:27:22.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001747391 2024-04-23T16:27:22.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:27:22.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700150+0000", 2024-04-23T16:27:22.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.19991759000000001 2024-04-23T16:27:22.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700193+0000", 2024-04-23T16:27:22.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2684e-05 2024-04-23T16:27:22.905 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.905 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.905 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700235+0000", 2024-04-23T16:27:22.905 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1990000000000003e-05 2024-04-23T16:27:22.905 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.905 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.905 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.905 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.905 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9642 4.2 4:4040f281:::benchmark_data_smithi188_101370_object9641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:27:22.905 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.499186+0000", 2024-04-23T16:27:22.905 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.3711119370000002, 2024-04-23T16:27:22.905 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.201217806, 2024-04-23T16:27:22.906 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.906 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.906 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.906 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:22.906 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:27:22.906 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:27:22.906 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9642 2024-04-23T16:27:22.906 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.906 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.906 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.906 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.906 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.499186+0000", 2024-04-23T16:27:22.906 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.906 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.907 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.907 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.499186+0000", 2024-04-23T16:27:22.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.907 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.907 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.907 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.499187+0000", 2024-04-23T16:27:22.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7000000000000004e-07 2024-04-23T16:27:22.907 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.907 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.907 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.499231+0000", 2024-04-23T16:27:22.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.464e-05 2024-04-23T16:27:22.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.499238+0000", 2024-04-23T16:27:22.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.075e-06 2024-04-23T16:27:22.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.499245+0000", 2024-04-23T16:27:22.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4540000000000004e-06 2024-04-23T16:27:22.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.499433+0000", 2024-04-23T16:27:22.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018828200000000001 2024-04-23T16:27:22.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.499543+0000", 2024-04-23T16:27:22.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110109 2024-04-23T16:27:22.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:27:22.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.499661+0000", 2024-04-23T16:27:22.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118144 2024-04-23T16:27:22.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:27:22.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.500369+0000", 2024-04-23T16:27:22.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070764199999999997 2024-04-23T16:27:22.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:27:22.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700291+0000", 2024-04-23T16:27:22.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.19992175200000001 2024-04-23T16:27:22.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700362+0000", 2024-04-23T16:27:22.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0798999999999998e-05 2024-04-23T16:27:22.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700404+0000", 2024-04-23T16:27:22.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2138999999999998e-05 2024-04-23T16:27:22.911 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.911 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.911 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.912 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9678 4.2 4:4327109c:::benchmark_data_smithi188_101370_object9677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:27:22.912 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.506291+0000", 2024-04-23T16:27:22.912 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.3640070240000002, 2024-04-23T16:27:22.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.19453815399999999, 2024-04-23T16:27:22.912 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.912 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.912 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.912 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:22.912 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:27:22.912 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:27:22.912 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9678 2024-04-23T16:27:22.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.912 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.506291+0000", 2024-04-23T16:27:22.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.506291+0000", 2024-04-23T16:27:22.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.506293+0000", 2024-04-23T16:27:22.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1900000000000002e-06 2024-04-23T16:27:22.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.506316+0000", 2024-04-23T16:27:22.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3050000000000001e-05 2024-04-23T16:27:22.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.506330+0000", 2024-04-23T16:27:22.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3499000000000001e-05 2024-04-23T16:27:22.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.506338+0000", 2024-04-23T16:27:22.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2919999999999996e-06 2024-04-23T16:27:22.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.506427+0000", 2024-04-23T16:27:22.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8792e-05 2024-04-23T16:27:22.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.506566+0000", 2024-04-23T16:27:22.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000139157 2024-04-23T16:27:22.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:27:22.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.506718+0000", 2024-04-23T16:27:22.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015196400000000001 2024-04-23T16:27:22.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:27:22.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.507457+0000", 2024-04-23T16:27:22.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00073881100000000002 2024-04-23T16:27:22.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:27:22.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700756+0000", 2024-04-23T16:27:22.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.193299309 2024-04-23T16:27:22.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700789+0000", 2024-04-23T16:27:22.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2954000000000003e-05 2024-04-23T16:27:22.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700829+0000", 2024-04-23T16:27:22.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0136000000000003e-05 2024-04-23T16:27:22.917 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.917 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.917 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.917 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9706 4.2 4:4f243e4e:::benchmark_data_smithi188_101370_object9705:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:27:22.917 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.511210+0000", 2024-04-23T16:27:22.918 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.3590875809999998, 2024-04-23T16:27:22.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18977317099999999, 2024-04-23T16:27:22.918 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.918 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.918 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.918 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:22.918 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:27:22.918 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:27:22.918 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9706 2024-04-23T16:27:22.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.918 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.511210+0000", 2024-04-23T16:27:22.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.511210+0000", 2024-04-23T16:27:22.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.511213+0000", 2024-04-23T16:27:22.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4320000000000002e-06 2024-04-23T16:27:22.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.511239+0000", 2024-04-23T16:27:22.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6143999999999999e-05 2024-04-23T16:27:22.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.511251+0000", 2024-04-23T16:27:22.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2251e-05 2024-04-23T16:27:22.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.511260+0000", 2024-04-23T16:27:22.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.49e-06 2024-04-23T16:27:22.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.511330+0000", 2024-04-23T16:27:22.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0468000000000004e-05 2024-04-23T16:27:22.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.511479+0000", 2024-04-23T16:27:22.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000148937 2024-04-23T16:27:22.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:27:22.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.511664+0000", 2024-04-23T16:27:22.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018490999999999999 2024-04-23T16:27:22.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:27:22.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.512361+0000", 2024-04-23T16:27:22.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069734400000000002 2024-04-23T16:27:22.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:27:22.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700884+0000", 2024-04-23T16:27:22.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18852321499999999 2024-04-23T16:27:22.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700948+0000", 2024-04-23T16:27:22.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3482e-05 2024-04-23T16:27:22.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700983+0000", 2024-04-23T16:27:22.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5497999999999998e-05 2024-04-23T16:27:22.923 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.923 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.923 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.923 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9717 4.2 4:43e0c227:::benchmark_data_smithi188_101370_object9716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:27:22.923 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.514316+0000", 2024-04-23T16:27:22.924 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.355982107, 2024-04-23T16:27:22.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18925007299999999, 2024-04-23T16:27:22.924 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.924 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.924 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.924 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:22.924 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:27:22.924 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:27:22.924 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9717 2024-04-23T16:27:22.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.924 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.514316+0000", 2024-04-23T16:27:22.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.514316+0000", 2024-04-23T16:27:22.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.514318+0000", 2024-04-23T16:27:22.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0140000000000001e-06 2024-04-23T16:27:22.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.514343+0000", 2024-04-23T16:27:22.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5795e-05 2024-04-23T16:27:22.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.514359+0000", 2024-04-23T16:27:22.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5460000000000001e-05 2024-04-23T16:27:22.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.514368+0000", 2024-04-23T16:27:22.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2299999999999997e-06 2024-04-23T16:27:22.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.514416+0000", 2024-04-23T16:27:22.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8299e-05 2024-04-23T16:27:22.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.514509+0000", 2024-04-23T16:27:22.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2499999999999999e-05 2024-04-23T16:27:22.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:27:22.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.514594+0000", 2024-04-23T16:27:22.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4890999999999997e-05 2024-04-23T16:27:22.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:27:22.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515958+0000", 2024-04-23T16:27:22.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001363733 2024-04-23T16:27:22.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:27:22.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703457+0000", 2024-04-23T16:27:22.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.187498991 2024-04-23T16:27:22.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703515+0000", 2024-04-23T16:27:22.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8575000000000001e-05 2024-04-23T16:27:22.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.929 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703566+0000", 2024-04-23T16:27:22.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0584999999999998e-05 2024-04-23T16:27:22.929 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.929 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.929 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.929 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9719 4.2 4:4ce29c52:::benchmark_data_smithi188_101370_object9718:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:27:22.929 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.515092+0000", 2024-04-23T16:27:22.929 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.35520591, 2024-04-23T16:27:22.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18860664399999999, 2024-04-23T16:27:22.929 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.930 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.930 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.930 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:22.930 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:27:22.930 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:27:22.930 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9719 2024-04-23T16:27:22.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.930 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515092+0000", 2024-04-23T16:27:22.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515092+0000", 2024-04-23T16:27:22.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515096+0000", 2024-04-23T16:27:22.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6129999999999997e-06 2024-04-23T16:27:22.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515143+0000", 2024-04-23T16:27:22.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6971999999999997e-05 2024-04-23T16:27:22.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515155+0000", 2024-04-23T16:27:22.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1296e-05 2024-04-23T16:27:22.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515161+0000", 2024-04-23T16:27:22.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7299999999999999e-06 2024-04-23T16:27:22.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515211+0000", 2024-04-23T16:27:22.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9413000000000003e-05 2024-04-23T16:27:22.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515338+0000", 2024-04-23T16:27:22.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012697199999999999 2024-04-23T16:27:22.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:27:22.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515484+0000", 2024-04-23T16:27:22.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014637999999999999 2024-04-23T16:27:22.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:27:22.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.516155+0000", 2024-04-23T16:27:22.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067116799999999998 2024-04-23T16:27:22.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:27:22.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703628+0000", 2024-04-23T16:27:22.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18747253899999999 2024-04-23T16:27:22.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703666+0000", 2024-04-23T16:27:22.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7966000000000002e-05 2024-04-23T16:27:22.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703699+0000", 2024-04-23T16:27:22.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2595000000000002e-05 2024-04-23T16:27:22.935 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.935 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.935 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.935 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9737 4.2 4:4045b26d:::benchmark_data_smithi188_101370_object9736:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:27:22.935 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.520122+0000", 2024-04-23T16:27:22.935 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.3501755069999999, 2024-04-23T16:27:22.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.183681758, 2024-04-23T16:27:22.935 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.936 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.936 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.936 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:22.936 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:27:22.936 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:27:22.936 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9737 2024-04-23T16:27:22.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.936 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.520122+0000", 2024-04-23T16:27:22.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.520122+0000", 2024-04-23T16:27:22.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.520125+0000", 2024-04-23T16:27:22.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7269999999999999e-06 2024-04-23T16:27:22.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.520180+0000", 2024-04-23T16:27:22.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4645999999999998e-05 2024-04-23T16:27:22.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.520201+0000", 2024-04-23T16:27:22.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1299999999999999e-05 2024-04-23T16:27:22.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.520213+0000", 2024-04-23T16:27:22.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.163e-05 2024-04-23T16:27:22.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.520284+0000", 2024-04-23T16:27:22.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1364000000000003e-05 2024-04-23T16:27:22.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.520417+0000", 2024-04-23T16:27:22.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013339800000000001 2024-04-23T16:27:22.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:27:22.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.520520+0000", 2024-04-23T16:27:22.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000103069 2024-04-23T16:27:22.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:27:22.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.521128+0000", 2024-04-23T16:27:22.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00060738999999999997 2024-04-23T16:27:22.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:27:22.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703738+0000", 2024-04-23T16:27:22.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18261016399999999 2024-04-23T16:27:22.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703768+0000", 2024-04-23T16:27:22.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.972e-05 2024-04-23T16:27:22.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703804+0000", 2024-04-23T16:27:22.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6350000000000003e-05 2024-04-23T16:27:22.941 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.941 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.941 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.941 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9757 4.2 4:4e85d964:::benchmark_data_smithi188_101370_object9756:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:27:22.941 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.526315+0000", 2024-04-23T16:27:22.941 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.3439826200000002, 2024-04-23T16:27:22.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.177584518, 2024-04-23T16:27:22.941 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.941 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.941 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.941 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:22.942 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:27:22.942 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:27:22.942 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9757 2024-04-23T16:27:22.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.942 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.526315+0000", 2024-04-23T16:27:22.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.526315+0000", 2024-04-23T16:27:22.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.526317+0000", 2024-04-23T16:27:22.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1409999999999999e-06 2024-04-23T16:27:22.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.526364+0000", 2024-04-23T16:27:22.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7074999999999999e-05 2024-04-23T16:27:22.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.526380+0000", 2024-04-23T16:27:22.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5789000000000001e-05 2024-04-23T16:27:22.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.526389+0000", 2024-04-23T16:27:22.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.602e-06 2024-04-23T16:27:22.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.526439+0000", 2024-04-23T16:27:22.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0466000000000001e-05 2024-04-23T16:27:22.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.526523+0000", 2024-04-23T16:27:22.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3437999999999996e-05 2024-04-23T16:27:22.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:27:22.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.526609+0000", 2024-04-23T16:27:22.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6278000000000003e-05 2024-04-23T16:27:22.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:27:22.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527210+0000", 2024-04-23T16:27:22.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00060086700000000005 2024-04-23T16:27:22.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:27:22.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703840+0000", 2024-04-23T16:27:22.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.176630227 2024-04-23T16:27:22.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703873+0000", 2024-04-23T16:27:22.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3002e-05 2024-04-23T16:27:22.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703900+0000", 2024-04-23T16:27:22.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6633000000000001e-05 2024-04-23T16:27:22.947 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.947 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.947 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.947 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9762 4.2 4:44b975ec:::benchmark_data_smithi188_101370_object9761:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:27:22.947 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.527254+0000", 2024-04-23T16:27:22.947 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.3430437419999999, 2024-04-23T16:27:22.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.176807251, 2024-04-23T16:27:22.947 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.947 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.947 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.948 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:22.948 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:27:22.948 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:27:22.948 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9762 2024-04-23T16:27:22.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.948 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527254+0000", 2024-04-23T16:27:22.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527254+0000", 2024-04-23T16:27:22.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527256+0000", 2024-04-23T16:27:22.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.61e-06 2024-04-23T16:27:22.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527305+0000", 2024-04-23T16:27:22.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9036000000000003e-05 2024-04-23T16:27:22.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527316+0000", 2024-04-23T16:27:22.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0879e-05 2024-04-23T16:27:22.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527327+0000", 2024-04-23T16:27:22.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1083999999999999e-05 2024-04-23T16:27:22.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527382+0000", 2024-04-23T16:27:22.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5754000000000002e-05 2024-04-23T16:27:22.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.951 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527478+0000", 2024-04-23T16:27:22.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5166000000000003e-05 2024-04-23T16:27:22.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:27:22.951 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527577+0000", 2024-04-23T16:27:22.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9499000000000004e-05 2024-04-23T16:27:22.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:27:22.951 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.528059+0000", 2024-04-23T16:27:22.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048219199999999999 2024-04-23T16:27:22.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:27:22.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703969+0000", 2024-04-23T16:27:22.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17590968200000001 2024-04-23T16:27:22.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704024+0000", 2024-04-23T16:27:22.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5161999999999998e-05 2024-04-23T16:27:22.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704061+0000", 2024-04-23T16:27:22.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7187000000000002e-05 2024-04-23T16:27:22.953 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.953 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.953 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.953 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9774 4.2 4:4c2552b2:::benchmark_data_smithi188_101370_object9773:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:27:22.953 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.530865+0000", 2024-04-23T16:27:22.953 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.3394330000000001, 2024-04-23T16:27:22.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17334628899999999, 2024-04-23T16:27:22.953 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.953 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.953 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.953 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:22.953 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:27:22.954 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:27:22.954 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9774 2024-04-23T16:27:22.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.954 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.530865+0000", 2024-04-23T16:27:22.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.530865+0000", 2024-04-23T16:27:22.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.530867+0000", 2024-04-23T16:27:22.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9360000000000002e-06 2024-04-23T16:27:22.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.530910+0000", 2024-04-23T16:27:22.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2886000000000002e-05 2024-04-23T16:27:22.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.530922+0000", 2024-04-23T16:27:22.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2036e-05 2024-04-23T16:27:22.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.530928+0000", 2024-04-23T16:27:22.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6320000000000002e-06 2024-04-23T16:27:22.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.530982+0000", 2024-04-23T16:27:22.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3974000000000002e-05 2024-04-23T16:27:22.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.531080+0000", 2024-04-23T16:27:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8171000000000006e-05 2024-04-23T16:27:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:27:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.531178+0000", 2024-04-23T16:27:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7570000000000003e-05 2024-04-23T16:27:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:27:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.531768+0000", 2024-04-23T16:27:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00058953499999999997 2024-04-23T16:27:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:27:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704122+0000", 2024-04-23T16:27:22.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17235446900000001 2024-04-23T16:27:22.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704172+0000", 2024-04-23T16:27:22.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9787000000000003e-05 2024-04-23T16:27:22.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704211+0000", 2024-04-23T16:27:22.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9292999999999998e-05 2024-04-23T16:27:22.958 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.958 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.959 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.959 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9793 4.2 4:48f6dcf1:::benchmark_data_smithi188_101370_object9792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:27:22.959 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.537491+0000", 2024-04-23T16:27:22.959 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.3328066010000001, 2024-04-23T16:27:22.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.16686599799999999, 2024-04-23T16:27:22.959 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.959 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.959 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.959 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:22.959 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:27:22.959 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:27:22.960 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9793 2024-04-23T16:27:22.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.960 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.960 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.537491+0000", 2024-04-23T16:27:22.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.537491+0000", 2024-04-23T16:27:22.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.537492+0000", 2024-04-23T16:27:22.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.175e-06 2024-04-23T16:27:22.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.537510+0000", 2024-04-23T16:27:22.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7261999999999998e-05 2024-04-23T16:27:22.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.537520+0000", 2024-04-23T16:27:22.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0305e-05 2024-04-23T16:27:22.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.537528+0000", 2024-04-23T16:27:22.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8499999999999994e-06 2024-04-23T16:27:22.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.537580+0000", 2024-04-23T16:27:22.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2641e-05 2024-04-23T16:27:22.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.537709+0000", 2024-04-23T16:27:22.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012869499999999999 2024-04-23T16:27:22.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:27:22.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.537853+0000", 2024-04-23T16:27:22.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014435199999999999 2024-04-23T16:27:22.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:27:22.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.538386+0000", 2024-04-23T16:27:22.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053206100000000001 2024-04-23T16:27:22.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:27:22.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704280+0000", 2024-04-23T16:27:22.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.165894508 2024-04-23T16:27:22.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704328+0000", 2024-04-23T16:27:22.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8025999999999997e-05 2024-04-23T16:27:22.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704357+0000", 2024-04-23T16:27:22.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9122999999999999e-05 2024-04-23T16:27:22.965 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.965 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.965 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.965 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9797 4.2 4:483b6d3f:::benchmark_data_smithi188_101370_object9796:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:27:22.965 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.539175+0000", 2024-04-23T16:27:22.965 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.3311228719999999, 2024-04-23T16:27:22.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.16532619300000001, 2024-04-23T16:27:22.965 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.965 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.966 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.966 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:22.966 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:27:22.966 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:27:22.966 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9797 2024-04-23T16:27:22.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.966 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539175+0000", 2024-04-23T16:27:22.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539175+0000", 2024-04-23T16:27:22.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539176+0000", 2024-04-23T16:27:22.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2049999999999999e-06 2024-04-23T16:27:22.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539195+0000", 2024-04-23T16:27:22.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8383999999999999e-05 2024-04-23T16:27:22.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539203+0000", 2024-04-23T16:27:22.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6780000000000006e-06 2024-04-23T16:27:22.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539209+0000", 2024-04-23T16:27:22.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5380000000000002e-06 2024-04-23T16:27:22.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539256+0000", 2024-04-23T16:27:22.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7411999999999999e-05 2024-04-23T16:27:22.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539370+0000", 2024-04-23T16:27:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000113519 2024-04-23T16:27:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:27:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539523+0000", 2024-04-23T16:27:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000153194 2024-04-23T16:27:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:27:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539941+0000", 2024-04-23T16:27:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00041838500000000001 2024-04-23T16:27:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:27:22.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704408+0000", 2024-04-23T16:27:22.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.16446627899999999 2024-04-23T16:27:22.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704463+0000", 2024-04-23T16:27:22.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5241999999999997e-05 2024-04-23T16:27:22.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704501+0000", 2024-04-23T16:27:22.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8356999999999999e-05 2024-04-23T16:27:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9807 4.2 4:4cb83167:::benchmark_data_smithi188_101370_object9806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:27:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.543468+0000", 2024-04-23T16:27:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.326830078, 2024-04-23T16:27:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.16123652299999999, 2024-04-23T16:27:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:27:22.972 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:27:22.972 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9807 2024-04-23T16:27:22.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.972 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.543468+0000", 2024-04-23T16:27:22.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.543468+0000", 2024-04-23T16:27:22.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.543469+0000", 2024-04-23T16:27:22.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.282e-06 2024-04-23T16:27:22.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.543484+0000", 2024-04-23T16:27:22.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.484e-05 2024-04-23T16:27:22.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.543493+0000", 2024-04-23T16:27:22.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9789999999999999e-06 2024-04-23T16:27:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.543498+0000", 2024-04-23T16:27:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6030000000000004e-06 2024-04-23T16:27:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.543545+0000", 2024-04-23T16:27:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6105e-05 2024-04-23T16:27:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.543678+0000", 2024-04-23T16:27:22.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013313600000000001 2024-04-23T16:27:22.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:27:22.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.543827+0000", 2024-04-23T16:27:22.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014967800000000001 2024-04-23T16:27:22.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:27:22.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.544302+0000", 2024-04-23T16:27:22.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047469599999999998 2024-04-23T16:27:22.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:27:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704574+0000", 2024-04-23T16:27:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.160272426 2024-04-23T16:27:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704630+0000", 2024-04-23T16:27:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5161999999999998e-05 2024-04-23T16:27:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704704+0000", 2024-04-23T16:27:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4616000000000005e-05 2024-04-23T16:27:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.977 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.977 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9825 4.2 4:4e8045d9:::benchmark_data_smithi188_101370_object9824:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:27:22.977 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.554216+0000", 2024-04-23T16:27:22.977 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.3160819410000002, 2024-04-23T16:27:22.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.15061554099999999, 2024-04-23T16:27:22.977 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.977 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.977 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.977 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:22.977 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:27:22.977 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:27:22.977 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9825 2024-04-23T16:27:22.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.978 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554216+0000", 2024-04-23T16:27:22.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554216+0000", 2024-04-23T16:27:22.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554218+0000", 2024-04-23T16:27:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.826e-06 2024-04-23T16:27:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554235+0000", 2024-04-23T16:27:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7374e-05 2024-04-23T16:27:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554245+0000", 2024-04-23T16:27:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7860000000000008e-06 2024-04-23T16:27:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554251+0000", 2024-04-23T16:27:22.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6620000000000002e-06 2024-04-23T16:27:22.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554296+0000", 2024-04-23T16:27:22.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5716000000000001e-05 2024-04-23T16:27:22.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554377+0000", 2024-04-23T16:27:22.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1083999999999998e-05 2024-04-23T16:27:22.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:27:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554462+0000", 2024-04-23T16:27:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4827999999999995e-05 2024-04-23T16:27:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:27:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554901+0000", 2024-04-23T16:27:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043888800000000002 2024-04-23T16:27:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:27:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704750+0000", 2024-04-23T16:27:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14984945499999999 2024-04-23T16:27:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704795+0000", 2024-04-23T16:27:22.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4477000000000001e-05 2024-04-23T16:27:22.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704831+0000", 2024-04-23T16:27:22.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6445000000000001e-05 2024-04-23T16:27:22.982 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.982 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.982 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.982 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9839 4.2 4:41a36203:::benchmark_data_smithi188_101370_object9838:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:27:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.576043+0000", 2024-04-23T16:27:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.2942551419999999, 2024-04-23T16:27:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.129074616, 2024-04-23T16:27:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:27:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:27:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:27:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9839 2024-04-23T16:27:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576043+0000", 2024-04-23T16:27:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576043+0000", 2024-04-23T16:27:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576044+0000", 2024-04-23T16:27:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.28e-06 2024-04-23T16:27:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:22.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576081+0000", 2024-04-23T16:27:22.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7279e-05 2024-04-23T16:27:22.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:22.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576091+0000", 2024-04-23T16:27:22.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9399999999999997e-06 2024-04-23T16:27:22.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:22.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576105+0000", 2024-04-23T16:27:22.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3919e-05 2024-04-23T16:27:22.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:22.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576146+0000", 2024-04-23T16:27:22.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0756000000000001e-05 2024-04-23T16:27:22.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:27:22.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576226+0000", 2024-04-23T16:27:22.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9969999999999995e-05 2024-04-23T16:27:22.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:27:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576309+0000", 2024-04-23T16:27:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2930999999999996e-05 2024-04-23T16:27:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:27:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576755+0000", 2024-04-23T16:27:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044604099999999999 2024-04-23T16:27:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:27:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.705043+0000", 2024-04-23T16:27:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.12828847199999999 2024-04-23T16:27:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:27:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.705082+0000", 2024-04-23T16:27:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8763999999999997e-05 2024-04-23T16:27:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.705117+0000", 2024-04-23T16:27:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5264e-05 2024-04-23T16:27:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:22.989 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:27:22.989 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:27:23.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:22 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/507065735' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-23T16:27:23.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:22 smithi103 ceph-mon[39629]: osdmap e283: 12 total, 12 up, 10 in 2024-04-23T16:27:23.027 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:23.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:22 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/507065735' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-23T16:27:23.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:22 smithi148 ceph-mon[44520]: osdmap e283: 12 total, 12 up, 10 in 2024-04-23T16:27:23.608 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-23T16:27:23.817 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 6 16 27617 27601 287.484 242 0.00296643 0.00347562 2024-04-23T16:27:23.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:23 smithi188 ceph-mon[44101]: pgmap v817: 17 pgs: 17 active+clean; 647 MiB data, 2.8 GiB used, 891 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-23T16:27:24.060 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:27:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:23 smithi103 ceph-mon[39629]: pgmap v817: 17 pgs: 17 active+clean; 647 MiB data, 2.8 GiB used, 891 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-23T16:27:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:23 smithi148 ceph-mon[44520]: pgmap v817: 17 pgs: 17 active+clean; 647 MiB data, 2.8 GiB used, 891 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-23T16:27:24.818 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 7 16 31246 31230 278.814 226.812 0.00417103 0.00358421 2024-04-23T16:27:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:25 smithi103 ceph-mon[39629]: osdmap e284: 12 total, 12 up, 10 in 2024-04-23T16:27:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:25 smithi103 ceph-mon[39629]: pgmap v819: 17 pgs: 17 active+clean; 1.4 GiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 243 MiB/s wr, 3.88k op/s 2024-04-23T16:27:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:25 smithi148 ceph-mon[44520]: osdmap e284: 12 total, 12 up, 10 in 2024-04-23T16:27:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:25 smithi148 ceph-mon[44520]: pgmap v819: 17 pgs: 17 active+clean; 1.4 GiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 243 MiB/s wr, 3.88k op/s 2024-04-23T16:27:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:25 smithi188 ceph-mon[44101]: osdmap e284: 12 total, 12 up, 10 in 2024-04-23T16:27:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:25 smithi188 ceph-mon[44101]: pgmap v819: 17 pgs: 17 active+clean; 1.4 GiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 243 MiB/s wr, 3.88k op/s 2024-04-23T16:27:25.693 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:27:25.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:25.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:27:25.798 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:27:25.819 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 8 16 34781 34765 271.542 220.938 0.00690798 0.00367997 2024-04-23T16:27:25.845 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:25.876 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:27:25.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:25.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:27:25.982 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:27:26.025 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:26.054 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:27:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:26.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:27:26.159 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:27:26.197 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:26.222 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:27:26.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:26.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:27:26.327 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:27:26.365 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:26.390 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:27:26.411 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:27:26.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:26.417 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:27:26.518 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:27:26.553 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:26.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:26 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4013482239' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:27:26.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:26 smithi188 ceph-mon[44101]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:27:26.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:26 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:27:26.679 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:27:26.819 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 9 15 38108 38093 264.481 208 0.00819232 0.00377878 2024-04-23T16:27:26.907 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:27:26.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:26.911 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:27:27.012 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:27:27.071 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:27.090 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:26 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4013482239' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:27:27.091 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:26 smithi103 ceph-mon[39629]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:27:27.091 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:26 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:27:27.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:26 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4013482239' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:27:27.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:26 smithi148 ceph-mon[44520]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:27:27.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:26 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:27:27.131 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:27:27.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:27.136 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:27:27.213 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:27:27.213 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:27:27.214 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-04-23T16:27:27.214 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:27:27.237 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:27:27.277 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:27.301 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:27:27.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:27.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:27:27.408 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:27:27.447 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:27.477 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:27:27.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:27.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:27:27.581 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:27:27.618 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:27.646 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:27:27.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:27.658 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:27:27.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:27 smithi188 ceph-mon[44101]: pgmap v820: 17 pgs: 17 active+clean; 1.9 GiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 239 MiB/s wr, 3.83k op/s 2024-04-23T16:27:27.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:27 smithi188 ceph-mon[44101]: from='client.35432 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:27:27.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:27 smithi188 ceph-mon[44101]: osdmap e285: 12 total, 12 up, 10 in 2024-04-23T16:27:27.758 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:27:27.799 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:27.819 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 10 16 41316 41300 258.076 200.438 0.00672839 0.003873 2024-04-23T16:27:27.985 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-23T16:27:28.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:27 smithi103 ceph-mon[39629]: pgmap v820: 17 pgs: 17 active+clean; 1.9 GiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 239 MiB/s wr, 3.83k op/s 2024-04-23T16:27:28.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:27 smithi103 ceph-mon[39629]: from='client.35432 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:27:28.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:27 smithi103 ceph-mon[39629]: osdmap e285: 12 total, 12 up, 10 in 2024-04-23T16:27:28.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:27 smithi148 ceph-mon[44520]: pgmap v820: 17 pgs: 17 active+clean; 1.9 GiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 239 MiB/s wr, 3.83k op/s 2024-04-23T16:27:28.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:27 smithi148 ceph-mon[44520]: from='client.35432 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:27:28.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:27 smithi148 ceph-mon[44520]: osdmap e285: 12 total, 12 up, 10 in 2024-04-23T16:27:28.370 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:27:28.819 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 11 16 44431 44415 252.312 194.688 0.00374561 0.00396174 2024-04-23T16:27:29.257 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:27:29.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:29.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:27:29.325 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:27:29.352 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:27:29.364 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:27:29.379 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:27:29.404 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:27:29.408 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:29.433 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:27:29.434 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:27:29.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:29.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:27:29.463 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:27:29.490 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:27:29.522 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:27:29.538 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:27:29.552 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:27:29.576 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:29.582 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:27:29.613 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:27:29.641 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:27:29.644 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:27:29.644 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:27:29.644 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:27:29.644 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:27:29.644 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:27:29.644 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:27:29.645 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:27:29.645 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:27:29.645 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:27:29.645 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:27:29.645 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:27:29.645 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:27:29.683 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:29 smithi148 ceph-mon[44520]: pgmap v822: 17 pgs: 17 active+clean; 2.0 GiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 195 MiB/s wr, 3.13k op/s 2024-04-23T16:27:29.683 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:29 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3302643948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:27:29.683 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:29 smithi188 ceph-mon[44101]: pgmap v822: 17 pgs: 17 active+clean; 2.0 GiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 195 MiB/s wr, 3.13k op/s 2024-04-23T16:27:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:29 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3302643948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:27:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:29.782 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:27:29.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:29.786 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:27:29.819 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 12 15 47477 47462 247.155 190.438 0.00801325 0.00404379 2024-04-23T16:27:29.886 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:27:29.926 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:30.073 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:29 smithi103 ceph-mon[39629]: pgmap v822: 17 pgs: 17 active+clean; 2.0 GiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 195 MiB/s wr, 3.13k op/s 2024-04-23T16:27:30.073 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:29 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3302643948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:27:30.073 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.132 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:27:30.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:30.137 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:27:30.238 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:27:30.282 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:30.313 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:27:30.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:30.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:27:30.418 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:27:30.460 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:30.479 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:27:30.479 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:27:30.487 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:27:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:30.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:27:30.591 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:27:30.637 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:30.671 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:27:30.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:30.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:27:30.708 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10, 1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1] 2024-04-23T16:27:30.708 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:27:30.708 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-23T16:27:30.708 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-23T16:27:30.774 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:27:30.778 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:27:30.819 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 13 16 50286 50270 241.643 175.5 0.0033711 0.00413629 2024-04-23T16:27:30.823 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.825 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.825 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.825 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.825 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.825 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.825 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.825 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.825 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.825 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.826 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.837 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:30.887 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:30.922 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:27:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:30.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:27:31.039 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:27:31.076 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:31.114 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:27:31.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:31.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:31.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:31.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:31.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:31.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:31.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:31.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:31.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:31.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:31.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:31.156 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:27:31.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:31.163 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:27:31.198 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-23T16:27:31.264 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:27:31.308 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:31.342 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:27:31.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:31.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:27:31.448 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:27:31.490 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:31.534 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:27:31.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:31.537 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:27:31.633 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:27:31.638 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:27:31.685 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:31.729 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:27:31.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:31.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:27:31.819 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 14 16 52889 52873 236.003 162.688 0.00721478 0.00423598 2024-04-23T16:27:31.836 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:27:31.885 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:31.910 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:31 smithi188 ceph-mon[44101]: pgmap v823: 17 pgs: 17 active+clean; 2.7 GiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 259 MiB/s wr, 4.14k op/s 2024-04-23T16:27:31.919 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:27:31.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:31.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:27:31.957 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:27:32.031 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:27:32.068 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:31 smithi103 ceph-mon[39629]: pgmap v823: 17 pgs: 17 active+clean; 2.7 GiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 259 MiB/s wr, 4.14k op/s 2024-04-23T16:27:32.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:31 smithi148 ceph-mon[44520]: pgmap v823: 17 pgs: 17 active+clean; 2.7 GiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 259 MiB/s wr, 4.14k op/s 2024-04-23T16:27:32.557 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:27:32.819 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 15 16 55589 55573 231.518 168.75 0.00905906 0.00431698 2024-04-23T16:27:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:32 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:27:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:32 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:27:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:32 smithi103 ceph-mon[39629]: osdmap e286: 12 total, 12 up, 10 in 2024-04-23T16:27:32.904 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:32 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:27:32.905 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:32 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:27:32.905 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:32 smithi188 ceph-mon[44101]: osdmap e286: 12 total, 12 up, 10 in 2024-04-23T16:27:32.923 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:27:32.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:32.927 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:27:33.028 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:27:33.064 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:32 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:27:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:32 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:27:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:32 smithi148 ceph-mon[44520]: osdmap e286: 12 total, 12 up, 10 in 2024-04-23T16:27:33.520 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:27:33.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:33.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:27:33.583 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:27:33.583 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:27:33.624 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:27:33.672 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:33.705 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:27:33.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:33.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:27:33.811 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:27:33.819 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 16 16 58210 58194 227.286 163.812 0.0099432 0.00439806 2024-04-23T16:27:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:33 smithi103 ceph-mon[39629]: pgmap v825: 17 pgs: 17 active+clean; 2.9 GiB data, 7.5 GiB used, 887 GiB / 894 GiB avail; 185 MiB/s wr, 2.95k op/s 2024-04-23T16:27:33.854 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:33.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:33 smithi188 ceph-mon[44101]: pgmap v825: 17 pgs: 17 active+clean; 2.9 GiB data, 7.5 GiB used, 887 GiB / 894 GiB avail; 185 MiB/s wr, 2.95k op/s 2024-04-23T16:27:33.986 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:27:34.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:33 smithi148 ceph-mon[44520]: pgmap v825: 17 pgs: 17 active+clean; 2.9 GiB data, 7.5 GiB used, 887 GiB / 894 GiB avail; 185 MiB/s wr, 2.95k op/s 2024-04-23T16:27:34.400 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-23T16:27:34.416 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:27:34.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:34.419 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:27:34.520 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:27:34.574 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:34.599 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:27:34.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:34.603 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:27:34.704 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:27:34.739 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:34.768 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:27:34.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:34.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:27:34.798 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:27:34.819 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 17 16 60703 60687 223.082 155.812 0.0100647 0.00448131 2024-04-23T16:27:34.874 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:27:34.911 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:35.066 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:27:35.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:35.073 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:27:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:35 smithi103 ceph-mon[39629]: osdmap e287: 12 total, 12 up, 10 in 2024-04-23T16:27:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:35 smithi103 ceph-mon[39629]: pgmap v827: 17 pgs: 17 active+clean; 3.4 GiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 194 MiB/s wr, 3.11k op/s 2024-04-23T16:27:35.174 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:27:35.221 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:35.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:35 smithi188 ceph-mon[44101]: osdmap e287: 12 total, 12 up, 10 in 2024-04-23T16:27:35.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:35 smithi188 ceph-mon[44101]: pgmap v827: 17 pgs: 17 active+clean; 3.4 GiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 194 MiB/s wr, 3.11k op/s 2024-04-23T16:27:35.278 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:27:35.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:35.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:27:35.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:35 smithi148 ceph-mon[44520]: osdmap e287: 12 total, 12 up, 10 in 2024-04-23T16:27:35.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:35 smithi148 ceph-mon[44520]: pgmap v827: 17 pgs: 17 active+clean; 3.4 GiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 194 MiB/s wr, 3.11k op/s 2024-04-23T16:27:35.383 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:27:35.430 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:35.457 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:27:35.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:35.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:27:35.564 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:27:35.604 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:35.638 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:27:35.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:35.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:27:35.742 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:27:35.790 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:35.819 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 18 16 63139 63123 219.146 152.25 0.00568927 0.00456124 2024-04-23T16:27:35.824 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:27:35.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:35.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:27:35.929 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:27:35.968 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:36.176 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:27:36.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:36.182 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:27:36.283 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:27:36.336 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:36.365 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:27:36.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:36.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:27:36.470 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:27:36.513 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:36.606 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:27:36.609 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2403379480' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:27:36.609 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:36 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:27:36.820 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 19 16 65457 65441 215.237 144.875 0.0072368 0.0046448 2024-04-23T16:27:36.822 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:27:36.822 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:27:36.847 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2403379480' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:27:36.847 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:36 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:27:36.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2403379480' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:27:36.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:36 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:27:37.098 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:27:37.099 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:27:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "epoch": 288, 2024-04-23T16:27:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "fsid": "8e3594f2-018b-11ef-bc93-c7b262605968", 2024-04-23T16:27:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "created": "2024-04-23T16:08:13.477590+0000", 2024-04-23T16:27:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "modified": "2024-04-23T16:27:36.594622+0000", 2024-04-23T16:27:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "last_up_change": "2024-04-23T16:27:00.729195+0000", 2024-04-23T16:27:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "last_in_change": "2024-04-23T16:24:50.783418+0000", 2024-04-23T16:27:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-23T16:27:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "flags_num": 5806080, 2024-04-23T16:27:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "flags_set": [ 2024-04-23T16:27:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "nodeep-scrub", 2024-04-23T16:27:37.099 INFO:teuthology.orchestra.run.smithi103.stdout: "noscrub", 2024-04-23T16:27:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "pglog_hardlimit", 2024-04-23T16:27:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "purged_snapdirs", 2024-04-23T16:27:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_deletes", 2024-04-23T16:27:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "sortbitwise" 2024-04-23T16:27:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:27:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_version": 28, 2024-04-23T16:27:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "full_ratio": 0.94999998807907104, 2024-04-23T16:27:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-23T16:27:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-23T16:27:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_snapshot": "", 2024-04-23T16:27:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_max": 4, 2024-04-23T16:27:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "max_osd": 12, 2024-04-23T16:27:37.100 INFO:teuthology.orchestra.run.smithi103.stdout: "require_min_compat_client": "luminous", 2024-04-23T16:27:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "min_compat_client": "jewel", 2024-04-23T16:27:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "require_osd_release": "squid", 2024-04-23T16:27:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "allow_crimson": false, 2024-04-23T16:27:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "pools": [ 2024-04-23T16:27:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-04-23T16:27:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_name": ".mgr", 2024-04-23T16:27:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "create_time": "2024-04-23T16:11:22.416777+0000", 2024-04-23T16:27:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-04-23T16:27:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "flags_names": "hashpspool", 2024-04-23T16:27:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "type": 1, 2024-04-23T16:27:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 2, 2024-04-23T16:27:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "min_size": 2, 2024-04-23T16:27:37.101 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_rule": 0, 2024-04-23T16:27:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_count": 0, 2024-04-23T16:27:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_target": 0, 2024-04-23T16:27:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T16:27:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T16:27:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "object_hash": 2, 2024-04-23T16:27:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_autoscale_mode": "off", 2024-04-23T16:27:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num": 1, 2024-04-23T16:27:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_placement_num": 1, 2024-04-23T16:27:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_placement_num_target": 1, 2024-04-23T16:27:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_target": 1, 2024-04-23T16:27:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_pending": 1, 2024-04-23T16:27:37.102 INFO:teuthology.orchestra.run.smithi103.stdout: "last_pg_merge_meta": { 2024-04-23T16:27:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "source_pgid": "0.0", 2024-04-23T16:27:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "ready_epoch": 0, 2024-04-23T16:27:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "last_epoch_started": 0, 2024-04-23T16:27:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "last_epoch_clean": 0, 2024-04-23T16:27:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "source_version": "0'0", 2024-04-23T16:27:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "target_version": "0'0" 2024-04-23T16:27:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "last_change": "17", 2024-04-23T16:27:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend": "0", 2024-04-23T16:27:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T16:27:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T16:27:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "auid": 0, 2024-04-23T16:27:37.103 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_mode": "selfmanaged", 2024-04-23T16:27:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_seq": 0, 2024-04-23T16:27:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_epoch": 0, 2024-04-23T16:27:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_snaps": [], 2024-04-23T16:27:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "removed_snaps": "[]", 2024-04-23T16:27:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "quota_max_bytes": 0, 2024-04-23T16:27:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "quota_max_objects": 0, 2024-04-23T16:27:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "tiers": [], 2024-04-23T16:27:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "tier_of": -1, 2024-04-23T16:27:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "read_tier": -1, 2024-04-23T16:27:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "write_tier": -1, 2024-04-23T16:27:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_mode": "none", 2024-04-23T16:27:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "target_max_bytes": 0, 2024-04-23T16:27:37.104 INFO:teuthology.orchestra.run.smithi103.stdout: "target_max_objects": 0, 2024-04-23T16:27:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T16:27:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T16:27:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T16:27:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_min_flush_age": 0, 2024-04-23T16:27:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_min_evict_age": 0, 2024-04-23T16:27:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "erasure_code_profile": "", 2024-04-23T16:27:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_params": { 2024-04-23T16:27:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "none" 2024-04-23T16:27:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_period": 0, 2024-04-23T16:27:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_count": 0, 2024-04-23T16:27:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "use_gmt_hitset": true, 2024-04-23T16:27:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "min_read_recency_for_promote": 0, 2024-04-23T16:27:37.105 INFO:teuthology.orchestra.run.smithi103.stdout: "min_write_recency_for_promote": 0, 2024-04-23T16:27:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T16:27:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_search_last_n": 0, 2024-04-23T16:27:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: "grade_table": [], 2024-04-23T16:27:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: "stripe_width": 0, 2024-04-23T16:27:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: "expected_num_objects": 0, 2024-04-23T16:27:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: "fast_read": false, 2024-04-23T16:27:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: "options": { 2024-04-23T16:27:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_max": 32, 2024-04-23T16:27:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_min": 1 2024-04-23T16:27:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: "application_metadata": { 2024-04-23T16:27:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": {} 2024-04-23T16:27:37.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: "read_balance": { 2024-04-23T16:27:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: "score_type": "Fair distribution", 2024-04-23T16:27:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: "score_acting": 11.760000228881836, 2024-04-23T16:27:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: "score_stable": 11.760000228881836, 2024-04-23T16:27:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: "optimal_score": 0.17000000178813934, 2024-04-23T16:27:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: "raw_score_acting": 2, 2024-04-23T16:27:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: "raw_score_stable": 2, 2024-04-23T16:27:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity_weighted": 0.98181778192520142, 2024-04-23T16:27:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: "average_primary_affinity": 0.98000001907348633, 2024-04-23T16:27:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-04-23T16:27:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.108 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 4, 2024-04-23T16:27:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_name": "unique_pool_2", 2024-04-23T16:27:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: "create_time": "2024-04-23T16:27:07.774581+0000", 2024-04-23T16:27:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-04-23T16:27:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: "flags_names": "hashpspool", 2024-04-23T16:27:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: "type": 1, 2024-04-23T16:27:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 2, 2024-04-23T16:27:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: "min_size": 2, 2024-04-23T16:27:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_rule": 0, 2024-04-23T16:27:37.109 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_count": 0, 2024-04-23T16:27:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_target": 0, 2024-04-23T16:27:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T16:27:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T16:27:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: "object_hash": 2, 2024-04-23T16:27:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_autoscale_mode": "off", 2024-04-23T16:27:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num": 16, 2024-04-23T16:27:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_placement_num": 16, 2024-04-23T16:27:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_placement_num_target": 16, 2024-04-23T16:27:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_target": 16, 2024-04-23T16:27:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_pending": 16, 2024-04-23T16:27:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "last_pg_merge_meta": { 2024-04-23T16:27:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "source_pgid": "0.0", 2024-04-23T16:27:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "ready_epoch": 0, 2024-04-23T16:27:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "last_epoch_started": 0, 2024-04-23T16:27:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "last_epoch_clean": 0, 2024-04-23T16:27:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "source_version": "0'0", 2024-04-23T16:27:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "target_version": "0'0" 2024-04-23T16:27:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "last_change": "281", 2024-04-23T16:27:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend": "0", 2024-04-23T16:27:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T16:27:37.111 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T16:27:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "auid": 0, 2024-04-23T16:27:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_mode": "selfmanaged", 2024-04-23T16:27:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_seq": 0, 2024-04-23T16:27:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_epoch": 0, 2024-04-23T16:27:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_snaps": [], 2024-04-23T16:27:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "removed_snaps": "[]", 2024-04-23T16:27:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "quota_max_bytes": 0, 2024-04-23T16:27:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "quota_max_objects": 0, 2024-04-23T16:27:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "tiers": [], 2024-04-23T16:27:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "tier_of": -1, 2024-04-23T16:27:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "read_tier": -1, 2024-04-23T16:27:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "write_tier": -1, 2024-04-23T16:27:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_mode": "none", 2024-04-23T16:27:37.112 INFO:teuthology.orchestra.run.smithi103.stdout: "target_max_bytes": 0, 2024-04-23T16:27:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "target_max_objects": 0, 2024-04-23T16:27:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T16:27:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T16:27:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T16:27:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_min_flush_age": 0, 2024-04-23T16:27:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_min_evict_age": 0, 2024-04-23T16:27:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "erasure_code_profile": "", 2024-04-23T16:27:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_params": { 2024-04-23T16:27:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "none" 2024-04-23T16:27:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_period": 0, 2024-04-23T16:27:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_count": 0, 2024-04-23T16:27:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "use_gmt_hitset": true, 2024-04-23T16:27:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "min_read_recency_for_promote": 0, 2024-04-23T16:27:37.113 INFO:teuthology.orchestra.run.smithi103.stdout: "min_write_recency_for_promote": 0, 2024-04-23T16:27:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T16:27:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_search_last_n": 0, 2024-04-23T16:27:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "grade_table": [], 2024-04-23T16:27:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "stripe_width": 0, 2024-04-23T16:27:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "expected_num_objects": 0, 2024-04-23T16:27:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "fast_read": false, 2024-04-23T16:27:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "options": {}, 2024-04-23T16:27:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "application_metadata": { 2024-04-23T16:27:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "rados": {} 2024-04-23T16:27:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "read_balance": { 2024-04-23T16:27:37.114 INFO:teuthology.orchestra.run.smithi103.stdout: "score_type": "Fair distribution", 2024-04-23T16:27:37.115 INFO:teuthology.orchestra.run.smithi103.stdout: "score_acting": 3, 2024-04-23T16:27:37.115 INFO:teuthology.orchestra.run.smithi103.stdout: "score_stable": 3, 2024-04-23T16:27:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: "optimal_score": 0.75, 2024-04-23T16:27:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: "raw_score_acting": 2.25, 2024-04-23T16:27:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: "raw_score_stable": 2.25, 2024-04-23T16:27:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity_weighted": 0.98181778192520142, 2024-04-23T16:27:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: "average_primary_affinity": 0.98000001907348633, 2024-04-23T16:27:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-04-23T16:27:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:27:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: "osds": [ 2024-04-23T16:27:37.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 0, 2024-04-23T16:27:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "4c084047-8f01-4b86-a60c-5592735b5862", 2024-04-23T16:27:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:27:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:27:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:27:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:27:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:27:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:27:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 8, 2024-04-23T16:27:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 274, 2024-04-23T16:27:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:27:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:27:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:27:37.117 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6801", 2024-04-23T16:27:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:27:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:27:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6802", 2024-04-23T16:27:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:27:37.118 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:27:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6804", 2024-04-23T16:27:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:27:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:27:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6803", 2024-04-23T16:27:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:27:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6801/4274040097", 2024-04-23T16:27:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6802/4274040097", 2024-04-23T16:27:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6804/4274040097", 2024-04-23T16:27:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6803/4274040097", 2024-04-23T16:27:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:27:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:27:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:27:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 1, 2024-04-23T16:27:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "3438f2bd-5fb7-432c-b383-29e18c442f20", 2024-04-23T16:27:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:27:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 0, 2024-04-23T16:27:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 0, 2024-04-23T16:27:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:27:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 13, 2024-04-23T16:27:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 232, 2024-04-23T16:27:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 271, 2024-04-23T16:27:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 192, 2024-04-23T16:27:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 233, 2024-04-23T16:27:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:27:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:27:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6805", 2024-04-23T16:27:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 859282187 2024-04-23T16:27:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:27:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6806", 2024-04-23T16:27:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 859282187 2024-04-23T16:27:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:27:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6808", 2024-04-23T16:27:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 859282187 2024-04-23T16:27:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:27:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6807", 2024-04-23T16:27:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 859282187 2024-04-23T16:27:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6805/859282187", 2024-04-23T16:27:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6806/859282187", 2024-04-23T16:27:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6808/859282187", 2024-04-23T16:27:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6807/859282187", 2024-04-23T16:27:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:27:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:27:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:27:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 2, 2024-04-23T16:27:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "c17fffb4-8d17-4546-ba24-e5951a2a37fe", 2024-04-23T16:27:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:27:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:27:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:27:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:27:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:27:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:27:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 21, 2024-04-23T16:27:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 274, 2024-04-23T16:27:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:27:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:27:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:27:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6809", 2024-04-23T16:27:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 424240088 2024-04-23T16:27:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:27:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6810", 2024-04-23T16:27:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 424240088 2024-04-23T16:27:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:27:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6812", 2024-04-23T16:27:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 424240088 2024-04-23T16:27:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:27:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6811", 2024-04-23T16:27:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 424240088 2024-04-23T16:27:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6809/424240088", 2024-04-23T16:27:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6810/424240088", 2024-04-23T16:27:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6812/424240088", 2024-04-23T16:27:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6811/424240088", 2024-04-23T16:27:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:27:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:27:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:27:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 3, 2024-04-23T16:27:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "cee91a3d-55dc-4e89-b568-86821632bb1c", 2024-04-23T16:27:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:27:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:27:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:27:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:27:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:27:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:27:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 27, 2024-04-23T16:27:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 274, 2024-04-23T16:27:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:27:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:27:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:27:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6813", 2024-04-23T16:27:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1895183074 2024-04-23T16:27:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:27:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6814", 2024-04-23T16:27:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1895183074 2024-04-23T16:27:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:27:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6816", 2024-04-23T16:27:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1895183074 2024-04-23T16:27:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:27:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6815", 2024-04-23T16:27:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1895183074 2024-04-23T16:27:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6813/1895183074", 2024-04-23T16:27:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6814/1895183074", 2024-04-23T16:27:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6816/1895183074", 2024-04-23T16:27:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6815/1895183074", 2024-04-23T16:27:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:27:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:27:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:27:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 4, 2024-04-23T16:27:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "61d5385b-2c82-42f8-93c3-8f1baf71c65e", 2024-04-23T16:27:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:27:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:27:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:27:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:27:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:27:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:27:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 33, 2024-04-23T16:27:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 274, 2024-04-23T16:27:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:27:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:27:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:27:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6800", 2024-04-23T16:27:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 794945071 2024-04-23T16:27:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:27:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6801", 2024-04-23T16:27:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 794945071 2024-04-23T16:27:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:27:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6803", 2024-04-23T16:27:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 794945071 2024-04-23T16:27:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:27:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6802", 2024-04-23T16:27:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 794945071 2024-04-23T16:27:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6800/794945071", 2024-04-23T16:27:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6801/794945071", 2024-04-23T16:27:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6803/794945071", 2024-04-23T16:27:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6802/794945071", 2024-04-23T16:27:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:27:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:27:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:27:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 5, 2024-04-23T16:27:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "5d492189-c3cd-47ad-a12a-ed4d275bcfe3", 2024-04-23T16:27:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:27:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:27:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:27:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:27:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:27:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:27:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 38, 2024-04-23T16:27:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 192, 2024-04-23T16:27:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:27:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:27:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:27:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6804", 2024-04-23T16:27:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:27:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:27:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6805", 2024-04-23T16:27:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:27:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:27:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6807", 2024-04-23T16:27:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:27:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:27:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6806", 2024-04-23T16:27:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:27:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6804/3547200711", 2024-04-23T16:27:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6805/3547200711", 2024-04-23T16:27:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6807/3547200711", 2024-04-23T16:27:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6806/3547200711", 2024-04-23T16:27:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:27:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:27:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:27:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 6, 2024-04-23T16:27:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "89f2379e-cf14-4e69-8747-7535db81b70b", 2024-04-23T16:27:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:27:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:27:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:27:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:27:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:27:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:27:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 47, 2024-04-23T16:27:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 274, 2024-04-23T16:27:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:27:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:27:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:27:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6808", 2024-04-23T16:27:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:27:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:27:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6809", 2024-04-23T16:27:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:27:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:27:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6811", 2024-04-23T16:27:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:27:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:27:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6810", 2024-04-23T16:27:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:27:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6808/366919758", 2024-04-23T16:27:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6809/366919758", 2024-04-23T16:27:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6811/366919758", 2024-04-23T16:27:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6810/366919758", 2024-04-23T16:27:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:27:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:27:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:27:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 7, 2024-04-23T16:27:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "f27b035a-558f-4528-80ea-2a00d03f6539", 2024-04-23T16:27:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:27:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:27:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:27:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 0.7818145751953125, 2024-04-23T16:27:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:27:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:27:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 56, 2024-04-23T16:27:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 192, 2024-04-23T16:27:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:27:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:27:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:27:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6812", 2024-04-23T16:27:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:27:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:27:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6813", 2024-04-23T16:27:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:27:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:27:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6815", 2024-04-23T16:27:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:27:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:27:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6814", 2024-04-23T16:27:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:27:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6812/3719550888", 2024-04-23T16:27:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6813/3719550888", 2024-04-23T16:27:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6815/3719550888", 2024-04-23T16:27:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6814/3719550888", 2024-04-23T16:27:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:27:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:27:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:27:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 8, 2024-04-23T16:27:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed", 2024-04-23T16:27:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:27:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:27:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:27:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:27:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:27:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:27:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 66, 2024-04-23T16:27:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 192, 2024-04-23T16:27:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:27:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:27:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:27:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6800", 2024-04-23T16:27:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:27:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:27:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6801", 2024-04-23T16:27:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:27:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:27:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6803", 2024-04-23T16:27:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:27:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:27:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6802", 2024-04-23T16:27:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:27:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6800/1151408595", 2024-04-23T16:27:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6801/1151408595", 2024-04-23T16:27:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6803/1151408595", 2024-04-23T16:27:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6802/1151408595", 2024-04-23T16:27:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:27:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:27:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:27:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 9, 2024-04-23T16:27:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "124d9b05-11eb-4622-b6a2-c4d5bd60b7e6", 2024-04-23T16:27:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:27:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:27:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:27:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:27:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:27:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:27:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 73, 2024-04-23T16:27:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 274, 2024-04-23T16:27:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:27:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:27:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:27:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6804", 2024-04-23T16:27:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:27:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:27:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6805", 2024-04-23T16:27:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:27:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:27:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6807", 2024-04-23T16:27:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:27:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:27:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6806", 2024-04-23T16:27:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:27:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6804/306501862", 2024-04-23T16:27:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6805/306501862", 2024-04-23T16:27:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6807/306501862", 2024-04-23T16:27:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6806/306501862", 2024-04-23T16:27:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:27:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:27:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:27:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 10, 2024-04-23T16:27:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "3dfb9070-e534-4ea4-a25e-ee1c51297447", 2024-04-23T16:27:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:27:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 0, 2024-04-23T16:27:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 0, 2024-04-23T16:27:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:27:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:27:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:27:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 81, 2024-04-23T16:27:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 0, 2024-04-23T16:27:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:27:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:27:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:27:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6808", 2024-04-23T16:27:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:27:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:27:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6809", 2024-04-23T16:27:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:27:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:27:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6811", 2024-04-23T16:27:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:27:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:27:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6810", 2024-04-23T16:27:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:27:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6808/1231263138", 2024-04-23T16:27:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6809/1231263138", 2024-04-23T16:27:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6811/1231263138", 2024-04-23T16:27:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6810/1231263138", 2024-04-23T16:27:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:27:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:27:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:27:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 11, 2024-04-23T16:27:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "bc163925-f623-4f63-ab89-35e2d511a730", 2024-04-23T16:27:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:27:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:27:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:27:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:27:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:27:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:27:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 90, 2024-04-23T16:27:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 274, 2024-04-23T16:27:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:27:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:27:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:27:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6812", 2024-04-23T16:27:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:27:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:27:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6813", 2024-04-23T16:27:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:27:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:27:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6815", 2024-04-23T16:27:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:27:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:27:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:27:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:27:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6814", 2024-04-23T16:27:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:27:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6812/457992661", 2024-04-23T16:27:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6813/457992661", 2024-04-23T16:27:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6815/457992661", 2024-04-23T16:27:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6814/457992661", 2024-04-23T16:27:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:27:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:27:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:27:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:27:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:27:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_xinfo": [ 2024-04-23T16:27:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 0, 2024-04-23T16:27:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:27:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:27:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:27:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:27:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:27:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:26:48.649852+0000", 2024-04-23T16:27:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:27:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 1, 2024-04-23T16:27:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:27:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:27:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:27:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:27:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 65536, 2024-04-23T16:27:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:27:00.293433+0000", 2024-04-23T16:27:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 232 2024-04-23T16:27:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 2, 2024-04-23T16:27:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:27:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:27:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:27:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:27:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:27:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:26:30.393856+0000", 2024-04-23T16:27:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:27:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 3, 2024-04-23T16:27:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:27:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:27:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:27:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:27:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:27:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:26:30.409557+0000", 2024-04-23T16:27:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:27:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 4, 2024-04-23T16:27:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:27:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:27:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:27:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:27:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:27:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:26:36.961902+0000", 2024-04-23T16:27:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:27:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 5, 2024-04-23T16:27:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:27:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:27:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:27:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:27:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:27:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:27:05.081617+0000", 2024-04-23T16:27:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:27:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 6, 2024-04-23T16:27:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:27:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:27:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:27:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:27:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:27:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:26:35.509986+0000", 2024-04-23T16:27:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:27:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 7, 2024-04-23T16:27:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:27:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:27:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:27:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:27:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:27:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:27:31.235811+0000", 2024-04-23T16:27:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:27:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 8, 2024-04-23T16:27:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:27:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:27:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:27:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:27:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:27:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:27:10.485962+0000", 2024-04-23T16:27:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:27:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 9, 2024-04-23T16:27:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:27:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:27:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:27:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:27:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:27:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:27:20.706699+0000", 2024-04-23T16:27:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:27:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 10, 2024-04-23T16:27:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:27:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:27:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:27:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:27:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 65536, 2024-04-23T16:27:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:27:11.081185+0000", 2024-04-23T16:27:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:27:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:27:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 11, 2024-04-23T16:27:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:27:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:27:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:27:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:27:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:27:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:27:07.125024+0000", 2024-04-23T16:27:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:27:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:27:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_upmap": [], 2024-04-23T16:27:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_upmap_items": [], 2024-04-23T16:27:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_upmap_primaries": [], 2024-04-23T16:27:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_temp": [], 2024-04-23T16:27:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_temp": [], 2024-04-23T16:27:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "blocklist": { 2024-04-23T16:27:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:6800/3535470771": "2024-04-24T16:09:14.324448+0000", 2024-04-23T16:27:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/1188058303": "2024-04-24T16:09:14.324448+0000", 2024-04-23T16:27:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/1510553721": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:27:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/281182801": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:27:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:6800/2049124518": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:27:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/466241963": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:27:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/2726838698": "2024-04-24T16:09:14.324448+0000", 2024-04-23T16:27:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/596230573": "2024-04-24T16:09:14.324448+0000" 2024-04-23T16:27:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: "range_blocklist": {}, 2024-04-23T16:27:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: "erasure_code_profiles": { 2024-04-23T16:27:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: "default": { 2024-04-23T16:27:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: "crush-failure-domain": "osd", 2024-04-23T16:27:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: "k": "2", 2024-04-23T16:27:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: "m": "1", 2024-04-23T16:27:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: "plugin": "jerasure", 2024-04-23T16:27:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: "technique": "reed_sol_van" 2024-04-23T16:27:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:27:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "removed_snaps_queue": [], 2024-04-23T16:27:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "new_removed_snaps": [], 2024-04-23T16:27:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "new_purged_snaps": [], 2024-04-23T16:27:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_node_flags": {}, 2024-04-23T16:27:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "device_class_flags": {}, 2024-04-23T16:27:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_mode": { 2024-04-23T16:27:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_mode_enabled": false, 2024-04-23T16:27:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_bucket_count": 0, 2024-04-23T16:27:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "degraded_stretch_mode": 0, 2024-04-23T16:27:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "recovering_stretch_mode": 0, 2024-04-23T16:27:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_mode_bucket": 0 2024-04-23T16:27:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:27:37.177 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:27:37.822 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:27:37.821486 min lat: 0.00105502 max lat: 0.204864 avg lat: 0.00472479 2024-04-23T16:27:37.822 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:27:37.822 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 20 16 67736 67720 211.578 142.438 0.00817245 0.00472479 2024-04-23T16:27:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:37 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2403379480' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:27:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:37 smithi148 ceph-mon[44520]: osdmap e288: 12 total, 12 up, 10 in 2024-04-23T16:27:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:37 smithi148 ceph-mon[44520]: pgmap v829: 17 pgs: 17 active+clean; 3.7 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 178 MiB/s wr, 2.85k op/s 2024-04-23T16:27:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:37 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2321582370' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:27:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:37.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:37 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3750065031' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-23T16:27:37.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:37.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:37.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:37.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:37 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2403379480' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:27:37.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:37 smithi188 ceph-mon[44101]: osdmap e288: 12 total, 12 up, 10 in 2024-04-23T16:27:37.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:37 smithi188 ceph-mon[44101]: pgmap v829: 17 pgs: 17 active+clean; 3.7 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 178 MiB/s wr, 2.85k op/s 2024-04-23T16:27:37.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:37 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2321582370' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:27:37.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:37.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:37.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:37.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:37.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:37 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3750065031' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-23T16:27:37.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:37.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:37.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.063 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:27:38.064 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:27:38.064 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:27:38.064 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:27:38.064 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:27:38.064 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:27:38.064 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:27:38.064 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:27:38.064 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:27:38.064 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:27:38.064 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:27:38.064 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:27:38.065 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:27:38.065 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:27:38.065 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:27:38.065 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:27:38.065 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:27:38.065 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:27:38.065 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:27:38.065 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:27:38.065 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:27:38.065 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:27:38.065 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:27:38.065 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:27:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:37 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2403379480' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:27:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:37 smithi103 ceph-mon[39629]: osdmap e288: 12 total, 12 up, 10 in 2024-04-23T16:27:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:37 smithi103 ceph-mon[39629]: pgmap v829: 17 pgs: 17 active+clean; 3.7 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 178 MiB/s wr, 2.85k op/s 2024-04-23T16:27:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:37 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2321582370' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:27:38.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:37 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3750065031' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-23T16:27:38.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.783 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:27:38.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:38.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:27:38.821 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 21 16 70071 70055 208.452 145.938 0.00759633 0.0047954 2024-04-23T16:27:38.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.888 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:27:38.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.923 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:38.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:38.964 INFO:tasks.thrashosds.thrasher:j is {'epoch': 288, 'fsid': '8e3594f2-018b-11ef-bc93-c7b262605968', 'created': '2024-04-23T16:08:13.477590+0000', 'modified': '2024-04-23T16:27:36.594622+0000', 'last_up_change': '2024-04-23T16:27:00.729195+0000', 'last_in_change': '2024-04-23T16:24:50.783418+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T16:11:22.416777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9818177819252014, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-04-23T16:27:07.774581+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '281', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9818177819252014, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}], 'osds': [{'osd': 0, 'uuid': '4c084047-8f01-4b86-a60c-5592735b5862', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6801', 'nonce': 4274040097}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6802', 'nonce': 4274040097}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6804', 'nonce': 4274040097}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6803', 'nonce': 4274040097}]}, 'public_addr': '172.21.15.103:6801/4274040097', 'cluster_addr': '172.21.15.103:6802/4274040097', 'heartbeat_back_addr': '172.21.15.103:6804/4274040097', 'heartbeat_front_addr': '172.21.15.103:6803/4274040097', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '3438f2bd-5fb7-432c-b383-29e18c442f20', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 232, 'up_from': 271, 'up_thru': 192, 'down_at': 233, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6805', 'nonce': 859282187}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6806', 'nonce': 859282187}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6808', 'nonce': 859282187}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6807', 'nonce': 859282187}]}, 'public_addr': '172.21.15.103:6805/859282187', 'cluster_addr': '172.21.15.103:6806/859282187', 'heartbeat_back_addr': '172.21.15.103:6808/859282187', 'heartbeat_front_addr': '172.21.15.103:6807/859282187', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c17fffb4-8d17-4546-ba24-e5951a2a37fe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6809', 'nonce': 424240088}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6810', 'nonce': 424240088}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6812', 'nonce': 424240088}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6811', 'nonce': 424240088}]}, 'public_addr': '172.21.15.103:6809/424240088', 'cluster_addr': '172.21.15.103:6810/424240088', 'heartbeat_back_addr': '172.21.15.103:6812/424240088', 'heartbeat_front_addr': '172.21.15.103:6811/424240088', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'cee91a3d-55dc-4e89-b568-86821632bb1c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6813', 'nonce': 1895183074}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6814', 'nonce': 1895183074}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6816', 'nonce': 1895183074}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6815', 'nonce': 1895183074}]}, 'public_addr': '172.21.15.103:6813/1895183074', 'cluster_addr': '172.21.15.103:6814/1895183074', 'heartbeat_back_addr': '172.21.15.103:6816/1895183074', 'heartbeat_front_addr': '172.21.15.103:6815/1895183074', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '61d5385b-2c82-42f8-93c3-8f1baf71c65e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6800', 'nonce': 794945071}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6801', 'nonce': 794945071}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6803', 'nonce': 794945071}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6802', 'nonce': 794945071}]}, 'public_addr': '172.21.15.148:6800/794945071', 'cluster_addr': '172.21.15.148:6801/794945071', 'heartbeat_back_addr': '172.21.15.148:6803/794945071', 'heartbeat_front_addr': '172.21.15.148:6802/794945071', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '5d492189-c3cd-47ad-a12a-ed4d275bcfe3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 192, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6804', 'nonce': 3547200711}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6805', 'nonce': 3547200711}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6807', 'nonce': 3547200711}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6806', 'nonce': 3547200711}]}, 'public_addr': '172.21.15.148:6804/3547200711', 'cluster_addr': '172.21.15.148:6805/3547200711', 'heartbeat_back_addr': '172.21.15.148:6807/3547200711', 'heartbeat_front_addr': '172.21.15.148:6806/3547200711', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '89f2379e-cf14-4e69-8747-7535db81b70b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6808', 'nonce': 366919758}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6809', 'nonce': 366919758}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6811', 'nonce': 366919758}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6810', 'nonce': 366919758}]}, 'public_addr': '172.21.15.148:6808/366919758', 'cluster_addr': '172.21.15.148:6809/366919758', 'heartbeat_back_addr': '172.21.15.148:6811/366919758', 'heartbeat_front_addr': '172.21.15.148:6810/366919758', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f27b035a-558f-4528-80ea-2a00d03f6539', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7818145751953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 192, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6812', 'nonce': 3719550888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6813', 'nonce': 3719550888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6815', 'nonce': 3719550888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6814', 'nonce': 3719550888}]}, 'public_addr': '172.21.15.148:6812/3719550888', 'cluster_addr': '172.21.15.148:6813/3719550888', 'heartbeat_back_addr': '172.21.15.148:6815/3719550888', 'heartbeat_front_addr': '172.21.15.148:6814/3719550888', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 192, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6800', 'nonce': 1151408595}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6801', 'nonce': 1151408595}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6803', 'nonce': 1151408595}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6802', 'nonce': 1151408595}]}, 'public_addr': '172.21.15.188:6800/1151408595', 'cluster_addr': '172.21.15.188:6801/1151408595', 'heartbeat_back_addr': '172.21.15.188:6803/1151408595', 'heartbeat_front_addr': '172.21.15.188:6802/1151408595', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '124d9b05-11eb-4622-b6a2-c4d5bd60b7e6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6804', 'nonce': 306501862}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6805', 'nonce': 306501862}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6807', 'nonce': 306501862}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6806', 'nonce': 306501862}]}, 'public_addr': '172.21.15.188:6804/306501862', 'cluster_addr': '172.21.15.188:6805/306501862', 'heartbeat_back_addr': '172.21.15.188:6807/306501862', 'heartbeat_front_addr': '172.21.15.188:6806/306501862', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '3dfb9070-e534-4ea4-a25e-ee1c51297447', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6808', 'nonce': 1231263138}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6809', 'nonce': 1231263138}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6811', 'nonce': 1231263138}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6810', 'nonce': 1231263138}]}, 'public_addr': '172.21.15.188:6808/1231263138', 'cluster_addr': '172.21.15.188:6809/1231263138', 'heartbeat_back_addr': '172.21.15.188:6811/1231263138', 'heartbeat_front_addr': '172.21.15.188:6810/1231263138', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'bc163925-f623-4f63-ab89-35e2d511a730', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6812', 'nonce': 457992661}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6813', 'nonce': 457992661}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6815', 'nonce': 457992661}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6814', 'nonce': 457992661}]}, 'public_addr': '172.21.15.188:6812/457992661', 'cluster_addr': '172.21.15.188:6813/457992661', 'heartbeat_back_addr': '172.21.15.188:6815/457992661', 'heartbeat_front_addr': '172.21.15.188:6814/457992661', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:26:48.649852+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-23T16:27:00.293433+0000', 'dead_epoch': 232}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:26:30.393856+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:26:30.409557+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:26:36.961902+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:27:05.081617+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:26:35.509986+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:27:31.235811+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:27:10.485962+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:27:20.706699+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-23T16:27:11.081185+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:27:07.125024+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.103:6800/3535470771': '2024-04-24T16:09:14.324448+0000', '172.21.15.103:0/1188058303': '2024-04-24T16:09:14.324448+0000', '172.21.15.103:0/1510553721': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:0/281182801': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:6800/2049124518': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:0/466241963': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:0/2726838698': '2024-04-24T16:09:14.324448+0000', '172.21.15.103:0/596230573': '2024-04-24T16:09:14.324448+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-23T16:27:38.964 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:27:39.663 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:27:39.822 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 22 16 72065 72049 204.642 124.625 0.0792115 0.00487322 2024-04-23T16:27:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:39 smithi103 ceph-mon[39629]: pgmap v830: 17 pgs: 17 active+clean; 3.7 GiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-04-23T16:27:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:39 smithi103 ceph-mon[39629]: osdmap e289: 12 total, 12 up, 10 in 2024-04-23T16:27:39.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:39 smithi188 ceph-mon[44101]: pgmap v830: 17 pgs: 17 active+clean; 3.7 GiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-04-23T16:27:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:39 smithi188 ceph-mon[44101]: osdmap e289: 12 total, 12 up, 10 in 2024-04-23T16:27:40.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:39 smithi148 ceph-mon[44520]: pgmap v830: 17 pgs: 17 active+clean; 3.7 GiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-04-23T16:27:40.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:39 smithi148 ceph-mon[44520]: osdmap e289: 12 total, 12 up, 10 in 2024-04-23T16:27:40.796 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:27:40.822 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 23 16 74355 74339 201.967 143.125 0.00998336 0.00494973 2024-04-23T16:27:41.383 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:27:41.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:41.389 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:27:41.391 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-23T16:27:41.493 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:27:41.526 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:41.530 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:27:41.551 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:27:41.555 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:27:41.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:41.556 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:27:41.657 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:27:41.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:41 smithi188 ceph-mon[44101]: pgmap v832: 17 pgs: 17 active+clean; 4.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 168 MiB/s wr, 2.68k op/s 2024-04-23T16:27:41.705 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:41.739 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:27:41.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:41.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:27:41.802 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:27:41.824 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 24 16 76337 76321 198.697 123.875 0.00948199 0.00503098 2024-04-23T16:27:41.844 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:27:41.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:41 smithi103 ceph-mon[39629]: pgmap v832: 17 pgs: 17 active+clean; 4.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 168 MiB/s wr, 2.68k op/s 2024-04-23T16:27:41.880 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:42.086 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:27:42.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:42.091 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:27:42.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:41 smithi148 ceph-mon[44520]: pgmap v832: 17 pgs: 17 active+clean; 4.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 168 MiB/s wr, 2.68k op/s 2024-04-23T16:27:42.128 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:27:42.142 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:27:42.192 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:27:42.229 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:42.687 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:27:42.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:42.692 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:27:42.793 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:27:42.824 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 25 16 78455 78439 196.044 132.375 0.00330957 0.00509925 2024-04-23T16:27:42.839 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:42.872 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:27:42.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:42.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:27:42.977 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:27:43.013 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:43.044 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:27:43.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:43.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:27:43.153 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:27:43.193 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:43.811 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:27:43.811 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:27:43.812 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:27:43.812 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:27:43.812 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:27:43.824 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 26 16 80556 80540 193.554 131.312 0.00324737 0.00516515 2024-04-23T16:27:43.833 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:43 smithi188 ceph-mon[44101]: from='client.15531 v1:172.21.15.103:0/3457600470' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:27:43.834 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:43 smithi188 ceph-mon[44101]: pgmap v833: 17 pgs: 17 active+clean; 4.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-04-23T16:27:44.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:43 smithi103 ceph-mon[39629]: from='client.15531 v1:172.21.15.103:0/3457600470' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:27:44.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:43 smithi103 ceph-mon[39629]: pgmap v833: 17 pgs: 17 active+clean; 4.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-04-23T16:27:44.047 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:27:44.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:44.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:27:44.080 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:27:44.090 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":832,"stamp":"2024-04-23T16:27:40.633377+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4541252128,"num_objects":69287,"num_object_clones":0,"num_object_copies":138574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69287,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":69466,"num_write_kb":4437178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69424,"ondisk_log_size":69424,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":937451520,"kb_used":10969748,"kb_used_data":8833764,"kb_used_omap":15,"kb_used_meta":2135856,"kb_avail":926481772,"statfs":{"total":959950356480,"available":948717334528,"internally_reserved":0,"allocated":9045774336,"data_stored":9041117298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15948,"internal_metadata":2187116980},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1164181504,"num_objects":17764,"num_object_clones":0,"num_object_copies":35528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17764,"num_write_kb":1136896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.618335"},"pg_stats":[{"pgid":"4.f","version":"289'4745","reported_seq":4784,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-23T16:27:40.458253+0000","last_change":"2024-04-23T16:27:08.790969+0000","last_active":"2024-04-23T16:27:40.458253+0000","last_peered":"2024-04-23T16:27:40.458253+0000","last_clean":"2024-04-23T16:27:40.458253+0000","last_became_active":"2024-04-23T16:27:08.790680+0000","last_became_peered":"2024-04-23T16:27:08.790680+0000","last_unstale":"2024-04-23T16:27:40.458253+0000","last_undegraded":"2024-04-23T16:27:40.458253+0000","last_fullsized":"2024-04-23T16:27:40.458253+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":4745,"log_dups_size":0,"ondisk_log_size":4745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:12.841142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310968320,"num_objects":4745,"num_object_clones":0,"num_object_copies":9490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4745,"num_write_kb":303680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"288'4274","reported_seq":4310,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-23T16:27:38.904760+0000","last_change":"2024-04-23T16:27:08.790855+0000","last_active":"2024-04-23T16:27:38.904760+0000","last_peered":"2024-04-23T16:27:38.904760+0000","last_clean":"2024-04-23T16:27:38.904760+0000","last_became_active":"2024-04-23T16:27:08.790531+0000","last_became_peered":"2024-04-23T16:27:08.790531+0000","last_unstale":"2024-04-23T16:27:38.904760+0000","last_undegraded":"2024-04-23T16:27:38.904760+0000","last_fullsized":"2024-04-23T16:27:38.904760+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":4274,"log_dups_size":0,"ondisk_log_size":4274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:18.893001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280100864,"num_objects":4274,"num_object_clones":0,"num_object_copies":8548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4274,"num_write_kb":273536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"288'4385","reported_seq":4421,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-23T16:27:38.893530+0000","last_change":"2024-04-23T16:27:08.789322+0000","last_active":"2024-04-23T16:27:38.893530+0000","last_peered":"2024-04-23T16:27:38.893530+0000","last_clean":"2024-04-23T16:27:38.893530+0000","last_became_active":"2024-04-23T16:27:08.789015+0000","last_became_peered":"2024-04-23T16:27:08.789015+0000","last_unstale":"2024-04-23T16:27:38.893530+0000","last_undegraded":"2024-04-23T16:27:38.893530+0000","last_fullsized":"2024-04-23T16:27:38.893530+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":4385,"log_dups_size":0,"ondisk_log_size":4385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:18.065354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287375360,"num_objects":4385,"num_object_clones":0,"num_object_copies":8770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4385,"num_write_kb":280640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"288'4402","reported_seq":4438,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-23T16:27:38.913507+0000","last_change":"2024-04-23T16:27:08.790981+0000","last_active":"2024-04-23T16:27:38.913507+0000","last_peered":"2024-04-23T16:27:38.913507+0000","last_clean":"2024-04-23T16:27:38.913507+0000","last_became_active":"2024-04-23T16:27:08.790714+0000","last_became_peered":"2024-04-23T16:27:08.790714+0000","last_unstale":"2024-04-23T16:27:38.913507+0000","last_undegraded":"2024-04-23T16:27:38.913507+0000","last_fullsized":"2024-04-23T16:27:38.913507+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":4402,"log_dups_size":0,"ondisk_log_size":4402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:10.950834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288489472,"num_objects":4402,"num_object_clones":0,"num_object_copies":8804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4402,"num_write_kb":281728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"288'4399","reported_seq":4436,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-23T16:27:38.760896+0000","last_change":"2024-04-23T16:27:08.791044+0000","last_active":"2024-04-23T16:27:38.760896+0000","last_peered":"2024-04-23T16:27:38.760896+0000","last_clean":"2024-04-23T16:27:38.760896+0000","last_became_active":"2024-04-23T16:27:08.790772+0000","last_became_peered":"2024-04-23T16:27:08.790772+0000","last_unstale":"2024-04-23T16:27:38.760896+0000","last_undegraded":"2024-04-23T16:27:38.760896+0000","last_fullsized":"2024-04-23T16:27:38.760896+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":4399,"log_dups_size":0,"ondisk_log_size":4399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:28.679532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288292864,"num_objects":4399,"num_object_clones":0,"num_object_copies":8798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4399,"num_write_kb":281536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"289'4474","reported_seq":4512,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-23T16:27:39.113633+0000","last_change":"2024-04-23T16:27:08.792705+0000","last_active":"2024-04-23T16:27:39.113633+0000","last_peered":"2024-04-23T16:27:39.113633+0000","last_clean":"2024-04-23T16:27:39.113633+0000","last_became_active":"2024-04-23T16:27:08.792382+0000","last_became_peered":"2024-04-23T16:27:08.792382+0000","last_unstale":"2024-04-23T16:27:39.113633+0000","last_undegraded":"2024-04-23T16:27:39.113633+0000","last_fullsized":"2024-04-23T16:27:39.113633+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":4474,"log_dups_size":0,"ondisk_log_size":4474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:09.350646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293208064,"num_objects":4474,"num_object_clones":0,"num_object_copies":8948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4474,"num_write_kb":286336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"287'4027","reported_seq":4060,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-04-23T16:27:36.436760+0000","last_change":"2024-04-23T16:27:08.790068+0000","last_active":"2024-04-23T16:27:36.436760+0000","last_peered":"2024-04-23T16:27:36.436760+0000","last_clean":"2024-04-23T16:27:36.436760+0000","last_became_active":"2024-04-23T16:27:08.789822+0000","last_became_peered":"2024-04-23T16:27:08.789822+0000","last_unstale":"2024-04-23T16:27:36.436760+0000","last_undegraded":"2024-04-23T16:27:36.436760+0000","last_fullsized":"2024-04-23T16:27:36.436760+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":4027,"log_dups_size":0,"ondisk_log_size":4027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:17.457267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263913472,"num_objects":4027,"num_object_clones":0,"num_object_copies":8054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4027,"num_write_kb":257728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"287'4030","reported_seq":4063,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-04-23T16:27:36.413712+0000","last_change":"2024-04-23T16:27:08.790983+0000","last_active":"2024-04-23T16:27:36.413712+0000","last_peered":"2024-04-23T16:27:36.413712+0000","last_clean":"2024-04-23T16:27:36.413712+0000","last_became_active":"2024-04-23T16:27:08.790727+0000","last_became_peered":"2024-04-23T16:27:08.790727+0000","last_unstale":"2024-04-23T16:27:36.413712+0000","last_undegraded":"2024-04-23T16:27:36.413712+0000","last_fullsized":"2024-04-23T16:27:36.413712+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":4030,"log_dups_size":0,"ondisk_log_size":4030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:09.076024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264110080,"num_objects":4030,"num_object_clones":0,"num_object_copies":8060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4030,"num_write_kb":257920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"288'4325","reported_seq":4362,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-23T16:27:38.780109+0000","last_change":"2024-04-23T16:27:08.791657+0000","last_active":"2024-04-23T16:27:38.780109+0000","last_peered":"2024-04-23T16:27:38.780109+0000","last_clean":"2024-04-23T16:27:38.780109+0000","last_became_active":"2024-04-23T16:27:08.791328+0000","last_became_peered":"2024-04-23T16:27:08.791328+0000","last_unstale":"2024-04-23T16:27:38.780109+0000","last_undegraded":"2024-04-23T16:27:38.780109+0000","last_fullsized":"2024-04-23T16:27:38.780109+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":4325,"log_dups_size":0,"ondisk_log_size":4325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:35.446374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283443200,"num_objects":4325,"num_object_clones":0,"num_object_copies":8650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4325,"num_write_kb":276800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"288'4429","reported_seq":4464,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-23T16:27:38.809348+0000","last_change":"2024-04-23T16:27:08.789417+0000","last_active":"2024-04-23T16:27:38.809348+0000","last_peered":"2024-04-23T16:27:38.809348+0000","last_clean":"2024-04-23T16:27:38.809348+0000","last_became_active":"2024-04-23T16:27:08.789141+0000","last_became_peered":"2024-04-23T16:27:08.789141+0000","last_unstale":"2024-04-23T16:27:38.809348+0000","last_undegraded":"2024-04-23T16:27:38.809348+0000","last_fullsized":"2024-04-23T16:27:38.809348+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":4429,"log_dups_size":0,"ondisk_log_size":4429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:15.404514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290258944,"num_objects":4429,"num_object_clones":0,"num_object_copies":8858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4429,"num_write_kb":283456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"288'4295","reported_seq":4332,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-23T16:27:38.569105+0000","last_change":"2024-04-23T16:27:08.790564+0000","last_active":"2024-04-23T16:27:38.569105+0000","last_peered":"2024-04-23T16:27:38.569105+0000","last_clean":"2024-04-23T16:27:38.569105+0000","last_became_active":"2024-04-23T16:27:08.790294+0000","last_became_peered":"2024-04-23T16:27:08.790294+0000","last_unstale":"2024-04-23T16:27:38.569105+0000","last_undegraded":"2024-04-23T16:27:38.569105+0000","last_fullsized":"2024-04-23T16:27:38.569105+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":4295,"log_dups_size":0,"ondisk_log_size":4295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:31.526680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281477120,"num_objects":4295,"num_object_clones":0,"num_object_copies":8590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4295,"num_write_kb":274880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"287'4119","reported_seq":4152,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-04-23T16:27:36.433814+0000","last_change":"2024-04-23T16:27:08.791003+0000","last_active":"2024-04-23T16:27:36.433814+0000","last_peered":"2024-04-23T16:27:36.433814+0000","last_clean":"2024-04-23T16:27:36.433814+0000","last_became_active":"2024-04-23T16:27:08.790759+0000","last_became_peered":"2024-04-23T16:27:08.790759+0000","last_unstale":"2024-04-23T16:27:36.433814+0000","last_undegraded":"2024-04-23T16:27:36.433814+0000","last_fullsized":"2024-04-23T16:27:36.433814+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":4119,"log_dups_size":0,"ondisk_log_size":4119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:23.398358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269942784,"num_objects":4119,"num_object_clones":0,"num_object_copies":8238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4119,"num_write_kb":263616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"288'4315","reported_seq":4350,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-23T16:27:38.835251+0000","last_change":"2024-04-23T16:27:08.859737+0000","last_active":"2024-04-23T16:27:38.835251+0000","last_peered":"2024-04-23T16:27:38.835251+0000","last_clean":"2024-04-23T16:27:38.835251+0000","last_became_active":"2024-04-23T16:27:08.859470+0000","last_became_peered":"2024-04-23T16:27:08.859470+0000","last_unstale":"2024-04-23T16:27:38.835251+0000","last_undegraded":"2024-04-23T16:27:38.835251+0000","last_fullsized":"2024-04-23T16:27:38.835251+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":4315,"log_dups_size":0,"ondisk_log_size":4315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:36.345105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282787840,"num_objects":4315,"num_object_clones":0,"num_object_copies":8630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4315,"num_write_kb":276160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"287'4048","reported_seq":4081,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-04-23T16:27:36.431175+0000","last_change":"2024-04-23T16:27:08.859830+0000","last_active":"2024-04-23T16:27:36.431175+0000","last_peered":"2024-04-23T16:27:36.431175+0000","last_clean":"2024-04-23T16:27:36.431175+0000","last_became_active":"2024-04-23T16:27:08.859549+0000","last_became_peered":"2024-04-23T16:27:08.859549+0000","last_unstale":"2024-04-23T16:27:36.431175+0000","last_undegraded":"2024-04-23T16:27:36.431175+0000","last_fullsized":"2024-04-23T16:27:36.431175+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":4048,"log_dups_size":0,"ondisk_log_size":4048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:28.867751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265289728,"num_objects":4048,"num_object_clones":0,"num_object_copies":8096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4048,"num_write_kb":259072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"289'4664","reported_seq":4702,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-23T16:27:40.465267+0000","last_change":"2024-04-23T16:27:08.791209+0000","last_active":"2024-04-23T16:27:40.465267+0000","last_peered":"2024-04-23T16:27:40.465267+0000","last_clean":"2024-04-23T16:27:40.465267+0000","last_became_active":"2024-04-23T16:27:08.790935+0000","last_became_peered":"2024-04-23T16:27:08.790935+0000","last_unstale":"2024-04-23T16:27:40.465267+0000","last_undegraded":"2024-04-23T16:27:40.465267+0000","last_fullsized":"2024-04-23T16:27:40.465267+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":4664,"log_dups_size":0,"ondisk_log_size":4664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:31.848457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305659904,"num_objects":4664,"num_object_clones":0,"num_object_copies":9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4664,"num_write_kb":298496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"289'4354","reported_seq":4391,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-23T16:27:39.113321+0000","last_change":"2024-04-23T16:27:08.791436+0000","last_active":"2024-04-23T16:27:39.113321+0000","last_peered":"2024-04-23T16:27:39.113321+0000","last_clean":"2024-04-23T16:27:39.113321+0000","last_became_active":"2024-04-23T16:27:08.791180+0000","last_became_peered":"2024-04-23T16:27:08.791180+0000","last_unstale":"2024-04-23T16:27:39.113321+0000","last_undegraded":"2024-04-23T16:27:39.113321+0000","last_fullsized":"2024-04-23T16:27:39.113321+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":4354,"log_dups_size":0,"ondisk_log_size":4354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:18.494567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285343744,"num_objects":4354,"num_object_clones":0,"num_object_copies":8708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4354,"num_write_kb":278656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":786,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-23T16:27:39.026574+0000","last_change":"2024-04-23T16:27:05.751536+0000","last_active":"2024-04-23T16:27:39.026574+0000","last_peered":"2024-04-23T16:27:39.026574+0000","last_clean":"2024-04-23T16:27:39.026574+0000","last_became_active":"2024-04-23T16:15:36.918552+0000","last_became_peered":"2024-04-23T16:15:36.918552+0000","last_unstale":"2024-04-23T16:27:39.026574+0000","last_undegraded":"2024-04-23T16:27:39.026574+0000","last_fullsized":"2024-04-23T16:27:39.026574+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:27:05.751391+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:27:05.751391+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:17.706740+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":4540661760,"num_objects":69285,"num_object_clones":0,"num_object_copies":138570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69285,"num_write_kb":4434240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9163636736,"data_stored":9163636736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69285,"ondisk_log_size":69285,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056787,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1878296,"kb_used_data":1550032,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":91866856,"statfs":{"total":95995035648,"available":94071660544,"internally_reserved":0,"allocated":1587232768,"data_stored":1586770864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":336132548},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351130,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612768,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1017376,"kb_used_data":827924,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92727776,"statfs":{"total":95995035648,"available":94953242624,"internally_reserved":0,"allocated":847794176,"data_stored":847329790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":193984960},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841699,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":140384,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93604768,"statfs":{"total":95995035648,"available":95851282432,"internally_reserved":0,"allocated":937984,"data_stored":473598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168745,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1622644,"kb_used_data":1442920,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92122508,"statfs":{"total":95995035648,"available":94333448192,"internally_reserved":0,"allocated":1477550080,"data_stored":1477076088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":184023498},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463087,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":925472,"kb_used_data":822100,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92819680,"statfs":{"total":95995035648,"available":95047352320,"internally_reserved":0,"allocated":841830400,"data_stored":841366014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137225,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738572,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":790940,"kb_used_data":552916,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":92954212,"statfs":{"total":95995035648,"available":95185113088,"internally_reserved":0,"allocated":566185984,"data_stored":565721598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313409,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":669216,"kb_used_data":539476,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93075936,"statfs":{"total":95995035648,"available":95309758464,"internally_reserved":0,"allocated":552423424,"data_stored":551959038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117181,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1738988,"kb_used_data":1464864,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92006164,"statfs":{"total":95995035648,"available":94214311936,"internally_reserved":0,"allocated":1500020736,"data_stored":1499550296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":280689095},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920951,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1402272,"kb_used_data":1100372,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92342880,"statfs":{"total":95995035648,"available":94559109120,"internally_reserved":0,"allocated":1126780928,"data_stored":1126316542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757426,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":784160,"kb_used_data":532244,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":92960992,"statfs":{"total":95995035648,"available":95192055808,"internally_reserved":0,"allocated":545017856,"data_stored":544553470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":571932672,"data_stored":571932672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":569638912,"data_stored":569638912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1518469120,"data_stored":1518469120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1146421248,"data_stored":1146421248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":558235648,"data_stored":558235648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":859439104,"data_stored":859439104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1482227712,"data_stored":1482227712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":859832320,"data_stored":859832320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1597440000,"data_stored":1597440000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:27:44.092 INFO:tasks.thrashosds.thrasher:Setting 4.6 to [6, 9] 2024-04-23T16:27:44.092 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.6', '6', '9'] 2024-04-23T16:27:44.092 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pg-upmap 4.6 6 9 2024-04-23T16:27:44.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:43 smithi148 ceph-mon[44520]: from='client.15531 v1:172.21.15.103:0/3457600470' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:27:44.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:43 smithi148 ceph-mon[44520]: pgmap v833: 17 pgs: 17 active+clean; 4.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-04-23T16:27:44.155 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:27:44.197 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:44.554 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:27:44.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:44.561 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-23T16:27:44.562 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:27:44.663 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:27:44.701 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:44.725 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:27:44.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:44.731 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:27:44.824 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 27 16 82580 82564 191.071 126.5 0.00685973 0.0052322 2024-04-23T16:27:44.832 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:27:44.874 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:44.901 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:27:44.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:44.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:27:44.962 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:27:45.006 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:27:45.050 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:45.314 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:27:45.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:45.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:27:45.418 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:27:45.453 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:45.714 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:27:45.824 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 28 16 84554 84538 188.653 123.375 0.00717575 0.00529887 2024-04-23T16:27:45.928 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:45 smithi188 ceph-mon[44101]: pgmap v834: 17 pgs: 17 active+clean; 4.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-04-23T16:27:45.928 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:45 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2807327284' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:27:46.009 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:27:46.012 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:45 smithi103 ceph-mon[39629]: pgmap v834: 17 pgs: 17 active+clean; 4.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-04-23T16:27:46.012 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:45 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2807327284' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:27:46.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:46.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:27:46.114 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:27:46.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:45 smithi148 ceph-mon[44520]: pgmap v834: 17 pgs: 17 active+clean; 4.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-04-23T16:27:46.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:45 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2807327284' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:27:46.152 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:46.313 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:27:46.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:46.317 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:27:46.418 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:27:46.460 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:46.711 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:46 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:27:46.711 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:46 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2807327284' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:27:46.711 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:46 smithi103 ceph-mon[39629]: osdmap e290: 12 total, 12 up, 10 in 2024-04-23T16:27:46.825 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 29 16 86520 86504 186.385 122.875 0.00831182 0.0053634 2024-04-23T16:27:46.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:46 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:27:46.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:46 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2807327284' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:27:46.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:46 smithi188 ceph-mon[44101]: osdmap e290: 12 total, 12 up, 10 in 2024-04-23T16:27:46.992 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:27:46.992 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-04-23T16:27:46.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-04-23T16:27:46.992 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:27:46.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:46.993 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208917 3.4 3.eacf0934 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:27:46.993 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085422+0000", 2024-04-23T16:27:46.993 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 175.90222163600001, 2024-04-23T16:27:46.993 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70691126599999998, 2024-04-23T16:27:46.993 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:46.994 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:46.994 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:46.994 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:46.994 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:46.994 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:46.994 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208917 2024-04-23T16:27:46.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:46.995 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:46.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:46.995 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:46.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085422+0000", 2024-04-23T16:27:46.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:46.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:46.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:46.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:46.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085422+0000", 2024-04-23T16:27:46.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:46.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:46.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:46.997 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:46.997 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085423+0000", 2024-04-23T16:27:46.997 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2309999999999999e-06 2024-04-23T16:27:46.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:46.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:46.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:46.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085428+0000", 2024-04-23T16:27:46.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2870000000000004e-06 2024-04-23T16:27:46.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:46.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:46.999 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:46.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085437+0000", 2024-04-23T16:27:46.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2150000000000007e-06 2024-04-23T16:27:47.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.000 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.000 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085444+0000", 2024-04-23T16:27:47.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9179999999999997e-06 2024-04-23T16:27:47.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.000 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085498+0000", 2024-04-23T16:27:47.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4088000000000001e-05 2024-04-23T16:27:47.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085509+0000", 2024-04-23T16:27:47.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1009000000000001e-05 2024-04-23T16:27:47.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792295+0000", 2024-04-23T16:27:47.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70678534800000004 2024-04-23T16:27:47.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792333+0000", 2024-04-23T16:27:47.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8170000000000002e-05 2024-04-23T16:27:47.002 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.002 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.002 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.002 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208925 3.8 3.a4efd988 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:27:47.002 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085458+0000", 2024-04-23T16:27:47.002 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 175.902185777, 2024-04-23T16:27:47.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70667339100000004, 2024-04-23T16:27:47.003 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:47.003 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:47.003 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:47.003 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:47.003 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:47.003 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:47.003 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208925 2024-04-23T16:27:47.003 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.003 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:47.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:47.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085458+0000", 2024-04-23T16:27:47.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:47.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085458+0000", 2024-04-23T16:27:47.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:47.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085458+0000", 2024-04-23T16:27:47.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6000000000000003e-07 2024-04-23T16:27:47.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.005 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.005 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:47.005 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085462+0000", 2024-04-23T16:27:47.005 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4450000000000001e-06 2024-04-23T16:27:47.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.005 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.006 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:47.006 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085468+0000", 2024-04-23T16:27:47.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4489999999999996e-06 2024-04-23T16:27:47.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.006 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.006 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085473+0000", 2024-04-23T16:27:47.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7260000000000002e-06 2024-04-23T16:27:47.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085516+0000", 2024-04-23T16:27:47.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3013000000000003e-05 2024-04-23T16:27:47.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085523+0000", 2024-04-23T16:27:47.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0169999999999999e-06 2024-04-23T16:27:47.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792056+0000", 2024-04-23T16:27:47.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70653289799999996 2024-04-23T16:27:47.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792131+0000", 2024-04-23T16:27:47.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5183000000000002e-05 2024-04-23T16:27:47.010 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.010 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.011 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.011 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208935 3.4 3.ef358d64 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:27:47.011 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085484+0000", 2024-04-23T16:27:47.011 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 175.902159703, 2024-04-23T16:27:47.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70688482200000002, 2024-04-23T16:27:47.012 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:47.012 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:47.012 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:47.012 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:47.012 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:47.013 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:47.013 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208935 2024-04-23T16:27:47.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.013 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:47.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:47.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085484+0000", 2024-04-23T16:27:47.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:47.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085484+0000", 2024-04-23T16:27:47.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:47.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085484+0000", 2024-04-23T16:27:47.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8500000000000001e-07 2024-04-23T16:27:47.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:47.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085488+0000", 2024-04-23T16:27:47.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3009999999999998e-06 2024-04-23T16:27:47.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:47.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085492+0000", 2024-04-23T16:27:47.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1749999999999998e-06 2024-04-23T16:27:47.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085496+0000", 2024-04-23T16:27:47.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4190000000000004e-06 2024-04-23T16:27:47.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085542+0000", 2024-04-23T16:27:47.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5799e-05 2024-04-23T16:27:47.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085551+0000", 2024-04-23T16:27:47.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0860000000000008e-06 2024-04-23T16:27:47.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792360+0000", 2024-04-23T16:27:47.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70680912900000004 2024-04-23T16:27:47.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792368+0000", 2024-04-23T16:27:47.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2279999999999998e-06 2024-04-23T16:27:47.019 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.019 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.019 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.019 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208938 3.4 3.b83bb9a4 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:27:47.019 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085502+0000", 2024-04-23T16:27:47.020 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 175.90214104699999, 2024-04-23T16:27:47.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70692120599999997, 2024-04-23T16:27:47.020 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:47.020 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:47.020 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:47.020 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:47.020 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:47.020 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:47.020 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208938 2024-04-23T16:27:47.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.020 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:47.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:47.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085502+0000", 2024-04-23T16:27:47.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:47.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085502+0000", 2024-04-23T16:27:47.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:47.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085503+0000", 2024-04-23T16:27:47.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2300000000000001e-07 2024-04-23T16:27:47.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085505+0000", 2024-04-23T16:27:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.469e-06 2024-04-23T16:27:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085511+0000", 2024-04-23T16:27:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7819999999999999e-06 2024-04-23T16:27:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085517+0000", 2024-04-23T16:27:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7740000000000001e-06 2024-04-23T16:27:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085578+0000", 2024-04-23T16:27:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1475000000000003e-05 2024-04-23T16:27:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085586+0000", 2024-04-23T16:27:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8140000000000001e-06 2024-04-23T16:27:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792413+0000", 2024-04-23T16:27:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70682651100000005 2024-04-23T16:27:47.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792424+0000", 2024-04-23T16:27:47.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0858e-05 2024-04-23T16:27:47.024 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.024 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.024 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.025 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208939 3.8 3.5b6d9248 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:27:47.025 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085539+0000", 2024-04-23T16:27:47.025 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 175.902104818, 2024-04-23T16:27:47.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70665678499999995, 2024-04-23T16:27:47.025 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:47.025 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:47.025 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:47.025 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:47.025 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:47.026 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:47.026 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208939 2024-04-23T16:27:47.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.026 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:47.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:47.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085539+0000", 2024-04-23T16:27:47.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:47.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085539+0000", 2024-04-23T16:27:47.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:47.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085539+0000", 2024-04-23T16:27:47.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0200000000000001e-07 2024-04-23T16:27:47.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:47.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085543+0000", 2024-04-23T16:27:47.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4209999999999999e-06 2024-04-23T16:27:47.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:47.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085549+0000", 2024-04-23T16:27:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.959e-06 2024-04-23T16:27:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085554+0000", 2024-04-23T16:27:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7189999999999998e-06 2024-04-23T16:27:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085587+0000", 2024-04-23T16:27:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2540999999999999e-05 2024-04-23T16:27:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085591+0000", 2024-04-23T16:27:47.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8510000000000001e-06 2024-04-23T16:27:47.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792182+0000", 2024-04-23T16:27:47.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70659176199999996 2024-04-23T16:27:47.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792195+0000", 2024-04-23T16:27:47.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2830000000000001e-05 2024-04-23T16:27:47.029 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.029 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.030 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.030 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208948 3.8 3.9d5e98c8 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:27:47.030 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085565+0000", 2024-04-23T16:27:47.030 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 175.90207841599999, 2024-04-23T16:27:47.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70682216099999995, 2024-04-23T16:27:47.030 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:47.030 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:47.030 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:47.030 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:47.030 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:47.030 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:47.030 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208948 2024-04-23T16:27:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085565+0000", 2024-04-23T16:27:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085565+0000", 2024-04-23T16:27:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085566+0000", 2024-04-23T16:27:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2600000000000002e-07 2024-04-23T16:27:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085568+0000", 2024-04-23T16:27:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7769999999999998e-06 2024-04-23T16:27:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085575+0000", 2024-04-23T16:27:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0230000000000004e-06 2024-04-23T16:27:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085580+0000", 2024-04-23T16:27:47.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5499999999999996e-06 2024-04-23T16:27:47.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085609+0000", 2024-04-23T16:27:47.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9332999999999999e-05 2024-04-23T16:27:47.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085613+0000", 2024-04-23T16:27:47.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7799999999999998e-06 2024-04-23T16:27:47.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792374+0000", 2024-04-23T16:27:47.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70676071600000001 2024-04-23T16:27:47.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792387+0000", 2024-04-23T16:27:47.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3256e-05 2024-04-23T16:27:47.034 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.034 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.034 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208949 3.8 3.32f0e298 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:27:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085593+0000", 2024-04-23T16:27:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 175.90204984900001, 2024-04-23T16:27:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70685236200000001, 2024-04-23T16:27:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208949 2024-04-23T16:27:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.036 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:47.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:47.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085593+0000", 2024-04-23T16:27:47.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:47.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085593+0000", 2024-04-23T16:27:47.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:47.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085594+0000", 2024-04-23T16:27:47.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4000000000000002e-07 2024-04-23T16:27:47.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:47.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085596+0000", 2024-04-23T16:27:47.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4480000000000001e-06 2024-04-23T16:27:47.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:47.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085603+0000", 2024-04-23T16:27:47.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2899999999999999e-06 2024-04-23T16:27:47.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085607+0000", 2024-04-23T16:27:47.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3610000000000003e-06 2024-04-23T16:27:47.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085651+0000", 2024-04-23T16:27:47.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4075000000000001e-05 2024-04-23T16:27:47.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085659+0000", 2024-04-23T16:27:47.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1079999999999993e-06 2024-04-23T16:27:47.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792436+0000", 2024-04-23T16:27:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70677668500000002 2024-04-23T16:27:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792446+0000", 2024-04-23T16:27:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9550000000000004e-06 2024-04-23T16:27:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.041 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208950 3.4 3.4636ca44 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:27:47.041 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085613+0000", 2024-04-23T16:27:47.041 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 175.902030438, 2024-04-23T16:27:47.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70686611700000002, 2024-04-23T16:27:47.041 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:47.041 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:47.041 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:47.041 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:47.041 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:47.041 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:47.041 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208950 2024-04-23T16:27:47.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.041 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085613+0000", 2024-04-23T16:27:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085613+0000", 2024-04-23T16:27:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085613+0000", 2024-04-23T16:27:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4799999999999999e-07 2024-04-23T16:27:47.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:47.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085615+0000", 2024-04-23T16:27:47.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.951e-06 2024-04-23T16:27:47.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:47.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085621+0000", 2024-04-23T16:27:47.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6890000000000001e-06 2024-04-23T16:27:47.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085624+0000", 2024-04-23T16:27:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4960000000000001e-06 2024-04-23T16:27:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085659+0000", 2024-04-23T16:27:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4516e-05 2024-04-23T16:27:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085665+0000", 2024-04-23T16:27:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.643e-06 2024-04-23T16:27:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792468+0000", 2024-04-23T16:27:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70680334700000003 2024-04-23T16:27:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792479+0000", 2024-04-23T16:27:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1127e-05 2024-04-23T16:27:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208951 3.4 3.7f30f4f4 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:27:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085630+0000", 2024-04-23T16:27:47.046 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 175.90201292, 2024-04-23T16:27:47.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70690233800000002, 2024-04-23T16:27:47.046 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:47.046 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:47.046 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:47.046 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:47.046 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:47.046 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:47.046 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208951 2024-04-23T16:27:47.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.046 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:47.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:47.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085630+0000", 2024-04-23T16:27:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085630+0000", 2024-04-23T16:27:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085631+0000", 2024-04-23T16:27:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8299999999999997e-07 2024-04-23T16:27:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:47.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085633+0000", 2024-04-23T16:27:47.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3970000000000001e-06 2024-04-23T16:27:47.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:47.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085639+0000", 2024-04-23T16:27:47.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7810000000000002e-06 2024-04-23T16:27:47.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085644+0000", 2024-04-23T16:27:47.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7299999999999996e-06 2024-04-23T16:27:47.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085693+0000", 2024-04-23T16:27:47.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9537999999999999e-05 2024-04-23T16:27:47.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085702+0000", 2024-04-23T16:27:47.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1309999999999995e-06 2024-04-23T16:27:47.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792520+0000", 2024-04-23T16:27:47.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70681792300000001 2024-04-23T16:27:47.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792533+0000", 2024-04-23T16:27:47.050 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2354999999999999e-05 2024-04-23T16:27:47.050 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.050 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.050 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.050 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208989 3.8 3.631ac508 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:27:47.050 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085652+0000", 2024-04-23T16:27:47.050 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 175.901991337, 2024-04-23T16:27:47.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70683001599999995, 2024-04-23T16:27:47.051 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:47.051 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:47.051 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:47.051 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:47.051 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:47.051 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:47.051 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208989 2024-04-23T16:27:47.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.051 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:47.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:47.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085652+0000", 2024-04-23T16:27:47.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:47.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085652+0000", 2024-04-23T16:27:47.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:47.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085653+0000", 2024-04-23T16:27:47.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9699999999999996e-07 2024-04-23T16:27:47.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:47.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085655+0000", 2024-04-23T16:27:47.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7810000000000001e-06 2024-04-23T16:27:47.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:47.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085661+0000", 2024-04-23T16:27:47.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1930000000000001e-06 2024-04-23T16:27:47.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085666+0000", 2024-04-23T16:27:47.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6579999999999999e-06 2024-04-23T16:27:47.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085695+0000", 2024-04-23T16:27:47.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8711999999999999e-05 2024-04-23T16:27:47.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085699+0000", 2024-04-23T16:27:47.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1500000000000001e-06 2024-04-23T16:27:47.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792474+0000", 2024-04-23T16:27:47.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70677516500000004 2024-04-23T16:27:47.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792482+0000", 2024-04-23T16:27:47.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7600000000000002e-06 2024-04-23T16:27:47.055 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.055 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.055 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.055 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208990 3.4 3.b71d3f14 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:27:47.055 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085676+0000", 2024-04-23T16:27:47.055 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 175.901967351, 2024-04-23T16:27:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70695150200000001, 2024-04-23T16:27:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208990 2024-04-23T16:27:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085676+0000", 2024-04-23T16:27:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085676+0000", 2024-04-23T16:27:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085677+0000", 2024-04-23T16:27:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6300000000000005e-07 2024-04-23T16:27:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085680+0000", 2024-04-23T16:27:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9799999999999998e-06 2024-04-23T16:27:47.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:47.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085686+0000", 2024-04-23T16:27:47.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9669999999999998e-06 2024-04-23T16:27:47.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085691+0000", 2024-04-23T16:27:47.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2000000000000002e-06 2024-04-23T16:27:47.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085730+0000", 2024-04-23T16:27:47.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9388999999999998e-05 2024-04-23T16:27:47.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085737+0000", 2024-04-23T16:27:47.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7129999999999999e-06 2024-04-23T16:27:47.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792616+0000", 2024-04-23T16:27:47.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70687957599999995 2024-04-23T16:27:47.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792627+0000", 2024-04-23T16:27:47.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1014e-05 2024-04-23T16:27:47.060 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.060 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.060 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.060 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208925 3.8 3.a4efd988 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:27:47.061 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820364+0000", 2024-04-23T16:27:47.061 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 174.167279044, 2024-04-23T16:27:47.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55176175500000002, 2024-04-23T16:27:47.061 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:47.061 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:47.061 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:47.061 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:47.061 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:47.061 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:47.061 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208925 2024-04-23T16:27:47.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.061 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:47.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:47.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820364+0000", 2024-04-23T16:27:47.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:47.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820364+0000", 2024-04-23T16:27:47.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:47.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820368+0000", 2024-04-23T16:27:47.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8060000000000001e-06 2024-04-23T16:27:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820379+0000", 2024-04-23T16:27:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0634e-05 2024-04-23T16:27:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820394+0000", 2024-04-23T16:27:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5554e-05 2024-04-23T16:27:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820408+0000", 2024-04-23T16:27:47.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3508e-05 2024-04-23T16:27:47.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820486+0000", 2024-04-23T16:27:47.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7844999999999998e-05 2024-04-23T16:27:47.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820495+0000", 2024-04-23T16:27:47.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3610000000000007e-06 2024-04-23T16:27:47.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372103+0000", 2024-04-23T16:27:47.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55160793100000005 2024-04-23T16:27:47.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372126+0000", 2024-04-23T16:27:47.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3116e-05 2024-04-23T16:27:47.065 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.065 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.065 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.065 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208939 3.8 3.5b6d9248 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:27:47.066 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820453+0000", 2024-04-23T16:27:47.066 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 174.16719003099999, 2024-04-23T16:27:47.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55183821600000005, 2024-04-23T16:27:47.066 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:47.067 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:47.067 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:47.067 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:47.068 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:47.068 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:47.068 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208939 2024-04-23T16:27:47.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.068 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:47.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:47.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820453+0000", 2024-04-23T16:27:47.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820453+0000", 2024-04-23T16:27:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820454+0000", 2024-04-23T16:27:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0270000000000001e-06 2024-04-23T16:27:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820458+0000", 2024-04-23T16:27:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6430000000000001e-06 2024-04-23T16:27:47.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:47.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820465+0000", 2024-04-23T16:27:47.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2679999999999997e-06 2024-04-23T16:27:47.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820471+0000", 2024-04-23T16:27:47.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9529999999999996e-06 2024-04-23T16:27:47.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820615+0000", 2024-04-23T16:27:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014426700000000001 2024-04-23T16:27:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820622+0000", 2024-04-23T16:27:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2570000000000004e-06 2024-04-23T16:27:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372284+0000", 2024-04-23T16:27:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55166234199999997 2024-04-23T16:27:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372292+0000", 2024-04-23T16:27:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4590000000000001e-06 2024-04-23T16:27:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208948 3.8 3.9d5e98c8 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:27:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820480+0000", 2024-04-23T16:27:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 174.167163754, 2024-04-23T16:27:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55184634399999999, 2024-04-23T16:27:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208948 2024-04-23T16:27:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820480+0000", 2024-04-23T16:27:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:47.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820480+0000", 2024-04-23T16:27:47.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:47.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820480+0000", 2024-04-23T16:27:47.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4199999999999996e-07 2024-04-23T16:27:47.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:47.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820482+0000", 2024-04-23T16:27:47.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1919999999999999e-06 2024-04-23T16:27:47.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820487+0000", 2024-04-23T16:27:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7990000000000001e-06 2024-04-23T16:27:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820492+0000", 2024-04-23T16:27:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1869999999999996e-06 2024-04-23T16:27:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820654+0000", 2024-04-23T16:27:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016204500000000001 2024-04-23T16:27:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820659+0000", 2024-04-23T16:27:47.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8019999999999999e-06 2024-04-23T16:27:47.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372321+0000", 2024-04-23T16:27:47.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.551661548 2024-04-23T16:27:47.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372326+0000", 2024-04-23T16:27:47.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2290000000000002e-06 2024-04-23T16:27:47.077 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.077 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.077 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.077 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208949 3.8 3.32f0e298 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:27:47.077 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820498+0000", 2024-04-23T16:27:47.077 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 174.16714494999999, 2024-04-23T16:27:47.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55185574800000003, 2024-04-23T16:27:47.077 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:47.078 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:47.078 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:47.078 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:47.078 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:47.078 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:47.078 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208949 2024-04-23T16:27:47.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.079 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:47.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:47.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820498+0000", 2024-04-23T16:27:47.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:47.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820498+0000", 2024-04-23T16:27:47.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820499+0000", 2024-04-23T16:27:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6499999999999999e-07 2024-04-23T16:27:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820501+0000", 2024-04-23T16:27:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1090000000000001e-06 2024-04-23T16:27:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820506+0000", 2024-04-23T16:27:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.305e-06 2024-04-23T16:27:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820511+0000", 2024-04-23T16:27:47.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1869999999999996e-06 2024-04-23T16:27:47.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820688+0000", 2024-04-23T16:27:47.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000176353 2024-04-23T16:27:47.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820696+0000", 2024-04-23T16:27:47.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9410000000000007e-06 2024-04-23T16:27:47.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372350+0000", 2024-04-23T16:27:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55165407099999997 2024-04-23T16:27:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372354+0000", 2024-04-23T16:27:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3170000000000003e-06 2024-04-23T16:27:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.083 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208989 3.8 3.631ac508 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:27:47.083 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820517+0000", 2024-04-23T16:27:47.083 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 174.16712644699999, 2024-04-23T16:27:47.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55195477900000001, 2024-04-23T16:27:47.083 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:47.083 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:47.083 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:47.083 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:47.083 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:47.083 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:47.083 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208989 2024-04-23T16:27:47.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.083 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:47.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:47.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820517+0000", 2024-04-23T16:27:47.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:47.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820517+0000", 2024-04-23T16:27:47.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:47.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820517+0000", 2024-04-23T16:27:47.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2500000000000001e-07 2024-04-23T16:27:47.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:47.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820519+0000", 2024-04-23T16:27:47.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0760000000000001e-06 2024-04-23T16:27:47.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:47.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820524+0000", 2024-04-23T16:27:47.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.814e-06 2024-04-23T16:27:47.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820529+0000", 2024-04-23T16:27:47.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5809999999999998e-06 2024-04-23T16:27:47.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820738+0000", 2024-04-23T16:27:47.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020955600000000001 2024-04-23T16:27:47.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820748+0000", 2024-04-23T16:27:47.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9760000000000007e-06 2024-04-23T16:27:47.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372466+0000", 2024-04-23T16:27:47.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55171814399999997 2024-04-23T16:27:47.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372472+0000", 2024-04-23T16:27:47.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2070000000000003e-06 2024-04-23T16:27:47.087 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.087 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.087 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208935 3.4 3.ef358d64 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:27:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820748+0000", 2024-04-23T16:27:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 174.166895727, 2024-04-23T16:27:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.12019948499999999, 2024-04-23T16:27:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208935 2024-04-23T16:27:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:47.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:47.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820748+0000", 2024-04-23T16:27:47.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:47.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820748+0000", 2024-04-23T16:27:47.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:47.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820749+0000", 2024-04-23T16:27:47.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2619999999999999e-06 2024-04-23T16:27:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820753+0000", 2024-04-23T16:27:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5839999999999996e-06 2024-04-23T16:27:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820762+0000", 2024-04-23T16:27:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0439999999999996e-06 2024-04-23T16:27:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820770+0000", 2024-04-23T16:27:47.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3280000000000006e-06 2024-04-23T16:27:47.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820865+0000", 2024-04-23T16:27:47.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5311999999999998e-05 2024-04-23T16:27:47.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820882+0000", 2024-04-23T16:27:47.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7266999999999999e-05 2024-04-23T16:27:47.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.940935+0000", 2024-04-23T16:27:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.120052908 2024-04-23T16:27:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.940947+0000", 2024-04-23T16:27:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1780000000000001e-05 2024-04-23T16:27:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208938 3.4 3.b83bb9a4 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:27:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820786+0000", 2024-04-23T16:27:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 174.16685782900001, 2024-04-23T16:27:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.120213972, 2024-04-23T16:27:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208938 2024-04-23T16:27:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820786+0000", 2024-04-23T16:27:47.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:47.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820786+0000", 2024-04-23T16:27:47.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:47.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820787+0000", 2024-04-23T16:27:47.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.094e-06 2024-04-23T16:27:47.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:47.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820791+0000", 2024-04-23T16:27:47.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4700000000000004e-06 2024-04-23T16:27:47.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:47.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820800+0000", 2024-04-23T16:27:47.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4059999999999994e-06 2024-04-23T16:27:47.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820809+0000", 2024-04-23T16:27:47.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1689999999999992e-06 2024-04-23T16:27:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820932+0000", 2024-04-23T16:27:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012330800000000001 2024-04-23T16:27:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820942+0000", 2024-04-23T16:27:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9480000000000003e-06 2024-04-23T16:27:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.940991+0000", 2024-04-23T16:27:47.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.120049106 2024-04-23T16:27:47.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.940999+0000", 2024-04-23T16:27:47.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4710000000000007e-06 2024-04-23T16:27:47.097 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.097 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.097 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.097 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208951 3.4 3.7f30f4f4 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:27:47.097 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820918+0000", 2024-04-23T16:27:47.097 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 174.166725434, 2024-04-23T16:27:47.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.120268738, 2024-04-23T16:27:47.098 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:47.098 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:47.098 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:47.098 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:47.098 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:47.098 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:47.098 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208951 2024-04-23T16:27:47.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.098 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:47.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:47.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820918+0000", 2024-04-23T16:27:47.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:47.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820918+0000", 2024-04-23T16:27:47.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:47.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820919+0000", 2024-04-23T16:27:47.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9500000000000001e-07 2024-04-23T16:27:47.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820925+0000", 2024-04-23T16:27:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.835e-06 2024-04-23T16:27:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820933+0000", 2024-04-23T16:27:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3739999999999998e-06 2024-04-23T16:27:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820941+0000", 2024-04-23T16:27:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8220000000000007e-06 2024-04-23T16:27:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.821053+0000", 2024-04-23T16:27:47.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000112617 2024-04-23T16:27:47.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.821059+0000", 2024-04-23T16:27:47.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1479999999999997e-06 2024-04-23T16:27:47.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.941182+0000", 2024-04-23T16:27:47.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.12012339399999999 2024-04-23T16:27:47.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.941187+0000", 2024-04-23T16:27:47.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6530000000000002e-06 2024-04-23T16:27:47.102 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.102 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.102 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.102 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208990 3.4 3.b71d3f14 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:27:47.102 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820957+0000", 2024-04-23T16:27:47.102 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 174.166686172, 2024-04-23T16:27:47.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.120262297, 2024-04-23T16:27:47.102 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:27:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:27:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:27:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:27:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:27:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:27:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208990 2024-04-23T16:27:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:27:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:27:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820957+0000", 2024-04-23T16:27:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:27:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820957+0000", 2024-04-23T16:27:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:27:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:27:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820959+0000", 2024-04-23T16:27:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3430000000000001e-06 2024-04-23T16:27:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:27:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820963+0000", 2024-04-23T16:27:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2400000000000001e-06 2024-04-23T16:27:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:27:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820975+0000", 2024-04-23T16:27:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1783999999999999e-05 2024-04-23T16:27:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:27:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820985+0000", 2024-04-23T16:27:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0105000000000001e-05 2024-04-23T16:27:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.821085+0000", 2024-04-23T16:27:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100806 2024-04-23T16:27:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:27:47.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.821102+0000", 2024-04-23T16:27:47.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6283e-05 2024-04-23T16:27:47.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:27:47.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.941213+0000", 2024-04-23T16:27:47.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.12011137199999999 2024-04-23T16:27:47.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:27:47.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:27:47.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:27:47.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.941219+0000", 2024-04-23T16:27:47.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3640000000000004e-06 2024-04-23T16:27:47.107 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.107 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.107 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.107 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:27:47.107 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:27:47.107 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:27:47.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:46 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:27:47.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:46 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2807327284' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:27:47.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:46 smithi148 ceph-mon[44520]: osdmap e290: 12 total, 12 up, 10 in 2024-04-23T16:27:47.764 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-23T16:27:47.824 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 30 16 88525 88509 184.349 125.312 0.00937953 0.005423 2024-04-23T16:27:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:47 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:27:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:47 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:27:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:47 smithi188 ceph-mon[44101]: pgmap v836: 17 pgs: 17 active+clean; 5.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-04-23T16:27:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:47 smithi188 ceph-mon[44101]: osdmap e291: 12 total, 12 up, 10 in 2024-04-23T16:27:48.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:47 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:27:48.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:47 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:27:48.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:47 smithi103 ceph-mon[39629]: pgmap v836: 17 pgs: 17 active+clean; 5.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-04-23T16:27:48.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:47 smithi103 ceph-mon[39629]: osdmap e291: 12 total, 12 up, 10 in 2024-04-23T16:27:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:47 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:27:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:47 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:27:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:47 smithi148 ceph-mon[44520]: pgmap v836: 17 pgs: 17 active+clean; 5.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-04-23T16:27:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:47 smithi148 ceph-mon[44520]: osdmap e291: 12 total, 12 up, 10 in 2024-04-23T16:27:48.126 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:27:48.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:48.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:27:48.187 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:27:48.232 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:27:48.269 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:48.294 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:27:48.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:48.298 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:27:48.399 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:27:48.454 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:48.469 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:27:48.478 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:27:48.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:48.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:27:48.583 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:27:48.621 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:48.654 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:27:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:48.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:27:48.759 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:27:48.797 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:48.819 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:27:48.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:48.823 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:27:48.824 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 31 15 90424 90409 182.233 118.75 0.0135347 0.00548578 2024-04-23T16:27:48.924 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:27:48.972 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:49.000 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:27:49.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:49.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:27:49.107 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:27:49.167 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:49.192 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:27:49.219 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:27:49.247 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:27:49.274 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:27:49.309 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:27:49.341 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:27:49.371 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:27:49.378 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:27:49.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:49.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:27:49.405 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:27:49.435 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:27:49.459 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:27:49.484 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:27:49.487 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:27:49.518 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:27:49.524 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:27:49.524 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:27:49.524 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:27:49.525 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:27:49.525 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:27:49.525 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:27:49.525 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:27:49.525 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:27:49.525 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:27:49.525 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:27:49.525 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:27:49.525 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:27:49.525 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:49.731 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:27:49.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:49.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:27:49.824 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 32 16 92330 92314 180.259 119.062 0.00639526 0.00554631 2024-04-23T16:27:49.835 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:27:49.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:49 smithi103 ceph-mon[39629]: pgmap v838: 17 pgs: 17 active+clean; 5.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-04-23T16:27:49.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:49 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1866023705' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:27:49.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:49.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:49.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:49.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:49.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:49.888 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:49.915 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:27:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:49.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:27:49.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:49 smithi188 ceph-mon[44101]: pgmap v838: 17 pgs: 17 active+clean; 5.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-04-23T16:27:49.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:49 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1866023705' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:27:49.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:49.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:49.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:49.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:49.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:50.027 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:27:50.071 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:49 smithi148 ceph-mon[44520]: pgmap v838: 17 pgs: 17 active+clean; 5.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-04-23T16:27:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:49 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1866023705' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:27:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:50.176 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:27:50.814 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:27:50.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:50.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:27:50.824 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 33 16 94233 94217 178.401 118.938 0.00996748 0.00560387 2024-04-23T16:27:50.917 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:27:50.948 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:50.975 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:27:50.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:50.981 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:27:51.082 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:27:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.115 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:51.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:27:51.139 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:27:51.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:51.143 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:27:51.244 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:27:51.275 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:51.311 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:27:51.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:51.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:27:51.420 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:27:51.458 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:51.824 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 34 16 96117 96101 176.617 117.75 0.0040929 0.00566053 2024-04-23T16:27:52.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:51 smithi103 ceph-mon[39629]: pgmap v839: 17 pgs: 17 active+clean; 5.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 139 MiB/s wr, 2.22k op/s 2024-04-23T16:27:52.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:51 smithi148 ceph-mon[44520]: pgmap v839: 17 pgs: 17 active+clean; 5.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 139 MiB/s wr, 2.22k op/s 2024-04-23T16:27:52.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:51 smithi188 ceph-mon[44101]: pgmap v839: 17 pgs: 17 active+clean; 5.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 139 MiB/s wr, 2.22k op/s 2024-04-23T16:27:52.665 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:27:52.715 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:27:52.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:52.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:27:52.766 INFO:teuthology.orchestra.run.smithi103.stderr:set 4.6 pg_upmap mapping to [6,9] 2024-04-23T16:27:52.824 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:27:52.825 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 35 16 97936 97920 174.819 113.688 0.00419085 0.0057165 2024-04-23T16:27:52.854 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:52.877 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:27:52.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:52.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:27:52.981 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:27:53.006 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:52 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:27:53.007 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:52 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:27:53.007 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:52 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2968363455' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [6, 9]}]: dispatch 2024-04-23T16:27:53.007 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:52 smithi148 ceph-mon[44520]: from='client.25411 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [6, 9]}]: dispatch 2024-04-23T16:27:53.007 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:53.026 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:27:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:53.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:27:53.029 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:52 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:27:53.029 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:52 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:27:53.029 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:52 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2968363455' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [6, 9]}]: dispatch 2024-04-23T16:27:53.029 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:52 smithi103 ceph-mon[39629]: from='client.25411 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [6, 9]}]: dispatch 2024-04-23T16:27:53.130 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:27:53.156 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:53.174 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:27:53.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:53.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:27:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:52 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:27:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:52 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:27:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:52 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2968363455' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [6, 9]}]: dispatch 2024-04-23T16:27:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:52 smithi188 ceph-mon[44101]: from='client.25411 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [6, 9]}]: dispatch 2024-04-23T16:27:53.278 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:27:53.305 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:53.324 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:27:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:53.326 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:27:53.427 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:27:53.457 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:53.477 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:27:53.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:53.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:27:53.580 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:27:53.608 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:53.705 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:27:53.705 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:27:53.706 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:27:53.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:53.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:27:53.810 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:27:53.825 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 36 16 98084 98068 170.221 9.25 0.00324971 0.00571496 2024-04-23T16:27:53.838 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:53.856 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:27:53.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:53.859 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:27:53.870 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:27:53.959 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:27:53.993 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:54.013 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:27:54.015 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:53 smithi148 ceph-mon[44520]: pgmap v840: 17 pgs: 17 active+clean; 5.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-04-23T16:27:54.015 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:53 smithi148 ceph-mon[44520]: from='client.25411 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [6, 9]}]': finished 2024-04-23T16:27:54.015 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:53 smithi148 ceph-mon[44520]: osdmap e292: 12 total, 12 up, 10 in 2024-04-23T16:27:54.015 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:53 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:27:54.015 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:53 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:27:54.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:54.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:27:54.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:53 smithi103 ceph-mon[39629]: pgmap v840: 17 pgs: 17 active+clean; 5.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-04-23T16:27:54.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:53 smithi103 ceph-mon[39629]: from='client.25411 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [6, 9]}]': finished 2024-04-23T16:27:54.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:53 smithi103 ceph-mon[39629]: osdmap e292: 12 total, 12 up, 10 in 2024-04-23T16:27:54.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:53 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:27:54.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:53 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:27:54.117 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:27:54.145 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:53 smithi188 ceph-mon[44101]: pgmap v840: 17 pgs: 17 active+clean; 5.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-04-23T16:27:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:53 smithi188 ceph-mon[44101]: from='client.25411 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [6, 9]}]': finished 2024-04-23T16:27:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:53 smithi188 ceph-mon[44101]: osdmap e292: 12 total, 12 up, 10 in 2024-04-23T16:27:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:53 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:27:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:53 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:27:54.593 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:27:54.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:54.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:27:54.697 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:27:54.729 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:54.751 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:27:54.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:54.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:27:54.825 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 37 16 98084 98068 165.621 0 - 0.00571496 2024-04-23T16:27:54.854 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:27:54.883 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:54.903 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:27:54.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:54.906 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:27:55.010 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:27:55.035 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:54 smithi103 ceph-mon[39629]: osdmap e293: 12 total, 12 up, 10 in 2024-04-23T16:27:55.036 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:55.090 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:27:55.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:55.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:27:55.116 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-23T16:27:55.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:54 smithi148 ceph-mon[44520]: osdmap e293: 12 total, 12 up, 10 in 2024-04-23T16:27:55.194 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:27:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:54 smithi188 ceph-mon[44101]: osdmap e293: 12 total, 12 up, 10 in 2024-04-23T16:27:55.226 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:55.248 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:27:55.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:55.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:27:55.352 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:27:55.382 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:55.460 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:27:55.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:55.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:27:55.566 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:27:55.600 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:55.622 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:27:55.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:55.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:27:55.731 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:27:55.770 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:55 smithi103 ceph-mon[39629]: pgmap v843: 17 pgs: 17 active+clean; 5.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 110 MiB/s wr, 1.77k op/s 2024-04-23T16:27:55.770 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:55 smithi103 ceph-mon[39629]: osdmap e294: 12 total, 12 up, 10 in 2024-04-23T16:27:55.772 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:55.825 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 38 16 98748 98732 162.354 20.75 0.0328193 0.00615773 2024-04-23T16:27:55.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:55 smithi188 ceph-mon[44101]: pgmap v843: 17 pgs: 17 active+clean; 5.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 110 MiB/s wr, 1.77k op/s 2024-04-23T16:27:55.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:55 smithi188 ceph-mon[44101]: osdmap e294: 12 total, 12 up, 10 in 2024-04-23T16:27:55.981 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:27:55.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:55.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:27:56.016 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:27:56.087 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:27:56.088 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:27:56.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:55 smithi148 ceph-mon[44520]: pgmap v843: 17 pgs: 17 active+clean; 5.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 110 MiB/s wr, 1.77k op/s 2024-04-23T16:27:56.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:55 smithi148 ceph-mon[44520]: osdmap e294: 12 total, 12 up, 10 in 2024-04-23T16:27:56.134 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:56.170 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:27:56.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:56.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:27:56.275 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:27:56.332 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:56.353 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:27:56.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:56.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:27:56.457 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:27:56.501 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:56.703 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:27:56.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:56.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:27:56.809 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:27:56.826 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 39 16 100241 100225 160.58 93.3125 0.00821927 0.00622636 2024-04-23T16:27:56.843 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:56.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:56.871 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:27:56.871 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:27:56.972 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:27:57.009 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:57.826 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:27:57.826190 min lat: 0.00105502 max lat: 2.61652 avg lat: 0.00628972 2024-04-23T16:27:57.826 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:27:57.826 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 40 16 101758 101742 158.936 94.8125 0.0103462 0.00628972 2024-04-23T16:27:57.972 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:27:57.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:57.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:27:58.081 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:27:58.106 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:57 smithi103 ceph-mon[39629]: pgmap v845: 17 pgs: 17 active+clean; 6.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-04-23T16:27:58.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:57 smithi148 ceph-mon[44520]: pgmap v845: 17 pgs: 17 active+clean; 6.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-04-23T16:27:58.133 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:58.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:57 smithi188 ceph-mon[44101]: pgmap v845: 17 pgs: 17 active+clean; 6.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-04-23T16:27:58.253 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:27:58.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:58.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:27:58.362 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:27:58.388 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:58.414 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:27:58.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:58.419 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:27:58.520 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:27:58.566 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:58.601 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:27:58.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:58.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:27:58.709 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:27:58.754 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:58.784 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:27:58.786 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:27:58.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:58.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:27:58.826 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 41 16 103151 103135 157.183 87.0625 0.00496762 0.00636052 2024-04-23T16:27:58.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:58 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4209127582' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:27:58.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:58 smithi148 ceph-mon[44520]: from='client.25417 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:27:58.890 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:27:58.942 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:58.970 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:27:58.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:58.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:27:59.079 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:27:59.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:58 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4209127582' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:27:59.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:58 smithi103 ceph-mon[39629]: from='client.25417 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:27:59.106 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:58 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4209127582' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:27:59.107 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:58 smithi188 ceph-mon[44101]: from='client.25417 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:27:59.111 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:59.138 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:27:59.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:59.141 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:27:59.242 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:27:59.273 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:59.296 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:27:59.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:59.299 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:27:59.400 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:27:59.433 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:59.462 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:27:59.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:27:59.466 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:27:59.476 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10, 1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1] 2024-04-23T16:27:59.476 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:27:59.477 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-04-23T16:27:59.477 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:27:59.566 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:27:59.618 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:27:59.789 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:59 smithi103 ceph-mon[39629]: pgmap v846: 17 pgs: 17 active+clean; 6.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 61 MiB/s wr, 975 op/s 2024-04-23T16:27:59.826 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 42 16 104530 104514 155.492 86.1875 0.0290069 0.00642742 2024-04-23T16:28:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:59 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:28:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:59 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:28:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:59 smithi103 ceph-mon[39629]: from='client.25417 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:28:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:59 smithi103 ceph-mon[39629]: osdmap e295: 12 total, 12 up, 10 in 2024-04-23T16:28:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:59 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1101702956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:28:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:27:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:00.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:59 smithi148 ceph-mon[44520]: pgmap v846: 17 pgs: 17 active+clean; 6.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 61 MiB/s wr, 975 op/s 2024-04-23T16:28:00.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:59 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:28:00.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:59 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:28:00.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:59 smithi148 ceph-mon[44520]: from='client.25417 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:28:00.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:59 smithi148 ceph-mon[44520]: osdmap e295: 12 total, 12 up, 10 in 2024-04-23T16:28:00.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:59 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1101702956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:28:00.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:00.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:27:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:59 smithi188 ceph-mon[44101]: pgmap v846: 17 pgs: 17 active+clean; 6.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 61 MiB/s wr, 975 op/s 2024-04-23T16:28:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:59 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:28:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:59 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:28:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:59 smithi188 ceph-mon[44101]: from='client.25417 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:28:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:59 smithi188 ceph-mon[44101]: osdmap e295: 12 total, 12 up, 10 in 2024-04-23T16:28:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:59 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1101702956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:28:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:27:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:00.826 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 43 16 105916 105900 153.891 86.625 0.00883758 0.00649579 2024-04-23T16:28:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:01.826 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 44 16 107375 107359 152.466 91.1875 0.00518793 0.00655717 2024-04-23T16:28:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:01 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:28:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:01 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:28:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:01 smithi103 ceph-mon[39629]: pgmap v848: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 91 MiB/s wr, 1.46k op/s; 6064/205306 objects misplaced (2.954%); 838 KiB/s, 13 objects/s recovering 2024-04-23T16:28:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:01 smithi103 ceph-mon[39629]: osdmap e296: 12 total, 12 up, 10 in 2024-04-23T16:28:02.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:01 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:28:02.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:01 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:28:02.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:01 smithi148 ceph-mon[44520]: pgmap v848: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 91 MiB/s wr, 1.46k op/s; 6064/205306 objects misplaced (2.954%); 838 KiB/s, 13 objects/s recovering 2024-04-23T16:28:02.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:01 smithi148 ceph-mon[44520]: osdmap e296: 12 total, 12 up, 10 in 2024-04-23T16:28:02.187 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:28:02.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:02.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:28:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:01 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:28:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:01 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:28:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:01 smithi188 ceph-mon[44101]: pgmap v848: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 91 MiB/s wr, 1.46k op/s; 6064/205306 objects misplaced (2.954%); 838 KiB/s, 13 objects/s recovering 2024-04-23T16:28:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:01 smithi188 ceph-mon[44101]: osdmap e296: 12 total, 12 up, 10 in 2024-04-23T16:28:02.292 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:28:02.326 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:02.827 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 45 16 108770 108754 151.015 87.1875 0.0105185 0.00662016 2024-04-23T16:28:03.367 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:28:03.367 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:28:03.367 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:28:03.367 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:28:03.367 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:28:03.367 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:28:03.367 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:28:03.367 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:28:03.367 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:28:03.368 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:28:03.368 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:28:03.368 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:28:03.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:28:03.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:28:03.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:28:03.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:28:03.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:28:03.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:28:03.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:28:03.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:28:03.368 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:28:03.369 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:28:03.369 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:28:03.369 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:28:03.827 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 46 16 110211 110195 149.69 90.0625 0.00675648 0.00667808 2024-04-23T16:28:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:03 smithi103 ceph-mon[39629]: pgmap v850: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 63 MiB/s wr, 1.01k op/s; 6064/208696 objects misplaced (2.906%); 731 KiB/s, 11 objects/s recovering 2024-04-23T16:28:04.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:03 smithi148 ceph-mon[44520]: pgmap v850: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 63 MiB/s wr, 1.01k op/s; 6064/208696 objects misplaced (2.906%); 731 KiB/s, 11 objects/s recovering 2024-04-23T16:28:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:03 smithi188 ceph-mon[44101]: pgmap v850: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 63 MiB/s wr, 1.01k op/s; 6064/208696 objects misplaced (2.906%); 731 KiB/s, 11 objects/s recovering 2024-04-23T16:28:04.486 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:28:04.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:04.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:28:04.591 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:28:04.625 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:04.651 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:28:04.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:04.659 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:28:04.760 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:28:04.791 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:04.816 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:28:04.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:04.819 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:28:04.827 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 47 15 111676 111661 148.455 91.625 0.0192528 0.0067339 2024-04-23T16:28:04.921 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:28:04.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:04 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:28:04.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:04 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:28:04.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:04 smithi188 ceph-mon[44101]: osdmap e297: 12 total, 12 up, 10 in 2024-04-23T16:28:04.985 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:05.009 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:28:05.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:05.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:28:05.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:04 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:28:05.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:04 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:28:05.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:04 smithi103 ceph-mon[39629]: osdmap e297: 12 total, 12 up, 10 in 2024-04-23T16:28:05.113 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:28:05.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:04 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:28:05.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:04 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:28:05.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:04 smithi148 ceph-mon[44520]: osdmap e297: 12 total, 12 up, 10 in 2024-04-23T16:28:05.163 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:28:05.166 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:05.272 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:28:05.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:05.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:28:05.376 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:28:05.435 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:05.459 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:28:05.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:05.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:28:05.564 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:28:05.596 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:05.617 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:28:05.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:05.620 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:28:05.721 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:28:05.756 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:05.781 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:28:05.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:05.786 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:28:05.819 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:05 smithi148 ceph-mon[44520]: pgmap v852: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 112 MiB/s wr, 1.79k op/s; 5932/217472 objects misplaced (2.728%); 2.3 MiB/s, 36 objects/s recovering 2024-04-23T16:28:05.819 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:05 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:28:05.819 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:05 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:28:05.820 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:05 smithi103 ceph-mon[39629]: pgmap v852: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 112 MiB/s wr, 1.79k op/s; 5932/217472 objects misplaced (2.728%); 2.3 MiB/s, 36 objects/s recovering 2024-04-23T16:28:05.820 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:05 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:28:05.820 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:05 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:28:05.827 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 48 16 113019 113003 147.109 83.875 0.0045476 0.00679447 2024-04-23T16:28:05.887 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:28:05.928 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:05.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:05 smithi188 ceph-mon[44101]: pgmap v852: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 112 MiB/s wr, 1.79k op/s; 5932/217472 objects misplaced (2.728%); 2.3 MiB/s, 36 objects/s recovering 2024-04-23T16:28:05.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:05 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:28:05.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:05 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:28:06.193 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:28:06.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:06.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:28:06.297 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:28:06.328 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:06.358 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:28:06.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:06.364 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:28:06.464 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:28:06.495 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:06.520 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:28:06.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:06.523 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:28:06.624 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:28:06.661 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:06.827 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 49 16 114434 114418 145.912 88.4375 0.013863 0.00685161 2024-04-23T16:28:07.027 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:28:07.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:06 smithi103 ceph-mon[39629]: osdmap e298: 12 total, 12 up, 10 in 2024-04-23T16:28:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:06 smithi148 ceph-mon[44520]: osdmap e298: 12 total, 12 up, 10 in 2024-04-23T16:28:07.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:06 smithi188 ceph-mon[44101]: osdmap e298: 12 total, 12 up, 10 in 2024-04-23T16:28:07.827 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 50 16 115828 115812 144.736 87.125 0.00475018 0.00690783 2024-04-23T16:28:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:07 smithi103 ceph-mon[39629]: pgmap v854: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 92 MiB/s wr, 1.48k op/s; 5932/223038 objects misplaced (2.660%); 1.4 MiB/s, 21 objects/s recovering 2024-04-23T16:28:08.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:07 smithi148 ceph-mon[44520]: pgmap v854: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 92 MiB/s wr, 1.48k op/s; 5932/223038 objects misplaced (2.660%); 1.4 MiB/s, 21 objects/s recovering 2024-04-23T16:28:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:07 smithi188 ceph-mon[44101]: pgmap v854: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 92 MiB/s wr, 1.48k op/s; 5932/223038 objects misplaced (2.660%); 1.4 MiB/s, 21 objects/s recovering 2024-04-23T16:28:08.225 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:28:08.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:08.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:28:08.329 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:28:08.365 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:08.389 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:28:08.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:08.392 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:28:08.493 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:28:08.535 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:08.564 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:28:08.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:08.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:28:08.668 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:28:08.708 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:08.827 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 51 16 117201 117185 143.581 85.8125 0.0150591 0.00696308 2024-04-23T16:28:09.076 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:28:09.076 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:28:09.076 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-23T16:28:09.076 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-23T16:28:09.076 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:28:09.418 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:28:09.421 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:28:09.472 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:28:09.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:09.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:28:09.577 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:28:09.606 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:09.629 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:28:09.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:09.633 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:28:09.728 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:28:09.729 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:28:09.734 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:28:09.827 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 52 16 118553 118537 142.445 84.5 0.00757556 0.00701838 2024-04-23T16:28:09.834 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:09.873 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:28:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:09.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:28:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:09 smithi188 ceph-mon[44101]: pgmap v855: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 71 MiB/s wr, 1.13k op/s; 5932/223038 objects misplaced (2.660%); 1.1 MiB/s, 16 objects/s recovering 2024-04-23T16:28:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:09 smithi188 ceph-mon[44101]: osdmap e299: 12 total, 12 up, 10 in 2024-04-23T16:28:09.980 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:28:10.010 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:10.032 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:28:10.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:10.035 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:28:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:09 smithi103 ceph-mon[39629]: pgmap v855: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 71 MiB/s wr, 1.13k op/s; 5932/223038 objects misplaced (2.660%); 1.1 MiB/s, 16 objects/s recovering 2024-04-23T16:28:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:09 smithi103 ceph-mon[39629]: osdmap e299: 12 total, 12 up, 10 in 2024-04-23T16:28:10.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:09 smithi148 ceph-mon[44520]: pgmap v855: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 71 MiB/s wr, 1.13k op/s; 5932/223038 objects misplaced (2.660%); 1.1 MiB/s, 16 objects/s recovering 2024-04-23T16:28:10.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:09 smithi148 ceph-mon[44520]: osdmap e299: 12 total, 12 up, 10 in 2024-04-23T16:28:10.135 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:28:10.170 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:10.192 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:28:10.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:10.196 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:28:10.264 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":856,"stamp":"2024-04-23T16:28:09.023699+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7533888032,"num_objects":114951,"num_object_clones":0,"num_object_copies":229902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5798,"num_objects_unfound":0,"num_objects_dirty":114951,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":115130,"num_write_kb":7359674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":364,"num_bytes_recovered":25446528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115088,"ondisk_log_size":115088,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":937451520,"kb_used":17015300,"kb_used_data":14657176,"kb_used_omap":15,"kb_used_meta":2358000,"kb_avail":920436220,"statfs":{"total":959950356480,"available":942526689280,"internally_reserved":0,"allocated":15008948224,"data_stored":15003723134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15948,"internal_metadata":2414592436},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,8,8,6,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":694878208,"num_objects":10603,"num_object_clones":0,"num_object_copies":21206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-266,"num_objects_unfound":0,"num_objects_dirty":10603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10603,"num_write_kb":678592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":266,"num_bytes_recovered":17432576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.385593"},"pg_stats":[{"pgid":"4.f","version":"297'7170","reported_seq":7225,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-04-23T16:28:05.411817+0000","last_change":"2024-04-23T16:27:08.790969+0000","last_active":"2024-04-23T16:28:05.411817+0000","last_peered":"2024-04-23T16:28:05.411817+0000","last_clean":"2024-04-23T16:28:05.411817+0000","last_became_active":"2024-04-23T16:27:08.790680+0000","last_became_peered":"2024-04-23T16:27:08.790680+0000","last_unstale":"2024-04-23T16:28:05.411817+0000","last_undegraded":"2024-04-23T16:28:05.411817+0000","last_fullsized":"2024-04-23T16:28:05.411817+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":7170,"log_dups_size":0,"ondisk_log_size":7170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:12.841142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":469893120,"num_objects":7170,"num_object_clones":0,"num_object_copies":14340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7170,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"298'7187","reported_seq":7243,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-04-23T16:28:08.924606+0000","last_change":"2024-04-23T16:27:08.790855+0000","last_active":"2024-04-23T16:28:08.924606+0000","last_peered":"2024-04-23T16:28:08.924606+0000","last_clean":"2024-04-23T16:28:08.924606+0000","last_became_active":"2024-04-23T16:27:08.790531+0000","last_became_peered":"2024-04-23T16:27:08.790531+0000","last_unstale":"2024-04-23T16:28:08.924606+0000","last_undegraded":"2024-04-23T16:28:08.924606+0000","last_fullsized":"2024-04-23T16:28:08.924606+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":7187,"log_dups_size":0,"ondisk_log_size":7187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:18.893001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471007232,"num_objects":7187,"num_object_clones":0,"num_object_copies":14374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7187,"num_write_kb":459968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"298'7312","reported_seq":7368,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-04-23T16:28:08.922525+0000","last_change":"2024-04-23T16:27:08.789322+0000","last_active":"2024-04-23T16:28:08.922525+0000","last_peered":"2024-04-23T16:28:08.922525+0000","last_clean":"2024-04-23T16:28:08.922525+0000","last_became_active":"2024-04-23T16:27:08.789015+0000","last_became_peered":"2024-04-23T16:27:08.789015+0000","last_unstale":"2024-04-23T16:28:08.922525+0000","last_undegraded":"2024-04-23T16:28:08.922525+0000","last_fullsized":"2024-04-23T16:28:08.922525+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":7312,"log_dups_size":0,"ondisk_log_size":7312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:18.065354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479199232,"num_objects":7312,"num_object_clones":0,"num_object_copies":14624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7312,"num_write_kb":467968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"298'7322","reported_seq":7378,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-04-23T16:28:08.912624+0000","last_change":"2024-04-23T16:27:08.790981+0000","last_active":"2024-04-23T16:28:08.912624+0000","last_peered":"2024-04-23T16:28:08.912624+0000","last_clean":"2024-04-23T16:28:08.912624+0000","last_became_active":"2024-04-23T16:27:08.790714+0000","last_became_peered":"2024-04-23T16:27:08.790714+0000","last_unstale":"2024-04-23T16:28:08.912624+0000","last_undegraded":"2024-04-23T16:28:08.912624+0000","last_fullsized":"2024-04-23T16:28:08.912624+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":7322,"log_dups_size":0,"ondisk_log_size":7322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:10.950834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479854592,"num_objects":7322,"num_object_clones":0,"num_object_copies":14644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7322,"num_write_kb":468608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"298'7423","reported_seq":7480,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-04-23T16:28:08.788291+0000","last_change":"2024-04-23T16:27:08.791044+0000","last_active":"2024-04-23T16:28:08.788291+0000","last_peered":"2024-04-23T16:28:08.788291+0000","last_clean":"2024-04-23T16:28:08.788291+0000","last_became_active":"2024-04-23T16:27:08.790772+0000","last_became_peered":"2024-04-23T16:27:08.790772+0000","last_unstale":"2024-04-23T16:28:08.788291+0000","last_undegraded":"2024-04-23T16:28:08.788291+0000","last_fullsized":"2024-04-23T16:28:08.788291+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":7423,"log_dups_size":0,"ondisk_log_size":7423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:28.679532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486473728,"num_objects":7423,"num_object_clones":0,"num_object_copies":14846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7423,"num_write_kb":475072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"297'6989","reported_seq":7043,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-04-23T16:28:04.109219+0000","last_change":"2024-04-23T16:27:08.792705+0000","last_active":"2024-04-23T16:28:04.109219+0000","last_peered":"2024-04-23T16:28:04.109219+0000","last_clean":"2024-04-23T16:28:04.109219+0000","last_became_active":"2024-04-23T16:27:08.792382+0000","last_became_peered":"2024-04-23T16:27:08.792382+0000","last_unstale":"2024-04-23T16:28:04.109219+0000","last_undegraded":"2024-04-23T16:28:04.109219+0000","last_fullsized":"2024-04-23T16:28:04.109219+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":6989,"log_dups_size":0,"ondisk_log_size":6989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:09.350646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458031104,"num_objects":6989,"num_object_clones":0,"num_object_copies":13978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6989,"num_write_kb":447296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"298'7152","reported_seq":7207,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-04-23T16:28:06.461082+0000","last_change":"2024-04-23T16:27:08.790068+0000","last_active":"2024-04-23T16:28:06.461082+0000","last_peered":"2024-04-23T16:28:06.461082+0000","last_clean":"2024-04-23T16:28:06.461082+0000","last_became_active":"2024-04-23T16:27:08.789822+0000","last_became_peered":"2024-04-23T16:27:08.789822+0000","last_unstale":"2024-04-23T16:28:06.461082+0000","last_undegraded":"2024-04-23T16:28:06.461082+0000","last_fullsized":"2024-04-23T16:28:06.461082+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":7152,"log_dups_size":0,"ondisk_log_size":7152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:17.457267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468713472,"num_objects":7152,"num_object_clones":0,"num_object_copies":14304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7152,"num_write_kb":457728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"298'7089","reported_seq":7144,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-04-23T16:28:06.460747+0000","last_change":"2024-04-23T16:27:08.790983+0000","last_active":"2024-04-23T16:28:06.460747+0000","last_peered":"2024-04-23T16:28:06.460747+0000","last_clean":"2024-04-23T16:28:06.460747+0000","last_became_active":"2024-04-23T16:27:08.790727+0000","last_became_peered":"2024-04-23T16:27:08.790727+0000","last_unstale":"2024-04-23T16:28:06.460747+0000","last_undegraded":"2024-04-23T16:28:06.460747+0000","last_fullsized":"2024-04-23T16:28:06.460747+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":7089,"log_dups_size":0,"ondisk_log_size":7089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:09.076024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464584704,"num_objects":7089,"num_object_clones":0,"num_object_copies":14178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7089,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"298'7299","reported_seq":7356,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-04-23T16:28:08.768041+0000","last_change":"2024-04-23T16:27:08.791657+0000","last_active":"2024-04-23T16:28:08.768041+0000","last_peered":"2024-04-23T16:28:08.768041+0000","last_clean":"2024-04-23T16:28:08.768041+0000","last_became_active":"2024-04-23T16:27:08.791328+0000","last_became_peered":"2024-04-23T16:27:08.791328+0000","last_unstale":"2024-04-23T16:28:08.768041+0000","last_undegraded":"2024-04-23T16:28:08.768041+0000","last_fullsized":"2024-04-23T16:28:08.768041+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":7299,"log_dups_size":0,"ondisk_log_size":7299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:35.446374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478347264,"num_objects":7299,"num_object_clones":0,"num_object_copies":14598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7299,"num_write_kb":467136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"298'7401","reported_seq":8171,"reported_epoch":298,"state":"active+recovering+remapped","last_fresh":"2024-04-23T16:28:08.644341+0000","last_change":"2024-04-23T16:27:55.413627+0000","last_active":"2024-04-23T16:28:08.644341+0000","last_peered":"2024-04-23T16:28:08.644341+0000","last_clean":"2024-04-23T16:27:52.738131+0000","last_became_active":"2024-04-23T16:27:55.352467+0000","last_became_peered":"2024-04-23T16:27:55.352467+0000","last_unstale":"2024-04-23T16:28:08.644341+0000","last_undegraded":"2024-04-23T16:28:08.644341+0000","last_fullsized":"2024-04-23T16:28:08.644341+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":7401,"log_dups_size":0,"ondisk_log_size":7401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485031936,"num_objects":7401,"num_object_clones":0,"num_object_copies":14802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5798,"num_objects_unfound":0,"num_objects_dirty":7401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7401,"num_write_kb":473664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":356,"num_bytes_recovered":23330816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[2,9],"avail_no_missing":["2","9"],"object_location_counts":[{"shards":"2,9","objects":1603}],"blocked_by":[],"up_primary":6,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"298'7214","reported_seq":7271,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-04-23T16:28:08.610731+0000","last_change":"2024-04-23T16:27:08.790564+0000","last_active":"2024-04-23T16:28:08.610731+0000","last_peered":"2024-04-23T16:28:08.610731+0000","last_clean":"2024-04-23T16:28:08.610731+0000","last_became_active":"2024-04-23T16:27:08.790294+0000","last_became_peered":"2024-04-23T16:27:08.790294+0000","last_unstale":"2024-04-23T16:28:08.610731+0000","last_undegraded":"2024-04-23T16:28:08.610731+0000","last_fullsized":"2024-04-23T16:28:08.610731+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":7214,"log_dups_size":0,"ondisk_log_size":7214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:31.526680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472776704,"num_objects":7214,"num_object_clones":0,"num_object_copies":14428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7214,"num_write_kb":461696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"298'7069","reported_seq":7124,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-04-23T16:28:06.472509+0000","last_change":"2024-04-23T16:27:08.791003+0000","last_active":"2024-04-23T16:28:06.472509+0000","last_peered":"2024-04-23T16:28:06.472509+0000","last_clean":"2024-04-23T16:28:06.472509+0000","last_became_active":"2024-04-23T16:27:08.790759+0000","last_became_peered":"2024-04-23T16:27:08.790759+0000","last_unstale":"2024-04-23T16:28:06.472509+0000","last_undegraded":"2024-04-23T16:28:06.472509+0000","last_fullsized":"2024-04-23T16:28:06.472509+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":7069,"log_dups_size":0,"ondisk_log_size":7069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:23.398358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463273984,"num_objects":7069,"num_object_clones":0,"num_object_copies":14138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7069,"num_write_kb":452416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"298'7196","reported_seq":7251,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-04-23T16:28:08.821534+0000","last_change":"2024-04-23T16:27:08.859737+0000","last_active":"2024-04-23T16:28:08.821534+0000","last_peered":"2024-04-23T16:28:08.821534+0000","last_clean":"2024-04-23T16:28:08.821534+0000","last_became_active":"2024-04-23T16:27:08.859470+0000","last_became_peered":"2024-04-23T16:27:08.859470+0000","last_unstale":"2024-04-23T16:28:08.821534+0000","last_undegraded":"2024-04-23T16:28:08.821534+0000","last_fullsized":"2024-04-23T16:28:08.821534+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":7196,"log_dups_size":0,"ondisk_log_size":7196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:36.345105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471597056,"num_objects":7196,"num_object_clones":0,"num_object_copies":14392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7196,"num_write_kb":460544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"298'7098","reported_seq":7153,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-04-23T16:28:06.441848+0000","last_change":"2024-04-23T16:27:08.859830+0000","last_active":"2024-04-23T16:28:06.441848+0000","last_peered":"2024-04-23T16:28:06.441848+0000","last_clean":"2024-04-23T16:28:06.441848+0000","last_became_active":"2024-04-23T16:27:08.859549+0000","last_became_peered":"2024-04-23T16:27:08.859549+0000","last_unstale":"2024-04-23T16:28:06.441848+0000","last_undegraded":"2024-04-23T16:28:06.441848+0000","last_fullsized":"2024-04-23T16:28:06.441848+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":7098,"log_dups_size":0,"ondisk_log_size":7098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:28.867751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465174528,"num_objects":7098,"num_object_clones":0,"num_object_copies":14196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7098,"num_write_kb":454272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"297'7210","reported_seq":7265,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-04-23T16:28:05.412319+0000","last_change":"2024-04-23T16:27:08.791209+0000","last_active":"2024-04-23T16:28:05.412319+0000","last_peered":"2024-04-23T16:28:05.412319+0000","last_clean":"2024-04-23T16:28:05.412319+0000","last_became_active":"2024-04-23T16:27:08.790935+0000","last_became_peered":"2024-04-23T16:27:08.790935+0000","last_unstale":"2024-04-23T16:28:05.412319+0000","last_undegraded":"2024-04-23T16:28:05.412319+0000","last_fullsized":"2024-04-23T16:28:05.412319+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":7210,"log_dups_size":0,"ondisk_log_size":7210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:31.848457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472514560,"num_objects":7210,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7210,"num_write_kb":461440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"297'6818","reported_seq":6872,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-04-23T16:28:04.124354+0000","last_change":"2024-04-23T16:27:08.791436+0000","last_active":"2024-04-23T16:28:04.124354+0000","last_peered":"2024-04-23T16:28:04.124354+0000","last_clean":"2024-04-23T16:28:04.124354+0000","last_became_active":"2024-04-23T16:27:08.791180+0000","last_became_peered":"2024-04-23T16:27:08.791180+0000","last_unstale":"2024-04-23T16:28:04.124354+0000","last_undegraded":"2024-04-23T16:28:04.124354+0000","last_fullsized":"2024-04-23T16:28:04.124354+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":274,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:27:07.774961+0000","last_clean_scrub_stamp":"2024-04-23T16:27:07.774961+0000","objects_scrubbed":0,"log_size":6818,"log_dups_size":0,"ondisk_log_size":6818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:18.494567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446824448,"num_objects":6818,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6818,"num_write_kb":436352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":803,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-04-23T16:28:03.816947+0000","last_change":"2024-04-23T16:27:05.751536+0000","last_active":"2024-04-23T16:28:03.816947+0000","last_peered":"2024-04-23T16:28:03.816947+0000","last_clean":"2024-04-23T16:28:03.816947+0000","last_became_active":"2024-04-23T16:15:36.918552+0000","last_became_peered":"2024-04-23T16:15:36.918552+0000","last_unstale":"2024-04-23T16:28:03.816947+0000","last_undegraded":"2024-04-23T16:28:03.816947+0000","last_fullsized":"2024-04-23T16:28:03.816947+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:27:05.751391+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:27:05.751391+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:17.706740+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":7533297664,"num_objects":114949,"num_object_clones":0,"num_object_copies":229898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5798,"num_objects_unfound":0,"num_objects_dirty":114949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114949,"num_write_kb":7356736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":356,"num_bytes_recovered":23330816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15058206720,"data_stored":15058206720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114949,"ondisk_log_size":114949,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056794,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3077920,"kb_used_data":2731604,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":90667232,"statfs":{"total":95995035648,"available":92843245568,"internally_reserved":0,"allocated":2797162496,"data_stored":2796634749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":354613700},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351135,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612774,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1644952,"kb_used_data":1397196,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92100200,"statfs":{"total":95995035648,"available":94310604800,"internally_reserved":0,"allocated":1430728704,"data_stored":1430204529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":253688256},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841705,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":140504,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93604648,"statfs":{"total":95995035648,"available":95851159552,"internally_reserved":0,"allocated":1060864,"data_stored":536689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168750,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2485972,"kb_used_data":2242376,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":91259180,"statfs":{"total":95995035648,"available":93449400320,"internally_reserved":0,"allocated":2296193024,"data_stored":2295671243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":249428426},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463092,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1465152,"kb_used_data":1365940,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92280000,"statfs":{"total":95995035648,"available":94494720000,"internally_reserved":0,"allocated":1398722560,"data_stored":1398210277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137231,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738578,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1170196,"kb_used_data":940492,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":92574956,"statfs":{"total":95995035648,"available":94796754944,"internally_reserved":0,"allocated":963063808,"data_stored":962539633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313415,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1067092,"kb_used_data":927628,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92678060,"statfs":{"total":95995035648,"available":94902333440,"internally_reserved":0,"allocated":949891072,"data_stored":949366897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117186,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2554636,"kb_used_data":2266624,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91190516,"statfs":{"total":95995035648,"available":93379088384,"internally_reserved":0,"allocated":2321022976,"data_stored":2320504747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":294910407},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920957,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2176024,"kb_used_data":1865804,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":91569128,"statfs":{"total":95995035648,"available":93766787072,"internally_reserved":0,"allocated":1910583296,"data_stored":1910059121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757432,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1232852,"kb_used_data":918476,"kb_used_omap":1,"kb_used_meta":314366,"kb_avail":92512300,"statfs":{"total":95995035648,"available":94732595200,"internally_reserved":0,"allocated":940519424,"data_stored":939995249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":321911237},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":964886528,"data_stored":964886528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":958136320,"data_stored":958136320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2323251200,"data_stored":2323251200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1917714432,"data_stored":1917714432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":945094656,"data_stored":945094656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1418067968,"data_stored":1418067968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2294939648,"data_stored":2294939648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1436483584,"data_stored":1436483584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2799632384,"data_stored":2799632384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:28:10.265 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-04-23T16:28:10.266 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-04-23T16:28:10.297 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:28:10.335 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:10.364 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:28:10.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:10.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:28:10.470 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:28:10.511 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:10.598 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:28:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:10.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:28:10.706 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:28:10.741 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:10.767 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:28:10.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:10.770 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:28:10.827 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 53 16 119874 119858 141.315 82.5625 0.0105708 0.00707462 2024-04-23T16:28:10.871 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:28:10.907 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:11.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:10 smithi148 ceph-mon[44520]: from='client.25429 v1:172.21.15.103:0/2802141694' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:28:11.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:10 smithi188 ceph-mon[44101]: from='client.25429 v1:172.21.15.103:0/2802141694' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:28:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:10 smithi103 ceph-mon[39629]: from='client.25429 v1:172.21.15.103:0/2802141694' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:28:11.765 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:28:11.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:11.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:28:11.785 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-23T16:28:11.788 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:28:11.807 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:28:11.827 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 54 16 121236 121220 140.274 85.125 0.0130263 0.00712734 2024-04-23T16:28:11.887 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:28:11.918 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:11.949 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:28:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:11.954 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:28:12.055 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:28:12.096 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:12.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:11 smithi103 ceph-mon[39629]: pgmap v857: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 96 MiB/s wr, 1.53k op/s; 5798/233224 objects misplaced (2.486%); 1.2 MiB/s, 19 objects/s recovering 2024-04-23T16:28:12.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:11 smithi103 ceph-mon[39629]: 4.8 scrub starts 2024-04-23T16:28:12.122 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:28:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:11 smithi148 ceph-mon[44520]: pgmap v857: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 96 MiB/s wr, 1.53k op/s; 5798/233224 objects misplaced (2.486%); 1.2 MiB/s, 19 objects/s recovering 2024-04-23T16:28:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:11 smithi148 ceph-mon[44520]: 4.8 scrub starts 2024-04-23T16:28:12.127 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:11 smithi188 ceph-mon[44101]: pgmap v857: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 96 MiB/s wr, 1.53k op/s; 5798/233224 objects misplaced (2.486%); 1.2 MiB/s, 19 objects/s recovering 2024-04-23T16:28:12.128 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:11 smithi188 ceph-mon[44101]: 4.8 scrub starts 2024-04-23T16:28:12.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:12.129 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:28:12.230 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:28:12.274 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:12.403 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:28:12.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:12.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:28:12.508 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:28:12.546 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:12.827 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 55 16 122545 122529 139.211 81.8125 0.00529072 0.00718164 2024-04-23T16:28:13.067 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:12 smithi103 ceph-mon[39629]: pgmap v858: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 15 active+clean; 7.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 75 MiB/s wr, 1.21k op/s; 5798/236778 objects misplaced (2.449%); 1.0 MiB/s, 16 objects/s recovering 2024-04-23T16:28:13.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:12 smithi188 ceph-mon[44101]: pgmap v858: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 15 active+clean; 7.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 75 MiB/s wr, 1.21k op/s; 5798/236778 objects misplaced (2.449%); 1.0 MiB/s, 16 objects/s recovering 2024-04-23T16:28:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:12 smithi148 ceph-mon[44520]: pgmap v858: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 15 active+clean; 7.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 75 MiB/s wr, 1.21k op/s; 5798/236778 objects misplaced (2.449%); 1.0 MiB/s, 16 objects/s recovering 2024-04-23T16:28:13.707 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:28:13.739 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:28:13.772 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:28:13.814 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:28:13.827 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 56 16 123861 123845 138.194 82.25 0.00491333 0.00723409 2024-04-23T16:28:13.842 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:28:13.875 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:28:13.905 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:28:13.939 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:28:13.968 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:28:14.012 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:28:14.016 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:28:14.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:14.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:28:14.054 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:28:14.088 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:28:14.093 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:28:14.093 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:28:14.093 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:28:14.093 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:28:14.093 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:28:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:28:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:28:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:28:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:28:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:28:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:28:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:28:14.127 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:28:14.181 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:13 smithi148 ceph-mon[44520]: 4.8 scrub ok 2024-04-23T16:28:14.181 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:13 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1681161629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:28:14.181 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:14.212 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:28:14.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:14.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:28:14.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:13 smithi188 ceph-mon[44101]: 4.8 scrub ok 2024-04-23T16:28:14.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:13 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1681161629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:28:14.316 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:28:14.347 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:14.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:13 smithi103 ceph-mon[39629]: 4.8 scrub ok 2024-04-23T16:28:14.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:13 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1681161629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:28:14.371 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:28:14.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:14.375 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:28:14.476 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:28:14.516 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:14.618 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:28:14.623 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:28:14.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:14.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:28:14.727 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:28:14.771 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:14.811 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:28:14.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:14.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:28:14.827 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 57 16 125113 125097 137.142 78.25 0.00404182 0.00728975 2024-04-23T16:28:14.917 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:28:14.956 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:14.988 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:28:14.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:14.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:28:15.055 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:15 smithi148 ceph-mon[44520]: 4.f scrub starts 2024-04-23T16:28:15.055 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.055 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.055 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.055 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.055 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.055 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.055 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.055 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.056 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.056 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.056 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.056 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.056 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:15 smithi148 ceph-mon[44520]: pgmap v859: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 15 active+clean; 7.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 91 MiB/s wr, 1.46k op/s; 5642/245192 objects misplaced (2.301%); 2.1 MiB/s, 32 objects/s recovering 2024-04-23T16:28:15.056 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:15 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:28:15.092 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:28:15.153 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:15.189 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:28:15.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:15.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:28:15.294 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:28:15.324 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:15 smithi103 ceph-mon[39629]: 4.f scrub starts 2024-04-23T16:28:15.324 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.324 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.324 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.324 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.324 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.324 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.324 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.324 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.324 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:15 smithi103 ceph-mon[39629]: pgmap v859: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 15 active+clean; 7.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 91 MiB/s wr, 1.46k op/s; 5642/245192 objects misplaced (2.301%); 2.1 MiB/s, 32 objects/s recovering 2024-04-23T16:28:15.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:15 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:28:15.332 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:15.361 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:28:15.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:15.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:28:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:15 smithi188 ceph-mon[44101]: 4.f scrub starts 2024-04-23T16:28:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:15.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:15 smithi188 ceph-mon[44101]: pgmap v859: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 15 active+clean; 7.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 91 MiB/s wr, 1.46k op/s; 5642/245192 objects misplaced (2.301%); 2.1 MiB/s, 32 objects/s recovering 2024-04-23T16:28:15.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:15 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:28:15.489 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:28:15.516 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:15.545 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:28:15.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:15.548 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:28:15.648 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:28:15.710 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:15.747 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:28:15.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:15.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:28:15.828 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 58 16 126195 126179 135.944 67.625 0.00579439 0.00735428 2024-04-23T16:28:15.857 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:28:15.888 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:15.910 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:28:15.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:15.912 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:28:16.013 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:28:16.049 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:16.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:16 smithi188 ceph-mon[44101]: 4.f scrub ok 2024-04-23T16:28:16.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:28:16.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:28:16.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:28:16.313 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:16 smithi103 ceph-mon[39629]: 4.f scrub ok 2024-04-23T16:28:16.313 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:28:16.313 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:28:16.313 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:28:16.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:16 smithi148 ceph-mon[44520]: 4.f scrub ok 2024-04-23T16:28:16.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:28:16.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:28:16.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:28:16.606 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:28:16.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:16.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:28:16.711 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:28:16.752 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:16.789 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:28:16.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:16.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:28:16.828 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 59 16 127394 127378 134.91 74.9375 0.0206285 0.00741093 2024-04-23T16:28:16.898 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:28:16.934 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:16.961 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:28:16.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:16.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:28:17.067 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:28:17.096 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:17.121 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:28:17.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:17.127 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:28:17.228 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:28:17.240 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:17 smithi103 ceph-mon[39629]: 4.c scrub starts 2024-04-23T16:28:17.240 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:17 smithi103 ceph-mon[39629]: pgmap v860: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 84 MiB/s wr, 1.34k op/s; 5642/249808 objects misplaced (2.259%); 1.8 MiB/s, 28 objects/s recovering 2024-04-23T16:28:17.281 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:17.304 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:28:17.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:17.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:28:17.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:17 smithi148 ceph-mon[44520]: 4.c scrub starts 2024-04-23T16:28:17.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:17 smithi148 ceph-mon[44520]: pgmap v860: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 84 MiB/s wr, 1.34k op/s; 5642/249808 objects misplaced (2.259%); 1.8 MiB/s, 28 objects/s recovering 2024-04-23T16:28:17.408 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:28:17.466 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:17.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:17 smithi188 ceph-mon[44101]: 4.c scrub starts 2024-04-23T16:28:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:17 smithi188 ceph-mon[44101]: pgmap v860: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 84 MiB/s wr, 1.34k op/s; 5642/249808 objects misplaced (2.259%); 1.8 MiB/s, 28 objects/s recovering 2024-04-23T16:28:17.500 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:28:17.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:17.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:28:17.603 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:28:17.642 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:17.828 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:28:17.827815 min lat: 0.00105502 max lat: 2.61652 avg lat: 0.00746375 2024-04-23T16:28:17.828 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:28:17.828 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 60 16 128625 128609 133.944 76.9375 0.00669478 0.00746375 2024-04-23T16:28:18.086 INFO:teuthology.orchestra.run.smithi103.stderr:set pool 1 pg_num to 11 2024-04-23T16:28:18.213 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:28:18.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:18.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:28:18.316 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:28:18.347 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:18 smithi103 ceph-mon[39629]: 4.c scrub ok 2024-04-23T16:28:18.347 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:18 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1346008054' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-23T16:28:18.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:18 smithi103 ceph-mon[39629]: from='client.25441 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-23T16:28:18.349 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:18.354 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:28:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:18 smithi148 ceph-mon[44520]: 4.c scrub ok 2024-04-23T16:28:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:18 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1346008054' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-23T16:28:18.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:18 smithi148 ceph-mon[44520]: from='client.25441 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-23T16:28:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:18 smithi188 ceph-mon[44101]: 4.c scrub ok 2024-04-23T16:28:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:18 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1346008054' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-23T16:28:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:18 smithi188 ceph-mon[44101]: from='client.25441 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-23T16:28:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:28:18.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:18.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:28:18.686 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:28:18.718 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:18.740 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:28:18.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:18.744 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:28:18.830 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 61 16 129933 129917 133.088 81.75 0.00747023 0.00751261 2024-04-23T16:28:18.846 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:28:18.888 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:18.914 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:28:18.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:18.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:28:19.021 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:28:19.035 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:28:19.080 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:19.111 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:28:19.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:19.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:28:19.215 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:28:19.247 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:19.270 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:28:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:19.273 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:28:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:19 smithi103 ceph-mon[39629]: from='client.25441 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-23T16:28:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:19 smithi103 ceph-mon[39629]: osdmap e300: 12 total, 12 up, 10 in 2024-04-23T16:28:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:19 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1234238956' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:28:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:19 smithi103 ceph-mon[39629]: from='client.25444 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:28:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:19 smithi103 ceph-mon[39629]: pgmap v862: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 65 MiB/s wr, 1.03k op/s; 5642/249808 objects misplaced (2.259%); 1.0 MiB/s, 16 objects/s recovering 2024-04-23T16:28:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:19 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-23T16:28:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:19 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:28:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:19 smithi103 ceph-mon[39629]: from='client.25444 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:28:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:19 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-23T16:28:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:19 smithi103 ceph-mon[39629]: osdmap e301: 12 total, 12 up, 10 in 2024-04-23T16:28:19.373 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:28:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:19 smithi148 ceph-mon[44520]: from='client.25441 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-23T16:28:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:19 smithi148 ceph-mon[44520]: osdmap e300: 12 total, 12 up, 10 in 2024-04-23T16:28:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:19 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1234238956' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:28:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:19 smithi148 ceph-mon[44520]: from='client.25444 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:28:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:19 smithi148 ceph-mon[44520]: pgmap v862: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 65 MiB/s wr, 1.03k op/s; 5642/249808 objects misplaced (2.259%); 1.0 MiB/s, 16 objects/s recovering 2024-04-23T16:28:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:19 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-23T16:28:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:19 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:28:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:19 smithi148 ceph-mon[44520]: from='client.25444 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:28:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:19 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-23T16:28:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:19 smithi148 ceph-mon[44520]: osdmap e301: 12 total, 12 up, 10 in 2024-04-23T16:28:19.406 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:19 smithi188 ceph-mon[44101]: from='client.25441 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-23T16:28:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:19 smithi188 ceph-mon[44101]: osdmap e300: 12 total, 12 up, 10 in 2024-04-23T16:28:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:19 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1234238956' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:28:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:19 smithi188 ceph-mon[44101]: from='client.25444 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:28:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:19 smithi188 ceph-mon[44101]: pgmap v862: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 65 MiB/s wr, 1.03k op/s; 5642/249808 objects misplaced (2.259%); 1.0 MiB/s, 16 objects/s recovering 2024-04-23T16:28:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:19 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-23T16:28:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:19 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:28:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:19 smithi188 ceph-mon[44101]: from='client.25444 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:28:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:19 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-23T16:28:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:19 smithi188 ceph-mon[44101]: osdmap e301: 12 total, 12 up, 10 in 2024-04-23T16:28:19.828 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 62 16 131135 131119 132.153 75.125 0.0108589 0.00756552 2024-04-23T16:28:20.828 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 63 16 132433 132417 131.343 81.125 0.0117906 0.00761191 2024-04-23T16:28:21.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:21 smithi103 ceph-mon[39629]: osdmap e302: 12 total, 12 up, 10 in 2024-04-23T16:28:21.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:21 smithi103 ceph-mon[39629]: pgmap v865: 27 pgs: 1 peering, 1 active+recovering+remapped, 10 unknown, 15 active+clean; 7.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 72 MiB/s wr, 1.15k op/s; 5505/258952 objects misplaced (2.126%); 1.4 MiB/s, 22 objects/s recovering 2024-04-23T16:28:21.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:21 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-23T16:28:21.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:21 smithi148 ceph-mon[44520]: osdmap e302: 12 total, 12 up, 10 in 2024-04-23T16:28:21.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:21 smithi148 ceph-mon[44520]: pgmap v865: 27 pgs: 1 peering, 1 active+recovering+remapped, 10 unknown, 15 active+clean; 7.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 72 MiB/s wr, 1.15k op/s; 5505/258952 objects misplaced (2.126%); 1.4 MiB/s, 22 objects/s recovering 2024-04-23T16:28:21.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:21 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-23T16:28:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:21 smithi188 ceph-mon[44101]: osdmap e302: 12 total, 12 up, 10 in 2024-04-23T16:28:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:21 smithi188 ceph-mon[44101]: pgmap v865: 27 pgs: 1 peering, 1 active+recovering+remapped, 10 unknown, 15 active+clean; 7.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 72 MiB/s wr, 1.15k op/s; 5505/258952 objects misplaced (2.126%); 1.4 MiB/s, 22 objects/s recovering 2024-04-23T16:28:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:21 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-23T16:28:21.828 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 64 16 133700 133684 130.528 79.1875 0.011145 0.00765995 2024-04-23T16:28:22.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:22 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-23T16:28:22.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:22 smithi103 ceph-mon[39629]: osdmap e303: 12 total, 12 up, 10 in 2024-04-23T16:28:22.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:22 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-23T16:28:22.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:22 smithi148 ceph-mon[44520]: osdmap e303: 12 total, 12 up, 10 in 2024-04-23T16:28:22.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:22 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-23T16:28:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:22 smithi188 ceph-mon[44101]: osdmap e303: 12 total, 12 up, 10 in 2024-04-23T16:28:22.828 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 65 16 134946 134930 129.718 77.875 0.00901769 0.00770733 2024-04-23T16:28:23.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:23 smithi103 ceph-mon[39629]: osdmap e304: 12 total, 12 up, 10 in 2024-04-23T16:28:23.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:23 smithi103 ceph-mon[39629]: pgmap v868: 27 pgs: 2 peering, 1 active+recovering+remapped, 9 unknown, 15 active+clean; 8.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 96 MiB/s wr, 1.53k op/s; 5505/262084 objects misplaced (2.100%); 2.1 MiB/s, 34 objects/s recovering 2024-04-23T16:28:23.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:23 smithi148 ceph-mon[44520]: osdmap e304: 12 total, 12 up, 10 in 2024-04-23T16:28:23.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:23 smithi148 ceph-mon[44520]: pgmap v868: 27 pgs: 2 peering, 1 active+recovering+remapped, 9 unknown, 15 active+clean; 8.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 96 MiB/s wr, 1.53k op/s; 5505/262084 objects misplaced (2.100%); 2.1 MiB/s, 34 objects/s recovering 2024-04-23T16:28:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:23 smithi188 ceph-mon[44101]: osdmap e304: 12 total, 12 up, 10 in 2024-04-23T16:28:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:23 smithi188 ceph-mon[44101]: pgmap v868: 27 pgs: 2 peering, 1 active+recovering+remapped, 9 unknown, 15 active+clean; 8.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 96 MiB/s wr, 1.53k op/s; 5505/262084 objects misplaced (2.100%); 2.1 MiB/s, 34 objects/s recovering 2024-04-23T16:28:23.828 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 66 16 136181 136165 128.922 77.1875 0.0113793 0.0077552 2024-04-23T16:28:23.957 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10, 1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1] 2024-04-23T16:28:23.957 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:28:23.957 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-23T16:28:23.957 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-23T16:28:24.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:24 smithi103 ceph-mon[39629]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-23T16:28:24.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:24 smithi103 ceph-mon[39629]: osdmap e305: 12 total, 12 up, 10 in 2024-04-23T16:28:24.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:24 smithi103 ceph-mon[39629]: osdmap e306: 12 total, 12 up, 10 in 2024-04-23T16:28:24.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:24 smithi148 ceph-mon[44520]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-23T16:28:24.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:24 smithi148 ceph-mon[44520]: osdmap e305: 12 total, 12 up, 10 in 2024-04-23T16:28:24.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:24 smithi148 ceph-mon[44520]: osdmap e306: 12 total, 12 up, 10 in 2024-04-23T16:28:24.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:24 smithi188 ceph-mon[44101]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-23T16:28:24.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:24 smithi188 ceph-mon[44101]: osdmap e305: 12 total, 12 up, 10 in 2024-04-23T16:28:24.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:24 smithi188 ceph-mon[44101]: osdmap e306: 12 total, 12 up, 10 in 2024-04-23T16:28:24.828 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 67 16 137397 137381 128.132 76 0.0389876 0.00780256 2024-04-23T16:28:24.940 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:28:24.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:24.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:28:24.963 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:28:25.045 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:28:25.078 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:25.154 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:28:25.160 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:25 smithi103 ceph-mon[39629]: pgmap v871: 27 pgs: 4 peering, 1 active+recovering+remapped, 7 unknown, 15 active+clean; 8.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 87 MiB/s wr, 1.38k op/s; 5363/270034 objects misplaced (1.986%); 2.2 MiB/s, 35 objects/s recovering 2024-04-23T16:28:25.161 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:25 smithi103 ceph-mon[39629]: osdmap e307: 12 total, 12 up, 10 in 2024-04-23T16:28:25.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:25.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:28:25.262 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:28:25.292 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:25.338 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:28:25.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:25.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:28:25.343 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:25 smithi148 ceph-mon[44520]: pgmap v871: 27 pgs: 4 peering, 1 active+recovering+remapped, 7 unknown, 15 active+clean; 8.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 87 MiB/s wr, 1.38k op/s; 5363/270034 objects misplaced (1.986%); 2.2 MiB/s, 35 objects/s recovering 2024-04-23T16:28:25.344 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:25 smithi148 ceph-mon[44520]: osdmap e307: 12 total, 12 up, 10 in 2024-04-23T16:28:25.444 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:28:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:25 smithi188 ceph-mon[44101]: pgmap v871: 27 pgs: 4 peering, 1 active+recovering+remapped, 7 unknown, 15 active+clean; 8.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 87 MiB/s wr, 1.38k op/s; 5363/270034 objects misplaced (1.986%); 2.2 MiB/s, 35 objects/s recovering 2024-04-23T16:28:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:25 smithi188 ceph-mon[44101]: osdmap e307: 12 total, 12 up, 10 in 2024-04-23T16:28:25.496 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:25.525 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:28:25.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:25.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:28:25.631 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:28:25.669 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:25.693 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:28:25.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:25.695 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:28:25.796 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:28:25.828 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 68 16 138475 138459 127.239 67.375 0.010432 0.00785761 2024-04-23T16:28:25.846 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:25.880 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:28:25.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:25.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:28:25.994 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:28:26.025 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:26.053 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:28:26.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:26.056 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:28:26.157 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:28:26.190 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:26.218 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:28:26.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:26.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:28:26.322 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:28:26.361 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:26.830 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 69 16 139640 139624 126.45 72.8125 0.00942892 0.00790625 2024-04-23T16:28:26.948 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:28:27.072 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:26 smithi103 ceph-mon[39629]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-23T16:28:27.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:26 smithi148 ceph-mon[44520]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-23T16:28:27.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:26 smithi188 ceph-mon[44101]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-23T16:28:27.624 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:28:27.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:27.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:28:27.667 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:28:27.668 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:28:27.728 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:28:27.784 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:27.818 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:28:27.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:27.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:28:27.829 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 70 16 140830 140814 125.706 74.375 0.00471708 0.0079536 2024-04-23T16:28:27.922 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:28:27.960 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:27.982 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:28:27.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:27.985 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:28:28.086 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:28:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:27 smithi103 ceph-mon[39629]: pgmap v873: 27 pgs: 2 peering, 1 active+recovering+remapped, 6 unknown, 18 active+clean; 8.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 84 MiB/s wr, 1.35k op/s; 5363/274490 objects misplaced (1.954%); 2.0 MiB/s, 31 objects/s recovering 2024-04-23T16:28:28.120 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:28.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:27 smithi148 ceph-mon[44520]: pgmap v873: 27 pgs: 2 peering, 1 active+recovering+remapped, 6 unknown, 18 active+clean; 8.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 84 MiB/s wr, 1.35k op/s; 5363/274490 objects misplaced (1.954%); 2.0 MiB/s, 31 objects/s recovering 2024-04-23T16:28:28.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:27 smithi188 ceph-mon[44101]: pgmap v873: 27 pgs: 2 peering, 1 active+recovering+remapped, 6 unknown, 18 active+clean; 8.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 84 MiB/s wr, 1.35k op/s; 5363/274490 objects misplaced (1.954%); 2.0 MiB/s, 31 objects/s recovering 2024-04-23T16:28:28.761 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:28 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:28:28.762 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:28 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:28:28.829 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 71 15 142042 142027 125.003 75.8125 0.00948116 0.00799802 2024-04-23T16:28:28.973 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:28:28.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:28.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:28:29.027 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-23T16:28:29.072 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:28:29.082 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:28:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:28 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:28:29.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:28 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:28:29.145 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:28 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:28:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:28 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:28:29.692 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:28:29.712 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:28:29.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:29.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:28:29.822 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:28:29.829 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 72 15 143228 143213 124.296 74.125 0.0129667 0.00804286 2024-04-23T16:28:29.879 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:29.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:29.908 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:28:29.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:28:29.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:29 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:28:29.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:29 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:28:29.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:29 smithi103 ceph-mon[39629]: pgmap v874: 27 pgs: 2 peering, 1 active+recovering+remapped, 5 unknown, 19 active+clean; 8.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 65 MiB/s wr, 1.03k op/s; 5363/274490 objects misplaced (1.954%); 1.5 MiB/s, 23 objects/s recovering 2024-04-23T16:28:29.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:29 smithi103 ceph-mon[39629]: osdmap e308: 12 total, 12 up, 10 in 2024-04-23T16:28:29.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:29 smithi103 ceph-mon[39629]: osdmap e309: 12 total, 12 up, 10 in 2024-04-23T16:28:30.009 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:28:30.036 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:29 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:28:30.037 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:29 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:28:30.037 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:29 smithi188 ceph-mon[44101]: pgmap v874: 27 pgs: 2 peering, 1 active+recovering+remapped, 5 unknown, 19 active+clean; 8.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 65 MiB/s wr, 1.03k op/s; 5363/274490 objects misplaced (1.954%); 1.5 MiB/s, 23 objects/s recovering 2024-04-23T16:28:30.037 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:29 smithi188 ceph-mon[44101]: osdmap e308: 12 total, 12 up, 10 in 2024-04-23T16:28:30.037 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:29 smithi188 ceph-mon[44101]: osdmap e309: 12 total, 12 up, 10 in 2024-04-23T16:28:30.038 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:30.063 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:28:30.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:30.068 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:28:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:29 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:28:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:29 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:28:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:29 smithi148 ceph-mon[44520]: pgmap v874: 27 pgs: 2 peering, 1 active+recovering+remapped, 5 unknown, 19 active+clean; 8.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 65 MiB/s wr, 1.03k op/s; 5363/274490 objects misplaced (1.954%); 1.5 MiB/s, 23 objects/s recovering 2024-04-23T16:28:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:29 smithi148 ceph-mon[44520]: osdmap e308: 12 total, 12 up, 10 in 2024-04-23T16:28:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:29 smithi148 ceph-mon[44520]: osdmap e309: 12 total, 12 up, 10 in 2024-04-23T16:28:30.170 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:28:30.200 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:30.223 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:28:30.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:30.229 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:28:30.330 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:28:30.364 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:30.386 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:28:30.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:30.389 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:28:30.490 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:28:30.543 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:30.621 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:28:30.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:30.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:28:30.653 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:28:30.681 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:28:30.714 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:28:30.727 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:28:30.751 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:28:30.787 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:30.796 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:28:30.814 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:28:30.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:30.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:28:30.829 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 73 16 144363 144347 123.565 70.875 0.0107326 0.00809175 2024-04-23T16:28:30.835 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:28:30.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:30 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2206756362' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:28:30.873 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:28:30.906 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:28:30.921 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:28:30.938 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:28:30.966 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:30.966 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:28:30.998 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:28:31.030 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:28:31.033 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:28:31.033 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:28:31.033 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:28:31.033 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:28:31.033 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:28:31.033 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:28:31.033 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:28:31.033 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:28:31.033 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:28:31.033 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:28:31.033 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:28:31.033 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:28:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:30 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2206756362' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:28:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:30 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2206756362' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:28:31.535 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:28:31.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:31.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:28:31.642 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:28:31.675 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:31.698 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:28:31.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:31.701 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:28:31.779 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.779 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:31 smithi103 ceph-mon[39629]: pgmap v877: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 8.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 69 MiB/s wr, 1.11k op/s; 5212/283332 objects misplaced (1.840%); 1.6 MiB/s, 25 objects/s recovering 2024-04-23T16:28:31.779 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.779 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.779 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.779 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.779 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.779 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.779 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.779 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.779 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.779 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.779 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:31 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:28:31.780 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:31 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:28:31.801 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:28:31.829 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 74 16 145535 145519 122.885 73.25 0.010836 0.0081362 2024-04-23T16:28:31.840 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:31.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:31 smithi188 ceph-mon[44101]: pgmap v877: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 8.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 69 MiB/s wr, 1.11k op/s; 5212/283332 objects misplaced (1.840%); 1.6 MiB/s, 25 objects/s recovering 2024-04-23T16:28:31.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:31.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:31 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:28:31.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:31 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:28:32.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:32.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:31 smithi148 ceph-mon[44520]: pgmap v877: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 8.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 69 MiB/s wr, 1.11k op/s; 5212/283332 objects misplaced (1.840%); 1.6 MiB/s, 25 objects/s recovering 2024-04-23T16:28:32.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:32.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:32.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:32.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:32.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:32.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:32.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:32.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:32.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:32.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:32.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:31 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:28:32.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:31 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:28:32.594 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:28:32.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:32.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:28:32.698 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:28:32.736 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:32.755 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:28:32.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:32.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:28:32.829 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 75 16 146698 146682 122.215 72.6875 0.00543849 0.00818052 2024-04-23T16:28:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:32 smithi103 ceph-mon[39629]: osdmap e310: 12 total, 12 up, 10 in 2024-04-23T16:28:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:32 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:28:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:32 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:28:32.860 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:28:32.911 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:32.948 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:28:32.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:32.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:28:33.053 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:28:33.113 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:33.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:32 smithi148 ceph-mon[44520]: osdmap e310: 12 total, 12 up, 10 in 2024-04-23T16:28:33.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:32 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:28:33.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:32 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:28:33.149 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:28:33.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:33.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:28:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:32 smithi188 ceph-mon[44101]: osdmap e310: 12 total, 12 up, 10 in 2024-04-23T16:28:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:32 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:28:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:32 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:28:33.254 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:28:33.306 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:33.339 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:28:33.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:33.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:28:33.446 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:28:33.499 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:33.531 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:28:33.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:33.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:28:33.571 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:28:33.571 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:28:33.571 INFO:teuthology.orchestra.run.smithi103.stdout: "epoch": 311, 2024-04-23T16:28:33.571 INFO:teuthology.orchestra.run.smithi103.stdout: "fsid": "8e3594f2-018b-11ef-bc93-c7b262605968", 2024-04-23T16:28:33.571 INFO:teuthology.orchestra.run.smithi103.stdout: "created": "2024-04-23T16:08:13.477590+0000", 2024-04-23T16:28:33.571 INFO:teuthology.orchestra.run.smithi103.stdout: "modified": "2024-04-23T16:28:32.787463+0000", 2024-04-23T16:28:33.572 INFO:teuthology.orchestra.run.smithi103.stdout: "last_up_change": "2024-04-23T16:27:00.729195+0000", 2024-04-23T16:28:33.572 INFO:teuthology.orchestra.run.smithi103.stdout: "last_in_change": "2024-04-23T16:24:50.783418+0000", 2024-04-23T16:28:33.572 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-23T16:28:33.572 INFO:teuthology.orchestra.run.smithi103.stdout: "flags_num": 5801984, 2024-04-23T16:28:33.572 INFO:teuthology.orchestra.run.smithi103.stdout: "flags_set": [ 2024-04-23T16:28:33.572 INFO:teuthology.orchestra.run.smithi103.stdout: "noscrub", 2024-04-23T16:28:33.572 INFO:teuthology.orchestra.run.smithi103.stdout: "pglog_hardlimit", 2024-04-23T16:28:33.572 INFO:teuthology.orchestra.run.smithi103.stdout: "purged_snapdirs", 2024-04-23T16:28:33.572 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_deletes", 2024-04-23T16:28:33.572 INFO:teuthology.orchestra.run.smithi103.stdout: "sortbitwise" 2024-04-23T16:28:33.572 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:28:33.572 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_version": 28, 2024-04-23T16:28:33.572 INFO:teuthology.orchestra.run.smithi103.stdout: "full_ratio": 0.94999998807907104, 2024-04-23T16:28:33.572 INFO:teuthology.orchestra.run.smithi103.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-23T16:28:33.573 INFO:teuthology.orchestra.run.smithi103.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-23T16:28:33.573 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_snapshot": "", 2024-04-23T16:28:33.573 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_max": 4, 2024-04-23T16:28:33.573 INFO:teuthology.orchestra.run.smithi103.stdout: "max_osd": 12, 2024-04-23T16:28:33.573 INFO:teuthology.orchestra.run.smithi103.stdout: "require_min_compat_client": "luminous", 2024-04-23T16:28:33.573 INFO:teuthology.orchestra.run.smithi103.stdout: "min_compat_client": "luminous", 2024-04-23T16:28:33.573 INFO:teuthology.orchestra.run.smithi103.stdout: "require_osd_release": "squid", 2024-04-23T16:28:33.573 INFO:teuthology.orchestra.run.smithi103.stdout: "allow_crimson": false, 2024-04-23T16:28:33.573 INFO:teuthology.orchestra.run.smithi103.stdout: "pools": [ 2024-04-23T16:28:33.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.573 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-04-23T16:28:33.573 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_name": ".mgr", 2024-04-23T16:28:33.573 INFO:teuthology.orchestra.run.smithi103.stdout: "create_time": "2024-04-23T16:11:22.416777+0000", 2024-04-23T16:28:33.573 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-04-23T16:28:33.574 INFO:teuthology.orchestra.run.smithi103.stdout: "flags_names": "hashpspool", 2024-04-23T16:28:33.574 INFO:teuthology.orchestra.run.smithi103.stdout: "type": 1, 2024-04-23T16:28:33.574 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 2, 2024-04-23T16:28:33.574 INFO:teuthology.orchestra.run.smithi103.stdout: "min_size": 2, 2024-04-23T16:28:33.574 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_rule": 0, 2024-04-23T16:28:33.574 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_count": 0, 2024-04-23T16:28:33.574 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_target": 0, 2024-04-23T16:28:33.574 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T16:28:33.574 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T16:28:33.574 INFO:teuthology.orchestra.run.smithi103.stdout: "object_hash": 2, 2024-04-23T16:28:33.574 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_autoscale_mode": "off", 2024-04-23T16:28:33.574 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num": 11, 2024-04-23T16:28:33.575 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_placement_num": 11, 2024-04-23T16:28:33.575 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_placement_num_target": 11, 2024-04-23T16:28:33.575 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_target": 11, 2024-04-23T16:28:33.575 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_pending": 11, 2024-04-23T16:28:33.575 INFO:teuthology.orchestra.run.smithi103.stdout: "last_pg_merge_meta": { 2024-04-23T16:28:33.575 INFO:teuthology.orchestra.run.smithi103.stdout: "source_pgid": "0.0", 2024-04-23T16:28:33.575 INFO:teuthology.orchestra.run.smithi103.stdout: "ready_epoch": 0, 2024-04-23T16:28:33.575 INFO:teuthology.orchestra.run.smithi103.stdout: "last_epoch_started": 0, 2024-04-23T16:28:33.575 INFO:teuthology.orchestra.run.smithi103.stdout: "last_epoch_clean": 0, 2024-04-23T16:28:33.575 INFO:teuthology.orchestra.run.smithi103.stdout: "source_version": "0'0", 2024-04-23T16:28:33.575 INFO:teuthology.orchestra.run.smithi103.stdout: "target_version": "0'0" 2024-04-23T16:28:33.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.576 INFO:teuthology.orchestra.run.smithi103.stdout: "last_change": "303", 2024-04-23T16:28:33.576 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend": "0", 2024-04-23T16:28:33.576 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T16:28:33.576 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend_preluminous": "301", 2024-04-23T16:28:33.576 INFO:teuthology.orchestra.run.smithi103.stdout: "auid": 0, 2024-04-23T16:28:33.576 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_mode": "selfmanaged", 2024-04-23T16:28:33.576 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_seq": 0, 2024-04-23T16:28:33.576 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_epoch": 0, 2024-04-23T16:28:33.576 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_snaps": [], 2024-04-23T16:28:33.576 INFO:teuthology.orchestra.run.smithi103.stdout: "removed_snaps": "[]", 2024-04-23T16:28:33.576 INFO:teuthology.orchestra.run.smithi103.stdout: "quota_max_bytes": 0, 2024-04-23T16:28:33.576 INFO:teuthology.orchestra.run.smithi103.stdout: "quota_max_objects": 0, 2024-04-23T16:28:33.576 INFO:teuthology.orchestra.run.smithi103.stdout: "tiers": [], 2024-04-23T16:28:33.577 INFO:teuthology.orchestra.run.smithi103.stdout: "tier_of": -1, 2024-04-23T16:28:33.577 INFO:teuthology.orchestra.run.smithi103.stdout: "read_tier": -1, 2024-04-23T16:28:33.577 INFO:teuthology.orchestra.run.smithi103.stdout: "write_tier": -1, 2024-04-23T16:28:33.577 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_mode": "none", 2024-04-23T16:28:33.577 INFO:teuthology.orchestra.run.smithi103.stdout: "target_max_bytes": 0, 2024-04-23T16:28:33.577 INFO:teuthology.orchestra.run.smithi103.stdout: "target_max_objects": 0, 2024-04-23T16:28:33.577 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T16:28:33.577 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T16:28:33.577 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T16:28:33.577 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_min_flush_age": 0, 2024-04-23T16:28:33.577 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_min_evict_age": 0, 2024-04-23T16:28:33.577 INFO:teuthology.orchestra.run.smithi103.stdout: "erasure_code_profile": "", 2024-04-23T16:28:33.577 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_params": { 2024-04-23T16:28:33.578 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "none" 2024-04-23T16:28:33.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.578 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_period": 0, 2024-04-23T16:28:33.578 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_count": 0, 2024-04-23T16:28:33.578 INFO:teuthology.orchestra.run.smithi103.stdout: "use_gmt_hitset": true, 2024-04-23T16:28:33.578 INFO:teuthology.orchestra.run.smithi103.stdout: "min_read_recency_for_promote": 0, 2024-04-23T16:28:33.578 INFO:teuthology.orchestra.run.smithi103.stdout: "min_write_recency_for_promote": 0, 2024-04-23T16:28:33.578 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T16:28:33.578 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_search_last_n": 0, 2024-04-23T16:28:33.578 INFO:teuthology.orchestra.run.smithi103.stdout: "grade_table": [], 2024-04-23T16:28:33.578 INFO:teuthology.orchestra.run.smithi103.stdout: "stripe_width": 0, 2024-04-23T16:28:33.578 INFO:teuthology.orchestra.run.smithi103.stdout: "expected_num_objects": 0, 2024-04-23T16:28:33.579 INFO:teuthology.orchestra.run.smithi103.stdout: "fast_read": false, 2024-04-23T16:28:33.579 INFO:teuthology.orchestra.run.smithi103.stdout: "options": { 2024-04-23T16:28:33.579 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_max": 32, 2024-04-23T16:28:33.579 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_min": 1 2024-04-23T16:28:33.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.579 INFO:teuthology.orchestra.run.smithi103.stdout: "application_metadata": { 2024-04-23T16:28:33.579 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": {} 2024-04-23T16:28:33.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.579 INFO:teuthology.orchestra.run.smithi103.stdout: "read_balance": { 2024-04-23T16:28:33.579 INFO:teuthology.orchestra.run.smithi103.stdout: "score_type": "Fair distribution", 2024-04-23T16:28:33.579 INFO:teuthology.orchestra.run.smithi103.stdout: "score_acting": 3.2899999618530273, 2024-04-23T16:28:33.579 INFO:teuthology.orchestra.run.smithi103.stdout: "score_stable": 3.2899999618530273, 2024-04-23T16:28:33.579 INFO:teuthology.orchestra.run.smithi103.stdout: "optimal_score": 0.82999998331069946, 2024-04-23T16:28:33.580 INFO:teuthology.orchestra.run.smithi103.stdout: "raw_score_acting": 2.7300000190734863, 2024-04-23T16:28:33.580 INFO:teuthology.orchestra.run.smithi103.stdout: "raw_score_stable": 2.7300000190734863, 2024-04-23T16:28:33.580 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity_weighted": 0.98181778192520142, 2024-04-23T16:28:33.580 INFO:teuthology.orchestra.run.smithi103.stdout: "average_primary_affinity": 0.98000001907348633, 2024-04-23T16:28:33.580 INFO:teuthology.orchestra.run.smithi103.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-04-23T16:28:33.580 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.580 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 4, 2024-04-23T16:28:33.580 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_name": "unique_pool_2", 2024-04-23T16:28:33.580 INFO:teuthology.orchestra.run.smithi103.stdout: "create_time": "2024-04-23T16:27:07.774581+0000", 2024-04-23T16:28:33.580 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-04-23T16:28:33.580 INFO:teuthology.orchestra.run.smithi103.stdout: "flags_names": "hashpspool", 2024-04-23T16:28:33.580 INFO:teuthology.orchestra.run.smithi103.stdout: "type": 1, 2024-04-23T16:28:33.581 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 2, 2024-04-23T16:28:33.581 INFO:teuthology.orchestra.run.smithi103.stdout: "min_size": 2, 2024-04-23T16:28:33.581 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_rule": 0, 2024-04-23T16:28:33.581 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_count": 0, 2024-04-23T16:28:33.581 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_target": 0, 2024-04-23T16:28:33.581 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T16:28:33.581 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T16:28:33.581 INFO:teuthology.orchestra.run.smithi103.stdout: "object_hash": 2, 2024-04-23T16:28:33.581 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_autoscale_mode": "off", 2024-04-23T16:28:33.581 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num": 16, 2024-04-23T16:28:33.581 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_placement_num": 16, 2024-04-23T16:28:33.581 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_placement_num_target": 16, 2024-04-23T16:28:33.581 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_target": 16, 2024-04-23T16:28:33.581 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_pending": 16, 2024-04-23T16:28:33.582 INFO:teuthology.orchestra.run.smithi103.stdout: "last_pg_merge_meta": { 2024-04-23T16:28:33.582 INFO:teuthology.orchestra.run.smithi103.stdout: "source_pgid": "0.0", 2024-04-23T16:28:33.582 INFO:teuthology.orchestra.run.smithi103.stdout: "ready_epoch": 0, 2024-04-23T16:28:33.582 INFO:teuthology.orchestra.run.smithi103.stdout: "last_epoch_started": 0, 2024-04-23T16:28:33.582 INFO:teuthology.orchestra.run.smithi103.stdout: "last_epoch_clean": 0, 2024-04-23T16:28:33.582 INFO:teuthology.orchestra.run.smithi103.stdout: "source_version": "0'0", 2024-04-23T16:28:33.582 INFO:teuthology.orchestra.run.smithi103.stdout: "target_version": "0'0" 2024-04-23T16:28:33.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.582 INFO:teuthology.orchestra.run.smithi103.stdout: "last_change": "281", 2024-04-23T16:28:33.582 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend": "0", 2024-04-23T16:28:33.582 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T16:28:33.582 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T16:28:33.582 INFO:teuthology.orchestra.run.smithi103.stdout: "auid": 0, 2024-04-23T16:28:33.583 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_mode": "selfmanaged", 2024-04-23T16:28:33.583 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_seq": 0, 2024-04-23T16:28:33.583 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_epoch": 0, 2024-04-23T16:28:33.583 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_snaps": [], 2024-04-23T16:28:33.583 INFO:teuthology.orchestra.run.smithi103.stdout: "removed_snaps": "[]", 2024-04-23T16:28:33.583 INFO:teuthology.orchestra.run.smithi103.stdout: "quota_max_bytes": 0, 2024-04-23T16:28:33.583 INFO:teuthology.orchestra.run.smithi103.stdout: "quota_max_objects": 0, 2024-04-23T16:28:33.583 INFO:teuthology.orchestra.run.smithi103.stdout: "tiers": [], 2024-04-23T16:28:33.583 INFO:teuthology.orchestra.run.smithi103.stdout: "tier_of": -1, 2024-04-23T16:28:33.583 INFO:teuthology.orchestra.run.smithi103.stdout: "read_tier": -1, 2024-04-23T16:28:33.583 INFO:teuthology.orchestra.run.smithi103.stdout: "write_tier": -1, 2024-04-23T16:28:33.583 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_mode": "none", 2024-04-23T16:28:33.583 INFO:teuthology.orchestra.run.smithi103.stdout: "target_max_bytes": 0, 2024-04-23T16:28:33.583 INFO:teuthology.orchestra.run.smithi103.stdout: "target_max_objects": 0, 2024-04-23T16:28:33.583 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T16:28:33.584 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T16:28:33.584 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T16:28:33.584 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_min_flush_age": 0, 2024-04-23T16:28:33.584 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_min_evict_age": 0, 2024-04-23T16:28:33.584 INFO:teuthology.orchestra.run.smithi103.stdout: "erasure_code_profile": "", 2024-04-23T16:28:33.584 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_params": { 2024-04-23T16:28:33.584 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "none" 2024-04-23T16:28:33.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.584 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_period": 0, 2024-04-23T16:28:33.584 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_count": 0, 2024-04-23T16:28:33.584 INFO:teuthology.orchestra.run.smithi103.stdout: "use_gmt_hitset": true, 2024-04-23T16:28:33.584 INFO:teuthology.orchestra.run.smithi103.stdout: "min_read_recency_for_promote": 0, 2024-04-23T16:28:33.584 INFO:teuthology.orchestra.run.smithi103.stdout: "min_write_recency_for_promote": 0, 2024-04-23T16:28:33.584 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T16:28:33.584 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_search_last_n": 0, 2024-04-23T16:28:33.585 INFO:teuthology.orchestra.run.smithi103.stdout: "grade_table": [], 2024-04-23T16:28:33.585 INFO:teuthology.orchestra.run.smithi103.stdout: "stripe_width": 0, 2024-04-23T16:28:33.585 INFO:teuthology.orchestra.run.smithi103.stdout: "expected_num_objects": 0, 2024-04-23T16:28:33.585 INFO:teuthology.orchestra.run.smithi103.stdout: "fast_read": false, 2024-04-23T16:28:33.585 INFO:teuthology.orchestra.run.smithi103.stdout: "options": {}, 2024-04-23T16:28:33.585 INFO:teuthology.orchestra.run.smithi103.stdout: "application_metadata": { 2024-04-23T16:28:33.585 INFO:teuthology.orchestra.run.smithi103.stdout: "rados": {} 2024-04-23T16:28:33.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.585 INFO:teuthology.orchestra.run.smithi103.stdout: "read_balance": { 2024-04-23T16:28:33.585 INFO:teuthology.orchestra.run.smithi103.stdout: "score_type": "Fair distribution", 2024-04-23T16:28:33.585 INFO:teuthology.orchestra.run.smithi103.stdout: "score_acting": 3, 2024-04-23T16:28:33.585 INFO:teuthology.orchestra.run.smithi103.stdout: "score_stable": 3, 2024-04-23T16:28:33.585 INFO:teuthology.orchestra.run.smithi103.stdout: "optimal_score": 0.75, 2024-04-23T16:28:33.585 INFO:teuthology.orchestra.run.smithi103.stdout: "raw_score_acting": 2.25, 2024-04-23T16:28:33.585 INFO:teuthology.orchestra.run.smithi103.stdout: "raw_score_stable": 2.25, 2024-04-23T16:28:33.586 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity_weighted": 0.98181778192520142, 2024-04-23T16:28:33.586 INFO:teuthology.orchestra.run.smithi103.stdout: "average_primary_affinity": 0.98000001907348633, 2024-04-23T16:28:33.586 INFO:teuthology.orchestra.run.smithi103.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-04-23T16:28:33.586 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.586 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.586 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:28:33.586 INFO:teuthology.orchestra.run.smithi103.stdout: "osds": [ 2024-04-23T16:28:33.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.586 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 0, 2024-04-23T16:28:33.589 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "4c084047-8f01-4b86-a60c-5592735b5862", 2024-04-23T16:28:33.590 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:33.590 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:33.590 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:33.590 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:33.590 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:33.590 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:33.590 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 8, 2024-04-23T16:28:33.590 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 306, 2024-04-23T16:28:33.590 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:33.590 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:33.591 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:33.591 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.591 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.591 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6801", 2024-04-23T16:28:33.591 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:28:33.591 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.591 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.591 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:33.591 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.591 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.591 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6802", 2024-04-23T16:28:33.592 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:28:33.592 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.592 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.592 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:33.592 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.592 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.592 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6804", 2024-04-23T16:28:33.592 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:28:33.592 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.592 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.592 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:33.593 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.593 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.593 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6803", 2024-04-23T16:28:33.593 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:28:33.593 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.593 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.593 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6801/4274040097", 2024-04-23T16:28:33.593 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6802/4274040097", 2024-04-23T16:28:33.593 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6804/4274040097", 2024-04-23T16:28:33.593 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6803/4274040097", 2024-04-23T16:28:33.593 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:33.593 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:33.594 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:33.594 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.594 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 1, 2024-04-23T16:28:33.594 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "3438f2bd-5fb7-432c-b383-29e18c442f20", 2024-04-23T16:28:33.594 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:33.594 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 0, 2024-04-23T16:28:33.594 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 0, 2024-04-23T16:28:33.594 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:33.594 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 13, 2024-04-23T16:28:33.594 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 232, 2024-04-23T16:28:33.594 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 271, 2024-04-23T16:28:33.594 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 192, 2024-04-23T16:28:33.595 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 233, 2024-04-23T16:28:33.595 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:33.595 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:33.595 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.595 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.595 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6805", 2024-04-23T16:28:33.595 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 859282187 2024-04-23T16:28:33.595 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.595 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.595 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:33.595 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.596 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.596 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6806", 2024-04-23T16:28:33.596 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 859282187 2024-04-23T16:28:33.596 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.596 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.596 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:33.596 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.596 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.596 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6808", 2024-04-23T16:28:33.596 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 859282187 2024-04-23T16:28:33.596 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.596 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.597 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:33.597 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.597 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.597 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6807", 2024-04-23T16:28:33.597 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 859282187 2024-04-23T16:28:33.597 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.597 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.597 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6805/859282187", 2024-04-23T16:28:33.597 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6806/859282187", 2024-04-23T16:28:33.597 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6808/859282187", 2024-04-23T16:28:33.597 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6807/859282187", 2024-04-23T16:28:33.598 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:33.598 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:33.598 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:33.598 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.598 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 2, 2024-04-23T16:28:33.598 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "c17fffb4-8d17-4546-ba24-e5951a2a37fe", 2024-04-23T16:28:33.598 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:33.598 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:33.598 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:33.598 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:33.598 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:33.598 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:33.599 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 21, 2024-04-23T16:28:33.599 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 306, 2024-04-23T16:28:33.599 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:33.599 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:33.599 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:33.599 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.599 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.599 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6809", 2024-04-23T16:28:33.599 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 424240088 2024-04-23T16:28:33.599 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.599 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.599 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:33.600 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.600 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.600 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6810", 2024-04-23T16:28:33.600 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 424240088 2024-04-23T16:28:33.600 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.600 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.600 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:33.600 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.600 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.600 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6812", 2024-04-23T16:28:33.600 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 424240088 2024-04-23T16:28:33.601 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.601 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.601 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:33.601 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.601 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.601 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6811", 2024-04-23T16:28:33.601 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 424240088 2024-04-23T16:28:33.601 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.601 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.601 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6809/424240088", 2024-04-23T16:28:33.601 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6810/424240088", 2024-04-23T16:28:33.602 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6812/424240088", 2024-04-23T16:28:33.602 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6811/424240088", 2024-04-23T16:28:33.602 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:33.602 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:33.602 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:33.602 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.602 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 3, 2024-04-23T16:28:33.602 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "cee91a3d-55dc-4e89-b568-86821632bb1c", 2024-04-23T16:28:33.602 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:33.602 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:33.602 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:33.602 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:33.603 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:33.603 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:33.603 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 27, 2024-04-23T16:28:33.603 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 274, 2024-04-23T16:28:33.603 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:33.603 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:33.603 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:33.603 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.603 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.603 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6813", 2024-04-23T16:28:33.603 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1895183074 2024-04-23T16:28:33.603 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.603 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.604 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:33.604 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.604 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.604 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6814", 2024-04-23T16:28:33.604 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1895183074 2024-04-23T16:28:33.604 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.604 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.604 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:33.604 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.604 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.604 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6816", 2024-04-23T16:28:33.605 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1895183074 2024-04-23T16:28:33.605 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.605 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.605 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:33.605 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.605 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.605 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6815", 2024-04-23T16:28:33.605 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1895183074 2024-04-23T16:28:33.605 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.605 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.605 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6813/1895183074", 2024-04-23T16:28:33.606 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6814/1895183074", 2024-04-23T16:28:33.606 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6816/1895183074", 2024-04-23T16:28:33.606 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6815/1895183074", 2024-04-23T16:28:33.606 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:33.606 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:33.606 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:33.606 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.606 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 4, 2024-04-23T16:28:33.606 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "61d5385b-2c82-42f8-93c3-8f1baf71c65e", 2024-04-23T16:28:33.606 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:33.606 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:33.607 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:33.607 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:33.607 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:33.607 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:33.607 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 33, 2024-04-23T16:28:33.607 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 274, 2024-04-23T16:28:33.607 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:33.607 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:33.607 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:33.607 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.607 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.607 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6800", 2024-04-23T16:28:33.607 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 794945071 2024-04-23T16:28:33.608 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.608 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.608 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:33.608 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.608 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.608 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6801", 2024-04-23T16:28:33.608 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 794945071 2024-04-23T16:28:33.608 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.608 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.608 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:33.609 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.609 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.609 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6803", 2024-04-23T16:28:33.609 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 794945071 2024-04-23T16:28:33.609 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.609 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.609 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:33.609 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.609 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.609 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6802", 2024-04-23T16:28:33.609 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 794945071 2024-04-23T16:28:33.610 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.610 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.610 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6800/794945071", 2024-04-23T16:28:33.610 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6801/794945071", 2024-04-23T16:28:33.610 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6803/794945071", 2024-04-23T16:28:33.610 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6802/794945071", 2024-04-23T16:28:33.610 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:33.610 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:33.610 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:33.610 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.610 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 5, 2024-04-23T16:28:33.611 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "5d492189-c3cd-47ad-a12a-ed4d275bcfe3", 2024-04-23T16:28:33.611 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:33.611 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:33.611 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:33.611 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:33.611 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:33.611 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:33.611 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 38, 2024-04-23T16:28:33.611 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 306, 2024-04-23T16:28:33.611 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:33.611 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:33.611 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:33.611 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.612 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.612 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6804", 2024-04-23T16:28:33.612 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:28:33.612 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.612 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.612 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:33.612 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.612 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.612 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6805", 2024-04-23T16:28:33.612 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:28:33.612 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.612 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.613 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:33.613 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.613 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.613 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6807", 2024-04-23T16:28:33.613 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:28:33.613 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.613 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.613 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:33.613 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.613 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.613 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6806", 2024-04-23T16:28:33.614 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:28:33.614 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.614 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.614 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6804/3547200711", 2024-04-23T16:28:33.614 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6805/3547200711", 2024-04-23T16:28:33.614 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6807/3547200711", 2024-04-23T16:28:33.614 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6806/3547200711", 2024-04-23T16:28:33.614 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:33.614 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:33.614 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:33.614 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.615 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 6, 2024-04-23T16:28:33.615 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "89f2379e-cf14-4e69-8747-7535db81b70b", 2024-04-23T16:28:33.615 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:33.615 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:33.615 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:33.615 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:33.615 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:33.615 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:33.615 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 47, 2024-04-23T16:28:33.615 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 306, 2024-04-23T16:28:33.615 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:33.615 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:33.615 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:33.615 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.616 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.616 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6808", 2024-04-23T16:28:33.616 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:28:33.616 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.616 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.616 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:33.616 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.616 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.616 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6809", 2024-04-23T16:28:33.616 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:28:33.616 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.617 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.617 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:33.617 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.617 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.617 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6811", 2024-04-23T16:28:33.617 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:28:33.617 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.617 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.617 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:33.617 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.618 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.618 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6810", 2024-04-23T16:28:33.618 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:28:33.618 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.618 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.618 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6808/366919758", 2024-04-23T16:28:33.618 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6809/366919758", 2024-04-23T16:28:33.618 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6811/366919758", 2024-04-23T16:28:33.618 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6810/366919758", 2024-04-23T16:28:33.618 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:33.618 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:33.618 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:33.618 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.619 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 7, 2024-04-23T16:28:33.619 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "f27b035a-558f-4528-80ea-2a00d03f6539", 2024-04-23T16:28:33.619 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:33.619 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:33.619 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:33.619 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 0.7818145751953125, 2024-04-23T16:28:33.619 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:33.619 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:33.619 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 56, 2024-04-23T16:28:33.619 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 305, 2024-04-23T16:28:33.619 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:33.620 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:33.620 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:33.620 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.620 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.620 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6812", 2024-04-23T16:28:33.620 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:28:33.620 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.620 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.620 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:33.620 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.620 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.621 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6813", 2024-04-23T16:28:33.621 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:28:33.621 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.621 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.621 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:33.621 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.621 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.621 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6815", 2024-04-23T16:28:33.621 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:28:33.621 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.621 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.622 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:33.622 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.622 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.622 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6814", 2024-04-23T16:28:33.622 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:28:33.622 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.622 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.622 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6812/3719550888", 2024-04-23T16:28:33.622 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6813/3719550888", 2024-04-23T16:28:33.622 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6815/3719550888", 2024-04-23T16:28:33.622 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6814/3719550888", 2024-04-23T16:28:33.623 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:33.623 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:33.623 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:33.623 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.623 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 8, 2024-04-23T16:28:33.623 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed", 2024-04-23T16:28:33.623 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:33.623 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:33.623 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:33.623 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:33.623 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:33.623 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:33.623 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 66, 2024-04-23T16:28:33.624 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 192, 2024-04-23T16:28:33.624 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:33.624 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:33.624 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:33.624 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.624 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.624 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6800", 2024-04-23T16:28:33.624 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:28:33.624 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.624 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.624 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:33.624 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.625 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.625 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6801", 2024-04-23T16:28:33.625 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:28:33.625 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.625 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.625 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:33.625 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.625 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.625 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6803", 2024-04-23T16:28:33.625 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:28:33.625 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.625 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.626 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:33.626 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.626 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.626 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6802", 2024-04-23T16:28:33.626 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:28:33.626 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.626 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.626 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6800/1151408595", 2024-04-23T16:28:33.626 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6801/1151408595", 2024-04-23T16:28:33.626 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6803/1151408595", 2024-04-23T16:28:33.626 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6802/1151408595", 2024-04-23T16:28:33.626 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:33.627 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:33.627 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:33.627 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.627 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 9, 2024-04-23T16:28:33.627 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "124d9b05-11eb-4622-b6a2-c4d5bd60b7e6", 2024-04-23T16:28:33.627 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:33.627 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:33.627 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:33.627 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:33.627 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:33.627 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:33.627 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 73, 2024-04-23T16:28:33.627 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 306, 2024-04-23T16:28:33.628 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:33.628 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:33.628 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:33.628 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.628 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.628 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6804", 2024-04-23T16:28:33.628 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:28:33.628 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.628 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.628 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:33.628 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.629 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.629 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6805", 2024-04-23T16:28:33.629 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:28:33.629 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.629 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.629 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:33.629 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.629 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.629 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6807", 2024-04-23T16:28:33.629 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:28:33.630 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.630 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.630 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:33.630 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.630 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.630 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6806", 2024-04-23T16:28:33.630 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:28:33.630 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.630 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.630 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6804/306501862", 2024-04-23T16:28:33.630 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6805/306501862", 2024-04-23T16:28:33.631 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6807/306501862", 2024-04-23T16:28:33.631 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6806/306501862", 2024-04-23T16:28:33.631 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:33.631 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:33.631 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:33.631 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.631 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 10, 2024-04-23T16:28:33.631 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "3dfb9070-e534-4ea4-a25e-ee1c51297447", 2024-04-23T16:28:33.631 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:33.631 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 0, 2024-04-23T16:28:33.631 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 0, 2024-04-23T16:28:33.631 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:33.632 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:33.632 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:33.632 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 81, 2024-04-23T16:28:33.632 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 0, 2024-04-23T16:28:33.632 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:33.632 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:33.632 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:33.632 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.632 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.632 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6808", 2024-04-23T16:28:33.632 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:28:33.632 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.632 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.633 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:33.633 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.633 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.633 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6809", 2024-04-23T16:28:33.633 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:28:33.633 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.633 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.633 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:33.633 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.633 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.633 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6811", 2024-04-23T16:28:33.634 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:28:33.634 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.634 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.634 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:33.634 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.634 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.634 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6810", 2024-04-23T16:28:33.634 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:28:33.634 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.634 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.634 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6808/1231263138", 2024-04-23T16:28:33.635 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6809/1231263138", 2024-04-23T16:28:33.635 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6811/1231263138", 2024-04-23T16:28:33.635 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6810/1231263138", 2024-04-23T16:28:33.635 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:33.635 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:33.635 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:33.635 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.635 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 11, 2024-04-23T16:28:33.635 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "bc163925-f623-4f63-ab89-35e2d511a730", 2024-04-23T16:28:33.635 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:33.635 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:33.635 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:33.636 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:33.636 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:33.636 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:33.636 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 90, 2024-04-23T16:28:33.636 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 303, 2024-04-23T16:28:33.636 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:33.636 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:33.636 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:33.636 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.636 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.636 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6812", 2024-04-23T16:28:33.636 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:28:33.636 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.637 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.637 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:33.637 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.637 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.637 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6813", 2024-04-23T16:28:33.637 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:28:33.637 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.637 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.637 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:33.637 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.638 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.638 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6815", 2024-04-23T16:28:33.638 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:28:33.638 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.638 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.638 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:33.638 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:33.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.638 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:33.638 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6814", 2024-04-23T16:28:33.638 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:28:33.638 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.638 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.639 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6812/457992661", 2024-04-23T16:28:33.639 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6813/457992661", 2024-04-23T16:28:33.639 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6815/457992661", 2024-04-23T16:28:33.639 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6814/457992661", 2024-04-23T16:28:33.639 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:33.639 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:33.639 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:33.639 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.639 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.639 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:28:33.639 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_xinfo": [ 2024-04-23T16:28:33.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.639 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 0, 2024-04-23T16:28:33.640 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:33.640 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:33.640 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:33.640 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:33.640 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:33.640 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:28:04.955347+0000", 2024-04-23T16:28:33.640 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:33.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.640 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 1, 2024-04-23T16:28:33.640 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:33.640 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:33.640 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:33.640 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:33.640 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 65536, 2024-04-23T16:28:33.641 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:28:03.447957+0000", 2024-04-23T16:28:33.641 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 232 2024-04-23T16:28:33.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.641 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 2, 2024-04-23T16:28:33.641 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:33.641 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:33.641 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:33.641 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:33.641 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:33.641 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:27:46.330329+0000", 2024-04-23T16:28:33.641 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:33.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.642 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 3, 2024-04-23T16:28:33.642 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:33.642 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:33.642 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:33.642 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:33.642 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:33.642 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:28:00.560710+0000", 2024-04-23T16:28:33.642 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:33.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.642 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 4, 2024-04-23T16:28:33.642 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:33.642 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:33.642 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:33.643 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:33.643 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:33.643 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:27:53.012954+0000", 2024-04-23T16:28:33.643 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:33.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.643 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 5, 2024-04-23T16:28:33.643 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:33.643 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:33.643 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:33.643 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:33.643 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:33.643 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:28:31.926363+0000", 2024-04-23T16:28:33.643 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:33.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.644 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 6, 2024-04-23T16:28:33.644 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:33.644 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:33.644 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:33.644 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:33.644 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:33.644 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:27:52.236962+0000", 2024-04-23T16:28:33.644 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:33.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.644 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 7, 2024-04-23T16:28:33.644 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:33.644 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:33.645 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:33.645 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:33.645 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:33.645 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:27:31.235811+0000", 2024-04-23T16:28:33.645 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:33.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.645 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 8, 2024-04-23T16:28:33.645 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:33.645 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:33.645 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:33.645 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:33.645 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:33.645 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:28:31.427008+0000", 2024-04-23T16:28:33.646 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:33.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.646 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 9, 2024-04-23T16:28:33.646 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:33.646 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:33.646 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:33.646 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:33.646 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:33.646 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:27:20.706699+0000", 2024-04-23T16:28:33.646 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:33.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.646 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 10, 2024-04-23T16:28:33.646 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:33.647 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:33.647 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:33.647 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:33.647 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 65536, 2024-04-23T16:28:33.647 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:28:28.494775+0000", 2024-04-23T16:28:33.647 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:33.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.647 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 11, 2024-04-23T16:28:33.647 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:33.647 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:33.647 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:33.647 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:33.647 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:33.648 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:28:28.083599+0000", 2024-04-23T16:28:33.648 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:33.648 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.648 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:28:33.648 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_upmap": [ 2024-04-23T16:28:33.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.648 INFO:teuthology.orchestra.run.smithi103.stdout: "pgid": "4.6", 2024-04-23T16:28:33.648 INFO:teuthology.orchestra.run.smithi103.stdout: "osds": [ 2024-04-23T16:28:33.648 INFO:teuthology.orchestra.run.smithi103.stdout: 6, 2024-04-23T16:28:33.648 INFO:teuthology.orchestra.run.smithi103.stdout: 9 2024-04-23T16:28:33.648 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.648 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.648 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:28:33.648 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_upmap_items": [], 2024-04-23T16:28:33.649 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_upmap_primaries": [], 2024-04-23T16:28:33.649 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_temp": [ 2024-04-23T16:28:33.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:33.649 INFO:teuthology.orchestra.run.smithi103.stdout: "pgid": "4.6", 2024-04-23T16:28:33.649 INFO:teuthology.orchestra.run.smithi103.stdout: "osds": [ 2024-04-23T16:28:33.649 INFO:teuthology.orchestra.run.smithi103.stdout: 2, 2024-04-23T16:28:33.649 INFO:teuthology.orchestra.run.smithi103.stdout: 9 2024-04-23T16:28:33.649 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:33.649 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.649 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:28:33.649 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_temp": [], 2024-04-23T16:28:33.649 INFO:teuthology.orchestra.run.smithi103.stdout: "blocklist": { 2024-04-23T16:28:33.649 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:6800/3535470771": "2024-04-24T16:09:14.324448+0000", 2024-04-23T16:28:33.649 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/1188058303": "2024-04-24T16:09:14.324448+0000", 2024-04-23T16:28:33.650 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/1510553721": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:28:33.650 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/281182801": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:28:33.650 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:6800/2049124518": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:28:33.650 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/466241963": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:28:33.650 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/2726838698": "2024-04-24T16:09:14.324448+0000", 2024-04-23T16:28:33.650 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/596230573": "2024-04-24T16:09:14.324448+0000" 2024-04-23T16:28:33.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.650 INFO:teuthology.orchestra.run.smithi103.stdout: "range_blocklist": {}, 2024-04-23T16:28:33.650 INFO:teuthology.orchestra.run.smithi103.stdout: "erasure_code_profiles": { 2024-04-23T16:28:33.650 INFO:teuthology.orchestra.run.smithi103.stdout: "default": { 2024-04-23T16:28:33.650 INFO:teuthology.orchestra.run.smithi103.stdout: "crush-failure-domain": "osd", 2024-04-23T16:28:33.650 INFO:teuthology.orchestra.run.smithi103.stdout: "k": "2", 2024-04-23T16:28:33.650 INFO:teuthology.orchestra.run.smithi103.stdout: "m": "1", 2024-04-23T16:28:33.650 INFO:teuthology.orchestra.run.smithi103.stdout: "plugin": "jerasure", 2024-04-23T16:28:33.651 INFO:teuthology.orchestra.run.smithi103.stdout: "technique": "reed_sol_van" 2024-04-23T16:28:33.651 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:33.651 INFO:teuthology.orchestra.run.smithi103.stdout: "removed_snaps_queue": [], 2024-04-23T16:28:33.651 INFO:teuthology.orchestra.run.smithi103.stdout: "new_removed_snaps": [], 2024-04-23T16:28:33.651 INFO:teuthology.orchestra.run.smithi103.stdout: "new_purged_snaps": [], 2024-04-23T16:28:33.651 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_node_flags": {}, 2024-04-23T16:28:33.651 INFO:teuthology.orchestra.run.smithi103.stdout: "device_class_flags": {}, 2024-04-23T16:28:33.651 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_mode": { 2024-04-23T16:28:33.651 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_mode_enabled": false, 2024-04-23T16:28:33.651 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_bucket_count": 0, 2024-04-23T16:28:33.651 INFO:teuthology.orchestra.run.smithi103.stdout: "degraded_stretch_mode": 0, 2024-04-23T16:28:33.651 INFO:teuthology.orchestra.run.smithi103.stdout: "recovering_stretch_mode": 0, 2024-04-23T16:28:33.651 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_mode_bucket": 0 2024-04-23T16:28:33.652 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:33.652 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:28:33.652 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:28:33.653 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:28:33.715 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:33.772 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:28:33.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:33.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:28:33.829 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 76 16 147844 147828 121.55 71.625 0.0080336 0.00822573 2024-04-23T16:28:33.878 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:28:33.920 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:33.956 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:28:33.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:33.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:28:34.064 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:28:34.089 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:33 smithi188 ceph-mon[44101]: pgmap v879: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 8.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 61 MiB/s wr, 980 op/s; 5212/286258 objects misplaced (1.821%); 1.6 MiB/s, 25 objects/s recovering 2024-04-23T16:28:34.090 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:33 smithi188 ceph-mon[44101]: osdmap e311: 12 total, 12 up, 10 in 2024-04-23T16:28:34.090 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:33 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1563865535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-23T16:28:34.092 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:33 smithi103 ceph-mon[39629]: pgmap v879: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 8.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 61 MiB/s wr, 980 op/s; 5212/286258 objects misplaced (1.821%); 1.6 MiB/s, 25 objects/s recovering 2024-04-23T16:28:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:33 smithi103 ceph-mon[39629]: osdmap e311: 12 total, 12 up, 10 in 2024-04-23T16:28:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:33 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1563865535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-23T16:28:34.114 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:28:34.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:34.118 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:28:34.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:33 smithi148 ceph-mon[44520]: pgmap v879: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 8.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 61 MiB/s wr, 980 op/s; 5212/286258 objects misplaced (1.821%); 1.6 MiB/s, 25 objects/s recovering 2024-04-23T16:28:34.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:33 smithi148 ceph-mon[44520]: osdmap e311: 12 total, 12 up, 10 in 2024-04-23T16:28:34.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:33 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1563865535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-23T16:28:34.218 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:28:34.248 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:34.270 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:28:34.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:34.276 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:28:34.377 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:28:34.407 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:34.511 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:28:34.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:34.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:28:34.617 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:28:34.643 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:28:34.656 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:34.741 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:28:34.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:34.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:28:34.829 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 77 16 148960 148944 120.877 69.75 0.00566058 0.00827017 2024-04-23T16:28:34.846 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:28:34.907 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:35.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:35 smithi103 ceph-mon[39629]: pgmap v881: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s; 5069/293484 objects misplaced (1.727%); 1.5 MiB/s, 24 objects/s recovering 2024-04-23T16:28:35.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:35 smithi148 ceph-mon[44520]: pgmap v881: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s; 5069/293484 objects misplaced (1.727%); 1.5 MiB/s, 24 objects/s recovering 2024-04-23T16:28:35.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:35 smithi188 ceph-mon[44101]: pgmap v881: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s; 5069/293484 objects misplaced (1.727%); 1.5 MiB/s, 24 objects/s recovering 2024-04-23T16:28:35.829 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 78 16 150143 150127 120.275 73.9375 0.00894799 0.00831268 2024-04-23T16:28:36.066 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:28:36.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:36.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:28:36.179 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:28:36.221 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:36.435 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:28:36.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:36.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:28:36.466 INFO:tasks.thrashosds.thrasher:j is {'epoch': 311, 'fsid': '8e3594f2-018b-11ef-bc93-c7b262605968', 'created': '2024-04-23T16:08:13.477590+0000', 'modified': '2024-04-23T16:28:32.787463+0000', 'last_up_change': '2024-04-23T16:27:00.729195+0000', 'last_in_change': '2024-04-23T16:24:50.783418+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T16:11:22.416777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '303', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '301', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.2899999618530273, 'score_stable': 3.2899999618530273, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 2.7300000190734863, 'raw_score_stable': 2.7300000190734863, 'primary_affinity_weighted': 0.9818177819252014, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-04-23T16:27:07.774581+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '281', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9818177819252014, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}], 'osds': [{'osd': 0, 'uuid': '4c084047-8f01-4b86-a60c-5592735b5862', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6801', 'nonce': 4274040097}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6802', 'nonce': 4274040097}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6804', 'nonce': 4274040097}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6803', 'nonce': 4274040097}]}, 'public_addr': '172.21.15.103:6801/4274040097', 'cluster_addr': '172.21.15.103:6802/4274040097', 'heartbeat_back_addr': '172.21.15.103:6804/4274040097', 'heartbeat_front_addr': '172.21.15.103:6803/4274040097', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '3438f2bd-5fb7-432c-b383-29e18c442f20', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 232, 'up_from': 271, 'up_thru': 192, 'down_at': 233, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6805', 'nonce': 859282187}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6806', 'nonce': 859282187}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6808', 'nonce': 859282187}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6807', 'nonce': 859282187}]}, 'public_addr': '172.21.15.103:6805/859282187', 'cluster_addr': '172.21.15.103:6806/859282187', 'heartbeat_back_addr': '172.21.15.103:6808/859282187', 'heartbeat_front_addr': '172.21.15.103:6807/859282187', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c17fffb4-8d17-4546-ba24-e5951a2a37fe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6809', 'nonce': 424240088}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6810', 'nonce': 424240088}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6812', 'nonce': 424240088}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6811', 'nonce': 424240088}]}, 'public_addr': '172.21.15.103:6809/424240088', 'cluster_addr': '172.21.15.103:6810/424240088', 'heartbeat_back_addr': '172.21.15.103:6812/424240088', 'heartbeat_front_addr': '172.21.15.103:6811/424240088', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'cee91a3d-55dc-4e89-b568-86821632bb1c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6813', 'nonce': 1895183074}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6814', 'nonce': 1895183074}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6816', 'nonce': 1895183074}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6815', 'nonce': 1895183074}]}, 'public_addr': '172.21.15.103:6813/1895183074', 'cluster_addr': '172.21.15.103:6814/1895183074', 'heartbeat_back_addr': '172.21.15.103:6816/1895183074', 'heartbeat_front_addr': '172.21.15.103:6815/1895183074', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '61d5385b-2c82-42f8-93c3-8f1baf71c65e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6800', 'nonce': 794945071}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6801', 'nonce': 794945071}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6803', 'nonce': 794945071}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6802', 'nonce': 794945071}]}, 'public_addr': '172.21.15.148:6800/794945071', 'cluster_addr': '172.21.15.148:6801/794945071', 'heartbeat_back_addr': '172.21.15.148:6803/794945071', 'heartbeat_front_addr': '172.21.15.148:6802/794945071', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '5d492189-c3cd-47ad-a12a-ed4d275bcfe3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6804', 'nonce': 3547200711}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6805', 'nonce': 3547200711}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6807', 'nonce': 3547200711}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6806', 'nonce': 3547200711}]}, 'public_addr': '172.21.15.148:6804/3547200711', 'cluster_addr': '172.21.15.148:6805/3547200711', 'heartbeat_back_addr': '172.21.15.148:6807/3547200711', 'heartbeat_front_addr': '172.21.15.148:6806/3547200711', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '89f2379e-cf14-4e69-8747-7535db81b70b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6808', 'nonce': 366919758}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6809', 'nonce': 366919758}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6811', 'nonce': 366919758}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6810', 'nonce': 366919758}]}, 'public_addr': '172.21.15.148:6808/366919758', 'cluster_addr': '172.21.15.148:6809/366919758', 'heartbeat_back_addr': '172.21.15.148:6811/366919758', 'heartbeat_front_addr': '172.21.15.148:6810/366919758', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f27b035a-558f-4528-80ea-2a00d03f6539', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7818145751953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 305, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6812', 'nonce': 3719550888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6813', 'nonce': 3719550888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6815', 'nonce': 3719550888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6814', 'nonce': 3719550888}]}, 'public_addr': '172.21.15.148:6812/3719550888', 'cluster_addr': '172.21.15.148:6813/3719550888', 'heartbeat_back_addr': '172.21.15.148:6815/3719550888', 'heartbeat_front_addr': '172.21.15.148:6814/3719550888', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 192, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6800', 'nonce': 1151408595}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6801', 'nonce': 1151408595}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6803', 'nonce': 1151408595}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6802', 'nonce': 1151408595}]}, 'public_addr': '172.21.15.188:6800/1151408595', 'cluster_addr': '172.21.15.188:6801/1151408595', 'heartbeat_back_addr': '172.21.15.188:6803/1151408595', 'heartbeat_front_addr': '172.21.15.188:6802/1151408595', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '124d9b05-11eb-4622-b6a2-c4d5bd60b7e6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6804', 'nonce': 306501862}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6805', 'nonce': 306501862}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6807', 'nonce': 306501862}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6806', 'nonce': 306501862}]}, 'public_addr': '172.21.15.188:6804/306501862', 'cluster_addr': '172.21.15.188:6805/306501862', 'heartbeat_back_addr': '172.21.15.188:6807/306501862', 'heartbeat_front_addr': '172.21.15.188:6806/306501862', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '3dfb9070-e534-4ea4-a25e-ee1c51297447', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6808', 'nonce': 1231263138}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6809', 'nonce': 1231263138}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6811', 'nonce': 1231263138}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6810', 'nonce': 1231263138}]}, 'public_addr': '172.21.15.188:6808/1231263138', 'cluster_addr': '172.21.15.188:6809/1231263138', 'heartbeat_back_addr': '172.21.15.188:6811/1231263138', 'heartbeat_front_addr': '172.21.15.188:6810/1231263138', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'bc163925-f623-4f63-ab89-35e2d511a730', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 303, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6812', 'nonce': 457992661}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6813', 'nonce': 457992661}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6815', 'nonce': 457992661}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6814', 'nonce': 457992661}]}, 'public_addr': '172.21.15.188:6812/457992661', 'cluster_addr': '172.21.15.188:6813/457992661', 'heartbeat_back_addr': '172.21.15.188:6815/457992661', 'heartbeat_front_addr': '172.21.15.188:6814/457992661', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:28:04.955347+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-23T16:28:03.447957+0000', 'dead_epoch': 232}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:27:46.330329+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:28:00.560710+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:27:53.012954+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:28:31.926363+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:27:52.236962+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:27:31.235811+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:28:31.427008+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:27:20.706699+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-23T16:28:28.494775+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:28:28.083599+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.6', 'osds': [6, 9]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '4.6', 'osds': [2, 9]}], 'primary_temp': [], 'blocklist': {'172.21.15.103:6800/3535470771': '2024-04-24T16:09:14.324448+0000', '172.21.15.103:0/1188058303': '2024-04-24T16:09:14.324448+0000', '172.21.15.103:0/1510553721': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:0/281182801': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:6800/2049124518': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:0/466241963': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:0/2726838698': '2024-04-24T16:09:14.324448+0000', '172.21.15.103:0/596230573': '2024-04-24T16:09:14.324448+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-23T16:28:36.466 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-23T16:28:36.540 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:28:36.588 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:36.628 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:28:36.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:36.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:28:36.733 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:28:36.795 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:36.829 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 79 16 151267 151251 119.642 70.25 0.011791 0.00835644 2024-04-23T16:28:36.834 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:28:36.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:36.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:28:36.942 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:28:36.996 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:37.202 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:28:37.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:37.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:28:37.239 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:28:37.239 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:28:37.239 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-23T16:28:37.239 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:28:37.310 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:28:37.342 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:37.829 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:28:37.829514 min lat: 0.00105502 max lat: 2.61652 avg lat: 0.0083989 2024-04-23T16:28:37.830 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:28:37.830 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 80 16 152413 152397 119.042 71.625 0.00451211 0.0083989 2024-04-23T16:28:37.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:37 smithi188 ceph-mon[44101]: pgmap v882: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s; 5069/297758 objects misplaced (1.702%); 1.2 MiB/s, 18 objects/s recovering 2024-04-23T16:28:37.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:37 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:28:37.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:37 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:28:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:37 smithi103 ceph-mon[39629]: pgmap v882: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s; 5069/297758 objects misplaced (1.702%); 1.2 MiB/s, 18 objects/s recovering 2024-04-23T16:28:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:37 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:28:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:37 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:28:38.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:37 smithi148 ceph-mon[44520]: pgmap v882: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s; 5069/297758 objects misplaced (1.702%); 1.2 MiB/s, 18 objects/s recovering 2024-04-23T16:28:38.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:37 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:28:38.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:37 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:28:38.830 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 81 16 153551 153535 118.45 71.125 0.00920435 0.0084408 2024-04-23T16:28:38.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:38 smithi188 ceph-mon[44101]: osdmap e312: 12 total, 12 up, 10 in 2024-04-23T16:28:39.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:38 smithi103 ceph-mon[39629]: osdmap e312: 12 total, 12 up, 10 in 2024-04-23T16:28:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:38 smithi148 ceph-mon[44520]: osdmap e312: 12 total, 12 up, 10 in 2024-04-23T16:28:39.830 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 82 16 154664 154648 117.854 69.5625 0.0104539 0.00848309 2024-04-23T16:28:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:39 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:28:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:39 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:28:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:39 smithi188 ceph-mon[44101]: pgmap v884: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 52 MiB/s wr, 838 op/s; 5069/297758 objects misplaced (1.702%); 1.3 MiB/s, 20 objects/s recovering 2024-04-23T16:28:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:39 smithi188 ceph-mon[44101]: osdmap e313: 12 total, 12 up, 10 in 2024-04-23T16:28:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:39 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2742457903' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:28:40.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:39 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:28:40.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:39 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:28:40.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:39 smithi103 ceph-mon[39629]: pgmap v884: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 52 MiB/s wr, 838 op/s; 5069/297758 objects misplaced (1.702%); 1.3 MiB/s, 20 objects/s recovering 2024-04-23T16:28:40.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:39 smithi103 ceph-mon[39629]: osdmap e313: 12 total, 12 up, 10 in 2024-04-23T16:28:40.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:39 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2742457903' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:28:40.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:39 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:28:40.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:39 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:28:40.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:39 smithi148 ceph-mon[44520]: pgmap v884: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 52 MiB/s wr, 838 op/s; 5069/297758 objects misplaced (1.702%); 1.3 MiB/s, 20 objects/s recovering 2024-04-23T16:28:40.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:39 smithi148 ceph-mon[44520]: osdmap e313: 12 total, 12 up, 10 in 2024-04-23T16:28:40.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:39 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2742457903' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:28:40.830 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 83 16 155795 155779 117.285 70.6875 0.00536557 0.00852493 2024-04-23T16:28:40.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:40 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:28:40.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:40 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2742457903' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:28:40.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:40 smithi188 ceph-mon[44101]: osdmap e314: 12 total, 12 up, 10 in 2024-04-23T16:28:41.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:40 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:28:41.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:40 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2742457903' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:28:41.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:40 smithi103 ceph-mon[39629]: osdmap e314: 12 total, 12 up, 10 in 2024-04-23T16:28:41.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:40 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:28:41.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:40 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2742457903' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:28:41.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:40 smithi148 ceph-mon[44520]: osdmap e314: 12 total, 12 up, 10 in 2024-04-23T16:28:41.302 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:28:41.467 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10, 1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1] 2024-04-23T16:28:41.467 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:28:41.467 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-23T16:28:41.468 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-23T16:28:41.515 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:28:41.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:41.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:28:41.625 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:28:41.675 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:41.830 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 84 16 156910 156894 116.719 69.6875 0.0104815 0.00856626 2024-04-23T16:28:41.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:41 smithi103 ceph-mon[39629]: pgmap v887: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s; 4935/306390 objects misplaced (1.611%); 1.4 MiB/s, 22 objects/s recovering 2024-04-23T16:28:42.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:41 smithi148 ceph-mon[44520]: pgmap v887: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s; 4935/306390 objects misplaced (1.611%); 1.4 MiB/s, 22 objects/s recovering 2024-04-23T16:28:42.181 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:28:42.236 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:41 smithi188 ceph-mon[44101]: pgmap v887: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s; 4935/306390 objects misplaced (1.611%); 1.4 MiB/s, 22 objects/s recovering 2024-04-23T16:28:42.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:42.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:28:42.338 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:28:42.371 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:42.393 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:28:42.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:42.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:28:42.498 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:28:42.529 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:42.562 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:28:42.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:42.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:28:42.570 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:28:42.611 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-23T16:28:42.667 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:28:42.695 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:42.722 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:28:42.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:42.726 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:28:42.827 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:28:42.830 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 85 16 158009 157993 116.154 68.6875 0.00935185 0.00860773 2024-04-23T16:28:42.890 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:42.917 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:28:42.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:42.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:28:43.021 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:28:43.059 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:43.167 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:28:43.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:43.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:28:43.281 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:28:43.331 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:43.380 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:28:43.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:43.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:28:43.485 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:28:43.522 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:43.547 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:28:43.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:43.552 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:28:43.605 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:28:43.615 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:28:43.652 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:28:43.695 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:43.830 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 86 16 159061 159045 115.568 65.75 0.00582566 0.00865054 2024-04-23T16:28:43.951 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:28:43.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:43.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:28:43.958 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:43 smithi103 ceph-mon[39629]: pgmap v888: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 60 MiB/s wr, 952 op/s; 4935/309192 objects misplaced (1.596%); 1.4 MiB/s, 22 objects/s recovering 2024-04-23T16:28:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:43 smithi188 ceph-mon[44101]: pgmap v888: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 60 MiB/s wr, 952 op/s; 4935/309192 objects misplaced (1.596%); 1.4 MiB/s, 22 objects/s recovering 2024-04-23T16:28:44.058 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:28:44.117 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:43 smithi148 ceph-mon[44520]: pgmap v888: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 60 MiB/s wr, 952 op/s; 4935/309192 objects misplaced (1.596%); 1.4 MiB/s, 22 objects/s recovering 2024-04-23T16:28:44.124 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:44.167 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:28:44.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:44.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:28:44.238 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:28:44.273 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:28:44.303 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:44.325 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:28:44.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:44.328 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:28:44.429 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:28:44.474 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:44.512 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:28:44.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:44.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:28:44.620 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:28:44.653 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:44.680 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:28:44.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:44.691 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:28:44.792 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:28:44.821 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:44.830 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 87 16 160124 160108 115.003 66.4375 0.0406863 0.00869374 2024-04-23T16:28:44.843 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:28:44.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:44.847 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:28:44.948 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:28:44.989 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:45.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:45 smithi103 ceph-mon[39629]: osdmap e315: 12 total, 12 up, 10 in 2024-04-23T16:28:45.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:45 smithi103 ceph-mon[39629]: pgmap v890: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 96 MiB/s wr, 1.53k op/s; 4776/316154 objects misplaced (1.511%); 3.1 MiB/s, 48 objects/s recovering 2024-04-23T16:28:45.350 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:28:45.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:45.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:28:45.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:45 smithi148 ceph-mon[44520]: osdmap e315: 12 total, 12 up, 10 in 2024-04-23T16:28:45.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:45 smithi148 ceph-mon[44520]: pgmap v890: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 96 MiB/s wr, 1.53k op/s; 4776/316154 objects misplaced (1.511%); 3.1 MiB/s, 48 objects/s recovering 2024-04-23T16:28:45.454 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:28:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:45 smithi188 ceph-mon[44101]: osdmap e315: 12 total, 12 up, 10 in 2024-04-23T16:28:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:45 smithi188 ceph-mon[44101]: pgmap v890: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 96 MiB/s wr, 1.53k op/s; 4776/316154 objects misplaced (1.511%); 3.1 MiB/s, 48 objects/s recovering 2024-04-23T16:28:45.497 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:45.830 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 88 16 161255 161239 114.499 70.6875 0.0102268 0.00873187 2024-04-23T16:28:45.986 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:28:45.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:45.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:28:46.098 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:28:46.131 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:46.177 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:28:46.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:46.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:28:46.285 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:28:46.326 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:46.721 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:46 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3125794274' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:28:46.726 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:28:46.740 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:28:46.767 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:28:46.807 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:28:46.830 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 89 16 162358 162342 113.987 68.9375 0.00631925 0.00877127 2024-04-23T16:28:46.844 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:28:46.872 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:28:46.919 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:28:46.952 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:28:46.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:46 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3125794274' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:28:46.992 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:28:47.026 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:28:47.056 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:28:47.092 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:28:47.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:28:47.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:28:47.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:28:47.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:28:47.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:28:47.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:28:47.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:28:47.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:28:47.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:28:47.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:28:47.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:28:47.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:28:47.100 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:28:47.100 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:28:47.100 INFO:teuthology.orchestra.run.smithi103.stdout: "epoch": 315, 2024-04-23T16:28:47.100 INFO:teuthology.orchestra.run.smithi103.stdout: "fsid": "8e3594f2-018b-11ef-bc93-c7b262605968", 2024-04-23T16:28:47.100 INFO:teuthology.orchestra.run.smithi103.stdout: "created": "2024-04-23T16:08:13.477590+0000", 2024-04-23T16:28:47.101 INFO:teuthology.orchestra.run.smithi103.stdout: "modified": "2024-04-23T16:28:44.038128+0000", 2024-04-23T16:28:47.101 INFO:teuthology.orchestra.run.smithi103.stdout: "last_up_change": "2024-04-23T16:27:00.729195+0000", 2024-04-23T16:28:47.101 INFO:teuthology.orchestra.run.smithi103.stdout: "last_in_change": "2024-04-23T16:24:50.783418+0000", 2024-04-23T16:28:47.101 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-23T16:28:47.101 INFO:teuthology.orchestra.run.smithi103.stdout: "flags_num": 5806080, 2024-04-23T16:28:47.101 INFO:teuthology.orchestra.run.smithi103.stdout: "flags_set": [ 2024-04-23T16:28:47.101 INFO:teuthology.orchestra.run.smithi103.stdout: "nodeep-scrub", 2024-04-23T16:28:47.101 INFO:teuthology.orchestra.run.smithi103.stdout: "noscrub", 2024-04-23T16:28:47.101 INFO:teuthology.orchestra.run.smithi103.stdout: "pglog_hardlimit", 2024-04-23T16:28:47.101 INFO:teuthology.orchestra.run.smithi103.stdout: "purged_snapdirs", 2024-04-23T16:28:47.101 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_deletes", 2024-04-23T16:28:47.101 INFO:teuthology.orchestra.run.smithi103.stdout: "sortbitwise" 2024-04-23T16:28:47.101 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:28:47.102 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_version": 28, 2024-04-23T16:28:47.102 INFO:teuthology.orchestra.run.smithi103.stdout: "full_ratio": 0.94999998807907104, 2024-04-23T16:28:47.102 INFO:teuthology.orchestra.run.smithi103.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-23T16:28:47.102 INFO:teuthology.orchestra.run.smithi103.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-23T16:28:47.102 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_snapshot": "", 2024-04-23T16:28:47.102 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_max": 4, 2024-04-23T16:28:47.102 INFO:teuthology.orchestra.run.smithi103.stdout: "max_osd": 12, 2024-04-23T16:28:47.102 INFO:teuthology.orchestra.run.smithi103.stdout: "require_min_compat_client": "luminous", 2024-04-23T16:28:47.102 INFO:teuthology.orchestra.run.smithi103.stdout: "min_compat_client": "luminous", 2024-04-23T16:28:47.102 INFO:teuthology.orchestra.run.smithi103.stdout: "require_osd_release": "squid", 2024-04-23T16:28:47.102 INFO:teuthology.orchestra.run.smithi103.stdout: "allow_crimson": false, 2024-04-23T16:28:47.102 INFO:teuthology.orchestra.run.smithi103.stdout: "pools": [ 2024-04-23T16:28:47.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.102 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-04-23T16:28:47.103 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_name": ".mgr", 2024-04-23T16:28:47.103 INFO:teuthology.orchestra.run.smithi103.stdout: "create_time": "2024-04-23T16:11:22.416777+0000", 2024-04-23T16:28:47.103 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-04-23T16:28:47.103 INFO:teuthology.orchestra.run.smithi103.stdout: "flags_names": "hashpspool", 2024-04-23T16:28:47.103 INFO:teuthology.orchestra.run.smithi103.stdout: "type": 1, 2024-04-23T16:28:47.103 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 2, 2024-04-23T16:28:47.103 INFO:teuthology.orchestra.run.smithi103.stdout: "min_size": 2, 2024-04-23T16:28:47.103 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_rule": 0, 2024-04-23T16:28:47.103 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_count": 0, 2024-04-23T16:28:47.103 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_target": 0, 2024-04-23T16:28:47.103 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T16:28:47.103 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T16:28:47.103 INFO:teuthology.orchestra.run.smithi103.stdout: "object_hash": 2, 2024-04-23T16:28:47.104 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_autoscale_mode": "off", 2024-04-23T16:28:47.104 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num": 11, 2024-04-23T16:28:47.104 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_placement_num": 11, 2024-04-23T16:28:47.104 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_placement_num_target": 11, 2024-04-23T16:28:47.104 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_target": 11, 2024-04-23T16:28:47.104 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_pending": 11, 2024-04-23T16:28:47.104 INFO:teuthology.orchestra.run.smithi103.stdout: "last_pg_merge_meta": { 2024-04-23T16:28:47.104 INFO:teuthology.orchestra.run.smithi103.stdout: "source_pgid": "0.0", 2024-04-23T16:28:47.104 INFO:teuthology.orchestra.run.smithi103.stdout: "ready_epoch": 0, 2024-04-23T16:28:47.104 INFO:teuthology.orchestra.run.smithi103.stdout: "last_epoch_started": 0, 2024-04-23T16:28:47.104 INFO:teuthology.orchestra.run.smithi103.stdout: "last_epoch_clean": 0, 2024-04-23T16:28:47.104 INFO:teuthology.orchestra.run.smithi103.stdout: "source_version": "0'0", 2024-04-23T16:28:47.104 INFO:teuthology.orchestra.run.smithi103.stdout: "target_version": "0'0" 2024-04-23T16:28:47.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.105 INFO:teuthology.orchestra.run.smithi103.stdout: "last_change": "303", 2024-04-23T16:28:47.105 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend": "0", 2024-04-23T16:28:47.105 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T16:28:47.105 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend_preluminous": "301", 2024-04-23T16:28:47.105 INFO:teuthology.orchestra.run.smithi103.stdout: "auid": 0, 2024-04-23T16:28:47.105 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_mode": "selfmanaged", 2024-04-23T16:28:47.105 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_seq": 0, 2024-04-23T16:28:47.105 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_epoch": 0, 2024-04-23T16:28:47.105 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_snaps": [], 2024-04-23T16:28:47.105 INFO:teuthology.orchestra.run.smithi103.stdout: "removed_snaps": "[]", 2024-04-23T16:28:47.105 INFO:teuthology.orchestra.run.smithi103.stdout: "quota_max_bytes": 0, 2024-04-23T16:28:47.105 INFO:teuthology.orchestra.run.smithi103.stdout: "quota_max_objects": 0, 2024-04-23T16:28:47.105 INFO:teuthology.orchestra.run.smithi103.stdout: "tiers": [], 2024-04-23T16:28:47.105 INFO:teuthology.orchestra.run.smithi103.stdout: "tier_of": -1, 2024-04-23T16:28:47.106 INFO:teuthology.orchestra.run.smithi103.stdout: "read_tier": -1, 2024-04-23T16:28:47.106 INFO:teuthology.orchestra.run.smithi103.stdout: "write_tier": -1, 2024-04-23T16:28:47.106 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_mode": "none", 2024-04-23T16:28:47.106 INFO:teuthology.orchestra.run.smithi103.stdout: "target_max_bytes": 0, 2024-04-23T16:28:47.106 INFO:teuthology.orchestra.run.smithi103.stdout: "target_max_objects": 0, 2024-04-23T16:28:47.106 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T16:28:47.106 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T16:28:47.106 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T16:28:47.106 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_min_flush_age": 0, 2024-04-23T16:28:47.106 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_min_evict_age": 0, 2024-04-23T16:28:47.106 INFO:teuthology.orchestra.run.smithi103.stdout: "erasure_code_profile": "", 2024-04-23T16:28:47.106 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_params": { 2024-04-23T16:28:47.106 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "none" 2024-04-23T16:28:47.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.107 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_period": 0, 2024-04-23T16:28:47.107 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_count": 0, 2024-04-23T16:28:47.107 INFO:teuthology.orchestra.run.smithi103.stdout: "use_gmt_hitset": true, 2024-04-23T16:28:47.107 INFO:teuthology.orchestra.run.smithi103.stdout: "min_read_recency_for_promote": 0, 2024-04-23T16:28:47.107 INFO:teuthology.orchestra.run.smithi103.stdout: "min_write_recency_for_promote": 0, 2024-04-23T16:28:47.107 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T16:28:47.107 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_search_last_n": 0, 2024-04-23T16:28:47.107 INFO:teuthology.orchestra.run.smithi103.stdout: "grade_table": [], 2024-04-23T16:28:47.107 INFO:teuthology.orchestra.run.smithi103.stdout: "stripe_width": 0, 2024-04-23T16:28:47.107 INFO:teuthology.orchestra.run.smithi103.stdout: "expected_num_objects": 0, 2024-04-23T16:28:47.107 INFO:teuthology.orchestra.run.smithi103.stdout: "fast_read": false, 2024-04-23T16:28:47.108 INFO:teuthology.orchestra.run.smithi103.stdout: "options": { 2024-04-23T16:28:47.108 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_max": 32, 2024-04-23T16:28:47.108 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_min": 1 2024-04-23T16:28:47.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.108 INFO:teuthology.orchestra.run.smithi103.stdout: "application_metadata": { 2024-04-23T16:28:47.108 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": {} 2024-04-23T16:28:47.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.108 INFO:teuthology.orchestra.run.smithi103.stdout: "read_balance": { 2024-04-23T16:28:47.108 INFO:teuthology.orchestra.run.smithi103.stdout: "score_type": "Fair distribution", 2024-04-23T16:28:47.108 INFO:teuthology.orchestra.run.smithi103.stdout: "score_acting": 3.2899999618530273, 2024-04-23T16:28:47.108 INFO:teuthology.orchestra.run.smithi103.stdout: "score_stable": 3.2899999618530273, 2024-04-23T16:28:47.108 INFO:teuthology.orchestra.run.smithi103.stdout: "optimal_score": 0.82999998331069946, 2024-04-23T16:28:47.108 INFO:teuthology.orchestra.run.smithi103.stdout: "raw_score_acting": 2.7300000190734863, 2024-04-23T16:28:47.108 INFO:teuthology.orchestra.run.smithi103.stdout: "raw_score_stable": 2.7300000190734863, 2024-04-23T16:28:47.109 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity_weighted": 0.98181778192520142, 2024-04-23T16:28:47.109 INFO:teuthology.orchestra.run.smithi103.stdout: "average_primary_affinity": 0.98000001907348633, 2024-04-23T16:28:47.109 INFO:teuthology.orchestra.run.smithi103.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-04-23T16:28:47.109 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.109 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 4, 2024-04-23T16:28:47.109 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_name": "unique_pool_2", 2024-04-23T16:28:47.109 INFO:teuthology.orchestra.run.smithi103.stdout: "create_time": "2024-04-23T16:27:07.774581+0000", 2024-04-23T16:28:47.109 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-04-23T16:28:47.109 INFO:teuthology.orchestra.run.smithi103.stdout: "flags_names": "hashpspool", 2024-04-23T16:28:47.110 INFO:teuthology.orchestra.run.smithi103.stdout: "type": 1, 2024-04-23T16:28:47.110 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 2, 2024-04-23T16:28:47.110 INFO:teuthology.orchestra.run.smithi103.stdout: "min_size": 2, 2024-04-23T16:28:47.110 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_rule": 0, 2024-04-23T16:28:47.110 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_count": 0, 2024-04-23T16:28:47.110 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_target": 0, 2024-04-23T16:28:47.110 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T16:28:47.110 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T16:28:47.110 INFO:teuthology.orchestra.run.smithi103.stdout: "object_hash": 2, 2024-04-23T16:28:47.110 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_autoscale_mode": "off", 2024-04-23T16:28:47.110 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num": 16, 2024-04-23T16:28:47.110 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_placement_num": 16, 2024-04-23T16:28:47.110 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_placement_num_target": 16, 2024-04-23T16:28:47.111 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_target": 16, 2024-04-23T16:28:47.111 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_pending": 16, 2024-04-23T16:28:47.111 INFO:teuthology.orchestra.run.smithi103.stdout: "last_pg_merge_meta": { 2024-04-23T16:28:47.111 INFO:teuthology.orchestra.run.smithi103.stdout: "source_pgid": "0.0", 2024-04-23T16:28:47.111 INFO:teuthology.orchestra.run.smithi103.stdout: "ready_epoch": 0, 2024-04-23T16:28:47.111 INFO:teuthology.orchestra.run.smithi103.stdout: "last_epoch_started": 0, 2024-04-23T16:28:47.111 INFO:teuthology.orchestra.run.smithi103.stdout: "last_epoch_clean": 0, 2024-04-23T16:28:47.111 INFO:teuthology.orchestra.run.smithi103.stdout: "source_version": "0'0", 2024-04-23T16:28:47.111 INFO:teuthology.orchestra.run.smithi103.stdout: "target_version": "0'0" 2024-04-23T16:28:47.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.111 INFO:teuthology.orchestra.run.smithi103.stdout: "last_change": "281", 2024-04-23T16:28:47.111 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend": "0", 2024-04-23T16:28:47.111 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T16:28:47.111 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T16:28:47.111 INFO:teuthology.orchestra.run.smithi103.stdout: "auid": 0, 2024-04-23T16:28:47.112 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_mode": "selfmanaged", 2024-04-23T16:28:47.112 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_seq": 0, 2024-04-23T16:28:47.112 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_epoch": 0, 2024-04-23T16:28:47.112 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_snaps": [], 2024-04-23T16:28:47.112 INFO:teuthology.orchestra.run.smithi103.stdout: "removed_snaps": "[]", 2024-04-23T16:28:47.112 INFO:teuthology.orchestra.run.smithi103.stdout: "quota_max_bytes": 0, 2024-04-23T16:28:47.112 INFO:teuthology.orchestra.run.smithi103.stdout: "quota_max_objects": 0, 2024-04-23T16:28:47.112 INFO:teuthology.orchestra.run.smithi103.stdout: "tiers": [], 2024-04-23T16:28:47.112 INFO:teuthology.orchestra.run.smithi103.stdout: "tier_of": -1, 2024-04-23T16:28:47.112 INFO:teuthology.orchestra.run.smithi103.stdout: "read_tier": -1, 2024-04-23T16:28:47.113 INFO:teuthology.orchestra.run.smithi103.stdout: "write_tier": -1, 2024-04-23T16:28:47.113 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_mode": "none", 2024-04-23T16:28:47.113 INFO:teuthology.orchestra.run.smithi103.stdout: "target_max_bytes": 0, 2024-04-23T16:28:47.113 INFO:teuthology.orchestra.run.smithi103.stdout: "target_max_objects": 0, 2024-04-23T16:28:47.113 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T16:28:47.113 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T16:28:47.113 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T16:28:47.113 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_min_flush_age": 0, 2024-04-23T16:28:47.113 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_min_evict_age": 0, 2024-04-23T16:28:47.113 INFO:teuthology.orchestra.run.smithi103.stdout: "erasure_code_profile": "", 2024-04-23T16:28:47.113 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_params": { 2024-04-23T16:28:47.113 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "none" 2024-04-23T16:28:47.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.114 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_period": 0, 2024-04-23T16:28:47.114 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_count": 0, 2024-04-23T16:28:47.114 INFO:teuthology.orchestra.run.smithi103.stdout: "use_gmt_hitset": true, 2024-04-23T16:28:47.114 INFO:teuthology.orchestra.run.smithi103.stdout: "min_read_recency_for_promote": 0, 2024-04-23T16:28:47.114 INFO:teuthology.orchestra.run.smithi103.stdout: "min_write_recency_for_promote": 0, 2024-04-23T16:28:47.114 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T16:28:47.114 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_search_last_n": 0, 2024-04-23T16:28:47.114 INFO:teuthology.orchestra.run.smithi103.stdout: "grade_table": [], 2024-04-23T16:28:47.114 INFO:teuthology.orchestra.run.smithi103.stdout: "stripe_width": 0, 2024-04-23T16:28:47.114 INFO:teuthology.orchestra.run.smithi103.stdout: "expected_num_objects": 0, 2024-04-23T16:28:47.114 INFO:teuthology.orchestra.run.smithi103.stdout: "fast_read": false, 2024-04-23T16:28:47.114 INFO:teuthology.orchestra.run.smithi103.stdout: "options": {}, 2024-04-23T16:28:47.114 INFO:teuthology.orchestra.run.smithi103.stdout: "application_metadata": { 2024-04-23T16:28:47.115 INFO:teuthology.orchestra.run.smithi103.stdout: "rados": {} 2024-04-23T16:28:47.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.115 INFO:teuthology.orchestra.run.smithi103.stdout: "read_balance": { 2024-04-23T16:28:47.115 INFO:teuthology.orchestra.run.smithi103.stdout: "score_type": "Fair distribution", 2024-04-23T16:28:47.115 INFO:teuthology.orchestra.run.smithi103.stdout: "score_acting": 3, 2024-04-23T16:28:47.115 INFO:teuthology.orchestra.run.smithi103.stdout: "score_stable": 3, 2024-04-23T16:28:47.115 INFO:teuthology.orchestra.run.smithi103.stdout: "optimal_score": 0.75, 2024-04-23T16:28:47.115 INFO:teuthology.orchestra.run.smithi103.stdout: "raw_score_acting": 2.25, 2024-04-23T16:28:47.115 INFO:teuthology.orchestra.run.smithi103.stdout: "raw_score_stable": 2.25, 2024-04-23T16:28:47.115 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity_weighted": 0.98181778192520142, 2024-04-23T16:28:47.115 INFO:teuthology.orchestra.run.smithi103.stdout: "average_primary_affinity": 0.98000001907348633, 2024-04-23T16:28:47.115 INFO:teuthology.orchestra.run.smithi103.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-04-23T16:28:47.115 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.115 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.116 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:28:47.116 INFO:teuthology.orchestra.run.smithi103.stdout: "osds": [ 2024-04-23T16:28:47.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.116 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 0, 2024-04-23T16:28:47.116 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "4c084047-8f01-4b86-a60c-5592735b5862", 2024-04-23T16:28:47.116 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:47.116 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:47.116 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:47.116 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:47.116 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:47.116 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:47.116 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 8, 2024-04-23T16:28:47.116 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 306, 2024-04-23T16:28:47.116 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:47.116 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:47.117 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:47.117 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.117 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.117 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6801", 2024-04-23T16:28:47.117 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:28:47.117 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.117 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.117 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:47.117 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.117 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.117 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6802", 2024-04-23T16:28:47.118 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:28:47.118 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.118 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.118 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:47.118 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.118 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.118 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6804", 2024-04-23T16:28:47.118 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:28:47.118 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.118 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.119 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:47.119 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.119 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.119 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6803", 2024-04-23T16:28:47.119 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:28:47.119 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.119 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.119 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6801/4274040097", 2024-04-23T16:28:47.119 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6802/4274040097", 2024-04-23T16:28:47.119 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6804/4274040097", 2024-04-23T16:28:47.119 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6803/4274040097", 2024-04-23T16:28:47.120 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:47.120 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:47.120 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:47.120 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.120 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 1, 2024-04-23T16:28:47.120 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "3438f2bd-5fb7-432c-b383-29e18c442f20", 2024-04-23T16:28:47.120 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:47.120 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 0, 2024-04-23T16:28:47.120 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 0, 2024-04-23T16:28:47.120 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:47.120 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 13, 2024-04-23T16:28:47.120 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 232, 2024-04-23T16:28:47.120 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 271, 2024-04-23T16:28:47.121 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 192, 2024-04-23T16:28:47.121 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 233, 2024-04-23T16:28:47.121 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:47.121 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:47.121 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.121 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.121 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6805", 2024-04-23T16:28:47.121 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 859282187 2024-04-23T16:28:47.121 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.121 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.121 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:47.121 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.122 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.122 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6806", 2024-04-23T16:28:47.122 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 859282187 2024-04-23T16:28:47.122 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.122 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.122 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:47.122 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.122 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.122 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6808", 2024-04-23T16:28:47.123 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 859282187 2024-04-23T16:28:47.123 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.123 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.123 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:47.123 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.123 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.123 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6807", 2024-04-23T16:28:47.123 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 859282187 2024-04-23T16:28:47.123 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.123 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.123 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6805/859282187", 2024-04-23T16:28:47.124 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6806/859282187", 2024-04-23T16:28:47.124 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6808/859282187", 2024-04-23T16:28:47.124 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6807/859282187", 2024-04-23T16:28:47.124 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:47.124 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:47.124 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:47.124 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.124 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 2, 2024-04-23T16:28:47.124 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "c17fffb4-8d17-4546-ba24-e5951a2a37fe", 2024-04-23T16:28:47.124 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:47.124 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:47.124 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:47.124 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:47.125 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:47.125 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:47.125 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 21, 2024-04-23T16:28:47.125 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 306, 2024-04-23T16:28:47.125 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:47.125 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:47.125 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:47.125 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.125 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.125 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6809", 2024-04-23T16:28:47.125 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 424240088 2024-04-23T16:28:47.125 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.125 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.126 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:47.126 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.126 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.126 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6810", 2024-04-23T16:28:47.126 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 424240088 2024-04-23T16:28:47.126 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.126 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.126 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:47.126 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.127 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.127 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6812", 2024-04-23T16:28:47.127 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 424240088 2024-04-23T16:28:47.127 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.127 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.127 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:47.127 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.127 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.127 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6811", 2024-04-23T16:28:47.127 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 424240088 2024-04-23T16:28:47.127 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.127 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.128 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6809/424240088", 2024-04-23T16:28:47.128 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6810/424240088", 2024-04-23T16:28:47.128 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6812/424240088", 2024-04-23T16:28:47.128 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6811/424240088", 2024-04-23T16:28:47.128 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:47.128 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:47.128 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:47.128 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.128 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 3, 2024-04-23T16:28:47.128 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "cee91a3d-55dc-4e89-b568-86821632bb1c", 2024-04-23T16:28:47.128 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:47.128 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:47.129 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:47.129 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:47.129 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:47.129 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:47.129 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 27, 2024-04-23T16:28:47.129 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 274, 2024-04-23T16:28:47.129 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:47.129 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:47.129 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:47.129 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.129 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.129 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6813", 2024-04-23T16:28:47.129 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1895183074 2024-04-23T16:28:47.129 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.130 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.130 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:47.130 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.130 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.130 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6814", 2024-04-23T16:28:47.130 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1895183074 2024-04-23T16:28:47.130 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.130 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.130 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:47.130 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.131 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.131 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6816", 2024-04-23T16:28:47.131 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1895183074 2024-04-23T16:28:47.131 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.131 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.131 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:47.131 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.131 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.131 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6815", 2024-04-23T16:28:47.131 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1895183074 2024-04-23T16:28:47.131 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.131 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.132 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6813/1895183074", 2024-04-23T16:28:47.132 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6814/1895183074", 2024-04-23T16:28:47.132 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6816/1895183074", 2024-04-23T16:28:47.132 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6815/1895183074", 2024-04-23T16:28:47.132 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:47.132 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:47.132 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:47.132 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.132 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 4, 2024-04-23T16:28:47.133 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "61d5385b-2c82-42f8-93c3-8f1baf71c65e", 2024-04-23T16:28:47.133 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:47.133 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:47.133 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:47.133 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:47.133 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:47.133 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:47.133 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 33, 2024-04-23T16:28:47.133 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 274, 2024-04-23T16:28:47.133 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:47.133 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:47.134 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:47.134 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.134 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.134 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6800", 2024-04-23T16:28:47.134 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 794945071 2024-04-23T16:28:47.134 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.134 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.134 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:47.134 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.134 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.134 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6801", 2024-04-23T16:28:47.135 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 794945071 2024-04-23T16:28:47.135 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.135 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.135 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:47.135 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.135 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.135 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6803", 2024-04-23T16:28:47.135 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 794945071 2024-04-23T16:28:47.135 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.135 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.135 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:47.136 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.136 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.136 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6802", 2024-04-23T16:28:47.136 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 794945071 2024-04-23T16:28:47.136 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.136 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.136 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6800/794945071", 2024-04-23T16:28:47.136 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6801/794945071", 2024-04-23T16:28:47.136 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6803/794945071", 2024-04-23T16:28:47.136 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6802/794945071", 2024-04-23T16:28:47.136 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:47.136 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:47.136 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:47.137 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.137 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 5, 2024-04-23T16:28:47.137 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "5d492189-c3cd-47ad-a12a-ed4d275bcfe3", 2024-04-23T16:28:47.137 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:47.137 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:47.137 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:47.137 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:47.137 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:47.137 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:47.137 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 38, 2024-04-23T16:28:47.137 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 306, 2024-04-23T16:28:47.137 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:47.138 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:47.138 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:47.138 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.138 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.138 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6804", 2024-04-23T16:28:47.138 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:28:47.138 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.138 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.138 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:47.138 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.139 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.139 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6805", 2024-04-23T16:28:47.139 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:28:47.139 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.139 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.139 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:47.139 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.139 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.139 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6807", 2024-04-23T16:28:47.139 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:28:47.139 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.139 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.140 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:47.140 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.140 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.140 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6806", 2024-04-23T16:28:47.140 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:28:47.140 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.140 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.140 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6804/3547200711", 2024-04-23T16:28:47.140 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6805/3547200711", 2024-04-23T16:28:47.140 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6807/3547200711", 2024-04-23T16:28:47.140 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6806/3547200711", 2024-04-23T16:28:47.141 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:47.141 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:47.141 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:47.141 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.141 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 6, 2024-04-23T16:28:47.141 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "89f2379e-cf14-4e69-8747-7535db81b70b", 2024-04-23T16:28:47.141 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:47.141 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:47.141 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:47.141 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:47.141 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:47.141 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:47.141 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 47, 2024-04-23T16:28:47.142 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 306, 2024-04-23T16:28:47.142 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:47.142 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:47.142 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:47.142 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.142 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.142 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6808", 2024-04-23T16:28:47.142 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:28:47.142 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.142 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.142 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:47.143 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.143 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.143 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6809", 2024-04-23T16:28:47.143 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:28:47.143 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.143 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.143 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:47.143 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.143 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.143 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6811", 2024-04-23T16:28:47.144 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:28:47.144 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.144 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.144 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:47.144 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.144 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.144 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6810", 2024-04-23T16:28:47.144 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:28:47.144 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.144 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.144 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6808/366919758", 2024-04-23T16:28:47.145 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6809/366919758", 2024-04-23T16:28:47.145 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6811/366919758", 2024-04-23T16:28:47.145 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6810/366919758", 2024-04-23T16:28:47.145 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:47.145 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:47.145 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:47.145 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.145 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 7, 2024-04-23T16:28:47.145 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "f27b035a-558f-4528-80ea-2a00d03f6539", 2024-04-23T16:28:47.145 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:47.145 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:47.145 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:47.146 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 0.7818145751953125, 2024-04-23T16:28:47.146 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:47.146 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:47.146 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 56, 2024-04-23T16:28:47.146 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 305, 2024-04-23T16:28:47.146 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:47.146 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:47.146 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:47.146 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.146 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.146 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6812", 2024-04-23T16:28:47.146 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:28:47.146 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.147 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.147 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:47.147 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.147 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.147 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6813", 2024-04-23T16:28:47.147 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:28:47.147 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.147 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.147 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:47.147 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.148 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.148 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6815", 2024-04-23T16:28:47.148 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:28:47.148 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.148 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.148 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:47.148 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.148 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.148 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6814", 2024-04-23T16:28:47.148 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:28:47.148 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.148 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.149 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6812/3719550888", 2024-04-23T16:28:47.149 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6813/3719550888", 2024-04-23T16:28:47.149 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6815/3719550888", 2024-04-23T16:28:47.149 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6814/3719550888", 2024-04-23T16:28:47.149 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:47.149 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:47.149 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:47.149 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.149 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 8, 2024-04-23T16:28:47.149 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed", 2024-04-23T16:28:47.149 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:47.149 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:47.150 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:47.150 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:47.150 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:47.150 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:47.150 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 66, 2024-04-23T16:28:47.150 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 192, 2024-04-23T16:28:47.150 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:47.150 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:47.150 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:47.150 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.150 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.151 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6800", 2024-04-23T16:28:47.151 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:28:47.151 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.152 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.152 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:47.152 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.152 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.152 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6801", 2024-04-23T16:28:47.152 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:28:47.152 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.152 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.152 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:47.152 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.152 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.153 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6803", 2024-04-23T16:28:47.153 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:28:47.153 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.153 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.153 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:47.153 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.153 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.153 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6802", 2024-04-23T16:28:47.153 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:28:47.153 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.153 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.153 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6800/1151408595", 2024-04-23T16:28:47.154 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6801/1151408595", 2024-04-23T16:28:47.154 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6803/1151408595", 2024-04-23T16:28:47.154 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6802/1151408595", 2024-04-23T16:28:47.154 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:47.154 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:47.154 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:47.154 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.154 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 9, 2024-04-23T16:28:47.154 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "124d9b05-11eb-4622-b6a2-c4d5bd60b7e6", 2024-04-23T16:28:47.154 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:47.154 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:47.154 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:47.155 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:47.155 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:47.155 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:47.155 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 73, 2024-04-23T16:28:47.155 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 306, 2024-04-23T16:28:47.155 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:47.155 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:47.155 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:47.155 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.155 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.155 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6804", 2024-04-23T16:28:47.155 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:28:47.155 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.156 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.156 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:47.156 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.156 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.156 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6805", 2024-04-23T16:28:47.156 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:28:47.156 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.157 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.157 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:47.157 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.157 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.157 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6807", 2024-04-23T16:28:47.157 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:28:47.157 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.157 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.157 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:47.157 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.158 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.158 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6806", 2024-04-23T16:28:47.158 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:28:47.158 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.158 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.158 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6804/306501862", 2024-04-23T16:28:47.158 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6805/306501862", 2024-04-23T16:28:47.158 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6807/306501862", 2024-04-23T16:28:47.158 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6806/306501862", 2024-04-23T16:28:47.158 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:47.158 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:47.158 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:47.158 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.159 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 10, 2024-04-23T16:28:47.159 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "3dfb9070-e534-4ea4-a25e-ee1c51297447", 2024-04-23T16:28:47.159 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:47.159 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 0, 2024-04-23T16:28:47.159 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 0, 2024-04-23T16:28:47.159 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:47.159 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:47.159 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:47.159 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 81, 2024-04-23T16:28:47.159 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 0, 2024-04-23T16:28:47.159 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:47.159 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:47.160 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:47.160 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.160 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.160 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6808", 2024-04-23T16:28:47.160 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:28:47.160 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.160 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.160 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:47.160 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.160 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.160 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6809", 2024-04-23T16:28:47.160 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:28:47.161 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.161 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.161 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:47.161 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.161 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.161 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6811", 2024-04-23T16:28:47.161 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:28:47.161 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.161 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.161 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:47.161 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.162 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.162 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6810", 2024-04-23T16:28:47.162 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:28:47.162 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.162 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.162 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6808/1231263138", 2024-04-23T16:28:47.162 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6809/1231263138", 2024-04-23T16:28:47.162 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6811/1231263138", 2024-04-23T16:28:47.162 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6810/1231263138", 2024-04-23T16:28:47.162 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:47.162 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:47.162 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:47.163 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.163 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 11, 2024-04-23T16:28:47.163 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "bc163925-f623-4f63-ab89-35e2d511a730", 2024-04-23T16:28:47.163 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:28:47.163 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:28:47.163 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:28:47.163 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:28:47.163 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:28:47.163 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:28:47.163 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 90, 2024-04-23T16:28:47.163 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 303, 2024-04-23T16:28:47.163 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:28:47.164 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:28:47.164 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:28:47.164 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.164 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.164 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6812", 2024-04-23T16:28:47.164 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:28:47.164 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.164 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.164 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:28:47.164 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.164 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.165 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6813", 2024-04-23T16:28:47.165 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:28:47.165 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.165 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.165 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:28:47.165 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.165 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.165 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6815", 2024-04-23T16:28:47.165 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:28:47.165 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.165 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.165 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:28:47.166 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:28:47.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.166 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:28:47.166 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6814", 2024-04-23T16:28:47.166 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:28:47.166 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.166 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.166 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6812/457992661", 2024-04-23T16:28:47.166 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6813/457992661", 2024-04-23T16:28:47.166 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6815/457992661", 2024-04-23T16:28:47.167 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6814/457992661", 2024-04-23T16:28:47.167 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:28:47.167 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:28:47.167 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:28:47.167 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.167 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.167 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:28:47.167 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_xinfo": [ 2024-04-23T16:28:47.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.168 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 0, 2024-04-23T16:28:47.168 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:47.168 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:47.168 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:47.169 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:47.169 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:47.169 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:28:04.955347+0000", 2024-04-23T16:28:47.169 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:47.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.169 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 1, 2024-04-23T16:28:47.169 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:47.169 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:47.169 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:47.169 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:47.169 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 65536, 2024-04-23T16:28:47.170 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:28:03.447957+0000", 2024-04-23T16:28:47.170 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 232 2024-04-23T16:28:47.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.170 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 2, 2024-04-23T16:28:47.170 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:47.170 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:47.170 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:47.170 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:47.170 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:47.170 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:27:46.330329+0000", 2024-04-23T16:28:47.170 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:47.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.171 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 3, 2024-04-23T16:28:47.171 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:47.171 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:47.171 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:47.171 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:47.171 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:47.171 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:28:00.560710+0000", 2024-04-23T16:28:47.171 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:47.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.171 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 4, 2024-04-23T16:28:47.171 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:47.171 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:47.171 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:47.171 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:47.172 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:47.172 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:27:53.012954+0000", 2024-04-23T16:28:47.172 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:47.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.172 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 5, 2024-04-23T16:28:47.172 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:47.172 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:47.172 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:47.172 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:47.172 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:47.172 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:28:31.926363+0000", 2024-04-23T16:28:47.172 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:47.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.173 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 6, 2024-04-23T16:28:47.173 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:47.173 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:47.173 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:47.173 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:47.173 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:47.173 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:27:52.236962+0000", 2024-04-23T16:28:47.173 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:47.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.173 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 7, 2024-04-23T16:28:47.173 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:47.173 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:47.173 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:47.173 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:47.174 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:47.174 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:28:38.400231+0000", 2024-04-23T16:28:47.174 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:47.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.174 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 8, 2024-04-23T16:28:47.174 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:47.174 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:47.174 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:47.174 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:47.174 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:47.174 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:28:31.427008+0000", 2024-04-23T16:28:47.174 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:47.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.175 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 9, 2024-04-23T16:28:47.175 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:47.175 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:47.175 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:47.175 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:47.175 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:47.175 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:28:36.872460+0000", 2024-04-23T16:28:47.175 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:47.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.175 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 10, 2024-04-23T16:28:47.175 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:47.175 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:47.175 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:47.176 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:47.176 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 65536, 2024-04-23T16:28:47.176 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:28:28.494775+0000", 2024-04-23T16:28:47.176 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:47.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.176 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 11, 2024-04-23T16:28:47.176 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:28:47.176 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:28:47.176 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:28:47.176 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:28:47.176 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:28:47.176 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:28:28.083599+0000", 2024-04-23T16:28:47.176 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:28:47.177 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.177 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:28:47.177 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_upmap": [ 2024-04-23T16:28:47.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.177 INFO:teuthology.orchestra.run.smithi103.stdout: "pgid": "4.6", 2024-04-23T16:28:47.177 INFO:teuthology.orchestra.run.smithi103.stdout: "osds": [ 2024-04-23T16:28:47.177 INFO:teuthology.orchestra.run.smithi103.stdout: 6, 2024-04-23T16:28:47.177 INFO:teuthology.orchestra.run.smithi103.stdout: 9 2024-04-23T16:28:47.177 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.177 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.177 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:28:47.177 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_upmap_items": [], 2024-04-23T16:28:47.177 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_upmap_primaries": [], 2024-04-23T16:28:47.177 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_temp": [ 2024-04-23T16:28:47.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:28:47.177 INFO:teuthology.orchestra.run.smithi103.stdout: "pgid": "4.6", 2024-04-23T16:28:47.178 INFO:teuthology.orchestra.run.smithi103.stdout: "osds": [ 2024-04-23T16:28:47.178 INFO:teuthology.orchestra.run.smithi103.stdout: 2, 2024-04-23T16:28:47.178 INFO:teuthology.orchestra.run.smithi103.stdout: 9 2024-04-23T16:28:47.178 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:28:47.178 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.178 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:28:47.178 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_temp": [], 2024-04-23T16:28:47.178 INFO:teuthology.orchestra.run.smithi103.stdout: "blocklist": { 2024-04-23T16:28:47.178 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:6800/3535470771": "2024-04-24T16:09:14.324448+0000", 2024-04-23T16:28:47.178 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/1188058303": "2024-04-24T16:09:14.324448+0000", 2024-04-23T16:28:47.178 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/1510553721": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:28:47.178 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/281182801": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:28:47.178 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:6800/2049124518": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:28:47.178 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/466241963": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:28:47.179 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/2726838698": "2024-04-24T16:09:14.324448+0000", 2024-04-23T16:28:47.179 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/596230573": "2024-04-24T16:09:14.324448+0000" 2024-04-23T16:28:47.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.179 INFO:teuthology.orchestra.run.smithi103.stdout: "range_blocklist": {}, 2024-04-23T16:28:47.179 INFO:teuthology.orchestra.run.smithi103.stdout: "erasure_code_profiles": { 2024-04-23T16:28:47.179 INFO:teuthology.orchestra.run.smithi103.stdout: "default": { 2024-04-23T16:28:47.179 INFO:teuthology.orchestra.run.smithi103.stdout: "crush-failure-domain": "osd", 2024-04-23T16:28:47.179 INFO:teuthology.orchestra.run.smithi103.stdout: "k": "2", 2024-04-23T16:28:47.179 INFO:teuthology.orchestra.run.smithi103.stdout: "m": "1", 2024-04-23T16:28:47.179 INFO:teuthology.orchestra.run.smithi103.stdout: "plugin": "jerasure", 2024-04-23T16:28:47.179 INFO:teuthology.orchestra.run.smithi103.stdout: "technique": "reed_sol_van" 2024-04-23T16:28:47.179 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:28:47.179 INFO:teuthology.orchestra.run.smithi103.stdout: "removed_snaps_queue": [], 2024-04-23T16:28:47.179 INFO:teuthology.orchestra.run.smithi103.stdout: "new_removed_snaps": [], 2024-04-23T16:28:47.180 INFO:teuthology.orchestra.run.smithi103.stdout: "new_purged_snaps": [], 2024-04-23T16:28:47.180 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_node_flags": {}, 2024-04-23T16:28:47.180 INFO:teuthology.orchestra.run.smithi103.stdout: "device_class_flags": {}, 2024-04-23T16:28:47.180 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_mode": { 2024-04-23T16:28:47.180 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_mode_enabled": false, 2024-04-23T16:28:47.180 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_bucket_count": 0, 2024-04-23T16:28:47.180 INFO:teuthology.orchestra.run.smithi103.stdout: "degraded_stretch_mode": 0, 2024-04-23T16:28:47.180 INFO:teuthology.orchestra.run.smithi103.stdout: "recovering_stretch_mode": 0, 2024-04-23T16:28:47.180 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_mode_bucket": 0 2024-04-23T16:28:47.180 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:28:47.180 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:28:47.181 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:46 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3125794274' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:28:47.391 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:28:47.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:47.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:28:47.499 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:28:47.563 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:47.608 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:28:47.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:47.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:28:47.714 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:28:47.722 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:47 smithi103 ceph-mon[39629]: pgmap v891: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.8 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 82 MiB/s wr, 1.32k op/s; 4776/320202 objects misplaced (1.492%); 1.3 MiB/s, 20 objects/s recovering 2024-04-23T16:28:47.722 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.722 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.722 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.722 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.723 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.723 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.723 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.723 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.723 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.723 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.723 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.723 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:47 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4189086854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-23T16:28:47.759 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:28:47.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:47.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:28:47.830 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 90 10 163446 163436 113.48 68.375 0.0450308 0.00881014 2024-04-23T16:28:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:47 smithi148 ceph-mon[44520]: pgmap v891: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.8 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 82 MiB/s wr, 1.32k op/s; 4776/320202 objects misplaced (1.492%); 1.3 MiB/s, 20 objects/s recovering 2024-04-23T16:28:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:47 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4189086854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-23T16:28:47.892 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:28:47.918 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:47.936 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:28:47.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:47.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:28:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:47 smithi188 ceph-mon[44101]: pgmap v891: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.8 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 82 MiB/s wr, 1.32k op/s; 4776/320202 objects misplaced (1.492%); 1.3 MiB/s, 20 objects/s recovering 2024-04-23T16:28:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:47.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:47 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4189086854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-23T16:28:48.043 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:28:48.070 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:48.089 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:28:48.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:48.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:28:48.193 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:28:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:48.238 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:28:48.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:48.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:28:48.341 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:28:48.371 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:48.392 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:28:48.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:48.395 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:28:48.496 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:28:48.523 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:48.542 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:28:48.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:48.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:28:48.646 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:28:48.672 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:48.692 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:28:48.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:48.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:28:48.796 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:28:48.828 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:48.830 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Total time run: 90.0161 2024-04-23T16:28:48.831 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Total writes made: 163446 2024-04-23T16:28:48.831 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Write size: 65536 2024-04-23T16:28:48.831 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Object size: 65536 2024-04-23T16:28:48.831 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Bandwidth (MB/sec): 113.484 2024-04-23T16:28:48.831 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev Bandwidth: 64.3939 2024-04-23T16:28:48.831 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max bandwidth (MB/sec): 380.312 2024-04-23T16:28:48.831 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min bandwidth (MB/sec): 0 2024-04-23T16:28:48.831 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Average IOPS: 1815 2024-04-23T16:28:48.831 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev IOPS: 1030.3 2024-04-23T16:28:48.831 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max IOPS: 6085 2024-04-23T16:28:48.832 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min IOPS: 0 2024-04-23T16:28:48.832 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Average Latency(s): 0.00881105 2024-04-23T16:28:48.832 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev Latency(s): 0.0264207 2024-04-23T16:28:48.832 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max latency(s): 2.61652 2024-04-23T16:28:48.832 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min latency(s): 0.00105502 2024-04-23T16:28:48.838 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Cleaning up (deleting benchmark objects) 2024-04-23T16:28:48.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:48.857 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:28:48.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:48.862 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:28:48.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:48.962 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:28:49.003 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:49.025 INFO:tasks.thrashosds.thrasher:j is {'epoch': 315, 'fsid': '8e3594f2-018b-11ef-bc93-c7b262605968', 'created': '2024-04-23T16:08:13.477590+0000', 'modified': '2024-04-23T16:28:44.038128+0000', 'last_up_change': '2024-04-23T16:27:00.729195+0000', 'last_in_change': '2024-04-23T16:24:50.783418+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T16:11:22.416777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '303', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '301', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.2899999618530273, 'score_stable': 3.2899999618530273, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 2.7300000190734863, 'raw_score_stable': 2.7300000190734863, 'primary_affinity_weighted': 0.9818177819252014, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-04-23T16:27:07.774581+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '281', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9818177819252014, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}], 'osds': [{'osd': 0, 'uuid': '4c084047-8f01-4b86-a60c-5592735b5862', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6801', 'nonce': 4274040097}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6802', 'nonce': 4274040097}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6804', 'nonce': 4274040097}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6803', 'nonce': 4274040097}]}, 'public_addr': '172.21.15.103:6801/4274040097', 'cluster_addr': '172.21.15.103:6802/4274040097', 'heartbeat_back_addr': '172.21.15.103:6804/4274040097', 'heartbeat_front_addr': '172.21.15.103:6803/4274040097', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '3438f2bd-5fb7-432c-b383-29e18c442f20', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 232, 'up_from': 271, 'up_thru': 192, 'down_at': 233, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6805', 'nonce': 859282187}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6806', 'nonce': 859282187}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6808', 'nonce': 859282187}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6807', 'nonce': 859282187}]}, 'public_addr': '172.21.15.103:6805/859282187', 'cluster_addr': '172.21.15.103:6806/859282187', 'heartbeat_back_addr': '172.21.15.103:6808/859282187', 'heartbeat_front_addr': '172.21.15.103:6807/859282187', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c17fffb4-8d17-4546-ba24-e5951a2a37fe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6809', 'nonce': 424240088}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6810', 'nonce': 424240088}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6812', 'nonce': 424240088}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6811', 'nonce': 424240088}]}, 'public_addr': '172.21.15.103:6809/424240088', 'cluster_addr': '172.21.15.103:6810/424240088', 'heartbeat_back_addr': '172.21.15.103:6812/424240088', 'heartbeat_front_addr': '172.21.15.103:6811/424240088', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'cee91a3d-55dc-4e89-b568-86821632bb1c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6813', 'nonce': 1895183074}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6814', 'nonce': 1895183074}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6816', 'nonce': 1895183074}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6815', 'nonce': 1895183074}]}, 'public_addr': '172.21.15.103:6813/1895183074', 'cluster_addr': '172.21.15.103:6814/1895183074', 'heartbeat_back_addr': '172.21.15.103:6816/1895183074', 'heartbeat_front_addr': '172.21.15.103:6815/1895183074', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '61d5385b-2c82-42f8-93c3-8f1baf71c65e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6800', 'nonce': 794945071}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6801', 'nonce': 794945071}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6803', 'nonce': 794945071}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6802', 'nonce': 794945071}]}, 'public_addr': '172.21.15.148:6800/794945071', 'cluster_addr': '172.21.15.148:6801/794945071', 'heartbeat_back_addr': '172.21.15.148:6803/794945071', 'heartbeat_front_addr': '172.21.15.148:6802/794945071', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '5d492189-c3cd-47ad-a12a-ed4d275bcfe3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6804', 'nonce': 3547200711}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6805', 'nonce': 3547200711}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6807', 'nonce': 3547200711}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6806', 'nonce': 3547200711}]}, 'public_addr': '172.21.15.148:6804/3547200711', 'cluster_addr': '172.21.15.148:6805/3547200711', 'heartbeat_back_addr': '172.21.15.148:6807/3547200711', 'heartbeat_front_addr': '172.21.15.148:6806/3547200711', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '89f2379e-cf14-4e69-8747-7535db81b70b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6808', 'nonce': 366919758}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6809', 'nonce': 366919758}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6811', 'nonce': 366919758}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6810', 'nonce': 366919758}]}, 'public_addr': '172.21.15.148:6808/366919758', 'cluster_addr': '172.21.15.148:6809/366919758', 'heartbeat_back_addr': '172.21.15.148:6811/366919758', 'heartbeat_front_addr': '172.21.15.148:6810/366919758', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f27b035a-558f-4528-80ea-2a00d03f6539', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7818145751953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 305, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6812', 'nonce': 3719550888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6813', 'nonce': 3719550888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6815', 'nonce': 3719550888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6814', 'nonce': 3719550888}]}, 'public_addr': '172.21.15.148:6812/3719550888', 'cluster_addr': '172.21.15.148:6813/3719550888', 'heartbeat_back_addr': '172.21.15.148:6815/3719550888', 'heartbeat_front_addr': '172.21.15.148:6814/3719550888', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 192, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6800', 'nonce': 1151408595}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6801', 'nonce': 1151408595}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6803', 'nonce': 1151408595}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6802', 'nonce': 1151408595}]}, 'public_addr': '172.21.15.188:6800/1151408595', 'cluster_addr': '172.21.15.188:6801/1151408595', 'heartbeat_back_addr': '172.21.15.188:6803/1151408595', 'heartbeat_front_addr': '172.21.15.188:6802/1151408595', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '124d9b05-11eb-4622-b6a2-c4d5bd60b7e6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6804', 'nonce': 306501862}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6805', 'nonce': 306501862}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6807', 'nonce': 306501862}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6806', 'nonce': 306501862}]}, 'public_addr': '172.21.15.188:6804/306501862', 'cluster_addr': '172.21.15.188:6805/306501862', 'heartbeat_back_addr': '172.21.15.188:6807/306501862', 'heartbeat_front_addr': '172.21.15.188:6806/306501862', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '3dfb9070-e534-4ea4-a25e-ee1c51297447', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6808', 'nonce': 1231263138}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6809', 'nonce': 1231263138}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6811', 'nonce': 1231263138}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6810', 'nonce': 1231263138}]}, 'public_addr': '172.21.15.188:6808/1231263138', 'cluster_addr': '172.21.15.188:6809/1231263138', 'heartbeat_back_addr': '172.21.15.188:6811/1231263138', 'heartbeat_front_addr': '172.21.15.188:6810/1231263138', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'bc163925-f623-4f63-ab89-35e2d511a730', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 303, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6812', 'nonce': 457992661}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6813', 'nonce': 457992661}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6815', 'nonce': 457992661}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6814', 'nonce': 457992661}]}, 'public_addr': '172.21.15.188:6812/457992661', 'cluster_addr': '172.21.15.188:6813/457992661', 'heartbeat_back_addr': '172.21.15.188:6815/457992661', 'heartbeat_front_addr': '172.21.15.188:6814/457992661', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:28:04.955347+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-23T16:28:03.447957+0000', 'dead_epoch': 232}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:27:46.330329+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:28:00.560710+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:27:53.012954+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:28:31.926363+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:27:52.236962+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:28:38.400231+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:28:31.427008+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:28:36.872460+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-23T16:28:28.494775+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:28:28.083599+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.6', 'osds': [6, 9]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '4.6', 'osds': [2, 9]}], 'primary_temp': [], 'blocklist': {'172.21.15.103:6800/3535470771': '2024-04-24T16:09:14.324448+0000', '172.21.15.103:0/1188058303': '2024-04-24T16:09:14.324448+0000', '172.21.15.103:0/1510553721': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:0/281182801': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:6800/2049124518': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:0/466241963': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:0/2726838698': '2024-04-24T16:09:14.324448+0000', '172.21.15.103:0/596230573': '2024-04-24T16:09:14.324448+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-23T16:28:49.025 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.6 2024-04-23T16:28:49.025 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd rm-pg-upmap 4.6 2024-04-23T16:28:49.026 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:28:49.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:49.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:28:49.131 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:28:49.160 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:28:49.161 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:49.184 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:28:49.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:49.188 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:28:49.289 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:28:49.318 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:49.339 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:28:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:49.342 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:28:49.442 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:28:49.480 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:49.511 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:28:49.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:49.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:28:49.617 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:28:49.665 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:49.685 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:28:49.686 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:28:49.694 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:28:49.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:49.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:28:49.804 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:28:49.835 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:49.859 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:28:49.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:49.862 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:28:49.963 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:28:49.995 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:49 smithi103 ceph-mon[39629]: pgmap v892: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.8 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 53 MiB/s wr, 855 op/s; 4776/320202 objects misplaced (1.492%); 1.1 MiB/s, 17 objects/s recovering 2024-04-23T16:28:49.995 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:50.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:49 smithi148 ceph-mon[44520]: pgmap v892: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.8 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 53 MiB/s wr, 855 op/s; 4776/320202 objects misplaced (1.492%); 1.1 MiB/s, 17 objects/s recovering 2024-04-23T16:28:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:49 smithi188 ceph-mon[44101]: pgmap v892: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.8 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 53 MiB/s wr, 855 op/s; 4776/320202 objects misplaced (1.492%); 1.1 MiB/s, 17 objects/s recovering 2024-04-23T16:28:50.832 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-23T16:28:50.850 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:28:50.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:50.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:28:50.881 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:28:50.942 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:28:50.956 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:28:51.006 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:51.220 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:28:51.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:51.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:28:51.333 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:28:51.365 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:51.390 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:28:51.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:51.398 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:28:51.499 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:28:51.546 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:51.580 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:28:51.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:51.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:28:51.691 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:28:51.729 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:51.854 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:51 smithi103 ceph-mon[39629]: pgmap v893: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 102 B/s rd, 62 MiB/s wr, 1.02k op/s; 4572/325612 objects misplaced (1.404%); 2.3 MiB/s, 36 objects/s recovering 2024-04-23T16:28:51.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:51 smithi148 ceph-mon[44520]: pgmap v893: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 102 B/s rd, 62 MiB/s wr, 1.02k op/s; 4572/325612 objects misplaced (1.404%); 2.3 MiB/s, 36 objects/s recovering 2024-04-23T16:28:52.123 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:28:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:51 smithi188 ceph-mon[44101]: pgmap v893: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 102 B/s rd, 62 MiB/s wr, 1.02k op/s; 4572/325612 objects misplaced (1.404%); 2.3 MiB/s, 36 objects/s recovering 2024-04-23T16:28:52.589 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:28:52.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:52.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:28:52.694 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:28:52.742 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:52.774 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:28:52.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:52.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:28:52.880 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:28:52.912 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:52.936 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:28:52.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:52.941 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:28:53.042 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:28:53.072 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:53.104 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:28:53.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:53.108 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:28:53.209 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:28:53.254 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:28:53.261 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:53.299 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:28:53.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:53.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:28:53.403 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:28:53.432 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:53.457 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:28:53.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:53.460 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:28:53.561 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:28:53.591 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:53.613 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:28:53.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:53.616 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:28:53.717 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:28:53.776 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:53.817 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:28:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:53.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:28:53.924 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:28:53.956 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:54.023 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:28:54.025 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:53 smithi103 ceph-mon[39629]: pgmap v894: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 102 B/s rd, 55 MiB/s wr, 993 op/s; 4572/324724 objects misplaced (1.408%); 2.3 MiB/s, 36 objects/s recovering 2024-04-23T16:28:54.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:54.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:28:54.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:53 smithi148 ceph-mon[44520]: pgmap v894: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 102 B/s rd, 55 MiB/s wr, 993 op/s; 4572/324724 objects misplaced (1.408%); 2.3 MiB/s, 36 objects/s recovering 2024-04-23T16:28:54.127 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:28:54.160 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:53 smithi188 ceph-mon[44101]: pgmap v894: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 102 B/s rd, 55 MiB/s wr, 993 op/s; 4572/324724 objects misplaced (1.408%); 2.3 MiB/s, 36 objects/s recovering 2024-04-23T16:28:54.812 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:28:54.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:54.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:28:54.921 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:28:54.955 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:54.997 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:28:55.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:55.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:28:55.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:55 smithi103 ceph-mon[39629]: pgmap v895: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.7 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 96 B/s rd, 36 MiB/s wr, 1.01k op/s; 4411/317718 objects misplaced (1.388%); 2.2 MiB/s, 34 objects/s recovering 2024-04-23T16:28:55.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:55 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/913458470' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.6"}]: dispatch 2024-04-23T16:28:55.066 INFO:teuthology.orchestra.run.smithi103.stderr:clear 4.6 pg_upmap mapping 2024-04-23T16:28:55.102 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:28:55.150 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:55.309 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:28:55.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:55.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:28:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:55 smithi148 ceph-mon[44520]: pgmap v895: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.7 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 96 B/s rd, 36 MiB/s wr, 1.01k op/s; 4411/317718 objects misplaced (1.388%); 2.2 MiB/s, 34 objects/s recovering 2024-04-23T16:28:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:55 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/913458470' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.6"}]: dispatch 2024-04-23T16:28:55.413 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:28:55.440 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:55.462 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:28:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:55 smithi188 ceph-mon[44101]: pgmap v895: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.7 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 96 B/s rd, 36 MiB/s wr, 1.01k op/s; 4411/317718 objects misplaced (1.388%); 2.2 MiB/s, 34 objects/s recovering 2024-04-23T16:28:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:55 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/913458470' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.6"}]: dispatch 2024-04-23T16:28:55.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:55.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:28:55.584 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:28:55.611 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:55.629 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:28:55.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:55.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:28:55.733 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:28:55.760 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:55.779 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:28:55.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:55.782 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:28:55.883 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:28:55.909 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:55.927 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:28:55.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:55.930 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:28:56.031 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:28:56.058 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:56.069 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:28:56.079 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:28:56.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:56.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:28:56.182 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:28:56.208 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:56.226 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:28:56.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:56.229 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:28:56.330 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:28:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:56 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/913458470' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.6"}]': finished 2024-04-23T16:28:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:56 smithi103 ceph-mon[39629]: osdmap e316: 12 total, 12 up, 10 in 2024-04-23T16:28:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:56 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2224784814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:28:56.355 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:56 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/913458470' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.6"}]': finished 2024-04-23T16:28:56.355 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:56 smithi188 ceph-mon[44101]: osdmap e316: 12 total, 12 up, 10 in 2024-04-23T16:28:56.355 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:56 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2224784814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:28:56.357 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:56.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:56 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/913458470' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.6"}]': finished 2024-04-23T16:28:56.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:56 smithi148 ceph-mon[44520]: osdmap e316: 12 total, 12 up, 10 in 2024-04-23T16:28:56.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:56 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2224784814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:28:56.375 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:28:56.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:56.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:28:56.479 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:28:56.505 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:56.523 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:28:56.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:56.526 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:28:56.570 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:28:56.627 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:28:56.655 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:56.674 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:28:56.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:56.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:28:56.778 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:28:56.805 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:56.824 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:28:56.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:56.827 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:28:56.928 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:28:56.955 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:57.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:57 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:28:57.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:57 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2224784814' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:28:57.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:57 smithi103 ceph-mon[39629]: osdmap e317: 12 total, 12 up, 10 in 2024-04-23T16:28:57.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:57 smithi103 ceph-mon[39629]: pgmap v898: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 127 B/s rd, 26 MiB/s wr, 1.20k op/s; 4411/314356 objects misplaced (1.403%); 2.9 MiB/s, 45 objects/s recovering 2024-04-23T16:28:57.086 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:28:57.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:57.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:28:57.192 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:28:57.232 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:57 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:28:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:57 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2224784814' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:28:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:57 smithi148 ceph-mon[44520]: osdmap e317: 12 total, 12 up, 10 in 2024-04-23T16:28:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:57 smithi148 ceph-mon[44520]: pgmap v898: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 127 B/s rd, 26 MiB/s wr, 1.20k op/s; 4411/314356 objects misplaced (1.403%); 2.9 MiB/s, 45 objects/s recovering 2024-04-23T16:28:57.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:57 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:28:57.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:57 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2224784814' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:28:57.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:57 smithi188 ceph-mon[44101]: osdmap e317: 12 total, 12 up, 10 in 2024-04-23T16:28:57.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:57 smithi188 ceph-mon[44101]: pgmap v898: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 127 B/s rd, 26 MiB/s wr, 1.20k op/s; 4411/314356 objects misplaced (1.403%); 2.9 MiB/s, 45 objects/s recovering 2024-04-23T16:28:58.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:58 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:28:58.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:58 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:28:58.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:58 smithi103 ceph-mon[39629]: osdmap e318: 12 total, 12 up, 10 in 2024-04-23T16:28:58.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:58 smithi103 ceph-mon[39629]: 1.4 scrub starts 2024-04-23T16:28:58.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:58 smithi103 ceph-mon[39629]: 1.4 scrub ok 2024-04-23T16:28:58.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:58 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:28:58.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:58 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:28:58.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:58 smithi148 ceph-mon[44520]: osdmap e318: 12 total, 12 up, 10 in 2024-04-23T16:28:58.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:58 smithi148 ceph-mon[44520]: 1.4 scrub starts 2024-04-23T16:28:58.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:58 smithi148 ceph-mon[44520]: 1.4 scrub ok 2024-04-23T16:28:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:58 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:28:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:58 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:28:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:58 smithi188 ceph-mon[44101]: osdmap e318: 12 total, 12 up, 10 in 2024-04-23T16:28:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:58 smithi188 ceph-mon[44101]: 1.4 scrub starts 2024-04-23T16:28:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:58 smithi188 ceph-mon[44101]: 1.4 scrub ok 2024-04-23T16:28:58.695 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:28:58.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:58.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:28:58.808 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:28:58.860 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:58.900 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:28:58.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:58.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:28:59.006 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:28:59.040 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:59.071 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:28:59.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:59.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:28:59.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:59 smithi148 ceph-mon[44520]: 1.5 scrub starts 2024-04-23T16:28:59.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:59 smithi148 ceph-mon[44520]: 1.5 scrub ok 2024-04-23T16:28:59.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:59 smithi148 ceph-mon[44520]: 1.7 scrub starts 2024-04-23T16:28:59.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:59 smithi148 ceph-mon[44520]: 1.7 scrub ok 2024-04-23T16:28:59.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:59 smithi148 ceph-mon[44520]: 4.9 scrub starts 2024-04-23T16:28:59.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:59 smithi148 ceph-mon[44520]: pgmap v900: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 863 op/s; 4411/314356 objects misplaced (1.403%); 1.7 MiB/s, 26 objects/s recovering 2024-04-23T16:28:59.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:28:59 smithi148 ceph-mon[44520]: osdmap e319: 12 total, 12 up, 10 in 2024-04-23T16:28:59.179 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:28:59.247 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:59.296 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:28:59.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:59.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:28:59.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:59 smithi103 ceph-mon[39629]: 1.5 scrub starts 2024-04-23T16:28:59.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:59 smithi103 ceph-mon[39629]: 1.5 scrub ok 2024-04-23T16:28:59.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:59 smithi103 ceph-mon[39629]: 1.7 scrub starts 2024-04-23T16:28:59.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:59 smithi103 ceph-mon[39629]: 1.7 scrub ok 2024-04-23T16:28:59.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:59 smithi103 ceph-mon[39629]: 4.9 scrub starts 2024-04-23T16:28:59.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:59 smithi103 ceph-mon[39629]: pgmap v900: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 863 op/s; 4411/314356 objects misplaced (1.403%); 1.7 MiB/s, 26 objects/s recovering 2024-04-23T16:28:59.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:28:59 smithi103 ceph-mon[39629]: osdmap e319: 12 total, 12 up, 10 in 2024-04-23T16:28:59.405 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:28:59.440 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:59.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:59 smithi188 ceph-mon[44101]: 1.5 scrub starts 2024-04-23T16:28:59.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:59 smithi188 ceph-mon[44101]: 1.5 scrub ok 2024-04-23T16:28:59.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:59 smithi188 ceph-mon[44101]: 1.7 scrub starts 2024-04-23T16:28:59.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:59 smithi188 ceph-mon[44101]: 1.7 scrub ok 2024-04-23T16:28:59.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:59 smithi188 ceph-mon[44101]: 4.9 scrub starts 2024-04-23T16:28:59.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:59 smithi188 ceph-mon[44101]: pgmap v900: 27 pgs: 1 active+recovering+remapped, 26 active+clean; 9.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 863 op/s; 4411/314356 objects misplaced (1.403%); 1.7 MiB/s, 26 objects/s recovering 2024-04-23T16:28:59.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:28:59 smithi188 ceph-mon[44101]: osdmap e319: 12 total, 12 up, 10 in 2024-04-23T16:28:59.465 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:28:59.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:59.467 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:28:59.568 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:28:59.601 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:28:59.621 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:59.654 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:28:59.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:59.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:28:59.759 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:28:59.801 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:28:59.826 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:28:59.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:28:59.829 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:28:59.930 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:28:59.994 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:00.032 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:29:00.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:00.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:29:00.058 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:29:00.094 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:29:00.124 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:29:00.141 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:29:00.149 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:29:00.149 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:29:00.165 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:29:00.197 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:29:00.199 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:00.235 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:29:00.269 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:29:00.304 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:29:00.343 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:29:00.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:00 smithi103 ceph-mon[39629]: 4.0 scrub starts 2024-04-23T16:29:00.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:00 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:29:00.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:00 smithi103 ceph-mon[39629]: 4.d scrub starts 2024-04-23T16:29:00.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:00 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:29:00.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:00 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:29:00.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:00 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:29:00.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:00 smithi103 ceph-mon[39629]: 4.3 scrub starts 2024-04-23T16:29:00.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:00 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:29:00.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:00 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:29:00.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:00 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/14305642' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:29:00.371 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:29:00.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:00 smithi148 ceph-mon[44520]: 4.0 scrub starts 2024-04-23T16:29:00.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:00 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:29:00.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:00 smithi148 ceph-mon[44520]: 4.d scrub starts 2024-04-23T16:29:00.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:00 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:29:00.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:00 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:29:00.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:00 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:29:00.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:00 smithi148 ceph-mon[44520]: 4.3 scrub starts 2024-04-23T16:29:00.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:00 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:29:00.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:00 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:29:00.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:00 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/14305642' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:29:00.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:00.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:29:00.384 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:29:00.413 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:29:00.438 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:29:00.452 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:29:00.452 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:29:00.452 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:29:00.452 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:29:00.452 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:29:00.452 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:29:00.452 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:29:00.452 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:29:00.453 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:29:00.453 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:29:00.453 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:29:00.453 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:29:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:00 smithi188 ceph-mon[44101]: 4.0 scrub starts 2024-04-23T16:29:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:00 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:29:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:00 smithi188 ceph-mon[44101]: 4.d scrub starts 2024-04-23T16:29:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:00 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:29:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:00 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:29:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:00 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:29:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:00 smithi188 ceph-mon[44101]: 4.3 scrub starts 2024-04-23T16:29:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:00 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:29:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:00 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:29:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:00 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/14305642' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:29:00.481 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:29:00.539 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:00.571 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:29:00.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:00.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:29:00.678 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:29:00.711 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:01.027 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10, 1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1] 2024-04-23T16:29:01.028 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:29:01.028 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-23T16:29:01.028 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 1.000000 2024-04-23T16:29:01.028 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 6 1 2024-04-23T16:29:01.066 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:29:01.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:01.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:29:01.118 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-23T16:29:01.142 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:29:01.171 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:29:01.241 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:01.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:01 smithi103 ceph-mon[39629]: 4.b scrub starts 2024-04-23T16:29:01.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:01 smithi103 ceph-mon[39629]: 4.9 scrub ok 2024-04-23T16:29:01.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:01 smithi103 ceph-mon[39629]: 1.8 scrub starts 2024-04-23T16:29:01.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:01 smithi103 ceph-mon[39629]: 1.8 scrub ok 2024-04-23T16:29:01.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:01 smithi103 ceph-mon[39629]: 1.9 scrub starts 2024-04-23T16:29:01.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:01 smithi103 ceph-mon[39629]: 1.9 scrub ok 2024-04-23T16:29:01.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:01 smithi103 ceph-mon[39629]: 4.0 scrub ok 2024-04-23T16:29:01.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:01 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T16:29:01.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:01 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T16:29:01.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:01 smithi103 ceph-mon[39629]: pgmap v902: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 9.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 803 op/s; 278 KiB/s, 6 objects/s recovering 2024-04-23T16:29:01.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:01 smithi148 ceph-mon[44520]: 4.b scrub starts 2024-04-23T16:29:01.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:01 smithi148 ceph-mon[44520]: 4.9 scrub ok 2024-04-23T16:29:01.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:01 smithi148 ceph-mon[44520]: 1.8 scrub starts 2024-04-23T16:29:01.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:01 smithi148 ceph-mon[44520]: 1.8 scrub ok 2024-04-23T16:29:01.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:01 smithi148 ceph-mon[44520]: 1.9 scrub starts 2024-04-23T16:29:01.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:01 smithi148 ceph-mon[44520]: 1.9 scrub ok 2024-04-23T16:29:01.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:01 smithi148 ceph-mon[44520]: 4.0 scrub ok 2024-04-23T16:29:01.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:01 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T16:29:01.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:01 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T16:29:01.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:01 smithi148 ceph-mon[44520]: pgmap v902: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 9.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 803 op/s; 278 KiB/s, 6 objects/s recovering 2024-04-23T16:29:01.396 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:29:01.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:01.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:29:01.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:01 smithi188 ceph-mon[44101]: 4.b scrub starts 2024-04-23T16:29:01.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:01 smithi188 ceph-mon[44101]: 4.9 scrub ok 2024-04-23T16:29:01.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:01 smithi188 ceph-mon[44101]: 1.8 scrub starts 2024-04-23T16:29:01.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:01 smithi188 ceph-mon[44101]: 1.8 scrub ok 2024-04-23T16:29:01.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:01 smithi188 ceph-mon[44101]: 1.9 scrub starts 2024-04-23T16:29:01.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:01 smithi188 ceph-mon[44101]: 1.9 scrub ok 2024-04-23T16:29:01.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:01 smithi188 ceph-mon[44101]: 4.0 scrub ok 2024-04-23T16:29:01.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:01 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T16:29:01.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:01 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T16:29:01.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:01.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:01 smithi188 ceph-mon[44101]: pgmap v902: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 9.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 803 op/s; 278 KiB/s, 6 objects/s recovering 2024-04-23T16:29:01.512 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:29:01.548 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:02.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:02.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:02.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:02.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:02.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:02 smithi148 ceph-mon[44520]: 4.d scrub ok 2024-04-23T16:29:02.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:02 smithi148 ceph-mon[44520]: 4.3 scrub ok 2024-04-23T16:29:02.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:02 smithi148 ceph-mon[44520]: 4.e scrub starts 2024-04-23T16:29:02.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:02 smithi148 ceph-mon[44520]: 1.a scrub starts 2024-04-23T16:29:02.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:02 smithi148 ceph-mon[44520]: 1.a scrub ok 2024-04-23T16:29:02.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:02.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:02.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:02.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:02.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:02 smithi188 ceph-mon[44101]: 4.d scrub ok 2024-04-23T16:29:02.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:02 smithi188 ceph-mon[44101]: 4.3 scrub ok 2024-04-23T16:29:02.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:02 smithi188 ceph-mon[44101]: 4.e scrub starts 2024-04-23T16:29:02.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:02 smithi188 ceph-mon[44101]: 1.a scrub starts 2024-04-23T16:29:02.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:02 smithi188 ceph-mon[44101]: 1.a scrub ok 2024-04-23T16:29:02.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:02.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:02.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:02.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:02.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:02 smithi103 ceph-mon[39629]: 4.d scrub ok 2024-04-23T16:29:02.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:02 smithi103 ceph-mon[39629]: 4.3 scrub ok 2024-04-23T16:29:02.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:02 smithi103 ceph-mon[39629]: 4.e scrub starts 2024-04-23T16:29:02.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:02 smithi103 ceph-mon[39629]: 1.a scrub starts 2024-04-23T16:29:02.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:02 smithi103 ceph-mon[39629]: 1.a scrub ok 2024-04-23T16:29:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:03 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:29:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:03 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:29:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:03 smithi188 ceph-mon[44101]: 4.b scrub ok 2024-04-23T16:29:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:03 smithi188 ceph-mon[44101]: osdmap e320: 12 total, 12 up, 10 in 2024-04-23T16:29:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:03 smithi188 ceph-mon[44101]: 4.a scrub starts 2024-04-23T16:29:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:03 smithi188 ceph-mon[44101]: pgmap v904: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 9.3 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 671 op/s; 175 KiB/s, 5 objects/s recovering 2024-04-23T16:29:03.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:03 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:29:03.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:03 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:29:03.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:03 smithi103 ceph-mon[39629]: 4.b scrub ok 2024-04-23T16:29:03.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:03 smithi103 ceph-mon[39629]: osdmap e320: 12 total, 12 up, 10 in 2024-04-23T16:29:03.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:03 smithi103 ceph-mon[39629]: 4.a scrub starts 2024-04-23T16:29:03.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:03 smithi103 ceph-mon[39629]: pgmap v904: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 9.3 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 671 op/s; 175 KiB/s, 5 objects/s recovering 2024-04-23T16:29:03.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:03 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:29:03.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:03 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:29:03.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:03 smithi148 ceph-mon[44520]: 4.b scrub ok 2024-04-23T16:29:03.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:03 smithi148 ceph-mon[44520]: osdmap e320: 12 total, 12 up, 10 in 2024-04-23T16:29:03.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:03 smithi148 ceph-mon[44520]: 4.a scrub starts 2024-04-23T16:29:03.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:03 smithi148 ceph-mon[44520]: pgmap v904: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 9.3 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 671 op/s; 175 KiB/s, 5 objects/s recovering 2024-04-23T16:29:04.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:04 smithi188 ceph-mon[44101]: 4.7 scrub starts 2024-04-23T16:29:04.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:04 smithi188 ceph-mon[44101]: 4.e scrub ok 2024-04-23T16:29:04.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:04 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:29:04.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:04 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:29:04.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:04 smithi188 ceph-mon[44101]: osdmap e321: 12 total, 12 up, 10 in 2024-04-23T16:29:04.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:04 smithi188 ceph-mon[44101]: 4.4 scrub starts 2024-04-23T16:29:04.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:04 smithi188 ceph-mon[44101]: 4.1 scrub starts 2024-04-23T16:29:04.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:04 smithi188 ceph-mon[44101]: 4.5 scrub starts 2024-04-23T16:29:04.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:04 smithi103 ceph-mon[39629]: 4.7 scrub starts 2024-04-23T16:29:04.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:04 smithi103 ceph-mon[39629]: 4.e scrub ok 2024-04-23T16:29:04.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:04 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:29:04.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:04 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:29:04.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:04 smithi103 ceph-mon[39629]: osdmap e321: 12 total, 12 up, 10 in 2024-04-23T16:29:04.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:04 smithi103 ceph-mon[39629]: 4.4 scrub starts 2024-04-23T16:29:04.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:04 smithi103 ceph-mon[39629]: 4.1 scrub starts 2024-04-23T16:29:04.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:04 smithi103 ceph-mon[39629]: 4.5 scrub starts 2024-04-23T16:29:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:04 smithi148 ceph-mon[44520]: 4.7 scrub starts 2024-04-23T16:29:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:04 smithi148 ceph-mon[44520]: 4.e scrub ok 2024-04-23T16:29:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:04 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:29:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:04 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:29:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:04 smithi148 ceph-mon[44520]: osdmap e321: 12 total, 12 up, 10 in 2024-04-23T16:29:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:04 smithi148 ceph-mon[44520]: 4.4 scrub starts 2024-04-23T16:29:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:04 smithi148 ceph-mon[44520]: 4.1 scrub starts 2024-04-23T16:29:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:04 smithi148 ceph-mon[44520]: 4.5 scrub starts 2024-04-23T16:29:05.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:05 smithi188 ceph-mon[44101]: 4.7 scrub ok 2024-04-23T16:29:05.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:05 smithi188 ceph-mon[44101]: pgmap v906: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 9.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s; 50 KiB/s, 4 objects/s recovering 2024-04-23T16:29:05.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:05 smithi188 ceph-mon[44101]: 4.4 scrub ok 2024-04-23T16:29:05.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:05 smithi103 ceph-mon[39629]: 4.7 scrub ok 2024-04-23T16:29:05.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:05 smithi103 ceph-mon[39629]: pgmap v906: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 9.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s; 50 KiB/s, 4 objects/s recovering 2024-04-23T16:29:05.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:05 smithi103 ceph-mon[39629]: 4.4 scrub ok 2024-04-23T16:29:05.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:05 smithi148 ceph-mon[44520]: 4.7 scrub ok 2024-04-23T16:29:05.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:05 smithi148 ceph-mon[44520]: pgmap v906: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 9.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s; 50 KiB/s, 4 objects/s recovering 2024-04-23T16:29:05.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:05 smithi148 ceph-mon[44520]: 4.4 scrub ok 2024-04-23T16:29:06.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:06 smithi188 ceph-mon[44101]: 4.a scrub ok 2024-04-23T16:29:06.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:06 smithi188 ceph-mon[44101]: 4.1 scrub ok 2024-04-23T16:29:06.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:06 smithi188 ceph-mon[44101]: 4.5 scrub ok 2024-04-23T16:29:06.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:06 smithi103 ceph-mon[39629]: 4.a scrub ok 2024-04-23T16:29:06.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:06 smithi103 ceph-mon[39629]: 4.1 scrub ok 2024-04-23T16:29:06.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:06 smithi103 ceph-mon[39629]: 4.5 scrub ok 2024-04-23T16:29:06.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:06 smithi148 ceph-mon[44520]: 4.a scrub ok 2024-04-23T16:29:06.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:06 smithi148 ceph-mon[44520]: 4.1 scrub ok 2024-04-23T16:29:06.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:06 smithi148 ceph-mon[44520]: 4.5 scrub ok 2024-04-23T16:29:07.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:07 smithi188 ceph-mon[44101]: pgmap v907: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 9.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.00k op/s 2024-04-23T16:29:07.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:07 smithi103 ceph-mon[39629]: pgmap v907: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 9.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.00k op/s 2024-04-23T16:29:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:07 smithi148 ceph-mon[44520]: pgmap v907: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 9.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.00k op/s 2024-04-23T16:29:08.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:08 smithi188 ceph-mon[44101]: 4.6 scrub starts 2024-04-23T16:29:08.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:08 smithi103 ceph-mon[39629]: 4.6 scrub starts 2024-04-23T16:29:08.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:08 smithi148 ceph-mon[44520]: 4.6 scrub starts 2024-04-23T16:29:08.817 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:29:08.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:08.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:29:08.921 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:29:08.957 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:09.187 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:29:09.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:09.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:29:09.297 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:29:09.332 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:09.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:09 smithi188 ceph-mon[44101]: pgmap v908: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 9.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 837 op/s 2024-04-23T16:29:09.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:09 smithi188 ceph-mon[44101]: 4.6 scrub ok 2024-04-23T16:29:09.534 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:29:09.538 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:09 smithi103 ceph-mon[39629]: pgmap v908: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 9.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 837 op/s 2024-04-23T16:29:09.538 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:09 smithi103 ceph-mon[39629]: 4.6 scrub ok 2024-04-23T16:29:09.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:09.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:29:09.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:09 smithi148 ceph-mon[44520]: pgmap v908: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 9.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 837 op/s 2024-04-23T16:29:09.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:09 smithi148 ceph-mon[44520]: 4.6 scrub ok 2024-04-23T16:29:09.640 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:29:09.677 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:09.735 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:29:09.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:09.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:29:09.841 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:29:09.855 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:29:09.888 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:09.904 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:29:09.910 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:29:09.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:09.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:29:10.016 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:10.055 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:10.077 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:10.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:10.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:10.182 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:29:10.215 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:10.250 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:29:10.262 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:10 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/218587924' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:29:10.262 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:10 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:29:10.262 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:10 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:29:10.464 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:10 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/218587924' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:29:10.464 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:10 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:29:10.465 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:10 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:29:10.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:10 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/218587924' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:29:10.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:10 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:29:10.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:10 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:29:10.580 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:29:10.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:10.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:29:10.684 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:29:10.717 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:10.739 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:29:10.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:10.743 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:29:10.844 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:29:10.875 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:10.956 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:29:10.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:10.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:29:11.060 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:29:11.102 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:11.133 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:29:11.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:11.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:29:11.217 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:29:11.239 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:29:11.277 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:11.303 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:29:11.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:11.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:29:11.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:11 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/218587924' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:29:11.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:11 smithi148 ceph-mon[44520]: osdmap e322: 12 total, 12 up, 10 in 2024-04-23T16:29:11.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:11 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:29:11.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:11 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:29:11.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:11 smithi148 ceph-mon[44520]: pgmap v910: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 8.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-23T16:29:11.409 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:29:11.457 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:11.510 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:29:11.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:11.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:29:11.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:11 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/218587924' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:29:11.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:11 smithi103 ceph-mon[39629]: osdmap e322: 12 total, 12 up, 10 in 2024-04-23T16:29:11.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:11 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:29:11.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:11 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:29:11.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:11 smithi103 ceph-mon[39629]: pgmap v910: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 8.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-23T16:29:11.615 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:29:11.658 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:11.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:11 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/218587924' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:29:11.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:11 smithi188 ceph-mon[44101]: osdmap e322: 12 total, 12 up, 10 in 2024-04-23T16:29:11.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:11 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:29:11.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:11 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:29:11.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:11 smithi188 ceph-mon[44101]: pgmap v910: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 8.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-23T16:29:11.812 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:29:11.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:11.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:29:11.917 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:11.948 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:11.989 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:11.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:11.991 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:12.092 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:29:12.133 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:12.285 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:29:12.288 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:12 smithi103 ceph-mon[39629]: osdmap e323: 12 total, 12 up, 10 in 2024-04-23T16:29:12.288 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:12 smithi103 ceph-mon[39629]: 4.2 scrub starts 2024-04-23T16:29:12.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:12.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:29:12.392 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:29:12.421 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:12.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:12 smithi148 ceph-mon[44520]: osdmap e323: 12 total, 12 up, 10 in 2024-04-23T16:29:12.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:12 smithi148 ceph-mon[44520]: 4.2 scrub starts 2024-04-23T16:29:12.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:12 smithi188 ceph-mon[44101]: osdmap e323: 12 total, 12 up, 10 in 2024-04-23T16:29:12.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:12 smithi188 ceph-mon[44101]: 4.2 scrub starts 2024-04-23T16:29:13.565 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:13 smithi103 ceph-mon[39629]: pgmap v912: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 8.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-23T16:29:13.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:13 smithi148 ceph-mon[44520]: pgmap v912: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 8.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-23T16:29:13.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:13 smithi188 ceph-mon[44101]: pgmap v912: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 8.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-23T16:29:13.843 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:29:14.302 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:14 smithi103 ceph-mon[39629]: 4.2 scrub ok 2024-04-23T16:29:14.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:14 smithi148 ceph-mon[44520]: 4.2 scrub ok 2024-04-23T16:29:14.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:14 smithi188 ceph-mon[44101]: 4.2 scrub ok 2024-04-23T16:29:15.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:15 smithi103 ceph-mon[39629]: pgmap v913: 27 pgs: 27 active+clean; 8.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-23T16:29:15.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:15 smithi148 ceph-mon[44520]: pgmap v913: 27 pgs: 27 active+clean; 8.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-23T16:29:15.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:15 smithi188 ceph-mon[44101]: pgmap v913: 27 pgs: 27 active+clean; 8.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-23T16:29:15.831 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:29:15.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:15.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:29:15.936 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:29:15.979 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:16.006 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:29:16.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:16.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:29:16.113 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:29:16.148 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:16.179 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:29:16.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:16.185 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:29:16.287 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:29:16.333 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:16.347 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:29:16.369 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:29:16.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:16.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:29:16.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:16 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:29:16.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:16 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:29:16.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:16 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:29:16.475 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:29:16.505 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:16.531 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:29:16.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:16.535 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:29:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:16 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:29:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:16 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:29:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:16 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:29:16.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:16 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:29:16.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:16 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:29:16.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:16 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:29:16.635 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:29:16.695 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:16.901 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:29:16.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:16.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:29:17.004 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:29:17.059 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:17.100 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:29:17.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:17.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:29:17.210 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:29:17.247 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:17.284 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:29:17.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:17.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:29:17.346 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:17 smithi148 ceph-mon[44520]: osdmap e324: 12 total, 12 up, 10 in 2024-04-23T16:29:17.347 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:17 smithi148 ceph-mon[44520]: pgmap v915: 27 pgs: 27 active+clean; 8.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.34k op/s 2024-04-23T16:29:17.347 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:29:17.347 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:29:17.347 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:29:17.347 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:17 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/164019290' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:29:17.347 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:17 smithi148 ceph-mon[44520]: from='client.25501 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:29:17.388 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:17.417 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:17.443 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:17.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:17.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:17.546 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:29:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:17 smithi103 ceph-mon[39629]: osdmap e324: 12 total, 12 up, 10 in 2024-04-23T16:29:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:17 smithi103 ceph-mon[39629]: pgmap v915: 27 pgs: 27 active+clean; 8.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.34k op/s 2024-04-23T16:29:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:29:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:29:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:29:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:17 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/164019290' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:29:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:17 smithi103 ceph-mon[39629]: from='client.25501 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:29:17.601 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:17.648 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:29:17.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:17.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:29:17.703 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:29:17.703 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:29:17.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.704 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:194832 4.4 4:21d372c3:::benchmark_data_smithi188_101370_object31383:head [delete] snapc 0=[] ondisk+write+known_if_redirected e324)", 2024-04-23T16:29:17.704 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:29:17.215904+0000", 2024-04-23T16:29:17.704 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.020578744999999999, 2024-04-23T16:29:17.704 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020602042000000001, 2024-04-23T16:29:17.704 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:17.704 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:17.704 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:29:17.704 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:17.704 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:17.704 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:17.704 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 194832 2024-04-23T16:29:17.704 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.704 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:17.704 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.705 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:17.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.215904+0000", 2024-04-23T16:29:17.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:17.705 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.705 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.705 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:17.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.215904+0000", 2024-04-23T16:29:17.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:17.705 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.705 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.705 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:17.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.215907+0000", 2024-04-23T16:29:17.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3999999999999999e-06 2024-04-23T16:29:17.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.706 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.706 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:17.706 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.215914+0000", 2024-04-23T16:29:17.706 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8249999999999999e-06 2024-04-23T16:29:17.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.706 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.706 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:17.706 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.215927+0000", 2024-04-23T16:29:17.706 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3369e-05 2024-04-23T16:29:17.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.706 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.706 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:17.707 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.215936+0000", 2024-04-23T16:29:17.707 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9309999999999997e-06 2024-04-23T16:29:17.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.707 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:17.707 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.216239+0000", 2024-04-23T16:29:17.707 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00030364500000000001 2024-04-23T16:29:17.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.707 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:17.707 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.216380+0000", 2024-04-23T16:29:17.707 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014040600000000001 2024-04-23T16:29:17.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.708 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:29:17.708 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.216515+0000", 2024-04-23T16:29:17.708 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013559600000000001 2024-04-23T16:29:17.708 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.708 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:17.708 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.221372+0000", 2024-04-23T16:29:17.708 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0048567619999999997 2024-04-23T16:29:17.708 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:17.708 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:17.708 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:17.708 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.708 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:194839 4.4 4:2374f14f:::benchmark_data_smithi188_101370_object31390:head [delete] snapc 0=[] ondisk+write+known_if_redirected e324)", 2024-04-23T16:29:17.708 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:29:17.224481+0000", 2024-04-23T16:29:17.709 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.012001883, 2024-04-23T16:29:17.709 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012080047999999999, 2024-04-23T16:29:17.709 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:17.709 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:17.709 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:29:17.709 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:17.709 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:17.709 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:17.709 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 194839 2024-04-23T16:29:17.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.709 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:17.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:17.709 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.224481+0000", 2024-04-23T16:29:17.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:17.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:17.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.224481+0000", 2024-04-23T16:29:17.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:17.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:17.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.224484+0000", 2024-04-23T16:29:17.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.418e-06 2024-04-23T16:29:17.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:17.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.224490+0000", 2024-04-23T16:29:17.711 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6510000000000004e-06 2024-04-23T16:29:17.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.711 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:17.711 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.224502+0000", 2024-04-23T16:29:17.711 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1367e-05 2024-04-23T16:29:17.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.711 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:17.711 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.224509+0000", 2024-04-23T16:29:17.711 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-23T16:29:17.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.711 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:17.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.224566+0000", 2024-04-23T16:29:17.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6994999999999999e-05 2024-04-23T16:29:17.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.712 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:17.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.224695+0000", 2024-04-23T16:29:17.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000128725 2024-04-23T16:29:17.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.712 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:29:17.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.224818+0000", 2024-04-23T16:29:17.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012286999999999999 2024-04-23T16:29:17.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:17.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:17.712 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:17.713 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:17.229182+0000", 2024-04-23T16:29:17.713 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043639439999999998 2024-04-23T16:29:17.713 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:17.713 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:17.713 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:17.713 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:17.713 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:29:17.713 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 2 2024-04-23T16:29:17.713 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:29:17.713 INFO:teuthology.orchestra.run.smithi103.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-23T16:29:17.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:17 smithi188 ceph-mon[44101]: osdmap e324: 12 total, 12 up, 10 in 2024-04-23T16:29:17.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:17 smithi188 ceph-mon[44101]: pgmap v915: 27 pgs: 27 active+clean; 8.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.34k op/s 2024-04-23T16:29:17.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:29:17.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:29:17.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:29:17.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:17 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/164019290' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:29:17.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:17 smithi188 ceph-mon[44101]: from='client.25501 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:29:17.755 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:29:17.808 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:17.844 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:29:17.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:17.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:29:17.954 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:17.985 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:18.009 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:18.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:18.015 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:18.116 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:29:18.170 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:18.204 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:29:18.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:18.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:29:18.315 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:29:18.347 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:18.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:18 smithi188 ceph-mon[44101]: from='client.25501 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:29:18.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:18 smithi188 ceph-mon[44101]: osdmap e325: 12 total, 12 up, 10 in 2024-04-23T16:29:19.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:18 smithi103 ceph-mon[39629]: from='client.25501 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:29:19.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:18 smithi103 ceph-mon[39629]: osdmap e325: 12 total, 12 up, 10 in 2024-04-23T16:29:19.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:18 smithi148 ceph-mon[44520]: from='client.25501 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:29:19.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:18 smithi148 ceph-mon[44520]: osdmap e325: 12 total, 12 up, 10 in 2024-04-23T16:29:19.167 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-23T16:29:19.393 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:29:19.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:19.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:29:19.501 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:19.535 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:19.558 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:19.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:19.566 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:19.652 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:29:19.667 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:19.708 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:19.736 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:19.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:19.746 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:19.847 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:29:19.886 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:19.921 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:29:19.945 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:29:19.972 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:29:19.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:19 smithi188 ceph-mon[44101]: pgmap v917: 27 pgs: 27 active+clean; 8.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-23T16:29:20.000 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:29:20.003 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:19 smithi103 ceph-mon[39629]: pgmap v917: 27 pgs: 27 active+clean; 8.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-23T16:29:20.036 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:29:20.073 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:29:20.100 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:29:20.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:19 smithi148 ceph-mon[44520]: pgmap v917: 27 pgs: 27 active+clean; 8.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-23T16:29:20.130 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:29:20.151 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:29:20.181 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:29:20.211 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:29:20.235 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:29:20.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:29:20.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:29:20.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:29:20.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:29:20.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:29:20.239 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:29:20.239 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:29:20.239 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:29:20.239 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:29:20.239 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:29:20.239 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:29:20.239 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:29:20.452 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:29:20.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:20.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:29:20.559 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:29:20.598 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:20.625 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:29:20.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:20.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:29:20.729 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:29:20.776 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:20.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:20 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/48972944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:29:20.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:20.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:20.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:20.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:20.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:20.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:20.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:20.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:20.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:20.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:20 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/48972944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:29:21.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:20 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/48972944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:29:21.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.534 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:29:21.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:21.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:29:21.638 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:29:21.672 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:21.696 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:29:21.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:21.701 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:29:21.802 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:29:21.847 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:21.883 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:29:21.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:21.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:29:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:21 smithi188 ceph-mon[44101]: pgmap v918: 27 pgs: 27 active+clean; 8.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.36k op/s 2024-04-23T16:29:21.985 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:29:22.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:22.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:22.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:21 smithi148 ceph-mon[44520]: pgmap v918: 27 pgs: 27 active+clean; 8.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.36k op/s 2024-04-23T16:29:22.026 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:22.057 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:29:22.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:22.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:29:22.087 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:29:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:21 smithi103 ceph-mon[39629]: pgmap v918: 27 pgs: 27 active+clean; 8.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.36k op/s 2024-04-23T16:29:22.161 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:29:22.222 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:22.265 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:29:22.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:22.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:29:22.370 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:22.403 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:22.428 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:22.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:22.431 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:22.532 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:29:22.568 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:23.022 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:29:23.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:23.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:29:23.131 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:29:23.168 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:23.194 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:29:23.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:23.197 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:29:23.235 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:29:23.298 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:29:23.328 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:23.355 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:29:23.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:23.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:29:23.460 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:29:23.491 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:23.512 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:29:23.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:23.514 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:29:23.615 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:29:23.659 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:23.701 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:29:23.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:23.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:29:23.814 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:29:23.852 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:23.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:23 smithi188 ceph-mon[44101]: pgmap v919: 27 pgs: 27 active+clean; 7.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-23T16:29:23.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:23 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:29:23.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:23 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:29:23.985 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:29:23.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:23.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:29:23.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:23 smithi103 ceph-mon[39629]: pgmap v919: 27 pgs: 27 active+clean; 7.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-23T16:29:23.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:23 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:29:23.991 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:23 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:29:24.089 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:29:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:23 smithi148 ceph-mon[44520]: pgmap v919: 27 pgs: 27 active+clean; 7.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-23T16:29:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:23 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:29:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:23 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:29:24.125 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:24.287 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:29:24.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:24.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:29:24.389 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:29:24.442 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:24.736 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:29:25.075 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:24 smithi103 ceph-mon[39629]: osdmap e326: 12 total, 12 up, 10 in 2024-04-23T16:29:25.075 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:24 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:29:25.075 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:24 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:29:25.075 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:24 smithi103 ceph-mon[39629]: osdmap e327: 12 total, 12 up, 10 in 2024-04-23T16:29:25.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1022578035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:29:25.089 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:29:25.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:24 smithi148 ceph-mon[44520]: osdmap e326: 12 total, 12 up, 10 in 2024-04-23T16:29:25.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:24 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:29:25.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:24 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:29:25.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:24 smithi148 ceph-mon[44520]: osdmap e327: 12 total, 12 up, 10 in 2024-04-23T16:29:25.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:24 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1022578035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:29:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:24 smithi188 ceph-mon[44101]: osdmap e326: 12 total, 12 up, 10 in 2024-04-23T16:29:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:24 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:29:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:24 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:29:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:24 smithi188 ceph-mon[44101]: osdmap e327: 12 total, 12 up, 10 in 2024-04-23T16:29:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1022578035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:29:25.487 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10, 1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1] 2024-04-23T16:29:25.487 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:29:25.488 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-04-23T16:29:25.488 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-23T16:29:25.488 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-23T16:29:25.488 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-04-23T16:29:26.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:25 smithi103 ceph-mon[39629]: pgmap v922: 27 pgs: 27 active+clean; 7.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.46k op/s 2024-04-23T16:29:26.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:25 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:29:26.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:25 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1022578035' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:29:26.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:25 smithi103 ceph-mon[39629]: osdmap e328: 12 total, 12 up, 10 in 2024-04-23T16:29:26.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:25 smithi148 ceph-mon[44520]: pgmap v922: 27 pgs: 27 active+clean; 7.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.46k op/s 2024-04-23T16:29:26.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:25 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:29:26.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:25 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1022578035' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:29:26.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:25 smithi148 ceph-mon[44520]: osdmap e328: 12 total, 12 up, 10 in 2024-04-23T16:29:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:25 smithi188 ceph-mon[44101]: pgmap v922: 27 pgs: 27 active+clean; 7.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.46k op/s 2024-04-23T16:29:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:25 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:29:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:25 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1022578035' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:29:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:25 smithi188 ceph-mon[44101]: osdmap e328: 12 total, 12 up, 10 in 2024-04-23T16:29:26.303 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:29:26.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:26.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:29:26.407 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:29:26.456 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:26.487 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:29:26.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:26.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:29:26.594 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:26.625 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:26.633 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:29:26.633 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:29:26.649 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:26.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:26.652 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:26.753 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:29:26.786 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:26.812 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:29:26.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:26.818 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:29:26.918 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:29:26.949 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:27.150 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:29:27.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:27.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:29:27.257 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:29:27.294 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:27.858 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:29:27.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:27.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:29:27.962 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:29:27.991 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:27 smithi188 ceph-mon[44101]: pgmap v924: 27 pgs: 27 active+clean; 7.5 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.32k op/s 2024-04-23T16:29:27.992 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:28.016 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:29:28.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:28.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:29:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:27 smithi103 ceph-mon[39629]: pgmap v924: 27 pgs: 27 active+clean; 7.5 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.32k op/s 2024-04-23T16:29:28.122 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:28.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:27 smithi148 ceph-mon[44520]: pgmap v924: 27 pgs: 27 active+clean; 7.5 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.32k op/s 2024-04-23T16:29:28.153 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:28.175 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:28.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:28.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:28.283 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:29:28.315 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:28.337 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:29:28.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:28.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:29:28.422 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-23T16:29:28.442 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:29:28.500 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:28.527 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:29:28.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:28.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:29:28.636 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:29:28.684 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:28.721 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:29:28.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:28.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:29:28.826 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:29:28.878 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:28.900 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:29:28.910 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:29:28.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:28.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:29:28.977 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:29:29.014 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:29:29.014 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:29:29.041 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:29:29.044 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:29.066 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:29:29.068 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:29:29.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:29.071 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:29:29.108 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:29:29.138 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:29:29.172 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:29:29.173 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:29:29.214 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:29.222 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:29:29.242 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:29:29.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:29.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:29:29.261 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:29:29.291 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:29:29.320 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:29:29.347 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:29:29.349 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:29:29.354 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:29:29.354 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:29:29.354 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:29:29.354 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:29:29.354 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:29:29.354 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:29:29.354 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:29:29.354 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:29:29.355 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:29:29.355 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:29:29.355 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:29:29.355 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:29:29.413 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:29.451 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:29:29.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:29.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:29:29.533 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:29:29.558 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:29:29.628 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:29.680 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:29:29.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:29.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:29:29.747 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:29 smithi103 ceph-mon[39629]: pgmap v925: 27 pgs: 27 active+clean; 7.5 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-23T16:29:29.747 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:29 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3250951675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:29:29.747 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:29.747 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:29.748 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:29.748 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:29 smithi103 ceph-mon[39629]: osdmap e329: 12 total, 12 up, 10 in 2024-04-23T16:29:29.748 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:29.748 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:29.748 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:29.748 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:29.748 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:29.748 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:29.786 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:29:29.854 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:29.889 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:29:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:29.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:29:30.000 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:29:30.030 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:29 smithi188 ceph-mon[44101]: pgmap v925: 27 pgs: 27 active+clean; 7.5 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-23T16:29:30.030 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:29 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3250951675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:29:30.030 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:30.031 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:30.031 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:30.031 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:29 smithi188 ceph-mon[44101]: osdmap e329: 12 total, 12 up, 10 in 2024-04-23T16:29:30.031 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:30.031 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:30.031 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:30.031 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:30.031 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:30.031 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:30.033 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:30.056 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:29:30.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:30.060 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:29:30.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:29 smithi148 ceph-mon[44520]: pgmap v925: 27 pgs: 27 active+clean; 7.5 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-23T16:29:30.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:29 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3250951675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:29:30.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:30.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:30.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:30.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:29 smithi148 ceph-mon[44520]: osdmap e329: 12 total, 12 up, 10 in 2024-04-23T16:29:30.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:30.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:30.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:30.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:30.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:30.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:30.161 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:29:30.199 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:30.651 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:29:30.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:30.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:29:30.757 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:29:30.803 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:30.852 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:29:30.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:30.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:29:30.957 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:29:31.032 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:31.032 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:31.032 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:31.037 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:31.062 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:29:31.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:31.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:29:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:31.170 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:29:31.198 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:31.198 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:31.198 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:31.200 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:31.224 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:29:31.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:31.228 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:29:31.252 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:29:31.328 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:29:31.402 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:31.442 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:29:31.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:31.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:29:31.550 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:29:31.596 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:31.620 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:29:31.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:31.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:29:31.728 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:31.759 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:31 smithi103 ceph-mon[39629]: pgmap v927: 27 pgs: 27 active+clean; 7.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-23T16:29:31.765 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:31.790 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:31.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:31.793 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:31.894 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:29:31.937 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:31.974 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:29:31.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:31.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:29:32.045 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:29:32.045 INFO:teuthology.orchestra.run.smithi103.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-23T16:29:32.045 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:29:32.082 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:29:32.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:31 smithi148 ceph-mon[44520]: pgmap v927: 27 pgs: 27 active+clean; 7.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-23T16:29:32.135 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:32.153 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:29:32.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:32.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:29:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:31 smithi188 ceph-mon[44101]: pgmap v927: 27 pgs: 27 active+clean; 7.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-23T16:29:32.258 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:29:32.309 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:32.344 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:29:32.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:32.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:29:32.448 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:29:32.489 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:32.643 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:29:32.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:32.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:29:32.747 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:29:32.777 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:33.198 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:29:33.281 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:29:33.334 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:29:33.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:33.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:29:33.449 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:29:33.485 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:33.509 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:29:33.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:33.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:29:33.614 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:29:33.646 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:33.682 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:29:33.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:33.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:29:33.792 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:29:33.821 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:33.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:33 smithi103 ceph-mon[39629]: pgmap v928: 27 pgs: 27 active+clean; 7.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-23T16:29:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:33 smithi148 ceph-mon[44520]: pgmap v928: 27 pgs: 27 active+clean; 7.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-23T16:29:34.176 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:29:34.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:34.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:29:34.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:33 smithi188 ceph-mon[44101]: pgmap v928: 27 pgs: 27 active+clean; 7.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-23T16:29:34.281 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:34.320 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:34.345 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:34.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:34.349 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:34.450 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:29:34.490 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:34.515 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:29:34.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:34.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:29:34.624 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:29:34.656 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:34.746 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:29:34.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:34.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:29:34.851 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:29:34.884 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:34.905 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:29:34.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:34.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:29:35.011 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:29:35.069 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:35.128 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:29:35.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:35.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:29:35.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:35 smithi188 ceph-mon[44101]: pgmap v929: 27 pgs: 27 active+clean; 6.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.32k op/s 2024-04-23T16:29:35.234 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:29:35.264 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:35.292 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:29:35.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:35.295 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:29:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:35 smithi103 ceph-mon[39629]: pgmap v929: 27 pgs: 27 active+clean; 6.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.32k op/s 2024-04-23T16:29:35.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:35 smithi148 ceph-mon[44520]: pgmap v929: 27 pgs: 27 active+clean; 6.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.32k op/s 2024-04-23T16:29:35.396 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:29:35.434 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:35.613 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:29:36.493 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:29:36.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:36.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:29:36.593 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:29:36.599 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:29:36.666 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:36.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/713185717' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:29:36.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/713185717' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:29:36.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/713185717' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:29:37.333 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:29:37.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:37.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:29:37.437 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:29:37.466 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:37.788 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:37 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:29:37.789 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:37 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/713185717' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:29:37.789 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:37 smithi103 ceph-mon[39629]: osdmap e330: 12 total, 12 up, 10 in 2024-04-23T16:29:37.789 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:37 smithi103 ceph-mon[39629]: pgmap v931: 27 pgs: 27 active+clean; 6.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.58k op/s 2024-04-23T16:29:37.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:37 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:29:37.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:37 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/713185717' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:29:37.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:37 smithi148 ceph-mon[44520]: osdmap e330: 12 total, 12 up, 10 in 2024-04-23T16:29:37.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:37 smithi148 ceph-mon[44520]: pgmap v931: 27 pgs: 27 active+clean; 6.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.58k op/s 2024-04-23T16:29:37.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:37 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:29:37.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:37 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/713185717' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:29:37.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:37 smithi188 ceph-mon[44101]: osdmap e330: 12 total, 12 up, 10 in 2024-04-23T16:29:37.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:37 smithi188 ceph-mon[44101]: pgmap v931: 27 pgs: 27 active+clean; 6.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.58k op/s 2024-04-23T16:29:38.314 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10, 1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1] 2024-04-23T16:29:38.314 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:29:38.314 INFO:tasks.thrashosds.thrasher:inject_pause on osd.8 2024-04-23T16:29:38.314 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-23T16:29:38.315 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-23T16:29:38.315 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 config set filestore_inject_stall 3 2024-04-23T16:29:38.465 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:29:38.466 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:29:38.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:29:38.466 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:29:38.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.466 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.14862.0:269181 2.3 e168/110 2:c3209f67:::benchmark_data_smithi188_77627_object99832:head v 168'16692, mlcod=168'16690)", 2024-04-23T16:29:38.466 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:20:49.556842+0000", 2024-04-23T16:29:38.466 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 527.243079416, 2024-04-23T16:29:38.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.070806754999999999, 2024-04-23T16:29:38.466 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.466 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.466 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.466 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.556842+0000", 2024-04-23T16:29:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.556842+0000", 2024-04-23T16:29:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.556843+0000", 2024-04-23T16:29:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2470000000000001e-06 2024-04-23T16:29:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.556850+0000", 2024-04-23T16:29:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.55e-06 2024-04-23T16:29:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.556859+0000", 2024-04-23T16:29:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3139999999999993e-06 2024-04-23T16:29:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.556866+0000", 2024-04-23T16:29:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.72e-06 2024-04-23T16:29:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.556910+0000", 2024-04-23T16:29:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4044e-05 2024-04-23T16:29:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.556929+0000", 2024-04-23T16:29:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9258000000000001e-05 2024-04-23T16:29:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.627603+0000", 2024-04-23T16:29:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.070673949999999999 2024-04-23T16:29:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:20:49.627649+0000", 2024-04-23T16:29:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5671999999999999e-05 2024-04-23T16:29:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.471 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:105164 4.6 4:60b85178:::benchmark_data_smithi188_101370_object105163:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e295)", 2024-04-23T16:29:38.471 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:28:00.251784+0000", 2024-04-23T16:29:38.471 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 96.548138011000006, 2024-04-23T16:29:38.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.072133548000000006, 2024-04-23T16:29:38.471 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.471 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.471 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.471 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:38.471 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:38.471 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:38.471 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 105164 2024-04-23T16:29:38.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.251784+0000", 2024-04-23T16:29:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.251784+0000", 2024-04-23T16:29:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.251785+0000", 2024-04-23T16:29:38.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.266e-06 2024-04-23T16:29:38.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.251838+0000", 2024-04-23T16:29:38.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3102000000000001e-05 2024-04-23T16:29:38.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.251846+0000", 2024-04-23T16:29:38.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4239999999999993e-06 2024-04-23T16:29:38.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.251853+0000", 2024-04-23T16:29:38.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.127e-06 2024-04-23T16:29:38.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.262698+0000", 2024-04-23T16:29:38.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010845852 2024-04-23T16:29:38.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.262845+0000", 2024-04-23T16:29:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014687200000000001 2024-04-23T16:29:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6,9", 2024-04-23T16:29:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.263004+0000", 2024-04-23T16:29:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015825100000000001 2024-04-23T16:29:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.269326+0000", 2024-04-23T16:29:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0063228299999999998 2024-04-23T16:29:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.273058+0000", 2024-04-23T16:29:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037320700000000001 2024-04-23T16:29:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.321805+0000", 2024-04-23T16:29:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.048746663000000003 2024-04-23T16:29:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.321871+0000", 2024-04-23T16:29:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5436000000000002e-05 2024-04-23T16:29:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.323917+0000", 2024-04-23T16:29:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020466550000000001 2024-04-23T16:29:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:105179 4.6 4:6561370a:::benchmark_data_smithi188_101370_object105178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e295)", 2024-04-23T16:29:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:28:00.260284+0000", 2024-04-23T16:29:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 96.539637671999998, 2024-04-23T16:29:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.075289554999999994, 2024-04-23T16:29:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.478 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.478 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.478 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:38.478 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:38.478 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:38.478 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 105179 2024-04-23T16:29:38.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.478 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.260284+0000", 2024-04-23T16:29:38.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.260284+0000", 2024-04-23T16:29:38.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.260286+0000", 2024-04-23T16:29:38.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7600000000000001e-06 2024-04-23T16:29:38.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.260318+0000", 2024-04-23T16:29:38.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1773000000000002e-05 2024-04-23T16:29:38.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.260325+0000", 2024-04-23T16:29:38.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.622e-06 2024-04-23T16:29:38.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.260332+0000", 2024-04-23T16:29:38.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8630000000000002e-06 2024-04-23T16:29:38.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.265295+0000", 2024-04-23T16:29:38.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004962976 2024-04-23T16:29:38.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.265395+0000", 2024-04-23T16:29:38.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100109 2024-04-23T16:29:38.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6,9", 2024-04-23T16:29:38.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.265503+0000", 2024-04-23T16:29:38.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107831 2024-04-23T16:29:38.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:38.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.269335+0000", 2024-04-23T16:29:38.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038320429999999998 2024-04-23T16:29:38.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.273101+0000", 2024-04-23T16:29:38.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00376633 2024-04-23T16:29:38.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.333094+0000", 2024-04-23T16:29:38.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.059993021000000001 2024-04-23T16:29:38.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.333152+0000", 2024-04-23T16:29:38.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7417999999999998e-05 2024-04-23T16:29:38.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.335574+0000", 2024-04-23T16:29:38.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002421809 2024-04-23T16:29:38.484 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.484 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.484 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.484 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:105188 4.6 4:61ee61fd:::benchmark_data_smithi188_101370_object105187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e295)", 2024-04-23T16:29:38.484 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:28:00.264068+0000", 2024-04-23T16:29:38.484 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 96.535853384999996, 2024-04-23T16:29:38.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.07376133, 2024-04-23T16:29:38.484 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.484 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.484 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.485 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:38.485 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:38.485 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:38.485 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 105188 2024-04-23T16:29:38.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.485 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.264068+0000", 2024-04-23T16:29:38.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.264068+0000", 2024-04-23T16:29:38.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.264070+0000", 2024-04-23T16:29:38.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.621e-06 2024-04-23T16:29:38.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.264120+0000", 2024-04-23T16:29:38.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0155999999999999e-05 2024-04-23T16:29:38.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.264130+0000", 2024-04-23T16:29:38.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0349e-05 2024-04-23T16:29:38.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.264137+0000", 2024-04-23T16:29:38.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1210000000000003e-06 2024-04-23T16:29:38.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.267234+0000", 2024-04-23T16:29:38.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030972899999999999 2024-04-23T16:29:38.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.267325+0000", 2024-04-23T16:29:38.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1629e-05 2024-04-23T16:29:38.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6,9", 2024-04-23T16:29:38.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.267424+0000", 2024-04-23T16:29:38.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8072999999999994e-05 2024-04-23T16:29:38.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:38.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.269490+0000", 2024-04-23T16:29:38.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002066905 2024-04-23T16:29:38.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.273133+0000", 2024-04-23T16:29:38.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0036424169999999998 2024-04-23T16:29:38.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.335644+0000", 2024-04-23T16:29:38.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.062510938000000002 2024-04-23T16:29:38.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.335705+0000", 2024-04-23T16:29:38.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0767000000000003e-05 2024-04-23T16:29:38.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.337830+0000", 2024-04-23T16:29:38.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0021250639999999999 2024-04-23T16:29:38.490 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.490 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.490 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.490 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:105198 4.6 4:623868bc:::benchmark_data_smithi188_101370_object105197:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e295)", 2024-04-23T16:29:38.490 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:28:00.275353+0000", 2024-04-23T16:29:38.491 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 96.524568392000006, 2024-04-23T16:29:38.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.071458009000000003, 2024-04-23T16:29:38.491 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.491 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.491 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.491 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:38.491 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:38.491 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:38.491 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 105198 2024-04-23T16:29:38.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.491 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.275353+0000", 2024-04-23T16:29:38.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.275353+0000", 2024-04-23T16:29:38.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.275356+0000", 2024-04-23T16:29:38.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4370000000000001e-06 2024-04-23T16:29:38.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.275405+0000", 2024-04-23T16:29:38.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9691999999999998e-05 2024-04-23T16:29:38.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.275418+0000", 2024-04-23T16:29:38.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2765999999999999e-05 2024-04-23T16:29:38.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.275426+0000", 2024-04-23T16:29:38.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0660000000000004e-06 2024-04-23T16:29:38.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.277459+0000", 2024-04-23T16:29:38.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020328379999999999 2024-04-23T16:29:38.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.277564+0000", 2024-04-23T16:29:38.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010493900000000001 2024-04-23T16:29:38.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6,9", 2024-04-23T16:29:38.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.277668+0000", 2024-04-23T16:29:38.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000104221 2024-04-23T16:29:38.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:38.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.291842+0000", 2024-04-23T16:29:38.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014174203 2024-04-23T16:29:38.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.291879+0000", 2024-04-23T16:29:38.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6720000000000001e-05 2024-04-23T16:29:38.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.344680+0000", 2024-04-23T16:29:38.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.052800387999999997 2024-04-23T16:29:38.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.344716+0000", 2024-04-23T16:29:38.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6943999999999997e-05 2024-04-23T16:29:38.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.346811+0000", 2024-04-23T16:29:38.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002094795 2024-04-23T16:29:38.497 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.497 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.497 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.497 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:105205 4.6 4:6595ee81:::benchmark_data_smithi188_101370_object105204:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e295)", 2024-04-23T16:29:38.497 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:28:00.279459+0000", 2024-04-23T16:29:38.497 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 96.520462948000002, 2024-04-23T16:29:38.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.089592385999999996, 2024-04-23T16:29:38.497 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.497 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.497 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.497 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:38.498 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:38.498 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:38.498 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 105205 2024-04-23T16:29:38.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.498 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.279459+0000", 2024-04-23T16:29:38.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.279459+0000", 2024-04-23T16:29:38.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.279460+0000", 2024-04-23T16:29:38.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.553e-06 2024-04-23T16:29:38.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.279519+0000", 2024-04-23T16:29:38.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8559e-05 2024-04-23T16:29:38.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.279533+0000", 2024-04-23T16:29:38.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4089e-05 2024-04-23T16:29:38.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.279542+0000", 2024-04-23T16:29:38.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8400000000000001e-06 2024-04-23T16:29:38.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.291903+0000", 2024-04-23T16:29:38.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012361633 2024-04-23T16:29:38.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.291999+0000", 2024-04-23T16:29:38.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5820000000000001e-05 2024-04-23T16:29:38.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6,9", 2024-04-23T16:29:38.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.292101+0000", 2024-04-23T16:29:38.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101304 2024-04-23T16:29:38.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:38.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.294669+0000", 2024-04-23T16:29:38.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0025688180000000001 2024-04-23T16:29:38.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.295045+0000", 2024-04-23T16:29:38.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00037594700000000001 2024-04-23T16:29:38.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.367171+0000", 2024-04-23T16:29:38.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.072125599999999998 2024-04-23T16:29:38.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.367209+0000", 2024-04-23T16:29:38.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7796000000000002e-05 2024-04-23T16:29:38.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.369051+0000", 2024-04-23T16:29:38.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001842427 2024-04-23T16:29:38.503 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.503 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.503 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.503 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:105837 4.6 4:6ef22d6b:::benchmark_data_smithi188_101370_object105836:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e295)", 2024-04-23T16:29:38.503 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:28:00.739761+0000", 2024-04-23T16:29:38.503 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 96.060160339000007, 2024-04-23T16:29:38.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.072974128999999999, 2024-04-23T16:29:38.504 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.504 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.504 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.504 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:38.504 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:38.504 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:38.504 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 105837 2024-04-23T16:29:38.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.504 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.739761+0000", 2024-04-23T16:29:38.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.739761+0000", 2024-04-23T16:29:38.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.739763+0000", 2024-04-23T16:29:38.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.387e-06 2024-04-23T16:29:38.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.739821+0000", 2024-04-23T16:29:38.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8560999999999997e-05 2024-04-23T16:29:38.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.739828+0000", 2024-04-23T16:29:38.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0439999999999998e-06 2024-04-23T16:29:38.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.739834+0000", 2024-04-23T16:29:38.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2499999999999997e-06 2024-04-23T16:29:38.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.746087+0000", 2024-04-23T16:29:38.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062534169999999998 2024-04-23T16:29:38.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.746226+0000", 2024-04-23T16:29:38.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139424 2024-04-23T16:29:38.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6,9", 2024-04-23T16:29:38.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.746418+0000", 2024-04-23T16:29:38.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000191285 2024-04-23T16:29:38.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.761617+0000", 2024-04-23T16:29:38.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01519969 2024-04-23T16:29:38.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:38.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.761634+0000", 2024-04-23T16:29:38.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6512e-05 2024-04-23T16:29:38.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.781755+0000", 2024-04-23T16:29:38.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020121061999999999 2024-04-23T16:29:38.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.781790+0000", 2024-04-23T16:29:38.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5370000000000002e-05 2024-04-23T16:29:38.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:00.812735+0000", 2024-04-23T16:29:38.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.030945126999999999 2024-04-23T16:29:38.509 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.509 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.509 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.509 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:106425 4.6 4:65e4c8b4:::benchmark_data_smithi188_101370_object106424:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e296)", 2024-04-23T16:29:38.509 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:28:01.156682+0000", 2024-04-23T16:29:38.510 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 95.643239863999995, 2024-04-23T16:29:38.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.071028697000000002, 2024-04-23T16:29:38.510 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.510 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.510 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.510 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:38.510 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:38.510 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:38.510 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 106425 2024-04-23T16:29:38.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.510 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:01.156682+0000", 2024-04-23T16:29:38.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:01.156682+0000", 2024-04-23T16:29:38.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:01.156683+0000", 2024-04-23T16:29:38.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.172e-06 2024-04-23T16:29:38.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:01.156767+0000", 2024-04-23T16:29:38.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4240999999999995e-05 2024-04-23T16:29:38.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:01.156775+0000", 2024-04-23T16:29:38.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8809999999999996e-06 2024-04-23T16:29:38.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:01.156780+0000", 2024-04-23T16:29:38.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2379999999999997e-06 2024-04-23T16:29:38.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:01.160052+0000", 2024-04-23T16:29:38.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0032714290000000002 2024-04-23T16:29:38.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:01.160189+0000", 2024-04-23T16:29:38.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013703700000000001 2024-04-23T16:29:38.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6,9", 2024-04-23T16:29:38.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:01.160344+0000", 2024-04-23T16:29:38.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000155468 2024-04-23T16:29:38.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:38.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:01.163549+0000", 2024-04-23T16:29:38.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0032047740000000001 2024-04-23T16:29:38.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:01.168563+0000", 2024-04-23T16:29:38.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.005013834 2024-04-23T16:29:38.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:01.225365+0000", 2024-04-23T16:29:38.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.056802436999999997 2024-04-23T16:29:38.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:01.225435+0000", 2024-04-23T16:29:38.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9487999999999996e-05 2024-04-23T16:29:38.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:01.227710+0000", 2024-04-23T16:29:38.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002275698 2024-04-23T16:29:38.515 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.515 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.515 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.516 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:114525 4.6 4:62835b99:::benchmark_data_smithi188_101370_object114524:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e298)", 2024-04-23T16:29:38.516 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:28:06.887553+0000", 2024-04-23T16:29:38.516 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 89.912369032000001, 2024-04-23T16:29:38.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.070106138999999998, 2024-04-23T16:29:38.516 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.516 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.516 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.516 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:38.516 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:38.516 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:38.516 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114525 2024-04-23T16:29:38.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.517 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:06.887553+0000", 2024-04-23T16:29:38.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:06.887553+0000", 2024-04-23T16:29:38.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:06.887555+0000", 2024-04-23T16:29:38.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5639999999999999e-06 2024-04-23T16:29:38.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:06.887602+0000", 2024-04-23T16:29:38.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7061999999999998e-05 2024-04-23T16:29:38.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:06.887612+0000", 2024-04-23T16:29:38.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0129000000000001e-05 2024-04-23T16:29:38.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:06.887618+0000", 2024-04-23T16:29:38.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7019999999999998e-06 2024-04-23T16:29:38.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:06.905195+0000", 2024-04-23T16:29:38.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017576784000000002 2024-04-23T16:29:38.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:06.905291+0000", 2024-04-23T16:29:38.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5680999999999994e-05 2024-04-23T16:29:38.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6,9", 2024-04-23T16:29:38.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:06.905394+0000", 2024-04-23T16:29:38.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010313 2024-04-23T16:29:38.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:06.911429+0000", 2024-04-23T16:29:38.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060357140000000002 2024-04-23T16:29:38.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:38.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:06.911448+0000", 2024-04-23T16:29:38.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8899e-05 2024-04-23T16:29:38.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:06.955537+0000", 2024-04-23T16:29:38.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.044088212000000002 2024-04-23T16:29:38.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:06.955581+0000", 2024-04-23T16:29:38.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4155e-05 2024-04-23T16:29:38.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:06.957659+0000", 2024-04-23T16:29:38.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002078107 2024-04-23T16:29:38.522 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.522 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.522 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.522 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:151765 4.6 4:6cc75f21:::benchmark_data_smithi188_101370_object151764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e311)", 2024-04-23T16:29:38.523 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:28:37.274498+0000", 2024-04-23T16:29:38.523 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.52542416, 2024-04-23T16:29:38.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.070695552999999994, 2024-04-23T16:29:38.523 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.523 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.523 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.523 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:38.523 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:38.523 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:38.523 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 151765 2024-04-23T16:29:38.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.523 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:37.274498+0000", 2024-04-23T16:29:38.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:37.274498+0000", 2024-04-23T16:29:38.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:37.274499+0000", 2024-04-23T16:29:38.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6819999999999999e-06 2024-04-23T16:29:38.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:37.274565+0000", 2024-04-23T16:29:38.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6146e-05 2024-04-23T16:29:38.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:37.274575+0000", 2024-04-23T16:29:38.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3899999999999999e-06 2024-04-23T16:29:38.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:37.274581+0000", 2024-04-23T16:29:38.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1809999999999998e-06 2024-04-23T16:29:38.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:37.284159+0000", 2024-04-23T16:29:38.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0095776249999999993 2024-04-23T16:29:38.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:37.284269+0000", 2024-04-23T16:29:38.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110207 2024-04-23T16:29:38.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6,9", 2024-04-23T16:29:38.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:37.284380+0000", 2024-04-23T16:29:38.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000111168 2024-04-23T16:29:38.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:37.293977+0000", 2024-04-23T16:29:38.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0095973819999999998 2024-04-23T16:29:38.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:38.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:37.293995+0000", 2024-04-23T16:29:38.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8138000000000002e-05 2024-04-23T16:29:38.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:37.343067+0000", 2024-04-23T16:29:38.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.049071686000000003 2024-04-23T16:29:38.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:37.343109+0000", 2024-04-23T16:29:38.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1464000000000001e-05 2024-04-23T16:29:38.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:37.345193+0000", 2024-04-23T16:29:38.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020844840000000002 2024-04-23T16:29:38.529 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.529 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.529 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.529 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:157513 4.6 4:616d49ef:::benchmark_data_smithi188_101370_object157512:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e314)", 2024-04-23T16:29:38.529 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:28:42.360467+0000", 2024-04-23T16:29:38.529 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 54.439454867999999, 2024-04-23T16:29:38.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.067735152000000007, 2024-04-23T16:29:38.529 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.529 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.529 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.529 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:38.529 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:38.530 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:38.530 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 157513 2024-04-23T16:29:38.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.530 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.360467+0000", 2024-04-23T16:29:38.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.360467+0000", 2024-04-23T16:29:38.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.360467+0000", 2024-04-23T16:29:38.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2300000000000001e-07 2024-04-23T16:29:38.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.360535+0000", 2024-04-23T16:29:38.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7404999999999996e-05 2024-04-23T16:29:38.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.360544+0000", 2024-04-23T16:29:38.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1479999999999995e-06 2024-04-23T16:29:38.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.360552+0000", 2024-04-23T16:29:38.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7530000000000001e-06 2024-04-23T16:29:38.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.368541+0000", 2024-04-23T16:29:38.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079896700000000008 2024-04-23T16:29:38.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.368696+0000", 2024-04-23T16:29:38.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000154575 2024-04-23T16:29:38.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6,9", 2024-04-23T16:29:38.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.368874+0000", 2024-04-23T16:29:38.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017816699999999999 2024-04-23T16:29:38.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:38.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.384903+0000", 2024-04-23T16:29:38.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016028754999999999 2024-04-23T16:29:38.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.384942+0000", 2024-04-23T16:29:38.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9483000000000001e-05 2024-04-23T16:29:38.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.426076+0000", 2024-04-23T16:29:38.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.041133282 2024-04-23T16:29:38.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.426142+0000", 2024-04-23T16:29:38.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6291000000000006e-05 2024-04-23T16:29:38.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.428202+0000", 2024-04-23T16:29:38.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020600000000000002 2024-04-23T16:29:38.535 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.535 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.535 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.535 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:157516 4.6 4:6b6d9de3:::benchmark_data_smithi188_101370_object157515:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e314)", 2024-04-23T16:29:38.535 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:28:42.360877+0000", 2024-04-23T16:29:38.535 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 54.439045061999998, 2024-04-23T16:29:38.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.068662817000000001, 2024-04-23T16:29:38.535 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.535 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.535 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.536 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:38.536 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:38.536 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:38.536 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 157516 2024-04-23T16:29:38.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.536 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.360877+0000", 2024-04-23T16:29:38.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.360877+0000", 2024-04-23T16:29:38.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.360878+0000", 2024-04-23T16:29:38.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.204e-06 2024-04-23T16:29:38.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.360954+0000", 2024-04-23T16:29:38.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6639999999999998e-05 2024-04-23T16:29:38.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.360963+0000", 2024-04-23T16:29:38.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0240000000000003e-06 2024-04-23T16:29:38.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.360970+0000", 2024-04-23T16:29:38.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4309999999999999e-06 2024-04-23T16:29:38.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.375507+0000", 2024-04-23T16:29:38.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014537179000000001 2024-04-23T16:29:38.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.375615+0000", 2024-04-23T16:29:38.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107773 2024-04-23T16:29:38.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6,9", 2024-04-23T16:29:38.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.375727+0000", 2024-04-23T16:29:38.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011243600000000001 2024-04-23T16:29:38.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.384982+0000", 2024-04-23T16:29:38.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0092549190000000003 2024-04-23T16:29:38.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:38.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.384992+0000", 2024-04-23T16:29:38.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5689999999999994e-06 2024-04-23T16:29:38.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.428259+0000", 2024-04-23T16:29:38.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.043267242999999997 2024-04-23T16:29:38.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.428299+0000", 2024-04-23T16:29:38.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0132999999999997e-05 2024-04-23T16:29:38.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:42.429539+0000", 2024-04-23T16:29:38.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012402660000000001 2024-04-23T16:29:38.541 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.541 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.541 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.541 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:158996 4.6 4:6a18d6a1:::benchmark_data_smithi188_101370_object158995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e314)", 2024-04-23T16:29:38.541 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:28:43.767982+0000", 2024-04-23T16:29:38.541 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 53.031939749999999, 2024-04-23T16:29:38.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.067467888000000004, 2024-04-23T16:29:38.542 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.542 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.542 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.542 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:38.542 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:38.542 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:38.542 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 158996 2024-04-23T16:29:38.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.542 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:43.767982+0000", 2024-04-23T16:29:38.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:43.767982+0000", 2024-04-23T16:29:38.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:43.767983+0000", 2024-04-23T16:29:38.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.172e-06 2024-04-23T16:29:38.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:43.768070+0000", 2024-04-23T16:29:38.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7096999999999996e-05 2024-04-23T16:29:38.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:43.768084+0000", 2024-04-23T16:29:38.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3563999999999999e-05 2024-04-23T16:29:38.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:43.768091+0000", 2024-04-23T16:29:38.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9800000000000001e-06 2024-04-23T16:29:38.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:43.780879+0000", 2024-04-23T16:29:38.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012788532 2024-04-23T16:29:38.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:43.781030+0000", 2024-04-23T16:29:38.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015102099999999999 2024-04-23T16:29:38.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6,9", 2024-04-23T16:29:38.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:43.781199+0000", 2024-04-23T16:29:38.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000168703 2024-04-23T16:29:38.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:43.788771+0000", 2024-04-23T16:29:38.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075721190000000004 2024-04-23T16:29:38.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:38.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:43.792355+0000", 2024-04-23T16:29:38.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003583475 2024-04-23T16:29:38.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:43.833427+0000", 2024-04-23T16:29:38.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.041072601 2024-04-23T16:29:38.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:43.833471+0000", 2024-04-23T16:29:38.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3491000000000001e-05 2024-04-23T16:29:38.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:43.835450+0000", 2024-04-23T16:29:38.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019791330000000001 2024-04-23T16:29:38.547 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.547 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.547 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.548 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:160038 4.6 4:6c16be6e:::benchmark_data_smithi188_101370_object160037:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e315)", 2024-04-23T16:29:38.548 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:28:44.741866+0000", 2024-04-23T16:29:38.548 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 52.058055299999999, 2024-04-23T16:29:38.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.068501461999999999, 2024-04-23T16:29:38.548 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.548 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.548 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.548 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:38.548 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:38.548 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:38.548 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 160038 2024-04-23T16:29:38.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.548 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.741866+0000", 2024-04-23T16:29:38.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.741866+0000", 2024-04-23T16:29:38.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.741867+0000", 2024-04-23T16:29:38.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7299999999999995e-07 2024-04-23T16:29:38.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.741940+0000", 2024-04-23T16:29:38.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2727999999999999e-05 2024-04-23T16:29:38.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.741951+0000", 2024-04-23T16:29:38.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0825000000000001e-05 2024-04-23T16:29:38.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.741959+0000", 2024-04-23T16:29:38.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1249999999999993e-06 2024-04-23T16:29:38.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.760349+0000", 2024-04-23T16:29:38.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018389895 2024-04-23T16:29:38.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.760506+0000", 2024-04-23T16:29:38.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157057 2024-04-23T16:29:38.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6,9", 2024-04-23T16:29:38.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.760656+0000", 2024-04-23T16:29:38.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000149877 2024-04-23T16:29:38.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.764574+0000", 2024-04-23T16:29:38.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039185219999999998 2024-04-23T16:29:38.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:38.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.767891+0000", 2024-04-23T16:29:38.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0033172589999999999 2024-04-23T16:29:38.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.808369+0000", 2024-04-23T16:29:38.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.040478082999999998 2024-04-23T16:29:38.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.808429+0000", 2024-04-23T16:29:38.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9363000000000001e-05 2024-04-23T16:29:38.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.810368+0000", 2024-04-23T16:29:38.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019390550000000001 2024-04-23T16:29:38.553 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.553 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.554 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.554 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:160044 4.6 4:69cb8e31:::benchmark_data_smithi188_101370_object160043:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e315)", 2024-04-23T16:29:38.554 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:28:44.746682+0000", 2024-04-23T16:29:38.554 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 52.053240062, 2024-04-23T16:29:38.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.072274601999999993, 2024-04-23T16:29:38.554 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.554 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.554 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.554 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:38.554 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:38.554 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:38.555 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 160044 2024-04-23T16:29:38.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.555 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.746682+0000", 2024-04-23T16:29:38.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.746682+0000", 2024-04-23T16:29:38.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.746683+0000", 2024-04-23T16:29:38.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.575e-06 2024-04-23T16:29:38.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.746739+0000", 2024-04-23T16:29:38.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6104999999999999e-05 2024-04-23T16:29:38.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.746751+0000", 2024-04-23T16:29:38.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1989e-05 2024-04-23T16:29:38.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.746760+0000", 2024-04-23T16:29:38.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6689999999999995e-06 2024-04-23T16:29:38.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.767921+0000", 2024-04-23T16:29:38.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.021161276 2024-04-23T16:29:38.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.768016+0000", 2024-04-23T16:29:38.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4368000000000002e-05 2024-04-23T16:29:38.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6,9", 2024-04-23T16:29:38.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.768124+0000", 2024-04-23T16:29:38.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108329 2024-04-23T16:29:38.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:38.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.778226+0000", 2024-04-23T16:29:38.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010101943 2024-04-23T16:29:38.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.778299+0000", 2024-04-23T16:29:38.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2733999999999999e-05 2024-04-23T16:29:38.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.816770+0000", 2024-04-23T16:29:38.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.038471565999999999 2024-04-23T16:29:38.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.816805+0000", 2024-04-23T16:29:38.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4774000000000003e-05 2024-04-23T16:29:38.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:44.818956+0000", 2024-04-23T16:29:38.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0021512739999999999 2024-04-23T16:29:38.560 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.560 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.560 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.560 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:161332 4.6 4:60c5939c:::benchmark_data_smithi188_101370_object161331:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e315)", 2024-04-23T16:29:38.560 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:28:45.897581+0000", 2024-04-23T16:29:38.560 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 50.902340500999998, 2024-04-23T16:29:38.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.069837516000000002, 2024-04-23T16:29:38.560 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.560 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.560 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.560 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:38.561 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:38.561 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:38.561 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 161332 2024-04-23T16:29:38.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.561 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:45.897581+0000", 2024-04-23T16:29:38.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:45.897581+0000", 2024-04-23T16:29:38.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:45.897582+0000", 2024-04-23T16:29:38.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4700000000000001e-07 2024-04-23T16:29:38.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:45.897607+0000", 2024-04-23T16:29:38.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5177999999999998e-05 2024-04-23T16:29:38.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:45.897613+0000", 2024-04-23T16:29:38.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6540000000000004e-06 2024-04-23T16:29:38.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:45.897616+0000", 2024-04-23T16:29:38.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.766e-06 2024-04-23T16:29:38.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:45.920706+0000", 2024-04-23T16:29:38.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.023089657 2024-04-23T16:29:38.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:45.920808+0000", 2024-04-23T16:29:38.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010162600000000001 2024-04-23T16:29:38.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6,9", 2024-04-23T16:29:38.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:45.920918+0000", 2024-04-23T16:29:38.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110858 2024-04-23T16:29:38.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:38.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:45.934640+0000", 2024-04-23T16:29:38.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013721975000000001 2024-04-23T16:29:38.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:45.934688+0000", 2024-04-23T16:29:38.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7280000000000001e-05 2024-04-23T16:29:38.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:45.965611+0000", 2024-04-23T16:29:38.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.030923346000000001 2024-04-23T16:29:38.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:45.965649+0000", 2024-04-23T16:29:38.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7648999999999998e-05 2024-04-23T16:29:38.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:28:45.967419+0000", 2024-04-23T16:29:38.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00176998 2024-04-23T16:29:38.566 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.566 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.566 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.566 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:178452 4.4 4:22d51dae:::benchmark_data_smithi188_101370_object15003:head [delete] snapc 0=[] ondisk+write+known_if_redirected e321)", 2024-04-23T16:29:38.566 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:29:04.002816+0000", 2024-04-23T16:29:38.566 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.797106046000003, 2024-04-23T16:29:38.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.082418783999999995, 2024-04-23T16:29:38.567 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.567 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.567 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.567 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:38.567 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:38.567 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:38.567 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 178452 2024-04-23T16:29:38.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.567 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.002816+0000", 2024-04-23T16:29:38.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.002816+0000", 2024-04-23T16:29:38.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.002817+0000", 2024-04-23T16:29:38.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8020000000000001e-06 2024-04-23T16:29:38.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.002823+0000", 2024-04-23T16:29:38.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6080000000000003e-06 2024-04-23T16:29:38.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.002833+0000", 2024-04-23T16:29:38.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0412e-05 2024-04-23T16:29:38.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.002842+0000", 2024-04-23T16:29:38.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1659999999999994e-06 2024-04-23T16:29:38.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.054679+0000", 2024-04-23T16:29:38.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.051836947000000001 2024-04-23T16:29:38.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.054818+0000", 2024-04-23T16:29:38.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013972499999999999 2024-04-23T16:29:38.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:29:38.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.054951+0000", 2024-04-23T16:29:38.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000132736 2024-04-23T16:29:38.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.072144+0000", 2024-04-23T16:29:38.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017193322 2024-04-23T16:29:38.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:38.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.085134+0000", 2024-04-23T16:29:38.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012989469 2024-04-23T16:29:38.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.085171+0000", 2024-04-23T16:29:38.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7013e-05 2024-04-23T16:29:38.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.085234+0000", 2024-04-23T16:29:38.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3584000000000007e-05 2024-04-23T16:29:38.572 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.572 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.572 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.572 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:178454 4.4 4:21559ea0:::benchmark_data_smithi188_101370_object15005:head [delete] snapc 0=[] ondisk+write+known_if_redirected e321)", 2024-04-23T16:29:38.572 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:29:04.003418+0000", 2024-04-23T16:29:38.572 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.796503436999998, 2024-04-23T16:29:38.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.082372069000000006, 2024-04-23T16:29:38.572 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.573 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.573 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.573 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:38.573 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:38.573 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:38.573 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 178454 2024-04-23T16:29:38.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.573 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.003418+0000", 2024-04-23T16:29:38.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.003418+0000", 2024-04-23T16:29:38.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.003419+0000", 2024-04-23T16:29:38.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1599999999999999e-06 2024-04-23T16:29:38.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.003423+0000", 2024-04-23T16:29:38.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8190000000000002e-06 2024-04-23T16:29:38.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.054539+0000", 2024-04-23T16:29:38.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.051115756999999998 2024-04-23T16:29:38.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.054557+0000", 2024-04-23T16:29:38.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7693000000000002e-05 2024-04-23T16:29:38.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.067202+0000", 2024-04-23T16:29:38.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012645682 2024-04-23T16:29:38.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.067335+0000", 2024-04-23T16:29:38.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013255400000000001 2024-04-23T16:29:38.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:29:38.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.067479+0000", 2024-04-23T16:29:38.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000144191 2024-04-23T16:29:38.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:38.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.085247+0000", 2024-04-23T16:29:38.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017768254000000001 2024-04-23T16:29:38.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.085723+0000", 2024-04-23T16:29:38.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000475561 2024-04-23T16:29:38.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.085755+0000", 2024-04-23T16:29:38.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1798000000000003e-05 2024-04-23T16:29:38.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:04.085790+0000", 2024-04-23T16:29:38.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5599999999999998e-05 2024-04-23T16:29:38.578 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.578 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.578 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.578 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:190408 4.4 4:28566a71:::benchmark_data_smithi188_101370_object26959:head [delete] snapc 0=[] ondisk+write+known_if_redirected e323)", 2024-04-23T16:29:38.578 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:29:13.629341+0000", 2024-04-23T16:29:38.578 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.170580380000001, 2024-04-23T16:29:38.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.071825427999999997, 2024-04-23T16:29:38.578 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.578 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.578 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.578 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:38.578 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:38.579 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:38.579 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 190408 2024-04-23T16:29:38.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.579 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.629341+0000", 2024-04-23T16:29:38.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.629341+0000", 2024-04-23T16:29:38.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.629343+0000", 2024-04-23T16:29:38.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9080000000000002e-06 2024-04-23T16:29:38.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.629348+0000", 2024-04-23T16:29:38.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2410000000000003e-06 2024-04-23T16:29:38.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.629359+0000", 2024-04-23T16:29:38.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0417999999999999e-05 2024-04-23T16:29:38.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.629366+0000", 2024-04-23T16:29:38.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6329999999999996e-06 2024-04-23T16:29:38.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.632947+0000", 2024-04-23T16:29:38.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035803319999999999 2024-04-23T16:29:38.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.633061+0000", 2024-04-23T16:29:38.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114622 2024-04-23T16:29:38.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:29:38.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.633168+0000", 2024-04-23T16:29:38.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000106283 2024-04-23T16:29:38.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:38.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.641140+0000", 2024-04-23T16:29:38.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079722049999999996 2024-04-23T16:29:38.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.701085+0000", 2024-04-23T16:29:38.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.059945270000000002 2024-04-23T16:29:38.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.701121+0000", 2024-04-23T16:29:38.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6146999999999998e-05 2024-04-23T16:29:38.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.701167+0000", 2024-04-23T16:29:38.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5368999999999999e-05 2024-04-23T16:29:38.583 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.584 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.584 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.584 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15486.0:190414 4.4 4:23f14902:::benchmark_data_smithi188_101370_object26965:head [delete] snapc 0=[] ondisk+write+known_if_redirected e323)", 2024-04-23T16:29:38.584 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:29:13.634376+0000", 2024-04-23T16:29:38.584 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.165546124999999, 2024-04-23T16:29:38.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.077138793999999997, 2024-04-23T16:29:38.584 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:29:38.584 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:29:38.584 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:29:38.584 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:29:38.584 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15486", 2024-04-23T16:29:38.584 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:29:38.584 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 190414 2024-04-23T16:29:38.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.585 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:29:38.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:29:38.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.634376+0000", 2024-04-23T16:29:38.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:29:38.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.634376+0000", 2024-04-23T16:29:38.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:29:38.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:29:38.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.634377+0000", 2024-04-23T16:29:38.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.35e-06 2024-04-23T16:29:38.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:29:38.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.634382+0000", 2024-04-23T16:29:38.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1340000000000003e-06 2024-04-23T16:29:38.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:29:38.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.634390+0000", 2024-04-23T16:29:38.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9810000000000003e-06 2024-04-23T16:29:38.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:29:38.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.634395+0000", 2024-04-23T16:29:38.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2669999999999997e-06 2024-04-23T16:29:38.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:29:38.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.637820+0000", 2024-04-23T16:29:38.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034242439999999999 2024-04-23T16:29:38.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:29:38.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.637891+0000", 2024-04-23T16:29:38.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1106999999999995e-05 2024-04-23T16:29:38.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-23T16:29:38.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.637956+0000", 2024-04-23T16:29:38.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4900000000000005e-05 2024-04-23T16:29:38.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:29:38.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.641291+0000", 2024-04-23T16:29:38.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0033352669999999998 2024-04-23T16:29:38.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:29:38.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.711407+0000", 2024-04-23T16:29:38.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.070116204000000001 2024-04-23T16:29:38.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:29:38.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.711468+0000", 2024-04-23T16:29:38.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0739999999999998e-05 2024-04-23T16:29:38.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:29:38.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:29:38.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:29:38.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:29:13.711514+0000", 2024-04-23T16:29:38.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6600000000000001e-05 2024-04-23T16:29:38.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.589 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:29:38.589 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:29:38.589 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:29:38.657 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:29:38.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:38 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:29:38.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:38 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:29:38.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:38 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:29:38.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:38 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:29:38.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:38 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:29:38.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:38 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:29:39.802 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:39 smithi103 ceph-mon[39629]: osdmap e331: 12 total, 12 up, 10 in 2024-04-23T16:29:39.802 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:39 smithi103 ceph-mon[39629]: pgmap v933: 27 pgs: 27 active+clean; 6.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 982 op/s 2024-04-23T16:29:39.825 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:29:39.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:39.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:29:39.832 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-23T16:29:39.930 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:29:39.971 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:39 smithi188 ceph-mon[44101]: osdmap e331: 12 total, 12 up, 10 in 2024-04-23T16:29:39.971 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:39 smithi188 ceph-mon[44101]: pgmap v933: 27 pgs: 27 active+clean; 6.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 982 op/s 2024-04-23T16:29:39.974 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:40.103 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:29:40.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:40.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:29:40.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:39 smithi148 ceph-mon[44520]: osdmap e331: 12 total, 12 up, 10 in 2024-04-23T16:29:40.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:39 smithi148 ceph-mon[44520]: pgmap v933: 27 pgs: 27 active+clean; 6.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 982 op/s 2024-04-23T16:29:40.157 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:29:40.188 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:29:40.214 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:29:40.216 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:29:40.243 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:29:40.245 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:29:40.266 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:40.285 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:29:40.299 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:29:40.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:40.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:29:40.309 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:29:40.344 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:29:40.375 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:29:40.407 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:29:40.408 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:29:40.441 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:29:40.468 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:40.473 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:29:40.506 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:29:40.508 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:29:40.508 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:29:40.508 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:29:40.508 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:29:40.508 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:29:40.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:29:40.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:29:40.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:29:40.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:29:40.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:29:40.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:29:40.509 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:29:40.580 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:29:40.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:40.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:29:40.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:40 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2172518882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:29:40.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:40.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:40.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:40.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:40.686 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:29:40.716 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:40 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2172518882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:29:40.716 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:40.716 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:40.716 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:40.716 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:40.718 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:40.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:40 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2172518882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:29:40.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:40.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:40.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:40.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:40.785 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:29:40.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:40.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:29:40.807 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:29:40.808 INFO:teuthology.orchestra.run.smithi188.stdout: "success": "" 2024-04-23T16:29:40.808 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:29:40.889 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:40.929 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:41.177 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:29:41.632 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:41 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:41.633 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:41 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:41.633 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:41 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:41.633 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:41 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:41.633 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:41 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:41.633 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:41 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:41.633 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:41 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:41.633 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:41 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:41.633 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:41 smithi188 ceph-mon[44101]: pgmap v934: 27 pgs: 27 active+clean; 6.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.38k op/s 2024-04-23T16:29:41.782 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:41.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:41.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:41.890 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:29:41.920 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:41.946 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:29:41.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:41.949 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:29:42.050 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:29:42.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:41 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:42.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:41 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:42.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:41 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:42.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:41 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:42.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:41 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:42.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:41 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:42.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:41 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:42.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:41 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:42.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:41 smithi103 ceph-mon[39629]: pgmap v934: 27 pgs: 27 active+clean; 6.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.38k op/s 2024-04-23T16:29:42.080 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:42.105 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:29:42.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:42.110 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:29:42.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:41 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:42.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:41 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:42.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:41 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:42.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:41 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:42.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:41 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:42.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:41 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:42.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:41 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:42.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:41 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:42.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:41 smithi148 ceph-mon[44520]: pgmap v934: 27 pgs: 27 active+clean; 6.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.38k op/s 2024-04-23T16:29:42.210 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:29:42.244 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:42.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:42 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:29:42.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:42 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:29:42.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:42 smithi188 ceph-mon[44101]: osdmap e332: 12 total, 12 up, 10 in 2024-04-23T16:29:43.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:42 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:29:43.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:42 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:29:43.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:42 smithi103 ceph-mon[39629]: osdmap e332: 12 total, 12 up, 10 in 2024-04-23T16:29:43.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:42 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:29:43.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:42 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:29:43.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:42 smithi148 ceph-mon[44520]: osdmap e332: 12 total, 12 up, 10 in 2024-04-23T16:29:43.866 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:29:43.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:43 smithi188 ceph-mon[44101]: pgmap v936: 27 pgs: 27 active+clean; 6.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.32k op/s 2024-04-23T16:29:44.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:43 smithi103 ceph-mon[39629]: pgmap v936: 27 pgs: 27 active+clean; 6.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.32k op/s 2024-04-23T16:29:44.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:43 smithi148 ceph-mon[44520]: pgmap v936: 27 pgs: 27 active+clean; 6.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.32k op/s 2024-04-23T16:29:45.243 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:29:45.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:45.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:29:45.255 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:29:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:45 smithi103 ceph-mon[39629]: pgmap v937: 27 pgs: 27 active+clean; 6.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.29k op/s 2024-04-23T16:29:45.353 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:45.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:45 smithi148 ceph-mon[44520]: pgmap v937: 27 pgs: 27 active+clean; 6.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.29k op/s 2024-04-23T16:29:45.382 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:45 smithi188 ceph-mon[44101]: pgmap v937: 27 pgs: 27 active+clean; 6.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.29k op/s 2024-04-23T16:29:45.384 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:45.409 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:45.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:45.414 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:45.515 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:29:45.561 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:45.593 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:29:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:45.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:29:45.699 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:29:45.734 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:46.752 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10, 1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1] 2024-04-23T16:29:46.752 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:29:46.752 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-04-23T16:29:46.752 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-23T16:29:46.753 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-23T16:29:46.753 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-04-23T16:29:47.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:47 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:29:47.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:47 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:29:47.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:47 smithi188 ceph-mon[44101]: osdmap e333: 12 total, 12 up, 10 in 2024-04-23T16:29:47.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:47 smithi188 ceph-mon[44101]: pgmap v939: 27 pgs: 27 active+clean; 5.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.59k op/s 2024-04-23T16:29:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:47 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:29:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:47 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:29:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:47 smithi103 ceph-mon[39629]: osdmap e333: 12 total, 12 up, 10 in 2024-04-23T16:29:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:47 smithi103 ceph-mon[39629]: pgmap v939: 27 pgs: 27 active+clean; 5.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.59k op/s 2024-04-23T16:29:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:47 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:29:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:47 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:29:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:47 smithi148 ceph-mon[44520]: osdmap e333: 12 total, 12 up, 10 in 2024-04-23T16:29:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:47 smithi148 ceph-mon[44520]: pgmap v939: 27 pgs: 27 active+clean; 5.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.59k op/s 2024-04-23T16:29:48.020 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:29:48.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:48.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:29:48.195 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:29:48.230 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:48.387 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:29:48.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:48.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:29:48.491 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:48.505 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:29:48.525 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:48.549 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:48.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:48.553 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:48.561 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:29:48.561 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:29:48.654 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:29:48.707 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:48.742 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:29:48.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:48.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:29:48.846 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:29:48.900 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:48.932 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:29:48.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:48.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:29:49.038 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:29:49.075 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:49.099 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:29:49.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:49.107 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:29:49.209 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:29:49.244 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:49.326 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:29:49.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:49.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:29:49.430 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:29:49.462 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:49.919 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:29:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:49.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:29:49.956 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-23T16:29:49.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:49 smithi188 ceph-mon[44101]: pgmap v940: 27 pgs: 27 active+clean; 5.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 988 op/s 2024-04-23T16:29:50.026 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:29:50.027 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:49 smithi103 ceph-mon[39629]: pgmap v940: 27 pgs: 27 active+clean; 5.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 988 op/s 2024-04-23T16:29:50.064 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:49 smithi148 ceph-mon[44520]: pgmap v940: 27 pgs: 27 active+clean; 5.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 988 op/s 2024-04-23T16:29:50.268 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:29:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:50.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:29:50.375 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:50.405 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:50.428 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:50.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:50.435 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:50.536 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:29:50.564 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:51.412 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:29:51.424 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:29:51.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:51.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:29:51.532 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:29:51.571 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:51.606 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:29:51.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:51.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:29:51.714 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:51.723 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:51 smithi103 ceph-mon[39629]: pgmap v941: 27 pgs: 27 active+clean; 5.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.41k op/s 2024-04-23T16:29:51.750 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:51.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:51.775 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:51.775 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:51.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:51 smithi148 ceph-mon[44520]: pgmap v941: 27 pgs: 27 active+clean; 5.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.41k op/s 2024-04-23T16:29:51.876 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:29:51.906 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:51.934 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:29:51.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:51.937 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:29:51.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:51 smithi188 ceph-mon[44101]: pgmap v941: 27 pgs: 27 active+clean; 5.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.41k op/s 2024-04-23T16:29:52.037 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:29:52.082 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:52.117 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:29:52.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:52.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:29:52.229 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:29:52.272 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:52.303 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:29:52.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:52.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:29:52.415 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:29:52.454 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:52.772 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:29:52.794 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:29:52.809 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:29:52.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:52.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:29:52.918 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:29:52.968 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:53.009 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:29:53.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:53.023 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:52 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:29:53.023 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:52 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:29:53.023 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:52 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1229216630' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:29:53.023 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:52 smithi148 ceph-mon[44520]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:29:53.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:29:53.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:52 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:29:53.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:52 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:29:53.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:52 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1229216630' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:29:53.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:52 smithi103 ceph-mon[39629]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:29:53.125 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:29:53.182 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:52 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:29:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:52 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:29:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:52 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1229216630' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:29:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:52 smithi188 ceph-mon[44101]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:29:53.242 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:29:53.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:53.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:29:53.349 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:53.377 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:53.403 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:53.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:53.410 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:53.511 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:29:53.567 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:53.590 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:29:53.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:53.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:29:53.696 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:29:53.731 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:53.765 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:29:53.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:53.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:29:53.876 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:29:53.879 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:29:53.879 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:29:53.879 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:29:53.879 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:29:53.880 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:29:53.880 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:29:53.880 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:29:53.880 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:29:53.881 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:53 smithi148 ceph-mon[44520]: pgmap v942: 27 pgs: 27 active+clean; 5.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-23T16:29:53.881 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:53 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:29:53.881 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:53 smithi148 ceph-mon[44520]: from='client.25528 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:29:53.881 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:53 smithi148 ceph-mon[44520]: osdmap e334: 12 total, 12 up, 10 in 2024-04-23T16:29:53.881 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:53 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/768222302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:29:53.895 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:29:53.911 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:53.926 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:29:53.936 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:29:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:53.941 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:29:53.949 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:29:53.979 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:29:53.980 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:29:53.980 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:29:53.981 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:29:53.981 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:29:53.981 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:29:53.981 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:29:53.981 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:29:53.981 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:29:53.981 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:29:53.981 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:29:53.981 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:29:53.981 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:29:54.008 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:53 smithi103 ceph-mon[39629]: pgmap v942: 27 pgs: 27 active+clean; 5.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-23T16:29:54.008 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:53 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:29:54.008 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:53 smithi103 ceph-mon[39629]: from='client.25528 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:29:54.008 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:53 smithi103 ceph-mon[39629]: osdmap e334: 12 total, 12 up, 10 in 2024-04-23T16:29:54.008 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:53 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/768222302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:29:54.042 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:29:54.068 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:53 smithi188 ceph-mon[44101]: pgmap v942: 27 pgs: 27 active+clean; 5.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-23T16:29:54.068 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:53 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:29:54.068 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:53 smithi188 ceph-mon[44101]: from='client.25528 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:29:54.069 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:53 smithi188 ceph-mon[44101]: osdmap e334: 12 total, 12 up, 10 in 2024-04-23T16:29:54.069 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:53 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/768222302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:29:54.070 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:54.098 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:29:54.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:54.101 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:29:54.202 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:29:54.233 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:54.263 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:29:54.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:54.269 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:29:54.370 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:29:54.407 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:54.450 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:29:54.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:54.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:29:54.554 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:29:54.620 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:54.643 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:29:54.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:54.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:29:54.749 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:29:54.780 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:54.806 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:29:54.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:54.812 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:29:54.913 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:29:54.950 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:54.983 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:29:54.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:54.988 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:29:55.085 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.085 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:55 smithi188 ceph-mon[44101]: 4.8 scrub starts 2024-04-23T16:29:55.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:55 smithi188 ceph-mon[44101]: pgmap v944: 27 pgs: 27 active+clean; 5.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.44k op/s 2024-04-23T16:29:55.089 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:29:55.134 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:55.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:55.181 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:29:55.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:29:55.282 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:29:55.289 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.289 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.289 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.289 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.289 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.289 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.289 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.289 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.289 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.290 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.290 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.290 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.290 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:55 smithi103 ceph-mon[39629]: 4.8 scrub starts 2024-04-23T16:29:55.290 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:55 smithi103 ceph-mon[39629]: pgmap v944: 27 pgs: 27 active+clean; 5.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.44k op/s 2024-04-23T16:29:55.330 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:55.361 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:29:55.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:55.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:29:55.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.366 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.366 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.366 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.366 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.366 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.366 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.366 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.366 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.369 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.369 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.369 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:29:55.370 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:55 smithi148 ceph-mon[44520]: 4.8 scrub starts 2024-04-23T16:29:55.370 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:55 smithi148 ceph-mon[44520]: pgmap v944: 27 pgs: 27 active+clean; 5.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.44k op/s 2024-04-23T16:29:55.466 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:29:55.526 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:55.557 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:29:55.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:55.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:29:55.662 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:29:55.697 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:55.958 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:29:55.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:55.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:29:56.062 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:56.097 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:56.132 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:56.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:56.135 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:56.236 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:29:56.299 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:56.329 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:29:56.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:56.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:29:56.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:29:56.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:56 smithi188 ceph-mon[44101]: 4.f scrub starts 2024-04-23T16:29:56.479 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:56 smithi103 ceph-mon[39629]: 4.f scrub starts 2024-04-23T16:29:56.481 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:56.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:56 smithi148 ceph-mon[44520]: 4.f scrub starts 2024-04-23T16:29:57.029 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:29:57.050 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:29:57.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:57.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:29:57.155 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:29:57.201 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:57.228 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:29:57.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:57.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:29:57.332 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:29:57.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:57 smithi103 ceph-mon[39629]: 4.8 scrub ok 2024-04-23T16:29:57.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:57 smithi103 ceph-mon[39629]: pgmap v945: 27 pgs: 27 active+clean; 5.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:29:57.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:57 smithi103 ceph-mon[39629]: 4.f scrub ok 2024-04-23T16:29:57.365 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:57.388 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:29:57.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:57.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:29:57.493 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:29:57.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:57 smithi188 ceph-mon[44101]: 4.8 scrub ok 2024-04-23T16:29:57.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:57 smithi188 ceph-mon[44101]: pgmap v945: 27 pgs: 27 active+clean; 5.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:29:57.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:57 smithi188 ceph-mon[44101]: 4.f scrub ok 2024-04-23T16:29:57.526 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:57.548 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:29:57.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:57.555 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:29:57.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:57 smithi148 ceph-mon[44520]: 4.8 scrub ok 2024-04-23T16:29:57.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:57 smithi148 ceph-mon[44520]: pgmap v945: 27 pgs: 27 active+clean; 5.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:29:57.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:57 smithi148 ceph-mon[44520]: 4.f scrub ok 2024-04-23T16:29:57.655 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:29:57.709 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:57.722 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:29:57.722 INFO:teuthology.orchestra.run.smithi103.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-23T16:29:57.722 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:29:57.970 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:29:57.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:57.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:29:58.076 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:29:58.110 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:58.569 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:29:58.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:58.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:29:58.597 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:29:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:29:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-23T16:29:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-23T16:29:58.597 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:29:58.676 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:58.697 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:29:58.710 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:58.738 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:58.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:58.742 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:58.843 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:58.872 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:58.900 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:58.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:58.905 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:59.006 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:29:59.035 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:59.066 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:29:59.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:59.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:29:59.172 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:29:59.202 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:59.229 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:29:59.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:59.233 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:29:59.334 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:29:59.362 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:59.385 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:29:59.387 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:29:59.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:59.390 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:29:59.490 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:29:59.523 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:59.558 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:29:59.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:59.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:29:59.672 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:29:59.704 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:59.729 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:29:59.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:59.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:29:59.834 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:29:59.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:29:59 smithi103 ceph-mon[39629]: pgmap v946: 27 pgs: 27 active+clean; 5.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:29:59.864 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:29:59.885 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:29:59.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:29:59.888 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:29:59.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:29:59 smithi188 ceph-mon[44101]: pgmap v946: 27 pgs: 27 active+clean; 5.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:29:59.990 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:30:00.019 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:00.043 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:30:00.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:00.051 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:30:00.073 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-23T16:30:00.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:29:59 smithi148 ceph-mon[44520]: pgmap v946: 27 pgs: 27 active+clean; 5.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:30:00.152 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:30:00.196 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:00.306 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:30:00.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:00.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:30:00.420 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:30:00.481 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:00.616 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:30:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:00.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:30:00.722 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:30:00.758 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:00.785 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:30:00.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:00.790 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:30:00.892 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:30:00.921 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:00 smithi103 ceph-mon[39629]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-04-23T16:30:00.921 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:00 smithi103 ceph-mon[39629]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-04-23T16:30:00.924 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:00.966 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:30:00.966 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:00.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:00.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:30:01.068 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:30:01.102 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:00 smithi188 ceph-mon[44101]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-04-23T16:30:01.102 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:00 smithi188 ceph-mon[44101]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-04-23T16:30:01.105 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:00 smithi148 ceph-mon[44520]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-04-23T16:30:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:00 smithi148 ceph-mon[44520]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-04-23T16:30:01.130 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:30:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:01.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:30:01.235 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:30:01.306 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:01.330 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:30:01.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:01.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:30:01.439 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:30:01.468 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:01.495 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:30:01.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:01.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:30:01.598 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:30:01.641 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:01.797 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:30:01.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:01.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:30:01.902 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:30:01.937 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:01 smithi103 ceph-mon[39629]: pgmap v947: 27 pgs: 27 active+clean; 4.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-23T16:30:01.937 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:01 smithi103 ceph-mon[39629]: 4.c scrub starts 2024-04-23T16:30:01.950 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:01.980 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:30:01.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:01.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:30:02.085 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:30:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:01 smithi148 ceph-mon[44520]: pgmap v947: 27 pgs: 27 active+clean; 4.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-23T16:30:02.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:01 smithi148 ceph-mon[44520]: 4.c scrub starts 2024-04-23T16:30:02.127 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:01 smithi188 ceph-mon[44101]: pgmap v947: 27 pgs: 27 active+clean; 4.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-23T16:30:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:01 smithi188 ceph-mon[44101]: 4.c scrub starts 2024-04-23T16:30:02.259 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:30:02.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:02.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:30:02.362 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:30:02.392 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:02.522 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:30:02.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:02.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:30:02.630 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:30:02.669 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:02.697 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:30:02.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:02.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:30:02.811 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:30:02.844 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:02 smithi103 ceph-mon[39629]: 4.c scrub ok 2024-04-23T16:30:02.870 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:30:02.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:02.877 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:30:02.978 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:30:03.009 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:02 smithi188 ceph-mon[44101]: 4.c scrub ok 2024-04-23T16:30:03.010 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:03.041 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:30:03.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:03.044 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:30:03.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:02 smithi148 ceph-mon[44520]: 4.c scrub ok 2024-04-23T16:30:03.144 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:30:03.193 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:03.223 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:30:03.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:03.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:30:03.333 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:30:03.370 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:03.391 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:03.406 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:30:03.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:03.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:30:03.457 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:30:03.489 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:30:03.515 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:30:03.515 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:30:03.543 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:30:03.548 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:03.568 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:30:03.596 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:30:03.636 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:30:03.664 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:30:03.687 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:30:03.711 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:30:03.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:03.718 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:30:03.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:30:03.745 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:30:03.777 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:30:03.784 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:30:03.784 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:30:03.784 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:30:03.784 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:30:03.784 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:30:03.784 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:30:03.784 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:30:03.784 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:30:03.784 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:30:03.784 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:30:03.784 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:30:03.784 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:30:03.820 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:30:03.853 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:03.877 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:30:03.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:03.884 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:30:03.985 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:30:04.018 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:03 smithi188 ceph-mon[44101]: pgmap v948: 27 pgs: 27 active+clean; 4.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:30:04.018 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:03 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1691203062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:04.018 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.018 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.020 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:04.044 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:30:04.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:03 smithi103 ceph-mon[39629]: pgmap v948: 27 pgs: 27 active+clean; 4.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:30:04.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:03 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1691203062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:04.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.047 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:04.048 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:30:04.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:03 smithi148 ceph-mon[44520]: pgmap v948: 27 pgs: 27 active+clean; 4.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:30:04.167 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:03 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1691203062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:04.167 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.167 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.167 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:30:04.168 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10, 1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1] 2024-04-23T16:30:04.168 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-23T16:30:04.202 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:04.227 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:30:04.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:04.230 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:30:04.331 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:30:04.375 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:04.423 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:30:04.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:04.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:30:04.544 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:30:04.599 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:04.631 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:30:04.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:04.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:30:04.739 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:30:04.754 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:04 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:30:04.755 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:04 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:30:04.755 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.755 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.755 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.755 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.755 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.755 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.755 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.755 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.755 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.755 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:04 smithi103 ceph-mon[39629]: osdmap e335: 12 total, 12 up, 10 in 2024-04-23T16:30:04.755 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.755 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:04 smithi103 ceph-mon[39629]: 4.9 scrub starts 2024-04-23T16:30:04.771 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:04.796 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:30:04.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:04.799 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:30:04.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:04 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:30:04.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:04 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:30:04.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:04 smithi148 ceph-mon[44520]: osdmap e335: 12 total, 12 up, 10 in 2024-04-23T16:30:04.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:04.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:04 smithi148 ceph-mon[44520]: 4.9 scrub starts 2024-04-23T16:30:04.900 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:30:04.945 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:05.119 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:30:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:04 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:30:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:04 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:30:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:05.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:05.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:05.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:05.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:05.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:04 smithi188 ceph-mon[44101]: osdmap e335: 12 total, 12 up, 10 in 2024-04-23T16:30:05.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:05.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:04 smithi188 ceph-mon[44101]: 4.9 scrub starts 2024-04-23T16:30:05.910 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:30:05.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:05.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:30:06.016 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:30:06.043 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:05 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:30:06.043 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:05 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:30:06.043 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:05 smithi188 ceph-mon[44101]: pgmap v950: 27 pgs: 27 active+clean; 4.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:30:06.043 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:05 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3078651309' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:30:06.043 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:05 smithi188 ceph-mon[44101]: from='client.25540 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:30:06.043 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:05 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:30:06.043 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:05 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:30:06.043 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:05 smithi188 ceph-mon[44101]: from='client.25540 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:30:06.043 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:05 smithi188 ceph-mon[44101]: osdmap e336: 12 total, 12 up, 10 in 2024-04-23T16:30:06.043 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:05 smithi188 ceph-mon[44101]: 4.9 scrub ok 2024-04-23T16:30:06.043 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:05 smithi188 ceph-mon[44101]: 1.4 scrub starts 2024-04-23T16:30:06.043 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:05 smithi188 ceph-mon[44101]: 1.4 scrub ok 2024-04-23T16:30:06.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:05 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:30:06.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:05 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:30:06.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:05 smithi103 ceph-mon[39629]: pgmap v950: 27 pgs: 27 active+clean; 4.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:30:06.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:05 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3078651309' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:30:06.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:05 smithi103 ceph-mon[39629]: from='client.25540 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:30:06.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:05 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:30:06.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:05 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:30:06.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:05 smithi103 ceph-mon[39629]: from='client.25540 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:30:06.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:05 smithi103 ceph-mon[39629]: osdmap e336: 12 total, 12 up, 10 in 2024-04-23T16:30:06.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:05 smithi103 ceph-mon[39629]: 4.9 scrub ok 2024-04-23T16:30:06.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:05 smithi103 ceph-mon[39629]: 1.4 scrub starts 2024-04-23T16:30:06.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:05 smithi103 ceph-mon[39629]: 1.4 scrub ok 2024-04-23T16:30:06.046 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:06.069 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:30:06.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:06.073 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:30:06.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:05 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:30:06.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:05 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:30:06.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:05 smithi148 ceph-mon[44520]: pgmap v950: 27 pgs: 27 active+clean; 4.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:30:06.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:05 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3078651309' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:30:06.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:05 smithi148 ceph-mon[44520]: from='client.25540 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:30:06.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:05 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:30:06.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:05 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:30:06.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:05 smithi148 ceph-mon[44520]: from='client.25540 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:30:06.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:05 smithi148 ceph-mon[44520]: osdmap e336: 12 total, 12 up, 10 in 2024-04-23T16:30:06.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:05 smithi148 ceph-mon[44520]: 4.9 scrub ok 2024-04-23T16:30:06.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:05 smithi148 ceph-mon[44520]: 1.4 scrub starts 2024-04-23T16:30:06.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:05 smithi148 ceph-mon[44520]: 1.4 scrub ok 2024-04-23T16:30:06.174 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:30:06.202 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:06.556 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:30:06.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:06.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:30:06.661 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:30:06.693 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:06.714 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:30:06.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:06.719 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:30:06.820 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:30:06.854 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:06 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:30:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:06 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:30:07.088 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:06 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:30:07.088 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:06 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:30:07.092 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:07.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:06 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:30:07.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:06 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:30:07.228 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:30:07.904 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:07 smithi103 ceph-mon[39629]: 4.e scrub starts 2024-04-23T16:30:07.904 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:07 smithi103 ceph-mon[39629]: pgmap v952: 27 pgs: 27 active+clean; 4.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.33k op/s 2024-04-23T16:30:07.904 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:07 smithi103 ceph-mon[39629]: osdmap e337: 12 total, 12 up, 10 in 2024-04-23T16:30:08.027 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:30:08.027 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:30:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:07 smithi148 ceph-mon[44520]: 4.e scrub starts 2024-04-23T16:30:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:07 smithi148 ceph-mon[44520]: pgmap v952: 27 pgs: 27 active+clean; 4.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.33k op/s 2024-04-23T16:30:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:07 smithi148 ceph-mon[44520]: osdmap e337: 12 total, 12 up, 10 in 2024-04-23T16:30:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:07 smithi188 ceph-mon[44101]: 4.e scrub starts 2024-04-23T16:30:08.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:07 smithi188 ceph-mon[44101]: pgmap v952: 27 pgs: 27 active+clean; 4.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.33k op/s 2024-04-23T16:30:08.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:07 smithi188 ceph-mon[44101]: osdmap e337: 12 total, 12 up, 10 in 2024-04-23T16:30:08.615 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:30:08.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:08.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:30:08.727 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:30:08.773 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:08.811 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:30:08.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:08.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:30:08.918 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:30:08.948 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:08 smithi103 ceph-mon[39629]: 4.e scrub ok 2024-04-23T16:30:08.951 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:09.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:08 smithi148 ceph-mon[44520]: 4.e scrub ok 2024-04-23T16:30:09.132 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:30:09.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:08 smithi188 ceph-mon[44101]: 4.e scrub ok 2024-04-23T16:30:09.291 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:09.802 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-23T16:30:10.026 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:09 smithi103 ceph-mon[39629]: pgmap v954: 27 pgs: 27 active+clean; 4.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-23T16:30:10.026 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:09 smithi103 ceph-mon[39629]: 4.5 scrub starts 2024-04-23T16:30:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:09 smithi148 ceph-mon[44520]: pgmap v954: 27 pgs: 27 active+clean; 4.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-23T16:30:10.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:09 smithi148 ceph-mon[44520]: 4.5 scrub starts 2024-04-23T16:30:10.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:09 smithi188 ceph-mon[44101]: pgmap v954: 27 pgs: 27 active+clean; 4.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-23T16:30:10.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:09 smithi188 ceph-mon[44101]: 4.5 scrub starts 2024-04-23T16:30:10.249 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:30:10.304 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:30:10.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:10.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:30:10.408 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:30:10.456 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:10.527 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:30:10.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:10.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:30:10.639 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:30:10.680 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:10.896 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:30:10.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:10.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:30:11.000 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:30:11.027 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:11.044 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:11.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:10 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:30:11.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:10 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:30:11.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:10 smithi103 ceph-mon[39629]: 4.5 scrub ok 2024-04-23T16:30:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:10 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:30:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:10 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:30:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:10 smithi148 ceph-mon[44520]: 4.5 scrub ok 2024-04-23T16:30:11.161 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:30:11.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:11.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:30:11.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:10 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:30:11.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:10 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:30:11.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:10 smithi188 ceph-mon[44101]: 4.5 scrub ok 2024-04-23T16:30:11.267 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:30:11.301 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:11.327 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:30:11.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:11.330 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:30:11.431 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:30:11.493 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:11.736 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:30:11.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:11.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:30:11.828 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.0 to 1 (10000) 2024-04-23T16:30:11.850 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:30:11.887 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:11.912 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:30:11.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:11.915 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:30:12.016 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:30:12.055 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:12.088 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:11 smithi148 ceph-mon[44520]: pgmap v955: 27 pgs: 27 active+clean; 4.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.55k op/s 2024-04-23T16:30:12.088 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:11 smithi148 ceph-mon[44520]: osdmap e338: 12 total, 12 up, 10 in 2024-04-23T16:30:12.088 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:11 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/558043745' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:30:12.088 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:11 smithi148 ceph-mon[44520]: from='client.25546 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:30:12.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:11 smithi103 ceph-mon[39629]: pgmap v955: 27 pgs: 27 active+clean; 4.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.55k op/s 2024-04-23T16:30:12.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:11 smithi103 ceph-mon[39629]: osdmap e338: 12 total, 12 up, 10 in 2024-04-23T16:30:12.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:11 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/558043745' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:30:12.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:11 smithi103 ceph-mon[39629]: from='client.25546 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:30:12.110 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:30:12.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:12.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:30:12.214 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:30:12.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:11 smithi188 ceph-mon[44101]: pgmap v955: 27 pgs: 27 active+clean; 4.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.55k op/s 2024-04-23T16:30:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:11 smithi188 ceph-mon[44101]: osdmap e338: 12 total, 12 up, 10 in 2024-04-23T16:30:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:11 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/558043745' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:30:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:11 smithi188 ceph-mon[44101]: from='client.25546 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:30:12.246 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:12.269 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:30:12.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:12.275 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:30:12.376 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:30:12.406 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:12.429 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:30:12.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:12.431 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:30:12.446 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:30:12.446 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:30:12.532 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:30:12.579 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:12.730 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:30:12.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:12.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:30:12.809 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-23T16:30:12.835 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:30:12.869 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:12.897 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:30:12.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:12.900 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:30:13.001 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:30:13.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:12 smithi103 ceph-mon[39629]: from='client.25546 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:30:13.029 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:12 smithi103 ceph-mon[39629]: osdmap e339: 12 total, 12 up, 10 in 2024-04-23T16:30:13.038 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:12 smithi148 ceph-mon[44520]: from='client.25546 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:30:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:12 smithi148 ceph-mon[44520]: osdmap e339: 12 total, 12 up, 10 in 2024-04-23T16:30:13.203 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-23T16:30:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:12 smithi188 ceph-mon[44101]: from='client.25546 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:30:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:12 smithi188 ceph-mon[44101]: osdmap e339: 12 total, 12 up, 10 in 2024-04-23T16:30:13.247 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:30:13.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:13.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:30:13.355 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:30:13.444 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:13.482 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:30:13.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:13.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:30:13.590 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:30:13.622 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:13.644 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:30:13.646 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:30:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:13.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:30:13.754 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:30:13.792 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:13.843 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:13 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T16:30:13.843 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:13 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T16:30:13.843 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:13 smithi103 ceph-mon[39629]: pgmap v958: 27 pgs: 27 active+clean; 4.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:30:13.903 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:30:13.930 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:30:13.938 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:30:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:13.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:30:13.964 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:30:13.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:13 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T16:30:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:13 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T16:30:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:13 smithi188 ceph-mon[44101]: pgmap v958: 27 pgs: 27 active+clean; 4.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:30:13.995 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:30:14.025 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:30:14.044 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:30:14.061 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:30:14.092 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:30:14.108 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:13 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T16:30:14.108 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:13 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T16:30:14.108 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:13 smithi148 ceph-mon[44520]: pgmap v958: 27 pgs: 27 active+clean; 4.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:30:14.115 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:30:14.118 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:14.145 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:30:14.173 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:30:14.207 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:30:14.235 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:30:14.236 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:30:14.237 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:30:14.237 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:30:14.237 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:30:14.237 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:30:14.237 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:30:14.237 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:30:14.237 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:30:14.237 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:30:14.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:30:14.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:30:14.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:30:14.245 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:30:14.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:14.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:30:14.356 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:30:14.396 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:14.612 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:30:14.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:14.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:30:14.726 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:30:14.788 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:14.845 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:14 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/171373218' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:14.845 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:14.845 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:14.845 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:14.845 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:14.846 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:14.846 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:14.846 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:14.846 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:14.846 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:14.846 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:14.846 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:14 smithi103 ceph-mon[39629]: 4.3 scrub starts 2024-04-23T16:30:14.851 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:14 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/171373218' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:14 smithi148 ceph-mon[44520]: 4.3 scrub starts 2024-04-23T16:30:15.158 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:30:15.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:15.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:30:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:14 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/171373218' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:14 smithi188 ceph-mon[44101]: 4.3 scrub starts 2024-04-23T16:30:15.271 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:30:15.306 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:15.333 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:30:15.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:15.336 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:30:15.437 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:30:15.498 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:15.533 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:30:15.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:15.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:30:15.643 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:30:15.684 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:15.859 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.859 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.859 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:15 smithi148 ceph-mon[44520]: pgmap v959: 27 pgs: 27 active+clean; 3.8 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:30:15.859 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:15 smithi148 ceph-mon[44520]: 4.3 scrub ok 2024-04-23T16:30:15.859 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:15 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2292753731' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:30:15.888 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:30:15.889 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:30:15.889 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2024-04-23T16:30:15.889 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2024-04-23T16:30:15.889 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:30:15.987 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.987 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:15.987 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:15 smithi103 ceph-mon[39629]: pgmap v959: 27 pgs: 27 active+clean; 3.8 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:30:15.988 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:15 smithi103 ceph-mon[39629]: 4.3 scrub ok 2024-04-23T16:30:15.988 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:15 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2292753731' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:30:16.046 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:30:16.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:16.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:30:16.097 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:30:16.153 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:30:16.194 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:16.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:16.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:16.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:15 smithi188 ceph-mon[44101]: pgmap v959: 27 pgs: 27 active+clean; 3.8 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:30:16.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:15 smithi188 ceph-mon[44101]: 4.3 scrub ok 2024-04-23T16:30:16.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:15 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2292753731' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:30:16.350 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:30:16.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:16.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:30:16.455 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:30:16.502 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:16.853 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:16 smithi148 ceph-mon[44520]: 4.0 scrub starts 2024-04-23T16:30:16.853 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:16 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:30:16.853 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:16 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2292753731' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:30:16.853 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:16 smithi148 ceph-mon[44520]: osdmap e340: 12 total, 12 up, 10 in 2024-04-23T16:30:17.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:16 smithi103 ceph-mon[39629]: 4.0 scrub starts 2024-04-23T16:30:17.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:16 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:30:17.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:16 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2292753731' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:30:17.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:16 smithi103 ceph-mon[39629]: osdmap e340: 12 total, 12 up, 10 in 2024-04-23T16:30:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:16 smithi188 ceph-mon[44101]: 4.0 scrub starts 2024-04-23T16:30:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:16 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:30:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:16 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2292753731' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:30:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:16 smithi188 ceph-mon[44101]: osdmap e340: 12 total, 12 up, 10 in 2024-04-23T16:30:18.025 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:30:18.058 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:17 smithi103 ceph-mon[39629]: pgmap v961: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 3.8 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 988 op/s 2024-04-23T16:30:18.059 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:30:18.059 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:30:18.059 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:17 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:30:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:17 smithi148 ceph-mon[44520]: pgmap v961: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 3.8 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 988 op/s 2024-04-23T16:30:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:30:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:30:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:17 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:30:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:17 smithi188 ceph-mon[44101]: pgmap v961: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 3.8 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 988 op/s 2024-04-23T16:30:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:30:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:30:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:17 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:30:18.595 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:30:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:18.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:30:18.699 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:30:18.767 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:18.794 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:18.795 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:30:18.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:18.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:30:18.871 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=2847700666,config=2679774077,health=3340005642,kv=3797444253}) 2024-04-23T16:30:18.901 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:30:18.951 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:19.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=2847700666,config=2679774077,health=3340005642,kv=3797444253}) 2024-04-23T16:30:19.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=3906364575,logm=4043461221}) 2024-04-23T16:30:19.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1417921882}) 2024-04-23T16:30:19.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1759377402}) 2024-04-23T16:30:19.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1422568931}) 2024-04-23T16:30:19.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2326363252}) 2024-04-23T16:30:19.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3897412310}) 2024-04-23T16:30:19.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3727854629}) 2024-04-23T16:30:19.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=395374961}) 2024-04-23T16:30:19.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2247838021}) 2024-04-23T16:30:19.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=625758054}) 2024-04-23T16:30:19.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=360877913}) 2024-04-23T16:30:19.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=584871906}) 2024-04-23T16:30:19.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1550369926}) 2024-04-23T16:30:19.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1559724944}) 2024-04-23T16:30:19.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3750494779}) 2024-04-23T16:30:19.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2929027992}) 2024-04-23T16:30:19.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3604102342}) 2024-04-23T16:30:19.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=364750742}) 2024-04-23T16:30:19.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2121518362}) 2024-04-23T16:30:19.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=359220904}) 2024-04-23T16:30:19.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3121062435}) 2024-04-23T16:30:19.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3873587307}) 2024-04-23T16:30:19.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3399758023}) 2024-04-23T16:30:19.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=247067328}) 2024-04-23T16:30:19.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339682550}) 2024-04-23T16:30:19.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=952424124}) 2024-04-23T16:30:19.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=549606571}) 2024-04-23T16:30:19.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2345705251}) 2024-04-23T16:30:19.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1647791611}) 2024-04-23T16:30:19.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=394422706}) 2024-04-23T16:30:19.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3986703888}) 2024-04-23T16:30:19.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=587115525}) 2024-04-23T16:30:19.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3823962889}) 2024-04-23T16:30:19.106 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=773915743}) 2024-04-23T16:30:19.106 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=768961032}) 2024-04-23T16:30:19.106 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3792569385}) 2024-04-23T16:30:19.106 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3673302774}) 2024-04-23T16:30:19.106 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=589912204}) 2024-04-23T16:30:19.106 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1378972603}) 2024-04-23T16:30:19.106 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3512458167}) 2024-04-23T16:30:19.106 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=409650961}) 2024-04-23T16:30:19.106 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=464596766}) 2024-04-23T16:30:19.106 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3828462282}) 2024-04-23T16:30:19.106 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1632034699}) 2024-04-23T16:30:19.107 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=41,mdsmap=3,mgr=19,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=24} crc {logm=3295920963,mdsmap=2840952586,mgr=1765502803,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=1094608572,mon_config_key=2498862838}) 2024-04-23T16:30:19.107 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3261505249}) 2024-04-23T16:30:19.107 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=708368997}) 2024-04-23T16:30:19.107 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=63,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=19} crc {mon_config_key=1226261546,monmap=2133572193,osd_metadata=3355936916,osd_pg_creating=1757136664,osdmap=1151098161}) 2024-04-23T16:30:19.107 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=88} crc {osdmap=651753627}) 2024-04-23T16:30:19.107 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:30:19.107 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:30:19.107 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:18 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:30:19.112 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:30:19.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:19.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:30:19.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=3906364575,logm=4043461221}) 2024-04-23T16:30:19.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1417921882}) 2024-04-23T16:30:19.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1759377402}) 2024-04-23T16:30:19.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1422568931}) 2024-04-23T16:30:19.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2326363252}) 2024-04-23T16:30:19.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3897412310}) 2024-04-23T16:30:19.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3727854629}) 2024-04-23T16:30:19.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=395374961}) 2024-04-23T16:30:19.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2247838021}) 2024-04-23T16:30:19.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=625758054}) 2024-04-23T16:30:19.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=360877913}) 2024-04-23T16:30:19.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=584871906}) 2024-04-23T16:30:19.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1550369926}) 2024-04-23T16:30:19.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1559724944}) 2024-04-23T16:30:19.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3750494779}) 2024-04-23T16:30:19.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2929027992}) 2024-04-23T16:30:19.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3604102342}) 2024-04-23T16:30:19.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=364750742}) 2024-04-23T16:30:19.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2121518362}) 2024-04-23T16:30:19.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=359220904}) 2024-04-23T16:30:19.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3121062435}) 2024-04-23T16:30:19.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3873587307}) 2024-04-23T16:30:19.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3399758023}) 2024-04-23T16:30:19.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=247067328}) 2024-04-23T16:30:19.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339682550}) 2024-04-23T16:30:19.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=952424124}) 2024-04-23T16:30:19.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=549606571}) 2024-04-23T16:30:19.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2345705251}) 2024-04-23T16:30:19.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1647791611}) 2024-04-23T16:30:19.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=394422706}) 2024-04-23T16:30:19.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3986703888}) 2024-04-23T16:30:19.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=587115525}) 2024-04-23T16:30:19.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3823962889}) 2024-04-23T16:30:19.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=773915743}) 2024-04-23T16:30:19.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=768961032}) 2024-04-23T16:30:19.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3792569385}) 2024-04-23T16:30:19.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3673302774}) 2024-04-23T16:30:19.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=589912204}) 2024-04-23T16:30:19.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1378972603}) 2024-04-23T16:30:19.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3512458167}) 2024-04-23T16:30:19.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=409650961}) 2024-04-23T16:30:19.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=464596766}) 2024-04-23T16:30:19.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3828462282}) 2024-04-23T16:30:19.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1632034699}) 2024-04-23T16:30:19.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=41,mdsmap=3,mgr=19,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=24} crc {logm=3295920963,mdsmap=2840952586,mgr=1765502803,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=1094608572,mon_config_key=2498862838}) 2024-04-23T16:30:19.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3261505249}) 2024-04-23T16:30:19.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=708368997}) 2024-04-23T16:30:19.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=63,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=19} crc {mon_config_key=1226261546,monmap=2133572193,osd_metadata=3355936916,osd_pg_creating=1757136664,osdmap=1151098161}) 2024-04-23T16:30:19.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=88} crc {osdmap=651753627}) 2024-04-23T16:30:19.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:30:19.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:30:19.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:18 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:30:19.220 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:30:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=2847700666,config=2679774077,health=3340005642,kv=3797444253}) 2024-04-23T16:30:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=3906364575,logm=4043461221}) 2024-04-23T16:30:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1417921882}) 2024-04-23T16:30:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1759377402}) 2024-04-23T16:30:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1422568931}) 2024-04-23T16:30:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2326363252}) 2024-04-23T16:30:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3897412310}) 2024-04-23T16:30:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3727854629}) 2024-04-23T16:30:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=395374961}) 2024-04-23T16:30:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2247838021}) 2024-04-23T16:30:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=625758054}) 2024-04-23T16:30:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=360877913}) 2024-04-23T16:30:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=584871906}) 2024-04-23T16:30:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1550369926}) 2024-04-23T16:30:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1559724944}) 2024-04-23T16:30:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3750494779}) 2024-04-23T16:30:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2929027992}) 2024-04-23T16:30:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3604102342}) 2024-04-23T16:30:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=364750742}) 2024-04-23T16:30:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2121518362}) 2024-04-23T16:30:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=359220904}) 2024-04-23T16:30:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3121062435}) 2024-04-23T16:30:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3873587307}) 2024-04-23T16:30:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3399758023}) 2024-04-23T16:30:19.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=247067328}) 2024-04-23T16:30:19.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339682550}) 2024-04-23T16:30:19.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=952424124}) 2024-04-23T16:30:19.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=549606571}) 2024-04-23T16:30:19.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2345705251}) 2024-04-23T16:30:19.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1647791611}) 2024-04-23T16:30:19.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=394422706}) 2024-04-23T16:30:19.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3986703888}) 2024-04-23T16:30:19.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=587115525}) 2024-04-23T16:30:19.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3823962889}) 2024-04-23T16:30:19.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=773915743}) 2024-04-23T16:30:19.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=768961032}) 2024-04-23T16:30:19.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3792569385}) 2024-04-23T16:30:19.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3673302774}) 2024-04-23T16:30:19.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=589912204}) 2024-04-23T16:30:19.230 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1378972603}) 2024-04-23T16:30:19.230 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3512458167}) 2024-04-23T16:30:19.230 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=409650961}) 2024-04-23T16:30:19.230 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=464596766}) 2024-04-23T16:30:19.230 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3828462282}) 2024-04-23T16:30:19.230 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1632034699}) 2024-04-23T16:30:19.230 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=41,mdsmap=3,mgr=19,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=24} crc {logm=3295920963,mdsmap=2840952586,mgr=1765502803,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=1094608572,mon_config_key=2498862838}) 2024-04-23T16:30:19.230 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3261505249}) 2024-04-23T16:30:19.230 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=708368997}) 2024-04-23T16:30:19.230 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=63,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=19} crc {mon_config_key=1226261546,monmap=2133572193,osd_metadata=3355936916,osd_pg_creating=1757136664,osdmap=1151098161}) 2024-04-23T16:30:19.230 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=88} crc {osdmap=651753627}) 2024-04-23T16:30:19.230 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:30:19.230 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:30:19.230 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:18 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:30:19.252 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:19.808 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:30:19.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:19.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:30:19.883 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.2 to 1 (10000) 2024-04-23T16:30:19.912 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:30:19.944 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:19.966 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:30:19.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:19.969 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:30:20.070 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:30:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:19 smithi103 ceph-mon[39629]: pgmap v962: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 3.7 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 931 op/s 2024-04-23T16:30:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:19 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2231376730' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:30:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:19 smithi103 ceph-mon[39629]: from='client.25558 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:30:20.104 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:20.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:19 smithi148 ceph-mon[44520]: pgmap v962: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 3.7 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 931 op/s 2024-04-23T16:30:20.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:19 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2231376730' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:30:20.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:19 smithi148 ceph-mon[44520]: from='client.25558 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:30:20.181 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:30:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:19 smithi188 ceph-mon[44101]: pgmap v962: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 3.7 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 931 op/s 2024-04-23T16:30:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:19 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2231376730' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:30:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:19 smithi188 ceph-mon[44101]: from='client.25558 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:30:20.555 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:30:20.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:20.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:30:20.662 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:30:20.690 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:20.713 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:30:20.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:20.716 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:30:20.818 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:30:20.849 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:20.874 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:30:20.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:20.877 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:30:20.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:20 smithi188 ceph-mon[44101]: from='client.25558 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-23T16:30:20.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:20 smithi188 ceph-mon[44101]: osdmap e341: 12 total, 12 up, 10 in 2024-04-23T16:30:20.978 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:30:21.018 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:21.219 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:20 smithi148 ceph-mon[44520]: from='client.25558 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-23T16:30:21.219 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:20 smithi148 ceph-mon[44520]: osdmap e341: 12 total, 12 up, 10 in 2024-04-23T16:30:21.264 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-23T16:30:21.301 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:20 smithi103 ceph-mon[39629]: from='client.25558 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-23T16:30:21.301 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:20 smithi103 ceph-mon[39629]: osdmap e341: 12 total, 12 up, 10 in 2024-04-23T16:30:21.332 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:21.345 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-23T16:30:21.661 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:30:21.676 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:30:21.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:21.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:30:21.780 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:30:21.813 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:21.836 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:30:21.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:21.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:30:21.896 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:21 smithi103 ceph-mon[39629]: pgmap v964: 27 pgs: 27 active+clean; 3.4 GiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.29k op/s 2024-04-23T16:30:21.944 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:30:22.006 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:22.008 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:30:22.037 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:30:22.043 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:30:22.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:22.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:30:22.062 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:30:22.089 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:30:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:30:22.148 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:30:22.149 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:30:22.178 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:30:22.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:21 smithi188 ceph-mon[44101]: pgmap v964: 27 pgs: 27 active+clean; 3.4 GiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.29k op/s 2024-04-23T16:30:22.184 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:22.205 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:30:22.213 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:30:22.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:22.216 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:30:22.228 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:30:22.257 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:30:22.286 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:30:22.307 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:30:22.308 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:30:22.309 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:30:22.309 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:30:22.309 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:30:22.309 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:30:22.309 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:30:22.309 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:30:22.309 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:30:22.309 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:30:22.309 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:30:22.309 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:30:22.309 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:30:22.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:21 smithi148 ceph-mon[44520]: pgmap v964: 27 pgs: 27 active+clean; 3.4 GiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.29k op/s 2024-04-23T16:30:22.316 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:30:22.346 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:22.370 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:30:22.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:22.372 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:30:22.474 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:30:22.514 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:22.646 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:30:22.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:22.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:30:22.758 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:30:22.800 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:23.054 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:30:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:23.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:30:23.057 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:22 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:30:23.057 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:22 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:30:23.057 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:22 smithi103 ceph-mon[39629]: osdmap e342: 12 total, 12 up, 10 in 2024-04-23T16:30:23.057 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:22 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/598288441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:23.058 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.058 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.058 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.058 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.058 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.061 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.061 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.110 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:22 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:30:23.110 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:22 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:30:23.110 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:22 smithi148 ceph-mon[44520]: osdmap e342: 12 total, 12 up, 10 in 2024-04-23T16:30:23.110 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:22 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/598288441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:23.110 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.111 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.111 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.111 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.111 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.111 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.111 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.111 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.111 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.157 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:30:23.186 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:23.215 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:30:23.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:23.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:30:23.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:22 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:30:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:22 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:30:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:22 smithi188 ceph-mon[44101]: osdmap e342: 12 total, 12 up, 10 in 2024-04-23T16:30:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:22 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/598288441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:23.321 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:30:23.354 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:23.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:23.385 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:30:23.386 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:30:23.487 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:30:23.566 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:23.573 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:23.656 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:30:23.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:23.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:30:23.765 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:30:23.815 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:23.843 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:30:23.843 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:30:23.851 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:30:23.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:23.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:30:23.958 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:30:23.989 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:24.013 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:30:24.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:24.018 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:30:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:23 smithi103 ceph-mon[39629]: pgmap v966: 27 pgs: 27 active+clean; 3.3 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-23T16:30:24.119 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:30:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:23 smithi148 ceph-mon[44520]: pgmap v966: 27 pgs: 27 active+clean; 3.3 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-23T16:30:24.159 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:24.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:24.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:24.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:24.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:23 smithi188 ceph-mon[44101]: pgmap v966: 27 pgs: 27 active+clean; 3.3 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-23T16:30:24.421 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:30:24.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:24.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:30:24.525 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:30:24.553 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:24.571 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-23T16:30:24.575 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:30:24.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:24.583 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:30:24.684 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:30:24.742 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:24.780 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:30:24.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:24.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:30:24.886 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:30:24.931 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:24.991 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:30:24.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:24.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:30:25.029 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:30:25.070 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:30:25.102 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:30:25.106 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:25 smithi148 ceph-mon[44520]: pgmap v967: 27 pgs: 27 active+clean; 3.1 GiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.39k op/s 2024-04-23T16:30:25.140 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:25.168 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:30:25.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:25.175 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:30:25.276 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:30:25.308 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:25.329 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:30:25.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:25.333 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:30:25.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:25 smithi103 ceph-mon[39629]: pgmap v967: 27 pgs: 27 active+clean; 3.1 GiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.39k op/s 2024-04-23T16:30:25.433 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:30:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:25 smithi188 ceph-mon[44101]: pgmap v967: 27 pgs: 27 active+clean; 3.1 GiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.39k op/s 2024-04-23T16:30:25.476 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:25.946 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:30:25.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:25.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:30:26.053 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:30:26.099 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:26.461 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:30:26.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:26.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:30:26.565 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:30:26.600 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:26.676 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:30:26.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:26.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:30:26.740 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:26.782 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:30:26.819 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:27.075 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:30:27.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:27.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:30:27.081 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:30:27.081 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:30:27.181 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:30:27.211 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:27.756 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:30:27.762 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.3 to 1 (10000) 2024-04-23T16:30:27.837 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:27 smithi148 ceph-mon[44520]: pgmap v968: 27 pgs: 27 active+clean; 3.1 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-23T16:30:27.837 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:27 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1487272091' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:30:27.838 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:27 smithi148 ceph-mon[44520]: from='client.25567 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:30:27.838 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:27 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1138884398' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:30:27.838 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:27 smithi148 ceph-mon[44520]: from='client.25573 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:30:27.881 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-23T16:30:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:27 smithi103 ceph-mon[39629]: pgmap v968: 27 pgs: 27 active+clean; 3.1 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-23T16:30:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:27 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1487272091' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:30:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:27 smithi103 ceph-mon[39629]: from='client.25567 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:30:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:27 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1138884398' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:30:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:27 smithi103 ceph-mon[39629]: from='client.25573 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:30:28.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:27 smithi188 ceph-mon[44101]: pgmap v968: 27 pgs: 27 active+clean; 3.1 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-23T16:30:28.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:27 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1487272091' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:30:28.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:27 smithi188 ceph-mon[44101]: from='client.25567 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:30:28.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:27 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1138884398' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:30:28.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:27 smithi188 ceph-mon[44101]: from='client.25573 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:30:28.310 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:30:28.795 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:30:28.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:28.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:30:28.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:28 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:30:28.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:28 smithi148 ceph-mon[44520]: from='client.25567 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-23T16:30:28.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:28 smithi148 ceph-mon[44520]: from='client.25573 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:30:28.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:28 smithi148 ceph-mon[44520]: osdmap e343: 12 total, 12 up, 10 in 2024-04-23T16:30:28.902 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:30:28.971 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:29.027 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:28 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:30:29.027 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:28 smithi103 ceph-mon[39629]: from='client.25567 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-23T16:30:29.027 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:28 smithi103 ceph-mon[39629]: from='client.25573 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:30:29.027 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:28 smithi103 ceph-mon[39629]: osdmap e343: 12 total, 12 up, 10 in 2024-04-23T16:30:29.197 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:30:29.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:29.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:30:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:28 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:30:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:28 smithi188 ceph-mon[44101]: from='client.25567 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-23T16:30:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:28 smithi188 ceph-mon[44101]: from='client.25573 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:30:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:28 smithi188 ceph-mon[44101]: osdmap e343: 12 total, 12 up, 10 in 2024-04-23T16:30:29.289 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:30:29.306 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:30:29.317 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:30:29.340 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:30:29.352 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:29.367 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:30:29.394 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:30:29.430 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:30:29.455 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:30:29.481 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:30:29.507 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:30:29.529 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:30:29.555 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:30:29.587 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:30:29.588 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:30:29.588 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:30:29.588 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:30:29.589 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:30:29.589 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:30:29.589 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:30:29.589 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:30:29.589 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:30:29.589 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:30:29.589 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:30:29.589 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:30:29.589 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:30:29.625 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:30:29.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:29.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:30:29.731 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:30:29.762 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:29.784 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:30:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:29.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:30:29.888 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:30:29.923 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:29 smithi148 ceph-mon[44520]: pgmap v970: 27 pgs: 27 active+clean; 3.0 GiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 933 op/s 2024-04-23T16:30:29.924 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:29 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:30:29.924 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:29 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:30:29.924 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:29 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3005225948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:29.924 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:29.924 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:29.924 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:29.925 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:29.925 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:29.929 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:29.957 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:30:29.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:29.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:30:30.065 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:30:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:29 smithi103 ceph-mon[39629]: pgmap v970: 27 pgs: 27 active+clean; 3.0 GiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 933 op/s 2024-04-23T16:30:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:29 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:30:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:29 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:30:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:29 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3005225948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.122 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:30.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:29 smithi188 ceph-mon[44101]: pgmap v970: 27 pgs: 27 active+clean; 3.0 GiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 933 op/s 2024-04-23T16:30:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:29 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:30:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:29 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:30:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:29 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3005225948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.245 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:30:30.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:30.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:30:30.337 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:30:30.337 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:30:30.338 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2024-04-23T16:30:30.338 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2024-04-23T16:30:30.338 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:30:30.351 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:30:30.381 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:30.405 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:30:30.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:30.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:30:30.510 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:30:30.578 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:30.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:30 smithi103 ceph-mon[39629]: osdmap e344: 12 total, 12 up, 10 in 2024-04-23T16:30:30.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:30.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:30 smithi148 ceph-mon[44520]: osdmap e344: 12 total, 12 up, 10 in 2024-04-23T16:30:30.997 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-23T16:30:31.125 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-23T16:30:31.152 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:30:31.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:31.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:30:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:30 smithi188 ceph-mon[44101]: osdmap e344: 12 total, 12 up, 10 in 2024-04-23T16:30:31.257 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:30:31.334 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:31.384 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:30:31.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:31.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:30:31.492 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:30:31.522 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:31.545 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:30:31.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:31.554 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:30:31.571 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:30:31.658 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:30:31.682 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:31.692 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:31.716 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:30:31.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:31.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:30:31.771 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:31 smithi103 ceph-mon[39629]: pgmap v972: 27 pgs: 27 active+clean; 2.7 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:30:31.775 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:31 smithi188 ceph-mon[44101]: pgmap v972: 27 pgs: 27 active+clean; 2.7 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:30:31.824 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:30:31.856 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:31.980 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:30:31.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:31.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:30:32.085 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:30:32.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:31 smithi148 ceph-mon[44520]: pgmap v972: 27 pgs: 27 active+clean; 2.7 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:30:32.149 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:32.563 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:30:32.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:32 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:30:32.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:32 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:30:32.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:32 smithi103 ceph-mon[39629]: osdmap e345: 12 total, 12 up, 10 in 2024-04-23T16:30:32.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:32 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:30:32.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:32 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:30:32.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:32 smithi148 ceph-mon[44520]: osdmap e345: 12 total, 12 up, 10 in 2024-04-23T16:30:32.933 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:30:32.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:32.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:30:33.044 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:30:33.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:32 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:30:33.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:32 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:30:33.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:32 smithi188 ceph-mon[44101]: osdmap e345: 12 total, 12 up, 10 in 2024-04-23T16:30:33.074 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:33.096 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:30:33.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:33.103 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:30:33.204 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:30:33.238 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:33.489 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:30:33.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:33.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:30:33.542 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:30:33.593 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:30:33.598 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:30:33.598 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2024-04-23T16:30:33.598 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2024-04-23T16:30:33.598 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:30:33.598 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.598 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:122027 3.5 3:abb47304:::benchmark_data_smithi188_89276_object122026:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-23T16:30:33.598 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:15.787723+0000", 2024-04-23T16:30:33.598 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 437.80185471099998, 2024-04-23T16:30:33.598 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.21095762000000001, 2024-04-23T16:30:33.598 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.598 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.599 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.599 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:30:33.599 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:30:33.599 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:30:33.599 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 122027 2024-04-23T16:30:33.599 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.599 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.599 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.599 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.599 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.787723+0000", 2024-04-23T16:30:33.599 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.599 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.599 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.599 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.600 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.787723+0000", 2024-04-23T16:30:33.600 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.600 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.600 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.600 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.600 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.787723+0000", 2024-04-23T16:30:33.600 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8000000000000004e-07 2024-04-23T16:30:33.600 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.600 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.600 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.600 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.787799+0000", 2024-04-23T16:30:33.600 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6030000000000002e-05 2024-04-23T16:30:33.600 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.600 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.601 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.601 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.787810+0000", 2024-04-23T16:30:33.601 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0111e-05 2024-04-23T16:30:33.601 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.601 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.601 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.601 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.787820+0000", 2024-04-23T16:30:33.601 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0071000000000001e-05 2024-04-23T16:30:33.601 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.608 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.608 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.608 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.790677+0000", 2024-04-23T16:30:33.608 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0028577210000000001 2024-04-23T16:30:33.608 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.608 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.608 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.608 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.790801+0000", 2024-04-23T16:30:33.608 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012331000000000001 2024-04-23T16:30:33.608 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.608 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.609 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:30:33.609 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.790892+0000", 2024-04-23T16:30:33.609 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1520000000000005e-05 2024-04-23T16:30:33.609 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.609 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.609 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:30:33.610 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.797054+0000", 2024-04-23T16:30:33.610 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0061618949999999997 2024-04-23T16:30:33.610 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.610 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.610 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:30:33.610 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.998592+0000", 2024-04-23T16:30:33.610 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.201537405 2024-04-23T16:30:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.998630+0000", 2024-04-23T16:30:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8013999999999999e-05 2024-04-23T16:30:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.998680+0000", 2024-04-23T16:30:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0662999999999999e-05 2024-04-23T16:30:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:122031 3.5 3:ae893ae0:::benchmark_data_smithi188_89276_object122030:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-23T16:30:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:15.787855+0000", 2024-04-23T16:30:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 437.80172224, 2024-04-23T16:30:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.22571445400000001, 2024-04-23T16:30:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:30:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:30:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:30:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 122031 2024-04-23T16:30:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.787855+0000", 2024-04-23T16:30:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.787855+0000", 2024-04-23T16:30:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.787856+0000", 2024-04-23T16:30:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3e-06 2024-04-23T16:30:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.787916+0000", 2024-04-23T16:30:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9228999999999998e-05 2024-04-23T16:30:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.787924+0000", 2024-04-23T16:30:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8279999999999992e-06 2024-04-23T16:30:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.787930+0000", 2024-04-23T16:30:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0530000000000003e-06 2024-04-23T16:30:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.793559+0000", 2024-04-23T16:30:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0056281559999999996 2024-04-23T16:30:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.793635+0000", 2024-04-23T16:30:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6412e-05 2024-04-23T16:30:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:30:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.793734+0000", 2024-04-23T16:30:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8924000000000004e-05 2024-04-23T16:30:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:30:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.800208+0000", 2024-04-23T16:30:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0064736849999999999 2024-04-23T16:30:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:30:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.013471+0000", 2024-04-23T16:30:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.21326363700000001 2024-04-23T16:30:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.617 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.617 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.617 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.013528+0000", 2024-04-23T16:30:33.617 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6586999999999999e-05 2024-04-23T16:30:33.617 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.617 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.013570+0000", 2024-04-23T16:30:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1643000000000001e-05 2024-04-23T16:30:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:122035 3.c 3:389b6e84:::benchmark_data_smithi188_89276_object122034:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-23T16:30:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:15.791031+0000", 2024-04-23T16:30:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 437.79854636200002, 2024-04-23T16:30:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.207649587, 2024-04-23T16:30:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:30:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:30:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:30:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 122035 2024-04-23T16:30:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.791031+0000", 2024-04-23T16:30:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.620 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.791031+0000", 2024-04-23T16:30:33.620 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.620 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.620 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.620 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.620 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.791032+0000", 2024-04-23T16:30:33.620 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3179999999999999e-06 2024-04-23T16:30:33.620 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.620 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.620 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.620 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.791081+0000", 2024-04-23T16:30:33.620 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.85e-05 2024-04-23T16:30:33.620 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.620 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.621 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.621 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.791090+0000", 2024-04-23T16:30:33.621 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6600000000000007e-06 2024-04-23T16:30:33.621 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.621 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.621 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.621 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.791097+0000", 2024-04-23T16:30:33.621 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4849999999999997e-06 2024-04-23T16:30:33.621 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.621 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.621 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.621 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.791136+0000", 2024-04-23T16:30:33.621 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9332999999999998e-05 2024-04-23T16:30:33.621 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.622 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.622 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.622 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.791232+0000", 2024-04-23T16:30:33.622 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5625999999999996e-05 2024-04-23T16:30:33.622 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.622 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.622 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:30:33.622 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.791320+0000", 2024-04-23T16:30:33.622 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8506999999999998e-05 2024-04-23T16:30:33.622 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.622 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.622 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:30:33.622 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.793957+0000", 2024-04-23T16:30:33.622 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0026364689999999998 2024-04-23T16:30:33.623 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.623 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.623 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:30:33.623 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.998579+0000", 2024-04-23T16:30:33.623 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20462203600000001 2024-04-23T16:30:33.623 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.623 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.623 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.623 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.998644+0000", 2024-04-23T16:30:33.623 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4753999999999997e-05 2024-04-23T16:30:33.623 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.623 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.623 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.623 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.998681+0000", 2024-04-23T16:30:33.624 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6899e-05 2024-04-23T16:30:33.624 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.624 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.624 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.624 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.624 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.624 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:122055 3.5 3:a80c6850:::benchmark_data_smithi188_89276_object122054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-23T16:30:33.624 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:15.806914+0000", 2024-04-23T16:30:33.624 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 437.78266359399998, 2024-04-23T16:30:33.624 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20677689299999999, 2024-04-23T16:30:33.624 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.624 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.624 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.624 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:30:33.624 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:30:33.625 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:30:33.625 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 122055 2024-04-23T16:30:33.625 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.625 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.625 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.625 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.625 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.806914+0000", 2024-04-23T16:30:33.625 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.625 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.625 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.625 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.625 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.806914+0000", 2024-04-23T16:30:33.625 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.625 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.626 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.626 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.626 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.806916+0000", 2024-04-23T16:30:33.626 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.119e-06 2024-04-23T16:30:33.626 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.626 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.626 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.626 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.806958+0000", 2024-04-23T16:30:33.626 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1884000000000001e-05 2024-04-23T16:30:33.626 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.626 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.626 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.626 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.806968+0000", 2024-04-23T16:30:33.627 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0427e-05 2024-04-23T16:30:33.627 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.627 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.627 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.627 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.806990+0000", 2024-04-23T16:30:33.627 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2364000000000001e-05 2024-04-23T16:30:33.627 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.627 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.627 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.627 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.807039+0000", 2024-04-23T16:30:33.627 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8483999999999999e-05 2024-04-23T16:30:33.627 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.627 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.627 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.628 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.807142+0000", 2024-04-23T16:30:33.628 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000103001 2024-04-23T16:30:33.628 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.628 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.628 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:30:33.628 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.807242+0000", 2024-04-23T16:30:33.628 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000100342 2024-04-23T16:30:33.628 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.628 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.628 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:30:33.628 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.816578+0000", 2024-04-23T16:30:33.628 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0093354570000000001 2024-04-23T16:30:33.628 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.628 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.628 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:30:33.629 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.013608+0000", 2024-04-23T16:30:33.629 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19703056999999999 2024-04-23T16:30:33.629 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.629 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.629 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.629 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.013655+0000", 2024-04-23T16:30:33.629 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6987000000000003e-05 2024-04-23T16:30:33.629 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.629 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.629 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.629 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.013691+0000", 2024-04-23T16:30:33.629 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5258e-05 2024-04-23T16:30:33.629 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.629 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.630 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.630 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.630 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.630 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:122056 3.5 3:af2fb776:::benchmark_data_smithi188_89276_object122055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-23T16:30:33.630 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:15.807047+0000", 2024-04-23T16:30:33.630 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 437.78253004800001, 2024-04-23T16:30:33.630 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.212037635, 2024-04-23T16:30:33.630 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.630 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.630 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.630 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:30:33.630 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:30:33.630 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:30:33.631 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 122056 2024-04-23T16:30:33.631 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.631 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.631 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.631 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.631 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.807047+0000", 2024-04-23T16:30:33.631 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.631 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.632 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.632 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.632 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.807047+0000", 2024-04-23T16:30:33.632 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.632 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.632 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.632 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.632 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.807048+0000", 2024-04-23T16:30:33.632 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.78e-07 2024-04-23T16:30:33.632 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.632 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.632 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.632 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.807103+0000", 2024-04-23T16:30:33.632 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4369000000000001e-05 2024-04-23T16:30:33.632 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.633 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.633 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.633 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.807109+0000", 2024-04-23T16:30:33.633 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3600000000000001e-06 2024-04-23T16:30:33.633 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.633 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.633 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.633 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.807114+0000", 2024-04-23T16:30:33.633 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9470000000000002e-06 2024-04-23T16:30:33.633 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.633 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.633 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.633 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.810223+0000", 2024-04-23T16:30:33.633 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0031094059999999999 2024-04-23T16:30:33.634 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.634 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.634 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.634 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.810356+0000", 2024-04-23T16:30:33.634 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000133046 2024-04-23T16:30:33.634 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.634 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.634 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:30:33.634 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.810509+0000", 2024-04-23T16:30:33.634 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015293199999999999 2024-04-23T16:30:33.634 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.634 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.634 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:30:33.634 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.816586+0000", 2024-04-23T16:30:33.634 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0060771339999999997 2024-04-23T16:30:33.635 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.635 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.635 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:30:33.635 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.019039+0000", 2024-04-23T16:30:33.635 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.202452142 2024-04-23T16:30:33.635 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.635 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.635 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.635 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.019066+0000", 2024-04-23T16:30:33.635 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7212999999999998e-05 2024-04-23T16:30:33.635 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.635 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.635 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.635 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.019085+0000", 2024-04-23T16:30:33.636 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9108000000000001e-05 2024-04-23T16:30:33.636 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.636 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.636 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.636 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.636 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.636 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:122057 3.c 3:3f32ce20:::benchmark_data_smithi188_89276_object122056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-23T16:30:33.636 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:15.807218+0000", 2024-04-23T16:30:33.636 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 437.78235897299999, 2024-04-23T16:30:33.636 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20162126699999999, 2024-04-23T16:30:33.636 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.636 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.637 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.637 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:30:33.637 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:30:33.637 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:30:33.637 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 122057 2024-04-23T16:30:33.637 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.637 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.637 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.637 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.637 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.807218+0000", 2024-04-23T16:30:33.637 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.637 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.637 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.637 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.638 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.807218+0000", 2024-04-23T16:30:33.638 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.638 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.638 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.638 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.638 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.807219+0000", 2024-04-23T16:30:33.638 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1480000000000001e-06 2024-04-23T16:30:33.638 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.638 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.638 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.638 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.807296+0000", 2024-04-23T16:30:33.638 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6958000000000004e-05 2024-04-23T16:30:33.638 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.638 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.639 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.639 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.807305+0000", 2024-04-23T16:30:33.639 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6920000000000008e-06 2024-04-23T16:30:33.639 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.639 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.639 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.639 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.807312+0000", 2024-04-23T16:30:33.639 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7220000000000002e-06 2024-04-23T16:30:33.639 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.639 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.639 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.639 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.807361+0000", 2024-04-23T16:30:33.639 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9607000000000001e-05 2024-04-23T16:30:33.639 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.640 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.640 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.640 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.807453+0000", 2024-04-23T16:30:33.640 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1909000000000004e-05 2024-04-23T16:30:33.640 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.640 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.640 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:30:33.640 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.807539+0000", 2024-04-23T16:30:33.640 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5928999999999997e-05 2024-04-23T16:30:33.640 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.640 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.640 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:30:33.640 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.810486+0000", 2024-04-23T16:30:33.640 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0029467999999999999 2024-04-23T16:30:33.641 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.641 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.641 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:30:33.641 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.008746+0000", 2024-04-23T16:30:33.641 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.198260348 2024-04-23T16:30:33.641 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.641 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.641 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.641 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.008798+0000", 2024-04-23T16:30:33.641 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1070999999999999e-05 2024-04-23T16:30:33.641 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.641 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.641 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.641 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.008840+0000", 2024-04-23T16:30:33.641 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2082999999999997e-05 2024-04-23T16:30:33.642 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.642 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.642 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.642 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.642 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.642 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:122063 3.c 3:33a9930b:::benchmark_data_smithi188_89276_object122062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-23T16:30:33.642 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:15.813884+0000", 2024-04-23T16:30:33.642 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 437.77569300599998, 2024-04-23T16:30:33.642 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.195058015, 2024-04-23T16:30:33.642 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.642 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.642 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.642 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:30:33.642 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:30:33.643 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:30:33.643 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 122063 2024-04-23T16:30:33.643 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.643 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.643 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.643 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.643 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.813884+0000", 2024-04-23T16:30:33.643 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.643 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.643 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.643 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.643 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.813884+0000", 2024-04-23T16:30:33.643 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.643 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.644 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.644 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.644 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.813886+0000", 2024-04-23T16:30:33.644 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.438e-06 2024-04-23T16:30:33.644 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.644 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.644 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.644 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.813954+0000", 2024-04-23T16:30:33.644 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8572999999999996e-05 2024-04-23T16:30:33.644 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.644 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.644 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.644 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.813967+0000", 2024-04-23T16:30:33.644 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2812e-05 2024-04-23T16:30:33.644 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.645 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.645 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.645 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.813975+0000", 2024-04-23T16:30:33.645 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2970000000000004e-06 2024-04-23T16:30:33.645 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.645 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.645 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.645 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.814035+0000", 2024-04-23T16:30:33.645 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0003999999999997e-05 2024-04-23T16:30:33.645 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.645 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.645 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.645 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.814162+0000", 2024-04-23T16:30:33.645 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000126472 2024-04-23T16:30:33.645 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.646 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.646 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:30:33.646 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.814290+0000", 2024-04-23T16:30:33.646 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012812300000000001 2024-04-23T16:30:33.646 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.646 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.646 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:30:33.646 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.817242+0000", 2024-04-23T16:30:33.646 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002951979 2024-04-23T16:30:33.646 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.646 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.646 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:30:33.646 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.008874+0000", 2024-04-23T16:30:33.646 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19163248499999999 2024-04-23T16:30:33.647 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.647 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.647 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.647 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.008908+0000", 2024-04-23T16:30:33.647 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3448999999999998e-05 2024-04-23T16:30:33.647 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.647 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.647 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.647 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.008942+0000", 2024-04-23T16:30:33.647 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4382999999999999e-05 2024-04-23T16:30:33.647 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.647 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.647 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.647 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.648 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.648 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:122067 3.5 3:a742c091:::benchmark_data_smithi188_89276_object122066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-23T16:30:33.648 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:15.817627+0000", 2024-04-23T16:30:33.648 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 437.77195070800002, 2024-04-23T16:30:33.648 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20159436999999999, 2024-04-23T16:30:33.648 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.648 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.648 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.648 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:30:33.648 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:30:33.648 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:30:33.648 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 122067 2024-04-23T16:30:33.648 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.648 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.649 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.649 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.649 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.817627+0000", 2024-04-23T16:30:33.649 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.649 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.649 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.649 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.649 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.817627+0000", 2024-04-23T16:30:33.649 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.649 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.649 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.649 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.649 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.817628+0000", 2024-04-23T16:30:33.649 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.858e-06 2024-04-23T16:30:33.649 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.650 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.650 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.650 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.817698+0000", 2024-04-23T16:30:33.650 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9060999999999995e-05 2024-04-23T16:30:33.650 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.650 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.650 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.650 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.817707+0000", 2024-04-23T16:30:33.650 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.054e-06 2024-04-23T16:30:33.650 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.650 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.650 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.650 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.817712+0000", 2024-04-23T16:30:33.650 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9009999999999999e-06 2024-04-23T16:30:33.651 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.651 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.651 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.651 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.817757+0000", 2024-04-23T16:30:33.651 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4672999999999997e-05 2024-04-23T16:30:33.651 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.651 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.651 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.651 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.817884+0000", 2024-04-23T16:30:33.651 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012722100000000001 2024-04-23T16:30:33.651 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.651 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.651 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:30:33.651 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.818029+0000", 2024-04-23T16:30:33.652 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014467000000000001 2024-04-23T16:30:33.652 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.652 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.652 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:30:33.652 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.824427+0000", 2024-04-23T16:30:33.652 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0063978669999999998 2024-04-23T16:30:33.652 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.652 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.652 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:30:33.652 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.019157+0000", 2024-04-23T16:30:33.652 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19472973499999999 2024-04-23T16:30:33.652 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.652 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.652 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.653 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.019187+0000", 2024-04-23T16:30:33.653 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0153000000000001e-05 2024-04-23T16:30:33.653 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.653 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.653 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.653 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.019221+0000", 2024-04-23T16:30:33.653 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4177000000000002e-05 2024-04-23T16:30:33.653 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.653 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.653 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.653 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.653 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.653 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:122070 3.5 3:a8c5d78b:::benchmark_data_smithi188_89276_object122069:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-23T16:30:33.653 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:15.818970+0000", 2024-04-23T16:30:33.654 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 437.77060772300001, 2024-04-23T16:30:33.654 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20780275400000001, 2024-04-23T16:30:33.654 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.654 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.654 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.654 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:30:33.654 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:30:33.654 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:30:33.654 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 122070 2024-04-23T16:30:33.654 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.654 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.654 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.654 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.654 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.818970+0000", 2024-04-23T16:30:33.655 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.655 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.655 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.655 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.655 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.818970+0000", 2024-04-23T16:30:33.655 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.655 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.655 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.655 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.655 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.818971+0000", 2024-04-23T16:30:33.655 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.274e-06 2024-04-23T16:30:33.655 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.655 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.655 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.656 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.819071+0000", 2024-04-23T16:30:33.656 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000100268 2024-04-23T16:30:33.656 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.656 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.656 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.656 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.819082+0000", 2024-04-23T16:30:33.656 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1246e-05 2024-04-23T16:30:33.656 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.656 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.656 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.656 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.819090+0000", 2024-04-23T16:30:33.656 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8720000000000002e-06 2024-04-23T16:30:33.656 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.656 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.656 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.657 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.821051+0000", 2024-04-23T16:30:33.657 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0019605849999999999 2024-04-23T16:30:33.657 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.657 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.657 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.657 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.821173+0000", 2024-04-23T16:30:33.657 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012235300000000001 2024-04-23T16:30:33.657 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.657 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.657 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:30:33.657 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.821313+0000", 2024-04-23T16:30:33.657 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014013400000000001 2024-04-23T16:30:33.657 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.657 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.657 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:30:33.658 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.824562+0000", 2024-04-23T16:30:33.658 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032491450000000002 2024-04-23T16:30:33.658 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.658 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.658 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:30:33.658 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.026665+0000", 2024-04-23T16:30:33.658 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20210276999999999 2024-04-23T16:30:33.658 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.658 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.658 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.658 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.026721+0000", 2024-04-23T16:30:33.658 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6141999999999999e-05 2024-04-23T16:30:33.658 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.658 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.659 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.659 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.026772+0000", 2024-04-23T16:30:33.659 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0964999999999997e-05 2024-04-23T16:30:33.659 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.659 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.659 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.659 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.659 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.659 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:122081 3.5 3:a7bde7c5:::benchmark_data_smithi188_89276_object122080:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-23T16:30:33.659 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:15.827164+0000", 2024-04-23T16:30:33.659 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 437.76241352400001, 2024-04-23T16:30:33.660 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20006768799999999, 2024-04-23T16:30:33.660 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.660 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.660 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.660 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:30:33.660 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:30:33.660 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:30:33.660 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 122081 2024-04-23T16:30:33.660 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.660 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.660 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.660 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.660 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.827164+0000", 2024-04-23T16:30:33.661 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.661 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.661 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.661 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.662 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.827164+0000", 2024-04-23T16:30:33.662 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.662 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.662 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.662 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.662 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.827166+0000", 2024-04-23T16:30:33.662 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.934e-06 2024-04-23T16:30:33.662 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.662 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.662 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.662 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.827246+0000", 2024-04-23T16:30:33.662 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.9983999999999999e-05 2024-04-23T16:30:33.662 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.663 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.663 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.663 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.827257+0000", 2024-04-23T16:30:33.663 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1538e-05 2024-04-23T16:30:33.663 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.663 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.663 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.663 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.827267+0000", 2024-04-23T16:30:33.663 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5249999999999994e-06 2024-04-23T16:30:33.663 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.663 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.663 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.664 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.827313+0000", 2024-04-23T16:30:33.664 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5930999999999998e-05 2024-04-23T16:30:33.664 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.664 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.664 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.664 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.827406+0000", 2024-04-23T16:30:33.664 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3621000000000004e-05 2024-04-23T16:30:33.664 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.664 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.664 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:30:33.664 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.827508+0000", 2024-04-23T16:30:33.664 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000102118 2024-04-23T16:30:33.664 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.664 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.665 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:30:33.665 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.830233+0000", 2024-04-23T16:30:33.665 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0027242159999999998 2024-04-23T16:30:33.665 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.665 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.665 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:30:33.665 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.027136+0000", 2024-04-23T16:30:33.665 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19690311999999999 2024-04-23T16:30:33.665 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.665 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.665 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.665 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.027190+0000", 2024-04-23T16:30:33.665 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4447999999999998e-05 2024-04-23T16:30:33.665 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.666 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.666 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.666 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.027231+0000", 2024-04-23T16:30:33.666 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1253e-05 2024-04-23T16:30:33.666 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.666 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.666 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.666 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.666 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.666 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:122084 3.5 3:a0c15bd1:::benchmark_data_smithi188_89276_object122083:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-23T16:30:33.666 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:15.831874+0000", 2024-04-23T16:30:33.666 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 437.75770322199998, 2024-04-23T16:30:33.666 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.195459721, 2024-04-23T16:30:33.666 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.667 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.667 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.667 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:30:33.667 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:30:33.667 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:30:33.667 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 122084 2024-04-23T16:30:33.667 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.667 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.667 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.667 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.667 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.831874+0000", 2024-04-23T16:30:33.667 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.667 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.667 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.668 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.668 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.831874+0000", 2024-04-23T16:30:33.668 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.668 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.668 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.668 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.668 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.831876+0000", 2024-04-23T16:30:33.668 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.609e-06 2024-04-23T16:30:33.668 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.668 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.668 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.668 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.831930+0000", 2024-04-23T16:30:33.668 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4468000000000001e-05 2024-04-23T16:30:33.668 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.668 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.669 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.669 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.831941+0000", 2024-04-23T16:30:33.669 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1199000000000001e-05 2024-04-23T16:30:33.669 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.669 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.669 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.669 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.831949+0000", 2024-04-23T16:30:33.669 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.9729999999999997e-06 2024-04-23T16:30:33.669 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.669 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.669 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.669 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.831987+0000", 2024-04-23T16:30:33.669 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8041999999999999e-05 2024-04-23T16:30:33.669 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.669 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.670 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.670 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.832075+0000", 2024-04-23T16:30:33.670 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7659999999999998e-05 2024-04-23T16:30:33.670 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.670 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.670 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:30:33.670 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.832170+0000", 2024-04-23T16:30:33.670 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5235999999999994e-05 2024-04-23T16:30:33.670 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.670 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.670 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:30:33.670 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.835636+0000", 2024-04-23T16:30:33.670 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0034661689999999998 2024-04-23T16:30:33.670 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.671 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.671 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:30:33.671 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.027268+0000", 2024-04-23T16:30:33.671 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19163128900000001 2024-04-23T16:30:33.671 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.671 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.671 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.671 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.027298+0000", 2024-04-23T16:30:33.671 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0275999999999999e-05 2024-04-23T16:30:33.671 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.671 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.671 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.671 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.027334+0000", 2024-04-23T16:30:33.671 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5800000000000003e-05 2024-04-23T16:30:33.671 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.672 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.672 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.672 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.672 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.672 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:122085 3.5 3:acdffea3:::benchmark_data_smithi188_89276_object122084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-23T16:30:33.672 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:15.831962+0000", 2024-04-23T16:30:33.672 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 437.75761561899998, 2024-04-23T16:30:33.672 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19545873799999999, 2024-04-23T16:30:33.672 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.672 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.672 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.672 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:30:33.672 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:30:33.672 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:30:33.673 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 122085 2024-04-23T16:30:33.673 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.673 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.673 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.673 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.673 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.831962+0000", 2024-04-23T16:30:33.673 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.673 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.673 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.673 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.674 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.831962+0000", 2024-04-23T16:30:33.674 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.674 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.674 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.674 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.674 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.831962+0000", 2024-04-23T16:30:33.674 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8199999999999999e-07 2024-04-23T16:30:33.674 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.674 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.674 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.674 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.832008+0000", 2024-04-23T16:30:33.674 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5210000000000003e-05 2024-04-23T16:30:33.674 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.675 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.675 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.675 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.832013+0000", 2024-04-23T16:30:33.675 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8000000000000004e-06 2024-04-23T16:30:33.675 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.675 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.675 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.675 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.832017+0000", 2024-04-23T16:30:33.675 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1269999999999996e-06 2024-04-23T16:30:33.675 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.675 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.675 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.675 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.833864+0000", 2024-04-23T16:30:33.675 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001846667 2024-04-23T16:30:33.676 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.676 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.676 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.676 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.833991+0000", 2024-04-23T16:30:33.676 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012711999999999999 2024-04-23T16:30:33.676 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.676 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.676 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:30:33.676 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.834138+0000", 2024-04-23T16:30:33.676 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014632299999999999 2024-04-23T16:30:33.676 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.676 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.676 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:30:33.676 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.837388+0000", 2024-04-23T16:30:33.676 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003250237 2024-04-23T16:30:33.677 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.677 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.677 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:30:33.677 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.027361+0000", 2024-04-23T16:30:33.677 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.18997319200000001 2024-04-23T16:30:33.677 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.677 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.677 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.677 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.027389+0000", 2024-04-23T16:30:33.677 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8062e-05 2024-04-23T16:30:33.677 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.677 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.677 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.677 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.027420+0000", 2024-04-23T16:30:33.678 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1318000000000001e-05 2024-04-23T16:30:33.678 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.678 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.678 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.678 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.678 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.678 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:122087 3.5 3:aadc8321:::benchmark_data_smithi188_89276_object122086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-23T16:30:33.678 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:15.835574+0000", 2024-04-23T16:30:33.678 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 437.75400323899999, 2024-04-23T16:30:33.678 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19712622699999999, 2024-04-23T16:30:33.678 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.678 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.678 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.678 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:30:33.679 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:30:33.679 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:30:33.679 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 122087 2024-04-23T16:30:33.679 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.679 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.679 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.679 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.679 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.835574+0000", 2024-04-23T16:30:33.679 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.679 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.679 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.679 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.679 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.835574+0000", 2024-04-23T16:30:33.679 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.680 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.680 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.680 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.680 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.835575+0000", 2024-04-23T16:30:33.680 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.124e-06 2024-04-23T16:30:33.680 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.680 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.680 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.680 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.835634+0000", 2024-04-23T16:30:33.680 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8525e-05 2024-04-23T16:30:33.680 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.680 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.680 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.680 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.835659+0000", 2024-04-23T16:30:33.681 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5341999999999999e-05 2024-04-23T16:30:33.681 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.681 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.681 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.681 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.835664+0000", 2024-04-23T16:30:33.681 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5789999999999997e-06 2024-04-23T16:30:33.681 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.681 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.681 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.681 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.835704+0000", 2024-04-23T16:30:33.681 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0596999999999998e-05 2024-04-23T16:30:33.681 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.681 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.681 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.681 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.835819+0000", 2024-04-23T16:30:33.682 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011503899999999999 2024-04-23T16:30:33.682 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.682 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.682 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:30:33.682 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.835961+0000", 2024-04-23T16:30:33.682 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014166800000000001 2024-04-23T16:30:33.682 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.682 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.682 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:30:33.682 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.837544+0000", 2024-04-23T16:30:33.682 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0015829679999999999 2024-04-23T16:30:33.682 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.682 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.682 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:30:33.683 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.032637+0000", 2024-04-23T16:30:33.683 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19509323200000001 2024-04-23T16:30:33.683 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.683 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.683 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.683 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.032674+0000", 2024-04-23T16:30:33.683 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7286000000000002e-05 2024-04-23T16:30:33.683 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.683 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.683 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.683 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.032700+0000", 2024-04-23T16:30:33.683 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5867000000000002e-05 2024-04-23T16:30:33.683 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.683 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.684 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.684 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.684 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.684 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:122094 3.5 3:a6b7f65b:::benchmark_data_smithi188_89276_object122093:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-23T16:30:33.684 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:23:15.845813+0000", 2024-04-23T16:30:33.684 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 437.74376427499999, 2024-04-23T16:30:33.684 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.187034275, 2024-04-23T16:30:33.684 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.684 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.684 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.684 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:30:33.684 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:30:33.684 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:30:33.684 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 122094 2024-04-23T16:30:33.684 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.685 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.685 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.685 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.685 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.845813+0000", 2024-04-23T16:30:33.685 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.685 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.685 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.685 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.685 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.845813+0000", 2024-04-23T16:30:33.685 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.685 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.685 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.685 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.685 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.845822+0000", 2024-04-23T16:30:33.685 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7700000000000007e-06 2024-04-23T16:30:33.686 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.686 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.686 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.686 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.845880+0000", 2024-04-23T16:30:33.686 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8131999999999998e-05 2024-04-23T16:30:33.686 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.686 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.686 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.686 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.845892+0000", 2024-04-23T16:30:33.686 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2085e-05 2024-04-23T16:30:33.686 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.686 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.686 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.686 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.845901+0000", 2024-04-23T16:30:33.687 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8249999999999994e-06 2024-04-23T16:30:33.687 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.687 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.688 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.688 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.845960+0000", 2024-04-23T16:30:33.688 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9255000000000001e-05 2024-04-23T16:30:33.688 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.688 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.688 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.688 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.846073+0000", 2024-04-23T16:30:33.688 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000112506 2024-04-23T16:30:33.688 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.688 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.688 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:30:33.688 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.846158+0000", 2024-04-23T16:30:33.688 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5872999999999996e-05 2024-04-23T16:30:33.689 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.689 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.689 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:30:33.689 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:15.848843+0000", 2024-04-23T16:30:33.689 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0026842989999999998 2024-04-23T16:30:33.689 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.689 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.689 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:30:33.689 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.032748+0000", 2024-04-23T16:30:33.689 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.18390505700000001 2024-04-23T16:30:33.689 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.689 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.689 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.689 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.032805+0000", 2024-04-23T16:30:33.690 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.681e-05 2024-04-23T16:30:33.690 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.690 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.690 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.690 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:23:16.032847+0000", 2024-04-23T16:30:33.690 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2663000000000001e-05 2024-04-23T16:30:33.690 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.690 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.690 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.690 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.690 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.690 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.15486.0:190486 4.2 e323/274 4:4b455819:::benchmark_data_smithi188_101370_object27037:head v 323'11690, mlcod=323'11688)", 2024-04-23T16:30:33.690 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:29:13.724565+0000", 2024-04-23T16:30:33.690 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 79.865012167000003, 2024-04-23T16:30:33.691 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.029182217, 2024-04-23T16:30:33.691 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.691 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.691 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.691 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.691 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.691 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.691 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:13.724565+0000", 2024-04-23T16:30:33.691 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.691 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.691 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.691 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.691 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:13.724565+0000", 2024-04-23T16:30:33.691 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.692 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.692 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.692 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.692 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:13.724567+0000", 2024-04-23T16:30:33.692 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.753e-06 2024-04-23T16:30:33.692 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.692 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.692 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.692 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:13.724575+0000", 2024-04-23T16:30:33.692 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0749999999999998e-06 2024-04-23T16:30:33.692 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.692 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.692 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.692 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:13.724583+0000", 2024-04-23T16:30:33.693 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7079999999999997e-06 2024-04-23T16:30:33.693 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.693 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.693 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.693 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:13.724589+0000", 2024-04-23T16:30:33.693 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7399999999999998e-06 2024-04-23T16:30:33.693 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.693 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.693 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.693 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:13.724643+0000", 2024-04-23T16:30:33.693 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3338999999999999e-05 2024-04-23T16:30:33.693 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.693 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.693 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.694 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:13.724662+0000", 2024-04-23T16:30:33.694 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9675000000000001e-05 2024-04-23T16:30:33.694 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.694 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.694 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.694 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:13.753695+0000", 2024-04-23T16:30:33.694 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.029032644999999999 2024-04-23T16:30:33.694 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.694 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.694 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.694 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:13.753747+0000", 2024-04-23T16:30:33.694 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2281999999999999e-05 2024-04-23T16:30:33.694 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.694 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.695 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.695 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.695 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.695 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.15486.0:192501 4.2 e323/274 4:4dbcd0bf:::benchmark_data_smithi188_101370_object29052:head v 323'11815, mlcod=323'11813)", 2024-04-23T16:30:33.695 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:29:15.264466+0000", 2024-04-23T16:30:33.695 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 78.325110835999993, 2024-04-23T16:30:33.695 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.027113558999999999, 2024-04-23T16:30:33.695 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.695 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.695 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.695 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.695 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.695 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.695 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:15.264466+0000", 2024-04-23T16:30:33.695 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.696 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.696 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.696 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.696 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:15.264466+0000", 2024-04-23T16:30:33.696 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.696 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.696 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.696 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.696 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:15.264469+0000", 2024-04-23T16:30:33.696 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3159999999999999e-06 2024-04-23T16:30:33.696 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.696 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.696 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.696 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:15.264479+0000", 2024-04-23T16:30:33.697 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0091e-05 2024-04-23T16:30:33.697 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.697 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.697 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.697 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:15.264499+0000", 2024-04-23T16:30:33.697 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0369000000000001e-05 2024-04-23T16:30:33.697 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.697 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.697 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.697 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:15.264509+0000", 2024-04-23T16:30:33.697 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9590000000000007e-06 2024-04-23T16:30:33.697 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.697 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.697 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.698 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:15.264734+0000", 2024-04-23T16:30:33.698 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00022494100000000001 2024-04-23T16:30:33.698 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.698 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.698 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.698 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:15.264755+0000", 2024-04-23T16:30:33.698 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0387999999999999e-05 2024-04-23T16:30:33.698 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.698 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.698 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.698 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:15.291517+0000", 2024-04-23T16:30:33.698 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026762166 2024-04-23T16:30:33.698 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.698 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.699 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.699 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:15.291580+0000", 2024-04-23T16:30:33.699 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3329000000000003e-05 2024-04-23T16:30:33.699 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.699 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.699 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.699 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.699 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.699 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.15486.0:244837 4.1 e334/274 4:86182be1:::benchmark_data_smithi188_101370_object81388:head v 334'15251, mlcod=334'15247)", 2024-04-23T16:30:33.699 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:29:56.958584+0000", 2024-04-23T16:30:33.699 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 36.630992812000002, 2024-04-23T16:30:33.699 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.027970325000000001, 2024-04-23T16:30:33.699 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.699 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.699 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.700 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.700 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.700 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.700 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:56.958584+0000", 2024-04-23T16:30:33.700 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.700 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.700 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.700 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.700 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:56.958584+0000", 2024-04-23T16:30:33.700 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.700 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.700 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.700 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.700 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:56.958586+0000", 2024-04-23T16:30:33.700 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5370000000000001e-06 2024-04-23T16:30:33.701 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.701 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.701 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.701 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:56.958594+0000", 2024-04-23T16:30:33.701 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8050000000000007e-06 2024-04-23T16:30:33.701 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.701 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.701 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.701 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:56.958604+0000", 2024-04-23T16:30:33.701 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7200000000000001e-06 2024-04-23T16:30:33.701 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.701 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.701 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.701 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:56.958611+0000", 2024-04-23T16:30:33.702 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6799999999999993e-06 2024-04-23T16:30:33.702 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.702 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.702 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.702 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:56.967644+0000", 2024-04-23T16:30:33.702 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0090328979999999993 2024-04-23T16:30:33.702 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.702 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.702 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.702 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:56.967667+0000", 2024-04-23T16:30:33.702 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2447e-05 2024-04-23T16:30:33.702 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.702 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.702 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.703 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:56.986510+0000", 2024-04-23T16:30:33.703 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018843660000000002 2024-04-23T16:30:33.703 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.703 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.703 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.703 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:29:56.986555+0000", 2024-04-23T16:30:33.703 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4577999999999999e-05 2024-04-23T16:30:33.703 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.703 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.703 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.703 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.703 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.703 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.15486.0:284232 4.2 e344/274 4:4597e941:::benchmark_data_smithi188_101370_object120783:head v 344'17452, mlcod=344'17450)", 2024-04-23T16:30:33.703 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:30:29.880017+0000", 2024-04-23T16:30:33.703 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 3.7095605090000001, 2024-04-23T16:30:33.704 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033395872, 2024-04-23T16:30:33.704 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.704 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.704 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.704 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.704 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.704 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.704 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:29.880017+0000", 2024-04-23T16:30:33.704 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.704 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.704 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.704 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.704 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:29.880017+0000", 2024-04-23T16:30:33.704 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.705 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.705 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.705 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.705 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:29.880018+0000", 2024-04-23T16:30:33.705 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5349999999999999e-06 2024-04-23T16:30:33.705 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.705 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.705 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.705 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:29.880026+0000", 2024-04-23T16:30:33.705 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8439999999999998e-06 2024-04-23T16:30:33.705 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.705 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.705 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.705 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:29.880039+0000", 2024-04-23T16:30:33.706 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2376e-05 2024-04-23T16:30:33.706 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.706 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.706 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.706 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:29.880045+0000", 2024-04-23T16:30:33.706 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1070000000000001e-06 2024-04-23T16:30:33.706 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.706 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.706 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.706 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:29.880094+0000", 2024-04-23T16:30:33.706 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9404999999999999e-05 2024-04-23T16:30:33.706 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.706 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.706 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.707 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:29.880114+0000", 2024-04-23T16:30:33.707 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9743000000000001e-05 2024-04-23T16:30:33.707 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.707 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.707 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.707 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:29.913375+0000", 2024-04-23T16:30:33.707 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033261691000000003 2024-04-23T16:30:33.707 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.707 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.707 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.707 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:29.913413+0000", 2024-04-23T16:30:33.707 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7171e-05 2024-04-23T16:30:33.707 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.707 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.707 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.708 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.708 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.708 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.15486.0:286876 4.1 e345/274 4:82e03517:::benchmark_data_smithi188_101370_object123427:head v 345'17940, mlcod=345'17937)", 2024-04-23T16:30:33.708 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:30:32.196843+0000", 2024-04-23T16:30:33.708 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 1.3927345930000001, 2024-04-23T16:30:33.708 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028173776000000001, 2024-04-23T16:30:33.708 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.708 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.708 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.708 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.708 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.708 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.708 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.196843+0000", 2024-04-23T16:30:33.708 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.709 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.709 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.709 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.709 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.196843+0000", 2024-04-23T16:30:33.709 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.709 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.709 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.709 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.709 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.196845+0000", 2024-04-23T16:30:33.709 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.069e-06 2024-04-23T16:30:33.709 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.709 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.709 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.709 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.196854+0000", 2024-04-23T16:30:33.710 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2869999999999993e-06 2024-04-23T16:30:33.710 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.710 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.710 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.710 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.196865+0000", 2024-04-23T16:30:33.710 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0895e-05 2024-04-23T16:30:33.710 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.710 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.710 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.710 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.196872+0000", 2024-04-23T16:30:33.710 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5410000000000003e-06 2024-04-23T16:30:33.710 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.710 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.710 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.711 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.197208+0000", 2024-04-23T16:30:33.711 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00033572499999999999 2024-04-23T16:30:33.711 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.711 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.711 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.711 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.197227+0000", 2024-04-23T16:30:33.711 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8304e-05 2024-04-23T16:30:33.711 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.711 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.711 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.711 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.224960+0000", 2024-04-23T16:30:33.711 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.027733121999999999 2024-04-23T16:30:33.711 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.711 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.712 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.712 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.225016+0000", 2024-04-23T16:30:33.712 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6833000000000003e-05 2024-04-23T16:30:33.712 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.712 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.712 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.712 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.712 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.712 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.15486.0:286897 4.1 e345/274 4:808912ac:::benchmark_data_smithi188_101370_object123448:head v 345'17945, mlcod=345'17938)", 2024-04-23T16:30:33.712 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:30:32.213512+0000", 2024-04-23T16:30:33.712 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 1.3760650800000001, 2024-04-23T16:30:33.712 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.035241857000000001, 2024-04-23T16:30:33.712 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:33.712 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:33.713 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:30:33.713 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:33.713 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.713 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:33.713 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.213512+0000", 2024-04-23T16:30:33.713 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.713 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.713 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.713 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:33.713 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.213512+0000", 2024-04-23T16:30:33.713 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:33.713 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.713 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.713 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:33.713 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.213514+0000", 2024-04-23T16:30:33.714 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1179999999999999e-06 2024-04-23T16:30:33.714 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.714 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.714 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:33.714 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.213524+0000", 2024-04-23T16:30:33.714 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8819999999999996e-06 2024-04-23T16:30:33.714 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.714 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.714 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:33.714 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.213537+0000", 2024-04-23T16:30:33.714 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2724999999999999e-05 2024-04-23T16:30:33.714 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.714 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.714 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:33.715 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.213545+0000", 2024-04-23T16:30:33.715 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7419999999999997e-06 2024-04-23T16:30:33.715 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.715 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.715 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:33.715 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.220523+0000", 2024-04-23T16:30:33.715 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0069778669999999996 2024-04-23T16:30:33.715 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.715 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.715 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:33.715 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.220546+0000", 2024-04-23T16:30:33.715 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3193e-05 2024-04-23T16:30:33.715 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.715 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.717 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:30:33.717 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.248702+0000", 2024-04-23T16:30:33.717 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028155804 2024-04-23T16:30:33.718 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:33.718 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:33.719 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:30:33.719 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:32.248754+0000", 2024-04-23T16:30:33.719 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2525999999999998e-05 2024-04-23T16:30:33.719 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.719 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.719 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.719 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:33.719 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:33.719 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:30:33.720 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:33.804 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:33.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:33 smithi103 ceph-mon[39629]: pgmap v974: 27 pgs: 27 active+clean; 2.6 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:30:33.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:33 smithi148 ceph-mon[44520]: pgmap v974: 27 pgs: 27 active+clean; 2.6 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:30:33.936 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:30:33.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:33.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:30:34.042 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:30:34.068 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:33 smithi188 ceph-mon[44101]: pgmap v974: 27 pgs: 27 active+clean; 2.6 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:30:34.070 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:34.102 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:30:34.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:34.107 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:30:34.208 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:30:34.249 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:34.378 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-23T16:30:34.706 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:30:34.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:34.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:30:34.729 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:34.811 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:30:34.840 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:30:34.843 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:34.873 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:30:34.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:34.877 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:30:34.978 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:30:35.009 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:35.032 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:30:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:35.037 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:30:35.100 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.4 to 1 (10000) 2024-04-23T16:30:35.138 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:30:35.174 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:35.200 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:30:35.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:35.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:30:35.304 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:30:35.333 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:35.362 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:30:35.363 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:35 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3522834997' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-23T16:30:35.363 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:35 smithi188 ceph-mon[44101]: pgmap v975: 27 pgs: 27 active+clean; 2.4 GiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.37k op/s 2024-04-23T16:30:35.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:35.364 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:30:35.382 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:35 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3522834997' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-23T16:30:35.382 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:35 smithi148 ceph-mon[44520]: pgmap v975: 27 pgs: 27 active+clean; 2.4 GiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.37k op/s 2024-04-23T16:30:35.465 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:30:35.484 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:35 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3522834997' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-23T16:30:35.484 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:35 smithi103 ceph-mon[39629]: pgmap v975: 27 pgs: 27 active+clean; 2.4 GiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.37k op/s 2024-04-23T16:30:35.538 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:35.754 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:30:35.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:35.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:30:35.858 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:30:35.918 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:35.954 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:30:35.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:35.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:30:36.065 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:30:36.097 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:36.310 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:30:36.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:36.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:30:36.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3522834997' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-23T16:30:36.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:36 smithi148 ceph-mon[44520]: osdmap e346: 12 total, 12 up, 10 in 2024-04-23T16:30:36.401 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3522834997' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-23T16:30:36.401 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:36 smithi103 ceph-mon[39629]: osdmap e346: 12 total, 12 up, 10 in 2024-04-23T16:30:36.415 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:30:36.448 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3522834997' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-23T16:30:36.448 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:36 smithi188 ceph-mon[44101]: osdmap e346: 12 total, 12 up, 10 in 2024-04-23T16:30:36.449 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:36.474 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:30:36.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:36.478 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:30:36.581 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:30:36.617 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:36.777 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:30:36.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:36.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:30:36.881 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:30:36.913 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:36.942 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:30:36.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:36.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:30:37.049 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:30:37.079 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:37.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:37 smithi148 ceph-mon[44520]: pgmap v977: 27 pgs: 27 active+clean; 2.3 GiB data, 7.4 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 941 op/s 2024-04-23T16:30:37.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:37 smithi188 ceph-mon[44101]: pgmap v977: 27 pgs: 27 active+clean; 2.3 GiB data, 7.4 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 941 op/s 2024-04-23T16:30:37.301 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:30:37.302 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:30:37.302 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:37.302 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:292959 4.a 4:500fc257:::benchmark_data_smithi188_101370_object129510:head [delete] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-04-23T16:30:37.302 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:30:37.285961+0000", 2024-04-23T16:30:37.302 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.014214315, 2024-04-23T16:30:37.302 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014238186, 2024-04-23T16:30:37.302 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:37.302 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:37.302 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:30:37.302 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:30:37.302 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:30:37.302 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:30:37.303 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 292959 2024-04-23T16:30:37.303 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:37.303 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:37.303 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:37.303 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:37.303 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:37.285961+0000", 2024-04-23T16:30:37.303 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:37.303 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:37.303 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:37.303 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:37.303 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:37.285961+0000", 2024-04-23T16:30:37.303 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:37.303 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:37.303 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:37.304 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:37.304 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:37.285962+0000", 2024-04-23T16:30:37.304 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5570000000000001e-06 2024-04-23T16:30:37.304 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:37.304 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:37.304 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:37.304 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:37.285966+0000", 2024-04-23T16:30:37.304 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1699999999999999e-06 2024-04-23T16:30:37.304 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:37.304 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:37.304 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:37.304 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:37.285975+0000", 2024-04-23T16:30:37.304 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3960000000000006e-06 2024-04-23T16:30:37.304 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:37.305 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:37.305 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:37.305 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:37.285982+0000", 2024-04-23T16:30:37.305 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4379999999999998e-06 2024-04-23T16:30:37.305 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:37.305 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:37.305 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:37.305 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:37.286052+0000", 2024-04-23T16:30:37.305 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9391e-05 2024-04-23T16:30:37.305 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:37.305 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:37.305 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:37.305 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:37.286182+0000", 2024-04-23T16:30:37.305 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013028599999999999 2024-04-23T16:30:37.305 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:37.306 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:37.306 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:30:37.306 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:37.286311+0000", 2024-04-23T16:30:37.306 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012874800000000001 2024-04-23T16:30:37.306 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:37.306 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:37.306 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:30:37.306 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:37.296650+0000", 2024-04-23T16:30:37.306 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010339324 2024-04-23T16:30:37.306 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:37.306 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:37.306 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:37.306 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:37.306 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-04-23T16:30:37.307 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 1 2024-04-23T16:30:37.307 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:30:37.337 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:30:37.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:37 smithi103 ceph-mon[39629]: pgmap v977: 27 pgs: 27 active+clean; 2.3 GiB data, 7.4 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 941 op/s 2024-04-23T16:30:37.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:37.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:30:37.447 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:30:37.476 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:37.498 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:30:37.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:37.503 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:30:37.605 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:30:37.661 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:37.970 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-23T16:30:38.027 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:30:38.027 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:30:38.027 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:30:38.027 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:30:38.027 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:30:38.027 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:30:38.027 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:30:38.027 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:30:38.027 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:30:38.033 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:30:38.056 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:30:38.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:38.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:30:38.060 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:30:38.089 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:30:38.090 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:30:38.090 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:30:38.090 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:30:38.090 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:30:38.090 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:30:38.091 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:30:38.091 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:30:38.091 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:30:38.091 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:30:38.091 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:30:38.091 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:30:38.091 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:30:38.093 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-23T16:30:38.161 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:30:38.189 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:38.213 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:30:38.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:38.217 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:30:38.318 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:30:38.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:38 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/69140014' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:38.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:38.348 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:38.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:38 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/69140014' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:38.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:38.381 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:30:38.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:38.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:30:38.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:38 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/69140014' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:38.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:38.489 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:30:38.543 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:38.549 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:30:38.605 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:30:38.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:38.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:30:38.719 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:30:38.766 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:38.845 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:30:38.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:38.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:30:38.954 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:30:38.994 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:39.099 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:30:39.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:39.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:30:39.206 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:30:39.242 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:39.284 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:30:39.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:39.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:30:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:39 smithi103 ceph-mon[39629]: pgmap v978: 27 pgs: 27 active+clean; 2.2 GiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 995 op/s 2024-04-23T16:30:39.391 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:30:39.463 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.463 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.463 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.463 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.463 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.463 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.463 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.463 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.464 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.464 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.464 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.464 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:39 smithi148 ceph-mon[44520]: pgmap v978: 27 pgs: 27 active+clean; 2.2 GiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 995 op/s 2024-04-23T16:30:39.464 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:39 smithi188 ceph-mon[44101]: pgmap v978: 27 pgs: 27 active+clean; 2.2 GiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 995 op/s 2024-04-23T16:30:39.481 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:39.930 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:30:39.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:39.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:30:40.038 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:30:40.070 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:40.094 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:30:40.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:40.099 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:30:40.155 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:30:40.201 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:30:40.231 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:40.264 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:30:40.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:40.270 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:30:40.289 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:40 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3269698475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:30:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:40 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3269698475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:30:40.370 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:30:40.431 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:40.467 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:30:40.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:40 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3269698475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:30:40.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:40.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:30:40.581 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:30:40.622 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:40.662 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:30:40.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:40.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:30:40.665 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:30:40.666 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:30:40.766 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:30:40.798 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:40.821 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:30:40.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:40.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:30:40.925 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:30:40.956 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:40.978 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:30:40.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:40.981 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:30:41.082 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:30:41.128 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:30:41.162 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:41.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:41 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:30:41.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:41 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3269698475' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:30:41.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:41 smithi148 ceph-mon[44520]: osdmap e347: 12 total, 12 up, 10 in 2024-04-23T16:30:41.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:41 smithi148 ceph-mon[44520]: pgmap v980: 27 pgs: 27 active+clean; 2.0 GiB data, 6.7 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:30:41.419 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:30:41.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:41.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:30:41.427 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-23T16:30:41.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:41 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:30:41.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:41 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3269698475' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:30:41.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:41 smithi188 ceph-mon[44101]: osdmap e347: 12 total, 12 up, 10 in 2024-04-23T16:30:41.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:41 smithi188 ceph-mon[44101]: pgmap v980: 27 pgs: 27 active+clean; 2.0 GiB data, 6.7 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:30:41.526 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:30:41.555 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:41.580 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:30:41.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:41.584 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:30:41.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:41 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:30:41.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:41 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3269698475' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:30:41.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:41 smithi103 ceph-mon[39629]: osdmap e347: 12 total, 12 up, 10 in 2024-04-23T16:30:41.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:41 smithi103 ceph-mon[39629]: pgmap v980: 27 pgs: 27 active+clean; 2.0 GiB data, 6.7 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:30:41.685 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:30:41.724 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:41.759 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:30:41.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:41.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:30:41.863 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:30:41.897 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:41.900 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:30:41.975 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:30:41.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:41.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:30:42.084 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:30:42.125 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:42.250 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:30:42.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:42.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:30:42.360 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:30:42.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:42 smithi148 ceph-mon[44520]: 4.0 scrub starts 2024-04-23T16:30:42.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:42 smithi148 ceph-mon[44520]: 1.8 scrub starts 2024-04-23T16:30:42.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:42 smithi148 ceph-mon[44520]: 1.8 scrub ok 2024-04-23T16:30:42.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:42 smithi148 ceph-mon[44520]: 4.0 scrub ok 2024-04-23T16:30:42.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:42 smithi148 ceph-mon[44520]: 4.b scrub starts 2024-04-23T16:30:42.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:42 smithi148 ceph-mon[44520]: 4.b scrub ok 2024-04-23T16:30:42.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:42 smithi148 ceph-mon[44520]: 1.a scrub starts 2024-04-23T16:30:42.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:42 smithi148 ceph-mon[44520]: 1.a scrub ok 2024-04-23T16:30:42.401 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:42.436 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:30:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:42.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:30:42.447 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:42.448 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:42 smithi103 ceph-mon[39629]: 4.0 scrub starts 2024-04-23T16:30:42.448 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:42 smithi103 ceph-mon[39629]: 1.8 scrub starts 2024-04-23T16:30:42.448 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:42 smithi103 ceph-mon[39629]: 1.8 scrub ok 2024-04-23T16:30:42.448 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:42 smithi103 ceph-mon[39629]: 4.0 scrub ok 2024-04-23T16:30:42.448 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:42 smithi103 ceph-mon[39629]: 4.b scrub starts 2024-04-23T16:30:42.449 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:42 smithi103 ceph-mon[39629]: 4.b scrub ok 2024-04-23T16:30:42.449 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:42 smithi103 ceph-mon[39629]: 1.a scrub starts 2024-04-23T16:30:42.449 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:42 smithi103 ceph-mon[39629]: 1.a scrub ok 2024-04-23T16:30:42.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:42 smithi188 ceph-mon[44101]: 4.0 scrub starts 2024-04-23T16:30:42.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:42 smithi188 ceph-mon[44101]: 1.8 scrub starts 2024-04-23T16:30:42.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:42 smithi188 ceph-mon[44101]: 1.8 scrub ok 2024-04-23T16:30:42.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:42 smithi188 ceph-mon[44101]: 4.0 scrub ok 2024-04-23T16:30:42.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:42 smithi188 ceph-mon[44101]: 4.b scrub starts 2024-04-23T16:30:42.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:42 smithi188 ceph-mon[44101]: 4.b scrub ok 2024-04-23T16:30:42.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:42 smithi188 ceph-mon[44101]: 1.a scrub starts 2024-04-23T16:30:42.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:42 smithi188 ceph-mon[44101]: 1.a scrub ok 2024-04-23T16:30:42.547 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:30:42.588 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:42.716 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:30:42.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:42.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:30:42.819 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:30:42.851 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:42.881 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:30:42.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:42.887 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:30:42.988 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:30:43.023 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:43.049 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:30:43.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:43.054 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:30:43.155 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:30:43.159 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.5 to 1 (10000) 2024-04-23T16:30:43.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:43 smithi188 ceph-mon[44101]: 4.d scrub starts 2024-04-23T16:30:43.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:43 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1230464271' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-23T16:30:43.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:43 smithi188 ceph-mon[44101]: pgmap v981: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.9 GiB data, 6.5 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-23T16:30:43.223 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:43 smithi103 ceph-mon[39629]: 4.d scrub starts 2024-04-23T16:30:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:43 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1230464271' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-23T16:30:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:43 smithi103 ceph-mon[39629]: pgmap v981: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.9 GiB data, 6.5 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-23T16:30:43.366 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:43 smithi148 ceph-mon[44520]: 4.d scrub starts 2024-04-23T16:30:43.366 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:43 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1230464271' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-23T16:30:43.366 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:43 smithi148 ceph-mon[44520]: pgmap v981: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.9 GiB data, 6.5 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-23T16:30:43.374 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:30:43.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:43.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:30:43.480 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:30:43.511 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:43.533 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:30:43.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:43.539 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:30:43.639 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:30:43.673 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:43.784 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:30:44.105 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:30:44.105 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:30:44.115 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-23T16:30:44.125 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:30:44.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:44.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:30:44.232 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:30:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:44.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:44 smithi103 ceph-mon[39629]: 4.d scrub ok 2024-04-23T16:30:44.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:44 smithi103 ceph-mon[39629]: 4.7 scrub starts 2024-04-23T16:30:44.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:44 smithi103 ceph-mon[39629]: 4.7 scrub ok 2024-04-23T16:30:44.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:44 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1230464271' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-23T16:30:44.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:44 smithi103 ceph-mon[39629]: osdmap e348: 12 total, 12 up, 10 in 2024-04-23T16:30:44.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:44 smithi103 ceph-mon[39629]: 4.a scrub starts 2024-04-23T16:30:44.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:44 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:30:44.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:44 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:30:44.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:44 smithi103 ceph-mon[39629]: 1.5 scrub starts 2024-04-23T16:30:44.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:44 smithi103 ceph-mon[39629]: 1.5 scrub ok 2024-04-23T16:30:44.358 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:30:44.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:44.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:30:44.468 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:30:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:44 smithi188 ceph-mon[44101]: 4.d scrub ok 2024-04-23T16:30:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:44 smithi188 ceph-mon[44101]: 4.7 scrub starts 2024-04-23T16:30:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:44 smithi188 ceph-mon[44101]: 4.7 scrub ok 2024-04-23T16:30:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:44 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1230464271' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-23T16:30:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:44 smithi188 ceph-mon[44101]: osdmap e348: 12 total, 12 up, 10 in 2024-04-23T16:30:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:44 smithi188 ceph-mon[44101]: 4.a scrub starts 2024-04-23T16:30:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:44 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:30:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:44 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:30:44.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:44 smithi188 ceph-mon[44101]: 1.5 scrub starts 2024-04-23T16:30:44.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:44 smithi188 ceph-mon[44101]: 1.5 scrub ok 2024-04-23T16:30:44.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:44 smithi148 ceph-mon[44520]: 4.d scrub ok 2024-04-23T16:30:44.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:44 smithi148 ceph-mon[44520]: 4.7 scrub starts 2024-04-23T16:30:44.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:44 smithi148 ceph-mon[44520]: 4.7 scrub ok 2024-04-23T16:30:44.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:44 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1230464271' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-23T16:30:44.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:44 smithi148 ceph-mon[44520]: osdmap e348: 12 total, 12 up, 10 in 2024-04-23T16:30:44.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:44 smithi148 ceph-mon[44520]: 4.a scrub starts 2024-04-23T16:30:44.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:44 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:30:44.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:44 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:30:44.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:44 smithi148 ceph-mon[44520]: 1.5 scrub starts 2024-04-23T16:30:44.503 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:44 smithi148 ceph-mon[44520]: 1.5 scrub ok 2024-04-23T16:30:44.503 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:44.525 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:30:44.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:44.531 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:30:44.631 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:30:44.686 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:44.691 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:44.836 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:30:44.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:44.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:30:44.885 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-23T16:30:44.949 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:30:44.981 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:45.025 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:30:45.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:45.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:30:45.134 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:30:45.177 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:45.346 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:30:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:45 smithi103 ceph-mon[39629]: 4.a scrub ok 2024-04-23T16:30:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:45 smithi103 ceph-mon[39629]: pgmap v983: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.7 GiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.37k op/s 2024-04-23T16:30:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:45 smithi103 ceph-mon[39629]: 4.4 scrub starts 2024-04-23T16:30:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:45 smithi188 ceph-mon[44101]: 4.a scrub ok 2024-04-23T16:30:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:45 smithi188 ceph-mon[44101]: pgmap v983: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.7 GiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.37k op/s 2024-04-23T16:30:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:45 smithi188 ceph-mon[44101]: 4.4 scrub starts 2024-04-23T16:30:45.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:45 smithi148 ceph-mon[44520]: 4.a scrub ok 2024-04-23T16:30:45.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:45 smithi148 ceph-mon[44520]: pgmap v983: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.7 GiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.37k op/s 2024-04-23T16:30:45.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:45 smithi148 ceph-mon[44520]: 4.4 scrub starts 2024-04-23T16:30:45.736 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:30:45.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:45.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:30:45.840 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:30:45.871 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:46.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:46 smithi148 ceph-mon[44520]: 4.4 scrub ok 2024-04-23T16:30:46.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:46 smithi148 ceph-mon[44520]: 4.6 scrub starts 2024-04-23T16:30:46.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:46 smithi188 ceph-mon[44101]: 4.4 scrub ok 2024-04-23T16:30:46.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:46 smithi188 ceph-mon[44101]: 4.6 scrub starts 2024-04-23T16:30:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:46 smithi103 ceph-mon[39629]: 4.4 scrub ok 2024-04-23T16:30:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:46 smithi103 ceph-mon[39629]: 4.6 scrub starts 2024-04-23T16:30:47.067 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:47.089 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:30:47.118 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:30:47.145 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:30:47.172 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:30:47.197 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:30:47.231 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:30:47.267 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:30:47.293 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:30:47.316 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:30:47.347 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:30:47.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:47 smithi148 ceph-mon[44520]: 4.6 scrub ok 2024-04-23T16:30:47.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:47 smithi148 ceph-mon[44520]: 4.1 scrub starts 2024-04-23T16:30:47.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:47 smithi148 ceph-mon[44520]: pgmap v984: 27 pgs: 27 active+clean; 1.6 GiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:30:47.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:47 smithi148 ceph-mon[44520]: 4.2 scrub starts 2024-04-23T16:30:47.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:47 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4017848144' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:47.377 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:30:47.410 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:30:47.411 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:30:47.411 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:30:47.411 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:30:47.411 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:30:47.411 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:30:47.411 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:30:47.411 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:30:47.411 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:30:47.411 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:30:47.411 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:30:47.412 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:30:47.412 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:30:47.458 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:30:47.458 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:30:47.458 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.458 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.15486.0:304633 4.a e348/274 4:512d5600:::benchmark_data_smithi188_101370_object141184:head v 348'19139, mlcod=348'19134)", 2024-04-23T16:30:47.458 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:30:47.435485+0000", 2024-04-23T16:30:47.458 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.015391238999999999, 2024-04-23T16:30:47.459 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015418645999999999, 2024-04-23T16:30:47.459 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:47.459 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:47.459 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:30:47.459 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:47.459 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.459 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:47.459 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.435485+0000", 2024-04-23T16:30:47.459 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:47.459 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.459 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.459 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:47.459 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.435485+0000", 2024-04-23T16:30:47.459 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:47.460 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.460 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.460 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:47.460 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.435487+0000", 2024-04-23T16:30:47.460 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2470000000000001e-06 2024-04-23T16:30:47.460 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.460 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.460 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:47.460 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.435494+0000", 2024-04-23T16:30:47.460 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6199999999999999e-06 2024-04-23T16:30:47.460 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.460 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.460 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:47.461 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.435505+0000", 2024-04-23T16:30:47.461 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0593e-05 2024-04-23T16:30:47.461 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.461 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.461 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:47.461 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.435512+0000", 2024-04-23T16:30:47.461 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0450000000000003e-06 2024-04-23T16:30:47.461 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.461 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.461 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:47.461 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.436340+0000", 2024-04-23T16:30:47.461 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00082838599999999997 2024-04-23T16:30:47.461 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.461 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.461 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:47.462 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.436357+0000", 2024-04-23T16:30:47.462 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6631000000000001e-05 2024-04-23T16:30:47.462 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:47.462 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:47.462 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:47.462 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.462 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.462 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.15486.0:304644 4.0 e348/274 4:0c5f17c6:::benchmark_data_smithi188_101370_object141195:head v 348'19330, mlcod=348'19328)", 2024-04-23T16:30:47.462 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:30:47.441933+0000", 2024-04-23T16:30:47.462 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.0089433710000000003, 2024-04-23T16:30:47.462 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.009036957, 2024-04-23T16:30:47.462 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:47.462 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:47.463 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:30:47.463 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:47.463 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.463 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:47.463 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.441933+0000", 2024-04-23T16:30:47.463 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:47.463 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.463 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.463 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:47.463 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.441933+0000", 2024-04-23T16:30:47.463 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:47.463 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.463 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.464 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:47.464 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.441935+0000", 2024-04-23T16:30:47.464 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.562e-06 2024-04-23T16:30:47.464 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.464 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.464 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:47.464 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.441941+0000", 2024-04-23T16:30:47.464 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5080000000000002e-06 2024-04-23T16:30:47.464 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.464 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.464 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:47.464 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.441954+0000", 2024-04-23T16:30:47.464 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2544000000000001e-05 2024-04-23T16:30:47.464 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.465 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.465 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:47.465 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.441960+0000", 2024-04-23T16:30:47.465 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1700000000000002e-06 2024-04-23T16:30:47.465 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.465 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.465 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:47.465 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.442010+0000", 2024-04-23T16:30:47.465 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9886999999999999e-05 2024-04-23T16:30:47.465 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.465 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.465 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:47.465 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.442030+0000", 2024-04-23T16:30:47.465 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0347e-05 2024-04-23T16:30:47.466 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:47.466 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:47.466 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:47.466 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.466 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.466 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.15486.0:304634 4.a e348/274 4:5fd8a011:::benchmark_data_smithi188_101370_object141185:head v 348'19140, mlcod=348'19134)", 2024-04-23T16:30:47.466 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:30:47.445336+0000", 2024-04-23T16:30:47.466 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.0055401499999999998, 2024-04-23T16:30:47.466 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0057108339999999997, 2024-04-23T16:30:47.466 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:47.466 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:47.466 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:30:47.466 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:47.466 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.467 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:47.467 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.445336+0000", 2024-04-23T16:30:47.467 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:47.467 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.467 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.467 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:47.467 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.445336+0000", 2024-04-23T16:30:47.467 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:47.467 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.467 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.467 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:47.467 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.445338+0000", 2024-04-23T16:30:47.467 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.319e-06 2024-04-23T16:30:47.467 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.468 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.468 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:47.468 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.445347+0000", 2024-04-23T16:30:47.468 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1370000000000008e-06 2024-04-23T16:30:47.468 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.468 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.468 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:47.468 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.445362+0000", 2024-04-23T16:30:47.468 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4701e-05 2024-04-23T16:30:47.468 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.468 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.468 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:47.468 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.445369+0000", 2024-04-23T16:30:47.468 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7100000000000007e-06 2024-04-23T16:30:47.469 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.469 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.469 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:47.469 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.445478+0000", 2024-04-23T16:30:47.469 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010864799999999999 2024-04-23T16:30:47.469 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.469 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.469 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:47.469 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.445499+0000", 2024-04-23T16:30:47.469 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0596999999999999e-05 2024-04-23T16:30:47.469 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:47.469 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:47.469 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:47.469 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.470 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.470 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.15486.0:304645 4.a e348/274 4:51ec89c4:::benchmark_data_smithi188_101370_object141196:head v 348'19141, mlcod=348'19134)", 2024-04-23T16:30:47.470 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:30:47.445939+0000", 2024-04-23T16:30:47.470 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.0049372449999999998, 2024-04-23T16:30:47.470 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.005171274, 2024-04-23T16:30:47.470 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:47.470 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:47.470 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:30:47.470 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:47.470 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.470 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:47.470 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.445939+0000", 2024-04-23T16:30:47.470 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:47.470 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.471 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.471 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:47.471 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.445939+0000", 2024-04-23T16:30:47.471 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:47.471 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.471 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.471 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:47.471 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.445941+0000", 2024-04-23T16:30:47.471 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.466e-06 2024-04-23T16:30:47.471 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.471 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.471 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:47.471 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.445948+0000", 2024-04-23T16:30:47.471 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7229999999999998e-06 2024-04-23T16:30:47.472 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.472 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.472 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:47.472 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.445957+0000", 2024-04-23T16:30:47.472 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8600000000000005e-06 2024-04-23T16:30:47.472 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.472 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.472 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:47.472 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.445964+0000", 2024-04-23T16:30:47.472 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7769999999999997e-06 2024-04-23T16:30:47.472 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.472 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.472 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:47.472 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.449865+0000", 2024-04-23T16:30:47.473 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0039007790000000001 2024-04-23T16:30:47.473 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.473 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.473 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:47.473 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.449898+0000", 2024-04-23T16:30:47.473 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3297999999999999e-05 2024-04-23T16:30:47.473 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:47.473 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:47.473 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:47.473 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.473 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.473 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.15486.0:304649 4.8 e348/274 4:1db08e62:::benchmark_data_smithi188_101370_object141200:head v 348'18923, mlcod=348'18921)", 2024-04-23T16:30:47.474 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:30:47.446566+0000", 2024-04-23T16:30:47.474 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.0043109489999999997, 2024-04-23T16:30:47.474 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045853200000000004, 2024-04-23T16:30:47.474 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:30:47.474 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:30:47.474 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:30:47.474 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:30:47.474 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.474 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:30:47.474 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.446566+0000", 2024-04-23T16:30:47.474 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:47.474 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.474 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.474 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:30:47.475 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.446566+0000", 2024-04-23T16:30:47.475 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:30:47.475 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.475 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.475 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:30:47.475 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.446567+0000", 2024-04-23T16:30:47.475 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3009999999999999e-06 2024-04-23T16:30:47.475 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.475 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.475 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:30:47.475 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.446578+0000", 2024-04-23T16:30:47.475 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0746e-05 2024-04-23T16:30:47.475 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.475 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.476 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:30:47.476 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.446592+0000", 2024-04-23T16:30:47.476 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4141000000000001e-05 2024-04-23T16:30:47.476 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.476 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.476 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:30:47.476 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.446601+0000", 2024-04-23T16:30:47.476 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9330000000000007e-06 2024-04-23T16:30:47.476 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.476 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.476 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:30:47.476 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.446688+0000", 2024-04-23T16:30:47.476 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7533000000000004e-05 2024-04-23T16:30:47.476 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:30:47.477 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:30:47.477 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:30:47.477 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:47.446715+0000", 2024-04-23T16:30:47.477 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7100000000000001e-05 2024-04-23T16:30:47.477 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:47.477 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:30:47.477 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:47.477 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:30:47.477 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-04-23T16:30:47.477 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 5 2024-04-23T16:30:47.477 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:30:47.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:47 smithi188 ceph-mon[44101]: 4.6 scrub ok 2024-04-23T16:30:47.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:47 smithi188 ceph-mon[44101]: 4.1 scrub starts 2024-04-23T16:30:47.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:47 smithi188 ceph-mon[44101]: pgmap v984: 27 pgs: 27 active+clean; 1.6 GiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:30:47.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:47 smithi188 ceph-mon[44101]: 4.2 scrub starts 2024-04-23T16:30:47.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:47 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4017848144' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:47.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:47 smithi103 ceph-mon[39629]: 4.6 scrub ok 2024-04-23T16:30:47.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:47 smithi103 ceph-mon[39629]: 4.1 scrub starts 2024-04-23T16:30:47.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:47 smithi103 ceph-mon[39629]: pgmap v984: 27 pgs: 27 active+clean; 1.6 GiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:30:47.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:47 smithi103 ceph-mon[39629]: 4.2 scrub starts 2024-04-23T16:30:47.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:47 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4017848144' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:48.229 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:30:48.257 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:48 smithi103 ceph-mon[39629]: 4.1 scrub ok 2024-04-23T16:30:48.257 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:48 smithi103 ceph-mon[39629]: 4.2 scrub ok 2024-04-23T16:30:48.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:48 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:30:48.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:48 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:30:48.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:48 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3520037140' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:30:48.351 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-23T16:30:48.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:48 smithi188 ceph-mon[44101]: 4.1 scrub ok 2024-04-23T16:30:48.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:48 smithi188 ceph-mon[44101]: 4.2 scrub ok 2024-04-23T16:30:48.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:48 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:30:48.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:48 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:30:48.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:48 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3520037140' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:30:48.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:48 smithi148 ceph-mon[44520]: 4.1 scrub ok 2024-04-23T16:30:48.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:48 smithi148 ceph-mon[44520]: 4.2 scrub ok 2024-04-23T16:30:48.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:48.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:48 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:30:48.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:48 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:30:48.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:48 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3520037140' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:30:48.776 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:30:49.309 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:49.309 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:49.309 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:49 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:30:49.309 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:49 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:30:49.310 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:49 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:30:49.310 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:49 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:30:49.310 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:49 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3520037140' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:30:49.310 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:49 smithi103 ceph-mon[39629]: osdmap e349: 12 total, 12 up, 10 in 2024-04-23T16:30:49.310 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:49 smithi103 ceph-mon[39629]: 1.9 scrub starts 2024-04-23T16:30:49.310 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:49 smithi103 ceph-mon[39629]: 1.9 scrub ok 2024-04-23T16:30:49.310 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:49 smithi103 ceph-mon[39629]: 1.7 scrub starts 2024-04-23T16:30:49.310 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:49 smithi103 ceph-mon[39629]: 1.7 scrub ok 2024-04-23T16:30:49.310 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:49 smithi103 ceph-mon[39629]: pgmap v986: 27 pgs: 27 active+clean; 1.5 GiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 989 op/s 2024-04-23T16:30:49.327 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:30:49.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:49.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:30:49.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:49.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:49.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:49 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:30:49.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:49 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:30:49.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:49 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:30:49.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:49 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:30:49.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:49 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3520037140' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:30:49.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:49 smithi148 ceph-mon[44520]: osdmap e349: 12 total, 12 up, 10 in 2024-04-23T16:30:49.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:49 smithi148 ceph-mon[44520]: 1.9 scrub starts 2024-04-23T16:30:49.350 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:49 smithi148 ceph-mon[44520]: 1.9 scrub ok 2024-04-23T16:30:49.350 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:49 smithi148 ceph-mon[44520]: 1.7 scrub starts 2024-04-23T16:30:49.350 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:49 smithi148 ceph-mon[44520]: 1.7 scrub ok 2024-04-23T16:30:49.350 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:49 smithi148 ceph-mon[44520]: pgmap v986: 27 pgs: 27 active+clean; 1.5 GiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 989 op/s 2024-04-23T16:30:49.431 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:30:49.463 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:49.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:49.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:49.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:49 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:30:49.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:49 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:30:49.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:49 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:30:49.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:49 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:30:49.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:49 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3520037140' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:30:49.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:49 smithi188 ceph-mon[44101]: osdmap e349: 12 total, 12 up, 10 in 2024-04-23T16:30:49.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:49 smithi188 ceph-mon[44101]: 1.9 scrub starts 2024-04-23T16:30:49.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:49 smithi188 ceph-mon[44101]: 1.9 scrub ok 2024-04-23T16:30:49.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:49 smithi188 ceph-mon[44101]: 1.7 scrub starts 2024-04-23T16:30:49.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:49 smithi188 ceph-mon[44101]: 1.7 scrub ok 2024-04-23T16:30:49.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:49 smithi188 ceph-mon[44101]: pgmap v986: 27 pgs: 27 active+clean; 1.5 GiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 989 op/s 2024-04-23T16:30:49.522 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:30:49.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:49.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:30:49.627 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:30:49.661 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:49.683 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:30:49.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:49.688 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:30:49.789 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:30:49.827 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:49.855 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:30:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:49.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:30:49.959 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:30:49.987 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:50.013 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:30:50.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:50.018 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:30:50.119 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:30:50.191 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:50.273 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:30:50.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:50.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:30:50.374 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:30:50.386 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:30:50.454 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:50.498 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:30:50.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:50.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:30:50.606 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:30:50.686 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:50.726 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:30:50.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:50.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:30:50.779 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:30:50.779 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:30:50.779 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2024-04-23T16:30:50.779 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2024-04-23T16:30:50.779 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:30:50.828 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:30:50.860 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:50.883 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:30:50.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:50.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:30:50.988 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:30:51.021 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:51.042 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:30:51.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:51.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:30:51.077 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:30:51.149 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:30:51.207 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:51.241 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:30:51.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:51.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:30:51.323 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:51.350 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:30:51.399 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:51.546 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-23T16:30:51.656 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:30:51.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:51.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:30:51.751 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:51 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:30:51.751 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:51 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:30:51.751 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:51 smithi148 ceph-mon[44520]: pgmap v987: 27 pgs: 27 active+clean; 1.3 GiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:30:51.751 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:51 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2943460784' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:30:51.751 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:51 smithi148 ceph-mon[44520]: from='client.25618 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:30:51.759 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:30:51.762 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.6 to 1 (10000) 2024-04-23T16:30:51.798 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:51.994 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:30:52.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:51 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:30:52.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:51 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:30:52.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:51 smithi103 ceph-mon[39629]: pgmap v987: 27 pgs: 27 active+clean; 1.3 GiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:30:52.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:51 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2943460784' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:30:52.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:51 smithi103 ceph-mon[39629]: from='client.25618 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:30:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:51 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:30:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:51 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:30:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:51 smithi188 ceph-mon[44101]: pgmap v987: 27 pgs: 27 active+clean; 1.3 GiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:30:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:51 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2943460784' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:30:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:51 smithi188 ceph-mon[44101]: from='client.25618 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:30:52.480 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:30:52.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:52.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:30:52.584 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:30:52.617 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:52.639 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:30:52.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:52.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:30:52.745 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:30:52.770 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:52 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:30:52.770 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:52 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:30:52.770 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:52 smithi103 ceph-mon[39629]: from='client.25618 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:30:52.770 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:52 smithi103 ceph-mon[39629]: osdmap e350: 12 total, 12 up, 10 in 2024-04-23T16:30:52.782 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:52.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:52 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:30:52.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:52 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:30:52.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:52 smithi148 ceph-mon[44520]: from='client.25618 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:30:52.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:52 smithi148 ceph-mon[44520]: osdmap e350: 12 total, 12 up, 10 in 2024-04-23T16:30:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:52 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:30:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:52 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:30:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:52 smithi188 ceph-mon[44101]: from='client.25618 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:30:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:52 smithi188 ceph-mon[44101]: osdmap e350: 12 total, 12 up, 10 in 2024-04-23T16:30:53.140 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:30:53.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:53.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:30:53.247 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:30:53.276 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:53.633 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:53.638 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:30:53.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:53.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:30:53.672 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-23T16:30:53.747 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:30:53.799 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:53.900 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:53 smithi103 ceph-mon[39629]: pgmap v989: 27 pgs: 27 active+clean; 1.2 GiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 994 op/s 2024-04-23T16:30:53.901 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:53 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:30:53.901 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:53 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:30:54.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:53 smithi148 ceph-mon[44520]: pgmap v989: 27 pgs: 27 active+clean; 1.2 GiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 994 op/s 2024-04-23T16:30:54.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:53 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:30:54.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:53 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:30:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:53 smithi188 ceph-mon[44101]: pgmap v989: 27 pgs: 27 active+clean; 1.2 GiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 994 op/s 2024-04-23T16:30:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:53 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:30:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:53 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:30:54.468 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:30:54.493 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:30:54.521 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:30:54.561 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:30:54.593 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:30:54.627 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:30:54.661 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:30:54.686 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:30:54.719 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:30:54.744 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:30:54.766 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:30:54.792 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:30:54.796 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:30:54.796 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:30:54.797 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:30:54.797 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:30:54.797 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:30:54.797 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:30:54.797 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:30:54.797 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:30:54.797 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:30:54.797 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:30:54.797 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:30:54.797 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:30:54.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:54 smithi103 ceph-mon[39629]: osdmap e351: 12 total, 12 up, 10 in 2024-04-23T16:30:54.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:54 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/912036225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:54.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:55.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:54 smithi148 ceph-mon[44520]: osdmap e351: 12 total, 12 up, 10 in 2024-04-23T16:30:55.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:54 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/912036225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:55.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:55.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:54 smithi188 ceph-mon[44101]: osdmap e351: 12 total, 12 up, 10 in 2024-04-23T16:30:55.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:54 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/912036225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:30:55.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:55 smithi103 ceph-mon[39629]: pgmap v991: 27 pgs: 27 active+clean; 980 MiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.47k op/s 2024-04-23T16:30:56.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:55 smithi148 ceph-mon[44520]: pgmap v991: 27 pgs: 27 active+clean; 980 MiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.47k op/s 2024-04-23T16:30:56.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:55 smithi188 ceph-mon[44101]: pgmap v991: 27 pgs: 27 active+clean; 980 MiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.47k op/s 2024-04-23T16:30:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:30:56.279 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:30:56.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:56.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:30:56.391 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:30:56.421 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:56.674 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:30:56.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:56.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:30:56.762 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:56.784 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:30:56.829 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:57.036 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:56 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:30:57.036 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:56 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:30:57.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:56 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:30:57.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:56 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:30:57.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:56 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:30:57.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:56 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:30:57.726 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:30:57.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:57.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:30:57.752 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:30:57.830 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:30:57.864 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:57.887 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:30:57.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:57.895 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:30:57.996 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:30:58.038 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:57 smithi148 ceph-mon[44520]: pgmap v992: 27 pgs: 27 active+clean; 933 MiB data, 4.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:30:58.039 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:57 smithi148 ceph-mon[44520]: osdmap e352: 12 total, 12 up, 10 in 2024-04-23T16:30:58.039 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:57 smithi148 ceph-mon[44520]: 4.8 scrub starts 2024-04-23T16:30:58.039 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:57 smithi148 ceph-mon[44520]: 4.8 scrub ok 2024-04-23T16:30:58.046 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:58.078 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:30:58.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:58.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:30:58.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:57 smithi103 ceph-mon[39629]: pgmap v992: 27 pgs: 27 active+clean; 933 MiB data, 4.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:30:58.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:57 smithi103 ceph-mon[39629]: osdmap e352: 12 total, 12 up, 10 in 2024-04-23T16:30:58.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:57 smithi103 ceph-mon[39629]: 4.8 scrub starts 2024-04-23T16:30:58.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:57 smithi103 ceph-mon[39629]: 4.8 scrub ok 2024-04-23T16:30:58.121 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:30:58.184 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:30:58.216 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:58.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:57 smithi188 ceph-mon[44101]: pgmap v992: 27 pgs: 27 active+clean; 933 MiB data, 4.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-23T16:30:58.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:57 smithi188 ceph-mon[44101]: osdmap e352: 12 total, 12 up, 10 in 2024-04-23T16:30:58.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:57 smithi188 ceph-mon[44101]: 4.8 scrub starts 2024-04-23T16:30:58.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:57 smithi188 ceph-mon[44101]: 4.8 scrub ok 2024-04-23T16:30:58.244 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:30:58.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:30:58.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:30:58.349 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:30:58.377 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:30:58.377 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:30:58.381 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:30:58.842 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:30:58.966 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:58 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:30:58.966 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:58 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:30:58.967 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:58 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4084042323' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:30:58.967 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:58 smithi103 ceph-mon[39629]: from='client.25630 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:30:59.094 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:58 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:30:59.094 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:58 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:30:59.095 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:58 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4084042323' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:30:59.095 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:58 smithi148 ceph-mon[44520]: from='client.25630 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:30:59.132 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-23T16:30:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:58 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:30:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:58 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:30:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:58 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4084042323' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:30:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:58 smithi188 ceph-mon[44101]: from='client.25630 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:30:59.489 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:31:00.024 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:59 smithi188 ceph-mon[44101]: pgmap v994: 27 pgs: 27 active+clean; 840 MiB data, 4.3 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 851 op/s 2024-04-23T16:31:00.025 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:59 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:31:00.025 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:59 smithi188 ceph-mon[44101]: from='client.25630 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:31:00.025 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:30:59 smithi188 ceph-mon[44101]: osdmap e353: 12 total, 12 up, 10 in 2024-04-23T16:31:00.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:59 smithi103 ceph-mon[39629]: pgmap v994: 27 pgs: 27 active+clean; 840 MiB data, 4.3 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 851 op/s 2024-04-23T16:31:00.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:59 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:31:00.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:59 smithi103 ceph-mon[39629]: from='client.25630 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:31:00.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:30:59 smithi103 ceph-mon[39629]: osdmap e353: 12 total, 12 up, 10 in 2024-04-23T16:31:00.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:59 smithi148 ceph-mon[44520]: pgmap v994: 27 pgs: 27 active+clean; 840 MiB data, 4.3 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 851 op/s 2024-04-23T16:31:00.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:59 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:31:00.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:59 smithi148 ceph-mon[44520]: from='client.25630 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:31:00.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:30:59 smithi148 ceph-mon[44520]: osdmap e353: 12 total, 12 up, 10 in 2024-04-23T16:31:00.854 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.7 to 1 (10000) 2024-04-23T16:31:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:00 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:31:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:00 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:31:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:00 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2545740512' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:31:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:00 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:31:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:00 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:31:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:00 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2545740512' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:31:01.126 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:00 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:31:01.126 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:00 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:31:01.126 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:00 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2545740512' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:31:01.460 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:31:01.460 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:31:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:01 smithi103 ceph-mon[39629]: pgmap v996: 27 pgs: 27 active+clean; 606 MiB data, 3.9 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.40k op/s 2024-04-23T16:31:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:01 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2545740512' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:31:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:01 smithi103 ceph-mon[39629]: osdmap e354: 12 total, 12 up, 10 in 2024-04-23T16:31:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:01 smithi148 ceph-mon[44520]: pgmap v996: 27 pgs: 27 active+clean; 606 MiB data, 3.9 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.40k op/s 2024-04-23T16:31:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:01 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2545740512' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:31:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:01 smithi148 ceph-mon[44520]: osdmap e354: 12 total, 12 up, 10 in 2024-04-23T16:31:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:01 smithi188 ceph-mon[44101]: pgmap v996: 27 pgs: 27 active+clean; 606 MiB data, 3.9 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.40k op/s 2024-04-23T16:31:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:01 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2545740512' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:31:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:01 smithi188 ceph-mon[44101]: osdmap e354: 12 total, 12 up, 10 in 2024-04-23T16:31:02.320 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-23T16:31:02.712 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:31:02.816 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:31:04.116 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:03 smithi188 ceph-mon[44101]: pgmap v998: 27 pgs: 27 active+clean; 469 MiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:31:04.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:03 smithi148 ceph-mon[44520]: pgmap v998: 27 pgs: 27 active+clean; 469 MiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:31:04.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:03 smithi103 ceph-mon[39629]: pgmap v998: 27 pgs: 27 active+clean; 469 MiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:31:04.752 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:31:04.752 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:31:05.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:05 smithi148 ceph-mon[44520]: pgmap v999: 27 pgs: 27 active+clean; 231 MiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:31:05.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:05 smithi188 ceph-mon[44101]: pgmap v999: 27 pgs: 27 active+clean; 231 MiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:31:05.519 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-23T16:31:05.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:05 smithi103 ceph-mon[39629]: pgmap v999: 27 pgs: 27 active+clean; 231 MiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-23T16:31:05.882 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:31:06.082 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Removed 163446 objects 2024-04-23T16:31:06.082 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Clean up completed and total clean up time :137.244 2024-04-23T16:31:06.083 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:31:06.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:06.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:31:06.332 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:31:06.362 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:06.382 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:31:06.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:06.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:31:06.485 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:31:06.514 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:07.383 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 8 1 2024-04-23T16:31:07.424 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:31:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:07 smithi103 ceph-mon[39629]: pgmap v1000: 27 pgs: 27 active+clean; 183 MiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-23T16:31:08.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:07 smithi148 ceph-mon[44520]: pgmap v1000: 27 pgs: 27 active+clean; 183 MiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-23T16:31:08.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:07 smithi188 ceph-mon[44101]: pgmap v1000: 27 pgs: 27 active+clean; 183 MiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-23T16:31:08.765 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:31:08.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:08.768 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:31:08.870 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:31:08.902 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:09.580 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-04-23T16:31:09.581 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-04-23T16:31:09.782 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:31:09.782 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:31:09.854 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:31:09.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:09.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:31:09.880 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:09 smithi103 ceph-mon[39629]: pgmap v1001: 27 pgs: 27 active+clean; 101 MiB data, 2.9 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:31:09.959 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:31:09.988 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:10.006 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:31:10.008 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:09 smithi188 ceph-mon[44101]: pgmap v1001: 27 pgs: 27 active+clean; 101 MiB data, 2.9 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:31:10.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:10.010 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:31:10.111 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:31:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:09 smithi148 ceph-mon[44520]: pgmap v1001: 27 pgs: 27 active+clean; 101 MiB data, 2.9 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:31:10.145 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:10.167 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:31:10.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:10.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:31:10.271 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:31:10.300 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:10.423 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:31:10.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:10.426 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:31:10.440 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-23T16:31:10.526 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:31:10.552 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:10.571 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:31:10.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:10.574 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:31:10.675 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:31:10.707 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:10.722 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:31:10.722 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:31:10.722 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:31:10.722 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:31:10.722 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:31:10.722 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:31:10.728 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:31:10.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:10.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:31:10.742 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:31:10.745 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:31:10.762 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:31:10.782 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:31:10.802 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:31:10.821 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:31:10.832 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:31:10.841 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:31:10.842 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:31:10.842 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:31:10.842 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:31:10.842 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:31:10.842 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:31:10.842 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:31:10.842 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:31:10.843 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:31:10.843 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:31:10.843 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:31:10.843 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:31:10.843 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:31:10.862 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:10.882 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:31:10.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:10.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:31:10.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:10 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2102034242' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:31:10.986 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:31:11.012 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:10 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2102034242' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:31:11.014 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:11.034 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:31:11.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:11.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:31:11.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:10 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2102034242' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:31:11.137 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:31:11.166 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:11.515 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:31:11.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:11.517 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:31:11.618 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:31:11.650 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:11.738 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:31:11.749 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:31:11.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:11.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:31:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:11 smithi103 ceph-mon[39629]: pgmap v1002: 27 pgs: 27 active+clean; 4.8 MiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 962 op/s 2024-04-23T16:31:11.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.852 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:31:11.879 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:11.955 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.956 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.956 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.956 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:11 smithi188 ceph-mon[44101]: pgmap v1002: 27 pgs: 27 active+clean; 4.8 MiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 962 op/s 2024-04-23T16:31:11.956 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.956 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.956 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.956 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.956 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.956 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.956 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.956 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:11.956 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:11 smithi148 ceph-mon[44520]: pgmap v1002: 27 pgs: 27 active+clean; 4.8 MiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 962 op/s 2024-04-23T16:31:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:12.427 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:31:12.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:12.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:31:12.461 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:31:12.461 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:31:12.461 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-04-23T16:31:12.462 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:31:12.532 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:31:12.560 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:12.634 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:31:12.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:12.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:31:12.738 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:31:12.771 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:12.792 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:31:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:12.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:31:12.896 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:31:12.927 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:12.946 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:31:12.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:12.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:31:13.050 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:31:13.081 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:13.179 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:31:13.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:13.182 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:31:13.196 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-23T16:31:13.228 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:31:13.283 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:31:13.312 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:13.364 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:31:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:13.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:31:13.484 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:31:13.496 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:31:13.516 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:13.538 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:31:13.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:13.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:31:13.642 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:31:13.673 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:13.694 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:31:13.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:13.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:31:13.757 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:31:13.798 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:31:13.826 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:13 smithi148 ceph-mon[44520]: pgmap v1003: 27 pgs: 27 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 757 op/s 2024-04-23T16:31:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:13 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2178327206' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:31:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:13 smithi148 ceph-mon[44520]: from='client.25648 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:31:13.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:13 smithi103 ceph-mon[39629]: pgmap v1003: 27 pgs: 27 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 757 op/s 2024-04-23T16:31:13.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:13 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2178327206' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:31:13.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:13 smithi103 ceph-mon[39629]: from='client.25648 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:31:13.958 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:31:13.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:13.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:31:13.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:13 smithi188 ceph-mon[44101]: pgmap v1003: 27 pgs: 27 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 757 op/s 2024-04-23T16:31:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:13 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2178327206' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:31:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:13 smithi188 ceph-mon[44101]: from='client.25648 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:31:14.062 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:31:14.089 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:14.213 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:31:14.638 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:31:14.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:14.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:31:14.741 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:31:14.774 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:14.796 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:31:14.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:14.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:31:14.899 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:31:14.927 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:14 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:31:14.927 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:14 smithi103 ceph-mon[39629]: from='client.25648 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:31:14.927 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:14 smithi103 ceph-mon[39629]: osdmap e355: 12 total, 12 up, 10 in 2024-04-23T16:31:14.929 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:14.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:14 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:31:14.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:14 smithi188 ceph-mon[44101]: from='client.25648 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:31:14.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:14 smithi188 ceph-mon[44101]: osdmap e355: 12 total, 12 up, 10 in 2024-04-23T16:31:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:14 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:31:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:14 smithi148 ceph-mon[44520]: from='client.25648 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:31:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:14 smithi148 ceph-mon[44520]: osdmap e355: 12 total, 12 up, 10 in 2024-04-23T16:31:15.176 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:31:15.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:15.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:31:15.280 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:31:15.313 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:15.319 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:31:15.320 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:31:15.320 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:31:15.320 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:31:15.320 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:31:15.334 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:31:15.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:15.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:31:15.438 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:31:15.465 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:15.549 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:31:15.565 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:31:15.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:15.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:31:15.667 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:31:15.699 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:15.721 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:31:15.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:15.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:31:15.748 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:15 smithi103 ceph-mon[39629]: pgmap v1005: 27 pgs: 27 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 368 op/s 2024-04-23T16:31:15.824 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:31:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:15.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:15 smithi188 ceph-mon[44101]: pgmap v1005: 27 pgs: 27 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 368 op/s 2024-04-23T16:31:15.987 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-23T16:31:16.005 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:31:16.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:16.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:31:16.109 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:31:16.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:15 smithi148 ceph-mon[44520]: pgmap v1005: 27 pgs: 27 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 368 op/s 2024-04-23T16:31:16.140 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:16.246 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:31:16.623 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:31:16.790 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:31:16.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:16.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:31:16.894 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:31:16.920 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:17.017 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:31:17.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:17.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:31:17.121 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:31:17.149 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:17.405 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:31:17.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:17.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:31:17.413 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:31:17.509 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:31:17.539 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:17.559 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:31:17.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:17.564 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:31:17.665 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:31:17.691 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:17.755 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'unique_pool_2' removed 2024-04-23T16:31:17.758 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:17 smithi103 ceph-mon[39629]: pgmap v1006: 27 pgs: 27 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 292 op/s 2024-04-23T16:31:17.758 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:17 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1692674935' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:31:17.758 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:17 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1934638148' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:31:17.760 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.8 to 1 (10000) 2024-04-23T16:31:17.806 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:17 smithi188 ceph-mon[44101]: pgmap v1006: 27 pgs: 27 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 292 op/s 2024-04-23T16:31:17.807 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:17 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1692674935' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:31:17.807 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:17 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1934638148' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:31:17.814 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:31:17.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:17.817 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:31:17.918 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:31:17.946 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:18.023 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:31:18.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:18.026 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:31:18.078 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:31:18.078 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-04-23T16:31:18.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-04-23T16:31:18.078 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:31:18.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.079 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9614 4.2 4:4f13b2dc:::benchmark_data_smithi188_101370_object9613:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:31:18.079 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.494209+0000", 2024-04-23T16:31:18.079 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 238.580482727, 2024-04-23T16:31:18.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.20416315199999999, 2024-04-23T16:31:18.079 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.079 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.079 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.079 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.079 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.079 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.079 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9614 2024-04-23T16:31:18.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.079 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.494209+0000", 2024-04-23T16:31:18.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.494209+0000", 2024-04-23T16:31:18.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.494216+0000", 2024-04-23T16:31:18.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6649999999999997e-06 2024-04-23T16:31:18.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.494245+0000", 2024-04-23T16:31:18.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9430000000000001e-05 2024-04-23T16:31:18.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.494262+0000", 2024-04-23T16:31:18.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6810999999999999e-05 2024-04-23T16:31:18.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.494275+0000", 2024-04-23T16:31:18.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3341e-05 2024-04-23T16:31:18.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.494337+0000", 2024-04-23T16:31:18.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2297000000000003e-05 2024-04-23T16:31:18.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.494486+0000", 2024-04-23T16:31:18.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000148887 2024-04-23T16:31:18.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:31:18.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.494645+0000", 2024-04-23T16:31:18.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015817899999999999 2024-04-23T16:31:18.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.495299+0000", 2024-04-23T16:31:18.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00065442099999999997 2024-04-23T16:31:18.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.698303+0000", 2024-04-23T16:31:18.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.20300422500000001 2024-04-23T16:31:18.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.698340+0000", 2024-04-23T16:31:18.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6795999999999998e-05 2024-04-23T16:31:18.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.698372+0000", 2024-04-23T16:31:18.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2100000000000001e-05 2024-04-23T16:31:18.085 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.085 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.085 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.085 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9636 4.2 4:420d7b21:::benchmark_data_smithi188_101370_object9635:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:31:18.086 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.498108+0000", 2024-04-23T16:31:18.086 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 238.576583462, 2024-04-23T16:31:18.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.202126424, 2024-04-23T16:31:18.086 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.086 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.086 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.086 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.086 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.086 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.086 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9636 2024-04-23T16:31:18.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.086 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.498108+0000", 2024-04-23T16:31:18.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.498108+0000", 2024-04-23T16:31:18.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.498109+0000", 2024-04-23T16:31:18.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7899999999999997e-07 2024-04-23T16:31:18.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.498139+0000", 2024-04-23T16:31:18.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9977000000000001e-05 2024-04-23T16:31:18.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.498151+0000", 2024-04-23T16:31:18.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2261e-05 2024-04-23T16:31:18.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.498158+0000", 2024-04-23T16:31:18.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7619999999999998e-06 2024-04-23T16:31:18.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.498221+0000", 2024-04-23T16:31:18.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3248000000000002e-05 2024-04-23T16:31:18.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.498340+0000", 2024-04-23T16:31:18.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011917900000000001 2024-04-23T16:31:18.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:31:18.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.498485+0000", 2024-04-23T16:31:18.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000144563 2024-04-23T16:31:18.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.500232+0000", 2024-04-23T16:31:18.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001747391 2024-04-23T16:31:18.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700150+0000", 2024-04-23T16:31:18.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.19991759000000001 2024-04-23T16:31:18.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700193+0000", 2024-04-23T16:31:18.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2684e-05 2024-04-23T16:31:18.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700235+0000", 2024-04-23T16:31:18.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1990000000000003e-05 2024-04-23T16:31:18.092 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.092 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.092 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.092 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9642 4.2 4:4040f281:::benchmark_data_smithi188_101370_object9641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:31:18.092 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.499186+0000", 2024-04-23T16:31:18.092 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 238.57550578499999, 2024-04-23T16:31:18.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.201217806, 2024-04-23T16:31:18.092 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.093 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.093 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.093 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.093 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.093 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.093 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9642 2024-04-23T16:31:18.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.093 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.499186+0000", 2024-04-23T16:31:18.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.499186+0000", 2024-04-23T16:31:18.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.499187+0000", 2024-04-23T16:31:18.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7000000000000004e-07 2024-04-23T16:31:18.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.499231+0000", 2024-04-23T16:31:18.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.464e-05 2024-04-23T16:31:18.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.499238+0000", 2024-04-23T16:31:18.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.075e-06 2024-04-23T16:31:18.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.499245+0000", 2024-04-23T16:31:18.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4540000000000004e-06 2024-04-23T16:31:18.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.499433+0000", 2024-04-23T16:31:18.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018828200000000001 2024-04-23T16:31:18.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.499543+0000", 2024-04-23T16:31:18.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110109 2024-04-23T16:31:18.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:31:18.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.499661+0000", 2024-04-23T16:31:18.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118144 2024-04-23T16:31:18.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.500369+0000", 2024-04-23T16:31:18.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070764199999999997 2024-04-23T16:31:18.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700291+0000", 2024-04-23T16:31:18.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.19992175200000001 2024-04-23T16:31:18.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700362+0000", 2024-04-23T16:31:18.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0798999999999998e-05 2024-04-23T16:31:18.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700404+0000", 2024-04-23T16:31:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2138999999999998e-05 2024-04-23T16:31:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9678 4.2 4:4327109c:::benchmark_data_smithi188_101370_object9677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:31:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.506291+0000", 2024-04-23T16:31:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 238.56840087200001, 2024-04-23T16:31:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.19453815399999999, 2024-04-23T16:31:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9678 2024-04-23T16:31:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.506291+0000", 2024-04-23T16:31:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.506291+0000", 2024-04-23T16:31:18.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.506293+0000", 2024-04-23T16:31:18.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1900000000000002e-06 2024-04-23T16:31:18.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.506316+0000", 2024-04-23T16:31:18.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3050000000000001e-05 2024-04-23T16:31:18.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.506330+0000", 2024-04-23T16:31:18.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3499000000000001e-05 2024-04-23T16:31:18.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.506338+0000", 2024-04-23T16:31:18.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2919999999999996e-06 2024-04-23T16:31:18.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.506427+0000", 2024-04-23T16:31:18.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8792e-05 2024-04-23T16:31:18.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.506566+0000", 2024-04-23T16:31:18.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000139157 2024-04-23T16:31:18.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:31:18.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.506718+0000", 2024-04-23T16:31:18.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015196400000000001 2024-04-23T16:31:18.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.507457+0000", 2024-04-23T16:31:18.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00073881100000000002 2024-04-23T16:31:18.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700756+0000", 2024-04-23T16:31:18.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.193299309 2024-04-23T16:31:18.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700789+0000", 2024-04-23T16:31:18.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2954000000000003e-05 2024-04-23T16:31:18.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700829+0000", 2024-04-23T16:31:18.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0136000000000003e-05 2024-04-23T16:31:18.105 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.105 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.105 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.106 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9706 4.2 4:4f243e4e:::benchmark_data_smithi188_101370_object9705:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:31:18.106 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.511210+0000", 2024-04-23T16:31:18.106 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 238.56348142900001, 2024-04-23T16:31:18.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18977317099999999, 2024-04-23T16:31:18.106 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.106 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.106 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.106 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.106 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.106 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.107 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9706 2024-04-23T16:31:18.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.107 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.511210+0000", 2024-04-23T16:31:18.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.511210+0000", 2024-04-23T16:31:18.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.511213+0000", 2024-04-23T16:31:18.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4320000000000002e-06 2024-04-23T16:31:18.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.511239+0000", 2024-04-23T16:31:18.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6143999999999999e-05 2024-04-23T16:31:18.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.511251+0000", 2024-04-23T16:31:18.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2251e-05 2024-04-23T16:31:18.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.511260+0000", 2024-04-23T16:31:18.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.49e-06 2024-04-23T16:31:18.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.511330+0000", 2024-04-23T16:31:18.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0468000000000004e-05 2024-04-23T16:31:18.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.511479+0000", 2024-04-23T16:31:18.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000148937 2024-04-23T16:31:18.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:31:18.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.511664+0000", 2024-04-23T16:31:18.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018490999999999999 2024-04-23T16:31:18.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.512361+0000", 2024-04-23T16:31:18.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069734400000000002 2024-04-23T16:31:18.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700884+0000", 2024-04-23T16:31:18.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18852321499999999 2024-04-23T16:31:18.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700948+0000", 2024-04-23T16:31:18.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3482e-05 2024-04-23T16:31:18.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.700983+0000", 2024-04-23T16:31:18.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5497999999999998e-05 2024-04-23T16:31:18.112 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.112 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.112 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.112 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9717 4.2 4:43e0c227:::benchmark_data_smithi188_101370_object9716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:31:18.112 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.514316+0000", 2024-04-23T16:31:18.113 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 238.56037595500001, 2024-04-23T16:31:18.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18925007299999999, 2024-04-23T16:31:18.113 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.113 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.113 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.113 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.113 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.113 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.113 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9717 2024-04-23T16:31:18.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.113 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.514316+0000", 2024-04-23T16:31:18.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.514316+0000", 2024-04-23T16:31:18.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.514318+0000", 2024-04-23T16:31:18.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0140000000000001e-06 2024-04-23T16:31:18.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.514343+0000", 2024-04-23T16:31:18.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5795e-05 2024-04-23T16:31:18.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.514359+0000", 2024-04-23T16:31:18.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5460000000000001e-05 2024-04-23T16:31:18.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.514368+0000", 2024-04-23T16:31:18.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2299999999999997e-06 2024-04-23T16:31:18.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.514416+0000", 2024-04-23T16:31:18.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8299e-05 2024-04-23T16:31:18.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.514509+0000", 2024-04-23T16:31:18.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2499999999999999e-05 2024-04-23T16:31:18.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:31:18.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.514594+0000", 2024-04-23T16:31:18.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4890999999999997e-05 2024-04-23T16:31:18.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515958+0000", 2024-04-23T16:31:18.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001363733 2024-04-23T16:31:18.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703457+0000", 2024-04-23T16:31:18.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.187498991 2024-04-23T16:31:18.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703515+0000", 2024-04-23T16:31:18.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8575000000000001e-05 2024-04-23T16:31:18.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703566+0000", 2024-04-23T16:31:18.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0584999999999998e-05 2024-04-23T16:31:18.119 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.119 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.119 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.119 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9719 4.2 4:4ce29c52:::benchmark_data_smithi188_101370_object9718:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:31:18.119 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.515092+0000", 2024-04-23T16:31:18.119 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 238.55959975799999, 2024-04-23T16:31:18.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18860664399999999, 2024-04-23T16:31:18.120 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.120 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.120 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.120 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.120 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.120 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.120 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9719 2024-04-23T16:31:18.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.120 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515092+0000", 2024-04-23T16:31:18.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515092+0000", 2024-04-23T16:31:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515096+0000", 2024-04-23T16:31:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6129999999999997e-06 2024-04-23T16:31:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515143+0000", 2024-04-23T16:31:18.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6971999999999997e-05 2024-04-23T16:31:18.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515155+0000", 2024-04-23T16:31:18.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1296e-05 2024-04-23T16:31:18.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515161+0000", 2024-04-23T16:31:18.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7299999999999999e-06 2024-04-23T16:31:18.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515211+0000", 2024-04-23T16:31:18.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9413000000000003e-05 2024-04-23T16:31:18.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515338+0000", 2024-04-23T16:31:18.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012697199999999999 2024-04-23T16:31:18.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:31:18.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.515484+0000", 2024-04-23T16:31:18.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014637999999999999 2024-04-23T16:31:18.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.516155+0000", 2024-04-23T16:31:18.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067116799999999998 2024-04-23T16:31:18.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703628+0000", 2024-04-23T16:31:18.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18747253899999999 2024-04-23T16:31:18.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703666+0000", 2024-04-23T16:31:18.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7966000000000002e-05 2024-04-23T16:31:18.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703699+0000", 2024-04-23T16:31:18.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2595000000000002e-05 2024-04-23T16:31:18.125 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.125 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.126 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.126 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9737 4.2 4:4045b26d:::benchmark_data_smithi188_101370_object9736:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:31:18.126 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.520122+0000", 2024-04-23T16:31:18.126 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 238.55456935500001, 2024-04-23T16:31:18.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.183681758, 2024-04-23T16:31:18.126 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.126 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.126 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.126 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.126 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.127 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.127 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9737 2024-04-23T16:31:18.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.127 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.520122+0000", 2024-04-23T16:31:18.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.520122+0000", 2024-04-23T16:31:18.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.520125+0000", 2024-04-23T16:31:18.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7269999999999999e-06 2024-04-23T16:31:18.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.520180+0000", 2024-04-23T16:31:18.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4645999999999998e-05 2024-04-23T16:31:18.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.520201+0000", 2024-04-23T16:31:18.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1299999999999999e-05 2024-04-23T16:31:18.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.520213+0000", 2024-04-23T16:31:18.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.163e-05 2024-04-23T16:31:18.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.520284+0000", 2024-04-23T16:31:18.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1364000000000003e-05 2024-04-23T16:31:18.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.520417+0000", 2024-04-23T16:31:18.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013339800000000001 2024-04-23T16:31:18.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:31:18.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.520520+0000", 2024-04-23T16:31:18.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000103069 2024-04-23T16:31:18.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.521128+0000", 2024-04-23T16:31:18.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00060738999999999997 2024-04-23T16:31:18.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703738+0000", 2024-04-23T16:31:18.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18261016399999999 2024-04-23T16:31:18.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703768+0000", 2024-04-23T16:31:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.972e-05 2024-04-23T16:31:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703804+0000", 2024-04-23T16:31:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6350000000000003e-05 2024-04-23T16:31:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9757 4.2 4:4e85d964:::benchmark_data_smithi188_101370_object9756:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:31:18.133 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.526315+0000", 2024-04-23T16:31:18.133 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 238.54837646799999, 2024-04-23T16:31:18.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.177584518, 2024-04-23T16:31:18.133 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.133 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.133 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.133 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.133 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.133 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.133 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9757 2024-04-23T16:31:18.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.134 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.526315+0000", 2024-04-23T16:31:18.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.526315+0000", 2024-04-23T16:31:18.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.526317+0000", 2024-04-23T16:31:18.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1409999999999999e-06 2024-04-23T16:31:18.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.526364+0000", 2024-04-23T16:31:18.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7074999999999999e-05 2024-04-23T16:31:18.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.526380+0000", 2024-04-23T16:31:18.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5789000000000001e-05 2024-04-23T16:31:18.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.526389+0000", 2024-04-23T16:31:18.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.602e-06 2024-04-23T16:31:18.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.526439+0000", 2024-04-23T16:31:18.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0466000000000001e-05 2024-04-23T16:31:18.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.526523+0000", 2024-04-23T16:31:18.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3437999999999996e-05 2024-04-23T16:31:18.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:31:18.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.526609+0000", 2024-04-23T16:31:18.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6278000000000003e-05 2024-04-23T16:31:18.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527210+0000", 2024-04-23T16:31:18.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00060086700000000005 2024-04-23T16:31:18.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703840+0000", 2024-04-23T16:31:18.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.176630227 2024-04-23T16:31:18.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703873+0000", 2024-04-23T16:31:18.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3002e-05 2024-04-23T16:31:18.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703900+0000", 2024-04-23T16:31:18.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6633000000000001e-05 2024-04-23T16:31:18.139 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.139 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.139 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.139 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9762 4.2 4:44b975ec:::benchmark_data_smithi188_101370_object9761:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:31:18.139 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.527254+0000", 2024-04-23T16:31:18.139 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 238.54743758999999, 2024-04-23T16:31:18.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.176807251, 2024-04-23T16:31:18.140 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.140 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.140 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.140 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.140 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.140 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.140 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9762 2024-04-23T16:31:18.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.140 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527254+0000", 2024-04-23T16:31:18.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527254+0000", 2024-04-23T16:31:18.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527256+0000", 2024-04-23T16:31:18.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.61e-06 2024-04-23T16:31:18.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527305+0000", 2024-04-23T16:31:18.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9036000000000003e-05 2024-04-23T16:31:18.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527316+0000", 2024-04-23T16:31:18.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0879e-05 2024-04-23T16:31:18.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527327+0000", 2024-04-23T16:31:18.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1083999999999999e-05 2024-04-23T16:31:18.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527382+0000", 2024-04-23T16:31:18.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5754000000000002e-05 2024-04-23T16:31:18.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527478+0000", 2024-04-23T16:31:18.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5166000000000003e-05 2024-04-23T16:31:18.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:31:18.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.527577+0000", 2024-04-23T16:31:18.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9499000000000004e-05 2024-04-23T16:31:18.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.528059+0000", 2024-04-23T16:31:18.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048219199999999999 2024-04-23T16:31:18.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.703969+0000", 2024-04-23T16:31:18.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17590968200000001 2024-04-23T16:31:18.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704024+0000", 2024-04-23T16:31:18.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5161999999999998e-05 2024-04-23T16:31:18.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704061+0000", 2024-04-23T16:31:18.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7187000000000002e-05 2024-04-23T16:31:18.145 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.146 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.146 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.146 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9774 4.2 4:4c2552b2:::benchmark_data_smithi188_101370_object9773:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:31:18.146 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.530865+0000", 2024-04-23T16:31:18.146 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 238.54382684800001, 2024-04-23T16:31:18.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17334628899999999, 2024-04-23T16:31:18.146 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.146 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.146 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.146 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.147 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.147 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.147 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9774 2024-04-23T16:31:18.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.147 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.530865+0000", 2024-04-23T16:31:18.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.530865+0000", 2024-04-23T16:31:18.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.530867+0000", 2024-04-23T16:31:18.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9360000000000002e-06 2024-04-23T16:31:18.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.530910+0000", 2024-04-23T16:31:18.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2886000000000002e-05 2024-04-23T16:31:18.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.530922+0000", 2024-04-23T16:31:18.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2036e-05 2024-04-23T16:31:18.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.530928+0000", 2024-04-23T16:31:18.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6320000000000002e-06 2024-04-23T16:31:18.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.530982+0000", 2024-04-23T16:31:18.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3974000000000002e-05 2024-04-23T16:31:18.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.531080+0000", 2024-04-23T16:31:18.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8171000000000006e-05 2024-04-23T16:31:18.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:31:18.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.531178+0000", 2024-04-23T16:31:18.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7570000000000003e-05 2024-04-23T16:31:18.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.531768+0000", 2024-04-23T16:31:18.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00058953499999999997 2024-04-23T16:31:18.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704122+0000", 2024-04-23T16:31:18.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17235446900000001 2024-04-23T16:31:18.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704172+0000", 2024-04-23T16:31:18.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9787000000000003e-05 2024-04-23T16:31:18.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704211+0000", 2024-04-23T16:31:18.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9292999999999998e-05 2024-04-23T16:31:18.152 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.152 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.152 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.153 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9793 4.2 4:48f6dcf1:::benchmark_data_smithi188_101370_object9792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:31:18.153 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.537491+0000", 2024-04-23T16:31:18.153 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 238.53720044900001, 2024-04-23T16:31:18.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.16686599799999999, 2024-04-23T16:31:18.153 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.153 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.153 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.153 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.153 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.153 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.153 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9793 2024-04-23T16:31:18.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.154 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.537491+0000", 2024-04-23T16:31:18.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.537491+0000", 2024-04-23T16:31:18.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.537492+0000", 2024-04-23T16:31:18.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.175e-06 2024-04-23T16:31:18.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.537510+0000", 2024-04-23T16:31:18.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7261999999999998e-05 2024-04-23T16:31:18.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.537520+0000", 2024-04-23T16:31:18.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0305e-05 2024-04-23T16:31:18.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.537528+0000", 2024-04-23T16:31:18.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8499999999999994e-06 2024-04-23T16:31:18.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.537580+0000", 2024-04-23T16:31:18.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2641e-05 2024-04-23T16:31:18.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.537709+0000", 2024-04-23T16:31:18.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012869499999999999 2024-04-23T16:31:18.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:31:18.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.537853+0000", 2024-04-23T16:31:18.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014435199999999999 2024-04-23T16:31:18.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.538386+0000", 2024-04-23T16:31:18.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053206100000000001 2024-04-23T16:31:18.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704280+0000", 2024-04-23T16:31:18.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.165894508 2024-04-23T16:31:18.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704328+0000", 2024-04-23T16:31:18.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8025999999999997e-05 2024-04-23T16:31:18.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704357+0000", 2024-04-23T16:31:18.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9122999999999999e-05 2024-04-23T16:31:18.159 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.159 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.159 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.159 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9797 4.2 4:483b6d3f:::benchmark_data_smithi188_101370_object9796:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:31:18.159 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.539175+0000", 2024-04-23T16:31:18.160 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 238.53551672, 2024-04-23T16:31:18.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.16532619300000001, 2024-04-23T16:31:18.160 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.160 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.160 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.160 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.160 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.160 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.160 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9797 2024-04-23T16:31:18.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.160 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539175+0000", 2024-04-23T16:31:18.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539175+0000", 2024-04-23T16:31:18.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539176+0000", 2024-04-23T16:31:18.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2049999999999999e-06 2024-04-23T16:31:18.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539195+0000", 2024-04-23T16:31:18.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8383999999999999e-05 2024-04-23T16:31:18.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539203+0000", 2024-04-23T16:31:18.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6780000000000006e-06 2024-04-23T16:31:18.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539209+0000", 2024-04-23T16:31:18.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5380000000000002e-06 2024-04-23T16:31:18.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539256+0000", 2024-04-23T16:31:18.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7411999999999999e-05 2024-04-23T16:31:18.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539370+0000", 2024-04-23T16:31:18.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000113519 2024-04-23T16:31:18.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:31:18.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539523+0000", 2024-04-23T16:31:18.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000153194 2024-04-23T16:31:18.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.539941+0000", 2024-04-23T16:31:18.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00041838500000000001 2024-04-23T16:31:18.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704408+0000", 2024-04-23T16:31:18.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.16446627899999999 2024-04-23T16:31:18.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704463+0000", 2024-04-23T16:31:18.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5241999999999997e-05 2024-04-23T16:31:18.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704501+0000", 2024-04-23T16:31:18.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8356999999999999e-05 2024-04-23T16:31:18.166 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.166 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.166 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.166 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9807 4.2 4:4cb83167:::benchmark_data_smithi188_101370_object9806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:31:18.166 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.543468+0000", 2024-04-23T16:31:18.166 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 238.531223926, 2024-04-23T16:31:18.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.16123652299999999, 2024-04-23T16:31:18.166 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.166 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.166 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.167 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.167 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.167 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.167 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9807 2024-04-23T16:31:18.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.167 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.543468+0000", 2024-04-23T16:31:18.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.543468+0000", 2024-04-23T16:31:18.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.543469+0000", 2024-04-23T16:31:18.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.282e-06 2024-04-23T16:31:18.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.543484+0000", 2024-04-23T16:31:18.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.484e-05 2024-04-23T16:31:18.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.543493+0000", 2024-04-23T16:31:18.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9789999999999999e-06 2024-04-23T16:31:18.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.543498+0000", 2024-04-23T16:31:18.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6030000000000004e-06 2024-04-23T16:31:18.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.543545+0000", 2024-04-23T16:31:18.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6105e-05 2024-04-23T16:31:18.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.543678+0000", 2024-04-23T16:31:18.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013313600000000001 2024-04-23T16:31:18.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:31:18.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.543827+0000", 2024-04-23T16:31:18.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014967800000000001 2024-04-23T16:31:18.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.544302+0000", 2024-04-23T16:31:18.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047469599999999998 2024-04-23T16:31:18.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704574+0000", 2024-04-23T16:31:18.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.160272426 2024-04-23T16:31:18.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704630+0000", 2024-04-23T16:31:18.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5161999999999998e-05 2024-04-23T16:31:18.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704704+0000", 2024-04-23T16:31:18.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4616000000000005e-05 2024-04-23T16:31:18.172 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.172 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.172 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.172 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9825 4.2 4:4e8045d9:::benchmark_data_smithi188_101370_object9824:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:31:18.173 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.554216+0000", 2024-04-23T16:31:18.173 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 238.52047578899999, 2024-04-23T16:31:18.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.15061554099999999, 2024-04-23T16:31:18.173 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.173 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.173 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.173 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.173 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.173 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.173 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9825 2024-04-23T16:31:18.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.173 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554216+0000", 2024-04-23T16:31:18.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554216+0000", 2024-04-23T16:31:18.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554218+0000", 2024-04-23T16:31:18.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.826e-06 2024-04-23T16:31:18.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554235+0000", 2024-04-23T16:31:18.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7374e-05 2024-04-23T16:31:18.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554245+0000", 2024-04-23T16:31:18.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7860000000000008e-06 2024-04-23T16:31:18.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554251+0000", 2024-04-23T16:31:18.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6620000000000002e-06 2024-04-23T16:31:18.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554296+0000", 2024-04-23T16:31:18.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5716000000000001e-05 2024-04-23T16:31:18.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554377+0000", 2024-04-23T16:31:18.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1083999999999998e-05 2024-04-23T16:31:18.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:31:18.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554462+0000", 2024-04-23T16:31:18.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4827999999999995e-05 2024-04-23T16:31:18.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.554901+0000", 2024-04-23T16:31:18.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043888800000000002 2024-04-23T16:31:18.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704750+0000", 2024-04-23T16:31:18.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14984945499999999 2024-04-23T16:31:18.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704795+0000", 2024-04-23T16:31:18.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4477000000000001e-05 2024-04-23T16:31:18.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.704831+0000", 2024-04-23T16:31:18.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6445000000000001e-05 2024-04-23T16:31:18.179 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.179 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.179 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.179 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:9839 4.2 4:41a36203:::benchmark_data_smithi188_101370_object9838:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:31:18.180 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:19.576043+0000", 2024-04-23T16:31:18.180 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 238.49864898999999, 2024-04-23T16:31:18.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.129074616, 2024-04-23T16:31:18.180 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.180 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.180 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.180 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.180 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.180 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.180 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9839 2024-04-23T16:31:18.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.180 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576043+0000", 2024-04-23T16:31:18.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576043+0000", 2024-04-23T16:31:18.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576044+0000", 2024-04-23T16:31:18.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.28e-06 2024-04-23T16:31:18.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576081+0000", 2024-04-23T16:31:18.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7279e-05 2024-04-23T16:31:18.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576091+0000", 2024-04-23T16:31:18.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9399999999999997e-06 2024-04-23T16:31:18.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576105+0000", 2024-04-23T16:31:18.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3919e-05 2024-04-23T16:31:18.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576146+0000", 2024-04-23T16:31:18.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0756000000000001e-05 2024-04-23T16:31:18.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576226+0000", 2024-04-23T16:31:18.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9969999999999995e-05 2024-04-23T16:31:18.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:31:18.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576309+0000", 2024-04-23T16:31:18.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2930999999999996e-05 2024-04-23T16:31:18.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.576755+0000", 2024-04-23T16:31:18.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044604099999999999 2024-04-23T16:31:18.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.705043+0000", 2024-04-23T16:31:18.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.12828847199999999 2024-04-23T16:31:18.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.705082+0000", 2024-04-23T16:31:18.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8763999999999997e-05 2024-04-23T16:31:18.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:19.705117+0000", 2024-04-23T16:31:18.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5264e-05 2024-04-23T16:31:18.186 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.186 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.186 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.186 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:67253 4.2 4:4f895511:::benchmark_data_smithi188_101370_object67252:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e288)", 2024-04-23T16:31:18.186 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:27:37.576346+0000", 2024-04-23T16:31:18.186 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 220.498345796, 2024-04-23T16:31:18.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.048550018, 2024-04-23T16:31:18.186 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.186 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.187 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.187 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.187 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.187 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.187 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 67253 2024-04-23T16:31:18.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.187 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:37.576346+0000", 2024-04-23T16:31:18.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:37.576346+0000", 2024-04-23T16:31:18.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:37.576346+0000", 2024-04-23T16:31:18.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.37e-07 2024-04-23T16:31:18.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:37.576363+0000", 2024-04-23T16:31:18.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6092000000000001e-05 2024-04-23T16:31:18.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:37.576367+0000", 2024-04-23T16:31:18.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3420000000000001e-06 2024-04-23T16:31:18.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:37.576372+0000", 2024-04-23T16:31:18.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3279999999999996e-06 2024-04-23T16:31:18.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:37.578640+0000", 2024-04-23T16:31:18.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002267972 2024-04-23T16:31:18.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:37.578746+0000", 2024-04-23T16:31:18.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0001055 2024-04-23T16:31:18.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:31:18.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:37.578842+0000", 2024-04-23T16:31:18.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6582999999999993e-05 2024-04-23T16:31:18.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:37.624730+0000", 2024-04-23T16:31:18.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.045888101000000001 2024-04-23T16:31:18.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:37.624785+0000", 2024-04-23T16:31:18.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5059999999999998e-05 2024-04-23T16:31:18.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:37.624849+0000", 2024-04-23T16:31:18.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3334e-05 2024-04-23T16:31:18.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:27:37.624896+0000", 2024-04-23T16:31:18.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7068999999999999e-05 2024-04-23T16:31:18.192 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.192 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.192 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.193 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:178449 4.6 4:647e9c02:::benchmark_data_smithi188_101370_object15000:head [delete] snapc 0=[] ondisk+write+known_if_redirected e321)", 2024-04-23T16:31:18.193 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:29:04.000341+0000", 2024-04-23T16:31:18.193 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 134.074350592, 2024-04-23T16:31:18.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.071347912999999999, 2024-04-23T16:31:18.193 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.193 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.193 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.193 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.193 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.193 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.194 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 178449 2024-04-23T16:31:18.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.194 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.000341+0000", 2024-04-23T16:31:18.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.000341+0000", 2024-04-23T16:31:18.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.000343+0000", 2024-04-23T16:31:18.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6390000000000001e-06 2024-04-23T16:31:18.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.000348+0000", 2024-04-23T16:31:18.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.826e-06 2024-04-23T16:31:18.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.000357+0000", 2024-04-23T16:31:18.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7419999999999992e-06 2024-04-23T16:31:18.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.000363+0000", 2024-04-23T16:31:18.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8869999999999997e-06 2024-04-23T16:31:18.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.000408+0000", 2024-04-23T16:31:18.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5300000000000003e-05 2024-04-23T16:31:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.000492+0000", 2024-04-23T16:31:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3633000000000004e-05 2024-04-23T16:31:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 2", 2024-04-23T16:31:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.000563+0000", 2024-04-23T16:31:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1177e-05 2024-04-23T16:31:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.007540+0000", 2024-04-23T16:31:18.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0069772419999999998 2024-04-23T16:31:18.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.071581+0000", 2024-04-23T16:31:18.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.064040785000000003 2024-04-23T16:31:18.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.071639+0000", 2024-04-23T16:31:18.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8078999999999997e-05 2024-04-23T16:31:18.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.071689+0000", 2024-04-23T16:31:18.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9602999999999999e-05 2024-04-23T16:31:18.199 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.199 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.199 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.199 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:178453 4.6 4:6a27016b:::benchmark_data_smithi188_101370_object15004:head [delete] snapc 0=[] ondisk+write+known_if_redirected e321)", 2024-04-23T16:31:18.199 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:29:04.002864+0000", 2024-04-23T16:31:18.199 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 134.07182806500001, 2024-04-23T16:31:18.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.068974024999999994, 2024-04-23T16:31:18.200 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.200 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.200 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.200 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.200 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.200 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.200 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 178453 2024-04-23T16:31:18.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.200 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.002864+0000", 2024-04-23T16:31:18.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.002864+0000", 2024-04-23T16:31:18.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.002865+0000", 2024-04-23T16:31:18.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3659999999999999e-06 2024-04-23T16:31:18.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.002869+0000", 2024-04-23T16:31:18.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5199999999999999e-06 2024-04-23T16:31:18.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.002878+0000", 2024-04-23T16:31:18.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3720000000000005e-06 2024-04-23T16:31:18.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.002884+0000", 2024-04-23T16:31:18.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4969999999999998e-06 2024-04-23T16:31:18.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.004185+0000", 2024-04-23T16:31:18.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013002339999999999 2024-04-23T16:31:18.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.004269+0000", 2024-04-23T16:31:18.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4327999999999996e-05 2024-04-23T16:31:18.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 2", 2024-04-23T16:31:18.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.004338+0000", 2024-04-23T16:31:18.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8959000000000002e-05 2024-04-23T16:31:18.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.007694+0000", 2024-04-23T16:31:18.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033563400000000002 2024-04-23T16:31:18.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.071744+0000", 2024-04-23T16:31:18.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.064049769000000006 2024-04-23T16:31:18.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.071798+0000", 2024-04-23T16:31:18.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3891000000000003e-05 2024-04-23T16:31:18.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.071838+0000", 2024-04-23T16:31:18.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9749000000000002e-05 2024-04-23T16:31:18.206 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.206 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.206 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.206 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15486.0:178468 4.6 4:6b9dcdcc:::benchmark_data_smithi188_101370_object15019:head [delete] snapc 0=[] ondisk+write+known_if_redirected e321)", 2024-04-23T16:31:18.206 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:29:04.022261+0000", 2024-04-23T16:31:18.206 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 134.05243066400001, 2024-04-23T16:31:18.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.049710870999999997, 2024-04-23T16:31:18.206 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:18.206 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:18.206 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:31:18.207 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:18.207 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15486", 2024-04-23T16:31:18.207 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:31:18.207 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 178468 2024-04-23T16:31:18.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.207 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:18.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:18.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.022261+0000", 2024-04-23T16:31:18.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:18.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.022261+0000", 2024-04-23T16:31:18.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:18.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:18.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.022264+0000", 2024-04-23T16:31:18.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9160000000000001e-06 2024-04-23T16:31:18.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:18.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.022271+0000", 2024-04-23T16:31:18.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1130000000000003e-06 2024-04-23T16:31:18.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:18.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.022289+0000", 2024-04-23T16:31:18.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7799999999999999e-05 2024-04-23T16:31:18.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:18.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.022298+0000", 2024-04-23T16:31:18.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4919999999999999e-06 2024-04-23T16:31:18.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:18.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.022358+0000", 2024-04-23T16:31:18.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9598000000000001e-05 2024-04-23T16:31:18.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:31:18.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.022476+0000", 2024-04-23T16:31:18.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118194 2024-04-23T16:31:18.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 2", 2024-04-23T16:31:18.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.022594+0000", 2024-04-23T16:31:18.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000117558 2024-04-23T16:31:18.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:31:18.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.027495+0000", 2024-04-23T16:31:18.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049017610000000001 2024-04-23T16:31:18.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:31:18.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.071885+0000", 2024-04-23T16:31:18.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.044389862000000002 2024-04-23T16:31:18.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:31:18.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.071934+0000", 2024-04-23T16:31:18.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8569999999999997e-05 2024-04-23T16:31:18.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:18.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:18.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:18.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:29:04.071972+0000", 2024-04-23T16:31:18.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8006999999999997e-05 2024-04-23T16:31:18.212 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.212 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.212 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.212 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:18.212 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:18.212 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:31:18.213 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:31:18.214 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:17 smithi148 ceph-mon[44520]: pgmap v1006: 27 pgs: 27 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 292 op/s 2024-04-23T16:31:18.214 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:17 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1692674935' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:31:18.214 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:17 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1934638148' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:31:18.244 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:18.758 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-23T16:31:19.042 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:31:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:18 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1692674935' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:31:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:18 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1934638148' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:31:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:18 smithi103 ceph-mon[39629]: osdmap e356: 12 total, 12 up, 10 in 2024-04-23T16:31:19.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:18 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1692674935' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:31:19.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:18 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1934638148' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:31:19.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:18 smithi148 ceph-mon[44520]: osdmap e356: 12 total, 12 up, 10 in 2024-04-23T16:31:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:18 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1692674935' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:31:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:18 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1934638148' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:31:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:18 smithi188 ceph-mon[44101]: osdmap e356: 12 total, 12 up, 10 in 2024-04-23T16:31:19.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:19.452 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:31:19.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:31:19.554 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:31:19.582 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:19.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:19 smithi188 ceph-mon[44101]: pgmap v1008: 11 pgs: 11 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:19.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:19 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:31:19.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:19 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:31:19.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:19 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:31:19.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:19 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:31:19.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:19 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:31:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:19 smithi103 ceph-mon[39629]: pgmap v1008: 11 pgs: 11 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:19 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:31:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:19 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:31:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:19 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:31:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:19 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:31:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:19 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:31:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:19 smithi148 ceph-mon[44520]: pgmap v1008: 11 pgs: 11 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:19 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:31:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:19 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:31:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:19 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:31:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:19 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:31:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:19 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:31:20.148 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 9 1 2024-04-23T16:31:20.331 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:31:20.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:20.334 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:31:20.435 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:31:20.463 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:20.614 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:31:20.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:20.617 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:31:20.718 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:31:20.749 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:20.771 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-23T16:31:20.771 INFO:tasks.radosbench:Beginning radosbench... 2024-04-23T16:31:20.772 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-04-23T16:31:20.772 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-04-23T16:31:20.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:20.773 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:31:20.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:31:20.874 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:31:20.878 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:31:20.879 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:31:20.879 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-04-23T16:31:20.879 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:31:20.900 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:20 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:31:20.901 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:20 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:31:20.901 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:20 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:31:20.903 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:21.035 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:20 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:31:21.035 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:20 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:31:21.036 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:20 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:31:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:20 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:31:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:20 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:31:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:20 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:31:21.309 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:31:21.333 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:31:21.353 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:31:21.372 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:31:21.392 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:31:21.412 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:31:21.431 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:31:21.452 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:31:21.472 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:31:21.491 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:31:21.514 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:31:21.536 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:31:21.537 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:31:21.537 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:31:21.538 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:31:21.538 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:31:21.538 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:31:21.538 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:31:21.538 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:31:21.538 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:31:21.538 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:31:21.538 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:31:21.538 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:31:21.538 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:31:21.541 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-23T16:31:21.551 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:31:21.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:21.555 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:31:21.655 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:31:21.686 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:21.705 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:31:21.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:21.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:31:21.811 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:31:21.813 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:31:21.840 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:21 smithi188 ceph-mon[44101]: pgmap v1009: 11 pgs: 11 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:21.840 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:21 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3644247527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:31:21.840 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:21.840 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:21.840 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:21.840 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:21.840 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:21.842 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:21.878 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:31:21.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:21.880 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:31:21.981 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:31:22.007 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:21 smithi103 ceph-mon[39629]: pgmap v1009: 11 pgs: 11 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:21 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3644247527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:31:22.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:22.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:22.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:22.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:22.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:22.105 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:31:22.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:22.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:31:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:21 smithi148 ceph-mon[44520]: pgmap v1009: 11 pgs: 11 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:21 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3644247527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:31:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:22.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:22.209 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:31:22.242 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:22.262 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:31:22.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:22.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:31:22.366 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:31:22.394 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:22.431 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:31:22.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:22.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:31:22.535 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:31:22.557 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:31:22.558 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:31:22.564 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:22.586 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:31:22.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:22.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:31:22.690 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:31:22.722 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:22.744 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:31:22.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:22.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:31:22.770 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:22.771 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:22.771 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:22.771 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:22.771 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:22.771 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:22.771 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:22.772 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:31:22.848 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:31:22.875 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:23.036 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:22 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4080635346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:31:23.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:23.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:23.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:23.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:23.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:23.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:23.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:23.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:22 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4080635346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:31:23.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:23.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:23.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:23.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:23.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:23.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:23.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:23.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:22 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4080635346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:31:23.224 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:31:23.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:23.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:31:23.328 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:31:23.355 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:23.601 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:31:23.601 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:31:23.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:23 smithi188 ceph-mon[44101]: pgmap v1010: 11 pgs: 11 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:23.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:23 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:31:23.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:23 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4080635346' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:31:23.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:23 smithi188 ceph-mon[44101]: osdmap e357: 12 total, 12 up, 10 in 2024-04-23T16:31:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:23 smithi103 ceph-mon[39629]: pgmap v1010: 11 pgs: 11 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:23 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:31:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:23 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4080635346' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:31:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:23 smithi103 ceph-mon[39629]: osdmap e357: 12 total, 12 up, 10 in 2024-04-23T16:31:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:23 smithi148 ceph-mon[44520]: pgmap v1010: 11 pgs: 11 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:23 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:31:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:23 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4080635346' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:31:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:23 smithi148 ceph-mon[44520]: osdmap e357: 12 total, 12 up, 10 in 2024-04-23T16:31:24.421 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-23T16:31:24.455 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:31:24.603 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:31:24.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:24.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:31:24.682 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:31:24.707 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:31:24.735 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:24.933 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:31:24.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:24.936 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:31:25.037 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:31:25.066 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:25.316 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:31:25.320 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:25 smithi103 ceph-mon[39629]: pgmap v1012: 11 pgs: 11 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:25.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:25.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:31:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:25 smithi148 ceph-mon[44520]: pgmap v1012: 11 pgs: 11 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:25.421 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:31:25.448 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:25 smithi188 ceph-mon[44101]: pgmap v1012: 11 pgs: 11 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:25.450 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:25.471 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:31:25.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:25.474 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:31:25.575 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:31:25.601 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:25.949 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:31:25.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:25.952 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:31:26.053 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:31:26.080 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:26.114 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.9 to 1 (10000) 2024-04-23T16:31:26.217 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:26 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3680320484' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:31:26.229 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:31:26.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:26.232 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:31:26.333 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:31:26.362 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:26 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3680320484' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:31:26.426 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:31:26.435 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:31:26.441 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:26 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3680320484' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:31:26.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:26.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:31:26.519 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:31:26.519 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:31:26.542 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:31:26.574 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:26.596 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:31:26.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:26.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:31:26.700 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:31:26.729 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:26.748 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:31:26.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:26.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:31:26.852 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:31:26.884 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:26.905 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:31:26.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:26.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:31:27.009 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:31:27.039 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:27.059 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:31:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:27.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:31:27.084 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:31:27.153 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-23T16:31:27.167 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:31:27.194 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:27 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3680320484' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:31:27.194 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:27 smithi148 ceph-mon[44520]: osdmap e358: 12 total, 12 up, 10 in 2024-04-23T16:31:27.194 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:27 smithi148 ceph-mon[44520]: pgmap v1014: 11 pgs: 11 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:27.194 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:27 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T16:31:27.194 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:27 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T16:31:27.195 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:27.215 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:31:27.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:27.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:31:27.319 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:31:27.351 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:27.367 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:27 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3680320484' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:31:27.367 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:27 smithi103 ceph-mon[39629]: osdmap e358: 12 total, 12 up, 10 in 2024-04-23T16:31:27.367 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:27 smithi103 ceph-mon[39629]: pgmap v1014: 11 pgs: 11 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:27.367 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:27 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T16:31:27.367 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:27 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T16:31:27.368 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 11 1 2024-04-23T16:31:27.411 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:31:27.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:27 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3680320484' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:31:27.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:27 smithi188 ceph-mon[44101]: osdmap e358: 12 total, 12 up, 10 in 2024-04-23T16:31:27.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:27 smithi188 ceph-mon[44101]: pgmap v1014: 11 pgs: 11 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:27.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:27 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T16:31:27.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:27 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T16:31:27.656 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:31:27.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:27.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:31:27.760 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:31:27.788 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:28.133 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'unique_pool_3' created 2024-04-23T16:31:28.215 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:31:28.217 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:28 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3459082054' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-23T16:31:28.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:28.218 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:31:28.319 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:31:28.348 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:28.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:28 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3459082054' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-23T16:31:28.446 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:31:28.447 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:28 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3459082054' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-23T16:31:28.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:28.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:31:28.550 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:31:28.577 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:28.675 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:31:28.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:28.678 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:31:28.779 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:31:28.812 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:28.833 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:31:28.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:28.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:31:28.877 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-04-23T16:31:28.937 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:31:28.963 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:29 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:31:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:29 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:31:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:29 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3459082054' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-04-23T16:31:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:29 smithi188 ceph-mon[44101]: osdmap e359: 12 total, 12 up, 10 in 2024-04-23T16:31:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:29 smithi188 ceph-mon[44101]: pgmap v1016: 27 pgs: 16 unknown, 11 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:29.308 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:31:29.308 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:31:29.411 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:31:29.413 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:29 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:31:29.413 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:29 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:31:29.413 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:29 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3459082054' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-04-23T16:31:29.413 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:29 smithi103 ceph-mon[39629]: osdmap e359: 12 total, 12 up, 10 in 2024-04-23T16:31:29.413 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:29 smithi103 ceph-mon[39629]: pgmap v1016: 27 pgs: 16 unknown, 11 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:29.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:29.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:31:29.515 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:31:29.542 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:29.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:29 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:31:29.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:29 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:31:29.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:29 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3459082054' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-04-23T16:31:29.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:29 smithi148 ceph-mon[44520]: osdmap e359: 12 total, 12 up, 10 in 2024-04-23T16:31:29.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:29 smithi148 ceph-mon[44520]: pgmap v1016: 27 pgs: 16 unknown, 11 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:29.740 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:31:29.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:29.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:31:29.843 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:31:29.876 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:29.897 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:31:29.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:29.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:31:29.906 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:31:29.960 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-23T16:31:30.003 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:31:30.029 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:30.046 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:31:30.046 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:31:30.051 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:31:30.052 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:31:30.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:30.056 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:31:30.061 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:31:30.083 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:31:30.103 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:31:30.124 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:31:30.148 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:31:30.156 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:31:30.168 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:31:30.185 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:30.189 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:31:30.211 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:31:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:30 smithi188 ceph-mon[44101]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:31:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:30 smithi188 ceph-mon[44101]: osdmap e360: 12 total, 12 up, 10 in 2024-04-23T16:31:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:30 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3297908920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:31:30.230 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:31:30.231 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:31:30.251 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:31:30.252 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:31:30.252 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:31:30.252 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:31:30.252 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:31:30.252 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:31:30.252 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:31:30.252 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:31:30.252 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:31:30.252 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:31:30.252 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:31:30.253 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:31:30.253 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:31:30.258 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:31:30.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:30.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:31:30.363 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:31:30.390 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:30.462 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:31:30.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:30.465 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:31:30.566 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:31:30.593 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:30.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:30 smithi103 ceph-mon[39629]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:31:30.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:30 smithi103 ceph-mon[39629]: osdmap e360: 12 total, 12 up, 10 in 2024-04-23T16:31:30.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:30 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3297908920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:31:30.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:30 smithi148 ceph-mon[44520]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:31:30.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:30 smithi148 ceph-mon[44520]: osdmap e360: 12 total, 12 up, 10 in 2024-04-23T16:31:30.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:30 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3297908920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:31:30.665 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:31:30.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:30.668 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:31:30.769 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:31:30.798 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:31.234 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:31:31.247 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:31:31.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:31.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:31:31.316 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:31 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:31:31.316 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:31 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:31:31.316 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:31 smithi188 ceph-mon[44101]: 1.4 scrub starts 2024-04-23T16:31:31.316 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.316 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.316 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.316 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:31 smithi188 ceph-mon[44101]: 1.4 scrub ok 2024-04-23T16:31:31.316 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.316 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.316 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.316 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:31 smithi188 ceph-mon[44101]: osdmap e361: 12 total, 12 up, 10 in 2024-04-23T16:31:31.316 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.317 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.317 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.317 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.317 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.317 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.317 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:31 smithi188 ceph-mon[44101]: pgmap v1019: 27 pgs: 1 creating+activating, 7 creating+peering, 4 unknown, 15 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:31.350 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:31:31.383 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:31.405 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:31:31.406 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:31 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:31:31.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:31 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:31:31.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:31 smithi148 ceph-mon[44520]: 1.4 scrub starts 2024-04-23T16:31:31.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:31 smithi148 ceph-mon[44520]: 1.4 scrub ok 2024-04-23T16:31:31.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:31 smithi148 ceph-mon[44520]: osdmap e361: 12 total, 12 up, 10 in 2024-04-23T16:31:31.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.408 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.408 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.408 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.408 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:31 smithi148 ceph-mon[44520]: pgmap v1019: 27 pgs: 1 creating+activating, 7 creating+peering, 4 unknown, 15 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:31.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:31.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:31:31.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:31 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:31:31.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:31 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:31:31.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:31 smithi103 ceph-mon[39629]: 1.4 scrub starts 2024-04-23T16:31:31.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:31 smithi103 ceph-mon[39629]: 1.4 scrub ok 2024-04-23T16:31:31.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:31 smithi103 ceph-mon[39629]: osdmap e361: 12 total, 12 up, 10 in 2024-04-23T16:31:31.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:31.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:31 smithi103 ceph-mon[39629]: pgmap v1019: 27 pgs: 1 creating+activating, 7 creating+peering, 4 unknown, 15 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:31.510 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:31:31.537 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:31.610 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:31:31.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:31.613 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:31:31.714 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:31:31.744 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:31.795 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:31:31.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:31.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:31:31.900 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:31:31.917 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:31:31.917 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:31:31.917 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:31:31.917 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:31:31.917 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:31:31.927 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:32.050 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:31:32.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:32.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:31:32.154 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:31:32.181 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:32 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:31:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:32 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:31:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:32 smithi188 ceph-mon[44101]: osdmap e362: 12 total, 12 up, 10 in 2024-04-23T16:31:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:32 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:31:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:32 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:31:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:32 smithi103 ceph-mon[39629]: osdmap e362: 12 total, 12 up, 10 in 2024-04-23T16:31:32.527 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-23T16:31:32.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:32 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:31:32.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:32 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:31:32.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:32 smithi148 ceph-mon[44520]: osdmap e362: 12 total, 12 up, 10 in 2024-04-23T16:31:32.791 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:31:33.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:33 smithi103 ceph-mon[39629]: pgmap v1021: 27 pgs: 1 creating+activating, 7 creating+peering, 2 unknown, 17 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:33.302 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:33 smithi188 ceph-mon[44101]: pgmap v1021: 27 pgs: 1 creating+activating, 7 creating+peering, 2 unknown, 17 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:33.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:33 smithi148 ceph-mon[44520]: pgmap v1021: 27 pgs: 1 creating+activating, 7 creating+peering, 2 unknown, 17 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:34.132 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:31:34.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:34.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:31:34.238 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:31:34.268 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:34.288 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:31:34.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:34.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:31:34.392 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:31:34.423 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:34.445 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:31:34.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:34.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:31:34.525 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:31:34.533 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:31:34.533 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-04-23T16:31:34.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-04-23T16:31:34.534 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:31:34.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.534 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208917 3.4 3.eacf0934 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:31:34.534 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085422+0000", 2024-04-23T16:31:34.534 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 403.44565986800001, 2024-04-23T16:31:34.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70691126599999998, 2024-04-23T16:31:34.534 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.534 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.534 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.534 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.534 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.534 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.535 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208917 2024-04-23T16:31:34.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.535 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085422+0000", 2024-04-23T16:31:34.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085422+0000", 2024-04-23T16:31:34.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.536 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085423+0000", 2024-04-23T16:31:34.536 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2309999999999999e-06 2024-04-23T16:31:34.536 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.536 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.536 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.536 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085428+0000", 2024-04-23T16:31:34.536 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2870000000000004e-06 2024-04-23T16:31:34.536 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.536 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.536 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.536 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085437+0000", 2024-04-23T16:31:34.536 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2150000000000007e-06 2024-04-23T16:31:34.536 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085444+0000", 2024-04-23T16:31:34.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9179999999999997e-06 2024-04-23T16:31:34.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085498+0000", 2024-04-23T16:31:34.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4088000000000001e-05 2024-04-23T16:31:34.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085509+0000", 2024-04-23T16:31:34.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1009000000000001e-05 2024-04-23T16:31:34.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792295+0000", 2024-04-23T16:31:34.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70678534800000004 2024-04-23T16:31:34.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792333+0000", 2024-04-23T16:31:34.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8170000000000002e-05 2024-04-23T16:31:34.538 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.538 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.538 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.539 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.539 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208925 3.8 3.a4efd988 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:31:34.539 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085458+0000", 2024-04-23T16:31:34.539 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 403.44562400900003, 2024-04-23T16:31:34.539 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70667339100000004, 2024-04-23T16:31:34.539 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.539 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.539 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.539 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.539 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.539 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.539 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208925 2024-04-23T16:31:34.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.540 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.540 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.540 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085458+0000", 2024-04-23T16:31:34.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.540 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.540 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085458+0000", 2024-04-23T16:31:34.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085458+0000", 2024-04-23T16:31:34.541 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6000000000000003e-07 2024-04-23T16:31:34.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085462+0000", 2024-04-23T16:31:34.542 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4450000000000001e-06 2024-04-23T16:31:34.542 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.542 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.542 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.542 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085468+0000", 2024-04-23T16:31:34.542 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4489999999999996e-06 2024-04-23T16:31:34.542 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.542 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.542 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.542 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085473+0000", 2024-04-23T16:31:34.542 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7260000000000002e-06 2024-04-23T16:31:34.542 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.542 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.543 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.543 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085516+0000", 2024-04-23T16:31:34.543 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3013000000000003e-05 2024-04-23T16:31:34.543 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.543 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.544 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.544 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085523+0000", 2024-04-23T16:31:34.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0169999999999999e-06 2024-04-23T16:31:34.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.544 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.544 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.544 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792056+0000", 2024-04-23T16:31:34.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70653289799999996 2024-04-23T16:31:34.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.544 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.544 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.544 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792131+0000", 2024-04-23T16:31:34.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5183000000000002e-05 2024-04-23T16:31:34.545 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.545 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.545 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.545 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.545 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.545 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208935 3.4 3.ef358d64 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:31:34.546 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085484+0000", 2024-04-23T16:31:34.546 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 403.44559793500002, 2024-04-23T16:31:34.546 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70688482200000002, 2024-04-23T16:31:34.546 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.546 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.546 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.546 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.546 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.546 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.547 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208935 2024-04-23T16:31:34.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.547 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085484+0000", 2024-04-23T16:31:34.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085484+0000", 2024-04-23T16:31:34.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085484+0000", 2024-04-23T16:31:34.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8500000000000001e-07 2024-04-23T16:31:34.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085488+0000", 2024-04-23T16:31:34.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3009999999999998e-06 2024-04-23T16:31:34.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085492+0000", 2024-04-23T16:31:34.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1749999999999998e-06 2024-04-23T16:31:34.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.549 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.549 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085496+0000", 2024-04-23T16:31:34.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4190000000000004e-06 2024-04-23T16:31:34.549 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.549 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.549 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085542+0000", 2024-04-23T16:31:34.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5799e-05 2024-04-23T16:31:34.549 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.549 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.549 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085551+0000", 2024-04-23T16:31:34.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0860000000000008e-06 2024-04-23T16:31:34.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792360+0000", 2024-04-23T16:31:34.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70680912900000004 2024-04-23T16:31:34.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792368+0000", 2024-04-23T16:31:34.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2279999999999998e-06 2024-04-23T16:31:34.550 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.550 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.551 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.551 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.551 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208938 3.4 3.b83bb9a4 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:31:34.551 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085502+0000", 2024-04-23T16:31:34.551 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 403.44557927900001, 2024-04-23T16:31:34.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70692120599999997, 2024-04-23T16:31:34.551 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.551 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.551 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.551 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.551 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.551 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.551 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208938 2024-04-23T16:31:34.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.552 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.552 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.552 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085502+0000", 2024-04-23T16:31:34.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.552 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.552 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085502+0000", 2024-04-23T16:31:34.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.552 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085503+0000", 2024-04-23T16:31:34.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2300000000000001e-07 2024-04-23T16:31:34.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085505+0000", 2024-04-23T16:31:34.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.469e-06 2024-04-23T16:31:34.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085511+0000", 2024-04-23T16:31:34.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7819999999999999e-06 2024-04-23T16:31:34.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085517+0000", 2024-04-23T16:31:34.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7740000000000001e-06 2024-04-23T16:31:34.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085578+0000", 2024-04-23T16:31:34.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1475000000000003e-05 2024-04-23T16:31:34.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085586+0000", 2024-04-23T16:31:34.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8140000000000001e-06 2024-04-23T16:31:34.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792413+0000", 2024-04-23T16:31:34.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70682651100000005 2024-04-23T16:31:34.555 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792424+0000", 2024-04-23T16:31:34.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0858e-05 2024-04-23T16:31:34.555 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.555 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.555 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.555 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.555 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208939 3.8 3.5b6d9248 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:31:34.556 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085539+0000", 2024-04-23T16:31:34.556 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 403.44554305000003, 2024-04-23T16:31:34.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70665678499999995, 2024-04-23T16:31:34.556 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.556 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.556 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.556 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.556 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.556 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.556 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208939 2024-04-23T16:31:34.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.556 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.557 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085539+0000", 2024-04-23T16:31:34.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.557 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.557 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085539+0000", 2024-04-23T16:31:34.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.557 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.557 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085539+0000", 2024-04-23T16:31:34.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0200000000000001e-07 2024-04-23T16:31:34.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085543+0000", 2024-04-23T16:31:34.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4209999999999999e-06 2024-04-23T16:31:34.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.558 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085549+0000", 2024-04-23T16:31:34.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.959e-06 2024-04-23T16:31:34.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.558 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085554+0000", 2024-04-23T16:31:34.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7189999999999998e-06 2024-04-23T16:31:34.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085587+0000", 2024-04-23T16:31:34.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2540999999999999e-05 2024-04-23T16:31:34.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085591+0000", 2024-04-23T16:31:34.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8510000000000001e-06 2024-04-23T16:31:34.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792182+0000", 2024-04-23T16:31:34.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70659176199999996 2024-04-23T16:31:34.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.560 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792195+0000", 2024-04-23T16:31:34.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2830000000000001e-05 2024-04-23T16:31:34.560 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.560 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.560 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.560 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208948 3.8 3.9d5e98c8 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:31:34.560 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085565+0000", 2024-04-23T16:31:34.560 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 403.44551664800002, 2024-04-23T16:31:34.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70682216099999995, 2024-04-23T16:31:34.561 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.561 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.561 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.561 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.561 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.561 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.561 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208948 2024-04-23T16:31:34.561 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.561 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.561 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.561 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.561 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085565+0000", 2024-04-23T16:31:34.561 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.561 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.561 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.562 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.562 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085565+0000", 2024-04-23T16:31:34.562 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.562 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.562 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085566+0000", 2024-04-23T16:31:34.562 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2600000000000002e-07 2024-04-23T16:31:34.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.562 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.562 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085568+0000", 2024-04-23T16:31:34.562 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7769999999999998e-06 2024-04-23T16:31:34.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.563 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.563 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085575+0000", 2024-04-23T16:31:34.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0230000000000004e-06 2024-04-23T16:31:34.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.563 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.563 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085580+0000", 2024-04-23T16:31:34.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5499999999999996e-06 2024-04-23T16:31:34.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.563 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.563 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085609+0000", 2024-04-23T16:31:34.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9332999999999999e-05 2024-04-23T16:31:34.564 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.564 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.564 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085613+0000", 2024-04-23T16:31:34.564 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7799999999999998e-06 2024-04-23T16:31:34.564 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.564 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.564 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792374+0000", 2024-04-23T16:31:34.564 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70676071600000001 2024-04-23T16:31:34.564 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.564 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.564 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792387+0000", 2024-04-23T16:31:34.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3256e-05 2024-04-23T16:31:34.565 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.565 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.565 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.565 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.565 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.565 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208949 3.8 3.32f0e298 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:31:34.565 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085593+0000", 2024-04-23T16:31:34.565 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 403.44548808100001, 2024-04-23T16:31:34.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70685236200000001, 2024-04-23T16:31:34.565 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.565 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.565 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.565 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.566 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.566 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.566 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208949 2024-04-23T16:31:34.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.566 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085593+0000", 2024-04-23T16:31:34.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085593+0000", 2024-04-23T16:31:34.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.567 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085594+0000", 2024-04-23T16:31:34.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4000000000000002e-07 2024-04-23T16:31:34.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.567 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085596+0000", 2024-04-23T16:31:34.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4480000000000001e-06 2024-04-23T16:31:34.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.567 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085603+0000", 2024-04-23T16:31:34.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2899999999999999e-06 2024-04-23T16:31:34.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085607+0000", 2024-04-23T16:31:34.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3610000000000003e-06 2024-04-23T16:31:34.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085651+0000", 2024-04-23T16:31:34.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4075000000000001e-05 2024-04-23T16:31:34.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.569 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085659+0000", 2024-04-23T16:31:34.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1079999999999993e-06 2024-04-23T16:31:34.569 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.569 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.569 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.569 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792436+0000", 2024-04-23T16:31:34.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70677668500000002 2024-04-23T16:31:34.569 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.569 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.569 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.569 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792446+0000", 2024-04-23T16:31:34.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9550000000000004e-06 2024-04-23T16:31:34.569 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.569 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.569 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.570 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208950 3.4 3.4636ca44 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:31:34.570 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085613+0000", 2024-04-23T16:31:34.570 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 403.44546867000003, 2024-04-23T16:31:34.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70686611700000002, 2024-04-23T16:31:34.570 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.570 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.570 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.570 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.570 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.570 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.570 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208950 2024-04-23T16:31:34.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.571 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085613+0000", 2024-04-23T16:31:34.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085613+0000", 2024-04-23T16:31:34.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085613+0000", 2024-04-23T16:31:34.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4799999999999999e-07 2024-04-23T16:31:34.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085615+0000", 2024-04-23T16:31:34.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.951e-06 2024-04-23T16:31:34.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085621+0000", 2024-04-23T16:31:34.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6890000000000001e-06 2024-04-23T16:31:34.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.573 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.573 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085624+0000", 2024-04-23T16:31:34.573 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4960000000000001e-06 2024-04-23T16:31:34.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.573 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.573 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085659+0000", 2024-04-23T16:31:34.573 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4516e-05 2024-04-23T16:31:34.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.573 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.573 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085665+0000", 2024-04-23T16:31:34.573 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.643e-06 2024-04-23T16:31:34.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.574 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.574 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792468+0000", 2024-04-23T16:31:34.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70680334700000003 2024-04-23T16:31:34.574 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.574 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.574 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.574 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792479+0000", 2024-04-23T16:31:34.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1127e-05 2024-04-23T16:31:34.574 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.574 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.574 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.574 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.574 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.574 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208951 3.4 3.7f30f4f4 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:31:34.575 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085630+0000", 2024-04-23T16:31:34.575 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 403.44545115199998, 2024-04-23T16:31:34.575 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70690233800000002, 2024-04-23T16:31:34.575 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.575 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.575 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.575 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.575 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.575 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.575 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208951 2024-04-23T16:31:34.575 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.575 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.575 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.575 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.576 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085630+0000", 2024-04-23T16:31:34.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.576 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.576 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.576 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085630+0000", 2024-04-23T16:31:34.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.576 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.576 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.576 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085631+0000", 2024-04-23T16:31:34.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8299999999999997e-07 2024-04-23T16:31:34.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.576 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.577 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085633+0000", 2024-04-23T16:31:34.577 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3970000000000001e-06 2024-04-23T16:31:34.577 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.577 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.577 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085639+0000", 2024-04-23T16:31:34.577 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7810000000000002e-06 2024-04-23T16:31:34.577 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.577 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.577 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085644+0000", 2024-04-23T16:31:34.577 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7299999999999996e-06 2024-04-23T16:31:34.577 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.577 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.578 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.578 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085693+0000", 2024-04-23T16:31:34.578 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9537999999999999e-05 2024-04-23T16:31:34.578 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.578 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.578 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.578 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085702+0000", 2024-04-23T16:31:34.578 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1309999999999995e-06 2024-04-23T16:31:34.578 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.578 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.578 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.578 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792520+0000", 2024-04-23T16:31:34.578 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70681792300000001 2024-04-23T16:31:34.579 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.579 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.579 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.579 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792533+0000", 2024-04-23T16:31:34.579 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2354999999999999e-05 2024-04-23T16:31:34.579 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.579 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.579 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.579 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.579 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.579 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208989 3.8 3.631ac508 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:31:34.579 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085652+0000", 2024-04-23T16:31:34.579 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 403.445429569, 2024-04-23T16:31:34.579 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70683001599999995, 2024-04-23T16:31:34.580 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.580 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.580 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.580 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.580 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.580 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.580 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208989 2024-04-23T16:31:34.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.580 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.580 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.580 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.580 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085652+0000", 2024-04-23T16:31:34.580 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.581 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085652+0000", 2024-04-23T16:31:34.581 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.581 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.581 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085653+0000", 2024-04-23T16:31:34.581 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9699999999999996e-07 2024-04-23T16:31:34.581 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.581 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085655+0000", 2024-04-23T16:31:34.581 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7810000000000001e-06 2024-04-23T16:31:34.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.582 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085661+0000", 2024-04-23T16:31:34.582 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1930000000000001e-06 2024-04-23T16:31:34.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.582 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085666+0000", 2024-04-23T16:31:34.582 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6579999999999999e-06 2024-04-23T16:31:34.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.582 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085695+0000", 2024-04-23T16:31:34.583 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8711999999999999e-05 2024-04-23T16:31:34.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.583 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.583 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.583 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085699+0000", 2024-04-23T16:31:34.583 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1500000000000001e-06 2024-04-23T16:31:34.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.583 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.583 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.583 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792474+0000", 2024-04-23T16:31:34.583 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70677516500000004 2024-04-23T16:31:34.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.583 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.584 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.584 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792482+0000", 2024-04-23T16:31:34.584 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7600000000000002e-06 2024-04-23T16:31:34.584 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.584 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.584 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.584 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.584 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.584 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208990 3.4 3.b71d3f14 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:31:34.584 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085676+0000", 2024-04-23T16:31:34.584 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 403.44540558300002, 2024-04-23T16:31:34.584 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70695150200000001, 2024-04-23T16:31:34.584 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.585 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.585 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.585 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.585 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.585 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.585 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208990 2024-04-23T16:31:34.585 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.585 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.585 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.585 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.585 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085676+0000", 2024-04-23T16:31:34.585 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.585 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.585 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.586 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.586 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085676+0000", 2024-04-23T16:31:34.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.586 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.586 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.586 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.586 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085677+0000", 2024-04-23T16:31:34.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6300000000000005e-07 2024-04-23T16:31:34.586 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.586 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.586 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.586 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085680+0000", 2024-04-23T16:31:34.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9799999999999998e-06 2024-04-23T16:31:34.586 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.587 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.587 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.587 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085686+0000", 2024-04-23T16:31:34.587 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9669999999999998e-06 2024-04-23T16:31:34.587 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.587 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.587 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.587 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085691+0000", 2024-04-23T16:31:34.587 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2000000000000002e-06 2024-04-23T16:31:34.587 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.587 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.587 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.587 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085730+0000", 2024-04-23T16:31:34.587 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9388999999999998e-05 2024-04-23T16:31:34.588 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.588 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.588 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.588 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085737+0000", 2024-04-23T16:31:34.588 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7129999999999999e-06 2024-04-23T16:31:34.588 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.588 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.588 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.588 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792616+0000", 2024-04-23T16:31:34.588 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70687957599999995 2024-04-23T16:31:34.588 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.588 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.588 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.589 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792627+0000", 2024-04-23T16:31:34.589 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1014e-05 2024-04-23T16:31:34.589 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.589 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.589 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.589 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.589 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.589 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208925 3.8 3.a4efd988 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:31:34.589 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820364+0000", 2024-04-23T16:31:34.589 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 401.71071727600003, 2024-04-23T16:31:34.589 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55176175500000002, 2024-04-23T16:31:34.589 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.589 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.589 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.590 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.590 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.590 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.590 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208925 2024-04-23T16:31:34.590 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.590 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.590 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.590 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.590 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820364+0000", 2024-04-23T16:31:34.590 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.590 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.590 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.590 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.591 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820364+0000", 2024-04-23T16:31:34.591 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.591 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.591 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.591 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.591 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820368+0000", 2024-04-23T16:31:34.591 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8060000000000001e-06 2024-04-23T16:31:34.591 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.591 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.591 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.591 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820379+0000", 2024-04-23T16:31:34.591 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0634e-05 2024-04-23T16:31:34.591 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.591 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.592 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.592 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820394+0000", 2024-04-23T16:31:34.592 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5554e-05 2024-04-23T16:31:34.592 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.592 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.592 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820408+0000", 2024-04-23T16:31:34.592 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3508e-05 2024-04-23T16:31:34.592 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.592 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.592 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820486+0000", 2024-04-23T16:31:34.592 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7844999999999998e-05 2024-04-23T16:31:34.592 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.592 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.593 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820495+0000", 2024-04-23T16:31:34.593 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3610000000000007e-06 2024-04-23T16:31:34.593 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.593 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.593 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.593 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372103+0000", 2024-04-23T16:31:34.593 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55160793100000005 2024-04-23T16:31:34.593 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.593 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.593 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.593 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372126+0000", 2024-04-23T16:31:34.593 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3116e-05 2024-04-23T16:31:34.593 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.593 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.593 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.593 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.594 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.594 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208939 3.8 3.5b6d9248 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:31:34.594 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820453+0000", 2024-04-23T16:31:34.594 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 401.71062826299999, 2024-04-23T16:31:34.594 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55183821600000005, 2024-04-23T16:31:34.594 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.594 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.594 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.594 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.594 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.594 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.594 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208939 2024-04-23T16:31:34.594 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.594 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.595 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.595 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.595 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820453+0000", 2024-04-23T16:31:34.595 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.595 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.595 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.595 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.595 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820453+0000", 2024-04-23T16:31:34.595 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.595 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.595 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.595 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.595 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820454+0000", 2024-04-23T16:31:34.595 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0270000000000001e-06 2024-04-23T16:31:34.596 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.596 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.596 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.596 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820458+0000", 2024-04-23T16:31:34.596 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6430000000000001e-06 2024-04-23T16:31:34.596 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.596 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.596 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.596 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820465+0000", 2024-04-23T16:31:34.596 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2679999999999997e-06 2024-04-23T16:31:34.596 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.596 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.596 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.596 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820471+0000", 2024-04-23T16:31:34.597 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9529999999999996e-06 2024-04-23T16:31:34.597 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.597 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.597 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.597 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820615+0000", 2024-04-23T16:31:34.597 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014426700000000001 2024-04-23T16:31:34.597 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.597 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.597 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.597 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820622+0000", 2024-04-23T16:31:34.597 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2570000000000004e-06 2024-04-23T16:31:34.597 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.597 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.597 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.598 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372284+0000", 2024-04-23T16:31:34.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55166234199999997 2024-04-23T16:31:34.598 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.598 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.598 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.598 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372292+0000", 2024-04-23T16:31:34.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4590000000000001e-06 2024-04-23T16:31:34.598 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.598 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.598 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.598 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.598 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.599 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208948 3.8 3.9d5e98c8 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:31:34.599 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820480+0000", 2024-04-23T16:31:34.599 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 401.71060198599997, 2024-04-23T16:31:34.599 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55184634399999999, 2024-04-23T16:31:34.599 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.599 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.599 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.599 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.599 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.599 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.599 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208948 2024-04-23T16:31:34.599 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.599 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.599 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820480+0000", 2024-04-23T16:31:34.600 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820480+0000", 2024-04-23T16:31:34.600 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820480+0000", 2024-04-23T16:31:34.600 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4199999999999996e-07 2024-04-23T16:31:34.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.601 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.601 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.601 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820482+0000", 2024-04-23T16:31:34.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1919999999999999e-06 2024-04-23T16:31:34.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.601 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.601 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.601 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820487+0000", 2024-04-23T16:31:34.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7990000000000001e-06 2024-04-23T16:31:34.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.601 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.601 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.601 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820492+0000", 2024-04-23T16:31:34.602 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1869999999999996e-06 2024-04-23T16:31:34.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.602 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820654+0000", 2024-04-23T16:31:34.602 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016204500000000001 2024-04-23T16:31:34.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.602 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820659+0000", 2024-04-23T16:31:34.602 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8019999999999999e-06 2024-04-23T16:31:34.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.603 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372321+0000", 2024-04-23T16:31:34.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.551661548 2024-04-23T16:31:34.603 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.603 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.603 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.603 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372326+0000", 2024-04-23T16:31:34.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2290000000000002e-06 2024-04-23T16:31:34.603 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.603 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.603 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.603 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.603 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.603 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208949 3.8 3.32f0e298 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:31:34.603 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820498+0000", 2024-04-23T16:31:34.603 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 401.71058318199999, 2024-04-23T16:31:34.604 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55185574800000003, 2024-04-23T16:31:34.604 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.604 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.604 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.604 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.604 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.604 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.604 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208949 2024-04-23T16:31:34.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.604 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.604 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.604 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.604 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820498+0000", 2024-04-23T16:31:34.605 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.605 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.605 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.605 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.605 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820498+0000", 2024-04-23T16:31:34.605 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.605 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.605 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.605 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.605 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820499+0000", 2024-04-23T16:31:34.605 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6499999999999999e-07 2024-04-23T16:31:34.605 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.605 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.605 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.606 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820501+0000", 2024-04-23T16:31:34.606 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1090000000000001e-06 2024-04-23T16:31:34.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.606 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.606 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.606 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820506+0000", 2024-04-23T16:31:34.606 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.305e-06 2024-04-23T16:31:34.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.606 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.606 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.606 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820511+0000", 2024-04-23T16:31:34.606 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1869999999999996e-06 2024-04-23T16:31:34.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.606 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.607 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.607 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820688+0000", 2024-04-23T16:31:34.607 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000176353 2024-04-23T16:31:34.607 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.607 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.607 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.607 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820696+0000", 2024-04-23T16:31:34.607 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9410000000000007e-06 2024-04-23T16:31:34.607 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.607 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.607 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.607 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372350+0000", 2024-04-23T16:31:34.607 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55165407099999997 2024-04-23T16:31:34.607 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.608 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.608 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.608 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372354+0000", 2024-04-23T16:31:34.608 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3170000000000003e-06 2024-04-23T16:31:34.608 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.608 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.608 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.608 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.608 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.608 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208989 3.8 3.631ac508 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:31:34.608 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820517+0000", 2024-04-23T16:31:34.608 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 401.71056467900002, 2024-04-23T16:31:34.608 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55195477900000001, 2024-04-23T16:31:34.608 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.609 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.609 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.609 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.609 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.609 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.609 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208989 2024-04-23T16:31:34.609 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.609 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.609 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.609 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.609 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820517+0000", 2024-04-23T16:31:34.609 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.609 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.610 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.610 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.610 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820517+0000", 2024-04-23T16:31:34.610 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.610 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.610 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.610 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.610 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820517+0000", 2024-04-23T16:31:34.610 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2500000000000001e-07 2024-04-23T16:31:34.610 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.610 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.610 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.610 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820519+0000", 2024-04-23T16:31:34.610 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0760000000000001e-06 2024-04-23T16:31:34.611 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.611 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.611 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.611 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820524+0000", 2024-04-23T16:31:34.611 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.814e-06 2024-04-23T16:31:34.611 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.611 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.611 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.611 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820529+0000", 2024-04-23T16:31:34.611 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5809999999999998e-06 2024-04-23T16:31:34.611 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.611 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.611 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.611 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820738+0000", 2024-04-23T16:31:34.611 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020955600000000001 2024-04-23T16:31:34.612 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.612 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.612 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.612 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820748+0000", 2024-04-23T16:31:34.612 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9760000000000007e-06 2024-04-23T16:31:34.612 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.612 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.612 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.612 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372466+0000", 2024-04-23T16:31:34.612 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55171814399999997 2024-04-23T16:31:34.612 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.612 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.612 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.612 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372472+0000", 2024-04-23T16:31:34.612 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2070000000000003e-06 2024-04-23T16:31:34.613 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.613 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.613 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.613 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.613 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.613 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208935 3.4 3.ef358d64 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:31:34.613 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820748+0000", 2024-04-23T16:31:34.613 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 401.71033395900002, 2024-04-23T16:31:34.613 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.12019948499999999, 2024-04-23T16:31:34.613 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.613 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.613 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.613 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.613 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.613 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.613 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208935 2024-04-23T16:31:34.614 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.614 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.614 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.614 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.614 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820748+0000", 2024-04-23T16:31:34.614 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.614 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.614 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.614 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.614 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820748+0000", 2024-04-23T16:31:34.614 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.614 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.614 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.615 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.615 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820749+0000", 2024-04-23T16:31:34.615 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2619999999999999e-06 2024-04-23T16:31:34.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.615 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.615 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.615 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820753+0000", 2024-04-23T16:31:34.615 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5839999999999996e-06 2024-04-23T16:31:34.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.615 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.615 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.615 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820762+0000", 2024-04-23T16:31:34.615 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0439999999999996e-06 2024-04-23T16:31:34.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.616 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.616 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.616 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820770+0000", 2024-04-23T16:31:34.616 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3280000000000006e-06 2024-04-23T16:31:34.616 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.616 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.616 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.616 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820865+0000", 2024-04-23T16:31:34.616 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5311999999999998e-05 2024-04-23T16:31:34.616 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.616 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.616 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.616 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820882+0000", 2024-04-23T16:31:34.616 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7266999999999999e-05 2024-04-23T16:31:34.617 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.617 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.617 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.617 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.940935+0000", 2024-04-23T16:31:34.617 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.120052908 2024-04-23T16:31:34.617 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.617 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.617 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.617 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.940947+0000", 2024-04-23T16:31:34.617 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1780000000000001e-05 2024-04-23T16:31:34.617 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.617 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.617 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.618 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.618 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.618 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208938 3.4 3.b83bb9a4 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:31:34.618 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820786+0000", 2024-04-23T16:31:34.618 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 401.71029606100001, 2024-04-23T16:31:34.618 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.120213972, 2024-04-23T16:31:34.618 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.618 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.618 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.618 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.618 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.618 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.618 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208938 2024-04-23T16:31:34.619 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.619 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.619 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.619 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.619 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820786+0000", 2024-04-23T16:31:34.619 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.619 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.619 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.619 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.619 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820786+0000", 2024-04-23T16:31:34.619 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.619 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.619 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.619 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.619 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820787+0000", 2024-04-23T16:31:34.620 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.094e-06 2024-04-23T16:31:34.620 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.620 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.620 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.620 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820791+0000", 2024-04-23T16:31:34.620 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4700000000000004e-06 2024-04-23T16:31:34.620 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.620 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.620 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.620 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820800+0000", 2024-04-23T16:31:34.620 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4059999999999994e-06 2024-04-23T16:31:34.620 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.620 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.620 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.621 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820809+0000", 2024-04-23T16:31:34.621 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1689999999999992e-06 2024-04-23T16:31:34.621 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.621 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.621 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.621 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820932+0000", 2024-04-23T16:31:34.621 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012330800000000001 2024-04-23T16:31:34.621 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.621 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.621 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.621 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820942+0000", 2024-04-23T16:31:34.621 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9480000000000003e-06 2024-04-23T16:31:34.621 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.621 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.622 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.622 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.940991+0000", 2024-04-23T16:31:34.622 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.120049106 2024-04-23T16:31:34.622 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.622 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.622 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.622 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.940999+0000", 2024-04-23T16:31:34.622 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4710000000000007e-06 2024-04-23T16:31:34.622 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.622 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.622 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.622 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.622 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.622 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208951 3.4 3.7f30f4f4 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:31:34.623 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820918+0000", 2024-04-23T16:31:34.623 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 401.71016366600003, 2024-04-23T16:31:34.623 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.120268738, 2024-04-23T16:31:34.623 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.623 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.623 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.623 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.623 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.623 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.623 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208951 2024-04-23T16:31:34.623 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.623 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.623 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.623 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.624 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820918+0000", 2024-04-23T16:31:34.624 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.624 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.624 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.624 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.624 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820918+0000", 2024-04-23T16:31:34.624 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.624 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.624 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.624 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.624 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820919+0000", 2024-04-23T16:31:34.624 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9500000000000001e-07 2024-04-23T16:31:34.624 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.624 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.625 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.625 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820925+0000", 2024-04-23T16:31:34.625 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.835e-06 2024-04-23T16:31:34.625 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.625 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.625 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.625 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820933+0000", 2024-04-23T16:31:34.625 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3739999999999998e-06 2024-04-23T16:31:34.625 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.625 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.625 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.625 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820941+0000", 2024-04-23T16:31:34.625 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8220000000000007e-06 2024-04-23T16:31:34.625 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.626 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.626 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.626 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.821053+0000", 2024-04-23T16:31:34.626 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000112617 2024-04-23T16:31:34.626 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.626 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.626 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.626 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.821059+0000", 2024-04-23T16:31:34.626 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1479999999999997e-06 2024-04-23T16:31:34.626 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.626 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.626 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.626 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.941182+0000", 2024-04-23T16:31:34.626 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.12012339399999999 2024-04-23T16:31:34.627 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.627 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.627 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.627 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.941187+0000", 2024-04-23T16:31:34.627 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6530000000000002e-06 2024-04-23T16:31:34.627 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.627 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.627 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.627 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.627 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.627 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208990 3.4 3.b71d3f14 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:31:34.627 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820957+0000", 2024-04-23T16:31:34.627 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 401.710124404, 2024-04-23T16:31:34.628 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.120262297, 2024-04-23T16:31:34.628 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:31:34.628 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:31:34.628 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:31:34.628 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:31:34.628 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:31:34.628 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:31:34.628 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208990 2024-04-23T16:31:34.628 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.628 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:31:34.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.628 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:31:34.628 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820957+0000", 2024-04-23T16:31:34.628 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.629 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.629 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:31:34.629 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820957+0000", 2024-04-23T16:31:34.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:31:34.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.629 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.629 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:31:34.629 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820959+0000", 2024-04-23T16:31:34.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3430000000000001e-06 2024-04-23T16:31:34.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.629 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.629 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:31:34.629 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820963+0000", 2024-04-23T16:31:34.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2400000000000001e-06 2024-04-23T16:31:34.630 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.630 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.630 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:31:34.630 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820975+0000", 2024-04-23T16:31:34.630 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1783999999999999e-05 2024-04-23T16:31:34.630 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.630 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.630 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:31:34.630 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820985+0000", 2024-04-23T16:31:34.630 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0105000000000001e-05 2024-04-23T16:31:34.630 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.630 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.630 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.631 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.821085+0000", 2024-04-23T16:31:34.631 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100806 2024-04-23T16:31:34.631 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.631 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.631 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:31:34.631 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.821102+0000", 2024-04-23T16:31:34.631 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6283e-05 2024-04-23T16:31:34.631 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.631 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.631 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:31:34.631 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.941213+0000", 2024-04-23T16:31:34.631 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.12011137199999999 2024-04-23T16:31:34.631 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:31:34.631 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:31:34.632 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:31:34.632 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.941219+0000", 2024-04-23T16:31:34.632 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3640000000000004e-06 2024-04-23T16:31:34.632 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.632 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.632 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.632 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:31:34.632 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:31:34.632 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:31:34.632 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:31:34.662 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:35.115 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.11 to 1 (10000) 2024-04-23T16:31:35.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:35 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3805554703' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:31:35.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:35 smithi188 ceph-mon[44101]: pgmap v1022: 27 pgs: 27 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:35.265 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-23T16:31:35.291 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:35 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3805554703' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:31:35.291 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:35 smithi103 ceph-mon[39629]: pgmap v1022: 27 pgs: 27 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:35.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:35 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3805554703' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:31:35.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:35 smithi148 ceph-mon[44520]: pgmap v1022: 27 pgs: 27 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:35.710 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:31:35.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:35.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:31:35.815 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:31:35.842 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:35.990 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:31:35.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:35.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:31:36.099 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:31:36.125 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:36 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:31:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:36 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:31:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3805554703' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:31:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:36 smithi103 ceph-mon[39629]: osdmap e363: 12 total, 12 up, 10 in 2024-04-23T16:31:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:36 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:31:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:36 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:31:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3805554703' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:31:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:36 smithi148 ceph-mon[44520]: osdmap e363: 12 total, 12 up, 10 in 2024-04-23T16:31:36.381 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-04-23T16:31:36.381 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-04-23T16:31:36.381 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-23T16:31:36.381 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T16:31:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:36 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:31:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:36 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:31:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3805554703' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:31:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:36 smithi188 ceph-mon[44101]: osdmap e363: 12 total, 12 up, 10 in 2024-04-23T16:31:36.574 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:31:36.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:36.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:31:36.678 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:31:36.715 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:36.737 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:31:36.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:36.740 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:31:36.841 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:31:36.870 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:36.942 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:31:36.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:36.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:31:37.046 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:31:37.067 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:31:37.079 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:37.100 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:31:37.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:37.103 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:31:37.126 INFO:teuthology.orchestra.run.smithi103.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-04-23T16:31:37.205 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:31:37.235 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:37.334 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:31:37.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:37.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:31:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:37 smithi148 ceph-mon[44520]: pgmap v1024: 27 pgs: 27 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:37 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1223434979' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:31:37.380 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:37 smithi103 ceph-mon[39629]: pgmap v1024: 27 pgs: 27 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:37.380 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:37 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1223434979' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:31:37.438 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:31:37.465 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:37 smithi188 ceph-mon[44101]: pgmap v1024: 27 pgs: 27 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:37 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1223434979' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:31:37.588 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:31:37.589 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:31:37.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:37.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:31:37.593 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:31:37.693 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:31:37.726 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:37.747 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:31:37.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:37.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:31:37.852 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:31:37.882 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:37.903 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:31:37.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:37.906 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:31:38.007 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:31:38.035 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:38.132 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:31:38.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:38.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:31:38.136 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:31:38.237 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:31:38.265 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:38 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1223434979' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:31:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:38 smithi103 ceph-mon[39629]: osdmap e364: 12 total, 12 up, 10 in 2024-04-23T16:31:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:38 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3438241045' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:31:38.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:38 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1223434979' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:31:38.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:38 smithi188 ceph-mon[44101]: osdmap e364: 12 total, 12 up, 10 in 2024-04-23T16:31:38.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:38 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3438241045' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:31:38.614 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:31:38.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:38.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:31:38.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:38 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1223434979' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:31:38.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:38 smithi148 ceph-mon[44520]: osdmap e364: 12 total, 12 up, 10 in 2024-04-23T16:31:38.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:38 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3438241045' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:31:38.717 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:31:38.745 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:38.764 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:31:38.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:38.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:31:38.869 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:31:38.896 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:39.143 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:31:39.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:39.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:31:39.170 DEBUG:teuthology.orchestra.run.smithi188:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-04-23T16:31:39.172 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-23T16:31:39.173 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-23T16:31:39.247 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:31:39.271 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:hints = 1 2024-04-23T16:31:39.271 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-23T16:31:39.272 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Object prefix: benchmark_data_smithi188_113957 2024-04-23T16:31:39.272 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:31:39.272 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 0 0 0 0 0 0 - 0 2024-04-23T16:31:39.274 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:39 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:31:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:39 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3438241045' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:31:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:39 smithi188 ceph-mon[44101]: osdmap e365: 12 total, 12 up, 10 in 2024-04-23T16:31:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:39 smithi188 ceph-mon[44101]: pgmap v1027: 27 pgs: 27 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:39 smithi188 ceph-mon[44101]: osdmap e366: 12 total, 12 up, 10 in 2024-04-23T16:31:39.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:39 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:31:39.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:39 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3438241045' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:31:39.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:39 smithi103 ceph-mon[39629]: osdmap e365: 12 total, 12 up, 10 in 2024-04-23T16:31:39.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:39 smithi103 ceph-mon[39629]: pgmap v1027: 27 pgs: 27 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:39.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:39 smithi103 ceph-mon[39629]: osdmap e366: 12 total, 12 up, 10 in 2024-04-23T16:31:39.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:39 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:31:39.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:39 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3438241045' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:31:39.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:39 smithi148 ceph-mon[44520]: osdmap e365: 12 total, 12 up, 10 in 2024-04-23T16:31:39.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:39 smithi148 ceph-mon[44520]: pgmap v1027: 27 pgs: 27 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-04-23T16:31:39.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:39 smithi148 ceph-mon[44520]: osdmap e366: 12 total, 12 up, 10 in 2024-04-23T16:31:39.931 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:31:39.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:39.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:31:40.045 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:31:40.080 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:40.108 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:31:40.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:40.111 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:31:40.212 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:31:40.254 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:40.272 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 1 16 6255 6239 389.916 389.938 0.002563 0.00255861 2024-04-23T16:31:40.281 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:31:40.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:40.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:31:40.387 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:31:40.426 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:40 smithi103 ceph-mon[39629]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:31:40.426 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:40 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:31:40.428 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:40.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:40 smithi188 ceph-mon[44101]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:31:40.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:40 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:31:40.534 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:31:40.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:40.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:31:40.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:40 smithi148 ceph-mon[44520]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:31:40.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:40 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:31:40.641 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:31:40.683 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:41.174 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:31:41.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:41.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:31:41.272 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 2 16 11778 11762 367.538 345.188 0.00370902 0.00271692 2024-04-23T16:31:41.282 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:31:41.323 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:41.430 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:31:41.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:41.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:31:41.434 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:41 smithi103 ceph-mon[39629]: pgmap v1029: 27 pgs: 27 active+clean; 22 MiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 3.9 MiB/s wr, 62 op/s 2024-04-23T16:31:41.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:41 smithi188 ceph-mon[44101]: pgmap v1029: 27 pgs: 27 active+clean; 22 MiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 3.9 MiB/s wr, 62 op/s 2024-04-23T16:31:41.535 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:31:41.577 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:41.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:41 smithi148 ceph-mon[44520]: pgmap v1029: 27 pgs: 27 active+clean; 22 MiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 3.9 MiB/s wr, 62 op/s 2024-04-23T16:31:42.272 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 3 15 16700 16685 347.581 307.688 0.00342768 0.0028745 2024-04-23T16:31:42.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:42 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:31:42.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:42 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:31:42.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:42 smithi188 ceph-mon[44101]: osdmap e367: 12 total, 12 up, 10 in 2024-04-23T16:31:42.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:42 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:31:42.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:42 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:31:42.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:42 smithi103 ceph-mon[39629]: osdmap e367: 12 total, 12 up, 10 in 2024-04-23T16:31:42.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:42 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:31:42.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:42 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:31:42.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:42 smithi148 ceph-mon[44520]: osdmap e367: 12 total, 12 up, 10 in 2024-04-23T16:31:43.272 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 4 16 21042 21026 328.509 271.312 0.0029922 0.00304153 2024-04-23T16:31:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:43 smithi103 ceph-mon[39629]: pgmap v1031: 27 pgs: 27 active+clean; 132 MiB data, 2.8 GiB used, 891 GiB / 894 GiB avail; 24 MiB/s wr, 377 op/s 2024-04-23T16:31:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:43 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/379857255' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:31:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:43 smithi188 ceph-mon[44101]: pgmap v1031: 27 pgs: 27 active+clean; 132 MiB data, 2.8 GiB used, 891 GiB / 894 GiB avail; 24 MiB/s wr, 377 op/s 2024-04-23T16:31:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:43 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/379857255' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:31:43.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:43 smithi148 ceph-mon[44520]: pgmap v1031: 27 pgs: 27 active+clean; 132 MiB data, 2.8 GiB used, 891 GiB / 894 GiB avail; 24 MiB/s wr, 377 op/s 2024-04-23T16:31:43.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:43 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/379857255' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:31:44.272 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 5 16 25134 25118 313.95 255.75 0.00166348 0.00315473 2024-04-23T16:31:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:44 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:31:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:44 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:31:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:44 smithi188 ceph-mon[44101]: osdmap e368: 12 total, 12 up, 10 in 2024-04-23T16:31:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:44 smithi188 ceph-mon[44101]: osdmap e369: 12 total, 12 up, 10 in 2024-04-23T16:31:44.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:44 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:31:44.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:44 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:31:44.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:44 smithi103 ceph-mon[39629]: osdmap e368: 12 total, 12 up, 10 in 2024-04-23T16:31:44.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:44 smithi103 ceph-mon[39629]: osdmap e369: 12 total, 12 up, 10 in 2024-04-23T16:31:44.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:44 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:31:44.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:44 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:31:44.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:44 smithi148 ceph-mon[44520]: osdmap e368: 12 total, 12 up, 10 in 2024-04-23T16:31:44.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:44 smithi148 ceph-mon[44520]: osdmap e369: 12 total, 12 up, 10 in 2024-04-23T16:31:45.272 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 6 16 28732 28716 299.101 224.875 0.0027025 0.00334176 2024-04-23T16:31:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:45 smithi188 ceph-mon[44101]: pgmap v1034: 27 pgs: 27 active+clean; 1.3 GiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 240 MiB/s wr, 3.84k op/s 2024-04-23T16:31:45.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:45 smithi103 ceph-mon[39629]: pgmap v1034: 27 pgs: 27 active+clean; 1.3 GiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 240 MiB/s wr, 3.84k op/s 2024-04-23T16:31:45.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:45 smithi148 ceph-mon[44520]: pgmap v1034: 27 pgs: 27 active+clean; 1.3 GiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 240 MiB/s wr, 3.84k op/s 2024-04-23T16:31:46.272 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 7 16 32261 32245 287.879 220.562 0.0056745 0.00347191 2024-04-23T16:31:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:46 smithi188 ceph-mon[44101]: 1.1 deep-scrub starts 2024-04-23T16:31:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:46 smithi188 ceph-mon[44101]: 1.1 deep-scrub ok 2024-04-23T16:31:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:46 smithi103 ceph-mon[39629]: 1.1 deep-scrub starts 2024-04-23T16:31:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:46 smithi103 ceph-mon[39629]: 1.1 deep-scrub ok 2024-04-23T16:31:46.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:46 smithi148 ceph-mon[44520]: 1.1 deep-scrub starts 2024-04-23T16:31:46.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:46 smithi148 ceph-mon[44520]: 1.1 deep-scrub ok 2024-04-23T16:31:46.733 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:31:46.733 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":367,"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","created":"2024-04-23T16:08:13.477590+0000","modified":"2024-04-23T16:31:41.126686+0000","last_up_change":"2024-04-23T16:27:00.729195+0000","last_in_change":"2024-04-23T16:24:50.783418+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T16:11:22.416777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"303","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"301","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.2899999618530273,"score_stable":3.2899999618530273,"optimal_score":0.82999998331069946,"raw_score_acting":2.7300000190734863,"raw_score_stable":2.7300000190734863,"primary_affinity_weighted":0.98181778192520142,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2024-04-23T16:31:27.271087+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"364","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.2699999809265137,"score_stable":2.2699999809265137,"optimal_score":0.82999998331069946,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.98181778192520142,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}}],"osds":[{"osd":0,"uuid":"4c084047-8f01-4b86-a60c-5592735b5862","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":359,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6801","nonce":4274040097}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6802","nonce":4274040097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6804","nonce":4274040097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6803","nonce":4274040097}]},"public_addr":"172.21.15.103:6801/4274040097","cluster_addr":"172.21.15.103:6802/4274040097","heartbeat_back_addr":"172.21.15.103:6804/4274040097","heartbeat_front_addr":"172.21.15.103:6803/4274040097","state":["exists","up"]},{"osd":1,"uuid":"3438f2bd-5fb7-432c-b383-29e18c442f20","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":232,"up_from":271,"up_thru":192,"down_at":233,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6805","nonce":859282187}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6806","nonce":859282187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6808","nonce":859282187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6807","nonce":859282187}]},"public_addr":"172.21.15.103:6805/859282187","cluster_addr":"172.21.15.103:6806/859282187","heartbeat_back_addr":"172.21.15.103:6808/859282187","heartbeat_front_addr":"172.21.15.103:6807/859282187","state":["exists","up"]},{"osd":2,"uuid":"c17fffb4-8d17-4546-ba24-e5951a2a37fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":359,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6809","nonce":424240088}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6810","nonce":424240088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6812","nonce":424240088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6811","nonce":424240088}]},"public_addr":"172.21.15.103:6809/424240088","cluster_addr":"172.21.15.103:6810/424240088","heartbeat_back_addr":"172.21.15.103:6812/424240088","heartbeat_front_addr":"172.21.15.103:6811/424240088","state":["exists","up"]},{"osd":3,"uuid":"cee91a3d-55dc-4e89-b568-86821632bb1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":274,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6813","nonce":1895183074}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6814","nonce":1895183074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6816","nonce":1895183074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6815","nonce":1895183074}]},"public_addr":"172.21.15.103:6813/1895183074","cluster_addr":"172.21.15.103:6814/1895183074","heartbeat_back_addr":"172.21.15.103:6816/1895183074","heartbeat_front_addr":"172.21.15.103:6815/1895183074","state":["exists","up"]},{"osd":4,"uuid":"61d5385b-2c82-42f8-93c3-8f1baf71c65e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":359,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6800","nonce":794945071}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6801","nonce":794945071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6803","nonce":794945071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6802","nonce":794945071}]},"public_addr":"172.21.15.148:6800/794945071","cluster_addr":"172.21.15.148:6801/794945071","heartbeat_back_addr":"172.21.15.148:6803/794945071","heartbeat_front_addr":"172.21.15.148:6802/794945071","state":["exists","up"]},{"osd":5,"uuid":"5d492189-c3cd-47ad-a12a-ed4d275bcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":359,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6804","nonce":3547200711}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6805","nonce":3547200711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6807","nonce":3547200711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6806","nonce":3547200711}]},"public_addr":"172.21.15.148:6804/3547200711","cluster_addr":"172.21.15.148:6805/3547200711","heartbeat_back_addr":"172.21.15.148:6807/3547200711","heartbeat_front_addr":"172.21.15.148:6806/3547200711","state":["exists","up"]},{"osd":6,"uuid":"89f2379e-cf14-4e69-8747-7535db81b70b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":359,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6808","nonce":366919758}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6809","nonce":366919758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6811","nonce":366919758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6810","nonce":366919758}]},"public_addr":"172.21.15.148:6808/366919758","cluster_addr":"172.21.15.148:6809/366919758","heartbeat_back_addr":"172.21.15.148:6811/366919758","heartbeat_front_addr":"172.21.15.148:6810/366919758","state":["exists","up"]},{"osd":7,"uuid":"f27b035a-558f-4528-80ea-2a00d03f6539","up":1,"in":1,"weight":1,"primary_affinity":0.7818145751953125,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":359,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6812","nonce":3719550888}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6813","nonce":3719550888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6815","nonce":3719550888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6814","nonce":3719550888}]},"public_addr":"172.21.15.148:6812/3719550888","cluster_addr":"172.21.15.148:6813/3719550888","heartbeat_back_addr":"172.21.15.148:6815/3719550888","heartbeat_front_addr":"172.21.15.148:6814/3719550888","state":["exists","up"]},{"osd":8,"uuid":"a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6800","nonce":1151408595}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6801","nonce":1151408595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6803","nonce":1151408595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6802","nonce":1151408595}]},"public_addr":"172.21.15.188:6800/1151408595","cluster_addr":"172.21.15.188:6801/1151408595","heartbeat_back_addr":"172.21.15.188:6803/1151408595","heartbeat_front_addr":"172.21.15.188:6802/1151408595","state":["exists","up"]},{"osd":9,"uuid":"124d9b05-11eb-4622-b6a2-c4d5bd60b7e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":359,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6804","nonce":306501862}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6805","nonce":306501862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6807","nonce":306501862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6806","nonce":306501862}]},"public_addr":"172.21.15.188:6804/306501862","cluster_addr":"172.21.15.188:6805/306501862","heartbeat_back_addr":"172.21.15.188:6807/306501862","heartbeat_front_addr":"172.21.15.188:6806/306501862","state":["exists","up"]},{"osd":10,"uuid":"3dfb9070-e534-4ea4-a25e-ee1c51297447","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6808","nonce":1231263138}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6809","nonce":1231263138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6811","nonce":1231263138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6810","nonce":1231263138}]},"public_addr":"172.21.15.188:6808/1231263138","cluster_addr":"172.21.15.188:6809/1231263138","heartbeat_back_addr":"172.21.15.188:6811/1231263138","heartbeat_front_addr":"172.21.15.188:6810/1231263138","state":["exists","up"]},{"osd":11,"uuid":"bc163925-f623-4f63-ab89-35e2d511a730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":359,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6812","nonce":457992661}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6813","nonce":457992661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6815","nonce":457992661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6814","nonce":457992661}]},"public_addr":"172.21.15.188:6812/457992661","cluster_addr":"172.21.15.188:6813/457992661","heartbeat_back_addr":"172.21.15.188:6815/457992661","heartbeat_front_addr":"172.21.15.188:6814/457992661","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:30:51.626612+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-23T16:30:28.966805+0000","dead_epoch":232},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:30:31.689111+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:31:34.915684+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:31:40.895239+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:30:57.867548+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:31:31.115616+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:31:29.555432+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:31:00.104899+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:31:28.093896+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-23T16:30:52.892014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:30:56.195875+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6800/3535470771":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1188058303":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1510553721":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/281182801":"2024-04-24T16:08:44.450632+0000","172.21.15.103:6800/2049124518":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/466241963":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/2726838698":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/596230573":"2024-04-24T16:09:14.324448+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T16:31:47.272 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 8 16 35900 35884 280.321 227.438 0.00415304 0.00356581 2024-04-23T16:31:47.425 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:47 smithi103 ceph-mon[39629]: 1.8 scrub starts 2024-04-23T16:31:47.425 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:47 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:31:47.425 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:47 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:31:47.425 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:47 smithi103 ceph-mon[39629]: 1.8 scrub ok 2024-04-23T16:31:47.425 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:47 smithi103 ceph-mon[39629]: osdmap e370: 12 total, 12 up, 10 in 2024-04-23T16:31:47.426 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:47 smithi103 ceph-mon[39629]: pgmap v1036: 27 pgs: 27 active+clean; 1.4 GiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 254 MiB/s wr, 4.06k op/s 2024-04-23T16:31:47.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:47 smithi188 ceph-mon[44101]: 1.8 scrub starts 2024-04-23T16:31:47.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:47 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:31:47.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:47 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:31:47.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:47 smithi188 ceph-mon[44101]: 1.8 scrub ok 2024-04-23T16:31:47.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:47 smithi188 ceph-mon[44101]: osdmap e370: 12 total, 12 up, 10 in 2024-04-23T16:31:47.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:47 smithi188 ceph-mon[44101]: pgmap v1036: 27 pgs: 27 active+clean; 1.4 GiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 254 MiB/s wr, 4.06k op/s 2024-04-23T16:31:47.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:47 smithi148 ceph-mon[44520]: 1.8 scrub starts 2024-04-23T16:31:47.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:47 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:31:47.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:47 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:31:47.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:47 smithi148 ceph-mon[44520]: 1.8 scrub ok 2024-04-23T16:31:47.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:47 smithi148 ceph-mon[44520]: osdmap e370: 12 total, 12 up, 10 in 2024-04-23T16:31:47.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:47 smithi148 ceph-mon[44520]: pgmap v1036: 27 pgs: 27 active+clean; 1.4 GiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 254 MiB/s wr, 4.06k op/s 2024-04-23T16:31:47.932 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-23T16:31:48.170 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:31:48.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:48.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:31:48.180 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:31:48.180 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:31:48.180 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-23T16:31:48.180 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:31:48.273 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 9 16 39303 39287 272.804 212.688 0.00496709 0.00366333 2024-04-23T16:31:48.275 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:31:48.310 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:48.335 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:31:48.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:48.341 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:31:48.442 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:31:48.476 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:48.500 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:31:48.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:48.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:31:48.605 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:31:48.651 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:48.666 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:31:49.010 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:31:49.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:49.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:31:49.119 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:31:49.155 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:49.273 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 10 16 42445 42429 265.16 196.375 0.00552239 0.00376951 2024-04-23T16:31:49.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:49 smithi103 ceph-mon[39629]: pgmap v1037: 27 pgs: 27 active+clean; 1.7 GiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 264 MiB/s wr, 4.23k op/s 2024-04-23T16:31:50.116 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-23T16:31:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:49 smithi148 ceph-mon[44520]: pgmap v1037: 27 pgs: 27 active+clean; 1.7 GiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 264 MiB/s wr, 4.23k op/s 2024-04-23T16:31:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:49 smithi188 ceph-mon[44101]: pgmap v1037: 27 pgs: 27 active+clean; 1.7 GiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 264 MiB/s wr, 4.23k op/s 2024-04-23T16:31:50.273 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 11 16 45492 45476 258.365 190.438 0.00873461 0.0038683 2024-04-23T16:31:50.314 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:31:50.318 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:31:50.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:50.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:31:50.392 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:31:50.427 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:31:50.428 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:31:50.458 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:31:50.459 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:50.483 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:31:50.484 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:31:50.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:50.488 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:31:50.519 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:31:50.553 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:31:50.587 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:31:50.588 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:31:50.622 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:31:50.637 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:50.681 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:31:50.682 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:31:50.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:50.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:31:50.709 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:31:50.736 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:31:50.765 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:31:50.771 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:31:50.771 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:31:50.771 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:31:50.772 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:31:50.772 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:31:50.772 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:31:50.772 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:31:50.772 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:31:50.772 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:31:50.772 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:31:50.772 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:31:50.772 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:31:50.787 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:31:50.820 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:50.847 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:31:50.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:50.850 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:31:50.946 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:50 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3048187109' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:31:50.946 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:50.946 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:50.946 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:50.946 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:50.951 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:31:50.999 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:51.030 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:31:51.034 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:50 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3048187109' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:31:51.034 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:51.034 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:51.034 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:51.034 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:51.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:51.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:31:51.136 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:31:51.175 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:51.195 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:31:51.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:51.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:31:51.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:50 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3048187109' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:31:51.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:51.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:51.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:51.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:51.273 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 12 16 48209 48193 250.985 169.812 0.00481144 0.00398282 2024-04-23T16:31:51.299 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:31:51.330 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:51.355 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:31:51.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:51.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:31:51.458 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:31:51.501 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:51.533 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:31:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:51.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:31:51.645 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:31:51.692 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:51.717 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:31:51.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:51.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:31:51.822 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:31:51.854 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:52.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:51 smithi103 ceph-mon[39629]: pgmap v1038: 27 pgs: 27 active+clean; 2.5 GiB data, 7.5 GiB used, 886 GiB / 894 GiB avail; 320 MiB/s wr, 5.12k op/s 2024-04-23T16:31:52.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.112 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:31:52.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:52.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:31:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:51 smithi148 ceph-mon[44520]: pgmap v1038: 27 pgs: 27 active+clean; 2.5 GiB data, 7.5 GiB used, 886 GiB / 894 GiB avail; 320 MiB/s wr, 5.12k op/s 2024-04-23T16:31:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.217 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:31:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:51 smithi188 ceph-mon[44101]: pgmap v1038: 27 pgs: 27 active+clean; 2.5 GiB data, 7.5 GiB used, 886 GiB / 894 GiB avail; 320 MiB/s wr, 5.12k op/s 2024-04-23T16:31:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:31:52.260 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:52.273 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 13 15 50856 50841 244.408 165.5 0.00636112 0.00408959 2024-04-23T16:31:52.291 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:31:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:52.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:31:52.396 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:31:52.435 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:52.466 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:31:52.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:52.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:31:52.571 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:31:52.612 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:52.668 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:31:52.674 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:31:52.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:52.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:31:52.784 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:31:52.816 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:52.841 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:31:52.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:52.846 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:31:52.947 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:31:52.984 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:52 smithi103 ceph-mon[39629]: 1.5 deep-scrub starts 2024-04-23T16:31:52.984 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:52 smithi103 ceph-mon[39629]: 1.5 deep-scrub ok 2024-04-23T16:31:52.990 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:53.023 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:31:53.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:53.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:31:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:52 smithi148 ceph-mon[44520]: 1.5 deep-scrub starts 2024-04-23T16:31:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:52 smithi148 ceph-mon[44520]: 1.5 deep-scrub ok 2024-04-23T16:31:53.128 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:31:53.178 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:53.205 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:31:53.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:53.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:31:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:52 smithi188 ceph-mon[44101]: 1.5 deep-scrub starts 2024-04-23T16:31:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:52 smithi188 ceph-mon[44101]: 1.5 deep-scrub ok 2024-04-23T16:31:53.273 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 14 16 53501 53485 238.753 165.25 0.00259078 0.00418698 2024-04-23T16:31:53.308 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:31:53.355 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:53.386 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:31:53.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:53.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:31:53.493 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:31:53.528 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:53.552 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:31:53.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:53.555 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:31:53.656 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:31:53.697 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:53.728 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:31:53.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:53.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:31:53.832 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:31:53.879 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:53.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:53 smithi188 ceph-mon[44101]: pgmap v1039: 27 pgs: 27 active+clean; 2.6 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 200 MiB/s wr, 3.19k op/s 2024-04-23T16:31:54.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:53 smithi103 ceph-mon[39629]: pgmap v1039: 27 pgs: 27 active+clean; 2.6 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 200 MiB/s wr, 3.19k op/s 2024-04-23T16:31:54.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:53 smithi148 ceph-mon[44520]: pgmap v1039: 27 pgs: 27 active+clean; 2.6 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 200 MiB/s wr, 3.19k op/s 2024-04-23T16:31:54.273 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 15 16 56057 56041 233.486 159.75 0.00463951 0.00428166 2024-04-23T16:31:54.719 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:31:54.743 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:31:54.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:54.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:31:54.799 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:31:54.855 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:31:54.896 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:54.931 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:31:54.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:54.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:31:55.041 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:31:55.060 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:54 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1268280851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:31:55.060 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:54 smithi103 ceph-mon[39629]: from='client.25732 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:31:55.073 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:54 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1268280851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:31:55.073 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:54 smithi148 ceph-mon[44520]: from='client.25732 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:31:55.074 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:55.097 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:31:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:55.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:31:55.203 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:31:55.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:54 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1268280851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:31:55.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:54 smithi188 ceph-mon[44101]: from='client.25732 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:31:55.234 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:55.259 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:31:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:55.262 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:31:55.273 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 16 16 58465 58449 228.298 150.5 0.00642061 0.00437865 2024-04-23T16:31:55.362 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:31:55.394 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:55.456 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:31:55.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:55.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:31:55.567 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:31:55.598 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:55.622 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:31:55.622 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:31:55.623 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:31:55.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:55.627 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:31:55.729 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:31:55.773 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:55.812 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:31:55.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:55.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:31:55.916 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:31:55.946 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:55 smithi188 ceph-mon[44101]: 1.0 deep-scrub starts 2024-04-23T16:31:55.946 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:55 smithi188 ceph-mon[44101]: 1.0 deep-scrub ok 2024-04-23T16:31:55.947 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:55 smithi188 ceph-mon[44101]: 1.a scrub starts 2024-04-23T16:31:55.947 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:55 smithi188 ceph-mon[44101]: pgmap v1040: 27 pgs: 27 active+clean; 3.3 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 202 MiB/s wr, 3.23k op/s 2024-04-23T16:31:55.947 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:55 smithi188 ceph-mon[44101]: 1.a scrub ok 2024-04-23T16:31:55.947 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:55 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:31:55.947 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:55 smithi188 ceph-mon[44101]: from='client.25732 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:31:55.947 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:55 smithi188 ceph-mon[44101]: osdmap e371: 12 total, 12 up, 10 in 2024-04-23T16:31:55.947 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:55.968 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:31:55.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:55.971 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:31:56.072 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:31:56.093 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:55 smithi103 ceph-mon[39629]: 1.0 deep-scrub starts 2024-04-23T16:31:56.093 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:55 smithi103 ceph-mon[39629]: 1.0 deep-scrub ok 2024-04-23T16:31:56.093 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:55 smithi103 ceph-mon[39629]: 1.a scrub starts 2024-04-23T16:31:56.093 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:55 smithi103 ceph-mon[39629]: pgmap v1040: 27 pgs: 27 active+clean; 3.3 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 202 MiB/s wr, 3.23k op/s 2024-04-23T16:31:56.093 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:55 smithi103 ceph-mon[39629]: 1.a scrub ok 2024-04-23T16:31:56.093 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:55 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:31:56.093 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:55 smithi103 ceph-mon[39629]: from='client.25732 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:31:56.093 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:55 smithi103 ceph-mon[39629]: osdmap e371: 12 total, 12 up, 10 in 2024-04-23T16:31:56.100 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:56.124 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:31:56.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:55 smithi148 ceph-mon[44520]: 1.0 deep-scrub starts 2024-04-23T16:31:56.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:55 smithi148 ceph-mon[44520]: 1.0 deep-scrub ok 2024-04-23T16:31:56.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:55 smithi148 ceph-mon[44520]: 1.a scrub starts 2024-04-23T16:31:56.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:55 smithi148 ceph-mon[44520]: pgmap v1040: 27 pgs: 27 active+clean; 3.3 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 202 MiB/s wr, 3.23k op/s 2024-04-23T16:31:56.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:55 smithi148 ceph-mon[44520]: 1.a scrub ok 2024-04-23T16:31:56.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:55 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:31:56.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:55 smithi148 ceph-mon[44520]: from='client.25732 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:31:56.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:55 smithi148 ceph-mon[44520]: osdmap e371: 12 total, 12 up, 10 in 2024-04-23T16:31:56.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:56.126 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:31:56.227 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:31:56.258 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:56.273 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 17 16 60837 60821 223.589 148.25 0.011398 0.0044705 2024-04-23T16:31:56.281 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:31:56.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:56.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:31:56.386 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:31:56.419 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:56.442 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:31:56.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:56.445 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:31:56.546 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:31:56.575 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:56.601 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:31:56.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:56.604 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:31:56.705 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:31:56.738 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:56.894 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:31:56.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:56.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:31:56.933 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-23T16:31:57.001 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:31:57.005 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:31:57.074 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:57.273 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 18 16 63258 63242 219.573 151.312 0.008983 0.0045527 2024-04-23T16:31:57.427 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:31:57.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:57.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:31:57.536 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:31:57.567 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:57.592 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:31:57.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:57.596 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:31:57.697 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:31:57.738 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:57.764 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:31:57.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:57.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:31:57.869 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:31:57.905 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:57.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:57 smithi188 ceph-mon[44101]: pgmap v1042: 27 pgs: 27 active+clean; 3.3 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 198 MiB/s wr, 3.17k op/s 2024-04-23T16:31:57.990 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:31:57.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:57.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:31:57.995 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:57 smithi103 ceph-mon[39629]: pgmap v1042: 27 pgs: 27 active+clean; 3.3 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 198 MiB/s wr, 3.17k op/s 2024-04-23T16:31:58.096 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:31:58.124 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:31:58.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:57 smithi148 ceph-mon[44520]: pgmap v1042: 27 pgs: 27 active+clean; 3.3 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 198 MiB/s wr, 3.17k op/s 2024-04-23T16:31:58.130 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:58.154 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:31:58.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:58.158 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:31:58.259 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:31:58.274 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 19 16 65601 65585 215.723 146.438 0.00395474 0.0046332 2024-04-23T16:31:58.304 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:58.331 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:31:58.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:58.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:31:58.441 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:31:58.484 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:58.510 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:31:58.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:58.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:31:58.581 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:31:58.617 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:31:58.660 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:58.693 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:31:58.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:58.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:31:58.780 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:58 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:31:58.780 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:58 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:31:58.797 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:31:58.837 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:58.879 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:31:58.886 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:31:58.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:58.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:31:58.996 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:31:59.041 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:59.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:58 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:31:59.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:58 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:31:59.152 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:31:59.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:59.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:31:59.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:58 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:31:59.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:58 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:31:59.258 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:31:59.273 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:31:59.273480 min lat: 0.0010294 max lat: 0.0742725 avg lat: 0.00472005 2024-04-23T16:31:59.274 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:31:59.274 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 20 16 67797 67781 211.799 137.25 0.00723726 0.00472005 2024-04-23T16:31:59.297 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:59.345 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:31:59.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:59.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:31:59.450 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:31:59.501 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:59.564 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:31:59.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:31:59.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:31:59.670 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:31:59.713 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:31:59.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:59 smithi103 ceph-mon[39629]: pgmap v1043: 27 pgs: 27 active+clean; 3.5 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 188 MiB/s wr, 3.01k op/s 2024-04-23T16:31:59.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:31:59 smithi103 ceph-mon[39629]: osdmap e372: 12 total, 12 up, 10 in 2024-04-23T16:32:00.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:59 smithi148 ceph-mon[44520]: pgmap v1043: 27 pgs: 27 active+clean; 3.5 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 188 MiB/s wr, 3.01k op/s 2024-04-23T16:32:00.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:31:59 smithi148 ceph-mon[44520]: osdmap e372: 12 total, 12 up, 10 in 2024-04-23T16:32:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:59 smithi188 ceph-mon[44101]: pgmap v1043: 27 pgs: 27 active+clean; 3.5 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 188 MiB/s wr, 3.01k op/s 2024-04-23T16:32:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:31:59 smithi188 ceph-mon[44101]: osdmap e372: 12 total, 12 up, 10 in 2024-04-23T16:32:00.268 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:32:00.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:00.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:32:00.273 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 21 15 70071 70056 208.484 142.188 0.00458627 0.0047947 2024-04-23T16:32:00.373 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:32:00.412 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:00.435 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:32:00.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:00.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:32:00.443 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:32:00.475 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:32:00.504 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:32:00.533 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:32:00.538 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:32:00.573 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:32:00.576 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:00.607 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:32:00.643 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:32:00.676 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:32:00.705 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:32:00.735 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:32:00.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:00.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:32:00.738 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:32:00.769 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:32:00.803 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:32:00.811 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:32:00.811 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:32:00.811 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:32:00.811 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:32:00.811 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:32:00.811 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:32:00.811 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:32:00.812 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:32:00.812 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:32:00.812 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:32:00.812 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:32:00.812 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:32:00.839 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:32:00.840 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:00 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3502939186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:32:00.840 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:00.840 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:00.888 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:00.926 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:32:00.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:00.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:32:01.033 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:32:01.087 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:00 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3502939186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:32:01.087 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:01.087 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:01.089 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:01.117 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:32:01.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:01.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:32:01.223 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:32:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:00 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3502939186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:32:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:01.269 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:01.273 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 22 16 72201 72185 205.055 133.062 0.0083153 0.00487497 2024-04-23T16:32:01.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:01.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:01.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:01.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:01.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:01.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:01.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:01.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:01 smithi103 ceph-mon[39629]: pgmap v1045: 27 pgs: 27 active+clean; 4.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 187 MiB/s wr, 2.99k op/s 2024-04-23T16:32:01.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:01.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:01.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:01 smithi148 ceph-mon[44520]: pgmap v1045: 27 pgs: 27 active+clean; 4.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 187 MiB/s wr, 2.99k op/s 2024-04-23T16:32:02.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:01 smithi188 ceph-mon[44101]: pgmap v1045: 27 pgs: 27 active+clean; 4.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 187 MiB/s wr, 2.99k op/s 2024-04-23T16:32:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:02.274 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 23 16 74428 74412 202.191 139.188 0.00876617 0.00494417 2024-04-23T16:32:02.557 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:32:02.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:02.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:32:02.664 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:32:02.703 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:02.734 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:32:02.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:02.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:32:02.839 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:32:02.873 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:02.898 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:32:02.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:02.904 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:32:02.923 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:32:02.923 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:32:02.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:32:02.923 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-23T16:32:02.923 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:32:03.005 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:32:03.051 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:03.081 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:32:03.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:03.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:32:03.187 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:32:03.226 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:03.249 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:32:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:03.251 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:32:03.274 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 24 16 76548 76532 199.286 132.5 0.00833011 0.00501657 2024-04-23T16:32:03.353 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:32:03.386 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:03.408 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:32:03.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:03.411 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:32:03.512 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:32:03.545 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:03.695 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:32:03.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:03.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:32:03.799 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:32:03.827 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:04.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:03 smithi103 ceph-mon[39629]: pgmap v1046: 27 pgs: 27 active+clean; 4.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-04-23T16:32:04.075 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:32:04.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:04.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:32:04.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:03 smithi148 ceph-mon[44520]: pgmap v1046: 27 pgs: 27 active+clean; 4.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-04-23T16:32:04.179 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:32:04.210 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:03 smithi188 ceph-mon[44101]: pgmap v1046: 27 pgs: 27 active+clean; 4.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-04-23T16:32:04.230 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:32:04.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:04.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:32:04.274 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 25 16 77044 77028 192.555 31 0.00327257 0.0050213 2024-04-23T16:32:04.328 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:32:04.334 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:32:04.362 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:04.382 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:32:04.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:04.385 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:32:04.486 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:32:04.514 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:04.945 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-23T16:32:04.963 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:32:04.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:04.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:32:05.036 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:32:05.069 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:32:05.102 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:05.123 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:32:05.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:05.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:32:05.134 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:32:05.227 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:32:05.270 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:05.274 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 26 16 77185 77169 185.487 8.8125 0.0189076 0.00539028 2024-04-23T16:32:05.303 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:32:05.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:05.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:32:05.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:05 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3061650119' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:32:05.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:05 smithi103 ceph-mon[39629]: from='client.25744 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:32:05.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:05 smithi103 ceph-mon[39629]: pgmap v1047: 27 pgs: 27 active+clean; 4.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-04-23T16:32:05.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:05 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3061650119' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:32:05.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:05 smithi148 ceph-mon[44520]: from='client.25744 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:32:05.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:05 smithi148 ceph-mon[44520]: pgmap v1047: 27 pgs: 27 active+clean; 4.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-04-23T16:32:05.413 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:32:05.451 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:05.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:05 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3061650119' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:32:05.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:05 smithi188 ceph-mon[44101]: from='client.25744 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:32:05.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:05 smithi188 ceph-mon[44101]: pgmap v1047: 27 pgs: 27 active+clean; 4.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-04-23T16:32:05.481 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:32:05.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:05.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:32:05.586 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:32:05.639 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:05.677 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:32:05.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:05.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:32:05.782 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:05.822 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:05.976 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:05.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:05.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:06.081 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:32:06.129 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:06.158 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:32:06.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:06.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:32:06.265 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:32:06.274 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 27 16 79229 79213 183.349 127.75 0.00762312 0.00545246 2024-04-23T16:32:06.303 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:06.326 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:32:06.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:06.329 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:32:06.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:06 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:32:06.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:06 smithi103 ceph-mon[39629]: from='client.25744 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:32:06.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:06 smithi103 ceph-mon[39629]: osdmap e373: 12 total, 12 up, 10 in 2024-04-23T16:32:06.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:06 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:32:06.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:06 smithi148 ceph-mon[44520]: from='client.25744 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:32:06.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:06 smithi148 ceph-mon[44520]: osdmap e373: 12 total, 12 up, 10 in 2024-04-23T16:32:06.430 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:32:06.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:06 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:32:06.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:06 smithi188 ceph-mon[44101]: from='client.25744 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:32:06.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:06 smithi188 ceph-mon[44101]: osdmap e373: 12 total, 12 up, 10 in 2024-04-23T16:32:06.477 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:06.613 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:32:06.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:06.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:32:06.723 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:32:06.768 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:06.798 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:32:06.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:06.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:32:06.905 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:32:06.958 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:32:07.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:07.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:32:07.101 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:32:07.123 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:32:07.143 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:07.172 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:32:07.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:07.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:32:07.274 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 28 16 81263 81247 181.34 127.125 0.00666584 0.00551282 2024-04-23T16:32:07.278 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:32:07.308 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:07.330 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:32:07.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:07.333 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:32:07.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:07 smithi103 ceph-mon[39629]: pgmap v1049: 27 pgs: 27 active+clean; 4.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 147 MiB/s wr, 2.36k op/s 2024-04-23T16:32:07.434 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:07 smithi188 ceph-mon[44101]: pgmap v1049: 27 pgs: 27 active+clean; 4.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 147 MiB/s wr, 2.36k op/s 2024-04-23T16:32:07.485 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:07.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:07 smithi148 ceph-mon[44520]: pgmap v1049: 27 pgs: 27 active+clean; 4.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 147 MiB/s wr, 2.36k op/s 2024-04-23T16:32:07.745 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:07.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:07.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:07.854 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:32:07.893 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:07.933 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-23T16:32:07.933 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:32:08.056 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:32:08.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:08.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:32:08.161 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:32:08.203 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:08.232 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:32:08.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:08.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:32:08.274 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 29 16 83250 83234 179.369 124.188 0.0118958 0.00557352 2024-04-23T16:32:08.339 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:32:08.379 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:08.410 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:32:08.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:08.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:32:08.518 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:08.519 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:32:08.560 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:08.560 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:32:08.591 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:32:08.617 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:32:08.647 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:32:08.650 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:32:08.691 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:32:08.731 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:32:08.765 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:32:08.792 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:32:08.824 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:32:08.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:08 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2706805290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:32:08.857 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:32:08.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:08 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2706805290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:32:08.892 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:32:08.901 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:32:08.901 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:32:08.901 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:32:08.901 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:32:08.901 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:32:08.902 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:32:08.902 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:32:08.902 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:32:08.902 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:32:08.902 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:32:08.902 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:32:08.902 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:32:08.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:08 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2706805290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:32:09.274 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 30 16 85144 85128 177.336 118.375 0.00340312 0.00563735 2024-04-23T16:32:09.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:09 smithi103 ceph-mon[39629]: pgmap v1050: 27 pgs: 27 active+clean; 4.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-04-23T16:32:09.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:09 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:32:09.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:09 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:32:09.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:09 smithi103 ceph-mon[39629]: osdmap e374: 12 total, 12 up, 10 in 2024-04-23T16:32:09.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:09 smithi148 ceph-mon[44520]: pgmap v1050: 27 pgs: 27 active+clean; 4.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-04-23T16:32:09.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:09 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:32:09.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:09 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:32:09.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:09 smithi148 ceph-mon[44520]: osdmap e374: 12 total, 12 up, 10 in 2024-04-23T16:32:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:09 smithi188 ceph-mon[44101]: pgmap v1050: 27 pgs: 27 active+clean; 4.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-04-23T16:32:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:09 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:32:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:09 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:32:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:09.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:09 smithi188 ceph-mon[44101]: osdmap e374: 12 total, 12 up, 10 in 2024-04-23T16:32:10.274 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 31 16 87128 87112 175.615 124 0.00890075 0.0056928 2024-04-23T16:32:10.524 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:10.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:10.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:10.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:10.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:10.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:10.635 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:32:10.694 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:10.724 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:32:10.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:10.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:32:10.840 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:32:10.872 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:32:10.872 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:32:10.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:10.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:10.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:10.888 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:32:10.889 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:10.930 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:32:10.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:10.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:32:10.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:10.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:10.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:11.034 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:32:11.065 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:11.088 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:32:11.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:11.092 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:32:11.121 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:32:11.193 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:32:11.225 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:11.250 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:32:11.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:11.255 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:32:11.274 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 32 16 89016 89000 173.814 118 0.0170812 0.00575182 2024-04-23T16:32:11.356 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:32:11.410 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:11.667 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:32:11.667 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:11 smithi103 ceph-mon[39629]: pgmap v1052: 27 pgs: 27 active+clean; 5.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-04-23T16:32:11.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:11.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:32:11.774 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:32:11.809 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:11.835 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:32:11.836 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:11 smithi188 ceph-mon[44101]: pgmap v1052: 27 pgs: 27 active+clean; 5.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-04-23T16:32:11.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:11.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:32:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:11 smithi148 ceph-mon[44520]: pgmap v1052: 27 pgs: 27 active+clean; 5.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-04-23T16:32:11.939 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:11.976 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:12.184 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:12.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:12.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:12.274 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 33 16 90903 90887 172.121 117.938 0.00756889 0.00580856 2024-04-23T16:32:12.290 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:32:12.352 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:12.359 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:32:12.378 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:32:12.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:12.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:32:12.483 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:32:12.517 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:12.540 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:32:12.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:12.544 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:32:12.645 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:32:12.676 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:12.699 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:32:12.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:12.705 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:32:12.806 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:32:12.847 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:12.875 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:32:12.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:12.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:32:12.981 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:32:13.011 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:13.037 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:32:13.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:13.039 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:32:13.140 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:32:13.174 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:13.197 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:32:13.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:13.201 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:32:13.275 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 34 16 92705 92689 170.371 112.625 0.00387229 0.00586806 2024-04-23T16:32:13.303 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:32:13.339 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:13.605 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:13 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:32:13.605 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:13 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:32:13.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:13 smithi103 ceph-mon[39629]: osdmap e375: 12 total, 12 up, 10 in 2024-04-23T16:32:13.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:13 smithi103 ceph-mon[39629]: pgmap v1054: 27 pgs: 27 active+clean; 5.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-04-23T16:32:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:13 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:32:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:13 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:32:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:13 smithi148 ceph-mon[44520]: osdmap e375: 12 total, 12 up, 10 in 2024-04-23T16:32:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:13 smithi148 ceph-mon[44520]: pgmap v1054: 27 pgs: 27 active+clean; 5.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-04-23T16:32:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:13 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:32:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:13 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:32:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:13 smithi188 ceph-mon[44101]: osdmap e375: 12 total, 12 up, 10 in 2024-04-23T16:32:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:13 smithi188 ceph-mon[44101]: pgmap v1054: 27 pgs: 27 active+clean; 5.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-04-23T16:32:14.275 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 35 16 94495 94479 168.699 111.875 0.003835 0.00592635 2024-04-23T16:32:15.275 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 36 15 96189 96174 166.955 105.938 0.0150513 0.00598645 2024-04-23T16:32:15.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:15 smithi148 ceph-mon[44520]: osdmap e376: 12 total, 12 up, 10 in 2024-04-23T16:32:15.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:15 smithi148 ceph-mon[44520]: pgmap v1056: 27 pgs: 27 active+clean; 5.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-04-23T16:32:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:15 smithi188 ceph-mon[44101]: osdmap e376: 12 total, 12 up, 10 in 2024-04-23T16:32:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:15 smithi188 ceph-mon[44101]: pgmap v1056: 27 pgs: 27 active+clean; 5.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-04-23T16:32:15.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:15 smithi103 ceph-mon[39629]: osdmap e376: 12 total, 12 up, 10 in 2024-04-23T16:32:15.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:15 smithi103 ceph-mon[39629]: pgmap v1056: 27 pgs: 27 active+clean; 5.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-04-23T16:32:16.275 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 37 16 97947 97931 165.411 109.812 0.0120859 0.00604408 2024-04-23T16:32:17.275 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 38 16 99734 99718 163.997 111.688 0.00656387 0.00609643 2024-04-23T16:32:18.000 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:32:18.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:18.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:32:18.007 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:17 smithi103 ceph-mon[39629]: pgmap v1057: 27 pgs: 27 active+clean; 5.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-04-23T16:32:18.107 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:32:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:17 smithi148 ceph-mon[44520]: pgmap v1057: 27 pgs: 27 active+clean; 5.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-04-23T16:32:18.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:17 smithi188 ceph-mon[44101]: pgmap v1057: 27 pgs: 27 active+clean; 5.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-04-23T16:32:18.140 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:18.163 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:32:18.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:18.166 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:32:18.267 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:32:18.275 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 39 15 101528 101513 162.668 112.188 0.00775337 0.0061462 2024-04-23T16:32:18.313 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:19.275 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:32:19.275100 min lat: 0.0010294 max lat: 1.8332 avg lat: 0.00620975 2024-04-23T16:32:19.275 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:32:19.275 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 40 16 103061 103045 160.995 95.75 0.00728187 0.00620975 2024-04-23T16:32:20.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:19 smithi103 ceph-mon[39629]: pgmap v1058: 27 pgs: 27 active+clean; 5.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-23T16:32:20.066 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:32:20.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:20.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:32:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:19 smithi148 ceph-mon[44520]: pgmap v1058: 27 pgs: 27 active+clean; 5.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-23T16:32:20.177 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:32:20.219 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:20.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:19 smithi188 ceph-mon[44101]: pgmap v1058: 27 pgs: 27 active+clean; 5.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-23T16:32:20.275 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 41 16 104703 104687 159.571 102.625 0.00334323 0.00626443 2024-04-23T16:32:21.076 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:32:21.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:21.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:32:21.112 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-23T16:32:21.191 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:32:21.229 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:21.255 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:32:21.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:21.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:32:21.275 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 42 15 106249 106234 158.074 96.6875 0.0125617 0.00632494 2024-04-23T16:32:21.358 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:32:21.386 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:21.408 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:32:21.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:21.411 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:32:21.512 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:32:21.564 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:21.601 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:32:21.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:21.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:32:21.707 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:32:21.738 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:21.763 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:32:21.764 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:21 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:32:21.764 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:21 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:32:21.764 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:21 smithi103 ceph-mon[39629]: pgmap v1059: 27 pgs: 27 active+clean; 6.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 130 MiB/s wr, 2.07k op/s 2024-04-23T16:32:21.764 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:21 smithi103 ceph-mon[39629]: osdmap e377: 12 total, 12 up, 10 in 2024-04-23T16:32:21.764 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:21 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:32:21.764 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:21 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:32:21.764 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:21 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:32:21.764 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:21 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:32:21.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:21.767 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:32:21.867 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:32:21.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:21 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:32:21.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:21 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:32:21.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:21 smithi148 ceph-mon[44520]: pgmap v1059: 27 pgs: 27 active+clean; 6.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 130 MiB/s wr, 2.07k op/s 2024-04-23T16:32:21.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:21 smithi148 ceph-mon[44520]: osdmap e377: 12 total, 12 up, 10 in 2024-04-23T16:32:21.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:21 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:32:21.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:21 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:32:21.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:21 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:32:21.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:21 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:32:21.899 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:21.905 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:32:21.918 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:32:21.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:21.924 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:32:22.025 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:32:22.097 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:22.222 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:32:22.224 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:32:22.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:21 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:32:22.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:21 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:32:22.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:21 smithi188 ceph-mon[44101]: pgmap v1059: 27 pgs: 27 active+clean; 6.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 130 MiB/s wr, 2.07k op/s 2024-04-23T16:32:22.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:21 smithi188 ceph-mon[44101]: osdmap e377: 12 total, 12 up, 10 in 2024-04-23T16:32:22.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:21 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:32:22.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:21 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:32:22.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:21 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:32:22.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:21 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:32:22.275 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 43 16 107749 107733 156.576 93.6875 0.00820244 0.00638539 2024-04-23T16:32:22.766 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:32:22.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:22.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:32:22.876 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:32:22.904 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:22.929 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:32:22.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:22.933 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:32:23.034 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:23.089 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:23.276 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 44 15 109338 109323 155.276 99.375 0.00755052 0.00643857 2024-04-23T16:32:23.580 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:32:23.780 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:32:24.090 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:23 smithi103 ceph-mon[39629]: from='client.15819 v1:172.21.15.103:0/2809592834' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:32:24.090 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:23 smithi103 ceph-mon[39629]: pgmap v1061: 27 pgs: 27 active+clean; 6.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-23T16:32:24.090 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:23 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1686427396' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:32:24.125 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1060,"stamp":"2024-04-23T16:32:20.762368+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6693323296,"num_objects":102125,"num_object_clones":0,"num_object_copies":204250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102123,"num_write_kb":6535872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103652,"ondisk_log_size":103652,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":16112060,"kb_used_data":12979304,"kb_used_omap":15,"kb_used_meta":3132656,"kb_avail":921339460,"statfs":{"total":959950356480,"available":943451607040,"internally_reserved":0,"allocated":13290807296,"data_stored":13285941714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15948,"internal_metadata":3207840180},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1105264640,"num_objects":16865,"num_object_clones":0,"num_object_copies":33730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16865,"num_write_kb":1079360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.069095"},"pg_stats":[{"pgid":"5.f","version":"376'6405","reported_seq":6447,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:18.333588+0000","last_change":"2024-04-23T16:31:29.141207+0000","last_active":"2024-04-23T16:32:18.333588+0000","last_peered":"2024-04-23T16:32:18.333588+0000","last_clean":"2024-04-23T16:32:18.333588+0000","last_became_active":"2024-04-23T16:31:29.140923+0000","last_became_peered":"2024-04-23T16:31:29.140923+0000","last_unstale":"2024-04-23T16:32:18.333588+0000","last_undegraded":"2024-04-23T16:32:18.333588+0000","last_fullsized":"2024-04-23T16:32:18.333588+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":6405,"log_dups_size":0,"ondisk_log_size":6405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:45.436434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419758080,"num_objects":6405,"num_object_clones":0,"num_object_copies":12810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6405,"num_write_kb":409920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"376'6462","reported_seq":6505,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:18.951723+0000","last_change":"2024-04-23T16:31:29.138155+0000","last_active":"2024-04-23T16:32:18.951723+0000","last_peered":"2024-04-23T16:32:18.951723+0000","last_clean":"2024-04-23T16:32:18.951723+0000","last_became_active":"2024-04-23T16:31:29.137890+0000","last_became_peered":"2024-04-23T16:31:29.137890+0000","last_unstale":"2024-04-23T16:32:18.951723+0000","last_undegraded":"2024-04-23T16:32:18.951723+0000","last_fullsized":"2024-04-23T16:32:18.951723+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":6462,"log_dups_size":0,"ondisk_log_size":6462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:28.628097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423493632,"num_objects":6462,"num_object_clones":0,"num_object_copies":12924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6462,"num_write_kb":413568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"376'6488","reported_seq":6531,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:18.867950+0000","last_change":"2024-04-23T16:31:29.311079+0000","last_active":"2024-04-23T16:32:18.867950+0000","last_peered":"2024-04-23T16:32:18.867950+0000","last_clean":"2024-04-23T16:32:18.867950+0000","last_became_active":"2024-04-23T16:31:29.310887+0000","last_became_peered":"2024-04-23T16:31:29.310887+0000","last_unstale":"2024-04-23T16:32:18.867950+0000","last_undegraded":"2024-04-23T16:32:18.867950+0000","last_fullsized":"2024-04-23T16:32:18.867950+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":6488,"log_dups_size":0,"ondisk_log_size":6488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:38.952306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425197568,"num_objects":6488,"num_object_clones":0,"num_object_copies":12976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6488,"num_write_kb":415232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"376'6559","reported_seq":6601,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:19.963313+0000","last_change":"2024-04-23T16:31:29.141851+0000","last_active":"2024-04-23T16:32:19.963313+0000","last_peered":"2024-04-23T16:32:19.963313+0000","last_clean":"2024-04-23T16:32:19.963313+0000","last_became_active":"2024-04-23T16:31:29.141551+0000","last_became_peered":"2024-04-23T16:31:29.141551+0000","last_unstale":"2024-04-23T16:32:19.963313+0000","last_undegraded":"2024-04-23T16:32:19.963313+0000","last_fullsized":"2024-04-23T16:32:19.963313+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":6559,"log_dups_size":0,"ondisk_log_size":6559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:32.122461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429850624,"num_objects":6559,"num_object_clones":0,"num_object_copies":13118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6559,"num_write_kb":419776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"376'6016","reported_seq":6059,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:16.722220+0000","last_change":"2024-04-23T16:31:29.139319+0000","last_active":"2024-04-23T16:32:16.722220+0000","last_peered":"2024-04-23T16:32:16.722220+0000","last_clean":"2024-04-23T16:32:16.722220+0000","last_became_active":"2024-04-23T16:31:29.139045+0000","last_became_peered":"2024-04-23T16:31:29.139045+0000","last_unstale":"2024-04-23T16:32:16.722220+0000","last_undegraded":"2024-04-23T16:32:16.722220+0000","last_fullsized":"2024-04-23T16:32:16.722220+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":6016,"log_dups_size":0,"ondisk_log_size":6016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:51.133750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394264576,"num_objects":6016,"num_object_clones":0,"num_object_copies":12032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6016,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"376'6447","reported_seq":6490,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:18.966053+0000","last_change":"2024-04-23T16:31:29.311223+0000","last_active":"2024-04-23T16:32:18.966053+0000","last_peered":"2024-04-23T16:32:18.966053+0000","last_clean":"2024-04-23T16:32:18.966053+0000","last_became_active":"2024-04-23T16:31:29.310906+0000","last_became_peered":"2024-04-23T16:31:29.310906+0000","last_unstale":"2024-04-23T16:32:18.966053+0000","last_undegraded":"2024-04-23T16:32:18.966053+0000","last_fullsized":"2024-04-23T16:32:18.966053+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":6447,"log_dups_size":0,"ondisk_log_size":6447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:50.356210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422510592,"num_objects":6447,"num_object_clones":0,"num_object_copies":12894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":412608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":996,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:14.122591+0000","last_change":"2024-04-23T16:31:54.712643+0000","last_active":"2024-04-23T16:32:14.122591+0000","last_peered":"2024-04-23T16:32:14.122591+0000","last_clean":"2024-04-23T16:32:14.122591+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:32:14.122591+0000","last_undegraded":"2024-04-23T16:32:14.122591+0000","last_fullsized":"2024-04-23T16:32:14.122591+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:54.712524+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:54.712524+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:21.984104+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"376'6517","reported_seq":6559,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:19.145988+0000","last_change":"2024-04-23T16:31:29.141936+0000","last_active":"2024-04-23T16:32:19.145988+0000","last_peered":"2024-04-23T16:32:19.145988+0000","last_clean":"2024-04-23T16:32:19.145988+0000","last_became_active":"2024-04-23T16:31:29.141640+0000","last_became_peered":"2024-04-23T16:31:29.141640+0000","last_unstale":"2024-04-23T16:32:19.145988+0000","last_undegraded":"2024-04-23T16:32:19.145988+0000","last_fullsized":"2024-04-23T16:32:19.145988+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":6517,"log_dups_size":0,"ondisk_log_size":6517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:35.266902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427098112,"num_objects":6517,"num_object_clones":0,"num_object_copies":13034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6517,"num_write_kb":417088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":995,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:14.123786+0000","last_change":"2024-04-23T16:31:45.498938+0000","last_active":"2024-04-23T16:32:14.123786+0000","last_peered":"2024-04-23T16:32:14.123786+0000","last_clean":"2024-04-23T16:32:14.123786+0000","last_became_active":"2024-04-23T16:28:25.050007+0000","last_became_peered":"2024-04-23T16:28:25.050007+0000","last_unstale":"2024-04-23T16:32:14.123786+0000","last_undegraded":"2024-04-23T16:32:14.123786+0000","last_fullsized":"2024-04-23T16:32:14.123786+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.498820+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.498820+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:51.991697+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"376'6075","reported_seq":6118,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:16.715939+0000","last_change":"2024-04-23T16:31:29.140476+0000","last_active":"2024-04-23T16:32:16.715939+0000","last_peered":"2024-04-23T16:32:16.715939+0000","last_clean":"2024-04-23T16:32:16.715939+0000","last_became_active":"2024-04-23T16:31:29.140251+0000","last_became_peered":"2024-04-23T16:31:29.140251+0000","last_unstale":"2024-04-23T16:32:16.715939+0000","last_undegraded":"2024-04-23T16:32:16.715939+0000","last_fullsized":"2024-04-23T16:32:16.715939+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":6075,"log_dups_size":0,"ondisk_log_size":6075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:52.891704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398131200,"num_objects":6075,"num_object_clones":0,"num_object_copies":12150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6075,"num_write_kb":388800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":989,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:14.130787+0000","last_change":"2024-04-23T16:30:48.312533+0000","last_active":"2024-04-23T16:32:14.130787+0000","last_peered":"2024-04-23T16:32:14.130787+0000","last_clean":"2024-04-23T16:32:14.130787+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:32:14.130787+0000","last_undegraded":"2024-04-23T16:32:14.130787+0000","last_fullsized":"2024-04-23T16:32:14.130787+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:30:48.312402+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:30:48.312402+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:17.394541+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"376'6479","reported_seq":6521,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:18.887370+0000","last_change":"2024-04-23T16:31:29.140069+0000","last_active":"2024-04-23T16:32:18.887370+0000","last_peered":"2024-04-23T16:32:18.887370+0000","last_clean":"2024-04-23T16:32:18.887370+0000","last_became_active":"2024-04-23T16:31:29.139806+0000","last_became_peered":"2024-04-23T16:31:29.139806+0000","last_unstale":"2024-04-23T16:32:18.887370+0000","last_undegraded":"2024-04-23T16:32:18.887370+0000","last_fullsized":"2024-04-23T16:32:18.887370+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":6479,"log_dups_size":0,"ondisk_log_size":6479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:38.160135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424607744,"num_objects":6479,"num_object_clones":0,"num_object_copies":12958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6479,"num_write_kb":414656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":987,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:14.125485+0000","last_change":"2024-04-23T16:30:47.896287+0000","last_active":"2024-04-23T16:32:14.125485+0000","last_peered":"2024-04-23T16:32:14.125485+0000","last_clean":"2024-04-23T16:32:14.125485+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:32:14.125485+0000","last_undegraded":"2024-04-23T16:32:14.125485+0000","last_fullsized":"2024-04-23T16:32:14.125485+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:30:47.896137+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:30:47.896137+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:17.822102+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"376'6212","reported_seq":6254,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:18.334401+0000","last_change":"2024-04-23T16:31:29.140042+0000","last_active":"2024-04-23T16:32:18.334401+0000","last_peered":"2024-04-23T16:32:18.334401+0000","last_clean":"2024-04-23T16:32:18.334401+0000","last_became_active":"2024-04-23T16:31:29.139709+0000","last_became_peered":"2024-04-23T16:31:29.139709+0000","last_unstale":"2024-04-23T16:32:18.334401+0000","last_undegraded":"2024-04-23T16:32:18.334401+0000","last_fullsized":"2024-04-23T16:32:18.334401+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":6212,"log_dups_size":0,"ondisk_log_size":6212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:51.163966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407109632,"num_objects":6212,"num_object_clones":0,"num_object_copies":12424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6212,"num_write_kb":397568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":988,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:14.127511+0000","last_change":"2024-04-23T16:30:47.512248+0000","last_active":"2024-04-23T16:32:14.127511+0000","last_peered":"2024-04-23T16:32:14.127511+0000","last_clean":"2024-04-23T16:32:14.127511+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:32:14.127511+0000","last_undegraded":"2024-04-23T16:32:14.127511+0000","last_fullsized":"2024-04-23T16:32:14.127511+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:30:47.512120+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:30:47.512120+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:16.516058+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"376'6475","reported_seq":6518,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:18.948177+0000","last_change":"2024-04-23T16:31:29.139868+0000","last_active":"2024-04-23T16:32:18.948177+0000","last_peered":"2024-04-23T16:32:18.948177+0000","last_clean":"2024-04-23T16:32:18.948177+0000","last_became_active":"2024-04-23T16:31:29.139606+0000","last_became_peered":"2024-04-23T16:31:29.139606+0000","last_unstale":"2024-04-23T16:32:18.948177+0000","last_undegraded":"2024-04-23T16:32:18.948177+0000","last_fullsized":"2024-04-23T16:32:18.948177+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":6475,"log_dups_size":0,"ondisk_log_size":6475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:51.545504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424345600,"num_objects":6475,"num_object_clones":0,"num_object_copies":12950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6475,"num_write_kb":414400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":995,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:14.122594+0000","last_change":"2024-04-23T16:31:45.883868+0000","last_active":"2024-04-23T16:32:14.122594+0000","last_peered":"2024-04-23T16:32:14.122594+0000","last_clean":"2024-04-23T16:32:14.122594+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:32:14.122594+0000","last_undegraded":"2024-04-23T16:32:14.122594+0000","last_fullsized":"2024-04-23T16:32:14.122594+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.883736+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:07.109005+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.2","version":"376'6402","reported_seq":6445,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:19.407954+0000","last_change":"2024-04-23T16:31:29.138797+0000","last_active":"2024-04-23T16:32:19.407954+0000","last_peered":"2024-04-23T16:32:19.407954+0000","last_clean":"2024-04-23T16:32:19.407954+0000","last_became_active":"2024-04-23T16:31:29.138430+0000","last_became_peered":"2024-04-23T16:31:29.138430+0000","last_unstale":"2024-04-23T16:32:19.407954+0000","last_undegraded":"2024-04-23T16:32:19.407954+0000","last_fullsized":"2024-04-23T16:32:19.407954+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":6402,"log_dups_size":0,"ondisk_log_size":6402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:31.211006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419561472,"num_objects":6402,"num_object_clones":0,"num_object_copies":12804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6402,"num_write_kb":409728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":996,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:14.122740+0000","last_change":"2024-04-23T16:31:26.858471+0000","last_active":"2024-04-23T16:32:14.122740+0000","last_peered":"2024-04-23T16:32:14.122740+0000","last_clean":"2024-04-23T16:32:14.122740+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:32:14.122740+0000","last_undegraded":"2024-04-23T16:32:14.122740+0000","last_fullsized":"2024-04-23T16:32:14.122740+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:26.858333+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:26.858333+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:43.789314+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"376'6344","reported_seq":6386,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:18.957343+0000","last_change":"2024-04-23T16:31:29.139641+0000","last_active":"2024-04-23T16:32:18.957343+0000","last_peered":"2024-04-23T16:32:18.957343+0000","last_clean":"2024-04-23T16:32:18.957343+0000","last_became_active":"2024-04-23T16:31:29.139398+0000","last_became_peered":"2024-04-23T16:31:29.139398+0000","last_unstale":"2024-04-23T16:32:18.957343+0000","last_undegraded":"2024-04-23T16:32:18.957343+0000","last_fullsized":"2024-04-23T16:32:18.957343+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":6344,"log_dups_size":0,"ondisk_log_size":6344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:41.180907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415760384,"num_objects":6344,"num_object_clones":0,"num_object_copies":12688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6344,"num_write_kb":406016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":988,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:14.127512+0000","last_change":"2024-04-23T16:30:48.463114+0000","last_active":"2024-04-23T16:32:14.127512+0000","last_peered":"2024-04-23T16:32:14.127512+0000","last_clean":"2024-04-23T16:32:14.127512+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:32:14.127512+0000","last_undegraded":"2024-04-23T16:32:14.127512+0000","last_fullsized":"2024-04-23T16:32:14.127512+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:30:48.463036+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:30:48.463036+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:16.516021+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"376'6345","reported_seq":6386,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:19.158884+0000","last_change":"2024-04-23T16:31:29.139304+0000","last_active":"2024-04-23T16:32:19.158884+0000","last_peered":"2024-04-23T16:32:19.158884+0000","last_clean":"2024-04-23T16:32:19.158884+0000","last_became_active":"2024-04-23T16:31:29.138968+0000","last_became_peered":"2024-04-23T16:31:29.138968+0000","last_unstale":"2024-04-23T16:32:19.158884+0000","last_undegraded":"2024-04-23T16:32:19.158884+0000","last_fullsized":"2024-04-23T16:32:19.158884+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":6345,"log_dups_size":0,"ondisk_log_size":6345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:41.287078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415825920,"num_objects":6345,"num_object_clones":0,"num_object_copies":12690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6345,"num_write_kb":406080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":887,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:14.122665+0000","last_change":"2024-04-23T16:31:30.064468+0000","last_active":"2024-04-23T16:32:14.122665+0000","last_peered":"2024-04-23T16:32:14.122665+0000","last_clean":"2024-04-23T16:32:14.122665+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:32:14.122665+0000","last_undegraded":"2024-04-23T16:32:14.122665+0000","last_fullsized":"2024-04-23T16:32:14.122665+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:30.064354+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:30.064354+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:49.450707+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"376'6496","reported_seq":6537,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:19.160800+0000","last_change":"2024-04-23T16:31:29.310501+0000","last_active":"2024-04-23T16:32:19.160800+0000","last_peered":"2024-04-23T16:32:19.160800+0000","last_clean":"2024-04-23T16:32:19.160800+0000","last_became_active":"2024-04-23T16:31:29.310246+0000","last_became_peered":"2024-04-23T16:31:29.310246+0000","last_unstale":"2024-04-23T16:32:19.160800+0000","last_undegraded":"2024-04-23T16:32:19.160800+0000","last_fullsized":"2024-04-23T16:32:19.160800+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":6496,"log_dups_size":0,"ondisk_log_size":6496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:36.052300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425721856,"num_objects":6496,"num_object_clones":0,"num_object_copies":12992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6496,"num_write_kb":415744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":995,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:14.124138+0000","last_change":"2024-04-23T16:31:51.684967+0000","last_active":"2024-04-23T16:32:14.124138+0000","last_peered":"2024-04-23T16:32:14.124138+0000","last_clean":"2024-04-23T16:32:14.124138+0000","last_became_active":"2024-04-23T16:28:25.050105+0000","last_became_peered":"2024-04-23T16:28:25.050105+0000","last_unstale":"2024-04-23T16:32:14.124138+0000","last_undegraded":"2024-04-23T16:32:14.124138+0000","last_fullsized":"2024-04-23T16:32:14.124138+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:31:51.684868+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:09.608871+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.4","version":"376'6401","reported_seq":6444,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:18.874466+0000","last_change":"2024-04-23T16:31:29.139993+0000","last_active":"2024-04-23T16:32:18.874466+0000","last_peered":"2024-04-23T16:32:18.874466+0000","last_clean":"2024-04-23T16:32:18.874466+0000","last_became_active":"2024-04-23T16:31:29.139762+0000","last_became_peered":"2024-04-23T16:31:29.139762+0000","last_unstale":"2024-04-23T16:32:18.874466+0000","last_undegraded":"2024-04-23T16:32:18.874466+0000","last_fullsized":"2024-04-23T16:32:18.874466+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":6401,"log_dups_size":0,"ondisk_log_size":6401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:34.629312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419495936,"num_objects":6401,"num_object_clones":0,"num_object_copies":12802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6401,"num_write_kb":409664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":994,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-04-23T16:32:14.130601+0000","last_change":"2024-04-23T16:31:54.636517+0000","last_active":"2024-04-23T16:32:14.130601+0000","last_peered":"2024-04-23T16:32:14.130601+0000","last_clean":"2024-04-23T16:32:14.130601+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:32:14.130601+0000","last_undegraded":"2024-04-23T16:32:14.130601+0000","last_fullsized":"2024-04-23T16:32:14.130601+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:31:54.636376+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:54.835653+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":6692732928,"num_objects":102123,"num_object_clones":0,"num_object_copies":204246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102123,"num_write_kb":6535872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13406502912,"data_stored":13406502912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102123,"ondisk_log_size":102123,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056844,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1240584,"kb_used_data":763772,"kb_used_omap":1,"kb_used_meta":476798,"kb_avail":92504568,"statfs":{"total":95995035648,"available":94724677632,"internally_reserved":0,"allocated":782102528,"data_stored":781616673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":488241604},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351186,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612824,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1521156,"kb_used_data":1219260,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92223996,"statfs":{"total":95995035648,"available":94437371904,"internally_reserved":0,"allocated":1248522240,"data_stored":1248036393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":309131712},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841755,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":969732,"kb_used_data":808060,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":92775420,"statfs":{"total":95995035648,"available":95002030080,"internally_reserved":0,"allocated":827453440,"data_stored":826967593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168801,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":742724,"kb_used_data":418620,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93002428,"statfs":{"total":95995035648,"available":95234486272,"internally_reserved":0,"allocated":428666880,"data_stored":428181033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":331872714},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463143,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1429956,"kb_used_data":1218300,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92315196,"statfs":{"total":95995035648,"available":94530760704,"internally_reserved":0,"allocated":1247539200,"data_stored":1247053353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":216725958},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137282,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738628,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3145924,"kb_used_data":2817660,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":90599228,"statfs":{"total":95995035648,"available":92773609472,"internally_reserved":0,"allocated":2885283840,"data_stored":2884797993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313465,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1439052,"kb_used_data":1232964,"kb_used_omap":1,"kb_used_meta":206078,"kb_avail":92306100,"statfs":{"total":95995035648,"available":94521446400,"internally_reserved":0,"allocated":1262555136,"data_stored":1262065737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":211024325},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117237,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":787972,"kb_used_data":427772,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92957180,"statfs":{"total":95995035648,"available":95188152320,"internally_reserved":0,"allocated":438038528,"data_stored":437552681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":368835015},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921007,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2454788,"kb_used_data":2048764,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":91290364,"statfs":{"total":95995035648,"available":93481332736,"internally_reserved":0,"allocated":2097934336,"data_stored":2097448489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757482,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2380172,"kb_used_data":2024132,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":91364980,"statfs":{"total":95995035648,"available":93557739520,"internally_reserved":0,"allocated":2072711168,"data_stored":2072221769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2930114560,"data_stored":2930114560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1269301248,"data_stored":1269301248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":437190656,"data_stored":437190656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2119106560,"data_stored":2119106560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2075000832,"data_stored":2075000832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1265500160,"data_stored":1265500160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":429850624,"data_stored":429850624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":831193088,"data_stored":831193088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1256849408,"data_stored":1256849408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":792395776,"data_stored":792395776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:32:24.127 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:32:24.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:23 smithi148 ceph-mon[44520]: from='client.15819 v1:172.21.15.103:0/2809592834' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:32:24.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:23 smithi148 ceph-mon[44520]: pgmap v1061: 27 pgs: 27 active+clean; 6.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-23T16:32:24.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:23 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1686427396' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:32:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:23 smithi188 ceph-mon[44101]: from='client.15819 v1:172.21.15.103:0/2809592834' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:32:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:23 smithi188 ceph-mon[44101]: pgmap v1061: 27 pgs: 27 active+clean; 6.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-23T16:32:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:23 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1686427396' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:32:24.276 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 45 16 110972 110956 154.093 102.062 0.00914021 0.0064884 2024-04-23T16:32:24.349 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:24.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:24.456 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:32:24.487 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:24.511 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:32:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:24.516 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:32:24.618 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:32:24.659 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:24.699 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:32:24.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:24.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:32:24.767 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:24 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:32:24.768 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1686427396' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:32:24.768 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:24 smithi188 ceph-mon[44101]: osdmap e378: 12 total, 12 up, 10 in 2024-04-23T16:32:24.805 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:32:24.836 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:24.858 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:32:24.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:24.865 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:32:24.966 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:32:25.018 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:24 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:32:25.019 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:24 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1686427396' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:32:25.019 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:24 smithi148 ceph-mon[44520]: osdmap e378: 12 total, 12 up, 10 in 2024-04-23T16:32:25.019 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:25.054 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:32:25.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:25.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:32:25.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:24 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:32:25.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1686427396' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:32:25.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:24 smithi103 ceph-mon[39629]: osdmap e378: 12 total, 12 up, 10 in 2024-04-23T16:32:25.163 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:32:25.211 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:25.240 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:32:25.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:25.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:32:25.276 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 46 16 112449 112433 152.75 92.3125 0.00386311 0.0065411 2024-04-23T16:32:25.343 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:32:25.394 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:25.424 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:32:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:25.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:32:25.530 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:32:25.593 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:25.634 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:32:25.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:25.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:32:25.740 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:32:25.771 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:25.796 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:32:25.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:25.799 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:32:25.900 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:32:25.954 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:26.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:25 smithi103 ceph-mon[39629]: pgmap v1063: 27 pgs: 27 active+clean; 6.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-04-23T16:32:26.052 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:25 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:32:26.052 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:25 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:32:26.060 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:32:26.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:26.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:32:26.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:25 smithi148 ceph-mon[44520]: pgmap v1063: 27 pgs: 27 active+clean; 6.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-04-23T16:32:26.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:25 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:32:26.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:25 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:32:26.171 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:32:26.223 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:25 smithi188 ceph-mon[44101]: pgmap v1063: 27 pgs: 27 active+clean; 6.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-04-23T16:32:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:25 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:32:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:25 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:32:26.265 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:32:26.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:26.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:32:26.275 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 47 16 113918 113902 151.453 91.8125 0.00508771 0.00660097 2024-04-23T16:32:26.369 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:32:26.402 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:26.485 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:32:26.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:26.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:32:26.593 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:32:26.627 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:26.652 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:32:26.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:26.656 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:32:26.726 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:32:26.756 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:32:26.786 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:26.793 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:32:26.798 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:26 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:32:26.798 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:26 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:32:26.798 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:26 smithi103 ceph-mon[39629]: osdmap e379: 12 total, 12 up, 10 in 2024-04-23T16:32:26.798 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:26 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4094529468' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:32:26.809 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:32:26.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:26.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:32:26.825 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:32:26.854 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:32:26.883 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:32:26.912 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:32:26.916 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:32:26.942 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:26 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:32:26.942 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:26 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:32:26.942 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:26 smithi188 ceph-mon[44101]: osdmap e379: 12 total, 12 up, 10 in 2024-04-23T16:32:26.942 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:26 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4094529468' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:32:26.944 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:26.945 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:32:26.969 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:32:26.970 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:32:26.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:26.974 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:32:27.010 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:32:27.044 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:32:27.068 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:32:27.075 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:32:27.098 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:32:27.121 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:27.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:26 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:32:27.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:26 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:32:27.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:26 smithi148 ceph-mon[44520]: osdmap e379: 12 total, 12 up, 10 in 2024-04-23T16:32:27.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:26 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4094529468' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:32:27.142 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:32:27.158 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:32:27.158 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:32:27.158 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:32:27.158 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:32:27.158 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:32:27.159 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:32:27.159 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:32:27.159 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:32:27.159 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:32:27.159 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:32:27.159 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:32:27.159 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:32:27.276 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 48 16 115437 115421 150.276 94.9375 0.00402803 0.00665276 2024-04-23T16:32:27.809 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:27 smithi103 ceph-mon[39629]: pgmap v1065: 27 pgs: 27 active+clean; 6.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-23T16:32:27.809 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:27.809 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:27.809 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:27.809 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:27.809 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:27.809 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:27.810 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:27.810 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:27.810 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:27.810 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:27.810 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:27 smithi148 ceph-mon[44520]: pgmap v1065: 27 pgs: 27 active+clean; 6.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-23T16:32:28.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:27 smithi188 ceph-mon[44101]: pgmap v1065: 27 pgs: 27 active+clean; 6.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-23T16:32:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.276 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 49 16 117031 117015 149.242 99.625 0.00476892 0.00669924 2024-04-23T16:32:28.291 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:32:28.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:28.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:32:28.399 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:28.442 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:28.515 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:32:28.812 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:28.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:28.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:28.816 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:28.816 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:28 smithi103 ceph-mon[39629]: 1.7 scrub starts 2024-04-23T16:32:28.816 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:28 smithi103 ceph-mon[39629]: 1.7 scrub ok 2024-04-23T16:32:28.916 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:32:28.945 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:28.972 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:32:28.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:28.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:32:29.078 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:32:29.109 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:29.109 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:28 smithi188 ceph-mon[44101]: 1.7 scrub starts 2024-04-23T16:32:29.110 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:28 smithi188 ceph-mon[44101]: 1.7 scrub ok 2024-04-23T16:32:29.111 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:29.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:29.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:28 smithi148 ceph-mon[44520]: 1.7 scrub starts 2024-04-23T16:32:29.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:28 smithi148 ceph-mon[44520]: 1.7 scrub ok 2024-04-23T16:32:29.134 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:32:29.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:29.138 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:32:29.239 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:29.276 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 50 16 118536 118520 148.138 94.0625 0.00473127 0.0067489 2024-04-23T16:32:29.276 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:29.323 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:29.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:29.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:29.428 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:32:29.490 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:29.507 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:32:29.507 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:32:29.507 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-23T16:32:29.508 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-23T16:32:29.508 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:32:29.518 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:32:29.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:29.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:32:29.624 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:32:29.661 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:29.686 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:32:29.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:29.689 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:32:29.790 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:32:29.817 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:29 smithi103 ceph-mon[39629]: pgmap v1066: 27 pgs: 27 active+clean; 6.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-23T16:32:29.817 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:29 smithi103 ceph-mon[39629]: osdmap e380: 12 total, 12 up, 10 in 2024-04-23T16:32:29.822 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:29.823 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:29 smithi148 ceph-mon[44520]: pgmap v1066: 27 pgs: 27 active+clean; 6.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-23T16:32:29.823 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:29 smithi148 ceph-mon[44520]: osdmap e380: 12 total, 12 up, 10 in 2024-04-23T16:32:29.850 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:32:29.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:29.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:32:29.954 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:32:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:29 smithi188 ceph-mon[44101]: pgmap v1066: 27 pgs: 27 active+clean; 6.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-23T16:32:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:29 smithi188 ceph-mon[44101]: osdmap e380: 12 total, 12 up, 10 in 2024-04-23T16:32:30.007 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:30.033 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:32:30.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:30.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:32:30.150 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:32:30.195 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:30.276 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 51 16 119980 119964 147.003 90.25 0.00510079 0.00680131 2024-04-23T16:32:30.305 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:32:30.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:30.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:32:30.331 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:32:30.416 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:32:30.462 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:30.490 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:32:30.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:30.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:32:30.598 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:32:30.635 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:30.673 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:32:30.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:30.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:32:30.781 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:32:30.819 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:30.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:30 smithi103 ceph-mon[39629]: 1.9 scrub starts 2024-04-23T16:32:30.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:30 smithi103 ceph-mon[39629]: 5.b scrub starts 2024-04-23T16:32:30.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:30 smithi103 ceph-mon[39629]: 1.9 scrub ok 2024-04-23T16:32:30.849 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:32:30.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:30.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:32:30.957 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:32:30.987 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:30.998 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:32:31.011 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:32:31.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:31.014 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:32:31.115 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:32:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:30 smithi148 ceph-mon[44520]: 1.9 scrub starts 2024-04-23T16:32:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:30 smithi148 ceph-mon[44520]: 5.b scrub starts 2024-04-23T16:32:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:30 smithi148 ceph-mon[44520]: 1.9 scrub ok 2024-04-23T16:32:31.178 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:31.219 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:32:31.223 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:32:31.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:31.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:30 smithi188 ceph-mon[44101]: 1.9 scrub starts 2024-04-23T16:32:31.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:30 smithi188 ceph-mon[44101]: 5.b scrub starts 2024-04-23T16:32:31.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:30 smithi188 ceph-mon[44101]: 1.9 scrub ok 2024-04-23T16:32:31.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:32:31.227 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:32:31.276 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 52 16 121393 121377 145.874 88.3125 0.00353319 0.00685347 2024-04-23T16:32:31.327 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:32:31.366 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:31.399 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:32:31.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:31.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:32:31.504 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:32:31.516 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-23T16:32:31.545 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:31.760 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:32:31.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:31.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:32:31.826 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:31 smithi103 ceph-mon[39629]: 5.b scrub ok 2024-04-23T16:32:31.827 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:31 smithi103 ceph-mon[39629]: pgmap v1068: 27 pgs: 27 active+clean; 7.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 123 MiB/s wr, 1.97k op/s 2024-04-23T16:32:31.867 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:31.913 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:32.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:31 smithi148 ceph-mon[44520]: 5.b scrub ok 2024-04-23T16:32:32.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:31 smithi148 ceph-mon[44520]: pgmap v1068: 27 pgs: 27 active+clean; 7.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 123 MiB/s wr, 1.97k op/s 2024-04-23T16:32:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:31 smithi188 ceph-mon[44101]: 5.b scrub ok 2024-04-23T16:32:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:31 smithi188 ceph-mon[44101]: pgmap v1068: 27 pgs: 27 active+clean; 7.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 123 MiB/s wr, 1.97k op/s 2024-04-23T16:32:32.276 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 53 16 122884 122868 144.88 93.1875 0.0135036 0.00690085 2024-04-23T16:32:32.625 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1068,"stamp":"2024-04-23T16:32:30.697196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7714046496,"num_objects":117700,"num_object_clones":0,"num_object_copies":235400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117698,"num_write_kb":7532672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119227,"ondisk_log_size":119227,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":18127292,"kb_used_data":14930652,"kb_used_omap":15,"kb_used_meta":3196528,"kb_avail":919324228,"statfs":{"total":959950356480,"available":941388009472,"internally_reserved":0,"allocated":15288987648,"data_stored":15283935342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15948,"internal_metadata":3273245108},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,3,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":891944960,"num_objects":13610,"num_object_clones":0,"num_object_copies":27220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13610,"num_write_kb":871040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.921542"},"pg_stats":[{"pgid":"5.f","version":"379'7405","reported_seq":7452,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-23T16:32:28.371895+0000","last_change":"2024-04-23T16:31:29.141207+0000","last_active":"2024-04-23T16:32:28.371895+0000","last_peered":"2024-04-23T16:32:28.371895+0000","last_clean":"2024-04-23T16:32:28.371895+0000","last_became_active":"2024-04-23T16:31:29.140923+0000","last_became_peered":"2024-04-23T16:31:29.140923+0000","last_unstale":"2024-04-23T16:32:28.371895+0000","last_undegraded":"2024-04-23T16:32:28.371895+0000","last_fullsized":"2024-04-23T16:32:28.371895+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":7405,"log_dups_size":0,"ondisk_log_size":7405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:45.436434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485294080,"num_objects":7405,"num_object_clones":0,"num_object_copies":14810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7405,"num_write_kb":473920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"379'7439","reported_seq":7488,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-23T16:32:28.968197+0000","last_change":"2024-04-23T16:31:29.138155+0000","last_active":"2024-04-23T16:32:28.968197+0000","last_peered":"2024-04-23T16:32:28.968197+0000","last_clean":"2024-04-23T16:32:28.968197+0000","last_became_active":"2024-04-23T16:31:29.137890+0000","last_became_peered":"2024-04-23T16:31:29.137890+0000","last_unstale":"2024-04-23T16:32:28.968197+0000","last_undegraded":"2024-04-23T16:32:28.968197+0000","last_fullsized":"2024-04-23T16:32:28.968197+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":7439,"log_dups_size":0,"ondisk_log_size":7439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487522304,"num_objects":7439,"num_object_clones":0,"num_object_copies":14878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7439,"num_write_kb":476096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"379'7483","reported_seq":7532,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-23T16:32:28.890379+0000","last_change":"2024-04-23T16:31:29.311079+0000","last_active":"2024-04-23T16:32:28.890379+0000","last_peered":"2024-04-23T16:32:28.890379+0000","last_clean":"2024-04-23T16:32:28.890379+0000","last_became_active":"2024-04-23T16:31:29.310887+0000","last_became_peered":"2024-04-23T16:31:29.310887+0000","last_unstale":"2024-04-23T16:32:28.890379+0000","last_undegraded":"2024-04-23T16:32:28.890379+0000","last_fullsized":"2024-04-23T16:32:28.890379+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":7483,"log_dups_size":0,"ondisk_log_size":7483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:38.952306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490405888,"num_objects":7483,"num_object_clones":0,"num_object_copies":14966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7483,"num_write_kb":478912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"380'7525","reported_seq":7575,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-04-23T16:32:29.957621+0000","last_change":"2024-04-23T16:31:29.141851+0000","last_active":"2024-04-23T16:32:29.957621+0000","last_peered":"2024-04-23T16:32:29.957621+0000","last_clean":"2024-04-23T16:32:29.957621+0000","last_became_active":"2024-04-23T16:31:29.141551+0000","last_became_peered":"2024-04-23T16:31:29.141551+0000","last_unstale":"2024-04-23T16:32:29.957621+0000","last_undegraded":"2024-04-23T16:32:29.957621+0000","last_fullsized":"2024-04-23T16:32:29.957621+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":7525,"log_dups_size":0,"ondisk_log_size":7525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:32.122461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":493158400,"num_objects":7525,"num_object_clones":0,"num_object_copies":15050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7525,"num_write_kb":481600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"379'6959","reported_seq":7008,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-23T16:32:26.688962+0000","last_change":"2024-04-23T16:31:29.139319+0000","last_active":"2024-04-23T16:32:26.688962+0000","last_peered":"2024-04-23T16:32:26.688962+0000","last_clean":"2024-04-23T16:32:26.688962+0000","last_became_active":"2024-04-23T16:31:29.139045+0000","last_became_peered":"2024-04-23T16:31:29.139045+0000","last_unstale":"2024-04-23T16:32:26.688962+0000","last_undegraded":"2024-04-23T16:32:26.688962+0000","last_fullsized":"2024-04-23T16:32:26.688962+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":6959,"log_dups_size":0,"ondisk_log_size":6959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:51.133750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456065024,"num_objects":6959,"num_object_clones":0,"num_object_copies":13918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6959,"num_write_kb":445376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"379'7427","reported_seq":7476,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-23T16:32:28.986137+0000","last_change":"2024-04-23T16:31:29.311223+0000","last_active":"2024-04-23T16:32:28.986137+0000","last_peered":"2024-04-23T16:32:28.986137+0000","last_clean":"2024-04-23T16:32:28.986137+0000","last_became_active":"2024-04-23T16:31:29.310906+0000","last_became_peered":"2024-04-23T16:31:29.310906+0000","last_unstale":"2024-04-23T16:32:28.986137+0000","last_undegraded":"2024-04-23T16:32:28.986137+0000","last_fullsized":"2024-04-23T16:32:28.986137+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":7427,"log_dups_size":0,"ondisk_log_size":7427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:50.356210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486735872,"num_objects":7427,"num_object_clones":0,"num_object_copies":14854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7427,"num_write_kb":475328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1002,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-23T16:32:25.789205+0000","last_change":"2024-04-23T16:31:54.712643+0000","last_active":"2024-04-23T16:32:25.789205+0000","last_peered":"2024-04-23T16:32:25.789205+0000","last_clean":"2024-04-23T16:32:25.789205+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:32:25.789205+0000","last_undegraded":"2024-04-23T16:32:25.789205+0000","last_fullsized":"2024-04-23T16:32:25.789205+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:54.712524+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:54.712524+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:21.984104+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"380'7498","reported_seq":7548,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-04-23T16:32:29.162228+0000","last_change":"2024-04-23T16:31:29.141936+0000","last_active":"2024-04-23T16:32:29.162228+0000","last_peered":"2024-04-23T16:32:29.162228+0000","last_clean":"2024-04-23T16:32:29.162228+0000","last_became_active":"2024-04-23T16:31:29.141640+0000","last_became_peered":"2024-04-23T16:31:29.141640+0000","last_unstale":"2024-04-23T16:32:29.162228+0000","last_undegraded":"2024-04-23T16:32:29.162228+0000","last_fullsized":"2024-04-23T16:32:29.162228+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":7498,"log_dups_size":0,"ondisk_log_size":7498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:35.266902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491388928,"num_objects":7498,"num_object_clones":0,"num_object_copies":14996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7498,"num_write_kb":479872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1001,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-23T16:32:25.790904+0000","last_change":"2024-04-23T16:31:45.498938+0000","last_active":"2024-04-23T16:32:25.790904+0000","last_peered":"2024-04-23T16:32:25.790904+0000","last_clean":"2024-04-23T16:32:25.790904+0000","last_became_active":"2024-04-23T16:28:25.050007+0000","last_became_peered":"2024-04-23T16:28:25.050007+0000","last_unstale":"2024-04-23T16:32:25.790904+0000","last_undegraded":"2024-04-23T16:32:25.790904+0000","last_fullsized":"2024-04-23T16:32:25.790904+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.498820+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.498820+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:51.991697+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"379'7053","reported_seq":7102,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-23T16:32:26.719261+0000","last_change":"2024-04-23T16:31:29.140476+0000","last_active":"2024-04-23T16:32:26.719261+0000","last_peered":"2024-04-23T16:32:26.719261+0000","last_clean":"2024-04-23T16:32:26.719261+0000","last_became_active":"2024-04-23T16:31:29.140251+0000","last_became_peered":"2024-04-23T16:31:29.140251+0000","last_unstale":"2024-04-23T16:32:26.719261+0000","last_undegraded":"2024-04-23T16:32:26.719261+0000","last_fullsized":"2024-04-23T16:32:26.719261+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":7053,"log_dups_size":0,"ondisk_log_size":7053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:52.891704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462225408,"num_objects":7053,"num_object_clones":0,"num_object_copies":14106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7053,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1005,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-04-23T16:32:29.517904+0000","last_change":"2024-04-23T16:32:29.517947+0000","last_active":"2024-04-23T16:32:29.517904+0000","last_peered":"2024-04-23T16:32:29.517904+0000","last_clean":"2024-04-23T16:32:29.517904+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:32:29.517904+0000","last_undegraded":"2024-04-23T16:32:29.517904+0000","last_fullsized":"2024-04-23T16:32:29.517904+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:32:29.517800+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:32:29.517800+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:58.100582+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"379'7474","reported_seq":7523,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-23T16:32:28.881248+0000","last_change":"2024-04-23T16:31:29.140069+0000","last_active":"2024-04-23T16:32:28.881248+0000","last_peered":"2024-04-23T16:32:28.881248+0000","last_clean":"2024-04-23T16:32:28.881248+0000","last_became_active":"2024-04-23T16:31:29.139806+0000","last_became_peered":"2024-04-23T16:31:29.139806+0000","last_unstale":"2024-04-23T16:32:28.881248+0000","last_undegraded":"2024-04-23T16:32:28.881248+0000","last_fullsized":"2024-04-23T16:32:28.881248+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":7474,"log_dups_size":0,"ondisk_log_size":7474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:38.160135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489816064,"num_objects":7474,"num_object_clones":0,"num_object_copies":14948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7474,"num_write_kb":478336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1001,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-23T16:32:25.788431+0000","last_change":"2024-04-23T16:32:25.045027+0000","last_active":"2024-04-23T16:32:25.788431+0000","last_peered":"2024-04-23T16:32:25.788431+0000","last_clean":"2024-04-23T16:32:25.788431+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:32:25.788431+0000","last_undegraded":"2024-04-23T16:32:25.788431+0000","last_fullsized":"2024-04-23T16:32:25.788431+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:32:25.044929+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:32:25.044929+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:27.766158+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"379'7193","reported_seq":7241,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-23T16:32:28.387492+0000","last_change":"2024-04-23T16:31:29.140042+0000","last_active":"2024-04-23T16:32:28.387492+0000","last_peered":"2024-04-23T16:32:28.387492+0000","last_clean":"2024-04-23T16:32:28.387492+0000","last_became_active":"2024-04-23T16:31:29.139709+0000","last_became_peered":"2024-04-23T16:31:29.139709+0000","last_unstale":"2024-04-23T16:32:28.387492+0000","last_undegraded":"2024-04-23T16:32:28.387492+0000","last_fullsized":"2024-04-23T16:32:28.387492+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":7193,"log_dups_size":0,"ondisk_log_size":7193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:51.163966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471400448,"num_objects":7193,"num_object_clones":0,"num_object_copies":14386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7193,"num_write_kb":460352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":996,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-04-23T16:32:29.132010+0000","last_change":"2024-04-23T16:30:47.512248+0000","last_active":"2024-04-23T16:32:29.132010+0000","last_peered":"2024-04-23T16:32:29.132010+0000","last_clean":"2024-04-23T16:32:29.132010+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:32:29.132010+0000","last_undegraded":"2024-04-23T16:32:29.132010+0000","last_fullsized":"2024-04-23T16:32:29.132010+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:30:47.512120+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:30:47.512120+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"379'7439","reported_seq":7488,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-23T16:32:28.954957+0000","last_change":"2024-04-23T16:31:29.139868+0000","last_active":"2024-04-23T16:32:28.954957+0000","last_peered":"2024-04-23T16:32:28.954957+0000","last_clean":"2024-04-23T16:32:28.954957+0000","last_became_active":"2024-04-23T16:31:29.139606+0000","last_became_peered":"2024-04-23T16:31:29.139606+0000","last_unstale":"2024-04-23T16:32:28.954957+0000","last_undegraded":"2024-04-23T16:32:28.954957+0000","last_fullsized":"2024-04-23T16:32:28.954957+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":7439,"log_dups_size":0,"ondisk_log_size":7439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:51.545504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487522304,"num_objects":7439,"num_object_clones":0,"num_object_copies":14878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7439,"num_write_kb":476096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1001,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-23T16:32:25.789205+0000","last_change":"2024-04-23T16:31:45.883868+0000","last_active":"2024-04-23T16:32:25.789205+0000","last_peered":"2024-04-23T16:32:25.789205+0000","last_clean":"2024-04-23T16:32:25.789205+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:32:25.789205+0000","last_undegraded":"2024-04-23T16:32:25.789205+0000","last_fullsized":"2024-04-23T16:32:25.789205+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.883736+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:07.109005+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.2","version":"380'7343","reported_seq":7394,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-04-23T16:32:29.429628+0000","last_change":"2024-04-23T16:31:29.138797+0000","last_active":"2024-04-23T16:32:29.429628+0000","last_peered":"2024-04-23T16:32:29.429628+0000","last_clean":"2024-04-23T16:32:29.429628+0000","last_became_active":"2024-04-23T16:31:29.138430+0000","last_became_peered":"2024-04-23T16:31:29.138430+0000","last_unstale":"2024-04-23T16:32:29.429628+0000","last_undegraded":"2024-04-23T16:32:29.429628+0000","last_fullsized":"2024-04-23T16:32:29.429628+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":7343,"log_dups_size":0,"ondisk_log_size":7343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:31.211006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481230848,"num_objects":7343,"num_object_clones":0,"num_object_copies":14686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7343,"num_write_kb":469952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1002,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-23T16:32:25.789257+0000","last_change":"2024-04-23T16:31:26.858471+0000","last_active":"2024-04-23T16:32:25.789257+0000","last_peered":"2024-04-23T16:32:25.789257+0000","last_clean":"2024-04-23T16:32:25.789257+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:32:25.789257+0000","last_undegraded":"2024-04-23T16:32:25.789257+0000","last_fullsized":"2024-04-23T16:32:25.789257+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:26.858333+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:26.858333+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:43.789314+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"379'7330","reported_seq":7379,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-23T16:32:28.938433+0000","last_change":"2024-04-23T16:31:29.139641+0000","last_active":"2024-04-23T16:32:28.938433+0000","last_peered":"2024-04-23T16:32:28.938433+0000","last_clean":"2024-04-23T16:32:28.938433+0000","last_became_active":"2024-04-23T16:31:29.139398+0000","last_became_peered":"2024-04-23T16:31:29.139398+0000","last_unstale":"2024-04-23T16:32:28.938433+0000","last_undegraded":"2024-04-23T16:32:28.938433+0000","last_fullsized":"2024-04-23T16:32:28.938433+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":7330,"log_dups_size":0,"ondisk_log_size":7330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:41.180907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480378880,"num_objects":7330,"num_object_clones":0,"num_object_copies":14660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7330,"num_write_kb":469120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1004,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-04-23T16:32:29.131876+0000","last_change":"2024-04-23T16:32:28.560894+0000","last_active":"2024-04-23T16:32:29.131876+0000","last_peered":"2024-04-23T16:32:29.131876+0000","last_clean":"2024-04-23T16:32:29.131876+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:32:29.131876+0000","last_undegraded":"2024-04-23T16:32:29.131876+0000","last_fullsized":"2024-04-23T16:32:29.131876+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:32:28.560754+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:32:28.560754+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:52.803146+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"380'7258","reported_seq":7308,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-04-23T16:32:29.139773+0000","last_change":"2024-04-23T16:31:29.139304+0000","last_active":"2024-04-23T16:32:29.139773+0000","last_peered":"2024-04-23T16:32:29.139773+0000","last_clean":"2024-04-23T16:32:29.139773+0000","last_became_active":"2024-04-23T16:31:29.138968+0000","last_became_peered":"2024-04-23T16:31:29.138968+0000","last_unstale":"2024-04-23T16:32:29.139773+0000","last_undegraded":"2024-04-23T16:32:29.139773+0000","last_fullsized":"2024-04-23T16:32:29.139773+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":7258,"log_dups_size":0,"ondisk_log_size":7258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:41.287078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475660288,"num_objects":7258,"num_object_clones":0,"num_object_copies":14516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7258,"num_write_kb":464512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":893,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-23T16:32:25.787623+0000","last_change":"2024-04-23T16:31:30.064468+0000","last_active":"2024-04-23T16:32:25.787623+0000","last_peered":"2024-04-23T16:32:25.787623+0000","last_clean":"2024-04-23T16:32:25.787623+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:32:25.787623+0000","last_undegraded":"2024-04-23T16:32:25.787623+0000","last_fullsized":"2024-04-23T16:32:25.787623+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:30.064354+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:30.064354+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:49.450707+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"380'7516","reported_seq":7566,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-04-23T16:32:29.170532+0000","last_change":"2024-04-23T16:31:29.310501+0000","last_active":"2024-04-23T16:32:29.170532+0000","last_peered":"2024-04-23T16:32:29.170532+0000","last_clean":"2024-04-23T16:32:29.170532+0000","last_became_active":"2024-04-23T16:31:29.310246+0000","last_became_peered":"2024-04-23T16:31:29.310246+0000","last_unstale":"2024-04-23T16:32:29.170532+0000","last_undegraded":"2024-04-23T16:32:29.170532+0000","last_fullsized":"2024-04-23T16:32:29.170532+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":7516,"log_dups_size":0,"ondisk_log_size":7516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:36.052300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":492568576,"num_objects":7516,"num_object_clones":0,"num_object_copies":15032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7516,"num_write_kb":481024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1001,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-23T16:32:25.789210+0000","last_change":"2024-04-23T16:31:51.684967+0000","last_active":"2024-04-23T16:32:25.789210+0000","last_peered":"2024-04-23T16:32:25.789210+0000","last_clean":"2024-04-23T16:32:25.789210+0000","last_became_active":"2024-04-23T16:28:25.050105+0000","last_became_peered":"2024-04-23T16:28:25.050105+0000","last_unstale":"2024-04-23T16:32:25.789210+0000","last_undegraded":"2024-04-23T16:32:25.789210+0000","last_fullsized":"2024-04-23T16:32:25.789210+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:31:51.684868+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:09.608871+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.4","version":"379'7356","reported_seq":7405,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-23T16:32:28.871790+0000","last_change":"2024-04-23T16:31:29.139993+0000","last_active":"2024-04-23T16:32:28.871790+0000","last_peered":"2024-04-23T16:32:28.871790+0000","last_clean":"2024-04-23T16:32:28.871790+0000","last_became_active":"2024-04-23T16:31:29.139762+0000","last_became_peered":"2024-04-23T16:31:29.139762+0000","last_unstale":"2024-04-23T16:32:28.871790+0000","last_undegraded":"2024-04-23T16:32:28.871790+0000","last_fullsized":"2024-04-23T16:32:28.871790+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:31:28.121358+0000","objects_scrubbed":0,"log_size":7356,"log_dups_size":0,"ondisk_log_size":7356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:34.629312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482082816,"num_objects":7356,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7356,"num_write_kb":470784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1002,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-04-23T16:32:29.140136+0000","last_change":"2024-04-23T16:31:54.636517+0000","last_active":"2024-04-23T16:32:29.140136+0000","last_peered":"2024-04-23T16:32:29.140136+0000","last_clean":"2024-04-23T16:32:29.140136+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:32:29.140136+0000","last_undegraded":"2024-04-23T16:32:29.140136+0000","last_fullsized":"2024-04-23T16:32:29.140136+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:31:54.636376+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:54.835653+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":7713456128,"num_objects":117698,"num_object_clones":0,"num_object_copies":235396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117698,"num_write_kb":7532672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15415312384,"data_stored":15415312384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117698,"ondisk_log_size":117698,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056846,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1369776,"kb_used_data":887392,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":92375376,"statfs":{"total":95995035648,"available":94592385024,"internally_reserved":0,"allocated":908689408,"data_stored":908185491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":493943236},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351188,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612826,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1703848,"kb_used_data":1406112,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92041304,"statfs":{"total":95995035648,"available":94250295296,"internally_reserved":0,"allocated":1439858688,"data_stored":1439354779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":304871872},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841757,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1105448,"kb_used_data":935456,"kb_used_omap":1,"kb_used_meta":169982,"kb_avail":92639704,"statfs":{"total":95995035648,"available":94863056896,"internally_reserved":0,"allocated":957906944,"data_stored":957403035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":174062021},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168803,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":807544,"kb_used_data":479276,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92937608,"statfs":{"total":95995035648,"available":95168110592,"internally_reserved":0,"allocated":490778624,"data_stored":490268661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":336132554},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463145,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1694648,"kb_used_data":1420524,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92050504,"statfs":{"total":95995035648,"available":94259716096,"internally_reserved":0,"allocated":1454616576,"data_stored":1454106613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280689094},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137284,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738630,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3568232,"kb_used_data":3248288,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":90176920,"statfs":{"total":95995035648,"available":92341166080,"internally_reserved":0,"allocated":3326246912,"data_stored":3325743003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313467,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1620592,"kb_used_data":1422824,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":92124560,"statfs":{"total":95995035648,"available":94335549440,"internally_reserved":0,"allocated":1456971776,"data_stored":1456464315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117238,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":824028,"kb_used_data":459668,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":92921124,"statfs":{"total":95995035648,"available":95151230976,"internally_reserved":0,"allocated":470700032,"data_stored":470202127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921009,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2767336,"kb_used_data":2361312,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":90977816,"statfs":{"total":95995035648,"available":93161283584,"internally_reserved":0,"allocated":2417983488,"data_stored":2417479579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757484,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2665840,"kb_used_data":2309800,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":91079312,"statfs":{"total":95995035648,"available":93265215488,"internally_reserved":0,"allocated":2365235200,"data_stored":2364727739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3368550400,"data_stored":3368550400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1462632448,"data_stored":1462632448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":470089728,"data_stored":470089728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2439905280,"data_stored":2439905280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2396979200,"data_stored":2396979200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1454243840,"data_stored":1454243840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":493223936,"data_stored":493223936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":961544192,"data_stored":961544192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1449852928,"data_stored":1449852928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":918290432,"data_stored":918290432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:32:32.627 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-23T16:32:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:32 smithi103 ceph-mon[39629]: from='client.15837 v1:172.21.15.103:0/2782927504' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:32:33.067 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:33.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:33.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:33.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:32 smithi148 ceph-mon[44520]: from='client.15837 v1:172.21.15.103:0/2782927504' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:32:33.177 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:32:33.206 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:32 smithi188 ceph-mon[44101]: from='client.15837 v1:172.21.15.103:0/2782927504' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:32:33.208 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:33.230 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:32:33.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:33.236 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:32:33.276 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 54 16 124388 124372 143.937 94 0.00577108 0.00694618 2024-04-23T16:32:33.337 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:32:33.375 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:33.404 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:32:33.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:33.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:32:33.508 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:33.544 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:33.656 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:33.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:33.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:33.688 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:32:33.709 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:32:33.765 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:32:33.793 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:33.820 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:32:33.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:33.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:32:33.925 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:32:33.962 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:34.044 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:32:34.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:34.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:32:34.095 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:33 smithi103 ceph-mon[39629]: pgmap v1069: 27 pgs: 27 active+clean; 7.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-23T16:32:34.096 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:33 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:32:34.096 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:33 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:32:34.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:33 smithi148 ceph-mon[44520]: pgmap v1069: 27 pgs: 27 active+clean; 7.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-23T16:32:34.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:33 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:32:34.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:33 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:32:34.148 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:32:34.179 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:33 smithi188 ceph-mon[44101]: pgmap v1069: 27 pgs: 27 active+clean; 7.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-23T16:32:34.179 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:33 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:32:34.179 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:33 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:32:34.181 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:34.201 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:32:34.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:34.206 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:32:34.276 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 55 16 125789 125773 142.912 87.5625 0.00782439 0.00699593 2024-04-23T16:32:34.307 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:32:34.351 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:34.410 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:32:34.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:34.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:32:34.517 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:32:34.549 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:34.572 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:32:34.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:34.575 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:32:34.677 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:32:34.707 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:34.730 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:32:34.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:34.734 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:32:34.835 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:32:34.866 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:34.888 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:32:34.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:34.892 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:32:34.993 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:32:35.036 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:35.074 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:32:35.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:35.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:32:35.153 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:35 smithi188 ceph-mon[44101]: pgmap v1070: 27 pgs: 27 active+clean; 7.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-23T16:32:35.190 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:32:35.237 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:35.273 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:32:35.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:35.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:32:35.277 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 56 16 127201 127185 141.936 88.25 0.0104164 0.00704395 2024-04-23T16:32:35.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:35 smithi103 ceph-mon[39629]: pgmap v1070: 27 pgs: 27 active+clean; 7.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-23T16:32:35.377 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:35.430 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:35.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:35 smithi148 ceph-mon[44520]: pgmap v1070: 27 pgs: 27 active+clean; 7.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-23T16:32:36.177 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:36 smithi103 ceph-mon[39629]: 5.4 scrub starts 2024-04-23T16:32:36.177 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:36 smithi103 ceph-mon[39629]: 5.2 scrub starts 2024-04-23T16:32:36.177 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3055422735' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:32:36.177 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:36 smithi103 ceph-mon[39629]: from='client.25777 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:32:36.277 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 57 16 128388 128372 140.747 74.1875 0.00653134 0.00710351 2024-04-23T16:32:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:36 smithi188 ceph-mon[44101]: 5.4 scrub starts 2024-04-23T16:32:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:36 smithi188 ceph-mon[44101]: 5.2 scrub starts 2024-04-23T16:32:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3055422735' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:32:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:36 smithi188 ceph-mon[44101]: from='client.25777 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:32:36.505 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:32:36.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:36 smithi148 ceph-mon[44520]: 5.4 scrub starts 2024-04-23T16:32:36.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:36 smithi148 ceph-mon[44520]: 5.2 scrub starts 2024-04-23T16:32:36.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3055422735' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:32:36.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:36 smithi148 ceph-mon[44520]: from='client.25777 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:32:37.277 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 58 16 129484 129468 139.502 68.5 0.00608731 0.00716682 2024-04-23T16:32:37.304 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:37.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:37.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:37.409 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:37.442 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:37 smithi103 ceph-mon[39629]: 5.9 scrub starts 2024-04-23T16:32:37.443 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:37 smithi103 ceph-mon[39629]: 5.4 scrub ok 2024-04-23T16:32:37.443 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:37 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:32:37.443 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:37 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:32:37.443 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:37 smithi103 ceph-mon[39629]: from='client.25777 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:32:37.443 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:37 smithi103 ceph-mon[39629]: osdmap e381: 12 total, 12 up, 10 in 2024-04-23T16:32:37.443 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:37 smithi103 ceph-mon[39629]: 5.c scrub starts 2024-04-23T16:32:37.443 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:37 smithi103 ceph-mon[39629]: pgmap v1072: 27 pgs: 27 active+clean; 7.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-23T16:32:37.445 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:37 smithi188 ceph-mon[44101]: 5.9 scrub starts 2024-04-23T16:32:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:37 smithi188 ceph-mon[44101]: 5.4 scrub ok 2024-04-23T16:32:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:37 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:32:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:37 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:32:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:37 smithi188 ceph-mon[44101]: from='client.25777 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:32:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:37 smithi188 ceph-mon[44101]: osdmap e381: 12 total, 12 up, 10 in 2024-04-23T16:32:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:37 smithi188 ceph-mon[44101]: 5.c scrub starts 2024-04-23T16:32:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:37 smithi188 ceph-mon[44101]: pgmap v1072: 27 pgs: 27 active+clean; 7.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-23T16:32:37.607 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:37.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:37.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:37.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:37 smithi148 ceph-mon[44520]: 5.9 scrub starts 2024-04-23T16:32:37.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:37 smithi148 ceph-mon[44520]: 5.4 scrub ok 2024-04-23T16:32:37.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:37 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:32:37.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:37 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:32:37.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:37 smithi148 ceph-mon[44520]: from='client.25777 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:32:37.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:37 smithi148 ceph-mon[44520]: osdmap e381: 12 total, 12 up, 10 in 2024-04-23T16:32:37.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:37 smithi148 ceph-mon[44520]: 5.c scrub starts 2024-04-23T16:32:37.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:37 smithi148 ceph-mon[44520]: pgmap v1072: 27 pgs: 27 active+clean; 7.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-23T16:32:37.714 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:32:37.760 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:38.112 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:32:38.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:38.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:32:38.216 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:32:38.246 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:38.267 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:32:38.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:38.270 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:32:38.276 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 59 16 130742 130726 138.47 78.625 0.0120378 0.0072203 2024-04-23T16:32:38.371 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:32:38.413 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:38.445 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:32:38.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:38.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:32:38.451 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:38 smithi148 ceph-mon[44520]: 5.2 scrub ok 2024-04-23T16:32:38.451 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:38 smithi148 ceph-mon[44520]: 5.1 scrub starts 2024-04-23T16:32:38.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:38 smithi188 ceph-mon[44101]: 5.2 scrub ok 2024-04-23T16:32:38.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:38 smithi188 ceph-mon[44101]: 5.1 scrub starts 2024-04-23T16:32:38.501 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:32:38.532 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:32:38.552 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:38.566 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:32:38.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:38 smithi103 ceph-mon[39629]: 5.2 scrub ok 2024-04-23T16:32:38.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:38 smithi103 ceph-mon[39629]: 5.1 scrub starts 2024-04-23T16:32:38.591 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:38.616 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:32:38.638 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:32:38.675 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:32:38.716 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:32:38.742 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:32:38.771 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:32:38.797 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:32:38.825 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:32:38.856 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:32:38.857 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:32:38.857 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:32:38.857 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:32:38.857 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:32:38.857 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:32:38.857 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:32:38.857 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:32:38.857 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:32:38.858 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:32:38.858 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:32:38.858 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:32:38.858 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:32:38.864 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:38.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:38.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:38.968 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:32:39.043 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:39.085 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:32:39.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:39.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:32:39.194 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:32:39.203 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:39 smithi103 ceph-mon[39629]: 5.9 scrub ok 2024-04-23T16:32:39.203 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:39 smithi103 ceph-mon[39629]: 5.c scrub ok 2024-04-23T16:32:39.203 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:39 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/583233417' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:32:39.203 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:39.203 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:39.203 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:39 smithi103 ceph-mon[39629]: pgmap v1073: 27 pgs: 27 active+clean; 7.7 GiB data, 19 GiB used, 876 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-23T16:32:39.204 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:39.204 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:39.204 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:39 smithi103 ceph-mon[39629]: osdmap e382: 12 total, 12 up, 10 in 2024-04-23T16:32:39.214 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:32:39.214 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:32:39.237 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:39.267 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:32:39.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:39.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:32:39.277 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:32:39.276714 min lat: 0.0010294 max lat: 1.8332 avg lat: 0.00727266 2024-04-23T16:32:39.277 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:32:39.277 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 60 15 132006 131991 137.48 79.0625 0.0235195 0.00727266 2024-04-23T16:32:39.335 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:32:39.374 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:32:39.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:39 smithi148 ceph-mon[44520]: 5.9 scrub ok 2024-04-23T16:32:39.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:39 smithi148 ceph-mon[44520]: 5.c scrub ok 2024-04-23T16:32:39.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:39 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/583233417' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:32:39.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:39.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:39.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:39 smithi148 ceph-mon[44520]: pgmap v1073: 27 pgs: 27 active+clean; 7.7 GiB data, 19 GiB used, 876 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-23T16:32:39.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:39.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:39.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:39 smithi148 ceph-mon[44520]: osdmap e382: 12 total, 12 up, 10 in 2024-04-23T16:32:39.425 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:39.454 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:32:39.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:39.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:32:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:39 smithi188 ceph-mon[44101]: 5.9 scrub ok 2024-04-23T16:32:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:39 smithi188 ceph-mon[44101]: 5.c scrub ok 2024-04-23T16:32:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:39 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/583233417' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:32:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:39 smithi188 ceph-mon[44101]: pgmap v1073: 27 pgs: 27 active+clean; 7.7 GiB data, 19 GiB used, 876 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-23T16:32:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:39 smithi188 ceph-mon[44101]: osdmap e382: 12 total, 12 up, 10 in 2024-04-23T16:32:39.559 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:32:39.609 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:39.639 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:32:39.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:39.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:32:39.746 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:32:39.798 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:40.007 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:32:40.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:40.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:32:40.112 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:40.151 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:40.277 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 61 15 133388 133373 136.642 86.375 0.0233091 0.00731703 2024-04-23T16:32:40.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:40 smithi103 ceph-mon[39629]: 5.1 scrub ok 2024-04-23T16:32:40.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:40 smithi188 ceph-mon[44101]: 5.1 scrub ok 2024-04-23T16:32:40.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:40.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:40 smithi148 ceph-mon[44520]: 5.1 scrub ok 2024-04-23T16:32:41.113 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:41.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:41.139 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-23T16:32:41.195 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:32:41.218 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:32:41.273 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:41.277 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 62 16 134759 134743 135.819 85.625 0.0152206 0.00736113 2024-04-23T16:32:41.303 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:32:41.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:41.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:32:41.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:41 smithi103 ceph-mon[39629]: pgmap v1075: 27 pgs: 27 active+clean; 8.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-04-23T16:32:41.407 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:32:41.459 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:41.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:41 smithi188 ceph-mon[44101]: pgmap v1075: 27 pgs: 27 active+clean; 8.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-04-23T16:32:41.485 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:32:41.486 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:41 smithi148 ceph-mon[44520]: pgmap v1075: 27 pgs: 27 active+clean; 8.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-04-23T16:32:41.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:41.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:32:41.589 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:32:41.626 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:41.709 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:32:41.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:41.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:32:41.814 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:41.858 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:42.045 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:32:42.124 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:42.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:42.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:42.164 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:32:42.227 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:42.277 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 63 16 135954 135938 134.848 74.6875 0.005774 0.00741401 2024-04-23T16:32:42.281 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:42.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:42 smithi103 ceph-mon[39629]: 5.6 scrub starts 2024-04-23T16:32:42.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:42 smithi148 ceph-mon[44520]: 5.6 scrub starts 2024-04-23T16:32:42.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:42 smithi188 ceph-mon[44101]: 5.6 scrub starts 2024-04-23T16:32:43.277 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 64 16 136915 136899 133.68 60.0625 0.0121094 0.00747857 2024-04-23T16:32:43.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:43 smithi103 ceph-mon[39629]: 5.3 deep-scrub starts 2024-04-23T16:32:43.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:43 smithi103 ceph-mon[39629]: 5.0 scrub starts 2024-04-23T16:32:43.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:43 smithi103 ceph-mon[39629]: 5.a scrub starts 2024-04-23T16:32:43.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:43 smithi103 ceph-mon[39629]: pgmap v1076: 27 pgs: 27 active+clean; 8.1 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 59 MiB/s wr, 947 op/s 2024-04-23T16:32:43.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:43 smithi103 ceph-mon[39629]: 5.6 scrub ok 2024-04-23T16:32:43.451 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:43.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:43.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:43.557 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:32:43.600 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:43.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:43 smithi148 ceph-mon[44520]: 5.3 deep-scrub starts 2024-04-23T16:32:43.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:43 smithi148 ceph-mon[44520]: 5.0 scrub starts 2024-04-23T16:32:43.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:43 smithi148 ceph-mon[44520]: 5.a scrub starts 2024-04-23T16:32:43.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:43 smithi148 ceph-mon[44520]: pgmap v1076: 27 pgs: 27 active+clean; 8.1 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 59 MiB/s wr, 947 op/s 2024-04-23T16:32:43.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:43 smithi148 ceph-mon[44520]: 5.6 scrub ok 2024-04-23T16:32:43.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:43 smithi188 ceph-mon[44101]: 5.3 deep-scrub starts 2024-04-23T16:32:43.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:43 smithi188 ceph-mon[44101]: 5.0 scrub starts 2024-04-23T16:32:43.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:43 smithi188 ceph-mon[44101]: 5.a scrub starts 2024-04-23T16:32:43.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:43 smithi188 ceph-mon[44101]: pgmap v1076: 27 pgs: 27 active+clean; 8.1 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 59 MiB/s wr, 947 op/s 2024-04-23T16:32:43.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:43 smithi188 ceph-mon[44101]: 5.6 scrub ok 2024-04-23T16:32:43.859 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:32:43.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:43.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:32:43.967 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:32:44.000 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:44.101 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:32:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:44.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:32:44.209 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:32:44.253 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:44.277 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 65 16 137897 137881 132.567 61.375 0.0169539 0.00754115 2024-04-23T16:32:44.587 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:44 smithi103 ceph-mon[39629]: 5.a scrub ok 2024-04-23T16:32:44.611 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:32:44.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:44.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:32:44.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:44 smithi148 ceph-mon[44520]: 5.a scrub ok 2024-04-23T16:32:44.627 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:32:44.721 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:32:44.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:44 smithi188 ceph-mon[44101]: 5.a scrub ok 2024-04-23T16:32:44.749 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:44.769 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:32:44.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:44.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:32:44.873 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:32:44.924 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:44.956 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:32:44.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:44.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:32:45.061 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:45.094 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:45.272 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:45 smithi103 ceph-mon[39629]: 5.0 scrub ok 2024-04-23T16:32:45.272 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:45 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2922329217' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:32:45.272 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:45 smithi103 ceph-mon[39629]: from='client.35735 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:32:45.272 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:45 smithi103 ceph-mon[39629]: pgmap v1077: 27 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 25 active+clean; 8.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-23T16:32:45.277 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 66 16 138962 138946 131.567 66.5625 0.0276023 0.0075993 2024-04-23T16:32:45.280 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:45 smithi148 ceph-mon[44520]: 5.0 scrub ok 2024-04-23T16:32:45.281 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:45 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2922329217' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:32:45.281 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:45 smithi148 ceph-mon[44520]: from='client.35735 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:32:45.281 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:45 smithi148 ceph-mon[44520]: pgmap v1077: 27 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 25 active+clean; 8.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-23T16:32:45.293 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:32:45.298 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:45.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:45.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:45.403 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:32:45.430 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:45.451 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:32:45.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:45.454 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:32:45.555 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:32:45.589 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:45.673 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:32:45.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:45.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:32:45.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:45 smithi188 ceph-mon[44101]: 5.0 scrub ok 2024-04-23T16:32:45.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:45 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2922329217' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:32:45.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:45 smithi188 ceph-mon[44101]: from='client.35735 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:32:45.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:45 smithi188 ceph-mon[44101]: pgmap v1077: 27 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 25 active+clean; 8.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-23T16:32:45.779 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:45.811 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:46.014 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:46.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:46.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:46.119 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:46.153 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:46.277 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 67 16 140059 140043 130.627 68.5625 0.0129379 0.007653 2024-04-23T16:32:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:46 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:32:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:46 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:32:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:46 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:32:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:46 smithi103 ceph-mon[39629]: from='client.35735 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:32:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:46 smithi103 ceph-mon[39629]: osdmap e383: 12 total, 12 up, 10 in 2024-04-23T16:32:46.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:46 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:32:46.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:46 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:32:46.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:46 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:32:46.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:46 smithi148 ceph-mon[44520]: from='client.35735 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:32:46.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:46 smithi148 ceph-mon[44520]: osdmap e383: 12 total, 12 up, 10 in 2024-04-23T16:32:46.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:46 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:32:46.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:46 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:32:46.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:46 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:32:46.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:46 smithi188 ceph-mon[44101]: from='client.35735 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:32:46.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:46 smithi188 ceph-mon[44101]: osdmap e383: 12 total, 12 up, 10 in 2024-04-23T16:32:46.812 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:46.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:46.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:46.915 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:32:46.960 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:47.277 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 68 16 141192 141176 129.747 70.8125 0.0145347 0.00770538 2024-04-23T16:32:47.388 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:32:47.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:47.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:32:47.492 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:32:47.536 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:47 smithi148 ceph-mon[44520]: pgmap v1079: 27 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 25 active+clean; 8.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-23T16:32:47.537 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:47.568 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:32:47.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:47.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:32:47.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:47 smithi103 ceph-mon[39629]: pgmap v1079: 27 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 25 active+clean; 8.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-23T16:32:47.673 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:32:47.705 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:47.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:47 smithi188 ceph-mon[44101]: pgmap v1079: 27 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 25 active+clean; 8.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-23T16:32:47.739 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:32:47.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:47.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:32:47.844 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:32:47.887 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:47.918 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:32:47.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:47.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:32:48.024 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:32:48.062 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:48.103 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:32:48.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:48.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:32:48.122 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:32:48.122 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:32:48.122 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-23T16:32:48.122 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:32:48.175 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:32:48.175 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:32:48.175 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:32:48.175 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:32:48.175 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:32:48.175 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:32:48.180 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:32:48.213 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:32:48.214 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:32:48.244 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:48.249 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:32:48.266 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:32:48.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:48.269 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:32:48.273 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:32:48.277 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 69 16 142225 142209 128.802 64.5625 0.0159785 0.00776187 2024-04-23T16:32:48.308 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:32:48.337 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:32:48.339 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:32:48.339 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:32:48.339 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:32:48.339 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:32:48.339 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:32:48.339 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:32:48.339 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:32:48.339 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:32:48.339 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:32:48.339 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:32:48.340 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:32:48.340 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:32:48.370 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:32:48.422 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:48.450 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:32:48.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:48.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:32:48.555 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:48.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:48 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:32:48.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:48 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:32:48.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:48 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3922390788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:32:48.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:48.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:48.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:48.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:48.586 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:48.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:48 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:32:48.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:48 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:32:48.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:48 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3922390788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:32:48.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:48.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:48.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:48.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:48.673 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:48.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:48.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:48.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:48 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:32:48.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:48 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:32:48.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:48 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3922390788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:32:48.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:48.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:48.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:48.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:48.777 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:32:48.806 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:48.832 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:32:48.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:48.835 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:32:48.936 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:32:48.993 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:49.033 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:32:49.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:49.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:32:49.142 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:49.182 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:49.193 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-23T16:32:49.215 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:49.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:49.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:49.277 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 70 16 143220 143204 127.85 62.1875 0.00473836 0.00781749 2024-04-23T16:32:49.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:49 smithi148 ceph-mon[44520]: osdmap e384: 12 total, 12 up, 10 in 2024-04-23T16:32:49.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:49 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:32:49.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:49 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:32:49.306 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:49 smithi148 ceph-mon[44520]: pgmap v1081: 27 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 25 active+clean; 8.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 57 MiB/s wr, 915 op/s 2024-04-23T16:32:49.319 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:32:49.353 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:49.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:32:49.475 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.475 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.475 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.475 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.476 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.476 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.476 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:49 smithi103 ceph-mon[39629]: osdmap e384: 12 total, 12 up, 10 in 2024-04-23T16:32:49.476 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.476 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.476 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:49 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:32:49.476 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:49 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:32:49.476 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:49 smithi103 ceph-mon[39629]: pgmap v1081: 27 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 25 active+clean; 8.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 57 MiB/s wr, 915 op/s 2024-04-23T16:32:49.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:49 smithi188 ceph-mon[44101]: osdmap e384: 12 total, 12 up, 10 in 2024-04-23T16:32:49.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:32:49.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:49 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:32:49.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:49 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:32:49.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:49 smithi188 ceph-mon[44101]: pgmap v1081: 27 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 25 active+clean; 8.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 57 MiB/s wr, 915 op/s 2024-04-23T16:32:49.925 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:32:49.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:49.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:32:49.954 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:32:50.030 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:32:50.090 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:50.120 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:32:50.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:50.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:32:50.225 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:32:50.263 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:50.278 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 71 16 144166 144150 126.882 59.125 0.00449498 0.00787871 2024-04-23T16:32:50.300 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:32:50.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:50.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:32:50.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:50 smithi148 ceph-mon[44520]: osdmap e385: 12 total, 12 up, 10 in 2024-04-23T16:32:50.411 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:32:50.454 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:50.490 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:32:50.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:50.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:32:50.504 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:50 smithi103 ceph-mon[39629]: osdmap e385: 12 total, 12 up, 10 in 2024-04-23T16:32:50.594 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:32:50.641 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:50.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:50 smithi188 ceph-mon[44101]: osdmap e385: 12 total, 12 up, 10 in 2024-04-23T16:32:50.829 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:32:50.847 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:32:50.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:50.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:32:50.956 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:50.999 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:51.130 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:51.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:51.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:51.236 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:32:51.276 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:51.278 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 72 16 145183 145167 126.003 63.5625 0.00573689 0.00793463 2024-04-23T16:32:51.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:51 smithi103 ceph-mon[39629]: pgmap v1083: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 59 MiB/s wr, 944 op/s 2024-04-23T16:32:51.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:51 smithi148 ceph-mon[44520]: pgmap v1083: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 59 MiB/s wr, 944 op/s 2024-04-23T16:32:51.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:51 smithi188 ceph-mon[44101]: pgmap v1083: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 59 MiB/s wr, 944 op/s 2024-04-23T16:32:51.886 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:32:52.278 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 73 15 146204 146189 125.152 63.875 0.00884736 0.00798845 2024-04-23T16:32:52.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:52 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:32:52.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:52 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:32:52.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:52 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:32:52.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:52 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:32:52.627 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-04-23T16:32:52.628 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1] 2024-04-23T16:32:52.628 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-23T16:32:52.628 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.2 2024-04-23T16:32:52.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:52 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:32:52.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:52 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:32:53.100 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:32:52 smithi103 systemd[1]: Stopping Ceph osd.2 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T16:32:53.278 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 74 16 147068 147052 124.189 53.9375 0.00540993 0.00805039 2024-04-23T16:32:53.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:53 smithi103 ceph-mon[39629]: osdmap e386: 12 total, 12 up, 10 in 2024-04-23T16:32:53.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:53 smithi103 ceph-mon[39629]: pgmap v1085: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 62 MiB/s wr, 994 op/s 2024-04-23T16:32:53.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:53 smithi148 ceph-mon[44520]: osdmap e386: 12 total, 12 up, 10 in 2024-04-23T16:32:53.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:53 smithi148 ceph-mon[44520]: pgmap v1085: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 62 MiB/s wr, 994 op/s 2024-04-23T16:32:53.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:53 smithi188 ceph-mon[44101]: osdmap e386: 12 total, 12 up, 10 in 2024-04-23T16:32:53.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:53 smithi188 ceph-mon[44101]: pgmap v1085: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 62 MiB/s wr, 994 op/s 2024-04-23T16:32:54.278 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 75 16 148041 148025 123.344 60.8125 0.0057067 0.00810167 2024-04-23T16:32:55.278 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 76 15 149124 149109 122.613 67.75 0.0114777 0.00815414 2024-04-23T16:32:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:55 smithi188 ceph-mon[44101]: pgmap v1086: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 9.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-04-23T16:32:55.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:55 smithi103 ceph-mon[39629]: pgmap v1086: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 9.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-04-23T16:32:55.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:55 smithi148 ceph-mon[44520]: pgmap v1086: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 9.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-04-23T16:32:56.278 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 77 16 150133 150117 121.838 63 0.0102505 0.00820581 2024-04-23T16:32:57.278 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 78 15 151137 151122 121.082 62.8125 0.0640519 0.00825713 2024-04-23T16:32:57.931 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:32:57.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:57.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:32:58.036 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:32:58.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:57 smithi188 ceph-mon[44101]: pgmap v1087: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 9.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-04-23T16:32:58.064 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:58.089 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:32:58.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:58.093 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:32:58.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:57 smithi103 ceph-mon[39629]: pgmap v1087: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 9.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-04-23T16:32:58.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:57 smithi148 ceph-mon[44520]: pgmap v1087: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 9.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-04-23T16:32:58.194 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:32:58.222 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:58.242 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:32:58.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:58.245 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:32:58.278 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 79 16 152058 152042 120.277 57.5 0.0441284 0.00831103 2024-04-23T16:32:58.346 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:32:58.374 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:58.396 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:32:58.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:58.399 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:32:58.499 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:32:58.526 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:58.546 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:32:58.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:58.549 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:32:58.649 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:32:58.697 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:58.738 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:32:58.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:58.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:32:58.847 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:32:58.880 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:58.905 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:32:58.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:58.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:32:59.011 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:32:59.040 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:59.060 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:32:59.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:59.064 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:32:59.165 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:32:59.181 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:32:58 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[72723]: 2024-04-23T16:32:58.865+0000 7f2db6326700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T16:32:59.181 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:32:58 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[72723]: 2024-04-23T16:32:58.865+0000 7f2db6326700 -1 osd.2 386 *** Got signal Terminated *** 2024-04-23T16:32:59.181 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:32:58 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[72723]: 2024-04-23T16:32:58.865+0000 7f2db6326700 -1 osd.2 386 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T16:32:59.188 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:32:59.205 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:59.227 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:32:59.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:59.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:32:59.278 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:32:59.278344 min lat: 0.0010294 max lat: 1.8332 avg lat: 0.00836351 2024-04-23T16:32:59.278 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:32:59.279 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 80 16 152997 152981 119.507 58.6875 0.00479681 0.00836351 2024-04-23T16:32:59.334 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:32:59.361 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:59.385 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:32:59.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:59.389 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:32:59.490 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:32:59.519 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:59.538 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:32:59.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:59.542 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:32:59.643 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:32:59.673 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:59.695 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:32:59.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:59.701 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:32:59.759 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:59 smithi103 ceph-mon[39629]: pgmap v1088: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 9.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 45 MiB/s wr, 720 op/s 2024-04-23T16:32:59.759 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:32:59 smithi103 ceph-mon[39629]: osd.2 marked itself down and dead 2024-04-23T16:32:59.801 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:32:59.837 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:32:59.966 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:32:59.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:32:59.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:32:59.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:59 smithi188 ceph-mon[44101]: pgmap v1088: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 9.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 45 MiB/s wr, 720 op/s 2024-04-23T16:32:59.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:32:59 smithi188 ceph-mon[44101]: osd.2 marked itself down and dead 2024-04-23T16:33:00.071 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:33:00.095 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:59 smithi148 ceph-mon[44520]: pgmap v1088: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 9.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 45 MiB/s wr, 720 op/s 2024-04-23T16:33:00.095 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:32:59 smithi148 ceph-mon[44520]: osd.2 marked itself down and dead 2024-04-23T16:33:00.097 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:00.116 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:33:00.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:00.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:33:00.220 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:00.246 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:00.269 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:00.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:00.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:00.278 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 81 16 153594 153578 118.492 37.3125 0.0457902 0.00839412 2024-04-23T16:33:00.373 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:33:00.409 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:00.774 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:33:01.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:00 smithi103 ceph-mon[39629]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T16:33:01.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:00 smithi103 ceph-mon[39629]: osdmap e387: 12 total, 11 up, 10 in 2024-04-23T16:33:01.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:00 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3143449896' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:33:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:00 smithi148 ceph-mon[44520]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T16:33:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:00 smithi148 ceph-mon[44520]: osdmap e387: 12 total, 11 up, 10 in 2024-04-23T16:33:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:00 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3143449896' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:33:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:00 smithi188 ceph-mon[44101]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T16:33:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:00 smithi188 ceph-mon[44101]: osdmap e387: 12 total, 11 up, 10 in 2024-04-23T16:33:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:00 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3143449896' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:33:01.251 INFO:teuthology.orchestra.run.smithi103.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-23T16:33:01.279 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 82 16 153594 153578 117.047 0 - 0.00839412 2024-04-23T16:33:01.459 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:33:01.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:01.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:33:01.563 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:33:01.595 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:01.616 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:33:01.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:01.618 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:33:01.720 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:33:01.749 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:01.770 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:33:01.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:01.772 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:33:01.873 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:01.905 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:01.926 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:01.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:01.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:01 smithi188 ceph-mon[44101]: pgmap v1090: 27 pgs: 1 active+clean+scrubbing+deep, 4 stale+active+clean, 22 active+clean; 9.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-23T16:33:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:01 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:33:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:01 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3143449896' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:33:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:01 smithi188 ceph-mon[44101]: osdmap e388: 12 total, 11 up, 10 in 2024-04-23T16:33:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:01 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:33:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:01 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:33:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:01 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:33:01.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:01 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:33:02.027 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:01 smithi103 ceph-mon[39629]: pgmap v1090: 27 pgs: 1 active+clean+scrubbing+deep, 4 stale+active+clean, 22 active+clean; 9.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-23T16:33:02.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:01 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:33:02.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:01 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3143449896' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:33:02.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:01 smithi103 ceph-mon[39629]: osdmap e388: 12 total, 11 up, 10 in 2024-04-23T16:33:02.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:01 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:33:02.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:01 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:33:02.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:01 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:33:02.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:01 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:33:02.030 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:33:02.054 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T16:33:02.055 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-23T16:33:02.059 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:02.079 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:33:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:02.082 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:33:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:01 smithi148 ceph-mon[44520]: pgmap v1090: 27 pgs: 1 active+clean+scrubbing+deep, 4 stale+active+clean, 22 active+clean; 9.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-23T16:33:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:01 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:33:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:01 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3143449896' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:33:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:01 smithi148 ceph-mon[44520]: osdmap e388: 12 total, 11 up, 10 in 2024-04-23T16:33:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:01 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:33:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:01 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:33:02.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:01 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:33:02.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:01 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:33:02.182 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:33:02.209 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:02.279 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 83 16 153594 153578 115.636 0 - 0.00839412 2024-04-23T16:33:02.464 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:33:02.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:02.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:33:02.568 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:33:02.595 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:02.668 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:33:02.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:02.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:33:02.774 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:02.806 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:02.828 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:02.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:02.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:02.932 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:02.962 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:02.982 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:02.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:02.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:03.086 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:33:03.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:02 smithi103 ceph-mon[39629]: osdmap e389: 12 total, 11 up, 10 in 2024-04-23T16:33:03.100 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:33:02 smithi103 bash[184968]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2 2024-04-23T16:33:03.118 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:03.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:02 smithi148 ceph-mon[44520]: osdmap e389: 12 total, 11 up, 10 in 2024-04-23T16:33:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:02 smithi188 ceph-mon[44101]: osdmap e389: 12 total, 11 up, 10 in 2024-04-23T16:33:03.279 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 84 16 153594 153578 114.26 0 - 0.00839412 2024-04-23T16:33:03.777 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:33:03.796 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:33:03.815 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:33:03.835 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:33:03.856 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:33:03.870 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:33:03.875 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:33:03.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:03.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:33:03.896 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:33:03.916 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:33:03.936 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:33:03.956 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:33:03.976 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:33:03.977 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:33:03.977 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:33:03.977 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:33:03.977 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:33:03.977 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:33:03.977 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:33:03.977 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:33:03.977 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:33:03.977 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:33:03.977 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:33:03.977 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:33:03.977 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:33:03.987 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:04.019 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:04.041 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:04.043 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:03 smithi148 ceph-mon[44520]: pgmap v1093: 27 pgs: 1 active+clean+scrubbing+deep, 4 stale+active+clean, 22 active+clean; 9.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 55 MiB/s wr, 885 op/s 2024-04-23T16:33:04.043 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:03 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/450801568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:33:04.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:04.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:03 smithi103 ceph-mon[39629]: pgmap v1093: 27 pgs: 1 active+clean+scrubbing+deep, 4 stale+active+clean, 22 active+clean; 9.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 55 MiB/s wr, 885 op/s 2024-04-23T16:33:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:03 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/450801568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:33:04.145 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:33:04.174 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:04.193 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:33:04.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:04.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:33:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:03 smithi188 ceph-mon[44101]: pgmap v1093: 27 pgs: 1 active+clean+scrubbing+deep, 4 stale+active+clean, 22 active+clean; 9.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 55 MiB/s wr, 885 op/s 2024-04-23T16:33:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:03 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/450801568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:33:04.279 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 85 16 153594 153578 112.915 0 - 0.00839412 2024-04-23T16:33:04.298 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:33:04.329 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:04.351 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:33:04.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:04.354 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:33:04.455 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:04.483 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:04.503 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:04.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:04.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:04.607 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:33:04.635 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:04.656 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:33:04.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:04.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:33:04.767 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:33:04.799 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:04.820 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:33:04.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:04.823 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:33:04.924 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:33:04.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:04.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:04.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:04.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:04.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:04.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:04.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:04.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:04.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:04.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:04.951 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:04.979 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:33:04.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:04.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:33:05.002 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:33:05.003 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:33:05.056 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:33:05.083 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:33:05.112 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.112 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.112 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.113 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.115 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:05.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:05.136 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:33:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:05.139 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:33:05.240 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:33:05.267 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:05.279 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 86 16 153594 153578 111.602 0 - 0.00839412 2024-04-23T16:33:05.365 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:33:05.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:05.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:33:05.469 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:05.503 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:05.523 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:05.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:05.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:05.627 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:05.656 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:05.677 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:05.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:05.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:05.781 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:33:05.810 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:06.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:06.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:05 smithi103 ceph-mon[39629]: pgmap v1094: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 1.7 KiB/s rd, 50 MiB/s wr, 798 op/s; 28953/307160 objects degraded (9.426%); 31 KiB/s, 0 objects/s recovering 2024-04-23T16:33:06.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:05 smithi103 ceph-mon[39629]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-23T16:33:06.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:05 smithi103 ceph-mon[39629]: Health check failed: Degraded data redundancy: 28953/307160 objects degraded (9.426%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:06.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:05 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:33:06.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:05 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:33:06.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:06.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:05 smithi148 ceph-mon[44520]: pgmap v1094: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 1.7 KiB/s rd, 50 MiB/s wr, 798 op/s; 28953/307160 objects degraded (9.426%); 31 KiB/s, 0 objects/s recovering 2024-04-23T16:33:06.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:05 smithi148 ceph-mon[44520]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-23T16:33:06.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:05 smithi148 ceph-mon[44520]: Health check failed: Degraded data redundancy: 28953/307160 objects degraded (9.426%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:06.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:05 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:33:06.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:05 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:33:06.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:06.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:05 smithi188 ceph-mon[44101]: pgmap v1094: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 1.7 KiB/s rd, 50 MiB/s wr, 798 op/s; 28953/307160 objects degraded (9.426%); 31 KiB/s, 0 objects/s recovering 2024-04-23T16:33:06.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:05 smithi188 ceph-mon[44101]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-23T16:33:06.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:05 smithi188 ceph-mon[44101]: Health check failed: Degraded data redundancy: 28953/307160 objects degraded (9.426%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:06.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:05 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:33:06.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:05 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:33:06.279 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 87 16 153594 153578 110.32 0 - 0.00839412 2024-04-23T16:33:06.292 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:33:07.004 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:33:07.161 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:33:07.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:07.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:33:07.267 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:33:07.280 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 88 16 153594 153578 109.066 0 - 0.00839412 2024-04-23T16:33:07.291 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:07 smithi103 ceph-mon[39629]: osdmap e390: 12 total, 11 up, 10 in 2024-04-23T16:33:07.292 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:07 smithi103 ceph-mon[39629]: pgmap v1096: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 1.7 KiB/s rd, 11 MiB/s wr, 182 op/s; 28953/307160 objects degraded (9.426%); 31 KiB/s, 0 objects/s recovering 2024-04-23T16:33:07.294 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:07 smithi188 ceph-mon[44101]: osdmap e390: 12 total, 11 up, 10 in 2024-04-23T16:33:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:07 smithi188 ceph-mon[44101]: pgmap v1096: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 1.7 KiB/s rd, 11 MiB/s wr, 182 op/s; 28953/307160 objects degraded (9.426%); 31 KiB/s, 0 objects/s recovering 2024-04-23T16:33:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:07 smithi148 ceph-mon[44520]: osdmap e390: 12 total, 11 up, 10 in 2024-04-23T16:33:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:07 smithi148 ceph-mon[44520]: pgmap v1096: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 1.7 KiB/s rd, 11 MiB/s wr, 182 op/s; 28953/307160 objects degraded (9.426%); 31 KiB/s, 0 objects/s recovering 2024-04-23T16:33:07.672 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.2.service' 2024-04-23T16:33:07.752 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T16:33:07.752 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-23T16:33:07.753 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd down 2 2024-04-23T16:33:07.946 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:33:07.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:07.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:33:08.049 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:33:08.078 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:08.182 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:33:08.182 INFO:teuthology.orchestra.run.smithi103.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-23T16:33:08.203 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:33:08.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:08.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:33:08.280 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 89 16 153594 153578 107.84 0 - 0.00839412 2024-04-23T16:33:08.308 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:33:08.341 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:08.362 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:33:08.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:08.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:33:08.466 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:08.498 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:08.519 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:08.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:08.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:08.623 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:33:08.653 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:08.674 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:33:08.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:08.676 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:33:08.777 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:33:08.806 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:08.826 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:33:08.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:08.828 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:33:08.929 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:33:08.957 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:09.192 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T16:33:09.193 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-23T16:33:09.206 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:33:09.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:09.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:33:09.280 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 90 16 153594 153578 106.642 0 - 0.00839412 2024-04-23T16:33:09.310 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:33:09.342 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:09.363 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:33:09.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:09.366 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:33:09.464 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:33:09.466 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:33:09.496 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:09.517 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:33:09.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:09.520 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:33:09.620 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:33:09.648 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:09.667 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:33:09.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:09.670 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:33:09.772 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:09.801 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:09.820 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:09.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:09.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:09.924 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:09.952 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:09.972 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:09 smithi148 ceph-mon[44520]: pgmap v1097: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 1.3 KiB/s rd, 8.4 MiB/s wr, 137 op/s; 28953/307160 objects degraded (9.426%); 24 KiB/s, 0 objects/s recovering 2024-04-23T16:33:09.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:09 smithi188 ceph-mon[44101]: pgmap v1097: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 1.3 KiB/s rd, 8.4 MiB/s wr, 137 op/s; 28953/307160 objects degraded (9.426%); 24 KiB/s, 0 objects/s recovering 2024-04-23T16:33:10.092 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:09 smithi103 ceph-mon[39629]: pgmap v1097: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 1.3 KiB/s rd, 8.4 MiB/s wr, 137 op/s; 28953/307160 objects degraded (9.426%); 24 KiB/s, 0 objects/s recovering 2024-04-23T16:33:10.167 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:33:10.280 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 91 16 153594 153578 105.47 0 - 0.00839412 2024-04-23T16:33:10.802 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:10.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:10.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:10.907 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:33:10.940 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:11.035 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:33:11.035 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:33:11.036 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:33:11.036 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:33:11.037 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:33:11.039 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:33:11.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:11.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:33:11.059 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:33:11.079 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:33:11.100 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:33:11.121 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:33:11.140 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:33:11.141 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:33:11.161 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:33:11.161 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:33:11.161 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:33:11.161 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:33:11.161 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:33:11.161 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:33:11.161 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:33:11.161 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:33:11.162 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:33:11.162 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:33:11.162 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:33:11.162 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:33:11.162 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:33:11.168 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:11.280 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 92 16 153594 153578 104.323 0 - 0.00839412 2024-04-23T16:33:11.326 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:33:11.326 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:33:11.326 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2024-04-23T16:33:11.326 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:33:11.419 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:33:11.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:11.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:33:11.523 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:33:11.556 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:11.577 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:33:11.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:11.580 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:33:11.681 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:11.712 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:11.731 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:11.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:11.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:11.836 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:33:11.865 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:11 smithi188 ceph-mon[44101]: pgmap v1098: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 1.1 KiB/s rd, 4.9 MiB/s wr, 81 op/s; 28953/307160 objects degraded (9.426%); 21 KiB/s, 0 objects/s recovering 2024-04-23T16:33:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:11 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3951185029' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:33:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:11.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.013 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:11 smithi148 ceph-mon[44520]: pgmap v1098: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 1.1 KiB/s rd, 4.9 MiB/s wr, 81 op/s; 28953/307160 objects degraded (9.426%); 21 KiB/s, 0 objects/s recovering 2024-04-23T16:33:12.013 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:11 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3951185029' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:33:12.013 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.013 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.014 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.014 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.014 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.014 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.014 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.014 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.014 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.022 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:11 smithi103 ceph-mon[39629]: pgmap v1098: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 1.1 KiB/s rd, 4.9 MiB/s wr, 81 op/s; 28953/307160 objects degraded (9.426%); 21 KiB/s, 0 objects/s recovering 2024-04-23T16:33:12.022 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:11 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3951185029' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:33:12.022 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.022 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.022 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.022 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.039 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-23T16:33:12.222 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:33:12.280 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 93 16 153594 153578 103.202 0 - 0.00839412 2024-04-23T16:33:12.298 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:33:12.612 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:33:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:12.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:33:12.640 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:33:12.641 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:33:12.718 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:33:12.745 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:12.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.942 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.942 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:12.943 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:33:12.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:12.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:33:13.048 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:33:13.077 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:13.078 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:13.080 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:13.101 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:33:13.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:13.104 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:33:13.205 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:33:13.234 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:33:13.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:13.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:33:13.281 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 94 16 153594 153578 102.104 0 - 0.00839412 2024-04-23T16:33:13.356 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:33:13.388 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:13.410 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:33:13.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:13.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:33:13.514 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:33:13.542 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:13.563 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:33:13.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:13.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:33:13.666 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:13.696 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:13.772 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:33:13.793 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:13.796 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:13 smithi103 ceph-mon[39629]: pgmap v1099: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 1023 B/s rd, 4.4 MiB/s wr, 72 op/s; 28953/307160 objects degraded (9.426%); 19 KiB/s, 0 objects/s recovering 2024-04-23T16:33:13.796 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:13 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3154996422' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:33:13.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:13.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:13.899 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:13.927 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:13.980 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:13.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:13.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:14.084 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:33:14.114 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:33:14.114 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:33:14.114 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2024-04-23T16:33:14.114 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2024-04-23T16:33:14.114 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:33:14.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:13 smithi188 ceph-mon[44101]: pgmap v1099: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 1023 B/s rd, 4.4 MiB/s wr, 72 op/s; 28953/307160 objects degraded (9.426%); 19 KiB/s, 0 objects/s recovering 2024-04-23T16:33:14.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:13 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3154996422' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:33:14.116 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:14.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:13 smithi148 ceph-mon[44520]: pgmap v1099: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 1023 B/s rd, 4.4 MiB/s wr, 72 op/s; 28953/307160 objects degraded (9.426%); 19 KiB/s, 0 objects/s recovering 2024-04-23T16:33:14.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:13 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3154996422' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:33:14.138 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:33:14.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:14.142 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:33:14.243 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:33:14.271 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:14.281 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 95 16 153594 153578 101.029 0 - 0.00839412 2024-04-23T16:33:14.520 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:33:14.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:14.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:33:14.624 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:14.642 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:33:14.652 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:14.776 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:14.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:14.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:14.782 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:14 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:33:14.783 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:14 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3154996422' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:33:14.783 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:14 smithi103 ceph-mon[39629]: osdmap e391: 12 total, 11 up, 10 in 2024-04-23T16:33:14.802 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-23T16:33:14.879 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:14.912 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:14.932 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:14.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:14.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:15.035 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:15.061 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:14 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:33:15.061 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:14 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3154996422' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:33:15.061 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:14 smithi148 ceph-mon[44520]: osdmap e391: 12 total, 11 up, 10 in 2024-04-23T16:33:15.063 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:15.083 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:15.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:15.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:15.111 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:33:15.187 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:15.215 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:14 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:33:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:14 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3154996422' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:33:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:14 smithi188 ceph-mon[44101]: osdmap e391: 12 total, 11 up, 10 in 2024-04-23T16:33:15.281 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 96 16 153594 153578 99.9764 0 - 0.00839412 2024-04-23T16:33:15.363 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:15.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:15.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:15.466 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:15.492 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:15.943 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:15.943 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:15 smithi148 ceph-mon[44520]: pgmap v1101: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:15.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:15.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:16.045 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:16.074 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:15 smithi103 ceph-mon[39629]: pgmap v1101: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:16.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:15 smithi188 ceph-mon[44101]: pgmap v1101: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:16.239 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:33:16.281 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 97 16 153594 153578 98.9456 0 - 0.00839412 2024-04-23T16:33:16.334 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2 is already down. 2024-04-23T16:33:16.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:16 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T16:33:16.329+0000 7f135b1c7700 -1 mon.a@0(leader).osd e391 definitely_dead 0 2024-04-23T16:33:16.623 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:16.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:16.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:16.727 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:33:16.758 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:16.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:16 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4035413517' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["2"]}]: dispatch 2024-04-23T16:33:16.893 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:33:16.893 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2024-04-23T16:33:16.893 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2024-04-23T16:33:16.893 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:33:16.893 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.893 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:283055 3.3 3:c80795a4:::benchmark_data_smithi188_89276_object114411:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:33:16.893 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:09.381356+0000", 2024-04-23T16:33:16.894 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 427.508190512, 2024-04-23T16:33:16.894 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.22286794500000001, 2024-04-23T16:33:16.894 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:16.894 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:16.894 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:16.894 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:16.894 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:33:16.894 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:33:16.894 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 283055 2024-04-23T16:33:16.894 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.894 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:16.894 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.894 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:16.895 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381356+0000", 2024-04-23T16:33:16.895 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.895 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.895 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.895 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:16.895 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381356+0000", 2024-04-23T16:33:16.895 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.895 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.895 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.895 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:16.895 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381358+0000", 2024-04-23T16:33:16.896 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9300000000000002e-06 2024-04-23T16:33:16.896 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.896 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.896 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:16.896 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381364+0000", 2024-04-23T16:33:16.896 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2099999999999998e-06 2024-04-23T16:33:16.896 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.896 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.896 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:16.896 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381375+0000", 2024-04-23T16:33:16.896 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1060000000000001e-05 2024-04-23T16:33:16.896 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.896 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.896 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:16.897 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381382+0000", 2024-04-23T16:33:16.897 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5860000000000001e-06 2024-04-23T16:33:16.897 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.897 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.897 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:16.897 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381435+0000", 2024-04-23T16:33:16.897 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2787000000000001e-05 2024-04-23T16:33:16.897 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.897 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.897 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:16.897 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381576+0000", 2024-04-23T16:33:16.897 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000141256 2024-04-23T16:33:16.897 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.898 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.898 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:33:16.898 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381695+0000", 2024-04-23T16:33:16.898 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000119413 2024-04-23T16:33:16.898 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.898 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.898 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:16.898 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.396183+0000", 2024-04-23T16:33:16.898 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014487451 2024-04-23T16:33:16.898 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.898 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.898 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:16.898 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.604137+0000", 2024-04-23T16:33:16.899 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20795429000000001 2024-04-23T16:33:16.899 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.899 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.899 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:16.899 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.604181+0000", 2024-04-23T16:33:16.899 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3563000000000003e-05 2024-04-23T16:33:16.899 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.899 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.899 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:16.899 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.604224+0000", 2024-04-23T16:33:16.899 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3399000000000002e-05 2024-04-23T16:33:16.899 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.899 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:16.899 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.900 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.900 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.900 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:283057 3.3 3:cdfb5812:::benchmark_data_smithi188_89276_object114413:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:33:16.900 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:09.381405+0000", 2024-04-23T16:33:16.900 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 427.50814205299997, 2024-04-23T16:33:16.900 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.22294652200000001, 2024-04-23T16:33:16.900 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:16.900 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:16.900 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:16.900 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:16.900 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:33:16.900 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:33:16.900 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 283057 2024-04-23T16:33:16.901 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.901 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:16.901 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.901 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:16.901 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381405+0000", 2024-04-23T16:33:16.901 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.901 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.901 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.901 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:16.901 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381405+0000", 2024-04-23T16:33:16.901 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.901 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.901 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.901 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:16.902 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381405+0000", 2024-04-23T16:33:16.902 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9100000000000004e-07 2024-04-23T16:33:16.902 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.902 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.902 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:16.902 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381408+0000", 2024-04-23T16:33:16.902 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.695e-06 2024-04-23T16:33:16.902 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.902 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.902 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:16.902 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381414+0000", 2024-04-23T16:33:16.902 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8189999999999997e-06 2024-04-23T16:33:16.902 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.903 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.903 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:16.903 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381418+0000", 2024-04-23T16:33:16.903 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7680000000000002e-06 2024-04-23T16:33:16.903 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.903 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.903 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:16.903 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.386409+0000", 2024-04-23T16:33:16.903 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0049914310000000002 2024-04-23T16:33:16.903 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.903 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.903 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:16.903 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.386516+0000", 2024-04-23T16:33:16.903 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010747300000000001 2024-04-23T16:33:16.903 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.904 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.904 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:33:16.904 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.386620+0000", 2024-04-23T16:33:16.904 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010332600000000001 2024-04-23T16:33:16.904 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.904 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.904 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:16.904 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.396194+0000", 2024-04-23T16:33:16.904 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.009573781 2024-04-23T16:33:16.904 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.904 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.904 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:16.904 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.604283+0000", 2024-04-23T16:33:16.904 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20808908300000001 2024-04-23T16:33:16.905 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.905 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.905 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:16.905 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.604314+0000", 2024-04-23T16:33:16.905 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0997000000000002e-05 2024-04-23T16:33:16.905 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.905 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.905 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:16.905 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.604351+0000", 2024-04-23T16:33:16.905 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7558000000000002e-05 2024-04-23T16:33:16.905 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.905 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:16.905 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.906 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.906 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.906 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:283059 3.3 3:cd8d2e79:::benchmark_data_smithi188_89276_object114415:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:33:16.906 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:09.381492+0000", 2024-04-23T16:33:16.906 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 427.50805517100002, 2024-04-23T16:33:16.906 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.22299651700000001, 2024-04-23T16:33:16.906 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:16.906 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:16.906 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:16.906 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:16.906 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:33:16.907 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:33:16.907 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 283059 2024-04-23T16:33:16.907 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.907 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:16.907 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.907 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:16.907 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381492+0000", 2024-04-23T16:33:16.907 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.908 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.908 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.908 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:16.908 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381492+0000", 2024-04-23T16:33:16.908 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.908 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.908 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.908 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:16.908 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381493+0000", 2024-04-23T16:33:16.908 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.136e-06 2024-04-23T16:33:16.908 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.908 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.908 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:16.908 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381497+0000", 2024-04-23T16:33:16.909 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9199999999999997e-06 2024-04-23T16:33:16.909 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.909 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.909 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:16.909 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381505+0000", 2024-04-23T16:33:16.909 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.011e-06 2024-04-23T16:33:16.909 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.909 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.909 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:16.909 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.381513+0000", 2024-04-23T16:33:16.909 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4810000000000006e-06 2024-04-23T16:33:16.909 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.909 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.909 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:16.910 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.391290+0000", 2024-04-23T16:33:16.910 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0097767040000000006 2024-04-23T16:33:16.910 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.910 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.910 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:16.910 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.391406+0000", 2024-04-23T16:33:16.910 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000116353 2024-04-23T16:33:16.910 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.910 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.910 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:33:16.910 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.391528+0000", 2024-04-23T16:33:16.910 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012218400000000001 2024-04-23T16:33:16.910 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.911 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.911 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:16.911 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.396347+0000", 2024-04-23T16:33:16.911 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0048190960000000001 2024-04-23T16:33:16.911 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.911 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.911 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:16.911 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.604392+0000", 2024-04-23T16:33:16.911 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20804426300000001 2024-04-23T16:33:16.911 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.911 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.911 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:16.911 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.604441+0000", 2024-04-23T16:33:16.911 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8931000000000003e-05 2024-04-23T16:33:16.911 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.912 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.912 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:16.912 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.604488+0000", 2024-04-23T16:33:16.912 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7438000000000002e-05 2024-04-23T16:33:16.912 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.912 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:16.912 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.912 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.912 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.912 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:283081 3.3 3:ca861c37:::benchmark_data_smithi188_89276_object114437:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:33:16.912 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:09.405748+0000", 2024-04-23T16:33:16.912 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 427.48379872999999, 2024-04-23T16:33:16.912 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20958642199999999, 2024-04-23T16:33:16.913 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:16.913 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:16.913 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:16.913 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:16.913 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:33:16.913 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:33:16.913 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 283081 2024-04-23T16:33:16.913 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.913 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:16.913 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.913 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:16.913 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405748+0000", 2024-04-23T16:33:16.913 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.913 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.913 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.914 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:16.914 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405748+0000", 2024-04-23T16:33:16.914 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.914 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.914 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.914 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:16.914 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405750+0000", 2024-04-23T16:33:16.914 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2259999999999999e-06 2024-04-23T16:33:16.914 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.914 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.914 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:16.914 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405756+0000", 2024-04-23T16:33:16.914 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0519999999999998e-06 2024-04-23T16:33:16.915 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.915 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.915 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:16.915 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405768+0000", 2024-04-23T16:33:16.915 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.188e-05 2024-04-23T16:33:16.915 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.915 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.915 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:16.915 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405776+0000", 2024-04-23T16:33:16.915 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0250000000000002e-06 2024-04-23T16:33:16.915 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.915 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.915 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:16.915 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405837+0000", 2024-04-23T16:33:16.916 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1134000000000001e-05 2024-04-23T16:33:16.916 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.916 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.916 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:16.916 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405938+0000", 2024-04-23T16:33:16.916 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000100917 2024-04-23T16:33:16.916 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.916 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.916 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:33:16.916 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.406015+0000", 2024-04-23T16:33:16.916 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6728000000000001e-05 2024-04-23T16:33:16.916 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.916 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.916 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:16.917 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.413731+0000", 2024-04-23T16:33:16.917 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0077160579999999996 2024-04-23T16:33:16.917 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.917 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.917 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:16.917 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.615248+0000", 2024-04-23T16:33:16.917 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.201516636 2024-04-23T16:33:16.917 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.917 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.917 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:16.917 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.615286+0000", 2024-04-23T16:33:16.917 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8609999999999998e-05 2024-04-23T16:33:16.917 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.917 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.918 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:16.918 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.615334+0000", 2024-04-23T16:33:16.918 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8155999999999998e-05 2024-04-23T16:33:16.918 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.918 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:16.918 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.918 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.918 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.918 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:283082 3.3 3:c2076386:::benchmark_data_smithi188_89276_object114438:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:33:16.918 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:09.405795+0000", 2024-04-23T16:33:16.918 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 427.48375178399999, 2024-04-23T16:33:16.918 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.214633718, 2024-04-23T16:33:16.918 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:16.918 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:16.919 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:16.919 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:16.919 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:33:16.919 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:33:16.919 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 283082 2024-04-23T16:33:16.919 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.919 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:16.919 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.919 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:16.919 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405795+0000", 2024-04-23T16:33:16.919 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.919 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.919 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.919 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:16.920 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405795+0000", 2024-04-23T16:33:16.920 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.920 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.920 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.920 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:16.920 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405796+0000", 2024-04-23T16:33:16.920 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6899999999999996e-07 2024-04-23T16:33:16.920 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.920 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.920 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:16.920 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405798+0000", 2024-04-23T16:33:16.920 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7099999999999999e-06 2024-04-23T16:33:16.920 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.920 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.921 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:16.921 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405811+0000", 2024-04-23T16:33:16.921 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2969000000000001e-05 2024-04-23T16:33:16.921 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.921 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.921 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:16.921 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.405817+0000", 2024-04-23T16:33:16.921 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6230000000000002e-06 2024-04-23T16:33:16.921 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.921 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.921 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:16.921 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.409560+0000", 2024-04-23T16:33:16.922 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0037428079999999998 2024-04-23T16:33:16.922 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.922 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.922 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:16.922 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.409651+0000", 2024-04-23T16:33:16.922 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1077999999999998e-05 2024-04-23T16:33:16.922 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.922 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.922 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:33:16.922 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.409728+0000", 2024-04-23T16:33:16.922 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6837999999999999e-05 2024-04-23T16:33:16.922 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.923 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.923 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:16.923 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.421862+0000", 2024-04-23T16:33:16.923 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01213412 2024-04-23T16:33:16.923 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.923 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.923 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:16.923 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.620338+0000", 2024-04-23T16:33:16.923 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19847571999999999 2024-04-23T16:33:16.923 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.923 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.923 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:16.923 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.620388+0000", 2024-04-23T16:33:16.923 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0874000000000001e-05 2024-04-23T16:33:16.924 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.924 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.924 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:16.924 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.620429+0000", 2024-04-23T16:33:16.924 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0209000000000001e-05 2024-04-23T16:33:16.924 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.924 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:16.924 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.924 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.924 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.924 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:283085 3.3 3:c5fb6a0c:::benchmark_data_smithi188_89276_object114441:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:33:16.924 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:09.409895+0000", 2024-04-23T16:33:16.924 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 427.47965159900002, 2024-04-23T16:33:16.924 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.21995653700000001, 2024-04-23T16:33:16.925 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:16.925 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:16.925 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:16.925 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:16.925 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:33:16.925 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:33:16.925 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 283085 2024-04-23T16:33:16.925 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.925 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:16.925 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.925 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:16.925 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.409895+0000", 2024-04-23T16:33:16.925 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.925 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.926 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.926 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:16.926 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.409895+0000", 2024-04-23T16:33:16.926 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.926 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.926 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.926 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:16.926 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.409897+0000", 2024-04-23T16:33:16.926 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.666e-06 2024-04-23T16:33:16.926 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.926 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.926 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:16.926 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.409902+0000", 2024-04-23T16:33:16.927 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4909999999999996e-06 2024-04-23T16:33:16.927 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.927 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.927 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:16.927 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.409913+0000", 2024-04-23T16:33:16.927 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0317e-05 2024-04-23T16:33:16.927 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.927 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.927 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:16.927 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.409918+0000", 2024-04-23T16:33:16.927 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5369999999999997e-06 2024-04-23T16:33:16.927 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.927 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.927 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:16.928 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.413761+0000", 2024-04-23T16:33:16.928 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0038424539999999999 2024-04-23T16:33:16.928 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.928 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.928 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:16.928 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.413880+0000", 2024-04-23T16:33:16.928 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011968099999999999 2024-04-23T16:33:16.928 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.928 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.928 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:33:16.928 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.413993+0000", 2024-04-23T16:33:16.928 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000112883 2024-04-23T16:33:16.928 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.928 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.929 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:16.929 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.421867+0000", 2024-04-23T16:33:16.929 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0078734059999999995 2024-04-23T16:33:16.929 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.929 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.929 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:16.929 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.629780+0000", 2024-04-23T16:33:16.929 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20791299999999999 2024-04-23T16:33:16.929 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.929 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.929 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:16.929 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.629811+0000", 2024-04-23T16:33:16.929 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1387999999999998e-05 2024-04-23T16:33:16.929 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.930 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.930 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:16.930 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.629852+0000", 2024-04-23T16:33:16.930 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0713999999999997e-05 2024-04-23T16:33:16.930 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.930 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:16.930 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.930 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.930 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.930 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:283092 3.3 3:c6d6d9fb:::benchmark_data_smithi188_89276_object114448:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:33:16.930 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:09.416225+0000", 2024-04-23T16:33:16.930 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 427.47332195899997, 2024-04-23T16:33:16.930 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.21378214000000001, 2024-04-23T16:33:16.931 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:16.931 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:16.931 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:16.931 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:16.931 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:33:16.931 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:33:16.931 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 283092 2024-04-23T16:33:16.931 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.931 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:16.931 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.931 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:16.931 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.416225+0000", 2024-04-23T16:33:16.931 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.931 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.932 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.932 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:16.932 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.416225+0000", 2024-04-23T16:33:16.932 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.932 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.932 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.932 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:16.932 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.416227+0000", 2024-04-23T16:33:16.932 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4540000000000001e-06 2024-04-23T16:33:16.932 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.932 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.932 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:16.932 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.416233+0000", 2024-04-23T16:33:16.932 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7250000000000002e-06 2024-04-23T16:33:16.932 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.933 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.933 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:16.933 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.416243+0000", 2024-04-23T16:33:16.933 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0395e-05 2024-04-23T16:33:16.933 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.933 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.933 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:16.933 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.416249+0000", 2024-04-23T16:33:16.933 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7470000000000002e-06 2024-04-23T16:33:16.933 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.933 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.933 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:16.933 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.418349+0000", 2024-04-23T16:33:16.933 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0020998269999999999 2024-04-23T16:33:16.934 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.934 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.934 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:16.934 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.418427+0000", 2024-04-23T16:33:16.934 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7626999999999994e-05 2024-04-23T16:33:16.934 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.934 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.934 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:33:16.934 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.418503+0000", 2024-04-23T16:33:16.934 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6625999999999994e-05 2024-04-23T16:33:16.934 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.934 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.934 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:16.934 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.422039+0000", 2024-04-23T16:33:16.935 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0035362990000000001 2024-04-23T16:33:16.935 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.935 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.935 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:16.935 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.629907+0000", 2024-04-23T16:33:16.935 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20786766100000001 2024-04-23T16:33:16.935 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.935 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.935 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:16.935 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.629954+0000", 2024-04-23T16:33:16.935 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7329e-05 2024-04-23T16:33:16.935 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.935 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.935 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:16.935 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.630007+0000", 2024-04-23T16:33:16.936 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2450000000000001e-05 2024-04-23T16:33:16.936 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.936 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:16.936 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.936 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.936 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.936 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:283111 3.3 3:ccaa2632:::benchmark_data_smithi188_89276_object114467:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:33:16.936 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:09.445340+0000", 2024-04-23T16:33:16.936 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 427.44420664199998, 2024-04-23T16:33:16.936 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.18482868599999999, 2024-04-23T16:33:16.936 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:16.936 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:16.936 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:16.937 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:16.937 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:33:16.937 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:33:16.937 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 283111 2024-04-23T16:33:16.937 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.937 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:16.937 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.937 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:16.937 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.445340+0000", 2024-04-23T16:33:16.937 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.937 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.937 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.937 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:16.937 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.445340+0000", 2024-04-23T16:33:16.938 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.938 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.938 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.938 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:16.938 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.445344+0000", 2024-04-23T16:33:16.938 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7730000000000001e-06 2024-04-23T16:33:16.938 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.938 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.938 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:16.938 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.445356+0000", 2024-04-23T16:33:16.938 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1943000000000001e-05 2024-04-23T16:33:16.938 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.938 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.938 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:16.939 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.445382+0000", 2024-04-23T16:33:16.939 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6063000000000001e-05 2024-04-23T16:33:16.939 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.939 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.939 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:16.939 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.445397+0000", 2024-04-23T16:33:16.939 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.486e-05 2024-04-23T16:33:16.939 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.939 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.939 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:16.939 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.445514+0000", 2024-04-23T16:33:16.939 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011703 2024-04-23T16:33:16.939 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.939 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.940 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:16.940 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.445600+0000", 2024-04-23T16:33:16.940 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6274999999999996e-05 2024-04-23T16:33:16.940 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.940 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.940 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:33:16.940 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.445677+0000", 2024-04-23T16:33:16.940 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6563999999999994e-05 2024-04-23T16:33:16.940 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.940 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.940 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:16.940 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.449211+0000", 2024-04-23T16:33:16.940 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0035343829999999999 2024-04-23T16:33:16.940 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.941 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.941 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:16.941 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.630075+0000", 2024-04-23T16:33:16.941 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.18086406799999999 2024-04-23T16:33:16.941 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.941 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.941 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:16.941 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.630125+0000", 2024-04-23T16:33:16.941 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0098e-05 2024-04-23T16:33:16.941 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.941 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.941 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:16.941 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.630169+0000", 2024-04-23T16:33:16.942 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3628999999999999e-05 2024-04-23T16:33:16.942 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.942 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:16.942 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.942 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.942 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.942 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:283115 3.3 3:c783fede:::benchmark_data_smithi188_89276_object114471:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:33:16.942 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:09.462085+0000", 2024-04-23T16:33:16.942 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 427.42746206300001, 2024-04-23T16:33:16.942 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.175558731, 2024-04-23T16:33:16.942 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:16.942 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:16.942 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:16.942 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:16.943 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:33:16.943 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:33:16.943 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 283115 2024-04-23T16:33:16.943 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.943 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:16.943 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.943 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:16.943 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.462085+0000", 2024-04-23T16:33:16.943 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.943 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.943 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.943 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:16.943 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.462085+0000", 2024-04-23T16:33:16.943 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.944 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.944 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.944 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:16.944 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.462086+0000", 2024-04-23T16:33:16.944 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.463e-06 2024-04-23T16:33:16.944 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.944 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.944 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:16.944 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.462091+0000", 2024-04-23T16:33:16.944 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5549999999999996e-06 2024-04-23T16:33:16.944 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.944 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.944 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:16.944 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.462100+0000", 2024-04-23T16:33:16.945 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.329e-06 2024-04-23T16:33:16.945 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.945 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.945 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:16.945 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.462122+0000", 2024-04-23T16:33:16.945 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2436999999999999e-05 2024-04-23T16:33:16.945 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.945 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.945 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:16.945 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.462196+0000", 2024-04-23T16:33:16.945 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3334e-05 2024-04-23T16:33:16.945 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.945 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.945 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:16.945 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.462273+0000", 2024-04-23T16:33:16.946 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7674000000000002e-05 2024-04-23T16:33:16.946 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.946 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.946 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:33:16.946 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.462343+0000", 2024-04-23T16:33:16.946 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9408999999999999e-05 2024-04-23T16:33:16.946 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.946 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.946 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:16.946 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.464641+0000", 2024-04-23T16:33:16.946 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002298394 2024-04-23T16:33:16.946 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.947 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.947 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:16.947 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.637545+0000", 2024-04-23T16:33:16.947 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.172903328 2024-04-23T16:33:16.947 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.947 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.947 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:16.947 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.637598+0000", 2024-04-23T16:33:16.947 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3278e-05 2024-04-23T16:33:16.947 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.947 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.947 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:16.947 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:09.637643+0000", 2024-04-23T16:33:16.947 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5529999999999999e-05 2024-04-23T16:33:16.947 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.948 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:16.948 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.948 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.948 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.948 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12658 4.d 4:be7ebf30:::benchmark_data_smithi188_101370_object12657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:33:16.948 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.230769+0000", 2024-04-23T16:33:16.948 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 356.65877768799999, 2024-04-23T16:33:16.948 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.204651206, 2024-04-23T16:33:16.948 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:16.948 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:16.948 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:16.948 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:16.948 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:33:16.948 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:33:16.949 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12658 2024-04-23T16:33:16.949 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.949 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:16.949 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.949 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:16.949 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.230769+0000", 2024-04-23T16:33:16.949 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.949 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.949 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.949 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:16.949 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.230769+0000", 2024-04-23T16:33:16.949 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.949 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.949 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.950 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:16.950 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.230770+0000", 2024-04-23T16:33:16.950 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.133e-06 2024-04-23T16:33:16.950 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.950 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.950 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:16.950 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.230846+0000", 2024-04-23T16:33:16.950 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5947999999999999e-05 2024-04-23T16:33:16.950 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.950 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.950 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:16.950 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.230853+0000", 2024-04-23T16:33:16.950 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7299999999999999e-06 2024-04-23T16:33:16.951 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.951 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.951 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:16.951 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.230858+0000", 2024-04-23T16:33:16.951 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3940000000000004e-06 2024-04-23T16:33:16.951 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.951 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.951 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:16.951 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.230908+0000", 2024-04-23T16:33:16.951 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9320999999999998e-05 2024-04-23T16:33:16.951 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.951 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.951 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:16.951 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.231012+0000", 2024-04-23T16:33:16.952 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000104196 2024-04-23T16:33:16.952 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.952 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.952 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:33:16.952 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.231122+0000", 2024-04-23T16:33:16.952 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000110728 2024-04-23T16:33:16.952 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.952 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.952 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:16.952 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.232467+0000", 2024-04-23T16:33:16.952 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0013445950000000001 2024-04-23T16:33:16.952 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.952 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.952 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:16.953 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.435342+0000", 2024-04-23T16:33:16.953 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20287460399999999 2024-04-23T16:33:16.953 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.953 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.953 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:16.953 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.435384+0000", 2024-04-23T16:33:16.953 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1915000000000001e-05 2024-04-23T16:33:16.953 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.953 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.953 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:16.953 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.435420+0000", 2024-04-23T16:33:16.953 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6641999999999999e-05 2024-04-23T16:33:16.953 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.954 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:16.954 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.954 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.954 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.954 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12698 4.d 4:bce91761:::benchmark_data_smithi188_101370_object12697:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:33:16.954 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.238200+0000", 2024-04-23T16:33:16.954 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 356.65134664700003, 2024-04-23T16:33:16.954 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19737084899999999, 2024-04-23T16:33:16.954 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:16.954 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:16.954 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:16.954 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:16.954 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:33:16.954 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:33:16.955 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12698 2024-04-23T16:33:16.955 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.955 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:16.955 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.955 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:16.955 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.238200+0000", 2024-04-23T16:33:16.955 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.955 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.955 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.955 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:16.955 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.238200+0000", 2024-04-23T16:33:16.955 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.955 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.956 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.956 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:16.956 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.238202+0000", 2024-04-23T16:33:16.956 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9910000000000001e-06 2024-04-23T16:33:16.956 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.956 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.956 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:16.956 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.238232+0000", 2024-04-23T16:33:16.956 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9935999999999999e-05 2024-04-23T16:33:16.956 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.956 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.956 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:16.956 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.238243+0000", 2024-04-23T16:33:16.956 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0747000000000001e-05 2024-04-23T16:33:16.957 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.957 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.957 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:16.957 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.238248+0000", 2024-04-23T16:33:16.957 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.716e-06 2024-04-23T16:33:16.957 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.957 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.957 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:16.957 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.238296+0000", 2024-04-23T16:33:16.957 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7821000000000002e-05 2024-04-23T16:33:16.957 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.957 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.957 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:16.957 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.238394+0000", 2024-04-23T16:33:16.958 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7775999999999994e-05 2024-04-23T16:33:16.958 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.958 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.958 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:33:16.958 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.238499+0000", 2024-04-23T16:33:16.958 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000104931 2024-04-23T16:33:16.958 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.958 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.958 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:16.958 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.239359+0000", 2024-04-23T16:33:16.958 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00086016700000000001 2024-04-23T16:33:16.958 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.958 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.958 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:16.959 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.435482+0000", 2024-04-23T16:33:16.959 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19612264500000001 2024-04-23T16:33:16.959 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.959 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.959 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:16.959 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.435542+0000", 2024-04-23T16:33:16.959 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0174999999999999e-05 2024-04-23T16:33:16.959 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.959 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.959 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:16.959 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.435571+0000", 2024-04-23T16:33:16.959 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8943999999999999e-05 2024-04-23T16:33:16.959 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.959 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:16.960 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.960 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.960 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.960 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12758 4.d 4:b109748d:::benchmark_data_smithi188_101370_object12757:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:33:16.960 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.251139+0000", 2024-04-23T16:33:16.960 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 356.63840734299998, 2024-04-23T16:33:16.960 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.185709438, 2024-04-23T16:33:16.960 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:16.960 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:16.960 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:16.960 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:16.960 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:33:16.960 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:33:16.960 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12758 2024-04-23T16:33:16.961 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.961 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:16.961 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.961 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:16.961 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251139+0000", 2024-04-23T16:33:16.961 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.961 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.961 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.961 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:16.961 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251139+0000", 2024-04-23T16:33:16.961 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.961 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.961 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.961 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:16.962 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251141+0000", 2024-04-23T16:33:16.962 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.951e-06 2024-04-23T16:33:16.962 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.962 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.962 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:16.962 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251179+0000", 2024-04-23T16:33:16.962 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7208e-05 2024-04-23T16:33:16.962 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.962 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.962 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:16.962 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251189+0000", 2024-04-23T16:33:16.962 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0556e-05 2024-04-23T16:33:16.963 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.963 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.963 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:16.963 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251197+0000", 2024-04-23T16:33:16.963 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7880000000000007e-06 2024-04-23T16:33:16.963 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.963 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.963 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:16.963 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251246+0000", 2024-04-23T16:33:16.963 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9379000000000003e-05 2024-04-23T16:33:16.963 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.964 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:16.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251329+0000", 2024-04-23T16:33:16.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2973000000000007e-05 2024-04-23T16:33:16.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.964 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:33:16.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251466+0000", 2024-04-23T16:33:16.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000137112 2024-04-23T16:33:16.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.964 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:16.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.252783+0000", 2024-04-23T16:33:16.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001316371 2024-04-23T16:33:16.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.965 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:16.965 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.436792+0000", 2024-04-23T16:33:16.965 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.18400925300000001 2024-04-23T16:33:16.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.965 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:16.965 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.436824+0000", 2024-04-23T16:33:16.965 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2264999999999997e-05 2024-04-23T16:33:16.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.965 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:16.965 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.436849+0000", 2024-04-23T16:33:16.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4581999999999999e-05 2024-04-23T16:33:16.966 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.966 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:16.966 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.966 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12759 4.d 4:bdc675b3:::benchmark_data_smithi188_101370_object12758:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:33:16.966 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.251342+0000", 2024-04-23T16:33:16.966 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 356.63820512799998, 2024-04-23T16:33:16.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.18565109299999999, 2024-04-23T16:33:16.966 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:16.966 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:16.966 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:16.967 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:16.967 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:33:16.967 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:33:16.967 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12759 2024-04-23T16:33:16.967 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.967 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:16.967 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.967 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:16.967 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251342+0000", 2024-04-23T16:33:16.967 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.967 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.967 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.967 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:16.967 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251342+0000", 2024-04-23T16:33:16.968 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.968 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.968 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:16.968 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251343+0000", 2024-04-23T16:33:16.968 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2529999999999999e-06 2024-04-23T16:33:16.968 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.968 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:16.968 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251484+0000", 2024-04-23T16:33:16.968 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014138099999999999 2024-04-23T16:33:16.968 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.968 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:16.969 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251492+0000", 2024-04-23T16:33:16.969 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2530000000000004e-06 2024-04-23T16:33:16.969 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.969 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.969 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:16.969 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251500+0000", 2024-04-23T16:33:16.969 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0940000000000002e-06 2024-04-23T16:33:16.969 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.969 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.969 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:16.969 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.252084+0000", 2024-04-23T16:33:16.969 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00058474700000000004 2024-04-23T16:33:16.969 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.969 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:16.970 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.252195+0000", 2024-04-23T16:33:16.970 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011098 2024-04-23T16:33:16.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:33:16.970 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.252335+0000", 2024-04-23T16:33:16.970 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013999600000000001 2024-04-23T16:33:16.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:16.970 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.252972+0000", 2024-04-23T16:33:16.970 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00063641100000000003 2024-04-23T16:33:16.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:16.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.436896+0000", 2024-04-23T16:33:16.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.18392444599999999 2024-04-23T16:33:16.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:16.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.436950+0000", 2024-04-23T16:33:16.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4018999999999999e-05 2024-04-23T16:33:16.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:16.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.436993+0000", 2024-04-23T16:33:16.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2512999999999998e-05 2024-04-23T16:33:16.972 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.972 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:16.972 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.972 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12789 4.d 4:ba9dcfb2:::benchmark_data_smithi188_101370_object12788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:33:16.972 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.257550+0000", 2024-04-23T16:33:16.972 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 356.63199674100002, 2024-04-23T16:33:16.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.17956171100000001, 2024-04-23T16:33:16.972 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:16.972 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:16.972 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:16.972 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:16.972 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:33:16.973 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:33:16.973 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12789 2024-04-23T16:33:16.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.973 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:16.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.973 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:16.973 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.257550+0000", 2024-04-23T16:33:16.973 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.973 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:16.973 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.257550+0000", 2024-04-23T16:33:16.973 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:16.974 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.257551+0000", 2024-04-23T16:33:16.974 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.24e-06 2024-04-23T16:33:16.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:16.974 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.257612+0000", 2024-04-23T16:33:16.974 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0853999999999997e-05 2024-04-23T16:33:16.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:16.974 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.257621+0000", 2024-04-23T16:33:16.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1770000000000004e-06 2024-04-23T16:33:16.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.975 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:16.975 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.257627+0000", 2024-04-23T16:33:16.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0560000000000001e-06 2024-04-23T16:33:16.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.975 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:16.975 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.257667+0000", 2024-04-23T16:33:16.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9279999999999997e-05 2024-04-23T16:33:16.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.975 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:16.975 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.257774+0000", 2024-04-23T16:33:16.976 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107816 2024-04-23T16:33:16.976 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:33:16.976 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.257890+0000", 2024-04-23T16:33:16.976 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011557000000000001 2024-04-23T16:33:16.976 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:16.976 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.259226+0000", 2024-04-23T16:33:16.976 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0013363769999999999 2024-04-23T16:33:16.976 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:16.977 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437047+0000", 2024-04-23T16:33:16.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.17782076899999999 2024-04-23T16:33:16.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.977 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:16.977 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437079+0000", 2024-04-23T16:33:16.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1715999999999999e-05 2024-04-23T16:33:16.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.977 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:16.977 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437112+0000", 2024-04-23T16:33:16.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2855999999999998e-05 2024-04-23T16:33:16.977 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.977 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:16.978 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.978 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.978 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12849 4.d 4:b91fa107:::benchmark_data_smithi188_101370_object12848:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:33:16.978 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.271006+0000", 2024-04-23T16:33:16.978 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 356.61854026200001, 2024-04-23T16:33:16.978 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.16626290699999999, 2024-04-23T16:33:16.978 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:16.978 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:16.978 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:16.978 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:16.978 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:33:16.978 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:33:16.978 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12849 2024-04-23T16:33:16.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.979 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:16.979 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.979 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:16.979 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.271006+0000", 2024-04-23T16:33:16.979 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.979 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.979 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:16.979 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.271006+0000", 2024-04-23T16:33:16.979 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.980 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.980 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:16.980 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.271008+0000", 2024-04-23T16:33:16.980 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6249999999999999e-06 2024-04-23T16:33:16.980 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.980 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.980 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:16.980 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.271058+0000", 2024-04-23T16:33:16.980 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0324000000000001e-05 2024-04-23T16:33:16.980 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.980 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.980 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:16.980 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.271070+0000", 2024-04-23T16:33:16.980 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2112e-05 2024-04-23T16:33:16.981 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.981 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.981 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:16.981 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.271078+0000", 2024-04-23T16:33:16.981 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.9899999999999997e-06 2024-04-23T16:33:16.981 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.981 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.981 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:16.981 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.271124+0000", 2024-04-23T16:33:16.981 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5089000000000001e-05 2024-04-23T16:33:16.981 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.982 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.982 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:16.982 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.271208+0000", 2024-04-23T16:33:16.983 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4578000000000002e-05 2024-04-23T16:33:16.983 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.983 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.983 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:33:16.983 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.271422+0000", 2024-04-23T16:33:16.983 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000213463 2024-04-23T16:33:16.983 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.983 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.983 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:16.983 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.272355+0000", 2024-04-23T16:33:16.983 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00093360399999999999 2024-04-23T16:33:16.983 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.984 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:16.984 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437172+0000", 2024-04-23T16:33:16.984 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.164816616 2024-04-23T16:33:16.984 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.984 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:16.984 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437228+0000", 2024-04-23T16:33:16.984 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6047000000000001e-05 2024-04-23T16:33:16.984 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.984 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:16.984 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437269+0000", 2024-04-23T16:33:16.985 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1458999999999997e-05 2024-04-23T16:33:16.985 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.985 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:16.985 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.985 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.985 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.985 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12862 4.d 4:b829ec64:::benchmark_data_smithi188_101370_object12861:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:33:16.985 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.273139+0000", 2024-04-23T16:33:16.985 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 356.61640725799998, 2024-04-23T16:33:16.985 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.16464094500000001, 2024-04-23T16:33:16.985 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:16.985 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:16.985 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:16.985 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:16.986 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:33:16.986 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:33:16.986 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12862 2024-04-23T16:33:16.986 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.986 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:16.986 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.986 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:16.986 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.273139+0000", 2024-04-23T16:33:16.986 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.986 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.986 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.986 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:16.986 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.273139+0000", 2024-04-23T16:33:16.986 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.987 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:16.987 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.273141+0000", 2024-04-23T16:33:16.987 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6920000000000001e-06 2024-04-23T16:33:16.987 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:16.987 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.273241+0000", 2024-04-23T16:33:16.987 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000100157 2024-04-23T16:33:16.987 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:16.987 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.273252+0000", 2024-04-23T16:33:16.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1012000000000001e-05 2024-04-23T16:33:16.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.988 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:16.988 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.273260+0000", 2024-04-23T16:33:16.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.028e-06 2024-04-23T16:33:16.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.988 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:16.988 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.273309+0000", 2024-04-23T16:33:16.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8310000000000003e-05 2024-04-23T16:33:16.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.988 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:16.989 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.273427+0000", 2024-04-23T16:33:16.989 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000118149 2024-04-23T16:33:16.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.989 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:33:16.989 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.273629+0000", 2024-04-23T16:33:16.989 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00020209399999999999 2024-04-23T16:33:16.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.989 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:16.989 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.274363+0000", 2024-04-23T16:33:16.989 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00073423200000000003 2024-04-23T16:33:16.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:16.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437621+0000", 2024-04-23T16:33:16.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.163258073 2024-04-23T16:33:16.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:16.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437676+0000", 2024-04-23T16:33:16.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5061e-05 2024-04-23T16:33:16.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:16.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437780+0000", 2024-04-23T16:33:16.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000104137 2024-04-23T16:33:16.990 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.991 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:16.991 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.991 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12906 4.d 4:bb06cb71:::benchmark_data_smithi188_101370_object12905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:33:16.991 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.283740+0000", 2024-04-23T16:33:16.991 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 356.60580713100001, 2024-04-23T16:33:16.991 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.15417470999999999, 2024-04-23T16:33:16.991 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:16.991 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:16.991 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:16.991 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:16.991 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:33:16.991 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:33:16.992 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12906 2024-04-23T16:33:16.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.992 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:16.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:16.992 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.283740+0000", 2024-04-23T16:33:16.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:16.992 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.283740+0000", 2024-04-23T16:33:16.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.993 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:16.993 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.283741+0000", 2024-04-23T16:33:16.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.55e-06 2024-04-23T16:33:16.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.993 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:16.993 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.283821+0000", 2024-04-23T16:33:16.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0289000000000003e-05 2024-04-23T16:33:16.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.993 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:16.993 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.283834+0000", 2024-04-23T16:33:16.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2799000000000001e-05 2024-04-23T16:33:16.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.994 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:16.994 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.283842+0000", 2024-04-23T16:33:16.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0390000000000005e-06 2024-04-23T16:33:16.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.994 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:16.994 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.283896+0000", 2024-04-23T16:33:16.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3856000000000001e-05 2024-04-23T16:33:16.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.994 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:16.994 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.284029+0000", 2024-04-23T16:33:16.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000133079 2024-04-23T16:33:16.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:33:16.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.284168+0000", 2024-04-23T16:33:16.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013844000000000001 2024-04-23T16:33:16.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:16.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.284799+0000", 2024-04-23T16:33:16.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00063107599999999995 2024-04-23T16:33:16.996 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.996 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.996 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:16.996 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437820+0000", 2024-04-23T16:33:16.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.15302147299999999 2024-04-23T16:33:16.996 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.996 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.996 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:16.996 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437872+0000", 2024-04-23T16:33:16.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2173999999999999e-05 2024-04-23T16:33:16.996 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.996 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.996 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:16.997 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437914+0000", 2024-04-23T16:33:16.997 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1934999999999997e-05 2024-04-23T16:33:16.997 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.997 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:16.997 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:16.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.997 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12952 4.d 4:b4108fc3:::benchmark_data_smithi188_101370_object12951:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:33:16.997 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.294705+0000", 2024-04-23T16:33:16.997 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 356.59484120600001, 2024-04-23T16:33:16.997 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.14389808100000001, 2024-04-23T16:33:16.997 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:16.997 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:16.998 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:16.998 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:16.998 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:33:16.998 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:33:16.998 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12952 2024-04-23T16:33:16.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.998 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:16.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.998 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:16.998 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294705+0000", 2024-04-23T16:33:16.998 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.999 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:16.999 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294705+0000", 2024-04-23T16:33:16.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:16.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.999 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:16.999 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294707+0000", 2024-04-23T16:33:16.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5400000000000001e-06 2024-04-23T16:33:16.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:16.999 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:16.999 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294786+0000", 2024-04-23T16:33:16.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.9336000000000001e-05 2024-04-23T16:33:16.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:16.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.000 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:17.000 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294797+0000", 2024-04-23T16:33:17.000 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0261999999999999e-05 2024-04-23T16:33:17.000 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.000 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:17.000 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294806+0000", 2024-04-23T16:33:17.000 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3640000000000005e-06 2024-04-23T16:33:17.000 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.000 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:17.000 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294850+0000", 2024-04-23T16:33:17.000 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4348999999999998e-05 2024-04-23T16:33:17.001 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.001 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.001 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:17.001 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294956+0000", 2024-04-23T16:33:17.001 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010563799999999999 2024-04-23T16:33:17.001 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.001 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.001 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:33:17.001 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.295084+0000", 2024-04-23T16:33:17.001 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012836999999999999 2024-04-23T16:33:17.001 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.001 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.001 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:17.001 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.296273+0000", 2024-04-23T16:33:17.002 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0011887899999999999 2024-04-23T16:33:17.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.002 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.002 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:17.002 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.438514+0000", 2024-04-23T16:33:17.002 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.142240754 2024-04-23T16:33:17.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.002 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.002 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:17.002 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.438566+0000", 2024-04-23T16:33:17.002 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.223e-05 2024-04-23T16:33:17.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.002 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:17.003 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.438604+0000", 2024-04-23T16:33:17.003 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7447999999999998e-05 2024-04-23T16:33:17.003 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:17.003 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:17.003 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:17.003 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.003 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12954 4.d 4:b54e759f:::benchmark_data_smithi188_101370_object12953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:33:17.003 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.294975+0000", 2024-04-23T16:33:17.003 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 356.594572153, 2024-04-23T16:33:17.003 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.14378174599999999, 2024-04-23T16:33:17.003 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:17.003 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:17.004 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:17.004 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:17.004 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:33:17.004 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:33:17.004 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12954 2024-04-23T16:33:17.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.004 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:17.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.004 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:17.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294975+0000", 2024-04-23T16:33:17.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:17.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.004 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:17.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294975+0000", 2024-04-23T16:33:17.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:17.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:17.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294976+0000", 2024-04-23T16:33:17.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3740000000000001e-06 2024-04-23T16:33:17.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:17.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.295058+0000", 2024-04-23T16:33:17.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1865000000000003e-05 2024-04-23T16:33:17.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:17.006 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.295067+0000", 2024-04-23T16:33:17.006 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2410000000000001e-06 2024-04-23T16:33:17.006 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.006 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.006 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:17.006 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.295075+0000", 2024-04-23T16:33:17.006 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5700000000000004e-06 2024-04-23T16:33:17.006 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.006 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.006 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:17.006 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.295662+0000", 2024-04-23T16:33:17.007 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00058759600000000002 2024-04-23T16:33:17.007 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.007 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:17.007 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.295781+0000", 2024-04-23T16:33:17.007 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011877100000000001 2024-04-23T16:33:17.007 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.007 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:33:17.007 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.295921+0000", 2024-04-23T16:33:17.007 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013954200000000001 2024-04-23T16:33:17.007 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.008 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:17.008 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.296878+0000", 2024-04-23T16:33:17.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000957988 2024-04-23T16:33:17.008 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.008 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.008 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:17.008 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.438655+0000", 2024-04-23T16:33:17.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.141776021 2024-04-23T16:33:17.008 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.008 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.008 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:17.008 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.438716+0000", 2024-04-23T16:33:17.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1402000000000006e-05 2024-04-23T16:33:17.008 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.009 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.009 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:17.009 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.438756+0000", 2024-04-23T16:33:17.009 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0376000000000001e-05 2024-04-23T16:33:17.009 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:17.009 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:17.009 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:17.009 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.009 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.009 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:13024 4.d 4:b1add8c7:::benchmark_data_smithi188_101370_object13023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:33:17.009 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.315294+0000", 2024-04-23T16:33:17.009 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 356.57425247800001, 2024-04-23T16:33:17.009 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.124104795, 2024-04-23T16:33:17.009 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:17.010 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:17.010 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:17.010 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:17.010 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:33:17.010 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:33:17.010 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 13024 2024-04-23T16:33:17.010 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.010 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:17.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.010 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:17.010 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.315294+0000", 2024-04-23T16:33:17.010 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:17.010 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.011 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:17.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.315294+0000", 2024-04-23T16:33:17.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:17.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.011 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.011 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:17.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.315296+0000", 2024-04-23T16:33:17.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.418e-06 2024-04-23T16:33:17.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.011 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.011 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:17.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.315352+0000", 2024-04-23T16:33:17.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6385000000000003e-05 2024-04-23T16:33:17.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:17.012 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.315361+0000", 2024-04-23T16:33:17.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5120000000000008e-06 2024-04-23T16:33:17.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:17.012 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.315366+0000", 2024-04-23T16:33:17.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0730000000000004e-06 2024-04-23T16:33:17.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:17.012 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.315411+0000", 2024-04-23T16:33:17.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5886000000000001e-05 2024-04-23T16:33:17.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.013 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.013 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:17.013 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.315520+0000", 2024-04-23T16:33:17.013 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000108687 2024-04-23T16:33:17.013 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.013 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.013 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:33:17.013 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.315627+0000", 2024-04-23T16:33:17.013 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000106735 2024-04-23T16:33:17.013 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.013 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.013 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:17.013 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.316337+0000", 2024-04-23T16:33:17.013 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00071045099999999996 2024-04-23T16:33:17.014 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.014 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.014 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:17.014 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.439333+0000", 2024-04-23T16:33:17.014 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.122995585 2024-04-23T16:33:17.014 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.014 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.014 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:17.014 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.439371+0000", 2024-04-23T16:33:17.014 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8538999999999998e-05 2024-04-23T16:33:17.014 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:17.014 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:17.014 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:17.015 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.439399+0000", 2024-04-23T16:33:17.015 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7523999999999999e-05 2024-04-23T16:33:17.015 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:17.015 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:17.015 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:17.015 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:17.015 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:17.015 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:33:17.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:16 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4035413517' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["2"]}]: dispatch 2024-04-23T16:33:17.216 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:33:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:16 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4035413517' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["2"]}]: dispatch 2024-04-23T16:33:17.281 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 98 16 153594 153578 97.936 0 - 0.00839412 2024-04-23T16:33:17.508 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:33:17.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:17.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:33:17.612 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:33:17.641 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-23T16:33:17.644 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:17.666 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:33:17.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:17.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:33:17.730 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 2, 3, 4, 5, 6, 7, 8, 9, 11] 2024-04-23T16:33:17.730 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd out 2 2024-04-23T16:33:17.770 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:33:17.777 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:33:17.816 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:17.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:17 smithi148 ceph-mon[44520]: pgmap v1102: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:17.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:17 smithi148 ceph-mon[44520]: 5.f scrub starts 2024-04-23T16:33:17.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:17 smithi148 ceph-mon[44520]: 5.3 scrub starts 2024-04-23T16:33:17.935 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:33:17.968 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:33:17.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:17.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:33:18.075 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:33:18.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:17 smithi103 ceph-mon[39629]: pgmap v1102: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:18.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:17 smithi103 ceph-mon[39629]: 5.f scrub starts 2024-04-23T16:33:18.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:17 smithi103 ceph-mon[39629]: 5.3 scrub starts 2024-04-23T16:33:18.101 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:17 smithi188 ceph-mon[44101]: pgmap v1102: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:17 smithi188 ceph-mon[44101]: 5.f scrub starts 2024-04-23T16:33:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:17 smithi188 ceph-mon[44101]: 5.3 scrub starts 2024-04-23T16:33:18.281 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 99 16 153594 153578 96.9467 0 - 0.00839412 2024-04-23T16:33:18.449 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:33:18.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:18.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:33:18.553 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:18.581 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:18.703 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:18.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:18.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:18.806 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:18.831 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:18 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:33:18.831 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:18 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:33:18.831 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:18 smithi148 ceph-mon[44520]: 5.f scrub ok 2024-04-23T16:33:18.831 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:18 smithi148 ceph-mon[44520]: 5.7 scrub starts 2024-04-23T16:33:18.831 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:18 smithi148 ceph-mon[44520]: 5.3 scrub ok 2024-04-23T16:33:18.833 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:19.058 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:18 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:33:19.058 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:18 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:33:19.058 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:18 smithi103 ceph-mon[39629]: 5.f scrub ok 2024-04-23T16:33:19.058 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:18 smithi103 ceph-mon[39629]: 5.7 scrub starts 2024-04-23T16:33:19.058 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:18 smithi103 ceph-mon[39629]: 5.3 scrub ok 2024-04-23T16:33:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:18 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:33:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:18 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:33:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:18 smithi188 ceph-mon[44101]: 5.f scrub ok 2024-04-23T16:33:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:18 smithi188 ceph-mon[44101]: 5.7 scrub starts 2024-04-23T16:33:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:18 smithi188 ceph-mon[44101]: 5.3 scrub ok 2024-04-23T16:33:19.281 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:33:19.281362 min lat: 0.0010294 max lat: 1.8332 avg lat: 0.00839412 2024-04-23T16:33:19.282 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:33:19.282 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 100 16 153594 153578 95.9772 0 - 0.00839412 2024-04-23T16:33:19.282 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:19.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:19.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:19.386 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:33:19.413 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:19.431 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:33:19.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:19.433 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:33:19.534 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:33:19.563 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:19.774 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:33:19.775 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:33:19.775 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:33:19.775 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2024-04-23T16:33:19.775 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:33:19.800 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:33:19.811 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:33:19.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:19.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:33:19.820 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:33:19.841 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:33:19.863 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:33:19.889 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:33:19.908 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:33:19.914 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:19.935 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:33:19.942 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:19.955 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:33:19.975 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:33:19.994 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:33:19.995 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:33:19.995 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:33:19.995 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:33:19.995 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:33:19.995 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:33:19.995 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:33:19.995 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:33:19.996 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:33:19.996 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:33:19.996 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:33:19.996 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:33:19.996 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:33:20.040 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:20.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:19 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T16:33:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:19 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T16:33:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:19 smithi103 ceph-mon[39629]: 5.7 scrub ok 2024-04-23T16:33:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:19 smithi103 ceph-mon[39629]: pgmap v1103: 27 pgs: 1 active+clean+scrubbing, 4 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:19 smithi103 ceph-mon[39629]: 5.5 scrub starts 2024-04-23T16:33:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:19 smithi103 ceph-mon[39629]: 1.4 scrub starts 2024-04-23T16:33:20.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:19 smithi103 ceph-mon[39629]: 1.4 scrub ok 2024-04-23T16:33:20.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:19 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4016499569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:33:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:19 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T16:33:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:19 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T16:33:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:19 smithi148 ceph-mon[44520]: 5.7 scrub ok 2024-04-23T16:33:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:19 smithi148 ceph-mon[44520]: pgmap v1103: 27 pgs: 1 active+clean+scrubbing, 4 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:19 smithi148 ceph-mon[44520]: 5.5 scrub starts 2024-04-23T16:33:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:19 smithi148 ceph-mon[44520]: 1.4 scrub starts 2024-04-23T16:33:20.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:19 smithi148 ceph-mon[44520]: 1.4 scrub ok 2024-04-23T16:33:20.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:19 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4016499569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:33:20.143 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:33:20.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:19 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T16:33:20.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:19 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T16:33:20.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:19 smithi188 ceph-mon[44101]: 5.7 scrub ok 2024-04-23T16:33:20.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:19 smithi188 ceph-mon[44101]: pgmap v1103: 27 pgs: 1 active+clean+scrubbing, 4 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:20.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:19 smithi188 ceph-mon[44101]: 5.5 scrub starts 2024-04-23T16:33:20.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:19 smithi188 ceph-mon[44101]: 1.4 scrub starts 2024-04-23T16:33:20.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:19 smithi188 ceph-mon[44101]: 1.4 scrub ok 2024-04-23T16:33:20.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:19 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4016499569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:33:20.170 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:20.188 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:33:20.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:20.191 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:33:20.281 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 101 16 153594 153578 95.0269 0 - 0.00839412 2024-04-23T16:33:20.292 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:33:20.322 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:20.450 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-23T16:33:20.571 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:33:20.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:20.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:33:20.675 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:20.702 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:20.712 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:33:20.723 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:20.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:20.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:20.826 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:33:20.836 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.836 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.836 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.836 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.836 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.836 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.837 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.837 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.837 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:20 smithi103 ceph-mon[39629]: 5.5 scrub ok 2024-04-23T16:33:20.837 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.837 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.855 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:20.876 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:33:20.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:20.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:33:20.962 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.962 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.962 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.962 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.962 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.962 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.962 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.963 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.963 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:20 smithi148 ceph-mon[44520]: 5.5 scrub ok 2024-04-23T16:33:20.963 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.963 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:20.979 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:33:21.006 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:21.072 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:33:21.079 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:33:21.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:21.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:33:21.183 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:33:21.211 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:20 smithi188 ceph-mon[44101]: 5.5 scrub ok 2024-04-23T16:33:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:21.282 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 102 16 153594 153578 94.0952 0 - 0.00839412 2024-04-23T16:33:21.559 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:33:21.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:21.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:33:21.663 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:33:21.690 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:21.710 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:33:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:21.713 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:33:21.811 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:33:21.813 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:33:21.837 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:21.837 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:21 smithi103 ceph-mon[39629]: pgmap v1104: 27 pgs: 1 active+clean+scrubbing, 4 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:21.837 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:21 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1378824819' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:33:21.837 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:21 smithi103 ceph-mon[39629]: from='client.35783 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:33:21.837 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:21 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:33:21.841 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:21.862 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:33:21.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:21.865 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:33:21.894 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:33:21.895 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:33:21.965 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:21.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:21 smithi188 ceph-mon[44101]: pgmap v1104: 27 pgs: 1 active+clean+scrubbing, 4 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:21 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1378824819' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:33:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:21 smithi188 ceph-mon[44101]: from='client.35783 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:33:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:21 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:33:21.992 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:22.012 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:22.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:22.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:22.118 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:33:22.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:21 smithi148 ceph-mon[44520]: pgmap v1104: 27 pgs: 1 active+clean+scrubbing, 4 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:21 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1378824819' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:33:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:21 smithi148 ceph-mon[44520]: from='client.35783 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:33:22.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:21 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:33:22.145 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:22.282 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 103 16 153594 153578 93.1816 0 - 0.00839412 2024-04-23T16:33:22.293 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:33:22.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:22.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:33:22.398 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:33:22.424 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:22.442 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:33:22.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:22.445 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:33:22.546 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:33:22.550 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:33:22.550 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:33:22.576 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:22.595 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:33:22.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:22.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:33:22.701 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:22.728 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:22.800 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:22.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:22.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:22.905 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:33:22.935 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:23.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:22 smithi103 ceph-mon[39629]: 5.e scrub starts 2024-04-23T16:33:23.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:22 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:33:23.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:22 smithi103 ceph-mon[39629]: from='client.35783 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:33:23.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:22 smithi103 ceph-mon[39629]: osdmap e392: 12 total, 11 up, 10 in 2024-04-23T16:33:23.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:22 smithi148 ceph-mon[44520]: 5.e scrub starts 2024-04-23T16:33:23.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:22 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:33:23.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:22 smithi148 ceph-mon[44520]: from='client.35783 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:33:23.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:22 smithi148 ceph-mon[44520]: osdmap e392: 12 total, 11 up, 10 in 2024-04-23T16:33:23.223 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-23T16:33:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:22 smithi188 ceph-mon[44101]: 5.e scrub starts 2024-04-23T16:33:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:22 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:33:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:22 smithi188 ceph-mon[44101]: from='client.35783 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:33:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:22 smithi188 ceph-mon[44101]: osdmap e392: 12 total, 11 up, 10 in 2024-04-23T16:33:23.282 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 104 16 153594 153578 92.2856 0 - 0.00839412 2024-04-23T16:33:23.383 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:33:23.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:23.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:33:23.484 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:33:23.486 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:33:23.513 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:23.712 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:33:23.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:23.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:33:23.816 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:23.844 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:23.896 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:33:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:23 smithi103 ceph-mon[39629]: 5.e scrub ok 2024-04-23T16:33:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:23 smithi103 ceph-mon[39629]: pgmap v1106: 27 pgs: 1 active+clean+scrubbing, 4 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:23 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:33:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:23 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:33:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:23 smithi103 ceph-mon[39629]: osdmap e393: 12 total, 11 up, 10 in 2024-04-23T16:33:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:23 smithi148 ceph-mon[44520]: 5.e scrub ok 2024-04-23T16:33:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:23 smithi148 ceph-mon[44520]: pgmap v1106: 27 pgs: 1 active+clean+scrubbing, 4 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:23 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:33:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:23 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:33:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:23 smithi148 ceph-mon[44520]: osdmap e393: 12 total, 11 up, 10 in 2024-04-23T16:33:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:23 smithi188 ceph-mon[44101]: 5.e scrub ok 2024-04-23T16:33:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:23 smithi188 ceph-mon[44101]: pgmap v1106: 27 pgs: 1 active+clean+scrubbing, 4 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:23 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:33:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:23 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:33:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:23 smithi188 ceph-mon[44101]: osdmap e393: 12 total, 11 up, 10 in 2024-04-23T16:33:24.282 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 105 16 153594 153578 91.4067 0 - 0.00839412 2024-04-23T16:33:24.498 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:24 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T16:33:24.235+0000 7f135b1c7700 -1 mon.a@0(leader).osd e393 definitely_dead 0 2024-04-23T16:33:24.549 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:33:24.722 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:24.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:24.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:24.826 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:33:24.840 INFO:teuthology.orchestra.run.smithi103.stderr:marked out osd.2. 2024-04-23T16:33:24.854 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:24.896 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:33:24.952 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:33:24.954 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:24 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4249587218' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-23T16:33:24.954 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:24 smithi148 ceph-mon[44520]: from='client.35789 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-23T16:33:24.954 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:24 smithi148 ceph-mon[44520]: Client client.admin marked osd.2 out, after it was down for 24 seconds 2024-04-23T16:33:24.954 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:24 smithi148 ceph-mon[44520]: 5.8 scrub starts 2024-04-23T16:33:24.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:24.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:33:25.057 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:25.085 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:25.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4249587218' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-23T16:33:25.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:24 smithi103 ceph-mon[39629]: from='client.35789 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-23T16:33:25.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:24 smithi103 ceph-mon[39629]: Client client.admin marked osd.2 out, after it was down for 24 seconds 2024-04-23T16:33:25.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:24 smithi103 ceph-mon[39629]: 5.8 scrub starts 2024-04-23T16:33:25.104 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:25.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:25.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:25.207 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4249587218' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-23T16:33:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:24 smithi188 ceph-mon[44101]: from='client.35789 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-23T16:33:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:24 smithi188 ceph-mon[44101]: Client client.admin marked osd.2 out, after it was down for 24 seconds 2024-04-23T16:33:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:24 smithi188 ceph-mon[44101]: 5.8 scrub starts 2024-04-23T16:33:25.228 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:33:25.229 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:33:25.237 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:25.256 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:25.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:25.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:25.282 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 106 16 153594 153578 90.5443 0 - 0.00839412 2024-04-23T16:33:25.359 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:25.387 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:25.460 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:25.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:25.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:25.565 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:25.592 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:25.630 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:25.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:25.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:25.737 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:25.749 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-04-23T16:33:25.750 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi103.front.sepia.ceph.com 2024-04-23T16:33:25.750 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.2 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-04-23T16:33:25.764 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:25 smithi148 ceph-mon[44520]: pgmap v1108: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:25 smithi148 ceph-mon[44520]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T16:33:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:25 smithi148 ceph-mon[44520]: from='client.35789 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-04-23T16:33:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:25 smithi148 ceph-mon[44520]: osdmap e394: 12 total, 11 up, 9 in 2024-04-23T16:33:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:25 smithi148 ceph-mon[44520]: 5.8 scrub ok 2024-04-23T16:33:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:25 smithi148 ceph-mon[44520]: 5.d scrub starts 2024-04-23T16:33:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:25 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:33:25.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:25 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:33:25.948 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-23T16:33:25.963 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:25.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:26.002 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:25 smithi103 ceph-mon[39629]: pgmap v1108: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:26.003 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:25 smithi103 ceph-mon[39629]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T16:33:26.003 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:25 smithi103 ceph-mon[39629]: from='client.35789 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-04-23T16:33:26.003 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:25 smithi103 ceph-mon[39629]: osdmap e394: 12 total, 11 up, 9 in 2024-04-23T16:33:26.003 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:25 smithi103 ceph-mon[39629]: 5.8 scrub ok 2024-04-23T16:33:26.003 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:25 smithi103 ceph-mon[39629]: 5.d scrub starts 2024-04-23T16:33:26.003 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:25 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:33:26.003 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:25 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:33:26.007 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.2/config 2024-04-23T16:33:26.066 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:33:26.093 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:25 smithi188 ceph-mon[44101]: pgmap v1108: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:25 smithi188 ceph-mon[44101]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T16:33:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:25 smithi188 ceph-mon[44101]: from='client.35789 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-04-23T16:33:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:25 smithi188 ceph-mon[44101]: osdmap e394: 12 total, 11 up, 9 in 2024-04-23T16:33:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:25 smithi188 ceph-mon[44101]: 5.8 scrub ok 2024-04-23T16:33:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:25 smithi188 ceph-mon[44101]: 5.d scrub starts 2024-04-23T16:33:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:25 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:33:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:25 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:33:26.240 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:33:26.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:26.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:33:26.282 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:33:26.282 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 107 16 153595 153579 89.6986 0.00240385 26.3219 0.00856545 2024-04-23T16:33:26.344 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:26.375 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:26.574 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:26.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:26.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:26.679 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:33:26.709 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:26.815 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:33:26.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:26.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:33:26.919 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:33:26.946 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:27.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:26 smithi103 ceph-mon[39629]: osdmap e395: 12 total, 11 up, 9 in 2024-04-23T16:33:27.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:26 smithi103 ceph-mon[39629]: 1.5 scrub starts 2024-04-23T16:33:27.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:26 smithi103 ceph-mon[39629]: 5.d scrub ok 2024-04-23T16:33:27.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:26 smithi103 ceph-mon[39629]: 1.5 scrub ok 2024-04-23T16:33:27.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:26 smithi148 ceph-mon[44520]: osdmap e395: 12 total, 11 up, 9 in 2024-04-23T16:33:27.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:26 smithi148 ceph-mon[44520]: 1.5 scrub starts 2024-04-23T16:33:27.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:26 smithi148 ceph-mon[44520]: 5.d scrub ok 2024-04-23T16:33:27.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:26 smithi148 ceph-mon[44520]: 1.5 scrub ok 2024-04-23T16:33:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:26 smithi188 ceph-mon[44101]: osdmap e395: 12 total, 11 up, 9 in 2024-04-23T16:33:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:26 smithi188 ceph-mon[44101]: 1.5 scrub starts 2024-04-23T16:33:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:26 smithi188 ceph-mon[44101]: 5.d scrub ok 2024-04-23T16:33:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:26 smithi188 ceph-mon[44101]: 1.5 scrub ok 2024-04-23T16:33:27.282 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Total time run: 107.238 2024-04-23T16:33:27.283 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Total writes made: 153595 2024-04-23T16:33:27.283 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Write size: 65536 2024-04-23T16:33:27.283 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Object size: 65536 2024-04-23T16:33:27.283 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Bandwidth (MB/sec): 89.518 2024-04-23T16:33:27.283 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev Bandwidth: 78.2751 2024-04-23T16:33:27.283 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max bandwidth (MB/sec): 389.938 2024-04-23T16:33:27.283 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min bandwidth (MB/sec): 0 2024-04-23T16:33:27.283 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Average IOPS: 1432 2024-04-23T16:33:27.283 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev IOPS: 1252.4 2024-04-23T16:33:27.283 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max IOPS: 6239 2024-04-23T16:33:27.283 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min IOPS: 0 2024-04-23T16:33:27.283 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Average Latency(s): 0.0111675 2024-04-23T16:33:27.283 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev Latency(s): 0.272195 2024-04-23T16:33:27.284 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max latency(s): 26.734 2024-04-23T16:33:27.284 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min latency(s): 0.0010294 2024-04-23T16:33:27.289 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Cleaning up (deleting benchmark objects) 2024-04-23T16:33:27.561 INFO:teuthology.orchestra.run.smithi103.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:33:27.871 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:33:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:27 smithi103 ceph-mon[39629]: pgmap v1111: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:28.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:27 smithi148 ceph-mon[44520]: pgmap v1111: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:27 smithi188 ceph-mon[44101]: pgmap v1111: 27 pgs: 4 undersized+degraded+peered, 1 undersized+peered, 22 active+clean; 9.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 28953/307160 objects degraded (9.426%) 2024-04-23T16:33:28.348 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:33:28.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:28.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:33:28.453 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:33:28.481 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:28.500 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:33:28.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:28.503 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:33:28.543 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:33:28.543 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:33:28.603 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:28.676 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:28.805 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:28.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:28.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:28.910 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:28.968 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:29.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:28 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 28753/306706 objects degraded (9.375%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:28 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 28753/306706 objects degraded (9.375%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:29.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:28 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 28753/306706 objects degraded (9.375%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:29.959 INFO:teuthology.orchestra.run.smithi103.stdout:5.a 2024-04-23T16:33:29.959 INFO:teuthology.orchestra.run.smithi103.stdout:5.6 2024-04-23T16:33:29.960 INFO:teuthology.orchestra.run.smithi103.stdout:5.4 2024-04-23T16:33:29.960 INFO:teuthology.orchestra.run.smithi103.stdout:1.5 2024-04-23T16:33:29.960 INFO:teuthology.orchestra.run.smithi103.stdout:1.8 2024-04-23T16:33:30.156 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:29 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:33:30.156 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:29 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:33:30.156 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:29 smithi103 ceph-mon[39629]: pgmap v1112: 27 pgs: 3 active+recovering+degraded, 1 undersized+peered, 23 active+clean; 9.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 171 KiB/s wr, 43 op/s; 28753/306706 objects degraded (9.375%); 2.1 MiB/s, 33 objects/s recovering 2024-04-23T16:33:30.156 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:33:30.156 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:33:30.156 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:33:30.156 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:33:30.156 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:33:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:29 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:33:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:29 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:33:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:29 smithi188 ceph-mon[44101]: pgmap v1112: 27 pgs: 3 active+recovering+degraded, 1 undersized+peered, 23 active+clean; 9.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 171 KiB/s wr, 43 op/s; 28753/306706 objects degraded (9.375%); 2.1 MiB/s, 33 objects/s recovering 2024-04-23T16:33:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:33:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:33:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:33:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:33:30.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:33:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:29 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:33:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:29 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:33:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:29 smithi148 ceph-mon[44520]: pgmap v1112: 27 pgs: 3 active+recovering+degraded, 1 undersized+peered, 23 active+clean; 9.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 171 KiB/s wr, 43 op/s; 28753/306706 objects degraded (9.375%); 2.1 MiB/s, 33 objects/s recovering 2024-04-23T16:33:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:33:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:33:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:33:30.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:33:30.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:33:30.493 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:30.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:30.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:30.522 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:33:30.542 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:33:30.547 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-23T16:33:30.562 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:33:30.584 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:33:30.596 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:30.624 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:30.632 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:33:30.658 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:33:30.669 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:30.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:30.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:30.684 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:33:30.707 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:33:30.729 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:33:30.755 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:33:30.775 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:30.776 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:33:30.776 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:33:30.776 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:33:30.776 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:33:30.777 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:33:30.777 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:33:30.777 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:33:30.777 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:33:30.777 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:33:30.777 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:33:30.777 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:33:30.777 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:33:30.777 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:33:30.876 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:30.912 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:30.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:30.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:31.001 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:33:31.016 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:33:31.078 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:31.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:30 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3009452940' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:33:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:31.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:31.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:30 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3009452940' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:33:31.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:31.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:31.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:31.168 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:33:31.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:31.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:33:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:30 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3009452940' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:33:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:31.274 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:31.337 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:31.640 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.2 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 5.a --file /var/log/ceph/exp.5.a.2 2024-04-23T16:33:32.098 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.2/config 2024-04-23T16:33:32.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:31 smithi188 ceph-mon[44101]: pgmap v1113: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 9.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 104 KiB/s wr, 124 op/s; 28753/305232 objects degraded (9.420%); 1.6 MiB/s, 25 objects/s recovering 2024-04-23T16:33:32.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:31 smithi188 ceph-mon[44101]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-23T16:33:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:31 smithi103 ceph-mon[39629]: pgmap v1113: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 9.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 104 KiB/s wr, 124 op/s; 28753/305232 objects degraded (9.420%); 1.6 MiB/s, 25 objects/s recovering 2024-04-23T16:33:32.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:31 smithi103 ceph-mon[39629]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-23T16:33:32.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:31 smithi148 ceph-mon[44520]: pgmap v1113: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 9.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 104 KiB/s wr, 124 op/s; 28753/305232 objects degraded (9.420%); 1.6 MiB/s, 25 objects/s recovering 2024-04-23T16:33:32.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:32.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:31 smithi148 ceph-mon[44520]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-23T16:33:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:32 smithi188 ceph-mon[44101]: 1.a scrub starts 2024-04-23T16:33:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:32 smithi188 ceph-mon[44101]: 1.a scrub ok 2024-04-23T16:33:33.332 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:33.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:33.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:33.339 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:32 smithi148 ceph-mon[44520]: 1.a scrub starts 2024-04-23T16:33:33.339 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:32 smithi148 ceph-mon[44520]: 1.a scrub ok 2024-04-23T16:33:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:32 smithi103 ceph-mon[39629]: 1.a scrub starts 2024-04-23T16:33:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:32 smithi103 ceph-mon[39629]: 1.a scrub ok 2024-04-23T16:33:33.438 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:33.501 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:33.552 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:33:33.552 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:33:33.661 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:33.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:33.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:33.768 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:33:33.798 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:33.827 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:33:33.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:33.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:33:33.933 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:33:33.962 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:33.982 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:33:33.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:33.985 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:33:34.056 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:33 smithi103 ceph-mon[39629]: pgmap v1114: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 9.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 103 KiB/s wr, 166 op/s; 28753/304534 objects degraded (9.442%); 1.6 MiB/s, 24 objects/s recovering 2024-04-23T16:33:34.056 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:33 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:33:34.056 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:33 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:33:34.086 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:33:34.114 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:34.139 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:33:34.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:34.149 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:33:34.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:33 smithi188 ceph-mon[44101]: pgmap v1114: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 9.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 103 KiB/s wr, 166 op/s; 28753/304534 objects degraded (9.442%); 1.6 MiB/s, 24 objects/s recovering 2024-04-23T16:33:34.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:33 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:33:34.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:33 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:33:34.250 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:33:34.281 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:34.300 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:33:34.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:34.311 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:33:34.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:33 smithi148 ceph-mon[44520]: pgmap v1114: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 9.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 103 KiB/s wr, 166 op/s; 28753/304534 objects degraded (9.442%); 1.6 MiB/s, 24 objects/s recovering 2024-04-23T16:33:34.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:33 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:33:34.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:33 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:33:34.412 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:33:34.445 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:34.471 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:33:34.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:34.475 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:33:34.577 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:34.629 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:34.895 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:34.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:34.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:35.005 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:35.052 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:34 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 28753/304534 objects degraded (9.442%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:34 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3898270567' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:33:35.154 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:33:35.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:34 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 28753/304534 objects degraded (9.442%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:35.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:34 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3898270567' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:33:35.229 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:35.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:35.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:35.236 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:34 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 28753/304534 objects degraded (9.442%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:35.236 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:34 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3898270567' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:33:35.294 INFO:teuthology.orchestra.run.smithi103.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:33:35.328 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:33:35.328 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:33:35.328 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2024-04-23T16:33:35.328 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2024-04-23T16:33:35.329 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:33:35.336 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:35.417 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:35.552 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:35.553 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:33:35.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:35.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:35.665 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:35.726 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:36.093 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-23T16:33:36.098 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:36.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:36.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:36.211 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:33:36.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:35 smithi188 ceph-mon[44101]: pgmap v1115: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 9.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 83 KiB/s wr, 392 op/s; 28217/299440 objects degraded (9.423%); 4.7 MiB/s, 74 objects/s recovering 2024-04-23T16:33:36.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:35 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:33:36.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:35 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3898270567' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:33:36.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:35 smithi188 ceph-mon[44101]: osdmap e396: 12 total, 11 up, 9 in 2024-04-23T16:33:36.238 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:36.263 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:33:36.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:36.267 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:33:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:35 smithi103 ceph-mon[39629]: pgmap v1115: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 9.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 83 KiB/s wr, 392 op/s; 28217/299440 objects degraded (9.423%); 4.7 MiB/s, 74 objects/s recovering 2024-04-23T16:33:36.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:35 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:33:36.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:35 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3898270567' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:33:36.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:35 smithi103 ceph-mon[39629]: osdmap e396: 12 total, 11 up, 9 in 2024-04-23T16:33:36.367 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:33:36.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:35 smithi148 ceph-mon[44520]: pgmap v1115: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 9.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 83 KiB/s wr, 392 op/s; 28217/299440 objects degraded (9.423%); 4.7 MiB/s, 74 objects/s recovering 2024-04-23T16:33:36.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:35 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:33:36.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:35 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3898270567' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:33:36.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:35 smithi148 ceph-mon[44520]: osdmap e396: 12 total, 11 up, 9 in 2024-04-23T16:33:36.395 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:36.415 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:33:36.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:36.418 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:33:36.520 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:36.579 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:33:36.606 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:36.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:36.643 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:36.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:36.744 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:33:36.776 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:36.799 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:33:36.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:36.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:33:36.817 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:33:36.905 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:33:36.959 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:36.995 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:33:37.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:37.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:33:37.103 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:33:37.140 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:37.164 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:33:37.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:37.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:33:37.271 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:33:37.299 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:37.318 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:33:37.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:37.321 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:33:37.422 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:33:37.451 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:37.473 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:33:37.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:37.476 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:33:37.575 INFO:teuthology.orchestra.run.smithi103.stderr:Exporting 5.a info 5.a( v 386'9712 (0'0,386'9712] local-lis/les=359/360 n=9712 ec=359/359 lis/c=359/359 les/c/f=360/360/0 sis=359) 2024-04-23T16:33:37.576 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:33:37.606 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:37.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5000aecc:::benchmark_data_smithi188_113957_object25214:head# 2024-04-23T16:33:37.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5000b1fb:::benchmark_data_smithi188_113957_object88294:head# 2024-04-23T16:33:37.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5000d1fa:::benchmark_data_smithi188_113957_object63134:head# 2024-04-23T16:33:37.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5001785e:::benchmark_data_smithi188_113957_object576:head# 2024-04-23T16:33:37.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5002d4fb:::benchmark_data_smithi188_113957_object110729:head# 2024-04-23T16:33:37.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5002d6c2:::benchmark_data_smithi188_113957_object108243:head# 2024-04-23T16:33:37.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5002e6ba:::benchmark_data_smithi188_113957_object137087:head# 2024-04-23T16:33:37.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5003e9da:::benchmark_data_smithi188_113957_object90852:head# 2024-04-23T16:33:37.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5003ea6b:::benchmark_data_smithi188_113957_object59076:head# 2024-04-23T16:33:37.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50049178:::benchmark_data_smithi188_113957_object39155:head# 2024-04-23T16:33:37.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50053d67:::benchmark_data_smithi188_113957_object83545:head# 2024-04-23T16:33:37.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50055a33:::benchmark_data_smithi188_113957_object142656:head# 2024-04-23T16:33:37.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5005c6fb:::benchmark_data_smithi188_113957_object140615:head# 2024-04-23T16:33:37.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5005db0d:::benchmark_data_smithi188_113957_object89814:head# 2024-04-23T16:33:37.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5006ec38:::benchmark_data_smithi188_113957_object73297:head# 2024-04-23T16:33:37.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5007914a:::benchmark_data_smithi188_113957_object24056:head# 2024-04-23T16:33:37.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50094b33:::benchmark_data_smithi188_113957_object43552:head# 2024-04-23T16:33:37.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50095dd8:::benchmark_data_smithi188_113957_object102563:head# 2024-04-23T16:33:37.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5009d793:::benchmark_data_smithi188_113957_object8521:head# 2024-04-23T16:33:37.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5009f36f:::benchmark_data_smithi188_113957_object120695:head# 2024-04-23T16:33:37.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:500a1e91:::benchmark_data_smithi188_113957_object112154:head# 2024-04-23T16:33:37.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:500a557c:::benchmark_data_smithi188_113957_object34865:head# 2024-04-23T16:33:37.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:500a75b7:::benchmark_data_smithi188_113957_object118329:head# 2024-04-23T16:33:37.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:500afe81:::benchmark_data_smithi188_113957_object78463:head# 2024-04-23T16:33:37.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:500b0565:::benchmark_data_smithi188_113957_object103091:head# 2024-04-23T16:33:37.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:500bffaf:::benchmark_data_smithi188_113957_object24049:head# 2024-04-23T16:33:37.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:500c54ba:::benchmark_data_smithi188_113957_object20722:head# 2024-04-23T16:33:37.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:500cb56d:::benchmark_data_smithi188_113957_object109029:head# 2024-04-23T16:33:37.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:500ce5a8:::benchmark_data_smithi188_113957_object26326:head# 2024-04-23T16:33:37.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:500d2af8:::benchmark_data_smithi188_113957_object100892:head# 2024-04-23T16:33:37.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:500d33b2:::benchmark_data_smithi188_113957_object118095:head# 2024-04-23T16:33:37.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:500e0df9:::benchmark_data_smithi188_113957_object4057:head# 2024-04-23T16:33:37.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:500e1b23:::benchmark_data_smithi188_113957_object35892:head# 2024-04-23T16:33:37.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:500ee6ac:::benchmark_data_smithi188_113957_object57714:head# 2024-04-23T16:33:37.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:500f467e:::benchmark_data_smithi188_113957_object121458:head# 2024-04-23T16:33:37.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50105912:::benchmark_data_smithi188_113957_object126891:head# 2024-04-23T16:33:37.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5011fbb6:::benchmark_data_smithi188_113957_object142365:head# 2024-04-23T16:33:37.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:501236a6:::benchmark_data_smithi188_113957_object3380:head# 2024-04-23T16:33:37.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:501301da:::benchmark_data_smithi188_113957_object44650:head# 2024-04-23T16:33:37.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50133de5:::benchmark_data_smithi188_113957_object64266:head# 2024-04-23T16:33:37.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5013e6fb:::benchmark_data_smithi188_113957_object91942:head# 2024-04-23T16:33:37.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50141b29:::benchmark_data_smithi188_113957_object144942:head# 2024-04-23T16:33:37.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50147160:::benchmark_data_smithi188_113957_object133034:head# 2024-04-23T16:33:37.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5014f597:::benchmark_data_smithi188_113957_object51409:head# 2024-04-23T16:33:37.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:501504bc:::benchmark_data_smithi188_113957_object125796:head# 2024-04-23T16:33:37.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50155661:::benchmark_data_smithi188_113957_object46012:head# 2024-04-23T16:33:37.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5015b3aa:::benchmark_data_smithi188_113957_object131397:head# 2024-04-23T16:33:37.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5015dac2:::benchmark_data_smithi188_113957_object112924:head# 2024-04-23T16:33:37.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50163301:::benchmark_data_smithi188_113957_object24122:head# 2024-04-23T16:33:37.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5016692b:::benchmark_data_smithi188_113957_object4333:head# 2024-04-23T16:33:37.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5016d351:::benchmark_data_smithi188_113957_object97877:head# 2024-04-23T16:33:37.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50170301:::benchmark_data_smithi188_113957_object767:head# 2024-04-23T16:33:37.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5017a617:::benchmark_data_smithi188_113957_object34256:head# 2024-04-23T16:33:37.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50180741:::benchmark_data_smithi188_113957_object57907:head# 2024-04-23T16:33:37.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5018538b:::benchmark_data_smithi188_113957_object56575:head# 2024-04-23T16:33:37.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50186012:::benchmark_data_smithi188_113957_object84046:head# 2024-04-23T16:33:37.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50192224:::benchmark_data_smithi188_113957_object99466:head# 2024-04-23T16:33:37.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5019a0b5:::benchmark_data_smithi188_113957_object118194:head# 2024-04-23T16:33:37.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5019abaa:::benchmark_data_smithi188_113957_object54246:head# 2024-04-23T16:33:37.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:501a6e23:::benchmark_data_smithi188_113957_object85200:head# 2024-04-23T16:33:37.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:501a8ca8:::benchmark_data_smithi188_113957_object52461:head# 2024-04-23T16:33:37.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:501aee86:::benchmark_data_smithi188_113957_object140340:head# 2024-04-23T16:33:37.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:501c9c38:::benchmark_data_smithi188_113957_object150006:head# 2024-04-23T16:33:37.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:501ca56c:::benchmark_data_smithi188_113957_object113692:head# 2024-04-23T16:33:37.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:501cfd48:::benchmark_data_smithi188_113957_object16928:head# 2024-04-23T16:33:37.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:501d9b78:::benchmark_data_smithi188_113957_object141793:head# 2024-04-23T16:33:37.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:501e52c0:::benchmark_data_smithi188_113957_object120183:head# 2024-04-23T16:33:37.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:501ebbac:::benchmark_data_smithi188_113957_object70479:head# 2024-04-23T16:33:37.661 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:501fa391:::benchmark_data_smithi188_113957_object74444:head# 2024-04-23T16:33:37.661 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50206a75:::benchmark_data_smithi188_113957_object31902:head# 2024-04-23T16:33:37.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502094ec:::benchmark_data_smithi188_113957_object66503:head# 2024-04-23T16:33:37.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5020fad2:::benchmark_data_smithi188_113957_object110213:head# 2024-04-23T16:33:37.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5021351b:::benchmark_data_smithi188_113957_object142004:head# 2024-04-23T16:33:37.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502167fc:::benchmark_data_smithi188_113957_object91258:head# 2024-04-23T16:33:37.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50219437:::benchmark_data_smithi188_113957_object11230:head# 2024-04-23T16:33:37.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50223fb0:::benchmark_data_smithi188_113957_object144689:head# 2024-04-23T16:33:37.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5022439e:::benchmark_data_smithi188_113957_object86313:head# 2024-04-23T16:33:37.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50226a7a:::benchmark_data_smithi188_113957_object73357:head# 2024-04-23T16:33:37.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5022858d:::benchmark_data_smithi188_113957_object122708:head# 2024-04-23T16:33:37.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5023ae17:::benchmark_data_smithi188_113957_object53567:head# 2024-04-23T16:33:37.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5023e0e8:::benchmark_data_smithi188_113957_object96847:head# 2024-04-23T16:33:37.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5024242a:::benchmark_data_smithi188_113957_object127771:head# 2024-04-23T16:33:37.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50243bc8:::benchmark_data_smithi188_113957_object152477:head# 2024-04-23T16:33:37.668 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50244c0a:::benchmark_data_smithi188_113957_object146516:head# 2024-04-23T16:33:37.668 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50244ea6:::benchmark_data_smithi188_113957_object5239:head# 2024-04-23T16:33:37.669 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5024b2c5:::benchmark_data_smithi188_113957_object151640:head# 2024-04-23T16:33:37.669 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5024e505:::benchmark_data_smithi188_113957_object46064:head# 2024-04-23T16:33:37.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5026758a:::benchmark_data_smithi188_113957_object115505:head# 2024-04-23T16:33:37.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5026dd3c:::benchmark_data_smithi188_113957_object143309:head# 2024-04-23T16:33:37.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50272fe3:::benchmark_data_smithi188_113957_object139349:head# 2024-04-23T16:33:37.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502759e6:::benchmark_data_smithi188_113957_object124882:head# 2024-04-23T16:33:37.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50276568:::benchmark_data_smithi188_113957_object84720:head# 2024-04-23T16:33:37.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5027c812:::benchmark_data_smithi188_113957_object29768:head# 2024-04-23T16:33:37.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502847db:::benchmark_data_smithi188_113957_object4102:head# 2024-04-23T16:33:37.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50291df1:::benchmark_data_smithi188_113957_object100204:head# 2024-04-23T16:33:37.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5029d62d:::benchmark_data_smithi188_113957_object97228:head# 2024-04-23T16:33:37.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5029f49c:::benchmark_data_smithi188_113957_object128641:head# 2024-04-23T16:33:37.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502a4a49:::benchmark_data_smithi188_113957_object51156:head# 2024-04-23T16:33:37.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502a99cc:::benchmark_data_smithi188_113957_object75234:head# 2024-04-23T16:33:37.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502aa9d3:::benchmark_data_smithi188_113957_object119983:head# 2024-04-23T16:33:37.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502aae6c:::benchmark_data_smithi188_113957_object106654:head# 2024-04-23T16:33:37.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502af0e3:::benchmark_data_smithi188_113957_object23161:head# 2024-04-23T16:33:37.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502b5bc9:::benchmark_data_smithi188_113957_object79950:head# 2024-04-23T16:33:37.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502b5c9c:::benchmark_data_smithi188_113957_object74396:head# 2024-04-23T16:33:37.678 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502b5e03:::benchmark_data_smithi188_113957_object132188:head# 2024-04-23T16:33:37.678 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502c3c8f:::benchmark_data_smithi188_113957_object57260:head# 2024-04-23T16:33:37.679 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502c8aa1:::benchmark_data_smithi188_113957_object12772:head# 2024-04-23T16:33:37.679 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502cd8e5:::benchmark_data_smithi188_113957_object128251:head# 2024-04-23T16:33:37.680 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502e0ebc:::benchmark_data_smithi188_113957_object34542:head# 2024-04-23T16:33:37.680 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502f1fee:::benchmark_data_smithi188_113957_object29518:head# 2024-04-23T16:33:37.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502f3e10:::benchmark_data_smithi188_113957_object133225:head# 2024-04-23T16:33:37.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502f83f3:::benchmark_data_smithi188_113957_object62180:head# 2024-04-23T16:33:37.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502fbe27:::benchmark_data_smithi188_113957_object66158:head# 2024-04-23T16:33:37.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:502fd17f:::benchmark_data_smithi188_113957_object134092:head# 2024-04-23T16:33:37.683 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50300311:::benchmark_data_smithi188_113957_object27818:head# 2024-04-23T16:33:37.683 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:503008de:::benchmark_data_smithi188_113957_object87560:head# 2024-04-23T16:33:37.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50306ebe:::benchmark_data_smithi188_113957_object83540:head# 2024-04-23T16:33:37.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5030aa44:::benchmark_data_smithi188_113957_object120509:head# 2024-04-23T16:33:37.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5031e78a:::benchmark_data_smithi188_113957_object76842:head# 2024-04-23T16:33:37.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50320092:::benchmark_data_smithi188_113957_object7863:head# 2024-04-23T16:33:37.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50325392:::benchmark_data_smithi188_113957_object131703:head# 2024-04-23T16:33:37.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50328553:::benchmark_data_smithi188_113957_object108775:head# 2024-04-23T16:33:37.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50329319:::benchmark_data_smithi188_113957_object140154:head# 2024-04-23T16:33:37.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5032c908:::benchmark_data_smithi188_113957_object126186:head# 2024-04-23T16:33:37.689 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5033a60a:::benchmark_data_smithi188_113957_object96745:head# 2024-04-23T16:33:37.689 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5034046b:::benchmark_data_smithi188_113957_object74619:head# 2024-04-23T16:33:37.689 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5034c963:::benchmark_data_smithi188_113957_object59746:head# 2024-04-23T16:33:37.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50357126:::benchmark_data_smithi188_113957_object39807:head# 2024-04-23T16:33:37.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5036b891:::benchmark_data_smithi188_113957_object32984:head# 2024-04-23T16:33:37.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5036fb33:::benchmark_data_smithi188_113957_object26055:head# 2024-04-23T16:33:37.691 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:503746fa:::benchmark_data_smithi188_113957_object86256:head# 2024-04-23T16:33:37.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50379675:::benchmark_data_smithi188_113957_object33754:head# 2024-04-23T16:33:37.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50379b99:::benchmark_data_smithi188_113957_object116991:head# 2024-04-23T16:33:37.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50385eac:::benchmark_data_smithi188_113957_object6687:head# 2024-04-23T16:33:37.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50387064:::benchmark_data_smithi188_113957_object90738:head# 2024-04-23T16:33:37.694 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5038b827:::benchmark_data_smithi188_113957_object23313:head# 2024-04-23T16:33:37.694 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:503a28ba:::benchmark_data_smithi188_113957_object131031:head# 2024-04-23T16:33:37.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:503a94be:::benchmark_data_smithi188_113957_object76840:head# 2024-04-23T16:33:37.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:503a9ac9:::benchmark_data_smithi188_113957_object46868:head# 2024-04-23T16:33:37.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:503ad802:::benchmark_data_smithi188_113957_object123601:head# 2024-04-23T16:33:37.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:503b45d1:::benchmark_data_smithi188_113957_object60381:head# 2024-04-23T16:33:37.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:503cd587:::benchmark_data_smithi188_113957_object115473:head# 2024-04-23T16:33:37.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:503e07d6:::benchmark_data_smithi188_113957_object42571:head# 2024-04-23T16:33:37.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:503e1ff6:::benchmark_data_smithi188_113957_object68126:head# 2024-04-23T16:33:37.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:503ecd30:::benchmark_data_smithi188_113957_object52640:head# 2024-04-23T16:33:37.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:503fe0c2:::benchmark_data_smithi188_113957_object119609:head# 2024-04-23T16:33:37.699 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:503ff1f8:::benchmark_data_smithi188_113957_object45520:head# 2024-04-23T16:33:37.699 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504031e1:::benchmark_data_smithi188_113957_object68840:head# 2024-04-23T16:33:37.699 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50409a09:::benchmark_data_smithi188_113957_object116446:head# 2024-04-23T16:33:37.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50411833:::benchmark_data_smithi188_113957_object146925:head# 2024-04-23T16:33:37.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50413e16:::benchmark_data_smithi188_113957_object81943:head# 2024-04-23T16:33:37.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50415219:::benchmark_data_smithi188_113957_object130234:head# 2024-04-23T16:33:37.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5041ab5f:::benchmark_data_smithi188_113957_object128837:head# 2024-04-23T16:33:37.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5041bcdd:::benchmark_data_smithi188_113957_object111134:head# 2024-04-23T16:33:37.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50425d86:::benchmark_data_smithi188_113957_object71424:head# 2024-04-23T16:33:37.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50437003:::benchmark_data_smithi188_113957_object137997:head# 2024-04-23T16:33:37.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50439ad9:::benchmark_data_smithi188_113957_object151019:head# 2024-04-23T16:33:37.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5044671c:::benchmark_data_smithi188_113957_object101294:head# 2024-04-23T16:33:37.706 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:33:37.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5044cd0a:::benchmark_data_smithi188_113957_object153123:head# 2024-04-23T16:33:37.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5045696c:::benchmark_data_smithi188_113957_object10588:head# 2024-04-23T16:33:37.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50457304:::benchmark_data_smithi188_113957_object945:head# 2024-04-23T16:33:37.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5045ad5d:::benchmark_data_smithi188_113957_object13242:head# 2024-04-23T16:33:37.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5045c1bb:::benchmark_data_smithi188_113957_object43442:head# 2024-04-23T16:33:37.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5045e0a8:::benchmark_data_smithi188_113957_object50351:head# 2024-04-23T16:33:37.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:37.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:33:37.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50461fd5:::benchmark_data_smithi188_113957_object35755:head# 2024-04-23T16:33:37.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50464493:::benchmark_data_smithi188_113957_object76557:head# 2024-04-23T16:33:37.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5046c4a3:::benchmark_data_smithi188_113957_object70128:head# 2024-04-23T16:33:37.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50470e1a:::benchmark_data_smithi188_113957_object49211:head# 2024-04-23T16:33:37.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5047cd42:::benchmark_data_smithi188_113957_object80187:head# 2024-04-23T16:33:37.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50480269:::benchmark_data_smithi188_113957_object68752:head# 2024-04-23T16:33:37.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5048351f:::benchmark_data_smithi188_113957_object105326:head# 2024-04-23T16:33:37.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5048cff3:::benchmark_data_smithi188_113957_object51617:head# 2024-04-23T16:33:37.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5048f8d0:::benchmark_data_smithi188_113957_object3230:head# 2024-04-23T16:33:37.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50492d46:::benchmark_data_smithi188_113957_object8667:head# 2024-04-23T16:33:37.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504936de:::benchmark_data_smithi188_113957_object15157:head# 2024-04-23T16:33:37.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504988e6:::benchmark_data_smithi188_113957_object76719:head# 2024-04-23T16:33:37.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504a76d3:::benchmark_data_smithi188_113957_object149473:head# 2024-04-23T16:33:37.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504ad1f3:::benchmark_data_smithi188_113957_object116188:head# 2024-04-23T16:33:37.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504b45c0:::benchmark_data_smithi188_113957_object14702:head# 2024-04-23T16:33:37.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504bbeb7:::benchmark_data_smithi188_113957_object128681:head# 2024-04-23T16:33:37.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504c0f45:::benchmark_data_smithi188_113957_object102497:head# 2024-04-23T16:33:37.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504d1175:::benchmark_data_smithi188_113957_object136255:head# 2024-04-23T16:33:37.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504d3e2e:::benchmark_data_smithi188_113957_object93737:head# 2024-04-23T16:33:37.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504d8a41:::benchmark_data_smithi188_113957_object1726:head# 2024-04-23T16:33:37.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504dc19e:::benchmark_data_smithi188_113957_object107885:head# 2024-04-23T16:33:37.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504ddafa:::benchmark_data_smithi188_113957_object27239:head# 2024-04-23T16:33:37.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504de971:::benchmark_data_smithi188_113957_object39895:head# 2024-04-23T16:33:37.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504e2928:::benchmark_data_smithi188_113957_object96576:head# 2024-04-23T16:33:37.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504e6b0b:::benchmark_data_smithi188_113957_object40449:head# 2024-04-23T16:33:37.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504ec570:::benchmark_data_smithi188_113957_object45756:head# 2024-04-23T16:33:37.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504edf5d:::benchmark_data_smithi188_113957_object85298:head# 2024-04-23T16:33:37.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504f2f2b:::benchmark_data_smithi188_113957_object53222:head# 2024-04-23T16:33:37.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504f7e13:::benchmark_data_smithi188_113957_object28936:head# 2024-04-23T16:33:37.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504f9fc9:::benchmark_data_smithi188_113957_object89666:head# 2024-04-23T16:33:37.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:504fec48:::benchmark_data_smithi188_113957_object7513:head# 2024-04-23T16:33:37.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5050eb83:::benchmark_data_smithi188_113957_object133202:head# 2024-04-23T16:33:37.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5050f24e:::benchmark_data_smithi188_113957_object131196:head# 2024-04-23T16:33:37.725 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:505131c3:::benchmark_data_smithi188_113957_object78097:head# 2024-04-23T16:33:37.725 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50513cae:::benchmark_data_smithi188_113957_object103344:head# 2024-04-23T16:33:37.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50515d81:::benchmark_data_smithi188_113957_object29942:head# 2024-04-23T16:33:37.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5051ab3f:::benchmark_data_smithi188_113957_object40779:head# 2024-04-23T16:33:37.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5051abfe:::benchmark_data_smithi188_113957_object1392:head# 2024-04-23T16:33:37.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:505243a7:::benchmark_data_smithi188_113957_object150573:head# 2024-04-23T16:33:37.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:505285cc:::benchmark_data_smithi188_113957_object62791:head# 2024-04-23T16:33:37.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5052a38d:::benchmark_data_smithi188_113957_object110981:head# 2024-04-23T16:33:37.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:505491c0:::benchmark_data_smithi188_113957_object44575:head# 2024-04-23T16:33:37.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5054fadf:::benchmark_data_smithi188_113957_object21287:head# 2024-04-23T16:33:37.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50556f69:::benchmark_data_smithi188_113957_object4375:head# 2024-04-23T16:33:37.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5055718b:::benchmark_data_smithi188_113957_object21725:head# 2024-04-23T16:33:37.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50559dbb:::benchmark_data_smithi188_113957_object100478:head# 2024-04-23T16:33:37.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5055a719:::benchmark_data_smithi188_113957_object110645:head# 2024-04-23T16:33:37.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50563499:::benchmark_data_smithi188_113957_object96342:head# 2024-04-23T16:33:37.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5056370c:::benchmark_data_smithi188_113957_object139452:head# 2024-04-23T16:33:37.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:505648fb:::benchmark_data_smithi188_113957_object26430:head# 2024-04-23T16:33:37.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50564e6a:::benchmark_data_smithi188_113957_object149242:head# 2024-04-23T16:33:37.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5056c403:::benchmark_data_smithi188_113957_object74868:head# 2024-04-23T16:33:37.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50572568:::benchmark_data_smithi188_113957_object38810:head# 2024-04-23T16:33:37.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50573e8f:::benchmark_data_smithi188_113957_object103561:head# 2024-04-23T16:33:37.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5057b998:::benchmark_data_smithi188_113957_object56631:head# 2024-04-23T16:33:37.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5057ed55:::benchmark_data_smithi188_113957_object88595:head# 2024-04-23T16:33:37.735 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50585b2e:::benchmark_data_smithi188_113957_object10802:head# 2024-04-23T16:33:37.735 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5058830c:::benchmark_data_smithi188_113957_object32951:head# 2024-04-23T16:33:37.735 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:505901f7:::benchmark_data_smithi188_113957_object67458:head# 2024-04-23T16:33:37.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:505951a3:::benchmark_data_smithi188_113957_object23361:head# 2024-04-23T16:33:37.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5059f177:::benchmark_data_smithi188_113957_object20278:head# 2024-04-23T16:33:37.737 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:505a09e8:::benchmark_data_smithi188_113957_object72346:head# 2024-04-23T16:33:37.737 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:505a683d:::benchmark_data_smithi188_113957_object94888:head# 2024-04-23T16:33:37.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:505a6ab7:::benchmark_data_smithi188_113957_object20173:head# 2024-04-23T16:33:37.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:505ae16c:::benchmark_data_smithi188_113957_object30540:head# 2024-04-23T16:33:37.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:505b0f4b:::benchmark_data_smithi188_113957_object48688:head# 2024-04-23T16:33:37.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:505cd687:::benchmark_data_smithi188_113957_object29943:head# 2024-04-23T16:33:37.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:505cdcfc:::benchmark_data_smithi188_113957_object104929:head# 2024-04-23T16:33:37.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:505e2d3f:::benchmark_data_smithi188_113957_object101922:head# 2024-04-23T16:33:37.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:505f450e:::benchmark_data_smithi188_113957_object117573:head# 2024-04-23T16:33:37.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5060012a:::benchmark_data_smithi188_113957_object80149:head# 2024-04-23T16:33:37.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50602ab2:::benchmark_data_smithi188_113957_object139308:head# 2024-04-23T16:33:37.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50603644:::benchmark_data_smithi188_113957_object122095:head# 2024-04-23T16:33:37.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50605b5c:::benchmark_data_smithi188_113957_object131516:head# 2024-04-23T16:33:37.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50607485:::benchmark_data_smithi188_113957_object1189:head# 2024-04-23T16:33:37.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50608292:::benchmark_data_smithi188_113957_object74086:head# 2024-04-23T16:33:37.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50612f4b:::benchmark_data_smithi188_113957_object138188:head# 2024-04-23T16:33:37.744 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50614827:::benchmark_data_smithi188_113957_object90008:head# 2024-04-23T16:33:37.744 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50618274:::benchmark_data_smithi188_113957_object9078:head# 2024-04-23T16:33:37.745 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506197bb:::benchmark_data_smithi188_113957_object59575:head# 2024-04-23T16:33:37.745 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50621593:::benchmark_data_smithi188_113957_object95828:head# 2024-04-23T16:33:37.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5062e232:::benchmark_data_smithi188_113957_object29395:head# 2024-04-23T16:33:37.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506367dc:::benchmark_data_smithi188_113957_object26792:head# 2024-04-23T16:33:37.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50637634:::benchmark_data_smithi188_113957_object139593:head# 2024-04-23T16:33:37.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506392c1:::benchmark_data_smithi188_113957_object32204:head# 2024-04-23T16:33:37.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5063b02f:::benchmark_data_smithi188_113957_object39395:head# 2024-04-23T16:33:37.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5064140b:::benchmark_data_smithi188_113957_object122795:head# 2024-04-23T16:33:37.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50642564:::benchmark_data_smithi188_113957_object128471:head# 2024-04-23T16:33:37.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50648b42:::benchmark_data_smithi188_113957_object1040:head# 2024-04-23T16:33:37.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50648b5d:::benchmark_data_smithi188_113957_object110384:head# 2024-04-23T16:33:37.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5064b628:::benchmark_data_smithi188_113957_object116359:head# 2024-04-23T16:33:37.750 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5064f6f6:::benchmark_data_smithi188_113957_object58742:head# 2024-04-23T16:33:37.750 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5064fadd:::benchmark_data_smithi188_113957_object629:head# 2024-04-23T16:33:37.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506570d2:::benchmark_data_smithi188_113957_object83696:head# 2024-04-23T16:33:37.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50662eb1:::benchmark_data_smithi188_113957_object109449:head# 2024-04-23T16:33:37.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5066cc92:::benchmark_data_smithi188_113957_object60646:head# 2024-04-23T16:33:37.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5067a74c:::benchmark_data_smithi188_113957_object57981:head# 2024-04-23T16:33:37.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5067bf25:::benchmark_data_smithi188_113957_object8814:head# 2024-04-23T16:33:37.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5067c775:::benchmark_data_smithi188_113957_object90304:head# 2024-04-23T16:33:37.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506841eb:::benchmark_data_smithi188_113957_object1785:head# 2024-04-23T16:33:37.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50689834:::benchmark_data_smithi188_113957_object71203:head# 2024-04-23T16:33:37.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5068b38d:::benchmark_data_smithi188_113957_object6034:head# 2024-04-23T16:33:37.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5068cdca:::benchmark_data_smithi188_113957_object47201:head# 2024-04-23T16:33:37.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5068fc3a:::benchmark_data_smithi188_113957_object117619:head# 2024-04-23T16:33:37.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50692f5c:::benchmark_data_smithi188_113957_object129878:head# 2024-04-23T16:33:37.756 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506989f6:::benchmark_data_smithi188_113957_object46819:head# 2024-04-23T16:33:37.756 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5069f9f0:::benchmark_data_smithi188_113957_object61543:head# 2024-04-23T16:33:37.757 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506a6a57:::benchmark_data_smithi188_113957_object24313:head# 2024-04-23T16:33:37.757 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506ac60a:::benchmark_data_smithi188_113957_object46256:head# 2024-04-23T16:33:37.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506bb9ac:::benchmark_data_smithi188_113957_object69153:head# 2024-04-23T16:33:37.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506bf8b4:::benchmark_data_smithi188_113957_object10703:head# 2024-04-23T16:33:37.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506c6a1f:::benchmark_data_smithi188_113957_object100851:head# 2024-04-23T16:33:37.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506cb347:::benchmark_data_smithi188_113957_object79517:head# 2024-04-23T16:33:37.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506d17d7:::benchmark_data_smithi188_113957_object66729:head# 2024-04-23T16:33:37.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506d3c6a:::benchmark_data_smithi188_113957_object143692:head# 2024-04-23T16:33:37.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506d8a5a:::benchmark_data_smithi188_113957_object134588:head# 2024-04-23T16:33:37.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506e374f:::benchmark_data_smithi188_113957_object56080:head# 2024-04-23T16:33:37.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506e9bcb:::benchmark_data_smithi188_113957_object62307:head# 2024-04-23T16:33:37.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506eb1de:::benchmark_data_smithi188_113957_object52049:head# 2024-04-23T16:33:37.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506f659e:::benchmark_data_smithi188_113957_object93815:head# 2024-04-23T16:33:37.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:506fe96e:::benchmark_data_smithi188_113957_object14674:head# 2024-04-23T16:33:37.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50706f2e:::benchmark_data_smithi188_113957_object86000:head# 2024-04-23T16:33:37.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5070a063:::benchmark_data_smithi188_113957_object101740:head# 2024-04-23T16:33:37.764 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5071ae6d:::benchmark_data_smithi188_113957_object88574:head# 2024-04-23T16:33:37.764 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5072194c:::benchmark_data_smithi188_113957_object136179:head# 2024-04-23T16:33:37.765 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5072ad2f:::benchmark_data_smithi188_113957_object106860:head# 2024-04-23T16:33:37.765 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5072dded:::benchmark_data_smithi188_113957_object36605:head# 2024-04-23T16:33:37.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5072e5ca:::benchmark_data_smithi188_113957_object141157:head# 2024-04-23T16:33:37.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5073b2c0:::benchmark_data_smithi188_113957_object58190:head# 2024-04-23T16:33:37.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5073b91a:::benchmark_data_smithi188_113957_object148432:head# 2024-04-23T16:33:37.767 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5074295a:::benchmark_data_smithi188_113957_object116171:head# 2024-04-23T16:33:37.767 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50748b41:::benchmark_data_smithi188_113957_object34942:head# 2024-04-23T16:33:37.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5074fdf1:::benchmark_data_smithi188_113957_object51330:head# 2024-04-23T16:33:37.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50754044:::benchmark_data_smithi188_113957_object64612:head# 2024-04-23T16:33:37.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5075bdc2:::benchmark_data_smithi188_113957_object93348:head# 2024-04-23T16:33:37.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5075bfe2:::benchmark_data_smithi188_113957_object60794:head# 2024-04-23T16:33:37.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5075ea21:::benchmark_data_smithi188_113957_object31129:head# 2024-04-23T16:33:37.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50774ccc:::benchmark_data_smithi188_113957_object47641:head# 2024-04-23T16:33:37.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50776ff6:::benchmark_data_smithi188_113957_object32138:head# 2024-04-23T16:33:37.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50778d59:::benchmark_data_smithi188_113957_object70920:head# 2024-04-23T16:33:37.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50782c02:::benchmark_data_smithi188_113957_object19126:head# 2024-04-23T16:33:37.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507893d4:::benchmark_data_smithi188_113957_object140279:head# 2024-04-23T16:33:37.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50795a3e:::benchmark_data_smithi188_113957_object133862:head# 2024-04-23T16:33:37.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50799145:::benchmark_data_smithi188_113957_object3241:head# 2024-04-23T16:33:37.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507a445f:::benchmark_data_smithi188_113957_object118906:head# 2024-04-23T16:33:37.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507a68cf:::benchmark_data_smithi188_113957_object122511:head# 2024-04-23T16:33:37.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507ae352:::benchmark_data_smithi188_113957_object89624:head# 2024-04-23T16:33:37.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507b3f5d:::benchmark_data_smithi188_113957_object135301:head# 2024-04-23T16:33:37.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507b8295:::benchmark_data_smithi188_113957_object8698:head# 2024-04-23T16:33:37.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507c0390:::benchmark_data_smithi188_113957_object44457:head# 2024-04-23T16:33:37.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507c29a9:::benchmark_data_smithi188_113957_object117352:head# 2024-04-23T16:33:37.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507cc113:::benchmark_data_smithi188_113957_object42461:head# 2024-04-23T16:33:37.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507cdae0:::benchmark_data_smithi188_113957_object92776:head# 2024-04-23T16:33:37.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507d28e7:::benchmark_data_smithi188_113957_object114068:head# 2024-04-23T16:33:37.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507d4614:::benchmark_data_smithi188_113957_object95850:head# 2024-04-23T16:33:37.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507d64d0:::benchmark_data_smithi188_113957_object42565:head# 2024-04-23T16:33:37.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507d7d3f:::benchmark_data_smithi188_113957_object27226:head# 2024-04-23T16:33:37.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507d8bd8:::benchmark_data_smithi188_113957_object99958:head# 2024-04-23T16:33:37.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507d922c:::benchmark_data_smithi188_113957_object48087:head# 2024-04-23T16:33:37.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507d9fb7:::benchmark_data_smithi188_113957_object74412:head# 2024-04-23T16:33:37.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507dc51f:::benchmark_data_smithi188_113957_object6236:head# 2024-04-23T16:33:37.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507e9da2:::benchmark_data_smithi188_113957_object6054:head# 2024-04-23T16:33:37.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507ea31c:::benchmark_data_smithi188_113957_object11266:head# 2024-04-23T16:33:37.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507f01d1:::benchmark_data_smithi188_113957_object133982:head# 2024-04-23T16:33:37.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:507f8dbf:::benchmark_data_smithi188_113957_object19910:head# 2024-04-23T16:33:37.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5080520b:::benchmark_data_smithi188_113957_object26408:head# 2024-04-23T16:33:37.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50805d9d:::benchmark_data_smithi188_113957_object54975:head# 2024-04-23T16:33:37.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50805f3c:::benchmark_data_smithi188_113957_object34641:head# 2024-04-23T16:33:37.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50821d4f:::benchmark_data_smithi188_113957_object7574:head# 2024-04-23T16:33:37.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50823d2e:::benchmark_data_smithi188_113957_object137780:head# 2024-04-23T16:33:37.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5082ec9b:::benchmark_data_smithi188_113957_object126505:head# 2024-04-23T16:33:37.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50834397:::benchmark_data_smithi188_113957_object138754:head# 2024-04-23T16:33:37.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50834831:::benchmark_data_smithi188_113957_object18823:head# 2024-04-23T16:33:37.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50834e5c:::benchmark_data_smithi188_113957_object36267:head# 2024-04-23T16:33:37.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50842811:::benchmark_data_smithi188_113957_object76889:head# 2024-04-23T16:33:37.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50846144:::benchmark_data_smithi188_113957_object64633:head# 2024-04-23T16:33:37.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5085238f:::benchmark_data_smithi188_113957_object42248:head# 2024-04-23T16:33:37.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5085799d:::benchmark_data_smithi188_113957_object135355:head# 2024-04-23T16:33:37.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:508617ce:::benchmark_data_smithi188_113957_object133221:head# 2024-04-23T16:33:37.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5086f448:::benchmark_data_smithi188_113957_object53933:head# 2024-04-23T16:33:37.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50872fda:::benchmark_data_smithi188_113957_object2573:head# 2024-04-23T16:33:37.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50874d5f:::benchmark_data_smithi188_113957_object17235:head# 2024-04-23T16:33:37.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:508757f1:::benchmark_data_smithi188_113957_object19645:head# 2024-04-23T16:33:37.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50879d6a:::benchmark_data_smithi188_113957_object29530:head# 2024-04-23T16:33:37.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5088b44a:::benchmark_data_smithi188_113957_object101853:head# 2024-04-23T16:33:37.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50891d95:::benchmark_data_smithi188_113957_object107526:head# 2024-04-23T16:33:37.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:508b589c:::benchmark_data_smithi188_113957_object14397:head# 2024-04-23T16:33:37.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:508ba08a:::benchmark_data_smithi188_113957_object83239:head# 2024-04-23T16:33:37.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:508bdbce:::benchmark_data_smithi188_113957_object61287:head# 2024-04-23T16:33:37.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:508c4b6c:::benchmark_data_smithi188_113957_object130221:head# 2024-04-23T16:33:37.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:508cc536:::benchmark_data_smithi188_113957_object130163:head# 2024-04-23T16:33:37.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:508d5d14:::benchmark_data_smithi188_113957_object55024:head# 2024-04-23T16:33:37.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:508e03ec:::benchmark_data_smithi188_113957_object129594:head# 2024-04-23T16:33:37.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:508e2e54:::benchmark_data_smithi188_113957_object127838:head# 2024-04-23T16:33:37.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:508e554d:::benchmark_data_smithi188_113957_object131477:head# 2024-04-23T16:33:37.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:508ec647:::benchmark_data_smithi188_113957_object64908:head# 2024-04-23T16:33:37.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:508f38ad:::benchmark_data_smithi188_113957_object63947:head# 2024-04-23T16:33:37.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:508fda59:::benchmark_data_smithi188_113957_object30617:head# 2024-04-23T16:33:37.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:508ff668:::benchmark_data_smithi188_113957_object85036:head# 2024-04-23T16:33:37.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50900b81:::benchmark_data_smithi188_113957_object37830:head# 2024-04-23T16:33:37.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509020f9:::benchmark_data_smithi188_113957_object114488:head# 2024-04-23T16:33:37.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509064f6:::benchmark_data_smithi188_113957_object106536:head# 2024-04-23T16:33:37.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5090a6f0:::benchmark_data_smithi188_113957_object146530:head# 2024-04-23T16:33:37.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5090c48d:::benchmark_data_smithi188_113957_object84657:head# 2024-04-23T16:33:37.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50912471:::benchmark_data_smithi188_113957_object110195:head# 2024-04-23T16:33:37.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50914f70:::benchmark_data_smithi188_113957_object27765:head# 2024-04-23T16:33:37.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5091dcf0:::benchmark_data_smithi188_113957_object112394:head# 2024-04-23T16:33:37.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50921a51:::benchmark_data_smithi188_113957_object145073:head# 2024-04-23T16:33:37.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50925286:::benchmark_data_smithi188_113957_object95564:head# 2024-04-23T16:33:37.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50928b68:::benchmark_data_smithi188_113957_object142095:head# 2024-04-23T16:33:37.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5092904c:::benchmark_data_smithi188_113957_object20514:head# 2024-04-23T16:33:37.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5092e29b:::benchmark_data_smithi188_113957_object103922:head# 2024-04-23T16:33:37.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50936750:::benchmark_data_smithi188_113957_object107972:head# 2024-04-23T16:33:37.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50943dad:::benchmark_data_smithi188_113957_object54429:head# 2024-04-23T16:33:37.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5096aa02:::benchmark_data_smithi188_113957_object94559:head# 2024-04-23T16:33:37.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5096ed23:::benchmark_data_smithi188_113957_object43034:head# 2024-04-23T16:33:37.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509761c4:::benchmark_data_smithi188_113957_object56284:head# 2024-04-23T16:33:37.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50977a8f:::benchmark_data_smithi188_113957_object65493:head# 2024-04-23T16:33:37.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509806dd:::benchmark_data_smithi188_113957_object91409:head# 2024-04-23T16:33:37.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50989e27:::benchmark_data_smithi188_113957_object152942:head# 2024-04-23T16:33:37.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5099521d:::benchmark_data_smithi188_113957_object59301:head# 2024-04-23T16:33:37.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50995e32:::benchmark_data_smithi188_113957_object33281:head# 2024-04-23T16:33:37.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509abeb5:::benchmark_data_smithi188_113957_object51552:head# 2024-04-23T16:33:37.810 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:33:37.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509afb05:::benchmark_data_smithi188_113957_object136407:head# 2024-04-23T16:33:37.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509b16ea:::benchmark_data_smithi188_113957_object29511:head# 2024-04-23T16:33:37.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509b9bb2:::benchmark_data_smithi188_113957_object140083:head# 2024-04-23T16:33:37.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509bd9bc:::benchmark_data_smithi188_113957_object55965:head# 2024-04-23T16:33:37.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509bf0bc:::benchmark_data_smithi188_113957_object128048:head# 2024-04-23T16:33:37.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509c0a03:::benchmark_data_smithi188_113957_object92196:head# 2024-04-23T16:33:37.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509c7e10:::benchmark_data_smithi188_113957_object42119:head# 2024-04-23T16:33:37.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509c7eea:::benchmark_data_smithi188_113957_object55182:head# 2024-04-23T16:33:37.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509c928f:::benchmark_data_smithi188_113957_object87201:head# 2024-04-23T16:33:37.814 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509cb43f:::benchmark_data_smithi188_113957_object138546:head# 2024-04-23T16:33:37.814 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509d351d:::benchmark_data_smithi188_113957_object153172:head# 2024-04-23T16:33:37.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509de4a1:::benchmark_data_smithi188_113957_object128145:head# 2024-04-23T16:33:37.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509e1e82:::benchmark_data_smithi188_113957_object47070:head# 2024-04-23T16:33:37.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509e484e:::benchmark_data_smithi188_113957_object30832:head# 2024-04-23T16:33:37.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509e5a73:::benchmark_data_smithi188_113957_object150382:head# 2024-04-23T16:33:37.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509e754d:::benchmark_data_smithi188_113957_object127441:head# 2024-04-23T16:33:37.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509f1ac2:::benchmark_data_smithi188_113957_object130478:head# 2024-04-23T16:33:37.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509f5bc1:::benchmark_data_smithi188_113957_object16986:head# 2024-04-23T16:33:37.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509f7986:::benchmark_data_smithi188_113957_object21285:head# 2024-04-23T16:33:37.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509fb181:::benchmark_data_smithi188_113957_object51414:head# 2024-04-23T16:33:37.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:509fe1c8:::benchmark_data_smithi188_113957_object42018:head# 2024-04-23T16:33:37.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a0461b:::benchmark_data_smithi188_113957_object51213:head# 2024-04-23T16:33:37.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a0b4d8:::benchmark_data_smithi188_113957_object85485:head# 2024-04-23T16:33:37.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a176d9:::benchmark_data_smithi188_113957_object87817:head# 2024-04-23T16:33:37.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a198d5:::benchmark_data_smithi188_113957_object109411:head# 2024-04-23T16:33:37.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a1eae3:::benchmark_data_smithi188_113957_object98239:head# 2024-04-23T16:33:37.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a22734:::benchmark_data_smithi188_113957_object18181:head# 2024-04-23T16:33:37.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a2ef80:::benchmark_data_smithi188_113957_object61204:head# 2024-04-23T16:33:37.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a46118:::benchmark_data_smithi188_113957_object82129:head# 2024-04-23T16:33:37.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a51a80:::benchmark_data_smithi188_113957_object63439:head# 2024-04-23T16:33:37.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a5e674:::benchmark_data_smithi188_113957_object97286:head# 2024-04-23T16:33:37.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a6b962:::benchmark_data_smithi188_113957_object122839:head# 2024-04-23T16:33:37.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a72697:::benchmark_data_smithi188_113957_object32568:head# 2024-04-23T16:33:37.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a793e4:::benchmark_data_smithi188_113957_object8723:head# 2024-04-23T16:33:37.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a7d66b:::benchmark_data_smithi188_113957_object62449:head# 2024-04-23T16:33:37.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a8359a:::benchmark_data_smithi188_113957_object19695:head# 2024-04-23T16:33:37.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a88d6f:::benchmark_data_smithi188_113957_object33236:head# 2024-04-23T16:33:37.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a8ef14:::benchmark_data_smithi188_113957_object128896:head# 2024-04-23T16:33:37.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a93dc4:::benchmark_data_smithi188_113957_object23215:head# 2024-04-23T16:33:37.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a9507b:::benchmark_data_smithi188_113957_object149233:head# 2024-04-23T16:33:37.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a958de:::benchmark_data_smithi188_113957_object124414:head# 2024-04-23T16:33:37.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a96aa2:::benchmark_data_smithi188_113957_object67747:head# 2024-04-23T16:33:37.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a9799e:::benchmark_data_smithi188_113957_object8950:head# 2024-04-23T16:33:37.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50a9929a:::benchmark_data_smithi188_113957_object18801:head# 2024-04-23T16:33:37.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50aa0416:::benchmark_data_smithi188_113957_object97149:head# 2024-04-23T16:33:37.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50aa1193:::benchmark_data_smithi188_113957_object37545:head# 2024-04-23T16:33:37.832 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50aacbd3:::benchmark_data_smithi188_113957_object147585:head# 2024-04-23T16:33:37.832 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ab1184:::benchmark_data_smithi188_113957_object150085:head# 2024-04-23T16:33:37.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ab46f4:::benchmark_data_smithi188_113957_object84161:head# 2024-04-23T16:33:37.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50abae45:::benchmark_data_smithi188_113957_object120439:head# 2024-04-23T16:33:37.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ac2d15:::benchmark_data_smithi188_113957_object90240:head# 2024-04-23T16:33:37.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ac4dd4:::benchmark_data_smithi188_113957_object70956:head# 2024-04-23T16:33:37.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ac6cf7:::benchmark_data_smithi188_113957_object88319:head# 2024-04-23T16:33:37.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50acaa9b:::benchmark_data_smithi188_113957_object14269:head# 2024-04-23T16:33:37.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ad4803:::benchmark_data_smithi188_113957_object117131:head# 2024-04-23T16:33:37.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ad5f27:::benchmark_data_smithi188_113957_object9960:head# 2024-04-23T16:33:37.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ad8004:::benchmark_data_smithi188_113957_object128849:head# 2024-04-23T16:33:37.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ae7cd7:::benchmark_data_smithi188_113957_object140531:head# 2024-04-23T16:33:37.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50aeafb9:::benchmark_data_smithi188_113957_object130775:head# 2024-04-23T16:33:37.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50aeb709:::benchmark_data_smithi188_113957_object77088:head# 2024-04-23T16:33:37.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50aee2e9:::benchmark_data_smithi188_113957_object148565:head# 2024-04-23T16:33:37.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50afadca:::benchmark_data_smithi188_113957_object16587:head# 2024-04-23T16:33:37.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b0ae12:::benchmark_data_smithi188_113957_object99872:head# 2024-04-23T16:33:37.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b0c334:::benchmark_data_smithi188_113957_object10037:head# 2024-04-23T16:33:37.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b0dcdd:::benchmark_data_smithi188_113957_object89554:head# 2024-04-23T16:33:37.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b1330e:::benchmark_data_smithi188_113957_object150275:head# 2024-04-23T16:33:37.840 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:37.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b22b7c:::benchmark_data_smithi188_113957_object71821:head# 2024-04-23T16:33:37.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b37df1:::benchmark_data_smithi188_113957_object66388:head# 2024-04-23T16:33:37.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b3d2f4:::benchmark_data_smithi188_113957_object16105:head# 2024-04-23T16:33:37.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b47438:::benchmark_data_smithi188_113957_object123662:head# 2024-04-23T16:33:37.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b5d6d4:::benchmark_data_smithi188_113957_object62519:head# 2024-04-23T16:33:37.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b6222c:::benchmark_data_smithi188_113957_object94240:head# 2024-04-23T16:33:37.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b71857:::benchmark_data_smithi188_113957_object101407:head# 2024-04-23T16:33:37.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b74f0c:::benchmark_data_smithi188_113957_object60688:head# 2024-04-23T16:33:37.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b7b957:::benchmark_data_smithi188_113957_object79502:head# 2024-04-23T16:33:37.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b7bf54:::benchmark_data_smithi188_113957_object74548:head# 2024-04-23T16:33:37.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b7ffa5:::benchmark_data_smithi188_113957_object127008:head# 2024-04-23T16:33:37.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b90f33:::benchmark_data_smithi188_113957_object29195:head# 2024-04-23T16:33:37.845 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b93fec:::benchmark_data_smithi188_113957_object27366:head# 2024-04-23T16:33:37.845 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b95026:::benchmark_data_smithi188_113957_object82605:head# 2024-04-23T16:33:37.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b9647e:::benchmark_data_smithi188_113957_object24515:head# 2024-04-23T16:33:37.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b9a36c:::benchmark_data_smithi188_113957_object133304:head# 2024-04-23T16:33:37.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b9bcc7:::benchmark_data_smithi188_113957_object106449:head# 2024-04-23T16:33:37.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50b9c070:::benchmark_data_smithi188_113957_object32343:head# 2024-04-23T16:33:37.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ba6d61:::benchmark_data_smithi188_113957_object35095:head# 2024-04-23T16:33:37.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50bb4b9c:::benchmark_data_smithi188_113957_object100626:head# 2024-04-23T16:33:37.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50bb77eb:::benchmark_data_smithi188_113957_object59669:head# 2024-04-23T16:33:37.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50bb8a3c:::benchmark_data_smithi188_113957_object130104:head# 2024-04-23T16:33:37.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50bc4811:::benchmark_data_smithi188_113957_object135037:head# 2024-04-23T16:33:37.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50bc6a12:::benchmark_data_smithi188_113957_object117803:head# 2024-04-23T16:33:37.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50bca88c:::benchmark_data_smithi188_113957_object105866:head# 2024-04-23T16:33:37.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50bcc6d7:::benchmark_data_smithi188_113957_object126932:head# 2024-04-23T16:33:37.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50bd1d2e:::benchmark_data_smithi188_113957_object24001:head# 2024-04-23T16:33:37.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50bd79d0:::benchmark_data_smithi188_113957_object48082:head# 2024-04-23T16:33:37.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50be833a:::benchmark_data_smithi188_113957_object21844:head# 2024-04-23T16:33:37.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50bec0c5:::benchmark_data_smithi188_113957_object42622:head# 2024-04-23T16:33:37.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50bf0199:::benchmark_data_smithi188_113957_object76125:head# 2024-04-23T16:33:37.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50bf4889:::benchmark_data_smithi188_113957_object55212:head# 2024-04-23T16:33:37.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50bf586a:::benchmark_data_smithi188_113957_object7163:head# 2024-04-23T16:33:37.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50bf5b3e:::benchmark_data_smithi188_113957_object5957:head# 2024-04-23T16:33:37.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50bfa7db:::benchmark_data_smithi188_113957_object43602:head# 2024-04-23T16:33:37.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50bfb612:::benchmark_data_smithi188_113957_object152811:head# 2024-04-23T16:33:37.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50bfed26:::benchmark_data_smithi188_113957_object83597:head# 2024-04-23T16:33:37.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50bfeeef:::benchmark_data_smithi188_113957_object4024:head# 2024-04-23T16:33:37.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50c04f31:::benchmark_data_smithi188_113957_object134025:head# 2024-04-23T16:33:37.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50c0ab36:::benchmark_data_smithi188_113957_object76340:head# 2024-04-23T16:33:37.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50c1f5d0:::benchmark_data_smithi188_113957_object148126:head# 2024-04-23T16:33:37.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50c230b8:::benchmark_data_smithi188_113957_object25956:head# 2024-04-23T16:33:37.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50c25179:::benchmark_data_smithi188_113957_object147448:head# 2024-04-23T16:33:37.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50c2962d:::benchmark_data_smithi188_113957_object23572:head# 2024-04-23T16:33:37.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50c347aa:::benchmark_data_smithi188_113957_object118747:head# 2024-04-23T16:33:37.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50c39a5f:::benchmark_data_smithi188_113957_object76959:head# 2024-04-23T16:33:37.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50c4f7cc:::benchmark_data_smithi188_113957_object4635:head# 2024-04-23T16:33:37.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50c5100e:::benchmark_data_smithi188_113957_object116872:head# 2024-04-23T16:33:37.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50c6513c:::benchmark_data_smithi188_113957_object30944:head# 2024-04-23T16:33:37.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50c76aeb:::benchmark_data_smithi188_113957_object130883:head# 2024-04-23T16:33:37.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50c87038:::benchmark_data_smithi188_113957_object60961:head# 2024-04-23T16:33:37.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50c89de4:::benchmark_data_smithi188_113957_object45765:head# 2024-04-23T16:33:37.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50c8e8fd:::benchmark_data_smithi188_113957_object101568:head# 2024-04-23T16:33:37.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50c930d7:::benchmark_data_smithi188_113957_object16995:head# 2024-04-23T16:33:37.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ca2df1:::benchmark_data_smithi188_113957_object144709:head# 2024-04-23T16:33:37.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50cb8e8c:::benchmark_data_smithi188_113957_object107427:head# 2024-04-23T16:33:37.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50cbbf73:::benchmark_data_smithi188_113957_object100517:head# 2024-04-23T16:33:37.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50cbd2df:::benchmark_data_smithi188_113957_object55600:head# 2024-04-23T16:33:37.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50cc74e9:::benchmark_data_smithi188_113957_object19085:head# 2024-04-23T16:33:37.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ccc14a:::benchmark_data_smithi188_113957_object25947:head# 2024-04-23T16:33:37.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50cd021a:::benchmark_data_smithi188_113957_object132836:head# 2024-04-23T16:33:37.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50cd21d5:::benchmark_data_smithi188_113957_object24349:head# 2024-04-23T16:33:37.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50cd5e9c:::benchmark_data_smithi188_113957_object49792:head# 2024-04-23T16:33:37.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ce2913:::benchmark_data_smithi188_113957_object84632:head# 2024-04-23T16:33:37.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ce33c5:::benchmark_data_smithi188_113957_object25252:head# 2024-04-23T16:33:37.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50cf209a:::benchmark_data_smithi188_113957_object117323:head# 2024-04-23T16:33:37.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50cf5ac1:::benchmark_data_smithi188_113957_object27807:head# 2024-04-23T16:33:37.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d0943b:::benchmark_data_smithi188_113957_object5648:head# 2024-04-23T16:33:37.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d0bbda:::benchmark_data_smithi188_113957_object152014:head# 2024-04-23T16:33:37.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d105a4:::benchmark_data_smithi188_113957_object16457:head# 2024-04-23T16:33:37.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d1b595:::benchmark_data_smithi188_113957_object41967:head# 2024-04-23T16:33:37.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d1d046:::benchmark_data_smithi188_113957_object138502:head# 2024-04-23T16:33:37.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d1d5dd:::benchmark_data_smithi188_113957_object45278:head# 2024-04-23T16:33:37.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d1e7dd:::benchmark_data_smithi188_113957_object2791:head# 2024-04-23T16:33:37.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d1fa70:::benchmark_data_smithi188_113957_object33245:head# 2024-04-23T16:33:37.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d21dde:::benchmark_data_smithi188_113957_object59092:head# 2024-04-23T16:33:37.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d25ccf:::benchmark_data_smithi188_113957_object141121:head# 2024-04-23T16:33:37.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d30e32:::benchmark_data_smithi188_113957_object79728:head# 2024-04-23T16:33:37.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d3162c:::benchmark_data_smithi188_113957_object143625:head# 2024-04-23T16:33:37.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d32701:::benchmark_data_smithi188_113957_object76157:head# 2024-04-23T16:33:37.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d35d68:::benchmark_data_smithi188_113957_object23744:head# 2024-04-23T16:33:37.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d37f3c:::benchmark_data_smithi188_113957_object26204:head# 2024-04-23T16:33:37.875 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d3fd2f:::benchmark_data_smithi188_113957_object147675:head# 2024-04-23T16:33:37.876 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d48db7:::benchmark_data_smithi188_113957_object37898:head# 2024-04-23T16:33:37.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d53139:::benchmark_data_smithi188_113957_object30248:head# 2024-04-23T16:33:37.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d6939e:::benchmark_data_smithi188_113957_object126313:head# 2024-04-23T16:33:37.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d69fdd:::benchmark_data_smithi188_113957_object110021:head# 2024-04-23T16:33:37.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d6ab21:::benchmark_data_smithi188_113957_object17317:head# 2024-04-23T16:33:37.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d6dcc0:::benchmark_data_smithi188_113957_object28310:head# 2024-04-23T16:33:37.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d732a1:::benchmark_data_smithi188_113957_object11703:head# 2024-04-23T16:33:37.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d7f000:::benchmark_data_smithi188_113957_object55216:head# 2024-04-23T16:33:37.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50d8e23e:::benchmark_data_smithi188_113957_object87177:head# 2024-04-23T16:33:37.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50da0847:::benchmark_data_smithi188_113957_object86499:head# 2024-04-23T16:33:37.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50da1354:::benchmark_data_smithi188_113957_object80704:head# 2024-04-23T16:33:37.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50da2d5a:::benchmark_data_smithi188_113957_object63224:head# 2024-04-23T16:33:37.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50da2fdc:::benchmark_data_smithi188_113957_object112274:head# 2024-04-23T16:33:37.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50da5071:::benchmark_data_smithi188_113957_object123536:head# 2024-04-23T16:33:37.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50dba284:::benchmark_data_smithi188_113957_object86423:head# 2024-04-23T16:33:37.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50dbda5f:::benchmark_data_smithi188_113957_object34996:head# 2024-04-23T16:33:37.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50dbeb67:::benchmark_data_smithi188_113957_object47824:head# 2024-04-23T16:33:37.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50dc2110:::benchmark_data_smithi188_113957_object152834:head# 2024-04-23T16:33:37.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ddac4b:::benchmark_data_smithi188_113957_object67637:head# 2024-04-23T16:33:37.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ddc280:::benchmark_data_smithi188_113957_object110900:head# 2024-04-23T16:33:37.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50de6d75:::benchmark_data_smithi188_113957_object43449:head# 2024-04-23T16:33:37.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50dea46f:::benchmark_data_smithi188_113957_object101872:head# 2024-04-23T16:33:37.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50deb021:::benchmark_data_smithi188_113957_object151729:head# 2024-04-23T16:33:37.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50df0c7a:::benchmark_data_smithi188_113957_object133364:head# 2024-04-23T16:33:37.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50df82ca:::benchmark_data_smithi188_113957_object121500:head# 2024-04-23T16:33:37.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50dfae04:::benchmark_data_smithi188_113957_object99080:head# 2024-04-23T16:33:37.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50e030d4:::benchmark_data_smithi188_113957_object2098:head# 2024-04-23T16:33:37.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50e07c36:::benchmark_data_smithi188_113957_object4862:head# 2024-04-23T16:33:37.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50e0b423:::benchmark_data_smithi188_113957_object18739:head# 2024-04-23T16:33:37.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50e25653:::benchmark_data_smithi188_113957_object35631:head# 2024-04-23T16:33:37.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50e2fa6d:::benchmark_data_smithi188_113957_object48628:head# 2024-04-23T16:33:37.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50e32006:::benchmark_data_smithi188_113957_object104793:head# 2024-04-23T16:33:37.891 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50e35246:::benchmark_data_smithi188_113957_object134241:head# 2024-04-23T16:33:37.891 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50e44616:::benchmark_data_smithi188_113957_object151172:head# 2024-04-23T16:33:37.892 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50e51dbe:::benchmark_data_smithi188_113957_object97806:head# 2024-04-23T16:33:37.892 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50e54270:::benchmark_data_smithi188_113957_object122134:head# 2024-04-23T16:33:37.892 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50e568c5:::benchmark_data_smithi188_113957_object82443:head# 2024-04-23T16:33:37.893 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50e56a6b:::benchmark_data_smithi188_113957_object2703:head# 2024-04-23T16:33:37.893 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50e6cbd4:::benchmark_data_smithi188_113957_object130083:head# 2024-04-23T16:33:37.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50e73057:::benchmark_data_smithi188_113957_object134899:head# 2024-04-23T16:33:37.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50e75302:::benchmark_data_smithi188_113957_object983:head# 2024-04-23T16:33:37.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50e9bd26:::benchmark_data_smithi188_113957_object105003:head# 2024-04-23T16:33:37.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50e9f198:::benchmark_data_smithi188_113957_object2927:head# 2024-04-23T16:33:37.896 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50eae8e3:::benchmark_data_smithi188_113957_object56419:head# 2024-04-23T16:33:37.896 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ee63e8:::benchmark_data_smithi188_113957_object88228:head# 2024-04-23T16:33:37.896 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ee7137:::benchmark_data_smithi188_113957_object37107:head# 2024-04-23T16:33:37.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50eef07f:::benchmark_data_smithi188_113957_object143429:head# 2024-04-23T16:33:37.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ef14c6:::benchmark_data_smithi188_113957_object123242:head# 2024-04-23T16:33:37.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ef94c0:::benchmark_data_smithi188_113957_object150358:head# 2024-04-23T16:33:37.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ef988e:::benchmark_data_smithi188_113957_object106109:head# 2024-04-23T16:33:37.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50efbf2c:::benchmark_data_smithi188_113957_object39738:head# 2024-04-23T16:33:37.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f0005d:::benchmark_data_smithi188_113957_object29523:head# 2024-04-23T16:33:37.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f080d3:::benchmark_data_smithi188_113957_object28677:head# 2024-04-23T16:33:37.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f0e33c:::benchmark_data_smithi188_113957_object131552:head# 2024-04-23T16:33:37.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f13ecc:::benchmark_data_smithi188_113957_object18647:head# 2024-04-23T16:33:37.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f17453:::benchmark_data_smithi188_113957_object117908:head# 2024-04-23T16:33:37.904 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f1a3a1:::benchmark_data_smithi188_113957_object69265:head# 2024-04-23T16:33:37.904 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f1f4c1:::benchmark_data_smithi188_113957_object96829:head# 2024-04-23T16:33:37.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f25fcf:::benchmark_data_smithi188_113957_object41929:head# 2024-04-23T16:33:37.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f2a753:::benchmark_data_smithi188_113957_object150972:head# 2024-04-23T16:33:37.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f2c852:::benchmark_data_smithi188_113957_object116010:head# 2024-04-23T16:33:37.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f33230:::benchmark_data_smithi188_113957_object20631:head# 2024-04-23T16:33:37.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f37e96:::benchmark_data_smithi188_113957_object74085:head# 2024-04-23T16:33:37.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f3b9bc:::benchmark_data_smithi188_113957_object134653:head# 2024-04-23T16:33:37.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f61f61:::benchmark_data_smithi188_113957_object34954:head# 2024-04-23T16:33:37.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f64718:::benchmark_data_smithi188_113957_object122008:head# 2024-04-23T16:33:37.909 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f65d09:::benchmark_data_smithi188_113957_object83858:head# 2024-04-23T16:33:37.909 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f67489:::benchmark_data_smithi188_113957_object64123:head# 2024-04-23T16:33:37.910 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f6d393:::benchmark_data_smithi188_113957_object24994:head# 2024-04-23T16:33:37.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f70881:::benchmark_data_smithi188_113957_object66138:head# 2024-04-23T16:33:37.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f70989:::benchmark_data_smithi188_113957_object129724:head# 2024-04-23T16:33:37.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f7178e:::benchmark_data_smithi188_113957_object107925:head# 2024-04-23T16:33:37.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f74e2c:::benchmark_data_smithi188_113957_object61878:head# 2024-04-23T16:33:37.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f8c4d6:::benchmark_data_smithi188_113957_object33472:head# 2024-04-23T16:33:37.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f8fb8c:::benchmark_data_smithi188_113957_object66181:head# 2024-04-23T16:33:37.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f94a3b:::benchmark_data_smithi188_113957_object132533:head# 2024-04-23T16:33:37.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f955cf:::benchmark_data_smithi188_113957_object4996:head# 2024-04-23T16:33:37.914 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50f9602b:::benchmark_data_smithi188_113957_object15394:head# 2024-04-23T16:33:37.914 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50fa38aa:::benchmark_data_smithi188_113957_object91019:head# 2024-04-23T16:33:37.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50fb7ebf:::benchmark_data_smithi188_113957_object47478:head# 2024-04-23T16:33:37.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50fbd613:::benchmark_data_smithi188_113957_object23877:head# 2024-04-23T16:33:37.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50fd45c4:::benchmark_data_smithi188_113957_object135348:head# 2024-04-23T16:33:37.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50fe0ded:::benchmark_data_smithi188_113957_object83142:head# 2024-04-23T16:33:37.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50fe46e9:::benchmark_data_smithi188_113957_object137814:head# 2024-04-23T16:33:37.917 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50feb14e:::benchmark_data_smithi188_113957_object31410:head# 2024-04-23T16:33:37.917 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50fed46e:::benchmark_data_smithi188_113957_object73346:head# 2024-04-23T16:33:37.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ff001d:::benchmark_data_smithi188_113957_object126305:head# 2024-04-23T16:33:37.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ff01bb:::benchmark_data_smithi188_113957_object37321:head# 2024-04-23T16:33:37.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:50ff6f32:::benchmark_data_smithi188_113957_object61368:head# 2024-04-23T16:33:37.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510005f0:::benchmark_data_smithi188_113957_object90191:head# 2024-04-23T16:33:37.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51003c57:::benchmark_data_smithi188_113957_object86991:head# 2024-04-23T16:33:37.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510043eb:::benchmark_data_smithi188_113957_object9945:head# 2024-04-23T16:33:37.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5100d392:::benchmark_data_smithi188_113957_object13207:head# 2024-04-23T16:33:37.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510145c6:::benchmark_data_smithi188_113957_object410:head# 2024-04-23T16:33:37.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51016d92:::benchmark_data_smithi188_113957_object62882:head# 2024-04-23T16:33:37.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510184f6:::benchmark_data_smithi188_113957_object140621:head# 2024-04-23T16:33:37.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51023017:::benchmark_data_smithi188_113957_object77586:head# 2024-04-23T16:33:37.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51029c96:::benchmark_data_smithi188_113957_object74997:head# 2024-04-23T16:33:37.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5102e71a:::benchmark_data_smithi188_113957_object141548:head# 2024-04-23T16:33:37.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51031a85:::benchmark_data_smithi188_113957_object98377:head# 2024-04-23T16:33:37.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51032e34:::benchmark_data_smithi188_113957_object69565:head# 2024-04-23T16:33:37.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5103f3b9:::benchmark_data_smithi188_113957_object101840:head# 2024-04-23T16:33:37.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51040329:::benchmark_data_smithi188_113957_object62354:head# 2024-04-23T16:33:37.925 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510461a5:::benchmark_data_smithi188_113957_object27428:head# 2024-04-23T16:33:37.925 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5104893f:::benchmark_data_smithi188_113957_object65275:head# 2024-04-23T16:33:37.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5104c415:::benchmark_data_smithi188_113957_object66246:head# 2024-04-23T16:33:37.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51051811:::benchmark_data_smithi188_113957_object41278:head# 2024-04-23T16:33:37.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5105ca81:::benchmark_data_smithi188_113957_object38993:head# 2024-04-23T16:33:37.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5105dec8:::benchmark_data_smithi188_113957_object21053:head# 2024-04-23T16:33:37.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51064de2:::benchmark_data_smithi188_113957_object39516:head# 2024-04-23T16:33:37.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51068702:::benchmark_data_smithi188_113957_object85334:head# 2024-04-23T16:33:37.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5106dd31:::benchmark_data_smithi188_113957_object115601:head# 2024-04-23T16:33:37.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51075a03:::benchmark_data_smithi188_113957_object65030:head# 2024-04-23T16:33:37.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51078f22:::benchmark_data_smithi188_113957_object15074:head# 2024-04-23T16:33:37.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5107f3b6:::benchmark_data_smithi188_113957_object140299:head# 2024-04-23T16:33:37.930 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51094611:::benchmark_data_smithi188_113957_object11002:head# 2024-04-23T16:33:37.930 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5109b57f:::benchmark_data_smithi188_113957_object40233:head# 2024-04-23T16:33:37.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510a6d98:::benchmark_data_smithi188_113957_object109558:head# 2024-04-23T16:33:37.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510a8779:::benchmark_data_smithi188_113957_object55303:head# 2024-04-23T16:33:37.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510ab898:::benchmark_data_smithi188_113957_object75271:head# 2024-04-23T16:33:37.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510ac940:::benchmark_data_smithi188_113957_object24735:head# 2024-04-23T16:33:37.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510b2cbc:::benchmark_data_smithi188_113957_object56636:head# 2024-04-23T16:33:37.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510b8057:::benchmark_data_smithi188_113957_object75953:head# 2024-04-23T16:33:37.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510c046a:::benchmark_data_smithi188_113957_object153483:head# 2024-04-23T16:33:37.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510cadcb:::benchmark_data_smithi188_113957_object76799:head# 2024-04-23T16:33:37.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510cbf41:::benchmark_data_smithi188_113957_object87792:head# 2024-04-23T16:33:37.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510cd6ba:::benchmark_data_smithi188_113957_object145397:head# 2024-04-23T16:33:37.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510d356e:::benchmark_data_smithi188_113957_object14103:head# 2024-04-23T16:33:37.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510da4d7:::benchmark_data_smithi188_113957_object126098:head# 2024-04-23T16:33:37.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510e2088:::benchmark_data_smithi188_113957_object5676:head# 2024-04-23T16:33:37.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510e2a7c:::benchmark_data_smithi188_113957_object128128:head# 2024-04-23T16:33:37.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510e6c5d:::benchmark_data_smithi188_113957_object16085:head# 2024-04-23T16:33:37.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510f74dc:::benchmark_data_smithi188_113957_object118606:head# 2024-04-23T16:33:37.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:510f95bf:::benchmark_data_smithi188_113957_object68247:head# 2024-04-23T16:33:37.939 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5110262f:::benchmark_data_smithi188_113957_object68180:head# 2024-04-23T16:33:37.939 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5110978f:::benchmark_data_smithi188_113957_object115014:head# 2024-04-23T16:33:37.940 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5110dfaa:::benchmark_data_smithi188_113957_object25093:head# 2024-04-23T16:33:37.940 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51117a21:::benchmark_data_smithi188_113957_object56069:head# 2024-04-23T16:33:37.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5111c428:::benchmark_data_smithi188_113957_object86219:head# 2024-04-23T16:33:37.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5111d9af:::benchmark_data_smithi188_113957_object96744:head# 2024-04-23T16:33:37.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5112207d:::benchmark_data_smithi188_113957_object64459:head# 2024-04-23T16:33:37.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51128a5b:::benchmark_data_smithi188_113957_object104030:head# 2024-04-23T16:33:37.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51133f90:::benchmark_data_smithi188_113957_object53277:head# 2024-04-23T16:33:37.943 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51143ece:::benchmark_data_smithi188_113957_object12672:head# 2024-04-23T16:33:37.943 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51145a80:::benchmark_data_smithi188_113957_object85168:head# 2024-04-23T16:33:37.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:511483cb:::benchmark_data_smithi188_113957_object23808:head# 2024-04-23T16:33:37.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51149a68:::benchmark_data_smithi188_113957_object50790:head# 2024-04-23T16:33:37.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:511531c1:::benchmark_data_smithi188_113957_object94660:head# 2024-04-23T16:33:37.945 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51157c11:::benchmark_data_smithi188_113957_object30882:head# 2024-04-23T16:33:37.945 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5115fa65:::benchmark_data_smithi188_113957_object277:head# 2024-04-23T16:33:37.946 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:511732bc:::benchmark_data_smithi188_113957_object111729:head# 2024-04-23T16:33:37.947 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:511818a5:::benchmark_data_smithi188_113957_object85594:head# 2024-04-23T16:33:37.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51189969:::benchmark_data_smithi188_113957_object54924:head# 2024-04-23T16:33:37.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51191e31:::benchmark_data_smithi188_113957_object13863:head# 2024-04-23T16:33:37.949 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51194119:::benchmark_data_smithi188_113957_object84185:head# 2024-04-23T16:33:37.949 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51194538:::benchmark_data_smithi188_113957_object54702:head# 2024-04-23T16:33:37.950 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:511ac0a2:::benchmark_data_smithi188_113957_object88108:head# 2024-04-23T16:33:37.950 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:511af551:::benchmark_data_smithi188_113957_object115419:head# 2024-04-23T16:33:37.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:511d83b4:::benchmark_data_smithi188_113957_object122005:head# 2024-04-23T16:33:37.953 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:511d8e5f:::benchmark_data_smithi188_113957_object30102:head# 2024-04-23T16:33:37.953 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:511d97d8:::benchmark_data_smithi188_113957_object72546:head# 2024-04-23T16:33:37.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:511daa86:::benchmark_data_smithi188_113957_object143906:head# 2024-04-23T16:33:37.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:511e44ca:::benchmark_data_smithi188_113957_object32224:head# 2024-04-23T16:33:37.955 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51201a09:::benchmark_data_smithi188_113957_object18656:head# 2024-04-23T16:33:37.955 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51206c43:::benchmark_data_smithi188_113957_object18676:head# 2024-04-23T16:33:37.956 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51208ef7:::benchmark_data_smithi188_113957_object109144:head# 2024-04-23T16:33:37.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5120e346:::benchmark_data_smithi188_113957_object140710:head# 2024-04-23T16:33:37.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5120f4ce:::benchmark_data_smithi188_113957_object140035:head# 2024-04-23T16:33:37.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:512142cc:::benchmark_data_smithi188_113957_object21349:head# 2024-04-23T16:33:37.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5121763b:::benchmark_data_smithi188_113957_object142409:head# 2024-04-23T16:33:37.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5121844a:::benchmark_data_smithi188_113957_object90441:head# 2024-04-23T16:33:37.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5122607d:::benchmark_data_smithi188_113957_object39496:head# 2024-04-23T16:33:37.959 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:512292f3:::benchmark_data_smithi188_113957_object131812:head# 2024-04-23T16:33:37.959 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5122c969:::benchmark_data_smithi188_113957_object87498:head# 2024-04-23T16:33:37.960 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5122d324:::benchmark_data_smithi188_113957_object42623:head# 2024-04-23T16:33:37.960 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51233235:::benchmark_data_smithi188_113957_object19898:head# 2024-04-23T16:33:37.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51233550:::benchmark_data_smithi188_113957_object88807:head# 2024-04-23T16:33:37.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5124056d:::benchmark_data_smithi188_113957_object139293:head# 2024-04-23T16:33:37.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51244f4f:::benchmark_data_smithi188_113957_object127266:head# 2024-04-23T16:33:37.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5124ddce:::benchmark_data_smithi188_113957_object75356:head# 2024-04-23T16:33:37.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5124e8dc:::benchmark_data_smithi188_113957_object142678:head# 2024-04-23T16:33:37.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51255be0:::benchmark_data_smithi188_113957_object3165:head# 2024-04-23T16:33:37.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5125615e:::benchmark_data_smithi188_113957_object46072:head# 2024-04-23T16:33:37.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51257954:::benchmark_data_smithi188_113957_object34749:head# 2024-04-23T16:33:37.965 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51259bc1:::benchmark_data_smithi188_113957_object82730:head# 2024-04-23T16:33:37.967 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5127242d:::benchmark_data_smithi188_113957_object51320:head# 2024-04-23T16:33:37.967 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51274da6:::benchmark_data_smithi188_113957_object122340:head# 2024-04-23T16:33:37.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51283cb4:::benchmark_data_smithi188_113957_object39280:head# 2024-04-23T16:33:37.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51294fdb:::benchmark_data_smithi188_113957_object74194:head# 2024-04-23T16:33:37.969 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5129eae4:::benchmark_data_smithi188_113957_object103249:head# 2024-04-23T16:33:37.969 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:512a89b9:::benchmark_data_smithi188_113957_object56848:head# 2024-04-23T16:33:37.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:512af9da:::benchmark_data_smithi188_113957_object124256:head# 2024-04-23T16:33:37.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:512afcc8:::benchmark_data_smithi188_113957_object125322:head# 2024-04-23T16:33:37.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:512b3d39:::benchmark_data_smithi188_113957_object50598:head# 2024-04-23T16:33:37.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:512bce26:::benchmark_data_smithi188_113957_object71321:head# 2024-04-23T16:33:37.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:512c0ef9:::benchmark_data_smithi188_113957_object63043:head# 2024-04-23T16:33:37.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:512db80b:::benchmark_data_smithi188_113957_object101051:head# 2024-04-23T16:33:37.973 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:512dbb52:::benchmark_data_smithi188_113957_object72614:head# 2024-04-23T16:33:37.973 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:512e8195:::benchmark_data_smithi188_113957_object143893:head# 2024-04-23T16:33:37.973 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:512e9ffb:::benchmark_data_smithi188_113957_object110478:head# 2024-04-23T16:33:37.974 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:512f6d32:::benchmark_data_smithi188_113957_object46856:head# 2024-04-23T16:33:37.974 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:512fa26e:::benchmark_data_smithi188_113957_object152637:head# 2024-04-23T16:33:37.974 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51302247:::benchmark_data_smithi188_113957_object74493:head# 2024-04-23T16:33:37.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:513081ee:::benchmark_data_smithi188_113957_object132737:head# 2024-04-23T16:33:37.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5130b0e6:::benchmark_data_smithi188_113957_object105424:head# 2024-04-23T16:33:37.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51314a00:::benchmark_data_smithi188_113957_object80459:head# 2024-04-23T16:33:37.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5131d43d:::benchmark_data_smithi188_113957_object135872:head# 2024-04-23T16:33:37.977 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5132de20:::benchmark_data_smithi188_113957_object90375:head# 2024-04-23T16:33:37.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5132e266:::benchmark_data_smithi188_113957_object79596:head# 2024-04-23T16:33:37.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5132e73c:::benchmark_data_smithi188_113957_object13964:head# 2024-04-23T16:33:37.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5133fe96:::benchmark_data_smithi188_113957_object50001:head# 2024-04-23T16:33:37.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5134d42b:::benchmark_data_smithi188_113957_object132156:head# 2024-04-23T16:33:37.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5134e66e:::benchmark_data_smithi188_113957_object51642:head# 2024-04-23T16:33:37.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5134efbf:::benchmark_data_smithi188_113957_object51460:head# 2024-04-23T16:33:37.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51352326:::benchmark_data_smithi188_113957_object68249:head# 2024-04-23T16:33:37.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51352406:::benchmark_data_smithi188_113957_object53781:head# 2024-04-23T16:33:37.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51352615:::benchmark_data_smithi188_113957_object27478:head# 2024-04-23T16:33:37.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5135a3fc:::benchmark_data_smithi188_113957_object74961:head# 2024-04-23T16:33:37.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5135c2f8:::benchmark_data_smithi188_113957_object149723:head# 2024-04-23T16:33:37.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:513714e0:::benchmark_data_smithi188_113957_object109890:head# 2024-04-23T16:33:37.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51373ab8:::benchmark_data_smithi188_113957_object145772:head# 2024-04-23T16:33:37.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51380db1:::benchmark_data_smithi188_113957_object49834:head# 2024-04-23T16:33:37.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51381df7:::benchmark_data_smithi188_113957_object100972:head# 2024-04-23T16:33:37.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5138a589:::benchmark_data_smithi188_113957_object13463:head# 2024-04-23T16:33:37.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5138c74c:::benchmark_data_smithi188_113957_object132071:head# 2024-04-23T16:33:37.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51398bfa:::benchmark_data_smithi188_113957_object1512:head# 2024-04-23T16:33:37.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:513a6fa3:::benchmark_data_smithi188_113957_object29810:head# 2024-04-23T16:33:37.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:513bbeb9:::benchmark_data_smithi188_113957_object152715:head# 2024-04-23T16:33:37.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:513ceb50:::benchmark_data_smithi188_113957_object97374:head# 2024-04-23T16:33:37.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:513cf96b:::benchmark_data_smithi188_113957_object117743:head# 2024-04-23T16:33:37.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:513d991b:::benchmark_data_smithi188_113957_object126713:head# 2024-04-23T16:33:37.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:513da6b2:::benchmark_data_smithi188_113957_object35433:head# 2024-04-23T16:33:37.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:513f424b:::benchmark_data_smithi188_113957_object32023:head# 2024-04-23T16:33:37.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51400100:::benchmark_data_smithi188_113957_object7123:head# 2024-04-23T16:33:37.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5140560d:::benchmark_data_smithi188_113957_object136154:head# 2024-04-23T16:33:37.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5141b158:::benchmark_data_smithi188_113957_object123268:head# 2024-04-23T16:33:37.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5141d8f0:::benchmark_data_smithi188_113957_object118898:head# 2024-04-23T16:33:37.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5141ee2a:::benchmark_data_smithi188_113957_object44423:head# 2024-04-23T16:33:37.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5141f31f:::benchmark_data_smithi188_113957_object114286:head# 2024-04-23T16:33:37.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5142530b:::benchmark_data_smithi188_113957_object63735:head# 2024-04-23T16:33:37.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5144f7cd:::benchmark_data_smithi188_113957_object82352:head# 2024-04-23T16:33:37.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5145af04:::benchmark_data_smithi188_113957_object92384:head# 2024-04-23T16:33:37.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51460caf:::benchmark_data_smithi188_113957_object53334:head# 2024-04-23T16:33:37.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51477fdd:::benchmark_data_smithi188_113957_object129895:head# 2024-04-23T16:33:37.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5147ea08:::benchmark_data_smithi188_113957_object34364:head# 2024-04-23T16:33:37.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51481246:::benchmark_data_smithi188_113957_object21810:head# 2024-04-23T16:33:37.997 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51489347:::benchmark_data_smithi188_113957_object105162:head# 2024-04-23T16:33:37.997 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5148cd8f:::benchmark_data_smithi188_113957_object153456:head# 2024-04-23T16:33:38.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5149635f:::benchmark_data_smithi188_113957_object77369:head# 2024-04-23T16:33:38.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51497d38:::benchmark_data_smithi188_113957_object14066:head# 2024-04-23T16:33:38.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:514af384:::benchmark_data_smithi188_113957_object135132:head# 2024-04-23T16:33:38.001 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:514bc252:::benchmark_data_smithi188_113957_object68869:head# 2024-04-23T16:33:38.001 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:514bc7bc:::benchmark_data_smithi188_113957_object70004:head# 2024-04-23T16:33:38.002 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:514bd046:::benchmark_data_smithi188_113957_object54651:head# 2024-04-23T16:33:38.002 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:514cab87:::benchmark_data_smithi188_113957_object56294:head# 2024-04-23T16:33:38.003 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:514f69a8:::benchmark_data_smithi188_113957_object136753:head# 2024-04-23T16:33:38.003 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51504834:::benchmark_data_smithi188_113957_object22954:head# 2024-04-23T16:33:38.003 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5150e706:::benchmark_data_smithi188_113957_object70937:head# 2024-04-23T16:33:38.004 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51512ee7:::benchmark_data_smithi188_113957_object79434:head# 2024-04-23T16:33:38.004 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5151dacf:::benchmark_data_smithi188_113957_object66579:head# 2024-04-23T16:33:38.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5152377d:::benchmark_data_smithi188_113957_object31780:head# 2024-04-23T16:33:38.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51528525:::benchmark_data_smithi188_113957_object135571:head# 2024-04-23T16:33:38.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51539e11:::benchmark_data_smithi188_113957_object139372:head# 2024-04-23T16:33:38.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:515479fa:::benchmark_data_smithi188_113957_object113027:head# 2024-04-23T16:33:38.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5154a711:::benchmark_data_smithi188_113957_object81377:head# 2024-04-23T16:33:38.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51554b09:::benchmark_data_smithi188_113957_object56810:head# 2024-04-23T16:33:38.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51559c78:::benchmark_data_smithi188_113957_object118824:head# 2024-04-23T16:33:38.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51562c44:::benchmark_data_smithi188_113957_object55884:head# 2024-04-23T16:33:38.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51574da2:::benchmark_data_smithi188_113957_object113603:head# 2024-04-23T16:33:38.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51586e9a:::benchmark_data_smithi188_113957_object4443:head# 2024-04-23T16:33:38.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51587027:::benchmark_data_smithi188_113957_object44207:head# 2024-04-23T16:33:38.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51596a23:::benchmark_data_smithi188_113957_object43263:head# 2024-04-23T16:33:38.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5159ccb3:::benchmark_data_smithi188_113957_object113576:head# 2024-04-23T16:33:38.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:515ab018:::benchmark_data_smithi188_113957_object77132:head# 2024-04-23T16:33:38.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:515b38a2:::benchmark_data_smithi188_113957_object74645:head# 2024-04-23T16:33:38.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:515ba7e7:::benchmark_data_smithi188_113957_object122531:head# 2024-04-23T16:33:38.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:515cbf93:::benchmark_data_smithi188_113957_object66428:head# 2024-04-23T16:33:38.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:515d4d12:::benchmark_data_smithi188_113957_object34154:head# 2024-04-23T16:33:38.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:515d6a2c:::benchmark_data_smithi188_113957_object74045:head# 2024-04-23T16:33:38.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:515d7a36:::benchmark_data_smithi188_113957_object46997:head# 2024-04-23T16:33:38.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:515ef5fc:::benchmark_data_smithi188_113957_object143415:head# 2024-04-23T16:33:38.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:515f88b4:::benchmark_data_smithi188_113957_object98438:head# 2024-04-23T16:33:38.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:515fabd5:::benchmark_data_smithi188_113957_object113895:head# 2024-04-23T16:33:38.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:515fb279:::benchmark_data_smithi188_113957_object6989:head# 2024-04-23T16:33:38.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51607da9:::benchmark_data_smithi188_113957_object134251:head# 2024-04-23T16:33:38.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5160ac5c:::benchmark_data_smithi188_113957_object1788:head# 2024-04-23T16:33:38.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51611edb:::benchmark_data_smithi188_113957_object36696:head# 2024-04-23T16:33:38.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516172be:::benchmark_data_smithi188_113957_object61157:head# 2024-04-23T16:33:38.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51623526:::benchmark_data_smithi188_113957_object135856:head# 2024-04-23T16:33:38.018 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51625d59:::benchmark_data_smithi188_113957_object30810:head# 2024-04-23T16:33:38.018 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51626298:::benchmark_data_smithi188_113957_object80762:head# 2024-04-23T16:33:38.019 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51637142:::benchmark_data_smithi188_113957_object101579:head# 2024-04-23T16:33:38.019 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51637dac:::benchmark_data_smithi188_113957_object112656:head# 2024-04-23T16:33:38.019 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5163863b:::benchmark_data_smithi188_113957_object115977:head# 2024-04-23T16:33:38.020 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5163a7b5:::benchmark_data_smithi188_113957_object30227:head# 2024-04-23T16:33:38.020 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5163b1f0:::benchmark_data_smithi188_113957_object89834:head# 2024-04-23T16:33:38.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5163eaef:::benchmark_data_smithi188_113957_object75303:head# 2024-04-23T16:33:38.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51646b87:::benchmark_data_smithi188_113957_object55608:head# 2024-04-23T16:33:38.022 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51647a2a:::benchmark_data_smithi188_113957_object81974:head# 2024-04-23T16:33:38.022 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51649c38:::benchmark_data_smithi188_113957_object96528:head# 2024-04-23T16:33:38.023 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5164f57a:::benchmark_data_smithi188_113957_object129280:head# 2024-04-23T16:33:38.023 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51652114:::benchmark_data_smithi188_113957_object150318:head# 2024-04-23T16:33:38.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51655bd6:::benchmark_data_smithi188_113957_object38606:head# 2024-04-23T16:33:38.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516569fc:::benchmark_data_smithi188_113957_object128717:head# 2024-04-23T16:33:38.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516629ca:::benchmark_data_smithi188_113957_object31725:head# 2024-04-23T16:33:38.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516793c6:::benchmark_data_smithi188_113957_object63301:head# 2024-04-23T16:33:38.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5167e475:::benchmark_data_smithi188_113957_object141901:head# 2024-04-23T16:33:38.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5167f7e5:::benchmark_data_smithi188_113957_object25150:head# 2024-04-23T16:33:38.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5168b96b:::benchmark_data_smithi188_113957_object16669:head# 2024-04-23T16:33:38.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51691863:::benchmark_data_smithi188_113957_object103938:head# 2024-04-23T16:33:38.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516938a2:::benchmark_data_smithi188_113957_object96164:head# 2024-04-23T16:33:38.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51699208:::benchmark_data_smithi188_113957_object13802:head# 2024-04-23T16:33:38.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5169c2e6:::benchmark_data_smithi188_113957_object87450:head# 2024-04-23T16:33:38.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516a2fd0:::benchmark_data_smithi188_113957_object79656:head# 2024-04-23T16:33:38.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516a3d6c:::benchmark_data_smithi188_113957_object45642:head# 2024-04-23T16:33:38.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516a51e0:::benchmark_data_smithi188_113957_object62784:head# 2024-04-23T16:33:38.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516a6d7f:::benchmark_data_smithi188_113957_object31724:head# 2024-04-23T16:33:38.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516acbff:::benchmark_data_smithi188_113957_object65256:head# 2024-04-23T16:33:38.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516b194f:::benchmark_data_smithi188_113957_object5910:head# 2024-04-23T16:33:38.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516b492a:::benchmark_data_smithi188_113957_object133528:head# 2024-04-23T16:33:38.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516b4bd3:::benchmark_data_smithi188_113957_object110741:head# 2024-04-23T16:33:38.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516b77fc:::benchmark_data_smithi188_113957_object55570:head# 2024-04-23T16:33:38.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516ba6a6:::benchmark_data_smithi188_113957_object96008:head# 2024-04-23T16:33:38.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516c46b7:::benchmark_data_smithi188_113957_object57986:head# 2024-04-23T16:33:38.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516c69e1:::benchmark_data_smithi188_113957_object5276:head# 2024-04-23T16:33:38.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516dfc05:::benchmark_data_smithi188_113957_object34623:head# 2024-04-23T16:33:38.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516ea325:::benchmark_data_smithi188_113957_object125233:head# 2024-04-23T16:33:38.034 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516eccee:::benchmark_data_smithi188_113957_object21648:head# 2024-04-23T16:33:38.034 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516ee77e:::benchmark_data_smithi188_113957_object113808:head# 2024-04-23T16:33:38.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516f350e:::benchmark_data_smithi188_113957_object104582:head# 2024-04-23T16:33:38.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516f4eb3:::benchmark_data_smithi188_113957_object28959:head# 2024-04-23T16:33:38.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516f9c33:::benchmark_data_smithi188_113957_object98967:head# 2024-04-23T16:33:38.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:516fc2a9:::benchmark_data_smithi188_113957_object87935:head# 2024-04-23T16:33:38.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51706449:::benchmark_data_smithi188_113957_object135215:head# 2024-04-23T16:33:38.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:517133d6:::benchmark_data_smithi188_113957_object14862:head# 2024-04-23T16:33:38.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51719875:::benchmark_data_smithi188_113957_object88643:head# 2024-04-23T16:33:38.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51719c1c:::benchmark_data_smithi188_113957_object44564:head# 2024-04-23T16:33:38.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5171bede:::benchmark_data_smithi188_113957_object32374:head# 2024-04-23T16:33:38.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5171e6bd:::benchmark_data_smithi188_113957_object143506:head# 2024-04-23T16:33:38.040 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5172a0ed:::benchmark_data_smithi188_113957_object22684:head# 2024-04-23T16:33:38.040 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51733793:::benchmark_data_smithi188_113957_object131402:head# 2024-04-23T16:33:38.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51737773:::benchmark_data_smithi188_113957_object142741:head# 2024-04-23T16:33:38.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5173d412:::benchmark_data_smithi188_113957_object27081:head# 2024-04-23T16:33:38.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5174ddf6:::benchmark_data_smithi188_113957_object44515:head# 2024-04-23T16:33:38.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5174fb42:::benchmark_data_smithi188_113957_object139396:head# 2024-04-23T16:33:38.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51751a25:::benchmark_data_smithi188_113957_object10781:head# 2024-04-23T16:33:38.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:517537b4:::benchmark_data_smithi188_113957_object143012:head# 2024-04-23T16:33:38.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5175f139:::benchmark_data_smithi188_113957_object45296:head# 2024-04-23T16:33:38.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51764eb7:::benchmark_data_smithi188_113957_object92364:head# 2024-04-23T16:33:38.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:517669ca:::benchmark_data_smithi188_113957_object45792:head# 2024-04-23T16:33:38.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5176bbf0:::benchmark_data_smithi188_113957_object53655:head# 2024-04-23T16:33:38.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5176c24f:::benchmark_data_smithi188_113957_object63761:head# 2024-04-23T16:33:38.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51776500:::benchmark_data_smithi188_113957_object78190:head# 2024-04-23T16:33:38.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5177e648:::benchmark_data_smithi188_113957_object1647:head# 2024-04-23T16:33:38.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51780394:::benchmark_data_smithi188_113957_object74670:head# 2024-04-23T16:33:38.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5178751a:::benchmark_data_smithi188_113957_object52547:head# 2024-04-23T16:33:38.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5178840c:::benchmark_data_smithi188_113957_object135634:head# 2024-04-23T16:33:38.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5178afbd:::benchmark_data_smithi188_113957_object92226:head# 2024-04-23T16:33:38.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5179712e:::benchmark_data_smithi188_113957_object7166:head# 2024-04-23T16:33:38.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5179a92e:::benchmark_data_smithi188_113957_object86457:head# 2024-04-23T16:33:38.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:517a65b3:::benchmark_data_smithi188_113957_object120344:head# 2024-04-23T16:33:38.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:517a73a5:::benchmark_data_smithi188_113957_object148399:head# 2024-04-23T16:33:38.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:517a9d4d:::benchmark_data_smithi188_113957_object94685:head# 2024-04-23T16:33:38.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:517b2141:::benchmark_data_smithi188_113957_object4879:head# 2024-04-23T16:33:38.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:517d247f:::benchmark_data_smithi188_113957_object127538:head# 2024-04-23T16:33:38.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:517d3e07:::benchmark_data_smithi188_113957_object137789:head# 2024-04-23T16:33:38.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:517d95e5:::benchmark_data_smithi188_113957_object121528:head# 2024-04-23T16:33:38.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:517e5bc4:::benchmark_data_smithi188_113957_object41017:head# 2024-04-23T16:33:38.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:517fb4a4:::benchmark_data_smithi188_113957_object4230:head# 2024-04-23T16:33:38.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:518029e7:::benchmark_data_smithi188_113957_object102761:head# 2024-04-23T16:33:38.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5180b936:::benchmark_data_smithi188_113957_object109463:head# 2024-04-23T16:33:38.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5181b58f:::benchmark_data_smithi188_113957_object128270:head# 2024-04-23T16:33:38.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5181fd47:::benchmark_data_smithi188_113957_object46375:head# 2024-04-23T16:33:38.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51820eb8:::benchmark_data_smithi188_113957_object42287:head# 2024-04-23T16:33:38.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51827f55:::benchmark_data_smithi188_113957_object72303:head# 2024-04-23T16:33:38.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51830348:::benchmark_data_smithi188_113957_object49014:head# 2024-04-23T16:33:38.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5183a603:::benchmark_data_smithi188_113957_object103232:head# 2024-04-23T16:33:38.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5184a0c5:::benchmark_data_smithi188_113957_object60120:head# 2024-04-23T16:33:38.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5184aadd:::benchmark_data_smithi188_113957_object144496:head# 2024-04-23T16:33:38.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5184f5d0:::benchmark_data_smithi188_113957_object77430:head# 2024-04-23T16:33:38.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5184f6f8:::benchmark_data_smithi188_113957_object137076:head# 2024-04-23T16:33:38.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5187062f:::benchmark_data_smithi188_113957_object151360:head# 2024-04-23T16:33:38.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5187177b:::benchmark_data_smithi188_113957_object23135:head# 2024-04-23T16:33:38.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5187684d:::benchmark_data_smithi188_113957_object102776:head# 2024-04-23T16:33:38.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:518769ca:::benchmark_data_smithi188_113957_object43483:head# 2024-04-23T16:33:38.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51879449:::benchmark_data_smithi188_113957_object96429:head# 2024-04-23T16:33:38.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5187c191:::benchmark_data_smithi188_113957_object142873:head# 2024-04-23T16:33:38.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51884886:::benchmark_data_smithi188_113957_object46851:head# 2024-04-23T16:33:38.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5188d6cf:::benchmark_data_smithi188_113957_object43240:head# 2024-04-23T16:33:38.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:518968c6:::benchmark_data_smithi188_113957_object41205:head# 2024-04-23T16:33:38.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51898fb7:::benchmark_data_smithi188_113957_object152595:head# 2024-04-23T16:33:38.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5189901b:::benchmark_data_smithi188_113957_object136777:head# 2024-04-23T16:33:38.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5189b96c:::benchmark_data_smithi188_113957_object71306:head# 2024-04-23T16:33:38.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5189eee4:::benchmark_data_smithi188_113957_object123442:head# 2024-04-23T16:33:38.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:518a5d11:::benchmark_data_smithi188_113957_object108504:head# 2024-04-23T16:33:38.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:518aa553:::benchmark_data_smithi188_113957_object20925:head# 2024-04-23T16:33:38.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:518afd0c:::benchmark_data_smithi188_113957_object137812:head# 2024-04-23T16:33:38.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:518b1efb:::benchmark_data_smithi188_113957_object4072:head# 2024-04-23T16:33:38.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:518c16f6:::benchmark_data_smithi188_113957_object18331:head# 2024-04-23T16:33:38.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:518c8c49:::benchmark_data_smithi188_113957_object93:head# 2024-04-23T16:33:38.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:518d5448:::benchmark_data_smithi188_113957_object44449:head# 2024-04-23T16:33:38.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:518ecd68:::benchmark_data_smithi188_113957_object51160:head# 2024-04-23T16:33:38.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:518f220c:::benchmark_data_smithi188_113957_object133243:head# 2024-04-23T16:33:38.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:518f571b:::benchmark_data_smithi188_113957_object55808:head# 2024-04-23T16:33:38.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:518f6c1b:::benchmark_data_smithi188_113957_object88785:head# 2024-04-23T16:33:38.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519024de:::benchmark_data_smithi188_113957_object129451:head# 2024-04-23T16:33:38.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51903687:::benchmark_data_smithi188_113957_object102921:head# 2024-04-23T16:33:38.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5190bfe2:::benchmark_data_smithi188_113957_object75294:head# 2024-04-23T16:33:38.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5190e700:::benchmark_data_smithi188_113957_object112700:head# 2024-04-23T16:33:38.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51916ad7:::benchmark_data_smithi188_113957_object93300:head# 2024-04-23T16:33:38.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51921cdf:::benchmark_data_smithi188_113957_object2844:head# 2024-04-23T16:33:38.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519244e3:::benchmark_data_smithi188_113957_object109738:head# 2024-04-23T16:33:38.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51928b34:::benchmark_data_smithi188_113957_object140131:head# 2024-04-23T16:33:38.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51931d17:::benchmark_data_smithi188_113957_object38473:head# 2024-04-23T16:33:38.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5193daa3:::benchmark_data_smithi188_113957_object1657:head# 2024-04-23T16:33:38.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5193fbf9:::benchmark_data_smithi188_113957_object86285:head# 2024-04-23T16:33:38.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51945567:::benchmark_data_smithi188_113957_object91316:head# 2024-04-23T16:33:38.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5195beb2:::benchmark_data_smithi188_113957_object8281:head# 2024-04-23T16:33:38.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5195ed93:::benchmark_data_smithi188_113957_object111735:head# 2024-04-23T16:33:38.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51969207:::benchmark_data_smithi188_113957_object6341:head# 2024-04-23T16:33:38.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51969c87:::benchmark_data_smithi188_113957_object58374:head# 2024-04-23T16:33:38.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5196b6c8:::benchmark_data_smithi188_113957_object78100:head# 2024-04-23T16:33:38.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5196b710:::benchmark_data_smithi188_113957_object102944:head# 2024-04-23T16:33:38.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51974002:::benchmark_data_smithi188_113957_object67904:head# 2024-04-23T16:33:38.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519744a3:::benchmark_data_smithi188_113957_object48142:head# 2024-04-23T16:33:38.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51983351:::benchmark_data_smithi188_113957_object67343:head# 2024-04-23T16:33:38.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5198b3c3:::benchmark_data_smithi188_113957_object84487:head# 2024-04-23T16:33:38.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51991e94:::benchmark_data_smithi188_113957_object112812:head# 2024-04-23T16:33:38.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51994ef1:::benchmark_data_smithi188_113957_object128097:head# 2024-04-23T16:33:38.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5199c365:::benchmark_data_smithi188_113957_object32079:head# 2024-04-23T16:33:38.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519a4c66:::benchmark_data_smithi188_113957_object76753:head# 2024-04-23T16:33:38.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519a63e3:::benchmark_data_smithi188_113957_object126433:head# 2024-04-23T16:33:38.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519a71bb:::benchmark_data_smithi188_113957_object148848:head# 2024-04-23T16:33:38.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519ab99b:::benchmark_data_smithi188_113957_object148915:head# 2024-04-23T16:33:38.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519b30a1:::benchmark_data_smithi188_113957_object54736:head# 2024-04-23T16:33:38.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519b3655:::benchmark_data_smithi188_113957_object26825:head# 2024-04-23T16:33:38.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519baa39:::benchmark_data_smithi188_113957_object65843:head# 2024-04-23T16:33:38.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519bcbda:::benchmark_data_smithi188_113957_object30015:head# 2024-04-23T16:33:38.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519d274e:::benchmark_data_smithi188_113957_object32234:head# 2024-04-23T16:33:38.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519df366:::benchmark_data_smithi188_113957_object42598:head# 2024-04-23T16:33:38.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519e357b:::benchmark_data_smithi188_113957_object131491:head# 2024-04-23T16:33:38.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519f6388:::benchmark_data_smithi188_113957_object68978:head# 2024-04-23T16:33:38.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519f8474:::benchmark_data_smithi188_113957_object19150:head# 2024-04-23T16:33:38.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519f9a08:::benchmark_data_smithi188_113957_object146495:head# 2024-04-23T16:33:38.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519fa244:::benchmark_data_smithi188_113957_object125928:head# 2024-04-23T16:33:38.087 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519fc613:::benchmark_data_smithi188_113957_object131415:head# 2024-04-23T16:33:38.087 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:519fed97:::benchmark_data_smithi188_113957_object544:head# 2024-04-23T16:33:38.088 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a05bfd:::benchmark_data_smithi188_113957_object132889:head# 2024-04-23T16:33:38.088 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a0abc9:::benchmark_data_smithi188_113957_object13987:head# 2024-04-23T16:33:38.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a16a81:::benchmark_data_smithi188_113957_object85277:head# 2024-04-23T16:33:38.089 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:37 smithi148 ceph-mon[44520]: pgmap v1117: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 9.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 82 KiB/s wr, 406 op/s; 28217/299060 objects degraded (9.435%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:33:38.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a1d138:::benchmark_data_smithi188_113957_object51983:head# 2024-04-23T16:33:38.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a20795:::benchmark_data_smithi188_113957_object8336:head# 2024-04-23T16:33:38.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a272aa:::benchmark_data_smithi188_113957_object118933:head# 2024-04-23T16:33:38.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a27329:::benchmark_data_smithi188_113957_object40053:head# 2024-04-23T16:33:38.091 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a35106:::benchmark_data_smithi188_113957_object50330:head# 2024-04-23T16:33:38.091 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a41466:::benchmark_data_smithi188_113957_object7158:head# 2024-04-23T16:33:38.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a4309b:::benchmark_data_smithi188_113957_object111732:head# 2024-04-23T16:33:38.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a4d544:::benchmark_data_smithi188_113957_object64509:head# 2024-04-23T16:33:38.093 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a58dcf:::benchmark_data_smithi188_113957_object2324:head# 2024-04-23T16:33:38.093 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a5c17f:::benchmark_data_smithi188_113957_object92048:head# 2024-04-23T16:33:38.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a60bc7:::benchmark_data_smithi188_113957_object89080:head# 2024-04-23T16:33:38.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a6c41b:::benchmark_data_smithi188_113957_object45132:head# 2024-04-23T16:33:38.095 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a7022d:::benchmark_data_smithi188_113957_object2455:head# 2024-04-23T16:33:38.095 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a73ad2:::benchmark_data_smithi188_113957_object49850:head# 2024-04-23T16:33:38.096 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a749fd:::benchmark_data_smithi188_113957_object21312:head# 2024-04-23T16:33:38.096 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a75188:::benchmark_data_smithi188_113957_object11189:head# 2024-04-23T16:33:38.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a77f00:::benchmark_data_smithi188_113957_object137879:head# 2024-04-23T16:33:38.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a7ab9a:::benchmark_data_smithi188_113957_object4874:head# 2024-04-23T16:33:38.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a7ed62:::benchmark_data_smithi188_113957_object53083:head# 2024-04-23T16:33:38.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a824da:::benchmark_data_smithi188_113957_object12770:head# 2024-04-23T16:33:38.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a8cee0:::benchmark_data_smithi188_113957_object135198:head# 2024-04-23T16:33:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:37 smithi103 ceph-mon[39629]: pgmap v1117: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 9.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 82 KiB/s wr, 406 op/s; 28217/299060 objects degraded (9.435%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:33:38.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a8e25c:::benchmark_data_smithi188_113957_object34077:head# 2024-04-23T16:33:38.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a9cffa:::benchmark_data_smithi188_113957_object78142:head# 2024-04-23T16:33:38.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51a9d6eb:::benchmark_data_smithi188_113957_object128413:head# 2024-04-23T16:33:38.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51aa352f:::benchmark_data_smithi188_113957_object81317:head# 2024-04-23T16:33:38.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51aa598d:::benchmark_data_smithi188_113957_object20134:head# 2024-04-23T16:33:38.102 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51aaf659:::benchmark_data_smithi188_113957_object146224:head# 2024-04-23T16:33:38.102 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ab3333:::benchmark_data_smithi188_113957_object149420:head# 2024-04-23T16:33:38.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ab7979:::benchmark_data_smithi188_113957_object106519:head# 2024-04-23T16:33:38.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ab7fe9:::benchmark_data_smithi188_113957_object111739:head# 2024-04-23T16:33:38.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ab9a41:::benchmark_data_smithi188_113957_object141236:head# 2024-04-23T16:33:38.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ac75c1:::benchmark_data_smithi188_113957_object98150:head# 2024-04-23T16:33:38.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ac7af5:::benchmark_data_smithi188_113957_object125914:head# 2024-04-23T16:33:38.105 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ac9648:::benchmark_data_smithi188_113957_object133691:head# 2024-04-23T16:33:38.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51acb528:::benchmark_data_smithi188_113957_object1922:head# 2024-04-23T16:33:38.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ad026e:::benchmark_data_smithi188_113957_object99976:head# 2024-04-23T16:33:38.107 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51af2dcc:::benchmark_data_smithi188_113957_object50702:head# 2024-04-23T16:33:38.107 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51afc693:::benchmark_data_smithi188_113957_object54733:head# 2024-04-23T16:33:38.107 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51afe272:::benchmark_data_smithi188_113957_object66237:head# 2024-04-23T16:33:38.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b01eaa:::benchmark_data_smithi188_113957_object26081:head# 2024-04-23T16:33:38.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b090b7:::benchmark_data_smithi188_113957_object99858:head# 2024-04-23T16:33:38.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b1f036:::benchmark_data_smithi188_113957_object134572:head# 2024-04-23T16:33:38.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b23ba7:::benchmark_data_smithi188_113957_object43490:head# 2024-04-23T16:33:38.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b25892:::benchmark_data_smithi188_113957_object81066:head# 2024-04-23T16:33:38.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b26660:::benchmark_data_smithi188_113957_object40557:head# 2024-04-23T16:33:38.111 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b29dec:::benchmark_data_smithi188_113957_object32028:head# 2024-04-23T16:33:38.111 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b30c64:::benchmark_data_smithi188_113957_object75739:head# 2024-04-23T16:33:38.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b337bc:::benchmark_data_smithi188_113957_object37647:head# 2024-04-23T16:33:38.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b46143:::benchmark_data_smithi188_113957_object27892:head# 2024-04-23T16:33:38.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b48b2f:::benchmark_data_smithi188_113957_object50684:head# 2024-04-23T16:33:38.113 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b49fd1:::benchmark_data_smithi188_113957_object40126:head# 2024-04-23T16:33:38.113 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b522ee:::benchmark_data_smithi188_113957_object20577:head# 2024-04-23T16:33:38.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b527bd:::benchmark_data_smithi188_113957_object120873:head# 2024-04-23T16:33:38.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b5817e:::benchmark_data_smithi188_113957_object42960:head# 2024-04-23T16:33:38.114 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:33:38.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b5f103:::benchmark_data_smithi188_113957_object34981:head# 2024-04-23T16:33:38.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b7c1c1:::benchmark_data_smithi188_113957_object118663:head# 2024-04-23T16:33:38.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b857b5:::benchmark_data_smithi188_113957_object86942:head# 2024-04-23T16:33:38.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b87fc7:::benchmark_data_smithi188_113957_object103498:head# 2024-04-23T16:33:38.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b8ef89:::benchmark_data_smithi188_113957_object97494:head# 2024-04-23T16:33:38.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b92078:::benchmark_data_smithi188_113957_object153243:head# 2024-04-23T16:33:38.118 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b9329c:::benchmark_data_smithi188_113957_object14820:head# 2024-04-23T16:33:38.118 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51b9cca1:::benchmark_data_smithi188_113957_object17687:head# 2024-04-23T16:33:38.118 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ba0bc5:::benchmark_data_smithi188_113957_object130879:head# 2024-04-23T16:33:38.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ba84f8:::benchmark_data_smithi188_113957_object90599:head# 2024-04-23T16:33:38.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51baa930:::benchmark_data_smithi188_113957_object86194:head# 2024-04-23T16:33:38.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51bab594:::benchmark_data_smithi188_113957_object108850:head# 2024-04-23T16:33:38.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51bad794:::benchmark_data_smithi188_113957_object106340:head# 2024-04-23T16:33:38.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51bba13d:::benchmark_data_smithi188_113957_object112607:head# 2024-04-23T16:33:38.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51bc6a75:::benchmark_data_smithi188_113957_object60496:head# 2024-04-23T16:33:38.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51bcd2cd:::benchmark_data_smithi188_113957_object93973:head# 2024-04-23T16:33:38.122 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51bd729a:::benchmark_data_smithi188_113957_object135674:head# 2024-04-23T16:33:38.122 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51bd7e45:::benchmark_data_smithi188_113957_object1669:head# 2024-04-23T16:33:38.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51bea053:::benchmark_data_smithi188_113957_object20325:head# 2024-04-23T16:33:38.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51beb405:::benchmark_data_smithi188_113957_object111455:head# 2024-04-23T16:33:38.124 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51bed45b:::benchmark_data_smithi188_113957_object111895:head# 2024-04-23T16:33:38.124 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51bef6bc:::benchmark_data_smithi188_113957_object97952:head# 2024-04-23T16:33:38.124 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51bfe7b3:::benchmark_data_smithi188_113957_object64745:head# 2024-04-23T16:33:38.125 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c0259a:::benchmark_data_smithi188_113957_object9088:head# 2024-04-23T16:33:38.125 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c0a5a7:::benchmark_data_smithi188_113957_object124368:head# 2024-04-23T16:33:38.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c0c329:::benchmark_data_smithi188_113957_object152187:head# 2024-04-23T16:33:38.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c0d1ff:::benchmark_data_smithi188_113957_object111328:head# 2024-04-23T16:33:38.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c1776c:::benchmark_data_smithi188_113957_object89967:head# 2024-04-23T16:33:38.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c1dfab:::benchmark_data_smithi188_113957_object45466:head# 2024-04-23T16:33:38.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c1ed61:::benchmark_data_smithi188_113957_object60691:head# 2024-04-23T16:33:38.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c3e546:::benchmark_data_smithi188_113957_object116410:head# 2024-04-23T16:33:38.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c4db81:::benchmark_data_smithi188_113957_object139259:head# 2024-04-23T16:33:38.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c541c9:::benchmark_data_smithi188_113957_object35254:head# 2024-04-23T16:33:38.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c544cf:::benchmark_data_smithi188_113957_object71559:head# 2024-04-23T16:33:38.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c559ac:::benchmark_data_smithi188_113957_object12872:head# 2024-04-23T16:33:38.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c5610b:::benchmark_data_smithi188_113957_object83062:head# 2024-04-23T16:33:38.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c5c63d:::benchmark_data_smithi188_113957_object75902:head# 2024-04-23T16:33:38.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c618da:::benchmark_data_smithi188_113957_object85740:head# 2024-04-23T16:33:38.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c64e50:::benchmark_data_smithi188_113957_object14080:head# 2024-04-23T16:33:38.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c66df8:::benchmark_data_smithi188_113957_object95468:head# 2024-04-23T16:33:38.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c6b200:::benchmark_data_smithi188_113957_object2374:head# 2024-04-23T16:33:38.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c7147b:::benchmark_data_smithi188_113957_object95774:head# 2024-04-23T16:33:38.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c7a40a:::benchmark_data_smithi188_113957_object35058:head# 2024-04-23T16:33:38.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c8071a:::benchmark_data_smithi188_113957_object88999:head# 2024-04-23T16:33:38.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c853d8:::benchmark_data_smithi188_113957_object82189:head# 2024-04-23T16:33:38.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c85594:::benchmark_data_smithi188_113957_object68776:head# 2024-04-23T16:33:38.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c8620c:::benchmark_data_smithi188_113957_object51607:head# 2024-04-23T16:33:38.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c87fac:::benchmark_data_smithi188_113957_object21376:head# 2024-04-23T16:33:38.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c89896:::benchmark_data_smithi188_113957_object7327:head# 2024-04-23T16:33:38.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c9a043:::benchmark_data_smithi188_113957_object130631:head# 2024-04-23T16:33:38.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51c9b236:::benchmark_data_smithi188_113957_object131215:head# 2024-04-23T16:33:38.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ca818a:::benchmark_data_smithi188_113957_object21082:head# 2024-04-23T16:33:38.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51cab89d:::benchmark_data_smithi188_113957_object76593:head# 2024-04-23T16:33:38.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51cb77c5:::benchmark_data_smithi188_113957_object35397:head# 2024-04-23T16:33:38.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51cccaed:::benchmark_data_smithi188_113957_object51239:head# 2024-04-23T16:33:38.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51cd168a:::benchmark_data_smithi188_113957_object130679:head# 2024-04-23T16:33:38.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51cd94a4:::benchmark_data_smithi188_113957_object41355:head# 2024-04-23T16:33:38.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51cde0aa:::benchmark_data_smithi188_113957_object59442:head# 2024-04-23T16:33:38.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ce5aff:::benchmark_data_smithi188_113957_object100504:head# 2024-04-23T16:33:38.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ce8a74:::benchmark_data_smithi188_113957_object85801:head# 2024-04-23T16:33:38.142 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51d08e4a:::benchmark_data_smithi188_113957_object61365:head# 2024-04-23T16:33:38.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51d0ab82:::benchmark_data_smithi188_113957_object96782:head# 2024-04-23T16:33:38.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51d0cab2:::benchmark_data_smithi188_113957_object118170:head# 2024-04-23T16:33:38.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51d1f46d:::benchmark_data_smithi188_113957_object39551:head# 2024-04-23T16:33:38.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51d30a00:::benchmark_data_smithi188_113957_object83692:head# 2024-04-23T16:33:38.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51d4ca54:::benchmark_data_smithi188_113957_object128824:head# 2024-04-23T16:33:38.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51d4e14d:::benchmark_data_smithi188_113957_object129960:head# 2024-04-23T16:33:38.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51d53f77:::benchmark_data_smithi188_113957_object90025:head# 2024-04-23T16:33:38.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51d560c2:::benchmark_data_smithi188_113957_object134997:head# 2024-04-23T16:33:38.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51d66f80:::benchmark_data_smithi188_113957_object39772:head# 2024-04-23T16:33:38.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51d69caf:::benchmark_data_smithi188_113957_object82048:head# 2024-04-23T16:33:38.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51d74541:::benchmark_data_smithi188_113957_object26829:head# 2024-04-23T16:33:38.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51d74e05:::benchmark_data_smithi188_113957_object122834:head# 2024-04-23T16:33:38.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51d7af0c:::benchmark_data_smithi188_113957_object117580:head# 2024-04-23T16:33:38.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51d87386:::benchmark_data_smithi188_113957_object140704:head# 2024-04-23T16:33:38.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51d8b0df:::benchmark_data_smithi188_113957_object63073:head# 2024-04-23T16:33:38.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51d9dd00:::benchmark_data_smithi188_113957_object4133:head# 2024-04-23T16:33:38.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51da3b40:::benchmark_data_smithi188_113957_object117452:head# 2024-04-23T16:33:38.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51da9bf4:::benchmark_data_smithi188_113957_object121558:head# 2024-04-23T16:33:38.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51db0c77:::benchmark_data_smithi188_113957_object74386:head# 2024-04-23T16:33:38.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51dc8d6b:::benchmark_data_smithi188_113957_object32431:head# 2024-04-23T16:33:38.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51dcca4b:::benchmark_data_smithi188_113957_object143098:head# 2024-04-23T16:33:38.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51dd4870:::benchmark_data_smithi188_113957_object112878:head# 2024-04-23T16:33:38.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51dd6c0e:::benchmark_data_smithi188_113957_object145227:head# 2024-04-23T16:33:38.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ddde98:::benchmark_data_smithi188_113957_object82995:head# 2024-04-23T16:33:38.155 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51de442a:::benchmark_data_smithi188_113957_object95245:head# 2024-04-23T16:33:38.155 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51de899a:::benchmark_data_smithi188_113957_object28417:head# 2024-04-23T16:33:38.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51dfd622:::benchmark_data_smithi188_113957_object84626:head# 2024-04-23T16:33:38.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e092d6:::benchmark_data_smithi188_113957_object62972:head# 2024-04-23T16:33:38.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e0a9da:::benchmark_data_smithi188_113957_object152514:head# 2024-04-23T16:33:38.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e0c501:::benchmark_data_smithi188_113957_object36128:head# 2024-04-23T16:33:38.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e0da78:::benchmark_data_smithi188_113957_object149946:head# 2024-04-23T16:33:38.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e0ef74:::benchmark_data_smithi188_113957_object95808:head# 2024-04-23T16:33:38.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e10247:::benchmark_data_smithi188_113957_object128287:head# 2024-04-23T16:33:38.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e108f1:::benchmark_data_smithi188_113957_object144934:head# 2024-04-23T16:33:38.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e22771:::benchmark_data_smithi188_113957_object30375:head# 2024-04-23T16:33:38.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e22804:::benchmark_data_smithi188_113957_object93692:head# 2024-04-23T16:33:38.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e29a90:::benchmark_data_smithi188_113957_object7127:head# 2024-04-23T16:33:38.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e3469e:::benchmark_data_smithi188_113957_object90717:head# 2024-04-23T16:33:38.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e34d83:::benchmark_data_smithi188_113957_object75586:head# 2024-04-23T16:33:38.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e355aa:::benchmark_data_smithi188_113957_object37914:head# 2024-04-23T16:33:38.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e3b863:::benchmark_data_smithi188_113957_object52057:head# 2024-04-23T16:33:38.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e3c0f5:::benchmark_data_smithi188_113957_object7428:head# 2024-04-23T16:33:38.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e3d7bc:::benchmark_data_smithi188_113957_object145545:head# 2024-04-23T16:33:38.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e51c74:::benchmark_data_smithi188_113957_object70466:head# 2024-04-23T16:33:38.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e5960c:::benchmark_data_smithi188_113957_object114909:head# 2024-04-23T16:33:38.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e5a56f:::benchmark_data_smithi188_113957_object8162:head# 2024-04-23T16:33:38.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e5ea7a:::benchmark_data_smithi188_113957_object85327:head# 2024-04-23T16:33:38.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e5f6d4:::benchmark_data_smithi188_113957_object110199:head# 2024-04-23T16:33:38.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e622df:::benchmark_data_smithi188_113957_object72492:head# 2024-04-23T16:33:38.167 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e6586f:::benchmark_data_smithi188_113957_object16734:head# 2024-04-23T16:33:38.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e873b3:::benchmark_data_smithi188_113957_object2091:head# 2024-04-23T16:33:38.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e89408:::benchmark_data_smithi188_113957_object116068:head# 2024-04-23T16:33:38.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e89e07:::benchmark_data_smithi188_113957_object1327:head# 2024-04-23T16:33:38.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e92d2e:::benchmark_data_smithi188_113957_object45746:head# 2024-04-23T16:33:38.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e973e4:::benchmark_data_smithi188_113957_object139505:head# 2024-04-23T16:33:38.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e981c0:::benchmark_data_smithi188_113957_object74151:head# 2024-04-23T16:33:38.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51e98ecd:::benchmark_data_smithi188_113957_object118599:head# 2024-04-23T16:33:38.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ea73a6:::benchmark_data_smithi188_113957_object64404:head# 2024-04-23T16:33:38.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51eb9b5d:::benchmark_data_smithi188_113957_object67403:head# 2024-04-23T16:33:38.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51eba23e:::benchmark_data_smithi188_113957_object46931:head# 2024-04-23T16:33:38.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ebe5bd:::benchmark_data_smithi188_113957_object45469:head# 2024-04-23T16:33:38.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ec7472:::benchmark_data_smithi188_113957_object149902:head# 2024-04-23T16:33:38.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ec960d:::benchmark_data_smithi188_113957_object31223:head# 2024-04-23T16:33:38.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ed64d6:::benchmark_data_smithi188_113957_object46532:head# 2024-04-23T16:33:38.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ed97dc:::benchmark_data_smithi188_113957_object135988:head# 2024-04-23T16:33:38.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ee38c6:::benchmark_data_smithi188_113957_object114690:head# 2024-04-23T16:33:38.176 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ee71af:::benchmark_data_smithi188_113957_object118817:head# 2024-04-23T16:33:38.176 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f08af5:::benchmark_data_smithi188_113957_object151155:head# 2024-04-23T16:33:38.177 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f09f50:::benchmark_data_smithi188_113957_object67383:head# 2024-04-23T16:33:38.177 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f126bb:::benchmark_data_smithi188_113957_object54197:head# 2024-04-23T16:33:38.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f149c4:::benchmark_data_smithi188_113957_object31188:head# 2024-04-23T16:33:38.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f1f6bf:::benchmark_data_smithi188_113957_object134730:head# 2024-04-23T16:33:38.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f2562b:::benchmark_data_smithi188_113957_object91673:head# 2024-04-23T16:33:38.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f30d0f:::benchmark_data_smithi188_113957_object49734:head# 2024-04-23T16:33:38.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f32955:::benchmark_data_smithi188_113957_object10938:head# 2024-04-23T16:33:38.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f36335:::benchmark_data_smithi188_113957_object57481:head# 2024-04-23T16:33:38.181 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f3ab4d:::benchmark_data_smithi188_113957_object62695:head# 2024-04-23T16:33:38.181 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f3dc0c:::benchmark_data_smithi188_113957_object93693:head# 2024-04-23T16:33:38.181 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f3e735:::benchmark_data_smithi188_113957_object86235:head# 2024-04-23T16:33:38.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f3f026:::benchmark_data_smithi188_113957_object26719:head# 2024-04-23T16:33:38.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f4e58c:::benchmark_data_smithi188_113957_object127848:head# 2024-04-23T16:33:38.183 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f5103b:::benchmark_data_smithi188_113957_object11140:head# 2024-04-23T16:33:38.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f53fc5:::benchmark_data_smithi188_113957_object20558:head# 2024-04-23T16:33:38.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f58ee6:::benchmark_data_smithi188_113957_object9313:head# 2024-04-23T16:33:38.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f6982c:::benchmark_data_smithi188_113957_object15073:head# 2024-04-23T16:33:38.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f71fd0:::benchmark_data_smithi188_113957_object82530:head# 2024-04-23T16:33:38.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f7634b:::benchmark_data_smithi188_113957_object55469:head# 2024-04-23T16:33:38.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f7bbda:::benchmark_data_smithi188_113957_object125214:head# 2024-04-23T16:33:38.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f7f806:::benchmark_data_smithi188_113957_object135473:head# 2024-04-23T16:33:38.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f865f1:::benchmark_data_smithi188_113957_object92693:head# 2024-04-23T16:33:38.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f90331:::benchmark_data_smithi188_113957_object138823:head# 2024-04-23T16:33:38.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f94ca2:::benchmark_data_smithi188_113957_object105819:head# 2024-04-23T16:33:38.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f99118:::benchmark_data_smithi188_113957_object120972:head# 2024-04-23T16:33:38.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f9a9de:::benchmark_data_smithi188_113957_object121834:head# 2024-04-23T16:33:38.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51f9dcbd:::benchmark_data_smithi188_113957_object36991:head# 2024-04-23T16:33:38.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51fa8aa4:::benchmark_data_smithi188_113957_object112070:head# 2024-04-23T16:33:38.192 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51fa8abb:::benchmark_data_smithi188_113957_object152197:head# 2024-04-23T16:33:38.192 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51fad575:::benchmark_data_smithi188_113957_object77400:head# 2024-04-23T16:33:38.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51fadd6d:::benchmark_data_smithi188_113957_object68594:head# 2024-04-23T16:33:38.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51fb1377:::benchmark_data_smithi188_113957_object34344:head# 2024-04-23T16:33:38.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51fc2c36:::benchmark_data_smithi188_113957_object138194:head# 2024-04-23T16:33:38.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51fcd4b4:::benchmark_data_smithi188_113957_object96600:head# 2024-04-23T16:33:38.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51fdde8c:::benchmark_data_smithi188_113957_object110446:head# 2024-04-23T16:33:38.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51fe290c:::benchmark_data_smithi188_113957_object136279:head# 2024-04-23T16:33:38.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51fe5852:::benchmark_data_smithi188_113957_object152215:head# 2024-04-23T16:33:38.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51fe7c81:::benchmark_data_smithi188_113957_object122285:head# 2024-04-23T16:33:38.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51feb6e4:::benchmark_data_smithi188_113957_object43829:head# 2024-04-23T16:33:38.197 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ff4c4e:::benchmark_data_smithi188_113957_object130263:head# 2024-04-23T16:33:38.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ff74d0:::benchmark_data_smithi188_113957_object145301:head# 2024-04-23T16:33:38.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:51ff7ec4:::benchmark_data_smithi188_113957_object78525:head# 2024-04-23T16:33:38.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52007584:::benchmark_data_smithi188_113957_object75790:head# 2024-04-23T16:33:38.199 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:520078be:::benchmark_data_smithi188_113957_object152789:head# 2024-04-23T16:33:38.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5200a140:::benchmark_data_smithi188_113957_object151417:head# 2024-04-23T16:33:38.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5200ae6c:::benchmark_data_smithi188_113957_object45097:head# 2024-04-23T16:33:38.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5200e47c:::benchmark_data_smithi188_113957_object131807:head# 2024-04-23T16:33:38.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5200f9c2:::benchmark_data_smithi188_113957_object107873:head# 2024-04-23T16:33:38.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52015e0f:::benchmark_data_smithi188_113957_object22375:head# 2024-04-23T16:33:38.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5201d10d:::benchmark_data_smithi188_113957_object113191:head# 2024-04-23T16:33:38.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5202617b:::benchmark_data_smithi188_113957_object7823:head# 2024-04-23T16:33:38.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52027f18:::benchmark_data_smithi188_113957_object90752:head# 2024-04-23T16:33:38.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52030ac4:::benchmark_data_smithi188_113957_object139232:head# 2024-04-23T16:33:38.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52033e9e:::benchmark_data_smithi188_113957_object113267:head# 2024-04-23T16:33:38.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52043e34:::benchmark_data_smithi188_113957_object46958:head# 2024-04-23T16:33:38.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5204c550:::benchmark_data_smithi188_113957_object85587:head# 2024-04-23T16:33:38.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5204c6f4:::benchmark_data_smithi188_113957_object104245:head# 2024-04-23T16:33:38.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52052962:::benchmark_data_smithi188_113957_object79583:head# 2024-04-23T16:33:38.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5205494a:::benchmark_data_smithi188_113957_object37706:head# 2024-04-23T16:33:38.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52057531:::benchmark_data_smithi188_113957_object5646:head# 2024-04-23T16:33:38.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52064134:::benchmark_data_smithi188_113957_object100535:head# 2024-04-23T16:33:38.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52064b00:::benchmark_data_smithi188_113957_object148882:head# 2024-04-23T16:33:38.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52077a4c:::benchmark_data_smithi188_113957_object131112:head# 2024-04-23T16:33:38.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5207d825:::benchmark_data_smithi188_113957_object52722:head# 2024-04-23T16:33:38.212 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5208ff7d:::benchmark_data_smithi188_113957_object88861:head# 2024-04-23T16:33:38.212 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52090b25:::benchmark_data_smithi188_113957_object13150:head# 2024-04-23T16:33:38.213 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5209107c:::benchmark_data_smithi188_113957_object8467:head# 2024-04-23T16:33:38.213 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5209bf50:::benchmark_data_smithi188_113957_object5808:head# 2024-04-23T16:33:38.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5209fe55:::benchmark_data_smithi188_113957_object140583:head# 2024-04-23T16:33:38.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:520a32bb:::benchmark_data_smithi188_113957_object77215:head# 2024-04-23T16:33:38.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:520b22cd:::benchmark_data_smithi188_113957_object96506:head# 2024-04-23T16:33:38.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:520b239b:::benchmark_data_smithi188_113957_object149069:head# 2024-04-23T16:33:38.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:520b6aec:::benchmark_data_smithi188_113957_object18655:head# 2024-04-23T16:33:38.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:520ba87e:::benchmark_data_smithi188_113957_object100184:head# 2024-04-23T16:33:38.216 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:520ce57a:::benchmark_data_smithi188_113957_object60678:head# 2024-04-23T16:33:38.216 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:520d42de:::benchmark_data_smithi188_113957_object59030:head# 2024-04-23T16:33:38.217 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:520d9105:::benchmark_data_smithi188_113957_object86828:head# 2024-04-23T16:33:38.217 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:520d9a51:::benchmark_data_smithi188_113957_object7296:head# 2024-04-23T16:33:38.217 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:520dd72c:::benchmark_data_smithi188_113957_object90313:head# 2024-04-23T16:33:38.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:520e08d5:::benchmark_data_smithi188_113957_object86126:head# 2024-04-23T16:33:38.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:520eadf0:::benchmark_data_smithi188_113957_object130340:head# 2024-04-23T16:33:38.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:520ff9ca:::benchmark_data_smithi188_113957_object32355:head# 2024-04-23T16:33:38.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52101e8a:::benchmark_data_smithi188_113957_object123782:head# 2024-04-23T16:33:38.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5210f609:::benchmark_data_smithi188_113957_object119761:head# 2024-04-23T16:33:38.221 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52111d23:::benchmark_data_smithi188_113957_object99312:head# 2024-04-23T16:33:38.221 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52114b83:::benchmark_data_smithi188_113957_object87341:head# 2024-04-23T16:33:38.222 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:521187c9:::benchmark_data_smithi188_113957_object446:head# 2024-04-23T16:33:38.222 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5211b08b:::benchmark_data_smithi188_113957_object153116:head# 2024-04-23T16:33:38.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5212065b:::benchmark_data_smithi188_113957_object126935:head# 2024-04-23T16:33:38.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52121e7b:::benchmark_data_smithi188_113957_object61918:head# 2024-04-23T16:33:38.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:37 smithi188 ceph-mon[44101]: pgmap v1117: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 9.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 82 KiB/s wr, 406 op/s; 28217/299060 objects degraded (9.435%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:33:38.224 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52122ec6:::benchmark_data_smithi188_113957_object148277:head# 2024-04-23T16:33:38.224 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5212cc6b:::benchmark_data_smithi188_113957_object145302:head# 2024-04-23T16:33:38.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5213cacb:::benchmark_data_smithi188_113957_object2945:head# 2024-04-23T16:33:38.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52148d05:::benchmark_data_smithi188_113957_object39600:head# 2024-04-23T16:33:38.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52150c03:::benchmark_data_smithi188_113957_object3701:head# 2024-04-23T16:33:38.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52152692:::benchmark_data_smithi188_113957_object48092:head# 2024-04-23T16:33:38.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5216a1a6:::benchmark_data_smithi188_113957_object31128:head# 2024-04-23T16:33:38.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5216ae0c:::benchmark_data_smithi188_113957_object68778:head# 2024-04-23T16:33:38.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5217386f:::benchmark_data_smithi188_113957_object25741:head# 2024-04-23T16:33:38.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5217614c:::benchmark_data_smithi188_113957_object107753:head# 2024-04-23T16:33:38.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5217d878:::benchmark_data_smithi188_113957_object61866:head# 2024-04-23T16:33:38.231 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52180308:::benchmark_data_smithi188_113957_object16758:head# 2024-04-23T16:33:38.231 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52187291:::benchmark_data_smithi188_113957_object141917:head# 2024-04-23T16:33:38.231 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5218c7bd:::benchmark_data_smithi188_113957_object1918:head# 2024-04-23T16:33:38.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52190964:::benchmark_data_smithi188_113957_object47435:head# 2024-04-23T16:33:38.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:521967d9:::benchmark_data_smithi188_113957_object15601:head# 2024-04-23T16:33:38.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5219b5ec:::benchmark_data_smithi188_113957_object16495:head# 2024-04-23T16:33:38.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:521aa929:::benchmark_data_smithi188_113957_object31894:head# 2024-04-23T16:33:38.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:521ae99c:::benchmark_data_smithi188_113957_object12240:head# 2024-04-23T16:33:38.234 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:521af5fe:::benchmark_data_smithi188_113957_object119820:head# 2024-04-23T16:33:38.234 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:521b349e:::benchmark_data_smithi188_113957_object84689:head# 2024-04-23T16:33:38.235 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:521c06b7:::benchmark_data_smithi188_113957_object11187:head# 2024-04-23T16:33:38.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:521d1737:::benchmark_data_smithi188_113957_object59857:head# 2024-04-23T16:33:38.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:521d3f90:::benchmark_data_smithi188_113957_object63432:head# 2024-04-23T16:33:38.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:521d6b9f:::benchmark_data_smithi188_113957_object138780:head# 2024-04-23T16:33:38.237 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:521d718a:::benchmark_data_smithi188_113957_object147940:head# 2024-04-23T16:33:38.237 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:521e87ef:::benchmark_data_smithi188_113957_object79557:head# 2024-04-23T16:33:38.237 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:521e89a8:::benchmark_data_smithi188_113957_object130322:head# 2024-04-23T16:33:38.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:521e8b1d:::benchmark_data_smithi188_113957_object142719:head# 2024-04-23T16:33:38.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:521eca3b:::benchmark_data_smithi188_113957_object58535:head# 2024-04-23T16:33:38.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:521f0713:::benchmark_data_smithi188_113957_object37645:head# 2024-04-23T16:33:38.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:521f5e40:::benchmark_data_smithi188_113957_object149607:head# 2024-04-23T16:33:38.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:521f94f5:::benchmark_data_smithi188_113957_object46185:head# 2024-04-23T16:33:38.240 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5220b7e5:::benchmark_data_smithi188_113957_object140714:head# 2024-04-23T16:33:38.240 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5221166e:::benchmark_data_smithi188_113957_object83585:head# 2024-04-23T16:33:38.241 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52225b92:::benchmark_data_smithi188_113957_object23309:head# 2024-04-23T16:33:38.241 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52228e27:::benchmark_data_smithi188_113957_object140200:head# 2024-04-23T16:33:38.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5222aa6d:::benchmark_data_smithi188_113957_object148626:head# 2024-04-23T16:33:38.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5222f7f4:::benchmark_data_smithi188_113957_object122162:head# 2024-04-23T16:33:38.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:522359e0:::benchmark_data_smithi188_113957_object96289:head# 2024-04-23T16:33:38.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5225d390:::benchmark_data_smithi188_113957_object68049:head# 2024-04-23T16:33:38.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5226db88:::benchmark_data_smithi188_113957_object9977:head# 2024-04-23T16:33:38.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52275877:::benchmark_data_smithi188_113957_object32285:head# 2024-04-23T16:33:38.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52275a44:::benchmark_data_smithi188_113957_object62634:head# 2024-04-23T16:33:38.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52276683:::benchmark_data_smithi188_113957_object56047:head# 2024-04-23T16:33:38.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:522909d9:::benchmark_data_smithi188_113957_object80780:head# 2024-04-23T16:33:38.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:522949f5:::benchmark_data_smithi188_113957_object3526:head# 2024-04-23T16:33:38.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5229bf34:::benchmark_data_smithi188_113957_object103517:head# 2024-04-23T16:33:38.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5229ca0a:::benchmark_data_smithi188_113957_object56367:head# 2024-04-23T16:33:38.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:522a2704:::benchmark_data_smithi188_113957_object130841:head# 2024-04-23T16:33:38.248 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:522aafb6:::benchmark_data_smithi188_113957_object85425:head# 2024-04-23T16:33:38.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:522ad75c:::benchmark_data_smithi188_113957_object147864:head# 2024-04-23T16:33:38.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:522b4da0:::benchmark_data_smithi188_113957_object75210:head# 2024-04-23T16:33:38.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:522ce784:::benchmark_data_smithi188_113957_object55520:head# 2024-04-23T16:33:38.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:522d0f21:::benchmark_data_smithi188_113957_object77330:head# 2024-04-23T16:33:38.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:522de997:::benchmark_data_smithi188_113957_object130801:head# 2024-04-23T16:33:38.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:522e410a:::benchmark_data_smithi188_113957_object35868:head# 2024-04-23T16:33:38.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:522e4c3c:::benchmark_data_smithi188_113957_object54593:head# 2024-04-23T16:33:38.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:522e5257:::benchmark_data_smithi188_113957_object3656:head# 2024-04-23T16:33:38.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:522eb588:::benchmark_data_smithi188_113957_object105678:head# 2024-04-23T16:33:38.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:522f34b3:::benchmark_data_smithi188_113957_object42133:head# 2024-04-23T16:33:38.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:522f7107:::benchmark_data_smithi188_113957_object31357:head# 2024-04-23T16:33:38.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5230440c:::benchmark_data_smithi188_113957_object27280:head# 2024-04-23T16:33:38.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5232062c:::benchmark_data_smithi188_113957_object7726:head# 2024-04-23T16:33:38.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:523233ac:::benchmark_data_smithi188_113957_object91893:head# 2024-04-23T16:33:38.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52325a1a:::benchmark_data_smithi188_113957_object105862:head# 2024-04-23T16:33:38.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52325e8f:::benchmark_data_smithi188_113957_object130900:head# 2024-04-23T16:33:38.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:523364c0:::benchmark_data_smithi188_113957_object35421:head# 2024-04-23T16:33:38.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:523424ef:::benchmark_data_smithi188_113957_object142978:head# 2024-04-23T16:33:38.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:523488f9:::benchmark_data_smithi188_113957_object107155:head# 2024-04-23T16:33:38.258 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5234b013:::benchmark_data_smithi188_113957_object20897:head# 2024-04-23T16:33:38.258 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5234ff27:::benchmark_data_smithi188_113957_object145809:head# 2024-04-23T16:33:38.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5235131d:::benchmark_data_smithi188_113957_object54918:head# 2024-04-23T16:33:38.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52355a66:::benchmark_data_smithi188_113957_object34386:head# 2024-04-23T16:33:38.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52359550:::benchmark_data_smithi188_113957_object133626:head# 2024-04-23T16:33:38.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5235bc64:::benchmark_data_smithi188_113957_object24819:head# 2024-04-23T16:33:38.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52361127:::benchmark_data_smithi188_113957_object7178:head# 2024-04-23T16:33:38.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5236448b:::benchmark_data_smithi188_113957_object9732:head# 2024-04-23T16:33:38.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5236e4f2:::benchmark_data_smithi188_113957_object9325:head# 2024-04-23T16:33:38.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52375455:::benchmark_data_smithi188_113957_object34530:head# 2024-04-23T16:33:38.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52376bc7:::benchmark_data_smithi188_113957_object90031:head# 2024-04-23T16:33:38.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5237a01f:::benchmark_data_smithi188_113957_object60707:head# 2024-04-23T16:33:38.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5237d23f:::benchmark_data_smithi188_113957_object126954:head# 2024-04-23T16:33:38.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5238186a:::benchmark_data_smithi188_113957_object72027:head# 2024-04-23T16:33:38.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52384617:::benchmark_data_smithi188_113957_object14371:head# 2024-04-23T16:33:38.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5238c6a0:::benchmark_data_smithi188_113957_object149220:head# 2024-04-23T16:33:38.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5239c0ca:::benchmark_data_smithi188_113957_object22100:head# 2024-04-23T16:33:38.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:523a0b89:::benchmark_data_smithi188_113957_object45338:head# 2024-04-23T16:33:38.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:523a6be9:::benchmark_data_smithi188_113957_object29369:head# 2024-04-23T16:33:38.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:523bdd52:::benchmark_data_smithi188_113957_object22752:head# 2024-04-23T16:33:38.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:523c2504:::benchmark_data_smithi188_113957_object65373:head# 2024-04-23T16:33:38.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:523c50e9:::benchmark_data_smithi188_113957_object55172:head# 2024-04-23T16:33:38.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:523cd450:::benchmark_data_smithi188_113957_object50355:head# 2024-04-23T16:33:38.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:523f7051:::benchmark_data_smithi188_113957_object28351:head# 2024-04-23T16:33:38.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:523f8936:::benchmark_data_smithi188_113957_object103839:head# 2024-04-23T16:33:38.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:523fa31b:::benchmark_data_smithi188_113957_object141049:head# 2024-04-23T16:33:38.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:523fc094:::benchmark_data_smithi188_113957_object127281:head# 2024-04-23T16:33:38.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:523feb1b:::benchmark_data_smithi188_113957_object94595:head# 2024-04-23T16:33:38.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:524072fe:::benchmark_data_smithi188_113957_object92447:head# 2024-04-23T16:33:38.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5240dbe2:::benchmark_data_smithi188_113957_object105135:head# 2024-04-23T16:33:38.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5241b17c:::benchmark_data_smithi188_113957_object64250:head# 2024-04-23T16:33:38.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5241b699:::benchmark_data_smithi188_113957_object105375:head# 2024-04-23T16:33:38.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52420f84:::benchmark_data_smithi188_113957_object91505:head# 2024-04-23T16:33:38.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52422d5e:::benchmark_data_smithi188_113957_object85631:head# 2024-04-23T16:33:38.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:524252bb:::benchmark_data_smithi188_113957_object71311:head# 2024-04-23T16:33:38.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:524302f9:::benchmark_data_smithi188_113957_object103519:head# 2024-04-23T16:33:38.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5243685c:::benchmark_data_smithi188_113957_object74488:head# 2024-04-23T16:33:38.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:524371ee:::benchmark_data_smithi188_113957_object50844:head# 2024-04-23T16:33:38.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52447e94:::benchmark_data_smithi188_113957_object142093:head# 2024-04-23T16:33:38.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5244d143:::benchmark_data_smithi188_113957_object110331:head# 2024-04-23T16:33:38.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5245305d:::benchmark_data_smithi188_113957_object5541:head# 2024-04-23T16:33:38.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5245f49a:::benchmark_data_smithi188_113957_object37510:head# 2024-04-23T16:33:38.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52460453:::benchmark_data_smithi188_113957_object3934:head# 2024-04-23T16:33:38.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52466dd1:::benchmark_data_smithi188_113957_object32122:head# 2024-04-23T16:33:38.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:524671bb:::benchmark_data_smithi188_113957_object9891:head# 2024-04-23T16:33:38.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5246969c:::benchmark_data_smithi188_113957_object124241:head# 2024-04-23T16:33:38.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52471548:::benchmark_data_smithi188_113957_object144753:head# 2024-04-23T16:33:38.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52474525:::benchmark_data_smithi188_113957_object4900:head# 2024-04-23T16:33:38.287 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5247ab19:::benchmark_data_smithi188_113957_object137473:head# 2024-04-23T16:33:38.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5247c8d8:::benchmark_data_smithi188_113957_object5217:head# 2024-04-23T16:33:38.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5248406a:::benchmark_data_smithi188_113957_object52560:head# 2024-04-23T16:33:38.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52484846:::benchmark_data_smithi188_113957_object25308:head# 2024-04-23T16:33:38.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5248d2df:::benchmark_data_smithi188_113957_object46121:head# 2024-04-23T16:33:38.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5248e96f:::benchmark_data_smithi188_113957_object47178:head# 2024-04-23T16:33:38.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52499490:::benchmark_data_smithi188_113957_object75187:head# 2024-04-23T16:33:38.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5249f7bd:::benchmark_data_smithi188_113957_object46400:head# 2024-04-23T16:33:38.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:524a0beb:::benchmark_data_smithi188_113957_object100544:head# 2024-04-23T16:33:38.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:524b103b:::benchmark_data_smithi188_113957_object150809:head# 2024-04-23T16:33:38.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:524b531d:::benchmark_data_smithi188_113957_object92680:head# 2024-04-23T16:33:38.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:524c0a14:::benchmark_data_smithi188_113957_object104209:head# 2024-04-23T16:33:38.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:524e113d:::benchmark_data_smithi188_113957_object62507:head# 2024-04-23T16:33:38.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:524e8b42:::benchmark_data_smithi188_113957_object90107:head# 2024-04-23T16:33:38.295 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:524f424b:::benchmark_data_smithi188_113957_object100476:head# 2024-04-23T16:33:38.295 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:524f47c5:::benchmark_data_smithi188_113957_object19221:head# 2024-04-23T16:33:38.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:524f8c8d:::benchmark_data_smithi188_113957_object82982:head# 2024-04-23T16:33:38.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:524fa6b3:::benchmark_data_smithi188_113957_object74907:head# 2024-04-23T16:33:38.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:524fb75b:::benchmark_data_smithi188_113957_object57035:head# 2024-04-23T16:33:38.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:524fc2db:::benchmark_data_smithi188_113957_object107950:head# 2024-04-23T16:33:38.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:524fd792:::benchmark_data_smithi188_113957_object4822:head# 2024-04-23T16:33:38.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5250f900:::benchmark_data_smithi188_113957_object41609:head# 2024-04-23T16:33:38.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:525180b0:::benchmark_data_smithi188_113957_object95417:head# 2024-04-23T16:33:38.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52518493:::benchmark_data_smithi188_113957_object27335:head# 2024-04-23T16:33:38.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5252278b:::benchmark_data_smithi188_113957_object13086:head# 2024-04-23T16:33:38.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52533d90:::benchmark_data_smithi188_113957_object143198:head# 2024-04-23T16:33:38.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5254841a:::benchmark_data_smithi188_113957_object43433:head# 2024-04-23T16:33:38.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5254cdcd:::benchmark_data_smithi188_113957_object116161:head# 2024-04-23T16:33:38.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5255dd4e:::benchmark_data_smithi188_113957_object64668:head# 2024-04-23T16:33:38.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5255f77b:::benchmark_data_smithi188_113957_object78377:head# 2024-04-23T16:33:38.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:525617e5:::benchmark_data_smithi188_113957_object103361:head# 2024-04-23T16:33:38.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:525670f7:::benchmark_data_smithi188_113957_object74341:head# 2024-04-23T16:33:38.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5256a01c:::benchmark_data_smithi188_113957_object53145:head# 2024-04-23T16:33:38.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5256bd2a:::benchmark_data_smithi188_113957_object17071:head# 2024-04-23T16:33:38.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5256f7d1:::benchmark_data_smithi188_113957_object8574:head# 2024-04-23T16:33:38.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5257e967:::benchmark_data_smithi188_113957_object19748:head# 2024-04-23T16:33:38.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52583dcd:::benchmark_data_smithi188_113957_object2528:head# 2024-04-23T16:33:38.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5258756e:::benchmark_data_smithi188_113957_object9899:head# 2024-04-23T16:33:38.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5258918c:::benchmark_data_smithi188_113957_object117745:head# 2024-04-23T16:33:38.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52590b5e:::benchmark_data_smithi188_113957_object36270:head# 2024-04-23T16:33:38.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5259113f:::benchmark_data_smithi188_113957_object115684:head# 2024-04-23T16:33:38.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52595403:::benchmark_data_smithi188_113957_object15448:head# 2024-04-23T16:33:38.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52598b52:::benchmark_data_smithi188_113957_object12388:head# 2024-04-23T16:33:38.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:525a4fa3:::benchmark_data_smithi188_113957_object70427:head# 2024-04-23T16:33:38.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:525ac089:::benchmark_data_smithi188_113957_object121919:head# 2024-04-23T16:33:38.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:525ad065:::benchmark_data_smithi188_113957_object1578:head# 2024-04-23T16:33:38.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:525bb162:::benchmark_data_smithi188_113957_object106111:head# 2024-04-23T16:33:38.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:525c43c5:::benchmark_data_smithi188_113957_object981:head# 2024-04-23T16:33:38.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:525c654a:::benchmark_data_smithi188_113957_object59796:head# 2024-04-23T16:33:38.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:525cc68c:::benchmark_data_smithi188_113957_object89436:head# 2024-04-23T16:33:38.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:525e3c12:::benchmark_data_smithi188_113957_object47310:head# 2024-04-23T16:33:38.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:525eadf1:::benchmark_data_smithi188_113957_object115956:head# 2024-04-23T16:33:38.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:525f0bd2:::benchmark_data_smithi188_113957_object138048:head# 2024-04-23T16:33:38.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:525f3427:::benchmark_data_smithi188_113957_object53558:head# 2024-04-23T16:33:38.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:525f8775:::benchmark_data_smithi188_113957_object153336:head# 2024-04-23T16:33:38.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:526042ac:::benchmark_data_smithi188_113957_object129687:head# 2024-04-23T16:33:38.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:526066fb:::benchmark_data_smithi188_113957_object91810:head# 2024-04-23T16:33:38.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5260ded6:::benchmark_data_smithi188_113957_object57388:head# 2024-04-23T16:33:38.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:526169a3:::benchmark_data_smithi188_113957_object4376:head# 2024-04-23T16:33:38.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:526169ea:::benchmark_data_smithi188_113957_object5483:head# 2024-04-23T16:33:38.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5261be34:::benchmark_data_smithi188_113957_object121411:head# 2024-04-23T16:33:38.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5261d8f5:::benchmark_data_smithi188_113957_object8799:head# 2024-04-23T16:33:38.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5261ed1d:::benchmark_data_smithi188_113957_object70623:head# 2024-04-23T16:33:38.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5261ef47:::benchmark_data_smithi188_113957_object138347:head# 2024-04-23T16:33:38.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5262cc93:::benchmark_data_smithi188_113957_object132019:head# 2024-04-23T16:33:38.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5262cda3:::benchmark_data_smithi188_113957_object7053:head# 2024-04-23T16:33:38.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5263b7ab:::benchmark_data_smithi188_113957_object80576:head# 2024-04-23T16:33:38.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5263fcac:::benchmark_data_smithi188_113957_object119046:head# 2024-04-23T16:33:38.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5264a2f9:::benchmark_data_smithi188_113957_object14500:head# 2024-04-23T16:33:38.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5264eeb6:::benchmark_data_smithi188_113957_object83440:head# 2024-04-23T16:33:38.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52650450:::benchmark_data_smithi188_113957_object40470:head# 2024-04-23T16:33:38.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52651b25:::benchmark_data_smithi188_113957_object122991:head# 2024-04-23T16:33:38.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5266050f:::benchmark_data_smithi188_113957_object52633:head# 2024-04-23T16:33:38.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52663c08:::benchmark_data_smithi188_113957_object54506:head# 2024-04-23T16:33:38.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:526661f5:::benchmark_data_smithi188_113957_object56861:head# 2024-04-23T16:33:38.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5266f1a1:::benchmark_data_smithi188_113957_object31668:head# 2024-04-23T16:33:38.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5266fcdb:::benchmark_data_smithi188_113957_object42033:head# 2024-04-23T16:33:38.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52671aed:::benchmark_data_smithi188_113957_object103470:head# 2024-04-23T16:33:38.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5267ad21:::benchmark_data_smithi188_113957_object105631:head# 2024-04-23T16:33:38.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52698638:::benchmark_data_smithi188_113957_object11780:head# 2024-04-23T16:33:38.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5269f667:::benchmark_data_smithi188_113957_object84987:head# 2024-04-23T16:33:38.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:526b2b7c:::benchmark_data_smithi188_113957_object126432:head# 2024-04-23T16:33:38.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:526c2009:::benchmark_data_smithi188_113957_object72130:head# 2024-04-23T16:33:38.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:526c906b:::benchmark_data_smithi188_113957_object67033:head# 2024-04-23T16:33:38.326 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:526ca625:::benchmark_data_smithi188_113957_object52832:head# 2024-04-23T16:33:38.326 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:526cfeea:::benchmark_data_smithi188_113957_object74110:head# 2024-04-23T16:33:38.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:526d246d:::benchmark_data_smithi188_113957_object147176:head# 2024-04-23T16:33:38.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:526d4227:::benchmark_data_smithi188_113957_object95990:head# 2024-04-23T16:33:38.328 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:526d9525:::benchmark_data_smithi188_113957_object83232:head# 2024-04-23T16:33:38.328 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:526dc44a:::benchmark_data_smithi188_113957_object37771:head# 2024-04-23T16:33:38.328 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:526ebbf1:::benchmark_data_smithi188_113957_object81482:head# 2024-04-23T16:33:38.329 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:526f5900:::benchmark_data_smithi188_113957_object5526:head# 2024-04-23T16:33:38.329 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:526f8595:::benchmark_data_smithi188_113957_object151348:head# 2024-04-23T16:33:38.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:526f8dfe:::benchmark_data_smithi188_113957_object105159:head# 2024-04-23T16:33:38.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:526f95a3:::benchmark_data_smithi188_113957_object17838:head# 2024-04-23T16:33:38.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5270c198:::benchmark_data_smithi188_113957_object123547:head# 2024-04-23T16:33:38.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52710f44:::benchmark_data_smithi188_113957_object61423:head# 2024-04-23T16:33:38.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5271b132:::benchmark_data_smithi188_113957_object83301:head# 2024-04-23T16:33:38.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52720c2a:::benchmark_data_smithi188_113957_object138953:head# 2024-04-23T16:33:38.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:527242b1:::benchmark_data_smithi188_113957_object32821:head# 2024-04-23T16:33:38.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5273b36c:::benchmark_data_smithi188_113957_object27517:head# 2024-04-23T16:33:38.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5273e578:::benchmark_data_smithi188_113957_object38110:head# 2024-04-23T16:33:38.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5274a4ee:::benchmark_data_smithi188_113957_object48543:head# 2024-04-23T16:33:38.334 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5274c1df:::benchmark_data_smithi188_113957_object110969:head# 2024-04-23T16:33:38.334 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5274cde7:::benchmark_data_smithi188_113957_object104901:head# 2024-04-23T16:33:38.334 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52756eef:::benchmark_data_smithi188_113957_object101749:head# 2024-04-23T16:33:38.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:527594bb:::benchmark_data_smithi188_113957_object59006:head# 2024-04-23T16:33:38.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52766348:::benchmark_data_smithi188_113957_object89576:head# 2024-04-23T16:33:38.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5276a5be:::benchmark_data_smithi188_113957_object149162:head# 2024-04-23T16:33:38.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5276a6f8:::benchmark_data_smithi188_113957_object97908:head# 2024-04-23T16:33:38.337 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:527813b3:::benchmark_data_smithi188_113957_object117448:head# 2024-04-23T16:33:38.337 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:527817c8:::benchmark_data_smithi188_113957_object66842:head# 2024-04-23T16:33:38.338 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52784b9a:::benchmark_data_smithi188_113957_object140360:head# 2024-04-23T16:33:38.338 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52786d86:::benchmark_data_smithi188_113957_object141865:head# 2024-04-23T16:33:38.338 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5278fb95:::benchmark_data_smithi188_113957_object89965:head# 2024-04-23T16:33:38.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52791296:::benchmark_data_smithi188_113957_object66209:head# 2024-04-23T16:33:38.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:527915f5:::benchmark_data_smithi188_113957_object102687:head# 2024-04-23T16:33:38.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5279282a:::benchmark_data_smithi188_113957_object61971:head# 2024-04-23T16:33:38.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:527a1e0e:::benchmark_data_smithi188_113957_object8660:head# 2024-04-23T16:33:38.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:527a3d91:::benchmark_data_smithi188_113957_object64048:head# 2024-04-23T16:33:38.341 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:527b1e30:::benchmark_data_smithi188_113957_object136831:head# 2024-04-23T16:33:38.342 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:527b9d28:::benchmark_data_smithi188_113957_object22526:head# 2024-04-23T16:33:38.342 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:527bb8bf:::benchmark_data_smithi188_113957_object21454:head# 2024-04-23T16:33:38.342 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:527beaff:::benchmark_data_smithi188_113957_object104496:head# 2024-04-23T16:33:38.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:527c2c75:::benchmark_data_smithi188_113957_object3269:head# 2024-04-23T16:33:38.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:527e0410:::benchmark_data_smithi188_113957_object15902:head# 2024-04-23T16:33:38.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:527e22ba:::benchmark_data_smithi188_113957_object16864:head# 2024-04-23T16:33:38.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:527f8b02:::benchmark_data_smithi188_113957_object70737:head# 2024-04-23T16:33:38.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52800204:::benchmark_data_smithi188_113957_object81788:head# 2024-04-23T16:33:38.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5280f212:::benchmark_data_smithi188_113957_object66983:head# 2024-04-23T16:33:38.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:528185a7:::benchmark_data_smithi188_113957_object103959:head# 2024-04-23T16:33:38.346 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5281caca:::benchmark_data_smithi188_113957_object135736:head# 2024-04-23T16:33:38.346 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52833543:::benchmark_data_smithi188_113957_object68618:head# 2024-04-23T16:33:38.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52834aaa:::benchmark_data_smithi188_113957_object48691:head# 2024-04-23T16:33:38.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5283d6ec:::benchmark_data_smithi188_113957_object135182:head# 2024-04-23T16:33:38.348 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5283dcdc:::benchmark_data_smithi188_113957_object125371:head# 2024-04-23T16:33:38.348 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:528494f2:::benchmark_data_smithi188_113957_object131485:head# 2024-04-23T16:33:38.348 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52852235:::benchmark_data_smithi188_113957_object147971:head# 2024-04-23T16:33:38.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52852834:::benchmark_data_smithi188_113957_object7820:head# 2024-04-23T16:33:38.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52853b78:::benchmark_data_smithi188_113957_object105463:head# 2024-04-23T16:33:38.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52857973:::benchmark_data_smithi188_113957_object142697:head# 2024-04-23T16:33:38.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:528582b7:::benchmark_data_smithi188_113957_object103000:head# 2024-04-23T16:33:38.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5285f488:::benchmark_data_smithi188_113957_object139974:head# 2024-04-23T16:33:38.353 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52865df0:::benchmark_data_smithi188_113957_object3419:head# 2024-04-23T16:33:38.353 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52866027:::benchmark_data_smithi188_113957_object88000:head# 2024-04-23T16:33:38.353 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52876b92:::benchmark_data_smithi188_113957_object103067:head# 2024-04-23T16:33:38.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:528784eb:::benchmark_data_smithi188_113957_object35399:head# 2024-04-23T16:33:38.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5287b5c9:::benchmark_data_smithi188_113957_object60714:head# 2024-04-23T16:33:38.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52888c1c:::benchmark_data_smithi188_113957_object61340:head# 2024-04-23T16:33:38.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52888ee7:::benchmark_data_smithi188_113957_object50650:head# 2024-04-23T16:33:38.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52892316:::benchmark_data_smithi188_113957_object65331:head# 2024-04-23T16:33:38.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5289547e:::benchmark_data_smithi188_113957_object144334:head# 2024-04-23T16:33:38.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:528ae044:::benchmark_data_smithi188_113957_object97385:head# 2024-04-23T16:33:38.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:528aebbf:::benchmark_data_smithi188_113957_object73367:head# 2024-04-23T16:33:38.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:528b8178:::benchmark_data_smithi188_113957_object63405:head# 2024-04-23T16:33:38.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:528c53d7:::benchmark_data_smithi188_113957_object66144:head# 2024-04-23T16:33:38.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:528d2624:::benchmark_data_smithi188_113957_object127587:head# 2024-04-23T16:33:38.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:528f2b4a:::benchmark_data_smithi188_113957_object83407:head# 2024-04-23T16:33:38.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52903bfb:::benchmark_data_smithi188_113957_object83673:head# 2024-04-23T16:33:38.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5290d368:::benchmark_data_smithi188_113957_object81871:head# 2024-04-23T16:33:38.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:529172f3:::benchmark_data_smithi188_113957_object80290:head# 2024-04-23T16:33:38.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:529180b2:::benchmark_data_smithi188_113957_object69479:head# 2024-04-23T16:33:38.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5291ba2f:::benchmark_data_smithi188_113957_object25965:head# 2024-04-23T16:33:38.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5292a402:::benchmark_data_smithi188_113957_object96882:head# 2024-04-23T16:33:38.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52931302:::benchmark_data_smithi188_113957_object111174:head# 2024-04-23T16:33:38.361 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52935397:::benchmark_data_smithi188_113957_object149933:head# 2024-04-23T16:33:38.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52938a7c:::benchmark_data_smithi188_113957_object107740:head# 2024-04-23T16:33:38.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5293b242:::benchmark_data_smithi188_113957_object58105:head# 2024-04-23T16:33:38.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5293e13f:::benchmark_data_smithi188_113957_object53995:head# 2024-04-23T16:33:38.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5294020d:::benchmark_data_smithi188_113957_object120074:head# 2024-04-23T16:33:38.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52943647:::benchmark_data_smithi188_113957_object55140:head# 2024-04-23T16:33:38.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5294a234:::benchmark_data_smithi188_113957_object11791:head# 2024-04-23T16:33:38.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52958890:::benchmark_data_smithi188_113957_object23046:head# 2024-04-23T16:33:38.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5295ae1b:::benchmark_data_smithi188_113957_object36896:head# 2024-04-23T16:33:38.366 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5296004f:::benchmark_data_smithi188_113957_object94662:head# 2024-04-23T16:33:38.366 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:529901dc:::benchmark_data_smithi188_113957_object76983:head# 2024-04-23T16:33:38.367 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52990a0f:::benchmark_data_smithi188_113957_object61600:head# 2024-04-23T16:33:38.367 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52994b25:::benchmark_data_smithi188_113957_object145430:head# 2024-04-23T16:33:38.368 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5299635d:::benchmark_data_smithi188_113957_object89843:head# 2024-04-23T16:33:38.368 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:33:38.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5299bf71:::benchmark_data_smithi188_113957_object12759:head# 2024-04-23T16:33:38.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:529a269e:::benchmark_data_smithi188_113957_object87065:head# 2024-04-23T16:33:38.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:529a3118:::benchmark_data_smithi188_113957_object27126:head# 2024-04-23T16:33:38.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:529aef78:::benchmark_data_smithi188_113957_object53204:head# 2024-04-23T16:33:38.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:529bd5cc:::benchmark_data_smithi188_113957_object42531:head# 2024-04-23T16:33:38.371 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:529c8e2b:::benchmark_data_smithi188_113957_object152327:head# 2024-04-23T16:33:38.371 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:529c8ff7:::benchmark_data_smithi188_113957_object39928:head# 2024-04-23T16:33:38.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:38.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:33:38.373 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:529c9a38:::benchmark_data_smithi188_113957_object137973:head# 2024-04-23T16:33:38.373 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:529dfd1e:::benchmark_data_smithi188_113957_object60594:head# 2024-04-23T16:33:38.373 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:529e38c8:::benchmark_data_smithi188_113957_object94299:head# 2024-04-23T16:33:38.373 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:529e94ca:::benchmark_data_smithi188_113957_object122331:head# 2024-04-23T16:33:38.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:529f7870:::benchmark_data_smithi188_113957_object133964:head# 2024-04-23T16:33:38.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:529fe528:::benchmark_data_smithi188_113957_object67333:head# 2024-04-23T16:33:38.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a1f634:::benchmark_data_smithi188_113957_object110302:head# 2024-04-23T16:33:38.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a1fefa:::benchmark_data_smithi188_113957_object100190:head# 2024-04-23T16:33:38.375 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a2c2ac:::benchmark_data_smithi188_113957_object95284:head# 2024-04-23T16:33:38.375 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a42faa:::benchmark_data_smithi188_113957_object148865:head# 2024-04-23T16:33:38.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a4e047:::benchmark_data_smithi188_113957_object11199:head# 2024-04-23T16:33:38.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a507b3:::benchmark_data_smithi188_113957_object133731:head# 2024-04-23T16:33:38.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a53b45:::benchmark_data_smithi188_113957_object69857:head# 2024-04-23T16:33:38.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a5ebc2:::benchmark_data_smithi188_113957_object55168:head# 2024-04-23T16:33:38.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a6128e:::benchmark_data_smithi188_113957_object41096:head# 2024-04-23T16:33:38.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a64f7c:::benchmark_data_smithi188_113957_object41064:head# 2024-04-23T16:33:38.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a6d4af:::benchmark_data_smithi188_113957_object840:head# 2024-04-23T16:33:38.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a6d653:::benchmark_data_smithi188_113957_object66683:head# 2024-04-23T16:33:38.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a6fe53:::benchmark_data_smithi188_113957_object69355:head# 2024-04-23T16:33:38.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a77d41:::benchmark_data_smithi188_113957_object123320:head# 2024-04-23T16:33:38.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a7c7a6:::benchmark_data_smithi188_113957_object38497:head# 2024-04-23T16:33:38.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a8609b:::benchmark_data_smithi188_113957_object60980:head# 2024-04-23T16:33:38.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a8aefa:::benchmark_data_smithi188_113957_object126407:head# 2024-04-23T16:33:38.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a8db3a:::benchmark_data_smithi188_113957_object29422:head# 2024-04-23T16:33:38.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a8f947:::benchmark_data_smithi188_113957_object144460:head# 2024-04-23T16:33:38.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a9165f:::benchmark_data_smithi188_113957_object105176:head# 2024-04-23T16:33:38.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a918a9:::benchmark_data_smithi188_113957_object49610:head# 2024-04-23T16:33:38.387 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52a955ef:::benchmark_data_smithi188_113957_object126437:head# 2024-04-23T16:33:38.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52aa12fe:::benchmark_data_smithi188_113957_object61378:head# 2024-04-23T16:33:38.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52aa94d9:::benchmark_data_smithi188_113957_object40522:head# 2024-04-23T16:33:38.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52aa9680:::benchmark_data_smithi188_113957_object80548:head# 2024-04-23T16:33:38.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52aaeb62:::benchmark_data_smithi188_113957_object112201:head# 2024-04-23T16:33:38.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ab1b2b:::benchmark_data_smithi188_113957_object30573:head# 2024-04-23T16:33:38.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ab550a:::benchmark_data_smithi188_113957_object86203:head# 2024-04-23T16:33:38.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ab804d:::benchmark_data_smithi188_113957_object36355:head# 2024-04-23T16:33:38.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ab9e47:::benchmark_data_smithi188_113957_object19456:head# 2024-04-23T16:33:38.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ac3d35:::benchmark_data_smithi188_113957_object15472:head# 2024-04-23T16:33:38.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52aca522:::benchmark_data_smithi188_113957_object52068:head# 2024-04-23T16:33:38.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ad2839:::benchmark_data_smithi188_113957_object101244:head# 2024-04-23T16:33:38.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52adb05b:::benchmark_data_smithi188_113957_object53603:head# 2024-04-23T16:33:38.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52adbd1a:::benchmark_data_smithi188_113957_object141199:head# 2024-04-23T16:33:38.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52aec290:::benchmark_data_smithi188_113957_object103191:head# 2024-04-23T16:33:38.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52af48a0:::benchmark_data_smithi188_113957_object13722:head# 2024-04-23T16:33:38.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52af83e9:::benchmark_data_smithi188_113957_object58549:head# 2024-04-23T16:33:38.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52af934d:::benchmark_data_smithi188_113957_object81353:head# 2024-04-23T16:33:38.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52af9b6a:::benchmark_data_smithi188_113957_object44759:head# 2024-04-23T16:33:38.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52b0adb6:::benchmark_data_smithi188_113957_object110721:head# 2024-04-23T16:33:38.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52b15ef3:::benchmark_data_smithi188_113957_object104617:head# 2024-04-23T16:33:38.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52b20cd1:::benchmark_data_smithi188_113957_object18570:head# 2024-04-23T16:33:38.395 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52b21b76:::benchmark_data_smithi188_113957_object76933:head# 2024-04-23T16:33:38.395 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52b3247c:::benchmark_data_smithi188_113957_object47936:head# 2024-04-23T16:33:38.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52b349f0:::benchmark_data_smithi188_113957_object11423:head# 2024-04-23T16:33:38.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52b3e770:::benchmark_data_smithi188_113957_object16327:head# 2024-04-23T16:33:38.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52b43c59:::benchmark_data_smithi188_113957_object16413:head# 2024-04-23T16:33:38.397 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52b4952e:::benchmark_data_smithi188_113957_object78226:head# 2024-04-23T16:33:38.397 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52b54872:::benchmark_data_smithi188_113957_object104930:head# 2024-04-23T16:33:38.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52b5b209:::benchmark_data_smithi188_113957_object122439:head# 2024-04-23T16:33:38.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52b69ee4:::benchmark_data_smithi188_113957_object28766:head# 2024-04-23T16:33:38.399 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52b6c603:::benchmark_data_smithi188_113957_object151943:head# 2024-04-23T16:33:38.399 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52b755c4:::benchmark_data_smithi188_113957_object91703:head# 2024-04-23T16:33:38.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52b8548d:::benchmark_data_smithi188_113957_object74042:head# 2024-04-23T16:33:38.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52b8ecf9:::benchmark_data_smithi188_113957_object30278:head# 2024-04-23T16:33:38.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52b93a97:::benchmark_data_smithi188_113957_object110486:head# 2024-04-23T16:33:38.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52b96c6d:::benchmark_data_smithi188_113957_object100911:head# 2024-04-23T16:33:38.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ba5bd9:::benchmark_data_smithi188_113957_object104542:head# 2024-04-23T16:33:38.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ba7088:::benchmark_data_smithi188_113957_object44687:head# 2024-04-23T16:33:38.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ba7539:::benchmark_data_smithi188_113957_object97063:head# 2024-04-23T16:33:38.403 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ba9d75:::benchmark_data_smithi188_113957_object40236:head# 2024-04-23T16:33:38.403 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52bb6210:::benchmark_data_smithi188_113957_object9996:head# 2024-04-23T16:33:38.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52bb708b:::benchmark_data_smithi188_113957_object71726:head# 2024-04-23T16:33:38.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52bb8063:::benchmark_data_smithi188_113957_object99037:head# 2024-04-23T16:33:38.405 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52bb9b6b:::benchmark_data_smithi188_113957_object99930:head# 2024-04-23T16:33:38.405 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52bcc79c:::benchmark_data_smithi188_113957_object30796:head# 2024-04-23T16:33:38.405 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52bd0ed7:::benchmark_data_smithi188_113957_object1431:head# 2024-04-23T16:33:38.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52bd1e82:::benchmark_data_smithi188_113957_object23707:head# 2024-04-23T16:33:38.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52bd2edf:::benchmark_data_smithi188_113957_object46735:head# 2024-04-23T16:33:38.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52bd5f57:::benchmark_data_smithi188_113957_object30792:head# 2024-04-23T16:33:38.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52be350e:::benchmark_data_smithi188_113957_object108041:head# 2024-04-23T16:33:38.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52bec8c3:::benchmark_data_smithi188_113957_object66553:head# 2024-04-23T16:33:38.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52bf8f6f:::benchmark_data_smithi188_113957_object12726:head# 2024-04-23T16:33:38.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52bfb21f:::benchmark_data_smithi188_113957_object129200:head# 2024-04-23T16:33:38.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52bfed64:::benchmark_data_smithi188_113957_object91290:head# 2024-04-23T16:33:38.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52c0d76c:::benchmark_data_smithi188_113957_object139692:head# 2024-04-23T16:33:38.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52c155da:::benchmark_data_smithi188_113957_object3527:head# 2024-04-23T16:33:38.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52c16dec:::benchmark_data_smithi188_113957_object37947:head# 2024-04-23T16:33:38.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52c1842c:::benchmark_data_smithi188_113957_object3572:head# 2024-04-23T16:33:38.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52c3754a:::benchmark_data_smithi188_113957_object50088:head# 2024-04-23T16:33:38.412 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52c39311:::benchmark_data_smithi188_113957_object127471:head# 2024-04-23T16:33:38.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52c408ec:::benchmark_data_smithi188_113957_object107895:head# 2024-04-23T16:33:38.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52c42089:::benchmark_data_smithi188_113957_object36435:head# 2024-04-23T16:33:38.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52c51581:::benchmark_data_smithi188_113957_object24503:head# 2024-04-23T16:33:38.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52c555cb:::benchmark_data_smithi188_113957_object31644:head# 2024-04-23T16:33:38.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52c70cc4:::benchmark_data_smithi188_113957_object133882:head# 2024-04-23T16:33:38.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52c755e4:::benchmark_data_smithi188_113957_object67471:head# 2024-04-23T16:33:38.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52c7b265:::benchmark_data_smithi188_113957_object96998:head# 2024-04-23T16:33:38.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52c88ee2:::benchmark_data_smithi188_113957_object94313:head# 2024-04-23T16:33:38.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52c8c514:::benchmark_data_smithi188_113957_object31754:head# 2024-04-23T16:33:38.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52c9c58b:::benchmark_data_smithi188_113957_object8377:head# 2024-04-23T16:33:38.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52c9d038:::benchmark_data_smithi188_113957_object64872:head# 2024-04-23T16:33:38.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ca23b0:::benchmark_data_smithi188_113957_object5951:head# 2024-04-23T16:33:38.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52cacd49:::benchmark_data_smithi188_113957_object145724:head# 2024-04-23T16:33:38.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52caeaf7:::benchmark_data_smithi188_113957_object52965:head# 2024-04-23T16:33:38.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52cb0d26:::benchmark_data_smithi188_113957_object79146:head# 2024-04-23T16:33:38.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52cb5768:::benchmark_data_smithi188_113957_object48386:head# 2024-04-23T16:33:38.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52cb5a4f:::benchmark_data_smithi188_113957_object75995:head# 2024-04-23T16:33:38.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52cbc071:::benchmark_data_smithi188_113957_object38175:head# 2024-04-23T16:33:38.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52cbd93e:::benchmark_data_smithi188_113957_object13920:head# 2024-04-23T16:33:38.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52cc2ab0:::benchmark_data_smithi188_113957_object85952:head# 2024-04-23T16:33:38.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52cc2ef9:::benchmark_data_smithi188_113957_object87346:head# 2024-04-23T16:33:38.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ccb75f:::benchmark_data_smithi188_113957_object28309:head# 2024-04-23T16:33:38.426 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ccbebd:::benchmark_data_smithi188_113957_object40124:head# 2024-04-23T16:33:38.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52cce1af:::benchmark_data_smithi188_113957_object116780:head# 2024-04-23T16:33:38.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ccf8ea:::benchmark_data_smithi188_113957_object105432:head# 2024-04-23T16:33:38.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52cdd152:::benchmark_data_smithi188_113957_object19524:head# 2024-04-23T16:33:38.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52cddafa:::benchmark_data_smithi188_113957_object89343:head# 2024-04-23T16:33:38.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ce5099:::benchmark_data_smithi188_113957_object125406:head# 2024-04-23T16:33:38.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52cf7e13:::benchmark_data_smithi188_113957_object81002:head# 2024-04-23T16:33:38.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52cf9c24:::benchmark_data_smithi188_113957_object113583:head# 2024-04-23T16:33:38.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52cfd76d:::benchmark_data_smithi188_113957_object142709:head# 2024-04-23T16:33:38.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52cfd9ce:::benchmark_data_smithi188_113957_object100960:head# 2024-04-23T16:33:38.434 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d0c40d:::benchmark_data_smithi188_113957_object49046:head# 2024-04-23T16:33:38.434 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d19a07:::benchmark_data_smithi188_113957_object63520:head# 2024-04-23T16:33:38.435 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d1b902:::benchmark_data_smithi188_113957_object12557:head# 2024-04-23T16:33:38.435 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d1c0b8:::benchmark_data_smithi188_113957_object90670:head# 2024-04-23T16:33:38.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d21567:::benchmark_data_smithi188_113957_object130211:head# 2024-04-23T16:33:38.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d233c3:::benchmark_data_smithi188_113957_object106022:head# 2024-04-23T16:33:38.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d292ca:::benchmark_data_smithi188_113957_object80036:head# 2024-04-23T16:33:38.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d2a735:::benchmark_data_smithi188_113957_object101364:head# 2024-04-23T16:33:38.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d2b518:::benchmark_data_smithi188_113957_object69863:head# 2024-04-23T16:33:38.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d2ba4f:::benchmark_data_smithi188_113957_object106042:head# 2024-04-23T16:33:38.438 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d31fa5:::benchmark_data_smithi188_113957_object104877:head# 2024-04-23T16:33:38.438 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d37fbb:::benchmark_data_smithi188_113957_object126752:head# 2024-04-23T16:33:38.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d56534:::benchmark_data_smithi188_113957_object136488:head# 2024-04-23T16:33:38.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d59a47:::benchmark_data_smithi188_113957_object28840:head# 2024-04-23T16:33:38.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d5a2b4:::benchmark_data_smithi188_113957_object1608:head# 2024-04-23T16:33:38.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d65f98:::benchmark_data_smithi188_113957_object68319:head# 2024-04-23T16:33:38.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d6607f:::benchmark_data_smithi188_113957_object75237:head# 2024-04-23T16:33:38.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d6beb6:::benchmark_data_smithi188_113957_object110992:head# 2024-04-23T16:33:38.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d71fd5:::benchmark_data_smithi188_113957_object144180:head# 2024-04-23T16:33:38.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d7c161:::benchmark_data_smithi188_113957_object104109:head# 2024-04-23T16:33:38.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d8b0ea:::benchmark_data_smithi188_113957_object148202:head# 2024-04-23T16:33:38.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d8d8d9:::benchmark_data_smithi188_113957_object145074:head# 2024-04-23T16:33:38.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d908f5:::benchmark_data_smithi188_113957_object91227:head# 2024-04-23T16:33:38.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d953f7:::benchmark_data_smithi188_113957_object43944:head# 2024-04-23T16:33:38.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52d9888e:::benchmark_data_smithi188_113957_object81609:head# 2024-04-23T16:33:38.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52da294a:::benchmark_data_smithi188_113957_object121984:head# 2024-04-23T16:33:38.446 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52da54d3:::benchmark_data_smithi188_113957_object84924:head# 2024-04-23T16:33:38.446 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52dadee7:::benchmark_data_smithi188_113957_object52884:head# 2024-04-23T16:33:38.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52dae5c5:::benchmark_data_smithi188_113957_object99020:head# 2024-04-23T16:33:38.447 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:33:38.448 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52db050e:::benchmark_data_smithi188_113957_object37446:head# 2024-04-23T16:33:38.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52db0a4d:::benchmark_data_smithi188_113957_object36039:head# 2024-04-23T16:33:38.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52db999c:::benchmark_data_smithi188_113957_object122706:head# 2024-04-23T16:33:38.451 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52dc2c4d:::benchmark_data_smithi188_113957_object111195:head# 2024-04-23T16:33:38.451 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52dcd3d8:::benchmark_data_smithi188_113957_object14528:head# 2024-04-23T16:33:38.452 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52dd41d6:::benchmark_data_smithi188_113957_object75327:head# 2024-04-23T16:33:38.452 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52dd6d70:::benchmark_data_smithi188_113957_object127098:head# 2024-04-23T16:33:38.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ddfaab:::benchmark_data_smithi188_113957_object79443:head# 2024-04-23T16:33:38.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52df801b:::benchmark_data_smithi188_113957_object131117:head# 2024-04-23T16:33:38.454 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52df9c11:::benchmark_data_smithi188_113957_object80653:head# 2024-04-23T16:33:38.454 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e037d5:::benchmark_data_smithi188_113957_object83686:head# 2024-04-23T16:33:38.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e06411:::benchmark_data_smithi188_113957_object20350:head# 2024-04-23T16:33:38.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e101a5:::benchmark_data_smithi188_113957_object100023:head# 2024-04-23T16:33:38.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e1baef:::benchmark_data_smithi188_113957_object24885:head# 2024-04-23T16:33:38.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e1d232:::benchmark_data_smithi188_113957_object35325:head# 2024-04-23T16:33:38.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e288ea:::benchmark_data_smithi188_113957_object132658:head# 2024-04-23T16:33:38.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e28c67:::benchmark_data_smithi188_113957_object13772:head# 2024-04-23T16:33:38.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e2e58c:::benchmark_data_smithi188_113957_object23262:head# 2024-04-23T16:33:38.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e31785:::benchmark_data_smithi188_113957_object104960:head# 2024-04-23T16:33:38.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e3bcad:::benchmark_data_smithi188_113957_object74394:head# 2024-04-23T16:33:38.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e500e6:::benchmark_data_smithi188_113957_object77864:head# 2024-04-23T16:33:38.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e51fcb:::benchmark_data_smithi188_113957_object51483:head# 2024-04-23T16:33:38.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e52aa8:::benchmark_data_smithi188_113957_object11726:head# 2024-04-23T16:33:38.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e5a797:::benchmark_data_smithi188_113957_object87475:head# 2024-04-23T16:33:38.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e5b68d:::benchmark_data_smithi188_113957_object142828:head# 2024-04-23T16:33:38.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e5e9dd:::benchmark_data_smithi188_113957_object152706:head# 2024-04-23T16:33:38.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e63f6b:::benchmark_data_smithi188_113957_object73155:head# 2024-04-23T16:33:38.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e66b66:::benchmark_data_smithi188_113957_object147660:head# 2024-04-23T16:33:38.473 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:33:38.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e691dc:::benchmark_data_smithi188_113957_object149136:head# 2024-04-23T16:33:38.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e75e51:::benchmark_data_smithi188_113957_object142663:head# 2024-04-23T16:33:38.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e78613:::benchmark_data_smithi188_113957_object130149:head# 2024-04-23T16:33:38.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e8028b:::benchmark_data_smithi188_113957_object107289:head# 2024-04-23T16:33:38.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e9916e:::benchmark_data_smithi188_113957_object23075:head# 2024-04-23T16:33:38.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52e9fce2:::benchmark_data_smithi188_113957_object15643:head# 2024-04-23T16:33:38.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ea0fd2:::benchmark_data_smithi188_113957_object96945:head# 2024-04-23T16:33:38.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ea1327:::benchmark_data_smithi188_113957_object105731:head# 2024-04-23T16:33:38.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52eb296b:::benchmark_data_smithi188_113957_object65419:head# 2024-04-23T16:33:38.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52eb8d67:::benchmark_data_smithi188_113957_object48722:head# 2024-04-23T16:33:38.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ebab59:::benchmark_data_smithi188_113957_object59988:head# 2024-04-23T16:33:38.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ebab91:::benchmark_data_smithi188_113957_object43479:head# 2024-04-23T16:33:38.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ec9f3d:::benchmark_data_smithi188_113957_object5846:head# 2024-04-23T16:33:38.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ecb999:::benchmark_data_smithi188_113957_object143303:head# 2024-04-23T16:33:38.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ecbb51:::benchmark_data_smithi188_113957_object123454:head# 2024-04-23T16:33:38.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ed0b26:::benchmark_data_smithi188_113957_object62479:head# 2024-04-23T16:33:38.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ed2040:::benchmark_data_smithi188_113957_object77087:head# 2024-04-23T16:33:38.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ed69f3:::benchmark_data_smithi188_113957_object143751:head# 2024-04-23T16:33:38.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52edffda:::benchmark_data_smithi188_113957_object51705:head# 2024-04-23T16:33:38.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ee391e:::benchmark_data_smithi188_113957_object74805:head# 2024-04-23T16:33:38.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ee5950:::benchmark_data_smithi188_113957_object105789:head# 2024-04-23T16:33:38.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52eea819:::benchmark_data_smithi188_113957_object89823:head# 2024-04-23T16:33:38.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52eed09b:::benchmark_data_smithi188_113957_object61895:head# 2024-04-23T16:33:38.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52eee006:::benchmark_data_smithi188_113957_object133742:head# 2024-04-23T16:33:38.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ef0943:::benchmark_data_smithi188_113957_object84785:head# 2024-04-23T16:33:38.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ef4782:::benchmark_data_smithi188_113957_object123336:head# 2024-04-23T16:33:38.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f02429:::benchmark_data_smithi188_113957_object141318:head# 2024-04-23T16:33:38.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f047c2:::benchmark_data_smithi188_113957_object8558:head# 2024-04-23T16:33:38.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f118af:::benchmark_data_smithi188_113957_object96696:head# 2024-04-23T16:33:38.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f16bf5:::benchmark_data_smithi188_113957_object32097:head# 2024-04-23T16:33:38.486 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f1edfe:::benchmark_data_smithi188_113957_object99217:head# 2024-04-23T16:33:38.486 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f2ce97:::benchmark_data_smithi188_113957_object80318:head# 2024-04-23T16:33:38.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f35e86:::benchmark_data_smithi188_113957_object15500:head# 2024-04-23T16:33:38.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f39602:::benchmark_data_smithi188_113957_object38877:head# 2024-04-23T16:33:38.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f3a5b1:::benchmark_data_smithi188_113957_object97066:head# 2024-04-23T16:33:38.488 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f3f6f8:::benchmark_data_smithi188_113957_object29788:head# 2024-04-23T16:33:38.488 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f43bb1:::benchmark_data_smithi188_113957_object98080:head# 2024-04-23T16:33:38.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f4592f:::benchmark_data_smithi188_113957_object20997:head# 2024-04-23T16:33:38.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f4596e:::benchmark_data_smithi188_113957_object151753:head# 2024-04-23T16:33:38.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f4aa5c:::benchmark_data_smithi188_113957_object26457:head# 2024-04-23T16:33:38.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f4ff40:::benchmark_data_smithi188_113957_object116901:head# 2024-04-23T16:33:38.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f518e6:::benchmark_data_smithi188_113957_object69136:head# 2024-04-23T16:33:38.491 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f537f5:::benchmark_data_smithi188_113957_object118542:head# 2024-04-23T16:33:38.491 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f54f14:::benchmark_data_smithi188_113957_object133162:head# 2024-04-23T16:33:38.492 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f562dc:::benchmark_data_smithi188_113957_object3086:head# 2024-04-23T16:33:38.492 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f58be4:::benchmark_data_smithi188_113957_object63010:head# 2024-04-23T16:33:38.492 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f68514:::benchmark_data_smithi188_113957_object120158:head# 2024-04-23T16:33:38.493 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f6d812:::benchmark_data_smithi188_113957_object48314:head# 2024-04-23T16:33:38.493 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f75597:::benchmark_data_smithi188_113957_object63505:head# 2024-04-23T16:33:38.494 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f771d9:::benchmark_data_smithi188_113957_object29857:head# 2024-04-23T16:33:38.494 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f7a26d:::benchmark_data_smithi188_113957_object116847:head# 2024-04-23T16:33:38.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f823b2:::benchmark_data_smithi188_113957_object117233:head# 2024-04-23T16:33:38.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52f859df:::benchmark_data_smithi188_113957_object57471:head# 2024-04-23T16:33:38.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52fa5d73:::benchmark_data_smithi188_113957_object52185:head# 2024-04-23T16:33:38.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52fad502:::benchmark_data_smithi188_113957_object101673:head# 2024-04-23T16:33:38.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52fb2f1e:::benchmark_data_smithi188_113957_object39943:head# 2024-04-23T16:33:38.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52fb88af:::benchmark_data_smithi188_113957_object74820:head# 2024-04-23T16:33:38.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52fbfa3b:::benchmark_data_smithi188_113957_object67982:head# 2024-04-23T16:33:38.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52fc7ccb:::benchmark_data_smithi188_113957_object129920:head# 2024-04-23T16:33:38.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52fcd34e:::benchmark_data_smithi188_113957_object6904:head# 2024-04-23T16:33:38.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52fd14f5:::benchmark_data_smithi188_113957_object131435:head# 2024-04-23T16:33:38.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52fd5296:::benchmark_data_smithi188_113957_object15701:head# 2024-04-23T16:33:38.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ff8d77:::benchmark_data_smithi188_113957_object17212:head# 2024-04-23T16:33:38.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ff9219:::benchmark_data_smithi188_113957_object142940:head# 2024-04-23T16:33:38.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ffd641:::benchmark_data_smithi188_113957_object135476:head# 2024-04-23T16:33:38.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:52ffec7f:::benchmark_data_smithi188_113957_object1342:head# 2024-04-23T16:33:38.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5300557f:::benchmark_data_smithi188_113957_object58445:head# 2024-04-23T16:33:38.502 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:38.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53010327:::benchmark_data_smithi188_113957_object52085:head# 2024-04-23T16:33:38.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:530148d2:::benchmark_data_smithi188_113957_object21896:head# 2024-04-23T16:33:38.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5301a77b:::benchmark_data_smithi188_113957_object104024:head# 2024-04-23T16:33:38.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:530408c4:::benchmark_data_smithi188_113957_object80834:head# 2024-04-23T16:33:38.504 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53046e2d:::benchmark_data_smithi188_113957_object148445:head# 2024-04-23T16:33:38.504 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5304ce7f:::benchmark_data_smithi188_113957_object26601:head# 2024-04-23T16:33:38.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5305c70c:::benchmark_data_smithi188_113957_object127847:head# 2024-04-23T16:33:38.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5305cbf8:::benchmark_data_smithi188_113957_object147977:head# 2024-04-23T16:33:38.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5305df11:::benchmark_data_smithi188_113957_object67751:head# 2024-04-23T16:33:38.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53069f53:::benchmark_data_smithi188_113957_object35275:head# 2024-04-23T16:33:38.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5307f825:::benchmark_data_smithi188_113957_object60045:head# 2024-04-23T16:33:38.507 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53085aa3:::benchmark_data_smithi188_113957_object111497:head# 2024-04-23T16:33:38.507 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:530888f0:::benchmark_data_smithi188_113957_object117393:head# 2024-04-23T16:33:38.508 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5308a9dc:::benchmark_data_smithi188_113957_object137751:head# 2024-04-23T16:33:38.508 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:530967fc:::benchmark_data_smithi188_113957_object116911:head# 2024-04-23T16:33:38.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5309bc4e:::benchmark_data_smithi188_113957_object14112:head# 2024-04-23T16:33:38.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:530a7000:::benchmark_data_smithi188_113957_object131950:head# 2024-04-23T16:33:38.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:530a91c6:::benchmark_data_smithi188_113957_object144929:head# 2024-04-23T16:33:38.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:530c3814:::benchmark_data_smithi188_113957_object58093:head# 2024-04-23T16:33:38.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:530c6be8:::benchmark_data_smithi188_113957_object17495:head# 2024-04-23T16:33:38.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:530c738f:::benchmark_data_smithi188_113957_object30281:head# 2024-04-23T16:33:38.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:530cbddc:::benchmark_data_smithi188_113957_object27181:head# 2024-04-23T16:33:38.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:530d3144:::benchmark_data_smithi188_113957_object95512:head# 2024-04-23T16:33:38.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:530d410e:::benchmark_data_smithi188_113957_object65060:head# 2024-04-23T16:33:38.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:530d4563:::benchmark_data_smithi188_113957_object73711:head# 2024-04-23T16:33:38.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:530db7f8:::benchmark_data_smithi188_113957_object30343:head# 2024-04-23T16:33:38.516 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:530dc5c5:::benchmark_data_smithi188_113957_object149294:head# 2024-04-23T16:33:38.516 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:530eeb89:::benchmark_data_smithi188_113957_object87311:head# 2024-04-23T16:33:38.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:530eff52:::benchmark_data_smithi188_113957_object95976:head# 2024-04-23T16:33:38.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:530f550a:::benchmark_data_smithi188_113957_object78360:head# 2024-04-23T16:33:38.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:530fec3e:::benchmark_data_smithi188_113957_object96362:head# 2024-04-23T16:33:38.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53101b39:::benchmark_data_smithi188_113957_object62197:head# 2024-04-23T16:33:38.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531165ba:::benchmark_data_smithi188_113957_object64435:head# 2024-04-23T16:33:38.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531186ad:::benchmark_data_smithi188_113957_object47334:head# 2024-04-23T16:33:38.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5311b9aa:::benchmark_data_smithi188_113957_object151363:head# 2024-04-23T16:33:38.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5312534c:::benchmark_data_smithi188_113957_object37816:head# 2024-04-23T16:33:38.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53128775:::benchmark_data_smithi188_113957_object63895:head# 2024-04-23T16:33:38.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53137af1:::benchmark_data_smithi188_113957_object19350:head# 2024-04-23T16:33:38.521 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53140ec9:::benchmark_data_smithi188_113957_object34476:head# 2024-04-23T16:33:38.521 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5314251b:::benchmark_data_smithi188_113957_object97631:head# 2024-04-23T16:33:38.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53142687:::benchmark_data_smithi188_113957_object5444:head# 2024-04-23T16:33:38.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53146f76:::benchmark_data_smithi188_113957_object106967:head# 2024-04-23T16:33:38.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53151e75:::benchmark_data_smithi188_113957_object71453:head# 2024-04-23T16:33:38.523 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53168cf8:::benchmark_data_smithi188_113957_object68288:head# 2024-04-23T16:33:38.523 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53177069:::benchmark_data_smithi188_113957_object21670:head# 2024-04-23T16:33:38.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53177cc7:::benchmark_data_smithi188_113957_object84139:head# 2024-04-23T16:33:38.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5317f7d2:::benchmark_data_smithi188_113957_object92673:head# 2024-04-23T16:33:38.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53188728:::benchmark_data_smithi188_113957_object8828:head# 2024-04-23T16:33:38.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5319f969:::benchmark_data_smithi188_113957_object107324:head# 2024-04-23T16:33:38.526 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:33:38.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531b4a03:::benchmark_data_smithi188_113957_object126616:head# 2024-04-23T16:33:38.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531b5c8f:::benchmark_data_smithi188_113957_object99576:head# 2024-04-23T16:33:38.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531ba70b:::benchmark_data_smithi188_113957_object97351:head# 2024-04-23T16:33:38.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531bc99f:::benchmark_data_smithi188_113957_object126937:head# 2024-04-23T16:33:38.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531bf7a5:::benchmark_data_smithi188_113957_object133017:head# 2024-04-23T16:33:38.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531c5686:::benchmark_data_smithi188_113957_object11574:head# 2024-04-23T16:33:38.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531cb76e:::benchmark_data_smithi188_113957_object151277:head# 2024-04-23T16:33:38.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:38.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:33:38.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531cecc8:::benchmark_data_smithi188_113957_object77717:head# 2024-04-23T16:33:38.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531d1a2f:::benchmark_data_smithi188_113957_object122637:head# 2024-04-23T16:33:38.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531d28a4:::benchmark_data_smithi188_113957_object34404:head# 2024-04-23T16:33:38.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531d6a63:::benchmark_data_smithi188_113957_object25307:head# 2024-04-23T16:33:38.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531dcbdd:::benchmark_data_smithi188_113957_object12288:head# 2024-04-23T16:33:38.531 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531dd265:::benchmark_data_smithi188_113957_object107124:head# 2024-04-23T16:33:38.531 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531e06f3:::benchmark_data_smithi188_113957_object81593:head# 2024-04-23T16:33:38.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531e69f2:::benchmark_data_smithi188_113957_object134582:head# 2024-04-23T16:33:38.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531f5da4:::benchmark_data_smithi188_113957_object150651:head# 2024-04-23T16:33:38.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531fa0a2:::benchmark_data_smithi188_113957_object113682:head# 2024-04-23T16:33:38.533 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531fa318:::benchmark_data_smithi188_113957_object113714:head# 2024-04-23T16:33:38.533 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531fea6c:::benchmark_data_smithi188_113957_object116965:head# 2024-04-23T16:33:38.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:531fee3b:::benchmark_data_smithi188_113957_object127908:head# 2024-04-23T16:33:38.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5320309c:::benchmark_data_smithi188_113957_object77452:head# 2024-04-23T16:33:38.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53204044:::benchmark_data_smithi188_113957_object137266:head# 2024-04-23T16:33:38.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53215846:::benchmark_data_smithi188_113957_object52911:head# 2024-04-23T16:33:38.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53219e7e:::benchmark_data_smithi188_113957_object104383:head# 2024-04-23T16:33:38.536 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53231a97:::benchmark_data_smithi188_113957_object138129:head# 2024-04-23T16:33:38.536 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53231c71:::benchmark_data_smithi188_113957_object108950:head# 2024-04-23T16:33:38.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53239e85:::benchmark_data_smithi188_113957_object111507:head# 2024-04-23T16:33:38.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5323ed18:::benchmark_data_smithi188_113957_object33665:head# 2024-04-23T16:33:38.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5323f05c:::benchmark_data_smithi188_113957_object53058:head# 2024-04-23T16:33:38.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:532419e3:::benchmark_data_smithi188_113957_object145927:head# 2024-04-23T16:33:38.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53243c58:::benchmark_data_smithi188_113957_object91168:head# 2024-04-23T16:33:38.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5324d4f4:::benchmark_data_smithi188_113957_object31024:head# 2024-04-23T16:33:38.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53250710:::benchmark_data_smithi188_113957_object151087:head# 2024-04-23T16:33:38.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5326621b:::benchmark_data_smithi188_113957_object150207:head# 2024-04-23T16:33:38.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53266366:::benchmark_data_smithi188_113957_object63891:head# 2024-04-23T16:33:38.541 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5326fade:::benchmark_data_smithi188_113957_object33731:head# 2024-04-23T16:33:38.541 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:532746ad:::benchmark_data_smithi188_113957_object121790:head# 2024-04-23T16:33:38.541 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53274e52:::benchmark_data_smithi188_113957_object63222:head# 2024-04-23T16:33:38.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53288130:::benchmark_data_smithi188_113957_object136776:head# 2024-04-23T16:33:38.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5328b599:::benchmark_data_smithi188_113957_object70235:head# 2024-04-23T16:33:38.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53290b25:::benchmark_data_smithi188_113957_object153180:head# 2024-04-23T16:33:38.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5329209a:::benchmark_data_smithi188_113957_object131219:head# 2024-04-23T16:33:38.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5329c1f5:::benchmark_data_smithi188_113957_object30196:head# 2024-04-23T16:33:38.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:532ac422:::benchmark_data_smithi188_113957_object25059:head# 2024-04-23T16:33:38.547 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:532bd787:::benchmark_data_smithi188_113957_object72437:head# 2024-04-23T16:33:38.547 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:532c7bea:::benchmark_data_smithi188_113957_object51471:head# 2024-04-23T16:33:38.547 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:532c7f18:::benchmark_data_smithi188_113957_object144909:head# 2024-04-23T16:33:38.548 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:532df6bf:::benchmark_data_smithi188_113957_object33424:head# 2024-04-23T16:33:38.548 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:532e4764:::benchmark_data_smithi188_113957_object1160:head# 2024-04-23T16:33:38.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:532ee4eb:::benchmark_data_smithi188_113957_object74154:head# 2024-04-23T16:33:38.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:532f5c71:::benchmark_data_smithi188_113957_object46029:head# 2024-04-23T16:33:38.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5330ff14:::benchmark_data_smithi188_113957_object150776:head# 2024-04-23T16:33:38.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53315ae2:::benchmark_data_smithi188_113957_object104223:head# 2024-04-23T16:33:38.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:533184c6:::benchmark_data_smithi188_113957_object32963:head# 2024-04-23T16:33:38.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5331cffa:::benchmark_data_smithi188_113957_object123625:head# 2024-04-23T16:33:38.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5331d368:::benchmark_data_smithi188_113957_object30207:head# 2024-04-23T16:33:38.552 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53326c44:::benchmark_data_smithi188_113957_object86475:head# 2024-04-23T16:33:38.553 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5332d3c9:::benchmark_data_smithi188_113957_object100580:head# 2024-04-23T16:33:38.553 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53333e6a:::benchmark_data_smithi188_113957_object131596:head# 2024-04-23T16:33:38.553 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5333e7e3:::benchmark_data_smithi188_113957_object132516:head# 2024-04-23T16:33:38.554 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:533457b1:::benchmark_data_smithi188_113957_object67559:head# 2024-04-23T16:33:38.554 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5335c62a:::benchmark_data_smithi188_113957_object122385:head# 2024-04-23T16:33:38.555 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53363447:::benchmark_data_smithi188_113957_object101007:head# 2024-04-23T16:33:38.555 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53367d1f:::benchmark_data_smithi188_113957_object97353:head# 2024-04-23T16:33:38.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53369074:::benchmark_data_smithi188_113957_object81658:head# 2024-04-23T16:33:38.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5336a7b4:::benchmark_data_smithi188_113957_object94933:head# 2024-04-23T16:33:38.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5336a893:::benchmark_data_smithi188_113957_object28858:head# 2024-04-23T16:33:38.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5336e8df:::benchmark_data_smithi188_113957_object40526:head# 2024-04-23T16:33:38.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5337f5ec:::benchmark_data_smithi188_113957_object36034:head# 2024-04-23T16:33:38.558 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53396bec:::benchmark_data_smithi188_113957_object125239:head# 2024-04-23T16:33:38.558 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5339e7e4:::benchmark_data_smithi188_113957_object148446:head# 2024-04-23T16:33:38.558 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:533a1655:::benchmark_data_smithi188_113957_object80379:head# 2024-04-23T16:33:38.559 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:533ab6d8:::benchmark_data_smithi188_113957_object69609:head# 2024-04-23T16:33:38.559 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:533ae59e:::benchmark_data_smithi188_113957_object72509:head# 2024-04-23T16:33:38.560 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:533b0755:::benchmark_data_smithi188_113957_object40431:head# 2024-04-23T16:33:38.560 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:533c3daf:::benchmark_data_smithi188_113957_object30194:head# 2024-04-23T16:33:38.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:533c4b3b:::benchmark_data_smithi188_113957_object19715:head# 2024-04-23T16:33:38.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:533cf277:::benchmark_data_smithi188_113957_object90450:head# 2024-04-23T16:33:38.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:533db209:::benchmark_data_smithi188_113957_object114154:head# 2024-04-23T16:33:38.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:533e6070:::benchmark_data_smithi188_113957_object127089:head# 2024-04-23T16:33:38.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:533e71da:::benchmark_data_smithi188_113957_object127605:head# 2024-04-23T16:33:38.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:533e7bc1:::benchmark_data_smithi188_113957_object32172:head# 2024-04-23T16:33:38.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:533f4f4a:::benchmark_data_smithi188_113957_object34689:head# 2024-04-23T16:33:38.564 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:533f9711:::benchmark_data_smithi188_113957_object51763:head# 2024-04-23T16:33:38.564 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:533fa2cc:::benchmark_data_smithi188_113957_object31822:head# 2024-04-23T16:33:38.565 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:533fc189:::benchmark_data_smithi188_113957_object79142:head# 2024-04-23T16:33:38.566 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:533ffae8:::benchmark_data_smithi188_113957_object122912:head# 2024-04-23T16:33:38.566 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53406970:::benchmark_data_smithi188_113957_object114650:head# 2024-04-23T16:33:38.566 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5340ad0d:::benchmark_data_smithi188_113957_object70351:head# 2024-04-23T16:33:38.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5340c4e3:::benchmark_data_smithi188_113957_object97462:head# 2024-04-23T16:33:38.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5340fef6:::benchmark_data_smithi188_113957_object104975:head# 2024-04-23T16:33:38.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534109d6:::benchmark_data_smithi188_113957_object86471:head# 2024-04-23T16:33:38.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53411b52:::benchmark_data_smithi188_113957_object26746:head# 2024-04-23T16:33:38.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534125aa:::benchmark_data_smithi188_113957_object33727:head# 2024-04-23T16:33:38.569 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534179a6:::benchmark_data_smithi188_113957_object88677:head# 2024-04-23T16:33:38.570 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53423aaf:::benchmark_data_smithi188_113957_object109350:head# 2024-04-23T16:33:38.570 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534255e5:::benchmark_data_smithi188_113957_object36475:head# 2024-04-23T16:33:38.571 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53428433:::benchmark_data_smithi188_113957_object31404:head# 2024-04-23T16:33:38.571 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5342c3bf:::benchmark_data_smithi188_113957_object130380:head# 2024-04-23T16:33:38.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5342f059:::benchmark_data_smithi188_113957_object77895:head# 2024-04-23T16:33:38.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53432dd9:::benchmark_data_smithi188_113957_object52931:head# 2024-04-23T16:33:38.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53433a0e:::benchmark_data_smithi188_113957_object111031:head# 2024-04-23T16:33:38.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534468d6:::benchmark_data_smithi188_113957_object75323:head# 2024-04-23T16:33:38.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5344a8cb:::benchmark_data_smithi188_113957_object150456:head# 2024-04-23T16:33:38.574 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5344c916:::benchmark_data_smithi188_113957_object120810:head# 2024-04-23T16:33:38.574 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5345a39b:::benchmark_data_smithi188_113957_object115379:head# 2024-04-23T16:33:38.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534630e7:::benchmark_data_smithi188_113957_object129153:head# 2024-04-23T16:33:38.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53463c94:::benchmark_data_smithi188_113957_object99904:head# 2024-04-23T16:33:38.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53465285:::benchmark_data_smithi188_113957_object806:head# 2024-04-23T16:33:38.576 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5347261a:::benchmark_data_smithi188_113957_object67426:head# 2024-04-23T16:33:38.576 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53478ac5:::benchmark_data_smithi188_113957_object120999:head# 2024-04-23T16:33:38.577 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5347cc4d:::benchmark_data_smithi188_113957_object92093:head# 2024-04-23T16:33:38.577 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534809c0:::benchmark_data_smithi188_113957_object150884:head# 2024-04-23T16:33:38.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53481cd6:::benchmark_data_smithi188_113957_object110602:head# 2024-04-23T16:33:38.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534836e4:::benchmark_data_smithi188_113957_object104733:head# 2024-04-23T16:33:38.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5348c8c1:::benchmark_data_smithi188_113957_object82840:head# 2024-04-23T16:33:38.579 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5348c8eb:::benchmark_data_smithi188_113957_object36920:head# 2024-04-23T16:33:38.579 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53497dcd:::benchmark_data_smithi188_113957_object152520:head# 2024-04-23T16:33:38.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5349ce43:::benchmark_data_smithi188_113957_object99683:head# 2024-04-23T16:33:38.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5349e2a1:::benchmark_data_smithi188_113957_object4206:head# 2024-04-23T16:33:38.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534a8be8:::benchmark_data_smithi188_113957_object76218:head# 2024-04-23T16:33:38.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534a9f29:::benchmark_data_smithi188_113957_object103201:head# 2024-04-23T16:33:38.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534aa4e4:::benchmark_data_smithi188_113957_object15216:head# 2024-04-23T16:33:38.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534ad44e:::benchmark_data_smithi188_113957_object131566:head# 2024-04-23T16:33:38.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534b4e51:::benchmark_data_smithi188_113957_object111023:head# 2024-04-23T16:33:38.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534b9b8c:::benchmark_data_smithi188_113957_object126151:head# 2024-04-23T16:33:38.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534d8bae:::benchmark_data_smithi188_113957_object121360:head# 2024-04-23T16:33:38.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534d9380:::benchmark_data_smithi188_113957_object144204:head# 2024-04-23T16:33:38.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534d9be6:::benchmark_data_smithi188_113957_object104045:head# 2024-04-23T16:33:38.585 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534dfeb9:::benchmark_data_smithi188_113957_object38710:head# 2024-04-23T16:33:38.585 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534f7acf:::benchmark_data_smithi188_113957_object26817:head# 2024-04-23T16:33:38.585 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:534fd587:::benchmark_data_smithi188_113957_object134663:head# 2024-04-23T16:33:38.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5350747b:::benchmark_data_smithi188_113957_object108974:head# 2024-04-23T16:33:38.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5350d364:::benchmark_data_smithi188_113957_object78345:head# 2024-04-23T16:33:38.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53516bd4:::benchmark_data_smithi188_113957_object67127:head# 2024-04-23T16:33:38.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5351b8b3:::benchmark_data_smithi188_113957_object11562:head# 2024-04-23T16:33:38.588 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:535214f0:::benchmark_data_smithi188_113957_object72014:head# 2024-04-23T16:33:38.588 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:535215db:::benchmark_data_smithi188_113957_object144696:head# 2024-04-23T16:33:38.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53525eaf:::benchmark_data_smithi188_113957_object27584:head# 2024-04-23T16:33:38.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5352a037:::benchmark_data_smithi188_113957_object56629:head# 2024-04-23T16:33:38.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5352f779:::benchmark_data_smithi188_113957_object143500:head# 2024-04-23T16:33:38.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5353544c:::benchmark_data_smithi188_113957_object89995:head# 2024-04-23T16:33:38.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53535b89:::benchmark_data_smithi188_113957_object121457:head# 2024-04-23T16:33:38.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53536369:::benchmark_data_smithi188_113957_object153051:head# 2024-04-23T16:33:38.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5354baf7:::benchmark_data_smithi188_113957_object48429:head# 2024-04-23T16:33:38.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5354ca12:::benchmark_data_smithi188_113957_object29349:head# 2024-04-23T16:33:38.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5355e808:::benchmark_data_smithi188_113957_object30859:head# 2024-04-23T16:33:38.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5355ee27:::benchmark_data_smithi188_113957_object121892:head# 2024-04-23T16:33:38.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:535623d8:::benchmark_data_smithi188_113957_object24042:head# 2024-04-23T16:33:38.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:535678b6:::benchmark_data_smithi188_113957_object21634:head# 2024-04-23T16:33:38.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5356bbe1:::benchmark_data_smithi188_113957_object126424:head# 2024-04-23T16:33:38.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53573642:::benchmark_data_smithi188_113957_object61806:head# 2024-04-23T16:33:38.596 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53580881:::benchmark_data_smithi188_113957_object52406:head# 2024-04-23T16:33:38.596 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53580b2f:::benchmark_data_smithi188_113957_object8675:head# 2024-04-23T16:33:38.596 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53591c48:::benchmark_data_smithi188_113957_object119224:head# 2024-04-23T16:33:38.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:535985ed:::benchmark_data_smithi188_113957_object88336:head# 2024-04-23T16:33:38.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53598b59:::benchmark_data_smithi188_113957_object81462:head# 2024-04-23T16:33:38.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5359a3bb:::benchmark_data_smithi188_113957_object7396:head# 2024-04-23T16:33:38.598 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:535a3893:::benchmark_data_smithi188_113957_object78856:head# 2024-04-23T16:33:38.598 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:535a8a86:::benchmark_data_smithi188_113957_object29914:head# 2024-04-23T16:33:38.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:535b064f:::benchmark_data_smithi188_113957_object6617:head# 2024-04-23T16:33:38.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:535b4398:::benchmark_data_smithi188_113957_object108125:head# 2024-04-23T16:33:38.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:535b89ad:::benchmark_data_smithi188_113957_object89744:head# 2024-04-23T16:33:38.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:535c3c08:::benchmark_data_smithi188_113957_object94918:head# 2024-04-23T16:33:38.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:535d671c:::benchmark_data_smithi188_113957_object151814:head# 2024-04-23T16:33:38.602 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:535edf45:::benchmark_data_smithi188_113957_object148936:head# 2024-04-23T16:33:38.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:535fa0cd:::benchmark_data_smithi188_113957_object70025:head# 2024-04-23T16:33:38.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:535fb5a7:::benchmark_data_smithi188_113957_object52479:head# 2024-04-23T16:33:38.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53612cd7:::benchmark_data_smithi188_113957_object96501:head# 2024-04-23T16:33:38.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53612d79:::benchmark_data_smithi188_113957_object93395:head# 2024-04-23T16:33:38.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:536242b3:::benchmark_data_smithi188_113957_object150957:head# 2024-04-23T16:33:38.605 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:536356c2:::benchmark_data_smithi188_113957_object50708:head# 2024-04-23T16:33:38.605 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5364ead0:::benchmark_data_smithi188_113957_object110319:head# 2024-04-23T16:33:38.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53656ab8:::benchmark_data_smithi188_113957_object84170:head# 2024-04-23T16:33:38.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5365a1e9:::benchmark_data_smithi188_113957_object2933:head# 2024-04-23T16:33:38.607 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5365ca70:::benchmark_data_smithi188_113957_object2650:head# 2024-04-23T16:33:38.607 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:536613e1:::benchmark_data_smithi188_113957_object51281:head# 2024-04-23T16:33:38.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53669e30:::benchmark_data_smithi188_113957_object41040:head# 2024-04-23T16:33:38.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5366d34d:::benchmark_data_smithi188_113957_object45430:head# 2024-04-23T16:33:38.609 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53674c30:::benchmark_data_smithi188_113957_object115195:head# 2024-04-23T16:33:38.609 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53678d19:::benchmark_data_smithi188_113957_object134100:head# 2024-04-23T16:33:38.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5367fa79:::benchmark_data_smithi188_113957_object30171:head# 2024-04-23T16:33:38.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53685e5b:::benchmark_data_smithi188_113957_object7091:head# 2024-04-23T16:33:38.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53686cd4:::benchmark_data_smithi188_113957_object133586:head# 2024-04-23T16:33:38.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:536891d3:::benchmark_data_smithi188_113957_object37092:head# 2024-04-23T16:33:38.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53696d7e:::benchmark_data_smithi188_113957_object13499:head# 2024-04-23T16:33:38.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:536bdecb:::benchmark_data_smithi188_113957_object94994:head# 2024-04-23T16:33:38.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:536bf14f:::benchmark_data_smithi188_113957_object39209:head# 2024-04-23T16:33:38.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:536ce97e:::benchmark_data_smithi188_113957_object78652:head# 2024-04-23T16:33:38.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:536d8532:::benchmark_data_smithi188_113957_object129391:head# 2024-04-23T16:33:38.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:536dde01:::benchmark_data_smithi188_113957_object121252:head# 2024-04-23T16:33:38.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:536ed31e:::benchmark_data_smithi188_113957_object15208:head# 2024-04-23T16:33:38.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:536fd1ef:::benchmark_data_smithi188_113957_object101898:head# 2024-04-23T16:33:38.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:536ff8a7:::benchmark_data_smithi188_113957_object769:head# 2024-04-23T16:33:38.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:536ffbb6:::benchmark_data_smithi188_113957_object111896:head# 2024-04-23T16:33:38.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5370bfd3:::benchmark_data_smithi188_113957_object39185:head# 2024-04-23T16:33:38.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53710a40:::benchmark_data_smithi188_113957_object120523:head# 2024-04-23T16:33:38.616 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:537111c3:::benchmark_data_smithi188_113957_object101345:head# 2024-04-23T16:33:38.616 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53714283:::benchmark_data_smithi188_113957_object145701:head# 2024-04-23T16:33:38.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53714416:::benchmark_data_smithi188_113957_object36166:head# 2024-04-23T16:33:38.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53714938:::benchmark_data_smithi188_113957_object113547:head# 2024-04-23T16:33:38.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5371af46:::benchmark_data_smithi188_113957_object106870:head# 2024-04-23T16:33:38.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5372a37c:::benchmark_data_smithi188_113957_object63958:head# 2024-04-23T16:33:38.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:537323eb:::benchmark_data_smithi188_113957_object143312:head# 2024-04-23T16:33:38.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53732b79:::benchmark_data_smithi188_113957_object120968:head# 2024-04-23T16:33:38.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5373fbad:::benchmark_data_smithi188_113957_object54201:head# 2024-04-23T16:33:38.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5374000b:::benchmark_data_smithi188_113957_object13967:head# 2024-04-23T16:33:38.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5374dab0:::benchmark_data_smithi188_113957_object85808:head# 2024-04-23T16:33:38.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53751bee:::benchmark_data_smithi188_113957_object4614:head# 2024-04-23T16:33:38.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53752a48:::benchmark_data_smithi188_113957_object105621:head# 2024-04-23T16:33:38.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53757c0b:::benchmark_data_smithi188_113957_object58355:head# 2024-04-23T16:33:38.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53771aec:::benchmark_data_smithi188_113957_object74192:head# 2024-04-23T16:33:38.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53776270:::benchmark_data_smithi188_113957_object124283:head# 2024-04-23T16:33:38.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5377f60f:::benchmark_data_smithi188_113957_object143797:head# 2024-04-23T16:33:38.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53789f23:::benchmark_data_smithi188_113957_object151582:head# 2024-04-23T16:33:38.624 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5379ef92:::benchmark_data_smithi188_113957_object19360:head# 2024-04-23T16:33:38.624 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:537a0524:::benchmark_data_smithi188_113957_object78136:head# 2024-04-23T16:33:38.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:537a924a:::benchmark_data_smithi188_113957_object45752:head# 2024-04-23T16:33:38.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:537abe9e:::benchmark_data_smithi188_113957_object32244:head# 2024-04-23T16:33:38.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:537b0520:::benchmark_data_smithi188_113957_object48405:head# 2024-04-23T16:33:38.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:537b53b0:::benchmark_data_smithi188_113957_object32418:head# 2024-04-23T16:33:38.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:537c093a:::benchmark_data_smithi188_113957_object144338:head# 2024-04-23T16:33:38.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:537c48bf:::benchmark_data_smithi188_113957_object124420:head# 2024-04-23T16:33:38.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:537c839c:::benchmark_data_smithi188_113957_object6530:head# 2024-04-23T16:33:38.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:537cdc9a:::benchmark_data_smithi188_113957_object80639:head# 2024-04-23T16:33:38.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:537d3bed:::benchmark_data_smithi188_113957_object144992:head# 2024-04-23T16:33:38.629 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:33:38.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:537dc7a6:::benchmark_data_smithi188_113957_object56826:head# 2024-04-23T16:33:38.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:537dd1ad:::benchmark_data_smithi188_113957_object104449:head# 2024-04-23T16:33:38.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:537ec1ac:::benchmark_data_smithi188_113957_object28522:head# 2024-04-23T16:33:38.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:537ec70e:::benchmark_data_smithi188_113957_object34803:head# 2024-04-23T16:33:38.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:537f98ff:::benchmark_data_smithi188_113957_object63311:head# 2024-04-23T16:33:38.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5380013d:::benchmark_data_smithi188_113957_object110348:head# 2024-04-23T16:33:38.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5380d6e8:::benchmark_data_smithi188_113957_object137993:head# 2024-04-23T16:33:38.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53811cc2:::benchmark_data_smithi188_113957_object100494:head# 2024-04-23T16:33:38.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53812277:::benchmark_data_smithi188_113957_object79143:head# 2024-04-23T16:33:38.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53817478:::benchmark_data_smithi188_113957_object109116:head# 2024-04-23T16:33:38.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53819a04:::benchmark_data_smithi188_113957_object5115:head# 2024-04-23T16:33:38.635 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5381f9e8:::benchmark_data_smithi188_113957_object100067:head# 2024-04-23T16:33:38.635 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53827466:::benchmark_data_smithi188_113957_object49707:head# 2024-04-23T16:33:38.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53828752:::benchmark_data_smithi188_113957_object106893:head# 2024-04-23T16:33:38.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:538310f1:::benchmark_data_smithi188_113957_object33010:head# 2024-04-23T16:33:38.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53832ea5:::benchmark_data_smithi188_113957_object7776:head# 2024-04-23T16:33:38.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53837c3d:::benchmark_data_smithi188_113957_object31194:head# 2024-04-23T16:33:38.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5383cf98:::benchmark_data_smithi188_113957_object72780:head# 2024-04-23T16:33:38.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5383e26f:::benchmark_data_smithi188_113957_object97630:head# 2024-04-23T16:33:38.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5384493c:::benchmark_data_smithi188_113957_object137106:head# 2024-04-23T16:33:38.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:538453e0:::benchmark_data_smithi188_113957_object58148:head# 2024-04-23T16:33:38.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:538459f2:::benchmark_data_smithi188_113957_object136113:head# 2024-04-23T16:33:38.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53876e11:::benchmark_data_smithi188_113957_object81101:head# 2024-04-23T16:33:38.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53878a94:::benchmark_data_smithi188_113957_object150694:head# 2024-04-23T16:33:38.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5387ab76:::benchmark_data_smithi188_113957_object37428:head# 2024-04-23T16:33:38.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5387cbca:::benchmark_data_smithi188_113957_object49537:head# 2024-04-23T16:33:38.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5387dbce:::benchmark_data_smithi188_113957_object95858:head# 2024-04-23T16:33:38.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5387e08e:::benchmark_data_smithi188_113957_object20139:head# 2024-04-23T16:33:38.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5388ac9d:::benchmark_data_smithi188_113957_object68801:head# 2024-04-23T16:33:38.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5388b3b6:::benchmark_data_smithi188_113957_object151826:head# 2024-04-23T16:33:38.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5388f6fb:::benchmark_data_smithi188_113957_object141738:head# 2024-04-23T16:33:38.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53897d99:::benchmark_data_smithi188_113957_object118810:head# 2024-04-23T16:33:38.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:538a330f:::benchmark_data_smithi188_113957_object57126:head# 2024-04-23T16:33:38.645 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:538a4bef:::benchmark_data_smithi188_113957_object4711:head# 2024-04-23T16:33:38.645 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:538a6d06:::benchmark_data_smithi188_113957_object11294:head# 2024-04-23T16:33:38.645 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:538a9a9f:::benchmark_data_smithi188_113957_object68011:head# 2024-04-23T16:33:38.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:538b9689:::benchmark_data_smithi188_113957_object81053:head# 2024-04-23T16:33:38.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:538b9ca6:::benchmark_data_smithi188_113957_object6006:head# 2024-04-23T16:33:38.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:538ca99a:::benchmark_data_smithi188_113957_object92396:head# 2024-04-23T16:33:38.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:538dda59:::benchmark_data_smithi188_113957_object47731:head# 2024-04-23T16:33:38.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:538dfec0:::benchmark_data_smithi188_113957_object50645:head# 2024-04-23T16:33:38.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:538e9906:::benchmark_data_smithi188_113957_object19263:head# 2024-04-23T16:33:38.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:538ef6e3:::benchmark_data_smithi188_113957_object76632:head# 2024-04-23T16:33:38.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:538f54f9:::benchmark_data_smithi188_113957_object72209:head# 2024-04-23T16:33:38.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:538ff93b:::benchmark_data_smithi188_113957_object110335:head# 2024-04-23T16:33:38.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53904e89:::benchmark_data_smithi188_113957_object145443:head# 2024-04-23T16:33:38.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53905376:::benchmark_data_smithi188_113957_object108995:head# 2024-04-23T16:33:38.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:539101ef:::benchmark_data_smithi188_113957_object2781:head# 2024-04-23T16:33:38.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5392330c:::benchmark_data_smithi188_113957_object99563:head# 2024-04-23T16:33:38.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53928c89:::benchmark_data_smithi188_113957_object106841:head# 2024-04-23T16:33:38.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5392e2a7:::benchmark_data_smithi188_113957_object133104:head# 2024-04-23T16:33:38.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5393d880:::benchmark_data_smithi188_113957_object55171:head# 2024-04-23T16:33:38.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5393dbdd:::benchmark_data_smithi188_113957_object43606:head# 2024-04-23T16:33:38.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53941fb2:::benchmark_data_smithi188_113957_object6351:head# 2024-04-23T16:33:38.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5395aa39:::benchmark_data_smithi188_113957_object103503:head# 2024-04-23T16:33:38.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53962fa3:::benchmark_data_smithi188_113957_object80932:head# 2024-04-23T16:33:38.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53964364:::benchmark_data_smithi188_113957_object36571:head# 2024-04-23T16:33:38.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5396f75d:::benchmark_data_smithi188_113957_object32190:head# 2024-04-23T16:33:38.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5396ff82:::benchmark_data_smithi188_113957_object22537:head# 2024-04-23T16:33:38.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:539773ea:::benchmark_data_smithi188_113957_object54905:head# 2024-04-23T16:33:38.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5397959b:::benchmark_data_smithi188_113957_object147915:head# 2024-04-23T16:33:38.657 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:38.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5397c471:::benchmark_data_smithi188_113957_object135025:head# 2024-04-23T16:33:38.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5398100f:::benchmark_data_smithi188_113957_object67870:head# 2024-04-23T16:33:38.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5399125a:::benchmark_data_smithi188_113957_object136201:head# 2024-04-23T16:33:38.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53992810:::benchmark_data_smithi188_113957_object94659:head# 2024-04-23T16:33:38.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5399fc23:::benchmark_data_smithi188_113957_object39577:head# 2024-04-23T16:33:38.661 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:539a7a7f:::benchmark_data_smithi188_113957_object82206:head# 2024-04-23T16:33:38.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:539ad625:::benchmark_data_smithi188_113957_object43097:head# 2024-04-23T16:33:38.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:539b0164:::benchmark_data_smithi188_113957_object83889:head# 2024-04-23T16:33:38.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:539b6f14:::benchmark_data_smithi188_113957_object106304:head# 2024-04-23T16:33:38.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:539b77c1:::benchmark_data_smithi188_113957_object128458:head# 2024-04-23T16:33:38.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:539ba7c0:::benchmark_data_smithi188_113957_object19911:head# 2024-04-23T16:33:38.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:539bdde9:::benchmark_data_smithi188_113957_object41409:head# 2024-04-23T16:33:38.668 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:539c1efd:::benchmark_data_smithi188_113957_object64907:head# 2024-04-23T16:33:38.669 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:539c8eaa:::benchmark_data_smithi188_113957_object32249:head# 2024-04-23T16:33:38.669 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:539d2b4e:::benchmark_data_smithi188_113957_object2160:head# 2024-04-23T16:33:38.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:539e1b50:::benchmark_data_smithi188_113957_object37660:head# 2024-04-23T16:33:38.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:539e327e:::benchmark_data_smithi188_113957_object46744:head# 2024-04-23T16:33:38.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:539ea435:::benchmark_data_smithi188_113957_object96958:head# 2024-04-23T16:33:38.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:539f4c71:::benchmark_data_smithi188_113957_object76956:head# 2024-04-23T16:33:38.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:539f637a:::benchmark_data_smithi188_113957_object4020:head# 2024-04-23T16:33:38.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a016fe:::benchmark_data_smithi188_113957_object26434:head# 2024-04-23T16:33:38.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a0ce4a:::benchmark_data_smithi188_113957_object59397:head# 2024-04-23T16:33:38.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a0df40:::benchmark_data_smithi188_113957_object115258:head# 2024-04-23T16:33:38.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a0fa08:::benchmark_data_smithi188_113957_object17818:head# 2024-04-23T16:33:38.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a284ee:::benchmark_data_smithi188_113957_object58284:head# 2024-04-23T16:33:38.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a32b12:::benchmark_data_smithi188_113957_object117858:head# 2024-04-23T16:33:38.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a42ddc:::benchmark_data_smithi188_113957_object23851:head# 2024-04-23T16:33:38.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a4f02b:::benchmark_data_smithi188_113957_object22518:head# 2024-04-23T16:33:38.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a4f116:::benchmark_data_smithi188_113957_object30238:head# 2024-04-23T16:33:38.679 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a62f92:::benchmark_data_smithi188_113957_object34139:head# 2024-04-23T16:33:38.679 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a67f49:::benchmark_data_smithi188_113957_object4913:head# 2024-04-23T16:33:38.679 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a6ca1a:::benchmark_data_smithi188_113957_object152903:head# 2024-04-23T16:33:38.680 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a71b93:::benchmark_data_smithi188_113957_object145811:head# 2024-04-23T16:33:38.680 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:33:38.680 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a73c0b:::benchmark_data_smithi188_113957_object88723:head# 2024-04-23T16:33:38.680 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a78acc:::benchmark_data_smithi188_113957_object82425:head# 2024-04-23T16:33:38.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a7a94a:::benchmark_data_smithi188_113957_object110251:head# 2024-04-23T16:33:38.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a846f1:::benchmark_data_smithi188_113957_object124910:head# 2024-04-23T16:33:38.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a87776:::benchmark_data_smithi188_113957_object73487:head# 2024-04-23T16:33:38.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a8a02a:::benchmark_data_smithi188_113957_object2036:head# 2024-04-23T16:33:38.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:38.683 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:33:38.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a96911:::benchmark_data_smithi188_113957_object137594:head# 2024-04-23T16:33:38.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a9d8f8:::benchmark_data_smithi188_113957_object125146:head# 2024-04-23T16:33:38.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53a9e25f:::benchmark_data_smithi188_113957_object152008:head# 2024-04-23T16:33:38.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53aa0cb9:::benchmark_data_smithi188_113957_object134755:head# 2024-04-23T16:33:38.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53aa5a3e:::benchmark_data_smithi188_113957_object134285:head# 2024-04-23T16:33:38.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53aab04a:::benchmark_data_smithi188_113957_object36433:head# 2024-04-23T16:33:38.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53aabe27:::benchmark_data_smithi188_113957_object124624:head# 2024-04-23T16:33:38.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53aac26c:::benchmark_data_smithi188_113957_object108854:head# 2024-04-23T16:33:38.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53aafe3c:::benchmark_data_smithi188_113957_object47836:head# 2024-04-23T16:33:38.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ab1d54:::benchmark_data_smithi188_113957_object82316:head# 2024-04-23T16:33:38.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ab25cb:::benchmark_data_smithi188_113957_object101249:head# 2024-04-23T16:33:38.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ab4910:::benchmark_data_smithi188_113957_object26795:head# 2024-04-23T16:33:38.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ac576d:::benchmark_data_smithi188_113957_object30344:head# 2024-04-23T16:33:38.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ac9847:::benchmark_data_smithi188_113957_object63366:head# 2024-04-23T16:33:38.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ad2065:::benchmark_data_smithi188_113957_object52801:head# 2024-04-23T16:33:38.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ae01a7:::benchmark_data_smithi188_113957_object105406:head# 2024-04-23T16:33:38.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53af323e:::benchmark_data_smithi188_113957_object18881:head# 2024-04-23T16:33:38.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b0304e:::benchmark_data_smithi188_113957_object9427:head# 2024-04-23T16:33:38.691 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b0334a:::benchmark_data_smithi188_113957_object139176:head# 2024-04-23T16:33:38.691 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b04445:::benchmark_data_smithi188_113957_object113281:head# 2024-04-23T16:33:38.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b07c17:::benchmark_data_smithi188_113957_object66142:head# 2024-04-23T16:33:38.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b08efb:::benchmark_data_smithi188_113957_object79570:head# 2024-04-23T16:33:38.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b0c9d9:::benchmark_data_smithi188_113957_object81357:head# 2024-04-23T16:33:38.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b0ce72:::benchmark_data_smithi188_113957_object72138:head# 2024-04-23T16:33:38.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b1128d:::benchmark_data_smithi188_113957_object77189:head# 2024-04-23T16:33:38.694 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b1ea14:::benchmark_data_smithi188_113957_object96051:head# 2024-04-23T16:33:38.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b20e67:::benchmark_data_smithi188_113957_object146917:head# 2024-04-23T16:33:38.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b241fe:::benchmark_data_smithi188_113957_object151659:head# 2024-04-23T16:33:38.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b27f5d:::benchmark_data_smithi188_113957_object38417:head# 2024-04-23T16:33:38.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b2a699:::benchmark_data_smithi188_113957_object104594:head# 2024-04-23T16:33:38.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b2becf:::benchmark_data_smithi188_113957_object110520:head# 2024-04-23T16:33:38.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b302a1:::benchmark_data_smithi188_113957_object80980:head# 2024-04-23T16:33:38.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b314b2:::benchmark_data_smithi188_113957_object23364:head# 2024-04-23T16:33:38.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b371b8:::benchmark_data_smithi188_113957_object28141:head# 2024-04-23T16:33:38.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b4b98f:::benchmark_data_smithi188_113957_object88944:head# 2024-04-23T16:33:38.699 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b52a64:::benchmark_data_smithi188_113957_object90779:head# 2024-04-23T16:33:38.699 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b5a372:::benchmark_data_smithi188_113957_object107455:head# 2024-04-23T16:33:38.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b5e523:::benchmark_data_smithi188_113957_object77325:head# 2024-04-23T16:33:38.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b6f246:::benchmark_data_smithi188_113957_object55441:head# 2024-04-23T16:33:38.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b79d05:::benchmark_data_smithi188_113957_object55371:head# 2024-04-23T16:33:38.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b82249:::benchmark_data_smithi188_113957_object100941:head# 2024-04-23T16:33:38.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b85b54:::benchmark_data_smithi188_113957_object99389:head# 2024-04-23T16:33:38.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b866f6:::benchmark_data_smithi188_113957_object126084:head# 2024-04-23T16:33:38.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b88583:::benchmark_data_smithi188_113957_object111081:head# 2024-04-23T16:33:38.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53b9c617:::benchmark_data_smithi188_113957_object120449:head# 2024-04-23T16:33:38.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ba42ff:::benchmark_data_smithi188_113957_object137726:head# 2024-04-23T16:33:38.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53babaa7:::benchmark_data_smithi188_113957_object135670:head# 2024-04-23T16:33:38.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53bc989e:::benchmark_data_smithi188_113957_object142380:head# 2024-04-23T16:33:38.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53bcac3b:::benchmark_data_smithi188_113957_object146904:head# 2024-04-23T16:33:38.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53bd0f9f:::benchmark_data_smithi188_113957_object94471:head# 2024-04-23T16:33:38.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53bd9ac3:::benchmark_data_smithi188_113957_object134499:head# 2024-04-23T16:33:38.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53bdb56f:::benchmark_data_smithi188_113957_object150217:head# 2024-04-23T16:33:38.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53bdcfc5:::benchmark_data_smithi188_113957_object97722:head# 2024-04-23T16:33:38.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53be3dd4:::benchmark_data_smithi188_113957_object51915:head# 2024-04-23T16:33:38.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53be94de:::benchmark_data_smithi188_113957_object91875:head# 2024-04-23T16:33:38.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53be9aa5:::benchmark_data_smithi188_113957_object27692:head# 2024-04-23T16:33:38.709 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53bef699:::benchmark_data_smithi188_113957_object62976:head# 2024-04-23T16:33:38.709 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53bf7985:::benchmark_data_smithi188_113957_object24650:head# 2024-04-23T16:33:38.709 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c09a39:::benchmark_data_smithi188_113957_object100282:head# 2024-04-23T16:33:38.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c12ed0:::benchmark_data_smithi188_113957_object53524:head# 2024-04-23T16:33:38.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c26b49:::benchmark_data_smithi188_113957_object113920:head# 2024-04-23T16:33:38.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c2b040:::benchmark_data_smithi188_113957_object123998:head# 2024-04-23T16:33:38.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c3670d:::benchmark_data_smithi188_113957_object81565:head# 2024-04-23T16:33:38.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c3690d:::benchmark_data_smithi188_113957_object23741:head# 2024-04-23T16:33:38.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c37b80:::benchmark_data_smithi188_113957_object96969:head# 2024-04-23T16:33:38.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c3a963:::benchmark_data_smithi188_113957_object41507:head# 2024-04-23T16:33:38.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c3d3bc:::benchmark_data_smithi188_113957_object62098:head# 2024-04-23T16:33:38.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c3ff6c:::benchmark_data_smithi188_113957_object152397:head# 2024-04-23T16:33:38.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c47db9:::benchmark_data_smithi188_113957_object128302:head# 2024-04-23T16:33:38.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c4f953:::benchmark_data_smithi188_113957_object10113:head# 2024-04-23T16:33:38.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c5721c:::benchmark_data_smithi188_113957_object5079:head# 2024-04-23T16:33:38.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c652d3:::benchmark_data_smithi188_113957_object71216:head# 2024-04-23T16:33:38.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c69046:::benchmark_data_smithi188_113957_object83399:head# 2024-04-23T16:33:38.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c6b85d:::benchmark_data_smithi188_113957_object6809:head# 2024-04-23T16:33:38.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c75030:::benchmark_data_smithi188_113957_object47021:head# 2024-04-23T16:33:38.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c7991f:::benchmark_data_smithi188_113957_object85011:head# 2024-04-23T16:33:38.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c84f8c:::benchmark_data_smithi188_113957_object54871:head# 2024-04-23T16:33:38.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c89b06:::benchmark_data_smithi188_113957_object78465:head# 2024-04-23T16:33:38.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c963be:::benchmark_data_smithi188_113957_object80377:head# 2024-04-23T16:33:38.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53c9b9c5:::benchmark_data_smithi188_113957_object11567:head# 2024-04-23T16:33:38.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53cb5c34:::benchmark_data_smithi188_113957_object114502:head# 2024-04-23T16:33:38.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ccbb08:::benchmark_data_smithi188_113957_object28459:head# 2024-04-23T16:33:38.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ccf08d:::benchmark_data_smithi188_113957_object32411:head# 2024-04-23T16:33:38.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53cd1a6f:::benchmark_data_smithi188_113957_object137455:head# 2024-04-23T16:33:38.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53cd5cf2:::benchmark_data_smithi188_113957_object125990:head# 2024-04-23T16:33:38.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53cd61a1:::benchmark_data_smithi188_113957_object124623:head# 2024-04-23T16:33:38.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53cec3af:::benchmark_data_smithi188_113957_object80759:head# 2024-04-23T16:33:38.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53cf1a61:::benchmark_data_smithi188_113957_object609:head# 2024-04-23T16:33:38.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53cf6a19:::benchmark_data_smithi188_113957_object132549:head# 2024-04-23T16:33:38.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53cf6cd1:::benchmark_data_smithi188_113957_object10909:head# 2024-04-23T16:33:38.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53cf7dda:::benchmark_data_smithi188_113957_object79620:head# 2024-04-23T16:33:38.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53cf9efd:::benchmark_data_smithi188_113957_object90298:head# 2024-04-23T16:33:38.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53cff3b5:::benchmark_data_smithi188_113957_object29496:head# 2024-04-23T16:33:38.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53cffbdf:::benchmark_data_smithi188_113957_object4266:head# 2024-04-23T16:33:38.725 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d085e9:::benchmark_data_smithi188_113957_object116525:head# 2024-04-23T16:33:38.725 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d0b12c:::benchmark_data_smithi188_113957_object139733:head# 2024-04-23T16:33:38.725 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d0be85:::benchmark_data_smithi188_113957_object57990:head# 2024-04-23T16:33:38.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d10da1:::benchmark_data_smithi188_113957_object93697:head# 2024-04-23T16:33:38.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d1dffd:::benchmark_data_smithi188_113957_object33367:head# 2024-04-23T16:33:38.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d1f577:::benchmark_data_smithi188_113957_object128412:head# 2024-04-23T16:33:38.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d1fc01:::benchmark_data_smithi188_113957_object79758:head# 2024-04-23T16:33:38.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d24c27:::benchmark_data_smithi188_113957_object141707:head# 2024-04-23T16:33:38.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d32328:::benchmark_data_smithi188_113957_object109627:head# 2024-04-23T16:33:38.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d32346:::benchmark_data_smithi188_113957_object114263:head# 2024-04-23T16:33:38.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d371ec:::benchmark_data_smithi188_113957_object34932:head# 2024-04-23T16:33:38.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d3aaa6:::benchmark_data_smithi188_113957_object27555:head# 2024-04-23T16:33:38.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d40fcf:::benchmark_data_smithi188_113957_object13794:head# 2024-04-23T16:33:38.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d526c8:::benchmark_data_smithi188_113957_object128472:head# 2024-04-23T16:33:38.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d5527d:::benchmark_data_smithi188_113957_object121120:head# 2024-04-23T16:33:38.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d57f44:::benchmark_data_smithi188_113957_object80555:head# 2024-04-23T16:33:38.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d586b6:::benchmark_data_smithi188_113957_object59828:head# 2024-04-23T16:33:38.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d58edc:::benchmark_data_smithi188_113957_object14822:head# 2024-04-23T16:33:38.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d6728b:::benchmark_data_smithi188_113957_object9343:head# 2024-04-23T16:33:38.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d7ed42:::benchmark_data_smithi188_113957_object24316:head# 2024-04-23T16:33:38.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d7f047:::benchmark_data_smithi188_113957_object126707:head# 2024-04-23T16:33:38.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d88ba7:::benchmark_data_smithi188_113957_object64408:head# 2024-04-23T16:33:38.735 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53d99e53:::benchmark_data_smithi188_113957_object14176:head# 2024-04-23T16:33:38.735 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53da3e8d:::benchmark_data_smithi188_113957_object77904:head# 2024-04-23T16:33:38.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53dac305:::benchmark_data_smithi188_113957_object51222:head# 2024-04-23T16:33:38.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53db1b4c:::benchmark_data_smithi188_113957_object32556:head# 2024-04-23T16:33:38.737 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53db78fe:::benchmark_data_smithi188_113957_object108742:head# 2024-04-23T16:33:38.737 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53db8ce9:::benchmark_data_smithi188_113957_object88620:head# 2024-04-23T16:33:38.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53db9250:::benchmark_data_smithi188_113957_object83158:head# 2024-04-23T16:33:38.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53dbb82a:::benchmark_data_smithi188_113957_object51135:head# 2024-04-23T16:33:38.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53dc18c1:::benchmark_data_smithi188_113957_object109447:head# 2024-04-23T16:33:38.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53dc1d58:::benchmark_data_smithi188_113957_object142194:head# 2024-04-23T16:33:38.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53dca14a:::benchmark_data_smithi188_113957_object77259:head# 2024-04-23T16:33:38.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53dd0db1:::benchmark_data_smithi188_113957_object32477:head# 2024-04-23T16:33:38.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53dd2d96:::benchmark_data_smithi188_113957_object133371:head# 2024-04-23T16:33:38.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53dd6182:::benchmark_data_smithi188_113957_object79980:head# 2024-04-23T16:33:38.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ddd930:::benchmark_data_smithi188_113957_object115324:head# 2024-04-23T16:33:38.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ddddec:::benchmark_data_smithi188_113957_object108536:head# 2024-04-23T16:33:38.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53deab25:::benchmark_data_smithi188_113957_object21195:head# 2024-04-23T16:33:38.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53dec1df:::benchmark_data_smithi188_113957_object108517:head# 2024-04-23T16:33:38.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53dfb314:::benchmark_data_smithi188_113957_object80623:head# 2024-04-23T16:33:38.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53dfcba7:::benchmark_data_smithi188_113957_object12735:head# 2024-04-23T16:33:38.744 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53dfee3e:::benchmark_data_smithi188_113957_object139055:head# 2024-04-23T16:33:38.744 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53dff203:::benchmark_data_smithi188_113957_object65556:head# 2024-04-23T16:33:38.745 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53e046c7:::benchmark_data_smithi188_113957_object46236:head# 2024-04-23T16:33:38.745 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53e0791e:::benchmark_data_smithi188_113957_object98279:head# 2024-04-23T16:33:38.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53e0e100:::benchmark_data_smithi188_113957_object153194:head# 2024-04-23T16:33:38.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53e15379:::benchmark_data_smithi188_113957_object146676:head# 2024-04-23T16:33:38.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53e17960:::benchmark_data_smithi188_113957_object42382:head# 2024-04-23T16:33:38.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53e1e144:::benchmark_data_smithi188_113957_object141926:head# 2024-04-23T16:33:38.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53e1e52f:::benchmark_data_smithi188_113957_object89697:head# 2024-04-23T16:33:38.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53e21ba7:::benchmark_data_smithi188_113957_object151379:head# 2024-04-23T16:33:38.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53e30a38:::benchmark_data_smithi188_113957_object15960:head# 2024-04-23T16:33:38.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53e44431:::benchmark_data_smithi188_113957_object92807:head# 2024-04-23T16:33:38.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53e4d5fa:::benchmark_data_smithi188_113957_object1395:head# 2024-04-23T16:33:38.750 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53e521da:::benchmark_data_smithi188_113957_object145851:head# 2024-04-23T16:33:38.750 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53e545d9:::benchmark_data_smithi188_113957_object42823:head# 2024-04-23T16:33:38.750 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53e5cb7f:::benchmark_data_smithi188_113957_object93317:head# 2024-04-23T16:33:38.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53e5d49e:::benchmark_data_smithi188_113957_object148554:head# 2024-04-23T16:33:38.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53e63439:::benchmark_data_smithi188_113957_object97387:head# 2024-04-23T16:33:38.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53e82619:::benchmark_data_smithi188_113957_object82939:head# 2024-04-23T16:33:38.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53e8575e:::benchmark_data_smithi188_113957_object118678:head# 2024-04-23T16:33:38.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53e8d136:::benchmark_data_smithi188_113957_object88936:head# 2024-04-23T16:33:38.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53eabdf3:::benchmark_data_smithi188_113957_object84078:head# 2024-04-23T16:33:38.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53eabe2d:::benchmark_data_smithi188_113957_object72500:head# 2024-04-23T16:33:38.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53eae542:::benchmark_data_smithi188_113957_object72846:head# 2024-04-23T16:33:38.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53eb638c:::benchmark_data_smithi188_113957_object36760:head# 2024-04-23T16:33:38.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53eb9be8:::benchmark_data_smithi188_113957_object24267:head# 2024-04-23T16:33:38.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53eb9dd7:::benchmark_data_smithi188_113957_object139963:head# 2024-04-23T16:33:38.756 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ec7250:::benchmark_data_smithi188_113957_object141653:head# 2024-04-23T16:33:38.756 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53eca293:::benchmark_data_smithi188_113957_object39982:head# 2024-04-23T16:33:38.756 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53eca447:::benchmark_data_smithi188_113957_object89890:head# 2024-04-23T16:33:38.757 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ecb91f:::benchmark_data_smithi188_113957_object28235:head# 2024-04-23T16:33:38.757 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ed0fba:::benchmark_data_smithi188_113957_object148845:head# 2024-04-23T16:33:38.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ed441e:::benchmark_data_smithi188_113957_object108563:head# 2024-04-23T16:33:38.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ed55f5:::benchmark_data_smithi188_113957_object9969:head# 2024-04-23T16:33:38.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ee6fe6:::benchmark_data_smithi188_113957_object141828:head# 2024-04-23T16:33:38.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53eedf28:::benchmark_data_smithi188_113957_object133184:head# 2024-04-23T16:33:38.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53eee072:::benchmark_data_smithi188_113957_object62486:head# 2024-04-23T16:33:38.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53eee251:::benchmark_data_smithi188_113957_object143094:head# 2024-04-23T16:33:38.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ef3417:::benchmark_data_smithi188_113957_object21240:head# 2024-04-23T16:33:38.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f02958:::benchmark_data_smithi188_113957_object11154:head# 2024-04-23T16:33:38.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f05463:::benchmark_data_smithi188_113957_object49200:head# 2024-04-23T16:33:38.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f07e59:::benchmark_data_smithi188_113957_object38409:head# 2024-04-23T16:33:38.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f0cca5:::benchmark_data_smithi188_113957_object72049:head# 2024-04-23T16:33:38.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f0d8a2:::benchmark_data_smithi188_113957_object38859:head# 2024-04-23T16:33:38.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f0e1fa:::benchmark_data_smithi188_113957_object117498:head# 2024-04-23T16:33:38.764 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f0eb17:::benchmark_data_smithi188_113957_object118415:head# 2024-04-23T16:33:38.764 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f13274:::benchmark_data_smithi188_113957_object136908:head# 2024-04-23T16:33:38.765 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f1b50a:::benchmark_data_smithi188_113957_object49191:head# 2024-04-23T16:33:38.765 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f2c277:::benchmark_data_smithi188_113957_object49976:head# 2024-04-23T16:33:38.765 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f2d1ad:::benchmark_data_smithi188_113957_object87655:head# 2024-04-23T16:33:38.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f37936:::benchmark_data_smithi188_113957_object101087:head# 2024-04-23T16:33:38.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f39d66:::benchmark_data_smithi188_113957_object68598:head# 2024-04-23T16:33:38.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f3bbdc:::benchmark_data_smithi188_113957_object134348:head# 2024-04-23T16:33:38.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f5c55a:::benchmark_data_smithi188_113957_object65868:head# 2024-04-23T16:33:38.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f606f9:::benchmark_data_smithi188_113957_object41048:head# 2024-04-23T16:33:38.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f61e2a:::benchmark_data_smithi188_113957_object100436:head# 2024-04-23T16:33:38.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f62008:::benchmark_data_smithi188_113957_object85445:head# 2024-04-23T16:33:38.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f63c78:::benchmark_data_smithi188_113957_object126191:head# 2024-04-23T16:33:38.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f6ec35:::benchmark_data_smithi188_113957_object116268:head# 2024-04-23T16:33:38.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f7ea13:::benchmark_data_smithi188_113957_object43039:head# 2024-04-23T16:33:38.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f8fc9f:::benchmark_data_smithi188_113957_object131649:head# 2024-04-23T16:33:38.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f95c2b:::benchmark_data_smithi188_113957_object99876:head# 2024-04-23T16:33:38.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53f9f591:::benchmark_data_smithi188_113957_object66532:head# 2024-04-23T16:33:38.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53fa8841:::benchmark_data_smithi188_113957_object113093:head# 2024-04-23T16:33:38.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53fa916e:::benchmark_data_smithi188_113957_object38581:head# 2024-04-23T16:33:38.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53fbaa3a:::benchmark_data_smithi188_113957_object144446:head# 2024-04-23T16:33:38.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53fd7e4f:::benchmark_data_smithi188_113957_object88424:head# 2024-04-23T16:33:38.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53fd9fe5:::benchmark_data_smithi188_113957_object98770:head# 2024-04-23T16:33:38.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53fe0752:::benchmark_data_smithi188_113957_object105648:head# 2024-04-23T16:33:38.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53fe334b:::benchmark_data_smithi188_113957_object60886:head# 2024-04-23T16:33:38.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53fee7f2:::benchmark_data_smithi188_113957_object110151:head# 2024-04-23T16:33:38.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ff4181:::benchmark_data_smithi188_113957_object138996:head# 2024-04-23T16:33:38.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ff44c7:::benchmark_data_smithi188_113957_object38759:head# 2024-04-23T16:33:38.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ff46a9:::benchmark_data_smithi188_113957_object7392:head# 2024-04-23T16:33:38.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:53ffa6cb:::benchmark_data_smithi188_113957_object5752:head# 2024-04-23T16:33:38.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54004db2:::benchmark_data_smithi188_113957_object108555:head# 2024-04-23T16:33:38.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5401836f:::benchmark_data_smithi188_113957_object96866:head# 2024-04-23T16:33:38.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540197de:::benchmark_data_smithi188_113957_object81282:head# 2024-04-23T16:33:38.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54032504:::benchmark_data_smithi188_113957_object36298:head# 2024-04-23T16:33:38.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54035435:::benchmark_data_smithi188_113957_object31466:head# 2024-04-23T16:33:38.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5403ac8f:::benchmark_data_smithi188_113957_object58269:head# 2024-04-23T16:33:38.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5404ab11:::benchmark_data_smithi188_113957_object84392:head# 2024-04-23T16:33:38.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5404d004:::benchmark_data_smithi188_113957_object26409:head# 2024-04-23T16:33:38.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54053f7a:::benchmark_data_smithi188_113957_object21281:head# 2024-04-23T16:33:38.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54054a3d:::benchmark_data_smithi188_113957_object33806:head# 2024-04-23T16:33:38.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54056445:::benchmark_data_smithi188_113957_object136785:head# 2024-04-23T16:33:38.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5405677d:::benchmark_data_smithi188_113957_object105745:head# 2024-04-23T16:33:38.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5405b5ab:::benchmark_data_smithi188_113957_object44928:head# 2024-04-23T16:33:38.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5405e303:::benchmark_data_smithi188_113957_object2687:head# 2024-04-23T16:33:38.783 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:33:38.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540633b3:::benchmark_data_smithi188_113957_object120208:head# 2024-04-23T16:33:38.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54065216:::benchmark_data_smithi188_113957_object79345:head# 2024-04-23T16:33:38.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54065258:::benchmark_data_smithi188_113957_object90347:head# 2024-04-23T16:33:38.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54065f53:::benchmark_data_smithi188_113957_object59290:head# 2024-04-23T16:33:38.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5406aa54:::benchmark_data_smithi188_113957_object11475:head# 2024-04-23T16:33:38.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5406c9b4:::benchmark_data_smithi188_113957_object9530:head# 2024-04-23T16:33:38.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5406de04:::benchmark_data_smithi188_113957_object8037:head# 2024-04-23T16:33:38.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5406eaa7:::benchmark_data_smithi188_113957_object34053:head# 2024-04-23T16:33:38.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5406f2d8:::benchmark_data_smithi188_113957_object20504:head# 2024-04-23T16:33:38.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5408a1af:::benchmark_data_smithi188_113957_object22479:head# 2024-04-23T16:33:38.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5408c311:::benchmark_data_smithi188_113957_object95091:head# 2024-04-23T16:33:38.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5409234a:::benchmark_data_smithi188_113957_object77516:head# 2024-04-23T16:33:38.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54098ded:::benchmark_data_smithi188_113957_object55099:head# 2024-04-23T16:33:38.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54099d68:::benchmark_data_smithi188_113957_object62373:head# 2024-04-23T16:33:38.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5409a451:::benchmark_data_smithi188_113957_object65094:head# 2024-04-23T16:33:38.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5409f9d8:::benchmark_data_smithi188_113957_object39884:head# 2024-04-23T16:33:38.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540a18d1:::benchmark_data_smithi188_113957_object135392:head# 2024-04-23T16:33:38.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540a5346:::benchmark_data_smithi188_113957_object125236:head# 2024-04-23T16:33:38.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540a654e:::benchmark_data_smithi188_113957_object147370:head# 2024-04-23T16:33:38.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540abdf8:::benchmark_data_smithi188_113957_object64537:head# 2024-04-23T16:33:38.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540accdc:::benchmark_data_smithi188_113957_object46273:head# 2024-04-23T16:33:38.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540b509f:::benchmark_data_smithi188_113957_object138619:head# 2024-04-23T16:33:38.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540b9db8:::benchmark_data_smithi188_113957_object76917:head# 2024-04-23T16:33:38.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540bda55:::benchmark_data_smithi188_113957_object146344:head# 2024-04-23T16:33:38.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540c08ec:::benchmark_data_smithi188_113957_object20038:head# 2024-04-23T16:33:38.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540c69f3:::benchmark_data_smithi188_113957_object30586:head# 2024-04-23T16:33:38.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540c912b:::benchmark_data_smithi188_113957_object142326:head# 2024-04-23T16:33:38.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540dddff:::benchmark_data_smithi188_113957_object122631:head# 2024-04-23T16:33:38.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540e2057:::benchmark_data_smithi188_113957_object87880:head# 2024-04-23T16:33:38.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540e2bff:::benchmark_data_smithi188_113957_object142220:head# 2024-04-23T16:33:38.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540e8177:::benchmark_data_smithi188_113957_object104824:head# 2024-04-23T16:33:38.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540e9218:::benchmark_data_smithi188_113957_object104808:head# 2024-04-23T16:33:38.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540efb7e:::benchmark_data_smithi188_113957_object60784:head# 2024-04-23T16:33:38.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540fd7ae:::benchmark_data_smithi188_113957_object24861:head# 2024-04-23T16:33:38.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:540fde04:::benchmark_data_smithi188_113957_object81596:head# 2024-04-23T16:33:38.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5410023c:::benchmark_data_smithi188_113957_object111397:head# 2024-04-23T16:33:38.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5410b1ca:::benchmark_data_smithi188_113957_object95362:head# 2024-04-23T16:33:38.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541153c5:::benchmark_data_smithi188_113957_object31065:head# 2024-04-23T16:33:38.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54121aca:::benchmark_data_smithi188_113957_object59837:head# 2024-04-23T16:33:38.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5412369f:::benchmark_data_smithi188_113957_object88629:head# 2024-04-23T16:33:38.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5412c223:::benchmark_data_smithi188_113957_object97549:head# 2024-04-23T16:33:38.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54130019:::benchmark_data_smithi188_113957_object52890:head# 2024-04-23T16:33:38.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5413098e:::benchmark_data_smithi188_113957_object114185:head# 2024-04-23T16:33:38.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54136255:::benchmark_data_smithi188_113957_object78470:head# 2024-04-23T16:33:38.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541469c9:::benchmark_data_smithi188_113957_object78165:head# 2024-04-23T16:33:38.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5414e889:::benchmark_data_smithi188_113957_object148007:head# 2024-04-23T16:33:38.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54154607:::benchmark_data_smithi188_113957_object63544:head# 2024-04-23T16:33:38.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54175af9:::benchmark_data_smithi188_113957_object106141:head# 2024-04-23T16:33:38.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54176973:::benchmark_data_smithi188_113957_object14248:head# 2024-04-23T16:33:38.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54178271:::benchmark_data_smithi188_113957_object32867:head# 2024-04-23T16:33:38.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5417966a:::benchmark_data_smithi188_113957_object117668:head# 2024-04-23T16:33:38.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5417b6c6:::benchmark_data_smithi188_113957_object112920:head# 2024-04-23T16:33:38.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5417cf44:::benchmark_data_smithi188_113957_object86865:head# 2024-04-23T16:33:38.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541815eb:::benchmark_data_smithi188_113957_object110458:head# 2024-04-23T16:33:38.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541945f9:::benchmark_data_smithi188_113957_object100647:head# 2024-04-23T16:33:38.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54196ec5:::benchmark_data_smithi188_113957_object122101:head# 2024-04-23T16:33:38.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541a1b0b:::benchmark_data_smithi188_113957_object8534:head# 2024-04-23T16:33:38.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541a416a:::benchmark_data_smithi188_113957_object90195:head# 2024-04-23T16:33:38.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541a872c:::benchmark_data_smithi188_113957_object89245:head# 2024-04-23T16:33:38.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541ae0e1:::benchmark_data_smithi188_113957_object13059:head# 2024-04-23T16:33:38.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541af46a:::benchmark_data_smithi188_113957_object69493:head# 2024-04-23T16:33:38.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541af990:::benchmark_data_smithi188_113957_object128162:head# 2024-04-23T16:33:38.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541b039a:::benchmark_data_smithi188_113957_object58837:head# 2024-04-23T16:33:38.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541b3b57:::benchmark_data_smithi188_113957_object70619:head# 2024-04-23T16:33:38.814 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541bb0aa:::benchmark_data_smithi188_113957_object103076:head# 2024-04-23T16:33:38.814 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541c2014:::benchmark_data_smithi188_113957_object81721:head# 2024-04-23T16:33:38.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541c6374:::benchmark_data_smithi188_113957_object107445:head# 2024-04-23T16:33:38.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541e81c9:::benchmark_data_smithi188_113957_object46460:head# 2024-04-23T16:33:38.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541ed1af:::benchmark_data_smithi188_113957_object142126:head# 2024-04-23T16:33:38.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541f0c71:::benchmark_data_smithi188_113957_object133232:head# 2024-04-23T16:33:38.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541f3525:::benchmark_data_smithi188_113957_object128091:head# 2024-04-23T16:33:38.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541f597b:::benchmark_data_smithi188_113957_object30197:head# 2024-04-23T16:33:38.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541f8949:::benchmark_data_smithi188_113957_object47223:head# 2024-04-23T16:33:38.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541f9e3e:::benchmark_data_smithi188_113957_object117667:head# 2024-04-23T16:33:38.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541faf59:::benchmark_data_smithi188_113957_object105538:head# 2024-04-23T16:33:38.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541fb3f1:::benchmark_data_smithi188_113957_object67542:head# 2024-04-23T16:33:38.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541fe30e:::benchmark_data_smithi188_113957_object9156:head# 2024-04-23T16:33:38.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541ff0e0:::benchmark_data_smithi188_113957_object72171:head# 2024-04-23T16:33:38.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:541ff31f:::benchmark_data_smithi188_113957_object145126:head# 2024-04-23T16:33:38.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54208c91:::benchmark_data_smithi188_113957_object38317:head# 2024-04-23T16:33:38.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5420a1d3:::benchmark_data_smithi188_113957_object139234:head# 2024-04-23T16:33:38.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:542179a7:::benchmark_data_smithi188_113957_object57277:head# 2024-04-23T16:33:38.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54222c79:::benchmark_data_smithi188_113957_object44414:head# 2024-04-23T16:33:38.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54229ad0:::benchmark_data_smithi188_113957_object150447:head# 2024-04-23T16:33:38.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5422b98e:::benchmark_data_smithi188_113957_object30035:head# 2024-04-23T16:33:38.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5422ef02:::benchmark_data_smithi188_113957_object130560:head# 2024-04-23T16:33:38.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54231bd7:::benchmark_data_smithi188_113957_object135759:head# 2024-04-23T16:33:38.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54237a9a:::benchmark_data_smithi188_113957_object151721:head# 2024-04-23T16:33:38.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5423b03b:::benchmark_data_smithi188_113957_object28907:head# 2024-04-23T16:33:38.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54256fbe:::benchmark_data_smithi188_113957_object45599:head# 2024-04-23T16:33:38.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54265bd7:::benchmark_data_smithi188_113957_object25977:head# 2024-04-23T16:33:38.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:542742af:::benchmark_data_smithi188_113957_object98208:head# 2024-04-23T16:33:38.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5427679c:::benchmark_data_smithi188_113957_object79486:head# 2024-04-23T16:33:38.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5427ac03:::benchmark_data_smithi188_113957_object553:head# 2024-04-23T16:33:38.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5428062f:::benchmark_data_smithi188_113957_object76893:head# 2024-04-23T16:33:38.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:542813a9:::benchmark_data_smithi188_113957_object146263:head# 2024-04-23T16:33:38.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54281959:::benchmark_data_smithi188_113957_object141869:head# 2024-04-23T16:33:38.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54286625:::benchmark_data_smithi188_113957_object100619:head# 2024-04-23T16:33:38.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54296b2b:::benchmark_data_smithi188_113957_object2365:head# 2024-04-23T16:33:38.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54298d7d:::benchmark_data_smithi188_113957_object1607:head# 2024-04-23T16:33:38.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:542a03c4:::benchmark_data_smithi188_113957_object111886:head# 2024-04-23T16:33:38.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:542a5db4:::benchmark_data_smithi188_113957_object35567:head# 2024-04-23T16:33:38.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:542a7680:::benchmark_data_smithi188_113957_object104113:head# 2024-04-23T16:33:38.832 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:542a792b:::benchmark_data_smithi188_113957_object127899:head# 2024-04-23T16:33:38.832 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:542ad384:::benchmark_data_smithi188_113957_object63852:head# 2024-04-23T16:33:38.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:542add16:::benchmark_data_smithi188_113957_object71040:head# 2024-04-23T16:33:38.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:542c0cd8:::benchmark_data_smithi188_113957_object116655:head# 2024-04-23T16:33:38.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:542d967e:::benchmark_data_smithi188_113957_object2894:head# 2024-04-23T16:33:38.834 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:542dc8b9:::benchmark_data_smithi188_113957_object85184:head# 2024-04-23T16:33:38.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:542f24dd:::benchmark_data_smithi188_113957_object103128:head# 2024-04-23T16:33:38.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:542f2eb5:::benchmark_data_smithi188_113957_object112208:head# 2024-04-23T16:33:38.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:542f6eb9:::benchmark_data_smithi188_113957_object68953:head# 2024-04-23T16:33:38.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:542f75d2:::benchmark_data_smithi188_113957_object98881:head# 2024-04-23T16:33:38.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:542f97a8:::benchmark_data_smithi188_113957_object84962:head# 2024-04-23T16:33:38.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5430674b:::benchmark_data_smithi188_113957_object12680:head# 2024-04-23T16:33:38.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54308a58:::benchmark_data_smithi188_113957_object3650:head# 2024-04-23T16:33:38.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5430d755:::benchmark_data_smithi188_113957_object65712:head# 2024-04-23T16:33:38.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5430f7a0:::benchmark_data_smithi188_113957_object110858:head# 2024-04-23T16:33:38.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54313e22:::benchmark_data_smithi188_113957_object92644:head# 2024-04-23T16:33:38.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:543201d5:::benchmark_data_smithi188_113957_object16467:head# 2024-04-23T16:33:38.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54321bfc:::benchmark_data_smithi188_113957_object73912:head# 2024-04-23T16:33:38.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54321de8:::benchmark_data_smithi188_113957_object13088:head# 2024-04-23T16:33:38.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54325201:::benchmark_data_smithi188_113957_object91832:head# 2024-04-23T16:33:38.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5432ac4b:::benchmark_data_smithi188_113957_object94045:head# 2024-04-23T16:33:38.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5432f537:::benchmark_data_smithi188_113957_object68525:head# 2024-04-23T16:33:38.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:543315cd:::benchmark_data_smithi188_113957_object142789:head# 2024-04-23T16:33:38.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5433e76c:::benchmark_data_smithi188_113957_object125461:head# 2024-04-23T16:33:38.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:543528c0:::benchmark_data_smithi188_113957_object66126:head# 2024-04-23T16:33:38.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:543665a3:::benchmark_data_smithi188_113957_object540:head# 2024-04-23T16:33:38.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54379160:::benchmark_data_smithi188_113957_object136652:head# 2024-04-23T16:33:38.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5437c752:::benchmark_data_smithi188_113957_object110471:head# 2024-04-23T16:33:38.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5437e5e3:::benchmark_data_smithi188_113957_object117674:head# 2024-04-23T16:33:38.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54383bcf:::benchmark_data_smithi188_113957_object111740:head# 2024-04-23T16:33:38.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54386304:::benchmark_data_smithi188_113957_object57803:head# 2024-04-23T16:33:38.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54391b51:::benchmark_data_smithi188_113957_object115938:head# 2024-04-23T16:33:38.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54395f46:::benchmark_data_smithi188_113957_object147398:head# 2024-04-23T16:33:38.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54397e9a:::benchmark_data_smithi188_113957_object121112:head# 2024-04-23T16:33:38.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:543a33fd:::benchmark_data_smithi188_113957_object58827:head# 2024-04-23T16:33:38.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:543ad8bb:::benchmark_data_smithi188_113957_object96850:head# 2024-04-23T16:33:38.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:543afb31:::benchmark_data_smithi188_113957_object103400:head# 2024-04-23T16:33:38.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:543be64d:::benchmark_data_smithi188_113957_object36099:head# 2024-04-23T16:33:38.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:543beb01:::benchmark_data_smithi188_113957_object76061:head# 2024-04-23T16:33:38.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:543c0c43:::benchmark_data_smithi188_113957_object151195:head# 2024-04-23T16:33:38.849 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:38.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:543c2edc:::benchmark_data_smithi188_113957_object72328:head# 2024-04-23T16:33:38.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:543cd217:::benchmark_data_smithi188_113957_object98102:head# 2024-04-23T16:33:38.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:543ce81f:::benchmark_data_smithi188_113957_object58908:head# 2024-04-23T16:33:38.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:543d76e5:::benchmark_data_smithi188_113957_object152878:head# 2024-04-23T16:33:38.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:543dc5aa:::benchmark_data_smithi188_113957_object98952:head# 2024-04-23T16:33:38.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:543f2b7d:::benchmark_data_smithi188_113957_object1719:head# 2024-04-23T16:33:38.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:543fb63c:::benchmark_data_smithi188_113957_object104481:head# 2024-04-23T16:33:38.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54400f78:::benchmark_data_smithi188_113957_object53128:head# 2024-04-23T16:33:38.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544062e1:::benchmark_data_smithi188_113957_object44282:head# 2024-04-23T16:33:38.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5440f1a2:::benchmark_data_smithi188_113957_object60324:head# 2024-04-23T16:33:38.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5441b75e:::benchmark_data_smithi188_113957_object139851:head# 2024-04-23T16:33:38.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5441b8b3:::benchmark_data_smithi188_113957_object41385:head# 2024-04-23T16:33:38.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5441c40f:::benchmark_data_smithi188_113957_object137590:head# 2024-04-23T16:33:38.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544253db:::benchmark_data_smithi188_113957_object30301:head# 2024-04-23T16:33:38.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5442ec12:::benchmark_data_smithi188_113957_object2184:head# 2024-04-23T16:33:38.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54433fd1:::benchmark_data_smithi188_113957_object124240:head# 2024-04-23T16:33:38.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5443885f:::benchmark_data_smithi188_113957_object107914:head# 2024-04-23T16:33:38.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5443a60e:::benchmark_data_smithi188_113957_object141879:head# 2024-04-23T16:33:38.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5443c5ca:::benchmark_data_smithi188_113957_object100381:head# 2024-04-23T16:33:38.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54442fdc:::benchmark_data_smithi188_113957_object79353:head# 2024-04-23T16:33:38.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54466d75:::benchmark_data_smithi188_113957_object103406:head# 2024-04-23T16:33:38.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544733dd:::benchmark_data_smithi188_113957_object125478:head# 2024-04-23T16:33:38.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5447e4c6:::benchmark_data_smithi188_113957_object32495:head# 2024-04-23T16:33:38.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5448734f:::benchmark_data_smithi188_113957_object10842:head# 2024-04-23T16:33:38.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544973ef:::benchmark_data_smithi188_113957_object85119:head# 2024-04-23T16:33:38.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544a3cba:::benchmark_data_smithi188_113957_object139109:head# 2024-04-23T16:33:38.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544ab906:::benchmark_data_smithi188_113957_object125647:head# 2024-04-23T16:33:38.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544b22e5:::benchmark_data_smithi188_113957_object130665:head# 2024-04-23T16:33:38.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544b466c:::benchmark_data_smithi188_113957_object89865:head# 2024-04-23T16:33:38.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544b56ea:::benchmark_data_smithi188_113957_object65036:head# 2024-04-23T16:33:38.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544c0ebe:::benchmark_data_smithi188_113957_object110522:head# 2024-04-23T16:33:38.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544c4be2:::benchmark_data_smithi188_113957_object145993:head# 2024-04-23T16:33:38.869 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:33:38.869 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2024-04-23T16:33:38.869 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2024-04-23T16:33:38.869 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:33:38.869 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.869 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:246025 3.a 3:5c8509c6:::benchmark_data_smithi188_89276_object77381:head [delete] snapc 0=[] ondisk+write+known_if_redirected e251)", 2024-04-23T16:33:38.869 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:25:39.979578+0000", 2024-04-23T16:33:38.869 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 478.88169291600002, 2024-04-23T16:33:38.869 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.080820345000000002, 2024-04-23T16:33:38.870 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:38.870 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:38.870 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:38.870 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:38.870 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:33:38.870 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:33:38.870 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 246025 2024-04-23T16:33:38.870 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.870 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:38.870 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.870 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:38.870 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:39.979578+0000", 2024-04-23T16:33:38.870 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.871 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.871 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.871 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:38.871 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:39.979578+0000", 2024-04-23T16:33:38.871 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.871 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.871 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.871 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:38.871 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:39.979579+0000", 2024-04-23T16:33:38.871 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7099999999999999e-06 2024-04-23T16:33:38.871 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.871 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.872 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:38.872 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:39.979585+0000", 2024-04-23T16:33:38.872 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7409999999999998e-06 2024-04-23T16:33:38.872 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.872 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.872 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:38.872 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:39.979595+0000", 2024-04-23T16:33:38.872 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0317e-05 2024-04-23T16:33:38.872 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.873 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.873 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:38.873 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:39.979602+0000", 2024-04-23T16:33:38.873 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0609999999999998e-06 2024-04-23T16:33:38.873 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.873 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.873 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:38.873 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:39.979649+0000", 2024-04-23T16:33:38.873 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6978999999999999e-05 2024-04-23T16:33:38.873 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.873 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.873 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:38.874 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:39.979726+0000", 2024-04-23T16:33:38.874 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7154e-05 2024-04-23T16:33:38.874 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.874 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.874 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:38.874 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:39.979792+0000", 2024-04-23T16:33:38.874 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5080999999999996e-05 2024-04-23T16:33:38.874 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.874 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.874 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:38.874 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:39.983267+0000", 2024-04-23T16:33:38.875 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0034758749999999998 2024-04-23T16:33:38.875 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.875 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.875 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:38.875 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:40.060286+0000", 2024-04-23T16:33:38.876 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.077018141999999998 2024-04-23T16:33:38.876 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.877 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.877 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:38.877 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:40.060344+0000", 2024-04-23T16:33:38.877 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8820999999999998e-05 2024-04-23T16:33:38.877 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.877 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.878 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:38.878 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:25:40.060398+0000", 2024-04-23T16:33:38.878 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3464000000000002e-05 2024-04-23T16:33:38.878 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.878 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:38.878 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.878 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.878 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.878 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:70258 4.5 4:ae7ceeba:::benchmark_data_smithi188_101370_object70257:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e288)", 2024-04-23T16:33:38.879 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:38.898686+0000", 2024-04-23T16:33:38.879 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 359.96258485999999, 2024-04-23T16:33:38.879 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.074931969000000001, 2024-04-23T16:33:38.879 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:38.879 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:38.879 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:38.879 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:38.879 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:33:38.879 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:33:38.879 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 70258 2024-04-23T16:33:38.880 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.880 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:38.880 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.880 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:38.880 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.898686+0000", 2024-04-23T16:33:38.882 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.883 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.883 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.883 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:38.883 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.898686+0000", 2024-04-23T16:33:38.884 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.884 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.884 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.884 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:38.884 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.898688+0000", 2024-04-23T16:33:38.885 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.979e-06 2024-04-23T16:33:38.885 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.885 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.885 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:38.885 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.898740+0000", 2024-04-23T16:33:38.886 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2045000000000001e-05 2024-04-23T16:33:38.886 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.886 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.886 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:38.886 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.898749+0000", 2024-04-23T16:33:38.886 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7599999999999997e-06 2024-04-23T16:33:38.886 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.886 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.886 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:38.886 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.898757+0000", 2024-04-23T16:33:38.886 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4900000000000003e-06 2024-04-23T16:33:38.886 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.887 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.887 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:38.887 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.900642+0000", 2024-04-23T16:33:38.887 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0018855300000000001 2024-04-23T16:33:38.887 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.887 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.887 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:38.887 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.900757+0000", 2024-04-23T16:33:38.887 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000114824 2024-04-23T16:33:38.887 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.887 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.887 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:33:38.887 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.900890+0000", 2024-04-23T16:33:38.888 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013328099999999999 2024-04-23T16:33:38.888 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.888 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.888 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:38.888 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.906685+0000", 2024-04-23T16:33:38.888 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0057942200000000001 2024-04-23T16:33:38.888 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.888 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.888 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:38.888 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.973552+0000", 2024-04-23T16:33:38.888 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.066867001999999995 2024-04-23T16:33:38.888 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.888 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.889 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:38.889 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.973585+0000", 2024-04-23T16:33:38.889 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2818000000000003e-05 2024-04-23T16:33:38.889 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.889 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.889 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:38.889 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.973618+0000", 2024-04-23T16:33:38.889 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3019999999999999e-05 2024-04-23T16:33:38.889 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.889 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:38.889 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.889 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.890 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.890 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:70261 4.5 4:a9238a77:::benchmark_data_smithi188_101370_object70260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e288)", 2024-04-23T16:33:38.890 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:38.899453+0000", 2024-04-23T16:33:38.890 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 359.96181766900003, 2024-04-23T16:33:38.890 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.074320675000000003, 2024-04-23T16:33:38.890 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:38.890 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:38.890 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:38.890 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:38.890 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:33:38.890 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:33:38.890 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 70261 2024-04-23T16:33:38.890 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.890 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:38.891 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.891 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:38.891 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.899453+0000", 2024-04-23T16:33:38.891 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.891 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.891 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.891 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:38.891 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.899453+0000", 2024-04-23T16:33:38.891 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.891 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.891 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.891 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:38.891 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.899454+0000", 2024-04-23T16:33:38.892 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.437e-06 2024-04-23T16:33:38.892 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.892 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.892 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:38.892 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.899508+0000", 2024-04-23T16:33:38.892 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3582999999999998e-05 2024-04-23T16:33:38.892 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.892 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.892 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:38.892 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.899520+0000", 2024-04-23T16:33:38.892 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1950000000000001e-05 2024-04-23T16:33:38.892 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.892 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.892 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:38.893 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.899527+0000", 2024-04-23T16:33:38.893 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.396e-06 2024-04-23T16:33:38.893 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.893 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.893 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:38.893 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.902591+0000", 2024-04-23T16:33:38.893 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0030636909999999999 2024-04-23T16:33:38.893 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.893 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.894 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:38.894 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.902689+0000", 2024-04-23T16:33:38.894 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7959000000000002e-05 2024-04-23T16:33:38.894 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.894 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.894 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:33:38.894 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.902819+0000", 2024-04-23T16:33:38.894 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012998 2024-04-23T16:33:38.894 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.894 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.894 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:38.895 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.906689+0000", 2024-04-23T16:33:38.895 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0038702519999999998 2024-04-23T16:33:38.895 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.895 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.895 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:38.895 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.973673+0000", 2024-04-23T16:33:38.895 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.066984112999999998 2024-04-23T16:33:38.895 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.895 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.895 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:38.895 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.973731+0000", 2024-04-23T16:33:38.895 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7683000000000003e-05 2024-04-23T16:33:38.895 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.896 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.896 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:38.896 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.973773+0000", 2024-04-23T16:33:38.896 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2630999999999999e-05 2024-04-23T16:33:38.896 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.896 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:38.896 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.896 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.896 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.896 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:70264 4.5 4:a69a5e6e:::benchmark_data_smithi188_101370_object70263:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e288)", 2024-04-23T16:33:38.896 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:38.902021+0000", 2024-04-23T16:33:38.896 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 359.95924896700001, 2024-04-23T16:33:38.896 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.071839927999999997, 2024-04-23T16:33:38.896 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:38.897 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:38.897 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:38.897 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:38.897 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:33:38.897 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:33:38.897 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 70264 2024-04-23T16:33:38.897 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.897 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:38.897 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.897 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:38.897 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.902021+0000", 2024-04-23T16:33:38.897 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.897 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.897 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.898 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:38.898 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.902021+0000", 2024-04-23T16:33:38.898 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.898 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.898 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.898 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:38.898 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.902023+0000", 2024-04-23T16:33:38.898 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2729999999999999e-06 2024-04-23T16:33:38.898 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.898 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.898 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:38.899 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.902052+0000", 2024-04-23T16:33:38.899 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9053000000000001e-05 2024-04-23T16:33:38.899 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.899 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.899 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:38.899 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.902060+0000", 2024-04-23T16:33:38.899 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2500000000000006e-06 2024-04-23T16:33:38.899 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.899 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.899 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:38.899 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.902066+0000", 2024-04-23T16:33:38.899 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5790000000000003e-06 2024-04-23T16:33:38.900 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.900 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.900 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:38.900 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.904452+0000", 2024-04-23T16:33:38.900 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0023865090000000002 2024-04-23T16:33:38.900 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.900 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.900 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:38.900 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.904561+0000", 2024-04-23T16:33:38.900 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010875299999999999 2024-04-23T16:33:38.900 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.900 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.900 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:33:38.900 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.904694+0000", 2024-04-23T16:33:38.901 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000132743 2024-04-23T16:33:38.901 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.901 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.901 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:38.901 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.908497+0000", 2024-04-23T16:33:38.901 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0038033229999999999 2024-04-23T16:33:38.901 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.901 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.901 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:38.901 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.973808+0000", 2024-04-23T16:33:38.901 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.065311548999999997 2024-04-23T16:33:38.901 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.901 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.901 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:38.902 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.973837+0000", 2024-04-23T16:33:38.902 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8445999999999998e-05 2024-04-23T16:33:38.902 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.902 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.902 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:38.902 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.973861+0000", 2024-04-23T16:33:38.902 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4450000000000001e-05 2024-04-23T16:33:38.902 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.902 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:38.902 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.902 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.903 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.903 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:70270 4.5 4:a2c5ee34:::benchmark_data_smithi188_101370_object70269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e288)", 2024-04-23T16:33:38.903 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:38.905527+0000", 2024-04-23T16:33:38.903 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 359.95574349200001, 2024-04-23T16:33:38.903 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.072635020999999994, 2024-04-23T16:33:38.903 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:38.903 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:38.904 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:38.904 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:38.904 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:33:38.904 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:33:38.904 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 70270 2024-04-23T16:33:38.905 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.905 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:38.905 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.905 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:38.905 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.905527+0000", 2024-04-23T16:33:38.905 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.905 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.905 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.905 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:38.905 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.905527+0000", 2024-04-23T16:33:38.905 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.906 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.906 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.906 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:38.906 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.905528+0000", 2024-04-23T16:33:38.906 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4799999999999997e-07 2024-04-23T16:33:38.906 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.906 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.906 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:38.906 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.905597+0000", 2024-04-23T16:33:38.906 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9469999999999997e-05 2024-04-23T16:33:38.906 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.906 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.907 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:38.907 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.905605+0000", 2024-04-23T16:33:38.907 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7680000000000008e-06 2024-04-23T16:33:38.907 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.907 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.907 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:38.907 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.905615+0000", 2024-04-23T16:33:38.907 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9369999999999999e-06 2024-04-23T16:33:38.907 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.907 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.907 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:38.907 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.906709+0000", 2024-04-23T16:33:38.908 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0010942910000000001 2024-04-23T16:33:38.908 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.908 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.908 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:38.908 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.906783+0000", 2024-04-23T16:33:38.908 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3764999999999996e-05 2024-04-23T16:33:38.908 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.908 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.908 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:33:38.908 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.906860+0000", 2024-04-23T16:33:38.908 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6917999999999998e-05 2024-04-23T16:33:38.909 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.909 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.909 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:38.909 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.908591+0000", 2024-04-23T16:33:38.909 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0017309789999999999 2024-04-23T16:33:38.909 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.909 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.909 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:38.909 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.978065+0000", 2024-04-23T16:33:38.909 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.069474525999999995 2024-04-23T16:33:38.909 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.909 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.910 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:38.910 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.978121+0000", 2024-04-23T16:33:38.910 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5643000000000002e-05 2024-04-23T16:33:38.910 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.910 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.910 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:38.910 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.978162+0000", 2024-04-23T16:33:38.910 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0976000000000002e-05 2024-04-23T16:33:38.910 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.910 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:38.910 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.910 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.911 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.911 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:72050 4.5 4:a40c3676:::benchmark_data_smithi188_101370_object72049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e289)", 2024-04-23T16:33:38.911 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:39.744552+0000", 2024-04-23T16:33:38.911 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 359.11671819200001, 2024-04-23T16:33:38.911 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.078171600999999993, 2024-04-23T16:33:38.911 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:38.911 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:38.911 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:38.912 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:38.912 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:33:38.912 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:33:38.912 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 72050 2024-04-23T16:33:38.912 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.912 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:38.913 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.913 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:38.914 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.744552+0000", 2024-04-23T16:33:38.914 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.914 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.914 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.914 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:38.914 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.744552+0000", 2024-04-23T16:33:38.914 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.914 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.914 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.914 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:38.915 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.744554+0000", 2024-04-23T16:33:38.915 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7889999999999999e-06 2024-04-23T16:33:38.915 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.915 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.915 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:38.915 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.744598+0000", 2024-04-23T16:33:38.915 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4375000000000001e-05 2024-04-23T16:33:38.915 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.915 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.915 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:38.915 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.744608+0000", 2024-04-23T16:33:38.915 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9899999999999992e-06 2024-04-23T16:33:38.916 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.916 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.916 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:38.916 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.744614+0000", 2024-04-23T16:33:38.916 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6300000000000003e-06 2024-04-23T16:33:38.916 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.916 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.916 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:38.916 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.744661+0000", 2024-04-23T16:33:38.917 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6709000000000003e-05 2024-04-23T16:33:38.917 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.917 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.917 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:38.917 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.744757+0000", 2024-04-23T16:33:38.917 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6151999999999997e-05 2024-04-23T16:33:38.917 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.918 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.918 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:33:38.918 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.744855+0000", 2024-04-23T16:33:38.918 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7646999999999995e-05 2024-04-23T16:33:38.918 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.918 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.918 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:38.918 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.746687+0000", 2024-04-23T16:33:38.919 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0018320949999999999 2024-04-23T16:33:38.919 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.919 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.919 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:38.919 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.822629+0000", 2024-04-23T16:33:38.919 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.075942280000000001 2024-04-23T16:33:38.919 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.919 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.919 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:38.919 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.822686+0000", 2024-04-23T16:33:38.920 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6610999999999997e-05 2024-04-23T16:33:38.920 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.920 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.920 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:38.920 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.822724+0000", 2024-04-23T16:33:38.920 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8322999999999999e-05 2024-04-23T16:33:38.920 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.920 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:38.920 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.920 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.921 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.921 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.15486.0:105198 4.6 e295/293 4:623868bc:::benchmark_data_smithi188_101370_object105197:head v 295'6627, mlcod=281'131)", 2024-04-23T16:33:38.921 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:28:00.277692+0000", 2024-04-23T16:33:38.921 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 338.583578445, 2024-04-23T16:33:38.921 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.066640286000000007, 2024-04-23T16:33:38.921 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:38.921 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:38.921 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:38.921 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:38.922 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.922 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:38.922 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.277692+0000", 2024-04-23T16:33:38.922 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.922 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.922 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.922 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:38.923 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.277692+0000", 2024-04-23T16:33:38.923 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.923 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.923 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.923 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:38.924 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.277694+0000", 2024-04-23T16:33:38.924 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8109999999999999e-06 2024-04-23T16:33:38.924 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.924 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.924 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:38.924 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.277751+0000", 2024-04-23T16:33:38.925 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7343999999999998e-05 2024-04-23T16:33:38.925 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.925 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.925 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:38.925 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.277761+0000", 2024-04-23T16:33:38.925 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0267e-05 2024-04-23T16:33:38.925 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.925 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.925 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:38.925 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.277769+0000", 2024-04-23T16:33:38.925 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5229999999999998e-06 2024-04-23T16:33:38.926 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.926 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.926 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:38.926 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.324380+0000", 2024-04-23T16:33:38.926 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.046610780999999997 2024-04-23T16:33:38.926 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.926 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.926 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:38.926 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.324402+0000", 2024-04-23T16:33:38.927 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2171000000000002e-05 2024-04-23T16:33:38.927 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.927 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.927 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:38.927 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.344288+0000", 2024-04-23T16:33:38.927 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019885968 2024-04-23T16:33:38.927 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.927 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.927 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:38.927 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.344332+0000", 2024-04-23T16:33:38.928 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4421e-05 2024-04-23T16:33:38.928 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.928 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:38.928 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.928 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.928 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.928 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPush(4.6 295/293 [PushOp(4:6d6703c0:::benchmark_data_smithi188_101370_object2146:head, version: 281'134, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6d6703c0:::benchmark_data_smithi188_101370_object2146:head@281'134, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T16:33:38.928 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:28:00.311233+0000", 2024-04-23T16:33:38.928 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 338.55003702400001, 2024-04-23T16:33:38.929 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.111358305, 2024-04-23T16:33:38.929 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:38.929 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:38.929 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:33:38.929 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:38.929 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.929 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:38.929 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.311233+0000", 2024-04-23T16:33:38.929 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.929 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.929 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.929 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:38.930 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.311233+0000", 2024-04-23T16:33:38.930 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.930 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.930 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.930 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:38.930 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.311235+0000", 2024-04-23T16:33:38.930 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5009999999999999e-06 2024-04-23T16:33:38.930 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.930 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.930 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:38.930 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.311285+0000", 2024-04-23T16:33:38.931 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0590999999999997e-05 2024-04-23T16:33:38.931 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.931 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.931 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:38.931 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.311305+0000", 2024-04-23T16:33:38.931 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9247000000000001e-05 2024-04-23T16:33:38.931 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.931 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.931 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:38.931 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.311321+0000", 2024-04-23T16:33:38.931 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6240999999999999e-05 2024-04-23T16:33:38.931 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.932 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.932 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:38.932 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.420586+0000", 2024-04-23T16:33:38.932 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.10926459299999999 2024-04-23T16:33:38.932 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.932 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.932 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:38.932 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.420594+0000", 2024-04-23T16:33:38.932 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1070000000000005e-06 2024-04-23T16:33:38.932 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.933 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.933 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:38.933 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.422592+0000", 2024-04-23T16:33:38.933 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0019980250000000001 2024-04-23T16:33:38.933 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.933 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:38.933 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.933 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.933 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.933 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPush(4.6 297/293 [PushOp(4:6b111983:::benchmark_data_smithi188_101370_object4279:head, version: 281'273, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6b111983:::benchmark_data_smithi188_101370_object4279:head@281'273, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T16:33:38.934 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:28:05.437696+0000", 2024-04-23T16:33:38.934 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 333.42357455400003, 2024-04-23T16:33:38.934 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.068626678999999996, 2024-04-23T16:33:38.934 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:38.934 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:38.934 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:33:38.934 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:38.934 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.934 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:38.935 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:05.437696+0000", 2024-04-23T16:33:38.935 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.935 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.935 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.935 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:38.935 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:05.437696+0000", 2024-04-23T16:33:38.935 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.935 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.935 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.935 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:38.936 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:05.437697+0000", 2024-04-23T16:33:38.936 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7899999999999997e-07 2024-04-23T16:33:38.936 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.936 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.936 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:38.936 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:05.437749+0000", 2024-04-23T16:33:38.936 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1990999999999997e-05 2024-04-23T16:33:38.936 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.936 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.936 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:38.936 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:05.437769+0000", 2024-04-23T16:33:38.936 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0409000000000001e-05 2024-04-23T16:33:38.937 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.937 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.937 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:38.937 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:05.437787+0000", 2024-04-23T16:33:38.937 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8321e-05 2024-04-23T16:33:38.937 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.937 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.937 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:38.937 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:05.504146+0000", 2024-04-23T16:33:38.937 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.066358640999999996 2024-04-23T16:33:38.937 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.937 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.938 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:38.938 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:05.504164+0000", 2024-04-23T16:33:38.938 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.787e-05 2024-04-23T16:33:38.938 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.938 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.938 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:38.938 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:05.506323+0000", 2024-04-23T16:33:38.938 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021586679999999999 2024-04-23T16:33:38.938 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.938 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:38.939 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.939 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.939 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.939 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPush(4.6 298/293 [PushOp(4:64d5b26c:::benchmark_data_smithi188_101370_object5480:head, version: 281'356, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:64d5b26c:::benchmark_data_smithi188_101370_object5480:head@281'356, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T16:33:38.939 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:28:08.553825+0000", 2024-04-23T16:33:38.939 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 330.30744529999998, 2024-04-23T16:33:38.939 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.073229198999999995, 2024-04-23T16:33:38.939 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:38.939 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:38.939 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:33:38.939 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:38.939 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.940 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:38.940 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:08.553825+0000", 2024-04-23T16:33:38.940 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.940 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.940 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.940 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:38.940 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:08.553825+0000", 2024-04-23T16:33:38.940 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.940 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.940 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.940 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:38.940 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:08.553827+0000", 2024-04-23T16:33:38.940 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0420000000000001e-06 2024-04-23T16:33:38.941 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.941 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.941 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:38.941 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:08.553882+0000", 2024-04-23T16:33:38.941 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5133000000000002e-05 2024-04-23T16:33:38.941 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.941 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.941 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:38.941 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:08.553903+0000", 2024-04-23T16:33:38.941 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0823999999999999e-05 2024-04-23T16:33:38.941 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.941 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.941 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:38.942 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:08.553916+0000", 2024-04-23T16:33:38.942 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2645e-05 2024-04-23T16:33:38.942 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.942 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.942 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:38.942 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:08.624484+0000", 2024-04-23T16:33:38.942 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.070567932 2024-04-23T16:33:38.942 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.942 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.942 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:38.942 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:08.624505+0000", 2024-04-23T16:33:38.942 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1484999999999998e-05 2024-04-23T16:33:38.943 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.943 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.943 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:38.943 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:08.627054+0000", 2024-04-23T16:33:38.943 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025491379999999998 2024-04-23T16:33:38.943 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.943 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:38.943 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.943 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.943 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.943 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPush(4.6 311/293 [PushOp(4:6c219b0a:::benchmark_data_smithi188_101370_object16938:head, version: 282'1106, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6c219b0a:::benchmark_data_smithi188_101370_object16938:head@282'1106, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T16:33:38.944 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:28:34.443852+0000", 2024-04-23T16:33:38.944 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 304.417418914, 2024-04-23T16:33:38.944 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.10967393, 2024-04-23T16:33:38.944 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:38.944 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:38.944 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:33:38.944 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:38.944 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.944 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:38.944 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:34.443852+0000", 2024-04-23T16:33:38.944 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.945 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.945 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.945 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:38.945 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:34.443852+0000", 2024-04-23T16:33:38.945 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.945 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.945 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.945 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:38.945 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:34.443852+0000", 2024-04-23T16:33:38.945 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5300000000000002e-07 2024-04-23T16:33:38.945 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.945 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.946 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:38.946 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:34.443899+0000", 2024-04-23T16:33:38.946 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6904999999999999e-05 2024-04-23T16:33:38.946 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.946 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.946 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:38.946 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:34.443918+0000", 2024-04-23T16:33:38.946 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8258e-05 2024-04-23T16:33:38.946 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.946 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.947 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:38.947 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:34.443937+0000", 2024-04-23T16:33:38.947 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9046e-05 2024-04-23T16:33:38.947 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.947 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.947 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:38.947 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:34.538038+0000", 2024-04-23T16:33:38.947 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.094101406999999998 2024-04-23T16:33:38.947 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.947 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.947 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:38.947 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:34.538052+0000", 2024-04-23T16:33:38.948 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4122e-05 2024-04-23T16:33:38.948 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.948 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.948 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:38.948 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:34.553525+0000", 2024-04-23T16:33:38.948 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015473239 2024-04-23T16:33:38.948 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.948 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:38.948 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.948 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.948 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.949 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPush(4.6 314/293 [PushOp(4:6246f4d9:::benchmark_data_smithi188_101370_object20226:head, version: 283'1332, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6246f4d9:::benchmark_data_smithi188_101370_object20226:head@283'1332, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T16:33:38.949 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:28:41.952574+0000", 2024-04-23T16:33:38.949 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 296.90869627500001, 2024-04-23T16:33:38.949 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.097604073999999999, 2024-04-23T16:33:38.949 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:38.949 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:38.949 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:33:38.950 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:38.950 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.950 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:38.950 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:41.952574+0000", 2024-04-23T16:33:38.950 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.950 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.950 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.950 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:38.951 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:41.952574+0000", 2024-04-23T16:33:38.951 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.951 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.951 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.951 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:38.951 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:41.952575+0000", 2024-04-23T16:33:38.951 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.155e-06 2024-04-23T16:33:38.951 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.951 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.952 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:38.952 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:41.952622+0000", 2024-04-23T16:33:38.952 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6393000000000001e-05 2024-04-23T16:33:38.952 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.952 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.952 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:38.952 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:41.952643+0000", 2024-04-23T16:33:38.952 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1670000000000001e-05 2024-04-23T16:33:38.952 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.952 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.952 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:38.953 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:41.952659+0000", 2024-04-23T16:33:38.953 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5248e-05 2024-04-23T16:33:38.953 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.953 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.953 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:38.953 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.039224+0000", 2024-04-23T16:33:38.953 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.086565248999999997 2024-04-23T16:33:38.953 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.953 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.953 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:38.953 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.039237+0000", 2024-04-23T16:33:38.954 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2687000000000001e-05 2024-04-23T16:33:38.954 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.954 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.954 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:38.954 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.050178+0000", 2024-04-23T16:33:38.954 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010941672 2024-04-23T16:33:38.954 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.954 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:38.954 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.954 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.954 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.955 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPush(4.6 314/293 [PushOp(4:612d6aba:::benchmark_data_smithi188_101370_object20393:head, version: 283'1348, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:612d6aba:::benchmark_data_smithi188_101370_object20393:head@283'1348, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T16:33:38.955 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:28:42.683224+0000", 2024-04-23T16:33:38.955 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 296.17804690600002, 2024-04-23T16:33:38.955 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.072648131000000005, 2024-04-23T16:33:38.955 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:38.955 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:38.955 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:33:38.955 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:38.955 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.955 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:38.955 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.683224+0000", 2024-04-23T16:33:38.956 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.956 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.956 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.956 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:38.956 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.683224+0000", 2024-04-23T16:33:38.956 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.956 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.956 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.957 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:38.957 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.683224+0000", 2024-04-23T16:33:38.957 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8800000000000003e-07 2024-04-23T16:33:38.957 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.957 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.957 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:38.957 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.683284+0000", 2024-04-23T16:33:38.957 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0491000000000001e-05 2024-04-23T16:33:38.957 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.957 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.957 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:38.957 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.683301+0000", 2024-04-23T16:33:38.958 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6651999999999999e-05 2024-04-23T16:33:38.958 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.958 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.958 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:38.958 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.683315+0000", 2024-04-23T16:33:38.958 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3658e-05 2024-04-23T16:33:38.958 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.958 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.958 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:38.958 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.752838+0000", 2024-04-23T16:33:38.959 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.069522908999999994 2024-04-23T16:33:38.959 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.959 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.959 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:38.959 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.752851+0000", 2024-04-23T16:33:38.959 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3722e-05 2024-04-23T16:33:38.959 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.959 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.959 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:38.959 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.755872+0000", 2024-04-23T16:33:38.959 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0030202110000000001 2024-04-23T16:33:38.960 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.960 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:38.960 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.960 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.960 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.960 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:26106 5.2 5:461a31c8:::benchmark_data_smithi188_113957_object26105:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e369)", 2024-04-23T16:33:38.960 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:31:44.502808+0000", 2024-04-23T16:33:38.960 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 114.35846198900001, 2024-04-23T16:33:38.960 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.065193778999999993, 2024-04-23T16:33:38.960 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:38.960 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:38.961 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:38.961 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:38.961 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:33:38.961 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:38.961 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 26106 2024-04-23T16:33:38.961 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.961 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:38.961 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.961 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:38.961 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.502808+0000", 2024-04-23T16:33:38.961 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.961 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.961 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.962 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:38.962 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.502808+0000", 2024-04-23T16:33:38.962 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.962 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.962 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.962 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:38.962 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.502810+0000", 2024-04-23T16:33:38.962 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1739999999999999e-06 2024-04-23T16:33:38.962 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.962 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.962 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:38.963 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.503950+0000", 2024-04-23T16:33:38.963 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0011402719999999999 2024-04-23T16:33:38.963 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.963 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.963 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:38.963 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.503962+0000", 2024-04-23T16:33:38.963 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2483999999999999e-05 2024-04-23T16:33:38.963 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.963 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.963 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:38.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.503971+0000", 2024-04-23T16:33:38.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3950000000000001e-06 2024-04-23T16:33:38.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.964 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:38.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.504260+0000", 2024-04-23T16:33:38.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00028959999999999999 2024-04-23T16:33:38.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.964 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:38.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.504348+0000", 2024-04-23T16:33:38.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7793999999999993e-05 2024-04-23T16:33:38.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.965 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:33:38.965 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.504452+0000", 2024-04-23T16:33:38.965 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000104307 2024-04-23T16:33:38.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.965 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:38.965 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.507204+0000", 2024-04-23T16:33:38.965 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002751093 2024-04-23T16:33:38.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.966 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:38.966 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.567809+0000", 2024-04-23T16:33:38.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.060605277999999999 2024-04-23T16:33:38.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.966 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:38.966 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.567958+0000", 2024-04-23T16:33:38.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014895699999999999 2024-04-23T16:33:38.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.967 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.967 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:38.967 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.568002+0000", 2024-04-23T16:33:38.967 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4425000000000002e-05 2024-04-23T16:33:38.967 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.967 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:38.967 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.967 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.967 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.967 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:26117 5.2 5:47254ac5:::benchmark_data_smithi188_113957_object26116:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e369)", 2024-04-23T16:33:38.968 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:31:44.505370+0000", 2024-04-23T16:33:38.968 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 114.35589994999999, 2024-04-23T16:33:38.968 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.064646837999999998, 2024-04-23T16:33:38.968 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:38.968 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:38.968 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:38.968 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:38.968 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:33:38.968 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:38.968 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 26117 2024-04-23T16:33:38.968 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.969 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:38.969 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.969 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:38.969 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505370+0000", 2024-04-23T16:33:38.969 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.969 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.969 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.969 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:38.969 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505370+0000", 2024-04-23T16:33:38.969 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.969 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:38.970 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505372+0000", 2024-04-23T16:33:38.970 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4279999999999999e-06 2024-04-23T16:33:38.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:38.970 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505397+0000", 2024-04-23T16:33:38.970 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5253999999999999e-05 2024-04-23T16:33:38.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:38.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505405+0000", 2024-04-23T16:33:38.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1079999999999993e-06 2024-04-23T16:33:38.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:38.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505412+0000", 2024-04-23T16:33:38.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7819999999999996e-06 2024-04-23T16:33:38.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.972 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:38.972 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505472+0000", 2024-04-23T16:33:38.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0075000000000003e-05 2024-04-23T16:33:38.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.972 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:38.972 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505587+0000", 2024-04-23T16:33:38.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000115147 2024-04-23T16:33:38.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.972 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:33:38.973 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505706+0000", 2024-04-23T16:33:38.973 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000118366 2024-04-23T16:33:38.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.973 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:38.973 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.507946+0000", 2024-04-23T16:33:38.973 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0022402659999999999 2024-04-23T16:33:38.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.973 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:38.973 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.569916+0000", 2024-04-23T16:33:38.974 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.061970372000000003 2024-04-23T16:33:38.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:38.974 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.569976+0000", 2024-04-23T16:33:38.974 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9670000000000003e-05 2024-04-23T16:33:38.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:38.974 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.570017+0000", 2024-04-23T16:33:38.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1369999999999999e-05 2024-04-23T16:33:38.975 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.975 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:38.975 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.975 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:26124 5.2 5:4ffdf5ef:::benchmark_data_smithi188_113957_object26123:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e369)", 2024-04-23T16:33:38.975 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:31:44.505687+0000", 2024-04-23T16:33:38.975 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 114.355583291, 2024-04-23T16:33:38.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.064651243999999997, 2024-04-23T16:33:38.975 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:38.976 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:38.976 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:38.976 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:38.976 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:33:38.976 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:38.976 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 26124 2024-04-23T16:33:38.976 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.976 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:38.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:38.976 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505687+0000", 2024-04-23T16:33:38.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.977 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:38.977 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505687+0000", 2024-04-23T16:33:38.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.978 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.978 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:38.978 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505688+0000", 2024-04-23T16:33:38.978 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.341e-06 2024-04-23T16:33:38.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.978 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.978 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:38.978 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505779+0000", 2024-04-23T16:33:38.978 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0470000000000006e-05 2024-04-23T16:33:38.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.979 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.979 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:38.979 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505789+0000", 2024-04-23T16:33:38.979 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0052e-05 2024-04-23T16:33:38.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.979 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.979 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:38.979 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505797+0000", 2024-04-23T16:33:38.979 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5870000000000004e-06 2024-04-23T16:33:38.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.980 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.980 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:38.980 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.506467+0000", 2024-04-23T16:33:38.980 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00067056699999999995 2024-04-23T16:33:38.980 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.980 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.980 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:38.980 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.506580+0000", 2024-04-23T16:33:38.980 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000113348 2024-04-23T16:33:38.980 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.980 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.980 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:33:38.981 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.506721+0000", 2024-04-23T16:33:38.981 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014030999999999999 2024-04-23T16:33:38.981 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.981 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.981 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:38.981 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.507959+0000", 2024-04-23T16:33:38.981 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012383299999999999 2024-04-23T16:33:38.981 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.981 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.981 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:38.981 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.570235+0000", 2024-04-23T16:33:38.981 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.062276146999999997 2024-04-23T16:33:38.982 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.982 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.982 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:38.982 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.570291+0000", 2024-04-23T16:33:38.982 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5427000000000003e-05 2024-04-23T16:33:38.982 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.982 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.982 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:38.982 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.570338+0000", 2024-04-23T16:33:38.982 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7664999999999998e-05 2024-04-23T16:33:38.982 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.982 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:38.983 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.983 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.983 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.983 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:145609 5.3 e385/359 5:ce32bf88:::benchmark_data_smithi188_113957_object145608:head v 385'10187, mlcod=385'10182)", 2024-04-23T16:33:38.983 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:32:51.714951+0000", 2024-04-23T16:33:38.983 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 47.146319912999999, 2024-04-23T16:33:38.983 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052001789, 2024-04-23T16:33:38.983 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:38.983 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:38.983 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:38.983 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:38.983 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.984 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:38.984 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.714951+0000", 2024-04-23T16:33:38.984 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.984 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.984 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:38.984 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.714951+0000", 2024-04-23T16:33:38.984 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.984 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.984 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:38.985 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.714952+0000", 2024-04-23T16:33:38.985 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7540000000000001e-06 2024-04-23T16:33:38.985 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.985 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.985 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:38.985 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.715057+0000", 2024-04-23T16:33:38.985 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000104467 2024-04-23T16:33:38.985 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.985 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.985 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:38.985 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.715068+0000", 2024-04-23T16:33:38.985 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1758e-05 2024-04-23T16:33:38.985 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.986 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.986 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:38.986 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.715076+0000", 2024-04-23T16:33:38.986 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.926e-06 2024-04-23T16:33:38.986 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.986 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.986 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:38.986 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.717231+0000", 2024-04-23T16:33:38.986 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021546489999999998 2024-04-23T16:33:38.986 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.986 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:38.987 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.717248+0000", 2024-04-23T16:33:38.987 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7365000000000001e-05 2024-04-23T16:33:38.987 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:38.987 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.766907+0000", 2024-04-23T16:33:38.987 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.049658901999999998 2024-04-23T16:33:38.987 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:38.987 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.766952+0000", 2024-04-23T16:33:38.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4968e-05 2024-04-23T16:33:38.988 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.988 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:38.988 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.988 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:146469 5.2 5:432c4c0b:::benchmark_data_smithi188_113957_object146468:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-23T16:33:38.988 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:32:52.553527+0000", 2024-04-23T16:33:38.988 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 46.307743838, 2024-04-23T16:33:38.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.051413853000000002, 2024-04-23T16:33:38.988 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:38.988 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:38.989 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:38.989 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:38.989 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:33:38.989 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:38.989 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 146469 2024-04-23T16:33:38.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.989 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:38.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.989 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:38.989 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.553527+0000", 2024-04-23T16:33:38.989 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:38.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.553527+0000", 2024-04-23T16:33:38.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:38.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.553530+0000", 2024-04-23T16:33:38.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0520000000000002e-06 2024-04-23T16:33:38.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.991 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:38.991 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.553606+0000", 2024-04-23T16:33:38.991 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6798000000000006e-05 2024-04-23T16:33:38.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.991 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:38.991 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.553618+0000", 2024-04-23T16:33:38.991 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1508999999999999e-05 2024-04-23T16:33:38.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.991 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:38.991 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.553627+0000", 2024-04-23T16:33:38.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0469999999999999e-06 2024-04-23T16:33:38.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:38.992 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.553708+0000", 2024-04-23T16:33:38.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0921999999999995e-05 2024-04-23T16:33:38.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:38.992 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.553879+0000", 2024-04-23T16:33:38.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000171423 2024-04-23T16:33:38.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.993 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:33:38.993 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.554038+0000", 2024-04-23T16:33:38.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015865000000000001 2024-04-23T16:33:38.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.993 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:38.993 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.558576+0000", 2024-04-23T16:33:38.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045381409999999999 2024-04-23T16:33:38.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.994 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:38.994 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.604819+0000", 2024-04-23T16:33:38.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.046243228999999997 2024-04-23T16:33:38.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.994 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:38.994 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.604889+0000", 2024-04-23T16:33:38.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9470999999999999e-05 2024-04-23T16:33:38.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:38.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.604940+0000", 2024-04-23T16:33:38.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1610999999999998e-05 2024-04-23T16:33:38.995 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.995 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:38.995 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:38.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.995 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:148177 5.3 e386/359 5:caa34e22:::benchmark_data_smithi188_113957_object148176:head v 386'10606, mlcod=386'10603)", 2024-04-23T16:33:38.996 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:32:54.408471+0000", 2024-04-23T16:33:38.996 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 44.452799532999997, 2024-04-23T16:33:38.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.051610369000000003, 2024-04-23T16:33:38.996 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:38.996 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:38.996 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:38.996 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:38.996 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.996 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:38.996 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.408471+0000", 2024-04-23T16:33:38.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.997 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:38.997 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.408471+0000", 2024-04-23T16:33:38.997 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:38.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.997 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:38.997 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.408473+0000", 2024-04-23T16:33:38.998 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1619999999999998e-06 2024-04-23T16:33:38.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.998 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:38.998 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.408504+0000", 2024-04-23T16:33:38.998 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1285999999999998e-05 2024-04-23T16:33:38.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.998 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:38.998 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.408515+0000", 2024-04-23T16:33:38.998 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0531e-05 2024-04-23T16:33:38.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.999 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:38.999 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.408523+0000", 2024-04-23T16:33:38.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.072e-06 2024-04-23T16:33:38.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:38.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:38.999 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:39.000 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.433140+0000", 2024-04-23T16:33:39.000 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.024616842 2024-04-23T16:33:39.000 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:39.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:39.000 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:39.000 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.433159+0000", 2024-04-23T16:33:39.000 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8978e-05 2024-04-23T16:33:39.000 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:39.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:39.000 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:39.001 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.460041+0000", 2024-04-23T16:33:39.001 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026882593 2024-04-23T16:33:39.001 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:39.001 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:39.001 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:39.001 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.460081+0000", 2024-04-23T16:33:39.001 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9904999999999998e-05 2024-04-23T16:33:39.001 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:39.001 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:39.001 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:39.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:39.002 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:39.002 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:151546 5.2 5:497e7a6e:::benchmark_data_smithi188_113957_object151545:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-23T16:33:39.002 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:32:57.686710+0000", 2024-04-23T16:33:39.002 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 41.174560180999997, 2024-04-23T16:33:39.002 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.067105949999999998, 2024-04-23T16:33:39.002 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:39.002 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:39.002 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:39.002 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:39.002 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:33:39.003 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:39.003 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 151546 2024-04-23T16:33:39.003 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:39.003 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:39.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:39.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:39.003 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.686710+0000", 2024-04-23T16:33:39.003 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:39.003 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:39.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:39.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:39.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.686710+0000", 2024-04-23T16:33:39.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:39.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:39.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:39.004 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:39.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.686713+0000", 2024-04-23T16:33:39.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3530000000000002e-06 2024-04-23T16:33:39.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:39.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:39.004 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:39.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.686768+0000", 2024-04-23T16:33:39.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5173999999999997e-05 2024-04-23T16:33:39.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:39.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:39.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:39.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.686780+0000", 2024-04-23T16:33:39.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2160999999999999e-05 2024-04-23T16:33:39.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:39.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:39.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:39.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.686788+0000", 2024-04-23T16:33:39.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7559999999999999e-06 2024-04-23T16:33:39.006 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:39.006 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:39.006 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:39.006 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.686846+0000", 2024-04-23T16:33:39.006 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8121000000000001e-05 2024-04-23T16:33:39.006 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:39.006 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:39.006 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:39.006 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.686974+0000", 2024-04-23T16:33:39.006 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012791199999999999 2024-04-23T16:33:39.007 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:39.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:39.007 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:33:39.007 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.687128+0000", 2024-04-23T16:33:39.007 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015439699999999999 2024-04-23T16:33:39.007 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:39.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:39.007 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:39.007 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.691955+0000", 2024-04-23T16:33:39.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0048266560000000003 2024-04-23T16:33:39.008 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:39.008 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:39.008 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:39.008 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.753708+0000", 2024-04-23T16:33:39.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.061752870000000001 2024-04-23T16:33:39.008 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:39.008 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:39.008 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:39.008 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.753766+0000", 2024-04-23T16:33:39.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8745000000000001e-05 2024-04-23T16:33:39.009 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:39.009 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:39.009 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:39.009 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.753816+0000", 2024-04-23T16:33:39.009 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9805000000000002e-05 2024-04-23T16:33:39.009 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:39.009 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:39.009 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:39.009 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:39.009 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:39.010 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:33:39.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544c5a0a:::benchmark_data_smithi188_113957_object105636:head# 2024-04-23T16:33:39.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544d2d4e:::benchmark_data_smithi188_113957_object108073:head# 2024-04-23T16:33:39.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544d9fd6:::benchmark_data_smithi188_113957_object102727:head# 2024-04-23T16:33:39.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544dcb12:::benchmark_data_smithi188_113957_object74327:head# 2024-04-23T16:33:39.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:39.013 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:33:39.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:33:39.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544dcecb:::benchmark_data_smithi188_113957_object148561:head# 2024-04-23T16:33:39.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544e3714:::benchmark_data_smithi188_113957_object140405:head# 2024-04-23T16:33:39.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544ea324:::benchmark_data_smithi188_113957_object17627:head# 2024-04-23T16:33:39.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544ecfba:::benchmark_data_smithi188_113957_object37962:head# 2024-04-23T16:33:39.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544ee66d:::benchmark_data_smithi188_113957_object87309:head# 2024-04-23T16:33:39.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:544ef4b3:::benchmark_data_smithi188_113957_object31433:head# 2024-04-23T16:33:39.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5451b9c7:::benchmark_data_smithi188_113957_object85126:head# 2024-04-23T16:33:39.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54527079:::benchmark_data_smithi188_113957_object59000:head# 2024-04-23T16:33:39.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5452e1d6:::benchmark_data_smithi188_113957_object104025:head# 2024-04-23T16:33:39.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5452e8a2:::benchmark_data_smithi188_113957_object102693:head# 2024-04-23T16:33:39.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5452f3d0:::benchmark_data_smithi188_113957_object149358:head# 2024-04-23T16:33:39.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5453c998:::benchmark_data_smithi188_113957_object112035:head# 2024-04-23T16:33:39.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5454b08e:::benchmark_data_smithi188_113957_object83938:head# 2024-04-23T16:33:39.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5454ef6c:::benchmark_data_smithi188_113957_object110625:head# 2024-04-23T16:33:39.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5454f0f3:::benchmark_data_smithi188_113957_object145460:head# 2024-04-23T16:33:39.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5455cc40:::benchmark_data_smithi188_113957_object58363:head# 2024-04-23T16:33:39.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5455f504:::benchmark_data_smithi188_113957_object130231:head# 2024-04-23T16:33:39.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:545607ad:::benchmark_data_smithi188_113957_object18438:head# 2024-04-23T16:33:39.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5457de8f:::benchmark_data_smithi188_113957_object83593:head# 2024-04-23T16:33:39.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5457e106:::benchmark_data_smithi188_113957_object33318:head# 2024-04-23T16:33:39.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5457fac8:::benchmark_data_smithi188_113957_object82155:head# 2024-04-23T16:33:39.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54585bfe:::benchmark_data_smithi188_113957_object80221:head# 2024-04-23T16:33:39.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54585cea:::benchmark_data_smithi188_113957_object109609:head# 2024-04-23T16:33:39.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54586246:::benchmark_data_smithi188_113957_object97784:head# 2024-04-23T16:33:39.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:545883b1:::benchmark_data_smithi188_113957_object62594:head# 2024-04-23T16:33:39.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54592c83:::benchmark_data_smithi188_113957_object29068:head# 2024-04-23T16:33:39.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5459e189:::benchmark_data_smithi188_113957_object54984:head# 2024-04-23T16:33:39.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:545a3bf8:::benchmark_data_smithi188_113957_object128003:head# 2024-04-23T16:33:39.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:545af2de:::benchmark_data_smithi188_113957_object103540:head# 2024-04-23T16:33:39.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:545bae0c:::benchmark_data_smithi188_113957_object137985:head# 2024-04-23T16:33:39.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:545bf3bf:::benchmark_data_smithi188_113957_object24397:head# 2024-04-23T16:33:39.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:545c20bb:::benchmark_data_smithi188_113957_object658:head# 2024-04-23T16:33:39.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:545c5e3a:::benchmark_data_smithi188_113957_object89087:head# 2024-04-23T16:33:39.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:545cbc62:::benchmark_data_smithi188_113957_object114223:head# 2024-04-23T16:33:39.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:545d32f6:::benchmark_data_smithi188_113957_object89027:head# 2024-04-23T16:33:39.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:545d85d9:::benchmark_data_smithi188_113957_object80967:head# 2024-04-23T16:33:39.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:545d91db:::benchmark_data_smithi188_113957_object70672:head# 2024-04-23T16:33:39.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:545ee934:::benchmark_data_smithi188_113957_object58743:head# 2024-04-23T16:33:39.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:545f4fa3:::benchmark_data_smithi188_113957_object67488:head# 2024-04-23T16:33:39.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:545f9dc6:::benchmark_data_smithi188_113957_object27616:head# 2024-04-23T16:33:39.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:545fb7db:::benchmark_data_smithi188_113957_object67316:head# 2024-04-23T16:33:39.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5460256e:::benchmark_data_smithi188_113957_object60778:head# 2024-04-23T16:33:39.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5461a3de:::benchmark_data_smithi188_113957_object126015:head# 2024-04-23T16:33:39.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5461aa3e:::benchmark_data_smithi188_113957_object139033:head# 2024-04-23T16:33:39.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5461b6f3:::benchmark_data_smithi188_113957_object41237:head# 2024-04-23T16:33:39.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546210eb:::benchmark_data_smithi188_113957_object21484:head# 2024-04-23T16:33:39.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546245c8:::benchmark_data_smithi188_113957_object30303:head# 2024-04-23T16:33:39.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54626065:::benchmark_data_smithi188_113957_object131235:head# 2024-04-23T16:33:39.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546394c3:::benchmark_data_smithi188_113957_object148843:head# 2024-04-23T16:33:39.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54639c17:::benchmark_data_smithi188_113957_object33462:head# 2024-04-23T16:33:39.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5463ce25:::benchmark_data_smithi188_113957_object110935:head# 2024-04-23T16:33:39.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54641fa7:::benchmark_data_smithi188_113957_object89036:head# 2024-04-23T16:33:39.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546436d4:::benchmark_data_smithi188_113957_object111804:head# 2024-04-23T16:33:39.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546473ef:::benchmark_data_smithi188_113957_object61915:head# 2024-04-23T16:33:39.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54649216:::benchmark_data_smithi188_113957_object107817:head# 2024-04-23T16:33:39.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5464c6bf:::benchmark_data_smithi188_113957_object48046:head# 2024-04-23T16:33:39.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5464dcb0:::benchmark_data_smithi188_113957_object127952:head# 2024-04-23T16:33:39.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5464f279:::benchmark_data_smithi188_113957_object128150:head# 2024-04-23T16:33:39.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5464fb8c:::benchmark_data_smithi188_113957_object118043:head# 2024-04-23T16:33:39.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54653839:::benchmark_data_smithi188_113957_object144713:head# 2024-04-23T16:33:39.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546562a9:::benchmark_data_smithi188_113957_object50665:head# 2024-04-23T16:33:39.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5465dd6c:::benchmark_data_smithi188_113957_object149485:head# 2024-04-23T16:33:39.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546668a7:::benchmark_data_smithi188_113957_object31877:head# 2024-04-23T16:33:39.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5467047f:::benchmark_data_smithi188_113957_object89280:head# 2024-04-23T16:33:39.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546793b9:::benchmark_data_smithi188_113957_object66764:head# 2024-04-23T16:33:39.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54682cb1:::benchmark_data_smithi188_113957_object123240:head# 2024-04-23T16:33:39.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5468615a:::benchmark_data_smithi188_113957_object62143:head# 2024-04-23T16:33:39.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54690ed3:::benchmark_data_smithi188_113957_object151885:head# 2024-04-23T16:33:39.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5469320e:::benchmark_data_smithi188_113957_object10553:head# 2024-04-23T16:33:39.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54693639:::benchmark_data_smithi188_113957_object117808:head# 2024-04-23T16:33:39.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5469cf72:::benchmark_data_smithi188_113957_object90186:head# 2024-04-23T16:33:39.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546a3246:::benchmark_data_smithi188_113957_object80991:head# 2024-04-23T16:33:39.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546b0cfc:::benchmark_data_smithi188_113957_object82017:head# 2024-04-23T16:33:39.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546b3944:::benchmark_data_smithi188_113957_object41208:head# 2024-04-23T16:33:39.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546ba5f9:::benchmark_data_smithi188_113957_object75105:head# 2024-04-23T16:33:39.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546c6b5a:::benchmark_data_smithi188_113957_object15477:head# 2024-04-23T16:33:39.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546c6fc7:::benchmark_data_smithi188_113957_object138582:head# 2024-04-23T16:33:39.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546d1781:::benchmark_data_smithi188_113957_object110678:head# 2024-04-23T16:33:39.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546d19ed:::benchmark_data_smithi188_113957_object152099:head# 2024-04-23T16:33:39.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546d5808:::benchmark_data_smithi188_113957_object119876:head# 2024-04-23T16:33:39.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546d869e:::benchmark_data_smithi188_113957_object13062:head# 2024-04-23T16:33:39.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546d9381:::benchmark_data_smithi188_113957_object118159:head# 2024-04-23T16:33:39.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546dc772:::benchmark_data_smithi188_113957_object88199:head# 2024-04-23T16:33:39.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546e8c7b:::benchmark_data_smithi188_113957_object119697:head# 2024-04-23T16:33:39.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546f3c65:::benchmark_data_smithi188_113957_object83133:head# 2024-04-23T16:33:39.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546f5edc:::benchmark_data_smithi188_113957_object4482:head# 2024-04-23T16:33:39.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:546fd932:::benchmark_data_smithi188_113957_object85839:head# 2024-04-23T16:33:39.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54705e6d:::benchmark_data_smithi188_113957_object127817:head# 2024-04-23T16:33:39.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5470a109:::benchmark_data_smithi188_113957_object32648:head# 2024-04-23T16:33:39.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5470d450:::benchmark_data_smithi188_113957_object16744:head# 2024-04-23T16:33:39.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54717861:::benchmark_data_smithi188_113957_object123125:head# 2024-04-23T16:33:39.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54717f35:::benchmark_data_smithi188_113957_object116070:head# 2024-04-23T16:33:39.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54718bf8:::benchmark_data_smithi188_113957_object54441:head# 2024-04-23T16:33:39.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5471dc00:::benchmark_data_smithi188_113957_object98337:head# 2024-04-23T16:33:39.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5471f6c1:::benchmark_data_smithi188_113957_object41364:head# 2024-04-23T16:33:39.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54723605:::benchmark_data_smithi188_113957_object98814:head# 2024-04-23T16:33:39.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54728274:::benchmark_data_smithi188_113957_object131490:head# 2024-04-23T16:33:39.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5472fce2:::benchmark_data_smithi188_113957_object39147:head# 2024-04-23T16:33:39.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54744331:::benchmark_data_smithi188_113957_object119648:head# 2024-04-23T16:33:39.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54744a14:::benchmark_data_smithi188_113957_object115887:head# 2024-04-23T16:33:39.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54755dfc:::benchmark_data_smithi188_113957_object72240:head# 2024-04-23T16:33:39.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:547615da:::benchmark_data_smithi188_113957_object115976:head# 2024-04-23T16:33:39.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54766ec8:::benchmark_data_smithi188_113957_object35162:head# 2024-04-23T16:33:39.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54770220:::benchmark_data_smithi188_113957_object6980:head# 2024-04-23T16:33:39.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5477485f:::benchmark_data_smithi188_113957_object44198:head# 2024-04-23T16:33:39.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5477b88b:::benchmark_data_smithi188_113957_object76770:head# 2024-04-23T16:33:39.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5477fc0e:::benchmark_data_smithi188_113957_object143544:head# 2024-04-23T16:33:39.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5478504f:::benchmark_data_smithi188_113957_object52557:head# 2024-04-23T16:33:39.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5478d2f8:::benchmark_data_smithi188_113957_object91630:head# 2024-04-23T16:33:39.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5479b879:::benchmark_data_smithi188_113957_object32570:head# 2024-04-23T16:33:39.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5479d428:::benchmark_data_smithi188_113957_object1442:head# 2024-04-23T16:33:39.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:547a465a:::benchmark_data_smithi188_113957_object46730:head# 2024-04-23T16:33:39.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:547a7c69:::benchmark_data_smithi188_113957_object60811:head# 2024-04-23T16:33:39.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:547b1acd:::benchmark_data_smithi188_113957_object883:head# 2024-04-23T16:33:39.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:547b632a:::benchmark_data_smithi188_113957_object136496:head# 2024-04-23T16:33:39.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:547b740c:::benchmark_data_smithi188_113957_object62792:head# 2024-04-23T16:33:39.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:547c2fad:::benchmark_data_smithi188_113957_object144849:head# 2024-04-23T16:33:39.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:547d8a23:::benchmark_data_smithi188_113957_object115286:head# 2024-04-23T16:33:39.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:547e2ea0:::benchmark_data_smithi188_113957_object90139:head# 2024-04-23T16:33:39.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:547e6131:::benchmark_data_smithi188_113957_object71873:head# 2024-04-23T16:33:39.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:547ed9a9:::benchmark_data_smithi188_113957_object60251:head# 2024-04-23T16:33:39.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:547f2d3d:::benchmark_data_smithi188_113957_object130743:head# 2024-04-23T16:33:39.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:547f7eed:::benchmark_data_smithi188_113957_object134313:head# 2024-04-23T16:33:39.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:547f8ce1:::benchmark_data_smithi188_113957_object22482:head# 2024-04-23T16:33:39.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54801778:::benchmark_data_smithi188_113957_object150026:head# 2024-04-23T16:33:39.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548031b4:::benchmark_data_smithi188_113957_object32323:head# 2024-04-23T16:33:39.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54813877:::benchmark_data_smithi188_113957_object96470:head# 2024-04-23T16:33:39.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5481463f:::benchmark_data_smithi188_113957_object28792:head# 2024-04-23T16:33:39.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54816c93:::benchmark_data_smithi188_113957_object84838:head# 2024-04-23T16:33:39.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548201b7:::benchmark_data_smithi188_113957_object133414:head# 2024-04-23T16:33:39.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54828a6d:::benchmark_data_smithi188_113957_object62673:head# 2024-04-23T16:33:39.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5482d580:::benchmark_data_smithi188_113957_object8550:head# 2024-04-23T16:33:39.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548374e6:::benchmark_data_smithi188_113957_object101996:head# 2024-04-23T16:33:39.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548406cf:::benchmark_data_smithi188_113957_object36217:head# 2024-04-23T16:33:39.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548459b8:::benchmark_data_smithi188_113957_object109239:head# 2024-04-23T16:33:39.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54850faf:::benchmark_data_smithi188_113957_object20500:head# 2024-04-23T16:33:39.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5485669f:::benchmark_data_smithi188_113957_object144320:head# 2024-04-23T16:33:39.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5485acff:::benchmark_data_smithi188_113957_object116134:head# 2024-04-23T16:33:39.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548696c7:::benchmark_data_smithi188_113957_object2400:head# 2024-04-23T16:33:39.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54873878:::benchmark_data_smithi188_113957_object109880:head# 2024-04-23T16:33:39.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54874b74:::benchmark_data_smithi188_113957_object83772:head# 2024-04-23T16:33:39.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5487b9a3:::benchmark_data_smithi188_113957_object46636:head# 2024-04-23T16:33:39.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5487cf47:::benchmark_data_smithi188_113957_object72954:head# 2024-04-23T16:33:39.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5488668a:::benchmark_data_smithi188_113957_object143291:head# 2024-04-23T16:33:39.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5488b36e:::benchmark_data_smithi188_113957_object88364:head# 2024-04-23T16:33:39.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54894638:::benchmark_data_smithi188_113957_object92671:head# 2024-04-23T16:33:39.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5489c330:::benchmark_data_smithi188_113957_object35688:head# 2024-04-23T16:33:39.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5489fa46:::benchmark_data_smithi188_113957_object5622:head# 2024-04-23T16:33:39.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548a73c1:::benchmark_data_smithi188_113957_object48140:head# 2024-04-23T16:33:39.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548a811f:::benchmark_data_smithi188_113957_object61631:head# 2024-04-23T16:33:39.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548ad1bf:::benchmark_data_smithi188_113957_object127319:head# 2024-04-23T16:33:39.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548adfac:::benchmark_data_smithi188_113957_object146683:head# 2024-04-23T16:33:39.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548ae720:::benchmark_data_smithi188_113957_object26475:head# 2024-04-23T16:33:39.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548b2c41:::benchmark_data_smithi188_113957_object150222:head# 2024-04-23T16:33:39.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548daf32:::benchmark_data_smithi188_113957_object95927:head# 2024-04-23T16:33:39.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548e1c13:::benchmark_data_smithi188_113957_object91288:head# 2024-04-23T16:33:39.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548e28e4:::benchmark_data_smithi188_113957_object15051:head# 2024-04-23T16:33:39.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548e360d:::benchmark_data_smithi188_113957_object105964:head# 2024-04-23T16:33:39.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548e580e:::benchmark_data_smithi188_113957_object90242:head# 2024-04-23T16:33:39.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548ec162:::benchmark_data_smithi188_113957_object103593:head# 2024-04-23T16:33:39.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548f23bf:::benchmark_data_smithi188_113957_object126211:head# 2024-04-23T16:33:39.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548f5b22:::benchmark_data_smithi188_113957_object59034:head# 2024-04-23T16:33:39.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548f81eb:::benchmark_data_smithi188_113957_object28173:head# 2024-04-23T16:33:39.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:548fbe07:::benchmark_data_smithi188_113957_object146644:head# 2024-04-23T16:33:39.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54902cdd:::benchmark_data_smithi188_113957_object51397:head# 2024-04-23T16:33:39.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54907c45:::benchmark_data_smithi188_113957_object41626:head# 2024-04-23T16:33:39.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54910e72:::benchmark_data_smithi188_113957_object4178:head# 2024-04-23T16:33:39.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54912ec0:::benchmark_data_smithi188_113957_object61446:head# 2024-04-23T16:33:39.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5491e00b:::benchmark_data_smithi188_113957_object72634:head# 2024-04-23T16:33:39.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549235e0:::benchmark_data_smithi188_113957_object76459:head# 2024-04-23T16:33:39.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5492821b:::benchmark_data_smithi188_113957_object102745:head# 2024-04-23T16:33:39.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549387ba:::benchmark_data_smithi188_113957_object16530:head# 2024-04-23T16:33:39.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5493a665:::benchmark_data_smithi188_113957_object88924:head# 2024-04-23T16:33:39.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5494308d:::benchmark_data_smithi188_113957_object7020:head# 2024-04-23T16:33:39.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549439c0:::benchmark_data_smithi188_113957_object56573:head# 2024-04-23T16:33:39.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549454cb:::benchmark_data_smithi188_113957_object20294:head# 2024-04-23T16:33:39.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54947f3f:::benchmark_data_smithi188_113957_object45443:head# 2024-04-23T16:33:39.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54951bed:::benchmark_data_smithi188_113957_object79010:head# 2024-04-23T16:33:39.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5495856f:::benchmark_data_smithi188_113957_object91354:head# 2024-04-23T16:33:39.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54959a8a:::benchmark_data_smithi188_113957_object129273:head# 2024-04-23T16:33:39.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5495b8a5:::benchmark_data_smithi188_113957_object30460:head# 2024-04-23T16:33:39.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54964d82:::benchmark_data_smithi188_113957_object74629:head# 2024-04-23T16:33:39.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549651f8:::benchmark_data_smithi188_113957_object110434:head# 2024-04-23T16:33:39.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54966840:::benchmark_data_smithi188_113957_object5855:head# 2024-04-23T16:33:39.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549692d8:::benchmark_data_smithi188_113957_object72100:head# 2024-04-23T16:33:39.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5496a700:::benchmark_data_smithi188_113957_object151390:head# 2024-04-23T16:33:39.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5496d1a6:::benchmark_data_smithi188_113957_object108676:head# 2024-04-23T16:33:39.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5496e6f3:::benchmark_data_smithi188_113957_object13665:head# 2024-04-23T16:33:39.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54973b1d:::benchmark_data_smithi188_113957_object60732:head# 2024-04-23T16:33:39.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5497c468:::benchmark_data_smithi188_113957_object12755:head# 2024-04-23T16:33:39.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5497e83b:::benchmark_data_smithi188_113957_object91311:head# 2024-04-23T16:33:39.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549881ef:::benchmark_data_smithi188_113957_object24846:head# 2024-04-23T16:33:39.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54990535:::benchmark_data_smithi188_113957_object102487:head# 2024-04-23T16:33:39.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5499093f:::benchmark_data_smithi188_113957_object82660:head# 2024-04-23T16:33:39.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54990b56:::benchmark_data_smithi188_113957_object139134:head# 2024-04-23T16:33:39.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549954b4:::benchmark_data_smithi188_113957_object9001:head# 2024-04-23T16:33:39.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549ae5e3:::benchmark_data_smithi188_113957_object29442:head# 2024-04-23T16:33:39.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549b0e91:::benchmark_data_smithi188_113957_object23170:head# 2024-04-23T16:33:39.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549b731b:::benchmark_data_smithi188_113957_object51768:head# 2024-04-23T16:33:39.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549b9993:::benchmark_data_smithi188_113957_object26844:head# 2024-04-23T16:33:39.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549c95fc:::benchmark_data_smithi188_113957_object138107:head# 2024-04-23T16:33:39.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549ca392:::benchmark_data_smithi188_113957_object146374:head# 2024-04-23T16:33:39.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549cac1a:::benchmark_data_smithi188_113957_object93653:head# 2024-04-23T16:33:39.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549ce5dc:::benchmark_data_smithi188_113957_object122011:head# 2024-04-23T16:33:39.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549d1e3f:::benchmark_data_smithi188_113957_object35643:head# 2024-04-23T16:33:39.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549dd2d3:::benchmark_data_smithi188_113957_object17223:head# 2024-04-23T16:33:39.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549e3932:::benchmark_data_smithi188_113957_object55188:head# 2024-04-23T16:33:39.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549e7ad2:::benchmark_data_smithi188_113957_object127329:head# 2024-04-23T16:33:39.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549e845f:::benchmark_data_smithi188_113957_object88356:head# 2024-04-23T16:33:39.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549ea553:::benchmark_data_smithi188_113957_object96556:head# 2024-04-23T16:33:39.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549ee49d:::benchmark_data_smithi188_113957_object115820:head# 2024-04-23T16:33:39.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549f23a3:::benchmark_data_smithi188_113957_object46582:head# 2024-04-23T16:33:39.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549f6084:::benchmark_data_smithi188_113957_object91089:head# 2024-04-23T16:33:39.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:549fa02a:::benchmark_data_smithi188_113957_object12485:head# 2024-04-23T16:33:39.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a012cd:::benchmark_data_smithi188_113957_object21665:head# 2024-04-23T16:33:39.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a0d8cf:::benchmark_data_smithi188_113957_object146710:head# 2024-04-23T16:33:39.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a1281d:::benchmark_data_smithi188_113957_object45990:head# 2024-04-23T16:33:39.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a1f1a8:::benchmark_data_smithi188_113957_object99821:head# 2024-04-23T16:33:39.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a27573:::benchmark_data_smithi188_113957_object39460:head# 2024-04-23T16:33:39.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a33492:::benchmark_data_smithi188_113957_object64077:head# 2024-04-23T16:33:39.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a3b39e:::benchmark_data_smithi188_113957_object18566:head# 2024-04-23T16:33:39.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a3b623:::benchmark_data_smithi188_113957_object868:head# 2024-04-23T16:33:39.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a4acd4:::benchmark_data_smithi188_113957_object55086:head# 2024-04-23T16:33:39.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a4b2e7:::benchmark_data_smithi188_113957_object137439:head# 2024-04-23T16:33:39.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a52d68:::benchmark_data_smithi188_113957_object131637:head# 2024-04-23T16:33:39.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a54334:::benchmark_data_smithi188_113957_object63743:head# 2024-04-23T16:33:39.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a57319:::benchmark_data_smithi188_113957_object152704:head# 2024-04-23T16:33:39.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a5b41c:::benchmark_data_smithi188_113957_object859:head# 2024-04-23T16:33:39.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a61508:::benchmark_data_smithi188_113957_object63661:head# 2024-04-23T16:33:39.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a6354f:::benchmark_data_smithi188_113957_object129569:head# 2024-04-23T16:33:39.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a647f3:::benchmark_data_smithi188_113957_object87409:head# 2024-04-23T16:33:39.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a76290:::benchmark_data_smithi188_113957_object97674:head# 2024-04-23T16:33:39.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a76300:::benchmark_data_smithi188_113957_object95986:head# 2024-04-23T16:33:39.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a778d0:::benchmark_data_smithi188_113957_object144927:head# 2024-04-23T16:33:39.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a79cb9:::benchmark_data_smithi188_113957_object25756:head# 2024-04-23T16:33:39.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a7a9dc:::benchmark_data_smithi188_113957_object152024:head# 2024-04-23T16:33:39.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a7db61:::benchmark_data_smithi188_113957_object44316:head# 2024-04-23T16:33:39.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a827a9:::benchmark_data_smithi188_113957_object57368:head# 2024-04-23T16:33:39.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a98204:::benchmark_data_smithi188_113957_object136083:head# 2024-04-23T16:33:39.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54a98d8c:::benchmark_data_smithi188_113957_object105786:head# 2024-04-23T16:33:39.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54aa4085:::benchmark_data_smithi188_113957_object30270:head# 2024-04-23T16:33:39.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54aa6338:::benchmark_data_smithi188_113957_object117569:head# 2024-04-23T16:33:39.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ab09d3:::benchmark_data_smithi188_113957_object139573:head# 2024-04-23T16:33:39.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ab74c8:::benchmark_data_smithi188_113957_object130532:head# 2024-04-23T16:33:39.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ac3ba3:::benchmark_data_smithi188_113957_object46682:head# 2024-04-23T16:33:39.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ad5972:::benchmark_data_smithi188_113957_object113934:head# 2024-04-23T16:33:39.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54adc5ac:::benchmark_data_smithi188_113957_object88578:head# 2024-04-23T16:33:39.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54add534:::benchmark_data_smithi188_113957_object133818:head# 2024-04-23T16:33:39.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ae5dc2:::benchmark_data_smithi188_113957_object26470:head# 2024-04-23T16:33:39.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54af4b14:::benchmark_data_smithi188_113957_object69576:head# 2024-04-23T16:33:39.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54af65fc:::benchmark_data_smithi188_113957_object46122:head# 2024-04-23T16:33:39.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54b2756e:::benchmark_data_smithi188_113957_object107666:head# 2024-04-23T16:33:39.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54b2dcb5:::benchmark_data_smithi188_113957_object77740:head# 2024-04-23T16:33:39.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54b30575:::benchmark_data_smithi188_113957_object8259:head# 2024-04-23T16:33:39.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54b33b25:::benchmark_data_smithi188_113957_object8830:head# 2024-04-23T16:33:39.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54b53a2a:::benchmark_data_smithi188_113957_object53645:head# 2024-04-23T16:33:39.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54b63bac:::benchmark_data_smithi188_113957_object56154:head# 2024-04-23T16:33:39.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54b73c90:::benchmark_data_smithi188_113957_object113652:head# 2024-04-23T16:33:39.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54b74417:::benchmark_data_smithi188_113957_object124726:head# 2024-04-23T16:33:39.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54b75049:::benchmark_data_smithi188_113957_object93794:head# 2024-04-23T16:33:39.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54b75c5c:::benchmark_data_smithi188_113957_object150471:head# 2024-04-23T16:33:39.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54b77090:::benchmark_data_smithi188_113957_object87439:head# 2024-04-23T16:33:39.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54b816af:::benchmark_data_smithi188_113957_object105346:head# 2024-04-23T16:33:39.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54b8537a:::benchmark_data_smithi188_113957_object131588:head# 2024-04-23T16:33:39.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54b8f2f1:::benchmark_data_smithi188_113957_object135279:head# 2024-04-23T16:33:39.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54bab253:::benchmark_data_smithi188_113957_object145591:head# 2024-04-23T16:33:39.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54bc9f52:::benchmark_data_smithi188_113957_object40433:head# 2024-04-23T16:33:39.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54bceada:::benchmark_data_smithi188_113957_object132503:head# 2024-04-23T16:33:39.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54bcf204:::benchmark_data_smithi188_113957_object117268:head# 2024-04-23T16:33:39.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54bd3329:::benchmark_data_smithi188_113957_object22090:head# 2024-04-23T16:33:39.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54bd410c:::benchmark_data_smithi188_113957_object141294:head# 2024-04-23T16:33:39.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54be98a1:::benchmark_data_smithi188_113957_object125219:head# 2024-04-23T16:33:39.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54bef75e:::benchmark_data_smithi188_113957_object99392:head# 2024-04-23T16:33:39.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54befb9f:::benchmark_data_smithi188_113957_object144395:head# 2024-04-23T16:33:39.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54bf83fb:::benchmark_data_smithi188_113957_object70912:head# 2024-04-23T16:33:39.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54bf9436:::benchmark_data_smithi188_113957_object97701:head# 2024-04-23T16:33:39.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54bfa0cf:::benchmark_data_smithi188_113957_object137763:head# 2024-04-23T16:33:39.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c0fadc:::benchmark_data_smithi188_113957_object79426:head# 2024-04-23T16:33:39.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c13422:::benchmark_data_smithi188_113957_object124560:head# 2024-04-23T16:33:39.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c20b1e:::benchmark_data_smithi188_113957_object101837:head# 2024-04-23T16:33:39.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c21d4f:::benchmark_data_smithi188_113957_object110423:head# 2024-04-23T16:33:39.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c240be:::benchmark_data_smithi188_113957_object45371:head# 2024-04-23T16:33:39.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c2b641:::benchmark_data_smithi188_113957_object39757:head# 2024-04-23T16:33:39.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c345d6:::benchmark_data_smithi188_113957_object139672:head# 2024-04-23T16:33:39.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c3adf2:::benchmark_data_smithi188_113957_object71260:head# 2024-04-23T16:33:39.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c3e8bd:::benchmark_data_smithi188_113957_object55381:head# 2024-04-23T16:33:39.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c462fa:::benchmark_data_smithi188_113957_object69585:head# 2024-04-23T16:33:39.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c5c505:::benchmark_data_smithi188_113957_object107064:head# 2024-04-23T16:33:39.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c68200:::benchmark_data_smithi188_113957_object91670:head# 2024-04-23T16:33:39.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c6f214:::benchmark_data_smithi188_113957_object98176:head# 2024-04-23T16:33:39.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c7394d:::benchmark_data_smithi188_113957_object101329:head# 2024-04-23T16:33:39.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c7c213:::benchmark_data_smithi188_113957_object45458:head# 2024-04-23T16:33:39.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c90062:::benchmark_data_smithi188_113957_object112489:head# 2024-04-23T16:33:39.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c91e05:::benchmark_data_smithi188_113957_object104210:head# 2024-04-23T16:33:39.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c9389b:::benchmark_data_smithi188_113957_object29379:head# 2024-04-23T16:33:39.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c99488:::benchmark_data_smithi188_113957_object26617:head# 2024-04-23T16:33:39.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c9ba47:::benchmark_data_smithi188_113957_object47410:head# 2024-04-23T16:33:39.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54c9dbb0:::benchmark_data_smithi188_113957_object152090:head# 2024-04-23T16:33:39.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ca0552:::benchmark_data_smithi188_113957_object65776:head# 2024-04-23T16:33:39.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54cab723:::benchmark_data_smithi188_113957_object107006:head# 2024-04-23T16:33:39.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54cb001e:::benchmark_data_smithi188_113957_object13704:head# 2024-04-23T16:33:39.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54cb6349:::benchmark_data_smithi188_113957_object144702:head# 2024-04-23T16:33:39.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54cb7abe:::benchmark_data_smithi188_113957_object4372:head# 2024-04-23T16:33:39.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54cbd4e9:::benchmark_data_smithi188_113957_object18677:head# 2024-04-23T16:33:39.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54cc1d92:::benchmark_data_smithi188_113957_object108307:head# 2024-04-23T16:33:39.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54cc360c:::benchmark_data_smithi188_113957_object5329:head# 2024-04-23T16:33:39.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54cc72ac:::benchmark_data_smithi188_113957_object150101:head# 2024-04-23T16:33:39.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ccc1b3:::benchmark_data_smithi188_113957_object131405:head# 2024-04-23T16:33:39.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ccdf62:::benchmark_data_smithi188_113957_object86729:head# 2024-04-23T16:33:39.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54cce867:::benchmark_data_smithi188_113957_object68763:head# 2024-04-23T16:33:39.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54cced0d:::benchmark_data_smithi188_113957_object19881:head# 2024-04-23T16:33:39.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ccf989:::benchmark_data_smithi188_113957_object122261:head# 2024-04-23T16:33:39.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54cd2723:::benchmark_data_smithi188_113957_object136927:head# 2024-04-23T16:33:39.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54cdf390:::benchmark_data_smithi188_113957_object74343:head# 2024-04-23T16:33:39.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ce7c59:::benchmark_data_smithi188_113957_object112875:head# 2024-04-23T16:33:39.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ce8667:::benchmark_data_smithi188_113957_object108087:head# 2024-04-23T16:33:39.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ce92f9:::benchmark_data_smithi188_113957_object38849:head# 2024-04-23T16:33:39.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54cec5a7:::benchmark_data_smithi188_113957_object26544:head# 2024-04-23T16:33:39.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54cfa597:::benchmark_data_smithi188_113957_object122004:head# 2024-04-23T16:33:39.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d068bb:::benchmark_data_smithi188_113957_object56441:head# 2024-04-23T16:33:39.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d0c720:::benchmark_data_smithi188_113957_object21791:head# 2024-04-23T16:33:39.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d0f63e:::benchmark_data_smithi188_113957_object152619:head# 2024-04-23T16:33:39.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d1212f:::benchmark_data_smithi188_113957_object65251:head# 2024-04-23T16:33:39.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d1b828:::benchmark_data_smithi188_113957_object87127:head# 2024-04-23T16:33:39.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d2635b:::benchmark_data_smithi188_113957_object12904:head# 2024-04-23T16:33:39.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d2cd5d:::benchmark_data_smithi188_113957_object91367:head# 2024-04-23T16:33:39.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d2e636:::benchmark_data_smithi188_113957_object98895:head# 2024-04-23T16:33:39.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d35ed1:::benchmark_data_smithi188_113957_object131054:head# 2024-04-23T16:33:39.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d36b67:::benchmark_data_smithi188_113957_object9170:head# 2024-04-23T16:33:39.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d3e59d:::benchmark_data_smithi188_113957_object4016:head# 2024-04-23T16:33:39.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d40a02:::benchmark_data_smithi188_113957_object15211:head# 2024-04-23T16:33:39.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d50948:::benchmark_data_smithi188_113957_object3028:head# 2024-04-23T16:33:39.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d51d6c:::benchmark_data_smithi188_113957_object21822:head# 2024-04-23T16:33:39.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d57187:::benchmark_data_smithi188_113957_object10607:head# 2024-04-23T16:33:39.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d581c4:::benchmark_data_smithi188_113957_object23455:head# 2024-04-23T16:33:39.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d64535:::benchmark_data_smithi188_113957_object133561:head# 2024-04-23T16:33:39.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d64f94:::benchmark_data_smithi188_113957_object43137:head# 2024-04-23T16:33:39.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d6eda7:::benchmark_data_smithi188_113957_object20101:head# 2024-04-23T16:33:39.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d7db86:::benchmark_data_smithi188_113957_object103627:head# 2024-04-23T16:33:39.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d80693:::benchmark_data_smithi188_113957_object52413:head# 2024-04-23T16:33:39.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d8344e:::benchmark_data_smithi188_113957_object11402:head# 2024-04-23T16:33:39.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d89e52:::benchmark_data_smithi188_113957_object149831:head# 2024-04-23T16:33:39.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d94c38:::benchmark_data_smithi188_113957_object119261:head# 2024-04-23T16:33:39.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d9a12e:::benchmark_data_smithi188_113957_object66148:head# 2024-04-23T16:33:39.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54d9de98:::benchmark_data_smithi188_113957_object150990:head# 2024-04-23T16:33:39.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54da051b:::benchmark_data_smithi188_113957_object67325:head# 2024-04-23T16:33:39.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54da8fbc:::benchmark_data_smithi188_113957_object28668:head# 2024-04-23T16:33:39.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54dac0cc:::benchmark_data_smithi188_113957_object42910:head# 2024-04-23T16:33:39.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54db07c2:::benchmark_data_smithi188_113957_object763:head# 2024-04-23T16:33:39.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54dbeb07:::benchmark_data_smithi188_113957_object119762:head# 2024-04-23T16:33:39.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54dc7797:::benchmark_data_smithi188_113957_object686:head# 2024-04-23T16:33:39.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54dc8d05:::benchmark_data_smithi188_113957_object97077:head# 2024-04-23T16:33:39.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54dc9a0b:::benchmark_data_smithi188_113957_object122227:head# 2024-04-23T16:33:39.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54dd3089:::benchmark_data_smithi188_113957_object148361:head# 2024-04-23T16:33:39.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54de0ed7:::benchmark_data_smithi188_113957_object98235:head# 2024-04-23T16:33:39.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54de5091:::benchmark_data_smithi188_113957_object50632:head# 2024-04-23T16:33:39.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54de6679:::benchmark_data_smithi188_113957_object50754:head# 2024-04-23T16:33:39.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54dece0e:::benchmark_data_smithi188_113957_object97643:head# 2024-04-23T16:33:39.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54dedcb4:::benchmark_data_smithi188_113957_object80424:head# 2024-04-23T16:33:39.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54df44ec:::benchmark_data_smithi188_113957_object61754:head# 2024-04-23T16:33:39.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54df8366:::benchmark_data_smithi188_113957_object152465:head# 2024-04-23T16:33:39.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e133d3:::benchmark_data_smithi188_113957_object32864:head# 2024-04-23T16:33:39.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e1eae5:::benchmark_data_smithi188_113957_object138644:head# 2024-04-23T16:33:39.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e31782:::benchmark_data_smithi188_113957_object98907:head# 2024-04-23T16:33:39.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e340a0:::benchmark_data_smithi188_113957_object6858:head# 2024-04-23T16:33:39.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e3a49d:::benchmark_data_smithi188_113957_object119368:head# 2024-04-23T16:33:39.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e3e274:::benchmark_data_smithi188_113957_object39665:head# 2024-04-23T16:33:39.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e3f9f2:::benchmark_data_smithi188_113957_object73605:head# 2024-04-23T16:33:39.096 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e40685:::benchmark_data_smithi188_113957_object73075:head# 2024-04-23T16:33:39.096 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e45e32:::benchmark_data_smithi188_113957_object94277:head# 2024-04-23T16:33:39.096 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e46658:::benchmark_data_smithi188_113957_object124518:head# 2024-04-23T16:33:39.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e4a826:::benchmark_data_smithi188_113957_object82860:head# 2024-04-23T16:33:39.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e4b4c0:::benchmark_data_smithi188_113957_object23525:head# 2024-04-23T16:33:39.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e4e6c9:::benchmark_data_smithi188_113957_object83542:head# 2024-04-23T16:33:39.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e5c921:::benchmark_data_smithi188_113957_object152798:head# 2024-04-23T16:33:39.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e5cac7:::benchmark_data_smithi188_113957_object109973:head# 2024-04-23T16:33:39.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e5e081:::benchmark_data_smithi188_113957_object119965:head# 2024-04-23T16:33:39.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e698a5:::benchmark_data_smithi188_113957_object60153:head# 2024-04-23T16:33:39.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e6cfbd:::benchmark_data_smithi188_113957_object80070:head# 2024-04-23T16:33:39.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e6e55c:::benchmark_data_smithi188_113957_object38577:head# 2024-04-23T16:33:39.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e6fcfb:::benchmark_data_smithi188_113957_object76256:head# 2024-04-23T16:33:39.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e77ea1:::benchmark_data_smithi188_113957_object23272:head# 2024-04-23T16:33:39.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e83015:::benchmark_data_smithi188_113957_object16279:head# 2024-04-23T16:33:39.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e88584:::benchmark_data_smithi188_113957_object102784:head# 2024-04-23T16:33:39.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e9aae9:::benchmark_data_smithi188_113957_object125121:head# 2024-04-23T16:33:39.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e9c2e6:::benchmark_data_smithi188_113957_object140590:head# 2024-04-23T16:33:39.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54e9cb65:::benchmark_data_smithi188_113957_object34776:head# 2024-04-23T16:33:39.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ea36c1:::benchmark_data_smithi188_113957_object3396:head# 2024-04-23T16:33:39.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ea49fe:::benchmark_data_smithi188_113957_object146600:head# 2024-04-23T16:33:39.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ea5007:::benchmark_data_smithi188_113957_object33172:head# 2024-04-23T16:33:39.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ebb1e3:::benchmark_data_smithi188_113957_object46394:head# 2024-04-23T16:33:39.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ebe458:::benchmark_data_smithi188_113957_object33790:head# 2024-04-23T16:33:39.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ec2039:::benchmark_data_smithi188_113957_object6784:head# 2024-04-23T16:33:39.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ec3cbd:::benchmark_data_smithi188_113957_object53967:head# 2024-04-23T16:33:39.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ec7d6d:::benchmark_data_smithi188_113957_object107323:head# 2024-04-23T16:33:39.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ec9f70:::benchmark_data_smithi188_113957_object114869:head# 2024-04-23T16:33:39.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ecc3d4:::benchmark_data_smithi188_113957_object142430:head# 2024-04-23T16:33:39.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ed98be:::benchmark_data_smithi188_113957_object25258:head# 2024-04-23T16:33:39.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54edd426:::benchmark_data_smithi188_113957_object136340:head# 2024-04-23T16:33:39.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54eec0ff:::benchmark_data_smithi188_113957_object24062:head# 2024-04-23T16:33:39.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54eec178:::benchmark_data_smithi188_113957_object11305:head# 2024-04-23T16:33:39.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54eec418:::benchmark_data_smithi188_113957_object143972:head# 2024-04-23T16:33:39.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ef30b2:::benchmark_data_smithi188_113957_object132853:head# 2024-04-23T16:33:39.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ef55ef:::benchmark_data_smithi188_113957_object47477:head# 2024-04-23T16:33:39.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54efc623:::benchmark_data_smithi188_113957_object87069:head# 2024-04-23T16:33:39.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f01730:::benchmark_data_smithi188_113957_object27630:head# 2024-04-23T16:33:39.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f030e8:::benchmark_data_smithi188_113957_object58639:head# 2024-04-23T16:33:39.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f18108:::benchmark_data_smithi188_113957_object25503:head# 2024-04-23T16:33:39.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f2a00f:::benchmark_data_smithi188_113957_object129286:head# 2024-04-23T16:33:39.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f2cf69:::benchmark_data_smithi188_113957_object117721:head# 2024-04-23T16:33:39.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f2e26d:::benchmark_data_smithi188_113957_object7011:head# 2024-04-23T16:33:39.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f392aa:::benchmark_data_smithi188_113957_object79817:head# 2024-04-23T16:33:39.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f41de4:::benchmark_data_smithi188_113957_object47719:head# 2024-04-23T16:33:39.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f47ff1:::benchmark_data_smithi188_113957_object113624:head# 2024-04-23T16:33:39.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f56041:::benchmark_data_smithi188_113957_object50535:head# 2024-04-23T16:33:39.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f5700c:::benchmark_data_smithi188_113957_object124018:head# 2024-04-23T16:33:39.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f671f0:::benchmark_data_smithi188_113957_object112590:head# 2024-04-23T16:33:39.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f675ca:::benchmark_data_smithi188_113957_object61889:head# 2024-04-23T16:33:39.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f68fe6:::benchmark_data_smithi188_113957_object55123:head# 2024-04-23T16:33:39.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f6b4bb:::benchmark_data_smithi188_113957_object128738:head# 2024-04-23T16:33:39.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f6c263:::benchmark_data_smithi188_113957_object73230:head# 2024-04-23T16:33:39.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f75ee8:::benchmark_data_smithi188_113957_object120525:head# 2024-04-23T16:33:39.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f8ced1:::benchmark_data_smithi188_113957_object69539:head# 2024-04-23T16:33:39.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f8cfc8:::benchmark_data_smithi188_113957_object120992:head# 2024-04-23T16:33:39.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f8eb09:::benchmark_data_smithi188_113957_object102604:head# 2024-04-23T16:33:39.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f92f03:::benchmark_data_smithi188_113957_object89558:head# 2024-04-23T16:33:39.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f9578e:::benchmark_data_smithi188_113957_object71706:head# 2024-04-23T16:33:39.102 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54f986f6:::benchmark_data_smithi188_113957_object25960:head# 2024-04-23T16:33:39.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54fa86dc:::benchmark_data_smithi188_113957_object105114:head# 2024-04-23T16:33:39.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54fa9f79:::benchmark_data_smithi188_113957_object135413:head# 2024-04-23T16:33:39.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54fb14aa:::benchmark_data_smithi188_113957_object106741:head# 2024-04-23T16:33:39.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54fb2bfe:::benchmark_data_smithi188_113957_object40040:head# 2024-04-23T16:33:39.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54fb4eb5:::benchmark_data_smithi188_113957_object133815:head# 2024-04-23T16:33:39.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54fbc553:::benchmark_data_smithi188_113957_object21318:head# 2024-04-23T16:33:39.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54fbe9ad:::benchmark_data_smithi188_113957_object16570:head# 2024-04-23T16:33:39.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:39 smithi103 ceph-mon[39629]: pgmap v1118: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 9.0 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 518 op/s; 27626/296312 objects degraded (9.323%); 7.0 MiB/s, 112 objects/s recovering 2024-04-23T16:33:39.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:39 smithi103 ceph-mon[39629]: osdmap e397: 12 total, 11 up, 9 in 2024-04-23T16:33:39.107 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54fbeda5:::benchmark_data_smithi188_113957_object106828:head# 2024-04-23T16:33:39.107 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54fc9365:::benchmark_data_smithi188_113957_object109751:head# 2024-04-23T16:33:39.107 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54fce245:::benchmark_data_smithi188_113957_object29945:head# 2024-04-23T16:33:39.107 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54fdb90c:::benchmark_data_smithi188_113957_object137497:head# 2024-04-23T16:33:39.107 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54fdea4b:::benchmark_data_smithi188_113957_object1494:head# 2024-04-23T16:33:39.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54fdf2cc:::benchmark_data_smithi188_113957_object91811:head# 2024-04-23T16:33:39.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54fea976:::benchmark_data_smithi188_113957_object121570:head# 2024-04-23T16:33:39.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:54ff2ae6:::benchmark_data_smithi188_113957_object78960:head# 2024-04-23T16:33:39.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55010991:::benchmark_data_smithi188_113957_object56834:head# 2024-04-23T16:33:39.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550114dd:::benchmark_data_smithi188_113957_object42:head# 2024-04-23T16:33:39.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55011c13:::benchmark_data_smithi188_113957_object1681:head# 2024-04-23T16:33:39.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5502098f:::benchmark_data_smithi188_113957_object108965:head# 2024-04-23T16:33:39.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55021678:::benchmark_data_smithi188_113957_object15200:head# 2024-04-23T16:33:39.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55024724:::benchmark_data_smithi188_113957_object91338:head# 2024-04-23T16:33:39.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5502b20d:::benchmark_data_smithi188_113957_object107336:head# 2024-04-23T16:33:39.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55033178:::benchmark_data_smithi188_113957_object54574:head# 2024-04-23T16:33:39.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5503caaa:::benchmark_data_smithi188_113957_object129089:head# 2024-04-23T16:33:39.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55046a48:::benchmark_data_smithi188_113957_object41902:head# 2024-04-23T16:33:39.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55051e3c:::benchmark_data_smithi188_113957_object40642:head# 2024-04-23T16:33:39.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550539e4:::benchmark_data_smithi188_113957_object3119:head# 2024-04-23T16:33:39.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55056664:::benchmark_data_smithi188_113957_object131519:head# 2024-04-23T16:33:39.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5505c25c:::benchmark_data_smithi188_113957_object104537:head# 2024-04-23T16:33:39.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5506061f:::benchmark_data_smithi188_113957_object148938:head# 2024-04-23T16:33:39.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55060662:::benchmark_data_smithi188_113957_object60065:head# 2024-04-23T16:33:39.111 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55066953:::benchmark_data_smithi188_113957_object119488:head# 2024-04-23T16:33:39.111 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55068d42:::benchmark_data_smithi188_113957_object96407:head# 2024-04-23T16:33:39.111 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55070d71:::benchmark_data_smithi188_113957_object111163:head# 2024-04-23T16:33:39.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55076c58:::benchmark_data_smithi188_113957_object134723:head# 2024-04-23T16:33:39.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550777d1:::benchmark_data_smithi188_113957_object70222:head# 2024-04-23T16:33:39.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55080934:::benchmark_data_smithi188_113957_object69992:head# 2024-04-23T16:33:39.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55081593:::benchmark_data_smithi188_113957_object55828:head# 2024-04-23T16:33:39.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55081ef4:::benchmark_data_smithi188_113957_object62573:head# 2024-04-23T16:33:39.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5508213b:::benchmark_data_smithi188_113957_object102450:head# 2024-04-23T16:33:39.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5508299e:::benchmark_data_smithi188_113957_object145653:head# 2024-04-23T16:33:39.115 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:33:39.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55083e1f:::benchmark_data_smithi188_113957_object147938:head# 2024-04-23T16:33:39.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55085422:::benchmark_data_smithi188_113957_object134401:head# 2024-04-23T16:33:39.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550951c9:::benchmark_data_smithi188_113957_object71507:head# 2024-04-23T16:33:39.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550a1d6e:::benchmark_data_smithi188_113957_object118744:head# 2024-04-23T16:33:39.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550a29e1:::benchmark_data_smithi188_113957_object18139:head# 2024-04-23T16:33:39.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550a304b:::benchmark_data_smithi188_113957_object60909:head# 2024-04-23T16:33:39.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550a31a1:::benchmark_data_smithi188_113957_object103866:head# 2024-04-23T16:33:39.118 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550a4f94:::benchmark_data_smithi188_113957_object6174:head# 2024-04-23T16:33:39.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550adc0c:::benchmark_data_smithi188_113957_object50880:head# 2024-04-23T16:33:39.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550b3108:::benchmark_data_smithi188_113957_object16789:head# 2024-04-23T16:33:39.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550b8b38:::benchmark_data_smithi188_113957_object33585:head# 2024-04-23T16:33:39.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550c3983:::benchmark_data_smithi188_113957_object99523:head# 2024-04-23T16:33:39.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550c769f:::benchmark_data_smithi188_113957_object81901:head# 2024-04-23T16:33:39.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550cc5cf:::benchmark_data_smithi188_113957_object21231:head# 2024-04-23T16:33:39.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550cd423:::benchmark_data_smithi188_113957_object66555:head# 2024-04-23T16:33:39.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550cf1f3:::benchmark_data_smithi188_113957_object116089:head# 2024-04-23T16:33:39.122 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550daee6:::benchmark_data_smithi188_113957_object51081:head# 2024-04-23T16:33:39.122 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550ed300:::benchmark_data_smithi188_113957_object51473:head# 2024-04-23T16:33:39.122 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550f500c:::benchmark_data_smithi188_113957_object32126:head# 2024-04-23T16:33:39.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:550faaeb:::benchmark_data_smithi188_113957_object144471:head# 2024-04-23T16:33:39.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:551059f5:::benchmark_data_smithi188_113957_object33025:head# 2024-04-23T16:33:39.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5510ad99:::benchmark_data_smithi188_113957_object120137:head# 2024-04-23T16:33:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:39 smithi148 ceph-mon[44520]: pgmap v1118: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 9.0 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 518 op/s; 27626/296312 objects degraded (9.323%); 7.0 MiB/s, 112 objects/s recovering 2024-04-23T16:33:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:39 smithi148 ceph-mon[44520]: osdmap e397: 12 total, 11 up, 9 in 2024-04-23T16:33:39.125 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5510d2e6:::benchmark_data_smithi188_113957_object102235:head# 2024-04-23T16:33:39.125 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55114371:::benchmark_data_smithi188_113957_object39051:head# 2024-04-23T16:33:39.125 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5512b00a:::benchmark_data_smithi188_113957_object23912:head# 2024-04-23T16:33:39.125 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5512f10b:::benchmark_data_smithi188_113957_object39315:head# 2024-04-23T16:33:39.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5513349c:::benchmark_data_smithi188_113957_object89606:head# 2024-04-23T16:33:39.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55134f99:::benchmark_data_smithi188_113957_object128576:head# 2024-04-23T16:33:39.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5514032c:::benchmark_data_smithi188_113957_object107801:head# 2024-04-23T16:33:39.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:551439d1:::benchmark_data_smithi188_113957_object37309:head# 2024-04-23T16:33:39.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55147856:::benchmark_data_smithi188_113957_object33167:head# 2024-04-23T16:33:39.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:551485d0:::benchmark_data_smithi188_113957_object86886:head# 2024-04-23T16:33:39.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55157647:::benchmark_data_smithi188_113957_object79331:head# 2024-04-23T16:33:39.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:551618b1:::benchmark_data_smithi188_113957_object61263:head# 2024-04-23T16:33:39.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55176347:::benchmark_data_smithi188_113957_object2279:head# 2024-04-23T16:33:39.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5519fa31:::benchmark_data_smithi188_113957_object88202:head# 2024-04-23T16:33:39.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:551a1c62:::benchmark_data_smithi188_113957_object65915:head# 2024-04-23T16:33:39.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:551a98dc:::benchmark_data_smithi188_113957_object149507:head# 2024-04-23T16:33:39.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:551b1108:::benchmark_data_smithi188_113957_object58996:head# 2024-04-23T16:33:39.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:551ba4b6:::benchmark_data_smithi188_113957_object68113:head# 2024-04-23T16:33:39.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:551c26c9:::benchmark_data_smithi188_113957_object6127:head# 2024-04-23T16:33:39.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:551c348b:::benchmark_data_smithi188_113957_object62560:head# 2024-04-23T16:33:39.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:551c5b1d:::benchmark_data_smithi188_113957_object12244:head# 2024-04-23T16:33:39.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:551cf2e7:::benchmark_data_smithi188_113957_object106503:head# 2024-04-23T16:33:39.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:551ee632:::benchmark_data_smithi188_113957_object92717:head# 2024-04-23T16:33:39.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:551f523c:::benchmark_data_smithi188_113957_object90867:head# 2024-04-23T16:33:39.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5520bbbf:::benchmark_data_smithi188_113957_object13958:head# 2024-04-23T16:33:39.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55211228:::benchmark_data_smithi188_113957_object142335:head# 2024-04-23T16:33:39.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55211a20:::benchmark_data_smithi188_113957_object116162:head# 2024-04-23T16:33:39.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55225a1d:::benchmark_data_smithi188_113957_object139470:head# 2024-04-23T16:33:39.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5522dbff:::benchmark_data_smithi188_113957_object46292:head# 2024-04-23T16:33:39.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5523752a:::benchmark_data_smithi188_113957_object135869:head# 2024-04-23T16:33:39.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55237a94:::benchmark_data_smithi188_113957_object123859:head# 2024-04-23T16:33:39.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5523b08d:::benchmark_data_smithi188_113957_object6183:head# 2024-04-23T16:33:39.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5523c173:::benchmark_data_smithi188_113957_object30416:head# 2024-04-23T16:33:39.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5524498f:::benchmark_data_smithi188_113957_object138480:head# 2024-04-23T16:33:39.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55246b27:::benchmark_data_smithi188_113957_object84222:head# 2024-04-23T16:33:39.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5524bcb6:::benchmark_data_smithi188_113957_object56738:head# 2024-04-23T16:33:39.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5524ed4c:::benchmark_data_smithi188_113957_object115378:head# 2024-04-23T16:33:39.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55250085:::benchmark_data_smithi188_113957_object7845:head# 2024-04-23T16:33:39.142 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55258ba0:::benchmark_data_smithi188_113957_object57219:head# 2024-04-23T16:33:39.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:552610da:::benchmark_data_smithi188_113957_object116700:head# 2024-04-23T16:33:39.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55267846:::benchmark_data_smithi188_113957_object146124:head# 2024-04-23T16:33:39.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5526e5e0:::benchmark_data_smithi188_113957_object62718:head# 2024-04-23T16:33:39.144 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:39.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:552818fc:::benchmark_data_smithi188_113957_object98248:head# 2024-04-23T16:33:39.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55283fb8:::benchmark_data_smithi188_113957_object83085:head# 2024-04-23T16:33:39.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5528a32c:::benchmark_data_smithi188_113957_object58366:head# 2024-04-23T16:33:39.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5528caa7:::benchmark_data_smithi188_113957_object141303:head# 2024-04-23T16:33:39.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5528ec87:::benchmark_data_smithi188_113957_object81910:head# 2024-04-23T16:33:39.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55295c38:::benchmark_data_smithi188_113957_object76180:head# 2024-04-23T16:33:39.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5529a2ca:::benchmark_data_smithi188_113957_object6832:head# 2024-04-23T16:33:39.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5529d512:::benchmark_data_smithi188_113957_object143886:head# 2024-04-23T16:33:39.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:552aac7d:::benchmark_data_smithi188_113957_object36158:head# 2024-04-23T16:33:39.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:552ab278:::benchmark_data_smithi188_113957_object146884:head# 2024-04-23T16:33:39.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:552b6020:::benchmark_data_smithi188_113957_object40984:head# 2024-04-23T16:33:39.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:552bde8f:::benchmark_data_smithi188_113957_object54168:head# 2024-04-23T16:33:39.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:552c7855:::benchmark_data_smithi188_113957_object37122:head# 2024-04-23T16:33:39.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:552c7b8b:::benchmark_data_smithi188_113957_object105518:head# 2024-04-23T16:33:39.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:552c9124:::benchmark_data_smithi188_113957_object140895:head# 2024-04-23T16:33:39.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:552caf97:::benchmark_data_smithi188_113957_object145582:head# 2024-04-23T16:33:39.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:552cd443:::benchmark_data_smithi188_113957_object53611:head# 2024-04-23T16:33:39.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:552d694d:::benchmark_data_smithi188_113957_object81413:head# 2024-04-23T16:33:39.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:552e7f16:::benchmark_data_smithi188_113957_object127853:head# 2024-04-23T16:33:39.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:552ebeea:::benchmark_data_smithi188_113957_object144228:head# 2024-04-23T16:33:39.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5530a9ad:::benchmark_data_smithi188_113957_object64789:head# 2024-04-23T16:33:39.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5531bcba:::benchmark_data_smithi188_113957_object21540:head# 2024-04-23T16:33:39.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5533c9c1:::benchmark_data_smithi188_113957_object4648:head# 2024-04-23T16:33:39.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5533ce43:::benchmark_data_smithi188_113957_object65511:head# 2024-04-23T16:33:39.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5533ff77:::benchmark_data_smithi188_113957_object120162:head# 2024-04-23T16:33:39.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55355167:::benchmark_data_smithi188_113957_object38573:head# 2024-04-23T16:33:39.155 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5535b5f4:::benchmark_data_smithi188_113957_object68889:head# 2024-04-23T16:33:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553620a0:::benchmark_data_smithi188_113957_object73614:head# 2024-04-23T16:33:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55370a66:::benchmark_data_smithi188_113957_object46536:head# 2024-04-23T16:33:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5537bafa:::benchmark_data_smithi188_113957_object62656:head# 2024-04-23T16:33:39.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5537fea3:::benchmark_data_smithi188_113957_object1245:head# 2024-04-23T16:33:39.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553853af:::benchmark_data_smithi188_113957_object100915:head# 2024-04-23T16:33:39.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5539024a:::benchmark_data_smithi188_113957_object103742:head# 2024-04-23T16:33:39.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553926ca:::benchmark_data_smithi188_113957_object3166:head# 2024-04-23T16:33:39.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55395102:::benchmark_data_smithi188_113957_object90707:head# 2024-04-23T16:33:39.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553a412b:::benchmark_data_smithi188_113957_object89189:head# 2024-04-23T16:33:39.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553a8709:::benchmark_data_smithi188_113957_object85542:head# 2024-04-23T16:33:39.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553b256e:::benchmark_data_smithi188_113957_object59949:head# 2024-04-23T16:33:39.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553b4aaa:::benchmark_data_smithi188_113957_object125459:head# 2024-04-23T16:33:39.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553b63da:::benchmark_data_smithi188_113957_object108240:head# 2024-04-23T16:33:39.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553bb021:::benchmark_data_smithi188_113957_object139928:head# 2024-04-23T16:33:39.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553bb8b1:::benchmark_data_smithi188_113957_object79358:head# 2024-04-23T16:33:39.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553d32ee:::benchmark_data_smithi188_113957_object149927:head# 2024-04-23T16:33:39.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553d78de:::benchmark_data_smithi188_113957_object64917:head# 2024-04-23T16:33:39.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553dd5e6:::benchmark_data_smithi188_113957_object22337:head# 2024-04-23T16:33:39.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553e195a:::benchmark_data_smithi188_113957_object75906:head# 2024-04-23T16:33:39.166 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:33:39.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553e4def:::benchmark_data_smithi188_113957_object68499:head# 2024-04-23T16:33:39.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553ed765:::benchmark_data_smithi188_113957_object89650:head# 2024-04-23T16:33:39.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:39.170 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:33:39.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553f09ee:::benchmark_data_smithi188_113957_object114921:head# 2024-04-23T16:33:39.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553f2b34:::benchmark_data_smithi188_113957_object92070:head# 2024-04-23T16:33:39.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553f4836:::benchmark_data_smithi188_113957_object25087:head# 2024-04-23T16:33:39.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553f647b:::benchmark_data_smithi188_113957_object139449:head# 2024-04-23T16:33:39.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553f724c:::benchmark_data_smithi188_113957_object59371:head# 2024-04-23T16:33:39.177 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:553fa479:::benchmark_data_smithi188_113957_object106702:head# 2024-04-23T16:33:39.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5540308d:::benchmark_data_smithi188_113957_object88176:head# 2024-04-23T16:33:39.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55404475:::benchmark_data_smithi188_113957_object97361:head# 2024-04-23T16:33:39.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5540a39b:::benchmark_data_smithi188_113957_object53870:head# 2024-04-23T16:33:39.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5540f3f7:::benchmark_data_smithi188_113957_object44560:head# 2024-04-23T16:33:39.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554119e2:::benchmark_data_smithi188_113957_object144570:head# 2024-04-23T16:33:39.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5541751c:::benchmark_data_smithi188_113957_object126357:head# 2024-04-23T16:33:39.181 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5541c043:::benchmark_data_smithi188_113957_object111285:head# 2024-04-23T16:33:39.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5541cd26:::benchmark_data_smithi188_113957_object144459:head# 2024-04-23T16:33:39.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5541f98c:::benchmark_data_smithi188_113957_object133597:head# 2024-04-23T16:33:39.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554201f7:::benchmark_data_smithi188_113957_object36686:head# 2024-04-23T16:33:39.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554218ad:::benchmark_data_smithi188_113957_object139793:head# 2024-04-23T16:33:39.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5542b291:::benchmark_data_smithi188_113957_object130587:head# 2024-04-23T16:33:39.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5543119b:::benchmark_data_smithi188_113957_object11468:head# 2024-04-23T16:33:39.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55441881:::benchmark_data_smithi188_113957_object136258:head# 2024-04-23T16:33:39.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55443686:::benchmark_data_smithi188_113957_object62265:head# 2024-04-23T16:33:39.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5544a4ce:::benchmark_data_smithi188_113957_object87864:head# 2024-04-23T16:33:39.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55452612:::benchmark_data_smithi188_113957_object38408:head# 2024-04-23T16:33:39.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55452a83:::benchmark_data_smithi188_113957_object73504:head# 2024-04-23T16:33:39.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5545541c:::benchmark_data_smithi188_113957_object136204:head# 2024-04-23T16:33:39.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5545913f:::benchmark_data_smithi188_113957_object82271:head# 2024-04-23T16:33:39.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5545f392:::benchmark_data_smithi188_113957_object75560:head# 2024-04-23T16:33:39.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5545fdb0:::benchmark_data_smithi188_113957_object19142:head# 2024-04-23T16:33:39.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55461741:::benchmark_data_smithi188_113957_object102151:head# 2024-04-23T16:33:39.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55462c20:::benchmark_data_smithi188_113957_object118423:head# 2024-04-23T16:33:39.197 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554656a6:::benchmark_data_smithi188_113957_object38366:head# 2024-04-23T16:33:39.197 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5546fa00:::benchmark_data_smithi188_113957_object130436:head# 2024-04-23T16:33:39.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554716d5:::benchmark_data_smithi188_113957_object99925:head# 2024-04-23T16:33:39.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55476c2f:::benchmark_data_smithi188_113957_object98111:head# 2024-04-23T16:33:39.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55482f2f:::benchmark_data_smithi188_113957_object79494:head# 2024-04-23T16:33:39.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5548453b:::benchmark_data_smithi188_113957_object56005:head# 2024-04-23T16:33:39.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55485531:::benchmark_data_smithi188_113957_object91573:head# 2024-04-23T16:33:39.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55489592:::benchmark_data_smithi188_113957_object90208:head# 2024-04-23T16:33:39.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5548b4ca:::benchmark_data_smithi188_113957_object94270:head# 2024-04-23T16:33:39.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5549652a:::benchmark_data_smithi188_113957_object119096:head# 2024-04-23T16:33:39.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5549876c:::benchmark_data_smithi188_113957_object3046:head# 2024-04-23T16:33:39.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554a1244:::benchmark_data_smithi188_113957_object20836:head# 2024-04-23T16:33:39.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554a9095:::benchmark_data_smithi188_113957_object21284:head# 2024-04-23T16:33:39.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554a989d:::benchmark_data_smithi188_113957_object21345:head# 2024-04-23T16:33:39.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554aa3c2:::benchmark_data_smithi188_113957_object75519:head# 2024-04-23T16:33:39.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554b8a4b:::benchmark_data_smithi188_113957_object61884:head# 2024-04-23T16:33:39.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554bd0cd:::benchmark_data_smithi188_113957_object132880:head# 2024-04-23T16:33:39.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554c22c9:::benchmark_data_smithi188_113957_object131497:head# 2024-04-23T16:33:39.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554cb6c6:::benchmark_data_smithi188_113957_object34879:head# 2024-04-23T16:33:39.206 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554d5caf:::benchmark_data_smithi188_113957_object25166:head# 2024-04-23T16:33:39.207 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554e4f23:::benchmark_data_smithi188_113957_object97875:head# 2024-04-23T16:33:39.207 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554e5142:::benchmark_data_smithi188_113957_object6596:head# 2024-04-23T16:33:39.207 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554edae6:::benchmark_data_smithi188_113957_object56562:head# 2024-04-23T16:33:39.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554f3552:::benchmark_data_smithi188_113957_object135840:head# 2024-04-23T16:33:39.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554f4765:::benchmark_data_smithi188_113957_object41202:head# 2024-04-23T16:33:39.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554f8818:::benchmark_data_smithi188_113957_object84913:head# 2024-04-23T16:33:39.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:554fd75a:::benchmark_data_smithi188_113957_object127386:head# 2024-04-23T16:33:39.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555034e7:::benchmark_data_smithi188_113957_object132124:head# 2024-04-23T16:33:39.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555047c7:::benchmark_data_smithi188_113957_object138301:head# 2024-04-23T16:33:39.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55508ef9:::benchmark_data_smithi188_113957_object50163:head# 2024-04-23T16:33:39.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5550afac:::benchmark_data_smithi188_113957_object83876:head# 2024-04-23T16:33:39.212 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5550f044:::benchmark_data_smithi188_113957_object59389:head# 2024-04-23T16:33:39.212 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55515f55:::benchmark_data_smithi188_113957_object110658:head# 2024-04-23T16:33:39.212 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55517056:::benchmark_data_smithi188_113957_object92981:head# 2024-04-23T16:33:39.213 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55521946:::benchmark_data_smithi188_113957_object97912:head# 2024-04-23T16:33:39.213 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5552c3dd:::benchmark_data_smithi188_113957_object67469:head# 2024-04-23T16:33:39.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55530f0f:::benchmark_data_smithi188_113957_object2525:head# 2024-04-23T16:33:39.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555354b4:::benchmark_data_smithi188_113957_object23562:head# 2024-04-23T16:33:39.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555356bf:::benchmark_data_smithi188_113957_object153383:head# 2024-04-23T16:33:39.217 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5553edd0:::benchmark_data_smithi188_113957_object66766:head# 2024-04-23T16:33:39.217 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5554887e:::benchmark_data_smithi188_113957_object82809:head# 2024-04-23T16:33:39.217 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555562d1:::benchmark_data_smithi188_113957_object39331:head# 2024-04-23T16:33:39.217 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555594d1:::benchmark_data_smithi188_113957_object136978:head# 2024-04-23T16:33:39.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5555cfba:::benchmark_data_smithi188_113957_object62512:head# 2024-04-23T16:33:39.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5555d784:::benchmark_data_smithi188_113957_object32875:head# 2024-04-23T16:33:39.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5555d985:::benchmark_data_smithi188_113957_object60506:head# 2024-04-23T16:33:39.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5556be61:::benchmark_data_smithi188_113957_object35781:head# 2024-04-23T16:33:39.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5556d917:::benchmark_data_smithi188_113957_object88487:head# 2024-04-23T16:33:39.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5557534a:::benchmark_data_smithi188_113957_object116129:head# 2024-04-23T16:33:39.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555753e6:::benchmark_data_smithi188_113957_object50987:head# 2024-04-23T16:33:39.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5557f883:::benchmark_data_smithi188_113957_object134408:head# 2024-04-23T16:33:39.221 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55582a75:::benchmark_data_smithi188_113957_object119861:head# 2024-04-23T16:33:39.222 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55590f46:::benchmark_data_smithi188_113957_object8707:head# 2024-04-23T16:33:39.222 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555a6da2:::benchmark_data_smithi188_113957_object145231:head# 2024-04-23T16:33:39.222 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555a97d2:::benchmark_data_smithi188_113957_object104186:head# 2024-04-23T16:33:39.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555aa9e3:::benchmark_data_smithi188_113957_object112465:head# 2024-04-23T16:33:39.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555abdac:::benchmark_data_smithi188_113957_object131062:head# 2024-04-23T16:33:39.224 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555b5242:::benchmark_data_smithi188_113957_object49165:head# 2024-04-23T16:33:39.224 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555c3048:::benchmark_data_smithi188_113957_object141708:head# 2024-04-23T16:33:39.225 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555cfff2:::benchmark_data_smithi188_113957_object65322:head# 2024-04-23T16:33:39.225 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555d007c:::benchmark_data_smithi188_113957_object134551:head# 2024-04-23T16:33:39.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555d7cce:::benchmark_data_smithi188_113957_object50688:head# 2024-04-23T16:33:39.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555de223:::benchmark_data_smithi188_113957_object37277:head# 2024-04-23T16:33:39.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555e5683:::benchmark_data_smithi188_113957_object69035:head# 2024-04-23T16:33:39.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555eb516:::benchmark_data_smithi188_113957_object105205:head# 2024-04-23T16:33:39.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555eb6c2:::benchmark_data_smithi188_113957_object51982:head# 2024-04-23T16:33:39.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555f2e26:::benchmark_data_smithi188_113957_object36043:head# 2024-04-23T16:33:39.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:555f4365:::benchmark_data_smithi188_113957_object98025:head# 2024-04-23T16:33:39.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55605097:::benchmark_data_smithi188_113957_object54457:head# 2024-04-23T16:33:39.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5560b3f9:::benchmark_data_smithi188_113957_object107518:head# 2024-04-23T16:33:39.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55611733:::benchmark_data_smithi188_113957_object22119:head# 2024-04-23T16:33:39.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:556151f0:::benchmark_data_smithi188_113957_object102012:head# 2024-04-23T16:33:39.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5563878e:::benchmark_data_smithi188_113957_object110267:head# 2024-04-23T16:33:39.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5563b4e5:::benchmark_data_smithi188_113957_object98339:head# 2024-04-23T16:33:39.231 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55648f15:::benchmark_data_smithi188_113957_object15321:head# 2024-04-23T16:33:39.231 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55655d6c:::benchmark_data_smithi188_113957_object90702:head# 2024-04-23T16:33:39.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5565c2c2:::benchmark_data_smithi188_113957_object107591:head# 2024-04-23T16:33:39.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55660a12:::benchmark_data_smithi188_113957_object48648:head# 2024-04-23T16:33:39.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55664e81:::benchmark_data_smithi188_113957_object23796:head# 2024-04-23T16:33:39.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:556668d5:::benchmark_data_smithi188_113957_object34805:head# 2024-04-23T16:33:39.234 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55666c79:::benchmark_data_smithi188_113957_object109852:head# 2024-04-23T16:33:39.234 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55668499:::benchmark_data_smithi188_113957_object37483:head# 2024-04-23T16:33:39.234 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55668a72:::benchmark_data_smithi188_113957_object81712:head# 2024-04-23T16:33:39.235 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:33:39.235 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:33:39.235 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:33:39.235 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:33:39.235 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:33:39.235 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:33:39.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55668f0a:::benchmark_data_smithi188_113957_object91995:head# 2024-04-23T16:33:39.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5566c85c:::benchmark_data_smithi188_113957_object55020:head# 2024-04-23T16:33:39.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55681f53:::benchmark_data_smithi188_113957_object43502:head# 2024-04-23T16:33:39.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55682c7b:::benchmark_data_smithi188_113957_object118283:head# 2024-04-23T16:33:39.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55684ec5:::benchmark_data_smithi188_113957_object139465:head# 2024-04-23T16:33:39.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:556a17b5:::benchmark_data_smithi188_113957_object69116:head# 2024-04-23T16:33:39.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:556a1cc9:::benchmark_data_smithi188_113957_object57463:head# 2024-04-23T16:33:39.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:556b637c:::benchmark_data_smithi188_113957_object146036:head# 2024-04-23T16:33:39.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:556bd1be:::benchmark_data_smithi188_113957_object33006:head# 2024-04-23T16:33:39.240 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:556c39ef:::benchmark_data_smithi188_113957_object125613:head# 2024-04-23T16:33:39.240 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:556d3aa7:::benchmark_data_smithi188_113957_object50094:head# 2024-04-23T16:33:39.241 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:556dce14:::benchmark_data_smithi188_113957_object15482:head# 2024-04-23T16:33:39.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:556e174b:::benchmark_data_smithi188_113957_object130189:head# 2024-04-23T16:33:39.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:556e7794:::benchmark_data_smithi188_113957_object151869:head# 2024-04-23T16:33:39.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:556e8265:::benchmark_data_smithi188_113957_object75700:head# 2024-04-23T16:33:39.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:556ef0ec:::benchmark_data_smithi188_113957_object150816:head# 2024-04-23T16:33:39.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:556f699a:::benchmark_data_smithi188_113957_object41599:head# 2024-04-23T16:33:39.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:556fbbf3:::benchmark_data_smithi188_113957_object114524:head# 2024-04-23T16:33:39.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:556fdb33:::benchmark_data_smithi188_113957_object42103:head# 2024-04-23T16:33:39.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5570bc91:::benchmark_data_smithi188_113957_object124982:head# 2024-04-23T16:33:39.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5570d663:::benchmark_data_smithi188_113957_object7761:head# 2024-04-23T16:33:39.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5571f985:::benchmark_data_smithi188_113957_object49719:head# 2024-04-23T16:33:39.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55720340:::benchmark_data_smithi188_113957_object134507:head# 2024-04-23T16:33:39.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5572dae9:::benchmark_data_smithi188_113957_object88682:head# 2024-04-23T16:33:39.248 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5572e3be:::benchmark_data_smithi188_113957_object112951:head# 2024-04-23T16:33:39.248 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:557338d8:::benchmark_data_smithi188_113957_object69744:head# 2024-04-23T16:33:39.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5573933b:::benchmark_data_smithi188_113957_object148860:head# 2024-04-23T16:33:39.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:557443f7:::benchmark_data_smithi188_113957_object142520:head# 2024-04-23T16:33:39.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5575b8f5:::benchmark_data_smithi188_113957_object35326:head# 2024-04-23T16:33:39.250 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:33:39.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5575ba08:::benchmark_data_smithi188_113957_object59505:head# 2024-04-23T16:33:39.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5575c5d5:::benchmark_data_smithi188_113957_object81436:head# 2024-04-23T16:33:39.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5575db80:::benchmark_data_smithi188_113957_object108589:head# 2024-04-23T16:33:39.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5576c38f:::benchmark_data_smithi188_113957_object125231:head# 2024-04-23T16:33:39.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5577df55:::benchmark_data_smithi188_113957_object68979:head# 2024-04-23T16:33:39.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55787810:::benchmark_data_smithi188_113957_object16603:head# 2024-04-23T16:33:39.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:557884dd:::benchmark_data_smithi188_113957_object145140:head# 2024-04-23T16:33:39.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5578887d:::benchmark_data_smithi188_113957_object122851:head# 2024-04-23T16:33:39.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5578cd74:::benchmark_data_smithi188_113957_object119472:head# 2024-04-23T16:33:39.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5578fb50:::benchmark_data_smithi188_113957_object23210:head# 2024-04-23T16:33:39.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55790921:::benchmark_data_smithi188_113957_object102830:head# 2024-04-23T16:33:39.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5579ded6:::benchmark_data_smithi188_113957_object32428:head# 2024-04-23T16:33:39.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5579fa41:::benchmark_data_smithi188_113957_object100930:head# 2024-04-23T16:33:39.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:557a01a8:::benchmark_data_smithi188_113957_object8408:head# 2024-04-23T16:33:39.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:557aab32:::benchmark_data_smithi188_113957_object12413:head# 2024-04-23T16:33:39.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:557b4224:::benchmark_data_smithi188_113957_object128565:head# 2024-04-23T16:33:39.262 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:557b50c2:::benchmark_data_smithi188_113957_object59126:head# 2024-04-23T16:33:39.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:557c5e70:::benchmark_data_smithi188_113957_object75141:head# 2024-04-23T16:33:39.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:557d03d1:::benchmark_data_smithi188_113957_object31756:head# 2024-04-23T16:33:39.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:557dde19:::benchmark_data_smithi188_113957_object146845:head# 2024-04-23T16:33:39.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:557e7cbb:::benchmark_data_smithi188_113957_object84229:head# 2024-04-23T16:33:39.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:557efdc0:::benchmark_data_smithi188_113957_object24626:head# 2024-04-23T16:33:39.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:557f0b85:::benchmark_data_smithi188_113957_object92231:head# 2024-04-23T16:33:39.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:557fa1f8:::benchmark_data_smithi188_113957_object11563:head# 2024-04-23T16:33:39.266 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55806fe7:::benchmark_data_smithi188_113957_object33667:head# 2024-04-23T16:33:39.267 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5580b4ed:::benchmark_data_smithi188_113957_object111461:head# 2024-04-23T16:33:39.267 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5580e78a:::benchmark_data_smithi188_113957_object79892:head# 2024-04-23T16:33:39.267 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5581842d:::benchmark_data_smithi188_113957_object53169:head# 2024-04-23T16:33:39.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5581b8b6:::benchmark_data_smithi188_113957_object113842:head# 2024-04-23T16:33:39.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55821510:::benchmark_data_smithi188_113957_object39957:head# 2024-04-23T16:33:39.271 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:33:39.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558247b5:::benchmark_data_smithi188_113957_object62205:head# 2024-04-23T16:33:39.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55825308:::benchmark_data_smithi188_113957_object10370:head# 2024-04-23T16:33:39.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55825c77:::benchmark_data_smithi188_113957_object70997:head# 2024-04-23T16:33:39.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55835ddc:::benchmark_data_smithi188_113957_object60269:head# 2024-04-23T16:33:39.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55836d4b:::benchmark_data_smithi188_113957_object116051:head# 2024-04-23T16:33:39.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5583c262:::benchmark_data_smithi188_113957_object72295:head# 2024-04-23T16:33:39.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55845296:::benchmark_data_smithi188_113957_object103302:head# 2024-04-23T16:33:39.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5584793f:::benchmark_data_smithi188_113957_object65873:head# 2024-04-23T16:33:39.274 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:33:39.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5584e629:::benchmark_data_smithi188_113957_object109666:head# 2024-04-23T16:33:39.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5585ac42:::benchmark_data_smithi188_113957_object127260:head# 2024-04-23T16:33:39.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5585c1de:::benchmark_data_smithi188_113957_object94107:head# 2024-04-23T16:33:39.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5585c55a:::benchmark_data_smithi188_113957_object60200:head# 2024-04-23T16:33:39.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5585f9ea:::benchmark_data_smithi188_113957_object43023:head# 2024-04-23T16:33:39.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5586027b:::benchmark_data_smithi188_113957_object114195:head# 2024-04-23T16:33:39.277 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55863659:::benchmark_data_smithi188_113957_object37538:head# 2024-04-23T16:33:39.277 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5586ce5e:::benchmark_data_smithi188_113957_object140679:head# 2024-04-23T16:33:39.277 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5588239b:::benchmark_data_smithi188_113957_object3214:head# 2024-04-23T16:33:39.277 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55890873:::benchmark_data_smithi188_113957_object27186:head# 2024-04-23T16:33:39.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55897386:::benchmark_data_smithi188_113957_object71730:head# 2024-04-23T16:33:39.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5589a10c:::benchmark_data_smithi188_113957_object147267:head# 2024-04-23T16:33:39.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5589c053:::benchmark_data_smithi188_113957_object100360:head# 2024-04-23T16:33:39.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558a0186:::benchmark_data_smithi188_113957_object69575:head# 2024-04-23T16:33:39.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558a0c78:::benchmark_data_smithi188_113957_object33469:head# 2024-04-23T16:33:39.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558a282a:::benchmark_data_smithi188_113957_object18721:head# 2024-04-23T16:33:39.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558a448a:::benchmark_data_smithi188_113957_object135695:head# 2024-04-23T16:33:39.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558abcef:::benchmark_data_smithi188_113957_object28956:head# 2024-04-23T16:33:39.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558acd13:::benchmark_data_smithi188_113957_object97044:head# 2024-04-23T16:33:39.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558af6de:::benchmark_data_smithi188_113957_object85313:head# 2024-04-23T16:33:39.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558b3736:::benchmark_data_smithi188_113957_object48356:head# 2024-04-23T16:33:39.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558b4061:::benchmark_data_smithi188_113957_object21886:head# 2024-04-23T16:33:39.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558b64cf:::benchmark_data_smithi188_113957_object64883:head# 2024-04-23T16:33:39.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558b6936:::benchmark_data_smithi188_113957_object147679:head# 2024-04-23T16:33:39.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558b7f66:::benchmark_data_smithi188_113957_object53282:head# 2024-04-23T16:33:39.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558baee7:::benchmark_data_smithi188_113957_object71269:head# 2024-04-23T16:33:39.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558bc523:::benchmark_data_smithi188_113957_object47849:head# 2024-04-23T16:33:39.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558c66fd:::benchmark_data_smithi188_113957_object847:head# 2024-04-23T16:33:39.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558c9c1b:::benchmark_data_smithi188_113957_object84697:head# 2024-04-23T16:33:39.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558ccff4:::benchmark_data_smithi188_113957_object93189:head# 2024-04-23T16:33:39.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558cff9b:::benchmark_data_smithi188_113957_object71519:head# 2024-04-23T16:33:39.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558d2b0a:::benchmark_data_smithi188_113957_object149634:head# 2024-04-23T16:33:39.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558d7268:::benchmark_data_smithi188_113957_object95083:head# 2024-04-23T16:33:39.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558de549:::benchmark_data_smithi188_113957_object102106:head# 2024-04-23T16:33:39.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558e2574:::benchmark_data_smithi188_113957_object151154:head# 2024-04-23T16:33:39.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558e9cf8:::benchmark_data_smithi188_113957_object103737:head# 2024-04-23T16:33:39.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558ed926:::benchmark_data_smithi188_113957_object38960:head# 2024-04-23T16:33:39.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558f6d1a:::benchmark_data_smithi188_113957_object89915:head# 2024-04-23T16:33:39.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:558f7f9e:::benchmark_data_smithi188_113957_object105446:head# 2024-04-23T16:33:39.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:559078d1:::benchmark_data_smithi188_113957_object131801:head# 2024-04-23T16:33:39.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55909666:::benchmark_data_smithi188_113957_object129839:head# 2024-04-23T16:33:39.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5591993f:::benchmark_data_smithi188_113957_object49650:head# 2024-04-23T16:33:39.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5591e8fc:::benchmark_data_smithi188_113957_object19022:head# 2024-04-23T16:33:39.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5591fc55:::benchmark_data_smithi188_113957_object91836:head# 2024-04-23T16:33:39.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:559200b3:::benchmark_data_smithi188_113957_object129826:head# 2024-04-23T16:33:39.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55923d64:::benchmark_data_smithi188_113957_object129213:head# 2024-04-23T16:33:39.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55926f2f:::benchmark_data_smithi188_113957_object58556:head# 2024-04-23T16:33:39.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5593068c:::benchmark_data_smithi188_113957_object36911:head# 2024-04-23T16:33:39.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5593244f:::benchmark_data_smithi188_113957_object140028:head# 2024-04-23T16:33:39.295 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:559337fa:::benchmark_data_smithi188_113957_object89700:head# 2024-04-23T16:33:39.297 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:33:39.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5593bca7:::benchmark_data_smithi188_113957_object98481:head# 2024-04-23T16:33:39.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5593d352:::benchmark_data_smithi188_113957_object26552:head# 2024-04-23T16:33:39.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5593d96c:::benchmark_data_smithi188_113957_object87714:head# 2024-04-23T16:33:39.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5594a200:::benchmark_data_smithi188_113957_object131518:head# 2024-04-23T16:33:39.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55955aee:::benchmark_data_smithi188_113957_object51519:head# 2024-04-23T16:33:39.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55959e84:::benchmark_data_smithi188_113957_object124824:head# 2024-04-23T16:33:39.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:559602b3:::benchmark_data_smithi188_113957_object103096:head# 2024-04-23T16:33:39.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55965540:::benchmark_data_smithi188_113957_object24944:head# 2024-04-23T16:33:39.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5596d2de:::benchmark_data_smithi188_113957_object104757:head# 2024-04-23T16:33:39.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55971129:::benchmark_data_smithi188_113957_object140133:head# 2024-04-23T16:33:39.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55977503:::benchmark_data_smithi188_113957_object74671:head# 2024-04-23T16:33:39.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5598148f:::benchmark_data_smithi188_113957_object128538:head# 2024-04-23T16:33:39.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:559847e6:::benchmark_data_smithi188_113957_object153492:head# 2024-04-23T16:33:39.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5598631a:::benchmark_data_smithi188_113957_object61132:head# 2024-04-23T16:33:39.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5598fde9:::benchmark_data_smithi188_113957_object95136:head# 2024-04-23T16:33:39.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:559a6b15:::benchmark_data_smithi188_113957_object9769:head# 2024-04-23T16:33:39.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:559aeff5:::benchmark_data_smithi188_113957_object77967:head# 2024-04-23T16:33:39.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:559b0083:::benchmark_data_smithi188_113957_object9211:head# 2024-04-23T16:33:39.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:559c03a9:::benchmark_data_smithi188_113957_object133302:head# 2024-04-23T16:33:39.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:559c80c2:::benchmark_data_smithi188_113957_object114784:head# 2024-04-23T16:33:39.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:559ca98a:::benchmark_data_smithi188_113957_object74885:head# 2024-04-23T16:33:39.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:559cfff7:::benchmark_data_smithi188_113957_object61113:head# 2024-04-23T16:33:39.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:559d233d:::benchmark_data_smithi188_113957_object61335:head# 2024-04-23T16:33:39.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:559d98f9:::benchmark_data_smithi188_113957_object87120:head# 2024-04-23T16:33:39.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:559ec83e:::benchmark_data_smithi188_113957_object59902:head# 2024-04-23T16:33:39.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:559ef89c:::benchmark_data_smithi188_113957_object127966:head# 2024-04-23T16:33:39.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:559f26c3:::benchmark_data_smithi188_113957_object22343:head# 2024-04-23T16:33:39.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a0203f:::benchmark_data_smithi188_113957_object70407:head# 2024-04-23T16:33:39.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a05c1d:::benchmark_data_smithi188_113957_object89516:head# 2024-04-23T16:33:39.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a05dc4:::benchmark_data_smithi188_113957_object118225:head# 2024-04-23T16:33:39.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a0960b:::benchmark_data_smithi188_113957_object81110:head# 2024-04-23T16:33:39.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a0abc2:::benchmark_data_smithi188_113957_object105843:head# 2024-04-23T16:33:39.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a0bc02:::benchmark_data_smithi188_113957_object96257:head# 2024-04-23T16:33:39.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a15c85:::benchmark_data_smithi188_113957_object131827:head# 2024-04-23T16:33:39.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a1b715:::benchmark_data_smithi188_113957_object53855:head# 2024-04-23T16:33:39.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a21a1f:::benchmark_data_smithi188_113957_object135907:head# 2024-04-23T16:33:39.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a22b02:::benchmark_data_smithi188_113957_object103132:head# 2024-04-23T16:33:39.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a231b0:::benchmark_data_smithi188_113957_object52499:head# 2024-04-23T16:33:39.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a2442e:::benchmark_data_smithi188_113957_object100712:head# 2024-04-23T16:33:39.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a2594b:::benchmark_data_smithi188_113957_object88742:head# 2024-04-23T16:33:39.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a28726:::benchmark_data_smithi188_113957_object61840:head# 2024-04-23T16:33:39.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a30164:::benchmark_data_smithi188_113957_object38214:head# 2024-04-23T16:33:39.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a42b8f:::benchmark_data_smithi188_113957_object143543:head# 2024-04-23T16:33:39.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a4b833:::benchmark_data_smithi188_113957_object97428:head# 2024-04-23T16:33:39.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a4de2d:::benchmark_data_smithi188_113957_object43072:head# 2024-04-23T16:33:39.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a57f6d:::benchmark_data_smithi188_113957_object13434:head# 2024-04-23T16:33:39.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a5867e:::benchmark_data_smithi188_113957_object125714:head# 2024-04-23T16:33:39.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a59836:::benchmark_data_smithi188_113957_object82094:head# 2024-04-23T16:33:39.320 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:33:39.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a63cdc:::benchmark_data_smithi188_113957_object129374:head# 2024-04-23T16:33:39.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a68900:::benchmark_data_smithi188_113957_object15432:head# 2024-04-23T16:33:39.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a6c3cb:::benchmark_data_smithi188_113957_object91026:head# 2024-04-23T16:33:39.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a6e654:::benchmark_data_smithi188_113957_object26211:head# 2024-04-23T16:33:39.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a769fd:::benchmark_data_smithi188_113957_object132028:head# 2024-04-23T16:33:39.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a77bca:::benchmark_data_smithi188_113957_object19099:head# 2024-04-23T16:33:39.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a79d93:::benchmark_data_smithi188_113957_object132187:head# 2024-04-23T16:33:39.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a80d03:::benchmark_data_smithi188_113957_object66081:head# 2024-04-23T16:33:39.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a845ef:::benchmark_data_smithi188_113957_object102748:head# 2024-04-23T16:33:39.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a9695d:::benchmark_data_smithi188_113957_object121588:head# 2024-04-23T16:33:39.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a99739:::benchmark_data_smithi188_113957_object148035:head# 2024-04-23T16:33:39.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55a9995f:::benchmark_data_smithi188_113957_object122083:head# 2024-04-23T16:33:39.326 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55aabb10:::benchmark_data_smithi188_113957_object34064:head# 2024-04-23T16:33:39.326 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55ab2598:::benchmark_data_smithi188_113957_object55715:head# 2024-04-23T16:33:39.326 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55abc2ed:::benchmark_data_smithi188_113957_object79355:head# 2024-04-23T16:33:39.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55aca8fc:::benchmark_data_smithi188_113957_object50063:head# 2024-04-23T16:33:39.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55ad1f61:::benchmark_data_smithi188_113957_object61563:head# 2024-04-23T16:33:39.329 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55ad3dce:::benchmark_data_smithi188_113957_object19494:head# 2024-04-23T16:33:39.329 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55ad6e2f:::benchmark_data_smithi188_113957_object57947:head# 2024-04-23T16:33:39.329 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55ada704:::benchmark_data_smithi188_113957_object9275:head# 2024-04-23T16:33:39.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55ae68d1:::benchmark_data_smithi188_113957_object22992:head# 2024-04-23T16:33:39.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55ae99bf:::benchmark_data_smithi188_113957_object21072:head# 2024-04-23T16:33:39.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55aec01a:::benchmark_data_smithi188_113957_object621:head# 2024-04-23T16:33:39.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55aee280:::benchmark_data_smithi188_113957_object124031:head# 2024-04-23T16:33:39.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b006d6:::benchmark_data_smithi188_113957_object68480:head# 2024-04-23T16:33:39.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b0e127:::benchmark_data_smithi188_113957_object1082:head# 2024-04-23T16:33:39.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b0e301:::benchmark_data_smithi188_113957_object135111:head# 2024-04-23T16:33:39.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b17406:::benchmark_data_smithi188_113957_object64394:head# 2024-04-23T16:33:39.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b19abb:::benchmark_data_smithi188_113957_object8912:head# 2024-04-23T16:33:39.334 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b1e538:::benchmark_data_smithi188_113957_object4117:head# 2024-04-23T16:33:39.334 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b20615:::benchmark_data_smithi188_113957_object82804:head# 2024-04-23T16:33:39.334 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b206b5:::benchmark_data_smithi188_113957_object15914:head# 2024-04-23T16:33:39.334 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b2275d:::benchmark_data_smithi188_113957_object148374:head# 2024-04-23T16:33:39.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b2c57c:::benchmark_data_smithi188_113957_object146457:head# 2024-04-23T16:33:39.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b2ca42:::benchmark_data_smithi188_113957_object124717:head# 2024-04-23T16:33:39.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b34495:::benchmark_data_smithi188_113957_object82114:head# 2024-04-23T16:33:39.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b4131e:::benchmark_data_smithi188_113957_object86193:head# 2024-04-23T16:33:39.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b4544b:::benchmark_data_smithi188_113957_object150012:head# 2024-04-23T16:33:39.337 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b493fb:::benchmark_data_smithi188_113957_object37492:head# 2024-04-23T16:33:39.337 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b4d9fa:::benchmark_data_smithi188_113957_object33234:head# 2024-04-23T16:33:39.337 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b62bf7:::benchmark_data_smithi188_113957_object115367:head# 2024-04-23T16:33:39.338 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b63650:::benchmark_data_smithi188_113957_object63348:head# 2024-04-23T16:33:39.338 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b67d3a:::benchmark_data_smithi188_113957_object146961:head# 2024-04-23T16:33:39.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b6c2cc:::benchmark_data_smithi188_113957_object97921:head# 2024-04-23T16:33:39.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b7353e:::benchmark_data_smithi188_113957_object127204:head# 2024-04-23T16:33:39.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b737a3:::benchmark_data_smithi188_113957_object13423:head# 2024-04-23T16:33:39.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b7dd32:::benchmark_data_smithi188_113957_object13332:head# 2024-04-23T16:33:39.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b7e060:::benchmark_data_smithi188_113957_object153212:head# 2024-04-23T16:33:39.341 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b7f894:::benchmark_data_smithi188_113957_object11913:head# 2024-04-23T16:33:39.341 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b8a806:::benchmark_data_smithi188_113957_object24457:head# 2024-04-23T16:33:39.342 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:39.343 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:33:39.343 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:33:39.344 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:33:39.344 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:33:39.344 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:33:39.344 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:33:39.344 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:33:39.344 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:33:39.344 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:33:39.344 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:33:39.344 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:33:39.344 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:33:39.344 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:33:39.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b953f9:::benchmark_data_smithi188_113957_object109662:head# 2024-04-23T16:33:39.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b9a209:::benchmark_data_smithi188_113957_object68629:head# 2024-04-23T16:33:39.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b9bb51:::benchmark_data_smithi188_113957_object94575:head# 2024-04-23T16:33:39.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b9bd9d:::benchmark_data_smithi188_113957_object85371:head# 2024-04-23T16:33:39.346 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b9e2e0:::benchmark_data_smithi188_113957_object138391:head# 2024-04-23T16:33:39.346 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b9ea20:::benchmark_data_smithi188_113957_object46481:head# 2024-04-23T16:33:39.346 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55b9ebc0:::benchmark_data_smithi188_113957_object141941:head# 2024-04-23T16:33:39.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55baeaa5:::benchmark_data_smithi188_113957_object18246:head# 2024-04-23T16:33:39.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55bb3633:::benchmark_data_smithi188_113957_object126810:head# 2024-04-23T16:33:39.348 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55bc28c9:::benchmark_data_smithi188_113957_object5155:head# 2024-04-23T16:33:39.348 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55bc6882:::benchmark_data_smithi188_113957_object57746:head# 2024-04-23T16:33:39.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55bc9ce0:::benchmark_data_smithi188_113957_object121980:head# 2024-04-23T16:33:39.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55bcda90:::benchmark_data_smithi188_113957_object16366:head# 2024-04-23T16:33:39.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55bcfa1e:::benchmark_data_smithi188_113957_object16747:head# 2024-04-23T16:33:39.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55bdbe96:::benchmark_data_smithi188_113957_object57321:head# 2024-04-23T16:33:39.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55be10b4:::benchmark_data_smithi188_113957_object103633:head# 2024-04-23T16:33:39.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55be153c:::benchmark_data_smithi188_113957_object146832:head# 2024-04-23T16:33:39.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55be4872:::benchmark_data_smithi188_113957_object9953:head# 2024-04-23T16:33:39.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55be723e:::benchmark_data_smithi188_113957_object46387:head# 2024-04-23T16:33:39.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55be9037:::benchmark_data_smithi188_113957_object23217:head# 2024-04-23T16:33:39.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55befb1d:::benchmark_data_smithi188_113957_object2577:head# 2024-04-23T16:33:39.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55bf82a2:::benchmark_data_smithi188_113957_object109521:head# 2024-04-23T16:33:39.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c106b2:::benchmark_data_smithi188_113957_object108011:head# 2024-04-23T16:33:39.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c163b8:::benchmark_data_smithi188_113957_object24701:head# 2024-04-23T16:33:39.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c22299:::benchmark_data_smithi188_113957_object18876:head# 2024-04-23T16:33:39.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c2763e:::benchmark_data_smithi188_113957_object57765:head# 2024-04-23T16:33:39.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c2afc5:::benchmark_data_smithi188_113957_object119928:head# 2024-04-23T16:33:39.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c2b4a8:::benchmark_data_smithi188_113957_object124053:head# 2024-04-23T16:33:39.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c3663e:::benchmark_data_smithi188_113957_object149744:head# 2024-04-23T16:33:39.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c4292e:::benchmark_data_smithi188_113957_object26749:head# 2024-04-23T16:33:39.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c522d0:::benchmark_data_smithi188_113957_object149314:head# 2024-04-23T16:33:39.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c5347d:::benchmark_data_smithi188_113957_object141538:head# 2024-04-23T16:33:39.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c65a31:::benchmark_data_smithi188_113957_object133433:head# 2024-04-23T16:33:39.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c6ca9d:::benchmark_data_smithi188_113957_object113185:head# 2024-04-23T16:33:39.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c6d5a5:::benchmark_data_smithi188_113957_object112012:head# 2024-04-23T16:33:39.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c73292:::benchmark_data_smithi188_113957_object121883:head# 2024-04-23T16:33:39.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c7490a:::benchmark_data_smithi188_113957_object33229:head# 2024-04-23T16:33:39.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c7b792:::benchmark_data_smithi188_113957_object60575:head# 2024-04-23T16:33:39.361 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c84c78:::benchmark_data_smithi188_113957_object95325:head# 2024-04-23T16:33:39.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c86916:::benchmark_data_smithi188_113957_object72273:head# 2024-04-23T16:33:39.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c897df:::benchmark_data_smithi188_113957_object59830:head# 2024-04-23T16:33:39.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c918fb:::benchmark_data_smithi188_113957_object21718:head# 2024-04-23T16:33:39.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c91cf7:::benchmark_data_smithi188_113957_object120313:head# 2024-04-23T16:33:39.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55c97a0b:::benchmark_data_smithi188_113957_object19283:head# 2024-04-23T16:33:39.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55cab02a:::benchmark_data_smithi188_113957_object79609:head# 2024-04-23T16:33:39.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55cad639:::benchmark_data_smithi188_113957_object2020:head# 2024-04-23T16:33:39.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55cbc0ff:::benchmark_data_smithi188_113957_object81701:head# 2024-04-23T16:33:39.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55cbd04d:::benchmark_data_smithi188_113957_object109405:head# 2024-04-23T16:33:39.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55cc10ac:::benchmark_data_smithi188_113957_object53016:head# 2024-04-23T16:33:39.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55cc157d:::benchmark_data_smithi188_113957_object14019:head# 2024-04-23T16:33:39.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55cc1ae2:::benchmark_data_smithi188_113957_object70226:head# 2024-04-23T16:33:39.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55cd27c0:::benchmark_data_smithi188_113957_object54152:head# 2024-04-23T16:33:39.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55cd351c:::benchmark_data_smithi188_113957_object131705:head# 2024-04-23T16:33:39.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55cd3686:::benchmark_data_smithi188_113957_object4027:head# 2024-04-23T16:33:39.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55cd406e:::benchmark_data_smithi188_113957_object79199:head# 2024-04-23T16:33:39.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55cdccf1:::benchmark_data_smithi188_113957_object149207:head# 2024-04-23T16:33:39.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55cdfeda:::benchmark_data_smithi188_113957_object9418:head# 2024-04-23T16:33:39.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55ce4c66:::benchmark_data_smithi188_113957_object78208:head# 2024-04-23T16:33:39.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55ce526a:::benchmark_data_smithi188_113957_object74650:head# 2024-04-23T16:33:39.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55ce7c93:::benchmark_data_smithi188_113957_object12838:head# 2024-04-23T16:33:39.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55cef9db:::benchmark_data_smithi188_113957_object30491:head# 2024-04-23T16:33:39.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55cf46d8:::benchmark_data_smithi188_113957_object126163:head# 2024-04-23T16:33:39.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55cf5451:::benchmark_data_smithi188_113957_object135548:head# 2024-04-23T16:33:39.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d025cf:::benchmark_data_smithi188_113957_object48803:head# 2024-04-23T16:33:39.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d04607:::benchmark_data_smithi188_113957_object143831:head# 2024-04-23T16:33:39.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d21110:::benchmark_data_smithi188_113957_object128731:head# 2024-04-23T16:33:39.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d24259:::benchmark_data_smithi188_113957_object57249:head# 2024-04-23T16:33:39.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d25280:::benchmark_data_smithi188_113957_object151879:head# 2024-04-23T16:33:39.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d2f932:::benchmark_data_smithi188_113957_object96355:head# 2024-04-23T16:33:39.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d33497:::benchmark_data_smithi188_113957_object115346:head# 2024-04-23T16:33:39.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d36012:::benchmark_data_smithi188_113957_object90603:head# 2024-04-23T16:33:39.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d362d0:::benchmark_data_smithi188_113957_object90594:head# 2024-04-23T16:33:39.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d3c0a9:::benchmark_data_smithi188_113957_object25378:head# 2024-04-23T16:33:39.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d43111:::benchmark_data_smithi188_113957_object64182:head# 2024-04-23T16:33:39.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d53ad8:::benchmark_data_smithi188_113957_object74508:head# 2024-04-23T16:33:39.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d5618e:::benchmark_data_smithi188_113957_object111161:head# 2024-04-23T16:33:39.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d5dc80:::benchmark_data_smithi188_113957_object83270:head# 2024-04-23T16:33:39.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d6d6f1:::benchmark_data_smithi188_113957_object8386:head# 2024-04-23T16:33:39.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d72810:::benchmark_data_smithi188_113957_object140018:head# 2024-04-23T16:33:39.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d88991:::benchmark_data_smithi188_113957_object138685:head# 2024-04-23T16:33:39.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d9378a:::benchmark_data_smithi188_113957_object34849:head# 2024-04-23T16:33:39.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d98c31:::benchmark_data_smithi188_113957_object42579:head# 2024-04-23T16:33:39.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55d9cf36:::benchmark_data_smithi188_113957_object152537:head# 2024-04-23T16:33:39.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55daee72:::benchmark_data_smithi188_113957_object117172:head# 2024-04-23T16:33:39.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55db5aaa:::benchmark_data_smithi188_113957_object100648:head# 2024-04-23T16:33:39.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55db6984:::benchmark_data_smithi188_113957_object91387:head# 2024-04-23T16:33:39.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55dba7e4:::benchmark_data_smithi188_113957_object92310:head# 2024-04-23T16:33:39.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55dbcd55:::benchmark_data_smithi188_113957_object84042:head# 2024-04-23T16:33:39.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55dbd249:::benchmark_data_smithi188_113957_object129531:head# 2024-04-23T16:33:39.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55dc16c4:::benchmark_data_smithi188_113957_object94053:head# 2024-04-23T16:33:39.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55dc9e78:::benchmark_data_smithi188_113957_object61355:head# 2024-04-23T16:33:39.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55de953b:::benchmark_data_smithi188_113957_object69033:head# 2024-04-23T16:33:39.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55df0938:::benchmark_data_smithi188_113957_object27999:head# 2024-04-23T16:33:39.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e02565:::benchmark_data_smithi188_113957_object38433:head# 2024-04-23T16:33:39.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e124aa:::benchmark_data_smithi188_113957_object112407:head# 2024-04-23T16:33:39.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e1d26d:::benchmark_data_smithi188_113957_object69886:head# 2024-04-23T16:33:39.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e21130:::benchmark_data_smithi188_113957_object99966:head# 2024-04-23T16:33:39.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e23a1e:::benchmark_data_smithi188_113957_object72177:head# 2024-04-23T16:33:39.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e2776b:::benchmark_data_smithi188_113957_object74566:head# 2024-04-23T16:33:39.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e290f2:::benchmark_data_smithi188_113957_object31581:head# 2024-04-23T16:33:39.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e2ebb2:::benchmark_data_smithi188_113957_object128440:head# 2024-04-23T16:33:39.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e2f2e4:::benchmark_data_smithi188_113957_object141699:head# 2024-04-23T16:33:39.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e30b32:::benchmark_data_smithi188_113957_object97456:head# 2024-04-23T16:33:39.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e3136f:::benchmark_data_smithi188_113957_object131138:head# 2024-04-23T16:33:39.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e3784e:::benchmark_data_smithi188_113957_object13431:head# 2024-04-23T16:33:39.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e416e7:::benchmark_data_smithi188_113957_object123641:head# 2024-04-23T16:33:39.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e42442:::benchmark_data_smithi188_113957_object90477:head# 2024-04-23T16:33:39.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e45227:::benchmark_data_smithi188_113957_object123091:head# 2024-04-23T16:33:39.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e4b39a:::benchmark_data_smithi188_113957_object56618:head# 2024-04-23T16:33:39.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e4c4ed:::benchmark_data_smithi188_113957_object3842:head# 2024-04-23T16:33:39.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e4ec98:::benchmark_data_smithi188_113957_object29310:head# 2024-04-23T16:33:39.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e65137:::benchmark_data_smithi188_113957_object568:head# 2024-04-23T16:33:39.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e74d09:::benchmark_data_smithi188_113957_object29791:head# 2024-04-23T16:33:39.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e7c116:::benchmark_data_smithi188_113957_object22386:head# 2024-04-23T16:33:39.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e84c2f:::benchmark_data_smithi188_113957_object66796:head# 2024-04-23T16:33:39.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e86bee:::benchmark_data_smithi188_113957_object124602:head# 2024-04-23T16:33:39.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e88bfa:::benchmark_data_smithi188_113957_object121643:head# 2024-04-23T16:33:39.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e8a4c0:::benchmark_data_smithi188_113957_object133940:head# 2024-04-23T16:33:39.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e9388e:::benchmark_data_smithi188_113957_object51097:head# 2024-04-23T16:33:39.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e989e9:::benchmark_data_smithi188_113957_object128022:head# 2024-04-23T16:33:39.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e9ea8c:::benchmark_data_smithi188_113957_object50243:head# 2024-04-23T16:33:39.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55e9f07a:::benchmark_data_smithi188_113957_object51946:head# 2024-04-23T16:33:39.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55eacc0d:::benchmark_data_smithi188_113957_object69984:head# 2024-04-23T16:33:39.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55ec3820:::benchmark_data_smithi188_113957_object141796:head# 2024-04-23T16:33:39.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55ec4faf:::benchmark_data_smithi188_113957_object105821:head# 2024-04-23T16:33:39.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55ed3193:::benchmark_data_smithi188_113957_object143454:head# 2024-04-23T16:33:39.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55edb98a:::benchmark_data_smithi188_113957_object41603:head# 2024-04-23T16:33:39.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55ee2a05:::benchmark_data_smithi188_113957_object86157:head# 2024-04-23T16:33:39.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55ee4ed9:::benchmark_data_smithi188_113957_object118322:head# 2024-04-23T16:33:39.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55ef2902:::benchmark_data_smithi188_113957_object74070:head# 2024-04-23T16:33:39.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55ef8b74:::benchmark_data_smithi188_113957_object84333:head# 2024-04-23T16:33:39.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55efe33f:::benchmark_data_smithi188_113957_object72314:head# 2024-04-23T16:33:39.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55f03593:::benchmark_data_smithi188_113957_object21832:head# 2024-04-23T16:33:39.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55f0e710:::benchmark_data_smithi188_113957_object109209:head# 2024-04-23T16:33:39.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55f1c131:::benchmark_data_smithi188_113957_object22567:head# 2024-04-23T16:33:39.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55f1e1f0:::benchmark_data_smithi188_113957_object23728:head# 2024-04-23T16:33:39.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55f1fa27:::benchmark_data_smithi188_113957_object86755:head# 2024-04-23T16:33:39.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55f23b1f:::benchmark_data_smithi188_113957_object13448:head# 2024-04-23T16:33:39.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55f2984d:::benchmark_data_smithi188_113957_object33706:head# 2024-04-23T16:33:39.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55f38810:::benchmark_data_smithi188_113957_object131254:head# 2024-04-23T16:33:39.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55f3cced:::benchmark_data_smithi188_113957_object96559:head# 2024-04-23T16:33:39.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55f50ebb:::benchmark_data_smithi188_113957_object91171:head# 2024-04-23T16:33:39.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55f60c32:::benchmark_data_smithi188_113957_object144461:head# 2024-04-23T16:33:39.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55f67bac:::benchmark_data_smithi188_113957_object148291:head# 2024-04-23T16:33:39.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55f6d7cb:::benchmark_data_smithi188_113957_object132338:head# 2024-04-23T16:33:39.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55f7bb75:::benchmark_data_smithi188_113957_object27269:head# 2024-04-23T16:33:39.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55f84803:::benchmark_data_smithi188_113957_object26887:head# 2024-04-23T16:33:39.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55f85e79:::benchmark_data_smithi188_113957_object90820:head# 2024-04-23T16:33:39.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55f98632:::benchmark_data_smithi188_113957_object91291:head# 2024-04-23T16:33:39.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55fad332:::benchmark_data_smithi188_113957_object30242:head# 2024-04-23T16:33:39.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55fbb767:::benchmark_data_smithi188_113957_object124192:head# 2024-04-23T16:33:39.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55fbcb11:::benchmark_data_smithi188_113957_object42993:head# 2024-04-23T16:33:39.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55fc789a:::benchmark_data_smithi188_113957_object132587:head# 2024-04-23T16:33:39.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55fc78b7:::benchmark_data_smithi188_113957_object120800:head# 2024-04-23T16:33:39.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55fc7fe9:::benchmark_data_smithi188_113957_object226:head# 2024-04-23T16:33:39.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55fcb3e7:::benchmark_data_smithi188_113957_object142740:head# 2024-04-23T16:33:39.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55fd817e:::benchmark_data_smithi188_113957_object37644:head# 2024-04-23T16:33:39.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55fe0b06:::benchmark_data_smithi188_113957_object139566:head# 2024-04-23T16:33:39.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55fe4ed1:::benchmark_data_smithi188_113957_object48593:head# 2024-04-23T16:33:39.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55fe5230:::benchmark_data_smithi188_113957_object30932:head# 2024-04-23T16:33:39.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55fe5c44:::benchmark_data_smithi188_113957_object103760:head# 2024-04-23T16:33:39.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:55fe86fe:::benchmark_data_smithi188_113957_object56937:head# 2024-04-23T16:33:39.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56007a0f:::benchmark_data_smithi188_113957_object116:head# 2024-04-23T16:33:39.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5600c46d:::benchmark_data_smithi188_113957_object119366:head# 2024-04-23T16:33:39.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56010da3:::benchmark_data_smithi188_113957_object728:head# 2024-04-23T16:33:39.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5601225f:::benchmark_data_smithi188_113957_object22851:head# 2024-04-23T16:33:39.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5602497d:::benchmark_data_smithi188_113957_object111207:head# 2024-04-23T16:33:39.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5603c4f0:::benchmark_data_smithi188_113957_object50570:head# 2024-04-23T16:33:39.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56040f6f:::benchmark_data_smithi188_113957_object3972:head# 2024-04-23T16:33:39.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5605eda7:::benchmark_data_smithi188_113957_object110223:head# 2024-04-23T16:33:39.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56070803:::benchmark_data_smithi188_113957_object121564:head# 2024-04-23T16:33:39.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5607122a:::benchmark_data_smithi188_113957_object27928:head# 2024-04-23T16:33:39.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56073264:::benchmark_data_smithi188_113957_object3536:head# 2024-04-23T16:33:39.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56075e15:::benchmark_data_smithi188_113957_object125815:head# 2024-04-23T16:33:39.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:560815cb:::benchmark_data_smithi188_113957_object91465:head# 2024-04-23T16:33:39.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:560860e2:::benchmark_data_smithi188_113957_object40097:head# 2024-04-23T16:33:39.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5608bb94:::benchmark_data_smithi188_113957_object45236:head# 2024-04-23T16:33:39.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5608f7f2:::benchmark_data_smithi188_113957_object43359:head# 2024-04-23T16:33:39.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5609ba1a:::benchmark_data_smithi188_113957_object46480:head# 2024-04-23T16:33:39.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5609ed32:::benchmark_data_smithi188_113957_object7746:head# 2024-04-23T16:33:39.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:560a55a9:::benchmark_data_smithi188_113957_object100193:head# 2024-04-23T16:33:39.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:560b2ee9:::benchmark_data_smithi188_113957_object23567:head# 2024-04-23T16:33:39.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:560c9407:::benchmark_data_smithi188_113957_object116848:head# 2024-04-23T16:33:39.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:560ce526:::benchmark_data_smithi188_113957_object111472:head# 2024-04-23T16:33:39.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:560d8c7d:::benchmark_data_smithi188_113957_object39880:head# 2024-04-23T16:33:39.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:560dc1e6:::benchmark_data_smithi188_113957_object25656:head# 2024-04-23T16:33:39.431 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:560dc54d:::benchmark_data_smithi188_113957_object14457:head# 2024-04-23T16:33:39.431 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:560e3a41:::benchmark_data_smithi188_113957_object53308:head# 2024-04-23T16:33:39.431 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:560e7cb8:::benchmark_data_smithi188_113957_object45117:head# 2024-04-23T16:33:39.431 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:560e7d96:::benchmark_data_smithi188_113957_object49444:head# 2024-04-23T16:33:39.432 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:560fe1ce:::benchmark_data_smithi188_113957_object88233:head# 2024-04-23T16:33:39.432 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56119de1:::benchmark_data_smithi188_113957_object66808:head# 2024-04-23T16:33:39.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5611ff46:::benchmark_data_smithi188_113957_object87008:head# 2024-04-23T16:33:39.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:561212ef:::benchmark_data_smithi188_113957_object112648:head# 2024-04-23T16:33:39.434 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5612d56d:::benchmark_data_smithi188_113957_object63608:head# 2024-04-23T16:33:39.434 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56130bf1:::benchmark_data_smithi188_113957_object71325:head# 2024-04-23T16:33:39.435 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56136f60:::benchmark_data_smithi188_113957_object19270:head# 2024-04-23T16:33:39.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56138a59:::benchmark_data_smithi188_113957_object90784:head# 2024-04-23T16:33:39.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5613d951:::benchmark_data_smithi188_113957_object108653:head# 2024-04-23T16:33:39.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:561464c7:::benchmark_data_smithi188_113957_object23502:head# 2024-04-23T16:33:39.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56154975:::benchmark_data_smithi188_113957_object136241:head# 2024-04-23T16:33:39.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5615d7c6:::benchmark_data_smithi188_113957_object122356:head# 2024-04-23T16:33:39.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5615e949:::benchmark_data_smithi188_113957_object32098:head# 2024-04-23T16:33:39.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5616b709:::benchmark_data_smithi188_113957_object148385:head# 2024-04-23T16:33:39.438 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56178dd2:::benchmark_data_smithi188_113957_object43820:head# 2024-04-23T16:33:39.438 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:561795e3:::benchmark_data_smithi188_113957_object118326:head# 2024-04-23T16:33:39.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56181559:::benchmark_data_smithi188_113957_object9458:head# 2024-04-23T16:33:39.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5618ad7e:::benchmark_data_smithi188_113957_object3606:head# 2024-04-23T16:33:39.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5618cfc5:::benchmark_data_smithi188_113957_object54236:head# 2024-04-23T16:33:39.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:561921bd:::benchmark_data_smithi188_113957_object57365:head# 2024-04-23T16:33:39.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5619c115:::benchmark_data_smithi188_113957_object61055:head# 2024-04-23T16:33:39.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:561b465c:::benchmark_data_smithi188_113957_object96449:head# 2024-04-23T16:33:39.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:561b5b3a:::benchmark_data_smithi188_113957_object15365:head# 2024-04-23T16:33:39.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:561bf969:::benchmark_data_smithi188_113957_object123408:head# 2024-04-23T16:33:39.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:561bfd0c:::benchmark_data_smithi188_113957_object56879:head# 2024-04-23T16:33:39.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:561cc869:::benchmark_data_smithi188_113957_object62476:head# 2024-04-23T16:33:39.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:561d9bbc:::benchmark_data_smithi188_113957_object40708:head# 2024-04-23T16:33:39.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:561dd015:::benchmark_data_smithi188_113957_object105215:head# 2024-04-23T16:33:39.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:561ddaeb:::benchmark_data_smithi188_113957_object30285:head# 2024-04-23T16:33:39.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:561e2b3d:::benchmark_data_smithi188_113957_object92749:head# 2024-04-23T16:33:39.446 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:561e8df9:::benchmark_data_smithi188_113957_object52891:head# 2024-04-23T16:33:39.446 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:561ed0f5:::benchmark_data_smithi188_113957_object71134:head# 2024-04-23T16:33:39.446 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:561edccd:::benchmark_data_smithi188_113957_object115764:head# 2024-04-23T16:33:39.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:561f020d:::benchmark_data_smithi188_113957_object137218:head# 2024-04-23T16:33:39.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:561f405a:::benchmark_data_smithi188_113957_object4270:head# 2024-04-23T16:33:39.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:561fd1bf:::benchmark_data_smithi188_113957_object68854:head# 2024-04-23T16:33:39.448 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56211d99:::benchmark_data_smithi188_113957_object42444:head# 2024-04-23T16:33:39.448 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56223f99:::benchmark_data_smithi188_113957_object28187:head# 2024-04-23T16:33:39.449 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5622e4b1:::benchmark_data_smithi188_113957_object93522:head# 2024-04-23T16:33:39.449 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5622e500:::benchmark_data_smithi188_113957_object124701:head# 2024-04-23T16:33:39.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5622ff3f:::benchmark_data_smithi188_113957_object129976:head# 2024-04-23T16:33:39.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56235d9d:::benchmark_data_smithi188_113957_object78888:head# 2024-04-23T16:33:39.451 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5623bfdd:::benchmark_data_smithi188_113957_object35045:head# 2024-04-23T16:33:39.451 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56240ccb:::benchmark_data_smithi188_113957_object122964:head# 2024-04-23T16:33:39.451 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56242b6f:::benchmark_data_smithi188_113957_object81648:head# 2024-04-23T16:33:39.452 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56242d46:::benchmark_data_smithi188_113957_object31820:head# 2024-04-23T16:33:39.452 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5624955f:::benchmark_data_smithi188_113957_object105853:head# 2024-04-23T16:33:39.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5624c907:::benchmark_data_smithi188_113957_object4347:head# 2024-04-23T16:33:39.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5624d661:::benchmark_data_smithi188_113957_object118846:head# 2024-04-23T16:33:39.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56251457:::benchmark_data_smithi188_113957_object72296:head# 2024-04-23T16:33:39.454 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56254e65:::benchmark_data_smithi188_113957_object116467:head# 2024-04-23T16:33:39.454 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56255051:::benchmark_data_smithi188_113957_object111435:head# 2024-04-23T16:33:39.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56258d75:::benchmark_data_smithi188_113957_object62051:head# 2024-04-23T16:33:39.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:562599f8:::benchmark_data_smithi188_113957_object84847:head# 2024-04-23T16:33:39.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5625bc31:::benchmark_data_smithi188_113957_object40811:head# 2024-04-23T16:33:39.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56263799:::benchmark_data_smithi188_113957_object68612:head# 2024-04-23T16:33:39.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:562673cb:::benchmark_data_smithi188_113957_object140100:head# 2024-04-23T16:33:39.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5626dc62:::benchmark_data_smithi188_113957_object85528:head# 2024-04-23T16:33:39.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56274713:::benchmark_data_smithi188_113957_object81449:head# 2024-04-23T16:33:39.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5627a2da:::benchmark_data_smithi188_113957_object80604:head# 2024-04-23T16:33:39.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5628dc4e:::benchmark_data_smithi188_113957_object57304:head# 2024-04-23T16:33:39.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:562961ee:::benchmark_data_smithi188_113957_object73890:head# 2024-04-23T16:33:39.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5629e79a:::benchmark_data_smithi188_113957_object14459:head# 2024-04-23T16:33:39.460 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:562a1a56:::benchmark_data_smithi188_113957_object101915:head# 2024-04-23T16:33:39.460 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:562a5f60:::benchmark_data_smithi188_113957_object148714:head# 2024-04-23T16:33:39.460 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:562accad:::benchmark_data_smithi188_113957_object90888:head# 2024-04-23T16:33:39.461 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:562bba7c:::benchmark_data_smithi188_113957_object83:head# 2024-04-23T16:33:39.461 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:562c69d0:::benchmark_data_smithi188_113957_object71501:head# 2024-04-23T16:33:39.462 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:562d4b5e:::benchmark_data_smithi188_113957_object4650:head# 2024-04-23T16:33:39.462 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:562d634d:::benchmark_data_smithi188_113957_object81362:head# 2024-04-23T16:33:39.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:562e6d12:::benchmark_data_smithi188_113957_object92584:head# 2024-04-23T16:33:39.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:562eed03:::benchmark_data_smithi188_113957_object38847:head# 2024-04-23T16:33:39.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:562f4a50:::benchmark_data_smithi188_113957_object55129:head# 2024-04-23T16:33:39.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:562f4cd0:::benchmark_data_smithi188_113957_object55548:head# 2024-04-23T16:33:39.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:563008cb:::benchmark_data_smithi188_113957_object145146:head# 2024-04-23T16:33:39.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5630174e:::benchmark_data_smithi188_113957_object150232:head# 2024-04-23T16:33:39.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56302dd9:::benchmark_data_smithi188_113957_object105403:head# 2024-04-23T16:33:39.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56307301:::benchmark_data_smithi188_113957_object119230:head# 2024-04-23T16:33:39.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5630b99e:::benchmark_data_smithi188_113957_object123326:head# 2024-04-23T16:33:39.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5631138c:::benchmark_data_smithi188_113957_object69595:head# 2024-04-23T16:33:39.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:563197f2:::benchmark_data_smithi188_113957_object61414:head# 2024-04-23T16:33:39.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5631b9a9:::benchmark_data_smithi188_113957_object22862:head# 2024-04-23T16:33:39.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5632b01d:::benchmark_data_smithi188_113957_object78372:head# 2024-04-23T16:33:39.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5632e3b8:::benchmark_data_smithi188_113957_object137023:head# 2024-04-23T16:33:39.469 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56333091:::benchmark_data_smithi188_113957_object26082:head# 2024-04-23T16:33:39.469 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:563369cd:::benchmark_data_smithi188_113957_object89216:head# 2024-04-23T16:33:39.470 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56345351:::benchmark_data_smithi188_113957_object46659:head# 2024-04-23T16:33:39.470 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56348ec9:::benchmark_data_smithi188_113957_object119249:head# 2024-04-23T16:33:39.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56356ba1:::benchmark_data_smithi188_113957_object31922:head# 2024-04-23T16:33:39.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5635afa8:::benchmark_data_smithi188_113957_object143965:head# 2024-04-23T16:33:39.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56361e2d:::benchmark_data_smithi188_113957_object27209:head# 2024-04-23T16:33:39.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5636bc96:::benchmark_data_smithi188_113957_object7610:head# 2024-04-23T16:33:39.473 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5637083c:::benchmark_data_smithi188_113957_object145061:head# 2024-04-23T16:33:39.473 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56373e49:::benchmark_data_smithi188_113957_object91003:head# 2024-04-23T16:33:39.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:39 smithi188 ceph-mon[44101]: pgmap v1118: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 9.0 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 518 op/s; 27626/296312 objects degraded (9.323%); 7.0 MiB/s, 112 objects/s recovering 2024-04-23T16:33:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:39 smithi188 ceph-mon[44101]: osdmap e397: 12 total, 11 up, 9 in 2024-04-23T16:33:39.474 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5637baac:::benchmark_data_smithi188_113957_object115510:head# 2024-04-23T16:33:39.474 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:563812b5:::benchmark_data_smithi188_113957_object77668:head# 2024-04-23T16:33:39.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56386637:::benchmark_data_smithi188_113957_object57492:head# 2024-04-23T16:33:39.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:563a6d3e:::benchmark_data_smithi188_113957_object55783:head# 2024-04-23T16:33:39.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:563aac39:::benchmark_data_smithi188_113957_object29929:head# 2024-04-23T16:33:39.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:563b27b1:::benchmark_data_smithi188_113957_object123677:head# 2024-04-23T16:33:39.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:563b50d7:::benchmark_data_smithi188_113957_object86353:head# 2024-04-23T16:33:39.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:563b998f:::benchmark_data_smithi188_113957_object18392:head# 2024-04-23T16:33:39.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:563ca06d:::benchmark_data_smithi188_113957_object82389:head# 2024-04-23T16:33:39.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:563cea74:::benchmark_data_smithi188_113957_object67165:head# 2024-04-23T16:33:39.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:563cf643:::benchmark_data_smithi188_113957_object144112:head# 2024-04-23T16:33:39.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:563dd3aa:::benchmark_data_smithi188_113957_object62201:head# 2024-04-23T16:33:39.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:563e6321:::benchmark_data_smithi188_113957_object109432:head# 2024-04-23T16:33:39.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:563e90e0:::benchmark_data_smithi188_113957_object1221:head# 2024-04-23T16:33:39.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:563f1deb:::benchmark_data_smithi188_113957_object119852:head# 2024-04-23T16:33:39.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:563fd536:::benchmark_data_smithi188_113957_object148436:head# 2024-04-23T16:33:39.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:563ff608:::benchmark_data_smithi188_113957_object18841:head# 2024-04-23T16:33:39.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56429cf4:::benchmark_data_smithi188_113957_object118864:head# 2024-04-23T16:33:39.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56434432:::benchmark_data_smithi188_113957_object14936:head# 2024-04-23T16:33:39.481 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5643f662:::benchmark_data_smithi188_113957_object118477:head# 2024-04-23T16:33:39.481 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5644b7af:::benchmark_data_smithi188_113957_object76387:head# 2024-04-23T16:33:39.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56454df9:::benchmark_data_smithi188_113957_object22384:head# 2024-04-23T16:33:39.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5645cebb:::benchmark_data_smithi188_113957_object15013:head# 2024-04-23T16:33:39.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56466b99:::benchmark_data_smithi188_113957_object143155:head# 2024-04-23T16:33:39.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56469ba1:::benchmark_data_smithi188_113957_object36236:head# 2024-04-23T16:33:39.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5646b774:::benchmark_data_smithi188_113957_object95392:head# 2024-04-23T16:33:39.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5646efb9:::benchmark_data_smithi188_113957_object96290:head# 2024-04-23T16:33:39.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5647ecb9:::benchmark_data_smithi188_113957_object16331:head# 2024-04-23T16:33:39.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5648888f:::benchmark_data_smithi188_113957_object141235:head# 2024-04-23T16:33:39.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5648c2ad:::benchmark_data_smithi188_113957_object146657:head# 2024-04-23T16:33:39.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5648d5f8:::benchmark_data_smithi188_113957_object139425:head# 2024-04-23T16:33:39.486 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:564a4e83:::benchmark_data_smithi188_113957_object57892:head# 2024-04-23T16:33:39.486 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:564a9174:::benchmark_data_smithi188_113957_object105947:head# 2024-04-23T16:33:39.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:564ae3bf:::benchmark_data_smithi188_113957_object30744:head# 2024-04-23T16:33:39.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:564b9d1c:::benchmark_data_smithi188_113957_object111200:head# 2024-04-23T16:33:39.488 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:564b9e2f:::benchmark_data_smithi188_113957_object134840:head# 2024-04-23T16:33:39.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:564bb485:::benchmark_data_smithi188_113957_object26219:head# 2024-04-23T16:33:39.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:564ca2b9:::benchmark_data_smithi188_113957_object152940:head# 2024-04-23T16:33:39.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:564cb418:::benchmark_data_smithi188_113957_object75925:head# 2024-04-23T16:33:39.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:564d311f:::benchmark_data_smithi188_113957_object19364:head# 2024-04-23T16:33:39.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:564d75dc:::benchmark_data_smithi188_113957_object85190:head# 2024-04-23T16:33:39.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:564d83de:::benchmark_data_smithi188_113957_object89058:head# 2024-04-23T16:33:39.491 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:564de16c:::benchmark_data_smithi188_113957_object109839:head# 2024-04-23T16:33:39.491 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:564eb423:::benchmark_data_smithi188_113957_object86910:head# 2024-04-23T16:33:39.492 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:564ed4b8:::benchmark_data_smithi188_113957_object150698:head# 2024-04-23T16:33:39.492 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:564f05bd:::benchmark_data_smithi188_113957_object14956:head# 2024-04-23T16:33:39.493 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:564f158c:::benchmark_data_smithi188_113957_object22999:head# 2024-04-23T16:33:39.493 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:565008ff:::benchmark_data_smithi188_113957_object97193:head# 2024-04-23T16:33:39.493 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5650151b:::benchmark_data_smithi188_113957_object97126:head# 2024-04-23T16:33:39.494 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56509385:::benchmark_data_smithi188_113957_object64391:head# 2024-04-23T16:33:39.494 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56520254:::benchmark_data_smithi188_113957_object129012:head# 2024-04-23T16:33:39.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56524a9c:::benchmark_data_smithi188_113957_object141535:head# 2024-04-23T16:33:39.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56533af0:::benchmark_data_smithi188_113957_object57236:head# 2024-04-23T16:33:39.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:565385eb:::benchmark_data_smithi188_113957_object80729:head# 2024-04-23T16:33:39.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5653a980:::benchmark_data_smithi188_113957_object80720:head# 2024-04-23T16:33:39.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56542960:::benchmark_data_smithi188_113957_object7798:head# 2024-04-23T16:33:39.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56551dc2:::benchmark_data_smithi188_113957_object152385:head# 2024-04-23T16:33:39.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5655cb37:::benchmark_data_smithi188_113957_object70064:head# 2024-04-23T16:33:39.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56567d81:::benchmark_data_smithi188_113957_object32047:head# 2024-04-23T16:33:39.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5656951c:::benchmark_data_smithi188_113957_object93105:head# 2024-04-23T16:33:39.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5656b429:::benchmark_data_smithi188_113957_object12079:head# 2024-04-23T16:33:39.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:565708a0:::benchmark_data_smithi188_113957_object148401:head# 2024-04-23T16:33:39.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56570c03:::benchmark_data_smithi188_113957_object55603:head# 2024-04-23T16:33:39.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5657148a:::benchmark_data_smithi188_113957_object76254:head# 2024-04-23T16:33:39.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56577efe:::benchmark_data_smithi188_113957_object53041:head# 2024-04-23T16:33:39.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5657a866:::benchmark_data_smithi188_113957_object144421:head# 2024-04-23T16:33:39.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5657cd1f:::benchmark_data_smithi188_113957_object138664:head# 2024-04-23T16:33:39.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56580872:::benchmark_data_smithi188_113957_object1192:head# 2024-04-23T16:33:39.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56584ee5:::benchmark_data_smithi188_113957_object143661:head# 2024-04-23T16:33:39.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5658a9c3:::benchmark_data_smithi188_113957_object121575:head# 2024-04-23T16:33:39.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5658f855:::benchmark_data_smithi188_113957_object56271:head# 2024-04-23T16:33:39.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56593ac5:::benchmark_data_smithi188_113957_object50203:head# 2024-04-23T16:33:39.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5659814d:::benchmark_data_smithi188_113957_object102953:head# 2024-04-23T16:33:39.504 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:565a4a41:::benchmark_data_smithi188_113957_object151422:head# 2024-04-23T16:33:39.504 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:565a594c:::benchmark_data_smithi188_113957_object46702:head# 2024-04-23T16:33:39.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:565a5bbf:::benchmark_data_smithi188_113957_object65038:head# 2024-04-23T16:33:39.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:565bee12:::benchmark_data_smithi188_113957_object69235:head# 2024-04-23T16:33:39.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:565c56bf:::benchmark_data_smithi188_113957_object122458:head# 2024-04-23T16:33:39.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:565d588a:::benchmark_data_smithi188_113957_object151518:head# 2024-04-23T16:33:39.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:565d5ca4:::benchmark_data_smithi188_113957_object74791:head# 2024-04-23T16:33:39.507 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:565dbea7:::benchmark_data_smithi188_113957_object5738:head# 2024-04-23T16:33:39.507 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:565dd54d:::benchmark_data_smithi188_113957_object67679:head# 2024-04-23T16:33:39.508 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:565e0c78:::benchmark_data_smithi188_113957_object65316:head# 2024-04-23T16:33:39.509 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:565e11b7:::benchmark_data_smithi188_113957_object79392:head# 2024-04-23T16:33:39.509 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:565e3ffa:::benchmark_data_smithi188_113957_object127621:head# 2024-04-23T16:33:39.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:565ef85f:::benchmark_data_smithi188_113957_object82679:head# 2024-04-23T16:33:39.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:565f95d5:::benchmark_data_smithi188_113957_object105319:head# 2024-04-23T16:33:39.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:565f9da2:::benchmark_data_smithi188_113957_object77006:head# 2024-04-23T16:33:39.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:565fb883:::benchmark_data_smithi188_113957_object86242:head# 2024-04-23T16:33:39.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:566021a6:::benchmark_data_smithi188_113957_object30575:head# 2024-04-23T16:33:39.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5660740c:::benchmark_data_smithi188_113957_object51672:head# 2024-04-23T16:33:39.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5660c0d4:::benchmark_data_smithi188_113957_object69674:head# 2024-04-23T16:33:39.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56610d28:::benchmark_data_smithi188_113957_object44350:head# 2024-04-23T16:33:39.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:566137b5:::benchmark_data_smithi188_113957_object123117:head# 2024-04-23T16:33:39.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56622d8a:::benchmark_data_smithi188_113957_object60126:head# 2024-04-23T16:33:39.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56625df7:::benchmark_data_smithi188_113957_object134317:head# 2024-04-23T16:33:39.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56635f2b:::benchmark_data_smithi188_113957_object55094:head# 2024-04-23T16:33:39.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5663af1f:::benchmark_data_smithi188_113957_object77050:head# 2024-04-23T16:33:39.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56659ec8:::benchmark_data_smithi188_113957_object109864:head# 2024-04-23T16:33:39.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56662a2d:::benchmark_data_smithi188_113957_object140716:head# 2024-04-23T16:33:39.516 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56665a61:::benchmark_data_smithi188_113957_object106562:head# 2024-04-23T16:33:39.516 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5666a724:::benchmark_data_smithi188_113957_object52961:head# 2024-04-23T16:33:39.516 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56673363:::benchmark_data_smithi188_113957_object144553:head# 2024-04-23T16:33:39.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5667514f:::benchmark_data_smithi188_113957_object143083:head# 2024-04-23T16:33:39.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56677fd0:::benchmark_data_smithi188_113957_object120310:head# 2024-04-23T16:33:39.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:566785e3:::benchmark_data_smithi188_113957_object99716:head# 2024-04-23T16:33:39.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5667dce0:::benchmark_data_smithi188_113957_object84757:head# 2024-04-23T16:33:39.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56684e4a:::benchmark_data_smithi188_113957_object17586:head# 2024-04-23T16:33:39.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56687322:::benchmark_data_smithi188_113957_object110587:head# 2024-04-23T16:33:39.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5668a528:::benchmark_data_smithi188_113957_object110927:head# 2024-04-23T16:33:39.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5668c067:::benchmark_data_smithi188_113957_object47854:head# 2024-04-23T16:33:39.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5669111f:::benchmark_data_smithi188_113957_object3333:head# 2024-04-23T16:33:39.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56691abc:::benchmark_data_smithi188_113957_object118272:head# 2024-04-23T16:33:39.521 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5669c0b3:::benchmark_data_smithi188_113957_object67592:head# 2024-04-23T16:33:39.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:566a768e:::benchmark_data_smithi188_113957_object85205:head# 2024-04-23T16:33:39.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:566b2cad:::benchmark_data_smithi188_113957_object142227:head# 2024-04-23T16:33:39.523 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:566b83b6:::benchmark_data_smithi188_113957_object120323:head# 2024-04-23T16:33:39.523 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:566bb7ff:::benchmark_data_smithi188_113957_object52133:head# 2024-04-23T16:33:39.523 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:566c72ff:::benchmark_data_smithi188_113957_object117595:head# 2024-04-23T16:33:39.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:566dca15:::benchmark_data_smithi188_113957_object17835:head# 2024-04-23T16:33:39.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:566dcca0:::benchmark_data_smithi188_113957_object134272:head# 2024-04-23T16:33:39.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:566f64fc:::benchmark_data_smithi188_113957_object108153:head# 2024-04-23T16:33:39.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5670520d:::benchmark_data_smithi188_113957_object7400:head# 2024-04-23T16:33:39.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5670ca65:::benchmark_data_smithi188_113957_object55780:head# 2024-04-23T16:33:39.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5670fea8:::benchmark_data_smithi188_113957_object70357:head# 2024-04-23T16:33:39.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56719575:::benchmark_data_smithi188_113957_object18333:head# 2024-04-23T16:33:39.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5671a30b:::benchmark_data_smithi188_113957_object101611:head# 2024-04-23T16:33:39.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5671a5ae:::benchmark_data_smithi188_113957_object44880:head# 2024-04-23T16:33:39.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5671df73:::benchmark_data_smithi188_113957_object38744:head# 2024-04-23T16:33:39.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56736b38:::benchmark_data_smithi188_113957_object20150:head# 2024-04-23T16:33:39.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5673be0e:::benchmark_data_smithi188_113957_object114451:head# 2024-04-23T16:33:39.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5673c108:::benchmark_data_smithi188_113957_object55668:head# 2024-04-23T16:33:39.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5673ef40:::benchmark_data_smithi188_113957_object131469:head# 2024-04-23T16:33:39.531 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56744bea:::benchmark_data_smithi188_113957_object50234:head# 2024-04-23T16:33:39.531 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56745e81:::benchmark_data_smithi188_113957_object2163:head# 2024-04-23T16:33:39.531 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56746743:::benchmark_data_smithi188_113957_object96813:head# 2024-04-23T16:33:39.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:567468cc:::benchmark_data_smithi188_113957_object52498:head# 2024-04-23T16:33:39.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5674bd95:::benchmark_data_smithi188_113957_object33896:head# 2024-04-23T16:33:39.533 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5675af0a:::benchmark_data_smithi188_113957_object19883:head# 2024-04-23T16:33:39.533 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56761ac2:::benchmark_data_smithi188_113957_object84020:head# 2024-04-23T16:33:39.533 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56777f37:::benchmark_data_smithi188_113957_object80083:head# 2024-04-23T16:33:39.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56785465:::benchmark_data_smithi188_113957_object92227:head# 2024-04-23T16:33:39.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56790c94:::benchmark_data_smithi188_113957_object2624:head# 2024-04-23T16:33:39.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5679844e:::benchmark_data_smithi188_113957_object55586:head# 2024-04-23T16:33:39.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56798b87:::benchmark_data_smithi188_113957_object97416:head# 2024-04-23T16:33:39.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:567a68f1:::benchmark_data_smithi188_113957_object35009:head# 2024-04-23T16:33:39.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:567a8e84:::benchmark_data_smithi188_113957_object106357:head# 2024-04-23T16:33:39.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:567a8fd3:::benchmark_data_smithi188_113957_object14076:head# 2024-04-23T16:33:39.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:567b41dc:::benchmark_data_smithi188_113957_object25552:head# 2024-04-23T16:33:39.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:567b5f69:::benchmark_data_smithi188_113957_object40022:head# 2024-04-23T16:33:39.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:567b622a:::benchmark_data_smithi188_113957_object74966:head# 2024-04-23T16:33:39.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:567b6503:::benchmark_data_smithi188_113957_object28684:head# 2024-04-23T16:33:39.541 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:567bc9d0:::benchmark_data_smithi188_113957_object115930:head# 2024-04-23T16:33:39.541 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:567c0308:::benchmark_data_smithi188_113957_object103693:head# 2024-04-23T16:33:39.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:567d8eb5:::benchmark_data_smithi188_113957_object11937:head# 2024-04-23T16:33:39.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:567dcffe:::benchmark_data_smithi188_113957_object127791:head# 2024-04-23T16:33:39.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:567e5761:::benchmark_data_smithi188_113957_object41881:head# 2024-04-23T16:33:39.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568065fd:::benchmark_data_smithi188_113957_object83273:head# 2024-04-23T16:33:39.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5680aa92:::benchmark_data_smithi188_113957_object121093:head# 2024-04-23T16:33:39.544 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56826a54:::benchmark_data_smithi188_113957_object68655:head# 2024-04-23T16:33:39.544 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5682dc9b:::benchmark_data_smithi188_113957_object21958:head# 2024-04-23T16:33:39.544 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568390cd:::benchmark_data_smithi188_113957_object147601:head# 2024-04-23T16:33:39.545 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56840984:::benchmark_data_smithi188_113957_object19522:head# 2024-04-23T16:33:39.545 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568452eb:::benchmark_data_smithi188_113957_object47127:head# 2024-04-23T16:33:39.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56845539:::benchmark_data_smithi188_113957_object122734:head# 2024-04-23T16:33:39.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56848450:::benchmark_data_smithi188_113957_object24729:head# 2024-04-23T16:33:39.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568519d9:::benchmark_data_smithi188_113957_object43447:head# 2024-04-23T16:33:39.547 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56856229:::benchmark_data_smithi188_113957_object114669:head# 2024-04-23T16:33:39.547 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56863cb7:::benchmark_data_smithi188_113957_object151008:head# 2024-04-23T16:33:39.548 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5686a640:::benchmark_data_smithi188_113957_object123710:head# 2024-04-23T16:33:39.548 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568716ee:::benchmark_data_smithi188_113957_object131949:head# 2024-04-23T16:33:39.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56879a9e:::benchmark_data_smithi188_113957_object114956:head# 2024-04-23T16:33:39.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56884338:::benchmark_data_smithi188_113957_object17528:head# 2024-04-23T16:33:39.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5688f158:::benchmark_data_smithi188_113957_object11615:head# 2024-04-23T16:33:39.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56891b4d:::benchmark_data_smithi188_113957_object13976:head# 2024-04-23T16:33:39.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5689efff:::benchmark_data_smithi188_113957_object75031:head# 2024-04-23T16:33:39.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568a0739:::benchmark_data_smithi188_113957_object54553:head# 2024-04-23T16:33:39.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568a2de2:::benchmark_data_smithi188_113957_object140079:head# 2024-04-23T16:33:39.552 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568abc2e:::benchmark_data_smithi188_113957_object8583:head# 2024-04-23T16:33:39.552 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568abee3:::benchmark_data_smithi188_113957_object128:head# 2024-04-23T16:33:39.552 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568aff3e:::benchmark_data_smithi188_113957_object98783:head# 2024-04-23T16:33:39.553 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568b15d0:::benchmark_data_smithi188_113957_object55700:head# 2024-04-23T16:33:39.553 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568bfc8d:::benchmark_data_smithi188_113957_object102276:head# 2024-04-23T16:33:39.554 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568c1b56:::benchmark_data_smithi188_113957_object78299:head# 2024-04-23T16:33:39.554 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568c5391:::benchmark_data_smithi188_113957_object131331:head# 2024-04-23T16:33:39.555 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568c67c5:::benchmark_data_smithi188_113957_object84646:head# 2024-04-23T16:33:39.555 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568c9443:::benchmark_data_smithi188_113957_object53357:head# 2024-04-23T16:33:39.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568ce61e:::benchmark_data_smithi188_113957_object127630:head# 2024-04-23T16:33:39.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568d1cd9:::benchmark_data_smithi188_113957_object34214:head# 2024-04-23T16:33:39.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568d94a9:::benchmark_data_smithi188_113957_object38304:head# 2024-04-23T16:33:39.559 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568e0dd3:::benchmark_data_smithi188_113957_object115104:head# 2024-04-23T16:33:39.560 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568e40d6:::benchmark_data_smithi188_113957_object58586:head# 2024-04-23T16:33:39.560 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568e9c9c:::benchmark_data_smithi188_113957_object103906:head# 2024-04-23T16:33:39.560 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568f0c99:::benchmark_data_smithi188_113957_object33924:head# 2024-04-23T16:33:39.560 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568fbced:::benchmark_data_smithi188_113957_object58971:head# 2024-04-23T16:33:39.560 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568fe87d:::benchmark_data_smithi188_113957_object45327:head# 2024-04-23T16:33:39.560 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:568ff552:::benchmark_data_smithi188_113957_object43870:head# 2024-04-23T16:33:39.560 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5690691a:::benchmark_data_smithi188_113957_object147418:head# 2024-04-23T16:33:39.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5690f4ec:::benchmark_data_smithi188_113957_object148748:head# 2024-04-23T16:33:39.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5691042e:::benchmark_data_smithi188_113957_object14947:head# 2024-04-23T16:33:39.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56915a09:::benchmark_data_smithi188_113957_object63109:head# 2024-04-23T16:33:39.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56916b31:::benchmark_data_smithi188_113957_object47916:head# 2024-04-23T16:33:39.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56916f37:::benchmark_data_smithi188_113957_object91404:head# 2024-04-23T16:33:39.564 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56917190:::benchmark_data_smithi188_113957_object142893:head# 2024-04-23T16:33:39.564 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56919abc:::benchmark_data_smithi188_113957_object92512:head# 2024-04-23T16:33:39.564 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5691e20e:::benchmark_data_smithi188_113957_object58791:head# 2024-04-23T16:33:39.565 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56923fdf:::benchmark_data_smithi188_113957_object26590:head# 2024-04-23T16:33:39.565 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5692befb:::benchmark_data_smithi188_113957_object123967:head# 2024-04-23T16:33:39.566 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5692f7f1:::benchmark_data_smithi188_113957_object35028:head# 2024-04-23T16:33:39.566 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5693268b:::benchmark_data_smithi188_113957_object86568:head# 2024-04-23T16:33:39.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56935832:::benchmark_data_smithi188_113957_object39235:head# 2024-04-23T16:33:39.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56942654:::benchmark_data_smithi188_113957_object79409:head# 2024-04-23T16:33:39.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5694530d:::benchmark_data_smithi188_113957_object120303:head# 2024-04-23T16:33:39.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5694a3bd:::benchmark_data_smithi188_113957_object118865:head# 2024-04-23T16:33:39.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5694d475:::benchmark_data_smithi188_113957_object134017:head# 2024-04-23T16:33:39.569 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5695574a:::benchmark_data_smithi188_113957_object109518:head# 2024-04-23T16:33:39.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56968537:::benchmark_data_smithi188_113957_object92667:head# 2024-04-23T16:33:39.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5696cb24:::benchmark_data_smithi188_113957_object29167:head# 2024-04-23T16:33:39.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56970c7f:::benchmark_data_smithi188_113957_object21074:head# 2024-04-23T16:33:39.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:569731e6:::benchmark_data_smithi188_113957_object45013:head# 2024-04-23T16:33:39.574 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5697c1dc:::benchmark_data_smithi188_113957_object18639:head# 2024-04-23T16:33:39.574 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56984e5c:::benchmark_data_smithi188_113957_object94809:head# 2024-04-23T16:33:39.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5698a46c:::benchmark_data_smithi188_113957_object77309:head# 2024-04-23T16:33:39.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5698b96e:::benchmark_data_smithi188_113957_object21995:head# 2024-04-23T16:33:39.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56990ee3:::benchmark_data_smithi188_113957_object109902:head# 2024-04-23T16:33:39.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56993a58:::benchmark_data_smithi188_113957_object86482:head# 2024-04-23T16:33:39.576 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56993f50:::benchmark_data_smithi188_113957_object27290:head# 2024-04-23T16:33:39.576 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56996c78:::benchmark_data_smithi188_113957_object38369:head# 2024-04-23T16:33:39.576 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5699f2a9:::benchmark_data_smithi188_113957_object121437:head# 2024-04-23T16:33:39.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:569a2a07:::benchmark_data_smithi188_113957_object40060:head# 2024-04-23T16:33:39.579 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:569a5a0d:::benchmark_data_smithi188_113957_object82581:head# 2024-04-23T16:33:39.579 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:569a66ae:::benchmark_data_smithi188_113957_object12758:head# 2024-04-23T16:33:39.579 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:569ab4ef:::benchmark_data_smithi188_113957_object81903:head# 2024-04-23T16:33:39.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:569cf0d6:::benchmark_data_smithi188_113957_object296:head# 2024-04-23T16:33:39.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:569d808c:::benchmark_data_smithi188_113957_object54644:head# 2024-04-23T16:33:39.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:569d8d36:::benchmark_data_smithi188_113957_object146518:head# 2024-04-23T16:33:39.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:569d8ecc:::benchmark_data_smithi188_113957_object147624:head# 2024-04-23T16:33:39.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:569e2cf1:::benchmark_data_smithi188_113957_object100085:head# 2024-04-23T16:33:39.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:569e85d9:::benchmark_data_smithi188_113957_object101481:head# 2024-04-23T16:33:39.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:569e9353:::benchmark_data_smithi188_113957_object99862:head# 2024-04-23T16:33:39.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:569f8981:::benchmark_data_smithi188_113957_object29080:head# 2024-04-23T16:33:39.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a03da5:::benchmark_data_smithi188_113957_object16089:head# 2024-04-23T16:33:39.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a0d3bd:::benchmark_data_smithi188_113957_object85848:head# 2024-04-23T16:33:39.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a0eefd:::benchmark_data_smithi188_113957_object93769:head# 2024-04-23T16:33:39.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a138ef:::benchmark_data_smithi188_113957_object23687:head# 2024-04-23T16:33:39.585 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a14c93:::benchmark_data_smithi188_113957_object145626:head# 2024-04-23T16:33:39.585 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a18b10:::benchmark_data_smithi188_113957_object27547:head# 2024-04-23T16:33:39.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a1c15e:::benchmark_data_smithi188_113957_object76900:head# 2024-04-23T16:33:39.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a27ec1:::benchmark_data_smithi188_113957_object28242:head# 2024-04-23T16:33:39.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a297d2:::benchmark_data_smithi188_113957_object149929:head# 2024-04-23T16:33:39.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a3042a:::benchmark_data_smithi188_113957_object37459:head# 2024-04-23T16:33:39.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a3135c:::benchmark_data_smithi188_113957_object31143:head# 2024-04-23T16:33:39.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a3654f:::benchmark_data_smithi188_113957_object136837:head# 2024-04-23T16:33:39.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a37ade:::benchmark_data_smithi188_113957_object11841:head# 2024-04-23T16:33:39.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a40671:::benchmark_data_smithi188_113957_object29500:head# 2024-04-23T16:33:39.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a4597b:::benchmark_data_smithi188_113957_object151257:head# 2024-04-23T16:33:39.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a4b104:::benchmark_data_smithi188_113957_object84276:head# 2024-04-23T16:33:39.592 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a4dec8:::benchmark_data_smithi188_113957_object67743:head# 2024-04-23T16:33:39.592 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a5ca9e:::benchmark_data_smithi188_113957_object71600:head# 2024-04-23T16:33:39.592 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a5d50b:::benchmark_data_smithi188_113957_object28014:head# 2024-04-23T16:33:39.593 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a66714:::benchmark_data_smithi188_113957_object90293:head# 2024-04-23T16:33:39.593 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a675d6:::benchmark_data_smithi188_113957_object46260:head# 2024-04-23T16:33:39.594 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a6810f:::benchmark_data_smithi188_113957_object60650:head# 2024-04-23T16:33:39.594 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a697cc:::benchmark_data_smithi188_113957_object31711:head# 2024-04-23T16:33:39.594 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a6b485:::benchmark_data_smithi188_113957_object146654:head# 2024-04-23T16:33:39.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a73862:::benchmark_data_smithi188_113957_object3051:head# 2024-04-23T16:33:39.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a75fe8:::benchmark_data_smithi188_113957_object74146:head# 2024-04-23T16:33:39.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a8c3c8:::benchmark_data_smithi188_113957_object77540:head# 2024-04-23T16:33:39.596 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a8d873:::benchmark_data_smithi188_113957_object66358:head# 2024-04-23T16:33:39.596 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a8f3fe:::benchmark_data_smithi188_113957_object114785:head# 2024-04-23T16:33:39.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a937db:::benchmark_data_smithi188_113957_object150202:head# 2024-04-23T16:33:39.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a94354:::benchmark_data_smithi188_113957_object97345:head# 2024-04-23T16:33:39.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56a9ab97:::benchmark_data_smithi188_113957_object26953:head# 2024-04-23T16:33:39.598 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56aa28d0:::benchmark_data_smithi188_113957_object47272:head# 2024-04-23T16:33:39.598 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56aa324c:::benchmark_data_smithi188_113957_object148082:head# 2024-04-23T16:33:39.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56ac478e:::benchmark_data_smithi188_113957_object104399:head# 2024-04-23T16:33:39.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56aca95b:::benchmark_data_smithi188_113957_object10505:head# 2024-04-23T16:33:39.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56acf3a7:::benchmark_data_smithi188_113957_object68576:head# 2024-04-23T16:33:39.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56acfd96:::benchmark_data_smithi188_113957_object122575:head# 2024-04-23T16:33:39.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56ad0151:::benchmark_data_smithi188_113957_object61704:head# 2024-04-23T16:33:39.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56ada056:::benchmark_data_smithi188_113957_object122689:head# 2024-04-23T16:33:39.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56ae64ec:::benchmark_data_smithi188_113957_object104592:head# 2024-04-23T16:33:39.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56ae7bf0:::benchmark_data_smithi188_113957_object92248:head# 2024-04-23T16:33:39.602 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56af00bf:::benchmark_data_smithi188_113957_object61163:head# 2024-04-23T16:33:39.602 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56af9ad6:::benchmark_data_smithi188_113957_object67773:head# 2024-04-23T16:33:39.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b0026a:::benchmark_data_smithi188_113957_object87707:head# 2024-04-23T16:33:39.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b06452:::benchmark_data_smithi188_113957_object30351:head# 2024-04-23T16:33:39.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b0bec4:::benchmark_data_smithi188_113957_object142462:head# 2024-04-23T16:33:39.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b0d595:::benchmark_data_smithi188_113957_object125818:head# 2024-04-23T16:33:39.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b19cda:::benchmark_data_smithi188_113957_object98032:head# 2024-04-23T16:33:39.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b22ade:::benchmark_data_smithi188_113957_object56806:head# 2024-04-23T16:33:39.605 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b24402:::benchmark_data_smithi188_113957_object47682:head# 2024-04-23T16:33:39.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b25e51:::benchmark_data_smithi188_113957_object1091:head# 2024-04-23T16:33:39.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b290c0:::benchmark_data_smithi188_113957_object71171:head# 2024-04-23T16:33:39.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b2a8f7:::benchmark_data_smithi188_113957_object9215:head# 2024-04-23T16:33:39.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b2f271:::benchmark_data_smithi188_113957_object54459:head# 2024-04-23T16:33:39.607 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b30f89:::benchmark_data_smithi188_113957_object141278:head# 2024-04-23T16:33:39.607 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b38747:::benchmark_data_smithi188_113957_object149062:head# 2024-04-23T16:33:39.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b38ebe:::benchmark_data_smithi188_113957_object60903:head# 2024-04-23T16:33:39.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b43564:::benchmark_data_smithi188_113957_object48027:head# 2024-04-23T16:33:39.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b512f7:::benchmark_data_smithi188_113957_object12921:head# 2024-04-23T16:33:39.609 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b518f0:::benchmark_data_smithi188_113957_object90531:head# 2024-04-23T16:33:39.609 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b558c0:::benchmark_data_smithi188_113957_object30277:head# 2024-04-23T16:33:39.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b55cff:::benchmark_data_smithi188_113957_object125762:head# 2024-04-23T16:33:39.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b5a02e:::benchmark_data_smithi188_113957_object99800:head# 2024-04-23T16:33:39.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b5e5a8:::benchmark_data_smithi188_113957_object42901:head# 2024-04-23T16:33:39.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b6394b:::benchmark_data_smithi188_113957_object101794:head# 2024-04-23T16:33:39.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b699f9:::benchmark_data_smithi188_113957_object85495:head# 2024-04-23T16:33:39.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b6a247:::benchmark_data_smithi188_113957_object97506:head# 2024-04-23T16:33:39.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b773ea:::benchmark_data_smithi188_113957_object46828:head# 2024-04-23T16:33:39.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b7ff99:::benchmark_data_smithi188_113957_object131323:head# 2024-04-23T16:33:39.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b80c21:::benchmark_data_smithi188_113957_object19703:head# 2024-04-23T16:33:39.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b8e80e:::benchmark_data_smithi188_113957_object127129:head# 2024-04-23T16:33:39.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b9afa9:::benchmark_data_smithi188_113957_object117353:head# 2024-04-23T16:33:39.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56b9c9fe:::benchmark_data_smithi188_113957_object120520:head# 2024-04-23T16:33:39.616 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56ba0d54:::benchmark_data_smithi188_113957_object103585:head# 2024-04-23T16:33:39.616 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:33:39.617 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56babc6f:::benchmark_data_smithi188_113957_object132115:head# 2024-04-23T16:33:39.617 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56bae99e:::benchmark_data_smithi188_113957_object132436:head# 2024-04-23T16:33:39.617 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56bba03f:::benchmark_data_smithi188_113957_object93826:head# 2024-04-23T16:33:39.618 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56bbcb3c:::benchmark_data_smithi188_113957_object32198:head# 2024-04-23T16:33:39.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56bd2521:::benchmark_data_smithi188_113957_object48879:head# 2024-04-23T16:33:39.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56bdbbd9:::benchmark_data_smithi188_113957_object77406:head# 2024-04-23T16:33:39.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56beb9d6:::benchmark_data_smithi188_113957_object69803:head# 2024-04-23T16:33:39.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56bf19e4:::benchmark_data_smithi188_113957_object85300:head# 2024-04-23T16:33:39.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56bf2b53:::benchmark_data_smithi188_113957_object128109:head# 2024-04-23T16:33:39.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:39.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56bf348d:::benchmark_data_smithi188_113957_object77575:head# 2024-04-23T16:33:39.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:33:39.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56bf39a0:::benchmark_data_smithi188_113957_object100531:head# 2024-04-23T16:33:39.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56bf5f0a:::benchmark_data_smithi188_113957_object69379:head# 2024-04-23T16:33:39.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56bf8df3:::benchmark_data_smithi188_113957_object92304:head# 2024-04-23T16:33:39.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56bfacf2:::benchmark_data_smithi188_113957_object103650:head# 2024-04-23T16:33:39.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c04ded:::benchmark_data_smithi188_113957_object9793:head# 2024-04-23T16:33:39.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c13f99:::benchmark_data_smithi188_113957_object16579:head# 2024-04-23T16:33:39.624 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c21ed2:::benchmark_data_smithi188_113957_object10006:head# 2024-04-23T16:33:39.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c2ab02:::benchmark_data_smithi188_113957_object101885:head# 2024-04-23T16:33:39.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c2ad32:::benchmark_data_smithi188_113957_object123647:head# 2024-04-23T16:33:39.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c34a05:::benchmark_data_smithi188_113957_object150224:head# 2024-04-23T16:33:39.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c36de4:::benchmark_data_smithi188_113957_object53701:head# 2024-04-23T16:33:39.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c3efd8:::benchmark_data_smithi188_113957_object8766:head# 2024-04-23T16:33:39.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c3fb49:::benchmark_data_smithi188_113957_object117236:head# 2024-04-23T16:33:39.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c42daf:::benchmark_data_smithi188_113957_object126739:head# 2024-04-23T16:33:39.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c50546:::benchmark_data_smithi188_113957_object50185:head# 2024-04-23T16:33:39.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c6069d:::benchmark_data_smithi188_113957_object128991:head# 2024-04-23T16:33:39.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c6f941:::benchmark_data_smithi188_113957_object30253:head# 2024-04-23T16:33:39.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c7a32c:::benchmark_data_smithi188_113957_object125807:head# 2024-04-23T16:33:39.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c7bb63:::benchmark_data_smithi188_113957_object38685:head# 2024-04-23T16:33:39.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c7ec4a:::benchmark_data_smithi188_113957_object69363:head# 2024-04-23T16:33:39.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c8306c:::benchmark_data_smithi188_113957_object113094:head# 2024-04-23T16:33:39.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c8a757:::benchmark_data_smithi188_113957_object4736:head# 2024-04-23T16:33:39.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c8be37:::benchmark_data_smithi188_113957_object70724:head# 2024-04-23T16:33:39.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56c939e3:::benchmark_data_smithi188_113957_object119622:head# 2024-04-23T16:33:39.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56ca539b:::benchmark_data_smithi188_113957_object136698:head# 2024-04-23T16:33:39.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56cb90bc:::benchmark_data_smithi188_113957_object13075:head# 2024-04-23T16:33:39.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56cbcbd2:::benchmark_data_smithi188_113957_object114083:head# 2024-04-23T16:33:39.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56cbd379:::benchmark_data_smithi188_113957_object51721:head# 2024-04-23T16:33:39.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56cbee46:::benchmark_data_smithi188_113957_object124537:head# 2024-04-23T16:33:39.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56cc1a4b:::benchmark_data_smithi188_113957_object14139:head# 2024-04-23T16:33:39.635 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56cc4bb8:::benchmark_data_smithi188_113957_object141117:head# 2024-04-23T16:33:39.635 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56cc6322:::benchmark_data_smithi188_113957_object17925:head# 2024-04-23T16:33:39.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56cd8be7:::benchmark_data_smithi188_113957_object149213:head# 2024-04-23T16:33:39.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56ce1cbd:::benchmark_data_smithi188_113957_object110130:head# 2024-04-23T16:33:39.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56ce6474:::benchmark_data_smithi188_113957_object25836:head# 2024-04-23T16:33:39.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56cf22f8:::benchmark_data_smithi188_113957_object2429:head# 2024-04-23T16:33:39.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56cf36d2:::benchmark_data_smithi188_113957_object58575:head# 2024-04-23T16:33:39.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d04d96:::benchmark_data_smithi188_113957_object2343:head# 2024-04-23T16:33:39.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d0fca3:::benchmark_data_smithi188_113957_object11463:head# 2024-04-23T16:33:39.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d109c9:::benchmark_data_smithi188_113957_object69758:head# 2024-04-23T16:33:39.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d10a71:::benchmark_data_smithi188_113957_object59011:head# 2024-04-23T16:33:39.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d118c6:::benchmark_data_smithi188_113957_object94612:head# 2024-04-23T16:33:39.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d14788:::benchmark_data_smithi188_113957_object123750:head# 2024-04-23T16:33:39.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d1c09c:::benchmark_data_smithi188_113957_object19700:head# 2024-04-23T16:33:39.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d1c1e4:::benchmark_data_smithi188_113957_object79824:head# 2024-04-23T16:33:39.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d21832:::benchmark_data_smithi188_113957_object68916:head# 2024-04-23T16:33:39.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d269a8:::benchmark_data_smithi188_113957_object43848:head# 2024-04-23T16:33:39.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d293bc:::benchmark_data_smithi188_113957_object88283:head# 2024-04-23T16:33:39.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d2d006:::benchmark_data_smithi188_113957_object146132:head# 2024-04-23T16:33:39.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d30cc8:::benchmark_data_smithi188_113957_object135202:head# 2024-04-23T16:33:39.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d326ba:::benchmark_data_smithi188_113957_object49538:head# 2024-04-23T16:33:39.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d360b1:::benchmark_data_smithi188_113957_object61125:head# 2024-04-23T16:33:39.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d3aae3:::benchmark_data_smithi188_113957_object142548:head# 2024-04-23T16:33:39.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d49ed2:::benchmark_data_smithi188_113957_object84623:head# 2024-04-23T16:33:39.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d50b67:::benchmark_data_smithi188_113957_object91719:head# 2024-04-23T16:33:39.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d5187e:::benchmark_data_smithi188_113957_object45697:head# 2024-04-23T16:33:39.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d60e69:::benchmark_data_smithi188_113957_object40298:head# 2024-04-23T16:33:39.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d720d1:::benchmark_data_smithi188_113957_object36026:head# 2024-04-23T16:33:39.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d72330:::benchmark_data_smithi188_113957_object66789:head# 2024-04-23T16:33:39.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d74c1e:::benchmark_data_smithi188_113957_object90322:head# 2024-04-23T16:33:39.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d7c2bb:::benchmark_data_smithi188_113957_object15396:head# 2024-04-23T16:33:39.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d859b9:::benchmark_data_smithi188_113957_object106392:head# 2024-04-23T16:33:39.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d86ddb:::benchmark_data_smithi188_113957_object140398:head# 2024-04-23T16:33:39.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d8f07a:::benchmark_data_smithi188_113957_object7161:head# 2024-04-23T16:33:39.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d99621:::benchmark_data_smithi188_113957_object148824:head# 2024-04-23T16:33:39.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d9d1e1:::benchmark_data_smithi188_113957_object100996:head# 2024-04-23T16:33:39.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56d9d3c8:::benchmark_data_smithi188_113957_object108003:head# 2024-04-23T16:33:39.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56dac983:::benchmark_data_smithi188_113957_object70593:head# 2024-04-23T16:33:39.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56daff7b:::benchmark_data_smithi188_113957_object100450:head# 2024-04-23T16:33:39.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56db2bb6:::benchmark_data_smithi188_113957_object56601:head# 2024-04-23T16:33:39.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56db473e:::benchmark_data_smithi188_113957_object55089:head# 2024-04-23T16:33:39.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56dba1f4:::benchmark_data_smithi188_113957_object67484:head# 2024-04-23T16:33:39.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56dbe9a9:::benchmark_data_smithi188_113957_object55902:head# 2024-04-23T16:33:39.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56dc6159:::benchmark_data_smithi188_113957_object147534:head# 2024-04-23T16:33:39.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56dcc8cc:::benchmark_data_smithi188_113957_object126669:head# 2024-04-23T16:33:39.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56dd08f0:::benchmark_data_smithi188_113957_object87771:head# 2024-04-23T16:33:39.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56dd1369:::benchmark_data_smithi188_113957_object95970:head# 2024-04-23T16:33:39.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56dd772e:::benchmark_data_smithi188_113957_object151612:head# 2024-04-23T16:33:39.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56de90a2:::benchmark_data_smithi188_113957_object67497:head# 2024-04-23T16:33:39.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56def2aa:::benchmark_data_smithi188_113957_object105529:head# 2024-04-23T16:33:39.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56df290f:::benchmark_data_smithi188_113957_object55510:head# 2024-04-23T16:33:39.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56e0ab45:::benchmark_data_smithi188_113957_object20254:head# 2024-04-23T16:33:39.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56e0bea7:::benchmark_data_smithi188_113957_object36235:head# 2024-04-23T16:33:39.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56e19fe1:::benchmark_data_smithi188_113957_object149690:head# 2024-04-23T16:33:39.661 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56e27099:::benchmark_data_smithi188_113957_object30636:head# 2024-04-23T16:33:39.661 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56e2c2f4:::benchmark_data_smithi188_113957_object72312:head# 2024-04-23T16:33:39.661 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56e2c5cc:::benchmark_data_smithi188_113957_object37607:head# 2024-04-23T16:33:39.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56e5c37c:::benchmark_data_smithi188_113957_object148241:head# 2024-04-23T16:33:39.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56e5ebea:::benchmark_data_smithi188_113957_object103143:head# 2024-04-23T16:33:39.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56e60e8f:::benchmark_data_smithi188_113957_object120015:head# 2024-04-23T16:33:39.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56e6e14d:::benchmark_data_smithi188_113957_object30298:head# 2024-04-23T16:33:39.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56e7ac95:::benchmark_data_smithi188_113957_object107854:head# 2024-04-23T16:33:39.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56e8d3b5:::benchmark_data_smithi188_113957_object93967:head# 2024-04-23T16:33:39.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56e98d35:::benchmark_data_smithi188_113957_object92020:head# 2024-04-23T16:33:39.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56e9d3d8:::benchmark_data_smithi188_113957_object8204:head# 2024-04-23T16:33:39.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56ea42eb:::benchmark_data_smithi188_113957_object100514:head# 2024-04-23T16:33:39.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56ea7594:::benchmark_data_smithi188_113957_object89011:head# 2024-04-23T16:33:39.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56eac1e7:::benchmark_data_smithi188_113957_object12865:head# 2024-04-23T16:33:39.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56ec4db6:::benchmark_data_smithi188_113957_object27333:head# 2024-04-23T16:33:39.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56ec5b4b:::benchmark_data_smithi188_113957_object117146:head# 2024-04-23T16:33:39.668 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56ed46a9:::benchmark_data_smithi188_113957_object31758:head# 2024-04-23T16:33:39.668 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56ee0127:::benchmark_data_smithi188_113957_object109612:head# 2024-04-23T16:33:39.668 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56ee7410:::benchmark_data_smithi188_113957_object105128:head# 2024-04-23T16:33:39.669 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56ef66c9:::benchmark_data_smithi188_113957_object19534:head# 2024-04-23T16:33:39.669 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f04b13:::benchmark_data_smithi188_113957_object54747:head# 2024-04-23T16:33:39.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f093de:::benchmark_data_smithi188_113957_object50793:head# 2024-04-23T16:33:39.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f0d55d:::benchmark_data_smithi188_113957_object22749:head# 2024-04-23T16:33:39.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f103d3:::benchmark_data_smithi188_113957_object124940:head# 2024-04-23T16:33:39.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f1552b:::benchmark_data_smithi188_113957_object96982:head# 2024-04-23T16:33:39.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f21b36:::benchmark_data_smithi188_113957_object74156:head# 2024-04-23T16:33:39.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f233ed:::benchmark_data_smithi188_113957_object243:head# 2024-04-23T16:33:39.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f28768:::benchmark_data_smithi188_113957_object38198:head# 2024-04-23T16:33:39.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f29113:::benchmark_data_smithi188_113957_object118156:head# 2024-04-23T16:33:39.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f2f800:::benchmark_data_smithi188_113957_object48409:head# 2024-04-23T16:33:39.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f3649c:::benchmark_data_smithi188_113957_object17948:head# 2024-04-23T16:33:39.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f55660:::benchmark_data_smithi188_113957_object4397:head# 2024-04-23T16:33:39.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f580ba:::benchmark_data_smithi188_113957_object17698:head# 2024-04-23T16:33:39.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f5a7f4:::benchmark_data_smithi188_113957_object55144:head# 2024-04-23T16:33:39.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f604d6:::benchmark_data_smithi188_113957_object44435:head# 2024-04-23T16:33:39.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f615b3:::benchmark_data_smithi188_113957_object54660:head# 2024-04-23T16:33:39.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f66827:::benchmark_data_smithi188_113957_object36878:head# 2024-04-23T16:33:39.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f67603:::benchmark_data_smithi188_113957_object25665:head# 2024-04-23T16:33:39.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f6b2a8:::benchmark_data_smithi188_113957_object9369:head# 2024-04-23T16:33:39.678 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f6c1cc:::benchmark_data_smithi188_113957_object3306:head# 2024-04-23T16:33:39.678 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f6cf8e:::benchmark_data_smithi188_113957_object50245:head# 2024-04-23T16:33:39.678 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f6d9e3:::benchmark_data_smithi188_113957_object867:head# 2024-04-23T16:33:39.679 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f731dd:::benchmark_data_smithi188_113957_object139868:head# 2024-04-23T16:33:39.679 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f7872e:::benchmark_data_smithi188_113957_object111905:head# 2024-04-23T16:33:39.680 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f7b2d0:::benchmark_data_smithi188_113957_object61169:head# 2024-04-23T16:33:39.680 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f7b6fb:::benchmark_data_smithi188_113957_object46422:head# 2024-04-23T16:33:39.680 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f7dfa4:::benchmark_data_smithi188_113957_object120671:head# 2024-04-23T16:33:39.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f80a23:::benchmark_data_smithi188_113957_object52928:head# 2024-04-23T16:33:39.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f8c33a:::benchmark_data_smithi188_113957_object13401:head# 2024-04-23T16:33:39.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f8c8a1:::benchmark_data_smithi188_113957_object142375:head# 2024-04-23T16:33:39.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f9999e:::benchmark_data_smithi188_113957_object1892:head# 2024-04-23T16:33:39.683 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f9b224:::benchmark_data_smithi188_113957_object57331:head# 2024-04-23T16:33:39.683 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f9da86:::benchmark_data_smithi188_113957_object86447:head# 2024-04-23T16:33:39.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f9e859:::benchmark_data_smithi188_113957_object121828:head# 2024-04-23T16:33:39.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56f9e9dc:::benchmark_data_smithi188_113957_object135131:head# 2024-04-23T16:33:39.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56fa5dd1:::benchmark_data_smithi188_113957_object30010:head# 2024-04-23T16:33:39.685 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-23T16:33:39.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56fa7c34:::benchmark_data_smithi188_113957_object15478:head# 2024-04-23T16:33:39.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56fab5ef:::benchmark_data_smithi188_113957_object113879:head# 2024-04-23T16:33:39.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56facf59:::benchmark_data_smithi188_113957_object144775:head# 2024-04-23T16:33:39.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56fb1cf8:::benchmark_data_smithi188_113957_object90116:head# 2024-04-23T16:33:39.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56fb3acd:::benchmark_data_smithi188_113957_object35445:head# 2024-04-23T16:33:39.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56fbe7df:::benchmark_data_smithi188_113957_object113031:head# 2024-04-23T16:33:39.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56fc4d01:::benchmark_data_smithi188_113957_object80018:head# 2024-04-23T16:33:39.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56fc5ba7:::benchmark_data_smithi188_113957_object6621:head# 2024-04-23T16:33:39.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56fc9532:::benchmark_data_smithi188_113957_object141333:head# 2024-04-23T16:33:39.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56fca945:::benchmark_data_smithi188_113957_object73801:head# 2024-04-23T16:33:39.689 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56fcb0a2:::benchmark_data_smithi188_113957_object82910:head# 2024-04-23T16:33:39.689 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56fdadc4:::benchmark_data_smithi188_113957_object28960:head# 2024-04-23T16:33:39.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56fe66c0:::benchmark_data_smithi188_113957_object144982:head# 2024-04-23T16:33:39.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56fe82b5:::benchmark_data_smithi188_113957_object152050:head# 2024-04-23T16:33:39.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:56ff5785:::benchmark_data_smithi188_113957_object44324:head# 2024-04-23T16:33:39.691 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570000bb:::benchmark_data_smithi188_113957_object93495:head# 2024-04-23T16:33:39.691 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570009ea:::benchmark_data_smithi188_113957_object64637:head# 2024-04-23T16:33:39.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57000cea:::benchmark_data_smithi188_113957_object51886:head# 2024-04-23T16:33:39.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57001f49:::benchmark_data_smithi188_113957_object117770:head# 2024-04-23T16:33:39.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57011d54:::benchmark_data_smithi188_113957_object45116:head# 2024-04-23T16:33:39.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570132d3:::benchmark_data_smithi188_113957_object20770:head# 2024-04-23T16:33:39.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5701d24d:::benchmark_data_smithi188_113957_object27635:head# 2024-04-23T16:33:39.694 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570289df:::benchmark_data_smithi188_113957_object32278:head# 2024-04-23T16:33:39.694 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57028c86:::benchmark_data_smithi188_113957_object4186:head# 2024-04-23T16:33:39.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5702a9d4:::benchmark_data_smithi188_113957_object74542:head# 2024-04-23T16:33:39.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5702fd55:::benchmark_data_smithi188_113957_object96261:head# 2024-04-23T16:33:39.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5703f36d:::benchmark_data_smithi188_113957_object62764:head# 2024-04-23T16:33:39.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5704077a:::benchmark_data_smithi188_113957_object17490:head# 2024-04-23T16:33:39.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5704502d:::benchmark_data_smithi188_113957_object26031:head# 2024-04-23T16:33:39.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57046249:::benchmark_data_smithi188_113957_object78067:head# 2024-04-23T16:33:39.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57054508:::benchmark_data_smithi188_113957_object143384:head# 2024-04-23T16:33:39.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570552a0:::benchmark_data_smithi188_113957_object132134:head# 2024-04-23T16:33:39.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57055bda:::benchmark_data_smithi188_113957_object15888:head# 2024-04-23T16:33:39.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5705f266:::benchmark_data_smithi188_113957_object75170:head# 2024-04-23T16:33:39.699 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5705ffd4:::benchmark_data_smithi188_113957_object65250:head# 2024-04-23T16:33:39.699 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570639ec:::benchmark_data_smithi188_113957_object32729:head# 2024-04-23T16:33:39.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570647f0:::benchmark_data_smithi188_113957_object119636:head# 2024-04-23T16:33:39.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5707b304:::benchmark_data_smithi188_113957_object128868:head# 2024-04-23T16:33:39.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570858bf:::benchmark_data_smithi188_113957_object134669:head# 2024-04-23T16:33:39.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57086e32:::benchmark_data_smithi188_113957_object61714:head# 2024-04-23T16:33:39.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5708bad3:::benchmark_data_smithi188_113957_object87589:head# 2024-04-23T16:33:39.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5708f4bb:::benchmark_data_smithi188_113957_object125672:head# 2024-04-23T16:33:39.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57097a20:::benchmark_data_smithi188_113957_object116682:head# 2024-04-23T16:33:39.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5709800b:::benchmark_data_smithi188_113957_object377:head# 2024-04-23T16:33:39.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5709f1c4:::benchmark_data_smithi188_113957_object51390:head# 2024-04-23T16:33:39.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570a0a78:::benchmark_data_smithi188_113957_object101770:head# 2024-04-23T16:33:39.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570a14bd:::benchmark_data_smithi188_113957_object113400:head# 2024-04-23T16:33:39.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570a1e61:::benchmark_data_smithi188_113957_object15702:head# 2024-04-23T16:33:39.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570abffb:::benchmark_data_smithi188_113957_object37747:head# 2024-04-23T16:33:39.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570adc6b:::benchmark_data_smithi188_113957_object148393:head# 2024-04-23T16:33:39.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570c1a10:::benchmark_data_smithi188_113957_object126136:head# 2024-04-23T16:33:39.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570c2692:::benchmark_data_smithi188_113957_object112966:head# 2024-04-23T16:33:39.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570c5927:::benchmark_data_smithi188_113957_object84874:head# 2024-04-23T16:33:39.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570ceca3:::benchmark_data_smithi188_113957_object13891:head# 2024-04-23T16:33:39.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570d6f94:::benchmark_data_smithi188_113957_object135439:head# 2024-04-23T16:33:39.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570e41b5:::benchmark_data_smithi188_113957_object16808:head# 2024-04-23T16:33:39.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570e840f:::benchmark_data_smithi188_113957_object19821:head# 2024-04-23T16:33:39.709 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:570fc220:::benchmark_data_smithi188_113957_object52047:head# 2024-04-23T16:33:39.709 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5710f8d4:::benchmark_data_smithi188_113957_object143070:head# 2024-04-23T16:33:39.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5711e478:::benchmark_data_smithi188_113957_object141972:head# 2024-04-23T16:33:39.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57128d0c:::benchmark_data_smithi188_113957_object16207:head# 2024-04-23T16:33:39.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5712b16c:::benchmark_data_smithi188_113957_object78565:head# 2024-04-23T16:33:39.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5712f027:::benchmark_data_smithi188_113957_object53118:head# 2024-04-23T16:33:39.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5713b0b7:::benchmark_data_smithi188_113957_object49326:head# 2024-04-23T16:33:39.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5713f04b:::benchmark_data_smithi188_113957_object117042:head# 2024-04-23T16:33:39.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57140b28:::benchmark_data_smithi188_113957_object56637:head# 2024-04-23T16:33:39.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57141338:::benchmark_data_smithi188_113957_object75163:head# 2024-04-23T16:33:39.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5714711a:::benchmark_data_smithi188_113957_object67487:head# 2024-04-23T16:33:39.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5714f633:::benchmark_data_smithi188_113957_object96182:head# 2024-04-23T16:33:39.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57150d7a:::benchmark_data_smithi188_113957_object29781:head# 2024-04-23T16:33:39.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57154f9d:::benchmark_data_smithi188_113957_object45185:head# 2024-04-23T16:33:39.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5715cb81:::benchmark_data_smithi188_113957_object124850:head# 2024-04-23T16:33:39.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5715dcdd:::benchmark_data_smithi188_113957_object59823:head# 2024-04-23T16:33:39.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5715eecf:::benchmark_data_smithi188_113957_object110765:head# 2024-04-23T16:33:39.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5716255b:::benchmark_data_smithi188_113957_object74866:head# 2024-04-23T16:33:39.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57165504:::benchmark_data_smithi188_113957_object73931:head# 2024-04-23T16:33:39.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5716572d:::benchmark_data_smithi188_113957_object28598:head# 2024-04-23T16:33:39.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57167e91:::benchmark_data_smithi188_113957_object104205:head# 2024-04-23T16:33:39.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5716c53a:::benchmark_data_smithi188_113957_object5056:head# 2024-04-23T16:33:39.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5716cca3:::benchmark_data_smithi188_113957_object79375:head# 2024-04-23T16:33:39.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5716d45b:::benchmark_data_smithi188_113957_object61186:head# 2024-04-23T16:33:39.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5716f04c:::benchmark_data_smithi188_113957_object146756:head# 2024-04-23T16:33:39.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57170502:::benchmark_data_smithi188_113957_object132333:head# 2024-04-23T16:33:39.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57172746:::benchmark_data_smithi188_113957_object67337:head# 2024-04-23T16:33:39.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:571731e4:::benchmark_data_smithi188_113957_object139122:head# 2024-04-23T16:33:39.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5717a6e3:::benchmark_data_smithi188_113957_object7619:head# 2024-04-23T16:33:39.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5718210e:::benchmark_data_smithi188_113957_object65356:head# 2024-04-23T16:33:39.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57185fc1:::benchmark_data_smithi188_113957_object110269:head# 2024-04-23T16:33:39.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5719ae5e:::benchmark_data_smithi188_113957_object34213:head# 2024-04-23T16:33:39.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5719eccc:::benchmark_data_smithi188_113957_object65686:head# 2024-04-23T16:33:39.723 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:33:39.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:571a3fa5:::benchmark_data_smithi188_113957_object128071:head# 2024-04-23T16:33:39.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:571a4e6a:::benchmark_data_smithi188_113957_object96659:head# 2024-04-23T16:33:39.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:571af5be:::benchmark_data_smithi188_113957_object131209:head# 2024-04-23T16:33:39.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:571b861a:::benchmark_data_smithi188_113957_object35466:head# 2024-04-23T16:33:39.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:571c1c55:::benchmark_data_smithi188_113957_object38102:head# 2024-04-23T16:33:39.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:571d07a9:::benchmark_data_smithi188_113957_object74479:head# 2024-04-23T16:33:39.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:571d7ffd:::benchmark_data_smithi188_113957_object52868:head# 2024-04-23T16:33:39.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572004e0:::benchmark_data_smithi188_113957_object133833:head# 2024-04-23T16:33:39.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572012c1:::benchmark_data_smithi188_113957_object106509:head# 2024-04-23T16:33:39.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57201560:::benchmark_data_smithi188_113957_object45053:head# 2024-04-23T16:33:39.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57206f94:::benchmark_data_smithi188_113957_object141290:head# 2024-04-23T16:33:39.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5721acb3:::benchmark_data_smithi188_113957_object512:head# 2024-04-23T16:33:39.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5721db84:::benchmark_data_smithi188_113957_object19235:head# 2024-04-23T16:33:39.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5722e5b9:::benchmark_data_smithi188_113957_object113743:head# 2024-04-23T16:33:39.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572345c8:::benchmark_data_smithi188_113957_object28223:head# 2024-04-23T16:33:39.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572353ad:::benchmark_data_smithi188_113957_object145678:head# 2024-04-23T16:33:39.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57246388:::benchmark_data_smithi188_113957_object68830:head# 2024-04-23T16:33:39.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57249276:::benchmark_data_smithi188_113957_object70723:head# 2024-04-23T16:33:39.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5724f313:::benchmark_data_smithi188_113957_object43033:head# 2024-04-23T16:33:39.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57267dc1:::benchmark_data_smithi188_113957_object13687:head# 2024-04-23T16:33:39.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5726c85a:::benchmark_data_smithi188_113957_object9077:head# 2024-04-23T16:33:39.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57274daa:::benchmark_data_smithi188_113957_object128024:head# 2024-04-23T16:33:39.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5728ebae:::benchmark_data_smithi188_113957_object81184:head# 2024-04-23T16:33:39.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5728eea3:::benchmark_data_smithi188_113957_object116925:head# 2024-04-23T16:33:39.735 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57290b38:::benchmark_data_smithi188_113957_object113657:head# 2024-04-23T16:33:39.735 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57290cfa:::benchmark_data_smithi188_113957_object1658:head# 2024-04-23T16:33:39.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57298a23:::benchmark_data_smithi188_113957_object39030:head# 2024-04-23T16:33:39.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5729a95a:::benchmark_data_smithi188_113957_object103098:head# 2024-04-23T16:33:39.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5729c4d5:::benchmark_data_smithi188_113957_object83638:head# 2024-04-23T16:33:39.737 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5729eeab:::benchmark_data_smithi188_113957_object25489:head# 2024-04-23T16:33:39.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572a749c:::benchmark_data_smithi188_113957_object112187:head# 2024-04-23T16:33:39.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572abc69:::benchmark_data_smithi188_113957_object97574:head# 2024-04-23T16:33:39.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572b27ed:::benchmark_data_smithi188_113957_object5905:head# 2024-04-23T16:33:39.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572b5b41:::benchmark_data_smithi188_113957_object80652:head# 2024-04-23T16:33:39.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572b955e:::benchmark_data_smithi188_113957_object129716:head# 2024-04-23T16:33:39.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572bc268:::benchmark_data_smithi188_113957_object114805:head# 2024-04-23T16:33:39.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572bd48a:::benchmark_data_smithi188_113957_object3496:head# 2024-04-23T16:33:39.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572bf121:::benchmark_data_smithi188_113957_object73010:head# 2024-04-23T16:33:39.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572c1bcb:::benchmark_data_smithi188_113957_object129198:head# 2024-04-23T16:33:39.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572c8123:::benchmark_data_smithi188_113957_object135543:head# 2024-04-23T16:33:39.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572cab43:::benchmark_data_smithi188_113957_object87056:head# 2024-04-23T16:33:39.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572cfdf8:::benchmark_data_smithi188_113957_object6039:head# 2024-04-23T16:33:39.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572d1014:::benchmark_data_smithi188_113957_object108031:head# 2024-04-23T16:33:39.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572d51b1:::benchmark_data_smithi188_113957_object89656:head# 2024-04-23T16:33:39.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572da272:::benchmark_data_smithi188_113957_object33874:head# 2024-04-23T16:33:39.744 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572dd6a3:::benchmark_data_smithi188_113957_object130080:head# 2024-04-23T16:33:39.744 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572e2dac:::benchmark_data_smithi188_113957_object85239:head# 2024-04-23T16:33:39.744 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572e816e:::benchmark_data_smithi188_113957_object107987:head# 2024-04-23T16:33:39.745 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572e8f71:::benchmark_data_smithi188_113957_object75722:head# 2024-04-23T16:33:39.745 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572f5426:::benchmark_data_smithi188_113957_object32728:head# 2024-04-23T16:33:39.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572f630d:::benchmark_data_smithi188_113957_object46354:head# 2024-04-23T16:33:39.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572f7d92:::benchmark_data_smithi188_113957_object87965:head# 2024-04-23T16:33:39.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:572fd2f1:::benchmark_data_smithi188_113957_object14327:head# 2024-04-23T16:33:39.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57302288:::benchmark_data_smithi188_113957_object110035:head# 2024-04-23T16:33:39.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57302c8a:::benchmark_data_smithi188_113957_object91567:head# 2024-04-23T16:33:39.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5730c6fd:::benchmark_data_smithi188_113957_object35501:head# 2024-04-23T16:33:39.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57311934:::benchmark_data_smithi188_113957_object1881:head# 2024-04-23T16:33:39.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57312d57:::benchmark_data_smithi188_113957_object3510:head# 2024-04-23T16:33:39.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5731d724:::benchmark_data_smithi188_113957_object41440:head# 2024-04-23T16:33:39.750 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5731dc68:::benchmark_data_smithi188_113957_object98572:head# 2024-04-23T16:33:39.750 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573293ff:::benchmark_data_smithi188_113957_object114836:head# 2024-04-23T16:33:39.750 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5732b21a:::benchmark_data_smithi188_113957_object6319:head# 2024-04-23T16:33:39.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57335969:::benchmark_data_smithi188_113957_object116130:head# 2024-04-23T16:33:39.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57341f08:::benchmark_data_smithi188_113957_object33931:head# 2024-04-23T16:33:39.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57347aab:::benchmark_data_smithi188_113957_object147019:head# 2024-04-23T16:33:39.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57348a89:::benchmark_data_smithi188_113957_object28565:head# 2024-04-23T16:33:39.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5734f6f5:::benchmark_data_smithi188_113957_object86527:head# 2024-04-23T16:33:39.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573536f3:::benchmark_data_smithi188_113957_object101328:head# 2024-04-23T16:33:39.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5735a0fd:::benchmark_data_smithi188_113957_object67095:head# 2024-04-23T16:33:39.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5736137d:::benchmark_data_smithi188_113957_object120454:head# 2024-04-23T16:33:39.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5736156e:::benchmark_data_smithi188_113957_object13115:head# 2024-04-23T16:33:39.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5736253c:::benchmark_data_smithi188_113957_object24745:head# 2024-04-23T16:33:39.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5736ad30:::benchmark_data_smithi188_113957_object112741:head# 2024-04-23T16:33:39.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5737b107:::benchmark_data_smithi188_113957_object18387:head# 2024-04-23T16:33:39.756 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:39.756 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5737ca2d:::benchmark_data_smithi188_113957_object139953:head# 2024-04-23T16:33:39.756 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5737e07c:::benchmark_data_smithi188_113957_object151744:head# 2024-04-23T16:33:39.756 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5738ddf9:::benchmark_data_smithi188_113957_object38434:head# 2024-04-23T16:33:39.757 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57391af7:::benchmark_data_smithi188_113957_object121720:head# 2024-04-23T16:33:39.757 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57395a87:::benchmark_data_smithi188_113957_object59448:head# 2024-04-23T16:33:39.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5739a3ea:::benchmark_data_smithi188_113957_object104727:head# 2024-04-23T16:33:39.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5739aedd:::benchmark_data_smithi188_113957_object52659:head# 2024-04-23T16:33:39.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5739b80e:::benchmark_data_smithi188_113957_object103346:head# 2024-04-23T16:33:39.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5739f8bc:::benchmark_data_smithi188_113957_object81183:head# 2024-04-23T16:33:39.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573a094b:::benchmark_data_smithi188_113957_object42166:head# 2024-04-23T16:33:39.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573a2f9b:::benchmark_data_smithi188_113957_object108771:head# 2024-04-23T16:33:39.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573a5536:::benchmark_data_smithi188_113957_object134014:head# 2024-04-23T16:33:39.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573af11f:::benchmark_data_smithi188_113957_object84965:head# 2024-04-23T16:33:39.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573af9d2:::benchmark_data_smithi188_113957_object70236:head# 2024-04-23T16:33:39.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573bbad5:::benchmark_data_smithi188_113957_object20023:head# 2024-04-23T16:33:39.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573bf991:::benchmark_data_smithi188_113957_object47843:head# 2024-04-23T16:33:39.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573c445a:::benchmark_data_smithi188_113957_object69168:head# 2024-04-23T16:33:39.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573cc7e2:::benchmark_data_smithi188_113957_object119468:head# 2024-04-23T16:33:39.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573d8d79:::benchmark_data_smithi188_113957_object133726:head# 2024-04-23T16:33:39.764 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573d92c1:::benchmark_data_smithi188_113957_object113809:head# 2024-04-23T16:33:39.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573e7ac8:::benchmark_data_smithi188_113957_object82743:head# 2024-04-23T16:33:39.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573ea506:::benchmark_data_smithi188_113957_object139064:head# 2024-04-23T16:33:39.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573eba63:::benchmark_data_smithi188_113957_object28873:head# 2024-04-23T16:33:39.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573f0738:::benchmark_data_smithi188_113957_object76644:head# 2024-04-23T16:33:39.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573f285e:::benchmark_data_smithi188_113957_object66376:head# 2024-04-23T16:33:39.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573f9892:::benchmark_data_smithi188_113957_object69318:head# 2024-04-23T16:33:39.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:573fcd11:::benchmark_data_smithi188_113957_object60748:head# 2024-04-23T16:33:39.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57400383:::benchmark_data_smithi188_113957_object133924:head# 2024-04-23T16:33:39.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574085fa:::benchmark_data_smithi188_113957_object135702:head# 2024-04-23T16:33:39.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5740b384:::benchmark_data_smithi188_113957_object77753:head# 2024-04-23T16:33:39.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57414678:::benchmark_data_smithi188_113957_object31540:head# 2024-04-23T16:33:39.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57415aa4:::benchmark_data_smithi188_113957_object7026:head# 2024-04-23T16:33:39.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5741a8fd:::benchmark_data_smithi188_113957_object3175:head# 2024-04-23T16:33:39.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574200c6:::benchmark_data_smithi188_113957_object123416:head# 2024-04-23T16:33:39.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5742d2d3:::benchmark_data_smithi188_113957_object57021:head# 2024-04-23T16:33:39.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5743c050:::benchmark_data_smithi188_113957_object5013:head# 2024-04-23T16:33:39.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5743eba7:::benchmark_data_smithi188_113957_object9084:head# 2024-04-23T16:33:39.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57444e9f:::benchmark_data_smithi188_113957_object33417:head# 2024-04-23T16:33:39.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57446d26:::benchmark_data_smithi188_113957_object26042:head# 2024-04-23T16:33:39.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57448378:::benchmark_data_smithi188_113957_object128724:head# 2024-04-23T16:33:39.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5744951d:::benchmark_data_smithi188_113957_object7775:head# 2024-04-23T16:33:39.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574597e9:::benchmark_data_smithi188_113957_object88757:head# 2024-04-23T16:33:39.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57467633:::benchmark_data_smithi188_113957_object17657:head# 2024-04-23T16:33:39.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5746befc:::benchmark_data_smithi188_113957_object61402:head# 2024-04-23T16:33:39.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5746eac5:::benchmark_data_smithi188_113957_object53990:head# 2024-04-23T16:33:39.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5746f976:::benchmark_data_smithi188_113957_object97823:head# 2024-04-23T16:33:39.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5748455c:::benchmark_data_smithi188_113957_object151844:head# 2024-04-23T16:33:39.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5748e08b:::benchmark_data_smithi188_113957_object105273:head# 2024-04-23T16:33:39.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574948d8:::benchmark_data_smithi188_113957_object130627:head# 2024-04-23T16:33:39.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574961d0:::benchmark_data_smithi188_113957_object35343:head# 2024-04-23T16:33:39.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57496be4:::benchmark_data_smithi188_113957_object27273:head# 2024-04-23T16:33:39.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574ab1b5:::benchmark_data_smithi188_113957_object107590:head# 2024-04-23T16:33:39.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574b54f0:::benchmark_data_smithi188_113957_object24035:head# 2024-04-23T16:33:39.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574ba09a:::benchmark_data_smithi188_113957_object94572:head# 2024-04-23T16:33:39.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574c670e:::benchmark_data_smithi188_113957_object144618:head# 2024-04-23T16:33:39.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574c7db4:::benchmark_data_smithi188_113957_object136513:head# 2024-04-23T16:33:39.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574d6ecd:::benchmark_data_smithi188_113957_object13885:head# 2024-04-23T16:33:39.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574d728a:::benchmark_data_smithi188_113957_object60880:head# 2024-04-23T16:33:39.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574e33c7:::benchmark_data_smithi188_113957_object148799:head# 2024-04-23T16:33:39.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574e6153:::benchmark_data_smithi188_113957_object39319:head# 2024-04-23T16:33:39.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574e61fc:::benchmark_data_smithi188_113957_object38928:head# 2024-04-23T16:33:39.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574e8dd3:::benchmark_data_smithi188_113957_object11397:head# 2024-04-23T16:33:39.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574ec714:::benchmark_data_smithi188_113957_object44133:head# 2024-04-23T16:33:39.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574ed7c3:::benchmark_data_smithi188_113957_object128428:head# 2024-04-23T16:33:39.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574ee984:::benchmark_data_smithi188_113957_object40757:head# 2024-04-23T16:33:39.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574f569e:::benchmark_data_smithi188_113957_object15010:head# 2024-04-23T16:33:39.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574f59ac:::benchmark_data_smithi188_113957_object88561:head# 2024-04-23T16:33:39.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574fb635:::benchmark_data_smithi188_113957_object38743:head# 2024-04-23T16:33:39.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:574fd865:::benchmark_data_smithi188_113957_object55330:head# 2024-04-23T16:33:39.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5750c0bc:::benchmark_data_smithi188_113957_object56642:head# 2024-04-23T16:33:39.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5750df0a:::benchmark_data_smithi188_113957_object14406:head# 2024-04-23T16:33:39.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:575165f4:::benchmark_data_smithi188_113957_object13975:head# 2024-04-23T16:33:39.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57518189:::benchmark_data_smithi188_113957_object40078:head# 2024-04-23T16:33:39.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5751adef:::benchmark_data_smithi188_113957_object107641:head# 2024-04-23T16:33:39.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57538c9b:::benchmark_data_smithi188_113957_object113234:head# 2024-04-23T16:33:39.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:575413e4:::benchmark_data_smithi188_113957_object10924:head# 2024-04-23T16:33:39.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5754236f:::benchmark_data_smithi188_113957_object136414:head# 2024-04-23T16:33:39.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5754c0fb:::benchmark_data_smithi188_113957_object78277:head# 2024-04-23T16:33:39.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5754c710:::benchmark_data_smithi188_113957_object102629:head# 2024-04-23T16:33:39.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5754d1b0:::benchmark_data_smithi188_113957_object4656:head# 2024-04-23T16:33:39.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5755592d:::benchmark_data_smithi188_113957_object68392:head# 2024-04-23T16:33:39.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:575587f2:::benchmark_data_smithi188_113957_object29861:head# 2024-04-23T16:33:39.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5755d132:::benchmark_data_smithi188_113957_object113977:head# 2024-04-23T16:33:39.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5755fef8:::benchmark_data_smithi188_113957_object71438:head# 2024-04-23T16:33:39.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57560c53:::benchmark_data_smithi188_113957_object130175:head# 2024-04-23T16:33:39.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:575681b3:::benchmark_data_smithi188_113957_object107999:head# 2024-04-23T16:33:39.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5756823d:::benchmark_data_smithi188_113957_object74540:head# 2024-04-23T16:33:39.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5756f844:::benchmark_data_smithi188_113957_object93410:head# 2024-04-23T16:33:39.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:575733bb:::benchmark_data_smithi188_113957_object79934:head# 2024-04-23T16:33:39.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57582abd:::benchmark_data_smithi188_113957_object71902:head# 2024-04-23T16:33:39.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57597fd3:::benchmark_data_smithi188_113957_object1792:head# 2024-04-23T16:33:39.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5759b67c:::benchmark_data_smithi188_113957_object111301:head# 2024-04-23T16:33:39.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:575a873a:::benchmark_data_smithi188_113957_object70954:head# 2024-04-23T16:33:39.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:575a94ba:::benchmark_data_smithi188_113957_object110102:head# 2024-04-23T16:33:39.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:575b1c2f:::benchmark_data_smithi188_113957_object124070:head# 2024-04-23T16:33:39.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:575b9d8e:::benchmark_data_smithi188_113957_object6861:head# 2024-04-23T16:33:39.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:575beb5d:::benchmark_data_smithi188_113957_object94881:head# 2024-04-23T16:33:39.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:575c701a:::benchmark_data_smithi188_113957_object127091:head# 2024-04-23T16:33:39.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:575d225f:::benchmark_data_smithi188_113957_object97135:head# 2024-04-23T16:33:39.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:575d2717:::benchmark_data_smithi188_113957_object57620:head# 2024-04-23T16:33:39.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:575d65b7:::benchmark_data_smithi188_113957_object2521:head# 2024-04-23T16:33:39.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:575dcf28:::benchmark_data_smithi188_113957_object71247:head# 2024-04-23T16:33:39.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:575e71bd:::benchmark_data_smithi188_113957_object65643:head# 2024-04-23T16:33:39.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:575ee3ea:::benchmark_data_smithi188_113957_object53080:head# 2024-04-23T16:33:39.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:575f1573:::benchmark_data_smithi188_113957_object79702:head# 2024-04-23T16:33:39.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57604e42:::benchmark_data_smithi188_113957_object64368:head# 2024-04-23T16:33:39.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576092d0:::benchmark_data_smithi188_113957_object120956:head# 2024-04-23T16:33:39.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57609872:::benchmark_data_smithi188_113957_object23271:head# 2024-04-23T16:33:39.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5760c71b:::benchmark_data_smithi188_113957_object146507:head# 2024-04-23T16:33:39.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5760c80c:::benchmark_data_smithi188_113957_object76401:head# 2024-04-23T16:33:39.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5760d3a9:::benchmark_data_smithi188_113957_object120422:head# 2024-04-23T16:33:39.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576237d4:::benchmark_data_smithi188_113957_object148143:head# 2024-04-23T16:33:39.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57624bd3:::benchmark_data_smithi188_113957_object35788:head# 2024-04-23T16:33:39.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57626730:::benchmark_data_smithi188_113957_object78833:head# 2024-04-23T16:33:39.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5763b539:::benchmark_data_smithi188_113957_object79210:head# 2024-04-23T16:33:39.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57644b07:::benchmark_data_smithi188_113957_object98132:head# 2024-04-23T16:33:39.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57647693:::benchmark_data_smithi188_113957_object93262:head# 2024-04-23T16:33:39.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5764c2d8:::benchmark_data_smithi188_113957_object93955:head# 2024-04-23T16:33:39.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5765bd9c:::benchmark_data_smithi188_113957_object127953:head# 2024-04-23T16:33:39.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57662e9e:::benchmark_data_smithi188_113957_object141441:head# 2024-04-23T16:33:39.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5766b551:::benchmark_data_smithi188_113957_object33895:head# 2024-04-23T16:33:39.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57678ac1:::benchmark_data_smithi188_113957_object21480:head# 2024-04-23T16:33:39.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5767d771:::benchmark_data_smithi188_113957_object126587:head# 2024-04-23T16:33:39.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5767e808:::benchmark_data_smithi188_113957_object129221:head# 2024-04-23T16:33:39.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57685dc5:::benchmark_data_smithi188_113957_object11845:head# 2024-04-23T16:33:39.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5768afae:::benchmark_data_smithi188_113957_object128236:head# 2024-04-23T16:33:39.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576915eb:::benchmark_data_smithi188_113957_object73218:head# 2024-04-23T16:33:39.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576920ff:::benchmark_data_smithi188_113957_object36090:head# 2024-04-23T16:33:39.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576939a5:::benchmark_data_smithi188_113957_object134339:head# 2024-04-23T16:33:39.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5769e2ac:::benchmark_data_smithi188_113957_object143858:head# 2024-04-23T16:33:39.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5769e691:::benchmark_data_smithi188_113957_object18301:head# 2024-04-23T16:33:39.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576aa988:::benchmark_data_smithi188_113957_object1435:head# 2024-04-23T16:33:39.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576ab6ab:::benchmark_data_smithi188_113957_object94081:head# 2024-04-23T16:33:39.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576b5186:::benchmark_data_smithi188_113957_object76847:head# 2024-04-23T16:33:39.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576bc57d:::benchmark_data_smithi188_113957_object141822:head# 2024-04-23T16:33:39.814 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576c29f3:::benchmark_data_smithi188_113957_object36397:head# 2024-04-23T16:33:39.814 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576c42ea:::benchmark_data_smithi188_113957_object71766:head# 2024-04-23T16:33:39.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576c5f07:::benchmark_data_smithi188_113957_object59256:head# 2024-04-23T16:33:39.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576cace4:::benchmark_data_smithi188_113957_object122299:head# 2024-04-23T16:33:39.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576cce04:::benchmark_data_smithi188_113957_object125100:head# 2024-04-23T16:33:39.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576cd365:::benchmark_data_smithi188_113957_object145555:head# 2024-04-23T16:33:39.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576d65db:::benchmark_data_smithi188_113957_object1593:head# 2024-04-23T16:33:39.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576d813a:::benchmark_data_smithi188_113957_object78944:head# 2024-04-23T16:33:39.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576d8f7c:::benchmark_data_smithi188_113957_object1243:head# 2024-04-23T16:33:39.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576e517a:::benchmark_data_smithi188_113957_object137241:head# 2024-04-23T16:33:39.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576e8764:::benchmark_data_smithi188_113957_object89980:head# 2024-04-23T16:33:39.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576fa95f:::benchmark_data_smithi188_113957_object153295:head# 2024-04-23T16:33:39.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:576fb195:::benchmark_data_smithi188_113957_object84979:head# 2024-04-23T16:33:39.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57708a3a:::benchmark_data_smithi188_113957_object4146:head# 2024-04-23T16:33:39.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5770adb6:::benchmark_data_smithi188_113957_object52041:head# 2024-04-23T16:33:39.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5770af9e:::benchmark_data_smithi188_113957_object43843:head# 2024-04-23T16:33:39.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5770f897:::benchmark_data_smithi188_113957_object32921:head# 2024-04-23T16:33:39.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57711e26:::benchmark_data_smithi188_113957_object31121:head# 2024-04-23T16:33:39.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57718c06:::benchmark_data_smithi188_113957_object57730:head# 2024-04-23T16:33:39.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5771e1f0:::benchmark_data_smithi188_113957_object113014:head# 2024-04-23T16:33:39.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57723bde:::benchmark_data_smithi188_113957_object102816:head# 2024-04-23T16:33:39.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57733afd:::benchmark_data_smithi188_113957_object30177:head# 2024-04-23T16:33:39.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57738630:::benchmark_data_smithi188_113957_object135773:head# 2024-04-23T16:33:39.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5773d475:::benchmark_data_smithi188_113957_object51844:head# 2024-04-23T16:33:39.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57748b39:::benchmark_data_smithi188_113957_object74316:head# 2024-04-23T16:33:39.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57751892:::benchmark_data_smithi188_113957_object114464:head# 2024-04-23T16:33:39.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57752883:::benchmark_data_smithi188_113957_object148873:head# 2024-04-23T16:33:39.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57756b8b:::benchmark_data_smithi188_113957_object48563:head# 2024-04-23T16:33:39.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5776d2d9:::benchmark_data_smithi188_113957_object80122:head# 2024-04-23T16:33:39.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:577772e0:::benchmark_data_smithi188_113957_object15333:head# 2024-04-23T16:33:39.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5777a242:::benchmark_data_smithi188_113957_object96214:head# 2024-04-23T16:33:39.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5777e8e4:::benchmark_data_smithi188_113957_object34794:head# 2024-04-23T16:33:39.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:577827b3:::benchmark_data_smithi188_113957_object2750:head# 2024-04-23T16:33:39.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57783dfd:::benchmark_data_smithi188_113957_object1146:head# 2024-04-23T16:33:39.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57789957:::benchmark_data_smithi188_113957_object2722:head# 2024-04-23T16:33:39.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5778f0c6:::benchmark_data_smithi188_113957_object35975:head# 2024-04-23T16:33:39.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:577923ec:::benchmark_data_smithi188_113957_object65536:head# 2024-04-23T16:33:39.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5779337d:::benchmark_data_smithi188_113957_object77268:head# 2024-04-23T16:33:39.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57794f9c:::benchmark_data_smithi188_113957_object83309:head# 2024-04-23T16:33:39.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5779f19a:::benchmark_data_smithi188_113957_object146373:head# 2024-04-23T16:33:39.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:577a0dbc:::benchmark_data_smithi188_113957_object144529:head# 2024-04-23T16:33:39.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:577b08df:::benchmark_data_smithi188_113957_object49044:head# 2024-04-23T16:33:39.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:577b27b7:::benchmark_data_smithi188_113957_object45648:head# 2024-04-23T16:33:39.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:577bd70d:::benchmark_data_smithi188_113957_object45380:head# 2024-04-23T16:33:39.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:577bebc5:::benchmark_data_smithi188_113957_object26355:head# 2024-04-23T16:33:39.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:577c431d:::benchmark_data_smithi188_113957_object892:head# 2024-04-23T16:33:39.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:577c54eb:::benchmark_data_smithi188_113957_object67468:head# 2024-04-23T16:33:39.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:577cffda:::benchmark_data_smithi188_113957_object35870:head# 2024-04-23T16:33:39.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:577e48e5:::benchmark_data_smithi188_113957_object107158:head# 2024-04-23T16:33:39.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:577eb984:::benchmark_data_smithi188_113957_object81867:head# 2024-04-23T16:33:39.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:577f419f:::benchmark_data_smithi188_113957_object102873:head# 2024-04-23T16:33:39.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:577f7297:::benchmark_data_smithi188_113957_object100404:head# 2024-04-23T16:33:39.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57800fe3:::benchmark_data_smithi188_113957_object62014:head# 2024-04-23T16:33:39.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5780d1e0:::benchmark_data_smithi188_113957_object144614:head# 2024-04-23T16:33:39.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5781af37:::benchmark_data_smithi188_113957_object108487:head# 2024-04-23T16:33:39.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5781beb9:::benchmark_data_smithi188_113957_object91074:head# 2024-04-23T16:33:39.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5781cc30:::benchmark_data_smithi188_113957_object28385:head# 2024-04-23T16:33:39.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5781ead5:::benchmark_data_smithi188_113957_object31748:head# 2024-04-23T16:33:39.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5781ff72:::benchmark_data_smithi188_113957_object12625:head# 2024-04-23T16:33:39.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57822a52:::benchmark_data_smithi188_113957_object104738:head# 2024-04-23T16:33:39.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57828cb5:::benchmark_data_smithi188_113957_object12640:head# 2024-04-23T16:33:39.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5782fd44:::benchmark_data_smithi188_113957_object138495:head# 2024-04-23T16:33:39.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57833877:::benchmark_data_smithi188_113957_object26513:head# 2024-04-23T16:33:39.845 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57849a9d:::benchmark_data_smithi188_113957_object16405:head# 2024-04-23T16:33:39.845 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5784ac91:::benchmark_data_smithi188_113957_object130320:head# 2024-04-23T16:33:39.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5784c1ff:::benchmark_data_smithi188_113957_object48353:head# 2024-04-23T16:33:39.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5785c62f:::benchmark_data_smithi188_113957_object126519:head# 2024-04-23T16:33:39.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5785e53a:::benchmark_data_smithi188_113957_object42870:head# 2024-04-23T16:33:39.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57861a39:::benchmark_data_smithi188_113957_object87066:head# 2024-04-23T16:33:39.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5786e821:::benchmark_data_smithi188_113957_object5885:head# 2024-04-23T16:33:39.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5786faab:::benchmark_data_smithi188_113957_object46023:head# 2024-04-23T16:33:39.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57870230:::benchmark_data_smithi188_113957_object53857:head# 2024-04-23T16:33:39.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5787ad1f:::benchmark_data_smithi188_113957_object53409:head# 2024-04-23T16:33:39.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5788b2d8:::benchmark_data_smithi188_113957_object5199:head# 2024-04-23T16:33:39.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5788c7cd:::benchmark_data_smithi188_113957_object30484:head# 2024-04-23T16:33:39.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5788f1e3:::benchmark_data_smithi188_113957_object62889:head# 2024-04-23T16:33:39.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57891f83:::benchmark_data_smithi188_113957_object100213:head# 2024-04-23T16:33:39.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:578a2af3:::benchmark_data_smithi188_113957_object34799:head# 2024-04-23T16:33:39.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:578a828d:::benchmark_data_smithi188_113957_object115288:head# 2024-04-23T16:33:39.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:578a9b96:::benchmark_data_smithi188_113957_object79996:head# 2024-04-23T16:33:39.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:578b5622:::benchmark_data_smithi188_113957_object150796:head# 2024-04-23T16:33:39.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:578ba0e9:::benchmark_data_smithi188_113957_object123749:head# 2024-04-23T16:33:39.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:578c1de8:::benchmark_data_smithi188_113957_object78804:head# 2024-04-23T16:33:39.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:578cb083:::benchmark_data_smithi188_113957_object147437:head# 2024-04-23T16:33:39.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:578d62a6:::benchmark_data_smithi188_113957_object61152:head# 2024-04-23T16:33:39.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:578eb74a:::benchmark_data_smithi188_113957_object54896:head# 2024-04-23T16:33:39.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:578f66e9:::benchmark_data_smithi188_113957_object96301:head# 2024-04-23T16:33:39.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:578f9ef0:::benchmark_data_smithi188_113957_object89774:head# 2024-04-23T16:33:39.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:578fc749:::benchmark_data_smithi188_113957_object17013:head# 2024-04-23T16:33:39.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57905a5c:::benchmark_data_smithi188_113957_object141788:head# 2024-04-23T16:33:39.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5790cd4b:::benchmark_data_smithi188_113957_object75798:head# 2024-04-23T16:33:39.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5791091e:::benchmark_data_smithi188_113957_object142461:head# 2024-04-23T16:33:39.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5791a1cc:::benchmark_data_smithi188_113957_object17720:head# 2024-04-23T16:33:39.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57932d67:::benchmark_data_smithi188_113957_object66643:head# 2024-04-23T16:33:39.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57940ae1:::benchmark_data_smithi188_113957_object63507:head# 2024-04-23T16:33:39.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5794bce5:::benchmark_data_smithi188_113957_object149573:head# 2024-04-23T16:33:39.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5794de8c:::benchmark_data_smithi188_113957_object134441:head# 2024-04-23T16:33:39.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5795430c:::benchmark_data_smithi188_113957_object136821:head# 2024-04-23T16:33:39.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5795870f:::benchmark_data_smithi188_113957_object135692:head# 2024-04-23T16:33:39.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57958fda:::benchmark_data_smithi188_113957_object57376:head# 2024-04-23T16:33:39.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5795a392:::benchmark_data_smithi188_113957_object49596:head# 2024-04-23T16:33:39.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57963525:::benchmark_data_smithi188_113957_object74445:head# 2024-04-23T16:33:39.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57973d59:::benchmark_data_smithi188_113957_object84324:head# 2024-04-23T16:33:39.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57977c56:::benchmark_data_smithi188_113957_object47689:head# 2024-04-23T16:33:39.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5798e821:::benchmark_data_smithi188_113957_object56536:head# 2024-04-23T16:33:39.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57992907:::benchmark_data_smithi188_113957_object98601:head# 2024-04-23T16:33:39.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5799a1f2:::benchmark_data_smithi188_113957_object111743:head# 2024-04-23T16:33:39.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5799a29a:::benchmark_data_smithi188_113957_object79361:head# 2024-04-23T16:33:39.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5799f954:::benchmark_data_smithi188_113957_object77210:head# 2024-04-23T16:33:39.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:579c798b:::benchmark_data_smithi188_113957_object110564:head# 2024-04-23T16:33:39.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:579dc720:::benchmark_data_smithi188_113957_object79551:head# 2024-04-23T16:33:39.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:579e2120:::benchmark_data_smithi188_113957_object91216:head# 2024-04-23T16:33:39.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:579fb8cd:::benchmark_data_smithi188_113957_object25689:head# 2024-04-23T16:33:39.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:579fdab8:::benchmark_data_smithi188_113957_object108600:head# 2024-04-23T16:33:39.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a043fe:::benchmark_data_smithi188_113957_object108051:head# 2024-04-23T16:33:39.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a04605:::benchmark_data_smithi188_113957_object32247:head# 2024-04-23T16:33:39.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a06aa3:::benchmark_data_smithi188_113957_object20344:head# 2024-04-23T16:33:39.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a0ed07:::benchmark_data_smithi188_113957_object113169:head# 2024-04-23T16:33:39.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a17307:::benchmark_data_smithi188_113957_object66843:head# 2024-04-23T16:33:39.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a178f7:::benchmark_data_smithi188_113957_object24159:head# 2024-04-23T16:33:39.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a297f6:::benchmark_data_smithi188_113957_object110284:head# 2024-04-23T16:33:39.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a2eedf:::benchmark_data_smithi188_113957_object111233:head# 2024-04-23T16:33:39.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a356ba:::benchmark_data_smithi188_113957_object100979:head# 2024-04-23T16:33:39.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a457a1:::benchmark_data_smithi188_113957_object26801:head# 2024-04-23T16:33:39.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a58cd7:::benchmark_data_smithi188_113957_object129495:head# 2024-04-23T16:33:39.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a5c7ce:::benchmark_data_smithi188_113957_object76297:head# 2024-04-23T16:33:39.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a5ca32:::benchmark_data_smithi188_113957_object16132:head# 2024-04-23T16:33:39.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a5dd53:::benchmark_data_smithi188_113957_object151466:head# 2024-04-23T16:33:39.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a5e276:::benchmark_data_smithi188_113957_object140931:head# 2024-04-23T16:33:39.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a69639:::benchmark_data_smithi188_113957_object52804:head# 2024-04-23T16:33:39.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a6e3b9:::benchmark_data_smithi188_113957_object93127:head# 2024-04-23T16:33:39.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a747c4:::benchmark_data_smithi188_113957_object130440:head# 2024-04-23T16:33:39.875 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a88a3b:::benchmark_data_smithi188_113957_object84470:head# 2024-04-23T16:33:39.875 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a8fd61:::benchmark_data_smithi188_113957_object144084:head# 2024-04-23T16:33:39.875 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57a9cb4b:::benchmark_data_smithi188_113957_object132979:head# 2024-04-23T16:33:39.876 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57aa10c3:::benchmark_data_smithi188_113957_object5803:head# 2024-04-23T16:33:39.876 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57aa7dde:::benchmark_data_smithi188_113957_object76248:head# 2024-04-23T16:33:39.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ab2d5d:::benchmark_data_smithi188_113957_object424:head# 2024-04-23T16:33:39.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ab3e02:::benchmark_data_smithi188_113957_object4227:head# 2024-04-23T16:33:39.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ab4102:::benchmark_data_smithi188_113957_object119764:head# 2024-04-23T16:33:39.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ab6c03:::benchmark_data_smithi188_113957_object71570:head# 2024-04-23T16:33:39.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57abab2a:::benchmark_data_smithi188_113957_object129316:head# 2024-04-23T16:33:39.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57abb311:::benchmark_data_smithi188_113957_object21:head# 2024-04-23T16:33:39.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57abc94e:::benchmark_data_smithi188_113957_object1144:head# 2024-04-23T16:33:39.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ac60f6:::benchmark_data_smithi188_113957_object60584:head# 2024-04-23T16:33:39.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ac98a9:::benchmark_data_smithi188_113957_object74796:head# 2024-04-23T16:33:39.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ad0894:::benchmark_data_smithi188_113957_object142586:head# 2024-04-23T16:33:39.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ad0a3a:::benchmark_data_smithi188_113957_object116943:head# 2024-04-23T16:33:39.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ad2b28:::benchmark_data_smithi188_113957_object90029:head# 2024-04-23T16:33:39.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ad75a4:::benchmark_data_smithi188_113957_object111682:head# 2024-04-23T16:33:39.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57adda78:::benchmark_data_smithi188_113957_object75082:head# 2024-04-23T16:33:39.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57adfa39:::benchmark_data_smithi188_113957_object77000:head# 2024-04-23T16:33:39.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ae7730:::benchmark_data_smithi188_113957_object81346:head# 2024-04-23T16:33:39.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57aeabeb:::benchmark_data_smithi188_113957_object69898:head# 2024-04-23T16:33:39.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57aee28a:::benchmark_data_smithi188_113957_object4301:head# 2024-04-23T16:33:39.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57af2165:::benchmark_data_smithi188_113957_object102888:head# 2024-04-23T16:33:39.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57af65fd:::benchmark_data_smithi188_113957_object149208:head# 2024-04-23T16:33:39.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57af760d:::benchmark_data_smithi188_113957_object117646:head# 2024-04-23T16:33:39.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b08d4f:::benchmark_data_smithi188_113957_object99558:head# 2024-04-23T16:33:39.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b164fd:::benchmark_data_smithi188_113957_object139017:head# 2024-04-23T16:33:39.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b17856:::benchmark_data_smithi188_113957_object91767:head# 2024-04-23T16:33:39.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b245a9:::benchmark_data_smithi188_113957_object56912:head# 2024-04-23T16:33:39.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b251a3:::benchmark_data_smithi188_113957_object22063:head# 2024-04-23T16:33:39.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b2606c:::benchmark_data_smithi188_113957_object126586:head# 2024-04-23T16:33:39.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b27bf6:::benchmark_data_smithi188_113957_object895:head# 2024-04-23T16:33:39.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b2cebe:::benchmark_data_smithi188_113957_object116014:head# 2024-04-23T16:33:39.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b2e1f1:::benchmark_data_smithi188_113957_object149791:head# 2024-04-23T16:33:39.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b3e995:::benchmark_data_smithi188_113957_object51276:head# 2024-04-23T16:33:39.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b45d70:::benchmark_data_smithi188_113957_object97772:head# 2024-04-23T16:33:39.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b49cc3:::benchmark_data_smithi188_113957_object56913:head# 2024-04-23T16:33:39.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b530ad:::benchmark_data_smithi188_113957_object126714:head# 2024-04-23T16:33:39.891 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b5a5ab:::benchmark_data_smithi188_113957_object73961:head# 2024-04-23T16:33:39.891 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b67d67:::benchmark_data_smithi188_113957_object128597:head# 2024-04-23T16:33:39.892 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b6c087:::benchmark_data_smithi188_113957_object20066:head# 2024-04-23T16:33:39.892 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b71219:::benchmark_data_smithi188_113957_object136500:head# 2024-04-23T16:33:39.893 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b7dd48:::benchmark_data_smithi188_113957_object7063:head# 2024-04-23T16:33:39.893 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b9235c:::benchmark_data_smithi188_113957_object2594:head# 2024-04-23T16:33:39.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b93705:::benchmark_data_smithi188_113957_object129303:head# 2024-04-23T16:33:39.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b974e1:::benchmark_data_smithi188_113957_object25396:head# 2024-04-23T16:33:39.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57b99c6f:::benchmark_data_smithi188_113957_object89147:head# 2024-04-23T16:33:39.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ba06eb:::benchmark_data_smithi188_113957_object51791:head# 2024-04-23T16:33:39.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57bb875a:::benchmark_data_smithi188_113957_object91097:head# 2024-04-23T16:33:39.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57bbbaf5:::benchmark_data_smithi188_113957_object92126:head# 2024-04-23T16:33:39.896 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57bc22a3:::benchmark_data_smithi188_113957_object150987:head# 2024-04-23T16:33:39.896 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57bc5b13:::benchmark_data_smithi188_113957_object67057:head# 2024-04-23T16:33:39.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57be0a22:::benchmark_data_smithi188_113957_object48818:head# 2024-04-23T16:33:39.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57be5104:::benchmark_data_smithi188_113957_object131433:head# 2024-04-23T16:33:39.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57be8918:::benchmark_data_smithi188_113957_object152016:head# 2024-04-23T16:33:39.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57be8f92:::benchmark_data_smithi188_113957_object33379:head# 2024-04-23T16:33:39.899 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57bf2c39:::benchmark_data_smithi188_113957_object82363:head# 2024-04-23T16:33:39.899 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57bf90ad:::benchmark_data_smithi188_113957_object71423:head# 2024-04-23T16:33:39.900 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c05a40:::benchmark_data_smithi188_113957_object152312:head# 2024-04-23T16:33:39.900 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c13201:::benchmark_data_smithi188_113957_object102946:head# 2024-04-23T16:33:39.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c1ce1a:::benchmark_data_smithi188_113957_object41707:head# 2024-04-23T16:33:39.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c230d2:::benchmark_data_smithi188_113957_object106657:head# 2024-04-23T16:33:39.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c25e96:::benchmark_data_smithi188_113957_object107396:head# 2024-04-23T16:33:39.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c2780f:::benchmark_data_smithi188_113957_object98962:head# 2024-04-23T16:33:39.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c31f05:::benchmark_data_smithi188_113957_object91654:head# 2024-04-23T16:33:39.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c3849a:::benchmark_data_smithi188_113957_object114281:head# 2024-04-23T16:33:39.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c3a392:::benchmark_data_smithi188_113957_object52365:head# 2024-04-23T16:33:39.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c3e65a:::benchmark_data_smithi188_113957_object51447:head# 2024-04-23T16:33:39.904 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c4683b:::benchmark_data_smithi188_113957_object96183:head# 2024-04-23T16:33:39.904 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c4de45:::benchmark_data_smithi188_113957_object46827:head# 2024-04-23T16:33:39.904 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c537fd:::benchmark_data_smithi188_113957_object95605:head# 2024-04-23T16:33:39.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c5f23d:::benchmark_data_smithi188_113957_object64769:head# 2024-04-23T16:33:39.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c61ba4:::benchmark_data_smithi188_113957_object142000:head# 2024-04-23T16:33:39.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c698fb:::benchmark_data_smithi188_113957_object114527:head# 2024-04-23T16:33:39.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c6c264:::benchmark_data_smithi188_113957_object16663:head# 2024-04-23T16:33:39.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c852ad:::benchmark_data_smithi188_113957_object126120:head# 2024-04-23T16:33:39.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c8c5f6:::benchmark_data_smithi188_113957_object138978:head# 2024-04-23T16:33:39.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57c9b11c:::benchmark_data_smithi188_113957_object101125:head# 2024-04-23T16:33:39.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ca2be2:::benchmark_data_smithi188_113957_object61909:head# 2024-04-23T16:33:39.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ca5213:::benchmark_data_smithi188_113957_object75202:head# 2024-04-23T16:33:39.909 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57caff50:::benchmark_data_smithi188_113957_object103273:head# 2024-04-23T16:33:39.909 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57cbeca1:::benchmark_data_smithi188_113957_object130465:head# 2024-04-23T16:33:39.909 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ccc64f:::benchmark_data_smithi188_113957_object2952:head# 2024-04-23T16:33:39.910 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57cd1d0f:::benchmark_data_smithi188_113957_object90593:head# 2024-04-23T16:33:39.910 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57cd4e74:::benchmark_data_smithi188_113957_object49423:head# 2024-04-23T16:33:39.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57cdbdf6:::benchmark_data_smithi188_113957_object82469:head# 2024-04-23T16:33:39.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ce333e:::benchmark_data_smithi188_113957_object102329:head# 2024-04-23T16:33:39.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ce512e:::benchmark_data_smithi188_113957_object67211:head# 2024-04-23T16:33:39.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ced4c1:::benchmark_data_smithi188_113957_object145106:head# 2024-04-23T16:33:39.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57cf5d23:::benchmark_data_smithi188_113957_object112546:head# 2024-04-23T16:33:39.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57d04835:::benchmark_data_smithi188_113957_object13656:head# 2024-04-23T16:33:39.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57d08a97:::benchmark_data_smithi188_113957_object15849:head# 2024-04-23T16:33:39.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57d0daf9:::benchmark_data_smithi188_113957_object123596:head# 2024-04-23T16:33:39.914 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57d1601d:::benchmark_data_smithi188_113957_object85424:head# 2024-04-23T16:33:39.914 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57d215c6:::benchmark_data_smithi188_113957_object118040:head# 2024-04-23T16:33:39.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57d275d5:::benchmark_data_smithi188_113957_object30166:head# 2024-04-23T16:33:39.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57d2e470:::benchmark_data_smithi188_113957_object38281:head# 2024-04-23T16:33:39.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57d30d2e:::benchmark_data_smithi188_113957_object21741:head# 2024-04-23T16:33:39.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57d36aa0:::benchmark_data_smithi188_113957_object38139:head# 2024-04-23T16:33:39.917 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57d37efd:::benchmark_data_smithi188_113957_object127480:head# 2024-04-23T16:33:39.917 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57d3d442:::benchmark_data_smithi188_113957_object44502:head# 2024-04-23T16:33:39.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57d67768:::benchmark_data_smithi188_113957_object105288:head# 2024-04-23T16:33:39.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57d67b07:::benchmark_data_smithi188_113957_object82989:head# 2024-04-23T16:33:39.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57d7e883:::benchmark_data_smithi188_113957_object255:head# 2024-04-23T16:33:39.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57d80ca5:::benchmark_data_smithi188_113957_object18360:head# 2024-04-23T16:33:39.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57d86933:::benchmark_data_smithi188_113957_object133046:head# 2024-04-23T16:33:39.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57d87c63:::benchmark_data_smithi188_113957_object40313:head# 2024-04-23T16:33:39.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57d88791:::benchmark_data_smithi188_113957_object56396:head# 2024-04-23T16:33:39.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57d9e08e:::benchmark_data_smithi188_113957_object60717:head# 2024-04-23T16:33:39.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57da15db:::benchmark_data_smithi188_113957_object103764:head# 2024-04-23T16:33:39.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57dafb31:::benchmark_data_smithi188_113957_object65418:head# 2024-04-23T16:33:39.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57db1827:::benchmark_data_smithi188_113957_object65003:head# 2024-04-23T16:33:39.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57dc42c9:::benchmark_data_smithi188_113957_object3135:head# 2024-04-23T16:33:39.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57dc7932:::benchmark_data_smithi188_113957_object40545:head# 2024-04-23T16:33:39.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57de05ae:::benchmark_data_smithi188_113957_object145781:head# 2024-04-23T16:33:39.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57de5fc9:::benchmark_data_smithi188_113957_object131961:head# 2024-04-23T16:33:39.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57de7612:::benchmark_data_smithi188_113957_object30845:head# 2024-04-23T16:33:39.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57de8dd8:::benchmark_data_smithi188_113957_object19920:head# 2024-04-23T16:33:39.925 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57deb9a6:::benchmark_data_smithi188_113957_object125686:head# 2024-04-23T16:33:39.925 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57defb69:::benchmark_data_smithi188_113957_object133679:head# 2024-04-23T16:33:39.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57dfbe06:::benchmark_data_smithi188_113957_object33114:head# 2024-04-23T16:33:39.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57dfef7b:::benchmark_data_smithi188_113957_object119582:head# 2024-04-23T16:33:39.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57e1a4d7:::benchmark_data_smithi188_113957_object85670:head# 2024-04-23T16:33:39.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57e2336d:::benchmark_data_smithi188_113957_object12431:head# 2024-04-23T16:33:39.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57e36b40:::benchmark_data_smithi188_113957_object136672:head# 2024-04-23T16:33:39.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57e3f827:::benchmark_data_smithi188_113957_object47880:head# 2024-04-23T16:33:39.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57e40c40:::benchmark_data_smithi188_113957_object133375:head# 2024-04-23T16:33:39.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57e41e0b:::benchmark_data_smithi188_113957_object130098:head# 2024-04-23T16:33:39.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57e445ec:::benchmark_data_smithi188_113957_object134323:head# 2024-04-23T16:33:39.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57e4645b:::benchmark_data_smithi188_113957_object37168:head# 2024-04-23T16:33:39.930 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57e5775e:::benchmark_data_smithi188_113957_object75468:head# 2024-04-23T16:33:39.930 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57e5abd1:::benchmark_data_smithi188_113957_object13147:head# 2024-04-23T16:33:39.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57e6d1f8:::benchmark_data_smithi188_113957_object37039:head# 2024-04-23T16:33:39.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57e6e872:::benchmark_data_smithi188_113957_object15935:head# 2024-04-23T16:33:39.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57e719e6:::benchmark_data_smithi188_113957_object43777:head# 2024-04-23T16:33:39.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57e7271c:::benchmark_data_smithi188_113957_object86503:head# 2024-04-23T16:33:39.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57e7fd99:::benchmark_data_smithi188_113957_object82296:head# 2024-04-23T16:33:39.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57e8a2fb:::benchmark_data_smithi188_113957_object35740:head# 2024-04-23T16:33:39.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57e91521:::benchmark_data_smithi188_113957_object84633:head# 2024-04-23T16:33:39.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ebdce6:::benchmark_data_smithi188_113957_object31488:head# 2024-04-23T16:33:39.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ec5ee9:::benchmark_data_smithi188_113957_object8090:head# 2024-04-23T16:33:39.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ec6ede:::benchmark_data_smithi188_113957_object137742:head# 2024-04-23T16:33:39.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ecfcde:::benchmark_data_smithi188_113957_object55133:head# 2024-04-23T16:33:39.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ee6d36:::benchmark_data_smithi188_113957_object120038:head# 2024-04-23T16:33:39.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57eeaa18:::benchmark_data_smithi188_113957_object9064:head# 2024-04-23T16:33:39.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57eefd5b:::benchmark_data_smithi188_113957_object106010:head# 2024-04-23T16:33:39.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ef3e63:::benchmark_data_smithi188_113957_object82350:head# 2024-04-23T16:33:39.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ef75e0:::benchmark_data_smithi188_113957_object27601:head# 2024-04-23T16:33:39.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ef9395:::benchmark_data_smithi188_113957_object48319:head# 2024-04-23T16:33:39.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f039f9:::benchmark_data_smithi188_113957_object143077:head# 2024-04-23T16:33:39.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f05545:::benchmark_data_smithi188_113957_object84546:head# 2024-04-23T16:33:39.939 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f14759:::benchmark_data_smithi188_113957_object86881:head# 2024-04-23T16:33:39.939 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f17121:::benchmark_data_smithi188_113957_object140061:head# 2024-04-23T16:33:39.940 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f1a29e:::benchmark_data_smithi188_113957_object148861:head# 2024-04-23T16:33:39.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f1bf0a:::benchmark_data_smithi188_113957_object104653:head# 2024-04-23T16:33:39.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f1bf6f:::benchmark_data_smithi188_113957_object114002:head# 2024-04-23T16:33:39.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f202f6:::benchmark_data_smithi188_113957_object19100:head# 2024-04-23T16:33:39.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f21ec1:::benchmark_data_smithi188_113957_object39838:head# 2024-04-23T16:33:39.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f2e3fa:::benchmark_data_smithi188_113957_object27957:head# 2024-04-23T16:33:39.943 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f375dc:::benchmark_data_smithi188_113957_object101411:head# 2024-04-23T16:33:39.943 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f436bf:::benchmark_data_smithi188_113957_object70359:head# 2024-04-23T16:33:39.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f482d3:::benchmark_data_smithi188_113957_object28866:head# 2024-04-23T16:33:39.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f486e0:::benchmark_data_smithi188_113957_object105380:head# 2024-04-23T16:33:39.945 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f50d77:::benchmark_data_smithi188_113957_object39802:head# 2024-04-23T16:33:39.945 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f51d02:::benchmark_data_smithi188_113957_object98804:head# 2024-04-23T16:33:39.945 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f5301e:::benchmark_data_smithi188_113957_object17406:head# 2024-04-23T16:33:39.947 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f67fbd:::benchmark_data_smithi188_113957_object52453:head# 2024-04-23T16:33:39.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f75fae:::benchmark_data_smithi188_113957_object54898:head# 2024-04-23T16:33:39.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f7c79b:::benchmark_data_smithi188_113957_object130693:head# 2024-04-23T16:33:39.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f82ab0:::benchmark_data_smithi188_113957_object7153:head# 2024-04-23T16:33:39.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57f8f06a:::benchmark_data_smithi188_113957_object102492:head# 2024-04-23T16:33:39.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57fa1ca4:::benchmark_data_smithi188_113957_object4637:head# 2024-04-23T16:33:39.949 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57fa9780:::benchmark_data_smithi188_113957_object46658:head# 2024-04-23T16:33:39.949 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57fa9bb9:::benchmark_data_smithi188_113957_object49484:head# 2024-04-23T16:33:39.949 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57fb6af4:::benchmark_data_smithi188_113957_object51260:head# 2024-04-23T16:33:39.949 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57fb6cd0:::benchmark_data_smithi188_113957_object100951:head# 2024-04-23T16:33:39.950 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57fbab73:::benchmark_data_smithi188_113957_object26614:head# 2024-04-23T16:33:39.950 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57fbc8e8:::benchmark_data_smithi188_113957_object109256:head# 2024-04-23T16:33:39.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57fbe8fd:::benchmark_data_smithi188_113957_object42238:head# 2024-04-23T16:33:39.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57fc0ecc:::benchmark_data_smithi188_113957_object37007:head# 2024-04-23T16:33:39.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57fc45e9:::benchmark_data_smithi188_113957_object118297:head# 2024-04-23T16:33:39.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57fc6ca5:::benchmark_data_smithi188_113957_object124862:head# 2024-04-23T16:33:39.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57fcf7bd:::benchmark_data_smithi188_113957_object75467:head# 2024-04-23T16:33:39.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57fd3bdb:::benchmark_data_smithi188_113957_object67939:head# 2024-04-23T16:33:39.953 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57fd5dbf:::benchmark_data_smithi188_113957_object66270:head# 2024-04-23T16:33:39.953 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57fd6f1b:::benchmark_data_smithi188_113957_object152791:head# 2024-04-23T16:33:39.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57fda357:::benchmark_data_smithi188_113957_object26099:head# 2024-04-23T16:33:39.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57fe2ff7:::benchmark_data_smithi188_113957_object102844:head# 2024-04-23T16:33:39.955 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57fe5552:::benchmark_data_smithi188_113957_object47289:head# 2024-04-23T16:33:39.955 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ff1bbf:::benchmark_data_smithi188_113957_object78181:head# 2024-04-23T16:33:39.956 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:57ff96ce:::benchmark_data_smithi188_113957_object45361:head# 2024-04-23T16:33:39.956 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58007301:::benchmark_data_smithi188_113957_object117317:head# 2024-04-23T16:33:39.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5800e7ea:::benchmark_data_smithi188_113957_object108477:head# 2024-04-23T16:33:39.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58018fdd:::benchmark_data_smithi188_113957_object82330:head# 2024-04-23T16:33:39.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5801ae8b:::benchmark_data_smithi188_113957_object133271:head# 2024-04-23T16:33:39.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58023b19:::benchmark_data_smithi188_113957_object31303:head# 2024-04-23T16:33:39.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5803353c:::benchmark_data_smithi188_113957_object77392:head# 2024-04-23T16:33:39.959 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5803d7d2:::benchmark_data_smithi188_113957_object66605:head# 2024-04-23T16:33:39.959 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58041828:::benchmark_data_smithi188_113957_object92939:head# 2024-04-23T16:33:39.959 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5804539c:::benchmark_data_smithi188_113957_object25171:head# 2024-04-23T16:33:39.960 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58047c1e:::benchmark_data_smithi188_113957_object107288:head# 2024-04-23T16:33:39.960 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58048d39:::benchmark_data_smithi188_113957_object110654:head# 2024-04-23T16:33:39.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5804b5c2:::benchmark_data_smithi188_113957_object84382:head# 2024-04-23T16:33:39.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5804c544:::benchmark_data_smithi188_113957_object57435:head# 2024-04-23T16:33:39.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58051f9e:::benchmark_data_smithi188_113957_object130734:head# 2024-04-23T16:33:39.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58058395:::benchmark_data_smithi188_113957_object134670:head# 2024-04-23T16:33:39.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58059ddb:::benchmark_data_smithi188_113957_object54014:head# 2024-04-23T16:33:39.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5805fc91:::benchmark_data_smithi188_113957_object27376:head# 2024-04-23T16:33:39.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58060475:::benchmark_data_smithi188_113957_object144556:head# 2024-04-23T16:33:39.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58062a9f:::benchmark_data_smithi188_113957_object30130:head# 2024-04-23T16:33:39.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5807128d:::benchmark_data_smithi188_113957_object142658:head# 2024-04-23T16:33:39.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:580725e4:::benchmark_data_smithi188_113957_object128862:head# 2024-04-23T16:33:39.965 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58073a08:::benchmark_data_smithi188_113957_object33269:head# 2024-04-23T16:33:39.966 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58074b16:::benchmark_data_smithi188_113957_object127385:head# 2024-04-23T16:33:39.967 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:580759aa:::benchmark_data_smithi188_113957_object144945:head# 2024-04-23T16:33:39.967 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58086069:::benchmark_data_smithi188_113957_object46336:head# 2024-04-23T16:33:39.967 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:580871d1:::benchmark_data_smithi188_113957_object99121:head# 2024-04-23T16:33:39.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58088108:::benchmark_data_smithi188_113957_object139262:head# 2024-04-23T16:33:39.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58090bcc:::benchmark_data_smithi188_113957_object13812:head# 2024-04-23T16:33:39.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58091cc3:::benchmark_data_smithi188_113957_object101912:head# 2024-04-23T16:33:39.969 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:580a5034:::benchmark_data_smithi188_113957_object36301:head# 2024-04-23T16:33:39.969 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:580a9710:::benchmark_data_smithi188_113957_object10377:head# 2024-04-23T16:33:39.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:580b81b7:::benchmark_data_smithi188_113957_object125939:head# 2024-04-23T16:33:39.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:580bfd7d:::benchmark_data_smithi188_113957_object138002:head# 2024-04-23T16:33:39.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:580c56fc:::benchmark_data_smithi188_113957_object144546:head# 2024-04-23T16:33:39.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:580c6a7d:::benchmark_data_smithi188_113957_object61813:head# 2024-04-23T16:33:39.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:580cdc9c:::benchmark_data_smithi188_113957_object68203:head# 2024-04-23T16:33:39.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:580da1c5:::benchmark_data_smithi188_113957_object25127:head# 2024-04-23T16:33:39.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:580e5702:::benchmark_data_smithi188_113957_object42302:head# 2024-04-23T16:33:39.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:580efd60:::benchmark_data_smithi188_113957_object119412:head# 2024-04-23T16:33:39.973 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:580f1846:::benchmark_data_smithi188_113957_object136632:head# 2024-04-23T16:33:39.973 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:580f52a1:::benchmark_data_smithi188_113957_object98880:head# 2024-04-23T16:33:39.974 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:580fc87b:::benchmark_data_smithi188_113957_object130204:head# 2024-04-23T16:33:39.974 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:580fd78a:::benchmark_data_smithi188_113957_object7334:head# 2024-04-23T16:33:39.974 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58108348:::benchmark_data_smithi188_113957_object89434:head# 2024-04-23T16:33:39.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5810b06a:::benchmark_data_smithi188_113957_object10560:head# 2024-04-23T16:33:39.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5812a634:::benchmark_data_smithi188_113957_object11267:head# 2024-04-23T16:33:39.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5812caac:::benchmark_data_smithi188_113957_object38440:head# 2024-04-23T16:33:39.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58131a3c:::benchmark_data_smithi188_113957_object108806:head# 2024-04-23T16:33:39.977 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5813d075:::benchmark_data_smithi188_113957_object101433:head# 2024-04-23T16:33:39.977 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58142b99:::benchmark_data_smithi188_113957_object23120:head# 2024-04-23T16:33:39.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:581453b8:::benchmark_data_smithi188_113957_object10099:head# 2024-04-23T16:33:39.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5814ad7d:::benchmark_data_smithi188_113957_object84609:head# 2024-04-23T16:33:39.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58151bb6:::benchmark_data_smithi188_113957_object21599:head# 2024-04-23T16:33:39.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5815aba6:::benchmark_data_smithi188_113957_object16613:head# 2024-04-23T16:33:39.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5815afde:::benchmark_data_smithi188_113957_object123217:head# 2024-04-23T16:33:39.980 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5815e3f6:::benchmark_data_smithi188_113957_object17898:head# 2024-04-23T16:33:39.980 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5815ea4a:::benchmark_data_smithi188_113957_object153113:head# 2024-04-23T16:33:39.980 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:581665cc:::benchmark_data_smithi188_113957_object14236:head# 2024-04-23T16:33:39.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58178d81:::benchmark_data_smithi188_113957_object139238:head# 2024-04-23T16:33:39.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5817f235:::benchmark_data_smithi188_113957_object92478:head# 2024-04-23T16:33:39.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:581821e6:::benchmark_data_smithi188_113957_object134642:head# 2024-04-23T16:33:39.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58182565:::benchmark_data_smithi188_113957_object47598:head# 2024-04-23T16:33:39.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58192d41:::benchmark_data_smithi188_113957_object30911:head# 2024-04-23T16:33:39.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58198125:::benchmark_data_smithi188_113957_object31590:head# 2024-04-23T16:33:39.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5819efb1:::benchmark_data_smithi188_113957_object16601:head# 2024-04-23T16:33:39.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:581a1cd2:::benchmark_data_smithi188_113957_object13210:head# 2024-04-23T16:33:39.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:581bb4db:::benchmark_data_smithi188_113957_object106742:head# 2024-04-23T16:33:39.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:581bc38c:::benchmark_data_smithi188_113957_object33736:head# 2024-04-23T16:33:39.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:581bdaa6:::benchmark_data_smithi188_113957_object69246:head# 2024-04-23T16:33:39.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:581dbed1:::benchmark_data_smithi188_113957_object28099:head# 2024-04-23T16:33:39.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:581e016b:::benchmark_data_smithi188_113957_object125551:head# 2024-04-23T16:33:39.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:581f2f71:::benchmark_data_smithi188_113957_object88482:head# 2024-04-23T16:33:39.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58201084:::benchmark_data_smithi188_113957_object90144:head# 2024-04-23T16:33:39.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58202ba5:::benchmark_data_smithi188_113957_object111919:head# 2024-04-23T16:33:39.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58208abe:::benchmark_data_smithi188_113957_object109667:head# 2024-04-23T16:33:39.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5821336a:::benchmark_data_smithi188_113957_object43616:head# 2024-04-23T16:33:39.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5822c6b0:::benchmark_data_smithi188_113957_object92266:head# 2024-04-23T16:33:39.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5823aac0:::benchmark_data_smithi188_113957_object47763:head# 2024-04-23T16:33:39.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58247bc9:::benchmark_data_smithi188_113957_object63953:head# 2024-04-23T16:33:39.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:582553f5:::benchmark_data_smithi188_113957_object22241:head# 2024-04-23T16:33:39.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5825db45:::benchmark_data_smithi188_113957_object30476:head# 2024-04-23T16:33:39.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58261bc6:::benchmark_data_smithi188_113957_object71856:head# 2024-04-23T16:33:39.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5827d3af:::benchmark_data_smithi188_113957_object126919:head# 2024-04-23T16:33:39.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:582872f3:::benchmark_data_smithi188_113957_object140723:head# 2024-04-23T16:33:39.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5828e201:::benchmark_data_smithi188_113957_object81603:head# 2024-04-23T16:33:39.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5829cc2d:::benchmark_data_smithi188_113957_object35184:head# 2024-04-23T16:33:39.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5829fb9e:::benchmark_data_smithi188_113957_object89867:head# 2024-04-23T16:33:39.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:582a5484:::benchmark_data_smithi188_113957_object90194:head# 2024-04-23T16:33:39.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:582a87ea:::benchmark_data_smithi188_113957_object149672:head# 2024-04-23T16:33:39.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:582b0b23:::benchmark_data_smithi188_113957_object122653:head# 2024-04-23T16:33:39.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:582b4ed4:::benchmark_data_smithi188_113957_object4029:head# 2024-04-23T16:33:39.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:582bd1b8:::benchmark_data_smithi188_113957_object91718:head# 2024-04-23T16:33:39.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:582c0844:::benchmark_data_smithi188_113957_object126348:head# 2024-04-23T16:33:39.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:582c6594:::benchmark_data_smithi188_113957_object2752:head# 2024-04-23T16:33:39.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:582d0dd9:::benchmark_data_smithi188_113957_object19533:head# 2024-04-23T16:33:39.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:582d3d49:::benchmark_data_smithi188_113957_object138132:head# 2024-04-23T16:33:39.997 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:582f9125:::benchmark_data_smithi188_113957_object42857:head# 2024-04-23T16:33:39.997 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:582fa491:::benchmark_data_smithi188_113957_object142726:head# 2024-04-23T16:33:39.997 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:583011cc:::benchmark_data_smithi188_113957_object117372:head# 2024-04-23T16:33:39.998 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58303ddc:::benchmark_data_smithi188_113957_object50942:head# 2024-04-23T16:33:39.998 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:583072c9:::benchmark_data_smithi188_113957_object61997:head# 2024-04-23T16:33:39.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58309916:::benchmark_data_smithi188_113957_object21902:head# 2024-04-23T16:33:39.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58312adb:::benchmark_data_smithi188_113957_object40382:head# 2024-04-23T16:33:40.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58313b64:::benchmark_data_smithi188_113957_object102160:head# 2024-04-23T16:33:40.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58316c2c:::benchmark_data_smithi188_113957_object96458:head# 2024-04-23T16:33:40.001 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58322366:::benchmark_data_smithi188_113957_object39602:head# 2024-04-23T16:33:40.001 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5832c8a6:::benchmark_data_smithi188_113957_object21610:head# 2024-04-23T16:33:40.001 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5832d169:::benchmark_data_smithi188_113957_object17035:head# 2024-04-23T16:33:40.002 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58339242:::benchmark_data_smithi188_113957_object83564:head# 2024-04-23T16:33:40.002 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5833d3f3:::benchmark_data_smithi188_113957_object89528:head# 2024-04-23T16:33:40.004 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58349524:::benchmark_data_smithi188_113957_object15727:head# 2024-04-23T16:33:40.004 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5834a379:::benchmark_data_smithi188_113957_object45820:head# 2024-04-23T16:33:40.004 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:583581b0:::benchmark_data_smithi188_113957_object6360:head# 2024-04-23T16:33:40.004 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5836b995:::benchmark_data_smithi188_113957_object143734:head# 2024-04-23T16:33:40.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58371423:::benchmark_data_smithi188_113957_object107202:head# 2024-04-23T16:33:40.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58378e5d:::benchmark_data_smithi188_113957_object141020:head# 2024-04-23T16:33:40.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5838e8a6:::benchmark_data_smithi188_113957_object21178:head# 2024-04-23T16:33:40.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58392d37:::benchmark_data_smithi188_113957_object128632:head# 2024-04-23T16:33:40.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58394cbe:::benchmark_data_smithi188_113957_object45170:head# 2024-04-23T16:33:40.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5839b946:::benchmark_data_smithi188_113957_object124617:head# 2024-04-23T16:33:40.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:583a09c8:::benchmark_data_smithi188_113957_object147699:head# 2024-04-23T16:33:40.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:583a239c:::benchmark_data_smithi188_113957_object50613:head# 2024-04-23T16:33:40.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:583a7907:::benchmark_data_smithi188_113957_object104645:head# 2024-04-23T16:33:40.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:583ac4f3:::benchmark_data_smithi188_113957_object25760:head# 2024-04-23T16:33:40.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:583b81f5:::benchmark_data_smithi188_113957_object139450:head# 2024-04-23T16:33:40.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:583b8435:::benchmark_data_smithi188_113957_object135833:head# 2024-04-23T16:33:40.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:583b8f4c:::benchmark_data_smithi188_113957_object100338:head# 2024-04-23T16:33:40.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:583bc214:::benchmark_data_smithi188_113957_object77849:head# 2024-04-23T16:33:40.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:583bd25a:::benchmark_data_smithi188_113957_object73475:head# 2024-04-23T16:33:40.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:583c50dd:::benchmark_data_smithi188_113957_object146749:head# 2024-04-23T16:33:40.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:583d3a4b:::benchmark_data_smithi188_113957_object139607:head# 2024-04-23T16:33:40.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:583d898f:::benchmark_data_smithi188_113957_object11838:head# 2024-04-23T16:33:40.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:583deaf2:::benchmark_data_smithi188_113957_object109343:head# 2024-04-23T16:33:40.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:583e754d:::benchmark_data_smithi188_113957_object26183:head# 2024-04-23T16:33:40.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:583e9e66:::benchmark_data_smithi188_113957_object87134:head# 2024-04-23T16:33:40.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:583ec2d1:::benchmark_data_smithi188_113957_object80468:head# 2024-04-23T16:33:40.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5840c3fa:::benchmark_data_smithi188_113957_object122054:head# 2024-04-23T16:33:40.013 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:40 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:33:40.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:40 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:33:40.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:40 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/25397802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:33:40.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:40 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 27626/296312 objects degraded (9.323%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:40.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.015 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.015 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5840ec8d:::benchmark_data_smithi188_113957_object19968:head# 2024-04-23T16:33:40.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58419d45:::benchmark_data_smithi188_113957_object104421:head# 2024-04-23T16:33:40.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5841a570:::benchmark_data_smithi188_113957_object129577:head# 2024-04-23T16:33:40.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5841c79e:::benchmark_data_smithi188_113957_object51126:head# 2024-04-23T16:33:40.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58421698:::benchmark_data_smithi188_113957_object136536:head# 2024-04-23T16:33:40.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5843bcd9:::benchmark_data_smithi188_113957_object99912:head# 2024-04-23T16:33:40.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5844aeaf:::benchmark_data_smithi188_113957_object97486:head# 2024-04-23T16:33:40.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5844d8b7:::benchmark_data_smithi188_113957_object48946:head# 2024-04-23T16:33:40.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58455935:::benchmark_data_smithi188_113957_object89437:head# 2024-04-23T16:33:40.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58465884:::benchmark_data_smithi188_113957_object92075:head# 2024-04-23T16:33:40.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5846d6fa:::benchmark_data_smithi188_113957_object30290:head# 2024-04-23T16:33:40.018 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58471649:::benchmark_data_smithi188_113957_object27401:head# 2024-04-23T16:33:40.018 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5848aa7b:::benchmark_data_smithi188_113957_object107389:head# 2024-04-23T16:33:40.019 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5848c663:::benchmark_data_smithi188_113957_object61305:head# 2024-04-23T16:33:40.019 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584accf7:::benchmark_data_smithi188_113957_object55803:head# 2024-04-23T16:33:40.019 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584afb58:::benchmark_data_smithi188_113957_object28477:head# 2024-04-23T16:33:40.020 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584b5ba4:::benchmark_data_smithi188_113957_object55291:head# 2024-04-23T16:33:40.020 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584ba18c:::benchmark_data_smithi188_113957_object123304:head# 2024-04-23T16:33:40.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584ba2a3:::benchmark_data_smithi188_113957_object143068:head# 2024-04-23T16:33:40.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584be48f:::benchmark_data_smithi188_113957_object22131:head# 2024-04-23T16:33:40.022 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584c13f0:::benchmark_data_smithi188_113957_object96130:head# 2024-04-23T16:33:40.022 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584c5d46:::benchmark_data_smithi188_113957_object97079:head# 2024-04-23T16:33:40.022 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584c888e:::benchmark_data_smithi188_113957_object115598:head# 2024-04-23T16:33:40.023 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584c9048:::benchmark_data_smithi188_113957_object44929:head# 2024-04-23T16:33:40.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584d3975:::benchmark_data_smithi188_113957_object153497:head# 2024-04-23T16:33:40.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584d569d:::benchmark_data_smithi188_113957_object15911:head# 2024-04-23T16:33:40.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584d713d:::benchmark_data_smithi188_113957_object140392:head# 2024-04-23T16:33:40.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584dc194:::benchmark_data_smithi188_113957_object17266:head# 2024-04-23T16:33:40.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584dc6af:::benchmark_data_smithi188_113957_object107508:head# 2024-04-23T16:33:40.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584dd1a8:::benchmark_data_smithi188_113957_object118392:head# 2024-04-23T16:33:40.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584ded34:::benchmark_data_smithi188_113957_object15978:head# 2024-04-23T16:33:40.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584e438c:::benchmark_data_smithi188_113957_object28969:head# 2024-04-23T16:33:40.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584f0f56:::benchmark_data_smithi188_113957_object145109:head# 2024-04-23T16:33:40.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584f35c4:::benchmark_data_smithi188_113957_object119348:head# 2024-04-23T16:33:40.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584f3746:::benchmark_data_smithi188_113957_object96292:head# 2024-04-23T16:33:40.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:584fe7a4:::benchmark_data_smithi188_113957_object38761:head# 2024-04-23T16:33:40.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58505439:::benchmark_data_smithi188_113957_object53393:head# 2024-04-23T16:33:40.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:585137ec:::benchmark_data_smithi188_113957_object43934:head# 2024-04-23T16:33:40.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58515fa0:::benchmark_data_smithi188_113957_object103268:head# 2024-04-23T16:33:40.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5851c39d:::benchmark_data_smithi188_113957_object91590:head# 2024-04-23T16:33:40.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5852c9be:::benchmark_data_smithi188_113957_object123805:head# 2024-04-23T16:33:40.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:585397ce:::benchmark_data_smithi188_113957_object55640:head# 2024-04-23T16:33:40.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:585427df:::benchmark_data_smithi188_113957_object138897:head# 2024-04-23T16:33:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:585452f1:::benchmark_data_smithi188_113957_object46249:head# 2024-04-23T16:33:40.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58546670:::benchmark_data_smithi188_113957_object59847:head# 2024-04-23T16:33:40.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58548d4a:::benchmark_data_smithi188_113957_object38277:head# 2024-04-23T16:33:40.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5854aca5:::benchmark_data_smithi188_113957_object68128:head# 2024-04-23T16:33:40.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58557a88:::benchmark_data_smithi188_113957_object106951:head# 2024-04-23T16:33:40.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:585774e2:::benchmark_data_smithi188_113957_object149260:head# 2024-04-23T16:33:40.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58581d7d:::benchmark_data_smithi188_113957_object135825:head# 2024-04-23T16:33:40.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58587316:::benchmark_data_smithi188_113957_object42987:head# 2024-04-23T16:33:40.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:585891b0:::benchmark_data_smithi188_113957_object107404:head# 2024-04-23T16:33:40.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5858a650:::benchmark_data_smithi188_113957_object143502:head# 2024-04-23T16:33:40.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5859332c:::benchmark_data_smithi188_113957_object81919:head# 2024-04-23T16:33:40.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5859862f:::benchmark_data_smithi188_113957_object115804:head# 2024-04-23T16:33:40.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:585a15b4:::benchmark_data_smithi188_113957_object60875:head# 2024-04-23T16:33:40.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:585a2294:::benchmark_data_smithi188_113957_object72497:head# 2024-04-23T16:33:40.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:585b9dc7:::benchmark_data_smithi188_113957_object65861:head# 2024-04-23T16:33:40.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:585be0dd:::benchmark_data_smithi188_113957_object93292:head# 2024-04-23T16:33:40.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:585c9ba5:::benchmark_data_smithi188_113957_object114339:head# 2024-04-23T16:33:40.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:585cad81:::benchmark_data_smithi188_113957_object82377:head# 2024-04-23T16:33:40.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:585cf7f7:::benchmark_data_smithi188_113957_object64254:head# 2024-04-23T16:33:40.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:585d13a7:::benchmark_data_smithi188_113957_object49018:head# 2024-04-23T16:33:40.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:585de950:::benchmark_data_smithi188_113957_object35250:head# 2024-04-23T16:33:40.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:585f5af0:::benchmark_data_smithi188_113957_object12827:head# 2024-04-23T16:33:40.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:585fdb8c:::benchmark_data_smithi188_113957_object85554:head# 2024-04-23T16:33:40.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58601820:::benchmark_data_smithi188_113957_object63653:head# 2024-04-23T16:33:40.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5860918a:::benchmark_data_smithi188_113957_object95957:head# 2024-04-23T16:33:40.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:586093b3:::benchmark_data_smithi188_113957_object130476:head# 2024-04-23T16:33:40.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:586095b8:::benchmark_data_smithi188_113957_object16401:head# 2024-04-23T16:33:40.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5860b4c2:::benchmark_data_smithi188_113957_object88205:head# 2024-04-23T16:33:40.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5860f334:::benchmark_data_smithi188_113957_object98027:head# 2024-04-23T16:33:40.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:586114d7:::benchmark_data_smithi188_113957_object84341:head# 2024-04-23T16:33:40.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58612b16:::benchmark_data_smithi188_113957_object21507:head# 2024-04-23T16:33:40.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5862a162:::benchmark_data_smithi188_113957_object45227:head# 2024-04-23T16:33:40.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5862d424:::benchmark_data_smithi188_113957_object6743:head# 2024-04-23T16:33:40.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5862f380:::benchmark_data_smithi188_113957_object50668:head# 2024-04-23T16:33:40.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58639bbe:::benchmark_data_smithi188_113957_object35917:head# 2024-04-23T16:33:40.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5863aff9:::benchmark_data_smithi188_113957_object54390:head# 2024-04-23T16:33:40.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5863d825:::benchmark_data_smithi188_113957_object117923:head# 2024-04-23T16:33:40.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5863f1e0:::benchmark_data_smithi188_113957_object7635:head# 2024-04-23T16:33:40.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58647ff5:::benchmark_data_smithi188_113957_object61741:head# 2024-04-23T16:33:40.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5864dc6b:::benchmark_data_smithi188_113957_object101461:head# 2024-04-23T16:33:40.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5864dd97:::benchmark_data_smithi188_113957_object98258:head# 2024-04-23T16:33:40.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5864e178:::benchmark_data_smithi188_113957_object59012:head# 2024-04-23T16:33:40.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58654ef1:::benchmark_data_smithi188_113957_object120300:head# 2024-04-23T16:33:40.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5866b4e9:::benchmark_data_smithi188_113957_object64738:head# 2024-04-23T16:33:40.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5866e76c:::benchmark_data_smithi188_113957_object105861:head# 2024-04-23T16:33:40.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5867b990:::benchmark_data_smithi188_113957_object109976:head# 2024-04-23T16:33:40.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5867f786:::benchmark_data_smithi188_113957_object143911:head# 2024-04-23T16:33:40.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:586807e7:::benchmark_data_smithi188_113957_object6347:head# 2024-04-23T16:33:40.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:586823af:::benchmark_data_smithi188_113957_object103786:head# 2024-04-23T16:33:40.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58686790:::benchmark_data_smithi188_113957_object19807:head# 2024-04-23T16:33:40.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:586969f5:::benchmark_data_smithi188_113957_object57789:head# 2024-04-23T16:33:40.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5869a2d3:::benchmark_data_smithi188_113957_object109327:head# 2024-04-23T16:33:40.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5869b108:::benchmark_data_smithi188_113957_object12507:head# 2024-04-23T16:33:40.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5869c35b:::benchmark_data_smithi188_113957_object46326:head# 2024-04-23T16:33:40.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:586a4499:::benchmark_data_smithi188_113957_object35050:head# 2024-04-23T16:33:40.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:586a8651:::benchmark_data_smithi188_113957_object84268:head# 2024-04-23T16:33:40.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:586b0500:::benchmark_data_smithi188_113957_object101933:head# 2024-04-23T16:33:40.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:586b6591:::benchmark_data_smithi188_113957_object129327:head# 2024-04-23T16:33:40.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:586bf793:::benchmark_data_smithi188_113957_object12798:head# 2024-04-23T16:33:40.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:586c6613:::benchmark_data_smithi188_113957_object32229:head# 2024-04-23T16:33:40.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:586cb5c8:::benchmark_data_smithi188_113957_object619:head# 2024-04-23T16:33:40.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:586db4f6:::benchmark_data_smithi188_113957_object94744:head# 2024-04-23T16:33:40.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:586e8c5d:::benchmark_data_smithi188_113957_object13928:head# 2024-04-23T16:33:40.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:586ebaf6:::benchmark_data_smithi188_113957_object140389:head# 2024-04-23T16:33:40.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:586f885d:::benchmark_data_smithi188_113957_object8289:head# 2024-04-23T16:33:40.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58700eae:::benchmark_data_smithi188_113957_object145867:head# 2024-04-23T16:33:40.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58726dc7:::benchmark_data_smithi188_113957_object84480:head# 2024-04-23T16:33:40.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5872b537:::benchmark_data_smithi188_113957_object99395:head# 2024-04-23T16:33:40.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5872f998:::benchmark_data_smithi188_113957_object49982:head# 2024-04-23T16:33:40.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58730a49:::benchmark_data_smithi188_113957_object65320:head# 2024-04-23T16:33:40.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587327a3:::benchmark_data_smithi188_113957_object39284:head# 2024-04-23T16:33:40.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5873d77a:::benchmark_data_smithi188_113957_object78442:head# 2024-04-23T16:33:40.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587424e8:::benchmark_data_smithi188_113957_object49479:head# 2024-04-23T16:33:40.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58742b1e:::benchmark_data_smithi188_113957_object135267:head# 2024-04-23T16:33:40.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58748fd8:::benchmark_data_smithi188_113957_object133250:head# 2024-04-23T16:33:40.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5874c485:::benchmark_data_smithi188_113957_object18437:head# 2024-04-23T16:33:40.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587575c8:::benchmark_data_smithi188_113957_object132345:head# 2024-04-23T16:33:40.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58760ff6:::benchmark_data_smithi188_113957_object139225:head# 2024-04-23T16:33:40.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587646c4:::benchmark_data_smithi188_113957_object103936:head# 2024-04-23T16:33:40.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5876b04a:::benchmark_data_smithi188_113957_object8722:head# 2024-04-23T16:33:40.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5876b0d7:::benchmark_data_smithi188_113957_object44257:head# 2024-04-23T16:33:40.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5877dee7:::benchmark_data_smithi188_113957_object54834:head# 2024-04-23T16:33:40.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58789604:::benchmark_data_smithi188_113957_object118826:head# 2024-04-23T16:33:40.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587897f8:::benchmark_data_smithi188_113957_object1111:head# 2024-04-23T16:33:40.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587962e8:::benchmark_data_smithi188_113957_object113308:head# 2024-04-23T16:33:40.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587a98ab:::benchmark_data_smithi188_113957_object2285:head# 2024-04-23T16:33:40.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587aecff:::benchmark_data_smithi188_113957_object77403:head# 2024-04-23T16:33:40.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587b09ed:::benchmark_data_smithi188_113957_object141861:head# 2024-04-23T16:33:40.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587b1bdb:::benchmark_data_smithi188_113957_object109170:head# 2024-04-23T16:33:40.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587b1e35:::benchmark_data_smithi188_113957_object91036:head# 2024-04-23T16:33:40.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587b7e46:::benchmark_data_smithi188_113957_object104535:head# 2024-04-23T16:33:40.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587be6bc:::benchmark_data_smithi188_113957_object68320:head# 2024-04-23T16:33:40.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587be729:::benchmark_data_smithi188_113957_object131133:head# 2024-04-23T16:33:40.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587c0b44:::benchmark_data_smithi188_113957_object139153:head# 2024-04-23T16:33:40.087 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587ca75b:::benchmark_data_smithi188_113957_object126778:head# 2024-04-23T16:33:40.087 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587cd8d5:::benchmark_data_smithi188_113957_object84679:head# 2024-04-23T16:33:40.087 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587ead3e:::benchmark_data_smithi188_113957_object43404:head# 2024-04-23T16:33:40.088 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587ee470:::benchmark_data_smithi188_113957_object96997:head# 2024-04-23T16:33:40.088 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587f05fb:::benchmark_data_smithi188_113957_object136395:head# 2024-04-23T16:33:40.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587f0c05:::benchmark_data_smithi188_113957_object38831:head# 2024-04-23T16:33:40.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587f18a7:::benchmark_data_smithi188_113957_object116086:head# 2024-04-23T16:33:40.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:587f4fa1:::benchmark_data_smithi188_113957_object83437:head# 2024-04-23T16:33:40.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5880319d:::benchmark_data_smithi188_113957_object19836:head# 2024-04-23T16:33:40.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5880eeca:::benchmark_data_smithi188_113957_object146221:head# 2024-04-23T16:33:40.091 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5881484c:::benchmark_data_smithi188_113957_object76652:head# 2024-04-23T16:33:40.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58818dc5:::benchmark_data_smithi188_113957_object108899:head# 2024-04-23T16:33:40.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5882d556:::benchmark_data_smithi188_113957_object60852:head# 2024-04-23T16:33:40.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58831788:::benchmark_data_smithi188_113957_object3506:head# 2024-04-23T16:33:40.093 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58839dc2:::benchmark_data_smithi188_113957_object115454:head# 2024-04-23T16:33:40.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5883b2c9:::benchmark_data_smithi188_113957_object125812:head# 2024-04-23T16:33:40.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5883cb15:::benchmark_data_smithi188_113957_object108637:head# 2024-04-23T16:33:40.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5884bf47:::benchmark_data_smithi188_113957_object3205:head# 2024-04-23T16:33:40.095 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5884e6c6:::benchmark_data_smithi188_113957_object35561:head# 2024-04-23T16:33:40.095 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5884f500:::benchmark_data_smithi188_113957_object152961:head# 2024-04-23T16:33:40.096 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58860eb0:::benchmark_data_smithi188_113957_object122941:head# 2024-04-23T16:33:40.096 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58864214:::benchmark_data_smithi188_113957_object52147:head# 2024-04-23T16:33:40.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58871b99:::benchmark_data_smithi188_113957_object66648:head# 2024-04-23T16:33:40.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58873dbb:::benchmark_data_smithi188_113957_object71971:head# 2024-04-23T16:33:40.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58873dea:::benchmark_data_smithi188_113957_object83734:head# 2024-04-23T16:33:40.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58877221:::benchmark_data_smithi188_113957_object17513:head# 2024-04-23T16:33:40.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58878201:::benchmark_data_smithi188_113957_object102239:head# 2024-04-23T16:33:40.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5888d14e:::benchmark_data_smithi188_113957_object57820:head# 2024-04-23T16:33:40.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58896a58:::benchmark_data_smithi188_113957_object47910:head# 2024-04-23T16:33:40.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58896cb1:::benchmark_data_smithi188_113957_object25818:head# 2024-04-23T16:33:40.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5889d3d3:::benchmark_data_smithi188_113957_object23042:head# 2024-04-23T16:33:40.102 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588a1799:::benchmark_data_smithi188_113957_object116341:head# 2024-04-23T16:33:40.102 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588a1ee0:::benchmark_data_smithi188_113957_object42206:head# 2024-04-23T16:33:40.102 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588a3667:::benchmark_data_smithi188_113957_object62478:head# 2024-04-23T16:33:40.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588a6022:::benchmark_data_smithi188_113957_object104911:head# 2024-04-23T16:33:40.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588a6af2:::benchmark_data_smithi188_113957_object34974:head# 2024-04-23T16:33:40.105 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588abbaa:::benchmark_data_smithi188_113957_object7168:head# 2024-04-23T16:33:40.105 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588acf15:::benchmark_data_smithi188_113957_object117399:head# 2024-04-23T16:33:40.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588b8d76:::benchmark_data_smithi188_113957_object51036:head# 2024-04-23T16:33:40.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588bc5a1:::benchmark_data_smithi188_113957_object143550:head# 2024-04-23T16:33:40.107 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588c59ee:::benchmark_data_smithi188_113957_object121416:head# 2024-04-23T16:33:40.107 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588d2011:::benchmark_data_smithi188_113957_object61521:head# 2024-04-23T16:33:40.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588d3733:::benchmark_data_smithi188_113957_object16214:head# 2024-04-23T16:33:40.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588d836c:::benchmark_data_smithi188_113957_object49070:head# 2024-04-23T16:33:40.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588dab13:::benchmark_data_smithi188_113957_object76748:head# 2024-04-23T16:33:40.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588e42ce:::benchmark_data_smithi188_113957_object122625:head# 2024-04-23T16:33:40.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588eb8a5:::benchmark_data_smithi188_113957_object93517:head# 2024-04-23T16:33:40.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588f1487:::benchmark_data_smithi188_113957_object46350:head# 2024-04-23T16:33:40.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588f3368:::benchmark_data_smithi188_113957_object90697:head# 2024-04-23T16:33:40.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588f5623:::benchmark_data_smithi188_113957_object92600:head# 2024-04-23T16:33:40.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:588ff667:::benchmark_data_smithi188_113957_object101801:head# 2024-04-23T16:33:40.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:589089c2:::benchmark_data_smithi188_113957_object22566:head# 2024-04-23T16:33:40.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5890f800:::benchmark_data_smithi188_113957_object26006:head# 2024-04-23T16:33:40.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58911912:::benchmark_data_smithi188_113957_object64120:head# 2024-04-23T16:33:40.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5891eabb:::benchmark_data_smithi188_113957_object34809:head# 2024-04-23T16:33:40.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5891ebc8:::benchmark_data_smithi188_113957_object46581:head# 2024-04-23T16:33:40.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:589250ee:::benchmark_data_smithi188_113957_object120549:head# 2024-04-23T16:33:40.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58925816:::benchmark_data_smithi188_113957_object98408:head# 2024-04-23T16:33:40.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5892d14c:::benchmark_data_smithi188_113957_object22515:head# 2024-04-23T16:33:40.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5892fa55:::benchmark_data_smithi188_113957_object145356:head# 2024-04-23T16:33:40.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5893200c:::benchmark_data_smithi188_113957_object64622:head# 2024-04-23T16:33:40.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58935269:::benchmark_data_smithi188_113957_object102897:head# 2024-04-23T16:33:40.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58940952:::benchmark_data_smithi188_113957_object116590:head# 2024-04-23T16:33:40.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5894a8b4:::benchmark_data_smithi188_113957_object23530:head# 2024-04-23T16:33:40.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5894c68c:::benchmark_data_smithi188_113957_object43124:head# 2024-04-23T16:33:40.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58952771:::benchmark_data_smithi188_113957_object34043:head# 2024-04-23T16:33:40.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58952d27:::benchmark_data_smithi188_113957_object142314:head# 2024-04-23T16:33:40.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5895669d:::benchmark_data_smithi188_113957_object23484:head# 2024-04-23T16:33:40.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5896a7ef:::benchmark_data_smithi188_113957_object150378:head# 2024-04-23T16:33:40.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5898bc40:::benchmark_data_smithi188_113957_object47595:head# 2024-04-23T16:33:40.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58995fb7:::benchmark_data_smithi188_113957_object4223:head# 2024-04-23T16:33:40.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58996de3:::benchmark_data_smithi188_113957_object127277:head# 2024-04-23T16:33:40.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:589a98e9:::benchmark_data_smithi188_113957_object38058:head# 2024-04-23T16:33:40.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:589ad526:::benchmark_data_smithi188_113957_object112214:head# 2024-04-23T16:33:40.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:589c0360:::benchmark_data_smithi188_113957_object108663:head# 2024-04-23T16:33:40.122 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:589c2964:::benchmark_data_smithi188_113957_object95085:head# 2024-04-23T16:33:40.122 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:589c525e:::benchmark_data_smithi188_113957_object122526:head# 2024-04-23T16:33:40.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:589d1698:::benchmark_data_smithi188_113957_object80636:head# 2024-04-23T16:33:40.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:589d2906:::benchmark_data_smithi188_113957_object151979:head# 2024-04-23T16:33:40.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:589d343a:::benchmark_data_smithi188_113957_object25550:head# 2024-04-23T16:33:40.124 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:589d4c3f:::benchmark_data_smithi188_113957_object152485:head# 2024-04-23T16:33:40.124 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:589e1bc5:::benchmark_data_smithi188_113957_object28830:head# 2024-04-23T16:33:40.125 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:589e9ee5:::benchmark_data_smithi188_113957_object95690:head# 2024-04-23T16:33:40.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:589eff36:::benchmark_data_smithi188_113957_object83331:head# 2024-04-23T16:33:40.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a0c6e5:::benchmark_data_smithi188_113957_object55729:head# 2024-04-23T16:33:40.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a124c1:::benchmark_data_smithi188_113957_object42861:head# 2024-04-23T16:33:40.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a1be8c:::benchmark_data_smithi188_113957_object314:head# 2024-04-23T16:33:40.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a1e2b4:::benchmark_data_smithi188_113957_object51687:head# 2024-04-23T16:33:40.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a21224:::benchmark_data_smithi188_113957_object87462:head# 2024-04-23T16:33:40.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a2313d:::benchmark_data_smithi188_113957_object37784:head# 2024-04-23T16:33:40.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a25967:::benchmark_data_smithi188_113957_object105923:head# 2024-04-23T16:33:40.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a268f4:::benchmark_data_smithi188_113957_object91747:head# 2024-04-23T16:33:40.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a31a06:::benchmark_data_smithi188_113957_object90449:head# 2024-04-23T16:33:40.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a386f5:::benchmark_data_smithi188_113957_object61760:head# 2024-04-23T16:33:40.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a39973:::benchmark_data_smithi188_113957_object136095:head# 2024-04-23T16:33:40.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a43393:::benchmark_data_smithi188_113957_object22869:head# 2024-04-23T16:33:40.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a4b344:::benchmark_data_smithi188_113957_object105234:head# 2024-04-23T16:33:40.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a5032d:::benchmark_data_smithi188_113957_object78859:head# 2024-04-23T16:33:40.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a50f4b:::benchmark_data_smithi188_113957_object14351:head# 2024-04-23T16:33:40.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a51ad2:::benchmark_data_smithi188_113957_object145501:head# 2024-04-23T16:33:40.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a58891:::benchmark_data_smithi188_113957_object36564:head# 2024-04-23T16:33:40.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a5c8ba:::benchmark_data_smithi188_113957_object96948:head# 2024-04-23T16:33:40.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a5db8b:::benchmark_data_smithi188_113957_object119398:head# 2024-04-23T16:33:40.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a5f7cf:::benchmark_data_smithi188_113957_object20449:head# 2024-04-23T16:33:40.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a612c2:::benchmark_data_smithi188_113957_object40832:head# 2024-04-23T16:33:40.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a62389:::benchmark_data_smithi188_113957_object102781:head# 2024-04-23T16:33:40.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a70890:::benchmark_data_smithi188_113957_object42578:head# 2024-04-23T16:33:40.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a83369:::benchmark_data_smithi188_113957_object103501:head# 2024-04-23T16:33:40.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a83aee:::benchmark_data_smithi188_113957_object21662:head# 2024-04-23T16:33:40.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a86a00:::benchmark_data_smithi188_113957_object120198:head# 2024-04-23T16:33:40.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a8df29:::benchmark_data_smithi188_113957_object47235:head# 2024-04-23T16:33:40.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a92c72:::benchmark_data_smithi188_113957_object87105:head# 2024-04-23T16:33:40.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a96432:::benchmark_data_smithi188_113957_object54801:head# 2024-04-23T16:33:40.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58a9dcf9:::benchmark_data_smithi188_113957_object64417:head# 2024-04-23T16:33:40.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ab20d8:::benchmark_data_smithi188_113957_object97237:head# 2024-04-23T16:33:40.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ab46f2:::benchmark_data_smithi188_113957_object117954:head# 2024-04-23T16:33:40.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58abcbd8:::benchmark_data_smithi188_113957_object24225:head# 2024-04-23T16:33:40.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ac8934:::benchmark_data_smithi188_113957_object55135:head# 2024-04-23T16:33:40.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58acdce5:::benchmark_data_smithi188_113957_object99230:head# 2024-04-23T16:33:40.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58adb1d8:::benchmark_data_smithi188_113957_object28298:head# 2024-04-23T16:33:40.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58add20f:::benchmark_data_smithi188_113957_object143584:head# 2024-04-23T16:33:40.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ae2cda:::benchmark_data_smithi188_113957_object54469:head# 2024-04-23T16:33:40.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ae3b55:::benchmark_data_smithi188_113957_object20174:head# 2024-04-23T16:33:40.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ae7d57:::benchmark_data_smithi188_113957_object100508:head# 2024-04-23T16:33:40.142 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58aeb0c1:::benchmark_data_smithi188_113957_object71800:head# 2024-04-23T16:33:40.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58af9292:::benchmark_data_smithi188_113957_object152243:head# 2024-04-23T16:33:40.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b0aed6:::benchmark_data_smithi188_113957_object30766:head# 2024-04-23T16:33:40.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b22b33:::benchmark_data_smithi188_113957_object63771:head# 2024-04-23T16:33:40.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b29eab:::benchmark_data_smithi188_113957_object140487:head# 2024-04-23T16:33:40.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b2f4ad:::benchmark_data_smithi188_113957_object103863:head# 2024-04-23T16:33:40.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b34d06:::benchmark_data_smithi188_113957_object73512:head# 2024-04-23T16:33:40.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b351e8:::benchmark_data_smithi188_113957_object47392:head# 2024-04-23T16:33:40.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b3745b:::benchmark_data_smithi188_113957_object78090:head# 2024-04-23T16:33:40.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b42131:::benchmark_data_smithi188_113957_object26797:head# 2024-04-23T16:33:40.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b4316a:::benchmark_data_smithi188_113957_object940:head# 2024-04-23T16:33:40.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b47b4f:::benchmark_data_smithi188_113957_object22973:head# 2024-04-23T16:33:40.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b53270:::benchmark_data_smithi188_113957_object123985:head# 2024-04-23T16:33:40.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b53eaf:::benchmark_data_smithi188_113957_object15139:head# 2024-04-23T16:33:40.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b5f277:::benchmark_data_smithi188_113957_object130811:head# 2024-04-23T16:33:40.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b68354:::benchmark_data_smithi188_113957_object47870:head# 2024-04-23T16:33:40.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b7392b:::benchmark_data_smithi188_113957_object99717:head# 2024-04-23T16:33:40.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b74972:::benchmark_data_smithi188_113957_object96521:head# 2024-04-23T16:33:40.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b76f23:::benchmark_data_smithi188_113957_object92181:head# 2024-04-23T16:33:40.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b7a658:::benchmark_data_smithi188_113957_object57404:head# 2024-04-23T16:33:40.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b8a0d4:::benchmark_data_smithi188_113957_object33376:head# 2024-04-23T16:33:40.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b93222:::benchmark_data_smithi188_113957_object42687:head# 2024-04-23T16:33:40.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58b983ba:::benchmark_data_smithi188_113957_object67112:head# 2024-04-23T16:33:40.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ba1120:::benchmark_data_smithi188_113957_object2261:head# 2024-04-23T16:33:40.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ba21b8:::benchmark_data_smithi188_113957_object28929:head# 2024-04-23T16:33:40.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58bbfe42:::benchmark_data_smithi188_113957_object126765:head# 2024-04-23T16:33:40.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58bc0add:::benchmark_data_smithi188_113957_object131595:head# 2024-04-23T16:33:40.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58bc3f1b:::benchmark_data_smithi188_113957_object76001:head# 2024-04-23T16:33:40.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58bc9172:::benchmark_data_smithi188_113957_object61699:head# 2024-04-23T16:33:40.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58bd4c43:::benchmark_data_smithi188_113957_object144322:head# 2024-04-23T16:33:40.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58bd77ca:::benchmark_data_smithi188_113957_object32391:head# 2024-04-23T16:33:40.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58bd9060:::benchmark_data_smithi188_113957_object38556:head# 2024-04-23T16:33:40.155 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58bdcd07:::benchmark_data_smithi188_113957_object64189:head# 2024-04-23T16:33:40.155 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58bdd88e:::benchmark_data_smithi188_113957_object138064:head# 2024-04-23T16:33:40.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58be1ef5:::benchmark_data_smithi188_113957_object63260:head# 2024-04-23T16:33:40.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58be65e4:::benchmark_data_smithi188_113957_object83561:head# 2024-04-23T16:33:40.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58be8718:::benchmark_data_smithi188_113957_object36323:head# 2024-04-23T16:33:40.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58beea73:::benchmark_data_smithi188_113957_object38650:head# 2024-04-23T16:33:40.158 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58bfd8d3:::benchmark_data_smithi188_113957_object72163:head# 2024-04-23T16:33:40.158 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c049d4:::benchmark_data_smithi188_113957_object22153:head# 2024-04-23T16:33:40.158 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c08380:::benchmark_data_smithi188_113957_object20512:head# 2024-04-23T16:33:40.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c0885c:::benchmark_data_smithi188_113957_object54938:head# 2024-04-23T16:33:40.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c0e2cb:::benchmark_data_smithi188_113957_object90353:head# 2024-04-23T16:33:40.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c14214:::benchmark_data_smithi188_113957_object128997:head# 2024-04-23T16:33:40.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c16583:::benchmark_data_smithi188_113957_object29219:head# 2024-04-23T16:33:40.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c177f8:::benchmark_data_smithi188_113957_object128836:head# 2024-04-23T16:33:40.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c1bad0:::benchmark_data_smithi188_113957_object38892:head# 2024-04-23T16:33:40.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c2164b:::benchmark_data_smithi188_113957_object19355:head# 2024-04-23T16:33:40.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c21fe4:::benchmark_data_smithi188_113957_object36936:head# 2024-04-23T16:33:40.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c2f0e7:::benchmark_data_smithi188_113957_object115853:head# 2024-04-23T16:33:40.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c2faa7:::benchmark_data_smithi188_113957_object148737:head# 2024-04-23T16:33:40.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c3223d:::benchmark_data_smithi188_113957_object104588:head# 2024-04-23T16:33:40.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c365d7:::benchmark_data_smithi188_113957_object68338:head# 2024-04-23T16:33:40.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c36d43:::benchmark_data_smithi188_113957_object113769:head# 2024-04-23T16:33:40.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c3908e:::benchmark_data_smithi188_113957_object40362:head# 2024-04-23T16:33:40.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c412f0:::benchmark_data_smithi188_113957_object61360:head# 2024-04-23T16:33:40.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c44b90:::benchmark_data_smithi188_113957_object118984:head# 2024-04-23T16:33:40.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c658b1:::benchmark_data_smithi188_113957_object62165:head# 2024-04-23T16:33:40.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c66488:::benchmark_data_smithi188_113957_object75915:head# 2024-04-23T16:33:40.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c79b0e:::benchmark_data_smithi188_113957_object119853:head# 2024-04-23T16:33:40.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58c84118:::benchmark_data_smithi188_113957_object108494:head# 2024-04-23T16:33:40.167 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ca0280:::benchmark_data_smithi188_113957_object83135:head# 2024-04-23T16:33:40.167 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ca5bc7:::benchmark_data_smithi188_113957_object92350:head# 2024-04-23T16:33:40.167 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ca9945:::benchmark_data_smithi188_113957_object131038:head# 2024-04-23T16:33:40.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58cb1b43:::benchmark_data_smithi188_113957_object137662:head# 2024-04-23T16:33:40.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58cb240f:::benchmark_data_smithi188_113957_object21564:head# 2024-04-23T16:33:40.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58cb6fe0:::benchmark_data_smithi188_113957_object103428:head# 2024-04-23T16:33:40.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58cbafff:::benchmark_data_smithi188_113957_object139198:head# 2024-04-23T16:33:40.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58cbed69:::benchmark_data_smithi188_113957_object128978:head# 2024-04-23T16:33:40.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58cc8a2f:::benchmark_data_smithi188_113957_object107821:head# 2024-04-23T16:33:40.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58cd025f:::benchmark_data_smithi188_113957_object119283:head# 2024-04-23T16:33:40.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58cd26a4:::benchmark_data_smithi188_113957_object7150:head# 2024-04-23T16:33:40.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58cd34ca:::benchmark_data_smithi188_113957_object114792:head# 2024-04-23T16:33:40.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ce8d25:::benchmark_data_smithi188_113957_object94704:head# 2024-04-23T16:33:40.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58cfd972:::benchmark_data_smithi188_113957_object122265:head# 2024-04-23T16:33:40.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58cff73f:::benchmark_data_smithi188_113957_object34534:head# 2024-04-23T16:33:40.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d0962f:::benchmark_data_smithi188_113957_object139612:head# 2024-04-23T16:33:40.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d0fd4c:::benchmark_data_smithi188_113957_object70682:head# 2024-04-23T16:33:40.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d18a7c:::benchmark_data_smithi188_113957_object89492:head# 2024-04-23T16:33:40.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d24b0d:::benchmark_data_smithi188_113957_object49062:head# 2024-04-23T16:33:40.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d25d46:::benchmark_data_smithi188_113957_object50577:head# 2024-04-23T16:33:40.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d2662b:::benchmark_data_smithi188_113957_object99606:head# 2024-04-23T16:33:40.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d29687:::benchmark_data_smithi188_113957_object22420:head# 2024-04-23T16:33:40.176 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d2a423:::benchmark_data_smithi188_113957_object57476:head# 2024-04-23T16:33:40.176 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:33:40.177 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d2fe49:::benchmark_data_smithi188_113957_object59097:head# 2024-04-23T16:33:40.177 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d3155b:::benchmark_data_smithi188_113957_object77940:head# 2024-04-23T16:33:40.177 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d3325c:::benchmark_data_smithi188_113957_object145274:head# 2024-04-23T16:33:40.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d3538d:::benchmark_data_smithi188_113957_object35970:head# 2024-04-23T16:33:40.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d37894:::benchmark_data_smithi188_113957_object79678:head# 2024-04-23T16:33:40.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d47526:::benchmark_data_smithi188_113957_object4181:head# 2024-04-23T16:33:40.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d47c2f:::benchmark_data_smithi188_113957_object109978:head# 2024-04-23T16:33:40.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d4c18b:::benchmark_data_smithi188_113957_object74651:head# 2024-04-23T16:33:40.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d540bc:::benchmark_data_smithi188_113957_object126985:head# 2024-04-23T16:33:40.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d54ec5:::benchmark_data_smithi188_113957_object147983:head# 2024-04-23T16:33:40.181 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d5aadd:::benchmark_data_smithi188_113957_object125966:head# 2024-04-23T16:33:40.181 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d5d5b3:::benchmark_data_smithi188_113957_object36695:head# 2024-04-23T16:33:40.181 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d5ed19:::benchmark_data_smithi188_113957_object84285:head# 2024-04-23T16:33:40.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d60a35:::benchmark_data_smithi188_113957_object23160:head# 2024-04-23T16:33:40.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d63ee1:::benchmark_data_smithi188_113957_object116271:head# 2024-04-23T16:33:40.183 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d69f1a:::benchmark_data_smithi188_113957_object70772:head# 2024-04-23T16:33:40.183 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d75719:::benchmark_data_smithi188_113957_object146534:head# 2024-04-23T16:33:40.183 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58d7873c:::benchmark_data_smithi188_113957_object11541:head# 2024-04-23T16:33:40.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58da82cc:::benchmark_data_smithi188_113957_object122586:head# 2024-04-23T16:33:40.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58da8ee6:::benchmark_data_smithi188_113957_object146297:head# 2024-04-23T16:33:40.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58daa4c9:::benchmark_data_smithi188_113957_object41110:head# 2024-04-23T16:33:40.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58db862e:::benchmark_data_smithi188_113957_object57928:head# 2024-04-23T16:33:40.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58db9dcb:::benchmark_data_smithi188_113957_object88292:head# 2024-04-23T16:33:40.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58dbd227:::benchmark_data_smithi188_113957_object127916:head# 2024-04-23T16:33:40.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58dc161b:::benchmark_data_smithi188_113957_object130852:head# 2024-04-23T16:33:40.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58dc2c66:::benchmark_data_smithi188_113957_object135347:head# 2024-04-23T16:33:40.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58dc8e60:::benchmark_data_smithi188_113957_object76757:head# 2024-04-23T16:33:40.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58dcb77f:::benchmark_data_smithi188_113957_object124421:head# 2024-04-23T16:33:40.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58dcecac:::benchmark_data_smithi188_113957_object61072:head# 2024-04-23T16:33:40.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58de73c3:::benchmark_data_smithi188_113957_object14634:head# 2024-04-23T16:33:40.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58e07521:::benchmark_data_smithi188_113957_object39653:head# 2024-04-23T16:33:40.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58e0c25d:::benchmark_data_smithi188_113957_object2062:head# 2024-04-23T16:33:40.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58e29840:::benchmark_data_smithi188_113957_object147649:head# 2024-04-23T16:33:40.192 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58e2ed95:::benchmark_data_smithi188_113957_object50604:head# 2024-04-23T16:33:40.192 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58e307fa:::benchmark_data_smithi188_113957_object95279:head# 2024-04-23T16:33:40.192 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58e5430a:::benchmark_data_smithi188_113957_object45951:head# 2024-04-23T16:33:40.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58e62ffb:::benchmark_data_smithi188_113957_object52926:head# 2024-04-23T16:33:40.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58e645ae:::benchmark_data_smithi188_113957_object109094:head# 2024-04-23T16:33:40.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58e71551:::benchmark_data_smithi188_113957_object77835:head# 2024-04-23T16:33:40.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58e79b35:::benchmark_data_smithi188_113957_object130150:head# 2024-04-23T16:33:40.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58e88b54:::benchmark_data_smithi188_113957_object66661:head# 2024-04-23T16:33:40.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58e89d8c:::benchmark_data_smithi188_113957_object42353:head# 2024-04-23T16:33:40.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58e924c4:::benchmark_data_smithi188_113957_object13894:head# 2024-04-23T16:33:40.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58e9735e:::benchmark_data_smithi188_113957_object17343:head# 2024-04-23T16:33:40.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58eb0760:::benchmark_data_smithi188_113957_object78004:head# 2024-04-23T16:33:40.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58eb151d:::benchmark_data_smithi188_113957_object143770:head# 2024-04-23T16:33:40.197 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58eb8302:::benchmark_data_smithi188_113957_object42664:head# 2024-04-23T16:33:40.197 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ec46ec:::benchmark_data_smithi188_113957_object93757:head# 2024-04-23T16:33:40.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ec885d:::benchmark_data_smithi188_113957_object50186:head# 2024-04-23T16:33:40.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ecc7dc:::benchmark_data_smithi188_113957_object83961:head# 2024-04-23T16:33:40.199 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ece08d:::benchmark_data_smithi188_113957_object63641:head# 2024-04-23T16:33:40.199 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ed32cf:::benchmark_data_smithi188_113957_object88810:head# 2024-04-23T16:33:40.199 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ed7b08:::benchmark_data_smithi188_113957_object18848:head# 2024-04-23T16:33:40.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ed9683:::benchmark_data_smithi188_113957_object51058:head# 2024-04-23T16:33:40.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58edcd64:::benchmark_data_smithi188_113957_object34693:head# 2024-04-23T16:33:40.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ee5d4f:::benchmark_data_smithi188_113957_object83131:head# 2024-04-23T16:33:40.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ef5534:::benchmark_data_smithi188_113957_object110608:head# 2024-04-23T16:33:40.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f07019:::benchmark_data_smithi188_113957_object147465:head# 2024-04-23T16:33:40.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f09e71:::benchmark_data_smithi188_113957_object19873:head# 2024-04-23T16:33:40.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f0b78f:::benchmark_data_smithi188_113957_object146616:head# 2024-04-23T16:33:40.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f169b3:::benchmark_data_smithi188_113957_object16309:head# 2024-04-23T16:33:40.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f19a07:::benchmark_data_smithi188_113957_object150051:head# 2024-04-23T16:33:40.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f1b514:::benchmark_data_smithi188_113957_object139805:head# 2024-04-23T16:33:40.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f22309:::benchmark_data_smithi188_113957_object139750:head# 2024-04-23T16:33:40.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f2ac4f:::benchmark_data_smithi188_113957_object23992:head# 2024-04-23T16:33:40.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f2cc84:::benchmark_data_smithi188_113957_object107080:head# 2024-04-23T16:33:40.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f2d5d9:::benchmark_data_smithi188_113957_object4907:head# 2024-04-23T16:33:40.206 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f3478f:::benchmark_data_smithi188_113957_object71659:head# 2024-04-23T16:33:40.206 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f3859e:::benchmark_data_smithi188_113957_object92065:head# 2024-04-23T16:33:40.206 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f3944f:::benchmark_data_smithi188_113957_object18140:head# 2024-04-23T16:33:40.207 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f40c41:::benchmark_data_smithi188_113957_object86225:head# 2024-04-23T16:33:40.207 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f4979e:::benchmark_data_smithi188_113957_object29800:head# 2024-04-23T16:33:40.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f4b999:::benchmark_data_smithi188_113957_object68257:head# 2024-04-23T16:33:40.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f5529e:::benchmark_data_smithi188_113957_object18485:head# 2024-04-23T16:33:40.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f5a45d:::benchmark_data_smithi188_113957_object38528:head# 2024-04-23T16:33:40.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f628df:::benchmark_data_smithi188_113957_object75728:head# 2024-04-23T16:33:40.209 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:33:40.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f6e8e1:::benchmark_data_smithi188_113957_object93644:head# 2024-04-23T16:33:40.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f72982:::benchmark_data_smithi188_113957_object49642:head# 2024-04-23T16:33:40.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f76bef:::benchmark_data_smithi188_113957_object39971:head# 2024-04-23T16:33:40.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f7e472:::benchmark_data_smithi188_113957_object147633:head# 2024-04-23T16:33:40.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f80e0f:::benchmark_data_smithi188_113957_object110123:head# 2024-04-23T16:33:40.212 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58f9b7b9:::benchmark_data_smithi188_113957_object126969:head# 2024-04-23T16:33:40.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:40.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:33:40.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58fa1bde:::benchmark_data_smithi188_113957_object7335:head# 2024-04-23T16:33:40.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58fa7b3c:::benchmark_data_smithi188_113957_object18024:head# 2024-04-23T16:33:40.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58fa8ea8:::benchmark_data_smithi188_113957_object137633:head# 2024-04-23T16:33:40.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58fa9304:::benchmark_data_smithi188_113957_object33718:head# 2024-04-23T16:33:40.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58faaf2f:::benchmark_data_smithi188_113957_object107254:head# 2024-04-23T16:33:40.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58fb0d7e:::benchmark_data_smithi188_113957_object97123:head# 2024-04-23T16:33:40.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58fb3217:::benchmark_data_smithi188_113957_object104260:head# 2024-04-23T16:33:40.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58fb6a86:::benchmark_data_smithi188_113957_object24708:head# 2024-04-23T16:33:40.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58fb72d7:::benchmark_data_smithi188_113957_object46512:head# 2024-04-23T16:33:40.216 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58fc368a:::benchmark_data_smithi188_113957_object97530:head# 2024-04-23T16:33:40.216 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58fc5281:::benchmark_data_smithi188_113957_object94560:head# 2024-04-23T16:33:40.216 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58fcb521:::benchmark_data_smithi188_113957_object9246:head# 2024-04-23T16:33:40.217 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58fcf838:::benchmark_data_smithi188_113957_object114747:head# 2024-04-23T16:33:40.217 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58fdd95b:::benchmark_data_smithi188_113957_object10251:head# 2024-04-23T16:33:40.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58fdefa4:::benchmark_data_smithi188_113957_object80197:head# 2024-04-23T16:33:40.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58fe956f:::benchmark_data_smithi188_113957_object148754:head# 2024-04-23T16:33:40.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ff04c2:::benchmark_data_smithi188_113957_object14819:head# 2024-04-23T16:33:40.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ff3722:::benchmark_data_smithi188_113957_object116612:head# 2024-04-23T16:33:40.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ff3df8:::benchmark_data_smithi188_113957_object140521:head# 2024-04-23T16:33:40.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ffab92:::benchmark_data_smithi188_113957_object152325:head# 2024-04-23T16:33:40.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:58ffe3e2:::benchmark_data_smithi188_113957_object50565:head# 2024-04-23T16:33:40.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5900f981:::benchmark_data_smithi188_113957_object63292:head# 2024-04-23T16:33:40.221 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5901355e:::benchmark_data_smithi188_113957_object145043:head# 2024-04-23T16:33:40.221 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59029e6b:::benchmark_data_smithi188_113957_object47663:head# 2024-04-23T16:33:40.222 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59048ae9:::benchmark_data_smithi188_113957_object99000:head# 2024-04-23T16:33:40.222 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59057390:::benchmark_data_smithi188_113957_object86286:head# 2024-04-23T16:33:40.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590606cb:::benchmark_data_smithi188_113957_object109561:head# 2024-04-23T16:33:40.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590634b5:::benchmark_data_smithi188_113957_object95843:head# 2024-04-23T16:33:40.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590774a9:::benchmark_data_smithi188_113957_object86738:head# 2024-04-23T16:33:40.224 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590783be:::benchmark_data_smithi188_113957_object144513:head# 2024-04-23T16:33:40.224 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5907fea1:::benchmark_data_smithi188_113957_object74415:head# 2024-04-23T16:33:40.225 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5908b495:::benchmark_data_smithi188_113957_object151029:head# 2024-04-23T16:33:40.225 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5908bf4c:::benchmark_data_smithi188_113957_object7295:head# 2024-04-23T16:33:40.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5908ee9f:::benchmark_data_smithi188_113957_object75037:head# 2024-04-23T16:33:40.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59094935:::benchmark_data_smithi188_113957_object136701:head# 2024-04-23T16:33:40.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590a8d13:::benchmark_data_smithi188_113957_object43421:head# 2024-04-23T16:33:40.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590afeea:::benchmark_data_smithi188_113957_object97687:head# 2024-04-23T16:33:40.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590b02b2:::benchmark_data_smithi188_113957_object99361:head# 2024-04-23T16:33:40.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590b2b68:::benchmark_data_smithi188_113957_object143721:head# 2024-04-23T16:33:40.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590c0924:::benchmark_data_smithi188_113957_object120164:head# 2024-04-23T16:33:40.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590c290c:::benchmark_data_smithi188_113957_object131337:head# 2024-04-23T16:33:40.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590c4a3f:::benchmark_data_smithi188_113957_object105994:head# 2024-04-23T16:33:40.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590c8144:::benchmark_data_smithi188_113957_object76566:head# 2024-04-23T16:33:40.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590c9748:::benchmark_data_smithi188_113957_object97867:head# 2024-04-23T16:33:40.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590d1222:::benchmark_data_smithi188_113957_object101226:head# 2024-04-23T16:33:40.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590d3a26:::benchmark_data_smithi188_113957_object36523:head# 2024-04-23T16:33:40.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590d3a3a:::benchmark_data_smithi188_113957_object115827:head# 2024-04-23T16:33:40.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590d5e07:::benchmark_data_smithi188_113957_object19318:head# 2024-04-23T16:33:40.234 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590e1c25:::benchmark_data_smithi188_113957_object50729:head# 2024-04-23T16:33:40.234 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590ed616:::benchmark_data_smithi188_113957_object71120:head# 2024-04-23T16:33:40.235 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590f31f2:::benchmark_data_smithi188_113957_object139133:head# 2024-04-23T16:33:40.235 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590f7deb:::benchmark_data_smithi188_113957_object29636:head# 2024-04-23T16:33:40.235 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:590fd1d8:::benchmark_data_smithi188_113957_object95410:head# 2024-04-23T16:33:40.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59113d35:::benchmark_data_smithi188_113957_object75254:head# 2024-04-23T16:33:40.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59116c67:::benchmark_data_smithi188_113957_object38233:head# 2024-04-23T16:33:40.237 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59127576:::benchmark_data_smithi188_113957_object29194:head# 2024-04-23T16:33:40.237 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59134349:::benchmark_data_smithi188_113957_object146222:head# 2024-04-23T16:33:40.237 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5913f746:::benchmark_data_smithi188_113957_object109016:head# 2024-04-23T16:33:40.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59142196:::benchmark_data_smithi188_113957_object20889:head# 2024-04-23T16:33:40.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59143a49:::benchmark_data_smithi188_113957_object20668:head# 2024-04-23T16:33:40.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5914511d:::benchmark_data_smithi188_113957_object41786:head# 2024-04-23T16:33:40.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59147613:::benchmark_data_smithi188_113957_object125388:head# 2024-04-23T16:33:40.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5914eee6:::benchmark_data_smithi188_113957_object106107:head# 2024-04-23T16:33:40.240 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5914fcf0:::benchmark_data_smithi188_113957_object139965:head# 2024-04-23T16:33:40.240 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:591510c2:::benchmark_data_smithi188_113957_object183:head# 2024-04-23T16:33:40.241 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5915bcae:::benchmark_data_smithi188_113957_object42403:head# 2024-04-23T16:33:40.241 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5915f2ee:::benchmark_data_smithi188_113957_object21997:head# 2024-04-23T16:33:40.241 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59167855:::benchmark_data_smithi188_113957_object132991:head# 2024-04-23T16:33:40.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59173af0:::benchmark_data_smithi188_113957_object104125:head# 2024-04-23T16:33:40.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:591740e1:::benchmark_data_smithi188_113957_object84190:head# 2024-04-23T16:33:40.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5917a4b8:::benchmark_data_smithi188_113957_object25726:head# 2024-04-23T16:33:40.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59184134:::benchmark_data_smithi188_113957_object108626:head# 2024-04-23T16:33:40.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5919491a:::benchmark_data_smithi188_113957_object30077:head# 2024-04-23T16:33:40.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59195b37:::benchmark_data_smithi188_113957_object78014:head# 2024-04-23T16:33:40.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59199b96:::benchmark_data_smithi188_113957_object151653:head# 2024-04-23T16:33:40.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5919ae54:::benchmark_data_smithi188_113957_object16268:head# 2024-04-23T16:33:40.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:591a046d:::benchmark_data_smithi188_113957_object100665:head# 2024-04-23T16:33:40.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:591a2ce7:::benchmark_data_smithi188_113957_object54338:head# 2024-04-23T16:33:40.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:591aadfd:::benchmark_data_smithi188_113957_object12977:head# 2024-04-23T16:33:40.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:591af7ac:::benchmark_data_smithi188_113957_object57183:head# 2024-04-23T16:33:40.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:591b677d:::benchmark_data_smithi188_113957_object69725:head# 2024-04-23T16:33:40.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:591c3d50:::benchmark_data_smithi188_113957_object24137:head# 2024-04-23T16:33:40.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:591d6796:::benchmark_data_smithi188_113957_object143341:head# 2024-04-23T16:33:40.248 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:591d6e62:::benchmark_data_smithi188_113957_object31535:head# 2024-04-23T16:33:40.248 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:591dabb3:::benchmark_data_smithi188_113957_object8005:head# 2024-04-23T16:33:40.248 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:591dc8fc:::benchmark_data_smithi188_113957_object33407:head# 2024-04-23T16:33:40.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:591ea2be:::benchmark_data_smithi188_113957_object110887:head# 2024-04-23T16:33:40.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:591ea2cf:::benchmark_data_smithi188_113957_object14781:head# 2024-04-23T16:33:40.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:591f4b2d:::benchmark_data_smithi188_113957_object120594:head# 2024-04-23T16:33:40.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:591f7caf:::benchmark_data_smithi188_113957_object88844:head# 2024-04-23T16:33:40.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59206aa4:::benchmark_data_smithi188_113957_object133295:head# 2024-04-23T16:33:40.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5921d36e:::benchmark_data_smithi188_113957_object99041:head# 2024-04-23T16:33:40.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5921d979:::benchmark_data_smithi188_113957_object18366:head# 2024-04-23T16:33:40.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5921f033:::benchmark_data_smithi188_113957_object146392:head# 2024-04-23T16:33:40.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5922549b:::benchmark_data_smithi188_113957_object72819:head# 2024-04-23T16:33:40.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59225f5e:::benchmark_data_smithi188_113957_object48296:head# 2024-04-23T16:33:40.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5922f55b:::benchmark_data_smithi188_113957_object64828:head# 2024-04-23T16:33:40.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5923cafb:::benchmark_data_smithi188_113957_object52495:head# 2024-04-23T16:33:40.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592446cf:::benchmark_data_smithi188_113957_object86693:head# 2024-04-23T16:33:40.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59248537:::benchmark_data_smithi188_113957_object93085:head# 2024-04-23T16:33:40.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5924acb9:::benchmark_data_smithi188_113957_object90896:head# 2024-04-23T16:33:40.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592516b9:::benchmark_data_smithi188_113957_object41666:head# 2024-04-23T16:33:40.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5925234b:::benchmark_data_smithi188_113957_object12580:head# 2024-04-23T16:33:40.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59252f95:::benchmark_data_smithi188_113957_object137514:head# 2024-04-23T16:33:40.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59254260:::benchmark_data_smithi188_113957_object57618:head# 2024-04-23T16:33:40.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5925b53b:::benchmark_data_smithi188_113957_object31481:head# 2024-04-23T16:33:40.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5925be25:::benchmark_data_smithi188_113957_object148234:head# 2024-04-23T16:33:40.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5926a267:::benchmark_data_smithi188_113957_object46269:head# 2024-04-23T16:33:40.258 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5926a5eb:::benchmark_data_smithi188_113957_object33157:head# 2024-04-23T16:33:40.258 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592748cf:::benchmark_data_smithi188_113957_object19955:head# 2024-04-23T16:33:40.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59276468:::benchmark_data_smithi188_113957_object115897:head# 2024-04-23T16:33:40.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59277ce5:::benchmark_data_smithi188_113957_object64938:head# 2024-04-23T16:33:40.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5927af46:::benchmark_data_smithi188_113957_object20524:head# 2024-04-23T16:33:40.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59283fe2:::benchmark_data_smithi188_113957_object94347:head# 2024-04-23T16:33:40.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592853ec:::benchmark_data_smithi188_113957_object77711:head# 2024-04-23T16:33:40.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59286e09:::benchmark_data_smithi188_113957_object128799:head# 2024-04-23T16:33:40.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59294e0a:::benchmark_data_smithi188_113957_object95711:head# 2024-04-23T16:33:40.262 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592956da:::benchmark_data_smithi188_113957_object147247:head# 2024-04-23T16:33:40.262 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592a0eb5:::benchmark_data_smithi188_113957_object94373:head# 2024-04-23T16:33:40.262 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592a521d:::benchmark_data_smithi188_113957_object136240:head# 2024-04-23T16:33:40.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592aedb0:::benchmark_data_smithi188_113957_object120036:head# 2024-04-23T16:33:40.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592b5f69:::benchmark_data_smithi188_113957_object41664:head# 2024-04-23T16:33:40.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592ba768:::benchmark_data_smithi188_113957_object73714:head# 2024-04-23T16:33:40.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592bb830:::benchmark_data_smithi188_113957_object6926:head# 2024-04-23T16:33:40.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592bbffb:::benchmark_data_smithi188_113957_object62614:head# 2024-04-23T16:33:40.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592bcc04:::benchmark_data_smithi188_113957_object147930:head# 2024-04-23T16:33:40.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592bce13:::benchmark_data_smithi188_113957_object40565:head# 2024-04-23T16:33:40.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592c4d60:::benchmark_data_smithi188_113957_object141059:head# 2024-04-23T16:33:40.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592c6003:::benchmark_data_smithi188_113957_object26766:head# 2024-04-23T16:33:40.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592c67b5:::benchmark_data_smithi188_113957_object1504:head# 2024-04-23T16:33:40.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592cba15:::benchmark_data_smithi188_113957_object103234:head# 2024-04-23T16:33:40.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592cf82a:::benchmark_data_smithi188_113957_object77834:head# 2024-04-23T16:33:40.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592d0e36:::benchmark_data_smithi188_113957_object114255:head# 2024-04-23T16:33:40.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592e118b:::benchmark_data_smithi188_113957_object19260:head# 2024-04-23T16:33:40.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592ebe41:::benchmark_data_smithi188_113957_object149318:head# 2024-04-23T16:33:40.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592f63dc:::benchmark_data_smithi188_113957_object48917:head# 2024-04-23T16:33:40.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592f6f84:::benchmark_data_smithi188_113957_object42357:head# 2024-04-23T16:33:40.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:592fa333:::benchmark_data_smithi188_113957_object130766:head# 2024-04-23T16:33:40.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59302559:::benchmark_data_smithi188_113957_object75996:head# 2024-04-23T16:33:40.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5930948c:::benchmark_data_smithi188_113957_object12820:head# 2024-04-23T16:33:40.277 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5930a0ab:::benchmark_data_smithi188_113957_object49709:head# 2024-04-23T16:33:40.277 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5930cb69:::benchmark_data_smithi188_113957_object42605:head# 2024-04-23T16:33:40.277 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5930e144:::benchmark_data_smithi188_113957_object87653:head# 2024-04-23T16:33:40.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5930f79a:::benchmark_data_smithi188_113957_object48977:head# 2024-04-23T16:33:40.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5933f927:::benchmark_data_smithi188_113957_object74491:head# 2024-04-23T16:33:40.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59343c4c:::benchmark_data_smithi188_113957_object72019:head# 2024-04-23T16:33:40.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59357f72:::benchmark_data_smithi188_113957_object32690:head# 2024-04-23T16:33:40.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:593584b1:::benchmark_data_smithi188_113957_object122129:head# 2024-04-23T16:33:40.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5935860c:::benchmark_data_smithi188_113957_object128069:head# 2024-04-23T16:33:40.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:593595ad:::benchmark_data_smithi188_113957_object61470:head# 2024-04-23T16:33:40.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5935ceda:::benchmark_data_smithi188_113957_object150991:head# 2024-04-23T16:33:40.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59366b6e:::benchmark_data_smithi188_113957_object51762:head# 2024-04-23T16:33:40.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59369592:::benchmark_data_smithi188_113957_object143808:head# 2024-04-23T16:33:40.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5936efdb:::benchmark_data_smithi188_113957_object96841:head# 2024-04-23T16:33:40.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59370601:::benchmark_data_smithi188_113957_object78363:head# 2024-04-23T16:33:40.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59371aba:::benchmark_data_smithi188_113957_object97405:head# 2024-04-23T16:33:40.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5937680c:::benchmark_data_smithi188_113957_object61148:head# 2024-04-23T16:33:40.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5937adc0:::benchmark_data_smithi188_113957_object145197:head# 2024-04-23T16:33:40.287 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5937af2f:::benchmark_data_smithi188_113957_object94593:head# 2024-04-23T16:33:40.287 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5937b484:::benchmark_data_smithi188_113957_object6066:head# 2024-04-23T16:33:40.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5937cc02:::benchmark_data_smithi188_113957_object107215:head# 2024-04-23T16:33:40.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5937dd97:::benchmark_data_smithi188_113957_object112530:head# 2024-04-23T16:33:40.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59384fc2:::benchmark_data_smithi188_113957_object123663:head# 2024-04-23T16:33:40.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59388fc0:::benchmark_data_smithi188_113957_object132820:head# 2024-04-23T16:33:40.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5938fef6:::benchmark_data_smithi188_113957_object104551:head# 2024-04-23T16:33:40.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59399408:::benchmark_data_smithi188_113957_object151867:head# 2024-04-23T16:33:40.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:593a7810:::benchmark_data_smithi188_113957_object7214:head# 2024-04-23T16:33:40.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:593ae712:::benchmark_data_smithi188_113957_object100995:head# 2024-04-23T16:33:40.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:593b520d:::benchmark_data_smithi188_113957_object8933:head# 2024-04-23T16:33:40.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:593b5215:::benchmark_data_smithi188_113957_object144467:head# 2024-04-23T16:33:40.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:593b8453:::benchmark_data_smithi188_113957_object72331:head# 2024-04-23T16:33:40.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:593b987e:::benchmark_data_smithi188_113957_object129462:head# 2024-04-23T16:33:40.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:593bfd4c:::benchmark_data_smithi188_113957_object17152:head# 2024-04-23T16:33:40.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:593c96b9:::benchmark_data_smithi188_113957_object107271:head# 2024-04-23T16:33:40.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:593eb2ac:::benchmark_data_smithi188_113957_object76293:head# 2024-04-23T16:33:40.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:593eddb3:::benchmark_data_smithi188_113957_object26539:head# 2024-04-23T16:33:40.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:593efdcd:::benchmark_data_smithi188_113957_object38715:head# 2024-04-23T16:33:40.295 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:593f5f05:::benchmark_data_smithi188_113957_object4075:head# 2024-04-23T16:33:40.295 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:593ffd40:::benchmark_data_smithi188_113957_object73277:head# 2024-04-23T16:33:40.295 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59401941:::benchmark_data_smithi188_113957_object54381:head# 2024-04-23T16:33:40.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594040f3:::benchmark_data_smithi188_113957_object64122:head# 2024-04-23T16:33:40.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59411049:::benchmark_data_smithi188_113957_object112854:head# 2024-04-23T16:33:40.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594123d5:::benchmark_data_smithi188_113957_object36390:head# 2024-04-23T16:33:40.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59416665:::benchmark_data_smithi188_113957_object14919:head# 2024-04-23T16:33:40.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5942433b:::benchmark_data_smithi188_113957_object144749:head# 2024-04-23T16:33:40.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5942c1aa:::benchmark_data_smithi188_113957_object51627:head# 2024-04-23T16:33:40.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5943342b:::benchmark_data_smithi188_113957_object58117:head# 2024-04-23T16:33:40.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59435f99:::benchmark_data_smithi188_113957_object106395:head# 2024-04-23T16:33:40.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59437f27:::benchmark_data_smithi188_113957_object7353:head# 2024-04-23T16:33:40.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59438657:::benchmark_data_smithi188_113957_object99867:head# 2024-04-23T16:33:40.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5943b7ee:::benchmark_data_smithi188_113957_object72530:head# 2024-04-23T16:33:40.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59446154:::benchmark_data_smithi188_113957_object74015:head# 2024-04-23T16:33:40.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5944bf56:::benchmark_data_smithi188_113957_object81080:head# 2024-04-23T16:33:40.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5944c5ab:::benchmark_data_smithi188_113957_object118840:head# 2024-04-23T16:33:40.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59450fb2:::benchmark_data_smithi188_113957_object151492:head# 2024-04-23T16:33:40.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59453869:::benchmark_data_smithi188_113957_object123511:head# 2024-04-23T16:33:40.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59461db7:::benchmark_data_smithi188_113957_object117839:head# 2024-04-23T16:33:40.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59463daf:::benchmark_data_smithi188_113957_object92500:head# 2024-04-23T16:33:40.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594753d2:::benchmark_data_smithi188_113957_object140281:head# 2024-04-23T16:33:40.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5947afa7:::benchmark_data_smithi188_113957_object144552:head# 2024-04-23T16:33:40.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5947dbf5:::benchmark_data_smithi188_113957_object96148:head# 2024-04-23T16:33:40.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594872f1:::benchmark_data_smithi188_113957_object84363:head# 2024-04-23T16:33:40.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594880e1:::benchmark_data_smithi188_113957_object45103:head# 2024-04-23T16:33:40.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594adb72:::benchmark_data_smithi188_113957_object16091:head# 2024-04-23T16:33:40.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594b9b3b:::benchmark_data_smithi188_113957_object140730:head# 2024-04-23T16:33:40.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594c41da:::benchmark_data_smithi188_113957_object9373:head# 2024-04-23T16:33:40.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594ce330:::benchmark_data_smithi188_113957_object20130:head# 2024-04-23T16:33:40.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594d74f4:::benchmark_data_smithi188_113957_object104161:head# 2024-04-23T16:33:40.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594d9387:::benchmark_data_smithi188_113957_object43554:head# 2024-04-23T16:33:40.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594d9dde:::benchmark_data_smithi188_113957_object139955:head# 2024-04-23T16:33:40.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594dc1e7:::benchmark_data_smithi188_113957_object144385:head# 2024-04-23T16:33:40.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594dde5e:::benchmark_data_smithi188_113957_object30659:head# 2024-04-23T16:33:40.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594dfa48:::benchmark_data_smithi188_113957_object62346:head# 2024-04-23T16:33:40.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594e407b:::benchmark_data_smithi188_113957_object116470:head# 2024-04-23T16:33:40.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594e70b9:::benchmark_data_smithi188_113957_object118672:head# 2024-04-23T16:33:40.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594e7737:::benchmark_data_smithi188_113957_object122165:head# 2024-04-23T16:33:40.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594f3597:::benchmark_data_smithi188_113957_object77526:head# 2024-04-23T16:33:40.313 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:33:40.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594fa067:::benchmark_data_smithi188_113957_object82136:head# 2024-04-23T16:33:40.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:594fbbf5:::benchmark_data_smithi188_113957_object127751:head# 2024-04-23T16:33:40.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59501d48:::benchmark_data_smithi188_113957_object39758:head# 2024-04-23T16:33:40.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5950284e:::benchmark_data_smithi188_113957_object91339:head# 2024-04-23T16:33:40.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59503644:::benchmark_data_smithi188_113957_object6954:head# 2024-04-23T16:33:40.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59508517:::benchmark_data_smithi188_113957_object36706:head# 2024-04-23T16:33:40.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5950ac3a:::benchmark_data_smithi188_113957_object32186:head# 2024-04-23T16:33:40.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5950d5e1:::benchmark_data_smithi188_113957_object67423:head# 2024-04-23T16:33:40.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5950edce:::benchmark_data_smithi188_113957_object100768:head# 2024-04-23T16:33:40.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59515dc6:::benchmark_data_smithi188_113957_object95820:head# 2024-04-23T16:33:40.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5951b096:::benchmark_data_smithi188_113957_object122255:head# 2024-04-23T16:33:40.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5951b755:::benchmark_data_smithi188_113957_object120878:head# 2024-04-23T16:33:40.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5951b8fc:::benchmark_data_smithi188_113957_object14174:head# 2024-04-23T16:33:40.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5951c9ca:::benchmark_data_smithi188_113957_object68890:head# 2024-04-23T16:33:40.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595258f0:::benchmark_data_smithi188_113957_object84568:head# 2024-04-23T16:33:40.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59527dd1:::benchmark_data_smithi188_113957_object116046:head# 2024-04-23T16:33:40.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5953f45d:::benchmark_data_smithi188_113957_object132700:head# 2024-04-23T16:33:40.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595455a8:::benchmark_data_smithi188_113957_object40135:head# 2024-04-23T16:33:40.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5954a672:::benchmark_data_smithi188_113957_object123300:head# 2024-04-23T16:33:40.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5954f2ac:::benchmark_data_smithi188_113957_object74850:head# 2024-04-23T16:33:40.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5956ed2e:::benchmark_data_smithi188_113957_object136250:head# 2024-04-23T16:33:40.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59582854:::benchmark_data_smithi188_113957_object3839:head# 2024-04-23T16:33:40.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595863ad:::benchmark_data_smithi188_113957_object146411:head# 2024-04-23T16:33:40.326 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59596355:::benchmark_data_smithi188_113957_object91689:head# 2024-04-23T16:33:40.326 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5959d4d6:::benchmark_data_smithi188_113957_object89999:head# 2024-04-23T16:33:40.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595a0405:::benchmark_data_smithi188_113957_object134398:head# 2024-04-23T16:33:40.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595a26fa:::benchmark_data_smithi188_113957_object131428:head# 2024-04-23T16:33:40.328 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595a342f:::benchmark_data_smithi188_113957_object144901:head# 2024-04-23T16:33:40.328 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595a43f0:::benchmark_data_smithi188_113957_object127427:head# 2024-04-23T16:33:40.329 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595a5768:::benchmark_data_smithi188_113957_object140674:head# 2024-04-23T16:33:40.329 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595aef84:::benchmark_data_smithi188_113957_object6958:head# 2024-04-23T16:33:40.329 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595b0eef:::benchmark_data_smithi188_113957_object141334:head# 2024-04-23T16:33:40.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595b62f1:::benchmark_data_smithi188_113957_object49004:head# 2024-04-23T16:33:40.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595c0a97:::benchmark_data_smithi188_113957_object57307:head# 2024-04-23T16:33:40.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595ce1d2:::benchmark_data_smithi188_113957_object75931:head# 2024-04-23T16:33:40.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595dc850:::benchmark_data_smithi188_113957_object69188:head# 2024-04-23T16:33:40.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595dcac1:::benchmark_data_smithi188_113957_object97576:head# 2024-04-23T16:33:40.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595dfe36:::benchmark_data_smithi188_113957_object61382:head# 2024-04-23T16:33:40.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595ecbfb:::benchmark_data_smithi188_113957_object116968:head# 2024-04-23T16:33:40.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595f3ce3:::benchmark_data_smithi188_113957_object131799:head# 2024-04-23T16:33:40.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595fa8ea:::benchmark_data_smithi188_113957_object107933:head# 2024-04-23T16:33:40.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:595fd4df:::benchmark_data_smithi188_113957_object140043:head# 2024-04-23T16:33:40.334 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59604c9a:::benchmark_data_smithi188_113957_object92946:head# 2024-04-23T16:33:40.334 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5960950f:::benchmark_data_smithi188_113957_object65385:head# 2024-04-23T16:33:40.334 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5960a99b:::benchmark_data_smithi188_113957_object39328:head# 2024-04-23T16:33:40.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59612725:::benchmark_data_smithi188_113957_object118409:head# 2024-04-23T16:33:40.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5961338e:::benchmark_data_smithi188_113957_object37648:head# 2024-04-23T16:33:40.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59617819:::benchmark_data_smithi188_113957_object98035:head# 2024-04-23T16:33:40.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5961a519:::benchmark_data_smithi188_113957_object64436:head# 2024-04-23T16:33:40.338 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59620a97:::benchmark_data_smithi188_113957_object83906:head# 2024-04-23T16:33:40.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:596249b1:::benchmark_data_smithi188_113957_object86620:head# 2024-04-23T16:33:40.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59626480:::benchmark_data_smithi188_113957_object121503:head# 2024-04-23T16:33:40.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59638898:::benchmark_data_smithi188_113957_object98365:head# 2024-04-23T16:33:40.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5963c4f8:::benchmark_data_smithi188_113957_object119168:head# 2024-04-23T16:33:40.341 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:596423ee:::benchmark_data_smithi188_113957_object20145:head# 2024-04-23T16:33:40.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:40 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:33:40.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:40 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:33:40.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:40 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/25397802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:33:40.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:40 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 27626/296312 objects degraded (9.323%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:40.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59643c2b:::benchmark_data_smithi188_113957_object109740:head# 2024-04-23T16:33:40.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59652270:::benchmark_data_smithi188_113957_object121203:head# 2024-04-23T16:33:40.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59654b1e:::benchmark_data_smithi188_113957_object50732:head# 2024-04-23T16:33:40.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:596672b9:::benchmark_data_smithi188_113957_object31519:head# 2024-04-23T16:33:40.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5967737d:::benchmark_data_smithi188_113957_object72956:head# 2024-04-23T16:33:40.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59677be0:::benchmark_data_smithi188_113957_object18886:head# 2024-04-23T16:33:40.344 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:40.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59691134:::benchmark_data_smithi188_113957_object34621:head# 2024-04-23T16:33:40.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5969af85:::benchmark_data_smithi188_113957_object92681:head# 2024-04-23T16:33:40.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:596b8ba0:::benchmark_data_smithi188_113957_object42550:head# 2024-04-23T16:33:40.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:596bc705:::benchmark_data_smithi188_113957_object137452:head# 2024-04-23T16:33:40.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:596c2c88:::benchmark_data_smithi188_113957_object43846:head# 2024-04-23T16:33:40.346 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:596c4c55:::benchmark_data_smithi188_113957_object150171:head# 2024-04-23T16:33:40.346 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:596cbd5f:::benchmark_data_smithi188_113957_object68406:head# 2024-04-23T16:33:40.346 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:596cd488:::benchmark_data_smithi188_113957_object26669:head# 2024-04-23T16:33:40.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:596d3a1d:::benchmark_data_smithi188_113957_object143761:head# 2024-04-23T16:33:40.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:596d57c1:::benchmark_data_smithi188_113957_object90015:head# 2024-04-23T16:33:40.348 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:596d6433:::benchmark_data_smithi188_113957_object714:head# 2024-04-23T16:33:40.348 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:596d8692:::benchmark_data_smithi188_113957_object59172:head# 2024-04-23T16:33:40.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:596d96df:::benchmark_data_smithi188_113957_object35673:head# 2024-04-23T16:33:40.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:596ea340:::benchmark_data_smithi188_113957_object7071:head# 2024-04-23T16:33:40.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:596f170a:::benchmark_data_smithi188_113957_object11447:head# 2024-04-23T16:33:40.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59707a86:::benchmark_data_smithi188_113957_object76075:head# 2024-04-23T16:33:40.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59709ed8:::benchmark_data_smithi188_113957_object105851:head# 2024-04-23T16:33:40.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5970bac9:::benchmark_data_smithi188_113957_object152848:head# 2024-04-23T16:33:40.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59721aec:::benchmark_data_smithi188_113957_object125883:head# 2024-04-23T16:33:40.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59737f61:::benchmark_data_smithi188_113957_object91776:head# 2024-04-23T16:33:40.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5974e0d4:::benchmark_data_smithi188_113957_object106160:head# 2024-04-23T16:33:40.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59752139:::benchmark_data_smithi188_113957_object129778:head# 2024-04-23T16:33:40.353 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5975e107:::benchmark_data_smithi188_113957_object18617:head# 2024-04-23T16:33:40.353 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5976082a:::benchmark_data_smithi188_113957_object43774:head# 2024-04-23T16:33:40.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:597639f6:::benchmark_data_smithi188_113957_object144710:head# 2024-04-23T16:33:40.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59774f65:::benchmark_data_smithi188_113957_object66154:head# 2024-04-23T16:33:40.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59780a35:::benchmark_data_smithi188_113957_object11414:head# 2024-04-23T16:33:40.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59782210:::benchmark_data_smithi188_113957_object136004:head# 2024-04-23T16:33:40.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5979613a:::benchmark_data_smithi188_113957_object27171:head# 2024-04-23T16:33:40.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5979de20:::benchmark_data_smithi188_113957_object35321:head# 2024-04-23T16:33:40.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5979f1be:::benchmark_data_smithi188_113957_object94800:head# 2024-04-23T16:33:40.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:597a3450:::benchmark_data_smithi188_113957_object121407:head# 2024-04-23T16:33:40.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:597ab1c2:::benchmark_data_smithi188_113957_object116172:head# 2024-04-23T16:33:40.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:597accbf:::benchmark_data_smithi188_113957_object91911:head# 2024-04-23T16:33:40.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:597ae803:::benchmark_data_smithi188_113957_object152720:head# 2024-04-23T16:33:40.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:597b9e64:::benchmark_data_smithi188_113957_object117563:head# 2024-04-23T16:33:40.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:597c9f2d:::benchmark_data_smithi188_113957_object123520:head# 2024-04-23T16:33:40.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:597caea6:::benchmark_data_smithi188_113957_object72081:head# 2024-04-23T16:33:40.363 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:597d551f:::benchmark_data_smithi188_113957_object110790:head# 2024-04-23T16:33:40.363 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:597dd751:::benchmark_data_smithi188_113957_object1837:head# 2024-04-23T16:33:40.363 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:597e785e:::benchmark_data_smithi188_113957_object107883:head# 2024-04-23T16:33:40.363 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:597f90ff:::benchmark_data_smithi188_113957_object54911:head# 2024-04-23T16:33:40.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:597fb3f2:::benchmark_data_smithi188_113957_object95060:head# 2024-04-23T16:33:40.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:597fb511:::benchmark_data_smithi188_113957_object75712:head# 2024-04-23T16:33:40.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:597fd66b:::benchmark_data_smithi188_113957_object119491:head# 2024-04-23T16:33:40.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59800f00:::benchmark_data_smithi188_113957_object37422:head# 2024-04-23T16:33:40.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5980c5d5:::benchmark_data_smithi188_113957_object1606:head# 2024-04-23T16:33:40.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5981144c:::benchmark_data_smithi188_113957_object74494:head# 2024-04-23T16:33:40.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59816f86:::benchmark_data_smithi188_113957_object152862:head# 2024-04-23T16:33:40.366 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598293ba:::benchmark_data_smithi188_113957_object49209:head# 2024-04-23T16:33:40.366 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5982b706:::benchmark_data_smithi188_113957_object89276:head# 2024-04-23T16:33:40.367 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5982bbe8:::benchmark_data_smithi188_113957_object88837:head# 2024-04-23T16:33:40.367 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5982d6fa:::benchmark_data_smithi188_113957_object70274:head# 2024-04-23T16:33:40.367 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59836abe:::benchmark_data_smithi188_113957_object92914:head# 2024-04-23T16:33:40.368 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:33:40.368 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5983e598:::benchmark_data_smithi188_113957_object106973:head# 2024-04-23T16:33:40.368 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5984900c:::benchmark_data_smithi188_113957_object98960:head# 2024-04-23T16:33:40.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59851ef1:::benchmark_data_smithi188_113957_object130183:head# 2024-04-23T16:33:40.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59856d9b:::benchmark_data_smithi188_113957_object143136:head# 2024-04-23T16:33:40.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598692cb:::benchmark_data_smithi188_113957_object88408:head# 2024-04-23T16:33:40.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5986f08e:::benchmark_data_smithi188_113957_object18669:head# 2024-04-23T16:33:40.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59871447:::benchmark_data_smithi188_113957_object91539:head# 2024-04-23T16:33:40.371 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5987b47b:::benchmark_data_smithi188_113957_object139446:head# 2024-04-23T16:33:40.371 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59887a86:::benchmark_data_smithi188_113957_object35003:head# 2024-04-23T16:33:40.372 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598892b2:::benchmark_data_smithi188_113957_object56905:head# 2024-04-23T16:33:40.372 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5988dd69:::benchmark_data_smithi188_113957_object13881:head# 2024-04-23T16:33:40.373 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5988f265:::benchmark_data_smithi188_113957_object140001:head# 2024-04-23T16:33:40.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:40.373 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:33:40.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59893b51:::benchmark_data_smithi188_113957_object83459:head# 2024-04-23T16:33:40.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5989b845:::benchmark_data_smithi188_113957_object114857:head# 2024-04-23T16:33:40.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598a6333:::benchmark_data_smithi188_113957_object91064:head# 2024-04-23T16:33:40.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:40 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:33:40.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:40 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:33:40.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:40 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/25397802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:33:40.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:40 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 27626/296312 objects degraded (9.323%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:40.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:40.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598a6431:::benchmark_data_smithi188_113957_object152433:head# 2024-04-23T16:33:40.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598ac039:::benchmark_data_smithi188_113957_object78403:head# 2024-04-23T16:33:40.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598ad413:::benchmark_data_smithi188_113957_object15980:head# 2024-04-23T16:33:40.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598b4d69:::benchmark_data_smithi188_113957_object34728:head# 2024-04-23T16:33:40.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598b5235:::benchmark_data_smithi188_113957_object136671:head# 2024-04-23T16:33:40.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598b854b:::benchmark_data_smithi188_113957_object46210:head# 2024-04-23T16:33:40.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598bead3:::benchmark_data_smithi188_113957_object74489:head# 2024-04-23T16:33:40.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598c0c86:::benchmark_data_smithi188_113957_object70710:head# 2024-04-23T16:33:40.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598c1632:::benchmark_data_smithi188_113957_object89846:head# 2024-04-23T16:33:40.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598c2a9a:::benchmark_data_smithi188_113957_object126884:head# 2024-04-23T16:33:40.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598c3b93:::benchmark_data_smithi188_113957_object62032:head# 2024-04-23T16:33:40.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598ccd51:::benchmark_data_smithi188_113957_object47134:head# 2024-04-23T16:33:40.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598e39a0:::benchmark_data_smithi188_113957_object113591:head# 2024-04-23T16:33:40.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598e5c76:::benchmark_data_smithi188_113957_object119840:head# 2024-04-23T16:33:40.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598e64a9:::benchmark_data_smithi188_113957_object14020:head# 2024-04-23T16:33:40.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598f53b5:::benchmark_data_smithi188_113957_object28078:head# 2024-04-23T16:33:40.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:598fe280:::benchmark_data_smithi188_113957_object119788:head# 2024-04-23T16:33:40.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5990d742:::benchmark_data_smithi188_113957_object139378:head# 2024-04-23T16:33:40.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59919a41:::benchmark_data_smithi188_113957_object134808:head# 2024-04-23T16:33:40.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5991f8ce:::benchmark_data_smithi188_113957_object62868:head# 2024-04-23T16:33:40.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:599268a0:::benchmark_data_smithi188_113957_object48780:head# 2024-04-23T16:33:40.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:599270c4:::benchmark_data_smithi188_113957_object132217:head# 2024-04-23T16:33:40.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59931032:::benchmark_data_smithi188_113957_object130982:head# 2024-04-23T16:33:40.387 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59933c34:::benchmark_data_smithi188_113957_object144222:head# 2024-04-23T16:33:40.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59933f0c:::benchmark_data_smithi188_113957_object87885:head# 2024-04-23T16:33:40.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:599347ce:::benchmark_data_smithi188_113957_object66551:head# 2024-04-23T16:33:40.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59946143:::benchmark_data_smithi188_113957_object49466:head# 2024-04-23T16:33:40.389 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5994e0b2:::benchmark_data_smithi188_113957_object102473:head# 2024-04-23T16:33:40.389 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59954d74:::benchmark_data_smithi188_113957_object98450:head# 2024-04-23T16:33:40.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59958664:::benchmark_data_smithi188_113957_object89203:head# 2024-04-23T16:33:40.391 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59958dcd:::benchmark_data_smithi188_113957_object18730:head# 2024-04-23T16:33:40.391 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5996f436:::benchmark_data_smithi188_113957_object146427:head# 2024-04-23T16:33:40.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59972335:::benchmark_data_smithi188_113957_object19929:head# 2024-04-23T16:33:40.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59978c2e:::benchmark_data_smithi188_113957_object66044:head# 2024-04-23T16:33:40.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5997c938:::benchmark_data_smithi188_113957_object91984:head# 2024-04-23T16:33:40.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59981de3:::benchmark_data_smithi188_113957_object143147:head# 2024-04-23T16:33:40.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5998d703:::benchmark_data_smithi188_113957_object86965:head# 2024-04-23T16:33:40.395 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59993bd5:::benchmark_data_smithi188_113957_object130653:head# 2024-04-23T16:33:40.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5999d1b3:::benchmark_data_smithi188_113957_object56257:head# 2024-04-23T16:33:40.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:599a4084:::benchmark_data_smithi188_113957_object3898:head# 2024-04-23T16:33:40.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:599a47d6:::benchmark_data_smithi188_113957_object49666:head# 2024-04-23T16:33:40.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:599a5856:::benchmark_data_smithi188_113957_object31174:head# 2024-04-23T16:33:40.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:599ad436:::benchmark_data_smithi188_113957_object102456:head# 2024-04-23T16:33:40.397 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:599b052f:::benchmark_data_smithi188_113957_object44521:head# 2024-04-23T16:33:40.397 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:599b6ee9:::benchmark_data_smithi188_113957_object101510:head# 2024-04-23T16:33:40.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:599c17fb:::benchmark_data_smithi188_113957_object70141:head# 2024-04-23T16:33:40.399 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:599c294d:::benchmark_data_smithi188_113957_object134020:head# 2024-04-23T16:33:40.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:599e40cc:::benchmark_data_smithi188_113957_object98502:head# 2024-04-23T16:33:40.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:599e9648:::benchmark_data_smithi188_113957_object94691:head# 2024-04-23T16:33:40.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:599eb36f:::benchmark_data_smithi188_113957_object136715:head# 2024-04-23T16:33:40.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:599fd775:::benchmark_data_smithi188_113957_object138955:head# 2024-04-23T16:33:40.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:599fe654:::benchmark_data_smithi188_113957_object66561:head# 2024-04-23T16:33:40.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a00d8e:::benchmark_data_smithi188_113957_object57572:head# 2024-04-23T16:33:40.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a072b6:::benchmark_data_smithi188_113957_object45877:head# 2024-04-23T16:33:40.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a08cd6:::benchmark_data_smithi188_113957_object150824:head# 2024-04-23T16:33:40.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a0e946:::benchmark_data_smithi188_113957_object52521:head# 2024-04-23T16:33:40.403 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a16d21:::benchmark_data_smithi188_113957_object81905:head# 2024-04-23T16:33:40.403 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a1dcdc:::benchmark_data_smithi188_113957_object52389:head# 2024-04-23T16:33:40.403 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a249ae:::benchmark_data_smithi188_113957_object22244:head# 2024-04-23T16:33:40.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a2558f:::benchmark_data_smithi188_113957_object30963:head# 2024-04-23T16:33:40.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a30b5d:::benchmark_data_smithi188_113957_object36063:head# 2024-04-23T16:33:40.405 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a32a7d:::benchmark_data_smithi188_113957_object45193:head# 2024-04-23T16:33:40.405 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a3fb51:::benchmark_data_smithi188_113957_object11091:head# 2024-04-23T16:33:40.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a410c3:::benchmark_data_smithi188_113957_object100111:head# 2024-04-23T16:33:40.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a42daa:::benchmark_data_smithi188_113957_object77314:head# 2024-04-23T16:33:40.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a54e0f:::benchmark_data_smithi188_113957_object152117:head# 2024-04-23T16:33:40.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a5b3bc:::benchmark_data_smithi188_113957_object98099:head# 2024-04-23T16:33:40.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a5f5fd:::benchmark_data_smithi188_113957_object77567:head# 2024-04-23T16:33:40.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a62438:::benchmark_data_smithi188_113957_object122854:head# 2024-04-23T16:33:40.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a65c9d:::benchmark_data_smithi188_113957_object78267:head# 2024-04-23T16:33:40.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a68246:::benchmark_data_smithi188_113957_object100788:head# 2024-04-23T16:33:40.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a6eb0e:::benchmark_data_smithi188_113957_object66508:head# 2024-04-23T16:33:40.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a6fd19:::benchmark_data_smithi188_113957_object82107:head# 2024-04-23T16:33:40.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a726d5:::benchmark_data_smithi188_113957_object13689:head# 2024-04-23T16:33:40.412 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a7df1f:::benchmark_data_smithi188_113957_object77106:head# 2024-04-23T16:33:40.412 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a7f463:::benchmark_data_smithi188_113957_object68237:head# 2024-04-23T16:33:40.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a82e19:::benchmark_data_smithi188_113957_object80167:head# 2024-04-23T16:33:40.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a8b158:::benchmark_data_smithi188_113957_object64146:head# 2024-04-23T16:33:40.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a8d1c3:::benchmark_data_smithi188_113957_object68541:head# 2024-04-23T16:33:40.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a98657:::benchmark_data_smithi188_113957_object76958:head# 2024-04-23T16:33:40.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59a9c2c9:::benchmark_data_smithi188_113957_object30784:head# 2024-04-23T16:33:40.414 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59aa1545:::benchmark_data_smithi188_113957_object62006:head# 2024-04-23T16:33:40.414 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59aa3554:::benchmark_data_smithi188_113957_object90882:head# 2024-04-23T16:33:40.414 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59aac73b:::benchmark_data_smithi188_113957_object56640:head# 2024-04-23T16:33:40.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ab1cf0:::benchmark_data_smithi188_113957_object100233:head# 2024-04-23T16:33:40.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ab6b42:::benchmark_data_smithi188_113957_object135710:head# 2024-04-23T16:33:40.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ab724d:::benchmark_data_smithi188_113957_object35065:head# 2024-04-23T16:33:40.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ac6137:::benchmark_data_smithi188_113957_object85189:head# 2024-04-23T16:33:40.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ace9ee:::benchmark_data_smithi188_113957_object83814:head# 2024-04-23T16:33:40.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59acefe8:::benchmark_data_smithi188_113957_object143676:head# 2024-04-23T16:33:40.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ad55e2:::benchmark_data_smithi188_113957_object105654:head# 2024-04-23T16:33:40.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59af5881:::benchmark_data_smithi188_113957_object72365:head# 2024-04-23T16:33:40.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59af788c:::benchmark_data_smithi188_113957_object143259:head# 2024-04-23T16:33:40.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59afc02f:::benchmark_data_smithi188_113957_object134449:head# 2024-04-23T16:33:40.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59affeb2:::benchmark_data_smithi188_113957_object123597:head# 2024-04-23T16:33:40.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b07c07:::benchmark_data_smithi188_113957_object122147:head# 2024-04-23T16:33:40.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b08009:::benchmark_data_smithi188_113957_object113978:head# 2024-04-23T16:33:40.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b0e3ae:::benchmark_data_smithi188_113957_object135021:head# 2024-04-23T16:33:40.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b1c541:::benchmark_data_smithi188_113957_object62181:head# 2024-04-23T16:33:40.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b1e30b:::benchmark_data_smithi188_113957_object26147:head# 2024-04-23T16:33:40.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b1e6ea:::benchmark_data_smithi188_113957_object73033:head# 2024-04-23T16:33:40.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b28696:::benchmark_data_smithi188_113957_object49048:head# 2024-04-23T16:33:40.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b2bc46:::benchmark_data_smithi188_113957_object145027:head# 2024-04-23T16:33:40.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b2bdb4:::benchmark_data_smithi188_113957_object13338:head# 2024-04-23T16:33:40.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b2f757:::benchmark_data_smithi188_113957_object9469:head# 2024-04-23T16:33:40.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b39746:::benchmark_data_smithi188_113957_object87106:head# 2024-04-23T16:33:40.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b451d3:::benchmark_data_smithi188_113957_object30706:head# 2024-04-23T16:33:40.426 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b4cd78:::benchmark_data_smithi188_113957_object78732:head# 2024-04-23T16:33:40.426 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b536be:::benchmark_data_smithi188_113957_object10838:head# 2024-04-23T16:33:40.426 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b59f95:::benchmark_data_smithi188_113957_object104124:head# 2024-04-23T16:33:40.426 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b6f91b:::benchmark_data_smithi188_113957_object75282:head# 2024-04-23T16:33:40.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b7141e:::benchmark_data_smithi188_113957_object150701:head# 2024-04-23T16:33:40.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b77f92:::benchmark_data_smithi188_113957_object152114:head# 2024-04-23T16:33:40.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b79bcb:::benchmark_data_smithi188_113957_object132723:head# 2024-04-23T16:33:40.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b84408:::benchmark_data_smithi188_113957_object75690:head# 2024-04-23T16:33:40.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b85e05:::benchmark_data_smithi188_113957_object56319:head# 2024-04-23T16:33:40.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b9a301:::benchmark_data_smithi188_113957_object44298:head# 2024-04-23T16:33:40.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b9b26e:::benchmark_data_smithi188_113957_object44474:head# 2024-04-23T16:33:40.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59b9f9ca:::benchmark_data_smithi188_113957_object88373:head# 2024-04-23T16:33:40.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ba88c0:::benchmark_data_smithi188_113957_object20392:head# 2024-04-23T16:33:40.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ba8dde:::benchmark_data_smithi188_113957_object4982:head# 2024-04-23T16:33:40.431 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59bb7cb5:::benchmark_data_smithi188_113957_object14788:head# 2024-04-23T16:33:40.431 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59bbdf28:::benchmark_data_smithi188_113957_object134343:head# 2024-04-23T16:33:40.432 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59bbe630:::benchmark_data_smithi188_113957_object56314:head# 2024-04-23T16:33:40.432 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59bbf6d1:::benchmark_data_smithi188_113957_object20293:head# 2024-04-23T16:33:40.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59bc3b36:::benchmark_data_smithi188_113957_object100255:head# 2024-04-23T16:33:40.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59bc89a6:::benchmark_data_smithi188_113957_object8876:head# 2024-04-23T16:33:40.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59bcd274:::benchmark_data_smithi188_113957_object131862:head# 2024-04-23T16:33:40.434 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59bcea2f:::benchmark_data_smithi188_113957_object51434:head# 2024-04-23T16:33:40.434 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59bd7baf:::benchmark_data_smithi188_113957_object22058:head# 2024-04-23T16:33:40.435 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59bddd6f:::benchmark_data_smithi188_113957_object35502:head# 2024-04-23T16:33:40.435 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59bdfbd9:::benchmark_data_smithi188_113957_object65475:head# 2024-04-23T16:33:40.435 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59be5fb9:::benchmark_data_smithi188_113957_object12657:head# 2024-04-23T16:33:40.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59bedb89:::benchmark_data_smithi188_113957_object97515:head# 2024-04-23T16:33:40.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59bf1d94:::benchmark_data_smithi188_113957_object84908:head# 2024-04-23T16:33:40.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59bfcabc:::benchmark_data_smithi188_113957_object10535:head# 2024-04-23T16:33:40.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c132cc:::benchmark_data_smithi188_113957_object74039:head# 2024-04-23T16:33:40.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c1601b:::benchmark_data_smithi188_113957_object142381:head# 2024-04-23T16:33:40.438 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c19e33:::benchmark_data_smithi188_113957_object130884:head# 2024-04-23T16:33:40.438 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c1c5fb:::benchmark_data_smithi188_113957_object123587:head# 2024-04-23T16:33:40.438 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c1f715:::benchmark_data_smithi188_113957_object108576:head# 2024-04-23T16:33:40.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c2229f:::benchmark_data_smithi188_113957_object11572:head# 2024-04-23T16:33:40.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c26838:::benchmark_data_smithi188_113957_object81434:head# 2024-04-23T16:33:40.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c2b4b9:::benchmark_data_smithi188_113957_object43493:head# 2024-04-23T16:33:40.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c32809:::benchmark_data_smithi188_113957_object23850:head# 2024-04-23T16:33:40.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c38db3:::benchmark_data_smithi188_113957_object10284:head# 2024-04-23T16:33:40.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c3d5b8:::benchmark_data_smithi188_113957_object6723:head# 2024-04-23T16:33:40.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c45c98:::benchmark_data_smithi188_113957_object28212:head# 2024-04-23T16:33:40.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c476c8:::benchmark_data_smithi188_113957_object77031:head# 2024-04-23T16:33:40.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c4c52e:::benchmark_data_smithi188_113957_object29476:head# 2024-04-23T16:33:40.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c54062:::benchmark_data_smithi188_113957_object127569:head# 2024-04-23T16:33:40.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c5878c:::benchmark_data_smithi188_113957_object55698:head# 2024-04-23T16:33:40.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c6014b:::benchmark_data_smithi188_113957_object7859:head# 2024-04-23T16:33:40.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c63861:::benchmark_data_smithi188_113957_object34298:head# 2024-04-23T16:33:40.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c734ba:::benchmark_data_smithi188_113957_object142508:head# 2024-04-23T16:33:40.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c761df:::benchmark_data_smithi188_113957_object126994:head# 2024-04-23T16:33:40.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c8cbd3:::benchmark_data_smithi188_113957_object17408:head# 2024-04-23T16:33:40.446 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c938cc:::benchmark_data_smithi188_113957_object88960:head# 2024-04-23T16:33:40.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c952ee:::benchmark_data_smithi188_113957_object130044:head# 2024-04-23T16:33:40.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59c99f4d:::benchmark_data_smithi188_113957_object89272:head# 2024-04-23T16:33:40.448 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ca0298:::benchmark_data_smithi188_113957_object57863:head# 2024-04-23T16:33:40.448 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ca3f8f:::benchmark_data_smithi188_113957_object114411:head# 2024-04-23T16:33:40.448 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ca400f:::benchmark_data_smithi188_113957_object25269:head# 2024-04-23T16:33:40.449 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59cabfd2:::benchmark_data_smithi188_113957_object17283:head# 2024-04-23T16:33:40.449 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59cb8897:::benchmark_data_smithi188_113957_object25205:head# 2024-04-23T16:33:40.449 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ccc050:::benchmark_data_smithi188_113957_object144495:head# 2024-04-23T16:33:40.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59cd0804:::benchmark_data_smithi188_113957_object101702:head# 2024-04-23T16:33:40.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59cd5b01:::benchmark_data_smithi188_113957_object20735:head# 2024-04-23T16:33:40.451 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59cdd16f:::benchmark_data_smithi188_113957_object142015:head# 2024-04-23T16:33:40.451 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ceac5c:::benchmark_data_smithi188_113957_object118282:head# 2024-04-23T16:33:40.452 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59cec7ce:::benchmark_data_smithi188_113957_object93944:head# 2024-04-23T16:33:40.452 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59cf2577:::benchmark_data_smithi188_113957_object40226:head# 2024-04-23T16:33:40.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59cf5f55:::benchmark_data_smithi188_113957_object4408:head# 2024-04-23T16:33:40.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59cf67e3:::benchmark_data_smithi188_113957_object3095:head# 2024-04-23T16:33:40.454 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59cfe45b:::benchmark_data_smithi188_113957_object47433:head# 2024-04-23T16:33:40.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59cfeedd:::benchmark_data_smithi188_113957_object82707:head# 2024-04-23T16:33:40.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59cff00e:::benchmark_data_smithi188_113957_object59326:head# 2024-04-23T16:33:40.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d011be:::benchmark_data_smithi188_113957_object55073:head# 2024-04-23T16:33:40.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d08a7f:::benchmark_data_smithi188_113957_object78773:head# 2024-04-23T16:33:40.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d16581:::benchmark_data_smithi188_113957_object97557:head# 2024-04-23T16:33:40.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d17606:::benchmark_data_smithi188_113957_object135765:head# 2024-04-23T16:33:40.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d22f2b:::benchmark_data_smithi188_113957_object128915:head# 2024-04-23T16:33:40.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d2cb84:::benchmark_data_smithi188_113957_object70701:head# 2024-04-23T16:33:40.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d30063:::benchmark_data_smithi188_113957_object105387:head# 2024-04-23T16:33:40.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d3130e:::benchmark_data_smithi188_113957_object72680:head# 2024-04-23T16:33:40.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d31ed7:::benchmark_data_smithi188_113957_object48854:head# 2024-04-23T16:33:40.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d37ad0:::benchmark_data_smithi188_113957_object97864:head# 2024-04-23T16:33:40.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d3fcee:::benchmark_data_smithi188_113957_object66813:head# 2024-04-23T16:33:40.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d40773:::benchmark_data_smithi188_113957_object121381:head# 2024-04-23T16:33:40.460 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d49277:::benchmark_data_smithi188_113957_object143871:head# 2024-04-23T16:33:40.460 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d5d585:::benchmark_data_smithi188_113957_object128323:head# 2024-04-23T16:33:40.460 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d6634a:::benchmark_data_smithi188_113957_object89376:head# 2024-04-23T16:33:40.461 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d6f4b4:::benchmark_data_smithi188_113957_object61698:head# 2024-04-23T16:33:40.461 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d706f7:::benchmark_data_smithi188_113957_object87161:head# 2024-04-23T16:33:40.462 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d711e9:::benchmark_data_smithi188_113957_object12898:head# 2024-04-23T16:33:40.462 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d862a7:::benchmark_data_smithi188_113957_object86666:head# 2024-04-23T16:33:40.462 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d9c9bc:::benchmark_data_smithi188_113957_object87926:head# 2024-04-23T16:33:40.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d9e32c:::benchmark_data_smithi188_113957_object141963:head# 2024-04-23T16:33:40.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59d9eff0:::benchmark_data_smithi188_113957_object128039:head# 2024-04-23T16:33:40.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59da0c7a:::benchmark_data_smithi188_113957_object68748:head# 2024-04-23T16:33:40.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59da2983:::benchmark_data_smithi188_113957_object131277:head# 2024-04-23T16:33:40.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59da535e:::benchmark_data_smithi188_113957_object37187:head# 2024-04-23T16:33:40.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59da8c55:::benchmark_data_smithi188_113957_object27835:head# 2024-04-23T16:33:40.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59dafaad:::benchmark_data_smithi188_113957_object21751:head# 2024-04-23T16:33:40.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59dc07d3:::benchmark_data_smithi188_113957_object100877:head# 2024-04-23T16:33:40.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59dc290d:::benchmark_data_smithi188_113957_object40973:head# 2024-04-23T16:33:40.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59dc78da:::benchmark_data_smithi188_113957_object107509:head# 2024-04-23T16:33:40.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59dd580f:::benchmark_data_smithi188_113957_object15687:head# 2024-04-23T16:33:40.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59dd5e7c:::benchmark_data_smithi188_113957_object52124:head# 2024-04-23T16:33:40.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59de28c8:::benchmark_data_smithi188_113957_object148039:head# 2024-04-23T16:33:40.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59deef39:::benchmark_data_smithi188_113957_object60296:head# 2024-04-23T16:33:40.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59dff661:::benchmark_data_smithi188_113957_object62829:head# 2024-04-23T16:33:40.469 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59e061f4:::benchmark_data_smithi188_113957_object113911:head# 2024-04-23T16:33:40.469 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59e0e2b0:::benchmark_data_smithi188_113957_object77395:head# 2024-04-23T16:33:40.470 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59e0e484:::benchmark_data_smithi188_113957_object76861:head# 2024-04-23T16:33:40.470 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59e20367:::benchmark_data_smithi188_113957_object143281:head# 2024-04-23T16:33:40.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59e22195:::benchmark_data_smithi188_113957_object20726:head# 2024-04-23T16:33:40.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59e23af4:::benchmark_data_smithi188_113957_object68336:head# 2024-04-23T16:33:40.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59e26b02:::benchmark_data_smithi188_113957_object74659:head# 2024-04-23T16:33:40.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59e26f6c:::benchmark_data_smithi188_113957_object20497:head# 2024-04-23T16:33:40.473 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59e38fd9:::benchmark_data_smithi188_113957_object29838:head# 2024-04-23T16:33:40.473 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59e4734c:::benchmark_data_smithi188_113957_object88053:head# 2024-04-23T16:33:40.474 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:40.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59e5c87a:::benchmark_data_smithi188_113957_object120936:head# 2024-04-23T16:33:40.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59e6999b:::benchmark_data_smithi188_113957_object57479:head# 2024-04-23T16:33:40.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59e6c2f0:::benchmark_data_smithi188_113957_object92903:head# 2024-04-23T16:33:40.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59e6f95e:::benchmark_data_smithi188_113957_object117202:head# 2024-04-23T16:33:40.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59e729d9:::benchmark_data_smithi188_113957_object147676:head# 2024-04-23T16:33:40.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59e7702c:::benchmark_data_smithi188_113957_object107342:head# 2024-04-23T16:33:40.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59e79f53:::benchmark_data_smithi188_113957_object20451:head# 2024-04-23T16:33:40.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59e8db00:::benchmark_data_smithi188_113957_object64517:head# 2024-04-23T16:33:40.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ea03b5:::benchmark_data_smithi188_113957_object91661:head# 2024-04-23T16:33:40.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ea91e5:::benchmark_data_smithi188_113957_object150307:head# 2024-04-23T16:33:40.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59eab948:::benchmark_data_smithi188_113957_object88871:head# 2024-04-23T16:33:40.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59eac620:::benchmark_data_smithi188_113957_object142531:head# 2024-04-23T16:33:40.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59eb1b1d:::benchmark_data_smithi188_113957_object13723:head# 2024-04-23T16:33:40.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59eb6fb9:::benchmark_data_smithi188_113957_object38262:head# 2024-04-23T16:33:40.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59eba14c:::benchmark_data_smithi188_113957_object54979:head# 2024-04-23T16:33:40.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ec211e:::benchmark_data_smithi188_113957_object318:head# 2024-04-23T16:33:40.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ed6579:::benchmark_data_smithi188_113957_object97744:head# 2024-04-23T16:33:40.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ee1a5b:::benchmark_data_smithi188_113957_object3054:head# 2024-04-23T16:33:40.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ee4347:::benchmark_data_smithi188_113957_object73657:head# 2024-04-23T16:33:40.481 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ee8768:::benchmark_data_smithi188_113957_object105618:head# 2024-04-23T16:33:40.481 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59eee949:::benchmark_data_smithi188_113957_object145512:head# 2024-04-23T16:33:40.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59f0027c:::benchmark_data_smithi188_113957_object70:head# 2024-04-23T16:33:40.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59f02947:::benchmark_data_smithi188_113957_object139389:head# 2024-04-23T16:33:40.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59f0e10b:::benchmark_data_smithi188_113957_object71084:head# 2024-04-23T16:33:40.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59f14cf2:::benchmark_data_smithi188_113957_object42908:head# 2024-04-23T16:33:40.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59f24b6f:::benchmark_data_smithi188_113957_object36681:head# 2024-04-23T16:33:40.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59f2798a:::benchmark_data_smithi188_113957_object59657:head# 2024-04-23T16:33:40.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59f2826d:::benchmark_data_smithi188_113957_object151709:head# 2024-04-23T16:33:40.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59f2f4f7:::benchmark_data_smithi188_113957_object123914:head# 2024-04-23T16:33:40.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59f510c4:::benchmark_data_smithi188_113957_object26027:head# 2024-04-23T16:33:40.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59f624b1:::benchmark_data_smithi188_113957_object43122:head# 2024-04-23T16:33:40.486 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59f6ed1a:::benchmark_data_smithi188_113957_object105226:head# 2024-04-23T16:33:40.486 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59f70ef2:::benchmark_data_smithi188_113957_object79181:head# 2024-04-23T16:33:40.486 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59f77778:::benchmark_data_smithi188_113957_object7152:head# 2024-04-23T16:33:40.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59f7fba8:::benchmark_data_smithi188_113957_object22370:head# 2024-04-23T16:33:40.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59f8b962:::benchmark_data_smithi188_113957_object12057:head# 2024-04-23T16:33:40.488 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59f9fe74:::benchmark_data_smithi188_113957_object120730:head# 2024-04-23T16:33:40.488 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59fa6044:::benchmark_data_smithi188_113957_object59727:head# 2024-04-23T16:33:40.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59fa63ec:::benchmark_data_smithi188_113957_object60877:head# 2024-04-23T16:33:40.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59fa7cd1:::benchmark_data_smithi188_113957_object129619:head# 2024-04-23T16:33:40.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59faac06:::benchmark_data_smithi188_113957_object98564:head# 2024-04-23T16:33:40.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59fab2ca:::benchmark_data_smithi188_113957_object127345:head# 2024-04-23T16:33:40.491 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59fab49e:::benchmark_data_smithi188_113957_object45720:head# 2024-04-23T16:33:40.492 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59fb0a64:::benchmark_data_smithi188_113957_object17250:head# 2024-04-23T16:33:40.492 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59fb1212:::benchmark_data_smithi188_113957_object146723:head# 2024-04-23T16:33:40.493 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59fb52d4:::benchmark_data_smithi188_113957_object150833:head# 2024-04-23T16:33:40.493 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59fbecd1:::benchmark_data_smithi188_113957_object27776:head# 2024-04-23T16:33:40.494 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59fc1c39:::benchmark_data_smithi188_113957_object27864:head# 2024-04-23T16:33:40.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59fc5306:::benchmark_data_smithi188_113957_object107631:head# 2024-04-23T16:33:40.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59fcd3f6:::benchmark_data_smithi188_113957_object132704:head# 2024-04-23T16:33:40.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59fce32c:::benchmark_data_smithi188_113957_object53243:head# 2024-04-23T16:33:40.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59fdd03d:::benchmark_data_smithi188_113957_object4006:head# 2024-04-23T16:33:40.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59fdd7d3:::benchmark_data_smithi188_113957_object21993:head# 2024-04-23T16:33:40.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59fde2bb:::benchmark_data_smithi188_113957_object111528:head# 2024-04-23T16:33:40.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59fe5147:::benchmark_data_smithi188_113957_object70396:head# 2024-04-23T16:33:40.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59feb2b3:::benchmark_data_smithi188_113957_object150952:head# 2024-04-23T16:33:40.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59fec33c:::benchmark_data_smithi188_113957_object85117:head# 2024-04-23T16:33:40.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ff47c2:::benchmark_data_smithi188_113957_object68165:head# 2024-04-23T16:33:40.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:59ffafd3:::benchmark_data_smithi188_113957_object143395:head# 2024-04-23T16:33:40.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a004446:::benchmark_data_smithi188_113957_object143177:head# 2024-04-23T16:33:40.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a014593:::benchmark_data_smithi188_113957_object49632:head# 2024-04-23T16:33:40.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a016f3f:::benchmark_data_smithi188_113957_object126528:head# 2024-04-23T16:33:40.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a01a6dd:::benchmark_data_smithi188_113957_object79864:head# 2024-04-23T16:33:40.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a03bacb:::benchmark_data_smithi188_113957_object131562:head# 2024-04-23T16:33:40.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a03d7be:::benchmark_data_smithi188_113957_object72113:head# 2024-04-23T16:33:40.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a048556:::benchmark_data_smithi188_113957_object78411:head# 2024-04-23T16:33:40.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a04bc6c:::benchmark_data_smithi188_113957_object101334:head# 2024-04-23T16:33:40.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a0508a0:::benchmark_data_smithi188_113957_object98658:head# 2024-04-23T16:33:40.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a050e30:::benchmark_data_smithi188_113957_object78341:head# 2024-04-23T16:33:40.504 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a05c68a:::benchmark_data_smithi188_113957_object141867:head# 2024-04-23T16:33:40.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a05eb8e:::benchmark_data_smithi188_113957_object82656:head# 2024-04-23T16:33:40.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a05f745:::benchmark_data_smithi188_113957_object13793:head# 2024-04-23T16:33:40.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a062207:::benchmark_data_smithi188_113957_object115298:head# 2024-04-23T16:33:40.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a068451:::benchmark_data_smithi188_113957_object96857:head# 2024-04-23T16:33:40.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a06aeb2:::benchmark_data_smithi188_113957_object27402:head# 2024-04-23T16:33:40.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a07668d:::benchmark_data_smithi188_113957_object13685:head# 2024-04-23T16:33:40.507 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a07d2f7:::benchmark_data_smithi188_113957_object54140:head# 2024-04-23T16:33:40.508 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a08feef:::benchmark_data_smithi188_113957_object31608:head# 2024-04-23T16:33:40.508 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a0a5363:::benchmark_data_smithi188_113957_object51142:head# 2024-04-23T16:33:40.509 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a0a7689:::benchmark_data_smithi188_113957_object16333:head# 2024-04-23T16:33:40.509 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a0ae4f0:::benchmark_data_smithi188_113957_object79186:head# 2024-04-23T16:33:40.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a0b6e00:::benchmark_data_smithi188_113957_object45705:head# 2024-04-23T16:33:40.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a0bf3a9:::benchmark_data_smithi188_113957_object9729:head# 2024-04-23T16:33:40.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a0c330a:::benchmark_data_smithi188_113957_object67436:head# 2024-04-23T16:33:40.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a0d3892:::benchmark_data_smithi188_113957_object31054:head# 2024-04-23T16:33:40.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a0f2e70:::benchmark_data_smithi188_113957_object108239:head# 2024-04-23T16:33:40.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a10d4a2:::benchmark_data_smithi188_113957_object64440:head# 2024-04-23T16:33:40.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a1101d4:::benchmark_data_smithi188_113957_object127734:head# 2024-04-23T16:33:40.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a113f2b:::benchmark_data_smithi188_113957_object32488:head# 2024-04-23T16:33:40.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a12a228:::benchmark_data_smithi188_113957_object151860:head# 2024-04-23T16:33:40.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a12d845:::benchmark_data_smithi188_113957_object80038:head# 2024-04-23T16:33:40.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a138fdd:::benchmark_data_smithi188_113957_object80332:head# 2024-04-23T16:33:40.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a140e6f:::benchmark_data_smithi188_113957_object135239:head# 2024-04-23T16:33:40.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a151e0e:::benchmark_data_smithi188_113957_object69815:head# 2024-04-23T16:33:40.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a15d48b:::benchmark_data_smithi188_113957_object68554:head# 2024-04-23T16:33:40.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a163ced:::benchmark_data_smithi188_113957_object72795:head# 2024-04-23T16:33:40.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a16582b:::benchmark_data_smithi188_113957_object29478:head# 2024-04-23T16:33:40.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a167576:::benchmark_data_smithi188_113957_object110087:head# 2024-04-23T16:33:40.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a16d7c6:::benchmark_data_smithi188_113957_object73412:head# 2024-04-23T16:33:40.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a16da1d:::benchmark_data_smithi188_113957_object17110:head# 2024-04-23T16:33:40.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a1995d8:::benchmark_data_smithi188_113957_object1745:head# 2024-04-23T16:33:40.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a199c91:::benchmark_data_smithi188_113957_object121894:head# 2024-04-23T16:33:40.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a1b10f2:::benchmark_data_smithi188_113957_object96394:head# 2024-04-23T16:33:40.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a1b2812:::benchmark_data_smithi188_113957_object17853:head# 2024-04-23T16:33:40.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a1b4bef:::benchmark_data_smithi188_113957_object115300:head# 2024-04-23T16:33:40.523 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a1b87b7:::benchmark_data_smithi188_113957_object114722:head# 2024-04-23T16:33:40.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a1bd363:::benchmark_data_smithi188_113957_object68472:head# 2024-04-23T16:33:40.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a1bf81f:::benchmark_data_smithi188_113957_object29541:head# 2024-04-23T16:33:40.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a1d0e9b:::benchmark_data_smithi188_113957_object92073:head# 2024-04-23T16:33:40.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a1dc978:::benchmark_data_smithi188_113957_object89029:head# 2024-04-23T16:33:40.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a1e1ae1:::benchmark_data_smithi188_113957_object144808:head# 2024-04-23T16:33:40.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a1e4e06:::benchmark_data_smithi188_113957_object45645:head# 2024-04-23T16:33:40.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a1ed67e:::benchmark_data_smithi188_113957_object21843:head# 2024-04-23T16:33:40.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a1f0a3d:::benchmark_data_smithi188_113957_object97686:head# 2024-04-23T16:33:40.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a1f73b7:::benchmark_data_smithi188_113957_object47405:head# 2024-04-23T16:33:40.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a227f7f:::benchmark_data_smithi188_113957_object73615:head# 2024-04-23T16:33:40.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a23d6e5:::benchmark_data_smithi188_113957_object132267:head# 2024-04-23T16:33:40.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a23efc1:::benchmark_data_smithi188_113957_object2922:head# 2024-04-23T16:33:40.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a24095b:::benchmark_data_smithi188_113957_object122604:head# 2024-04-23T16:33:40.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a2470b2:::benchmark_data_smithi188_113957_object123289:head# 2024-04-23T16:33:40.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a249db2:::benchmark_data_smithi188_113957_object60323:head# 2024-04-23T16:33:40.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a24f801:::benchmark_data_smithi188_113957_object123774:head# 2024-04-23T16:33:40.531 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a25020e:::benchmark_data_smithi188_113957_object55938:head# 2024-04-23T16:33:40.531 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a25c6cc:::benchmark_data_smithi188_113957_object60454:head# 2024-04-23T16:33:40.531 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a2627cd:::benchmark_data_smithi188_113957_object128679:head# 2024-04-23T16:33:40.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a262e88:::benchmark_data_smithi188_113957_object15834:head# 2024-04-23T16:33:40.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a263205:::benchmark_data_smithi188_113957_object39779:head# 2024-04-23T16:33:40.533 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a26f567:::benchmark_data_smithi188_113957_object21949:head# 2024-04-23T16:33:40.533 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a26fcb9:::benchmark_data_smithi188_113957_object82:head# 2024-04-23T16:33:40.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a2708a9:::benchmark_data_smithi188_113957_object11718:head# 2024-04-23T16:33:40.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a273004:::benchmark_data_smithi188_113957_object11198:head# 2024-04-23T16:33:40.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a27ea40:::benchmark_data_smithi188_113957_object112660:head# 2024-04-23T16:33:40.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a287e87:::benchmark_data_smithi188_113957_object134464:head# 2024-04-23T16:33:40.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a290211:::benchmark_data_smithi188_113957_object138730:head# 2024-04-23T16:33:40.536 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a290ea1:::benchmark_data_smithi188_113957_object41783:head# 2024-04-23T16:33:40.536 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a2965f6:::benchmark_data_smithi188_113957_object141432:head# 2024-04-23T16:33:40.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a2a448f:::benchmark_data_smithi188_113957_object23768:head# 2024-04-23T16:33:40.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a2b7d51:::benchmark_data_smithi188_113957_object125255:head# 2024-04-23T16:33:40.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a2be851:::benchmark_data_smithi188_113957_object101012:head# 2024-04-23T16:33:40.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a2cb6bf:::benchmark_data_smithi188_113957_object142269:head# 2024-04-23T16:33:40.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a2cd588:::benchmark_data_smithi188_113957_object32390:head# 2024-04-23T16:33:40.541 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a2d129d:::benchmark_data_smithi188_113957_object114802:head# 2024-04-23T16:33:40.541 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a2d1d27:::benchmark_data_smithi188_113957_object116775:head# 2024-04-23T16:33:40.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a2d9734:::benchmark_data_smithi188_113957_object146336:head# 2024-04-23T16:33:40.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a2de7a2:::benchmark_data_smithi188_113957_object29247:head# 2024-04-23T16:33:40.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a2e09ab:::benchmark_data_smithi188_113957_object65800:head# 2024-04-23T16:33:40.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a2e7f2c:::benchmark_data_smithi188_113957_object117066:head# 2024-04-23T16:33:40.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a2e936b:::benchmark_data_smithi188_113957_object150797:head# 2024-04-23T16:33:40.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a2f839b:::benchmark_data_smithi188_113957_object27426:head# 2024-04-23T16:33:40.544 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a2fec0c:::benchmark_data_smithi188_113957_object112503:head# 2024-04-23T16:33:40.544 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a2ff37a:::benchmark_data_smithi188_113957_object27979:head# 2024-04-23T16:33:40.545 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a305ecc:::benchmark_data_smithi188_113957_object5290:head# 2024-04-23T16:33:40.545 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a32812d:::benchmark_data_smithi188_113957_object9116:head# 2024-04-23T16:33:40.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a33b8c8:::benchmark_data_smithi188_113957_object13023:head# 2024-04-23T16:33:40.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3402ca:::benchmark_data_smithi188_113957_object71637:head# 2024-04-23T16:33:40.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a343de1:::benchmark_data_smithi188_113957_object131757:head# 2024-04-23T16:33:40.547 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a34c0e7:::benchmark_data_smithi188_113957_object67999:head# 2024-04-23T16:33:40.547 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a362c0b:::benchmark_data_smithi188_113957_object15404:head# 2024-04-23T16:33:40.548 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a362fb6:::benchmark_data_smithi188_113957_object58234:head# 2024-04-23T16:33:40.548 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3676d8:::benchmark_data_smithi188_113957_object134311:head# 2024-04-23T16:33:40.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a372dee:::benchmark_data_smithi188_113957_object140602:head# 2024-04-23T16:33:40.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a376715:::benchmark_data_smithi188_113957_object63137:head# 2024-04-23T16:33:40.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a378289:::benchmark_data_smithi188_113957_object146740:head# 2024-04-23T16:33:40.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a37aab3:::benchmark_data_smithi188_113957_object63643:head# 2024-04-23T16:33:40.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a380acd:::benchmark_data_smithi188_113957_object135353:head# 2024-04-23T16:33:40.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a396313:::benchmark_data_smithi188_113957_object96249:head# 2024-04-23T16:33:40.553 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a39eafd:::benchmark_data_smithi188_113957_object40843:head# 2024-04-23T16:33:40.554 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3a2ca7:::benchmark_data_smithi188_113957_object147123:head# 2024-04-23T16:33:40.554 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3ac5a9:::benchmark_data_smithi188_113957_object22278:head# 2024-04-23T16:33:40.555 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3b0e04:::benchmark_data_smithi188_113957_object3133:head# 2024-04-23T16:33:40.555 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3b0ff5:::benchmark_data_smithi188_113957_object101812:head# 2024-04-23T16:33:40.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3b457c:::benchmark_data_smithi188_113957_object120872:head# 2024-04-23T16:33:40.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3b4a02:::benchmark_data_smithi188_113957_object29338:head# 2024-04-23T16:33:40.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3ba333:::benchmark_data_smithi188_113957_object111899:head# 2024-04-23T16:33:40.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3bde8c:::benchmark_data_smithi188_113957_object75801:head# 2024-04-23T16:33:40.558 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3bf4a8:::benchmark_data_smithi188_113957_object61385:head# 2024-04-23T16:33:40.558 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3c0ab8:::benchmark_data_smithi188_113957_object26064:head# 2024-04-23T16:33:40.558 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3c49ab:::benchmark_data_smithi188_113957_object23724:head# 2024-04-23T16:33:40.558 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3cb919:::benchmark_data_smithi188_113957_object3483:head# 2024-04-23T16:33:40.559 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3d3047:::benchmark_data_smithi188_113957_object18525:head# 2024-04-23T16:33:40.559 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3d7725:::benchmark_data_smithi188_113957_object106047:head# 2024-04-23T16:33:40.559 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3dca2b:::benchmark_data_smithi188_113957_object129321:head# 2024-04-23T16:33:40.560 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3dee65:::benchmark_data_smithi188_113957_object29135:head# 2024-04-23T16:33:40.560 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3ee66d:::benchmark_data_smithi188_113957_object8714:head# 2024-04-23T16:33:40.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3f2719:::benchmark_data_smithi188_113957_object124131:head# 2024-04-23T16:33:40.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3f8f01:::benchmark_data_smithi188_113957_object61390:head# 2024-04-23T16:33:40.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a3ff3f8:::benchmark_data_smithi188_113957_object95974:head# 2024-04-23T16:33:40.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a4068b1:::benchmark_data_smithi188_113957_object18715:head# 2024-04-23T16:33:40.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a417cde:::benchmark_data_smithi188_113957_object93735:head# 2024-04-23T16:33:40.564 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a41d9bd:::benchmark_data_smithi188_113957_object35160:head# 2024-04-23T16:33:40.564 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a41ff6d:::benchmark_data_smithi188_113957_object107412:head# 2024-04-23T16:33:40.565 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a42c3b7:::benchmark_data_smithi188_113957_object127861:head# 2024-04-23T16:33:40.565 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a43a392:::benchmark_data_smithi188_113957_object138350:head# 2024-04-23T16:33:40.566 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a43c58e:::benchmark_data_smithi188_113957_object125795:head# 2024-04-23T16:33:40.566 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a43fd76:::benchmark_data_smithi188_113957_object146117:head# 2024-04-23T16:33:40.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a4415a4:::benchmark_data_smithi188_113957_object102780:head# 2024-04-23T16:33:40.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a4418fc:::benchmark_data_smithi188_113957_object67697:head# 2024-04-23T16:33:40.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a445a66:::benchmark_data_smithi188_113957_object57012:head# 2024-04-23T16:33:40.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a4468b8:::benchmark_data_smithi188_113957_object134327:head# 2024-04-23T16:33:40.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a44720c:::benchmark_data_smithi188_113957_object42577:head# 2024-04-23T16:33:40.569 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a44d07b:::benchmark_data_smithi188_113957_object29985:head# 2024-04-23T16:33:40.569 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a450dee:::benchmark_data_smithi188_113957_object70513:head# 2024-04-23T16:33:40.570 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a453ebf:::benchmark_data_smithi188_113957_object11799:head# 2024-04-23T16:33:40.570 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a45d196:::benchmark_data_smithi188_113957_object102486:head# 2024-04-23T16:33:40.570 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a45e223:::benchmark_data_smithi188_113957_object103229:head# 2024-04-23T16:33:40.571 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a461e36:::benchmark_data_smithi188_113957_object74778:head# 2024-04-23T16:33:40.571 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a480f3f:::benchmark_data_smithi188_113957_object106390:head# 2024-04-23T16:33:40.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a49c8d8:::benchmark_data_smithi188_113957_object30711:head# 2024-04-23T16:33:40.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a4a9d3a:::benchmark_data_smithi188_113957_object24290:head# 2024-04-23T16:33:40.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a4acd50:::benchmark_data_smithi188_113957_object84936:head# 2024-04-23T16:33:40.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a4b5c04:::benchmark_data_smithi188_113957_object4229:head# 2024-04-23T16:33:40.574 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:40.574 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a4b5d39:::benchmark_data_smithi188_113957_object42369:head# 2024-04-23T16:33:40.574 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a4b867b:::benchmark_data_smithi188_113957_object144009:head# 2024-04-23T16:33:40.576 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a4bf5d2:::benchmark_data_smithi188_113957_object81473:head# 2024-04-23T16:33:40.576 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a4c5a4e:::benchmark_data_smithi188_113957_object46513:head# 2024-04-23T16:33:40.576 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a4c65c3:::benchmark_data_smithi188_113957_object74388:head# 2024-04-23T16:33:40.577 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a4cd0bd:::benchmark_data_smithi188_113957_object124175:head# 2024-04-23T16:33:40.577 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a4dc27b:::benchmark_data_smithi188_113957_object60089:head# 2024-04-23T16:33:40.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a4fc54f:::benchmark_data_smithi188_113957_object15164:head# 2024-04-23T16:33:40.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a4ff77f:::benchmark_data_smithi188_113957_object1956:head# 2024-04-23T16:33:40.579 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a504128:::benchmark_data_smithi188_113957_object98177:head# 2024-04-23T16:33:40.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a505b74:::benchmark_data_smithi188_113957_object147498:head# 2024-04-23T16:33:40.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a50de81:::benchmark_data_smithi188_113957_object66852:head# 2024-04-23T16:33:40.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a512711:::benchmark_data_smithi188_113957_object108241:head# 2024-04-23T16:33:40.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a5181cb:::benchmark_data_smithi188_113957_object120062:head# 2024-04-23T16:33:40.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a519a10:::benchmark_data_smithi188_113957_object77760:head# 2024-04-23T16:33:40.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a51cf52:::benchmark_data_smithi188_113957_object109035:head# 2024-04-23T16:33:40.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a5223f3:::benchmark_data_smithi188_113957_object98410:head# 2024-04-23T16:33:40.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a525840:::benchmark_data_smithi188_113957_object110076:head# 2024-04-23T16:33:40.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a533677:::benchmark_data_smithi188_113957_object14364:head# 2024-04-23T16:33:40.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a537105:::benchmark_data_smithi188_113957_object17562:head# 2024-04-23T16:33:40.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a54ce96:::benchmark_data_smithi188_113957_object70813:head# 2024-04-23T16:33:40.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a55346d:::benchmark_data_smithi188_113957_object92362:head# 2024-04-23T16:33:40.585 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a5669e3:::benchmark_data_smithi188_113957_object88862:head# 2024-04-23T16:33:40.585 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a56ce19:::benchmark_data_smithi188_113957_object77857:head# 2024-04-23T16:33:40.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a56e294:::benchmark_data_smithi188_113957_object116927:head# 2024-04-23T16:33:40.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a58342d:::benchmark_data_smithi188_113957_object152951:head# 2024-04-23T16:33:40.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a585220:::benchmark_data_smithi188_113957_object145857:head# 2024-04-23T16:33:40.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a586389:::benchmark_data_smithi188_113957_object101069:head# 2024-04-23T16:33:40.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a59b35d:::benchmark_data_smithi188_113957_object152884:head# 2024-04-23T16:33:40.588 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a59f6c7:::benchmark_data_smithi188_113957_object99772:head# 2024-04-23T16:33:40.588 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a5a3bde:::benchmark_data_smithi188_113957_object128439:head# 2024-04-23T16:33:40.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a5a5c65:::benchmark_data_smithi188_113957_object136437:head# 2024-04-23T16:33:40.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a5c0bdb:::benchmark_data_smithi188_113957_object66953:head# 2024-04-23T16:33:40.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a5c7807:::benchmark_data_smithi188_113957_object78889:head# 2024-04-23T16:33:40.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a5c8106:::benchmark_data_smithi188_113957_object60118:head# 2024-04-23T16:33:40.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a5e6f6a:::benchmark_data_smithi188_113957_object26393:head# 2024-04-23T16:33:40.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a5ee94f:::benchmark_data_smithi188_113957_object132789:head# 2024-04-23T16:33:40.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a5f2ec1:::benchmark_data_smithi188_113957_object134726:head# 2024-04-23T16:33:40.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a606325:::benchmark_data_smithi188_113957_object104074:head# 2024-04-23T16:33:40.592 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a607f0d:::benchmark_data_smithi188_113957_object73673:head# 2024-04-23T16:33:40.592 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a60bc66:::benchmark_data_smithi188_113957_object38494:head# 2024-04-23T16:33:40.592 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a60ce51:::benchmark_data_smithi188_113957_object13954:head# 2024-04-23T16:33:40.593 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a61046e:::benchmark_data_smithi188_113957_object4480:head# 2024-04-23T16:33:40.593 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a613612:::benchmark_data_smithi188_113957_object146762:head# 2024-04-23T16:33:40.594 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a61a870:::benchmark_data_smithi188_113957_object44142:head# 2024-04-23T16:33:40.594 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a6230e0:::benchmark_data_smithi188_113957_object143837:head# 2024-04-23T16:33:40.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a629342:::benchmark_data_smithi188_113957_object152409:head# 2024-04-23T16:33:40.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a629e6e:::benchmark_data_smithi188_113957_object37916:head# 2024-04-23T16:33:40.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a635bf5:::benchmark_data_smithi188_113957_object106717:head# 2024-04-23T16:33:40.596 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a636a48:::benchmark_data_smithi188_113957_object21510:head# 2024-04-23T16:33:40.596 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a637c82:::benchmark_data_smithi188_113957_object29184:head# 2024-04-23T16:33:40.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a64b1a8:::benchmark_data_smithi188_113957_object13857:head# 2024-04-23T16:33:40.598 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a65b73b:::benchmark_data_smithi188_113957_object33783:head# 2024-04-23T16:33:40.598 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a67074c:::benchmark_data_smithi188_113957_object35374:head# 2024-04-23T16:33:40.598 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a6723ef:::benchmark_data_smithi188_113957_object45718:head# 2024-04-23T16:33:40.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a68361f:::benchmark_data_smithi188_113957_object93851:head# 2024-04-23T16:33:40.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a689e11:::benchmark_data_smithi188_113957_object46998:head# 2024-04-23T16:33:40.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a68f433:::benchmark_data_smithi188_113957_object85819:head# 2024-04-23T16:33:40.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a6901b6:::benchmark_data_smithi188_113957_object95855:head# 2024-04-23T16:33:40.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a6919ba:::benchmark_data_smithi188_113957_object148010:head# 2024-04-23T16:33:40.602 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a692b77:::benchmark_data_smithi188_113957_object110655:head# 2024-04-23T16:33:40.602 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a69a57e:::benchmark_data_smithi188_113957_object13315:head# 2024-04-23T16:33:40.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a69bbe4:::benchmark_data_smithi188_113957_object88912:head# 2024-04-23T16:33:40.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a6a0f44:::benchmark_data_smithi188_113957_object147059:head# 2024-04-23T16:33:40.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a6a40f4:::benchmark_data_smithi188_113957_object48871:head# 2024-04-23T16:33:40.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a6be88f:::benchmark_data_smithi188_113957_object93658:head# 2024-04-23T16:33:40.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a6c7329:::benchmark_data_smithi188_113957_object115343:head# 2024-04-23T16:33:40.604 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:40.605 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a6cab64:::benchmark_data_smithi188_113957_object29293:head# 2024-04-23T16:33:40.605 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a6cd580:::benchmark_data_smithi188_113957_object112768:head# 2024-04-23T16:33:40.605 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a6ce5b5:::benchmark_data_smithi188_113957_object75802:head# 2024-04-23T16:33:40.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a6cf82e:::benchmark_data_smithi188_113957_object28932:head# 2024-04-23T16:33:40.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a6d2ffa:::benchmark_data_smithi188_113957_object96897:head# 2024-04-23T16:33:40.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a6d924e:::benchmark_data_smithi188_113957_object19276:head# 2024-04-23T16:33:40.607 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a6e5257:::benchmark_data_smithi188_113957_object122058:head# 2024-04-23T16:33:40.607 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a6eb3f0:::benchmark_data_smithi188_113957_object122382:head# 2024-04-23T16:33:40.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:40.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a6efb32:::benchmark_data_smithi188_113957_object124927:head# 2024-04-23T16:33:40.609 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a6fca64:::benchmark_data_smithi188_113957_object40597:head# 2024-04-23T16:33:40.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:40.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a6fe604:::benchmark_data_smithi188_113957_object49858:head# 2024-04-23T16:33:40.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a70040f:::benchmark_data_smithi188_113957_object84766:head# 2024-04-23T16:33:40.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a702505:::benchmark_data_smithi188_113957_object44227:head# 2024-04-23T16:33:40.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a7085b6:::benchmark_data_smithi188_113957_object67684:head# 2024-04-23T16:33:40.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a70e209:::benchmark_data_smithi188_113957_object123113:head# 2024-04-23T16:33:40.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a71999e:::benchmark_data_smithi188_113957_object51997:head# 2024-04-23T16:33:40.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a71b632:::benchmark_data_smithi188_113957_object36436:head# 2024-04-23T16:33:40.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a71dae7:::benchmark_data_smithi188_113957_object143677:head# 2024-04-23T16:33:40.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a7297e9:::benchmark_data_smithi188_113957_object140618:head# 2024-04-23T16:33:40.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a729d97:::benchmark_data_smithi188_113957_object136406:head# 2024-04-23T16:33:40.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a72ad89:::benchmark_data_smithi188_113957_object87479:head# 2024-04-23T16:33:40.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a732169:::benchmark_data_smithi188_113957_object120633:head# 2024-04-23T16:33:40.617 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a73590b:::benchmark_data_smithi188_113957_object116845:head# 2024-04-23T16:33:40.617 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a737946:::benchmark_data_smithi188_113957_object27865:head# 2024-04-23T16:33:40.617 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a73fa18:::benchmark_data_smithi188_113957_object111329:head# 2024-04-23T16:33:40.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a744200:::benchmark_data_smithi188_113957_object37822:head# 2024-04-23T16:33:40.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a748f01:::benchmark_data_smithi188_113957_object110561:head# 2024-04-23T16:33:40.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a74e43f:::benchmark_data_smithi188_113957_object94600:head# 2024-04-23T16:33:40.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a755e67:::benchmark_data_smithi188_113957_object37331:head# 2024-04-23T16:33:40.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a764acd:::benchmark_data_smithi188_113957_object36395:head# 2024-04-23T16:33:40.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a768a6e:::benchmark_data_smithi188_113957_object48931:head# 2024-04-23T16:33:40.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a76bb7a:::benchmark_data_smithi188_113957_object123860:head# 2024-04-23T16:33:40.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a777181:::benchmark_data_smithi188_113957_object100449:head# 2024-04-23T16:33:40.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a7837b3:::benchmark_data_smithi188_113957_object37232:head# 2024-04-23T16:33:40.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a7895fb:::benchmark_data_smithi188_113957_object100483:head# 2024-04-23T16:33:40.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a78ee25:::benchmark_data_smithi188_113957_object116265:head# 2024-04-23T16:33:40.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a7906b6:::benchmark_data_smithi188_113957_object71384:head# 2024-04-23T16:33:40.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a797e78:::benchmark_data_smithi188_113957_object18394:head# 2024-04-23T16:33:40.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a79a2a5:::benchmark_data_smithi188_113957_object140721:head# 2024-04-23T16:33:40.624 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a79db3d:::benchmark_data_smithi188_113957_object130158:head# 2024-04-23T16:33:40.624 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a7b01fc:::benchmark_data_smithi188_113957_object2110:head# 2024-04-23T16:33:40.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a7b6ceb:::benchmark_data_smithi188_113957_object32258:head# 2024-04-23T16:33:40.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a7c29f4:::benchmark_data_smithi188_113957_object5358:head# 2024-04-23T16:33:40.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a7d2580:::benchmark_data_smithi188_113957_object28502:head# 2024-04-23T16:33:40.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a7d6815:::benchmark_data_smithi188_113957_object30525:head# 2024-04-23T16:33:40.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a7ea47a:::benchmark_data_smithi188_113957_object106872:head# 2024-04-23T16:33:40.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a7ed06a:::benchmark_data_smithi188_113957_object84781:head# 2024-04-23T16:33:40.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a7ed512:::benchmark_data_smithi188_113957_object96279:head# 2024-04-23T16:33:40.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a7f0c04:::benchmark_data_smithi188_113957_object98297:head# 2024-04-23T16:33:40.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a7f1ec7:::benchmark_data_smithi188_113957_object101108:head# 2024-04-23T16:33:40.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a7fa091:::benchmark_data_smithi188_113957_object24713:head# 2024-04-23T16:33:40.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a80db3b:::benchmark_data_smithi188_113957_object137653:head# 2024-04-23T16:33:40.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a81d54e:::benchmark_data_smithi188_113957_object77961:head# 2024-04-23T16:33:40.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a81d95c:::benchmark_data_smithi188_113957_object18557:head# 2024-04-23T16:33:40.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a828708:::benchmark_data_smithi188_113957_object27817:head# 2024-04-23T16:33:40.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a829184:::benchmark_data_smithi188_113957_object114573:head# 2024-04-23T16:33:40.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a8317d0:::benchmark_data_smithi188_113957_object44356:head# 2024-04-23T16:33:40.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a83689c:::benchmark_data_smithi188_113957_object19282:head# 2024-04-23T16:33:40.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a83a7f6:::benchmark_data_smithi188_113957_object23730:head# 2024-04-23T16:33:40.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a8471d1:::benchmark_data_smithi188_113957_object140824:head# 2024-04-23T16:33:40.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a848717:::benchmark_data_smithi188_113957_object5439:head# 2024-04-23T16:33:40.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a850f96:::benchmark_data_smithi188_113957_object131687:head# 2024-04-23T16:33:40.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a853732:::benchmark_data_smithi188_113957_object42124:head# 2024-04-23T16:33:40.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a8676ac:::benchmark_data_smithi188_113957_object9583:head# 2024-04-23T16:33:40.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a868fd3:::benchmark_data_smithi188_113957_object88077:head# 2024-04-23T16:33:40.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a86c119:::benchmark_data_smithi188_113957_object119796:head# 2024-04-23T16:33:40.635 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a89222d:::benchmark_data_smithi188_113957_object120546:head# 2024-04-23T16:33:40.635 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a89bed0:::benchmark_data_smithi188_113957_object90184:head# 2024-04-23T16:33:40.635 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a8a6f16:::benchmark_data_smithi188_113957_object132718:head# 2024-04-23T16:33:40.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a8ab3fb:::benchmark_data_smithi188_113957_object8786:head# 2024-04-23T16:33:40.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a8be68d:::benchmark_data_smithi188_113957_object122836:head# 2024-04-23T16:33:40.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a8c055b:::benchmark_data_smithi188_113957_object112137:head# 2024-04-23T16:33:40.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a8ef04e:::benchmark_data_smithi188_113957_object147708:head# 2024-04-23T16:33:40.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a8f8604:::benchmark_data_smithi188_113957_object27648:head# 2024-04-23T16:33:40.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a8fec63:::benchmark_data_smithi188_113957_object19969:head# 2024-04-23T16:33:40.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a8ff456:::benchmark_data_smithi188_113957_object2660:head# 2024-04-23T16:33:40.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a90c08d:::benchmark_data_smithi188_113957_object131933:head# 2024-04-23T16:33:40.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a91306e:::benchmark_data_smithi188_113957_object143489:head# 2024-04-23T16:33:40.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a91db8c:::benchmark_data_smithi188_113957_object138696:head# 2024-04-23T16:33:40.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a91e6b2:::benchmark_data_smithi188_113957_object11741:head# 2024-04-23T16:33:40.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a92aaf8:::benchmark_data_smithi188_113957_object27556:head# 2024-04-23T16:33:40.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a93539c:::benchmark_data_smithi188_113957_object71067:head# 2024-04-23T16:33:40.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a936872:::benchmark_data_smithi188_113957_object107364:head# 2024-04-23T16:33:40.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a93cb06:::benchmark_data_smithi188_113957_object151148:head# 2024-04-23T16:33:40.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a93f826:::benchmark_data_smithi188_113957_object82932:head# 2024-04-23T16:33:40.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9428ec:::benchmark_data_smithi188_113957_object36519:head# 2024-04-23T16:33:40.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9483f1:::benchmark_data_smithi188_113957_object87570:head# 2024-04-23T16:33:40.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a95176c:::benchmark_data_smithi188_113957_object128974:head# 2024-04-23T16:33:40.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a95b7db:::benchmark_data_smithi188_113957_object114658:head# 2024-04-23T16:33:40.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a95bf53:::benchmark_data_smithi188_113957_object103286:head# 2024-04-23T16:33:40.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a95c900:::benchmark_data_smithi188_113957_object132194:head# 2024-04-23T16:33:40.645 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9602b5:::benchmark_data_smithi188_113957_object92775:head# 2024-04-23T16:33:40.645 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a969894:::benchmark_data_smithi188_113957_object149545:head# 2024-04-23T16:33:40.645 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a97054d:::benchmark_data_smithi188_113957_object94221:head# 2024-04-23T16:33:40.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9872bd:::benchmark_data_smithi188_113957_object77294:head# 2024-04-23T16:33:40.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9909de:::benchmark_data_smithi188_113957_object90616:head# 2024-04-23T16:33:40.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9a186a:::benchmark_data_smithi188_113957_object87782:head# 2024-04-23T16:33:40.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9a44bd:::benchmark_data_smithi188_113957_object7791:head# 2024-04-23T16:33:40.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9ad0f9:::benchmark_data_smithi188_113957_object102756:head# 2024-04-23T16:33:40.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9b4c0e:::benchmark_data_smithi188_113957_object39277:head# 2024-04-23T16:33:40.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9b86b6:::benchmark_data_smithi188_113957_object41766:head# 2024-04-23T16:33:40.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9bf7d7:::benchmark_data_smithi188_113957_object118772:head# 2024-04-23T16:33:40.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9bfbff:::benchmark_data_smithi188_113957_object33978:head# 2024-04-23T16:33:40.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9c85e5:::benchmark_data_smithi188_113957_object20640:head# 2024-04-23T16:33:40.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9cb756:::benchmark_data_smithi188_113957_object32432:head# 2024-04-23T16:33:40.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9cb863:::benchmark_data_smithi188_113957_object69131:head# 2024-04-23T16:33:40.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9dba56:::benchmark_data_smithi188_113957_object82290:head# 2024-04-23T16:33:40.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9ddbc7:::benchmark_data_smithi188_113957_object69442:head# 2024-04-23T16:33:40.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9ddbf6:::benchmark_data_smithi188_113957_object91705:head# 2024-04-23T16:33:40.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9e851b:::benchmark_data_smithi188_113957_object24647:head# 2024-04-23T16:33:40.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9f0112:::benchmark_data_smithi188_113957_object20837:head# 2024-04-23T16:33:40.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9f0f3c:::benchmark_data_smithi188_113957_object15066:head# 2024-04-23T16:33:40.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9fcaf1:::benchmark_data_smithi188_113957_object134377:head# 2024-04-23T16:33:40.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9fcb0b:::benchmark_data_smithi188_113957_object111463:head# 2024-04-23T16:33:40.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5a9fd0bc:::benchmark_data_smithi188_113957_object56675:head# 2024-04-23T16:33:40.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa02716:::benchmark_data_smithi188_113957_object4588:head# 2024-04-23T16:33:40.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa06674:::benchmark_data_smithi188_113957_object61400:head# 2024-04-23T16:33:40.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa075fe:::benchmark_data_smithi188_113957_object93615:head# 2024-04-23T16:33:40.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa09528:::benchmark_data_smithi188_113957_object148640:head# 2024-04-23T16:33:40.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa0d09e:::benchmark_data_smithi188_113957_object80398:head# 2024-04-23T16:33:40.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa1b42b:::benchmark_data_smithi188_113957_object144330:head# 2024-04-23T16:33:40.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa20aaa:::benchmark_data_smithi188_113957_object50480:head# 2024-04-23T16:33:40.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa2e508:::benchmark_data_smithi188_113957_object136690:head# 2024-04-23T16:33:40.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa37ccd:::benchmark_data_smithi188_113957_object133613:head# 2024-04-23T16:33:40.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa39845:::benchmark_data_smithi188_113957_object105668:head# 2024-04-23T16:33:40.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa39dde:::benchmark_data_smithi188_113957_object94168:head# 2024-04-23T16:33:40.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa429cb:::benchmark_data_smithi188_113957_object80363:head# 2024-04-23T16:33:40.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa4a86c:::benchmark_data_smithi188_113957_object67889:head# 2024-04-23T16:33:40.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa4c53e:::benchmark_data_smithi188_113957_object117596:head# 2024-04-23T16:33:40.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa4dfdb:::benchmark_data_smithi188_113957_object151176:head# 2024-04-23T16:33:40.661 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa4eed0:::benchmark_data_smithi188_113957_object115641:head# 2024-04-23T16:33:40.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa5a109:::benchmark_data_smithi188_113957_object9937:head# 2024-04-23T16:33:40.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa652d5:::benchmark_data_smithi188_113957_object114628:head# 2024-04-23T16:33:40.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa7d0a5:::benchmark_data_smithi188_113957_object17643:head# 2024-04-23T16:33:40.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa99399:::benchmark_data_smithi188_113957_object46067:head# 2024-04-23T16:33:40.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa9a800:::benchmark_data_smithi188_113957_object58911:head# 2024-04-23T16:33:40.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa9c3cb:::benchmark_data_smithi188_113957_object130691:head# 2024-04-23T16:33:40.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aa9d6a7:::benchmark_data_smithi188_113957_object26089:head# 2024-04-23T16:33:40.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aaae189:::benchmark_data_smithi188_113957_object14601:head# 2024-04-23T16:33:40.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aab335d:::benchmark_data_smithi188_113957_object39625:head# 2024-04-23T16:33:40.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aab8b49:::benchmark_data_smithi188_113957_object65394:head# 2024-04-23T16:33:40.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aabf71a:::benchmark_data_smithi188_113957_object150746:head# 2024-04-23T16:33:40.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aacdc2a:::benchmark_data_smithi188_113957_object20332:head# 2024-04-23T16:33:40.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aacef54:::benchmark_data_smithi188_113957_object61451:head# 2024-04-23T16:33:40.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aad9a06:::benchmark_data_smithi188_113957_object23373:head# 2024-04-23T16:33:40.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aae3487:::benchmark_data_smithi188_113957_object139004:head# 2024-04-23T16:33:40.668 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aae798a:::benchmark_data_smithi188_113957_object111268:head# 2024-04-23T16:33:40.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aafae6f:::benchmark_data_smithi188_113957_object42705:head# 2024-04-23T16:33:40.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aafec99:::benchmark_data_smithi188_113957_object137517:head# 2024-04-23T16:33:40.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab0ad34:::benchmark_data_smithi188_113957_object104249:head# 2024-04-23T16:33:40.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab0aec4:::benchmark_data_smithi188_113957_object14807:head# 2024-04-23T16:33:40.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab0dbb9:::benchmark_data_smithi188_113957_object7948:head# 2024-04-23T16:33:40.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab140c8:::benchmark_data_smithi188_113957_object12667:head# 2024-04-23T16:33:40.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab16195:::benchmark_data_smithi188_113957_object12532:head# 2024-04-23T16:33:40.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab2a77b:::benchmark_data_smithi188_113957_object58036:head# 2024-04-23T16:33:40.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab2b66b:::benchmark_data_smithi188_113957_object39471:head# 2024-04-23T16:33:40.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab3a83a:::benchmark_data_smithi188_113957_object142472:head# 2024-04-23T16:33:40.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab44876:::benchmark_data_smithi188_113957_object127932:head# 2024-04-23T16:33:40.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab4ee10:::benchmark_data_smithi188_113957_object21223:head# 2024-04-23T16:33:40.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab516ac:::benchmark_data_smithi188_113957_object46245:head# 2024-04-23T16:33:40.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab6c3a7:::benchmark_data_smithi188_113957_object96058:head# 2024-04-23T16:33:40.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab74206:::benchmark_data_smithi188_113957_object98926:head# 2024-04-23T16:33:40.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab7ba93:::benchmark_data_smithi188_113957_object6090:head# 2024-04-23T16:33:40.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab8204e:::benchmark_data_smithi188_113957_object35223:head# 2024-04-23T16:33:40.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab82478:::benchmark_data_smithi188_113957_object5012:head# 2024-04-23T16:33:40.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab88e78:::benchmark_data_smithi188_113957_object51529:head# 2024-04-23T16:33:40.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab8bdbf:::benchmark_data_smithi188_113957_object121144:head# 2024-04-23T16:33:40.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab8c9f6:::benchmark_data_smithi188_113957_object106448:head# 2024-04-23T16:33:40.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab93ebb:::benchmark_data_smithi188_113957_object117857:head# 2024-04-23T16:33:40.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab9430b:::benchmark_data_smithi188_113957_object3927:head# 2024-04-23T16:33:40.678 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ab976c7:::benchmark_data_smithi188_113957_object64826:head# 2024-04-23T16:33:40.678 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aba5539:::benchmark_data_smithi188_113957_object77198:head# 2024-04-23T16:33:40.678 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5abb0a56:::benchmark_data_smithi188_113957_object126153:head# 2024-04-23T16:33:40.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5abc4288:::benchmark_data_smithi188_113957_object109395:head# 2024-04-23T16:33:40.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5abc8785:::benchmark_data_smithi188_113957_object58669:head# 2024-04-23T16:33:40.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5abcf9d9:::benchmark_data_smithi188_113957_object124821:head# 2024-04-23T16:33:40.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5abd0863:::benchmark_data_smithi188_113957_object5030:head# 2024-04-23T16:33:40.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5abd5755:::benchmark_data_smithi188_113957_object21444:head# 2024-04-23T16:33:40.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5abdc020:::benchmark_data_smithi188_113957_object135934:head# 2024-04-23T16:33:40.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5abde79f:::benchmark_data_smithi188_113957_object125540:head# 2024-04-23T16:33:40.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5abdef5a:::benchmark_data_smithi188_113957_object28725:head# 2024-04-23T16:33:40.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5abe61ad:::benchmark_data_smithi188_113957_object10220:head# 2024-04-23T16:33:40.683 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5abe687f:::benchmark_data_smithi188_113957_object120128:head# 2024-04-23T16:33:40.683 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5abe9103:::benchmark_data_smithi188_113957_object137699:head# 2024-04-23T16:33:40.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5abeb11d:::benchmark_data_smithi188_113957_object121627:head# 2024-04-23T16:33:40.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5abec261:::benchmark_data_smithi188_113957_object127275:head# 2024-04-23T16:33:40.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5abef06a:::benchmark_data_smithi188_113957_object104862:head# 2024-04-23T16:33:40.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5abf18a4:::benchmark_data_smithi188_113957_object13436:head# 2024-04-23T16:33:40.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5abfc149:::benchmark_data_smithi188_113957_object80405:head# 2024-04-23T16:33:40.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac02c7d:::benchmark_data_smithi188_113957_object77992:head# 2024-04-23T16:33:40.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac07b24:::benchmark_data_smithi188_113957_object20939:head# 2024-04-23T16:33:40.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac0aa37:::benchmark_data_smithi188_113957_object26880:head# 2024-04-23T16:33:40.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac12826:::benchmark_data_smithi188_113957_object50773:head# 2024-04-23T16:33:40.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac13e89:::benchmark_data_smithi188_113957_object51195:head# 2024-04-23T16:33:40.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac1add8:::benchmark_data_smithi188_113957_object9964:head# 2024-04-23T16:33:40.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac2c306:::benchmark_data_smithi188_113957_object118149:head# 2024-04-23T16:33:40.689 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac2c360:::benchmark_data_smithi188_113957_object130957:head# 2024-04-23T16:33:40.689 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac2d59e:::benchmark_data_smithi188_113957_object40737:head# 2024-04-23T16:33:40.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac2ead9:::benchmark_data_smithi188_113957_object811:head# 2024-04-23T16:33:40.691 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac3a3b3:::benchmark_data_smithi188_113957_object85543:head# 2024-04-23T16:33:40.691 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac3dcfa:::benchmark_data_smithi188_113957_object135013:head# 2024-04-23T16:33:40.691 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac40875:::benchmark_data_smithi188_113957_object7732:head# 2024-04-23T16:33:40.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac48d44:::benchmark_data_smithi188_113957_object3706:head# 2024-04-23T16:33:40.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac4ffdb:::benchmark_data_smithi188_113957_object145948:head# 2024-04-23T16:33:40.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac50e68:::benchmark_data_smithi188_113957_object101784:head# 2024-04-23T16:33:40.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac50f78:::benchmark_data_smithi188_113957_object38881:head# 2024-04-23T16:33:40.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac557bd:::benchmark_data_smithi188_113957_object26221:head# 2024-04-23T16:33:40.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac56914:::benchmark_data_smithi188_113957_object149836:head# 2024-04-23T16:33:40.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac5f54d:::benchmark_data_smithi188_113957_object84359:head# 2024-04-23T16:33:40.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac6e97f:::benchmark_data_smithi188_113957_object5943:head# 2024-04-23T16:33:40.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac894ca:::benchmark_data_smithi188_113957_object39645:head# 2024-04-23T16:33:40.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac8b126:::benchmark_data_smithi188_113957_object42536:head# 2024-04-23T16:33:40.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac9adb4:::benchmark_data_smithi188_113957_object57007:head# 2024-04-23T16:33:40.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac9b1b0:::benchmark_data_smithi188_113957_object2081:head# 2024-04-23T16:33:40.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac9d9da:::benchmark_data_smithi188_113957_object147980:head# 2024-04-23T16:33:40.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac9ddea:::benchmark_data_smithi188_113957_object145533:head# 2024-04-23T16:33:40.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac9e24d:::benchmark_data_smithi188_113957_object127278:head# 2024-04-23T16:33:40.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ac9e259:::benchmark_data_smithi188_113957_object14782:head# 2024-04-23T16:33:40.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aca0c3b:::benchmark_data_smithi188_113957_object43946:head# 2024-04-23T16:33:40.699 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5acba531:::benchmark_data_smithi188_113957_object105002:head# 2024-04-23T16:33:40.699 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5acc4565:::benchmark_data_smithi188_113957_object73870:head# 2024-04-23T16:33:40.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5acc6ab2:::benchmark_data_smithi188_113957_object94377:head# 2024-04-23T16:33:40.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5acd9187:::benchmark_data_smithi188_113957_object38898:head# 2024-04-23T16:33:40.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5acdb09a:::benchmark_data_smithi188_113957_object70341:head# 2024-04-23T16:33:40.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ace434c:::benchmark_data_smithi188_113957_object70692:head# 2024-04-23T16:33:40.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5acfb329:::benchmark_data_smithi188_113957_object69538:head# 2024-04-23T16:33:40.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5acfe2c3:::benchmark_data_smithi188_113957_object107683:head# 2024-04-23T16:33:40.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad1a612:::benchmark_data_smithi188_113957_object61165:head# 2024-04-23T16:33:40.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad1c3ce:::benchmark_data_smithi188_113957_object123246:head# 2024-04-23T16:33:40.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad1f996:::benchmark_data_smithi188_113957_object63003:head# 2024-04-23T16:33:40.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad201d3:::benchmark_data_smithi188_113957_object93513:head# 2024-04-23T16:33:40.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad20da5:::benchmark_data_smithi188_113957_object76369:head# 2024-04-23T16:33:40.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad2ceae:::benchmark_data_smithi188_113957_object22312:head# 2024-04-23T16:33:40.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad312fb:::benchmark_data_smithi188_113957_object121250:head# 2024-04-23T16:33:40.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad33ee5:::benchmark_data_smithi188_113957_object121578:head# 2024-04-23T16:33:40.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad36f62:::benchmark_data_smithi188_113957_object79084:head# 2024-04-23T16:33:40.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad430fd:::benchmark_data_smithi188_113957_object102281:head# 2024-04-23T16:33:40.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad4658d:::benchmark_data_smithi188_113957_object79403:head# 2024-04-23T16:33:40.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad46d7e:::benchmark_data_smithi188_113957_object54646:head# 2024-04-23T16:33:40.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad49034:::benchmark_data_smithi188_113957_object69520:head# 2024-04-23T16:33:40.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad4980a:::benchmark_data_smithi188_113957_object18701:head# 2024-04-23T16:33:40.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad4de88:::benchmark_data_smithi188_113957_object105012:head# 2024-04-23T16:33:40.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad523a8:::benchmark_data_smithi188_113957_object4047:head# 2024-04-23T16:33:40.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad5e0f7:::benchmark_data_smithi188_113957_object89762:head# 2024-04-23T16:33:40.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad60c25:::benchmark_data_smithi188_113957_object71388:head# 2024-04-23T16:33:40.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad6a0ae:::benchmark_data_smithi188_113957_object148405:head# 2024-04-23T16:33:40.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad6daad:::benchmark_data_smithi188_113957_object51448:head# 2024-04-23T16:33:40.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad760e5:::benchmark_data_smithi188_113957_object38620:head# 2024-04-23T16:33:40.711 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:40.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad7922e:::benchmark_data_smithi188_113957_object12978:head# 2024-04-23T16:33:40.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad7a715:::benchmark_data_smithi188_113957_object41373:head# 2024-04-23T16:33:40.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad7b571:::benchmark_data_smithi188_113957_object30893:head# 2024-04-23T16:33:40.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad7f4d4:::benchmark_data_smithi188_113957_object823:head# 2024-04-23T16:33:40.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad7ff0e:::benchmark_data_smithi188_113957_object39800:head# 2024-04-23T16:33:40.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad80859:::benchmark_data_smithi188_113957_object3688:head# 2024-04-23T16:33:40.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad8bdd7:::benchmark_data_smithi188_113957_object47901:head# 2024-04-23T16:33:40.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad8f72c:::benchmark_data_smithi188_113957_object57645:head# 2024-04-23T16:33:40.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad987ee:::benchmark_data_smithi188_113957_object103117:head# 2024-04-23T16:33:40.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ad9927f:::benchmark_data_smithi188_113957_object30269:head# 2024-04-23T16:33:40.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ada7a6f:::benchmark_data_smithi188_113957_object98261:head# 2024-04-23T16:33:40.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5adaa279:::benchmark_data_smithi188_113957_object90049:head# 2024-04-23T16:33:40.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5adae12a:::benchmark_data_smithi188_113957_object83235:head# 2024-04-23T16:33:40.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5adb43b5:::benchmark_data_smithi188_113957_object142180:head# 2024-04-23T16:33:40.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5adb8942:::benchmark_data_smithi188_113957_object124650:head# 2024-04-23T16:33:40.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5adc164b:::benchmark_data_smithi188_113957_object12498:head# 2024-04-23T16:33:40.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5adc8566:::benchmark_data_smithi188_113957_object96060:head# 2024-04-23T16:33:40.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5adcd60e:::benchmark_data_smithi188_113957_object43935:head# 2024-04-23T16:33:40.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ade6679:::benchmark_data_smithi188_113957_object29384:head# 2024-04-23T16:33:40.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5adea7a9:::benchmark_data_smithi188_113957_object76499:head# 2024-04-23T16:33:40.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aded5b9:::benchmark_data_smithi188_113957_object5428:head# 2024-04-23T16:33:40.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aded9b8:::benchmark_data_smithi188_113957_object32463:head# 2024-04-23T16:33:40.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5adfe365:::benchmark_data_smithi188_113957_object139995:head# 2024-04-23T16:33:40.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5adfeb79:::benchmark_data_smithi188_113957_object126960:head# 2024-04-23T16:33:40.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae01b89:::benchmark_data_smithi188_113957_object88061:head# 2024-04-23T16:33:40.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae03902:::benchmark_data_smithi188_113957_object8133:head# 2024-04-23T16:33:40.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae0a82a:::benchmark_data_smithi188_113957_object2995:head# 2024-04-23T16:33:40.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae0e144:::benchmark_data_smithi188_113957_object112353:head# 2024-04-23T16:33:40.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae0eb82:::benchmark_data_smithi188_113957_object76642:head# 2024-04-23T16:33:40.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae22215:::benchmark_data_smithi188_113957_object149942:head# 2024-04-23T16:33:40.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae22fe5:::benchmark_data_smithi188_113957_object32721:head# 2024-04-23T16:33:40.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae34de5:::benchmark_data_smithi188_113957_object28528:head# 2024-04-23T16:33:40.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae35464:::benchmark_data_smithi188_113957_object130696:head# 2024-04-23T16:33:40.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae364af:::benchmark_data_smithi188_113957_object144567:head# 2024-04-23T16:33:40.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae3d540:::benchmark_data_smithi188_113957_object135530:head# 2024-04-23T16:33:40.725 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae3f88e:::benchmark_data_smithi188_113957_object128904:head# 2024-04-23T16:33:40.725 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae45047:::benchmark_data_smithi188_113957_object77214:head# 2024-04-23T16:33:40.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae4fa8d:::benchmark_data_smithi188_113957_object6672:head# 2024-04-23T16:33:40.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae53e80:::benchmark_data_smithi188_113957_object68808:head# 2024-04-23T16:33:40.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae5f008:::benchmark_data_smithi188_113957_object12934:head# 2024-04-23T16:33:40.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae5f952:::benchmark_data_smithi188_113957_object147687:head# 2024-04-23T16:33:40.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae69ad5:::benchmark_data_smithi188_113957_object85546:head# 2024-04-23T16:33:40.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae72760:::benchmark_data_smithi188_113957_object112260:head# 2024-04-23T16:33:40.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae7edda:::benchmark_data_smithi188_113957_object80286:head# 2024-04-23T16:33:40.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae87335:::benchmark_data_smithi188_113957_object110400:head# 2024-04-23T16:33:40.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae8ae91:::benchmark_data_smithi188_113957_object112491:head# 2024-04-23T16:33:40.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae917c0:::benchmark_data_smithi188_113957_object56793:head# 2024-04-23T16:33:40.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ae9e96b:::benchmark_data_smithi188_113957_object79971:head# 2024-04-23T16:33:40.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aea1eaf:::benchmark_data_smithi188_113957_object30261:head# 2024-04-23T16:33:40.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aea2ee8:::benchmark_data_smithi188_113957_object9274:head# 2024-04-23T16:33:40.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aeb05b2:::benchmark_data_smithi188_113957_object11923:head# 2024-04-23T16:33:40.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aebd959:::benchmark_data_smithi188_113957_object101957:head# 2024-04-23T16:33:40.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aecc86b:::benchmark_data_smithi188_113957_object144249:head# 2024-04-23T16:33:40.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aeccbc1:::benchmark_data_smithi188_113957_object19913:head# 2024-04-23T16:33:40.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aed7e14:::benchmark_data_smithi188_113957_object11038:head# 2024-04-23T16:33:40.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aedb5db:::benchmark_data_smithi188_113957_object136674:head# 2024-04-23T16:33:40.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aede738:::benchmark_data_smithi188_113957_object46055:head# 2024-04-23T16:33:40.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aedef62:::benchmark_data_smithi188_113957_object65954:head# 2024-04-23T16:33:40.735 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aedf586:::benchmark_data_smithi188_113957_object120266:head# 2024-04-23T16:33:40.735 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aedfbf3:::benchmark_data_smithi188_113957_object5384:head# 2024-04-23T16:33:40.735 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aeec623:::benchmark_data_smithi188_113957_object142142:head# 2024-04-23T16:33:40.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aeef03d:::benchmark_data_smithi188_113957_object76011:head# 2024-04-23T16:33:40.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aeeff06:::benchmark_data_smithi188_113957_object31270:head# 2024-04-23T16:33:40.737 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af02d25:::benchmark_data_smithi188_113957_object121843:head# 2024-04-23T16:33:40.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af06029:::benchmark_data_smithi188_113957_object11569:head# 2024-04-23T16:33:40.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af1b2c1:::benchmark_data_smithi188_113957_object52211:head# 2024-04-23T16:33:40.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af1c860:::benchmark_data_smithi188_113957_object44441:head# 2024-04-23T16:33:40.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af21354:::benchmark_data_smithi188_113957_object66096:head# 2024-04-23T16:33:40.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af24927:::benchmark_data_smithi188_113957_object137554:head# 2024-04-23T16:33:40.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af26dc3:::benchmark_data_smithi188_113957_object107957:head# 2024-04-23T16:33:40.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af2b3ce:::benchmark_data_smithi188_113957_object81311:head# 2024-04-23T16:33:40.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af2ccec:::benchmark_data_smithi188_113957_object104547:head# 2024-04-23T16:33:40.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af36cd9:::benchmark_data_smithi188_113957_object4703:head# 2024-04-23T16:33:40.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af3aba3:::benchmark_data_smithi188_113957_object55791:head# 2024-04-23T16:33:40.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af3c5f9:::benchmark_data_smithi188_113957_object119946:head# 2024-04-23T16:33:40.744 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af494b5:::benchmark_data_smithi188_113957_object29975:head# 2024-04-23T16:33:40.744 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af4df46:::benchmark_data_smithi188_113957_object62062:head# 2024-04-23T16:33:40.745 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af52ab2:::benchmark_data_smithi188_113957_object152257:head# 2024-04-23T16:33:40.745 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af55c58:::benchmark_data_smithi188_113957_object77359:head# 2024-04-23T16:33:40.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af5ab6e:::benchmark_data_smithi188_113957_object106660:head# 2024-04-23T16:33:40.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af5df34:::benchmark_data_smithi188_113957_object142863:head# 2024-04-23T16:33:40.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af5e8e5:::benchmark_data_smithi188_113957_object145795:head# 2024-04-23T16:33:40.750 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af6b81d:::benchmark_data_smithi188_113957_object109299:head# 2024-04-23T16:33:40.750 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af7123e:::benchmark_data_smithi188_113957_object14354:head# 2024-04-23T16:33:40.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af79f1b:::benchmark_data_smithi188_113957_object90212:head# 2024-04-23T16:33:40.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af80580:::benchmark_data_smithi188_113957_object47181:head# 2024-04-23T16:33:40.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af84eb0:::benchmark_data_smithi188_113957_object150579:head# 2024-04-23T16:33:40.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af88f67:::benchmark_data_smithi188_113957_object3571:head# 2024-04-23T16:33:40.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af899b2:::benchmark_data_smithi188_113957_object86534:head# 2024-04-23T16:33:40.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af8ce14:::benchmark_data_smithi188_113957_object114535:head# 2024-04-23T16:33:40.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af93b36:::benchmark_data_smithi188_113957_object148519:head# 2024-04-23T16:33:40.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5af9bab7:::benchmark_data_smithi188_113957_object108436:head# 2024-04-23T16:33:40.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5afa6ff5:::benchmark_data_smithi188_113957_object15505:head# 2024-04-23T16:33:40.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5afc5077:::benchmark_data_smithi188_113957_object95464:head# 2024-04-23T16:33:40.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5afd062f:::benchmark_data_smithi188_113957_object13080:head# 2024-04-23T16:33:40.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5afd1be5:::benchmark_data_smithi188_113957_object23797:head# 2024-04-23T16:33:40.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5afdc42f:::benchmark_data_smithi188_113957_object36913:head# 2024-04-23T16:33:40.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5afe37d8:::benchmark_data_smithi188_113957_object124159:head# 2024-04-23T16:33:40.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5afed319:::benchmark_data_smithi188_113957_object1491:head# 2024-04-23T16:33:40.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aff11fa:::benchmark_data_smithi188_113957_object16812:head# 2024-04-23T16:33:40.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aff2c81:::benchmark_data_smithi188_113957_object147800:head# 2024-04-23T16:33:40.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aff769e:::benchmark_data_smithi188_113957_object144073:head# 2024-04-23T16:33:40.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5aff993f:::benchmark_data_smithi188_113957_object22260:head# 2024-04-23T16:33:40.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5affcf50:::benchmark_data_smithi188_113957_object73268:head# 2024-04-23T16:33:40.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5affd4c5:::benchmark_data_smithi188_113957_object124712:head# 2024-04-23T16:33:40.764 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b001169:::benchmark_data_smithi188_113957_object125068:head# 2024-04-23T16:33:40.764 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b004f35:::benchmark_data_smithi188_113957_object20611:head# 2024-04-23T16:33:40.765 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b00b827:::benchmark_data_smithi188_113957_object102109:head# 2024-04-23T16:33:40.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b01965d:::benchmark_data_smithi188_113957_object78317:head# 2024-04-23T16:33:40.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b01b241:::benchmark_data_smithi188_113957_object4160:head# 2024-04-23T16:33:40.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b01c6a7:::benchmark_data_smithi188_113957_object48989:head# 2024-04-23T16:33:40.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b02620e:::benchmark_data_smithi188_113957_object24027:head# 2024-04-23T16:33:40.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b028349:::benchmark_data_smithi188_113957_object99837:head# 2024-04-23T16:33:40.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b029354:::benchmark_data_smithi188_113957_object71161:head# 2024-04-23T16:33:40.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b02b93a:::benchmark_data_smithi188_113957_object24717:head# 2024-04-23T16:33:40.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b02bf32:::benchmark_data_smithi188_113957_object2647:head# 2024-04-23T16:33:40.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b033c19:::benchmark_data_smithi188_113957_object3061:head# 2024-04-23T16:33:40.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b0362eb:::benchmark_data_smithi188_113957_object3107:head# 2024-04-23T16:33:40.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b0366db:::benchmark_data_smithi188_113957_object67004:head# 2024-04-23T16:33:40.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b03b18c:::benchmark_data_smithi188_113957_object32416:head# 2024-04-23T16:33:40.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b03ff45:::benchmark_data_smithi188_113957_object55157:head# 2024-04-23T16:33:40.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b0440b7:::benchmark_data_smithi188_113957_object48826:head# 2024-04-23T16:33:40.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b0458e9:::benchmark_data_smithi188_113957_object45613:head# 2024-04-23T16:33:40.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b04dc4a:::benchmark_data_smithi188_113957_object124487:head# 2024-04-23T16:33:40.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b0530c7:::benchmark_data_smithi188_113957_object72391:head# 2024-04-23T16:33:40.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b05ebfd:::benchmark_data_smithi188_113957_object34026:head# 2024-04-23T16:33:40.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b06d0fc:::benchmark_data_smithi188_113957_object88353:head# 2024-04-23T16:33:40.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b0825af:::benchmark_data_smithi188_113957_object63693:head# 2024-04-23T16:33:40.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b089656:::benchmark_data_smithi188_113957_object73207:head# 2024-04-23T16:33:40.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b08af99:::benchmark_data_smithi188_113957_object124406:head# 2024-04-23T16:33:40.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b095638:::benchmark_data_smithi188_113957_object108680:head# 2024-04-23T16:33:40.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b098885:::benchmark_data_smithi188_113957_object56744:head# 2024-04-23T16:33:40.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b098e49:::benchmark_data_smithi188_113957_object21391:head# 2024-04-23T16:33:40.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b099a05:::benchmark_data_smithi188_113957_object27942:head# 2024-04-23T16:33:40.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b0af591:::benchmark_data_smithi188_113957_object56920:head# 2024-04-23T16:33:40.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b0b370a:::benchmark_data_smithi188_113957_object141069:head# 2024-04-23T16:33:40.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b0b6a0c:::benchmark_data_smithi188_113957_object87687:head# 2024-04-23T16:33:40.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b0c4df2:::benchmark_data_smithi188_113957_object63577:head# 2024-04-23T16:33:40.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b0c58d0:::benchmark_data_smithi188_113957_object76810:head# 2024-04-23T16:33:40.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b0d096e:::benchmark_data_smithi188_113957_object114264:head# 2024-04-23T16:33:40.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b0d09a7:::benchmark_data_smithi188_113957_object102146:head# 2024-04-23T16:33:40.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b0d6e65:::benchmark_data_smithi188_113957_object93771:head# 2024-04-23T16:33:40.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b0ec35b:::benchmark_data_smithi188_113957_object8423:head# 2024-04-23T16:33:40.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b0f3297:::benchmark_data_smithi188_113957_object127517:head# 2024-04-23T16:33:40.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b0fcaff:::benchmark_data_smithi188_113957_object138588:head# 2024-04-23T16:33:40.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b0fe3ed:::benchmark_data_smithi188_113957_object60316:head# 2024-04-23T16:33:40.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b10939b:::benchmark_data_smithi188_113957_object93099:head# 2024-04-23T16:33:40.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b11501a:::benchmark_data_smithi188_113957_object82072:head# 2024-04-23T16:33:40.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b115917:::benchmark_data_smithi188_113957_object56272:head# 2024-04-23T16:33:40.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b130524:::benchmark_data_smithi188_113957_object12614:head# 2024-04-23T16:33:40.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b13d1f1:::benchmark_data_smithi188_113957_object147893:head# 2024-04-23T16:33:40.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b14ec81:::benchmark_data_smithi188_113957_object59073:head# 2024-04-23T16:33:40.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b154576:::benchmark_data_smithi188_113957_object50214:head# 2024-04-23T16:33:40.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1545de:::benchmark_data_smithi188_113957_object127597:head# 2024-04-23T16:33:40.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b15a2c0:::benchmark_data_smithi188_113957_object117411:head# 2024-04-23T16:33:40.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1672d3:::benchmark_data_smithi188_113957_object128481:head# 2024-04-23T16:33:40.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b16897b:::benchmark_data_smithi188_113957_object76190:head# 2024-04-23T16:33:40.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b16b6ad:::benchmark_data_smithi188_113957_object119656:head# 2024-04-23T16:33:40.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b170b8d:::benchmark_data_smithi188_113957_object45809:head# 2024-04-23T16:33:40.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b17a294:::benchmark_data_smithi188_113957_object58787:head# 2024-04-23T16:33:40.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b190d01:::benchmark_data_smithi188_113957_object124334:head# 2024-04-23T16:33:40.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b19554c:::benchmark_data_smithi188_113957_object144327:head# 2024-04-23T16:33:40.789 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:40.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1989fc:::benchmark_data_smithi188_113957_object87995:head# 2024-04-23T16:33:40.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1a1929:::benchmark_data_smithi188_113957_object125531:head# 2024-04-23T16:33:40.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1a36a6:::benchmark_data_smithi188_113957_object44951:head# 2024-04-23T16:33:40.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1a3bc7:::benchmark_data_smithi188_113957_object22920:head# 2024-04-23T16:33:40.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1a6d35:::benchmark_data_smithi188_113957_object101255:head# 2024-04-23T16:33:40.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1a80d6:::benchmark_data_smithi188_113957_object23665:head# 2024-04-23T16:33:40.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1ac2af:::benchmark_data_smithi188_113957_object44144:head# 2024-04-23T16:33:40.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1bdd3f:::benchmark_data_smithi188_113957_object16338:head# 2024-04-23T16:33:40.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1bf4b9:::benchmark_data_smithi188_113957_object48976:head# 2024-04-23T16:33:40.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1c0015:::benchmark_data_smithi188_113957_object76775:head# 2024-04-23T16:33:40.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1d599a:::benchmark_data_smithi188_113957_object14072:head# 2024-04-23T16:33:40.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1d675f:::benchmark_data_smithi188_113957_object108038:head# 2024-04-23T16:33:40.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1d8447:::benchmark_data_smithi188_113957_object130818:head# 2024-04-23T16:33:40.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1dba62:::benchmark_data_smithi188_113957_object28592:head# 2024-04-23T16:33:40.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1dc000:::benchmark_data_smithi188_113957_object15942:head# 2024-04-23T16:33:40.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1dc477:::benchmark_data_smithi188_113957_object54139:head# 2024-04-23T16:33:40.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1df49e:::benchmark_data_smithi188_113957_object67261:head# 2024-04-23T16:33:40.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1e00c0:::benchmark_data_smithi188_113957_object151356:head# 2024-04-23T16:33:40.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1e2b3f:::benchmark_data_smithi188_113957_object110480:head# 2024-04-23T16:33:40.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1e4d60:::benchmark_data_smithi188_113957_object13063:head# 2024-04-23T16:33:40.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1e7b08:::benchmark_data_smithi188_113957_object122487:head# 2024-04-23T16:33:40.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1f4122:::benchmark_data_smithi188_113957_object115293:head# 2024-04-23T16:33:40.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b1ffe5e:::benchmark_data_smithi188_113957_object103024:head# 2024-04-23T16:33:40.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b20327e:::benchmark_data_smithi188_113957_object141712:head# 2024-04-23T16:33:40.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b212252:::benchmark_data_smithi188_113957_object31884:head# 2024-04-23T16:33:40.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b218223:::benchmark_data_smithi188_113957_object73023:head# 2024-04-23T16:33:40.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2224e9:::benchmark_data_smithi188_113957_object152355:head# 2024-04-23T16:33:40.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b224e89:::benchmark_data_smithi188_113957_object21614:head# 2024-04-23T16:33:40.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b227ec7:::benchmark_data_smithi188_113957_object78415:head# 2024-04-23T16:33:40.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b229028:::benchmark_data_smithi188_113957_object125774:head# 2024-04-23T16:33:40.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b229412:::benchmark_data_smithi188_113957_object64897:head# 2024-04-23T16:33:40.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2382ee:::benchmark_data_smithi188_113957_object83607:head# 2024-04-23T16:33:40.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b23b540:::benchmark_data_smithi188_113957_object50736:head# 2024-04-23T16:33:40.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2437c7:::benchmark_data_smithi188_113957_object3151:head# 2024-04-23T16:33:40.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b24b7b3:::benchmark_data_smithi188_113957_object129444:head# 2024-04-23T16:33:40.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b256a2d:::benchmark_data_smithi188_113957_object134121:head# 2024-04-23T16:33:40.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b25984d:::benchmark_data_smithi188_113957_object44826:head# 2024-04-23T16:33:40.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b26746f:::benchmark_data_smithi188_113957_object49911:head# 2024-04-23T16:33:40.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b26f62f:::benchmark_data_smithi188_113957_object10174:head# 2024-04-23T16:33:40.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b271e9c:::benchmark_data_smithi188_113957_object101180:head# 2024-04-23T16:33:40.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b272c3c:::benchmark_data_smithi188_113957_object143681:head# 2024-04-23T16:33:40.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b286976:::benchmark_data_smithi188_113957_object110333:head# 2024-04-23T16:33:40.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2920af:::benchmark_data_smithi188_113957_object8526:head# 2024-04-23T16:33:40.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2a0f0d:::benchmark_data_smithi188_113957_object79226:head# 2024-04-23T16:33:40.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2a6a40:::benchmark_data_smithi188_113957_object44073:head# 2024-04-23T16:33:40.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2b193c:::benchmark_data_smithi188_113957_object31185:head# 2024-04-23T16:33:40.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2c6238:::benchmark_data_smithi188_113957_object85568:head# 2024-04-23T16:33:40.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2cce8d:::benchmark_data_smithi188_113957_object50345:head# 2024-04-23T16:33:40.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2d1ea1:::benchmark_data_smithi188_113957_object128723:head# 2024-04-23T16:33:40.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2d4a59:::benchmark_data_smithi188_113957_object99744:head# 2024-04-23T16:33:40.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2d9df3:::benchmark_data_smithi188_113957_object76650:head# 2024-04-23T16:33:40.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2de567:::benchmark_data_smithi188_113957_object3931:head# 2024-04-23T16:33:40.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2e29b3:::benchmark_data_smithi188_113957_object108817:head# 2024-04-23T16:33:40.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2ea644:::benchmark_data_smithi188_113957_object102166:head# 2024-04-23T16:33:40.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2eb0d2:::benchmark_data_smithi188_113957_object78792:head# 2024-04-23T16:33:40.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2ee149:::benchmark_data_smithi188_113957_object98560:head# 2024-04-23T16:33:40.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2eee0b:::benchmark_data_smithi188_113957_object137839:head# 2024-04-23T16:33:40.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2f0e3b:::benchmark_data_smithi188_113957_object75897:head# 2024-04-23T16:33:40.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2f501d:::benchmark_data_smithi188_113957_object11731:head# 2024-04-23T16:33:40.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2fbbdb:::benchmark_data_smithi188_113957_object46443:head# 2024-04-23T16:33:40.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b2fd879:::benchmark_data_smithi188_113957_object117207:head# 2024-04-23T16:33:40.814 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b30606c:::benchmark_data_smithi188_113957_object98781:head# 2024-04-23T16:33:40.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b30ea3b:::benchmark_data_smithi188_113957_object2777:head# 2024-04-23T16:33:40.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b31f308:::benchmark_data_smithi188_113957_object117897:head# 2024-04-23T16:33:40.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b325310:::benchmark_data_smithi188_113957_object688:head# 2024-04-23T16:33:40.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b3449e2:::benchmark_data_smithi188_113957_object50301:head# 2024-04-23T16:33:40.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b349024:::benchmark_data_smithi188_113957_object110447:head# 2024-04-23T16:33:40.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b34b0ec:::benchmark_data_smithi188_113957_object87652:head# 2024-04-23T16:33:40.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b3519ba:::benchmark_data_smithi188_113957_object111852:head# 2024-04-23T16:33:40.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b352960:::benchmark_data_smithi188_113957_object95122:head# 2024-04-23T16:33:40.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b354ca8:::benchmark_data_smithi188_113957_object25095:head# 2024-04-23T16:33:40.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b35c856:::benchmark_data_smithi188_113957_object14957:head# 2024-04-23T16:33:40.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b362679:::benchmark_data_smithi188_113957_object139027:head# 2024-04-23T16:33:40.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b377277:::benchmark_data_smithi188_113957_object89012:head# 2024-04-23T16:33:40.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b3854cd:::benchmark_data_smithi188_113957_object101628:head# 2024-04-23T16:33:40.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b395c76:::benchmark_data_smithi188_113957_object71778:head# 2024-04-23T16:33:40.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b396a93:::benchmark_data_smithi188_113957_object71228:head# 2024-04-23T16:33:40.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b398f3d:::benchmark_data_smithi188_113957_object71979:head# 2024-04-23T16:33:40.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b39d00d:::benchmark_data_smithi188_113957_object46424:head# 2024-04-23T16:33:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b3a13b7:::benchmark_data_smithi188_113957_object52478:head# 2024-04-23T16:33:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b3c35e7:::benchmark_data_smithi188_113957_object29557:head# 2024-04-23T16:33:40.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b3c7653:::benchmark_data_smithi188_113957_object30789:head# 2024-04-23T16:33:40.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b3c7b2d:::benchmark_data_smithi188_113957_object37258:head# 2024-04-23T16:33:40.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b3ca917:::benchmark_data_smithi188_113957_object133264:head# 2024-04-23T16:33:40.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b3cc15b:::benchmark_data_smithi188_113957_object129326:head# 2024-04-23T16:33:40.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b3d6f0f:::benchmark_data_smithi188_113957_object63051:head# 2024-04-23T16:33:40.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b3e89d5:::benchmark_data_smithi188_113957_object38611:head# 2024-04-23T16:33:40.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b3edb4d:::benchmark_data_smithi188_113957_object78145:head# 2024-04-23T16:33:40.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b3f79fa:::benchmark_data_smithi188_113957_object150451:head# 2024-04-23T16:33:40.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b4006f4:::benchmark_data_smithi188_113957_object117021:head# 2024-04-23T16:33:40.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b4071c3:::benchmark_data_smithi188_113957_object60359:head# 2024-04-23T16:33:40.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b40fabd:::benchmark_data_smithi188_113957_object90233:head# 2024-04-23T16:33:40.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b41edfd:::benchmark_data_smithi188_113957_object139819:head# 2024-04-23T16:33:40.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b4272ec:::benchmark_data_smithi188_113957_object114017:head# 2024-04-23T16:33:40.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b437965:::benchmark_data_smithi188_113957_object2610:head# 2024-04-23T16:33:40.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b457527:::benchmark_data_smithi188_113957_object130277:head# 2024-04-23T16:33:40.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b4578a2:::benchmark_data_smithi188_113957_object7844:head# 2024-04-23T16:33:40.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b46dfe8:::benchmark_data_smithi188_113957_object116190:head# 2024-04-23T16:33:40.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b4765df:::benchmark_data_smithi188_113957_object31939:head# 2024-04-23T16:33:40.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b482e40:::benchmark_data_smithi188_113957_object37946:head# 2024-04-23T16:33:40.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b48f104:::benchmark_data_smithi188_113957_object13955:head# 2024-04-23T16:33:40.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b493dbe:::benchmark_data_smithi188_113957_object7842:head# 2024-04-23T16:33:40.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b49d03b:::benchmark_data_smithi188_113957_object124356:head# 2024-04-23T16:33:40.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b4a1327:::benchmark_data_smithi188_113957_object118860:head# 2024-04-23T16:33:40.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b4a2494:::benchmark_data_smithi188_113957_object136013:head# 2024-04-23T16:33:40.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b4a9be9:::benchmark_data_smithi188_113957_object63514:head# 2024-04-23T16:33:40.845 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b4aab66:::benchmark_data_smithi188_113957_object4095:head# 2024-04-23T16:33:40.845 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b4ac600:::benchmark_data_smithi188_113957_object92177:head# 2024-04-23T16:33:40.845 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b4b3acb:::benchmark_data_smithi188_113957_object23177:head# 2024-04-23T16:33:40.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b4eb297:::benchmark_data_smithi188_113957_object116831:head# 2024-04-23T16:33:40.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b4ef5e0:::benchmark_data_smithi188_113957_object4739:head# 2024-04-23T16:33:40.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b4fb85d:::benchmark_data_smithi188_113957_object27400:head# 2024-04-23T16:33:40.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b4ff347:::benchmark_data_smithi188_113957_object20764:head# 2024-04-23T16:33:40.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b502851:::benchmark_data_smithi188_113957_object121619:head# 2024-04-23T16:33:40.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b50d897:::benchmark_data_smithi188_113957_object37894:head# 2024-04-23T16:33:40.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b50e329:::benchmark_data_smithi188_113957_object85436:head# 2024-04-23T16:33:40.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b528d72:::benchmark_data_smithi188_113957_object61264:head# 2024-04-23T16:33:40.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b52d0ca:::benchmark_data_smithi188_113957_object113162:head# 2024-04-23T16:33:40.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b538de0:::benchmark_data_smithi188_113957_object17827:head# 2024-04-23T16:33:40.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b539989:::benchmark_data_smithi188_113957_object150552:head# 2024-04-23T16:33:40.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b53c745:::benchmark_data_smithi188_113957_object70311:head# 2024-04-23T16:33:40.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b53e569:::benchmark_data_smithi188_113957_object25985:head# 2024-04-23T16:33:40.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b53ebf5:::benchmark_data_smithi188_113957_object82200:head# 2024-04-23T16:33:40.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b54f9d4:::benchmark_data_smithi188_113957_object33140:head# 2024-04-23T16:33:40.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b55055a:::benchmark_data_smithi188_113957_object87461:head# 2024-04-23T16:33:40.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b555d82:::benchmark_data_smithi188_113957_object97484:head# 2024-04-23T16:33:40.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b55d2c4:::benchmark_data_smithi188_113957_object150045:head# 2024-04-23T16:33:40.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b55d934:::benchmark_data_smithi188_113957_object119138:head# 2024-04-23T16:33:40.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b55f89a:::benchmark_data_smithi188_113957_object138279:head# 2024-04-23T16:33:40.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b561ca0:::benchmark_data_smithi188_113957_object82510:head# 2024-04-23T16:33:40.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b565953:::benchmark_data_smithi188_113957_object34207:head# 2024-04-23T16:33:40.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b566508:::benchmark_data_smithi188_113957_object114011:head# 2024-04-23T16:33:40.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b567a9d:::benchmark_data_smithi188_113957_object98772:head# 2024-04-23T16:33:40.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b56b942:::benchmark_data_smithi188_113957_object45684:head# 2024-04-23T16:33:40.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b576382:::benchmark_data_smithi188_113957_object63001:head# 2024-04-23T16:33:40.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b5784e7:::benchmark_data_smithi188_113957_object37855:head# 2024-04-23T16:33:40.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b57bccc:::benchmark_data_smithi188_113957_object63310:head# 2024-04-23T16:33:40.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b57c712:::benchmark_data_smithi188_113957_object44111:head# 2024-04-23T16:33:40.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b57eb46:::benchmark_data_smithi188_113957_object57614:head# 2024-04-23T16:33:40.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b58320d:::benchmark_data_smithi188_113957_object121444:head# 2024-04-23T16:33:40.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b58e5f2:::benchmark_data_smithi188_113957_object140903:head# 2024-04-23T16:33:40.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b58e960:::benchmark_data_smithi188_113957_object40983:head# 2024-04-23T16:33:40.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b594050:::benchmark_data_smithi188_113957_object28578:head# 2024-04-23T16:33:40.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b59507c:::benchmark_data_smithi188_113957_object2616:head# 2024-04-23T16:33:40.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b596641:::benchmark_data_smithi188_113957_object87207:head# 2024-04-23T16:33:40.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b5975ae:::benchmark_data_smithi188_113957_object126460:head# 2024-04-23T16:33:40.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b5976a6:::benchmark_data_smithi188_113957_object22746:head# 2024-04-23T16:33:40.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b597cc7:::benchmark_data_smithi188_113957_object46061:head# 2024-04-23T16:33:40.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b59a217:::benchmark_data_smithi188_113957_object30867:head# 2024-04-23T16:33:40.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b5a12f4:::benchmark_data_smithi188_113957_object85379:head# 2024-04-23T16:33:40.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b5a4be6:::benchmark_data_smithi188_113957_object96287:head# 2024-04-23T16:33:40.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b5a8549:::benchmark_data_smithi188_113957_object126422:head# 2024-04-23T16:33:40.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b5abfe6:::benchmark_data_smithi188_113957_object112360:head# 2024-04-23T16:33:40.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b5ae2dc:::benchmark_data_smithi188_113957_object60928:head# 2024-04-23T16:33:40.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b5c0889:::benchmark_data_smithi188_113957_object64392:head# 2024-04-23T16:33:40.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b5c3453:::benchmark_data_smithi188_113957_object144331:head# 2024-04-23T16:33:40.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b5c9c9b:::benchmark_data_smithi188_113957_object36578:head# 2024-04-23T16:33:40.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b5c9f18:::benchmark_data_smithi188_113957_object87275:head# 2024-04-23T16:33:40.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b5cbfa2:::benchmark_data_smithi188_113957_object37588:head# 2024-04-23T16:33:40.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b5cef7e:::benchmark_data_smithi188_113957_object51971:head# 2024-04-23T16:33:40.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b5d674c:::benchmark_data_smithi188_113957_object136727:head# 2024-04-23T16:33:40.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b5f5ead:::benchmark_data_smithi188_113957_object89929:head# 2024-04-23T16:33:40.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b5f6110:::benchmark_data_smithi188_113957_object98719:head# 2024-04-23T16:33:40.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b5f76c9:::benchmark_data_smithi188_113957_object60175:head# 2024-04-23T16:33:40.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b5f8f23:::benchmark_data_smithi188_113957_object78686:head# 2024-04-23T16:33:40.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b6083cf:::benchmark_data_smithi188_113957_object28513:head# 2024-04-23T16:33:40.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b60bafb:::benchmark_data_smithi188_113957_object92593:head# 2024-04-23T16:33:40.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b61587b:::benchmark_data_smithi188_113957_object59502:head# 2024-04-23T16:33:40.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b6209ce:::benchmark_data_smithi188_113957_object144198:head# 2024-04-23T16:33:40.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b64755b:::benchmark_data_smithi188_113957_object12027:head# 2024-04-23T16:33:40.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b65288a:::benchmark_data_smithi188_113957_object111786:head# 2024-04-23T16:33:40.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b65c405:::benchmark_data_smithi188_113957_object114433:head# 2024-04-23T16:33:40.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b663d61:::benchmark_data_smithi188_113957_object62473:head# 2024-04-23T16:33:40.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b66407b:::benchmark_data_smithi188_113957_object88247:head# 2024-04-23T16:33:40.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b667ea3:::benchmark_data_smithi188_113957_object92134:head# 2024-04-23T16:33:40.875 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b679da0:::benchmark_data_smithi188_113957_object17102:head# 2024-04-23T16:33:40.875 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b682613:::benchmark_data_smithi188_113957_object88953:head# 2024-04-23T16:33:40.876 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b682d76:::benchmark_data_smithi188_113957_object81973:head# 2024-04-23T16:33:40.876 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b68b22b:::benchmark_data_smithi188_113957_object10880:head# 2024-04-23T16:33:40.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b691151:::benchmark_data_smithi188_113957_object8069:head# 2024-04-23T16:33:40.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b6957cc:::benchmark_data_smithi188_113957_object113840:head# 2024-04-23T16:33:40.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b6a4968:::benchmark_data_smithi188_113957_object76577:head# 2024-04-23T16:33:40.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b6a5e14:::benchmark_data_smithi188_113957_object134639:head# 2024-04-23T16:33:40.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b6ab884:::benchmark_data_smithi188_113957_object67953:head# 2024-04-23T16:33:40.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b6adf4f:::benchmark_data_smithi188_113957_object32039:head# 2024-04-23T16:33:40.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b6b8e81:::benchmark_data_smithi188_113957_object32836:head# 2024-04-23T16:33:40.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b6ba975:::benchmark_data_smithi188_113957_object117461:head# 2024-04-23T16:33:40.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b6bc03d:::benchmark_data_smithi188_113957_object88038:head# 2024-04-23T16:33:40.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b6e05ea:::benchmark_data_smithi188_113957_object137807:head# 2024-04-23T16:33:40.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b6e445a:::benchmark_data_smithi188_113957_object22265:head# 2024-04-23T16:33:40.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b6ee6a5:::benchmark_data_smithi188_113957_object68056:head# 2024-04-23T16:33:40.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b6f3384:::benchmark_data_smithi188_113957_object151931:head# 2024-04-23T16:33:40.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b6f4ea5:::benchmark_data_smithi188_113957_object115162:head# 2024-04-23T16:33:40.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b6f6d54:::benchmark_data_smithi188_113957_object56501:head# 2024-04-23T16:33:40.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b6fd7c3:::benchmark_data_smithi188_113957_object26631:head# 2024-04-23T16:33:40.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b6fe3b8:::benchmark_data_smithi188_113957_object123762:head# 2024-04-23T16:33:40.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b700ee1:::benchmark_data_smithi188_113957_object104622:head# 2024-04-23T16:33:40.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b707ada:::benchmark_data_smithi188_113957_object54297:head# 2024-04-23T16:33:40.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b70a52e:::benchmark_data_smithi188_113957_object148563:head# 2024-04-23T16:33:40.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b70b709:::benchmark_data_smithi188_113957_object74872:head# 2024-04-23T16:33:40.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b70e238:::benchmark_data_smithi188_113957_object52227:head# 2024-04-23T16:33:40.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b7113ab:::benchmark_data_smithi188_113957_object142289:head# 2024-04-23T16:33:40.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b71462e:::benchmark_data_smithi188_113957_object133084:head# 2024-04-23T16:33:40.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b7166ec:::benchmark_data_smithi188_113957_object11509:head# 2024-04-23T16:33:40.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b71e9eb:::benchmark_data_smithi188_113957_object121098:head# 2024-04-23T16:33:40.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b7202c8:::benchmark_data_smithi188_113957_object18062:head# 2024-04-23T16:33:40.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b727288:::benchmark_data_smithi188_113957_object64677:head# 2024-04-23T16:33:40.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b72e7ec:::benchmark_data_smithi188_113957_object65470:head# 2024-04-23T16:33:40.891 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b738ef6:::benchmark_data_smithi188_113957_object18233:head# 2024-04-23T16:33:40.891 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b753e6b:::benchmark_data_smithi188_113957_object136659:head# 2024-04-23T16:33:40.892 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b75471d:::benchmark_data_smithi188_113957_object121393:head# 2024-04-23T16:33:40.892 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b761d6b:::benchmark_data_smithi188_113957_object96425:head# 2024-04-23T16:33:40.892 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b76820a:::benchmark_data_smithi188_113957_object14481:head# 2024-04-23T16:33:40.893 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b768568:::benchmark_data_smithi188_113957_object120079:head# 2024-04-23T16:33:40.893 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b76de63:::benchmark_data_smithi188_113957_object64556:head# 2024-04-23T16:33:40.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b775c6e:::benchmark_data_smithi188_113957_object83090:head# 2024-04-23T16:33:40.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b77dff5:::benchmark_data_smithi188_113957_object80765:head# 2024-04-23T16:33:40.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b781701:::benchmark_data_smithi188_113957_object143481:head# 2024-04-23T16:33:40.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b7850fb:::benchmark_data_smithi188_113957_object137529:head# 2024-04-23T16:33:40.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b78a450:::benchmark_data_smithi188_113957_object84844:head# 2024-04-23T16:33:40.896 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b78c09a:::benchmark_data_smithi188_113957_object6495:head# 2024-04-23T16:33:40.896 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b79c6ac:::benchmark_data_smithi188_113957_object115587:head# 2024-04-23T16:33:40.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b79ff47:::benchmark_data_smithi188_113957_object143149:head# 2024-04-23T16:33:40.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b7a1508:::benchmark_data_smithi188_113957_object120957:head# 2024-04-23T16:33:40.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b7a25b9:::benchmark_data_smithi188_113957_object149999:head# 2024-04-23T16:33:40.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b7a58a1:::benchmark_data_smithi188_113957_object36:head# 2024-04-23T16:33:40.899 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b7a7e1e:::benchmark_data_smithi188_113957_object145562:head# 2024-04-23T16:33:40.899 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b7ab97b:::benchmark_data_smithi188_113957_object80324:head# 2024-04-23T16:33:40.900 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b7b6b95:::benchmark_data_smithi188_113957_object90990:head# 2024-04-23T16:33:40.900 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b7bf007:::benchmark_data_smithi188_113957_object46431:head# 2024-04-23T16:33:40.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b7dc8a3:::benchmark_data_smithi188_113957_object92452:head# 2024-04-23T16:33:40.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b7e100a:::benchmark_data_smithi188_113957_object17623:head# 2024-04-23T16:33:40.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b7f8f06:::benchmark_data_smithi188_113957_object143996:head# 2024-04-23T16:33:40.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b8031d2:::benchmark_data_smithi188_113957_object150582:head# 2024-04-23T16:33:40.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b817dfc:::benchmark_data_smithi188_113957_object105930:head# 2024-04-23T16:33:40.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b81853d:::benchmark_data_smithi188_113957_object147540:head# 2024-04-23T16:33:40.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b81ab6f:::benchmark_data_smithi188_113957_object113582:head# 2024-04-23T16:33:40.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b81f5c0:::benchmark_data_smithi188_113957_object83342:head# 2024-04-23T16:33:40.904 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b820541:::benchmark_data_smithi188_113957_object34511:head# 2024-04-23T16:33:40.904 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b82b24d:::benchmark_data_smithi188_113957_object39373:head# 2024-04-23T16:33:40.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b8370f1:::benchmark_data_smithi188_113957_object88343:head# 2024-04-23T16:33:40.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b846068:::benchmark_data_smithi188_113957_object40387:head# 2024-04-23T16:33:40.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b8548fb:::benchmark_data_smithi188_113957_object115532:head# 2024-04-23T16:33:40.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b861667:::benchmark_data_smithi188_113957_object34182:head# 2024-04-23T16:33:40.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b863520:::benchmark_data_smithi188_113957_object72703:head# 2024-04-23T16:33:40.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b86788b:::benchmark_data_smithi188_113957_object109475:head# 2024-04-23T16:33:40.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b867975:::benchmark_data_smithi188_113957_object17856:head# 2024-04-23T16:33:40.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b86b93b:::benchmark_data_smithi188_113957_object59458:head# 2024-04-23T16:33:40.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b86cf96:::benchmark_data_smithi188_113957_object71544:head# 2024-04-23T16:33:40.909 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b877e9b:::benchmark_data_smithi188_113957_object72304:head# 2024-04-23T16:33:40.909 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b87ac1a:::benchmark_data_smithi188_113957_object77079:head# 2024-04-23T16:33:40.910 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b87de36:::benchmark_data_smithi188_113957_object117050:head# 2024-04-23T16:33:40.910 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b87f5a6:::benchmark_data_smithi188_113957_object78820:head# 2024-04-23T16:33:40.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b88c81e:::benchmark_data_smithi188_113957_object19563:head# 2024-04-23T16:33:40.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b88e61a:::benchmark_data_smithi188_113957_object127183:head# 2024-04-23T16:33:40.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b89dfc3:::benchmark_data_smithi188_113957_object23863:head# 2024-04-23T16:33:40.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b89e424:::benchmark_data_smithi188_113957_object65994:head# 2024-04-23T16:33:40.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b8b0503:::benchmark_data_smithi188_113957_object98596:head# 2024-04-23T16:33:40.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b8b4663:::benchmark_data_smithi188_113957_object5482:head# 2024-04-23T16:33:40.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b8b5b1d:::benchmark_data_smithi188_113957_object22701:head# 2024-04-23T16:33:40.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b8bc5ab:::benchmark_data_smithi188_113957_object100929:head# 2024-04-23T16:33:40.914 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b8beecf:::benchmark_data_smithi188_113957_object30080:head# 2024-04-23T16:33:40.914 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b8c408d:::benchmark_data_smithi188_113957_object42285:head# 2024-04-23T16:33:40.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b8c6390:::benchmark_data_smithi188_113957_object82717:head# 2024-04-23T16:33:40.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b8cc335:::benchmark_data_smithi188_113957_object142047:head# 2024-04-23T16:33:40.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b8d88a0:::benchmark_data_smithi188_113957_object60658:head# 2024-04-23T16:33:40.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b8db770:::benchmark_data_smithi188_113957_object5124:head# 2024-04-23T16:33:40.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b8e3bc3:::benchmark_data_smithi188_113957_object32632:head# 2024-04-23T16:33:40.917 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b908d53:::benchmark_data_smithi188_113957_object8245:head# 2024-04-23T16:33:40.917 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b9245aa:::benchmark_data_smithi188_113957_object123027:head# 2024-04-23T16:33:40.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b927e76:::benchmark_data_smithi188_113957_object38728:head# 2024-04-23T16:33:40.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b929d0f:::benchmark_data_smithi188_113957_object121003:head# 2024-04-23T16:33:40.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b92b927:::benchmark_data_smithi188_113957_object90460:head# 2024-04-23T16:33:40.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b92d17a:::benchmark_data_smithi188_113957_object147626:head# 2024-04-23T16:33:40.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b932fbe:::benchmark_data_smithi188_113957_object121414:head# 2024-04-23T16:33:40.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b93e791:::benchmark_data_smithi188_113957_object25765:head# 2024-04-23T16:33:40.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b942351:::benchmark_data_smithi188_113957_object142100:head# 2024-04-23T16:33:40.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b9496db:::benchmark_data_smithi188_113957_object54803:head# 2024-04-23T16:33:40.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b94b971:::benchmark_data_smithi188_113957_object63212:head# 2024-04-23T16:33:40.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b94d5ea:::benchmark_data_smithi188_113957_object6456:head# 2024-04-23T16:33:40.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b96e553:::benchmark_data_smithi188_113957_object79368:head# 2024-04-23T16:33:40.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b9733fe:::benchmark_data_smithi188_113957_object150602:head# 2024-04-23T16:33:40.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b97dc2f:::benchmark_data_smithi188_113957_object131104:head# 2024-04-23T16:33:40.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b982573:::benchmark_data_smithi188_113957_object9639:head# 2024-04-23T16:33:40.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b98a656:::benchmark_data_smithi188_113957_object20595:head# 2024-04-23T16:33:40.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b993ccb:::benchmark_data_smithi188_113957_object23791:head# 2024-04-23T16:33:40.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b99b6fb:::benchmark_data_smithi188_113957_object84807:head# 2024-04-23T16:33:40.925 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b99bac9:::benchmark_data_smithi188_113957_object21309:head# 2024-04-23T16:33:40.925 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b99c9ea:::benchmark_data_smithi188_113957_object44187:head# 2024-04-23T16:33:40.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b9a0aeb:::benchmark_data_smithi188_113957_object143519:head# 2024-04-23T16:33:40.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b9a315b:::benchmark_data_smithi188_113957_object68994:head# 2024-04-23T16:33:40.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b9a835a:::benchmark_data_smithi188_113957_object144577:head# 2024-04-23T16:33:40.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b9af8f9:::benchmark_data_smithi188_113957_object66163:head# 2024-04-23T16:33:40.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b9c917c:::benchmark_data_smithi188_113957_object85446:head# 2024-04-23T16:33:40.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b9cb4d1:::benchmark_data_smithi188_113957_object74461:head# 2024-04-23T16:33:40.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b9cd60b:::benchmark_data_smithi188_113957_object11608:head# 2024-04-23T16:33:40.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b9d5226:::benchmark_data_smithi188_113957_object141015:head# 2024-04-23T16:33:40.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b9dae10:::benchmark_data_smithi188_113957_object30219:head# 2024-04-23T16:33:40.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b9ddb42:::benchmark_data_smithi188_113957_object120788:head# 2024-04-23T16:33:40.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b9e18da:::benchmark_data_smithi188_113957_object116141:head# 2024-04-23T16:33:40.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b9e288a:::benchmark_data_smithi188_113957_object57705:head# 2024-04-23T16:33:40.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b9e8f2e:::benchmark_data_smithi188_113957_object40170:head# 2024-04-23T16:33:40.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b9e9ca1:::benchmark_data_smithi188_113957_object149027:head# 2024-04-23T16:33:40.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5b9f44a3:::benchmark_data_smithi188_113957_object120367:head# 2024-04-23T16:33:40.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba02414:::benchmark_data_smithi188_113957_object47253:head# 2024-04-23T16:33:40.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba029c7:::benchmark_data_smithi188_113957_object150832:head# 2024-04-23T16:33:40.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba0e897:::benchmark_data_smithi188_113957_object114117:head# 2024-04-23T16:33:40.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba0ff39:::benchmark_data_smithi188_113957_object152665:head# 2024-04-23T16:33:40.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba2dd4c:::benchmark_data_smithi188_113957_object84835:head# 2024-04-23T16:33:40.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba3b289:::benchmark_data_smithi188_113957_object81166:head# 2024-04-23T16:33:40.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba3bdc7:::benchmark_data_smithi188_113957_object33751:head# 2024-04-23T16:33:40.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba3c0b2:::benchmark_data_smithi188_113957_object131274:head# 2024-04-23T16:33:40.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba3c436:::benchmark_data_smithi188_113957_object105714:head# 2024-04-23T16:33:40.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba3cb22:::benchmark_data_smithi188_113957_object69119:head# 2024-04-23T16:33:40.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba517f4:::benchmark_data_smithi188_113957_object13438:head# 2024-04-23T16:33:40.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba54f3e:::benchmark_data_smithi188_113957_object12384:head# 2024-04-23T16:33:40.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba58a65:::benchmark_data_smithi188_113957_object80698:head# 2024-04-23T16:33:40.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba5a05d:::benchmark_data_smithi188_113957_object48848:head# 2024-04-23T16:33:40.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba62952:::benchmark_data_smithi188_113957_object43152:head# 2024-04-23T16:33:40.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba6547c:::benchmark_data_smithi188_113957_object133642:head# 2024-04-23T16:33:40.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba715bb:::benchmark_data_smithi188_113957_object87970:head# 2024-04-23T16:33:40.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba74b60:::benchmark_data_smithi188_113957_object123398:head# 2024-04-23T16:33:40.939 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba7c638:::benchmark_data_smithi188_113957_object77174:head# 2024-04-23T16:33:40.939 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba826ab:::benchmark_data_smithi188_113957_object23014:head# 2024-04-23T16:33:40.939 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba883e4:::benchmark_data_smithi188_113957_object6446:head# 2024-04-23T16:33:40.940 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba91335:::benchmark_data_smithi188_113957_object98700:head# 2024-04-23T16:33:40.940 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba9b40b:::benchmark_data_smithi188_113957_object140584:head# 2024-04-23T16:33:40.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba9d6f2:::benchmark_data_smithi188_113957_object38204:head# 2024-04-23T16:33:40.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba9de21:::benchmark_data_smithi188_113957_object63592:head# 2024-04-23T16:33:40.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ba9e303:::benchmark_data_smithi188_113957_object14159:head# 2024-04-23T16:33:40.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5baa4087:::benchmark_data_smithi188_113957_object101686:head# 2024-04-23T16:33:40.942 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:33:40.943 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:33:40.943 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5baa7a95:::benchmark_data_smithi188_113957_object82479:head# 2024-04-23T16:33:40.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bab14b6:::benchmark_data_smithi188_113957_object15099:head# 2024-04-23T16:33:40.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bab623c:::benchmark_data_smithi188_113957_object129250:head# 2024-04-23T16:33:40.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bac1d46:::benchmark_data_smithi188_113957_object61313:head# 2024-04-23T16:33:40.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bac9398:::benchmark_data_smithi188_113957_object110568:head# 2024-04-23T16:33:40.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bacfbd0:::benchmark_data_smithi188_113957_object49019:head# 2024-04-23T16:33:40.945 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5badbf32:::benchmark_data_smithi188_113957_object44875:head# 2024-04-23T16:33:40.945 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bade4ae:::benchmark_data_smithi188_113957_object984:head# 2024-04-23T16:33:40.946 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bae4e1d:::benchmark_data_smithi188_113957_object78725:head# 2024-04-23T16:33:40.946 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bae6fac:::benchmark_data_smithi188_113957_object121055:head# 2024-04-23T16:33:40.946 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5baedc78:::benchmark_data_smithi188_113957_object131344:head# 2024-04-23T16:33:40.947 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bafb1a0:::benchmark_data_smithi188_113957_object51582:head# 2024-04-23T16:33:40.947 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5baff623:::benchmark_data_smithi188_113957_object5597:head# 2024-04-23T16:33:40.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb00ae9:::benchmark_data_smithi188_113957_object75400:head# 2024-04-23T16:33:40.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb29c8a:::benchmark_data_smithi188_113957_object137909:head# 2024-04-23T16:33:40.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb2fa9c:::benchmark_data_smithi188_113957_object33491:head# 2024-04-23T16:33:40.949 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb37d4f:::benchmark_data_smithi188_113957_object139346:head# 2024-04-23T16:33:40.949 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb394ae:::benchmark_data_smithi188_113957_object43291:head# 2024-04-23T16:33:40.949 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb3cb27:::benchmark_data_smithi188_113957_object56914:head# 2024-04-23T16:33:40.950 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb462bf:::benchmark_data_smithi188_113957_object32767:head# 2024-04-23T16:33:40.950 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb4e2ee:::benchmark_data_smithi188_113957_object115782:head# 2024-04-23T16:33:40.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb4f039:::benchmark_data_smithi188_113957_object87116:head# 2024-04-23T16:33:40.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb4fa65:::benchmark_data_smithi188_113957_object58921:head# 2024-04-23T16:33:40.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb5566e:::benchmark_data_smithi188_113957_object34354:head# 2024-04-23T16:33:40.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb5b191:::benchmark_data_smithi188_113957_object43082:head# 2024-04-23T16:33:40.953 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb60271:::benchmark_data_smithi188_113957_object20955:head# 2024-04-23T16:33:40.953 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb61f6a:::benchmark_data_smithi188_113957_object129075:head# 2024-04-23T16:33:40.953 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb6340a:::benchmark_data_smithi188_113957_object35204:head# 2024-04-23T16:33:40.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb661c6:::benchmark_data_smithi188_113957_object107353:head# 2024-04-23T16:33:40.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb6e699:::benchmark_data_smithi188_113957_object142920:head# 2024-04-23T16:33:40.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb755d4:::benchmark_data_smithi188_113957_object73360:head# 2024-04-23T16:33:40.955 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb8234c:::benchmark_data_smithi188_113957_object100302:head# 2024-04-23T16:33:40.956 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb85fe5:::benchmark_data_smithi188_113957_object93032:head# 2024-04-23T16:33:40.956 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb8eb05:::benchmark_data_smithi188_113957_object105911:head# 2024-04-23T16:33:40.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb93448:::benchmark_data_smithi188_113957_object117759:head# 2024-04-23T16:33:40.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb9b9db:::benchmark_data_smithi188_113957_object24760:head# 2024-04-23T16:33:40.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb9c580:::benchmark_data_smithi188_113957_object7043:head# 2024-04-23T16:33:40.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bb9dd2c:::benchmark_data_smithi188_113957_object90853:head# 2024-04-23T16:33:40.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bbb236a:::benchmark_data_smithi188_113957_object60308:head# 2024-04-23T16:33:40.959 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bbb8be3:::benchmark_data_smithi188_113957_object96410:head# 2024-04-23T16:33:40.959 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bbb9bd1:::benchmark_data_smithi188_113957_object47275:head# 2024-04-23T16:33:40.960 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bbbe0fb:::benchmark_data_smithi188_113957_object120698:head# 2024-04-23T16:33:40.960 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bbbebfa:::benchmark_data_smithi188_113957_object70974:head# 2024-04-23T16:33:40.960 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bbbf18e:::benchmark_data_smithi188_113957_object64626:head# 2024-04-23T16:33:40.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bbc69a9:::benchmark_data_smithi188_113957_object82095:head# 2024-04-23T16:33:40.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bbcff65:::benchmark_data_smithi188_113957_object20876:head# 2024-04-23T16:33:40.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bbd7997:::benchmark_data_smithi188_113957_object41964:head# 2024-04-23T16:33:40.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bbe19db:::benchmark_data_smithi188_113957_object84669:head# 2024-04-23T16:33:40.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bbe6c35:::benchmark_data_smithi188_113957_object86696:head# 2024-04-23T16:33:40.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bbe7f46:::benchmark_data_smithi188_113957_object96880:head# 2024-04-23T16:33:40.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bbee640:::benchmark_data_smithi188_113957_object136959:head# 2024-04-23T16:33:40.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bbf2bd2:::benchmark_data_smithi188_113957_object133287:head# 2024-04-23T16:33:40.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bbf602a:::benchmark_data_smithi188_113957_object85584:head# 2024-04-23T16:33:40.965 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bbf7087:::benchmark_data_smithi188_113957_object49458:head# 2024-04-23T16:33:40.965 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bbf79f3:::benchmark_data_smithi188_113957_object103171:head# 2024-04-23T16:33:40.965 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bbfd879:::benchmark_data_smithi188_113957_object145201:head# 2024-04-23T16:33:40.966 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc172c0:::benchmark_data_smithi188_113957_object66431:head# 2024-04-23T16:33:40.967 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc1ec4a:::benchmark_data_smithi188_113957_object62140:head# 2024-04-23T16:33:40.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc1f28a:::benchmark_data_smithi188_113957_object103004:head# 2024-04-23T16:33:40.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc2180f:::benchmark_data_smithi188_113957_object140686:head# 2024-04-23T16:33:40.969 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc2200e:::benchmark_data_smithi188_113957_object117598:head# 2024-04-23T16:33:40.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc26412:::benchmark_data_smithi188_113957_object57806:head# 2024-04-23T16:33:40.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc26570:::benchmark_data_smithi188_113957_object148022:head# 2024-04-23T16:33:40.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc29c2a:::benchmark_data_smithi188_113957_object42211:head# 2024-04-23T16:33:40.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc2e9f9:::benchmark_data_smithi188_113957_object9020:head# 2024-04-23T16:33:40.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc2ea73:::benchmark_data_smithi188_113957_object59584:head# 2024-04-23T16:33:40.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc33d5c:::benchmark_data_smithi188_113957_object48431:head# 2024-04-23T16:33:40.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc37ec0:::benchmark_data_smithi188_113957_object42882:head# 2024-04-23T16:33:40.973 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc3c112:::benchmark_data_smithi188_113957_object93850:head# 2024-04-23T16:33:40.973 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc3e842:::benchmark_data_smithi188_113957_object64573:head# 2024-04-23T16:33:40.974 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc3fa0a:::benchmark_data_smithi188_113957_object101224:head# 2024-04-23T16:33:40.974 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc42e61:::benchmark_data_smithi188_113957_object82422:head# 2024-04-23T16:33:40.974 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc64a79:::benchmark_data_smithi188_113957_object65501:head# 2024-04-23T16:33:40.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc6df9c:::benchmark_data_smithi188_113957_object152705:head# 2024-04-23T16:33:40.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc6fb47:::benchmark_data_smithi188_113957_object63906:head# 2024-04-23T16:33:40.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc779b6:::benchmark_data_smithi188_113957_object76048:head# 2024-04-23T16:33:40.977 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc7b638:::benchmark_data_smithi188_113957_object131553:head# 2024-04-23T16:33:40.977 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc847d4:::benchmark_data_smithi188_113957_object22555:head# 2024-04-23T16:33:40.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc886d5:::benchmark_data_smithi188_113957_object116785:head# 2024-04-23T16:33:40.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc8c1bb:::benchmark_data_smithi188_113957_object14374:head# 2024-04-23T16:33:40.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc93623:::benchmark_data_smithi188_113957_object100608:head# 2024-04-23T16:33:40.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc9eb6a:::benchmark_data_smithi188_113957_object18210:head# 2024-04-23T16:33:40.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bc9ec8c:::benchmark_data_smithi188_113957_object124475:head# 2024-04-23T16:33:40.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bca44ba:::benchmark_data_smithi188_113957_object32483:head# 2024-04-23T16:33:40.980 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bcb095c:::benchmark_data_smithi188_113957_object30604:head# 2024-04-23T16:33:40.980 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bcb2258:::benchmark_data_smithi188_113957_object4030:head# 2024-04-23T16:33:40.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bcb63ca:::benchmark_data_smithi188_113957_object28992:head# 2024-04-23T16:33:40.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bcb8602:::benchmark_data_smithi188_113957_object51929:head# 2024-04-23T16:33:40.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bcba0cd:::benchmark_data_smithi188_113957_object15842:head# 2024-04-23T16:33:40.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bcba6ca:::benchmark_data_smithi188_113957_object129179:head# 2024-04-23T16:33:40.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bcbb633:::benchmark_data_smithi188_113957_object131042:head# 2024-04-23T16:33:40.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bcc5309:::benchmark_data_smithi188_113957_object139984:head# 2024-04-23T16:33:40.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bcce2a4:::benchmark_data_smithi188_113957_object43401:head# 2024-04-23T16:33:40.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bcd2dd5:::benchmark_data_smithi188_113957_object16706:head# 2024-04-23T16:33:40.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bcd8509:::benchmark_data_smithi188_113957_object102636:head# 2024-04-23T16:33:40.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bcd999e:::benchmark_data_smithi188_113957_object143459:head# 2024-04-23T16:33:40.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bce1d1c:::benchmark_data_smithi188_113957_object13657:head# 2024-04-23T16:33:40.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bcf0cbc:::benchmark_data_smithi188_113957_object90380:head# 2024-04-23T16:33:40.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bcf473c:::benchmark_data_smithi188_113957_object146823:head# 2024-04-23T16:33:40.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bcfa70e:::benchmark_data_smithi188_113957_object91013:head# 2024-04-23T16:33:40.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd018a2:::benchmark_data_smithi188_113957_object6995:head# 2024-04-23T16:33:40.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd02c54:::benchmark_data_smithi188_113957_object12695:head# 2024-04-23T16:33:40.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd08377:::benchmark_data_smithi188_113957_object69046:head# 2024-04-23T16:33:40.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd0908f:::benchmark_data_smithi188_113957_object133610:head# 2024-04-23T16:33:40.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd0e266:::benchmark_data_smithi188_113957_object114843:head# 2024-04-23T16:33:40.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd13b7a:::benchmark_data_smithi188_113957_object80063:head# 2024-04-23T16:33:40.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd16d7b:::benchmark_data_smithi188_113957_object23335:head# 2024-04-23T16:33:40.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd19c0d:::benchmark_data_smithi188_113957_object82496:head# 2024-04-23T16:33:40.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd2afbc:::benchmark_data_smithi188_113957_object78175:head# 2024-04-23T16:33:40.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd341c6:::benchmark_data_smithi188_113957_object145155:head# 2024-04-23T16:33:40.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd37f54:::benchmark_data_smithi188_113957_object65067:head# 2024-04-23T16:33:40.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd38348:::benchmark_data_smithi188_113957_object89895:head# 2024-04-23T16:33:40.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd3b09b:::benchmark_data_smithi188_113957_object113592:head# 2024-04-23T16:33:40.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd3e9e6:::benchmark_data_smithi188_113957_object115998:head# 2024-04-23T16:33:40.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd48dfd:::benchmark_data_smithi188_113957_object32080:head# 2024-04-23T16:33:40.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd4f159:::benchmark_data_smithi188_113957_object89713:head# 2024-04-23T16:33:40.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd4f48a:::benchmark_data_smithi188_113957_object117855:head# 2024-04-23T16:33:40.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd4f5ed:::benchmark_data_smithi188_113957_object23583:head# 2024-04-23T16:33:40.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd5a7b7:::benchmark_data_smithi188_113957_object30429:head# 2024-04-23T16:33:40.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd8caa3:::benchmark_data_smithi188_113957_object48298:head# 2024-04-23T16:33:40.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd8cfd3:::benchmark_data_smithi188_113957_object103615:head# 2024-04-23T16:33:40.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd8ffce:::benchmark_data_smithi188_113957_object50442:head# 2024-04-23T16:33:40.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd9550d:::benchmark_data_smithi188_113957_object8165:head# 2024-04-23T16:33:40.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bd9fcb1:::benchmark_data_smithi188_113957_object34668:head# 2024-04-23T16:33:40.997 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bda02b6:::benchmark_data_smithi188_113957_object116059:head# 2024-04-23T16:33:40.997 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bda4358:::benchmark_data_smithi188_113957_object24868:head# 2024-04-23T16:33:40.998 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bda719b:::benchmark_data_smithi188_113957_object115808:head# 2024-04-23T16:33:40.998 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bdb64dc:::benchmark_data_smithi188_113957_object125626:head# 2024-04-23T16:33:40.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bdc85a9:::benchmark_data_smithi188_113957_object1399:head# 2024-04-23T16:33:40.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bdd8a32:::benchmark_data_smithi188_113957_object67726:head# 2024-04-23T16:33:40.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bddb5f0:::benchmark_data_smithi188_113957_object39631:head# 2024-04-23T16:33:41.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bded3e9:::benchmark_data_smithi188_113957_object100389:head# 2024-04-23T16:33:41.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bdf999e:::benchmark_data_smithi188_113957_object11904:head# 2024-04-23T16:33:41.001 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be0a2cb:::benchmark_data_smithi188_113957_object126444:head# 2024-04-23T16:33:41.001 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be146e4:::benchmark_data_smithi188_113957_object1051:head# 2024-04-23T16:33:41.001 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be1a96a:::benchmark_data_smithi188_113957_object80968:head# 2024-04-23T16:33:41.002 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be1eb4f:::benchmark_data_smithi188_113957_object108410:head# 2024-04-23T16:33:41.003 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be23414:::benchmark_data_smithi188_113957_object131009:head# 2024-04-23T16:33:41.003 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be26c31:::benchmark_data_smithi188_113957_object83659:head# 2024-04-23T16:33:41.004 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be26e68:::benchmark_data_smithi188_113957_object147328:head# 2024-04-23T16:33:41.004 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be2a791:::benchmark_data_smithi188_113957_object129516:head# 2024-04-23T16:33:41.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be2aade:::benchmark_data_smithi188_113957_object26185:head# 2024-04-23T16:33:41.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be2e440:::benchmark_data_smithi188_113957_object82093:head# 2024-04-23T16:33:41.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be2ee10:::benchmark_data_smithi188_113957_object103402:head# 2024-04-23T16:33:41.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be32112:::benchmark_data_smithi188_113957_object93258:head# 2024-04-23T16:33:41.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be38aac:::benchmark_data_smithi188_113957_object50361:head# 2024-04-23T16:33:41.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be3e656:::benchmark_data_smithi188_113957_object137666:head# 2024-04-23T16:33:41.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be429df:::benchmark_data_smithi188_113957_object97492:head# 2024-04-23T16:33:41.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be462df:::benchmark_data_smithi188_113957_object131451:head# 2024-04-23T16:33:41.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be51fab:::benchmark_data_smithi188_113957_object127354:head# 2024-04-23T16:33:41.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be5d7f5:::benchmark_data_smithi188_113957_object48473:head# 2024-04-23T16:33:41.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be6bf85:::benchmark_data_smithi188_113957_object110903:head# 2024-04-23T16:33:41.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be6e295:::benchmark_data_smithi188_113957_object39996:head# 2024-04-23T16:33:41.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be6e44a:::benchmark_data_smithi188_113957_object106355:head# 2024-04-23T16:33:41.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be6f0be:::benchmark_data_smithi188_113957_object134040:head# 2024-04-23T16:33:41.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be74b9b:::benchmark_data_smithi188_113957_object111702:head# 2024-04-23T16:33:41.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be75520:::benchmark_data_smithi188_113957_object35812:head# 2024-04-23T16:33:41.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be86fad:::benchmark_data_smithi188_113957_object119760:head# 2024-04-23T16:33:41.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be8949e:::benchmark_data_smithi188_113957_object79129:head# 2024-04-23T16:33:41.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be8aed1:::benchmark_data_smithi188_113957_object128619:head# 2024-04-23T16:33:41.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be8e41f:::benchmark_data_smithi188_113957_object59937:head# 2024-04-23T16:33:41.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be8fe3e:::benchmark_data_smithi188_113957_object76860:head# 2024-04-23T16:33:41.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be92899:::benchmark_data_smithi188_113957_object51992:head# 2024-04-23T16:33:41.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be94d71:::benchmark_data_smithi188_113957_object106557:head# 2024-04-23T16:33:41.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be96f45:::benchmark_data_smithi188_113957_object132619:head# 2024-04-23T16:33:41.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5be9e74d:::benchmark_data_smithi188_113957_object92456:head# 2024-04-23T16:33:41.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bea06f7:::benchmark_data_smithi188_113957_object142927:head# 2024-04-23T16:33:41.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bea0d83:::benchmark_data_smithi188_113957_object83247:head# 2024-04-23T16:33:41.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bea81f4:::benchmark_data_smithi188_113957_object68083:head# 2024-04-23T16:33:41.018 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5beae3e5:::benchmark_data_smithi188_113957_object83280:head# 2024-04-23T16:33:41.019 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5beb4791:::benchmark_data_smithi188_113957_object57180:head# 2024-04-23T16:33:41.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5beb5a87:::benchmark_data_smithi188_113957_object118071:head# 2024-04-23T16:33:41.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5beb674f:::benchmark_data_smithi188_113957_object18376:head# 2024-04-23T16:33:41.022 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bebcd3a:::benchmark_data_smithi188_113957_object14902:head# 2024-04-23T16:33:41.022 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bec5c40:::benchmark_data_smithi188_113957_object98690:head# 2024-04-23T16:33:41.023 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bec9722:::benchmark_data_smithi188_113957_object4473:head# 2024-04-23T16:33:41.023 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bed070a:::benchmark_data_smithi188_113957_object58381:head# 2024-04-23T16:33:41.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bed07a1:::benchmark_data_smithi188_113957_object7529:head# 2024-04-23T16:33:41.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bed83fc:::benchmark_data_smithi188_113957_object82393:head# 2024-04-23T16:33:41.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bedb054:::benchmark_data_smithi188_113957_object39212:head# 2024-04-23T16:33:41.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bedccba:::benchmark_data_smithi188_113957_object105021:head# 2024-04-23T16:33:41.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bee1475:::benchmark_data_smithi188_113957_object33654:head# 2024-04-23T16:33:41.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bee5774:::benchmark_data_smithi188_113957_object9370:head# 2024-04-23T16:33:41.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5befeb67:::benchmark_data_smithi188_113957_object120859:head# 2024-04-23T16:33:41.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf00861:::benchmark_data_smithi188_113957_object78678:head# 2024-04-23T16:33:41.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf00c45:::benchmark_data_smithi188_113957_object139534:head# 2024-04-23T16:33:41.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf04436:::benchmark_data_smithi188_113957_object22815:head# 2024-04-23T16:33:41.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf04923:::benchmark_data_smithi188_113957_object120667:head# 2024-04-23T16:33:41.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf11811:::benchmark_data_smithi188_113957_object142105:head# 2024-04-23T16:33:41.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf17143:::benchmark_data_smithi188_113957_object30243:head# 2024-04-23T16:33:41.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf18a47:::benchmark_data_smithi188_113957_object135746:head# 2024-04-23T16:33:41.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf1dc0f:::benchmark_data_smithi188_113957_object99051:head# 2024-04-23T16:33:41.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf265f8:::benchmark_data_smithi188_113957_object123674:head# 2024-04-23T16:33:41.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf2d5af:::benchmark_data_smithi188_113957_object71523:head# 2024-04-23T16:33:41.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf3496b:::benchmark_data_smithi188_113957_object34628:head# 2024-04-23T16:33:41.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf3c8a5:::benchmark_data_smithi188_113957_object19892:head# 2024-04-23T16:33:41.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf498af:::benchmark_data_smithi188_113957_object10552:head# 2024-04-23T16:33:41.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf4b769:::benchmark_data_smithi188_113957_object15879:head# 2024-04-23T16:33:41.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf65eb4:::benchmark_data_smithi188_113957_object8472:head# 2024-04-23T16:33:41.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf7324e:::benchmark_data_smithi188_113957_object37154:head# 2024-04-23T16:33:41.034 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf75526:::benchmark_data_smithi188_113957_object138609:head# 2024-04-23T16:33:41.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf79318:::benchmark_data_smithi188_113957_object433:head# 2024-04-23T16:33:41.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf7a0b8:::benchmark_data_smithi188_113957_object26684:head# 2024-04-23T16:33:41.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf88451:::benchmark_data_smithi188_113957_object82873:head# 2024-04-23T16:33:41.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf8de0d:::benchmark_data_smithi188_113957_object129466:head# 2024-04-23T16:33:41.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf934b0:::benchmark_data_smithi188_113957_object62236:head# 2024-04-23T16:33:41.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf93f92:::benchmark_data_smithi188_113957_object30899:head# 2024-04-23T16:33:41.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf93fc4:::benchmark_data_smithi188_113957_object123706:head# 2024-04-23T16:33:41.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf973b6:::benchmark_data_smithi188_113957_object19105:head# 2024-04-23T16:33:41.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bf9cef4:::benchmark_data_smithi188_113957_object53442:head# 2024-04-23T16:33:41.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bfaab9b:::benchmark_data_smithi188_113957_object91597:head# 2024-04-23T16:33:41.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bfb7663:::benchmark_data_smithi188_113957_object64041:head# 2024-04-23T16:33:41.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bfbd362:::benchmark_data_smithi188_113957_object144452:head# 2024-04-23T16:33:41.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bfcb1c0:::benchmark_data_smithi188_113957_object16244:head# 2024-04-23T16:33:41.040 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bfcd603:::benchmark_data_smithi188_113957_object9833:head# 2024-04-23T16:33:41.040 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bfdbe58:::benchmark_data_smithi188_113957_object44788:head# 2024-04-23T16:33:41.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bfe044e:::benchmark_data_smithi188_113957_object91398:head# 2024-04-23T16:33:41.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bfec2c5:::benchmark_data_smithi188_113957_object90247:head# 2024-04-23T16:33:41.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5bffbca1:::benchmark_data_smithi188_113957_object44620:head# 2024-04-23T16:33:41.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c00116e:::benchmark_data_smithi188_113957_object53887:head# 2024-04-23T16:33:41.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c006fde:::benchmark_data_smithi188_113957_object67433:head# 2024-04-23T16:33:41.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c007581:::benchmark_data_smithi188_113957_object108386:head# 2024-04-23T16:33:41.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c00ed0f:::benchmark_data_smithi188_113957_object82341:head# 2024-04-23T16:33:41.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0136e2:::benchmark_data_smithi188_113957_object59860:head# 2024-04-23T16:33:41.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c013f4d:::benchmark_data_smithi188_113957_object60574:head# 2024-04-23T16:33:41.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c01f67c:::benchmark_data_smithi188_113957_object68644:head# 2024-04-23T16:33:41.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c02b8d3:::benchmark_data_smithi188_113957_object44325:head# 2024-04-23T16:33:41.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0385f5:::benchmark_data_smithi188_113957_object108741:head# 2024-04-23T16:33:41.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c03a641:::benchmark_data_smithi188_113957_object107188:head# 2024-04-23T16:33:41.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c03a916:::benchmark_data_smithi188_113957_object78125:head# 2024-04-23T16:33:41.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c042d0d:::benchmark_data_smithi188_113957_object79929:head# 2024-04-23T16:33:41.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c04dd12:::benchmark_data_smithi188_113957_object63122:head# 2024-04-23T16:33:41.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c05bb34:::benchmark_data_smithi188_113957_object149610:head# 2024-04-23T16:33:41.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c076330:::benchmark_data_smithi188_113957_object73530:head# 2024-04-23T16:33:41.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c07745e:::benchmark_data_smithi188_113957_object74228:head# 2024-04-23T16:33:41.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c083496:::benchmark_data_smithi188_113957_object132309:head# 2024-04-23T16:33:41.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c087af0:::benchmark_data_smithi188_113957_object80139:head# 2024-04-23T16:33:41.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c08abff:::benchmark_data_smithi188_113957_object128728:head# 2024-04-23T16:33:41.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c08af28:::benchmark_data_smithi188_113957_object95684:head# 2024-04-23T16:33:41.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c08e15b:::benchmark_data_smithi188_113957_object104561:head# 2024-04-23T16:33:41.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c08fb00:::benchmark_data_smithi188_113957_object29562:head# 2024-04-23T16:33:41.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c092262:::benchmark_data_smithi188_113957_object102591:head# 2024-04-23T16:33:41.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0955a7:::benchmark_data_smithi188_113957_object85830:head# 2024-04-23T16:33:41.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c09a670:::benchmark_data_smithi188_113957_object142017:head# 2024-04-23T16:33:41.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0a5ac1:::benchmark_data_smithi188_113957_object58170:head# 2024-04-23T16:33:41.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0a73d0:::benchmark_data_smithi188_113957_object3617:head# 2024-04-23T16:33:41.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0aa52b:::benchmark_data_smithi188_113957_object31859:head# 2024-04-23T16:33:41.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0b904f:::benchmark_data_smithi188_113957_object142101:head# 2024-04-23T16:33:41.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0b9180:::benchmark_data_smithi188_113957_object27498:head# 2024-04-23T16:33:41.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0be4af:::benchmark_data_smithi188_113957_object51134:head# 2024-04-23T16:33:41.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0be6eb:::benchmark_data_smithi188_113957_object86754:head# 2024-04-23T16:33:41.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0c697a:::benchmark_data_smithi188_113957_object16463:head# 2024-04-23T16:33:41.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0c8fe0:::benchmark_data_smithi188_113957_object148616:head# 2024-04-23T16:33:41.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0d2a6d:::benchmark_data_smithi188_113957_object111223:head# 2024-04-23T16:33:41.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0d4153:::benchmark_data_smithi188_113957_object4562:head# 2024-04-23T16:33:41.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0d7993:::benchmark_data_smithi188_113957_object5868:head# 2024-04-23T16:33:41.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0d90cf:::benchmark_data_smithi188_113957_object20005:head# 2024-04-23T16:33:41.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0dabf6:::benchmark_data_smithi188_113957_object8637:head# 2024-04-23T16:33:41.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0dc38b:::benchmark_data_smithi188_113957_object8952:head# 2024-04-23T16:33:41.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0ded5e:::benchmark_data_smithi188_113957_object48120:head# 2024-04-23T16:33:41.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0e0642:::benchmark_data_smithi188_113957_object49985:head# 2024-04-23T16:33:41.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0e400d:::benchmark_data_smithi188_113957_object128155:head# 2024-04-23T16:33:41.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0e9c93:::benchmark_data_smithi188_113957_object18747:head# 2024-04-23T16:33:41.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0ea29d:::benchmark_data_smithi188_113957_object72679:head# 2024-04-23T16:33:41.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0eeae8:::benchmark_data_smithi188_113957_object125983:head# 2024-04-23T16:33:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0f7442:::benchmark_data_smithi188_113957_object32960:head# 2024-04-23T16:33:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c0f7fd5:::benchmark_data_smithi188_113957_object122576:head# 2024-04-23T16:33:41.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c100956:::benchmark_data_smithi188_113957_object72371:head# 2024-04-23T16:33:41.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c115e00:::benchmark_data_smithi188_113957_object31979:head# 2024-04-23T16:33:41.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c115fcc:::benchmark_data_smithi188_113957_object75238:head# 2024-04-23T16:33:41.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c1195d1:::benchmark_data_smithi188_113957_object20555:head# 2024-04-23T16:33:41.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c11d879:::benchmark_data_smithi188_113957_object132284:head# 2024-04-23T16:33:41.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c11e501:::benchmark_data_smithi188_113957_object127405:head# 2024-04-23T16:33:41.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c125a16:::benchmark_data_smithi188_113957_object17407:head# 2024-04-23T16:33:41.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c125c4f:::benchmark_data_smithi188_113957_object87090:head# 2024-04-23T16:33:41.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c128901:::benchmark_data_smithi188_113957_object6031:head# 2024-04-23T16:33:41.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c13b6c1:::benchmark_data_smithi188_113957_object27097:head# 2024-04-23T16:33:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c14bcf8:::benchmark_data_smithi188_113957_object61670:head# 2024-04-23T16:33:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c14ef15:::benchmark_data_smithi188_113957_object77211:head# 2024-04-23T16:33:41.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c159b66:::benchmark_data_smithi188_113957_object5471:head# 2024-04-23T16:33:41.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c15f622:::benchmark_data_smithi188_113957_object23350:head# 2024-04-23T16:33:41.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c15f6f1:::benchmark_data_smithi188_113957_object24746:head# 2024-04-23T16:33:41.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c165818:::benchmark_data_smithi188_113957_object58610:head# 2024-04-23T16:33:41.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c16c30e:::benchmark_data_smithi188_113957_object16202:head# 2024-04-23T16:33:41.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c16ed8c:::benchmark_data_smithi188_113957_object8519:head# 2024-04-23T16:33:41.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c177973:::benchmark_data_smithi188_113957_object144975:head# 2024-04-23T16:33:41.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c17928d:::benchmark_data_smithi188_113957_object65044:head# 2024-04-23T16:33:41.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c1834c3:::benchmark_data_smithi188_113957_object27777:head# 2024-04-23T16:33:41.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c189652:::benchmark_data_smithi188_113957_object126562:head# 2024-04-23T16:33:41.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c18ca75:::benchmark_data_smithi188_113957_object150806:head# 2024-04-23T16:33:41.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c18e3d7:::benchmark_data_smithi188_113957_object19594:head# 2024-04-23T16:33:41.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c190551:::benchmark_data_smithi188_113957_object135452:head# 2024-04-23T16:33:41.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c1912ff:::benchmark_data_smithi188_113957_object134777:head# 2024-04-23T16:33:41.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c196fec:::benchmark_data_smithi188_113957_object22837:head# 2024-04-23T16:33:41.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c19c289:::benchmark_data_smithi188_113957_object99877:head# 2024-04-23T16:33:41.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c19ebfa:::benchmark_data_smithi188_113957_object123458:head# 2024-04-23T16:33:41.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c1a52e7:::benchmark_data_smithi188_113957_object111744:head# 2024-04-23T16:33:41.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c1b1728:::benchmark_data_smithi188_113957_object54451:head# 2024-04-23T16:33:41.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c1b6193:::benchmark_data_smithi188_113957_object52894:head# 2024-04-23T16:33:41.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c1bf026:::benchmark_data_smithi188_113957_object151294:head# 2024-04-23T16:33:41.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c1c5420:::benchmark_data_smithi188_113957_object73397:head# 2024-04-23T16:33:41.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c1d05c9:::benchmark_data_smithi188_113957_object1871:head# 2024-04-23T16:33:41.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c1de0cf:::benchmark_data_smithi188_113957_object82494:head# 2024-04-23T16:33:41.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c1e62aa:::benchmark_data_smithi188_113957_object106236:head# 2024-04-23T16:33:41.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c1ea435:::benchmark_data_smithi188_113957_object16890:head# 2024-04-23T16:33:41.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c1ee329:::benchmark_data_smithi188_113957_object9931:head# 2024-04-23T16:33:41.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c1f4d43:::benchmark_data_smithi188_113957_object75740:head# 2024-04-23T16:33:41.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c206dd6:::benchmark_data_smithi188_113957_object13815:head# 2024-04-23T16:33:41.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2077c8:::benchmark_data_smithi188_113957_object11516:head# 2024-04-23T16:33:41.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c208c91:::benchmark_data_smithi188_113957_object127533:head# 2024-04-23T16:33:41.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2095ae:::benchmark_data_smithi188_113957_object110454:head# 2024-04-23T16:33:41.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c20e5e9:::benchmark_data_smithi188_113957_object87488:head# 2024-04-23T16:33:41.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c20fd85:::benchmark_data_smithi188_113957_object127050:head# 2024-04-23T16:33:41.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c210866:::benchmark_data_smithi188_113957_object64119:head# 2024-04-23T16:33:41.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c226fec:::benchmark_data_smithi188_113957_object99063:head# 2024-04-23T16:33:41.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2280ae:::benchmark_data_smithi188_113957_object50832:head# 2024-04-23T16:33:41.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c22b34d:::benchmark_data_smithi188_113957_object3628:head# 2024-04-23T16:33:41.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2335ee:::benchmark_data_smithi188_113957_object35612:head# 2024-04-23T16:33:41.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c233a58:::benchmark_data_smithi188_113957_object69620:head# 2024-04-23T16:33:41.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c239fe6:::benchmark_data_smithi188_113957_object149857:head# 2024-04-23T16:33:41.087 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2408d8:::benchmark_data_smithi188_113957_object104123:head# 2024-04-23T16:33:41.087 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c242bac:::benchmark_data_smithi188_113957_object94172:head# 2024-04-23T16:33:41.088 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c244907:::benchmark_data_smithi188_113957_object93819:head# 2024-04-23T16:33:41.088 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c247cf2:::benchmark_data_smithi188_113957_object99828:head# 2024-04-23T16:33:41.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c24916a:::benchmark_data_smithi188_113957_object21320:head# 2024-04-23T16:33:41.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c24aac7:::benchmark_data_smithi188_113957_object49416:head# 2024-04-23T16:33:41.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c24e227:::benchmark_data_smithi188_113957_object68591:head# 2024-04-23T16:33:41.091 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c253906:::benchmark_data_smithi188_113957_object115992:head# 2024-04-23T16:33:41.091 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c257659:::benchmark_data_smithi188_113957_object109175:head# 2024-04-23T16:33:41.091 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c25a6f2:::benchmark_data_smithi188_113957_object135587:head# 2024-04-23T16:33:41.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c262b3c:::benchmark_data_smithi188_113957_object54563:head# 2024-04-23T16:33:41.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c26c2db:::benchmark_data_smithi188_113957_object119544:head# 2024-04-23T16:33:41.093 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c283432:::benchmark_data_smithi188_113957_object41529:head# 2024-04-23T16:33:41.093 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c28925d:::benchmark_data_smithi188_113957_object102474:head# 2024-04-23T16:33:41.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c28a041:::benchmark_data_smithi188_113957_object11679:head# 2024-04-23T16:33:41.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c28aa99:::benchmark_data_smithi188_113957_object63257:head# 2024-04-23T16:33:41.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c28f402:::benchmark_data_smithi188_113957_object16908:head# 2024-04-23T16:33:41.095 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c290fbc:::benchmark_data_smithi188_113957_object10059:head# 2024-04-23T16:33:41.095 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c291e9c:::benchmark_data_smithi188_113957_object133455:head# 2024-04-23T16:33:41.095 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c298248:::benchmark_data_smithi188_113957_object79668:head# 2024-04-23T16:33:41.096 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c29a650:::benchmark_data_smithi188_113957_object89143:head# 2024-04-23T16:33:41.096 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2ac69a:::benchmark_data_smithi188_113957_object111550:head# 2024-04-23T16:33:41.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2add12:::benchmark_data_smithi188_113957_object153368:head# 2024-04-23T16:33:41.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2ae29c:::benchmark_data_smithi188_113957_object14291:head# 2024-04-23T16:33:41.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2bb1ad:::benchmark_data_smithi188_113957_object28177:head# 2024-04-23T16:33:41.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2c25be:::benchmark_data_smithi188_113957_object9406:head# 2024-04-23T16:33:41.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2c4070:::benchmark_data_smithi188_113957_object25635:head# 2024-04-23T16:33:41.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2cbf34:::benchmark_data_smithi188_113957_object61231:head# 2024-04-23T16:33:41.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2d03d8:::benchmark_data_smithi188_113957_object54452:head# 2024-04-23T16:33:41.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2d5c58:::benchmark_data_smithi188_113957_object106881:head# 2024-04-23T16:33:41.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2ddbd5:::benchmark_data_smithi188_113957_object40337:head# 2024-04-23T16:33:41.102 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2e3721:::benchmark_data_smithi188_113957_object70002:head# 2024-04-23T16:33:41.102 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2ec5c7:::benchmark_data_smithi188_113957_object66674:head# 2024-04-23T16:33:41.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2eda6e:::benchmark_data_smithi188_113957_object26553:head# 2024-04-23T16:33:41.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2f5169:::benchmark_data_smithi188_113957_object56593:head# 2024-04-23T16:33:41.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2f8909:::benchmark_data_smithi188_113957_object56924:head# 2024-04-23T16:33:41.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2f9017:::benchmark_data_smithi188_113957_object67927:head# 2024-04-23T16:33:41.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2ff25b:::benchmark_data_smithi188_113957_object6466:head# 2024-04-23T16:33:41.105 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c2ff482:::benchmark_data_smithi188_113957_object142231:head# 2024-04-23T16:33:41.105 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c3036bb:::benchmark_data_smithi188_113957_object23930:head# 2024-04-23T16:33:41.105 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c30939a:::benchmark_data_smithi188_113957_object63660:head# 2024-04-23T16:33:41.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c311879:::benchmark_data_smithi188_113957_object65957:head# 2024-04-23T16:33:41.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c3151f3:::benchmark_data_smithi188_113957_object93805:head# 2024-04-23T16:33:41.107 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c319278:::benchmark_data_smithi188_113957_object57823:head# 2024-04-23T16:33:41.107 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c31fd07:::benchmark_data_smithi188_113957_object34158:head# 2024-04-23T16:33:41.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c32c5ba:::benchmark_data_smithi188_113957_object126779:head# 2024-04-23T16:33:41.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c336d6a:::benchmark_data_smithi188_113957_object97365:head# 2024-04-23T16:33:41.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c342b0b:::benchmark_data_smithi188_113957_object45050:head# 2024-04-23T16:33:41.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c348692:::benchmark_data_smithi188_113957_object99154:head# 2024-04-23T16:33:41.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c34b9dc:::benchmark_data_smithi188_113957_object67284:head# 2024-04-23T16:33:41.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c3549ab:::benchmark_data_smithi188_113957_object81056:head# 2024-04-23T16:33:41.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c356ff1:::benchmark_data_smithi188_113957_object73337:head# 2024-04-23T16:33:41.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c35bbd4:::benchmark_data_smithi188_113957_object119663:head# 2024-04-23T16:33:41.111 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c37bbb7:::benchmark_data_smithi188_113957_object24410:head# 2024-04-23T16:33:41.111 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c398525:::benchmark_data_smithi188_113957_object56995:head# 2024-04-23T16:33:41.111 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c39f120:::benchmark_data_smithi188_113957_object132496:head# 2024-04-23T16:33:41.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c3a1fbf:::benchmark_data_smithi188_113957_object31736:head# 2024-04-23T16:33:41.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c3a9ce7:::benchmark_data_smithi188_113957_object56332:head# 2024-04-23T16:33:41.113 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c3b6503:::benchmark_data_smithi188_113957_object61396:head# 2024-04-23T16:33:41.113 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c3bd561:::benchmark_data_smithi188_113957_object30691:head# 2024-04-23T16:33:41.113 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c3be106:::benchmark_data_smithi188_113957_object132048:head# 2024-04-23T16:33:41.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c3c4ba2:::benchmark_data_smithi188_113957_object4716:head# 2024-04-23T16:33:41.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c3c8237:::benchmark_data_smithi188_113957_object70346:head# 2024-04-23T16:33:41.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c3c8759:::benchmark_data_smithi188_113957_object28967:head# 2024-04-23T16:33:41.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c3d9857:::benchmark_data_smithi188_113957_object62745:head# 2024-04-23T16:33:41.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c3db4c1:::benchmark_data_smithi188_113957_object123314:head# 2024-04-23T16:33:41.118 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c3dc1e1:::benchmark_data_smithi188_113957_object41545:head# 2024-04-23T16:33:41.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c3f458b:::benchmark_data_smithi188_113957_object29351:head# 2024-04-23T16:33:41.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c3f5493:::benchmark_data_smithi188_113957_object54662:head# 2024-04-23T16:33:41.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c3fb20e:::benchmark_data_smithi188_113957_object64964:head# 2024-04-23T16:33:41.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c3fd88c:::benchmark_data_smithi188_113957_object42388:head# 2024-04-23T16:33:41.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c40188e:::benchmark_data_smithi188_113957_object133340:head# 2024-04-23T16:33:41.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c405a23:::benchmark_data_smithi188_113957_object21550:head# 2024-04-23T16:33:41.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c411f2e:::benchmark_data_smithi188_113957_object139990:head# 2024-04-23T16:33:41.122 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c41336d:::benchmark_data_smithi188_113957_object43832:head# 2024-04-23T16:33:41.122 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c420709:::benchmark_data_smithi188_113957_object29817:head# 2024-04-23T16:33:41.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c4210a3:::benchmark_data_smithi188_113957_object28534:head# 2024-04-23T16:33:41.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c425a4b:::benchmark_data_smithi188_113957_object93642:head# 2024-04-23T16:33:41.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c434686:::benchmark_data_smithi188_113957_object113901:head# 2024-04-23T16:33:41.124 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c43e976:::benchmark_data_smithi188_113957_object129549:head# 2024-04-23T16:33:41.124 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c447a76:::benchmark_data_smithi188_113957_object10503:head# 2024-04-23T16:33:41.125 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c44b68c:::benchmark_data_smithi188_113957_object32015:head# 2024-04-23T16:33:41.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:41 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:41.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:41 smithi148 ceph-mon[44520]: pgmap v1120: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.9 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 727 op/s; 27626/292884 objects degraded (9.432%); 8.8 MiB/s, 140 objects/s recovering 2024-04-23T16:33:41.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c454b58:::benchmark_data_smithi188_113957_object132286:head# 2024-04-23T16:33:41.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c45add9:::benchmark_data_smithi188_113957_object140402:head# 2024-04-23T16:33:41.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c45b75d:::benchmark_data_smithi188_113957_object112221:head# 2024-04-23T16:33:41.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c45dd5e:::benchmark_data_smithi188_113957_object102948:head# 2024-04-23T16:33:41.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c4633cb:::benchmark_data_smithi188_113957_object79387:head# 2024-04-23T16:33:41.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c469cd9:::benchmark_data_smithi188_113957_object30708:head# 2024-04-23T16:33:41.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c474e91:::benchmark_data_smithi188_113957_object70612:head# 2024-04-23T16:33:41.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c474faf:::benchmark_data_smithi188_113957_object40327:head# 2024-04-23T16:33:41.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c475e20:::benchmark_data_smithi188_113957_object136499:head# 2024-04-23T16:33:41.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c4763de:::benchmark_data_smithi188_113957_object68810:head# 2024-04-23T16:33:41.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c47cbeb:::benchmark_data_smithi188_113957_object43559:head# 2024-04-23T16:33:41.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c4854b1:::benchmark_data_smithi188_113957_object70072:head# 2024-04-23T16:33:41.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c486160:::benchmark_data_smithi188_113957_object126680:head# 2024-04-23T16:33:41.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c48ba20:::benchmark_data_smithi188_113957_object44897:head# 2024-04-23T16:33:41.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c48c004:::benchmark_data_smithi188_113957_object27548:head# 2024-04-23T16:33:41.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c48e025:::benchmark_data_smithi188_113957_object109502:head# 2024-04-23T16:33:41.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c48fcc6:::benchmark_data_smithi188_113957_object61239:head# 2024-04-23T16:33:41.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c49a610:::benchmark_data_smithi188_113957_object68432:head# 2024-04-23T16:33:41.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c4b19df:::benchmark_data_smithi188_113957_object110668:head# 2024-04-23T16:33:41.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c4b2332:::benchmark_data_smithi188_113957_object152066:head# 2024-04-23T16:33:41.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c4b74d9:::benchmark_data_smithi188_113957_object125096:head# 2024-04-23T16:33:41.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c4bcb26:::benchmark_data_smithi188_113957_object49195:head# 2024-04-23T16:33:41.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c4c830a:::benchmark_data_smithi188_113957_object114273:head# 2024-04-23T16:33:41.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c4c9395:::benchmark_data_smithi188_113957_object66598:head# 2024-04-23T16:33:41.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c4cc2d9:::benchmark_data_smithi188_113957_object69284:head# 2024-04-23T16:33:41.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c4ce210:::benchmark_data_smithi188_113957_object151063:head# 2024-04-23T16:33:41.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c4d1aaf:::benchmark_data_smithi188_113957_object56333:head# 2024-04-23T16:33:41.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c4d7974:::benchmark_data_smithi188_113957_object6781:head# 2024-04-23T16:33:41.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c4edfd2:::benchmark_data_smithi188_113957_object23379:head# 2024-04-23T16:33:41.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c4fa861:::benchmark_data_smithi188_113957_object51526:head# 2024-04-23T16:33:41.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5088e6:::benchmark_data_smithi188_113957_object45749:head# 2024-04-23T16:33:41.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c509646:::benchmark_data_smithi188_113957_object59909:head# 2024-04-23T16:33:41.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c51302f:::benchmark_data_smithi188_113957_object100286:head# 2024-04-23T16:33:41.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c517cbf:::benchmark_data_smithi188_113957_object9908:head# 2024-04-23T16:33:41.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c51c5ff:::benchmark_data_smithi188_113957_object137096:head# 2024-04-23T16:33:41.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c51e453:::benchmark_data_smithi188_113957_object26371:head# 2024-04-23T16:33:41.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c520143:::benchmark_data_smithi188_113957_object131809:head# 2024-04-23T16:33:41.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c52699c:::benchmark_data_smithi188_113957_object40542:head# 2024-04-23T16:33:41.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c533b81:::benchmark_data_smithi188_113957_object10625:head# 2024-04-23T16:33:41.142 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5492cf:::benchmark_data_smithi188_113957_object32137:head# 2024-04-23T16:33:41.142 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c54c418:::benchmark_data_smithi188_113957_object112461:head# 2024-04-23T16:33:41.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c54c9c2:::benchmark_data_smithi188_113957_object81646:head# 2024-04-23T16:33:41.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c553112:::benchmark_data_smithi188_113957_object86571:head# 2024-04-23T16:33:41.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c55914c:::benchmark_data_smithi188_113957_object129641:head# 2024-04-23T16:33:41.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5596b5:::benchmark_data_smithi188_113957_object90067:head# 2024-04-23T16:33:41.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c56885e:::benchmark_data_smithi188_113957_object75582:head# 2024-04-23T16:33:41.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5708c8:::benchmark_data_smithi188_113957_object131180:head# 2024-04-23T16:33:41.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c573c90:::benchmark_data_smithi188_113957_object98453:head# 2024-04-23T16:33:41.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5820e0:::benchmark_data_smithi188_113957_object143386:head# 2024-04-23T16:33:41.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c58397f:::benchmark_data_smithi188_113957_object108082:head# 2024-04-23T16:33:41.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c585957:::benchmark_data_smithi188_113957_object36051:head# 2024-04-23T16:33:41.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c59284d:::benchmark_data_smithi188_113957_object66057:head# 2024-04-23T16:33:41.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c59351f:::benchmark_data_smithi188_113957_object148580:head# 2024-04-23T16:33:41.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c595d88:::benchmark_data_smithi188_113957_object12010:head# 2024-04-23T16:33:41.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5a0e7b:::benchmark_data_smithi188_113957_object10281:head# 2024-04-23T16:33:41.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5a5534:::benchmark_data_smithi188_113957_object55340:head# 2024-04-23T16:33:41.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5ae346:::benchmark_data_smithi188_113957_object71370:head# 2024-04-23T16:33:41.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5aeedd:::benchmark_data_smithi188_113957_object98775:head# 2024-04-23T16:33:41.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5af80a:::benchmark_data_smithi188_113957_object88326:head# 2024-04-23T16:33:41.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5b2d58:::benchmark_data_smithi188_113957_object108712:head# 2024-04-23T16:33:41.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5b47c1:::benchmark_data_smithi188_113957_object78870:head# 2024-04-23T16:33:41.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5c8b7d:::benchmark_data_smithi188_113957_object45358:head# 2024-04-23T16:33:41.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5c9d2f:::benchmark_data_smithi188_113957_object138448:head# 2024-04-23T16:33:41.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5ccde5:::benchmark_data_smithi188_113957_object73119:head# 2024-04-23T16:33:41.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5d4911:::benchmark_data_smithi188_113957_object148908:head# 2024-04-23T16:33:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5d5cb3:::benchmark_data_smithi188_113957_object94934:head# 2024-04-23T16:33:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5dc08a:::benchmark_data_smithi188_113957_object75068:head# 2024-04-23T16:33:41.155 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5dd1c9:::benchmark_data_smithi188_113957_object90172:head# 2024-04-23T16:33:41.155 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5e115c:::benchmark_data_smithi188_113957_object36871:head# 2024-04-23T16:33:41.155 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5e9bc3:::benchmark_data_smithi188_113957_object50225:head# 2024-04-23T16:33:41.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5eb44f:::benchmark_data_smithi188_113957_object77426:head# 2024-04-23T16:33:41.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5ee2f5:::benchmark_data_smithi188_113957_object129522:head# 2024-04-23T16:33:41.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5eeda4:::benchmark_data_smithi188_113957_object129995:head# 2024-04-23T16:33:41.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c5f12b9:::benchmark_data_smithi188_113957_object68747:head# 2024-04-23T16:33:41.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c606ed6:::benchmark_data_smithi188_113957_object98247:head# 2024-04-23T16:33:41.158 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c60e38c:::benchmark_data_smithi188_113957_object16637:head# 2024-04-23T16:33:41.158 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:41.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c614c18:::benchmark_data_smithi188_113957_object112548:head# 2024-04-23T16:33:41.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c61b7ae:::benchmark_data_smithi188_113957_object6775:head# 2024-04-23T16:33:41.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c62448e:::benchmark_data_smithi188_113957_object152195:head# 2024-04-23T16:33:41.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c62516d:::benchmark_data_smithi188_113957_object129914:head# 2024-04-23T16:33:41.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c629ae9:::benchmark_data_smithi188_113957_object89742:head# 2024-04-23T16:33:41.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c62b6eb:::benchmark_data_smithi188_113957_object35842:head# 2024-04-23T16:33:41.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c634b5d:::benchmark_data_smithi188_113957_object56173:head# 2024-04-23T16:33:41.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c63a020:::benchmark_data_smithi188_113957_object9426:head# 2024-04-23T16:33:41.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c64c023:::benchmark_data_smithi188_113957_object23103:head# 2024-04-23T16:33:41.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c6514f8:::benchmark_data_smithi188_113957_object123186:head# 2024-04-23T16:33:41.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c661e4a:::benchmark_data_smithi188_113957_object16573:head# 2024-04-23T16:33:41.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c662391:::benchmark_data_smithi188_113957_object34946:head# 2024-04-23T16:33:41.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c6653ac:::benchmark_data_smithi188_113957_object45471:head# 2024-04-23T16:33:41.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c671bdf:::benchmark_data_smithi188_113957_object58809:head# 2024-04-23T16:33:41.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c672ae5:::benchmark_data_smithi188_113957_object16373:head# 2024-04-23T16:33:41.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:41.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c6793b8:::benchmark_data_smithi188_113957_object138169:head# 2024-04-23T16:33:41.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c682c8c:::benchmark_data_smithi188_113957_object98692:head# 2024-04-23T16:33:41.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:41.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c6963c9:::benchmark_data_smithi188_113957_object103172:head# 2024-04-23T16:33:41.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c69942e:::benchmark_data_smithi188_113957_object115595:head# 2024-04-23T16:33:41.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c6a83f9:::benchmark_data_smithi188_113957_object4678:head# 2024-04-23T16:33:41.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c6a9bb4:::benchmark_data_smithi188_113957_object87966:head# 2024-04-23T16:33:41.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c6ae872:::benchmark_data_smithi188_113957_object40264:head# 2024-04-23T16:33:41.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c6b6953:::benchmark_data_smithi188_113957_object126169:head# 2024-04-23T16:33:41.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c6c1308:::benchmark_data_smithi188_113957_object124342:head# 2024-04-23T16:33:41.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c6c75d0:::benchmark_data_smithi188_113957_object11609:head# 2024-04-23T16:33:41.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c6cd3b2:::benchmark_data_smithi188_113957_object82321:head# 2024-04-23T16:33:41.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c6d6c6e:::benchmark_data_smithi188_113957_object123974:head# 2024-04-23T16:33:41.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c6f4607:::benchmark_data_smithi188_113957_object27747:head# 2024-04-23T16:33:41.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c6fe41f:::benchmark_data_smithi188_113957_object23098:head# 2024-04-23T16:33:41.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c710934:::benchmark_data_smithi188_113957_object137959:head# 2024-04-23T16:33:41.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c711661:::benchmark_data_smithi188_113957_object25040:head# 2024-04-23T16:33:41.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c712bc1:::benchmark_data_smithi188_113957_object30408:head# 2024-04-23T16:33:41.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c714b18:::benchmark_data_smithi188_113957_object67676:head# 2024-04-23T16:33:41.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c719ca7:::benchmark_data_smithi188_113957_object52028:head# 2024-04-23T16:33:41.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c71f1f7:::benchmark_data_smithi188_113957_object19988:head# 2024-04-23T16:33:41.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c720bb6:::benchmark_data_smithi188_113957_object143:head# 2024-04-23T16:33:41.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c721852:::benchmark_data_smithi188_113957_object66940:head# 2024-04-23T16:33:41.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c72984c:::benchmark_data_smithi188_113957_object60960:head# 2024-04-23T16:33:41.176 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c72a6c0:::benchmark_data_smithi188_113957_object65139:head# 2024-04-23T16:33:41.177 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c72fe23:::benchmark_data_smithi188_113957_object93198:head# 2024-04-23T16:33:41.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c73198e:::benchmark_data_smithi188_113957_object8522:head# 2024-04-23T16:33:41.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c731d3e:::benchmark_data_smithi188_113957_object12665:head# 2024-04-23T16:33:41.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c73ac00:::benchmark_data_smithi188_113957_object118518:head# 2024-04-23T16:33:41.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c73cdda:::benchmark_data_smithi188_113957_object69866:head# 2024-04-23T16:33:41.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c73e840:::benchmark_data_smithi188_113957_object119016:head# 2024-04-23T16:33:41.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c73e939:::benchmark_data_smithi188_113957_object112748:head# 2024-04-23T16:33:41.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c744c37:::benchmark_data_smithi188_113957_object59791:head# 2024-04-23T16:33:41.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c74e992:::benchmark_data_smithi188_113957_object1891:head# 2024-04-23T16:33:41.181 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c74e9dd:::benchmark_data_smithi188_113957_object129794:head# 2024-04-23T16:33:41.181 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c756f0d:::benchmark_data_smithi188_113957_object21998:head# 2024-04-23T16:33:41.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c766700:::benchmark_data_smithi188_113957_object60813:head# 2024-04-23T16:33:41.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c772b0b:::benchmark_data_smithi188_113957_object38396:head# 2024-04-23T16:33:41.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c775915:::benchmark_data_smithi188_113957_object55971:head# 2024-04-23T16:33:41.183 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c77bfdc:::benchmark_data_smithi188_113957_object137475:head# 2024-04-23T16:33:41.183 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:41 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:41.183 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:41 smithi103 ceph-mon[39629]: pgmap v1120: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.9 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 727 op/s; 27626/292884 objects degraded (9.432%); 8.8 MiB/s, 140 objects/s recovering 2024-04-23T16:33:41.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c7801b7:::benchmark_data_smithi188_113957_object54727:head# 2024-04-23T16:33:41.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c7889df:::benchmark_data_smithi188_113957_object45216:head# 2024-04-23T16:33:41.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c793c85:::benchmark_data_smithi188_113957_object6785:head# 2024-04-23T16:33:41.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c79b28a:::benchmark_data_smithi188_113957_object50860:head# 2024-04-23T16:33:41.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c7ae476:::benchmark_data_smithi188_113957_object141270:head# 2024-04-23T16:33:41.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c7b5494:::benchmark_data_smithi188_113957_object146513:head# 2024-04-23T16:33:41.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c7be5bb:::benchmark_data_smithi188_113957_object33260:head# 2024-04-23T16:33:41.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c7c4588:::benchmark_data_smithi188_113957_object92123:head# 2024-04-23T16:33:41.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c7c55b9:::benchmark_data_smithi188_113957_object70350:head# 2024-04-23T16:33:41.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c7cdb40:::benchmark_data_smithi188_113957_object26391:head# 2024-04-23T16:33:41.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c7d23f8:::benchmark_data_smithi188_113957_object68749:head# 2024-04-23T16:33:41.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c7d5a88:::benchmark_data_smithi188_113957_object5261:head# 2024-04-23T16:33:41.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c7dbbd8:::benchmark_data_smithi188_113957_object104303:head# 2024-04-23T16:33:41.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c7df7df:::benchmark_data_smithi188_113957_object140447:head# 2024-04-23T16:33:41.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c7f38cb:::benchmark_data_smithi188_113957_object97004:head# 2024-04-23T16:33:41.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c7f3c6d:::benchmark_data_smithi188_113957_object52163:head# 2024-04-23T16:33:41.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c7fb864:::benchmark_data_smithi188_113957_object26466:head# 2024-04-23T16:33:41.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c805fce:::benchmark_data_smithi188_113957_object95258:head# 2024-04-23T16:33:41.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c80fc50:::benchmark_data_smithi188_113957_object76728:head# 2024-04-23T16:33:41.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c814b64:::benchmark_data_smithi188_113957_object153330:head# 2024-04-23T16:33:41.192 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c81902a:::benchmark_data_smithi188_113957_object106184:head# 2024-04-23T16:33:41.192 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8195f3:::benchmark_data_smithi188_113957_object152299:head# 2024-04-23T16:33:41.192 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c81b64f:::benchmark_data_smithi188_113957_object66371:head# 2024-04-23T16:33:41.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8218fd:::benchmark_data_smithi188_113957_object55178:head# 2024-04-23T16:33:41.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c827408:::benchmark_data_smithi188_113957_object133788:head# 2024-04-23T16:33:41.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c82b93f:::benchmark_data_smithi188_113957_object64090:head# 2024-04-23T16:33:41.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c832181:::benchmark_data_smithi188_113957_object51326:head# 2024-04-23T16:33:41.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c83aa57:::benchmark_data_smithi188_113957_object88147:head# 2024-04-23T16:33:41.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c83d2c1:::benchmark_data_smithi188_113957_object50843:head# 2024-04-23T16:33:41.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c845697:::benchmark_data_smithi188_113957_object38114:head# 2024-04-23T16:33:41.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c849d29:::benchmark_data_smithi188_113957_object136478:head# 2024-04-23T16:33:41.197 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c854fcc:::benchmark_data_smithi188_113957_object83529:head# 2024-04-23T16:33:41.197 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8613e7:::benchmark_data_smithi188_113957_object147769:head# 2024-04-23T16:33:41.197 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c869ecf:::benchmark_data_smithi188_113957_object119061:head# 2024-04-23T16:33:41.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c876b80:::benchmark_data_smithi188_113957_object131103:head# 2024-04-23T16:33:41.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c877819:::benchmark_data_smithi188_113957_object70655:head# 2024-04-23T16:33:41.199 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c87a4c8:::benchmark_data_smithi188_113957_object118568:head# 2024-04-23T16:33:41.199 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c87d560:::benchmark_data_smithi188_113957_object77677:head# 2024-04-23T16:33:41.199 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c88a399:::benchmark_data_smithi188_113957_object92501:head# 2024-04-23T16:33:41.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c899c5d:::benchmark_data_smithi188_113957_object112776:head# 2024-04-23T16:33:41.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c899f7a:::benchmark_data_smithi188_113957_object5022:head# 2024-04-23T16:33:41.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8a0cde:::benchmark_data_smithi188_113957_object37125:head# 2024-04-23T16:33:41.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8a444f:::benchmark_data_smithi188_113957_object129422:head# 2024-04-23T16:33:41.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8a73cd:::benchmark_data_smithi188_113957_object124373:head# 2024-04-23T16:33:41.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8bad9b:::benchmark_data_smithi188_113957_object79305:head# 2024-04-23T16:33:41.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8bb973:::benchmark_data_smithi188_113957_object5807:head# 2024-04-23T16:33:41.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8bd0f9:::benchmark_data_smithi188_113957_object104732:head# 2024-04-23T16:33:41.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8bf7f4:::benchmark_data_smithi188_113957_object149084:head# 2024-04-23T16:33:41.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8c0353:::benchmark_data_smithi188_113957_object47423:head# 2024-04-23T16:33:41.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8c7663:::benchmark_data_smithi188_113957_object94044:head# 2024-04-23T16:33:41.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8c7ede:::benchmark_data_smithi188_113957_object72799:head# 2024-04-23T16:33:41.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8d3695:::benchmark_data_smithi188_113957_object69170:head# 2024-04-23T16:33:41.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8e2c6c:::benchmark_data_smithi188_113957_object86452:head# 2024-04-23T16:33:41.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8e5b5d:::benchmark_data_smithi188_113957_object39241:head# 2024-04-23T16:33:41.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8e6243:::benchmark_data_smithi188_113957_object93962:head# 2024-04-23T16:33:41.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8ed484:::benchmark_data_smithi188_113957_object100217:head# 2024-04-23T16:33:41.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8f1d7e:::benchmark_data_smithi188_113957_object97516:head# 2024-04-23T16:33:41.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c8fb614:::benchmark_data_smithi188_113957_object148155:head# 2024-04-23T16:33:41.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c901aa3:::benchmark_data_smithi188_113957_object80500:head# 2024-04-23T16:33:41.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c90989c:::benchmark_data_smithi188_113957_object58433:head# 2024-04-23T16:33:41.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c911c72:::benchmark_data_smithi188_113957_object30589:head# 2024-04-23T16:33:41.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c91222d:::benchmark_data_smithi188_113957_object98097:head# 2024-04-23T16:33:41.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c91b345:::benchmark_data_smithi188_113957_object20479:head# 2024-04-23T16:33:41.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c91c503:::benchmark_data_smithi188_113957_object117643:head# 2024-04-23T16:33:41.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c91c6ad:::benchmark_data_smithi188_113957_object133118:head# 2024-04-23T16:33:41.212 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c9275cd:::benchmark_data_smithi188_113957_object134708:head# 2024-04-23T16:33:41.212 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c92883e:::benchmark_data_smithi188_113957_object88961:head# 2024-04-23T16:33:41.212 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c929d88:::benchmark_data_smithi188_113957_object118295:head# 2024-04-23T16:33:41.213 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c93f7e2:::benchmark_data_smithi188_113957_object15801:head# 2024-04-23T16:33:41.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c94231d:::benchmark_data_smithi188_113957_object9267:head# 2024-04-23T16:33:41.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c9449fc:::benchmark_data_smithi188_113957_object149500:head# 2024-04-23T16:33:41.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c946ce4:::benchmark_data_smithi188_113957_object37570:head# 2024-04-23T16:33:41.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c949060:::benchmark_data_smithi188_113957_object127544:head# 2024-04-23T16:33:41.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c949087:::benchmark_data_smithi188_113957_object134454:head# 2024-04-23T16:33:41.216 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c94f4f0:::benchmark_data_smithi188_113957_object50512:head# 2024-04-23T16:33:41.216 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c951911:::benchmark_data_smithi188_113957_object64810:head# 2024-04-23T16:33:41.217 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c96330d:::benchmark_data_smithi188_113957_object8578:head# 2024-04-23T16:33:41.217 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c965ff0:::benchmark_data_smithi188_113957_object135308:head# 2024-04-23T16:33:41.217 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c968bf4:::benchmark_data_smithi188_113957_object76385:head# 2024-04-23T16:33:41.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c96a202:::benchmark_data_smithi188_113957_object82215:head# 2024-04-23T16:33:41.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c971b2d:::benchmark_data_smithi188_113957_object62597:head# 2024-04-23T16:33:41.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c973f03:::benchmark_data_smithi188_113957_object56325:head# 2024-04-23T16:33:41.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c9861a4:::benchmark_data_smithi188_113957_object3019:head# 2024-04-23T16:33:41.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c98d9ec:::benchmark_data_smithi188_113957_object132083:head# 2024-04-23T16:33:41.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c9902cd:::benchmark_data_smithi188_113957_object86891:head# 2024-04-23T16:33:41.224 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c995199:::benchmark_data_smithi188_113957_object46218:head# 2024-04-23T16:33:41.224 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c996bde:::benchmark_data_smithi188_113957_object7956:head# 2024-04-23T16:33:41.225 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c9a2787:::benchmark_data_smithi188_113957_object35347:head# 2024-04-23T16:33:41.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c9a64aa:::benchmark_data_smithi188_113957_object12360:head# 2024-04-23T16:33:41.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c9a92b1:::benchmark_data_smithi188_113957_object74455:head# 2024-04-23T16:33:41.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c9ae713:::benchmark_data_smithi188_113957_object120467:head# 2024-04-23T16:33:41.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c9b0042:::benchmark_data_smithi188_113957_object104704:head# 2024-04-23T16:33:41.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c9b0335:::benchmark_data_smithi188_113957_object55759:head# 2024-04-23T16:33:41.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c9b1cf9:::benchmark_data_smithi188_113957_object88904:head# 2024-04-23T16:33:41.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c9cf01c:::benchmark_data_smithi188_113957_object96180:head# 2024-04-23T16:33:41.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c9d0f6e:::benchmark_data_smithi188_113957_object17132:head# 2024-04-23T16:33:41.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c9e3e76:::benchmark_data_smithi188_113957_object115122:head# 2024-04-23T16:33:41.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c9f4d6d:::benchmark_data_smithi188_113957_object143305:head# 2024-04-23T16:33:41.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5c9fda72:::benchmark_data_smithi188_113957_object125782:head# 2024-04-23T16:33:41.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca0a2e9:::benchmark_data_smithi188_113957_object80116:head# 2024-04-23T16:33:41.231 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca0b4df:::benchmark_data_smithi188_113957_object85588:head# 2024-04-23T16:33:41.231 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca12a63:::benchmark_data_smithi188_113957_object4845:head# 2024-04-23T16:33:41.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca148e5:::benchmark_data_smithi188_113957_object145393:head# 2024-04-23T16:33:41.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca1ff6f:::benchmark_data_smithi188_113957_object9108:head# 2024-04-23T16:33:41.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca2466d:::benchmark_data_smithi188_113957_object75944:head# 2024-04-23T16:33:41.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca253ef:::benchmark_data_smithi188_113957_object29250:head# 2024-04-23T16:33:41.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca2f7b6:::benchmark_data_smithi188_113957_object58696:head# 2024-04-23T16:33:41.234 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca30fd2:::benchmark_data_smithi188_113957_object33418:head# 2024-04-23T16:33:41.234 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca372fe:::benchmark_data_smithi188_113957_object27849:head# 2024-04-23T16:33:41.234 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca379d6:::benchmark_data_smithi188_113957_object39828:head# 2024-04-23T16:33:41.235 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca3a9de:::benchmark_data_smithi188_113957_object78361:head# 2024-04-23T16:33:41.235 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca4294b:::benchmark_data_smithi188_113957_object50478:head# 2024-04-23T16:33:41.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca4aa5a:::benchmark_data_smithi188_113957_object50951:head# 2024-04-23T16:33:41.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca528a0:::benchmark_data_smithi188_113957_object119454:head# 2024-04-23T16:33:41.237 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca71e7b:::benchmark_data_smithi188_113957_object109439:head# 2024-04-23T16:33:41.237 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca73380:::benchmark_data_smithi188_113957_object27238:head# 2024-04-23T16:33:41.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca73dfe:::benchmark_data_smithi188_113957_object81503:head# 2024-04-23T16:33:41.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca84b9f:::benchmark_data_smithi188_113957_object401:head# 2024-04-23T16:33:41.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca89270:::benchmark_data_smithi188_113957_object56225:head# 2024-04-23T16:33:41.240 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ca98ec8:::benchmark_data_smithi188_113957_object123803:head# 2024-04-23T16:33:41.240 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5caa0980:::benchmark_data_smithi188_113957_object122257:head# 2024-04-23T16:33:41.241 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5caaaa9a:::benchmark_data_smithi188_113957_object68097:head# 2024-04-23T16:33:41.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cab0125:::benchmark_data_smithi188_113957_object46058:head# 2024-04-23T16:33:41.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cab33b3:::benchmark_data_smithi188_113957_object23095:head# 2024-04-23T16:33:41.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cab936a:::benchmark_data_smithi188_113957_object95399:head# 2024-04-23T16:33:41.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cac38bb:::benchmark_data_smithi188_113957_object128254:head# 2024-04-23T16:33:41.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5caceda1:::benchmark_data_smithi188_113957_object116111:head# 2024-04-23T16:33:41.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cad03b7:::benchmark_data_smithi188_113957_object2775:head# 2024-04-23T16:33:41.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cad7d35:::benchmark_data_smithi188_113957_object102570:head# 2024-04-23T16:33:41.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cadc6c0:::benchmark_data_smithi188_113957_object62600:head# 2024-04-23T16:33:41.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cadd51b:::benchmark_data_smithi188_113957_object150103:head# 2024-04-23T16:33:41.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cadf308:::benchmark_data_smithi188_113957_object305:head# 2024-04-23T16:33:41.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5caf7655:::benchmark_data_smithi188_113957_object80188:head# 2024-04-23T16:33:41.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5caf9167:::benchmark_data_smithi188_113957_object42323:head# 2024-04-23T16:33:41.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5caf9c1c:::benchmark_data_smithi188_113957_object64198:head# 2024-04-23T16:33:41.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cafa48d:::benchmark_data_smithi188_113957_object106514:head# 2024-04-23T16:33:41.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cafa58b:::benchmark_data_smithi188_113957_object8004:head# 2024-04-23T16:33:41.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cafd317:::benchmark_data_smithi188_113957_object137214:head# 2024-04-23T16:33:41.248 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb05434:::benchmark_data_smithi188_113957_object675:head# 2024-04-23T16:33:41.248 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb11616:::benchmark_data_smithi188_113957_object96711:head# 2024-04-23T16:33:41.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb16cea:::benchmark_data_smithi188_113957_object14926:head# 2024-04-23T16:33:41.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb1e29b:::benchmark_data_smithi188_113957_object106474:head# 2024-04-23T16:33:41.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb20b7c:::benchmark_data_smithi188_113957_object41311:head# 2024-04-23T16:33:41.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb35214:::benchmark_data_smithi188_113957_object29973:head# 2024-04-23T16:33:41.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb3f854:::benchmark_data_smithi188_113957_object22686:head# 2024-04-23T16:33:41.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb40d36:::benchmark_data_smithi188_113957_object75092:head# 2024-04-23T16:33:41.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb4e649:::benchmark_data_smithi188_113957_object68162:head# 2024-04-23T16:33:41.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb51888:::benchmark_data_smithi188_113957_object127065:head# 2024-04-23T16:33:41.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb585cd:::benchmark_data_smithi188_113957_object33453:head# 2024-04-23T16:33:41.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb595c0:::benchmark_data_smithi188_113957_object87658:head# 2024-04-23T16:33:41.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb59cbe:::benchmark_data_smithi188_113957_object73168:head# 2024-04-23T16:33:41.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb5b8dd:::benchmark_data_smithi188_113957_object124117:head# 2024-04-23T16:33:41.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb65aff:::benchmark_data_smithi188_113957_object43532:head# 2024-04-23T16:33:41.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb72f9c:::benchmark_data_smithi188_113957_object131698:head# 2024-04-23T16:33:41.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb774a2:::benchmark_data_smithi188_113957_object87240:head# 2024-04-23T16:33:41.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb77805:::benchmark_data_smithi188_113957_object42118:head# 2024-04-23T16:33:41.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb86a77:::benchmark_data_smithi188_113957_object15891:head# 2024-04-23T16:33:41.258 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb8c060:::benchmark_data_smithi188_113957_object75693:head# 2024-04-23T16:33:41.258 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cb8f713:::benchmark_data_smithi188_113957_object96367:head# 2024-04-23T16:33:41.258 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cba1713:::benchmark_data_smithi188_113957_object26794:head# 2024-04-23T16:33:41.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cbafa43:::benchmark_data_smithi188_113957_object124231:head# 2024-04-23T16:33:41.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cbb18b2:::benchmark_data_smithi188_113957_object74634:head# 2024-04-23T16:33:41.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cbb28db:::benchmark_data_smithi188_113957_object105039:head# 2024-04-23T16:33:41.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cbb525b:::benchmark_data_smithi188_113957_object9378:head# 2024-04-23T16:33:41.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cbb5521:::benchmark_data_smithi188_113957_object58703:head# 2024-04-23T16:33:41.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cbb5f39:::benchmark_data_smithi188_113957_object118489:head# 2024-04-23T16:33:41.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cbe65eb:::benchmark_data_smithi188_113957_object95421:head# 2024-04-23T16:33:41.262 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cbefb51:::benchmark_data_smithi188_113957_object23274:head# 2024-04-23T16:33:41.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cbf38d8:::benchmark_data_smithi188_113957_object43348:head# 2024-04-23T16:33:41.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cbf4d76:::benchmark_data_smithi188_113957_object24946:head# 2024-04-23T16:33:41.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cbfab92:::benchmark_data_smithi188_113957_object86315:head# 2024-04-23T16:33:41.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc04f6d:::benchmark_data_smithi188_113957_object14594:head# 2024-04-23T16:33:41.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc05718:::benchmark_data_smithi188_113957_object126931:head# 2024-04-23T16:33:41.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc08beb:::benchmark_data_smithi188_113957_object7303:head# 2024-04-23T16:33:41.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc0ac54:::benchmark_data_smithi188_113957_object37977:head# 2024-04-23T16:33:41.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc150d8:::benchmark_data_smithi188_113957_object121568:head# 2024-04-23T16:33:41.266 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc17079:::benchmark_data_smithi188_113957_object138461:head# 2024-04-23T16:33:41.266 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc194b8:::benchmark_data_smithi188_113957_object75559:head# 2024-04-23T16:33:41.267 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc1a0ce:::benchmark_data_smithi188_113957_object62971:head# 2024-04-23T16:33:41.268 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:33:41.269 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc1d289:::benchmark_data_smithi188_113957_object96162:head# 2024-04-23T16:33:41.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc1f8d6:::benchmark_data_smithi188_113957_object78581:head# 2024-04-23T16:33:41.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc1ff48:::benchmark_data_smithi188_113957_object97107:head# 2024-04-23T16:33:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc2045d:::benchmark_data_smithi188_113957_object33175:head# 2024-04-23T16:33:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc29e8e:::benchmark_data_smithi188_113957_object67764:head# 2024-04-23T16:33:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc2ea11:::benchmark_data_smithi188_113957_object115221:head# 2024-04-23T16:33:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc3af77:::benchmark_data_smithi188_113957_object121258:head# 2024-04-23T16:33:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc3f33d:::benchmark_data_smithi188_113957_object127571:head# 2024-04-23T16:33:41.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc4c9b1:::benchmark_data_smithi188_113957_object145566:head# 2024-04-23T16:33:41.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc518fe:::benchmark_data_smithi188_113957_object32618:head# 2024-04-23T16:33:41.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc5cf47:::benchmark_data_smithi188_113957_object102403:head# 2024-04-23T16:33:41.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc6749f:::benchmark_data_smithi188_113957_object26449:head# 2024-04-23T16:33:41.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc67c86:::benchmark_data_smithi188_113957_object52187:head# 2024-04-23T16:33:41.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc6aeda:::benchmark_data_smithi188_113957_object30412:head# 2024-04-23T16:33:41.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc6d3b0:::benchmark_data_smithi188_113957_object116949:head# 2024-04-23T16:33:41.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc6d41e:::benchmark_data_smithi188_113957_object117930:head# 2024-04-23T16:33:41.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc7365e:::benchmark_data_smithi188_113957_object46778:head# 2024-04-23T16:33:41.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc73857:::benchmark_data_smithi188_113957_object60774:head# 2024-04-23T16:33:41.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc7f92b:::benchmark_data_smithi188_113957_object111294:head# 2024-04-23T16:33:41.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc8b214:::benchmark_data_smithi188_113957_object33192:head# 2024-04-23T16:33:41.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc99ad3:::benchmark_data_smithi188_113957_object71193:head# 2024-04-23T16:33:41.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc9ef8c:::benchmark_data_smithi188_113957_object43105:head# 2024-04-23T16:33:41.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cc9f907:::benchmark_data_smithi188_113957_object12692:head# 2024-04-23T16:33:41.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cca3f24:::benchmark_data_smithi188_113957_object99933:head# 2024-04-23T16:33:41.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cca56ed:::benchmark_data_smithi188_113957_object60474:head# 2024-04-23T16:33:41.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ccadd24:::benchmark_data_smithi188_113957_object150182:head# 2024-04-23T16:33:41.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ccb82a7:::benchmark_data_smithi188_113957_object102247:head# 2024-04-23T16:33:41.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ccc2d01:::benchmark_data_smithi188_113957_object93932:head# 2024-04-23T16:33:41.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ccc5874:::benchmark_data_smithi188_113957_object75255:head# 2024-04-23T16:33:41.287 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ccd4928:::benchmark_data_smithi188_113957_object113676:head# 2024-04-23T16:33:41.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ccd6520:::benchmark_data_smithi188_113957_object149562:head# 2024-04-23T16:33:41.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ccd7b89:::benchmark_data_smithi188_113957_object8122:head# 2024-04-23T16:33:41.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ccdc11a:::benchmark_data_smithi188_113957_object56494:head# 2024-04-23T16:33:41.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ccdc9ce:::benchmark_data_smithi188_113957_object104987:head# 2024-04-23T16:33:41.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ccee253:::benchmark_data_smithi188_113957_object33476:head# 2024-04-23T16:33:41.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ccf6758:::benchmark_data_smithi188_113957_object13591:head# 2024-04-23T16:33:41.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ccf802b:::benchmark_data_smithi188_113957_object979:head# 2024-04-23T16:33:41.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cd0016a:::benchmark_data_smithi188_113957_object134111:head# 2024-04-23T16:33:41.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cd0621a:::benchmark_data_smithi188_113957_object150110:head# 2024-04-23T16:33:41.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cd08e48:::benchmark_data_smithi188_113957_object71871:head# 2024-04-23T16:33:41.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cd0bace:::benchmark_data_smithi188_113957_object84655:head# 2024-04-23T16:33:41.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cd12742:::benchmark_data_smithi188_113957_object69316:head# 2024-04-23T16:33:41.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cd31a69:::benchmark_data_smithi188_113957_object15810:head# 2024-04-23T16:33:41.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cd3ae48:::benchmark_data_smithi188_113957_object20848:head# 2024-04-23T16:33:41.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cd3dd95:::benchmark_data_smithi188_113957_object136271:head# 2024-04-23T16:33:41.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cd4afbc:::benchmark_data_smithi188_113957_object79999:head# 2024-04-23T16:33:41.295 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cd5e5a0:::benchmark_data_smithi188_113957_object125109:head# 2024-04-23T16:33:41.295 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cd5f89b:::benchmark_data_smithi188_113957_object25538:head# 2024-04-23T16:33:41.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cd6e5dd:::benchmark_data_smithi188_113957_object140418:head# 2024-04-23T16:33:41.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cd78ddd:::benchmark_data_smithi188_113957_object128961:head# 2024-04-23T16:33:41.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cd7fc0b:::benchmark_data_smithi188_113957_object115135:head# 2024-04-23T16:33:41.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cd91afb:::benchmark_data_smithi188_113957_object39381:head# 2024-04-23T16:33:41.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cda54a4:::benchmark_data_smithi188_113957_object153039:head# 2024-04-23T16:33:41.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cdac090:::benchmark_data_smithi188_113957_object70084:head# 2024-04-23T16:33:41.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cdb5b52:::benchmark_data_smithi188_113957_object96353:head# 2024-04-23T16:33:41.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cdb8f01:::benchmark_data_smithi188_113957_object99202:head# 2024-04-23T16:33:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cdb9a99:::benchmark_data_smithi188_113957_object133545:head# 2024-04-23T16:33:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cdba5ae:::benchmark_data_smithi188_113957_object36444:head# 2024-04-23T16:33:41.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cdc127a:::benchmark_data_smithi188_113957_object10926:head# 2024-04-23T16:33:41.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cdc9d3c:::benchmark_data_smithi188_113957_object142958:head# 2024-04-23T16:33:41.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cdca065:::benchmark_data_smithi188_113957_object80350:head# 2024-04-23T16:33:41.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cdcd662:::benchmark_data_smithi188_113957_object75440:head# 2024-04-23T16:33:41.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cded1f5:::benchmark_data_smithi188_113957_object23105:head# 2024-04-23T16:33:41.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cdee7d7:::benchmark_data_smithi188_113957_object145586:head# 2024-04-23T16:33:41.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cdeef5b:::benchmark_data_smithi188_113957_object81736:head# 2024-04-23T16:33:41.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cdefe6d:::benchmark_data_smithi188_113957_object89546:head# 2024-04-23T16:33:41.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce006c2:::benchmark_data_smithi188_113957_object39700:head# 2024-04-23T16:33:41.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce00d56:::benchmark_data_smithi188_113957_object42246:head# 2024-04-23T16:33:41.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce0e038:::benchmark_data_smithi188_113957_object82467:head# 2024-04-23T16:33:41.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce1fb78:::benchmark_data_smithi188_113957_object111143:head# 2024-04-23T16:33:41.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce26fd8:::benchmark_data_smithi188_113957_object43422:head# 2024-04-23T16:33:41.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce2c27a:::benchmark_data_smithi188_113957_object144604:head# 2024-04-23T16:33:41.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce2cb0d:::benchmark_data_smithi188_113957_object146215:head# 2024-04-23T16:33:41.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce32524:::benchmark_data_smithi188_113957_object114358:head# 2024-04-23T16:33:41.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce35e01:::benchmark_data_smithi188_113957_object81971:head# 2024-04-23T16:33:41.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce3a751:::benchmark_data_smithi188_113957_object8540:head# 2024-04-23T16:33:41.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce41ddf:::benchmark_data_smithi188_113957_object87440:head# 2024-04-23T16:33:41.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce4b4cb:::benchmark_data_smithi188_113957_object135315:head# 2024-04-23T16:33:41.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce5b5fa:::benchmark_data_smithi188_113957_object35059:head# 2024-04-23T16:33:41.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce5ffe5:::benchmark_data_smithi188_113957_object5373:head# 2024-04-23T16:33:41.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce6e904:::benchmark_data_smithi188_113957_object1931:head# 2024-04-23T16:33:41.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce78977:::benchmark_data_smithi188_113957_object33420:head# 2024-04-23T16:33:41.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce7e62d:::benchmark_data_smithi188_113957_object37274:head# 2024-04-23T16:33:41.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce81265:::benchmark_data_smithi188_113957_object128002:head# 2024-04-23T16:33:41.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce8e1fd:::benchmark_data_smithi188_113957_object124792:head# 2024-04-23T16:33:41.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce90dec:::benchmark_data_smithi188_113957_object101828:head# 2024-04-23T16:33:41.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce96363:::benchmark_data_smithi188_113957_object101608:head# 2024-04-23T16:33:41.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce963aa:::benchmark_data_smithi188_113957_object71602:head# 2024-04-23T16:33:41.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ce9f6ff:::benchmark_data_smithi188_113957_object131114:head# 2024-04-23T16:33:41.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cea0dd9:::benchmark_data_smithi188_113957_object29579:head# 2024-04-23T16:33:41.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cea6024:::benchmark_data_smithi188_113957_object13601:head# 2024-04-23T16:33:41.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cea6d65:::benchmark_data_smithi188_113957_object19981:head# 2024-04-23T16:33:41.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ceac30b:::benchmark_data_smithi188_113957_object80877:head# 2024-04-23T16:33:41.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cead6eb:::benchmark_data_smithi188_113957_object85600:head# 2024-04-23T16:33:41.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ceaf649:::benchmark_data_smithi188_113957_object100891:head# 2024-04-23T16:33:41.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ceb54ae:::benchmark_data_smithi188_113957_object102326:head# 2024-04-23T16:33:41.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cebbce2:::benchmark_data_smithi188_113957_object33342:head# 2024-04-23T16:33:41.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cebe132:::benchmark_data_smithi188_113957_object30255:head# 2024-04-23T16:33:41.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cebe4e9:::benchmark_data_smithi188_113957_object147667:head# 2024-04-23T16:33:41.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cebf09c:::benchmark_data_smithi188_113957_object3115:head# 2024-04-23T16:33:41.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cec322b:::benchmark_data_smithi188_113957_object23052:head# 2024-04-23T16:33:41.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cec718b:::benchmark_data_smithi188_113957_object59542:head# 2024-04-23T16:33:41.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cec8804:::benchmark_data_smithi188_113957_object19785:head# 2024-04-23T16:33:41.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cecac46:::benchmark_data_smithi188_113957_object791:head# 2024-04-23T16:33:41.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ced26eb:::benchmark_data_smithi188_113957_object4926:head# 2024-04-23T16:33:41.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cedc568:::benchmark_data_smithi188_113957_object30294:head# 2024-04-23T16:33:41.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cedf6f5:::benchmark_data_smithi188_113957_object150070:head# 2024-04-23T16:33:41.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cef0fc7:::benchmark_data_smithi188_113957_object28406:head# 2024-04-23T16:33:41.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cef18ae:::benchmark_data_smithi188_113957_object126647:head# 2024-04-23T16:33:41.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cef2c00:::benchmark_data_smithi188_113957_object43365:head# 2024-04-23T16:33:41.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cef95e5:::benchmark_data_smithi188_113957_object27810:head# 2024-04-23T16:33:41.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cef9b6d:::benchmark_data_smithi188_113957_object144066:head# 2024-04-23T16:33:41.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf08f33:::benchmark_data_smithi188_113957_object55851:head# 2024-04-23T16:33:41.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf0e470:::benchmark_data_smithi188_113957_object111877:head# 2024-04-23T16:33:41.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf14484:::benchmark_data_smithi188_113957_object89191:head# 2024-04-23T16:33:41.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf2af59:::benchmark_data_smithi188_113957_object106761:head# 2024-04-23T16:33:41.326 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf2e9b3:::benchmark_data_smithi188_113957_object149375:head# 2024-04-23T16:33:41.326 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf33db5:::benchmark_data_smithi188_113957_object67500:head# 2024-04-23T16:33:41.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf3915d:::benchmark_data_smithi188_113957_object43255:head# 2024-04-23T16:33:41.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf3a05e:::benchmark_data_smithi188_113957_object88581:head# 2024-04-23T16:33:41.328 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf3b9a4:::benchmark_data_smithi188_113957_object34792:head# 2024-04-23T16:33:41.328 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf46c1f:::benchmark_data_smithi188_113957_object35090:head# 2024-04-23T16:33:41.328 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf4897b:::benchmark_data_smithi188_113957_object16017:head# 2024-04-23T16:33:41.329 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf5f887:::benchmark_data_smithi188_113957_object77734:head# 2024-04-23T16:33:41.329 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf65f24:::benchmark_data_smithi188_113957_object101833:head# 2024-04-23T16:33:41.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf689ce:::benchmark_data_smithi188_113957_object121240:head# 2024-04-23T16:33:41.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf6b76d:::benchmark_data_smithi188_113957_object145429:head# 2024-04-23T16:33:41.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf6fca1:::benchmark_data_smithi188_113957_object116311:head# 2024-04-23T16:33:41.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf9416a:::benchmark_data_smithi188_113957_object38133:head# 2024-04-23T16:33:41.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf95585:::benchmark_data_smithi188_113957_object149755:head# 2024-04-23T16:33:41.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf9bfa9:::benchmark_data_smithi188_113957_object2913:head# 2024-04-23T16:33:41.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cf9da8a:::benchmark_data_smithi188_113957_object61465:head# 2024-04-23T16:33:41.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cfa48b8:::benchmark_data_smithi188_113957_object144095:head# 2024-04-23T16:33:41.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cfb01c5:::benchmark_data_smithi188_113957_object33428:head# 2024-04-23T16:33:41.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cfb05d8:::benchmark_data_smithi188_113957_object135203:head# 2024-04-23T16:33:41.334 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cfb222b:::benchmark_data_smithi188_113957_object87101:head# 2024-04-23T16:33:41.334 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cfb99ce:::benchmark_data_smithi188_113957_object14055:head# 2024-04-23T16:33:41.337 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cfbe510:::benchmark_data_smithi188_113957_object137428:head# 2024-04-23T16:33:41.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cfc0ca1:::benchmark_data_smithi188_113957_object54684:head# 2024-04-23T16:33:41.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cfc3f3d:::benchmark_data_smithi188_113957_object119877:head# 2024-04-23T16:33:41.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cfc5d40:::benchmark_data_smithi188_113957_object38315:head# 2024-04-23T16:33:41.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cfc66b1:::benchmark_data_smithi188_113957_object58759:head# 2024-04-23T16:33:41.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cfd3566:::benchmark_data_smithi188_113957_object23948:head# 2024-04-23T16:33:41.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cfd54f6:::benchmark_data_smithi188_113957_object86733:head# 2024-04-23T16:33:41.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cff1557:::benchmark_data_smithi188_113957_object73897:head# 2024-04-23T16:33:41.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cff5929:::benchmark_data_smithi188_113957_object147909:head# 2024-04-23T16:33:41.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cff6c8e:::benchmark_data_smithi188_113957_object118839:head# 2024-04-23T16:33:41.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5cffea2d:::benchmark_data_smithi188_113957_object66835:head# 2024-04-23T16:33:41.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d00637e:::benchmark_data_smithi188_113957_object92180:head# 2024-04-23T16:33:41.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0070d8:::benchmark_data_smithi188_113957_object95348:head# 2024-04-23T16:33:41.342 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0094d8:::benchmark_data_smithi188_113957_object134970:head# 2024-04-23T16:33:41.342 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d01553c:::benchmark_data_smithi188_113957_object114783:head# 2024-04-23T16:33:41.342 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d01f93b:::benchmark_data_smithi188_113957_object116746:head# 2024-04-23T16:33:41.342 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d02b731:::benchmark_data_smithi188_113957_object107977:head# 2024-04-23T16:33:41.342 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d046921:::benchmark_data_smithi188_113957_object45222:head# 2024-04-23T16:33:41.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0486af:::benchmark_data_smithi188_113957_object1479:head# 2024-04-23T16:33:41.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d048d1a:::benchmark_data_smithi188_113957_object97265:head# 2024-04-23T16:33:41.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d048ff7:::benchmark_data_smithi188_113957_object50892:head# 2024-04-23T16:33:41.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d04cdb3:::benchmark_data_smithi188_113957_object58997:head# 2024-04-23T16:33:41.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d04f614:::benchmark_data_smithi188_113957_object133802:head# 2024-04-23T16:33:41.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d04fb1e:::benchmark_data_smithi188_113957_object116278:head# 2024-04-23T16:33:41.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d050117:::benchmark_data_smithi188_113957_object89230:head# 2024-04-23T16:33:41.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d05dd6c:::benchmark_data_smithi188_113957_object1280:head# 2024-04-23T16:33:41.346 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d05de5c:::benchmark_data_smithi188_113957_object25471:head# 2024-04-23T16:33:41.346 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d05ffc6:::benchmark_data_smithi188_113957_object103958:head# 2024-04-23T16:33:41.346 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d062034:::benchmark_data_smithi188_113957_object106839:head# 2024-04-23T16:33:41.347 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:41.348 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d06cac8:::benchmark_data_smithi188_113957_object60522:head# 2024-04-23T16:33:41.348 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d07a4ec:::benchmark_data_smithi188_113957_object149055:head# 2024-04-23T16:33:41.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d08085e:::benchmark_data_smithi188_113957_object129314:head# 2024-04-23T16:33:41.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d083ea8:::benchmark_data_smithi188_113957_object56643:head# 2024-04-23T16:33:41.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d08d8dc:::benchmark_data_smithi188_113957_object101588:head# 2024-04-23T16:33:41.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d098e21:::benchmark_data_smithi188_113957_object146149:head# 2024-04-23T16:33:41.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d09fa51:::benchmark_data_smithi188_113957_object59192:head# 2024-04-23T16:33:41.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0a3873:::benchmark_data_smithi188_113957_object36843:head# 2024-04-23T16:33:41.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0acf2d:::benchmark_data_smithi188_113957_object69249:head# 2024-04-23T16:33:41.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0af967:::benchmark_data_smithi188_113957_object28965:head# 2024-04-23T16:33:41.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0b057e:::benchmark_data_smithi188_113957_object79148:head# 2024-04-23T16:33:41.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0b3399:::benchmark_data_smithi188_113957_object57215:head# 2024-04-23T16:33:41.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0bc34b:::benchmark_data_smithi188_113957_object2749:head# 2024-04-23T16:33:41.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0c3eff:::benchmark_data_smithi188_113957_object38765:head# 2024-04-23T16:33:41.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0c8117:::benchmark_data_smithi188_113957_object30870:head# 2024-04-23T16:33:41.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0ce7b2:::benchmark_data_smithi188_113957_object123015:head# 2024-04-23T16:33:41.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0d37a1:::benchmark_data_smithi188_113957_object30929:head# 2024-04-23T16:33:41.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0dae89:::benchmark_data_smithi188_113957_object112072:head# 2024-04-23T16:33:41.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0dd3ab:::benchmark_data_smithi188_113957_object139831:head# 2024-04-23T16:33:41.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0ea0ca:::benchmark_data_smithi188_113957_object146251:head# 2024-04-23T16:33:41.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0f2717:::benchmark_data_smithi188_113957_object130471:head# 2024-04-23T16:33:41.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0f324c:::benchmark_data_smithi188_113957_object23045:head# 2024-04-23T16:33:41.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0f8777:::benchmark_data_smithi188_113957_object29774:head# 2024-04-23T16:33:41.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d0fcf15:::benchmark_data_smithi188_113957_object417:head# 2024-04-23T16:33:41.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1095c6:::benchmark_data_smithi188_113957_object81537:head# 2024-04-23T16:33:41.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d10cb5b:::benchmark_data_smithi188_113957_object150762:head# 2024-04-23T16:33:41.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d10cb67:::benchmark_data_smithi188_113957_object87128:head# 2024-04-23T16:33:41.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d10e092:::benchmark_data_smithi188_113957_object127614:head# 2024-04-23T16:33:41.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1110e4:::benchmark_data_smithi188_113957_object81896:head# 2024-04-23T16:33:41.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d113a14:::benchmark_data_smithi188_113957_object64362:head# 2024-04-23T16:33:41.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1147a0:::benchmark_data_smithi188_113957_object23401:head# 2024-04-23T16:33:41.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d117ae1:::benchmark_data_smithi188_113957_object106204:head# 2024-04-23T16:33:41.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d11ae7c:::benchmark_data_smithi188_113957_object40491:head# 2024-04-23T16:33:41.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d126757:::benchmark_data_smithi188_113957_object13007:head# 2024-04-23T16:33:41.361 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1321e7:::benchmark_data_smithi188_113957_object122615:head# 2024-04-23T16:33:41.362 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1334db:::benchmark_data_smithi188_113957_object25094:head# 2024-04-23T16:33:41.362 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d13e4ec:::benchmark_data_smithi188_113957_object82175:head# 2024-04-23T16:33:41.363 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d14ada8:::benchmark_data_smithi188_113957_object91948:head# 2024-04-23T16:33:41.363 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d151185:::benchmark_data_smithi188_113957_object109584:head# 2024-04-23T16:33:41.363 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1535d2:::benchmark_data_smithi188_113957_object127759:head# 2024-04-23T16:33:41.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d160d4d:::benchmark_data_smithi188_113957_object42460:head# 2024-04-23T16:33:41.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d168ce5:::benchmark_data_smithi188_113957_object28991:head# 2024-04-23T16:33:41.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d173d0f:::benchmark_data_smithi188_113957_object138719:head# 2024-04-23T16:33:41.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d180d91:::benchmark_data_smithi188_113957_object34488:head# 2024-04-23T16:33:41.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d182f3c:::benchmark_data_smithi188_113957_object114385:head# 2024-04-23T16:33:41.366 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d185b98:::benchmark_data_smithi188_113957_object98791:head# 2024-04-23T16:33:41.366 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d186486:::benchmark_data_smithi188_113957_object70026:head# 2024-04-23T16:33:41.366 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d186f46:::benchmark_data_smithi188_113957_object153250:head# 2024-04-23T16:33:41.367 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1885b5:::benchmark_data_smithi188_113957_object33034:head# 2024-04-23T16:33:41.368 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d18c63b:::benchmark_data_smithi188_113957_object139500:head# 2024-04-23T16:33:41.368 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d18f170:::benchmark_data_smithi188_113957_object22032:head# 2024-04-23T16:33:41.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1948d0:::benchmark_data_smithi188_113957_object27605:head# 2024-04-23T16:33:41.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1a0cd1:::benchmark_data_smithi188_113957_object56984:head# 2024-04-23T16:33:41.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1a7e71:::benchmark_data_smithi188_113957_object68446:head# 2024-04-23T16:33:41.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1b1caa:::benchmark_data_smithi188_113957_object123389:head# 2024-04-23T16:33:41.371 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1b3b87:::benchmark_data_smithi188_113957_object73498:head# 2024-04-23T16:33:41.371 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1b635c:::benchmark_data_smithi188_113957_object125852:head# 2024-04-23T16:33:41.371 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1b66d2:::benchmark_data_smithi188_113957_object37270:head# 2024-04-23T16:33:41.372 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1c01ab:::benchmark_data_smithi188_113957_object78222:head# 2024-04-23T16:33:41.372 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1c261a:::benchmark_data_smithi188_113957_object84992:head# 2024-04-23T16:33:41.373 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1c2939:::benchmark_data_smithi188_113957_object115593:head# 2024-04-23T16:33:41.373 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1d21fc:::benchmark_data_smithi188_113957_object81249:head# 2024-04-23T16:33:41.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1d7453:::benchmark_data_smithi188_113957_object38068:head# 2024-04-23T16:33:41.375 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1d97b2:::benchmark_data_smithi188_113957_object108382:head# 2024-04-23T16:33:41.375 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1dd943:::benchmark_data_smithi188_113957_object82822:head# 2024-04-23T16:33:41.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1de885:::benchmark_data_smithi188_113957_object42386:head# 2024-04-23T16:33:41.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1e11c6:::benchmark_data_smithi188_113957_object90311:head# 2024-04-23T16:33:41.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1e4c0b:::benchmark_data_smithi188_113957_object131639:head# 2024-04-23T16:33:41.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1f4947:::benchmark_data_smithi188_113957_object76989:head# 2024-04-23T16:33:41.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1f5643:::benchmark_data_smithi188_113957_object56613:head# 2024-04-23T16:33:41.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1f5d31:::benchmark_data_smithi188_113957_object81991:head# 2024-04-23T16:33:41.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1f7e46:::benchmark_data_smithi188_113957_object124025:head# 2024-04-23T16:33:41.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1fbbc8:::benchmark_data_smithi188_113957_object93655:head# 2024-04-23T16:33:41.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d1fe2b3:::benchmark_data_smithi188_113957_object52822:head# 2024-04-23T16:33:41.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d203dee:::benchmark_data_smithi188_113957_object57453:head# 2024-04-23T16:33:41.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d216fee:::benchmark_data_smithi188_113957_object43955:head# 2024-04-23T16:33:41.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d21874c:::benchmark_data_smithi188_113957_object149593:head# 2024-04-23T16:33:41.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d218b2d:::benchmark_data_smithi188_113957_object80703:head# 2024-04-23T16:33:41.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d220779:::benchmark_data_smithi188_113957_object145773:head# 2024-04-23T16:33:41.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d220885:::benchmark_data_smithi188_113957_object6524:head# 2024-04-23T16:33:41.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d22942f:::benchmark_data_smithi188_113957_object88637:head# 2024-04-23T16:33:41.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d22ab04:::benchmark_data_smithi188_113957_object10709:head# 2024-04-23T16:33:41.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d22c467:::benchmark_data_smithi188_113957_object117755:head# 2024-04-23T16:33:41.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d22f52b:::benchmark_data_smithi188_113957_object85945:head# 2024-04-23T16:33:41.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2392ee:::benchmark_data_smithi188_113957_object51114:head# 2024-04-23T16:33:41.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d23a8bc:::benchmark_data_smithi188_113957_object65674:head# 2024-04-23T16:33:41.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d23cef5:::benchmark_data_smithi188_113957_object110921:head# 2024-04-23T16:33:41.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d24e2e2:::benchmark_data_smithi188_113957_object16033:head# 2024-04-23T16:33:41.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d24ff3d:::benchmark_data_smithi188_113957_object53197:head# 2024-04-23T16:33:41.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d252ad4:::benchmark_data_smithi188_113957_object132912:head# 2024-04-23T16:33:41.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d253cff:::benchmark_data_smithi188_113957_object25755:head# 2024-04-23T16:33:41.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d257554:::benchmark_data_smithi188_113957_object40883:head# 2024-04-23T16:33:41.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2623be:::benchmark_data_smithi188_113957_object66595:head# 2024-04-23T16:33:41.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d268a4b:::benchmark_data_smithi188_113957_object74886:head# 2024-04-23T16:33:41.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2731e4:::benchmark_data_smithi188_113957_object34024:head# 2024-04-23T16:33:41.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2744c5:::benchmark_data_smithi188_113957_object82545:head# 2024-04-23T16:33:41.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d274630:::benchmark_data_smithi188_113957_object84452:head# 2024-04-23T16:33:41.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d27911b:::benchmark_data_smithi188_113957_object112151:head# 2024-04-23T16:33:41.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d27a8bd:::benchmark_data_smithi188_113957_object69389:head# 2024-04-23T16:33:41.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d27cdb5:::benchmark_data_smithi188_113957_object22228:head# 2024-04-23T16:33:41.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d291e95:::benchmark_data_smithi188_113957_object52959:head# 2024-04-23T16:33:41.395 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d292e30:::benchmark_data_smithi188_113957_object60675:head# 2024-04-23T16:33:41.395 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d298ead:::benchmark_data_smithi188_113957_object112387:head# 2024-04-23T16:33:41.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d29aaf5:::benchmark_data_smithi188_113957_object76894:head# 2024-04-23T16:33:41.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2a1d11:::benchmark_data_smithi188_113957_object45365:head# 2024-04-23T16:33:41.397 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2a978a:::benchmark_data_smithi188_113957_object41915:head# 2024-04-23T16:33:41.397 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2af72b:::benchmark_data_smithi188_113957_object146161:head# 2024-04-23T16:33:41.397 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2b3332:::benchmark_data_smithi188_113957_object11157:head# 2024-04-23T16:33:41.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2c1539:::benchmark_data_smithi188_113957_object102760:head# 2024-04-23T16:33:41.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2ca466:::benchmark_data_smithi188_113957_object79986:head# 2024-04-23T16:33:41.399 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2d5259:::benchmark_data_smithi188_113957_object42542:head# 2024-04-23T16:33:41.399 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2d787a:::benchmark_data_smithi188_113957_object67269:head# 2024-04-23T16:33:41.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2d989b:::benchmark_data_smithi188_113957_object143578:head# 2024-04-23T16:33:41.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2db626:::benchmark_data_smithi188_113957_object24026:head# 2024-04-23T16:33:41.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2e091b:::benchmark_data_smithi188_113957_object4203:head# 2024-04-23T16:33:41.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2ecaa1:::benchmark_data_smithi188_113957_object14430:head# 2024-04-23T16:33:41.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2f3bf4:::benchmark_data_smithi188_113957_object25889:head# 2024-04-23T16:33:41.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2f5906:::benchmark_data_smithi188_113957_object52487:head# 2024-04-23T16:33:41.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2f7e49:::benchmark_data_smithi188_113957_object86142:head# 2024-04-23T16:33:41.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2fac2c:::benchmark_data_smithi188_113957_object28718:head# 2024-04-23T16:33:41.403 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d2fc3ef:::benchmark_data_smithi188_113957_object133067:head# 2024-04-23T16:33:41.403 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d304a13:::benchmark_data_smithi188_113957_object141110:head# 2024-04-23T16:33:41.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d30ea5b:::benchmark_data_smithi188_113957_object125684:head# 2024-04-23T16:33:41.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d332d03:::benchmark_data_smithi188_113957_object25847:head# 2024-04-23T16:33:41.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d33c80f:::benchmark_data_smithi188_113957_object95432:head# 2024-04-23T16:33:41.405 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d344480:::benchmark_data_smithi188_113957_object5189:head# 2024-04-23T16:33:41.405 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d346344:::benchmark_data_smithi188_113957_object4077:head# 2024-04-23T16:33:41.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d355e25:::benchmark_data_smithi188_113957_object17742:head# 2024-04-23T16:33:41.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d35c0d4:::benchmark_data_smithi188_113957_object150817:head# 2024-04-23T16:33:41.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d35d05e:::benchmark_data_smithi188_113957_object120964:head# 2024-04-23T16:33:41.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d35d84d:::benchmark_data_smithi188_113957_object70408:head# 2024-04-23T16:33:41.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d364c65:::benchmark_data_smithi188_113957_object139555:head# 2024-04-23T16:33:41.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d364faa:::benchmark_data_smithi188_113957_object9438:head# 2024-04-23T16:33:41.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d3663a7:::benchmark_data_smithi188_113957_object58520:head# 2024-04-23T16:33:41.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d36c7bf:::benchmark_data_smithi188_113957_object122466:head# 2024-04-23T16:33:41.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d370495:::benchmark_data_smithi188_113957_object94425:head# 2024-04-23T16:33:41.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d370875:::benchmark_data_smithi188_113957_object79606:head# 2024-04-23T16:33:41.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d382b81:::benchmark_data_smithi188_113957_object27449:head# 2024-04-23T16:33:41.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d38d571:::benchmark_data_smithi188_113957_object94690:head# 2024-04-23T16:33:41.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d38de99:::benchmark_data_smithi188_113957_object101446:head# 2024-04-23T16:33:41.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d38fabd:::benchmark_data_smithi188_113957_object22147:head# 2024-04-23T16:33:41.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d3954fc:::benchmark_data_smithi188_113957_object31634:head# 2024-04-23T16:33:41.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d39ac37:::benchmark_data_smithi188_113957_object84782:head# 2024-04-23T16:33:41.412 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d39df2f:::benchmark_data_smithi188_113957_object143952:head# 2024-04-23T16:33:41.412 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d3a186f:::benchmark_data_smithi188_113957_object31677:head# 2024-04-23T16:33:41.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d3a1da4:::benchmark_data_smithi188_113957_object25915:head# 2024-04-23T16:33:41.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d3a78ad:::benchmark_data_smithi188_113957_object135657:head# 2024-04-23T16:33:41.414 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d3ac931:::benchmark_data_smithi188_113957_object84183:head# 2024-04-23T16:33:41.414 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d3aef1f:::benchmark_data_smithi188_113957_object112709:head# 2024-04-23T16:33:41.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d3b2887:::benchmark_data_smithi188_113957_object48733:head# 2024-04-23T16:33:41.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d3b6b16:::benchmark_data_smithi188_113957_object32966:head# 2024-04-23T16:33:41.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d3c23a5:::benchmark_data_smithi188_113957_object14555:head# 2024-04-23T16:33:41.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d3c846b:::benchmark_data_smithi188_113957_object134414:head# 2024-04-23T16:33:41.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d3d0b27:::benchmark_data_smithi188_113957_object39095:head# 2024-04-23T16:33:41.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d3d1d23:::benchmark_data_smithi188_113957_object28479:head# 2024-04-23T16:33:41.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d3d769e:::benchmark_data_smithi188_113957_object77312:head# 2024-04-23T16:33:41.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d3d8b34:::benchmark_data_smithi188_113957_object33715:head# 2024-04-23T16:33:41.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d3e2ecf:::benchmark_data_smithi188_113957_object33900:head# 2024-04-23T16:33:41.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d3e2fb3:::benchmark_data_smithi188_113957_object151034:head# 2024-04-23T16:33:41.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d3e9f6a:::benchmark_data_smithi188_113957_object52258:head# 2024-04-23T16:33:41.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d407098:::benchmark_data_smithi188_113957_object45224:head# 2024-04-23T16:33:41.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d40819f:::benchmark_data_smithi188_113957_object101994:head# 2024-04-23T16:33:41.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d40e4fe:::benchmark_data_smithi188_113957_object109323:head# 2024-04-23T16:33:41.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d41a595:::benchmark_data_smithi188_113957_object17101:head# 2024-04-23T16:33:41.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d41ef5e:::benchmark_data_smithi188_113957_object3217:head# 2024-04-23T16:33:41.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d41fb1f:::benchmark_data_smithi188_113957_object141921:head# 2024-04-23T16:33:41.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d423173:::benchmark_data_smithi188_113957_object136816:head# 2024-04-23T16:33:41.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d424946:::benchmark_data_smithi188_113957_object80061:head# 2024-04-23T16:33:41.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d436ea8:::benchmark_data_smithi188_113957_object94568:head# 2024-04-23T16:33:41.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d43a393:::benchmark_data_smithi188_113957_object3325:head# 2024-04-23T16:33:41.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d444ce8:::benchmark_data_smithi188_113957_object43306:head# 2024-04-23T16:33:41.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d445f71:::benchmark_data_smithi188_113957_object127303:head# 2024-04-23T16:33:41.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d449328:::benchmark_data_smithi188_113957_object58528:head# 2024-04-23T16:33:41.426 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d44c1a2:::benchmark_data_smithi188_113957_object128670:head# 2024-04-23T16:33:41.426 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d4534f3:::benchmark_data_smithi188_113957_object131230:head# 2024-04-23T16:33:41.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d4576ed:::benchmark_data_smithi188_113957_object4411:head# 2024-04-23T16:33:41.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d4613cf:::benchmark_data_smithi188_113957_object59070:head# 2024-04-23T16:33:41.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d462df8:::benchmark_data_smithi188_113957_object5426:head# 2024-04-23T16:33:41.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d471b18:::benchmark_data_smithi188_113957_object40800:head# 2024-04-23T16:33:41.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d474cf5:::benchmark_data_smithi188_113957_object136315:head# 2024-04-23T16:33:41.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d480b98:::benchmark_data_smithi188_113957_object111984:head# 2024-04-23T16:33:41.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d494841:::benchmark_data_smithi188_113957_object12737:head# 2024-04-23T16:33:41.431 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d497c15:::benchmark_data_smithi188_113957_object93050:head# 2024-04-23T16:33:41.431 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d49c803:::benchmark_data_smithi188_113957_object84853:head# 2024-04-23T16:33:41.432 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d4a0958:::benchmark_data_smithi188_113957_object42540:head# 2024-04-23T16:33:41.432 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d4a468b:::benchmark_data_smithi188_113957_object153087:head# 2024-04-23T16:33:41.432 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d4b2388:::benchmark_data_smithi188_113957_object10092:head# 2024-04-23T16:33:41.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d4bc339:::benchmark_data_smithi188_113957_object36776:head# 2024-04-23T16:33:41.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d4c9dfe:::benchmark_data_smithi188_113957_object87340:head# 2024-04-23T16:33:41.434 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d4cf2e7:::benchmark_data_smithi188_113957_object83305:head# 2024-04-23T16:33:41.434 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d4da93c:::benchmark_data_smithi188_113957_object120840:head# 2024-04-23T16:33:41.434 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d4e8ba1:::benchmark_data_smithi188_113957_object94037:head# 2024-04-23T16:33:41.435 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d4eafbb:::benchmark_data_smithi188_113957_object14078:head# 2024-04-23T16:33:41.435 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d4ec61e:::benchmark_data_smithi188_113957_object153099:head# 2024-04-23T16:33:41.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d4f20b5:::benchmark_data_smithi188_113957_object67858:head# 2024-04-23T16:33:41.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d4f44c7:::benchmark_data_smithi188_113957_object12835:head# 2024-04-23T16:33:41.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d4fe694:::benchmark_data_smithi188_113957_object76965:head# 2024-04-23T16:33:41.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5021b6:::benchmark_data_smithi188_113957_object87231:head# 2024-04-23T16:33:41.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5046c1:::benchmark_data_smithi188_113957_object146733:head# 2024-04-23T16:33:41.438 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d504a5a:::benchmark_data_smithi188_113957_object103238:head# 2024-04-23T16:33:41.438 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d50cd48:::benchmark_data_smithi188_113957_object128826:head# 2024-04-23T16:33:41.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d515387:::benchmark_data_smithi188_113957_object91152:head# 2024-04-23T16:33:41.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d51678f:::benchmark_data_smithi188_113957_object39227:head# 2024-04-23T16:33:41.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d51ec3b:::benchmark_data_smithi188_113957_object73608:head# 2024-04-23T16:33:41.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d521d50:::benchmark_data_smithi188_113957_object29121:head# 2024-04-23T16:33:41.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d524639:::benchmark_data_smithi188_113957_object28637:head# 2024-04-23T16:33:41.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d52abf7:::benchmark_data_smithi188_113957_object98223:head# 2024-04-23T16:33:41.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d52ca8f:::benchmark_data_smithi188_113957_object148231:head# 2024-04-23T16:33:41.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d52f4ba:::benchmark_data_smithi188_113957_object70283:head# 2024-04-23T16:33:41.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5306cc:::benchmark_data_smithi188_113957_object73121:head# 2024-04-23T16:33:41.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5325ce:::benchmark_data_smithi188_113957_object63340:head# 2024-04-23T16:33:41.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d534847:::benchmark_data_smithi188_113957_object98688:head# 2024-04-23T16:33:41.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d539157:::benchmark_data_smithi188_113957_object25776:head# 2024-04-23T16:33:41.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d53a2c8:::benchmark_data_smithi188_113957_object24502:head# 2024-04-23T16:33:41.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d542b38:::benchmark_data_smithi188_113957_object151455:head# 2024-04-23T16:33:41.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d555e22:::benchmark_data_smithi188_113957_object111674:head# 2024-04-23T16:33:41.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d557b47:::benchmark_data_smithi188_113957_object146173:head# 2024-04-23T16:33:41.446 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d55b54a:::benchmark_data_smithi188_113957_object136922:head# 2024-04-23T16:33:41.446 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d55faa8:::benchmark_data_smithi188_113957_object10152:head# 2024-04-23T16:33:41.446 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d560738:::benchmark_data_smithi188_113957_object29712:head# 2024-04-23T16:33:41.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d578f36:::benchmark_data_smithi188_113957_object3889:head# 2024-04-23T16:33:41.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d57acb9:::benchmark_data_smithi188_113957_object149734:head# 2024-04-23T16:33:41.448 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d581beb:::benchmark_data_smithi188_113957_object112184:head# 2024-04-23T16:33:41.448 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d587ffe:::benchmark_data_smithi188_113957_object112869:head# 2024-04-23T16:33:41.449 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5a729f:::benchmark_data_smithi188_113957_object6427:head# 2024-04-23T16:33:41.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5ad044:::benchmark_data_smithi188_113957_object53457:head# 2024-04-23T16:33:41.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5b6e24:::benchmark_data_smithi188_113957_object63140:head# 2024-04-23T16:33:41.451 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5bc901:::benchmark_data_smithi188_113957_object63688:head# 2024-04-23T16:33:41.451 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5cc158:::benchmark_data_smithi188_113957_object57105:head# 2024-04-23T16:33:41.451 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5cc361:::benchmark_data_smithi188_113957_object115231:head# 2024-04-23T16:33:41.452 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5cf744:::benchmark_data_smithi188_113957_object41505:head# 2024-04-23T16:33:41.452 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5d264a:::benchmark_data_smithi188_113957_object64737:head# 2024-04-23T16:33:41.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5d2d7c:::benchmark_data_smithi188_113957_object81309:head# 2024-04-23T16:33:41.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5d377e:::benchmark_data_smithi188_113957_object38419:head# 2024-04-23T16:33:41.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5d5e72:::benchmark_data_smithi188_113957_object111639:head# 2024-04-23T16:33:41.454 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5dd8e5:::benchmark_data_smithi188_113957_object109135:head# 2024-04-23T16:33:41.454 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5de0a3:::benchmark_data_smithi188_113957_object84743:head# 2024-04-23T16:33:41.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5e2a89:::benchmark_data_smithi188_113957_object16218:head# 2024-04-23T16:33:41.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5e2dc4:::benchmark_data_smithi188_113957_object144140:head# 2024-04-23T16:33:41.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5e91ab:::benchmark_data_smithi188_113957_object91235:head# 2024-04-23T16:33:41.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5ed4f9:::benchmark_data_smithi188_113957_object12112:head# 2024-04-23T16:33:41.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5f92fd:::benchmark_data_smithi188_113957_object72068:head# 2024-04-23T16:33:41.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d5fc895:::benchmark_data_smithi188_113957_object86494:head# 2024-04-23T16:33:41.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d60a62a:::benchmark_data_smithi188_113957_object93081:head# 2024-04-23T16:33:41.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d60c6df:::benchmark_data_smithi188_113957_object102501:head# 2024-04-23T16:33:41.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d60efc3:::benchmark_data_smithi188_113957_object111171:head# 2024-04-23T16:33:41.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d61cf86:::benchmark_data_smithi188_113957_object55552:head# 2024-04-23T16:33:41.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d632055:::benchmark_data_smithi188_113957_object142605:head# 2024-04-23T16:33:41.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d63ab68:::benchmark_data_smithi188_113957_object58131:head# 2024-04-23T16:33:41.460 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d64af3d:::benchmark_data_smithi188_113957_object129745:head# 2024-04-23T16:33:41.460 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d64f54a:::benchmark_data_smithi188_113957_object82407:head# 2024-04-23T16:33:41.460 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d650db2:::benchmark_data_smithi188_113957_object61278:head# 2024-04-23T16:33:41.461 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d654722:::benchmark_data_smithi188_113957_object142905:head# 2024-04-23T16:33:41.461 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d676e8a:::benchmark_data_smithi188_113957_object97402:head# 2024-04-23T16:33:41.461 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d67b99a:::benchmark_data_smithi188_113957_object138112:head# 2024-04-23T16:33:41.462 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d6868a9:::benchmark_data_smithi188_113957_object149185:head# 2024-04-23T16:33:41.462 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d692464:::benchmark_data_smithi188_113957_object62959:head# 2024-04-23T16:33:41.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d6931a0:::benchmark_data_smithi188_113957_object110563:head# 2024-04-23T16:33:41.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d697840:::benchmark_data_smithi188_113957_object15117:head# 2024-04-23T16:33:41.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d698d42:::benchmark_data_smithi188_113957_object115372:head# 2024-04-23T16:33:41.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d699123:::benchmark_data_smithi188_113957_object101783:head# 2024-04-23T16:33:41.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d6be1a9:::benchmark_data_smithi188_113957_object53944:head# 2024-04-23T16:33:41.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d6be32f:::benchmark_data_smithi188_113957_object143695:head# 2024-04-23T16:33:41.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d6c5978:::benchmark_data_smithi188_113957_object170:head# 2024-04-23T16:33:41.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d6d7b5f:::benchmark_data_smithi188_113957_object37217:head# 2024-04-23T16:33:41.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d6da432:::benchmark_data_smithi188_113957_object51509:head# 2024-04-23T16:33:41.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d6db002:::benchmark_data_smithi188_113957_object117900:head# 2024-04-23T16:33:41.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d6e2de4:::benchmark_data_smithi188_113957_object32653:head# 2024-04-23T16:33:41.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d6e6b1e:::benchmark_data_smithi188_113957_object82399:head# 2024-04-23T16:33:41.469 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d6e7ebb:::benchmark_data_smithi188_113957_object133593:head# 2024-04-23T16:33:41.469 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d6eef78:::benchmark_data_smithi188_113957_object58320:head# 2024-04-23T16:33:41.470 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d6ef39f:::benchmark_data_smithi188_113957_object143172:head# 2024-04-23T16:33:41.470 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d6f0149:::benchmark_data_smithi188_113957_object112304:head# 2024-04-23T16:33:41.470 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d6f372a:::benchmark_data_smithi188_113957_object32596:head# 2024-04-23T16:33:41.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d702c6e:::benchmark_data_smithi188_113957_object76562:head# 2024-04-23T16:33:41.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d704fe7:::benchmark_data_smithi188_113957_object153464:head# 2024-04-23T16:33:41.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d70d9e8:::benchmark_data_smithi188_113957_object12327:head# 2024-04-23T16:33:41.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d70e135:::benchmark_data_smithi188_113957_object142651:head# 2024-04-23T16:33:41.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d70f62a:::benchmark_data_smithi188_113957_object153363:head# 2024-04-23T16:33:41.473 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7100fd:::benchmark_data_smithi188_113957_object18365:head# 2024-04-23T16:33:41.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:41 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:41.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:41 smithi188 ceph-mon[44101]: pgmap v1120: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.9 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 727 op/s; 27626/292884 objects degraded (9.432%); 8.8 MiB/s, 140 objects/s recovering 2024-04-23T16:33:41.474 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d711cb0:::benchmark_data_smithi188_113957_object61738:head# 2024-04-23T16:33:41.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7145d1:::benchmark_data_smithi188_113957_object73458:head# 2024-04-23T16:33:41.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d718d5b:::benchmark_data_smithi188_113957_object94435:head# 2024-04-23T16:33:41.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d719338:::benchmark_data_smithi188_113957_object16051:head# 2024-04-23T16:33:41.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d71e63c:::benchmark_data_smithi188_113957_object103167:head# 2024-04-23T16:33:41.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d725b00:::benchmark_data_smithi188_113957_object151594:head# 2024-04-23T16:33:41.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d73b235:::benchmark_data_smithi188_113957_object3603:head# 2024-04-23T16:33:41.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d73dcd4:::benchmark_data_smithi188_113957_object130297:head# 2024-04-23T16:33:41.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d73e659:::benchmark_data_smithi188_113957_object147432:head# 2024-04-23T16:33:41.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d747176:::benchmark_data_smithi188_113957_object7060:head# 2024-04-23T16:33:41.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d74dbcc:::benchmark_data_smithi188_113957_object153367:head# 2024-04-23T16:33:41.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d751dbe:::benchmark_data_smithi188_113957_object132725:head# 2024-04-23T16:33:41.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7598de:::benchmark_data_smithi188_113957_object93865:head# 2024-04-23T16:33:41.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d768135:::benchmark_data_smithi188_113957_object124505:head# 2024-04-23T16:33:41.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d771724:::benchmark_data_smithi188_113957_object125221:head# 2024-04-23T16:33:41.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7836d6:::benchmark_data_smithi188_113957_object21650:head# 2024-04-23T16:33:41.481 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d78ccca:::benchmark_data_smithi188_113957_object4245:head# 2024-04-23T16:33:41.481 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d79115f:::benchmark_data_smithi188_113957_object73937:head# 2024-04-23T16:33:41.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7969d2:::benchmark_data_smithi188_113957_object41597:head# 2024-04-23T16:33:41.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d79ae4b:::benchmark_data_smithi188_113957_object153104:head# 2024-04-23T16:33:41.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7a6e94:::benchmark_data_smithi188_113957_object85044:head# 2024-04-23T16:33:41.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7adb0f:::benchmark_data_smithi188_113957_object140742:head# 2024-04-23T16:33:41.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7b7296:::benchmark_data_smithi188_113957_object89133:head# 2024-04-23T16:33:41.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7cfd29:::benchmark_data_smithi188_113957_object133178:head# 2024-04-23T16:33:41.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7d193c:::benchmark_data_smithi188_113957_object15438:head# 2024-04-23T16:33:41.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7d8ad2:::benchmark_data_smithi188_113957_object32275:head# 2024-04-23T16:33:41.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7e19fc:::benchmark_data_smithi188_113957_object121493:head# 2024-04-23T16:33:41.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7e1b29:::benchmark_data_smithi188_113957_object80924:head# 2024-04-23T16:33:41.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7e3ff5:::benchmark_data_smithi188_113957_object121378:head# 2024-04-23T16:33:41.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7e89f2:::benchmark_data_smithi188_113957_object148868:head# 2024-04-23T16:33:41.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7f1b17:::benchmark_data_smithi188_113957_object61697:head# 2024-04-23T16:33:41.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7f23a2:::benchmark_data_smithi188_113957_object21103:head# 2024-04-23T16:33:41.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7f59ce:::benchmark_data_smithi188_113957_object59132:head# 2024-04-23T16:33:41.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7faf63:::benchmark_data_smithi188_113957_object54503:head# 2024-04-23T16:33:41.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7fbfda:::benchmark_data_smithi188_113957_object119848:head# 2024-04-23T16:33:41.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7fcdb1:::benchmark_data_smithi188_113957_object142701:head# 2024-04-23T16:33:41.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7fd777:::benchmark_data_smithi188_113957_object25464:head# 2024-04-23T16:33:41.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d7ffeb5:::benchmark_data_smithi188_113957_object61519:head# 2024-04-23T16:33:41.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d8030a2:::benchmark_data_smithi188_113957_object21637:head# 2024-04-23T16:33:41.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d80c7dc:::benchmark_data_smithi188_113957_object75044:head# 2024-04-23T16:33:41.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d81c032:::benchmark_data_smithi188_113957_object102631:head# 2024-04-23T16:33:41.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d81f290:::benchmark_data_smithi188_113957_object133297:head# 2024-04-23T16:33:41.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d82c4ad:::benchmark_data_smithi188_113957_object44820:head# 2024-04-23T16:33:41.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d8424d0:::benchmark_data_smithi188_113957_object93821:head# 2024-04-23T16:33:41.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d849842:::benchmark_data_smithi188_113957_object137565:head# 2024-04-23T16:33:41.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d84c985:::benchmark_data_smithi188_113957_object120468:head# 2024-04-23T16:33:41.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d84cc72:::benchmark_data_smithi188_113957_object85122:head# 2024-04-23T16:33:41.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d8506c2:::benchmark_data_smithi188_113957_object128887:head# 2024-04-23T16:33:41.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d85b25f:::benchmark_data_smithi188_113957_object51332:head# 2024-04-23T16:33:41.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d85e29a:::benchmark_data_smithi188_113957_object105396:head# 2024-04-23T16:33:41.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d86c41e:::benchmark_data_smithi188_113957_object132643:head# 2024-04-23T16:33:41.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d87cca3:::benchmark_data_smithi188_113957_object139265:head# 2024-04-23T16:33:41.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d889ed1:::benchmark_data_smithi188_113957_object25948:head# 2024-04-23T16:33:41.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d89c2de:::benchmark_data_smithi188_113957_object114456:head# 2024-04-23T16:33:41.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d8a9a34:::benchmark_data_smithi188_113957_object15683:head# 2024-04-23T16:33:41.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d8ac281:::benchmark_data_smithi188_113957_object70373:head# 2024-04-23T16:33:41.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d8ae72a:::benchmark_data_smithi188_113957_object127337:head# 2024-04-23T16:33:41.504 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d8baad1:::benchmark_data_smithi188_113957_object118092:head# 2024-04-23T16:33:41.504 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d8bce5e:::benchmark_data_smithi188_113957_object114582:head# 2024-04-23T16:33:41.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d8bfdc6:::benchmark_data_smithi188_113957_object52011:head# 2024-04-23T16:33:41.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d8d9ec3:::benchmark_data_smithi188_113957_object69412:head# 2024-04-23T16:33:41.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d8dc670:::benchmark_data_smithi188_113957_object79624:head# 2024-04-23T16:33:41.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d8e5a8a:::benchmark_data_smithi188_113957_object8597:head# 2024-04-23T16:33:41.507 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d8e6441:::benchmark_data_smithi188_113957_object80012:head# 2024-04-23T16:33:41.507 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d8ea9ca:::benchmark_data_smithi188_113957_object124502:head# 2024-04-23T16:33:41.509 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d8eb546:::benchmark_data_smithi188_113957_object81073:head# 2024-04-23T16:33:41.509 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d8ee46f:::benchmark_data_smithi188_113957_object42904:head# 2024-04-23T16:33:41.509 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d8f65dd:::benchmark_data_smithi188_113957_object77020:head# 2024-04-23T16:33:41.509 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d8f687e:::benchmark_data_smithi188_113957_object10499:head# 2024-04-23T16:33:41.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d8ffb7b:::benchmark_data_smithi188_113957_object127322:head# 2024-04-23T16:33:41.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d90447f:::benchmark_data_smithi188_113957_object62310:head# 2024-04-23T16:33:41.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d91e7e5:::benchmark_data_smithi188_113957_object140514:head# 2024-04-23T16:33:41.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d9233b1:::benchmark_data_smithi188_113957_object18407:head# 2024-04-23T16:33:41.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d925b6b:::benchmark_data_smithi188_113957_object37434:head# 2024-04-23T16:33:41.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d926982:::benchmark_data_smithi188_113957_object119140:head# 2024-04-23T16:33:41.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d928545:::benchmark_data_smithi188_113957_object30262:head# 2024-04-23T16:33:41.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d92d6a5:::benchmark_data_smithi188_113957_object20862:head# 2024-04-23T16:33:41.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d92fc93:::benchmark_data_smithi188_113957_object89201:head# 2024-04-23T16:33:41.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d940feb:::benchmark_data_smithi188_113957_object125731:head# 2024-04-23T16:33:41.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d94cc73:::benchmark_data_smithi188_113957_object10403:head# 2024-04-23T16:33:41.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d95626c:::benchmark_data_smithi188_113957_object44719:head# 2024-04-23T16:33:41.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d957aa7:::benchmark_data_smithi188_113957_object139358:head# 2024-04-23T16:33:41.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d9589fc:::benchmark_data_smithi188_113957_object137392:head# 2024-04-23T16:33:41.516 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d95c84d:::benchmark_data_smithi188_113957_object745:head# 2024-04-23T16:33:41.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d967502:::benchmark_data_smithi188_113957_object8078:head# 2024-04-23T16:33:41.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d96786e:::benchmark_data_smithi188_113957_object76779:head# 2024-04-23T16:33:41.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d96a011:::benchmark_data_smithi188_113957_object127559:head# 2024-04-23T16:33:41.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d96ca81:::benchmark_data_smithi188_113957_object136992:head# 2024-04-23T16:33:41.521 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d96ffd4:::benchmark_data_smithi188_113957_object49578:head# 2024-04-23T16:33:41.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d9853db:::benchmark_data_smithi188_113957_object114859:head# 2024-04-23T16:33:41.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d99e8c5:::benchmark_data_smithi188_113957_object138065:head# 2024-04-23T16:33:41.523 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d99e979:::benchmark_data_smithi188_113957_object70113:head# 2024-04-23T16:33:41.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d9a978f:::benchmark_data_smithi188_113957_object139643:head# 2024-04-23T16:33:41.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d9ab40d:::benchmark_data_smithi188_113957_object72309:head# 2024-04-23T16:33:41.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d9b0ab2:::benchmark_data_smithi188_113957_object86724:head# 2024-04-23T16:33:41.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d9b4044:::benchmark_data_smithi188_113957_object31413:head# 2024-04-23T16:33:41.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d9bb8c5:::benchmark_data_smithi188_113957_object108762:head# 2024-04-23T16:33:41.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d9d6bc1:::benchmark_data_smithi188_113957_object19335:head# 2024-04-23T16:33:41.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d9d864c:::benchmark_data_smithi188_113957_object124114:head# 2024-04-23T16:33:41.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d9da76c:::benchmark_data_smithi188_113957_object80410:head# 2024-04-23T16:33:41.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d9df807:::benchmark_data_smithi188_113957_object58627:head# 2024-04-23T16:33:41.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d9df8c0:::benchmark_data_smithi188_113957_object137408:head# 2024-04-23T16:33:41.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d9eb6e9:::benchmark_data_smithi188_113957_object120047:head# 2024-04-23T16:33:41.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5d9f4d82:::benchmark_data_smithi188_113957_object81185:head# 2024-04-23T16:33:41.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5da030ac:::benchmark_data_smithi188_113957_object26109:head# 2024-04-23T16:33:41.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5da0c3fa:::benchmark_data_smithi188_113957_object149379:head# 2024-04-23T16:33:41.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5da16bec:::benchmark_data_smithi188_113957_object66087:head# 2024-04-23T16:33:41.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5da1ce47:::benchmark_data_smithi188_113957_object121628:head# 2024-04-23T16:33:41.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5da2866b:::benchmark_data_smithi188_113957_object136214:head# 2024-04-23T16:33:41.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5da2ddf1:::benchmark_data_smithi188_113957_object90724:head# 2024-04-23T16:33:41.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5da40e7b:::benchmark_data_smithi188_113957_object16554:head# 2024-04-23T16:33:41.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5da416f5:::benchmark_data_smithi188_113957_object65910:head# 2024-04-23T16:33:41.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5da4d8a4:::benchmark_data_smithi188_113957_object142361:head# 2024-04-23T16:33:41.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5da573c5:::benchmark_data_smithi188_113957_object128115:head# 2024-04-23T16:33:41.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5da5b203:::benchmark_data_smithi188_113957_object54750:head# 2024-04-23T16:33:41.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5da60bd8:::benchmark_data_smithi188_113957_object58304:head# 2024-04-23T16:33:41.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5da61b86:::benchmark_data_smithi188_113957_object69357:head# 2024-04-23T16:33:41.533 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5da6e766:::benchmark_data_smithi188_113957_object66702:head# 2024-04-23T16:33:41.533 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5da76aa9:::benchmark_data_smithi188_113957_object83950:head# 2024-04-23T16:33:41.533 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5da84a59:::benchmark_data_smithi188_113957_object46534:head# 2024-04-23T16:33:41.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5da99e4a:::benchmark_data_smithi188_113957_object65737:head# 2024-04-23T16:33:41.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5daa98ed:::benchmark_data_smithi188_113957_object120595:head# 2024-04-23T16:33:41.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dac2a36:::benchmark_data_smithi188_113957_object147794:head# 2024-04-23T16:33:41.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dac48c1:::benchmark_data_smithi188_113957_object153214:head# 2024-04-23T16:33:41.536 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dac9729:::benchmark_data_smithi188_113957_object150930:head# 2024-04-23T16:33:41.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dacddeb:::benchmark_data_smithi188_113957_object126289:head# 2024-04-23T16:33:41.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dad065e:::benchmark_data_smithi188_113957_object7257:head# 2024-04-23T16:33:41.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dad1e19:::benchmark_data_smithi188_113957_object58293:head# 2024-04-23T16:33:41.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dad2477:::benchmark_data_smithi188_113957_object111902:head# 2024-04-23T16:33:41.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dad8263:::benchmark_data_smithi188_113957_object101285:head# 2024-04-23T16:33:41.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dae3459:::benchmark_data_smithi188_113957_object59148:head# 2024-04-23T16:33:41.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dae6b15:::benchmark_data_smithi188_113957_object46380:head# 2024-04-23T16:33:41.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5daed293:::benchmark_data_smithi188_113957_object17935:head# 2024-04-23T16:33:41.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5daef02e:::benchmark_data_smithi188_113957_object60842:head# 2024-04-23T16:33:41.541 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5daf4ce7:::benchmark_data_smithi188_113957_object10376:head# 2024-04-23T16:33:41.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dafa02f:::benchmark_data_smithi188_113957_object93393:head# 2024-04-23T16:33:41.545 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dafe98a:::benchmark_data_smithi188_113957_object37881:head# 2024-04-23T16:33:41.545 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db03a7d:::benchmark_data_smithi188_113957_object42925:head# 2024-04-23T16:33:41.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db0406c:::benchmark_data_smithi188_113957_object133787:head# 2024-04-23T16:33:41.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db0765f:::benchmark_data_smithi188_113957_object125439:head# 2024-04-23T16:33:41.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db0bb03:::benchmark_data_smithi188_113957_object120244:head# 2024-04-23T16:33:41.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db11c68:::benchmark_data_smithi188_113957_object135016:head# 2024-04-23T16:33:41.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db15966:::benchmark_data_smithi188_113957_object79926:head# 2024-04-23T16:33:41.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db1a5e7:::benchmark_data_smithi188_113957_object147057:head# 2024-04-23T16:33:41.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db1fbc7:::benchmark_data_smithi188_113957_object95144:head# 2024-04-23T16:33:41.546 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db2bb8e:::benchmark_data_smithi188_113957_object57598:head# 2024-04-23T16:33:41.547 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db322e1:::benchmark_data_smithi188_113957_object121:head# 2024-04-23T16:33:41.547 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db49e3b:::benchmark_data_smithi188_113957_object42093:head# 2024-04-23T16:33:41.547 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db51416:::benchmark_data_smithi188_113957_object120340:head# 2024-04-23T16:33:41.547 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db5c3e1:::benchmark_data_smithi188_113957_object151434:head# 2024-04-23T16:33:41.548 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db5e447:::benchmark_data_smithi188_113957_object1260:head# 2024-04-23T16:33:41.548 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db5f251:::benchmark_data_smithi188_113957_object13248:head# 2024-04-23T16:33:41.548 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db6bbe6:::benchmark_data_smithi188_113957_object124440:head# 2024-04-23T16:33:41.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db71991:::benchmark_data_smithi188_113957_object81610:head# 2024-04-23T16:33:41.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db7b683:::benchmark_data_smithi188_113957_object22736:head# 2024-04-23T16:33:41.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db7d7ac:::benchmark_data_smithi188_113957_object69428:head# 2024-04-23T16:33:41.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db9199f:::benchmark_data_smithi188_113957_object37755:head# 2024-04-23T16:33:41.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5db98383:::benchmark_data_smithi188_113957_object27698:head# 2024-04-23T16:33:41.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dba6c63:::benchmark_data_smithi188_113957_object72323:head# 2024-04-23T16:33:41.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dbb3961:::benchmark_data_smithi188_113957_object82372:head# 2024-04-23T16:33:41.552 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dbb7ee2:::benchmark_data_smithi188_113957_object108836:head# 2024-04-23T16:33:41.553 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dbbc8d7:::benchmark_data_smithi188_113957_object29946:head# 2024-04-23T16:33:41.553 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dbcd787:::benchmark_data_smithi188_113957_object42710:head# 2024-04-23T16:33:41.553 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dbce6a0:::benchmark_data_smithi188_113957_object74727:head# 2024-04-23T16:33:41.553 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dbcfb41:::benchmark_data_smithi188_113957_object117842:head# 2024-04-23T16:33:41.553 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dbd6dd5:::benchmark_data_smithi188_113957_object53444:head# 2024-04-23T16:33:41.554 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dbe1bb7:::benchmark_data_smithi188_113957_object60343:head# 2024-04-23T16:33:41.554 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dbe692a:::benchmark_data_smithi188_113957_object81570:head# 2024-04-23T16:33:41.555 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dbe6fa1:::benchmark_data_smithi188_113957_object93801:head# 2024-04-23T16:33:41.555 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dbe96b0:::benchmark_data_smithi188_113957_object1170:head# 2024-04-23T16:33:41.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dbeb10e:::benchmark_data_smithi188_113957_object102102:head# 2024-04-23T16:33:41.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dbec2d4:::benchmark_data_smithi188_113957_object20529:head# 2024-04-23T16:33:41.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dbf50c7:::benchmark_data_smithi188_113957_object48322:head# 2024-04-23T16:33:41.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dbf899b:::benchmark_data_smithi188_113957_object70138:head# 2024-04-23T16:33:41.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dbfde87:::benchmark_data_smithi188_113957_object108429:head# 2024-04-23T16:33:41.558 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc0427a:::benchmark_data_smithi188_113957_object102263:head# 2024-04-23T16:33:41.558 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc10b17:::benchmark_data_smithi188_113957_object113646:head# 2024-04-23T16:33:41.559 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc1525f:::benchmark_data_smithi188_113957_object14755:head# 2024-04-23T16:33:41.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc23590:::benchmark_data_smithi188_113957_object89243:head# 2024-04-23T16:33:41.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc27043:::benchmark_data_smithi188_113957_object72839:head# 2024-04-23T16:33:41.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc284ef:::benchmark_data_smithi188_113957_object71781:head# 2024-04-23T16:33:41.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc31c73:::benchmark_data_smithi188_113957_object4504:head# 2024-04-23T16:33:41.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc342c4:::benchmark_data_smithi188_113957_object13561:head# 2024-04-23T16:33:41.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc34c70:::benchmark_data_smithi188_113957_object141451:head# 2024-04-23T16:33:41.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc35fbf:::benchmark_data_smithi188_113957_object88006:head# 2024-04-23T16:33:41.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc3fd68:::benchmark_data_smithi188_113957_object11513:head# 2024-04-23T16:33:41.564 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc44b8f:::benchmark_data_smithi188_113957_object34873:head# 2024-04-23T16:33:41.564 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc473d2:::benchmark_data_smithi188_113957_object55148:head# 2024-04-23T16:33:41.564 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc63e73:::benchmark_data_smithi188_113957_object105617:head# 2024-04-23T16:33:41.565 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc67258:::benchmark_data_smithi188_113957_object86264:head# 2024-04-23T16:33:41.565 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc7a3fc:::benchmark_data_smithi188_113957_object147385:head# 2024-04-23T16:33:41.566 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc8011f:::benchmark_data_smithi188_113957_object101954:head# 2024-04-23T16:33:41.566 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc8359f:::benchmark_data_smithi188_113957_object50201:head# 2024-04-23T16:33:41.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc83d6a:::benchmark_data_smithi188_113957_object125521:head# 2024-04-23T16:33:41.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc8b30b:::benchmark_data_smithi188_113957_object27304:head# 2024-04-23T16:33:41.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc94fe3:::benchmark_data_smithi188_113957_object2147:head# 2024-04-23T16:33:41.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc96143:::benchmark_data_smithi188_113957_object80883:head# 2024-04-23T16:33:41.569 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc9db9d:::benchmark_data_smithi188_113957_object118265:head# 2024-04-23T16:33:41.569 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc9dbe4:::benchmark_data_smithi188_113957_object106856:head# 2024-04-23T16:33:41.569 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dc9e790:::benchmark_data_smithi188_113957_object96380:head# 2024-04-23T16:33:41.570 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dca3bf0:::benchmark_data_smithi188_113957_object64241:head# 2024-04-23T16:33:41.570 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dca4f71:::benchmark_data_smithi188_113957_object75076:head# 2024-04-23T16:33:41.571 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dca8fd9:::benchmark_data_smithi188_113957_object9336:head# 2024-04-23T16:33:41.571 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dcad90c:::benchmark_data_smithi188_113957_object69531:head# 2024-04-23T16:33:41.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dcb3f0e:::benchmark_data_smithi188_113957_object18251:head# 2024-04-23T16:33:41.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dcc83cb:::benchmark_data_smithi188_113957_object55907:head# 2024-04-23T16:33:41.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dcc91b5:::benchmark_data_smithi188_113957_object18469:head# 2024-04-23T16:33:41.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dcd3e1c:::benchmark_data_smithi188_113957_object70983:head# 2024-04-23T16:33:41.574 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dcd5525:::benchmark_data_smithi188_113957_object115658:head# 2024-04-23T16:33:41.574 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dcd936e:::benchmark_data_smithi188_113957_object77163:head# 2024-04-23T16:33:41.574 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dcdc495:::benchmark_data_smithi188_113957_object24285:head# 2024-04-23T16:33:41.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dcdeb93:::benchmark_data_smithi188_113957_object93988:head# 2024-04-23T16:33:41.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dcea8a5:::benchmark_data_smithi188_113957_object146824:head# 2024-04-23T16:33:41.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dcf1e4e:::benchmark_data_smithi188_113957_object25958:head# 2024-04-23T16:33:41.576 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd12937:::benchmark_data_smithi188_113957_object32311:head# 2024-04-23T16:33:41.576 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd1415b:::benchmark_data_smithi188_113957_object10079:head# 2024-04-23T16:33:41.577 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd23e54:::benchmark_data_smithi188_113957_object115177:head# 2024-04-23T16:33:41.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd2594c:::benchmark_data_smithi188_113957_object26629:head# 2024-04-23T16:33:41.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd2a0b4:::benchmark_data_smithi188_113957_object67968:head# 2024-04-23T16:33:41.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd32688:::benchmark_data_smithi188_113957_object54370:head# 2024-04-23T16:33:41.579 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd409e5:::benchmark_data_smithi188_113957_object134295:head# 2024-04-23T16:33:41.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd41e3e:::benchmark_data_smithi188_113957_object10704:head# 2024-04-23T16:33:41.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd4c8aa:::benchmark_data_smithi188_113957_object43922:head# 2024-04-23T16:33:41.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd5198a:::benchmark_data_smithi188_113957_object116728:head# 2024-04-23T16:33:41.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd61891:::benchmark_data_smithi188_113957_object132251:head# 2024-04-23T16:33:41.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd62782:::benchmark_data_smithi188_113957_object121312:head# 2024-04-23T16:33:41.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd62e66:::benchmark_data_smithi188_113957_object118733:head# 2024-04-23T16:33:41.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd7487f:::benchmark_data_smithi188_113957_object95124:head# 2024-04-23T16:33:41.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd74b46:::benchmark_data_smithi188_113957_object48737:head# 2024-04-23T16:33:41.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd8a813:::benchmark_data_smithi188_113957_object141980:head# 2024-04-23T16:33:41.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd93baa:::benchmark_data_smithi188_113957_object107347:head# 2024-04-23T16:33:41.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd9d92e:::benchmark_data_smithi188_113957_object78913:head# 2024-04-23T16:33:41.585 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd9ef79:::benchmark_data_smithi188_113957_object34770:head# 2024-04-23T16:33:41.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dd9f601:::benchmark_data_smithi188_113957_object67833:head# 2024-04-23T16:33:41.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dda4a7f:::benchmark_data_smithi188_113957_object12475:head# 2024-04-23T16:33:41.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dda83df:::benchmark_data_smithi188_113957_object40813:head# 2024-04-23T16:33:41.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ddad1a3:::benchmark_data_smithi188_113957_object119346:head# 2024-04-23T16:33:41.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ddb52ef:::benchmark_data_smithi188_113957_object87574:head# 2024-04-23T16:33:41.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ddbaa9e:::benchmark_data_smithi188_113957_object9171:head# 2024-04-23T16:33:41.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dddeddd:::benchmark_data_smithi188_113957_object12020:head# 2024-04-23T16:33:41.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dde2c93:::benchmark_data_smithi188_113957_object98181:head# 2024-04-23T16:33:41.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dde6f94:::benchmark_data_smithi188_113957_object124082:head# 2024-04-23T16:33:41.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ddee485:::benchmark_data_smithi188_113957_object111383:head# 2024-04-23T16:33:41.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ddf6c2d:::benchmark_data_smithi188_113957_object45010:head# 2024-04-23T16:33:41.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ddfa0d7:::benchmark_data_smithi188_113957_object95390:head# 2024-04-23T16:33:41.592 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ddfcb35:::benchmark_data_smithi188_113957_object114280:head# 2024-04-23T16:33:41.592 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de0025b:::benchmark_data_smithi188_113957_object74292:head# 2024-04-23T16:33:41.593 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de0730c:::benchmark_data_smithi188_113957_object31821:head# 2024-04-23T16:33:41.593 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de0f26c:::benchmark_data_smithi188_113957_object99189:head# 2024-04-23T16:33:41.594 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de11948:::benchmark_data_smithi188_113957_object50457:head# 2024-04-23T16:33:41.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de1f0b5:::benchmark_data_smithi188_113957_object99213:head# 2024-04-23T16:33:41.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de28ed9:::benchmark_data_smithi188_113957_object33539:head# 2024-04-23T16:33:41.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de3712c:::benchmark_data_smithi188_113957_object117618:head# 2024-04-23T16:33:41.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de39150:::benchmark_data_smithi188_113957_object32037:head# 2024-04-23T16:33:41.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de3b778:::benchmark_data_smithi188_113957_object51605:head# 2024-04-23T16:33:41.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de3cc99:::benchmark_data_smithi188_113957_object77372:head# 2024-04-23T16:33:41.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de4a570:::benchmark_data_smithi188_113957_object106831:head# 2024-04-23T16:33:41.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de4c3b8:::benchmark_data_smithi188_113957_object144801:head# 2024-04-23T16:33:41.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de4e183:::benchmark_data_smithi188_113957_object121284:head# 2024-04-23T16:33:41.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de56985:::benchmark_data_smithi188_113957_object110752:head# 2024-04-23T16:33:41.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de5c510:::benchmark_data_smithi188_113957_object6271:head# 2024-04-23T16:33:41.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de626fe:::benchmark_data_smithi188_113957_object32241:head# 2024-04-23T16:33:41.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de79a05:::benchmark_data_smithi188_113957_object15492:head# 2024-04-23T16:33:41.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de7d3d6:::benchmark_data_smithi188_113957_object124410:head# 2024-04-23T16:33:41.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de7f606:::benchmark_data_smithi188_113957_object92698:head# 2024-04-23T16:33:41.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de87fab:::benchmark_data_smithi188_113957_object114568:head# 2024-04-23T16:33:41.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de978c5:::benchmark_data_smithi188_113957_object30833:head# 2024-04-23T16:33:41.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de98f21:::benchmark_data_smithi188_113957_object106446:head# 2024-04-23T16:33:41.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de9bc22:::benchmark_data_smithi188_113957_object113492:head# 2024-04-23T16:33:41.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5de9c228:::benchmark_data_smithi188_113957_object110619:head# 2024-04-23T16:33:41.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dea4ab6:::benchmark_data_smithi188_113957_object96756:head# 2024-04-23T16:33:41.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5debc6d4:::benchmark_data_smithi188_113957_object83269:head# 2024-04-23T16:33:41.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5debe4bd:::benchmark_data_smithi188_113957_object129871:head# 2024-04-23T16:33:41.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dece465:::benchmark_data_smithi188_113957_object122495:head# 2024-04-23T16:33:41.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5decf61d:::benchmark_data_smithi188_113957_object134201:head# 2024-04-23T16:33:41.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ded16c2:::benchmark_data_smithi188_113957_object41986:head# 2024-04-23T16:33:41.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ded4e62:::benchmark_data_smithi188_113957_object113311:head# 2024-04-23T16:33:41.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ded6daf:::benchmark_data_smithi188_113957_object54024:head# 2024-04-23T16:33:41.609 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ded9fd8:::benchmark_data_smithi188_113957_object89224:head# 2024-04-23T16:33:41.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dedb6fc:::benchmark_data_smithi188_113957_object101401:head# 2024-04-23T16:33:41.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dedc03b:::benchmark_data_smithi188_113957_object114293:head# 2024-04-23T16:33:41.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dede062:::benchmark_data_smithi188_113957_object39006:head# 2024-04-23T16:33:41.612 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:33:41.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dede304:::benchmark_data_smithi188_113957_object108237:head# 2024-04-23T16:33:41.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dedfb70:::benchmark_data_smithi188_113957_object131046:head# 2024-04-23T16:33:41.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dee230b:::benchmark_data_smithi188_113957_object45784:head# 2024-04-23T16:33:41.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5deef475:::benchmark_data_smithi188_113957_object144129:head# 2024-04-23T16:33:41.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5def4b01:::benchmark_data_smithi188_113957_object81015:head# 2024-04-23T16:33:41.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5def71d2:::benchmark_data_smithi188_113957_object137526:head# 2024-04-23T16:33:41.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5deff469:::benchmark_data_smithi188_113957_object123183:head# 2024-04-23T16:33:41.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5df19cb9:::benchmark_data_smithi188_113957_object122732:head# 2024-04-23T16:33:41.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5df1f041:::benchmark_data_smithi188_113957_object92276:head# 2024-04-23T16:33:41.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5df266d3:::benchmark_data_smithi188_113957_object21571:head# 2024-04-23T16:33:41.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5df2dac6:::benchmark_data_smithi188_113957_object75124:head# 2024-04-23T16:33:41.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5df4f2cb:::benchmark_data_smithi188_113957_object10424:head# 2024-04-23T16:33:41.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5df68cd4:::benchmark_data_smithi188_113957_object66887:head# 2024-04-23T16:33:41.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5df68f50:::benchmark_data_smithi188_113957_object112439:head# 2024-04-23T16:33:41.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5df6e438:::benchmark_data_smithi188_113957_object62949:head# 2024-04-23T16:33:41.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5df729f5:::benchmark_data_smithi188_113957_object100853:head# 2024-04-23T16:33:41.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5df72e65:::benchmark_data_smithi188_113957_object140788:head# 2024-04-23T16:33:41.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5df793bc:::benchmark_data_smithi188_113957_object98901:head# 2024-04-23T16:33:41.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5df8b900:::benchmark_data_smithi188_113957_object149689:head# 2024-04-23T16:33:41.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5df93f07:::benchmark_data_smithi188_113957_object74385:head# 2024-04-23T16:33:41.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5df9f1ef:::benchmark_data_smithi188_113957_object75658:head# 2024-04-23T16:33:41.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dfb15f8:::benchmark_data_smithi188_113957_object27723:head# 2024-04-23T16:33:41.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dfb7b20:::benchmark_data_smithi188_113957_object11831:head# 2024-04-23T16:33:41.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:41.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:33:41.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dfb928c:::benchmark_data_smithi188_113957_object32313:head# 2024-04-23T16:33:41.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dfbbf13:::benchmark_data_smithi188_113957_object132036:head# 2024-04-23T16:33:41.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dfcc625:::benchmark_data_smithi188_113957_object115988:head# 2024-04-23T16:33:41.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dfcd964:::benchmark_data_smithi188_113957_object18523:head# 2024-04-23T16:33:41.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dfe5362:::benchmark_data_smithi188_113957_object5203:head# 2024-04-23T16:33:41.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dfe5d44:::benchmark_data_smithi188_113957_object27306:head# 2024-04-23T16:33:41.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5dfedfac:::benchmark_data_smithi188_113957_object4787:head# 2024-04-23T16:33:41.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e0024b0:::benchmark_data_smithi188_113957_object68296:head# 2024-04-23T16:33:41.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e009dbf:::benchmark_data_smithi188_113957_object122498:head# 2024-04-23T16:33:41.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e00d78b:::benchmark_data_smithi188_113957_object40976:head# 2024-04-23T16:33:41.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e0123c9:::benchmark_data_smithi188_113957_object72860:head# 2024-04-23T16:33:41.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e016778:::benchmark_data_smithi188_113957_object65354:head# 2024-04-23T16:33:41.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e0224b1:::benchmark_data_smithi188_113957_object37024:head# 2024-04-23T16:33:41.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e0238a2:::benchmark_data_smithi188_113957_object24565:head# 2024-04-23T16:33:41.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e024d6c:::benchmark_data_smithi188_113957_object116878:head# 2024-04-23T16:33:41.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e03736c:::benchmark_data_smithi188_113957_object25570:head# 2024-04-23T16:33:41.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e03925c:::benchmark_data_smithi188_113957_object31563:head# 2024-04-23T16:33:41.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e03a715:::benchmark_data_smithi188_113957_object137792:head# 2024-04-23T16:33:41.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e03aa55:::benchmark_data_smithi188_113957_object105527:head# 2024-04-23T16:33:41.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e03b9f8:::benchmark_data_smithi188_113957_object48930:head# 2024-04-23T16:33:41.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e03d713:::benchmark_data_smithi188_113957_object134866:head# 2024-04-23T16:33:41.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e06ea4d:::benchmark_data_smithi188_113957_object8902:head# 2024-04-23T16:33:41.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e072103:::benchmark_data_smithi188_113957_object19677:head# 2024-04-23T16:33:41.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e075114:::benchmark_data_smithi188_113957_object126601:head# 2024-04-23T16:33:41.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e076c30:::benchmark_data_smithi188_113957_object2306:head# 2024-04-23T16:33:41.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e0780b7:::benchmark_data_smithi188_113957_object73712:head# 2024-04-23T16:33:41.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e080bf8:::benchmark_data_smithi188_113957_object142929:head# 2024-04-23T16:33:41.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e0912b1:::benchmark_data_smithi188_113957_object56432:head# 2024-04-23T16:33:41.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e09235f:::benchmark_data_smithi188_113957_object59444:head# 2024-04-23T16:33:41.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e0a3bcc:::benchmark_data_smithi188_113957_object26760:head# 2024-04-23T16:33:41.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e0b656d:::benchmark_data_smithi188_113957_object40299:head# 2024-04-23T16:33:41.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e0bcbf7:::benchmark_data_smithi188_113957_object80785:head# 2024-04-23T16:33:41.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e0cbd92:::benchmark_data_smithi188_113957_object49771:head# 2024-04-23T16:33:41.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e0d9514:::benchmark_data_smithi188_113957_object44629:head# 2024-04-23T16:33:41.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e0dd5c0:::benchmark_data_smithi188_113957_object4503:head# 2024-04-23T16:33:41.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e0fe7a5:::benchmark_data_smithi188_113957_object106888:head# 2024-04-23T16:33:41.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e0ffc85:::benchmark_data_smithi188_113957_object136256:head# 2024-04-23T16:33:41.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e101ca7:::benchmark_data_smithi188_113957_object105269:head# 2024-04-23T16:33:41.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e101f20:::benchmark_data_smithi188_113957_object137491:head# 2024-04-23T16:33:41.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e103ee9:::benchmark_data_smithi188_113957_object108058:head# 2024-04-23T16:33:41.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e104a85:::benchmark_data_smithi188_113957_object80642:head# 2024-04-23T16:33:41.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e104f41:::benchmark_data_smithi188_113957_object86809:head# 2024-04-23T16:33:41.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e106ac8:::benchmark_data_smithi188_113957_object14965:head# 2024-04-23T16:33:41.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1143c7:::benchmark_data_smithi188_113957_object70715:head# 2024-04-23T16:33:41.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e117818:::benchmark_data_smithi188_113957_object86128:head# 2024-04-23T16:33:41.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e11a4ea:::benchmark_data_smithi188_113957_object125452:head# 2024-04-23T16:33:41.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e11b5d0:::benchmark_data_smithi188_113957_object36427:head# 2024-04-23T16:33:41.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e11c2c6:::benchmark_data_smithi188_113957_object25207:head# 2024-04-23T16:33:41.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1211a4:::benchmark_data_smithi188_113957_object40144:head# 2024-04-23T16:33:41.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e12371d:::benchmark_data_smithi188_113957_object32913:head# 2024-04-23T16:33:41.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e12d73b:::benchmark_data_smithi188_113957_object19319:head# 2024-04-23T16:33:41.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e138fef:::benchmark_data_smithi188_113957_object91208:head# 2024-04-23T16:33:41.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e13b944:::benchmark_data_smithi188_113957_object29550:head# 2024-04-23T16:33:41.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e145b86:::benchmark_data_smithi188_113957_object131291:head# 2024-04-23T16:33:41.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e14a3b1:::benchmark_data_smithi188_113957_object121896:head# 2024-04-23T16:33:41.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e14f471:::benchmark_data_smithi188_113957_object109918:head# 2024-04-23T16:33:41.645 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e157071:::benchmark_data_smithi188_113957_object42351:head# 2024-04-23T16:33:41.645 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1573c6:::benchmark_data_smithi188_113957_object99036:head# 2024-04-23T16:33:41.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e15b670:::benchmark_data_smithi188_113957_object79525:head# 2024-04-23T16:33:41.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e15d92b:::benchmark_data_smithi188_113957_object87152:head# 2024-04-23T16:33:41.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1600fd:::benchmark_data_smithi188_113957_object85305:head# 2024-04-23T16:33:41.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e16606c:::benchmark_data_smithi188_113957_object24089:head# 2024-04-23T16:33:41.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1696e8:::benchmark_data_smithi188_113957_object97304:head# 2024-04-23T16:33:41.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e169f17:::benchmark_data_smithi188_113957_object86439:head# 2024-04-23T16:33:41.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e16d36d:::benchmark_data_smithi188_113957_object17952:head# 2024-04-23T16:33:41.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e183e79:::benchmark_data_smithi188_113957_object15523:head# 2024-04-23T16:33:41.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1904ab:::benchmark_data_smithi188_113957_object151011:head# 2024-04-23T16:33:41.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e193991:::benchmark_data_smithi188_113957_object92923:head# 2024-04-23T16:33:41.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1a1b8f:::benchmark_data_smithi188_113957_object19766:head# 2024-04-23T16:33:41.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1a5174:::benchmark_data_smithi188_113957_object39684:head# 2024-04-23T16:33:41.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1a559b:::benchmark_data_smithi188_113957_object97418:head# 2024-04-23T16:33:41.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1ab87c:::benchmark_data_smithi188_113957_object104072:head# 2024-04-23T16:33:41.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1ac5c2:::benchmark_data_smithi188_113957_object135537:head# 2024-04-23T16:33:41.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1b0d42:::benchmark_data_smithi188_113957_object59237:head# 2024-04-23T16:33:41.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1b3189:::benchmark_data_smithi188_113957_object75765:head# 2024-04-23T16:33:41.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1c26e9:::benchmark_data_smithi188_113957_object126995:head# 2024-04-23T16:33:41.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1d6ca1:::benchmark_data_smithi188_113957_object36940:head# 2024-04-23T16:33:41.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1ddbca:::benchmark_data_smithi188_113957_object15248:head# 2024-04-23T16:33:41.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1e1fa8:::benchmark_data_smithi188_113957_object49981:head# 2024-04-23T16:33:41.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1e4be0:::benchmark_data_smithi188_113957_object37630:head# 2024-04-23T16:33:41.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1e6b89:::benchmark_data_smithi188_113957_object20452:head# 2024-04-23T16:33:41.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1e6fa6:::benchmark_data_smithi188_113957_object49399:head# 2024-04-23T16:33:41.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1e87d4:::benchmark_data_smithi188_113957_object114092:head# 2024-04-23T16:33:41.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1e8aba:::benchmark_data_smithi188_113957_object32986:head# 2024-04-23T16:33:41.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1f03f1:::benchmark_data_smithi188_113957_object86680:head# 2024-04-23T16:33:41.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1f5244:::benchmark_data_smithi188_113957_object122194:head# 2024-04-23T16:33:41.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1f7c34:::benchmark_data_smithi188_113957_object13888:head# 2024-04-23T16:33:41.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e1fa8e8:::benchmark_data_smithi188_113957_object12915:head# 2024-04-23T16:33:41.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e212093:::benchmark_data_smithi188_113957_object108163:head# 2024-04-23T16:33:41.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2142ec:::benchmark_data_smithi188_113957_object41016:head# 2024-04-23T16:33:41.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e216541:::benchmark_data_smithi188_113957_object86426:head# 2024-04-23T16:33:41.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e22d759:::benchmark_data_smithi188_113957_object89073:head# 2024-04-23T16:33:41.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e22e497:::benchmark_data_smithi188_113957_object4220:head# 2024-04-23T16:33:41.661 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e23047c:::benchmark_data_smithi188_113957_object5082:head# 2024-04-23T16:33:41.661 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2306a5:::benchmark_data_smithi188_113957_object34622:head# 2024-04-23T16:33:41.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e232611:::benchmark_data_smithi188_113957_object31837:head# 2024-04-23T16:33:41.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e232a6b:::benchmark_data_smithi188_113957_object15444:head# 2024-04-23T16:33:41.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2362a1:::benchmark_data_smithi188_113957_object79330:head# 2024-04-23T16:33:41.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e23bf0d:::benchmark_data_smithi188_113957_object143462:head# 2024-04-23T16:33:41.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e243e39:::benchmark_data_smithi188_113957_object139553:head# 2024-04-23T16:33:41.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2461cb:::benchmark_data_smithi188_113957_object138826:head# 2024-04-23T16:33:41.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e24b7d4:::benchmark_data_smithi188_113957_object97583:head# 2024-04-23T16:33:41.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e25323a:::benchmark_data_smithi188_113957_object12332:head# 2024-04-23T16:33:41.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e25d44c:::benchmark_data_smithi188_113957_object141813:head# 2024-04-23T16:33:41.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e26107c:::benchmark_data_smithi188_113957_object111196:head# 2024-04-23T16:33:41.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e26dc03:::benchmark_data_smithi188_113957_object58773:head# 2024-04-23T16:33:41.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2750cc:::benchmark_data_smithi188_113957_object102664:head# 2024-04-23T16:33:41.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e275431:::benchmark_data_smithi188_113957_object103894:head# 2024-04-23T16:33:41.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e276954:::benchmark_data_smithi188_113957_object52358:head# 2024-04-23T16:33:41.668 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e276a96:::benchmark_data_smithi188_113957_object56329:head# 2024-04-23T16:33:41.668 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e28879b:::benchmark_data_smithi188_113957_object152823:head# 2024-04-23T16:33:41.669 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e28da7e:::benchmark_data_smithi188_113957_object143921:head# 2024-04-23T16:33:41.669 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e29714c:::benchmark_data_smithi188_113957_object25885:head# 2024-04-23T16:33:41.669 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2a1d32:::benchmark_data_smithi188_113957_object77074:head# 2024-04-23T16:33:41.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2b039c:::benchmark_data_smithi188_113957_object16108:head# 2024-04-23T16:33:41.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2b38a0:::benchmark_data_smithi188_113957_object127294:head# 2024-04-23T16:33:41.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2bb758:::benchmark_data_smithi188_113957_object109218:head# 2024-04-23T16:33:41.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2be29e:::benchmark_data_smithi188_113957_object44252:head# 2024-04-23T16:33:41.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2c11e6:::benchmark_data_smithi188_113957_object138135:head# 2024-04-23T16:33:41.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2c21ae:::benchmark_data_smithi188_113957_object91251:head# 2024-04-23T16:33:41.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2c7965:::benchmark_data_smithi188_113957_object70830:head# 2024-04-23T16:33:41.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2cb857:::benchmark_data_smithi188_113957_object23106:head# 2024-04-23T16:33:41.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2cca38:::benchmark_data_smithi188_113957_object118650:head# 2024-04-23T16:33:41.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2d29f0:::benchmark_data_smithi188_113957_object130441:head# 2024-04-23T16:33:41.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2d409e:::benchmark_data_smithi188_113957_object64602:head# 2024-04-23T16:33:41.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2d62b2:::benchmark_data_smithi188_113957_object131638:head# 2024-04-23T16:33:41.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2dd920:::benchmark_data_smithi188_113957_object112559:head# 2024-04-23T16:33:41.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2e37a3:::benchmark_data_smithi188_113957_object101455:head# 2024-04-23T16:33:41.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2e8bb0:::benchmark_data_smithi188_113957_object111622:head# 2024-04-23T16:33:41.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2fbee4:::benchmark_data_smithi188_113957_object147597:head# 2024-04-23T16:33:41.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2fcb2d:::benchmark_data_smithi188_113957_object18791:head# 2024-04-23T16:33:41.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e2ff4d9:::benchmark_data_smithi188_113957_object50850:head# 2024-04-23T16:33:41.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e307912:::benchmark_data_smithi188_113957_object31836:head# 2024-04-23T16:33:41.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e30e472:::benchmark_data_smithi188_113957_object81127:head# 2024-04-23T16:33:41.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e315ffd:::benchmark_data_smithi188_113957_object82776:head# 2024-04-23T16:33:41.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e3193c9:::benchmark_data_smithi188_113957_object132020:head# 2024-04-23T16:33:41.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e319de7:::benchmark_data_smithi188_113957_object102299:head# 2024-04-23T16:33:41.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e31b7da:::benchmark_data_smithi188_113957_object113266:head# 2024-04-23T16:33:41.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e31cb51:::benchmark_data_smithi188_113957_object8553:head# 2024-04-23T16:33:41.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e326619:::benchmark_data_smithi188_113957_object148940:head# 2024-04-23T16:33:41.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e327842:::benchmark_data_smithi188_113957_object56402:head# 2024-04-23T16:33:41.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e32cff3:::benchmark_data_smithi188_113957_object90263:head# 2024-04-23T16:33:41.683 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e33cbf9:::benchmark_data_smithi188_113957_object126122:head# 2024-04-23T16:33:41.683 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e343d7f:::benchmark_data_smithi188_113957_object4543:head# 2024-04-23T16:33:41.683 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e34c76b:::benchmark_data_smithi188_113957_object119738:head# 2024-04-23T16:33:41.683 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e352216:::benchmark_data_smithi188_113957_object82800:head# 2024-04-23T16:33:41.683 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e35628e:::benchmark_data_smithi188_113957_object83438:head# 2024-04-23T16:33:41.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e374907:::benchmark_data_smithi188_113957_object27936:head# 2024-04-23T16:33:41.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e375b6c:::benchmark_data_smithi188_113957_object101946:head# 2024-04-23T16:33:41.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e37ba63:::benchmark_data_smithi188_113957_object146775:head# 2024-04-23T16:33:41.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e37fe4a:::benchmark_data_smithi188_113957_object18593:head# 2024-04-23T16:33:41.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e382039:::benchmark_data_smithi188_113957_object97427:head# 2024-04-23T16:33:41.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e382548:::benchmark_data_smithi188_113957_object18743:head# 2024-04-23T16:33:41.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e392a3b:::benchmark_data_smithi188_113957_object2186:head# 2024-04-23T16:33:41.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e39eaec:::benchmark_data_smithi188_113957_object63736:head# 2024-04-23T16:33:41.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e3a2499:::benchmark_data_smithi188_113957_object81471:head# 2024-04-23T16:33:41.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e3cea6e:::benchmark_data_smithi188_113957_object83685:head# 2024-04-23T16:33:41.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e3e12b9:::benchmark_data_smithi188_113957_object109824:head# 2024-04-23T16:33:41.689 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e3f047d:::benchmark_data_smithi188_113957_object74600:head# 2024-04-23T16:33:41.689 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e3f2d55:::benchmark_data_smithi188_113957_object75221:head# 2024-04-23T16:33:41.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e3f95fa:::benchmark_data_smithi188_113957_object10470:head# 2024-04-23T16:33:41.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e3fbb7a:::benchmark_data_smithi188_113957_object5723:head# 2024-04-23T16:33:41.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e40c757:::benchmark_data_smithi188_113957_object117155:head# 2024-04-23T16:33:41.691 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e40cb03:::benchmark_data_smithi188_113957_object148943:head# 2024-04-23T16:33:41.694 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e41d115:::benchmark_data_smithi188_113957_object15393:head# 2024-04-23T16:33:41.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e420f65:::benchmark_data_smithi188_113957_object137902:head# 2024-04-23T16:33:41.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e42a314:::benchmark_data_smithi188_113957_object73824:head# 2024-04-23T16:33:41.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e43d638:::benchmark_data_smithi188_113957_object47307:head# 2024-04-23T16:33:41.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e43ea2b:::benchmark_data_smithi188_113957_object71615:head# 2024-04-23T16:33:41.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e43f218:::benchmark_data_smithi188_113957_object61419:head# 2024-04-23T16:33:41.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e44b6cf:::benchmark_data_smithi188_113957_object92676:head# 2024-04-23T16:33:41.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e44dc66:::benchmark_data_smithi188_113957_object49373:head# 2024-04-23T16:33:41.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e457acf:::benchmark_data_smithi188_113957_object56215:head# 2024-04-23T16:33:41.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e45d940:::benchmark_data_smithi188_113957_object138099:head# 2024-04-23T16:33:41.699 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e460cfb:::benchmark_data_smithi188_113957_object84465:head# 2024-04-23T16:33:41.699 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e469dbc:::benchmark_data_smithi188_113957_object58242:head# 2024-04-23T16:33:41.699 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e472bee:::benchmark_data_smithi188_113957_object67161:head# 2024-04-23T16:33:41.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e473efc:::benchmark_data_smithi188_113957_object105147:head# 2024-04-23T16:33:41.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e474e14:::benchmark_data_smithi188_113957_object59387:head# 2024-04-23T16:33:41.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e4781d3:::benchmark_data_smithi188_113957_object87704:head# 2024-04-23T16:33:41.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e478a8d:::benchmark_data_smithi188_113957_object110956:head# 2024-04-23T16:33:41.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e48293a:::benchmark_data_smithi188_113957_object54743:head# 2024-04-23T16:33:41.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e482ef7:::benchmark_data_smithi188_113957_object69399:head# 2024-04-23T16:33:41.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e488f89:::benchmark_data_smithi188_113957_object48151:head# 2024-04-23T16:33:41.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e48b046:::benchmark_data_smithi188_113957_object37454:head# 2024-04-23T16:33:41.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e48f168:::benchmark_data_smithi188_113957_object142723:head# 2024-04-23T16:33:41.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e492478:::benchmark_data_smithi188_113957_object126885:head# 2024-04-23T16:33:41.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e493395:::benchmark_data_smithi188_113957_object132498:head# 2024-04-23T16:33:41.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e4aeb67:::benchmark_data_smithi188_113957_object8713:head# 2024-04-23T16:33:41.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e4aeb87:::benchmark_data_smithi188_113957_object143851:head# 2024-04-23T16:33:41.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e4b77c7:::benchmark_data_smithi188_113957_object126720:head# 2024-04-23T16:33:41.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e4bbbb5:::benchmark_data_smithi188_113957_object24186:head# 2024-04-23T16:33:41.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e4bc623:::benchmark_data_smithi188_113957_object135324:head# 2024-04-23T16:33:41.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e4c6e20:::benchmark_data_smithi188_113957_object37348:head# 2024-04-23T16:33:41.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e4c87a0:::benchmark_data_smithi188_113957_object48437:head# 2024-04-23T16:33:41.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e4cd828:::benchmark_data_smithi188_113957_object3085:head# 2024-04-23T16:33:41.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e4cfc72:::benchmark_data_smithi188_113957_object62097:head# 2024-04-23T16:33:41.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e4d3bfb:::benchmark_data_smithi188_113957_object21487:head# 2024-04-23T16:33:41.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e4eeb80:::benchmark_data_smithi188_113957_object2564:head# 2024-04-23T16:33:41.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e4eeea2:::benchmark_data_smithi188_113957_object38836:head# 2024-04-23T16:33:41.709 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e4f9e4f:::benchmark_data_smithi188_113957_object122815:head# 2024-04-23T16:33:41.709 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e50478e:::benchmark_data_smithi188_113957_object85869:head# 2024-04-23T16:33:41.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e506852:::benchmark_data_smithi188_113957_object74054:head# 2024-04-23T16:33:41.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e509ce4:::benchmark_data_smithi188_113957_object63477:head# 2024-04-23T16:33:41.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e50a058:::benchmark_data_smithi188_113957_object44403:head# 2024-04-23T16:33:41.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e512a5e:::benchmark_data_smithi188_113957_object4828:head# 2024-04-23T16:33:41.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e5163a1:::benchmark_data_smithi188_113957_object101011:head# 2024-04-23T16:33:41.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e51c4db:::benchmark_data_smithi188_113957_object3466:head# 2024-04-23T16:33:41.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e51e9b3:::benchmark_data_smithi188_113957_object13435:head# 2024-04-23T16:33:41.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e51f4cf:::benchmark_data_smithi188_113957_object98313:head# 2024-04-23T16:33:41.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e525f4f:::benchmark_data_smithi188_113957_object144824:head# 2024-04-23T16:33:41.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e5281aa:::benchmark_data_smithi188_113957_object40847:head# 2024-04-23T16:33:41.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e54372b:::benchmark_data_smithi188_113957_object62138:head# 2024-04-23T16:33:41.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e559805:::benchmark_data_smithi188_113957_object135405:head# 2024-04-23T16:33:41.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e56dde8:::benchmark_data_smithi188_113957_object17884:head# 2024-04-23T16:33:41.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e57b3af:::benchmark_data_smithi188_113957_object103257:head# 2024-04-23T16:33:41.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e58a1c1:::benchmark_data_smithi188_113957_object118565:head# 2024-04-23T16:33:41.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e58cfe2:::benchmark_data_smithi188_113957_object54401:head# 2024-04-23T16:33:41.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e5949fc:::benchmark_data_smithi188_113957_object47591:head# 2024-04-23T16:33:41.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e5975a7:::benchmark_data_smithi188_113957_object124275:head# 2024-04-23T16:33:41.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e59de85:::benchmark_data_smithi188_113957_object33195:head# 2024-04-23T16:33:41.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e59eaae:::benchmark_data_smithi188_113957_object18544:head# 2024-04-23T16:33:41.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e5a31c2:::benchmark_data_smithi188_113957_object122893:head# 2024-04-23T16:33:41.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e5a83d1:::benchmark_data_smithi188_113957_object83921:head# 2024-04-23T16:33:41.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e5a8e4e:::benchmark_data_smithi188_113957_object63398:head# 2024-04-23T16:33:41.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e5b5bad:::benchmark_data_smithi188_113957_object2813:head# 2024-04-23T16:33:41.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e5e4b9a:::benchmark_data_smithi188_113957_object7703:head# 2024-04-23T16:33:41.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e5ef9c9:::benchmark_data_smithi188_113957_object140215:head# 2024-04-23T16:33:41.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e603b67:::benchmark_data_smithi188_113957_object77310:head# 2024-04-23T16:33:41.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e6066f8:::benchmark_data_smithi188_113957_object118557:head# 2024-04-23T16:33:41.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e607dca:::benchmark_data_smithi188_113957_object57832:head# 2024-04-23T16:33:41.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e639d60:::benchmark_data_smithi188_113957_object151102:head# 2024-04-23T16:33:41.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e6420b5:::benchmark_data_smithi188_113957_object1721:head# 2024-04-23T16:33:41.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e645352:::benchmark_data_smithi188_113957_object1771:head# 2024-04-23T16:33:41.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e65460c:::benchmark_data_smithi188_113957_object98105:head# 2024-04-23T16:33:41.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e6584ff:::benchmark_data_smithi188_113957_object108175:head# 2024-04-23T16:33:41.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e662ef6:::benchmark_data_smithi188_113957_object106378:head# 2024-04-23T16:33:41.724 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:33:41.725 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e66769e:::benchmark_data_smithi188_113957_object73816:head# 2024-04-23T16:33:41.725 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e6774bf:::benchmark_data_smithi188_113957_object148490:head# 2024-04-23T16:33:41.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e683422:::benchmark_data_smithi188_113957_object30319:head# 2024-04-23T16:33:41.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e685843:::benchmark_data_smithi188_113957_object147521:head# 2024-04-23T16:33:41.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e695020:::benchmark_data_smithi188_113957_object70164:head# 2024-04-23T16:33:41.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e6979d7:::benchmark_data_smithi188_113957_object149968:head# 2024-04-23T16:33:41.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e699982:::benchmark_data_smithi188_113957_object45930:head# 2024-04-23T16:33:41.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e6a77cf:::benchmark_data_smithi188_113957_object13041:head# 2024-04-23T16:33:41.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e6b27ad:::benchmark_data_smithi188_113957_object67730:head# 2024-04-23T16:33:41.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e6b7695:::benchmark_data_smithi188_113957_object27825:head# 2024-04-23T16:33:41.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e6cb06d:::benchmark_data_smithi188_113957_object51305:head# 2024-04-23T16:33:41.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e6cc0dc:::benchmark_data_smithi188_113957_object95999:head# 2024-04-23T16:33:41.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e6d91fc:::benchmark_data_smithi188_113957_object45202:head# 2024-04-23T16:33:41.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e6e2300:::benchmark_data_smithi188_113957_object49563:head# 2024-04-23T16:33:41.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e6eea12:::benchmark_data_smithi188_113957_object49844:head# 2024-04-23T16:33:41.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e6eebeb:::benchmark_data_smithi188_113957_object5628:head# 2024-04-23T16:33:41.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e6f81f4:::benchmark_data_smithi188_113957_object20581:head# 2024-04-23T16:33:41.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7001b6:::benchmark_data_smithi188_113957_object127335:head# 2024-04-23T16:33:41.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e71262f:::benchmark_data_smithi188_113957_object95707:head# 2024-04-23T16:33:41.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7129bf:::benchmark_data_smithi188_113957_object142325:head# 2024-04-23T16:33:41.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e71fde5:::benchmark_data_smithi188_113957_object67194:head# 2024-04-23T16:33:41.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e72243f:::benchmark_data_smithi188_113957_object6424:head# 2024-04-23T16:33:41.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e742130:::benchmark_data_smithi188_113957_object49761:head# 2024-04-23T16:33:41.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e742a58:::benchmark_data_smithi188_113957_object120731:head# 2024-04-23T16:33:41.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e746c8c:::benchmark_data_smithi188_113957_object53137:head# 2024-04-23T16:33:41.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e74a531:::benchmark_data_smithi188_113957_object52215:head# 2024-04-23T16:33:41.735 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e75178e:::benchmark_data_smithi188_113957_object79859:head# 2024-04-23T16:33:41.735 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e75c2af:::benchmark_data_smithi188_113957_object127926:head# 2024-04-23T16:33:41.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e75d18d:::benchmark_data_smithi188_113957_object85768:head# 2024-04-23T16:33:41.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e76260a:::benchmark_data_smithi188_113957_object33067:head# 2024-04-23T16:33:41.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e762e1b:::benchmark_data_smithi188_113957_object73473:head# 2024-04-23T16:33:41.737 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e76c3e6:::benchmark_data_smithi188_113957_object54948:head# 2024-04-23T16:33:41.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e771dbc:::benchmark_data_smithi188_113957_object117564:head# 2024-04-23T16:33:41.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e78a13d:::benchmark_data_smithi188_113957_object113511:head# 2024-04-23T16:33:41.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e78a348:::benchmark_data_smithi188_113957_object152103:head# 2024-04-23T16:33:41.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e79e514:::benchmark_data_smithi188_113957_object42689:head# 2024-04-23T16:33:41.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7a0298:::benchmark_data_smithi188_113957_object60051:head# 2024-04-23T16:33:41.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7ad008:::benchmark_data_smithi188_113957_object6378:head# 2024-04-23T16:33:41.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7b2f3a:::benchmark_data_smithi188_113957_object35294:head# 2024-04-23T16:33:41.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7b87d3:::benchmark_data_smithi188_113957_object122034:head# 2024-04-23T16:33:41.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7c870f:::benchmark_data_smithi188_113957_object99114:head# 2024-04-23T16:33:41.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7ce80d:::benchmark_data_smithi188_113957_object66012:head# 2024-04-23T16:33:41.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7cfc75:::benchmark_data_smithi188_113957_object138747:head# 2024-04-23T16:33:41.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7d3c21:::benchmark_data_smithi188_113957_object56709:head# 2024-04-23T16:33:41.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7d5a2b:::benchmark_data_smithi188_113957_object114980:head# 2024-04-23T16:33:41.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7d5acc:::benchmark_data_smithi188_113957_object130791:head# 2024-04-23T16:33:41.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7d6a88:::benchmark_data_smithi188_113957_object17036:head# 2024-04-23T16:33:41.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7ea9c7:::benchmark_data_smithi188_113957_object39501:head# 2024-04-23T16:33:41.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7eb068:::benchmark_data_smithi188_113957_object46638:head# 2024-04-23T16:33:41.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7ece0e:::benchmark_data_smithi188_113957_object28278:head# 2024-04-23T16:33:41.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7ef13a:::benchmark_data_smithi188_113957_object65977:head# 2024-04-23T16:33:41.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7ef1bb:::benchmark_data_smithi188_113957_object126463:head# 2024-04-23T16:33:41.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7f09f8:::benchmark_data_smithi188_113957_object146095:head# 2024-04-23T16:33:41.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7f0a48:::benchmark_data_smithi188_113957_object88009:head# 2024-04-23T16:33:41.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7f3935:::benchmark_data_smithi188_113957_object113063:head# 2024-04-23T16:33:41.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7f8c82:::benchmark_data_smithi188_113957_object57362:head# 2024-04-23T16:33:41.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7fd9a3:::benchmark_data_smithi188_113957_object109384:head# 2024-04-23T16:33:41.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e7fde27:::benchmark_data_smithi188_113957_object12570:head# 2024-04-23T16:33:41.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e803a64:::benchmark_data_smithi188_113957_object111860:head# 2024-04-23T16:33:41.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e807c19:::benchmark_data_smithi188_113957_object124209:head# 2024-04-23T16:33:41.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e810593:::benchmark_data_smithi188_113957_object91582:head# 2024-04-23T16:33:41.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e8169db:::benchmark_data_smithi188_113957_object100140:head# 2024-04-23T16:33:41.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e81cf65:::benchmark_data_smithi188_113957_object26745:head# 2024-04-23T16:33:41.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e81d13f:::benchmark_data_smithi188_113957_object17880:head# 2024-04-23T16:33:41.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e828465:::benchmark_data_smithi188_113957_object41309:head# 2024-04-23T16:33:41.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e82940f:::benchmark_data_smithi188_113957_object92753:head# 2024-04-23T16:33:41.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e82964a:::benchmark_data_smithi188_113957_object99859:head# 2024-04-23T16:33:41.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e832db2:::benchmark_data_smithi188_113957_object69582:head# 2024-04-23T16:33:41.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e838f90:::benchmark_data_smithi188_113957_object3149:head# 2024-04-23T16:33:41.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e848b81:::benchmark_data_smithi188_113957_object26036:head# 2024-04-23T16:33:41.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e8498eb:::benchmark_data_smithi188_113957_object26106:head# 2024-04-23T16:33:41.754 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:41.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e84e50b:::benchmark_data_smithi188_113957_object70893:head# 2024-04-23T16:33:41.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e851ca3:::benchmark_data_smithi188_113957_object108461:head# 2024-04-23T16:33:41.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e854c87:::benchmark_data_smithi188_113957_object42787:head# 2024-04-23T16:33:41.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e85c82b:::benchmark_data_smithi188_113957_object41855:head# 2024-04-23T16:33:41.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e861cf8:::benchmark_data_smithi188_113957_object135010:head# 2024-04-23T16:33:41.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e8628ed:::benchmark_data_smithi188_113957_object68177:head# 2024-04-23T16:33:41.756 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e86cfa0:::benchmark_data_smithi188_113957_object50905:head# 2024-04-23T16:33:41.756 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e879f07:::benchmark_data_smithi188_113957_object30928:head# 2024-04-23T16:33:41.757 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e88448a:::benchmark_data_smithi188_113957_object142591:head# 2024-04-23T16:33:41.757 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e8849c2:::benchmark_data_smithi188_113957_object37666:head# 2024-04-23T16:33:41.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e897913:::benchmark_data_smithi188_113957_object21805:head# 2024-04-23T16:33:41.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e89a599:::benchmark_data_smithi188_113957_object34436:head# 2024-04-23T16:33:41.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e89a8dd:::benchmark_data_smithi188_113957_object139799:head# 2024-04-23T16:33:41.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e89b8c7:::benchmark_data_smithi188_113957_object16630:head# 2024-04-23T16:33:41.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e89cc2c:::benchmark_data_smithi188_113957_object105313:head# 2024-04-23T16:33:41.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e89d598:::benchmark_data_smithi188_113957_object117948:head# 2024-04-23T16:33:41.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e89e7b6:::benchmark_data_smithi188_113957_object69496:head# 2024-04-23T16:33:41.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e8a1a1e:::benchmark_data_smithi188_113957_object135191:head# 2024-04-23T16:33:41.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e8a2eee:::benchmark_data_smithi188_113957_object45428:head# 2024-04-23T16:33:41.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e8b0554:::benchmark_data_smithi188_113957_object107990:head# 2024-04-23T16:33:41.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e8b8aec:::benchmark_data_smithi188_113957_object97471:head# 2024-04-23T16:33:41.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e8cdffb:::benchmark_data_smithi188_113957_object135921:head# 2024-04-23T16:33:41.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e8de167:::benchmark_data_smithi188_113957_object115972:head# 2024-04-23T16:33:41.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e8e5843:::benchmark_data_smithi188_113957_object9350:head# 2024-04-23T16:33:41.764 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e8e7f26:::benchmark_data_smithi188_113957_object132411:head# 2024-04-23T16:33:41.764 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e8e8235:::benchmark_data_smithi188_113957_object99334:head# 2024-04-23T16:33:41.764 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e8eb374:::benchmark_data_smithi188_113957_object4017:head# 2024-04-23T16:33:41.765 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e8fc5fd:::benchmark_data_smithi188_113957_object61595:head# 2024-04-23T16:33:41.767 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e8fda72:::benchmark_data_smithi188_113957_object64010:head# 2024-04-23T16:33:41.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e911cc3:::benchmark_data_smithi188_113957_object28810:head# 2024-04-23T16:33:41.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e9124f7:::benchmark_data_smithi188_113957_object119613:head# 2024-04-23T16:33:41.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e9131bd:::benchmark_data_smithi188_113957_object128375:head# 2024-04-23T16:33:41.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e929331:::benchmark_data_smithi188_113957_object136790:head# 2024-04-23T16:33:41.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e92b636:::benchmark_data_smithi188_113957_object116107:head# 2024-04-23T16:33:41.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e92cf84:::benchmark_data_smithi188_113957_object141291:head# 2024-04-23T16:33:41.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e930826:::benchmark_data_smithi188_113957_object7783:head# 2024-04-23T16:33:41.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e93fb97:::benchmark_data_smithi188_113957_object61232:head# 2024-04-23T16:33:41.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e9434e9:::benchmark_data_smithi188_113957_object82580:head# 2024-04-23T16:33:41.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e944dad:::benchmark_data_smithi188_113957_object97399:head# 2024-04-23T16:33:41.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e945a6c:::benchmark_data_smithi188_113957_object93756:head# 2024-04-23T16:33:41.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e947290:::benchmark_data_smithi188_113957_object118745:head# 2024-04-23T16:33:41.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e949e2e:::benchmark_data_smithi188_113957_object26493:head# 2024-04-23T16:33:41.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e94e170:::benchmark_data_smithi188_113957_object104663:head# 2024-04-23T16:33:41.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e950c42:::benchmark_data_smithi188_113957_object58943:head# 2024-04-23T16:33:41.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e955e92:::benchmark_data_smithi188_113957_object49793:head# 2024-04-23T16:33:41.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e957583:::benchmark_data_smithi188_113957_object108641:head# 2024-04-23T16:33:41.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e957a82:::benchmark_data_smithi188_113957_object22608:head# 2024-04-23T16:33:41.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e95bea0:::benchmark_data_smithi188_113957_object104219:head# 2024-04-23T16:33:41.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e95c089:::benchmark_data_smithi188_113957_object103259:head# 2024-04-23T16:33:41.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e9641f2:::benchmark_data_smithi188_113957_object123675:head# 2024-04-23T16:33:41.777 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:33:41.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e964947:::benchmark_data_smithi188_113957_object43947:head# 2024-04-23T16:33:41.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e965e64:::benchmark_data_smithi188_113957_object108619:head# 2024-04-23T16:33:41.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e973a2c:::benchmark_data_smithi188_113957_object147712:head# 2024-04-23T16:33:41.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e975140:::benchmark_data_smithi188_113957_object141048:head# 2024-04-23T16:33:41.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e9802e4:::benchmark_data_smithi188_113957_object37893:head# 2024-04-23T16:33:41.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e988234:::benchmark_data_smithi188_113957_object22347:head# 2024-04-23T16:33:41.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e98cddd:::benchmark_data_smithi188_113957_object144427:head# 2024-04-23T16:33:41.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:41.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e99c184:::benchmark_data_smithi188_113957_object2375:head# 2024-04-23T16:33:41.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e99c507:::benchmark_data_smithi188_113957_object102973:head# 2024-04-23T16:33:41.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e9afc25:::benchmark_data_smithi188_113957_object122209:head# 2024-04-23T16:33:41.782 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:33:41.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e9b0abd:::benchmark_data_smithi188_113957_object151312:head# 2024-04-23T16:33:41.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e9d40e3:::benchmark_data_smithi188_113957_object31249:head# 2024-04-23T16:33:41.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e9d4579:::benchmark_data_smithi188_113957_object96779:head# 2024-04-23T16:33:41.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e9da92a:::benchmark_data_smithi188_113957_object95299:head# 2024-04-23T16:33:41.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e9e4f82:::benchmark_data_smithi188_113957_object148083:head# 2024-04-23T16:33:41.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e9ed953:::benchmark_data_smithi188_113957_object74420:head# 2024-04-23T16:33:41.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5e9f2293:::benchmark_data_smithi188_113957_object52874:head# 2024-04-23T16:33:41.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea03bab:::benchmark_data_smithi188_113957_object140362:head# 2024-04-23T16:33:41.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea0905c:::benchmark_data_smithi188_113957_object35367:head# 2024-04-23T16:33:41.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea10ee1:::benchmark_data_smithi188_113957_object99450:head# 2024-04-23T16:33:41.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea13cbd:::benchmark_data_smithi188_113957_object40846:head# 2024-04-23T16:33:41.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea160a8:::benchmark_data_smithi188_113957_object24390:head# 2024-04-23T16:33:41.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea16a3d:::benchmark_data_smithi188_113957_object120345:head# 2024-04-23T16:33:41.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea1c145:::benchmark_data_smithi188_113957_object70497:head# 2024-04-23T16:33:41.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea24a1c:::benchmark_data_smithi188_113957_object74276:head# 2024-04-23T16:33:41.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea25a84:::benchmark_data_smithi188_113957_object121815:head# 2024-04-23T16:33:41.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea2ccba:::benchmark_data_smithi188_113957_object141692:head# 2024-04-23T16:33:41.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea3bb3d:::benchmark_data_smithi188_113957_object145882:head# 2024-04-23T16:33:41.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea3e9fa:::benchmark_data_smithi188_113957_object47616:head# 2024-04-23T16:33:41.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea4e2c0:::benchmark_data_smithi188_113957_object149958:head# 2024-04-23T16:33:41.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea556a7:::benchmark_data_smithi188_113957_object30990:head# 2024-04-23T16:33:41.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea593d7:::benchmark_data_smithi188_113957_object13943:head# 2024-04-23T16:33:41.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea59ab7:::benchmark_data_smithi188_113957_object144881:head# 2024-04-23T16:33:41.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea5cd57:::benchmark_data_smithi188_113957_object82327:head# 2024-04-23T16:33:41.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea60b17:::benchmark_data_smithi188_113957_object97740:head# 2024-04-23T16:33:41.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea60f5b:::benchmark_data_smithi188_113957_object32629:head# 2024-04-23T16:33:41.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea638d7:::benchmark_data_smithi188_113957_object10995:head# 2024-04-23T16:33:41.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea642d3:::benchmark_data_smithi188_113957_object72861:head# 2024-04-23T16:33:41.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea6813f:::benchmark_data_smithi188_113957_object64911:head# 2024-04-23T16:33:41.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea70adb:::benchmark_data_smithi188_113957_object51729:head# 2024-04-23T16:33:41.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea75d24:::benchmark_data_smithi188_113957_object81690:head# 2024-04-23T16:33:41.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea7a62c:::benchmark_data_smithi188_113957_object26233:head# 2024-04-23T16:33:41.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea7fe90:::benchmark_data_smithi188_113957_object119173:head# 2024-04-23T16:33:41.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea812f0:::benchmark_data_smithi188_113957_object62865:head# 2024-04-23T16:33:41.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea8a0d6:::benchmark_data_smithi188_113957_object43696:head# 2024-04-23T16:33:41.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea8bad2:::benchmark_data_smithi188_113957_object122578:head# 2024-04-23T16:33:41.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea9221d:::benchmark_data_smithi188_113957_object107067:head# 2024-04-23T16:33:41.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea9720f:::benchmark_data_smithi188_113957_object91476:head# 2024-04-23T16:33:41.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea99923:::benchmark_data_smithi188_113957_object44729:head# 2024-04-23T16:33:41.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea9b895:::benchmark_data_smithi188_113957_object79267:head# 2024-04-23T16:33:41.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ea9f611:::benchmark_data_smithi188_113957_object135421:head# 2024-04-23T16:33:41.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eaa3451:::benchmark_data_smithi188_113957_object140522:head# 2024-04-23T16:33:41.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eaaa77a:::benchmark_data_smithi188_113957_object82657:head# 2024-04-23T16:33:41.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eaaf66a:::benchmark_data_smithi188_113957_object40907:head# 2024-04-23T16:33:41.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eac2920:::benchmark_data_smithi188_113957_object72513:head# 2024-04-23T16:33:41.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eac7292:::benchmark_data_smithi188_113957_object48096:head# 2024-04-23T16:33:41.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ead7d14:::benchmark_data_smithi188_113957_object28806:head# 2024-04-23T16:33:41.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eae40d7:::benchmark_data_smithi188_113957_object67626:head# 2024-04-23T16:33:41.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eae9ae5:::benchmark_data_smithi188_113957_object23876:head# 2024-04-23T16:33:41.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eaea34c:::benchmark_data_smithi188_113957_object91605:head# 2024-04-23T16:33:41.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eaf9b49:::benchmark_data_smithi188_113957_object71908:head# 2024-04-23T16:33:41.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eaff217:::benchmark_data_smithi188_113957_object35586:head# 2024-04-23T16:33:41.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb0148f:::benchmark_data_smithi188_113957_object147617:head# 2024-04-23T16:33:41.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb0667e:::benchmark_data_smithi188_113957_object46928:head# 2024-04-23T16:33:41.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb0792b:::benchmark_data_smithi188_113957_object130864:head# 2024-04-23T16:33:41.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb1fe2c:::benchmark_data_smithi188_113957_object26019:head# 2024-04-23T16:33:41.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb261a9:::benchmark_data_smithi188_113957_object122995:head# 2024-04-23T16:33:41.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb2b227:::benchmark_data_smithi188_113957_object150778:head# 2024-04-23T16:33:41.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb32570:::benchmark_data_smithi188_113957_object4702:head# 2024-04-23T16:33:41.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb364fd:::benchmark_data_smithi188_113957_object86664:head# 2024-04-23T16:33:41.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb39b99:::benchmark_data_smithi188_113957_object74260:head# 2024-04-23T16:33:41.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb39fba:::benchmark_data_smithi188_113957_object96634:head# 2024-04-23T16:33:41.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb3b81e:::benchmark_data_smithi188_113957_object131811:head# 2024-04-23T16:33:41.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb40757:::benchmark_data_smithi188_113957_object70656:head# 2024-04-23T16:33:41.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb5027a:::benchmark_data_smithi188_113957_object88212:head# 2024-04-23T16:33:41.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb63667:::benchmark_data_smithi188_113957_object108472:head# 2024-04-23T16:33:41.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb6e372:::benchmark_data_smithi188_113957_object60338:head# 2024-04-23T16:33:41.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb7880f:::benchmark_data_smithi188_113957_object61036:head# 2024-04-23T16:33:41.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb7c1ad:::benchmark_data_smithi188_113957_object62990:head# 2024-04-23T16:33:41.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb7c603:::benchmark_data_smithi188_113957_object126610:head# 2024-04-23T16:33:41.814 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb7f8d2:::benchmark_data_smithi188_113957_object122254:head# 2024-04-23T16:33:41.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb7fa3a:::benchmark_data_smithi188_113957_object128861:head# 2024-04-23T16:33:41.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb81336:::benchmark_data_smithi188_113957_object144298:head# 2024-04-23T16:33:41.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb823a8:::benchmark_data_smithi188_113957_object44674:head# 2024-04-23T16:33:41.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eb91710:::benchmark_data_smithi188_113957_object143424:head# 2024-04-23T16:33:41.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eba80aa:::benchmark_data_smithi188_113957_object32943:head# 2024-04-23T16:33:41.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eba87c9:::benchmark_data_smithi188_113957_object134153:head# 2024-04-23T16:33:41.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ebb0c9e:::benchmark_data_smithi188_113957_object33055:head# 2024-04-23T16:33:41.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ebb4256:::benchmark_data_smithi188_113957_object47309:head# 2024-04-23T16:33:41.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ebb7c27:::benchmark_data_smithi188_113957_object5005:head# 2024-04-23T16:33:41.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ebc6267:::benchmark_data_smithi188_113957_object31873:head# 2024-04-23T16:33:41.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ebc8ef2:::benchmark_data_smithi188_113957_object11949:head# 2024-04-23T16:33:41.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ebcd3f3:::benchmark_data_smithi188_113957_object82558:head# 2024-04-23T16:33:41.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ebd7f7e:::benchmark_data_smithi188_113957_object17411:head# 2024-04-23T16:33:41.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ebd9545:::benchmark_data_smithi188_113957_object46111:head# 2024-04-23T16:33:41.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ebde932:::benchmark_data_smithi188_113957_object138794:head# 2024-04-23T16:33:41.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ebe5bfb:::benchmark_data_smithi188_113957_object149125:head# 2024-04-23T16:33:41.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ebe5c41:::benchmark_data_smithi188_113957_object110795:head# 2024-04-23T16:33:41.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ebebcdb:::benchmark_data_smithi188_113957_object124044:head# 2024-04-23T16:33:41.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ebee58f:::benchmark_data_smithi188_113957_object35289:head# 2024-04-23T16:33:41.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ebef38a:::benchmark_data_smithi188_113957_object141154:head# 2024-04-23T16:33:41.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ebf3215:::benchmark_data_smithi188_113957_object115088:head# 2024-04-23T16:33:41.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec063e3:::benchmark_data_smithi188_113957_object38508:head# 2024-04-23T16:33:41.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec09104:::benchmark_data_smithi188_113957_object134615:head# 2024-04-23T16:33:41.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec0b20b:::benchmark_data_smithi188_113957_object79837:head# 2024-04-23T16:33:41.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec16a8e:::benchmark_data_smithi188_113957_object27537:head# 2024-04-23T16:33:41.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec19855:::benchmark_data_smithi188_113957_object73977:head# 2024-04-23T16:33:41.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec1a66f:::benchmark_data_smithi188_113957_object79172:head# 2024-04-23T16:33:41.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec2651f:::benchmark_data_smithi188_113957_object115067:head# 2024-04-23T16:33:41.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec27b97:::benchmark_data_smithi188_113957_object51959:head# 2024-04-23T16:33:41.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec33c1d:::benchmark_data_smithi188_113957_object24568:head# 2024-04-23T16:33:41.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec37292:::benchmark_data_smithi188_113957_object115479:head# 2024-04-23T16:33:41.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec37689:::benchmark_data_smithi188_113957_object71294:head# 2024-04-23T16:33:41.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec3920f:::benchmark_data_smithi188_113957_object44428:head# 2024-04-23T16:33:41.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec3c3c7:::benchmark_data_smithi188_113957_object96487:head# 2024-04-23T16:33:41.832 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec4067f:::benchmark_data_smithi188_113957_object151384:head# 2024-04-23T16:33:41.832 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec43044:::benchmark_data_smithi188_113957_object9621:head# 2024-04-23T16:33:41.832 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec456de:::benchmark_data_smithi188_113957_object110791:head# 2024-04-23T16:33:41.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec4b6ae:::benchmark_data_smithi188_113957_object37602:head# 2024-04-23T16:33:41.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec51010:::benchmark_data_smithi188_113957_object35388:head# 2024-04-23T16:33:41.834 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec522bd:::benchmark_data_smithi188_113957_object5559:head# 2024-04-23T16:33:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec5ea41:::benchmark_data_smithi188_113957_object73813:head# 2024-04-23T16:33:41.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec610c7:::benchmark_data_smithi188_113957_object88278:head# 2024-04-23T16:33:41.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec62757:::benchmark_data_smithi188_113957_object19782:head# 2024-04-23T16:33:41.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec647d6:::benchmark_data_smithi188_113957_object64491:head# 2024-04-23T16:33:41.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec70557:::benchmark_data_smithi188_113957_object29126:head# 2024-04-23T16:33:41.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec75650:::benchmark_data_smithi188_113957_object58464:head# 2024-04-23T16:33:41.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec7c79b:::benchmark_data_smithi188_113957_object48159:head# 2024-04-23T16:33:41.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec87bd0:::benchmark_data_smithi188_113957_object141264:head# 2024-04-23T16:33:41.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec89e8d:::benchmark_data_smithi188_113957_object132692:head# 2024-04-23T16:33:41.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec9179e:::benchmark_data_smithi188_113957_object15494:head# 2024-04-23T16:33:41.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ec9e3c5:::benchmark_data_smithi188_113957_object21107:head# 2024-04-23T16:33:41.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ecbebc6:::benchmark_data_smithi188_113957_object96096:head# 2024-04-23T16:33:41.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ecbfcf1:::benchmark_data_smithi188_113957_object73690:head# 2024-04-23T16:33:41.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ecc00cb:::benchmark_data_smithi188_113957_object140:head# 2024-04-23T16:33:41.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eccc1ff:::benchmark_data_smithi188_113957_object110022:head# 2024-04-23T16:33:41.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eccf591:::benchmark_data_smithi188_113957_object37103:head# 2024-04-23T16:33:41.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ecd7c74:::benchmark_data_smithi188_113957_object108162:head# 2024-04-23T16:33:41.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ecd9870:::benchmark_data_smithi188_113957_object60162:head# 2024-04-23T16:33:41.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ecde0be:::benchmark_data_smithi188_113957_object29801:head# 2024-04-23T16:33:41.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ece9e45:::benchmark_data_smithi188_113957_object137350:head# 2024-04-23T16:33:41.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ecf0047:::benchmark_data_smithi188_113957_object10453:head# 2024-04-23T16:33:41.845 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ecf1373:::benchmark_data_smithi188_113957_object41829:head# 2024-04-23T16:33:41.845 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ecf5270:::benchmark_data_smithi188_113957_object67248:head# 2024-04-23T16:33:41.845 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ecf5415:::benchmark_data_smithi188_113957_object88079:head# 2024-04-23T16:33:41.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ecf7a36:::benchmark_data_smithi188_113957_object19261:head# 2024-04-23T16:33:41.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ecfb0b4:::benchmark_data_smithi188_113957_object146254:head# 2024-04-23T16:33:41.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed06015:::benchmark_data_smithi188_113957_object40767:head# 2024-04-23T16:33:41.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed08233:::benchmark_data_smithi188_113957_object54497:head# 2024-04-23T16:33:41.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed08550:::benchmark_data_smithi188_113957_object43425:head# 2024-04-23T16:33:41.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed1878c:::benchmark_data_smithi188_113957_object147639:head# 2024-04-23T16:33:41.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed19dd0:::benchmark_data_smithi188_113957_object53174:head# 2024-04-23T16:33:41.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed1ec73:::benchmark_data_smithi188_113957_object6664:head# 2024-04-23T16:33:41.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed1fb82:::benchmark_data_smithi188_113957_object102052:head# 2024-04-23T16:33:41.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed20713:::benchmark_data_smithi188_113957_object66466:head# 2024-04-23T16:33:41.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed2174f:::benchmark_data_smithi188_113957_object70903:head# 2024-04-23T16:33:41.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed2b347:::benchmark_data_smithi188_113957_object68428:head# 2024-04-23T16:33:41.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed30139:::benchmark_data_smithi188_113957_object34537:head# 2024-04-23T16:33:41.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed339a3:::benchmark_data_smithi188_113957_object131391:head# 2024-04-23T16:33:41.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed36453:::benchmark_data_smithi188_113957_object29040:head# 2024-04-23T16:33:41.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed3875e:::benchmark_data_smithi188_113957_object101187:head# 2024-04-23T16:33:41.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed3ad5f:::benchmark_data_smithi188_113957_object57217:head# 2024-04-23T16:33:41.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed3e5b1:::benchmark_data_smithi188_113957_object43847:head# 2024-04-23T16:33:41.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed47931:::benchmark_data_smithi188_113957_object56120:head# 2024-04-23T16:33:41.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed52273:::benchmark_data_smithi188_113957_object18404:head# 2024-04-23T16:33:41.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed52e31:::benchmark_data_smithi188_113957_object2032:head# 2024-04-23T16:33:41.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed59a36:::benchmark_data_smithi188_113957_object104278:head# 2024-04-23T16:33:41.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed5d0e7:::benchmark_data_smithi188_113957_object69522:head# 2024-04-23T16:33:41.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed6024d:::benchmark_data_smithi188_113957_object117429:head# 2024-04-23T16:33:41.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed60eda:::benchmark_data_smithi188_113957_object99486:head# 2024-04-23T16:33:41.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed6532c:::benchmark_data_smithi188_113957_object28435:head# 2024-04-23T16:33:41.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed7d276:::benchmark_data_smithi188_113957_object102804:head# 2024-04-23T16:33:41.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed80268:::benchmark_data_smithi188_113957_object85529:head# 2024-04-23T16:33:41.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed827d0:::benchmark_data_smithi188_113957_object97483:head# 2024-04-23T16:33:41.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed84bae:::benchmark_data_smithi188_113957_object43317:head# 2024-04-23T16:33:41.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed8d9de:::benchmark_data_smithi188_113957_object144956:head# 2024-04-23T16:33:41.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ed9dcc1:::benchmark_data_smithi188_113957_object6308:head# 2024-04-23T16:33:41.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eda997f:::benchmark_data_smithi188_113957_object150380:head# 2024-04-23T16:33:41.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5edafda0:::benchmark_data_smithi188_113957_object116414:head# 2024-04-23T16:33:41.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5edb288d:::benchmark_data_smithi188_113957_object135790:head# 2024-04-23T16:33:41.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5edb6e55:::benchmark_data_smithi188_113957_object56051:head# 2024-04-23T16:33:41.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5edc1e87:::benchmark_data_smithi188_113957_object119765:head# 2024-04-23T16:33:41.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5edc3644:::benchmark_data_smithi188_113957_object55030:head# 2024-04-23T16:33:41.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5edc45e5:::benchmark_data_smithi188_113957_object72022:head# 2024-04-23T16:33:41.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5edc51ca:::benchmark_data_smithi188_113957_object45753:head# 2024-04-23T16:33:41.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5edc76a7:::benchmark_data_smithi188_113957_object8758:head# 2024-04-23T16:33:41.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5edcce30:::benchmark_data_smithi188_113957_object75869:head# 2024-04-23T16:33:41.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5edd780b:::benchmark_data_smithi188_113957_object59810:head# 2024-04-23T16:33:41.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5edd995d:::benchmark_data_smithi188_113957_object45388:head# 2024-04-23T16:33:41.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ede1889:::benchmark_data_smithi188_113957_object38891:head# 2024-04-23T16:33:41.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ede6f4f:::benchmark_data_smithi188_113957_object71100:head# 2024-04-23T16:33:41.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ede992d:::benchmark_data_smithi188_113957_object68300:head# 2024-04-23T16:33:41.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5edf80fa:::benchmark_data_smithi188_113957_object64422:head# 2024-04-23T16:33:41.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5edffbb6:::benchmark_data_smithi188_113957_object106806:head# 2024-04-23T16:33:41.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ee20fa1:::benchmark_data_smithi188_113957_object66510:head# 2024-04-23T16:33:41.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ee27405:::benchmark_data_smithi188_113957_object23229:head# 2024-04-23T16:33:41.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ee29036:::benchmark_data_smithi188_113957_object113804:head# 2024-04-23T16:33:41.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ee3a4b5:::benchmark_data_smithi188_113957_object12070:head# 2024-04-23T16:33:41.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ee3cc8e:::benchmark_data_smithi188_113957_object71558:head# 2024-04-23T16:33:41.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ee3f83d:::benchmark_data_smithi188_113957_object87842:head# 2024-04-23T16:33:41.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ee400ea:::benchmark_data_smithi188_113957_object78702:head# 2024-04-23T16:33:41.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ee551e9:::benchmark_data_smithi188_113957_object39482:head# 2024-04-23T16:33:41.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ee5a1c4:::benchmark_data_smithi188_113957_object133356:head# 2024-04-23T16:33:41.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ee5d117:::benchmark_data_smithi188_113957_object10136:head# 2024-04-23T16:33:41.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ee63c61:::benchmark_data_smithi188_113957_object46287:head# 2024-04-23T16:33:41.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ee67064:::benchmark_data_smithi188_113957_object59716:head# 2024-04-23T16:33:41.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ee79344:::benchmark_data_smithi188_113957_object139073:head# 2024-04-23T16:33:41.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ee8330e:::benchmark_data_smithi188_113957_object147664:head# 2024-04-23T16:33:41.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ee89875:::benchmark_data_smithi188_113957_object97140:head# 2024-04-23T16:33:41.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ee89b17:::benchmark_data_smithi188_113957_object123882:head# 2024-04-23T16:33:41.875 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eeadbf4:::benchmark_data_smithi188_113957_object151554:head# 2024-04-23T16:33:41.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eeb56cf:::benchmark_data_smithi188_113957_object140582:head# 2024-04-23T16:33:41.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eec0e32:::benchmark_data_smithi188_113957_object34398:head# 2024-04-23T16:33:41.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eecc9e9:::benchmark_data_smithi188_113957_object109719:head# 2024-04-23T16:33:41.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eed6903:::benchmark_data_smithi188_113957_object58664:head# 2024-04-23T16:33:41.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eedd94a:::benchmark_data_smithi188_113957_object85102:head# 2024-04-23T16:33:41.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eee2391:::benchmark_data_smithi188_113957_object78262:head# 2024-04-23T16:33:41.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eef26bf:::benchmark_data_smithi188_113957_object139653:head# 2024-04-23T16:33:41.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eefc76d:::benchmark_data_smithi188_113957_object28464:head# 2024-04-23T16:33:41.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef05846:::benchmark_data_smithi188_113957_object38381:head# 2024-04-23T16:33:41.883 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:33:41.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef0aa6e:::benchmark_data_smithi188_113957_object43750:head# 2024-04-23T16:33:41.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef141de:::benchmark_data_smithi188_113957_object84457:head# 2024-04-23T16:33:41.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef166bc:::benchmark_data_smithi188_113957_object91992:head# 2024-04-23T16:33:41.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef19af1:::benchmark_data_smithi188_113957_object105005:head# 2024-04-23T16:33:41.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef19cbb:::benchmark_data_smithi188_113957_object42177:head# 2024-04-23T16:33:41.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef1aa7a:::benchmark_data_smithi188_113957_object55773:head# 2024-04-23T16:33:41.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef22541:::benchmark_data_smithi188_113957_object41185:head# 2024-04-23T16:33:41.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef22871:::benchmark_data_smithi188_113957_object26431:head# 2024-04-23T16:33:41.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef2e50e:::benchmark_data_smithi188_113957_object79306:head# 2024-04-23T16:33:41.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef31612:::benchmark_data_smithi188_113957_object53314:head# 2024-04-23T16:33:41.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef3f33f:::benchmark_data_smithi188_113957_object99656:head# 2024-04-23T16:33:41.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef40d78:::benchmark_data_smithi188_113957_object129:head# 2024-04-23T16:33:41.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef4cffc:::benchmark_data_smithi188_113957_object89736:head# 2024-04-23T16:33:41.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef57242:::benchmark_data_smithi188_113957_object103483:head# 2024-04-23T16:33:41.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef598be:::benchmark_data_smithi188_113957_object118801:head# 2024-04-23T16:33:41.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef5deee:::benchmark_data_smithi188_113957_object92891:head# 2024-04-23T16:33:41.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef60a95:::benchmark_data_smithi188_113957_object94042:head# 2024-04-23T16:33:41.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef61893:::benchmark_data_smithi188_113957_object338:head# 2024-04-23T16:33:41.891 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef6434f:::benchmark_data_smithi188_113957_object84395:head# 2024-04-23T16:33:41.893 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef6802e:::benchmark_data_smithi188_113957_object64379:head# 2024-04-23T16:33:41.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef69c95:::benchmark_data_smithi188_113957_object32982:head# 2024-04-23T16:33:41.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef6d493:::benchmark_data_smithi188_113957_object59039:head# 2024-04-23T16:33:41.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef70211:::benchmark_data_smithi188_113957_object118851:head# 2024-04-23T16:33:41.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef72325:::benchmark_data_smithi188_113957_object19614:head# 2024-04-23T16:33:41.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef75301:::benchmark_data_smithi188_113957_object104444:head# 2024-04-23T16:33:41.899 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef81201:::benchmark_data_smithi188_113957_object60972:head# 2024-04-23T16:33:41.899 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef88a9e:::benchmark_data_smithi188_113957_object7980:head# 2024-04-23T16:33:41.900 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef9286d:::benchmark_data_smithi188_113957_object121446:head# 2024-04-23T16:33:41.900 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ef9b081:::benchmark_data_smithi188_113957_object91445:head# 2024-04-23T16:33:41.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efa9f49:::benchmark_data_smithi188_113957_object87846:head# 2024-04-23T16:33:41.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efaa3ba:::benchmark_data_smithi188_113957_object74660:head# 2024-04-23T16:33:41.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efae8ae:::benchmark_data_smithi188_113957_object96457:head# 2024-04-23T16:33:41.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efb3c50:::benchmark_data_smithi188_113957_object141263:head# 2024-04-23T16:33:41.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efb41da:::benchmark_data_smithi188_113957_object41593:head# 2024-04-23T16:33:41.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efb6269:::benchmark_data_smithi188_113957_object115628:head# 2024-04-23T16:33:41.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efb8495:::benchmark_data_smithi188_113957_object56300:head# 2024-04-23T16:33:41.904 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efc255b:::benchmark_data_smithi188_113957_object86456:head# 2024-04-23T16:33:41.904 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efc2c52:::benchmark_data_smithi188_113957_object79131:head# 2024-04-23T16:33:41.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efc70ae:::benchmark_data_smithi188_113957_object75852:head# 2024-04-23T16:33:41.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efc932a:::benchmark_data_smithi188_113957_object98730:head# 2024-04-23T16:33:41.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efcc391:::benchmark_data_smithi188_113957_object53430:head# 2024-04-23T16:33:41.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efd39e0:::benchmark_data_smithi188_113957_object2103:head# 2024-04-23T16:33:41.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efd9cd5:::benchmark_data_smithi188_113957_object84094:head# 2024-04-23T16:33:41.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efe027e:::benchmark_data_smithi188_113957_object41817:head# 2024-04-23T16:33:41.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efe2cae:::benchmark_data_smithi188_113957_object98490:head# 2024-04-23T16:33:41.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efe4a96:::benchmark_data_smithi188_113957_object50482:head# 2024-04-23T16:33:41.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efeac35:::benchmark_data_smithi188_113957_object73366:head# 2024-04-23T16:33:41.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efec54e:::benchmark_data_smithi188_113957_object147829:head# 2024-04-23T16:33:41.909 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5efef530:::benchmark_data_smithi188_113957_object133776:head# 2024-04-23T16:33:41.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5eff237e:::benchmark_data_smithi188_113957_object2942:head# 2024-04-23T16:33:41.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5effbbf0:::benchmark_data_smithi188_113957_object136477:head# 2024-04-23T16:33:41.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f005ba9:::benchmark_data_smithi188_113957_object11795:head# 2024-04-23T16:33:41.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f00ee5e:::benchmark_data_smithi188_113957_object94541:head# 2024-04-23T16:33:41.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f01acf2:::benchmark_data_smithi188_113957_object115674:head# 2024-04-23T16:33:41.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f024fcd:::benchmark_data_smithi188_113957_object152718:head# 2024-04-23T16:33:41.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f02c959:::benchmark_data_smithi188_113957_object33607:head# 2024-04-23T16:33:41.913 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:41.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f02cd27:::benchmark_data_smithi188_113957_object46562:head# 2024-04-23T16:33:41.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f03519e:::benchmark_data_smithi188_113957_object9461:head# 2024-04-23T16:33:41.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f03b2fb:::benchmark_data_smithi188_113957_object142066:head# 2024-04-23T16:33:41.914 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f03e3d4:::benchmark_data_smithi188_113957_object79209:head# 2024-04-23T16:33:41.914 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f041115:::benchmark_data_smithi188_113957_object45708:head# 2024-04-23T16:33:41.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f04228c:::benchmark_data_smithi188_113957_object67159:head# 2024-04-23T16:33:41.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f04d468:::benchmark_data_smithi188_113957_object52719:head# 2024-04-23T16:33:41.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0619c3:::benchmark_data_smithi188_113957_object87347:head# 2024-04-23T16:33:41.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f07095d:::benchmark_data_smithi188_113957_object45919:head# 2024-04-23T16:33:41.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f07342c:::benchmark_data_smithi188_113957_object93881:head# 2024-04-23T16:33:41.917 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f07beb7:::benchmark_data_smithi188_113957_object126091:head# 2024-04-23T16:33:41.917 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0828aa:::benchmark_data_smithi188_113957_object45447:head# 2024-04-23T16:33:41.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f09c1bd:::benchmark_data_smithi188_113957_object143393:head# 2024-04-23T16:33:41.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0a19e4:::benchmark_data_smithi188_113957_object125869:head# 2024-04-23T16:33:41.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0a49ca:::benchmark_data_smithi188_113957_object10279:head# 2024-04-23T16:33:41.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0a8700:::benchmark_data_smithi188_113957_object79955:head# 2024-04-23T16:33:41.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0b0ec6:::benchmark_data_smithi188_113957_object23519:head# 2024-04-23T16:33:41.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0b54ba:::benchmark_data_smithi188_113957_object12839:head# 2024-04-23T16:33:41.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0b7d3f:::benchmark_data_smithi188_113957_object52175:head# 2024-04-23T16:33:41.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0b977d:::benchmark_data_smithi188_113957_object141994:head# 2024-04-23T16:33:41.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0bae4c:::benchmark_data_smithi188_113957_object31053:head# 2024-04-23T16:33:41.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0bc422:::benchmark_data_smithi188_113957_object21730:head# 2024-04-23T16:33:41.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0c29d9:::benchmark_data_smithi188_113957_object8829:head# 2024-04-23T16:33:41.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0c9fe7:::benchmark_data_smithi188_113957_object138962:head# 2024-04-23T16:33:41.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0d76d7:::benchmark_data_smithi188_113957_object39081:head# 2024-04-23T16:33:41.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0dc3e2:::benchmark_data_smithi188_113957_object43287:head# 2024-04-23T16:33:41.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0ddca4:::benchmark_data_smithi188_113957_object19288:head# 2024-04-23T16:33:41.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0df4cc:::benchmark_data_smithi188_113957_object59285:head# 2024-04-23T16:33:41.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0e553c:::benchmark_data_smithi188_113957_object79134:head# 2024-04-23T16:33:41.925 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0ec0bc:::benchmark_data_smithi188_113957_object15783:head# 2024-04-23T16:33:41.925 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0f1eb6:::benchmark_data_smithi188_113957_object112850:head# 2024-04-23T16:33:41.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0fb7c3:::benchmark_data_smithi188_113957_object73628:head# 2024-04-23T16:33:41.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f0fe8ed:::benchmark_data_smithi188_113957_object39435:head# 2024-04-23T16:33:41.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f101108:::benchmark_data_smithi188_113957_object95234:head# 2024-04-23T16:33:41.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f1012dd:::benchmark_data_smithi188_113957_object137371:head# 2024-04-23T16:33:41.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f10e205:::benchmark_data_smithi188_113957_object113355:head# 2024-04-23T16:33:41.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f116fe0:::benchmark_data_smithi188_113957_object20418:head# 2024-04-23T16:33:41.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f11c070:::benchmark_data_smithi188_113957_object10547:head# 2024-04-23T16:33:41.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f11fd1d:::benchmark_data_smithi188_113957_object132052:head# 2024-04-23T16:33:41.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f120a5b:::benchmark_data_smithi188_113957_object147094:head# 2024-04-23T16:33:41.930 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f121c92:::benchmark_data_smithi188_113957_object117054:head# 2024-04-23T16:33:41.930 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f12eb51:::benchmark_data_smithi188_113957_object55760:head# 2024-04-23T16:33:41.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f13e733:::benchmark_data_smithi188_113957_object105466:head# 2024-04-23T16:33:41.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f147221:::benchmark_data_smithi188_113957_object127320:head# 2024-04-23T16:33:41.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f14c7f0:::benchmark_data_smithi188_113957_object134850:head# 2024-04-23T16:33:41.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f1594d4:::benchmark_data_smithi188_113957_object88181:head# 2024-04-23T16:33:41.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f15cf92:::benchmark_data_smithi188_113957_object59568:head# 2024-04-23T16:33:41.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f15f590:::benchmark_data_smithi188_113957_object23978:head# 2024-04-23T16:33:41.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f161444:::benchmark_data_smithi188_113957_object95067:head# 2024-04-23T16:33:41.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f161903:::benchmark_data_smithi188_113957_object99013:head# 2024-04-23T16:33:41.934 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:33:41.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f1621f6:::benchmark_data_smithi188_113957_object43731:head# 2024-04-23T16:33:41.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f167f75:::benchmark_data_smithi188_113957_object46101:head# 2024-04-23T16:33:41.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f16c692:::benchmark_data_smithi188_113957_object77998:head# 2024-04-23T16:33:41.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f16f5b6:::benchmark_data_smithi188_113957_object26834:head# 2024-04-23T16:33:41.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f16f742:::benchmark_data_smithi188_113957_object67427:head# 2024-04-23T16:33:41.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f175ae8:::benchmark_data_smithi188_113957_object104497:head# 2024-04-23T16:33:41.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f17cd4b:::benchmark_data_smithi188_113957_object99609:head# 2024-04-23T16:33:41.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f184154:::benchmark_data_smithi188_113957_object58012:head# 2024-04-23T16:33:41.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f188746:::benchmark_data_smithi188_113957_object70681:head# 2024-04-23T16:33:41.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f1b29d1:::benchmark_data_smithi188_113957_object54339:head# 2024-04-23T16:33:41.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f1d2a7b:::benchmark_data_smithi188_113957_object90217:head# 2024-04-23T16:33:41.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f1d4d9d:::benchmark_data_smithi188_113957_object111150:head# 2024-04-23T16:33:41.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:41.940 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:33:41.940 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f1d8226:::benchmark_data_smithi188_113957_object23101:head# 2024-04-23T16:33:41.940 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f1da26f:::benchmark_data_smithi188_113957_object103079:head# 2024-04-23T16:33:41.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f1de382:::benchmark_data_smithi188_113957_object123561:head# 2024-04-23T16:33:41.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f1e754b:::benchmark_data_smithi188_113957_object39801:head# 2024-04-23T16:33:41.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f1ebbc0:::benchmark_data_smithi188_113957_object95821:head# 2024-04-23T16:33:41.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f1ee031:::benchmark_data_smithi188_113957_object34584:head# 2024-04-23T16:33:41.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f1ef57c:::benchmark_data_smithi188_113957_object112541:head# 2024-04-23T16:33:41.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f1f2937:::benchmark_data_smithi188_113957_object101053:head# 2024-04-23T16:33:41.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f2060b2:::benchmark_data_smithi188_113957_object83522:head# 2024-04-23T16:33:41.943 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f21cb37:::benchmark_data_smithi188_113957_object89966:head# 2024-04-23T16:33:41.943 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f21d011:::benchmark_data_smithi188_113957_object13742:head# 2024-04-23T16:33:41.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f227461:::benchmark_data_smithi188_113957_object87685:head# 2024-04-23T16:33:41.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f227be4:::benchmark_data_smithi188_113957_object111976:head# 2024-04-23T16:33:41.945 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f22dc10:::benchmark_data_smithi188_113957_object5919:head# 2024-04-23T16:33:41.945 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f22ff8f:::benchmark_data_smithi188_113957_object152671:head# 2024-04-23T16:33:41.945 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f233189:::benchmark_data_smithi188_113957_object108756:head# 2024-04-23T16:33:41.946 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f241152:::benchmark_data_smithi188_113957_object67623:head# 2024-04-23T16:33:41.946 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f24d7d3:::benchmark_data_smithi188_113957_object134095:head# 2024-04-23T16:33:41.946 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f252922:::benchmark_data_smithi188_113957_object63855:head# 2024-04-23T16:33:41.947 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f254e0e:::benchmark_data_smithi188_113957_object129342:head# 2024-04-23T16:33:41.947 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f255cda:::benchmark_data_smithi188_113957_object153144:head# 2024-04-23T16:33:41.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f269478:::benchmark_data_smithi188_113957_object34562:head# 2024-04-23T16:33:41.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f27fc2c:::benchmark_data_smithi188_113957_object110581:head# 2024-04-23T16:33:41.949 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f2806f9:::benchmark_data_smithi188_113957_object41932:head# 2024-04-23T16:33:41.949 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f28edf7:::benchmark_data_smithi188_113957_object9959:head# 2024-04-23T16:33:41.949 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f2939ff:::benchmark_data_smithi188_113957_object145314:head# 2024-04-23T16:33:41.950 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f299997:::benchmark_data_smithi188_113957_object107021:head# 2024-04-23T16:33:41.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f29a6c1:::benchmark_data_smithi188_113957_object98847:head# 2024-04-23T16:33:41.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f2ab85c:::benchmark_data_smithi188_113957_object41000:head# 2024-04-23T16:33:41.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f2ac8da:::benchmark_data_smithi188_113957_object96530:head# 2024-04-23T16:33:41.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f2c156c:::benchmark_data_smithi188_113957_object85188:head# 2024-04-23T16:33:41.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f2c209e:::benchmark_data_smithi188_113957_object27266:head# 2024-04-23T16:33:41.953 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f2cadcb:::benchmark_data_smithi188_113957_object79818:head# 2024-04-23T16:33:41.953 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f2d7192:::benchmark_data_smithi188_113957_object92114:head# 2024-04-23T16:33:41.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f2d83d9:::benchmark_data_smithi188_113957_object58401:head# 2024-04-23T16:33:41.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f2d8968:::benchmark_data_smithi188_113957_object85154:head# 2024-04-23T16:33:41.955 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f2df164:::benchmark_data_smithi188_113957_object143856:head# 2024-04-23T16:33:41.955 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f2e1327:::benchmark_data_smithi188_113957_object24571:head# 2024-04-23T16:33:41.955 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f2f0d09:::benchmark_data_smithi188_113957_object112634:head# 2024-04-23T16:33:41.956 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f2f8b56:::benchmark_data_smithi188_113957_object121662:head# 2024-04-23T16:33:41.956 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f300f6a:::benchmark_data_smithi188_113957_object12254:head# 2024-04-23T16:33:41.956 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f3070f0:::benchmark_data_smithi188_113957_object115803:head# 2024-04-23T16:33:41.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f319456:::benchmark_data_smithi188_113957_object54840:head# 2024-04-23T16:33:41.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f32bad8:::benchmark_data_smithi188_113957_object26791:head# 2024-04-23T16:33:41.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f330d47:::benchmark_data_smithi188_113957_object50762:head# 2024-04-23T16:33:41.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f33a235:::benchmark_data_smithi188_113957_object83930:head# 2024-04-23T16:33:41.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f33f91d:::benchmark_data_smithi188_113957_object127551:head# 2024-04-23T16:33:41.959 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f34091e:::benchmark_data_smithi188_113957_object40910:head# 2024-04-23T16:33:41.960 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f34693a:::benchmark_data_smithi188_113957_object141878:head# 2024-04-23T16:33:41.960 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f357e03:::benchmark_data_smithi188_113957_object69551:head# 2024-04-23T16:33:41.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f359746:::benchmark_data_smithi188_113957_object53794:head# 2024-04-23T16:33:41.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f35c57c:::benchmark_data_smithi188_113957_object121269:head# 2024-04-23T16:33:41.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f369e2b:::benchmark_data_smithi188_113957_object63571:head# 2024-04-23T16:33:41.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f36e34d:::benchmark_data_smithi188_113957_object18474:head# 2024-04-23T16:33:41.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f37388d:::benchmark_data_smithi188_113957_object89881:head# 2024-04-23T16:33:41.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f37ca4a:::benchmark_data_smithi188_113957_object7019:head# 2024-04-23T16:33:41.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f3875eb:::benchmark_data_smithi188_113957_object42503:head# 2024-04-23T16:33:41.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f389f2a:::benchmark_data_smithi188_113957_object86041:head# 2024-04-23T16:33:41.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f38ccf2:::benchmark_data_smithi188_113957_object140492:head# 2024-04-23T16:33:41.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f38fd20:::benchmark_data_smithi188_113957_object69689:head# 2024-04-23T16:33:41.965 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f394507:::benchmark_data_smithi188_113957_object119684:head# 2024-04-23T16:33:41.965 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f3a4cfc:::benchmark_data_smithi188_113957_object117214:head# 2024-04-23T16:33:41.965 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f3a5407:::benchmark_data_smithi188_113957_object130378:head# 2024-04-23T16:33:41.966 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f3afb3e:::benchmark_data_smithi188_113957_object92895:head# 2024-04-23T16:33:41.966 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f3b8f00:::benchmark_data_smithi188_113957_object145793:head# 2024-04-23T16:33:41.967 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f3be6af:::benchmark_data_smithi188_113957_object125295:head# 2024-04-23T16:33:41.967 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f3cc4d9:::benchmark_data_smithi188_113957_object77292:head# 2024-04-23T16:33:41.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f3d295f:::benchmark_data_smithi188_113957_object106096:head# 2024-04-23T16:33:41.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f3e0552:::benchmark_data_smithi188_113957_object124765:head# 2024-04-23T16:33:41.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f3e1155:::benchmark_data_smithi188_113957_object97195:head# 2024-04-23T16:33:41.969 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f3e4524:::benchmark_data_smithi188_113957_object57943:head# 2024-04-23T16:33:41.969 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f3e927d:::benchmark_data_smithi188_113957_object59858:head# 2024-04-23T16:33:41.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f3ea252:::benchmark_data_smithi188_113957_object139719:head# 2024-04-23T16:33:41.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f3ea3c2:::benchmark_data_smithi188_113957_object98905:head# 2024-04-23T16:33:41.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f3ec7a7:::benchmark_data_smithi188_113957_object6010:head# 2024-04-23T16:33:41.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f3faabd:::benchmark_data_smithi188_113957_object145714:head# 2024-04-23T16:33:41.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f3fe230:::benchmark_data_smithi188_113957_object126514:head# 2024-04-23T16:33:41.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f40398e:::benchmark_data_smithi188_113957_object60602:head# 2024-04-23T16:33:41.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f40700c:::benchmark_data_smithi188_113957_object49060:head# 2024-04-23T16:33:41.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f409dff:::benchmark_data_smithi188_113957_object90608:head# 2024-04-23T16:33:41.973 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f40b972:::benchmark_data_smithi188_113957_object6182:head# 2024-04-23T16:33:41.973 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f40fb30:::benchmark_data_smithi188_113957_object134725:head# 2024-04-23T16:33:41.974 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f414c7a:::benchmark_data_smithi188_113957_object141675:head# 2024-04-23T16:33:41.974 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f41b0f0:::benchmark_data_smithi188_113957_object85834:head# 2024-04-23T16:33:41.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f42afba:::benchmark_data_smithi188_113957_object46167:head# 2024-04-23T16:33:41.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f440885:::benchmark_data_smithi188_113957_object152663:head# 2024-04-23T16:33:41.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f4472d8:::benchmark_data_smithi188_113957_object149995:head# 2024-04-23T16:33:41.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f452081:::benchmark_data_smithi188_113957_object115269:head# 2024-04-23T16:33:41.977 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f45c2fa:::benchmark_data_smithi188_113957_object74128:head# 2024-04-23T16:33:41.977 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f47167b:::benchmark_data_smithi188_113957_object62715:head# 2024-04-23T16:33:41.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f476984:::benchmark_data_smithi188_113957_object59337:head# 2024-04-23T16:33:41.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f479199:::benchmark_data_smithi188_113957_object121108:head# 2024-04-23T16:33:41.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f47c42c:::benchmark_data_smithi188_113957_object18925:head# 2024-04-23T16:33:41.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f47c66d:::benchmark_data_smithi188_113957_object6383:head# 2024-04-23T16:33:41.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f48ba1c:::benchmark_data_smithi188_113957_object113698:head# 2024-04-23T16:33:41.980 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f49dc7d:::benchmark_data_smithi188_113957_object71940:head# 2024-04-23T16:33:41.980 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f49f3b9:::benchmark_data_smithi188_113957_object14714:head# 2024-04-23T16:33:41.980 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f4a4b1b:::benchmark_data_smithi188_113957_object73037:head# 2024-04-23T16:33:41.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f4ade10:::benchmark_data_smithi188_113957_object27988:head# 2024-04-23T16:33:41.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f4b9a07:::benchmark_data_smithi188_113957_object33329:head# 2024-04-23T16:33:41.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f4c7930:::benchmark_data_smithi188_113957_object96419:head# 2024-04-23T16:33:41.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f4c8205:::benchmark_data_smithi188_113957_object121095:head# 2024-04-23T16:33:41.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f4d01b1:::benchmark_data_smithi188_113957_object47750:head# 2024-04-23T16:33:41.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f4d4ceb:::benchmark_data_smithi188_113957_object68553:head# 2024-04-23T16:33:41.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f4e77f5:::benchmark_data_smithi188_113957_object54500:head# 2024-04-23T16:33:41.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f4f0175:::benchmark_data_smithi188_113957_object104717:head# 2024-04-23T16:33:41.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f4f1e70:::benchmark_data_smithi188_113957_object91277:head# 2024-04-23T16:33:41.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f4f970f:::benchmark_data_smithi188_113957_object33796:head# 2024-04-23T16:33:41.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f5008ce:::benchmark_data_smithi188_113957_object3618:head# 2024-04-23T16:33:41.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f509fd6:::benchmark_data_smithi188_113957_object11642:head# 2024-04-23T16:33:41.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f5116ba:::benchmark_data_smithi188_113957_object37248:head# 2024-04-23T16:33:41.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f511daa:::benchmark_data_smithi188_113957_object138257:head# 2024-04-23T16:33:41.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f519cac:::benchmark_data_smithi188_113957_object5129:head# 2024-04-23T16:33:41.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f5201cd:::benchmark_data_smithi188_113957_object10691:head# 2024-04-23T16:33:41.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f521fe8:::benchmark_data_smithi188_113957_object18538:head# 2024-04-23T16:33:41.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f530557:::benchmark_data_smithi188_113957_object26634:head# 2024-04-23T16:33:41.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f532d04:::benchmark_data_smithi188_113957_object54362:head# 2024-04-23T16:33:41.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f532d35:::benchmark_data_smithi188_113957_object104706:head# 2024-04-23T16:33:41.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f53bfef:::benchmark_data_smithi188_113957_object37866:head# 2024-04-23T16:33:41.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f541b8d:::benchmark_data_smithi188_113957_object151562:head# 2024-04-23T16:33:41.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f54686b:::benchmark_data_smithi188_113957_object88845:head# 2024-04-23T16:33:41.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f54933a:::benchmark_data_smithi188_113957_object48750:head# 2024-04-23T16:33:41.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f54f369:::benchmark_data_smithi188_113957_object29152:head# 2024-04-23T16:33:41.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f54f6f0:::benchmark_data_smithi188_113957_object61147:head# 2024-04-23T16:33:41.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f559a28:::benchmark_data_smithi188_113957_object121399:head# 2024-04-23T16:33:41.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f562609:::benchmark_data_smithi188_113957_object115415:head# 2024-04-23T16:33:41.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f56306a:::benchmark_data_smithi188_113957_object24559:head# 2024-04-23T16:33:41.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f56386d:::benchmark_data_smithi188_113957_object80594:head# 2024-04-23T16:33:41.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f579101:::benchmark_data_smithi188_113957_object2308:head# 2024-04-23T16:33:41.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f57b0ac:::benchmark_data_smithi188_113957_object30895:head# 2024-04-23T16:33:41.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f57c4a0:::benchmark_data_smithi188_113957_object101178:head# 2024-04-23T16:33:41.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f5915ad:::benchmark_data_smithi188_113957_object120963:head# 2024-04-23T16:33:41.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f59351d:::benchmark_data_smithi188_113957_object59615:head# 2024-04-23T16:33:41.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f598032:::benchmark_data_smithi188_113957_object141787:head# 2024-04-23T16:33:41.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f59a749:::benchmark_data_smithi188_113957_object87639:head# 2024-04-23T16:33:41.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f5aa87e:::benchmark_data_smithi188_113957_object50970:head# 2024-04-23T16:33:41.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f5ba281:::benchmark_data_smithi188_113957_object140286:head# 2024-04-23T16:33:41.997 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f5bd8df:::benchmark_data_smithi188_113957_object137941:head# 2024-04-23T16:33:41.997 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f5d019d:::benchmark_data_smithi188_113957_object76033:head# 2024-04-23T16:33:41.998 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f5d386c:::benchmark_data_smithi188_113957_object29769:head# 2024-04-23T16:33:41.998 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f5dc8cd:::benchmark_data_smithi188_113957_object117846:head# 2024-04-23T16:33:41.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f5ea3f1:::benchmark_data_smithi188_113957_object129111:head# 2024-04-23T16:33:41.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f5ed6d3:::benchmark_data_smithi188_113957_object39596:head# 2024-04-23T16:33:41.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f5ee71c:::benchmark_data_smithi188_113957_object2571:head# 2024-04-23T16:33:42.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f5f6443:::benchmark_data_smithi188_113957_object45189:head# 2024-04-23T16:33:42.001 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f5f6ac0:::benchmark_data_smithi188_113957_object76841:head# 2024-04-23T16:33:42.001 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f5ff722:::benchmark_data_smithi188_113957_object136348:head# 2024-04-23T16:33:42.002 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f601e42:::benchmark_data_smithi188_113957_object101578:head# 2024-04-23T16:33:42.002 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f60958f:::benchmark_data_smithi188_113957_object50184:head# 2024-04-23T16:33:42.002 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f60ee8d:::benchmark_data_smithi188_113957_object123120:head# 2024-04-23T16:33:42.003 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f60fc4b:::benchmark_data_smithi188_113957_object50961:head# 2024-04-23T16:33:42.003 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f619729:::benchmark_data_smithi188_113957_object64841:head# 2024-04-23T16:33:42.003 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f61b098:::benchmark_data_smithi188_113957_object35420:head# 2024-04-23T16:33:42.004 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f63098f:::benchmark_data_smithi188_113957_object146232:head# 2024-04-23T16:33:42.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f63841d:::benchmark_data_smithi188_113957_object117266:head# 2024-04-23T16:33:42.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f63ea47:::benchmark_data_smithi188_113957_object9796:head# 2024-04-23T16:33:42.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f645434:::benchmark_data_smithi188_113957_object106876:head# 2024-04-23T16:33:42.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f64c010:::benchmark_data_smithi188_113957_object134510:head# 2024-04-23T16:33:42.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f64d29a:::benchmark_data_smithi188_113957_object127186:head# 2024-04-23T16:33:42.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f64d9a5:::benchmark_data_smithi188_113957_object140992:head# 2024-04-23T16:33:42.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f653a8c:::benchmark_data_smithi188_113957_object13214:head# 2024-04-23T16:33:42.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f66ea3a:::benchmark_data_smithi188_113957_object139408:head# 2024-04-23T16:33:42.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f672a1e:::benchmark_data_smithi188_113957_object70412:head# 2024-04-23T16:33:42.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f679972:::benchmark_data_smithi188_113957_object53806:head# 2024-04-23T16:33:42.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f685131:::benchmark_data_smithi188_113957_object68919:head# 2024-04-23T16:33:42.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f68d226:::benchmark_data_smithi188_113957_object68666:head# 2024-04-23T16:33:42.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f68d94d:::benchmark_data_smithi188_113957_object25365:head# 2024-04-23T16:33:42.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f68e560:::benchmark_data_smithi188_113957_object2506:head# 2024-04-23T16:33:42.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f694ca0:::benchmark_data_smithi188_113957_object105864:head# 2024-04-23T16:33:42.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f6964bc:::benchmark_data_smithi188_113957_object3104:head# 2024-04-23T16:33:42.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f69d2d6:::benchmark_data_smithi188_113957_object111769:head# 2024-04-23T16:33:42.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f6a697f:::benchmark_data_smithi188_113957_object114682:head# 2024-04-23T16:33:42.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f6a850e:::benchmark_data_smithi188_113957_object80344:head# 2024-04-23T16:33:42.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f6ac9c3:::benchmark_data_smithi188_113957_object45789:head# 2024-04-23T16:33:42.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f6b2c14:::benchmark_data_smithi188_113957_object47867:head# 2024-04-23T16:33:42.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f6b5cd3:::benchmark_data_smithi188_113957_object139801:head# 2024-04-23T16:33:42.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f6c6450:::benchmark_data_smithi188_113957_object1509:head# 2024-04-23T16:33:42.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f6c7601:::benchmark_data_smithi188_113957_object111260:head# 2024-04-23T16:33:42.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f6c7dbc:::benchmark_data_smithi188_113957_object17126:head# 2024-04-23T16:33:42.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f6dbed0:::benchmark_data_smithi188_113957_object116489:head# 2024-04-23T16:33:42.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f6ec638:::benchmark_data_smithi188_113957_object33003:head# 2024-04-23T16:33:42.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f6eedcb:::benchmark_data_smithi188_113957_object140818:head# 2024-04-23T16:33:42.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f6efa7f:::benchmark_data_smithi188_113957_object42829:head# 2024-04-23T16:33:42.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f6f32dc:::benchmark_data_smithi188_113957_object20124:head# 2024-04-23T16:33:42.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f6f7a26:::benchmark_data_smithi188_113957_object149591:head# 2024-04-23T16:33:42.018 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f6f818b:::benchmark_data_smithi188_113957_object38510:head# 2024-04-23T16:33:42.018 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f6f9376:::benchmark_data_smithi188_113957_object96288:head# 2024-04-23T16:33:42.018 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f702498:::benchmark_data_smithi188_113957_object148441:head# 2024-04-23T16:33:42.019 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f704e85:::benchmark_data_smithi188_113957_object16984:head# 2024-04-23T16:33:42.019 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:42 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3968621477' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:33:42.020 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f706125:::benchmark_data_smithi188_113957_object143046:head# 2024-04-23T16:33:42.020 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f719ea4:::benchmark_data_smithi188_113957_object12526:head# 2024-04-23T16:33:42.020 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f71a932:::benchmark_data_smithi188_113957_object110886:head# 2024-04-23T16:33:42.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f71e7b8:::benchmark_data_smithi188_113957_object55872:head# 2024-04-23T16:33:42.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f7237b7:::benchmark_data_smithi188_113957_object129002:head# 2024-04-23T16:33:42.022 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f734548:::benchmark_data_smithi188_113957_object67367:head# 2024-04-23T16:33:42.022 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f7352c2:::benchmark_data_smithi188_113957_object130948:head# 2024-04-23T16:33:42.023 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f735f86:::benchmark_data_smithi188_113957_object105807:head# 2024-04-23T16:33:42.023 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f737b95:::benchmark_data_smithi188_113957_object127526:head# 2024-04-23T16:33:42.023 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f7381f4:::benchmark_data_smithi188_113957_object56823:head# 2024-04-23T16:33:42.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f740f8d:::benchmark_data_smithi188_113957_object45662:head# 2024-04-23T16:33:42.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f745c69:::benchmark_data_smithi188_113957_object78655:head# 2024-04-23T16:33:42.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f7505db:::benchmark_data_smithi188_113957_object98034:head# 2024-04-23T16:33:42.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f775292:::benchmark_data_smithi188_113957_object29725:head# 2024-04-23T16:33:42.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f775ba6:::benchmark_data_smithi188_113957_object149878:head# 2024-04-23T16:33:42.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f77cc52:::benchmark_data_smithi188_113957_object51742:head# 2024-04-23T16:33:42.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f78aef6:::benchmark_data_smithi188_113957_object37646:head# 2024-04-23T16:33:42.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f78c60e:::benchmark_data_smithi188_113957_object93372:head# 2024-04-23T16:33:42.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f79c8a1:::benchmark_data_smithi188_113957_object94082:head# 2024-04-23T16:33:42.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f7a0828:::benchmark_data_smithi188_113957_object44331:head# 2024-04-23T16:33:42.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f7a71ba:::benchmark_data_smithi188_113957_object120634:head# 2024-04-23T16:33:42.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f7a9555:::benchmark_data_smithi188_113957_object73661:head# 2024-04-23T16:33:42.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f7bdfc4:::benchmark_data_smithi188_113957_object49501:head# 2024-04-23T16:33:42.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f7be3ca:::benchmark_data_smithi188_113957_object49544:head# 2024-04-23T16:33:42.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f7c52fa:::benchmark_data_smithi188_113957_object133918:head# 2024-04-23T16:33:42.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f7cfcb7:::benchmark_data_smithi188_113957_object88251:head# 2024-04-23T16:33:42.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f7e3d2a:::benchmark_data_smithi188_113957_object112982:head# 2024-04-23T16:33:42.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f7ee3b9:::benchmark_data_smithi188_113957_object68050:head# 2024-04-23T16:33:42.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f7f0086:::benchmark_data_smithi188_113957_object98419:head# 2024-04-23T16:33:42.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f7f01c2:::benchmark_data_smithi188_113957_object97721:head# 2024-04-23T16:33:42.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f7fcbe4:::benchmark_data_smithi188_113957_object52377:head# 2024-04-23T16:33:42.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f7ff097:::benchmark_data_smithi188_113957_object33449:head# 2024-04-23T16:33:42.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f802091:::benchmark_data_smithi188_113957_object59496:head# 2024-04-23T16:33:42.036 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:33:42.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f802b6d:::benchmark_data_smithi188_113957_object128363:head# 2024-04-23T16:33:42.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f813831:::benchmark_data_smithi188_113957_object88802:head# 2024-04-23T16:33:42.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f81cd03:::benchmark_data_smithi188_113957_object88978:head# 2024-04-23T16:33:42.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f820d32:::benchmark_data_smithi188_113957_object28193:head# 2024-04-23T16:33:42.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f8237b5:::benchmark_data_smithi188_113957_object74636:head# 2024-04-23T16:33:42.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f82dacb:::benchmark_data_smithi188_113957_object81492:head# 2024-04-23T16:33:42.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f835f39:::benchmark_data_smithi188_113957_object105461:head# 2024-04-23T16:33:42.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f83640f:::benchmark_data_smithi188_113957_object15241:head# 2024-04-23T16:33:42.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f8399ab:::benchmark_data_smithi188_113957_object151355:head# 2024-04-23T16:33:42.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f840454:::benchmark_data_smithi188_113957_object18895:head# 2024-04-23T16:33:42.040 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f840fa6:::benchmark_data_smithi188_113957_object74447:head# 2024-04-23T16:33:42.040 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:33:42.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f844bbf:::benchmark_data_smithi188_113957_object106815:head# 2024-04-23T16:33:42.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f84933f:::benchmark_data_smithi188_113957_object31475:head# 2024-04-23T16:33:42.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f8552f4:::benchmark_data_smithi188_113957_object65219:head# 2024-04-23T16:33:42.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f86323a:::benchmark_data_smithi188_113957_object76388:head# 2024-04-23T16:33:42.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f864966:::benchmark_data_smithi188_113957_object136903:head# 2024-04-23T16:33:42.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f869480:::benchmark_data_smithi188_113957_object91684:head# 2024-04-23T16:33:42.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f87dcc6:::benchmark_data_smithi188_113957_object116199:head# 2024-04-23T16:33:42.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f87f314:::benchmark_data_smithi188_113957_object129712:head# 2024-04-23T16:33:42.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f87fc1c:::benchmark_data_smithi188_113957_object116326:head# 2024-04-23T16:33:42.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f889e58:::benchmark_data_smithi188_113957_object72388:head# 2024-04-23T16:33:42.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f893cec:::benchmark_data_smithi188_113957_object127410:head# 2024-04-23T16:33:42.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f895c71:::benchmark_data_smithi188_113957_object127411:head# 2024-04-23T16:33:42.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f89b4e8:::benchmark_data_smithi188_113957_object109639:head# 2024-04-23T16:33:42.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f89d519:::benchmark_data_smithi188_113957_object65195:head# 2024-04-23T16:33:42.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f8a1d33:::benchmark_data_smithi188_113957_object1687:head# 2024-04-23T16:33:42.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f8a8117:::benchmark_data_smithi188_113957_object86043:head# 2024-04-23T16:33:42.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f8ac0cf:::benchmark_data_smithi188_113957_object116570:head# 2024-04-23T16:33:42.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f8c6c3d:::benchmark_data_smithi188_113957_object58216:head# 2024-04-23T16:33:42.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f8c6e25:::benchmark_data_smithi188_113957_object30600:head# 2024-04-23T16:33:42.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f8cb159:::benchmark_data_smithi188_113957_object97844:head# 2024-04-23T16:33:42.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f8d9756:::benchmark_data_smithi188_113957_object9495:head# 2024-04-23T16:33:42.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f8dca4a:::benchmark_data_smithi188_113957_object125028:head# 2024-04-23T16:33:42.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f8edff1:::benchmark_data_smithi188_113957_object310:head# 2024-04-23T16:33:42.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f8f27b1:::benchmark_data_smithi188_113957_object111776:head# 2024-04-23T16:33:42.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f8fb07b:::benchmark_data_smithi188_113957_object39290:head# 2024-04-23T16:33:42.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f8fcaf6:::benchmark_data_smithi188_113957_object72599:head# 2024-04-23T16:33:42.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f8fdce8:::benchmark_data_smithi188_113957_object61552:head# 2024-04-23T16:33:42.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f908753:::benchmark_data_smithi188_113957_object95919:head# 2024-04-23T16:33:42.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f90da4c:::benchmark_data_smithi188_113957_object88518:head# 2024-04-23T16:33:42.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f910515:::benchmark_data_smithi188_113957_object354:head# 2024-04-23T16:33:42.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f910800:::benchmark_data_smithi188_113957_object135641:head# 2024-04-23T16:33:42.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f912ee8:::benchmark_data_smithi188_113957_object150795:head# 2024-04-23T16:33:42.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f91bd63:::benchmark_data_smithi188_113957_object136469:head# 2024-04-23T16:33:42.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f92d8e3:::benchmark_data_smithi188_113957_object107703:head# 2024-04-23T16:33:42.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f9384c6:::benchmark_data_smithi188_113957_object35542:head# 2024-04-23T16:33:42.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f93bc7b:::benchmark_data_smithi188_113957_object7537:head# 2024-04-23T16:33:42.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f93d076:::benchmark_data_smithi188_113957_object29987:head# 2024-04-23T16:33:42.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f93dcb9:::benchmark_data_smithi188_113957_object5493:head# 2024-04-23T16:33:42.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f93e9a6:::benchmark_data_smithi188_113957_object51955:head# 2024-04-23T16:33:42.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f93fc08:::benchmark_data_smithi188_113957_object149006:head# 2024-04-23T16:33:42.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f944fc8:::benchmark_data_smithi188_113957_object92587:head# 2024-04-23T16:33:42.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f950783:::benchmark_data_smithi188_113957_object83328:head# 2024-04-23T16:33:42.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f955091:::benchmark_data_smithi188_113957_object46493:head# 2024-04-23T16:33:42.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f95547d:::benchmark_data_smithi188_113957_object148747:head# 2024-04-23T16:33:42.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f95b385:::benchmark_data_smithi188_113957_object146869:head# 2024-04-23T16:33:42.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f95fed3:::benchmark_data_smithi188_113957_object142557:head# 2024-04-23T16:33:42.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f96208a:::benchmark_data_smithi188_113957_object33535:head# 2024-04-23T16:33:42.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f96306b:::benchmark_data_smithi188_113957_object62757:head# 2024-04-23T16:33:42.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f9647d9:::benchmark_data_smithi188_113957_object65458:head# 2024-04-23T16:33:42.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f968b82:::benchmark_data_smithi188_113957_object137124:head# 2024-04-23T16:33:42.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f969de7:::benchmark_data_smithi188_113957_object139807:head# 2024-04-23T16:33:42.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f97b31e:::benchmark_data_smithi188_113957_object41083:head# 2024-04-23T16:33:42.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f99e91a:::benchmark_data_smithi188_113957_object38566:head# 2024-04-23T16:33:42.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f9a3ac3:::benchmark_data_smithi188_113957_object149730:head# 2024-04-23T16:33:42.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f9a3e4b:::benchmark_data_smithi188_113957_object124412:head# 2024-04-23T16:33:42.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f9a44bb:::benchmark_data_smithi188_113957_object97580:head# 2024-04-23T16:33:42.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f9a9c46:::benchmark_data_smithi188_113957_object51165:head# 2024-04-23T16:33:42.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f9b8534:::benchmark_data_smithi188_113957_object106021:head# 2024-04-23T16:33:42.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f9c81b7:::benchmark_data_smithi188_113957_object100012:head# 2024-04-23T16:33:42.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f9d1a4b:::benchmark_data_smithi188_113957_object3147:head# 2024-04-23T16:33:42.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f9d223a:::benchmark_data_smithi188_113957_object48426:head# 2024-04-23T16:33:42.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f9d76f2:::benchmark_data_smithi188_113957_object147952:head# 2024-04-23T16:33:42.073 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:42.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f9db1cb:::benchmark_data_smithi188_113957_object58970:head# 2024-04-23T16:33:42.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f9e7f59:::benchmark_data_smithi188_113957_object103164:head# 2024-04-23T16:33:42.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f9f0d31:::benchmark_data_smithi188_113957_object98786:head# 2024-04-23T16:33:42.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f9f315f:::benchmark_data_smithi188_113957_object50107:head# 2024-04-23T16:33:42.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f9f3820:::benchmark_data_smithi188_113957_object47408:head# 2024-04-23T16:33:42.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5f9fe05b:::benchmark_data_smithi188_113957_object136530:head# 2024-04-23T16:33:42.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fa135e6:::benchmark_data_smithi188_113957_object1691:head# 2024-04-23T16:33:42.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fa1a4e9:::benchmark_data_smithi188_113957_object139636:head# 2024-04-23T16:33:42.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fa1d45e:::benchmark_data_smithi188_113957_object89662:head# 2024-04-23T16:33:42.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fa1d996:::benchmark_data_smithi188_113957_object95329:head# 2024-04-23T16:33:42.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fa1e617:::benchmark_data_smithi188_113957_object9118:head# 2024-04-23T16:33:42.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fa215c4:::benchmark_data_smithi188_113957_object119824:head# 2024-04-23T16:33:42.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fa24110:::benchmark_data_smithi188_113957_object20708:head# 2024-04-23T16:33:42.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fa33820:::benchmark_data_smithi188_113957_object24535:head# 2024-04-23T16:33:42.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fa416e2:::benchmark_data_smithi188_113957_object107934:head# 2024-04-23T16:33:42.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fa5860a:::benchmark_data_smithi188_113957_object86815:head# 2024-04-23T16:33:42.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fa58778:::benchmark_data_smithi188_113957_object138311:head# 2024-04-23T16:33:42.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fa5ab69:::benchmark_data_smithi188_113957_object48706:head# 2024-04-23T16:33:42.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fa6edd1:::benchmark_data_smithi188_113957_object32173:head# 2024-04-23T16:33:42.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fa743fd:::benchmark_data_smithi188_113957_object38459:head# 2024-04-23T16:33:42.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fa91300:::benchmark_data_smithi188_113957_object36488:head# 2024-04-23T16:33:42.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fa961dd:::benchmark_data_smithi188_113957_object131292:head# 2024-04-23T16:33:42.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fa98c66:::benchmark_data_smithi188_113957_object2804:head# 2024-04-23T16:33:42.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fa9be6b:::benchmark_data_smithi188_113957_object131798:head# 2024-04-23T16:33:42.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5faa3072:::benchmark_data_smithi188_113957_object6525:head# 2024-04-23T16:33:42.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fab8c24:::benchmark_data_smithi188_113957_object64380:head# 2024-04-23T16:33:42.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fabc7be:::benchmark_data_smithi188_113957_object107432:head# 2024-04-23T16:33:42.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fac1fbe:::benchmark_data_smithi188_113957_object78479:head# 2024-04-23T16:33:42.087 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5facd5b3:::benchmark_data_smithi188_113957_object48576:head# 2024-04-23T16:33:42.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5facdcfe:::benchmark_data_smithi188_113957_object46771:head# 2024-04-23T16:33:42.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5facfe74:::benchmark_data_smithi188_113957_object78554:head# 2024-04-23T16:33:42.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fad4bf6:::benchmark_data_smithi188_113957_object55579:head# 2024-04-23T16:33:42.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fad4f79:::benchmark_data_smithi188_113957_object81359:head# 2024-04-23T16:33:42.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fadc125:::benchmark_data_smithi188_113957_object25228:head# 2024-04-23T16:33:42.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fae2558:::benchmark_data_smithi188_113957_object148910:head# 2024-04-23T16:33:42.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5faedd7f:::benchmark_data_smithi188_113957_object107595:head# 2024-04-23T16:33:42.095 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5faeef46:::benchmark_data_smithi188_113957_object54828:head# 2024-04-23T16:33:42.095 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5faf3a31:::benchmark_data_smithi188_113957_object107794:head# 2024-04-23T16:33:42.095 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5faf6350:::benchmark_data_smithi188_113957_object115282:head# 2024-04-23T16:33:42.096 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb00625:::benchmark_data_smithi188_113957_object88330:head# 2024-04-23T16:33:42.096 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb0075f:::benchmark_data_smithi188_113957_object70778:head# 2024-04-23T16:33:42.096 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb052cd:::benchmark_data_smithi188_113957_object51578:head# 2024-04-23T16:33:42.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb1d89f:::benchmark_data_smithi188_113957_object11424:head# 2024-04-23T16:33:42.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb1dbd1:::benchmark_data_smithi188_113957_object30628:head# 2024-04-23T16:33:42.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb1ff96:::benchmark_data_smithi188_113957_object91800:head# 2024-04-23T16:33:42.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb20c68:::benchmark_data_smithi188_113957_object26052:head# 2024-04-23T16:33:42.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb24800:::benchmark_data_smithi188_113957_object97006:head# 2024-04-23T16:33:42.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb2582e:::benchmark_data_smithi188_113957_object74382:head# 2024-04-23T16:33:42.099 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb29286:::benchmark_data_smithi188_113957_object63417:head# 2024-04-23T16:33:42.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb3628f:::benchmark_data_smithi188_113957_object56411:head# 2024-04-23T16:33:42.100 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb3ef9a:::benchmark_data_smithi188_113957_object93195:head# 2024-04-23T16:33:42.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb432a1:::benchmark_data_smithi188_113957_object3038:head# 2024-04-23T16:33:42.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb528f1:::benchmark_data_smithi188_113957_object89328:head# 2024-04-23T16:33:42.102 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb55501:::benchmark_data_smithi188_113957_object73152:head# 2024-04-23T16:33:42.102 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb570f3:::benchmark_data_smithi188_113957_object26640:head# 2024-04-23T16:33:42.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb5bad5:::benchmark_data_smithi188_113957_object60319:head# 2024-04-23T16:33:42.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb68898:::benchmark_data_smithi188_113957_object84143:head# 2024-04-23T16:33:42.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb74495:::benchmark_data_smithi188_113957_object141853:head# 2024-04-23T16:33:42.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb7d4d7:::benchmark_data_smithi188_113957_object88405:head# 2024-04-23T16:33:42.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb804f9:::benchmark_data_smithi188_113957_object148981:head# 2024-04-23T16:33:42.105 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb88400:::benchmark_data_smithi188_113957_object126898:head# 2024-04-23T16:33:42.105 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb949de:::benchmark_data_smithi188_113957_object94687:head# 2024-04-23T16:33:42.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fb99d20:::benchmark_data_smithi188_113957_object138468:head# 2024-04-23T16:33:42.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fba32ff:::benchmark_data_smithi188_113957_object12510:head# 2024-04-23T16:33:42.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fbae5e1:::benchmark_data_smithi188_113957_object85826:head# 2024-04-23T16:33:42.107 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fbb2df7:::benchmark_data_smithi188_113957_object67784:head# 2024-04-23T16:33:42.107 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fbb57d9:::benchmark_data_smithi188_113957_object30923:head# 2024-04-23T16:33:42.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fbba091:::benchmark_data_smithi188_113957_object52584:head# 2024-04-23T16:33:42.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fbc0fb3:::benchmark_data_smithi188_113957_object58006:head# 2024-04-23T16:33:42.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fbc3278:::benchmark_data_smithi188_113957_object152201:head# 2024-04-23T16:33:42.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fbd2f80:::benchmark_data_smithi188_113957_object110253:head# 2024-04-23T16:33:42.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fbdcbf5:::benchmark_data_smithi188_113957_object153457:head# 2024-04-23T16:33:42.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fbde273:::benchmark_data_smithi188_113957_object140261:head# 2024-04-23T16:33:42.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fbf4989:::benchmark_data_smithi188_113957_object72301:head# 2024-04-23T16:33:42.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fbf7dc0:::benchmark_data_smithi188_113957_object31001:head# 2024-04-23T16:33:42.111 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fbfe31b:::benchmark_data_smithi188_113957_object93164:head# 2024-04-23T16:33:42.111 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc03d51:::benchmark_data_smithi188_113957_object29526:head# 2024-04-23T16:33:42.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc0d485:::benchmark_data_smithi188_113957_object54206:head# 2024-04-23T16:33:42.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc1199d:::benchmark_data_smithi188_113957_object92554:head# 2024-04-23T16:33:42.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc1ee08:::benchmark_data_smithi188_113957_object111560:head# 2024-04-23T16:33:42.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc2492e:::benchmark_data_smithi188_113957_object112294:head# 2024-04-23T16:33:42.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc2fcc4:::benchmark_data_smithi188_113957_object118681:head# 2024-04-23T16:33:42.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc3085b:::benchmark_data_smithi188_113957_object104842:head# 2024-04-23T16:33:42.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc3551e:::benchmark_data_smithi188_113957_object37098:head# 2024-04-23T16:33:42.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc3da93:::benchmark_data_smithi188_113957_object27626:head# 2024-04-23T16:33:42.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc4c7be:::benchmark_data_smithi188_113957_object23894:head# 2024-04-23T16:33:42.118 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc52889:::benchmark_data_smithi188_113957_object46773:head# 2024-04-23T16:33:42.118 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc55b64:::benchmark_data_smithi188_113957_object118544:head# 2024-04-23T16:33:42.118 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc57a16:::benchmark_data_smithi188_113957_object105113:head# 2024-04-23T16:33:42.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc580d7:::benchmark_data_smithi188_113957_object143602:head# 2024-04-23T16:33:42.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc67e4f:::benchmark_data_smithi188_113957_object83802:head# 2024-04-23T16:33:42.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc6b293:::benchmark_data_smithi188_113957_object112055:head# 2024-04-23T16:33:42.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc6b637:::benchmark_data_smithi188_113957_object59562:head# 2024-04-23T16:33:42.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc74826:::benchmark_data_smithi188_113957_object85912:head# 2024-04-23T16:33:42.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc787fe:::benchmark_data_smithi188_113957_object78769:head# 2024-04-23T16:33:42.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc904f8:::benchmark_data_smithi188_113957_object133137:head# 2024-04-23T16:33:42.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fc91af5:::benchmark_data_smithi188_113957_object42897:head# 2024-04-23T16:33:42.122 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fca1116:::benchmark_data_smithi188_113957_object52781:head# 2024-04-23T16:33:42.122 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fca4cd9:::benchmark_data_smithi188_113957_object145176:head# 2024-04-23T16:33:42.125 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fca5c4f:::benchmark_data_smithi188_113957_object88235:head# 2024-04-23T16:33:42.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fca7d5a:::benchmark_data_smithi188_113957_object15256:head# 2024-04-23T16:33:42.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fcaa477:::benchmark_data_smithi188_113957_object145428:head# 2024-04-23T16:33:42.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fcadb9a:::benchmark_data_smithi188_113957_object97933:head# 2024-04-23T16:33:42.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fcafa07:::benchmark_data_smithi188_113957_object12351:head# 2024-04-23T16:33:42.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fcbbe6d:::benchmark_data_smithi188_113957_object108014:head# 2024-04-23T16:33:42.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fcc11f1:::benchmark_data_smithi188_113957_object115962:head# 2024-04-23T16:33:42.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fcc6cf2:::benchmark_data_smithi188_113957_object132050:head# 2024-04-23T16:33:42.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fcc7d1d:::benchmark_data_smithi188_113957_object149306:head# 2024-04-23T16:33:42.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fccf98f:::benchmark_data_smithi188_113957_object112389:head# 2024-04-23T16:33:42.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fcdf1a9:::benchmark_data_smithi188_113957_object61412:head# 2024-04-23T16:33:42.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fce3453:::benchmark_data_smithi188_113957_object55653:head# 2024-04-23T16:33:42.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fce501d:::benchmark_data_smithi188_113957_object89003:head# 2024-04-23T16:33:42.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fce9b07:::benchmark_data_smithi188_113957_object72203:head# 2024-04-23T16:33:42.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fceb3b6:::benchmark_data_smithi188_113957_object51347:head# 2024-04-23T16:33:42.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fcee2f5:::benchmark_data_smithi188_113957_object64031:head# 2024-04-23T16:33:42.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fcf3990:::benchmark_data_smithi188_113957_object142994:head# 2024-04-23T16:33:42.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fcf53ed:::benchmark_data_smithi188_113957_object7650:head# 2024-04-23T16:33:42.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fcf8418:::benchmark_data_smithi188_113957_object49013:head# 2024-04-23T16:33:42.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fcfeb8a:::benchmark_data_smithi188_113957_object126701:head# 2024-04-23T16:33:42.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd018bf:::benchmark_data_smithi188_113957_object83336:head# 2024-04-23T16:33:42.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd03651:::benchmark_data_smithi188_113957_object76641:head# 2024-04-23T16:33:42.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd04a45:::benchmark_data_smithi188_113957_object48512:head# 2024-04-23T16:33:42.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd14d74:::benchmark_data_smithi188_113957_object106429:head# 2024-04-23T16:33:42.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd17311:::benchmark_data_smithi188_113957_object135258:head# 2024-04-23T16:33:42.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd1b3de:::benchmark_data_smithi188_113957_object113443:head# 2024-04-23T16:33:42.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd1d25f:::benchmark_data_smithi188_113957_object132407:head# 2024-04-23T16:33:42.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd20415:::benchmark_data_smithi188_113957_object150460:head# 2024-04-23T16:33:42.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd23ed7:::benchmark_data_smithi188_113957_object4252:head# 2024-04-23T16:33:42.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd24d7f:::benchmark_data_smithi188_113957_object129697:head# 2024-04-23T16:33:42.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd33909:::benchmark_data_smithi188_113957_object109621:head# 2024-04-23T16:33:42.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd357f1:::benchmark_data_smithi188_113957_object2838:head# 2024-04-23T16:33:42.142 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd41743:::benchmark_data_smithi188_113957_object133236:head# 2024-04-23T16:33:42.142 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd48e80:::benchmark_data_smithi188_113957_object130146:head# 2024-04-23T16:33:42.142 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd49a8b:::benchmark_data_smithi188_113957_object98545:head# 2024-04-23T16:33:42.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd4a02a:::benchmark_data_smithi188_113957_object132552:head# 2024-04-23T16:33:42.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd4b866:::benchmark_data_smithi188_113957_object87002:head# 2024-04-23T16:33:42.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd4c315:::benchmark_data_smithi188_113957_object26402:head# 2024-04-23T16:33:42.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd588bc:::benchmark_data_smithi188_113957_object95724:head# 2024-04-23T16:33:42.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd5a7e4:::benchmark_data_smithi188_113957_object45249:head# 2024-04-23T16:33:42.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd5e042:::benchmark_data_smithi188_113957_object61657:head# 2024-04-23T16:33:42.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd5e401:::benchmark_data_smithi188_113957_object71756:head# 2024-04-23T16:33:42.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd5eb50:::benchmark_data_smithi188_113957_object145739:head# 2024-04-23T16:33:42.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd60964:::benchmark_data_smithi188_113957_object1980:head# 2024-04-23T16:33:42.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd62bdf:::benchmark_data_smithi188_113957_object66384:head# 2024-04-23T16:33:42.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd6ffa6:::benchmark_data_smithi188_113957_object43274:head# 2024-04-23T16:33:42.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd772fb:::benchmark_data_smithi188_113957_object153489:head# 2024-04-23T16:33:42.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd7b9b9:::benchmark_data_smithi188_113957_object56001:head# 2024-04-23T16:33:42.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd7f0d3:::benchmark_data_smithi188_113957_object130453:head# 2024-04-23T16:33:42.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd821ea:::benchmark_data_smithi188_113957_object117970:head# 2024-04-23T16:33:42.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd897d5:::benchmark_data_smithi188_113957_object77868:head# 2024-04-23T16:33:42.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fd8f621:::benchmark_data_smithi188_113957_object27411:head# 2024-04-23T16:33:42.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fda1a12:::benchmark_data_smithi188_113957_object66464:head# 2024-04-23T16:33:42.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fda74e4:::benchmark_data_smithi188_113957_object125636:head# 2024-04-23T16:33:42.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fdac9a9:::benchmark_data_smithi188_113957_object118339:head# 2024-04-23T16:33:42.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fdaea41:::benchmark_data_smithi188_113957_object4877:head# 2024-04-23T16:33:42.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fdbca60:::benchmark_data_smithi188_113957_object24456:head# 2024-04-23T16:33:42.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fdc5c33:::benchmark_data_smithi188_113957_object70496:head# 2024-04-23T16:33:42.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fdcab9f:::benchmark_data_smithi188_113957_object32529:head# 2024-04-23T16:33:42.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fdcb7e9:::benchmark_data_smithi188_113957_object29059:head# 2024-04-23T16:33:42.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fdd63f5:::benchmark_data_smithi188_113957_object80011:head# 2024-04-23T16:33:42.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fdefe53:::benchmark_data_smithi188_113957_object76407:head# 2024-04-23T16:33:42.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fdf0232:::benchmark_data_smithi188_113957_object135343:head# 2024-04-23T16:33:42.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fdf2168:::benchmark_data_smithi188_113957_object49478:head# 2024-04-23T16:33:42.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe01129:::benchmark_data_smithi188_113957_object95525:head# 2024-04-23T16:33:42.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe04759:::benchmark_data_smithi188_113957_object34384:head# 2024-04-23T16:33:42.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe0937c:::benchmark_data_smithi188_113957_object129043:head# 2024-04-23T16:33:42.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe0ee8f:::benchmark_data_smithi188_113957_object43241:head# 2024-04-23T16:33:42.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe15fa5:::benchmark_data_smithi188_113957_object39643:head# 2024-04-23T16:33:42.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe1d2f0:::benchmark_data_smithi188_113957_object87564:head# 2024-04-23T16:33:42.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe1d720:::benchmark_data_smithi188_113957_object94502:head# 2024-04-23T16:33:42.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe1f2ec:::benchmark_data_smithi188_113957_object30131:head# 2024-04-23T16:33:42.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe227f0:::benchmark_data_smithi188_113957_object35777:head# 2024-04-23T16:33:42.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe290cd:::benchmark_data_smithi188_113957_object65446:head# 2024-04-23T16:33:42.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe2ab42:::benchmark_data_smithi188_113957_object17531:head# 2024-04-23T16:33:42.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe31d09:::benchmark_data_smithi188_113957_object80520:head# 2024-04-23T16:33:42.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe3facd:::benchmark_data_smithi188_113957_object147835:head# 2024-04-23T16:33:42.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe4e39e:::benchmark_data_smithi188_113957_object50356:head# 2024-04-23T16:33:42.167 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe59ac3:::benchmark_data_smithi188_113957_object150603:head# 2024-04-23T16:33:42.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe5a4b3:::benchmark_data_smithi188_113957_object103163:head# 2024-04-23T16:33:42.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe5ddd6:::benchmark_data_smithi188_113957_object56310:head# 2024-04-23T16:33:42.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe68d57:::benchmark_data_smithi188_113957_object39975:head# 2024-04-23T16:33:42.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe6cb06:::benchmark_data_smithi188_113957_object14556:head# 2024-04-23T16:33:42.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe81541:::benchmark_data_smithi188_113957_object133767:head# 2024-04-23T16:33:42.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe88c2a:::benchmark_data_smithi188_113957_object147970:head# 2024-04-23T16:33:42.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe8c0b4:::benchmark_data_smithi188_113957_object12258:head# 2024-04-23T16:33:42.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe91a52:::benchmark_data_smithi188_113957_object46241:head# 2024-04-23T16:33:42.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe92aec:::benchmark_data_smithi188_113957_object80171:head# 2024-04-23T16:33:42.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe92fcf:::benchmark_data_smithi188_113957_object145723:head# 2024-04-23T16:33:42.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe99b42:::benchmark_data_smithi188_113957_object62021:head# 2024-04-23T16:33:42.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fe9a6dd:::benchmark_data_smithi188_113957_object15288:head# 2024-04-23T16:33:42.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5febe5f5:::benchmark_data_smithi188_113957_object9472:head# 2024-04-23T16:33:42.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fec5674:::benchmark_data_smithi188_113957_object37677:head# 2024-04-23T16:33:42.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fec8957:::benchmark_data_smithi188_113957_object80885:head# 2024-04-23T16:33:42.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fed1bb6:::benchmark_data_smithi188_113957_object81615:head# 2024-04-23T16:33:42.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fedb388:::benchmark_data_smithi188_113957_object59573:head# 2024-04-23T16:33:42.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fee0491:::benchmark_data_smithi188_113957_object101584:head# 2024-04-23T16:33:42.176 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5feece4f:::benchmark_data_smithi188_113957_object88493:head# 2024-04-23T16:33:42.176 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fef148e:::benchmark_data_smithi188_113957_object16781:head# 2024-04-23T16:33:42.176 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fef859f:::benchmark_data_smithi188_113957_object116000:head# 2024-04-23T16:33:42.177 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff035b8:::benchmark_data_smithi188_113957_object66681:head# 2024-04-23T16:33:42.177 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff058e6:::benchmark_data_smithi188_113957_object152770:head# 2024-04-23T16:33:42.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff15c7f:::benchmark_data_smithi188_113957_object95837:head# 2024-04-23T16:33:42.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff1c554:::benchmark_data_smithi188_113957_object78853:head# 2024-04-23T16:33:42.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff24724:::benchmark_data_smithi188_113957_object119581:head# 2024-04-23T16:33:42.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff296ca:::benchmark_data_smithi188_113957_object90089:head# 2024-04-23T16:33:42.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff30028:::benchmark_data_smithi188_113957_object89782:head# 2024-04-23T16:33:42.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff3446b:::benchmark_data_smithi188_113957_object147266:head# 2024-04-23T16:33:42.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff3d471:::benchmark_data_smithi188_113957_object143918:head# 2024-04-23T16:33:42.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff3f095:::benchmark_data_smithi188_113957_object83460:head# 2024-04-23T16:33:42.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff3f1c6:::benchmark_data_smithi188_113957_object102314:head# 2024-04-23T16:33:42.181 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff4628c:::benchmark_data_smithi188_113957_object83809:head# 2024-04-23T16:33:42.181 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff4d3c8:::benchmark_data_smithi188_113957_object147817:head# 2024-04-23T16:33:42.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff54f2d:::benchmark_data_smithi188_113957_object47491:head# 2024-04-23T16:33:42.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff5acc8:::benchmark_data_smithi188_113957_object29095:head# 2024-04-23T16:33:42.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff5dd69:::benchmark_data_smithi188_113957_object58708:head# 2024-04-23T16:33:42.183 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff5f4a8:::benchmark_data_smithi188_113957_object42256:head# 2024-04-23T16:33:42.183 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff63f5e:::benchmark_data_smithi188_113957_object130123:head# 2024-04-23T16:33:42.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff64c0a:::benchmark_data_smithi188_113957_object34678:head# 2024-04-23T16:33:42.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff6540a:::benchmark_data_smithi188_113957_object43355:head# 2024-04-23T16:33:42.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff672b1:::benchmark_data_smithi188_113957_object43632:head# 2024-04-23T16:33:42.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff6b57b:::benchmark_data_smithi188_113957_object102533:head# 2024-04-23T16:33:42.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff81250:::benchmark_data_smithi188_113957_object25098:head# 2024-04-23T16:33:42.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff9639d:::benchmark_data_smithi188_113957_object16205:head# 2024-04-23T16:33:42.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff967c3:::benchmark_data_smithi188_113957_object95200:head# 2024-04-23T16:33:42.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff97a64:::benchmark_data_smithi188_113957_object85767:head# 2024-04-23T16:33:42.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff9daa0:::benchmark_data_smithi188_113957_object123954:head# 2024-04-23T16:33:42.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ff9ebf9:::benchmark_data_smithi188_113957_object52429:head# 2024-04-23T16:33:42.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ffacdff:::benchmark_data_smithi188_113957_object80940:head# 2024-04-23T16:33:42.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ffae8be:::benchmark_data_smithi188_113957_object30621:head# 2024-04-23T16:33:42.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ffb1a0c:::benchmark_data_smithi188_113957_object27180:head# 2024-04-23T16:33:42.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ffb77aa:::benchmark_data_smithi188_113957_object146942:head# 2024-04-23T16:33:42.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ffc4332:::benchmark_data_smithi188_113957_object25828:head# 2024-04-23T16:33:42.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ffcd61a:::benchmark_data_smithi188_113957_object30414:head# 2024-04-23T16:33:42.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ffe1be8:::benchmark_data_smithi188_113957_object37228:head# 2024-04-23T16:33:42.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ffec2a8:::benchmark_data_smithi188_113957_object37519:head# 2024-04-23T16:33:42.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fff680f:::benchmark_data_smithi188_113957_object144398:head# 2024-04-23T16:33:42.192 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fff8025:::benchmark_data_smithi188_113957_object62292:head# 2024-04-23T16:33:42.192 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5fffd9c3:::benchmark_data_smithi188_113957_object90862:head# 2024-04-23T16:33:42.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #5:5ffff964:::benchmark_data_smithi188_113957_object30407:head# 2024-04-23T16:33:42.196 INFO:teuthology.orchestra.run.smithi103.stderr:Export successful 2024-04-23T16:33:42.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:42 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3968621477' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:33:42.377 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:33:42.377 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:33:42.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:42 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3968621477' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:33:42.924 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:33:42.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:42.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:33:42.944 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:33:43.029 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:43.092 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:43 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:33:43.092 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:43 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3968621477' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:33:43.092 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:43 smithi148 ceph-mon[44520]: osdmap e398: 12 total, 11 up, 9 in 2024-04-23T16:33:43.092 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:43 smithi148 ceph-mon[44520]: pgmap v1122: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.9 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 461 op/s; 27626/292076 objects degraded (9.458%); 4.9 MiB/s, 78 objects/s recovering 2024-04-23T16:33:43.098 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:43.155 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:43.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:43.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:43.210 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-23T16:33:43.265 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:43.326 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:43 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:33:43.326 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:43 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3968621477' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:33:43.326 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:43 smithi103 ceph-mon[39629]: osdmap e398: 12 total, 11 up, 9 in 2024-04-23T16:33:43.326 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:43 smithi103 ceph-mon[39629]: pgmap v1122: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.9 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 461 op/s; 27626/292076 objects degraded (9.458%); 4.9 MiB/s, 78 objects/s recovering 2024-04-23T16:33:43.348 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:43.376 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:43.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:43.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:43 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:33:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:43 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3968621477' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:33:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:43 smithi188 ceph-mon[44101]: osdmap e398: 12 total, 11 up, 9 in 2024-04-23T16:33:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:43 smithi188 ceph-mon[44101]: pgmap v1122: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.9 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 461 op/s; 27626/292076 objects degraded (9.458%); 4.9 MiB/s, 78 objects/s recovering 2024-04-23T16:33:43.488 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:43.554 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:43.594 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:43.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:43.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:43.689 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:33:43.699 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:33:43.729 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:43.751 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:33:43.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:43.757 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:33:43.834 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.2 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 5.a 2024-04-23T16:33:43.857 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:43.917 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:33:43.928 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:43.994 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:43.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:43.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:44.078 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.2/config 2024-04-23T16:33:44.099 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:44.163 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:44.321 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:44.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:44.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:44.432 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:33:44.462 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:44.488 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:33:44.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:44.494 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:33:44.595 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:33:44.623 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:44.642 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:33:44.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:44.646 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:33:44.747 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:33:44.779 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:44.958 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:33:45.269 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:45 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:33:45.269 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:45 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:33:45.269 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:45 smithi148 ceph-mon[44520]: osdmap e399: 12 total, 11 up, 9 in 2024-04-23T16:33:45.269 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:33:45.269 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:33:45.269 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:33:45.269 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:33:45.270 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:33:45.270 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:33:45.270 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:33:45.270 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:45 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 27626/292076 objects degraded (9.458%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:45.270 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:33:45.270 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:33:45.270 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:33:45.270 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:33:45.270 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:33:45.270 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:33:45.270 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:33:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:45 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:33:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:45 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:33:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:45 smithi103 ceph-mon[39629]: osdmap e399: 12 total, 11 up, 9 in 2024-04-23T16:33:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:33:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:33:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:33:45.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:33:45.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:33:45.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:33:45.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:33:45.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:45 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 27626/292076 objects degraded (9.458%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:45.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:33:45.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:33:45.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:33:45.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:33:45.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:33:45.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:33:45.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:33:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:45 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:33:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:45 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:33:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:45 smithi188 ceph-mon[44101]: osdmap e399: 12 total, 11 up, 9 in 2024-04-23T16:33:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:33:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:33:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:33:45.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:33:45.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:33:45.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:33:45.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:33:45.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:45 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 27626/292076 objects degraded (9.458%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:45.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:33:45.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:33:45.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:33:45.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:33:45.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:33:45.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:33:45.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:33:45.937 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:33:45.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:45.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:33:45.985 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:33:45.986 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:33:46.042 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:46.122 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:46.262 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:46.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:46.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:46.365 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:33:46.449 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:46.752 INFO:teuthology.orchestra.run.smithi103.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:33:46.833 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-23T16:33:46.835 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:33:46.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:46.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:33:46.943 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:47.011 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:47.080 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:47.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:47.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:47.195 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:33:47.223 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:47.247 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:33:47.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:47.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:33:47.299 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:33:47.355 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:33:47.383 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:47.403 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:33:47.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:47.407 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:33:47.508 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:33:47.538 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:47.560 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:33:47.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:47.562 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:33:47.663 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:33:47.667 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:33:47.667 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:33:47.678 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:33:47.702 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:47.707 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:33:47.742 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:33:47.766 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:33:47.799 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:33:47.822 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:33:47.830 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:33:47.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:47.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:33:47.845 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:33:47.865 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:33:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:47 smithi148 ceph-mon[44520]: pgmap v1125: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 622 op/s; 27055/286662 objects degraded (9.438%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:33:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:47 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1563542716' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:33:47.885 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:33:47.886 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:33:47.886 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:33:47.886 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:33:47.886 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:33:47.886 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:33:47.886 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:33:47.886 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:33:47.886 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:33:47.886 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:33:47.886 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:33:47.886 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:33:47.887 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:33:47.902 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:33:47.935 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:33:47.967 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:47 smithi103 ceph-mon[39629]: pgmap v1125: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 622 op/s; 27055/286662 objects degraded (9.438%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:33:47.967 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:47 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1563542716' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:33:47.969 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:47 smithi188 ceph-mon[44101]: pgmap v1125: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 622 op/s; 27055/286662 objects degraded (9.438%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:33:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:47 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1563542716' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:33:48.168 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:33:48.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:48.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:33:48.273 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:33:48.301 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:48.321 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:33:48.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:48.324 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:33:48.425 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:33:48.456 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:48.477 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:33:48.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:48.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:33:48.582 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:33:48.612 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:48.714 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:33:48.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:48.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:33:48.831 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:33:48.862 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:48.882 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:33:48.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:48.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:33:48.986 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:33:49.014 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:49.041 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:33:49.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.051 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:49.051 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:33:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:33:49.152 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:49.206 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:49.332 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:49.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:49.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:49.438 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:33:49.470 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:49.524 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:33:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: marking collection for removal 2024-04-23T16:33:49.528 INFO:teuthology.orchestra.run.smithi103.stdout:setting '_remove' omap key 2024-04-23T16:33:49.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:49.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:33:49.533 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:33:49.534 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2024-04-23T16:33:49.534 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2024-04-23T16:33:49.534 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:33:49.534 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.534 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:186151 3.f 3:fe61d7fc:::benchmark_data_smithi188_89276_object17507:head [delete] snapc 0=[] ondisk+write+known_if_redirected e229)", 2024-04-23T16:33:49.534 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:24:11.193916+0000", 2024-04-23T16:33:49.535 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 578.33373532500002, 2024-04-23T16:33:49.535 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.058288421, 2024-04-23T16:33:49.535 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.535 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.535 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.536 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:49.536 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:33:49.536 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:33:49.536 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 186151 2024-04-23T16:33:49.536 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.536 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.537 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.537 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.537 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.193916+0000", 2024-04-23T16:33:49.537 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.537 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.537 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.538 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.538 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.193916+0000", 2024-04-23T16:33:49.538 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.538 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.538 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.539 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.539 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.193919+0000", 2024-04-23T16:33:49.539 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2869999999999999e-06 2024-04-23T16:33:49.539 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.539 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.539 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.540 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.193925+0000", 2024-04-23T16:33:49.540 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.088e-06 2024-04-23T16:33:49.540 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.540 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.540 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.540 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.193939+0000", 2024-04-23T16:33:49.541 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3930999999999999e-05 2024-04-23T16:33:49.541 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.541 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.541 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.541 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.193946+0000", 2024-04-23T16:33:49.541 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2989999999999999e-06 2024-04-23T16:33:49.542 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.542 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.542 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.542 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.194011+0000", 2024-04-23T16:33:49.542 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4572000000000005e-05 2024-04-23T16:33:49.542 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.543 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.543 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.543 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.194136+0000", 2024-04-23T16:33:49.543 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012519899999999999 2024-04-23T16:33:49.543 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.543 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.544 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:33:49.544 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.194256+0000", 2024-04-23T16:33:49.544 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000119793 2024-04-23T16:33:49.544 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.544 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.545 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:49.545 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.198004+0000", 2024-04-23T16:33:49.545 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0037486479999999998 2024-04-23T16:33:49.545 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.545 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.545 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:49.546 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.252101+0000", 2024-04-23T16:33:49.546 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.054097112000000003 2024-04-23T16:33:49.546 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.546 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.546 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.546 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.252156+0000", 2024-04-23T16:33:49.546 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4833999999999997e-05 2024-04-23T16:33:49.547 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.547 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.547 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.547 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.252205+0000", 2024-04-23T16:33:49.547 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8658000000000001e-05 2024-04-23T16:33:49.547 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.547 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.547 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.549 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.549 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.549 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:186161 3.f 3:f26e67fc:::benchmark_data_smithi188_89276_object17517:head [delete] snapc 0=[] ondisk+write+known_if_redirected e229)", 2024-04-23T16:33:49.550 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:24:11.204292+0000", 2024-04-23T16:33:49.550 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 578.32335975199999, 2024-04-23T16:33:49.550 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.05347905, 2024-04-23T16:33:49.550 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.550 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.550 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.550 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:49.550 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:33:49.551 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:33:49.551 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 186161 2024-04-23T16:33:49.551 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.551 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.551 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.551 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.551 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.204292+0000", 2024-04-23T16:33:49.551 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.551 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.551 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.551 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.551 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.204292+0000", 2024-04-23T16:33:49.551 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.552 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.552 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.552 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.552 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.204294+0000", 2024-04-23T16:33:49.552 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3089999999999998e-06 2024-04-23T16:33:49.552 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.552 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.552 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.552 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.204302+0000", 2024-04-23T16:33:49.552 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1550000000000001e-06 2024-04-23T16:33:49.552 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.552 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.552 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.553 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.204316+0000", 2024-04-23T16:33:49.553 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4632e-05 2024-04-23T16:33:49.553 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.553 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.553 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.553 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.204323+0000", 2024-04-23T16:33:49.553 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0870000000000001e-06 2024-04-23T16:33:49.553 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.553 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.553 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.553 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.204378+0000", 2024-04-23T16:33:49.553 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5238999999999997e-05 2024-04-23T16:33:49.553 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.553 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.554 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.554 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.204501+0000", 2024-04-23T16:33:49.554 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012240600000000001 2024-04-23T16:33:49.554 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.554 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.554 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:33:49.554 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.204619+0000", 2024-04-23T16:33:49.554 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000118387 2024-04-23T16:33:49.554 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.554 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.554 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:49.554 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.208499+0000", 2024-04-23T16:33:49.554 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0038792539999999999 2024-04-23T16:33:49.554 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.555 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.555 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:49.555 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.257696+0000", 2024-04-23T16:33:49.555 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.049197685999999997 2024-04-23T16:33:49.555 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.555 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.555 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.555 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.257729+0000", 2024-04-23T16:33:49.555 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2963000000000003e-05 2024-04-23T16:33:49.555 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.555 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.555 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.555 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:11.257771+0000", 2024-04-23T16:33:49.555 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1931999999999998e-05 2024-04-23T16:33:49.556 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.556 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.556 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.556 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.556 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.556 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:187427 3.f 3:f21f5638:::benchmark_data_smithi188_89276_object18783:head [delete] snapc 0=[] ondisk+write+known_if_redirected e229)", 2024-04-23T16:33:49.556 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:24:12.258172+0000", 2024-04-23T16:33:49.556 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 577.26948014499999, 2024-04-23T16:33:49.556 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.056675859000000002, 2024-04-23T16:33:49.556 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.556 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.556 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.556 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:49.556 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:33:49.557 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:33:49.557 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 187427 2024-04-23T16:33:49.557 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.557 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.557 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.557 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.557 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.258172+0000", 2024-04-23T16:33:49.557 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.557 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.557 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.557 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.557 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.258172+0000", 2024-04-23T16:33:49.557 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.557 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.557 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.558 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.558 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.258174+0000", 2024-04-23T16:33:49.558 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2630000000000002e-06 2024-04-23T16:33:49.558 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.558 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.558 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.558 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.258180+0000", 2024-04-23T16:33:49.558 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2140000000000001e-06 2024-04-23T16:33:49.558 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.558 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.558 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.558 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.258192+0000", 2024-04-23T16:33:49.559 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1994000000000001e-05 2024-04-23T16:33:49.559 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.559 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.559 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.559 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.258198+0000", 2024-04-23T16:33:49.559 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9920000000000004e-06 2024-04-23T16:33:49.559 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.559 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.559 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.559 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.261509+0000", 2024-04-23T16:33:49.559 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0033112229999999999 2024-04-23T16:33:49.559 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.559 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.559 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.559 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.261596+0000", 2024-04-23T16:33:49.560 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6254999999999993e-05 2024-04-23T16:33:49.560 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.560 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.560 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:33:49.560 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.261678+0000", 2024-04-23T16:33:49.560 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2259e-05 2024-04-23T16:33:49.560 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.560 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.560 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:49.560 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.269717+0000", 2024-04-23T16:33:49.560 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0080386859999999997 2024-04-23T16:33:49.560 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.560 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.560 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:49.561 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.314697+0000", 2024-04-23T16:33:49.561 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.044980458000000001 2024-04-23T16:33:49.561 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.561 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.561 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.561 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.314771+0000", 2024-04-23T16:33:49.561 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4269000000000003e-05 2024-04-23T16:33:49.561 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.561 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.561 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.561 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.314848+0000", 2024-04-23T16:33:49.561 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6246000000000001e-05 2024-04-23T16:33:49.561 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.562 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.562 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.562 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.562 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.562 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:187428 3.f 3:f58fa2bd:::benchmark_data_smithi188_89276_object18784:head [delete] snapc 0=[] ondisk+write+known_if_redirected e229)", 2024-04-23T16:33:49.562 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:24:12.258294+0000", 2024-04-23T16:33:49.562 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 577.26935776400001, 2024-04-23T16:33:49.562 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.056664345999999997, 2024-04-23T16:33:49.562 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.562 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.562 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.562 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:49.562 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:33:49.563 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:33:49.563 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 187428 2024-04-23T16:33:49.563 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.563 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.563 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.563 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.563 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.258294+0000", 2024-04-23T16:33:49.563 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.563 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.563 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.563 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.563 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.258294+0000", 2024-04-23T16:33:49.563 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.564 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.564 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.564 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.564 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.258295+0000", 2024-04-23T16:33:49.564 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4300000000000003e-07 2024-04-23T16:33:49.564 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.564 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.564 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.564 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.258297+0000", 2024-04-23T16:33:49.564 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.384e-06 2024-04-23T16:33:49.564 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.564 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.564 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.564 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.258303+0000", 2024-04-23T16:33:49.564 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.446e-06 2024-04-23T16:33:49.565 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.565 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.565 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.565 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.258306+0000", 2024-04-23T16:33:49.565 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8530000000000002e-06 2024-04-23T16:33:49.565 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.565 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.565 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.565 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.264826+0000", 2024-04-23T16:33:49.565 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0065197190000000002 2024-04-23T16:33:49.565 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.565 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.565 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.566 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.264931+0000", 2024-04-23T16:33:49.566 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000104858 2024-04-23T16:33:49.566 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.566 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.566 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:33:49.566 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.265029+0000", 2024-04-23T16:33:49.566 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7777999999999998e-05 2024-04-23T16:33:49.566 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.566 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.566 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:49.566 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.269720+0000", 2024-04-23T16:33:49.566 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00469113 2024-04-23T16:33:49.566 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.566 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.567 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:49.567 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.314890+0000", 2024-04-23T16:33:49.567 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.045169764000000001 2024-04-23T16:33:49.567 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.567 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.567 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.567 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.314921+0000", 2024-04-23T16:33:49.567 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1537e-05 2024-04-23T16:33:49.567 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.567 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.567 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.567 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:24:12.314958+0000", 2024-04-23T16:33:49.567 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7234000000000003e-05 2024-04-23T16:33:49.567 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.567 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.568 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.568 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.568 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.568 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15150.0:312876 3.f 3:f2d5cf85:::benchmark_data_smithi188_89276_object144232:head [delete] snapc 0=[] ondisk+write+known_if_redirected e264)", 2024-04-23T16:33:49.568 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:26:33.861538+0000", 2024-04-23T16:33:49.568 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 435.66611332399998, 2024-04-23T16:33:49.568 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052721704000000001, 2024-04-23T16:33:49.568 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.568 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.568 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.568 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:49.568 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15150", 2024-04-23T16:33:49.568 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:33:49.568 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 312876 2024-04-23T16:33:49.569 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.569 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.569 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.569 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.569 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:33.861538+0000", 2024-04-23T16:33:49.569 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.569 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.569 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.569 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.569 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:33.861538+0000", 2024-04-23T16:33:49.569 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.569 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.569 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.569 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.570 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:33.861541+0000", 2024-04-23T16:33:49.570 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1909999999999999e-06 2024-04-23T16:33:49.570 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.570 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.570 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.570 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:33.861547+0000", 2024-04-23T16:33:49.570 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0290000000000002e-06 2024-04-23T16:33:49.570 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.570 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.570 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.570 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:33.861561+0000", 2024-04-23T16:33:49.571 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4503e-05 2024-04-23T16:33:49.571 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.571 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.571 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.571 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:33.861570+0000", 2024-04-23T16:33:49.571 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4300000000000006e-06 2024-04-23T16:33:49.571 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.571 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.571 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.571 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:33.865572+0000", 2024-04-23T16:33:49.571 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.004002785 2024-04-23T16:33:49.571 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.571 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.571 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.571 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:33.865691+0000", 2024-04-23T16:33:49.572 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000118374 2024-04-23T16:33:49.572 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.572 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.572 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:33:49.572 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:33.865818+0000", 2024-04-23T16:33:49.572 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000127663 2024-04-23T16:33:49.572 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.572 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.572 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:49.572 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:33.881382+0000", 2024-04-23T16:33:49.572 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015563553000000001 2024-04-23T16:33:49.572 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.572 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.573 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:49.573 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:33.914165+0000", 2024-04-23T16:33:49.573 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.032783060000000003 2024-04-23T16:33:49.573 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.573 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.573 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.573 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:33.914213+0000", 2024-04-23T16:33:49.573 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8047000000000003e-05 2024-04-23T16:33:49.573 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.573 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.573 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.573 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:26:33.914260+0000", 2024-04-23T16:33:49.573 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7068999999999999e-05 2024-04-23T16:33:49.574 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.574 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.574 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.574 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.574 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.574 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.15486.0:267169 4.8 e339/274 4:1684fc24:::benchmark_data_smithi188_101370_object103720:head v 339'16618, mlcod=339'16614)", 2024-04-23T16:33:49.574 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:30:15.490781+0000", 2024-04-23T16:33:49.574 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 214.03687055899999, 2024-04-23T16:33:49.574 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052489635999999999, 2024-04-23T16:33:49.574 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.574 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.574 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.574 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.575 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.575 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.575 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:15.490781+0000", 2024-04-23T16:33:49.575 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.575 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.575 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.575 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.575 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:15.490781+0000", 2024-04-23T16:33:49.575 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.575 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.575 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.575 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.575 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:15.490782+0000", 2024-04-23T16:33:49.575 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2099999999999997e-07 2024-04-23T16:33:49.576 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.576 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.576 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.576 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:15.490786+0000", 2024-04-23T16:33:49.576 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4819999999999996e-06 2024-04-23T16:33:49.576 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.576 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.576 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.576 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:15.490791+0000", 2024-04-23T16:33:49.576 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8069999999999999e-06 2024-04-23T16:33:49.576 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.576 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.576 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.577 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:15.490795+0000", 2024-04-23T16:33:49.577 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1489999999999996e-06 2024-04-23T16:33:49.577 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.577 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.577 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.577 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:15.499501+0000", 2024-04-23T16:33:49.577 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0087056570000000003 2024-04-23T16:33:49.577 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.577 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.577 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.577 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:15.499523+0000", 2024-04-23T16:33:49.577 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2575e-05 2024-04-23T16:33:49.577 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.577 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.578 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.578 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:15.543227+0000", 2024-04-23T16:33:49.578 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.043703616000000001 2024-04-23T16:33:49.578 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.578 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.578 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.578 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:30:15.543271+0000", 2024-04-23T16:33:49.578 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3829000000000003e-05 2024-04-23T16:33:49.578 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.578 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.578 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.578 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.578 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.579 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:118766 5.9 5:976ced79:::benchmark_data_smithi188_113957_object118765:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e380)", 2024-04-23T16:33:49.579 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:32:29.428856+0000", 2024-04-23T16:33:49.579 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 80.098796027000006, 2024-04-23T16:33:49.579 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.088864259000000001, 2024-04-23T16:33:49.579 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.579 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.579 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.579 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:49.579 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:33:49.579 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:49.579 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 118766 2024-04-23T16:33:49.579 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.579 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.579 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.580 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.580 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.428856+0000", 2024-04-23T16:33:49.580 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.580 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.580 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.580 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.580 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.428856+0000", 2024-04-23T16:33:49.580 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.580 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.580 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.580 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.580 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.428859+0000", 2024-04-23T16:33:49.580 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7389999999999999e-06 2024-04-23T16:33:49.580 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.581 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.581 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.581 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.428919+0000", 2024-04-23T16:33:49.581 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0698000000000001e-05 2024-04-23T16:33:49.581 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.581 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.581 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.581 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.428937+0000", 2024-04-23T16:33:49.581 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7419999999999999e-05 2024-04-23T16:33:49.581 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.581 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.581 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.581 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.428947+0000", 2024-04-23T16:33:49.581 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.06e-05 2024-04-23T16:33:49.582 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.582 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.582 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.582 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.429008+0000", 2024-04-23T16:33:49.582 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0671000000000002e-05 2024-04-23T16:33:49.582 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.582 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.582 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.582 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.429157+0000", 2024-04-23T16:33:49.582 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014930299999999999 2024-04-23T16:33:49.582 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.582 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.582 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:33:49.583 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.429312+0000", 2024-04-23T16:33:49.583 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015511 2024-04-23T16:33:49.583 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.583 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.583 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:49.583 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.442243+0000", 2024-04-23T16:33:49.583 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012930222999999999 2024-04-23T16:33:49.583 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.583 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.583 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:49.583 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.517447+0000", 2024-04-23T16:33:49.583 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.075204328000000001 2024-04-23T16:33:49.583 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.583 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.584 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.584 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.517562+0000", 2024-04-23T16:33:49.584 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011535 2024-04-23T16:33:49.584 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.584 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.584 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.584 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.517720+0000", 2024-04-23T16:33:49.584 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015781700000000001 2024-04-23T16:33:49.584 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.584 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.584 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.584 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.584 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.584 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:118776 5.9 5:9b6ef62e:::benchmark_data_smithi188_113957_object118775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e380)", 2024-04-23T16:33:49.585 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:32:29.434471+0000", 2024-04-23T16:33:49.585 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 80.093180427999997, 2024-04-23T16:33:49.585 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.083390449000000005, 2024-04-23T16:33:49.585 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.585 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.585 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.585 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:49.585 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:33:49.585 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:49.585 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 118776 2024-04-23T16:33:49.585 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.585 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.585 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.585 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.586 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.434471+0000", 2024-04-23T16:33:49.586 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.586 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.586 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.586 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.586 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.434471+0000", 2024-04-23T16:33:49.586 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.586 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.586 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.586 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.586 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.434474+0000", 2024-04-23T16:33:49.586 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6019999999999998e-06 2024-04-23T16:33:49.586 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.586 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.587 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.587 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.434504+0000", 2024-04-23T16:33:49.587 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0295e-05 2024-04-23T16:33:49.587 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.587 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.587 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.587 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.434523+0000", 2024-04-23T16:33:49.587 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8893e-05 2024-04-23T16:33:49.587 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.587 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.587 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.587 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.434536+0000", 2024-04-23T16:33:49.587 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2446000000000001e-05 2024-04-23T16:33:49.588 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.588 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.588 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.588 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.434588+0000", 2024-04-23T16:33:49.588 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2048e-05 2024-04-23T16:33:49.588 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.588 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.588 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.588 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.434707+0000", 2024-04-23T16:33:49.588 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000119232 2024-04-23T16:33:49.588 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.588 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.588 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:33:49.588 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.434833+0000", 2024-04-23T16:33:49.589 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012571699999999999 2024-04-23T16:33:49.589 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.589 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.589 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:49.589 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.450207+0000", 2024-04-23T16:33:49.589 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015374820000000001 2024-04-23T16:33:49.589 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.589 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.589 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:49.589 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.517730+0000", 2024-04-23T16:33:49.589 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.067522896999999998 2024-04-23T16:33:49.589 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.589 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.590 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.590 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.517822+0000", 2024-04-23T16:33:49.590 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1748000000000003e-05 2024-04-23T16:33:49.590 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.590 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.590 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.590 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.517862+0000", 2024-04-23T16:33:49.590 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9750999999999999e-05 2024-04-23T16:33:49.590 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.590 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.590 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.590 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.590 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.590 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:118779 5.9 5:9fbf6f29:::benchmark_data_smithi188_113957_object118778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e380)", 2024-04-23T16:33:49.591 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:32:29.438485+0000", 2024-04-23T16:33:49.591 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 80.089167023000002, 2024-04-23T16:33:49.591 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.079463770000000003, 2024-04-23T16:33:49.591 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.591 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.591 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.591 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:49.591 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:33:49.591 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:49.591 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 118779 2024-04-23T16:33:49.591 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.591 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.591 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.592 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.592 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.438485+0000", 2024-04-23T16:33:49.592 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.592 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.592 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.592 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.592 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.438485+0000", 2024-04-23T16:33:49.592 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.592 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.592 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.592 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.592 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.438487+0000", 2024-04-23T16:33:49.592 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4480000000000001e-06 2024-04-23T16:33:49.592 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.593 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.593 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.593 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.438521+0000", 2024-04-23T16:33:49.593 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3562999999999997e-05 2024-04-23T16:33:49.593 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.593 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.593 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.593 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.438531+0000", 2024-04-23T16:33:49.593 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0175e-05 2024-04-23T16:33:49.593 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.593 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.593 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.593 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.438537+0000", 2024-04-23T16:33:49.593 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5119999999999997e-06 2024-04-23T16:33:49.593 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.594 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.594 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.594 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.438587+0000", 2024-04-23T16:33:49.594 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9382000000000003e-05 2024-04-23T16:33:49.594 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.594 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.594 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.594 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.438727+0000", 2024-04-23T16:33:49.594 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014028599999999999 2024-04-23T16:33:49.594 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.594 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.594 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:33:49.594 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.438873+0000", 2024-04-23T16:33:49.595 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000146236 2024-04-23T16:33:49.595 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.595 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.595 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:49.595 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.450283+0000", 2024-04-23T16:33:49.595 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011409913000000001 2024-04-23T16:33:49.595 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.595 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.595 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:49.595 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.517870+0000", 2024-04-23T16:33:49.595 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.067587003000000007 2024-04-23T16:33:49.595 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.595 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.595 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.596 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.517917+0000", 2024-04-23T16:33:49.596 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6499999999999999e-05 2024-04-23T16:33:49.596 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.596 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.596 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.596 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.517949+0000", 2024-04-23T16:33:49.596 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1751999999999997e-05 2024-04-23T16:33:49.596 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.596 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.596 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.596 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.596 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.596 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:118795 5.9 5:9a4b7f7e:::benchmark_data_smithi188_113957_object118794:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e380)", 2024-04-23T16:33:49.597 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:32:29.450772+0000", 2024-04-23T16:33:49.597 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 80.076879395999995, 2024-04-23T16:33:49.597 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.067269952999999993, 2024-04-23T16:33:49.597 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.597 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.597 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.597 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:49.597 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:33:49.597 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:49.597 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 118795 2024-04-23T16:33:49.597 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.597 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.597 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.597 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.597 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.450772+0000", 2024-04-23T16:33:49.598 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.598 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.598 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.598 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.598 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.450772+0000", 2024-04-23T16:33:49.598 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.598 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.598 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.598 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.598 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.450775+0000", 2024-04-23T16:33:49.598 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5550000000000001e-06 2024-04-23T16:33:49.598 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.598 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.599 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.599 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.450808+0000", 2024-04-23T16:33:49.599 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2657999999999998e-05 2024-04-23T16:33:49.599 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.599 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.599 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.599 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.450822+0000", 2024-04-23T16:33:49.599 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4857e-05 2024-04-23T16:33:49.599 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.599 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.599 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.599 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.450832+0000", 2024-04-23T16:33:49.599 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2359999999999993e-06 2024-04-23T16:33:49.599 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.600 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.600 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.600 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.450884+0000", 2024-04-23T16:33:49.600 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2778999999999997e-05 2024-04-23T16:33:49.600 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.600 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.600 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.600 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.451033+0000", 2024-04-23T16:33:49.600 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014821600000000001 2024-04-23T16:33:49.600 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.600 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.600 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:33:49.600 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.451186+0000", 2024-04-23T16:33:49.601 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000153306 2024-04-23T16:33:49.601 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.601 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.601 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:49.601 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.458786+0000", 2024-04-23T16:33:49.601 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0075997310000000002 2024-04-23T16:33:49.601 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.601 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.601 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:49.601 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.517956+0000", 2024-04-23T16:33:49.601 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.059170134999999999 2024-04-23T16:33:49.601 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.601 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.601 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.601 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.517999+0000", 2024-04-23T16:33:49.602 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3491000000000001e-05 2024-04-23T16:33:49.602 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.602 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.602 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.602 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.518042+0000", 2024-04-23T16:33:49.602 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2988999999999998e-05 2024-04-23T16:33:49.602 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.602 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.602 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.602 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.602 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.602 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:118812 5.9 5:982b9557:::benchmark_data_smithi188_113957_object118811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e380)", 2024-04-23T16:33:49.602 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:32:29.464169+0000", 2024-04-23T16:33:49.603 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 80.063482449999995, 2024-04-23T16:33:49.603 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.053935014000000003, 2024-04-23T16:33:49.603 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.603 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.603 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.603 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:49.603 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:33:49.603 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:49.603 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 118812 2024-04-23T16:33:49.603 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.603 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.603 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.603 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.603 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.464169+0000", 2024-04-23T16:33:49.604 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.604 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.604 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.604 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.604 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.464169+0000", 2024-04-23T16:33:49.604 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.604 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.604 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.604 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.604 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.464173+0000", 2024-04-23T16:33:49.604 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.574e-06 2024-04-23T16:33:49.604 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.604 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.605 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.605 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.464213+0000", 2024-04-23T16:33:49.605 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0361999999999997e-05 2024-04-23T16:33:49.605 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.605 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.605 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.605 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.464234+0000", 2024-04-23T16:33:49.605 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0231999999999999e-05 2024-04-23T16:33:49.605 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.605 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.605 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.605 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.464244+0000", 2024-04-23T16:33:49.605 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0893000000000001e-05 2024-04-23T16:33:49.605 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.606 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.606 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.606 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.464308+0000", 2024-04-23T16:33:49.606 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3245999999999997e-05 2024-04-23T16:33:49.606 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.606 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.606 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.606 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.464454+0000", 2024-04-23T16:33:49.606 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014671700000000001 2024-04-23T16:33:49.606 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.606 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.606 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:33:49.606 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.464593+0000", 2024-04-23T16:33:49.607 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013909999999999999 2024-04-23T16:33:49.607 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.607 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.607 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:49.607 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.471232+0000", 2024-04-23T16:33:49.607 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0066382840000000004 2024-04-23T16:33:49.607 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.607 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.607 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:49.607 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.518050+0000", 2024-04-23T16:33:49.607 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.046818729000000003 2024-04-23T16:33:49.607 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.607 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.607 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.608 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.518080+0000", 2024-04-23T16:33:49.608 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9805999999999999e-05 2024-04-23T16:33:49.608 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.608 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.608 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.608 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.518104+0000", 2024-04-23T16:33:49.608 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4071e-05 2024-04-23T16:33:49.608 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.608 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.608 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.608 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.608 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.608 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:118816 5.9 5:90084b05:::benchmark_data_smithi188_113957_object118815:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e380)", 2024-04-23T16:33:49.609 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:32:29.472239+0000", 2024-04-23T16:33:49.609 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 80.055412887000003, 2024-04-23T16:33:49.609 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.045920138999999999, 2024-04-23T16:33:49.609 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.609 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.609 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.609 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:49.609 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:33:49.609 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:49.609 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 118816 2024-04-23T16:33:49.609 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.609 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.609 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.609 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.609 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.472239+0000", 2024-04-23T16:33:49.610 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.610 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.610 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.610 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.610 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.472239+0000", 2024-04-23T16:33:49.610 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.610 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.610 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.610 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.610 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.472241+0000", 2024-04-23T16:33:49.610 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2979999999999999e-06 2024-04-23T16:33:49.610 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.610 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.611 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.611 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.472302+0000", 2024-04-23T16:33:49.611 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0647999999999999e-05 2024-04-23T16:33:49.611 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.611 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.611 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.611 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.472316+0000", 2024-04-23T16:33:49.611 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4100999999999999e-05 2024-04-23T16:33:49.611 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.611 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.611 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.611 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.472324+0000", 2024-04-23T16:33:49.611 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4060000000000005e-06 2024-04-23T16:33:49.611 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.612 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.612 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.612 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.472373+0000", 2024-04-23T16:33:49.612 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9008999999999998e-05 2024-04-23T16:33:49.612 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.612 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.612 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.612 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.472522+0000", 2024-04-23T16:33:49.612 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014837499999999999 2024-04-23T16:33:49.612 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.612 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.612 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:33:49.612 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.472615+0000", 2024-04-23T16:33:49.612 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3393000000000005e-05 2024-04-23T16:33:49.613 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.613 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.613 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:49.613 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.483244+0000", 2024-04-23T16:33:49.613 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010628449 2024-04-23T16:33:49.613 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.613 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.613 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:49.613 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.518109+0000", 2024-04-23T16:33:49.613 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.034865303 2024-04-23T16:33:49.613 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.613 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.613 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.614 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.518137+0000", 2024-04-23T16:33:49.614 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7665e-05 2024-04-23T16:33:49.614 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.614 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.614 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.614 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:29.518159+0000", 2024-04-23T16:33:49.614 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2492000000000001e-05 2024-04-23T16:33:49.614 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.614 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.614 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.614 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.614 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.614 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:128353 5.9 5:980a48e6:::benchmark_data_smithi188_113957_object128352:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e381)", 2024-04-23T16:33:49.614 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:32:36.250184+0000", 2024-04-23T16:33:49.615 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 73.277467569999999, 2024-04-23T16:33:49.615 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.044751312000000001, 2024-04-23T16:33:49.615 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.615 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.615 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.615 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:33:49.615 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:33:49.615 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:49.615 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 128353 2024-04-23T16:33:49.615 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.615 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.615 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.615 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.615 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:36.250184+0000", 2024-04-23T16:33:49.616 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.616 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.616 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.616 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.616 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:36.250184+0000", 2024-04-23T16:33:49.616 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.616 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.616 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.616 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.616 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:36.250186+0000", 2024-04-23T16:33:49.616 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.951e-06 2024-04-23T16:33:49.616 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.616 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.616 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.617 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:36.250215+0000", 2024-04-23T16:33:49.617 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8612e-05 2024-04-23T16:33:49.617 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.617 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.617 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.617 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:36.250224+0000", 2024-04-23T16:33:49.617 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3279999999999995e-06 2024-04-23T16:33:49.617 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.617 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.617 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.617 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:36.250232+0000", 2024-04-23T16:33:49.617 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.157e-06 2024-04-23T16:33:49.617 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.618 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.618 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.618 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:36.251221+0000", 2024-04-23T16:33:49.618 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00098907899999999991 2024-04-23T16:33:49.618 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.618 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.618 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.618 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:36.251326+0000", 2024-04-23T16:33:49.618 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000105091 2024-04-23T16:33:49.618 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.618 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.618 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:33:49.618 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:36.251425+0000", 2024-04-23T16:33:49.618 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8639000000000002e-05 2024-04-23T16:33:49.619 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.619 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.619 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:49.619 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:36.274037+0000", 2024-04-23T16:33:49.619 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022612303 2024-04-23T16:33:49.619 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.619 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.619 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:33:49.619 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:36.294795+0000", 2024-04-23T16:33:49.619 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020757701999999999 2024-04-23T16:33:49.619 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.619 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.619 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.620 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:36.294859+0000", 2024-04-23T16:33:49.620 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3742000000000001e-05 2024-04-23T16:33:49.620 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.620 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.620 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.620 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:36.294936+0000", 2024-04-23T16:33:49.620 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6707999999999998e-05 2024-04-23T16:33:49.620 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.620 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.620 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.620 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.620 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.620 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:157326 5.6 e395/394 5:6843a85e:::benchmark_data_smithi188_113957_object3728:head v 395'9944, mlcod=366'254)", 2024-04-23T16:33:49.620 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:33:33.204216+0000", 2024-04-23T16:33:49.621 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 16.323435574000001, 2024-04-23T16:33:49.621 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.043891675999999998, 2024-04-23T16:33:49.621 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.621 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.621 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.621 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.621 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.621 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.621 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:33.204216+0000", 2024-04-23T16:33:49.621 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.621 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.621 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.621 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.621 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:33.204216+0000", 2024-04-23T16:33:49.622 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.622 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.622 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.622 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.622 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:33.204217+0000", 2024-04-23T16:33:49.622 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.257e-06 2024-04-23T16:33:49.622 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.622 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.622 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.622 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:33.204225+0000", 2024-04-23T16:33:49.622 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7430000000000002e-06 2024-04-23T16:33:49.622 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.622 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.622 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.623 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:33.204237+0000", 2024-04-23T16:33:49.623 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2052000000000001e-05 2024-04-23T16:33:49.623 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.623 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.623 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.623 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:33.204244+0000", 2024-04-23T16:33:49.623 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8669999999999996e-06 2024-04-23T16:33:49.623 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.623 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.623 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.623 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:33.204330+0000", 2024-04-23T16:33:49.623 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5735000000000005e-05 2024-04-23T16:33:49.623 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.623 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.624 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.624 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:33.204362+0000", 2024-04-23T16:33:49.624 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2024999999999999e-05 2024-04-23T16:33:49.624 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.624 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.624 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.624 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:33.248054+0000", 2024-04-23T16:33:49.624 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.043692470999999997 2024-04-23T16:33:49.624 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.624 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.624 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.624 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:33.248108+0000", 2024-04-23T16:33:49.624 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3526000000000002e-05 2024-04-23T16:33:49.624 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.624 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.625 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.625 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.625 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.625 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:159094 5.6 e396/394 5:63fc129e:::benchmark_data_smithi188_113957_object5496:head v 396'10061, mlcod=366'368)", 2024-04-23T16:33:49.625 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:33:36.124513+0000", 2024-04-23T16:33:49.625 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 13.403139097, 2024-04-23T16:33:49.625 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.062629139, 2024-04-23T16:33:49.625 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.625 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.625 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.625 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.625 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.625 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.625 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.124513+0000", 2024-04-23T16:33:49.625 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.626 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.626 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.626 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.626 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.124513+0000", 2024-04-23T16:33:49.626 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.626 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.626 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.626 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.626 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.124514+0000", 2024-04-23T16:33:49.626 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1519999999999999e-06 2024-04-23T16:33:49.626 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.626 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.626 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.627 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.124522+0000", 2024-04-23T16:33:49.627 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.9889999999999992e-06 2024-04-23T16:33:49.627 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.627 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.627 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.627 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.124529+0000", 2024-04-23T16:33:49.627 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4939999999999997e-06 2024-04-23T16:33:49.627 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.627 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.627 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.627 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.124538+0000", 2024-04-23T16:33:49.627 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2409999999999995e-06 2024-04-23T16:33:49.627 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.627 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.628 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.628 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.137014+0000", 2024-04-23T16:33:49.628 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012476325999999999 2024-04-23T16:33:49.628 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.628 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.628 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.628 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.137046+0000", 2024-04-23T16:33:49.628 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1977000000000002e-05 2024-04-23T16:33:49.628 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.628 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.628 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.628 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.187102+0000", 2024-04-23T16:33:49.628 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.050056366999999997 2024-04-23T16:33:49.628 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.629 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.629 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.629 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.187142+0000", 2024-04-23T16:33:49.629 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9592999999999999e-05 2024-04-23T16:33:49.629 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.629 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.629 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.629 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.629 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.629 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:159121 5.6 e396/394 5:6a957fe7:::benchmark_data_smithi188_113957_object5523:head v 396'10062, mlcod=366'372)", 2024-04-23T16:33:49.629 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:33:36.139156+0000", 2024-04-23T16:33:49.629 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 13.388495597, 2024-04-23T16:33:49.629 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048008928999999999, 2024-04-23T16:33:49.629 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.630 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.630 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.630 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.630 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.630 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.630 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.139156+0000", 2024-04-23T16:33:49.630 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.630 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.630 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.630 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.630 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.139156+0000", 2024-04-23T16:33:49.630 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.630 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.631 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.631 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.631 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.139158+0000", 2024-04-23T16:33:49.631 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6670000000000001e-06 2024-04-23T16:33:49.631 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.631 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.631 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.631 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.139169+0000", 2024-04-23T16:33:49.631 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1343e-05 2024-04-23T16:33:49.631 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.631 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.631 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.631 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.139181+0000", 2024-04-23T16:33:49.631 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1572e-05 2024-04-23T16:33:49.632 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.632 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.632 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.632 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.139189+0000", 2024-04-23T16:33:49.632 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1110000000000008e-06 2024-04-23T16:33:49.632 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.632 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.632 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.632 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.148856+0000", 2024-04-23T16:33:49.632 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0096675519999999994 2024-04-23T16:33:49.632 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.632 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.632 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.632 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.148886+0000", 2024-04-23T16:33:49.633 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9521000000000001e-05 2024-04-23T16:33:49.633 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.633 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.633 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.633 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.187151+0000", 2024-04-23T16:33:49.633 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.038264738999999999 2024-04-23T16:33:49.633 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.633 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.633 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.633 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.187165+0000", 2024-04-23T16:33:49.633 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4423999999999999e-05 2024-04-23T16:33:49.633 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.633 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.634 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.634 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.634 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.634 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:159166 5.6 e396/394 5:6b32d413:::benchmark_data_smithi188_113957_object5568:head v 396'10066, mlcod=366'372)", 2024-04-23T16:33:49.634 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:33:36.242567+0000", 2024-04-23T16:33:49.634 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 13.285085168, 2024-04-23T16:33:49.634 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.054959486000000002, 2024-04-23T16:33:49.634 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.634 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.634 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.634 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.634 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.634 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.634 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.242567+0000", 2024-04-23T16:33:49.635 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.635 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.635 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.635 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.635 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.242567+0000", 2024-04-23T16:33:49.635 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.635 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.635 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.635 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.635 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.242567+0000", 2024-04-23T16:33:49.635 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4099999999999999e-07 2024-04-23T16:33:49.635 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.635 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.635 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.636 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.242572+0000", 2024-04-23T16:33:49.636 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8999999999999997e-06 2024-04-23T16:33:49.636 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.636 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.636 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.636 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.242577+0000", 2024-04-23T16:33:49.636 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1800000000000004e-06 2024-04-23T16:33:49.636 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.636 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.636 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.636 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.242582+0000", 2024-04-23T16:33:49.636 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6979999999999997e-06 2024-04-23T16:33:49.636 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.636 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.637 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.637 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.253123+0000", 2024-04-23T16:33:49.637 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010540739 2024-04-23T16:33:49.637 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.637 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.637 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.637 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.253148+0000", 2024-04-23T16:33:49.637 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5284999999999999e-05 2024-04-23T16:33:49.637 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.637 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.637 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.637 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.297220+0000", 2024-04-23T16:33:49.637 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.044072116000000001 2024-04-23T16:33:49.637 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.638 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.638 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.638 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.297526+0000", 2024-04-23T16:33:49.638 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000306027 2024-04-23T16:33:49.638 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.638 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.638 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.638 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.638 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.638 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:159328 5.6 e396/394 5:64270dec:::benchmark_data_smithi188_113957_object5730:head v 396'10077, mlcod=366'387)", 2024-04-23T16:33:49.638 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:33:36.522149+0000", 2024-04-23T16:33:49.638 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 13.005502551999999, 2024-04-23T16:33:49.638 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.060306527999999998, 2024-04-23T16:33:49.638 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.639 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.639 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.639 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.639 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.639 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.639 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.522149+0000", 2024-04-23T16:33:49.639 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.639 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.639 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.639 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.639 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.522149+0000", 2024-04-23T16:33:49.639 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.639 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.639 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.640 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.640 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.522151+0000", 2024-04-23T16:33:49.640 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.934e-06 2024-04-23T16:33:49.640 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.640 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.640 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.640 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.522165+0000", 2024-04-23T16:33:49.640 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4272e-05 2024-04-23T16:33:49.640 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.640 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.640 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.640 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.522174+0000", 2024-04-23T16:33:49.640 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7700000000000007e-06 2024-04-23T16:33:49.640 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.640 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.641 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.641 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.522182+0000", 2024-04-23T16:33:49.641 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8639999999999996e-06 2024-04-23T16:33:49.641 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.641 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.641 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.641 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.529750+0000", 2024-04-23T16:33:49.641 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0075678129999999996 2024-04-23T16:33:49.641 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.641 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.641 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.641 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.529776+0000", 2024-04-23T16:33:49.641 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6157e-05 2024-04-23T16:33:49.642 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.642 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.642 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.642 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.582387+0000", 2024-04-23T16:33:49.642 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052611316999999998 2024-04-23T16:33:49.642 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.642 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.642 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.642 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.582456+0000", 2024-04-23T16:33:49.642 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8400999999999998e-05 2024-04-23T16:33:49.642 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.642 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.642 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.643 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.643 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.643 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:159346 5.6 e396/394 5:69e117c1:::benchmark_data_smithi188_113957_object5748:head v 396'10078, mlcod=366'387)", 2024-04-23T16:33:49.643 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:33:36.526240+0000", 2024-04-23T16:33:49.643 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 13.001412237, 2024-04-23T16:33:49.643 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.068880489000000003, 2024-04-23T16:33:49.643 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.643 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.643 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.643 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.643 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.643 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.643 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.526240+0000", 2024-04-23T16:33:49.643 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.644 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.644 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.644 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.644 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.526240+0000", 2024-04-23T16:33:49.644 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.644 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.644 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.644 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.644 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.526242+0000", 2024-04-23T16:33:49.644 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6110000000000001e-06 2024-04-23T16:33:49.644 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.644 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.644 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.644 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.526256+0000", 2024-04-23T16:33:49.645 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3522e-05 2024-04-23T16:33:49.645 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.645 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.645 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.645 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.526270+0000", 2024-04-23T16:33:49.645 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4219999999999999e-05 2024-04-23T16:33:49.645 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.645 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.645 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.645 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.526281+0000", 2024-04-23T16:33:49.645 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1114999999999999e-05 2024-04-23T16:33:49.645 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.645 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.645 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.646 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.542611+0000", 2024-04-23T16:33:49.646 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016330112000000001 2024-04-23T16:33:49.646 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.646 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.646 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.646 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.542640+0000", 2024-04-23T16:33:49.646 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8790999999999999e-05 2024-04-23T16:33:49.646 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.646 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.646 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.646 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.595065+0000", 2024-04-23T16:33:49.646 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052424962999999998 2024-04-23T16:33:49.646 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.646 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.647 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.647 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:36.595120+0000", 2024-04-23T16:33:49.647 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5155000000000003e-05 2024-04-23T16:33:49.647 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.647 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.647 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.647 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.647 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.647 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:160000 5.6 e396/394 5:6d73a8a9:::benchmark_data_smithi188_113957_object6402:head v 396'10123, mlcod=366'429)", 2024-04-23T16:33:49.647 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:33:37.568914+0000", 2024-04-23T16:33:49.647 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 11.958737639000001, 2024-04-23T16:33:49.647 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.047431846, 2024-04-23T16:33:49.647 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:33:49.647 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:33:49.648 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:49.648 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:33:49.648 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.648 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:33:49.648 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:37.568914+0000", 2024-04-23T16:33:49.648 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.648 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.648 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.648 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:33:49.648 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:37.568914+0000", 2024-04-23T16:33:49.648 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:33:49.648 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.648 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.649 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:33:49.649 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:37.568923+0000", 2024-04-23T16:33:49.649 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4740000000000005e-06 2024-04-23T16:33:49.649 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.649 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.649 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:33:49.649 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:37.568935+0000", 2024-04-23T16:33:49.649 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2481e-05 2024-04-23T16:33:49.649 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.649 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.649 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:33:49.649 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:37.568953+0000", 2024-04-23T16:33:49.649 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8e-05 2024-04-23T16:33:49.649 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.650 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.650 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:33:49.650 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:37.568963+0000", 2024-04-23T16:33:49.650 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6090000000000007e-06 2024-04-23T16:33:49.650 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.650 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.650 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:33:49.650 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:37.599817+0000", 2024-04-23T16:33:49.650 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.030854056000000001 2024-04-23T16:33:49.650 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.650 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.650 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:33:49.650 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:37.599841+0000", 2024-04-23T16:33:49.650 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4363999999999999e-05 2024-04-23T16:33:49.651 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.651 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.651 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:33:49.651 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:37.616314+0000", 2024-04-23T16:33:49.651 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016473235999999999 2024-04-23T16:33:49.651 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:33:49.651 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:33:49.651 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:33:49.651 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:33:37.616346+0000", 2024-04-23T16:33:49.651 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1625999999999998e-05 2024-04-23T16:33:49.651 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.651 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.651 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.651 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:33:49.652 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:33:49.652 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:33:49.652 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:33:49.653 INFO:teuthology.orchestra.run.smithi103.stdout:finish_remove_pgs 5.a_head removing 5.a 2024-04-23T16:33:49.653 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:33:49.654 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:33:49.680 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:49.681 INFO:teuthology.orchestra.run.smithi103.stdout:Remove successful 2024-04-23T16:33:49.703 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:33:49.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:49.706 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:33:49.807 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:49.846 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:49 smithi148 ceph-mon[44520]: pgmap v1126: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 454 op/s; 26897/285606 objects degraded (9.418%); 5.7 MiB/s, 91 objects/s recovering 2024-04-23T16:33:49.846 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:49 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 26897/285606 objects degraded (9.418%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:49.848 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:49 smithi188 ceph-mon[44101]: pgmap v1126: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 454 op/s; 26897/285606 objects degraded (9.418%); 5.7 MiB/s, 91 objects/s recovering 2024-04-23T16:33:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:49 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 26897/285606 objects degraded (9.418%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:50.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:49 smithi103 ceph-mon[39629]: pgmap v1126: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 454 op/s; 26897/285606 objects degraded (9.418%); 5.7 MiB/s, 91 objects/s recovering 2024-04-23T16:33:50.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:49 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 26897/285606 objects degraded (9.418%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:50.327 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:50.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:50.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:50.360 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-23T16:33:50.442 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:33:50.481 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:50.510 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:33:50.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:50.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:33:50.618 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:50.682 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:33:50.687 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:50.737 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:50.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:50.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:50.846 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:33:50.874 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:51.223 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:33:51.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:51.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:33:51.249 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.2 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.5.a.2 2024-04-23T16:33:51.328 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:51.385 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:51.418 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:51.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:51.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:51.523 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:51.570 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.2/config 2024-04-23T16:33:51.574 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:51.608 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:51.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:51.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:51.655 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:33:51.715 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:33:51.750 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:51.796 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:51 smithi148 ceph-mon[44520]: pgmap v1127: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.6 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 662 op/s; 26477/280582 objects degraded (9.436%); 8.3 MiB/s, 132 objects/s recovering 2024-04-23T16:33:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:51 smithi103 ceph-mon[39629]: pgmap v1127: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.6 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 662 op/s; 26477/280582 objects degraded (9.436%); 8.3 MiB/s, 132 objects/s recovering 2024-04-23T16:33:51.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:51 smithi188 ceph-mon[44101]: pgmap v1127: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.6 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 662 op/s; 26477/280582 objects degraded (9.436%); 8.3 MiB/s, 132 objects/s recovering 2024-04-23T16:33:52.201 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:33:52.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:52.207 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:33:52.308 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:33:52.345 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:52.473 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:33:52.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:52.476 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:33:52.577 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:33:52.609 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:52.634 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:33:52.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:52.636 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:33:52.737 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:52.781 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:52.800 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:33:52.801 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:33:52.801 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:52.801 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:169447 5.a 5:57d08a97:::benchmark_data_smithi188_113957_object15849:head [delete] snapc 0=[] ondisk+write+known_if_redirected e399)", 2024-04-23T16:33:52.801 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:52.776614+0000", 2024-04-23T16:33:52.801 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.023176163, 2024-04-23T16:33:52.801 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023200486999999999, 2024-04-23T16:33:52.801 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:52.801 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:52.801 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:33:52.801 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:52.801 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:52.801 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:52.801 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 169447 2024-04-23T16:33:52.801 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:52.802 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:52.802 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:52.802 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:52.802 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.776614+0000", 2024-04-23T16:33:52.802 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:52.802 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:52.802 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:52.802 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:52.802 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.776614+0000", 2024-04-23T16:33:52.802 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:52.802 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:52.802 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:52.802 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:52.802 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.776616+0000", 2024-04-23T16:33:52.803 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5160000000000001e-06 2024-04-23T16:33:52.803 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:52.803 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:52.803 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:52.803 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.776624+0000", 2024-04-23T16:33:52.803 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6229999999999997e-06 2024-04-23T16:33:52.803 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:52.803 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:52.803 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:52.803 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.776634+0000", 2024-04-23T16:33:52.803 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0740999999999999e-05 2024-04-23T16:33:52.803 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:52.803 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:52.803 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:52.804 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.776643+0000", 2024-04-23T16:33:52.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7450000000000001e-06 2024-04-23T16:33:52.804 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:52.804 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:52.804 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:52.804 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.782512+0000", 2024-04-23T16:33:52.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0058683809999999998 2024-04-23T16:33:52.804 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:52.804 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:52.804 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:52.804 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.782595+0000", 2024-04-23T16:33:52.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3379000000000002e-05 2024-04-23T16:33:52.804 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:52.804 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:52.805 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:52.805 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.788202+0000", 2024-04-23T16:33:52.805 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0056067579999999999 2024-04-23T16:33:52.805 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:52.805 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:52.805 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:52.805 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.788316+0000", 2024-04-23T16:33:52.805 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000114515 2024-04-23T16:33:52.806 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:52.806 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:52.806 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:52.806 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.788499+0000", 2024-04-23T16:33:52.806 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018275100000000001 2024-04-23T16:33:52.806 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:52.806 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:52.806 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:52.806 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:52.806 INFO:teuthology.orchestra.run.smithi188.stdout: ], 2024-04-23T16:33:52.806 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 1 2024-04-23T16:33:52.807 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:33:52.810 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:33:52.814 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:52.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:52.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:52.925 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:52.980 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:53.008 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:53.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:53.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:53.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:52 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/161137909' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:33:53.115 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:52 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/161137909' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:33:53.180 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:53.200 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:52 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/161137909' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:33:53.210 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:53.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:53.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:53.275 INFO:teuthology.orchestra.run.smithi103.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:33:53.315 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:33:53.345 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:53.550 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:33:53.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:53.557 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:33:53.562 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-23T16:33:53.658 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:53.723 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:53.756 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:53.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:53.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:53.864 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:33:53.898 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:53.930 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:33:53.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:53.937 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:33:53.939 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:33:53.985 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:53 smithi103 ceph-mon[39629]: pgmap v1128: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 614 op/s; 26477/279782 objects degraded (9.463%); 7.2 MiB/s, 114 objects/s recovering 2024-04-23T16:33:53.986 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:53 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:33:53.986 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:53 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/161137909' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:33:53.986 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:53 smithi103 ceph-mon[39629]: osdmap e400: 12 total, 11 up, 9 in 2024-04-23T16:33:54.038 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:33:54.068 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:54.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:53 smithi148 ceph-mon[44520]: pgmap v1128: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 614 op/s; 26477/279782 objects degraded (9.463%); 7.2 MiB/s, 114 objects/s recovering 2024-04-23T16:33:54.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:53 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:33:54.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:53 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/161137909' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:33:54.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:53 smithi148 ceph-mon[44520]: osdmap e400: 12 total, 11 up, 9 in 2024-04-23T16:33:54.216 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:53 smithi188 ceph-mon[44101]: pgmap v1128: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 614 op/s; 26477/279782 objects degraded (9.463%); 7.2 MiB/s, 114 objects/s recovering 2024-04-23T16:33:54.216 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:53 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:33:54.216 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:53 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/161137909' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:33:54.216 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:53 smithi188 ceph-mon[44101]: osdmap e400: 12 total, 11 up, 9 in 2024-04-23T16:33:54.294 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:33:54.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:54.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:33:54.400 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:33:54.429 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:54.450 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:33:54.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:54.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:33:54.556 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:54.603 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:54.626 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:33:54.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:54.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:33:54.687 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:33:54.729 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:54.781 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:54.791 INFO:teuthology.orchestra.run.smithi103.stdout:get_pg_num_history pg_num_history pg_num_history(e386 pg_nums {1={301=11},4={274=16},5={359=16}} deleted_pools 356,4) 2024-04-23T16:33:54.791 INFO:teuthology.orchestra.run.smithi103.stdout:Importing pgid 5.a 2024-04-23T16:33:54.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5000aecc:::benchmark_data_smithi188_113957_object25214:head# 2024-04-23T16:33:54.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5000b1fb:::benchmark_data_smithi188_113957_object88294:head# 2024-04-23T16:33:54.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5000d1fa:::benchmark_data_smithi188_113957_object63134:head# 2024-04-23T16:33:54.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5001785e:::benchmark_data_smithi188_113957_object576:head# 2024-04-23T16:33:54.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5002d4fb:::benchmark_data_smithi188_113957_object110729:head# 2024-04-23T16:33:54.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5002d6c2:::benchmark_data_smithi188_113957_object108243:head# 2024-04-23T16:33:54.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5002e6ba:::benchmark_data_smithi188_113957_object137087:head# 2024-04-23T16:33:54.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5003e9da:::benchmark_data_smithi188_113957_object90852:head# 2024-04-23T16:33:54.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5003ea6b:::benchmark_data_smithi188_113957_object59076:head# 2024-04-23T16:33:54.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.803 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50049178:::benchmark_data_smithi188_113957_object39155:head# 2024-04-23T16:33:54.803 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50053d67:::benchmark_data_smithi188_113957_object83545:head# 2024-04-23T16:33:54.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50055a33:::benchmark_data_smithi188_113957_object142656:head# 2024-04-23T16:33:54.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5005c6fb:::benchmark_data_smithi188_113957_object140615:head# 2024-04-23T16:33:54.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5005db0d:::benchmark_data_smithi188_113957_object89814:head# 2024-04-23T16:33:54.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5006ec38:::benchmark_data_smithi188_113957_object73297:head# 2024-04-23T16:33:54.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5007914a:::benchmark_data_smithi188_113957_object24056:head# 2024-04-23T16:33:54.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50094b33:::benchmark_data_smithi188_113957_object43552:head# 2024-04-23T16:33:54.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50095dd8:::benchmark_data_smithi188_113957_object102563:head# 2024-04-23T16:33:54.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5009d793:::benchmark_data_smithi188_113957_object8521:head# 2024-04-23T16:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5009f36f:::benchmark_data_smithi188_113957_object120695:head# 2024-04-23T16:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:500a1e91:::benchmark_data_smithi188_113957_object112154:head# 2024-04-23T16:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:500a557c:::benchmark_data_smithi188_113957_object34865:head# 2024-04-23T16:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:500a75b7:::benchmark_data_smithi188_113957_object118329:head# 2024-04-23T16:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:500afe81:::benchmark_data_smithi188_113957_object78463:head# 2024-04-23T16:33:54.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:500b0565:::benchmark_data_smithi188_113957_object103091:head# 2024-04-23T16:33:54.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:500bffaf:::benchmark_data_smithi188_113957_object24049:head# 2024-04-23T16:33:54.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.817 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:500c54ba:::benchmark_data_smithi188_113957_object20722:head# 2024-04-23T16:33:54.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:500cb56d:::benchmark_data_smithi188_113957_object109029:head# 2024-04-23T16:33:54.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:500ce5a8:::benchmark_data_smithi188_113957_object26326:head# 2024-04-23T16:33:54.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.819 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:500d2af8:::benchmark_data_smithi188_113957_object100892:head# 2024-04-23T16:33:54.819 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.819 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:500d33b2:::benchmark_data_smithi188_113957_object118095:head# 2024-04-23T16:33:54.819 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.819 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:500e0df9:::benchmark_data_smithi188_113957_object4057:head# 2024-04-23T16:33:54.819 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:500e1b23:::benchmark_data_smithi188_113957_object35892:head# 2024-04-23T16:33:54.820 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:500ee6ac:::benchmark_data_smithi188_113957_object57714:head# 2024-04-23T16:33:54.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:500f467e:::benchmark_data_smithi188_113957_object121458:head# 2024-04-23T16:33:54.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50105912:::benchmark_data_smithi188_113957_object126891:head# 2024-04-23T16:33:54.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5011fbb6:::benchmark_data_smithi188_113957_object142365:head# 2024-04-23T16:33:54.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:501236a6:::benchmark_data_smithi188_113957_object3380:head# 2024-04-23T16:33:54.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:501301da:::benchmark_data_smithi188_113957_object44650:head# 2024-04-23T16:33:54.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50133de5:::benchmark_data_smithi188_113957_object64266:head# 2024-04-23T16:33:54.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5013e6fb:::benchmark_data_smithi188_113957_object91942:head# 2024-04-23T16:33:54.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.825 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50141b29:::benchmark_data_smithi188_113957_object144942:head# 2024-04-23T16:33:54.825 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.825 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50147160:::benchmark_data_smithi188_113957_object133034:head# 2024-04-23T16:33:54.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5014f597:::benchmark_data_smithi188_113957_object51409:head# 2024-04-23T16:33:54.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:501504bc:::benchmark_data_smithi188_113957_object125796:head# 2024-04-23T16:33:54.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50155661:::benchmark_data_smithi188_113957_object46012:head# 2024-04-23T16:33:54.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5015b3aa:::benchmark_data_smithi188_113957_object131397:head# 2024-04-23T16:33:54.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5015dac2:::benchmark_data_smithi188_113957_object112924:head# 2024-04-23T16:33:54.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50163301:::benchmark_data_smithi188_113957_object24122:head# 2024-04-23T16:33:54.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5016692b:::benchmark_data_smithi188_113957_object4333:head# 2024-04-23T16:33:54.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5016d351:::benchmark_data_smithi188_113957_object97877:head# 2024-04-23T16:33:54.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50170301:::benchmark_data_smithi188_113957_object767:head# 2024-04-23T16:33:54.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5017a617:::benchmark_data_smithi188_113957_object34256:head# 2024-04-23T16:33:54.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50180741:::benchmark_data_smithi188_113957_object57907:head# 2024-04-23T16:33:54.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5018538b:::benchmark_data_smithi188_113957_object56575:head# 2024-04-23T16:33:54.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50186012:::benchmark_data_smithi188_113957_object84046:head# 2024-04-23T16:33:54.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50192224:::benchmark_data_smithi188_113957_object99466:head# 2024-04-23T16:33:54.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5019a0b5:::benchmark_data_smithi188_113957_object118194:head# 2024-04-23T16:33:54.828 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.828 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5019abaa:::benchmark_data_smithi188_113957_object54246:head# 2024-04-23T16:33:54.828 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.828 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:501a6e23:::benchmark_data_smithi188_113957_object85200:head# 2024-04-23T16:33:54.828 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:501a8ca8:::benchmark_data_smithi188_113957_object52461:head# 2024-04-23T16:33:54.829 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:501aee86:::benchmark_data_smithi188_113957_object140340:head# 2024-04-23T16:33:54.829 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:501c9c38:::benchmark_data_smithi188_113957_object150006:head# 2024-04-23T16:33:54.829 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:501ca56c:::benchmark_data_smithi188_113957_object113692:head# 2024-04-23T16:33:54.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:501cfd48:::benchmark_data_smithi188_113957_object16928:head# 2024-04-23T16:33:54.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:501d9b78:::benchmark_data_smithi188_113957_object141793:head# 2024-04-23T16:33:54.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:501e52c0:::benchmark_data_smithi188_113957_object120183:head# 2024-04-23T16:33:54.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:501ebbac:::benchmark_data_smithi188_113957_object70479:head# 2024-04-23T16:33:54.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:501fa391:::benchmark_data_smithi188_113957_object74444:head# 2024-04-23T16:33:54.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50206a75:::benchmark_data_smithi188_113957_object31902:head# 2024-04-23T16:33:54.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502094ec:::benchmark_data_smithi188_113957_object66503:head# 2024-04-23T16:33:54.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5020fad2:::benchmark_data_smithi188_113957_object110213:head# 2024-04-23T16:33:54.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5021351b:::benchmark_data_smithi188_113957_object142004:head# 2024-04-23T16:33:54.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502167fc:::benchmark_data_smithi188_113957_object91258:head# 2024-04-23T16:33:54.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50219437:::benchmark_data_smithi188_113957_object11230:head# 2024-04-23T16:33:54.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.833 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50223fb0:::benchmark_data_smithi188_113957_object144689:head# 2024-04-23T16:33:54.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.833 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5022439e:::benchmark_data_smithi188_113957_object86313:head# 2024-04-23T16:33:54.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.833 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50226a7a:::benchmark_data_smithi188_113957_object73357:head# 2024-04-23T16:33:54.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.833 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5022858d:::benchmark_data_smithi188_113957_object122708:head# 2024-04-23T16:33:54.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.833 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5023ae17:::benchmark_data_smithi188_113957_object53567:head# 2024-04-23T16:33:54.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.833 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5023e0e8:::benchmark_data_smithi188_113957_object96847:head# 2024-04-23T16:33:54.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.833 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5024242a:::benchmark_data_smithi188_113957_object127771:head# 2024-04-23T16:33:54.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.834 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50243bc8:::benchmark_data_smithi188_113957_object152477:head# 2024-04-23T16:33:54.834 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:54.835 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:54.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:54.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50244c0a:::benchmark_data_smithi188_113957_object146516:head# 2024-04-23T16:33:54.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50244ea6:::benchmark_data_smithi188_113957_object5239:head# 2024-04-23T16:33:54.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5024b2c5:::benchmark_data_smithi188_113957_object151640:head# 2024-04-23T16:33:54.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5024e505:::benchmark_data_smithi188_113957_object46064:head# 2024-04-23T16:33:54.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5026758a:::benchmark_data_smithi188_113957_object115505:head# 2024-04-23T16:33:54.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5026dd3c:::benchmark_data_smithi188_113957_object143309:head# 2024-04-23T16:33:54.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50272fe3:::benchmark_data_smithi188_113957_object139349:head# 2024-04-23T16:33:54.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502759e6:::benchmark_data_smithi188_113957_object124882:head# 2024-04-23T16:33:54.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50276568:::benchmark_data_smithi188_113957_object84720:head# 2024-04-23T16:33:54.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5027c812:::benchmark_data_smithi188_113957_object29768:head# 2024-04-23T16:33:54.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502847db:::benchmark_data_smithi188_113957_object4102:head# 2024-04-23T16:33:54.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50291df1:::benchmark_data_smithi188_113957_object100204:head# 2024-04-23T16:33:54.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5029d62d:::benchmark_data_smithi188_113957_object97228:head# 2024-04-23T16:33:54.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5029f49c:::benchmark_data_smithi188_113957_object128641:head# 2024-04-23T16:33:54.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502a4a49:::benchmark_data_smithi188_113957_object51156:head# 2024-04-23T16:33:54.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502a99cc:::benchmark_data_smithi188_113957_object75234:head# 2024-04-23T16:33:54.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502aa9d3:::benchmark_data_smithi188_113957_object119983:head# 2024-04-23T16:33:54.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502aae6c:::benchmark_data_smithi188_113957_object106654:head# 2024-04-23T16:33:54.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502af0e3:::benchmark_data_smithi188_113957_object23161:head# 2024-04-23T16:33:54.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502b5bc9:::benchmark_data_smithi188_113957_object79950:head# 2024-04-23T16:33:54.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502b5c9c:::benchmark_data_smithi188_113957_object74396:head# 2024-04-23T16:33:54.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502b5e03:::benchmark_data_smithi188_113957_object132188:head# 2024-04-23T16:33:54.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502c3c8f:::benchmark_data_smithi188_113957_object57260:head# 2024-04-23T16:33:54.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502c8aa1:::benchmark_data_smithi188_113957_object12772:head# 2024-04-23T16:33:54.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502cd8e5:::benchmark_data_smithi188_113957_object128251:head# 2024-04-23T16:33:54.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502e0ebc:::benchmark_data_smithi188_113957_object34542:head# 2024-04-23T16:33:54.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502f1fee:::benchmark_data_smithi188_113957_object29518:head# 2024-04-23T16:33:54.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502f3e10:::benchmark_data_smithi188_113957_object133225:head# 2024-04-23T16:33:54.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502f83f3:::benchmark_data_smithi188_113957_object62180:head# 2024-04-23T16:33:54.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502fbe27:::benchmark_data_smithi188_113957_object66158:head# 2024-04-23T16:33:54.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:502fd17f:::benchmark_data_smithi188_113957_object134092:head# 2024-04-23T16:33:54.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50300311:::benchmark_data_smithi188_113957_object27818:head# 2024-04-23T16:33:54.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:503008de:::benchmark_data_smithi188_113957_object87560:head# 2024-04-23T16:33:54.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50306ebe:::benchmark_data_smithi188_113957_object83540:head# 2024-04-23T16:33:54.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5030aa44:::benchmark_data_smithi188_113957_object120509:head# 2024-04-23T16:33:54.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5031e78a:::benchmark_data_smithi188_113957_object76842:head# 2024-04-23T16:33:54.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50320092:::benchmark_data_smithi188_113957_object7863:head# 2024-04-23T16:33:54.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50325392:::benchmark_data_smithi188_113957_object131703:head# 2024-04-23T16:33:54.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50328553:::benchmark_data_smithi188_113957_object108775:head# 2024-04-23T16:33:54.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50329319:::benchmark_data_smithi188_113957_object140154:head# 2024-04-23T16:33:54.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5032c908:::benchmark_data_smithi188_113957_object126186:head# 2024-04-23T16:33:54.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5033a60a:::benchmark_data_smithi188_113957_object96745:head# 2024-04-23T16:33:54.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5034046b:::benchmark_data_smithi188_113957_object74619:head# 2024-04-23T16:33:54.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5034c963:::benchmark_data_smithi188_113957_object59746:head# 2024-04-23T16:33:54.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50357126:::benchmark_data_smithi188_113957_object39807:head# 2024-04-23T16:33:54.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5036b891:::benchmark_data_smithi188_113957_object32984:head# 2024-04-23T16:33:54.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5036fb33:::benchmark_data_smithi188_113957_object26055:head# 2024-04-23T16:33:54.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:503746fa:::benchmark_data_smithi188_113957_object86256:head# 2024-04-23T16:33:54.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50379675:::benchmark_data_smithi188_113957_object33754:head# 2024-04-23T16:33:54.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50379b99:::benchmark_data_smithi188_113957_object116991:head# 2024-04-23T16:33:54.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50385eac:::benchmark_data_smithi188_113957_object6687:head# 2024-04-23T16:33:54.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50387064:::benchmark_data_smithi188_113957_object90738:head# 2024-04-23T16:33:54.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5038b827:::benchmark_data_smithi188_113957_object23313:head# 2024-04-23T16:33:54.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:503a28ba:::benchmark_data_smithi188_113957_object131031:head# 2024-04-23T16:33:54.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:503a94be:::benchmark_data_smithi188_113957_object76840:head# 2024-04-23T16:33:54.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:503a9ac9:::benchmark_data_smithi188_113957_object46868:head# 2024-04-23T16:33:54.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:503ad802:::benchmark_data_smithi188_113957_object123601:head# 2024-04-23T16:33:54.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:503b45d1:::benchmark_data_smithi188_113957_object60381:head# 2024-04-23T16:33:54.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:503cd587:::benchmark_data_smithi188_113957_object115473:head# 2024-04-23T16:33:54.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:503e07d6:::benchmark_data_smithi188_113957_object42571:head# 2024-04-23T16:33:54.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:503e1ff6:::benchmark_data_smithi188_113957_object68126:head# 2024-04-23T16:33:54.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:503ecd30:::benchmark_data_smithi188_113957_object52640:head# 2024-04-23T16:33:54.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:503fe0c2:::benchmark_data_smithi188_113957_object119609:head# 2024-04-23T16:33:54.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:503ff1f8:::benchmark_data_smithi188_113957_object45520:head# 2024-04-23T16:33:54.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504031e1:::benchmark_data_smithi188_113957_object68840:head# 2024-04-23T16:33:54.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50409a09:::benchmark_data_smithi188_113957_object116446:head# 2024-04-23T16:33:54.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50411833:::benchmark_data_smithi188_113957_object146925:head# 2024-04-23T16:33:54.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50413e16:::benchmark_data_smithi188_113957_object81943:head# 2024-04-23T16:33:54.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50415219:::benchmark_data_smithi188_113957_object130234:head# 2024-04-23T16:33:54.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5041ab5f:::benchmark_data_smithi188_113957_object128837:head# 2024-04-23T16:33:54.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5041bcdd:::benchmark_data_smithi188_113957_object111134:head# 2024-04-23T16:33:54.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50425d86:::benchmark_data_smithi188_113957_object71424:head# 2024-04-23T16:33:54.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50437003:::benchmark_data_smithi188_113957_object137997:head# 2024-04-23T16:33:54.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50439ad9:::benchmark_data_smithi188_113957_object151019:head# 2024-04-23T16:33:54.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5044671c:::benchmark_data_smithi188_113957_object101294:head# 2024-04-23T16:33:54.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5044cd0a:::benchmark_data_smithi188_113957_object153123:head# 2024-04-23T16:33:54.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5045696c:::benchmark_data_smithi188_113957_object10588:head# 2024-04-23T16:33:54.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50457304:::benchmark_data_smithi188_113957_object945:head# 2024-04-23T16:33:54.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5045ad5d:::benchmark_data_smithi188_113957_object13242:head# 2024-04-23T16:33:54.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5045c1bb:::benchmark_data_smithi188_113957_object43442:head# 2024-04-23T16:33:54.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5045e0a8:::benchmark_data_smithi188_113957_object50351:head# 2024-04-23T16:33:54.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50461fd5:::benchmark_data_smithi188_113957_object35755:head# 2024-04-23T16:33:54.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50464493:::benchmark_data_smithi188_113957_object76557:head# 2024-04-23T16:33:54.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5046c4a3:::benchmark_data_smithi188_113957_object70128:head# 2024-04-23T16:33:54.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50470e1a:::benchmark_data_smithi188_113957_object49211:head# 2024-04-23T16:33:54.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5047cd42:::benchmark_data_smithi188_113957_object80187:head# 2024-04-23T16:33:54.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50480269:::benchmark_data_smithi188_113957_object68752:head# 2024-04-23T16:33:54.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5048351f:::benchmark_data_smithi188_113957_object105326:head# 2024-04-23T16:33:54.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5048cff3:::benchmark_data_smithi188_113957_object51617:head# 2024-04-23T16:33:54.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5048f8d0:::benchmark_data_smithi188_113957_object3230:head# 2024-04-23T16:33:54.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50492d46:::benchmark_data_smithi188_113957_object8667:head# 2024-04-23T16:33:54.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504936de:::benchmark_data_smithi188_113957_object15157:head# 2024-04-23T16:33:54.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504988e6:::benchmark_data_smithi188_113957_object76719:head# 2024-04-23T16:33:54.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504a76d3:::benchmark_data_smithi188_113957_object149473:head# 2024-04-23T16:33:54.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504ad1f3:::benchmark_data_smithi188_113957_object116188:head# 2024-04-23T16:33:54.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504b45c0:::benchmark_data_smithi188_113957_object14702:head# 2024-04-23T16:33:54.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504bbeb7:::benchmark_data_smithi188_113957_object128681:head# 2024-04-23T16:33:54.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504c0f45:::benchmark_data_smithi188_113957_object102497:head# 2024-04-23T16:33:54.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504d1175:::benchmark_data_smithi188_113957_object136255:head# 2024-04-23T16:33:54.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504d3e2e:::benchmark_data_smithi188_113957_object93737:head# 2024-04-23T16:33:54.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504d8a41:::benchmark_data_smithi188_113957_object1726:head# 2024-04-23T16:33:54.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504dc19e:::benchmark_data_smithi188_113957_object107885:head# 2024-04-23T16:33:54.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504ddafa:::benchmark_data_smithi188_113957_object27239:head# 2024-04-23T16:33:54.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504de971:::benchmark_data_smithi188_113957_object39895:head# 2024-04-23T16:33:54.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504e2928:::benchmark_data_smithi188_113957_object96576:head# 2024-04-23T16:33:54.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504e6b0b:::benchmark_data_smithi188_113957_object40449:head# 2024-04-23T16:33:54.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504ec570:::benchmark_data_smithi188_113957_object45756:head# 2024-04-23T16:33:54.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504edf5d:::benchmark_data_smithi188_113957_object85298:head# 2024-04-23T16:33:54.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504f2f2b:::benchmark_data_smithi188_113957_object53222:head# 2024-04-23T16:33:54.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504f7e13:::benchmark_data_smithi188_113957_object28936:head# 2024-04-23T16:33:54.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504f9fc9:::benchmark_data_smithi188_113957_object89666:head# 2024-04-23T16:33:54.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:504fec48:::benchmark_data_smithi188_113957_object7513:head# 2024-04-23T16:33:54.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5050eb83:::benchmark_data_smithi188_113957_object133202:head# 2024-04-23T16:33:54.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5050f24e:::benchmark_data_smithi188_113957_object131196:head# 2024-04-23T16:33:54.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:505131c3:::benchmark_data_smithi188_113957_object78097:head# 2024-04-23T16:33:54.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50513cae:::benchmark_data_smithi188_113957_object103344:head# 2024-04-23T16:33:54.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50515d81:::benchmark_data_smithi188_113957_object29942:head# 2024-04-23T16:33:54.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5051ab3f:::benchmark_data_smithi188_113957_object40779:head# 2024-04-23T16:33:54.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5051abfe:::benchmark_data_smithi188_113957_object1392:head# 2024-04-23T16:33:54.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:505243a7:::benchmark_data_smithi188_113957_object150573:head# 2024-04-23T16:33:54.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:505285cc:::benchmark_data_smithi188_113957_object62791:head# 2024-04-23T16:33:54.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5052a38d:::benchmark_data_smithi188_113957_object110981:head# 2024-04-23T16:33:54.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:505491c0:::benchmark_data_smithi188_113957_object44575:head# 2024-04-23T16:33:54.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5054fadf:::benchmark_data_smithi188_113957_object21287:head# 2024-04-23T16:33:54.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50556f69:::benchmark_data_smithi188_113957_object4375:head# 2024-04-23T16:33:54.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5055718b:::benchmark_data_smithi188_113957_object21725:head# 2024-04-23T16:33:54.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50559dbb:::benchmark_data_smithi188_113957_object100478:head# 2024-04-23T16:33:54.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5055a719:::benchmark_data_smithi188_113957_object110645:head# 2024-04-23T16:33:54.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50563499:::benchmark_data_smithi188_113957_object96342:head# 2024-04-23T16:33:54.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5056370c:::benchmark_data_smithi188_113957_object139452:head# 2024-04-23T16:33:54.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:505648fb:::benchmark_data_smithi188_113957_object26430:head# 2024-04-23T16:33:54.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50564e6a:::benchmark_data_smithi188_113957_object149242:head# 2024-04-23T16:33:54.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5056c403:::benchmark_data_smithi188_113957_object74868:head# 2024-04-23T16:33:54.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50572568:::benchmark_data_smithi188_113957_object38810:head# 2024-04-23T16:33:54.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50573e8f:::benchmark_data_smithi188_113957_object103561:head# 2024-04-23T16:33:54.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.881 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:54 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 26477/279782 objects degraded (9.463%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:54.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5057b998:::benchmark_data_smithi188_113957_object56631:head# 2024-04-23T16:33:54.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5057ed55:::benchmark_data_smithi188_113957_object88595:head# 2024-04-23T16:33:54.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50585b2e:::benchmark_data_smithi188_113957_object10802:head# 2024-04-23T16:33:54.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5058830c:::benchmark_data_smithi188_113957_object32951:head# 2024-04-23T16:33:54.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:505901f7:::benchmark_data_smithi188_113957_object67458:head# 2024-04-23T16:33:54.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:505951a3:::benchmark_data_smithi188_113957_object23361:head# 2024-04-23T16:33:54.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5059f177:::benchmark_data_smithi188_113957_object20278:head# 2024-04-23T16:33:54.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:505a09e8:::benchmark_data_smithi188_113957_object72346:head# 2024-04-23T16:33:54.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:505a683d:::benchmark_data_smithi188_113957_object94888:head# 2024-04-23T16:33:54.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:505a6ab7:::benchmark_data_smithi188_113957_object20173:head# 2024-04-23T16:33:54.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:505ae16c:::benchmark_data_smithi188_113957_object30540:head# 2024-04-23T16:33:54.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:505b0f4b:::benchmark_data_smithi188_113957_object48688:head# 2024-04-23T16:33:54.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:505cd687:::benchmark_data_smithi188_113957_object29943:head# 2024-04-23T16:33:54.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:505cdcfc:::benchmark_data_smithi188_113957_object104929:head# 2024-04-23T16:33:54.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:505e2d3f:::benchmark_data_smithi188_113957_object101922:head# 2024-04-23T16:33:54.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:505f450e:::benchmark_data_smithi188_113957_object117573:head# 2024-04-23T16:33:54.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5060012a:::benchmark_data_smithi188_113957_object80149:head# 2024-04-23T16:33:54.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50602ab2:::benchmark_data_smithi188_113957_object139308:head# 2024-04-23T16:33:54.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50603644:::benchmark_data_smithi188_113957_object122095:head# 2024-04-23T16:33:54.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50605b5c:::benchmark_data_smithi188_113957_object131516:head# 2024-04-23T16:33:54.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50607485:::benchmark_data_smithi188_113957_object1189:head# 2024-04-23T16:33:54.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50608292:::benchmark_data_smithi188_113957_object74086:head# 2024-04-23T16:33:54.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50612f4b:::benchmark_data_smithi188_113957_object138188:head# 2024-04-23T16:33:54.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50614827:::benchmark_data_smithi188_113957_object90008:head# 2024-04-23T16:33:54.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50618274:::benchmark_data_smithi188_113957_object9078:head# 2024-04-23T16:33:54.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506197bb:::benchmark_data_smithi188_113957_object59575:head# 2024-04-23T16:33:54.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50621593:::benchmark_data_smithi188_113957_object95828:head# 2024-04-23T16:33:54.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5062e232:::benchmark_data_smithi188_113957_object29395:head# 2024-04-23T16:33:54.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506367dc:::benchmark_data_smithi188_113957_object26792:head# 2024-04-23T16:33:54.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50637634:::benchmark_data_smithi188_113957_object139593:head# 2024-04-23T16:33:54.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506392c1:::benchmark_data_smithi188_113957_object32204:head# 2024-04-23T16:33:54.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5063b02f:::benchmark_data_smithi188_113957_object39395:head# 2024-04-23T16:33:54.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5064140b:::benchmark_data_smithi188_113957_object122795:head# 2024-04-23T16:33:54.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50642564:::benchmark_data_smithi188_113957_object128471:head# 2024-04-23T16:33:54.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50648b42:::benchmark_data_smithi188_113957_object1040:head# 2024-04-23T16:33:54.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50648b5d:::benchmark_data_smithi188_113957_object110384:head# 2024-04-23T16:33:54.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5064b628:::benchmark_data_smithi188_113957_object116359:head# 2024-04-23T16:33:54.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5064f6f6:::benchmark_data_smithi188_113957_object58742:head# 2024-04-23T16:33:54.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5064fadd:::benchmark_data_smithi188_113957_object629:head# 2024-04-23T16:33:54.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506570d2:::benchmark_data_smithi188_113957_object83696:head# 2024-04-23T16:33:54.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50662eb1:::benchmark_data_smithi188_113957_object109449:head# 2024-04-23T16:33:54.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5066cc92:::benchmark_data_smithi188_113957_object60646:head# 2024-04-23T16:33:54.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5067a74c:::benchmark_data_smithi188_113957_object57981:head# 2024-04-23T16:33:54.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5067bf25:::benchmark_data_smithi188_113957_object8814:head# 2024-04-23T16:33:54.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5067c775:::benchmark_data_smithi188_113957_object90304:head# 2024-04-23T16:33:54.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506841eb:::benchmark_data_smithi188_113957_object1785:head# 2024-04-23T16:33:54.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50689834:::benchmark_data_smithi188_113957_object71203:head# 2024-04-23T16:33:54.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5068b38d:::benchmark_data_smithi188_113957_object6034:head# 2024-04-23T16:33:54.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5068cdca:::benchmark_data_smithi188_113957_object47201:head# 2024-04-23T16:33:54.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5068fc3a:::benchmark_data_smithi188_113957_object117619:head# 2024-04-23T16:33:54.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50692f5c:::benchmark_data_smithi188_113957_object129878:head# 2024-04-23T16:33:54.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506989f6:::benchmark_data_smithi188_113957_object46819:head# 2024-04-23T16:33:54.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5069f9f0:::benchmark_data_smithi188_113957_object61543:head# 2024-04-23T16:33:54.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506a6a57:::benchmark_data_smithi188_113957_object24313:head# 2024-04-23T16:33:54.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506ac60a:::benchmark_data_smithi188_113957_object46256:head# 2024-04-23T16:33:54.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506bb9ac:::benchmark_data_smithi188_113957_object69153:head# 2024-04-23T16:33:54.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506bf8b4:::benchmark_data_smithi188_113957_object10703:head# 2024-04-23T16:33:54.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506c6a1f:::benchmark_data_smithi188_113957_object100851:head# 2024-04-23T16:33:54.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506cb347:::benchmark_data_smithi188_113957_object79517:head# 2024-04-23T16:33:54.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506d17d7:::benchmark_data_smithi188_113957_object66729:head# 2024-04-23T16:33:54.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506d3c6a:::benchmark_data_smithi188_113957_object143692:head# 2024-04-23T16:33:54.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506d8a5a:::benchmark_data_smithi188_113957_object134588:head# 2024-04-23T16:33:54.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506e374f:::benchmark_data_smithi188_113957_object56080:head# 2024-04-23T16:33:54.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506e9bcb:::benchmark_data_smithi188_113957_object62307:head# 2024-04-23T16:33:54.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506eb1de:::benchmark_data_smithi188_113957_object52049:head# 2024-04-23T16:33:54.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506f659e:::benchmark_data_smithi188_113957_object93815:head# 2024-04-23T16:33:54.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:506fe96e:::benchmark_data_smithi188_113957_object14674:head# 2024-04-23T16:33:54.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50706f2e:::benchmark_data_smithi188_113957_object86000:head# 2024-04-23T16:33:54.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5070a063:::benchmark_data_smithi188_113957_object101740:head# 2024-04-23T16:33:54.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5071ae6d:::benchmark_data_smithi188_113957_object88574:head# 2024-04-23T16:33:54.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5072194c:::benchmark_data_smithi188_113957_object136179:head# 2024-04-23T16:33:54.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5072ad2f:::benchmark_data_smithi188_113957_object106860:head# 2024-04-23T16:33:54.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5072dded:::benchmark_data_smithi188_113957_object36605:head# 2024-04-23T16:33:54.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5072e5ca:::benchmark_data_smithi188_113957_object141157:head# 2024-04-23T16:33:54.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5073b2c0:::benchmark_data_smithi188_113957_object58190:head# 2024-04-23T16:33:54.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5073b91a:::benchmark_data_smithi188_113957_object148432:head# 2024-04-23T16:33:54.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5074295a:::benchmark_data_smithi188_113957_object116171:head# 2024-04-23T16:33:54.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50748b41:::benchmark_data_smithi188_113957_object34942:head# 2024-04-23T16:33:54.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5074fdf1:::benchmark_data_smithi188_113957_object51330:head# 2024-04-23T16:33:54.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50754044:::benchmark_data_smithi188_113957_object64612:head# 2024-04-23T16:33:54.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5075bdc2:::benchmark_data_smithi188_113957_object93348:head# 2024-04-23T16:33:54.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5075bfe2:::benchmark_data_smithi188_113957_object60794:head# 2024-04-23T16:33:54.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5075ea21:::benchmark_data_smithi188_113957_object31129:head# 2024-04-23T16:33:54.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50774ccc:::benchmark_data_smithi188_113957_object47641:head# 2024-04-23T16:33:54.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50776ff6:::benchmark_data_smithi188_113957_object32138:head# 2024-04-23T16:33:54.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50778d59:::benchmark_data_smithi188_113957_object70920:head# 2024-04-23T16:33:54.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50782c02:::benchmark_data_smithi188_113957_object19126:head# 2024-04-23T16:33:54.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507893d4:::benchmark_data_smithi188_113957_object140279:head# 2024-04-23T16:33:54.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50795a3e:::benchmark_data_smithi188_113957_object133862:head# 2024-04-23T16:33:54.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50799145:::benchmark_data_smithi188_113957_object3241:head# 2024-04-23T16:33:54.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507a445f:::benchmark_data_smithi188_113957_object118906:head# 2024-04-23T16:33:54.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507a68cf:::benchmark_data_smithi188_113957_object122511:head# 2024-04-23T16:33:54.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507ae352:::benchmark_data_smithi188_113957_object89624:head# 2024-04-23T16:33:54.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507b3f5d:::benchmark_data_smithi188_113957_object135301:head# 2024-04-23T16:33:54.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507b8295:::benchmark_data_smithi188_113957_object8698:head# 2024-04-23T16:33:54.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507c0390:::benchmark_data_smithi188_113957_object44457:head# 2024-04-23T16:33:54.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507c29a9:::benchmark_data_smithi188_113957_object117352:head# 2024-04-23T16:33:54.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507cc113:::benchmark_data_smithi188_113957_object42461:head# 2024-04-23T16:33:54.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507cdae0:::benchmark_data_smithi188_113957_object92776:head# 2024-04-23T16:33:54.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507d28e7:::benchmark_data_smithi188_113957_object114068:head# 2024-04-23T16:33:54.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507d4614:::benchmark_data_smithi188_113957_object95850:head# 2024-04-23T16:33:54.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507d64d0:::benchmark_data_smithi188_113957_object42565:head# 2024-04-23T16:33:54.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507d7d3f:::benchmark_data_smithi188_113957_object27226:head# 2024-04-23T16:33:54.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507d8bd8:::benchmark_data_smithi188_113957_object99958:head# 2024-04-23T16:33:54.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507d922c:::benchmark_data_smithi188_113957_object48087:head# 2024-04-23T16:33:54.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507d9fb7:::benchmark_data_smithi188_113957_object74412:head# 2024-04-23T16:33:54.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507dc51f:::benchmark_data_smithi188_113957_object6236:head# 2024-04-23T16:33:54.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507e9da2:::benchmark_data_smithi188_113957_object6054:head# 2024-04-23T16:33:54.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507ea31c:::benchmark_data_smithi188_113957_object11266:head# 2024-04-23T16:33:54.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507f01d1:::benchmark_data_smithi188_113957_object133982:head# 2024-04-23T16:33:54.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:507f8dbf:::benchmark_data_smithi188_113957_object19910:head# 2024-04-23T16:33:54.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5080520b:::benchmark_data_smithi188_113957_object26408:head# 2024-04-23T16:33:54.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50805d9d:::benchmark_data_smithi188_113957_object54975:head# 2024-04-23T16:33:54.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50805f3c:::benchmark_data_smithi188_113957_object34641:head# 2024-04-23T16:33:54.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50821d4f:::benchmark_data_smithi188_113957_object7574:head# 2024-04-23T16:33:54.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50823d2e:::benchmark_data_smithi188_113957_object137780:head# 2024-04-23T16:33:54.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5082ec9b:::benchmark_data_smithi188_113957_object126505:head# 2024-04-23T16:33:54.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50834397:::benchmark_data_smithi188_113957_object138754:head# 2024-04-23T16:33:54.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50834831:::benchmark_data_smithi188_113957_object18823:head# 2024-04-23T16:33:54.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50834e5c:::benchmark_data_smithi188_113957_object36267:head# 2024-04-23T16:33:54.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50842811:::benchmark_data_smithi188_113957_object76889:head# 2024-04-23T16:33:54.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50846144:::benchmark_data_smithi188_113957_object64633:head# 2024-04-23T16:33:54.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5085238f:::benchmark_data_smithi188_113957_object42248:head# 2024-04-23T16:33:54.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5085799d:::benchmark_data_smithi188_113957_object135355:head# 2024-04-23T16:33:54.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:508617ce:::benchmark_data_smithi188_113957_object133221:head# 2024-04-23T16:33:54.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5086f448:::benchmark_data_smithi188_113957_object53933:head# 2024-04-23T16:33:54.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50872fda:::benchmark_data_smithi188_113957_object2573:head# 2024-04-23T16:33:54.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50874d5f:::benchmark_data_smithi188_113957_object17235:head# 2024-04-23T16:33:54.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:508757f1:::benchmark_data_smithi188_113957_object19645:head# 2024-04-23T16:33:54.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50879d6a:::benchmark_data_smithi188_113957_object29530:head# 2024-04-23T16:33:54.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5088b44a:::benchmark_data_smithi188_113957_object101853:head# 2024-04-23T16:33:54.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50891d95:::benchmark_data_smithi188_113957_object107526:head# 2024-04-23T16:33:54.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:508b589c:::benchmark_data_smithi188_113957_object14397:head# 2024-04-23T16:33:54.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:508ba08a:::benchmark_data_smithi188_113957_object83239:head# 2024-04-23T16:33:54.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:508bdbce:::benchmark_data_smithi188_113957_object61287:head# 2024-04-23T16:33:54.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:508c4b6c:::benchmark_data_smithi188_113957_object130221:head# 2024-04-23T16:33:54.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:508cc536:::benchmark_data_smithi188_113957_object130163:head# 2024-04-23T16:33:54.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:508d5d14:::benchmark_data_smithi188_113957_object55024:head# 2024-04-23T16:33:54.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:508e03ec:::benchmark_data_smithi188_113957_object129594:head# 2024-04-23T16:33:54.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:508e2e54:::benchmark_data_smithi188_113957_object127838:head# 2024-04-23T16:33:54.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:508e554d:::benchmark_data_smithi188_113957_object131477:head# 2024-04-23T16:33:54.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:508ec647:::benchmark_data_smithi188_113957_object64908:head# 2024-04-23T16:33:54.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:508f38ad:::benchmark_data_smithi188_113957_object63947:head# 2024-04-23T16:33:54.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:508fda59:::benchmark_data_smithi188_113957_object30617:head# 2024-04-23T16:33:54.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:508ff668:::benchmark_data_smithi188_113957_object85036:head# 2024-04-23T16:33:54.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50900b81:::benchmark_data_smithi188_113957_object37830:head# 2024-04-23T16:33:54.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509020f9:::benchmark_data_smithi188_113957_object114488:head# 2024-04-23T16:33:54.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509064f6:::benchmark_data_smithi188_113957_object106536:head# 2024-04-23T16:33:54.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5090a6f0:::benchmark_data_smithi188_113957_object146530:head# 2024-04-23T16:33:54.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5090c48d:::benchmark_data_smithi188_113957_object84657:head# 2024-04-23T16:33:54.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50912471:::benchmark_data_smithi188_113957_object110195:head# 2024-04-23T16:33:54.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50914f70:::benchmark_data_smithi188_113957_object27765:head# 2024-04-23T16:33:54.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5091dcf0:::benchmark_data_smithi188_113957_object112394:head# 2024-04-23T16:33:54.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50921a51:::benchmark_data_smithi188_113957_object145073:head# 2024-04-23T16:33:54.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50925286:::benchmark_data_smithi188_113957_object95564:head# 2024-04-23T16:33:54.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50928b68:::benchmark_data_smithi188_113957_object142095:head# 2024-04-23T16:33:54.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5092904c:::benchmark_data_smithi188_113957_object20514:head# 2024-04-23T16:33:54.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5092e29b:::benchmark_data_smithi188_113957_object103922:head# 2024-04-23T16:33:54.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50936750:::benchmark_data_smithi188_113957_object107972:head# 2024-04-23T16:33:54.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50943dad:::benchmark_data_smithi188_113957_object54429:head# 2024-04-23T16:33:54.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5096aa02:::benchmark_data_smithi188_113957_object94559:head# 2024-04-23T16:33:54.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5096ed23:::benchmark_data_smithi188_113957_object43034:head# 2024-04-23T16:33:54.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509761c4:::benchmark_data_smithi188_113957_object56284:head# 2024-04-23T16:33:54.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50977a8f:::benchmark_data_smithi188_113957_object65493:head# 2024-04-23T16:33:54.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509806dd:::benchmark_data_smithi188_113957_object91409:head# 2024-04-23T16:33:54.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50989e27:::benchmark_data_smithi188_113957_object152942:head# 2024-04-23T16:33:54.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5099521d:::benchmark_data_smithi188_113957_object59301:head# 2024-04-23T16:33:54.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50995e32:::benchmark_data_smithi188_113957_object33281:head# 2024-04-23T16:33:54.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509abeb5:::benchmark_data_smithi188_113957_object51552:head# 2024-04-23T16:33:54.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509afb05:::benchmark_data_smithi188_113957_object136407:head# 2024-04-23T16:33:54.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509b16ea:::benchmark_data_smithi188_113957_object29511:head# 2024-04-23T16:33:54.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509b9bb2:::benchmark_data_smithi188_113957_object140083:head# 2024-04-23T16:33:54.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509bd9bc:::benchmark_data_smithi188_113957_object55965:head# 2024-04-23T16:33:54.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509bf0bc:::benchmark_data_smithi188_113957_object128048:head# 2024-04-23T16:33:54.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509c0a03:::benchmark_data_smithi188_113957_object92196:head# 2024-04-23T16:33:54.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509c7e10:::benchmark_data_smithi188_113957_object42119:head# 2024-04-23T16:33:54.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509c7eea:::benchmark_data_smithi188_113957_object55182:head# 2024-04-23T16:33:54.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509c928f:::benchmark_data_smithi188_113957_object87201:head# 2024-04-23T16:33:54.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509cb43f:::benchmark_data_smithi188_113957_object138546:head# 2024-04-23T16:33:54.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509d351d:::benchmark_data_smithi188_113957_object153172:head# 2024-04-23T16:33:54.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509de4a1:::benchmark_data_smithi188_113957_object128145:head# 2024-04-23T16:33:54.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509e1e82:::benchmark_data_smithi188_113957_object47070:head# 2024-04-23T16:33:54.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509e484e:::benchmark_data_smithi188_113957_object30832:head# 2024-04-23T16:33:54.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509e5a73:::benchmark_data_smithi188_113957_object150382:head# 2024-04-23T16:33:54.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.962 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509e754d:::benchmark_data_smithi188_113957_object127441:head# 2024-04-23T16:33:54.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.962 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509f1ac2:::benchmark_data_smithi188_113957_object130478:head# 2024-04-23T16:33:54.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.962 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509f5bc1:::benchmark_data_smithi188_113957_object16986:head# 2024-04-23T16:33:54.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.962 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509f7986:::benchmark_data_smithi188_113957_object21285:head# 2024-04-23T16:33:54.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.962 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509fb181:::benchmark_data_smithi188_113957_object51414:head# 2024-04-23T16:33:54.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.962 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:509fe1c8:::benchmark_data_smithi188_113957_object42018:head# 2024-04-23T16:33:54.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a0461b:::benchmark_data_smithi188_113957_object51213:head# 2024-04-23T16:33:54.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a0b4d8:::benchmark_data_smithi188_113957_object85485:head# 2024-04-23T16:33:54.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a176d9:::benchmark_data_smithi188_113957_object87817:head# 2024-04-23T16:33:54.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a198d5:::benchmark_data_smithi188_113957_object109411:head# 2024-04-23T16:33:54.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a1eae3:::benchmark_data_smithi188_113957_object98239:head# 2024-04-23T16:33:54.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a22734:::benchmark_data_smithi188_113957_object18181:head# 2024-04-23T16:33:54.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a2ef80:::benchmark_data_smithi188_113957_object61204:head# 2024-04-23T16:33:54.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.964 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a46118:::benchmark_data_smithi188_113957_object82129:head# 2024-04-23T16:33:54.964 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.964 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a51a80:::benchmark_data_smithi188_113957_object63439:head# 2024-04-23T16:33:54.964 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.964 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a5e674:::benchmark_data_smithi188_113957_object97286:head# 2024-04-23T16:33:54.964 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.964 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a6b962:::benchmark_data_smithi188_113957_object122839:head# 2024-04-23T16:33:54.964 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.964 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a72697:::benchmark_data_smithi188_113957_object32568:head# 2024-04-23T16:33:54.964 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.964 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a793e4:::benchmark_data_smithi188_113957_object8723:head# 2024-04-23T16:33:54.964 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.964 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a7d66b:::benchmark_data_smithi188_113957_object62449:head# 2024-04-23T16:33:54.964 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.964 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a8359a:::benchmark_data_smithi188_113957_object19695:head# 2024-04-23T16:33:54.965 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.965 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a88d6f:::benchmark_data_smithi188_113957_object33236:head# 2024-04-23T16:33:54.965 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.965 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a8ef14:::benchmark_data_smithi188_113957_object128896:head# 2024-04-23T16:33:54.965 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.965 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a93dc4:::benchmark_data_smithi188_113957_object23215:head# 2024-04-23T16:33:54.965 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.965 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a9507b:::benchmark_data_smithi188_113957_object149233:head# 2024-04-23T16:33:54.965 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.965 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a958de:::benchmark_data_smithi188_113957_object124414:head# 2024-04-23T16:33:54.965 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.965 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a96aa2:::benchmark_data_smithi188_113957_object67747:head# 2024-04-23T16:33:54.965 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.965 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a9799e:::benchmark_data_smithi188_113957_object8950:head# 2024-04-23T16:33:54.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50a9929a:::benchmark_data_smithi188_113957_object18801:head# 2024-04-23T16:33:54.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50aa0416:::benchmark_data_smithi188_113957_object97149:head# 2024-04-23T16:33:54.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50aa1193:::benchmark_data_smithi188_113957_object37545:head# 2024-04-23T16:33:54.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50aacbd3:::benchmark_data_smithi188_113957_object147585:head# 2024-04-23T16:33:54.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ab1184:::benchmark_data_smithi188_113957_object150085:head# 2024-04-23T16:33:54.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ab46f4:::benchmark_data_smithi188_113957_object84161:head# 2024-04-23T16:33:54.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50abae45:::benchmark_data_smithi188_113957_object120439:head# 2024-04-23T16:33:54.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ac2d15:::benchmark_data_smithi188_113957_object90240:head# 2024-04-23T16:33:54.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ac4dd4:::benchmark_data_smithi188_113957_object70956:head# 2024-04-23T16:33:54.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ac6cf7:::benchmark_data_smithi188_113957_object88319:head# 2024-04-23T16:33:54.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50acaa9b:::benchmark_data_smithi188_113957_object14269:head# 2024-04-23T16:33:54.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ad4803:::benchmark_data_smithi188_113957_object117131:head# 2024-04-23T16:33:54.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ad5f27:::benchmark_data_smithi188_113957_object9960:head# 2024-04-23T16:33:54.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ad8004:::benchmark_data_smithi188_113957_object128849:head# 2024-04-23T16:33:54.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ae7cd7:::benchmark_data_smithi188_113957_object140531:head# 2024-04-23T16:33:54.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50aeafb9:::benchmark_data_smithi188_113957_object130775:head# 2024-04-23T16:33:54.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50aeb709:::benchmark_data_smithi188_113957_object77088:head# 2024-04-23T16:33:54.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50aee2e9:::benchmark_data_smithi188_113957_object148565:head# 2024-04-23T16:33:54.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50afadca:::benchmark_data_smithi188_113957_object16587:head# 2024-04-23T16:33:54.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b0ae12:::benchmark_data_smithi188_113957_object99872:head# 2024-04-23T16:33:54.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b0c334:::benchmark_data_smithi188_113957_object10037:head# 2024-04-23T16:33:54.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b0dcdd:::benchmark_data_smithi188_113957_object89554:head# 2024-04-23T16:33:54.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b1330e:::benchmark_data_smithi188_113957_object150275:head# 2024-04-23T16:33:54.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b22b7c:::benchmark_data_smithi188_113957_object71821:head# 2024-04-23T16:33:54.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b37df1:::benchmark_data_smithi188_113957_object66388:head# 2024-04-23T16:33:54.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b3d2f4:::benchmark_data_smithi188_113957_object16105:head# 2024-04-23T16:33:54.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b47438:::benchmark_data_smithi188_113957_object123662:head# 2024-04-23T16:33:54.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b5d6d4:::benchmark_data_smithi188_113957_object62519:head# 2024-04-23T16:33:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b6222c:::benchmark_data_smithi188_113957_object94240:head# 2024-04-23T16:33:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b71857:::benchmark_data_smithi188_113957_object101407:head# 2024-04-23T16:33:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b74f0c:::benchmark_data_smithi188_113957_object60688:head# 2024-04-23T16:33:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b7b957:::benchmark_data_smithi188_113957_object79502:head# 2024-04-23T16:33:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b7bf54:::benchmark_data_smithi188_113957_object74548:head# 2024-04-23T16:33:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b7ffa5:::benchmark_data_smithi188_113957_object127008:head# 2024-04-23T16:33:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b90f33:::benchmark_data_smithi188_113957_object29195:head# 2024-04-23T16:33:54.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b93fec:::benchmark_data_smithi188_113957_object27366:head# 2024-04-23T16:33:54.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b95026:::benchmark_data_smithi188_113957_object82605:head# 2024-04-23T16:33:54.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b9647e:::benchmark_data_smithi188_113957_object24515:head# 2024-04-23T16:33:54.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b9a36c:::benchmark_data_smithi188_113957_object133304:head# 2024-04-23T16:33:54.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b9bcc7:::benchmark_data_smithi188_113957_object106449:head# 2024-04-23T16:33:54.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50b9c070:::benchmark_data_smithi188_113957_object32343:head# 2024-04-23T16:33:54.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ba6d61:::benchmark_data_smithi188_113957_object35095:head# 2024-04-23T16:33:54.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50bb4b9c:::benchmark_data_smithi188_113957_object100626:head# 2024-04-23T16:33:54.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50bb77eb:::benchmark_data_smithi188_113957_object59669:head# 2024-04-23T16:33:54.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50bb8a3c:::benchmark_data_smithi188_113957_object130104:head# 2024-04-23T16:33:54.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50bc4811:::benchmark_data_smithi188_113957_object135037:head# 2024-04-23T16:33:54.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50bc6a12:::benchmark_data_smithi188_113957_object117803:head# 2024-04-23T16:33:54.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50bca88c:::benchmark_data_smithi188_113957_object105866:head# 2024-04-23T16:33:54.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50bcc6d7:::benchmark_data_smithi188_113957_object126932:head# 2024-04-23T16:33:54.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50bd1d2e:::benchmark_data_smithi188_113957_object24001:head# 2024-04-23T16:33:54.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50bd79d0:::benchmark_data_smithi188_113957_object48082:head# 2024-04-23T16:33:54.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50be833a:::benchmark_data_smithi188_113957_object21844:head# 2024-04-23T16:33:54.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50bec0c5:::benchmark_data_smithi188_113957_object42622:head# 2024-04-23T16:33:54.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50bf0199:::benchmark_data_smithi188_113957_object76125:head# 2024-04-23T16:33:54.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50bf4889:::benchmark_data_smithi188_113957_object55212:head# 2024-04-23T16:33:54.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50bf586a:::benchmark_data_smithi188_113957_object7163:head# 2024-04-23T16:33:54.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50bf5b3e:::benchmark_data_smithi188_113957_object5957:head# 2024-04-23T16:33:54.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50bfa7db:::benchmark_data_smithi188_113957_object43602:head# 2024-04-23T16:33:54.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50bfb612:::benchmark_data_smithi188_113957_object152811:head# 2024-04-23T16:33:54.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50bfed26:::benchmark_data_smithi188_113957_object83597:head# 2024-04-23T16:33:54.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50bfeeef:::benchmark_data_smithi188_113957_object4024:head# 2024-04-23T16:33:54.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50c04f31:::benchmark_data_smithi188_113957_object134025:head# 2024-04-23T16:33:54.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50c0ab36:::benchmark_data_smithi188_113957_object76340:head# 2024-04-23T16:33:54.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50c1f5d0:::benchmark_data_smithi188_113957_object148126:head# 2024-04-23T16:33:54.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50c230b8:::benchmark_data_smithi188_113957_object25956:head# 2024-04-23T16:33:54.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50c25179:::benchmark_data_smithi188_113957_object147448:head# 2024-04-23T16:33:54.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50c2962d:::benchmark_data_smithi188_113957_object23572:head# 2024-04-23T16:33:54.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50c347aa:::benchmark_data_smithi188_113957_object118747:head# 2024-04-23T16:33:54.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50c39a5f:::benchmark_data_smithi188_113957_object76959:head# 2024-04-23T16:33:54.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50c4f7cc:::benchmark_data_smithi188_113957_object4635:head# 2024-04-23T16:33:54.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50c5100e:::benchmark_data_smithi188_113957_object116872:head# 2024-04-23T16:33:54.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50c6513c:::benchmark_data_smithi188_113957_object30944:head# 2024-04-23T16:33:54.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50c76aeb:::benchmark_data_smithi188_113957_object130883:head# 2024-04-23T16:33:54.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50c87038:::benchmark_data_smithi188_113957_object60961:head# 2024-04-23T16:33:54.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50c89de4:::benchmark_data_smithi188_113957_object45765:head# 2024-04-23T16:33:54.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50c8e8fd:::benchmark_data_smithi188_113957_object101568:head# 2024-04-23T16:33:54.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50c930d7:::benchmark_data_smithi188_113957_object16995:head# 2024-04-23T16:33:54.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ca2df1:::benchmark_data_smithi188_113957_object144709:head# 2024-04-23T16:33:54.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50cb8e8c:::benchmark_data_smithi188_113957_object107427:head# 2024-04-23T16:33:54.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50cbbf73:::benchmark_data_smithi188_113957_object100517:head# 2024-04-23T16:33:54.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50cbd2df:::benchmark_data_smithi188_113957_object55600:head# 2024-04-23T16:33:54.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50cc74e9:::benchmark_data_smithi188_113957_object19085:head# 2024-04-23T16:33:54.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:54.977 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:33:55.019 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:55.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ccc14a:::benchmark_data_smithi188_113957_object25947:head# 2024-04-23T16:33:55.021 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50cd021a:::benchmark_data_smithi188_113957_object132836:head# 2024-04-23T16:33:55.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50cd21d5:::benchmark_data_smithi188_113957_object24349:head# 2024-04-23T16:33:55.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50cd5e9c:::benchmark_data_smithi188_113957_object49792:head# 2024-04-23T16:33:55.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ce2913:::benchmark_data_smithi188_113957_object84632:head# 2024-04-23T16:33:55.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ce33c5:::benchmark_data_smithi188_113957_object25252:head# 2024-04-23T16:33:55.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50cf209a:::benchmark_data_smithi188_113957_object117323:head# 2024-04-23T16:33:55.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50cf5ac1:::benchmark_data_smithi188_113957_object27807:head# 2024-04-23T16:33:55.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d0943b:::benchmark_data_smithi188_113957_object5648:head# 2024-04-23T16:33:55.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d0bbda:::benchmark_data_smithi188_113957_object152014:head# 2024-04-23T16:33:55.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d105a4:::benchmark_data_smithi188_113957_object16457:head# 2024-04-23T16:33:55.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d1b595:::benchmark_data_smithi188_113957_object41967:head# 2024-04-23T16:33:55.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d1d046:::benchmark_data_smithi188_113957_object138502:head# 2024-04-23T16:33:55.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d1d5dd:::benchmark_data_smithi188_113957_object45278:head# 2024-04-23T16:33:55.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d1e7dd:::benchmark_data_smithi188_113957_object2791:head# 2024-04-23T16:33:55.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d1fa70:::benchmark_data_smithi188_113957_object33245:head# 2024-04-23T16:33:55.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d21dde:::benchmark_data_smithi188_113957_object59092:head# 2024-04-23T16:33:55.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d25ccf:::benchmark_data_smithi188_113957_object141121:head# 2024-04-23T16:33:55.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d30e32:::benchmark_data_smithi188_113957_object79728:head# 2024-04-23T16:33:55.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d3162c:::benchmark_data_smithi188_113957_object143625:head# 2024-04-23T16:33:55.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d32701:::benchmark_data_smithi188_113957_object76157:head# 2024-04-23T16:33:55.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d35d68:::benchmark_data_smithi188_113957_object23744:head# 2024-04-23T16:33:55.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d37f3c:::benchmark_data_smithi188_113957_object26204:head# 2024-04-23T16:33:55.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d3fd2f:::benchmark_data_smithi188_113957_object147675:head# 2024-04-23T16:33:55.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d48db7:::benchmark_data_smithi188_113957_object37898:head# 2024-04-23T16:33:55.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d53139:::benchmark_data_smithi188_113957_object30248:head# 2024-04-23T16:33:55.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d6939e:::benchmark_data_smithi188_113957_object126313:head# 2024-04-23T16:33:55.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d69fdd:::benchmark_data_smithi188_113957_object110021:head# 2024-04-23T16:33:55.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d6ab21:::benchmark_data_smithi188_113957_object17317:head# 2024-04-23T16:33:55.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d6dcc0:::benchmark_data_smithi188_113957_object28310:head# 2024-04-23T16:33:55.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d732a1:::benchmark_data_smithi188_113957_object11703:head# 2024-04-23T16:33:55.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d7f000:::benchmark_data_smithi188_113957_object55216:head# 2024-04-23T16:33:55.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50d8e23e:::benchmark_data_smithi188_113957_object87177:head# 2024-04-23T16:33:55.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50da0847:::benchmark_data_smithi188_113957_object86499:head# 2024-04-23T16:33:55.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50da1354:::benchmark_data_smithi188_113957_object80704:head# 2024-04-23T16:33:55.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50da2d5a:::benchmark_data_smithi188_113957_object63224:head# 2024-04-23T16:33:55.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50da2fdc:::benchmark_data_smithi188_113957_object112274:head# 2024-04-23T16:33:55.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50da5071:::benchmark_data_smithi188_113957_object123536:head# 2024-04-23T16:33:55.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50dba284:::benchmark_data_smithi188_113957_object86423:head# 2024-04-23T16:33:55.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50dbda5f:::benchmark_data_smithi188_113957_object34996:head# 2024-04-23T16:33:55.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50dbeb67:::benchmark_data_smithi188_113957_object47824:head# 2024-04-23T16:33:55.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50dc2110:::benchmark_data_smithi188_113957_object152834:head# 2024-04-23T16:33:55.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ddac4b:::benchmark_data_smithi188_113957_object67637:head# 2024-04-23T16:33:55.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ddc280:::benchmark_data_smithi188_113957_object110900:head# 2024-04-23T16:33:55.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50de6d75:::benchmark_data_smithi188_113957_object43449:head# 2024-04-23T16:33:55.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50dea46f:::benchmark_data_smithi188_113957_object101872:head# 2024-04-23T16:33:55.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50deb021:::benchmark_data_smithi188_113957_object151729:head# 2024-04-23T16:33:55.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50df0c7a:::benchmark_data_smithi188_113957_object133364:head# 2024-04-23T16:33:55.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50df82ca:::benchmark_data_smithi188_113957_object121500:head# 2024-04-23T16:33:55.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50dfae04:::benchmark_data_smithi188_113957_object99080:head# 2024-04-23T16:33:55.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50e030d4:::benchmark_data_smithi188_113957_object2098:head# 2024-04-23T16:33:55.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50e07c36:::benchmark_data_smithi188_113957_object4862:head# 2024-04-23T16:33:55.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50e0b423:::benchmark_data_smithi188_113957_object18739:head# 2024-04-23T16:33:55.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50e25653:::benchmark_data_smithi188_113957_object35631:head# 2024-04-23T16:33:55.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50e2fa6d:::benchmark_data_smithi188_113957_object48628:head# 2024-04-23T16:33:55.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50e32006:::benchmark_data_smithi188_113957_object104793:head# 2024-04-23T16:33:55.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50e35246:::benchmark_data_smithi188_113957_object134241:head# 2024-04-23T16:33:55.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50e44616:::benchmark_data_smithi188_113957_object151172:head# 2024-04-23T16:33:55.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50e51dbe:::benchmark_data_smithi188_113957_object97806:head# 2024-04-23T16:33:55.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50e54270:::benchmark_data_smithi188_113957_object122134:head# 2024-04-23T16:33:55.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50e568c5:::benchmark_data_smithi188_113957_object82443:head# 2024-04-23T16:33:55.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50e56a6b:::benchmark_data_smithi188_113957_object2703:head# 2024-04-23T16:33:55.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50e6cbd4:::benchmark_data_smithi188_113957_object130083:head# 2024-04-23T16:33:55.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50e73057:::benchmark_data_smithi188_113957_object134899:head# 2024-04-23T16:33:55.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50e75302:::benchmark_data_smithi188_113957_object983:head# 2024-04-23T16:33:55.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50e9bd26:::benchmark_data_smithi188_113957_object105003:head# 2024-04-23T16:33:55.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50e9f198:::benchmark_data_smithi188_113957_object2927:head# 2024-04-23T16:33:55.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50eae8e3:::benchmark_data_smithi188_113957_object56419:head# 2024-04-23T16:33:55.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ee63e8:::benchmark_data_smithi188_113957_object88228:head# 2024-04-23T16:33:55.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ee7137:::benchmark_data_smithi188_113957_object37107:head# 2024-04-23T16:33:55.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50eef07f:::benchmark_data_smithi188_113957_object143429:head# 2024-04-23T16:33:55.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ef14c6:::benchmark_data_smithi188_113957_object123242:head# 2024-04-23T16:33:55.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ef94c0:::benchmark_data_smithi188_113957_object150358:head# 2024-04-23T16:33:55.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.032 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ef988e:::benchmark_data_smithi188_113957_object106109:head# 2024-04-23T16:33:55.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.032 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50efbf2c:::benchmark_data_smithi188_113957_object39738:head# 2024-04-23T16:33:55.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.032 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f0005d:::benchmark_data_smithi188_113957_object29523:head# 2024-04-23T16:33:55.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.032 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f080d3:::benchmark_data_smithi188_113957_object28677:head# 2024-04-23T16:33:55.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.032 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f0e33c:::benchmark_data_smithi188_113957_object131552:head# 2024-04-23T16:33:55.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.032 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f13ecc:::benchmark_data_smithi188_113957_object18647:head# 2024-04-23T16:33:55.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.032 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f17453:::benchmark_data_smithi188_113957_object117908:head# 2024-04-23T16:33:55.032 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.033 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f1a3a1:::benchmark_data_smithi188_113957_object69265:head# 2024-04-23T16:33:55.033 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.033 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f1f4c1:::benchmark_data_smithi188_113957_object96829:head# 2024-04-23T16:33:55.033 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.033 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f25fcf:::benchmark_data_smithi188_113957_object41929:head# 2024-04-23T16:33:55.033 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.033 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f2a753:::benchmark_data_smithi188_113957_object150972:head# 2024-04-23T16:33:55.033 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.033 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f2c852:::benchmark_data_smithi188_113957_object116010:head# 2024-04-23T16:33:55.033 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.033 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f33230:::benchmark_data_smithi188_113957_object20631:head# 2024-04-23T16:33:55.033 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.033 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f37e96:::benchmark_data_smithi188_113957_object74085:head# 2024-04-23T16:33:55.033 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.033 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f3b9bc:::benchmark_data_smithi188_113957_object134653:head# 2024-04-23T16:33:55.034 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.034 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f61f61:::benchmark_data_smithi188_113957_object34954:head# 2024-04-23T16:33:55.034 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.034 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f64718:::benchmark_data_smithi188_113957_object122008:head# 2024-04-23T16:33:55.034 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.034 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f65d09:::benchmark_data_smithi188_113957_object83858:head# 2024-04-23T16:33:55.034 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.034 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f67489:::benchmark_data_smithi188_113957_object64123:head# 2024-04-23T16:33:55.034 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.034 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f6d393:::benchmark_data_smithi188_113957_object24994:head# 2024-04-23T16:33:55.034 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.034 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f70881:::benchmark_data_smithi188_113957_object66138:head# 2024-04-23T16:33:55.034 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.034 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f70989:::benchmark_data_smithi188_113957_object129724:head# 2024-04-23T16:33:55.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f7178e:::benchmark_data_smithi188_113957_object107925:head# 2024-04-23T16:33:55.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f74e2c:::benchmark_data_smithi188_113957_object61878:head# 2024-04-23T16:33:55.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f8c4d6:::benchmark_data_smithi188_113957_object33472:head# 2024-04-23T16:33:55.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f8fb8c:::benchmark_data_smithi188_113957_object66181:head# 2024-04-23T16:33:55.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f94a3b:::benchmark_data_smithi188_113957_object132533:head# 2024-04-23T16:33:55.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f955cf:::benchmark_data_smithi188_113957_object4996:head# 2024-04-23T16:33:55.035 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.035 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50f9602b:::benchmark_data_smithi188_113957_object15394:head# 2024-04-23T16:33:55.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.036 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50fa38aa:::benchmark_data_smithi188_113957_object91019:head# 2024-04-23T16:33:55.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.036 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50fb7ebf:::benchmark_data_smithi188_113957_object47478:head# 2024-04-23T16:33:55.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.036 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50fbd613:::benchmark_data_smithi188_113957_object23877:head# 2024-04-23T16:33:55.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.036 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50fd45c4:::benchmark_data_smithi188_113957_object135348:head# 2024-04-23T16:33:55.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.036 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50fe0ded:::benchmark_data_smithi188_113957_object83142:head# 2024-04-23T16:33:55.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.036 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50fe46e9:::benchmark_data_smithi188_113957_object137814:head# 2024-04-23T16:33:55.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.036 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50feb14e:::benchmark_data_smithi188_113957_object31410:head# 2024-04-23T16:33:55.036 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.037 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50fed46e:::benchmark_data_smithi188_113957_object73346:head# 2024-04-23T16:33:55.037 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.037 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ff001d:::benchmark_data_smithi188_113957_object126305:head# 2024-04-23T16:33:55.037 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.037 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ff01bb:::benchmark_data_smithi188_113957_object37321:head# 2024-04-23T16:33:55.037 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.037 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:50ff6f32:::benchmark_data_smithi188_113957_object61368:head# 2024-04-23T16:33:55.037 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.037 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510005f0:::benchmark_data_smithi188_113957_object90191:head# 2024-04-23T16:33:55.037 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.037 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51003c57:::benchmark_data_smithi188_113957_object86991:head# 2024-04-23T16:33:55.037 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.037 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510043eb:::benchmark_data_smithi188_113957_object9945:head# 2024-04-23T16:33:55.037 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.037 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5100d392:::benchmark_data_smithi188_113957_object13207:head# 2024-04-23T16:33:55.038 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.038 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510145c6:::benchmark_data_smithi188_113957_object410:head# 2024-04-23T16:33:55.038 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.038 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51016d92:::benchmark_data_smithi188_113957_object62882:head# 2024-04-23T16:33:55.038 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.038 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510184f6:::benchmark_data_smithi188_113957_object140621:head# 2024-04-23T16:33:55.038 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.038 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51023017:::benchmark_data_smithi188_113957_object77586:head# 2024-04-23T16:33:55.038 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.038 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51029c96:::benchmark_data_smithi188_113957_object74997:head# 2024-04-23T16:33:55.038 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.038 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5102e71a:::benchmark_data_smithi188_113957_object141548:head# 2024-04-23T16:33:55.038 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.039 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51031a85:::benchmark_data_smithi188_113957_object98377:head# 2024-04-23T16:33:55.039 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.039 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51032e34:::benchmark_data_smithi188_113957_object69565:head# 2024-04-23T16:33:55.039 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.039 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5103f3b9:::benchmark_data_smithi188_113957_object101840:head# 2024-04-23T16:33:55.039 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.039 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51040329:::benchmark_data_smithi188_113957_object62354:head# 2024-04-23T16:33:55.039 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.039 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510461a5:::benchmark_data_smithi188_113957_object27428:head# 2024-04-23T16:33:55.039 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.039 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5104893f:::benchmark_data_smithi188_113957_object65275:head# 2024-04-23T16:33:55.039 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.039 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5104c415:::benchmark_data_smithi188_113957_object66246:head# 2024-04-23T16:33:55.039 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.040 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51051811:::benchmark_data_smithi188_113957_object41278:head# 2024-04-23T16:33:55.040 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.040 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5105ca81:::benchmark_data_smithi188_113957_object38993:head# 2024-04-23T16:33:55.040 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.040 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5105dec8:::benchmark_data_smithi188_113957_object21053:head# 2024-04-23T16:33:55.040 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.040 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51064de2:::benchmark_data_smithi188_113957_object39516:head# 2024-04-23T16:33:55.040 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.040 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51068702:::benchmark_data_smithi188_113957_object85334:head# 2024-04-23T16:33:55.040 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.040 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5106dd31:::benchmark_data_smithi188_113957_object115601:head# 2024-04-23T16:33:55.040 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.040 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51075a03:::benchmark_data_smithi188_113957_object65030:head# 2024-04-23T16:33:55.040 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.040 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51078f22:::benchmark_data_smithi188_113957_object15074:head# 2024-04-23T16:33:55.041 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.041 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5107f3b6:::benchmark_data_smithi188_113957_object140299:head# 2024-04-23T16:33:55.041 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.041 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51094611:::benchmark_data_smithi188_113957_object11002:head# 2024-04-23T16:33:55.041 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.041 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5109b57f:::benchmark_data_smithi188_113957_object40233:head# 2024-04-23T16:33:55.041 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.041 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510a6d98:::benchmark_data_smithi188_113957_object109558:head# 2024-04-23T16:33:55.041 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.041 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510a8779:::benchmark_data_smithi188_113957_object55303:head# 2024-04-23T16:33:55.041 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.041 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510ab898:::benchmark_data_smithi188_113957_object75271:head# 2024-04-23T16:33:55.041 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.041 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510ac940:::benchmark_data_smithi188_113957_object24735:head# 2024-04-23T16:33:55.041 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.042 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510b2cbc:::benchmark_data_smithi188_113957_object56636:head# 2024-04-23T16:33:55.042 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.042 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510b8057:::benchmark_data_smithi188_113957_object75953:head# 2024-04-23T16:33:55.042 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.042 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510c046a:::benchmark_data_smithi188_113957_object153483:head# 2024-04-23T16:33:55.042 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.042 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510cadcb:::benchmark_data_smithi188_113957_object76799:head# 2024-04-23T16:33:55.042 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.042 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510cbf41:::benchmark_data_smithi188_113957_object87792:head# 2024-04-23T16:33:55.042 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.042 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510cd6ba:::benchmark_data_smithi188_113957_object145397:head# 2024-04-23T16:33:55.042 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.042 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510d356e:::benchmark_data_smithi188_113957_object14103:head# 2024-04-23T16:33:55.043 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.043 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510da4d7:::benchmark_data_smithi188_113957_object126098:head# 2024-04-23T16:33:55.043 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.043 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510e2088:::benchmark_data_smithi188_113957_object5676:head# 2024-04-23T16:33:55.043 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.043 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510e2a7c:::benchmark_data_smithi188_113957_object128128:head# 2024-04-23T16:33:55.043 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.043 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510e6c5d:::benchmark_data_smithi188_113957_object16085:head# 2024-04-23T16:33:55.043 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.043 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510f74dc:::benchmark_data_smithi188_113957_object118606:head# 2024-04-23T16:33:55.043 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.043 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:510f95bf:::benchmark_data_smithi188_113957_object68247:head# 2024-04-23T16:33:55.043 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.043 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5110262f:::benchmark_data_smithi188_113957_object68180:head# 2024-04-23T16:33:55.043 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.044 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5110978f:::benchmark_data_smithi188_113957_object115014:head# 2024-04-23T16:33:55.044 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.044 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5110dfaa:::benchmark_data_smithi188_113957_object25093:head# 2024-04-23T16:33:55.044 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.044 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51117a21:::benchmark_data_smithi188_113957_object56069:head# 2024-04-23T16:33:55.044 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.044 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5111c428:::benchmark_data_smithi188_113957_object86219:head# 2024-04-23T16:33:55.044 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.044 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5111d9af:::benchmark_data_smithi188_113957_object96744:head# 2024-04-23T16:33:55.044 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.044 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5112207d:::benchmark_data_smithi188_113957_object64459:head# 2024-04-23T16:33:55.044 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.044 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51128a5b:::benchmark_data_smithi188_113957_object104030:head# 2024-04-23T16:33:55.044 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.045 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51133f90:::benchmark_data_smithi188_113957_object53277:head# 2024-04-23T16:33:55.045 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.045 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51143ece:::benchmark_data_smithi188_113957_object12672:head# 2024-04-23T16:33:55.045 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.045 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51145a80:::benchmark_data_smithi188_113957_object85168:head# 2024-04-23T16:33:55.045 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.045 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:511483cb:::benchmark_data_smithi188_113957_object23808:head# 2024-04-23T16:33:55.045 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.045 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51149a68:::benchmark_data_smithi188_113957_object50790:head# 2024-04-23T16:33:55.045 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.045 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:511531c1:::benchmark_data_smithi188_113957_object94660:head# 2024-04-23T16:33:55.045 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.045 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51157c11:::benchmark_data_smithi188_113957_object30882:head# 2024-04-23T16:33:55.045 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.046 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5115fa65:::benchmark_data_smithi188_113957_object277:head# 2024-04-23T16:33:55.046 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.046 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:511732bc:::benchmark_data_smithi188_113957_object111729:head# 2024-04-23T16:33:55.046 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.046 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:511818a5:::benchmark_data_smithi188_113957_object85594:head# 2024-04-23T16:33:55.046 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.046 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51189969:::benchmark_data_smithi188_113957_object54924:head# 2024-04-23T16:33:55.046 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.046 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51191e31:::benchmark_data_smithi188_113957_object13863:head# 2024-04-23T16:33:55.046 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.046 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51194119:::benchmark_data_smithi188_113957_object84185:head# 2024-04-23T16:33:55.046 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.046 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51194538:::benchmark_data_smithi188_113957_object54702:head# 2024-04-23T16:33:55.046 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.047 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:511ac0a2:::benchmark_data_smithi188_113957_object88108:head# 2024-04-23T16:33:55.047 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.047 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:511af551:::benchmark_data_smithi188_113957_object115419:head# 2024-04-23T16:33:55.047 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.047 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:511d83b4:::benchmark_data_smithi188_113957_object122005:head# 2024-04-23T16:33:55.047 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.047 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:511d8e5f:::benchmark_data_smithi188_113957_object30102:head# 2024-04-23T16:33:55.047 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.047 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:511d97d8:::benchmark_data_smithi188_113957_object72546:head# 2024-04-23T16:33:55.047 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.047 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:511daa86:::benchmark_data_smithi188_113957_object143906:head# 2024-04-23T16:33:55.047 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.047 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:511e44ca:::benchmark_data_smithi188_113957_object32224:head# 2024-04-23T16:33:55.047 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.047 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51201a09:::benchmark_data_smithi188_113957_object18656:head# 2024-04-23T16:33:55.048 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.048 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51206c43:::benchmark_data_smithi188_113957_object18676:head# 2024-04-23T16:33:55.048 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.048 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51208ef7:::benchmark_data_smithi188_113957_object109144:head# 2024-04-23T16:33:55.048 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.048 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5120e346:::benchmark_data_smithi188_113957_object140710:head# 2024-04-23T16:33:55.048 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.048 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5120f4ce:::benchmark_data_smithi188_113957_object140035:head# 2024-04-23T16:33:55.048 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.048 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:512142cc:::benchmark_data_smithi188_113957_object21349:head# 2024-04-23T16:33:55.048 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.048 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5121763b:::benchmark_data_smithi188_113957_object142409:head# 2024-04-23T16:33:55.048 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.048 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5121844a:::benchmark_data_smithi188_113957_object90441:head# 2024-04-23T16:33:55.049 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.049 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5122607d:::benchmark_data_smithi188_113957_object39496:head# 2024-04-23T16:33:55.049 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.049 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:512292f3:::benchmark_data_smithi188_113957_object131812:head# 2024-04-23T16:33:55.049 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.049 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5122c969:::benchmark_data_smithi188_113957_object87498:head# 2024-04-23T16:33:55.049 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.049 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5122d324:::benchmark_data_smithi188_113957_object42623:head# 2024-04-23T16:33:55.049 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.049 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51233235:::benchmark_data_smithi188_113957_object19898:head# 2024-04-23T16:33:55.049 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.049 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51233550:::benchmark_data_smithi188_113957_object88807:head# 2024-04-23T16:33:55.049 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.050 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5124056d:::benchmark_data_smithi188_113957_object139293:head# 2024-04-23T16:33:55.050 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.050 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51244f4f:::benchmark_data_smithi188_113957_object127266:head# 2024-04-23T16:33:55.050 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.050 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5124ddce:::benchmark_data_smithi188_113957_object75356:head# 2024-04-23T16:33:55.050 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.050 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5124e8dc:::benchmark_data_smithi188_113957_object142678:head# 2024-04-23T16:33:55.050 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.050 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51255be0:::benchmark_data_smithi188_113957_object3165:head# 2024-04-23T16:33:55.050 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.050 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5125615e:::benchmark_data_smithi188_113957_object46072:head# 2024-04-23T16:33:55.050 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.050 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51257954:::benchmark_data_smithi188_113957_object34749:head# 2024-04-23T16:33:55.051 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.051 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51259bc1:::benchmark_data_smithi188_113957_object82730:head# 2024-04-23T16:33:55.051 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.051 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5127242d:::benchmark_data_smithi188_113957_object51320:head# 2024-04-23T16:33:55.051 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.051 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51274da6:::benchmark_data_smithi188_113957_object122340:head# 2024-04-23T16:33:55.051 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.051 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51283cb4:::benchmark_data_smithi188_113957_object39280:head# 2024-04-23T16:33:55.051 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.051 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51294fdb:::benchmark_data_smithi188_113957_object74194:head# 2024-04-23T16:33:55.051 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.051 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5129eae4:::benchmark_data_smithi188_113957_object103249:head# 2024-04-23T16:33:55.051 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.051 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:512a89b9:::benchmark_data_smithi188_113957_object56848:head# 2024-04-23T16:33:55.052 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.052 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:512af9da:::benchmark_data_smithi188_113957_object124256:head# 2024-04-23T16:33:55.052 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.052 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:512afcc8:::benchmark_data_smithi188_113957_object125322:head# 2024-04-23T16:33:55.052 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.052 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:512b3d39:::benchmark_data_smithi188_113957_object50598:head# 2024-04-23T16:33:55.052 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.052 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:512bce26:::benchmark_data_smithi188_113957_object71321:head# 2024-04-23T16:33:55.052 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.052 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:512c0ef9:::benchmark_data_smithi188_113957_object63043:head# 2024-04-23T16:33:55.052 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.052 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:512db80b:::benchmark_data_smithi188_113957_object101051:head# 2024-04-23T16:33:55.052 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.052 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:512dbb52:::benchmark_data_smithi188_113957_object72614:head# 2024-04-23T16:33:55.053 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.053 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:512e8195:::benchmark_data_smithi188_113957_object143893:head# 2024-04-23T16:33:55.053 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.053 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:512e9ffb:::benchmark_data_smithi188_113957_object110478:head# 2024-04-23T16:33:55.053 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.053 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:512f6d32:::benchmark_data_smithi188_113957_object46856:head# 2024-04-23T16:33:55.053 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.053 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:512fa26e:::benchmark_data_smithi188_113957_object152637:head# 2024-04-23T16:33:55.053 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.053 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51302247:::benchmark_data_smithi188_113957_object74493:head# 2024-04-23T16:33:55.053 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.053 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:513081ee:::benchmark_data_smithi188_113957_object132737:head# 2024-04-23T16:33:55.053 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.053 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5130b0e6:::benchmark_data_smithi188_113957_object105424:head# 2024-04-23T16:33:55.054 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.054 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51314a00:::benchmark_data_smithi188_113957_object80459:head# 2024-04-23T16:33:55.054 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.054 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5131d43d:::benchmark_data_smithi188_113957_object135872:head# 2024-04-23T16:33:55.054 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.054 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5132de20:::benchmark_data_smithi188_113957_object90375:head# 2024-04-23T16:33:55.054 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.054 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5132e266:::benchmark_data_smithi188_113957_object79596:head# 2024-04-23T16:33:55.054 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.054 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5132e73c:::benchmark_data_smithi188_113957_object13964:head# 2024-04-23T16:33:55.054 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.054 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5133fe96:::benchmark_data_smithi188_113957_object50001:head# 2024-04-23T16:33:55.054 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.054 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5134d42b:::benchmark_data_smithi188_113957_object132156:head# 2024-04-23T16:33:55.055 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.055 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5134e66e:::benchmark_data_smithi188_113957_object51642:head# 2024-04-23T16:33:55.055 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.055 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5134efbf:::benchmark_data_smithi188_113957_object51460:head# 2024-04-23T16:33:55.055 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.055 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51352326:::benchmark_data_smithi188_113957_object68249:head# 2024-04-23T16:33:55.055 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.055 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51352406:::benchmark_data_smithi188_113957_object53781:head# 2024-04-23T16:33:55.055 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.055 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51352615:::benchmark_data_smithi188_113957_object27478:head# 2024-04-23T16:33:55.055 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.055 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5135a3fc:::benchmark_data_smithi188_113957_object74961:head# 2024-04-23T16:33:55.055 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.055 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5135c2f8:::benchmark_data_smithi188_113957_object149723:head# 2024-04-23T16:33:55.055 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.056 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:513714e0:::benchmark_data_smithi188_113957_object109890:head# 2024-04-23T16:33:55.056 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.056 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51373ab8:::benchmark_data_smithi188_113957_object145772:head# 2024-04-23T16:33:55.056 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.056 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51380db1:::benchmark_data_smithi188_113957_object49834:head# 2024-04-23T16:33:55.056 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.056 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51381df7:::benchmark_data_smithi188_113957_object100972:head# 2024-04-23T16:33:55.056 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.056 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5138a589:::benchmark_data_smithi188_113957_object13463:head# 2024-04-23T16:33:55.056 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.056 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5138c74c:::benchmark_data_smithi188_113957_object132071:head# 2024-04-23T16:33:55.056 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.056 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51398bfa:::benchmark_data_smithi188_113957_object1512:head# 2024-04-23T16:33:55.056 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.057 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:513a6fa3:::benchmark_data_smithi188_113957_object29810:head# 2024-04-23T16:33:55.057 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.057 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:513bbeb9:::benchmark_data_smithi188_113957_object152715:head# 2024-04-23T16:33:55.057 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.057 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:513ceb50:::benchmark_data_smithi188_113957_object97374:head# 2024-04-23T16:33:55.057 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.057 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:513cf96b:::benchmark_data_smithi188_113957_object117743:head# 2024-04-23T16:33:55.057 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.057 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:513d991b:::benchmark_data_smithi188_113957_object126713:head# 2024-04-23T16:33:55.057 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.057 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:513da6b2:::benchmark_data_smithi188_113957_object35433:head# 2024-04-23T16:33:55.057 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.057 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:513f424b:::benchmark_data_smithi188_113957_object32023:head# 2024-04-23T16:33:55.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51400100:::benchmark_data_smithi188_113957_object7123:head# 2024-04-23T16:33:55.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5140560d:::benchmark_data_smithi188_113957_object136154:head# 2024-04-23T16:33:55.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5141b158:::benchmark_data_smithi188_113957_object123268:head# 2024-04-23T16:33:55.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5141d8f0:::benchmark_data_smithi188_113957_object118898:head# 2024-04-23T16:33:55.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5141ee2a:::benchmark_data_smithi188_113957_object44423:head# 2024-04-23T16:33:55.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5141f31f:::benchmark_data_smithi188_113957_object114286:head# 2024-04-23T16:33:55.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5142530b:::benchmark_data_smithi188_113957_object63735:head# 2024-04-23T16:33:55.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5144f7cd:::benchmark_data_smithi188_113957_object82352:head# 2024-04-23T16:33:55.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5145af04:::benchmark_data_smithi188_113957_object92384:head# 2024-04-23T16:33:55.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51460caf:::benchmark_data_smithi188_113957_object53334:head# 2024-04-23T16:33:55.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51477fdd:::benchmark_data_smithi188_113957_object129895:head# 2024-04-23T16:33:55.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5147ea08:::benchmark_data_smithi188_113957_object34364:head# 2024-04-23T16:33:55.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51481246:::benchmark_data_smithi188_113957_object21810:head# 2024-04-23T16:33:55.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.060 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51489347:::benchmark_data_smithi188_113957_object105162:head# 2024-04-23T16:33:55.060 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.060 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5148cd8f:::benchmark_data_smithi188_113957_object153456:head# 2024-04-23T16:33:55.060 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.060 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5149635f:::benchmark_data_smithi188_113957_object77369:head# 2024-04-23T16:33:55.060 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.060 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51497d38:::benchmark_data_smithi188_113957_object14066:head# 2024-04-23T16:33:55.060 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.060 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:514af384:::benchmark_data_smithi188_113957_object135132:head# 2024-04-23T16:33:55.060 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.060 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:514bc252:::benchmark_data_smithi188_113957_object68869:head# 2024-04-23T16:33:55.060 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.060 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:514bc7bc:::benchmark_data_smithi188_113957_object70004:head# 2024-04-23T16:33:55.060 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.061 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:514bd046:::benchmark_data_smithi188_113957_object54651:head# 2024-04-23T16:33:55.061 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.061 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:514cab87:::benchmark_data_smithi188_113957_object56294:head# 2024-04-23T16:33:55.061 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.061 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:514f69a8:::benchmark_data_smithi188_113957_object136753:head# 2024-04-23T16:33:55.061 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.061 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51504834:::benchmark_data_smithi188_113957_object22954:head# 2024-04-23T16:33:55.061 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.061 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5150e706:::benchmark_data_smithi188_113957_object70937:head# 2024-04-23T16:33:55.061 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.061 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51512ee7:::benchmark_data_smithi188_113957_object79434:head# 2024-04-23T16:33:55.061 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.061 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5151dacf:::benchmark_data_smithi188_113957_object66579:head# 2024-04-23T16:33:55.061 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.062 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5152377d:::benchmark_data_smithi188_113957_object31780:head# 2024-04-23T16:33:55.062 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.062 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51528525:::benchmark_data_smithi188_113957_object135571:head# 2024-04-23T16:33:55.062 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.062 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51539e11:::benchmark_data_smithi188_113957_object139372:head# 2024-04-23T16:33:55.062 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.062 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:515479fa:::benchmark_data_smithi188_113957_object113027:head# 2024-04-23T16:33:55.062 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.062 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5154a711:::benchmark_data_smithi188_113957_object81377:head# 2024-04-23T16:33:55.062 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.062 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51554b09:::benchmark_data_smithi188_113957_object56810:head# 2024-04-23T16:33:55.062 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.062 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51559c78:::benchmark_data_smithi188_113957_object118824:head# 2024-04-23T16:33:55.063 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.063 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51562c44:::benchmark_data_smithi188_113957_object55884:head# 2024-04-23T16:33:55.063 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.063 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51574da2:::benchmark_data_smithi188_113957_object113603:head# 2024-04-23T16:33:55.063 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.063 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51586e9a:::benchmark_data_smithi188_113957_object4443:head# 2024-04-23T16:33:55.063 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.063 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51587027:::benchmark_data_smithi188_113957_object44207:head# 2024-04-23T16:33:55.063 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.063 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51596a23:::benchmark_data_smithi188_113957_object43263:head# 2024-04-23T16:33:55.063 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.063 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5159ccb3:::benchmark_data_smithi188_113957_object113576:head# 2024-04-23T16:33:55.063 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.063 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:515ab018:::benchmark_data_smithi188_113957_object77132:head# 2024-04-23T16:33:55.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:515b38a2:::benchmark_data_smithi188_113957_object74645:head# 2024-04-23T16:33:55.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:515ba7e7:::benchmark_data_smithi188_113957_object122531:head# 2024-04-23T16:33:55.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:515cbf93:::benchmark_data_smithi188_113957_object66428:head# 2024-04-23T16:33:55.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:515d4d12:::benchmark_data_smithi188_113957_object34154:head# 2024-04-23T16:33:55.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:515d6a2c:::benchmark_data_smithi188_113957_object74045:head# 2024-04-23T16:33:55.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:515d7a36:::benchmark_data_smithi188_113957_object46997:head# 2024-04-23T16:33:55.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:515ef5fc:::benchmark_data_smithi188_113957_object143415:head# 2024-04-23T16:33:55.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:515f88b4:::benchmark_data_smithi188_113957_object98438:head# 2024-04-23T16:33:55.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:515fabd5:::benchmark_data_smithi188_113957_object113895:head# 2024-04-23T16:33:55.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:515fb279:::benchmark_data_smithi188_113957_object6989:head# 2024-04-23T16:33:55.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51607da9:::benchmark_data_smithi188_113957_object134251:head# 2024-04-23T16:33:55.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5160ac5c:::benchmark_data_smithi188_113957_object1788:head# 2024-04-23T16:33:55.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51611edb:::benchmark_data_smithi188_113957_object36696:head# 2024-04-23T16:33:55.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516172be:::benchmark_data_smithi188_113957_object61157:head# 2024-04-23T16:33:55.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51623526:::benchmark_data_smithi188_113957_object135856:head# 2024-04-23T16:33:55.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51625d59:::benchmark_data_smithi188_113957_object30810:head# 2024-04-23T16:33:55.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51626298:::benchmark_data_smithi188_113957_object80762:head# 2024-04-23T16:33:55.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51637142:::benchmark_data_smithi188_113957_object101579:head# 2024-04-23T16:33:55.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51637dac:::benchmark_data_smithi188_113957_object112656:head# 2024-04-23T16:33:55.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5163863b:::benchmark_data_smithi188_113957_object115977:head# 2024-04-23T16:33:55.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.067 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5163a7b5:::benchmark_data_smithi188_113957_object30227:head# 2024-04-23T16:33:55.067 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.067 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5163b1f0:::benchmark_data_smithi188_113957_object89834:head# 2024-04-23T16:33:55.067 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.067 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5163eaef:::benchmark_data_smithi188_113957_object75303:head# 2024-04-23T16:33:55.067 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.067 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51646b87:::benchmark_data_smithi188_113957_object55608:head# 2024-04-23T16:33:55.067 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.067 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51647a2a:::benchmark_data_smithi188_113957_object81974:head# 2024-04-23T16:33:55.067 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.067 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51649c38:::benchmark_data_smithi188_113957_object96528:head# 2024-04-23T16:33:55.067 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5164f57a:::benchmark_data_smithi188_113957_object129280:head# 2024-04-23T16:33:55.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51652114:::benchmark_data_smithi188_113957_object150318:head# 2024-04-23T16:33:55.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51655bd6:::benchmark_data_smithi188_113957_object38606:head# 2024-04-23T16:33:55.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516569fc:::benchmark_data_smithi188_113957_object128717:head# 2024-04-23T16:33:55.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516629ca:::benchmark_data_smithi188_113957_object31725:head# 2024-04-23T16:33:55.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516793c6:::benchmark_data_smithi188_113957_object63301:head# 2024-04-23T16:33:55.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5167e475:::benchmark_data_smithi188_113957_object141901:head# 2024-04-23T16:33:55.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5167f7e5:::benchmark_data_smithi188_113957_object25150:head# 2024-04-23T16:33:55.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5168b96b:::benchmark_data_smithi188_113957_object16669:head# 2024-04-23T16:33:55.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51691863:::benchmark_data_smithi188_113957_object103938:head# 2024-04-23T16:33:55.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516938a2:::benchmark_data_smithi188_113957_object96164:head# 2024-04-23T16:33:55.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51699208:::benchmark_data_smithi188_113957_object13802:head# 2024-04-23T16:33:55.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5169c2e6:::benchmark_data_smithi188_113957_object87450:head# 2024-04-23T16:33:55.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516a2fd0:::benchmark_data_smithi188_113957_object79656:head# 2024-04-23T16:33:55.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516a3d6c:::benchmark_data_smithi188_113957_object45642:head# 2024-04-23T16:33:55.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516a51e0:::benchmark_data_smithi188_113957_object62784:head# 2024-04-23T16:33:55.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516a6d7f:::benchmark_data_smithi188_113957_object31724:head# 2024-04-23T16:33:55.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516acbff:::benchmark_data_smithi188_113957_object65256:head# 2024-04-23T16:33:55.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516b194f:::benchmark_data_smithi188_113957_object5910:head# 2024-04-23T16:33:55.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516b492a:::benchmark_data_smithi188_113957_object133528:head# 2024-04-23T16:33:55.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516b4bd3:::benchmark_data_smithi188_113957_object110741:head# 2024-04-23T16:33:55.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516b77fc:::benchmark_data_smithi188_113957_object55570:head# 2024-04-23T16:33:55.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516ba6a6:::benchmark_data_smithi188_113957_object96008:head# 2024-04-23T16:33:55.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516c46b7:::benchmark_data_smithi188_113957_object57986:head# 2024-04-23T16:33:55.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516c69e1:::benchmark_data_smithi188_113957_object5276:head# 2024-04-23T16:33:55.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516dfc05:::benchmark_data_smithi188_113957_object34623:head# 2024-04-23T16:33:55.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516ea325:::benchmark_data_smithi188_113957_object125233:head# 2024-04-23T16:33:55.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516eccee:::benchmark_data_smithi188_113957_object21648:head# 2024-04-23T16:33:55.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516ee77e:::benchmark_data_smithi188_113957_object113808:head# 2024-04-23T16:33:55.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516f350e:::benchmark_data_smithi188_113957_object104582:head# 2024-04-23T16:33:55.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516f4eb3:::benchmark_data_smithi188_113957_object28959:head# 2024-04-23T16:33:55.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516f9c33:::benchmark_data_smithi188_113957_object98967:head# 2024-04-23T16:33:55.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:516fc2a9:::benchmark_data_smithi188_113957_object87935:head# 2024-04-23T16:33:55.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51706449:::benchmark_data_smithi188_113957_object135215:head# 2024-04-23T16:33:55.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:517133d6:::benchmark_data_smithi188_113957_object14862:head# 2024-04-23T16:33:55.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51719875:::benchmark_data_smithi188_113957_object88643:head# 2024-04-23T16:33:55.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51719c1c:::benchmark_data_smithi188_113957_object44564:head# 2024-04-23T16:33:55.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5171bede:::benchmark_data_smithi188_113957_object32374:head# 2024-04-23T16:33:55.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5171e6bd:::benchmark_data_smithi188_113957_object143506:head# 2024-04-23T16:33:55.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5172a0ed:::benchmark_data_smithi188_113957_object22684:head# 2024-04-23T16:33:55.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.120 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51733793:::benchmark_data_smithi188_113957_object131402:head# 2024-04-23T16:33:55.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.120 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51737773:::benchmark_data_smithi188_113957_object142741:head# 2024-04-23T16:33:55.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.120 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5173d412:::benchmark_data_smithi188_113957_object27081:head# 2024-04-23T16:33:55.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.120 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5174ddf6:::benchmark_data_smithi188_113957_object44515:head# 2024-04-23T16:33:55.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.120 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5174fb42:::benchmark_data_smithi188_113957_object139396:head# 2024-04-23T16:33:55.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.120 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51751a25:::benchmark_data_smithi188_113957_object10781:head# 2024-04-23T16:33:55.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.120 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:517537b4:::benchmark_data_smithi188_113957_object143012:head# 2024-04-23T16:33:55.121 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.121 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5175f139:::benchmark_data_smithi188_113957_object45296:head# 2024-04-23T16:33:55.121 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.121 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51764eb7:::benchmark_data_smithi188_113957_object92364:head# 2024-04-23T16:33:55.121 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.121 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:517669ca:::benchmark_data_smithi188_113957_object45792:head# 2024-04-23T16:33:55.121 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.121 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5176bbf0:::benchmark_data_smithi188_113957_object53655:head# 2024-04-23T16:33:55.121 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.121 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5176c24f:::benchmark_data_smithi188_113957_object63761:head# 2024-04-23T16:33:55.121 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.121 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51776500:::benchmark_data_smithi188_113957_object78190:head# 2024-04-23T16:33:55.121 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.121 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5177e648:::benchmark_data_smithi188_113957_object1647:head# 2024-04-23T16:33:55.122 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.122 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51780394:::benchmark_data_smithi188_113957_object74670:head# 2024-04-23T16:33:55.122 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.122 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5178751a:::benchmark_data_smithi188_113957_object52547:head# 2024-04-23T16:33:55.122 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.122 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5178840c:::benchmark_data_smithi188_113957_object135634:head# 2024-04-23T16:33:55.122 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.122 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5178afbd:::benchmark_data_smithi188_113957_object92226:head# 2024-04-23T16:33:55.122 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.122 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5179712e:::benchmark_data_smithi188_113957_object7166:head# 2024-04-23T16:33:55.122 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.122 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5179a92e:::benchmark_data_smithi188_113957_object86457:head# 2024-04-23T16:33:55.122 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.123 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:517a65b3:::benchmark_data_smithi188_113957_object120344:head# 2024-04-23T16:33:55.123 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.123 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:517a73a5:::benchmark_data_smithi188_113957_object148399:head# 2024-04-23T16:33:55.123 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.123 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:517a9d4d:::benchmark_data_smithi188_113957_object94685:head# 2024-04-23T16:33:55.123 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.123 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:517b2141:::benchmark_data_smithi188_113957_object4879:head# 2024-04-23T16:33:55.123 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.123 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:517d247f:::benchmark_data_smithi188_113957_object127538:head# 2024-04-23T16:33:55.123 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.123 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:517d3e07:::benchmark_data_smithi188_113957_object137789:head# 2024-04-23T16:33:55.123 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.123 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:517d95e5:::benchmark_data_smithi188_113957_object121528:head# 2024-04-23T16:33:55.123 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.123 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:517e5bc4:::benchmark_data_smithi188_113957_object41017:head# 2024-04-23T16:33:55.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:517fb4a4:::benchmark_data_smithi188_113957_object4230:head# 2024-04-23T16:33:55.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:518029e7:::benchmark_data_smithi188_113957_object102761:head# 2024-04-23T16:33:55.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5180b936:::benchmark_data_smithi188_113957_object109463:head# 2024-04-23T16:33:55.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5181b58f:::benchmark_data_smithi188_113957_object128270:head# 2024-04-23T16:33:55.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5181fd47:::benchmark_data_smithi188_113957_object46375:head# 2024-04-23T16:33:55.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51820eb8:::benchmark_data_smithi188_113957_object42287:head# 2024-04-23T16:33:55.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51827f55:::benchmark_data_smithi188_113957_object72303:head# 2024-04-23T16:33:55.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51830348:::benchmark_data_smithi188_113957_object49014:head# 2024-04-23T16:33:55.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5183a603:::benchmark_data_smithi188_113957_object103232:head# 2024-04-23T16:33:55.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5184a0c5:::benchmark_data_smithi188_113957_object60120:head# 2024-04-23T16:33:55.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5184aadd:::benchmark_data_smithi188_113957_object144496:head# 2024-04-23T16:33:55.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5184f5d0:::benchmark_data_smithi188_113957_object77430:head# 2024-04-23T16:33:55.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5184f6f8:::benchmark_data_smithi188_113957_object137076:head# 2024-04-23T16:33:55.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5187062f:::benchmark_data_smithi188_113957_object151360:head# 2024-04-23T16:33:55.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5187177b:::benchmark_data_smithi188_113957_object23135:head# 2024-04-23T16:33:55.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5187684d:::benchmark_data_smithi188_113957_object102776:head# 2024-04-23T16:33:55.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:518769ca:::benchmark_data_smithi188_113957_object43483:head# 2024-04-23T16:33:55.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51879449:::benchmark_data_smithi188_113957_object96429:head# 2024-04-23T16:33:55.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5187c191:::benchmark_data_smithi188_113957_object142873:head# 2024-04-23T16:33:55.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51884886:::benchmark_data_smithi188_113957_object46851:head# 2024-04-23T16:33:55.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5188d6cf:::benchmark_data_smithi188_113957_object43240:head# 2024-04-23T16:33:55.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:518968c6:::benchmark_data_smithi188_113957_object41205:head# 2024-04-23T16:33:55.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51898fb7:::benchmark_data_smithi188_113957_object152595:head# 2024-04-23T16:33:55.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5189901b:::benchmark_data_smithi188_113957_object136777:head# 2024-04-23T16:33:55.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5189b96c:::benchmark_data_smithi188_113957_object71306:head# 2024-04-23T16:33:55.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5189eee4:::benchmark_data_smithi188_113957_object123442:head# 2024-04-23T16:33:55.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:518a5d11:::benchmark_data_smithi188_113957_object108504:head# 2024-04-23T16:33:55.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:518aa553:::benchmark_data_smithi188_113957_object20925:head# 2024-04-23T16:33:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:518afd0c:::benchmark_data_smithi188_113957_object137812:head# 2024-04-23T16:33:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:518b1efb:::benchmark_data_smithi188_113957_object4072:head# 2024-04-23T16:33:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:518c16f6:::benchmark_data_smithi188_113957_object18331:head# 2024-04-23T16:33:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:518c8c49:::benchmark_data_smithi188_113957_object93:head# 2024-04-23T16:33:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:518d5448:::benchmark_data_smithi188_113957_object44449:head# 2024-04-23T16:33:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:518ecd68:::benchmark_data_smithi188_113957_object51160:head# 2024-04-23T16:33:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:518f220c:::benchmark_data_smithi188_113957_object133243:head# 2024-04-23T16:33:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:518f571b:::benchmark_data_smithi188_113957_object55808:head# 2024-04-23T16:33:55.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:518f6c1b:::benchmark_data_smithi188_113957_object88785:head# 2024-04-23T16:33:55.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519024de:::benchmark_data_smithi188_113957_object129451:head# 2024-04-23T16:33:55.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51903687:::benchmark_data_smithi188_113957_object102921:head# 2024-04-23T16:33:55.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5190bfe2:::benchmark_data_smithi188_113957_object75294:head# 2024-04-23T16:33:55.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5190e700:::benchmark_data_smithi188_113957_object112700:head# 2024-04-23T16:33:55.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51916ad7:::benchmark_data_smithi188_113957_object93300:head# 2024-04-23T16:33:55.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51921cdf:::benchmark_data_smithi188_113957_object2844:head# 2024-04-23T16:33:55.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519244e3:::benchmark_data_smithi188_113957_object109738:head# 2024-04-23T16:33:55.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51928b34:::benchmark_data_smithi188_113957_object140131:head# 2024-04-23T16:33:55.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51931d17:::benchmark_data_smithi188_113957_object38473:head# 2024-04-23T16:33:55.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5193daa3:::benchmark_data_smithi188_113957_object1657:head# 2024-04-23T16:33:55.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5193fbf9:::benchmark_data_smithi188_113957_object86285:head# 2024-04-23T16:33:55.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51945567:::benchmark_data_smithi188_113957_object91316:head# 2024-04-23T16:33:55.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5195beb2:::benchmark_data_smithi188_113957_object8281:head# 2024-04-23T16:33:55.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5195ed93:::benchmark_data_smithi188_113957_object111735:head# 2024-04-23T16:33:55.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51969207:::benchmark_data_smithi188_113957_object6341:head# 2024-04-23T16:33:55.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51969c87:::benchmark_data_smithi188_113957_object58374:head# 2024-04-23T16:33:55.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5196b6c8:::benchmark_data_smithi188_113957_object78100:head# 2024-04-23T16:33:55.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5196b710:::benchmark_data_smithi188_113957_object102944:head# 2024-04-23T16:33:55.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51974002:::benchmark_data_smithi188_113957_object67904:head# 2024-04-23T16:33:55.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519744a3:::benchmark_data_smithi188_113957_object48142:head# 2024-04-23T16:33:55.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51983351:::benchmark_data_smithi188_113957_object67343:head# 2024-04-23T16:33:55.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5198b3c3:::benchmark_data_smithi188_113957_object84487:head# 2024-04-23T16:33:55.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51991e94:::benchmark_data_smithi188_113957_object112812:head# 2024-04-23T16:33:55.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51994ef1:::benchmark_data_smithi188_113957_object128097:head# 2024-04-23T16:33:55.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5199c365:::benchmark_data_smithi188_113957_object32079:head# 2024-04-23T16:33:55.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519a4c66:::benchmark_data_smithi188_113957_object76753:head# 2024-04-23T16:33:55.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519a63e3:::benchmark_data_smithi188_113957_object126433:head# 2024-04-23T16:33:55.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519a71bb:::benchmark_data_smithi188_113957_object148848:head# 2024-04-23T16:33:55.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519ab99b:::benchmark_data_smithi188_113957_object148915:head# 2024-04-23T16:33:55.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519b30a1:::benchmark_data_smithi188_113957_object54736:head# 2024-04-23T16:33:55.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519b3655:::benchmark_data_smithi188_113957_object26825:head# 2024-04-23T16:33:55.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519baa39:::benchmark_data_smithi188_113957_object65843:head# 2024-04-23T16:33:55.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519bcbda:::benchmark_data_smithi188_113957_object30015:head# 2024-04-23T16:33:55.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519d274e:::benchmark_data_smithi188_113957_object32234:head# 2024-04-23T16:33:55.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519df366:::benchmark_data_smithi188_113957_object42598:head# 2024-04-23T16:33:55.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519e357b:::benchmark_data_smithi188_113957_object131491:head# 2024-04-23T16:33:55.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519f6388:::benchmark_data_smithi188_113957_object68978:head# 2024-04-23T16:33:55.134 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.134 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519f8474:::benchmark_data_smithi188_113957_object19150:head# 2024-04-23T16:33:55.134 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.134 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519f9a08:::benchmark_data_smithi188_113957_object146495:head# 2024-04-23T16:33:55.134 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.134 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519fa244:::benchmark_data_smithi188_113957_object125928:head# 2024-04-23T16:33:55.134 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.134 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519fc613:::benchmark_data_smithi188_113957_object131415:head# 2024-04-23T16:33:55.134 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.134 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:519fed97:::benchmark_data_smithi188_113957_object544:head# 2024-04-23T16:33:55.134 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.134 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a05bfd:::benchmark_data_smithi188_113957_object132889:head# 2024-04-23T16:33:55.134 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.134 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a0abc9:::benchmark_data_smithi188_113957_object13987:head# 2024-04-23T16:33:55.135 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.135 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a16a81:::benchmark_data_smithi188_113957_object85277:head# 2024-04-23T16:33:55.135 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.135 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a1d138:::benchmark_data_smithi188_113957_object51983:head# 2024-04-23T16:33:55.135 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.135 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a20795:::benchmark_data_smithi188_113957_object8336:head# 2024-04-23T16:33:55.135 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.135 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a272aa:::benchmark_data_smithi188_113957_object118933:head# 2024-04-23T16:33:55.135 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.135 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a27329:::benchmark_data_smithi188_113957_object40053:head# 2024-04-23T16:33:55.135 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.135 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a35106:::benchmark_data_smithi188_113957_object50330:head# 2024-04-23T16:33:55.135 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.135 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a41466:::benchmark_data_smithi188_113957_object7158:head# 2024-04-23T16:33:55.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a4309b:::benchmark_data_smithi188_113957_object111732:head# 2024-04-23T16:33:55.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a4d544:::benchmark_data_smithi188_113957_object64509:head# 2024-04-23T16:33:55.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a58dcf:::benchmark_data_smithi188_113957_object2324:head# 2024-04-23T16:33:55.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a5c17f:::benchmark_data_smithi188_113957_object92048:head# 2024-04-23T16:33:55.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a60bc7:::benchmark_data_smithi188_113957_object89080:head# 2024-04-23T16:33:55.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a6c41b:::benchmark_data_smithi188_113957_object45132:head# 2024-04-23T16:33:55.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a7022d:::benchmark_data_smithi188_113957_object2455:head# 2024-04-23T16:33:55.136 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.136 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a73ad2:::benchmark_data_smithi188_113957_object49850:head# 2024-04-23T16:33:55.137 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.137 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a749fd:::benchmark_data_smithi188_113957_object21312:head# 2024-04-23T16:33:55.137 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.137 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a75188:::benchmark_data_smithi188_113957_object11189:head# 2024-04-23T16:33:55.137 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.137 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a77f00:::benchmark_data_smithi188_113957_object137879:head# 2024-04-23T16:33:55.137 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.137 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a7ab9a:::benchmark_data_smithi188_113957_object4874:head# 2024-04-23T16:33:55.137 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.137 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a7ed62:::benchmark_data_smithi188_113957_object53083:head# 2024-04-23T16:33:55.137 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.137 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a824da:::benchmark_data_smithi188_113957_object12770:head# 2024-04-23T16:33:55.137 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.137 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a8cee0:::benchmark_data_smithi188_113957_object135198:head# 2024-04-23T16:33:55.137 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.138 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a8e25c:::benchmark_data_smithi188_113957_object34077:head# 2024-04-23T16:33:55.138 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.138 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a9cffa:::benchmark_data_smithi188_113957_object78142:head# 2024-04-23T16:33:55.138 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.138 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51a9d6eb:::benchmark_data_smithi188_113957_object128413:head# 2024-04-23T16:33:55.138 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.138 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51aa352f:::benchmark_data_smithi188_113957_object81317:head# 2024-04-23T16:33:55.138 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.138 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51aa598d:::benchmark_data_smithi188_113957_object20134:head# 2024-04-23T16:33:55.138 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.138 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51aaf659:::benchmark_data_smithi188_113957_object146224:head# 2024-04-23T16:33:55.138 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.138 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ab3333:::benchmark_data_smithi188_113957_object149420:head# 2024-04-23T16:33:55.138 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ab7979:::benchmark_data_smithi188_113957_object106519:head# 2024-04-23T16:33:55.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ab7fe9:::benchmark_data_smithi188_113957_object111739:head# 2024-04-23T16:33:55.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ab9a41:::benchmark_data_smithi188_113957_object141236:head# 2024-04-23T16:33:55.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ac75c1:::benchmark_data_smithi188_113957_object98150:head# 2024-04-23T16:33:55.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ac7af5:::benchmark_data_smithi188_113957_object125914:head# 2024-04-23T16:33:55.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ac9648:::benchmark_data_smithi188_113957_object133691:head# 2024-04-23T16:33:55.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51acb528:::benchmark_data_smithi188_113957_object1922:head# 2024-04-23T16:33:55.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ad026e:::benchmark_data_smithi188_113957_object99976:head# 2024-04-23T16:33:55.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51af2dcc:::benchmark_data_smithi188_113957_object50702:head# 2024-04-23T16:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51afc693:::benchmark_data_smithi188_113957_object54733:head# 2024-04-23T16:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51afe272:::benchmark_data_smithi188_113957_object66237:head# 2024-04-23T16:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b01eaa:::benchmark_data_smithi188_113957_object26081:head# 2024-04-23T16:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b090b7:::benchmark_data_smithi188_113957_object99858:head# 2024-04-23T16:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b1f036:::benchmark_data_smithi188_113957_object134572:head# 2024-04-23T16:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b23ba7:::benchmark_data_smithi188_113957_object43490:head# 2024-04-23T16:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b25892:::benchmark_data_smithi188_113957_object81066:head# 2024-04-23T16:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b26660:::benchmark_data_smithi188_113957_object40557:head# 2024-04-23T16:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b29dec:::benchmark_data_smithi188_113957_object32028:head# 2024-04-23T16:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b30c64:::benchmark_data_smithi188_113957_object75739:head# 2024-04-23T16:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b337bc:::benchmark_data_smithi188_113957_object37647:head# 2024-04-23T16:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b46143:::benchmark_data_smithi188_113957_object27892:head# 2024-04-23T16:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b48b2f:::benchmark_data_smithi188_113957_object50684:head# 2024-04-23T16:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b49fd1:::benchmark_data_smithi188_113957_object40126:head# 2024-04-23T16:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b522ee:::benchmark_data_smithi188_113957_object20577:head# 2024-04-23T16:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b527bd:::benchmark_data_smithi188_113957_object120873:head# 2024-04-23T16:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b5817e:::benchmark_data_smithi188_113957_object42960:head# 2024-04-23T16:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b5f103:::benchmark_data_smithi188_113957_object34981:head# 2024-04-23T16:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b7c1c1:::benchmark_data_smithi188_113957_object118663:head# 2024-04-23T16:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b857b5:::benchmark_data_smithi188_113957_object86942:head# 2024-04-23T16:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b87fc7:::benchmark_data_smithi188_113957_object103498:head# 2024-04-23T16:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b8ef89:::benchmark_data_smithi188_113957_object97494:head# 2024-04-23T16:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b92078:::benchmark_data_smithi188_113957_object153243:head# 2024-04-23T16:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b9329c:::benchmark_data_smithi188_113957_object14820:head# 2024-04-23T16:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51b9cca1:::benchmark_data_smithi188_113957_object17687:head# 2024-04-23T16:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ba0bc5:::benchmark_data_smithi188_113957_object130879:head# 2024-04-23T16:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ba84f8:::benchmark_data_smithi188_113957_object90599:head# 2024-04-23T16:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51baa930:::benchmark_data_smithi188_113957_object86194:head# 2024-04-23T16:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51bab594:::benchmark_data_smithi188_113957_object108850:head# 2024-04-23T16:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51bad794:::benchmark_data_smithi188_113957_object106340:head# 2024-04-23T16:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51bba13d:::benchmark_data_smithi188_113957_object112607:head# 2024-04-23T16:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51bc6a75:::benchmark_data_smithi188_113957_object60496:head# 2024-04-23T16:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51bcd2cd:::benchmark_data_smithi188_113957_object93973:head# 2024-04-23T16:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51bd729a:::benchmark_data_smithi188_113957_object135674:head# 2024-04-23T16:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51bd7e45:::benchmark_data_smithi188_113957_object1669:head# 2024-04-23T16:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51bea053:::benchmark_data_smithi188_113957_object20325:head# 2024-04-23T16:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51beb405:::benchmark_data_smithi188_113957_object111455:head# 2024-04-23T16:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51bed45b:::benchmark_data_smithi188_113957_object111895:head# 2024-04-23T16:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51bef6bc:::benchmark_data_smithi188_113957_object97952:head# 2024-04-23T16:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51bfe7b3:::benchmark_data_smithi188_113957_object64745:head# 2024-04-23T16:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c0259a:::benchmark_data_smithi188_113957_object9088:head# 2024-04-23T16:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c0a5a7:::benchmark_data_smithi188_113957_object124368:head# 2024-04-23T16:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c0c329:::benchmark_data_smithi188_113957_object152187:head# 2024-04-23T16:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c0d1ff:::benchmark_data_smithi188_113957_object111328:head# 2024-04-23T16:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c1776c:::benchmark_data_smithi188_113957_object89967:head# 2024-04-23T16:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c1dfab:::benchmark_data_smithi188_113957_object45466:head# 2024-04-23T16:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c1ed61:::benchmark_data_smithi188_113957_object60691:head# 2024-04-23T16:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c3e546:::benchmark_data_smithi188_113957_object116410:head# 2024-04-23T16:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c4db81:::benchmark_data_smithi188_113957_object139259:head# 2024-04-23T16:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c541c9:::benchmark_data_smithi188_113957_object35254:head# 2024-04-23T16:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c544cf:::benchmark_data_smithi188_113957_object71559:head# 2024-04-23T16:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c559ac:::benchmark_data_smithi188_113957_object12872:head# 2024-04-23T16:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c5610b:::benchmark_data_smithi188_113957_object83062:head# 2024-04-23T16:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c5c63d:::benchmark_data_smithi188_113957_object75902:head# 2024-04-23T16:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c618da:::benchmark_data_smithi188_113957_object85740:head# 2024-04-23T16:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c64e50:::benchmark_data_smithi188_113957_object14080:head# 2024-04-23T16:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c66df8:::benchmark_data_smithi188_113957_object95468:head# 2024-04-23T16:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c6b200:::benchmark_data_smithi188_113957_object2374:head# 2024-04-23T16:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c7147b:::benchmark_data_smithi188_113957_object95774:head# 2024-04-23T16:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c7a40a:::benchmark_data_smithi188_113957_object35058:head# 2024-04-23T16:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c8071a:::benchmark_data_smithi188_113957_object88999:head# 2024-04-23T16:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c853d8:::benchmark_data_smithi188_113957_object82189:head# 2024-04-23T16:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c85594:::benchmark_data_smithi188_113957_object68776:head# 2024-04-23T16:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c8620c:::benchmark_data_smithi188_113957_object51607:head# 2024-04-23T16:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c87fac:::benchmark_data_smithi188_113957_object21376:head# 2024-04-23T16:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c89896:::benchmark_data_smithi188_113957_object7327:head# 2024-04-23T16:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c9a043:::benchmark_data_smithi188_113957_object130631:head# 2024-04-23T16:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51c9b236:::benchmark_data_smithi188_113957_object131215:head# 2024-04-23T16:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ca818a:::benchmark_data_smithi188_113957_object21082:head# 2024-04-23T16:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51cab89d:::benchmark_data_smithi188_113957_object76593:head# 2024-04-23T16:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51cb77c5:::benchmark_data_smithi188_113957_object35397:head# 2024-04-23T16:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51cccaed:::benchmark_data_smithi188_113957_object51239:head# 2024-04-23T16:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51cd168a:::benchmark_data_smithi188_113957_object130679:head# 2024-04-23T16:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51cd94a4:::benchmark_data_smithi188_113957_object41355:head# 2024-04-23T16:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51cde0aa:::benchmark_data_smithi188_113957_object59442:head# 2024-04-23T16:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ce5aff:::benchmark_data_smithi188_113957_object100504:head# 2024-04-23T16:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ce8a74:::benchmark_data_smithi188_113957_object85801:head# 2024-04-23T16:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51d08e4a:::benchmark_data_smithi188_113957_object61365:head# 2024-04-23T16:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51d0ab82:::benchmark_data_smithi188_113957_object96782:head# 2024-04-23T16:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51d0cab2:::benchmark_data_smithi188_113957_object118170:head# 2024-04-23T16:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51d1f46d:::benchmark_data_smithi188_113957_object39551:head# 2024-04-23T16:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51d30a00:::benchmark_data_smithi188_113957_object83692:head# 2024-04-23T16:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51d4ca54:::benchmark_data_smithi188_113957_object128824:head# 2024-04-23T16:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51d4e14d:::benchmark_data_smithi188_113957_object129960:head# 2024-04-23T16:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51d53f77:::benchmark_data_smithi188_113957_object90025:head# 2024-04-23T16:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51d560c2:::benchmark_data_smithi188_113957_object134997:head# 2024-04-23T16:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51d66f80:::benchmark_data_smithi188_113957_object39772:head# 2024-04-23T16:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51d69caf:::benchmark_data_smithi188_113957_object82048:head# 2024-04-23T16:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51d74541:::benchmark_data_smithi188_113957_object26829:head# 2024-04-23T16:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51d74e05:::benchmark_data_smithi188_113957_object122834:head# 2024-04-23T16:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51d7af0c:::benchmark_data_smithi188_113957_object117580:head# 2024-04-23T16:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51d87386:::benchmark_data_smithi188_113957_object140704:head# 2024-04-23T16:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51d8b0df:::benchmark_data_smithi188_113957_object63073:head# 2024-04-23T16:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51d9dd00:::benchmark_data_smithi188_113957_object4133:head# 2024-04-23T16:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51da3b40:::benchmark_data_smithi188_113957_object117452:head# 2024-04-23T16:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51da9bf4:::benchmark_data_smithi188_113957_object121558:head# 2024-04-23T16:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51db0c77:::benchmark_data_smithi188_113957_object74386:head# 2024-04-23T16:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51dc8d6b:::benchmark_data_smithi188_113957_object32431:head# 2024-04-23T16:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51dcca4b:::benchmark_data_smithi188_113957_object143098:head# 2024-04-23T16:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51dd4870:::benchmark_data_smithi188_113957_object112878:head# 2024-04-23T16:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51dd6c0e:::benchmark_data_smithi188_113957_object145227:head# 2024-04-23T16:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ddde98:::benchmark_data_smithi188_113957_object82995:head# 2024-04-23T16:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51de442a:::benchmark_data_smithi188_113957_object95245:head# 2024-04-23T16:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51de899a:::benchmark_data_smithi188_113957_object28417:head# 2024-04-23T16:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51dfd622:::benchmark_data_smithi188_113957_object84626:head# 2024-04-23T16:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e092d6:::benchmark_data_smithi188_113957_object62972:head# 2024-04-23T16:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e0a9da:::benchmark_data_smithi188_113957_object152514:head# 2024-04-23T16:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e0c501:::benchmark_data_smithi188_113957_object36128:head# 2024-04-23T16:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e0da78:::benchmark_data_smithi188_113957_object149946:head# 2024-04-23T16:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e0ef74:::benchmark_data_smithi188_113957_object95808:head# 2024-04-23T16:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e10247:::benchmark_data_smithi188_113957_object128287:head# 2024-04-23T16:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e108f1:::benchmark_data_smithi188_113957_object144934:head# 2024-04-23T16:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e22771:::benchmark_data_smithi188_113957_object30375:head# 2024-04-23T16:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e22804:::benchmark_data_smithi188_113957_object93692:head# 2024-04-23T16:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.156 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:54 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 26477/279782 objects degraded (9.463%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e29a90:::benchmark_data_smithi188_113957_object7127:head# 2024-04-23T16:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e3469e:::benchmark_data_smithi188_113957_object90717:head# 2024-04-23T16:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e34d83:::benchmark_data_smithi188_113957_object75586:head# 2024-04-23T16:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e355aa:::benchmark_data_smithi188_113957_object37914:head# 2024-04-23T16:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e3b863:::benchmark_data_smithi188_113957_object52057:head# 2024-04-23T16:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e3c0f5:::benchmark_data_smithi188_113957_object7428:head# 2024-04-23T16:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e3d7bc:::benchmark_data_smithi188_113957_object145545:head# 2024-04-23T16:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e51c74:::benchmark_data_smithi188_113957_object70466:head# 2024-04-23T16:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e5960c:::benchmark_data_smithi188_113957_object114909:head# 2024-04-23T16:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e5a56f:::benchmark_data_smithi188_113957_object8162:head# 2024-04-23T16:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e5ea7a:::benchmark_data_smithi188_113957_object85327:head# 2024-04-23T16:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e5f6d4:::benchmark_data_smithi188_113957_object110199:head# 2024-04-23T16:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e622df:::benchmark_data_smithi188_113957_object72492:head# 2024-04-23T16:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e6586f:::benchmark_data_smithi188_113957_object16734:head# 2024-04-23T16:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e873b3:::benchmark_data_smithi188_113957_object2091:head# 2024-04-23T16:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e89408:::benchmark_data_smithi188_113957_object116068:head# 2024-04-23T16:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e89e07:::benchmark_data_smithi188_113957_object1327:head# 2024-04-23T16:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e92d2e:::benchmark_data_smithi188_113957_object45746:head# 2024-04-23T16:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e973e4:::benchmark_data_smithi188_113957_object139505:head# 2024-04-23T16:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e981c0:::benchmark_data_smithi188_113957_object74151:head# 2024-04-23T16:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51e98ecd:::benchmark_data_smithi188_113957_object118599:head# 2024-04-23T16:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ea73a6:::benchmark_data_smithi188_113957_object64404:head# 2024-04-23T16:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51eb9b5d:::benchmark_data_smithi188_113957_object67403:head# 2024-04-23T16:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51eba23e:::benchmark_data_smithi188_113957_object46931:head# 2024-04-23T16:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ebe5bd:::benchmark_data_smithi188_113957_object45469:head# 2024-04-23T16:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ec7472:::benchmark_data_smithi188_113957_object149902:head# 2024-04-23T16:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ec960d:::benchmark_data_smithi188_113957_object31223:head# 2024-04-23T16:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ed64d6:::benchmark_data_smithi188_113957_object46532:head# 2024-04-23T16:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ed97dc:::benchmark_data_smithi188_113957_object135988:head# 2024-04-23T16:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ee38c6:::benchmark_data_smithi188_113957_object114690:head# 2024-04-23T16:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ee71af:::benchmark_data_smithi188_113957_object118817:head# 2024-04-23T16:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f08af5:::benchmark_data_smithi188_113957_object151155:head# 2024-04-23T16:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f09f50:::benchmark_data_smithi188_113957_object67383:head# 2024-04-23T16:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f126bb:::benchmark_data_smithi188_113957_object54197:head# 2024-04-23T16:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f149c4:::benchmark_data_smithi188_113957_object31188:head# 2024-04-23T16:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f1f6bf:::benchmark_data_smithi188_113957_object134730:head# 2024-04-23T16:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f2562b:::benchmark_data_smithi188_113957_object91673:head# 2024-04-23T16:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f30d0f:::benchmark_data_smithi188_113957_object49734:head# 2024-04-23T16:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f32955:::benchmark_data_smithi188_113957_object10938:head# 2024-04-23T16:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f36335:::benchmark_data_smithi188_113957_object57481:head# 2024-04-23T16:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f3ab4d:::benchmark_data_smithi188_113957_object62695:head# 2024-04-23T16:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f3dc0c:::benchmark_data_smithi188_113957_object93693:head# 2024-04-23T16:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f3e735:::benchmark_data_smithi188_113957_object86235:head# 2024-04-23T16:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f3f026:::benchmark_data_smithi188_113957_object26719:head# 2024-04-23T16:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f4e58c:::benchmark_data_smithi188_113957_object127848:head# 2024-04-23T16:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f5103b:::benchmark_data_smithi188_113957_object11140:head# 2024-04-23T16:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f53fc5:::benchmark_data_smithi188_113957_object20558:head# 2024-04-23T16:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f58ee6:::benchmark_data_smithi188_113957_object9313:head# 2024-04-23T16:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f6982c:::benchmark_data_smithi188_113957_object15073:head# 2024-04-23T16:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f71fd0:::benchmark_data_smithi188_113957_object82530:head# 2024-04-23T16:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f7634b:::benchmark_data_smithi188_113957_object55469:head# 2024-04-23T16:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f7bbda:::benchmark_data_smithi188_113957_object125214:head# 2024-04-23T16:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f7f806:::benchmark_data_smithi188_113957_object135473:head# 2024-04-23T16:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f865f1:::benchmark_data_smithi188_113957_object92693:head# 2024-04-23T16:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f90331:::benchmark_data_smithi188_113957_object138823:head# 2024-04-23T16:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f94ca2:::benchmark_data_smithi188_113957_object105819:head# 2024-04-23T16:33:55.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f99118:::benchmark_data_smithi188_113957_object120972:head# 2024-04-23T16:33:55.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f9a9de:::benchmark_data_smithi188_113957_object121834:head# 2024-04-23T16:33:55.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51f9dcbd:::benchmark_data_smithi188_113957_object36991:head# 2024-04-23T16:33:55.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51fa8aa4:::benchmark_data_smithi188_113957_object112070:head# 2024-04-23T16:33:55.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51fa8abb:::benchmark_data_smithi188_113957_object152197:head# 2024-04-23T16:33:55.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51fad575:::benchmark_data_smithi188_113957_object77400:head# 2024-04-23T16:33:55.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51fadd6d:::benchmark_data_smithi188_113957_object68594:head# 2024-04-23T16:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51fb1377:::benchmark_data_smithi188_113957_object34344:head# 2024-04-23T16:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51fc2c36:::benchmark_data_smithi188_113957_object138194:head# 2024-04-23T16:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51fcd4b4:::benchmark_data_smithi188_113957_object96600:head# 2024-04-23T16:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51fdde8c:::benchmark_data_smithi188_113957_object110446:head# 2024-04-23T16:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51fe290c:::benchmark_data_smithi188_113957_object136279:head# 2024-04-23T16:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51fe5852:::benchmark_data_smithi188_113957_object152215:head# 2024-04-23T16:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51fe7c81:::benchmark_data_smithi188_113957_object122285:head# 2024-04-23T16:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51feb6e4:::benchmark_data_smithi188_113957_object43829:head# 2024-04-23T16:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ff4c4e:::benchmark_data_smithi188_113957_object130263:head# 2024-04-23T16:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ff74d0:::benchmark_data_smithi188_113957_object145301:head# 2024-04-23T16:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:51ff7ec4:::benchmark_data_smithi188_113957_object78525:head# 2024-04-23T16:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52007584:::benchmark_data_smithi188_113957_object75790:head# 2024-04-23T16:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:520078be:::benchmark_data_smithi188_113957_object152789:head# 2024-04-23T16:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5200a140:::benchmark_data_smithi188_113957_object151417:head# 2024-04-23T16:33:55.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5200ae6c:::benchmark_data_smithi188_113957_object45097:head# 2024-04-23T16:33:55.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5200e47c:::benchmark_data_smithi188_113957_object131807:head# 2024-04-23T16:33:55.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5200f9c2:::benchmark_data_smithi188_113957_object107873:head# 2024-04-23T16:33:55.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52015e0f:::benchmark_data_smithi188_113957_object22375:head# 2024-04-23T16:33:55.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5201d10d:::benchmark_data_smithi188_113957_object113191:head# 2024-04-23T16:33:55.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5202617b:::benchmark_data_smithi188_113957_object7823:head# 2024-04-23T16:33:55.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52027f18:::benchmark_data_smithi188_113957_object90752:head# 2024-04-23T16:33:55.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52030ac4:::benchmark_data_smithi188_113957_object139232:head# 2024-04-23T16:33:55.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52033e9e:::benchmark_data_smithi188_113957_object113267:head# 2024-04-23T16:33:55.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52043e34:::benchmark_data_smithi188_113957_object46958:head# 2024-04-23T16:33:55.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5204c550:::benchmark_data_smithi188_113957_object85587:head# 2024-04-23T16:33:55.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5204c6f4:::benchmark_data_smithi188_113957_object104245:head# 2024-04-23T16:33:55.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52052962:::benchmark_data_smithi188_113957_object79583:head# 2024-04-23T16:33:55.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5205494a:::benchmark_data_smithi188_113957_object37706:head# 2024-04-23T16:33:55.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52057531:::benchmark_data_smithi188_113957_object5646:head# 2024-04-23T16:33:55.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52064134:::benchmark_data_smithi188_113957_object100535:head# 2024-04-23T16:33:55.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52064b00:::benchmark_data_smithi188_113957_object148882:head# 2024-04-23T16:33:55.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52077a4c:::benchmark_data_smithi188_113957_object131112:head# 2024-04-23T16:33:55.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5207d825:::benchmark_data_smithi188_113957_object52722:head# 2024-04-23T16:33:55.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5208ff7d:::benchmark_data_smithi188_113957_object88861:head# 2024-04-23T16:33:55.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52090b25:::benchmark_data_smithi188_113957_object13150:head# 2024-04-23T16:33:55.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5209107c:::benchmark_data_smithi188_113957_object8467:head# 2024-04-23T16:33:55.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5209bf50:::benchmark_data_smithi188_113957_object5808:head# 2024-04-23T16:33:55.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5209fe55:::benchmark_data_smithi188_113957_object140583:head# 2024-04-23T16:33:55.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:520a32bb:::benchmark_data_smithi188_113957_object77215:head# 2024-04-23T16:33:55.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:520b22cd:::benchmark_data_smithi188_113957_object96506:head# 2024-04-23T16:33:55.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:520b239b:::benchmark_data_smithi188_113957_object149069:head# 2024-04-23T16:33:55.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:520b6aec:::benchmark_data_smithi188_113957_object18655:head# 2024-04-23T16:33:55.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:520ba87e:::benchmark_data_smithi188_113957_object100184:head# 2024-04-23T16:33:55.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:520ce57a:::benchmark_data_smithi188_113957_object60678:head# 2024-04-23T16:33:55.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:520d42de:::benchmark_data_smithi188_113957_object59030:head# 2024-04-23T16:33:55.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:520d9105:::benchmark_data_smithi188_113957_object86828:head# 2024-04-23T16:33:55.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:520d9a51:::benchmark_data_smithi188_113957_object7296:head# 2024-04-23T16:33:55.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:520dd72c:::benchmark_data_smithi188_113957_object90313:head# 2024-04-23T16:33:55.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:520e08d5:::benchmark_data_smithi188_113957_object86126:head# 2024-04-23T16:33:55.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:520eadf0:::benchmark_data_smithi188_113957_object130340:head# 2024-04-23T16:33:55.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:520ff9ca:::benchmark_data_smithi188_113957_object32355:head# 2024-04-23T16:33:55.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52101e8a:::benchmark_data_smithi188_113957_object123782:head# 2024-04-23T16:33:55.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5210f609:::benchmark_data_smithi188_113957_object119761:head# 2024-04-23T16:33:55.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52111d23:::benchmark_data_smithi188_113957_object99312:head# 2024-04-23T16:33:55.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52114b83:::benchmark_data_smithi188_113957_object87341:head# 2024-04-23T16:33:55.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:521187c9:::benchmark_data_smithi188_113957_object446:head# 2024-04-23T16:33:55.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5211b08b:::benchmark_data_smithi188_113957_object153116:head# 2024-04-23T16:33:55.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5212065b:::benchmark_data_smithi188_113957_object126935:head# 2024-04-23T16:33:55.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52121e7b:::benchmark_data_smithi188_113957_object61918:head# 2024-04-23T16:33:55.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52122ec6:::benchmark_data_smithi188_113957_object148277:head# 2024-04-23T16:33:55.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5212cc6b:::benchmark_data_smithi188_113957_object145302:head# 2024-04-23T16:33:55.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5213cacb:::benchmark_data_smithi188_113957_object2945:head# 2024-04-23T16:33:55.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52148d05:::benchmark_data_smithi188_113957_object39600:head# 2024-04-23T16:33:55.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52150c03:::benchmark_data_smithi188_113957_object3701:head# 2024-04-23T16:33:55.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52152692:::benchmark_data_smithi188_113957_object48092:head# 2024-04-23T16:33:55.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5216a1a6:::benchmark_data_smithi188_113957_object31128:head# 2024-04-23T16:33:55.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5216ae0c:::benchmark_data_smithi188_113957_object68778:head# 2024-04-23T16:33:55.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5217386f:::benchmark_data_smithi188_113957_object25741:head# 2024-04-23T16:33:55.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5217614c:::benchmark_data_smithi188_113957_object107753:head# 2024-04-23T16:33:55.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5217d878:::benchmark_data_smithi188_113957_object61866:head# 2024-04-23T16:33:55.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52180308:::benchmark_data_smithi188_113957_object16758:head# 2024-04-23T16:33:55.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52187291:::benchmark_data_smithi188_113957_object141917:head# 2024-04-23T16:33:55.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5218c7bd:::benchmark_data_smithi188_113957_object1918:head# 2024-04-23T16:33:55.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52190964:::benchmark_data_smithi188_113957_object47435:head# 2024-04-23T16:33:55.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:521967d9:::benchmark_data_smithi188_113957_object15601:head# 2024-04-23T16:33:55.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5219b5ec:::benchmark_data_smithi188_113957_object16495:head# 2024-04-23T16:33:55.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:521aa929:::benchmark_data_smithi188_113957_object31894:head# 2024-04-23T16:33:55.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:521ae99c:::benchmark_data_smithi188_113957_object12240:head# 2024-04-23T16:33:55.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:521af5fe:::benchmark_data_smithi188_113957_object119820:head# 2024-04-23T16:33:55.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:521b349e:::benchmark_data_smithi188_113957_object84689:head# 2024-04-23T16:33:55.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:521c06b7:::benchmark_data_smithi188_113957_object11187:head# 2024-04-23T16:33:55.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:521d1737:::benchmark_data_smithi188_113957_object59857:head# 2024-04-23T16:33:55.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:521d3f90:::benchmark_data_smithi188_113957_object63432:head# 2024-04-23T16:33:55.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:521d6b9f:::benchmark_data_smithi188_113957_object138780:head# 2024-04-23T16:33:55.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:521d718a:::benchmark_data_smithi188_113957_object147940:head# 2024-04-23T16:33:55.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:521e87ef:::benchmark_data_smithi188_113957_object79557:head# 2024-04-23T16:33:55.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:521e89a8:::benchmark_data_smithi188_113957_object130322:head# 2024-04-23T16:33:55.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:521e8b1d:::benchmark_data_smithi188_113957_object142719:head# 2024-04-23T16:33:55.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:521eca3b:::benchmark_data_smithi188_113957_object58535:head# 2024-04-23T16:33:55.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:521f0713:::benchmark_data_smithi188_113957_object37645:head# 2024-04-23T16:33:55.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:521f5e40:::benchmark_data_smithi188_113957_object149607:head# 2024-04-23T16:33:55.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:521f94f5:::benchmark_data_smithi188_113957_object46185:head# 2024-04-23T16:33:55.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5220b7e5:::benchmark_data_smithi188_113957_object140714:head# 2024-04-23T16:33:55.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5221166e:::benchmark_data_smithi188_113957_object83585:head# 2024-04-23T16:33:55.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52225b92:::benchmark_data_smithi188_113957_object23309:head# 2024-04-23T16:33:55.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52228e27:::benchmark_data_smithi188_113957_object140200:head# 2024-04-23T16:33:55.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5222aa6d:::benchmark_data_smithi188_113957_object148626:head# 2024-04-23T16:33:55.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5222f7f4:::benchmark_data_smithi188_113957_object122162:head# 2024-04-23T16:33:55.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:522359e0:::benchmark_data_smithi188_113957_object96289:head# 2024-04-23T16:33:55.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5225d390:::benchmark_data_smithi188_113957_object68049:head# 2024-04-23T16:33:55.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5226db88:::benchmark_data_smithi188_113957_object9977:head# 2024-04-23T16:33:55.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52275877:::benchmark_data_smithi188_113957_object32285:head# 2024-04-23T16:33:55.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52275a44:::benchmark_data_smithi188_113957_object62634:head# 2024-04-23T16:33:55.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52276683:::benchmark_data_smithi188_113957_object56047:head# 2024-04-23T16:33:55.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:522909d9:::benchmark_data_smithi188_113957_object80780:head# 2024-04-23T16:33:55.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:522949f5:::benchmark_data_smithi188_113957_object3526:head# 2024-04-23T16:33:55.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5229bf34:::benchmark_data_smithi188_113957_object103517:head# 2024-04-23T16:33:55.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5229ca0a:::benchmark_data_smithi188_113957_object56367:head# 2024-04-23T16:33:55.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:522a2704:::benchmark_data_smithi188_113957_object130841:head# 2024-04-23T16:33:55.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:522aafb6:::benchmark_data_smithi188_113957_object85425:head# 2024-04-23T16:33:55.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:522ad75c:::benchmark_data_smithi188_113957_object147864:head# 2024-04-23T16:33:55.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:522b4da0:::benchmark_data_smithi188_113957_object75210:head# 2024-04-23T16:33:55.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:522ce784:::benchmark_data_smithi188_113957_object55520:head# 2024-04-23T16:33:55.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:522d0f21:::benchmark_data_smithi188_113957_object77330:head# 2024-04-23T16:33:55.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:522de997:::benchmark_data_smithi188_113957_object130801:head# 2024-04-23T16:33:55.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:522e410a:::benchmark_data_smithi188_113957_object35868:head# 2024-04-23T16:33:55.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:522e4c3c:::benchmark_data_smithi188_113957_object54593:head# 2024-04-23T16:33:55.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:522e5257:::benchmark_data_smithi188_113957_object3656:head# 2024-04-23T16:33:55.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:522eb588:::benchmark_data_smithi188_113957_object105678:head# 2024-04-23T16:33:55.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:522f34b3:::benchmark_data_smithi188_113957_object42133:head# 2024-04-23T16:33:55.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:522f7107:::benchmark_data_smithi188_113957_object31357:head# 2024-04-23T16:33:55.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5230440c:::benchmark_data_smithi188_113957_object27280:head# 2024-04-23T16:33:55.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5232062c:::benchmark_data_smithi188_113957_object7726:head# 2024-04-23T16:33:55.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:523233ac:::benchmark_data_smithi188_113957_object91893:head# 2024-04-23T16:33:55.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52325a1a:::benchmark_data_smithi188_113957_object105862:head# 2024-04-23T16:33:55.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52325e8f:::benchmark_data_smithi188_113957_object130900:head# 2024-04-23T16:33:55.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:523364c0:::benchmark_data_smithi188_113957_object35421:head# 2024-04-23T16:33:55.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:523424ef:::benchmark_data_smithi188_113957_object142978:head# 2024-04-23T16:33:55.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:523488f9:::benchmark_data_smithi188_113957_object107155:head# 2024-04-23T16:33:55.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5234b013:::benchmark_data_smithi188_113957_object20897:head# 2024-04-23T16:33:55.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5234ff27:::benchmark_data_smithi188_113957_object145809:head# 2024-04-23T16:33:55.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5235131d:::benchmark_data_smithi188_113957_object54918:head# 2024-04-23T16:33:55.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52355a66:::benchmark_data_smithi188_113957_object34386:head# 2024-04-23T16:33:55.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52359550:::benchmark_data_smithi188_113957_object133626:head# 2024-04-23T16:33:55.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5235bc64:::benchmark_data_smithi188_113957_object24819:head# 2024-04-23T16:33:55.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52361127:::benchmark_data_smithi188_113957_object7178:head# 2024-04-23T16:33:55.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5236448b:::benchmark_data_smithi188_113957_object9732:head# 2024-04-23T16:33:55.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5236e4f2:::benchmark_data_smithi188_113957_object9325:head# 2024-04-23T16:33:55.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52375455:::benchmark_data_smithi188_113957_object34530:head# 2024-04-23T16:33:55.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52376bc7:::benchmark_data_smithi188_113957_object90031:head# 2024-04-23T16:33:55.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5237a01f:::benchmark_data_smithi188_113957_object60707:head# 2024-04-23T16:33:55.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5237d23f:::benchmark_data_smithi188_113957_object126954:head# 2024-04-23T16:33:55.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5238186a:::benchmark_data_smithi188_113957_object72027:head# 2024-04-23T16:33:55.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52384617:::benchmark_data_smithi188_113957_object14371:head# 2024-04-23T16:33:55.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5238c6a0:::benchmark_data_smithi188_113957_object149220:head# 2024-04-23T16:33:55.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5239c0ca:::benchmark_data_smithi188_113957_object22100:head# 2024-04-23T16:33:55.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:523a0b89:::benchmark_data_smithi188_113957_object45338:head# 2024-04-23T16:33:55.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:523a6be9:::benchmark_data_smithi188_113957_object29369:head# 2024-04-23T16:33:55.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:523bdd52:::benchmark_data_smithi188_113957_object22752:head# 2024-04-23T16:33:55.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:523c2504:::benchmark_data_smithi188_113957_object65373:head# 2024-04-23T16:33:55.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:523c50e9:::benchmark_data_smithi188_113957_object55172:head# 2024-04-23T16:33:55.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:523cd450:::benchmark_data_smithi188_113957_object50355:head# 2024-04-23T16:33:55.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:523f7051:::benchmark_data_smithi188_113957_object28351:head# 2024-04-23T16:33:55.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:523f8936:::benchmark_data_smithi188_113957_object103839:head# 2024-04-23T16:33:55.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:523fa31b:::benchmark_data_smithi188_113957_object141049:head# 2024-04-23T16:33:55.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:523fc094:::benchmark_data_smithi188_113957_object127281:head# 2024-04-23T16:33:55.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:523feb1b:::benchmark_data_smithi188_113957_object94595:head# 2024-04-23T16:33:55.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:524072fe:::benchmark_data_smithi188_113957_object92447:head# 2024-04-23T16:33:55.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5240dbe2:::benchmark_data_smithi188_113957_object105135:head# 2024-04-23T16:33:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5241b17c:::benchmark_data_smithi188_113957_object64250:head# 2024-04-23T16:33:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5241b699:::benchmark_data_smithi188_113957_object105375:head# 2024-04-23T16:33:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52420f84:::benchmark_data_smithi188_113957_object91505:head# 2024-04-23T16:33:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52422d5e:::benchmark_data_smithi188_113957_object85631:head# 2024-04-23T16:33:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:524252bb:::benchmark_data_smithi188_113957_object71311:head# 2024-04-23T16:33:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:524302f9:::benchmark_data_smithi188_113957_object103519:head# 2024-04-23T16:33:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5243685c:::benchmark_data_smithi188_113957_object74488:head# 2024-04-23T16:33:55.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:524371ee:::benchmark_data_smithi188_113957_object50844:head# 2024-04-23T16:33:55.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52447e94:::benchmark_data_smithi188_113957_object142093:head# 2024-04-23T16:33:55.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5244d143:::benchmark_data_smithi188_113957_object110331:head# 2024-04-23T16:33:55.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5245305d:::benchmark_data_smithi188_113957_object5541:head# 2024-04-23T16:33:55.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5245f49a:::benchmark_data_smithi188_113957_object37510:head# 2024-04-23T16:33:55.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52460453:::benchmark_data_smithi188_113957_object3934:head# 2024-04-23T16:33:55.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52466dd1:::benchmark_data_smithi188_113957_object32122:head# 2024-04-23T16:33:55.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:524671bb:::benchmark_data_smithi188_113957_object9891:head# 2024-04-23T16:33:55.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5246969c:::benchmark_data_smithi188_113957_object124241:head# 2024-04-23T16:33:55.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52471548:::benchmark_data_smithi188_113957_object144753:head# 2024-04-23T16:33:55.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52474525:::benchmark_data_smithi188_113957_object4900:head# 2024-04-23T16:33:55.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5247ab19:::benchmark_data_smithi188_113957_object137473:head# 2024-04-23T16:33:55.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5247c8d8:::benchmark_data_smithi188_113957_object5217:head# 2024-04-23T16:33:55.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5248406a:::benchmark_data_smithi188_113957_object52560:head# 2024-04-23T16:33:55.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52484846:::benchmark_data_smithi188_113957_object25308:head# 2024-04-23T16:33:55.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5248d2df:::benchmark_data_smithi188_113957_object46121:head# 2024-04-23T16:33:55.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5248e96f:::benchmark_data_smithi188_113957_object47178:head# 2024-04-23T16:33:55.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52499490:::benchmark_data_smithi188_113957_object75187:head# 2024-04-23T16:33:55.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5249f7bd:::benchmark_data_smithi188_113957_object46400:head# 2024-04-23T16:33:55.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:524a0beb:::benchmark_data_smithi188_113957_object100544:head# 2024-04-23T16:33:55.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:524b103b:::benchmark_data_smithi188_113957_object150809:head# 2024-04-23T16:33:55.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:524b531d:::benchmark_data_smithi188_113957_object92680:head# 2024-04-23T16:33:55.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:524c0a14:::benchmark_data_smithi188_113957_object104209:head# 2024-04-23T16:33:55.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:524e113d:::benchmark_data_smithi188_113957_object62507:head# 2024-04-23T16:33:55.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:524e8b42:::benchmark_data_smithi188_113957_object90107:head# 2024-04-23T16:33:55.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:524f424b:::benchmark_data_smithi188_113957_object100476:head# 2024-04-23T16:33:55.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:524f47c5:::benchmark_data_smithi188_113957_object19221:head# 2024-04-23T16:33:55.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:524f8c8d:::benchmark_data_smithi188_113957_object82982:head# 2024-04-23T16:33:55.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:524fa6b3:::benchmark_data_smithi188_113957_object74907:head# 2024-04-23T16:33:55.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:524fb75b:::benchmark_data_smithi188_113957_object57035:head# 2024-04-23T16:33:55.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:524fc2db:::benchmark_data_smithi188_113957_object107950:head# 2024-04-23T16:33:55.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:524fd792:::benchmark_data_smithi188_113957_object4822:head# 2024-04-23T16:33:55.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5250f900:::benchmark_data_smithi188_113957_object41609:head# 2024-04-23T16:33:55.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:525180b0:::benchmark_data_smithi188_113957_object95417:head# 2024-04-23T16:33:55.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52518493:::benchmark_data_smithi188_113957_object27335:head# 2024-04-23T16:33:55.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5252278b:::benchmark_data_smithi188_113957_object13086:head# 2024-04-23T16:33:55.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52533d90:::benchmark_data_smithi188_113957_object143198:head# 2024-04-23T16:33:55.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5254841a:::benchmark_data_smithi188_113957_object43433:head# 2024-04-23T16:33:55.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5254cdcd:::benchmark_data_smithi188_113957_object116161:head# 2024-04-23T16:33:55.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5255dd4e:::benchmark_data_smithi188_113957_object64668:head# 2024-04-23T16:33:55.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5255f77b:::benchmark_data_smithi188_113957_object78377:head# 2024-04-23T16:33:55.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:525617e5:::benchmark_data_smithi188_113957_object103361:head# 2024-04-23T16:33:55.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:525670f7:::benchmark_data_smithi188_113957_object74341:head# 2024-04-23T16:33:55.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5256a01c:::benchmark_data_smithi188_113957_object53145:head# 2024-04-23T16:33:55.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5256bd2a:::benchmark_data_smithi188_113957_object17071:head# 2024-04-23T16:33:55.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5256f7d1:::benchmark_data_smithi188_113957_object8574:head# 2024-04-23T16:33:55.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5257e967:::benchmark_data_smithi188_113957_object19748:head# 2024-04-23T16:33:55.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52583dcd:::benchmark_data_smithi188_113957_object2528:head# 2024-04-23T16:33:55.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5258756e:::benchmark_data_smithi188_113957_object9899:head# 2024-04-23T16:33:55.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5258918c:::benchmark_data_smithi188_113957_object117745:head# 2024-04-23T16:33:55.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52590b5e:::benchmark_data_smithi188_113957_object36270:head# 2024-04-23T16:33:55.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5259113f:::benchmark_data_smithi188_113957_object115684:head# 2024-04-23T16:33:55.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52595403:::benchmark_data_smithi188_113957_object15448:head# 2024-04-23T16:33:55.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52598b52:::benchmark_data_smithi188_113957_object12388:head# 2024-04-23T16:33:55.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:525a4fa3:::benchmark_data_smithi188_113957_object70427:head# 2024-04-23T16:33:55.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:525ac089:::benchmark_data_smithi188_113957_object121919:head# 2024-04-23T16:33:55.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:525ad065:::benchmark_data_smithi188_113957_object1578:head# 2024-04-23T16:33:55.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:525bb162:::benchmark_data_smithi188_113957_object106111:head# 2024-04-23T16:33:55.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:525c43c5:::benchmark_data_smithi188_113957_object981:head# 2024-04-23T16:33:55.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:525c654a:::benchmark_data_smithi188_113957_object59796:head# 2024-04-23T16:33:55.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:525cc68c:::benchmark_data_smithi188_113957_object89436:head# 2024-04-23T16:33:55.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:525e3c12:::benchmark_data_smithi188_113957_object47310:head# 2024-04-23T16:33:55.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:525eadf1:::benchmark_data_smithi188_113957_object115956:head# 2024-04-23T16:33:55.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:525f0bd2:::benchmark_data_smithi188_113957_object138048:head# 2024-04-23T16:33:55.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:525f3427:::benchmark_data_smithi188_113957_object53558:head# 2024-04-23T16:33:55.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:525f8775:::benchmark_data_smithi188_113957_object153336:head# 2024-04-23T16:33:55.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:526042ac:::benchmark_data_smithi188_113957_object129687:head# 2024-04-23T16:33:55.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:526066fb:::benchmark_data_smithi188_113957_object91810:head# 2024-04-23T16:33:55.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5260ded6:::benchmark_data_smithi188_113957_object57388:head# 2024-04-23T16:33:55.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:526169a3:::benchmark_data_smithi188_113957_object4376:head# 2024-04-23T16:33:55.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:526169ea:::benchmark_data_smithi188_113957_object5483:head# 2024-04-23T16:33:55.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5261be34:::benchmark_data_smithi188_113957_object121411:head# 2024-04-23T16:33:55.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5261d8f5:::benchmark_data_smithi188_113957_object8799:head# 2024-04-23T16:33:55.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5261ed1d:::benchmark_data_smithi188_113957_object70623:head# 2024-04-23T16:33:55.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5261ef47:::benchmark_data_smithi188_113957_object138347:head# 2024-04-23T16:33:55.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5262cc93:::benchmark_data_smithi188_113957_object132019:head# 2024-04-23T16:33:55.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5262cda3:::benchmark_data_smithi188_113957_object7053:head# 2024-04-23T16:33:55.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5263b7ab:::benchmark_data_smithi188_113957_object80576:head# 2024-04-23T16:33:55.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5263fcac:::benchmark_data_smithi188_113957_object119046:head# 2024-04-23T16:33:55.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5264a2f9:::benchmark_data_smithi188_113957_object14500:head# 2024-04-23T16:33:55.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5264eeb6:::benchmark_data_smithi188_113957_object83440:head# 2024-04-23T16:33:55.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52650450:::benchmark_data_smithi188_113957_object40470:head# 2024-04-23T16:33:55.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52651b25:::benchmark_data_smithi188_113957_object122991:head# 2024-04-23T16:33:55.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5266050f:::benchmark_data_smithi188_113957_object52633:head# 2024-04-23T16:33:55.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52663c08:::benchmark_data_smithi188_113957_object54506:head# 2024-04-23T16:33:55.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:526661f5:::benchmark_data_smithi188_113957_object56861:head# 2024-04-23T16:33:55.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5266f1a1:::benchmark_data_smithi188_113957_object31668:head# 2024-04-23T16:33:55.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5266fcdb:::benchmark_data_smithi188_113957_object42033:head# 2024-04-23T16:33:55.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52671aed:::benchmark_data_smithi188_113957_object103470:head# 2024-04-23T16:33:55.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5267ad21:::benchmark_data_smithi188_113957_object105631:head# 2024-04-23T16:33:55.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52698638:::benchmark_data_smithi188_113957_object11780:head# 2024-04-23T16:33:55.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5269f667:::benchmark_data_smithi188_113957_object84987:head# 2024-04-23T16:33:55.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:526b2b7c:::benchmark_data_smithi188_113957_object126432:head# 2024-04-23T16:33:55.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:526c2009:::benchmark_data_smithi188_113957_object72130:head# 2024-04-23T16:33:55.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:526c906b:::benchmark_data_smithi188_113957_object67033:head# 2024-04-23T16:33:55.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:526ca625:::benchmark_data_smithi188_113957_object52832:head# 2024-04-23T16:33:55.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:526cfeea:::benchmark_data_smithi188_113957_object74110:head# 2024-04-23T16:33:55.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:526d246d:::benchmark_data_smithi188_113957_object147176:head# 2024-04-23T16:33:55.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:526d4227:::benchmark_data_smithi188_113957_object95990:head# 2024-04-23T16:33:55.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:526d9525:::benchmark_data_smithi188_113957_object83232:head# 2024-04-23T16:33:55.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:526dc44a:::benchmark_data_smithi188_113957_object37771:head# 2024-04-23T16:33:55.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:526ebbf1:::benchmark_data_smithi188_113957_object81482:head# 2024-04-23T16:33:55.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:526f5900:::benchmark_data_smithi188_113957_object5526:head# 2024-04-23T16:33:55.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:526f8595:::benchmark_data_smithi188_113957_object151348:head# 2024-04-23T16:33:55.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:526f8dfe:::benchmark_data_smithi188_113957_object105159:head# 2024-04-23T16:33:55.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:526f95a3:::benchmark_data_smithi188_113957_object17838:head# 2024-04-23T16:33:55.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5270c198:::benchmark_data_smithi188_113957_object123547:head# 2024-04-23T16:33:55.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52710f44:::benchmark_data_smithi188_113957_object61423:head# 2024-04-23T16:33:55.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5271b132:::benchmark_data_smithi188_113957_object83301:head# 2024-04-23T16:33:55.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52720c2a:::benchmark_data_smithi188_113957_object138953:head# 2024-04-23T16:33:55.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:527242b1:::benchmark_data_smithi188_113957_object32821:head# 2024-04-23T16:33:55.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5273b36c:::benchmark_data_smithi188_113957_object27517:head# 2024-04-23T16:33:55.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5273e578:::benchmark_data_smithi188_113957_object38110:head# 2024-04-23T16:33:55.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5274a4ee:::benchmark_data_smithi188_113957_object48543:head# 2024-04-23T16:33:55.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5274c1df:::benchmark_data_smithi188_113957_object110969:head# 2024-04-23T16:33:55.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5274cde7:::benchmark_data_smithi188_113957_object104901:head# 2024-04-23T16:33:55.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52756eef:::benchmark_data_smithi188_113957_object101749:head# 2024-04-23T16:33:55.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:527594bb:::benchmark_data_smithi188_113957_object59006:head# 2024-04-23T16:33:55.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52766348:::benchmark_data_smithi188_113957_object89576:head# 2024-04-23T16:33:55.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5276a5be:::benchmark_data_smithi188_113957_object149162:head# 2024-04-23T16:33:55.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5276a6f8:::benchmark_data_smithi188_113957_object97908:head# 2024-04-23T16:33:55.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:527813b3:::benchmark_data_smithi188_113957_object117448:head# 2024-04-23T16:33:55.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:527817c8:::benchmark_data_smithi188_113957_object66842:head# 2024-04-23T16:33:55.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52784b9a:::benchmark_data_smithi188_113957_object140360:head# 2024-04-23T16:33:55.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52786d86:::benchmark_data_smithi188_113957_object141865:head# 2024-04-23T16:33:55.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5278fb95:::benchmark_data_smithi188_113957_object89965:head# 2024-04-23T16:33:55.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52791296:::benchmark_data_smithi188_113957_object66209:head# 2024-04-23T16:33:55.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:527915f5:::benchmark_data_smithi188_113957_object102687:head# 2024-04-23T16:33:55.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5279282a:::benchmark_data_smithi188_113957_object61971:head# 2024-04-23T16:33:55.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:527a1e0e:::benchmark_data_smithi188_113957_object8660:head# 2024-04-23T16:33:55.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:527a3d91:::benchmark_data_smithi188_113957_object64048:head# 2024-04-23T16:33:55.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:527b1e30:::benchmark_data_smithi188_113957_object136831:head# 2024-04-23T16:33:55.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:527b9d28:::benchmark_data_smithi188_113957_object22526:head# 2024-04-23T16:33:55.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:527bb8bf:::benchmark_data_smithi188_113957_object21454:head# 2024-04-23T16:33:55.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:527beaff:::benchmark_data_smithi188_113957_object104496:head# 2024-04-23T16:33:55.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:527c2c75:::benchmark_data_smithi188_113957_object3269:head# 2024-04-23T16:33:55.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:527e0410:::benchmark_data_smithi188_113957_object15902:head# 2024-04-23T16:33:55.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:527e22ba:::benchmark_data_smithi188_113957_object16864:head# 2024-04-23T16:33:55.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:527f8b02:::benchmark_data_smithi188_113957_object70737:head# 2024-04-23T16:33:55.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52800204:::benchmark_data_smithi188_113957_object81788:head# 2024-04-23T16:33:55.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5280f212:::benchmark_data_smithi188_113957_object66983:head# 2024-04-23T16:33:55.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:528185a7:::benchmark_data_smithi188_113957_object103959:head# 2024-04-23T16:33:55.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5281caca:::benchmark_data_smithi188_113957_object135736:head# 2024-04-23T16:33:55.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52833543:::benchmark_data_smithi188_113957_object68618:head# 2024-04-23T16:33:55.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52834aaa:::benchmark_data_smithi188_113957_object48691:head# 2024-04-23T16:33:55.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5283d6ec:::benchmark_data_smithi188_113957_object135182:head# 2024-04-23T16:33:55.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5283dcdc:::benchmark_data_smithi188_113957_object125371:head# 2024-04-23T16:33:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:528494f2:::benchmark_data_smithi188_113957_object131485:head# 2024-04-23T16:33:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52852235:::benchmark_data_smithi188_113957_object147971:head# 2024-04-23T16:33:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52852834:::benchmark_data_smithi188_113957_object7820:head# 2024-04-23T16:33:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52853b78:::benchmark_data_smithi188_113957_object105463:head# 2024-04-23T16:33:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52857973:::benchmark_data_smithi188_113957_object142697:head# 2024-04-23T16:33:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:528582b7:::benchmark_data_smithi188_113957_object103000:head# 2024-04-23T16:33:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5285f488:::benchmark_data_smithi188_113957_object139974:head# 2024-04-23T16:33:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52865df0:::benchmark_data_smithi188_113957_object3419:head# 2024-04-23T16:33:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52866027:::benchmark_data_smithi188_113957_object88000:head# 2024-04-23T16:33:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52876b92:::benchmark_data_smithi188_113957_object103067:head# 2024-04-23T16:33:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:528784eb:::benchmark_data_smithi188_113957_object35399:head# 2024-04-23T16:33:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5287b5c9:::benchmark_data_smithi188_113957_object60714:head# 2024-04-23T16:33:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52888c1c:::benchmark_data_smithi188_113957_object61340:head# 2024-04-23T16:33:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52888ee7:::benchmark_data_smithi188_113957_object50650:head# 2024-04-23T16:33:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52892316:::benchmark_data_smithi188_113957_object65331:head# 2024-04-23T16:33:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5289547e:::benchmark_data_smithi188_113957_object144334:head# 2024-04-23T16:33:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:528ae044:::benchmark_data_smithi188_113957_object97385:head# 2024-04-23T16:33:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:528aebbf:::benchmark_data_smithi188_113957_object73367:head# 2024-04-23T16:33:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:528b8178:::benchmark_data_smithi188_113957_object63405:head# 2024-04-23T16:33:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:528c53d7:::benchmark_data_smithi188_113957_object66144:head# 2024-04-23T16:33:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:528d2624:::benchmark_data_smithi188_113957_object127587:head# 2024-04-23T16:33:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:528f2b4a:::benchmark_data_smithi188_113957_object83407:head# 2024-04-23T16:33:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52903bfb:::benchmark_data_smithi188_113957_object83673:head# 2024-04-23T16:33:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5290d368:::benchmark_data_smithi188_113957_object81871:head# 2024-04-23T16:33:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:529172f3:::benchmark_data_smithi188_113957_object80290:head# 2024-04-23T16:33:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:529180b2:::benchmark_data_smithi188_113957_object69479:head# 2024-04-23T16:33:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5291ba2f:::benchmark_data_smithi188_113957_object25965:head# 2024-04-23T16:33:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5292a402:::benchmark_data_smithi188_113957_object96882:head# 2024-04-23T16:33:55.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52931302:::benchmark_data_smithi188_113957_object111174:head# 2024-04-23T16:33:55.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52935397:::benchmark_data_smithi188_113957_object149933:head# 2024-04-23T16:33:55.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52938a7c:::benchmark_data_smithi188_113957_object107740:head# 2024-04-23T16:33:55.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5293b242:::benchmark_data_smithi188_113957_object58105:head# 2024-04-23T16:33:55.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5293e13f:::benchmark_data_smithi188_113957_object53995:head# 2024-04-23T16:33:55.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5294020d:::benchmark_data_smithi188_113957_object120074:head# 2024-04-23T16:33:55.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52943647:::benchmark_data_smithi188_113957_object55140:head# 2024-04-23T16:33:55.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5294a234:::benchmark_data_smithi188_113957_object11791:head# 2024-04-23T16:33:55.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52958890:::benchmark_data_smithi188_113957_object23046:head# 2024-04-23T16:33:55.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5295ae1b:::benchmark_data_smithi188_113957_object36896:head# 2024-04-23T16:33:55.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5296004f:::benchmark_data_smithi188_113957_object94662:head# 2024-04-23T16:33:55.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:529901dc:::benchmark_data_smithi188_113957_object76983:head# 2024-04-23T16:33:55.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52990a0f:::benchmark_data_smithi188_113957_object61600:head# 2024-04-23T16:33:55.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52994b25:::benchmark_data_smithi188_113957_object145430:head# 2024-04-23T16:33:55.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5299635d:::benchmark_data_smithi188_113957_object89843:head# 2024-04-23T16:33:55.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5299bf71:::benchmark_data_smithi188_113957_object12759:head# 2024-04-23T16:33:55.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:529a269e:::benchmark_data_smithi188_113957_object87065:head# 2024-04-23T16:33:55.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:529a3118:::benchmark_data_smithi188_113957_object27126:head# 2024-04-23T16:33:55.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:529aef78:::benchmark_data_smithi188_113957_object53204:head# 2024-04-23T16:33:55.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:529bd5cc:::benchmark_data_smithi188_113957_object42531:head# 2024-04-23T16:33:55.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:529c8e2b:::benchmark_data_smithi188_113957_object152327:head# 2024-04-23T16:33:55.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:529c8ff7:::benchmark_data_smithi188_113957_object39928:head# 2024-04-23T16:33:55.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:529c9a38:::benchmark_data_smithi188_113957_object137973:head# 2024-04-23T16:33:55.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:529dfd1e:::benchmark_data_smithi188_113957_object60594:head# 2024-04-23T16:33:55.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:529e38c8:::benchmark_data_smithi188_113957_object94299:head# 2024-04-23T16:33:55.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:529e94ca:::benchmark_data_smithi188_113957_object122331:head# 2024-04-23T16:33:55.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:529f7870:::benchmark_data_smithi188_113957_object133964:head# 2024-04-23T16:33:55.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:529fe528:::benchmark_data_smithi188_113957_object67333:head# 2024-04-23T16:33:55.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a1f634:::benchmark_data_smithi188_113957_object110302:head# 2024-04-23T16:33:55.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a1fefa:::benchmark_data_smithi188_113957_object100190:head# 2024-04-23T16:33:55.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a2c2ac:::benchmark_data_smithi188_113957_object95284:head# 2024-04-23T16:33:55.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a42faa:::benchmark_data_smithi188_113957_object148865:head# 2024-04-23T16:33:55.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a4e047:::benchmark_data_smithi188_113957_object11199:head# 2024-04-23T16:33:55.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a507b3:::benchmark_data_smithi188_113957_object133731:head# 2024-04-23T16:33:55.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a53b45:::benchmark_data_smithi188_113957_object69857:head# 2024-04-23T16:33:55.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a5ebc2:::benchmark_data_smithi188_113957_object55168:head# 2024-04-23T16:33:55.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a6128e:::benchmark_data_smithi188_113957_object41096:head# 2024-04-23T16:33:55.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a64f7c:::benchmark_data_smithi188_113957_object41064:head# 2024-04-23T16:33:55.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a6d4af:::benchmark_data_smithi188_113957_object840:head# 2024-04-23T16:33:55.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a6d653:::benchmark_data_smithi188_113957_object66683:head# 2024-04-23T16:33:55.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a6fe53:::benchmark_data_smithi188_113957_object69355:head# 2024-04-23T16:33:55.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a77d41:::benchmark_data_smithi188_113957_object123320:head# 2024-04-23T16:33:55.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a7c7a6:::benchmark_data_smithi188_113957_object38497:head# 2024-04-23T16:33:55.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a8609b:::benchmark_data_smithi188_113957_object60980:head# 2024-04-23T16:33:55.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a8aefa:::benchmark_data_smithi188_113957_object126407:head# 2024-04-23T16:33:55.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a8db3a:::benchmark_data_smithi188_113957_object29422:head# 2024-04-23T16:33:55.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a8f947:::benchmark_data_smithi188_113957_object144460:head# 2024-04-23T16:33:55.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a9165f:::benchmark_data_smithi188_113957_object105176:head# 2024-04-23T16:33:55.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a918a9:::benchmark_data_smithi188_113957_object49610:head# 2024-04-23T16:33:55.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52a955ef:::benchmark_data_smithi188_113957_object126437:head# 2024-04-23T16:33:55.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52aa12fe:::benchmark_data_smithi188_113957_object61378:head# 2024-04-23T16:33:55.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52aa94d9:::benchmark_data_smithi188_113957_object40522:head# 2024-04-23T16:33:55.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52aa9680:::benchmark_data_smithi188_113957_object80548:head# 2024-04-23T16:33:55.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52aaeb62:::benchmark_data_smithi188_113957_object112201:head# 2024-04-23T16:33:55.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ab1b2b:::benchmark_data_smithi188_113957_object30573:head# 2024-04-23T16:33:55.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ab550a:::benchmark_data_smithi188_113957_object86203:head# 2024-04-23T16:33:55.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ab804d:::benchmark_data_smithi188_113957_object36355:head# 2024-04-23T16:33:55.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ab9e47:::benchmark_data_smithi188_113957_object19456:head# 2024-04-23T16:33:55.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ac3d35:::benchmark_data_smithi188_113957_object15472:head# 2024-04-23T16:33:55.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52aca522:::benchmark_data_smithi188_113957_object52068:head# 2024-04-23T16:33:55.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ad2839:::benchmark_data_smithi188_113957_object101244:head# 2024-04-23T16:33:55.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52adb05b:::benchmark_data_smithi188_113957_object53603:head# 2024-04-23T16:33:55.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52adbd1a:::benchmark_data_smithi188_113957_object141199:head# 2024-04-23T16:33:55.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52aec290:::benchmark_data_smithi188_113957_object103191:head# 2024-04-23T16:33:55.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52af48a0:::benchmark_data_smithi188_113957_object13722:head# 2024-04-23T16:33:55.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52af83e9:::benchmark_data_smithi188_113957_object58549:head# 2024-04-23T16:33:55.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52af934d:::benchmark_data_smithi188_113957_object81353:head# 2024-04-23T16:33:55.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52af9b6a:::benchmark_data_smithi188_113957_object44759:head# 2024-04-23T16:33:55.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52b0adb6:::benchmark_data_smithi188_113957_object110721:head# 2024-04-23T16:33:55.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52b15ef3:::benchmark_data_smithi188_113957_object104617:head# 2024-04-23T16:33:55.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52b20cd1:::benchmark_data_smithi188_113957_object18570:head# 2024-04-23T16:33:55.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52b21b76:::benchmark_data_smithi188_113957_object76933:head# 2024-04-23T16:33:55.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52b3247c:::benchmark_data_smithi188_113957_object47936:head# 2024-04-23T16:33:55.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52b349f0:::benchmark_data_smithi188_113957_object11423:head# 2024-04-23T16:33:55.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.289 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:54 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 26477/279782 objects degraded (9.463%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:55.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:55.372 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:33:55.372 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:33:55.378 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52b3e770:::benchmark_data_smithi188_113957_object16327:head# 2024-04-23T16:33:55.378 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.378 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52b43c59:::benchmark_data_smithi188_113957_object16413:head# 2024-04-23T16:33:55.378 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52b4952e:::benchmark_data_smithi188_113957_object78226:head# 2024-04-23T16:33:55.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52b54872:::benchmark_data_smithi188_113957_object104930:head# 2024-04-23T16:33:55.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52b5b209:::benchmark_data_smithi188_113957_object122439:head# 2024-04-23T16:33:55.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52b69ee4:::benchmark_data_smithi188_113957_object28766:head# 2024-04-23T16:33:55.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52b6c603:::benchmark_data_smithi188_113957_object151943:head# 2024-04-23T16:33:55.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52b755c4:::benchmark_data_smithi188_113957_object91703:head# 2024-04-23T16:33:55.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.379 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52b8548d:::benchmark_data_smithi188_113957_object74042:head# 2024-04-23T16:33:55.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52b8ecf9:::benchmark_data_smithi188_113957_object30278:head# 2024-04-23T16:33:55.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52b93a97:::benchmark_data_smithi188_113957_object110486:head# 2024-04-23T16:33:55.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52b96c6d:::benchmark_data_smithi188_113957_object100911:head# 2024-04-23T16:33:55.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ba5bd9:::benchmark_data_smithi188_113957_object104542:head# 2024-04-23T16:33:55.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ba7088:::benchmark_data_smithi188_113957_object44687:head# 2024-04-23T16:33:55.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ba7539:::benchmark_data_smithi188_113957_object97063:head# 2024-04-23T16:33:55.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.380 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ba9d75:::benchmark_data_smithi188_113957_object40236:head# 2024-04-23T16:33:55.380 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52bb6210:::benchmark_data_smithi188_113957_object9996:head# 2024-04-23T16:33:55.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52bb708b:::benchmark_data_smithi188_113957_object71726:head# 2024-04-23T16:33:55.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52bb8063:::benchmark_data_smithi188_113957_object99037:head# 2024-04-23T16:33:55.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52bb9b6b:::benchmark_data_smithi188_113957_object99930:head# 2024-04-23T16:33:55.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52bcc79c:::benchmark_data_smithi188_113957_object30796:head# 2024-04-23T16:33:55.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52bd0ed7:::benchmark_data_smithi188_113957_object1431:head# 2024-04-23T16:33:55.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52bd1e82:::benchmark_data_smithi188_113957_object23707:head# 2024-04-23T16:33:55.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52bd2edf:::benchmark_data_smithi188_113957_object46735:head# 2024-04-23T16:33:55.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52bd5f57:::benchmark_data_smithi188_113957_object30792:head# 2024-04-23T16:33:55.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52be350e:::benchmark_data_smithi188_113957_object108041:head# 2024-04-23T16:33:55.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52bec8c3:::benchmark_data_smithi188_113957_object66553:head# 2024-04-23T16:33:55.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52bf8f6f:::benchmark_data_smithi188_113957_object12726:head# 2024-04-23T16:33:55.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52bfb21f:::benchmark_data_smithi188_113957_object129200:head# 2024-04-23T16:33:55.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52bfed64:::benchmark_data_smithi188_113957_object91290:head# 2024-04-23T16:33:55.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52c0d76c:::benchmark_data_smithi188_113957_object139692:head# 2024-04-23T16:33:55.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52c155da:::benchmark_data_smithi188_113957_object3527:head# 2024-04-23T16:33:55.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52c16dec:::benchmark_data_smithi188_113957_object37947:head# 2024-04-23T16:33:55.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52c1842c:::benchmark_data_smithi188_113957_object3572:head# 2024-04-23T16:33:55.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52c3754a:::benchmark_data_smithi188_113957_object50088:head# 2024-04-23T16:33:55.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52c39311:::benchmark_data_smithi188_113957_object127471:head# 2024-04-23T16:33:55.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52c408ec:::benchmark_data_smithi188_113957_object107895:head# 2024-04-23T16:33:55.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52c42089:::benchmark_data_smithi188_113957_object36435:head# 2024-04-23T16:33:55.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52c51581:::benchmark_data_smithi188_113957_object24503:head# 2024-04-23T16:33:55.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52c555cb:::benchmark_data_smithi188_113957_object31644:head# 2024-04-23T16:33:55.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52c70cc4:::benchmark_data_smithi188_113957_object133882:head# 2024-04-23T16:33:55.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52c755e4:::benchmark_data_smithi188_113957_object67471:head# 2024-04-23T16:33:55.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52c7b265:::benchmark_data_smithi188_113957_object96998:head# 2024-04-23T16:33:55.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52c88ee2:::benchmark_data_smithi188_113957_object94313:head# 2024-04-23T16:33:55.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52c8c514:::benchmark_data_smithi188_113957_object31754:head# 2024-04-23T16:33:55.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52c9c58b:::benchmark_data_smithi188_113957_object8377:head# 2024-04-23T16:33:55.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52c9d038:::benchmark_data_smithi188_113957_object64872:head# 2024-04-23T16:33:55.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ca23b0:::benchmark_data_smithi188_113957_object5951:head# 2024-04-23T16:33:55.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52cacd49:::benchmark_data_smithi188_113957_object145724:head# 2024-04-23T16:33:55.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52caeaf7:::benchmark_data_smithi188_113957_object52965:head# 2024-04-23T16:33:55.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52cb0d26:::benchmark_data_smithi188_113957_object79146:head# 2024-04-23T16:33:55.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52cb5768:::benchmark_data_smithi188_113957_object48386:head# 2024-04-23T16:33:55.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52cb5a4f:::benchmark_data_smithi188_113957_object75995:head# 2024-04-23T16:33:55.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52cbc071:::benchmark_data_smithi188_113957_object38175:head# 2024-04-23T16:33:55.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52cbd93e:::benchmark_data_smithi188_113957_object13920:head# 2024-04-23T16:33:55.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52cc2ab0:::benchmark_data_smithi188_113957_object85952:head# 2024-04-23T16:33:55.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.387 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52cc2ef9:::benchmark_data_smithi188_113957_object87346:head# 2024-04-23T16:33:55.387 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.387 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ccb75f:::benchmark_data_smithi188_113957_object28309:head# 2024-04-23T16:33:55.387 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.387 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ccbebd:::benchmark_data_smithi188_113957_object40124:head# 2024-04-23T16:33:55.387 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.387 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52cce1af:::benchmark_data_smithi188_113957_object116780:head# 2024-04-23T16:33:55.387 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.387 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ccf8ea:::benchmark_data_smithi188_113957_object105432:head# 2024-04-23T16:33:55.387 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.387 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52cdd152:::benchmark_data_smithi188_113957_object19524:head# 2024-04-23T16:33:55.387 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.387 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52cddafa:::benchmark_data_smithi188_113957_object89343:head# 2024-04-23T16:33:55.387 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ce5099:::benchmark_data_smithi188_113957_object125406:head# 2024-04-23T16:33:55.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52cf7e13:::benchmark_data_smithi188_113957_object81002:head# 2024-04-23T16:33:55.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52cf9c24:::benchmark_data_smithi188_113957_object113583:head# 2024-04-23T16:33:55.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52cfd76d:::benchmark_data_smithi188_113957_object142709:head# 2024-04-23T16:33:55.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52cfd9ce:::benchmark_data_smithi188_113957_object100960:head# 2024-04-23T16:33:55.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d0c40d:::benchmark_data_smithi188_113957_object49046:head# 2024-04-23T16:33:55.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d19a07:::benchmark_data_smithi188_113957_object63520:head# 2024-04-23T16:33:55.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d1b902:::benchmark_data_smithi188_113957_object12557:head# 2024-04-23T16:33:55.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d1c0b8:::benchmark_data_smithi188_113957_object90670:head# 2024-04-23T16:33:55.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d21567:::benchmark_data_smithi188_113957_object130211:head# 2024-04-23T16:33:55.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d233c3:::benchmark_data_smithi188_113957_object106022:head# 2024-04-23T16:33:55.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d292ca:::benchmark_data_smithi188_113957_object80036:head# 2024-04-23T16:33:55.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d2a735:::benchmark_data_smithi188_113957_object101364:head# 2024-04-23T16:33:55.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d2b518:::benchmark_data_smithi188_113957_object69863:head# 2024-04-23T16:33:55.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d2ba4f:::benchmark_data_smithi188_113957_object106042:head# 2024-04-23T16:33:55.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d31fa5:::benchmark_data_smithi188_113957_object104877:head# 2024-04-23T16:33:55.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d37fbb:::benchmark_data_smithi188_113957_object126752:head# 2024-04-23T16:33:55.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d56534:::benchmark_data_smithi188_113957_object136488:head# 2024-04-23T16:33:55.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d59a47:::benchmark_data_smithi188_113957_object28840:head# 2024-04-23T16:33:55.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d5a2b4:::benchmark_data_smithi188_113957_object1608:head# 2024-04-23T16:33:55.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d65f98:::benchmark_data_smithi188_113957_object68319:head# 2024-04-23T16:33:55.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d6607f:::benchmark_data_smithi188_113957_object75237:head# 2024-04-23T16:33:55.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d6beb6:::benchmark_data_smithi188_113957_object110992:head# 2024-04-23T16:33:55.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d71fd5:::benchmark_data_smithi188_113957_object144180:head# 2024-04-23T16:33:55.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d7c161:::benchmark_data_smithi188_113957_object104109:head# 2024-04-23T16:33:55.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d8b0ea:::benchmark_data_smithi188_113957_object148202:head# 2024-04-23T16:33:55.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d8d8d9:::benchmark_data_smithi188_113957_object145074:head# 2024-04-23T16:33:55.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d908f5:::benchmark_data_smithi188_113957_object91227:head# 2024-04-23T16:33:55.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d953f7:::benchmark_data_smithi188_113957_object43944:head# 2024-04-23T16:33:55.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52d9888e:::benchmark_data_smithi188_113957_object81609:head# 2024-04-23T16:33:55.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52da294a:::benchmark_data_smithi188_113957_object121984:head# 2024-04-23T16:33:55.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52da54d3:::benchmark_data_smithi188_113957_object84924:head# 2024-04-23T16:33:55.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52dadee7:::benchmark_data_smithi188_113957_object52884:head# 2024-04-23T16:33:55.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52dae5c5:::benchmark_data_smithi188_113957_object99020:head# 2024-04-23T16:33:55.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52db050e:::benchmark_data_smithi188_113957_object37446:head# 2024-04-23T16:33:55.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52db0a4d:::benchmark_data_smithi188_113957_object36039:head# 2024-04-23T16:33:55.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52db999c:::benchmark_data_smithi188_113957_object122706:head# 2024-04-23T16:33:55.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52dc2c4d:::benchmark_data_smithi188_113957_object111195:head# 2024-04-23T16:33:55.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52dcd3d8:::benchmark_data_smithi188_113957_object14528:head# 2024-04-23T16:33:55.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52dd41d6:::benchmark_data_smithi188_113957_object75327:head# 2024-04-23T16:33:55.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52dd6d70:::benchmark_data_smithi188_113957_object127098:head# 2024-04-23T16:33:55.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ddfaab:::benchmark_data_smithi188_113957_object79443:head# 2024-04-23T16:33:55.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52df801b:::benchmark_data_smithi188_113957_object131117:head# 2024-04-23T16:33:55.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52df9c11:::benchmark_data_smithi188_113957_object80653:head# 2024-04-23T16:33:55.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e037d5:::benchmark_data_smithi188_113957_object83686:head# 2024-04-23T16:33:55.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e06411:::benchmark_data_smithi188_113957_object20350:head# 2024-04-23T16:33:55.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e101a5:::benchmark_data_smithi188_113957_object100023:head# 2024-04-23T16:33:55.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e1baef:::benchmark_data_smithi188_113957_object24885:head# 2024-04-23T16:33:55.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e1d232:::benchmark_data_smithi188_113957_object35325:head# 2024-04-23T16:33:55.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e288ea:::benchmark_data_smithi188_113957_object132658:head# 2024-04-23T16:33:55.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e28c67:::benchmark_data_smithi188_113957_object13772:head# 2024-04-23T16:33:55.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e2e58c:::benchmark_data_smithi188_113957_object23262:head# 2024-04-23T16:33:55.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e31785:::benchmark_data_smithi188_113957_object104960:head# 2024-04-23T16:33:55.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e3bcad:::benchmark_data_smithi188_113957_object74394:head# 2024-04-23T16:33:55.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e500e6:::benchmark_data_smithi188_113957_object77864:head# 2024-04-23T16:33:55.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e51fcb:::benchmark_data_smithi188_113957_object51483:head# 2024-04-23T16:33:55.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e52aa8:::benchmark_data_smithi188_113957_object11726:head# 2024-04-23T16:33:55.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e5a797:::benchmark_data_smithi188_113957_object87475:head# 2024-04-23T16:33:55.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e5b68d:::benchmark_data_smithi188_113957_object142828:head# 2024-04-23T16:33:55.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e5e9dd:::benchmark_data_smithi188_113957_object152706:head# 2024-04-23T16:33:55.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e63f6b:::benchmark_data_smithi188_113957_object73155:head# 2024-04-23T16:33:55.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e66b66:::benchmark_data_smithi188_113957_object147660:head# 2024-04-23T16:33:55.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e691dc:::benchmark_data_smithi188_113957_object149136:head# 2024-04-23T16:33:55.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e75e51:::benchmark_data_smithi188_113957_object142663:head# 2024-04-23T16:33:55.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e78613:::benchmark_data_smithi188_113957_object130149:head# 2024-04-23T16:33:55.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e8028b:::benchmark_data_smithi188_113957_object107289:head# 2024-04-23T16:33:55.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e9916e:::benchmark_data_smithi188_113957_object23075:head# 2024-04-23T16:33:55.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52e9fce2:::benchmark_data_smithi188_113957_object15643:head# 2024-04-23T16:33:55.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ea0fd2:::benchmark_data_smithi188_113957_object96945:head# 2024-04-23T16:33:55.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ea1327:::benchmark_data_smithi188_113957_object105731:head# 2024-04-23T16:33:55.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52eb296b:::benchmark_data_smithi188_113957_object65419:head# 2024-04-23T16:33:55.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52eb8d67:::benchmark_data_smithi188_113957_object48722:head# 2024-04-23T16:33:55.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ebab59:::benchmark_data_smithi188_113957_object59988:head# 2024-04-23T16:33:55.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ebab91:::benchmark_data_smithi188_113957_object43479:head# 2024-04-23T16:33:55.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ec9f3d:::benchmark_data_smithi188_113957_object5846:head# 2024-04-23T16:33:55.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ecb999:::benchmark_data_smithi188_113957_object143303:head# 2024-04-23T16:33:55.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ecbb51:::benchmark_data_smithi188_113957_object123454:head# 2024-04-23T16:33:55.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ed0b26:::benchmark_data_smithi188_113957_object62479:head# 2024-04-23T16:33:55.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ed2040:::benchmark_data_smithi188_113957_object77087:head# 2024-04-23T16:33:55.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ed69f3:::benchmark_data_smithi188_113957_object143751:head# 2024-04-23T16:33:55.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52edffda:::benchmark_data_smithi188_113957_object51705:head# 2024-04-23T16:33:55.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ee391e:::benchmark_data_smithi188_113957_object74805:head# 2024-04-23T16:33:55.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ee5950:::benchmark_data_smithi188_113957_object105789:head# 2024-04-23T16:33:55.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52eea819:::benchmark_data_smithi188_113957_object89823:head# 2024-04-23T16:33:55.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52eed09b:::benchmark_data_smithi188_113957_object61895:head# 2024-04-23T16:33:55.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52eee006:::benchmark_data_smithi188_113957_object133742:head# 2024-04-23T16:33:55.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ef0943:::benchmark_data_smithi188_113957_object84785:head# 2024-04-23T16:33:55.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ef4782:::benchmark_data_smithi188_113957_object123336:head# 2024-04-23T16:33:55.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f02429:::benchmark_data_smithi188_113957_object141318:head# 2024-04-23T16:33:55.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f047c2:::benchmark_data_smithi188_113957_object8558:head# 2024-04-23T16:33:55.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f118af:::benchmark_data_smithi188_113957_object96696:head# 2024-04-23T16:33:55.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f16bf5:::benchmark_data_smithi188_113957_object32097:head# 2024-04-23T16:33:55.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f1edfe:::benchmark_data_smithi188_113957_object99217:head# 2024-04-23T16:33:55.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f2ce97:::benchmark_data_smithi188_113957_object80318:head# 2024-04-23T16:33:55.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f35e86:::benchmark_data_smithi188_113957_object15500:head# 2024-04-23T16:33:55.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f39602:::benchmark_data_smithi188_113957_object38877:head# 2024-04-23T16:33:55.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f3a5b1:::benchmark_data_smithi188_113957_object97066:head# 2024-04-23T16:33:55.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f3f6f8:::benchmark_data_smithi188_113957_object29788:head# 2024-04-23T16:33:55.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f43bb1:::benchmark_data_smithi188_113957_object98080:head# 2024-04-23T16:33:55.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f4592f:::benchmark_data_smithi188_113957_object20997:head# 2024-04-23T16:33:55.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f4596e:::benchmark_data_smithi188_113957_object151753:head# 2024-04-23T16:33:55.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f4aa5c:::benchmark_data_smithi188_113957_object26457:head# 2024-04-23T16:33:55.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f4ff40:::benchmark_data_smithi188_113957_object116901:head# 2024-04-23T16:33:55.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f518e6:::benchmark_data_smithi188_113957_object69136:head# 2024-04-23T16:33:55.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f537f5:::benchmark_data_smithi188_113957_object118542:head# 2024-04-23T16:33:55.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f54f14:::benchmark_data_smithi188_113957_object133162:head# 2024-04-23T16:33:55.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f562dc:::benchmark_data_smithi188_113957_object3086:head# 2024-04-23T16:33:55.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f58be4:::benchmark_data_smithi188_113957_object63010:head# 2024-04-23T16:33:55.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f68514:::benchmark_data_smithi188_113957_object120158:head# 2024-04-23T16:33:55.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f6d812:::benchmark_data_smithi188_113957_object48314:head# 2024-04-23T16:33:55.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f75597:::benchmark_data_smithi188_113957_object63505:head# 2024-04-23T16:33:55.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f771d9:::benchmark_data_smithi188_113957_object29857:head# 2024-04-23T16:33:55.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f7a26d:::benchmark_data_smithi188_113957_object116847:head# 2024-04-23T16:33:55.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f823b2:::benchmark_data_smithi188_113957_object117233:head# 2024-04-23T16:33:55.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52f859df:::benchmark_data_smithi188_113957_object57471:head# 2024-04-23T16:33:55.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52fa5d73:::benchmark_data_smithi188_113957_object52185:head# 2024-04-23T16:33:55.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52fad502:::benchmark_data_smithi188_113957_object101673:head# 2024-04-23T16:33:55.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52fb2f1e:::benchmark_data_smithi188_113957_object39943:head# 2024-04-23T16:33:55.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52fb88af:::benchmark_data_smithi188_113957_object74820:head# 2024-04-23T16:33:55.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52fbfa3b:::benchmark_data_smithi188_113957_object67982:head# 2024-04-23T16:33:55.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52fc7ccb:::benchmark_data_smithi188_113957_object129920:head# 2024-04-23T16:33:55.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52fcd34e:::benchmark_data_smithi188_113957_object6904:head# 2024-04-23T16:33:55.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52fd14f5:::benchmark_data_smithi188_113957_object131435:head# 2024-04-23T16:33:55.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52fd5296:::benchmark_data_smithi188_113957_object15701:head# 2024-04-23T16:33:55.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ff8d77:::benchmark_data_smithi188_113957_object17212:head# 2024-04-23T16:33:55.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ff9219:::benchmark_data_smithi188_113957_object142940:head# 2024-04-23T16:33:55.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ffd641:::benchmark_data_smithi188_113957_object135476:head# 2024-04-23T16:33:55.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:52ffec7f:::benchmark_data_smithi188_113957_object1342:head# 2024-04-23T16:33:55.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5300557f:::benchmark_data_smithi188_113957_object58445:head# 2024-04-23T16:33:55.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53010327:::benchmark_data_smithi188_113957_object52085:head# 2024-04-23T16:33:55.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:530148d2:::benchmark_data_smithi188_113957_object21896:head# 2024-04-23T16:33:55.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5301a77b:::benchmark_data_smithi188_113957_object104024:head# 2024-04-23T16:33:55.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:530408c4:::benchmark_data_smithi188_113957_object80834:head# 2024-04-23T16:33:55.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53046e2d:::benchmark_data_smithi188_113957_object148445:head# 2024-04-23T16:33:55.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5304ce7f:::benchmark_data_smithi188_113957_object26601:head# 2024-04-23T16:33:55.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5305c70c:::benchmark_data_smithi188_113957_object127847:head# 2024-04-23T16:33:55.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5305cbf8:::benchmark_data_smithi188_113957_object147977:head# 2024-04-23T16:33:55.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5305df11:::benchmark_data_smithi188_113957_object67751:head# 2024-04-23T16:33:55.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53069f53:::benchmark_data_smithi188_113957_object35275:head# 2024-04-23T16:33:55.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5307f825:::benchmark_data_smithi188_113957_object60045:head# 2024-04-23T16:33:55.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53085aa3:::benchmark_data_smithi188_113957_object111497:head# 2024-04-23T16:33:55.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:530888f0:::benchmark_data_smithi188_113957_object117393:head# 2024-04-23T16:33:55.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5308a9dc:::benchmark_data_smithi188_113957_object137751:head# 2024-04-23T16:33:55.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:530967fc:::benchmark_data_smithi188_113957_object116911:head# 2024-04-23T16:33:55.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5309bc4e:::benchmark_data_smithi188_113957_object14112:head# 2024-04-23T16:33:55.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:530a7000:::benchmark_data_smithi188_113957_object131950:head# 2024-04-23T16:33:55.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:530a91c6:::benchmark_data_smithi188_113957_object144929:head# 2024-04-23T16:33:55.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:530c3814:::benchmark_data_smithi188_113957_object58093:head# 2024-04-23T16:33:55.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:530c6be8:::benchmark_data_smithi188_113957_object17495:head# 2024-04-23T16:33:55.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:530c738f:::benchmark_data_smithi188_113957_object30281:head# 2024-04-23T16:33:55.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:530cbddc:::benchmark_data_smithi188_113957_object27181:head# 2024-04-23T16:33:55.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:530d3144:::benchmark_data_smithi188_113957_object95512:head# 2024-04-23T16:33:55.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:530d410e:::benchmark_data_smithi188_113957_object65060:head# 2024-04-23T16:33:55.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:530d4563:::benchmark_data_smithi188_113957_object73711:head# 2024-04-23T16:33:55.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:530db7f8:::benchmark_data_smithi188_113957_object30343:head# 2024-04-23T16:33:55.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:530dc5c5:::benchmark_data_smithi188_113957_object149294:head# 2024-04-23T16:33:55.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:530eeb89:::benchmark_data_smithi188_113957_object87311:head# 2024-04-23T16:33:55.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:530eff52:::benchmark_data_smithi188_113957_object95976:head# 2024-04-23T16:33:55.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:530f550a:::benchmark_data_smithi188_113957_object78360:head# 2024-04-23T16:33:55.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:530fec3e:::benchmark_data_smithi188_113957_object96362:head# 2024-04-23T16:33:55.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53101b39:::benchmark_data_smithi188_113957_object62197:head# 2024-04-23T16:33:55.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531165ba:::benchmark_data_smithi188_113957_object64435:head# 2024-04-23T16:33:55.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531186ad:::benchmark_data_smithi188_113957_object47334:head# 2024-04-23T16:33:55.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5311b9aa:::benchmark_data_smithi188_113957_object151363:head# 2024-04-23T16:33:55.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5312534c:::benchmark_data_smithi188_113957_object37816:head# 2024-04-23T16:33:55.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53128775:::benchmark_data_smithi188_113957_object63895:head# 2024-04-23T16:33:55.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53137af1:::benchmark_data_smithi188_113957_object19350:head# 2024-04-23T16:33:55.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53140ec9:::benchmark_data_smithi188_113957_object34476:head# 2024-04-23T16:33:55.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5314251b:::benchmark_data_smithi188_113957_object97631:head# 2024-04-23T16:33:55.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53142687:::benchmark_data_smithi188_113957_object5444:head# 2024-04-23T16:33:55.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53146f76:::benchmark_data_smithi188_113957_object106967:head# 2024-04-23T16:33:55.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53151e75:::benchmark_data_smithi188_113957_object71453:head# 2024-04-23T16:33:55.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53168cf8:::benchmark_data_smithi188_113957_object68288:head# 2024-04-23T16:33:55.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53177069:::benchmark_data_smithi188_113957_object21670:head# 2024-04-23T16:33:55.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53177cc7:::benchmark_data_smithi188_113957_object84139:head# 2024-04-23T16:33:55.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5317f7d2:::benchmark_data_smithi188_113957_object92673:head# 2024-04-23T16:33:55.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53188728:::benchmark_data_smithi188_113957_object8828:head# 2024-04-23T16:33:55.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5319f969:::benchmark_data_smithi188_113957_object107324:head# 2024-04-23T16:33:55.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531b4a03:::benchmark_data_smithi188_113957_object126616:head# 2024-04-23T16:33:55.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531b5c8f:::benchmark_data_smithi188_113957_object99576:head# 2024-04-23T16:33:55.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531ba70b:::benchmark_data_smithi188_113957_object97351:head# 2024-04-23T16:33:55.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531bc99f:::benchmark_data_smithi188_113957_object126937:head# 2024-04-23T16:33:55.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531bf7a5:::benchmark_data_smithi188_113957_object133017:head# 2024-04-23T16:33:55.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531c5686:::benchmark_data_smithi188_113957_object11574:head# 2024-04-23T16:33:55.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531cb76e:::benchmark_data_smithi188_113957_object151277:head# 2024-04-23T16:33:55.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531cecc8:::benchmark_data_smithi188_113957_object77717:head# 2024-04-23T16:33:55.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531d1a2f:::benchmark_data_smithi188_113957_object122637:head# 2024-04-23T16:33:55.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531d28a4:::benchmark_data_smithi188_113957_object34404:head# 2024-04-23T16:33:55.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531d6a63:::benchmark_data_smithi188_113957_object25307:head# 2024-04-23T16:33:55.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531dcbdd:::benchmark_data_smithi188_113957_object12288:head# 2024-04-23T16:33:55.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531dd265:::benchmark_data_smithi188_113957_object107124:head# 2024-04-23T16:33:55.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531e06f3:::benchmark_data_smithi188_113957_object81593:head# 2024-04-23T16:33:55.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531e69f2:::benchmark_data_smithi188_113957_object134582:head# 2024-04-23T16:33:55.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531f5da4:::benchmark_data_smithi188_113957_object150651:head# 2024-04-23T16:33:55.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531fa0a2:::benchmark_data_smithi188_113957_object113682:head# 2024-04-23T16:33:55.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531fa318:::benchmark_data_smithi188_113957_object113714:head# 2024-04-23T16:33:55.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531fea6c:::benchmark_data_smithi188_113957_object116965:head# 2024-04-23T16:33:55.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:531fee3b:::benchmark_data_smithi188_113957_object127908:head# 2024-04-23T16:33:55.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5320309c:::benchmark_data_smithi188_113957_object77452:head# 2024-04-23T16:33:55.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53204044:::benchmark_data_smithi188_113957_object137266:head# 2024-04-23T16:33:55.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53215846:::benchmark_data_smithi188_113957_object52911:head# 2024-04-23T16:33:55.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53219e7e:::benchmark_data_smithi188_113957_object104383:head# 2024-04-23T16:33:55.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53231a97:::benchmark_data_smithi188_113957_object138129:head# 2024-04-23T16:33:55.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53231c71:::benchmark_data_smithi188_113957_object108950:head# 2024-04-23T16:33:55.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53239e85:::benchmark_data_smithi188_113957_object111507:head# 2024-04-23T16:33:55.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5323ed18:::benchmark_data_smithi188_113957_object33665:head# 2024-04-23T16:33:55.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5323f05c:::benchmark_data_smithi188_113957_object53058:head# 2024-04-23T16:33:55.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:532419e3:::benchmark_data_smithi188_113957_object145927:head# 2024-04-23T16:33:55.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53243c58:::benchmark_data_smithi188_113957_object91168:head# 2024-04-23T16:33:55.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5324d4f4:::benchmark_data_smithi188_113957_object31024:head# 2024-04-23T16:33:55.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53250710:::benchmark_data_smithi188_113957_object151087:head# 2024-04-23T16:33:55.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5326621b:::benchmark_data_smithi188_113957_object150207:head# 2024-04-23T16:33:55.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53266366:::benchmark_data_smithi188_113957_object63891:head# 2024-04-23T16:33:55.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5326fade:::benchmark_data_smithi188_113957_object33731:head# 2024-04-23T16:33:55.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:532746ad:::benchmark_data_smithi188_113957_object121790:head# 2024-04-23T16:33:55.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53274e52:::benchmark_data_smithi188_113957_object63222:head# 2024-04-23T16:33:55.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53288130:::benchmark_data_smithi188_113957_object136776:head# 2024-04-23T16:33:55.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5328b599:::benchmark_data_smithi188_113957_object70235:head# 2024-04-23T16:33:55.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53290b25:::benchmark_data_smithi188_113957_object153180:head# 2024-04-23T16:33:55.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5329209a:::benchmark_data_smithi188_113957_object131219:head# 2024-04-23T16:33:55.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5329c1f5:::benchmark_data_smithi188_113957_object30196:head# 2024-04-23T16:33:55.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:532ac422:::benchmark_data_smithi188_113957_object25059:head# 2024-04-23T16:33:55.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:532bd787:::benchmark_data_smithi188_113957_object72437:head# 2024-04-23T16:33:55.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:532c7bea:::benchmark_data_smithi188_113957_object51471:head# 2024-04-23T16:33:55.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:532c7f18:::benchmark_data_smithi188_113957_object144909:head# 2024-04-23T16:33:55.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:532df6bf:::benchmark_data_smithi188_113957_object33424:head# 2024-04-23T16:33:55.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:532e4764:::benchmark_data_smithi188_113957_object1160:head# 2024-04-23T16:33:55.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:532ee4eb:::benchmark_data_smithi188_113957_object74154:head# 2024-04-23T16:33:55.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:532f5c71:::benchmark_data_smithi188_113957_object46029:head# 2024-04-23T16:33:55.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5330ff14:::benchmark_data_smithi188_113957_object150776:head# 2024-04-23T16:33:55.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53315ae2:::benchmark_data_smithi188_113957_object104223:head# 2024-04-23T16:33:55.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:533184c6:::benchmark_data_smithi188_113957_object32963:head# 2024-04-23T16:33:55.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5331cffa:::benchmark_data_smithi188_113957_object123625:head# 2024-04-23T16:33:55.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5331d368:::benchmark_data_smithi188_113957_object30207:head# 2024-04-23T16:33:55.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53326c44:::benchmark_data_smithi188_113957_object86475:head# 2024-04-23T16:33:55.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5332d3c9:::benchmark_data_smithi188_113957_object100580:head# 2024-04-23T16:33:55.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53333e6a:::benchmark_data_smithi188_113957_object131596:head# 2024-04-23T16:33:55.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5333e7e3:::benchmark_data_smithi188_113957_object132516:head# 2024-04-23T16:33:55.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:533457b1:::benchmark_data_smithi188_113957_object67559:head# 2024-04-23T16:33:55.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5335c62a:::benchmark_data_smithi188_113957_object122385:head# 2024-04-23T16:33:55.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53363447:::benchmark_data_smithi188_113957_object101007:head# 2024-04-23T16:33:55.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53367d1f:::benchmark_data_smithi188_113957_object97353:head# 2024-04-23T16:33:55.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53369074:::benchmark_data_smithi188_113957_object81658:head# 2024-04-23T16:33:55.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5336a7b4:::benchmark_data_smithi188_113957_object94933:head# 2024-04-23T16:33:55.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5336a893:::benchmark_data_smithi188_113957_object28858:head# 2024-04-23T16:33:55.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5336e8df:::benchmark_data_smithi188_113957_object40526:head# 2024-04-23T16:33:55.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5337f5ec:::benchmark_data_smithi188_113957_object36034:head# 2024-04-23T16:33:55.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53396bec:::benchmark_data_smithi188_113957_object125239:head# 2024-04-23T16:33:55.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5339e7e4:::benchmark_data_smithi188_113957_object148446:head# 2024-04-23T16:33:55.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:533a1655:::benchmark_data_smithi188_113957_object80379:head# 2024-04-23T16:33:55.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:533ab6d8:::benchmark_data_smithi188_113957_object69609:head# 2024-04-23T16:33:55.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:533ae59e:::benchmark_data_smithi188_113957_object72509:head# 2024-04-23T16:33:55.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:533b0755:::benchmark_data_smithi188_113957_object40431:head# 2024-04-23T16:33:55.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:533c3daf:::benchmark_data_smithi188_113957_object30194:head# 2024-04-23T16:33:55.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:533c4b3b:::benchmark_data_smithi188_113957_object19715:head# 2024-04-23T16:33:55.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:533cf277:::benchmark_data_smithi188_113957_object90450:head# 2024-04-23T16:33:55.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:533db209:::benchmark_data_smithi188_113957_object114154:head# 2024-04-23T16:33:55.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:533e6070:::benchmark_data_smithi188_113957_object127089:head# 2024-04-23T16:33:55.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:533e71da:::benchmark_data_smithi188_113957_object127605:head# 2024-04-23T16:33:55.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:533e7bc1:::benchmark_data_smithi188_113957_object32172:head# 2024-04-23T16:33:55.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:533f4f4a:::benchmark_data_smithi188_113957_object34689:head# 2024-04-23T16:33:55.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:533f9711:::benchmark_data_smithi188_113957_object51763:head# 2024-04-23T16:33:55.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:533fa2cc:::benchmark_data_smithi188_113957_object31822:head# 2024-04-23T16:33:55.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:533fc189:::benchmark_data_smithi188_113957_object79142:head# 2024-04-23T16:33:55.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:533ffae8:::benchmark_data_smithi188_113957_object122912:head# 2024-04-23T16:33:55.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53406970:::benchmark_data_smithi188_113957_object114650:head# 2024-04-23T16:33:55.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5340ad0d:::benchmark_data_smithi188_113957_object70351:head# 2024-04-23T16:33:55.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5340c4e3:::benchmark_data_smithi188_113957_object97462:head# 2024-04-23T16:33:55.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5340fef6:::benchmark_data_smithi188_113957_object104975:head# 2024-04-23T16:33:55.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534109d6:::benchmark_data_smithi188_113957_object86471:head# 2024-04-23T16:33:55.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53411b52:::benchmark_data_smithi188_113957_object26746:head# 2024-04-23T16:33:55.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534125aa:::benchmark_data_smithi188_113957_object33727:head# 2024-04-23T16:33:55.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534179a6:::benchmark_data_smithi188_113957_object88677:head# 2024-04-23T16:33:55.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53423aaf:::benchmark_data_smithi188_113957_object109350:head# 2024-04-23T16:33:55.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534255e5:::benchmark_data_smithi188_113957_object36475:head# 2024-04-23T16:33:55.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53428433:::benchmark_data_smithi188_113957_object31404:head# 2024-04-23T16:33:55.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5342c3bf:::benchmark_data_smithi188_113957_object130380:head# 2024-04-23T16:33:55.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5342f059:::benchmark_data_smithi188_113957_object77895:head# 2024-04-23T16:33:55.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53432dd9:::benchmark_data_smithi188_113957_object52931:head# 2024-04-23T16:33:55.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53433a0e:::benchmark_data_smithi188_113957_object111031:head# 2024-04-23T16:33:55.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534468d6:::benchmark_data_smithi188_113957_object75323:head# 2024-04-23T16:33:55.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5344a8cb:::benchmark_data_smithi188_113957_object150456:head# 2024-04-23T16:33:55.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5344c916:::benchmark_data_smithi188_113957_object120810:head# 2024-04-23T16:33:55.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5345a39b:::benchmark_data_smithi188_113957_object115379:head# 2024-04-23T16:33:55.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534630e7:::benchmark_data_smithi188_113957_object129153:head# 2024-04-23T16:33:55.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53463c94:::benchmark_data_smithi188_113957_object99904:head# 2024-04-23T16:33:55.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53465285:::benchmark_data_smithi188_113957_object806:head# 2024-04-23T16:33:55.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5347261a:::benchmark_data_smithi188_113957_object67426:head# 2024-04-23T16:33:55.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53478ac5:::benchmark_data_smithi188_113957_object120999:head# 2024-04-23T16:33:55.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5347cc4d:::benchmark_data_smithi188_113957_object92093:head# 2024-04-23T16:33:55.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534809c0:::benchmark_data_smithi188_113957_object150884:head# 2024-04-23T16:33:55.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53481cd6:::benchmark_data_smithi188_113957_object110602:head# 2024-04-23T16:33:55.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534836e4:::benchmark_data_smithi188_113957_object104733:head# 2024-04-23T16:33:55.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5348c8c1:::benchmark_data_smithi188_113957_object82840:head# 2024-04-23T16:33:55.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5348c8eb:::benchmark_data_smithi188_113957_object36920:head# 2024-04-23T16:33:55.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53497dcd:::benchmark_data_smithi188_113957_object152520:head# 2024-04-23T16:33:55.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5349ce43:::benchmark_data_smithi188_113957_object99683:head# 2024-04-23T16:33:55.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5349e2a1:::benchmark_data_smithi188_113957_object4206:head# 2024-04-23T16:33:55.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534a8be8:::benchmark_data_smithi188_113957_object76218:head# 2024-04-23T16:33:55.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534a9f29:::benchmark_data_smithi188_113957_object103201:head# 2024-04-23T16:33:55.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534aa4e4:::benchmark_data_smithi188_113957_object15216:head# 2024-04-23T16:33:55.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534ad44e:::benchmark_data_smithi188_113957_object131566:head# 2024-04-23T16:33:55.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534b4e51:::benchmark_data_smithi188_113957_object111023:head# 2024-04-23T16:33:55.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534b9b8c:::benchmark_data_smithi188_113957_object126151:head# 2024-04-23T16:33:55.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534d8bae:::benchmark_data_smithi188_113957_object121360:head# 2024-04-23T16:33:55.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534d9380:::benchmark_data_smithi188_113957_object144204:head# 2024-04-23T16:33:55.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534d9be6:::benchmark_data_smithi188_113957_object104045:head# 2024-04-23T16:33:55.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534dfeb9:::benchmark_data_smithi188_113957_object38710:head# 2024-04-23T16:33:55.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534f7acf:::benchmark_data_smithi188_113957_object26817:head# 2024-04-23T16:33:55.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:534fd587:::benchmark_data_smithi188_113957_object134663:head# 2024-04-23T16:33:55.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.432 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5350747b:::benchmark_data_smithi188_113957_object108974:head# 2024-04-23T16:33:55.432 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.432 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5350d364:::benchmark_data_smithi188_113957_object78345:head# 2024-04-23T16:33:55.432 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.432 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53516bd4:::benchmark_data_smithi188_113957_object67127:head# 2024-04-23T16:33:55.432 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.432 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5351b8b3:::benchmark_data_smithi188_113957_object11562:head# 2024-04-23T16:33:55.432 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.432 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:535214f0:::benchmark_data_smithi188_113957_object72014:head# 2024-04-23T16:33:55.432 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.432 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:535215db:::benchmark_data_smithi188_113957_object144696:head# 2024-04-23T16:33:55.432 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.432 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53525eaf:::benchmark_data_smithi188_113957_object27584:head# 2024-04-23T16:33:55.432 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5352a037:::benchmark_data_smithi188_113957_object56629:head# 2024-04-23T16:33:55.433 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5352f779:::benchmark_data_smithi188_113957_object143500:head# 2024-04-23T16:33:55.433 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5353544c:::benchmark_data_smithi188_113957_object89995:head# 2024-04-23T16:33:55.433 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53535b89:::benchmark_data_smithi188_113957_object121457:head# 2024-04-23T16:33:55.433 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53536369:::benchmark_data_smithi188_113957_object153051:head# 2024-04-23T16:33:55.433 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5354baf7:::benchmark_data_smithi188_113957_object48429:head# 2024-04-23T16:33:55.433 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5354ca12:::benchmark_data_smithi188_113957_object29349:head# 2024-04-23T16:33:55.433 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.434 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5355e808:::benchmark_data_smithi188_113957_object30859:head# 2024-04-23T16:33:55.434 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.434 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5355ee27:::benchmark_data_smithi188_113957_object121892:head# 2024-04-23T16:33:55.434 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.434 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:535623d8:::benchmark_data_smithi188_113957_object24042:head# 2024-04-23T16:33:55.434 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.434 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:535678b6:::benchmark_data_smithi188_113957_object21634:head# 2024-04-23T16:33:55.434 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.434 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5356bbe1:::benchmark_data_smithi188_113957_object126424:head# 2024-04-23T16:33:55.434 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.434 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53573642:::benchmark_data_smithi188_113957_object61806:head# 2024-04-23T16:33:55.434 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.435 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53580881:::benchmark_data_smithi188_113957_object52406:head# 2024-04-23T16:33:55.435 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.435 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53580b2f:::benchmark_data_smithi188_113957_object8675:head# 2024-04-23T16:33:55.435 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.435 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53591c48:::benchmark_data_smithi188_113957_object119224:head# 2024-04-23T16:33:55.435 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.435 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:535985ed:::benchmark_data_smithi188_113957_object88336:head# 2024-04-23T16:33:55.435 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.435 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53598b59:::benchmark_data_smithi188_113957_object81462:head# 2024-04-23T16:33:55.435 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.435 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5359a3bb:::benchmark_data_smithi188_113957_object7396:head# 2024-04-23T16:33:55.435 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.435 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:535a3893:::benchmark_data_smithi188_113957_object78856:head# 2024-04-23T16:33:55.435 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:535a8a86:::benchmark_data_smithi188_113957_object29914:head# 2024-04-23T16:33:55.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:535b064f:::benchmark_data_smithi188_113957_object6617:head# 2024-04-23T16:33:55.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:535b4398:::benchmark_data_smithi188_113957_object108125:head# 2024-04-23T16:33:55.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:535b89ad:::benchmark_data_smithi188_113957_object89744:head# 2024-04-23T16:33:55.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:535c3c08:::benchmark_data_smithi188_113957_object94918:head# 2024-04-23T16:33:55.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:535d671c:::benchmark_data_smithi188_113957_object151814:head# 2024-04-23T16:33:55.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:535edf45:::benchmark_data_smithi188_113957_object148936:head# 2024-04-23T16:33:55.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:535fa0cd:::benchmark_data_smithi188_113957_object70025:head# 2024-04-23T16:33:55.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:535fb5a7:::benchmark_data_smithi188_113957_object52479:head# 2024-04-23T16:33:55.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53612cd7:::benchmark_data_smithi188_113957_object96501:head# 2024-04-23T16:33:55.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53612d79:::benchmark_data_smithi188_113957_object93395:head# 2024-04-23T16:33:55.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:536242b3:::benchmark_data_smithi188_113957_object150957:head# 2024-04-23T16:33:55.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:536356c2:::benchmark_data_smithi188_113957_object50708:head# 2024-04-23T16:33:55.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5364ead0:::benchmark_data_smithi188_113957_object110319:head# 2024-04-23T16:33:55.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.438 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53656ab8:::benchmark_data_smithi188_113957_object84170:head# 2024-04-23T16:33:55.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.438 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5365a1e9:::benchmark_data_smithi188_113957_object2933:head# 2024-04-23T16:33:55.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.438 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5365ca70:::benchmark_data_smithi188_113957_object2650:head# 2024-04-23T16:33:55.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.438 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:536613e1:::benchmark_data_smithi188_113957_object51281:head# 2024-04-23T16:33:55.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.438 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53669e30:::benchmark_data_smithi188_113957_object41040:head# 2024-04-23T16:33:55.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.438 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5366d34d:::benchmark_data_smithi188_113957_object45430:head# 2024-04-23T16:33:55.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53674c30:::benchmark_data_smithi188_113957_object115195:head# 2024-04-23T16:33:55.439 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53678d19:::benchmark_data_smithi188_113957_object134100:head# 2024-04-23T16:33:55.439 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5367fa79:::benchmark_data_smithi188_113957_object30171:head# 2024-04-23T16:33:55.439 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53685e5b:::benchmark_data_smithi188_113957_object7091:head# 2024-04-23T16:33:55.439 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53686cd4:::benchmark_data_smithi188_113957_object133586:head# 2024-04-23T16:33:55.439 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:536891d3:::benchmark_data_smithi188_113957_object37092:head# 2024-04-23T16:33:55.439 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53696d7e:::benchmark_data_smithi188_113957_object13499:head# 2024-04-23T16:33:55.439 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:536bdecb:::benchmark_data_smithi188_113957_object94994:head# 2024-04-23T16:33:55.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:536bf14f:::benchmark_data_smithi188_113957_object39209:head# 2024-04-23T16:33:55.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:536ce97e:::benchmark_data_smithi188_113957_object78652:head# 2024-04-23T16:33:55.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:536d8532:::benchmark_data_smithi188_113957_object129391:head# 2024-04-23T16:33:55.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:536dde01:::benchmark_data_smithi188_113957_object121252:head# 2024-04-23T16:33:55.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:536ed31e:::benchmark_data_smithi188_113957_object15208:head# 2024-04-23T16:33:55.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:536fd1ef:::benchmark_data_smithi188_113957_object101898:head# 2024-04-23T16:33:55.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:536ff8a7:::benchmark_data_smithi188_113957_object769:head# 2024-04-23T16:33:55.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:536ffbb6:::benchmark_data_smithi188_113957_object111896:head# 2024-04-23T16:33:55.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5370bfd3:::benchmark_data_smithi188_113957_object39185:head# 2024-04-23T16:33:55.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53710a40:::benchmark_data_smithi188_113957_object120523:head# 2024-04-23T16:33:55.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:537111c3:::benchmark_data_smithi188_113957_object101345:head# 2024-04-23T16:33:55.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53714283:::benchmark_data_smithi188_113957_object145701:head# 2024-04-23T16:33:55.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53714416:::benchmark_data_smithi188_113957_object36166:head# 2024-04-23T16:33:55.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53714938:::benchmark_data_smithi188_113957_object113547:head# 2024-04-23T16:33:55.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5371af46:::benchmark_data_smithi188_113957_object106870:head# 2024-04-23T16:33:55.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5372a37c:::benchmark_data_smithi188_113957_object63958:head# 2024-04-23T16:33:55.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:537323eb:::benchmark_data_smithi188_113957_object143312:head# 2024-04-23T16:33:55.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53732b79:::benchmark_data_smithi188_113957_object120968:head# 2024-04-23T16:33:55.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5373fbad:::benchmark_data_smithi188_113957_object54201:head# 2024-04-23T16:33:55.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5374000b:::benchmark_data_smithi188_113957_object13967:head# 2024-04-23T16:33:55.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5374dab0:::benchmark_data_smithi188_113957_object85808:head# 2024-04-23T16:33:55.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53751bee:::benchmark_data_smithi188_113957_object4614:head# 2024-04-23T16:33:55.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53752a48:::benchmark_data_smithi188_113957_object105621:head# 2024-04-23T16:33:55.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53757c0b:::benchmark_data_smithi188_113957_object58355:head# 2024-04-23T16:33:55.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53771aec:::benchmark_data_smithi188_113957_object74192:head# 2024-04-23T16:33:55.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53776270:::benchmark_data_smithi188_113957_object124283:head# 2024-04-23T16:33:55.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5377f60f:::benchmark_data_smithi188_113957_object143797:head# 2024-04-23T16:33:55.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53789f23:::benchmark_data_smithi188_113957_object151582:head# 2024-04-23T16:33:55.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5379ef92:::benchmark_data_smithi188_113957_object19360:head# 2024-04-23T16:33:55.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:537a0524:::benchmark_data_smithi188_113957_object78136:head# 2024-04-23T16:33:55.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:537a924a:::benchmark_data_smithi188_113957_object45752:head# 2024-04-23T16:33:55.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:537abe9e:::benchmark_data_smithi188_113957_object32244:head# 2024-04-23T16:33:55.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:537b0520:::benchmark_data_smithi188_113957_object48405:head# 2024-04-23T16:33:55.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:537b53b0:::benchmark_data_smithi188_113957_object32418:head# 2024-04-23T16:33:55.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:537c093a:::benchmark_data_smithi188_113957_object144338:head# 2024-04-23T16:33:55.445 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.445 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:537c48bf:::benchmark_data_smithi188_113957_object124420:head# 2024-04-23T16:33:55.445 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.445 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:537c839c:::benchmark_data_smithi188_113957_object6530:head# 2024-04-23T16:33:55.445 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.445 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:537cdc9a:::benchmark_data_smithi188_113957_object80639:head# 2024-04-23T16:33:55.445 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.445 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:537d3bed:::benchmark_data_smithi188_113957_object144992:head# 2024-04-23T16:33:55.445 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.445 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:537dc7a6:::benchmark_data_smithi188_113957_object56826:head# 2024-04-23T16:33:55.445 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.445 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:537dd1ad:::benchmark_data_smithi188_113957_object104449:head# 2024-04-23T16:33:55.445 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.445 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:537ec1ac:::benchmark_data_smithi188_113957_object28522:head# 2024-04-23T16:33:55.446 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.446 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:537ec70e:::benchmark_data_smithi188_113957_object34803:head# 2024-04-23T16:33:55.446 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.446 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:537f98ff:::benchmark_data_smithi188_113957_object63311:head# 2024-04-23T16:33:55.446 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.446 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5380013d:::benchmark_data_smithi188_113957_object110348:head# 2024-04-23T16:33:55.446 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.446 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5380d6e8:::benchmark_data_smithi188_113957_object137993:head# 2024-04-23T16:33:55.446 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.446 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53811cc2:::benchmark_data_smithi188_113957_object100494:head# 2024-04-23T16:33:55.446 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.446 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53812277:::benchmark_data_smithi188_113957_object79143:head# 2024-04-23T16:33:55.446 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.447 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53817478:::benchmark_data_smithi188_113957_object109116:head# 2024-04-23T16:33:55.447 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.447 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53819a04:::benchmark_data_smithi188_113957_object5115:head# 2024-04-23T16:33:55.447 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.447 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5381f9e8:::benchmark_data_smithi188_113957_object100067:head# 2024-04-23T16:33:55.447 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.447 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53827466:::benchmark_data_smithi188_113957_object49707:head# 2024-04-23T16:33:55.447 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.447 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53828752:::benchmark_data_smithi188_113957_object106893:head# 2024-04-23T16:33:55.447 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.447 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:538310f1:::benchmark_data_smithi188_113957_object33010:head# 2024-04-23T16:33:55.447 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.447 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53832ea5:::benchmark_data_smithi188_113957_object7776:head# 2024-04-23T16:33:55.447 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53837c3d:::benchmark_data_smithi188_113957_object31194:head# 2024-04-23T16:33:55.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5383cf98:::benchmark_data_smithi188_113957_object72780:head# 2024-04-23T16:33:55.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5383e26f:::benchmark_data_smithi188_113957_object97630:head# 2024-04-23T16:33:55.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5384493c:::benchmark_data_smithi188_113957_object137106:head# 2024-04-23T16:33:55.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:538453e0:::benchmark_data_smithi188_113957_object58148:head# 2024-04-23T16:33:55.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:538459f2:::benchmark_data_smithi188_113957_object136113:head# 2024-04-23T16:33:55.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53876e11:::benchmark_data_smithi188_113957_object81101:head# 2024-04-23T16:33:55.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53878a94:::benchmark_data_smithi188_113957_object150694:head# 2024-04-23T16:33:55.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5387ab76:::benchmark_data_smithi188_113957_object37428:head# 2024-04-23T16:33:55.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5387cbca:::benchmark_data_smithi188_113957_object49537:head# 2024-04-23T16:33:55.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5387dbce:::benchmark_data_smithi188_113957_object95858:head# 2024-04-23T16:33:55.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5387e08e:::benchmark_data_smithi188_113957_object20139:head# 2024-04-23T16:33:55.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5388ac9d:::benchmark_data_smithi188_113957_object68801:head# 2024-04-23T16:33:55.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5388b3b6:::benchmark_data_smithi188_113957_object151826:head# 2024-04-23T16:33:55.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5388f6fb:::benchmark_data_smithi188_113957_object141738:head# 2024-04-23T16:33:55.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53897d99:::benchmark_data_smithi188_113957_object118810:head# 2024-04-23T16:33:55.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:538a330f:::benchmark_data_smithi188_113957_object57126:head# 2024-04-23T16:33:55.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:538a4bef:::benchmark_data_smithi188_113957_object4711:head# 2024-04-23T16:33:55.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:538a6d06:::benchmark_data_smithi188_113957_object11294:head# 2024-04-23T16:33:55.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:538a9a9f:::benchmark_data_smithi188_113957_object68011:head# 2024-04-23T16:33:55.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:538b9689:::benchmark_data_smithi188_113957_object81053:head# 2024-04-23T16:33:55.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.451 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:538b9ca6:::benchmark_data_smithi188_113957_object6006:head# 2024-04-23T16:33:55.451 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.451 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:538ca99a:::benchmark_data_smithi188_113957_object92396:head# 2024-04-23T16:33:55.451 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.451 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:538dda59:::benchmark_data_smithi188_113957_object47731:head# 2024-04-23T16:33:55.451 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.451 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:538dfec0:::benchmark_data_smithi188_113957_object50645:head# 2024-04-23T16:33:55.451 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.451 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:538e9906:::benchmark_data_smithi188_113957_object19263:head# 2024-04-23T16:33:55.451 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.451 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:538ef6e3:::benchmark_data_smithi188_113957_object76632:head# 2024-04-23T16:33:55.451 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.451 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:538f54f9:::benchmark_data_smithi188_113957_object72209:head# 2024-04-23T16:33:55.451 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.451 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:538ff93b:::benchmark_data_smithi188_113957_object110335:head# 2024-04-23T16:33:55.452 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.452 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53904e89:::benchmark_data_smithi188_113957_object145443:head# 2024-04-23T16:33:55.452 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.452 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53905376:::benchmark_data_smithi188_113957_object108995:head# 2024-04-23T16:33:55.452 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.452 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:539101ef:::benchmark_data_smithi188_113957_object2781:head# 2024-04-23T16:33:55.452 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.452 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5392330c:::benchmark_data_smithi188_113957_object99563:head# 2024-04-23T16:33:55.452 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.452 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53928c89:::benchmark_data_smithi188_113957_object106841:head# 2024-04-23T16:33:55.452 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.452 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5392e2a7:::benchmark_data_smithi188_113957_object133104:head# 2024-04-23T16:33:55.452 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.452 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5393d880:::benchmark_data_smithi188_113957_object55171:head# 2024-04-23T16:33:55.452 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5393dbdd:::benchmark_data_smithi188_113957_object43606:head# 2024-04-23T16:33:55.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53941fb2:::benchmark_data_smithi188_113957_object6351:head# 2024-04-23T16:33:55.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5395aa39:::benchmark_data_smithi188_113957_object103503:head# 2024-04-23T16:33:55.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53962fa3:::benchmark_data_smithi188_113957_object80932:head# 2024-04-23T16:33:55.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53964364:::benchmark_data_smithi188_113957_object36571:head# 2024-04-23T16:33:55.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5396f75d:::benchmark_data_smithi188_113957_object32190:head# 2024-04-23T16:33:55.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5396ff82:::benchmark_data_smithi188_113957_object22537:head# 2024-04-23T16:33:55.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:539773ea:::benchmark_data_smithi188_113957_object54905:head# 2024-04-23T16:33:55.454 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.454 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5397959b:::benchmark_data_smithi188_113957_object147915:head# 2024-04-23T16:33:55.454 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.454 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5397c471:::benchmark_data_smithi188_113957_object135025:head# 2024-04-23T16:33:55.454 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.454 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5398100f:::benchmark_data_smithi188_113957_object67870:head# 2024-04-23T16:33:55.454 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.454 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5399125a:::benchmark_data_smithi188_113957_object136201:head# 2024-04-23T16:33:55.454 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.454 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53992810:::benchmark_data_smithi188_113957_object94659:head# 2024-04-23T16:33:55.454 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.454 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5399fc23:::benchmark_data_smithi188_113957_object39577:head# 2024-04-23T16:33:55.454 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.454 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:539a7a7f:::benchmark_data_smithi188_113957_object82206:head# 2024-04-23T16:33:55.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:539ad625:::benchmark_data_smithi188_113957_object43097:head# 2024-04-23T16:33:55.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:539b0164:::benchmark_data_smithi188_113957_object83889:head# 2024-04-23T16:33:55.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:539b6f14:::benchmark_data_smithi188_113957_object106304:head# 2024-04-23T16:33:55.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:539b77c1:::benchmark_data_smithi188_113957_object128458:head# 2024-04-23T16:33:55.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:539ba7c0:::benchmark_data_smithi188_113957_object19911:head# 2024-04-23T16:33:55.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:539bdde9:::benchmark_data_smithi188_113957_object41409:head# 2024-04-23T16:33:55.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:539c1efd:::benchmark_data_smithi188_113957_object64907:head# 2024-04-23T16:33:55.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:539c8eaa:::benchmark_data_smithi188_113957_object32249:head# 2024-04-23T16:33:55.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:539d2b4e:::benchmark_data_smithi188_113957_object2160:head# 2024-04-23T16:33:55.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:539e1b50:::benchmark_data_smithi188_113957_object37660:head# 2024-04-23T16:33:55.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:539e327e:::benchmark_data_smithi188_113957_object46744:head# 2024-04-23T16:33:55.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:539ea435:::benchmark_data_smithi188_113957_object96958:head# 2024-04-23T16:33:55.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:539f4c71:::benchmark_data_smithi188_113957_object76956:head# 2024-04-23T16:33:55.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:539f637a:::benchmark_data_smithi188_113957_object4020:head# 2024-04-23T16:33:55.457 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.457 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a016fe:::benchmark_data_smithi188_113957_object26434:head# 2024-04-23T16:33:55.457 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.457 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a0ce4a:::benchmark_data_smithi188_113957_object59397:head# 2024-04-23T16:33:55.457 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.457 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a0df40:::benchmark_data_smithi188_113957_object115258:head# 2024-04-23T16:33:55.457 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.457 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a0fa08:::benchmark_data_smithi188_113957_object17818:head# 2024-04-23T16:33:55.457 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.457 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a284ee:::benchmark_data_smithi188_113957_object58284:head# 2024-04-23T16:33:55.457 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.457 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a32b12:::benchmark_data_smithi188_113957_object117858:head# 2024-04-23T16:33:55.457 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.457 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a42ddc:::benchmark_data_smithi188_113957_object23851:head# 2024-04-23T16:33:55.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a4f02b:::benchmark_data_smithi188_113957_object22518:head# 2024-04-23T16:33:55.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a4f116:::benchmark_data_smithi188_113957_object30238:head# 2024-04-23T16:33:55.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a62f92:::benchmark_data_smithi188_113957_object34139:head# 2024-04-23T16:33:55.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a67f49:::benchmark_data_smithi188_113957_object4913:head# 2024-04-23T16:33:55.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a6ca1a:::benchmark_data_smithi188_113957_object152903:head# 2024-04-23T16:33:55.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a71b93:::benchmark_data_smithi188_113957_object145811:head# 2024-04-23T16:33:55.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a73c0b:::benchmark_data_smithi188_113957_object88723:head# 2024-04-23T16:33:55.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a78acc:::benchmark_data_smithi188_113957_object82425:head# 2024-04-23T16:33:55.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a7a94a:::benchmark_data_smithi188_113957_object110251:head# 2024-04-23T16:33:55.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a846f1:::benchmark_data_smithi188_113957_object124910:head# 2024-04-23T16:33:55.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a87776:::benchmark_data_smithi188_113957_object73487:head# 2024-04-23T16:33:55.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a8a02a:::benchmark_data_smithi188_113957_object2036:head# 2024-04-23T16:33:55.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a96911:::benchmark_data_smithi188_113957_object137594:head# 2024-04-23T16:33:55.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a9d8f8:::benchmark_data_smithi188_113957_object125146:head# 2024-04-23T16:33:55.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53a9e25f:::benchmark_data_smithi188_113957_object152008:head# 2024-04-23T16:33:55.460 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.460 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53aa0cb9:::benchmark_data_smithi188_113957_object134755:head# 2024-04-23T16:33:55.460 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.460 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53aa5a3e:::benchmark_data_smithi188_113957_object134285:head# 2024-04-23T16:33:55.460 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.460 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53aab04a:::benchmark_data_smithi188_113957_object36433:head# 2024-04-23T16:33:55.460 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.460 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53aabe27:::benchmark_data_smithi188_113957_object124624:head# 2024-04-23T16:33:55.460 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.460 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53aac26c:::benchmark_data_smithi188_113957_object108854:head# 2024-04-23T16:33:55.460 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.460 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53aafe3c:::benchmark_data_smithi188_113957_object47836:head# 2024-04-23T16:33:55.460 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.460 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ab1d54:::benchmark_data_smithi188_113957_object82316:head# 2024-04-23T16:33:55.461 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.461 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ab25cb:::benchmark_data_smithi188_113957_object101249:head# 2024-04-23T16:33:55.461 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.461 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ab4910:::benchmark_data_smithi188_113957_object26795:head# 2024-04-23T16:33:55.461 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.461 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ac576d:::benchmark_data_smithi188_113957_object30344:head# 2024-04-23T16:33:55.461 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.461 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ac9847:::benchmark_data_smithi188_113957_object63366:head# 2024-04-23T16:33:55.461 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.461 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ad2065:::benchmark_data_smithi188_113957_object52801:head# 2024-04-23T16:33:55.461 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.461 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ae01a7:::benchmark_data_smithi188_113957_object105406:head# 2024-04-23T16:33:55.461 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.461 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53af323e:::benchmark_data_smithi188_113957_object18881:head# 2024-04-23T16:33:55.461 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b0304e:::benchmark_data_smithi188_113957_object9427:head# 2024-04-23T16:33:55.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b0334a:::benchmark_data_smithi188_113957_object139176:head# 2024-04-23T16:33:55.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b04445:::benchmark_data_smithi188_113957_object113281:head# 2024-04-23T16:33:55.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b07c17:::benchmark_data_smithi188_113957_object66142:head# 2024-04-23T16:33:55.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b08efb:::benchmark_data_smithi188_113957_object79570:head# 2024-04-23T16:33:55.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b0c9d9:::benchmark_data_smithi188_113957_object81357:head# 2024-04-23T16:33:55.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b0ce72:::benchmark_data_smithi188_113957_object72138:head# 2024-04-23T16:33:55.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b1128d:::benchmark_data_smithi188_113957_object77189:head# 2024-04-23T16:33:55.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b1ea14:::benchmark_data_smithi188_113957_object96051:head# 2024-04-23T16:33:55.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b20e67:::benchmark_data_smithi188_113957_object146917:head# 2024-04-23T16:33:55.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b241fe:::benchmark_data_smithi188_113957_object151659:head# 2024-04-23T16:33:55.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b27f5d:::benchmark_data_smithi188_113957_object38417:head# 2024-04-23T16:33:55.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b2a699:::benchmark_data_smithi188_113957_object104594:head# 2024-04-23T16:33:55.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b2becf:::benchmark_data_smithi188_113957_object110520:head# 2024-04-23T16:33:55.464 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.464 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b302a1:::benchmark_data_smithi188_113957_object80980:head# 2024-04-23T16:33:55.464 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.464 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b314b2:::benchmark_data_smithi188_113957_object23364:head# 2024-04-23T16:33:55.464 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.464 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b371b8:::benchmark_data_smithi188_113957_object28141:head# 2024-04-23T16:33:55.464 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.464 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b4b98f:::benchmark_data_smithi188_113957_object88944:head# 2024-04-23T16:33:55.464 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.464 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b52a64:::benchmark_data_smithi188_113957_object90779:head# 2024-04-23T16:33:55.464 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.464 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b5a372:::benchmark_data_smithi188_113957_object107455:head# 2024-04-23T16:33:55.464 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b5e523:::benchmark_data_smithi188_113957_object77325:head# 2024-04-23T16:33:55.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b6f246:::benchmark_data_smithi188_113957_object55441:head# 2024-04-23T16:33:55.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b79d05:::benchmark_data_smithi188_113957_object55371:head# 2024-04-23T16:33:55.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b82249:::benchmark_data_smithi188_113957_object100941:head# 2024-04-23T16:33:55.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b85b54:::benchmark_data_smithi188_113957_object99389:head# 2024-04-23T16:33:55.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b866f6:::benchmark_data_smithi188_113957_object126084:head# 2024-04-23T16:33:55.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b88583:::benchmark_data_smithi188_113957_object111081:head# 2024-04-23T16:33:55.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.466 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53b9c617:::benchmark_data_smithi188_113957_object120449:head# 2024-04-23T16:33:55.466 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.466 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ba42ff:::benchmark_data_smithi188_113957_object137726:head# 2024-04-23T16:33:55.466 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.466 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53babaa7:::benchmark_data_smithi188_113957_object135670:head# 2024-04-23T16:33:55.466 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.466 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53bc989e:::benchmark_data_smithi188_113957_object142380:head# 2024-04-23T16:33:55.466 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.466 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53bcac3b:::benchmark_data_smithi188_113957_object146904:head# 2024-04-23T16:33:55.466 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.466 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53bd0f9f:::benchmark_data_smithi188_113957_object94471:head# 2024-04-23T16:33:55.466 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.466 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53bd9ac3:::benchmark_data_smithi188_113957_object134499:head# 2024-04-23T16:33:55.466 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.467 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53bdb56f:::benchmark_data_smithi188_113957_object150217:head# 2024-04-23T16:33:55.467 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.467 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53bdcfc5:::benchmark_data_smithi188_113957_object97722:head# 2024-04-23T16:33:55.467 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.467 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53be3dd4:::benchmark_data_smithi188_113957_object51915:head# 2024-04-23T16:33:55.467 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.467 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53be94de:::benchmark_data_smithi188_113957_object91875:head# 2024-04-23T16:33:55.467 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.467 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53be9aa5:::benchmark_data_smithi188_113957_object27692:head# 2024-04-23T16:33:55.467 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.467 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53bef699:::benchmark_data_smithi188_113957_object62976:head# 2024-04-23T16:33:55.467 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.467 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53bf7985:::benchmark_data_smithi188_113957_object24650:head# 2024-04-23T16:33:55.467 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.468 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c09a39:::benchmark_data_smithi188_113957_object100282:head# 2024-04-23T16:33:55.468 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.468 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c12ed0:::benchmark_data_smithi188_113957_object53524:head# 2024-04-23T16:33:55.468 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.468 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c26b49:::benchmark_data_smithi188_113957_object113920:head# 2024-04-23T16:33:55.468 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.468 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c2b040:::benchmark_data_smithi188_113957_object123998:head# 2024-04-23T16:33:55.468 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.475 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:55.543 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:55.576 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:55.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:55.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:55.602 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c3670d:::benchmark_data_smithi188_113957_object81565:head# 2024-04-23T16:33:55.602 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.602 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c3690d:::benchmark_data_smithi188_113957_object23741:head# 2024-04-23T16:33:55.602 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.602 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c37b80:::benchmark_data_smithi188_113957_object96969:head# 2024-04-23T16:33:55.602 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.602 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c3a963:::benchmark_data_smithi188_113957_object41507:head# 2024-04-23T16:33:55.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c3d3bc:::benchmark_data_smithi188_113957_object62098:head# 2024-04-23T16:33:55.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c3ff6c:::benchmark_data_smithi188_113957_object152397:head# 2024-04-23T16:33:55.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c47db9:::benchmark_data_smithi188_113957_object128302:head# 2024-04-23T16:33:55.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c4f953:::benchmark_data_smithi188_113957_object10113:head# 2024-04-23T16:33:55.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c5721c:::benchmark_data_smithi188_113957_object5079:head# 2024-04-23T16:33:55.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c652d3:::benchmark_data_smithi188_113957_object71216:head# 2024-04-23T16:33:55.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c69046:::benchmark_data_smithi188_113957_object83399:head# 2024-04-23T16:33:55.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c6b85d:::benchmark_data_smithi188_113957_object6809:head# 2024-04-23T16:33:55.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c75030:::benchmark_data_smithi188_113957_object47021:head# 2024-04-23T16:33:55.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c7991f:::benchmark_data_smithi188_113957_object85011:head# 2024-04-23T16:33:55.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c84f8c:::benchmark_data_smithi188_113957_object54871:head# 2024-04-23T16:33:55.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c89b06:::benchmark_data_smithi188_113957_object78465:head# 2024-04-23T16:33:55.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c963be:::benchmark_data_smithi188_113957_object80377:head# 2024-04-23T16:33:55.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53c9b9c5:::benchmark_data_smithi188_113957_object11567:head# 2024-04-23T16:33:55.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.605 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53cb5c34:::benchmark_data_smithi188_113957_object114502:head# 2024-04-23T16:33:55.605 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.605 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ccbb08:::benchmark_data_smithi188_113957_object28459:head# 2024-04-23T16:33:55.605 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.605 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ccf08d:::benchmark_data_smithi188_113957_object32411:head# 2024-04-23T16:33:55.605 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.605 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53cd1a6f:::benchmark_data_smithi188_113957_object137455:head# 2024-04-23T16:33:55.605 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.605 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53cd5cf2:::benchmark_data_smithi188_113957_object125990:head# 2024-04-23T16:33:55.605 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.605 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53cd61a1:::benchmark_data_smithi188_113957_object124623:head# 2024-04-23T16:33:55.605 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.605 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53cec3af:::benchmark_data_smithi188_113957_object80759:head# 2024-04-23T16:33:55.605 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.605 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53cf1a61:::benchmark_data_smithi188_113957_object609:head# 2024-04-23T16:33:55.606 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.606 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53cf6a19:::benchmark_data_smithi188_113957_object132549:head# 2024-04-23T16:33:55.606 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.606 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53cf6cd1:::benchmark_data_smithi188_113957_object10909:head# 2024-04-23T16:33:55.606 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.606 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53cf7dda:::benchmark_data_smithi188_113957_object79620:head# 2024-04-23T16:33:55.606 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.606 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53cf9efd:::benchmark_data_smithi188_113957_object90298:head# 2024-04-23T16:33:55.606 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.606 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53cff3b5:::benchmark_data_smithi188_113957_object29496:head# 2024-04-23T16:33:55.606 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.606 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53cffbdf:::benchmark_data_smithi188_113957_object4266:head# 2024-04-23T16:33:55.607 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.607 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d085e9:::benchmark_data_smithi188_113957_object116525:head# 2024-04-23T16:33:55.607 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.607 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d0b12c:::benchmark_data_smithi188_113957_object139733:head# 2024-04-23T16:33:55.607 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.607 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d0be85:::benchmark_data_smithi188_113957_object57990:head# 2024-04-23T16:33:55.607 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.607 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d10da1:::benchmark_data_smithi188_113957_object93697:head# 2024-04-23T16:33:55.607 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.607 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d1dffd:::benchmark_data_smithi188_113957_object33367:head# 2024-04-23T16:33:55.607 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.607 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d1f577:::benchmark_data_smithi188_113957_object128412:head# 2024-04-23T16:33:55.607 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.607 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d1fc01:::benchmark_data_smithi188_113957_object79758:head# 2024-04-23T16:33:55.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d24c27:::benchmark_data_smithi188_113957_object141707:head# 2024-04-23T16:33:55.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d32328:::benchmark_data_smithi188_113957_object109627:head# 2024-04-23T16:33:55.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d32346:::benchmark_data_smithi188_113957_object114263:head# 2024-04-23T16:33:55.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d371ec:::benchmark_data_smithi188_113957_object34932:head# 2024-04-23T16:33:55.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d3aaa6:::benchmark_data_smithi188_113957_object27555:head# 2024-04-23T16:33:55.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d40fcf:::benchmark_data_smithi188_113957_object13794:head# 2024-04-23T16:33:55.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d526c8:::benchmark_data_smithi188_113957_object128472:head# 2024-04-23T16:33:55.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d5527d:::benchmark_data_smithi188_113957_object121120:head# 2024-04-23T16:33:55.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d57f44:::benchmark_data_smithi188_113957_object80555:head# 2024-04-23T16:33:55.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d586b6:::benchmark_data_smithi188_113957_object59828:head# 2024-04-23T16:33:55.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d58edc:::benchmark_data_smithi188_113957_object14822:head# 2024-04-23T16:33:55.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d6728b:::benchmark_data_smithi188_113957_object9343:head# 2024-04-23T16:33:55.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d7ed42:::benchmark_data_smithi188_113957_object24316:head# 2024-04-23T16:33:55.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d7f047:::benchmark_data_smithi188_113957_object126707:head# 2024-04-23T16:33:55.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d88ba7:::benchmark_data_smithi188_113957_object64408:head# 2024-04-23T16:33:55.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53d99e53:::benchmark_data_smithi188_113957_object14176:head# 2024-04-23T16:33:55.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53da3e8d:::benchmark_data_smithi188_113957_object77904:head# 2024-04-23T16:33:55.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53dac305:::benchmark_data_smithi188_113957_object51222:head# 2024-04-23T16:33:55.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53db1b4c:::benchmark_data_smithi188_113957_object32556:head# 2024-04-23T16:33:55.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53db78fe:::benchmark_data_smithi188_113957_object108742:head# 2024-04-23T16:33:55.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53db8ce9:::benchmark_data_smithi188_113957_object88620:head# 2024-04-23T16:33:55.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53db9250:::benchmark_data_smithi188_113957_object83158:head# 2024-04-23T16:33:55.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53dbb82a:::benchmark_data_smithi188_113957_object51135:head# 2024-04-23T16:33:55.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53dc18c1:::benchmark_data_smithi188_113957_object109447:head# 2024-04-23T16:33:55.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53dc1d58:::benchmark_data_smithi188_113957_object142194:head# 2024-04-23T16:33:55.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53dca14a:::benchmark_data_smithi188_113957_object77259:head# 2024-04-23T16:33:55.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53dd0db1:::benchmark_data_smithi188_113957_object32477:head# 2024-04-23T16:33:55.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53dd2d96:::benchmark_data_smithi188_113957_object133371:head# 2024-04-23T16:33:55.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53dd6182:::benchmark_data_smithi188_113957_object79980:head# 2024-04-23T16:33:55.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ddd930:::benchmark_data_smithi188_113957_object115324:head# 2024-04-23T16:33:55.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ddddec:::benchmark_data_smithi188_113957_object108536:head# 2024-04-23T16:33:55.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53deab25:::benchmark_data_smithi188_113957_object21195:head# 2024-04-23T16:33:55.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53dec1df:::benchmark_data_smithi188_113957_object108517:head# 2024-04-23T16:33:55.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53dfb314:::benchmark_data_smithi188_113957_object80623:head# 2024-04-23T16:33:55.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53dfcba7:::benchmark_data_smithi188_113957_object12735:head# 2024-04-23T16:33:55.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53dfee3e:::benchmark_data_smithi188_113957_object139055:head# 2024-04-23T16:33:55.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53dff203:::benchmark_data_smithi188_113957_object65556:head# 2024-04-23T16:33:55.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53e046c7:::benchmark_data_smithi188_113957_object46236:head# 2024-04-23T16:33:55.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53e0791e:::benchmark_data_smithi188_113957_object98279:head# 2024-04-23T16:33:55.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53e0e100:::benchmark_data_smithi188_113957_object153194:head# 2024-04-23T16:33:55.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53e15379:::benchmark_data_smithi188_113957_object146676:head# 2024-04-23T16:33:55.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53e17960:::benchmark_data_smithi188_113957_object42382:head# 2024-04-23T16:33:55.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53e1e144:::benchmark_data_smithi188_113957_object141926:head# 2024-04-23T16:33:55.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53e1e52f:::benchmark_data_smithi188_113957_object89697:head# 2024-04-23T16:33:55.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53e21ba7:::benchmark_data_smithi188_113957_object151379:head# 2024-04-23T16:33:55.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53e30a38:::benchmark_data_smithi188_113957_object15960:head# 2024-04-23T16:33:55.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53e44431:::benchmark_data_smithi188_113957_object92807:head# 2024-04-23T16:33:55.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53e4d5fa:::benchmark_data_smithi188_113957_object1395:head# 2024-04-23T16:33:55.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53e521da:::benchmark_data_smithi188_113957_object145851:head# 2024-04-23T16:33:55.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53e545d9:::benchmark_data_smithi188_113957_object42823:head# 2024-04-23T16:33:55.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53e5cb7f:::benchmark_data_smithi188_113957_object93317:head# 2024-04-23T16:33:55.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53e5d49e:::benchmark_data_smithi188_113957_object148554:head# 2024-04-23T16:33:55.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53e63439:::benchmark_data_smithi188_113957_object97387:head# 2024-04-23T16:33:55.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53e82619:::benchmark_data_smithi188_113957_object82939:head# 2024-04-23T16:33:55.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53e8575e:::benchmark_data_smithi188_113957_object118678:head# 2024-04-23T16:33:55.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53e8d136:::benchmark_data_smithi188_113957_object88936:head# 2024-04-23T16:33:55.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53eabdf3:::benchmark_data_smithi188_113957_object84078:head# 2024-04-23T16:33:55.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53eabe2d:::benchmark_data_smithi188_113957_object72500:head# 2024-04-23T16:33:55.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53eae542:::benchmark_data_smithi188_113957_object72846:head# 2024-04-23T16:33:55.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53eb638c:::benchmark_data_smithi188_113957_object36760:head# 2024-04-23T16:33:55.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53eb9be8:::benchmark_data_smithi188_113957_object24267:head# 2024-04-23T16:33:55.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53eb9dd7:::benchmark_data_smithi188_113957_object139963:head# 2024-04-23T16:33:55.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ec7250:::benchmark_data_smithi188_113957_object141653:head# 2024-04-23T16:33:55.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53eca293:::benchmark_data_smithi188_113957_object39982:head# 2024-04-23T16:33:55.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53eca447:::benchmark_data_smithi188_113957_object89890:head# 2024-04-23T16:33:55.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ecb91f:::benchmark_data_smithi188_113957_object28235:head# 2024-04-23T16:33:55.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ed0fba:::benchmark_data_smithi188_113957_object148845:head# 2024-04-23T16:33:55.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ed441e:::benchmark_data_smithi188_113957_object108563:head# 2024-04-23T16:33:55.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ed55f5:::benchmark_data_smithi188_113957_object9969:head# 2024-04-23T16:33:55.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ee6fe6:::benchmark_data_smithi188_113957_object141828:head# 2024-04-23T16:33:55.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53eedf28:::benchmark_data_smithi188_113957_object133184:head# 2024-04-23T16:33:55.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53eee072:::benchmark_data_smithi188_113957_object62486:head# 2024-04-23T16:33:55.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53eee251:::benchmark_data_smithi188_113957_object143094:head# 2024-04-23T16:33:55.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ef3417:::benchmark_data_smithi188_113957_object21240:head# 2024-04-23T16:33:55.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f02958:::benchmark_data_smithi188_113957_object11154:head# 2024-04-23T16:33:55.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f05463:::benchmark_data_smithi188_113957_object49200:head# 2024-04-23T16:33:55.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f07e59:::benchmark_data_smithi188_113957_object38409:head# 2024-04-23T16:33:55.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f0cca5:::benchmark_data_smithi188_113957_object72049:head# 2024-04-23T16:33:55.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f0d8a2:::benchmark_data_smithi188_113957_object38859:head# 2024-04-23T16:33:55.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f0e1fa:::benchmark_data_smithi188_113957_object117498:head# 2024-04-23T16:33:55.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f0eb17:::benchmark_data_smithi188_113957_object118415:head# 2024-04-23T16:33:55.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f13274:::benchmark_data_smithi188_113957_object136908:head# 2024-04-23T16:33:55.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f1b50a:::benchmark_data_smithi188_113957_object49191:head# 2024-04-23T16:33:55.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f2c277:::benchmark_data_smithi188_113957_object49976:head# 2024-04-23T16:33:55.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f2d1ad:::benchmark_data_smithi188_113957_object87655:head# 2024-04-23T16:33:55.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f37936:::benchmark_data_smithi188_113957_object101087:head# 2024-04-23T16:33:55.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f39d66:::benchmark_data_smithi188_113957_object68598:head# 2024-04-23T16:33:55.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f3bbdc:::benchmark_data_smithi188_113957_object134348:head# 2024-04-23T16:33:55.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f5c55a:::benchmark_data_smithi188_113957_object65868:head# 2024-04-23T16:33:55.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f606f9:::benchmark_data_smithi188_113957_object41048:head# 2024-04-23T16:33:55.621 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.621 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f61e2a:::benchmark_data_smithi188_113957_object100436:head# 2024-04-23T16:33:55.621 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.621 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f62008:::benchmark_data_smithi188_113957_object85445:head# 2024-04-23T16:33:55.621 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.621 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f63c78:::benchmark_data_smithi188_113957_object126191:head# 2024-04-23T16:33:55.621 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.621 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f6ec35:::benchmark_data_smithi188_113957_object116268:head# 2024-04-23T16:33:55.621 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.621 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f7ea13:::benchmark_data_smithi188_113957_object43039:head# 2024-04-23T16:33:55.621 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.621 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f8fc9f:::benchmark_data_smithi188_113957_object131649:head# 2024-04-23T16:33:55.621 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f95c2b:::benchmark_data_smithi188_113957_object99876:head# 2024-04-23T16:33:55.622 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53f9f591:::benchmark_data_smithi188_113957_object66532:head# 2024-04-23T16:33:55.622 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53fa8841:::benchmark_data_smithi188_113957_object113093:head# 2024-04-23T16:33:55.622 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53fa916e:::benchmark_data_smithi188_113957_object38581:head# 2024-04-23T16:33:55.622 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53fbaa3a:::benchmark_data_smithi188_113957_object144446:head# 2024-04-23T16:33:55.622 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53fd7e4f:::benchmark_data_smithi188_113957_object88424:head# 2024-04-23T16:33:55.622 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53fd9fe5:::benchmark_data_smithi188_113957_object98770:head# 2024-04-23T16:33:55.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53fe0752:::benchmark_data_smithi188_113957_object105648:head# 2024-04-23T16:33:55.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53fe334b:::benchmark_data_smithi188_113957_object60886:head# 2024-04-23T16:33:55.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53fee7f2:::benchmark_data_smithi188_113957_object110151:head# 2024-04-23T16:33:55.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ff4181:::benchmark_data_smithi188_113957_object138996:head# 2024-04-23T16:33:55.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ff44c7:::benchmark_data_smithi188_113957_object38759:head# 2024-04-23T16:33:55.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ff46a9:::benchmark_data_smithi188_113957_object7392:head# 2024-04-23T16:33:55.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:53ffa6cb:::benchmark_data_smithi188_113957_object5752:head# 2024-04-23T16:33:55.624 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.624 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54004db2:::benchmark_data_smithi188_113957_object108555:head# 2024-04-23T16:33:55.624 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.624 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5401836f:::benchmark_data_smithi188_113957_object96866:head# 2024-04-23T16:33:55.624 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.624 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540197de:::benchmark_data_smithi188_113957_object81282:head# 2024-04-23T16:33:55.624 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.624 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54032504:::benchmark_data_smithi188_113957_object36298:head# 2024-04-23T16:33:55.624 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.624 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54035435:::benchmark_data_smithi188_113957_object31466:head# 2024-04-23T16:33:55.624 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.624 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5403ac8f:::benchmark_data_smithi188_113957_object58269:head# 2024-04-23T16:33:55.624 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.625 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5404ab11:::benchmark_data_smithi188_113957_object84392:head# 2024-04-23T16:33:55.625 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.625 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5404d004:::benchmark_data_smithi188_113957_object26409:head# 2024-04-23T16:33:55.625 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.625 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54053f7a:::benchmark_data_smithi188_113957_object21281:head# 2024-04-23T16:33:55.625 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.625 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54054a3d:::benchmark_data_smithi188_113957_object33806:head# 2024-04-23T16:33:55.625 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.625 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54056445:::benchmark_data_smithi188_113957_object136785:head# 2024-04-23T16:33:55.625 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.625 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5405677d:::benchmark_data_smithi188_113957_object105745:head# 2024-04-23T16:33:55.625 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.625 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5405b5ab:::benchmark_data_smithi188_113957_object44928:head# 2024-04-23T16:33:55.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5405e303:::benchmark_data_smithi188_113957_object2687:head# 2024-04-23T16:33:55.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540633b3:::benchmark_data_smithi188_113957_object120208:head# 2024-04-23T16:33:55.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54065216:::benchmark_data_smithi188_113957_object79345:head# 2024-04-23T16:33:55.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54065258:::benchmark_data_smithi188_113957_object90347:head# 2024-04-23T16:33:55.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54065f53:::benchmark_data_smithi188_113957_object59290:head# 2024-04-23T16:33:55.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5406aa54:::benchmark_data_smithi188_113957_object11475:head# 2024-04-23T16:33:55.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5406c9b4:::benchmark_data_smithi188_113957_object9530:head# 2024-04-23T16:33:55.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5406de04:::benchmark_data_smithi188_113957_object8037:head# 2024-04-23T16:33:55.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5406eaa7:::benchmark_data_smithi188_113957_object34053:head# 2024-04-23T16:33:55.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5406f2d8:::benchmark_data_smithi188_113957_object20504:head# 2024-04-23T16:33:55.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5408a1af:::benchmark_data_smithi188_113957_object22479:head# 2024-04-23T16:33:55.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5408c311:::benchmark_data_smithi188_113957_object95091:head# 2024-04-23T16:33:55.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5409234a:::benchmark_data_smithi188_113957_object77516:head# 2024-04-23T16:33:55.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.628 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54098ded:::benchmark_data_smithi188_113957_object55099:head# 2024-04-23T16:33:55.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.628 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54099d68:::benchmark_data_smithi188_113957_object62373:head# 2024-04-23T16:33:55.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.628 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5409a451:::benchmark_data_smithi188_113957_object65094:head# 2024-04-23T16:33:55.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.628 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5409f9d8:::benchmark_data_smithi188_113957_object39884:head# 2024-04-23T16:33:55.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.628 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540a18d1:::benchmark_data_smithi188_113957_object135392:head# 2024-04-23T16:33:55.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.628 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540a5346:::benchmark_data_smithi188_113957_object125236:head# 2024-04-23T16:33:55.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540a654e:::benchmark_data_smithi188_113957_object147370:head# 2024-04-23T16:33:55.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540abdf8:::benchmark_data_smithi188_113957_object64537:head# 2024-04-23T16:33:55.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540accdc:::benchmark_data_smithi188_113957_object46273:head# 2024-04-23T16:33:55.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540b509f:::benchmark_data_smithi188_113957_object138619:head# 2024-04-23T16:33:55.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540b9db8:::benchmark_data_smithi188_113957_object76917:head# 2024-04-23T16:33:55.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540bda55:::benchmark_data_smithi188_113957_object146344:head# 2024-04-23T16:33:55.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540c08ec:::benchmark_data_smithi188_113957_object20038:head# 2024-04-23T16:33:55.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.630 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540c69f3:::benchmark_data_smithi188_113957_object30586:head# 2024-04-23T16:33:55.630 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.630 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540c912b:::benchmark_data_smithi188_113957_object142326:head# 2024-04-23T16:33:55.630 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.630 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540dddff:::benchmark_data_smithi188_113957_object122631:head# 2024-04-23T16:33:55.630 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.630 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540e2057:::benchmark_data_smithi188_113957_object87880:head# 2024-04-23T16:33:55.630 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.630 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540e2bff:::benchmark_data_smithi188_113957_object142220:head# 2024-04-23T16:33:55.630 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.630 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540e8177:::benchmark_data_smithi188_113957_object104824:head# 2024-04-23T16:33:55.630 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.631 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540e9218:::benchmark_data_smithi188_113957_object104808:head# 2024-04-23T16:33:55.631 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.631 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540efb7e:::benchmark_data_smithi188_113957_object60784:head# 2024-04-23T16:33:55.631 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.631 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540fd7ae:::benchmark_data_smithi188_113957_object24861:head# 2024-04-23T16:33:55.631 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.631 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:540fde04:::benchmark_data_smithi188_113957_object81596:head# 2024-04-23T16:33:55.631 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.631 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5410023c:::benchmark_data_smithi188_113957_object111397:head# 2024-04-23T16:33:55.631 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.631 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5410b1ca:::benchmark_data_smithi188_113957_object95362:head# 2024-04-23T16:33:55.631 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.631 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541153c5:::benchmark_data_smithi188_113957_object31065:head# 2024-04-23T16:33:55.631 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.631 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54121aca:::benchmark_data_smithi188_113957_object59837:head# 2024-04-23T16:33:55.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5412369f:::benchmark_data_smithi188_113957_object88629:head# 2024-04-23T16:33:55.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5412c223:::benchmark_data_smithi188_113957_object97549:head# 2024-04-23T16:33:55.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54130019:::benchmark_data_smithi188_113957_object52890:head# 2024-04-23T16:33:55.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5413098e:::benchmark_data_smithi188_113957_object114185:head# 2024-04-23T16:33:55.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54136255:::benchmark_data_smithi188_113957_object78470:head# 2024-04-23T16:33:55.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541469c9:::benchmark_data_smithi188_113957_object78165:head# 2024-04-23T16:33:55.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.633 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5414e889:::benchmark_data_smithi188_113957_object148007:head# 2024-04-23T16:33:55.633 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.633 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54154607:::benchmark_data_smithi188_113957_object63544:head# 2024-04-23T16:33:55.633 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.633 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54175af9:::benchmark_data_smithi188_113957_object106141:head# 2024-04-23T16:33:55.633 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.633 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54176973:::benchmark_data_smithi188_113957_object14248:head# 2024-04-23T16:33:55.633 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.633 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54178271:::benchmark_data_smithi188_113957_object32867:head# 2024-04-23T16:33:55.633 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.633 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5417966a:::benchmark_data_smithi188_113957_object117668:head# 2024-04-23T16:33:55.633 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.633 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5417b6c6:::benchmark_data_smithi188_113957_object112920:head# 2024-04-23T16:33:55.633 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5417cf44:::benchmark_data_smithi188_113957_object86865:head# 2024-04-23T16:33:55.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541815eb:::benchmark_data_smithi188_113957_object110458:head# 2024-04-23T16:33:55.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541945f9:::benchmark_data_smithi188_113957_object100647:head# 2024-04-23T16:33:55.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54196ec5:::benchmark_data_smithi188_113957_object122101:head# 2024-04-23T16:33:55.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541a1b0b:::benchmark_data_smithi188_113957_object8534:head# 2024-04-23T16:33:55.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541a416a:::benchmark_data_smithi188_113957_object90195:head# 2024-04-23T16:33:55.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541a872c:::benchmark_data_smithi188_113957_object89245:head# 2024-04-23T16:33:55.635 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.635 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541ae0e1:::benchmark_data_smithi188_113957_object13059:head# 2024-04-23T16:33:55.635 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.635 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541af46a:::benchmark_data_smithi188_113957_object69493:head# 2024-04-23T16:33:55.635 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.635 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541af990:::benchmark_data_smithi188_113957_object128162:head# 2024-04-23T16:33:55.635 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.635 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541b039a:::benchmark_data_smithi188_113957_object58837:head# 2024-04-23T16:33:55.635 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.635 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541b3b57:::benchmark_data_smithi188_113957_object70619:head# 2024-04-23T16:33:55.635 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.635 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541bb0aa:::benchmark_data_smithi188_113957_object103076:head# 2024-04-23T16:33:55.635 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.635 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541c2014:::benchmark_data_smithi188_113957_object81721:head# 2024-04-23T16:33:55.636 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.636 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541c6374:::benchmark_data_smithi188_113957_object107445:head# 2024-04-23T16:33:55.636 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.636 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541e81c9:::benchmark_data_smithi188_113957_object46460:head# 2024-04-23T16:33:55.636 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.636 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541ed1af:::benchmark_data_smithi188_113957_object142126:head# 2024-04-23T16:33:55.636 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.636 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541f0c71:::benchmark_data_smithi188_113957_object133232:head# 2024-04-23T16:33:55.636 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.637 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541f3525:::benchmark_data_smithi188_113957_object128091:head# 2024-04-23T16:33:55.637 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.637 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541f597b:::benchmark_data_smithi188_113957_object30197:head# 2024-04-23T16:33:55.637 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.637 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541f8949:::benchmark_data_smithi188_113957_object47223:head# 2024-04-23T16:33:55.637 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.637 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541f9e3e:::benchmark_data_smithi188_113957_object117667:head# 2024-04-23T16:33:55.637 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.637 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541faf59:::benchmark_data_smithi188_113957_object105538:head# 2024-04-23T16:33:55.637 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.637 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541fb3f1:::benchmark_data_smithi188_113957_object67542:head# 2024-04-23T16:33:55.637 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.637 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541fe30e:::benchmark_data_smithi188_113957_object9156:head# 2024-04-23T16:33:55.637 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541ff0e0:::benchmark_data_smithi188_113957_object72171:head# 2024-04-23T16:33:55.638 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:541ff31f:::benchmark_data_smithi188_113957_object145126:head# 2024-04-23T16:33:55.638 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54208c91:::benchmark_data_smithi188_113957_object38317:head# 2024-04-23T16:33:55.638 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5420a1d3:::benchmark_data_smithi188_113957_object139234:head# 2024-04-23T16:33:55.638 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:542179a7:::benchmark_data_smithi188_113957_object57277:head# 2024-04-23T16:33:55.638 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54222c79:::benchmark_data_smithi188_113957_object44414:head# 2024-04-23T16:33:55.638 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54229ad0:::benchmark_data_smithi188_113957_object150447:head# 2024-04-23T16:33:55.639 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.639 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5422b98e:::benchmark_data_smithi188_113957_object30035:head# 2024-04-23T16:33:55.639 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.639 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5422ef02:::benchmark_data_smithi188_113957_object130560:head# 2024-04-23T16:33:55.639 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.639 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54231bd7:::benchmark_data_smithi188_113957_object135759:head# 2024-04-23T16:33:55.639 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.639 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54237a9a:::benchmark_data_smithi188_113957_object151721:head# 2024-04-23T16:33:55.639 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.639 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5423b03b:::benchmark_data_smithi188_113957_object28907:head# 2024-04-23T16:33:55.639 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.639 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54256fbe:::benchmark_data_smithi188_113957_object45599:head# 2024-04-23T16:33:55.639 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.639 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54265bd7:::benchmark_data_smithi188_113957_object25977:head# 2024-04-23T16:33:55.639 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.640 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:542742af:::benchmark_data_smithi188_113957_object98208:head# 2024-04-23T16:33:55.640 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.640 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5427679c:::benchmark_data_smithi188_113957_object79486:head# 2024-04-23T16:33:55.640 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.640 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5427ac03:::benchmark_data_smithi188_113957_object553:head# 2024-04-23T16:33:55.640 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.640 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5428062f:::benchmark_data_smithi188_113957_object76893:head# 2024-04-23T16:33:55.640 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.640 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:542813a9:::benchmark_data_smithi188_113957_object146263:head# 2024-04-23T16:33:55.640 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.640 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54281959:::benchmark_data_smithi188_113957_object141869:head# 2024-04-23T16:33:55.640 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.640 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54286625:::benchmark_data_smithi188_113957_object100619:head# 2024-04-23T16:33:55.640 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.641 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54296b2b:::benchmark_data_smithi188_113957_object2365:head# 2024-04-23T16:33:55.641 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.641 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54298d7d:::benchmark_data_smithi188_113957_object1607:head# 2024-04-23T16:33:55.641 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.641 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:542a03c4:::benchmark_data_smithi188_113957_object111886:head# 2024-04-23T16:33:55.641 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.641 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:542a5db4:::benchmark_data_smithi188_113957_object35567:head# 2024-04-23T16:33:55.641 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.641 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:542a7680:::benchmark_data_smithi188_113957_object104113:head# 2024-04-23T16:33:55.641 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.641 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:542a792b:::benchmark_data_smithi188_113957_object127899:head# 2024-04-23T16:33:55.641 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.641 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:542ad384:::benchmark_data_smithi188_113957_object63852:head# 2024-04-23T16:33:55.641 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.641 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:542add16:::benchmark_data_smithi188_113957_object71040:head# 2024-04-23T16:33:55.642 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.642 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:542c0cd8:::benchmark_data_smithi188_113957_object116655:head# 2024-04-23T16:33:55.642 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.642 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:542d967e:::benchmark_data_smithi188_113957_object2894:head# 2024-04-23T16:33:55.642 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.642 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:542dc8b9:::benchmark_data_smithi188_113957_object85184:head# 2024-04-23T16:33:55.642 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.642 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:542f24dd:::benchmark_data_smithi188_113957_object103128:head# 2024-04-23T16:33:55.642 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.642 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:542f2eb5:::benchmark_data_smithi188_113957_object112208:head# 2024-04-23T16:33:55.642 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.642 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:542f6eb9:::benchmark_data_smithi188_113957_object68953:head# 2024-04-23T16:33:55.642 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:542f75d2:::benchmark_data_smithi188_113957_object98881:head# 2024-04-23T16:33:55.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:542f97a8:::benchmark_data_smithi188_113957_object84962:head# 2024-04-23T16:33:55.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5430674b:::benchmark_data_smithi188_113957_object12680:head# 2024-04-23T16:33:55.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54308a58:::benchmark_data_smithi188_113957_object3650:head# 2024-04-23T16:33:55.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5430d755:::benchmark_data_smithi188_113957_object65712:head# 2024-04-23T16:33:55.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5430f7a0:::benchmark_data_smithi188_113957_object110858:head# 2024-04-23T16:33:55.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54313e22:::benchmark_data_smithi188_113957_object92644:head# 2024-04-23T16:33:55.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.644 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:543201d5:::benchmark_data_smithi188_113957_object16467:head# 2024-04-23T16:33:55.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.644 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54321bfc:::benchmark_data_smithi188_113957_object73912:head# 2024-04-23T16:33:55.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.644 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54321de8:::benchmark_data_smithi188_113957_object13088:head# 2024-04-23T16:33:55.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.644 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54325201:::benchmark_data_smithi188_113957_object91832:head# 2024-04-23T16:33:55.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.644 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5432ac4b:::benchmark_data_smithi188_113957_object94045:head# 2024-04-23T16:33:55.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.644 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5432f537:::benchmark_data_smithi188_113957_object68525:head# 2024-04-23T16:33:55.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.644 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:543315cd:::benchmark_data_smithi188_113957_object142789:head# 2024-04-23T16:33:55.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.645 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5433e76c:::benchmark_data_smithi188_113957_object125461:head# 2024-04-23T16:33:55.645 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.645 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:543528c0:::benchmark_data_smithi188_113957_object66126:head# 2024-04-23T16:33:55.645 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.645 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:543665a3:::benchmark_data_smithi188_113957_object540:head# 2024-04-23T16:33:55.645 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.645 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54379160:::benchmark_data_smithi188_113957_object136652:head# 2024-04-23T16:33:55.645 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.645 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5437c752:::benchmark_data_smithi188_113957_object110471:head# 2024-04-23T16:33:55.645 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.645 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5437e5e3:::benchmark_data_smithi188_113957_object117674:head# 2024-04-23T16:33:55.645 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.645 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54383bcf:::benchmark_data_smithi188_113957_object111740:head# 2024-04-23T16:33:55.645 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.646 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54386304:::benchmark_data_smithi188_113957_object57803:head# 2024-04-23T16:33:55.646 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.646 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54391b51:::benchmark_data_smithi188_113957_object115938:head# 2024-04-23T16:33:55.646 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.646 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54395f46:::benchmark_data_smithi188_113957_object147398:head# 2024-04-23T16:33:55.646 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.646 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54397e9a:::benchmark_data_smithi188_113957_object121112:head# 2024-04-23T16:33:55.646 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.646 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:543a33fd:::benchmark_data_smithi188_113957_object58827:head# 2024-04-23T16:33:55.646 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.646 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:543ad8bb:::benchmark_data_smithi188_113957_object96850:head# 2024-04-23T16:33:55.646 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.646 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:543afb31:::benchmark_data_smithi188_113957_object103400:head# 2024-04-23T16:33:55.646 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:543be64d:::benchmark_data_smithi188_113957_object36099:head# 2024-04-23T16:33:55.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:543beb01:::benchmark_data_smithi188_113957_object76061:head# 2024-04-23T16:33:55.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:543c0c43:::benchmark_data_smithi188_113957_object151195:head# 2024-04-23T16:33:55.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:543c2edc:::benchmark_data_smithi188_113957_object72328:head# 2024-04-23T16:33:55.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:543cd217:::benchmark_data_smithi188_113957_object98102:head# 2024-04-23T16:33:55.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:543ce81f:::benchmark_data_smithi188_113957_object58908:head# 2024-04-23T16:33:55.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:543d76e5:::benchmark_data_smithi188_113957_object152878:head# 2024-04-23T16:33:55.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:543dc5aa:::benchmark_data_smithi188_113957_object98952:head# 2024-04-23T16:33:55.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:543f2b7d:::benchmark_data_smithi188_113957_object1719:head# 2024-04-23T16:33:55.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:543fb63c:::benchmark_data_smithi188_113957_object104481:head# 2024-04-23T16:33:55.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54400f78:::benchmark_data_smithi188_113957_object53128:head# 2024-04-23T16:33:55.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544062e1:::benchmark_data_smithi188_113957_object44282:head# 2024-04-23T16:33:55.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5440f1a2:::benchmark_data_smithi188_113957_object60324:head# 2024-04-23T16:33:55.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5441b75e:::benchmark_data_smithi188_113957_object139851:head# 2024-04-23T16:33:55.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5441b8b3:::benchmark_data_smithi188_113957_object41385:head# 2024-04-23T16:33:55.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5441c40f:::benchmark_data_smithi188_113957_object137590:head# 2024-04-23T16:33:55.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544253db:::benchmark_data_smithi188_113957_object30301:head# 2024-04-23T16:33:55.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5442ec12:::benchmark_data_smithi188_113957_object2184:head# 2024-04-23T16:33:55.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54433fd1:::benchmark_data_smithi188_113957_object124240:head# 2024-04-23T16:33:55.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5443885f:::benchmark_data_smithi188_113957_object107914:head# 2024-04-23T16:33:55.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5443a60e:::benchmark_data_smithi188_113957_object141879:head# 2024-04-23T16:33:55.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5443c5ca:::benchmark_data_smithi188_113957_object100381:head# 2024-04-23T16:33:55.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54442fdc:::benchmark_data_smithi188_113957_object79353:head# 2024-04-23T16:33:55.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54466d75:::benchmark_data_smithi188_113957_object103406:head# 2024-04-23T16:33:55.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544733dd:::benchmark_data_smithi188_113957_object125478:head# 2024-04-23T16:33:55.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5447e4c6:::benchmark_data_smithi188_113957_object32495:head# 2024-04-23T16:33:55.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5448734f:::benchmark_data_smithi188_113957_object10842:head# 2024-04-23T16:33:55.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544973ef:::benchmark_data_smithi188_113957_object85119:head# 2024-04-23T16:33:55.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544a3cba:::benchmark_data_smithi188_113957_object139109:head# 2024-04-23T16:33:55.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544ab906:::benchmark_data_smithi188_113957_object125647:head# 2024-04-23T16:33:55.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544b22e5:::benchmark_data_smithi188_113957_object130665:head# 2024-04-23T16:33:55.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544b466c:::benchmark_data_smithi188_113957_object89865:head# 2024-04-23T16:33:55.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544b56ea:::benchmark_data_smithi188_113957_object65036:head# 2024-04-23T16:33:55.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544c0ebe:::benchmark_data_smithi188_113957_object110522:head# 2024-04-23T16:33:55.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544c4be2:::benchmark_data_smithi188_113957_object145993:head# 2024-04-23T16:33:55.652 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.652 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544c5a0a:::benchmark_data_smithi188_113957_object105636:head# 2024-04-23T16:33:55.652 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.652 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544d2d4e:::benchmark_data_smithi188_113957_object108073:head# 2024-04-23T16:33:55.652 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.652 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544d9fd6:::benchmark_data_smithi188_113957_object102727:head# 2024-04-23T16:33:55.652 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.652 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544dcb12:::benchmark_data_smithi188_113957_object74327:head# 2024-04-23T16:33:55.652 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.652 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544dcecb:::benchmark_data_smithi188_113957_object148561:head# 2024-04-23T16:33:55.652 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.652 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544e3714:::benchmark_data_smithi188_113957_object140405:head# 2024-04-23T16:33:55.652 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.653 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544ea324:::benchmark_data_smithi188_113957_object17627:head# 2024-04-23T16:33:55.653 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.653 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544ecfba:::benchmark_data_smithi188_113957_object37962:head# 2024-04-23T16:33:55.653 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.653 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544ee66d:::benchmark_data_smithi188_113957_object87309:head# 2024-04-23T16:33:55.653 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.653 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:544ef4b3:::benchmark_data_smithi188_113957_object31433:head# 2024-04-23T16:33:55.653 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.653 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5451b9c7:::benchmark_data_smithi188_113957_object85126:head# 2024-04-23T16:33:55.653 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.653 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54527079:::benchmark_data_smithi188_113957_object59000:head# 2024-04-23T16:33:55.653 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.653 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5452e1d6:::benchmark_data_smithi188_113957_object104025:head# 2024-04-23T16:33:55.653 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.654 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5452e8a2:::benchmark_data_smithi188_113957_object102693:head# 2024-04-23T16:33:55.654 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.654 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5452f3d0:::benchmark_data_smithi188_113957_object149358:head# 2024-04-23T16:33:55.654 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.654 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5453c998:::benchmark_data_smithi188_113957_object112035:head# 2024-04-23T16:33:55.654 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.654 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5454b08e:::benchmark_data_smithi188_113957_object83938:head# 2024-04-23T16:33:55.654 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.654 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5454ef6c:::benchmark_data_smithi188_113957_object110625:head# 2024-04-23T16:33:55.654 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.654 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5454f0f3:::benchmark_data_smithi188_113957_object145460:head# 2024-04-23T16:33:55.654 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.654 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5455cc40:::benchmark_data_smithi188_113957_object58363:head# 2024-04-23T16:33:55.654 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5455f504:::benchmark_data_smithi188_113957_object130231:head# 2024-04-23T16:33:55.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:545607ad:::benchmark_data_smithi188_113957_object18438:head# 2024-04-23T16:33:55.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5457de8f:::benchmark_data_smithi188_113957_object83593:head# 2024-04-23T16:33:55.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5457e106:::benchmark_data_smithi188_113957_object33318:head# 2024-04-23T16:33:55.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5457fac8:::benchmark_data_smithi188_113957_object82155:head# 2024-04-23T16:33:55.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54585bfe:::benchmark_data_smithi188_113957_object80221:head# 2024-04-23T16:33:55.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54585cea:::benchmark_data_smithi188_113957_object109609:head# 2024-04-23T16:33:55.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54586246:::benchmark_data_smithi188_113957_object97784:head# 2024-04-23T16:33:55.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:545883b1:::benchmark_data_smithi188_113957_object62594:head# 2024-04-23T16:33:55.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54592c83:::benchmark_data_smithi188_113957_object29068:head# 2024-04-23T16:33:55.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5459e189:::benchmark_data_smithi188_113957_object54984:head# 2024-04-23T16:33:55.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:545a3bf8:::benchmark_data_smithi188_113957_object128003:head# 2024-04-23T16:33:55.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:545af2de:::benchmark_data_smithi188_113957_object103540:head# 2024-04-23T16:33:55.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:545bae0c:::benchmark_data_smithi188_113957_object137985:head# 2024-04-23T16:33:55.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:545bf3bf:::benchmark_data_smithi188_113957_object24397:head# 2024-04-23T16:33:55.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:545c20bb:::benchmark_data_smithi188_113957_object658:head# 2024-04-23T16:33:55.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:545c5e3a:::benchmark_data_smithi188_113957_object89087:head# 2024-04-23T16:33:55.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:545cbc62:::benchmark_data_smithi188_113957_object114223:head# 2024-04-23T16:33:55.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:545d32f6:::benchmark_data_smithi188_113957_object89027:head# 2024-04-23T16:33:55.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:545d85d9:::benchmark_data_smithi188_113957_object80967:head# 2024-04-23T16:33:55.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:545d91db:::benchmark_data_smithi188_113957_object70672:head# 2024-04-23T16:33:55.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.658 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:545ee934:::benchmark_data_smithi188_113957_object58743:head# 2024-04-23T16:33:55.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.658 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:545f4fa3:::benchmark_data_smithi188_113957_object67488:head# 2024-04-23T16:33:55.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.658 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:545f9dc6:::benchmark_data_smithi188_113957_object27616:head# 2024-04-23T16:33:55.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.658 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:545fb7db:::benchmark_data_smithi188_113957_object67316:head# 2024-04-23T16:33:55.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.658 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5460256e:::benchmark_data_smithi188_113957_object60778:head# 2024-04-23T16:33:55.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.658 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5461a3de:::benchmark_data_smithi188_113957_object126015:head# 2024-04-23T16:33:55.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.658 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5461aa3e:::benchmark_data_smithi188_113957_object139033:head# 2024-04-23T16:33:55.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.659 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5461b6f3:::benchmark_data_smithi188_113957_object41237:head# 2024-04-23T16:33:55.659 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.659 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546210eb:::benchmark_data_smithi188_113957_object21484:head# 2024-04-23T16:33:55.659 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.659 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546245c8:::benchmark_data_smithi188_113957_object30303:head# 2024-04-23T16:33:55.659 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.659 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54626065:::benchmark_data_smithi188_113957_object131235:head# 2024-04-23T16:33:55.659 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.659 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546394c3:::benchmark_data_smithi188_113957_object148843:head# 2024-04-23T16:33:55.659 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.659 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54639c17:::benchmark_data_smithi188_113957_object33462:head# 2024-04-23T16:33:55.659 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.659 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5463ce25:::benchmark_data_smithi188_113957_object110935:head# 2024-04-23T16:33:55.659 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.660 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54641fa7:::benchmark_data_smithi188_113957_object89036:head# 2024-04-23T16:33:55.660 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.660 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546436d4:::benchmark_data_smithi188_113957_object111804:head# 2024-04-23T16:33:55.660 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.660 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546473ef:::benchmark_data_smithi188_113957_object61915:head# 2024-04-23T16:33:55.660 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.660 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54649216:::benchmark_data_smithi188_113957_object107817:head# 2024-04-23T16:33:55.660 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.660 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5464c6bf:::benchmark_data_smithi188_113957_object48046:head# 2024-04-23T16:33:55.660 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.660 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5464dcb0:::benchmark_data_smithi188_113957_object127952:head# 2024-04-23T16:33:55.660 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.660 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5464f279:::benchmark_data_smithi188_113957_object128150:head# 2024-04-23T16:33:55.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5464fb8c:::benchmark_data_smithi188_113957_object118043:head# 2024-04-23T16:33:55.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54653839:::benchmark_data_smithi188_113957_object144713:head# 2024-04-23T16:33:55.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546562a9:::benchmark_data_smithi188_113957_object50665:head# 2024-04-23T16:33:55.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5465dd6c:::benchmark_data_smithi188_113957_object149485:head# 2024-04-23T16:33:55.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546668a7:::benchmark_data_smithi188_113957_object31877:head# 2024-04-23T16:33:55.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5467047f:::benchmark_data_smithi188_113957_object89280:head# 2024-04-23T16:33:55.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546793b9:::benchmark_data_smithi188_113957_object66764:head# 2024-04-23T16:33:55.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54682cb1:::benchmark_data_smithi188_113957_object123240:head# 2024-04-23T16:33:55.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5468615a:::benchmark_data_smithi188_113957_object62143:head# 2024-04-23T16:33:55.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54690ed3:::benchmark_data_smithi188_113957_object151885:head# 2024-04-23T16:33:55.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5469320e:::benchmark_data_smithi188_113957_object10553:head# 2024-04-23T16:33:55.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54693639:::benchmark_data_smithi188_113957_object117808:head# 2024-04-23T16:33:55.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5469cf72:::benchmark_data_smithi188_113957_object90186:head# 2024-04-23T16:33:55.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546a3246:::benchmark_data_smithi188_113957_object80991:head# 2024-04-23T16:33:55.663 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546b0cfc:::benchmark_data_smithi188_113957_object82017:head# 2024-04-23T16:33:55.663 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546b3944:::benchmark_data_smithi188_113957_object41208:head# 2024-04-23T16:33:55.663 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546ba5f9:::benchmark_data_smithi188_113957_object75105:head# 2024-04-23T16:33:55.663 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546c6b5a:::benchmark_data_smithi188_113957_object15477:head# 2024-04-23T16:33:55.663 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546c6fc7:::benchmark_data_smithi188_113957_object138582:head# 2024-04-23T16:33:55.663 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546d1781:::benchmark_data_smithi188_113957_object110678:head# 2024-04-23T16:33:55.663 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546d19ed:::benchmark_data_smithi188_113957_object152099:head# 2024-04-23T16:33:55.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546d5808:::benchmark_data_smithi188_113957_object119876:head# 2024-04-23T16:33:55.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546d869e:::benchmark_data_smithi188_113957_object13062:head# 2024-04-23T16:33:55.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546d9381:::benchmark_data_smithi188_113957_object118159:head# 2024-04-23T16:33:55.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546dc772:::benchmark_data_smithi188_113957_object88199:head# 2024-04-23T16:33:55.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546e8c7b:::benchmark_data_smithi188_113957_object119697:head# 2024-04-23T16:33:55.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546f3c65:::benchmark_data_smithi188_113957_object83133:head# 2024-04-23T16:33:55.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546f5edc:::benchmark_data_smithi188_113957_object4482:head# 2024-04-23T16:33:55.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:546fd932:::benchmark_data_smithi188_113957_object85839:head# 2024-04-23T16:33:55.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54705e6d:::benchmark_data_smithi188_113957_object127817:head# 2024-04-23T16:33:55.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5470a109:::benchmark_data_smithi188_113957_object32648:head# 2024-04-23T16:33:55.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5470d450:::benchmark_data_smithi188_113957_object16744:head# 2024-04-23T16:33:55.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54717861:::benchmark_data_smithi188_113957_object123125:head# 2024-04-23T16:33:55.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54717f35:::benchmark_data_smithi188_113957_object116070:head# 2024-04-23T16:33:55.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54718bf8:::benchmark_data_smithi188_113957_object54441:head# 2024-04-23T16:33:55.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5471dc00:::benchmark_data_smithi188_113957_object98337:head# 2024-04-23T16:33:55.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5471f6c1:::benchmark_data_smithi188_113957_object41364:head# 2024-04-23T16:33:55.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54723605:::benchmark_data_smithi188_113957_object98814:head# 2024-04-23T16:33:55.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54728274:::benchmark_data_smithi188_113957_object131490:head# 2024-04-23T16:33:55.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5472fce2:::benchmark_data_smithi188_113957_object39147:head# 2024-04-23T16:33:55.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54744331:::benchmark_data_smithi188_113957_object119648:head# 2024-04-23T16:33:55.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54744a14:::benchmark_data_smithi188_113957_object115887:head# 2024-04-23T16:33:55.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54755dfc:::benchmark_data_smithi188_113957_object72240:head# 2024-04-23T16:33:55.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:547615da:::benchmark_data_smithi188_113957_object115976:head# 2024-04-23T16:33:55.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54766ec8:::benchmark_data_smithi188_113957_object35162:head# 2024-04-23T16:33:55.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54770220:::benchmark_data_smithi188_113957_object6980:head# 2024-04-23T16:33:55.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5477485f:::benchmark_data_smithi188_113957_object44198:head# 2024-04-23T16:33:55.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.668 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5477b88b:::benchmark_data_smithi188_113957_object76770:head# 2024-04-23T16:33:55.668 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.668 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5477fc0e:::benchmark_data_smithi188_113957_object143544:head# 2024-04-23T16:33:55.668 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.668 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5478504f:::benchmark_data_smithi188_113957_object52557:head# 2024-04-23T16:33:55.668 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.668 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5478d2f8:::benchmark_data_smithi188_113957_object91630:head# 2024-04-23T16:33:55.668 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.668 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5479b879:::benchmark_data_smithi188_113957_object32570:head# 2024-04-23T16:33:55.668 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.668 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5479d428:::benchmark_data_smithi188_113957_object1442:head# 2024-04-23T16:33:55.668 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.668 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:547a465a:::benchmark_data_smithi188_113957_object46730:head# 2024-04-23T16:33:55.668 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:547a7c69:::benchmark_data_smithi188_113957_object60811:head# 2024-04-23T16:33:55.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:547b1acd:::benchmark_data_smithi188_113957_object883:head# 2024-04-23T16:33:55.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:547b632a:::benchmark_data_smithi188_113957_object136496:head# 2024-04-23T16:33:55.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:547b740c:::benchmark_data_smithi188_113957_object62792:head# 2024-04-23T16:33:55.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:547c2fad:::benchmark_data_smithi188_113957_object144849:head# 2024-04-23T16:33:55.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:547d8a23:::benchmark_data_smithi188_113957_object115286:head# 2024-04-23T16:33:55.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:547e2ea0:::benchmark_data_smithi188_113957_object90139:head# 2024-04-23T16:33:55.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:547e6131:::benchmark_data_smithi188_113957_object71873:head# 2024-04-23T16:33:55.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:547ed9a9:::benchmark_data_smithi188_113957_object60251:head# 2024-04-23T16:33:55.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:547f2d3d:::benchmark_data_smithi188_113957_object130743:head# 2024-04-23T16:33:55.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:547f7eed:::benchmark_data_smithi188_113957_object134313:head# 2024-04-23T16:33:55.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:547f8ce1:::benchmark_data_smithi188_113957_object22482:head# 2024-04-23T16:33:55.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54801778:::benchmark_data_smithi188_113957_object150026:head# 2024-04-23T16:33:55.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548031b4:::benchmark_data_smithi188_113957_object32323:head# 2024-04-23T16:33:55.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54813877:::benchmark_data_smithi188_113957_object96470:head# 2024-04-23T16:33:55.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5481463f:::benchmark_data_smithi188_113957_object28792:head# 2024-04-23T16:33:55.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54816c93:::benchmark_data_smithi188_113957_object84838:head# 2024-04-23T16:33:55.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548201b7:::benchmark_data_smithi188_113957_object133414:head# 2024-04-23T16:33:55.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54828a6d:::benchmark_data_smithi188_113957_object62673:head# 2024-04-23T16:33:55.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5482d580:::benchmark_data_smithi188_113957_object8550:head# 2024-04-23T16:33:55.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548374e6:::benchmark_data_smithi188_113957_object101996:head# 2024-04-23T16:33:55.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548406cf:::benchmark_data_smithi188_113957_object36217:head# 2024-04-23T16:33:55.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548459b8:::benchmark_data_smithi188_113957_object109239:head# 2024-04-23T16:33:55.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54850faf:::benchmark_data_smithi188_113957_object20500:head# 2024-04-23T16:33:55.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5485669f:::benchmark_data_smithi188_113957_object144320:head# 2024-04-23T16:33:55.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5485acff:::benchmark_data_smithi188_113957_object116134:head# 2024-04-23T16:33:55.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548696c7:::benchmark_data_smithi188_113957_object2400:head# 2024-04-23T16:33:55.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54873878:::benchmark_data_smithi188_113957_object109880:head# 2024-04-23T16:33:55.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54874b74:::benchmark_data_smithi188_113957_object83772:head# 2024-04-23T16:33:55.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5487b9a3:::benchmark_data_smithi188_113957_object46636:head# 2024-04-23T16:33:55.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5487cf47:::benchmark_data_smithi188_113957_object72954:head# 2024-04-23T16:33:55.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5488668a:::benchmark_data_smithi188_113957_object143291:head# 2024-04-23T16:33:55.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5488b36e:::benchmark_data_smithi188_113957_object88364:head# 2024-04-23T16:33:55.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.674 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54894638:::benchmark_data_smithi188_113957_object92671:head# 2024-04-23T16:33:55.674 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.674 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5489c330:::benchmark_data_smithi188_113957_object35688:head# 2024-04-23T16:33:55.674 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.674 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5489fa46:::benchmark_data_smithi188_113957_object5622:head# 2024-04-23T16:33:55.674 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.674 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548a73c1:::benchmark_data_smithi188_113957_object48140:head# 2024-04-23T16:33:55.674 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.674 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548a811f:::benchmark_data_smithi188_113957_object61631:head# 2024-04-23T16:33:55.674 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.674 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548ad1bf:::benchmark_data_smithi188_113957_object127319:head# 2024-04-23T16:33:55.674 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548adfac:::benchmark_data_smithi188_113957_object146683:head# 2024-04-23T16:33:55.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548ae720:::benchmark_data_smithi188_113957_object26475:head# 2024-04-23T16:33:55.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548b2c41:::benchmark_data_smithi188_113957_object150222:head# 2024-04-23T16:33:55.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548daf32:::benchmark_data_smithi188_113957_object95927:head# 2024-04-23T16:33:55.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548e1c13:::benchmark_data_smithi188_113957_object91288:head# 2024-04-23T16:33:55.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548e28e4:::benchmark_data_smithi188_113957_object15051:head# 2024-04-23T16:33:55.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548e360d:::benchmark_data_smithi188_113957_object105964:head# 2024-04-23T16:33:55.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548e580e:::benchmark_data_smithi188_113957_object90242:head# 2024-04-23T16:33:55.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548ec162:::benchmark_data_smithi188_113957_object103593:head# 2024-04-23T16:33:55.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548f23bf:::benchmark_data_smithi188_113957_object126211:head# 2024-04-23T16:33:55.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548f5b22:::benchmark_data_smithi188_113957_object59034:head# 2024-04-23T16:33:55.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548f81eb:::benchmark_data_smithi188_113957_object28173:head# 2024-04-23T16:33:55.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:548fbe07:::benchmark_data_smithi188_113957_object146644:head# 2024-04-23T16:33:55.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54902cdd:::benchmark_data_smithi188_113957_object51397:head# 2024-04-23T16:33:55.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.677 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54907c45:::benchmark_data_smithi188_113957_object41626:head# 2024-04-23T16:33:55.677 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.677 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54910e72:::benchmark_data_smithi188_113957_object4178:head# 2024-04-23T16:33:55.677 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.677 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54912ec0:::benchmark_data_smithi188_113957_object61446:head# 2024-04-23T16:33:55.677 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.677 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5491e00b:::benchmark_data_smithi188_113957_object72634:head# 2024-04-23T16:33:55.677 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.677 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549235e0:::benchmark_data_smithi188_113957_object76459:head# 2024-04-23T16:33:55.677 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.677 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5492821b:::benchmark_data_smithi188_113957_object102745:head# 2024-04-23T16:33:55.677 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.677 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549387ba:::benchmark_data_smithi188_113957_object16530:head# 2024-04-23T16:33:55.677 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5493a665:::benchmark_data_smithi188_113957_object88924:head# 2024-04-23T16:33:55.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5494308d:::benchmark_data_smithi188_113957_object7020:head# 2024-04-23T16:33:55.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549439c0:::benchmark_data_smithi188_113957_object56573:head# 2024-04-23T16:33:55.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549454cb:::benchmark_data_smithi188_113957_object20294:head# 2024-04-23T16:33:55.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54947f3f:::benchmark_data_smithi188_113957_object45443:head# 2024-04-23T16:33:55.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54951bed:::benchmark_data_smithi188_113957_object79010:head# 2024-04-23T16:33:55.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5495856f:::benchmark_data_smithi188_113957_object91354:head# 2024-04-23T16:33:55.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54959a8a:::benchmark_data_smithi188_113957_object129273:head# 2024-04-23T16:33:55.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5495b8a5:::benchmark_data_smithi188_113957_object30460:head# 2024-04-23T16:33:55.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54964d82:::benchmark_data_smithi188_113957_object74629:head# 2024-04-23T16:33:55.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549651f8:::benchmark_data_smithi188_113957_object110434:head# 2024-04-23T16:33:55.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54966840:::benchmark_data_smithi188_113957_object5855:head# 2024-04-23T16:33:55.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549692d8:::benchmark_data_smithi188_113957_object72100:head# 2024-04-23T16:33:55.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5496a700:::benchmark_data_smithi188_113957_object151390:head# 2024-04-23T16:33:55.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5496d1a6:::benchmark_data_smithi188_113957_object108676:head# 2024-04-23T16:33:55.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5496e6f3:::benchmark_data_smithi188_113957_object13665:head# 2024-04-23T16:33:55.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54973b1d:::benchmark_data_smithi188_113957_object60732:head# 2024-04-23T16:33:55.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5497c468:::benchmark_data_smithi188_113957_object12755:head# 2024-04-23T16:33:55.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5497e83b:::benchmark_data_smithi188_113957_object91311:head# 2024-04-23T16:33:55.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549881ef:::benchmark_data_smithi188_113957_object24846:head# 2024-04-23T16:33:55.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54990535:::benchmark_data_smithi188_113957_object102487:head# 2024-04-23T16:33:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5499093f:::benchmark_data_smithi188_113957_object82660:head# 2024-04-23T16:33:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54990b56:::benchmark_data_smithi188_113957_object139134:head# 2024-04-23T16:33:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549954b4:::benchmark_data_smithi188_113957_object9001:head# 2024-04-23T16:33:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549ae5e3:::benchmark_data_smithi188_113957_object29442:head# 2024-04-23T16:33:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549b0e91:::benchmark_data_smithi188_113957_object23170:head# 2024-04-23T16:33:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549b731b:::benchmark_data_smithi188_113957_object51768:head# 2024-04-23T16:33:55.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549b9993:::benchmark_data_smithi188_113957_object26844:head# 2024-04-23T16:33:55.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549c95fc:::benchmark_data_smithi188_113957_object138107:head# 2024-04-23T16:33:55.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549ca392:::benchmark_data_smithi188_113957_object146374:head# 2024-04-23T16:33:55.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549cac1a:::benchmark_data_smithi188_113957_object93653:head# 2024-04-23T16:33:55.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549ce5dc:::benchmark_data_smithi188_113957_object122011:head# 2024-04-23T16:33:55.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549d1e3f:::benchmark_data_smithi188_113957_object35643:head# 2024-04-23T16:33:55.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549dd2d3:::benchmark_data_smithi188_113957_object17223:head# 2024-04-23T16:33:55.683 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.683 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549e3932:::benchmark_data_smithi188_113957_object55188:head# 2024-04-23T16:33:55.683 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.683 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549e7ad2:::benchmark_data_smithi188_113957_object127329:head# 2024-04-23T16:33:55.683 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.683 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549e845f:::benchmark_data_smithi188_113957_object88356:head# 2024-04-23T16:33:55.683 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.683 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549ea553:::benchmark_data_smithi188_113957_object96556:head# 2024-04-23T16:33:55.683 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.683 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549ee49d:::benchmark_data_smithi188_113957_object115820:head# 2024-04-23T16:33:55.683 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.683 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549f23a3:::benchmark_data_smithi188_113957_object46582:head# 2024-04-23T16:33:55.683 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.683 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549f6084:::benchmark_data_smithi188_113957_object91089:head# 2024-04-23T16:33:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:549fa02a:::benchmark_data_smithi188_113957_object12485:head# 2024-04-23T16:33:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a012cd:::benchmark_data_smithi188_113957_object21665:head# 2024-04-23T16:33:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a0d8cf:::benchmark_data_smithi188_113957_object146710:head# 2024-04-23T16:33:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a1281d:::benchmark_data_smithi188_113957_object45990:head# 2024-04-23T16:33:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a1f1a8:::benchmark_data_smithi188_113957_object99821:head# 2024-04-23T16:33:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a27573:::benchmark_data_smithi188_113957_object39460:head# 2024-04-23T16:33:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a33492:::benchmark_data_smithi188_113957_object64077:head# 2024-04-23T16:33:55.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a3b39e:::benchmark_data_smithi188_113957_object18566:head# 2024-04-23T16:33:55.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a3b623:::benchmark_data_smithi188_113957_object868:head# 2024-04-23T16:33:55.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a4acd4:::benchmark_data_smithi188_113957_object55086:head# 2024-04-23T16:33:55.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a4b2e7:::benchmark_data_smithi188_113957_object137439:head# 2024-04-23T16:33:55.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a52d68:::benchmark_data_smithi188_113957_object131637:head# 2024-04-23T16:33:55.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a54334:::benchmark_data_smithi188_113957_object63743:head# 2024-04-23T16:33:55.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a57319:::benchmark_data_smithi188_113957_object152704:head# 2024-04-23T16:33:55.686 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.686 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a5b41c:::benchmark_data_smithi188_113957_object859:head# 2024-04-23T16:33:55.686 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.686 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a61508:::benchmark_data_smithi188_113957_object63661:head# 2024-04-23T16:33:55.686 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.686 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a6354f:::benchmark_data_smithi188_113957_object129569:head# 2024-04-23T16:33:55.686 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.686 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a647f3:::benchmark_data_smithi188_113957_object87409:head# 2024-04-23T16:33:55.686 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.686 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a76290:::benchmark_data_smithi188_113957_object97674:head# 2024-04-23T16:33:55.686 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.686 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a76300:::benchmark_data_smithi188_113957_object95986:head# 2024-04-23T16:33:55.686 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a778d0:::benchmark_data_smithi188_113957_object144927:head# 2024-04-23T16:33:55.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a79cb9:::benchmark_data_smithi188_113957_object25756:head# 2024-04-23T16:33:55.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a7a9dc:::benchmark_data_smithi188_113957_object152024:head# 2024-04-23T16:33:55.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a7db61:::benchmark_data_smithi188_113957_object44316:head# 2024-04-23T16:33:55.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a827a9:::benchmark_data_smithi188_113957_object57368:head# 2024-04-23T16:33:55.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a98204:::benchmark_data_smithi188_113957_object136083:head# 2024-04-23T16:33:55.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54a98d8c:::benchmark_data_smithi188_113957_object105786:head# 2024-04-23T16:33:55.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54aa4085:::benchmark_data_smithi188_113957_object30270:head# 2024-04-23T16:33:55.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54aa6338:::benchmark_data_smithi188_113957_object117569:head# 2024-04-23T16:33:55.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ab09d3:::benchmark_data_smithi188_113957_object139573:head# 2024-04-23T16:33:55.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ab74c8:::benchmark_data_smithi188_113957_object130532:head# 2024-04-23T16:33:55.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ac3ba3:::benchmark_data_smithi188_113957_object46682:head# 2024-04-23T16:33:55.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ad5972:::benchmark_data_smithi188_113957_object113934:head# 2024-04-23T16:33:55.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54adc5ac:::benchmark_data_smithi188_113957_object88578:head# 2024-04-23T16:33:55.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54add534:::benchmark_data_smithi188_113957_object133818:head# 2024-04-23T16:33:55.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ae5dc2:::benchmark_data_smithi188_113957_object26470:head# 2024-04-23T16:33:55.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54af4b14:::benchmark_data_smithi188_113957_object69576:head# 2024-04-23T16:33:55.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54af65fc:::benchmark_data_smithi188_113957_object46122:head# 2024-04-23T16:33:55.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54b2756e:::benchmark_data_smithi188_113957_object107666:head# 2024-04-23T16:33:55.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54b2dcb5:::benchmark_data_smithi188_113957_object77740:head# 2024-04-23T16:33:55.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54b30575:::benchmark_data_smithi188_113957_object8259:head# 2024-04-23T16:33:55.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54b33b25:::benchmark_data_smithi188_113957_object8830:head# 2024-04-23T16:33:55.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54b53a2a:::benchmark_data_smithi188_113957_object53645:head# 2024-04-23T16:33:55.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54b63bac:::benchmark_data_smithi188_113957_object56154:head# 2024-04-23T16:33:55.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54b73c90:::benchmark_data_smithi188_113957_object113652:head# 2024-04-23T16:33:55.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54b74417:::benchmark_data_smithi188_113957_object124726:head# 2024-04-23T16:33:55.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54b75049:::benchmark_data_smithi188_113957_object93794:head# 2024-04-23T16:33:55.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54b75c5c:::benchmark_data_smithi188_113957_object150471:head# 2024-04-23T16:33:55.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54b77090:::benchmark_data_smithi188_113957_object87439:head# 2024-04-23T16:33:55.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54b816af:::benchmark_data_smithi188_113957_object105346:head# 2024-04-23T16:33:55.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54b8537a:::benchmark_data_smithi188_113957_object131588:head# 2024-04-23T16:33:55.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54b8f2f1:::benchmark_data_smithi188_113957_object135279:head# 2024-04-23T16:33:55.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54bab253:::benchmark_data_smithi188_113957_object145591:head# 2024-04-23T16:33:55.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54bc9f52:::benchmark_data_smithi188_113957_object40433:head# 2024-04-23T16:33:55.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54bceada:::benchmark_data_smithi188_113957_object132503:head# 2024-04-23T16:33:55.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54bcf204:::benchmark_data_smithi188_113957_object117268:head# 2024-04-23T16:33:55.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54bd3329:::benchmark_data_smithi188_113957_object22090:head# 2024-04-23T16:33:55.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54bd410c:::benchmark_data_smithi188_113957_object141294:head# 2024-04-23T16:33:55.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54be98a1:::benchmark_data_smithi188_113957_object125219:head# 2024-04-23T16:33:55.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54bef75e:::benchmark_data_smithi188_113957_object99392:head# 2024-04-23T16:33:55.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54befb9f:::benchmark_data_smithi188_113957_object144395:head# 2024-04-23T16:33:55.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54bf83fb:::benchmark_data_smithi188_113957_object70912:head# 2024-04-23T16:33:55.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54bf9436:::benchmark_data_smithi188_113957_object97701:head# 2024-04-23T16:33:55.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54bfa0cf:::benchmark_data_smithi188_113957_object137763:head# 2024-04-23T16:33:55.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c0fadc:::benchmark_data_smithi188_113957_object79426:head# 2024-04-23T16:33:55.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c13422:::benchmark_data_smithi188_113957_object124560:head# 2024-04-23T16:33:55.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c20b1e:::benchmark_data_smithi188_113957_object101837:head# 2024-04-23T16:33:55.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c21d4f:::benchmark_data_smithi188_113957_object110423:head# 2024-04-23T16:33:55.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c240be:::benchmark_data_smithi188_113957_object45371:head# 2024-04-23T16:33:55.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c2b641:::benchmark_data_smithi188_113957_object39757:head# 2024-04-23T16:33:55.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.694 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c345d6:::benchmark_data_smithi188_113957_object139672:head# 2024-04-23T16:33:55.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.694 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c3adf2:::benchmark_data_smithi188_113957_object71260:head# 2024-04-23T16:33:55.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.694 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c3e8bd:::benchmark_data_smithi188_113957_object55381:head# 2024-04-23T16:33:55.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.694 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c462fa:::benchmark_data_smithi188_113957_object69585:head# 2024-04-23T16:33:55.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.694 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c5c505:::benchmark_data_smithi188_113957_object107064:head# 2024-04-23T16:33:55.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.694 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c68200:::benchmark_data_smithi188_113957_object91670:head# 2024-04-23T16:33:55.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c6f214:::benchmark_data_smithi188_113957_object98176:head# 2024-04-23T16:33:55.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c7394d:::benchmark_data_smithi188_113957_object101329:head# 2024-04-23T16:33:55.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c7c213:::benchmark_data_smithi188_113957_object45458:head# 2024-04-23T16:33:55.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c90062:::benchmark_data_smithi188_113957_object112489:head# 2024-04-23T16:33:55.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c91e05:::benchmark_data_smithi188_113957_object104210:head# 2024-04-23T16:33:55.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c9389b:::benchmark_data_smithi188_113957_object29379:head# 2024-04-23T16:33:55.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c99488:::benchmark_data_smithi188_113957_object26617:head# 2024-04-23T16:33:55.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.696 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c9ba47:::benchmark_data_smithi188_113957_object47410:head# 2024-04-23T16:33:55.696 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.696 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54c9dbb0:::benchmark_data_smithi188_113957_object152090:head# 2024-04-23T16:33:55.696 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.696 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ca0552:::benchmark_data_smithi188_113957_object65776:head# 2024-04-23T16:33:55.696 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.696 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54cab723:::benchmark_data_smithi188_113957_object107006:head# 2024-04-23T16:33:55.696 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.696 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54cb001e:::benchmark_data_smithi188_113957_object13704:head# 2024-04-23T16:33:55.696 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.696 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54cb6349:::benchmark_data_smithi188_113957_object144702:head# 2024-04-23T16:33:55.696 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.696 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54cb7abe:::benchmark_data_smithi188_113957_object4372:head# 2024-04-23T16:33:55.696 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.696 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54cbd4e9:::benchmark_data_smithi188_113957_object18677:head# 2024-04-23T16:33:55.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54cc1d92:::benchmark_data_smithi188_113957_object108307:head# 2024-04-23T16:33:55.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54cc360c:::benchmark_data_smithi188_113957_object5329:head# 2024-04-23T16:33:55.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54cc72ac:::benchmark_data_smithi188_113957_object150101:head# 2024-04-23T16:33:55.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ccc1b3:::benchmark_data_smithi188_113957_object131405:head# 2024-04-23T16:33:55.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ccdf62:::benchmark_data_smithi188_113957_object86729:head# 2024-04-23T16:33:55.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54cce867:::benchmark_data_smithi188_113957_object68763:head# 2024-04-23T16:33:55.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54cced0d:::benchmark_data_smithi188_113957_object19881:head# 2024-04-23T16:33:55.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ccf989:::benchmark_data_smithi188_113957_object122261:head# 2024-04-23T16:33:55.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54cd2723:::benchmark_data_smithi188_113957_object136927:head# 2024-04-23T16:33:55.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54cdf390:::benchmark_data_smithi188_113957_object74343:head# 2024-04-23T16:33:55.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ce7c59:::benchmark_data_smithi188_113957_object112875:head# 2024-04-23T16:33:55.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ce8667:::benchmark_data_smithi188_113957_object108087:head# 2024-04-23T16:33:55.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ce92f9:::benchmark_data_smithi188_113957_object38849:head# 2024-04-23T16:33:55.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54cec5a7:::benchmark_data_smithi188_113957_object26544:head# 2024-04-23T16:33:55.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.699 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54cfa597:::benchmark_data_smithi188_113957_object122004:head# 2024-04-23T16:33:55.699 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.699 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d068bb:::benchmark_data_smithi188_113957_object56441:head# 2024-04-23T16:33:55.699 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.699 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d0c720:::benchmark_data_smithi188_113957_object21791:head# 2024-04-23T16:33:55.699 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.699 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d0f63e:::benchmark_data_smithi188_113957_object152619:head# 2024-04-23T16:33:55.699 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.699 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d1212f:::benchmark_data_smithi188_113957_object65251:head# 2024-04-23T16:33:55.699 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.699 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d1b828:::benchmark_data_smithi188_113957_object87127:head# 2024-04-23T16:33:55.699 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.699 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d2635b:::benchmark_data_smithi188_113957_object12904:head# 2024-04-23T16:33:55.699 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d2cd5d:::benchmark_data_smithi188_113957_object91367:head# 2024-04-23T16:33:55.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d2e636:::benchmark_data_smithi188_113957_object98895:head# 2024-04-23T16:33:55.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d35ed1:::benchmark_data_smithi188_113957_object131054:head# 2024-04-23T16:33:55.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d36b67:::benchmark_data_smithi188_113957_object9170:head# 2024-04-23T16:33:55.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d3e59d:::benchmark_data_smithi188_113957_object4016:head# 2024-04-23T16:33:55.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d40a02:::benchmark_data_smithi188_113957_object15211:head# 2024-04-23T16:33:55.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d50948:::benchmark_data_smithi188_113957_object3028:head# 2024-04-23T16:33:55.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d51d6c:::benchmark_data_smithi188_113957_object21822:head# 2024-04-23T16:33:55.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d57187:::benchmark_data_smithi188_113957_object10607:head# 2024-04-23T16:33:55.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d581c4:::benchmark_data_smithi188_113957_object23455:head# 2024-04-23T16:33:55.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d64535:::benchmark_data_smithi188_113957_object133561:head# 2024-04-23T16:33:55.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d64f94:::benchmark_data_smithi188_113957_object43137:head# 2024-04-23T16:33:55.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d6eda7:::benchmark_data_smithi188_113957_object20101:head# 2024-04-23T16:33:55.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d7db86:::benchmark_data_smithi188_113957_object103627:head# 2024-04-23T16:33:55.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d80693:::benchmark_data_smithi188_113957_object52413:head# 2024-04-23T16:33:55.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d8344e:::benchmark_data_smithi188_113957_object11402:head# 2024-04-23T16:33:55.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d89e52:::benchmark_data_smithi188_113957_object149831:head# 2024-04-23T16:33:55.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d94c38:::benchmark_data_smithi188_113957_object119261:head# 2024-04-23T16:33:55.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d9a12e:::benchmark_data_smithi188_113957_object66148:head# 2024-04-23T16:33:55.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54d9de98:::benchmark_data_smithi188_113957_object150990:head# 2024-04-23T16:33:55.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54da051b:::benchmark_data_smithi188_113957_object67325:head# 2024-04-23T16:33:55.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54da8fbc:::benchmark_data_smithi188_113957_object28668:head# 2024-04-23T16:33:55.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54dac0cc:::benchmark_data_smithi188_113957_object42910:head# 2024-04-23T16:33:55.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54db07c2:::benchmark_data_smithi188_113957_object763:head# 2024-04-23T16:33:55.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54dbeb07:::benchmark_data_smithi188_113957_object119762:head# 2024-04-23T16:33:55.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54dc7797:::benchmark_data_smithi188_113957_object686:head# 2024-04-23T16:33:55.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54dc8d05:::benchmark_data_smithi188_113957_object97077:head# 2024-04-23T16:33:55.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54dc9a0b:::benchmark_data_smithi188_113957_object122227:head# 2024-04-23T16:33:55.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54dd3089:::benchmark_data_smithi188_113957_object148361:head# 2024-04-23T16:33:55.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54de0ed7:::benchmark_data_smithi188_113957_object98235:head# 2024-04-23T16:33:55.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54de5091:::benchmark_data_smithi188_113957_object50632:head# 2024-04-23T16:33:55.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54de6679:::benchmark_data_smithi188_113957_object50754:head# 2024-04-23T16:33:55.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54dece0e:::benchmark_data_smithi188_113957_object97643:head# 2024-04-23T16:33:55.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54dedcb4:::benchmark_data_smithi188_113957_object80424:head# 2024-04-23T16:33:55.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54df44ec:::benchmark_data_smithi188_113957_object61754:head# 2024-04-23T16:33:55.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54df8366:::benchmark_data_smithi188_113957_object152465:head# 2024-04-23T16:33:55.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e133d3:::benchmark_data_smithi188_113957_object32864:head# 2024-04-23T16:33:55.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e1eae5:::benchmark_data_smithi188_113957_object138644:head# 2024-04-23T16:33:55.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e31782:::benchmark_data_smithi188_113957_object98907:head# 2024-04-23T16:33:55.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e340a0:::benchmark_data_smithi188_113957_object6858:head# 2024-04-23T16:33:55.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e3a49d:::benchmark_data_smithi188_113957_object119368:head# 2024-04-23T16:33:55.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e3e274:::benchmark_data_smithi188_113957_object39665:head# 2024-04-23T16:33:55.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e3f9f2:::benchmark_data_smithi188_113957_object73605:head# 2024-04-23T16:33:55.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e40685:::benchmark_data_smithi188_113957_object73075:head# 2024-04-23T16:33:55.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e45e32:::benchmark_data_smithi188_113957_object94277:head# 2024-04-23T16:33:55.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e46658:::benchmark_data_smithi188_113957_object124518:head# 2024-04-23T16:33:55.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.710 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e4a826:::benchmark_data_smithi188_113957_object82860:head# 2024-04-23T16:33:55.710 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.710 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e4b4c0:::benchmark_data_smithi188_113957_object23525:head# 2024-04-23T16:33:55.710 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.710 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e4e6c9:::benchmark_data_smithi188_113957_object83542:head# 2024-04-23T16:33:55.710 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.710 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e5c921:::benchmark_data_smithi188_113957_object152798:head# 2024-04-23T16:33:55.710 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e5cac7:::benchmark_data_smithi188_113957_object109973:head# 2024-04-23T16:33:55.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e5e081:::benchmark_data_smithi188_113957_object119965:head# 2024-04-23T16:33:55.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e698a5:::benchmark_data_smithi188_113957_object60153:head# 2024-04-23T16:33:55.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e6cfbd:::benchmark_data_smithi188_113957_object80070:head# 2024-04-23T16:33:55.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e6e55c:::benchmark_data_smithi188_113957_object38577:head# 2024-04-23T16:33:55.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e6fcfb:::benchmark_data_smithi188_113957_object76256:head# 2024-04-23T16:33:55.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e77ea1:::benchmark_data_smithi188_113957_object23272:head# 2024-04-23T16:33:55.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e83015:::benchmark_data_smithi188_113957_object16279:head# 2024-04-23T16:33:55.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e88584:::benchmark_data_smithi188_113957_object102784:head# 2024-04-23T16:33:55.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e9aae9:::benchmark_data_smithi188_113957_object125121:head# 2024-04-23T16:33:55.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e9c2e6:::benchmark_data_smithi188_113957_object140590:head# 2024-04-23T16:33:55.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54e9cb65:::benchmark_data_smithi188_113957_object34776:head# 2024-04-23T16:33:55.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ea36c1:::benchmark_data_smithi188_113957_object3396:head# 2024-04-23T16:33:55.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ea49fe:::benchmark_data_smithi188_113957_object146600:head# 2024-04-23T16:33:55.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ea5007:::benchmark_data_smithi188_113957_object33172:head# 2024-04-23T16:33:55.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ebb1e3:::benchmark_data_smithi188_113957_object46394:head# 2024-04-23T16:33:55.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ebe458:::benchmark_data_smithi188_113957_object33790:head# 2024-04-23T16:33:55.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ec2039:::benchmark_data_smithi188_113957_object6784:head# 2024-04-23T16:33:55.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ec3cbd:::benchmark_data_smithi188_113957_object53967:head# 2024-04-23T16:33:55.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ec7d6d:::benchmark_data_smithi188_113957_object107323:head# 2024-04-23T16:33:55.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ec9f70:::benchmark_data_smithi188_113957_object114869:head# 2024-04-23T16:33:55.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ecc3d4:::benchmark_data_smithi188_113957_object142430:head# 2024-04-23T16:33:55.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.714 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ed98be:::benchmark_data_smithi188_113957_object25258:head# 2024-04-23T16:33:55.714 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.714 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54edd426:::benchmark_data_smithi188_113957_object136340:head# 2024-04-23T16:33:55.714 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.714 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54eec0ff:::benchmark_data_smithi188_113957_object24062:head# 2024-04-23T16:33:55.714 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.714 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54eec178:::benchmark_data_smithi188_113957_object11305:head# 2024-04-23T16:33:55.714 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.714 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54eec418:::benchmark_data_smithi188_113957_object143972:head# 2024-04-23T16:33:55.714 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.714 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ef30b2:::benchmark_data_smithi188_113957_object132853:head# 2024-04-23T16:33:55.714 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.714 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ef55ef:::benchmark_data_smithi188_113957_object47477:head# 2024-04-23T16:33:55.715 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54efc623:::benchmark_data_smithi188_113957_object87069:head# 2024-04-23T16:33:55.715 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f01730:::benchmark_data_smithi188_113957_object27630:head# 2024-04-23T16:33:55.715 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f030e8:::benchmark_data_smithi188_113957_object58639:head# 2024-04-23T16:33:55.715 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f18108:::benchmark_data_smithi188_113957_object25503:head# 2024-04-23T16:33:55.715 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f2a00f:::benchmark_data_smithi188_113957_object129286:head# 2024-04-23T16:33:55.715 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f2cf69:::benchmark_data_smithi188_113957_object117721:head# 2024-04-23T16:33:55.715 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f2e26d:::benchmark_data_smithi188_113957_object7011:head# 2024-04-23T16:33:55.716 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.716 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f392aa:::benchmark_data_smithi188_113957_object79817:head# 2024-04-23T16:33:55.716 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.716 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f41de4:::benchmark_data_smithi188_113957_object47719:head# 2024-04-23T16:33:55.716 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.716 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f47ff1:::benchmark_data_smithi188_113957_object113624:head# 2024-04-23T16:33:55.716 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.716 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f56041:::benchmark_data_smithi188_113957_object50535:head# 2024-04-23T16:33:55.716 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.716 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f5700c:::benchmark_data_smithi188_113957_object124018:head# 2024-04-23T16:33:55.716 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.716 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f671f0:::benchmark_data_smithi188_113957_object112590:head# 2024-04-23T16:33:55.716 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.716 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f675ca:::benchmark_data_smithi188_113957_object61889:head# 2024-04-23T16:33:55.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f68fe6:::benchmark_data_smithi188_113957_object55123:head# 2024-04-23T16:33:55.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f6b4bb:::benchmark_data_smithi188_113957_object128738:head# 2024-04-23T16:33:55.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f6c263:::benchmark_data_smithi188_113957_object73230:head# 2024-04-23T16:33:55.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f75ee8:::benchmark_data_smithi188_113957_object120525:head# 2024-04-23T16:33:55.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f8ced1:::benchmark_data_smithi188_113957_object69539:head# 2024-04-23T16:33:55.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f8cfc8:::benchmark_data_smithi188_113957_object120992:head# 2024-04-23T16:33:55.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f8eb09:::benchmark_data_smithi188_113957_object102604:head# 2024-04-23T16:33:55.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f92f03:::benchmark_data_smithi188_113957_object89558:head# 2024-04-23T16:33:55.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f9578e:::benchmark_data_smithi188_113957_object71706:head# 2024-04-23T16:33:55.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54f986f6:::benchmark_data_smithi188_113957_object25960:head# 2024-04-23T16:33:55.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54fa86dc:::benchmark_data_smithi188_113957_object105114:head# 2024-04-23T16:33:55.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54fa9f79:::benchmark_data_smithi188_113957_object135413:head# 2024-04-23T16:33:55.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54fb14aa:::benchmark_data_smithi188_113957_object106741:head# 2024-04-23T16:33:55.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54fb2bfe:::benchmark_data_smithi188_113957_object40040:head# 2024-04-23T16:33:55.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54fb4eb5:::benchmark_data_smithi188_113957_object133815:head# 2024-04-23T16:33:55.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54fbc553:::benchmark_data_smithi188_113957_object21318:head# 2024-04-23T16:33:55.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54fbe9ad:::benchmark_data_smithi188_113957_object16570:head# 2024-04-23T16:33:55.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54fbeda5:::benchmark_data_smithi188_113957_object106828:head# 2024-04-23T16:33:55.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54fc9365:::benchmark_data_smithi188_113957_object109751:head# 2024-04-23T16:33:55.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54fce245:::benchmark_data_smithi188_113957_object29945:head# 2024-04-23T16:33:55.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54fdb90c:::benchmark_data_smithi188_113957_object137497:head# 2024-04-23T16:33:55.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54fdea4b:::benchmark_data_smithi188_113957_object1494:head# 2024-04-23T16:33:55.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54fdf2cc:::benchmark_data_smithi188_113957_object91811:head# 2024-04-23T16:33:55.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54fea976:::benchmark_data_smithi188_113957_object121570:head# 2024-04-23T16:33:55.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:54ff2ae6:::benchmark_data_smithi188_113957_object78960:head# 2024-04-23T16:33:55.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55010991:::benchmark_data_smithi188_113957_object56834:head# 2024-04-23T16:33:55.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550114dd:::benchmark_data_smithi188_113957_object42:head# 2024-04-23T16:33:55.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55011c13:::benchmark_data_smithi188_113957_object1681:head# 2024-04-23T16:33:55.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5502098f:::benchmark_data_smithi188_113957_object108965:head# 2024-04-23T16:33:55.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55021678:::benchmark_data_smithi188_113957_object15200:head# 2024-04-23T16:33:55.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55024724:::benchmark_data_smithi188_113957_object91338:head# 2024-04-23T16:33:55.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5502b20d:::benchmark_data_smithi188_113957_object107336:head# 2024-04-23T16:33:55.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55033178:::benchmark_data_smithi188_113957_object54574:head# 2024-04-23T16:33:55.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5503caaa:::benchmark_data_smithi188_113957_object129089:head# 2024-04-23T16:33:55.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55046a48:::benchmark_data_smithi188_113957_object41902:head# 2024-04-23T16:33:55.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55051e3c:::benchmark_data_smithi188_113957_object40642:head# 2024-04-23T16:33:55.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550539e4:::benchmark_data_smithi188_113957_object3119:head# 2024-04-23T16:33:55.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55056664:::benchmark_data_smithi188_113957_object131519:head# 2024-04-23T16:33:55.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5505c25c:::benchmark_data_smithi188_113957_object104537:head# 2024-04-23T16:33:55.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5506061f:::benchmark_data_smithi188_113957_object148938:head# 2024-04-23T16:33:55.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55060662:::benchmark_data_smithi188_113957_object60065:head# 2024-04-23T16:33:55.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55066953:::benchmark_data_smithi188_113957_object119488:head# 2024-04-23T16:33:55.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55068d42:::benchmark_data_smithi188_113957_object96407:head# 2024-04-23T16:33:55.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55070d71:::benchmark_data_smithi188_113957_object111163:head# 2024-04-23T16:33:55.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55076c58:::benchmark_data_smithi188_113957_object134723:head# 2024-04-23T16:33:55.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550777d1:::benchmark_data_smithi188_113957_object70222:head# 2024-04-23T16:33:55.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55080934:::benchmark_data_smithi188_113957_object69992:head# 2024-04-23T16:33:55.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55081593:::benchmark_data_smithi188_113957_object55828:head# 2024-04-23T16:33:55.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55081ef4:::benchmark_data_smithi188_113957_object62573:head# 2024-04-23T16:33:55.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5508213b:::benchmark_data_smithi188_113957_object102450:head# 2024-04-23T16:33:55.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5508299e:::benchmark_data_smithi188_113957_object145653:head# 2024-04-23T16:33:55.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55083e1f:::benchmark_data_smithi188_113957_object147938:head# 2024-04-23T16:33:55.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55085422:::benchmark_data_smithi188_113957_object134401:head# 2024-04-23T16:33:55.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550951c9:::benchmark_data_smithi188_113957_object71507:head# 2024-04-23T16:33:55.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550a1d6e:::benchmark_data_smithi188_113957_object118744:head# 2024-04-23T16:33:55.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550a29e1:::benchmark_data_smithi188_113957_object18139:head# 2024-04-23T16:33:55.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550a304b:::benchmark_data_smithi188_113957_object60909:head# 2024-04-23T16:33:55.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550a31a1:::benchmark_data_smithi188_113957_object103866:head# 2024-04-23T16:33:55.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550a4f94:::benchmark_data_smithi188_113957_object6174:head# 2024-04-23T16:33:55.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550adc0c:::benchmark_data_smithi188_113957_object50880:head# 2024-04-23T16:33:55.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550b3108:::benchmark_data_smithi188_113957_object16789:head# 2024-04-23T16:33:55.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550b8b38:::benchmark_data_smithi188_113957_object33585:head# 2024-04-23T16:33:55.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550c3983:::benchmark_data_smithi188_113957_object99523:head# 2024-04-23T16:33:55.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550c769f:::benchmark_data_smithi188_113957_object81901:head# 2024-04-23T16:33:55.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550cc5cf:::benchmark_data_smithi188_113957_object21231:head# 2024-04-23T16:33:55.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550cd423:::benchmark_data_smithi188_113957_object66555:head# 2024-04-23T16:33:55.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550cf1f3:::benchmark_data_smithi188_113957_object116089:head# 2024-04-23T16:33:55.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550daee6:::benchmark_data_smithi188_113957_object51081:head# 2024-04-23T16:33:55.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550ed300:::benchmark_data_smithi188_113957_object51473:head# 2024-04-23T16:33:55.727 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550f500c:::benchmark_data_smithi188_113957_object32126:head# 2024-04-23T16:33:55.727 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:550faaeb:::benchmark_data_smithi188_113957_object144471:head# 2024-04-23T16:33:55.727 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:551059f5:::benchmark_data_smithi188_113957_object33025:head# 2024-04-23T16:33:55.727 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5510ad99:::benchmark_data_smithi188_113957_object120137:head# 2024-04-23T16:33:55.727 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5510d2e6:::benchmark_data_smithi188_113957_object102235:head# 2024-04-23T16:33:55.727 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55114371:::benchmark_data_smithi188_113957_object39051:head# 2024-04-23T16:33:55.727 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.728 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5512b00a:::benchmark_data_smithi188_113957_object23912:head# 2024-04-23T16:33:55.728 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.728 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5512f10b:::benchmark_data_smithi188_113957_object39315:head# 2024-04-23T16:33:55.728 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.728 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5513349c:::benchmark_data_smithi188_113957_object89606:head# 2024-04-23T16:33:55.728 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.728 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55134f99:::benchmark_data_smithi188_113957_object128576:head# 2024-04-23T16:33:55.728 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.728 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5514032c:::benchmark_data_smithi188_113957_object107801:head# 2024-04-23T16:33:55.728 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.728 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:551439d1:::benchmark_data_smithi188_113957_object37309:head# 2024-04-23T16:33:55.728 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.728 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55147856:::benchmark_data_smithi188_113957_object33167:head# 2024-04-23T16:33:55.728 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.729 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:551485d0:::benchmark_data_smithi188_113957_object86886:head# 2024-04-23T16:33:55.729 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.729 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55157647:::benchmark_data_smithi188_113957_object79331:head# 2024-04-23T16:33:55.729 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.729 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:551618b1:::benchmark_data_smithi188_113957_object61263:head# 2024-04-23T16:33:55.729 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.729 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55176347:::benchmark_data_smithi188_113957_object2279:head# 2024-04-23T16:33:55.729 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.729 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5519fa31:::benchmark_data_smithi188_113957_object88202:head# 2024-04-23T16:33:55.729 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.729 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:551a1c62:::benchmark_data_smithi188_113957_object65915:head# 2024-04-23T16:33:55.729 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.729 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:551a98dc:::benchmark_data_smithi188_113957_object149507:head# 2024-04-23T16:33:55.729 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:551b1108:::benchmark_data_smithi188_113957_object58996:head# 2024-04-23T16:33:55.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:551ba4b6:::benchmark_data_smithi188_113957_object68113:head# 2024-04-23T16:33:55.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:551c26c9:::benchmark_data_smithi188_113957_object6127:head# 2024-04-23T16:33:55.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:551c348b:::benchmark_data_smithi188_113957_object62560:head# 2024-04-23T16:33:55.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:551c5b1d:::benchmark_data_smithi188_113957_object12244:head# 2024-04-23T16:33:55.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:551cf2e7:::benchmark_data_smithi188_113957_object106503:head# 2024-04-23T16:33:55.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:551ee632:::benchmark_data_smithi188_113957_object92717:head# 2024-04-23T16:33:55.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:551f523c:::benchmark_data_smithi188_113957_object90867:head# 2024-04-23T16:33:55.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5520bbbf:::benchmark_data_smithi188_113957_object13958:head# 2024-04-23T16:33:55.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55211228:::benchmark_data_smithi188_113957_object142335:head# 2024-04-23T16:33:55.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55211a20:::benchmark_data_smithi188_113957_object116162:head# 2024-04-23T16:33:55.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55225a1d:::benchmark_data_smithi188_113957_object139470:head# 2024-04-23T16:33:55.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5522dbff:::benchmark_data_smithi188_113957_object46292:head# 2024-04-23T16:33:55.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5523752a:::benchmark_data_smithi188_113957_object135869:head# 2024-04-23T16:33:55.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55237a94:::benchmark_data_smithi188_113957_object123859:head# 2024-04-23T16:33:55.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5523b08d:::benchmark_data_smithi188_113957_object6183:head# 2024-04-23T16:33:55.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5523c173:::benchmark_data_smithi188_113957_object30416:head# 2024-04-23T16:33:55.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5524498f:::benchmark_data_smithi188_113957_object138480:head# 2024-04-23T16:33:55.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55246b27:::benchmark_data_smithi188_113957_object84222:head# 2024-04-23T16:33:55.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5524bcb6:::benchmark_data_smithi188_113957_object56738:head# 2024-04-23T16:33:55.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5524ed4c:::benchmark_data_smithi188_113957_object115378:head# 2024-04-23T16:33:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55250085:::benchmark_data_smithi188_113957_object7845:head# 2024-04-23T16:33:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55258ba0:::benchmark_data_smithi188_113957_object57219:head# 2024-04-23T16:33:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:552610da:::benchmark_data_smithi188_113957_object116700:head# 2024-04-23T16:33:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55267846:::benchmark_data_smithi188_113957_object146124:head# 2024-04-23T16:33:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5526e5e0:::benchmark_data_smithi188_113957_object62718:head# 2024-04-23T16:33:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:552818fc:::benchmark_data_smithi188_113957_object98248:head# 2024-04-23T16:33:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55283fb8:::benchmark_data_smithi188_113957_object83085:head# 2024-04-23T16:33:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5528a32c:::benchmark_data_smithi188_113957_object58366:head# 2024-04-23T16:33:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5528caa7:::benchmark_data_smithi188_113957_object141303:head# 2024-04-23T16:33:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5528ec87:::benchmark_data_smithi188_113957_object81910:head# 2024-04-23T16:33:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55295c38:::benchmark_data_smithi188_113957_object76180:head# 2024-04-23T16:33:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5529a2ca:::benchmark_data_smithi188_113957_object6832:head# 2024-04-23T16:33:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5529d512:::benchmark_data_smithi188_113957_object143886:head# 2024-04-23T16:33:55.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:552aac7d:::benchmark_data_smithi188_113957_object36158:head# 2024-04-23T16:33:55.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:552ab278:::benchmark_data_smithi188_113957_object146884:head# 2024-04-23T16:33:55.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:552b6020:::benchmark_data_smithi188_113957_object40984:head# 2024-04-23T16:33:55.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:552bde8f:::benchmark_data_smithi188_113957_object54168:head# 2024-04-23T16:33:55.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:552c7855:::benchmark_data_smithi188_113957_object37122:head# 2024-04-23T16:33:55.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:552c7b8b:::benchmark_data_smithi188_113957_object105518:head# 2024-04-23T16:33:55.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:552c9124:::benchmark_data_smithi188_113957_object140895:head# 2024-04-23T16:33:55.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:552caf97:::benchmark_data_smithi188_113957_object145582:head# 2024-04-23T16:33:55.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:552cd443:::benchmark_data_smithi188_113957_object53611:head# 2024-04-23T16:33:55.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:552d694d:::benchmark_data_smithi188_113957_object81413:head# 2024-04-23T16:33:55.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:552e7f16:::benchmark_data_smithi188_113957_object127853:head# 2024-04-23T16:33:55.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:552ebeea:::benchmark_data_smithi188_113957_object144228:head# 2024-04-23T16:33:55.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5530a9ad:::benchmark_data_smithi188_113957_object64789:head# 2024-04-23T16:33:55.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5531bcba:::benchmark_data_smithi188_113957_object21540:head# 2024-04-23T16:33:55.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.737 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5533c9c1:::benchmark_data_smithi188_113957_object4648:head# 2024-04-23T16:33:55.737 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.737 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5533ce43:::benchmark_data_smithi188_113957_object65511:head# 2024-04-23T16:33:55.737 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.737 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5533ff77:::benchmark_data_smithi188_113957_object120162:head# 2024-04-23T16:33:55.737 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.737 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55355167:::benchmark_data_smithi188_113957_object38573:head# 2024-04-23T16:33:55.737 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.737 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5535b5f4:::benchmark_data_smithi188_113957_object68889:head# 2024-04-23T16:33:55.737 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.737 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553620a0:::benchmark_data_smithi188_113957_object73614:head# 2024-04-23T16:33:55.737 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.737 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55370a66:::benchmark_data_smithi188_113957_object46536:head# 2024-04-23T16:33:55.737 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5537bafa:::benchmark_data_smithi188_113957_object62656:head# 2024-04-23T16:33:55.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5537fea3:::benchmark_data_smithi188_113957_object1245:head# 2024-04-23T16:33:55.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553853af:::benchmark_data_smithi188_113957_object100915:head# 2024-04-23T16:33:55.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5539024a:::benchmark_data_smithi188_113957_object103742:head# 2024-04-23T16:33:55.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553926ca:::benchmark_data_smithi188_113957_object3166:head# 2024-04-23T16:33:55.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55395102:::benchmark_data_smithi188_113957_object90707:head# 2024-04-23T16:33:55.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553a412b:::benchmark_data_smithi188_113957_object89189:head# 2024-04-23T16:33:55.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553a8709:::benchmark_data_smithi188_113957_object85542:head# 2024-04-23T16:33:55.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553b256e:::benchmark_data_smithi188_113957_object59949:head# 2024-04-23T16:33:55.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553b4aaa:::benchmark_data_smithi188_113957_object125459:head# 2024-04-23T16:33:55.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553b63da:::benchmark_data_smithi188_113957_object108240:head# 2024-04-23T16:33:55.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553bb021:::benchmark_data_smithi188_113957_object139928:head# 2024-04-23T16:33:55.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553bb8b1:::benchmark_data_smithi188_113957_object79358:head# 2024-04-23T16:33:55.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553d32ee:::benchmark_data_smithi188_113957_object149927:head# 2024-04-23T16:33:55.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.740 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553d78de:::benchmark_data_smithi188_113957_object64917:head# 2024-04-23T16:33:55.740 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.740 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553dd5e6:::benchmark_data_smithi188_113957_object22337:head# 2024-04-23T16:33:55.740 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.740 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553e195a:::benchmark_data_smithi188_113957_object75906:head# 2024-04-23T16:33:55.740 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.740 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553e4def:::benchmark_data_smithi188_113957_object68499:head# 2024-04-23T16:33:55.740 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.740 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553ed765:::benchmark_data_smithi188_113957_object89650:head# 2024-04-23T16:33:55.740 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.740 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553f09ee:::benchmark_data_smithi188_113957_object114921:head# 2024-04-23T16:33:55.740 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.740 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553f2b34:::benchmark_data_smithi188_113957_object92070:head# 2024-04-23T16:33:55.740 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.741 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553f4836:::benchmark_data_smithi188_113957_object25087:head# 2024-04-23T16:33:55.741 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.741 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553f647b:::benchmark_data_smithi188_113957_object139449:head# 2024-04-23T16:33:55.741 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.741 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553f724c:::benchmark_data_smithi188_113957_object59371:head# 2024-04-23T16:33:55.741 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.741 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:33:55.828 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:55 smithi148 ceph-mon[44520]: pgmap v1130: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.4 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 622 op/s; 25846/274602 objects degraded (9.412%); 7.6 MiB/s, 120 objects/s recovering 2024-04-23T16:33:55.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:553fa479:::benchmark_data_smithi188_113957_object106702:head# 2024-04-23T16:33:55.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5540308d:::benchmark_data_smithi188_113957_object88176:head# 2024-04-23T16:33:55.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55404475:::benchmark_data_smithi188_113957_object97361:head# 2024-04-23T16:33:55.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5540a39b:::benchmark_data_smithi188_113957_object53870:head# 2024-04-23T16:33:55.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5540f3f7:::benchmark_data_smithi188_113957_object44560:head# 2024-04-23T16:33:55.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554119e2:::benchmark_data_smithi188_113957_object144570:head# 2024-04-23T16:33:55.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5541751c:::benchmark_data_smithi188_113957_object126357:head# 2024-04-23T16:33:55.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5541c043:::benchmark_data_smithi188_113957_object111285:head# 2024-04-23T16:33:55.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5541cd26:::benchmark_data_smithi188_113957_object144459:head# 2024-04-23T16:33:55.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5541f98c:::benchmark_data_smithi188_113957_object133597:head# 2024-04-23T16:33:55.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554201f7:::benchmark_data_smithi188_113957_object36686:head# 2024-04-23T16:33:55.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554218ad:::benchmark_data_smithi188_113957_object139793:head# 2024-04-23T16:33:55.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5542b291:::benchmark_data_smithi188_113957_object130587:head# 2024-04-23T16:33:55.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5543119b:::benchmark_data_smithi188_113957_object11468:head# 2024-04-23T16:33:55.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55441881:::benchmark_data_smithi188_113957_object136258:head# 2024-04-23T16:33:55.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55443686:::benchmark_data_smithi188_113957_object62265:head# 2024-04-23T16:33:55.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5544a4ce:::benchmark_data_smithi188_113957_object87864:head# 2024-04-23T16:33:55.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55452612:::benchmark_data_smithi188_113957_object38408:head# 2024-04-23T16:33:55.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55452a83:::benchmark_data_smithi188_113957_object73504:head# 2024-04-23T16:33:55.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5545541c:::benchmark_data_smithi188_113957_object136204:head# 2024-04-23T16:33:55.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5545913f:::benchmark_data_smithi188_113957_object82271:head# 2024-04-23T16:33:55.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5545f392:::benchmark_data_smithi188_113957_object75560:head# 2024-04-23T16:33:55.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5545fdb0:::benchmark_data_smithi188_113957_object19142:head# 2024-04-23T16:33:55.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55461741:::benchmark_data_smithi188_113957_object102151:head# 2024-04-23T16:33:55.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55462c20:::benchmark_data_smithi188_113957_object118423:head# 2024-04-23T16:33:55.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554656a6:::benchmark_data_smithi188_113957_object38366:head# 2024-04-23T16:33:55.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5546fa00:::benchmark_data_smithi188_113957_object130436:head# 2024-04-23T16:33:55.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554716d5:::benchmark_data_smithi188_113957_object99925:head# 2024-04-23T16:33:55.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55476c2f:::benchmark_data_smithi188_113957_object98111:head# 2024-04-23T16:33:55.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55482f2f:::benchmark_data_smithi188_113957_object79494:head# 2024-04-23T16:33:55.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5548453b:::benchmark_data_smithi188_113957_object56005:head# 2024-04-23T16:33:55.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55485531:::benchmark_data_smithi188_113957_object91573:head# 2024-04-23T16:33:55.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55489592:::benchmark_data_smithi188_113957_object90208:head# 2024-04-23T16:33:55.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5548b4ca:::benchmark_data_smithi188_113957_object94270:head# 2024-04-23T16:33:55.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5549652a:::benchmark_data_smithi188_113957_object119096:head# 2024-04-23T16:33:55.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5549876c:::benchmark_data_smithi188_113957_object3046:head# 2024-04-23T16:33:55.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554a1244:::benchmark_data_smithi188_113957_object20836:head# 2024-04-23T16:33:55.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554a9095:::benchmark_data_smithi188_113957_object21284:head# 2024-04-23T16:33:55.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554a989d:::benchmark_data_smithi188_113957_object21345:head# 2024-04-23T16:33:55.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554aa3c2:::benchmark_data_smithi188_113957_object75519:head# 2024-04-23T16:33:55.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554b8a4b:::benchmark_data_smithi188_113957_object61884:head# 2024-04-23T16:33:55.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554bd0cd:::benchmark_data_smithi188_113957_object132880:head# 2024-04-23T16:33:55.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554c22c9:::benchmark_data_smithi188_113957_object131497:head# 2024-04-23T16:33:55.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554cb6c6:::benchmark_data_smithi188_113957_object34879:head# 2024-04-23T16:33:55.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554d5caf:::benchmark_data_smithi188_113957_object25166:head# 2024-04-23T16:33:55.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554e4f23:::benchmark_data_smithi188_113957_object97875:head# 2024-04-23T16:33:55.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554e5142:::benchmark_data_smithi188_113957_object6596:head# 2024-04-23T16:33:55.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554edae6:::benchmark_data_smithi188_113957_object56562:head# 2024-04-23T16:33:55.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554f3552:::benchmark_data_smithi188_113957_object135840:head# 2024-04-23T16:33:55.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554f4765:::benchmark_data_smithi188_113957_object41202:head# 2024-04-23T16:33:55.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554f8818:::benchmark_data_smithi188_113957_object84913:head# 2024-04-23T16:33:55.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:554fd75a:::benchmark_data_smithi188_113957_object127386:head# 2024-04-23T16:33:55.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555034e7:::benchmark_data_smithi188_113957_object132124:head# 2024-04-23T16:33:55.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555047c7:::benchmark_data_smithi188_113957_object138301:head# 2024-04-23T16:33:55.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55508ef9:::benchmark_data_smithi188_113957_object50163:head# 2024-04-23T16:33:55.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5550afac:::benchmark_data_smithi188_113957_object83876:head# 2024-04-23T16:33:55.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5550f044:::benchmark_data_smithi188_113957_object59389:head# 2024-04-23T16:33:55.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55515f55:::benchmark_data_smithi188_113957_object110658:head# 2024-04-23T16:33:55.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55517056:::benchmark_data_smithi188_113957_object92981:head# 2024-04-23T16:33:55.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55521946:::benchmark_data_smithi188_113957_object97912:head# 2024-04-23T16:33:55.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5552c3dd:::benchmark_data_smithi188_113957_object67469:head# 2024-04-23T16:33:55.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55530f0f:::benchmark_data_smithi188_113957_object2525:head# 2024-04-23T16:33:55.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555354b4:::benchmark_data_smithi188_113957_object23562:head# 2024-04-23T16:33:55.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555356bf:::benchmark_data_smithi188_113957_object153383:head# 2024-04-23T16:33:55.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5553edd0:::benchmark_data_smithi188_113957_object66766:head# 2024-04-23T16:33:55.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5554887e:::benchmark_data_smithi188_113957_object82809:head# 2024-04-23T16:33:55.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555562d1:::benchmark_data_smithi188_113957_object39331:head# 2024-04-23T16:33:55.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555594d1:::benchmark_data_smithi188_113957_object136978:head# 2024-04-23T16:33:55.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5555cfba:::benchmark_data_smithi188_113957_object62512:head# 2024-04-23T16:33:55.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5555d784:::benchmark_data_smithi188_113957_object32875:head# 2024-04-23T16:33:55.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5555d985:::benchmark_data_smithi188_113957_object60506:head# 2024-04-23T16:33:55.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5556be61:::benchmark_data_smithi188_113957_object35781:head# 2024-04-23T16:33:55.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5556d917:::benchmark_data_smithi188_113957_object88487:head# 2024-04-23T16:33:55.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5557534a:::benchmark_data_smithi188_113957_object116129:head# 2024-04-23T16:33:55.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555753e6:::benchmark_data_smithi188_113957_object50987:head# 2024-04-23T16:33:55.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5557f883:::benchmark_data_smithi188_113957_object134408:head# 2024-04-23T16:33:55.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55582a75:::benchmark_data_smithi188_113957_object119861:head# 2024-04-23T16:33:55.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55590f46:::benchmark_data_smithi188_113957_object8707:head# 2024-04-23T16:33:55.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555a6da2:::benchmark_data_smithi188_113957_object145231:head# 2024-04-23T16:33:55.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555a97d2:::benchmark_data_smithi188_113957_object104186:head# 2024-04-23T16:33:55.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555aa9e3:::benchmark_data_smithi188_113957_object112465:head# 2024-04-23T16:33:55.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555abdac:::benchmark_data_smithi188_113957_object131062:head# 2024-04-23T16:33:55.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555b5242:::benchmark_data_smithi188_113957_object49165:head# 2024-04-23T16:33:55.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555c3048:::benchmark_data_smithi188_113957_object141708:head# 2024-04-23T16:33:55.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555cfff2:::benchmark_data_smithi188_113957_object65322:head# 2024-04-23T16:33:55.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555d007c:::benchmark_data_smithi188_113957_object134551:head# 2024-04-23T16:33:55.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555d7cce:::benchmark_data_smithi188_113957_object50688:head# 2024-04-23T16:33:55.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555de223:::benchmark_data_smithi188_113957_object37277:head# 2024-04-23T16:33:55.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555e5683:::benchmark_data_smithi188_113957_object69035:head# 2024-04-23T16:33:55.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555eb516:::benchmark_data_smithi188_113957_object105205:head# 2024-04-23T16:33:55.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555eb6c2:::benchmark_data_smithi188_113957_object51982:head# 2024-04-23T16:33:55.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555f2e26:::benchmark_data_smithi188_113957_object36043:head# 2024-04-23T16:33:55.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:555f4365:::benchmark_data_smithi188_113957_object98025:head# 2024-04-23T16:33:55.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55605097:::benchmark_data_smithi188_113957_object54457:head# 2024-04-23T16:33:55.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5560b3f9:::benchmark_data_smithi188_113957_object107518:head# 2024-04-23T16:33:55.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55611733:::benchmark_data_smithi188_113957_object22119:head# 2024-04-23T16:33:55.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:556151f0:::benchmark_data_smithi188_113957_object102012:head# 2024-04-23T16:33:55.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5563878e:::benchmark_data_smithi188_113957_object110267:head# 2024-04-23T16:33:55.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5563b4e5:::benchmark_data_smithi188_113957_object98339:head# 2024-04-23T16:33:55.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55648f15:::benchmark_data_smithi188_113957_object15321:head# 2024-04-23T16:33:55.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55655d6c:::benchmark_data_smithi188_113957_object90702:head# 2024-04-23T16:33:55.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5565c2c2:::benchmark_data_smithi188_113957_object107591:head# 2024-04-23T16:33:55.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55660a12:::benchmark_data_smithi188_113957_object48648:head# 2024-04-23T16:33:55.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55664e81:::benchmark_data_smithi188_113957_object23796:head# 2024-04-23T16:33:55.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:556668d5:::benchmark_data_smithi188_113957_object34805:head# 2024-04-23T16:33:55.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55666c79:::benchmark_data_smithi188_113957_object109852:head# 2024-04-23T16:33:55.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55668499:::benchmark_data_smithi188_113957_object37483:head# 2024-04-23T16:33:55.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55668a72:::benchmark_data_smithi188_113957_object81712:head# 2024-04-23T16:33:55.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55668f0a:::benchmark_data_smithi188_113957_object91995:head# 2024-04-23T16:33:55.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5566c85c:::benchmark_data_smithi188_113957_object55020:head# 2024-04-23T16:33:55.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55681f53:::benchmark_data_smithi188_113957_object43502:head# 2024-04-23T16:33:55.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55682c7b:::benchmark_data_smithi188_113957_object118283:head# 2024-04-23T16:33:55.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55684ec5:::benchmark_data_smithi188_113957_object139465:head# 2024-04-23T16:33:55.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:556a17b5:::benchmark_data_smithi188_113957_object69116:head# 2024-04-23T16:33:55.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:556a1cc9:::benchmark_data_smithi188_113957_object57463:head# 2024-04-23T16:33:55.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:556b637c:::benchmark_data_smithi188_113957_object146036:head# 2024-04-23T16:33:55.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:556bd1be:::benchmark_data_smithi188_113957_object33006:head# 2024-04-23T16:33:55.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:556c39ef:::benchmark_data_smithi188_113957_object125613:head# 2024-04-23T16:33:55.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:556d3aa7:::benchmark_data_smithi188_113957_object50094:head# 2024-04-23T16:33:55.884 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:556dce14:::benchmark_data_smithi188_113957_object15482:head# 2024-04-23T16:33:55.884 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:556e174b:::benchmark_data_smithi188_113957_object130189:head# 2024-04-23T16:33:55.884 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:556e7794:::benchmark_data_smithi188_113957_object151869:head# 2024-04-23T16:33:55.884 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:556e8265:::benchmark_data_smithi188_113957_object75700:head# 2024-04-23T16:33:55.884 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:556ef0ec:::benchmark_data_smithi188_113957_object150816:head# 2024-04-23T16:33:55.884 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:556f699a:::benchmark_data_smithi188_113957_object41599:head# 2024-04-23T16:33:55.884 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:556fbbf3:::benchmark_data_smithi188_113957_object114524:head# 2024-04-23T16:33:55.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:556fdb33:::benchmark_data_smithi188_113957_object42103:head# 2024-04-23T16:33:55.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5570bc91:::benchmark_data_smithi188_113957_object124982:head# 2024-04-23T16:33:55.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5570d663:::benchmark_data_smithi188_113957_object7761:head# 2024-04-23T16:33:55.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5571f985:::benchmark_data_smithi188_113957_object49719:head# 2024-04-23T16:33:55.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55720340:::benchmark_data_smithi188_113957_object134507:head# 2024-04-23T16:33:55.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5572dae9:::benchmark_data_smithi188_113957_object88682:head# 2024-04-23T16:33:55.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5572e3be:::benchmark_data_smithi188_113957_object112951:head# 2024-04-23T16:33:55.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:557338d8:::benchmark_data_smithi188_113957_object69744:head# 2024-04-23T16:33:55.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5573933b:::benchmark_data_smithi188_113957_object148860:head# 2024-04-23T16:33:55.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:557443f7:::benchmark_data_smithi188_113957_object142520:head# 2024-04-23T16:33:55.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5575b8f5:::benchmark_data_smithi188_113957_object35326:head# 2024-04-23T16:33:55.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5575ba08:::benchmark_data_smithi188_113957_object59505:head# 2024-04-23T16:33:55.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5575c5d5:::benchmark_data_smithi188_113957_object81436:head# 2024-04-23T16:33:55.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5575db80:::benchmark_data_smithi188_113957_object108589:head# 2024-04-23T16:33:55.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5576c38f:::benchmark_data_smithi188_113957_object125231:head# 2024-04-23T16:33:55.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5577df55:::benchmark_data_smithi188_113957_object68979:head# 2024-04-23T16:33:55.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55787810:::benchmark_data_smithi188_113957_object16603:head# 2024-04-23T16:33:55.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:557884dd:::benchmark_data_smithi188_113957_object145140:head# 2024-04-23T16:33:55.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5578887d:::benchmark_data_smithi188_113957_object122851:head# 2024-04-23T16:33:55.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5578cd74:::benchmark_data_smithi188_113957_object119472:head# 2024-04-23T16:33:55.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5578fb50:::benchmark_data_smithi188_113957_object23210:head# 2024-04-23T16:33:55.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55790921:::benchmark_data_smithi188_113957_object102830:head# 2024-04-23T16:33:55.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5579ded6:::benchmark_data_smithi188_113957_object32428:head# 2024-04-23T16:33:55.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5579fa41:::benchmark_data_smithi188_113957_object100930:head# 2024-04-23T16:33:55.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:557a01a8:::benchmark_data_smithi188_113957_object8408:head# 2024-04-23T16:33:55.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:557aab32:::benchmark_data_smithi188_113957_object12413:head# 2024-04-23T16:33:55.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:557b4224:::benchmark_data_smithi188_113957_object128565:head# 2024-04-23T16:33:55.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:557b50c2:::benchmark_data_smithi188_113957_object59126:head# 2024-04-23T16:33:55.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:557c5e70:::benchmark_data_smithi188_113957_object75141:head# 2024-04-23T16:33:55.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:557d03d1:::benchmark_data_smithi188_113957_object31756:head# 2024-04-23T16:33:55.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:557dde19:::benchmark_data_smithi188_113957_object146845:head# 2024-04-23T16:33:55.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:557e7cbb:::benchmark_data_smithi188_113957_object84229:head# 2024-04-23T16:33:55.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:557efdc0:::benchmark_data_smithi188_113957_object24626:head# 2024-04-23T16:33:55.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:557f0b85:::benchmark_data_smithi188_113957_object92231:head# 2024-04-23T16:33:55.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:557fa1f8:::benchmark_data_smithi188_113957_object11563:head# 2024-04-23T16:33:55.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55806fe7:::benchmark_data_smithi188_113957_object33667:head# 2024-04-23T16:33:55.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5580b4ed:::benchmark_data_smithi188_113957_object111461:head# 2024-04-23T16:33:55.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5580e78a:::benchmark_data_smithi188_113957_object79892:head# 2024-04-23T16:33:55.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5581842d:::benchmark_data_smithi188_113957_object53169:head# 2024-04-23T16:33:55.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5581b8b6:::benchmark_data_smithi188_113957_object113842:head# 2024-04-23T16:33:55.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55821510:::benchmark_data_smithi188_113957_object39957:head# 2024-04-23T16:33:55.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558247b5:::benchmark_data_smithi188_113957_object62205:head# 2024-04-23T16:33:55.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55825308:::benchmark_data_smithi188_113957_object10370:head# 2024-04-23T16:33:55.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55825c77:::benchmark_data_smithi188_113957_object70997:head# 2024-04-23T16:33:55.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55835ddc:::benchmark_data_smithi188_113957_object60269:head# 2024-04-23T16:33:55.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55836d4b:::benchmark_data_smithi188_113957_object116051:head# 2024-04-23T16:33:55.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5583c262:::benchmark_data_smithi188_113957_object72295:head# 2024-04-23T16:33:55.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.892 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55845296:::benchmark_data_smithi188_113957_object103302:head# 2024-04-23T16:33:55.892 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.892 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5584793f:::benchmark_data_smithi188_113957_object65873:head# 2024-04-23T16:33:55.892 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.892 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5584e629:::benchmark_data_smithi188_113957_object109666:head# 2024-04-23T16:33:55.892 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.892 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5585ac42:::benchmark_data_smithi188_113957_object127260:head# 2024-04-23T16:33:55.892 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.892 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5585c1de:::benchmark_data_smithi188_113957_object94107:head# 2024-04-23T16:33:55.892 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.892 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5585c55a:::benchmark_data_smithi188_113957_object60200:head# 2024-04-23T16:33:55.892 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.892 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5585f9ea:::benchmark_data_smithi188_113957_object43023:head# 2024-04-23T16:33:55.892 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.893 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5586027b:::benchmark_data_smithi188_113957_object114195:head# 2024-04-23T16:33:55.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.893 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55863659:::benchmark_data_smithi188_113957_object37538:head# 2024-04-23T16:33:55.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.893 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5586ce5e:::benchmark_data_smithi188_113957_object140679:head# 2024-04-23T16:33:55.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.893 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5588239b:::benchmark_data_smithi188_113957_object3214:head# 2024-04-23T16:33:55.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.893 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55890873:::benchmark_data_smithi188_113957_object27186:head# 2024-04-23T16:33:55.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.893 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55897386:::benchmark_data_smithi188_113957_object71730:head# 2024-04-23T16:33:55.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.893 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5589a10c:::benchmark_data_smithi188_113957_object147267:head# 2024-04-23T16:33:55.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5589c053:::benchmark_data_smithi188_113957_object100360:head# 2024-04-23T16:33:55.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558a0186:::benchmark_data_smithi188_113957_object69575:head# 2024-04-23T16:33:55.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558a0c78:::benchmark_data_smithi188_113957_object33469:head# 2024-04-23T16:33:55.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558a282a:::benchmark_data_smithi188_113957_object18721:head# 2024-04-23T16:33:55.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558a448a:::benchmark_data_smithi188_113957_object135695:head# 2024-04-23T16:33:55.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558abcef:::benchmark_data_smithi188_113957_object28956:head# 2024-04-23T16:33:55.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558acd13:::benchmark_data_smithi188_113957_object97044:head# 2024-04-23T16:33:55.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558af6de:::benchmark_data_smithi188_113957_object85313:head# 2024-04-23T16:33:55.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558b3736:::benchmark_data_smithi188_113957_object48356:head# 2024-04-23T16:33:55.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558b4061:::benchmark_data_smithi188_113957_object21886:head# 2024-04-23T16:33:55.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558b64cf:::benchmark_data_smithi188_113957_object64883:head# 2024-04-23T16:33:55.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558b6936:::benchmark_data_smithi188_113957_object147679:head# 2024-04-23T16:33:55.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558b7f66:::benchmark_data_smithi188_113957_object53282:head# 2024-04-23T16:33:55.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558baee7:::benchmark_data_smithi188_113957_object71269:head# 2024-04-23T16:33:55.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558bc523:::benchmark_data_smithi188_113957_object47849:head# 2024-04-23T16:33:55.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558c66fd:::benchmark_data_smithi188_113957_object847:head# 2024-04-23T16:33:55.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558c9c1b:::benchmark_data_smithi188_113957_object84697:head# 2024-04-23T16:33:55.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558ccff4:::benchmark_data_smithi188_113957_object93189:head# 2024-04-23T16:33:55.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558cff9b:::benchmark_data_smithi188_113957_object71519:head# 2024-04-23T16:33:55.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558d2b0a:::benchmark_data_smithi188_113957_object149634:head# 2024-04-23T16:33:55.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558d7268:::benchmark_data_smithi188_113957_object95083:head# 2024-04-23T16:33:55.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558de549:::benchmark_data_smithi188_113957_object102106:head# 2024-04-23T16:33:55.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558e2574:::benchmark_data_smithi188_113957_object151154:head# 2024-04-23T16:33:55.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558e9cf8:::benchmark_data_smithi188_113957_object103737:head# 2024-04-23T16:33:55.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558ed926:::benchmark_data_smithi188_113957_object38960:head# 2024-04-23T16:33:55.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558f6d1a:::benchmark_data_smithi188_113957_object89915:head# 2024-04-23T16:33:55.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:558f7f9e:::benchmark_data_smithi188_113957_object105446:head# 2024-04-23T16:33:55.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:559078d1:::benchmark_data_smithi188_113957_object131801:head# 2024-04-23T16:33:55.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55909666:::benchmark_data_smithi188_113957_object129839:head# 2024-04-23T16:33:55.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5591993f:::benchmark_data_smithi188_113957_object49650:head# 2024-04-23T16:33:55.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5591e8fc:::benchmark_data_smithi188_113957_object19022:head# 2024-04-23T16:33:55.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5591fc55:::benchmark_data_smithi188_113957_object91836:head# 2024-04-23T16:33:55.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:559200b3:::benchmark_data_smithi188_113957_object129826:head# 2024-04-23T16:33:55.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55923d64:::benchmark_data_smithi188_113957_object129213:head# 2024-04-23T16:33:55.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.899 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55926f2f:::benchmark_data_smithi188_113957_object58556:head# 2024-04-23T16:33:55.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.899 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5593068c:::benchmark_data_smithi188_113957_object36911:head# 2024-04-23T16:33:55.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.899 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5593244f:::benchmark_data_smithi188_113957_object140028:head# 2024-04-23T16:33:55.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.899 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:559337fa:::benchmark_data_smithi188_113957_object89700:head# 2024-04-23T16:33:55.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.899 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5593bca7:::benchmark_data_smithi188_113957_object98481:head# 2024-04-23T16:33:55.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.899 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5593d352:::benchmark_data_smithi188_113957_object26552:head# 2024-04-23T16:33:55.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.899 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5593d96c:::benchmark_data_smithi188_113957_object87714:head# 2024-04-23T16:33:55.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5594a200:::benchmark_data_smithi188_113957_object131518:head# 2024-04-23T16:33:55.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55955aee:::benchmark_data_smithi188_113957_object51519:head# 2024-04-23T16:33:55.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55959e84:::benchmark_data_smithi188_113957_object124824:head# 2024-04-23T16:33:55.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:559602b3:::benchmark_data_smithi188_113957_object103096:head# 2024-04-23T16:33:55.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55965540:::benchmark_data_smithi188_113957_object24944:head# 2024-04-23T16:33:55.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5596d2de:::benchmark_data_smithi188_113957_object104757:head# 2024-04-23T16:33:55.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55971129:::benchmark_data_smithi188_113957_object140133:head# 2024-04-23T16:33:55.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.901 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55977503:::benchmark_data_smithi188_113957_object74671:head# 2024-04-23T16:33:55.901 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.901 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5598148f:::benchmark_data_smithi188_113957_object128538:head# 2024-04-23T16:33:55.901 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.901 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:559847e6:::benchmark_data_smithi188_113957_object153492:head# 2024-04-23T16:33:55.901 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.901 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5598631a:::benchmark_data_smithi188_113957_object61132:head# 2024-04-23T16:33:55.901 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.901 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5598fde9:::benchmark_data_smithi188_113957_object95136:head# 2024-04-23T16:33:55.901 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.901 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:559a6b15:::benchmark_data_smithi188_113957_object9769:head# 2024-04-23T16:33:55.901 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.901 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:559aeff5:::benchmark_data_smithi188_113957_object77967:head# 2024-04-23T16:33:55.901 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:559b0083:::benchmark_data_smithi188_113957_object9211:head# 2024-04-23T16:33:55.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:559c03a9:::benchmark_data_smithi188_113957_object133302:head# 2024-04-23T16:33:55.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:559c80c2:::benchmark_data_smithi188_113957_object114784:head# 2024-04-23T16:33:55.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:559ca98a:::benchmark_data_smithi188_113957_object74885:head# 2024-04-23T16:33:55.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:559cfff7:::benchmark_data_smithi188_113957_object61113:head# 2024-04-23T16:33:55.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:559d233d:::benchmark_data_smithi188_113957_object61335:head# 2024-04-23T16:33:55.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:559d98f9:::benchmark_data_smithi188_113957_object87120:head# 2024-04-23T16:33:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:559ec83e:::benchmark_data_smithi188_113957_object59902:head# 2024-04-23T16:33:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:559ef89c:::benchmark_data_smithi188_113957_object127966:head# 2024-04-23T16:33:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:559f26c3:::benchmark_data_smithi188_113957_object22343:head# 2024-04-23T16:33:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a0203f:::benchmark_data_smithi188_113957_object70407:head# 2024-04-23T16:33:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a05c1d:::benchmark_data_smithi188_113957_object89516:head# 2024-04-23T16:33:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a05dc4:::benchmark_data_smithi188_113957_object118225:head# 2024-04-23T16:33:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a0960b:::benchmark_data_smithi188_113957_object81110:head# 2024-04-23T16:33:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a0abc2:::benchmark_data_smithi188_113957_object105843:head# 2024-04-23T16:33:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a0bc02:::benchmark_data_smithi188_113957_object96257:head# 2024-04-23T16:33:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a15c85:::benchmark_data_smithi188_113957_object131827:head# 2024-04-23T16:33:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a1b715:::benchmark_data_smithi188_113957_object53855:head# 2024-04-23T16:33:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a21a1f:::benchmark_data_smithi188_113957_object135907:head# 2024-04-23T16:33:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a22b02:::benchmark_data_smithi188_113957_object103132:head# 2024-04-23T16:33:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a231b0:::benchmark_data_smithi188_113957_object52499:head# 2024-04-23T16:33:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a2442e:::benchmark_data_smithi188_113957_object100712:head# 2024-04-23T16:33:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a2594b:::benchmark_data_smithi188_113957_object88742:head# 2024-04-23T16:33:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a28726:::benchmark_data_smithi188_113957_object61840:head# 2024-04-23T16:33:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a30164:::benchmark_data_smithi188_113957_object38214:head# 2024-04-23T16:33:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a42b8f:::benchmark_data_smithi188_113957_object143543:head# 2024-04-23T16:33:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a4b833:::benchmark_data_smithi188_113957_object97428:head# 2024-04-23T16:33:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a4de2d:::benchmark_data_smithi188_113957_object43072:head# 2024-04-23T16:33:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a57f6d:::benchmark_data_smithi188_113957_object13434:head# 2024-04-23T16:33:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a5867e:::benchmark_data_smithi188_113957_object125714:head# 2024-04-23T16:33:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a59836:::benchmark_data_smithi188_113957_object82094:head# 2024-04-23T16:33:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a63cdc:::benchmark_data_smithi188_113957_object129374:head# 2024-04-23T16:33:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a68900:::benchmark_data_smithi188_113957_object15432:head# 2024-04-23T16:33:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a6c3cb:::benchmark_data_smithi188_113957_object91026:head# 2024-04-23T16:33:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a6e654:::benchmark_data_smithi188_113957_object26211:head# 2024-04-23T16:33:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a769fd:::benchmark_data_smithi188_113957_object132028:head# 2024-04-23T16:33:55.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a77bca:::benchmark_data_smithi188_113957_object19099:head# 2024-04-23T16:33:55.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a79d93:::benchmark_data_smithi188_113957_object132187:head# 2024-04-23T16:33:55.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a80d03:::benchmark_data_smithi188_113957_object66081:head# 2024-04-23T16:33:55.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a845ef:::benchmark_data_smithi188_113957_object102748:head# 2024-04-23T16:33:55.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a9695d:::benchmark_data_smithi188_113957_object121588:head# 2024-04-23T16:33:55.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.909 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a99739:::benchmark_data_smithi188_113957_object148035:head# 2024-04-23T16:33:55.909 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55a9995f:::benchmark_data_smithi188_113957_object122083:head# 2024-04-23T16:33:55.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55aabb10:::benchmark_data_smithi188_113957_object34064:head# 2024-04-23T16:33:55.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55ab2598:::benchmark_data_smithi188_113957_object55715:head# 2024-04-23T16:33:55.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55abc2ed:::benchmark_data_smithi188_113957_object79355:head# 2024-04-23T16:33:55.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55aca8fc:::benchmark_data_smithi188_113957_object50063:head# 2024-04-23T16:33:55.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55ad1f61:::benchmark_data_smithi188_113957_object61563:head# 2024-04-23T16:33:55.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55ad3dce:::benchmark_data_smithi188_113957_object19494:head# 2024-04-23T16:33:55.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55ad6e2f:::benchmark_data_smithi188_113957_object57947:head# 2024-04-23T16:33:55.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55ada704:::benchmark_data_smithi188_113957_object9275:head# 2024-04-23T16:33:55.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55ae68d1:::benchmark_data_smithi188_113957_object22992:head# 2024-04-23T16:33:55.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55ae99bf:::benchmark_data_smithi188_113957_object21072:head# 2024-04-23T16:33:55.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55aec01a:::benchmark_data_smithi188_113957_object621:head# 2024-04-23T16:33:55.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55aee280:::benchmark_data_smithi188_113957_object124031:head# 2024-04-23T16:33:55.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b006d6:::benchmark_data_smithi188_113957_object68480:head# 2024-04-23T16:33:55.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b0e127:::benchmark_data_smithi188_113957_object1082:head# 2024-04-23T16:33:55.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b0e301:::benchmark_data_smithi188_113957_object135111:head# 2024-04-23T16:33:55.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b17406:::benchmark_data_smithi188_113957_object64394:head# 2024-04-23T16:33:55.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b19abb:::benchmark_data_smithi188_113957_object8912:head# 2024-04-23T16:33:55.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b1e538:::benchmark_data_smithi188_113957_object4117:head# 2024-04-23T16:33:55.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b20615:::benchmark_data_smithi188_113957_object82804:head# 2024-04-23T16:33:55.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b206b5:::benchmark_data_smithi188_113957_object15914:head# 2024-04-23T16:33:55.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b2275d:::benchmark_data_smithi188_113957_object148374:head# 2024-04-23T16:33:55.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b2c57c:::benchmark_data_smithi188_113957_object146457:head# 2024-04-23T16:33:55.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b2ca42:::benchmark_data_smithi188_113957_object124717:head# 2024-04-23T16:33:55.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b34495:::benchmark_data_smithi188_113957_object82114:head# 2024-04-23T16:33:55.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b4131e:::benchmark_data_smithi188_113957_object86193:head# 2024-04-23T16:33:55.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b4544b:::benchmark_data_smithi188_113957_object150012:head# 2024-04-23T16:33:55.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b493fb:::benchmark_data_smithi188_113957_object37492:head# 2024-04-23T16:33:55.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.914 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b4d9fa:::benchmark_data_smithi188_113957_object33234:head# 2024-04-23T16:33:55.914 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.914 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b62bf7:::benchmark_data_smithi188_113957_object115367:head# 2024-04-23T16:33:55.914 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.914 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b63650:::benchmark_data_smithi188_113957_object63348:head# 2024-04-23T16:33:55.914 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.914 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b67d3a:::benchmark_data_smithi188_113957_object146961:head# 2024-04-23T16:33:55.914 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.914 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b6c2cc:::benchmark_data_smithi188_113957_object97921:head# 2024-04-23T16:33:55.914 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.914 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b7353e:::benchmark_data_smithi188_113957_object127204:head# 2024-04-23T16:33:55.914 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.914 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b737a3:::benchmark_data_smithi188_113957_object13423:head# 2024-04-23T16:33:55.915 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.915 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b7dd32:::benchmark_data_smithi188_113957_object13332:head# 2024-04-23T16:33:55.915 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.915 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b7e060:::benchmark_data_smithi188_113957_object153212:head# 2024-04-23T16:33:55.915 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.915 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b7f894:::benchmark_data_smithi188_113957_object11913:head# 2024-04-23T16:33:55.915 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.915 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b8a806:::benchmark_data_smithi188_113957_object24457:head# 2024-04-23T16:33:55.915 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.915 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b953f9:::benchmark_data_smithi188_113957_object109662:head# 2024-04-23T16:33:55.915 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.915 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b9a209:::benchmark_data_smithi188_113957_object68629:head# 2024-04-23T16:33:55.915 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.915 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b9bb51:::benchmark_data_smithi188_113957_object94575:head# 2024-04-23T16:33:55.916 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.916 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b9bd9d:::benchmark_data_smithi188_113957_object85371:head# 2024-04-23T16:33:55.916 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.916 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b9e2e0:::benchmark_data_smithi188_113957_object138391:head# 2024-04-23T16:33:55.916 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.916 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b9ea20:::benchmark_data_smithi188_113957_object46481:head# 2024-04-23T16:33:55.916 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.916 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55b9ebc0:::benchmark_data_smithi188_113957_object141941:head# 2024-04-23T16:33:55.916 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.916 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55baeaa5:::benchmark_data_smithi188_113957_object18246:head# 2024-04-23T16:33:55.916 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.916 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55bb3633:::benchmark_data_smithi188_113957_object126810:head# 2024-04-23T16:33:55.916 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.916 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55bc28c9:::benchmark_data_smithi188_113957_object5155:head# 2024-04-23T16:33:55.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55bc6882:::benchmark_data_smithi188_113957_object57746:head# 2024-04-23T16:33:55.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55bc9ce0:::benchmark_data_smithi188_113957_object121980:head# 2024-04-23T16:33:55.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55bcda90:::benchmark_data_smithi188_113957_object16366:head# 2024-04-23T16:33:55.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55bcfa1e:::benchmark_data_smithi188_113957_object16747:head# 2024-04-23T16:33:55.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55bdbe96:::benchmark_data_smithi188_113957_object57321:head# 2024-04-23T16:33:55.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55be10b4:::benchmark_data_smithi188_113957_object103633:head# 2024-04-23T16:33:55.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55be153c:::benchmark_data_smithi188_113957_object146832:head# 2024-04-23T16:33:55.918 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.918 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55be4872:::benchmark_data_smithi188_113957_object9953:head# 2024-04-23T16:33:55.918 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.918 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55be723e:::benchmark_data_smithi188_113957_object46387:head# 2024-04-23T16:33:55.918 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.918 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55be9037:::benchmark_data_smithi188_113957_object23217:head# 2024-04-23T16:33:55.918 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.918 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55befb1d:::benchmark_data_smithi188_113957_object2577:head# 2024-04-23T16:33:55.918 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.918 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55bf82a2:::benchmark_data_smithi188_113957_object109521:head# 2024-04-23T16:33:55.918 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.918 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c106b2:::benchmark_data_smithi188_113957_object108011:head# 2024-04-23T16:33:55.918 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.918 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c163b8:::benchmark_data_smithi188_113957_object24701:head# 2024-04-23T16:33:55.919 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.919 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c22299:::benchmark_data_smithi188_113957_object18876:head# 2024-04-23T16:33:55.919 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.919 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c2763e:::benchmark_data_smithi188_113957_object57765:head# 2024-04-23T16:33:55.919 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.919 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c2afc5:::benchmark_data_smithi188_113957_object119928:head# 2024-04-23T16:33:55.919 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.919 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c2b4a8:::benchmark_data_smithi188_113957_object124053:head# 2024-04-23T16:33:55.919 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.919 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c3663e:::benchmark_data_smithi188_113957_object149744:head# 2024-04-23T16:33:55.919 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.919 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c4292e:::benchmark_data_smithi188_113957_object26749:head# 2024-04-23T16:33:55.919 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.920 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c522d0:::benchmark_data_smithi188_113957_object149314:head# 2024-04-23T16:33:55.920 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.920 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c5347d:::benchmark_data_smithi188_113957_object141538:head# 2024-04-23T16:33:55.920 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.920 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c65a31:::benchmark_data_smithi188_113957_object133433:head# 2024-04-23T16:33:55.920 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.920 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c6ca9d:::benchmark_data_smithi188_113957_object113185:head# 2024-04-23T16:33:55.920 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.920 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c6d5a5:::benchmark_data_smithi188_113957_object112012:head# 2024-04-23T16:33:55.920 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.920 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c73292:::benchmark_data_smithi188_113957_object121883:head# 2024-04-23T16:33:55.920 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.920 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c7490a:::benchmark_data_smithi188_113957_object33229:head# 2024-04-23T16:33:55.920 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.921 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c7b792:::benchmark_data_smithi188_113957_object60575:head# 2024-04-23T16:33:55.921 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.921 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c84c78:::benchmark_data_smithi188_113957_object95325:head# 2024-04-23T16:33:55.921 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.921 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c86916:::benchmark_data_smithi188_113957_object72273:head# 2024-04-23T16:33:55.921 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.921 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c897df:::benchmark_data_smithi188_113957_object59830:head# 2024-04-23T16:33:55.921 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.921 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c918fb:::benchmark_data_smithi188_113957_object21718:head# 2024-04-23T16:33:55.921 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.921 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c91cf7:::benchmark_data_smithi188_113957_object120313:head# 2024-04-23T16:33:55.921 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.921 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55c97a0b:::benchmark_data_smithi188_113957_object19283:head# 2024-04-23T16:33:55.921 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.921 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55cab02a:::benchmark_data_smithi188_113957_object79609:head# 2024-04-23T16:33:55.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.922 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55cad639:::benchmark_data_smithi188_113957_object2020:head# 2024-04-23T16:33:55.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.922 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55cbc0ff:::benchmark_data_smithi188_113957_object81701:head# 2024-04-23T16:33:55.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.922 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55cbd04d:::benchmark_data_smithi188_113957_object109405:head# 2024-04-23T16:33:55.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.922 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55cc10ac:::benchmark_data_smithi188_113957_object53016:head# 2024-04-23T16:33:55.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.922 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55cc157d:::benchmark_data_smithi188_113957_object14019:head# 2024-04-23T16:33:55.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.922 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55cc1ae2:::benchmark_data_smithi188_113957_object70226:head# 2024-04-23T16:33:55.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55cd27c0:::benchmark_data_smithi188_113957_object54152:head# 2024-04-23T16:33:55.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55cd351c:::benchmark_data_smithi188_113957_object131705:head# 2024-04-23T16:33:55.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55cd3686:::benchmark_data_smithi188_113957_object4027:head# 2024-04-23T16:33:55.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55cd406e:::benchmark_data_smithi188_113957_object79199:head# 2024-04-23T16:33:55.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55cdccf1:::benchmark_data_smithi188_113957_object149207:head# 2024-04-23T16:33:55.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55cdfeda:::benchmark_data_smithi188_113957_object9418:head# 2024-04-23T16:33:55.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55ce4c66:::benchmark_data_smithi188_113957_object78208:head# 2024-04-23T16:33:55.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55ce526a:::benchmark_data_smithi188_113957_object74650:head# 2024-04-23T16:33:55.924 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55ce7c93:::benchmark_data_smithi188_113957_object12838:head# 2024-04-23T16:33:55.924 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55cef9db:::benchmark_data_smithi188_113957_object30491:head# 2024-04-23T16:33:55.924 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55cf46d8:::benchmark_data_smithi188_113957_object126163:head# 2024-04-23T16:33:55.924 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55cf5451:::benchmark_data_smithi188_113957_object135548:head# 2024-04-23T16:33:55.924 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d025cf:::benchmark_data_smithi188_113957_object48803:head# 2024-04-23T16:33:55.924 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d04607:::benchmark_data_smithi188_113957_object143831:head# 2024-04-23T16:33:55.924 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d21110:::benchmark_data_smithi188_113957_object128731:head# 2024-04-23T16:33:55.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d24259:::benchmark_data_smithi188_113957_object57249:head# 2024-04-23T16:33:55.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d25280:::benchmark_data_smithi188_113957_object151879:head# 2024-04-23T16:33:55.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d2f932:::benchmark_data_smithi188_113957_object96355:head# 2024-04-23T16:33:55.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d33497:::benchmark_data_smithi188_113957_object115346:head# 2024-04-23T16:33:55.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d36012:::benchmark_data_smithi188_113957_object90603:head# 2024-04-23T16:33:55.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d362d0:::benchmark_data_smithi188_113957_object90594:head# 2024-04-23T16:33:55.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d3c0a9:::benchmark_data_smithi188_113957_object25378:head# 2024-04-23T16:33:55.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d43111:::benchmark_data_smithi188_113957_object64182:head# 2024-04-23T16:33:55.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d53ad8:::benchmark_data_smithi188_113957_object74508:head# 2024-04-23T16:33:55.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d5618e:::benchmark_data_smithi188_113957_object111161:head# 2024-04-23T16:33:55.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d5dc80:::benchmark_data_smithi188_113957_object83270:head# 2024-04-23T16:33:55.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d6d6f1:::benchmark_data_smithi188_113957_object8386:head# 2024-04-23T16:33:55.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d72810:::benchmark_data_smithi188_113957_object140018:head# 2024-04-23T16:33:55.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d88991:::benchmark_data_smithi188_113957_object138685:head# 2024-04-23T16:33:55.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d9378a:::benchmark_data_smithi188_113957_object34849:head# 2024-04-23T16:33:55.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d98c31:::benchmark_data_smithi188_113957_object42579:head# 2024-04-23T16:33:55.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55d9cf36:::benchmark_data_smithi188_113957_object152537:head# 2024-04-23T16:33:55.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55daee72:::benchmark_data_smithi188_113957_object117172:head# 2024-04-23T16:33:55.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55db5aaa:::benchmark_data_smithi188_113957_object100648:head# 2024-04-23T16:33:55.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55db6984:::benchmark_data_smithi188_113957_object91387:head# 2024-04-23T16:33:55.928 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.928 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55dba7e4:::benchmark_data_smithi188_113957_object92310:head# 2024-04-23T16:33:55.928 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.928 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55dbcd55:::benchmark_data_smithi188_113957_object84042:head# 2024-04-23T16:33:55.928 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.928 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55dbd249:::benchmark_data_smithi188_113957_object129531:head# 2024-04-23T16:33:55.928 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.928 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55dc16c4:::benchmark_data_smithi188_113957_object94053:head# 2024-04-23T16:33:55.928 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.928 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55dc9e78:::benchmark_data_smithi188_113957_object61355:head# 2024-04-23T16:33:55.928 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.928 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55de953b:::benchmark_data_smithi188_113957_object69033:head# 2024-04-23T16:33:55.928 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.928 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55df0938:::benchmark_data_smithi188_113957_object27999:head# 2024-04-23T16:33:55.928 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.929 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e02565:::benchmark_data_smithi188_113957_object38433:head# 2024-04-23T16:33:55.929 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.929 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e124aa:::benchmark_data_smithi188_113957_object112407:head# 2024-04-23T16:33:55.929 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.929 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e1d26d:::benchmark_data_smithi188_113957_object69886:head# 2024-04-23T16:33:55.929 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.929 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e21130:::benchmark_data_smithi188_113957_object99966:head# 2024-04-23T16:33:55.929 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.929 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e23a1e:::benchmark_data_smithi188_113957_object72177:head# 2024-04-23T16:33:55.929 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.929 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e2776b:::benchmark_data_smithi188_113957_object74566:head# 2024-04-23T16:33:55.929 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.929 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e290f2:::benchmark_data_smithi188_113957_object31581:head# 2024-04-23T16:33:55.929 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e2ebb2:::benchmark_data_smithi188_113957_object128440:head# 2024-04-23T16:33:55.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e2f2e4:::benchmark_data_smithi188_113957_object141699:head# 2024-04-23T16:33:55.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e30b32:::benchmark_data_smithi188_113957_object97456:head# 2024-04-23T16:33:55.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e3136f:::benchmark_data_smithi188_113957_object131138:head# 2024-04-23T16:33:55.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e3784e:::benchmark_data_smithi188_113957_object13431:head# 2024-04-23T16:33:55.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e416e7:::benchmark_data_smithi188_113957_object123641:head# 2024-04-23T16:33:55.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e42442:::benchmark_data_smithi188_113957_object90477:head# 2024-04-23T16:33:55.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e45227:::benchmark_data_smithi188_113957_object123091:head# 2024-04-23T16:33:55.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e4b39a:::benchmark_data_smithi188_113957_object56618:head# 2024-04-23T16:33:55.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e4c4ed:::benchmark_data_smithi188_113957_object3842:head# 2024-04-23T16:33:55.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e4ec98:::benchmark_data_smithi188_113957_object29310:head# 2024-04-23T16:33:55.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e65137:::benchmark_data_smithi188_113957_object568:head# 2024-04-23T16:33:55.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e74d09:::benchmark_data_smithi188_113957_object29791:head# 2024-04-23T16:33:55.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e7c116:::benchmark_data_smithi188_113957_object22386:head# 2024-04-23T16:33:55.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e84c2f:::benchmark_data_smithi188_113957_object66796:head# 2024-04-23T16:33:55.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e86bee:::benchmark_data_smithi188_113957_object124602:head# 2024-04-23T16:33:55.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e88bfa:::benchmark_data_smithi188_113957_object121643:head# 2024-04-23T16:33:55.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e8a4c0:::benchmark_data_smithi188_113957_object133940:head# 2024-04-23T16:33:55.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e9388e:::benchmark_data_smithi188_113957_object51097:head# 2024-04-23T16:33:55.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e989e9:::benchmark_data_smithi188_113957_object128022:head# 2024-04-23T16:33:55.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e9ea8c:::benchmark_data_smithi188_113957_object50243:head# 2024-04-23T16:33:55.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55e9f07a:::benchmark_data_smithi188_113957_object51946:head# 2024-04-23T16:33:55.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55eacc0d:::benchmark_data_smithi188_113957_object69984:head# 2024-04-23T16:33:55.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55ec3820:::benchmark_data_smithi188_113957_object141796:head# 2024-04-23T16:33:55.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55ec4faf:::benchmark_data_smithi188_113957_object105821:head# 2024-04-23T16:33:55.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55ed3193:::benchmark_data_smithi188_113957_object143454:head# 2024-04-23T16:33:55.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55edb98a:::benchmark_data_smithi188_113957_object41603:head# 2024-04-23T16:33:55.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55ee2a05:::benchmark_data_smithi188_113957_object86157:head# 2024-04-23T16:33:55.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55ee4ed9:::benchmark_data_smithi188_113957_object118322:head# 2024-04-23T16:33:55.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55ef2902:::benchmark_data_smithi188_113957_object74070:head# 2024-04-23T16:33:55.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55ef8b74:::benchmark_data_smithi188_113957_object84333:head# 2024-04-23T16:33:55.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55efe33f:::benchmark_data_smithi188_113957_object72314:head# 2024-04-23T16:33:55.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55f03593:::benchmark_data_smithi188_113957_object21832:head# 2024-04-23T16:33:55.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55f0e710:::benchmark_data_smithi188_113957_object109209:head# 2024-04-23T16:33:55.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55f1c131:::benchmark_data_smithi188_113957_object22567:head# 2024-04-23T16:33:55.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55f1e1f0:::benchmark_data_smithi188_113957_object23728:head# 2024-04-23T16:33:55.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55f1fa27:::benchmark_data_smithi188_113957_object86755:head# 2024-04-23T16:33:55.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55f23b1f:::benchmark_data_smithi188_113957_object13448:head# 2024-04-23T16:33:55.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55f2984d:::benchmark_data_smithi188_113957_object33706:head# 2024-04-23T16:33:55.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55f38810:::benchmark_data_smithi188_113957_object131254:head# 2024-04-23T16:33:55.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55f3cced:::benchmark_data_smithi188_113957_object96559:head# 2024-04-23T16:33:55.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55f50ebb:::benchmark_data_smithi188_113957_object91171:head# 2024-04-23T16:33:55.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55f60c32:::benchmark_data_smithi188_113957_object144461:head# 2024-04-23T16:33:55.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55f67bac:::benchmark_data_smithi188_113957_object148291:head# 2024-04-23T16:33:55.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55f6d7cb:::benchmark_data_smithi188_113957_object132338:head# 2024-04-23T16:33:55.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55f7bb75:::benchmark_data_smithi188_113957_object27269:head# 2024-04-23T16:33:55.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55f84803:::benchmark_data_smithi188_113957_object26887:head# 2024-04-23T16:33:55.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55f85e79:::benchmark_data_smithi188_113957_object90820:head# 2024-04-23T16:33:55.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55f98632:::benchmark_data_smithi188_113957_object91291:head# 2024-04-23T16:33:55.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55fad332:::benchmark_data_smithi188_113957_object30242:head# 2024-04-23T16:33:55.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55fbb767:::benchmark_data_smithi188_113957_object124192:head# 2024-04-23T16:33:55.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55fbcb11:::benchmark_data_smithi188_113957_object42993:head# 2024-04-23T16:33:55.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55fc789a:::benchmark_data_smithi188_113957_object132587:head# 2024-04-23T16:33:55.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55fc78b7:::benchmark_data_smithi188_113957_object120800:head# 2024-04-23T16:33:55.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55fc7fe9:::benchmark_data_smithi188_113957_object226:head# 2024-04-23T16:33:55.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55fcb3e7:::benchmark_data_smithi188_113957_object142740:head# 2024-04-23T16:33:55.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55fd817e:::benchmark_data_smithi188_113957_object37644:head# 2024-04-23T16:33:55.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55fe0b06:::benchmark_data_smithi188_113957_object139566:head# 2024-04-23T16:33:55.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55fe4ed1:::benchmark_data_smithi188_113957_object48593:head# 2024-04-23T16:33:55.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55fe5230:::benchmark_data_smithi188_113957_object30932:head# 2024-04-23T16:33:55.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55fe5c44:::benchmark_data_smithi188_113957_object103760:head# 2024-04-23T16:33:55.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:55fe86fe:::benchmark_data_smithi188_113957_object56937:head# 2024-04-23T16:33:55.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56007a0f:::benchmark_data_smithi188_113957_object116:head# 2024-04-23T16:33:55.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5600c46d:::benchmark_data_smithi188_113957_object119366:head# 2024-04-23T16:33:55.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56010da3:::benchmark_data_smithi188_113957_object728:head# 2024-04-23T16:33:55.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5601225f:::benchmark_data_smithi188_113957_object22851:head# 2024-04-23T16:33:55.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5602497d:::benchmark_data_smithi188_113957_object111207:head# 2024-04-23T16:33:55.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5603c4f0:::benchmark_data_smithi188_113957_object50570:head# 2024-04-23T16:33:55.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56040f6f:::benchmark_data_smithi188_113957_object3972:head# 2024-04-23T16:33:55.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5605eda7:::benchmark_data_smithi188_113957_object110223:head# 2024-04-23T16:33:55.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56070803:::benchmark_data_smithi188_113957_object121564:head# 2024-04-23T16:33:55.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5607122a:::benchmark_data_smithi188_113957_object27928:head# 2024-04-23T16:33:55.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56073264:::benchmark_data_smithi188_113957_object3536:head# 2024-04-23T16:33:55.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56075e15:::benchmark_data_smithi188_113957_object125815:head# 2024-04-23T16:33:55.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:560815cb:::benchmark_data_smithi188_113957_object91465:head# 2024-04-23T16:33:55.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:560860e2:::benchmark_data_smithi188_113957_object40097:head# 2024-04-23T16:33:55.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5608bb94:::benchmark_data_smithi188_113957_object45236:head# 2024-04-23T16:33:55.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5608f7f2:::benchmark_data_smithi188_113957_object43359:head# 2024-04-23T16:33:55.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5609ba1a:::benchmark_data_smithi188_113957_object46480:head# 2024-04-23T16:33:55.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5609ed32:::benchmark_data_smithi188_113957_object7746:head# 2024-04-23T16:33:55.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:560a55a9:::benchmark_data_smithi188_113957_object100193:head# 2024-04-23T16:33:55.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:560b2ee9:::benchmark_data_smithi188_113957_object23567:head# 2024-04-23T16:33:55.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:560c9407:::benchmark_data_smithi188_113957_object116848:head# 2024-04-23T16:33:55.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:560ce526:::benchmark_data_smithi188_113957_object111472:head# 2024-04-23T16:33:55.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:560d8c7d:::benchmark_data_smithi188_113957_object39880:head# 2024-04-23T16:33:55.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:560dc1e6:::benchmark_data_smithi188_113957_object25656:head# 2024-04-23T16:33:55.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:560dc54d:::benchmark_data_smithi188_113957_object14457:head# 2024-04-23T16:33:55.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:560e3a41:::benchmark_data_smithi188_113957_object53308:head# 2024-04-23T16:33:55.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:560e7cb8:::benchmark_data_smithi188_113957_object45117:head# 2024-04-23T16:33:55.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:560e7d96:::benchmark_data_smithi188_113957_object49444:head# 2024-04-23T16:33:55.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:560fe1ce:::benchmark_data_smithi188_113957_object88233:head# 2024-04-23T16:33:55.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56119de1:::benchmark_data_smithi188_113957_object66808:head# 2024-04-23T16:33:55.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5611ff46:::benchmark_data_smithi188_113957_object87008:head# 2024-04-23T16:33:55.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:561212ef:::benchmark_data_smithi188_113957_object112648:head# 2024-04-23T16:33:55.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5612d56d:::benchmark_data_smithi188_113957_object63608:head# 2024-04-23T16:33:55.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56130bf1:::benchmark_data_smithi188_113957_object71325:head# 2024-04-23T16:33:55.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56136f60:::benchmark_data_smithi188_113957_object19270:head# 2024-04-23T16:33:55.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56138a59:::benchmark_data_smithi188_113957_object90784:head# 2024-04-23T16:33:55.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5613d951:::benchmark_data_smithi188_113957_object108653:head# 2024-04-23T16:33:55.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:561464c7:::benchmark_data_smithi188_113957_object23502:head# 2024-04-23T16:33:55.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56154975:::benchmark_data_smithi188_113957_object136241:head# 2024-04-23T16:33:55.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5615d7c6:::benchmark_data_smithi188_113957_object122356:head# 2024-04-23T16:33:55.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5615e949:::benchmark_data_smithi188_113957_object32098:head# 2024-04-23T16:33:55.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5616b709:::benchmark_data_smithi188_113957_object148385:head# 2024-04-23T16:33:55.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56178dd2:::benchmark_data_smithi188_113957_object43820:head# 2024-04-23T16:33:55.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:561795e3:::benchmark_data_smithi188_113957_object118326:head# 2024-04-23T16:33:55.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56181559:::benchmark_data_smithi188_113957_object9458:head# 2024-04-23T16:33:55.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5618ad7e:::benchmark_data_smithi188_113957_object3606:head# 2024-04-23T16:33:55.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5618cfc5:::benchmark_data_smithi188_113957_object54236:head# 2024-04-23T16:33:55.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:561921bd:::benchmark_data_smithi188_113957_object57365:head# 2024-04-23T16:33:55.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5619c115:::benchmark_data_smithi188_113957_object61055:head# 2024-04-23T16:33:55.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:561b465c:::benchmark_data_smithi188_113957_object96449:head# 2024-04-23T16:33:55.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:561b5b3a:::benchmark_data_smithi188_113957_object15365:head# 2024-04-23T16:33:55.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:561bf969:::benchmark_data_smithi188_113957_object123408:head# 2024-04-23T16:33:55.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:561bfd0c:::benchmark_data_smithi188_113957_object56879:head# 2024-04-23T16:33:55.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:561cc869:::benchmark_data_smithi188_113957_object62476:head# 2024-04-23T16:33:55.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:561d9bbc:::benchmark_data_smithi188_113957_object40708:head# 2024-04-23T16:33:55.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:561dd015:::benchmark_data_smithi188_113957_object105215:head# 2024-04-23T16:33:55.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:561ddaeb:::benchmark_data_smithi188_113957_object30285:head# 2024-04-23T16:33:55.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:561e2b3d:::benchmark_data_smithi188_113957_object92749:head# 2024-04-23T16:33:55.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:561e8df9:::benchmark_data_smithi188_113957_object52891:head# 2024-04-23T16:33:55.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:561ed0f5:::benchmark_data_smithi188_113957_object71134:head# 2024-04-23T16:33:55.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:561edccd:::benchmark_data_smithi188_113957_object115764:head# 2024-04-23T16:33:55.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:561f020d:::benchmark_data_smithi188_113957_object137218:head# 2024-04-23T16:33:55.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:561f405a:::benchmark_data_smithi188_113957_object4270:head# 2024-04-23T16:33:55.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:561fd1bf:::benchmark_data_smithi188_113957_object68854:head# 2024-04-23T16:33:55.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56211d99:::benchmark_data_smithi188_113957_object42444:head# 2024-04-23T16:33:55.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56223f99:::benchmark_data_smithi188_113957_object28187:head# 2024-04-23T16:33:55.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5622e4b1:::benchmark_data_smithi188_113957_object93522:head# 2024-04-23T16:33:55.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5622e500:::benchmark_data_smithi188_113957_object124701:head# 2024-04-23T16:33:55.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5622ff3f:::benchmark_data_smithi188_113957_object129976:head# 2024-04-23T16:33:55.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56235d9d:::benchmark_data_smithi188_113957_object78888:head# 2024-04-23T16:33:55.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5623bfdd:::benchmark_data_smithi188_113957_object35045:head# 2024-04-23T16:33:55.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56240ccb:::benchmark_data_smithi188_113957_object122964:head# 2024-04-23T16:33:55.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56242b6f:::benchmark_data_smithi188_113957_object81648:head# 2024-04-23T16:33:55.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56242d46:::benchmark_data_smithi188_113957_object31820:head# 2024-04-23T16:33:55.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5624955f:::benchmark_data_smithi188_113957_object105853:head# 2024-04-23T16:33:55.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5624c907:::benchmark_data_smithi188_113957_object4347:head# 2024-04-23T16:33:55.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5624d661:::benchmark_data_smithi188_113957_object118846:head# 2024-04-23T16:33:55.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56251457:::benchmark_data_smithi188_113957_object72296:head# 2024-04-23T16:33:55.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56254e65:::benchmark_data_smithi188_113957_object116467:head# 2024-04-23T16:33:55.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56255051:::benchmark_data_smithi188_113957_object111435:head# 2024-04-23T16:33:55.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56258d75:::benchmark_data_smithi188_113957_object62051:head# 2024-04-23T16:33:55.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:562599f8:::benchmark_data_smithi188_113957_object84847:head# 2024-04-23T16:33:55.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5625bc31:::benchmark_data_smithi188_113957_object40811:head# 2024-04-23T16:33:55.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56263799:::benchmark_data_smithi188_113957_object68612:head# 2024-04-23T16:33:55.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:562673cb:::benchmark_data_smithi188_113957_object140100:head# 2024-04-23T16:33:55.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5626dc62:::benchmark_data_smithi188_113957_object85528:head# 2024-04-23T16:33:55.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56274713:::benchmark_data_smithi188_113957_object81449:head# 2024-04-23T16:33:55.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5627a2da:::benchmark_data_smithi188_113957_object80604:head# 2024-04-23T16:33:55.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5628dc4e:::benchmark_data_smithi188_113957_object57304:head# 2024-04-23T16:33:55.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:562961ee:::benchmark_data_smithi188_113957_object73890:head# 2024-04-23T16:33:55.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5629e79a:::benchmark_data_smithi188_113957_object14459:head# 2024-04-23T16:33:55.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:562a1a56:::benchmark_data_smithi188_113957_object101915:head# 2024-04-23T16:33:55.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:562a5f60:::benchmark_data_smithi188_113957_object148714:head# 2024-04-23T16:33:55.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:562accad:::benchmark_data_smithi188_113957_object90888:head# 2024-04-23T16:33:55.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:562bba7c:::benchmark_data_smithi188_113957_object83:head# 2024-04-23T16:33:55.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:562c69d0:::benchmark_data_smithi188_113957_object71501:head# 2024-04-23T16:33:55.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:562d4b5e:::benchmark_data_smithi188_113957_object4650:head# 2024-04-23T16:33:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:562d634d:::benchmark_data_smithi188_113957_object81362:head# 2024-04-23T16:33:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:562e6d12:::benchmark_data_smithi188_113957_object92584:head# 2024-04-23T16:33:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:562eed03:::benchmark_data_smithi188_113957_object38847:head# 2024-04-23T16:33:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:562f4a50:::benchmark_data_smithi188_113957_object55129:head# 2024-04-23T16:33:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:562f4cd0:::benchmark_data_smithi188_113957_object55548:head# 2024-04-23T16:33:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:563008cb:::benchmark_data_smithi188_113957_object145146:head# 2024-04-23T16:33:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5630174e:::benchmark_data_smithi188_113957_object150232:head# 2024-04-23T16:33:55.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56302dd9:::benchmark_data_smithi188_113957_object105403:head# 2024-04-23T16:33:55.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56307301:::benchmark_data_smithi188_113957_object119230:head# 2024-04-23T16:33:55.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5630b99e:::benchmark_data_smithi188_113957_object123326:head# 2024-04-23T16:33:55.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5631138c:::benchmark_data_smithi188_113957_object69595:head# 2024-04-23T16:33:55.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:563197f2:::benchmark_data_smithi188_113957_object61414:head# 2024-04-23T16:33:55.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5631b9a9:::benchmark_data_smithi188_113957_object22862:head# 2024-04-23T16:33:55.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5632b01d:::benchmark_data_smithi188_113957_object78372:head# 2024-04-23T16:33:55.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5632e3b8:::benchmark_data_smithi188_113957_object137023:head# 2024-04-23T16:33:55.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56333091:::benchmark_data_smithi188_113957_object26082:head# 2024-04-23T16:33:55.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:563369cd:::benchmark_data_smithi188_113957_object89216:head# 2024-04-23T16:33:55.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56345351:::benchmark_data_smithi188_113957_object46659:head# 2024-04-23T16:33:55.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56348ec9:::benchmark_data_smithi188_113957_object119249:head# 2024-04-23T16:33:55.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56356ba1:::benchmark_data_smithi188_113957_object31922:head# 2024-04-23T16:33:55.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5635afa8:::benchmark_data_smithi188_113957_object143965:head# 2024-04-23T16:33:55.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56361e2d:::benchmark_data_smithi188_113957_object27209:head# 2024-04-23T16:33:55.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5636bc96:::benchmark_data_smithi188_113957_object7610:head# 2024-04-23T16:33:55.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5637083c:::benchmark_data_smithi188_113957_object145061:head# 2024-04-23T16:33:55.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56373e49:::benchmark_data_smithi188_113957_object91003:head# 2024-04-23T16:33:55.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5637baac:::benchmark_data_smithi188_113957_object115510:head# 2024-04-23T16:33:55.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:563812b5:::benchmark_data_smithi188_113957_object77668:head# 2024-04-23T16:33:55.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56386637:::benchmark_data_smithi188_113957_object57492:head# 2024-04-23T16:33:55.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:563a6d3e:::benchmark_data_smithi188_113957_object55783:head# 2024-04-23T16:33:55.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:563aac39:::benchmark_data_smithi188_113957_object29929:head# 2024-04-23T16:33:55.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:563b27b1:::benchmark_data_smithi188_113957_object123677:head# 2024-04-23T16:33:55.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:563b50d7:::benchmark_data_smithi188_113957_object86353:head# 2024-04-23T16:33:55.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:563b998f:::benchmark_data_smithi188_113957_object18392:head# 2024-04-23T16:33:55.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:563ca06d:::benchmark_data_smithi188_113957_object82389:head# 2024-04-23T16:33:55.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:563cea74:::benchmark_data_smithi188_113957_object67165:head# 2024-04-23T16:33:55.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:563cf643:::benchmark_data_smithi188_113957_object144112:head# 2024-04-23T16:33:55.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:563dd3aa:::benchmark_data_smithi188_113957_object62201:head# 2024-04-23T16:33:55.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:563e6321:::benchmark_data_smithi188_113957_object109432:head# 2024-04-23T16:33:55.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:563e90e0:::benchmark_data_smithi188_113957_object1221:head# 2024-04-23T16:33:55.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:563f1deb:::benchmark_data_smithi188_113957_object119852:head# 2024-04-23T16:33:55.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:563fd536:::benchmark_data_smithi188_113957_object148436:head# 2024-04-23T16:33:55.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:563ff608:::benchmark_data_smithi188_113957_object18841:head# 2024-04-23T16:33:55.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56429cf4:::benchmark_data_smithi188_113957_object118864:head# 2024-04-23T16:33:55.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56434432:::benchmark_data_smithi188_113957_object14936:head# 2024-04-23T16:33:55.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5643f662:::benchmark_data_smithi188_113957_object118477:head# 2024-04-23T16:33:55.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5644b7af:::benchmark_data_smithi188_113957_object76387:head# 2024-04-23T16:33:55.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56454df9:::benchmark_data_smithi188_113957_object22384:head# 2024-04-23T16:33:55.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5645cebb:::benchmark_data_smithi188_113957_object15013:head# 2024-04-23T16:33:55.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56466b99:::benchmark_data_smithi188_113957_object143155:head# 2024-04-23T16:33:55.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56469ba1:::benchmark_data_smithi188_113957_object36236:head# 2024-04-23T16:33:55.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5646b774:::benchmark_data_smithi188_113957_object95392:head# 2024-04-23T16:33:55.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5646efb9:::benchmark_data_smithi188_113957_object96290:head# 2024-04-23T16:33:55.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5647ecb9:::benchmark_data_smithi188_113957_object16331:head# 2024-04-23T16:33:55.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5648888f:::benchmark_data_smithi188_113957_object141235:head# 2024-04-23T16:33:55.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5648c2ad:::benchmark_data_smithi188_113957_object146657:head# 2024-04-23T16:33:55.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5648d5f8:::benchmark_data_smithi188_113957_object139425:head# 2024-04-23T16:33:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:564a4e83:::benchmark_data_smithi188_113957_object57892:head# 2024-04-23T16:33:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:564a9174:::benchmark_data_smithi188_113957_object105947:head# 2024-04-23T16:33:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:564ae3bf:::benchmark_data_smithi188_113957_object30744:head# 2024-04-23T16:33:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:564b9d1c:::benchmark_data_smithi188_113957_object111200:head# 2024-04-23T16:33:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:564b9e2f:::benchmark_data_smithi188_113957_object134840:head# 2024-04-23T16:33:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:564bb485:::benchmark_data_smithi188_113957_object26219:head# 2024-04-23T16:33:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:564ca2b9:::benchmark_data_smithi188_113957_object152940:head# 2024-04-23T16:33:55.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.962 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:564cb418:::benchmark_data_smithi188_113957_object75925:head# 2024-04-23T16:33:55.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.962 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:564d311f:::benchmark_data_smithi188_113957_object19364:head# 2024-04-23T16:33:55.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.962 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:564d75dc:::benchmark_data_smithi188_113957_object85190:head# 2024-04-23T16:33:55.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.962 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:564d83de:::benchmark_data_smithi188_113957_object89058:head# 2024-04-23T16:33:55.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.962 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:564de16c:::benchmark_data_smithi188_113957_object109839:head# 2024-04-23T16:33:55.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.962 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:564eb423:::benchmark_data_smithi188_113957_object86910:head# 2024-04-23T16:33:55.962 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.962 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:564ed4b8:::benchmark_data_smithi188_113957_object150698:head# 2024-04-23T16:33:55.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:564f05bd:::benchmark_data_smithi188_113957_object14956:head# 2024-04-23T16:33:55.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:564f158c:::benchmark_data_smithi188_113957_object22999:head# 2024-04-23T16:33:55.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:565008ff:::benchmark_data_smithi188_113957_object97193:head# 2024-04-23T16:33:55.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5650151b:::benchmark_data_smithi188_113957_object97126:head# 2024-04-23T16:33:55.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56509385:::benchmark_data_smithi188_113957_object64391:head# 2024-04-23T16:33:55.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56520254:::benchmark_data_smithi188_113957_object129012:head# 2024-04-23T16:33:55.963 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.963 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56524a9c:::benchmark_data_smithi188_113957_object141535:head# 2024-04-23T16:33:55.964 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.964 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56533af0:::benchmark_data_smithi188_113957_object57236:head# 2024-04-23T16:33:55.964 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.964 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:565385eb:::benchmark_data_smithi188_113957_object80729:head# 2024-04-23T16:33:55.964 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.964 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5653a980:::benchmark_data_smithi188_113957_object80720:head# 2024-04-23T16:33:55.964 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.964 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56542960:::benchmark_data_smithi188_113957_object7798:head# 2024-04-23T16:33:55.964 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.964 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56551dc2:::benchmark_data_smithi188_113957_object152385:head# 2024-04-23T16:33:55.964 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.964 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5655cb37:::benchmark_data_smithi188_113957_object70064:head# 2024-04-23T16:33:55.964 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.964 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56567d81:::benchmark_data_smithi188_113957_object32047:head# 2024-04-23T16:33:55.965 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.965 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5656951c:::benchmark_data_smithi188_113957_object93105:head# 2024-04-23T16:33:55.965 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.965 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5656b429:::benchmark_data_smithi188_113957_object12079:head# 2024-04-23T16:33:55.965 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.965 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:565708a0:::benchmark_data_smithi188_113957_object148401:head# 2024-04-23T16:33:55.965 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.965 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56570c03:::benchmark_data_smithi188_113957_object55603:head# 2024-04-23T16:33:55.965 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.965 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5657148a:::benchmark_data_smithi188_113957_object76254:head# 2024-04-23T16:33:55.965 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.965 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56577efe:::benchmark_data_smithi188_113957_object53041:head# 2024-04-23T16:33:55.965 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.965 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5657a866:::benchmark_data_smithi188_113957_object144421:head# 2024-04-23T16:33:55.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5657cd1f:::benchmark_data_smithi188_113957_object138664:head# 2024-04-23T16:33:55.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56580872:::benchmark_data_smithi188_113957_object1192:head# 2024-04-23T16:33:55.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56584ee5:::benchmark_data_smithi188_113957_object143661:head# 2024-04-23T16:33:55.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5658a9c3:::benchmark_data_smithi188_113957_object121575:head# 2024-04-23T16:33:55.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5658f855:::benchmark_data_smithi188_113957_object56271:head# 2024-04-23T16:33:55.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56593ac5:::benchmark_data_smithi188_113957_object50203:head# 2024-04-23T16:33:55.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5659814d:::benchmark_data_smithi188_113957_object102953:head# 2024-04-23T16:33:55.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:565a4a41:::benchmark_data_smithi188_113957_object151422:head# 2024-04-23T16:33:55.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:565a594c:::benchmark_data_smithi188_113957_object46702:head# 2024-04-23T16:33:55.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:565a5bbf:::benchmark_data_smithi188_113957_object65038:head# 2024-04-23T16:33:55.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:565bee12:::benchmark_data_smithi188_113957_object69235:head# 2024-04-23T16:33:55.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:565c56bf:::benchmark_data_smithi188_113957_object122458:head# 2024-04-23T16:33:55.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:565d588a:::benchmark_data_smithi188_113957_object151518:head# 2024-04-23T16:33:55.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:565d5ca4:::benchmark_data_smithi188_113957_object74791:head# 2024-04-23T16:33:55.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:565dbea7:::benchmark_data_smithi188_113957_object5738:head# 2024-04-23T16:33:55.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:565dd54d:::benchmark_data_smithi188_113957_object67679:head# 2024-04-23T16:33:55.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:565e0c78:::benchmark_data_smithi188_113957_object65316:head# 2024-04-23T16:33:55.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:565e11b7:::benchmark_data_smithi188_113957_object79392:head# 2024-04-23T16:33:55.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:565e3ffa:::benchmark_data_smithi188_113957_object127621:head# 2024-04-23T16:33:55.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:565ef85f:::benchmark_data_smithi188_113957_object82679:head# 2024-04-23T16:33:55.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:565f95d5:::benchmark_data_smithi188_113957_object105319:head# 2024-04-23T16:33:55.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:565f9da2:::benchmark_data_smithi188_113957_object77006:head# 2024-04-23T16:33:55.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:565fb883:::benchmark_data_smithi188_113957_object86242:head# 2024-04-23T16:33:55.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:566021a6:::benchmark_data_smithi188_113957_object30575:head# 2024-04-23T16:33:55.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5660740c:::benchmark_data_smithi188_113957_object51672:head# 2024-04-23T16:33:55.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5660c0d4:::benchmark_data_smithi188_113957_object69674:head# 2024-04-23T16:33:55.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56610d28:::benchmark_data_smithi188_113957_object44350:head# 2024-04-23T16:33:55.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:566137b5:::benchmark_data_smithi188_113957_object123117:head# 2024-04-23T16:33:55.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56622d8a:::benchmark_data_smithi188_113957_object60126:head# 2024-04-23T16:33:55.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56625df7:::benchmark_data_smithi188_113957_object134317:head# 2024-04-23T16:33:55.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56635f2b:::benchmark_data_smithi188_113957_object55094:head# 2024-04-23T16:33:55.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5663af1f:::benchmark_data_smithi188_113957_object77050:head# 2024-04-23T16:33:55.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56659ec8:::benchmark_data_smithi188_113957_object109864:head# 2024-04-23T16:33:55.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56662a2d:::benchmark_data_smithi188_113957_object140716:head# 2024-04-23T16:33:55.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56665a61:::benchmark_data_smithi188_113957_object106562:head# 2024-04-23T16:33:55.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5666a724:::benchmark_data_smithi188_113957_object52961:head# 2024-04-23T16:33:55.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56673363:::benchmark_data_smithi188_113957_object144553:head# 2024-04-23T16:33:55.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5667514f:::benchmark_data_smithi188_113957_object143083:head# 2024-04-23T16:33:55.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56677fd0:::benchmark_data_smithi188_113957_object120310:head# 2024-04-23T16:33:55.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:566785e3:::benchmark_data_smithi188_113957_object99716:head# 2024-04-23T16:33:55.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5667dce0:::benchmark_data_smithi188_113957_object84757:head# 2024-04-23T16:33:55.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56684e4a:::benchmark_data_smithi188_113957_object17586:head# 2024-04-23T16:33:55.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56687322:::benchmark_data_smithi188_113957_object110587:head# 2024-04-23T16:33:55.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5668a528:::benchmark_data_smithi188_113957_object110927:head# 2024-04-23T16:33:55.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5668c067:::benchmark_data_smithi188_113957_object47854:head# 2024-04-23T16:33:55.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5669111f:::benchmark_data_smithi188_113957_object3333:head# 2024-04-23T16:33:55.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56691abc:::benchmark_data_smithi188_113957_object118272:head# 2024-04-23T16:33:55.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5669c0b3:::benchmark_data_smithi188_113957_object67592:head# 2024-04-23T16:33:55.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:566a768e:::benchmark_data_smithi188_113957_object85205:head# 2024-04-23T16:33:55.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:566b2cad:::benchmark_data_smithi188_113957_object142227:head# 2024-04-23T16:33:55.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:566b83b6:::benchmark_data_smithi188_113957_object120323:head# 2024-04-23T16:33:55.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:566bb7ff:::benchmark_data_smithi188_113957_object52133:head# 2024-04-23T16:33:55.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:566c72ff:::benchmark_data_smithi188_113957_object117595:head# 2024-04-23T16:33:55.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:566dca15:::benchmark_data_smithi188_113957_object17835:head# 2024-04-23T16:33:55.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:566dcca0:::benchmark_data_smithi188_113957_object134272:head# 2024-04-23T16:33:55.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:566f64fc:::benchmark_data_smithi188_113957_object108153:head# 2024-04-23T16:33:55.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5670520d:::benchmark_data_smithi188_113957_object7400:head# 2024-04-23T16:33:55.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5670ca65:::benchmark_data_smithi188_113957_object55780:head# 2024-04-23T16:33:55.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5670fea8:::benchmark_data_smithi188_113957_object70357:head# 2024-04-23T16:33:55.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56719575:::benchmark_data_smithi188_113957_object18333:head# 2024-04-23T16:33:55.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5671a30b:::benchmark_data_smithi188_113957_object101611:head# 2024-04-23T16:33:55.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5671a5ae:::benchmark_data_smithi188_113957_object44880:head# 2024-04-23T16:33:55.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5671df73:::benchmark_data_smithi188_113957_object38744:head# 2024-04-23T16:33:55.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56736b38:::benchmark_data_smithi188_113957_object20150:head# 2024-04-23T16:33:55.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5673be0e:::benchmark_data_smithi188_113957_object114451:head# 2024-04-23T16:33:55.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5673c108:::benchmark_data_smithi188_113957_object55668:head# 2024-04-23T16:33:55.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5673ef40:::benchmark_data_smithi188_113957_object131469:head# 2024-04-23T16:33:55.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56744bea:::benchmark_data_smithi188_113957_object50234:head# 2024-04-23T16:33:55.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56745e81:::benchmark_data_smithi188_113957_object2163:head# 2024-04-23T16:33:55.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56746743:::benchmark_data_smithi188_113957_object96813:head# 2024-04-23T16:33:55.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:567468cc:::benchmark_data_smithi188_113957_object52498:head# 2024-04-23T16:33:55.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5674bd95:::benchmark_data_smithi188_113957_object33896:head# 2024-04-23T16:33:55.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5675af0a:::benchmark_data_smithi188_113957_object19883:head# 2024-04-23T16:33:55.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56761ac2:::benchmark_data_smithi188_113957_object84020:head# 2024-04-23T16:33:55.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56777f37:::benchmark_data_smithi188_113957_object80083:head# 2024-04-23T16:33:55.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56785465:::benchmark_data_smithi188_113957_object92227:head# 2024-04-23T16:33:55.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56790c94:::benchmark_data_smithi188_113957_object2624:head# 2024-04-23T16:33:55.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5679844e:::benchmark_data_smithi188_113957_object55586:head# 2024-04-23T16:33:55.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56798b87:::benchmark_data_smithi188_113957_object97416:head# 2024-04-23T16:33:55.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:567a68f1:::benchmark_data_smithi188_113957_object35009:head# 2024-04-23T16:33:55.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:567a8e84:::benchmark_data_smithi188_113957_object106357:head# 2024-04-23T16:33:55.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:567a8fd3:::benchmark_data_smithi188_113957_object14076:head# 2024-04-23T16:33:55.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:567b41dc:::benchmark_data_smithi188_113957_object25552:head# 2024-04-23T16:33:55.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:567b5f69:::benchmark_data_smithi188_113957_object40022:head# 2024-04-23T16:33:55.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.978 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:567b622a:::benchmark_data_smithi188_113957_object74966:head# 2024-04-23T16:33:55.978 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.978 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:567b6503:::benchmark_data_smithi188_113957_object28684:head# 2024-04-23T16:33:55.978 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.978 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:567bc9d0:::benchmark_data_smithi188_113957_object115930:head# 2024-04-23T16:33:55.978 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.978 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:567c0308:::benchmark_data_smithi188_113957_object103693:head# 2024-04-23T16:33:55.978 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.978 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:567d8eb5:::benchmark_data_smithi188_113957_object11937:head# 2024-04-23T16:33:55.978 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.978 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:567dcffe:::benchmark_data_smithi188_113957_object127791:head# 2024-04-23T16:33:55.978 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.979 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:567e5761:::benchmark_data_smithi188_113957_object41881:head# 2024-04-23T16:33:55.979 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.979 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568065fd:::benchmark_data_smithi188_113957_object83273:head# 2024-04-23T16:33:55.979 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.979 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5680aa92:::benchmark_data_smithi188_113957_object121093:head# 2024-04-23T16:33:55.979 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.979 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56826a54:::benchmark_data_smithi188_113957_object68655:head# 2024-04-23T16:33:55.979 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.979 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5682dc9b:::benchmark_data_smithi188_113957_object21958:head# 2024-04-23T16:33:55.979 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.979 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568390cd:::benchmark_data_smithi188_113957_object147601:head# 2024-04-23T16:33:55.979 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.979 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56840984:::benchmark_data_smithi188_113957_object19522:head# 2024-04-23T16:33:55.979 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.980 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568452eb:::benchmark_data_smithi188_113957_object47127:head# 2024-04-23T16:33:55.980 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.980 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56845539:::benchmark_data_smithi188_113957_object122734:head# 2024-04-23T16:33:55.980 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.980 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56848450:::benchmark_data_smithi188_113957_object24729:head# 2024-04-23T16:33:55.980 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.980 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568519d9:::benchmark_data_smithi188_113957_object43447:head# 2024-04-23T16:33:55.980 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.980 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56856229:::benchmark_data_smithi188_113957_object114669:head# 2024-04-23T16:33:55.980 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.980 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56863cb7:::benchmark_data_smithi188_113957_object151008:head# 2024-04-23T16:33:55.980 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.980 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5686a640:::benchmark_data_smithi188_113957_object123710:head# 2024-04-23T16:33:55.980 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.981 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568716ee:::benchmark_data_smithi188_113957_object131949:head# 2024-04-23T16:33:55.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.981 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56879a9e:::benchmark_data_smithi188_113957_object114956:head# 2024-04-23T16:33:55.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.981 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56884338:::benchmark_data_smithi188_113957_object17528:head# 2024-04-23T16:33:55.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.981 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5688f158:::benchmark_data_smithi188_113957_object11615:head# 2024-04-23T16:33:55.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.981 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56891b4d:::benchmark_data_smithi188_113957_object13976:head# 2024-04-23T16:33:55.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.981 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5689efff:::benchmark_data_smithi188_113957_object75031:head# 2024-04-23T16:33:55.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.981 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568a0739:::benchmark_data_smithi188_113957_object54553:head# 2024-04-23T16:33:55.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.981 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568a2de2:::benchmark_data_smithi188_113957_object140079:head# 2024-04-23T16:33:55.982 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.982 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568abc2e:::benchmark_data_smithi188_113957_object8583:head# 2024-04-23T16:33:55.982 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.982 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568abee3:::benchmark_data_smithi188_113957_object128:head# 2024-04-23T16:33:55.982 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.982 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568aff3e:::benchmark_data_smithi188_113957_object98783:head# 2024-04-23T16:33:55.982 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.982 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568b15d0:::benchmark_data_smithi188_113957_object55700:head# 2024-04-23T16:33:55.982 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.982 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568bfc8d:::benchmark_data_smithi188_113957_object102276:head# 2024-04-23T16:33:55.982 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.982 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568c1b56:::benchmark_data_smithi188_113957_object78299:head# 2024-04-23T16:33:55.982 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.982 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568c5391:::benchmark_data_smithi188_113957_object131331:head# 2024-04-23T16:33:55.983 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.983 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568c67c5:::benchmark_data_smithi188_113957_object84646:head# 2024-04-23T16:33:55.983 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.983 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568c9443:::benchmark_data_smithi188_113957_object53357:head# 2024-04-23T16:33:55.983 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.983 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568ce61e:::benchmark_data_smithi188_113957_object127630:head# 2024-04-23T16:33:55.983 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.983 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568d1cd9:::benchmark_data_smithi188_113957_object34214:head# 2024-04-23T16:33:55.983 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.983 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568d94a9:::benchmark_data_smithi188_113957_object38304:head# 2024-04-23T16:33:55.983 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.983 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568e0dd3:::benchmark_data_smithi188_113957_object115104:head# 2024-04-23T16:33:55.983 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.983 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568e40d6:::benchmark_data_smithi188_113957_object58586:head# 2024-04-23T16:33:55.984 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.984 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568e9c9c:::benchmark_data_smithi188_113957_object103906:head# 2024-04-23T16:33:55.984 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.984 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568f0c99:::benchmark_data_smithi188_113957_object33924:head# 2024-04-23T16:33:55.984 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.984 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568fbced:::benchmark_data_smithi188_113957_object58971:head# 2024-04-23T16:33:55.984 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.984 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568fe87d:::benchmark_data_smithi188_113957_object45327:head# 2024-04-23T16:33:55.984 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.984 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:568ff552:::benchmark_data_smithi188_113957_object43870:head# 2024-04-23T16:33:55.984 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.984 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5690691a:::benchmark_data_smithi188_113957_object147418:head# 2024-04-23T16:33:55.984 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.984 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5690f4ec:::benchmark_data_smithi188_113957_object148748:head# 2024-04-23T16:33:55.985 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.985 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5691042e:::benchmark_data_smithi188_113957_object14947:head# 2024-04-23T16:33:55.985 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.985 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56915a09:::benchmark_data_smithi188_113957_object63109:head# 2024-04-23T16:33:55.985 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.985 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56916b31:::benchmark_data_smithi188_113957_object47916:head# 2024-04-23T16:33:55.985 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.985 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56916f37:::benchmark_data_smithi188_113957_object91404:head# 2024-04-23T16:33:55.985 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.985 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56917190:::benchmark_data_smithi188_113957_object142893:head# 2024-04-23T16:33:55.985 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.985 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56919abc:::benchmark_data_smithi188_113957_object92512:head# 2024-04-23T16:33:55.985 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.985 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5691e20e:::benchmark_data_smithi188_113957_object58791:head# 2024-04-23T16:33:55.986 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.986 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56923fdf:::benchmark_data_smithi188_113957_object26590:head# 2024-04-23T16:33:55.986 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.986 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5692befb:::benchmark_data_smithi188_113957_object123967:head# 2024-04-23T16:33:55.986 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.986 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5692f7f1:::benchmark_data_smithi188_113957_object35028:head# 2024-04-23T16:33:55.986 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.986 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5693268b:::benchmark_data_smithi188_113957_object86568:head# 2024-04-23T16:33:55.986 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.986 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56935832:::benchmark_data_smithi188_113957_object39235:head# 2024-04-23T16:33:55.986 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.986 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56942654:::benchmark_data_smithi188_113957_object79409:head# 2024-04-23T16:33:55.986 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.986 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5694530d:::benchmark_data_smithi188_113957_object120303:head# 2024-04-23T16:33:55.987 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.987 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5694a3bd:::benchmark_data_smithi188_113957_object118865:head# 2024-04-23T16:33:55.987 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.987 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5694d475:::benchmark_data_smithi188_113957_object134017:head# 2024-04-23T16:33:55.987 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.987 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5695574a:::benchmark_data_smithi188_113957_object109518:head# 2024-04-23T16:33:55.987 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.987 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56968537:::benchmark_data_smithi188_113957_object92667:head# 2024-04-23T16:33:55.987 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.987 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5696cb24:::benchmark_data_smithi188_113957_object29167:head# 2024-04-23T16:33:55.987 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.987 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56970c7f:::benchmark_data_smithi188_113957_object21074:head# 2024-04-23T16:33:55.987 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.987 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:569731e6:::benchmark_data_smithi188_113957_object45013:head# 2024-04-23T16:33:55.987 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.988 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5697c1dc:::benchmark_data_smithi188_113957_object18639:head# 2024-04-23T16:33:55.988 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.988 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56984e5c:::benchmark_data_smithi188_113957_object94809:head# 2024-04-23T16:33:55.988 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.988 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5698a46c:::benchmark_data_smithi188_113957_object77309:head# 2024-04-23T16:33:55.988 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.988 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5698b96e:::benchmark_data_smithi188_113957_object21995:head# 2024-04-23T16:33:55.988 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.988 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56990ee3:::benchmark_data_smithi188_113957_object109902:head# 2024-04-23T16:33:55.988 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.988 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56993a58:::benchmark_data_smithi188_113957_object86482:head# 2024-04-23T16:33:55.988 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.988 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56993f50:::benchmark_data_smithi188_113957_object27290:head# 2024-04-23T16:33:55.988 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.989 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56996c78:::benchmark_data_smithi188_113957_object38369:head# 2024-04-23T16:33:55.989 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.989 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5699f2a9:::benchmark_data_smithi188_113957_object121437:head# 2024-04-23T16:33:55.989 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.989 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:569a2a07:::benchmark_data_smithi188_113957_object40060:head# 2024-04-23T16:33:55.989 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.989 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:569a5a0d:::benchmark_data_smithi188_113957_object82581:head# 2024-04-23T16:33:55.989 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.989 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:569a66ae:::benchmark_data_smithi188_113957_object12758:head# 2024-04-23T16:33:55.989 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.989 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:569ab4ef:::benchmark_data_smithi188_113957_object81903:head# 2024-04-23T16:33:55.989 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.989 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:569cf0d6:::benchmark_data_smithi188_113957_object296:head# 2024-04-23T16:33:55.989 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.989 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:569d808c:::benchmark_data_smithi188_113957_object54644:head# 2024-04-23T16:33:55.990 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.990 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:569d8d36:::benchmark_data_smithi188_113957_object146518:head# 2024-04-23T16:33:55.990 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.990 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:569d8ecc:::benchmark_data_smithi188_113957_object147624:head# 2024-04-23T16:33:55.990 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.990 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:569e2cf1:::benchmark_data_smithi188_113957_object100085:head# 2024-04-23T16:33:55.990 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.990 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:569e85d9:::benchmark_data_smithi188_113957_object101481:head# 2024-04-23T16:33:55.990 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.990 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:569e9353:::benchmark_data_smithi188_113957_object99862:head# 2024-04-23T16:33:55.990 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.990 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:569f8981:::benchmark_data_smithi188_113957_object29080:head# 2024-04-23T16:33:55.990 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.990 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a03da5:::benchmark_data_smithi188_113957_object16089:head# 2024-04-23T16:33:55.991 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.991 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a0d3bd:::benchmark_data_smithi188_113957_object85848:head# 2024-04-23T16:33:55.991 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.991 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a0eefd:::benchmark_data_smithi188_113957_object93769:head# 2024-04-23T16:33:55.991 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.991 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a138ef:::benchmark_data_smithi188_113957_object23687:head# 2024-04-23T16:33:55.991 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.991 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a14c93:::benchmark_data_smithi188_113957_object145626:head# 2024-04-23T16:33:55.991 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.991 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a18b10:::benchmark_data_smithi188_113957_object27547:head# 2024-04-23T16:33:55.991 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.991 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a1c15e:::benchmark_data_smithi188_113957_object76900:head# 2024-04-23T16:33:55.991 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.991 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a27ec1:::benchmark_data_smithi188_113957_object28242:head# 2024-04-23T16:33:55.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a297d2:::benchmark_data_smithi188_113957_object149929:head# 2024-04-23T16:33:55.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a3042a:::benchmark_data_smithi188_113957_object37459:head# 2024-04-23T16:33:55.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a3135c:::benchmark_data_smithi188_113957_object31143:head# 2024-04-23T16:33:55.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a3654f:::benchmark_data_smithi188_113957_object136837:head# 2024-04-23T16:33:55.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a37ade:::benchmark_data_smithi188_113957_object11841:head# 2024-04-23T16:33:55.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a40671:::benchmark_data_smithi188_113957_object29500:head# 2024-04-23T16:33:55.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.992 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a4597b:::benchmark_data_smithi188_113957_object151257:head# 2024-04-23T16:33:55.992 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.993 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a4b104:::benchmark_data_smithi188_113957_object84276:head# 2024-04-23T16:33:55.993 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.993 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a4dec8:::benchmark_data_smithi188_113957_object67743:head# 2024-04-23T16:33:55.993 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.993 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a5ca9e:::benchmark_data_smithi188_113957_object71600:head# 2024-04-23T16:33:55.993 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.993 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a5d50b:::benchmark_data_smithi188_113957_object28014:head# 2024-04-23T16:33:55.993 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.993 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a66714:::benchmark_data_smithi188_113957_object90293:head# 2024-04-23T16:33:55.993 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.993 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a675d6:::benchmark_data_smithi188_113957_object46260:head# 2024-04-23T16:33:55.993 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.993 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a6810f:::benchmark_data_smithi188_113957_object60650:head# 2024-04-23T16:33:55.993 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.994 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a697cc:::benchmark_data_smithi188_113957_object31711:head# 2024-04-23T16:33:55.994 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.994 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a6b485:::benchmark_data_smithi188_113957_object146654:head# 2024-04-23T16:33:55.994 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.994 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a73862:::benchmark_data_smithi188_113957_object3051:head# 2024-04-23T16:33:55.994 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.994 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a75fe8:::benchmark_data_smithi188_113957_object74146:head# 2024-04-23T16:33:55.994 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.994 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a8c3c8:::benchmark_data_smithi188_113957_object77540:head# 2024-04-23T16:33:55.994 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.994 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a8d873:::benchmark_data_smithi188_113957_object66358:head# 2024-04-23T16:33:55.994 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.994 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a8f3fe:::benchmark_data_smithi188_113957_object114785:head# 2024-04-23T16:33:55.995 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.995 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a937db:::benchmark_data_smithi188_113957_object150202:head# 2024-04-23T16:33:55.995 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.995 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a94354:::benchmark_data_smithi188_113957_object97345:head# 2024-04-23T16:33:55.995 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.995 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56a9ab97:::benchmark_data_smithi188_113957_object26953:head# 2024-04-23T16:33:55.995 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.995 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56aa28d0:::benchmark_data_smithi188_113957_object47272:head# 2024-04-23T16:33:55.995 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.995 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56aa324c:::benchmark_data_smithi188_113957_object148082:head# 2024-04-23T16:33:55.995 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.995 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56ac478e:::benchmark_data_smithi188_113957_object104399:head# 2024-04-23T16:33:55.995 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.995 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56aca95b:::benchmark_data_smithi188_113957_object10505:head# 2024-04-23T16:33:55.996 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.996 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56acf3a7:::benchmark_data_smithi188_113957_object68576:head# 2024-04-23T16:33:55.996 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.996 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56acfd96:::benchmark_data_smithi188_113957_object122575:head# 2024-04-23T16:33:55.996 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.996 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56ad0151:::benchmark_data_smithi188_113957_object61704:head# 2024-04-23T16:33:55.996 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.996 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56ada056:::benchmark_data_smithi188_113957_object122689:head# 2024-04-23T16:33:55.996 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.996 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56ae64ec:::benchmark_data_smithi188_113957_object104592:head# 2024-04-23T16:33:55.996 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.996 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56ae7bf0:::benchmark_data_smithi188_113957_object92248:head# 2024-04-23T16:33:55.996 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.996 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56af00bf:::benchmark_data_smithi188_113957_object61163:head# 2024-04-23T16:33:55.996 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.997 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56af9ad6:::benchmark_data_smithi188_113957_object67773:head# 2024-04-23T16:33:55.997 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.997 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b0026a:::benchmark_data_smithi188_113957_object87707:head# 2024-04-23T16:33:55.997 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.997 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b06452:::benchmark_data_smithi188_113957_object30351:head# 2024-04-23T16:33:55.997 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.997 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b0bec4:::benchmark_data_smithi188_113957_object142462:head# 2024-04-23T16:33:55.997 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.997 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b0d595:::benchmark_data_smithi188_113957_object125818:head# 2024-04-23T16:33:55.997 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.997 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b19cda:::benchmark_data_smithi188_113957_object98032:head# 2024-04-23T16:33:55.997 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.997 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b22ade:::benchmark_data_smithi188_113957_object56806:head# 2024-04-23T16:33:55.997 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.997 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b24402:::benchmark_data_smithi188_113957_object47682:head# 2024-04-23T16:33:55.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b25e51:::benchmark_data_smithi188_113957_object1091:head# 2024-04-23T16:33:55.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b290c0:::benchmark_data_smithi188_113957_object71171:head# 2024-04-23T16:33:55.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b2a8f7:::benchmark_data_smithi188_113957_object9215:head# 2024-04-23T16:33:55.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b2f271:::benchmark_data_smithi188_113957_object54459:head# 2024-04-23T16:33:55.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b30f89:::benchmark_data_smithi188_113957_object141278:head# 2024-04-23T16:33:55.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b38747:::benchmark_data_smithi188_113957_object149062:head# 2024-04-23T16:33:55.998 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.998 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b38ebe:::benchmark_data_smithi188_113957_object60903:head# 2024-04-23T16:33:55.999 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:55.999 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b43564:::benchmark_data_smithi188_113957_object48027:head# 2024-04-23T16:33:55.999 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.001 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:55 smithi188 ceph-mon[44101]: pgmap v1130: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.4 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 622 op/s; 25846/274602 objects degraded (9.412%); 7.6 MiB/s, 120 objects/s recovering 2024-04-23T16:33:56.002 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:33:56.002 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:33:56.002 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:33:56.002 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:33:56.002 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:33:56.014 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b512f7:::benchmark_data_smithi188_113957_object12921:head# 2024-04-23T16:33:56.014 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.014 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b518f0:::benchmark_data_smithi188_113957_object90531:head# 2024-04-23T16:33:56.014 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b558c0:::benchmark_data_smithi188_113957_object30277:head# 2024-04-23T16:33:56.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b55cff:::benchmark_data_smithi188_113957_object125762:head# 2024-04-23T16:33:56.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b5a02e:::benchmark_data_smithi188_113957_object99800:head# 2024-04-23T16:33:56.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b5e5a8:::benchmark_data_smithi188_113957_object42901:head# 2024-04-23T16:33:56.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b6394b:::benchmark_data_smithi188_113957_object101794:head# 2024-04-23T16:33:56.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b699f9:::benchmark_data_smithi188_113957_object85495:head# 2024-04-23T16:33:56.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b6a247:::benchmark_data_smithi188_113957_object97506:head# 2024-04-23T16:33:56.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b773ea:::benchmark_data_smithi188_113957_object46828:head# 2024-04-23T16:33:56.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b7ff99:::benchmark_data_smithi188_113957_object131323:head# 2024-04-23T16:33:56.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b80c21:::benchmark_data_smithi188_113957_object19703:head# 2024-04-23T16:33:56.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b8e80e:::benchmark_data_smithi188_113957_object127129:head# 2024-04-23T16:33:56.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b9afa9:::benchmark_data_smithi188_113957_object117353:head# 2024-04-23T16:33:56.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56b9c9fe:::benchmark_data_smithi188_113957_object120520:head# 2024-04-23T16:33:56.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56ba0d54:::benchmark_data_smithi188_113957_object103585:head# 2024-04-23T16:33:56.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.017 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56babc6f:::benchmark_data_smithi188_113957_object132115:head# 2024-04-23T16:33:56.017 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.017 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56bae99e:::benchmark_data_smithi188_113957_object132436:head# 2024-04-23T16:33:56.017 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.017 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56bba03f:::benchmark_data_smithi188_113957_object93826:head# 2024-04-23T16:33:56.017 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.017 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56bbcb3c:::benchmark_data_smithi188_113957_object32198:head# 2024-04-23T16:33:56.017 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.017 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56bd2521:::benchmark_data_smithi188_113957_object48879:head# 2024-04-23T16:33:56.017 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.017 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56bdbbd9:::benchmark_data_smithi188_113957_object77406:head# 2024-04-23T16:33:56.017 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.017 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56beb9d6:::benchmark_data_smithi188_113957_object69803:head# 2024-04-23T16:33:56.017 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.018 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56bf19e4:::benchmark_data_smithi188_113957_object85300:head# 2024-04-23T16:33:56.018 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.018 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56bf2b53:::benchmark_data_smithi188_113957_object128109:head# 2024-04-23T16:33:56.018 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.018 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56bf348d:::benchmark_data_smithi188_113957_object77575:head# 2024-04-23T16:33:56.018 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.018 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56bf39a0:::benchmark_data_smithi188_113957_object100531:head# 2024-04-23T16:33:56.018 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.018 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56bf5f0a:::benchmark_data_smithi188_113957_object69379:head# 2024-04-23T16:33:56.018 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.018 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56bf8df3:::benchmark_data_smithi188_113957_object92304:head# 2024-04-23T16:33:56.018 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.018 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56bfacf2:::benchmark_data_smithi188_113957_object103650:head# 2024-04-23T16:33:56.019 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.019 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c04ded:::benchmark_data_smithi188_113957_object9793:head# 2024-04-23T16:33:56.019 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.019 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c13f99:::benchmark_data_smithi188_113957_object16579:head# 2024-04-23T16:33:56.019 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.019 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c21ed2:::benchmark_data_smithi188_113957_object10006:head# 2024-04-23T16:33:56.019 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.019 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c2ab02:::benchmark_data_smithi188_113957_object101885:head# 2024-04-23T16:33:56.019 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.019 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c2ad32:::benchmark_data_smithi188_113957_object123647:head# 2024-04-23T16:33:56.019 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.019 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c34a05:::benchmark_data_smithi188_113957_object150224:head# 2024-04-23T16:33:56.019 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c36de4:::benchmark_data_smithi188_113957_object53701:head# 2024-04-23T16:33:56.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c3efd8:::benchmark_data_smithi188_113957_object8766:head# 2024-04-23T16:33:56.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c3fb49:::benchmark_data_smithi188_113957_object117236:head# 2024-04-23T16:33:56.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c42daf:::benchmark_data_smithi188_113957_object126739:head# 2024-04-23T16:33:56.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c50546:::benchmark_data_smithi188_113957_object50185:head# 2024-04-23T16:33:56.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c6069d:::benchmark_data_smithi188_113957_object128991:head# 2024-04-23T16:33:56.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c6f941:::benchmark_data_smithi188_113957_object30253:head# 2024-04-23T16:33:56.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c7a32c:::benchmark_data_smithi188_113957_object125807:head# 2024-04-23T16:33:56.021 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c7bb63:::benchmark_data_smithi188_113957_object38685:head# 2024-04-23T16:33:56.021 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c7ec4a:::benchmark_data_smithi188_113957_object69363:head# 2024-04-23T16:33:56.021 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c8306c:::benchmark_data_smithi188_113957_object113094:head# 2024-04-23T16:33:56.021 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c8a757:::benchmark_data_smithi188_113957_object4736:head# 2024-04-23T16:33:56.021 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c8be37:::benchmark_data_smithi188_113957_object70724:head# 2024-04-23T16:33:56.021 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56c939e3:::benchmark_data_smithi188_113957_object119622:head# 2024-04-23T16:33:56.021 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56ca539b:::benchmark_data_smithi188_113957_object136698:head# 2024-04-23T16:33:56.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56cb90bc:::benchmark_data_smithi188_113957_object13075:head# 2024-04-23T16:33:56.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56cbcbd2:::benchmark_data_smithi188_113957_object114083:head# 2024-04-23T16:33:56.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56cbd379:::benchmark_data_smithi188_113957_object51721:head# 2024-04-23T16:33:56.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56cbee46:::benchmark_data_smithi188_113957_object124537:head# 2024-04-23T16:33:56.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56cc1a4b:::benchmark_data_smithi188_113957_object14139:head# 2024-04-23T16:33:56.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56cc4bb8:::benchmark_data_smithi188_113957_object141117:head# 2024-04-23T16:33:56.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56cc6322:::benchmark_data_smithi188_113957_object17925:head# 2024-04-23T16:33:56.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56cd8be7:::benchmark_data_smithi188_113957_object149213:head# 2024-04-23T16:33:56.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56ce1cbd:::benchmark_data_smithi188_113957_object110130:head# 2024-04-23T16:33:56.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56ce6474:::benchmark_data_smithi188_113957_object25836:head# 2024-04-23T16:33:56.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56cf22f8:::benchmark_data_smithi188_113957_object2429:head# 2024-04-23T16:33:56.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56cf36d2:::benchmark_data_smithi188_113957_object58575:head# 2024-04-23T16:33:56.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d04d96:::benchmark_data_smithi188_113957_object2343:head# 2024-04-23T16:33:56.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d0fca3:::benchmark_data_smithi188_113957_object11463:head# 2024-04-23T16:33:56.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d109c9:::benchmark_data_smithi188_113957_object69758:head# 2024-04-23T16:33:56.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d10a71:::benchmark_data_smithi188_113957_object59011:head# 2024-04-23T16:33:56.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d118c6:::benchmark_data_smithi188_113957_object94612:head# 2024-04-23T16:33:56.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d14788:::benchmark_data_smithi188_113957_object123750:head# 2024-04-23T16:33:56.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d1c09c:::benchmark_data_smithi188_113957_object19700:head# 2024-04-23T16:33:56.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d1c1e4:::benchmark_data_smithi188_113957_object79824:head# 2024-04-23T16:33:56.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d21832:::benchmark_data_smithi188_113957_object68916:head# 2024-04-23T16:33:56.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d269a8:::benchmark_data_smithi188_113957_object43848:head# 2024-04-23T16:33:56.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d293bc:::benchmark_data_smithi188_113957_object88283:head# 2024-04-23T16:33:56.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d2d006:::benchmark_data_smithi188_113957_object146132:head# 2024-04-23T16:33:56.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d30cc8:::benchmark_data_smithi188_113957_object135202:head# 2024-04-23T16:33:56.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d326ba:::benchmark_data_smithi188_113957_object49538:head# 2024-04-23T16:33:56.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d360b1:::benchmark_data_smithi188_113957_object61125:head# 2024-04-23T16:33:56.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d3aae3:::benchmark_data_smithi188_113957_object142548:head# 2024-04-23T16:33:56.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d49ed2:::benchmark_data_smithi188_113957_object84623:head# 2024-04-23T16:33:56.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d50b67:::benchmark_data_smithi188_113957_object91719:head# 2024-04-23T16:33:56.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d5187e:::benchmark_data_smithi188_113957_object45697:head# 2024-04-23T16:33:56.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d60e69:::benchmark_data_smithi188_113957_object40298:head# 2024-04-23T16:33:56.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d720d1:::benchmark_data_smithi188_113957_object36026:head# 2024-04-23T16:33:56.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d72330:::benchmark_data_smithi188_113957_object66789:head# 2024-04-23T16:33:56.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d74c1e:::benchmark_data_smithi188_113957_object90322:head# 2024-04-23T16:33:56.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d7c2bb:::benchmark_data_smithi188_113957_object15396:head# 2024-04-23T16:33:56.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d859b9:::benchmark_data_smithi188_113957_object106392:head# 2024-04-23T16:33:56.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d86ddb:::benchmark_data_smithi188_113957_object140398:head# 2024-04-23T16:33:56.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d8f07a:::benchmark_data_smithi188_113957_object7161:head# 2024-04-23T16:33:56.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d99621:::benchmark_data_smithi188_113957_object148824:head# 2024-04-23T16:33:56.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d9d1e1:::benchmark_data_smithi188_113957_object100996:head# 2024-04-23T16:33:56.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56d9d3c8:::benchmark_data_smithi188_113957_object108003:head# 2024-04-23T16:33:56.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56dac983:::benchmark_data_smithi188_113957_object70593:head# 2024-04-23T16:33:56.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56daff7b:::benchmark_data_smithi188_113957_object100450:head# 2024-04-23T16:33:56.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56db2bb6:::benchmark_data_smithi188_113957_object56601:head# 2024-04-23T16:33:56.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56db473e:::benchmark_data_smithi188_113957_object55089:head# 2024-04-23T16:33:56.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56dba1f4:::benchmark_data_smithi188_113957_object67484:head# 2024-04-23T16:33:56.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56dbe9a9:::benchmark_data_smithi188_113957_object55902:head# 2024-04-23T16:33:56.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56dc6159:::benchmark_data_smithi188_113957_object147534:head# 2024-04-23T16:33:56.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56dcc8cc:::benchmark_data_smithi188_113957_object126669:head# 2024-04-23T16:33:56.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56dd08f0:::benchmark_data_smithi188_113957_object87771:head# 2024-04-23T16:33:56.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56dd1369:::benchmark_data_smithi188_113957_object95970:head# 2024-04-23T16:33:56.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56dd772e:::benchmark_data_smithi188_113957_object151612:head# 2024-04-23T16:33:56.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56de90a2:::benchmark_data_smithi188_113957_object67497:head# 2024-04-23T16:33:56.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56def2aa:::benchmark_data_smithi188_113957_object105529:head# 2024-04-23T16:33:56.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56df290f:::benchmark_data_smithi188_113957_object55510:head# 2024-04-23T16:33:56.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56e0ab45:::benchmark_data_smithi188_113957_object20254:head# 2024-04-23T16:33:56.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56e0bea7:::benchmark_data_smithi188_113957_object36235:head# 2024-04-23T16:33:56.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56e19fe1:::benchmark_data_smithi188_113957_object149690:head# 2024-04-23T16:33:56.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56e27099:::benchmark_data_smithi188_113957_object30636:head# 2024-04-23T16:33:56.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56e2c2f4:::benchmark_data_smithi188_113957_object72312:head# 2024-04-23T16:33:56.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56e2c5cc:::benchmark_data_smithi188_113957_object37607:head# 2024-04-23T16:33:56.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56e5c37c:::benchmark_data_smithi188_113957_object148241:head# 2024-04-23T16:33:56.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56e5ebea:::benchmark_data_smithi188_113957_object103143:head# 2024-04-23T16:33:56.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56e60e8f:::benchmark_data_smithi188_113957_object120015:head# 2024-04-23T16:33:56.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.031 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56e6e14d:::benchmark_data_smithi188_113957_object30298:head# 2024-04-23T16:33:56.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.031 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:56.032 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:33:56.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56e7ac95:::benchmark_data_smithi188_113957_object107854:head# 2024-04-23T16:33:56.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56e8d3b5:::benchmark_data_smithi188_113957_object93967:head# 2024-04-23T16:33:56.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56e98d35:::benchmark_data_smithi188_113957_object92020:head# 2024-04-23T16:33:56.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56e9d3d8:::benchmark_data_smithi188_113957_object8204:head# 2024-04-23T16:33:56.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56ea42eb:::benchmark_data_smithi188_113957_object100514:head# 2024-04-23T16:33:56.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56ea7594:::benchmark_data_smithi188_113957_object89011:head# 2024-04-23T16:33:56.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56eac1e7:::benchmark_data_smithi188_113957_object12865:head# 2024-04-23T16:33:56.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56ec4db6:::benchmark_data_smithi188_113957_object27333:head# 2024-04-23T16:33:56.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56ec5b4b:::benchmark_data_smithi188_113957_object117146:head# 2024-04-23T16:33:56.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56ed46a9:::benchmark_data_smithi188_113957_object31758:head# 2024-04-23T16:33:56.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56ee0127:::benchmark_data_smithi188_113957_object109612:head# 2024-04-23T16:33:56.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56ee7410:::benchmark_data_smithi188_113957_object105128:head# 2024-04-23T16:33:56.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56ef66c9:::benchmark_data_smithi188_113957_object19534:head# 2024-04-23T16:33:56.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f04b13:::benchmark_data_smithi188_113957_object54747:head# 2024-04-23T16:33:56.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f093de:::benchmark_data_smithi188_113957_object50793:head# 2024-04-23T16:33:56.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f0d55d:::benchmark_data_smithi188_113957_object22749:head# 2024-04-23T16:33:56.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f103d3:::benchmark_data_smithi188_113957_object124940:head# 2024-04-23T16:33:56.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f1552b:::benchmark_data_smithi188_113957_object96982:head# 2024-04-23T16:33:56.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f21b36:::benchmark_data_smithi188_113957_object74156:head# 2024-04-23T16:33:56.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f233ed:::benchmark_data_smithi188_113957_object243:head# 2024-04-23T16:33:56.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f28768:::benchmark_data_smithi188_113957_object38198:head# 2024-04-23T16:33:56.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f29113:::benchmark_data_smithi188_113957_object118156:head# 2024-04-23T16:33:56.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f2f800:::benchmark_data_smithi188_113957_object48409:head# 2024-04-23T16:33:56.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f3649c:::benchmark_data_smithi188_113957_object17948:head# 2024-04-23T16:33:56.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f55660:::benchmark_data_smithi188_113957_object4397:head# 2024-04-23T16:33:56.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f580ba:::benchmark_data_smithi188_113957_object17698:head# 2024-04-23T16:33:56.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f5a7f4:::benchmark_data_smithi188_113957_object55144:head# 2024-04-23T16:33:56.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f604d6:::benchmark_data_smithi188_113957_object44435:head# 2024-04-23T16:33:56.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f615b3:::benchmark_data_smithi188_113957_object54660:head# 2024-04-23T16:33:56.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f66827:::benchmark_data_smithi188_113957_object36878:head# 2024-04-23T16:33:56.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f67603:::benchmark_data_smithi188_113957_object25665:head# 2024-04-23T16:33:56.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f6b2a8:::benchmark_data_smithi188_113957_object9369:head# 2024-04-23T16:33:56.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f6c1cc:::benchmark_data_smithi188_113957_object3306:head# 2024-04-23T16:33:56.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f6cf8e:::benchmark_data_smithi188_113957_object50245:head# 2024-04-23T16:33:56.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f6d9e3:::benchmark_data_smithi188_113957_object867:head# 2024-04-23T16:33:56.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f731dd:::benchmark_data_smithi188_113957_object139868:head# 2024-04-23T16:33:56.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f7872e:::benchmark_data_smithi188_113957_object111905:head# 2024-04-23T16:33:56.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f7b2d0:::benchmark_data_smithi188_113957_object61169:head# 2024-04-23T16:33:56.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f7b6fb:::benchmark_data_smithi188_113957_object46422:head# 2024-04-23T16:33:56.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f7dfa4:::benchmark_data_smithi188_113957_object120671:head# 2024-04-23T16:33:56.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f80a23:::benchmark_data_smithi188_113957_object52928:head# 2024-04-23T16:33:56.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f8c33a:::benchmark_data_smithi188_113957_object13401:head# 2024-04-23T16:33:56.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f8c8a1:::benchmark_data_smithi188_113957_object142375:head# 2024-04-23T16:33:56.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f9999e:::benchmark_data_smithi188_113957_object1892:head# 2024-04-23T16:33:56.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f9b224:::benchmark_data_smithi188_113957_object57331:head# 2024-04-23T16:33:56.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f9da86:::benchmark_data_smithi188_113957_object86447:head# 2024-04-23T16:33:56.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f9e859:::benchmark_data_smithi188_113957_object121828:head# 2024-04-23T16:33:56.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56f9e9dc:::benchmark_data_smithi188_113957_object135131:head# 2024-04-23T16:33:56.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56fa5dd1:::benchmark_data_smithi188_113957_object30010:head# 2024-04-23T16:33:56.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56fa7c34:::benchmark_data_smithi188_113957_object15478:head# 2024-04-23T16:33:56.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56fab5ef:::benchmark_data_smithi188_113957_object113879:head# 2024-04-23T16:33:56.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56facf59:::benchmark_data_smithi188_113957_object144775:head# 2024-04-23T16:33:56.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56fb1cf8:::benchmark_data_smithi188_113957_object90116:head# 2024-04-23T16:33:56.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56fb3acd:::benchmark_data_smithi188_113957_object35445:head# 2024-04-23T16:33:56.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56fbe7df:::benchmark_data_smithi188_113957_object113031:head# 2024-04-23T16:33:56.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56fc4d01:::benchmark_data_smithi188_113957_object80018:head# 2024-04-23T16:33:56.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56fc5ba7:::benchmark_data_smithi188_113957_object6621:head# 2024-04-23T16:33:56.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56fc9532:::benchmark_data_smithi188_113957_object141333:head# 2024-04-23T16:33:56.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56fca945:::benchmark_data_smithi188_113957_object73801:head# 2024-04-23T16:33:56.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56fcb0a2:::benchmark_data_smithi188_113957_object82910:head# 2024-04-23T16:33:56.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56fdadc4:::benchmark_data_smithi188_113957_object28960:head# 2024-04-23T16:33:56.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56fe66c0:::benchmark_data_smithi188_113957_object144982:head# 2024-04-23T16:33:56.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56fe82b5:::benchmark_data_smithi188_113957_object152050:head# 2024-04-23T16:33:56.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:56ff5785:::benchmark_data_smithi188_113957_object44324:head# 2024-04-23T16:33:56.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570000bb:::benchmark_data_smithi188_113957_object93495:head# 2024-04-23T16:33:56.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570009ea:::benchmark_data_smithi188_113957_object64637:head# 2024-04-23T16:33:56.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57000cea:::benchmark_data_smithi188_113957_object51886:head# 2024-04-23T16:33:56.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57001f49:::benchmark_data_smithi188_113957_object117770:head# 2024-04-23T16:33:56.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57011d54:::benchmark_data_smithi188_113957_object45116:head# 2024-04-23T16:33:56.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570132d3:::benchmark_data_smithi188_113957_object20770:head# 2024-04-23T16:33:56.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5701d24d:::benchmark_data_smithi188_113957_object27635:head# 2024-04-23T16:33:56.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570289df:::benchmark_data_smithi188_113957_object32278:head# 2024-04-23T16:33:56.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57028c86:::benchmark_data_smithi188_113957_object4186:head# 2024-04-23T16:33:56.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5702a9d4:::benchmark_data_smithi188_113957_object74542:head# 2024-04-23T16:33:56.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5702fd55:::benchmark_data_smithi188_113957_object96261:head# 2024-04-23T16:33:56.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5703f36d:::benchmark_data_smithi188_113957_object62764:head# 2024-04-23T16:33:56.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5704077a:::benchmark_data_smithi188_113957_object17490:head# 2024-04-23T16:33:56.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5704502d:::benchmark_data_smithi188_113957_object26031:head# 2024-04-23T16:33:56.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57046249:::benchmark_data_smithi188_113957_object78067:head# 2024-04-23T16:33:56.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57054508:::benchmark_data_smithi188_113957_object143384:head# 2024-04-23T16:33:56.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570552a0:::benchmark_data_smithi188_113957_object132134:head# 2024-04-23T16:33:56.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57055bda:::benchmark_data_smithi188_113957_object15888:head# 2024-04-23T16:33:56.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5705f266:::benchmark_data_smithi188_113957_object75170:head# 2024-04-23T16:33:56.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5705ffd4:::benchmark_data_smithi188_113957_object65250:head# 2024-04-23T16:33:56.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570639ec:::benchmark_data_smithi188_113957_object32729:head# 2024-04-23T16:33:56.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570647f0:::benchmark_data_smithi188_113957_object119636:head# 2024-04-23T16:33:56.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5707b304:::benchmark_data_smithi188_113957_object128868:head# 2024-04-23T16:33:56.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570858bf:::benchmark_data_smithi188_113957_object134669:head# 2024-04-23T16:33:56.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57086e32:::benchmark_data_smithi188_113957_object61714:head# 2024-04-23T16:33:56.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5708bad3:::benchmark_data_smithi188_113957_object87589:head# 2024-04-23T16:33:56.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5708f4bb:::benchmark_data_smithi188_113957_object125672:head# 2024-04-23T16:33:56.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57097a20:::benchmark_data_smithi188_113957_object116682:head# 2024-04-23T16:33:56.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5709800b:::benchmark_data_smithi188_113957_object377:head# 2024-04-23T16:33:56.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5709f1c4:::benchmark_data_smithi188_113957_object51390:head# 2024-04-23T16:33:56.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570a0a78:::benchmark_data_smithi188_113957_object101770:head# 2024-04-23T16:33:56.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570a14bd:::benchmark_data_smithi188_113957_object113400:head# 2024-04-23T16:33:56.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570a1e61:::benchmark_data_smithi188_113957_object15702:head# 2024-04-23T16:33:56.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570abffb:::benchmark_data_smithi188_113957_object37747:head# 2024-04-23T16:33:56.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570adc6b:::benchmark_data_smithi188_113957_object148393:head# 2024-04-23T16:33:56.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570c1a10:::benchmark_data_smithi188_113957_object126136:head# 2024-04-23T16:33:56.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570c2692:::benchmark_data_smithi188_113957_object112966:head# 2024-04-23T16:33:56.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570c5927:::benchmark_data_smithi188_113957_object84874:head# 2024-04-23T16:33:56.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570ceca3:::benchmark_data_smithi188_113957_object13891:head# 2024-04-23T16:33:56.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570d6f94:::benchmark_data_smithi188_113957_object135439:head# 2024-04-23T16:33:56.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570e41b5:::benchmark_data_smithi188_113957_object16808:head# 2024-04-23T16:33:56.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570e840f:::benchmark_data_smithi188_113957_object19821:head# 2024-04-23T16:33:56.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:570fc220:::benchmark_data_smithi188_113957_object52047:head# 2024-04-23T16:33:56.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5710f8d4:::benchmark_data_smithi188_113957_object143070:head# 2024-04-23T16:33:56.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5711e478:::benchmark_data_smithi188_113957_object141972:head# 2024-04-23T16:33:56.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57128d0c:::benchmark_data_smithi188_113957_object16207:head# 2024-04-23T16:33:56.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5712b16c:::benchmark_data_smithi188_113957_object78565:head# 2024-04-23T16:33:56.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5712f027:::benchmark_data_smithi188_113957_object53118:head# 2024-04-23T16:33:56.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5713b0b7:::benchmark_data_smithi188_113957_object49326:head# 2024-04-23T16:33:56.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5713f04b:::benchmark_data_smithi188_113957_object117042:head# 2024-04-23T16:33:56.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57140b28:::benchmark_data_smithi188_113957_object56637:head# 2024-04-23T16:33:56.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57141338:::benchmark_data_smithi188_113957_object75163:head# 2024-04-23T16:33:56.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5714711a:::benchmark_data_smithi188_113957_object67487:head# 2024-04-23T16:33:56.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5714f633:::benchmark_data_smithi188_113957_object96182:head# 2024-04-23T16:33:56.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57150d7a:::benchmark_data_smithi188_113957_object29781:head# 2024-04-23T16:33:56.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57154f9d:::benchmark_data_smithi188_113957_object45185:head# 2024-04-23T16:33:56.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5715cb81:::benchmark_data_smithi188_113957_object124850:head# 2024-04-23T16:33:56.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5715dcdd:::benchmark_data_smithi188_113957_object59823:head# 2024-04-23T16:33:56.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5715eecf:::benchmark_data_smithi188_113957_object110765:head# 2024-04-23T16:33:56.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5716255b:::benchmark_data_smithi188_113957_object74866:head# 2024-04-23T16:33:56.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57165504:::benchmark_data_smithi188_113957_object73931:head# 2024-04-23T16:33:56.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5716572d:::benchmark_data_smithi188_113957_object28598:head# 2024-04-23T16:33:56.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57167e91:::benchmark_data_smithi188_113957_object104205:head# 2024-04-23T16:33:56.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5716c53a:::benchmark_data_smithi188_113957_object5056:head# 2024-04-23T16:33:56.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5716cca3:::benchmark_data_smithi188_113957_object79375:head# 2024-04-23T16:33:56.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5716d45b:::benchmark_data_smithi188_113957_object61186:head# 2024-04-23T16:33:56.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5716f04c:::benchmark_data_smithi188_113957_object146756:head# 2024-04-23T16:33:56.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57170502:::benchmark_data_smithi188_113957_object132333:head# 2024-04-23T16:33:56.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57172746:::benchmark_data_smithi188_113957_object67337:head# 2024-04-23T16:33:56.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:571731e4:::benchmark_data_smithi188_113957_object139122:head# 2024-04-23T16:33:56.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5717a6e3:::benchmark_data_smithi188_113957_object7619:head# 2024-04-23T16:33:56.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5718210e:::benchmark_data_smithi188_113957_object65356:head# 2024-04-23T16:33:56.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57185fc1:::benchmark_data_smithi188_113957_object110269:head# 2024-04-23T16:33:56.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5719ae5e:::benchmark_data_smithi188_113957_object34213:head# 2024-04-23T16:33:56.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5719eccc:::benchmark_data_smithi188_113957_object65686:head# 2024-04-23T16:33:56.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:571a3fa5:::benchmark_data_smithi188_113957_object128071:head# 2024-04-23T16:33:56.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:571a4e6a:::benchmark_data_smithi188_113957_object96659:head# 2024-04-23T16:33:56.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:571af5be:::benchmark_data_smithi188_113957_object131209:head# 2024-04-23T16:33:56.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:571b861a:::benchmark_data_smithi188_113957_object35466:head# 2024-04-23T16:33:56.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:571c1c55:::benchmark_data_smithi188_113957_object38102:head# 2024-04-23T16:33:56.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:571d07a9:::benchmark_data_smithi188_113957_object74479:head# 2024-04-23T16:33:56.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:571d7ffd:::benchmark_data_smithi188_113957_object52868:head# 2024-04-23T16:33:56.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572004e0:::benchmark_data_smithi188_113957_object133833:head# 2024-04-23T16:33:56.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572012c1:::benchmark_data_smithi188_113957_object106509:head# 2024-04-23T16:33:56.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57201560:::benchmark_data_smithi188_113957_object45053:head# 2024-04-23T16:33:56.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57206f94:::benchmark_data_smithi188_113957_object141290:head# 2024-04-23T16:33:56.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5721acb3:::benchmark_data_smithi188_113957_object512:head# 2024-04-23T16:33:56.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5721db84:::benchmark_data_smithi188_113957_object19235:head# 2024-04-23T16:33:56.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5722e5b9:::benchmark_data_smithi188_113957_object113743:head# 2024-04-23T16:33:56.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572345c8:::benchmark_data_smithi188_113957_object28223:head# 2024-04-23T16:33:56.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572353ad:::benchmark_data_smithi188_113957_object145678:head# 2024-04-23T16:33:56.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57246388:::benchmark_data_smithi188_113957_object68830:head# 2024-04-23T16:33:56.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57249276:::benchmark_data_smithi188_113957_object70723:head# 2024-04-23T16:33:56.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5724f313:::benchmark_data_smithi188_113957_object43033:head# 2024-04-23T16:33:56.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57267dc1:::benchmark_data_smithi188_113957_object13687:head# 2024-04-23T16:33:56.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5726c85a:::benchmark_data_smithi188_113957_object9077:head# 2024-04-23T16:33:56.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57274daa:::benchmark_data_smithi188_113957_object128024:head# 2024-04-23T16:33:56.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5728ebae:::benchmark_data_smithi188_113957_object81184:head# 2024-04-23T16:33:56.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5728eea3:::benchmark_data_smithi188_113957_object116925:head# 2024-04-23T16:33:56.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57290b38:::benchmark_data_smithi188_113957_object113657:head# 2024-04-23T16:33:56.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57290cfa:::benchmark_data_smithi188_113957_object1658:head# 2024-04-23T16:33:56.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57298a23:::benchmark_data_smithi188_113957_object39030:head# 2024-04-23T16:33:56.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5729a95a:::benchmark_data_smithi188_113957_object103098:head# 2024-04-23T16:33:56.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5729c4d5:::benchmark_data_smithi188_113957_object83638:head# 2024-04-23T16:33:56.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5729eeab:::benchmark_data_smithi188_113957_object25489:head# 2024-04-23T16:33:56.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572a749c:::benchmark_data_smithi188_113957_object112187:head# 2024-04-23T16:33:56.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572abc69:::benchmark_data_smithi188_113957_object97574:head# 2024-04-23T16:33:56.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572b27ed:::benchmark_data_smithi188_113957_object5905:head# 2024-04-23T16:33:56.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572b5b41:::benchmark_data_smithi188_113957_object80652:head# 2024-04-23T16:33:56.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572b955e:::benchmark_data_smithi188_113957_object129716:head# 2024-04-23T16:33:56.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572bc268:::benchmark_data_smithi188_113957_object114805:head# 2024-04-23T16:33:56.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572bd48a:::benchmark_data_smithi188_113957_object3496:head# 2024-04-23T16:33:56.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572bf121:::benchmark_data_smithi188_113957_object73010:head# 2024-04-23T16:33:56.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572c1bcb:::benchmark_data_smithi188_113957_object129198:head# 2024-04-23T16:33:56.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572c8123:::benchmark_data_smithi188_113957_object135543:head# 2024-04-23T16:33:56.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572cab43:::benchmark_data_smithi188_113957_object87056:head# 2024-04-23T16:33:56.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572cfdf8:::benchmark_data_smithi188_113957_object6039:head# 2024-04-23T16:33:56.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572d1014:::benchmark_data_smithi188_113957_object108031:head# 2024-04-23T16:33:56.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572d51b1:::benchmark_data_smithi188_113957_object89656:head# 2024-04-23T16:33:56.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572da272:::benchmark_data_smithi188_113957_object33874:head# 2024-04-23T16:33:56.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572dd6a3:::benchmark_data_smithi188_113957_object130080:head# 2024-04-23T16:33:56.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572e2dac:::benchmark_data_smithi188_113957_object85239:head# 2024-04-23T16:33:56.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572e816e:::benchmark_data_smithi188_113957_object107987:head# 2024-04-23T16:33:56.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572e8f71:::benchmark_data_smithi188_113957_object75722:head# 2024-04-23T16:33:56.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572f5426:::benchmark_data_smithi188_113957_object32728:head# 2024-04-23T16:33:56.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572f630d:::benchmark_data_smithi188_113957_object46354:head# 2024-04-23T16:33:56.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572f7d92:::benchmark_data_smithi188_113957_object87965:head# 2024-04-23T16:33:56.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:572fd2f1:::benchmark_data_smithi188_113957_object14327:head# 2024-04-23T16:33:56.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57302288:::benchmark_data_smithi188_113957_object110035:head# 2024-04-23T16:33:56.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57302c8a:::benchmark_data_smithi188_113957_object91567:head# 2024-04-23T16:33:56.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5730c6fd:::benchmark_data_smithi188_113957_object35501:head# 2024-04-23T16:33:56.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57311934:::benchmark_data_smithi188_113957_object1881:head# 2024-04-23T16:33:56.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57312d57:::benchmark_data_smithi188_113957_object3510:head# 2024-04-23T16:33:56.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5731d724:::benchmark_data_smithi188_113957_object41440:head# 2024-04-23T16:33:56.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5731dc68:::benchmark_data_smithi188_113957_object98572:head# 2024-04-23T16:33:56.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573293ff:::benchmark_data_smithi188_113957_object114836:head# 2024-04-23T16:33:56.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5732b21a:::benchmark_data_smithi188_113957_object6319:head# 2024-04-23T16:33:56.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57335969:::benchmark_data_smithi188_113957_object116130:head# 2024-04-23T16:33:56.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57341f08:::benchmark_data_smithi188_113957_object33931:head# 2024-04-23T16:33:56.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57347aab:::benchmark_data_smithi188_113957_object147019:head# 2024-04-23T16:33:56.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57348a89:::benchmark_data_smithi188_113957_object28565:head# 2024-04-23T16:33:56.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5734f6f5:::benchmark_data_smithi188_113957_object86527:head# 2024-04-23T16:33:56.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573536f3:::benchmark_data_smithi188_113957_object101328:head# 2024-04-23T16:33:56.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5735a0fd:::benchmark_data_smithi188_113957_object67095:head# 2024-04-23T16:33:56.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5736137d:::benchmark_data_smithi188_113957_object120454:head# 2024-04-23T16:33:56.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5736156e:::benchmark_data_smithi188_113957_object13115:head# 2024-04-23T16:33:56.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5736253c:::benchmark_data_smithi188_113957_object24745:head# 2024-04-23T16:33:56.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5736ad30:::benchmark_data_smithi188_113957_object112741:head# 2024-04-23T16:33:56.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5737b107:::benchmark_data_smithi188_113957_object18387:head# 2024-04-23T16:33:56.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5737ca2d:::benchmark_data_smithi188_113957_object139953:head# 2024-04-23T16:33:56.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5737e07c:::benchmark_data_smithi188_113957_object151744:head# 2024-04-23T16:33:56.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5738ddf9:::benchmark_data_smithi188_113957_object38434:head# 2024-04-23T16:33:56.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57391af7:::benchmark_data_smithi188_113957_object121720:head# 2024-04-23T16:33:56.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57395a87:::benchmark_data_smithi188_113957_object59448:head# 2024-04-23T16:33:56.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5739a3ea:::benchmark_data_smithi188_113957_object104727:head# 2024-04-23T16:33:56.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5739aedd:::benchmark_data_smithi188_113957_object52659:head# 2024-04-23T16:33:56.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5739b80e:::benchmark_data_smithi188_113957_object103346:head# 2024-04-23T16:33:56.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5739f8bc:::benchmark_data_smithi188_113957_object81183:head# 2024-04-23T16:33:56.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573a094b:::benchmark_data_smithi188_113957_object42166:head# 2024-04-23T16:33:56.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573a2f9b:::benchmark_data_smithi188_113957_object108771:head# 2024-04-23T16:33:56.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573a5536:::benchmark_data_smithi188_113957_object134014:head# 2024-04-23T16:33:56.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573af11f:::benchmark_data_smithi188_113957_object84965:head# 2024-04-23T16:33:56.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573af9d2:::benchmark_data_smithi188_113957_object70236:head# 2024-04-23T16:33:56.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573bbad5:::benchmark_data_smithi188_113957_object20023:head# 2024-04-23T16:33:56.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573bf991:::benchmark_data_smithi188_113957_object47843:head# 2024-04-23T16:33:56.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573c445a:::benchmark_data_smithi188_113957_object69168:head# 2024-04-23T16:33:56.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573cc7e2:::benchmark_data_smithi188_113957_object119468:head# 2024-04-23T16:33:56.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573d8d79:::benchmark_data_smithi188_113957_object133726:head# 2024-04-23T16:33:56.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573d92c1:::benchmark_data_smithi188_113957_object113809:head# 2024-04-23T16:33:56.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573e7ac8:::benchmark_data_smithi188_113957_object82743:head# 2024-04-23T16:33:56.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573ea506:::benchmark_data_smithi188_113957_object139064:head# 2024-04-23T16:33:56.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573eba63:::benchmark_data_smithi188_113957_object28873:head# 2024-04-23T16:33:56.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573f0738:::benchmark_data_smithi188_113957_object76644:head# 2024-04-23T16:33:56.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573f285e:::benchmark_data_smithi188_113957_object66376:head# 2024-04-23T16:33:56.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573f9892:::benchmark_data_smithi188_113957_object69318:head# 2024-04-23T16:33:56.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:573fcd11:::benchmark_data_smithi188_113957_object60748:head# 2024-04-23T16:33:56.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57400383:::benchmark_data_smithi188_113957_object133924:head# 2024-04-23T16:33:56.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574085fa:::benchmark_data_smithi188_113957_object135702:head# 2024-04-23T16:33:56.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5740b384:::benchmark_data_smithi188_113957_object77753:head# 2024-04-23T16:33:56.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57414678:::benchmark_data_smithi188_113957_object31540:head# 2024-04-23T16:33:56.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57415aa4:::benchmark_data_smithi188_113957_object7026:head# 2024-04-23T16:33:56.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5741a8fd:::benchmark_data_smithi188_113957_object3175:head# 2024-04-23T16:33:56.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574200c6:::benchmark_data_smithi188_113957_object123416:head# 2024-04-23T16:33:56.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5742d2d3:::benchmark_data_smithi188_113957_object57021:head# 2024-04-23T16:33:56.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5743c050:::benchmark_data_smithi188_113957_object5013:head# 2024-04-23T16:33:56.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5743eba7:::benchmark_data_smithi188_113957_object9084:head# 2024-04-23T16:33:56.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57444e9f:::benchmark_data_smithi188_113957_object33417:head# 2024-04-23T16:33:56.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57446d26:::benchmark_data_smithi188_113957_object26042:head# 2024-04-23T16:33:56.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57448378:::benchmark_data_smithi188_113957_object128724:head# 2024-04-23T16:33:56.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5744951d:::benchmark_data_smithi188_113957_object7775:head# 2024-04-23T16:33:56.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574597e9:::benchmark_data_smithi188_113957_object88757:head# 2024-04-23T16:33:56.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57467633:::benchmark_data_smithi188_113957_object17657:head# 2024-04-23T16:33:56.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5746befc:::benchmark_data_smithi188_113957_object61402:head# 2024-04-23T16:33:56.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5746eac5:::benchmark_data_smithi188_113957_object53990:head# 2024-04-23T16:33:56.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5746f976:::benchmark_data_smithi188_113957_object97823:head# 2024-04-23T16:33:56.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5748455c:::benchmark_data_smithi188_113957_object151844:head# 2024-04-23T16:33:56.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5748e08b:::benchmark_data_smithi188_113957_object105273:head# 2024-04-23T16:33:56.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574948d8:::benchmark_data_smithi188_113957_object130627:head# 2024-04-23T16:33:56.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574961d0:::benchmark_data_smithi188_113957_object35343:head# 2024-04-23T16:33:56.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57496be4:::benchmark_data_smithi188_113957_object27273:head# 2024-04-23T16:33:56.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574ab1b5:::benchmark_data_smithi188_113957_object107590:head# 2024-04-23T16:33:56.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574b54f0:::benchmark_data_smithi188_113957_object24035:head# 2024-04-23T16:33:56.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574ba09a:::benchmark_data_smithi188_113957_object94572:head# 2024-04-23T16:33:56.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574c670e:::benchmark_data_smithi188_113957_object144618:head# 2024-04-23T16:33:56.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574c7db4:::benchmark_data_smithi188_113957_object136513:head# 2024-04-23T16:33:56.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574d6ecd:::benchmark_data_smithi188_113957_object13885:head# 2024-04-23T16:33:56.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574d728a:::benchmark_data_smithi188_113957_object60880:head# 2024-04-23T16:33:56.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574e33c7:::benchmark_data_smithi188_113957_object148799:head# 2024-04-23T16:33:56.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574e6153:::benchmark_data_smithi188_113957_object39319:head# 2024-04-23T16:33:56.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574e61fc:::benchmark_data_smithi188_113957_object38928:head# 2024-04-23T16:33:56.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574e8dd3:::benchmark_data_smithi188_113957_object11397:head# 2024-04-23T16:33:56.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574ec714:::benchmark_data_smithi188_113957_object44133:head# 2024-04-23T16:33:56.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574ed7c3:::benchmark_data_smithi188_113957_object128428:head# 2024-04-23T16:33:56.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574ee984:::benchmark_data_smithi188_113957_object40757:head# 2024-04-23T16:33:56.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574f569e:::benchmark_data_smithi188_113957_object15010:head# 2024-04-23T16:33:56.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574f59ac:::benchmark_data_smithi188_113957_object88561:head# 2024-04-23T16:33:56.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574fb635:::benchmark_data_smithi188_113957_object38743:head# 2024-04-23T16:33:56.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:574fd865:::benchmark_data_smithi188_113957_object55330:head# 2024-04-23T16:33:56.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5750c0bc:::benchmark_data_smithi188_113957_object56642:head# 2024-04-23T16:33:56.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5750df0a:::benchmark_data_smithi188_113957_object14406:head# 2024-04-23T16:33:56.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:575165f4:::benchmark_data_smithi188_113957_object13975:head# 2024-04-23T16:33:56.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57518189:::benchmark_data_smithi188_113957_object40078:head# 2024-04-23T16:33:56.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5751adef:::benchmark_data_smithi188_113957_object107641:head# 2024-04-23T16:33:56.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57538c9b:::benchmark_data_smithi188_113957_object113234:head# 2024-04-23T16:33:56.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:575413e4:::benchmark_data_smithi188_113957_object10924:head# 2024-04-23T16:33:56.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5754236f:::benchmark_data_smithi188_113957_object136414:head# 2024-04-23T16:33:56.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5754c0fb:::benchmark_data_smithi188_113957_object78277:head# 2024-04-23T16:33:56.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5754c710:::benchmark_data_smithi188_113957_object102629:head# 2024-04-23T16:33:56.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5754d1b0:::benchmark_data_smithi188_113957_object4656:head# 2024-04-23T16:33:56.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5755592d:::benchmark_data_smithi188_113957_object68392:head# 2024-04-23T16:33:56.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:575587f2:::benchmark_data_smithi188_113957_object29861:head# 2024-04-23T16:33:56.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5755d132:::benchmark_data_smithi188_113957_object113977:head# 2024-04-23T16:33:56.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5755fef8:::benchmark_data_smithi188_113957_object71438:head# 2024-04-23T16:33:56.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57560c53:::benchmark_data_smithi188_113957_object130175:head# 2024-04-23T16:33:56.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:575681b3:::benchmark_data_smithi188_113957_object107999:head# 2024-04-23T16:33:56.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5756823d:::benchmark_data_smithi188_113957_object74540:head# 2024-04-23T16:33:56.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5756f844:::benchmark_data_smithi188_113957_object93410:head# 2024-04-23T16:33:56.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:575733bb:::benchmark_data_smithi188_113957_object79934:head# 2024-04-23T16:33:56.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57582abd:::benchmark_data_smithi188_113957_object71902:head# 2024-04-23T16:33:56.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57597fd3:::benchmark_data_smithi188_113957_object1792:head# 2024-04-23T16:33:56.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5759b67c:::benchmark_data_smithi188_113957_object111301:head# 2024-04-23T16:33:56.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:575a873a:::benchmark_data_smithi188_113957_object70954:head# 2024-04-23T16:33:56.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:575a94ba:::benchmark_data_smithi188_113957_object110102:head# 2024-04-23T16:33:56.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:575b1c2f:::benchmark_data_smithi188_113957_object124070:head# 2024-04-23T16:33:56.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:575b9d8e:::benchmark_data_smithi188_113957_object6861:head# 2024-04-23T16:33:56.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:575beb5d:::benchmark_data_smithi188_113957_object94881:head# 2024-04-23T16:33:56.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:575c701a:::benchmark_data_smithi188_113957_object127091:head# 2024-04-23T16:33:56.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:575d225f:::benchmark_data_smithi188_113957_object97135:head# 2024-04-23T16:33:56.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:575d2717:::benchmark_data_smithi188_113957_object57620:head# 2024-04-23T16:33:56.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:575d65b7:::benchmark_data_smithi188_113957_object2521:head# 2024-04-23T16:33:56.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:575dcf28:::benchmark_data_smithi188_113957_object71247:head# 2024-04-23T16:33:56.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:575e71bd:::benchmark_data_smithi188_113957_object65643:head# 2024-04-23T16:33:56.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:575ee3ea:::benchmark_data_smithi188_113957_object53080:head# 2024-04-23T16:33:56.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:575f1573:::benchmark_data_smithi188_113957_object79702:head# 2024-04-23T16:33:56.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57604e42:::benchmark_data_smithi188_113957_object64368:head# 2024-04-23T16:33:56.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576092d0:::benchmark_data_smithi188_113957_object120956:head# 2024-04-23T16:33:56.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57609872:::benchmark_data_smithi188_113957_object23271:head# 2024-04-23T16:33:56.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5760c71b:::benchmark_data_smithi188_113957_object146507:head# 2024-04-23T16:33:56.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5760c80c:::benchmark_data_smithi188_113957_object76401:head# 2024-04-23T16:33:56.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5760d3a9:::benchmark_data_smithi188_113957_object120422:head# 2024-04-23T16:33:56.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576237d4:::benchmark_data_smithi188_113957_object148143:head# 2024-04-23T16:33:56.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57624bd3:::benchmark_data_smithi188_113957_object35788:head# 2024-04-23T16:33:56.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57626730:::benchmark_data_smithi188_113957_object78833:head# 2024-04-23T16:33:56.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5763b539:::benchmark_data_smithi188_113957_object79210:head# 2024-04-23T16:33:56.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57644b07:::benchmark_data_smithi188_113957_object98132:head# 2024-04-23T16:33:56.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57647693:::benchmark_data_smithi188_113957_object93262:head# 2024-04-23T16:33:56.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5764c2d8:::benchmark_data_smithi188_113957_object93955:head# 2024-04-23T16:33:56.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5765bd9c:::benchmark_data_smithi188_113957_object127953:head# 2024-04-23T16:33:56.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57662e9e:::benchmark_data_smithi188_113957_object141441:head# 2024-04-23T16:33:56.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5766b551:::benchmark_data_smithi188_113957_object33895:head# 2024-04-23T16:33:56.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57678ac1:::benchmark_data_smithi188_113957_object21480:head# 2024-04-23T16:33:56.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5767d771:::benchmark_data_smithi188_113957_object126587:head# 2024-04-23T16:33:56.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5767e808:::benchmark_data_smithi188_113957_object129221:head# 2024-04-23T16:33:56.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57685dc5:::benchmark_data_smithi188_113957_object11845:head# 2024-04-23T16:33:56.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5768afae:::benchmark_data_smithi188_113957_object128236:head# 2024-04-23T16:33:56.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576915eb:::benchmark_data_smithi188_113957_object73218:head# 2024-04-23T16:33:56.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576920ff:::benchmark_data_smithi188_113957_object36090:head# 2024-04-23T16:33:56.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576939a5:::benchmark_data_smithi188_113957_object134339:head# 2024-04-23T16:33:56.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5769e2ac:::benchmark_data_smithi188_113957_object143858:head# 2024-04-23T16:33:56.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5769e691:::benchmark_data_smithi188_113957_object18301:head# 2024-04-23T16:33:56.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576aa988:::benchmark_data_smithi188_113957_object1435:head# 2024-04-23T16:33:56.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576ab6ab:::benchmark_data_smithi188_113957_object94081:head# 2024-04-23T16:33:56.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576b5186:::benchmark_data_smithi188_113957_object76847:head# 2024-04-23T16:33:56.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576bc57d:::benchmark_data_smithi188_113957_object141822:head# 2024-04-23T16:33:56.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576c29f3:::benchmark_data_smithi188_113957_object36397:head# 2024-04-23T16:33:56.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576c42ea:::benchmark_data_smithi188_113957_object71766:head# 2024-04-23T16:33:56.218 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576c5f07:::benchmark_data_smithi188_113957_object59256:head# 2024-04-23T16:33:56.218 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576cace4:::benchmark_data_smithi188_113957_object122299:head# 2024-04-23T16:33:56.218 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576cce04:::benchmark_data_smithi188_113957_object125100:head# 2024-04-23T16:33:56.218 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576cd365:::benchmark_data_smithi188_113957_object145555:head# 2024-04-23T16:33:56.218 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576d65db:::benchmark_data_smithi188_113957_object1593:head# 2024-04-23T16:33:56.218 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576d813a:::benchmark_data_smithi188_113957_object78944:head# 2024-04-23T16:33:56.218 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576d8f7c:::benchmark_data_smithi188_113957_object1243:head# 2024-04-23T16:33:56.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576e517a:::benchmark_data_smithi188_113957_object137241:head# 2024-04-23T16:33:56.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576e8764:::benchmark_data_smithi188_113957_object89980:head# 2024-04-23T16:33:56.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576fa95f:::benchmark_data_smithi188_113957_object153295:head# 2024-04-23T16:33:56.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:576fb195:::benchmark_data_smithi188_113957_object84979:head# 2024-04-23T16:33:56.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57708a3a:::benchmark_data_smithi188_113957_object4146:head# 2024-04-23T16:33:56.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5770adb6:::benchmark_data_smithi188_113957_object52041:head# 2024-04-23T16:33:56.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5770af9e:::benchmark_data_smithi188_113957_object43843:head# 2024-04-23T16:33:56.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5770f897:::benchmark_data_smithi188_113957_object32921:head# 2024-04-23T16:33:56.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57711e26:::benchmark_data_smithi188_113957_object31121:head# 2024-04-23T16:33:56.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57718c06:::benchmark_data_smithi188_113957_object57730:head# 2024-04-23T16:33:56.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5771e1f0:::benchmark_data_smithi188_113957_object113014:head# 2024-04-23T16:33:56.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57723bde:::benchmark_data_smithi188_113957_object102816:head# 2024-04-23T16:33:56.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57733afd:::benchmark_data_smithi188_113957_object30177:head# 2024-04-23T16:33:56.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57738630:::benchmark_data_smithi188_113957_object135773:head# 2024-04-23T16:33:56.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5773d475:::benchmark_data_smithi188_113957_object51844:head# 2024-04-23T16:33:56.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57748b39:::benchmark_data_smithi188_113957_object74316:head# 2024-04-23T16:33:56.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57751892:::benchmark_data_smithi188_113957_object114464:head# 2024-04-23T16:33:56.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57752883:::benchmark_data_smithi188_113957_object148873:head# 2024-04-23T16:33:56.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57756b8b:::benchmark_data_smithi188_113957_object48563:head# 2024-04-23T16:33:56.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5776d2d9:::benchmark_data_smithi188_113957_object80122:head# 2024-04-23T16:33:56.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:577772e0:::benchmark_data_smithi188_113957_object15333:head# 2024-04-23T16:33:56.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5777a242:::benchmark_data_smithi188_113957_object96214:head# 2024-04-23T16:33:56.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5777e8e4:::benchmark_data_smithi188_113957_object34794:head# 2024-04-23T16:33:56.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:577827b3:::benchmark_data_smithi188_113957_object2750:head# 2024-04-23T16:33:56.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57783dfd:::benchmark_data_smithi188_113957_object1146:head# 2024-04-23T16:33:56.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57789957:::benchmark_data_smithi188_113957_object2722:head# 2024-04-23T16:33:56.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5778f0c6:::benchmark_data_smithi188_113957_object35975:head# 2024-04-23T16:33:56.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:577923ec:::benchmark_data_smithi188_113957_object65536:head# 2024-04-23T16:33:56.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5779337d:::benchmark_data_smithi188_113957_object77268:head# 2024-04-23T16:33:56.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57794f9c:::benchmark_data_smithi188_113957_object83309:head# 2024-04-23T16:33:56.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5779f19a:::benchmark_data_smithi188_113957_object146373:head# 2024-04-23T16:33:56.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:577a0dbc:::benchmark_data_smithi188_113957_object144529:head# 2024-04-23T16:33:56.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:577b08df:::benchmark_data_smithi188_113957_object49044:head# 2024-04-23T16:33:56.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:577b27b7:::benchmark_data_smithi188_113957_object45648:head# 2024-04-23T16:33:56.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:577bd70d:::benchmark_data_smithi188_113957_object45380:head# 2024-04-23T16:33:56.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:577bebc5:::benchmark_data_smithi188_113957_object26355:head# 2024-04-23T16:33:56.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:577c431d:::benchmark_data_smithi188_113957_object892:head# 2024-04-23T16:33:56.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:577c54eb:::benchmark_data_smithi188_113957_object67468:head# 2024-04-23T16:33:56.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:577cffda:::benchmark_data_smithi188_113957_object35870:head# 2024-04-23T16:33:56.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:577e48e5:::benchmark_data_smithi188_113957_object107158:head# 2024-04-23T16:33:56.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:577eb984:::benchmark_data_smithi188_113957_object81867:head# 2024-04-23T16:33:56.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:577f419f:::benchmark_data_smithi188_113957_object102873:head# 2024-04-23T16:33:56.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:577f7297:::benchmark_data_smithi188_113957_object100404:head# 2024-04-23T16:33:56.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57800fe3:::benchmark_data_smithi188_113957_object62014:head# 2024-04-23T16:33:56.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5780d1e0:::benchmark_data_smithi188_113957_object144614:head# 2024-04-23T16:33:56.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5781af37:::benchmark_data_smithi188_113957_object108487:head# 2024-04-23T16:33:56.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5781beb9:::benchmark_data_smithi188_113957_object91074:head# 2024-04-23T16:33:56.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5781cc30:::benchmark_data_smithi188_113957_object28385:head# 2024-04-23T16:33:56.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5781ead5:::benchmark_data_smithi188_113957_object31748:head# 2024-04-23T16:33:56.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5781ff72:::benchmark_data_smithi188_113957_object12625:head# 2024-04-23T16:33:56.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57822a52:::benchmark_data_smithi188_113957_object104738:head# 2024-04-23T16:33:56.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57828cb5:::benchmark_data_smithi188_113957_object12640:head# 2024-04-23T16:33:56.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5782fd44:::benchmark_data_smithi188_113957_object138495:head# 2024-04-23T16:33:56.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57833877:::benchmark_data_smithi188_113957_object26513:head# 2024-04-23T16:33:56.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57849a9d:::benchmark_data_smithi188_113957_object16405:head# 2024-04-23T16:33:56.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5784ac91:::benchmark_data_smithi188_113957_object130320:head# 2024-04-23T16:33:56.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5784c1ff:::benchmark_data_smithi188_113957_object48353:head# 2024-04-23T16:33:56.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5785c62f:::benchmark_data_smithi188_113957_object126519:head# 2024-04-23T16:33:56.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5785e53a:::benchmark_data_smithi188_113957_object42870:head# 2024-04-23T16:33:56.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57861a39:::benchmark_data_smithi188_113957_object87066:head# 2024-04-23T16:33:56.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5786e821:::benchmark_data_smithi188_113957_object5885:head# 2024-04-23T16:33:56.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5786faab:::benchmark_data_smithi188_113957_object46023:head# 2024-04-23T16:33:56.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57870230:::benchmark_data_smithi188_113957_object53857:head# 2024-04-23T16:33:56.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5787ad1f:::benchmark_data_smithi188_113957_object53409:head# 2024-04-23T16:33:56.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5788b2d8:::benchmark_data_smithi188_113957_object5199:head# 2024-04-23T16:33:56.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5788c7cd:::benchmark_data_smithi188_113957_object30484:head# 2024-04-23T16:33:56.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5788f1e3:::benchmark_data_smithi188_113957_object62889:head# 2024-04-23T16:33:56.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57891f83:::benchmark_data_smithi188_113957_object100213:head# 2024-04-23T16:33:56.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:578a2af3:::benchmark_data_smithi188_113957_object34799:head# 2024-04-23T16:33:56.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:578a828d:::benchmark_data_smithi188_113957_object115288:head# 2024-04-23T16:33:56.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:578a9b96:::benchmark_data_smithi188_113957_object79996:head# 2024-04-23T16:33:56.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:578b5622:::benchmark_data_smithi188_113957_object150796:head# 2024-04-23T16:33:56.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:578ba0e9:::benchmark_data_smithi188_113957_object123749:head# 2024-04-23T16:33:56.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:578c1de8:::benchmark_data_smithi188_113957_object78804:head# 2024-04-23T16:33:56.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:578cb083:::benchmark_data_smithi188_113957_object147437:head# 2024-04-23T16:33:56.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:578d62a6:::benchmark_data_smithi188_113957_object61152:head# 2024-04-23T16:33:56.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:578eb74a:::benchmark_data_smithi188_113957_object54896:head# 2024-04-23T16:33:56.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:578f66e9:::benchmark_data_smithi188_113957_object96301:head# 2024-04-23T16:33:56.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:578f9ef0:::benchmark_data_smithi188_113957_object89774:head# 2024-04-23T16:33:56.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:578fc749:::benchmark_data_smithi188_113957_object17013:head# 2024-04-23T16:33:56.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57905a5c:::benchmark_data_smithi188_113957_object141788:head# 2024-04-23T16:33:56.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5790cd4b:::benchmark_data_smithi188_113957_object75798:head# 2024-04-23T16:33:56.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5791091e:::benchmark_data_smithi188_113957_object142461:head# 2024-04-23T16:33:56.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5791a1cc:::benchmark_data_smithi188_113957_object17720:head# 2024-04-23T16:33:56.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57932d67:::benchmark_data_smithi188_113957_object66643:head# 2024-04-23T16:33:56.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57940ae1:::benchmark_data_smithi188_113957_object63507:head# 2024-04-23T16:33:56.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5794bce5:::benchmark_data_smithi188_113957_object149573:head# 2024-04-23T16:33:56.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5794de8c:::benchmark_data_smithi188_113957_object134441:head# 2024-04-23T16:33:56.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5795430c:::benchmark_data_smithi188_113957_object136821:head# 2024-04-23T16:33:56.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5795870f:::benchmark_data_smithi188_113957_object135692:head# 2024-04-23T16:33:56.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57958fda:::benchmark_data_smithi188_113957_object57376:head# 2024-04-23T16:33:56.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5795a392:::benchmark_data_smithi188_113957_object49596:head# 2024-04-23T16:33:56.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57963525:::benchmark_data_smithi188_113957_object74445:head# 2024-04-23T16:33:56.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57973d59:::benchmark_data_smithi188_113957_object84324:head# 2024-04-23T16:33:56.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57977c56:::benchmark_data_smithi188_113957_object47689:head# 2024-04-23T16:33:56.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5798e821:::benchmark_data_smithi188_113957_object56536:head# 2024-04-23T16:33:56.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57992907:::benchmark_data_smithi188_113957_object98601:head# 2024-04-23T16:33:56.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5799a1f2:::benchmark_data_smithi188_113957_object111743:head# 2024-04-23T16:33:56.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5799a29a:::benchmark_data_smithi188_113957_object79361:head# 2024-04-23T16:33:56.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5799f954:::benchmark_data_smithi188_113957_object77210:head# 2024-04-23T16:33:56.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:579c798b:::benchmark_data_smithi188_113957_object110564:head# 2024-04-23T16:33:56.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:579dc720:::benchmark_data_smithi188_113957_object79551:head# 2024-04-23T16:33:56.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:579e2120:::benchmark_data_smithi188_113957_object91216:head# 2024-04-23T16:33:56.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:579fb8cd:::benchmark_data_smithi188_113957_object25689:head# 2024-04-23T16:33:56.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:579fdab8:::benchmark_data_smithi188_113957_object108600:head# 2024-04-23T16:33:56.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a043fe:::benchmark_data_smithi188_113957_object108051:head# 2024-04-23T16:33:56.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a04605:::benchmark_data_smithi188_113957_object32247:head# 2024-04-23T16:33:56.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a06aa3:::benchmark_data_smithi188_113957_object20344:head# 2024-04-23T16:33:56.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a0ed07:::benchmark_data_smithi188_113957_object113169:head# 2024-04-23T16:33:56.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a17307:::benchmark_data_smithi188_113957_object66843:head# 2024-04-23T16:33:56.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a178f7:::benchmark_data_smithi188_113957_object24159:head# 2024-04-23T16:33:56.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a297f6:::benchmark_data_smithi188_113957_object110284:head# 2024-04-23T16:33:56.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a2eedf:::benchmark_data_smithi188_113957_object111233:head# 2024-04-23T16:33:56.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a356ba:::benchmark_data_smithi188_113957_object100979:head# 2024-04-23T16:33:56.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a457a1:::benchmark_data_smithi188_113957_object26801:head# 2024-04-23T16:33:56.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a58cd7:::benchmark_data_smithi188_113957_object129495:head# 2024-04-23T16:33:56.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a5c7ce:::benchmark_data_smithi188_113957_object76297:head# 2024-04-23T16:33:56.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a5ca32:::benchmark_data_smithi188_113957_object16132:head# 2024-04-23T16:33:56.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a5dd53:::benchmark_data_smithi188_113957_object151466:head# 2024-04-23T16:33:56.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a5e276:::benchmark_data_smithi188_113957_object140931:head# 2024-04-23T16:33:56.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a69639:::benchmark_data_smithi188_113957_object52804:head# 2024-04-23T16:33:56.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a6e3b9:::benchmark_data_smithi188_113957_object93127:head# 2024-04-23T16:33:56.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a747c4:::benchmark_data_smithi188_113957_object130440:head# 2024-04-23T16:33:56.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a88a3b:::benchmark_data_smithi188_113957_object84470:head# 2024-04-23T16:33:56.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a8fd61:::benchmark_data_smithi188_113957_object144084:head# 2024-04-23T16:33:56.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57a9cb4b:::benchmark_data_smithi188_113957_object132979:head# 2024-04-23T16:33:56.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57aa10c3:::benchmark_data_smithi188_113957_object5803:head# 2024-04-23T16:33:56.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57aa7dde:::benchmark_data_smithi188_113957_object76248:head# 2024-04-23T16:33:56.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ab2d5d:::benchmark_data_smithi188_113957_object424:head# 2024-04-23T16:33:56.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ab3e02:::benchmark_data_smithi188_113957_object4227:head# 2024-04-23T16:33:56.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ab4102:::benchmark_data_smithi188_113957_object119764:head# 2024-04-23T16:33:56.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ab6c03:::benchmark_data_smithi188_113957_object71570:head# 2024-04-23T16:33:56.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57abab2a:::benchmark_data_smithi188_113957_object129316:head# 2024-04-23T16:33:56.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57abb311:::benchmark_data_smithi188_113957_object21:head# 2024-04-23T16:33:56.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57abc94e:::benchmark_data_smithi188_113957_object1144:head# 2024-04-23T16:33:56.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ac60f6:::benchmark_data_smithi188_113957_object60584:head# 2024-04-23T16:33:56.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ac98a9:::benchmark_data_smithi188_113957_object74796:head# 2024-04-23T16:33:56.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ad0894:::benchmark_data_smithi188_113957_object142586:head# 2024-04-23T16:33:56.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ad0a3a:::benchmark_data_smithi188_113957_object116943:head# 2024-04-23T16:33:56.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ad2b28:::benchmark_data_smithi188_113957_object90029:head# 2024-04-23T16:33:56.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ad75a4:::benchmark_data_smithi188_113957_object111682:head# 2024-04-23T16:33:56.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57adda78:::benchmark_data_smithi188_113957_object75082:head# 2024-04-23T16:33:56.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57adfa39:::benchmark_data_smithi188_113957_object77000:head# 2024-04-23T16:33:56.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ae7730:::benchmark_data_smithi188_113957_object81346:head# 2024-04-23T16:33:56.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57aeabeb:::benchmark_data_smithi188_113957_object69898:head# 2024-04-23T16:33:56.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57aee28a:::benchmark_data_smithi188_113957_object4301:head# 2024-04-23T16:33:56.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57af2165:::benchmark_data_smithi188_113957_object102888:head# 2024-04-23T16:33:56.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57af65fd:::benchmark_data_smithi188_113957_object149208:head# 2024-04-23T16:33:56.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57af760d:::benchmark_data_smithi188_113957_object117646:head# 2024-04-23T16:33:56.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b08d4f:::benchmark_data_smithi188_113957_object99558:head# 2024-04-23T16:33:56.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b164fd:::benchmark_data_smithi188_113957_object139017:head# 2024-04-23T16:33:56.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b17856:::benchmark_data_smithi188_113957_object91767:head# 2024-04-23T16:33:56.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b245a9:::benchmark_data_smithi188_113957_object56912:head# 2024-04-23T16:33:56.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b251a3:::benchmark_data_smithi188_113957_object22063:head# 2024-04-23T16:33:56.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b2606c:::benchmark_data_smithi188_113957_object126586:head# 2024-04-23T16:33:56.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b27bf6:::benchmark_data_smithi188_113957_object895:head# 2024-04-23T16:33:56.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b2cebe:::benchmark_data_smithi188_113957_object116014:head# 2024-04-23T16:33:56.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b2e1f1:::benchmark_data_smithi188_113957_object149791:head# 2024-04-23T16:33:56.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b3e995:::benchmark_data_smithi188_113957_object51276:head# 2024-04-23T16:33:56.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b45d70:::benchmark_data_smithi188_113957_object97772:head# 2024-04-23T16:33:56.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b49cc3:::benchmark_data_smithi188_113957_object56913:head# 2024-04-23T16:33:56.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b530ad:::benchmark_data_smithi188_113957_object126714:head# 2024-04-23T16:33:56.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b5a5ab:::benchmark_data_smithi188_113957_object73961:head# 2024-04-23T16:33:56.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b67d67:::benchmark_data_smithi188_113957_object128597:head# 2024-04-23T16:33:56.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b6c087:::benchmark_data_smithi188_113957_object20066:head# 2024-04-23T16:33:56.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b71219:::benchmark_data_smithi188_113957_object136500:head# 2024-04-23T16:33:56.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b7dd48:::benchmark_data_smithi188_113957_object7063:head# 2024-04-23T16:33:56.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b9235c:::benchmark_data_smithi188_113957_object2594:head# 2024-04-23T16:33:56.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b93705:::benchmark_data_smithi188_113957_object129303:head# 2024-04-23T16:33:56.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b974e1:::benchmark_data_smithi188_113957_object25396:head# 2024-04-23T16:33:56.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57b99c6f:::benchmark_data_smithi188_113957_object89147:head# 2024-04-23T16:33:56.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ba06eb:::benchmark_data_smithi188_113957_object51791:head# 2024-04-23T16:33:56.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57bb875a:::benchmark_data_smithi188_113957_object91097:head# 2024-04-23T16:33:56.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57bbbaf5:::benchmark_data_smithi188_113957_object92126:head# 2024-04-23T16:33:56.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57bc22a3:::benchmark_data_smithi188_113957_object150987:head# 2024-04-23T16:33:56.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57bc5b13:::benchmark_data_smithi188_113957_object67057:head# 2024-04-23T16:33:56.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57be0a22:::benchmark_data_smithi188_113957_object48818:head# 2024-04-23T16:33:56.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57be5104:::benchmark_data_smithi188_113957_object131433:head# 2024-04-23T16:33:56.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57be8918:::benchmark_data_smithi188_113957_object152016:head# 2024-04-23T16:33:56.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57be8f92:::benchmark_data_smithi188_113957_object33379:head# 2024-04-23T16:33:56.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57bf2c39:::benchmark_data_smithi188_113957_object82363:head# 2024-04-23T16:33:56.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57bf90ad:::benchmark_data_smithi188_113957_object71423:head# 2024-04-23T16:33:56.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c05a40:::benchmark_data_smithi188_113957_object152312:head# 2024-04-23T16:33:56.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c13201:::benchmark_data_smithi188_113957_object102946:head# 2024-04-23T16:33:56.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c1ce1a:::benchmark_data_smithi188_113957_object41707:head# 2024-04-23T16:33:56.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c230d2:::benchmark_data_smithi188_113957_object106657:head# 2024-04-23T16:33:56.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c25e96:::benchmark_data_smithi188_113957_object107396:head# 2024-04-23T16:33:56.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c2780f:::benchmark_data_smithi188_113957_object98962:head# 2024-04-23T16:33:56.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c31f05:::benchmark_data_smithi188_113957_object91654:head# 2024-04-23T16:33:56.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c3849a:::benchmark_data_smithi188_113957_object114281:head# 2024-04-23T16:33:56.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c3a392:::benchmark_data_smithi188_113957_object52365:head# 2024-04-23T16:33:56.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c3e65a:::benchmark_data_smithi188_113957_object51447:head# 2024-04-23T16:33:56.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c4683b:::benchmark_data_smithi188_113957_object96183:head# 2024-04-23T16:33:56.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c4de45:::benchmark_data_smithi188_113957_object46827:head# 2024-04-23T16:33:56.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c537fd:::benchmark_data_smithi188_113957_object95605:head# 2024-04-23T16:33:56.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c5f23d:::benchmark_data_smithi188_113957_object64769:head# 2024-04-23T16:33:56.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c61ba4:::benchmark_data_smithi188_113957_object142000:head# 2024-04-23T16:33:56.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c698fb:::benchmark_data_smithi188_113957_object114527:head# 2024-04-23T16:33:56.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c6c264:::benchmark_data_smithi188_113957_object16663:head# 2024-04-23T16:33:56.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c852ad:::benchmark_data_smithi188_113957_object126120:head# 2024-04-23T16:33:56.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c8c5f6:::benchmark_data_smithi188_113957_object138978:head# 2024-04-23T16:33:56.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57c9b11c:::benchmark_data_smithi188_113957_object101125:head# 2024-04-23T16:33:56.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ca2be2:::benchmark_data_smithi188_113957_object61909:head# 2024-04-23T16:33:56.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ca5213:::benchmark_data_smithi188_113957_object75202:head# 2024-04-23T16:33:56.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57caff50:::benchmark_data_smithi188_113957_object103273:head# 2024-04-23T16:33:56.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57cbeca1:::benchmark_data_smithi188_113957_object130465:head# 2024-04-23T16:33:56.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ccc64f:::benchmark_data_smithi188_113957_object2952:head# 2024-04-23T16:33:56.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57cd1d0f:::benchmark_data_smithi188_113957_object90593:head# 2024-04-23T16:33:56.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57cd4e74:::benchmark_data_smithi188_113957_object49423:head# 2024-04-23T16:33:56.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57cdbdf6:::benchmark_data_smithi188_113957_object82469:head# 2024-04-23T16:33:56.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ce333e:::benchmark_data_smithi188_113957_object102329:head# 2024-04-23T16:33:56.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ce512e:::benchmark_data_smithi188_113957_object67211:head# 2024-04-23T16:33:56.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ced4c1:::benchmark_data_smithi188_113957_object145106:head# 2024-04-23T16:33:56.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57cf5d23:::benchmark_data_smithi188_113957_object112546:head# 2024-04-23T16:33:56.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57d04835:::benchmark_data_smithi188_113957_object13656:head# 2024-04-23T16:33:56.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57d08a97:::benchmark_data_smithi188_113957_object15849:head# 2024-04-23T16:33:56.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57d0daf9:::benchmark_data_smithi188_113957_object123596:head# 2024-04-23T16:33:56.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57d1601d:::benchmark_data_smithi188_113957_object85424:head# 2024-04-23T16:33:56.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57d215c6:::benchmark_data_smithi188_113957_object118040:head# 2024-04-23T16:33:56.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57d275d5:::benchmark_data_smithi188_113957_object30166:head# 2024-04-23T16:33:56.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57d2e470:::benchmark_data_smithi188_113957_object38281:head# 2024-04-23T16:33:56.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57d30d2e:::benchmark_data_smithi188_113957_object21741:head# 2024-04-23T16:33:56.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57d36aa0:::benchmark_data_smithi188_113957_object38139:head# 2024-04-23T16:33:56.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57d37efd:::benchmark_data_smithi188_113957_object127480:head# 2024-04-23T16:33:56.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57d3d442:::benchmark_data_smithi188_113957_object44502:head# 2024-04-23T16:33:56.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57d67768:::benchmark_data_smithi188_113957_object105288:head# 2024-04-23T16:33:56.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57d67b07:::benchmark_data_smithi188_113957_object82989:head# 2024-04-23T16:33:56.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57d7e883:::benchmark_data_smithi188_113957_object255:head# 2024-04-23T16:33:56.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57d80ca5:::benchmark_data_smithi188_113957_object18360:head# 2024-04-23T16:33:56.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57d86933:::benchmark_data_smithi188_113957_object133046:head# 2024-04-23T16:33:56.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57d87c63:::benchmark_data_smithi188_113957_object40313:head# 2024-04-23T16:33:56.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57d88791:::benchmark_data_smithi188_113957_object56396:head# 2024-04-23T16:33:56.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57d9e08e:::benchmark_data_smithi188_113957_object60717:head# 2024-04-23T16:33:56.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57da15db:::benchmark_data_smithi188_113957_object103764:head# 2024-04-23T16:33:56.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57dafb31:::benchmark_data_smithi188_113957_object65418:head# 2024-04-23T16:33:56.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57db1827:::benchmark_data_smithi188_113957_object65003:head# 2024-04-23T16:33:56.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57dc42c9:::benchmark_data_smithi188_113957_object3135:head# 2024-04-23T16:33:56.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57dc7932:::benchmark_data_smithi188_113957_object40545:head# 2024-04-23T16:33:56.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57de05ae:::benchmark_data_smithi188_113957_object145781:head# 2024-04-23T16:33:56.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57de5fc9:::benchmark_data_smithi188_113957_object131961:head# 2024-04-23T16:33:56.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57de7612:::benchmark_data_smithi188_113957_object30845:head# 2024-04-23T16:33:56.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57de8dd8:::benchmark_data_smithi188_113957_object19920:head# 2024-04-23T16:33:56.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57deb9a6:::benchmark_data_smithi188_113957_object125686:head# 2024-04-23T16:33:56.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57defb69:::benchmark_data_smithi188_113957_object133679:head# 2024-04-23T16:33:56.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57dfbe06:::benchmark_data_smithi188_113957_object33114:head# 2024-04-23T16:33:56.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57dfef7b:::benchmark_data_smithi188_113957_object119582:head# 2024-04-23T16:33:56.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57e1a4d7:::benchmark_data_smithi188_113957_object85670:head# 2024-04-23T16:33:56.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57e2336d:::benchmark_data_smithi188_113957_object12431:head# 2024-04-23T16:33:56.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57e36b40:::benchmark_data_smithi188_113957_object136672:head# 2024-04-23T16:33:56.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57e3f827:::benchmark_data_smithi188_113957_object47880:head# 2024-04-23T16:33:56.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57e40c40:::benchmark_data_smithi188_113957_object133375:head# 2024-04-23T16:33:56.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57e41e0b:::benchmark_data_smithi188_113957_object130098:head# 2024-04-23T16:33:56.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57e445ec:::benchmark_data_smithi188_113957_object134323:head# 2024-04-23T16:33:56.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57e4645b:::benchmark_data_smithi188_113957_object37168:head# 2024-04-23T16:33:56.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57e5775e:::benchmark_data_smithi188_113957_object75468:head# 2024-04-23T16:33:56.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57e5abd1:::benchmark_data_smithi188_113957_object13147:head# 2024-04-23T16:33:56.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57e6d1f8:::benchmark_data_smithi188_113957_object37039:head# 2024-04-23T16:33:56.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57e6e872:::benchmark_data_smithi188_113957_object15935:head# 2024-04-23T16:33:56.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57e719e6:::benchmark_data_smithi188_113957_object43777:head# 2024-04-23T16:33:56.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57e7271c:::benchmark_data_smithi188_113957_object86503:head# 2024-04-23T16:33:56.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57e7fd99:::benchmark_data_smithi188_113957_object82296:head# 2024-04-23T16:33:56.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57e8a2fb:::benchmark_data_smithi188_113957_object35740:head# 2024-04-23T16:33:56.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57e91521:::benchmark_data_smithi188_113957_object84633:head# 2024-04-23T16:33:56.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ebdce6:::benchmark_data_smithi188_113957_object31488:head# 2024-04-23T16:33:56.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ec5ee9:::benchmark_data_smithi188_113957_object8090:head# 2024-04-23T16:33:56.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ec6ede:::benchmark_data_smithi188_113957_object137742:head# 2024-04-23T16:33:56.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ecfcde:::benchmark_data_smithi188_113957_object55133:head# 2024-04-23T16:33:56.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ee6d36:::benchmark_data_smithi188_113957_object120038:head# 2024-04-23T16:33:56.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57eeaa18:::benchmark_data_smithi188_113957_object9064:head# 2024-04-23T16:33:56.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57eefd5b:::benchmark_data_smithi188_113957_object106010:head# 2024-04-23T16:33:56.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ef3e63:::benchmark_data_smithi188_113957_object82350:head# 2024-04-23T16:33:56.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ef75e0:::benchmark_data_smithi188_113957_object27601:head# 2024-04-23T16:33:56.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ef9395:::benchmark_data_smithi188_113957_object48319:head# 2024-04-23T16:33:56.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f039f9:::benchmark_data_smithi188_113957_object143077:head# 2024-04-23T16:33:56.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f05545:::benchmark_data_smithi188_113957_object84546:head# 2024-04-23T16:33:56.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f14759:::benchmark_data_smithi188_113957_object86881:head# 2024-04-23T16:33:56.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f17121:::benchmark_data_smithi188_113957_object140061:head# 2024-04-23T16:33:56.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f1a29e:::benchmark_data_smithi188_113957_object148861:head# 2024-04-23T16:33:56.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f1bf0a:::benchmark_data_smithi188_113957_object104653:head# 2024-04-23T16:33:56.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f1bf6f:::benchmark_data_smithi188_113957_object114002:head# 2024-04-23T16:33:56.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f202f6:::benchmark_data_smithi188_113957_object19100:head# 2024-04-23T16:33:56.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f21ec1:::benchmark_data_smithi188_113957_object39838:head# 2024-04-23T16:33:56.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f2e3fa:::benchmark_data_smithi188_113957_object27957:head# 2024-04-23T16:33:56.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f375dc:::benchmark_data_smithi188_113957_object101411:head# 2024-04-23T16:33:56.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f436bf:::benchmark_data_smithi188_113957_object70359:head# 2024-04-23T16:33:56.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f482d3:::benchmark_data_smithi188_113957_object28866:head# 2024-04-23T16:33:56.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f486e0:::benchmark_data_smithi188_113957_object105380:head# 2024-04-23T16:33:56.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f50d77:::benchmark_data_smithi188_113957_object39802:head# 2024-04-23T16:33:56.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f51d02:::benchmark_data_smithi188_113957_object98804:head# 2024-04-23T16:33:56.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f5301e:::benchmark_data_smithi188_113957_object17406:head# 2024-04-23T16:33:56.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f67fbd:::benchmark_data_smithi188_113957_object52453:head# 2024-04-23T16:33:56.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f75fae:::benchmark_data_smithi188_113957_object54898:head# 2024-04-23T16:33:56.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f7c79b:::benchmark_data_smithi188_113957_object130693:head# 2024-04-23T16:33:56.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f82ab0:::benchmark_data_smithi188_113957_object7153:head# 2024-04-23T16:33:56.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57f8f06a:::benchmark_data_smithi188_113957_object102492:head# 2024-04-23T16:33:56.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57fa1ca4:::benchmark_data_smithi188_113957_object4637:head# 2024-04-23T16:33:56.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57fa9780:::benchmark_data_smithi188_113957_object46658:head# 2024-04-23T16:33:56.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57fa9bb9:::benchmark_data_smithi188_113957_object49484:head# 2024-04-23T16:33:56.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57fb6af4:::benchmark_data_smithi188_113957_object51260:head# 2024-04-23T16:33:56.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57fb6cd0:::benchmark_data_smithi188_113957_object100951:head# 2024-04-23T16:33:56.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57fbab73:::benchmark_data_smithi188_113957_object26614:head# 2024-04-23T16:33:56.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57fbc8e8:::benchmark_data_smithi188_113957_object109256:head# 2024-04-23T16:33:56.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57fbe8fd:::benchmark_data_smithi188_113957_object42238:head# 2024-04-23T16:33:56.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57fc0ecc:::benchmark_data_smithi188_113957_object37007:head# 2024-04-23T16:33:56.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57fc45e9:::benchmark_data_smithi188_113957_object118297:head# 2024-04-23T16:33:56.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57fc6ca5:::benchmark_data_smithi188_113957_object124862:head# 2024-04-23T16:33:56.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57fcf7bd:::benchmark_data_smithi188_113957_object75467:head# 2024-04-23T16:33:56.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57fd3bdb:::benchmark_data_smithi188_113957_object67939:head# 2024-04-23T16:33:56.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57fd5dbf:::benchmark_data_smithi188_113957_object66270:head# 2024-04-23T16:33:56.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57fd6f1b:::benchmark_data_smithi188_113957_object152791:head# 2024-04-23T16:33:56.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57fda357:::benchmark_data_smithi188_113957_object26099:head# 2024-04-23T16:33:56.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57fe2ff7:::benchmark_data_smithi188_113957_object102844:head# 2024-04-23T16:33:56.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57fe5552:::benchmark_data_smithi188_113957_object47289:head# 2024-04-23T16:33:56.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ff1bbf:::benchmark_data_smithi188_113957_object78181:head# 2024-04-23T16:33:56.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:57ff96ce:::benchmark_data_smithi188_113957_object45361:head# 2024-04-23T16:33:56.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58007301:::benchmark_data_smithi188_113957_object117317:head# 2024-04-23T16:33:56.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5800e7ea:::benchmark_data_smithi188_113957_object108477:head# 2024-04-23T16:33:56.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58018fdd:::benchmark_data_smithi188_113957_object82330:head# 2024-04-23T16:33:56.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5801ae8b:::benchmark_data_smithi188_113957_object133271:head# 2024-04-23T16:33:56.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58023b19:::benchmark_data_smithi188_113957_object31303:head# 2024-04-23T16:33:56.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5803353c:::benchmark_data_smithi188_113957_object77392:head# 2024-04-23T16:33:56.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5803d7d2:::benchmark_data_smithi188_113957_object66605:head# 2024-04-23T16:33:56.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58041828:::benchmark_data_smithi188_113957_object92939:head# 2024-04-23T16:33:56.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5804539c:::benchmark_data_smithi188_113957_object25171:head# 2024-04-23T16:33:56.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58047c1e:::benchmark_data_smithi188_113957_object107288:head# 2024-04-23T16:33:56.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58048d39:::benchmark_data_smithi188_113957_object110654:head# 2024-04-23T16:33:56.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5804b5c2:::benchmark_data_smithi188_113957_object84382:head# 2024-04-23T16:33:56.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5804c544:::benchmark_data_smithi188_113957_object57435:head# 2024-04-23T16:33:56.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58051f9e:::benchmark_data_smithi188_113957_object130734:head# 2024-04-23T16:33:56.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58058395:::benchmark_data_smithi188_113957_object134670:head# 2024-04-23T16:33:56.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58059ddb:::benchmark_data_smithi188_113957_object54014:head# 2024-04-23T16:33:56.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5805fc91:::benchmark_data_smithi188_113957_object27376:head# 2024-04-23T16:33:56.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58060475:::benchmark_data_smithi188_113957_object144556:head# 2024-04-23T16:33:56.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58062a9f:::benchmark_data_smithi188_113957_object30130:head# 2024-04-23T16:33:56.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5807128d:::benchmark_data_smithi188_113957_object142658:head# 2024-04-23T16:33:56.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:580725e4:::benchmark_data_smithi188_113957_object128862:head# 2024-04-23T16:33:56.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58073a08:::benchmark_data_smithi188_113957_object33269:head# 2024-04-23T16:33:56.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58074b16:::benchmark_data_smithi188_113957_object127385:head# 2024-04-23T16:33:56.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:580759aa:::benchmark_data_smithi188_113957_object144945:head# 2024-04-23T16:33:56.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58086069:::benchmark_data_smithi188_113957_object46336:head# 2024-04-23T16:33:56.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:580871d1:::benchmark_data_smithi188_113957_object99121:head# 2024-04-23T16:33:56.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58088108:::benchmark_data_smithi188_113957_object139262:head# 2024-04-23T16:33:56.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58090bcc:::benchmark_data_smithi188_113957_object13812:head# 2024-04-23T16:33:56.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58091cc3:::benchmark_data_smithi188_113957_object101912:head# 2024-04-23T16:33:56.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:580a5034:::benchmark_data_smithi188_113957_object36301:head# 2024-04-23T16:33:56.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:580a9710:::benchmark_data_smithi188_113957_object10377:head# 2024-04-23T16:33:56.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:580b81b7:::benchmark_data_smithi188_113957_object125939:head# 2024-04-23T16:33:56.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:580bfd7d:::benchmark_data_smithi188_113957_object138002:head# 2024-04-23T16:33:56.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:580c56fc:::benchmark_data_smithi188_113957_object144546:head# 2024-04-23T16:33:56.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:580c6a7d:::benchmark_data_smithi188_113957_object61813:head# 2024-04-23T16:33:56.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:580cdc9c:::benchmark_data_smithi188_113957_object68203:head# 2024-04-23T16:33:56.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:580da1c5:::benchmark_data_smithi188_113957_object25127:head# 2024-04-23T16:33:56.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:580e5702:::benchmark_data_smithi188_113957_object42302:head# 2024-04-23T16:33:56.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:580efd60:::benchmark_data_smithi188_113957_object119412:head# 2024-04-23T16:33:56.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:580f1846:::benchmark_data_smithi188_113957_object136632:head# 2024-04-23T16:33:56.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:580f52a1:::benchmark_data_smithi188_113957_object98880:head# 2024-04-23T16:33:56.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:580fc87b:::benchmark_data_smithi188_113957_object130204:head# 2024-04-23T16:33:56.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:580fd78a:::benchmark_data_smithi188_113957_object7334:head# 2024-04-23T16:33:56.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58108348:::benchmark_data_smithi188_113957_object89434:head# 2024-04-23T16:33:56.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5810b06a:::benchmark_data_smithi188_113957_object10560:head# 2024-04-23T16:33:56.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5812a634:::benchmark_data_smithi188_113957_object11267:head# 2024-04-23T16:33:56.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5812caac:::benchmark_data_smithi188_113957_object38440:head# 2024-04-23T16:33:56.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58131a3c:::benchmark_data_smithi188_113957_object108806:head# 2024-04-23T16:33:56.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5813d075:::benchmark_data_smithi188_113957_object101433:head# 2024-04-23T16:33:56.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58142b99:::benchmark_data_smithi188_113957_object23120:head# 2024-04-23T16:33:56.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:581453b8:::benchmark_data_smithi188_113957_object10099:head# 2024-04-23T16:33:56.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5814ad7d:::benchmark_data_smithi188_113957_object84609:head# 2024-04-23T16:33:56.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58151bb6:::benchmark_data_smithi188_113957_object21599:head# 2024-04-23T16:33:56.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5815aba6:::benchmark_data_smithi188_113957_object16613:head# 2024-04-23T16:33:56.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5815afde:::benchmark_data_smithi188_113957_object123217:head# 2024-04-23T16:33:56.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5815e3f6:::benchmark_data_smithi188_113957_object17898:head# 2024-04-23T16:33:56.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5815ea4a:::benchmark_data_smithi188_113957_object153113:head# 2024-04-23T16:33:56.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:581665cc:::benchmark_data_smithi188_113957_object14236:head# 2024-04-23T16:33:56.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58178d81:::benchmark_data_smithi188_113957_object139238:head# 2024-04-23T16:33:56.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5817f235:::benchmark_data_smithi188_113957_object92478:head# 2024-04-23T16:33:56.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:581821e6:::benchmark_data_smithi188_113957_object134642:head# 2024-04-23T16:33:56.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58182565:::benchmark_data_smithi188_113957_object47598:head# 2024-04-23T16:33:56.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58192d41:::benchmark_data_smithi188_113957_object30911:head# 2024-04-23T16:33:56.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58198125:::benchmark_data_smithi188_113957_object31590:head# 2024-04-23T16:33:56.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5819efb1:::benchmark_data_smithi188_113957_object16601:head# 2024-04-23T16:33:56.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:581a1cd2:::benchmark_data_smithi188_113957_object13210:head# 2024-04-23T16:33:56.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:581bb4db:::benchmark_data_smithi188_113957_object106742:head# 2024-04-23T16:33:56.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:581bc38c:::benchmark_data_smithi188_113957_object33736:head# 2024-04-23T16:33:56.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:581bdaa6:::benchmark_data_smithi188_113957_object69246:head# 2024-04-23T16:33:56.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:581dbed1:::benchmark_data_smithi188_113957_object28099:head# 2024-04-23T16:33:56.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:581e016b:::benchmark_data_smithi188_113957_object125551:head# 2024-04-23T16:33:56.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:581f2f71:::benchmark_data_smithi188_113957_object88482:head# 2024-04-23T16:33:56.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58201084:::benchmark_data_smithi188_113957_object90144:head# 2024-04-23T16:33:56.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58202ba5:::benchmark_data_smithi188_113957_object111919:head# 2024-04-23T16:33:56.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58208abe:::benchmark_data_smithi188_113957_object109667:head# 2024-04-23T16:33:56.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5821336a:::benchmark_data_smithi188_113957_object43616:head# 2024-04-23T16:33:56.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5822c6b0:::benchmark_data_smithi188_113957_object92266:head# 2024-04-23T16:33:56.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5823aac0:::benchmark_data_smithi188_113957_object47763:head# 2024-04-23T16:33:56.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58247bc9:::benchmark_data_smithi188_113957_object63953:head# 2024-04-23T16:33:56.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:582553f5:::benchmark_data_smithi188_113957_object22241:head# 2024-04-23T16:33:56.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5825db45:::benchmark_data_smithi188_113957_object30476:head# 2024-04-23T16:33:56.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58261bc6:::benchmark_data_smithi188_113957_object71856:head# 2024-04-23T16:33:56.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5827d3af:::benchmark_data_smithi188_113957_object126919:head# 2024-04-23T16:33:56.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:582872f3:::benchmark_data_smithi188_113957_object140723:head# 2024-04-23T16:33:56.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5828e201:::benchmark_data_smithi188_113957_object81603:head# 2024-04-23T16:33:56.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5829cc2d:::benchmark_data_smithi188_113957_object35184:head# 2024-04-23T16:33:56.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5829fb9e:::benchmark_data_smithi188_113957_object89867:head# 2024-04-23T16:33:56.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:582a5484:::benchmark_data_smithi188_113957_object90194:head# 2024-04-23T16:33:56.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:582a87ea:::benchmark_data_smithi188_113957_object149672:head# 2024-04-23T16:33:56.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:582b0b23:::benchmark_data_smithi188_113957_object122653:head# 2024-04-23T16:33:56.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:582b4ed4:::benchmark_data_smithi188_113957_object4029:head# 2024-04-23T16:33:56.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:582bd1b8:::benchmark_data_smithi188_113957_object91718:head# 2024-04-23T16:33:56.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:582c0844:::benchmark_data_smithi188_113957_object126348:head# 2024-04-23T16:33:56.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:582c6594:::benchmark_data_smithi188_113957_object2752:head# 2024-04-23T16:33:56.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:582d0dd9:::benchmark_data_smithi188_113957_object19533:head# 2024-04-23T16:33:56.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:582d3d49:::benchmark_data_smithi188_113957_object138132:head# 2024-04-23T16:33:56.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:582f9125:::benchmark_data_smithi188_113957_object42857:head# 2024-04-23T16:33:56.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:582fa491:::benchmark_data_smithi188_113957_object142726:head# 2024-04-23T16:33:56.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:583011cc:::benchmark_data_smithi188_113957_object117372:head# 2024-04-23T16:33:56.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58303ddc:::benchmark_data_smithi188_113957_object50942:head# 2024-04-23T16:33:56.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:583072c9:::benchmark_data_smithi188_113957_object61997:head# 2024-04-23T16:33:56.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58309916:::benchmark_data_smithi188_113957_object21902:head# 2024-04-23T16:33:56.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58312adb:::benchmark_data_smithi188_113957_object40382:head# 2024-04-23T16:33:56.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58313b64:::benchmark_data_smithi188_113957_object102160:head# 2024-04-23T16:33:56.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58316c2c:::benchmark_data_smithi188_113957_object96458:head# 2024-04-23T16:33:56.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58322366:::benchmark_data_smithi188_113957_object39602:head# 2024-04-23T16:33:56.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5832c8a6:::benchmark_data_smithi188_113957_object21610:head# 2024-04-23T16:33:56.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5832d169:::benchmark_data_smithi188_113957_object17035:head# 2024-04-23T16:33:56.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58339242:::benchmark_data_smithi188_113957_object83564:head# 2024-04-23T16:33:56.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5833d3f3:::benchmark_data_smithi188_113957_object89528:head# 2024-04-23T16:33:56.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58349524:::benchmark_data_smithi188_113957_object15727:head# 2024-04-23T16:33:56.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5834a379:::benchmark_data_smithi188_113957_object45820:head# 2024-04-23T16:33:56.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:583581b0:::benchmark_data_smithi188_113957_object6360:head# 2024-04-23T16:33:56.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5836b995:::benchmark_data_smithi188_113957_object143734:head# 2024-04-23T16:33:56.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58371423:::benchmark_data_smithi188_113957_object107202:head# 2024-04-23T16:33:56.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58378e5d:::benchmark_data_smithi188_113957_object141020:head# 2024-04-23T16:33:56.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5838e8a6:::benchmark_data_smithi188_113957_object21178:head# 2024-04-23T16:33:56.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58392d37:::benchmark_data_smithi188_113957_object128632:head# 2024-04-23T16:33:56.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58394cbe:::benchmark_data_smithi188_113957_object45170:head# 2024-04-23T16:33:56.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5839b946:::benchmark_data_smithi188_113957_object124617:head# 2024-04-23T16:33:56.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:583a09c8:::benchmark_data_smithi188_113957_object147699:head# 2024-04-23T16:33:56.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:583a239c:::benchmark_data_smithi188_113957_object50613:head# 2024-04-23T16:33:56.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:583a7907:::benchmark_data_smithi188_113957_object104645:head# 2024-04-23T16:33:56.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:583ac4f3:::benchmark_data_smithi188_113957_object25760:head# 2024-04-23T16:33:56.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:583b81f5:::benchmark_data_smithi188_113957_object139450:head# 2024-04-23T16:33:56.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:583b8435:::benchmark_data_smithi188_113957_object135833:head# 2024-04-23T16:33:56.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:583b8f4c:::benchmark_data_smithi188_113957_object100338:head# 2024-04-23T16:33:56.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:583bc214:::benchmark_data_smithi188_113957_object77849:head# 2024-04-23T16:33:56.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:583bd25a:::benchmark_data_smithi188_113957_object73475:head# 2024-04-23T16:33:56.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:583c50dd:::benchmark_data_smithi188_113957_object146749:head# 2024-04-23T16:33:56.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:583d3a4b:::benchmark_data_smithi188_113957_object139607:head# 2024-04-23T16:33:56.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:583d898f:::benchmark_data_smithi188_113957_object11838:head# 2024-04-23T16:33:56.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:583deaf2:::benchmark_data_smithi188_113957_object109343:head# 2024-04-23T16:33:56.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:583e754d:::benchmark_data_smithi188_113957_object26183:head# 2024-04-23T16:33:56.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:583e9e66:::benchmark_data_smithi188_113957_object87134:head# 2024-04-23T16:33:56.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:583ec2d1:::benchmark_data_smithi188_113957_object80468:head# 2024-04-23T16:33:56.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5840c3fa:::benchmark_data_smithi188_113957_object122054:head# 2024-04-23T16:33:56.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5840ec8d:::benchmark_data_smithi188_113957_object19968:head# 2024-04-23T16:33:56.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58419d45:::benchmark_data_smithi188_113957_object104421:head# 2024-04-23T16:33:56.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5841a570:::benchmark_data_smithi188_113957_object129577:head# 2024-04-23T16:33:56.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5841c79e:::benchmark_data_smithi188_113957_object51126:head# 2024-04-23T16:33:56.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58421698:::benchmark_data_smithi188_113957_object136536:head# 2024-04-23T16:33:56.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5843bcd9:::benchmark_data_smithi188_113957_object99912:head# 2024-04-23T16:33:56.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5844aeaf:::benchmark_data_smithi188_113957_object97486:head# 2024-04-23T16:33:56.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5844d8b7:::benchmark_data_smithi188_113957_object48946:head# 2024-04-23T16:33:56.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58455935:::benchmark_data_smithi188_113957_object89437:head# 2024-04-23T16:33:56.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58465884:::benchmark_data_smithi188_113957_object92075:head# 2024-04-23T16:33:56.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5846d6fa:::benchmark_data_smithi188_113957_object30290:head# 2024-04-23T16:33:56.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58471649:::benchmark_data_smithi188_113957_object27401:head# 2024-04-23T16:33:56.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5848aa7b:::benchmark_data_smithi188_113957_object107389:head# 2024-04-23T16:33:56.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5848c663:::benchmark_data_smithi188_113957_object61305:head# 2024-04-23T16:33:56.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584accf7:::benchmark_data_smithi188_113957_object55803:head# 2024-04-23T16:33:56.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584afb58:::benchmark_data_smithi188_113957_object28477:head# 2024-04-23T16:33:56.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584b5ba4:::benchmark_data_smithi188_113957_object55291:head# 2024-04-23T16:33:56.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584ba18c:::benchmark_data_smithi188_113957_object123304:head# 2024-04-23T16:33:56.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584ba2a3:::benchmark_data_smithi188_113957_object143068:head# 2024-04-23T16:33:56.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584be48f:::benchmark_data_smithi188_113957_object22131:head# 2024-04-23T16:33:56.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584c13f0:::benchmark_data_smithi188_113957_object96130:head# 2024-04-23T16:33:56.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584c5d46:::benchmark_data_smithi188_113957_object97079:head# 2024-04-23T16:33:56.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584c888e:::benchmark_data_smithi188_113957_object115598:head# 2024-04-23T16:33:56.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584c9048:::benchmark_data_smithi188_113957_object44929:head# 2024-04-23T16:33:56.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584d3975:::benchmark_data_smithi188_113957_object153497:head# 2024-04-23T16:33:56.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584d569d:::benchmark_data_smithi188_113957_object15911:head# 2024-04-23T16:33:56.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584d713d:::benchmark_data_smithi188_113957_object140392:head# 2024-04-23T16:33:56.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584dc194:::benchmark_data_smithi188_113957_object17266:head# 2024-04-23T16:33:56.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584dc6af:::benchmark_data_smithi188_113957_object107508:head# 2024-04-23T16:33:56.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584dd1a8:::benchmark_data_smithi188_113957_object118392:head# 2024-04-23T16:33:56.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584ded34:::benchmark_data_smithi188_113957_object15978:head# 2024-04-23T16:33:56.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584e438c:::benchmark_data_smithi188_113957_object28969:head# 2024-04-23T16:33:56.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584f0f56:::benchmark_data_smithi188_113957_object145109:head# 2024-04-23T16:33:56.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584f35c4:::benchmark_data_smithi188_113957_object119348:head# 2024-04-23T16:33:56.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584f3746:::benchmark_data_smithi188_113957_object96292:head# 2024-04-23T16:33:56.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:584fe7a4:::benchmark_data_smithi188_113957_object38761:head# 2024-04-23T16:33:56.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58505439:::benchmark_data_smithi188_113957_object53393:head# 2024-04-23T16:33:56.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:585137ec:::benchmark_data_smithi188_113957_object43934:head# 2024-04-23T16:33:56.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58515fa0:::benchmark_data_smithi188_113957_object103268:head# 2024-04-23T16:33:56.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5851c39d:::benchmark_data_smithi188_113957_object91590:head# 2024-04-23T16:33:56.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5852c9be:::benchmark_data_smithi188_113957_object123805:head# 2024-04-23T16:33:56.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:585397ce:::benchmark_data_smithi188_113957_object55640:head# 2024-04-23T16:33:56.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:585427df:::benchmark_data_smithi188_113957_object138897:head# 2024-04-23T16:33:56.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:585452f1:::benchmark_data_smithi188_113957_object46249:head# 2024-04-23T16:33:56.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58546670:::benchmark_data_smithi188_113957_object59847:head# 2024-04-23T16:33:56.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58548d4a:::benchmark_data_smithi188_113957_object38277:head# 2024-04-23T16:33:56.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5854aca5:::benchmark_data_smithi188_113957_object68128:head# 2024-04-23T16:33:56.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58557a88:::benchmark_data_smithi188_113957_object106951:head# 2024-04-23T16:33:56.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:585774e2:::benchmark_data_smithi188_113957_object149260:head# 2024-04-23T16:33:56.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58581d7d:::benchmark_data_smithi188_113957_object135825:head# 2024-04-23T16:33:56.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58587316:::benchmark_data_smithi188_113957_object42987:head# 2024-04-23T16:33:56.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:585891b0:::benchmark_data_smithi188_113957_object107404:head# 2024-04-23T16:33:56.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5858a650:::benchmark_data_smithi188_113957_object143502:head# 2024-04-23T16:33:56.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5859332c:::benchmark_data_smithi188_113957_object81919:head# 2024-04-23T16:33:56.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5859862f:::benchmark_data_smithi188_113957_object115804:head# 2024-04-23T16:33:56.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:585a15b4:::benchmark_data_smithi188_113957_object60875:head# 2024-04-23T16:33:56.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:585a2294:::benchmark_data_smithi188_113957_object72497:head# 2024-04-23T16:33:56.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:585b9dc7:::benchmark_data_smithi188_113957_object65861:head# 2024-04-23T16:33:56.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:585be0dd:::benchmark_data_smithi188_113957_object93292:head# 2024-04-23T16:33:56.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:585c9ba5:::benchmark_data_smithi188_113957_object114339:head# 2024-04-23T16:33:56.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:585cad81:::benchmark_data_smithi188_113957_object82377:head# 2024-04-23T16:33:56.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:585cf7f7:::benchmark_data_smithi188_113957_object64254:head# 2024-04-23T16:33:56.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:585d13a7:::benchmark_data_smithi188_113957_object49018:head# 2024-04-23T16:33:56.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:585de950:::benchmark_data_smithi188_113957_object35250:head# 2024-04-23T16:33:56.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:585f5af0:::benchmark_data_smithi188_113957_object12827:head# 2024-04-23T16:33:56.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:585fdb8c:::benchmark_data_smithi188_113957_object85554:head# 2024-04-23T16:33:56.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58601820:::benchmark_data_smithi188_113957_object63653:head# 2024-04-23T16:33:56.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5860918a:::benchmark_data_smithi188_113957_object95957:head# 2024-04-23T16:33:56.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:586093b3:::benchmark_data_smithi188_113957_object130476:head# 2024-04-23T16:33:56.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:586095b8:::benchmark_data_smithi188_113957_object16401:head# 2024-04-23T16:33:56.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5860b4c2:::benchmark_data_smithi188_113957_object88205:head# 2024-04-23T16:33:56.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5860f334:::benchmark_data_smithi188_113957_object98027:head# 2024-04-23T16:33:56.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:586114d7:::benchmark_data_smithi188_113957_object84341:head# 2024-04-23T16:33:56.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58612b16:::benchmark_data_smithi188_113957_object21507:head# 2024-04-23T16:33:56.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5862a162:::benchmark_data_smithi188_113957_object45227:head# 2024-04-23T16:33:56.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5862d424:::benchmark_data_smithi188_113957_object6743:head# 2024-04-23T16:33:56.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5862f380:::benchmark_data_smithi188_113957_object50668:head# 2024-04-23T16:33:56.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58639bbe:::benchmark_data_smithi188_113957_object35917:head# 2024-04-23T16:33:56.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5863aff9:::benchmark_data_smithi188_113957_object54390:head# 2024-04-23T16:33:56.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5863d825:::benchmark_data_smithi188_113957_object117923:head# 2024-04-23T16:33:56.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5863f1e0:::benchmark_data_smithi188_113957_object7635:head# 2024-04-23T16:33:56.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58647ff5:::benchmark_data_smithi188_113957_object61741:head# 2024-04-23T16:33:56.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5864dc6b:::benchmark_data_smithi188_113957_object101461:head# 2024-04-23T16:33:56.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5864dd97:::benchmark_data_smithi188_113957_object98258:head# 2024-04-23T16:33:56.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5864e178:::benchmark_data_smithi188_113957_object59012:head# 2024-04-23T16:33:56.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58654ef1:::benchmark_data_smithi188_113957_object120300:head# 2024-04-23T16:33:56.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5866b4e9:::benchmark_data_smithi188_113957_object64738:head# 2024-04-23T16:33:56.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5866e76c:::benchmark_data_smithi188_113957_object105861:head# 2024-04-23T16:33:56.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5867b990:::benchmark_data_smithi188_113957_object109976:head# 2024-04-23T16:33:56.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5867f786:::benchmark_data_smithi188_113957_object143911:head# 2024-04-23T16:33:56.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:586807e7:::benchmark_data_smithi188_113957_object6347:head# 2024-04-23T16:33:56.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:586823af:::benchmark_data_smithi188_113957_object103786:head# 2024-04-23T16:33:56.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58686790:::benchmark_data_smithi188_113957_object19807:head# 2024-04-23T16:33:56.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:586969f5:::benchmark_data_smithi188_113957_object57789:head# 2024-04-23T16:33:56.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5869a2d3:::benchmark_data_smithi188_113957_object109327:head# 2024-04-23T16:33:56.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5869b108:::benchmark_data_smithi188_113957_object12507:head# 2024-04-23T16:33:56.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5869c35b:::benchmark_data_smithi188_113957_object46326:head# 2024-04-23T16:33:56.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:586a4499:::benchmark_data_smithi188_113957_object35050:head# 2024-04-23T16:33:56.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:586a8651:::benchmark_data_smithi188_113957_object84268:head# 2024-04-23T16:33:56.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:586b0500:::benchmark_data_smithi188_113957_object101933:head# 2024-04-23T16:33:56.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:586b6591:::benchmark_data_smithi188_113957_object129327:head# 2024-04-23T16:33:56.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:586bf793:::benchmark_data_smithi188_113957_object12798:head# 2024-04-23T16:33:56.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:586c6613:::benchmark_data_smithi188_113957_object32229:head# 2024-04-23T16:33:56.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:586cb5c8:::benchmark_data_smithi188_113957_object619:head# 2024-04-23T16:33:56.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:586db4f6:::benchmark_data_smithi188_113957_object94744:head# 2024-04-23T16:33:56.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:586e8c5d:::benchmark_data_smithi188_113957_object13928:head# 2024-04-23T16:33:56.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:586ebaf6:::benchmark_data_smithi188_113957_object140389:head# 2024-04-23T16:33:56.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:586f885d:::benchmark_data_smithi188_113957_object8289:head# 2024-04-23T16:33:56.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58700eae:::benchmark_data_smithi188_113957_object145867:head# 2024-04-23T16:33:56.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58726dc7:::benchmark_data_smithi188_113957_object84480:head# 2024-04-23T16:33:56.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5872b537:::benchmark_data_smithi188_113957_object99395:head# 2024-04-23T16:33:56.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5872f998:::benchmark_data_smithi188_113957_object49982:head# 2024-04-23T16:33:56.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58730a49:::benchmark_data_smithi188_113957_object65320:head# 2024-04-23T16:33:56.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587327a3:::benchmark_data_smithi188_113957_object39284:head# 2024-04-23T16:33:56.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5873d77a:::benchmark_data_smithi188_113957_object78442:head# 2024-04-23T16:33:56.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587424e8:::benchmark_data_smithi188_113957_object49479:head# 2024-04-23T16:33:56.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58742b1e:::benchmark_data_smithi188_113957_object135267:head# 2024-04-23T16:33:56.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58748fd8:::benchmark_data_smithi188_113957_object133250:head# 2024-04-23T16:33:56.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5874c485:::benchmark_data_smithi188_113957_object18437:head# 2024-04-23T16:33:56.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587575c8:::benchmark_data_smithi188_113957_object132345:head# 2024-04-23T16:33:56.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58760ff6:::benchmark_data_smithi188_113957_object139225:head# 2024-04-23T16:33:56.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587646c4:::benchmark_data_smithi188_113957_object103936:head# 2024-04-23T16:33:56.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5876b04a:::benchmark_data_smithi188_113957_object8722:head# 2024-04-23T16:33:56.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5876b0d7:::benchmark_data_smithi188_113957_object44257:head# 2024-04-23T16:33:56.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5877dee7:::benchmark_data_smithi188_113957_object54834:head# 2024-04-23T16:33:56.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58789604:::benchmark_data_smithi188_113957_object118826:head# 2024-04-23T16:33:56.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587897f8:::benchmark_data_smithi188_113957_object1111:head# 2024-04-23T16:33:56.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587962e8:::benchmark_data_smithi188_113957_object113308:head# 2024-04-23T16:33:56.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587a98ab:::benchmark_data_smithi188_113957_object2285:head# 2024-04-23T16:33:56.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587aecff:::benchmark_data_smithi188_113957_object77403:head# 2024-04-23T16:33:56.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587b09ed:::benchmark_data_smithi188_113957_object141861:head# 2024-04-23T16:33:56.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587b1bdb:::benchmark_data_smithi188_113957_object109170:head# 2024-04-23T16:33:56.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587b1e35:::benchmark_data_smithi188_113957_object91036:head# 2024-04-23T16:33:56.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587b7e46:::benchmark_data_smithi188_113957_object104535:head# 2024-04-23T16:33:56.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587be6bc:::benchmark_data_smithi188_113957_object68320:head# 2024-04-23T16:33:56.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587be729:::benchmark_data_smithi188_113957_object131133:head# 2024-04-23T16:33:56.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587c0b44:::benchmark_data_smithi188_113957_object139153:head# 2024-04-23T16:33:56.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587ca75b:::benchmark_data_smithi188_113957_object126778:head# 2024-04-23T16:33:56.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587cd8d5:::benchmark_data_smithi188_113957_object84679:head# 2024-04-23T16:33:56.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587ead3e:::benchmark_data_smithi188_113957_object43404:head# 2024-04-23T16:33:56.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587ee470:::benchmark_data_smithi188_113957_object96997:head# 2024-04-23T16:33:56.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587f05fb:::benchmark_data_smithi188_113957_object136395:head# 2024-04-23T16:33:56.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587f0c05:::benchmark_data_smithi188_113957_object38831:head# 2024-04-23T16:33:56.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587f18a7:::benchmark_data_smithi188_113957_object116086:head# 2024-04-23T16:33:56.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:587f4fa1:::benchmark_data_smithi188_113957_object83437:head# 2024-04-23T16:33:56.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5880319d:::benchmark_data_smithi188_113957_object19836:head# 2024-04-23T16:33:56.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5880eeca:::benchmark_data_smithi188_113957_object146221:head# 2024-04-23T16:33:56.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5881484c:::benchmark_data_smithi188_113957_object76652:head# 2024-04-23T16:33:56.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58818dc5:::benchmark_data_smithi188_113957_object108899:head# 2024-04-23T16:33:56.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5882d556:::benchmark_data_smithi188_113957_object60852:head# 2024-04-23T16:33:56.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58831788:::benchmark_data_smithi188_113957_object3506:head# 2024-04-23T16:33:56.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58839dc2:::benchmark_data_smithi188_113957_object115454:head# 2024-04-23T16:33:56.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5883b2c9:::benchmark_data_smithi188_113957_object125812:head# 2024-04-23T16:33:56.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5883cb15:::benchmark_data_smithi188_113957_object108637:head# 2024-04-23T16:33:56.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5884bf47:::benchmark_data_smithi188_113957_object3205:head# 2024-04-23T16:33:56.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5884e6c6:::benchmark_data_smithi188_113957_object35561:head# 2024-04-23T16:33:56.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5884f500:::benchmark_data_smithi188_113957_object152961:head# 2024-04-23T16:33:56.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58860eb0:::benchmark_data_smithi188_113957_object122941:head# 2024-04-23T16:33:56.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58864214:::benchmark_data_smithi188_113957_object52147:head# 2024-04-23T16:33:56.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58871b99:::benchmark_data_smithi188_113957_object66648:head# 2024-04-23T16:33:56.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58873dbb:::benchmark_data_smithi188_113957_object71971:head# 2024-04-23T16:33:56.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58873dea:::benchmark_data_smithi188_113957_object83734:head# 2024-04-23T16:33:56.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58877221:::benchmark_data_smithi188_113957_object17513:head# 2024-04-23T16:33:56.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58878201:::benchmark_data_smithi188_113957_object102239:head# 2024-04-23T16:33:56.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5888d14e:::benchmark_data_smithi188_113957_object57820:head# 2024-04-23T16:33:56.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58896a58:::benchmark_data_smithi188_113957_object47910:head# 2024-04-23T16:33:56.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58896cb1:::benchmark_data_smithi188_113957_object25818:head# 2024-04-23T16:33:56.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5889d3d3:::benchmark_data_smithi188_113957_object23042:head# 2024-04-23T16:33:56.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588a1799:::benchmark_data_smithi188_113957_object116341:head# 2024-04-23T16:33:56.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588a1ee0:::benchmark_data_smithi188_113957_object42206:head# 2024-04-23T16:33:56.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588a3667:::benchmark_data_smithi188_113957_object62478:head# 2024-04-23T16:33:56.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588a6022:::benchmark_data_smithi188_113957_object104911:head# 2024-04-23T16:33:56.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588a6af2:::benchmark_data_smithi188_113957_object34974:head# 2024-04-23T16:33:56.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588abbaa:::benchmark_data_smithi188_113957_object7168:head# 2024-04-23T16:33:56.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588acf15:::benchmark_data_smithi188_113957_object117399:head# 2024-04-23T16:33:56.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588b8d76:::benchmark_data_smithi188_113957_object51036:head# 2024-04-23T16:33:56.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588bc5a1:::benchmark_data_smithi188_113957_object143550:head# 2024-04-23T16:33:56.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588c59ee:::benchmark_data_smithi188_113957_object121416:head# 2024-04-23T16:33:56.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588d2011:::benchmark_data_smithi188_113957_object61521:head# 2024-04-23T16:33:56.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588d3733:::benchmark_data_smithi188_113957_object16214:head# 2024-04-23T16:33:56.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588d836c:::benchmark_data_smithi188_113957_object49070:head# 2024-04-23T16:33:56.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588dab13:::benchmark_data_smithi188_113957_object76748:head# 2024-04-23T16:33:56.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588e42ce:::benchmark_data_smithi188_113957_object122625:head# 2024-04-23T16:33:56.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588eb8a5:::benchmark_data_smithi188_113957_object93517:head# 2024-04-23T16:33:56.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.316 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:55 smithi103 ceph-mon[39629]: pgmap v1130: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.4 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 622 op/s; 25846/274602 objects degraded (9.412%); 7.6 MiB/s, 120 objects/s recovering 2024-04-23T16:33:56.339 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588f1487:::benchmark_data_smithi188_113957_object46350:head# 2024-04-23T16:33:56.339 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.339 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588f3368:::benchmark_data_smithi188_113957_object90697:head# 2024-04-23T16:33:56.339 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588f5623:::benchmark_data_smithi188_113957_object92600:head# 2024-04-23T16:33:56.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:588ff667:::benchmark_data_smithi188_113957_object101801:head# 2024-04-23T16:33:56.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:589089c2:::benchmark_data_smithi188_113957_object22566:head# 2024-04-23T16:33:56.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5890f800:::benchmark_data_smithi188_113957_object26006:head# 2024-04-23T16:33:56.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58911912:::benchmark_data_smithi188_113957_object64120:head# 2024-04-23T16:33:56.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5891eabb:::benchmark_data_smithi188_113957_object34809:head# 2024-04-23T16:33:56.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5891ebc8:::benchmark_data_smithi188_113957_object46581:head# 2024-04-23T16:33:56.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.341 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:589250ee:::benchmark_data_smithi188_113957_object120549:head# 2024-04-23T16:33:56.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.341 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58925816:::benchmark_data_smithi188_113957_object98408:head# 2024-04-23T16:33:56.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.341 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5892d14c:::benchmark_data_smithi188_113957_object22515:head# 2024-04-23T16:33:56.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.341 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5892fa55:::benchmark_data_smithi188_113957_object145356:head# 2024-04-23T16:33:56.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.341 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5893200c:::benchmark_data_smithi188_113957_object64622:head# 2024-04-23T16:33:56.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.341 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58935269:::benchmark_data_smithi188_113957_object102897:head# 2024-04-23T16:33:56.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.341 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58940952:::benchmark_data_smithi188_113957_object116590:head# 2024-04-23T16:33:56.342 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.342 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5894a8b4:::benchmark_data_smithi188_113957_object23530:head# 2024-04-23T16:33:56.342 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.342 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5894c68c:::benchmark_data_smithi188_113957_object43124:head# 2024-04-23T16:33:56.342 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.342 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58952771:::benchmark_data_smithi188_113957_object34043:head# 2024-04-23T16:33:56.342 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.342 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58952d27:::benchmark_data_smithi188_113957_object142314:head# 2024-04-23T16:33:56.342 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.342 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5895669d:::benchmark_data_smithi188_113957_object23484:head# 2024-04-23T16:33:56.342 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.342 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5896a7ef:::benchmark_data_smithi188_113957_object150378:head# 2024-04-23T16:33:56.342 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.343 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5898bc40:::benchmark_data_smithi188_113957_object47595:head# 2024-04-23T16:33:56.343 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.343 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58995fb7:::benchmark_data_smithi188_113957_object4223:head# 2024-04-23T16:33:56.343 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.343 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58996de3:::benchmark_data_smithi188_113957_object127277:head# 2024-04-23T16:33:56.343 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.343 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:589a98e9:::benchmark_data_smithi188_113957_object38058:head# 2024-04-23T16:33:56.343 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.343 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:589ad526:::benchmark_data_smithi188_113957_object112214:head# 2024-04-23T16:33:56.343 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.343 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:589c0360:::benchmark_data_smithi188_113957_object108663:head# 2024-04-23T16:33:56.343 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.343 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:589c2964:::benchmark_data_smithi188_113957_object95085:head# 2024-04-23T16:33:56.343 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.344 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:589c525e:::benchmark_data_smithi188_113957_object122526:head# 2024-04-23T16:33:56.344 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.344 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:589d1698:::benchmark_data_smithi188_113957_object80636:head# 2024-04-23T16:33:56.344 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.344 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:589d2906:::benchmark_data_smithi188_113957_object151979:head# 2024-04-23T16:33:56.344 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.344 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:589d343a:::benchmark_data_smithi188_113957_object25550:head# 2024-04-23T16:33:56.344 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.344 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:589d4c3f:::benchmark_data_smithi188_113957_object152485:head# 2024-04-23T16:33:56.344 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.344 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:589e1bc5:::benchmark_data_smithi188_113957_object28830:head# 2024-04-23T16:33:56.344 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.344 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:589e9ee5:::benchmark_data_smithi188_113957_object95690:head# 2024-04-23T16:33:56.344 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.345 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:589eff36:::benchmark_data_smithi188_113957_object83331:head# 2024-04-23T16:33:56.345 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.345 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a0c6e5:::benchmark_data_smithi188_113957_object55729:head# 2024-04-23T16:33:56.345 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.345 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a124c1:::benchmark_data_smithi188_113957_object42861:head# 2024-04-23T16:33:56.345 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.345 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a1be8c:::benchmark_data_smithi188_113957_object314:head# 2024-04-23T16:33:56.345 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.345 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a1e2b4:::benchmark_data_smithi188_113957_object51687:head# 2024-04-23T16:33:56.345 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.345 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a21224:::benchmark_data_smithi188_113957_object87462:head# 2024-04-23T16:33:56.345 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.345 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a2313d:::benchmark_data_smithi188_113957_object37784:head# 2024-04-23T16:33:56.345 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.346 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a25967:::benchmark_data_smithi188_113957_object105923:head# 2024-04-23T16:33:56.346 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.346 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a268f4:::benchmark_data_smithi188_113957_object91747:head# 2024-04-23T16:33:56.346 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.346 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a31a06:::benchmark_data_smithi188_113957_object90449:head# 2024-04-23T16:33:56.346 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.346 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a386f5:::benchmark_data_smithi188_113957_object61760:head# 2024-04-23T16:33:56.346 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.346 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a39973:::benchmark_data_smithi188_113957_object136095:head# 2024-04-23T16:33:56.346 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.346 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a43393:::benchmark_data_smithi188_113957_object22869:head# 2024-04-23T16:33:56.346 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.346 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a4b344:::benchmark_data_smithi188_113957_object105234:head# 2024-04-23T16:33:56.346 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.347 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a5032d:::benchmark_data_smithi188_113957_object78859:head# 2024-04-23T16:33:56.347 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.347 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a50f4b:::benchmark_data_smithi188_113957_object14351:head# 2024-04-23T16:33:56.347 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.347 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a51ad2:::benchmark_data_smithi188_113957_object145501:head# 2024-04-23T16:33:56.347 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.347 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a58891:::benchmark_data_smithi188_113957_object36564:head# 2024-04-23T16:33:56.347 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.347 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a5c8ba:::benchmark_data_smithi188_113957_object96948:head# 2024-04-23T16:33:56.347 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.347 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a5db8b:::benchmark_data_smithi188_113957_object119398:head# 2024-04-23T16:33:56.347 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.347 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a5f7cf:::benchmark_data_smithi188_113957_object20449:head# 2024-04-23T16:33:56.347 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.348 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a612c2:::benchmark_data_smithi188_113957_object40832:head# 2024-04-23T16:33:56.348 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.348 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a62389:::benchmark_data_smithi188_113957_object102781:head# 2024-04-23T16:33:56.348 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.348 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a70890:::benchmark_data_smithi188_113957_object42578:head# 2024-04-23T16:33:56.348 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.348 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a83369:::benchmark_data_smithi188_113957_object103501:head# 2024-04-23T16:33:56.348 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.348 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a83aee:::benchmark_data_smithi188_113957_object21662:head# 2024-04-23T16:33:56.348 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.348 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a86a00:::benchmark_data_smithi188_113957_object120198:head# 2024-04-23T16:33:56.348 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.348 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a8df29:::benchmark_data_smithi188_113957_object47235:head# 2024-04-23T16:33:56.348 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.349 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a92c72:::benchmark_data_smithi188_113957_object87105:head# 2024-04-23T16:33:56.349 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.349 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a96432:::benchmark_data_smithi188_113957_object54801:head# 2024-04-23T16:33:56.349 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.349 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58a9dcf9:::benchmark_data_smithi188_113957_object64417:head# 2024-04-23T16:33:56.349 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.349 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ab20d8:::benchmark_data_smithi188_113957_object97237:head# 2024-04-23T16:33:56.349 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.349 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ab46f2:::benchmark_data_smithi188_113957_object117954:head# 2024-04-23T16:33:56.349 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.349 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58abcbd8:::benchmark_data_smithi188_113957_object24225:head# 2024-04-23T16:33:56.349 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.349 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ac8934:::benchmark_data_smithi188_113957_object55135:head# 2024-04-23T16:33:56.349 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58acdce5:::benchmark_data_smithi188_113957_object99230:head# 2024-04-23T16:33:56.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58adb1d8:::benchmark_data_smithi188_113957_object28298:head# 2024-04-23T16:33:56.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58add20f:::benchmark_data_smithi188_113957_object143584:head# 2024-04-23T16:33:56.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ae2cda:::benchmark_data_smithi188_113957_object54469:head# 2024-04-23T16:33:56.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ae3b55:::benchmark_data_smithi188_113957_object20174:head# 2024-04-23T16:33:56.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ae7d57:::benchmark_data_smithi188_113957_object100508:head# 2024-04-23T16:33:56.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58aeb0c1:::benchmark_data_smithi188_113957_object71800:head# 2024-04-23T16:33:56.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58af9292:::benchmark_data_smithi188_113957_object152243:head# 2024-04-23T16:33:56.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b0aed6:::benchmark_data_smithi188_113957_object30766:head# 2024-04-23T16:33:56.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b22b33:::benchmark_data_smithi188_113957_object63771:head# 2024-04-23T16:33:56.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b29eab:::benchmark_data_smithi188_113957_object140487:head# 2024-04-23T16:33:56.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b2f4ad:::benchmark_data_smithi188_113957_object103863:head# 2024-04-23T16:33:56.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b34d06:::benchmark_data_smithi188_113957_object73512:head# 2024-04-23T16:33:56.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b351e8:::benchmark_data_smithi188_113957_object47392:head# 2024-04-23T16:33:56.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b3745b:::benchmark_data_smithi188_113957_object78090:head# 2024-04-23T16:33:56.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b42131:::benchmark_data_smithi188_113957_object26797:head# 2024-04-23T16:33:56.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b4316a:::benchmark_data_smithi188_113957_object940:head# 2024-04-23T16:33:56.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b47b4f:::benchmark_data_smithi188_113957_object22973:head# 2024-04-23T16:33:56.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b53270:::benchmark_data_smithi188_113957_object123985:head# 2024-04-23T16:33:56.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b53eaf:::benchmark_data_smithi188_113957_object15139:head# 2024-04-23T16:33:56.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b5f277:::benchmark_data_smithi188_113957_object130811:head# 2024-04-23T16:33:56.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b68354:::benchmark_data_smithi188_113957_object47870:head# 2024-04-23T16:33:56.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b7392b:::benchmark_data_smithi188_113957_object99717:head# 2024-04-23T16:33:56.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b74972:::benchmark_data_smithi188_113957_object96521:head# 2024-04-23T16:33:56.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b76f23:::benchmark_data_smithi188_113957_object92181:head# 2024-04-23T16:33:56.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b7a658:::benchmark_data_smithi188_113957_object57404:head# 2024-04-23T16:33:56.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b8a0d4:::benchmark_data_smithi188_113957_object33376:head# 2024-04-23T16:33:56.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b93222:::benchmark_data_smithi188_113957_object42687:head# 2024-04-23T16:33:56.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58b983ba:::benchmark_data_smithi188_113957_object67112:head# 2024-04-23T16:33:56.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ba1120:::benchmark_data_smithi188_113957_object2261:head# 2024-04-23T16:33:56.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ba21b8:::benchmark_data_smithi188_113957_object28929:head# 2024-04-23T16:33:56.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58bbfe42:::benchmark_data_smithi188_113957_object126765:head# 2024-04-23T16:33:56.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58bc0add:::benchmark_data_smithi188_113957_object131595:head# 2024-04-23T16:33:56.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58bc3f1b:::benchmark_data_smithi188_113957_object76001:head# 2024-04-23T16:33:56.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58bc9172:::benchmark_data_smithi188_113957_object61699:head# 2024-04-23T16:33:56.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58bd4c43:::benchmark_data_smithi188_113957_object144322:head# 2024-04-23T16:33:56.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58bd77ca:::benchmark_data_smithi188_113957_object32391:head# 2024-04-23T16:33:56.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58bd9060:::benchmark_data_smithi188_113957_object38556:head# 2024-04-23T16:33:56.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58bdcd07:::benchmark_data_smithi188_113957_object64189:head# 2024-04-23T16:33:56.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58bdd88e:::benchmark_data_smithi188_113957_object138064:head# 2024-04-23T16:33:56.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58be1ef5:::benchmark_data_smithi188_113957_object63260:head# 2024-04-23T16:33:56.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58be65e4:::benchmark_data_smithi188_113957_object83561:head# 2024-04-23T16:33:56.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58be8718:::benchmark_data_smithi188_113957_object36323:head# 2024-04-23T16:33:56.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58beea73:::benchmark_data_smithi188_113957_object38650:head# 2024-04-23T16:33:56.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58bfd8d3:::benchmark_data_smithi188_113957_object72163:head# 2024-04-23T16:33:56.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c049d4:::benchmark_data_smithi188_113957_object22153:head# 2024-04-23T16:33:56.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c08380:::benchmark_data_smithi188_113957_object20512:head# 2024-04-23T16:33:56.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c0885c:::benchmark_data_smithi188_113957_object54938:head# 2024-04-23T16:33:56.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c0e2cb:::benchmark_data_smithi188_113957_object90353:head# 2024-04-23T16:33:56.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c14214:::benchmark_data_smithi188_113957_object128997:head# 2024-04-23T16:33:56.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c16583:::benchmark_data_smithi188_113957_object29219:head# 2024-04-23T16:33:56.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c177f8:::benchmark_data_smithi188_113957_object128836:head# 2024-04-23T16:33:56.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c1bad0:::benchmark_data_smithi188_113957_object38892:head# 2024-04-23T16:33:56.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c2164b:::benchmark_data_smithi188_113957_object19355:head# 2024-04-23T16:33:56.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c21fe4:::benchmark_data_smithi188_113957_object36936:head# 2024-04-23T16:33:56.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c2f0e7:::benchmark_data_smithi188_113957_object115853:head# 2024-04-23T16:33:56.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c2faa7:::benchmark_data_smithi188_113957_object148737:head# 2024-04-23T16:33:56.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c3223d:::benchmark_data_smithi188_113957_object104588:head# 2024-04-23T16:33:56.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c365d7:::benchmark_data_smithi188_113957_object68338:head# 2024-04-23T16:33:56.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c36d43:::benchmark_data_smithi188_113957_object113769:head# 2024-04-23T16:33:56.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c3908e:::benchmark_data_smithi188_113957_object40362:head# 2024-04-23T16:33:56.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c412f0:::benchmark_data_smithi188_113957_object61360:head# 2024-04-23T16:33:56.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c44b90:::benchmark_data_smithi188_113957_object118984:head# 2024-04-23T16:33:56.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c658b1:::benchmark_data_smithi188_113957_object62165:head# 2024-04-23T16:33:56.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c66488:::benchmark_data_smithi188_113957_object75915:head# 2024-04-23T16:33:56.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c79b0e:::benchmark_data_smithi188_113957_object119853:head# 2024-04-23T16:33:56.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58c84118:::benchmark_data_smithi188_113957_object108494:head# 2024-04-23T16:33:56.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ca0280:::benchmark_data_smithi188_113957_object83135:head# 2024-04-23T16:33:56.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ca5bc7:::benchmark_data_smithi188_113957_object92350:head# 2024-04-23T16:33:56.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ca9945:::benchmark_data_smithi188_113957_object131038:head# 2024-04-23T16:33:56.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58cb1b43:::benchmark_data_smithi188_113957_object137662:head# 2024-04-23T16:33:56.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58cb240f:::benchmark_data_smithi188_113957_object21564:head# 2024-04-23T16:33:56.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58cb6fe0:::benchmark_data_smithi188_113957_object103428:head# 2024-04-23T16:33:56.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58cbafff:::benchmark_data_smithi188_113957_object139198:head# 2024-04-23T16:33:56.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58cbed69:::benchmark_data_smithi188_113957_object128978:head# 2024-04-23T16:33:56.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58cc8a2f:::benchmark_data_smithi188_113957_object107821:head# 2024-04-23T16:33:56.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58cd025f:::benchmark_data_smithi188_113957_object119283:head# 2024-04-23T16:33:56.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58cd26a4:::benchmark_data_smithi188_113957_object7150:head# 2024-04-23T16:33:56.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58cd34ca:::benchmark_data_smithi188_113957_object114792:head# 2024-04-23T16:33:56.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ce8d25:::benchmark_data_smithi188_113957_object94704:head# 2024-04-23T16:33:56.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58cfd972:::benchmark_data_smithi188_113957_object122265:head# 2024-04-23T16:33:56.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58cff73f:::benchmark_data_smithi188_113957_object34534:head# 2024-04-23T16:33:56.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d0962f:::benchmark_data_smithi188_113957_object139612:head# 2024-04-23T16:33:56.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d0fd4c:::benchmark_data_smithi188_113957_object70682:head# 2024-04-23T16:33:56.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d18a7c:::benchmark_data_smithi188_113957_object89492:head# 2024-04-23T16:33:56.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d24b0d:::benchmark_data_smithi188_113957_object49062:head# 2024-04-23T16:33:56.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d25d46:::benchmark_data_smithi188_113957_object50577:head# 2024-04-23T16:33:56.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d2662b:::benchmark_data_smithi188_113957_object99606:head# 2024-04-23T16:33:56.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d29687:::benchmark_data_smithi188_113957_object22420:head# 2024-04-23T16:33:56.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d2a423:::benchmark_data_smithi188_113957_object57476:head# 2024-04-23T16:33:56.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d2fe49:::benchmark_data_smithi188_113957_object59097:head# 2024-04-23T16:33:56.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d3155b:::benchmark_data_smithi188_113957_object77940:head# 2024-04-23T16:33:56.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d3325c:::benchmark_data_smithi188_113957_object145274:head# 2024-04-23T16:33:56.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d3538d:::benchmark_data_smithi188_113957_object35970:head# 2024-04-23T16:33:56.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d37894:::benchmark_data_smithi188_113957_object79678:head# 2024-04-23T16:33:56.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d47526:::benchmark_data_smithi188_113957_object4181:head# 2024-04-23T16:33:56.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d47c2f:::benchmark_data_smithi188_113957_object109978:head# 2024-04-23T16:33:56.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d4c18b:::benchmark_data_smithi188_113957_object74651:head# 2024-04-23T16:33:56.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d540bc:::benchmark_data_smithi188_113957_object126985:head# 2024-04-23T16:33:56.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d54ec5:::benchmark_data_smithi188_113957_object147983:head# 2024-04-23T16:33:56.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d5aadd:::benchmark_data_smithi188_113957_object125966:head# 2024-04-23T16:33:56.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d5d5b3:::benchmark_data_smithi188_113957_object36695:head# 2024-04-23T16:33:56.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d5ed19:::benchmark_data_smithi188_113957_object84285:head# 2024-04-23T16:33:56.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d60a35:::benchmark_data_smithi188_113957_object23160:head# 2024-04-23T16:33:56.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d63ee1:::benchmark_data_smithi188_113957_object116271:head# 2024-04-23T16:33:56.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d69f1a:::benchmark_data_smithi188_113957_object70772:head# 2024-04-23T16:33:56.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d75719:::benchmark_data_smithi188_113957_object146534:head# 2024-04-23T16:33:56.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58d7873c:::benchmark_data_smithi188_113957_object11541:head# 2024-04-23T16:33:56.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58da82cc:::benchmark_data_smithi188_113957_object122586:head# 2024-04-23T16:33:56.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58da8ee6:::benchmark_data_smithi188_113957_object146297:head# 2024-04-23T16:33:56.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58daa4c9:::benchmark_data_smithi188_113957_object41110:head# 2024-04-23T16:33:56.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58db862e:::benchmark_data_smithi188_113957_object57928:head# 2024-04-23T16:33:56.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58db9dcb:::benchmark_data_smithi188_113957_object88292:head# 2024-04-23T16:33:56.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58dbd227:::benchmark_data_smithi188_113957_object127916:head# 2024-04-23T16:33:56.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58dc161b:::benchmark_data_smithi188_113957_object130852:head# 2024-04-23T16:33:56.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58dc2c66:::benchmark_data_smithi188_113957_object135347:head# 2024-04-23T16:33:56.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58dc8e60:::benchmark_data_smithi188_113957_object76757:head# 2024-04-23T16:33:56.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58dcb77f:::benchmark_data_smithi188_113957_object124421:head# 2024-04-23T16:33:56.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58dcecac:::benchmark_data_smithi188_113957_object61072:head# 2024-04-23T16:33:56.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58de73c3:::benchmark_data_smithi188_113957_object14634:head# 2024-04-23T16:33:56.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58e07521:::benchmark_data_smithi188_113957_object39653:head# 2024-04-23T16:33:56.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58e0c25d:::benchmark_data_smithi188_113957_object2062:head# 2024-04-23T16:33:56.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58e29840:::benchmark_data_smithi188_113957_object147649:head# 2024-04-23T16:33:56.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58e2ed95:::benchmark_data_smithi188_113957_object50604:head# 2024-04-23T16:33:56.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58e307fa:::benchmark_data_smithi188_113957_object95279:head# 2024-04-23T16:33:56.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58e5430a:::benchmark_data_smithi188_113957_object45951:head# 2024-04-23T16:33:56.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58e62ffb:::benchmark_data_smithi188_113957_object52926:head# 2024-04-23T16:33:56.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58e645ae:::benchmark_data_smithi188_113957_object109094:head# 2024-04-23T16:33:56.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58e71551:::benchmark_data_smithi188_113957_object77835:head# 2024-04-23T16:33:56.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58e79b35:::benchmark_data_smithi188_113957_object130150:head# 2024-04-23T16:33:56.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58e88b54:::benchmark_data_smithi188_113957_object66661:head# 2024-04-23T16:33:56.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58e89d8c:::benchmark_data_smithi188_113957_object42353:head# 2024-04-23T16:33:56.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58e924c4:::benchmark_data_smithi188_113957_object13894:head# 2024-04-23T16:33:56.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58e9735e:::benchmark_data_smithi188_113957_object17343:head# 2024-04-23T16:33:56.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58eb0760:::benchmark_data_smithi188_113957_object78004:head# 2024-04-23T16:33:56.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58eb151d:::benchmark_data_smithi188_113957_object143770:head# 2024-04-23T16:33:56.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58eb8302:::benchmark_data_smithi188_113957_object42664:head# 2024-04-23T16:33:56.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ec46ec:::benchmark_data_smithi188_113957_object93757:head# 2024-04-23T16:33:56.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ec885d:::benchmark_data_smithi188_113957_object50186:head# 2024-04-23T16:33:56.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ecc7dc:::benchmark_data_smithi188_113957_object83961:head# 2024-04-23T16:33:56.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ece08d:::benchmark_data_smithi188_113957_object63641:head# 2024-04-23T16:33:56.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ed32cf:::benchmark_data_smithi188_113957_object88810:head# 2024-04-23T16:33:56.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ed7b08:::benchmark_data_smithi188_113957_object18848:head# 2024-04-23T16:33:56.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ed9683:::benchmark_data_smithi188_113957_object51058:head# 2024-04-23T16:33:56.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58edcd64:::benchmark_data_smithi188_113957_object34693:head# 2024-04-23T16:33:56.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ee5d4f:::benchmark_data_smithi188_113957_object83131:head# 2024-04-23T16:33:56.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ef5534:::benchmark_data_smithi188_113957_object110608:head# 2024-04-23T16:33:56.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f07019:::benchmark_data_smithi188_113957_object147465:head# 2024-04-23T16:33:56.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f09e71:::benchmark_data_smithi188_113957_object19873:head# 2024-04-23T16:33:56.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f0b78f:::benchmark_data_smithi188_113957_object146616:head# 2024-04-23T16:33:56.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f169b3:::benchmark_data_smithi188_113957_object16309:head# 2024-04-23T16:33:56.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f19a07:::benchmark_data_smithi188_113957_object150051:head# 2024-04-23T16:33:56.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f1b514:::benchmark_data_smithi188_113957_object139805:head# 2024-04-23T16:33:56.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f22309:::benchmark_data_smithi188_113957_object139750:head# 2024-04-23T16:33:56.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f2ac4f:::benchmark_data_smithi188_113957_object23992:head# 2024-04-23T16:33:56.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f2cc84:::benchmark_data_smithi188_113957_object107080:head# 2024-04-23T16:33:56.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f2d5d9:::benchmark_data_smithi188_113957_object4907:head# 2024-04-23T16:33:56.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f3478f:::benchmark_data_smithi188_113957_object71659:head# 2024-04-23T16:33:56.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f3859e:::benchmark_data_smithi188_113957_object92065:head# 2024-04-23T16:33:56.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f3944f:::benchmark_data_smithi188_113957_object18140:head# 2024-04-23T16:33:56.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f40c41:::benchmark_data_smithi188_113957_object86225:head# 2024-04-23T16:33:56.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f4979e:::benchmark_data_smithi188_113957_object29800:head# 2024-04-23T16:33:56.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f4b999:::benchmark_data_smithi188_113957_object68257:head# 2024-04-23T16:33:56.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f5529e:::benchmark_data_smithi188_113957_object18485:head# 2024-04-23T16:33:56.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f5a45d:::benchmark_data_smithi188_113957_object38528:head# 2024-04-23T16:33:56.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f628df:::benchmark_data_smithi188_113957_object75728:head# 2024-04-23T16:33:56.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f6e8e1:::benchmark_data_smithi188_113957_object93644:head# 2024-04-23T16:33:56.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f72982:::benchmark_data_smithi188_113957_object49642:head# 2024-04-23T16:33:56.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f76bef:::benchmark_data_smithi188_113957_object39971:head# 2024-04-23T16:33:56.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f7e472:::benchmark_data_smithi188_113957_object147633:head# 2024-04-23T16:33:56.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f80e0f:::benchmark_data_smithi188_113957_object110123:head# 2024-04-23T16:33:56.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58f9b7b9:::benchmark_data_smithi188_113957_object126969:head# 2024-04-23T16:33:56.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58fa1bde:::benchmark_data_smithi188_113957_object7335:head# 2024-04-23T16:33:56.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58fa7b3c:::benchmark_data_smithi188_113957_object18024:head# 2024-04-23T16:33:56.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58fa8ea8:::benchmark_data_smithi188_113957_object137633:head# 2024-04-23T16:33:56.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58fa9304:::benchmark_data_smithi188_113957_object33718:head# 2024-04-23T16:33:56.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58faaf2f:::benchmark_data_smithi188_113957_object107254:head# 2024-04-23T16:33:56.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58fb0d7e:::benchmark_data_smithi188_113957_object97123:head# 2024-04-23T16:33:56.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58fb3217:::benchmark_data_smithi188_113957_object104260:head# 2024-04-23T16:33:56.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58fb6a86:::benchmark_data_smithi188_113957_object24708:head# 2024-04-23T16:33:56.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58fb72d7:::benchmark_data_smithi188_113957_object46512:head# 2024-04-23T16:33:56.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58fc368a:::benchmark_data_smithi188_113957_object97530:head# 2024-04-23T16:33:56.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58fc5281:::benchmark_data_smithi188_113957_object94560:head# 2024-04-23T16:33:56.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58fcb521:::benchmark_data_smithi188_113957_object9246:head# 2024-04-23T16:33:56.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58fcf838:::benchmark_data_smithi188_113957_object114747:head# 2024-04-23T16:33:56.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58fdd95b:::benchmark_data_smithi188_113957_object10251:head# 2024-04-23T16:33:56.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58fdefa4:::benchmark_data_smithi188_113957_object80197:head# 2024-04-23T16:33:56.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58fe956f:::benchmark_data_smithi188_113957_object148754:head# 2024-04-23T16:33:56.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ff04c2:::benchmark_data_smithi188_113957_object14819:head# 2024-04-23T16:33:56.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ff3722:::benchmark_data_smithi188_113957_object116612:head# 2024-04-23T16:33:56.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ff3df8:::benchmark_data_smithi188_113957_object140521:head# 2024-04-23T16:33:56.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ffab92:::benchmark_data_smithi188_113957_object152325:head# 2024-04-23T16:33:56.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:58ffe3e2:::benchmark_data_smithi188_113957_object50565:head# 2024-04-23T16:33:56.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5900f981:::benchmark_data_smithi188_113957_object63292:head# 2024-04-23T16:33:56.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5901355e:::benchmark_data_smithi188_113957_object145043:head# 2024-04-23T16:33:56.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59029e6b:::benchmark_data_smithi188_113957_object47663:head# 2024-04-23T16:33:56.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59048ae9:::benchmark_data_smithi188_113957_object99000:head# 2024-04-23T16:33:56.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59057390:::benchmark_data_smithi188_113957_object86286:head# 2024-04-23T16:33:56.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590606cb:::benchmark_data_smithi188_113957_object109561:head# 2024-04-23T16:33:56.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590634b5:::benchmark_data_smithi188_113957_object95843:head# 2024-04-23T16:33:56.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590774a9:::benchmark_data_smithi188_113957_object86738:head# 2024-04-23T16:33:56.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590783be:::benchmark_data_smithi188_113957_object144513:head# 2024-04-23T16:33:56.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5907fea1:::benchmark_data_smithi188_113957_object74415:head# 2024-04-23T16:33:56.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5908b495:::benchmark_data_smithi188_113957_object151029:head# 2024-04-23T16:33:56.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5908bf4c:::benchmark_data_smithi188_113957_object7295:head# 2024-04-23T16:33:56.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5908ee9f:::benchmark_data_smithi188_113957_object75037:head# 2024-04-23T16:33:56.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59094935:::benchmark_data_smithi188_113957_object136701:head# 2024-04-23T16:33:56.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590a8d13:::benchmark_data_smithi188_113957_object43421:head# 2024-04-23T16:33:56.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590afeea:::benchmark_data_smithi188_113957_object97687:head# 2024-04-23T16:33:56.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590b02b2:::benchmark_data_smithi188_113957_object99361:head# 2024-04-23T16:33:56.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590b2b68:::benchmark_data_smithi188_113957_object143721:head# 2024-04-23T16:33:56.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590c0924:::benchmark_data_smithi188_113957_object120164:head# 2024-04-23T16:33:56.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590c290c:::benchmark_data_smithi188_113957_object131337:head# 2024-04-23T16:33:56.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590c4a3f:::benchmark_data_smithi188_113957_object105994:head# 2024-04-23T16:33:56.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590c8144:::benchmark_data_smithi188_113957_object76566:head# 2024-04-23T16:33:56.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590c9748:::benchmark_data_smithi188_113957_object97867:head# 2024-04-23T16:33:56.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590d1222:::benchmark_data_smithi188_113957_object101226:head# 2024-04-23T16:33:56.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590d3a26:::benchmark_data_smithi188_113957_object36523:head# 2024-04-23T16:33:56.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590d3a3a:::benchmark_data_smithi188_113957_object115827:head# 2024-04-23T16:33:56.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590d5e07:::benchmark_data_smithi188_113957_object19318:head# 2024-04-23T16:33:56.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590e1c25:::benchmark_data_smithi188_113957_object50729:head# 2024-04-23T16:33:56.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590ed616:::benchmark_data_smithi188_113957_object71120:head# 2024-04-23T16:33:56.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590f31f2:::benchmark_data_smithi188_113957_object139133:head# 2024-04-23T16:33:56.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590f7deb:::benchmark_data_smithi188_113957_object29636:head# 2024-04-23T16:33:56.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:590fd1d8:::benchmark_data_smithi188_113957_object95410:head# 2024-04-23T16:33:56.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59113d35:::benchmark_data_smithi188_113957_object75254:head# 2024-04-23T16:33:56.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59116c67:::benchmark_data_smithi188_113957_object38233:head# 2024-04-23T16:33:56.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59127576:::benchmark_data_smithi188_113957_object29194:head# 2024-04-23T16:33:56.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59134349:::benchmark_data_smithi188_113957_object146222:head# 2024-04-23T16:33:56.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5913f746:::benchmark_data_smithi188_113957_object109016:head# 2024-04-23T16:33:56.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59142196:::benchmark_data_smithi188_113957_object20889:head# 2024-04-23T16:33:56.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59143a49:::benchmark_data_smithi188_113957_object20668:head# 2024-04-23T16:33:56.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5914511d:::benchmark_data_smithi188_113957_object41786:head# 2024-04-23T16:33:56.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59147613:::benchmark_data_smithi188_113957_object125388:head# 2024-04-23T16:33:56.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5914eee6:::benchmark_data_smithi188_113957_object106107:head# 2024-04-23T16:33:56.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5914fcf0:::benchmark_data_smithi188_113957_object139965:head# 2024-04-23T16:33:56.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:591510c2:::benchmark_data_smithi188_113957_object183:head# 2024-04-23T16:33:56.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5915bcae:::benchmark_data_smithi188_113957_object42403:head# 2024-04-23T16:33:56.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5915f2ee:::benchmark_data_smithi188_113957_object21997:head# 2024-04-23T16:33:56.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59167855:::benchmark_data_smithi188_113957_object132991:head# 2024-04-23T16:33:56.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59173af0:::benchmark_data_smithi188_113957_object104125:head# 2024-04-23T16:33:56.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:591740e1:::benchmark_data_smithi188_113957_object84190:head# 2024-04-23T16:33:56.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5917a4b8:::benchmark_data_smithi188_113957_object25726:head# 2024-04-23T16:33:56.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59184134:::benchmark_data_smithi188_113957_object108626:head# 2024-04-23T16:33:56.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5919491a:::benchmark_data_smithi188_113957_object30077:head# 2024-04-23T16:33:56.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59195b37:::benchmark_data_smithi188_113957_object78014:head# 2024-04-23T16:33:56.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59199b96:::benchmark_data_smithi188_113957_object151653:head# 2024-04-23T16:33:56.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5919ae54:::benchmark_data_smithi188_113957_object16268:head# 2024-04-23T16:33:56.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:591a046d:::benchmark_data_smithi188_113957_object100665:head# 2024-04-23T16:33:56.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:591a2ce7:::benchmark_data_smithi188_113957_object54338:head# 2024-04-23T16:33:56.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:591aadfd:::benchmark_data_smithi188_113957_object12977:head# 2024-04-23T16:33:56.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:591af7ac:::benchmark_data_smithi188_113957_object57183:head# 2024-04-23T16:33:56.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:591b677d:::benchmark_data_smithi188_113957_object69725:head# 2024-04-23T16:33:56.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:591c3d50:::benchmark_data_smithi188_113957_object24137:head# 2024-04-23T16:33:56.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:591d6796:::benchmark_data_smithi188_113957_object143341:head# 2024-04-23T16:33:56.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:591d6e62:::benchmark_data_smithi188_113957_object31535:head# 2024-04-23T16:33:56.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:591dabb3:::benchmark_data_smithi188_113957_object8005:head# 2024-04-23T16:33:56.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:591dc8fc:::benchmark_data_smithi188_113957_object33407:head# 2024-04-23T16:33:56.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:591ea2be:::benchmark_data_smithi188_113957_object110887:head# 2024-04-23T16:33:56.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:591ea2cf:::benchmark_data_smithi188_113957_object14781:head# 2024-04-23T16:33:56.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:591f4b2d:::benchmark_data_smithi188_113957_object120594:head# 2024-04-23T16:33:56.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:591f7caf:::benchmark_data_smithi188_113957_object88844:head# 2024-04-23T16:33:56.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59206aa4:::benchmark_data_smithi188_113957_object133295:head# 2024-04-23T16:33:56.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5921d36e:::benchmark_data_smithi188_113957_object99041:head# 2024-04-23T16:33:56.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5921d979:::benchmark_data_smithi188_113957_object18366:head# 2024-04-23T16:33:56.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5921f033:::benchmark_data_smithi188_113957_object146392:head# 2024-04-23T16:33:56.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5922549b:::benchmark_data_smithi188_113957_object72819:head# 2024-04-23T16:33:56.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59225f5e:::benchmark_data_smithi188_113957_object48296:head# 2024-04-23T16:33:56.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5922f55b:::benchmark_data_smithi188_113957_object64828:head# 2024-04-23T16:33:56.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5923cafb:::benchmark_data_smithi188_113957_object52495:head# 2024-04-23T16:33:56.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592446cf:::benchmark_data_smithi188_113957_object86693:head# 2024-04-23T16:33:56.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59248537:::benchmark_data_smithi188_113957_object93085:head# 2024-04-23T16:33:56.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5924acb9:::benchmark_data_smithi188_113957_object90896:head# 2024-04-23T16:33:56.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592516b9:::benchmark_data_smithi188_113957_object41666:head# 2024-04-23T16:33:56.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5925234b:::benchmark_data_smithi188_113957_object12580:head# 2024-04-23T16:33:56.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59252f95:::benchmark_data_smithi188_113957_object137514:head# 2024-04-23T16:33:56.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59254260:::benchmark_data_smithi188_113957_object57618:head# 2024-04-23T16:33:56.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5925b53b:::benchmark_data_smithi188_113957_object31481:head# 2024-04-23T16:33:56.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5925be25:::benchmark_data_smithi188_113957_object148234:head# 2024-04-23T16:33:56.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5926a267:::benchmark_data_smithi188_113957_object46269:head# 2024-04-23T16:33:56.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5926a5eb:::benchmark_data_smithi188_113957_object33157:head# 2024-04-23T16:33:56.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592748cf:::benchmark_data_smithi188_113957_object19955:head# 2024-04-23T16:33:56.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59276468:::benchmark_data_smithi188_113957_object115897:head# 2024-04-23T16:33:56.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59277ce5:::benchmark_data_smithi188_113957_object64938:head# 2024-04-23T16:33:56.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5927af46:::benchmark_data_smithi188_113957_object20524:head# 2024-04-23T16:33:56.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59283fe2:::benchmark_data_smithi188_113957_object94347:head# 2024-04-23T16:33:56.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592853ec:::benchmark_data_smithi188_113957_object77711:head# 2024-04-23T16:33:56.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59286e09:::benchmark_data_smithi188_113957_object128799:head# 2024-04-23T16:33:56.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59294e0a:::benchmark_data_smithi188_113957_object95711:head# 2024-04-23T16:33:56.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592956da:::benchmark_data_smithi188_113957_object147247:head# 2024-04-23T16:33:56.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592a0eb5:::benchmark_data_smithi188_113957_object94373:head# 2024-04-23T16:33:56.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592a521d:::benchmark_data_smithi188_113957_object136240:head# 2024-04-23T16:33:56.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592aedb0:::benchmark_data_smithi188_113957_object120036:head# 2024-04-23T16:33:56.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592b5f69:::benchmark_data_smithi188_113957_object41664:head# 2024-04-23T16:33:56.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592ba768:::benchmark_data_smithi188_113957_object73714:head# 2024-04-23T16:33:56.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592bb830:::benchmark_data_smithi188_113957_object6926:head# 2024-04-23T16:33:56.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592bbffb:::benchmark_data_smithi188_113957_object62614:head# 2024-04-23T16:33:56.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592bcc04:::benchmark_data_smithi188_113957_object147930:head# 2024-04-23T16:33:56.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592bce13:::benchmark_data_smithi188_113957_object40565:head# 2024-04-23T16:33:56.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592c4d60:::benchmark_data_smithi188_113957_object141059:head# 2024-04-23T16:33:56.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592c6003:::benchmark_data_smithi188_113957_object26766:head# 2024-04-23T16:33:56.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592c67b5:::benchmark_data_smithi188_113957_object1504:head# 2024-04-23T16:33:56.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592cba15:::benchmark_data_smithi188_113957_object103234:head# 2024-04-23T16:33:56.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592cf82a:::benchmark_data_smithi188_113957_object77834:head# 2024-04-23T16:33:56.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592d0e36:::benchmark_data_smithi188_113957_object114255:head# 2024-04-23T16:33:56.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592e118b:::benchmark_data_smithi188_113957_object19260:head# 2024-04-23T16:33:56.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592ebe41:::benchmark_data_smithi188_113957_object149318:head# 2024-04-23T16:33:56.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592f63dc:::benchmark_data_smithi188_113957_object48917:head# 2024-04-23T16:33:56.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592f6f84:::benchmark_data_smithi188_113957_object42357:head# 2024-04-23T16:33:56.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.425 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:33:56.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:592fa333:::benchmark_data_smithi188_113957_object130766:head# 2024-04-23T16:33:56.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59302559:::benchmark_data_smithi188_113957_object75996:head# 2024-04-23T16:33:56.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5930948c:::benchmark_data_smithi188_113957_object12820:head# 2024-04-23T16:33:56.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5930a0ab:::benchmark_data_smithi188_113957_object49709:head# 2024-04-23T16:33:56.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5930cb69:::benchmark_data_smithi188_113957_object42605:head# 2024-04-23T16:33:56.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5930e144:::benchmark_data_smithi188_113957_object87653:head# 2024-04-23T16:33:56.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5930f79a:::benchmark_data_smithi188_113957_object48977:head# 2024-04-23T16:33:56.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5933f927:::benchmark_data_smithi188_113957_object74491:head# 2024-04-23T16:33:56.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:56.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:33:56.533 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:33:56.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59343c4c:::benchmark_data_smithi188_113957_object72019:head# 2024-04-23T16:33:56.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59357f72:::benchmark_data_smithi188_113957_object32690:head# 2024-04-23T16:33:56.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:593584b1:::benchmark_data_smithi188_113957_object122129:head# 2024-04-23T16:33:56.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5935860c:::benchmark_data_smithi188_113957_object128069:head# 2024-04-23T16:33:56.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:593595ad:::benchmark_data_smithi188_113957_object61470:head# 2024-04-23T16:33:56.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5935ceda:::benchmark_data_smithi188_113957_object150991:head# 2024-04-23T16:33:56.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59366b6e:::benchmark_data_smithi188_113957_object51762:head# 2024-04-23T16:33:56.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59369592:::benchmark_data_smithi188_113957_object143808:head# 2024-04-23T16:33:56.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5936efdb:::benchmark_data_smithi188_113957_object96841:head# 2024-04-23T16:33:56.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59370601:::benchmark_data_smithi188_113957_object78363:head# 2024-04-23T16:33:56.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59371aba:::benchmark_data_smithi188_113957_object97405:head# 2024-04-23T16:33:56.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5937680c:::benchmark_data_smithi188_113957_object61148:head# 2024-04-23T16:33:56.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5937adc0:::benchmark_data_smithi188_113957_object145197:head# 2024-04-23T16:33:56.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5937af2f:::benchmark_data_smithi188_113957_object94593:head# 2024-04-23T16:33:56.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5937b484:::benchmark_data_smithi188_113957_object6066:head# 2024-04-23T16:33:56.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5937cc02:::benchmark_data_smithi188_113957_object107215:head# 2024-04-23T16:33:56.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5937dd97:::benchmark_data_smithi188_113957_object112530:head# 2024-04-23T16:33:56.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59384fc2:::benchmark_data_smithi188_113957_object123663:head# 2024-04-23T16:33:56.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59388fc0:::benchmark_data_smithi188_113957_object132820:head# 2024-04-23T16:33:56.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5938fef6:::benchmark_data_smithi188_113957_object104551:head# 2024-04-23T16:33:56.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59399408:::benchmark_data_smithi188_113957_object151867:head# 2024-04-23T16:33:56.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:593a7810:::benchmark_data_smithi188_113957_object7214:head# 2024-04-23T16:33:56.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:593ae712:::benchmark_data_smithi188_113957_object100995:head# 2024-04-23T16:33:56.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:593b520d:::benchmark_data_smithi188_113957_object8933:head# 2024-04-23T16:33:56.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:593b5215:::benchmark_data_smithi188_113957_object144467:head# 2024-04-23T16:33:56.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:593b8453:::benchmark_data_smithi188_113957_object72331:head# 2024-04-23T16:33:56.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:593b987e:::benchmark_data_smithi188_113957_object129462:head# 2024-04-23T16:33:56.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:593bfd4c:::benchmark_data_smithi188_113957_object17152:head# 2024-04-23T16:33:56.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:593c96b9:::benchmark_data_smithi188_113957_object107271:head# 2024-04-23T16:33:56.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:593eb2ac:::benchmark_data_smithi188_113957_object76293:head# 2024-04-23T16:33:56.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:593eddb3:::benchmark_data_smithi188_113957_object26539:head# 2024-04-23T16:33:56.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:593efdcd:::benchmark_data_smithi188_113957_object38715:head# 2024-04-23T16:33:56.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:593f5f05:::benchmark_data_smithi188_113957_object4075:head# 2024-04-23T16:33:56.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:593ffd40:::benchmark_data_smithi188_113957_object73277:head# 2024-04-23T16:33:56.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59401941:::benchmark_data_smithi188_113957_object54381:head# 2024-04-23T16:33:56.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594040f3:::benchmark_data_smithi188_113957_object64122:head# 2024-04-23T16:33:56.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59411049:::benchmark_data_smithi188_113957_object112854:head# 2024-04-23T16:33:56.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594123d5:::benchmark_data_smithi188_113957_object36390:head# 2024-04-23T16:33:56.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59416665:::benchmark_data_smithi188_113957_object14919:head# 2024-04-23T16:33:56.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5942433b:::benchmark_data_smithi188_113957_object144749:head# 2024-04-23T16:33:56.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5942c1aa:::benchmark_data_smithi188_113957_object51627:head# 2024-04-23T16:33:56.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5943342b:::benchmark_data_smithi188_113957_object58117:head# 2024-04-23T16:33:56.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59435f99:::benchmark_data_smithi188_113957_object106395:head# 2024-04-23T16:33:56.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59437f27:::benchmark_data_smithi188_113957_object7353:head# 2024-04-23T16:33:56.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59438657:::benchmark_data_smithi188_113957_object99867:head# 2024-04-23T16:33:56.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5943b7ee:::benchmark_data_smithi188_113957_object72530:head# 2024-04-23T16:33:56.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59446154:::benchmark_data_smithi188_113957_object74015:head# 2024-04-23T16:33:56.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5944bf56:::benchmark_data_smithi188_113957_object81080:head# 2024-04-23T16:33:56.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5944c5ab:::benchmark_data_smithi188_113957_object118840:head# 2024-04-23T16:33:56.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59450fb2:::benchmark_data_smithi188_113957_object151492:head# 2024-04-23T16:33:56.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59453869:::benchmark_data_smithi188_113957_object123511:head# 2024-04-23T16:33:56.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59461db7:::benchmark_data_smithi188_113957_object117839:head# 2024-04-23T16:33:56.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59463daf:::benchmark_data_smithi188_113957_object92500:head# 2024-04-23T16:33:56.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594753d2:::benchmark_data_smithi188_113957_object140281:head# 2024-04-23T16:33:56.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5947afa7:::benchmark_data_smithi188_113957_object144552:head# 2024-04-23T16:33:56.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5947dbf5:::benchmark_data_smithi188_113957_object96148:head# 2024-04-23T16:33:56.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594872f1:::benchmark_data_smithi188_113957_object84363:head# 2024-04-23T16:33:56.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594880e1:::benchmark_data_smithi188_113957_object45103:head# 2024-04-23T16:33:56.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594adb72:::benchmark_data_smithi188_113957_object16091:head# 2024-04-23T16:33:56.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594b9b3b:::benchmark_data_smithi188_113957_object140730:head# 2024-04-23T16:33:56.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594c41da:::benchmark_data_smithi188_113957_object9373:head# 2024-04-23T16:33:56.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594ce330:::benchmark_data_smithi188_113957_object20130:head# 2024-04-23T16:33:56.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594d74f4:::benchmark_data_smithi188_113957_object104161:head# 2024-04-23T16:33:56.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594d9387:::benchmark_data_smithi188_113957_object43554:head# 2024-04-23T16:33:56.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594d9dde:::benchmark_data_smithi188_113957_object139955:head# 2024-04-23T16:33:56.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594dc1e7:::benchmark_data_smithi188_113957_object144385:head# 2024-04-23T16:33:56.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594dde5e:::benchmark_data_smithi188_113957_object30659:head# 2024-04-23T16:33:56.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594dfa48:::benchmark_data_smithi188_113957_object62346:head# 2024-04-23T16:33:56.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594e407b:::benchmark_data_smithi188_113957_object116470:head# 2024-04-23T16:33:56.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594e70b9:::benchmark_data_smithi188_113957_object118672:head# 2024-04-23T16:33:56.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594e7737:::benchmark_data_smithi188_113957_object122165:head# 2024-04-23T16:33:56.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594f3597:::benchmark_data_smithi188_113957_object77526:head# 2024-04-23T16:33:56.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594fa067:::benchmark_data_smithi188_113957_object82136:head# 2024-04-23T16:33:56.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:594fbbf5:::benchmark_data_smithi188_113957_object127751:head# 2024-04-23T16:33:56.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59501d48:::benchmark_data_smithi188_113957_object39758:head# 2024-04-23T16:33:56.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5950284e:::benchmark_data_smithi188_113957_object91339:head# 2024-04-23T16:33:56.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59503644:::benchmark_data_smithi188_113957_object6954:head# 2024-04-23T16:33:56.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59508517:::benchmark_data_smithi188_113957_object36706:head# 2024-04-23T16:33:56.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5950ac3a:::benchmark_data_smithi188_113957_object32186:head# 2024-04-23T16:33:56.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5950d5e1:::benchmark_data_smithi188_113957_object67423:head# 2024-04-23T16:33:56.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5950edce:::benchmark_data_smithi188_113957_object100768:head# 2024-04-23T16:33:56.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59515dc6:::benchmark_data_smithi188_113957_object95820:head# 2024-04-23T16:33:56.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5951b096:::benchmark_data_smithi188_113957_object122255:head# 2024-04-23T16:33:56.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5951b755:::benchmark_data_smithi188_113957_object120878:head# 2024-04-23T16:33:56.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.621 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5951b8fc:::benchmark_data_smithi188_113957_object14174:head# 2024-04-23T16:33:56.621 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.621 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5951c9ca:::benchmark_data_smithi188_113957_object68890:head# 2024-04-23T16:33:56.621 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.621 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595258f0:::benchmark_data_smithi188_113957_object84568:head# 2024-04-23T16:33:56.621 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.621 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59527dd1:::benchmark_data_smithi188_113957_object116046:head# 2024-04-23T16:33:56.621 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.621 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5953f45d:::benchmark_data_smithi188_113957_object132700:head# 2024-04-23T16:33:56.621 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.621 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595455a8:::benchmark_data_smithi188_113957_object40135:head# 2024-04-23T16:33:56.621 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.621 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5954a672:::benchmark_data_smithi188_113957_object123300:head# 2024-04-23T16:33:56.621 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5954f2ac:::benchmark_data_smithi188_113957_object74850:head# 2024-04-23T16:33:56.622 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5956ed2e:::benchmark_data_smithi188_113957_object136250:head# 2024-04-23T16:33:56.622 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59582854:::benchmark_data_smithi188_113957_object3839:head# 2024-04-23T16:33:56.622 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595863ad:::benchmark_data_smithi188_113957_object146411:head# 2024-04-23T16:33:56.622 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59596355:::benchmark_data_smithi188_113957_object91689:head# 2024-04-23T16:33:56.622 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5959d4d6:::benchmark_data_smithi188_113957_object89999:head# 2024-04-23T16:33:56.622 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.622 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595a0405:::benchmark_data_smithi188_113957_object134398:head# 2024-04-23T16:33:56.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595a26fa:::benchmark_data_smithi188_113957_object131428:head# 2024-04-23T16:33:56.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595a342f:::benchmark_data_smithi188_113957_object144901:head# 2024-04-23T16:33:56.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595a43f0:::benchmark_data_smithi188_113957_object127427:head# 2024-04-23T16:33:56.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595a5768:::benchmark_data_smithi188_113957_object140674:head# 2024-04-23T16:33:56.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595aef84:::benchmark_data_smithi188_113957_object6958:head# 2024-04-23T16:33:56.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595b0eef:::benchmark_data_smithi188_113957_object141334:head# 2024-04-23T16:33:56.623 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.623 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595b62f1:::benchmark_data_smithi188_113957_object49004:head# 2024-04-23T16:33:56.624 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.624 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595c0a97:::benchmark_data_smithi188_113957_object57307:head# 2024-04-23T16:33:56.624 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.624 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595ce1d2:::benchmark_data_smithi188_113957_object75931:head# 2024-04-23T16:33:56.624 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.624 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595dc850:::benchmark_data_smithi188_113957_object69188:head# 2024-04-23T16:33:56.624 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.624 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595dcac1:::benchmark_data_smithi188_113957_object97576:head# 2024-04-23T16:33:56.624 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.624 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595dfe36:::benchmark_data_smithi188_113957_object61382:head# 2024-04-23T16:33:56.624 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.624 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595ecbfb:::benchmark_data_smithi188_113957_object116968:head# 2024-04-23T16:33:56.624 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595f3ce3:::benchmark_data_smithi188_113957_object131799:head# 2024-04-23T16:33:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595fa8ea:::benchmark_data_smithi188_113957_object107933:head# 2024-04-23T16:33:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:595fd4df:::benchmark_data_smithi188_113957_object140043:head# 2024-04-23T16:33:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59604c9a:::benchmark_data_smithi188_113957_object92946:head# 2024-04-23T16:33:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5960950f:::benchmark_data_smithi188_113957_object65385:head# 2024-04-23T16:33:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5960a99b:::benchmark_data_smithi188_113957_object39328:head# 2024-04-23T16:33:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59612725:::benchmark_data_smithi188_113957_object118409:head# 2024-04-23T16:33:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5961338e:::benchmark_data_smithi188_113957_object37648:head# 2024-04-23T16:33:56.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59617819:::benchmark_data_smithi188_113957_object98035:head# 2024-04-23T16:33:56.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5961a519:::benchmark_data_smithi188_113957_object64436:head# 2024-04-23T16:33:56.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59620a97:::benchmark_data_smithi188_113957_object83906:head# 2024-04-23T16:33:56.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:596249b1:::benchmark_data_smithi188_113957_object86620:head# 2024-04-23T16:33:56.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59626480:::benchmark_data_smithi188_113957_object121503:head# 2024-04-23T16:33:56.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.626 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59638898:::benchmark_data_smithi188_113957_object98365:head# 2024-04-23T16:33:56.626 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5963c4f8:::benchmark_data_smithi188_113957_object119168:head# 2024-04-23T16:33:56.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:596423ee:::benchmark_data_smithi188_113957_object20145:head# 2024-04-23T16:33:56.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59643c2b:::benchmark_data_smithi188_113957_object109740:head# 2024-04-23T16:33:56.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59652270:::benchmark_data_smithi188_113957_object121203:head# 2024-04-23T16:33:56.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59654b1e:::benchmark_data_smithi188_113957_object50732:head# 2024-04-23T16:33:56.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:596672b9:::benchmark_data_smithi188_113957_object31519:head# 2024-04-23T16:33:56.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.627 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5967737d:::benchmark_data_smithi188_113957_object72956:head# 2024-04-23T16:33:56.627 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.628 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59677be0:::benchmark_data_smithi188_113957_object18886:head# 2024-04-23T16:33:56.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.628 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59691134:::benchmark_data_smithi188_113957_object34621:head# 2024-04-23T16:33:56.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.628 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5969af85:::benchmark_data_smithi188_113957_object92681:head# 2024-04-23T16:33:56.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.628 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:596b8ba0:::benchmark_data_smithi188_113957_object42550:head# 2024-04-23T16:33:56.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.628 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:596bc705:::benchmark_data_smithi188_113957_object137452:head# 2024-04-23T16:33:56.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.628 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:596c2c88:::benchmark_data_smithi188_113957_object43846:head# 2024-04-23T16:33:56.628 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.628 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:596c4c55:::benchmark_data_smithi188_113957_object150171:head# 2024-04-23T16:33:56.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:596cbd5f:::benchmark_data_smithi188_113957_object68406:head# 2024-04-23T16:33:56.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:596cd488:::benchmark_data_smithi188_113957_object26669:head# 2024-04-23T16:33:56.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:596d3a1d:::benchmark_data_smithi188_113957_object143761:head# 2024-04-23T16:33:56.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:596d57c1:::benchmark_data_smithi188_113957_object90015:head# 2024-04-23T16:33:56.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:596d6433:::benchmark_data_smithi188_113957_object714:head# 2024-04-23T16:33:56.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:596d8692:::benchmark_data_smithi188_113957_object59172:head# 2024-04-23T16:33:56.629 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.629 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:596d96df:::benchmark_data_smithi188_113957_object35673:head# 2024-04-23T16:33:56.630 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.630 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:596ea340:::benchmark_data_smithi188_113957_object7071:head# 2024-04-23T16:33:56.630 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.630 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:596f170a:::benchmark_data_smithi188_113957_object11447:head# 2024-04-23T16:33:56.630 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.630 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59707a86:::benchmark_data_smithi188_113957_object76075:head# 2024-04-23T16:33:56.630 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.630 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59709ed8:::benchmark_data_smithi188_113957_object105851:head# 2024-04-23T16:33:56.630 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.630 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5970bac9:::benchmark_data_smithi188_113957_object152848:head# 2024-04-23T16:33:56.630 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.630 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59721aec:::benchmark_data_smithi188_113957_object125883:head# 2024-04-23T16:33:56.630 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.631 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59737f61:::benchmark_data_smithi188_113957_object91776:head# 2024-04-23T16:33:56.631 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.631 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5974e0d4:::benchmark_data_smithi188_113957_object106160:head# 2024-04-23T16:33:56.631 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.631 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59752139:::benchmark_data_smithi188_113957_object129778:head# 2024-04-23T16:33:56.631 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.631 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5975e107:::benchmark_data_smithi188_113957_object18617:head# 2024-04-23T16:33:56.631 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.631 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5976082a:::benchmark_data_smithi188_113957_object43774:head# 2024-04-23T16:33:56.631 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.631 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:597639f6:::benchmark_data_smithi188_113957_object144710:head# 2024-04-23T16:33:56.631 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.631 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59774f65:::benchmark_data_smithi188_113957_object66154:head# 2024-04-23T16:33:56.631 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59780a35:::benchmark_data_smithi188_113957_object11414:head# 2024-04-23T16:33:56.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59782210:::benchmark_data_smithi188_113957_object136004:head# 2024-04-23T16:33:56.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5979613a:::benchmark_data_smithi188_113957_object27171:head# 2024-04-23T16:33:56.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5979de20:::benchmark_data_smithi188_113957_object35321:head# 2024-04-23T16:33:56.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5979f1be:::benchmark_data_smithi188_113957_object94800:head# 2024-04-23T16:33:56.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:597a3450:::benchmark_data_smithi188_113957_object121407:head# 2024-04-23T16:33:56.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.632 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:597ab1c2:::benchmark_data_smithi188_113957_object116172:head# 2024-04-23T16:33:56.632 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.633 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:597accbf:::benchmark_data_smithi188_113957_object91911:head# 2024-04-23T16:33:56.633 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.633 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:597ae803:::benchmark_data_smithi188_113957_object152720:head# 2024-04-23T16:33:56.633 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.633 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:597b9e64:::benchmark_data_smithi188_113957_object117563:head# 2024-04-23T16:33:56.633 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.633 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:597c9f2d:::benchmark_data_smithi188_113957_object123520:head# 2024-04-23T16:33:56.633 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.633 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:597caea6:::benchmark_data_smithi188_113957_object72081:head# 2024-04-23T16:33:56.633 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.633 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:597d551f:::benchmark_data_smithi188_113957_object110790:head# 2024-04-23T16:33:56.633 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.633 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:597dd751:::benchmark_data_smithi188_113957_object1837:head# 2024-04-23T16:33:56.633 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:597e785e:::benchmark_data_smithi188_113957_object107883:head# 2024-04-23T16:33:56.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:597f90ff:::benchmark_data_smithi188_113957_object54911:head# 2024-04-23T16:33:56.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:597fb3f2:::benchmark_data_smithi188_113957_object95060:head# 2024-04-23T16:33:56.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:597fb511:::benchmark_data_smithi188_113957_object75712:head# 2024-04-23T16:33:56.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:597fd66b:::benchmark_data_smithi188_113957_object119491:head# 2024-04-23T16:33:56.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59800f00:::benchmark_data_smithi188_113957_object37422:head# 2024-04-23T16:33:56.634 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.634 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5980c5d5:::benchmark_data_smithi188_113957_object1606:head# 2024-04-23T16:33:56.635 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.635 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5981144c:::benchmark_data_smithi188_113957_object74494:head# 2024-04-23T16:33:56.635 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.635 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59816f86:::benchmark_data_smithi188_113957_object152862:head# 2024-04-23T16:33:56.635 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.635 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598293ba:::benchmark_data_smithi188_113957_object49209:head# 2024-04-23T16:33:56.635 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.635 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5982b706:::benchmark_data_smithi188_113957_object89276:head# 2024-04-23T16:33:56.635 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.635 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5982bbe8:::benchmark_data_smithi188_113957_object88837:head# 2024-04-23T16:33:56.635 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.635 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5982d6fa:::benchmark_data_smithi188_113957_object70274:head# 2024-04-23T16:33:56.635 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.635 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59836abe:::benchmark_data_smithi188_113957_object92914:head# 2024-04-23T16:33:56.636 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.636 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5983e598:::benchmark_data_smithi188_113957_object106973:head# 2024-04-23T16:33:56.636 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.636 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5984900c:::benchmark_data_smithi188_113957_object98960:head# 2024-04-23T16:33:56.636 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.636 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59851ef1:::benchmark_data_smithi188_113957_object130183:head# 2024-04-23T16:33:56.636 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.636 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59856d9b:::benchmark_data_smithi188_113957_object143136:head# 2024-04-23T16:33:56.636 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.636 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598692cb:::benchmark_data_smithi188_113957_object88408:head# 2024-04-23T16:33:56.636 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.636 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5986f08e:::benchmark_data_smithi188_113957_object18669:head# 2024-04-23T16:33:56.636 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.637 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59871447:::benchmark_data_smithi188_113957_object91539:head# 2024-04-23T16:33:56.637 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.637 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5987b47b:::benchmark_data_smithi188_113957_object139446:head# 2024-04-23T16:33:56.637 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.637 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59887a86:::benchmark_data_smithi188_113957_object35003:head# 2024-04-23T16:33:56.637 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.637 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598892b2:::benchmark_data_smithi188_113957_object56905:head# 2024-04-23T16:33:56.637 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.637 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5988dd69:::benchmark_data_smithi188_113957_object13881:head# 2024-04-23T16:33:56.637 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.637 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5988f265:::benchmark_data_smithi188_113957_object140001:head# 2024-04-23T16:33:56.637 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.637 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59893b51:::benchmark_data_smithi188_113957_object83459:head# 2024-04-23T16:33:56.637 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5989b845:::benchmark_data_smithi188_113957_object114857:head# 2024-04-23T16:33:56.638 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598a6333:::benchmark_data_smithi188_113957_object91064:head# 2024-04-23T16:33:56.638 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598a6431:::benchmark_data_smithi188_113957_object152433:head# 2024-04-23T16:33:56.638 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598ac039:::benchmark_data_smithi188_113957_object78403:head# 2024-04-23T16:33:56.638 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598ad413:::benchmark_data_smithi188_113957_object15980:head# 2024-04-23T16:33:56.638 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598b4d69:::benchmark_data_smithi188_113957_object34728:head# 2024-04-23T16:33:56.638 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.638 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598b5235:::benchmark_data_smithi188_113957_object136671:head# 2024-04-23T16:33:56.639 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.639 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598b854b:::benchmark_data_smithi188_113957_object46210:head# 2024-04-23T16:33:56.639 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.639 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598bead3:::benchmark_data_smithi188_113957_object74489:head# 2024-04-23T16:33:56.639 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.639 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598c0c86:::benchmark_data_smithi188_113957_object70710:head# 2024-04-23T16:33:56.639 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.639 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598c1632:::benchmark_data_smithi188_113957_object89846:head# 2024-04-23T16:33:56.639 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.639 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598c2a9a:::benchmark_data_smithi188_113957_object126884:head# 2024-04-23T16:33:56.639 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.639 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598c3b93:::benchmark_data_smithi188_113957_object62032:head# 2024-04-23T16:33:56.639 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.639 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598ccd51:::benchmark_data_smithi188_113957_object47134:head# 2024-04-23T16:33:56.639 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.640 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598e39a0:::benchmark_data_smithi188_113957_object113591:head# 2024-04-23T16:33:56.640 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.640 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598e5c76:::benchmark_data_smithi188_113957_object119840:head# 2024-04-23T16:33:56.640 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.640 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598e64a9:::benchmark_data_smithi188_113957_object14020:head# 2024-04-23T16:33:56.640 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.640 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598f53b5:::benchmark_data_smithi188_113957_object28078:head# 2024-04-23T16:33:56.640 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.640 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:598fe280:::benchmark_data_smithi188_113957_object119788:head# 2024-04-23T16:33:56.640 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.640 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5990d742:::benchmark_data_smithi188_113957_object139378:head# 2024-04-23T16:33:56.640 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.640 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59919a41:::benchmark_data_smithi188_113957_object134808:head# 2024-04-23T16:33:56.640 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.641 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5991f8ce:::benchmark_data_smithi188_113957_object62868:head# 2024-04-23T16:33:56.641 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.641 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:599268a0:::benchmark_data_smithi188_113957_object48780:head# 2024-04-23T16:33:56.641 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.641 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:599270c4:::benchmark_data_smithi188_113957_object132217:head# 2024-04-23T16:33:56.641 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.641 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59931032:::benchmark_data_smithi188_113957_object130982:head# 2024-04-23T16:33:56.641 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.641 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59933c34:::benchmark_data_smithi188_113957_object144222:head# 2024-04-23T16:33:56.641 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.641 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59933f0c:::benchmark_data_smithi188_113957_object87885:head# 2024-04-23T16:33:56.641 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.641 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:599347ce:::benchmark_data_smithi188_113957_object66551:head# 2024-04-23T16:33:56.641 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.642 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59946143:::benchmark_data_smithi188_113957_object49466:head# 2024-04-23T16:33:56.642 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.642 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5994e0b2:::benchmark_data_smithi188_113957_object102473:head# 2024-04-23T16:33:56.642 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.642 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59954d74:::benchmark_data_smithi188_113957_object98450:head# 2024-04-23T16:33:56.642 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.642 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59958664:::benchmark_data_smithi188_113957_object89203:head# 2024-04-23T16:33:56.642 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.642 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59958dcd:::benchmark_data_smithi188_113957_object18730:head# 2024-04-23T16:33:56.642 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.642 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5996f436:::benchmark_data_smithi188_113957_object146427:head# 2024-04-23T16:33:56.642 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.642 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59972335:::benchmark_data_smithi188_113957_object19929:head# 2024-04-23T16:33:56.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59978c2e:::benchmark_data_smithi188_113957_object66044:head# 2024-04-23T16:33:56.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5997c938:::benchmark_data_smithi188_113957_object91984:head# 2024-04-23T16:33:56.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59981de3:::benchmark_data_smithi188_113957_object143147:head# 2024-04-23T16:33:56.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5998d703:::benchmark_data_smithi188_113957_object86965:head# 2024-04-23T16:33:56.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59993bd5:::benchmark_data_smithi188_113957_object130653:head# 2024-04-23T16:33:56.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5999d1b3:::benchmark_data_smithi188_113957_object56257:head# 2024-04-23T16:33:56.643 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.643 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:599a4084:::benchmark_data_smithi188_113957_object3898:head# 2024-04-23T16:33:56.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.644 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:599a47d6:::benchmark_data_smithi188_113957_object49666:head# 2024-04-23T16:33:56.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.644 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:599a5856:::benchmark_data_smithi188_113957_object31174:head# 2024-04-23T16:33:56.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.644 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:599ad436:::benchmark_data_smithi188_113957_object102456:head# 2024-04-23T16:33:56.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.644 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:599b052f:::benchmark_data_smithi188_113957_object44521:head# 2024-04-23T16:33:56.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.644 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:599b6ee9:::benchmark_data_smithi188_113957_object101510:head# 2024-04-23T16:33:56.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.644 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:599c17fb:::benchmark_data_smithi188_113957_object70141:head# 2024-04-23T16:33:56.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.644 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:599c294d:::benchmark_data_smithi188_113957_object134020:head# 2024-04-23T16:33:56.644 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.645 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:599e40cc:::benchmark_data_smithi188_113957_object98502:head# 2024-04-23T16:33:56.645 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.645 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:599e9648:::benchmark_data_smithi188_113957_object94691:head# 2024-04-23T16:33:56.645 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.645 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:599eb36f:::benchmark_data_smithi188_113957_object136715:head# 2024-04-23T16:33:56.645 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.645 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:599fd775:::benchmark_data_smithi188_113957_object138955:head# 2024-04-23T16:33:56.645 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.645 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:599fe654:::benchmark_data_smithi188_113957_object66561:head# 2024-04-23T16:33:56.645 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.645 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a00d8e:::benchmark_data_smithi188_113957_object57572:head# 2024-04-23T16:33:56.645 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.645 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a072b6:::benchmark_data_smithi188_113957_object45877:head# 2024-04-23T16:33:56.645 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.646 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a08cd6:::benchmark_data_smithi188_113957_object150824:head# 2024-04-23T16:33:56.646 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.646 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a0e946:::benchmark_data_smithi188_113957_object52521:head# 2024-04-23T16:33:56.646 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.646 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a16d21:::benchmark_data_smithi188_113957_object81905:head# 2024-04-23T16:33:56.646 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.646 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a1dcdc:::benchmark_data_smithi188_113957_object52389:head# 2024-04-23T16:33:56.646 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.646 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a249ae:::benchmark_data_smithi188_113957_object22244:head# 2024-04-23T16:33:56.646 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.646 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a2558f:::benchmark_data_smithi188_113957_object30963:head# 2024-04-23T16:33:56.646 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.646 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a30b5d:::benchmark_data_smithi188_113957_object36063:head# 2024-04-23T16:33:56.646 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a32a7d:::benchmark_data_smithi188_113957_object45193:head# 2024-04-23T16:33:56.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a3fb51:::benchmark_data_smithi188_113957_object11091:head# 2024-04-23T16:33:56.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a410c3:::benchmark_data_smithi188_113957_object100111:head# 2024-04-23T16:33:56.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a42daa:::benchmark_data_smithi188_113957_object77314:head# 2024-04-23T16:33:56.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a54e0f:::benchmark_data_smithi188_113957_object152117:head# 2024-04-23T16:33:56.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a5b3bc:::benchmark_data_smithi188_113957_object98099:head# 2024-04-23T16:33:56.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.647 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a5f5fd:::benchmark_data_smithi188_113957_object77567:head# 2024-04-23T16:33:56.647 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a62438:::benchmark_data_smithi188_113957_object122854:head# 2024-04-23T16:33:56.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a65c9d:::benchmark_data_smithi188_113957_object78267:head# 2024-04-23T16:33:56.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a68246:::benchmark_data_smithi188_113957_object100788:head# 2024-04-23T16:33:56.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a6eb0e:::benchmark_data_smithi188_113957_object66508:head# 2024-04-23T16:33:56.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a6fd19:::benchmark_data_smithi188_113957_object82107:head# 2024-04-23T16:33:56.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a726d5:::benchmark_data_smithi188_113957_object13689:head# 2024-04-23T16:33:56.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.648 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a7df1f:::benchmark_data_smithi188_113957_object77106:head# 2024-04-23T16:33:56.648 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a7f463:::benchmark_data_smithi188_113957_object68237:head# 2024-04-23T16:33:56.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a82e19:::benchmark_data_smithi188_113957_object80167:head# 2024-04-23T16:33:56.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a8b158:::benchmark_data_smithi188_113957_object64146:head# 2024-04-23T16:33:56.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a8d1c3:::benchmark_data_smithi188_113957_object68541:head# 2024-04-23T16:33:56.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a98657:::benchmark_data_smithi188_113957_object76958:head# 2024-04-23T16:33:56.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59a9c2c9:::benchmark_data_smithi188_113957_object30784:head# 2024-04-23T16:33:56.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.649 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59aa1545:::benchmark_data_smithi188_113957_object62006:head# 2024-04-23T16:33:56.649 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59aa3554:::benchmark_data_smithi188_113957_object90882:head# 2024-04-23T16:33:56.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59aac73b:::benchmark_data_smithi188_113957_object56640:head# 2024-04-23T16:33:56.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ab1cf0:::benchmark_data_smithi188_113957_object100233:head# 2024-04-23T16:33:56.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ab6b42:::benchmark_data_smithi188_113957_object135710:head# 2024-04-23T16:33:56.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ab724d:::benchmark_data_smithi188_113957_object35065:head# 2024-04-23T16:33:56.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ac6137:::benchmark_data_smithi188_113957_object85189:head# 2024-04-23T16:33:56.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.650 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ace9ee:::benchmark_data_smithi188_113957_object83814:head# 2024-04-23T16:33:56.650 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59acefe8:::benchmark_data_smithi188_113957_object143676:head# 2024-04-23T16:33:56.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ad55e2:::benchmark_data_smithi188_113957_object105654:head# 2024-04-23T16:33:56.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59af5881:::benchmark_data_smithi188_113957_object72365:head# 2024-04-23T16:33:56.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59af788c:::benchmark_data_smithi188_113957_object143259:head# 2024-04-23T16:33:56.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59afc02f:::benchmark_data_smithi188_113957_object134449:head# 2024-04-23T16:33:56.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59affeb2:::benchmark_data_smithi188_113957_object123597:head# 2024-04-23T16:33:56.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.651 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b07c07:::benchmark_data_smithi188_113957_object122147:head# 2024-04-23T16:33:56.651 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.652 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b08009:::benchmark_data_smithi188_113957_object113978:head# 2024-04-23T16:33:56.652 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.652 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b0e3ae:::benchmark_data_smithi188_113957_object135021:head# 2024-04-23T16:33:56.652 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.652 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b1c541:::benchmark_data_smithi188_113957_object62181:head# 2024-04-23T16:33:56.652 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.652 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b1e30b:::benchmark_data_smithi188_113957_object26147:head# 2024-04-23T16:33:56.652 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.652 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b1e6ea:::benchmark_data_smithi188_113957_object73033:head# 2024-04-23T16:33:56.652 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.652 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b28696:::benchmark_data_smithi188_113957_object49048:head# 2024-04-23T16:33:56.652 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.652 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b2bc46:::benchmark_data_smithi188_113957_object145027:head# 2024-04-23T16:33:56.653 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.653 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b2bdb4:::benchmark_data_smithi188_113957_object13338:head# 2024-04-23T16:33:56.653 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.653 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b2f757:::benchmark_data_smithi188_113957_object9469:head# 2024-04-23T16:33:56.653 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.653 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b39746:::benchmark_data_smithi188_113957_object87106:head# 2024-04-23T16:33:56.653 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.653 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b451d3:::benchmark_data_smithi188_113957_object30706:head# 2024-04-23T16:33:56.653 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.653 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b4cd78:::benchmark_data_smithi188_113957_object78732:head# 2024-04-23T16:33:56.653 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.653 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b536be:::benchmark_data_smithi188_113957_object10838:head# 2024-04-23T16:33:56.653 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.653 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b59f95:::benchmark_data_smithi188_113957_object104124:head# 2024-04-23T16:33:56.654 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.654 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b6f91b:::benchmark_data_smithi188_113957_object75282:head# 2024-04-23T16:33:56.654 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.654 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b7141e:::benchmark_data_smithi188_113957_object150701:head# 2024-04-23T16:33:56.654 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.654 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b77f92:::benchmark_data_smithi188_113957_object152114:head# 2024-04-23T16:33:56.654 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.654 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b79bcb:::benchmark_data_smithi188_113957_object132723:head# 2024-04-23T16:33:56.654 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.654 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b84408:::benchmark_data_smithi188_113957_object75690:head# 2024-04-23T16:33:56.654 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.654 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b85e05:::benchmark_data_smithi188_113957_object56319:head# 2024-04-23T16:33:56.654 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b9a301:::benchmark_data_smithi188_113957_object44298:head# 2024-04-23T16:33:56.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b9b26e:::benchmark_data_smithi188_113957_object44474:head# 2024-04-23T16:33:56.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59b9f9ca:::benchmark_data_smithi188_113957_object88373:head# 2024-04-23T16:33:56.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ba88c0:::benchmark_data_smithi188_113957_object20392:head# 2024-04-23T16:33:56.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ba8dde:::benchmark_data_smithi188_113957_object4982:head# 2024-04-23T16:33:56.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59bb7cb5:::benchmark_data_smithi188_113957_object14788:head# 2024-04-23T16:33:56.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.655 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59bbdf28:::benchmark_data_smithi188_113957_object134343:head# 2024-04-23T16:33:56.655 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59bbe630:::benchmark_data_smithi188_113957_object56314:head# 2024-04-23T16:33:56.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59bbf6d1:::benchmark_data_smithi188_113957_object20293:head# 2024-04-23T16:33:56.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59bc3b36:::benchmark_data_smithi188_113957_object100255:head# 2024-04-23T16:33:56.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59bc89a6:::benchmark_data_smithi188_113957_object8876:head# 2024-04-23T16:33:56.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59bcd274:::benchmark_data_smithi188_113957_object131862:head# 2024-04-23T16:33:56.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59bcea2f:::benchmark_data_smithi188_113957_object51434:head# 2024-04-23T16:33:56.656 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.656 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59bd7baf:::benchmark_data_smithi188_113957_object22058:head# 2024-04-23T16:33:56.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59bddd6f:::benchmark_data_smithi188_113957_object35502:head# 2024-04-23T16:33:56.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59bdfbd9:::benchmark_data_smithi188_113957_object65475:head# 2024-04-23T16:33:56.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59be5fb9:::benchmark_data_smithi188_113957_object12657:head# 2024-04-23T16:33:56.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59bedb89:::benchmark_data_smithi188_113957_object97515:head# 2024-04-23T16:33:56.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59bf1d94:::benchmark_data_smithi188_113957_object84908:head# 2024-04-23T16:33:56.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59bfcabc:::benchmark_data_smithi188_113957_object10535:head# 2024-04-23T16:33:56.657 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.657 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c132cc:::benchmark_data_smithi188_113957_object74039:head# 2024-04-23T16:33:56.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.658 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c1601b:::benchmark_data_smithi188_113957_object142381:head# 2024-04-23T16:33:56.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.658 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c19e33:::benchmark_data_smithi188_113957_object130884:head# 2024-04-23T16:33:56.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.658 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c1c5fb:::benchmark_data_smithi188_113957_object123587:head# 2024-04-23T16:33:56.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.658 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c1f715:::benchmark_data_smithi188_113957_object108576:head# 2024-04-23T16:33:56.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.658 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c2229f:::benchmark_data_smithi188_113957_object11572:head# 2024-04-23T16:33:56.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.658 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c26838:::benchmark_data_smithi188_113957_object81434:head# 2024-04-23T16:33:56.658 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.659 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c2b4b9:::benchmark_data_smithi188_113957_object43493:head# 2024-04-23T16:33:56.659 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.659 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c32809:::benchmark_data_smithi188_113957_object23850:head# 2024-04-23T16:33:56.659 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.659 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c38db3:::benchmark_data_smithi188_113957_object10284:head# 2024-04-23T16:33:56.659 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.659 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c3d5b8:::benchmark_data_smithi188_113957_object6723:head# 2024-04-23T16:33:56.659 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.659 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c45c98:::benchmark_data_smithi188_113957_object28212:head# 2024-04-23T16:33:56.659 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.659 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c476c8:::benchmark_data_smithi188_113957_object77031:head# 2024-04-23T16:33:56.659 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.659 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c4c52e:::benchmark_data_smithi188_113957_object29476:head# 2024-04-23T16:33:56.659 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.659 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c54062:::benchmark_data_smithi188_113957_object127569:head# 2024-04-23T16:33:56.660 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.660 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c5878c:::benchmark_data_smithi188_113957_object55698:head# 2024-04-23T16:33:56.660 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.660 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c6014b:::benchmark_data_smithi188_113957_object7859:head# 2024-04-23T16:33:56.660 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.660 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c63861:::benchmark_data_smithi188_113957_object34298:head# 2024-04-23T16:33:56.660 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.660 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c734ba:::benchmark_data_smithi188_113957_object142508:head# 2024-04-23T16:33:56.660 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.660 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c761df:::benchmark_data_smithi188_113957_object126994:head# 2024-04-23T16:33:56.660 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.660 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c8cbd3:::benchmark_data_smithi188_113957_object17408:head# 2024-04-23T16:33:56.660 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.660 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c938cc:::benchmark_data_smithi188_113957_object88960:head# 2024-04-23T16:33:56.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c952ee:::benchmark_data_smithi188_113957_object130044:head# 2024-04-23T16:33:56.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59c99f4d:::benchmark_data_smithi188_113957_object89272:head# 2024-04-23T16:33:56.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ca0298:::benchmark_data_smithi188_113957_object57863:head# 2024-04-23T16:33:56.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ca3f8f:::benchmark_data_smithi188_113957_object114411:head# 2024-04-23T16:33:56.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ca400f:::benchmark_data_smithi188_113957_object25269:head# 2024-04-23T16:33:56.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59cabfd2:::benchmark_data_smithi188_113957_object17283:head# 2024-04-23T16:33:56.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.661 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59cb8897:::benchmark_data_smithi188_113957_object25205:head# 2024-04-23T16:33:56.661 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ccc050:::benchmark_data_smithi188_113957_object144495:head# 2024-04-23T16:33:56.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59cd0804:::benchmark_data_smithi188_113957_object101702:head# 2024-04-23T16:33:56.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59cd5b01:::benchmark_data_smithi188_113957_object20735:head# 2024-04-23T16:33:56.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59cdd16f:::benchmark_data_smithi188_113957_object142015:head# 2024-04-23T16:33:56.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ceac5c:::benchmark_data_smithi188_113957_object118282:head# 2024-04-23T16:33:56.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59cec7ce:::benchmark_data_smithi188_113957_object93944:head# 2024-04-23T16:33:56.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.662 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59cf2577:::benchmark_data_smithi188_113957_object40226:head# 2024-04-23T16:33:56.662 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59cf5f55:::benchmark_data_smithi188_113957_object4408:head# 2024-04-23T16:33:56.663 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59cf67e3:::benchmark_data_smithi188_113957_object3095:head# 2024-04-23T16:33:56.663 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59cfe45b:::benchmark_data_smithi188_113957_object47433:head# 2024-04-23T16:33:56.663 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59cfeedd:::benchmark_data_smithi188_113957_object82707:head# 2024-04-23T16:33:56.663 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59cff00e:::benchmark_data_smithi188_113957_object59326:head# 2024-04-23T16:33:56.663 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d011be:::benchmark_data_smithi188_113957_object55073:head# 2024-04-23T16:33:56.663 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.663 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d08a7f:::benchmark_data_smithi188_113957_object78773:head# 2024-04-23T16:33:56.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d16581:::benchmark_data_smithi188_113957_object97557:head# 2024-04-23T16:33:56.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d17606:::benchmark_data_smithi188_113957_object135765:head# 2024-04-23T16:33:56.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d22f2b:::benchmark_data_smithi188_113957_object128915:head# 2024-04-23T16:33:56.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d2cb84:::benchmark_data_smithi188_113957_object70701:head# 2024-04-23T16:33:56.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d30063:::benchmark_data_smithi188_113957_object105387:head# 2024-04-23T16:33:56.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d3130e:::benchmark_data_smithi188_113957_object72680:head# 2024-04-23T16:33:56.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.664 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d31ed7:::benchmark_data_smithi188_113957_object48854:head# 2024-04-23T16:33:56.664 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d37ad0:::benchmark_data_smithi188_113957_object97864:head# 2024-04-23T16:33:56.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d3fcee:::benchmark_data_smithi188_113957_object66813:head# 2024-04-23T16:33:56.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d40773:::benchmark_data_smithi188_113957_object121381:head# 2024-04-23T16:33:56.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d49277:::benchmark_data_smithi188_113957_object143871:head# 2024-04-23T16:33:56.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d5d585:::benchmark_data_smithi188_113957_object128323:head# 2024-04-23T16:33:56.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.665 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d6634a:::benchmark_data_smithi188_113957_object89376:head# 2024-04-23T16:33:56.665 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d6f4b4:::benchmark_data_smithi188_113957_object61698:head# 2024-04-23T16:33:56.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d706f7:::benchmark_data_smithi188_113957_object87161:head# 2024-04-23T16:33:56.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d711e9:::benchmark_data_smithi188_113957_object12898:head# 2024-04-23T16:33:56.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d862a7:::benchmark_data_smithi188_113957_object86666:head# 2024-04-23T16:33:56.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d9c9bc:::benchmark_data_smithi188_113957_object87926:head# 2024-04-23T16:33:56.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d9e32c:::benchmark_data_smithi188_113957_object141963:head# 2024-04-23T16:33:56.666 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.666 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59d9eff0:::benchmark_data_smithi188_113957_object128039:head# 2024-04-23T16:33:56.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59da0c7a:::benchmark_data_smithi188_113957_object68748:head# 2024-04-23T16:33:56.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59da2983:::benchmark_data_smithi188_113957_object131277:head# 2024-04-23T16:33:56.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59da535e:::benchmark_data_smithi188_113957_object37187:head# 2024-04-23T16:33:56.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59da8c55:::benchmark_data_smithi188_113957_object27835:head# 2024-04-23T16:33:56.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59dafaad:::benchmark_data_smithi188_113957_object21751:head# 2024-04-23T16:33:56.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59dc07d3:::benchmark_data_smithi188_113957_object100877:head# 2024-04-23T16:33:56.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.667 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59dc290d:::benchmark_data_smithi188_113957_object40973:head# 2024-04-23T16:33:56.667 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.668 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59dc78da:::benchmark_data_smithi188_113957_object107509:head# 2024-04-23T16:33:56.668 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.668 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59dd580f:::benchmark_data_smithi188_113957_object15687:head# 2024-04-23T16:33:56.668 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.668 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59dd5e7c:::benchmark_data_smithi188_113957_object52124:head# 2024-04-23T16:33:56.668 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.668 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59de28c8:::benchmark_data_smithi188_113957_object148039:head# 2024-04-23T16:33:56.668 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.668 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59deef39:::benchmark_data_smithi188_113957_object60296:head# 2024-04-23T16:33:56.668 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.668 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59dff661:::benchmark_data_smithi188_113957_object62829:head# 2024-04-23T16:33:56.668 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.668 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59e061f4:::benchmark_data_smithi188_113957_object113911:head# 2024-04-23T16:33:56.668 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59e0e2b0:::benchmark_data_smithi188_113957_object77395:head# 2024-04-23T16:33:56.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59e0e484:::benchmark_data_smithi188_113957_object76861:head# 2024-04-23T16:33:56.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59e20367:::benchmark_data_smithi188_113957_object143281:head# 2024-04-23T16:33:56.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59e22195:::benchmark_data_smithi188_113957_object20726:head# 2024-04-23T16:33:56.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59e23af4:::benchmark_data_smithi188_113957_object68336:head# 2024-04-23T16:33:56.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59e26b02:::benchmark_data_smithi188_113957_object74659:head# 2024-04-23T16:33:56.669 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.669 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59e26f6c:::benchmark_data_smithi188_113957_object20497:head# 2024-04-23T16:33:56.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59e38fd9:::benchmark_data_smithi188_113957_object29838:head# 2024-04-23T16:33:56.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59e4734c:::benchmark_data_smithi188_113957_object88053:head# 2024-04-23T16:33:56.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59e5c87a:::benchmark_data_smithi188_113957_object120936:head# 2024-04-23T16:33:56.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59e6999b:::benchmark_data_smithi188_113957_object57479:head# 2024-04-23T16:33:56.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59e6c2f0:::benchmark_data_smithi188_113957_object92903:head# 2024-04-23T16:33:56.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59e6f95e:::benchmark_data_smithi188_113957_object117202:head# 2024-04-23T16:33:56.670 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.670 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59e729d9:::benchmark_data_smithi188_113957_object147676:head# 2024-04-23T16:33:56.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59e7702c:::benchmark_data_smithi188_113957_object107342:head# 2024-04-23T16:33:56.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59e79f53:::benchmark_data_smithi188_113957_object20451:head# 2024-04-23T16:33:56.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59e8db00:::benchmark_data_smithi188_113957_object64517:head# 2024-04-23T16:33:56.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ea03b5:::benchmark_data_smithi188_113957_object91661:head# 2024-04-23T16:33:56.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ea91e5:::benchmark_data_smithi188_113957_object150307:head# 2024-04-23T16:33:56.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59eab948:::benchmark_data_smithi188_113957_object88871:head# 2024-04-23T16:33:56.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.671 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59eac620:::benchmark_data_smithi188_113957_object142531:head# 2024-04-23T16:33:56.671 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59eb1b1d:::benchmark_data_smithi188_113957_object13723:head# 2024-04-23T16:33:56.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59eb6fb9:::benchmark_data_smithi188_113957_object38262:head# 2024-04-23T16:33:56.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59eba14c:::benchmark_data_smithi188_113957_object54979:head# 2024-04-23T16:33:56.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ec211e:::benchmark_data_smithi188_113957_object318:head# 2024-04-23T16:33:56.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ed6579:::benchmark_data_smithi188_113957_object97744:head# 2024-04-23T16:33:56.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ee1a5b:::benchmark_data_smithi188_113957_object3054:head# 2024-04-23T16:33:56.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.672 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ee4347:::benchmark_data_smithi188_113957_object73657:head# 2024-04-23T16:33:56.672 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ee8768:::benchmark_data_smithi188_113957_object105618:head# 2024-04-23T16:33:56.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59eee949:::benchmark_data_smithi188_113957_object145512:head# 2024-04-23T16:33:56.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59f0027c:::benchmark_data_smithi188_113957_object70:head# 2024-04-23T16:33:56.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59f02947:::benchmark_data_smithi188_113957_object139389:head# 2024-04-23T16:33:56.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59f0e10b:::benchmark_data_smithi188_113957_object71084:head# 2024-04-23T16:33:56.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.673 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59f14cf2:::benchmark_data_smithi188_113957_object42908:head# 2024-04-23T16:33:56.673 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.674 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59f24b6f:::benchmark_data_smithi188_113957_object36681:head# 2024-04-23T16:33:56.674 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.674 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59f2798a:::benchmark_data_smithi188_113957_object59657:head# 2024-04-23T16:33:56.674 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.674 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59f2826d:::benchmark_data_smithi188_113957_object151709:head# 2024-04-23T16:33:56.674 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.674 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59f2f4f7:::benchmark_data_smithi188_113957_object123914:head# 2024-04-23T16:33:56.674 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.674 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59f510c4:::benchmark_data_smithi188_113957_object26027:head# 2024-04-23T16:33:56.674 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.674 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59f624b1:::benchmark_data_smithi188_113957_object43122:head# 2024-04-23T16:33:56.674 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.674 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59f6ed1a:::benchmark_data_smithi188_113957_object105226:head# 2024-04-23T16:33:56.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59f70ef2:::benchmark_data_smithi188_113957_object79181:head# 2024-04-23T16:33:56.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59f77778:::benchmark_data_smithi188_113957_object7152:head# 2024-04-23T16:33:56.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59f7fba8:::benchmark_data_smithi188_113957_object22370:head# 2024-04-23T16:33:56.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59f8b962:::benchmark_data_smithi188_113957_object12057:head# 2024-04-23T16:33:56.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59f9fe74:::benchmark_data_smithi188_113957_object120730:head# 2024-04-23T16:33:56.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59fa6044:::benchmark_data_smithi188_113957_object59727:head# 2024-04-23T16:33:56.675 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.675 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59fa63ec:::benchmark_data_smithi188_113957_object60877:head# 2024-04-23T16:33:56.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59fa7cd1:::benchmark_data_smithi188_113957_object129619:head# 2024-04-23T16:33:56.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59faac06:::benchmark_data_smithi188_113957_object98564:head# 2024-04-23T16:33:56.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59fab2ca:::benchmark_data_smithi188_113957_object127345:head# 2024-04-23T16:33:56.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59fab49e:::benchmark_data_smithi188_113957_object45720:head# 2024-04-23T16:33:56.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59fb0a64:::benchmark_data_smithi188_113957_object17250:head# 2024-04-23T16:33:56.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59fb1212:::benchmark_data_smithi188_113957_object146723:head# 2024-04-23T16:33:56.676 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.676 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59fb52d4:::benchmark_data_smithi188_113957_object150833:head# 2024-04-23T16:33:56.677 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.677 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59fbecd1:::benchmark_data_smithi188_113957_object27776:head# 2024-04-23T16:33:56.677 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.677 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59fc1c39:::benchmark_data_smithi188_113957_object27864:head# 2024-04-23T16:33:56.677 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.677 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59fc5306:::benchmark_data_smithi188_113957_object107631:head# 2024-04-23T16:33:56.677 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.677 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59fcd3f6:::benchmark_data_smithi188_113957_object132704:head# 2024-04-23T16:33:56.677 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.677 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59fce32c:::benchmark_data_smithi188_113957_object53243:head# 2024-04-23T16:33:56.677 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.677 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59fdd03d:::benchmark_data_smithi188_113957_object4006:head# 2024-04-23T16:33:56.677 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.677 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59fdd7d3:::benchmark_data_smithi188_113957_object21993:head# 2024-04-23T16:33:56.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59fde2bb:::benchmark_data_smithi188_113957_object111528:head# 2024-04-23T16:33:56.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59fe5147:::benchmark_data_smithi188_113957_object70396:head# 2024-04-23T16:33:56.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59feb2b3:::benchmark_data_smithi188_113957_object150952:head# 2024-04-23T16:33:56.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59fec33c:::benchmark_data_smithi188_113957_object85117:head# 2024-04-23T16:33:56.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ff47c2:::benchmark_data_smithi188_113957_object68165:head# 2024-04-23T16:33:56.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:59ffafd3:::benchmark_data_smithi188_113957_object143395:head# 2024-04-23T16:33:56.678 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.678 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a004446:::benchmark_data_smithi188_113957_object143177:head# 2024-04-23T16:33:56.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a014593:::benchmark_data_smithi188_113957_object49632:head# 2024-04-23T16:33:56.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a016f3f:::benchmark_data_smithi188_113957_object126528:head# 2024-04-23T16:33:56.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a01a6dd:::benchmark_data_smithi188_113957_object79864:head# 2024-04-23T16:33:56.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a03bacb:::benchmark_data_smithi188_113957_object131562:head# 2024-04-23T16:33:56.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a03d7be:::benchmark_data_smithi188_113957_object72113:head# 2024-04-23T16:33:56.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a048556:::benchmark_data_smithi188_113957_object78411:head# 2024-04-23T16:33:56.679 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.679 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a04bc6c:::benchmark_data_smithi188_113957_object101334:head# 2024-04-23T16:33:56.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a0508a0:::benchmark_data_smithi188_113957_object98658:head# 2024-04-23T16:33:56.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a050e30:::benchmark_data_smithi188_113957_object78341:head# 2024-04-23T16:33:56.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a05c68a:::benchmark_data_smithi188_113957_object141867:head# 2024-04-23T16:33:56.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a05eb8e:::benchmark_data_smithi188_113957_object82656:head# 2024-04-23T16:33:56.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a05f745:::benchmark_data_smithi188_113957_object13793:head# 2024-04-23T16:33:56.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a062207:::benchmark_data_smithi188_113957_object115298:head# 2024-04-23T16:33:56.680 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.680 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a068451:::benchmark_data_smithi188_113957_object96857:head# 2024-04-23T16:33:56.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a06aeb2:::benchmark_data_smithi188_113957_object27402:head# 2024-04-23T16:33:56.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a07668d:::benchmark_data_smithi188_113957_object13685:head# 2024-04-23T16:33:56.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a07d2f7:::benchmark_data_smithi188_113957_object54140:head# 2024-04-23T16:33:56.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a08feef:::benchmark_data_smithi188_113957_object31608:head# 2024-04-23T16:33:56.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a0a5363:::benchmark_data_smithi188_113957_object51142:head# 2024-04-23T16:33:56.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.681 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a0a7689:::benchmark_data_smithi188_113957_object16333:head# 2024-04-23T16:33:56.681 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a0ae4f0:::benchmark_data_smithi188_113957_object79186:head# 2024-04-23T16:33:56.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a0b6e00:::benchmark_data_smithi188_113957_object45705:head# 2024-04-23T16:33:56.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a0bf3a9:::benchmark_data_smithi188_113957_object9729:head# 2024-04-23T16:33:56.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a0c330a:::benchmark_data_smithi188_113957_object67436:head# 2024-04-23T16:33:56.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a0d3892:::benchmark_data_smithi188_113957_object31054:head# 2024-04-23T16:33:56.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a0f2e70:::benchmark_data_smithi188_113957_object108239:head# 2024-04-23T16:33:56.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.682 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a10d4a2:::benchmark_data_smithi188_113957_object64440:head# 2024-04-23T16:33:56.682 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.683 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a1101d4:::benchmark_data_smithi188_113957_object127734:head# 2024-04-23T16:33:56.683 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.683 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a113f2b:::benchmark_data_smithi188_113957_object32488:head# 2024-04-23T16:33:56.683 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.683 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a12a228:::benchmark_data_smithi188_113957_object151860:head# 2024-04-23T16:33:56.683 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.683 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a12d845:::benchmark_data_smithi188_113957_object80038:head# 2024-04-23T16:33:56.683 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.683 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a138fdd:::benchmark_data_smithi188_113957_object80332:head# 2024-04-23T16:33:56.683 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.683 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a140e6f:::benchmark_data_smithi188_113957_object135239:head# 2024-04-23T16:33:56.683 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.683 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a151e0e:::benchmark_data_smithi188_113957_object69815:head# 2024-04-23T16:33:56.683 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a15d48b:::benchmark_data_smithi188_113957_object68554:head# 2024-04-23T16:33:56.684 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a163ced:::benchmark_data_smithi188_113957_object72795:head# 2024-04-23T16:33:56.684 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a16582b:::benchmark_data_smithi188_113957_object29478:head# 2024-04-23T16:33:56.684 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a167576:::benchmark_data_smithi188_113957_object110087:head# 2024-04-23T16:33:56.684 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a16d7c6:::benchmark_data_smithi188_113957_object73412:head# 2024-04-23T16:33:56.684 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a16da1d:::benchmark_data_smithi188_113957_object17110:head# 2024-04-23T16:33:56.684 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.684 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a1995d8:::benchmark_data_smithi188_113957_object1745:head# 2024-04-23T16:33:56.684 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a199c91:::benchmark_data_smithi188_113957_object121894:head# 2024-04-23T16:33:56.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a1b10f2:::benchmark_data_smithi188_113957_object96394:head# 2024-04-23T16:33:56.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a1b2812:::benchmark_data_smithi188_113957_object17853:head# 2024-04-23T16:33:56.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a1b4bef:::benchmark_data_smithi188_113957_object115300:head# 2024-04-23T16:33:56.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a1b87b7:::benchmark_data_smithi188_113957_object114722:head# 2024-04-23T16:33:56.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a1bd363:::benchmark_data_smithi188_113957_object68472:head# 2024-04-23T16:33:56.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.685 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a1bf81f:::benchmark_data_smithi188_113957_object29541:head# 2024-04-23T16:33:56.685 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.686 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a1d0e9b:::benchmark_data_smithi188_113957_object92073:head# 2024-04-23T16:33:56.686 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.686 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a1dc978:::benchmark_data_smithi188_113957_object89029:head# 2024-04-23T16:33:56.686 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.686 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a1e1ae1:::benchmark_data_smithi188_113957_object144808:head# 2024-04-23T16:33:56.686 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.686 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a1e4e06:::benchmark_data_smithi188_113957_object45645:head# 2024-04-23T16:33:56.686 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.686 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a1ed67e:::benchmark_data_smithi188_113957_object21843:head# 2024-04-23T16:33:56.686 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.686 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a1f0a3d:::benchmark_data_smithi188_113957_object97686:head# 2024-04-23T16:33:56.686 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.686 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a1f73b7:::benchmark_data_smithi188_113957_object47405:head# 2024-04-23T16:33:56.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a227f7f:::benchmark_data_smithi188_113957_object73615:head# 2024-04-23T16:33:56.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a23d6e5:::benchmark_data_smithi188_113957_object132267:head# 2024-04-23T16:33:56.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.687 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a23efc1:::benchmark_data_smithi188_113957_object2922:head# 2024-04-23T16:33:56.687 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a24095b:::benchmark_data_smithi188_113957_object122604:head# 2024-04-23T16:33:56.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a2470b2:::benchmark_data_smithi188_113957_object123289:head# 2024-04-23T16:33:56.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a249db2:::benchmark_data_smithi188_113957_object60323:head# 2024-04-23T16:33:56.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a24f801:::benchmark_data_smithi188_113957_object123774:head# 2024-04-23T16:33:56.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a25020e:::benchmark_data_smithi188_113957_object55938:head# 2024-04-23T16:33:56.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a25c6cc:::benchmark_data_smithi188_113957_object60454:head# 2024-04-23T16:33:56.688 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.688 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a2627cd:::benchmark_data_smithi188_113957_object128679:head# 2024-04-23T16:33:56.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a262e88:::benchmark_data_smithi188_113957_object15834:head# 2024-04-23T16:33:56.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a263205:::benchmark_data_smithi188_113957_object39779:head# 2024-04-23T16:33:56.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a26f567:::benchmark_data_smithi188_113957_object21949:head# 2024-04-23T16:33:56.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a26fcb9:::benchmark_data_smithi188_113957_object82:head# 2024-04-23T16:33:56.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a2708a9:::benchmark_data_smithi188_113957_object11718:head# 2024-04-23T16:33:56.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a273004:::benchmark_data_smithi188_113957_object11198:head# 2024-04-23T16:33:56.689 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.689 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a27ea40:::benchmark_data_smithi188_113957_object112660:head# 2024-04-23T16:33:56.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a287e87:::benchmark_data_smithi188_113957_object134464:head# 2024-04-23T16:33:56.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a290211:::benchmark_data_smithi188_113957_object138730:head# 2024-04-23T16:33:56.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a290ea1:::benchmark_data_smithi188_113957_object41783:head# 2024-04-23T16:33:56.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a2965f6:::benchmark_data_smithi188_113957_object141432:head# 2024-04-23T16:33:56.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a2a448f:::benchmark_data_smithi188_113957_object23768:head# 2024-04-23T16:33:56.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a2b7d51:::benchmark_data_smithi188_113957_object125255:head# 2024-04-23T16:33:56.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a2be851:::benchmark_data_smithi188_113957_object101012:head# 2024-04-23T16:33:56.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a2cb6bf:::benchmark_data_smithi188_113957_object142269:head# 2024-04-23T16:33:56.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a2cd588:::benchmark_data_smithi188_113957_object32390:head# 2024-04-23T16:33:56.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a2d129d:::benchmark_data_smithi188_113957_object114802:head# 2024-04-23T16:33:56.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a2d1d27:::benchmark_data_smithi188_113957_object116775:head# 2024-04-23T16:33:56.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a2d9734:::benchmark_data_smithi188_113957_object146336:head# 2024-04-23T16:33:56.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a2de7a2:::benchmark_data_smithi188_113957_object29247:head# 2024-04-23T16:33:56.691 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a2e09ab:::benchmark_data_smithi188_113957_object65800:head# 2024-04-23T16:33:56.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a2e7f2c:::benchmark_data_smithi188_113957_object117066:head# 2024-04-23T16:33:56.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a2e936b:::benchmark_data_smithi188_113957_object150797:head# 2024-04-23T16:33:56.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a2f839b:::benchmark_data_smithi188_113957_object27426:head# 2024-04-23T16:33:56.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a2fec0c:::benchmark_data_smithi188_113957_object112503:head# 2024-04-23T16:33:56.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a2ff37a:::benchmark_data_smithi188_113957_object27979:head# 2024-04-23T16:33:56.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a305ecc:::benchmark_data_smithi188_113957_object5290:head# 2024-04-23T16:33:56.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a32812d:::benchmark_data_smithi188_113957_object9116:head# 2024-04-23T16:33:56.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a33b8c8:::benchmark_data_smithi188_113957_object13023:head# 2024-04-23T16:33:56.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3402ca:::benchmark_data_smithi188_113957_object71637:head# 2024-04-23T16:33:56.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a343de1:::benchmark_data_smithi188_113957_object131757:head# 2024-04-23T16:33:56.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a34c0e7:::benchmark_data_smithi188_113957_object67999:head# 2024-04-23T16:33:56.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a362c0b:::benchmark_data_smithi188_113957_object15404:head# 2024-04-23T16:33:56.693 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.693 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a362fb6:::benchmark_data_smithi188_113957_object58234:head# 2024-04-23T16:33:56.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.694 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3676d8:::benchmark_data_smithi188_113957_object134311:head# 2024-04-23T16:33:56.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.694 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a372dee:::benchmark_data_smithi188_113957_object140602:head# 2024-04-23T16:33:56.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.694 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a376715:::benchmark_data_smithi188_113957_object63137:head# 2024-04-23T16:33:56.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.694 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a378289:::benchmark_data_smithi188_113957_object146740:head# 2024-04-23T16:33:56.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.694 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a37aab3:::benchmark_data_smithi188_113957_object63643:head# 2024-04-23T16:33:56.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.694 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a380acd:::benchmark_data_smithi188_113957_object135353:head# 2024-04-23T16:33:56.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.694 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a396313:::benchmark_data_smithi188_113957_object96249:head# 2024-04-23T16:33:56.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a39eafd:::benchmark_data_smithi188_113957_object40843:head# 2024-04-23T16:33:56.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3a2ca7:::benchmark_data_smithi188_113957_object147123:head# 2024-04-23T16:33:56.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3ac5a9:::benchmark_data_smithi188_113957_object22278:head# 2024-04-23T16:33:56.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3b0e04:::benchmark_data_smithi188_113957_object3133:head# 2024-04-23T16:33:56.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3b0ff5:::benchmark_data_smithi188_113957_object101812:head# 2024-04-23T16:33:56.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3b457c:::benchmark_data_smithi188_113957_object120872:head# 2024-04-23T16:33:56.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3b4a02:::benchmark_data_smithi188_113957_object29338:head# 2024-04-23T16:33:56.696 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.696 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3ba333:::benchmark_data_smithi188_113957_object111899:head# 2024-04-23T16:33:56.696 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.696 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3bde8c:::benchmark_data_smithi188_113957_object75801:head# 2024-04-23T16:33:56.696 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.696 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3bf4a8:::benchmark_data_smithi188_113957_object61385:head# 2024-04-23T16:33:56.696 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.696 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3c0ab8:::benchmark_data_smithi188_113957_object26064:head# 2024-04-23T16:33:56.696 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.696 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3c49ab:::benchmark_data_smithi188_113957_object23724:head# 2024-04-23T16:33:56.696 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.696 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3cb919:::benchmark_data_smithi188_113957_object3483:head# 2024-04-23T16:33:56.696 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3d3047:::benchmark_data_smithi188_113957_object18525:head# 2024-04-23T16:33:56.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3d7725:::benchmark_data_smithi188_113957_object106047:head# 2024-04-23T16:33:56.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3dca2b:::benchmark_data_smithi188_113957_object129321:head# 2024-04-23T16:33:56.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3dee65:::benchmark_data_smithi188_113957_object29135:head# 2024-04-23T16:33:56.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3ee66d:::benchmark_data_smithi188_113957_object8714:head# 2024-04-23T16:33:56.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3f2719:::benchmark_data_smithi188_113957_object124131:head# 2024-04-23T16:33:56.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3f8f01:::benchmark_data_smithi188_113957_object61390:head# 2024-04-23T16:33:56.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a3ff3f8:::benchmark_data_smithi188_113957_object95974:head# 2024-04-23T16:33:56.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a4068b1:::benchmark_data_smithi188_113957_object18715:head# 2024-04-23T16:33:56.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a417cde:::benchmark_data_smithi188_113957_object93735:head# 2024-04-23T16:33:56.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a41d9bd:::benchmark_data_smithi188_113957_object35160:head# 2024-04-23T16:33:56.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a41ff6d:::benchmark_data_smithi188_113957_object107412:head# 2024-04-23T16:33:56.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a42c3b7:::benchmark_data_smithi188_113957_object127861:head# 2024-04-23T16:33:56.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.699 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a43a392:::benchmark_data_smithi188_113957_object138350:head# 2024-04-23T16:33:56.699 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.699 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a43c58e:::benchmark_data_smithi188_113957_object125795:head# 2024-04-23T16:33:56.699 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.699 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a43fd76:::benchmark_data_smithi188_113957_object146117:head# 2024-04-23T16:33:56.699 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.699 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a4415a4:::benchmark_data_smithi188_113957_object102780:head# 2024-04-23T16:33:56.699 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.699 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a4418fc:::benchmark_data_smithi188_113957_object67697:head# 2024-04-23T16:33:56.699 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.699 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a445a66:::benchmark_data_smithi188_113957_object57012:head# 2024-04-23T16:33:56.699 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.699 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a4468b8:::benchmark_data_smithi188_113957_object134327:head# 2024-04-23T16:33:56.699 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a44720c:::benchmark_data_smithi188_113957_object42577:head# 2024-04-23T16:33:56.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a44d07b:::benchmark_data_smithi188_113957_object29985:head# 2024-04-23T16:33:56.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a450dee:::benchmark_data_smithi188_113957_object70513:head# 2024-04-23T16:33:56.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a453ebf:::benchmark_data_smithi188_113957_object11799:head# 2024-04-23T16:33:56.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a45d196:::benchmark_data_smithi188_113957_object102486:head# 2024-04-23T16:33:56.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a45e223:::benchmark_data_smithi188_113957_object103229:head# 2024-04-23T16:33:56.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.700 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a461e36:::benchmark_data_smithi188_113957_object74778:head# 2024-04-23T16:33:56.700 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a480f3f:::benchmark_data_smithi188_113957_object106390:head# 2024-04-23T16:33:56.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a49c8d8:::benchmark_data_smithi188_113957_object30711:head# 2024-04-23T16:33:56.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a4a9d3a:::benchmark_data_smithi188_113957_object24290:head# 2024-04-23T16:33:56.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a4acd50:::benchmark_data_smithi188_113957_object84936:head# 2024-04-23T16:33:56.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a4b5c04:::benchmark_data_smithi188_113957_object4229:head# 2024-04-23T16:33:56.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a4b5d39:::benchmark_data_smithi188_113957_object42369:head# 2024-04-23T16:33:56.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a4b867b:::benchmark_data_smithi188_113957_object144009:head# 2024-04-23T16:33:56.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a4bf5d2:::benchmark_data_smithi188_113957_object81473:head# 2024-04-23T16:33:56.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a4c5a4e:::benchmark_data_smithi188_113957_object46513:head# 2024-04-23T16:33:56.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a4c65c3:::benchmark_data_smithi188_113957_object74388:head# 2024-04-23T16:33:56.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a4cd0bd:::benchmark_data_smithi188_113957_object124175:head# 2024-04-23T16:33:56.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a4dc27b:::benchmark_data_smithi188_113957_object60089:head# 2024-04-23T16:33:56.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a4fc54f:::benchmark_data_smithi188_113957_object15164:head# 2024-04-23T16:33:56.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a4ff77f:::benchmark_data_smithi188_113957_object1956:head# 2024-04-23T16:33:56.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a504128:::benchmark_data_smithi188_113957_object98177:head# 2024-04-23T16:33:56.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a505b74:::benchmark_data_smithi188_113957_object147498:head# 2024-04-23T16:33:56.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a50de81:::benchmark_data_smithi188_113957_object66852:head# 2024-04-23T16:33:56.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a512711:::benchmark_data_smithi188_113957_object108241:head# 2024-04-23T16:33:56.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a5181cb:::benchmark_data_smithi188_113957_object120062:head# 2024-04-23T16:33:56.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a519a10:::benchmark_data_smithi188_113957_object77760:head# 2024-04-23T16:33:56.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a51cf52:::benchmark_data_smithi188_113957_object109035:head# 2024-04-23T16:33:56.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a5223f3:::benchmark_data_smithi188_113957_object98410:head# 2024-04-23T16:33:56.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a525840:::benchmark_data_smithi188_113957_object110076:head# 2024-04-23T16:33:56.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a533677:::benchmark_data_smithi188_113957_object14364:head# 2024-04-23T16:33:56.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a537105:::benchmark_data_smithi188_113957_object17562:head# 2024-04-23T16:33:56.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a54ce96:::benchmark_data_smithi188_113957_object70813:head# 2024-04-23T16:33:56.704 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.704 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a55346d:::benchmark_data_smithi188_113957_object92362:head# 2024-04-23T16:33:56.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a5669e3:::benchmark_data_smithi188_113957_object88862:head# 2024-04-23T16:33:56.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a56ce19:::benchmark_data_smithi188_113957_object77857:head# 2024-04-23T16:33:56.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a56e294:::benchmark_data_smithi188_113957_object116927:head# 2024-04-23T16:33:56.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a58342d:::benchmark_data_smithi188_113957_object152951:head# 2024-04-23T16:33:56.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a585220:::benchmark_data_smithi188_113957_object145857:head# 2024-04-23T16:33:56.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.705 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a586389:::benchmark_data_smithi188_113957_object101069:head# 2024-04-23T16:33:56.705 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a59b35d:::benchmark_data_smithi188_113957_object152884:head# 2024-04-23T16:33:56.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a59f6c7:::benchmark_data_smithi188_113957_object99772:head# 2024-04-23T16:33:56.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a5a3bde:::benchmark_data_smithi188_113957_object128439:head# 2024-04-23T16:33:56.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a5a5c65:::benchmark_data_smithi188_113957_object136437:head# 2024-04-23T16:33:56.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a5c0bdb:::benchmark_data_smithi188_113957_object66953:head# 2024-04-23T16:33:56.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a5c7807:::benchmark_data_smithi188_113957_object78889:head# 2024-04-23T16:33:56.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a5c8106:::benchmark_data_smithi188_113957_object60118:head# 2024-04-23T16:33:56.707 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.707 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a5e6f6a:::benchmark_data_smithi188_113957_object26393:head# 2024-04-23T16:33:56.707 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.707 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a5ee94f:::benchmark_data_smithi188_113957_object132789:head# 2024-04-23T16:33:56.707 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.707 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a5f2ec1:::benchmark_data_smithi188_113957_object134726:head# 2024-04-23T16:33:56.707 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.707 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a606325:::benchmark_data_smithi188_113957_object104074:head# 2024-04-23T16:33:56.707 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.707 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a607f0d:::benchmark_data_smithi188_113957_object73673:head# 2024-04-23T16:33:56.707 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.707 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a60bc66:::benchmark_data_smithi188_113957_object38494:head# 2024-04-23T16:33:56.707 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.708 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a60ce51:::benchmark_data_smithi188_113957_object13954:head# 2024-04-23T16:33:56.708 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.708 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a61046e:::benchmark_data_smithi188_113957_object4480:head# 2024-04-23T16:33:56.708 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.708 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a613612:::benchmark_data_smithi188_113957_object146762:head# 2024-04-23T16:33:56.708 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.708 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a61a870:::benchmark_data_smithi188_113957_object44142:head# 2024-04-23T16:33:56.708 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.708 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a6230e0:::benchmark_data_smithi188_113957_object143837:head# 2024-04-23T16:33:56.708 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.708 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a629342:::benchmark_data_smithi188_113957_object152409:head# 2024-04-23T16:33:56.708 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.708 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a629e6e:::benchmark_data_smithi188_113957_object37916:head# 2024-04-23T16:33:56.708 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.708 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a635bf5:::benchmark_data_smithi188_113957_object106717:head# 2024-04-23T16:33:56.709 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.709 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a636a48:::benchmark_data_smithi188_113957_object21510:head# 2024-04-23T16:33:56.709 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.709 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a637c82:::benchmark_data_smithi188_113957_object29184:head# 2024-04-23T16:33:56.709 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.709 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a64b1a8:::benchmark_data_smithi188_113957_object13857:head# 2024-04-23T16:33:56.709 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.709 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a65b73b:::benchmark_data_smithi188_113957_object33783:head# 2024-04-23T16:33:56.709 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.709 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a67074c:::benchmark_data_smithi188_113957_object35374:head# 2024-04-23T16:33:56.709 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.709 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a6723ef:::benchmark_data_smithi188_113957_object45718:head# 2024-04-23T16:33:56.709 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.710 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a68361f:::benchmark_data_smithi188_113957_object93851:head# 2024-04-23T16:33:56.710 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.710 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a689e11:::benchmark_data_smithi188_113957_object46998:head# 2024-04-23T16:33:56.710 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.710 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a68f433:::benchmark_data_smithi188_113957_object85819:head# 2024-04-23T16:33:56.710 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.710 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a6901b6:::benchmark_data_smithi188_113957_object95855:head# 2024-04-23T16:33:56.710 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.710 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a6919ba:::benchmark_data_smithi188_113957_object148010:head# 2024-04-23T16:33:56.710 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.710 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a692b77:::benchmark_data_smithi188_113957_object110655:head# 2024-04-23T16:33:56.710 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.710 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a69a57e:::benchmark_data_smithi188_113957_object13315:head# 2024-04-23T16:33:56.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a69bbe4:::benchmark_data_smithi188_113957_object88912:head# 2024-04-23T16:33:56.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a6a0f44:::benchmark_data_smithi188_113957_object147059:head# 2024-04-23T16:33:56.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a6a40f4:::benchmark_data_smithi188_113957_object48871:head# 2024-04-23T16:33:56.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a6be88f:::benchmark_data_smithi188_113957_object93658:head# 2024-04-23T16:33:56.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a6c7329:::benchmark_data_smithi188_113957_object115343:head# 2024-04-23T16:33:56.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a6cab64:::benchmark_data_smithi188_113957_object29293:head# 2024-04-23T16:33:56.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a6cd580:::benchmark_data_smithi188_113957_object112768:head# 2024-04-23T16:33:56.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a6ce5b5:::benchmark_data_smithi188_113957_object75802:head# 2024-04-23T16:33:56.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a6cf82e:::benchmark_data_smithi188_113957_object28932:head# 2024-04-23T16:33:56.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a6d2ffa:::benchmark_data_smithi188_113957_object96897:head# 2024-04-23T16:33:56.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a6d924e:::benchmark_data_smithi188_113957_object19276:head# 2024-04-23T16:33:56.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a6e5257:::benchmark_data_smithi188_113957_object122058:head# 2024-04-23T16:33:56.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a6eb3f0:::benchmark_data_smithi188_113957_object122382:head# 2024-04-23T16:33:56.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a6efb32:::benchmark_data_smithi188_113957_object124927:head# 2024-04-23T16:33:56.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a6fca64:::benchmark_data_smithi188_113957_object40597:head# 2024-04-23T16:33:56.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a6fe604:::benchmark_data_smithi188_113957_object49858:head# 2024-04-23T16:33:56.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a70040f:::benchmark_data_smithi188_113957_object84766:head# 2024-04-23T16:33:56.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a702505:::benchmark_data_smithi188_113957_object44227:head# 2024-04-23T16:33:56.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a7085b6:::benchmark_data_smithi188_113957_object67684:head# 2024-04-23T16:33:56.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a70e209:::benchmark_data_smithi188_113957_object123113:head# 2024-04-23T16:33:56.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a71999e:::benchmark_data_smithi188_113957_object51997:head# 2024-04-23T16:33:56.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a71b632:::benchmark_data_smithi188_113957_object36436:head# 2024-04-23T16:33:56.714 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.714 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a71dae7:::benchmark_data_smithi188_113957_object143677:head# 2024-04-23T16:33:56.714 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.714 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a7297e9:::benchmark_data_smithi188_113957_object140618:head# 2024-04-23T16:33:56.714 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.714 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a729d97:::benchmark_data_smithi188_113957_object136406:head# 2024-04-23T16:33:56.714 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.714 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a72ad89:::benchmark_data_smithi188_113957_object87479:head# 2024-04-23T16:33:56.714 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.714 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a732169:::benchmark_data_smithi188_113957_object120633:head# 2024-04-23T16:33:56.714 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.714 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a73590b:::benchmark_data_smithi188_113957_object116845:head# 2024-04-23T16:33:56.715 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a737946:::benchmark_data_smithi188_113957_object27865:head# 2024-04-23T16:33:56.715 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a73fa18:::benchmark_data_smithi188_113957_object111329:head# 2024-04-23T16:33:56.715 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a744200:::benchmark_data_smithi188_113957_object37822:head# 2024-04-23T16:33:56.715 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a748f01:::benchmark_data_smithi188_113957_object110561:head# 2024-04-23T16:33:56.715 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a74e43f:::benchmark_data_smithi188_113957_object94600:head# 2024-04-23T16:33:56.715 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a755e67:::benchmark_data_smithi188_113957_object37331:head# 2024-04-23T16:33:56.715 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.715 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a764acd:::benchmark_data_smithi188_113957_object36395:head# 2024-04-23T16:33:56.716 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.716 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a768a6e:::benchmark_data_smithi188_113957_object48931:head# 2024-04-23T16:33:56.716 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.716 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a76bb7a:::benchmark_data_smithi188_113957_object123860:head# 2024-04-23T16:33:56.716 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.716 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a777181:::benchmark_data_smithi188_113957_object100449:head# 2024-04-23T16:33:56.716 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.716 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a7837b3:::benchmark_data_smithi188_113957_object37232:head# 2024-04-23T16:33:56.716 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.716 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a7895fb:::benchmark_data_smithi188_113957_object100483:head# 2024-04-23T16:33:56.716 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.716 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a78ee25:::benchmark_data_smithi188_113957_object116265:head# 2024-04-23T16:33:56.716 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.716 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a7906b6:::benchmark_data_smithi188_113957_object71384:head# 2024-04-23T16:33:56.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a797e78:::benchmark_data_smithi188_113957_object18394:head# 2024-04-23T16:33:56.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a79a2a5:::benchmark_data_smithi188_113957_object140721:head# 2024-04-23T16:33:56.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a79db3d:::benchmark_data_smithi188_113957_object130158:head# 2024-04-23T16:33:56.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a7b01fc:::benchmark_data_smithi188_113957_object2110:head# 2024-04-23T16:33:56.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a7b6ceb:::benchmark_data_smithi188_113957_object32258:head# 2024-04-23T16:33:56.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a7c29f4:::benchmark_data_smithi188_113957_object5358:head# 2024-04-23T16:33:56.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a7d2580:::benchmark_data_smithi188_113957_object28502:head# 2024-04-23T16:33:56.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a7d6815:::benchmark_data_smithi188_113957_object30525:head# 2024-04-23T16:33:56.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a7ea47a:::benchmark_data_smithi188_113957_object106872:head# 2024-04-23T16:33:56.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a7ed06a:::benchmark_data_smithi188_113957_object84781:head# 2024-04-23T16:33:56.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a7ed512:::benchmark_data_smithi188_113957_object96279:head# 2024-04-23T16:33:56.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a7f0c04:::benchmark_data_smithi188_113957_object98297:head# 2024-04-23T16:33:56.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a7f1ec7:::benchmark_data_smithi188_113957_object101108:head# 2024-04-23T16:33:56.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a7fa091:::benchmark_data_smithi188_113957_object24713:head# 2024-04-23T16:33:56.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a80db3b:::benchmark_data_smithi188_113957_object137653:head# 2024-04-23T16:33:56.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a81d54e:::benchmark_data_smithi188_113957_object77961:head# 2024-04-23T16:33:56.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a81d95c:::benchmark_data_smithi188_113957_object18557:head# 2024-04-23T16:33:56.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a828708:::benchmark_data_smithi188_113957_object27817:head# 2024-04-23T16:33:56.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a829184:::benchmark_data_smithi188_113957_object114573:head# 2024-04-23T16:33:56.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a8317d0:::benchmark_data_smithi188_113957_object44356:head# 2024-04-23T16:33:56.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a83689c:::benchmark_data_smithi188_113957_object19282:head# 2024-04-23T16:33:56.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a83a7f6:::benchmark_data_smithi188_113957_object23730:head# 2024-04-23T16:33:56.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a8471d1:::benchmark_data_smithi188_113957_object140824:head# 2024-04-23T16:33:56.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a848717:::benchmark_data_smithi188_113957_object5439:head# 2024-04-23T16:33:56.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a850f96:::benchmark_data_smithi188_113957_object131687:head# 2024-04-23T16:33:56.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a853732:::benchmark_data_smithi188_113957_object42124:head# 2024-04-23T16:33:56.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a8676ac:::benchmark_data_smithi188_113957_object9583:head# 2024-04-23T16:33:56.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a868fd3:::benchmark_data_smithi188_113957_object88077:head# 2024-04-23T16:33:56.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a86c119:::benchmark_data_smithi188_113957_object119796:head# 2024-04-23T16:33:56.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a89222d:::benchmark_data_smithi188_113957_object120546:head# 2024-04-23T16:33:56.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a89bed0:::benchmark_data_smithi188_113957_object90184:head# 2024-04-23T16:33:56.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a8a6f16:::benchmark_data_smithi188_113957_object132718:head# 2024-04-23T16:33:56.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a8ab3fb:::benchmark_data_smithi188_113957_object8786:head# 2024-04-23T16:33:56.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.721 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a8be68d:::benchmark_data_smithi188_113957_object122836:head# 2024-04-23T16:33:56.721 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a8c055b:::benchmark_data_smithi188_113957_object112137:head# 2024-04-23T16:33:56.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a8ef04e:::benchmark_data_smithi188_113957_object147708:head# 2024-04-23T16:33:56.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a8f8604:::benchmark_data_smithi188_113957_object27648:head# 2024-04-23T16:33:56.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a8fec63:::benchmark_data_smithi188_113957_object19969:head# 2024-04-23T16:33:56.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a8ff456:::benchmark_data_smithi188_113957_object2660:head# 2024-04-23T16:33:56.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a90c08d:::benchmark_data_smithi188_113957_object131933:head# 2024-04-23T16:33:56.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a91306e:::benchmark_data_smithi188_113957_object143489:head# 2024-04-23T16:33:56.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a91db8c:::benchmark_data_smithi188_113957_object138696:head# 2024-04-23T16:33:56.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a91e6b2:::benchmark_data_smithi188_113957_object11741:head# 2024-04-23T16:33:56.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a92aaf8:::benchmark_data_smithi188_113957_object27556:head# 2024-04-23T16:33:56.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a93539c:::benchmark_data_smithi188_113957_object71067:head# 2024-04-23T16:33:56.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a936872:::benchmark_data_smithi188_113957_object107364:head# 2024-04-23T16:33:56.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a93cb06:::benchmark_data_smithi188_113957_object151148:head# 2024-04-23T16:33:56.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a93f826:::benchmark_data_smithi188_113957_object82932:head# 2024-04-23T16:33:56.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9428ec:::benchmark_data_smithi188_113957_object36519:head# 2024-04-23T16:33:56.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9483f1:::benchmark_data_smithi188_113957_object87570:head# 2024-04-23T16:33:56.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a95176c:::benchmark_data_smithi188_113957_object128974:head# 2024-04-23T16:33:56.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a95b7db:::benchmark_data_smithi188_113957_object114658:head# 2024-04-23T16:33:56.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a95bf53:::benchmark_data_smithi188_113957_object103286:head# 2024-04-23T16:33:56.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a95c900:::benchmark_data_smithi188_113957_object132194:head# 2024-04-23T16:33:56.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9602b5:::benchmark_data_smithi188_113957_object92775:head# 2024-04-23T16:33:56.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a969894:::benchmark_data_smithi188_113957_object149545:head# 2024-04-23T16:33:56.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a97054d:::benchmark_data_smithi188_113957_object94221:head# 2024-04-23T16:33:56.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9872bd:::benchmark_data_smithi188_113957_object77294:head# 2024-04-23T16:33:56.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9909de:::benchmark_data_smithi188_113957_object90616:head# 2024-04-23T16:33:56.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9a186a:::benchmark_data_smithi188_113957_object87782:head# 2024-04-23T16:33:56.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9a44bd:::benchmark_data_smithi188_113957_object7791:head# 2024-04-23T16:33:56.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9ad0f9:::benchmark_data_smithi188_113957_object102756:head# 2024-04-23T16:33:56.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9b4c0e:::benchmark_data_smithi188_113957_object39277:head# 2024-04-23T16:33:56.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9b86b6:::benchmark_data_smithi188_113957_object41766:head# 2024-04-23T16:33:56.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9bf7d7:::benchmark_data_smithi188_113957_object118772:head# 2024-04-23T16:33:56.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9bfbff:::benchmark_data_smithi188_113957_object33978:head# 2024-04-23T16:33:56.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9c85e5:::benchmark_data_smithi188_113957_object20640:head# 2024-04-23T16:33:56.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9cb756:::benchmark_data_smithi188_113957_object32432:head# 2024-04-23T16:33:56.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9cb863:::benchmark_data_smithi188_113957_object69131:head# 2024-04-23T16:33:56.727 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9dba56:::benchmark_data_smithi188_113957_object82290:head# 2024-04-23T16:33:56.727 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9ddbc7:::benchmark_data_smithi188_113957_object69442:head# 2024-04-23T16:33:56.727 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9ddbf6:::benchmark_data_smithi188_113957_object91705:head# 2024-04-23T16:33:56.727 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9e851b:::benchmark_data_smithi188_113957_object24647:head# 2024-04-23T16:33:56.727 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9f0112:::benchmark_data_smithi188_113957_object20837:head# 2024-04-23T16:33:56.727 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.727 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9f0f3c:::benchmark_data_smithi188_113957_object15066:head# 2024-04-23T16:33:56.728 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.728 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9fcaf1:::benchmark_data_smithi188_113957_object134377:head# 2024-04-23T16:33:56.728 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.728 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9fcb0b:::benchmark_data_smithi188_113957_object111463:head# 2024-04-23T16:33:56.728 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.728 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5a9fd0bc:::benchmark_data_smithi188_113957_object56675:head# 2024-04-23T16:33:56.728 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.728 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa02716:::benchmark_data_smithi188_113957_object4588:head# 2024-04-23T16:33:56.728 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.728 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa06674:::benchmark_data_smithi188_113957_object61400:head# 2024-04-23T16:33:56.728 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.728 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa075fe:::benchmark_data_smithi188_113957_object93615:head# 2024-04-23T16:33:56.728 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.728 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa09528:::benchmark_data_smithi188_113957_object148640:head# 2024-04-23T16:33:56.729 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.729 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa0d09e:::benchmark_data_smithi188_113957_object80398:head# 2024-04-23T16:33:56.729 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.729 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa1b42b:::benchmark_data_smithi188_113957_object144330:head# 2024-04-23T16:33:56.729 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.729 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa20aaa:::benchmark_data_smithi188_113957_object50480:head# 2024-04-23T16:33:56.729 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.729 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa2e508:::benchmark_data_smithi188_113957_object136690:head# 2024-04-23T16:33:56.729 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.729 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa37ccd:::benchmark_data_smithi188_113957_object133613:head# 2024-04-23T16:33:56.729 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.729 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa39845:::benchmark_data_smithi188_113957_object105668:head# 2024-04-23T16:33:56.729 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.729 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa39dde:::benchmark_data_smithi188_113957_object94168:head# 2024-04-23T16:33:56.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa429cb:::benchmark_data_smithi188_113957_object80363:head# 2024-04-23T16:33:56.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa4a86c:::benchmark_data_smithi188_113957_object67889:head# 2024-04-23T16:33:56.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa4c53e:::benchmark_data_smithi188_113957_object117596:head# 2024-04-23T16:33:56.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa4dfdb:::benchmark_data_smithi188_113957_object151176:head# 2024-04-23T16:33:56.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa4eed0:::benchmark_data_smithi188_113957_object115641:head# 2024-04-23T16:33:56.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa5a109:::benchmark_data_smithi188_113957_object9937:head# 2024-04-23T16:33:56.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa652d5:::benchmark_data_smithi188_113957_object114628:head# 2024-04-23T16:33:56.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa7d0a5:::benchmark_data_smithi188_113957_object17643:head# 2024-04-23T16:33:56.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa99399:::benchmark_data_smithi188_113957_object46067:head# 2024-04-23T16:33:56.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa9a800:::benchmark_data_smithi188_113957_object58911:head# 2024-04-23T16:33:56.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa9c3cb:::benchmark_data_smithi188_113957_object130691:head# 2024-04-23T16:33:56.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aa9d6a7:::benchmark_data_smithi188_113957_object26089:head# 2024-04-23T16:33:56.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aaae189:::benchmark_data_smithi188_113957_object14601:head# 2024-04-23T16:33:56.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aab335d:::benchmark_data_smithi188_113957_object39625:head# 2024-04-23T16:33:56.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aab8b49:::benchmark_data_smithi188_113957_object65394:head# 2024-04-23T16:33:56.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aabf71a:::benchmark_data_smithi188_113957_object150746:head# 2024-04-23T16:33:56.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aacdc2a:::benchmark_data_smithi188_113957_object20332:head# 2024-04-23T16:33:56.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aacef54:::benchmark_data_smithi188_113957_object61451:head# 2024-04-23T16:33:56.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aad9a06:::benchmark_data_smithi188_113957_object23373:head# 2024-04-23T16:33:56.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aae3487:::benchmark_data_smithi188_113957_object139004:head# 2024-04-23T16:33:56.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aae798a:::benchmark_data_smithi188_113957_object111268:head# 2024-04-23T16:33:56.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aafae6f:::benchmark_data_smithi188_113957_object42705:head# 2024-04-23T16:33:56.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aafec99:::benchmark_data_smithi188_113957_object137517:head# 2024-04-23T16:33:56.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab0ad34:::benchmark_data_smithi188_113957_object104249:head# 2024-04-23T16:33:56.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab0aec4:::benchmark_data_smithi188_113957_object14807:head# 2024-04-23T16:33:56.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab0dbb9:::benchmark_data_smithi188_113957_object7948:head# 2024-04-23T16:33:56.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab140c8:::benchmark_data_smithi188_113957_object12667:head# 2024-04-23T16:33:56.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab16195:::benchmark_data_smithi188_113957_object12532:head# 2024-04-23T16:33:56.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab2a77b:::benchmark_data_smithi188_113957_object58036:head# 2024-04-23T16:33:56.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab2b66b:::benchmark_data_smithi188_113957_object39471:head# 2024-04-23T16:33:56.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab3a83a:::benchmark_data_smithi188_113957_object142472:head# 2024-04-23T16:33:56.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab44876:::benchmark_data_smithi188_113957_object127932:head# 2024-04-23T16:33:56.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab4ee10:::benchmark_data_smithi188_113957_object21223:head# 2024-04-23T16:33:56.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab516ac:::benchmark_data_smithi188_113957_object46245:head# 2024-04-23T16:33:56.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab6c3a7:::benchmark_data_smithi188_113957_object96058:head# 2024-04-23T16:33:56.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab74206:::benchmark_data_smithi188_113957_object98926:head# 2024-04-23T16:33:56.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab7ba93:::benchmark_data_smithi188_113957_object6090:head# 2024-04-23T16:33:56.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab8204e:::benchmark_data_smithi188_113957_object35223:head# 2024-04-23T16:33:56.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab82478:::benchmark_data_smithi188_113957_object5012:head# 2024-04-23T16:33:56.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab88e78:::benchmark_data_smithi188_113957_object51529:head# 2024-04-23T16:33:56.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab8bdbf:::benchmark_data_smithi188_113957_object121144:head# 2024-04-23T16:33:56.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab8c9f6:::benchmark_data_smithi188_113957_object106448:head# 2024-04-23T16:33:56.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab93ebb:::benchmark_data_smithi188_113957_object117857:head# 2024-04-23T16:33:56.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab9430b:::benchmark_data_smithi188_113957_object3927:head# 2024-04-23T16:33:56.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ab976c7:::benchmark_data_smithi188_113957_object64826:head# 2024-04-23T16:33:56.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aba5539:::benchmark_data_smithi188_113957_object77198:head# 2024-04-23T16:33:56.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5abb0a56:::benchmark_data_smithi188_113957_object126153:head# 2024-04-23T16:33:56.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5abc4288:::benchmark_data_smithi188_113957_object109395:head# 2024-04-23T16:33:56.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.737 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5abc8785:::benchmark_data_smithi188_113957_object58669:head# 2024-04-23T16:33:56.737 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.737 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5abcf9d9:::benchmark_data_smithi188_113957_object124821:head# 2024-04-23T16:33:56.737 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.737 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5abd0863:::benchmark_data_smithi188_113957_object5030:head# 2024-04-23T16:33:56.737 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.737 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5abd5755:::benchmark_data_smithi188_113957_object21444:head# 2024-04-23T16:33:56.737 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.737 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5abdc020:::benchmark_data_smithi188_113957_object135934:head# 2024-04-23T16:33:56.737 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.737 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5abde79f:::benchmark_data_smithi188_113957_object125540:head# 2024-04-23T16:33:56.737 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.737 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5abdef5a:::benchmark_data_smithi188_113957_object28725:head# 2024-04-23T16:33:56.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5abe61ad:::benchmark_data_smithi188_113957_object10220:head# 2024-04-23T16:33:56.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5abe687f:::benchmark_data_smithi188_113957_object120128:head# 2024-04-23T16:33:56.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5abe9103:::benchmark_data_smithi188_113957_object137699:head# 2024-04-23T16:33:56.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5abeb11d:::benchmark_data_smithi188_113957_object121627:head# 2024-04-23T16:33:56.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5abec261:::benchmark_data_smithi188_113957_object127275:head# 2024-04-23T16:33:56.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.738 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5abef06a:::benchmark_data_smithi188_113957_object104862:head# 2024-04-23T16:33:56.738 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5abf18a4:::benchmark_data_smithi188_113957_object13436:head# 2024-04-23T16:33:56.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5abfc149:::benchmark_data_smithi188_113957_object80405:head# 2024-04-23T16:33:56.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac02c7d:::benchmark_data_smithi188_113957_object77992:head# 2024-04-23T16:33:56.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac07b24:::benchmark_data_smithi188_113957_object20939:head# 2024-04-23T16:33:56.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac0aa37:::benchmark_data_smithi188_113957_object26880:head# 2024-04-23T16:33:56.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac12826:::benchmark_data_smithi188_113957_object50773:head# 2024-04-23T16:33:56.739 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.739 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac13e89:::benchmark_data_smithi188_113957_object51195:head# 2024-04-23T16:33:56.740 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.740 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac1add8:::benchmark_data_smithi188_113957_object9964:head# 2024-04-23T16:33:56.740 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.740 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac2c306:::benchmark_data_smithi188_113957_object118149:head# 2024-04-23T16:33:56.740 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.740 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac2c360:::benchmark_data_smithi188_113957_object130957:head# 2024-04-23T16:33:56.740 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.740 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac2d59e:::benchmark_data_smithi188_113957_object40737:head# 2024-04-23T16:33:56.740 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.740 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac2ead9:::benchmark_data_smithi188_113957_object811:head# 2024-04-23T16:33:56.740 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.740 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac3a3b3:::benchmark_data_smithi188_113957_object85543:head# 2024-04-23T16:33:56.740 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.740 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac3dcfa:::benchmark_data_smithi188_113957_object135013:head# 2024-04-23T16:33:56.741 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.741 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac40875:::benchmark_data_smithi188_113957_object7732:head# 2024-04-23T16:33:56.741 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.741 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac48d44:::benchmark_data_smithi188_113957_object3706:head# 2024-04-23T16:33:56.741 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.741 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac4ffdb:::benchmark_data_smithi188_113957_object145948:head# 2024-04-23T16:33:56.741 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.741 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac50e68:::benchmark_data_smithi188_113957_object101784:head# 2024-04-23T16:33:56.741 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.741 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac50f78:::benchmark_data_smithi188_113957_object38881:head# 2024-04-23T16:33:56.741 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.741 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac557bd:::benchmark_data_smithi188_113957_object26221:head# 2024-04-23T16:33:56.741 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.741 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac56914:::benchmark_data_smithi188_113957_object149836:head# 2024-04-23T16:33:56.742 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.742 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac5f54d:::benchmark_data_smithi188_113957_object84359:head# 2024-04-23T16:33:56.742 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.742 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac6e97f:::benchmark_data_smithi188_113957_object5943:head# 2024-04-23T16:33:56.742 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.742 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac894ca:::benchmark_data_smithi188_113957_object39645:head# 2024-04-23T16:33:56.742 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.742 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac8b126:::benchmark_data_smithi188_113957_object42536:head# 2024-04-23T16:33:56.742 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.742 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac9adb4:::benchmark_data_smithi188_113957_object57007:head# 2024-04-23T16:33:56.742 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.742 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac9b1b0:::benchmark_data_smithi188_113957_object2081:head# 2024-04-23T16:33:56.742 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.742 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac9d9da:::benchmark_data_smithi188_113957_object147980:head# 2024-04-23T16:33:56.743 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.743 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac9ddea:::benchmark_data_smithi188_113957_object145533:head# 2024-04-23T16:33:56.743 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.743 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac9e24d:::benchmark_data_smithi188_113957_object127278:head# 2024-04-23T16:33:56.743 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.743 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ac9e259:::benchmark_data_smithi188_113957_object14782:head# 2024-04-23T16:33:56.743 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.743 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aca0c3b:::benchmark_data_smithi188_113957_object43946:head# 2024-04-23T16:33:56.743 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.743 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5acba531:::benchmark_data_smithi188_113957_object105002:head# 2024-04-23T16:33:56.743 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.743 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5acc4565:::benchmark_data_smithi188_113957_object73870:head# 2024-04-23T16:33:56.743 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.743 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5acc6ab2:::benchmark_data_smithi188_113957_object94377:head# 2024-04-23T16:33:56.744 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.744 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5acd9187:::benchmark_data_smithi188_113957_object38898:head# 2024-04-23T16:33:56.744 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.744 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5acdb09a:::benchmark_data_smithi188_113957_object70341:head# 2024-04-23T16:33:56.744 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.744 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ace434c:::benchmark_data_smithi188_113957_object70692:head# 2024-04-23T16:33:56.744 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.744 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5acfb329:::benchmark_data_smithi188_113957_object69538:head# 2024-04-23T16:33:56.744 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.744 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5acfe2c3:::benchmark_data_smithi188_113957_object107683:head# 2024-04-23T16:33:56.744 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.744 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad1a612:::benchmark_data_smithi188_113957_object61165:head# 2024-04-23T16:33:56.744 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.744 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad1c3ce:::benchmark_data_smithi188_113957_object123246:head# 2024-04-23T16:33:56.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad1f996:::benchmark_data_smithi188_113957_object63003:head# 2024-04-23T16:33:56.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad201d3:::benchmark_data_smithi188_113957_object93513:head# 2024-04-23T16:33:56.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad20da5:::benchmark_data_smithi188_113957_object76369:head# 2024-04-23T16:33:56.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad2ceae:::benchmark_data_smithi188_113957_object22312:head# 2024-04-23T16:33:56.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad312fb:::benchmark_data_smithi188_113957_object121250:head# 2024-04-23T16:33:56.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad33ee5:::benchmark_data_smithi188_113957_object121578:head# 2024-04-23T16:33:56.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad36f62:::benchmark_data_smithi188_113957_object79084:head# 2024-04-23T16:33:56.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.746 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad430fd:::benchmark_data_smithi188_113957_object102281:head# 2024-04-23T16:33:56.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.746 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad4658d:::benchmark_data_smithi188_113957_object79403:head# 2024-04-23T16:33:56.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.746 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad46d7e:::benchmark_data_smithi188_113957_object54646:head# 2024-04-23T16:33:56.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.746 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad49034:::benchmark_data_smithi188_113957_object69520:head# 2024-04-23T16:33:56.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.746 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad4980a:::benchmark_data_smithi188_113957_object18701:head# 2024-04-23T16:33:56.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.746 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad4de88:::benchmark_data_smithi188_113957_object105012:head# 2024-04-23T16:33:56.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.747 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad523a8:::benchmark_data_smithi188_113957_object4047:head# 2024-04-23T16:33:56.747 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.747 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad5e0f7:::benchmark_data_smithi188_113957_object89762:head# 2024-04-23T16:33:56.747 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.747 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad60c25:::benchmark_data_smithi188_113957_object71388:head# 2024-04-23T16:33:56.747 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.747 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad6a0ae:::benchmark_data_smithi188_113957_object148405:head# 2024-04-23T16:33:56.747 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.747 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad6daad:::benchmark_data_smithi188_113957_object51448:head# 2024-04-23T16:33:56.747 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.747 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad760e5:::benchmark_data_smithi188_113957_object38620:head# 2024-04-23T16:33:56.747 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.747 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad7922e:::benchmark_data_smithi188_113957_object12978:head# 2024-04-23T16:33:56.747 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.748 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad7a715:::benchmark_data_smithi188_113957_object41373:head# 2024-04-23T16:33:56.748 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.748 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad7b571:::benchmark_data_smithi188_113957_object30893:head# 2024-04-23T16:33:56.748 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.748 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad7f4d4:::benchmark_data_smithi188_113957_object823:head# 2024-04-23T16:33:56.748 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.748 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad7ff0e:::benchmark_data_smithi188_113957_object39800:head# 2024-04-23T16:33:56.748 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.748 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad80859:::benchmark_data_smithi188_113957_object3688:head# 2024-04-23T16:33:56.748 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.748 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad8bdd7:::benchmark_data_smithi188_113957_object47901:head# 2024-04-23T16:33:56.748 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.748 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad8f72c:::benchmark_data_smithi188_113957_object57645:head# 2024-04-23T16:33:56.748 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad987ee:::benchmark_data_smithi188_113957_object103117:head# 2024-04-23T16:33:56.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ad9927f:::benchmark_data_smithi188_113957_object30269:head# 2024-04-23T16:33:56.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ada7a6f:::benchmark_data_smithi188_113957_object98261:head# 2024-04-23T16:33:56.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5adaa279:::benchmark_data_smithi188_113957_object90049:head# 2024-04-23T16:33:56.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5adae12a:::benchmark_data_smithi188_113957_object83235:head# 2024-04-23T16:33:56.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5adb43b5:::benchmark_data_smithi188_113957_object142180:head# 2024-04-23T16:33:56.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5adb8942:::benchmark_data_smithi188_113957_object124650:head# 2024-04-23T16:33:56.750 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.750 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5adc164b:::benchmark_data_smithi188_113957_object12498:head# 2024-04-23T16:33:56.750 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.750 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5adc8566:::benchmark_data_smithi188_113957_object96060:head# 2024-04-23T16:33:56.750 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.750 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5adcd60e:::benchmark_data_smithi188_113957_object43935:head# 2024-04-23T16:33:56.750 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.750 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ade6679:::benchmark_data_smithi188_113957_object29384:head# 2024-04-23T16:33:56.750 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.750 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5adea7a9:::benchmark_data_smithi188_113957_object76499:head# 2024-04-23T16:33:56.750 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.750 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aded5b9:::benchmark_data_smithi188_113957_object5428:head# 2024-04-23T16:33:56.750 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.750 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aded9b8:::benchmark_data_smithi188_113957_object32463:head# 2024-04-23T16:33:56.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5adfe365:::benchmark_data_smithi188_113957_object139995:head# 2024-04-23T16:33:56.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5adfeb79:::benchmark_data_smithi188_113957_object126960:head# 2024-04-23T16:33:56.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae01b89:::benchmark_data_smithi188_113957_object88061:head# 2024-04-23T16:33:56.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae03902:::benchmark_data_smithi188_113957_object8133:head# 2024-04-23T16:33:56.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae0a82a:::benchmark_data_smithi188_113957_object2995:head# 2024-04-23T16:33:56.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae0e144:::benchmark_data_smithi188_113957_object112353:head# 2024-04-23T16:33:56.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae0eb82:::benchmark_data_smithi188_113957_object76642:head# 2024-04-23T16:33:56.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae22215:::benchmark_data_smithi188_113957_object149942:head# 2024-04-23T16:33:56.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae22fe5:::benchmark_data_smithi188_113957_object32721:head# 2024-04-23T16:33:56.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae34de5:::benchmark_data_smithi188_113957_object28528:head# 2024-04-23T16:33:56.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae35464:::benchmark_data_smithi188_113957_object130696:head# 2024-04-23T16:33:56.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae364af:::benchmark_data_smithi188_113957_object144567:head# 2024-04-23T16:33:56.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae3d540:::benchmark_data_smithi188_113957_object135530:head# 2024-04-23T16:33:56.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.753 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae3f88e:::benchmark_data_smithi188_113957_object128904:head# 2024-04-23T16:33:56.753 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.753 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae45047:::benchmark_data_smithi188_113957_object77214:head# 2024-04-23T16:33:56.753 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.753 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae4fa8d:::benchmark_data_smithi188_113957_object6672:head# 2024-04-23T16:33:56.753 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.753 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae53e80:::benchmark_data_smithi188_113957_object68808:head# 2024-04-23T16:33:56.753 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.753 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae5f008:::benchmark_data_smithi188_113957_object12934:head# 2024-04-23T16:33:56.753 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.753 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae5f952:::benchmark_data_smithi188_113957_object147687:head# 2024-04-23T16:33:56.753 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.753 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae69ad5:::benchmark_data_smithi188_113957_object85546:head# 2024-04-23T16:33:56.753 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.754 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae72760:::benchmark_data_smithi188_113957_object112260:head# 2024-04-23T16:33:56.754 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.754 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae7edda:::benchmark_data_smithi188_113957_object80286:head# 2024-04-23T16:33:56.754 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.754 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae87335:::benchmark_data_smithi188_113957_object110400:head# 2024-04-23T16:33:56.754 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.754 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae8ae91:::benchmark_data_smithi188_113957_object112491:head# 2024-04-23T16:33:56.754 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.754 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae917c0:::benchmark_data_smithi188_113957_object56793:head# 2024-04-23T16:33:56.754 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.754 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ae9e96b:::benchmark_data_smithi188_113957_object79971:head# 2024-04-23T16:33:56.754 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.754 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aea1eaf:::benchmark_data_smithi188_113957_object30261:head# 2024-04-23T16:33:56.755 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.755 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aea2ee8:::benchmark_data_smithi188_113957_object9274:head# 2024-04-23T16:33:56.755 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.755 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aeb05b2:::benchmark_data_smithi188_113957_object11923:head# 2024-04-23T16:33:56.755 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.755 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aebd959:::benchmark_data_smithi188_113957_object101957:head# 2024-04-23T16:33:56.755 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.755 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aecc86b:::benchmark_data_smithi188_113957_object144249:head# 2024-04-23T16:33:56.755 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.755 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aeccbc1:::benchmark_data_smithi188_113957_object19913:head# 2024-04-23T16:33:56.755 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.755 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aed7e14:::benchmark_data_smithi188_113957_object11038:head# 2024-04-23T16:33:56.755 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.756 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aedb5db:::benchmark_data_smithi188_113957_object136674:head# 2024-04-23T16:33:56.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.756 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aede738:::benchmark_data_smithi188_113957_object46055:head# 2024-04-23T16:33:56.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.756 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aedef62:::benchmark_data_smithi188_113957_object65954:head# 2024-04-23T16:33:56.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.756 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aedf586:::benchmark_data_smithi188_113957_object120266:head# 2024-04-23T16:33:56.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.756 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aedfbf3:::benchmark_data_smithi188_113957_object5384:head# 2024-04-23T16:33:56.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.756 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aeec623:::benchmark_data_smithi188_113957_object142142:head# 2024-04-23T16:33:56.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.756 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aeef03d:::benchmark_data_smithi188_113957_object76011:head# 2024-04-23T16:33:56.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.757 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aeeff06:::benchmark_data_smithi188_113957_object31270:head# 2024-04-23T16:33:56.757 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.757 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af02d25:::benchmark_data_smithi188_113957_object121843:head# 2024-04-23T16:33:56.757 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.757 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af06029:::benchmark_data_smithi188_113957_object11569:head# 2024-04-23T16:33:56.757 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.757 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af1b2c1:::benchmark_data_smithi188_113957_object52211:head# 2024-04-23T16:33:56.757 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.757 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af1c860:::benchmark_data_smithi188_113957_object44441:head# 2024-04-23T16:33:56.757 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.757 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af21354:::benchmark_data_smithi188_113957_object66096:head# 2024-04-23T16:33:56.757 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.757 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af24927:::benchmark_data_smithi188_113957_object137554:head# 2024-04-23T16:33:56.757 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.757 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af26dc3:::benchmark_data_smithi188_113957_object107957:head# 2024-04-23T16:33:56.758 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.758 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af2b3ce:::benchmark_data_smithi188_113957_object81311:head# 2024-04-23T16:33:56.758 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.758 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af2ccec:::benchmark_data_smithi188_113957_object104547:head# 2024-04-23T16:33:56.758 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.758 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af36cd9:::benchmark_data_smithi188_113957_object4703:head# 2024-04-23T16:33:56.758 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.758 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af3aba3:::benchmark_data_smithi188_113957_object55791:head# 2024-04-23T16:33:56.758 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.758 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af3c5f9:::benchmark_data_smithi188_113957_object119946:head# 2024-04-23T16:33:56.758 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.758 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af494b5:::benchmark_data_smithi188_113957_object29975:head# 2024-04-23T16:33:56.758 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.759 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af4df46:::benchmark_data_smithi188_113957_object62062:head# 2024-04-23T16:33:56.759 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.759 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af52ab2:::benchmark_data_smithi188_113957_object152257:head# 2024-04-23T16:33:56.759 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.759 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af55c58:::benchmark_data_smithi188_113957_object77359:head# 2024-04-23T16:33:56.759 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.759 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af5ab6e:::benchmark_data_smithi188_113957_object106660:head# 2024-04-23T16:33:56.759 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.759 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af5df34:::benchmark_data_smithi188_113957_object142863:head# 2024-04-23T16:33:56.759 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.759 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af5e8e5:::benchmark_data_smithi188_113957_object145795:head# 2024-04-23T16:33:56.759 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.759 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af6b81d:::benchmark_data_smithi188_113957_object109299:head# 2024-04-23T16:33:56.759 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af7123e:::benchmark_data_smithi188_113957_object14354:head# 2024-04-23T16:33:56.760 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af79f1b:::benchmark_data_smithi188_113957_object90212:head# 2024-04-23T16:33:56.760 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af80580:::benchmark_data_smithi188_113957_object47181:head# 2024-04-23T16:33:56.760 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af84eb0:::benchmark_data_smithi188_113957_object150579:head# 2024-04-23T16:33:56.760 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af88f67:::benchmark_data_smithi188_113957_object3571:head# 2024-04-23T16:33:56.760 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af899b2:::benchmark_data_smithi188_113957_object86534:head# 2024-04-23T16:33:56.760 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af8ce14:::benchmark_data_smithi188_113957_object114535:head# 2024-04-23T16:33:56.760 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.761 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af93b36:::benchmark_data_smithi188_113957_object148519:head# 2024-04-23T16:33:56.761 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.761 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5af9bab7:::benchmark_data_smithi188_113957_object108436:head# 2024-04-23T16:33:56.761 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.761 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5afa6ff5:::benchmark_data_smithi188_113957_object15505:head# 2024-04-23T16:33:56.761 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.761 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5afc5077:::benchmark_data_smithi188_113957_object95464:head# 2024-04-23T16:33:56.761 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.761 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5afd062f:::benchmark_data_smithi188_113957_object13080:head# 2024-04-23T16:33:56.761 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.761 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5afd1be5:::benchmark_data_smithi188_113957_object23797:head# 2024-04-23T16:33:56.761 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.761 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5afdc42f:::benchmark_data_smithi188_113957_object36913:head# 2024-04-23T16:33:56.761 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.762 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5afe37d8:::benchmark_data_smithi188_113957_object124159:head# 2024-04-23T16:33:56.762 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.762 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5afed319:::benchmark_data_smithi188_113957_object1491:head# 2024-04-23T16:33:56.762 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.762 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aff11fa:::benchmark_data_smithi188_113957_object16812:head# 2024-04-23T16:33:56.762 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.762 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aff2c81:::benchmark_data_smithi188_113957_object147800:head# 2024-04-23T16:33:56.762 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.762 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aff769e:::benchmark_data_smithi188_113957_object144073:head# 2024-04-23T16:33:56.762 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.762 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5aff993f:::benchmark_data_smithi188_113957_object22260:head# 2024-04-23T16:33:56.762 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.762 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5affcf50:::benchmark_data_smithi188_113957_object73268:head# 2024-04-23T16:33:56.762 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.763 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5affd4c5:::benchmark_data_smithi188_113957_object124712:head# 2024-04-23T16:33:56.763 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.763 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b001169:::benchmark_data_smithi188_113957_object125068:head# 2024-04-23T16:33:56.763 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.763 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b004f35:::benchmark_data_smithi188_113957_object20611:head# 2024-04-23T16:33:56.763 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.763 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b00b827:::benchmark_data_smithi188_113957_object102109:head# 2024-04-23T16:33:56.763 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.763 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b01965d:::benchmark_data_smithi188_113957_object78317:head# 2024-04-23T16:33:56.763 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.763 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b01b241:::benchmark_data_smithi188_113957_object4160:head# 2024-04-23T16:33:56.763 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.763 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b01c6a7:::benchmark_data_smithi188_113957_object48989:head# 2024-04-23T16:33:56.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b02620e:::benchmark_data_smithi188_113957_object24027:head# 2024-04-23T16:33:56.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b028349:::benchmark_data_smithi188_113957_object99837:head# 2024-04-23T16:33:56.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b029354:::benchmark_data_smithi188_113957_object71161:head# 2024-04-23T16:33:56.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b02b93a:::benchmark_data_smithi188_113957_object24717:head# 2024-04-23T16:33:56.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b02bf32:::benchmark_data_smithi188_113957_object2647:head# 2024-04-23T16:33:56.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b033c19:::benchmark_data_smithi188_113957_object3061:head# 2024-04-23T16:33:56.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b0362eb:::benchmark_data_smithi188_113957_object3107:head# 2024-04-23T16:33:56.765 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.765 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b0366db:::benchmark_data_smithi188_113957_object67004:head# 2024-04-23T16:33:56.765 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.765 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b03b18c:::benchmark_data_smithi188_113957_object32416:head# 2024-04-23T16:33:56.765 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.765 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b03ff45:::benchmark_data_smithi188_113957_object55157:head# 2024-04-23T16:33:56.765 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.765 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b0440b7:::benchmark_data_smithi188_113957_object48826:head# 2024-04-23T16:33:56.765 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.765 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b0458e9:::benchmark_data_smithi188_113957_object45613:head# 2024-04-23T16:33:56.765 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.765 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b04dc4a:::benchmark_data_smithi188_113957_object124487:head# 2024-04-23T16:33:56.765 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b0530c7:::benchmark_data_smithi188_113957_object72391:head# 2024-04-23T16:33:56.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b05ebfd:::benchmark_data_smithi188_113957_object34026:head# 2024-04-23T16:33:56.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b06d0fc:::benchmark_data_smithi188_113957_object88353:head# 2024-04-23T16:33:56.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b0825af:::benchmark_data_smithi188_113957_object63693:head# 2024-04-23T16:33:56.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b089656:::benchmark_data_smithi188_113957_object73207:head# 2024-04-23T16:33:56.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b08af99:::benchmark_data_smithi188_113957_object124406:head# 2024-04-23T16:33:56.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.767 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b095638:::benchmark_data_smithi188_113957_object108680:head# 2024-04-23T16:33:56.767 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.767 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b098885:::benchmark_data_smithi188_113957_object56744:head# 2024-04-23T16:33:56.767 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.767 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b098e49:::benchmark_data_smithi188_113957_object21391:head# 2024-04-23T16:33:56.767 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.767 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b099a05:::benchmark_data_smithi188_113957_object27942:head# 2024-04-23T16:33:56.767 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.767 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b0af591:::benchmark_data_smithi188_113957_object56920:head# 2024-04-23T16:33:56.767 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.767 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b0b370a:::benchmark_data_smithi188_113957_object141069:head# 2024-04-23T16:33:56.767 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.767 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b0b6a0c:::benchmark_data_smithi188_113957_object87687:head# 2024-04-23T16:33:56.768 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.768 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b0c4df2:::benchmark_data_smithi188_113957_object63577:head# 2024-04-23T16:33:56.768 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.768 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b0c58d0:::benchmark_data_smithi188_113957_object76810:head# 2024-04-23T16:33:56.768 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.768 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b0d096e:::benchmark_data_smithi188_113957_object114264:head# 2024-04-23T16:33:56.768 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.768 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b0d09a7:::benchmark_data_smithi188_113957_object102146:head# 2024-04-23T16:33:56.768 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.768 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b0d6e65:::benchmark_data_smithi188_113957_object93771:head# 2024-04-23T16:33:56.768 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.768 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b0ec35b:::benchmark_data_smithi188_113957_object8423:head# 2024-04-23T16:33:56.768 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.769 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b0f3297:::benchmark_data_smithi188_113957_object127517:head# 2024-04-23T16:33:56.769 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.769 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b0fcaff:::benchmark_data_smithi188_113957_object138588:head# 2024-04-23T16:33:56.769 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.769 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b0fe3ed:::benchmark_data_smithi188_113957_object60316:head# 2024-04-23T16:33:56.769 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.769 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b10939b:::benchmark_data_smithi188_113957_object93099:head# 2024-04-23T16:33:56.769 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.769 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b11501a:::benchmark_data_smithi188_113957_object82072:head# 2024-04-23T16:33:56.769 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.769 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b115917:::benchmark_data_smithi188_113957_object56272:head# 2024-04-23T16:33:56.769 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.769 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b130524:::benchmark_data_smithi188_113957_object12614:head# 2024-04-23T16:33:56.769 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.770 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b13d1f1:::benchmark_data_smithi188_113957_object147893:head# 2024-04-23T16:33:56.770 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.770 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b14ec81:::benchmark_data_smithi188_113957_object59073:head# 2024-04-23T16:33:56.770 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.770 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b154576:::benchmark_data_smithi188_113957_object50214:head# 2024-04-23T16:33:56.770 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.770 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1545de:::benchmark_data_smithi188_113957_object127597:head# 2024-04-23T16:33:56.770 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.770 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b15a2c0:::benchmark_data_smithi188_113957_object117411:head# 2024-04-23T16:33:56.770 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.770 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1672d3:::benchmark_data_smithi188_113957_object128481:head# 2024-04-23T16:33:56.770 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.770 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b16897b:::benchmark_data_smithi188_113957_object76190:head# 2024-04-23T16:33:56.770 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.771 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b16b6ad:::benchmark_data_smithi188_113957_object119656:head# 2024-04-23T16:33:56.771 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.771 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b170b8d:::benchmark_data_smithi188_113957_object45809:head# 2024-04-23T16:33:56.771 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.771 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b17a294:::benchmark_data_smithi188_113957_object58787:head# 2024-04-23T16:33:56.771 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.771 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b190d01:::benchmark_data_smithi188_113957_object124334:head# 2024-04-23T16:33:56.771 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.771 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b19554c:::benchmark_data_smithi188_113957_object144327:head# 2024-04-23T16:33:56.771 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.771 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1989fc:::benchmark_data_smithi188_113957_object87995:head# 2024-04-23T16:33:56.771 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.771 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1a1929:::benchmark_data_smithi188_113957_object125531:head# 2024-04-23T16:33:56.771 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.772 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1a36a6:::benchmark_data_smithi188_113957_object44951:head# 2024-04-23T16:33:56.772 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.772 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1a3bc7:::benchmark_data_smithi188_113957_object22920:head# 2024-04-23T16:33:56.772 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.772 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1a6d35:::benchmark_data_smithi188_113957_object101255:head# 2024-04-23T16:33:56.772 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.772 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1a80d6:::benchmark_data_smithi188_113957_object23665:head# 2024-04-23T16:33:56.772 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.772 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1ac2af:::benchmark_data_smithi188_113957_object44144:head# 2024-04-23T16:33:56.772 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.772 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1bdd3f:::benchmark_data_smithi188_113957_object16338:head# 2024-04-23T16:33:56.772 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.772 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1bf4b9:::benchmark_data_smithi188_113957_object48976:head# 2024-04-23T16:33:56.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1c0015:::benchmark_data_smithi188_113957_object76775:head# 2024-04-23T16:33:56.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1d599a:::benchmark_data_smithi188_113957_object14072:head# 2024-04-23T16:33:56.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1d675f:::benchmark_data_smithi188_113957_object108038:head# 2024-04-23T16:33:56.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1d8447:::benchmark_data_smithi188_113957_object130818:head# 2024-04-23T16:33:56.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1dba62:::benchmark_data_smithi188_113957_object28592:head# 2024-04-23T16:33:56.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1dc000:::benchmark_data_smithi188_113957_object15942:head# 2024-04-23T16:33:56.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1dc477:::benchmark_data_smithi188_113957_object54139:head# 2024-04-23T16:33:56.774 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.774 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1df49e:::benchmark_data_smithi188_113957_object67261:head# 2024-04-23T16:33:56.774 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.774 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1e00c0:::benchmark_data_smithi188_113957_object151356:head# 2024-04-23T16:33:56.774 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.774 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1e2b3f:::benchmark_data_smithi188_113957_object110480:head# 2024-04-23T16:33:56.774 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.774 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1e4d60:::benchmark_data_smithi188_113957_object13063:head# 2024-04-23T16:33:56.774 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.774 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1e7b08:::benchmark_data_smithi188_113957_object122487:head# 2024-04-23T16:33:56.774 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.774 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1f4122:::benchmark_data_smithi188_113957_object115293:head# 2024-04-23T16:33:56.774 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.775 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b1ffe5e:::benchmark_data_smithi188_113957_object103024:head# 2024-04-23T16:33:56.775 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.775 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b20327e:::benchmark_data_smithi188_113957_object141712:head# 2024-04-23T16:33:56.775 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.775 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b212252:::benchmark_data_smithi188_113957_object31884:head# 2024-04-23T16:33:56.775 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.775 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b218223:::benchmark_data_smithi188_113957_object73023:head# 2024-04-23T16:33:56.775 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.775 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2224e9:::benchmark_data_smithi188_113957_object152355:head# 2024-04-23T16:33:56.775 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.775 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b224e89:::benchmark_data_smithi188_113957_object21614:head# 2024-04-23T16:33:56.775 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.775 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b227ec7:::benchmark_data_smithi188_113957_object78415:head# 2024-04-23T16:33:56.775 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.776 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b229028:::benchmark_data_smithi188_113957_object125774:head# 2024-04-23T16:33:56.776 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.776 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b229412:::benchmark_data_smithi188_113957_object64897:head# 2024-04-23T16:33:56.776 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.776 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2382ee:::benchmark_data_smithi188_113957_object83607:head# 2024-04-23T16:33:56.776 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.776 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b23b540:::benchmark_data_smithi188_113957_object50736:head# 2024-04-23T16:33:56.776 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.776 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2437c7:::benchmark_data_smithi188_113957_object3151:head# 2024-04-23T16:33:56.776 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.776 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b24b7b3:::benchmark_data_smithi188_113957_object129444:head# 2024-04-23T16:33:56.776 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.776 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b256a2d:::benchmark_data_smithi188_113957_object134121:head# 2024-04-23T16:33:56.776 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.777 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b25984d:::benchmark_data_smithi188_113957_object44826:head# 2024-04-23T16:33:56.777 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.777 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b26746f:::benchmark_data_smithi188_113957_object49911:head# 2024-04-23T16:33:56.777 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.777 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b26f62f:::benchmark_data_smithi188_113957_object10174:head# 2024-04-23T16:33:56.777 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.777 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b271e9c:::benchmark_data_smithi188_113957_object101180:head# 2024-04-23T16:33:56.777 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.777 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b272c3c:::benchmark_data_smithi188_113957_object143681:head# 2024-04-23T16:33:56.777 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.777 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b286976:::benchmark_data_smithi188_113957_object110333:head# 2024-04-23T16:33:56.777 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.777 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2920af:::benchmark_data_smithi188_113957_object8526:head# 2024-04-23T16:33:56.777 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.778 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2a0f0d:::benchmark_data_smithi188_113957_object79226:head# 2024-04-23T16:33:56.778 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.778 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2a6a40:::benchmark_data_smithi188_113957_object44073:head# 2024-04-23T16:33:56.778 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.778 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2b193c:::benchmark_data_smithi188_113957_object31185:head# 2024-04-23T16:33:56.778 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.778 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2c6238:::benchmark_data_smithi188_113957_object85568:head# 2024-04-23T16:33:56.778 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.778 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2cce8d:::benchmark_data_smithi188_113957_object50345:head# 2024-04-23T16:33:56.778 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.778 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2d1ea1:::benchmark_data_smithi188_113957_object128723:head# 2024-04-23T16:33:56.778 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.778 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2d4a59:::benchmark_data_smithi188_113957_object99744:head# 2024-04-23T16:33:56.778 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2d9df3:::benchmark_data_smithi188_113957_object76650:head# 2024-04-23T16:33:56.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2de567:::benchmark_data_smithi188_113957_object3931:head# 2024-04-23T16:33:56.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2e29b3:::benchmark_data_smithi188_113957_object108817:head# 2024-04-23T16:33:56.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2ea644:::benchmark_data_smithi188_113957_object102166:head# 2024-04-23T16:33:56.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2eb0d2:::benchmark_data_smithi188_113957_object78792:head# 2024-04-23T16:33:56.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2ee149:::benchmark_data_smithi188_113957_object98560:head# 2024-04-23T16:33:56.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2eee0b:::benchmark_data_smithi188_113957_object137839:head# 2024-04-23T16:33:56.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2f0e3b:::benchmark_data_smithi188_113957_object75897:head# 2024-04-23T16:33:56.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2f501d:::benchmark_data_smithi188_113957_object11731:head# 2024-04-23T16:33:56.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2fbbdb:::benchmark_data_smithi188_113957_object46443:head# 2024-04-23T16:33:56.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b2fd879:::benchmark_data_smithi188_113957_object117207:head# 2024-04-23T16:33:56.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b30606c:::benchmark_data_smithi188_113957_object98781:head# 2024-04-23T16:33:56.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b30ea3b:::benchmark_data_smithi188_113957_object2777:head# 2024-04-23T16:33:56.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b31f308:::benchmark_data_smithi188_113957_object117897:head# 2024-04-23T16:33:56.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b325310:::benchmark_data_smithi188_113957_object688:head# 2024-04-23T16:33:56.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b3449e2:::benchmark_data_smithi188_113957_object50301:head# 2024-04-23T16:33:56.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b349024:::benchmark_data_smithi188_113957_object110447:head# 2024-04-23T16:33:56.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b34b0ec:::benchmark_data_smithi188_113957_object87652:head# 2024-04-23T16:33:56.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b3519ba:::benchmark_data_smithi188_113957_object111852:head# 2024-04-23T16:33:56.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b352960:::benchmark_data_smithi188_113957_object95122:head# 2024-04-23T16:33:56.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b354ca8:::benchmark_data_smithi188_113957_object25095:head# 2024-04-23T16:33:56.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.782 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b35c856:::benchmark_data_smithi188_113957_object14957:head# 2024-04-23T16:33:56.782 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.782 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b362679:::benchmark_data_smithi188_113957_object139027:head# 2024-04-23T16:33:56.782 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.782 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b377277:::benchmark_data_smithi188_113957_object89012:head# 2024-04-23T16:33:56.782 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.782 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b3854cd:::benchmark_data_smithi188_113957_object101628:head# 2024-04-23T16:33:56.782 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.782 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b395c76:::benchmark_data_smithi188_113957_object71778:head# 2024-04-23T16:33:56.782 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.782 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b396a93:::benchmark_data_smithi188_113957_object71228:head# 2024-04-23T16:33:56.782 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.782 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b398f3d:::benchmark_data_smithi188_113957_object71979:head# 2024-04-23T16:33:56.783 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.783 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b39d00d:::benchmark_data_smithi188_113957_object46424:head# 2024-04-23T16:33:56.783 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.783 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b3a13b7:::benchmark_data_smithi188_113957_object52478:head# 2024-04-23T16:33:56.783 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.783 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b3c35e7:::benchmark_data_smithi188_113957_object29557:head# 2024-04-23T16:33:56.783 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.783 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b3c7653:::benchmark_data_smithi188_113957_object30789:head# 2024-04-23T16:33:56.783 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.783 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b3c7b2d:::benchmark_data_smithi188_113957_object37258:head# 2024-04-23T16:33:56.783 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.783 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b3ca917:::benchmark_data_smithi188_113957_object133264:head# 2024-04-23T16:33:56.783 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b3cc15b:::benchmark_data_smithi188_113957_object129326:head# 2024-04-23T16:33:56.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b3d6f0f:::benchmark_data_smithi188_113957_object63051:head# 2024-04-23T16:33:56.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b3e89d5:::benchmark_data_smithi188_113957_object38611:head# 2024-04-23T16:33:56.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b3edb4d:::benchmark_data_smithi188_113957_object78145:head# 2024-04-23T16:33:56.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b3f79fa:::benchmark_data_smithi188_113957_object150451:head# 2024-04-23T16:33:56.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b4006f4:::benchmark_data_smithi188_113957_object117021:head# 2024-04-23T16:33:56.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b4071c3:::benchmark_data_smithi188_113957_object60359:head# 2024-04-23T16:33:56.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b40fabd:::benchmark_data_smithi188_113957_object90233:head# 2024-04-23T16:33:56.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b41edfd:::benchmark_data_smithi188_113957_object139819:head# 2024-04-23T16:33:56.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b4272ec:::benchmark_data_smithi188_113957_object114017:head# 2024-04-23T16:33:56.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b437965:::benchmark_data_smithi188_113957_object2610:head# 2024-04-23T16:33:56.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b457527:::benchmark_data_smithi188_113957_object130277:head# 2024-04-23T16:33:56.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b4578a2:::benchmark_data_smithi188_113957_object7844:head# 2024-04-23T16:33:56.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b46dfe8:::benchmark_data_smithi188_113957_object116190:head# 2024-04-23T16:33:56.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b4765df:::benchmark_data_smithi188_113957_object31939:head# 2024-04-23T16:33:56.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b482e40:::benchmark_data_smithi188_113957_object37946:head# 2024-04-23T16:33:56.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b48f104:::benchmark_data_smithi188_113957_object13955:head# 2024-04-23T16:33:56.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b493dbe:::benchmark_data_smithi188_113957_object7842:head# 2024-04-23T16:33:56.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b49d03b:::benchmark_data_smithi188_113957_object124356:head# 2024-04-23T16:33:56.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b4a1327:::benchmark_data_smithi188_113957_object118860:head# 2024-04-23T16:33:56.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b4a2494:::benchmark_data_smithi188_113957_object136013:head# 2024-04-23T16:33:56.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b4a9be9:::benchmark_data_smithi188_113957_object63514:head# 2024-04-23T16:33:56.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b4aab66:::benchmark_data_smithi188_113957_object4095:head# 2024-04-23T16:33:56.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b4ac600:::benchmark_data_smithi188_113957_object92177:head# 2024-04-23T16:33:56.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b4b3acb:::benchmark_data_smithi188_113957_object23177:head# 2024-04-23T16:33:56.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b4eb297:::benchmark_data_smithi188_113957_object116831:head# 2024-04-23T16:33:56.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b4ef5e0:::benchmark_data_smithi188_113957_object4739:head# 2024-04-23T16:33:56.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b4fb85d:::benchmark_data_smithi188_113957_object27400:head# 2024-04-23T16:33:56.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b4ff347:::benchmark_data_smithi188_113957_object20764:head# 2024-04-23T16:33:56.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b502851:::benchmark_data_smithi188_113957_object121619:head# 2024-04-23T16:33:56.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b50d897:::benchmark_data_smithi188_113957_object37894:head# 2024-04-23T16:33:56.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b50e329:::benchmark_data_smithi188_113957_object85436:head# 2024-04-23T16:33:56.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b528d72:::benchmark_data_smithi188_113957_object61264:head# 2024-04-23T16:33:56.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b52d0ca:::benchmark_data_smithi188_113957_object113162:head# 2024-04-23T16:33:56.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b538de0:::benchmark_data_smithi188_113957_object17827:head# 2024-04-23T16:33:56.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b539989:::benchmark_data_smithi188_113957_object150552:head# 2024-04-23T16:33:56.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b53c745:::benchmark_data_smithi188_113957_object70311:head# 2024-04-23T16:33:56.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b53e569:::benchmark_data_smithi188_113957_object25985:head# 2024-04-23T16:33:56.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b53ebf5:::benchmark_data_smithi188_113957_object82200:head# 2024-04-23T16:33:56.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b54f9d4:::benchmark_data_smithi188_113957_object33140:head# 2024-04-23T16:33:56.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b55055a:::benchmark_data_smithi188_113957_object87461:head# 2024-04-23T16:33:56.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b555d82:::benchmark_data_smithi188_113957_object97484:head# 2024-04-23T16:33:56.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b55d2c4:::benchmark_data_smithi188_113957_object150045:head# 2024-04-23T16:33:56.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b55d934:::benchmark_data_smithi188_113957_object119138:head# 2024-04-23T16:33:56.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b55f89a:::benchmark_data_smithi188_113957_object138279:head# 2024-04-23T16:33:56.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b561ca0:::benchmark_data_smithi188_113957_object82510:head# 2024-04-23T16:33:56.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b565953:::benchmark_data_smithi188_113957_object34207:head# 2024-04-23T16:33:56.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b566508:::benchmark_data_smithi188_113957_object114011:head# 2024-04-23T16:33:56.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.791 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b567a9d:::benchmark_data_smithi188_113957_object98772:head# 2024-04-23T16:33:56.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.791 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b56b942:::benchmark_data_smithi188_113957_object45684:head# 2024-04-23T16:33:56.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.791 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b576382:::benchmark_data_smithi188_113957_object63001:head# 2024-04-23T16:33:56.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.791 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b5784e7:::benchmark_data_smithi188_113957_object37855:head# 2024-04-23T16:33:56.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.791 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b57bccc:::benchmark_data_smithi188_113957_object63310:head# 2024-04-23T16:33:56.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.791 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b57c712:::benchmark_data_smithi188_113957_object44111:head# 2024-04-23T16:33:56.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.791 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b57eb46:::benchmark_data_smithi188_113957_object57614:head# 2024-04-23T16:33:56.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b58320d:::benchmark_data_smithi188_113957_object121444:head# 2024-04-23T16:33:56.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b58e5f2:::benchmark_data_smithi188_113957_object140903:head# 2024-04-23T16:33:56.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b58e960:::benchmark_data_smithi188_113957_object40983:head# 2024-04-23T16:33:56.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b594050:::benchmark_data_smithi188_113957_object28578:head# 2024-04-23T16:33:56.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b59507c:::benchmark_data_smithi188_113957_object2616:head# 2024-04-23T16:33:56.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b596641:::benchmark_data_smithi188_113957_object87207:head# 2024-04-23T16:33:56.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b5975ae:::benchmark_data_smithi188_113957_object126460:head# 2024-04-23T16:33:56.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b5976a6:::benchmark_data_smithi188_113957_object22746:head# 2024-04-23T16:33:56.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b597cc7:::benchmark_data_smithi188_113957_object46061:head# 2024-04-23T16:33:56.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b59a217:::benchmark_data_smithi188_113957_object30867:head# 2024-04-23T16:33:56.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b5a12f4:::benchmark_data_smithi188_113957_object85379:head# 2024-04-23T16:33:56.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b5a4be6:::benchmark_data_smithi188_113957_object96287:head# 2024-04-23T16:33:56.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b5a8549:::benchmark_data_smithi188_113957_object126422:head# 2024-04-23T16:33:56.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b5abfe6:::benchmark_data_smithi188_113957_object112360:head# 2024-04-23T16:33:56.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b5ae2dc:::benchmark_data_smithi188_113957_object60928:head# 2024-04-23T16:33:56.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b5c0889:::benchmark_data_smithi188_113957_object64392:head# 2024-04-23T16:33:56.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b5c3453:::benchmark_data_smithi188_113957_object144331:head# 2024-04-23T16:33:56.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b5c9c9b:::benchmark_data_smithi188_113957_object36578:head# 2024-04-23T16:33:56.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b5c9f18:::benchmark_data_smithi188_113957_object87275:head# 2024-04-23T16:33:56.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b5cbfa2:::benchmark_data_smithi188_113957_object37588:head# 2024-04-23T16:33:56.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b5cef7e:::benchmark_data_smithi188_113957_object51971:head# 2024-04-23T16:33:56.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b5d674c:::benchmark_data_smithi188_113957_object136727:head# 2024-04-23T16:33:56.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b5f5ead:::benchmark_data_smithi188_113957_object89929:head# 2024-04-23T16:33:56.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b5f6110:::benchmark_data_smithi188_113957_object98719:head# 2024-04-23T16:33:56.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b5f76c9:::benchmark_data_smithi188_113957_object60175:head# 2024-04-23T16:33:56.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b5f8f23:::benchmark_data_smithi188_113957_object78686:head# 2024-04-23T16:33:56.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b6083cf:::benchmark_data_smithi188_113957_object28513:head# 2024-04-23T16:33:56.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b60bafb:::benchmark_data_smithi188_113957_object92593:head# 2024-04-23T16:33:56.796 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.796 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b61587b:::benchmark_data_smithi188_113957_object59502:head# 2024-04-23T16:33:56.796 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.796 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b6209ce:::benchmark_data_smithi188_113957_object144198:head# 2024-04-23T16:33:56.796 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.796 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b64755b:::benchmark_data_smithi188_113957_object12027:head# 2024-04-23T16:33:56.796 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.796 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b65288a:::benchmark_data_smithi188_113957_object111786:head# 2024-04-23T16:33:56.796 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.796 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b65c405:::benchmark_data_smithi188_113957_object114433:head# 2024-04-23T16:33:56.796 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.796 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b663d61:::benchmark_data_smithi188_113957_object62473:head# 2024-04-23T16:33:56.796 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.796 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b66407b:::benchmark_data_smithi188_113957_object88247:head# 2024-04-23T16:33:56.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b667ea3:::benchmark_data_smithi188_113957_object92134:head# 2024-04-23T16:33:56.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b679da0:::benchmark_data_smithi188_113957_object17102:head# 2024-04-23T16:33:56.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b682613:::benchmark_data_smithi188_113957_object88953:head# 2024-04-23T16:33:56.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b682d76:::benchmark_data_smithi188_113957_object81973:head# 2024-04-23T16:33:56.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b68b22b:::benchmark_data_smithi188_113957_object10880:head# 2024-04-23T16:33:56.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b691151:::benchmark_data_smithi188_113957_object8069:head# 2024-04-23T16:33:56.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b6957cc:::benchmark_data_smithi188_113957_object113840:head# 2024-04-23T16:33:56.798 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.798 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b6a4968:::benchmark_data_smithi188_113957_object76577:head# 2024-04-23T16:33:56.798 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.798 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b6a5e14:::benchmark_data_smithi188_113957_object134639:head# 2024-04-23T16:33:56.798 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.798 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b6ab884:::benchmark_data_smithi188_113957_object67953:head# 2024-04-23T16:33:56.798 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.798 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b6adf4f:::benchmark_data_smithi188_113957_object32039:head# 2024-04-23T16:33:56.798 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.798 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b6b8e81:::benchmark_data_smithi188_113957_object32836:head# 2024-04-23T16:33:56.798 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.798 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b6ba975:::benchmark_data_smithi188_113957_object117461:head# 2024-04-23T16:33:56.798 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.798 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b6bc03d:::benchmark_data_smithi188_113957_object88038:head# 2024-04-23T16:33:56.799 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.799 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b6e05ea:::benchmark_data_smithi188_113957_object137807:head# 2024-04-23T16:33:56.799 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.799 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b6e445a:::benchmark_data_smithi188_113957_object22265:head# 2024-04-23T16:33:56.799 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.799 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b6ee6a5:::benchmark_data_smithi188_113957_object68056:head# 2024-04-23T16:33:56.799 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.799 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b6f3384:::benchmark_data_smithi188_113957_object151931:head# 2024-04-23T16:33:56.799 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.799 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b6f4ea5:::benchmark_data_smithi188_113957_object115162:head# 2024-04-23T16:33:56.799 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.799 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b6f6d54:::benchmark_data_smithi188_113957_object56501:head# 2024-04-23T16:33:56.799 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.799 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b6fd7c3:::benchmark_data_smithi188_113957_object26631:head# 2024-04-23T16:33:56.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b6fe3b8:::benchmark_data_smithi188_113957_object123762:head# 2024-04-23T16:33:56.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b700ee1:::benchmark_data_smithi188_113957_object104622:head# 2024-04-23T16:33:56.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b707ada:::benchmark_data_smithi188_113957_object54297:head# 2024-04-23T16:33:56.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b70a52e:::benchmark_data_smithi188_113957_object148563:head# 2024-04-23T16:33:56.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b70b709:::benchmark_data_smithi188_113957_object74872:head# 2024-04-23T16:33:56.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b70e238:::benchmark_data_smithi188_113957_object52227:head# 2024-04-23T16:33:56.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b7113ab:::benchmark_data_smithi188_113957_object142289:head# 2024-04-23T16:33:56.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b71462e:::benchmark_data_smithi188_113957_object133084:head# 2024-04-23T16:33:56.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b7166ec:::benchmark_data_smithi188_113957_object11509:head# 2024-04-23T16:33:56.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b71e9eb:::benchmark_data_smithi188_113957_object121098:head# 2024-04-23T16:33:56.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b7202c8:::benchmark_data_smithi188_113957_object18062:head# 2024-04-23T16:33:56.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b727288:::benchmark_data_smithi188_113957_object64677:head# 2024-04-23T16:33:56.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b72e7ec:::benchmark_data_smithi188_113957_object65470:head# 2024-04-23T16:33:56.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b738ef6:::benchmark_data_smithi188_113957_object18233:head# 2024-04-23T16:33:56.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b753e6b:::benchmark_data_smithi188_113957_object136659:head# 2024-04-23T16:33:56.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b75471d:::benchmark_data_smithi188_113957_object121393:head# 2024-04-23T16:33:56.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b761d6b:::benchmark_data_smithi188_113957_object96425:head# 2024-04-23T16:33:56.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b76820a:::benchmark_data_smithi188_113957_object14481:head# 2024-04-23T16:33:56.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b768568:::benchmark_data_smithi188_113957_object120079:head# 2024-04-23T16:33:56.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b76de63:::benchmark_data_smithi188_113957_object64556:head# 2024-04-23T16:33:56.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b775c6e:::benchmark_data_smithi188_113957_object83090:head# 2024-04-23T16:33:56.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.803 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b77dff5:::benchmark_data_smithi188_113957_object80765:head# 2024-04-23T16:33:56.803 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.803 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b781701:::benchmark_data_smithi188_113957_object143481:head# 2024-04-23T16:33:56.803 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.803 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b7850fb:::benchmark_data_smithi188_113957_object137529:head# 2024-04-23T16:33:56.803 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.803 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b78a450:::benchmark_data_smithi188_113957_object84844:head# 2024-04-23T16:33:56.803 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.803 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b78c09a:::benchmark_data_smithi188_113957_object6495:head# 2024-04-23T16:33:56.803 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.803 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b79c6ac:::benchmark_data_smithi188_113957_object115587:head# 2024-04-23T16:33:56.803 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.803 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b79ff47:::benchmark_data_smithi188_113957_object143149:head# 2024-04-23T16:33:56.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b7a1508:::benchmark_data_smithi188_113957_object120957:head# 2024-04-23T16:33:56.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b7a25b9:::benchmark_data_smithi188_113957_object149999:head# 2024-04-23T16:33:56.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b7a58a1:::benchmark_data_smithi188_113957_object36:head# 2024-04-23T16:33:56.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b7a7e1e:::benchmark_data_smithi188_113957_object145562:head# 2024-04-23T16:33:56.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b7ab97b:::benchmark_data_smithi188_113957_object80324:head# 2024-04-23T16:33:56.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b7b6b95:::benchmark_data_smithi188_113957_object90990:head# 2024-04-23T16:33:56.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b7bf007:::benchmark_data_smithi188_113957_object46431:head# 2024-04-23T16:33:56.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b7dc8a3:::benchmark_data_smithi188_113957_object92452:head# 2024-04-23T16:33:56.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b7e100a:::benchmark_data_smithi188_113957_object17623:head# 2024-04-23T16:33:56.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b7f8f06:::benchmark_data_smithi188_113957_object143996:head# 2024-04-23T16:33:56.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b8031d2:::benchmark_data_smithi188_113957_object150582:head# 2024-04-23T16:33:56.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b817dfc:::benchmark_data_smithi188_113957_object105930:head# 2024-04-23T16:33:56.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b81853d:::benchmark_data_smithi188_113957_object147540:head# 2024-04-23T16:33:56.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b81ab6f:::benchmark_data_smithi188_113957_object113582:head# 2024-04-23T16:33:56.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b81f5c0:::benchmark_data_smithi188_113957_object83342:head# 2024-04-23T16:33:56.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b820541:::benchmark_data_smithi188_113957_object34511:head# 2024-04-23T16:33:56.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b82b24d:::benchmark_data_smithi188_113957_object39373:head# 2024-04-23T16:33:56.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b8370f1:::benchmark_data_smithi188_113957_object88343:head# 2024-04-23T16:33:56.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b846068:::benchmark_data_smithi188_113957_object40387:head# 2024-04-23T16:33:56.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b8548fb:::benchmark_data_smithi188_113957_object115532:head# 2024-04-23T16:33:56.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b861667:::benchmark_data_smithi188_113957_object34182:head# 2024-04-23T16:33:56.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b863520:::benchmark_data_smithi188_113957_object72703:head# 2024-04-23T16:33:56.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b86788b:::benchmark_data_smithi188_113957_object109475:head# 2024-04-23T16:33:56.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b867975:::benchmark_data_smithi188_113957_object17856:head# 2024-04-23T16:33:56.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b86b93b:::benchmark_data_smithi188_113957_object59458:head# 2024-04-23T16:33:56.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b86cf96:::benchmark_data_smithi188_113957_object71544:head# 2024-04-23T16:33:56.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b877e9b:::benchmark_data_smithi188_113957_object72304:head# 2024-04-23T16:33:56.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b87ac1a:::benchmark_data_smithi188_113957_object77079:head# 2024-04-23T16:33:56.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b87de36:::benchmark_data_smithi188_113957_object117050:head# 2024-04-23T16:33:56.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b87f5a6:::benchmark_data_smithi188_113957_object78820:head# 2024-04-23T16:33:56.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b88c81e:::benchmark_data_smithi188_113957_object19563:head# 2024-04-23T16:33:56.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b88e61a:::benchmark_data_smithi188_113957_object127183:head# 2024-04-23T16:33:56.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b89dfc3:::benchmark_data_smithi188_113957_object23863:head# 2024-04-23T16:33:56.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b89e424:::benchmark_data_smithi188_113957_object65994:head# 2024-04-23T16:33:56.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b8b0503:::benchmark_data_smithi188_113957_object98596:head# 2024-04-23T16:33:56.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b8b4663:::benchmark_data_smithi188_113957_object5482:head# 2024-04-23T16:33:56.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b8b5b1d:::benchmark_data_smithi188_113957_object22701:head# 2024-04-23T16:33:56.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b8bc5ab:::benchmark_data_smithi188_113957_object100929:head# 2024-04-23T16:33:56.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b8beecf:::benchmark_data_smithi188_113957_object30080:head# 2024-04-23T16:33:56.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b8c408d:::benchmark_data_smithi188_113957_object42285:head# 2024-04-23T16:33:56.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b8c6390:::benchmark_data_smithi188_113957_object82717:head# 2024-04-23T16:33:56.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b8cc335:::benchmark_data_smithi188_113957_object142047:head# 2024-04-23T16:33:56.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b8d88a0:::benchmark_data_smithi188_113957_object60658:head# 2024-04-23T16:33:56.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b8db770:::benchmark_data_smithi188_113957_object5124:head# 2024-04-23T16:33:56.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b8e3bc3:::benchmark_data_smithi188_113957_object32632:head# 2024-04-23T16:33:56.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b908d53:::benchmark_data_smithi188_113957_object8245:head# 2024-04-23T16:33:56.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b9245aa:::benchmark_data_smithi188_113957_object123027:head# 2024-04-23T16:33:56.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b927e76:::benchmark_data_smithi188_113957_object38728:head# 2024-04-23T16:33:56.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b929d0f:::benchmark_data_smithi188_113957_object121003:head# 2024-04-23T16:33:56.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b92b927:::benchmark_data_smithi188_113957_object90460:head# 2024-04-23T16:33:56.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b92d17a:::benchmark_data_smithi188_113957_object147626:head# 2024-04-23T16:33:56.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b932fbe:::benchmark_data_smithi188_113957_object121414:head# 2024-04-23T16:33:56.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b93e791:::benchmark_data_smithi188_113957_object25765:head# 2024-04-23T16:33:56.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b942351:::benchmark_data_smithi188_113957_object142100:head# 2024-04-23T16:33:56.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b9496db:::benchmark_data_smithi188_113957_object54803:head# 2024-04-23T16:33:56.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b94b971:::benchmark_data_smithi188_113957_object63212:head# 2024-04-23T16:33:56.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b94d5ea:::benchmark_data_smithi188_113957_object6456:head# 2024-04-23T16:33:56.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b96e553:::benchmark_data_smithi188_113957_object79368:head# 2024-04-23T16:33:56.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b9733fe:::benchmark_data_smithi188_113957_object150602:head# 2024-04-23T16:33:56.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b97dc2f:::benchmark_data_smithi188_113957_object131104:head# 2024-04-23T16:33:56.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b982573:::benchmark_data_smithi188_113957_object9639:head# 2024-04-23T16:33:56.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b98a656:::benchmark_data_smithi188_113957_object20595:head# 2024-04-23T16:33:56.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b993ccb:::benchmark_data_smithi188_113957_object23791:head# 2024-04-23T16:33:56.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b99b6fb:::benchmark_data_smithi188_113957_object84807:head# 2024-04-23T16:33:56.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b99bac9:::benchmark_data_smithi188_113957_object21309:head# 2024-04-23T16:33:56.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b99c9ea:::benchmark_data_smithi188_113957_object44187:head# 2024-04-23T16:33:56.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b9a0aeb:::benchmark_data_smithi188_113957_object143519:head# 2024-04-23T16:33:56.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b9a315b:::benchmark_data_smithi188_113957_object68994:head# 2024-04-23T16:33:56.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b9a835a:::benchmark_data_smithi188_113957_object144577:head# 2024-04-23T16:33:56.814 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b9af8f9:::benchmark_data_smithi188_113957_object66163:head# 2024-04-23T16:33:56.814 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b9c917c:::benchmark_data_smithi188_113957_object85446:head# 2024-04-23T16:33:56.814 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b9cb4d1:::benchmark_data_smithi188_113957_object74461:head# 2024-04-23T16:33:56.814 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b9cd60b:::benchmark_data_smithi188_113957_object11608:head# 2024-04-23T16:33:56.814 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b9d5226:::benchmark_data_smithi188_113957_object141015:head# 2024-04-23T16:33:56.814 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b9dae10:::benchmark_data_smithi188_113957_object30219:head# 2024-04-23T16:33:56.814 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b9ddb42:::benchmark_data_smithi188_113957_object120788:head# 2024-04-23T16:33:56.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b9e18da:::benchmark_data_smithi188_113957_object116141:head# 2024-04-23T16:33:56.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b9e288a:::benchmark_data_smithi188_113957_object57705:head# 2024-04-23T16:33:56.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b9e8f2e:::benchmark_data_smithi188_113957_object40170:head# 2024-04-23T16:33:56.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b9e9ca1:::benchmark_data_smithi188_113957_object149027:head# 2024-04-23T16:33:56.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5b9f44a3:::benchmark_data_smithi188_113957_object120367:head# 2024-04-23T16:33:56.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba02414:::benchmark_data_smithi188_113957_object47253:head# 2024-04-23T16:33:56.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba029c7:::benchmark_data_smithi188_113957_object150832:head# 2024-04-23T16:33:56.816 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.816 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba0e897:::benchmark_data_smithi188_113957_object114117:head# 2024-04-23T16:33:56.816 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.816 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba0ff39:::benchmark_data_smithi188_113957_object152665:head# 2024-04-23T16:33:56.816 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.816 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba2dd4c:::benchmark_data_smithi188_113957_object84835:head# 2024-04-23T16:33:56.816 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.816 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba3b289:::benchmark_data_smithi188_113957_object81166:head# 2024-04-23T16:33:56.816 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.816 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba3bdc7:::benchmark_data_smithi188_113957_object33751:head# 2024-04-23T16:33:56.816 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.816 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba3c0b2:::benchmark_data_smithi188_113957_object131274:head# 2024-04-23T16:33:56.816 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.817 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba3c436:::benchmark_data_smithi188_113957_object105714:head# 2024-04-23T16:33:56.817 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.817 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba3cb22:::benchmark_data_smithi188_113957_object69119:head# 2024-04-23T16:33:56.817 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.817 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba517f4:::benchmark_data_smithi188_113957_object13438:head# 2024-04-23T16:33:56.817 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.817 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba54f3e:::benchmark_data_smithi188_113957_object12384:head# 2024-04-23T16:33:56.817 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.817 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba58a65:::benchmark_data_smithi188_113957_object80698:head# 2024-04-23T16:33:56.817 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.817 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba5a05d:::benchmark_data_smithi188_113957_object48848:head# 2024-04-23T16:33:56.817 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.817 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba62952:::benchmark_data_smithi188_113957_object43152:head# 2024-04-23T16:33:56.817 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba6547c:::benchmark_data_smithi188_113957_object133642:head# 2024-04-23T16:33:56.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba715bb:::benchmark_data_smithi188_113957_object87970:head# 2024-04-23T16:33:56.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba74b60:::benchmark_data_smithi188_113957_object123398:head# 2024-04-23T16:33:56.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba7c638:::benchmark_data_smithi188_113957_object77174:head# 2024-04-23T16:33:56.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba826ab:::benchmark_data_smithi188_113957_object23014:head# 2024-04-23T16:33:56.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba883e4:::benchmark_data_smithi188_113957_object6446:head# 2024-04-23T16:33:56.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba91335:::benchmark_data_smithi188_113957_object98700:head# 2024-04-23T16:33:56.819 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.819 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba9b40b:::benchmark_data_smithi188_113957_object140584:head# 2024-04-23T16:33:56.819 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.819 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba9d6f2:::benchmark_data_smithi188_113957_object38204:head# 2024-04-23T16:33:56.819 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.819 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba9de21:::benchmark_data_smithi188_113957_object63592:head# 2024-04-23T16:33:56.819 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.819 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ba9e303:::benchmark_data_smithi188_113957_object14159:head# 2024-04-23T16:33:56.819 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.819 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5baa4087:::benchmark_data_smithi188_113957_object101686:head# 2024-04-23T16:33:56.819 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.819 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5baa7a95:::benchmark_data_smithi188_113957_object82479:head# 2024-04-23T16:33:56.819 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bab14b6:::benchmark_data_smithi188_113957_object15099:head# 2024-04-23T16:33:56.820 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:56.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:56.822 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-23T16:33:56.823 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:56.823 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:33:56.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:33:56.881 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:56 smithi148 ceph-mon[44520]: 5.2 scrub starts 2024-04-23T16:33:56.927 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:33:56.968 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:56.995 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:33:57.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:57.003 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:33:57.054 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bab623c:::benchmark_data_smithi188_113957_object129250:head# 2024-04-23T16:33:57.054 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.055 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bac1d46:::benchmark_data_smithi188_113957_object61313:head# 2024-04-23T16:33:57.055 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.055 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bac9398:::benchmark_data_smithi188_113957_object110568:head# 2024-04-23T16:33:57.055 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.055 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bacfbd0:::benchmark_data_smithi188_113957_object49019:head# 2024-04-23T16:33:57.055 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.055 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5badbf32:::benchmark_data_smithi188_113957_object44875:head# 2024-04-23T16:33:57.055 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.055 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bade4ae:::benchmark_data_smithi188_113957_object984:head# 2024-04-23T16:33:57.055 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.055 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bae4e1d:::benchmark_data_smithi188_113957_object78725:head# 2024-04-23T16:33:57.055 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.055 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bae6fac:::benchmark_data_smithi188_113957_object121055:head# 2024-04-23T16:33:57.055 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.056 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5baedc78:::benchmark_data_smithi188_113957_object131344:head# 2024-04-23T16:33:57.056 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.056 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bafb1a0:::benchmark_data_smithi188_113957_object51582:head# 2024-04-23T16:33:57.056 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.056 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5baff623:::benchmark_data_smithi188_113957_object5597:head# 2024-04-23T16:33:57.056 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.056 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb00ae9:::benchmark_data_smithi188_113957_object75400:head# 2024-04-23T16:33:57.056 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.056 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb29c8a:::benchmark_data_smithi188_113957_object137909:head# 2024-04-23T16:33:57.056 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.056 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb2fa9c:::benchmark_data_smithi188_113957_object33491:head# 2024-04-23T16:33:57.056 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.056 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb37d4f:::benchmark_data_smithi188_113957_object139346:head# 2024-04-23T16:33:57.057 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.057 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb394ae:::benchmark_data_smithi188_113957_object43291:head# 2024-04-23T16:33:57.057 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.057 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb3cb27:::benchmark_data_smithi188_113957_object56914:head# 2024-04-23T16:33:57.057 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.057 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb462bf:::benchmark_data_smithi188_113957_object32767:head# 2024-04-23T16:33:57.057 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.057 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb4e2ee:::benchmark_data_smithi188_113957_object115782:head# 2024-04-23T16:33:57.057 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.057 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb4f039:::benchmark_data_smithi188_113957_object87116:head# 2024-04-23T16:33:57.057 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.057 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb4fa65:::benchmark_data_smithi188_113957_object58921:head# 2024-04-23T16:33:57.057 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.057 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb5566e:::benchmark_data_smithi188_113957_object34354:head# 2024-04-23T16:33:57.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb5b191:::benchmark_data_smithi188_113957_object43082:head# 2024-04-23T16:33:57.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb60271:::benchmark_data_smithi188_113957_object20955:head# 2024-04-23T16:33:57.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb61f6a:::benchmark_data_smithi188_113957_object129075:head# 2024-04-23T16:33:57.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb6340a:::benchmark_data_smithi188_113957_object35204:head# 2024-04-23T16:33:57.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb661c6:::benchmark_data_smithi188_113957_object107353:head# 2024-04-23T16:33:57.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb6e699:::benchmark_data_smithi188_113957_object142920:head# 2024-04-23T16:33:57.058 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.058 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb755d4:::benchmark_data_smithi188_113957_object73360:head# 2024-04-23T16:33:57.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb8234c:::benchmark_data_smithi188_113957_object100302:head# 2024-04-23T16:33:57.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb85fe5:::benchmark_data_smithi188_113957_object93032:head# 2024-04-23T16:33:57.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb8eb05:::benchmark_data_smithi188_113957_object105911:head# 2024-04-23T16:33:57.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb93448:::benchmark_data_smithi188_113957_object117759:head# 2024-04-23T16:33:57.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb9b9db:::benchmark_data_smithi188_113957_object24760:head# 2024-04-23T16:33:57.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb9c580:::benchmark_data_smithi188_113957_object7043:head# 2024-04-23T16:33:57.059 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.059 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bb9dd2c:::benchmark_data_smithi188_113957_object90853:head# 2024-04-23T16:33:57.060 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.060 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bbb236a:::benchmark_data_smithi188_113957_object60308:head# 2024-04-23T16:33:57.060 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.060 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bbb8be3:::benchmark_data_smithi188_113957_object96410:head# 2024-04-23T16:33:57.060 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.060 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bbb9bd1:::benchmark_data_smithi188_113957_object47275:head# 2024-04-23T16:33:57.060 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.060 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bbbe0fb:::benchmark_data_smithi188_113957_object120698:head# 2024-04-23T16:33:57.060 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.060 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bbbebfa:::benchmark_data_smithi188_113957_object70974:head# 2024-04-23T16:33:57.060 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.060 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bbbf18e:::benchmark_data_smithi188_113957_object64626:head# 2024-04-23T16:33:57.060 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.061 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bbc69a9:::benchmark_data_smithi188_113957_object82095:head# 2024-04-23T16:33:57.061 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.061 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bbcff65:::benchmark_data_smithi188_113957_object20876:head# 2024-04-23T16:33:57.061 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.061 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bbd7997:::benchmark_data_smithi188_113957_object41964:head# 2024-04-23T16:33:57.061 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.061 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bbe19db:::benchmark_data_smithi188_113957_object84669:head# 2024-04-23T16:33:57.061 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.061 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bbe6c35:::benchmark_data_smithi188_113957_object86696:head# 2024-04-23T16:33:57.061 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.061 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bbe7f46:::benchmark_data_smithi188_113957_object96880:head# 2024-04-23T16:33:57.061 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.061 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bbee640:::benchmark_data_smithi188_113957_object136959:head# 2024-04-23T16:33:57.061 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.062 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bbf2bd2:::benchmark_data_smithi188_113957_object133287:head# 2024-04-23T16:33:57.062 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.062 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bbf602a:::benchmark_data_smithi188_113957_object85584:head# 2024-04-23T16:33:57.062 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.062 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bbf7087:::benchmark_data_smithi188_113957_object49458:head# 2024-04-23T16:33:57.062 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.062 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bbf79f3:::benchmark_data_smithi188_113957_object103171:head# 2024-04-23T16:33:57.062 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.062 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bbfd879:::benchmark_data_smithi188_113957_object145201:head# 2024-04-23T16:33:57.062 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.062 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc172c0:::benchmark_data_smithi188_113957_object66431:head# 2024-04-23T16:33:57.062 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.062 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc1ec4a:::benchmark_data_smithi188_113957_object62140:head# 2024-04-23T16:33:57.063 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.063 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc1f28a:::benchmark_data_smithi188_113957_object103004:head# 2024-04-23T16:33:57.063 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.063 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc2180f:::benchmark_data_smithi188_113957_object140686:head# 2024-04-23T16:33:57.063 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.063 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc2200e:::benchmark_data_smithi188_113957_object117598:head# 2024-04-23T16:33:57.063 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.063 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc26412:::benchmark_data_smithi188_113957_object57806:head# 2024-04-23T16:33:57.063 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.063 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc26570:::benchmark_data_smithi188_113957_object148022:head# 2024-04-23T16:33:57.063 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.063 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc29c2a:::benchmark_data_smithi188_113957_object42211:head# 2024-04-23T16:33:57.063 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.063 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc2e9f9:::benchmark_data_smithi188_113957_object9020:head# 2024-04-23T16:33:57.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc2ea73:::benchmark_data_smithi188_113957_object59584:head# 2024-04-23T16:33:57.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc33d5c:::benchmark_data_smithi188_113957_object48431:head# 2024-04-23T16:33:57.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc37ec0:::benchmark_data_smithi188_113957_object42882:head# 2024-04-23T16:33:57.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc3c112:::benchmark_data_smithi188_113957_object93850:head# 2024-04-23T16:33:57.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc3e842:::benchmark_data_smithi188_113957_object64573:head# 2024-04-23T16:33:57.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc3fa0a:::benchmark_data_smithi188_113957_object101224:head# 2024-04-23T16:33:57.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.064 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc42e61:::benchmark_data_smithi188_113957_object82422:head# 2024-04-23T16:33:57.064 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc64a79:::benchmark_data_smithi188_113957_object65501:head# 2024-04-23T16:33:57.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc6df9c:::benchmark_data_smithi188_113957_object152705:head# 2024-04-23T16:33:57.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc6fb47:::benchmark_data_smithi188_113957_object63906:head# 2024-04-23T16:33:57.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc779b6:::benchmark_data_smithi188_113957_object76048:head# 2024-04-23T16:33:57.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc7b638:::benchmark_data_smithi188_113957_object131553:head# 2024-04-23T16:33:57.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc847d4:::benchmark_data_smithi188_113957_object22555:head# 2024-04-23T16:33:57.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.065 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc886d5:::benchmark_data_smithi188_113957_object116785:head# 2024-04-23T16:33:57.065 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc8c1bb:::benchmark_data_smithi188_113957_object14374:head# 2024-04-23T16:33:57.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc93623:::benchmark_data_smithi188_113957_object100608:head# 2024-04-23T16:33:57.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc9eb6a:::benchmark_data_smithi188_113957_object18210:head# 2024-04-23T16:33:57.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bc9ec8c:::benchmark_data_smithi188_113957_object124475:head# 2024-04-23T16:33:57.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bca44ba:::benchmark_data_smithi188_113957_object32483:head# 2024-04-23T16:33:57.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bcb095c:::benchmark_data_smithi188_113957_object30604:head# 2024-04-23T16:33:57.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.066 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bcb2258:::benchmark_data_smithi188_113957_object4030:head# 2024-04-23T16:33:57.066 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.067 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bcb63ca:::benchmark_data_smithi188_113957_object28992:head# 2024-04-23T16:33:57.067 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.067 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bcb8602:::benchmark_data_smithi188_113957_object51929:head# 2024-04-23T16:33:57.067 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.067 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bcba0cd:::benchmark_data_smithi188_113957_object15842:head# 2024-04-23T16:33:57.067 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.067 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bcba6ca:::benchmark_data_smithi188_113957_object129179:head# 2024-04-23T16:33:57.067 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.067 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bcbb633:::benchmark_data_smithi188_113957_object131042:head# 2024-04-23T16:33:57.067 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.067 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bcc5309:::benchmark_data_smithi188_113957_object139984:head# 2024-04-23T16:33:57.067 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.067 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bcce2a4:::benchmark_data_smithi188_113957_object43401:head# 2024-04-23T16:33:57.067 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.068 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bcd2dd5:::benchmark_data_smithi188_113957_object16706:head# 2024-04-23T16:33:57.068 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.068 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bcd8509:::benchmark_data_smithi188_113957_object102636:head# 2024-04-23T16:33:57.068 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.068 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bcd999e:::benchmark_data_smithi188_113957_object143459:head# 2024-04-23T16:33:57.068 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.068 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bce1d1c:::benchmark_data_smithi188_113957_object13657:head# 2024-04-23T16:33:57.068 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.068 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bcf0cbc:::benchmark_data_smithi188_113957_object90380:head# 2024-04-23T16:33:57.068 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.068 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bcf473c:::benchmark_data_smithi188_113957_object146823:head# 2024-04-23T16:33:57.068 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.068 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bcfa70e:::benchmark_data_smithi188_113957_object91013:head# 2024-04-23T16:33:57.069 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.069 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd018a2:::benchmark_data_smithi188_113957_object6995:head# 2024-04-23T16:33:57.069 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.069 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd02c54:::benchmark_data_smithi188_113957_object12695:head# 2024-04-23T16:33:57.069 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.069 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd08377:::benchmark_data_smithi188_113957_object69046:head# 2024-04-23T16:33:57.069 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.069 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd0908f:::benchmark_data_smithi188_113957_object133610:head# 2024-04-23T16:33:57.069 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.069 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd0e266:::benchmark_data_smithi188_113957_object114843:head# 2024-04-23T16:33:57.069 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.069 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd13b7a:::benchmark_data_smithi188_113957_object80063:head# 2024-04-23T16:33:57.069 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.069 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd16d7b:::benchmark_data_smithi188_113957_object23335:head# 2024-04-23T16:33:57.070 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.070 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd19c0d:::benchmark_data_smithi188_113957_object82496:head# 2024-04-23T16:33:57.070 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.070 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd2afbc:::benchmark_data_smithi188_113957_object78175:head# 2024-04-23T16:33:57.070 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.070 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd341c6:::benchmark_data_smithi188_113957_object145155:head# 2024-04-23T16:33:57.070 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.070 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd37f54:::benchmark_data_smithi188_113957_object65067:head# 2024-04-23T16:33:57.070 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.070 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd38348:::benchmark_data_smithi188_113957_object89895:head# 2024-04-23T16:33:57.070 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.070 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd3b09b:::benchmark_data_smithi188_113957_object113592:head# 2024-04-23T16:33:57.070 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.070 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd3e9e6:::benchmark_data_smithi188_113957_object115998:head# 2024-04-23T16:33:57.071 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.071 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd48dfd:::benchmark_data_smithi188_113957_object32080:head# 2024-04-23T16:33:57.071 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.071 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd4f159:::benchmark_data_smithi188_113957_object89713:head# 2024-04-23T16:33:57.071 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.071 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd4f48a:::benchmark_data_smithi188_113957_object117855:head# 2024-04-23T16:33:57.071 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.071 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd4f5ed:::benchmark_data_smithi188_113957_object23583:head# 2024-04-23T16:33:57.071 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.071 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd5a7b7:::benchmark_data_smithi188_113957_object30429:head# 2024-04-23T16:33:57.071 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.071 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd8caa3:::benchmark_data_smithi188_113957_object48298:head# 2024-04-23T16:33:57.071 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.071 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd8cfd3:::benchmark_data_smithi188_113957_object103615:head# 2024-04-23T16:33:57.072 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.072 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd8ffce:::benchmark_data_smithi188_113957_object50442:head# 2024-04-23T16:33:57.072 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.072 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd9550d:::benchmark_data_smithi188_113957_object8165:head# 2024-04-23T16:33:57.072 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.072 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bd9fcb1:::benchmark_data_smithi188_113957_object34668:head# 2024-04-23T16:33:57.072 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.072 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bda02b6:::benchmark_data_smithi188_113957_object116059:head# 2024-04-23T16:33:57.072 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.072 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bda4358:::benchmark_data_smithi188_113957_object24868:head# 2024-04-23T16:33:57.072 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.072 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bda719b:::benchmark_data_smithi188_113957_object115808:head# 2024-04-23T16:33:57.072 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.072 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bdb64dc:::benchmark_data_smithi188_113957_object125626:head# 2024-04-23T16:33:57.073 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.073 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bdc85a9:::benchmark_data_smithi188_113957_object1399:head# 2024-04-23T16:33:57.073 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.073 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bdd8a32:::benchmark_data_smithi188_113957_object67726:head# 2024-04-23T16:33:57.073 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.073 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bddb5f0:::benchmark_data_smithi188_113957_object39631:head# 2024-04-23T16:33:57.073 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.073 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bded3e9:::benchmark_data_smithi188_113957_object100389:head# 2024-04-23T16:33:57.073 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.073 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bdf999e:::benchmark_data_smithi188_113957_object11904:head# 2024-04-23T16:33:57.073 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.073 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be0a2cb:::benchmark_data_smithi188_113957_object126444:head# 2024-04-23T16:33:57.073 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.073 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be146e4:::benchmark_data_smithi188_113957_object1051:head# 2024-04-23T16:33:57.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be1a96a:::benchmark_data_smithi188_113957_object80968:head# 2024-04-23T16:33:57.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be1eb4f:::benchmark_data_smithi188_113957_object108410:head# 2024-04-23T16:33:57.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be23414:::benchmark_data_smithi188_113957_object131009:head# 2024-04-23T16:33:57.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be26c31:::benchmark_data_smithi188_113957_object83659:head# 2024-04-23T16:33:57.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be26e68:::benchmark_data_smithi188_113957_object147328:head# 2024-04-23T16:33:57.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be2a791:::benchmark_data_smithi188_113957_object129516:head# 2024-04-23T16:33:57.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be2aade:::benchmark_data_smithi188_113957_object26185:head# 2024-04-23T16:33:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be2e440:::benchmark_data_smithi188_113957_object82093:head# 2024-04-23T16:33:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be2ee10:::benchmark_data_smithi188_113957_object103402:head# 2024-04-23T16:33:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be32112:::benchmark_data_smithi188_113957_object93258:head# 2024-04-23T16:33:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be38aac:::benchmark_data_smithi188_113957_object50361:head# 2024-04-23T16:33:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be3e656:::benchmark_data_smithi188_113957_object137666:head# 2024-04-23T16:33:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be429df:::benchmark_data_smithi188_113957_object97492:head# 2024-04-23T16:33:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be462df:::benchmark_data_smithi188_113957_object131451:head# 2024-04-23T16:33:57.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be51fab:::benchmark_data_smithi188_113957_object127354:head# 2024-04-23T16:33:57.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be5d7f5:::benchmark_data_smithi188_113957_object48473:head# 2024-04-23T16:33:57.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be6bf85:::benchmark_data_smithi188_113957_object110903:head# 2024-04-23T16:33:57.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be6e295:::benchmark_data_smithi188_113957_object39996:head# 2024-04-23T16:33:57.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be6e44a:::benchmark_data_smithi188_113957_object106355:head# 2024-04-23T16:33:57.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be6f0be:::benchmark_data_smithi188_113957_object134040:head# 2024-04-23T16:33:57.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be74b9b:::benchmark_data_smithi188_113957_object111702:head# 2024-04-23T16:33:57.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be75520:::benchmark_data_smithi188_113957_object35812:head# 2024-04-23T16:33:57.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be86fad:::benchmark_data_smithi188_113957_object119760:head# 2024-04-23T16:33:57.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be8949e:::benchmark_data_smithi188_113957_object79129:head# 2024-04-23T16:33:57.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be8aed1:::benchmark_data_smithi188_113957_object128619:head# 2024-04-23T16:33:57.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be8e41f:::benchmark_data_smithi188_113957_object59937:head# 2024-04-23T16:33:57.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be8fe3e:::benchmark_data_smithi188_113957_object76860:head# 2024-04-23T16:33:57.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be92899:::benchmark_data_smithi188_113957_object51992:head# 2024-04-23T16:33:57.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.078 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be94d71:::benchmark_data_smithi188_113957_object106557:head# 2024-04-23T16:33:57.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.078 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be96f45:::benchmark_data_smithi188_113957_object132619:head# 2024-04-23T16:33:57.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.078 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5be9e74d:::benchmark_data_smithi188_113957_object92456:head# 2024-04-23T16:33:57.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.078 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bea06f7:::benchmark_data_smithi188_113957_object142927:head# 2024-04-23T16:33:57.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.078 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bea0d83:::benchmark_data_smithi188_113957_object83247:head# 2024-04-23T16:33:57.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.078 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bea81f4:::benchmark_data_smithi188_113957_object68083:head# 2024-04-23T16:33:57.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.078 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5beae3e5:::benchmark_data_smithi188_113957_object83280:head# 2024-04-23T16:33:57.079 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.079 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5beb4791:::benchmark_data_smithi188_113957_object57180:head# 2024-04-23T16:33:57.079 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.079 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5beb5a87:::benchmark_data_smithi188_113957_object118071:head# 2024-04-23T16:33:57.079 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.079 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5beb674f:::benchmark_data_smithi188_113957_object18376:head# 2024-04-23T16:33:57.079 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.079 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bebcd3a:::benchmark_data_smithi188_113957_object14902:head# 2024-04-23T16:33:57.079 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.079 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bec5c40:::benchmark_data_smithi188_113957_object98690:head# 2024-04-23T16:33:57.079 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.079 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bec9722:::benchmark_data_smithi188_113957_object4473:head# 2024-04-23T16:33:57.079 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.079 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bed070a:::benchmark_data_smithi188_113957_object58381:head# 2024-04-23T16:33:57.080 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.080 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bed07a1:::benchmark_data_smithi188_113957_object7529:head# 2024-04-23T16:33:57.080 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.080 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bed83fc:::benchmark_data_smithi188_113957_object82393:head# 2024-04-23T16:33:57.080 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.080 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bedb054:::benchmark_data_smithi188_113957_object39212:head# 2024-04-23T16:33:57.080 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.080 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bedccba:::benchmark_data_smithi188_113957_object105021:head# 2024-04-23T16:33:57.080 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.080 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bee1475:::benchmark_data_smithi188_113957_object33654:head# 2024-04-23T16:33:57.080 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.080 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bee5774:::benchmark_data_smithi188_113957_object9370:head# 2024-04-23T16:33:57.080 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.080 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5befeb67:::benchmark_data_smithi188_113957_object120859:head# 2024-04-23T16:33:57.081 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.081 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf00861:::benchmark_data_smithi188_113957_object78678:head# 2024-04-23T16:33:57.081 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.081 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf00c45:::benchmark_data_smithi188_113957_object139534:head# 2024-04-23T16:33:57.081 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.081 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf04436:::benchmark_data_smithi188_113957_object22815:head# 2024-04-23T16:33:57.081 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.081 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf04923:::benchmark_data_smithi188_113957_object120667:head# 2024-04-23T16:33:57.081 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.081 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf11811:::benchmark_data_smithi188_113957_object142105:head# 2024-04-23T16:33:57.081 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.081 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf17143:::benchmark_data_smithi188_113957_object30243:head# 2024-04-23T16:33:57.081 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.081 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf18a47:::benchmark_data_smithi188_113957_object135746:head# 2024-04-23T16:33:57.082 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.082 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf1dc0f:::benchmark_data_smithi188_113957_object99051:head# 2024-04-23T16:33:57.082 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.082 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf265f8:::benchmark_data_smithi188_113957_object123674:head# 2024-04-23T16:33:57.082 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.082 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf2d5af:::benchmark_data_smithi188_113957_object71523:head# 2024-04-23T16:33:57.082 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.082 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf3496b:::benchmark_data_smithi188_113957_object34628:head# 2024-04-23T16:33:57.082 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.082 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf3c8a5:::benchmark_data_smithi188_113957_object19892:head# 2024-04-23T16:33:57.082 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.082 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf498af:::benchmark_data_smithi188_113957_object10552:head# 2024-04-23T16:33:57.082 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.083 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf4b769:::benchmark_data_smithi188_113957_object15879:head# 2024-04-23T16:33:57.083 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.083 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf65eb4:::benchmark_data_smithi188_113957_object8472:head# 2024-04-23T16:33:57.083 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.083 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf7324e:::benchmark_data_smithi188_113957_object37154:head# 2024-04-23T16:33:57.083 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.083 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf75526:::benchmark_data_smithi188_113957_object138609:head# 2024-04-23T16:33:57.083 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.083 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf79318:::benchmark_data_smithi188_113957_object433:head# 2024-04-23T16:33:57.083 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.083 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf7a0b8:::benchmark_data_smithi188_113957_object26684:head# 2024-04-23T16:33:57.083 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.083 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf88451:::benchmark_data_smithi188_113957_object82873:head# 2024-04-23T16:33:57.083 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.083 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf8de0d:::benchmark_data_smithi188_113957_object129466:head# 2024-04-23T16:33:57.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf934b0:::benchmark_data_smithi188_113957_object62236:head# 2024-04-23T16:33:57.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf93f92:::benchmark_data_smithi188_113957_object30899:head# 2024-04-23T16:33:57.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf93fc4:::benchmark_data_smithi188_113957_object123706:head# 2024-04-23T16:33:57.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf973b6:::benchmark_data_smithi188_113957_object19105:head# 2024-04-23T16:33:57.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bf9cef4:::benchmark_data_smithi188_113957_object53442:head# 2024-04-23T16:33:57.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bfaab9b:::benchmark_data_smithi188_113957_object91597:head# 2024-04-23T16:33:57.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bfb7663:::benchmark_data_smithi188_113957_object64041:head# 2024-04-23T16:33:57.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bfbd362:::benchmark_data_smithi188_113957_object144452:head# 2024-04-23T16:33:57.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bfcb1c0:::benchmark_data_smithi188_113957_object16244:head# 2024-04-23T16:33:57.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bfcd603:::benchmark_data_smithi188_113957_object9833:head# 2024-04-23T16:33:57.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bfdbe58:::benchmark_data_smithi188_113957_object44788:head# 2024-04-23T16:33:57.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bfe044e:::benchmark_data_smithi188_113957_object91398:head# 2024-04-23T16:33:57.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bfec2c5:::benchmark_data_smithi188_113957_object90247:head# 2024-04-23T16:33:57.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5bffbca1:::benchmark_data_smithi188_113957_object44620:head# 2024-04-23T16:33:57.086 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.086 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c00116e:::benchmark_data_smithi188_113957_object53887:head# 2024-04-23T16:33:57.086 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.086 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c006fde:::benchmark_data_smithi188_113957_object67433:head# 2024-04-23T16:33:57.086 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.086 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c007581:::benchmark_data_smithi188_113957_object108386:head# 2024-04-23T16:33:57.086 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.086 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c00ed0f:::benchmark_data_smithi188_113957_object82341:head# 2024-04-23T16:33:57.086 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.086 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0136e2:::benchmark_data_smithi188_113957_object59860:head# 2024-04-23T16:33:57.086 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.086 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c013f4d:::benchmark_data_smithi188_113957_object60574:head# 2024-04-23T16:33:57.086 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.086 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c01f67c:::benchmark_data_smithi188_113957_object68644:head# 2024-04-23T16:33:57.087 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.087 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c02b8d3:::benchmark_data_smithi188_113957_object44325:head# 2024-04-23T16:33:57.087 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.087 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0385f5:::benchmark_data_smithi188_113957_object108741:head# 2024-04-23T16:33:57.087 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.087 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c03a641:::benchmark_data_smithi188_113957_object107188:head# 2024-04-23T16:33:57.087 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.087 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c03a916:::benchmark_data_smithi188_113957_object78125:head# 2024-04-23T16:33:57.087 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.087 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c042d0d:::benchmark_data_smithi188_113957_object79929:head# 2024-04-23T16:33:57.087 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.087 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c04dd12:::benchmark_data_smithi188_113957_object63122:head# 2024-04-23T16:33:57.087 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.087 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c05bb34:::benchmark_data_smithi188_113957_object149610:head# 2024-04-23T16:33:57.088 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.088 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c076330:::benchmark_data_smithi188_113957_object73530:head# 2024-04-23T16:33:57.088 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.088 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c07745e:::benchmark_data_smithi188_113957_object74228:head# 2024-04-23T16:33:57.088 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.088 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c083496:::benchmark_data_smithi188_113957_object132309:head# 2024-04-23T16:33:57.088 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.088 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c087af0:::benchmark_data_smithi188_113957_object80139:head# 2024-04-23T16:33:57.088 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.088 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c08abff:::benchmark_data_smithi188_113957_object128728:head# 2024-04-23T16:33:57.088 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.088 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c08af28:::benchmark_data_smithi188_113957_object95684:head# 2024-04-23T16:33:57.088 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.088 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c08e15b:::benchmark_data_smithi188_113957_object104561:head# 2024-04-23T16:33:57.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.089 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c08fb00:::benchmark_data_smithi188_113957_object29562:head# 2024-04-23T16:33:57.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.089 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c092262:::benchmark_data_smithi188_113957_object102591:head# 2024-04-23T16:33:57.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.089 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0955a7:::benchmark_data_smithi188_113957_object85830:head# 2024-04-23T16:33:57.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.089 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c09a670:::benchmark_data_smithi188_113957_object142017:head# 2024-04-23T16:33:57.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.089 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0a5ac1:::benchmark_data_smithi188_113957_object58170:head# 2024-04-23T16:33:57.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.089 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0a73d0:::benchmark_data_smithi188_113957_object3617:head# 2024-04-23T16:33:57.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.089 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0aa52b:::benchmark_data_smithi188_113957_object31859:head# 2024-04-23T16:33:57.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0b904f:::benchmark_data_smithi188_113957_object142101:head# 2024-04-23T16:33:57.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0b9180:::benchmark_data_smithi188_113957_object27498:head# 2024-04-23T16:33:57.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0be4af:::benchmark_data_smithi188_113957_object51134:head# 2024-04-23T16:33:57.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0be6eb:::benchmark_data_smithi188_113957_object86754:head# 2024-04-23T16:33:57.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0c697a:::benchmark_data_smithi188_113957_object16463:head# 2024-04-23T16:33:57.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0c8fe0:::benchmark_data_smithi188_113957_object148616:head# 2024-04-23T16:33:57.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0d2a6d:::benchmark_data_smithi188_113957_object111223:head# 2024-04-23T16:33:57.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0d4153:::benchmark_data_smithi188_113957_object4562:head# 2024-04-23T16:33:57.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0d7993:::benchmark_data_smithi188_113957_object5868:head# 2024-04-23T16:33:57.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0d90cf:::benchmark_data_smithi188_113957_object20005:head# 2024-04-23T16:33:57.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0dabf6:::benchmark_data_smithi188_113957_object8637:head# 2024-04-23T16:33:57.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0dc38b:::benchmark_data_smithi188_113957_object8952:head# 2024-04-23T16:33:57.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0ded5e:::benchmark_data_smithi188_113957_object48120:head# 2024-04-23T16:33:57.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0e0642:::benchmark_data_smithi188_113957_object49985:head# 2024-04-23T16:33:57.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0e400d:::benchmark_data_smithi188_113957_object128155:head# 2024-04-23T16:33:57.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0e9c93:::benchmark_data_smithi188_113957_object18747:head# 2024-04-23T16:33:57.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0ea29d:::benchmark_data_smithi188_113957_object72679:head# 2024-04-23T16:33:57.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0eeae8:::benchmark_data_smithi188_113957_object125983:head# 2024-04-23T16:33:57.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0f7442:::benchmark_data_smithi188_113957_object32960:head# 2024-04-23T16:33:57.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c0f7fd5:::benchmark_data_smithi188_113957_object122576:head# 2024-04-23T16:33:57.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c100956:::benchmark_data_smithi188_113957_object72371:head# 2024-04-23T16:33:57.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c115e00:::benchmark_data_smithi188_113957_object31979:head# 2024-04-23T16:33:57.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c115fcc:::benchmark_data_smithi188_113957_object75238:head# 2024-04-23T16:33:57.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c1195d1:::benchmark_data_smithi188_113957_object20555:head# 2024-04-23T16:33:57.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c11d879:::benchmark_data_smithi188_113957_object132284:head# 2024-04-23T16:33:57.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c11e501:::benchmark_data_smithi188_113957_object127405:head# 2024-04-23T16:33:57.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c125a16:::benchmark_data_smithi188_113957_object17407:head# 2024-04-23T16:33:57.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c125c4f:::benchmark_data_smithi188_113957_object87090:head# 2024-04-23T16:33:57.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c128901:::benchmark_data_smithi188_113957_object6031:head# 2024-04-23T16:33:57.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c13b6c1:::benchmark_data_smithi188_113957_object27097:head# 2024-04-23T16:33:57.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c14bcf8:::benchmark_data_smithi188_113957_object61670:head# 2024-04-23T16:33:57.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c14ef15:::benchmark_data_smithi188_113957_object77211:head# 2024-04-23T16:33:57.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c159b66:::benchmark_data_smithi188_113957_object5471:head# 2024-04-23T16:33:57.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c15f622:::benchmark_data_smithi188_113957_object23350:head# 2024-04-23T16:33:57.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c15f6f1:::benchmark_data_smithi188_113957_object24746:head# 2024-04-23T16:33:57.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c165818:::benchmark_data_smithi188_113957_object58610:head# 2024-04-23T16:33:57.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c16c30e:::benchmark_data_smithi188_113957_object16202:head# 2024-04-23T16:33:57.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c16ed8c:::benchmark_data_smithi188_113957_object8519:head# 2024-04-23T16:33:57.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c177973:::benchmark_data_smithi188_113957_object144975:head# 2024-04-23T16:33:57.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c17928d:::benchmark_data_smithi188_113957_object65044:head# 2024-04-23T16:33:57.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c1834c3:::benchmark_data_smithi188_113957_object27777:head# 2024-04-23T16:33:57.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c189652:::benchmark_data_smithi188_113957_object126562:head# 2024-04-23T16:33:57.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c18ca75:::benchmark_data_smithi188_113957_object150806:head# 2024-04-23T16:33:57.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c18e3d7:::benchmark_data_smithi188_113957_object19594:head# 2024-04-23T16:33:57.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c190551:::benchmark_data_smithi188_113957_object135452:head# 2024-04-23T16:33:57.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c1912ff:::benchmark_data_smithi188_113957_object134777:head# 2024-04-23T16:33:57.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c196fec:::benchmark_data_smithi188_113957_object22837:head# 2024-04-23T16:33:57.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c19c289:::benchmark_data_smithi188_113957_object99877:head# 2024-04-23T16:33:57.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c19ebfa:::benchmark_data_smithi188_113957_object123458:head# 2024-04-23T16:33:57.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c1a52e7:::benchmark_data_smithi188_113957_object111744:head# 2024-04-23T16:33:57.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c1b1728:::benchmark_data_smithi188_113957_object54451:head# 2024-04-23T16:33:57.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c1b6193:::benchmark_data_smithi188_113957_object52894:head# 2024-04-23T16:33:57.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c1bf026:::benchmark_data_smithi188_113957_object151294:head# 2024-04-23T16:33:57.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c1c5420:::benchmark_data_smithi188_113957_object73397:head# 2024-04-23T16:33:57.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c1d05c9:::benchmark_data_smithi188_113957_object1871:head# 2024-04-23T16:33:57.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c1de0cf:::benchmark_data_smithi188_113957_object82494:head# 2024-04-23T16:33:57.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c1e62aa:::benchmark_data_smithi188_113957_object106236:head# 2024-04-23T16:33:57.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c1ea435:::benchmark_data_smithi188_113957_object16890:head# 2024-04-23T16:33:57.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c1ee329:::benchmark_data_smithi188_113957_object9931:head# 2024-04-23T16:33:57.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c1f4d43:::benchmark_data_smithi188_113957_object75740:head# 2024-04-23T16:33:57.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c206dd6:::benchmark_data_smithi188_113957_object13815:head# 2024-04-23T16:33:57.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2077c8:::benchmark_data_smithi188_113957_object11516:head# 2024-04-23T16:33:57.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c208c91:::benchmark_data_smithi188_113957_object127533:head# 2024-04-23T16:33:57.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2095ae:::benchmark_data_smithi188_113957_object110454:head# 2024-04-23T16:33:57.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c20e5e9:::benchmark_data_smithi188_113957_object87488:head# 2024-04-23T16:33:57.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c20fd85:::benchmark_data_smithi188_113957_object127050:head# 2024-04-23T16:33:57.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c210866:::benchmark_data_smithi188_113957_object64119:head# 2024-04-23T16:33:57.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c226fec:::benchmark_data_smithi188_113957_object99063:head# 2024-04-23T16:33:57.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2280ae:::benchmark_data_smithi188_113957_object50832:head# 2024-04-23T16:33:57.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c22b34d:::benchmark_data_smithi188_113957_object3628:head# 2024-04-23T16:33:57.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2335ee:::benchmark_data_smithi188_113957_object35612:head# 2024-04-23T16:33:57.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c233a58:::benchmark_data_smithi188_113957_object69620:head# 2024-04-23T16:33:57.102 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c239fe6:::benchmark_data_smithi188_113957_object149857:head# 2024-04-23T16:33:57.102 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2408d8:::benchmark_data_smithi188_113957_object104123:head# 2024-04-23T16:33:57.102 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c242bac:::benchmark_data_smithi188_113957_object94172:head# 2024-04-23T16:33:57.103 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.103 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c244907:::benchmark_data_smithi188_113957_object93819:head# 2024-04-23T16:33:57.103 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.103 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c247cf2:::benchmark_data_smithi188_113957_object99828:head# 2024-04-23T16:33:57.103 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.103 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c24916a:::benchmark_data_smithi188_113957_object21320:head# 2024-04-23T16:33:57.103 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.104 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c24aac7:::benchmark_data_smithi188_113957_object49416:head# 2024-04-23T16:33:57.104 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.104 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c24e227:::benchmark_data_smithi188_113957_object68591:head# 2024-04-23T16:33:57.104 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.104 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c253906:::benchmark_data_smithi188_113957_object115992:head# 2024-04-23T16:33:57.104 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.104 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c257659:::benchmark_data_smithi188_113957_object109175:head# 2024-04-23T16:33:57.105 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.105 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c25a6f2:::benchmark_data_smithi188_113957_object135587:head# 2024-04-23T16:33:57.105 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.105 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c262b3c:::benchmark_data_smithi188_113957_object54563:head# 2024-04-23T16:33:57.105 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.105 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c26c2db:::benchmark_data_smithi188_113957_object119544:head# 2024-04-23T16:33:57.105 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.105 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c283432:::benchmark_data_smithi188_113957_object41529:head# 2024-04-23T16:33:57.106 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.106 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c28925d:::benchmark_data_smithi188_113957_object102474:head# 2024-04-23T16:33:57.106 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.106 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c28a041:::benchmark_data_smithi188_113957_object11679:head# 2024-04-23T16:33:57.106 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.106 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c28aa99:::benchmark_data_smithi188_113957_object63257:head# 2024-04-23T16:33:57.106 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.106 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c28f402:::benchmark_data_smithi188_113957_object16908:head# 2024-04-23T16:33:57.107 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.107 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c290fbc:::benchmark_data_smithi188_113957_object10059:head# 2024-04-23T16:33:57.107 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.107 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c291e9c:::benchmark_data_smithi188_113957_object133455:head# 2024-04-23T16:33:57.107 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.107 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c298248:::benchmark_data_smithi188_113957_object79668:head# 2024-04-23T16:33:57.107 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.107 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c29a650:::benchmark_data_smithi188_113957_object89143:head# 2024-04-23T16:33:57.107 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2ac69a:::benchmark_data_smithi188_113957_object111550:head# 2024-04-23T16:33:57.108 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2add12:::benchmark_data_smithi188_113957_object153368:head# 2024-04-23T16:33:57.108 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2ae29c:::benchmark_data_smithi188_113957_object14291:head# 2024-04-23T16:33:57.108 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2bb1ad:::benchmark_data_smithi188_113957_object28177:head# 2024-04-23T16:33:57.108 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2c25be:::benchmark_data_smithi188_113957_object9406:head# 2024-04-23T16:33:57.108 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2c4070:::benchmark_data_smithi188_113957_object25635:head# 2024-04-23T16:33:57.108 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2cbf34:::benchmark_data_smithi188_113957_object61231:head# 2024-04-23T16:33:57.108 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2d03d8:::benchmark_data_smithi188_113957_object54452:head# 2024-04-23T16:33:57.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2d5c58:::benchmark_data_smithi188_113957_object106881:head# 2024-04-23T16:33:57.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2ddbd5:::benchmark_data_smithi188_113957_object40337:head# 2024-04-23T16:33:57.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2e3721:::benchmark_data_smithi188_113957_object70002:head# 2024-04-23T16:33:57.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2ec5c7:::benchmark_data_smithi188_113957_object66674:head# 2024-04-23T16:33:57.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2eda6e:::benchmark_data_smithi188_113957_object26553:head# 2024-04-23T16:33:57.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2f5169:::benchmark_data_smithi188_113957_object56593:head# 2024-04-23T16:33:57.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2f8909:::benchmark_data_smithi188_113957_object56924:head# 2024-04-23T16:33:57.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2f9017:::benchmark_data_smithi188_113957_object67927:head# 2024-04-23T16:33:57.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2ff25b:::benchmark_data_smithi188_113957_object6466:head# 2024-04-23T16:33:57.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c2ff482:::benchmark_data_smithi188_113957_object142231:head# 2024-04-23T16:33:57.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c3036bb:::benchmark_data_smithi188_113957_object23930:head# 2024-04-23T16:33:57.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c30939a:::benchmark_data_smithi188_113957_object63660:head# 2024-04-23T16:33:57.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c311879:::benchmark_data_smithi188_113957_object65957:head# 2024-04-23T16:33:57.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c3151f3:::benchmark_data_smithi188_113957_object93805:head# 2024-04-23T16:33:57.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c319278:::benchmark_data_smithi188_113957_object57823:head# 2024-04-23T16:33:57.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c31fd07:::benchmark_data_smithi188_113957_object34158:head# 2024-04-23T16:33:57.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c32c5ba:::benchmark_data_smithi188_113957_object126779:head# 2024-04-23T16:33:57.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c336d6a:::benchmark_data_smithi188_113957_object97365:head# 2024-04-23T16:33:57.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c342b0b:::benchmark_data_smithi188_113957_object45050:head# 2024-04-23T16:33:57.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c348692:::benchmark_data_smithi188_113957_object99154:head# 2024-04-23T16:33:57.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c34b9dc:::benchmark_data_smithi188_113957_object67284:head# 2024-04-23T16:33:57.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c3549ab:::benchmark_data_smithi188_113957_object81056:head# 2024-04-23T16:33:57.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c356ff1:::benchmark_data_smithi188_113957_object73337:head# 2024-04-23T16:33:57.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c35bbd4:::benchmark_data_smithi188_113957_object119663:head# 2024-04-23T16:33:57.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c37bbb7:::benchmark_data_smithi188_113957_object24410:head# 2024-04-23T16:33:57.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c398525:::benchmark_data_smithi188_113957_object56995:head# 2024-04-23T16:33:57.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c39f120:::benchmark_data_smithi188_113957_object132496:head# 2024-04-23T16:33:57.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c3a1fbf:::benchmark_data_smithi188_113957_object31736:head# 2024-04-23T16:33:57.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c3a9ce7:::benchmark_data_smithi188_113957_object56332:head# 2024-04-23T16:33:57.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c3b6503:::benchmark_data_smithi188_113957_object61396:head# 2024-04-23T16:33:57.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c3bd561:::benchmark_data_smithi188_113957_object30691:head# 2024-04-23T16:33:57.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c3be106:::benchmark_data_smithi188_113957_object132048:head# 2024-04-23T16:33:57.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c3c4ba2:::benchmark_data_smithi188_113957_object4716:head# 2024-04-23T16:33:57.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c3c8237:::benchmark_data_smithi188_113957_object70346:head# 2024-04-23T16:33:57.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c3c8759:::benchmark_data_smithi188_113957_object28967:head# 2024-04-23T16:33:57.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c3d9857:::benchmark_data_smithi188_113957_object62745:head# 2024-04-23T16:33:57.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c3db4c1:::benchmark_data_smithi188_113957_object123314:head# 2024-04-23T16:33:57.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c3dc1e1:::benchmark_data_smithi188_113957_object41545:head# 2024-04-23T16:33:57.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c3f458b:::benchmark_data_smithi188_113957_object29351:head# 2024-04-23T16:33:57.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c3f5493:::benchmark_data_smithi188_113957_object54662:head# 2024-04-23T16:33:57.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c3fb20e:::benchmark_data_smithi188_113957_object64964:head# 2024-04-23T16:33:57.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c3fd88c:::benchmark_data_smithi188_113957_object42388:head# 2024-04-23T16:33:57.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c40188e:::benchmark_data_smithi188_113957_object133340:head# 2024-04-23T16:33:57.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c405a23:::benchmark_data_smithi188_113957_object21550:head# 2024-04-23T16:33:57.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c411f2e:::benchmark_data_smithi188_113957_object139990:head# 2024-04-23T16:33:57.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c41336d:::benchmark_data_smithi188_113957_object43832:head# 2024-04-23T16:33:57.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c420709:::benchmark_data_smithi188_113957_object29817:head# 2024-04-23T16:33:57.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c4210a3:::benchmark_data_smithi188_113957_object28534:head# 2024-04-23T16:33:57.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c425a4b:::benchmark_data_smithi188_113957_object93642:head# 2024-04-23T16:33:57.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c434686:::benchmark_data_smithi188_113957_object113901:head# 2024-04-23T16:33:57.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c43e976:::benchmark_data_smithi188_113957_object129549:head# 2024-04-23T16:33:57.116 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.116 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c447a76:::benchmark_data_smithi188_113957_object10503:head# 2024-04-23T16:33:57.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c44b68c:::benchmark_data_smithi188_113957_object32015:head# 2024-04-23T16:33:57.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c454b58:::benchmark_data_smithi188_113957_object132286:head# 2024-04-23T16:33:57.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c45add9:::benchmark_data_smithi188_113957_object140402:head# 2024-04-23T16:33:57.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c45b75d:::benchmark_data_smithi188_113957_object112221:head# 2024-04-23T16:33:57.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c45dd5e:::benchmark_data_smithi188_113957_object102948:head# 2024-04-23T16:33:57.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.117 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c4633cb:::benchmark_data_smithi188_113957_object79387:head# 2024-04-23T16:33:57.117 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c469cd9:::benchmark_data_smithi188_113957_object30708:head# 2024-04-23T16:33:57.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c474e91:::benchmark_data_smithi188_113957_object70612:head# 2024-04-23T16:33:57.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c474faf:::benchmark_data_smithi188_113957_object40327:head# 2024-04-23T16:33:57.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c475e20:::benchmark_data_smithi188_113957_object136499:head# 2024-04-23T16:33:57.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c4763de:::benchmark_data_smithi188_113957_object68810:head# 2024-04-23T16:33:57.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c47cbeb:::benchmark_data_smithi188_113957_object43559:head# 2024-04-23T16:33:57.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.118 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c4854b1:::benchmark_data_smithi188_113957_object70072:head# 2024-04-23T16:33:57.118 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c486160:::benchmark_data_smithi188_113957_object126680:head# 2024-04-23T16:33:57.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c48ba20:::benchmark_data_smithi188_113957_object44897:head# 2024-04-23T16:33:57.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c48c004:::benchmark_data_smithi188_113957_object27548:head# 2024-04-23T16:33:57.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c48e025:::benchmark_data_smithi188_113957_object109502:head# 2024-04-23T16:33:57.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c48fcc6:::benchmark_data_smithi188_113957_object61239:head# 2024-04-23T16:33:57.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c49a610:::benchmark_data_smithi188_113957_object68432:head# 2024-04-23T16:33:57.119 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.119 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c4b19df:::benchmark_data_smithi188_113957_object110668:head# 2024-04-23T16:33:57.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.120 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c4b2332:::benchmark_data_smithi188_113957_object152066:head# 2024-04-23T16:33:57.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.120 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c4b74d9:::benchmark_data_smithi188_113957_object125096:head# 2024-04-23T16:33:57.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.120 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c4bcb26:::benchmark_data_smithi188_113957_object49195:head# 2024-04-23T16:33:57.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.120 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c4c830a:::benchmark_data_smithi188_113957_object114273:head# 2024-04-23T16:33:57.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.120 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c4c9395:::benchmark_data_smithi188_113957_object66598:head# 2024-04-23T16:33:57.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.120 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c4cc2d9:::benchmark_data_smithi188_113957_object69284:head# 2024-04-23T16:33:57.120 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.121 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c4ce210:::benchmark_data_smithi188_113957_object151063:head# 2024-04-23T16:33:57.121 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.121 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c4d1aaf:::benchmark_data_smithi188_113957_object56333:head# 2024-04-23T16:33:57.121 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.121 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c4d7974:::benchmark_data_smithi188_113957_object6781:head# 2024-04-23T16:33:57.121 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.121 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c4edfd2:::benchmark_data_smithi188_113957_object23379:head# 2024-04-23T16:33:57.121 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.121 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c4fa861:::benchmark_data_smithi188_113957_object51526:head# 2024-04-23T16:33:57.121 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.121 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5088e6:::benchmark_data_smithi188_113957_object45749:head# 2024-04-23T16:33:57.121 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.121 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c509646:::benchmark_data_smithi188_113957_object59909:head# 2024-04-23T16:33:57.121 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.122 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c51302f:::benchmark_data_smithi188_113957_object100286:head# 2024-04-23T16:33:57.122 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.122 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c517cbf:::benchmark_data_smithi188_113957_object9908:head# 2024-04-23T16:33:57.122 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.122 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c51c5ff:::benchmark_data_smithi188_113957_object137096:head# 2024-04-23T16:33:57.122 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.122 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c51e453:::benchmark_data_smithi188_113957_object26371:head# 2024-04-23T16:33:57.122 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.122 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c520143:::benchmark_data_smithi188_113957_object131809:head# 2024-04-23T16:33:57.122 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.122 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c52699c:::benchmark_data_smithi188_113957_object40542:head# 2024-04-23T16:33:57.122 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.122 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c533b81:::benchmark_data_smithi188_113957_object10625:head# 2024-04-23T16:33:57.123 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.123 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5492cf:::benchmark_data_smithi188_113957_object32137:head# 2024-04-23T16:33:57.123 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.123 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c54c418:::benchmark_data_smithi188_113957_object112461:head# 2024-04-23T16:33:57.123 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.123 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c54c9c2:::benchmark_data_smithi188_113957_object81646:head# 2024-04-23T16:33:57.123 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.123 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c553112:::benchmark_data_smithi188_113957_object86571:head# 2024-04-23T16:33:57.123 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.123 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c55914c:::benchmark_data_smithi188_113957_object129641:head# 2024-04-23T16:33:57.123 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.123 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5596b5:::benchmark_data_smithi188_113957_object90067:head# 2024-04-23T16:33:57.123 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.123 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c56885e:::benchmark_data_smithi188_113957_object75582:head# 2024-04-23T16:33:57.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5708c8:::benchmark_data_smithi188_113957_object131180:head# 2024-04-23T16:33:57.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c573c90:::benchmark_data_smithi188_113957_object98453:head# 2024-04-23T16:33:57.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5820e0:::benchmark_data_smithi188_113957_object143386:head# 2024-04-23T16:33:57.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c58397f:::benchmark_data_smithi188_113957_object108082:head# 2024-04-23T16:33:57.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c585957:::benchmark_data_smithi188_113957_object36051:head# 2024-04-23T16:33:57.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.124 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c59284d:::benchmark_data_smithi188_113957_object66057:head# 2024-04-23T16:33:57.124 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c59351f:::benchmark_data_smithi188_113957_object148580:head# 2024-04-23T16:33:57.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c595d88:::benchmark_data_smithi188_113957_object12010:head# 2024-04-23T16:33:57.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5a0e7b:::benchmark_data_smithi188_113957_object10281:head# 2024-04-23T16:33:57.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5a5534:::benchmark_data_smithi188_113957_object55340:head# 2024-04-23T16:33:57.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5ae346:::benchmark_data_smithi188_113957_object71370:head# 2024-04-23T16:33:57.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5aeedd:::benchmark_data_smithi188_113957_object98775:head# 2024-04-23T16:33:57.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.125 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5af80a:::benchmark_data_smithi188_113957_object88326:head# 2024-04-23T16:33:57.125 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5b2d58:::benchmark_data_smithi188_113957_object108712:head# 2024-04-23T16:33:57.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5b47c1:::benchmark_data_smithi188_113957_object78870:head# 2024-04-23T16:33:57.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5c8b7d:::benchmark_data_smithi188_113957_object45358:head# 2024-04-23T16:33:57.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5c9d2f:::benchmark_data_smithi188_113957_object138448:head# 2024-04-23T16:33:57.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5ccde5:::benchmark_data_smithi188_113957_object73119:head# 2024-04-23T16:33:57.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5d4911:::benchmark_data_smithi188_113957_object148908:head# 2024-04-23T16:33:57.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.126 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5d5cb3:::benchmark_data_smithi188_113957_object94934:head# 2024-04-23T16:33:57.126 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5dc08a:::benchmark_data_smithi188_113957_object75068:head# 2024-04-23T16:33:57.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5dd1c9:::benchmark_data_smithi188_113957_object90172:head# 2024-04-23T16:33:57.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5e115c:::benchmark_data_smithi188_113957_object36871:head# 2024-04-23T16:33:57.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5e9bc3:::benchmark_data_smithi188_113957_object50225:head# 2024-04-23T16:33:57.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5eb44f:::benchmark_data_smithi188_113957_object77426:head# 2024-04-23T16:33:57.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5ee2f5:::benchmark_data_smithi188_113957_object129522:head# 2024-04-23T16:33:57.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.127 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5eeda4:::benchmark_data_smithi188_113957_object129995:head# 2024-04-23T16:33:57.127 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c5f12b9:::benchmark_data_smithi188_113957_object68747:head# 2024-04-23T16:33:57.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c606ed6:::benchmark_data_smithi188_113957_object98247:head# 2024-04-23T16:33:57.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c60e38c:::benchmark_data_smithi188_113957_object16637:head# 2024-04-23T16:33:57.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c614c18:::benchmark_data_smithi188_113957_object112548:head# 2024-04-23T16:33:57.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c61b7ae:::benchmark_data_smithi188_113957_object6775:head# 2024-04-23T16:33:57.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c62448e:::benchmark_data_smithi188_113957_object152195:head# 2024-04-23T16:33:57.128 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.128 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c62516d:::benchmark_data_smithi188_113957_object129914:head# 2024-04-23T16:33:57.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c629ae9:::benchmark_data_smithi188_113957_object89742:head# 2024-04-23T16:33:57.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c62b6eb:::benchmark_data_smithi188_113957_object35842:head# 2024-04-23T16:33:57.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c634b5d:::benchmark_data_smithi188_113957_object56173:head# 2024-04-23T16:33:57.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c63a020:::benchmark_data_smithi188_113957_object9426:head# 2024-04-23T16:33:57.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c64c023:::benchmark_data_smithi188_113957_object23103:head# 2024-04-23T16:33:57.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c6514f8:::benchmark_data_smithi188_113957_object123186:head# 2024-04-23T16:33:57.129 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.129 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c661e4a:::benchmark_data_smithi188_113957_object16573:head# 2024-04-23T16:33:57.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c662391:::benchmark_data_smithi188_113957_object34946:head# 2024-04-23T16:33:57.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c6653ac:::benchmark_data_smithi188_113957_object45471:head# 2024-04-23T16:33:57.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c671bdf:::benchmark_data_smithi188_113957_object58809:head# 2024-04-23T16:33:57.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c672ae5:::benchmark_data_smithi188_113957_object16373:head# 2024-04-23T16:33:57.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c6793b8:::benchmark_data_smithi188_113957_object138169:head# 2024-04-23T16:33:57.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.130 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c682c8c:::benchmark_data_smithi188_113957_object98692:head# 2024-04-23T16:33:57.130 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c6963c9:::benchmark_data_smithi188_113957_object103172:head# 2024-04-23T16:33:57.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c69942e:::benchmark_data_smithi188_113957_object115595:head# 2024-04-23T16:33:57.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c6a83f9:::benchmark_data_smithi188_113957_object4678:head# 2024-04-23T16:33:57.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c6a9bb4:::benchmark_data_smithi188_113957_object87966:head# 2024-04-23T16:33:57.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c6ae872:::benchmark_data_smithi188_113957_object40264:head# 2024-04-23T16:33:57.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c6b6953:::benchmark_data_smithi188_113957_object126169:head# 2024-04-23T16:33:57.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.131 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c6c1308:::benchmark_data_smithi188_113957_object124342:head# 2024-04-23T16:33:57.131 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c6c75d0:::benchmark_data_smithi188_113957_object11609:head# 2024-04-23T16:33:57.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c6cd3b2:::benchmark_data_smithi188_113957_object82321:head# 2024-04-23T16:33:57.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c6d6c6e:::benchmark_data_smithi188_113957_object123974:head# 2024-04-23T16:33:57.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c6f4607:::benchmark_data_smithi188_113957_object27747:head# 2024-04-23T16:33:57.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c6fe41f:::benchmark_data_smithi188_113957_object23098:head# 2024-04-23T16:33:57.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c710934:::benchmark_data_smithi188_113957_object137959:head# 2024-04-23T16:33:57.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.132 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c711661:::benchmark_data_smithi188_113957_object25040:head# 2024-04-23T16:33:57.132 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c712bc1:::benchmark_data_smithi188_113957_object30408:head# 2024-04-23T16:33:57.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c714b18:::benchmark_data_smithi188_113957_object67676:head# 2024-04-23T16:33:57.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c719ca7:::benchmark_data_smithi188_113957_object52028:head# 2024-04-23T16:33:57.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c71f1f7:::benchmark_data_smithi188_113957_object19988:head# 2024-04-23T16:33:57.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c720bb6:::benchmark_data_smithi188_113957_object143:head# 2024-04-23T16:33:57.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c721852:::benchmark_data_smithi188_113957_object66940:head# 2024-04-23T16:33:57.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.133 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c72984c:::benchmark_data_smithi188_113957_object60960:head# 2024-04-23T16:33:57.133 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.134 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c72a6c0:::benchmark_data_smithi188_113957_object65139:head# 2024-04-23T16:33:57.134 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.134 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c72fe23:::benchmark_data_smithi188_113957_object93198:head# 2024-04-23T16:33:57.134 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.134 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c73198e:::benchmark_data_smithi188_113957_object8522:head# 2024-04-23T16:33:57.134 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.134 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c731d3e:::benchmark_data_smithi188_113957_object12665:head# 2024-04-23T16:33:57.134 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.134 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c73ac00:::benchmark_data_smithi188_113957_object118518:head# 2024-04-23T16:33:57.134 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.134 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c73cdda:::benchmark_data_smithi188_113957_object69866:head# 2024-04-23T16:33:57.134 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.134 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c73e840:::benchmark_data_smithi188_113957_object119016:head# 2024-04-23T16:33:57.135 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.135 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c73e939:::benchmark_data_smithi188_113957_object112748:head# 2024-04-23T16:33:57.135 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.135 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c744c37:::benchmark_data_smithi188_113957_object59791:head# 2024-04-23T16:33:57.137 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.138 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c74e992:::benchmark_data_smithi188_113957_object1891:head# 2024-04-23T16:33:57.138 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.138 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c74e9dd:::benchmark_data_smithi188_113957_object129794:head# 2024-04-23T16:33:57.138 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.138 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c756f0d:::benchmark_data_smithi188_113957_object21998:head# 2024-04-23T16:33:57.138 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.138 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c766700:::benchmark_data_smithi188_113957_object60813:head# 2024-04-23T16:33:57.138 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.138 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c772b0b:::benchmark_data_smithi188_113957_object38396:head# 2024-04-23T16:33:57.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c775915:::benchmark_data_smithi188_113957_object55971:head# 2024-04-23T16:33:57.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c77bfdc:::benchmark_data_smithi188_113957_object137475:head# 2024-04-23T16:33:57.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c7801b7:::benchmark_data_smithi188_113957_object54727:head# 2024-04-23T16:33:57.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c7889df:::benchmark_data_smithi188_113957_object45216:head# 2024-04-23T16:33:57.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c793c85:::benchmark_data_smithi188_113957_object6785:head# 2024-04-23T16:33:57.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c79b28a:::benchmark_data_smithi188_113957_object50860:head# 2024-04-23T16:33:57.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.139 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c7ae476:::benchmark_data_smithi188_113957_object141270:head# 2024-04-23T16:33:57.139 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c7b5494:::benchmark_data_smithi188_113957_object146513:head# 2024-04-23T16:33:57.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c7be5bb:::benchmark_data_smithi188_113957_object33260:head# 2024-04-23T16:33:57.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c7c4588:::benchmark_data_smithi188_113957_object92123:head# 2024-04-23T16:33:57.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c7c55b9:::benchmark_data_smithi188_113957_object70350:head# 2024-04-23T16:33:57.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c7cdb40:::benchmark_data_smithi188_113957_object26391:head# 2024-04-23T16:33:57.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c7d23f8:::benchmark_data_smithi188_113957_object68749:head# 2024-04-23T16:33:57.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.140 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c7d5a88:::benchmark_data_smithi188_113957_object5261:head# 2024-04-23T16:33:57.140 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c7dbbd8:::benchmark_data_smithi188_113957_object104303:head# 2024-04-23T16:33:57.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c7df7df:::benchmark_data_smithi188_113957_object140447:head# 2024-04-23T16:33:57.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c7f38cb:::benchmark_data_smithi188_113957_object97004:head# 2024-04-23T16:33:57.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c7f3c6d:::benchmark_data_smithi188_113957_object52163:head# 2024-04-23T16:33:57.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c7fb864:::benchmark_data_smithi188_113957_object26466:head# 2024-04-23T16:33:57.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c805fce:::benchmark_data_smithi188_113957_object95258:head# 2024-04-23T16:33:57.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.141 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c80fc50:::benchmark_data_smithi188_113957_object76728:head# 2024-04-23T16:33:57.141 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c814b64:::benchmark_data_smithi188_113957_object153330:head# 2024-04-23T16:33:57.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c81902a:::benchmark_data_smithi188_113957_object106184:head# 2024-04-23T16:33:57.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8195f3:::benchmark_data_smithi188_113957_object152299:head# 2024-04-23T16:33:57.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c81b64f:::benchmark_data_smithi188_113957_object66371:head# 2024-04-23T16:33:57.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8218fd:::benchmark_data_smithi188_113957_object55178:head# 2024-04-23T16:33:57.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c827408:::benchmark_data_smithi188_113957_object133788:head# 2024-04-23T16:33:57.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.142 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c82b93f:::benchmark_data_smithi188_113957_object64090:head# 2024-04-23T16:33:57.142 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c832181:::benchmark_data_smithi188_113957_object51326:head# 2024-04-23T16:33:57.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c83aa57:::benchmark_data_smithi188_113957_object88147:head# 2024-04-23T16:33:57.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c83d2c1:::benchmark_data_smithi188_113957_object50843:head# 2024-04-23T16:33:57.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c845697:::benchmark_data_smithi188_113957_object38114:head# 2024-04-23T16:33:57.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c849d29:::benchmark_data_smithi188_113957_object136478:head# 2024-04-23T16:33:57.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c854fcc:::benchmark_data_smithi188_113957_object83529:head# 2024-04-23T16:33:57.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.143 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8613e7:::benchmark_data_smithi188_113957_object147769:head# 2024-04-23T16:33:57.143 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c869ecf:::benchmark_data_smithi188_113957_object119061:head# 2024-04-23T16:33:57.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c876b80:::benchmark_data_smithi188_113957_object131103:head# 2024-04-23T16:33:57.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c877819:::benchmark_data_smithi188_113957_object70655:head# 2024-04-23T16:33:57.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c87a4c8:::benchmark_data_smithi188_113957_object118568:head# 2024-04-23T16:33:57.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c87d560:::benchmark_data_smithi188_113957_object77677:head# 2024-04-23T16:33:57.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c88a399:::benchmark_data_smithi188_113957_object92501:head# 2024-04-23T16:33:57.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.144 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c899c5d:::benchmark_data_smithi188_113957_object112776:head# 2024-04-23T16:33:57.144 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c899f7a:::benchmark_data_smithi188_113957_object5022:head# 2024-04-23T16:33:57.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8a0cde:::benchmark_data_smithi188_113957_object37125:head# 2024-04-23T16:33:57.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8a444f:::benchmark_data_smithi188_113957_object129422:head# 2024-04-23T16:33:57.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8a73cd:::benchmark_data_smithi188_113957_object124373:head# 2024-04-23T16:33:57.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8bad9b:::benchmark_data_smithi188_113957_object79305:head# 2024-04-23T16:33:57.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8bb973:::benchmark_data_smithi188_113957_object5807:head# 2024-04-23T16:33:57.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8bd0f9:::benchmark_data_smithi188_113957_object104732:head# 2024-04-23T16:33:57.145 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.145 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8bf7f4:::benchmark_data_smithi188_113957_object149084:head# 2024-04-23T16:33:57.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8c0353:::benchmark_data_smithi188_113957_object47423:head# 2024-04-23T16:33:57.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8c7663:::benchmark_data_smithi188_113957_object94044:head# 2024-04-23T16:33:57.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8c7ede:::benchmark_data_smithi188_113957_object72799:head# 2024-04-23T16:33:57.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8d3695:::benchmark_data_smithi188_113957_object69170:head# 2024-04-23T16:33:57.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8e2c6c:::benchmark_data_smithi188_113957_object86452:head# 2024-04-23T16:33:57.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.146 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8e5b5d:::benchmark_data_smithi188_113957_object39241:head# 2024-04-23T16:33:57.146 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8e6243:::benchmark_data_smithi188_113957_object93962:head# 2024-04-23T16:33:57.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8ed484:::benchmark_data_smithi188_113957_object100217:head# 2024-04-23T16:33:57.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8f1d7e:::benchmark_data_smithi188_113957_object97516:head# 2024-04-23T16:33:57.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c8fb614:::benchmark_data_smithi188_113957_object148155:head# 2024-04-23T16:33:57.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c901aa3:::benchmark_data_smithi188_113957_object80500:head# 2024-04-23T16:33:57.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c90989c:::benchmark_data_smithi188_113957_object58433:head# 2024-04-23T16:33:57.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c911c72:::benchmark_data_smithi188_113957_object30589:head# 2024-04-23T16:33:57.147 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.147 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c91222d:::benchmark_data_smithi188_113957_object98097:head# 2024-04-23T16:33:57.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c91b345:::benchmark_data_smithi188_113957_object20479:head# 2024-04-23T16:33:57.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c91c503:::benchmark_data_smithi188_113957_object117643:head# 2024-04-23T16:33:57.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c91c6ad:::benchmark_data_smithi188_113957_object133118:head# 2024-04-23T16:33:57.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c9275cd:::benchmark_data_smithi188_113957_object134708:head# 2024-04-23T16:33:57.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c92883e:::benchmark_data_smithi188_113957_object88961:head# 2024-04-23T16:33:57.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.148 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c929d88:::benchmark_data_smithi188_113957_object118295:head# 2024-04-23T16:33:57.148 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c93f7e2:::benchmark_data_smithi188_113957_object15801:head# 2024-04-23T16:33:57.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c94231d:::benchmark_data_smithi188_113957_object9267:head# 2024-04-23T16:33:57.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c9449fc:::benchmark_data_smithi188_113957_object149500:head# 2024-04-23T16:33:57.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c946ce4:::benchmark_data_smithi188_113957_object37570:head# 2024-04-23T16:33:57.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c949060:::benchmark_data_smithi188_113957_object127544:head# 2024-04-23T16:33:57.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c949087:::benchmark_data_smithi188_113957_object134454:head# 2024-04-23T16:33:57.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.149 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c94f4f0:::benchmark_data_smithi188_113957_object50512:head# 2024-04-23T16:33:57.149 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c951911:::benchmark_data_smithi188_113957_object64810:head# 2024-04-23T16:33:57.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c96330d:::benchmark_data_smithi188_113957_object8578:head# 2024-04-23T16:33:57.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c965ff0:::benchmark_data_smithi188_113957_object135308:head# 2024-04-23T16:33:57.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c968bf4:::benchmark_data_smithi188_113957_object76385:head# 2024-04-23T16:33:57.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c96a202:::benchmark_data_smithi188_113957_object82215:head# 2024-04-23T16:33:57.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c971b2d:::benchmark_data_smithi188_113957_object62597:head# 2024-04-23T16:33:57.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.150 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c973f03:::benchmark_data_smithi188_113957_object56325:head# 2024-04-23T16:33:57.150 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c9861a4:::benchmark_data_smithi188_113957_object3019:head# 2024-04-23T16:33:57.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c98d9ec:::benchmark_data_smithi188_113957_object132083:head# 2024-04-23T16:33:57.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c9902cd:::benchmark_data_smithi188_113957_object86891:head# 2024-04-23T16:33:57.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c995199:::benchmark_data_smithi188_113957_object46218:head# 2024-04-23T16:33:57.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c996bde:::benchmark_data_smithi188_113957_object7956:head# 2024-04-23T16:33:57.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c9a2787:::benchmark_data_smithi188_113957_object35347:head# 2024-04-23T16:33:57.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.151 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c9a64aa:::benchmark_data_smithi188_113957_object12360:head# 2024-04-23T16:33:57.151 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c9a92b1:::benchmark_data_smithi188_113957_object74455:head# 2024-04-23T16:33:57.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c9ae713:::benchmark_data_smithi188_113957_object120467:head# 2024-04-23T16:33:57.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c9b0042:::benchmark_data_smithi188_113957_object104704:head# 2024-04-23T16:33:57.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c9b0335:::benchmark_data_smithi188_113957_object55759:head# 2024-04-23T16:33:57.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c9b1cf9:::benchmark_data_smithi188_113957_object88904:head# 2024-04-23T16:33:57.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c9cf01c:::benchmark_data_smithi188_113957_object96180:head# 2024-04-23T16:33:57.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.152 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c9d0f6e:::benchmark_data_smithi188_113957_object17132:head# 2024-04-23T16:33:57.152 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c9e3e76:::benchmark_data_smithi188_113957_object115122:head# 2024-04-23T16:33:57.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c9f4d6d:::benchmark_data_smithi188_113957_object143305:head# 2024-04-23T16:33:57.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5c9fda72:::benchmark_data_smithi188_113957_object125782:head# 2024-04-23T16:33:57.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca0a2e9:::benchmark_data_smithi188_113957_object80116:head# 2024-04-23T16:33:57.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca0b4df:::benchmark_data_smithi188_113957_object85588:head# 2024-04-23T16:33:57.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca12a63:::benchmark_data_smithi188_113957_object4845:head# 2024-04-23T16:33:57.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.153 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca148e5:::benchmark_data_smithi188_113957_object145393:head# 2024-04-23T16:33:57.153 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca1ff6f:::benchmark_data_smithi188_113957_object9108:head# 2024-04-23T16:33:57.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca2466d:::benchmark_data_smithi188_113957_object75944:head# 2024-04-23T16:33:57.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca253ef:::benchmark_data_smithi188_113957_object29250:head# 2024-04-23T16:33:57.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca2f7b6:::benchmark_data_smithi188_113957_object58696:head# 2024-04-23T16:33:57.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca30fd2:::benchmark_data_smithi188_113957_object33418:head# 2024-04-23T16:33:57.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca372fe:::benchmark_data_smithi188_113957_object27849:head# 2024-04-23T16:33:57.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.154 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca379d6:::benchmark_data_smithi188_113957_object39828:head# 2024-04-23T16:33:57.154 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.155 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca3a9de:::benchmark_data_smithi188_113957_object78361:head# 2024-04-23T16:33:57.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.155 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca4294b:::benchmark_data_smithi188_113957_object50478:head# 2024-04-23T16:33:57.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.155 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca4aa5a:::benchmark_data_smithi188_113957_object50951:head# 2024-04-23T16:33:57.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.155 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca528a0:::benchmark_data_smithi188_113957_object119454:head# 2024-04-23T16:33:57.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.155 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca71e7b:::benchmark_data_smithi188_113957_object109439:head# 2024-04-23T16:33:57.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.155 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca73380:::benchmark_data_smithi188_113957_object27238:head# 2024-04-23T16:33:57.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.155 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca73dfe:::benchmark_data_smithi188_113957_object81503:head# 2024-04-23T16:33:57.155 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.155 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca84b9f:::benchmark_data_smithi188_113957_object401:head# 2024-04-23T16:33:57.156 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.156 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca89270:::benchmark_data_smithi188_113957_object56225:head# 2024-04-23T16:33:57.156 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.156 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ca98ec8:::benchmark_data_smithi188_113957_object123803:head# 2024-04-23T16:33:57.156 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.156 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5caa0980:::benchmark_data_smithi188_113957_object122257:head# 2024-04-23T16:33:57.156 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.156 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5caaaa9a:::benchmark_data_smithi188_113957_object68097:head# 2024-04-23T16:33:57.156 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.156 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cab0125:::benchmark_data_smithi188_113957_object46058:head# 2024-04-23T16:33:57.156 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.156 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cab33b3:::benchmark_data_smithi188_113957_object23095:head# 2024-04-23T16:33:57.156 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.156 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cab936a:::benchmark_data_smithi188_113957_object95399:head# 2024-04-23T16:33:57.157 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.157 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cac38bb:::benchmark_data_smithi188_113957_object128254:head# 2024-04-23T16:33:57.157 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.157 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5caceda1:::benchmark_data_smithi188_113957_object116111:head# 2024-04-23T16:33:57.157 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.157 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cad03b7:::benchmark_data_smithi188_113957_object2775:head# 2024-04-23T16:33:57.157 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.157 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cad7d35:::benchmark_data_smithi188_113957_object102570:head# 2024-04-23T16:33:57.157 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.157 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cadc6c0:::benchmark_data_smithi188_113957_object62600:head# 2024-04-23T16:33:57.157 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.157 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cadd51b:::benchmark_data_smithi188_113957_object150103:head# 2024-04-23T16:33:57.157 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.157 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cadf308:::benchmark_data_smithi188_113957_object305:head# 2024-04-23T16:33:57.157 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.158 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5caf7655:::benchmark_data_smithi188_113957_object80188:head# 2024-04-23T16:33:57.158 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.158 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5caf9167:::benchmark_data_smithi188_113957_object42323:head# 2024-04-23T16:33:57.158 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.158 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5caf9c1c:::benchmark_data_smithi188_113957_object64198:head# 2024-04-23T16:33:57.158 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.158 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cafa48d:::benchmark_data_smithi188_113957_object106514:head# 2024-04-23T16:33:57.158 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.158 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cafa58b:::benchmark_data_smithi188_113957_object8004:head# 2024-04-23T16:33:57.158 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.158 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cafd317:::benchmark_data_smithi188_113957_object137214:head# 2024-04-23T16:33:57.158 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.158 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb05434:::benchmark_data_smithi188_113957_object675:head# 2024-04-23T16:33:57.159 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.159 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb11616:::benchmark_data_smithi188_113957_object96711:head# 2024-04-23T16:33:57.159 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.159 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb16cea:::benchmark_data_smithi188_113957_object14926:head# 2024-04-23T16:33:57.159 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.159 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb1e29b:::benchmark_data_smithi188_113957_object106474:head# 2024-04-23T16:33:57.159 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.159 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb20b7c:::benchmark_data_smithi188_113957_object41311:head# 2024-04-23T16:33:57.159 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.159 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb35214:::benchmark_data_smithi188_113957_object29973:head# 2024-04-23T16:33:57.159 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.159 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb3f854:::benchmark_data_smithi188_113957_object22686:head# 2024-04-23T16:33:57.159 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.159 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb40d36:::benchmark_data_smithi188_113957_object75092:head# 2024-04-23T16:33:57.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb4e649:::benchmark_data_smithi188_113957_object68162:head# 2024-04-23T16:33:57.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb51888:::benchmark_data_smithi188_113957_object127065:head# 2024-04-23T16:33:57.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb585cd:::benchmark_data_smithi188_113957_object33453:head# 2024-04-23T16:33:57.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb595c0:::benchmark_data_smithi188_113957_object87658:head# 2024-04-23T16:33:57.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb59cbe:::benchmark_data_smithi188_113957_object73168:head# 2024-04-23T16:33:57.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb5b8dd:::benchmark_data_smithi188_113957_object124117:head# 2024-04-23T16:33:57.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb65aff:::benchmark_data_smithi188_113957_object43532:head# 2024-04-23T16:33:57.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb72f9c:::benchmark_data_smithi188_113957_object131698:head# 2024-04-23T16:33:57.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb774a2:::benchmark_data_smithi188_113957_object87240:head# 2024-04-23T16:33:57.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb77805:::benchmark_data_smithi188_113957_object42118:head# 2024-04-23T16:33:57.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb86a77:::benchmark_data_smithi188_113957_object15891:head# 2024-04-23T16:33:57.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb8c060:::benchmark_data_smithi188_113957_object75693:head# 2024-04-23T16:33:57.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cb8f713:::benchmark_data_smithi188_113957_object96367:head# 2024-04-23T16:33:57.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cba1713:::benchmark_data_smithi188_113957_object26794:head# 2024-04-23T16:33:57.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cbafa43:::benchmark_data_smithi188_113957_object124231:head# 2024-04-23T16:33:57.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cbb18b2:::benchmark_data_smithi188_113957_object74634:head# 2024-04-23T16:33:57.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cbb28db:::benchmark_data_smithi188_113957_object105039:head# 2024-04-23T16:33:57.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cbb525b:::benchmark_data_smithi188_113957_object9378:head# 2024-04-23T16:33:57.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cbb5521:::benchmark_data_smithi188_113957_object58703:head# 2024-04-23T16:33:57.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cbb5f39:::benchmark_data_smithi188_113957_object118489:head# 2024-04-23T16:33:57.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cbe65eb:::benchmark_data_smithi188_113957_object95421:head# 2024-04-23T16:33:57.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cbefb51:::benchmark_data_smithi188_113957_object23274:head# 2024-04-23T16:33:57.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cbf38d8:::benchmark_data_smithi188_113957_object43348:head# 2024-04-23T16:33:57.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cbf4d76:::benchmark_data_smithi188_113957_object24946:head# 2024-04-23T16:33:57.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cbfab92:::benchmark_data_smithi188_113957_object86315:head# 2024-04-23T16:33:57.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc04f6d:::benchmark_data_smithi188_113957_object14594:head# 2024-04-23T16:33:57.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc05718:::benchmark_data_smithi188_113957_object126931:head# 2024-04-23T16:33:57.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc08beb:::benchmark_data_smithi188_113957_object7303:head# 2024-04-23T16:33:57.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc0ac54:::benchmark_data_smithi188_113957_object37977:head# 2024-04-23T16:33:57.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc150d8:::benchmark_data_smithi188_113957_object121568:head# 2024-04-23T16:33:57.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc17079:::benchmark_data_smithi188_113957_object138461:head# 2024-04-23T16:33:57.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc194b8:::benchmark_data_smithi188_113957_object75559:head# 2024-04-23T16:33:57.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc1a0ce:::benchmark_data_smithi188_113957_object62971:head# 2024-04-23T16:33:57.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc1d289:::benchmark_data_smithi188_113957_object96162:head# 2024-04-23T16:33:57.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc1f8d6:::benchmark_data_smithi188_113957_object78581:head# 2024-04-23T16:33:57.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc1ff48:::benchmark_data_smithi188_113957_object97107:head# 2024-04-23T16:33:57.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc2045d:::benchmark_data_smithi188_113957_object33175:head# 2024-04-23T16:33:57.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc29e8e:::benchmark_data_smithi188_113957_object67764:head# 2024-04-23T16:33:57.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc2ea11:::benchmark_data_smithi188_113957_object115221:head# 2024-04-23T16:33:57.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc3af77:::benchmark_data_smithi188_113957_object121258:head# 2024-04-23T16:33:57.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc3f33d:::benchmark_data_smithi188_113957_object127571:head# 2024-04-23T16:33:57.166 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.166 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc4c9b1:::benchmark_data_smithi188_113957_object145566:head# 2024-04-23T16:33:57.166 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.166 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc518fe:::benchmark_data_smithi188_113957_object32618:head# 2024-04-23T16:33:57.166 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.166 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc5cf47:::benchmark_data_smithi188_113957_object102403:head# 2024-04-23T16:33:57.166 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.166 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc6749f:::benchmark_data_smithi188_113957_object26449:head# 2024-04-23T16:33:57.166 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.166 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc67c86:::benchmark_data_smithi188_113957_object52187:head# 2024-04-23T16:33:57.166 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.166 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc6aeda:::benchmark_data_smithi188_113957_object30412:head# 2024-04-23T16:33:57.166 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc6d3b0:::benchmark_data_smithi188_113957_object116949:head# 2024-04-23T16:33:57.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc6d41e:::benchmark_data_smithi188_113957_object117930:head# 2024-04-23T16:33:57.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc7365e:::benchmark_data_smithi188_113957_object46778:head# 2024-04-23T16:33:57.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc73857:::benchmark_data_smithi188_113957_object60774:head# 2024-04-23T16:33:57.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc7f92b:::benchmark_data_smithi188_113957_object111294:head# 2024-04-23T16:33:57.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc8b214:::benchmark_data_smithi188_113957_object33192:head# 2024-04-23T16:33:57.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc99ad3:::benchmark_data_smithi188_113957_object71193:head# 2024-04-23T16:33:57.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc9ef8c:::benchmark_data_smithi188_113957_object43105:head# 2024-04-23T16:33:57.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cc9f907:::benchmark_data_smithi188_113957_object12692:head# 2024-04-23T16:33:57.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cca3f24:::benchmark_data_smithi188_113957_object99933:head# 2024-04-23T16:33:57.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cca56ed:::benchmark_data_smithi188_113957_object60474:head# 2024-04-23T16:33:57.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ccadd24:::benchmark_data_smithi188_113957_object150182:head# 2024-04-23T16:33:57.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ccb82a7:::benchmark_data_smithi188_113957_object102247:head# 2024-04-23T16:33:57.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ccc2d01:::benchmark_data_smithi188_113957_object93932:head# 2024-04-23T16:33:57.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ccc5874:::benchmark_data_smithi188_113957_object75255:head# 2024-04-23T16:33:57.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ccd4928:::benchmark_data_smithi188_113957_object113676:head# 2024-04-23T16:33:57.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ccd6520:::benchmark_data_smithi188_113957_object149562:head# 2024-04-23T16:33:57.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ccd7b89:::benchmark_data_smithi188_113957_object8122:head# 2024-04-23T16:33:57.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ccdc11a:::benchmark_data_smithi188_113957_object56494:head# 2024-04-23T16:33:57.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ccdc9ce:::benchmark_data_smithi188_113957_object104987:head# 2024-04-23T16:33:57.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ccee253:::benchmark_data_smithi188_113957_object33476:head# 2024-04-23T16:33:57.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ccf6758:::benchmark_data_smithi188_113957_object13591:head# 2024-04-23T16:33:57.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ccf802b:::benchmark_data_smithi188_113957_object979:head# 2024-04-23T16:33:57.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cd0016a:::benchmark_data_smithi188_113957_object134111:head# 2024-04-23T16:33:57.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cd0621a:::benchmark_data_smithi188_113957_object150110:head# 2024-04-23T16:33:57.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cd08e48:::benchmark_data_smithi188_113957_object71871:head# 2024-04-23T16:33:57.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cd0bace:::benchmark_data_smithi188_113957_object84655:head# 2024-04-23T16:33:57.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cd12742:::benchmark_data_smithi188_113957_object69316:head# 2024-04-23T16:33:57.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cd31a69:::benchmark_data_smithi188_113957_object15810:head# 2024-04-23T16:33:57.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cd3ae48:::benchmark_data_smithi188_113957_object20848:head# 2024-04-23T16:33:57.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cd3dd95:::benchmark_data_smithi188_113957_object136271:head# 2024-04-23T16:33:57.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cd4afbc:::benchmark_data_smithi188_113957_object79999:head# 2024-04-23T16:33:57.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cd5e5a0:::benchmark_data_smithi188_113957_object125109:head# 2024-04-23T16:33:57.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cd5f89b:::benchmark_data_smithi188_113957_object25538:head# 2024-04-23T16:33:57.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cd6e5dd:::benchmark_data_smithi188_113957_object140418:head# 2024-04-23T16:33:57.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cd78ddd:::benchmark_data_smithi188_113957_object128961:head# 2024-04-23T16:33:57.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cd7fc0b:::benchmark_data_smithi188_113957_object115135:head# 2024-04-23T16:33:57.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cd91afb:::benchmark_data_smithi188_113957_object39381:head# 2024-04-23T16:33:57.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cda54a4:::benchmark_data_smithi188_113957_object153039:head# 2024-04-23T16:33:57.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cdac090:::benchmark_data_smithi188_113957_object70084:head# 2024-04-23T16:33:57.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cdb5b52:::benchmark_data_smithi188_113957_object96353:head# 2024-04-23T16:33:57.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cdb8f01:::benchmark_data_smithi188_113957_object99202:head# 2024-04-23T16:33:57.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cdb9a99:::benchmark_data_smithi188_113957_object133545:head# 2024-04-23T16:33:57.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cdba5ae:::benchmark_data_smithi188_113957_object36444:head# 2024-04-23T16:33:57.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cdc127a:::benchmark_data_smithi188_113957_object10926:head# 2024-04-23T16:33:57.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cdc9d3c:::benchmark_data_smithi188_113957_object142958:head# 2024-04-23T16:33:57.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cdca065:::benchmark_data_smithi188_113957_object80350:head# 2024-04-23T16:33:57.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cdcd662:::benchmark_data_smithi188_113957_object75440:head# 2024-04-23T16:33:57.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cded1f5:::benchmark_data_smithi188_113957_object23105:head# 2024-04-23T16:33:57.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cdee7d7:::benchmark_data_smithi188_113957_object145586:head# 2024-04-23T16:33:57.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cdeef5b:::benchmark_data_smithi188_113957_object81736:head# 2024-04-23T16:33:57.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cdefe6d:::benchmark_data_smithi188_113957_object89546:head# 2024-04-23T16:33:57.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce006c2:::benchmark_data_smithi188_113957_object39700:head# 2024-04-23T16:33:57.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce00d56:::benchmark_data_smithi188_113957_object42246:head# 2024-04-23T16:33:57.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce0e038:::benchmark_data_smithi188_113957_object82467:head# 2024-04-23T16:33:57.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce1fb78:::benchmark_data_smithi188_113957_object111143:head# 2024-04-23T16:33:57.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce26fd8:::benchmark_data_smithi188_113957_object43422:head# 2024-04-23T16:33:57.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce2c27a:::benchmark_data_smithi188_113957_object144604:head# 2024-04-23T16:33:57.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce2cb0d:::benchmark_data_smithi188_113957_object146215:head# 2024-04-23T16:33:57.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce32524:::benchmark_data_smithi188_113957_object114358:head# 2024-04-23T16:33:57.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce35e01:::benchmark_data_smithi188_113957_object81971:head# 2024-04-23T16:33:57.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce3a751:::benchmark_data_smithi188_113957_object8540:head# 2024-04-23T16:33:57.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce41ddf:::benchmark_data_smithi188_113957_object87440:head# 2024-04-23T16:33:57.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce4b4cb:::benchmark_data_smithi188_113957_object135315:head# 2024-04-23T16:33:57.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce5b5fa:::benchmark_data_smithi188_113957_object35059:head# 2024-04-23T16:33:57.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce5ffe5:::benchmark_data_smithi188_113957_object5373:head# 2024-04-23T16:33:57.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce6e904:::benchmark_data_smithi188_113957_object1931:head# 2024-04-23T16:33:57.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce78977:::benchmark_data_smithi188_113957_object33420:head# 2024-04-23T16:33:57.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce7e62d:::benchmark_data_smithi188_113957_object37274:head# 2024-04-23T16:33:57.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce81265:::benchmark_data_smithi188_113957_object128002:head# 2024-04-23T16:33:57.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce8e1fd:::benchmark_data_smithi188_113957_object124792:head# 2024-04-23T16:33:57.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce90dec:::benchmark_data_smithi188_113957_object101828:head# 2024-04-23T16:33:57.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce96363:::benchmark_data_smithi188_113957_object101608:head# 2024-04-23T16:33:57.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce963aa:::benchmark_data_smithi188_113957_object71602:head# 2024-04-23T16:33:57.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ce9f6ff:::benchmark_data_smithi188_113957_object131114:head# 2024-04-23T16:33:57.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cea0dd9:::benchmark_data_smithi188_113957_object29579:head# 2024-04-23T16:33:57.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cea6024:::benchmark_data_smithi188_113957_object13601:head# 2024-04-23T16:33:57.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cea6d65:::benchmark_data_smithi188_113957_object19981:head# 2024-04-23T16:33:57.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ceac30b:::benchmark_data_smithi188_113957_object80877:head# 2024-04-23T16:33:57.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cead6eb:::benchmark_data_smithi188_113957_object85600:head# 2024-04-23T16:33:57.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ceaf649:::benchmark_data_smithi188_113957_object100891:head# 2024-04-23T16:33:57.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ceb54ae:::benchmark_data_smithi188_113957_object102326:head# 2024-04-23T16:33:57.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cebbce2:::benchmark_data_smithi188_113957_object33342:head# 2024-04-23T16:33:57.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cebe132:::benchmark_data_smithi188_113957_object30255:head# 2024-04-23T16:33:57.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cebe4e9:::benchmark_data_smithi188_113957_object147667:head# 2024-04-23T16:33:57.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cebf09c:::benchmark_data_smithi188_113957_object3115:head# 2024-04-23T16:33:57.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cec322b:::benchmark_data_smithi188_113957_object23052:head# 2024-04-23T16:33:57.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cec718b:::benchmark_data_smithi188_113957_object59542:head# 2024-04-23T16:33:57.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cec8804:::benchmark_data_smithi188_113957_object19785:head# 2024-04-23T16:33:57.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cecac46:::benchmark_data_smithi188_113957_object791:head# 2024-04-23T16:33:57.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ced26eb:::benchmark_data_smithi188_113957_object4926:head# 2024-04-23T16:33:57.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cedc568:::benchmark_data_smithi188_113957_object30294:head# 2024-04-23T16:33:57.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cedf6f5:::benchmark_data_smithi188_113957_object150070:head# 2024-04-23T16:33:57.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cef0fc7:::benchmark_data_smithi188_113957_object28406:head# 2024-04-23T16:33:57.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cef18ae:::benchmark_data_smithi188_113957_object126647:head# 2024-04-23T16:33:57.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cef2c00:::benchmark_data_smithi188_113957_object43365:head# 2024-04-23T16:33:57.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cef95e5:::benchmark_data_smithi188_113957_object27810:head# 2024-04-23T16:33:57.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cef9b6d:::benchmark_data_smithi188_113957_object144066:head# 2024-04-23T16:33:57.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf08f33:::benchmark_data_smithi188_113957_object55851:head# 2024-04-23T16:33:57.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf0e470:::benchmark_data_smithi188_113957_object111877:head# 2024-04-23T16:33:57.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf14484:::benchmark_data_smithi188_113957_object89191:head# 2024-04-23T16:33:57.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf2af59:::benchmark_data_smithi188_113957_object106761:head# 2024-04-23T16:33:57.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf2e9b3:::benchmark_data_smithi188_113957_object149375:head# 2024-04-23T16:33:57.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf33db5:::benchmark_data_smithi188_113957_object67500:head# 2024-04-23T16:33:57.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf3915d:::benchmark_data_smithi188_113957_object43255:head# 2024-04-23T16:33:57.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf3a05e:::benchmark_data_smithi188_113957_object88581:head# 2024-04-23T16:33:57.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf3b9a4:::benchmark_data_smithi188_113957_object34792:head# 2024-04-23T16:33:57.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf46c1f:::benchmark_data_smithi188_113957_object35090:head# 2024-04-23T16:33:57.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf4897b:::benchmark_data_smithi188_113957_object16017:head# 2024-04-23T16:33:57.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf5f887:::benchmark_data_smithi188_113957_object77734:head# 2024-04-23T16:33:57.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf65f24:::benchmark_data_smithi188_113957_object101833:head# 2024-04-23T16:33:57.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf689ce:::benchmark_data_smithi188_113957_object121240:head# 2024-04-23T16:33:57.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf6b76d:::benchmark_data_smithi188_113957_object145429:head# 2024-04-23T16:33:57.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf6fca1:::benchmark_data_smithi188_113957_object116311:head# 2024-04-23T16:33:57.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf9416a:::benchmark_data_smithi188_113957_object38133:head# 2024-04-23T16:33:57.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf95585:::benchmark_data_smithi188_113957_object149755:head# 2024-04-23T16:33:57.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf9bfa9:::benchmark_data_smithi188_113957_object2913:head# 2024-04-23T16:33:57.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cf9da8a:::benchmark_data_smithi188_113957_object61465:head# 2024-04-23T16:33:57.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cfa48b8:::benchmark_data_smithi188_113957_object144095:head# 2024-04-23T16:33:57.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cfb01c5:::benchmark_data_smithi188_113957_object33428:head# 2024-04-23T16:33:57.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cfb05d8:::benchmark_data_smithi188_113957_object135203:head# 2024-04-23T16:33:57.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cfb222b:::benchmark_data_smithi188_113957_object87101:head# 2024-04-23T16:33:57.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cfb99ce:::benchmark_data_smithi188_113957_object14055:head# 2024-04-23T16:33:57.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cfbe510:::benchmark_data_smithi188_113957_object137428:head# 2024-04-23T16:33:57.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cfc0ca1:::benchmark_data_smithi188_113957_object54684:head# 2024-04-23T16:33:57.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cfc3f3d:::benchmark_data_smithi188_113957_object119877:head# 2024-04-23T16:33:57.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cfc5d40:::benchmark_data_smithi188_113957_object38315:head# 2024-04-23T16:33:57.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cfc66b1:::benchmark_data_smithi188_113957_object58759:head# 2024-04-23T16:33:57.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cfd3566:::benchmark_data_smithi188_113957_object23948:head# 2024-04-23T16:33:57.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cfd54f6:::benchmark_data_smithi188_113957_object86733:head# 2024-04-23T16:33:57.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cff1557:::benchmark_data_smithi188_113957_object73897:head# 2024-04-23T16:33:57.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cff5929:::benchmark_data_smithi188_113957_object147909:head# 2024-04-23T16:33:57.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cff6c8e:::benchmark_data_smithi188_113957_object118839:head# 2024-04-23T16:33:57.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5cffea2d:::benchmark_data_smithi188_113957_object66835:head# 2024-04-23T16:33:57.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d00637e:::benchmark_data_smithi188_113957_object92180:head# 2024-04-23T16:33:57.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0070d8:::benchmark_data_smithi188_113957_object95348:head# 2024-04-23T16:33:57.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0094d8:::benchmark_data_smithi188_113957_object134970:head# 2024-04-23T16:33:57.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d01553c:::benchmark_data_smithi188_113957_object114783:head# 2024-04-23T16:33:57.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d01f93b:::benchmark_data_smithi188_113957_object116746:head# 2024-04-23T16:33:57.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d02b731:::benchmark_data_smithi188_113957_object107977:head# 2024-04-23T16:33:57.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d046921:::benchmark_data_smithi188_113957_object45222:head# 2024-04-23T16:33:57.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0486af:::benchmark_data_smithi188_113957_object1479:head# 2024-04-23T16:33:57.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d048d1a:::benchmark_data_smithi188_113957_object97265:head# 2024-04-23T16:33:57.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d048ff7:::benchmark_data_smithi188_113957_object50892:head# 2024-04-23T16:33:57.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d04cdb3:::benchmark_data_smithi188_113957_object58997:head# 2024-04-23T16:33:57.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d04f614:::benchmark_data_smithi188_113957_object133802:head# 2024-04-23T16:33:57.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d04fb1e:::benchmark_data_smithi188_113957_object116278:head# 2024-04-23T16:33:57.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d050117:::benchmark_data_smithi188_113957_object89230:head# 2024-04-23T16:33:57.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d05dd6c:::benchmark_data_smithi188_113957_object1280:head# 2024-04-23T16:33:57.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d05de5c:::benchmark_data_smithi188_113957_object25471:head# 2024-04-23T16:33:57.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d05ffc6:::benchmark_data_smithi188_113957_object103958:head# 2024-04-23T16:33:57.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d062034:::benchmark_data_smithi188_113957_object106839:head# 2024-04-23T16:33:57.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d06cac8:::benchmark_data_smithi188_113957_object60522:head# 2024-04-23T16:33:57.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d07a4ec:::benchmark_data_smithi188_113957_object149055:head# 2024-04-23T16:33:57.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d08085e:::benchmark_data_smithi188_113957_object129314:head# 2024-04-23T16:33:57.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d083ea8:::benchmark_data_smithi188_113957_object56643:head# 2024-04-23T16:33:57.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d08d8dc:::benchmark_data_smithi188_113957_object101588:head# 2024-04-23T16:33:57.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d098e21:::benchmark_data_smithi188_113957_object146149:head# 2024-04-23T16:33:57.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d09fa51:::benchmark_data_smithi188_113957_object59192:head# 2024-04-23T16:33:57.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0a3873:::benchmark_data_smithi188_113957_object36843:head# 2024-04-23T16:33:57.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0acf2d:::benchmark_data_smithi188_113957_object69249:head# 2024-04-23T16:33:57.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0af967:::benchmark_data_smithi188_113957_object28965:head# 2024-04-23T16:33:57.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0b057e:::benchmark_data_smithi188_113957_object79148:head# 2024-04-23T16:33:57.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0b3399:::benchmark_data_smithi188_113957_object57215:head# 2024-04-23T16:33:57.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0bc34b:::benchmark_data_smithi188_113957_object2749:head# 2024-04-23T16:33:57.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0c3eff:::benchmark_data_smithi188_113957_object38765:head# 2024-04-23T16:33:57.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0c8117:::benchmark_data_smithi188_113957_object30870:head# 2024-04-23T16:33:57.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0ce7b2:::benchmark_data_smithi188_113957_object123015:head# 2024-04-23T16:33:57.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0d37a1:::benchmark_data_smithi188_113957_object30929:head# 2024-04-23T16:33:57.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0dae89:::benchmark_data_smithi188_113957_object112072:head# 2024-04-23T16:33:57.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0dd3ab:::benchmark_data_smithi188_113957_object139831:head# 2024-04-23T16:33:57.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0ea0ca:::benchmark_data_smithi188_113957_object146251:head# 2024-04-23T16:33:57.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0f2717:::benchmark_data_smithi188_113957_object130471:head# 2024-04-23T16:33:57.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0f324c:::benchmark_data_smithi188_113957_object23045:head# 2024-04-23T16:33:57.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0f8777:::benchmark_data_smithi188_113957_object29774:head# 2024-04-23T16:33:57.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d0fcf15:::benchmark_data_smithi188_113957_object417:head# 2024-04-23T16:33:57.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1095c6:::benchmark_data_smithi188_113957_object81537:head# 2024-04-23T16:33:57.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d10cb5b:::benchmark_data_smithi188_113957_object150762:head# 2024-04-23T16:33:57.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d10cb67:::benchmark_data_smithi188_113957_object87128:head# 2024-04-23T16:33:57.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d10e092:::benchmark_data_smithi188_113957_object127614:head# 2024-04-23T16:33:57.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1110e4:::benchmark_data_smithi188_113957_object81896:head# 2024-04-23T16:33:57.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d113a14:::benchmark_data_smithi188_113957_object64362:head# 2024-04-23T16:33:57.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1147a0:::benchmark_data_smithi188_113957_object23401:head# 2024-04-23T16:33:57.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d117ae1:::benchmark_data_smithi188_113957_object106204:head# 2024-04-23T16:33:57.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d11ae7c:::benchmark_data_smithi188_113957_object40491:head# 2024-04-23T16:33:57.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d126757:::benchmark_data_smithi188_113957_object13007:head# 2024-04-23T16:33:57.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1321e7:::benchmark_data_smithi188_113957_object122615:head# 2024-04-23T16:33:57.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1334db:::benchmark_data_smithi188_113957_object25094:head# 2024-04-23T16:33:57.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d13e4ec:::benchmark_data_smithi188_113957_object82175:head# 2024-04-23T16:33:57.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d14ada8:::benchmark_data_smithi188_113957_object91948:head# 2024-04-23T16:33:57.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d151185:::benchmark_data_smithi188_113957_object109584:head# 2024-04-23T16:33:57.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1535d2:::benchmark_data_smithi188_113957_object127759:head# 2024-04-23T16:33:57.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d160d4d:::benchmark_data_smithi188_113957_object42460:head# 2024-04-23T16:33:57.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d168ce5:::benchmark_data_smithi188_113957_object28991:head# 2024-04-23T16:33:57.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d173d0f:::benchmark_data_smithi188_113957_object138719:head# 2024-04-23T16:33:57.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d180d91:::benchmark_data_smithi188_113957_object34488:head# 2024-04-23T16:33:57.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d182f3c:::benchmark_data_smithi188_113957_object114385:head# 2024-04-23T16:33:57.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d185b98:::benchmark_data_smithi188_113957_object98791:head# 2024-04-23T16:33:57.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d186486:::benchmark_data_smithi188_113957_object70026:head# 2024-04-23T16:33:57.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d186f46:::benchmark_data_smithi188_113957_object153250:head# 2024-04-23T16:33:57.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1885b5:::benchmark_data_smithi188_113957_object33034:head# 2024-04-23T16:33:57.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d18c63b:::benchmark_data_smithi188_113957_object139500:head# 2024-04-23T16:33:57.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d18f170:::benchmark_data_smithi188_113957_object22032:head# 2024-04-23T16:33:57.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1948d0:::benchmark_data_smithi188_113957_object27605:head# 2024-04-23T16:33:57.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1a0cd1:::benchmark_data_smithi188_113957_object56984:head# 2024-04-23T16:33:57.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1a7e71:::benchmark_data_smithi188_113957_object68446:head# 2024-04-23T16:33:57.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1b1caa:::benchmark_data_smithi188_113957_object123389:head# 2024-04-23T16:33:57.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1b3b87:::benchmark_data_smithi188_113957_object73498:head# 2024-04-23T16:33:57.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1b635c:::benchmark_data_smithi188_113957_object125852:head# 2024-04-23T16:33:57.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1b66d2:::benchmark_data_smithi188_113957_object37270:head# 2024-04-23T16:33:57.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1c01ab:::benchmark_data_smithi188_113957_object78222:head# 2024-04-23T16:33:57.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1c261a:::benchmark_data_smithi188_113957_object84992:head# 2024-04-23T16:33:57.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1c2939:::benchmark_data_smithi188_113957_object115593:head# 2024-04-23T16:33:57.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1d21fc:::benchmark_data_smithi188_113957_object81249:head# 2024-04-23T16:33:57.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1d7453:::benchmark_data_smithi188_113957_object38068:head# 2024-04-23T16:33:57.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1d97b2:::benchmark_data_smithi188_113957_object108382:head# 2024-04-23T16:33:57.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1dd943:::benchmark_data_smithi188_113957_object82822:head# 2024-04-23T16:33:57.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1de885:::benchmark_data_smithi188_113957_object42386:head# 2024-04-23T16:33:57.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1e11c6:::benchmark_data_smithi188_113957_object90311:head# 2024-04-23T16:33:57.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1e4c0b:::benchmark_data_smithi188_113957_object131639:head# 2024-04-23T16:33:57.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1f4947:::benchmark_data_smithi188_113957_object76989:head# 2024-04-23T16:33:57.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1f5643:::benchmark_data_smithi188_113957_object56613:head# 2024-04-23T16:33:57.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1f5d31:::benchmark_data_smithi188_113957_object81991:head# 2024-04-23T16:33:57.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1f7e46:::benchmark_data_smithi188_113957_object124025:head# 2024-04-23T16:33:57.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1fbbc8:::benchmark_data_smithi188_113957_object93655:head# 2024-04-23T16:33:57.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d1fe2b3:::benchmark_data_smithi188_113957_object52822:head# 2024-04-23T16:33:57.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d203dee:::benchmark_data_smithi188_113957_object57453:head# 2024-04-23T16:33:57.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d216fee:::benchmark_data_smithi188_113957_object43955:head# 2024-04-23T16:33:57.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d21874c:::benchmark_data_smithi188_113957_object149593:head# 2024-04-23T16:33:57.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d218b2d:::benchmark_data_smithi188_113957_object80703:head# 2024-04-23T16:33:57.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d220779:::benchmark_data_smithi188_113957_object145773:head# 2024-04-23T16:33:57.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d220885:::benchmark_data_smithi188_113957_object6524:head# 2024-04-23T16:33:57.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d22942f:::benchmark_data_smithi188_113957_object88637:head# 2024-04-23T16:33:57.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d22ab04:::benchmark_data_smithi188_113957_object10709:head# 2024-04-23T16:33:57.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d22c467:::benchmark_data_smithi188_113957_object117755:head# 2024-04-23T16:33:57.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d22f52b:::benchmark_data_smithi188_113957_object85945:head# 2024-04-23T16:33:57.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2392ee:::benchmark_data_smithi188_113957_object51114:head# 2024-04-23T16:33:57.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d23a8bc:::benchmark_data_smithi188_113957_object65674:head# 2024-04-23T16:33:57.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d23cef5:::benchmark_data_smithi188_113957_object110921:head# 2024-04-23T16:33:57.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d24e2e2:::benchmark_data_smithi188_113957_object16033:head# 2024-04-23T16:33:57.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d24ff3d:::benchmark_data_smithi188_113957_object53197:head# 2024-04-23T16:33:57.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d252ad4:::benchmark_data_smithi188_113957_object132912:head# 2024-04-23T16:33:57.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d253cff:::benchmark_data_smithi188_113957_object25755:head# 2024-04-23T16:33:57.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d257554:::benchmark_data_smithi188_113957_object40883:head# 2024-04-23T16:33:57.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2623be:::benchmark_data_smithi188_113957_object66595:head# 2024-04-23T16:33:57.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d268a4b:::benchmark_data_smithi188_113957_object74886:head# 2024-04-23T16:33:57.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2731e4:::benchmark_data_smithi188_113957_object34024:head# 2024-04-23T16:33:57.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2744c5:::benchmark_data_smithi188_113957_object82545:head# 2024-04-23T16:33:57.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d274630:::benchmark_data_smithi188_113957_object84452:head# 2024-04-23T16:33:57.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d27911b:::benchmark_data_smithi188_113957_object112151:head# 2024-04-23T16:33:57.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d27a8bd:::benchmark_data_smithi188_113957_object69389:head# 2024-04-23T16:33:57.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d27cdb5:::benchmark_data_smithi188_113957_object22228:head# 2024-04-23T16:33:57.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d291e95:::benchmark_data_smithi188_113957_object52959:head# 2024-04-23T16:33:57.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d292e30:::benchmark_data_smithi188_113957_object60675:head# 2024-04-23T16:33:57.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d298ead:::benchmark_data_smithi188_113957_object112387:head# 2024-04-23T16:33:57.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d29aaf5:::benchmark_data_smithi188_113957_object76894:head# 2024-04-23T16:33:57.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2a1d11:::benchmark_data_smithi188_113957_object45365:head# 2024-04-23T16:33:57.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2a978a:::benchmark_data_smithi188_113957_object41915:head# 2024-04-23T16:33:57.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2af72b:::benchmark_data_smithi188_113957_object146161:head# 2024-04-23T16:33:57.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2b3332:::benchmark_data_smithi188_113957_object11157:head# 2024-04-23T16:33:57.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2c1539:::benchmark_data_smithi188_113957_object102760:head# 2024-04-23T16:33:57.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2ca466:::benchmark_data_smithi188_113957_object79986:head# 2024-04-23T16:33:57.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2d5259:::benchmark_data_smithi188_113957_object42542:head# 2024-04-23T16:33:57.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2d787a:::benchmark_data_smithi188_113957_object67269:head# 2024-04-23T16:33:57.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2d989b:::benchmark_data_smithi188_113957_object143578:head# 2024-04-23T16:33:57.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2db626:::benchmark_data_smithi188_113957_object24026:head# 2024-04-23T16:33:57.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2e091b:::benchmark_data_smithi188_113957_object4203:head# 2024-04-23T16:33:57.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2ecaa1:::benchmark_data_smithi188_113957_object14430:head# 2024-04-23T16:33:57.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2f3bf4:::benchmark_data_smithi188_113957_object25889:head# 2024-04-23T16:33:57.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2f5906:::benchmark_data_smithi188_113957_object52487:head# 2024-04-23T16:33:57.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2f7e49:::benchmark_data_smithi188_113957_object86142:head# 2024-04-23T16:33:57.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2fac2c:::benchmark_data_smithi188_113957_object28718:head# 2024-04-23T16:33:57.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d2fc3ef:::benchmark_data_smithi188_113957_object133067:head# 2024-04-23T16:33:57.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d304a13:::benchmark_data_smithi188_113957_object141110:head# 2024-04-23T16:33:57.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d30ea5b:::benchmark_data_smithi188_113957_object125684:head# 2024-04-23T16:33:57.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d332d03:::benchmark_data_smithi188_113957_object25847:head# 2024-04-23T16:33:57.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d33c80f:::benchmark_data_smithi188_113957_object95432:head# 2024-04-23T16:33:57.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d344480:::benchmark_data_smithi188_113957_object5189:head# 2024-04-23T16:33:57.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d346344:::benchmark_data_smithi188_113957_object4077:head# 2024-04-23T16:33:57.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d355e25:::benchmark_data_smithi188_113957_object17742:head# 2024-04-23T16:33:57.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d35c0d4:::benchmark_data_smithi188_113957_object150817:head# 2024-04-23T16:33:57.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d35d05e:::benchmark_data_smithi188_113957_object120964:head# 2024-04-23T16:33:57.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d35d84d:::benchmark_data_smithi188_113957_object70408:head# 2024-04-23T16:33:57.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d364c65:::benchmark_data_smithi188_113957_object139555:head# 2024-04-23T16:33:57.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d364faa:::benchmark_data_smithi188_113957_object9438:head# 2024-04-23T16:33:57.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d3663a7:::benchmark_data_smithi188_113957_object58520:head# 2024-04-23T16:33:57.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d36c7bf:::benchmark_data_smithi188_113957_object122466:head# 2024-04-23T16:33:57.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d370495:::benchmark_data_smithi188_113957_object94425:head# 2024-04-23T16:33:57.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d370875:::benchmark_data_smithi188_113957_object79606:head# 2024-04-23T16:33:57.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d382b81:::benchmark_data_smithi188_113957_object27449:head# 2024-04-23T16:33:57.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d38d571:::benchmark_data_smithi188_113957_object94690:head# 2024-04-23T16:33:57.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d38de99:::benchmark_data_smithi188_113957_object101446:head# 2024-04-23T16:33:57.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d38fabd:::benchmark_data_smithi188_113957_object22147:head# 2024-04-23T16:33:57.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d3954fc:::benchmark_data_smithi188_113957_object31634:head# 2024-04-23T16:33:57.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d39ac37:::benchmark_data_smithi188_113957_object84782:head# 2024-04-23T16:33:57.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d39df2f:::benchmark_data_smithi188_113957_object143952:head# 2024-04-23T16:33:57.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d3a186f:::benchmark_data_smithi188_113957_object31677:head# 2024-04-23T16:33:57.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d3a1da4:::benchmark_data_smithi188_113957_object25915:head# 2024-04-23T16:33:57.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d3a78ad:::benchmark_data_smithi188_113957_object135657:head# 2024-04-23T16:33:57.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d3ac931:::benchmark_data_smithi188_113957_object84183:head# 2024-04-23T16:33:57.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d3aef1f:::benchmark_data_smithi188_113957_object112709:head# 2024-04-23T16:33:57.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d3b2887:::benchmark_data_smithi188_113957_object48733:head# 2024-04-23T16:33:57.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d3b6b16:::benchmark_data_smithi188_113957_object32966:head# 2024-04-23T16:33:57.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d3c23a5:::benchmark_data_smithi188_113957_object14555:head# 2024-04-23T16:33:57.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d3c846b:::benchmark_data_smithi188_113957_object134414:head# 2024-04-23T16:33:57.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d3d0b27:::benchmark_data_smithi188_113957_object39095:head# 2024-04-23T16:33:57.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d3d1d23:::benchmark_data_smithi188_113957_object28479:head# 2024-04-23T16:33:57.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d3d769e:::benchmark_data_smithi188_113957_object77312:head# 2024-04-23T16:33:57.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d3d8b34:::benchmark_data_smithi188_113957_object33715:head# 2024-04-23T16:33:57.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d3e2ecf:::benchmark_data_smithi188_113957_object33900:head# 2024-04-23T16:33:57.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d3e2fb3:::benchmark_data_smithi188_113957_object151034:head# 2024-04-23T16:33:57.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d3e9f6a:::benchmark_data_smithi188_113957_object52258:head# 2024-04-23T16:33:57.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d407098:::benchmark_data_smithi188_113957_object45224:head# 2024-04-23T16:33:57.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d40819f:::benchmark_data_smithi188_113957_object101994:head# 2024-04-23T16:33:57.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d40e4fe:::benchmark_data_smithi188_113957_object109323:head# 2024-04-23T16:33:57.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d41a595:::benchmark_data_smithi188_113957_object17101:head# 2024-04-23T16:33:57.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d41ef5e:::benchmark_data_smithi188_113957_object3217:head# 2024-04-23T16:33:57.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d41fb1f:::benchmark_data_smithi188_113957_object141921:head# 2024-04-23T16:33:57.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d423173:::benchmark_data_smithi188_113957_object136816:head# 2024-04-23T16:33:57.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d424946:::benchmark_data_smithi188_113957_object80061:head# 2024-04-23T16:33:57.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d436ea8:::benchmark_data_smithi188_113957_object94568:head# 2024-04-23T16:33:57.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d43a393:::benchmark_data_smithi188_113957_object3325:head# 2024-04-23T16:33:57.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d444ce8:::benchmark_data_smithi188_113957_object43306:head# 2024-04-23T16:33:57.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d445f71:::benchmark_data_smithi188_113957_object127303:head# 2024-04-23T16:33:57.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d449328:::benchmark_data_smithi188_113957_object58528:head# 2024-04-23T16:33:57.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d44c1a2:::benchmark_data_smithi188_113957_object128670:head# 2024-04-23T16:33:57.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d4534f3:::benchmark_data_smithi188_113957_object131230:head# 2024-04-23T16:33:57.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d4576ed:::benchmark_data_smithi188_113957_object4411:head# 2024-04-23T16:33:57.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d4613cf:::benchmark_data_smithi188_113957_object59070:head# 2024-04-23T16:33:57.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d462df8:::benchmark_data_smithi188_113957_object5426:head# 2024-04-23T16:33:57.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d471b18:::benchmark_data_smithi188_113957_object40800:head# 2024-04-23T16:33:57.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d474cf5:::benchmark_data_smithi188_113957_object136315:head# 2024-04-23T16:33:57.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d480b98:::benchmark_data_smithi188_113957_object111984:head# 2024-04-23T16:33:57.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d494841:::benchmark_data_smithi188_113957_object12737:head# 2024-04-23T16:33:57.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d497c15:::benchmark_data_smithi188_113957_object93050:head# 2024-04-23T16:33:57.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d49c803:::benchmark_data_smithi188_113957_object84853:head# 2024-04-23T16:33:57.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d4a0958:::benchmark_data_smithi188_113957_object42540:head# 2024-04-23T16:33:57.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d4a468b:::benchmark_data_smithi188_113957_object153087:head# 2024-04-23T16:33:57.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d4b2388:::benchmark_data_smithi188_113957_object10092:head# 2024-04-23T16:33:57.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d4bc339:::benchmark_data_smithi188_113957_object36776:head# 2024-04-23T16:33:57.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d4c9dfe:::benchmark_data_smithi188_113957_object87340:head# 2024-04-23T16:33:57.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d4cf2e7:::benchmark_data_smithi188_113957_object83305:head# 2024-04-23T16:33:57.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d4da93c:::benchmark_data_smithi188_113957_object120840:head# 2024-04-23T16:33:57.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d4e8ba1:::benchmark_data_smithi188_113957_object94037:head# 2024-04-23T16:33:57.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d4eafbb:::benchmark_data_smithi188_113957_object14078:head# 2024-04-23T16:33:57.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d4ec61e:::benchmark_data_smithi188_113957_object153099:head# 2024-04-23T16:33:57.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d4f20b5:::benchmark_data_smithi188_113957_object67858:head# 2024-04-23T16:33:57.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d4f44c7:::benchmark_data_smithi188_113957_object12835:head# 2024-04-23T16:33:57.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d4fe694:::benchmark_data_smithi188_113957_object76965:head# 2024-04-23T16:33:57.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5021b6:::benchmark_data_smithi188_113957_object87231:head# 2024-04-23T16:33:57.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5046c1:::benchmark_data_smithi188_113957_object146733:head# 2024-04-23T16:33:57.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d504a5a:::benchmark_data_smithi188_113957_object103238:head# 2024-04-23T16:33:57.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d50cd48:::benchmark_data_smithi188_113957_object128826:head# 2024-04-23T16:33:57.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d515387:::benchmark_data_smithi188_113957_object91152:head# 2024-04-23T16:33:57.218 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d51678f:::benchmark_data_smithi188_113957_object39227:head# 2024-04-23T16:33:57.218 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d51ec3b:::benchmark_data_smithi188_113957_object73608:head# 2024-04-23T16:33:57.218 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d521d50:::benchmark_data_smithi188_113957_object29121:head# 2024-04-23T16:33:57.218 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d524639:::benchmark_data_smithi188_113957_object28637:head# 2024-04-23T16:33:57.218 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d52abf7:::benchmark_data_smithi188_113957_object98223:head# 2024-04-23T16:33:57.218 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d52ca8f:::benchmark_data_smithi188_113957_object148231:head# 2024-04-23T16:33:57.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d52f4ba:::benchmark_data_smithi188_113957_object70283:head# 2024-04-23T16:33:57.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5306cc:::benchmark_data_smithi188_113957_object73121:head# 2024-04-23T16:33:57.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5325ce:::benchmark_data_smithi188_113957_object63340:head# 2024-04-23T16:33:57.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d534847:::benchmark_data_smithi188_113957_object98688:head# 2024-04-23T16:33:57.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d539157:::benchmark_data_smithi188_113957_object25776:head# 2024-04-23T16:33:57.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d53a2c8:::benchmark_data_smithi188_113957_object24502:head# 2024-04-23T16:33:57.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d542b38:::benchmark_data_smithi188_113957_object151455:head# 2024-04-23T16:33:57.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d555e22:::benchmark_data_smithi188_113957_object111674:head# 2024-04-23T16:33:57.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d557b47:::benchmark_data_smithi188_113957_object146173:head# 2024-04-23T16:33:57.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d55b54a:::benchmark_data_smithi188_113957_object136922:head# 2024-04-23T16:33:57.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d55faa8:::benchmark_data_smithi188_113957_object10152:head# 2024-04-23T16:33:57.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d560738:::benchmark_data_smithi188_113957_object29712:head# 2024-04-23T16:33:57.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d578f36:::benchmark_data_smithi188_113957_object3889:head# 2024-04-23T16:33:57.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d57acb9:::benchmark_data_smithi188_113957_object149734:head# 2024-04-23T16:33:57.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d581beb:::benchmark_data_smithi188_113957_object112184:head# 2024-04-23T16:33:57.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d587ffe:::benchmark_data_smithi188_113957_object112869:head# 2024-04-23T16:33:57.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5a729f:::benchmark_data_smithi188_113957_object6427:head# 2024-04-23T16:33:57.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5ad044:::benchmark_data_smithi188_113957_object53457:head# 2024-04-23T16:33:57.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5b6e24:::benchmark_data_smithi188_113957_object63140:head# 2024-04-23T16:33:57.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5bc901:::benchmark_data_smithi188_113957_object63688:head# 2024-04-23T16:33:57.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5cc158:::benchmark_data_smithi188_113957_object57105:head# 2024-04-23T16:33:57.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5cc361:::benchmark_data_smithi188_113957_object115231:head# 2024-04-23T16:33:57.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5cf744:::benchmark_data_smithi188_113957_object41505:head# 2024-04-23T16:33:57.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5d264a:::benchmark_data_smithi188_113957_object64737:head# 2024-04-23T16:33:57.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5d2d7c:::benchmark_data_smithi188_113957_object81309:head# 2024-04-23T16:33:57.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5d377e:::benchmark_data_smithi188_113957_object38419:head# 2024-04-23T16:33:57.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5d5e72:::benchmark_data_smithi188_113957_object111639:head# 2024-04-23T16:33:57.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5dd8e5:::benchmark_data_smithi188_113957_object109135:head# 2024-04-23T16:33:57.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5de0a3:::benchmark_data_smithi188_113957_object84743:head# 2024-04-23T16:33:57.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5e2a89:::benchmark_data_smithi188_113957_object16218:head# 2024-04-23T16:33:57.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5e2dc4:::benchmark_data_smithi188_113957_object144140:head# 2024-04-23T16:33:57.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5e91ab:::benchmark_data_smithi188_113957_object91235:head# 2024-04-23T16:33:57.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5ed4f9:::benchmark_data_smithi188_113957_object12112:head# 2024-04-23T16:33:57.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5f92fd:::benchmark_data_smithi188_113957_object72068:head# 2024-04-23T16:33:57.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d5fc895:::benchmark_data_smithi188_113957_object86494:head# 2024-04-23T16:33:57.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d60a62a:::benchmark_data_smithi188_113957_object93081:head# 2024-04-23T16:33:57.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d60c6df:::benchmark_data_smithi188_113957_object102501:head# 2024-04-23T16:33:57.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d60efc3:::benchmark_data_smithi188_113957_object111171:head# 2024-04-23T16:33:57.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d61cf86:::benchmark_data_smithi188_113957_object55552:head# 2024-04-23T16:33:57.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d632055:::benchmark_data_smithi188_113957_object142605:head# 2024-04-23T16:33:57.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d63ab68:::benchmark_data_smithi188_113957_object58131:head# 2024-04-23T16:33:57.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d64af3d:::benchmark_data_smithi188_113957_object129745:head# 2024-04-23T16:33:57.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d64f54a:::benchmark_data_smithi188_113957_object82407:head# 2024-04-23T16:33:57.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d650db2:::benchmark_data_smithi188_113957_object61278:head# 2024-04-23T16:33:57.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d654722:::benchmark_data_smithi188_113957_object142905:head# 2024-04-23T16:33:57.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d676e8a:::benchmark_data_smithi188_113957_object97402:head# 2024-04-23T16:33:57.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d67b99a:::benchmark_data_smithi188_113957_object138112:head# 2024-04-23T16:33:57.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d6868a9:::benchmark_data_smithi188_113957_object149185:head# 2024-04-23T16:33:57.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d692464:::benchmark_data_smithi188_113957_object62959:head# 2024-04-23T16:33:57.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d6931a0:::benchmark_data_smithi188_113957_object110563:head# 2024-04-23T16:33:57.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d697840:::benchmark_data_smithi188_113957_object15117:head# 2024-04-23T16:33:57.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d698d42:::benchmark_data_smithi188_113957_object115372:head# 2024-04-23T16:33:57.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d699123:::benchmark_data_smithi188_113957_object101783:head# 2024-04-23T16:33:57.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d6be1a9:::benchmark_data_smithi188_113957_object53944:head# 2024-04-23T16:33:57.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d6be32f:::benchmark_data_smithi188_113957_object143695:head# 2024-04-23T16:33:57.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d6c5978:::benchmark_data_smithi188_113957_object170:head# 2024-04-23T16:33:57.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d6d7b5f:::benchmark_data_smithi188_113957_object37217:head# 2024-04-23T16:33:57.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d6da432:::benchmark_data_smithi188_113957_object51509:head# 2024-04-23T16:33:57.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d6db002:::benchmark_data_smithi188_113957_object117900:head# 2024-04-23T16:33:57.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d6e2de4:::benchmark_data_smithi188_113957_object32653:head# 2024-04-23T16:33:57.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d6e6b1e:::benchmark_data_smithi188_113957_object82399:head# 2024-04-23T16:33:57.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d6e7ebb:::benchmark_data_smithi188_113957_object133593:head# 2024-04-23T16:33:57.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d6eef78:::benchmark_data_smithi188_113957_object58320:head# 2024-04-23T16:33:57.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d6ef39f:::benchmark_data_smithi188_113957_object143172:head# 2024-04-23T16:33:57.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d6f0149:::benchmark_data_smithi188_113957_object112304:head# 2024-04-23T16:33:57.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d6f372a:::benchmark_data_smithi188_113957_object32596:head# 2024-04-23T16:33:57.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d702c6e:::benchmark_data_smithi188_113957_object76562:head# 2024-04-23T16:33:57.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d704fe7:::benchmark_data_smithi188_113957_object153464:head# 2024-04-23T16:33:57.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d70d9e8:::benchmark_data_smithi188_113957_object12327:head# 2024-04-23T16:33:57.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d70e135:::benchmark_data_smithi188_113957_object142651:head# 2024-04-23T16:33:57.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d70f62a:::benchmark_data_smithi188_113957_object153363:head# 2024-04-23T16:33:57.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7100fd:::benchmark_data_smithi188_113957_object18365:head# 2024-04-23T16:33:57.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d711cb0:::benchmark_data_smithi188_113957_object61738:head# 2024-04-23T16:33:57.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7145d1:::benchmark_data_smithi188_113957_object73458:head# 2024-04-23T16:33:57.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d718d5b:::benchmark_data_smithi188_113957_object94435:head# 2024-04-23T16:33:57.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d719338:::benchmark_data_smithi188_113957_object16051:head# 2024-04-23T16:33:57.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d71e63c:::benchmark_data_smithi188_113957_object103167:head# 2024-04-23T16:33:57.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d725b00:::benchmark_data_smithi188_113957_object151594:head# 2024-04-23T16:33:57.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d73b235:::benchmark_data_smithi188_113957_object3603:head# 2024-04-23T16:33:57.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d73dcd4:::benchmark_data_smithi188_113957_object130297:head# 2024-04-23T16:33:57.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d73e659:::benchmark_data_smithi188_113957_object147432:head# 2024-04-23T16:33:57.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d747176:::benchmark_data_smithi188_113957_object7060:head# 2024-04-23T16:33:57.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d74dbcc:::benchmark_data_smithi188_113957_object153367:head# 2024-04-23T16:33:57.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d751dbe:::benchmark_data_smithi188_113957_object132725:head# 2024-04-23T16:33:57.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7598de:::benchmark_data_smithi188_113957_object93865:head# 2024-04-23T16:33:57.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d768135:::benchmark_data_smithi188_113957_object124505:head# 2024-04-23T16:33:57.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d771724:::benchmark_data_smithi188_113957_object125221:head# 2024-04-23T16:33:57.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7836d6:::benchmark_data_smithi188_113957_object21650:head# 2024-04-23T16:33:57.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d78ccca:::benchmark_data_smithi188_113957_object4245:head# 2024-04-23T16:33:57.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d79115f:::benchmark_data_smithi188_113957_object73937:head# 2024-04-23T16:33:57.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7969d2:::benchmark_data_smithi188_113957_object41597:head# 2024-04-23T16:33:57.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d79ae4b:::benchmark_data_smithi188_113957_object153104:head# 2024-04-23T16:33:57.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7a6e94:::benchmark_data_smithi188_113957_object85044:head# 2024-04-23T16:33:57.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7adb0f:::benchmark_data_smithi188_113957_object140742:head# 2024-04-23T16:33:57.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7b7296:::benchmark_data_smithi188_113957_object89133:head# 2024-04-23T16:33:57.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7cfd29:::benchmark_data_smithi188_113957_object133178:head# 2024-04-23T16:33:57.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7d193c:::benchmark_data_smithi188_113957_object15438:head# 2024-04-23T16:33:57.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7d8ad2:::benchmark_data_smithi188_113957_object32275:head# 2024-04-23T16:33:57.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7e19fc:::benchmark_data_smithi188_113957_object121493:head# 2024-04-23T16:33:57.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7e1b29:::benchmark_data_smithi188_113957_object80924:head# 2024-04-23T16:33:57.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7e3ff5:::benchmark_data_smithi188_113957_object121378:head# 2024-04-23T16:33:57.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7e89f2:::benchmark_data_smithi188_113957_object148868:head# 2024-04-23T16:33:57.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7f1b17:::benchmark_data_smithi188_113957_object61697:head# 2024-04-23T16:33:57.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7f23a2:::benchmark_data_smithi188_113957_object21103:head# 2024-04-23T16:33:57.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7f59ce:::benchmark_data_smithi188_113957_object59132:head# 2024-04-23T16:33:57.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7faf63:::benchmark_data_smithi188_113957_object54503:head# 2024-04-23T16:33:57.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7fbfda:::benchmark_data_smithi188_113957_object119848:head# 2024-04-23T16:33:57.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7fcdb1:::benchmark_data_smithi188_113957_object142701:head# 2024-04-23T16:33:57.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7fd777:::benchmark_data_smithi188_113957_object25464:head# 2024-04-23T16:33:57.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d7ffeb5:::benchmark_data_smithi188_113957_object61519:head# 2024-04-23T16:33:57.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d8030a2:::benchmark_data_smithi188_113957_object21637:head# 2024-04-23T16:33:57.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d80c7dc:::benchmark_data_smithi188_113957_object75044:head# 2024-04-23T16:33:57.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d81c032:::benchmark_data_smithi188_113957_object102631:head# 2024-04-23T16:33:57.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d81f290:::benchmark_data_smithi188_113957_object133297:head# 2024-04-23T16:33:57.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d82c4ad:::benchmark_data_smithi188_113957_object44820:head# 2024-04-23T16:33:57.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d8424d0:::benchmark_data_smithi188_113957_object93821:head# 2024-04-23T16:33:57.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d849842:::benchmark_data_smithi188_113957_object137565:head# 2024-04-23T16:33:57.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d84c985:::benchmark_data_smithi188_113957_object120468:head# 2024-04-23T16:33:57.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d84cc72:::benchmark_data_smithi188_113957_object85122:head# 2024-04-23T16:33:57.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d8506c2:::benchmark_data_smithi188_113957_object128887:head# 2024-04-23T16:33:57.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d85b25f:::benchmark_data_smithi188_113957_object51332:head# 2024-04-23T16:33:57.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d85e29a:::benchmark_data_smithi188_113957_object105396:head# 2024-04-23T16:33:57.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d86c41e:::benchmark_data_smithi188_113957_object132643:head# 2024-04-23T16:33:57.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d87cca3:::benchmark_data_smithi188_113957_object139265:head# 2024-04-23T16:33:57.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d889ed1:::benchmark_data_smithi188_113957_object25948:head# 2024-04-23T16:33:57.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d89c2de:::benchmark_data_smithi188_113957_object114456:head# 2024-04-23T16:33:57.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d8a9a34:::benchmark_data_smithi188_113957_object15683:head# 2024-04-23T16:33:57.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d8ac281:::benchmark_data_smithi188_113957_object70373:head# 2024-04-23T16:33:57.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d8ae72a:::benchmark_data_smithi188_113957_object127337:head# 2024-04-23T16:33:57.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d8baad1:::benchmark_data_smithi188_113957_object118092:head# 2024-04-23T16:33:57.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d8bce5e:::benchmark_data_smithi188_113957_object114582:head# 2024-04-23T16:33:57.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d8bfdc6:::benchmark_data_smithi188_113957_object52011:head# 2024-04-23T16:33:57.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d8d9ec3:::benchmark_data_smithi188_113957_object69412:head# 2024-04-23T16:33:57.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d8dc670:::benchmark_data_smithi188_113957_object79624:head# 2024-04-23T16:33:57.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d8e5a8a:::benchmark_data_smithi188_113957_object8597:head# 2024-04-23T16:33:57.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d8e6441:::benchmark_data_smithi188_113957_object80012:head# 2024-04-23T16:33:57.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d8ea9ca:::benchmark_data_smithi188_113957_object124502:head# 2024-04-23T16:33:57.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d8eb546:::benchmark_data_smithi188_113957_object81073:head# 2024-04-23T16:33:57.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d8ee46f:::benchmark_data_smithi188_113957_object42904:head# 2024-04-23T16:33:57.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d8f65dd:::benchmark_data_smithi188_113957_object77020:head# 2024-04-23T16:33:57.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d8f687e:::benchmark_data_smithi188_113957_object10499:head# 2024-04-23T16:33:57.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d8ffb7b:::benchmark_data_smithi188_113957_object127322:head# 2024-04-23T16:33:57.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d90447f:::benchmark_data_smithi188_113957_object62310:head# 2024-04-23T16:33:57.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d91e7e5:::benchmark_data_smithi188_113957_object140514:head# 2024-04-23T16:33:57.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d9233b1:::benchmark_data_smithi188_113957_object18407:head# 2024-04-23T16:33:57.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d925b6b:::benchmark_data_smithi188_113957_object37434:head# 2024-04-23T16:33:57.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d926982:::benchmark_data_smithi188_113957_object119140:head# 2024-04-23T16:33:57.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d928545:::benchmark_data_smithi188_113957_object30262:head# 2024-04-23T16:33:57.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d92d6a5:::benchmark_data_smithi188_113957_object20862:head# 2024-04-23T16:33:57.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d92fc93:::benchmark_data_smithi188_113957_object89201:head# 2024-04-23T16:33:57.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d940feb:::benchmark_data_smithi188_113957_object125731:head# 2024-04-23T16:33:57.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d94cc73:::benchmark_data_smithi188_113957_object10403:head# 2024-04-23T16:33:57.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d95626c:::benchmark_data_smithi188_113957_object44719:head# 2024-04-23T16:33:57.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d957aa7:::benchmark_data_smithi188_113957_object139358:head# 2024-04-23T16:33:57.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d9589fc:::benchmark_data_smithi188_113957_object137392:head# 2024-04-23T16:33:57.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d95c84d:::benchmark_data_smithi188_113957_object745:head# 2024-04-23T16:33:57.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d967502:::benchmark_data_smithi188_113957_object8078:head# 2024-04-23T16:33:57.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d96786e:::benchmark_data_smithi188_113957_object76779:head# 2024-04-23T16:33:57.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d96a011:::benchmark_data_smithi188_113957_object127559:head# 2024-04-23T16:33:57.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d96ca81:::benchmark_data_smithi188_113957_object136992:head# 2024-04-23T16:33:57.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d96ffd4:::benchmark_data_smithi188_113957_object49578:head# 2024-04-23T16:33:57.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d9853db:::benchmark_data_smithi188_113957_object114859:head# 2024-04-23T16:33:57.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d99e8c5:::benchmark_data_smithi188_113957_object138065:head# 2024-04-23T16:33:57.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d99e979:::benchmark_data_smithi188_113957_object70113:head# 2024-04-23T16:33:57.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d9a978f:::benchmark_data_smithi188_113957_object139643:head# 2024-04-23T16:33:57.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d9ab40d:::benchmark_data_smithi188_113957_object72309:head# 2024-04-23T16:33:57.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d9b0ab2:::benchmark_data_smithi188_113957_object86724:head# 2024-04-23T16:33:57.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d9b4044:::benchmark_data_smithi188_113957_object31413:head# 2024-04-23T16:33:57.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d9bb8c5:::benchmark_data_smithi188_113957_object108762:head# 2024-04-23T16:33:57.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d9d6bc1:::benchmark_data_smithi188_113957_object19335:head# 2024-04-23T16:33:57.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d9d864c:::benchmark_data_smithi188_113957_object124114:head# 2024-04-23T16:33:57.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d9da76c:::benchmark_data_smithi188_113957_object80410:head# 2024-04-23T16:33:57.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d9df807:::benchmark_data_smithi188_113957_object58627:head# 2024-04-23T16:33:57.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d9df8c0:::benchmark_data_smithi188_113957_object137408:head# 2024-04-23T16:33:57.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d9eb6e9:::benchmark_data_smithi188_113957_object120047:head# 2024-04-23T16:33:57.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5d9f4d82:::benchmark_data_smithi188_113957_object81185:head# 2024-04-23T16:33:57.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5da030ac:::benchmark_data_smithi188_113957_object26109:head# 2024-04-23T16:33:57.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5da0c3fa:::benchmark_data_smithi188_113957_object149379:head# 2024-04-23T16:33:57.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5da16bec:::benchmark_data_smithi188_113957_object66087:head# 2024-04-23T16:33:57.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5da1ce47:::benchmark_data_smithi188_113957_object121628:head# 2024-04-23T16:33:57.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5da2866b:::benchmark_data_smithi188_113957_object136214:head# 2024-04-23T16:33:57.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5da2ddf1:::benchmark_data_smithi188_113957_object90724:head# 2024-04-23T16:33:57.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5da40e7b:::benchmark_data_smithi188_113957_object16554:head# 2024-04-23T16:33:57.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5da416f5:::benchmark_data_smithi188_113957_object65910:head# 2024-04-23T16:33:57.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5da4d8a4:::benchmark_data_smithi188_113957_object142361:head# 2024-04-23T16:33:57.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5da573c5:::benchmark_data_smithi188_113957_object128115:head# 2024-04-23T16:33:57.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5da5b203:::benchmark_data_smithi188_113957_object54750:head# 2024-04-23T16:33:57.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5da60bd8:::benchmark_data_smithi188_113957_object58304:head# 2024-04-23T16:33:57.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5da61b86:::benchmark_data_smithi188_113957_object69357:head# 2024-04-23T16:33:57.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5da6e766:::benchmark_data_smithi188_113957_object66702:head# 2024-04-23T16:33:57.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5da76aa9:::benchmark_data_smithi188_113957_object83950:head# 2024-04-23T16:33:57.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5da84a59:::benchmark_data_smithi188_113957_object46534:head# 2024-04-23T16:33:57.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5da99e4a:::benchmark_data_smithi188_113957_object65737:head# 2024-04-23T16:33:57.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5daa98ed:::benchmark_data_smithi188_113957_object120595:head# 2024-04-23T16:33:57.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dac2a36:::benchmark_data_smithi188_113957_object147794:head# 2024-04-23T16:33:57.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dac48c1:::benchmark_data_smithi188_113957_object153214:head# 2024-04-23T16:33:57.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dac9729:::benchmark_data_smithi188_113957_object150930:head# 2024-04-23T16:33:57.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dacddeb:::benchmark_data_smithi188_113957_object126289:head# 2024-04-23T16:33:57.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dad065e:::benchmark_data_smithi188_113957_object7257:head# 2024-04-23T16:33:57.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dad1e19:::benchmark_data_smithi188_113957_object58293:head# 2024-04-23T16:33:57.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dad2477:::benchmark_data_smithi188_113957_object111902:head# 2024-04-23T16:33:57.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dad8263:::benchmark_data_smithi188_113957_object101285:head# 2024-04-23T16:33:57.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dae3459:::benchmark_data_smithi188_113957_object59148:head# 2024-04-23T16:33:57.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dae6b15:::benchmark_data_smithi188_113957_object46380:head# 2024-04-23T16:33:57.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5daed293:::benchmark_data_smithi188_113957_object17935:head# 2024-04-23T16:33:57.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5daef02e:::benchmark_data_smithi188_113957_object60842:head# 2024-04-23T16:33:57.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5daf4ce7:::benchmark_data_smithi188_113957_object10376:head# 2024-04-23T16:33:57.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dafa02f:::benchmark_data_smithi188_113957_object93393:head# 2024-04-23T16:33:57.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dafe98a:::benchmark_data_smithi188_113957_object37881:head# 2024-04-23T16:33:57.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db03a7d:::benchmark_data_smithi188_113957_object42925:head# 2024-04-23T16:33:57.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db0406c:::benchmark_data_smithi188_113957_object133787:head# 2024-04-23T16:33:57.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db0765f:::benchmark_data_smithi188_113957_object125439:head# 2024-04-23T16:33:57.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db0bb03:::benchmark_data_smithi188_113957_object120244:head# 2024-04-23T16:33:57.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db11c68:::benchmark_data_smithi188_113957_object135016:head# 2024-04-23T16:33:57.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db15966:::benchmark_data_smithi188_113957_object79926:head# 2024-04-23T16:33:57.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db1a5e7:::benchmark_data_smithi188_113957_object147057:head# 2024-04-23T16:33:57.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db1fbc7:::benchmark_data_smithi188_113957_object95144:head# 2024-04-23T16:33:57.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db2bb8e:::benchmark_data_smithi188_113957_object57598:head# 2024-04-23T16:33:57.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db322e1:::benchmark_data_smithi188_113957_object121:head# 2024-04-23T16:33:57.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db49e3b:::benchmark_data_smithi188_113957_object42093:head# 2024-04-23T16:33:57.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db51416:::benchmark_data_smithi188_113957_object120340:head# 2024-04-23T16:33:57.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db5c3e1:::benchmark_data_smithi188_113957_object151434:head# 2024-04-23T16:33:57.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db5e447:::benchmark_data_smithi188_113957_object1260:head# 2024-04-23T16:33:57.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db5f251:::benchmark_data_smithi188_113957_object13248:head# 2024-04-23T16:33:57.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db6bbe6:::benchmark_data_smithi188_113957_object124440:head# 2024-04-23T16:33:57.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db71991:::benchmark_data_smithi188_113957_object81610:head# 2024-04-23T16:33:57.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db7b683:::benchmark_data_smithi188_113957_object22736:head# 2024-04-23T16:33:57.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db7d7ac:::benchmark_data_smithi188_113957_object69428:head# 2024-04-23T16:33:57.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db9199f:::benchmark_data_smithi188_113957_object37755:head# 2024-04-23T16:33:57.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5db98383:::benchmark_data_smithi188_113957_object27698:head# 2024-04-23T16:33:57.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dba6c63:::benchmark_data_smithi188_113957_object72323:head# 2024-04-23T16:33:57.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dbb3961:::benchmark_data_smithi188_113957_object82372:head# 2024-04-23T16:33:57.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dbb7ee2:::benchmark_data_smithi188_113957_object108836:head# 2024-04-23T16:33:57.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dbbc8d7:::benchmark_data_smithi188_113957_object29946:head# 2024-04-23T16:33:57.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dbcd787:::benchmark_data_smithi188_113957_object42710:head# 2024-04-23T16:33:57.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dbce6a0:::benchmark_data_smithi188_113957_object74727:head# 2024-04-23T16:33:57.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dbcfb41:::benchmark_data_smithi188_113957_object117842:head# 2024-04-23T16:33:57.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dbd6dd5:::benchmark_data_smithi188_113957_object53444:head# 2024-04-23T16:33:57.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dbe1bb7:::benchmark_data_smithi188_113957_object60343:head# 2024-04-23T16:33:57.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dbe692a:::benchmark_data_smithi188_113957_object81570:head# 2024-04-23T16:33:57.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dbe6fa1:::benchmark_data_smithi188_113957_object93801:head# 2024-04-23T16:33:57.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dbe96b0:::benchmark_data_smithi188_113957_object1170:head# 2024-04-23T16:33:57.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dbeb10e:::benchmark_data_smithi188_113957_object102102:head# 2024-04-23T16:33:57.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dbec2d4:::benchmark_data_smithi188_113957_object20529:head# 2024-04-23T16:33:57.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dbf50c7:::benchmark_data_smithi188_113957_object48322:head# 2024-04-23T16:33:57.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dbf899b:::benchmark_data_smithi188_113957_object70138:head# 2024-04-23T16:33:57.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dbfde87:::benchmark_data_smithi188_113957_object108429:head# 2024-04-23T16:33:57.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc0427a:::benchmark_data_smithi188_113957_object102263:head# 2024-04-23T16:33:57.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc10b17:::benchmark_data_smithi188_113957_object113646:head# 2024-04-23T16:33:57.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc1525f:::benchmark_data_smithi188_113957_object14755:head# 2024-04-23T16:33:57.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc23590:::benchmark_data_smithi188_113957_object89243:head# 2024-04-23T16:33:57.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc27043:::benchmark_data_smithi188_113957_object72839:head# 2024-04-23T16:33:57.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc284ef:::benchmark_data_smithi188_113957_object71781:head# 2024-04-23T16:33:57.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc31c73:::benchmark_data_smithi188_113957_object4504:head# 2024-04-23T16:33:57.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc342c4:::benchmark_data_smithi188_113957_object13561:head# 2024-04-23T16:33:57.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc34c70:::benchmark_data_smithi188_113957_object141451:head# 2024-04-23T16:33:57.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc35fbf:::benchmark_data_smithi188_113957_object88006:head# 2024-04-23T16:33:57.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc3fd68:::benchmark_data_smithi188_113957_object11513:head# 2024-04-23T16:33:57.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc44b8f:::benchmark_data_smithi188_113957_object34873:head# 2024-04-23T16:33:57.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc473d2:::benchmark_data_smithi188_113957_object55148:head# 2024-04-23T16:33:57.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc63e73:::benchmark_data_smithi188_113957_object105617:head# 2024-04-23T16:33:57.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc67258:::benchmark_data_smithi188_113957_object86264:head# 2024-04-23T16:33:57.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc7a3fc:::benchmark_data_smithi188_113957_object147385:head# 2024-04-23T16:33:57.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc8011f:::benchmark_data_smithi188_113957_object101954:head# 2024-04-23T16:33:57.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc8359f:::benchmark_data_smithi188_113957_object50201:head# 2024-04-23T16:33:57.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc83d6a:::benchmark_data_smithi188_113957_object125521:head# 2024-04-23T16:33:57.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc8b30b:::benchmark_data_smithi188_113957_object27304:head# 2024-04-23T16:33:57.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc94fe3:::benchmark_data_smithi188_113957_object2147:head# 2024-04-23T16:33:57.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc96143:::benchmark_data_smithi188_113957_object80883:head# 2024-04-23T16:33:57.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc9db9d:::benchmark_data_smithi188_113957_object118265:head# 2024-04-23T16:33:57.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc9dbe4:::benchmark_data_smithi188_113957_object106856:head# 2024-04-23T16:33:57.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dc9e790:::benchmark_data_smithi188_113957_object96380:head# 2024-04-23T16:33:57.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dca3bf0:::benchmark_data_smithi188_113957_object64241:head# 2024-04-23T16:33:57.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dca4f71:::benchmark_data_smithi188_113957_object75076:head# 2024-04-23T16:33:57.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dca8fd9:::benchmark_data_smithi188_113957_object9336:head# 2024-04-23T16:33:57.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dcad90c:::benchmark_data_smithi188_113957_object69531:head# 2024-04-23T16:33:57.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dcb3f0e:::benchmark_data_smithi188_113957_object18251:head# 2024-04-23T16:33:57.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dcc83cb:::benchmark_data_smithi188_113957_object55907:head# 2024-04-23T16:33:57.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dcc91b5:::benchmark_data_smithi188_113957_object18469:head# 2024-04-23T16:33:57.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dcd3e1c:::benchmark_data_smithi188_113957_object70983:head# 2024-04-23T16:33:57.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dcd5525:::benchmark_data_smithi188_113957_object115658:head# 2024-04-23T16:33:57.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dcd936e:::benchmark_data_smithi188_113957_object77163:head# 2024-04-23T16:33:57.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dcdc495:::benchmark_data_smithi188_113957_object24285:head# 2024-04-23T16:33:57.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dcdeb93:::benchmark_data_smithi188_113957_object93988:head# 2024-04-23T16:33:57.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dcea8a5:::benchmark_data_smithi188_113957_object146824:head# 2024-04-23T16:33:57.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dcf1e4e:::benchmark_data_smithi188_113957_object25958:head# 2024-04-23T16:33:57.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd12937:::benchmark_data_smithi188_113957_object32311:head# 2024-04-23T16:33:57.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd1415b:::benchmark_data_smithi188_113957_object10079:head# 2024-04-23T16:33:57.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd23e54:::benchmark_data_smithi188_113957_object115177:head# 2024-04-23T16:33:57.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd2594c:::benchmark_data_smithi188_113957_object26629:head# 2024-04-23T16:33:57.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd2a0b4:::benchmark_data_smithi188_113957_object67968:head# 2024-04-23T16:33:57.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd32688:::benchmark_data_smithi188_113957_object54370:head# 2024-04-23T16:33:57.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd409e5:::benchmark_data_smithi188_113957_object134295:head# 2024-04-23T16:33:57.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd41e3e:::benchmark_data_smithi188_113957_object10704:head# 2024-04-23T16:33:57.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd4c8aa:::benchmark_data_smithi188_113957_object43922:head# 2024-04-23T16:33:57.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd5198a:::benchmark_data_smithi188_113957_object116728:head# 2024-04-23T16:33:57.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd61891:::benchmark_data_smithi188_113957_object132251:head# 2024-04-23T16:33:57.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd62782:::benchmark_data_smithi188_113957_object121312:head# 2024-04-23T16:33:57.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd62e66:::benchmark_data_smithi188_113957_object118733:head# 2024-04-23T16:33:57.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd7487f:::benchmark_data_smithi188_113957_object95124:head# 2024-04-23T16:33:57.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd74b46:::benchmark_data_smithi188_113957_object48737:head# 2024-04-23T16:33:57.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd8a813:::benchmark_data_smithi188_113957_object141980:head# 2024-04-23T16:33:57.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd93baa:::benchmark_data_smithi188_113957_object107347:head# 2024-04-23T16:33:57.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd9d92e:::benchmark_data_smithi188_113957_object78913:head# 2024-04-23T16:33:57.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd9ef79:::benchmark_data_smithi188_113957_object34770:head# 2024-04-23T16:33:57.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dd9f601:::benchmark_data_smithi188_113957_object67833:head# 2024-04-23T16:33:57.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dda4a7f:::benchmark_data_smithi188_113957_object12475:head# 2024-04-23T16:33:57.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dda83df:::benchmark_data_smithi188_113957_object40813:head# 2024-04-23T16:33:57.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ddad1a3:::benchmark_data_smithi188_113957_object119346:head# 2024-04-23T16:33:57.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ddb52ef:::benchmark_data_smithi188_113957_object87574:head# 2024-04-23T16:33:57.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ddbaa9e:::benchmark_data_smithi188_113957_object9171:head# 2024-04-23T16:33:57.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dddeddd:::benchmark_data_smithi188_113957_object12020:head# 2024-04-23T16:33:57.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dde2c93:::benchmark_data_smithi188_113957_object98181:head# 2024-04-23T16:33:57.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dde6f94:::benchmark_data_smithi188_113957_object124082:head# 2024-04-23T16:33:57.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ddee485:::benchmark_data_smithi188_113957_object111383:head# 2024-04-23T16:33:57.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ddf6c2d:::benchmark_data_smithi188_113957_object45010:head# 2024-04-23T16:33:57.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ddfa0d7:::benchmark_data_smithi188_113957_object95390:head# 2024-04-23T16:33:57.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ddfcb35:::benchmark_data_smithi188_113957_object114280:head# 2024-04-23T16:33:57.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de0025b:::benchmark_data_smithi188_113957_object74292:head# 2024-04-23T16:33:57.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de0730c:::benchmark_data_smithi188_113957_object31821:head# 2024-04-23T16:33:57.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de0f26c:::benchmark_data_smithi188_113957_object99189:head# 2024-04-23T16:33:57.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de11948:::benchmark_data_smithi188_113957_object50457:head# 2024-04-23T16:33:57.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de1f0b5:::benchmark_data_smithi188_113957_object99213:head# 2024-04-23T16:33:57.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de28ed9:::benchmark_data_smithi188_113957_object33539:head# 2024-04-23T16:33:57.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de3712c:::benchmark_data_smithi188_113957_object117618:head# 2024-04-23T16:33:57.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de39150:::benchmark_data_smithi188_113957_object32037:head# 2024-04-23T16:33:57.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de3b778:::benchmark_data_smithi188_113957_object51605:head# 2024-04-23T16:33:57.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de3cc99:::benchmark_data_smithi188_113957_object77372:head# 2024-04-23T16:33:57.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de4a570:::benchmark_data_smithi188_113957_object106831:head# 2024-04-23T16:33:57.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de4c3b8:::benchmark_data_smithi188_113957_object144801:head# 2024-04-23T16:33:57.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de4e183:::benchmark_data_smithi188_113957_object121284:head# 2024-04-23T16:33:57.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de56985:::benchmark_data_smithi188_113957_object110752:head# 2024-04-23T16:33:57.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de5c510:::benchmark_data_smithi188_113957_object6271:head# 2024-04-23T16:33:57.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de626fe:::benchmark_data_smithi188_113957_object32241:head# 2024-04-23T16:33:57.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de79a05:::benchmark_data_smithi188_113957_object15492:head# 2024-04-23T16:33:57.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de7d3d6:::benchmark_data_smithi188_113957_object124410:head# 2024-04-23T16:33:57.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de7f606:::benchmark_data_smithi188_113957_object92698:head# 2024-04-23T16:33:57.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de87fab:::benchmark_data_smithi188_113957_object114568:head# 2024-04-23T16:33:57.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de978c5:::benchmark_data_smithi188_113957_object30833:head# 2024-04-23T16:33:57.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de98f21:::benchmark_data_smithi188_113957_object106446:head# 2024-04-23T16:33:57.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de9bc22:::benchmark_data_smithi188_113957_object113492:head# 2024-04-23T16:33:57.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5de9c228:::benchmark_data_smithi188_113957_object110619:head# 2024-04-23T16:33:57.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dea4ab6:::benchmark_data_smithi188_113957_object96756:head# 2024-04-23T16:33:57.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5debc6d4:::benchmark_data_smithi188_113957_object83269:head# 2024-04-23T16:33:57.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5debe4bd:::benchmark_data_smithi188_113957_object129871:head# 2024-04-23T16:33:57.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dece465:::benchmark_data_smithi188_113957_object122495:head# 2024-04-23T16:33:57.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5decf61d:::benchmark_data_smithi188_113957_object134201:head# 2024-04-23T16:33:57.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ded16c2:::benchmark_data_smithi188_113957_object41986:head# 2024-04-23T16:33:57.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ded4e62:::benchmark_data_smithi188_113957_object113311:head# 2024-04-23T16:33:57.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ded6daf:::benchmark_data_smithi188_113957_object54024:head# 2024-04-23T16:33:57.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ded9fd8:::benchmark_data_smithi188_113957_object89224:head# 2024-04-23T16:33:57.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dedb6fc:::benchmark_data_smithi188_113957_object101401:head# 2024-04-23T16:33:57.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dedc03b:::benchmark_data_smithi188_113957_object114293:head# 2024-04-23T16:33:57.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dede062:::benchmark_data_smithi188_113957_object39006:head# 2024-04-23T16:33:57.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dede304:::benchmark_data_smithi188_113957_object108237:head# 2024-04-23T16:33:57.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dedfb70:::benchmark_data_smithi188_113957_object131046:head# 2024-04-23T16:33:57.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dee230b:::benchmark_data_smithi188_113957_object45784:head# 2024-04-23T16:33:57.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5deef475:::benchmark_data_smithi188_113957_object144129:head# 2024-04-23T16:33:57.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5def4b01:::benchmark_data_smithi188_113957_object81015:head# 2024-04-23T16:33:57.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5def71d2:::benchmark_data_smithi188_113957_object137526:head# 2024-04-23T16:33:57.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5deff469:::benchmark_data_smithi188_113957_object123183:head# 2024-04-23T16:33:57.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5df19cb9:::benchmark_data_smithi188_113957_object122732:head# 2024-04-23T16:33:57.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5df1f041:::benchmark_data_smithi188_113957_object92276:head# 2024-04-23T16:33:57.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5df266d3:::benchmark_data_smithi188_113957_object21571:head# 2024-04-23T16:33:57.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5df2dac6:::benchmark_data_smithi188_113957_object75124:head# 2024-04-23T16:33:57.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5df4f2cb:::benchmark_data_smithi188_113957_object10424:head# 2024-04-23T16:33:57.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5df68cd4:::benchmark_data_smithi188_113957_object66887:head# 2024-04-23T16:33:57.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5df68f50:::benchmark_data_smithi188_113957_object112439:head# 2024-04-23T16:33:57.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5df6e438:::benchmark_data_smithi188_113957_object62949:head# 2024-04-23T16:33:57.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5df729f5:::benchmark_data_smithi188_113957_object100853:head# 2024-04-23T16:33:57.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5df72e65:::benchmark_data_smithi188_113957_object140788:head# 2024-04-23T16:33:57.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5df793bc:::benchmark_data_smithi188_113957_object98901:head# 2024-04-23T16:33:57.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5df8b900:::benchmark_data_smithi188_113957_object149689:head# 2024-04-23T16:33:57.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5df93f07:::benchmark_data_smithi188_113957_object74385:head# 2024-04-23T16:33:57.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5df9f1ef:::benchmark_data_smithi188_113957_object75658:head# 2024-04-23T16:33:57.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dfb15f8:::benchmark_data_smithi188_113957_object27723:head# 2024-04-23T16:33:57.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dfb7b20:::benchmark_data_smithi188_113957_object11831:head# 2024-04-23T16:33:57.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dfb928c:::benchmark_data_smithi188_113957_object32313:head# 2024-04-23T16:33:57.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dfbbf13:::benchmark_data_smithi188_113957_object132036:head# 2024-04-23T16:33:57.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dfcc625:::benchmark_data_smithi188_113957_object115988:head# 2024-04-23T16:33:57.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dfcd964:::benchmark_data_smithi188_113957_object18523:head# 2024-04-23T16:33:57.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dfe5362:::benchmark_data_smithi188_113957_object5203:head# 2024-04-23T16:33:57.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dfe5d44:::benchmark_data_smithi188_113957_object27306:head# 2024-04-23T16:33:57.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5dfedfac:::benchmark_data_smithi188_113957_object4787:head# 2024-04-23T16:33:57.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e0024b0:::benchmark_data_smithi188_113957_object68296:head# 2024-04-23T16:33:57.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e009dbf:::benchmark_data_smithi188_113957_object122498:head# 2024-04-23T16:33:57.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e00d78b:::benchmark_data_smithi188_113957_object40976:head# 2024-04-23T16:33:57.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e0123c9:::benchmark_data_smithi188_113957_object72860:head# 2024-04-23T16:33:57.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e016778:::benchmark_data_smithi188_113957_object65354:head# 2024-04-23T16:33:57.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e0224b1:::benchmark_data_smithi188_113957_object37024:head# 2024-04-23T16:33:57.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e0238a2:::benchmark_data_smithi188_113957_object24565:head# 2024-04-23T16:33:57.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e024d6c:::benchmark_data_smithi188_113957_object116878:head# 2024-04-23T16:33:57.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e03736c:::benchmark_data_smithi188_113957_object25570:head# 2024-04-23T16:33:57.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e03925c:::benchmark_data_smithi188_113957_object31563:head# 2024-04-23T16:33:57.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e03a715:::benchmark_data_smithi188_113957_object137792:head# 2024-04-23T16:33:57.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e03aa55:::benchmark_data_smithi188_113957_object105527:head# 2024-04-23T16:33:57.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e03b9f8:::benchmark_data_smithi188_113957_object48930:head# 2024-04-23T16:33:57.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e03d713:::benchmark_data_smithi188_113957_object134866:head# 2024-04-23T16:33:57.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e06ea4d:::benchmark_data_smithi188_113957_object8902:head# 2024-04-23T16:33:57.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e072103:::benchmark_data_smithi188_113957_object19677:head# 2024-04-23T16:33:57.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e075114:::benchmark_data_smithi188_113957_object126601:head# 2024-04-23T16:33:57.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e076c30:::benchmark_data_smithi188_113957_object2306:head# 2024-04-23T16:33:57.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e0780b7:::benchmark_data_smithi188_113957_object73712:head# 2024-04-23T16:33:57.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e080bf8:::benchmark_data_smithi188_113957_object142929:head# 2024-04-23T16:33:57.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e0912b1:::benchmark_data_smithi188_113957_object56432:head# 2024-04-23T16:33:57.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e09235f:::benchmark_data_smithi188_113957_object59444:head# 2024-04-23T16:33:57.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e0a3bcc:::benchmark_data_smithi188_113957_object26760:head# 2024-04-23T16:33:57.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e0b656d:::benchmark_data_smithi188_113957_object40299:head# 2024-04-23T16:33:57.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e0bcbf7:::benchmark_data_smithi188_113957_object80785:head# 2024-04-23T16:33:57.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e0cbd92:::benchmark_data_smithi188_113957_object49771:head# 2024-04-23T16:33:57.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e0d9514:::benchmark_data_smithi188_113957_object44629:head# 2024-04-23T16:33:57.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e0dd5c0:::benchmark_data_smithi188_113957_object4503:head# 2024-04-23T16:33:57.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e0fe7a5:::benchmark_data_smithi188_113957_object106888:head# 2024-04-23T16:33:57.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e0ffc85:::benchmark_data_smithi188_113957_object136256:head# 2024-04-23T16:33:57.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e101ca7:::benchmark_data_smithi188_113957_object105269:head# 2024-04-23T16:33:57.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e101f20:::benchmark_data_smithi188_113957_object137491:head# 2024-04-23T16:33:57.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e103ee9:::benchmark_data_smithi188_113957_object108058:head# 2024-04-23T16:33:57.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e104a85:::benchmark_data_smithi188_113957_object80642:head# 2024-04-23T16:33:57.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e104f41:::benchmark_data_smithi188_113957_object86809:head# 2024-04-23T16:33:57.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e106ac8:::benchmark_data_smithi188_113957_object14965:head# 2024-04-23T16:33:57.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1143c7:::benchmark_data_smithi188_113957_object70715:head# 2024-04-23T16:33:57.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e117818:::benchmark_data_smithi188_113957_object86128:head# 2024-04-23T16:33:57.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e11a4ea:::benchmark_data_smithi188_113957_object125452:head# 2024-04-23T16:33:57.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e11b5d0:::benchmark_data_smithi188_113957_object36427:head# 2024-04-23T16:33:57.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e11c2c6:::benchmark_data_smithi188_113957_object25207:head# 2024-04-23T16:33:57.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1211a4:::benchmark_data_smithi188_113957_object40144:head# 2024-04-23T16:33:57.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e12371d:::benchmark_data_smithi188_113957_object32913:head# 2024-04-23T16:33:57.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e12d73b:::benchmark_data_smithi188_113957_object19319:head# 2024-04-23T16:33:57.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e138fef:::benchmark_data_smithi188_113957_object91208:head# 2024-04-23T16:33:57.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e13b944:::benchmark_data_smithi188_113957_object29550:head# 2024-04-23T16:33:57.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e145b86:::benchmark_data_smithi188_113957_object131291:head# 2024-04-23T16:33:57.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e14a3b1:::benchmark_data_smithi188_113957_object121896:head# 2024-04-23T16:33:57.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e14f471:::benchmark_data_smithi188_113957_object109918:head# 2024-04-23T16:33:57.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e157071:::benchmark_data_smithi188_113957_object42351:head# 2024-04-23T16:33:57.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1573c6:::benchmark_data_smithi188_113957_object99036:head# 2024-04-23T16:33:57.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e15b670:::benchmark_data_smithi188_113957_object79525:head# 2024-04-23T16:33:57.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e15d92b:::benchmark_data_smithi188_113957_object87152:head# 2024-04-23T16:33:57.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1600fd:::benchmark_data_smithi188_113957_object85305:head# 2024-04-23T16:33:57.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e16606c:::benchmark_data_smithi188_113957_object24089:head# 2024-04-23T16:33:57.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1696e8:::benchmark_data_smithi188_113957_object97304:head# 2024-04-23T16:33:57.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e169f17:::benchmark_data_smithi188_113957_object86439:head# 2024-04-23T16:33:57.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e16d36d:::benchmark_data_smithi188_113957_object17952:head# 2024-04-23T16:33:57.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e183e79:::benchmark_data_smithi188_113957_object15523:head# 2024-04-23T16:33:57.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1904ab:::benchmark_data_smithi188_113957_object151011:head# 2024-04-23T16:33:57.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e193991:::benchmark_data_smithi188_113957_object92923:head# 2024-04-23T16:33:57.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1a1b8f:::benchmark_data_smithi188_113957_object19766:head# 2024-04-23T16:33:57.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1a5174:::benchmark_data_smithi188_113957_object39684:head# 2024-04-23T16:33:57.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1a559b:::benchmark_data_smithi188_113957_object97418:head# 2024-04-23T16:33:57.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1ab87c:::benchmark_data_smithi188_113957_object104072:head# 2024-04-23T16:33:57.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1ac5c2:::benchmark_data_smithi188_113957_object135537:head# 2024-04-23T16:33:57.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1b0d42:::benchmark_data_smithi188_113957_object59237:head# 2024-04-23T16:33:57.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1b3189:::benchmark_data_smithi188_113957_object75765:head# 2024-04-23T16:33:57.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1c26e9:::benchmark_data_smithi188_113957_object126995:head# 2024-04-23T16:33:57.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1d6ca1:::benchmark_data_smithi188_113957_object36940:head# 2024-04-23T16:33:57.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1ddbca:::benchmark_data_smithi188_113957_object15248:head# 2024-04-23T16:33:57.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1e1fa8:::benchmark_data_smithi188_113957_object49981:head# 2024-04-23T16:33:57.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1e4be0:::benchmark_data_smithi188_113957_object37630:head# 2024-04-23T16:33:57.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1e6b89:::benchmark_data_smithi188_113957_object20452:head# 2024-04-23T16:33:57.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1e6fa6:::benchmark_data_smithi188_113957_object49399:head# 2024-04-23T16:33:57.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1e87d4:::benchmark_data_smithi188_113957_object114092:head# 2024-04-23T16:33:57.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1e8aba:::benchmark_data_smithi188_113957_object32986:head# 2024-04-23T16:33:57.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1f03f1:::benchmark_data_smithi188_113957_object86680:head# 2024-04-23T16:33:57.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1f5244:::benchmark_data_smithi188_113957_object122194:head# 2024-04-23T16:33:57.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1f7c34:::benchmark_data_smithi188_113957_object13888:head# 2024-04-23T16:33:57.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e1fa8e8:::benchmark_data_smithi188_113957_object12915:head# 2024-04-23T16:33:57.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e212093:::benchmark_data_smithi188_113957_object108163:head# 2024-04-23T16:33:57.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2142ec:::benchmark_data_smithi188_113957_object41016:head# 2024-04-23T16:33:57.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e216541:::benchmark_data_smithi188_113957_object86426:head# 2024-04-23T16:33:57.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e22d759:::benchmark_data_smithi188_113957_object89073:head# 2024-04-23T16:33:57.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e22e497:::benchmark_data_smithi188_113957_object4220:head# 2024-04-23T16:33:57.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e23047c:::benchmark_data_smithi188_113957_object5082:head# 2024-04-23T16:33:57.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2306a5:::benchmark_data_smithi188_113957_object34622:head# 2024-04-23T16:33:57.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e232611:::benchmark_data_smithi188_113957_object31837:head# 2024-04-23T16:33:57.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e232a6b:::benchmark_data_smithi188_113957_object15444:head# 2024-04-23T16:33:57.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2362a1:::benchmark_data_smithi188_113957_object79330:head# 2024-04-23T16:33:57.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e23bf0d:::benchmark_data_smithi188_113957_object143462:head# 2024-04-23T16:33:57.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e243e39:::benchmark_data_smithi188_113957_object139553:head# 2024-04-23T16:33:57.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2461cb:::benchmark_data_smithi188_113957_object138826:head# 2024-04-23T16:33:57.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e24b7d4:::benchmark_data_smithi188_113957_object97583:head# 2024-04-23T16:33:57.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e25323a:::benchmark_data_smithi188_113957_object12332:head# 2024-04-23T16:33:57.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e25d44c:::benchmark_data_smithi188_113957_object141813:head# 2024-04-23T16:33:57.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e26107c:::benchmark_data_smithi188_113957_object111196:head# 2024-04-23T16:33:57.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e26dc03:::benchmark_data_smithi188_113957_object58773:head# 2024-04-23T16:33:57.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2750cc:::benchmark_data_smithi188_113957_object102664:head# 2024-04-23T16:33:57.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e275431:::benchmark_data_smithi188_113957_object103894:head# 2024-04-23T16:33:57.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e276954:::benchmark_data_smithi188_113957_object52358:head# 2024-04-23T16:33:57.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e276a96:::benchmark_data_smithi188_113957_object56329:head# 2024-04-23T16:33:57.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e28879b:::benchmark_data_smithi188_113957_object152823:head# 2024-04-23T16:33:57.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e28da7e:::benchmark_data_smithi188_113957_object143921:head# 2024-04-23T16:33:57.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e29714c:::benchmark_data_smithi188_113957_object25885:head# 2024-04-23T16:33:57.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2a1d32:::benchmark_data_smithi188_113957_object77074:head# 2024-04-23T16:33:57.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2b039c:::benchmark_data_smithi188_113957_object16108:head# 2024-04-23T16:33:57.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2b38a0:::benchmark_data_smithi188_113957_object127294:head# 2024-04-23T16:33:57.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2bb758:::benchmark_data_smithi188_113957_object109218:head# 2024-04-23T16:33:57.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2be29e:::benchmark_data_smithi188_113957_object44252:head# 2024-04-23T16:33:57.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2c11e6:::benchmark_data_smithi188_113957_object138135:head# 2024-04-23T16:33:57.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2c21ae:::benchmark_data_smithi188_113957_object91251:head# 2024-04-23T16:33:57.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2c7965:::benchmark_data_smithi188_113957_object70830:head# 2024-04-23T16:33:57.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2cb857:::benchmark_data_smithi188_113957_object23106:head# 2024-04-23T16:33:57.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2cca38:::benchmark_data_smithi188_113957_object118650:head# 2024-04-23T16:33:57.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2d29f0:::benchmark_data_smithi188_113957_object130441:head# 2024-04-23T16:33:57.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2d409e:::benchmark_data_smithi188_113957_object64602:head# 2024-04-23T16:33:57.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2d62b2:::benchmark_data_smithi188_113957_object131638:head# 2024-04-23T16:33:57.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2dd920:::benchmark_data_smithi188_113957_object112559:head# 2024-04-23T16:33:57.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2e37a3:::benchmark_data_smithi188_113957_object101455:head# 2024-04-23T16:33:57.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2e8bb0:::benchmark_data_smithi188_113957_object111622:head# 2024-04-23T16:33:57.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2fbee4:::benchmark_data_smithi188_113957_object147597:head# 2024-04-23T16:33:57.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2fcb2d:::benchmark_data_smithi188_113957_object18791:head# 2024-04-23T16:33:57.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e2ff4d9:::benchmark_data_smithi188_113957_object50850:head# 2024-04-23T16:33:57.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e307912:::benchmark_data_smithi188_113957_object31836:head# 2024-04-23T16:33:57.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e30e472:::benchmark_data_smithi188_113957_object81127:head# 2024-04-23T16:33:57.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e315ffd:::benchmark_data_smithi188_113957_object82776:head# 2024-04-23T16:33:57.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e3193c9:::benchmark_data_smithi188_113957_object132020:head# 2024-04-23T16:33:57.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e319de7:::benchmark_data_smithi188_113957_object102299:head# 2024-04-23T16:33:57.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e31b7da:::benchmark_data_smithi188_113957_object113266:head# 2024-04-23T16:33:57.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e31cb51:::benchmark_data_smithi188_113957_object8553:head# 2024-04-23T16:33:57.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e326619:::benchmark_data_smithi188_113957_object148940:head# 2024-04-23T16:33:57.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e327842:::benchmark_data_smithi188_113957_object56402:head# 2024-04-23T16:33:57.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e32cff3:::benchmark_data_smithi188_113957_object90263:head# 2024-04-23T16:33:57.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e33cbf9:::benchmark_data_smithi188_113957_object126122:head# 2024-04-23T16:33:57.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e343d7f:::benchmark_data_smithi188_113957_object4543:head# 2024-04-23T16:33:57.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e34c76b:::benchmark_data_smithi188_113957_object119738:head# 2024-04-23T16:33:57.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e352216:::benchmark_data_smithi188_113957_object82800:head# 2024-04-23T16:33:57.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e35628e:::benchmark_data_smithi188_113957_object83438:head# 2024-04-23T16:33:57.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e374907:::benchmark_data_smithi188_113957_object27936:head# 2024-04-23T16:33:57.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e375b6c:::benchmark_data_smithi188_113957_object101946:head# 2024-04-23T16:33:57.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e37ba63:::benchmark_data_smithi188_113957_object146775:head# 2024-04-23T16:33:57.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e37fe4a:::benchmark_data_smithi188_113957_object18593:head# 2024-04-23T16:33:57.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e382039:::benchmark_data_smithi188_113957_object97427:head# 2024-04-23T16:33:57.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e382548:::benchmark_data_smithi188_113957_object18743:head# 2024-04-23T16:33:57.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e392a3b:::benchmark_data_smithi188_113957_object2186:head# 2024-04-23T16:33:57.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e39eaec:::benchmark_data_smithi188_113957_object63736:head# 2024-04-23T16:33:57.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e3a2499:::benchmark_data_smithi188_113957_object81471:head# 2024-04-23T16:33:57.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e3cea6e:::benchmark_data_smithi188_113957_object83685:head# 2024-04-23T16:33:57.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e3e12b9:::benchmark_data_smithi188_113957_object109824:head# 2024-04-23T16:33:57.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e3f047d:::benchmark_data_smithi188_113957_object74600:head# 2024-04-23T16:33:57.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e3f2d55:::benchmark_data_smithi188_113957_object75221:head# 2024-04-23T16:33:57.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e3f95fa:::benchmark_data_smithi188_113957_object10470:head# 2024-04-23T16:33:57.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e3fbb7a:::benchmark_data_smithi188_113957_object5723:head# 2024-04-23T16:33:57.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e40c757:::benchmark_data_smithi188_113957_object117155:head# 2024-04-23T16:33:57.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e40cb03:::benchmark_data_smithi188_113957_object148943:head# 2024-04-23T16:33:57.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e41d115:::benchmark_data_smithi188_113957_object15393:head# 2024-04-23T16:33:57.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e420f65:::benchmark_data_smithi188_113957_object137902:head# 2024-04-23T16:33:57.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e42a314:::benchmark_data_smithi188_113957_object73824:head# 2024-04-23T16:33:57.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e43d638:::benchmark_data_smithi188_113957_object47307:head# 2024-04-23T16:33:57.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e43ea2b:::benchmark_data_smithi188_113957_object71615:head# 2024-04-23T16:33:57.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e43f218:::benchmark_data_smithi188_113957_object61419:head# 2024-04-23T16:33:57.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e44b6cf:::benchmark_data_smithi188_113957_object92676:head# 2024-04-23T16:33:57.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e44dc66:::benchmark_data_smithi188_113957_object49373:head# 2024-04-23T16:33:57.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e457acf:::benchmark_data_smithi188_113957_object56215:head# 2024-04-23T16:33:57.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e45d940:::benchmark_data_smithi188_113957_object138099:head# 2024-04-23T16:33:57.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e460cfb:::benchmark_data_smithi188_113957_object84465:head# 2024-04-23T16:33:57.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e469dbc:::benchmark_data_smithi188_113957_object58242:head# 2024-04-23T16:33:57.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e472bee:::benchmark_data_smithi188_113957_object67161:head# 2024-04-23T16:33:57.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e473efc:::benchmark_data_smithi188_113957_object105147:head# 2024-04-23T16:33:57.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e474e14:::benchmark_data_smithi188_113957_object59387:head# 2024-04-23T16:33:57.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e4781d3:::benchmark_data_smithi188_113957_object87704:head# 2024-04-23T16:33:57.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e478a8d:::benchmark_data_smithi188_113957_object110956:head# 2024-04-23T16:33:57.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e48293a:::benchmark_data_smithi188_113957_object54743:head# 2024-04-23T16:33:57.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e482ef7:::benchmark_data_smithi188_113957_object69399:head# 2024-04-23T16:33:57.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e488f89:::benchmark_data_smithi188_113957_object48151:head# 2024-04-23T16:33:57.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e48b046:::benchmark_data_smithi188_113957_object37454:head# 2024-04-23T16:33:57.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e48f168:::benchmark_data_smithi188_113957_object142723:head# 2024-04-23T16:33:57.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e492478:::benchmark_data_smithi188_113957_object126885:head# 2024-04-23T16:33:57.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e493395:::benchmark_data_smithi188_113957_object132498:head# 2024-04-23T16:33:57.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e4aeb67:::benchmark_data_smithi188_113957_object8713:head# 2024-04-23T16:33:57.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e4aeb87:::benchmark_data_smithi188_113957_object143851:head# 2024-04-23T16:33:57.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e4b77c7:::benchmark_data_smithi188_113957_object126720:head# 2024-04-23T16:33:57.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e4bbbb5:::benchmark_data_smithi188_113957_object24186:head# 2024-04-23T16:33:57.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e4bc623:::benchmark_data_smithi188_113957_object135324:head# 2024-04-23T16:33:57.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e4c6e20:::benchmark_data_smithi188_113957_object37348:head# 2024-04-23T16:33:57.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e4c87a0:::benchmark_data_smithi188_113957_object48437:head# 2024-04-23T16:33:57.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e4cd828:::benchmark_data_smithi188_113957_object3085:head# 2024-04-23T16:33:57.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e4cfc72:::benchmark_data_smithi188_113957_object62097:head# 2024-04-23T16:33:57.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e4d3bfb:::benchmark_data_smithi188_113957_object21487:head# 2024-04-23T16:33:57.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e4eeb80:::benchmark_data_smithi188_113957_object2564:head# 2024-04-23T16:33:57.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e4eeea2:::benchmark_data_smithi188_113957_object38836:head# 2024-04-23T16:33:57.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e4f9e4f:::benchmark_data_smithi188_113957_object122815:head# 2024-04-23T16:33:57.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e50478e:::benchmark_data_smithi188_113957_object85869:head# 2024-04-23T16:33:57.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e506852:::benchmark_data_smithi188_113957_object74054:head# 2024-04-23T16:33:57.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e509ce4:::benchmark_data_smithi188_113957_object63477:head# 2024-04-23T16:33:57.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e50a058:::benchmark_data_smithi188_113957_object44403:head# 2024-04-23T16:33:57.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e512a5e:::benchmark_data_smithi188_113957_object4828:head# 2024-04-23T16:33:57.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e5163a1:::benchmark_data_smithi188_113957_object101011:head# 2024-04-23T16:33:57.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e51c4db:::benchmark_data_smithi188_113957_object3466:head# 2024-04-23T16:33:57.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e51e9b3:::benchmark_data_smithi188_113957_object13435:head# 2024-04-23T16:33:57.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e51f4cf:::benchmark_data_smithi188_113957_object98313:head# 2024-04-23T16:33:57.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e525f4f:::benchmark_data_smithi188_113957_object144824:head# 2024-04-23T16:33:57.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e5281aa:::benchmark_data_smithi188_113957_object40847:head# 2024-04-23T16:33:57.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e54372b:::benchmark_data_smithi188_113957_object62138:head# 2024-04-23T16:33:57.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e559805:::benchmark_data_smithi188_113957_object135405:head# 2024-04-23T16:33:57.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e56dde8:::benchmark_data_smithi188_113957_object17884:head# 2024-04-23T16:33:57.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e57b3af:::benchmark_data_smithi188_113957_object103257:head# 2024-04-23T16:33:57.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e58a1c1:::benchmark_data_smithi188_113957_object118565:head# 2024-04-23T16:33:57.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e58cfe2:::benchmark_data_smithi188_113957_object54401:head# 2024-04-23T16:33:57.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e5949fc:::benchmark_data_smithi188_113957_object47591:head# 2024-04-23T16:33:57.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e5975a7:::benchmark_data_smithi188_113957_object124275:head# 2024-04-23T16:33:57.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e59de85:::benchmark_data_smithi188_113957_object33195:head# 2024-04-23T16:33:57.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e59eaae:::benchmark_data_smithi188_113957_object18544:head# 2024-04-23T16:33:57.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e5a31c2:::benchmark_data_smithi188_113957_object122893:head# 2024-04-23T16:33:57.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e5a83d1:::benchmark_data_smithi188_113957_object83921:head# 2024-04-23T16:33:57.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e5a8e4e:::benchmark_data_smithi188_113957_object63398:head# 2024-04-23T16:33:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e5b5bad:::benchmark_data_smithi188_113957_object2813:head# 2024-04-23T16:33:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e5e4b9a:::benchmark_data_smithi188_113957_object7703:head# 2024-04-23T16:33:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e5ef9c9:::benchmark_data_smithi188_113957_object140215:head# 2024-04-23T16:33:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e603b67:::benchmark_data_smithi188_113957_object77310:head# 2024-04-23T16:33:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e6066f8:::benchmark_data_smithi188_113957_object118557:head# 2024-04-23T16:33:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e607dca:::benchmark_data_smithi188_113957_object57832:head# 2024-04-23T16:33:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e639d60:::benchmark_data_smithi188_113957_object151102:head# 2024-04-23T16:33:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e6420b5:::benchmark_data_smithi188_113957_object1721:head# 2024-04-23T16:33:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e645352:::benchmark_data_smithi188_113957_object1771:head# 2024-04-23T16:33:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e65460c:::benchmark_data_smithi188_113957_object98105:head# 2024-04-23T16:33:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e6584ff:::benchmark_data_smithi188_113957_object108175:head# 2024-04-23T16:33:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e662ef6:::benchmark_data_smithi188_113957_object106378:head# 2024-04-23T16:33:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e66769e:::benchmark_data_smithi188_113957_object73816:head# 2024-04-23T16:33:57.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e6774bf:::benchmark_data_smithi188_113957_object148490:head# 2024-04-23T16:33:57.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e683422:::benchmark_data_smithi188_113957_object30319:head# 2024-04-23T16:33:57.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e685843:::benchmark_data_smithi188_113957_object147521:head# 2024-04-23T16:33:57.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e695020:::benchmark_data_smithi188_113957_object70164:head# 2024-04-23T16:33:57.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e6979d7:::benchmark_data_smithi188_113957_object149968:head# 2024-04-23T16:33:57.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e699982:::benchmark_data_smithi188_113957_object45930:head# 2024-04-23T16:33:57.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e6a77cf:::benchmark_data_smithi188_113957_object13041:head# 2024-04-23T16:33:57.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e6b27ad:::benchmark_data_smithi188_113957_object67730:head# 2024-04-23T16:33:57.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e6b7695:::benchmark_data_smithi188_113957_object27825:head# 2024-04-23T16:33:57.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e6cb06d:::benchmark_data_smithi188_113957_object51305:head# 2024-04-23T16:33:57.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e6cc0dc:::benchmark_data_smithi188_113957_object95999:head# 2024-04-23T16:33:57.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e6d91fc:::benchmark_data_smithi188_113957_object45202:head# 2024-04-23T16:33:57.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e6e2300:::benchmark_data_smithi188_113957_object49563:head# 2024-04-23T16:33:57.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e6eea12:::benchmark_data_smithi188_113957_object49844:head# 2024-04-23T16:33:57.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e6eebeb:::benchmark_data_smithi188_113957_object5628:head# 2024-04-23T16:33:57.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e6f81f4:::benchmark_data_smithi188_113957_object20581:head# 2024-04-23T16:33:57.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7001b6:::benchmark_data_smithi188_113957_object127335:head# 2024-04-23T16:33:57.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e71262f:::benchmark_data_smithi188_113957_object95707:head# 2024-04-23T16:33:57.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7129bf:::benchmark_data_smithi188_113957_object142325:head# 2024-04-23T16:33:57.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e71fde5:::benchmark_data_smithi188_113957_object67194:head# 2024-04-23T16:33:57.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e72243f:::benchmark_data_smithi188_113957_object6424:head# 2024-04-23T16:33:57.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e742130:::benchmark_data_smithi188_113957_object49761:head# 2024-04-23T16:33:57.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e742a58:::benchmark_data_smithi188_113957_object120731:head# 2024-04-23T16:33:57.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e746c8c:::benchmark_data_smithi188_113957_object53137:head# 2024-04-23T16:33:57.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e74a531:::benchmark_data_smithi188_113957_object52215:head# 2024-04-23T16:33:57.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e75178e:::benchmark_data_smithi188_113957_object79859:head# 2024-04-23T16:33:57.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e75c2af:::benchmark_data_smithi188_113957_object127926:head# 2024-04-23T16:33:57.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e75d18d:::benchmark_data_smithi188_113957_object85768:head# 2024-04-23T16:33:57.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e76260a:::benchmark_data_smithi188_113957_object33067:head# 2024-04-23T16:33:57.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e762e1b:::benchmark_data_smithi188_113957_object73473:head# 2024-04-23T16:33:57.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e76c3e6:::benchmark_data_smithi188_113957_object54948:head# 2024-04-23T16:33:57.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e771dbc:::benchmark_data_smithi188_113957_object117564:head# 2024-04-23T16:33:57.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e78a13d:::benchmark_data_smithi188_113957_object113511:head# 2024-04-23T16:33:57.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e78a348:::benchmark_data_smithi188_113957_object152103:head# 2024-04-23T16:33:57.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e79e514:::benchmark_data_smithi188_113957_object42689:head# 2024-04-23T16:33:57.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7a0298:::benchmark_data_smithi188_113957_object60051:head# 2024-04-23T16:33:57.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7ad008:::benchmark_data_smithi188_113957_object6378:head# 2024-04-23T16:33:57.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7b2f3a:::benchmark_data_smithi188_113957_object35294:head# 2024-04-23T16:33:57.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7b87d3:::benchmark_data_smithi188_113957_object122034:head# 2024-04-23T16:33:57.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7c870f:::benchmark_data_smithi188_113957_object99114:head# 2024-04-23T16:33:57.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7ce80d:::benchmark_data_smithi188_113957_object66012:head# 2024-04-23T16:33:57.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7cfc75:::benchmark_data_smithi188_113957_object138747:head# 2024-04-23T16:33:57.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7d3c21:::benchmark_data_smithi188_113957_object56709:head# 2024-04-23T16:33:57.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7d5a2b:::benchmark_data_smithi188_113957_object114980:head# 2024-04-23T16:33:57.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7d5acc:::benchmark_data_smithi188_113957_object130791:head# 2024-04-23T16:33:57.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7d6a88:::benchmark_data_smithi188_113957_object17036:head# 2024-04-23T16:33:57.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7ea9c7:::benchmark_data_smithi188_113957_object39501:head# 2024-04-23T16:33:57.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7eb068:::benchmark_data_smithi188_113957_object46638:head# 2024-04-23T16:33:57.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7ece0e:::benchmark_data_smithi188_113957_object28278:head# 2024-04-23T16:33:57.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7ef13a:::benchmark_data_smithi188_113957_object65977:head# 2024-04-23T16:33:57.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7ef1bb:::benchmark_data_smithi188_113957_object126463:head# 2024-04-23T16:33:57.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7f09f8:::benchmark_data_smithi188_113957_object146095:head# 2024-04-23T16:33:57.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7f0a48:::benchmark_data_smithi188_113957_object88009:head# 2024-04-23T16:33:57.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7f3935:::benchmark_data_smithi188_113957_object113063:head# 2024-04-23T16:33:57.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7f8c82:::benchmark_data_smithi188_113957_object57362:head# 2024-04-23T16:33:57.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7fd9a3:::benchmark_data_smithi188_113957_object109384:head# 2024-04-23T16:33:57.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e7fde27:::benchmark_data_smithi188_113957_object12570:head# 2024-04-23T16:33:57.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e803a64:::benchmark_data_smithi188_113957_object111860:head# 2024-04-23T16:33:57.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e807c19:::benchmark_data_smithi188_113957_object124209:head# 2024-04-23T16:33:57.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e810593:::benchmark_data_smithi188_113957_object91582:head# 2024-04-23T16:33:57.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e8169db:::benchmark_data_smithi188_113957_object100140:head# 2024-04-23T16:33:57.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e81cf65:::benchmark_data_smithi188_113957_object26745:head# 2024-04-23T16:33:57.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e81d13f:::benchmark_data_smithi188_113957_object17880:head# 2024-04-23T16:33:57.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e828465:::benchmark_data_smithi188_113957_object41309:head# 2024-04-23T16:33:57.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e82940f:::benchmark_data_smithi188_113957_object92753:head# 2024-04-23T16:33:57.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e82964a:::benchmark_data_smithi188_113957_object99859:head# 2024-04-23T16:33:57.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e832db2:::benchmark_data_smithi188_113957_object69582:head# 2024-04-23T16:33:57.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e838f90:::benchmark_data_smithi188_113957_object3149:head# 2024-04-23T16:33:57.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e848b81:::benchmark_data_smithi188_113957_object26036:head# 2024-04-23T16:33:57.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e8498eb:::benchmark_data_smithi188_113957_object26106:head# 2024-04-23T16:33:57.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e84e50b:::benchmark_data_smithi188_113957_object70893:head# 2024-04-23T16:33:57.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e851ca3:::benchmark_data_smithi188_113957_object108461:head# 2024-04-23T16:33:57.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.321 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:33:57.322 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:33:57.323 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:56 smithi188 ceph-mon[44101]: 5.2 scrub starts 2024-04-23T16:33:57.349 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e854c87:::benchmark_data_smithi188_113957_object42787:head# 2024-04-23T16:33:57.349 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.349 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e85c82b:::benchmark_data_smithi188_113957_object41855:head# 2024-04-23T16:33:57.349 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.349 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e861cf8:::benchmark_data_smithi188_113957_object135010:head# 2024-04-23T16:33:57.349 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.349 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e8628ed:::benchmark_data_smithi188_113957_object68177:head# 2024-04-23T16:33:57.349 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.349 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e86cfa0:::benchmark_data_smithi188_113957_object50905:head# 2024-04-23T16:33:57.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e879f07:::benchmark_data_smithi188_113957_object30928:head# 2024-04-23T16:33:57.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e88448a:::benchmark_data_smithi188_113957_object142591:head# 2024-04-23T16:33:57.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e8849c2:::benchmark_data_smithi188_113957_object37666:head# 2024-04-23T16:33:57.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e897913:::benchmark_data_smithi188_113957_object21805:head# 2024-04-23T16:33:57.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e89a599:::benchmark_data_smithi188_113957_object34436:head# 2024-04-23T16:33:57.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e89a8dd:::benchmark_data_smithi188_113957_object139799:head# 2024-04-23T16:33:57.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e89b8c7:::benchmark_data_smithi188_113957_object16630:head# 2024-04-23T16:33:57.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e89cc2c:::benchmark_data_smithi188_113957_object105313:head# 2024-04-23T16:33:57.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e89d598:::benchmark_data_smithi188_113957_object117948:head# 2024-04-23T16:33:57.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e89e7b6:::benchmark_data_smithi188_113957_object69496:head# 2024-04-23T16:33:57.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e8a1a1e:::benchmark_data_smithi188_113957_object135191:head# 2024-04-23T16:33:57.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e8a2eee:::benchmark_data_smithi188_113957_object45428:head# 2024-04-23T16:33:57.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e8b0554:::benchmark_data_smithi188_113957_object107990:head# 2024-04-23T16:33:57.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e8b8aec:::benchmark_data_smithi188_113957_object97471:head# 2024-04-23T16:33:57.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e8cdffb:::benchmark_data_smithi188_113957_object135921:head# 2024-04-23T16:33:57.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e8de167:::benchmark_data_smithi188_113957_object115972:head# 2024-04-23T16:33:57.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e8e5843:::benchmark_data_smithi188_113957_object9350:head# 2024-04-23T16:33:57.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e8e7f26:::benchmark_data_smithi188_113957_object132411:head# 2024-04-23T16:33:57.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e8e8235:::benchmark_data_smithi188_113957_object99334:head# 2024-04-23T16:33:57.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e8eb374:::benchmark_data_smithi188_113957_object4017:head# 2024-04-23T16:33:57.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e8fc5fd:::benchmark_data_smithi188_113957_object61595:head# 2024-04-23T16:33:57.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e8fda72:::benchmark_data_smithi188_113957_object64010:head# 2024-04-23T16:33:57.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e911cc3:::benchmark_data_smithi188_113957_object28810:head# 2024-04-23T16:33:57.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e9124f7:::benchmark_data_smithi188_113957_object119613:head# 2024-04-23T16:33:57.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e9131bd:::benchmark_data_smithi188_113957_object128375:head# 2024-04-23T16:33:57.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e929331:::benchmark_data_smithi188_113957_object136790:head# 2024-04-23T16:33:57.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e92b636:::benchmark_data_smithi188_113957_object116107:head# 2024-04-23T16:33:57.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e92cf84:::benchmark_data_smithi188_113957_object141291:head# 2024-04-23T16:33:57.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e930826:::benchmark_data_smithi188_113957_object7783:head# 2024-04-23T16:33:57.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e93fb97:::benchmark_data_smithi188_113957_object61232:head# 2024-04-23T16:33:57.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e9434e9:::benchmark_data_smithi188_113957_object82580:head# 2024-04-23T16:33:57.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e944dad:::benchmark_data_smithi188_113957_object97399:head# 2024-04-23T16:33:57.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e945a6c:::benchmark_data_smithi188_113957_object93756:head# 2024-04-23T16:33:57.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e947290:::benchmark_data_smithi188_113957_object118745:head# 2024-04-23T16:33:57.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e949e2e:::benchmark_data_smithi188_113957_object26493:head# 2024-04-23T16:33:57.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e94e170:::benchmark_data_smithi188_113957_object104663:head# 2024-04-23T16:33:57.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e950c42:::benchmark_data_smithi188_113957_object58943:head# 2024-04-23T16:33:57.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e955e92:::benchmark_data_smithi188_113957_object49793:head# 2024-04-23T16:33:57.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e957583:::benchmark_data_smithi188_113957_object108641:head# 2024-04-23T16:33:57.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e957a82:::benchmark_data_smithi188_113957_object22608:head# 2024-04-23T16:33:57.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e95bea0:::benchmark_data_smithi188_113957_object104219:head# 2024-04-23T16:33:57.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e95c089:::benchmark_data_smithi188_113957_object103259:head# 2024-04-23T16:33:57.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e9641f2:::benchmark_data_smithi188_113957_object123675:head# 2024-04-23T16:33:57.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e964947:::benchmark_data_smithi188_113957_object43947:head# 2024-04-23T16:33:57.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e965e64:::benchmark_data_smithi188_113957_object108619:head# 2024-04-23T16:33:57.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e973a2c:::benchmark_data_smithi188_113957_object147712:head# 2024-04-23T16:33:57.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e975140:::benchmark_data_smithi188_113957_object141048:head# 2024-04-23T16:33:57.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e9802e4:::benchmark_data_smithi188_113957_object37893:head# 2024-04-23T16:33:57.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e988234:::benchmark_data_smithi188_113957_object22347:head# 2024-04-23T16:33:57.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e98cddd:::benchmark_data_smithi188_113957_object144427:head# 2024-04-23T16:33:57.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e99c184:::benchmark_data_smithi188_113957_object2375:head# 2024-04-23T16:33:57.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e99c507:::benchmark_data_smithi188_113957_object102973:head# 2024-04-23T16:33:57.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e9afc25:::benchmark_data_smithi188_113957_object122209:head# 2024-04-23T16:33:57.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e9b0abd:::benchmark_data_smithi188_113957_object151312:head# 2024-04-23T16:33:57.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e9d40e3:::benchmark_data_smithi188_113957_object31249:head# 2024-04-23T16:33:57.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e9d4579:::benchmark_data_smithi188_113957_object96779:head# 2024-04-23T16:33:57.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e9da92a:::benchmark_data_smithi188_113957_object95299:head# 2024-04-23T16:33:57.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e9e4f82:::benchmark_data_smithi188_113957_object148083:head# 2024-04-23T16:33:57.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e9ed953:::benchmark_data_smithi188_113957_object74420:head# 2024-04-23T16:33:57.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5e9f2293:::benchmark_data_smithi188_113957_object52874:head# 2024-04-23T16:33:57.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea03bab:::benchmark_data_smithi188_113957_object140362:head# 2024-04-23T16:33:57.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea0905c:::benchmark_data_smithi188_113957_object35367:head# 2024-04-23T16:33:57.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea10ee1:::benchmark_data_smithi188_113957_object99450:head# 2024-04-23T16:33:57.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea13cbd:::benchmark_data_smithi188_113957_object40846:head# 2024-04-23T16:33:57.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea160a8:::benchmark_data_smithi188_113957_object24390:head# 2024-04-23T16:33:57.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea16a3d:::benchmark_data_smithi188_113957_object120345:head# 2024-04-23T16:33:57.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea1c145:::benchmark_data_smithi188_113957_object70497:head# 2024-04-23T16:33:57.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea24a1c:::benchmark_data_smithi188_113957_object74276:head# 2024-04-23T16:33:57.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea25a84:::benchmark_data_smithi188_113957_object121815:head# 2024-04-23T16:33:57.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea2ccba:::benchmark_data_smithi188_113957_object141692:head# 2024-04-23T16:33:57.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea3bb3d:::benchmark_data_smithi188_113957_object145882:head# 2024-04-23T16:33:57.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea3e9fa:::benchmark_data_smithi188_113957_object47616:head# 2024-04-23T16:33:57.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea4e2c0:::benchmark_data_smithi188_113957_object149958:head# 2024-04-23T16:33:57.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea556a7:::benchmark_data_smithi188_113957_object30990:head# 2024-04-23T16:33:57.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea593d7:::benchmark_data_smithi188_113957_object13943:head# 2024-04-23T16:33:57.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea59ab7:::benchmark_data_smithi188_113957_object144881:head# 2024-04-23T16:33:57.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea5cd57:::benchmark_data_smithi188_113957_object82327:head# 2024-04-23T16:33:57.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea60b17:::benchmark_data_smithi188_113957_object97740:head# 2024-04-23T16:33:57.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea60f5b:::benchmark_data_smithi188_113957_object32629:head# 2024-04-23T16:33:57.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea638d7:::benchmark_data_smithi188_113957_object10995:head# 2024-04-23T16:33:57.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea642d3:::benchmark_data_smithi188_113957_object72861:head# 2024-04-23T16:33:57.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea6813f:::benchmark_data_smithi188_113957_object64911:head# 2024-04-23T16:33:57.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea70adb:::benchmark_data_smithi188_113957_object51729:head# 2024-04-23T16:33:57.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea75d24:::benchmark_data_smithi188_113957_object81690:head# 2024-04-23T16:33:57.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea7a62c:::benchmark_data_smithi188_113957_object26233:head# 2024-04-23T16:33:57.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea7fe90:::benchmark_data_smithi188_113957_object119173:head# 2024-04-23T16:33:57.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea812f0:::benchmark_data_smithi188_113957_object62865:head# 2024-04-23T16:33:57.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea8a0d6:::benchmark_data_smithi188_113957_object43696:head# 2024-04-23T16:33:57.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea8bad2:::benchmark_data_smithi188_113957_object122578:head# 2024-04-23T16:33:57.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea9221d:::benchmark_data_smithi188_113957_object107067:head# 2024-04-23T16:33:57.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea9720f:::benchmark_data_smithi188_113957_object91476:head# 2024-04-23T16:33:57.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea99923:::benchmark_data_smithi188_113957_object44729:head# 2024-04-23T16:33:57.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea9b895:::benchmark_data_smithi188_113957_object79267:head# 2024-04-23T16:33:57.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ea9f611:::benchmark_data_smithi188_113957_object135421:head# 2024-04-23T16:33:57.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eaa3451:::benchmark_data_smithi188_113957_object140522:head# 2024-04-23T16:33:57.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eaaa77a:::benchmark_data_smithi188_113957_object82657:head# 2024-04-23T16:33:57.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eaaf66a:::benchmark_data_smithi188_113957_object40907:head# 2024-04-23T16:33:57.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eac2920:::benchmark_data_smithi188_113957_object72513:head# 2024-04-23T16:33:57.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eac7292:::benchmark_data_smithi188_113957_object48096:head# 2024-04-23T16:33:57.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ead7d14:::benchmark_data_smithi188_113957_object28806:head# 2024-04-23T16:33:57.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eae40d7:::benchmark_data_smithi188_113957_object67626:head# 2024-04-23T16:33:57.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eae9ae5:::benchmark_data_smithi188_113957_object23876:head# 2024-04-23T16:33:57.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eaea34c:::benchmark_data_smithi188_113957_object91605:head# 2024-04-23T16:33:57.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eaf9b49:::benchmark_data_smithi188_113957_object71908:head# 2024-04-23T16:33:57.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eaff217:::benchmark_data_smithi188_113957_object35586:head# 2024-04-23T16:33:57.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb0148f:::benchmark_data_smithi188_113957_object147617:head# 2024-04-23T16:33:57.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb0667e:::benchmark_data_smithi188_113957_object46928:head# 2024-04-23T16:33:57.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb0792b:::benchmark_data_smithi188_113957_object130864:head# 2024-04-23T16:33:57.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb1fe2c:::benchmark_data_smithi188_113957_object26019:head# 2024-04-23T16:33:57.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb261a9:::benchmark_data_smithi188_113957_object122995:head# 2024-04-23T16:33:57.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb2b227:::benchmark_data_smithi188_113957_object150778:head# 2024-04-23T16:33:57.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb32570:::benchmark_data_smithi188_113957_object4702:head# 2024-04-23T16:33:57.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb364fd:::benchmark_data_smithi188_113957_object86664:head# 2024-04-23T16:33:57.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb39b99:::benchmark_data_smithi188_113957_object74260:head# 2024-04-23T16:33:57.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb39fba:::benchmark_data_smithi188_113957_object96634:head# 2024-04-23T16:33:57.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb3b81e:::benchmark_data_smithi188_113957_object131811:head# 2024-04-23T16:33:57.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb40757:::benchmark_data_smithi188_113957_object70656:head# 2024-04-23T16:33:57.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb5027a:::benchmark_data_smithi188_113957_object88212:head# 2024-04-23T16:33:57.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb63667:::benchmark_data_smithi188_113957_object108472:head# 2024-04-23T16:33:57.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb6e372:::benchmark_data_smithi188_113957_object60338:head# 2024-04-23T16:33:57.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb7880f:::benchmark_data_smithi188_113957_object61036:head# 2024-04-23T16:33:57.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb7c1ad:::benchmark_data_smithi188_113957_object62990:head# 2024-04-23T16:33:57.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb7c603:::benchmark_data_smithi188_113957_object126610:head# 2024-04-23T16:33:57.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb7f8d2:::benchmark_data_smithi188_113957_object122254:head# 2024-04-23T16:33:57.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb7fa3a:::benchmark_data_smithi188_113957_object128861:head# 2024-04-23T16:33:57.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb81336:::benchmark_data_smithi188_113957_object144298:head# 2024-04-23T16:33:57.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb823a8:::benchmark_data_smithi188_113957_object44674:head# 2024-04-23T16:33:57.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eb91710:::benchmark_data_smithi188_113957_object143424:head# 2024-04-23T16:33:57.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eba80aa:::benchmark_data_smithi188_113957_object32943:head# 2024-04-23T16:33:57.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eba87c9:::benchmark_data_smithi188_113957_object134153:head# 2024-04-23T16:33:57.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ebb0c9e:::benchmark_data_smithi188_113957_object33055:head# 2024-04-23T16:33:57.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ebb4256:::benchmark_data_smithi188_113957_object47309:head# 2024-04-23T16:33:57.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ebb7c27:::benchmark_data_smithi188_113957_object5005:head# 2024-04-23T16:33:57.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ebc6267:::benchmark_data_smithi188_113957_object31873:head# 2024-04-23T16:33:57.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ebc8ef2:::benchmark_data_smithi188_113957_object11949:head# 2024-04-23T16:33:57.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ebcd3f3:::benchmark_data_smithi188_113957_object82558:head# 2024-04-23T16:33:57.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ebd7f7e:::benchmark_data_smithi188_113957_object17411:head# 2024-04-23T16:33:57.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ebd9545:::benchmark_data_smithi188_113957_object46111:head# 2024-04-23T16:33:57.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ebde932:::benchmark_data_smithi188_113957_object138794:head# 2024-04-23T16:33:57.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ebe5bfb:::benchmark_data_smithi188_113957_object149125:head# 2024-04-23T16:33:57.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ebe5c41:::benchmark_data_smithi188_113957_object110795:head# 2024-04-23T16:33:57.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ebebcdb:::benchmark_data_smithi188_113957_object124044:head# 2024-04-23T16:33:57.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ebee58f:::benchmark_data_smithi188_113957_object35289:head# 2024-04-23T16:33:57.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ebef38a:::benchmark_data_smithi188_113957_object141154:head# 2024-04-23T16:33:57.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ebf3215:::benchmark_data_smithi188_113957_object115088:head# 2024-04-23T16:33:57.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec063e3:::benchmark_data_smithi188_113957_object38508:head# 2024-04-23T16:33:57.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec09104:::benchmark_data_smithi188_113957_object134615:head# 2024-04-23T16:33:57.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec0b20b:::benchmark_data_smithi188_113957_object79837:head# 2024-04-23T16:33:57.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec16a8e:::benchmark_data_smithi188_113957_object27537:head# 2024-04-23T16:33:57.372 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.372 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec19855:::benchmark_data_smithi188_113957_object73977:head# 2024-04-23T16:33:57.372 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.372 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec1a66f:::benchmark_data_smithi188_113957_object79172:head# 2024-04-23T16:33:57.372 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.372 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec2651f:::benchmark_data_smithi188_113957_object115067:head# 2024-04-23T16:33:57.372 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.372 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec27b97:::benchmark_data_smithi188_113957_object51959:head# 2024-04-23T16:33:57.372 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.372 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec33c1d:::benchmark_data_smithi188_113957_object24568:head# 2024-04-23T16:33:57.372 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.372 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec37292:::benchmark_data_smithi188_113957_object115479:head# 2024-04-23T16:33:57.372 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.372 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec37689:::benchmark_data_smithi188_113957_object71294:head# 2024-04-23T16:33:57.373 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.373 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec3920f:::benchmark_data_smithi188_113957_object44428:head# 2024-04-23T16:33:57.373 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.373 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec3c3c7:::benchmark_data_smithi188_113957_object96487:head# 2024-04-23T16:33:57.373 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.373 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec4067f:::benchmark_data_smithi188_113957_object151384:head# 2024-04-23T16:33:57.373 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.373 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec43044:::benchmark_data_smithi188_113957_object9621:head# 2024-04-23T16:33:57.373 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.373 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec456de:::benchmark_data_smithi188_113957_object110791:head# 2024-04-23T16:33:57.373 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.373 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec4b6ae:::benchmark_data_smithi188_113957_object37602:head# 2024-04-23T16:33:57.373 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.373 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec51010:::benchmark_data_smithi188_113957_object35388:head# 2024-04-23T16:33:57.374 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.374 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec522bd:::benchmark_data_smithi188_113957_object5559:head# 2024-04-23T16:33:57.374 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.374 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec5ea41:::benchmark_data_smithi188_113957_object73813:head# 2024-04-23T16:33:57.374 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.374 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec610c7:::benchmark_data_smithi188_113957_object88278:head# 2024-04-23T16:33:57.374 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.374 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec62757:::benchmark_data_smithi188_113957_object19782:head# 2024-04-23T16:33:57.374 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.374 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec647d6:::benchmark_data_smithi188_113957_object64491:head# 2024-04-23T16:33:57.374 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.374 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec70557:::benchmark_data_smithi188_113957_object29126:head# 2024-04-23T16:33:57.374 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.374 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec75650:::benchmark_data_smithi188_113957_object58464:head# 2024-04-23T16:33:57.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec7c79b:::benchmark_data_smithi188_113957_object48159:head# 2024-04-23T16:33:57.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec87bd0:::benchmark_data_smithi188_113957_object141264:head# 2024-04-23T16:33:57.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec89e8d:::benchmark_data_smithi188_113957_object132692:head# 2024-04-23T16:33:57.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec9179e:::benchmark_data_smithi188_113957_object15494:head# 2024-04-23T16:33:57.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ec9e3c5:::benchmark_data_smithi188_113957_object21107:head# 2024-04-23T16:33:57.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ecbebc6:::benchmark_data_smithi188_113957_object96096:head# 2024-04-23T16:33:57.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ecbfcf1:::benchmark_data_smithi188_113957_object73690:head# 2024-04-23T16:33:57.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ecc00cb:::benchmark_data_smithi188_113957_object140:head# 2024-04-23T16:33:57.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eccc1ff:::benchmark_data_smithi188_113957_object110022:head# 2024-04-23T16:33:57.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eccf591:::benchmark_data_smithi188_113957_object37103:head# 2024-04-23T16:33:57.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ecd7c74:::benchmark_data_smithi188_113957_object108162:head# 2024-04-23T16:33:57.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ecd9870:::benchmark_data_smithi188_113957_object60162:head# 2024-04-23T16:33:57.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ecde0be:::benchmark_data_smithi188_113957_object29801:head# 2024-04-23T16:33:57.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ece9e45:::benchmark_data_smithi188_113957_object137350:head# 2024-04-23T16:33:57.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ecf0047:::benchmark_data_smithi188_113957_object10453:head# 2024-04-23T16:33:57.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ecf1373:::benchmark_data_smithi188_113957_object41829:head# 2024-04-23T16:33:57.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ecf5270:::benchmark_data_smithi188_113957_object67248:head# 2024-04-23T16:33:57.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ecf5415:::benchmark_data_smithi188_113957_object88079:head# 2024-04-23T16:33:57.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ecf7a36:::benchmark_data_smithi188_113957_object19261:head# 2024-04-23T16:33:57.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.378 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:56 smithi103 ceph-mon[39629]: 5.2 scrub starts 2024-04-23T16:33:57.380 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:57.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:57.434 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:33:57.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:33:57.468 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ecfb0b4:::benchmark_data_smithi188_113957_object146254:head# 2024-04-23T16:33:57.469 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.469 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed06015:::benchmark_data_smithi188_113957_object40767:head# 2024-04-23T16:33:57.469 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.469 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed08233:::benchmark_data_smithi188_113957_object54497:head# 2024-04-23T16:33:57.469 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.469 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed08550:::benchmark_data_smithi188_113957_object43425:head# 2024-04-23T16:33:57.469 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.469 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed1878c:::benchmark_data_smithi188_113957_object147639:head# 2024-04-23T16:33:57.469 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.469 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed19dd0:::benchmark_data_smithi188_113957_object53174:head# 2024-04-23T16:33:57.469 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.469 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed1ec73:::benchmark_data_smithi188_113957_object6664:head# 2024-04-23T16:33:57.470 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.470 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed1fb82:::benchmark_data_smithi188_113957_object102052:head# 2024-04-23T16:33:57.470 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.470 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed20713:::benchmark_data_smithi188_113957_object66466:head# 2024-04-23T16:33:57.470 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.470 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed2174f:::benchmark_data_smithi188_113957_object70903:head# 2024-04-23T16:33:57.470 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.470 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed2b347:::benchmark_data_smithi188_113957_object68428:head# 2024-04-23T16:33:57.470 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.470 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed30139:::benchmark_data_smithi188_113957_object34537:head# 2024-04-23T16:33:57.470 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.470 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed339a3:::benchmark_data_smithi188_113957_object131391:head# 2024-04-23T16:33:57.470 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed36453:::benchmark_data_smithi188_113957_object29040:head# 2024-04-23T16:33:57.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed3875e:::benchmark_data_smithi188_113957_object101187:head# 2024-04-23T16:33:57.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed3ad5f:::benchmark_data_smithi188_113957_object57217:head# 2024-04-23T16:33:57.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed3e5b1:::benchmark_data_smithi188_113957_object43847:head# 2024-04-23T16:33:57.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed47931:::benchmark_data_smithi188_113957_object56120:head# 2024-04-23T16:33:57.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed52273:::benchmark_data_smithi188_113957_object18404:head# 2024-04-23T16:33:57.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed52e31:::benchmark_data_smithi188_113957_object2032:head# 2024-04-23T16:33:57.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed59a36:::benchmark_data_smithi188_113957_object104278:head# 2024-04-23T16:33:57.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed5d0e7:::benchmark_data_smithi188_113957_object69522:head# 2024-04-23T16:33:57.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed6024d:::benchmark_data_smithi188_113957_object117429:head# 2024-04-23T16:33:57.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed60eda:::benchmark_data_smithi188_113957_object99486:head# 2024-04-23T16:33:57.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed6532c:::benchmark_data_smithi188_113957_object28435:head# 2024-04-23T16:33:57.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed7d276:::benchmark_data_smithi188_113957_object102804:head# 2024-04-23T16:33:57.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed80268:::benchmark_data_smithi188_113957_object85529:head# 2024-04-23T16:33:57.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed827d0:::benchmark_data_smithi188_113957_object97483:head# 2024-04-23T16:33:57.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed84bae:::benchmark_data_smithi188_113957_object43317:head# 2024-04-23T16:33:57.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed8d9de:::benchmark_data_smithi188_113957_object144956:head# 2024-04-23T16:33:57.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ed9dcc1:::benchmark_data_smithi188_113957_object6308:head# 2024-04-23T16:33:57.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eda997f:::benchmark_data_smithi188_113957_object150380:head# 2024-04-23T16:33:57.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5edafda0:::benchmark_data_smithi188_113957_object116414:head# 2024-04-23T16:33:57.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5edb288d:::benchmark_data_smithi188_113957_object135790:head# 2024-04-23T16:33:57.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5edb6e55:::benchmark_data_smithi188_113957_object56051:head# 2024-04-23T16:33:57.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5edc1e87:::benchmark_data_smithi188_113957_object119765:head# 2024-04-23T16:33:57.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5edc3644:::benchmark_data_smithi188_113957_object55030:head# 2024-04-23T16:33:57.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5edc45e5:::benchmark_data_smithi188_113957_object72022:head# 2024-04-23T16:33:57.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5edc51ca:::benchmark_data_smithi188_113957_object45753:head# 2024-04-23T16:33:57.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5edc76a7:::benchmark_data_smithi188_113957_object8758:head# 2024-04-23T16:33:57.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5edcce30:::benchmark_data_smithi188_113957_object75869:head# 2024-04-23T16:33:57.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5edd780b:::benchmark_data_smithi188_113957_object59810:head# 2024-04-23T16:33:57.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5edd995d:::benchmark_data_smithi188_113957_object45388:head# 2024-04-23T16:33:57.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ede1889:::benchmark_data_smithi188_113957_object38891:head# 2024-04-23T16:33:57.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ede6f4f:::benchmark_data_smithi188_113957_object71100:head# 2024-04-23T16:33:57.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ede992d:::benchmark_data_smithi188_113957_object68300:head# 2024-04-23T16:33:57.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5edf80fa:::benchmark_data_smithi188_113957_object64422:head# 2024-04-23T16:33:57.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5edffbb6:::benchmark_data_smithi188_113957_object106806:head# 2024-04-23T16:33:57.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ee20fa1:::benchmark_data_smithi188_113957_object66510:head# 2024-04-23T16:33:57.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ee27405:::benchmark_data_smithi188_113957_object23229:head# 2024-04-23T16:33:57.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ee29036:::benchmark_data_smithi188_113957_object113804:head# 2024-04-23T16:33:57.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ee3a4b5:::benchmark_data_smithi188_113957_object12070:head# 2024-04-23T16:33:57.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ee3cc8e:::benchmark_data_smithi188_113957_object71558:head# 2024-04-23T16:33:57.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ee3f83d:::benchmark_data_smithi188_113957_object87842:head# 2024-04-23T16:33:57.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ee400ea:::benchmark_data_smithi188_113957_object78702:head# 2024-04-23T16:33:57.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ee551e9:::benchmark_data_smithi188_113957_object39482:head# 2024-04-23T16:33:57.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ee5a1c4:::benchmark_data_smithi188_113957_object133356:head# 2024-04-23T16:33:57.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ee5d117:::benchmark_data_smithi188_113957_object10136:head# 2024-04-23T16:33:57.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ee63c61:::benchmark_data_smithi188_113957_object46287:head# 2024-04-23T16:33:57.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ee67064:::benchmark_data_smithi188_113957_object59716:head# 2024-04-23T16:33:57.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ee79344:::benchmark_data_smithi188_113957_object139073:head# 2024-04-23T16:33:57.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.478 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ee8330e:::benchmark_data_smithi188_113957_object147664:head# 2024-04-23T16:33:57.478 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.478 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ee89875:::benchmark_data_smithi188_113957_object97140:head# 2024-04-23T16:33:57.478 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.478 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ee89b17:::benchmark_data_smithi188_113957_object123882:head# 2024-04-23T16:33:57.478 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.478 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eeadbf4:::benchmark_data_smithi188_113957_object151554:head# 2024-04-23T16:33:57.478 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.478 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eeb56cf:::benchmark_data_smithi188_113957_object140582:head# 2024-04-23T16:33:57.478 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.478 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eec0e32:::benchmark_data_smithi188_113957_object34398:head# 2024-04-23T16:33:57.478 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.478 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eecc9e9:::benchmark_data_smithi188_113957_object109719:head# 2024-04-23T16:33:57.478 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eed6903:::benchmark_data_smithi188_113957_object58664:head# 2024-04-23T16:33:57.479 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eedd94a:::benchmark_data_smithi188_113957_object85102:head# 2024-04-23T16:33:57.479 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eee2391:::benchmark_data_smithi188_113957_object78262:head# 2024-04-23T16:33:57.479 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eef26bf:::benchmark_data_smithi188_113957_object139653:head# 2024-04-23T16:33:57.479 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eefc76d:::benchmark_data_smithi188_113957_object28464:head# 2024-04-23T16:33:57.479 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef05846:::benchmark_data_smithi188_113957_object38381:head# 2024-04-23T16:33:57.479 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef0aa6e:::benchmark_data_smithi188_113957_object43750:head# 2024-04-23T16:33:57.479 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.480 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef141de:::benchmark_data_smithi188_113957_object84457:head# 2024-04-23T16:33:57.480 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.480 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef166bc:::benchmark_data_smithi188_113957_object91992:head# 2024-04-23T16:33:57.480 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.480 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef19af1:::benchmark_data_smithi188_113957_object105005:head# 2024-04-23T16:33:57.480 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.480 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef19cbb:::benchmark_data_smithi188_113957_object42177:head# 2024-04-23T16:33:57.480 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.480 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef1aa7a:::benchmark_data_smithi188_113957_object55773:head# 2024-04-23T16:33:57.480 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.480 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef22541:::benchmark_data_smithi188_113957_object41185:head# 2024-04-23T16:33:57.480 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.480 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef22871:::benchmark_data_smithi188_113957_object26431:head# 2024-04-23T16:33:57.480 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.481 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef2e50e:::benchmark_data_smithi188_113957_object79306:head# 2024-04-23T16:33:57.481 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.481 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef31612:::benchmark_data_smithi188_113957_object53314:head# 2024-04-23T16:33:57.481 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.481 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef3f33f:::benchmark_data_smithi188_113957_object99656:head# 2024-04-23T16:33:57.481 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.481 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef40d78:::benchmark_data_smithi188_113957_object129:head# 2024-04-23T16:33:57.481 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.481 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef4cffc:::benchmark_data_smithi188_113957_object89736:head# 2024-04-23T16:33:57.481 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.481 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef57242:::benchmark_data_smithi188_113957_object103483:head# 2024-04-23T16:33:57.481 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.481 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef598be:::benchmark_data_smithi188_113957_object118801:head# 2024-04-23T16:33:57.481 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef5deee:::benchmark_data_smithi188_113957_object92891:head# 2024-04-23T16:33:57.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef60a95:::benchmark_data_smithi188_113957_object94042:head# 2024-04-23T16:33:57.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef61893:::benchmark_data_smithi188_113957_object338:head# 2024-04-23T16:33:57.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef6434f:::benchmark_data_smithi188_113957_object84395:head# 2024-04-23T16:33:57.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef6802e:::benchmark_data_smithi188_113957_object64379:head# 2024-04-23T16:33:57.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef69c95:::benchmark_data_smithi188_113957_object32982:head# 2024-04-23T16:33:57.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef6d493:::benchmark_data_smithi188_113957_object59039:head# 2024-04-23T16:33:57.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef70211:::benchmark_data_smithi188_113957_object118851:head# 2024-04-23T16:33:57.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef72325:::benchmark_data_smithi188_113957_object19614:head# 2024-04-23T16:33:57.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef75301:::benchmark_data_smithi188_113957_object104444:head# 2024-04-23T16:33:57.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef81201:::benchmark_data_smithi188_113957_object60972:head# 2024-04-23T16:33:57.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef88a9e:::benchmark_data_smithi188_113957_object7980:head# 2024-04-23T16:33:57.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef9286d:::benchmark_data_smithi188_113957_object121446:head# 2024-04-23T16:33:57.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ef9b081:::benchmark_data_smithi188_113957_object91445:head# 2024-04-23T16:33:57.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efa9f49:::benchmark_data_smithi188_113957_object87846:head# 2024-04-23T16:33:57.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efaa3ba:::benchmark_data_smithi188_113957_object74660:head# 2024-04-23T16:33:57.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efae8ae:::benchmark_data_smithi188_113957_object96457:head# 2024-04-23T16:33:57.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efb3c50:::benchmark_data_smithi188_113957_object141263:head# 2024-04-23T16:33:57.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efb41da:::benchmark_data_smithi188_113957_object41593:head# 2024-04-23T16:33:57.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efb6269:::benchmark_data_smithi188_113957_object115628:head# 2024-04-23T16:33:57.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efb8495:::benchmark_data_smithi188_113957_object56300:head# 2024-04-23T16:33:57.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efc255b:::benchmark_data_smithi188_113957_object86456:head# 2024-04-23T16:33:57.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efc2c52:::benchmark_data_smithi188_113957_object79131:head# 2024-04-23T16:33:57.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efc70ae:::benchmark_data_smithi188_113957_object75852:head# 2024-04-23T16:33:57.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efc932a:::benchmark_data_smithi188_113957_object98730:head# 2024-04-23T16:33:57.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efcc391:::benchmark_data_smithi188_113957_object53430:head# 2024-04-23T16:33:57.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efd39e0:::benchmark_data_smithi188_113957_object2103:head# 2024-04-23T16:33:57.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.486 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efd9cd5:::benchmark_data_smithi188_113957_object84094:head# 2024-04-23T16:33:57.486 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.486 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efe027e:::benchmark_data_smithi188_113957_object41817:head# 2024-04-23T16:33:57.486 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.486 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efe2cae:::benchmark_data_smithi188_113957_object98490:head# 2024-04-23T16:33:57.486 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.486 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efe4a96:::benchmark_data_smithi188_113957_object50482:head# 2024-04-23T16:33:57.486 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.486 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efeac35:::benchmark_data_smithi188_113957_object73366:head# 2024-04-23T16:33:57.486 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.486 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efec54e:::benchmark_data_smithi188_113957_object147829:head# 2024-04-23T16:33:57.486 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.486 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5efef530:::benchmark_data_smithi188_113957_object133776:head# 2024-04-23T16:33:57.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5eff237e:::benchmark_data_smithi188_113957_object2942:head# 2024-04-23T16:33:57.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5effbbf0:::benchmark_data_smithi188_113957_object136477:head# 2024-04-23T16:33:57.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f005ba9:::benchmark_data_smithi188_113957_object11795:head# 2024-04-23T16:33:57.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f00ee5e:::benchmark_data_smithi188_113957_object94541:head# 2024-04-23T16:33:57.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f01acf2:::benchmark_data_smithi188_113957_object115674:head# 2024-04-23T16:33:57.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f024fcd:::benchmark_data_smithi188_113957_object152718:head# 2024-04-23T16:33:57.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f02c959:::benchmark_data_smithi188_113957_object33607:head# 2024-04-23T16:33:57.488 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.488 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f02cd27:::benchmark_data_smithi188_113957_object46562:head# 2024-04-23T16:33:57.488 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.488 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f03519e:::benchmark_data_smithi188_113957_object9461:head# 2024-04-23T16:33:57.488 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.488 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f03b2fb:::benchmark_data_smithi188_113957_object142066:head# 2024-04-23T16:33:57.488 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.488 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f03e3d4:::benchmark_data_smithi188_113957_object79209:head# 2024-04-23T16:33:57.488 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.488 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f041115:::benchmark_data_smithi188_113957_object45708:head# 2024-04-23T16:33:57.488 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.488 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f04228c:::benchmark_data_smithi188_113957_object67159:head# 2024-04-23T16:33:57.488 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.489 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f04d468:::benchmark_data_smithi188_113957_object52719:head# 2024-04-23T16:33:57.489 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.489 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0619c3:::benchmark_data_smithi188_113957_object87347:head# 2024-04-23T16:33:57.489 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.489 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f07095d:::benchmark_data_smithi188_113957_object45919:head# 2024-04-23T16:33:57.489 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.489 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f07342c:::benchmark_data_smithi188_113957_object93881:head# 2024-04-23T16:33:57.489 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.489 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f07beb7:::benchmark_data_smithi188_113957_object126091:head# 2024-04-23T16:33:57.489 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.489 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0828aa:::benchmark_data_smithi188_113957_object45447:head# 2024-04-23T16:33:57.489 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.489 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f09c1bd:::benchmark_data_smithi188_113957_object143393:head# 2024-04-23T16:33:57.489 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.490 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0a19e4:::benchmark_data_smithi188_113957_object125869:head# 2024-04-23T16:33:57.490 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.490 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0a49ca:::benchmark_data_smithi188_113957_object10279:head# 2024-04-23T16:33:57.490 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.490 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0a8700:::benchmark_data_smithi188_113957_object79955:head# 2024-04-23T16:33:57.490 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.490 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0b0ec6:::benchmark_data_smithi188_113957_object23519:head# 2024-04-23T16:33:57.490 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.490 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0b54ba:::benchmark_data_smithi188_113957_object12839:head# 2024-04-23T16:33:57.490 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.490 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0b7d3f:::benchmark_data_smithi188_113957_object52175:head# 2024-04-23T16:33:57.490 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.490 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0b977d:::benchmark_data_smithi188_113957_object141994:head# 2024-04-23T16:33:57.490 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.491 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0bae4c:::benchmark_data_smithi188_113957_object31053:head# 2024-04-23T16:33:57.491 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.491 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0bc422:::benchmark_data_smithi188_113957_object21730:head# 2024-04-23T16:33:57.491 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.491 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0c29d9:::benchmark_data_smithi188_113957_object8829:head# 2024-04-23T16:33:57.491 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.491 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0c9fe7:::benchmark_data_smithi188_113957_object138962:head# 2024-04-23T16:33:57.491 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.491 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0d76d7:::benchmark_data_smithi188_113957_object39081:head# 2024-04-23T16:33:57.491 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.491 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0dc3e2:::benchmark_data_smithi188_113957_object43287:head# 2024-04-23T16:33:57.491 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.491 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0ddca4:::benchmark_data_smithi188_113957_object19288:head# 2024-04-23T16:33:57.491 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0df4cc:::benchmark_data_smithi188_113957_object59285:head# 2024-04-23T16:33:57.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0e553c:::benchmark_data_smithi188_113957_object79134:head# 2024-04-23T16:33:57.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0ec0bc:::benchmark_data_smithi188_113957_object15783:head# 2024-04-23T16:33:57.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0f1eb6:::benchmark_data_smithi188_113957_object112850:head# 2024-04-23T16:33:57.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0fb7c3:::benchmark_data_smithi188_113957_object73628:head# 2024-04-23T16:33:57.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f0fe8ed:::benchmark_data_smithi188_113957_object39435:head# 2024-04-23T16:33:57.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f101108:::benchmark_data_smithi188_113957_object95234:head# 2024-04-23T16:33:57.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f1012dd:::benchmark_data_smithi188_113957_object137371:head# 2024-04-23T16:33:57.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f10e205:::benchmark_data_smithi188_113957_object113355:head# 2024-04-23T16:33:57.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f116fe0:::benchmark_data_smithi188_113957_object20418:head# 2024-04-23T16:33:57.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f11c070:::benchmark_data_smithi188_113957_object10547:head# 2024-04-23T16:33:57.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f11fd1d:::benchmark_data_smithi188_113957_object132052:head# 2024-04-23T16:33:57.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f120a5b:::benchmark_data_smithi188_113957_object147094:head# 2024-04-23T16:33:57.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f121c92:::benchmark_data_smithi188_113957_object117054:head# 2024-04-23T16:33:57.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.494 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f12eb51:::benchmark_data_smithi188_113957_object55760:head# 2024-04-23T16:33:57.494 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.494 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f13e733:::benchmark_data_smithi188_113957_object105466:head# 2024-04-23T16:33:57.494 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.494 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f147221:::benchmark_data_smithi188_113957_object127320:head# 2024-04-23T16:33:57.494 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.494 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f14c7f0:::benchmark_data_smithi188_113957_object134850:head# 2024-04-23T16:33:57.494 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.494 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f1594d4:::benchmark_data_smithi188_113957_object88181:head# 2024-04-23T16:33:57.494 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.494 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f15cf92:::benchmark_data_smithi188_113957_object59568:head# 2024-04-23T16:33:57.494 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.494 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f15f590:::benchmark_data_smithi188_113957_object23978:head# 2024-04-23T16:33:57.494 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.495 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f161444:::benchmark_data_smithi188_113957_object95067:head# 2024-04-23T16:33:57.495 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.495 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f161903:::benchmark_data_smithi188_113957_object99013:head# 2024-04-23T16:33:57.495 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.495 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f1621f6:::benchmark_data_smithi188_113957_object43731:head# 2024-04-23T16:33:57.495 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.495 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f167f75:::benchmark_data_smithi188_113957_object46101:head# 2024-04-23T16:33:57.495 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.495 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f16c692:::benchmark_data_smithi188_113957_object77998:head# 2024-04-23T16:33:57.495 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.495 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f16f5b6:::benchmark_data_smithi188_113957_object26834:head# 2024-04-23T16:33:57.495 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.495 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f16f742:::benchmark_data_smithi188_113957_object67427:head# 2024-04-23T16:33:57.495 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.496 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f175ae8:::benchmark_data_smithi188_113957_object104497:head# 2024-04-23T16:33:57.496 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.496 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f17cd4b:::benchmark_data_smithi188_113957_object99609:head# 2024-04-23T16:33:57.496 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.496 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f184154:::benchmark_data_smithi188_113957_object58012:head# 2024-04-23T16:33:57.496 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.496 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f188746:::benchmark_data_smithi188_113957_object70681:head# 2024-04-23T16:33:57.496 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.496 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f1b29d1:::benchmark_data_smithi188_113957_object54339:head# 2024-04-23T16:33:57.496 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.496 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f1d2a7b:::benchmark_data_smithi188_113957_object90217:head# 2024-04-23T16:33:57.496 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.496 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f1d4d9d:::benchmark_data_smithi188_113957_object111150:head# 2024-04-23T16:33:57.496 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f1d8226:::benchmark_data_smithi188_113957_object23101:head# 2024-04-23T16:33:57.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f1da26f:::benchmark_data_smithi188_113957_object103079:head# 2024-04-23T16:33:57.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f1de382:::benchmark_data_smithi188_113957_object123561:head# 2024-04-23T16:33:57.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f1e754b:::benchmark_data_smithi188_113957_object39801:head# 2024-04-23T16:33:57.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f1ebbc0:::benchmark_data_smithi188_113957_object95821:head# 2024-04-23T16:33:57.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f1ee031:::benchmark_data_smithi188_113957_object34584:head# 2024-04-23T16:33:57.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f1ef57c:::benchmark_data_smithi188_113957_object112541:head# 2024-04-23T16:33:57.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f1f2937:::benchmark_data_smithi188_113957_object101053:head# 2024-04-23T16:33:57.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f2060b2:::benchmark_data_smithi188_113957_object83522:head# 2024-04-23T16:33:57.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f21cb37:::benchmark_data_smithi188_113957_object89966:head# 2024-04-23T16:33:57.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f21d011:::benchmark_data_smithi188_113957_object13742:head# 2024-04-23T16:33:57.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f227461:::benchmark_data_smithi188_113957_object87685:head# 2024-04-23T16:33:57.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f227be4:::benchmark_data_smithi188_113957_object111976:head# 2024-04-23T16:33:57.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f22dc10:::benchmark_data_smithi188_113957_object5919:head# 2024-04-23T16:33:57.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f22ff8f:::benchmark_data_smithi188_113957_object152671:head# 2024-04-23T16:33:57.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f233189:::benchmark_data_smithi188_113957_object108756:head# 2024-04-23T16:33:57.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f241152:::benchmark_data_smithi188_113957_object67623:head# 2024-04-23T16:33:57.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f24d7d3:::benchmark_data_smithi188_113957_object134095:head# 2024-04-23T16:33:57.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f252922:::benchmark_data_smithi188_113957_object63855:head# 2024-04-23T16:33:57.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f254e0e:::benchmark_data_smithi188_113957_object129342:head# 2024-04-23T16:33:57.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.500 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f255cda:::benchmark_data_smithi188_113957_object153144:head# 2024-04-23T16:33:57.500 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.500 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f269478:::benchmark_data_smithi188_113957_object34562:head# 2024-04-23T16:33:57.500 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.500 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f27fc2c:::benchmark_data_smithi188_113957_object110581:head# 2024-04-23T16:33:57.500 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.500 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f2806f9:::benchmark_data_smithi188_113957_object41932:head# 2024-04-23T16:33:57.500 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.500 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f28edf7:::benchmark_data_smithi188_113957_object9959:head# 2024-04-23T16:33:57.500 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.500 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f2939ff:::benchmark_data_smithi188_113957_object145314:head# 2024-04-23T16:33:57.500 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.500 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f299997:::benchmark_data_smithi188_113957_object107021:head# 2024-04-23T16:33:57.501 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.501 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f29a6c1:::benchmark_data_smithi188_113957_object98847:head# 2024-04-23T16:33:57.501 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.501 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f2ab85c:::benchmark_data_smithi188_113957_object41000:head# 2024-04-23T16:33:57.501 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.501 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f2ac8da:::benchmark_data_smithi188_113957_object96530:head# 2024-04-23T16:33:57.501 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.501 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f2c156c:::benchmark_data_smithi188_113957_object85188:head# 2024-04-23T16:33:57.501 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.501 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f2c209e:::benchmark_data_smithi188_113957_object27266:head# 2024-04-23T16:33:57.501 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.501 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f2cadcb:::benchmark_data_smithi188_113957_object79818:head# 2024-04-23T16:33:57.501 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.502 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f2d7192:::benchmark_data_smithi188_113957_object92114:head# 2024-04-23T16:33:57.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.502 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f2d83d9:::benchmark_data_smithi188_113957_object58401:head# 2024-04-23T16:33:57.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.502 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f2d8968:::benchmark_data_smithi188_113957_object85154:head# 2024-04-23T16:33:57.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.502 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f2df164:::benchmark_data_smithi188_113957_object143856:head# 2024-04-23T16:33:57.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.502 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f2e1327:::benchmark_data_smithi188_113957_object24571:head# 2024-04-23T16:33:57.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.502 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f2f0d09:::benchmark_data_smithi188_113957_object112634:head# 2024-04-23T16:33:57.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.502 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f2f8b56:::benchmark_data_smithi188_113957_object121662:head# 2024-04-23T16:33:57.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f300f6a:::benchmark_data_smithi188_113957_object12254:head# 2024-04-23T16:33:57.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f3070f0:::benchmark_data_smithi188_113957_object115803:head# 2024-04-23T16:33:57.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f319456:::benchmark_data_smithi188_113957_object54840:head# 2024-04-23T16:33:57.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f32bad8:::benchmark_data_smithi188_113957_object26791:head# 2024-04-23T16:33:57.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f330d47:::benchmark_data_smithi188_113957_object50762:head# 2024-04-23T16:33:57.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f33a235:::benchmark_data_smithi188_113957_object83930:head# 2024-04-23T16:33:57.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f33f91d:::benchmark_data_smithi188_113957_object127551:head# 2024-04-23T16:33:57.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f34091e:::benchmark_data_smithi188_113957_object40910:head# 2024-04-23T16:33:57.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f34693a:::benchmark_data_smithi188_113957_object141878:head# 2024-04-23T16:33:57.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f357e03:::benchmark_data_smithi188_113957_object69551:head# 2024-04-23T16:33:57.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f359746:::benchmark_data_smithi188_113957_object53794:head# 2024-04-23T16:33:57.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f35c57c:::benchmark_data_smithi188_113957_object121269:head# 2024-04-23T16:33:57.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f369e2b:::benchmark_data_smithi188_113957_object63571:head# 2024-04-23T16:33:57.505 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.505 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f36e34d:::benchmark_data_smithi188_113957_object18474:head# 2024-04-23T16:33:57.505 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.505 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f37388d:::benchmark_data_smithi188_113957_object89881:head# 2024-04-23T16:33:57.505 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.505 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f37ca4a:::benchmark_data_smithi188_113957_object7019:head# 2024-04-23T16:33:57.505 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.505 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f3875eb:::benchmark_data_smithi188_113957_object42503:head# 2024-04-23T16:33:57.505 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.505 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f389f2a:::benchmark_data_smithi188_113957_object86041:head# 2024-04-23T16:33:57.505 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.505 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f38ccf2:::benchmark_data_smithi188_113957_object140492:head# 2024-04-23T16:33:57.505 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.505 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f38fd20:::benchmark_data_smithi188_113957_object69689:head# 2024-04-23T16:33:57.506 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.506 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f394507:::benchmark_data_smithi188_113957_object119684:head# 2024-04-23T16:33:57.506 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.506 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f3a4cfc:::benchmark_data_smithi188_113957_object117214:head# 2024-04-23T16:33:57.506 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.506 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f3a5407:::benchmark_data_smithi188_113957_object130378:head# 2024-04-23T16:33:57.506 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.506 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f3afb3e:::benchmark_data_smithi188_113957_object92895:head# 2024-04-23T16:33:57.506 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.506 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f3b8f00:::benchmark_data_smithi188_113957_object145793:head# 2024-04-23T16:33:57.506 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.506 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f3be6af:::benchmark_data_smithi188_113957_object125295:head# 2024-04-23T16:33:57.506 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.506 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f3cc4d9:::benchmark_data_smithi188_113957_object77292:head# 2024-04-23T16:33:57.507 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f3d295f:::benchmark_data_smithi188_113957_object106096:head# 2024-04-23T16:33:57.507 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f3e0552:::benchmark_data_smithi188_113957_object124765:head# 2024-04-23T16:33:57.507 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f3e1155:::benchmark_data_smithi188_113957_object97195:head# 2024-04-23T16:33:57.507 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f3e4524:::benchmark_data_smithi188_113957_object57943:head# 2024-04-23T16:33:57.507 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f3e927d:::benchmark_data_smithi188_113957_object59858:head# 2024-04-23T16:33:57.507 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f3ea252:::benchmark_data_smithi188_113957_object139719:head# 2024-04-23T16:33:57.507 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f3ea3c2:::benchmark_data_smithi188_113957_object98905:head# 2024-04-23T16:33:57.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f3ec7a7:::benchmark_data_smithi188_113957_object6010:head# 2024-04-23T16:33:57.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f3faabd:::benchmark_data_smithi188_113957_object145714:head# 2024-04-23T16:33:57.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f3fe230:::benchmark_data_smithi188_113957_object126514:head# 2024-04-23T16:33:57.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f40398e:::benchmark_data_smithi188_113957_object60602:head# 2024-04-23T16:33:57.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f40700c:::benchmark_data_smithi188_113957_object49060:head# 2024-04-23T16:33:57.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f409dff:::benchmark_data_smithi188_113957_object90608:head# 2024-04-23T16:33:57.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f40b972:::benchmark_data_smithi188_113957_object6182:head# 2024-04-23T16:33:57.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f40fb30:::benchmark_data_smithi188_113957_object134725:head# 2024-04-23T16:33:57.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f414c7a:::benchmark_data_smithi188_113957_object141675:head# 2024-04-23T16:33:57.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f41b0f0:::benchmark_data_smithi188_113957_object85834:head# 2024-04-23T16:33:57.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f42afba:::benchmark_data_smithi188_113957_object46167:head# 2024-04-23T16:33:57.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f440885:::benchmark_data_smithi188_113957_object152663:head# 2024-04-23T16:33:57.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f4472d8:::benchmark_data_smithi188_113957_object149995:head# 2024-04-23T16:33:57.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f452081:::benchmark_data_smithi188_113957_object115269:head# 2024-04-23T16:33:57.510 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.510 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f45c2fa:::benchmark_data_smithi188_113957_object74128:head# 2024-04-23T16:33:57.510 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.510 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f47167b:::benchmark_data_smithi188_113957_object62715:head# 2024-04-23T16:33:57.510 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.510 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f476984:::benchmark_data_smithi188_113957_object59337:head# 2024-04-23T16:33:57.510 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.510 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f479199:::benchmark_data_smithi188_113957_object121108:head# 2024-04-23T16:33:57.510 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.510 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f47c42c:::benchmark_data_smithi188_113957_object18925:head# 2024-04-23T16:33:57.510 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.510 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f47c66d:::benchmark_data_smithi188_113957_object6383:head# 2024-04-23T16:33:57.510 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.511 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f48ba1c:::benchmark_data_smithi188_113957_object113698:head# 2024-04-23T16:33:57.511 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.511 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f49dc7d:::benchmark_data_smithi188_113957_object71940:head# 2024-04-23T16:33:57.511 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.511 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f49f3b9:::benchmark_data_smithi188_113957_object14714:head# 2024-04-23T16:33:57.511 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.511 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f4a4b1b:::benchmark_data_smithi188_113957_object73037:head# 2024-04-23T16:33:57.511 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.511 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f4ade10:::benchmark_data_smithi188_113957_object27988:head# 2024-04-23T16:33:57.511 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.511 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f4b9a07:::benchmark_data_smithi188_113957_object33329:head# 2024-04-23T16:33:57.511 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.511 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f4c7930:::benchmark_data_smithi188_113957_object96419:head# 2024-04-23T16:33:57.511 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f4c8205:::benchmark_data_smithi188_113957_object121095:head# 2024-04-23T16:33:57.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f4d01b1:::benchmark_data_smithi188_113957_object47750:head# 2024-04-23T16:33:57.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f4d4ceb:::benchmark_data_smithi188_113957_object68553:head# 2024-04-23T16:33:57.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f4e77f5:::benchmark_data_smithi188_113957_object54500:head# 2024-04-23T16:33:57.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f4f0175:::benchmark_data_smithi188_113957_object104717:head# 2024-04-23T16:33:57.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f4f1e70:::benchmark_data_smithi188_113957_object91277:head# 2024-04-23T16:33:57.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f4f970f:::benchmark_data_smithi188_113957_object33796:head# 2024-04-23T16:33:57.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f5008ce:::benchmark_data_smithi188_113957_object3618:head# 2024-04-23T16:33:57.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f509fd6:::benchmark_data_smithi188_113957_object11642:head# 2024-04-23T16:33:57.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f5116ba:::benchmark_data_smithi188_113957_object37248:head# 2024-04-23T16:33:57.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f511daa:::benchmark_data_smithi188_113957_object138257:head# 2024-04-23T16:33:57.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f519cac:::benchmark_data_smithi188_113957_object5129:head# 2024-04-23T16:33:57.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f5201cd:::benchmark_data_smithi188_113957_object10691:head# 2024-04-23T16:33:57.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f521fe8:::benchmark_data_smithi188_113957_object18538:head# 2024-04-23T16:33:57.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f530557:::benchmark_data_smithi188_113957_object26634:head# 2024-04-23T16:33:57.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f532d04:::benchmark_data_smithi188_113957_object54362:head# 2024-04-23T16:33:57.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f532d35:::benchmark_data_smithi188_113957_object104706:head# 2024-04-23T16:33:57.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f53bfef:::benchmark_data_smithi188_113957_object37866:head# 2024-04-23T16:33:57.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f541b8d:::benchmark_data_smithi188_113957_object151562:head# 2024-04-23T16:33:57.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.515 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f54686b:::benchmark_data_smithi188_113957_object88845:head# 2024-04-23T16:33:57.515 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.515 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f54933a:::benchmark_data_smithi188_113957_object48750:head# 2024-04-23T16:33:57.515 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.515 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f54f369:::benchmark_data_smithi188_113957_object29152:head# 2024-04-23T16:33:57.515 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.515 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f54f6f0:::benchmark_data_smithi188_113957_object61147:head# 2024-04-23T16:33:57.515 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.515 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f559a28:::benchmark_data_smithi188_113957_object121399:head# 2024-04-23T16:33:57.515 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.515 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f562609:::benchmark_data_smithi188_113957_object115415:head# 2024-04-23T16:33:57.515 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.515 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f56306a:::benchmark_data_smithi188_113957_object24559:head# 2024-04-23T16:33:57.515 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.516 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f56386d:::benchmark_data_smithi188_113957_object80594:head# 2024-04-23T16:33:57.516 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.516 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f579101:::benchmark_data_smithi188_113957_object2308:head# 2024-04-23T16:33:57.516 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.516 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f57b0ac:::benchmark_data_smithi188_113957_object30895:head# 2024-04-23T16:33:57.516 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.516 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f57c4a0:::benchmark_data_smithi188_113957_object101178:head# 2024-04-23T16:33:57.516 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.516 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f5915ad:::benchmark_data_smithi188_113957_object120963:head# 2024-04-23T16:33:57.516 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.516 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f59351d:::benchmark_data_smithi188_113957_object59615:head# 2024-04-23T16:33:57.516 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.516 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f598032:::benchmark_data_smithi188_113957_object141787:head# 2024-04-23T16:33:57.517 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.517 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f59a749:::benchmark_data_smithi188_113957_object87639:head# 2024-04-23T16:33:57.517 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.517 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f5aa87e:::benchmark_data_smithi188_113957_object50970:head# 2024-04-23T16:33:57.517 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.517 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f5ba281:::benchmark_data_smithi188_113957_object140286:head# 2024-04-23T16:33:57.517 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.517 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f5bd8df:::benchmark_data_smithi188_113957_object137941:head# 2024-04-23T16:33:57.517 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.517 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f5d019d:::benchmark_data_smithi188_113957_object76033:head# 2024-04-23T16:33:57.517 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.517 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f5d386c:::benchmark_data_smithi188_113957_object29769:head# 2024-04-23T16:33:57.517 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.518 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f5dc8cd:::benchmark_data_smithi188_113957_object117846:head# 2024-04-23T16:33:57.518 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.518 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f5ea3f1:::benchmark_data_smithi188_113957_object129111:head# 2024-04-23T16:33:57.518 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.518 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f5ed6d3:::benchmark_data_smithi188_113957_object39596:head# 2024-04-23T16:33:57.518 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.518 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f5ee71c:::benchmark_data_smithi188_113957_object2571:head# 2024-04-23T16:33:57.518 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.518 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f5f6443:::benchmark_data_smithi188_113957_object45189:head# 2024-04-23T16:33:57.518 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.518 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f5f6ac0:::benchmark_data_smithi188_113957_object76841:head# 2024-04-23T16:33:57.518 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.518 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f5ff722:::benchmark_data_smithi188_113957_object136348:head# 2024-04-23T16:33:57.518 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f601e42:::benchmark_data_smithi188_113957_object101578:head# 2024-04-23T16:33:57.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f60958f:::benchmark_data_smithi188_113957_object50184:head# 2024-04-23T16:33:57.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f60ee8d:::benchmark_data_smithi188_113957_object123120:head# 2024-04-23T16:33:57.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f60fc4b:::benchmark_data_smithi188_113957_object50961:head# 2024-04-23T16:33:57.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f619729:::benchmark_data_smithi188_113957_object64841:head# 2024-04-23T16:33:57.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f61b098:::benchmark_data_smithi188_113957_object35420:head# 2024-04-23T16:33:57.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f63098f:::benchmark_data_smithi188_113957_object146232:head# 2024-04-23T16:33:57.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f63841d:::benchmark_data_smithi188_113957_object117266:head# 2024-04-23T16:33:57.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f63ea47:::benchmark_data_smithi188_113957_object9796:head# 2024-04-23T16:33:57.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f645434:::benchmark_data_smithi188_113957_object106876:head# 2024-04-23T16:33:57.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f64c010:::benchmark_data_smithi188_113957_object134510:head# 2024-04-23T16:33:57.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f64d29a:::benchmark_data_smithi188_113957_object127186:head# 2024-04-23T16:33:57.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f64d9a5:::benchmark_data_smithi188_113957_object140992:head# 2024-04-23T16:33:57.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f653a8c:::benchmark_data_smithi188_113957_object13214:head# 2024-04-23T16:33:57.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f66ea3a:::benchmark_data_smithi188_113957_object139408:head# 2024-04-23T16:33:57.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f672a1e:::benchmark_data_smithi188_113957_object70412:head# 2024-04-23T16:33:57.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f679972:::benchmark_data_smithi188_113957_object53806:head# 2024-04-23T16:33:57.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f685131:::benchmark_data_smithi188_113957_object68919:head# 2024-04-23T16:33:57.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f68d226:::benchmark_data_smithi188_113957_object68666:head# 2024-04-23T16:33:57.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f68d94d:::benchmark_data_smithi188_113957_object25365:head# 2024-04-23T16:33:57.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f68e560:::benchmark_data_smithi188_113957_object2506:head# 2024-04-23T16:33:57.522 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.522 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f694ca0:::benchmark_data_smithi188_113957_object105864:head# 2024-04-23T16:33:57.522 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.522 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f6964bc:::benchmark_data_smithi188_113957_object3104:head# 2024-04-23T16:33:57.522 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.522 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f69d2d6:::benchmark_data_smithi188_113957_object111769:head# 2024-04-23T16:33:57.522 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.522 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f6a697f:::benchmark_data_smithi188_113957_object114682:head# 2024-04-23T16:33:57.522 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.522 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f6a850e:::benchmark_data_smithi188_113957_object80344:head# 2024-04-23T16:33:57.522 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.522 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f6ac9c3:::benchmark_data_smithi188_113957_object45789:head# 2024-04-23T16:33:57.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f6b2c14:::benchmark_data_smithi188_113957_object47867:head# 2024-04-23T16:33:57.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f6b5cd3:::benchmark_data_smithi188_113957_object139801:head# 2024-04-23T16:33:57.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f6c6450:::benchmark_data_smithi188_113957_object1509:head# 2024-04-23T16:33:57.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f6c7601:::benchmark_data_smithi188_113957_object111260:head# 2024-04-23T16:33:57.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f6c7dbc:::benchmark_data_smithi188_113957_object17126:head# 2024-04-23T16:33:57.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f6dbed0:::benchmark_data_smithi188_113957_object116489:head# 2024-04-23T16:33:57.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.523 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f6ec638:::benchmark_data_smithi188_113957_object33003:head# 2024-04-23T16:33:57.523 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.524 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f6eedcb:::benchmark_data_smithi188_113957_object140818:head# 2024-04-23T16:33:57.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.524 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f6efa7f:::benchmark_data_smithi188_113957_object42829:head# 2024-04-23T16:33:57.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.524 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f6f32dc:::benchmark_data_smithi188_113957_object20124:head# 2024-04-23T16:33:57.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.524 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f6f7a26:::benchmark_data_smithi188_113957_object149591:head# 2024-04-23T16:33:57.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.524 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f6f818b:::benchmark_data_smithi188_113957_object38510:head# 2024-04-23T16:33:57.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.524 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f6f9376:::benchmark_data_smithi188_113957_object96288:head# 2024-04-23T16:33:57.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.524 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f702498:::benchmark_data_smithi188_113957_object148441:head# 2024-04-23T16:33:57.524 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.525 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f704e85:::benchmark_data_smithi188_113957_object16984:head# 2024-04-23T16:33:57.525 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.525 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f706125:::benchmark_data_smithi188_113957_object143046:head# 2024-04-23T16:33:57.525 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.525 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f719ea4:::benchmark_data_smithi188_113957_object12526:head# 2024-04-23T16:33:57.525 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.525 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f71a932:::benchmark_data_smithi188_113957_object110886:head# 2024-04-23T16:33:57.525 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.525 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f71e7b8:::benchmark_data_smithi188_113957_object55872:head# 2024-04-23T16:33:57.525 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.525 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f7237b7:::benchmark_data_smithi188_113957_object129002:head# 2024-04-23T16:33:57.525 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.525 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f734548:::benchmark_data_smithi188_113957_object67367:head# 2024-04-23T16:33:57.526 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.526 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f7352c2:::benchmark_data_smithi188_113957_object130948:head# 2024-04-23T16:33:57.526 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.526 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f735f86:::benchmark_data_smithi188_113957_object105807:head# 2024-04-23T16:33:57.526 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.526 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f737b95:::benchmark_data_smithi188_113957_object127526:head# 2024-04-23T16:33:57.526 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.526 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f7381f4:::benchmark_data_smithi188_113957_object56823:head# 2024-04-23T16:33:57.526 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.526 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f740f8d:::benchmark_data_smithi188_113957_object45662:head# 2024-04-23T16:33:57.526 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.526 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f745c69:::benchmark_data_smithi188_113957_object78655:head# 2024-04-23T16:33:57.527 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.527 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f7505db:::benchmark_data_smithi188_113957_object98034:head# 2024-04-23T16:33:57.527 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.527 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f775292:::benchmark_data_smithi188_113957_object29725:head# 2024-04-23T16:33:57.527 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.527 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f775ba6:::benchmark_data_smithi188_113957_object149878:head# 2024-04-23T16:33:57.527 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.527 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f77cc52:::benchmark_data_smithi188_113957_object51742:head# 2024-04-23T16:33:57.527 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.527 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f78aef6:::benchmark_data_smithi188_113957_object37646:head# 2024-04-23T16:33:57.527 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.527 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f78c60e:::benchmark_data_smithi188_113957_object93372:head# 2024-04-23T16:33:57.528 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.528 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f79c8a1:::benchmark_data_smithi188_113957_object94082:head# 2024-04-23T16:33:57.528 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.528 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f7a0828:::benchmark_data_smithi188_113957_object44331:head# 2024-04-23T16:33:57.528 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.528 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f7a71ba:::benchmark_data_smithi188_113957_object120634:head# 2024-04-23T16:33:57.528 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.528 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f7a9555:::benchmark_data_smithi188_113957_object73661:head# 2024-04-23T16:33:57.528 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.528 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f7bdfc4:::benchmark_data_smithi188_113957_object49501:head# 2024-04-23T16:33:57.528 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.528 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f7be3ca:::benchmark_data_smithi188_113957_object49544:head# 2024-04-23T16:33:57.528 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.529 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f7c52fa:::benchmark_data_smithi188_113957_object133918:head# 2024-04-23T16:33:57.529 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.529 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f7cfcb7:::benchmark_data_smithi188_113957_object88251:head# 2024-04-23T16:33:57.529 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.529 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f7e3d2a:::benchmark_data_smithi188_113957_object112982:head# 2024-04-23T16:33:57.529 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.529 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f7ee3b9:::benchmark_data_smithi188_113957_object68050:head# 2024-04-23T16:33:57.529 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.529 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f7f0086:::benchmark_data_smithi188_113957_object98419:head# 2024-04-23T16:33:57.529 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.529 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f7f01c2:::benchmark_data_smithi188_113957_object97721:head# 2024-04-23T16:33:57.529 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.529 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f7fcbe4:::benchmark_data_smithi188_113957_object52377:head# 2024-04-23T16:33:57.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f7ff097:::benchmark_data_smithi188_113957_object33449:head# 2024-04-23T16:33:57.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f802091:::benchmark_data_smithi188_113957_object59496:head# 2024-04-23T16:33:57.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f802b6d:::benchmark_data_smithi188_113957_object128363:head# 2024-04-23T16:33:57.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f813831:::benchmark_data_smithi188_113957_object88802:head# 2024-04-23T16:33:57.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f81cd03:::benchmark_data_smithi188_113957_object88978:head# 2024-04-23T16:33:57.530 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.530 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f820d32:::benchmark_data_smithi188_113957_object28193:head# 2024-04-23T16:33:57.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f8237b5:::benchmark_data_smithi188_113957_object74636:head# 2024-04-23T16:33:57.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f82dacb:::benchmark_data_smithi188_113957_object81492:head# 2024-04-23T16:33:57.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f835f39:::benchmark_data_smithi188_113957_object105461:head# 2024-04-23T16:33:57.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f83640f:::benchmark_data_smithi188_113957_object15241:head# 2024-04-23T16:33:57.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f8399ab:::benchmark_data_smithi188_113957_object151355:head# 2024-04-23T16:33:57.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f840454:::benchmark_data_smithi188_113957_object18895:head# 2024-04-23T16:33:57.531 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.531 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f840fa6:::benchmark_data_smithi188_113957_object74447:head# 2024-04-23T16:33:57.532 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.532 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f844bbf:::benchmark_data_smithi188_113957_object106815:head# 2024-04-23T16:33:57.532 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.532 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f84933f:::benchmark_data_smithi188_113957_object31475:head# 2024-04-23T16:33:57.532 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.532 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f8552f4:::benchmark_data_smithi188_113957_object65219:head# 2024-04-23T16:33:57.532 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.532 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f86323a:::benchmark_data_smithi188_113957_object76388:head# 2024-04-23T16:33:57.532 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.532 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f864966:::benchmark_data_smithi188_113957_object136903:head# 2024-04-23T16:33:57.532 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.532 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f869480:::benchmark_data_smithi188_113957_object91684:head# 2024-04-23T16:33:57.532 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.533 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f87dcc6:::benchmark_data_smithi188_113957_object116199:head# 2024-04-23T16:33:57.533 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.533 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f87f314:::benchmark_data_smithi188_113957_object129712:head# 2024-04-23T16:33:57.533 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.533 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f87fc1c:::benchmark_data_smithi188_113957_object116326:head# 2024-04-23T16:33:57.533 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.533 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f889e58:::benchmark_data_smithi188_113957_object72388:head# 2024-04-23T16:33:57.533 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.533 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f893cec:::benchmark_data_smithi188_113957_object127410:head# 2024-04-23T16:33:57.533 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.533 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f895c71:::benchmark_data_smithi188_113957_object127411:head# 2024-04-23T16:33:57.533 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.533 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f89b4e8:::benchmark_data_smithi188_113957_object109639:head# 2024-04-23T16:33:57.533 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.534 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f89d519:::benchmark_data_smithi188_113957_object65195:head# 2024-04-23T16:33:57.534 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.534 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f8a1d33:::benchmark_data_smithi188_113957_object1687:head# 2024-04-23T16:33:57.534 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.534 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f8a8117:::benchmark_data_smithi188_113957_object86043:head# 2024-04-23T16:33:57.534 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.534 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f8ac0cf:::benchmark_data_smithi188_113957_object116570:head# 2024-04-23T16:33:57.534 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.534 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f8c6c3d:::benchmark_data_smithi188_113957_object58216:head# 2024-04-23T16:33:57.534 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.534 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f8c6e25:::benchmark_data_smithi188_113957_object30600:head# 2024-04-23T16:33:57.534 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.534 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f8cb159:::benchmark_data_smithi188_113957_object97844:head# 2024-04-23T16:33:57.534 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.535 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f8d9756:::benchmark_data_smithi188_113957_object9495:head# 2024-04-23T16:33:57.535 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.535 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f8dca4a:::benchmark_data_smithi188_113957_object125028:head# 2024-04-23T16:33:57.535 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.535 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f8edff1:::benchmark_data_smithi188_113957_object310:head# 2024-04-23T16:33:57.535 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.535 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f8f27b1:::benchmark_data_smithi188_113957_object111776:head# 2024-04-23T16:33:57.535 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.535 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f8fb07b:::benchmark_data_smithi188_113957_object39290:head# 2024-04-23T16:33:57.535 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.535 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f8fcaf6:::benchmark_data_smithi188_113957_object72599:head# 2024-04-23T16:33:57.535 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.535 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f8fdce8:::benchmark_data_smithi188_113957_object61552:head# 2024-04-23T16:33:57.536 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.536 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f908753:::benchmark_data_smithi188_113957_object95919:head# 2024-04-23T16:33:57.536 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.536 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f90da4c:::benchmark_data_smithi188_113957_object88518:head# 2024-04-23T16:33:57.536 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.536 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f910515:::benchmark_data_smithi188_113957_object354:head# 2024-04-23T16:33:57.536 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.536 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f910800:::benchmark_data_smithi188_113957_object135641:head# 2024-04-23T16:33:57.536 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.536 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f912ee8:::benchmark_data_smithi188_113957_object150795:head# 2024-04-23T16:33:57.536 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.536 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f91bd63:::benchmark_data_smithi188_113957_object136469:head# 2024-04-23T16:33:57.536 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.536 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f92d8e3:::benchmark_data_smithi188_113957_object107703:head# 2024-04-23T16:33:57.537 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.537 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f9384c6:::benchmark_data_smithi188_113957_object35542:head# 2024-04-23T16:33:57.537 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.537 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f93bc7b:::benchmark_data_smithi188_113957_object7537:head# 2024-04-23T16:33:57.537 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.537 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f93d076:::benchmark_data_smithi188_113957_object29987:head# 2024-04-23T16:33:57.537 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.537 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f93dcb9:::benchmark_data_smithi188_113957_object5493:head# 2024-04-23T16:33:57.537 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.537 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f93e9a6:::benchmark_data_smithi188_113957_object51955:head# 2024-04-23T16:33:57.537 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.537 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f93fc08:::benchmark_data_smithi188_113957_object149006:head# 2024-04-23T16:33:57.537 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.537 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f944fc8:::benchmark_data_smithi188_113957_object92587:head# 2024-04-23T16:33:57.538 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.538 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f950783:::benchmark_data_smithi188_113957_object83328:head# 2024-04-23T16:33:57.538 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.538 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f955091:::benchmark_data_smithi188_113957_object46493:head# 2024-04-23T16:33:57.538 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.538 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f95547d:::benchmark_data_smithi188_113957_object148747:head# 2024-04-23T16:33:57.538 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.538 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f95b385:::benchmark_data_smithi188_113957_object146869:head# 2024-04-23T16:33:57.538 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.538 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f95fed3:::benchmark_data_smithi188_113957_object142557:head# 2024-04-23T16:33:57.538 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.538 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f96208a:::benchmark_data_smithi188_113957_object33535:head# 2024-04-23T16:33:57.538 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.538 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f96306b:::benchmark_data_smithi188_113957_object62757:head# 2024-04-23T16:33:57.539 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.539 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f9647d9:::benchmark_data_smithi188_113957_object65458:head# 2024-04-23T16:33:57.539 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.539 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f968b82:::benchmark_data_smithi188_113957_object137124:head# 2024-04-23T16:33:57.539 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.539 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f969de7:::benchmark_data_smithi188_113957_object139807:head# 2024-04-23T16:33:57.539 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.539 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f97b31e:::benchmark_data_smithi188_113957_object41083:head# 2024-04-23T16:33:57.539 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.539 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f99e91a:::benchmark_data_smithi188_113957_object38566:head# 2024-04-23T16:33:57.539 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.539 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f9a3ac3:::benchmark_data_smithi188_113957_object149730:head# 2024-04-23T16:33:57.539 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.539 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f9a3e4b:::benchmark_data_smithi188_113957_object124412:head# 2024-04-23T16:33:57.540 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.540 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f9a44bb:::benchmark_data_smithi188_113957_object97580:head# 2024-04-23T16:33:57.540 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.540 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f9a9c46:::benchmark_data_smithi188_113957_object51165:head# 2024-04-23T16:33:57.540 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.540 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f9b8534:::benchmark_data_smithi188_113957_object106021:head# 2024-04-23T16:33:57.540 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.540 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f9c81b7:::benchmark_data_smithi188_113957_object100012:head# 2024-04-23T16:33:57.540 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.540 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f9d1a4b:::benchmark_data_smithi188_113957_object3147:head# 2024-04-23T16:33:57.540 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.540 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f9d223a:::benchmark_data_smithi188_113957_object48426:head# 2024-04-23T16:33:57.540 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.540 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f9d76f2:::benchmark_data_smithi188_113957_object147952:head# 2024-04-23T16:33:57.541 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.541 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f9db1cb:::benchmark_data_smithi188_113957_object58970:head# 2024-04-23T16:33:57.541 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.541 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f9e7f59:::benchmark_data_smithi188_113957_object103164:head# 2024-04-23T16:33:57.541 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.541 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f9f0d31:::benchmark_data_smithi188_113957_object98786:head# 2024-04-23T16:33:57.541 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.541 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f9f315f:::benchmark_data_smithi188_113957_object50107:head# 2024-04-23T16:33:57.541 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.541 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f9f3820:::benchmark_data_smithi188_113957_object47408:head# 2024-04-23T16:33:57.541 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.541 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5f9fe05b:::benchmark_data_smithi188_113957_object136530:head# 2024-04-23T16:33:57.541 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.541 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fa135e6:::benchmark_data_smithi188_113957_object1691:head# 2024-04-23T16:33:57.542 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.542 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fa1a4e9:::benchmark_data_smithi188_113957_object139636:head# 2024-04-23T16:33:57.542 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.542 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fa1d45e:::benchmark_data_smithi188_113957_object89662:head# 2024-04-23T16:33:57.542 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.542 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fa1d996:::benchmark_data_smithi188_113957_object95329:head# 2024-04-23T16:33:57.542 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.542 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fa1e617:::benchmark_data_smithi188_113957_object9118:head# 2024-04-23T16:33:57.542 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.542 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fa215c4:::benchmark_data_smithi188_113957_object119824:head# 2024-04-23T16:33:57.542 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.542 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fa24110:::benchmark_data_smithi188_113957_object20708:head# 2024-04-23T16:33:57.542 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.543 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fa33820:::benchmark_data_smithi188_113957_object24535:head# 2024-04-23T16:33:57.543 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.543 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fa416e2:::benchmark_data_smithi188_113957_object107934:head# 2024-04-23T16:33:57.543 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.543 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fa5860a:::benchmark_data_smithi188_113957_object86815:head# 2024-04-23T16:33:57.543 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.543 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fa58778:::benchmark_data_smithi188_113957_object138311:head# 2024-04-23T16:33:57.543 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.543 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fa5ab69:::benchmark_data_smithi188_113957_object48706:head# 2024-04-23T16:33:57.543 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.543 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fa6edd1:::benchmark_data_smithi188_113957_object32173:head# 2024-04-23T16:33:57.543 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.543 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fa743fd:::benchmark_data_smithi188_113957_object38459:head# 2024-04-23T16:33:57.543 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.544 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fa91300:::benchmark_data_smithi188_113957_object36488:head# 2024-04-23T16:33:57.544 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.544 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fa961dd:::benchmark_data_smithi188_113957_object131292:head# 2024-04-23T16:33:57.544 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.544 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fa98c66:::benchmark_data_smithi188_113957_object2804:head# 2024-04-23T16:33:57.544 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.544 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fa9be6b:::benchmark_data_smithi188_113957_object131798:head# 2024-04-23T16:33:57.544 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.544 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5faa3072:::benchmark_data_smithi188_113957_object6525:head# 2024-04-23T16:33:57.544 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.544 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fab8c24:::benchmark_data_smithi188_113957_object64380:head# 2024-04-23T16:33:57.544 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.544 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fabc7be:::benchmark_data_smithi188_113957_object107432:head# 2024-04-23T16:33:57.544 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.545 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fac1fbe:::benchmark_data_smithi188_113957_object78479:head# 2024-04-23T16:33:57.545 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.545 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5facd5b3:::benchmark_data_smithi188_113957_object48576:head# 2024-04-23T16:33:57.545 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.545 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5facdcfe:::benchmark_data_smithi188_113957_object46771:head# 2024-04-23T16:33:57.545 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.545 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5facfe74:::benchmark_data_smithi188_113957_object78554:head# 2024-04-23T16:33:57.545 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.545 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fad4bf6:::benchmark_data_smithi188_113957_object55579:head# 2024-04-23T16:33:57.545 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.545 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fad4f79:::benchmark_data_smithi188_113957_object81359:head# 2024-04-23T16:33:57.545 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.545 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fadc125:::benchmark_data_smithi188_113957_object25228:head# 2024-04-23T16:33:57.545 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.545 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fae2558:::benchmark_data_smithi188_113957_object148910:head# 2024-04-23T16:33:57.546 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.546 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5faedd7f:::benchmark_data_smithi188_113957_object107595:head# 2024-04-23T16:33:57.546 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.546 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5faeef46:::benchmark_data_smithi188_113957_object54828:head# 2024-04-23T16:33:57.546 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.546 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5faf3a31:::benchmark_data_smithi188_113957_object107794:head# 2024-04-23T16:33:57.546 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.546 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5faf6350:::benchmark_data_smithi188_113957_object115282:head# 2024-04-23T16:33:57.546 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.546 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb00625:::benchmark_data_smithi188_113957_object88330:head# 2024-04-23T16:33:57.546 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.546 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb0075f:::benchmark_data_smithi188_113957_object70778:head# 2024-04-23T16:33:57.546 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.547 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb052cd:::benchmark_data_smithi188_113957_object51578:head# 2024-04-23T16:33:57.547 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.547 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb1d89f:::benchmark_data_smithi188_113957_object11424:head# 2024-04-23T16:33:57.547 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.547 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb1dbd1:::benchmark_data_smithi188_113957_object30628:head# 2024-04-23T16:33:57.547 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.549 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb1ff96:::benchmark_data_smithi188_113957_object91800:head# 2024-04-23T16:33:57.549 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.549 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb20c68:::benchmark_data_smithi188_113957_object26052:head# 2024-04-23T16:33:57.549 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.550 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb24800:::benchmark_data_smithi188_113957_object97006:head# 2024-04-23T16:33:57.550 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.550 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb2582e:::benchmark_data_smithi188_113957_object74382:head# 2024-04-23T16:33:57.550 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.550 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb29286:::benchmark_data_smithi188_113957_object63417:head# 2024-04-23T16:33:57.550 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.550 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb3628f:::benchmark_data_smithi188_113957_object56411:head# 2024-04-23T16:33:57.550 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.550 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb3ef9a:::benchmark_data_smithi188_113957_object93195:head# 2024-04-23T16:33:57.550 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.550 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb432a1:::benchmark_data_smithi188_113957_object3038:head# 2024-04-23T16:33:57.550 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.550 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb528f1:::benchmark_data_smithi188_113957_object89328:head# 2024-04-23T16:33:57.551 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.551 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb55501:::benchmark_data_smithi188_113957_object73152:head# 2024-04-23T16:33:57.551 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.551 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb570f3:::benchmark_data_smithi188_113957_object26640:head# 2024-04-23T16:33:57.551 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.551 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb5bad5:::benchmark_data_smithi188_113957_object60319:head# 2024-04-23T16:33:57.551 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.551 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb68898:::benchmark_data_smithi188_113957_object84143:head# 2024-04-23T16:33:57.551 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.551 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb74495:::benchmark_data_smithi188_113957_object141853:head# 2024-04-23T16:33:57.551 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.551 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb7d4d7:::benchmark_data_smithi188_113957_object88405:head# 2024-04-23T16:33:57.551 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.552 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb804f9:::benchmark_data_smithi188_113957_object148981:head# 2024-04-23T16:33:57.552 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.552 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb88400:::benchmark_data_smithi188_113957_object126898:head# 2024-04-23T16:33:57.552 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.552 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb949de:::benchmark_data_smithi188_113957_object94687:head# 2024-04-23T16:33:57.552 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.552 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fb99d20:::benchmark_data_smithi188_113957_object138468:head# 2024-04-23T16:33:57.552 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.552 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fba32ff:::benchmark_data_smithi188_113957_object12510:head# 2024-04-23T16:33:57.552 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.552 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fbae5e1:::benchmark_data_smithi188_113957_object85826:head# 2024-04-23T16:33:57.552 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.552 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fbb2df7:::benchmark_data_smithi188_113957_object67784:head# 2024-04-23T16:33:57.552 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.552 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fbb57d9:::benchmark_data_smithi188_113957_object30923:head# 2024-04-23T16:33:57.553 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.553 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fbba091:::benchmark_data_smithi188_113957_object52584:head# 2024-04-23T16:33:57.553 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.553 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fbc0fb3:::benchmark_data_smithi188_113957_object58006:head# 2024-04-23T16:33:57.553 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.553 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fbc3278:::benchmark_data_smithi188_113957_object152201:head# 2024-04-23T16:33:57.553 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.553 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fbd2f80:::benchmark_data_smithi188_113957_object110253:head# 2024-04-23T16:33:57.553 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.553 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fbdcbf5:::benchmark_data_smithi188_113957_object153457:head# 2024-04-23T16:33:57.553 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.553 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fbde273:::benchmark_data_smithi188_113957_object140261:head# 2024-04-23T16:33:57.553 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.553 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fbf4989:::benchmark_data_smithi188_113957_object72301:head# 2024-04-23T16:33:57.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fbf7dc0:::benchmark_data_smithi188_113957_object31001:head# 2024-04-23T16:33:57.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fbfe31b:::benchmark_data_smithi188_113957_object93164:head# 2024-04-23T16:33:57.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc03d51:::benchmark_data_smithi188_113957_object29526:head# 2024-04-23T16:33:57.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc0d485:::benchmark_data_smithi188_113957_object54206:head# 2024-04-23T16:33:57.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc1199d:::benchmark_data_smithi188_113957_object92554:head# 2024-04-23T16:33:57.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc1ee08:::benchmark_data_smithi188_113957_object111560:head# 2024-04-23T16:33:57.554 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.554 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc2492e:::benchmark_data_smithi188_113957_object112294:head# 2024-04-23T16:33:57.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc2fcc4:::benchmark_data_smithi188_113957_object118681:head# 2024-04-23T16:33:57.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc3085b:::benchmark_data_smithi188_113957_object104842:head# 2024-04-23T16:33:57.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc3551e:::benchmark_data_smithi188_113957_object37098:head# 2024-04-23T16:33:57.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc3da93:::benchmark_data_smithi188_113957_object27626:head# 2024-04-23T16:33:57.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc4c7be:::benchmark_data_smithi188_113957_object23894:head# 2024-04-23T16:33:57.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc52889:::benchmark_data_smithi188_113957_object46773:head# 2024-04-23T16:33:57.555 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.555 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc55b64:::benchmark_data_smithi188_113957_object118544:head# 2024-04-23T16:33:57.556 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.556 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc57a16:::benchmark_data_smithi188_113957_object105113:head# 2024-04-23T16:33:57.556 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.556 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc580d7:::benchmark_data_smithi188_113957_object143602:head# 2024-04-23T16:33:57.556 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.556 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc67e4f:::benchmark_data_smithi188_113957_object83802:head# 2024-04-23T16:33:57.556 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.556 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc6b293:::benchmark_data_smithi188_113957_object112055:head# 2024-04-23T16:33:57.556 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.556 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc6b637:::benchmark_data_smithi188_113957_object59562:head# 2024-04-23T16:33:57.556 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.556 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc74826:::benchmark_data_smithi188_113957_object85912:head# 2024-04-23T16:33:57.556 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.556 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc787fe:::benchmark_data_smithi188_113957_object78769:head# 2024-04-23T16:33:57.557 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.557 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc904f8:::benchmark_data_smithi188_113957_object133137:head# 2024-04-23T16:33:57.557 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.557 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fc91af5:::benchmark_data_smithi188_113957_object42897:head# 2024-04-23T16:33:57.557 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.557 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fca1116:::benchmark_data_smithi188_113957_object52781:head# 2024-04-23T16:33:57.557 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.557 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fca4cd9:::benchmark_data_smithi188_113957_object145176:head# 2024-04-23T16:33:57.557 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.557 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fca5c4f:::benchmark_data_smithi188_113957_object88235:head# 2024-04-23T16:33:57.557 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.557 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fca7d5a:::benchmark_data_smithi188_113957_object15256:head# 2024-04-23T16:33:57.557 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.557 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fcaa477:::benchmark_data_smithi188_113957_object145428:head# 2024-04-23T16:33:57.558 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.558 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fcadb9a:::benchmark_data_smithi188_113957_object97933:head# 2024-04-23T16:33:57.558 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.558 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fcafa07:::benchmark_data_smithi188_113957_object12351:head# 2024-04-23T16:33:57.558 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.558 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fcbbe6d:::benchmark_data_smithi188_113957_object108014:head# 2024-04-23T16:33:57.558 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.558 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fcc11f1:::benchmark_data_smithi188_113957_object115962:head# 2024-04-23T16:33:57.558 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.558 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fcc6cf2:::benchmark_data_smithi188_113957_object132050:head# 2024-04-23T16:33:57.558 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.558 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fcc7d1d:::benchmark_data_smithi188_113957_object149306:head# 2024-04-23T16:33:57.558 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fccf98f:::benchmark_data_smithi188_113957_object112389:head# 2024-04-23T16:33:57.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fcdf1a9:::benchmark_data_smithi188_113957_object61412:head# 2024-04-23T16:33:57.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fce3453:::benchmark_data_smithi188_113957_object55653:head# 2024-04-23T16:33:57.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fce501d:::benchmark_data_smithi188_113957_object89003:head# 2024-04-23T16:33:57.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fce9b07:::benchmark_data_smithi188_113957_object72203:head# 2024-04-23T16:33:57.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fceb3b6:::benchmark_data_smithi188_113957_object51347:head# 2024-04-23T16:33:57.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.559 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fcee2f5:::benchmark_data_smithi188_113957_object64031:head# 2024-04-23T16:33:57.559 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.560 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fcf3990:::benchmark_data_smithi188_113957_object142994:head# 2024-04-23T16:33:57.560 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.560 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fcf53ed:::benchmark_data_smithi188_113957_object7650:head# 2024-04-23T16:33:57.560 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.560 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fcf8418:::benchmark_data_smithi188_113957_object49013:head# 2024-04-23T16:33:57.560 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.560 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fcfeb8a:::benchmark_data_smithi188_113957_object126701:head# 2024-04-23T16:33:57.560 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.560 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd018bf:::benchmark_data_smithi188_113957_object83336:head# 2024-04-23T16:33:57.560 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.560 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd03651:::benchmark_data_smithi188_113957_object76641:head# 2024-04-23T16:33:57.560 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.560 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd04a45:::benchmark_data_smithi188_113957_object48512:head# 2024-04-23T16:33:57.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd14d74:::benchmark_data_smithi188_113957_object106429:head# 2024-04-23T16:33:57.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd17311:::benchmark_data_smithi188_113957_object135258:head# 2024-04-23T16:33:57.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd1b3de:::benchmark_data_smithi188_113957_object113443:head# 2024-04-23T16:33:57.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd1d25f:::benchmark_data_smithi188_113957_object132407:head# 2024-04-23T16:33:57.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd20415:::benchmark_data_smithi188_113957_object150460:head# 2024-04-23T16:33:57.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd23ed7:::benchmark_data_smithi188_113957_object4252:head# 2024-04-23T16:33:57.561 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.561 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd24d7f:::benchmark_data_smithi188_113957_object129697:head# 2024-04-23T16:33:57.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd33909:::benchmark_data_smithi188_113957_object109621:head# 2024-04-23T16:33:57.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd357f1:::benchmark_data_smithi188_113957_object2838:head# 2024-04-23T16:33:57.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd41743:::benchmark_data_smithi188_113957_object133236:head# 2024-04-23T16:33:57.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd48e80:::benchmark_data_smithi188_113957_object130146:head# 2024-04-23T16:33:57.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd49a8b:::benchmark_data_smithi188_113957_object98545:head# 2024-04-23T16:33:57.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd4a02a:::benchmark_data_smithi188_113957_object132552:head# 2024-04-23T16:33:57.562 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.562 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd4b866:::benchmark_data_smithi188_113957_object87002:head# 2024-04-23T16:33:57.563 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.563 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd4c315:::benchmark_data_smithi188_113957_object26402:head# 2024-04-23T16:33:57.563 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.563 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd588bc:::benchmark_data_smithi188_113957_object95724:head# 2024-04-23T16:33:57.563 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.563 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd5a7e4:::benchmark_data_smithi188_113957_object45249:head# 2024-04-23T16:33:57.563 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.563 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd5e042:::benchmark_data_smithi188_113957_object61657:head# 2024-04-23T16:33:57.563 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.563 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd5e401:::benchmark_data_smithi188_113957_object71756:head# 2024-04-23T16:33:57.563 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.563 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd5eb50:::benchmark_data_smithi188_113957_object145739:head# 2024-04-23T16:33:57.563 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.563 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd60964:::benchmark_data_smithi188_113957_object1980:head# 2024-04-23T16:33:57.564 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.564 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd62bdf:::benchmark_data_smithi188_113957_object66384:head# 2024-04-23T16:33:57.564 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.564 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd6ffa6:::benchmark_data_smithi188_113957_object43274:head# 2024-04-23T16:33:57.564 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.564 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd772fb:::benchmark_data_smithi188_113957_object153489:head# 2024-04-23T16:33:57.564 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.564 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd7b9b9:::benchmark_data_smithi188_113957_object56001:head# 2024-04-23T16:33:57.564 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.564 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd7f0d3:::benchmark_data_smithi188_113957_object130453:head# 2024-04-23T16:33:57.564 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.564 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd821ea:::benchmark_data_smithi188_113957_object117970:head# 2024-04-23T16:33:57.564 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.564 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd897d5:::benchmark_data_smithi188_113957_object77868:head# 2024-04-23T16:33:57.565 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.565 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fd8f621:::benchmark_data_smithi188_113957_object27411:head# 2024-04-23T16:33:57.565 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.565 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fda1a12:::benchmark_data_smithi188_113957_object66464:head# 2024-04-23T16:33:57.565 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.565 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fda74e4:::benchmark_data_smithi188_113957_object125636:head# 2024-04-23T16:33:57.565 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.565 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fdac9a9:::benchmark_data_smithi188_113957_object118339:head# 2024-04-23T16:33:57.565 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.565 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fdaea41:::benchmark_data_smithi188_113957_object4877:head# 2024-04-23T16:33:57.565 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.565 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fdbca60:::benchmark_data_smithi188_113957_object24456:head# 2024-04-23T16:33:57.565 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.565 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fdc5c33:::benchmark_data_smithi188_113957_object70496:head# 2024-04-23T16:33:57.566 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.566 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fdcab9f:::benchmark_data_smithi188_113957_object32529:head# 2024-04-23T16:33:57.566 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.566 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fdcb7e9:::benchmark_data_smithi188_113957_object29059:head# 2024-04-23T16:33:57.566 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.566 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fdd63f5:::benchmark_data_smithi188_113957_object80011:head# 2024-04-23T16:33:57.566 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.566 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fdefe53:::benchmark_data_smithi188_113957_object76407:head# 2024-04-23T16:33:57.566 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.566 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fdf0232:::benchmark_data_smithi188_113957_object135343:head# 2024-04-23T16:33:57.566 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.566 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fdf2168:::benchmark_data_smithi188_113957_object49478:head# 2024-04-23T16:33:57.566 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.566 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe01129:::benchmark_data_smithi188_113957_object95525:head# 2024-04-23T16:33:57.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe04759:::benchmark_data_smithi188_113957_object34384:head# 2024-04-23T16:33:57.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe0937c:::benchmark_data_smithi188_113957_object129043:head# 2024-04-23T16:33:57.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe0ee8f:::benchmark_data_smithi188_113957_object43241:head# 2024-04-23T16:33:57.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe15fa5:::benchmark_data_smithi188_113957_object39643:head# 2024-04-23T16:33:57.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe1d2f0:::benchmark_data_smithi188_113957_object87564:head# 2024-04-23T16:33:57.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe1d720:::benchmark_data_smithi188_113957_object94502:head# 2024-04-23T16:33:57.567 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.567 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe1f2ec:::benchmark_data_smithi188_113957_object30131:head# 2024-04-23T16:33:57.568 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.568 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe227f0:::benchmark_data_smithi188_113957_object35777:head# 2024-04-23T16:33:57.568 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.568 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe290cd:::benchmark_data_smithi188_113957_object65446:head# 2024-04-23T16:33:57.568 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.568 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe2ab42:::benchmark_data_smithi188_113957_object17531:head# 2024-04-23T16:33:57.568 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.568 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe31d09:::benchmark_data_smithi188_113957_object80520:head# 2024-04-23T16:33:57.568 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.568 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe3facd:::benchmark_data_smithi188_113957_object147835:head# 2024-04-23T16:33:57.568 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.568 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe4e39e:::benchmark_data_smithi188_113957_object50356:head# 2024-04-23T16:33:57.568 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.569 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe59ac3:::benchmark_data_smithi188_113957_object150603:head# 2024-04-23T16:33:57.569 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.569 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe5a4b3:::benchmark_data_smithi188_113957_object103163:head# 2024-04-23T16:33:57.569 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.569 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe5ddd6:::benchmark_data_smithi188_113957_object56310:head# 2024-04-23T16:33:57.569 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.569 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe68d57:::benchmark_data_smithi188_113957_object39975:head# 2024-04-23T16:33:57.569 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.569 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe6cb06:::benchmark_data_smithi188_113957_object14556:head# 2024-04-23T16:33:57.569 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.569 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe81541:::benchmark_data_smithi188_113957_object133767:head# 2024-04-23T16:33:57.569 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.569 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe88c2a:::benchmark_data_smithi188_113957_object147970:head# 2024-04-23T16:33:57.569 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.570 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe8c0b4:::benchmark_data_smithi188_113957_object12258:head# 2024-04-23T16:33:57.570 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.570 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe91a52:::benchmark_data_smithi188_113957_object46241:head# 2024-04-23T16:33:57.570 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.570 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe92aec:::benchmark_data_smithi188_113957_object80171:head# 2024-04-23T16:33:57.570 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.570 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe92fcf:::benchmark_data_smithi188_113957_object145723:head# 2024-04-23T16:33:57.570 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.570 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe99b42:::benchmark_data_smithi188_113957_object62021:head# 2024-04-23T16:33:57.570 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.570 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fe9a6dd:::benchmark_data_smithi188_113957_object15288:head# 2024-04-23T16:33:57.570 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.570 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5febe5f5:::benchmark_data_smithi188_113957_object9472:head# 2024-04-23T16:33:57.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fec5674:::benchmark_data_smithi188_113957_object37677:head# 2024-04-23T16:33:57.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fec8957:::benchmark_data_smithi188_113957_object80885:head# 2024-04-23T16:33:57.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fed1bb6:::benchmark_data_smithi188_113957_object81615:head# 2024-04-23T16:33:57.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fedb388:::benchmark_data_smithi188_113957_object59573:head# 2024-04-23T16:33:57.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fee0491:::benchmark_data_smithi188_113957_object101584:head# 2024-04-23T16:33:57.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5feece4f:::benchmark_data_smithi188_113957_object88493:head# 2024-04-23T16:33:57.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.571 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fef148e:::benchmark_data_smithi188_113957_object16781:head# 2024-04-23T16:33:57.571 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fef859f:::benchmark_data_smithi188_113957_object116000:head# 2024-04-23T16:33:57.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff035b8:::benchmark_data_smithi188_113957_object66681:head# 2024-04-23T16:33:57.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff058e6:::benchmark_data_smithi188_113957_object152770:head# 2024-04-23T16:33:57.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff15c7f:::benchmark_data_smithi188_113957_object95837:head# 2024-04-23T16:33:57.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff1c554:::benchmark_data_smithi188_113957_object78853:head# 2024-04-23T16:33:57.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff24724:::benchmark_data_smithi188_113957_object119581:head# 2024-04-23T16:33:57.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff296ca:::benchmark_data_smithi188_113957_object90089:head# 2024-04-23T16:33:57.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff30028:::benchmark_data_smithi188_113957_object89782:head# 2024-04-23T16:33:57.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff3446b:::benchmark_data_smithi188_113957_object147266:head# 2024-04-23T16:33:57.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff3d471:::benchmark_data_smithi188_113957_object143918:head# 2024-04-23T16:33:57.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff3f095:::benchmark_data_smithi188_113957_object83460:head# 2024-04-23T16:33:57.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff3f1c6:::benchmark_data_smithi188_113957_object102314:head# 2024-04-23T16:33:57.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff4628c:::benchmark_data_smithi188_113957_object83809:head# 2024-04-23T16:33:57.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff4d3c8:::benchmark_data_smithi188_113957_object147817:head# 2024-04-23T16:33:57.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff54f2d:::benchmark_data_smithi188_113957_object47491:head# 2024-04-23T16:33:57.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff5acc8:::benchmark_data_smithi188_113957_object29095:head# 2024-04-23T16:33:57.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff5dd69:::benchmark_data_smithi188_113957_object58708:head# 2024-04-23T16:33:57.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff5f4a8:::benchmark_data_smithi188_113957_object42256:head# 2024-04-23T16:33:57.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff63f5e:::benchmark_data_smithi188_113957_object130123:head# 2024-04-23T16:33:57.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff64c0a:::benchmark_data_smithi188_113957_object34678:head# 2024-04-23T16:33:57.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff6540a:::benchmark_data_smithi188_113957_object43355:head# 2024-04-23T16:33:57.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff672b1:::benchmark_data_smithi188_113957_object43632:head# 2024-04-23T16:33:57.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff6b57b:::benchmark_data_smithi188_113957_object102533:head# 2024-04-23T16:33:57.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff81250:::benchmark_data_smithi188_113957_object25098:head# 2024-04-23T16:33:57.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff9639d:::benchmark_data_smithi188_113957_object16205:head# 2024-04-23T16:33:57.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff967c3:::benchmark_data_smithi188_113957_object95200:head# 2024-04-23T16:33:57.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff97a64:::benchmark_data_smithi188_113957_object85767:head# 2024-04-23T16:33:57.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff9daa0:::benchmark_data_smithi188_113957_object123954:head# 2024-04-23T16:33:57.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.576 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ff9ebf9:::benchmark_data_smithi188_113957_object52429:head# 2024-04-23T16:33:57.576 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.576 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ffacdff:::benchmark_data_smithi188_113957_object80940:head# 2024-04-23T16:33:57.576 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.576 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ffae8be:::benchmark_data_smithi188_113957_object30621:head# 2024-04-23T16:33:57.576 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.576 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ffb1a0c:::benchmark_data_smithi188_113957_object27180:head# 2024-04-23T16:33:57.576 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.576 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ffb77aa:::benchmark_data_smithi188_113957_object146942:head# 2024-04-23T16:33:57.576 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.576 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ffc4332:::benchmark_data_smithi188_113957_object25828:head# 2024-04-23T16:33:57.576 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.576 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ffcd61a:::benchmark_data_smithi188_113957_object30414:head# 2024-04-23T16:33:57.576 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.576 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ffe1be8:::benchmark_data_smithi188_113957_object37228:head# 2024-04-23T16:33:57.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ffec2a8:::benchmark_data_smithi188_113957_object37519:head# 2024-04-23T16:33:57.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fff680f:::benchmark_data_smithi188_113957_object144398:head# 2024-04-23T16:33:57.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fff8025:::benchmark_data_smithi188_113957_object62292:head# 2024-04-23T16:33:57.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5fffd9c3:::benchmark_data_smithi188_113957_object90862:head# 2024-04-23T16:33:57.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #5:5ffff964:::benchmark_data_smithi188_113957_object30407:head# 2024-04-23T16:33:57.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:33:57.577 INFO:teuthology.orchestra.run.smithi103.stdout:write_pg epoch 386 info 5.a( v 386'9712 (0'0,386'9712] local-lis/les=359/360 n=9712 ec=359/359 lis/c=359/359 les/c/f=360/360/0 sis=359) 2024-04-23T16:33:57.577 INFO:teuthology.orchestra.run.smithi103.stdout:Import successful 2024-04-23T16:33:57.578 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:33:57.606 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:57.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:57 smithi188 ceph-mon[44101]: pgmap v1131: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.4 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 619 op/s; 25846/274264 objects degraded (9.424%); 7.6 MiB/s, 120 objects/s recovering 2024-04-23T16:33:58.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:57 smithi103 ceph-mon[39629]: pgmap v1131: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.4 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 619 op/s; 25846/274264 objects degraded (9.424%); 7.6 MiB/s, 120 objects/s recovering 2024-04-23T16:33:58.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:57 smithi148 ceph-mon[44520]: pgmap v1131: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.4 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 619 op/s; 25846/274264 objects degraded (9.424%); 7.6 MiB/s, 120 objects/s recovering 2024-04-23T16:33:58.755 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:33:58.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:58.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:33:58.860 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:33:58.912 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:58.961 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:33:58.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:58.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:33:59.065 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:33:59.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:58 smithi103 ceph-mon[39629]: 5.2 scrub ok 2024-04-23T16:33:59.113 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:58 smithi148 ceph-mon[44520]: 5.2 scrub ok 2024-04-23T16:33:59.121 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:59.139 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:33:59.139 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-04-23T16:33:59.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-04-23T16:33:59.139 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:33:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:156747 5.a 5:5e838f90:::benchmark_data_smithi188_113957_object3149:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-04-23T16:33:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:32.184941+0000", 2024-04-23T16:33:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 26.950075481999999, 2024-04-23T16:33:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.141821367, 2024-04-23T16:33:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 156747 2024-04-23T16:33:59.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.141 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.184941+0000", 2024-04-23T16:33:59.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.184941+0000", 2024-04-23T16:33:59.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.184942+0000", 2024-04-23T16:33:59.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.271e-06 2024-04-23T16:33:59.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.184947+0000", 2024-04-23T16:33:59.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4880000000000001e-06 2024-04-23T16:33:59.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.184957+0000", 2024-04-23T16:33:59.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7429999999999997e-06 2024-04-23T16:33:59.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.184963+0000", 2024-04-23T16:33:59.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3589999999999996e-06 2024-04-23T16:33:59.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.205666+0000", 2024-04-23T16:33:59.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020702648000000001 2024-04-23T16:33:59.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.206061+0000", 2024-04-23T16:33:59.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00039514299999999998 2024-04-23T16:33:59.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.258827+0000", 2024-04-23T16:33:59.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.052766588000000003 2024-04-23T16:33:59.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.258902+0000", 2024-04-23T16:33:59.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4212e-05 2024-04-23T16:33:59.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.258990+0000", 2024-04-23T16:33:59.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8208999999999995e-05 2024-04-23T16:33:59.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.273139+0000", 2024-04-23T16:33:59.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014148770999999999 2024-04-23T16:33:59.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.323236+0000", 2024-04-23T16:33:59.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.050097326999999997 2024-04-23T16:33:59.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.323274+0000", 2024-04-23T16:33:59.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8163e-05 2024-04-23T16:33:59.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.326763+0000", 2024-04-23T16:33:59.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034884450000000002 2024-04-23T16:33:59.147 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.147 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.147 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.148 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:156749 5.a 5:5b2437c7:::benchmark_data_smithi188_113957_object3151:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-04-23T16:33:59.148 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:32.190391+0000", 2024-04-23T16:33:59.148 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 26.944625458000001, 2024-04-23T16:33:59.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.13927813, 2024-04-23T16:33:59.148 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.148 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.148 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.148 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.148 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.148 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.149 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 156749 2024-04-23T16:33:59.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.149 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.190391+0000", 2024-04-23T16:33:59.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.190391+0000", 2024-04-23T16:33:59.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.190394+0000", 2024-04-23T16:33:59.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3630000000000001e-06 2024-04-23T16:33:59.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.190402+0000", 2024-04-23T16:33:59.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3510000000000002e-06 2024-04-23T16:33:59.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.190414+0000", 2024-04-23T16:33:59.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2069000000000001e-05 2024-04-23T16:33:59.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.190422+0000", 2024-04-23T16:33:59.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7959999999999996e-06 2024-04-23T16:33:59.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.206100+0000", 2024-04-23T16:33:59.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015677857 2024-04-23T16:33:59.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.206331+0000", 2024-04-23T16:33:59.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00023119899999999999 2024-04-23T16:33:59.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.266227+0000", 2024-04-23T16:33:59.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.059895785999999999 2024-04-23T16:33:59.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.266302+0000", 2024-04-23T16:33:59.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5371999999999996e-05 2024-04-23T16:33:59.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.266395+0000", 2024-04-23T16:33:59.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2872000000000001e-05 2024-04-23T16:33:59.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.283106+0000", 2024-04-23T16:33:59.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016711388000000001 2024-04-23T16:33:59.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.326851+0000", 2024-04-23T16:33:59.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.043744693000000001 2024-04-23T16:33:59.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.326905+0000", 2024-04-23T16:33:59.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4230000000000001e-05 2024-04-23T16:33:59.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.329669+0000", 2024-04-23T16:33:59.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002764154 2024-04-23T16:33:59.155 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.155 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.155 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.155 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:156815 5.a 5:5d41ef5e:::benchmark_data_smithi188_113957_object3217:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-04-23T16:33:59.155 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:32.320381+0000", 2024-04-23T16:33:59.155 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 26.814635653, 2024-04-23T16:33:59.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.15085343900000001, 2024-04-23T16:33:59.155 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.156 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.156 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.156 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.156 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.156 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.156 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 156815 2024-04-23T16:33:59.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.156 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.320381+0000", 2024-04-23T16:33:59.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.320381+0000", 2024-04-23T16:33:59.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.320383+0000", 2024-04-23T16:33:59.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9769999999999999e-06 2024-04-23T16:33:59.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.320390+0000", 2024-04-23T16:33:59.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1119999999999998e-06 2024-04-23T16:33:59.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.320401+0000", 2024-04-23T16:33:59.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.11e-05 2024-04-23T16:33:59.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.320410+0000", 2024-04-23T16:33:59.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8890000000000008e-06 2024-04-23T16:33:59.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.341757+0000", 2024-04-23T16:33:59.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021347062999999999 2024-04-23T16:33:59.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.342070+0000", 2024-04-23T16:33:59.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000313195 2024-04-23T16:33:59.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.385596+0000", 2024-04-23T16:33:59.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.043525831000000001 2024-04-23T16:33:59.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.385663+0000", 2024-04-23T16:33:59.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7098e-05 2024-04-23T16:33:59.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.385745+0000", 2024-04-23T16:33:59.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2058999999999995e-05 2024-04-23T16:33:59.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.402422+0000", 2024-04-23T16:33:59.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016676265999999999 2024-04-23T16:33:59.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.468521+0000", 2024-04-23T16:33:59.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.066099028000000004 2024-04-23T16:33:59.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.468561+0000", 2024-04-23T16:33:59.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0627999999999998e-05 2024-04-23T16:33:59.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.471234+0000", 2024-04-23T16:33:59.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026731929999999999 2024-04-23T16:33:59.163 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.163 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.163 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.163 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:156828 5.a 5:5048f8d0:::benchmark_data_smithi188_113957_object3230:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-04-23T16:33:59.163 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:32.334897+0000", 2024-04-23T16:33:59.163 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 26.800119767999998, 2024-04-23T16:33:59.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14794104, 2024-04-23T16:33:59.163 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.164 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.164 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.164 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.164 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.164 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.164 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 156828 2024-04-23T16:33:59.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.164 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.334897+0000", 2024-04-23T16:33:59.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.334897+0000", 2024-04-23T16:33:59.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.334900+0000", 2024-04-23T16:33:59.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8600000000000001e-06 2024-04-23T16:33:59.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.334909+0000", 2024-04-23T16:33:59.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9570000000000008e-06 2024-04-23T16:33:59.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.334922+0000", 2024-04-23T16:33:59.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3064000000000001e-05 2024-04-23T16:33:59.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.334931+0000", 2024-04-23T16:33:59.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9779999999999994e-06 2024-04-23T16:33:59.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.342121+0000", 2024-04-23T16:33:59.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0071905399999999996 2024-04-23T16:33:59.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.342413+0000", 2024-04-23T16:33:59.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029153000000000002 2024-04-23T16:33:59.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.394561+0000", 2024-04-23T16:33:59.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.052148068999999998 2024-04-23T16:33:59.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.394632+0000", 2024-04-23T16:33:59.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1556000000000004e-05 2024-04-23T16:33:59.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.394718+0000", 2024-04-23T16:33:59.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5859999999999994e-05 2024-04-23T16:33:59.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.409643+0000", 2024-04-23T16:33:59.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014924642 2024-04-23T16:33:59.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.480552+0000", 2024-04-23T16:33:59.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.070909468000000003 2024-04-23T16:33:59.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.480585+0000", 2024-04-23T16:33:59.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3006999999999998e-05 2024-04-23T16:33:59.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.482838+0000", 2024-04-23T16:33:59.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022525090000000002 2024-04-23T16:33:59.170 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.170 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.171 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.171 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:156839 5.a 5:50799145:::benchmark_data_smithi188_113957_object3241:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-04-23T16:33:59.171 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:32.351342+0000", 2024-04-23T16:33:59.171 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 26.783674762, 2024-04-23T16:33:59.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.142521335, 2024-04-23T16:33:59.171 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.171 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.171 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.171 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.171 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.172 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.172 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 156839 2024-04-23T16:33:59.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.172 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.351342+0000", 2024-04-23T16:33:59.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.351342+0000", 2024-04-23T16:33:59.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.351345+0000", 2024-04-23T16:33:59.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6740000000000001e-06 2024-04-23T16:33:59.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.351352+0000", 2024-04-23T16:33:59.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5549999999999997e-06 2024-04-23T16:33:59.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.351365+0000", 2024-04-23T16:33:59.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3361e-05 2024-04-23T16:33:59.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.351375+0000", 2024-04-23T16:33:59.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3659999999999998e-06 2024-04-23T16:33:59.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.373287+0000", 2024-04-23T16:33:59.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021912003999999999 2024-04-23T16:33:59.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.373343+0000", 2024-04-23T16:33:59.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5849e-05 2024-04-23T16:33:59.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.409696+0000", 2024-04-23T16:33:59.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.036352954999999999 2024-04-23T16:33:59.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.409812+0000", 2024-04-23T16:33:59.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000116136 2024-04-23T16:33:59.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.409969+0000", 2024-04-23T16:33:59.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000156983 2024-04-23T16:33:59.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.425897+0000", 2024-04-23T16:33:59.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015928492999999998 2024-04-23T16:33:59.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.491320+0000", 2024-04-23T16:33:59.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.065422521999999997 2024-04-23T16:33:59.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.491369+0000", 2024-04-23T16:33:59.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9557e-05 2024-04-23T16:33:59.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:32.493863+0000", 2024-04-23T16:33:59.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00249388 2024-04-23T16:33:59.178 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.178 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.178 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.178 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:157801 5.a 5:5d2e091b:::benchmark_data_smithi188_113957_object4203:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-04-23T16:33:59.178 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:33.883301+0000", 2024-04-23T16:33:59.179 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.251716116000001, 2024-04-23T16:33:59.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14083684399999999, 2024-04-23T16:33:59.179 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.179 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.179 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.179 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.179 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.179 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.179 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 157801 2024-04-23T16:33:59.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.179 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.883301+0000", 2024-04-23T16:33:59.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.883301+0000", 2024-04-23T16:33:59.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.883302+0000", 2024-04-23T16:33:59.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5030000000000001e-06 2024-04-23T16:33:59.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.883308+0000", 2024-04-23T16:33:59.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5999999999999997e-06 2024-04-23T16:33:59.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.883318+0000", 2024-04-23T16:33:59.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0064000000000001e-05 2024-04-23T16:33:59.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.883324+0000", 2024-04-23T16:33:59.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7000000000000002e-06 2024-04-23T16:33:59.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.907233+0000", 2024-04-23T16:33:59.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023908365000000001 2024-04-23T16:33:59.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.907543+0000", 2024-04-23T16:33:59.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031017000000000002 2024-04-23T16:33:59.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.943047+0000", 2024-04-23T16:33:59.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.03550449 2024-04-23T16:33:59.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.943175+0000", 2024-04-23T16:33:59.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012736099999999999 2024-04-23T16:33:59.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.943336+0000", 2024-04-23T16:33:59.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016093800000000001 2024-04-23T16:33:59.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.976845+0000", 2024-04-23T16:33:59.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033508888000000001 2024-04-23T16:33:59.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:34.021080+0000", 2024-04-23T16:33:59.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.044234993 2024-04-23T16:33:59.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:34.021167+0000", 2024-04-23T16:33:59.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7131000000000003e-05 2024-04-23T16:33:59.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:34.024137+0000", 2024-04-23T16:33:59.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002970641 2024-04-23T16:33:59.186 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.186 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.186 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.186 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:157827 5.a 5:5a4b5c04:::benchmark_data_smithi188_113957_object4229:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-04-23T16:33:59.186 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:33.931505+0000", 2024-04-23T16:33:59.187 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.203512034999999, 2024-04-23T16:33:59.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.13924247000000001, 2024-04-23T16:33:59.187 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.187 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.187 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.187 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.187 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.187 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.187 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 157827 2024-04-23T16:33:59.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.187 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.931505+0000", 2024-04-23T16:33:59.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.931505+0000", 2024-04-23T16:33:59.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.931506+0000", 2024-04-23T16:33:59.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3939999999999999e-06 2024-04-23T16:33:59.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.931510+0000", 2024-04-23T16:33:59.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3320000000000002e-06 2024-04-23T16:33:59.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.931518+0000", 2024-04-23T16:33:59.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3749999999999997e-06 2024-04-23T16:33:59.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.931523+0000", 2024-04-23T16:33:59.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0749999999999997e-06 2024-04-23T16:33:59.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.934281+0000", 2024-04-23T16:33:59.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027577890000000001 2024-04-23T16:33:59.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.934727+0000", 2024-04-23T16:33:59.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044616300000000002 2024-04-23T16:33:59.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:34.009635+0000", 2024-04-23T16:33:59.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.074908077000000003 2024-04-23T16:33:59.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:34.009710+0000", 2024-04-23T16:33:59.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5128000000000003e-05 2024-04-23T16:33:59.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:34.009810+0000", 2024-04-23T16:33:59.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9689e-05 2024-04-23T16:33:59.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:34.026543+0000", 2024-04-23T16:33:59.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016733571999999999 2024-04-23T16:33:59.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:34.067531+0000", 2024-04-23T16:33:59.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.040987592000000003 2024-04-23T16:33:59.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:34.067563+0000", 2024-04-23T16:33:59.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2573999999999997e-05 2024-04-23T16:33:59.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:34.070747+0000", 2024-04-23T16:33:59.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031837100000000002 2024-04-23T16:33:59.194 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.194 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.194 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.194 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:157828 5.a 5:517fb4a4:::benchmark_data_smithi188_113957_object4230:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-04-23T16:33:59.194 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:33.932037+0000", 2024-04-23T16:33:59.194 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.202979563, 2024-04-23T16:33:59.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.141117887, 2024-04-23T16:33:59.198 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.198 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.198 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.198 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.198 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.198 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.198 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 157828 2024-04-23T16:33:59.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.198 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.932037+0000", 2024-04-23T16:33:59.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.932037+0000", 2024-04-23T16:33:59.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.932038+0000", 2024-04-23T16:33:59.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0860000000000001e-06 2024-04-23T16:33:59.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.932042+0000", 2024-04-23T16:33:59.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6789999999999998e-06 2024-04-23T16:33:59.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.932051+0000", 2024-04-23T16:33:59.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0100000000000001e-06 2024-04-23T16:33:59.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.932056+0000", 2024-04-23T16:33:59.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2000000000000002e-06 2024-04-23T16:33:59.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.934753+0000", 2024-04-23T16:33:59.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026968299999999999 2024-04-23T16:33:59.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:33.935024+0000", 2024-04-23T16:33:59.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00027064100000000001 2024-04-23T16:33:59.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:34.017568+0000", 2024-04-23T16:33:59.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.082544147999999998 2024-04-23T16:33:59.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:34.017640+0000", 2024-04-23T16:33:59.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2509000000000007e-05 2024-04-23T16:33:59.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:34.017736+0000", 2024-04-23T16:33:59.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6259000000000001e-05 2024-04-23T16:33:59.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:34.032373+0000", 2024-04-23T16:33:59.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014636708 2024-04-23T16:33:59.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:34.070795+0000", 2024-04-23T16:33:59.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.038421645999999997 2024-04-23T16:33:59.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:34.070832+0000", 2024-04-23T16:33:59.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7441000000000003e-05 2024-04-23T16:33:59.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:34.073155+0000", 2024-04-23T16:33:59.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023227299999999998 2024-04-23T16:33:59.206 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.206 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.206 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.206 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:160748 5.a 5:58cd26a4:::benchmark_data_smithi188_113957_object7150:head [delete] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-04-23T16:33:59.206 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:38.721059+0000", 2024-04-23T16:33:59.207 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.413957139000001, 2024-04-23T16:33:59.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.150199846, 2024-04-23T16:33:59.207 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.207 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.207 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.207 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.207 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.207 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.207 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 160748 2024-04-23T16:33:59.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.207 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.721059+0000", 2024-04-23T16:33:59.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.721059+0000", 2024-04-23T16:33:59.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.721062+0000", 2024-04-23T16:33:59.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7389999999999999e-06 2024-04-23T16:33:59.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.721068+0000", 2024-04-23T16:33:59.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.054e-06 2024-04-23T16:33:59.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.721080+0000", 2024-04-23T16:33:59.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1636e-05 2024-04-23T16:33:59.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.721087+0000", 2024-04-23T16:33:59.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1160000000000001e-06 2024-04-23T16:33:59.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.740581+0000", 2024-04-23T16:33:59.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019494431999999999 2024-04-23T16:33:59.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.740620+0000", 2024-04-23T16:33:59.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9038999999999997e-05 2024-04-23T16:33:59.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.798360+0000", 2024-04-23T16:33:59.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.057739288999999999 2024-04-23T16:33:59.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.798443+0000", 2024-04-23T16:33:59.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3112e-05 2024-04-23T16:33:59.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.798538+0000", 2024-04-23T16:33:59.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5532000000000006e-05 2024-04-23T16:33:59.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.813341+0000", 2024-04-23T16:33:59.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014802597000000001 2024-04-23T16:33:59.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.868889+0000", 2024-04-23T16:33:59.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.055548148999999998 2024-04-23T16:33:59.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.868947+0000", 2024-04-23T16:33:59.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8218999999999999e-05 2024-04-23T16:33:59.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.871259+0000", 2024-04-23T16:33:59.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002311932 2024-04-23T16:33:59.213 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.213 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.213 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.213 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:160750 5.a 5:59f77778:::benchmark_data_smithi188_113957_object7152:head [delete] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-04-23T16:33:59.214 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:38.724389+0000", 2024-04-23T16:33:59.214 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.410627272999999, 2024-04-23T16:33:59.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.139927302, 2024-04-23T16:33:59.214 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.214 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.214 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.214 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.214 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.214 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.214 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 160750 2024-04-23T16:33:59.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.214 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.724389+0000", 2024-04-23T16:33:59.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.724389+0000", 2024-04-23T16:33:59.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.724392+0000", 2024-04-23T16:33:59.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9299999999999999e-06 2024-04-23T16:33:59.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.724400+0000", 2024-04-23T16:33:59.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5870000000000004e-06 2024-04-23T16:33:59.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.724413+0000", 2024-04-23T16:33:59.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3305000000000001e-05 2024-04-23T16:33:59.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.724421+0000", 2024-04-23T16:33:59.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2130000000000008e-06 2024-04-23T16:33:59.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.740655+0000", 2024-04-23T16:33:59.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016234056 2024-04-23T16:33:59.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.740900+0000", 2024-04-23T16:33:59.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024452300000000002 2024-04-23T16:33:59.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.762643+0000", 2024-04-23T16:33:59.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021743225000000001 2024-04-23T16:33:59.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.762744+0000", 2024-04-23T16:33:59.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101227 2024-04-23T16:33:59.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.762892+0000", 2024-04-23T16:33:59.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000148018 2024-04-23T16:33:59.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.782625+0000", 2024-04-23T16:33:59.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019732619999999999 2024-04-23T16:33:59.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.219 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.861538+0000", 2024-04-23T16:33:59.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078912668000000005 2024-04-23T16:33:59.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.219 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.861574+0000", 2024-04-23T16:33:59.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5886000000000002e-05 2024-04-23T16:33:59.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.219 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.864317+0000", 2024-04-23T16:33:59.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002743044 2024-04-23T16:33:59.220 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.220 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.220 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.220 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.220 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:160751 5.a 5:57f82ab0:::benchmark_data_smithi188_113957_object7153:head [delete] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-04-23T16:33:59.220 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:38.724521+0000", 2024-04-23T16:33:59.220 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.410495408999999, 2024-04-23T16:33:59.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14184732799999999, 2024-04-23T16:33:59.220 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.220 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.220 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.220 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.220 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.221 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.221 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 160751 2024-04-23T16:33:59.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.221 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.724521+0000", 2024-04-23T16:33:59.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.724521+0000", 2024-04-23T16:33:59.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.724522+0000", 2024-04-23T16:33:59.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.099e-06 2024-04-23T16:33:59.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.724526+0000", 2024-04-23T16:33:59.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9670000000000003e-06 2024-04-23T16:33:59.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.724534+0000", 2024-04-23T16:33:59.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3579999999999997e-06 2024-04-23T16:33:59.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.224 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.224 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.724540+0000", 2024-04-23T16:33:59.224 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7789999999999998e-06 2024-04-23T16:33:59.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.225 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.740934+0000", 2024-04-23T16:33:59.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016393351 2024-04-23T16:33:59.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.741161+0000", 2024-04-23T16:33:59.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00022702700000000001 2024-04-23T16:33:59.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.226 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.770298+0000", 2024-04-23T16:33:59.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029136839000000001 2024-04-23T16:33:59.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.770367+0000", 2024-04-23T16:33:59.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9673999999999997e-05 2024-04-23T16:33:59.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.770457+0000", 2024-04-23T16:33:59.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9485000000000001e-05 2024-04-23T16:33:59.229 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.229 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.793920+0000", 2024-04-23T16:33:59.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023463690999999998 2024-04-23T16:33:59.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.230 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.230 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.864381+0000", 2024-04-23T16:33:59.230 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.070460989000000002 2024-04-23T16:33:59.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.231 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.231 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.231 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.864418+0000", 2024-04-23T16:33:59.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6879999999999999e-05 2024-04-23T16:33:59.231 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.231 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.232 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.232 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.866369+0000", 2024-04-23T16:33:59.232 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001950189 2024-04-23T16:33:59.232 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.232 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.232 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.233 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:160761 5.a 5:50bf586a:::benchmark_data_smithi188_113957_object7163:head [delete] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-04-23T16:33:59.233 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:38.751162+0000", 2024-04-23T16:33:59.233 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.383854643999999, 2024-04-23T16:33:59.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17909995200000001, 2024-04-23T16:33:59.233 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.234 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.234 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.234 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.234 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.234 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.234 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 160761 2024-04-23T16:33:59.235 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.235 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.751162+0000", 2024-04-23T16:33:59.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.236 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.236 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.751162+0000", 2024-04-23T16:33:59.236 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.237 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.237 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.751163+0000", 2024-04-23T16:33:59.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.415e-06 2024-04-23T16:33:59.237 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.237 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.237 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.237 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.751168+0000", 2024-04-23T16:33:59.238 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6550000000000003e-06 2024-04-23T16:33:59.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.238 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.238 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.751177+0000", 2024-04-23T16:33:59.238 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3249999999999997e-06 2024-04-23T16:33:59.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.751185+0000", 2024-04-23T16:33:59.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8180000000000004e-06 2024-04-23T16:33:59.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.753098+0000", 2024-04-23T16:33:59.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00191292 2024-04-23T16:33:59.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.753383+0000", 2024-04-23T16:33:59.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00028443000000000001 2024-04-23T16:33:59.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.241 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.241 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.838181+0000", 2024-04-23T16:33:59.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.084798011000000006 2024-04-23T16:33:59.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.241 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.838272+0000", 2024-04-23T16:33:59.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1055000000000001e-05 2024-04-23T16:33:59.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.838402+0000", 2024-04-23T16:33:59.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013013799999999999 2024-04-23T16:33:59.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.864331+0000", 2024-04-23T16:33:59.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025929526000000001 2024-04-23T16:33:59.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.927147+0000", 2024-04-23T16:33:59.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062815864999999999 2024-04-23T16:33:59.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.927182+0000", 2024-04-23T16:33:59.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5064999999999997e-05 2024-04-23T16:33:59.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.930262+0000", 2024-04-23T16:33:59.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030797289999999998 2024-04-23T16:33:59.246 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.246 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.246 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.247 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:160764 5.a 5:5179712e:::benchmark_data_smithi188_113957_object7166:head [delete] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-04-23T16:33:59.247 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:38.756328+0000", 2024-04-23T16:33:59.247 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.378688516, 2024-04-23T16:33:59.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17681493700000001, 2024-04-23T16:33:59.247 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.247 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.248 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.248 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.248 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.248 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.248 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 160764 2024-04-23T16:33:59.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.249 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756328+0000", 2024-04-23T16:33:59.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756328+0000", 2024-04-23T16:33:59.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756330+0000", 2024-04-23T16:33:59.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.013e-06 2024-04-23T16:33:59.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756335+0000", 2024-04-23T16:33:59.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2024-04-23T16:33:59.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756345+0000", 2024-04-23T16:33:59.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0122000000000001e-05 2024-04-23T16:33:59.252 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.252 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.252 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756351+0000", 2024-04-23T16:33:59.252 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6740000000000002e-06 2024-04-23T16:33:59.252 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.252 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756393+0000", 2024-04-23T16:33:59.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2258000000000001e-05 2024-04-23T16:33:59.253 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.253 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.253 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756643+0000", 2024-04-23T16:33:59.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000249497 2024-04-23T16:33:59.254 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.254 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.254 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.847557+0000", 2024-04-23T16:33:59.254 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.090914377000000005 2024-04-23T16:33:59.254 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.255 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.847648+0000", 2024-04-23T16:33:59.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0999000000000001e-05 2024-04-23T16:33:59.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.847805+0000", 2024-04-23T16:33:59.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015637100000000001 2024-04-23T16:33:59.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.256 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.864339+0000", 2024-04-23T16:33:59.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016534505000000001 2024-04-23T16:33:59.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.257 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.257 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.257 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.930330+0000", 2024-04-23T16:33:59.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.065990368999999993 2024-04-23T16:33:59.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.930385+0000", 2024-04-23T16:33:59.258 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5630000000000001e-05 2024-04-23T16:33:59.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.933143+0000", 2024-04-23T16:33:59.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027578680000000001 2024-04-23T16:33:59.259 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.259 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.259 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.259 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:160766 5.a 5:588abbaa:::benchmark_data_smithi188_113957_object7168:head [delete] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-04-23T16:33:59.259 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:38.757349+0000", 2024-04-23T16:33:59.259 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.377667973000001, 2024-04-23T16:33:59.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17851373300000001, 2024-04-23T16:33:59.259 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.259 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.259 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.259 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.260 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.260 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.260 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 160766 2024-04-23T16:33:59.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.260 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.260 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.260 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757349+0000", 2024-04-23T16:33:59.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.260 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.260 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757349+0000", 2024-04-23T16:33:59.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.261 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.261 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757350+0000", 2024-04-23T16:33:59.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0419999999999999e-06 2024-04-23T16:33:59.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.261 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.261 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757353+0000", 2024-04-23T16:33:59.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7790000000000002e-06 2024-04-23T16:33:59.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.261 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.261 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757361+0000", 2024-04-23T16:33:59.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4769999999999997e-06 2024-04-23T16:33:59.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757366+0000", 2024-04-23T16:33:59.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5550000000000002e-06 2024-04-23T16:33:59.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757405+0000", 2024-04-23T16:33:59.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8531000000000001e-05 2024-04-23T16:33:59.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757654+0000", 2024-04-23T16:33:59.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000249132 2024-04-23T16:33:59.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.263 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.858043+0000", 2024-04-23T16:33:59.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.100388963 2024-04-23T16:33:59.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.263 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.858117+0000", 2024-04-23T16:33:59.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4305000000000001e-05 2024-04-23T16:33:59.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.858202+0000", 2024-04-23T16:33:59.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4151000000000001e-05 2024-04-23T16:33:59.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.868822+0000", 2024-04-23T16:33:59.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010620076000000001 2024-04-23T16:33:59.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.933212+0000", 2024-04-23T16:33:59.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.064390728999999994 2024-04-23T16:33:59.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.933272+0000", 2024-04-23T16:33:59.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9135000000000002e-05 2024-04-23T16:33:59.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.935862+0000", 2024-04-23T16:33:59.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025908580000000001 2024-04-23T16:33:59.265 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.265 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.265 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.265 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:160776 5.a 5:52361127:::benchmark_data_smithi188_113957_object7178:head [delete] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-04-23T16:33:59.266 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:38.777848+0000", 2024-04-23T16:33:59.266 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.357168452, 2024-04-23T16:33:59.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.160716004, 2024-04-23T16:33:59.266 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.266 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.266 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.266 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.266 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.266 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.266 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 160776 2024-04-23T16:33:59.266 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.266 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.266 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.266 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.266 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.777848+0000", 2024-04-23T16:33:59.267 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.267 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.267 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.777848+0000", 2024-04-23T16:33:59.267 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.267 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.267 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.777850+0000", 2024-04-23T16:33:59.267 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.17e-06 2024-04-23T16:33:59.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.267 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.268 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.777857+0000", 2024-04-23T16:33:59.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4189999999999999e-06 2024-04-23T16:33:59.268 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.268 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.268 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.268 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.777867+0000", 2024-04-23T16:33:59.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0319999999999999e-05 2024-04-23T16:33:59.268 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.268 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.268 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.268 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.777874+0000", 2024-04-23T16:33:59.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.323e-06 2024-04-23T16:33:59.268 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.269 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.269 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.269 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.793866+0000", 2024-04-23T16:33:59.269 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015991498 2024-04-23T16:33:59.269 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.269 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.270 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.270 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.793911+0000", 2024-04-23T16:33:59.270 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4854000000000001e-05 2024-04-23T16:33:59.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.270 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.270 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.876843+0000", 2024-04-23T16:33:59.270 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.082932529000000005 2024-04-23T16:33:59.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.270 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.270 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.876951+0000", 2024-04-23T16:33:59.271 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000107656 2024-04-23T16:33:59.271 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.271 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.271 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.271 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.877046+0000", 2024-04-23T16:33:59.271 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4628000000000003e-05 2024-04-23T16:33:59.271 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.271 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.271 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.271 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.904700+0000", 2024-04-23T16:33:59.271 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027654773000000001 2024-04-23T16:33:59.271 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.271 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.271 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.272 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.935919+0000", 2024-04-23T16:33:59.272 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031218475999999998 2024-04-23T16:33:59.272 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.272 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.272 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.272 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.935961+0000", 2024-04-23T16:33:59.272 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1990000000000003e-05 2024-04-23T16:33:59.272 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.272 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.272 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.272 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.938564+0000", 2024-04-23T16:33:59.272 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002603368 2024-04-23T16:33:59.272 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.272 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.272 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.273 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.273 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.273 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:169046 5.a 5:52595403:::benchmark_data_smithi188_113957_object15448:head [delete] snapc 0=[] ondisk+write+known_if_redirected e399)", 2024-04-23T16:33:59.273 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:52.142355+0000", 2024-04-23T16:33:59.273 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.992662105, 2024-04-23T16:33:59.273 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.15251695200000001, 2024-04-23T16:33:59.273 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.273 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.273 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.273 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.273 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.273 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.273 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 169046 2024-04-23T16:33:59.273 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.274 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.274 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.274 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.274 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.142355+0000", 2024-04-23T16:33:59.274 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.274 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.274 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.274 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.142355+0000", 2024-04-23T16:33:59.274 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.274 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.274 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.275 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.142357+0000", 2024-04-23T16:33:59.275 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3989999999999998e-06 2024-04-23T16:33:59.275 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.275 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.275 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.275 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.142364+0000", 2024-04-23T16:33:59.275 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8700000000000003e-06 2024-04-23T16:33:59.275 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.275 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.275 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.275 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.142375+0000", 2024-04-23T16:33:59.275 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1080000000000001e-05 2024-04-23T16:33:59.275 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.275 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.276 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.142383+0000", 2024-04-23T16:33:59.276 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7500000000000003e-06 2024-04-23T16:33:59.276 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.276 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.160771+0000", 2024-04-23T16:33:59.276 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018388548000000001 2024-04-23T16:33:59.276 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.276 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.160845+0000", 2024-04-23T16:33:59.276 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3353000000000001e-05 2024-04-23T16:33:59.276 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.277 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.218859+0000", 2024-04-23T16:33:59.277 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.058014607000000003 2024-04-23T16:33:59.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.277 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.277 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.218934+0000", 2024-04-23T16:33:59.277 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5370000000000005e-05 2024-04-23T16:33:59.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.277 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.277 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.219028+0000", 2024-04-23T16:33:59.277 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3654999999999997e-05 2024-04-23T16:33:59.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.278 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.278 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.243832+0000", 2024-04-23T16:33:59.278 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024803623 2024-04-23T16:33:59.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.278 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.278 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.291335+0000", 2024-04-23T16:33:59.278 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.047503142999999998 2024-04-23T16:33:59.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.278 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.278 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.291398+0000", 2024-04-23T16:33:59.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3348000000000004e-05 2024-04-23T16:33:59.279 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.279 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.279 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.279 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.294871+0000", 2024-04-23T16:33:59.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034732059999999999 2024-04-23T16:33:59.279 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.279 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.279 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.279 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.279 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.279 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:169098 5.a 5:52f35e86:::benchmark_data_smithi188_113957_object15500:head [delete] snapc 0=[] ondisk+write+known_if_redirected e399)", 2024-04-23T16:33:59.279 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:52.226410+0000", 2024-04-23T16:33:59.279 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.9086069600000002, 2024-04-23T16:33:59.280 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.139840519, 2024-04-23T16:33:59.280 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.280 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.280 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.280 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.280 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.280 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.280 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 169098 2024-04-23T16:33:59.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.281 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.281 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.281 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.281 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.226410+0000", 2024-04-23T16:33:59.281 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.281 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.281 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.281 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.226410+0000", 2024-04-23T16:33:59.281 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.281 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.282 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.282 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.226413+0000", 2024-04-23T16:33:59.282 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4759999999999998e-06 2024-04-23T16:33:59.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.282 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.282 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.282 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.226425+0000", 2024-04-23T16:33:59.282 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2344000000000001e-05 2024-04-23T16:33:59.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.282 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.282 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.282 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.226445+0000", 2024-04-23T16:33:59.282 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9502999999999999e-05 2024-04-23T16:33:59.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.283 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.226460+0000", 2024-04-23T16:33:59.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5048999999999999e-05 2024-04-23T16:33:59.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.283 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.243876+0000", 2024-04-23T16:33:59.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017415854000000001 2024-04-23T16:33:59.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.283 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.243918+0000", 2024-04-23T16:33:59.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2514e-05 2024-04-23T16:33:59.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.284 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.284 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.284 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.328175+0000", 2024-04-23T16:33:59.284 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.084256517000000003 2024-04-23T16:33:59.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.284 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.284 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.284 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.328289+0000", 2024-04-23T16:33:59.284 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000114561 2024-04-23T16:33:59.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.284 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.284 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.284 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.328396+0000", 2024-04-23T16:33:59.285 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000106199 2024-04-23T16:33:59.285 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.285 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.285 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.285 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.339908+0000", 2024-04-23T16:33:59.285 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011512682999999999 2024-04-23T16:33:59.285 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.285 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.285 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.285 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.362886+0000", 2024-04-23T16:33:59.285 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022977146 2024-04-23T16:33:59.285 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.285 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.285 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.286 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.362925+0000", 2024-04-23T16:33:59.286 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9113000000000003e-05 2024-04-23T16:33:59.286 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.286 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.286 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.286 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.366250+0000", 2024-04-23T16:33:59.286 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00332556 2024-04-23T16:33:59.286 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.286 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.286 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.286 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.286 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.287 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:169103 5.a 5:5afa6ff5:::benchmark_data_smithi188_113957_object15505:head [delete] snapc 0=[] ondisk+write+known_if_redirected e399)", 2024-04-23T16:33:59.287 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:52.229820+0000", 2024-04-23T16:33:59.287 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.9051970689999997, 2024-04-23T16:33:59.287 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.15499592200000001, 2024-04-23T16:33:59.287 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.287 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.287 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.287 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.287 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.287 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.287 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 169103 2024-04-23T16:33:59.287 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.287 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.287 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.288 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.288 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.229820+0000", 2024-04-23T16:33:59.288 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.288 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.288 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.288 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.288 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.229820+0000", 2024-04-23T16:33:59.288 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.288 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.288 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.288 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.288 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.229822+0000", 2024-04-23T16:33:59.288 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.114e-06 2024-04-23T16:33:59.288 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.289 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.289 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.289 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.229829+0000", 2024-04-23T16:33:59.289 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7260000000000002e-06 2024-04-23T16:33:59.289 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.289 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.289 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.289 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.229839+0000", 2024-04-23T16:33:59.289 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4870000000000008e-06 2024-04-23T16:33:59.289 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.289 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.289 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.289 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.229846+0000", 2024-04-23T16:33:59.290 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5140000000000004e-06 2024-04-23T16:33:59.290 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.290 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.290 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.290 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.305245+0000", 2024-04-23T16:33:59.290 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.075398158000000007 2024-04-23T16:33:59.290 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.290 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.290 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.290 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.305686+0000", 2024-04-23T16:33:59.290 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044144699999999998 2024-04-23T16:33:59.290 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.290 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.290 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.291 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.347447+0000", 2024-04-23T16:33:59.291 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.041760661999999997 2024-04-23T16:33:59.291 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.291 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.291 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.291 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.347555+0000", 2024-04-23T16:33:59.291 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000108337 2024-04-23T16:33:59.291 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.292 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.292 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.292 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.347712+0000", 2024-04-23T16:33:59.292 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015675899999999999 2024-04-23T16:33:59.292 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.292 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.292 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.292 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.370838+0000", 2024-04-23T16:33:59.292 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023126049999999999 2024-04-23T16:33:59.292 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.292 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.293 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.293 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.381785+0000", 2024-04-23T16:33:59.293 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010947167000000001 2024-04-23T16:33:59.293 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.293 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.293 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.293 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.381820+0000", 2024-04-23T16:33:59.293 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4690000000000002e-05 2024-04-23T16:33:59.293 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.293 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.293 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.293 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:52.384815+0000", 2024-04-23T16:33:59.293 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029958110000000001 2024-04-23T16:33:59.293 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.293 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.294 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.294 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.294 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.294 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:172299 5.a 5:5ad4980a:::benchmark_data_smithi188_113957_object18701:head [delete] snapc 0=[] ondisk+write+known_if_redirected e400)", 2024-04-23T16:33:59.294 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:57.317409+0000", 2024-04-23T16:33:59.294 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 1.8176076719999998, 2024-04-23T16:33:59.294 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.15037547800000001, 2024-04-23T16:33:59.294 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.294 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.294 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.294 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.294 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.295 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.295 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 172299 2024-04-23T16:33:59.295 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.295 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.295 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.295 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.295 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:57.317409+0000", 2024-04-23T16:33:59.295 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.295 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.295 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.295 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.295 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:57.317409+0000", 2024-04-23T16:33:59.295 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.295 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.295 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.296 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.296 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:57.317411+0000", 2024-04-23T16:33:59.296 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.007e-06 2024-04-23T16:33:59.296 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.296 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.296 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.296 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:57.317417+0000", 2024-04-23T16:33:59.296 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5039999999999999e-06 2024-04-23T16:33:59.296 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.296 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.296 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.296 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:57.317428+0000", 2024-04-23T16:33:59.296 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0464e-05 2024-04-23T16:33:59.297 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.297 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.297 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.297 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:57.317436+0000", 2024-04-23T16:33:59.297 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3100000000000001e-06 2024-04-23T16:33:59.297 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.297 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.297 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.297 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:57.328265+0000", 2024-04-23T16:33:59.297 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010829248 2024-04-23T16:33:59.297 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.297 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.297 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.297 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:57.328337+0000", 2024-04-23T16:33:59.297 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1150999999999997e-05 2024-04-23T16:33:59.298 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.298 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.298 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.298 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:57.389296+0000", 2024-04-23T16:33:59.298 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.060959623999999997 2024-04-23T16:33:59.298 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.298 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.298 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.298 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:57.389402+0000", 2024-04-23T16:33:59.298 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000106068 2024-04-23T16:33:59.298 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.298 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.298 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.299 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:57.389555+0000", 2024-04-23T16:33:59.299 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015254 2024-04-23T16:33:59.299 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.299 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.299 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.299 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:57.409875+0000", 2024-04-23T16:33:59.299 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020320219 2024-04-23T16:33:59.299 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.299 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.299 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.299 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:57.464829+0000", 2024-04-23T16:33:59.299 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.054954271999999998 2024-04-23T16:33:59.299 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.300 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.300 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.300 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:57.464900+0000", 2024-04-23T16:33:59.300 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0877999999999995e-05 2024-04-23T16:33:59.300 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.300 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.300 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.300 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:57.467784+0000", 2024-04-23T16:33:59.300 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028841930000000002 2024-04-23T16:33:59.300 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.300 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.300 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.300 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.300 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.300 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:172859 5.a 5:5ecf7a36:::benchmark_data_smithi188_113957_object19261:head [delete] snapc 0=[] ondisk+write+known_if_redirected e400)", 2024-04-23T16:33:59.301 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:58.108070+0000", 2024-04-23T16:33:59.301 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 1.0269465870000001, 2024-04-23T16:33:59.301 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14096557400000001, 2024-04-23T16:33:59.301 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:33:59.301 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:33:59.301 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:33:59.301 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:33:59.301 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:33:59.301 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:33:59.301 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 172859 2024-04-23T16:33:59.302 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.302 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:33:59.302 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.302 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:33:59.302 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:58.108070+0000", 2024-04-23T16:33:59.302 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.302 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.302 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.303 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:33:59.303 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:58.108070+0000", 2024-04-23T16:33:59.303 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:33:59.303 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.303 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.303 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:33:59.303 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:58.108071+0000", 2024-04-23T16:33:59.303 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5099999999999998e-07 2024-04-23T16:33:59.303 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.303 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.303 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:33:59.304 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:58.108075+0000", 2024-04-23T16:33:59.304 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6729999999999998e-06 2024-04-23T16:33:59.304 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.304 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.304 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:33:59.304 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:58.108080+0000", 2024-04-23T16:33:59.304 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7679999999999997e-06 2024-04-23T16:33:59.304 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.304 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.304 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:33:59.304 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:58.108087+0000", 2024-04-23T16:33:59.304 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4320000000000004e-06 2024-04-23T16:33:59.304 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.304 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.305 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.305 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:58.123522+0000", 2024-04-23T16:33:59.305 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015435501000000001 2024-04-23T16:33:59.305 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.305 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.305 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:33:59.305 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:58.123570+0000", 2024-04-23T16:33:59.305 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7749e-05 2024-04-23T16:33:59.305 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.305 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.305 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:33:59.306 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:58.210809+0000", 2024-04-23T16:33:59.306 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.087238877000000006 2024-04-23T16:33:59.306 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.306 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.306 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:33:59.306 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:58.210887+0000", 2024-04-23T16:33:59.306 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8331000000000006e-05 2024-04-23T16:33:59.306 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.306 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.306 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:33:59.306 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:58.210983+0000", 2024-04-23T16:33:59.306 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5458000000000007e-05 2024-04-23T16:33:59.306 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.306 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.307 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:33:59.307 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:58.231171+0000", 2024-04-23T16:33:59.307 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020187914000000001 2024-04-23T16:33:59.307 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.307 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.307 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:33:59.307 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:58.246363+0000", 2024-04-23T16:33:59.307 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015192663 2024-04-23T16:33:59.307 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.307 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.307 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:33:59.307 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:58.246435+0000", 2024-04-23T16:33:59.307 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1731999999999997e-05 2024-04-23T16:33:59.307 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:33:59.308 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:33:59.308 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:33:59.308 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:58.249036+0000", 2024-04-23T16:33:59.308 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026006250000000001 2024-04-23T16:33:59.308 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.308 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.308 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.308 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:33:59.308 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:33:59.308 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:33:59.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:59.309 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:58 smithi188 ceph-mon[44101]: 5.2 scrub ok 2024-04-23T16:33:59.310 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:33:59.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:33:59.411 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:33:59.441 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:59.793 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:33:59.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:33:59.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:33:59.811 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/exp.5.a.2 2024-04-23T16:33:59.849 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:59 smithi188 ceph-mon[44101]: pgmap v1132: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.3 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 624 op/s; 25663/273112 objects degraded (9.397%); 7.7 MiB/s, 123 objects/s recovering 2024-04-23T16:33:59.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:33:59 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 25663/273112 objects degraded (9.397%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:33:59.900 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:33:59.959 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:33:59.962 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-23T16:33:59.988 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:00.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:00.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:00.012 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-04-23T16:34:00.012 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-23T16:34:00.012 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:34:00.038 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:59 smithi103 ceph-mon[39629]: pgmap v1132: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.3 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 624 op/s; 25663/273112 objects degraded (9.397%); 7.7 MiB/s, 123 objects/s recovering 2024-04-23T16:34:00.038 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:33:59 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 25663/273112 objects degraded (9.397%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:00.104 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:00.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:59 smithi148 ceph-mon[44520]: pgmap v1132: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.3 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 624 op/s; 25663/273112 objects degraded (9.397%); 7.7 MiB/s, 123 objects/s recovering 2024-04-23T16:34:00.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:33:59 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 25663/273112 objects degraded (9.397%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:00.149 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:00.163 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:34:00.177 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:00.193 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:34:00.206 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:00.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:00.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:00.221 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:34:00.267 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:34:00.286 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:34:00.310 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:34:00.338 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:34:00.354 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:00.375 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:34:00.401 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:34:00.408 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:34:00.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:00.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:34:00.422 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:34:00.452 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:34:00.490 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:34:00.520 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:34:00.521 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:34:00.521 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:34:00.521 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:34:00.521 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:34:00.521 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:34:00.521 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:34:00.521 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:34:00.521 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:34:00.521 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:34:00.521 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:34:00.522 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:34:00.522 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:34:00.522 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:34:00.551 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:00.581 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:34:00.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:00.585 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:34:00.685 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:34:00.718 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:00.817 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:34:00.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:00.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:34:00.921 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:00.995 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:01.022 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:01.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:01.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:00 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1936719244' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:34:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:00 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:34:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:00 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:34:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:00 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1936719244' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:34:01.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:00 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:34:01.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:00 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:34:01.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.137 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:01.178 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:01.209 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:01.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:01.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:00 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1936719244' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:34:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:00 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:34:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:00 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:34:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.318 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:01.362 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:01.395 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:01.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:01.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:01.428 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:34:01.429 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:34:01.505 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:01.584 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:01.629 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:01.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:01.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:01.737 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:34:01.767 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:01.867 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.867 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.868 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.868 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.868 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:01 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.868 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:01 smithi148 ceph-mon[44520]: pgmap v1133: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.2 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 641 op/s; 25263/267750 objects degraded (9.435%); 7.6 MiB/s, 121 objects/s recovering 2024-04-23T16:34:01.868 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:01 smithi148 ceph-mon[44520]: osdmap e401: 12 total, 11 up, 9 in 2024-04-23T16:34:01.869 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:01 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:34:01.869 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:01 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:34:01.870 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:34:01.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:01.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:34:01.885 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:01.972 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:01 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:01 smithi188 ceph-mon[44101]: pgmap v1133: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.2 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 641 op/s; 25263/267750 objects degraded (9.435%); 7.6 MiB/s, 121 objects/s recovering 2024-04-23T16:34:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:01 smithi188 ceph-mon[44101]: osdmap e401: 12 total, 11 up, 9 in 2024-04-23T16:34:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:01 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:34:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:01 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:34:02.016 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:02.066 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:02.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:02.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:01 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:02.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:01 smithi103 ceph-mon[39629]: pgmap v1133: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.2 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 641 op/s; 25263/267750 objects degraded (9.435%); 7.6 MiB/s, 121 objects/s recovering 2024-04-23T16:34:02.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:01 smithi103 ceph-mon[39629]: osdmap e401: 12 total, 11 up, 9 in 2024-04-23T16:34:02.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:01 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:34:02.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:01 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:34:02.173 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:34:02.206 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:02.228 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:34:02.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:02.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:34:02.335 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:02.436 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:02.475 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:02.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:02.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:02.578 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:34:02.614 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:02.662 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:34:02.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:02.667 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:34:02.692 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:34:02.692 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:34:02.768 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:02.869 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:02.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:02.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:03.017 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:03.078 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:03.101 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:03.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:03.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:03.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:02 smithi148 ceph-mon[44520]: 5.b scrub starts 2024-04-23T16:34:03.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:02 smithi148 ceph-mon[44520]: osdmap e402: 12 total, 11 up, 9 in 2024-04-23T16:34:03.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:02 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:34:03.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:02 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:34:03.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:02 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:34:03.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:02 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:34:03.150 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:02 smithi103 ceph-mon[39629]: 5.b scrub starts 2024-04-23T16:34:03.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:02 smithi103 ceph-mon[39629]: osdmap e402: 12 total, 11 up, 9 in 2024-04-23T16:34:03.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:02 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:34:03.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:02 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:34:03.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:02 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:34:03.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:02 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:34:03.208 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:34:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:02 smithi188 ceph-mon[44101]: 5.b scrub starts 2024-04-23T16:34:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:02 smithi188 ceph-mon[44101]: osdmap e402: 12 total, 11 up, 9 in 2024-04-23T16:34:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:02 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:34:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:02 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:34:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:02 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:34:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:02 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:34:03.255 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:03.286 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:34:03.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:03.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:34:03.390 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:34:03.423 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:03.427 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-23T16:34:03.430 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:34:03.446 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:34:03.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:03.449 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:34:03.549 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:34:03.595 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:03.622 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:34:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:03.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:34:03.728 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:03.766 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:03.770 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:34:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:03.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:03.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:03.903 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:34:03.933 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:04.005 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:34:04.008 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:03 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:34:04.008 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:03 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:34:04.009 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:03 smithi103 ceph-mon[39629]: 5.b scrub ok 2024-04-23T16:34:04.009 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:03 smithi103 ceph-mon[39629]: pgmap v1136: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.1 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 477 op/s; 25263/266964 objects degraded (9.463%); 4.6 MiB/s, 72 objects/s recovering 2024-04-23T16:34:04.009 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:03 smithi103 ceph-mon[39629]: osdmap e403: 12 total, 11 up, 9 in 2024-04-23T16:34:04.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:04.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:34:04.111 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:34:04.162 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:03 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:34:04.162 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:03 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:34:04.163 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:03 smithi148 ceph-mon[44520]: 5.b scrub ok 2024-04-23T16:34:04.163 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:03 smithi148 ceph-mon[44520]: pgmap v1136: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.1 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 477 op/s; 25263/266964 objects degraded (9.463%); 4.6 MiB/s, 72 objects/s recovering 2024-04-23T16:34:04.163 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:03 smithi148 ceph-mon[44520]: osdmap e403: 12 total, 11 up, 9 in 2024-04-23T16:34:04.166 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:04.204 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:34:04.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:04.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:34:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:03 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:34:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:03 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:34:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:03 smithi188 ceph-mon[44101]: 5.b scrub ok 2024-04-23T16:34:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:03 smithi188 ceph-mon[44101]: pgmap v1136: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.1 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 477 op/s; 25263/266964 objects degraded (9.463%); 4.6 MiB/s, 72 objects/s recovering 2024-04-23T16:34:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:03 smithi188 ceph-mon[44101]: osdmap e403: 12 total, 11 up, 9 in 2024-04-23T16:34:04.309 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:34:04.341 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:04.695 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:34:04.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:04.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:34:04.800 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:04.847 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:34:04.848 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:34:04.849 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:04.879 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:04.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:04.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:04.902 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:04.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:04 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 25263/266964 objects degraded (9.463%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:04.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:04 smithi188 ceph-mon[44101]: 1.7 scrub starts 2024-04-23T16:34:04.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:04 smithi188 ceph-mon[44101]: 1.7 scrub ok 2024-04-23T16:34:04.989 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:34:05.042 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:05.072 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:34:05.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:05.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:34:05.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:04 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 25263/266964 objects degraded (9.463%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:05.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:04 smithi148 ceph-mon[44520]: 1.7 scrub starts 2024-04-23T16:34:05.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:04 smithi148 ceph-mon[44520]: 1.7 scrub ok 2024-04-23T16:34:05.178 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:34:05.206 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:04 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 25263/266964 objects degraded (9.463%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:05.206 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:04 smithi103 ceph-mon[39629]: 1.7 scrub starts 2024-04-23T16:34:05.206 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:04 smithi103 ceph-mon[39629]: 1.7 scrub ok 2024-04-23T16:34:05.208 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:05.407 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:34:05.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:05.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:34:05.511 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:34:05.552 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:05.628 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:34:05.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:05.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:34:05.733 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:05.777 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:05.777 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:34:05.777 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:34:05.825 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:05.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:05.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:05.921 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:34:05.933 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:34:05.967 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:05.983 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1138,"stamp":"2024-04-23T16:34:04.716437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8578466360,"num_objects":130891,"num_object_clones":0,"num_object_copies":261782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130891,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176303,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4308,"num_bytes_recovered":282464800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162306,"ondisk_log_size":162306,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":18396232,"kb_used_data":15208452,"kb_used_omap":14,"kb_used_meta":3187569,"kb_avail":825310136,"statfs":{"total":863955320832,"available":845117579264,"internally_reserved":0,"allocated":15573454848,"data_stored":15570127391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3264071663},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-371261440,"num_objects":-5665,"num_object_clones":0,"num_object_copies":-11330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5665,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1018,"num_bytes_recovered":66715648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001407"},"pg_stats":[{"pgid":"5.f","version":"403'11174","reported_seq":11375,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:03.988496+0000","last_change":"2024-04-23T16:33:17.526419+0000","last_active":"2024-04-23T16:34:03.988496+0000","last_peered":"2024-04-23T16:34:03.988496+0000","last_clean":"2024-04-23T16:34:03.988496+0000","last_became_active":"2024-04-23T16:31:29.140923+0000","last_became_peered":"2024-04-23T16:31:29.140923+0000","last_unstale":"2024-04-23T16:34:03.988496+0000","last_undegraded":"2024-04-23T16:34:03.988496+0000","last_fullsized":"2024-04-23T16:34:03.988496+0000","mapping_epoch":359,"log_start":"367'1100","ondisk_log_start":"367'1100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9683","last_scrub_stamp":"2024-04-23T16:33:17.526283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.526283+0000","objects_scrubbed":9683,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:33.530461+0000","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536870912,"num_objects":8192,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11174,"num_write_kb":619712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"403'11033","reported_seq":11306,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:03.945825+0000","last_change":"2024-04-23T16:34:02.531358+0000","last_active":"2024-04-23T16:34:03.945825+0000","last_peered":"2024-04-23T16:34:03.945825+0000","last_clean":"2024-04-23T16:34:03.945825+0000","last_became_active":"2024-04-23T16:31:29.137890+0000","last_became_peered":"2024-04-23T16:31:29.137890+0000","last_unstale":"2024-04-23T16:34:03.945825+0000","last_undegraded":"2024-04-23T16:34:03.945825+0000","last_fullsized":"2024-04-23T16:34:03.945825+0000","mapping_epoch":359,"log_start":"367'1000","ondisk_log_start":"367'1000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'10975","last_scrub_stamp":"2024-04-23T16:34:02.531162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:02.531162+0000","objects_scrubbed":8314,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:09.558502+0000","scrub_duration":1066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539426816,"num_objects":8231,"num_object_clones":0,"num_object_copies":16462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11033,"num_write_kb":616448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"402'11077","reported_seq":13987,"reported_epoch":402,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:34:02.838879+0000","last_change":"2024-04-23T16:33:26.432192+0000","last_active":"2024-04-23T16:34:02.838879+0000","last_peered":"2024-04-23T16:34:02.838879+0000","last_clean":"2024-04-23T16:32:59.749530+0000","last_became_active":"2024-04-23T16:33:26.338644+0000","last_became_peered":"2024-04-23T16:33:26.338644+0000","last_unstale":"2024-04-23T16:34:02.838879+0000","last_undegraded":"2024-04-23T16:33:25.874847+0000","last_fullsized":"2024-04-23T16:34:02.838879+0000","mapping_epoch":394,"log_start":"367'1000","ondisk_log_start":"367'1000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8711","last_scrub_stamp":"2024-04-23T16:32:43.928914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.928914+0000","objects_scrubbed":8680,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":548208640,"num_objects":8365,"num_object_clones":0,"num_object_copies":16730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11077,"num_write_kb":622144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1357,"num_bytes_recovered":88932352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":["8"],"object_location_counts":[{"shards":"4,8","objects":10},{"shards":"8","objects":8355}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.9","version":"400'10939","reported_seq":11119,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-04-23T16:34:00.039729+0000","last_change":"2024-04-23T16:32:37.256944+0000","last_active":"2024-04-23T16:34:00.039729+0000","last_peered":"2024-04-23T16:34:00.039729+0000","last_clean":"2024-04-23T16:34:00.039729+0000","last_became_active":"2024-04-23T16:31:29.141551+0000","last_became_peered":"2024-04-23T16:31:29.141551+0000","last_unstale":"2024-04-23T16:34:00.039729+0000","last_undegraded":"2024-04-23T16:34:00.039729+0000","last_fullsized":"2024-04-23T16:34:00.039729+0000","mapping_epoch":359,"log_start":"367'900","ondisk_log_start":"367'900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8156","last_scrub_stamp":"2024-04-23T16:32:37.256787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:37.256787+0000","objects_scrubbed":8082,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":549257216,"num_objects":8381,"num_object_clones":0,"num_object_copies":16762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10939,"num_write_kb":618240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"401'10697","reported_seq":10892,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-23T16:34:01.601161+0000","last_change":"2024-04-23T16:33:25.105693+0000","last_active":"2024-04-23T16:34:01.601161+0000","last_peered":"2024-04-23T16:34:01.601161+0000","last_clean":"2024-04-23T16:34:01.601161+0000","last_became_active":"2024-04-23T16:31:29.139045+0000","last_became_peered":"2024-04-23T16:31:29.139045+0000","last_unstale":"2024-04-23T16:34:01.601161+0000","last_undegraded":"2024-04-23T16:34:01.601161+0000","last_fullsized":"2024-04-23T16:34:01.601161+0000","mapping_epoch":359,"log_start":"366'600","ondisk_log_start":"366'600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9367","last_scrub_stamp":"2024-04-23T16:33:25.105563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:25.105563+0000","objects_scrubbed":9367,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:25.368946+0000","scrub_duration":751,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526712832,"num_objects":8037,"num_object_clones":0,"num_object_copies":16074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10697,"num_write_kb":599488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"403'11001","reported_seq":11202,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:03.922907+0000","last_change":"2024-04-23T16:33:22.103034+0000","last_active":"2024-04-23T16:34:03.922907+0000","last_peered":"2024-04-23T16:34:03.922907+0000","last_clean":"2024-04-23T16:34:03.922907+0000","last_became_active":"2024-04-23T16:31:29.310906+0000","last_became_peered":"2024-04-23T16:31:29.310906+0000","last_unstale":"2024-04-23T16:34:03.922907+0000","last_undegraded":"2024-04-23T16:34:03.922907+0000","last_fullsized":"2024-04-23T16:34:03.922907+0000","mapping_epoch":359,"log_start":"367'1000","ondisk_log_start":"367'1000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9559","last_scrub_stamp":"2024-04-23T16:33:22.102857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:22.102857+0000","objects_scrubbed":9559,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:45.157065+0000","scrub_duration":758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":532086784,"num_objects":8119,"num_object_clones":0,"num_object_copies":16238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11001,"num_write_kb":611840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1059,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:02.892730+0000","last_change":"2024-04-23T16:33:31.424170+0000","last_active":"2024-04-23T16:34:02.892730+0000","last_peered":"2024-04-23T16:34:02.892730+0000","last_clean":"2024-04-23T16:34:02.892730+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:34:02.892730+0000","last_undegraded":"2024-04-23T16:34:02.892730+0000","last_fullsized":"2024-04-23T16:34:02.892730+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:31.424058+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:33:31.424058+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:35.317699+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"403'11222","reported_seq":11409,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:04.283796+0000","last_change":"2024-04-23T16:32:38.103756+0000","last_active":"2024-04-23T16:34:04.283796+0000","last_peered":"2024-04-23T16:34:04.283796+0000","last_clean":"2024-04-23T16:34:04.283796+0000","last_became_active":"2024-04-23T16:31:29.141640+0000","last_became_peered":"2024-04-23T16:31:29.141640+0000","last_unstale":"2024-04-23T16:34:04.283796+0000","last_undegraded":"2024-04-23T16:34:04.283796+0000","last_fullsized":"2024-04-23T16:34:04.283796+0000","mapping_epoch":359,"log_start":"367'1200","ondisk_log_start":"367'1200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8300","last_scrub_stamp":"2024-04-23T16:32:38.103547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:38.103547+0000","objects_scrubbed":8231,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539623424,"num_objects":8234,"num_object_clones":0,"num_object_copies":16468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11222,"num_write_kb":622592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1061,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:02.889736+0000","last_change":"2024-04-23T16:33:25.851379+0000","last_active":"2024-04-23T16:34:02.889736+0000","last_peered":"2024-04-23T16:34:02.889736+0000","last_clean":"2024-04-23T16:34:02.889736+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:34:02.889736+0000","last_undegraded":"2024-04-23T16:34:02.889736+0000","last_fullsized":"2024-04-23T16:34:02.889736+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.498820+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.498820+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"401'10914","reported_seq":11112,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-23T16:34:01.647916+0000","last_change":"2024-04-23T16:33:26.135212+0000","last_active":"2024-04-23T16:34:01.647916+0000","last_peered":"2024-04-23T16:34:01.647916+0000","last_clean":"2024-04-23T16:34:01.647916+0000","last_became_active":"2024-04-23T16:31:29.140251+0000","last_became_peered":"2024-04-23T16:31:29.140251+0000","last_unstale":"2024-04-23T16:34:01.647916+0000","last_undegraded":"2024-04-23T16:34:01.647916+0000","last_fullsized":"2024-04-23T16:34:01.647916+0000","mapping_epoch":359,"log_start":"367'900","ondisk_log_start":"367'900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9565","last_scrub_stamp":"2024-04-23T16:33:26.135088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:26.135088+0000","objects_scrubbed":9565,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:28.491332+0000","scrub_duration":759,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538509336,"num_objects":8218,"num_object_clones":0,"num_object_copies":16436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8218,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10914,"num_write_kb":612161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1046,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-04-23T16:33:52.836191+0000","last_change":"2024-04-23T16:32:29.517947+0000","last_active":"2024-04-23T16:33:52.836191+0000","last_peered":"2024-04-23T16:33:52.836191+0000","last_clean":"2024-04-23T16:33:52.836191+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:33:52.836191+0000","last_undegraded":"2024-04-23T16:33:52.836191+0000","last_fullsized":"2024-04-23T16:33:52.836191+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:32:29.517800+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:32:29.517800+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:58.100582+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"403'11196","reported_seq":14402,"reported_epoch":403,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:34:03.973433+0000","last_change":"2024-04-23T16:33:26.507399+0000","last_active":"2024-04-23T16:34:03.973433+0000","last_peered":"2024-04-23T16:34:03.973433+0000","last_clean":"2024-04-23T16:32:59.715575+0000","last_became_active":"2024-04-23T16:33:26.458611+0000","last_became_peered":"2024-04-23T16:33:26.458611+0000","last_unstale":"2024-04-23T16:34:03.973433+0000","last_undegraded":"2024-04-23T16:33:25.895472+0000","last_fullsized":"2024-04-23T16:34:03.973433+0000","mapping_epoch":394,"log_start":"367'1100","ondisk_log_start":"367'1100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8607","last_scrub_stamp":"2024-04-23T16:32:43.130806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.130806+0000","objects_scrubbed":8577,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536346624,"num_objects":8184,"num_object_clones":0,"num_object_copies":16368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11196,"num_write_kb":620160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1506,"num_bytes_recovered":98697216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":8181},{"shards":"5,7","objects":3}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1058,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:02.892396+0000","last_change":"2024-04-23T16:33:33.275883+0000","last_active":"2024-04-23T16:34:02.892396+0000","last_peered":"2024-04-23T16:34:02.892396+0000","last_clean":"2024-04-23T16:34:02.892396+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:34:02.892396+0000","last_undegraded":"2024-04-23T16:34:02.892396+0000","last_fullsized":"2024-04-23T16:34:02.892396+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:33.275799+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:33:33.275799+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:42.780661+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"403'10941","reported_seq":11139,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:03.979701+0000","last_change":"2024-04-23T16:33:18.576077+0000","last_active":"2024-04-23T16:34:03.979701+0000","last_peered":"2024-04-23T16:34:03.979701+0000","last_clean":"2024-04-23T16:34:03.979701+0000","last_became_active":"2024-04-23T16:31:29.139709+0000","last_became_peered":"2024-04-23T16:31:29.139709+0000","last_unstale":"2024-04-23T16:34:03.979701+0000","last_undegraded":"2024-04-23T16:34:03.979701+0000","last_fullsized":"2024-04-23T16:34:03.979701+0000","mapping_epoch":359,"log_start":"367'900","ondisk_log_start":"367'900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9472","last_scrub_stamp":"2024-04-23T16:33:18.575945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.575945+0000","objects_scrubbed":9472,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:42.303216+0000","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524484608,"num_objects":8003,"num_object_clones":0,"num_object_copies":16006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":606208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1059,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:02.906017+0000","last_change":"2024-04-23T16:34:02.399102+0000","last_active":"2024-04-23T16:34:02.906017+0000","last_peered":"2024-04-23T16:34:02.906017+0000","last_clean":"2024-04-23T16:34:02.906017+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:34:02.906017+0000","last_undegraded":"2024-04-23T16:34:02.906017+0000","last_fullsized":"2024-04-23T16:34:02.906017+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:02.398975+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:02.398975+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:23.945801+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"403'11098","reported_seq":11301,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:04.025672+0000","last_change":"2024-04-23T16:33:19.938285+0000","last_active":"2024-04-23T16:34:04.025672+0000","last_peered":"2024-04-23T16:34:04.025672+0000","last_clean":"2024-04-23T16:34:04.025672+0000","last_became_active":"2024-04-23T16:31:29.139606+0000","last_became_peered":"2024-04-23T16:31:29.139606+0000","last_unstale":"2024-04-23T16:34:04.025672+0000","last_undegraded":"2024-04-23T16:34:04.025672+0000","last_fullsized":"2024-04-23T16:34:04.025672+0000","mapping_epoch":359,"log_start":"367'1000","ondisk_log_start":"367'1000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9720","last_scrub_stamp":"2024-04-23T16:33:19.938119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:19.938119+0000","objects_scrubbed":9720,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:34.186612+0000","scrub_duration":794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":546701312,"num_objects":8342,"num_object_clones":0,"num_object_copies":16684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11098,"num_write_kb":622080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1058,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:02.892691+0000","last_change":"2024-04-23T16:33:28.454288+0000","last_active":"2024-04-23T16:34:02.892691+0000","last_peered":"2024-04-23T16:34:02.892691+0000","last_clean":"2024-04-23T16:34:02.892691+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:34:02.892691+0000","last_undegraded":"2024-04-23T16:34:02.892691+0000","last_fullsized":"2024-04-23T16:34:02.892691+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:28.454200+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:33:28.454200+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:51.316141+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.2","version":"403'10963","reported_seq":11241,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:04.466749+0000","last_change":"2024-04-23T16:33:57.879648+0000","last_active":"2024-04-23T16:34:04.466749+0000","last_peered":"2024-04-23T16:34:04.466749+0000","last_clean":"2024-04-23T16:34:04.466749+0000","last_became_active":"2024-04-23T16:31:29.138430+0000","last_became_peered":"2024-04-23T16:31:29.138430+0000","last_unstale":"2024-04-23T16:34:04.466749+0000","last_undegraded":"2024-04-23T16:34:04.466749+0000","last_fullsized":"2024-04-23T16:34:04.466749+0000","mapping_epoch":359,"log_start":"367'900","ondisk_log_start":"367'900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'10726","last_scrub_stamp":"2024-04-23T16:33:57.879516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:57.879516+0000","objects_scrubbed":8303,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:02.097116+0000","scrub_duration":1621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526450688,"num_objects":8033,"num_object_clones":0,"num_object_copies":16066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10963,"num_write_kb":607872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1059,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:02.892918+0000","last_change":"2024-04-23T16:33:18.422277+0000","last_active":"2024-04-23T16:34:02.892918+0000","last_peered":"2024-04-23T16:34:02.892918+0000","last_clean":"2024-04-23T16:34:02.892918+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:34:02.892918+0000","last_undegraded":"2024-04-23T16:34:02.892918+0000","last_fullsized":"2024-04-23T16:34:02.892918+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:18.422171+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.422171+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:22.642619+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"403'13233","reported_seq":13563,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:04.041791+0000","last_change":"2024-04-23T16:33:17.974701+0000","last_active":"2024-04-23T16:34:04.041791+0000","last_peered":"2024-04-23T16:34:04.041791+0000","last_clean":"2024-04-23T16:34:04.041791+0000","last_became_active":"2024-04-23T16:31:29.139398+0000","last_became_peered":"2024-04-23T16:31:29.139398+0000","last_unstale":"2024-04-23T16:34:04.041791+0000","last_undegraded":"2024-04-23T16:34:04.041791+0000","last_fullsized":"2024-04-23T16:34:04.041791+0000","mapping_epoch":359,"log_start":"370'3200","ondisk_log_start":"370'3200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'11838","last_scrub_stamp":"2024-04-23T16:33:17.974549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.974549+0000","objects_scrubbed":9564,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:28.837234+0000","scrub_duration":848,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535363584,"num_objects":8169,"num_object_clones":0,"num_object_copies":16338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10959,"num_write_kb":612096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1059,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:04.444064+0000","last_change":"2024-04-23T16:34:04.444103+0000","last_active":"2024-04-23T16:34:04.444064+0000","last_peered":"2024-04-23T16:34:04.444064+0000","last_clean":"2024-04-23T16:34:04.444064+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:34:04.444064+0000","last_undegraded":"2024-04-23T16:34:04.444064+0000","last_fullsized":"2024-04-23T16:34:04.444064+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:04.443975+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:04.443975+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:05.109970+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"403'10884","reported_seq":11073,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:04.243121+0000","last_change":"2024-04-23T16:32:43.947374+0000","last_active":"2024-04-23T16:34:04.243121+0000","last_peered":"2024-04-23T16:34:04.243121+0000","last_clean":"2024-04-23T16:34:04.243121+0000","last_became_active":"2024-04-23T16:31:29.138968+0000","last_became_peered":"2024-04-23T16:31:29.138968+0000","last_unstale":"2024-04-23T16:34:04.243121+0000","last_undegraded":"2024-04-23T16:34:04.243121+0000","last_fullsized":"2024-04-23T16:34:04.243121+0000","mapping_epoch":359,"log_start":"367'800","ondisk_log_start":"367'800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8482","last_scrub_stamp":"2024-04-23T16:32:43.947177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.947177+0000","objects_scrubbed":8428,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:13.699951+0000","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524943360,"num_objects":8010,"num_object_clones":0,"num_object_copies":16020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10884,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":946,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-23T16:34:00.873230+0000","last_change":"2024-04-23T16:33:19.443898+0000","last_active":"2024-04-23T16:34:00.873230+0000","last_peered":"2024-04-23T16:34:00.873230+0000","last_clean":"2024-04-23T16:34:00.873230+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:34:00.873230+0000","last_undegraded":"2024-04-23T16:34:00.873230+0000","last_fullsized":"2024-04-23T16:34:00.873230+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:19.443792+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:33:19.443792+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:29.564987+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"403'11204","reported_seq":11392,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:04.292641+0000","last_change":"2024-04-23T16:32:39.103230+0000","last_active":"2024-04-23T16:34:04.292641+0000","last_peered":"2024-04-23T16:34:04.292641+0000","last_clean":"2024-04-23T16:34:04.292641+0000","last_became_active":"2024-04-23T16:31:29.310246+0000","last_became_peered":"2024-04-23T16:31:29.310246+0000","last_unstale":"2024-04-23T16:34:04.292641+0000","last_undegraded":"2024-04-23T16:34:04.292641+0000","last_fullsized":"2024-04-23T16:34:04.292641+0000","mapping_epoch":359,"log_start":"367'1200","ondisk_log_start":"367'1200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8325","last_scrub_stamp":"2024-04-23T16:32:39.103059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:39.103059+0000","objects_scrubbed":8272,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:08.497880+0000","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":541196288,"num_objects":8258,"num_object_clones":0,"num_object_copies":16516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11204,"num_write_kb":622784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":872,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:02.905904+0000","last_change":"2024-04-23T16:33:26.153079+0000","last_active":"2024-04-23T16:34:02.905904+0000","last_peered":"2024-04-23T16:34:02.905904+0000","last_clean":"2024-04-23T16:34:02.905904+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:34:02.905904+0000","last_undegraded":"2024-04-23T16:34:02.905904+0000","last_fullsized":"2024-04-23T16:34:02.905904+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:26.152962+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:33:26.152962+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:53.292007+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"403'11001","reported_seq":14078,"reported_epoch":403,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:34:03.980424+0000","last_change":"2024-04-23T16:33:26.132487+0000","last_active":"2024-04-23T16:34:03.980424+0000","last_peered":"2024-04-23T16:34:03.980424+0000","last_clean":"2024-04-23T16:32:59.695133+0000","last_became_active":"2024-04-23T16:33:26.078062+0000","last_became_peered":"2024-04-23T16:33:26.078062+0000","last_unstale":"2024-04-23T16:34:03.980424+0000","last_undegraded":"2024-04-23T16:33:25.892009+0000","last_fullsized":"2024-04-23T16:34:03.980424+0000","mapping_epoch":394,"log_start":"367'1000","ondisk_log_start":"367'1000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'7968","last_scrub_stamp":"2024-04-23T16:32:36.102122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:36.102122+0000","objects_scrubbed":7926,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":531693568,"num_objects":8113,"num_object_clones":0,"num_object_copies":16226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11001,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1444,"num_bytes_recovered":94633984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":8109},{"shards":"5,8","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1051,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-04-23T16:33:52.830968+0000","last_change":"2024-04-23T16:33:17.354409+0000","last_active":"2024-04-23T16:33:52.830968+0000","last_peered":"2024-04-23T16:33:52.830968+0000","last_clean":"2024-04-23T16:33:52.830968+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:33:52.830968+0000","last_undegraded":"2024-04-23T16:33:52.830968+0000","last_fullsized":"2024-04-23T16:33:52.830968+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:17.354263+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.354263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:39.845454+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":8577875992,"num_objects":130889,"num_object_clones":0,"num_object_copies":261778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130889,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176303,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4307,"num_bytes_recovered":282263552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17414758400,"data_stored":17414750256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160777,"ondisk_log_size":160777,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056865,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1544272,"kb_used_data":1043844,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":92200880,"statfs":{"total":95995035648,"available":94413701120,"internally_reserved":0,"allocated":1068896256,"data_stored":1068530137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":512424388},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351206,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612845,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1891564,"kb_used_data":1563288,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":91853588,"statfs":{"total":95995035648,"available":94058074112,"internally_reserved":0,"allocated":1600806912,"data_stored":1600432907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":336132544},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841776,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1258780,"kb_used_data":1061004,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":92486372,"statfs":{"total":95995035648,"available":94706044928,"internally_reserved":0,"allocated":1086468096,"data_stored":1086100045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168821,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":859708,"kb_used_data":539700,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92885444,"statfs":{"total":95995035648,"available":95114694656,"internally_reserved":0,"allocated":552652800,"data_stored":552296757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":327612874},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463164,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1835560,"kb_used_data":1561432,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":91909592,"statfs":{"total":95995035648,"available":94115422208,"internally_reserved":0,"allocated":1598906368,"data_stored":1598532363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280689094},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137303,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738649,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4024812,"kb_used_data":3664600,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":89720340,"statfs":{"total":95995035648,"available":91873628160,"internally_reserved":0,"allocated":3752550400,"data_stored":3752176395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117257,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":905796,"kb_used_data":541436,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":92839356,"statfs":{"total":95995035648,"available":95067500544,"internally_reserved":0,"allocated":554430464,"data_stored":554070869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921028,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3079912,"kb_used_data":2629464,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":90665240,"statfs":{"total":95995035648,"available":92841205760,"internally_reserved":0,"allocated":2692571136,"data_stored":2692197131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757503,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2995828,"kb_used_data":2603684,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":90749324,"statfs":{"total":95995035648,"available":92927307776,"internally_reserved":0,"allocated":2666172416,"data_stored":2665790787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3735683072,"data_stored":3735683072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1886846976,"data_stored":1886846976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":551419904,"data_stored":551419904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2688483328,"data_stored":2688483328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2661814272,"data_stored":2661810200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1596850176,"data_stored":1596850176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":549519360,"data_stored":549519360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1083244544,"data_stored":1083244544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1595867136,"data_stored":1595867136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1065029632,"data_stored":1065025560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:34:05.985 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:34:06.026 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:34:06.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:06.029 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:34:06.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:05 smithi148 ceph-mon[44520]: pgmap v1138: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 943 op/s; 24645/261782 objects degraded (9.414%); 11 MiB/s, 169 objects/s recovering 2024-04-23T16:34:06.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:05 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:34:06.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:05 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:34:06.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:05 smithi148 ceph-mon[44520]: from='client.15942 v1:172.21.15.103:0/4282610292' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:34:06.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:05 smithi148 ceph-mon[44520]: osdmap e404: 12 total, 11 up, 9 in 2024-04-23T16:34:06.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:05 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3172110559' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:34:06.130 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:34:06.130 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:05 smithi188 ceph-mon[44101]: pgmap v1138: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 943 op/s; 24645/261782 objects degraded (9.414%); 11 MiB/s, 169 objects/s recovering 2024-04-23T16:34:06.130 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:05 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:34:06.130 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:05 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:34:06.130 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:05 smithi188 ceph-mon[44101]: from='client.15942 v1:172.21.15.103:0/4282610292' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:34:06.131 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:05 smithi188 ceph-mon[44101]: osdmap e404: 12 total, 11 up, 9 in 2024-04-23T16:34:06.131 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:05 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3172110559' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:34:06.158 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:05 smithi103 ceph-mon[39629]: pgmap v1138: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 943 op/s; 24645/261782 objects degraded (9.414%); 11 MiB/s, 169 objects/s recovering 2024-04-23T16:34:06.158 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:05 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:34:06.158 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:05 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:34:06.158 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:05 smithi103 ceph-mon[39629]: from='client.15942 v1:172.21.15.103:0/4282610292' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:34:06.158 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:05 smithi103 ceph-mon[39629]: osdmap e404: 12 total, 11 up, 9 in 2024-04-23T16:34:06.158 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:05 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3172110559' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:34:06.159 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:06.488 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-23T16:34:06.810 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:34:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:06 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:34:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:06 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3172110559' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:34:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:06 smithi103 ceph-mon[39629]: osdmap e405: 12 total, 11 up, 9 in 2024-04-23T16:34:07.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:06 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:34:07.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:06 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3172110559' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:34:07.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:06 smithi188 ceph-mon[44101]: osdmap e405: 12 total, 11 up, 9 in 2024-04-23T16:34:07.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:06 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:34:07.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:06 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3172110559' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:34:07.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:06 smithi148 ceph-mon[44520]: osdmap e405: 12 total, 11 up, 9 in 2024-04-23T16:34:07.705 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:07.716 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:34:07.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:07.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:34:07.822 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:34:07.851 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:08.018 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:07 smithi103 ceph-mon[39629]: pgmap v1141: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 577 op/s; 24645/261366 objects degraded (9.429%); 8.0 MiB/s, 127 objects/s recovering 2024-04-23T16:34:08.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:07 smithi188 ceph-mon[44101]: pgmap v1141: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 577 op/s; 24645/261366 objects degraded (9.429%); 8.0 MiB/s, 127 objects/s recovering 2024-04-23T16:34:08.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:07 smithi148 ceph-mon[44520]: pgmap v1141: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 8.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 577 op/s; 24645/261366 objects degraded (9.429%); 8.0 MiB/s, 127 objects/s recovering 2024-04-23T16:34:08.881 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:34:09.704 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:34:09.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:09.707 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:34:09.808 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:09.877 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:09.881 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:34:09.881 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:34:09.881 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:34:09.881 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:34:09.881 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:34:09.881 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:34:09.882 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:34:09.882 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:34:09.882 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:34:09.882 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:34:09.882 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:34:09.882 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:34:09.882 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:34:09.882 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:34:09.882 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:34:09.882 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:34:09.882 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:34:09.882 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:34:09.883 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:34:09.883 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:34:09.883 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:34:09.883 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:34:09.883 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:34:09.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:09 smithi188 ceph-mon[44101]: pgmap v1142: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.9 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 560 op/s; 24458/260234 objects degraded (9.398%); 8.4 MiB/s, 134 objects/s recovering 2024-04-23T16:34:09.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:09 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 24458/260234 objects degraded (9.398%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:09.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:09 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3481047981' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:34:09.943 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:09.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:09.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:10.053 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:09 smithi103 ceph-mon[39629]: pgmap v1142: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.9 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 560 op/s; 24458/260234 objects degraded (9.398%); 8.4 MiB/s, 134 objects/s recovering 2024-04-23T16:34:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:09 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 24458/260234 objects degraded (9.398%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:10.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:09 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3481047981' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:34:10.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:09 smithi148 ceph-mon[44520]: pgmap v1142: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.9 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 560 op/s; 24458/260234 objects degraded (9.398%); 8.4 MiB/s, 134 objects/s recovering 2024-04-23T16:34:10.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:09 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 24458/260234 objects degraded (9.398%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:10.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:09 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3481047981' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:34:10.128 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:10.174 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:10.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:10.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:10.286 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:34:10.320 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:10.398 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:34:10.399 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:34:10.401 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:34:10.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:10.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:34:10.504 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:34:10.535 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:10.886 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:34:10.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:10.892 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:34:10.941 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:34:10.942 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:34:10.992 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:34:11.025 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:11.135 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:34:11.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.140 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:11.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:34:11.195 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:11.243 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:11.304 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.306 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:11.327 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:11.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:11.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:11.430 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:34:11.460 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:34:11.460 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-04-23T16:34:11.460 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-04-23T16:34:11.460 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:34:11.460 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.460 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76742 5.e 5:735ec33a:::benchmark_data_smithi188_113957_object76741:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:34:11.460 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.360842+0000", 2024-04-23T16:34:11.460 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 128.09584199299999, 2024-04-23T16:34:11.460 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8321158799999999, 2024-04-23T16:34:11.460 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.460 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:34:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76742 2024-04-23T16:34:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360842+0000", 2024-04-23T16:34:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360842+0000", 2024-04-23T16:34:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360845+0000", 2024-04-23T16:34:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4940000000000002e-06 2024-04-23T16:34:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360880+0000", 2024-04-23T16:34:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4926999999999999e-05 2024-04-23T16:34:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360895+0000", 2024-04-23T16:34:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5508000000000001e-05 2024-04-23T16:34:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360904+0000", 2024-04-23T16:34:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1430000000000004e-06 2024-04-23T16:34:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.361056+0000", 2024-04-23T16:34:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015130699999999999 2024-04-23T16:34:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:34:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.361166+0000", 2024-04-23T16:34:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110179 2024-04-23T16:34:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:34:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.361311+0000", 2024-04-23T16:34:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014554300000000001 2024-04-23T16:34:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:34:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.368636+0000", 2024-04-23T16:34:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073243010000000001 2024-04-23T16:34:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:34:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.192829+0000", 2024-04-23T16:34:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8241933100000001 2024-04-23T16:34:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:34:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.192891+0000", 2024-04-23T16:34:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1778000000000004e-05 2024-04-23T16:34:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.192958+0000", 2024-04-23T16:34:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7390000000000004e-05 2024-04-23T16:34:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76743 5.e 5:7e57457a:::benchmark_data_smithi188_113957_object76742:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:34:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.360918+0000", 2024-04-23T16:34:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 128.09576589700001, 2024-04-23T16:34:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.832188693, 2024-04-23T16:34:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:34:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76743 2024-04-23T16:34:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360918+0000", 2024-04-23T16:34:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360918+0000", 2024-04-23T16:34:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360919+0000", 2024-04-23T16:34:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5000000000000001e-07 2024-04-23T16:34:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360941+0000", 2024-04-23T16:34:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1345999999999999e-05 2024-04-23T16:34:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360947+0000", 2024-04-23T16:34:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6379999999999998e-06 2024-04-23T16:34:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360953+0000", 2024-04-23T16:34:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9449999999999998e-06 2024-04-23T16:34:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.363820+0000", 2024-04-23T16:34:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028672939999999998 2024-04-23T16:34:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:34:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.363957+0000", 2024-04-23T16:34:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013641200000000001 2024-04-23T16:34:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:34:11.470 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.364114+0000", 2024-04-23T16:34:11.470 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015690300000000001 2024-04-23T16:34:11.470 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.470 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.470 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:34:11.470 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.368640+0000", 2024-04-23T16:34:11.470 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045266120000000002 2024-04-23T16:34:11.470 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.470 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.470 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:34:11.470 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.193003+0000", 2024-04-23T16:34:11.470 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.82436227 2024-04-23T16:34:11.470 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.470 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.470 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:34:11.471 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.193053+0000", 2024-04-23T16:34:11.471 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9811000000000001e-05 2024-04-23T16:34:11.471 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.471 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.471 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.471 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.193107+0000", 2024-04-23T16:34:11.471 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4611999999999998e-05 2024-04-23T16:34:11.471 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.471 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.471 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.471 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.471 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.471 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76746 5.e 5:71b55959:::benchmark_data_smithi188_113957_object76745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:34:11.471 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.364139+0000", 2024-04-23T16:34:11.472 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 128.092545753, 2024-04-23T16:34:11.472 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.83183684, 2024-04-23T16:34:11.472 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.472 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.472 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:34:11.472 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.472 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.472 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.472 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76746 2024-04-23T16:34:11.472 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.472 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.472 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.472 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.472 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.364139+0000", 2024-04-23T16:34:11.472 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.472 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.473 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.473 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.364139+0000", 2024-04-23T16:34:11.473 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.473 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.473 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.473 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.364140+0000", 2024-04-23T16:34:11.473 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7489999999999999e-06 2024-04-23T16:34:11.473 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.473 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.473 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.364164+0000", 2024-04-23T16:34:11.473 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.355e-05 2024-04-23T16:34:11.474 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.474 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.474 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.474 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.364173+0000", 2024-04-23T16:34:11.474 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8170000000000005e-06 2024-04-23T16:34:11.474 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.474 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.474 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.474 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.364180+0000", 2024-04-23T16:34:11.474 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7330000000000003e-06 2024-04-23T16:34:11.474 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.474 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.474 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.474 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.366059+0000", 2024-04-23T16:34:11.475 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018783630000000001 2024-04-23T16:34:11.475 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.475 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.475 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:34:11.475 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.366204+0000", 2024-04-23T16:34:11.475 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014560899999999999 2024-04-23T16:34:11.475 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.475 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.475 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:34:11.475 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.366354+0000", 2024-04-23T16:34:11.475 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000149982 2024-04-23T16:34:11.475 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.475 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.475 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:34:11.476 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.372229+0000", 2024-04-23T16:34:11.476 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005874893 2024-04-23T16:34:11.476 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.476 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.476 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:34:11.476 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.195882+0000", 2024-04-23T16:34:11.476 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.823652869 2024-04-23T16:34:11.476 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.476 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.476 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:34:11.476 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.195918+0000", 2024-04-23T16:34:11.476 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5555000000000001e-05 2024-04-23T16:34:11.476 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.476 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.476 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.477 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.195975+0000", 2024-04-23T16:34:11.477 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7720000000000003e-05 2024-04-23T16:34:11.477 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.477 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.477 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.477 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.477 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.477 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76759 5.e 5:79bad6f2:::benchmark_data_smithi188_113957_object76758:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:34:11.477 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.369738+0000", 2024-04-23T16:34:11.477 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 128.08694616099999, 2024-04-23T16:34:11.477 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.826393253, 2024-04-23T16:34:11.477 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.477 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.477 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:34:11.478 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.478 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.478 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.478 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76759 2024-04-23T16:34:11.478 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.478 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.478 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.478 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.478 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.369738+0000", 2024-04-23T16:34:11.478 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.478 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.478 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.478 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.478 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.369738+0000", 2024-04-23T16:34:11.479 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.479 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.479 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.479 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.479 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.369740+0000", 2024-04-23T16:34:11.479 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.863e-06 2024-04-23T16:34:11.479 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.479 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.479 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.479 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.369761+0000", 2024-04-23T16:34:11.479 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1452999999999999e-05 2024-04-23T16:34:11.479 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.479 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.479 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.479 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.369772+0000", 2024-04-23T16:34:11.480 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0964999999999999e-05 2024-04-23T16:34:11.480 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.480 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.480 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.480 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.369780+0000", 2024-04-23T16:34:11.480 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3429999999999998e-06 2024-04-23T16:34:11.480 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.480 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.480 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.480 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.369834+0000", 2024-04-23T16:34:11.480 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4416999999999998e-05 2024-04-23T16:34:11.480 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.480 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.480 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:34:11.481 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.369968+0000", 2024-04-23T16:34:11.481 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013420700000000001 2024-04-23T16:34:11.481 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.481 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.481 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:34:11.481 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.370114+0000", 2024-04-23T16:34:11.481 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014593599999999999 2024-04-23T16:34:11.481 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.481 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.481 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:34:11.481 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.372383+0000", 2024-04-23T16:34:11.481 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022682470000000001 2024-04-23T16:34:11.481 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.481 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.481 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:34:11.482 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.196026+0000", 2024-04-23T16:34:11.482 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8236435479999999 2024-04-23T16:34:11.482 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.482 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.482 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:34:11.482 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.196076+0000", 2024-04-23T16:34:11.482 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9665e-05 2024-04-23T16:34:11.482 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.482 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.482 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.482 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.196131+0000", 2024-04-23T16:34:11.482 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5609000000000002e-05 2024-04-23T16:34:11.482 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.482 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.482 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.483 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.483 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.483 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76778 5.e 5:7fc0c275:::benchmark_data_smithi188_113957_object76777:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:34:11.483 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.380005+0000", 2024-04-23T16:34:11.483 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 128.07667949200001, 2024-04-23T16:34:11.483 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8170255630000001, 2024-04-23T16:34:11.483 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.483 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.483 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:34:11.483 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.483 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.483 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.483 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76778 2024-04-23T16:34:11.483 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.483 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.484 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.484 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.484 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.380005+0000", 2024-04-23T16:34:11.484 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.484 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.484 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.484 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.484 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.380005+0000", 2024-04-23T16:34:11.484 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.486 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.486 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.486 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.486 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.380007+0000", 2024-04-23T16:34:11.486 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0839999999999999e-06 2024-04-23T16:34:11.486 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.486 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.486 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.486 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.380031+0000", 2024-04-23T16:34:11.487 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.438e-05 2024-04-23T16:34:11.487 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.487 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.487 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.487 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.380043+0000", 2024-04-23T16:34:11.487 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1313e-05 2024-04-23T16:34:11.487 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.487 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.487 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.487 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.380051+0000", 2024-04-23T16:34:11.487 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2190000000000004e-06 2024-04-23T16:34:11.487 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.487 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.487 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.487 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.380115+0000", 2024-04-23T16:34:11.488 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4021000000000002e-05 2024-04-23T16:34:11.488 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.488 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.488 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:34:11.488 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.380230+0000", 2024-04-23T16:34:11.488 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011500700000000001 2024-04-23T16:34:11.488 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.488 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.488 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:34:11.488 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.380351+0000", 2024-04-23T16:34:11.488 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000121514 2024-04-23T16:34:11.488 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.488 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.488 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:34:11.489 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.382807+0000", 2024-04-23T16:34:11.489 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002455942 2024-04-23T16:34:11.489 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.489 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.489 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:34:11.489 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.196947+0000", 2024-04-23T16:34:11.489 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8141396109999999 2024-04-23T16:34:11.489 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.489 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.489 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:34:11.489 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.196979+0000", 2024-04-23T16:34:11.489 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1775e-05 2024-04-23T16:34:11.489 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.489 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.489 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.490 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.197030+0000", 2024-04-23T16:34:11.490 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1697000000000003e-05 2024-04-23T16:34:11.490 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.490 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.490 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.490 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.490 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.490 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76851 5.e 5:7bc22d79:::benchmark_data_smithi188_113957_object76850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:34:11.490 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.419329+0000", 2024-04-23T16:34:11.490 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 128.037355034, 2024-04-23T16:34:11.490 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.777843713, 2024-04-23T16:34:11.490 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.490 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.490 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:34:11.491 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.491 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.491 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.491 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76851 2024-04-23T16:34:11.491 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.491 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.491 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.491 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.491 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.419329+0000", 2024-04-23T16:34:11.491 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.491 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.491 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.491 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.491 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.419329+0000", 2024-04-23T16:34:11.491 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.492 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.492 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.492 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.492 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.419331+0000", 2024-04-23T16:34:11.492 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6649999999999999e-06 2024-04-23T16:34:11.492 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.492 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.492 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.492 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.419352+0000", 2024-04-23T16:34:11.492 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0752999999999999e-05 2024-04-23T16:34:11.492 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.492 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.492 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.492 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.419363+0000", 2024-04-23T16:34:11.492 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0893000000000001e-05 2024-04-23T16:34:11.493 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.493 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.493 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.493 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.419369+0000", 2024-04-23T16:34:11.493 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2879999999999998e-06 2024-04-23T16:34:11.493 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.493 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.493 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.493 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.419416+0000", 2024-04-23T16:34:11.493 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6740999999999999e-05 2024-04-23T16:34:11.493 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.493 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.493 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:34:11.493 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.419522+0000", 2024-04-23T16:34:11.494 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010632399999999999 2024-04-23T16:34:11.494 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.494 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:34:11.494 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.419612+0000", 2024-04-23T16:34:11.494 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0290999999999993e-05 2024-04-23T16:34:11.494 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.494 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:34:11.494 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.421998+0000", 2024-04-23T16:34:11.494 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002385617 2024-04-23T16:34:11.494 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.494 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:34:11.495 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.197076+0000", 2024-04-23T16:34:11.495 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7750786189999999 2024-04-23T16:34:11.495 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.495 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.495 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:34:11.495 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.197123+0000", 2024-04-23T16:34:11.495 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6780999999999998e-05 2024-04-23T16:34:11.495 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.495 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.495 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.495 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.197173+0000", 2024-04-23T16:34:11.495 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9740999999999997e-05 2024-04-23T16:34:11.495 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.495 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.495 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.496 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.496 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.496 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76874 5.e 5:70e21036:::benchmark_data_smithi188_113957_object76873:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:34:11.496 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.430647+0000", 2024-04-23T16:34:11.496 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 128.02603725700001, 2024-04-23T16:34:11.496 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7694722189999998, 2024-04-23T16:34:11.496 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.496 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.496 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:34:11.496 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.496 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.496 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.496 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76874 2024-04-23T16:34:11.496 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.496 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.497 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.497 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.497 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.430647+0000", 2024-04-23T16:34:11.497 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.497 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.497 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.497 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.497 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.430647+0000", 2024-04-23T16:34:11.497 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.497 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.497 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.497 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.497 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.430649+0000", 2024-04-23T16:34:11.497 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5200000000000001e-06 2024-04-23T16:34:11.497 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.498 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.498 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.498 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.430669+0000", 2024-04-23T16:34:11.498 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0089e-05 2024-04-23T16:34:11.498 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.498 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.498 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.498 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.430679+0000", 2024-04-23T16:34:11.498 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0278999999999999e-05 2024-04-23T16:34:11.498 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.498 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.498 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.498 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.430686+0000", 2024-04-23T16:34:11.498 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3189999999999997e-06 2024-04-23T16:34:11.498 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.499 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.499 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.499 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.430745+0000", 2024-04-23T16:34:11.499 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8516e-05 2024-04-23T16:34:11.499 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.499 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.499 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:34:11.499 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.430892+0000", 2024-04-23T16:34:11.499 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014720099999999999 2024-04-23T16:34:11.499 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.499 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.499 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:34:11.499 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.430982+0000", 2024-04-23T16:34:11.499 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9639e-05 2024-04-23T16:34:11.499 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.500 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.500 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:34:11.500 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.432989+0000", 2024-04-23T16:34:11.500 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002007146 2024-04-23T16:34:11.500 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.500 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.500 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:34:11.500 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.200001+0000", 2024-04-23T16:34:11.500 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7670122859999999 2024-04-23T16:34:11.500 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.500 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.500 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:34:11.500 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.200050+0000", 2024-04-23T16:34:11.500 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9403000000000001e-05 2024-04-23T16:34:11.500 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.501 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.501 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.501 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.200119+0000", 2024-04-23T16:34:11.501 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8820999999999997e-05 2024-04-23T16:34:11.501 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.501 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.501 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.501 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.501 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.501 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76885 5.e 5:728c5bad:::benchmark_data_smithi188_113957_object76884:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:34:11.501 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.437272+0000", 2024-04-23T16:34:11.501 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 128.019412359, 2024-04-23T16:34:11.501 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.763021298, 2024-04-23T16:34:11.501 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.502 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.502 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:34:11.502 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.502 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.502 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.502 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76885 2024-04-23T16:34:11.502 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.502 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.502 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.502 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.502 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.437272+0000", 2024-04-23T16:34:11.502 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.502 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.502 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.502 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.503 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.437272+0000", 2024-04-23T16:34:11.503 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.503 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.503 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.503 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.503 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.437273+0000", 2024-04-23T16:34:11.503 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1230000000000001e-06 2024-04-23T16:34:11.503 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.503 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.503 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.503 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.437300+0000", 2024-04-23T16:34:11.503 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6961000000000002e-05 2024-04-23T16:34:11.503 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.503 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.503 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.504 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.437311+0000", 2024-04-23T16:34:11.504 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.115e-05 2024-04-23T16:34:11.504 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.504 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.504 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.504 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.437318+0000", 2024-04-23T16:34:11.504 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1040000000000001e-06 2024-04-23T16:34:11.504 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.504 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.504 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.504 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.437368+0000", 2024-04-23T16:34:11.504 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9421e-05 2024-04-23T16:34:11.504 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.504 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.505 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:34:11.505 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.437492+0000", 2024-04-23T16:34:11.505 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000123786 2024-04-23T16:34:11.505 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.505 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.505 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:34:11.505 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.437602+0000", 2024-04-23T16:34:11.505 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011039099999999999 2024-04-23T16:34:11.505 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.505 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.505 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:34:11.505 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.439385+0000", 2024-04-23T16:34:11.505 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001783527 2024-04-23T16:34:11.505 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.505 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.506 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:34:11.506 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.200172+0000", 2024-04-23T16:34:11.506 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7607869630000001 2024-04-23T16:34:11.506 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.506 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.506 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:34:11.506 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.200225+0000", 2024-04-23T16:34:11.506 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2633999999999998e-05 2024-04-23T16:34:11.506 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.506 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.506 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.506 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.200293+0000", 2024-04-23T16:34:11.506 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8238000000000006e-05 2024-04-23T16:34:11.506 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.507 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.507 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.507 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.507 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.507 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76889 5.e 5:7ef64d28:::benchmark_data_smithi188_113957_object76888:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:34:11.507 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.439666+0000", 2024-04-23T16:34:11.507 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 128.01701789800001, 2024-04-23T16:34:11.507 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7632851199999999, 2024-04-23T16:34:11.507 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.507 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.507 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:34:11.507 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.507 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.507 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.507 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76889 2024-04-23T16:34:11.508 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.508 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.508 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.508 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.508 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.439666+0000", 2024-04-23T16:34:11.508 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.508 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.508 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.508 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.508 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.439666+0000", 2024-04-23T16:34:11.508 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.508 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.508 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.508 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.508 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.439668+0000", 2024-04-23T16:34:11.509 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5319999999999999e-06 2024-04-23T16:34:11.509 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.509 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.509 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.509 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.439687+0000", 2024-04-23T16:34:11.509 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9388000000000001e-05 2024-04-23T16:34:11.509 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.509 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.509 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.509 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.439697+0000", 2024-04-23T16:34:11.509 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9310000000000003e-06 2024-04-23T16:34:11.509 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.509 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.509 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.509 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.439705+0000", 2024-04-23T16:34:11.510 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6690000000000006e-06 2024-04-23T16:34:11.510 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.510 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.510 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.510 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.439753+0000", 2024-04-23T16:34:11.510 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8050000000000002e-05 2024-04-23T16:34:11.510 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.510 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.510 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:34:11.510 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.439880+0000", 2024-04-23T16:34:11.510 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012687700000000001 2024-04-23T16:34:11.510 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.510 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.510 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:34:11.510 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.440021+0000", 2024-04-23T16:34:11.511 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014075200000000001 2024-04-23T16:34:11.511 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.511 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.511 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:34:11.511 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.442122+0000", 2024-04-23T16:34:11.511 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021011659999999998 2024-04-23T16:34:11.511 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.511 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.511 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:34:11.511 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.202835+0000", 2024-04-23T16:34:11.511 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7607130579999999 2024-04-23T16:34:11.511 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.511 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.511 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:34:11.511 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.202889+0000", 2024-04-23T16:34:11.512 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3702000000000002e-05 2024-04-23T16:34:11.512 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.512 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.512 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.512 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.202952+0000", 2024-04-23T16:34:11.512 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2995000000000002e-05 2024-04-23T16:34:11.512 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.512 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.512 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.512 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.512 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.512 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76900 5.e 5:74fb6a1f:::benchmark_data_smithi188_113957_object76899:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:34:11.512 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.444361+0000", 2024-04-23T16:34:11.512 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 128.01232346399999, 2024-04-23T16:34:11.513 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.758767958, 2024-04-23T16:34:11.513 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.513 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.513 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:34:11.513 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.513 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.513 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.513 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76900 2024-04-23T16:34:11.513 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.513 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.513 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.514 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.514 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.444361+0000", 2024-04-23T16:34:11.514 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.514 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.514 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.514 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.514 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.444361+0000", 2024-04-23T16:34:11.514 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.514 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.514 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.514 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.514 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.444362+0000", 2024-04-23T16:34:11.514 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2099999999999995e-07 2024-04-23T16:34:11.514 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.515 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.515 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.515 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.444399+0000", 2024-04-23T16:34:11.515 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7076000000000002e-05 2024-04-23T16:34:11.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.515 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.515 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.515 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.444407+0000", 2024-04-23T16:34:11.515 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8059999999999995e-06 2024-04-23T16:34:11.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.515 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.515 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.515 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.444411+0000", 2024-04-23T16:34:11.515 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6840000000000004e-06 2024-04-23T16:34:11.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.516 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.516 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.444455+0000", 2024-04-23T16:34:11.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3414999999999997e-05 2024-04-23T16:34:11.516 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.516 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:34:11.516 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.444572+0000", 2024-04-23T16:34:11.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011748600000000001 2024-04-23T16:34:11.516 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.516 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:34:11.516 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.444706+0000", 2024-04-23T16:34:11.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013363700000000001 2024-04-23T16:34:11.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.517 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.517 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:34:11.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.446856+0000", 2024-04-23T16:34:11.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021506070000000001 2024-04-23T16:34:11.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.517 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.517 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:34:11.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.203002+0000", 2024-04-23T16:34:11.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7561458669999999 2024-04-23T16:34:11.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.517 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.517 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:34:11.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.203064+0000", 2024-04-23T16:34:11.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2135000000000001e-05 2024-04-23T16:34:11.518 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.518 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.518 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.518 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.203129+0000", 2024-04-23T16:34:11.518 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4424000000000005e-05 2024-04-23T16:34:11.518 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.518 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.518 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.518 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.518 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.518 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76742 5.e 5:735ec33a:::benchmark_data_smithi188_113957_object76741:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:34:11.518 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.463775+0000", 2024-04-23T16:34:11.518 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 127.992909658, 2024-04-23T16:34:11.519 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.729131014, 2024-04-23T16:34:11.519 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.519 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.519 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:34:11.519 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.519 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.519 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.519 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76742 2024-04-23T16:34:11.519 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.519 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.519 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.519 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.519 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463775+0000", 2024-04-23T16:34:11.519 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.519 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.520 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.520 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.520 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463775+0000", 2024-04-23T16:34:11.520 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.520 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.520 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.520 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.520 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463777+0000", 2024-04-23T16:34:11.520 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9750000000000001e-06 2024-04-23T16:34:11.520 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.520 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.520 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.520 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463831+0000", 2024-04-23T16:34:11.520 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4188999999999999e-05 2024-04-23T16:34:11.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.521 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.521 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.521 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463842+0000", 2024-04-23T16:34:11.521 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1535000000000001e-05 2024-04-23T16:34:11.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.521 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.521 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.521 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463854+0000", 2024-04-23T16:34:11.521 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1851000000000001e-05 2024-04-23T16:34:11.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.521 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.521 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.521 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463921+0000", 2024-04-23T16:34:11.522 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6563999999999995e-05 2024-04-23T16:34:11.522 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.522 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:34:11.522 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463991+0000", 2024-04-23T16:34:11.522 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0281e-05 2024-04-23T16:34:11.522 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.522 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.522 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.192906+0000", 2024-04-23T16:34:11.522 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.728914619 2024-04-23T16:34:11.522 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.522 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.522 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.522 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.523 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.523 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76743 5.e 5:7e57457a:::benchmark_data_smithi188_113957_object76742:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:34:11.523 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.463873+0000", 2024-04-23T16:34:11.523 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 127.992811343, 2024-04-23T16:34:11.523 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7291898450000001, 2024-04-23T16:34:11.523 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.523 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.523 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:34:11.523 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.523 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.523 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.523 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76743 2024-04-23T16:34:11.523 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.523 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.524 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.524 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.524 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463873+0000", 2024-04-23T16:34:11.524 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.524 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.524 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.524 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.524 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463873+0000", 2024-04-23T16:34:11.524 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.524 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.524 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.524 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.524 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463875+0000", 2024-04-23T16:34:11.524 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.703e-06 2024-04-23T16:34:11.524 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.525 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.525 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463903+0000", 2024-04-23T16:34:11.525 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8789000000000002e-05 2024-04-23T16:34:11.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.525 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.525 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463915+0000", 2024-04-23T16:34:11.525 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1966e-05 2024-04-23T16:34:11.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.525 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.525 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463923+0000", 2024-04-23T16:34:11.525 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.79e-06 2024-04-23T16:34:11.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464029+0000", 2024-04-23T16:34:11.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105728 2024-04-23T16:34:11.526 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:34:11.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464079+0000", 2024-04-23T16:34:11.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0535999999999999e-05 2024-04-23T16:34:11.526 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.193063+0000", 2024-04-23T16:34:11.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.728983333 2024-04-23T16:34:11.527 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.527 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.527 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.527 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.527 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.527 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76746 5.e 5:71b55959:::benchmark_data_smithi188_113957_object76745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:34:11.527 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.463936+0000", 2024-04-23T16:34:11.527 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 127.992748615, 2024-04-23T16:34:11.527 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7319912259999999, 2024-04-23T16:34:11.527 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.527 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.527 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:34:11.527 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.527 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.528 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.528 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76746 2024-04-23T16:34:11.528 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.528 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.528 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.528 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463936+0000", 2024-04-23T16:34:11.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.528 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.528 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.528 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463936+0000", 2024-04-23T16:34:11.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.528 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.529 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.529 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463936+0000", 2024-04-23T16:34:11.529 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3499999999999995e-07 2024-04-23T16:34:11.529 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.529 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.529 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464009+0000", 2024-04-23T16:34:11.529 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2646999999999997e-05 2024-04-23T16:34:11.529 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.529 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.529 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464017+0000", 2024-04-23T16:34:11.529 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6070000000000002e-06 2024-04-23T16:34:11.529 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.530 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.530 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464032+0000", 2024-04-23T16:34:11.530 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5053e-05 2024-04-23T16:34:11.530 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.530 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.530 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.530 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464125+0000", 2024-04-23T16:34:11.530 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3388999999999996e-05 2024-04-23T16:34:11.530 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.530 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.530 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:34:11.530 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464191+0000", 2024-04-23T16:34:11.530 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6356e-05 2024-04-23T16:34:11.530 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.531 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.531 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.195927+0000", 2024-04-23T16:34:11.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7317354389999999 2024-04-23T16:34:11.531 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.531 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.531 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.531 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76759 5.e 5:79bad6f2:::benchmark_data_smithi188_113957_object76758:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:34:11.531 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.464041+0000", 2024-04-23T16:34:11.531 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 127.99264321699999, 2024-04-23T16:34:11.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.732046242, 2024-04-23T16:34:11.531 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.531 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.532 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:34:11.532 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.532 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.532 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.532 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76759 2024-04-23T16:34:11.532 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.532 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464041+0000", 2024-04-23T16:34:11.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.532 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464041+0000", 2024-04-23T16:34:11.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464042+0000", 2024-04-23T16:34:11.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7100000000000002e-07 2024-04-23T16:34:11.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464074+0000", 2024-04-23T16:34:11.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2184000000000002e-05 2024-04-23T16:34:11.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464080+0000", 2024-04-23T16:34:11.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8000000000000004e-06 2024-04-23T16:34:11.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.534 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464083+0000", 2024-04-23T16:34:11.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7469999999999999e-06 2024-04-23T16:34:11.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.534 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464238+0000", 2024-04-23T16:34:11.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015478099999999999 2024-04-23T16:34:11.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:34:11.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464289+0000", 2024-04-23T16:34:11.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0766999999999997e-05 2024-04-23T16:34:11.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.196087+0000", 2024-04-23T16:34:11.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.731798392 2024-04-23T16:34:11.535 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.535 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.535 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.535 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76778 5.e 5:7fc0c275:::benchmark_data_smithi188_113957_object76777:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:34:11.535 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.464091+0000", 2024-04-23T16:34:11.536 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 127.99259337700001, 2024-04-23T16:34:11.536 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7328962859999999, 2024-04-23T16:34:11.536 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.536 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.536 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:34:11.536 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.536 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.536 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.536 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76778 2024-04-23T16:34:11.536 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.536 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.536 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.536 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.536 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464091+0000", 2024-04-23T16:34:11.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464091+0000", 2024-04-23T16:34:11.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464091+0000", 2024-04-23T16:34:11.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8299999999999998e-07 2024-04-23T16:34:11.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464142+0000", 2024-04-23T16:34:11.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0362000000000003e-05 2024-04-23T16:34:11.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464148+0000", 2024-04-23T16:34:11.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8050000000000001e-06 2024-04-23T16:34:11.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464155+0000", 2024-04-23T16:34:11.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.241e-06 2024-04-23T16:34:11.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464329+0000", 2024-04-23T16:34:11.539 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017406400000000001 2024-04-23T16:34:11.539 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.539 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.539 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:34:11.539 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464391+0000", 2024-04-23T16:34:11.539 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1785999999999994e-05 2024-04-23T16:34:11.539 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.539 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.539 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.539 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.196987+0000", 2024-04-23T16:34:11.539 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7325966450000001 2024-04-23T16:34:11.539 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.539 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.539 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.539 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.540 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76851 5.e 5:7bc22d79:::benchmark_data_smithi188_113957_object76850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:34:11.540 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.464163+0000", 2024-04-23T16:34:11.540 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 127.992521525, 2024-04-23T16:34:11.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.732972556, 2024-04-23T16:34:11.540 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.540 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.540 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:34:11.540 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.540 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.540 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.540 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76851 2024-04-23T16:34:11.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.540 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464163+0000", 2024-04-23T16:34:11.541 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464163+0000", 2024-04-23T16:34:11.541 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464164+0000", 2024-04-23T16:34:11.541 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5799999999999998e-07 2024-04-23T16:34:11.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.542 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.542 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.542 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464203+0000", 2024-04-23T16:34:11.542 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9759999999999999e-05 2024-04-23T16:34:11.542 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.542 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.542 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.542 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464210+0000", 2024-04-23T16:34:11.542 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5610000000000004e-06 2024-04-23T16:34:11.542 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.542 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.542 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.542 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464216+0000", 2024-04-23T16:34:11.542 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9830000000000001e-06 2024-04-23T16:34:11.543 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.543 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.543 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.543 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464427+0000", 2024-04-23T16:34:11.543 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021098199999999999 2024-04-23T16:34:11.543 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.543 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.543 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:34:11.543 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464479+0000", 2024-04-23T16:34:11.543 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2076000000000001e-05 2024-04-23T16:34:11.543 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.543 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.543 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.543 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.197135+0000", 2024-04-23T16:34:11.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7326564360000001 2024-04-23T16:34:11.544 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.544 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.544 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.544 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.544 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76874 5.e 5:70e21036:::benchmark_data_smithi188_113957_object76873:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:34:11.544 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.464233+0000", 2024-04-23T16:34:11.544 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 127.992451415, 2024-04-23T16:34:11.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7358331840000001, 2024-04-23T16:34:11.544 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.544 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.544 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:34:11.544 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.545 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.545 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.545 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76874 2024-04-23T16:34:11.545 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.545 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.545 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.545 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.545 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464233+0000", 2024-04-23T16:34:11.545 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.545 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.545 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.545 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.545 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464233+0000", 2024-04-23T16:34:11.545 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.545 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.546 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.546 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.546 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464234+0000", 2024-04-23T16:34:11.546 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.85e-07 2024-04-23T16:34:11.546 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.546 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.546 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.546 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464272+0000", 2024-04-23T16:34:11.546 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8439999999999998e-05 2024-04-23T16:34:11.546 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.546 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.546 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.546 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464279+0000", 2024-04-23T16:34:11.546 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5599999999999999e-06 2024-04-23T16:34:11.546 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464284+0000", 2024-04-23T16:34:11.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.575e-06 2024-04-23T16:34:11.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464515+0000", 2024-04-23T16:34:11.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000231091 2024-04-23T16:34:11.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:34:11.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464574+0000", 2024-04-23T16:34:11.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9131e-05 2024-04-23T16:34:11.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.200066+0000", 2024-04-23T16:34:11.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.735491602 2024-04-23T16:34:11.548 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.548 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.548 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.548 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76885 5.e 5:728c5bad:::benchmark_data_smithi188_113957_object76884:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:34:11.548 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.464292+0000", 2024-04-23T16:34:11.548 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 127.99239238600001, 2024-04-23T16:34:11.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.735947431, 2024-04-23T16:34:11.548 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.549 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.549 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:34:11.549 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.549 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.549 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.549 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76885 2024-04-23T16:34:11.549 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.549 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.549 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.549 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464292+0000", 2024-04-23T16:34:11.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.549 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.549 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464292+0000", 2024-04-23T16:34:11.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464293+0000", 2024-04-23T16:34:11.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2699999999999999e-07 2024-04-23T16:34:11.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464411+0000", 2024-04-23T16:34:11.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011827 2024-04-23T16:34:11.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.551 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464417+0000", 2024-04-23T16:34:11.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3210000000000001e-06 2024-04-23T16:34:11.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.551 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.551 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.551 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464423+0000", 2024-04-23T16:34:11.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5740000000000003e-06 2024-04-23T16:34:11.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.551 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.551 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.551 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464611+0000", 2024-04-23T16:34:11.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000188553 2024-04-23T16:34:11.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.551 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.551 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:34:11.552 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464664+0000", 2024-04-23T16:34:11.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2893999999999999e-05 2024-04-23T16:34:11.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.552 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.552 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.200239+0000", 2024-04-23T16:34:11.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7355750919999999 2024-04-23T16:34:11.552 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.552 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.552 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.552 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76889 5.e 5:7ef64d28:::benchmark_data_smithi188_113957_object76888:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:34:11.552 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.464430+0000", 2024-04-23T16:34:11.553 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 127.99225397399999, 2024-04-23T16:34:11.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7384693900000001, 2024-04-23T16:34:11.553 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.553 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.553 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:34:11.553 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.553 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.553 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.553 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76889 2024-04-23T16:34:11.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.553 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464430+0000", 2024-04-23T16:34:11.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464430+0000", 2024-04-23T16:34:11.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464431+0000", 2024-04-23T16:34:11.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5199999999999996e-07 2024-04-23T16:34:11.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464470+0000", 2024-04-23T16:34:11.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9023000000000002e-05 2024-04-23T16:34:11.555 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464476+0000", 2024-04-23T16:34:11.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8250000000000001e-06 2024-04-23T16:34:11.555 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464482+0000", 2024-04-23T16:34:11.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7660000000000004e-06 2024-04-23T16:34:11.555 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464701+0000", 2024-04-23T16:34:11.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021927399999999999 2024-04-23T16:34:11.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:34:11.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464761+0000", 2024-04-23T16:34:11.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9636999999999998e-05 2024-04-23T16:34:11.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.202900+0000", 2024-04-23T16:34:11.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.738139213 2024-04-23T16:34:11.556 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.556 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.556 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.557 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76900 5.e 5:74fb6a1f:::benchmark_data_smithi188_113957_object76899:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:34:11.557 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.464509+0000", 2024-04-23T16:34:11.557 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 127.992175162, 2024-04-23T16:34:11.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7385653969999999, 2024-04-23T16:34:11.557 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:11.557 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:11.557 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:34:11.557 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:11.557 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:34:11.557 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:34:11.557 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76900 2024-04-23T16:34:11.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.557 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:11.558 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:11.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464509+0000", 2024-04-23T16:34:11.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.558 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:11.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464509+0000", 2024-04-23T16:34:11.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:11.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.558 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:11.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464510+0000", 2024-04-23T16:34:11.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7999999999999995e-07 2024-04-23T16:34:11.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:11.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464548+0000", 2024-04-23T16:34:11.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8022999999999998e-05 2024-04-23T16:34:11.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:11.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464554+0000", 2024-04-23T16:34:11.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9270000000000002e-06 2024-04-23T16:34:11.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:11.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464559+0000", 2024-04-23T16:34:11.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0819999999999998e-06 2024-04-23T16:34:11.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.560 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:11.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464786+0000", 2024-04-23T16:34:11.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00022706500000000001 2024-04-23T16:34:11.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.560 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:34:11.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464839+0000", 2024-04-23T16:34:11.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3056999999999997e-05 2024-04-23T16:34:11.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:11.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:11.560 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:11.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.203075+0000", 2024-04-23T16:34:11.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7382355629999999 2024-04-23T16:34:11.560 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.561 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.561 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.561 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:11.561 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:11.561 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:34:11.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:11.562 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:11.562 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:34:11.562 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:34:11.663 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:11.745 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:11.789 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:11.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:11.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:11.893 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:34:11.923 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:11 smithi148 ceph-mon[44520]: from='client.25903 v1:172.21.15.103:0/3521358300' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:34:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:11 smithi148 ceph-mon[44520]: pgmap v1143: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 750 op/s; 24021/254792 objects degraded (9.428%); 8.3 MiB/s, 132 objects/s recovering 2024-04-23T16:34:12.208 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:11 smithi188 ceph-mon[44101]: from='client.25903 v1:172.21.15.103:0/3521358300' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:34:12.208 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:11 smithi188 ceph-mon[44101]: pgmap v1143: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 750 op/s; 24021/254792 objects degraded (9.428%); 8.3 MiB/s, 132 objects/s recovering 2024-04-23T16:34:12.248 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-23T16:34:12.276 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:34:12.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:12.279 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:34:12.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:11 smithi103 ceph-mon[39629]: from='client.25903 v1:172.21.15.103:0/3521358300' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:34:12.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:11 smithi103 ceph-mon[39629]: pgmap v1143: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 750 op/s; 24021/254792 objects degraded (9.428%); 8.3 MiB/s, 132 objects/s recovering 2024-04-23T16:34:12.380 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:34:12.426 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:12.454 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:34:12.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:12.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:34:12.564 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:34:12.599 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:12.622 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:34:12.696 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:34:12.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:12.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:34:12.726 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1142,"stamp":"2024-04-23T16:34:08.717198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8527741496,"num_objects":130117,"num_object_clones":0,"num_object_copies":260234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130117,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177077,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4495,"num_bytes_recovered":294720032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162380,"ondisk_log_size":162380,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":18320104,"kb_used_data":15132384,"kb_used_omap":14,"kb_used_meta":3187569,"kb_avail":825386264,"statfs":{"total":863955320832,"available":845195534336,"internally_reserved":0,"allocated":15495561216,"data_stored":15492155807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3264071663},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-220528640,"num_objects":-3365,"num_object_clones":0,"num_object_copies":-6730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3365,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":805,"num_bytes_recovered":52756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001294"},"pg_stats":[{"pgid":"5.f","version":"403'11174","reported_seq":11375,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:03.988496+0000","last_change":"2024-04-23T16:33:17.526419+0000","last_active":"2024-04-23T16:34:03.988496+0000","last_peered":"2024-04-23T16:34:03.988496+0000","last_clean":"2024-04-23T16:34:03.988496+0000","last_became_active":"2024-04-23T16:31:29.140923+0000","last_became_peered":"2024-04-23T16:31:29.140923+0000","last_unstale":"2024-04-23T16:34:03.988496+0000","last_undegraded":"2024-04-23T16:34:03.988496+0000","last_fullsized":"2024-04-23T16:34:03.988496+0000","mapping_epoch":359,"log_start":"367'1100","ondisk_log_start":"367'1100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9683","last_scrub_stamp":"2024-04-23T16:33:17.526283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.526283+0000","objects_scrubbed":9683,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:33.530461+0000","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536870912,"num_objects":8192,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11174,"num_write_kb":619712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"403'11033","reported_seq":11306,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:03.945825+0000","last_change":"2024-04-23T16:34:02.531358+0000","last_active":"2024-04-23T16:34:03.945825+0000","last_peered":"2024-04-23T16:34:03.945825+0000","last_clean":"2024-04-23T16:34:03.945825+0000","last_became_active":"2024-04-23T16:31:29.137890+0000","last_became_peered":"2024-04-23T16:31:29.137890+0000","last_unstale":"2024-04-23T16:34:03.945825+0000","last_undegraded":"2024-04-23T16:34:03.945825+0000","last_fullsized":"2024-04-23T16:34:03.945825+0000","mapping_epoch":359,"log_start":"367'1000","ondisk_log_start":"367'1000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'10975","last_scrub_stamp":"2024-04-23T16:34:02.531162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:02.531162+0000","objects_scrubbed":8314,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:09.558502+0000","scrub_duration":1066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539426816,"num_objects":8231,"num_object_clones":0,"num_object_copies":16462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11033,"num_write_kb":616448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"405'11265","reported_seq":14555,"reported_epoch":405,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:34:07.863390+0000","last_change":"2024-04-23T16:33:26.432192+0000","last_active":"2024-04-23T16:34:07.863390+0000","last_peered":"2024-04-23T16:34:07.863390+0000","last_clean":"2024-04-23T16:32:59.749530+0000","last_became_active":"2024-04-23T16:33:26.338644+0000","last_became_peered":"2024-04-23T16:33:26.338644+0000","last_unstale":"2024-04-23T16:34:07.863390+0000","last_undegraded":"2024-04-23T16:33:25.874847+0000","last_fullsized":"2024-04-23T16:34:07.863390+0000","mapping_epoch":394,"log_start":"367'1200","ondisk_log_start":"367'1200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8711","last_scrub_stamp":"2024-04-23T16:32:43.928914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.928914+0000","objects_scrubbed":8680,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535887872,"num_objects":8177,"num_object_clones":0,"num_object_copies":16354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11265,"num_write_kb":622144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1544,"num_bytes_recovered":101187584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":["8"],"object_location_counts":[{"shards":"4,8","objects":9},{"shards":"8","objects":8168}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.9","version":"404'11147","reported_seq":11335,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-23T16:34:04.912863+0000","last_change":"2024-04-23T16:32:37.256944+0000","last_active":"2024-04-23T16:34:04.912863+0000","last_peered":"2024-04-23T16:34:04.912863+0000","last_clean":"2024-04-23T16:34:04.912863+0000","last_became_active":"2024-04-23T16:31:29.141551+0000","last_became_peered":"2024-04-23T16:31:29.141551+0000","last_unstale":"2024-04-23T16:34:04.912863+0000","last_undegraded":"2024-04-23T16:34:04.912863+0000","last_fullsized":"2024-04-23T16:34:04.912863+0000","mapping_epoch":359,"log_start":"367'1100","ondisk_log_start":"367'1100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8156","last_scrub_stamp":"2024-04-23T16:32:37.256787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:37.256787+0000","objects_scrubbed":8082,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535625728,"num_objects":8173,"num_object_clones":0,"num_object_copies":16346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11147,"num_write_kb":618240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"405'10889","reported_seq":11092,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:06.672969+0000","last_change":"2024-04-23T16:33:25.105693+0000","last_active":"2024-04-23T16:34:06.672969+0000","last_peered":"2024-04-23T16:34:06.672969+0000","last_clean":"2024-04-23T16:34:06.672969+0000","last_became_active":"2024-04-23T16:31:29.139045+0000","last_became_peered":"2024-04-23T16:31:29.139045+0000","last_unstale":"2024-04-23T16:34:06.672969+0000","last_undegraded":"2024-04-23T16:34:06.672969+0000","last_fullsized":"2024-04-23T16:34:06.672969+0000","mapping_epoch":359,"log_start":"367'800","ondisk_log_start":"367'800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9367","last_scrub_stamp":"2024-04-23T16:33:25.105563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:25.105563+0000","objects_scrubbed":9367,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:25.368946+0000","scrub_duration":751,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514129920,"num_objects":7845,"num_object_clones":0,"num_object_copies":15690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10889,"num_write_kb":599488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"403'11001","reported_seq":11202,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:03.922907+0000","last_change":"2024-04-23T16:33:22.103034+0000","last_active":"2024-04-23T16:34:03.922907+0000","last_peered":"2024-04-23T16:34:03.922907+0000","last_clean":"2024-04-23T16:34:03.922907+0000","last_became_active":"2024-04-23T16:31:29.310906+0000","last_became_peered":"2024-04-23T16:31:29.310906+0000","last_unstale":"2024-04-23T16:34:03.922907+0000","last_undegraded":"2024-04-23T16:34:03.922907+0000","last_fullsized":"2024-04-23T16:34:03.922907+0000","mapping_epoch":359,"log_start":"367'1000","ondisk_log_start":"367'1000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9559","last_scrub_stamp":"2024-04-23T16:33:22.102857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:22.102857+0000","objects_scrubbed":9559,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:45.157065+0000","scrub_duration":758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":532086784,"num_objects":8119,"num_object_clones":0,"num_object_copies":16238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11001,"num_write_kb":611840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1059,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:02.892730+0000","last_change":"2024-04-23T16:33:31.424170+0000","last_active":"2024-04-23T16:34:02.892730+0000","last_peered":"2024-04-23T16:34:02.892730+0000","last_clean":"2024-04-23T16:34:02.892730+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:34:02.892730+0000","last_undegraded":"2024-04-23T16:34:02.892730+0000","last_fullsized":"2024-04-23T16:34:02.892730+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:31.424058+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:33:31.424058+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:35.317699+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"403'11222","reported_seq":11409,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:04.283796+0000","last_change":"2024-04-23T16:32:38.103756+0000","last_active":"2024-04-23T16:34:04.283796+0000","last_peered":"2024-04-23T16:34:04.283796+0000","last_clean":"2024-04-23T16:34:04.283796+0000","last_became_active":"2024-04-23T16:31:29.141640+0000","last_became_peered":"2024-04-23T16:31:29.141640+0000","last_unstale":"2024-04-23T16:34:04.283796+0000","last_undegraded":"2024-04-23T16:34:04.283796+0000","last_fullsized":"2024-04-23T16:34:04.283796+0000","mapping_epoch":359,"log_start":"367'1200","ondisk_log_start":"367'1200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8300","last_scrub_stamp":"2024-04-23T16:32:38.103547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:38.103547+0000","objects_scrubbed":8231,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539623424,"num_objects":8234,"num_object_clones":0,"num_object_copies":16468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11222,"num_write_kb":622592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1061,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:02.889736+0000","last_change":"2024-04-23T16:33:25.851379+0000","last_active":"2024-04-23T16:34:02.889736+0000","last_peered":"2024-04-23T16:34:02.889736+0000","last_clean":"2024-04-23T16:34:02.889736+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:34:02.889736+0000","last_undegraded":"2024-04-23T16:34:02.889736+0000","last_fullsized":"2024-04-23T16:34:02.889736+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.498820+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.498820+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"405'11100","reported_seq":11306,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:06.738012+0000","last_change":"2024-04-23T16:33:26.135212+0000","last_active":"2024-04-23T16:34:06.738012+0000","last_peered":"2024-04-23T16:34:06.738012+0000","last_clean":"2024-04-23T16:34:06.738012+0000","last_became_active":"2024-04-23T16:31:29.140251+0000","last_became_peered":"2024-04-23T16:31:29.140251+0000","last_unstale":"2024-04-23T16:34:06.738012+0000","last_undegraded":"2024-04-23T16:34:06.738012+0000","last_fullsized":"2024-04-23T16:34:06.738012+0000","mapping_epoch":359,"log_start":"367'1000","ondisk_log_start":"367'1000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9565","last_scrub_stamp":"2024-04-23T16:33:26.135088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:26.135088+0000","objects_scrubbed":9565,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:28.491332+0000","scrub_duration":759,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526319640,"num_objects":8032,"num_object_clones":0,"num_object_copies":16064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8032,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11100,"num_write_kb":612161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1054,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-23T16:34:04.912916+0000","last_change":"2024-04-23T16:32:29.517947+0000","last_active":"2024-04-23T16:34:04.912916+0000","last_peered":"2024-04-23T16:34:04.912916+0000","last_clean":"2024-04-23T16:34:04.912916+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:34:04.912916+0000","last_undegraded":"2024-04-23T16:34:04.912916+0000","last_fullsized":"2024-04-23T16:34:04.912916+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:32:29.517800+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:32:29.517800+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"403'11196","reported_seq":14402,"reported_epoch":403,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:34:03.973433+0000","last_change":"2024-04-23T16:33:26.507399+0000","last_active":"2024-04-23T16:34:03.973433+0000","last_peered":"2024-04-23T16:34:03.973433+0000","last_clean":"2024-04-23T16:32:59.715575+0000","last_became_active":"2024-04-23T16:33:26.458611+0000","last_became_peered":"2024-04-23T16:33:26.458611+0000","last_unstale":"2024-04-23T16:34:03.973433+0000","last_undegraded":"2024-04-23T16:33:25.895472+0000","last_fullsized":"2024-04-23T16:34:03.973433+0000","mapping_epoch":394,"log_start":"367'1100","ondisk_log_start":"367'1100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8607","last_scrub_stamp":"2024-04-23T16:32:43.130806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.130806+0000","objects_scrubbed":8577,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536346624,"num_objects":8184,"num_object_clones":0,"num_object_copies":16368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11196,"num_write_kb":620160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1506,"num_bytes_recovered":98697216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":8181},{"shards":"5,7","objects":3}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1058,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:02.892396+0000","last_change":"2024-04-23T16:33:33.275883+0000","last_active":"2024-04-23T16:34:02.892396+0000","last_peered":"2024-04-23T16:34:02.892396+0000","last_clean":"2024-04-23T16:34:02.892396+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:34:02.892396+0000","last_undegraded":"2024-04-23T16:34:02.892396+0000","last_fullsized":"2024-04-23T16:34:02.892396+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:33.275799+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:33:33.275799+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:42.780661+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"403'10941","reported_seq":11139,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:03.979701+0000","last_change":"2024-04-23T16:33:18.576077+0000","last_active":"2024-04-23T16:34:03.979701+0000","last_peered":"2024-04-23T16:34:03.979701+0000","last_clean":"2024-04-23T16:34:03.979701+0000","last_became_active":"2024-04-23T16:31:29.139709+0000","last_became_peered":"2024-04-23T16:31:29.139709+0000","last_unstale":"2024-04-23T16:34:03.979701+0000","last_undegraded":"2024-04-23T16:34:03.979701+0000","last_fullsized":"2024-04-23T16:34:03.979701+0000","mapping_epoch":359,"log_start":"367'900","ondisk_log_start":"367'900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9472","last_scrub_stamp":"2024-04-23T16:33:18.575945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.575945+0000","objects_scrubbed":9472,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:42.303216+0000","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524484608,"num_objects":8003,"num_object_clones":0,"num_object_copies":16006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":606208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1059,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:02.906017+0000","last_change":"2024-04-23T16:34:02.399102+0000","last_active":"2024-04-23T16:34:02.906017+0000","last_peered":"2024-04-23T16:34:02.906017+0000","last_clean":"2024-04-23T16:34:02.906017+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:34:02.906017+0000","last_undegraded":"2024-04-23T16:34:02.906017+0000","last_fullsized":"2024-04-23T16:34:02.906017+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:02.398975+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:02.398975+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:23.945801+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"403'11098","reported_seq":11301,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:04.025672+0000","last_change":"2024-04-23T16:33:19.938285+0000","last_active":"2024-04-23T16:34:04.025672+0000","last_peered":"2024-04-23T16:34:04.025672+0000","last_clean":"2024-04-23T16:34:04.025672+0000","last_became_active":"2024-04-23T16:31:29.139606+0000","last_became_peered":"2024-04-23T16:31:29.139606+0000","last_unstale":"2024-04-23T16:34:04.025672+0000","last_undegraded":"2024-04-23T16:34:04.025672+0000","last_fullsized":"2024-04-23T16:34:04.025672+0000","mapping_epoch":359,"log_start":"367'1000","ondisk_log_start":"367'1000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9720","last_scrub_stamp":"2024-04-23T16:33:19.938119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:19.938119+0000","objects_scrubbed":9720,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:34.186612+0000","scrub_duration":794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":546701312,"num_objects":8342,"num_object_clones":0,"num_object_copies":16684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11098,"num_write_kb":622080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1058,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:02.892691+0000","last_change":"2024-04-23T16:33:28.454288+0000","last_active":"2024-04-23T16:34:02.892691+0000","last_peered":"2024-04-23T16:34:02.892691+0000","last_clean":"2024-04-23T16:34:02.892691+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:34:02.892691+0000","last_undegraded":"2024-04-23T16:34:02.892691+0000","last_fullsized":"2024-04-23T16:34:02.892691+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:28.454200+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:33:28.454200+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:51.316141+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.2","version":"403'10963","reported_seq":11241,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:04.466749+0000","last_change":"2024-04-23T16:33:57.879648+0000","last_active":"2024-04-23T16:34:04.466749+0000","last_peered":"2024-04-23T16:34:04.466749+0000","last_clean":"2024-04-23T16:34:04.466749+0000","last_became_active":"2024-04-23T16:31:29.138430+0000","last_became_peered":"2024-04-23T16:31:29.138430+0000","last_unstale":"2024-04-23T16:34:04.466749+0000","last_undegraded":"2024-04-23T16:34:04.466749+0000","last_fullsized":"2024-04-23T16:34:04.466749+0000","mapping_epoch":359,"log_start":"367'900","ondisk_log_start":"367'900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'10726","last_scrub_stamp":"2024-04-23T16:33:57.879516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:57.879516+0000","objects_scrubbed":8303,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:02.097116+0000","scrub_duration":1621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526450688,"num_objects":8033,"num_object_clones":0,"num_object_copies":16066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10963,"num_write_kb":607872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1059,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:02.892918+0000","last_change":"2024-04-23T16:33:18.422277+0000","last_active":"2024-04-23T16:34:02.892918+0000","last_peered":"2024-04-23T16:34:02.892918+0000","last_clean":"2024-04-23T16:34:02.892918+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:34:02.892918+0000","last_undegraded":"2024-04-23T16:34:02.892918+0000","last_fullsized":"2024-04-23T16:34:02.892918+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:18.422171+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.422171+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:22.642619+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"403'13233","reported_seq":13563,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:04.041791+0000","last_change":"2024-04-23T16:33:17.974701+0000","last_active":"2024-04-23T16:34:04.041791+0000","last_peered":"2024-04-23T16:34:04.041791+0000","last_clean":"2024-04-23T16:34:04.041791+0000","last_became_active":"2024-04-23T16:31:29.139398+0000","last_became_peered":"2024-04-23T16:31:29.139398+0000","last_unstale":"2024-04-23T16:34:04.041791+0000","last_undegraded":"2024-04-23T16:34:04.041791+0000","last_fullsized":"2024-04-23T16:34:04.041791+0000","mapping_epoch":359,"log_start":"370'3200","ondisk_log_start":"370'3200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'11838","last_scrub_stamp":"2024-04-23T16:33:17.974549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.974549+0000","objects_scrubbed":9564,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:28.837234+0000","scrub_duration":848,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535363584,"num_objects":8169,"num_object_clones":0,"num_object_copies":16338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10959,"num_write_kb":612096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1059,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:04.444064+0000","last_change":"2024-04-23T16:34:04.444103+0000","last_active":"2024-04-23T16:34:04.444064+0000","last_peered":"2024-04-23T16:34:04.444064+0000","last_clean":"2024-04-23T16:34:04.444064+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:34:04.444064+0000","last_undegraded":"2024-04-23T16:34:04.444064+0000","last_fullsized":"2024-04-23T16:34:04.444064+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:04.443975+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:04.443975+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:05.109970+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"403'10884","reported_seq":11073,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:04.243121+0000","last_change":"2024-04-23T16:32:43.947374+0000","last_active":"2024-04-23T16:34:04.243121+0000","last_peered":"2024-04-23T16:34:04.243121+0000","last_clean":"2024-04-23T16:34:04.243121+0000","last_became_active":"2024-04-23T16:31:29.138968+0000","last_became_peered":"2024-04-23T16:31:29.138968+0000","last_unstale":"2024-04-23T16:34:04.243121+0000","last_undegraded":"2024-04-23T16:34:04.243121+0000","last_fullsized":"2024-04-23T16:34:04.243121+0000","mapping_epoch":359,"log_start":"367'800","ondisk_log_start":"367'800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8482","last_scrub_stamp":"2024-04-23T16:32:43.947177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.947177+0000","objects_scrubbed":8428,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:13.699951+0000","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524943360,"num_objects":8010,"num_object_clones":0,"num_object_copies":16020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10884,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":954,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:05.941208+0000","last_change":"2024-04-23T16:33:19.443898+0000","last_active":"2024-04-23T16:34:05.941208+0000","last_peered":"2024-04-23T16:34:05.941208+0000","last_clean":"2024-04-23T16:34:05.941208+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:34:05.941208+0000","last_undegraded":"2024-04-23T16:34:05.941208+0000","last_fullsized":"2024-04-23T16:34:05.941208+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:19.443792+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:33:19.443792+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:29.564987+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"403'11204","reported_seq":11392,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:04.292641+0000","last_change":"2024-04-23T16:32:39.103230+0000","last_active":"2024-04-23T16:34:04.292641+0000","last_peered":"2024-04-23T16:34:04.292641+0000","last_clean":"2024-04-23T16:34:04.292641+0000","last_became_active":"2024-04-23T16:31:29.310246+0000","last_became_peered":"2024-04-23T16:31:29.310246+0000","last_unstale":"2024-04-23T16:34:04.292641+0000","last_undegraded":"2024-04-23T16:34:04.292641+0000","last_fullsized":"2024-04-23T16:34:04.292641+0000","mapping_epoch":359,"log_start":"367'1200","ondisk_log_start":"367'1200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8325","last_scrub_stamp":"2024-04-23T16:32:39.103059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:39.103059+0000","objects_scrubbed":8272,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:08.497880+0000","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":541196288,"num_objects":8258,"num_object_clones":0,"num_object_copies":16516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11204,"num_write_kb":622784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":872,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-23T16:34:02.905904+0000","last_change":"2024-04-23T16:33:26.153079+0000","last_active":"2024-04-23T16:34:02.905904+0000","last_peered":"2024-04-23T16:34:02.905904+0000","last_clean":"2024-04-23T16:34:02.905904+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:34:02.905904+0000","last_undegraded":"2024-04-23T16:34:02.905904+0000","last_fullsized":"2024-04-23T16:34:02.905904+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:26.152962+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:33:26.152962+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:53.292007+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"403'11001","reported_seq":14078,"reported_epoch":403,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:34:03.980424+0000","last_change":"2024-04-23T16:33:26.132487+0000","last_active":"2024-04-23T16:34:03.980424+0000","last_peered":"2024-04-23T16:34:03.980424+0000","last_clean":"2024-04-23T16:32:59.695133+0000","last_became_active":"2024-04-23T16:33:26.078062+0000","last_became_peered":"2024-04-23T16:33:26.078062+0000","last_unstale":"2024-04-23T16:34:03.980424+0000","last_undegraded":"2024-04-23T16:33:25.892009+0000","last_fullsized":"2024-04-23T16:34:03.980424+0000","mapping_epoch":394,"log_start":"367'1000","ondisk_log_start":"367'1000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'7968","last_scrub_stamp":"2024-04-23T16:32:36.102122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:36.102122+0000","objects_scrubbed":7926,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":531693568,"num_objects":8113,"num_object_clones":0,"num_object_copies":16226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11001,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1444,"num_bytes_recovered":94633984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":8109},{"shards":"5,8","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1059,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-23T16:34:04.912804+0000","last_change":"2024-04-23T16:33:17.354409+0000","last_active":"2024-04-23T16:34:04.912804+0000","last_peered":"2024-04-23T16:34:04.912804+0000","last_clean":"2024-04-23T16:34:04.912804+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:34:04.912804+0000","last_undegraded":"2024-04-23T16:34:04.912804+0000","last_fullsized":"2024-04-23T16:34:04.912804+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:17.354263+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.354263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:39.845454+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":8527151128,"num_objects":130115,"num_object_clones":0,"num_object_copies":260230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130115,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177077,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4494,"num_bytes_recovered":294518784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17338015744,"data_stored":17338007600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160851,"ondisk_log_size":160851,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056866,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1519936,"kb_used_data":1019508,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":92225216,"statfs":{"total":95995035648,"available":94438621184,"internally_reserved":0,"allocated":1043976192,"data_stored":1043586057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":512424388},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351207,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612845,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1891564,"kb_used_data":1563288,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":91853588,"statfs":{"total":95995035648,"available":94058074112,"internally_reserved":0,"allocated":1600806912,"data_stored":1600432907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":336132544},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841778,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1234620,"kb_used_data":1036848,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":92510532,"statfs":{"total":95995035648,"available":94730784768,"internally_reserved":0,"allocated":1061732352,"data_stored":1061346289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168822,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":845288,"kb_used_data":525336,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92899864,"statfs":{"total":95995035648,"available":95129460736,"internally_reserved":0,"allocated":537944064,"data_stored":537570059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":327612874},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463164,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1835560,"kb_used_data":1561432,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":91909592,"statfs":{"total":95995035648,"available":94115422208,"internally_reserved":0,"allocated":1598906368,"data_stored":1598532363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280689094},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137304,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738649,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4024812,"kb_used_data":3664600,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":89720340,"statfs":{"total":95995035648,"available":91873628160,"internally_reserved":0,"allocated":3752550400,"data_stored":3752176395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117258,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":892584,"kb_used_data":528224,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":92852568,"statfs":{"total":95995035648,"available":95081029632,"internally_reserved":0,"allocated":540901376,"data_stored":540523819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921028,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3079912,"kb_used_data":2629464,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":90665240,"statfs":{"total":95995035648,"available":92841205760,"internally_reserved":0,"allocated":2692571136,"data_stored":2692197131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757503,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2995828,"kb_used_data":2603684,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":90749324,"statfs":{"total":95995035648,"available":92927307776,"internally_reserved":0,"allocated":2666172416,"data_stored":2665790787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3735683072,"data_stored":3735683072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1886846976,"data_stored":1886846976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":536936448,"data_stored":536936448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2688483328,"data_stored":2688483328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2661814272,"data_stored":2661810200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1596850176,"data_stored":1596850176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":535953408,"data_stored":535953408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1059127296,"data_stored":1059127296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1595867136,"data_stored":1595867136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1040453632,"data_stored":1040449560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:34:12.728 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:34:12.807 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:34:12.836 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:12.910 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:34:12.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:12.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:34:12.943 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:34:13.020 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:34:13.050 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:13.302 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:34:13.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:13.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:34:13.344 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:13.406 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:13.467 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:13.524 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:13.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:13.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:13.631 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:13.669 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:13.718 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:13.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:13.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:13.828 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:34:13.860 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:13.958 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:34:13.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:13.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:34:14.062 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:14.097 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:13 smithi148 ceph-mon[44520]: pgmap v1144: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.8 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 489 op/s; 24021/253952 objects degraded (9.459%); 4.9 MiB/s, 77 objects/s recovering 2024-04-23T16:34:14.104 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:14.119 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:13 smithi103 ceph-mon[39629]: pgmap v1144: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.8 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 489 op/s; 24021/253952 objects degraded (9.459%); 4.9 MiB/s, 77 objects/s recovering 2024-04-23T16:34:14.148 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:14.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:14.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:14.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:13 smithi188 ceph-mon[44101]: pgmap v1144: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.8 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 489 op/s; 24021/253952 objects degraded (9.459%); 4.9 MiB/s, 77 objects/s recovering 2024-04-23T16:34:14.256 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:34:14.313 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:14.340 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:34:14.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:14.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:34:14.445 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:14.517 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:14.559 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:14.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:14.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:14.595 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:34:14.596 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:34:14.596 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-04-23T16:34:14.596 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:34:14.668 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:14.668 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:34:14.706 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:14.855 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:34:14.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:14.860 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:34:14.955 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:14 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 24021/253952 objects degraded (9.459%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:14.955 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:14 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2715792786' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:34:14.955 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:14 smithi188 ceph-mon[44101]: pgmap v1145: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 677 op/s; 23388/248478 objects degraded (9.413%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:34:14.961 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:34:14.990 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:15.161 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:34:15.270 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:14 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 24021/253952 objects degraded (9.459%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:15.270 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:14 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2715792786' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:34:15.270 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:14 smithi103 ceph-mon[39629]: pgmap v1145: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 677 op/s; 23388/248478 objects degraded (9.413%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:34:15.343 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:34:15.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:15.348 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:34:15.366 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-23T16:34:15.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:14 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 24021/253952 objects degraded (9.459%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:15.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:14 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2715792786' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:34:15.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:14 smithi148 ceph-mon[44520]: pgmap v1145: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 677 op/s; 23388/248478 objects degraded (9.413%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:34:15.449 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:34:15.480 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:15.581 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:34:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:15.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:34:15.685 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:34:15.712 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:15.752 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:34:16.164 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:34:16.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:16.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:34:16.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:15 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:34:16.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:15 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2715792786' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:34:16.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:15 smithi188 ceph-mon[44101]: osdmap e406: 12 total, 11 up, 9 in 2024-04-23T16:34:16.271 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:34:16.272 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:15 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:34:16.272 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:15 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2715792786' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:34:16.272 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:15 smithi103 ceph-mon[39629]: osdmap e406: 12 total, 11 up, 9 in 2024-04-23T16:34:16.302 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:16.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:15 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:34:16.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:15 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2715792786' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:34:16.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:15 smithi148 ceph-mon[44520]: osdmap e406: 12 total, 11 up, 9 in 2024-04-23T16:34:16.652 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:34:16.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:16.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:34:16.756 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:34:16.788 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:17.109 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:16 smithi188 ceph-mon[44101]: pgmap v1147: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 664 op/s; 23388/248080 objects degraded (9.428%); 7.9 MiB/s, 125 objects/s recovering 2024-04-23T16:34:17.140 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:34:17.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:17.144 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:34:17.245 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:34:17.274 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:17.295 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:34:17.318 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:34:17.341 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:34:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:16 smithi103 ceph-mon[39629]: pgmap v1147: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 664 op/s; 23388/248080 objects degraded (9.428%); 7.9 MiB/s, 125 objects/s recovering 2024-04-23T16:34:17.368 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:34:17.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:16 smithi148 ceph-mon[44520]: pgmap v1147: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 664 op/s; 23388/248080 objects degraded (9.428%); 7.9 MiB/s, 125 objects/s recovering 2024-04-23T16:34:17.403 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:34:17.426 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:34:17.427 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:34:17.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:17.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:34:17.453 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:34:17.474 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:34:17.496 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:34:17.517 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:34:17.530 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:34:17.542 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:34:17.542 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:34:17.542 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:34:17.542 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:34:17.543 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:34:17.543 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:34:17.543 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:34:17.543 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:34:17.543 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:34:17.543 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:34:17.543 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:34:17.543 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:34:17.543 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:34:17.561 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:17.694 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:34:17.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:17.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:34:17.745 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:34:17.745 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:34:17.800 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:34:17.853 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:17.892 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:34:17.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:17.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:34:18.002 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:18.057 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:18.091 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:18.103 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:17 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/381299892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:34:18.103 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.103 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.103 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.103 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.103 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.103 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:18.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:18.205 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:18.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:17 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/381299892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:34:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.302 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:18.341 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:18.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:18.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:17 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/381299892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:34:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.450 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:34:18.485 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:18.510 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-23T16:34:18.587 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:34:18.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:18.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:34:18.677 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:34:18.707 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:34:18.757 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:18.814 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:34:18.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:18.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:34:18.856 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:34:18.919 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:34:18.949 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:18.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:18.979 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:18 smithi103 ceph-mon[39629]: pgmap v1148: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 668 op/s; 23143/246858 objects degraded (9.375%); 8.2 MiB/s, 131 objects/s recovering 2024-04-23T16:34:19.014 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:34:19.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:19.017 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:34:19.066 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:34:19.068 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:34:19.118 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:34:19.145 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:19.221 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:34:19.222 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:34:19.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:18 smithi188 ceph-mon[44101]: pgmap v1148: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 668 op/s; 23143/246858 objects degraded (9.375%); 8.2 MiB/s, 131 objects/s recovering 2024-04-23T16:34:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:18 smithi148 ceph-mon[44520]: pgmap v1148: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 668 op/s; 23143/246858 objects degraded (9.375%); 8.2 MiB/s, 131 objects/s recovering 2024-04-23T16:34:19.499 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:34:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:19.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:34:19.603 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:20.153 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1148,"stamp":"2024-04-23T16:34:18.719257+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8089436728,"num_objects":123429,"num_object_clones":0,"num_object_copies":246858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123429,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183765,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5810,"num_bytes_recovered":380899872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162268,"ondisk_log_size":162268,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":17541040,"kb_used_data":14364144,"kb_used_omap":14,"kb_used_meta":3176497,"kb_avail":826165328,"statfs":{"total":863955320832,"available":845993295872,"internally_reserved":0,"allocated":14708883456,"data_stored":14705501949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3252733935},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,1,1,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-438304768,"num_objects":-6688,"num_object_clones":0,"num_object_copies":-13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1315,"num_bytes_recovered":86179840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002059"},"pg_stats":[{"pgid":"5.f","version":"405'11578","reported_seq":11782,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:14.172702+0000","last_change":"2024-04-23T16:33:17.526419+0000","last_active":"2024-04-23T16:34:14.172702+0000","last_peered":"2024-04-23T16:34:14.172702+0000","last_clean":"2024-04-23T16:34:14.172702+0000","last_became_active":"2024-04-23T16:31:29.140923+0000","last_became_peered":"2024-04-23T16:31:29.140923+0000","last_unstale":"2024-04-23T16:34:14.172702+0000","last_undegraded":"2024-04-23T16:34:14.172702+0000","last_fullsized":"2024-04-23T16:34:14.172702+0000","mapping_epoch":359,"log_start":"368'1500","ondisk_log_start":"368'1500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9683","last_scrub_stamp":"2024-04-23T16:33:17.526283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.526283+0000","objects_scrubbed":9683,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:33.530461+0000","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":510394368,"num_objects":7788,"num_object_clones":0,"num_object_copies":15576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11578,"num_write_kb":619712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"405'11429","reported_seq":11706,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:13.979405+0000","last_change":"2024-04-23T16:34:02.531358+0000","last_active":"2024-04-23T16:34:13.979405+0000","last_peered":"2024-04-23T16:34:13.979405+0000","last_clean":"2024-04-23T16:34:13.979405+0000","last_became_active":"2024-04-23T16:31:29.137890+0000","last_became_peered":"2024-04-23T16:31:29.137890+0000","last_unstale":"2024-04-23T16:34:13.979405+0000","last_undegraded":"2024-04-23T16:34:13.979405+0000","last_fullsized":"2024-04-23T16:34:13.979405+0000","mapping_epoch":359,"log_start":"368'1400","ondisk_log_start":"368'1400","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'10975","last_scrub_stamp":"2024-04-23T16:34:02.531162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:02.531162+0000","objects_scrubbed":8314,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:09.558502+0000","scrub_duration":1066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":513474560,"num_objects":7835,"num_object_clones":0,"num_object_copies":15670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11429,"num_write_kb":616448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"406'11730","reported_seq":15953,"reported_epoch":406,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:34:17.934116+0000","last_change":"2024-04-23T16:33:26.432192+0000","last_active":"2024-04-23T16:34:17.934116+0000","last_peered":"2024-04-23T16:34:17.934116+0000","last_clean":"2024-04-23T16:32:59.749530+0000","last_became_active":"2024-04-23T16:33:26.338644+0000","last_became_peered":"2024-04-23T16:33:26.338644+0000","last_unstale":"2024-04-23T16:34:17.934116+0000","last_undegraded":"2024-04-23T16:33:25.874847+0000","last_fullsized":"2024-04-23T16:34:17.934116+0000","mapping_epoch":394,"log_start":"369'1700","ondisk_log_start":"369'1700","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8711","last_scrub_stamp":"2024-04-23T16:32:43.928914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.928914+0000","objects_scrubbed":8680,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505413632,"num_objects":7712,"num_object_clones":0,"num_object_copies":15424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7702,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11730,"num_write_kb":622144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":131727360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":["8"],"object_location_counts":[{"shards":"4,8","objects":10},{"shards":"8","objects":7702}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.9","version":"405'11555","reported_seq":11745,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:15.064537+0000","last_change":"2024-04-23T16:32:37.256944+0000","last_active":"2024-04-23T16:34:15.064537+0000","last_peered":"2024-04-23T16:34:15.064537+0000","last_clean":"2024-04-23T16:34:15.064537+0000","last_became_active":"2024-04-23T16:31:29.141551+0000","last_became_peered":"2024-04-23T16:31:29.141551+0000","last_unstale":"2024-04-23T16:34:15.064537+0000","last_undegraded":"2024-04-23T16:34:15.064537+0000","last_fullsized":"2024-04-23T16:34:15.064537+0000","mapping_epoch":359,"log_start":"368'1500","ondisk_log_start":"368'1500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8156","last_scrub_stamp":"2024-04-23T16:32:37.256787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:37.256787+0000","objects_scrubbed":8082,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":508887040,"num_objects":7765,"num_object_clones":0,"num_object_copies":15530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11555,"num_write_kb":618240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"406'11288","reported_seq":11493,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-04-23T16:34:16.726838+0000","last_change":"2024-04-23T16:33:25.105693+0000","last_active":"2024-04-23T16:34:16.726838+0000","last_peered":"2024-04-23T16:34:16.726838+0000","last_clean":"2024-04-23T16:34:16.726838+0000","last_became_active":"2024-04-23T16:31:29.139045+0000","last_became_peered":"2024-04-23T16:31:29.139045+0000","last_unstale":"2024-04-23T16:34:16.726838+0000","last_undegraded":"2024-04-23T16:34:16.726838+0000","last_fullsized":"2024-04-23T16:34:16.726838+0000","mapping_epoch":359,"log_start":"367'1200","ondisk_log_start":"367'1200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9367","last_scrub_stamp":"2024-04-23T16:33:25.105563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:25.105563+0000","objects_scrubbed":9367,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:25.368946+0000","scrub_duration":751,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487981056,"num_objects":7446,"num_object_clones":0,"num_object_copies":14892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11288,"num_write_kb":599488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"405'11405","reported_seq":11610,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:14.002227+0000","last_change":"2024-04-23T16:33:22.103034+0000","last_active":"2024-04-23T16:34:14.002227+0000","last_peered":"2024-04-23T16:34:14.002227+0000","last_clean":"2024-04-23T16:34:14.002227+0000","last_became_active":"2024-04-23T16:31:29.310906+0000","last_became_peered":"2024-04-23T16:31:29.310906+0000","last_unstale":"2024-04-23T16:34:14.002227+0000","last_undegraded":"2024-04-23T16:34:14.002227+0000","last_fullsized":"2024-04-23T16:34:14.002227+0000","mapping_epoch":359,"log_start":"368'1400","ondisk_log_start":"368'1400","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9559","last_scrub_stamp":"2024-04-23T16:33:22.102857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:22.102857+0000","objects_scrubbed":9559,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:45.157065+0000","scrub_duration":758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505610240,"num_objects":7715,"num_object_clones":0,"num_object_copies":15430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11405,"num_write_kb":611840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1063,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:05.925663+0000","last_change":"2024-04-23T16:33:31.424170+0000","last_active":"2024-04-23T16:34:05.925663+0000","last_peered":"2024-04-23T16:34:05.925663+0000","last_clean":"2024-04-23T16:34:05.925663+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:34:05.925663+0000","last_undegraded":"2024-04-23T16:34:05.925663+0000","last_fullsized":"2024-04-23T16:34:05.925663+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:31.424058+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:33:31.424058+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:35.317699+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"405'11614","reported_seq":11805,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:14.287548+0000","last_change":"2024-04-23T16:32:38.103756+0000","last_active":"2024-04-23T16:34:14.287548+0000","last_peered":"2024-04-23T16:34:14.287548+0000","last_clean":"2024-04-23T16:34:14.287548+0000","last_became_active":"2024-04-23T16:31:29.141640+0000","last_became_peered":"2024-04-23T16:31:29.141640+0000","last_unstale":"2024-04-23T16:34:14.287548+0000","last_undegraded":"2024-04-23T16:34:14.287548+0000","last_fullsized":"2024-04-23T16:34:14.287548+0000","mapping_epoch":359,"log_start":"369'1600","ondisk_log_start":"369'1600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8300","last_scrub_stamp":"2024-04-23T16:32:38.103547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:38.103547+0000","objects_scrubbed":8231,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":513933312,"num_objects":7842,"num_object_clones":0,"num_object_copies":15684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11614,"num_write_kb":622592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1065,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:05.923719+0000","last_change":"2024-04-23T16:33:25.851379+0000","last_active":"2024-04-23T16:34:05.923719+0000","last_peered":"2024-04-23T16:34:05.923719+0000","last_clean":"2024-04-23T16:34:05.923719+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:34:05.923719+0000","last_undegraded":"2024-04-23T16:34:05.923719+0000","last_fullsized":"2024-04-23T16:34:05.923719+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.498820+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.498820+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"406'11488","reported_seq":11696,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-04-23T16:34:16.751124+0000","last_change":"2024-04-23T16:33:26.135212+0000","last_active":"2024-04-23T16:34:16.751124+0000","last_peered":"2024-04-23T16:34:16.751124+0000","last_clean":"2024-04-23T16:34:16.751124+0000","last_became_active":"2024-04-23T16:31:29.140251+0000","last_became_peered":"2024-04-23T16:31:29.140251+0000","last_unstale":"2024-04-23T16:34:16.751124+0000","last_undegraded":"2024-04-23T16:34:16.751124+0000","last_fullsized":"2024-04-23T16:34:16.751124+0000","mapping_epoch":359,"log_start":"368'1400","ondisk_log_start":"368'1400","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9565","last_scrub_stamp":"2024-04-23T16:33:26.135088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:26.135088+0000","objects_scrubbed":9565,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:28.491332+0000","scrub_duration":759,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":500891672,"num_objects":7644,"num_object_clones":0,"num_object_copies":15288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7644,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11488,"num_write_kb":612161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1056,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:05.936545+0000","last_change":"2024-04-23T16:32:29.517947+0000","last_active":"2024-04-23T16:34:05.936545+0000","last_peered":"2024-04-23T16:34:05.936545+0000","last_clean":"2024-04-23T16:34:05.936545+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:34:05.936545+0000","last_undegraded":"2024-04-23T16:34:05.936545+0000","last_fullsized":"2024-04-23T16:34:05.936545+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:32:29.517800+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:32:29.517800+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"405'11607","reported_seq":15640,"reported_epoch":405,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:34:14.165432+0000","last_change":"2024-04-23T16:33:26.507399+0000","last_active":"2024-04-23T16:34:14.165432+0000","last_peered":"2024-04-23T16:34:14.165432+0000","last_clean":"2024-04-23T16:32:59.715575+0000","last_became_active":"2024-04-23T16:33:26.458611+0000","last_became_peered":"2024-04-23T16:33:26.458611+0000","last_unstale":"2024-04-23T16:34:14.165432+0000","last_undegraded":"2024-04-23T16:33:25.895472+0000","last_fullsized":"2024-04-23T16:34:14.165432+0000","mapping_epoch":394,"log_start":"368'1600","ondisk_log_start":"368'1600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8607","last_scrub_stamp":"2024-04-23T16:32:43.130806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.130806+0000","objects_scrubbed":8577,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":509411328,"num_objects":7773,"num_object_clones":0,"num_object_copies":15546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11607,"num_write_kb":620160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1917,"num_bytes_recovered":125632512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":7770},{"shards":"5,7","objects":3}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1062,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:05.930639+0000","last_change":"2024-04-23T16:33:33.275883+0000","last_active":"2024-04-23T16:34:05.930639+0000","last_peered":"2024-04-23T16:34:05.930639+0000","last_clean":"2024-04-23T16:34:05.930639+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:34:05.930639+0000","last_undegraded":"2024-04-23T16:34:05.930639+0000","last_fullsized":"2024-04-23T16:34:05.930639+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:33.275799+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:33:33.275799+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:42.780661+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"405'11345","reported_seq":11548,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:14.036976+0000","last_change":"2024-04-23T16:33:18.576077+0000","last_active":"2024-04-23T16:34:14.036976+0000","last_peered":"2024-04-23T16:34:14.036976+0000","last_clean":"2024-04-23T16:34:14.036976+0000","last_became_active":"2024-04-23T16:31:29.139709+0000","last_became_peered":"2024-04-23T16:31:29.139709+0000","last_unstale":"2024-04-23T16:34:14.036976+0000","last_undegraded":"2024-04-23T16:34:14.036976+0000","last_fullsized":"2024-04-23T16:34:14.036976+0000","mapping_epoch":359,"log_start":"367'1300","ondisk_log_start":"367'1300","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9472","last_scrub_stamp":"2024-04-23T16:33:18.575945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.575945+0000","objects_scrubbed":9472,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:42.303216+0000","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498008064,"num_objects":7599,"num_object_clones":0,"num_object_copies":15198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11345,"num_write_kb":606208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1063,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:05.931379+0000","last_change":"2024-04-23T16:34:02.399102+0000","last_active":"2024-04-23T16:34:05.931379+0000","last_peered":"2024-04-23T16:34:05.931379+0000","last_clean":"2024-04-23T16:34:05.931379+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:34:05.931379+0000","last_undegraded":"2024-04-23T16:34:05.931379+0000","last_fullsized":"2024-04-23T16:34:05.931379+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:02.398975+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:02.398975+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:23.945801+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"405'11531","reported_seq":11738,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:14.050706+0000","last_change":"2024-04-23T16:33:19.938285+0000","last_active":"2024-04-23T16:34:14.050706+0000","last_peered":"2024-04-23T16:34:14.050706+0000","last_clean":"2024-04-23T16:34:14.050706+0000","last_became_active":"2024-04-23T16:31:29.139606+0000","last_became_peered":"2024-04-23T16:31:29.139606+0000","last_unstale":"2024-04-23T16:34:14.050706+0000","last_undegraded":"2024-04-23T16:34:14.050706+0000","last_fullsized":"2024-04-23T16:34:14.050706+0000","mapping_epoch":359,"log_start":"369'1500","ondisk_log_start":"369'1500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9720","last_scrub_stamp":"2024-04-23T16:33:19.938119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:19.938119+0000","objects_scrubbed":9720,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:34.186612+0000","scrub_duration":794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":518324224,"num_objects":7909,"num_object_clones":0,"num_object_copies":15818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11531,"num_write_kb":622080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1062,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:05.925708+0000","last_change":"2024-04-23T16:33:28.454288+0000","last_active":"2024-04-23T16:34:05.925708+0000","last_peered":"2024-04-23T16:34:05.925708+0000","last_clean":"2024-04-23T16:34:05.925708+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:34:05.925708+0000","last_undegraded":"2024-04-23T16:34:05.925708+0000","last_fullsized":"2024-04-23T16:34:05.925708+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:28.454200+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:33:28.454200+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:51.316141+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.2","version":"405'11389","reported_seq":11671,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:14.492668+0000","last_change":"2024-04-23T16:33:57.879648+0000","last_active":"2024-04-23T16:34:14.492668+0000","last_peered":"2024-04-23T16:34:14.492668+0000","last_clean":"2024-04-23T16:34:14.492668+0000","last_became_active":"2024-04-23T16:31:29.138430+0000","last_became_peered":"2024-04-23T16:31:29.138430+0000","last_unstale":"2024-04-23T16:34:14.492668+0000","last_undegraded":"2024-04-23T16:34:14.492668+0000","last_fullsized":"2024-04-23T16:34:14.492668+0000","mapping_epoch":359,"log_start":"367'1300","ondisk_log_start":"367'1300","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'10726","last_scrub_stamp":"2024-04-23T16:33:57.879516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:57.879516+0000","objects_scrubbed":8303,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:02.097116+0000","scrub_duration":1621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498532352,"num_objects":7607,"num_object_clones":0,"num_object_copies":15214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11389,"num_write_kb":607872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1063,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:05.925943+0000","last_change":"2024-04-23T16:33:18.422277+0000","last_active":"2024-04-23T16:34:05.925943+0000","last_peered":"2024-04-23T16:34:05.925943+0000","last_clean":"2024-04-23T16:34:05.925943+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:34:05.925943+0000","last_undegraded":"2024-04-23T16:34:05.925943+0000","last_fullsized":"2024-04-23T16:34:05.925943+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:18.422171+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.422171+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:22.642619+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"405'13676","reported_seq":14010,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:14.012526+0000","last_change":"2024-04-23T16:33:17.974701+0000","last_active":"2024-04-23T16:34:14.012526+0000","last_peered":"2024-04-23T16:34:14.012526+0000","last_clean":"2024-04-23T16:34:14.012526+0000","last_became_active":"2024-04-23T16:31:29.139398+0000","last_became_peered":"2024-04-23T16:31:29.139398+0000","last_unstale":"2024-04-23T16:34:14.012526+0000","last_undegraded":"2024-04-23T16:34:14.012526+0000","last_fullsized":"2024-04-23T16:34:14.012526+0000","mapping_epoch":359,"log_start":"371'3600","ondisk_log_start":"371'3600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'11838","last_scrub_stamp":"2024-04-23T16:33:17.974549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.974549+0000","objects_scrubbed":9564,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:28.837234+0000","scrub_duration":848,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506331136,"num_objects":7726,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11402,"num_write_kb":612096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1063,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:05.931388+0000","last_change":"2024-04-23T16:34:04.444103+0000","last_active":"2024-04-23T16:34:05.931388+0000","last_peered":"2024-04-23T16:34:05.931388+0000","last_clean":"2024-04-23T16:34:05.931388+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:34:05.931388+0000","last_undegraded":"2024-04-23T16:34:05.931388+0000","last_fullsized":"2024-04-23T16:34:05.931388+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:04.443975+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:04.443975+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:05.109970+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"405'11310","reported_seq":11503,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:14.287539+0000","last_change":"2024-04-23T16:32:43.947374+0000","last_active":"2024-04-23T16:34:14.287539+0000","last_peered":"2024-04-23T16:34:14.287539+0000","last_clean":"2024-04-23T16:34:14.287539+0000","last_became_active":"2024-04-23T16:31:29.138968+0000","last_became_peered":"2024-04-23T16:31:29.138968+0000","last_unstale":"2024-04-23T16:34:14.287539+0000","last_undegraded":"2024-04-23T16:34:14.287539+0000","last_fullsized":"2024-04-23T16:34:14.287539+0000","mapping_epoch":359,"log_start":"368'1300","ondisk_log_start":"368'1300","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8482","last_scrub_stamp":"2024-04-23T16:32:43.947177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.947177+0000","objects_scrubbed":8428,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":497025024,"num_objects":7584,"num_object_clones":0,"num_object_copies":15168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11310,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":956,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-04-23T16:34:15.159206+0000","last_change":"2024-04-23T16:33:19.443898+0000","last_active":"2024-04-23T16:34:15.159206+0000","last_peered":"2024-04-23T16:34:15.159206+0000","last_clean":"2024-04-23T16:34:15.159206+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:34:15.159206+0000","last_undegraded":"2024-04-23T16:34:15.159206+0000","last_fullsized":"2024-04-23T16:34:15.159206+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:19.443792+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:33:19.443792+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:29.564987+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"405'11656","reported_seq":11848,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:14.338032+0000","last_change":"2024-04-23T16:32:39.103230+0000","last_active":"2024-04-23T16:34:14.338032+0000","last_peered":"2024-04-23T16:34:14.338032+0000","last_clean":"2024-04-23T16:34:14.338032+0000","last_became_active":"2024-04-23T16:31:29.310246+0000","last_became_peered":"2024-04-23T16:31:29.310246+0000","last_unstale":"2024-04-23T16:34:14.338032+0000","last_undegraded":"2024-04-23T16:34:14.338032+0000","last_fullsized":"2024-04-23T16:34:14.338032+0000","mapping_epoch":359,"log_start":"369'1600","ondisk_log_start":"369'1600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8325","last_scrub_stamp":"2024-04-23T16:32:39.103059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:39.103059+0000","objects_scrubbed":8272,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":511574016,"num_objects":7806,"num_object_clones":0,"num_object_copies":15612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11656,"num_write_kb":622784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":876,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:05.931338+0000","last_change":"2024-04-23T16:33:26.153079+0000","last_active":"2024-04-23T16:34:05.931338+0000","last_peered":"2024-04-23T16:34:05.931338+0000","last_clean":"2024-04-23T16:34:05.931338+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:34:05.931338+0000","last_undegraded":"2024-04-23T16:34:05.931338+0000","last_fullsized":"2024-04-23T16:34:05.931338+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:26.152962+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:33:26.152962+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:53.292007+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"405'11438","reported_seq":15396,"reported_epoch":405,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:34:14.152471+0000","last_change":"2024-04-23T16:33:26.132487+0000","last_active":"2024-04-23T16:34:14.152471+0000","last_peered":"2024-04-23T16:34:14.152471+0000","last_clean":"2024-04-23T16:32:59.695133+0000","last_became_active":"2024-04-23T16:33:26.078062+0000","last_became_peered":"2024-04-23T16:33:26.078062+0000","last_unstale":"2024-04-23T16:34:14.152471+0000","last_undegraded":"2024-04-23T16:33:25.892009+0000","last_fullsized":"2024-04-23T16:34:14.152471+0000","mapping_epoch":394,"log_start":"368'1400","ondisk_log_start":"368'1400","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'7968","last_scrub_stamp":"2024-04-23T16:32:36.102122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:36.102122+0000","objects_scrubbed":7926,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503054336,"num_objects":7676,"num_object_clones":0,"num_object_copies":15352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7671,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11438,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1882,"num_bytes_recovered":123338752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":7671},{"shards":"5,8","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1061,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-23T16:34:05.936431+0000","last_change":"2024-04-23T16:33:17.354409+0000","last_active":"2024-04-23T16:34:05.936431+0000","last_peered":"2024-04-23T16:34:05.936431+0000","last_clean":"2024-04-23T16:34:05.936431+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:34:05.936431+0000","last_undegraded":"2024-04-23T16:34:05.936431+0000","last_fullsized":"2024-04-23T16:34:05.936431+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:17.354263+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.354263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:39.845454+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":8088846360,"num_objects":123427,"num_object_clones":0,"num_object_copies":246854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123427,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183765,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5809,"num_bytes_recovered":380698624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16549486592,"data_stored":16549478448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160739,"ondisk_log_size":160739,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1475340,"kb_used_data":970632,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":92269812,"statfs":{"total":95995035648,"available":94484287488,"internally_reserved":0,"allocated":993927168,"data_stored":993591187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":516684228},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351209,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612847,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1802104,"kb_used_data":1486320,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":91943048,"statfs":{"total":95995035648,"available":94149681152,"internally_reserved":0,"allocated":1521991680,"data_stored":1521605617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":323353024},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841780,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1173128,"kb_used_data":983556,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92572024,"statfs":{"total":95995035648,"available":94793752576,"internally_reserved":0,"allocated":1007161344,"data_stored":1006829435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,0,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168824,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":845176,"kb_used_data":498864,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92899976,"statfs":{"total":95995035648,"available":95129575424,"internally_reserved":0,"allocated":510836736,"data_stored":510450673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":354613706},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463166,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1742072,"kb_used_data":1480432,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":92003080,"statfs":{"total":95995035648,"available":94211153920,"internally_reserved":0,"allocated":1515962368,"data_stored":1515576305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":267909574},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137306,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738651,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3834232,"kb_used_data":3478192,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":89910920,"statfs":{"total":95995035648,"available":92068782080,"internally_reserved":0,"allocated":3561668608,"data_stored":3561282545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":863680,"kb_used_data":499320,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":92881472,"statfs":{"total":95995035648,"available":95110627328,"internally_reserved":0,"allocated":511303680,"data_stored":510914065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921030,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2935864,"kb_used_data":2493680,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":90809288,"statfs":{"total":95995035648,"available":92988710912,"internally_reserved":0,"allocated":2553528320,"data_stored":2553142257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":452721093},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757505,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2869444,"kb_used_data":2473148,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":90875708,"statfs":{"total":95995035648,"available":93056724992,"internally_reserved":0,"allocated":2532503552,"data_stored":2532109865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3544449024,"data_stored":3544449024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1886846976,"data_stored":1886846976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":508297216,"data_stored":508297216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2550792192,"data_stored":2550792192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2528382976,"data_stored":2528378904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1513226240,"data_stored":1513226240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":509149184,"data_stored":509149184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1002373120,"data_stored":1002373120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1517092864,"data_stored":1517092864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":988876800,"data_stored":988872728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:34:20.154 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.a is not active+clean 2024-04-23T16:34:20.154 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.a', 'version': "406'11730", 'reported_seq': 15953, 'reported_epoch': 406, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:34:17.934116+0000', 'last_change': '2024-04-23T16:33:26.432192+0000', 'last_active': '2024-04-23T16:34:17.934116+0000', 'last_peered': '2024-04-23T16:34:17.934116+0000', 'last_clean': '2024-04-23T16:32:59.749530+0000', 'last_became_active': '2024-04-23T16:33:26.338644+0000', 'last_became_peered': '2024-04-23T16:33:26.338644+0000', 'last_unstale': '2024-04-23T16:34:17.934116+0000', 'last_undegraded': '2024-04-23T16:33:25.874847+0000', 'last_fullsized': '2024-04-23T16:34:17.934116+0000', 'mapping_epoch': 394, 'log_start': "369'1700", 'ondisk_log_start': "369'1700", 'created': 359, 'last_epoch_clean': 360, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "382'8711", 'last_scrub_stamp': '2024-04-23T16:32:43.928914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:31:28.121358+0000', 'last_clean_scrub_stamp': '2024-04-23T16:32:43.928914+0000', 'objects_scrubbed': 8680, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1273, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 505413632, 'num_objects': 7712, 'num_object_clones': 0, 'num_object_copies': 15424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 7702, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7712, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11730, 'num_write_kb': 622144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2010, 'num_bytes_recovered': 131727360, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [8, 4], 'acting': [8, 4], 'avail_no_missing': ['8'], 'object_location_counts': [{'shards': '4,8', 'objects': 10}, {'shards': '8', 'objects': 7702}], 'blocked_by': [], 'up_primary': 8, 'acting_primary': 8, 'purged_snaps': []} 2024-04-23T16:34:20.154 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.6 is not active+clean 2024-04-23T16:34:20.155 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.6', 'version': "405'11607", 'reported_seq': 15640, 'reported_epoch': 405, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:34:14.165432+0000', 'last_change': '2024-04-23T16:33:26.507399+0000', 'last_active': '2024-04-23T16:34:14.165432+0000', 'last_peered': '2024-04-23T16:34:14.165432+0000', 'last_clean': '2024-04-23T16:32:59.715575+0000', 'last_became_active': '2024-04-23T16:33:26.458611+0000', 'last_became_peered': '2024-04-23T16:33:26.458611+0000', 'last_unstale': '2024-04-23T16:34:14.165432+0000', 'last_undegraded': '2024-04-23T16:33:25.895472+0000', 'last_fullsized': '2024-04-23T16:34:14.165432+0000', 'mapping_epoch': 394, 'log_start': "368'1600", 'ondisk_log_start': "368'1600", 'created': 359, 'last_epoch_clean': 360, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "382'8607", 'last_scrub_stamp': '2024-04-23T16:32:43.130806+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:31:28.121358+0000', 'last_clean_scrub_stamp': '2024-04-23T16:32:43.130806+0000', 'objects_scrubbed': 8577, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1484, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 509411328, 'num_objects': 7773, 'num_object_clones': 0, 'num_object_copies': 15546, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 7770, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7773, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11607, 'num_write_kb': 620160, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1917, 'num_bytes_recovered': 125632512, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': ['5'], 'object_location_counts': [{'shards': '5', 'objects': 7770}, {'shards': '5,7', 'objects': 3}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-23T16:34:20.155 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.4 is not active+clean 2024-04-23T16:34:20.155 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.4', 'version': "405'11438", 'reported_seq': 15396, 'reported_epoch': 405, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:34:14.152471+0000', 'last_change': '2024-04-23T16:33:26.132487+0000', 'last_active': '2024-04-23T16:34:14.152471+0000', 'last_peered': '2024-04-23T16:34:14.152471+0000', 'last_clean': '2024-04-23T16:32:59.695133+0000', 'last_became_active': '2024-04-23T16:33:26.078062+0000', 'last_became_peered': '2024-04-23T16:33:26.078062+0000', 'last_unstale': '2024-04-23T16:34:14.152471+0000', 'last_undegraded': '2024-04-23T16:33:25.892009+0000', 'last_fullsized': '2024-04-23T16:34:14.152471+0000', 'mapping_epoch': 394, 'log_start': "368'1400", 'ondisk_log_start': "368'1400", 'created': 359, 'last_epoch_clean': 360, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "380'7968", 'last_scrub_stamp': '2024-04-23T16:32:36.102122+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:31:28.121358+0000', 'last_clean_scrub_stamp': '2024-04-23T16:32:36.102122+0000', 'objects_scrubbed': 7926, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1466, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 503054336, 'num_objects': 7676, 'num_object_clones': 0, 'num_object_copies': 15352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 7671, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7676, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11438, 'num_write_kb': 611648, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1882, 'num_bytes_recovered': 123338752, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 8], 'acting': [5, 8], 'avail_no_missing': ['5'], 'object_location_counts': [{'shards': '5', 'objects': 7671}, {'shards': '5,8', 'objects': 5}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-23T16:34:20.155 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-23T16:34:20.156 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:19 smithi188 ceph-mon[44101]: from='client.15978 v1:172.21.15.103:0/218577960' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:34:20.156 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:19 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 23143/246858 objects degraded (9.375%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:20.158 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:20.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:19 smithi103 ceph-mon[39629]: from='client.15978 v1:172.21.15.103:0/218577960' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:34:20.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:19 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 23143/246858 objects degraded (9.375%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:20.467 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:19 smithi148 ceph-mon[44520]: from='client.15978 v1:172.21.15.103:0/218577960' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:34:20.467 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:19 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 23143/246858 objects degraded (9.375%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:20.795 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:34:20.795 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:34:21.035 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:21.196 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:21.208 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:21 smithi188 ceph-mon[44101]: pgmap v1149: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.4 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 659 op/s; 22734/241608 objects degraded (9.409%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:34:21.223 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:34:21.299 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:21.337 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:21 smithi103 ceph-mon[39629]: pgmap v1149: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.4 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 659 op/s; 22734/241608 objects degraded (9.409%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:34:21.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:21.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:21.509 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:21 smithi148 ceph-mon[44520]: pgmap v1149: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.4 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 659 op/s; 22734/241608 objects degraded (9.409%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:34:21.519 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:21.553 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-23T16:34:21.892 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:34:22.579 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:22.634 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:22.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:22.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:22.740 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:34:22.775 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:22.801 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:34:22.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:22.805 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:34:22.906 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:23.301 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:23.329 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:23.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:23.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:23.439 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:34:23.469 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:23.594 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:34:23.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:23.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:34:23.700 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:34:23.733 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:23.765 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:34:23.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:23.769 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:34:23.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:23 smithi103 ceph-mon[39629]: pgmap v1150: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.3 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 658 op/s; 22734/240784 objects degraded (9.442%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:34:23.869 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:34:23.907 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:23.925 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:34:23.925 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:34:23.929 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:34:23.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:23.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:34:23.952 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:23 smithi188 ceph-mon[44101]: pgmap v1150: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.3 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 658 op/s; 22734/240784 objects degraded (9.442%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:34:24.033 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:34:24.066 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:23 smithi148 ceph-mon[44520]: pgmap v1150: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.3 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 658 op/s; 22734/240784 objects degraded (9.442%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:34:24.189 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:24.191 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:34:24.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:24.199 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:34:24.300 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:34:24.331 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:24.333 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:34:24.333 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22734/240784 objects degraded (9.442%), 3 pgs degraded","count":3},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1446,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":407,"num_osds":12,"num_up_osds":11,"osd_up_since":1713889979,"num_in_osds":9,"osd_in_since":1713890004,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":24},{"state_name":"active+recovering+degraded","count":3}],"num_pgs":27,"num_pools":2,"num_objects":120392,"data_bytes":7890403896,"bytes_used":17631543296,"bytes_avail":846323777536,"bytes_total":863955320832,"degraded_objects":22734,"degraded_total":240784,"degraded_ratio":0.094416572529736176,"recovering_objects_per_sec":128,"recovering_bytes_per_sec":8432805,"recovering_keys_per_sec":0,"num_objects_recovered":1287,"num_bytes_recovered":84344832,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":658},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.103:8443/"}},"servicemap":{"epoch":32,"modified":"2024-04-23T16:32:56.701867+0000","services":{}},"progress_events":{}} 2024-04-23T16:34:24.622 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-23T16:34:24.808 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:34:24.964 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:34:24.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:24 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:34:24.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:24 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:34:24.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:24 smithi188 ceph-mon[44101]: osdmap e407: 12 total, 11 up, 9 in 2024-04-23T16:34:24.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/57110738' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:34:24.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:24 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 22734/240784 objects degraded (9.442%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:24.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/757844127' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:34:25.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:24 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:34:25.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:24 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:34:25.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:24 smithi103 ceph-mon[39629]: osdmap e407: 12 total, 11 up, 9 in 2024-04-23T16:34:25.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/57110738' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:34:25.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:24 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 22734/240784 objects degraded (9.442%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:25.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/757844127' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:34:25.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:24 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:34:25.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:24 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:34:25.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:24 smithi148 ceph-mon[44520]: osdmap e407: 12 total, 11 up, 9 in 2024-04-23T16:34:25.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:24 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/57110738' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:34:25.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:24 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 22734/240784 objects degraded (9.442%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:25.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:24 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/757844127' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:34:25.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:25 smithi188 ceph-mon[44101]: pgmap v1152: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 660 op/s; 22047/235440 objects degraded (9.364%); 8.8 MiB/s, 140 objects/s recovering 2024-04-23T16:34:25.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:25 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:34:25.800 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:25 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/57110738' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:34:25.801 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:25 smithi188 ceph-mon[44101]: osdmap e408: 12 total, 11 up, 9 in 2024-04-23T16:34:25.954 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:25 smithi103 ceph-mon[39629]: pgmap v1152: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 660 op/s; 22047/235440 objects degraded (9.364%); 8.8 MiB/s, 140 objects/s recovering 2024-04-23T16:34:25.954 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:25 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:34:25.954 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:25 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/57110738' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:34:25.954 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:25 smithi103 ceph-mon[39629]: osdmap e408: 12 total, 11 up, 9 in 2024-04-23T16:34:25.955 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-23T16:34:25.956 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:34:26.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:25 smithi148 ceph-mon[44520]: pgmap v1152: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 660 op/s; 22047/235440 objects degraded (9.364%); 8.8 MiB/s, 140 objects/s recovering 2024-04-23T16:34:26.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:25 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:34:26.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:25 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/57110738' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:34:26.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:25 smithi148 ceph-mon[44520]: osdmap e408: 12 total, 11 up, 9 in 2024-04-23T16:34:27.270 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:34:27.270 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:34:27.718 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:34:27.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:27.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:34:27.822 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:34:27.850 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:27.942 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:27 smithi103 ceph-mon[39629]: pgmap v1154: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 741 op/s; 22047/234990 objects degraded (9.382%); 8.6 MiB/s, 136 objects/s recovering 2024-04-23T16:34:27.943 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:27 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:34:27.943 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:27 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:34:27.951 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:34:27.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:27.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:34:27.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:27 smithi188 ceph-mon[44101]: pgmap v1154: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 741 op/s; 22047/234990 objects degraded (9.382%); 8.6 MiB/s, 136 objects/s recovering 2024-04-23T16:34:27.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:27 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:34:27.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:27 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:34:27.990 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-23T16:34:28.057 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:28.116 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:27 smithi148 ceph-mon[44520]: pgmap v1154: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 741 op/s; 22047/234990 objects degraded (9.382%); 8.6 MiB/s, 136 objects/s recovering 2024-04-23T16:34:28.116 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:27 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:34:28.116 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:27 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:34:28.125 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:28.154 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:28.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:28.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:28.200 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:28.244 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:34:28.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:34:28.269 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:34:28.291 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:28.295 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:34:28.332 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:34:28.342 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:34:28.363 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:34:28.389 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:34:28.410 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:34:28.432 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:34:28.453 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:34:28.473 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:34:28.492 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:34:28.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:28.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:34:28.496 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:34:28.497 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:34:28.497 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:34:28.497 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:34:28.497 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:34:28.497 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:34:28.497 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:34:28.497 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:34:28.497 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:34:28.497 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:34:28.497 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:34:28.497 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:34:28.497 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:34:28.592 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:34:28.596 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:34:28.666 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:28.914 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:28 smithi103 ceph-mon[39629]: osdmap e409: 12 total, 11 up, 9 in 2024-04-23T16:34:28.914 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:28 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4254663490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:34:28.914 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:28.914 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:28.914 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:28.914 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:28.914 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:28.914 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:28 smithi188 ceph-mon[44101]: osdmap e409: 12 total, 11 up, 9 in 2024-04-23T16:34:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:28 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4254663490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:34:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:28 smithi148 ceph-mon[44520]: osdmap e409: 12 total, 11 up, 9 in 2024-04-23T16:34:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:28 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4254663490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:34:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:29.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:29.928 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:34:29.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:29.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:34:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:29 smithi188 ceph-mon[44101]: pgmap v1156: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.1 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 579 op/s; 21849/233828 objects degraded (9.344%); 9.2 MiB/s, 147 objects/s recovering 2024-04-23T16:34:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:29 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 21849/233828 objects degraded (9.344%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:30.033 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:34:30.062 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:29 smithi103 ceph-mon[39629]: pgmap v1156: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.1 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 579 op/s; 21849/233828 objects degraded (9.344%); 9.2 MiB/s, 147 objects/s recovering 2024-04-23T16:34:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:29 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 21849/233828 objects degraded (9.344%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:29 smithi148 ceph-mon[44520]: pgmap v1156: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.1 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 579 op/s; 21849/233828 objects degraded (9.344%); 9.2 MiB/s, 147 objects/s recovering 2024-04-23T16:34:30.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:29 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 21849/233828 objects degraded (9.344%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:30.330 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:34:30.330 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-04-23T16:34:30.330 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-04-23T16:34:30.330 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:34:30.331 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.331 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208917 3.4 3.eacf0934 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:34:30.331 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085422+0000", 2024-04-23T16:34:30.331 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 579.24065559999997, 2024-04-23T16:34:30.331 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70691126599999998, 2024-04-23T16:34:30.331 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.332 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.332 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.332 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.332 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.332 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.332 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208917 2024-04-23T16:34:30.332 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.333 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.333 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.333 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.333 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085422+0000", 2024-04-23T16:34:30.333 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.333 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.334 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.334 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.334 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085422+0000", 2024-04-23T16:34:30.334 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.334 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.334 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.335 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.335 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085423+0000", 2024-04-23T16:34:30.335 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2309999999999999e-06 2024-04-23T16:34:30.335 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.335 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.335 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.335 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085428+0000", 2024-04-23T16:34:30.335 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2870000000000004e-06 2024-04-23T16:34:30.335 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.335 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.336 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.336 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085437+0000", 2024-04-23T16:34:30.336 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2150000000000007e-06 2024-04-23T16:34:30.336 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.336 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.336 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.336 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085444+0000", 2024-04-23T16:34:30.336 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9179999999999997e-06 2024-04-23T16:34:30.336 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.337 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.337 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.337 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085498+0000", 2024-04-23T16:34:30.337 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4088000000000001e-05 2024-04-23T16:34:30.337 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.337 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.337 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.337 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085509+0000", 2024-04-23T16:34:30.337 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1009000000000001e-05 2024-04-23T16:34:30.337 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.338 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.338 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.338 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792295+0000", 2024-04-23T16:34:30.338 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70678534800000004 2024-04-23T16:34:30.338 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.338 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.338 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.338 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792333+0000", 2024-04-23T16:34:30.338 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8170000000000002e-05 2024-04-23T16:34:30.339 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.339 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.339 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.339 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.339 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.339 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208925 3.8 3.a4efd988 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:34:30.339 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085458+0000", 2024-04-23T16:34:30.339 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 579.24061974100005, 2024-04-23T16:34:30.339 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70667339100000004, 2024-04-23T16:34:30.339 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.340 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.340 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.340 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.340 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.340 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.340 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208925 2024-04-23T16:34:30.340 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.340 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.340 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.340 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.341 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085458+0000", 2024-04-23T16:34:30.341 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.341 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.341 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.341 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.341 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085458+0000", 2024-04-23T16:34:30.341 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.341 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.341 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.341 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.342 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085458+0000", 2024-04-23T16:34:30.342 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6000000000000003e-07 2024-04-23T16:34:30.342 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.342 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.342 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.342 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085462+0000", 2024-04-23T16:34:30.342 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4450000000000001e-06 2024-04-23T16:34:30.342 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.342 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.342 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.343 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085468+0000", 2024-04-23T16:34:30.343 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4489999999999996e-06 2024-04-23T16:34:30.343 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.343 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.343 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.343 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085473+0000", 2024-04-23T16:34:30.343 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7260000000000002e-06 2024-04-23T16:34:30.343 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.343 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.343 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.344 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085516+0000", 2024-04-23T16:34:30.344 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3013000000000003e-05 2024-04-23T16:34:30.344 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.344 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.344 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.344 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085523+0000", 2024-04-23T16:34:30.344 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0169999999999999e-06 2024-04-23T16:34:30.344 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.344 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.345 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.345 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792056+0000", 2024-04-23T16:34:30.345 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70653289799999996 2024-04-23T16:34:30.345 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.345 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.345 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.345 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792131+0000", 2024-04-23T16:34:30.345 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5183000000000002e-05 2024-04-23T16:34:30.345 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.345 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.345 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.345 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.345 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.346 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208935 3.4 3.ef358d64 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:34:30.346 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085484+0000", 2024-04-23T16:34:30.346 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 579.24059366699998, 2024-04-23T16:34:30.346 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70688482200000002, 2024-04-23T16:34:30.346 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.346 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.346 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.346 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.346 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.346 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.346 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208935 2024-04-23T16:34:30.346 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.346 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.347 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.347 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.347 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085484+0000", 2024-04-23T16:34:30.347 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.347 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.347 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.347 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.347 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085484+0000", 2024-04-23T16:34:30.347 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.347 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.348 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.348 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.348 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085484+0000", 2024-04-23T16:34:30.348 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8500000000000001e-07 2024-04-23T16:34:30.348 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.348 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.348 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.348 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085488+0000", 2024-04-23T16:34:30.348 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3009999999999998e-06 2024-04-23T16:34:30.348 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.348 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.348 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.348 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085492+0000", 2024-04-23T16:34:30.348 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1749999999999998e-06 2024-04-23T16:34:30.348 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.349 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.349 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.349 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085496+0000", 2024-04-23T16:34:30.349 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4190000000000004e-06 2024-04-23T16:34:30.349 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.349 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.349 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.349 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085542+0000", 2024-04-23T16:34:30.350 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5799e-05 2024-04-23T16:34:30.350 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.350 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.350 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.350 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085551+0000", 2024-04-23T16:34:30.350 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0860000000000008e-06 2024-04-23T16:34:30.350 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.350 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.350 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.350 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792360+0000", 2024-04-23T16:34:30.350 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70680912900000004 2024-04-23T16:34:30.351 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.351 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.351 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.351 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792368+0000", 2024-04-23T16:34:30.351 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2279999999999998e-06 2024-04-23T16:34:30.351 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.351 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.351 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.351 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.351 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.351 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208938 3.4 3.b83bb9a4 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:34:30.351 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085502+0000", 2024-04-23T16:34:30.351 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 579.24057501100003, 2024-04-23T16:34:30.352 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70692120599999997, 2024-04-23T16:34:30.352 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.352 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.352 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.352 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.352 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.352 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.352 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208938 2024-04-23T16:34:30.352 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.352 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.352 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.352 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.352 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085502+0000", 2024-04-23T16:34:30.352 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.352 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.353 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.353 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.353 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085502+0000", 2024-04-23T16:34:30.353 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.353 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.353 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.353 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.353 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085503+0000", 2024-04-23T16:34:30.353 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2300000000000001e-07 2024-04-23T16:34:30.353 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.353 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.353 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.353 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085505+0000", 2024-04-23T16:34:30.354 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.469e-06 2024-04-23T16:34:30.354 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.354 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.354 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.354 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085511+0000", 2024-04-23T16:34:30.354 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7819999999999999e-06 2024-04-23T16:34:30.354 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.354 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.355 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.355 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085517+0000", 2024-04-23T16:34:30.355 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7740000000000001e-06 2024-04-23T16:34:30.355 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.355 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.355 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.355 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085578+0000", 2024-04-23T16:34:30.355 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1475000000000003e-05 2024-04-23T16:34:30.355 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.355 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.355 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.355 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085586+0000", 2024-04-23T16:34:30.356 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8140000000000001e-06 2024-04-23T16:34:30.356 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.356 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.356 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.356 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792413+0000", 2024-04-23T16:34:30.356 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70682651100000005 2024-04-23T16:34:30.356 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.356 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.356 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.356 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792424+0000", 2024-04-23T16:34:30.356 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0858e-05 2024-04-23T16:34:30.356 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.356 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.356 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.356 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.357 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.357 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208939 3.8 3.5b6d9248 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:34:30.357 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085539+0000", 2024-04-23T16:34:30.357 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 579.24053878200004, 2024-04-23T16:34:30.357 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70665678499999995, 2024-04-23T16:34:30.357 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.357 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.357 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.357 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.357 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.357 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.357 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208939 2024-04-23T16:34:30.357 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.357 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.358 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.358 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.358 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085539+0000", 2024-04-23T16:34:30.358 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.358 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.358 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.358 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.358 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085539+0000", 2024-04-23T16:34:30.358 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.358 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.358 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.358 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.358 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085539+0000", 2024-04-23T16:34:30.359 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0200000000000001e-07 2024-04-23T16:34:30.359 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.359 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.359 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.359 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085543+0000", 2024-04-23T16:34:30.359 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4209999999999999e-06 2024-04-23T16:34:30.359 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.359 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.359 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.359 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085549+0000", 2024-04-23T16:34:30.359 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.959e-06 2024-04-23T16:34:30.359 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.359 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.359 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.360 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085554+0000", 2024-04-23T16:34:30.360 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7189999999999998e-06 2024-04-23T16:34:30.360 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.360 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.360 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.360 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085587+0000", 2024-04-23T16:34:30.360 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2540999999999999e-05 2024-04-23T16:34:30.360 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.360 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.360 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.361 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085591+0000", 2024-04-23T16:34:30.361 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8510000000000001e-06 2024-04-23T16:34:30.361 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.361 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.361 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.361 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792182+0000", 2024-04-23T16:34:30.361 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70659176199999996 2024-04-23T16:34:30.361 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.361 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.361 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.361 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792195+0000", 2024-04-23T16:34:30.361 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2830000000000001e-05 2024-04-23T16:34:30.361 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.361 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.362 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.362 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.362 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.362 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208948 3.8 3.9d5e98c8 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:34:30.362 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085565+0000", 2024-04-23T16:34:30.362 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 579.24051238000004, 2024-04-23T16:34:30.362 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70682216099999995, 2024-04-23T16:34:30.362 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.362 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.362 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.362 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.362 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.362 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.363 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208948 2024-04-23T16:34:30.363 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.363 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.363 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.363 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.363 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085565+0000", 2024-04-23T16:34:30.363 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.363 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.363 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.363 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.363 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085565+0000", 2024-04-23T16:34:30.364 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.364 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.364 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.364 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085566+0000", 2024-04-23T16:34:30.364 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2600000000000002e-07 2024-04-23T16:34:30.364 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.364 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.365 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085568+0000", 2024-04-23T16:34:30.365 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7769999999999998e-06 2024-04-23T16:34:30.365 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.365 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.365 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.365 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085575+0000", 2024-04-23T16:34:30.365 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0230000000000004e-06 2024-04-23T16:34:30.365 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.365 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.366 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.366 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085580+0000", 2024-04-23T16:34:30.366 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5499999999999996e-06 2024-04-23T16:34:30.366 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.366 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.366 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.366 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085609+0000", 2024-04-23T16:34:30.366 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9332999999999999e-05 2024-04-23T16:34:30.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.367 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.367 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085613+0000", 2024-04-23T16:34:30.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7799999999999998e-06 2024-04-23T16:34:30.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.367 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.368 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792374+0000", 2024-04-23T16:34:30.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70676071600000001 2024-04-23T16:34:30.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.368 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.368 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.368 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792387+0000", 2024-04-23T16:34:30.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3256e-05 2024-04-23T16:34:30.368 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.368 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.369 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.369 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.369 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.369 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208949 3.8 3.32f0e298 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:34:30.369 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085593+0000", 2024-04-23T16:34:30.370 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 579.24048381299997, 2024-04-23T16:34:30.370 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70685236200000001, 2024-04-23T16:34:30.370 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.370 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.370 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.370 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.370 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.370 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.371 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208949 2024-04-23T16:34:30.371 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.371 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.371 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.371 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.371 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085593+0000", 2024-04-23T16:34:30.371 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.371 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.372 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.372 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.372 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085593+0000", 2024-04-23T16:34:30.372 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.372 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.372 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.372 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.372 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085594+0000", 2024-04-23T16:34:30.373 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4000000000000002e-07 2024-04-23T16:34:30.373 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.373 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.373 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.373 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085596+0000", 2024-04-23T16:34:30.373 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4480000000000001e-06 2024-04-23T16:34:30.373 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.373 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.373 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085603+0000", 2024-04-23T16:34:30.374 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2899999999999999e-06 2024-04-23T16:34:30.374 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.374 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085607+0000", 2024-04-23T16:34:30.374 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3610000000000003e-06 2024-04-23T16:34:30.374 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.375 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.375 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085651+0000", 2024-04-23T16:34:30.375 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4075000000000001e-05 2024-04-23T16:34:30.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.375 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.375 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085659+0000", 2024-04-23T16:34:30.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1079999999999993e-06 2024-04-23T16:34:30.376 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.376 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.376 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.376 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792436+0000", 2024-04-23T16:34:30.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70677668500000002 2024-04-23T16:34:30.376 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.376 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.377 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.377 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792446+0000", 2024-04-23T16:34:30.377 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9550000000000004e-06 2024-04-23T16:34:30.377 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.377 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.377 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.378 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.378 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.378 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208950 3.4 3.4636ca44 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:34:30.378 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085613+0000", 2024-04-23T16:34:30.378 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 579.24046440200004, 2024-04-23T16:34:30.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70686611700000002, 2024-04-23T16:34:30.378 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.378 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.379 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.379 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.379 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.379 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.379 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208950 2024-04-23T16:34:30.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.379 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.379 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.379 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085613+0000", 2024-04-23T16:34:30.379 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085613+0000", 2024-04-23T16:34:30.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085613+0000", 2024-04-23T16:34:30.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4799999999999999e-07 2024-04-23T16:34:30.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085615+0000", 2024-04-23T16:34:30.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.951e-06 2024-04-23T16:34:30.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.381 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.381 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.381 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085621+0000", 2024-04-23T16:34:30.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6890000000000001e-06 2024-04-23T16:34:30.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.381 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.381 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.381 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085624+0000", 2024-04-23T16:34:30.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4960000000000001e-06 2024-04-23T16:34:30.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.381 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.382 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085659+0000", 2024-04-23T16:34:30.382 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4516e-05 2024-04-23T16:34:30.382 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.382 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085665+0000", 2024-04-23T16:34:30.382 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.643e-06 2024-04-23T16:34:30.382 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.383 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792468+0000", 2024-04-23T16:34:30.383 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70680334700000003 2024-04-23T16:34:30.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.383 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.383 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792479+0000", 2024-04-23T16:34:30.383 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1127e-05 2024-04-23T16:34:30.383 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.383 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.383 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.383 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208951 3.4 3.7f30f4f4 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:34:30.383 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085630+0000", 2024-04-23T16:34:30.384 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 579.24044688399999, 2024-04-23T16:34:30.384 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70690233800000002, 2024-04-23T16:34:30.384 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.384 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.384 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.384 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.384 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.384 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.384 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208951 2024-04-23T16:34:30.384 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.384 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.384 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.384 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.384 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085630+0000", 2024-04-23T16:34:30.384 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.385 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.385 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.385 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085630+0000", 2024-04-23T16:34:30.385 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.385 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.385 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.385 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085631+0000", 2024-04-23T16:34:30.385 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8299999999999997e-07 2024-04-23T16:34:30.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.385 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.385 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.385 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085633+0000", 2024-04-23T16:34:30.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3970000000000001e-06 2024-04-23T16:34:30.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.386 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.386 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085639+0000", 2024-04-23T16:34:30.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7810000000000002e-06 2024-04-23T16:34:30.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.386 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.386 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085644+0000", 2024-04-23T16:34:30.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7299999999999996e-06 2024-04-23T16:34:30.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.386 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.387 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085693+0000", 2024-04-23T16:34:30.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9537999999999999e-05 2024-04-23T16:34:30.387 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.387 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.387 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.387 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085702+0000", 2024-04-23T16:34:30.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1309999999999995e-06 2024-04-23T16:34:30.387 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.387 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.387 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.387 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792520+0000", 2024-04-23T16:34:30.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70681792300000001 2024-04-23T16:34:30.387 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.387 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.388 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.388 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792533+0000", 2024-04-23T16:34:30.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2354999999999999e-05 2024-04-23T16:34:30.388 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.388 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.388 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.388 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.388 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208989 3.8 3.631ac508 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:34:30.388 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085652+0000", 2024-04-23T16:34:30.388 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 579.24042530099996, 2024-04-23T16:34:30.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70683001599999995, 2024-04-23T16:34:30.388 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.389 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.389 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.389 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.389 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.389 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.389 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208989 2024-04-23T16:34:30.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.389 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085652+0000", 2024-04-23T16:34:30.389 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.390 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085652+0000", 2024-04-23T16:34:30.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.390 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085653+0000", 2024-04-23T16:34:30.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9699999999999996e-07 2024-04-23T16:34:30.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.391 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085655+0000", 2024-04-23T16:34:30.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7810000000000001e-06 2024-04-23T16:34:30.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.391 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085661+0000", 2024-04-23T16:34:30.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1930000000000001e-06 2024-04-23T16:34:30.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.392 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.392 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085666+0000", 2024-04-23T16:34:30.392 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6579999999999999e-06 2024-04-23T16:34:30.392 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.392 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.392 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.392 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085695+0000", 2024-04-23T16:34:30.392 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8711999999999999e-05 2024-04-23T16:34:30.392 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.392 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.392 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.392 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085699+0000", 2024-04-23T16:34:30.392 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1500000000000001e-06 2024-04-23T16:34:30.392 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.393 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.393 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.393 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792474+0000", 2024-04-23T16:34:30.393 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70677516500000004 2024-04-23T16:34:30.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.393 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.393 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.393 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792482+0000", 2024-04-23T16:34:30.393 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7600000000000002e-06 2024-04-23T16:34:30.393 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.393 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.393 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.394 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.394 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208990 3.4 3.b71d3f14 (undecoded) ondisk+retry+write+known_if_redirected e241)", 2024-04-23T16:34:30.394 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:51.085676+0000", 2024-04-23T16:34:30.394 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 579.24040131499999, 2024-04-23T16:34:30.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70695150200000001, 2024-04-23T16:34:30.394 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.394 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.394 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.394 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.394 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.394 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.394 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208990 2024-04-23T16:34:30.394 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.394 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085676+0000", 2024-04-23T16:34:30.395 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085676+0000", 2024-04-23T16:34:30.395 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085677+0000", 2024-04-23T16:34:30.395 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6300000000000005e-07 2024-04-23T16:34:30.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.396 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.396 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.396 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085680+0000", 2024-04-23T16:34:30.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9799999999999998e-06 2024-04-23T16:34:30.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.396 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.396 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.396 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085686+0000", 2024-04-23T16:34:30.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9669999999999998e-06 2024-04-23T16:34:30.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.397 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.397 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.397 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085691+0000", 2024-04-23T16:34:30.397 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2000000000000002e-06 2024-04-23T16:34:30.397 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.397 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.397 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.397 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085730+0000", 2024-04-23T16:34:30.397 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9388999999999998e-05 2024-04-23T16:34:30.397 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.397 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.397 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.398 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.085737+0000", 2024-04-23T16:34:30.398 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7129999999999999e-06 2024-04-23T16:34:30.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.398 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.398 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792616+0000", 2024-04-23T16:34:30.398 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.70687957599999995 2024-04-23T16:34:30.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.398 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.398 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:51.792627+0000", 2024-04-23T16:34:30.398 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1014e-05 2024-04-23T16:34:30.398 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.398 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.399 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.399 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.399 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208925 3.8 3.a4efd988 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:34:30.399 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820364+0000", 2024-04-23T16:34:30.399 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 577.50571300800004, 2024-04-23T16:34:30.399 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55176175500000002, 2024-04-23T16:34:30.399 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.399 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.399 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.399 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.400 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.400 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.400 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208925 2024-04-23T16:34:30.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.400 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.400 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.400 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.400 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820364+0000", 2024-04-23T16:34:30.400 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.400 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.400 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.400 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820364+0000", 2024-04-23T16:34:30.400 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.401 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.401 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.401 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.401 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820368+0000", 2024-04-23T16:34:30.401 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8060000000000001e-06 2024-04-23T16:34:30.401 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.401 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.401 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.401 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820379+0000", 2024-04-23T16:34:30.401 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0634e-05 2024-04-23T16:34:30.401 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.401 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.401 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.401 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820394+0000", 2024-04-23T16:34:30.401 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5554e-05 2024-04-23T16:34:30.402 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.402 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.402 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.402 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820408+0000", 2024-04-23T16:34:30.402 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3508e-05 2024-04-23T16:34:30.402 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.402 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.402 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.402 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820486+0000", 2024-04-23T16:34:30.402 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7844999999999998e-05 2024-04-23T16:34:30.402 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.402 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.402 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.403 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820495+0000", 2024-04-23T16:34:30.403 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3610000000000007e-06 2024-04-23T16:34:30.403 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.403 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.403 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.403 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372103+0000", 2024-04-23T16:34:30.403 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55160793100000005 2024-04-23T16:34:30.403 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.403 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.403 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.403 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372126+0000", 2024-04-23T16:34:30.403 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3116e-05 2024-04-23T16:34:30.403 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.403 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.404 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.404 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.404 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208939 3.8 3.5b6d9248 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:34:30.404 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820453+0000", 2024-04-23T16:34:30.404 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 577.50562399499995, 2024-04-23T16:34:30.404 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55183821600000005, 2024-04-23T16:34:30.404 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.404 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.404 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.404 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.404 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.404 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.404 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208939 2024-04-23T16:34:30.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.405 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.405 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.405 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.405 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820453+0000", 2024-04-23T16:34:30.405 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.405 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.405 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.405 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.405 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820453+0000", 2024-04-23T16:34:30.405 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.405 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.405 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.405 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.405 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820454+0000", 2024-04-23T16:34:30.406 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0270000000000001e-06 2024-04-23T16:34:30.406 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.406 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.406 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.406 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820458+0000", 2024-04-23T16:34:30.406 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6430000000000001e-06 2024-04-23T16:34:30.406 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.406 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.406 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.406 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820465+0000", 2024-04-23T16:34:30.406 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2679999999999997e-06 2024-04-23T16:34:30.406 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.406 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.406 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.407 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820471+0000", 2024-04-23T16:34:30.407 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9529999999999996e-06 2024-04-23T16:34:30.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.407 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.407 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.407 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820615+0000", 2024-04-23T16:34:30.407 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014426700000000001 2024-04-23T16:34:30.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.407 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.408 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.408 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820622+0000", 2024-04-23T16:34:30.408 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2570000000000004e-06 2024-04-23T16:34:30.408 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.408 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.408 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.408 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372284+0000", 2024-04-23T16:34:30.408 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55166234199999997 2024-04-23T16:34:30.408 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.408 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.408 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.408 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372292+0000", 2024-04-23T16:34:30.408 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4590000000000001e-06 2024-04-23T16:34:30.408 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.409 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.409 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.409 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.409 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.409 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208948 3.8 3.9d5e98c8 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:34:30.409 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820480+0000", 2024-04-23T16:34:30.409 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 577.50559771799999, 2024-04-23T16:34:30.409 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55184634399999999, 2024-04-23T16:34:30.409 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.409 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.409 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.409 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.409 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.410 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.410 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208948 2024-04-23T16:34:30.410 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.410 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.410 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.410 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.410 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820480+0000", 2024-04-23T16:34:30.410 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.411 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.411 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.411 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820480+0000", 2024-04-23T16:34:30.411 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.411 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.411 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.411 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820480+0000", 2024-04-23T16:34:30.411 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4199999999999996e-07 2024-04-23T16:34:30.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.411 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.411 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.412 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820482+0000", 2024-04-23T16:34:30.412 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1919999999999999e-06 2024-04-23T16:34:30.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.412 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.412 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.412 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820487+0000", 2024-04-23T16:34:30.412 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7990000000000001e-06 2024-04-23T16:34:30.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.412 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.412 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.413 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820492+0000", 2024-04-23T16:34:30.413 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1869999999999996e-06 2024-04-23T16:34:30.413 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.413 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.413 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820654+0000", 2024-04-23T16:34:30.413 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016204500000000001 2024-04-23T16:34:30.413 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.413 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.413 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820659+0000", 2024-04-23T16:34:30.413 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8019999999999999e-06 2024-04-23T16:34:30.413 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.414 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.414 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372321+0000", 2024-04-23T16:34:30.414 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.551661548 2024-04-23T16:34:30.414 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.414 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.414 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.414 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372326+0000", 2024-04-23T16:34:30.414 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2290000000000002e-06 2024-04-23T16:34:30.414 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.414 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.414 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.414 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.414 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.415 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208949 3.8 3.32f0e298 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:34:30.415 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820498+0000", 2024-04-23T16:34:30.415 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 577.50557891400001, 2024-04-23T16:34:30.415 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55185574800000003, 2024-04-23T16:34:30.415 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.415 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.415 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.415 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.415 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.415 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.415 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208949 2024-04-23T16:34:30.415 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.416 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.416 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.416 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.416 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820498+0000", 2024-04-23T16:34:30.416 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.416 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.416 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.416 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.416 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820498+0000", 2024-04-23T16:34:30.416 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.416 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.416 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.416 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.416 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820499+0000", 2024-04-23T16:34:30.417 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6499999999999999e-07 2024-04-23T16:34:30.417 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.417 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.417 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.417 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820501+0000", 2024-04-23T16:34:30.417 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1090000000000001e-06 2024-04-23T16:34:30.417 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.417 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.417 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.417 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820506+0000", 2024-04-23T16:34:30.417 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.305e-06 2024-04-23T16:34:30.417 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.417 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.418 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.418 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820511+0000", 2024-04-23T16:34:30.418 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1869999999999996e-06 2024-04-23T16:34:30.418 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.418 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.418 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.418 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820688+0000", 2024-04-23T16:34:30.418 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000176353 2024-04-23T16:34:30.418 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.419 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.419 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820696+0000", 2024-04-23T16:34:30.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9410000000000007e-06 2024-04-23T16:34:30.419 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.419 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.419 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372350+0000", 2024-04-23T16:34:30.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55165407099999997 2024-04-23T16:34:30.419 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.419 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.419 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372354+0000", 2024-04-23T16:34:30.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3170000000000003e-06 2024-04-23T16:34:30.419 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.420 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.420 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.420 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.420 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.420 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208989 3.8 3.631ac508 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:34:30.420 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820517+0000", 2024-04-23T16:34:30.420 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 577.50556041100003, 2024-04-23T16:34:30.420 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55195477900000001, 2024-04-23T16:34:30.420 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.420 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.420 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.420 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.420 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.420 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.421 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208989 2024-04-23T16:34:30.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.421 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.421 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.421 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.421 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820517+0000", 2024-04-23T16:34:30.421 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.421 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.421 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.421 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820517+0000", 2024-04-23T16:34:30.421 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.421 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.422 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.422 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820517+0000", 2024-04-23T16:34:30.422 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2500000000000001e-07 2024-04-23T16:34:30.422 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.422 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.422 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.422 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820519+0000", 2024-04-23T16:34:30.422 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0760000000000001e-06 2024-04-23T16:34:30.422 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.422 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.423 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.423 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820524+0000", 2024-04-23T16:34:30.423 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.814e-06 2024-04-23T16:34:30.423 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.423 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.423 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.423 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820529+0000", 2024-04-23T16:34:30.423 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5809999999999998e-06 2024-04-23T16:34:30.423 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.423 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.423 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.423 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820738+0000", 2024-04-23T16:34:30.424 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00020955600000000001 2024-04-23T16:34:30.424 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.424 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.424 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.424 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820748+0000", 2024-04-23T16:34:30.424 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9760000000000007e-06 2024-04-23T16:34:30.424 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.424 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.424 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.424 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372466+0000", 2024-04-23T16:34:30.424 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.55171814399999997 2024-04-23T16:34:30.424 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.424 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.425 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.425 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:53.372472+0000", 2024-04-23T16:34:30.425 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2070000000000003e-06 2024-04-23T16:34:30.425 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.425 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.425 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.425 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.425 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.425 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208935 3.4 3.ef358d64 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:34:30.425 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820748+0000", 2024-04-23T16:34:30.425 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 577.50532969100004, 2024-04-23T16:34:30.425 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.12019948499999999, 2024-04-23T16:34:30.425 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.425 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.425 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.426 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.426 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.426 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.426 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208935 2024-04-23T16:34:30.426 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.426 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.426 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.426 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.426 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820748+0000", 2024-04-23T16:34:30.426 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.426 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.426 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.426 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.427 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820748+0000", 2024-04-23T16:34:30.427 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.427 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.427 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.427 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.427 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820749+0000", 2024-04-23T16:34:30.427 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2619999999999999e-06 2024-04-23T16:34:30.427 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.427 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.427 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.427 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820753+0000", 2024-04-23T16:34:30.427 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5839999999999996e-06 2024-04-23T16:34:30.427 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.427 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.428 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.428 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820762+0000", 2024-04-23T16:34:30.428 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0439999999999996e-06 2024-04-23T16:34:30.428 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.428 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.428 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.428 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820770+0000", 2024-04-23T16:34:30.428 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3280000000000006e-06 2024-04-23T16:34:30.428 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.428 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.429 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.429 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820865+0000", 2024-04-23T16:34:30.429 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5311999999999998e-05 2024-04-23T16:34:30.429 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.429 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.429 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.429 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820882+0000", 2024-04-23T16:34:30.429 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7266999999999999e-05 2024-04-23T16:34:30.429 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.429 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.429 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.429 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.940935+0000", 2024-04-23T16:34:30.429 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.120052908 2024-04-23T16:34:30.430 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.430 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.430 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.430 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.940947+0000", 2024-04-23T16:34:30.430 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1780000000000001e-05 2024-04-23T16:34:30.430 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.430 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.430 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.430 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.430 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.430 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208938 3.4 3.b83bb9a4 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:34:30.430 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820786+0000", 2024-04-23T16:34:30.430 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 577.50529179299997, 2024-04-23T16:34:30.431 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.120213972, 2024-04-23T16:34:30.431 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.431 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.431 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.431 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.431 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.431 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.431 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208938 2024-04-23T16:34:30.431 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.431 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.431 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.431 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.431 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820786+0000", 2024-04-23T16:34:30.431 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.431 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.432 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.432 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.432 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820786+0000", 2024-04-23T16:34:30.432 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.432 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.432 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.432 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.432 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820787+0000", 2024-04-23T16:34:30.432 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.094e-06 2024-04-23T16:34:30.432 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.432 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.432 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.432 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820791+0000", 2024-04-23T16:34:30.432 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4700000000000004e-06 2024-04-23T16:34:30.432 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.433 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.433 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.433 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820800+0000", 2024-04-23T16:34:30.433 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4059999999999994e-06 2024-04-23T16:34:30.433 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.433 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.433 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.433 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820809+0000", 2024-04-23T16:34:30.433 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1689999999999992e-06 2024-04-23T16:34:30.434 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.434 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.434 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.434 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820932+0000", 2024-04-23T16:34:30.434 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012330800000000001 2024-04-23T16:34:30.434 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.434 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.434 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.434 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820942+0000", 2024-04-23T16:34:30.434 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9480000000000003e-06 2024-04-23T16:34:30.434 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.434 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.434 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.435 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.940991+0000", 2024-04-23T16:34:30.435 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.120049106 2024-04-23T16:34:30.435 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.435 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.435 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.435 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.940999+0000", 2024-04-23T16:34:30.435 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4710000000000007e-06 2024-04-23T16:34:30.435 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.435 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.435 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.435 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.435 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.435 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208951 3.4 3.7f30f4f4 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:34:30.435 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820918+0000", 2024-04-23T16:34:30.436 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 577.50515939800005, 2024-04-23T16:34:30.436 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.120268738, 2024-04-23T16:34:30.436 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.436 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.436 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.436 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.436 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.436 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.436 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208951 2024-04-23T16:34:30.436 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.436 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.436 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.436 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.436 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820918+0000", 2024-04-23T16:34:30.437 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.437 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.437 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.437 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.437 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820918+0000", 2024-04-23T16:34:30.437 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.437 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.437 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.437 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.437 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820919+0000", 2024-04-23T16:34:30.437 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9500000000000001e-07 2024-04-23T16:34:30.437 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.437 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.437 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.438 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820925+0000", 2024-04-23T16:34:30.438 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.835e-06 2024-04-23T16:34:30.438 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.438 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.438 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.438 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820933+0000", 2024-04-23T16:34:30.438 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3739999999999998e-06 2024-04-23T16:34:30.438 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.438 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.438 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.438 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820941+0000", 2024-04-23T16:34:30.438 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8220000000000007e-06 2024-04-23T16:34:30.438 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.439 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.439 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.439 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.821053+0000", 2024-04-23T16:34:30.439 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000112617 2024-04-23T16:34:30.439 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.439 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.439 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.439 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.821059+0000", 2024-04-23T16:34:30.439 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1479999999999997e-06 2024-04-23T16:34:30.440 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.440 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.440 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.440 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.941182+0000", 2024-04-23T16:34:30.440 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.12012339399999999 2024-04-23T16:34:30.440 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.440 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.440 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.440 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.941187+0000", 2024-04-23T16:34:30.440 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6530000000000002e-06 2024-04-23T16:34:30.440 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.440 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.440 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.440 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.441 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.441 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.15150.0:208990 3.4 3.b71d3f14 (undecoded) ondisk+retry+write+known_if_redirected e243)", 2024-04-23T16:34:30.441 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:24:52.820957+0000", 2024-04-23T16:34:30.441 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 577.50512013599996, 2024-04-23T16:34:30.441 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.120262297, 2024-04-23T16:34:30.441 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:34:30.441 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:34:30.441 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:34:30.441 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:34:30.441 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.15150", 2024-04-23T16:34:30.441 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:34:30.441 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 208990 2024-04-23T16:34:30.441 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.441 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:34:30.441 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.442 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:34:30.442 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820957+0000", 2024-04-23T16:34:30.442 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.442 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.442 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.442 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:34:30.442 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820957+0000", 2024-04-23T16:34:30.442 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:34:30.442 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.442 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.442 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:34:30.442 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820959+0000", 2024-04-23T16:34:30.442 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3430000000000001e-06 2024-04-23T16:34:30.443 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.443 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.443 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:34:30.443 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820963+0000", 2024-04-23T16:34:30.443 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2400000000000001e-06 2024-04-23T16:34:30.443 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.443 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.443 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:34:30.443 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820975+0000", 2024-04-23T16:34:30.443 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1783999999999999e-05 2024-04-23T16:34:30.443 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.443 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.444 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:34:30.444 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.820985+0000", 2024-04-23T16:34:30.444 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0105000000000001e-05 2024-04-23T16:34:30.444 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.444 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.444 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.444 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.821085+0000", 2024-04-23T16:34:30.444 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100806 2024-04-23T16:34:30.445 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.445 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.445 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:34:30.445 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.821102+0000", 2024-04-23T16:34:30.445 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6283e-05 2024-04-23T16:34:30.445 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.445 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.446 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:34:30.446 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.941213+0000", 2024-04-23T16:34:30.446 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.12011137199999999 2024-04-23T16:34:30.446 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:34:30.446 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:34:30.446 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:34:30.446 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:24:52.941219+0000", 2024-04-23T16:34:30.446 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3640000000000004e-06 2024-04-23T16:34:30.446 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.447 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.447 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.447 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:34:30.447 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:34:30.447 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:34:32.012 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:34:32.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:32.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:34:32.040 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:34:32.040 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:34:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:31 smithi103 ceph-mon[39629]: pgmap v1157: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.0 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 867 op/s; 21419/228274 objects degraded (9.383%); 9.8 MiB/s, 156 objects/s recovering 2024-04-23T16:34:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:31 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:34:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:31 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:34:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:31 smithi103 ceph-mon[39629]: osdmap e410: 12 total, 11 up, 9 in 2024-04-23T16:34:32.118 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:34:32.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:31 smithi148 ceph-mon[44520]: pgmap v1157: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.0 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 867 op/s; 21419/228274 objects degraded (9.383%); 9.8 MiB/s, 156 objects/s recovering 2024-04-23T16:34:32.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:31 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:34:32.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:31 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:34:32.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:31 smithi148 ceph-mon[44520]: osdmap e410: 12 total, 11 up, 9 in 2024-04-23T16:34:32.214 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:32.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:31 smithi188 ceph-mon[44101]: pgmap v1157: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 7.0 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 867 op/s; 21419/228274 objects degraded (9.383%); 9.8 MiB/s, 156 objects/s recovering 2024-04-23T16:34:32.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:31 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:34:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:31 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:34:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:31 smithi188 ceph-mon[44101]: osdmap e410: 12 total, 11 up, 9 in 2024-04-23T16:34:32.276 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:32.283 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:34:32.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:32.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:34:32.392 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:34:32.425 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:32.533 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:34:32.535 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:34:34.041 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:34:34.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:33 smithi188 ceph-mon[44101]: from='client.25933 v1:172.21.15.103:0/726541890' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:34:34.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:33 smithi188 ceph-mon[44101]: pgmap v1159: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 6.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 506 op/s; 21419/227416 objects degraded (9.418%); 5.0 MiB/s, 79 objects/s recovering 2024-04-23T16:34:34.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:33 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:34:34.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:33 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:34:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:33 smithi148 ceph-mon[44520]: from='client.25933 v1:172.21.15.103:0/726541890' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:34:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:33 smithi148 ceph-mon[44520]: pgmap v1159: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 6.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 506 op/s; 21419/227416 objects degraded (9.418%); 5.0 MiB/s, 79 objects/s recovering 2024-04-23T16:34:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:33 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:34:34.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:33 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:34:34.147 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-23T16:34:34.174 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:33 smithi103 ceph-mon[39629]: from='client.25933 v1:172.21.15.103:0/726541890' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:34:34.174 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:33 smithi103 ceph-mon[39629]: pgmap v1159: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 6.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 506 op/s; 21419/227416 objects degraded (9.418%); 5.0 MiB/s, 79 objects/s recovering 2024-04-23T16:34:34.174 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:33 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:34:34.174 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:33 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:34:34.280 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:34:34.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:34.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:34:34.384 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:34.460 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:34.504 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:34.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:34.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:34.621 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:34:34.656 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:34.681 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:34:34.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:34.690 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:34:34.790 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:34:34.856 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:34.894 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:34:34.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:34.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:34:34.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:34 smithi188 ceph-mon[44101]: osdmap e411: 12 total, 11 up, 9 in 2024-04-23T16:34:34.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:34 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 21419/227416 objects degraded (9.418%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:35.005 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:34:35.057 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:35.108 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:34:35.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:35.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:34:35.111 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:34 smithi148 ceph-mon[44520]: osdmap e411: 12 total, 11 up, 9 in 2024-04-23T16:34:35.111 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:34 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 21419/227416 objects degraded (9.418%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:35.142 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:34 smithi103 ceph-mon[39629]: osdmap e411: 12 total, 11 up, 9 in 2024-04-23T16:34:35.143 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:34 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 21419/227416 objects degraded (9.418%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:35.179 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1158,"stamp":"2024-04-23T16:34:30.832237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7480476216,"num_objects":114137,"num_object_clones":0,"num_object_copies":228274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114137,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193057,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7534,"num_bytes_recovered":493883936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162360,"ondisk_log_size":162360,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":16490816,"kb_used_data":13265384,"kb_used_omap":14,"kb_used_meta":3225073,"kb_avail":827215552,"statfs":{"total":863955320832,"available":847068725248,"internally_reserved":0,"allocated":13583753216,"data_stored":13580600237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3302475759},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-234815488,"num_objects":-3583,"num_object_clones":0,"num_object_copies":-7166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-628,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3583,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":628,"num_bytes_recovered":41156608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.111789"},"pg_stats":[{"pgid":"5.f","version":"409'12149","reported_seq":12362,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:29.122907+0000","last_change":"2024-04-23T16:33:17.526419+0000","last_active":"2024-04-23T16:34:29.122907+0000","last_peered":"2024-04-23T16:34:29.122907+0000","last_clean":"2024-04-23T16:34:29.122907+0000","last_became_active":"2024-04-23T16:31:29.140923+0000","last_became_peered":"2024-04-23T16:31:29.140923+0000","last_unstale":"2024-04-23T16:34:29.122907+0000","last_undegraded":"2024-04-23T16:34:29.122907+0000","last_fullsized":"2024-04-23T16:34:29.122907+0000","mapping_epoch":359,"log_start":"370'2100","ondisk_log_start":"370'2100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9683","last_scrub_stamp":"2024-04-23T16:33:17.526283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.526283+0000","objects_scrubbed":9683,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:33.530461+0000","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472973312,"num_objects":7217,"num_object_clones":0,"num_object_copies":14434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12149,"num_write_kb":619712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"409'12060","reported_seq":12345,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:29.009196+0000","last_change":"2024-04-23T16:34:02.531358+0000","last_active":"2024-04-23T16:34:29.009196+0000","last_peered":"2024-04-23T16:34:29.009196+0000","last_clean":"2024-04-23T16:34:29.009196+0000","last_became_active":"2024-04-23T16:31:29.137890+0000","last_became_peered":"2024-04-23T16:31:29.137890+0000","last_unstale":"2024-04-23T16:34:29.009196+0000","last_undegraded":"2024-04-23T16:34:29.009196+0000","last_fullsized":"2024-04-23T16:34:29.009196+0000","mapping_epoch":359,"log_start":"370'2000","ondisk_log_start":"370'2000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'10975","last_scrub_stamp":"2024-04-23T16:34:02.531162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:02.531162+0000","objects_scrubbed":8314,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:09.558502+0000","scrub_duration":1066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472121344,"num_objects":7204,"num_object_clones":0,"num_object_copies":14408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12060,"num_write_kb":616448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"409'12159","reported_seq":17245,"reported_epoch":409,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:34:27.947153+0000","last_change":"2024-04-23T16:33:26.432192+0000","last_active":"2024-04-23T16:34:27.947153+0000","last_peered":"2024-04-23T16:34:27.947153+0000","last_clean":"2024-04-23T16:32:59.749530+0000","last_became_active":"2024-04-23T16:33:26.338644+0000","last_became_peered":"2024-04-23T16:33:26.338644+0000","last_unstale":"2024-04-23T16:34:27.947153+0000","last_undegraded":"2024-04-23T16:33:25.874847+0000","last_fullsized":"2024-04-23T16:34:27.947153+0000","mapping_epoch":394,"log_start":"370'2100","ondisk_log_start":"370'2100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8711","last_scrub_stamp":"2024-04-23T16:32:43.928914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.928914+0000","objects_scrubbed":8680,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477298688,"num_objects":7283,"num_object_clones":0,"num_object_copies":14566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12159,"num_write_kb":622144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2439,"num_bytes_recovered":159842304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":["8"],"object_location_counts":[{"shards":"4,8","objects":10},{"shards":"8","objects":7273}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.9","version":"409'12228","reported_seq":12426,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:30.083574+0000","last_change":"2024-04-23T16:32:37.256944+0000","last_active":"2024-04-23T16:34:30.083574+0000","last_peered":"2024-04-23T16:34:30.083574+0000","last_clean":"2024-04-23T16:34:30.083574+0000","last_became_active":"2024-04-23T16:31:29.141551+0000","last_became_peered":"2024-04-23T16:31:29.141551+0000","last_unstale":"2024-04-23T16:34:30.083574+0000","last_undegraded":"2024-04-23T16:34:30.083574+0000","last_fullsized":"2024-04-23T16:34:30.083574+0000","mapping_epoch":359,"log_start":"370'2200","ondisk_log_start":"370'2200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8156","last_scrub_stamp":"2024-04-23T16:32:37.256787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:37.256787+0000","objects_scrubbed":8082,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464781312,"num_objects":7092,"num_object_clones":0,"num_object_copies":14184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12228,"num_write_kb":618240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"408'11689","reported_seq":11898,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-23T16:34:26.719797+0000","last_change":"2024-04-23T16:33:25.105693+0000","last_active":"2024-04-23T16:34:26.719797+0000","last_peered":"2024-04-23T16:34:26.719797+0000","last_clean":"2024-04-23T16:34:26.719797+0000","last_became_active":"2024-04-23T16:31:29.139045+0000","last_became_peered":"2024-04-23T16:31:29.139045+0000","last_unstale":"2024-04-23T16:34:26.719797+0000","last_undegraded":"2024-04-23T16:34:26.719797+0000","last_fullsized":"2024-04-23T16:34:26.719797+0000","mapping_epoch":359,"log_start":"369'1600","ondisk_log_start":"369'1600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9367","last_scrub_stamp":"2024-04-23T16:33:25.105563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:25.105563+0000","objects_scrubbed":9367,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":751,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461701120,"num_objects":7045,"num_object_clones":0,"num_object_copies":14090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11689,"num_write_kb":599488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"409'11986","reported_seq":12199,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:28.904654+0000","last_change":"2024-04-23T16:33:22.103034+0000","last_active":"2024-04-23T16:34:28.904654+0000","last_peered":"2024-04-23T16:34:28.904654+0000","last_clean":"2024-04-23T16:34:28.904654+0000","last_became_active":"2024-04-23T16:31:29.310906+0000","last_became_peered":"2024-04-23T16:31:29.310906+0000","last_unstale":"2024-04-23T16:34:28.904654+0000","last_undegraded":"2024-04-23T16:34:28.904654+0000","last_fullsized":"2024-04-23T16:34:28.904654+0000","mapping_epoch":359,"log_start":"369'1900","ondisk_log_start":"369'1900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9559","last_scrub_stamp":"2024-04-23T16:33:22.102857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:22.102857+0000","objects_scrubbed":9559,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:45.157065+0000","scrub_duration":758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467533824,"num_objects":7134,"num_object_clones":0,"num_object_copies":14268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11986,"num_write_kb":611840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1071,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:27.817184+0000","last_change":"2024-04-23T16:33:31.424170+0000","last_active":"2024-04-23T16:34:27.817184+0000","last_peered":"2024-04-23T16:34:27.817184+0000","last_clean":"2024-04-23T16:34:27.817184+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:34:27.817184+0000","last_undegraded":"2024-04-23T16:34:27.817184+0000","last_fullsized":"2024-04-23T16:34:27.817184+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:31.424058+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:33:31.424058+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:35.317699+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"409'12207","reported_seq":12406,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:29.341941+0000","last_change":"2024-04-23T16:32:38.103756+0000","last_active":"2024-04-23T16:34:29.341941+0000","last_peered":"2024-04-23T16:34:29.341941+0000","last_clean":"2024-04-23T16:34:29.341941+0000","last_became_active":"2024-04-23T16:31:29.141640+0000","last_became_peered":"2024-04-23T16:31:29.141640+0000","last_unstale":"2024-04-23T16:34:29.341941+0000","last_undegraded":"2024-04-23T16:34:29.341941+0000","last_fullsized":"2024-04-23T16:34:29.341941+0000","mapping_epoch":359,"log_start":"370'2200","ondisk_log_start":"370'2200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8300","last_scrub_stamp":"2024-04-23T16:32:38.103547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:38.103547+0000","objects_scrubbed":8231,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475070464,"num_objects":7249,"num_object_clones":0,"num_object_copies":14498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12207,"num_write_kb":622592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1073,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:27.813820+0000","last_change":"2024-04-23T16:33:25.851379+0000","last_active":"2024-04-23T16:34:27.813820+0000","last_peered":"2024-04-23T16:34:27.813820+0000","last_clean":"2024-04-23T16:34:27.813820+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:34:27.813820+0000","last_undegraded":"2024-04-23T16:34:27.813820+0000","last_fullsized":"2024-04-23T16:34:27.813820+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.498820+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.498820+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"408'11882","reported_seq":12094,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-23T16:34:26.753169+0000","last_change":"2024-04-23T16:33:26.135212+0000","last_active":"2024-04-23T16:34:26.753169+0000","last_peered":"2024-04-23T16:34:26.753169+0000","last_clean":"2024-04-23T16:34:26.753169+0000","last_became_active":"2024-04-23T16:31:29.140251+0000","last_became_peered":"2024-04-23T16:31:29.140251+0000","last_unstale":"2024-04-23T16:34:26.753169+0000","last_undegraded":"2024-04-23T16:34:26.753169+0000","last_fullsized":"2024-04-23T16:34:26.753169+0000","mapping_epoch":359,"log_start":"369'1800","ondisk_log_start":"369'1800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9565","last_scrub_stamp":"2024-04-23T16:33:26.135088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:26.135088+0000","objects_scrubbed":9565,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:28.491332+0000","scrub_duration":759,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475070488,"num_objects":7250,"num_object_clones":0,"num_object_copies":14500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7250,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11882,"num_write_kb":612161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1064,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:27.824936+0000","last_change":"2024-04-23T16:32:29.517947+0000","last_active":"2024-04-23T16:34:27.824936+0000","last_peered":"2024-04-23T16:34:27.824936+0000","last_clean":"2024-04-23T16:34:27.824936+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:34:27.824936+0000","last_undegraded":"2024-04-23T16:34:27.824936+0000","last_fullsized":"2024-04-23T16:34:27.824936+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:32:29.517800+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:32:29.517800+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"409'12237","reported_seq":17539,"reported_epoch":409,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:34:29.258542+0000","last_change":"2024-04-23T16:33:26.507399+0000","last_active":"2024-04-23T16:34:29.258542+0000","last_peered":"2024-04-23T16:34:29.258542+0000","last_clean":"2024-04-23T16:32:59.715575+0000","last_became_active":"2024-04-23T16:33:26.458611+0000","last_became_peered":"2024-04-23T16:33:26.458611+0000","last_unstale":"2024-04-23T16:34:29.258542+0000","last_undegraded":"2024-04-23T16:33:25.895472+0000","last_fullsized":"2024-04-23T16:34:29.258542+0000","mapping_epoch":394,"log_start":"370'2200","ondisk_log_start":"370'2200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8607","last_scrub_stamp":"2024-04-23T16:32:43.130806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.130806+0000","objects_scrubbed":8577,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468123648,"num_objects":7143,"num_object_clones":0,"num_object_copies":14286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12237,"num_write_kb":620160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2547,"num_bytes_recovered":166920192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":7140},{"shards":"5,7","objects":3}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1070,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:27.812028+0000","last_change":"2024-04-23T16:33:33.275883+0000","last_active":"2024-04-23T16:34:27.812028+0000","last_peered":"2024-04-23T16:34:27.812028+0000","last_clean":"2024-04-23T16:34:27.812028+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:34:27.812028+0000","last_undegraded":"2024-04-23T16:34:27.812028+0000","last_fullsized":"2024-04-23T16:34:27.812028+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:33.275799+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:33:33.275799+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:42.780661+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"409'11957","reported_seq":12168,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:29.218081+0000","last_change":"2024-04-23T16:33:18.576077+0000","last_active":"2024-04-23T16:34:29.218081+0000","last_peered":"2024-04-23T16:34:29.218081+0000","last_clean":"2024-04-23T16:34:29.218081+0000","last_became_active":"2024-04-23T16:31:29.139709+0000","last_became_peered":"2024-04-23T16:31:29.139709+0000","last_unstale":"2024-04-23T16:34:29.218081+0000","last_undegraded":"2024-04-23T16:34:29.218081+0000","last_fullsized":"2024-04-23T16:34:29.218081+0000","mapping_epoch":359,"log_start":"369'1900","ondisk_log_start":"369'1900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9472","last_scrub_stamp":"2024-04-23T16:33:18.575945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.575945+0000","objects_scrubbed":9472,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:42.303216+0000","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457900032,"num_objects":6987,"num_object_clones":0,"num_object_copies":13974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11957,"num_write_kb":606208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1071,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:27.814031+0000","last_change":"2024-04-23T16:34:02.399102+0000","last_active":"2024-04-23T16:34:27.814031+0000","last_peered":"2024-04-23T16:34:27.814031+0000","last_clean":"2024-04-23T16:34:27.814031+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:34:27.814031+0000","last_undegraded":"2024-04-23T16:34:27.814031+0000","last_fullsized":"2024-04-23T16:34:27.814031+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:02.398975+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:02.398975+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:23.945801+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"409'12122","reported_seq":12337,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:29.065056+0000","last_change":"2024-04-23T16:33:19.938285+0000","last_active":"2024-04-23T16:34:29.065056+0000","last_peered":"2024-04-23T16:34:29.065056+0000","last_clean":"2024-04-23T16:34:29.065056+0000","last_became_active":"2024-04-23T16:31:29.139606+0000","last_became_peered":"2024-04-23T16:31:29.139606+0000","last_unstale":"2024-04-23T16:34:29.065056+0000","last_undegraded":"2024-04-23T16:34:29.065056+0000","last_fullsized":"2024-04-23T16:34:29.065056+0000","mapping_epoch":359,"log_start":"370'2100","ondisk_log_start":"370'2100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9720","last_scrub_stamp":"2024-04-23T16:33:19.938119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:19.938119+0000","objects_scrubbed":9720,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:34.186612+0000","scrub_duration":794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479592448,"num_objects":7318,"num_object_clones":0,"num_object_copies":14636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":622080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1070,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:27.817189+0000","last_change":"2024-04-23T16:33:28.454288+0000","last_active":"2024-04-23T16:34:27.817189+0000","last_peered":"2024-04-23T16:34:27.817189+0000","last_clean":"2024-04-23T16:34:27.817189+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:34:27.817189+0000","last_undegraded":"2024-04-23T16:34:27.817189+0000","last_fullsized":"2024-04-23T16:34:27.817189+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:28.454200+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:33:28.454200+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:51.316141+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.2","version":"409'12010","reported_seq":12300,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:29.536664+0000","last_change":"2024-04-23T16:33:57.879648+0000","last_active":"2024-04-23T16:34:29.536664+0000","last_peered":"2024-04-23T16:34:29.536664+0000","last_clean":"2024-04-23T16:34:29.536664+0000","last_became_active":"2024-04-23T16:31:29.138430+0000","last_became_peered":"2024-04-23T16:31:29.138430+0000","last_unstale":"2024-04-23T16:34:29.536664+0000","last_undegraded":"2024-04-23T16:34:29.536664+0000","last_fullsized":"2024-04-23T16:34:29.536664+0000","mapping_epoch":359,"log_start":"370'2000","ondisk_log_start":"370'2000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'10726","last_scrub_stamp":"2024-04-23T16:33:57.879516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:57.879516+0000","objects_scrubbed":8303,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:02.097116+0000","scrub_duration":1621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457834496,"num_objects":6986,"num_object_clones":0,"num_object_copies":13972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12010,"num_write_kb":607872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1071,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:27.817326+0000","last_change":"2024-04-23T16:33:18.422277+0000","last_active":"2024-04-23T16:34:27.817326+0000","last_peered":"2024-04-23T16:34:27.817326+0000","last_clean":"2024-04-23T16:34:27.817326+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:34:27.817326+0000","last_undegraded":"2024-04-23T16:34:27.817326+0000","last_fullsized":"2024-04-23T16:34:27.817326+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:18.422171+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.422171+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"409'14308","reported_seq":14650,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:29.071040+0000","last_change":"2024-04-23T16:33:17.974701+0000","last_active":"2024-04-23T16:34:29.071040+0000","last_peered":"2024-04-23T16:34:29.071040+0000","last_clean":"2024-04-23T16:34:29.071040+0000","last_became_active":"2024-04-23T16:31:29.139398+0000","last_became_peered":"2024-04-23T16:31:29.139398+0000","last_unstale":"2024-04-23T16:34:29.071040+0000","last_undegraded":"2024-04-23T16:34:29.071040+0000","last_fullsized":"2024-04-23T16:34:29.071040+0000","mapping_epoch":359,"log_start":"372'4300","ondisk_log_start":"372'4300","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'11838","last_scrub_stamp":"2024-04-23T16:33:17.974549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.974549+0000","objects_scrubbed":9564,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":848,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464912384,"num_objects":7094,"num_object_clones":0,"num_object_copies":14188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12034,"num_write_kb":612096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1071,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:27.813918+0000","last_change":"2024-04-23T16:34:04.444103+0000","last_active":"2024-04-23T16:34:27.813918+0000","last_peered":"2024-04-23T16:34:27.813918+0000","last_clean":"2024-04-23T16:34:27.813918+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:34:27.813918+0000","last_undegraded":"2024-04-23T16:34:27.813918+0000","last_fullsized":"2024-04-23T16:34:27.813918+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:04.443975+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:04.443975+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:05.109970+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"409'11981","reported_seq":12182,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:29.278612+0000","last_change":"2024-04-23T16:32:43.947374+0000","last_active":"2024-04-23T16:34:29.278612+0000","last_peered":"2024-04-23T16:34:29.278612+0000","last_clean":"2024-04-23T16:34:29.278612+0000","last_became_active":"2024-04-23T16:31:29.138968+0000","last_became_peered":"2024-04-23T16:31:29.138968+0000","last_unstale":"2024-04-23T16:34:29.278612+0000","last_undegraded":"2024-04-23T16:34:29.278612+0000","last_fullsized":"2024-04-23T16:34:29.278612+0000","mapping_epoch":359,"log_start":"369'1900","ondisk_log_start":"369'1900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8482","last_scrub_stamp":"2024-04-23T16:32:43.947177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.947177+0000","objects_scrubbed":8428,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453050368,"num_objects":6913,"num_object_clones":0,"num_object_copies":13826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11981,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":960,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-04-23T16:34:24.849109+0000","last_change":"2024-04-23T16:33:19.443898+0000","last_active":"2024-04-23T16:34:24.849109+0000","last_peered":"2024-04-23T16:34:24.849109+0000","last_clean":"2024-04-23T16:34:24.849109+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:34:24.849109+0000","last_undegraded":"2024-04-23T16:34:24.849109+0000","last_fullsized":"2024-04-23T16:34:24.849109+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:19.443792+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:33:19.443792+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:29.564987+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"409'12257","reported_seq":12455,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:29.351605+0000","last_change":"2024-04-23T16:32:39.103230+0000","last_active":"2024-04-23T16:34:29.351605+0000","last_peered":"2024-04-23T16:34:29.351605+0000","last_clean":"2024-04-23T16:34:29.351605+0000","last_became_active":"2024-04-23T16:31:29.310246+0000","last_became_peered":"2024-04-23T16:31:29.310246+0000","last_unstale":"2024-04-23T16:34:29.351605+0000","last_undegraded":"2024-04-23T16:34:29.351605+0000","last_fullsized":"2024-04-23T16:34:29.351605+0000","mapping_epoch":359,"log_start":"370'2200","ondisk_log_start":"370'2200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8325","last_scrub_stamp":"2024-04-23T16:32:39.103059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:39.103059+0000","objects_scrubbed":8272,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472186880,"num_objects":7205,"num_object_clones":0,"num_object_copies":14410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12257,"num_write_kb":622784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":884,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:27.813771+0000","last_change":"2024-04-23T16:33:26.153079+0000","last_active":"2024-04-23T16:34:27.813771+0000","last_peered":"2024-04-23T16:34:27.813771+0000","last_clean":"2024-04-23T16:34:27.813771+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:34:27.813771+0000","last_undegraded":"2024-04-23T16:34:27.813771+0000","last_fullsized":"2024-04-23T16:34:27.813771+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:26.152962+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:33:26.152962+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:53.292007+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"409'12099","reported_seq":17396,"reported_epoch":409,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:34:29.247703+0000","last_change":"2024-04-23T16:33:26.132487+0000","last_active":"2024-04-23T16:34:29.247703+0000","last_peered":"2024-04-23T16:34:29.247703+0000","last_clean":"2024-04-23T16:32:59.695133+0000","last_became_active":"2024-04-23T16:33:26.078062+0000","last_became_peered":"2024-04-23T16:33:26.078062+0000","last_unstale":"2024-04-23T16:34:29.247703+0000","last_undegraded":"2024-04-23T16:33:25.892009+0000","last_fullsized":"2024-04-23T16:34:29.247703+0000","mapping_epoch":394,"log_start":"369'2000","ondisk_log_start":"369'2000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'7968","last_scrub_stamp":"2024-04-23T16:32:36.102122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:36.102122+0000","objects_scrubbed":7926,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459735040,"num_objects":7015,"num_object_clones":0,"num_object_copies":14030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12099,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2547,"num_bytes_recovered":166920192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":7006},{"shards":"5,8","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1069,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-04-23T16:34:27.824801+0000","last_change":"2024-04-23T16:33:17.354409+0000","last_active":"2024-04-23T16:34:27.824801+0000","last_peered":"2024-04-23T16:34:27.824801+0000","last_clean":"2024-04-23T16:34:27.824801+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:34:27.824801+0000","last_undegraded":"2024-04-23T16:34:27.824801+0000","last_fullsized":"2024-04-23T16:34:27.824801+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:17.354263+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.354263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:39.845454+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":7479885848,"num_objects":114135,"num_object_clones":0,"num_object_copies":228270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114135,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193057,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7533,"num_bytes_recovered":493682688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15431901184,"data_stored":15431893040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160831,"ondisk_log_size":160831,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1427824,"kb_used_data":919072,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92317328,"statfs":{"total":95995035648,"available":94532943872,"internally_reserved":0,"allocated":941129728,"data_stored":940781689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":520944068},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351212,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612850,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1694708,"kb_used_data":1370600,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92050444,"statfs":{"total":95995035648,"available":94259654656,"internally_reserved":0,"allocated":1403494400,"data_stored":1403144429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":331872704},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841782,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1129188,"kb_used_data":931420,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":92615964,"statfs":{"total":95995035648,"available":94838747136,"internally_reserved":0,"allocated":953774080,"data_stored":953430113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168827,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":808688,"kb_used_data":456808,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":92936464,"statfs":{"total":95995035648,"available":95166939136,"internally_reserved":0,"allocated":467771392,"data_stored":467421421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":360315338},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463169,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1627312,"kb_used_data":1357352,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92117840,"statfs":{"total":95995035648,"available":94328668160,"internally_reserved":0,"allocated":1389928448,"data_stored":1389578477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":276429254},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137309,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738654,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3574452,"kb_used_data":3200360,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":90170700,"statfs":{"total":95995035648,"available":92334796800,"internally_reserved":0,"allocated":3277168640,"data_stored":3276818669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117263,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":809080,"kb_used_data":457200,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":92936072,"statfs":{"total":95995035648,"available":95166537728,"internally_reserved":0,"allocated":468172800,"data_stored":467819277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":360315335},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921033,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2753904,"kb_used_data":2299048,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":90991248,"statfs":{"total":95995035648,"available":93175037952,"internally_reserved":0,"allocated":2354225152,"data_stored":2353875181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757508,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2665660,"kb_used_data":2273524,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":91079492,"statfs":{"total":95995035648,"available":93265399808,"internally_reserved":0,"allocated":2328088576,"data_stored":2327730981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3262316544,"data_stored":3262316544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1886846976,"data_stored":1886846976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":466944000,"data_stored":466944000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2352545792,"data_stored":2352545792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2326466560,"data_stored":2326462488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1388314624,"data_stored":1388314624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":465108992,"data_stored":465108992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":948240384,"data_stored":948240384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1398407168,"data_stored":1398407168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":936710144,"data_stored":936706072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:34:35.192 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:35.211 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:34:35.242 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:35.261 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:34:35.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:35.264 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:34:35.365 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:35.431 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:35.470 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:35.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:35.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:35.574 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:34:35.602 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:35.624 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:34:35.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:35.631 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:34:35.732 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:34:35.770 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:35.790 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:34:35.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:35.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:34:35.888 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:34:35.896 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:34:35.927 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:36.078 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:34:36.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:35 smithi103 ceph-mon[39629]: pgmap v1161: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 6.8 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 908 op/s; 20798/221658 objects degraded (9.383%); 11 MiB/s, 180 objects/s recovering 2024-04-23T16:34:36.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:35 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1001550155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:34:36.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:36.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:34:36.182 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:34:36.210 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:36.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:35 smithi188 ceph-mon[44101]: pgmap v1161: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 6.8 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 908 op/s; 20798/221658 objects degraded (9.383%); 11 MiB/s, 180 objects/s recovering 2024-04-23T16:34:36.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:35 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1001550155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:34:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:35 smithi148 ceph-mon[44520]: pgmap v1161: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 6.8 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 908 op/s; 20798/221658 objects degraded (9.383%); 11 MiB/s, 180 objects/s recovering 2024-04-23T16:34:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:35 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1001550155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:34:36.804 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:37.031 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:36 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:34:37.032 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1001550155' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:34:37.032 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:36 smithi103 ceph-mon[39629]: osdmap e412: 12 total, 11 up, 9 in 2024-04-23T16:34:37.061 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:34:37.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:37.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:34:37.168 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:37.222 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:36 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:34:37.222 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1001550155' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:34:37.222 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:36 smithi148 ceph-mon[44520]: osdmap e412: 12 total, 11 up, 9 in 2024-04-23T16:34:37.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:36 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:34:37.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1001550155' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:34:37.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:36 smithi188 ceph-mon[44101]: osdmap e412: 12 total, 11 up, 9 in 2024-04-23T16:34:37.226 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:37.261 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:37.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:37.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:37.367 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:34:37.394 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:37.743 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:34:37.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:37.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:34:37.848 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:34:37.877 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:38.022 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:37 smithi103 ceph-mon[39629]: pgmap v1163: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 6.8 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 587 op/s; 20798/221226 objects degraded (9.401%); 6.5 MiB/s, 103 objects/s recovering 2024-04-23T16:34:38.025 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:34:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:38.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:34:38.129 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:34:38.181 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:34:38.189 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:37 smithi148 ceph-mon[44520]: pgmap v1163: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 6.8 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 587 op/s; 20798/221226 objects degraded (9.401%); 6.5 MiB/s, 103 objects/s recovering 2024-04-23T16:34:38.194 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:38.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:37 smithi188 ceph-mon[44101]: pgmap v1163: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 6.8 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 587 op/s; 20798/221226 objects degraded (9.401%); 6.5 MiB/s, 103 objects/s recovering 2024-04-23T16:34:38.283 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:34:38.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:38.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:34:38.394 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:34:38.429 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:38.453 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:34:38.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:38.458 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:34:38.559 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:34:38.588 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:38.613 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:34:38.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:38.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:34:38.717 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:34:38.750 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:38.807 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:34:38.874 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:34:38.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:38.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:34:38.983 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:34:39.008 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:38 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T16:34:39.009 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:38 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T16:34:39.009 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:38 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3590877557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:34:39.009 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:39.009 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:39.009 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:39.009 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:39.010 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:38 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T16:34:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:38 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T16:34:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:38 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3590877557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:34:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:39.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:38 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T16:34:39.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:38 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T16:34:39.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:38 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3590877557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:34:39.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:39.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:39.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:39.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:39.538 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:34:39.538 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:34:39.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:34:39.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:34:39.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:34:39.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:34:39.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:34:39.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:34:39.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:34:39.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:34:39.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:34:39.539 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:34:39.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:34:39.540 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:34:39.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:34:39.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:34:39.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:34:39.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:34:39.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:34:39.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:34:39.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:34:39.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:34:39.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:34:40.160 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:40.160 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:40.161 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:40.161 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:40.161 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:40.161 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:40.161 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:39 smithi103 ceph-mon[39629]: pgmap v1164: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 6.7 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 526 op/s; 20582/219970 objects degraded (9.357%); 6.6 MiB/s, 106 objects/s recovering 2024-04-23T16:34:40.161 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:39 smithi103 ceph-mon[39629]: 5.8 scrub starts 2024-04-23T16:34:40.161 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:39 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 20582/219970 objects degraded (9.357%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:40.161 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:39 smithi103 ceph-mon[39629]: 5.3 scrub starts 2024-04-23T16:34:40.166 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:34:40.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:40.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:34:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:39 smithi188 ceph-mon[44101]: pgmap v1164: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 6.7 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 526 op/s; 20582/219970 objects degraded (9.357%); 6.6 MiB/s, 106 objects/s recovering 2024-04-23T16:34:40.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:39 smithi188 ceph-mon[44101]: 5.8 scrub starts 2024-04-23T16:34:40.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:39 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 20582/219970 objects degraded (9.357%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:40.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:39 smithi188 ceph-mon[44101]: 5.3 scrub starts 2024-04-23T16:34:40.275 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:34:40.310 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:40.330 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:34:40.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:40.334 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:34:40.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:40.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:40.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:40.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:40.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:40.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:40.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:39 smithi148 ceph-mon[44520]: pgmap v1164: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 6.7 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 526 op/s; 20582/219970 objects degraded (9.357%); 6.6 MiB/s, 106 objects/s recovering 2024-04-23T16:34:40.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:39 smithi148 ceph-mon[44520]: 5.8 scrub starts 2024-04-23T16:34:40.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:39 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 20582/219970 objects degraded (9.357%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:40.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:39 smithi148 ceph-mon[44520]: 5.3 scrub starts 2024-04-23T16:34:40.435 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:34:40.449 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:34:40.449 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:34:40.467 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:40.486 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:34:40.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:40.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:34:40.590 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:34:40.621 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:40.648 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:34:40.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:40.655 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:34:40.756 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:34:40.799 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:40.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:40.865 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:34:40.865 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:34:40.901 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:34:40.902 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:34:40.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:40 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:34:40.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:40 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:34:40.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:40 smithi188 ceph-mon[44101]: 5.8 scrub ok 2024-04-23T16:34:40.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:40 smithi188 ceph-mon[44101]: 5.d scrub starts 2024-04-23T16:34:40.922 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:40 smithi188 ceph-mon[44101]: osdmap e413: 12 total, 11 up, 9 in 2024-04-23T16:34:40.922 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:40 smithi188 ceph-mon[44101]: 5.3 scrub ok 2024-04-23T16:34:40.966 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:34:41.003 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:41.028 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:34:41.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:41.034 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:34:41.135 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:41.220 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:40 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:34:41.220 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:40 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:34:41.220 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:40 smithi148 ceph-mon[44520]: 5.8 scrub ok 2024-04-23T16:34:41.221 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:40 smithi148 ceph-mon[44520]: 5.d scrub starts 2024-04-23T16:34:41.221 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:40 smithi148 ceph-mon[44520]: osdmap e413: 12 total, 11 up, 9 in 2024-04-23T16:34:41.221 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:40 smithi148 ceph-mon[44520]: 5.3 scrub ok 2024-04-23T16:34:41.222 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:41.239 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:41.243 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:41.251 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:41.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:41.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:41.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:40 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:34:41.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:40 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:34:41.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:40 smithi103 ceph-mon[39629]: 5.8 scrub ok 2024-04-23T16:34:41.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:40 smithi103 ceph-mon[39629]: 5.d scrub starts 2024-04-23T16:34:41.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:40 smithi103 ceph-mon[39629]: osdmap e413: 12 total, 11 up, 9 in 2024-04-23T16:34:41.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:40 smithi103 ceph-mon[39629]: 5.3 scrub ok 2024-04-23T16:34:41.357 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:34:41.386 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:41.410 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:34:41.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:41.416 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:34:41.517 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:34:41.553 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:41.756 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:34:41.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:41.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:34:41.861 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:34:41.891 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:41.918 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:34:41.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:41.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:34:42.022 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:34:42.089 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:42.164 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:34:42.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:42.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:34:42.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:41 smithi188 ceph-mon[44101]: pgmap v1166: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 6.5 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 910 op/s; 20123/214486 objects degraded (9.382%); 9.9 MiB/s, 158 objects/s recovering 2024-04-23T16:34:42.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:41 smithi188 ceph-mon[44101]: 5.d scrub ok 2024-04-23T16:34:42.270 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:34:42.300 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:42.327 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:34:42.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:42.330 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:34:42.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:41 smithi103 ceph-mon[39629]: pgmap v1166: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 6.5 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 910 op/s; 20123/214486 objects degraded (9.382%); 9.9 MiB/s, 158 objects/s recovering 2024-04-23T16:34:42.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:41 smithi103 ceph-mon[39629]: 5.d scrub ok 2024-04-23T16:34:42.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:41 smithi148 ceph-mon[44520]: pgmap v1166: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 6.5 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 910 op/s; 20123/214486 objects degraded (9.382%); 9.9 MiB/s, 158 objects/s recovering 2024-04-23T16:34:42.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:41 smithi148 ceph-mon[44520]: 5.d scrub ok 2024-04-23T16:34:42.431 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:34:42.460 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:42.483 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:34:42.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:42.489 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:34:42.590 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:42.674 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:42.731 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:42.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:42.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:42.843 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:42.902 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:34:42.911 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-23T16:34:42.957 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:42.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:42.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:43.068 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:43.138 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:43.184 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:43.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:43.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:43.296 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:34:43.328 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:43.354 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:34:43.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:43.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:34:43.459 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:34:43.488 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:43.514 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:34:43.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:43.517 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:34:43.618 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:34:43.652 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:43.675 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:34:43.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:43.684 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:34:43.785 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:34:43.822 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:43.956 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:34:43.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:43.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:34:44.061 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:34:44.087 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:43 smithi103 ceph-mon[39629]: pgmap v1167: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 6.5 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 502 op/s; 20123/213616 objects degraded (9.420%); 5.3 MiB/s, 84 objects/s recovering 2024-04-23T16:34:44.087 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:43 smithi103 ceph-mon[39629]: 5.5 scrub starts 2024-04-23T16:34:44.090 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:44.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:43 smithi188 ceph-mon[44101]: pgmap v1167: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 6.5 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 502 op/s; 20123/213616 objects degraded (9.420%); 5.3 MiB/s, 84 objects/s recovering 2024-04-23T16:34:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:43 smithi188 ceph-mon[44101]: 5.5 scrub starts 2024-04-23T16:34:44.291 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:34:44.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:44.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:34:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:43 smithi148 ceph-mon[44520]: pgmap v1167: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 6.5 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 502 op/s; 20123/213616 objects degraded (9.420%); 5.3 MiB/s, 84 objects/s recovering 2024-04-23T16:34:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:43 smithi148 ceph-mon[44520]: 5.5 scrub starts 2024-04-23T16:34:44.397 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:34:44.461 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:44.512 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:34:44.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:44.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:34:44.621 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:34:44.657 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:44.695 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:34:44.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:44.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:34:44.800 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:34:44.827 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:44.852 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:34:44.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:44.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:34:44.904 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:44.932 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:44 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 20123/213616 objects degraded (9.420%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:44.932 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:44 smithi103 ceph-mon[39629]: 5.5 scrub ok 2024-04-23T16:34:44.959 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:45.058 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:45.097 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:45.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:45.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:45.208 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:34:45.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:44 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 20123/213616 objects degraded (9.420%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:45.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:44 smithi188 ceph-mon[44101]: 5.5 scrub ok 2024-04-23T16:34:45.271 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:44 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 20123/213616 objects degraded (9.420%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:45.271 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:44 smithi148 ceph-mon[44520]: 5.5 scrub ok 2024-04-23T16:34:45.272 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:45.316 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:34:45.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:45.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:34:45.423 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:34:45.453 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:45.477 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:34:45.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:45.480 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:34:45.581 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:34:45.619 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:45.640 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:34:45.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:45.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:34:45.668 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:45.744 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:34:45.774 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:45.907 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:34:45.913 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:34:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:45 smithi103 ceph-mon[39629]: 1.a scrub starts 2024-04-23T16:34:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:45 smithi103 ceph-mon[39629]: pgmap v1168: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering+degraded, 23 active+clean; 6.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 749 op/s; 19486/207982 objects degraded (9.369%); 9.3 MiB/s, 148 objects/s recovering 2024-04-23T16:34:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:45 smithi103 ceph-mon[39629]: 1.a scrub ok 2024-04-23T16:34:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:45 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:34:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:45 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:34:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:34:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:45 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3028371438' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:34:46.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:45 smithi103 ceph-mon[39629]: osdmap e414: 12 total, 11 up, 9 in 2024-04-23T16:34:46.190 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:34:46.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:45 smithi188 ceph-mon[44101]: 1.a scrub starts 2024-04-23T16:34:46.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:45 smithi188 ceph-mon[44101]: pgmap v1168: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering+degraded, 23 active+clean; 6.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 749 op/s; 19486/207982 objects degraded (9.369%); 9.3 MiB/s, 148 objects/s recovering 2024-04-23T16:34:46.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:45 smithi188 ceph-mon[44101]: 1.a scrub ok 2024-04-23T16:34:46.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:45 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:34:46.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:45 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:34:46.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:34:46.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:45 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3028371438' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:34:46.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:45 smithi188 ceph-mon[44101]: osdmap e414: 12 total, 11 up, 9 in 2024-04-23T16:34:46.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:45 smithi148 ceph-mon[44520]: 1.a scrub starts 2024-04-23T16:34:46.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:45 smithi148 ceph-mon[44520]: pgmap v1168: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering+degraded, 23 active+clean; 6.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 749 op/s; 19486/207982 objects degraded (9.369%); 9.3 MiB/s, 148 objects/s recovering 2024-04-23T16:34:46.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:45 smithi148 ceph-mon[44520]: 1.a scrub ok 2024-04-23T16:34:46.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:45 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:34:46.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:45 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:34:46.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:34:46.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:45 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3028371438' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:34:46.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:45 smithi148 ceph-mon[44520]: osdmap e414: 12 total, 11 up, 9 in 2024-04-23T16:34:47.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:46 smithi103 ceph-mon[39629]: from='client.16026 v1:172.21.15.103:0/2093789954' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:34:47.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:46 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:34:47.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:46 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3028371438' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:34:47.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:46 smithi103 ceph-mon[39629]: osdmap e415: 12 total, 11 up, 9 in 2024-04-23T16:34:47.213 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1169,"stamp":"2024-04-23T16:34:45.177826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6800736824,"num_objects":103765,"num_object_clones":0,"num_object_copies":207530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103765,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":203429,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9467,"num_bytes_recovered":620565024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162332,"ondisk_log_size":162332,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":15316992,"kb_used_data":12090288,"kb_used_omap":14,"kb_used_meta":3226481,"kb_avail":828389376,"statfs":{"total":863955320832,"available":848270721024,"internally_reserved":0,"allocated":12380454912,"data_stored":12377073331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3303917551},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-448790528,"num_objects":-6848,"num_object_clones":0,"num_object_copies":-13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6848,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1312,"num_bytes_recovered":85983232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.454822"},"pg_stats":[{"pgid":"5.f","version":"413'12836","reported_seq":13057,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:44.349541+0000","last_change":"2024-04-23T16:33:17.526419+0000","last_active":"2024-04-23T16:34:44.349541+0000","last_peered":"2024-04-23T16:34:44.349541+0000","last_clean":"2024-04-23T16:34:44.349541+0000","last_became_active":"2024-04-23T16:31:29.140923+0000","last_became_peered":"2024-04-23T16:31:29.140923+0000","last_unstale":"2024-04-23T16:34:44.349541+0000","last_undegraded":"2024-04-23T16:34:44.349541+0000","last_fullsized":"2024-04-23T16:34:44.349541+0000","mapping_epoch":359,"log_start":"370'2800","ondisk_log_start":"370'2800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9683","last_scrub_stamp":"2024-04-23T16:33:17.526283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.526283+0000","objects_scrubbed":9683,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427950080,"num_objects":6530,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12836,"num_write_kb":619712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"413'12697","reported_seq":12990,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:44.019494+0000","last_change":"2024-04-23T16:34:02.531358+0000","last_active":"2024-04-23T16:34:44.019494+0000","last_peered":"2024-04-23T16:34:44.019494+0000","last_clean":"2024-04-23T16:34:44.019494+0000","last_became_active":"2024-04-23T16:31:29.137890+0000","last_became_peered":"2024-04-23T16:31:29.137890+0000","last_unstale":"2024-04-23T16:34:44.019494+0000","last_undegraded":"2024-04-23T16:34:44.019494+0000","last_fullsized":"2024-04-23T16:34:44.019494+0000","mapping_epoch":359,"log_start":"370'2600","ondisk_log_start":"370'2600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'10975","last_scrub_stamp":"2024-04-23T16:34:02.531162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:02.531162+0000","objects_scrubbed":8314,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:09.558502+0000","scrub_duration":1066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430374912,"num_objects":6567,"num_object_clones":0,"num_object_copies":13134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12697,"num_write_kb":616448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"413'12789","reported_seq":19151,"reported_epoch":413,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:34:43.006725+0000","last_change":"2024-04-23T16:33:26.432192+0000","last_active":"2024-04-23T16:34:43.006725+0000","last_peered":"2024-04-23T16:34:43.006725+0000","last_clean":"2024-04-23T16:32:59.749530+0000","last_became_active":"2024-04-23T16:33:26.338644+0000","last_became_peered":"2024-04-23T16:33:26.338644+0000","last_unstale":"2024-04-23T16:34:43.006725+0000","last_undegraded":"2024-04-23T16:33:25.874847+0000","last_fullsized":"2024-04-23T16:34:43.006725+0000","mapping_epoch":394,"log_start":"370'2700","ondisk_log_start":"370'2700","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8711","last_scrub_stamp":"2024-04-23T16:32:43.928914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.928914+0000","objects_scrubbed":8680,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436011008,"num_objects":6653,"num_object_clones":0,"num_object_copies":13306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12789,"num_write_kb":622144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3072,"num_bytes_recovered":201326592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":["8"],"object_location_counts":[{"shards":"4,8","objects":13},{"shards":"8","objects":6640}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.9","version":"413'12897","reported_seq":13103,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:45.029210+0000","last_change":"2024-04-23T16:32:37.256944+0000","last_active":"2024-04-23T16:34:45.029210+0000","last_peered":"2024-04-23T16:34:45.029210+0000","last_clean":"2024-04-23T16:34:45.029210+0000","last_became_active":"2024-04-23T16:31:29.141551+0000","last_became_peered":"2024-04-23T16:31:29.141551+0000","last_unstale":"2024-04-23T16:34:45.029210+0000","last_undegraded":"2024-04-23T16:34:45.029210+0000","last_fullsized":"2024-04-23T16:34:45.029210+0000","mapping_epoch":359,"log_start":"370'2800","ondisk_log_start":"370'2800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8156","last_scrub_stamp":"2024-04-23T16:32:37.256787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:37.256787+0000","objects_scrubbed":8082,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420937728,"num_objects":6423,"num_object_clones":0,"num_object_copies":12846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12897,"num_write_kb":618240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"413'12316","reported_seq":12608,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:41.692278+0000","last_change":"2024-04-23T16:34:39.934502+0000","last_active":"2024-04-23T16:34:41.692278+0000","last_peered":"2024-04-23T16:34:41.692278+0000","last_clean":"2024-04-23T16:34:41.692278+0000","last_became_active":"2024-04-23T16:31:29.139045+0000","last_became_peered":"2024-04-23T16:31:29.139045+0000","last_unstale":"2024-04-23T16:34:41.692278+0000","last_undegraded":"2024-04-23T16:34:41.692278+0000","last_fullsized":"2024-04-23T16:34:41.692278+0000","mapping_epoch":359,"log_start":"370'2300","ondisk_log_start":"370'2300","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'12219","last_scrub_stamp":"2024-04-23T16:34:39.934190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:39.934190+0000","objects_scrubbed":6529,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:59.649413+0000","scrub_duration":810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420610048,"num_objects":6418,"num_object_clones":0,"num_object_copies":12836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12316,"num_write_kb":599488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"413'12658","reported_seq":12879,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:44.019340+0000","last_change":"2024-04-23T16:33:22.103034+0000","last_active":"2024-04-23T16:34:44.019340+0000","last_peered":"2024-04-23T16:34:44.019340+0000","last_clean":"2024-04-23T16:34:44.019340+0000","last_became_active":"2024-04-23T16:31:29.310906+0000","last_became_peered":"2024-04-23T16:31:29.310906+0000","last_unstale":"2024-04-23T16:34:44.019340+0000","last_undegraded":"2024-04-23T16:34:44.019340+0000","last_fullsized":"2024-04-23T16:34:44.019340+0000","mapping_epoch":359,"log_start":"370'2600","ondisk_log_start":"370'2600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9559","last_scrub_stamp":"2024-04-23T16:33:22.102857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:22.102857+0000","objects_scrubbed":9559,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:45.157065+0000","scrub_duration":758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423493632,"num_objects":6462,"num_object_clones":0,"num_object_copies":12924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":611840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1079,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:40.173250+0000","last_change":"2024-04-23T16:33:31.424170+0000","last_active":"2024-04-23T16:34:40.173250+0000","last_peered":"2024-04-23T16:34:40.173250+0000","last_clean":"2024-04-23T16:34:40.173250+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:34:40.173250+0000","last_undegraded":"2024-04-23T16:34:40.173250+0000","last_fullsized":"2024-04-23T16:34:40.173250+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:31.424058+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:33:31.424058+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:40.594682+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"413'12859","reported_seq":13066,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:44.372644+0000","last_change":"2024-04-23T16:32:38.103756+0000","last_active":"2024-04-23T16:34:44.372644+0000","last_peered":"2024-04-23T16:34:44.372644+0000","last_clean":"2024-04-23T16:34:44.372644+0000","last_became_active":"2024-04-23T16:31:29.141640+0000","last_became_peered":"2024-04-23T16:31:29.141640+0000","last_unstale":"2024-04-23T16:34:44.372644+0000","last_undegraded":"2024-04-23T16:34:44.372644+0000","last_fullsized":"2024-04-23T16:34:44.372644+0000","mapping_epoch":359,"log_start":"370'2800","ondisk_log_start":"370'2800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8300","last_scrub_stamp":"2024-04-23T16:32:38.103547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:38.103547+0000","objects_scrubbed":8231,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432340992,"num_objects":6597,"num_object_clones":0,"num_object_copies":13194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12859,"num_write_kb":622592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1081,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:40.181958+0000","last_change":"2024-04-23T16:33:25.851379+0000","last_active":"2024-04-23T16:34:40.181958+0000","last_peered":"2024-04-23T16:34:40.181958+0000","last_clean":"2024-04-23T16:34:40.181958+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:34:40.181958+0000","last_undegraded":"2024-04-23T16:34:40.181958+0000","last_fullsized":"2024-04-23T16:34:40.181958+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.498820+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.498820+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"413'12531","reported_seq":12827,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:41.745776+0000","last_change":"2024-04-23T16:34:41.375046+0000","last_active":"2024-04-23T16:34:41.745776+0000","last_peered":"2024-04-23T16:34:41.745776+0000","last_clean":"2024-04-23T16:34:41.745776+0000","last_became_active":"2024-04-23T16:31:29.140251+0000","last_became_peered":"2024-04-23T16:31:29.140251+0000","last_unstale":"2024-04-23T16:34:41.745776+0000","last_undegraded":"2024-04-23T16:34:41.745776+0000","last_fullsized":"2024-04-23T16:34:41.745776+0000","mapping_epoch":359,"log_start":"370'2500","ondisk_log_start":"370'2500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12511","last_scrub_stamp":"2024-04-23T16:34:41.374888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:41.374888+0000","objects_scrubbed":6643,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:07.291855+0000","scrub_duration":1271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432537624,"num_objects":6601,"num_object_clones":0,"num_object_copies":13202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6601,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12531,"num_write_kb":612161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1072,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:40.176094+0000","last_change":"2024-04-23T16:32:29.517947+0000","last_active":"2024-04-23T16:34:40.176094+0000","last_peered":"2024-04-23T16:34:40.176094+0000","last_clean":"2024-04-23T16:34:40.176094+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:34:40.176094+0000","last_undegraded":"2024-04-23T16:34:40.176094+0000","last_fullsized":"2024-04-23T16:34:40.176094+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:32:29.517800+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:32:29.517800+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"413'12886","reported_seq":19498,"reported_epoch":413,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:34:44.330385+0000","last_change":"2024-04-23T16:33:26.507399+0000","last_active":"2024-04-23T16:34:44.330385+0000","last_peered":"2024-04-23T16:34:44.330385+0000","last_clean":"2024-04-23T16:32:59.715575+0000","last_became_active":"2024-04-23T16:33:26.458611+0000","last_became_peered":"2024-04-23T16:33:26.458611+0000","last_unstale":"2024-04-23T16:34:44.330385+0000","last_undegraded":"2024-04-23T16:33:25.895472+0000","last_fullsized":"2024-04-23T16:34:44.330385+0000","mapping_epoch":394,"log_start":"370'2800","ondisk_log_start":"370'2800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8607","last_scrub_stamp":"2024-04-23T16:32:43.130806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.130806+0000","objects_scrubbed":8577,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425590784,"num_objects":6494,"num_object_clones":0,"num_object_copies":12988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12886,"num_write_kb":620160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3198,"num_bytes_recovered":209584128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":6489},{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1079,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:40.180023+0000","last_change":"2024-04-23T16:33:33.275883+0000","last_active":"2024-04-23T16:34:40.180023+0000","last_peered":"2024-04-23T16:34:40.180023+0000","last_clean":"2024-04-23T16:34:40.180023+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:34:40.180023+0000","last_undegraded":"2024-04-23T16:34:40.180023+0000","last_fullsized":"2024-04-23T16:34:40.180023+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:33.275799+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:33:33.275799+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"413'12606","reported_seq":12825,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:44.357066+0000","last_change":"2024-04-23T16:33:18.576077+0000","last_active":"2024-04-23T16:34:44.357066+0000","last_peered":"2024-04-23T16:34:44.357066+0000","last_clean":"2024-04-23T16:34:44.357066+0000","last_became_active":"2024-04-23T16:31:29.139709+0000","last_became_peered":"2024-04-23T16:31:29.139709+0000","last_unstale":"2024-04-23T16:34:44.357066+0000","last_undegraded":"2024-04-23T16:34:44.357066+0000","last_fullsized":"2024-04-23T16:34:44.357066+0000","mapping_epoch":359,"log_start":"370'2600","ondisk_log_start":"370'2600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9472","last_scrub_stamp":"2024-04-23T16:33:18.575945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.575945+0000","objects_scrubbed":9472,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415367168,"num_objects":6338,"num_object_clones":0,"num_object_copies":12676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12606,"num_write_kb":606208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1080,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:40.182948+0000","last_change":"2024-04-23T16:34:02.399102+0000","last_active":"2024-04-23T16:34:40.182948+0000","last_peered":"2024-04-23T16:34:40.182948+0000","last_clean":"2024-04-23T16:34:40.182948+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:34:40.182948+0000","last_undegraded":"2024-04-23T16:34:40.182948+0000","last_fullsized":"2024-04-23T16:34:40.182948+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:02.398975+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:02.398975+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:23.945801+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"413'12739","reported_seq":13019,"reported_epoch":413,"state":"active+clean+scrubbing","last_fresh":"2024-04-23T16:34:44.110740+0000","last_change":"2024-04-23T16:34:43.323572+0000","last_active":"2024-04-23T16:34:44.110740+0000","last_peered":"2024-04-23T16:34:44.110740+0000","last_clean":"2024-04-23T16:34:44.110740+0000","last_became_active":"2024-04-23T16:31:29.139606+0000","last_became_peered":"2024-04-23T16:31:29.139606+0000","last_unstale":"2024-04-23T16:34:44.110740+0000","last_undegraded":"2024-04-23T16:34:44.110740+0000","last_fullsized":"2024-04-23T16:34:44.110740+0000","mapping_epoch":359,"log_start":"370'2700","ondisk_log_start":"370'2700","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9720","last_scrub_stamp":"2024-04-23T16:33:19.938119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:19.938119+0000","objects_scrubbed":5299,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439156736,"num_objects":6701,"num_object_clones":0,"num_object_copies":13402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12739,"num_write_kb":622080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1079,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:40.172977+0000","last_change":"2024-04-23T16:33:28.454288+0000","last_active":"2024-04-23T16:34:40.172977+0000","last_peered":"2024-04-23T16:34:40.172977+0000","last_clean":"2024-04-23T16:34:40.172977+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:34:40.172977+0000","last_undegraded":"2024-04-23T16:34:40.172977+0000","last_fullsized":"2024-04-23T16:34:40.172977+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:28.454200+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:33:28.454200+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:51.316141+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.2","version":"413'12637","reported_seq":12935,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:44.558196+0000","last_change":"2024-04-23T16:33:57.879648+0000","last_active":"2024-04-23T16:34:44.558196+0000","last_peered":"2024-04-23T16:34:44.558196+0000","last_clean":"2024-04-23T16:34:44.558196+0000","last_became_active":"2024-04-23T16:31:29.138430+0000","last_became_peered":"2024-04-23T16:31:29.138430+0000","last_unstale":"2024-04-23T16:34:44.558196+0000","last_undegraded":"2024-04-23T16:34:44.558196+0000","last_fullsized":"2024-04-23T16:34:44.558196+0000","mapping_epoch":359,"log_start":"370'2600","ondisk_log_start":"370'2600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'10726","last_scrub_stamp":"2024-04-23T16:33:57.879516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:57.879516+0000","objects_scrubbed":8303,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:02.097116+0000","scrub_duration":1621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416743424,"num_objects":6359,"num_object_clones":0,"num_object_copies":12718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12637,"num_write_kb":607872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1087,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:40.173082+0000","last_change":"2024-04-23T16:34:37.693617+0000","last_active":"2024-04-23T16:34:40.173082+0000","last_peered":"2024-04-23T16:34:40.173082+0000","last_clean":"2024-04-23T16:34:40.173082+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:34:40.173082+0000","last_undegraded":"2024-04-23T16:34:40.173082+0000","last_fullsized":"2024-04-23T16:34:40.173082+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:37.693482+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:37.693482+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:49.920873+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"413'14957","reported_seq":15381,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:44.004945+0000","last_change":"2024-04-23T16:34:40.604268+0000","last_active":"2024-04-23T16:34:44.004945+0000","last_peered":"2024-04-23T16:34:44.004945+0000","last_clean":"2024-04-23T16:34:44.004945+0000","last_became_active":"2024-04-23T16:31:29.139398+0000","last_became_peered":"2024-04-23T16:31:29.139398+0000","last_unstale":"2024-04-23T16:34:44.004945+0000","last_undegraded":"2024-04-23T16:34:44.004945+0000","last_fullsized":"2024-04-23T16:34:44.004945+0000","mapping_epoch":359,"log_start":"373'4900","ondisk_log_start":"373'4900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'14807","last_scrub_stamp":"2024-04-23T16:34:40.604136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:40.604136+0000","objects_scrubbed":6625,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:04.702557+0000","scrub_duration":1231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422379520,"num_objects":6445,"num_object_clones":0,"num_object_copies":12890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12683,"num_write_kb":612096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1079,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:40.182849+0000","last_change":"2024-04-23T16:34:04.444103+0000","last_active":"2024-04-23T16:34:40.182849+0000","last_peered":"2024-04-23T16:34:40.182849+0000","last_clean":"2024-04-23T16:34:40.182849+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:34:40.182849+0000","last_undegraded":"2024-04-23T16:34:40.182849+0000","last_fullsized":"2024-04-23T16:34:40.182849+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:04.443975+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:04.443975+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:05.109970+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"413'12641","reported_seq":12850,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:44.346566+0000","last_change":"2024-04-23T16:32:43.947374+0000","last_active":"2024-04-23T16:34:44.346566+0000","last_peered":"2024-04-23T16:34:44.346566+0000","last_clean":"2024-04-23T16:34:44.346566+0000","last_became_active":"2024-04-23T16:31:29.138968+0000","last_became_peered":"2024-04-23T16:31:29.138968+0000","last_unstale":"2024-04-23T16:34:44.346566+0000","last_undegraded":"2024-04-23T16:34:44.346566+0000","last_fullsized":"2024-04-23T16:34:44.346566+0000","mapping_epoch":359,"log_start":"370'2600","ondisk_log_start":"370'2600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8482","last_scrub_stamp":"2024-04-23T16:32:43.947177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.947177+0000","objects_scrubbed":8428,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409796608,"num_objects":6253,"num_object_clones":0,"num_object_copies":12506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12641,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":971,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:40.172233+0000","last_change":"2024-04-23T16:33:19.443898+0000","last_active":"2024-04-23T16:34:40.172233+0000","last_peered":"2024-04-23T16:34:40.172233+0000","last_clean":"2024-04-23T16:34:40.172233+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:34:40.172233+0000","last_undegraded":"2024-04-23T16:34:40.172233+0000","last_fullsized":"2024-04-23T16:34:40.172233+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:19.443792+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:33:19.443792+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"413'12905","reported_seq":13113,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:44.388638+0000","last_change":"2024-04-23T16:32:39.103230+0000","last_active":"2024-04-23T16:34:44.388638+0000","last_peered":"2024-04-23T16:34:44.388638+0000","last_clean":"2024-04-23T16:34:44.388638+0000","last_became_active":"2024-04-23T16:31:29.310246+0000","last_became_peered":"2024-04-23T16:31:29.310246+0000","last_unstale":"2024-04-23T16:34:44.388638+0000","last_undegraded":"2024-04-23T16:34:44.388638+0000","last_fullsized":"2024-04-23T16:34:44.388638+0000","mapping_epoch":359,"log_start":"370'2900","ondisk_log_start":"370'2900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8325","last_scrub_stamp":"2024-04-23T16:32:39.103059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:39.103059+0000","objects_scrubbed":8272,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429719552,"num_objects":6557,"num_object_clones":0,"num_object_copies":13114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12905,"num_write_kb":622784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":892,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:40.182738+0000","last_change":"2024-04-23T16:33:26.153079+0000","last_active":"2024-04-23T16:34:40.182738+0000","last_peered":"2024-04-23T16:34:40.182738+0000","last_clean":"2024-04-23T16:34:40.182738+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:34:40.182738+0000","last_undegraded":"2024-04-23T16:34:40.182738+0000","last_fullsized":"2024-04-23T16:34:40.182738+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:26.152962+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:33:26.152962+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:53.292007+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"413'12749","reported_seq":19352,"reported_epoch":413,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:34:44.330849+0000","last_change":"2024-04-23T16:33:26.132487+0000","last_active":"2024-04-23T16:34:44.330849+0000","last_peered":"2024-04-23T16:34:44.330849+0000","last_clean":"2024-04-23T16:32:59.695133+0000","last_became_active":"2024-04-23T16:33:26.078062+0000","last_became_peered":"2024-04-23T16:33:26.078062+0000","last_unstale":"2024-04-23T16:34:44.330849+0000","last_undegraded":"2024-04-23T16:33:25.892009+0000","last_fullsized":"2024-04-23T16:34:44.330849+0000","mapping_epoch":394,"log_start":"370'2700","ondisk_log_start":"370'2700","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'7968","last_scrub_stamp":"2024-04-23T16:32:36.102122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:36.102122+0000","objects_scrubbed":7926,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417136640,"num_objects":6365,"num_object_clones":0,"num_object_copies":12730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3196,"num_bytes_recovered":209453056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":6357},{"shards":"5,8","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1077,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-04-23T16:34:40.175981+0000","last_change":"2024-04-23T16:33:17.354409+0000","last_active":"2024-04-23T16:34:40.175981+0000","last_peered":"2024-04-23T16:34:40.175981+0000","last_clean":"2024-04-23T16:34:40.175981+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:34:40.175981+0000","last_undegraded":"2024-04-23T16:34:40.175981+0000","last_fullsized":"2024-04-23T16:34:40.175981+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:17.354263+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.354263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":6800146456,"num_objects":103763,"num_object_clones":0,"num_object_copies":207526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103763,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":203429,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9466,"num_bytes_recovered":620363776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14213390336,"data_stored":14213382192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160803,"ondisk_log_size":160803,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056873,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1334696,"kb_used_data":838428,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":92410456,"statfs":{"total":95995035648,"available":94628306944,"internally_reserved":0,"allocated":858550272,"data_stored":858172163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":508164548},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351215,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612853,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1561120,"kb_used_data":1245336,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92184032,"statfs":{"total":95995035648,"available":94396448768,"internally_reserved":0,"allocated":1275224064,"data_stored":1274850027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":323353024},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841785,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1038432,"kb_used_data":848984,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92706720,"statfs":{"total":95995035648,"available":94931681280,"internally_reserved":0,"allocated":869359616,"data_stored":868985579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168830,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":773984,"kb_used_data":413784,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92971168,"statfs":{"total":95995035648,"available":95202476032,"internally_reserved":0,"allocated":423714816,"data_stored":423340779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":368835018},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463172,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1494048,"kb_used_data":1232408,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":92251104,"statfs":{"total":95995035648,"available":94465130496,"internally_reserved":0,"allocated":1261985792,"data_stored":1261611755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":267909574},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137311,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738657,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3309408,"kb_used_data":2913112,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":90435744,"statfs":{"total":95995035648,"available":92606201856,"internally_reserved":0,"allocated":2983026688,"data_stored":2982652651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":783148,"kb_used_data":431264,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":92962004,"statfs":{"total":95995035648,"available":95193092096,"internally_reserved":0,"allocated":441614336,"data_stored":441236747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":360315335},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921036,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2544608,"kb_used_data":2090008,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":91200544,"statfs":{"total":95995035648,"available":93389357056,"internally_reserved":0,"allocated":2140168192,"data_stored":2139794155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757511,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2477548,"kb_used_data":2076964,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":91267604,"statfs":{"total":95995035648,"available":93458026496,"internally_reserved":0,"allocated":2126811136,"data_stored":2126429475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2967732224,"data_stored":2967732224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1886846976,"data_stored":1886846976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":440008704,"data_stored":440008704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2138701824,"data_stored":2138701824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2111705088,"data_stored":2111701016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1260257280,"data_stored":1260257280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":421462016,"data_stored":421462016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":863305728,"data_stored":863305728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1270218752,"data_stored":1270218752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":853151744,"data_stored":853147672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:34:47.215 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:34:47.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:46 smithi188 ceph-mon[44101]: from='client.16026 v1:172.21.15.103:0/2093789954' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:34:47.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:46 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:34:47.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:46 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3028371438' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:34:47.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:46 smithi188 ceph-mon[44101]: osdmap e415: 12 total, 11 up, 9 in 2024-04-23T16:34:47.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:46 smithi148 ceph-mon[44520]: from='client.16026 v1:172.21.15.103:0/2093789954' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:34:47.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:46 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:34:47.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:46 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3028371438' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:34:47.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:46 smithi148 ceph-mon[44520]: osdmap e415: 12 total, 11 up, 9 in 2024-04-23T16:34:47.725 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:34:47.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:47.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:34:47.831 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:34:47.983 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:47 smithi188 ceph-mon[44101]: pgmap v1171: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering+degraded, 23 active+clean; 6.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 530 op/s; 19486/207530 objects degraded (9.389%); 6.1 MiB/s, 97 objects/s recovering 2024-04-23T16:34:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:47 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:34:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:47 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:34:48.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:47 smithi103 ceph-mon[39629]: pgmap v1171: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering+degraded, 23 active+clean; 6.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 530 op/s; 19486/207530 objects degraded (9.389%); 6.1 MiB/s, 97 objects/s recovering 2024-04-23T16:34:48.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:47 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:34:48.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:47 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:34:48.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:47 smithi148 ceph-mon[44520]: pgmap v1171: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering+degraded, 23 active+clean; 6.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 530 op/s; 19486/207530 objects degraded (9.389%); 6.1 MiB/s, 97 objects/s recovering 2024-04-23T16:34:48.382 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:47 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:34:48.382 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:47 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:34:48.670 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:34:48.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:48.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:34:48.779 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:34:48.813 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:48.846 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:34:48.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:48.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:34:48.951 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:34:48.994 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:49.024 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:34:49.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:49.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:34:49.128 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:34:49.159 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:49.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:48 smithi188 ceph-mon[44101]: 5.e deep-scrub starts 2024-04-23T16:34:49.232 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:34:49.234 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:48 smithi103 ceph-mon[39629]: 5.e deep-scrub starts 2024-04-23T16:34:49.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:49.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:34:49.261 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:49.337 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:49.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:48 smithi148 ceph-mon[44520]: 5.e deep-scrub starts 2024-04-23T16:34:49.477 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:49.719 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:34:50.164 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:50.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:50.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:49 smithi188 ceph-mon[44101]: pgmap v1172: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering+degraded, 23 active+clean; 6.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 511 op/s; 19277/206294 objects degraded (9.344%); 6.6 MiB/s, 105 objects/s recovering 2024-04-23T16:34:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:49 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 19277/206294 objects degraded (9.344%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:50.268 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:50.322 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:49 smithi148 ceph-mon[44520]: pgmap v1172: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering+degraded, 23 active+clean; 6.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 511 op/s; 19277/206294 objects degraded (9.344%); 6.6 MiB/s, 105 objects/s recovering 2024-04-23T16:34:50.322 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:49 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 19277/206294 objects degraded (9.344%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:50.323 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:50.323 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:49 smithi103 ceph-mon[39629]: pgmap v1172: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering+degraded, 23 active+clean; 6.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 511 op/s; 19277/206294 objects degraded (9.344%); 6.6 MiB/s, 105 objects/s recovering 2024-04-23T16:34:50.323 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:49 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 19277/206294 objects degraded (9.344%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:50.348 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:50.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:50.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:50.453 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:34:50.496 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:50.517 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:34:50.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:50.522 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:34:50.623 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:34:50.658 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:50.678 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:34:50.678 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:34:50.687 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:34:50.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:50.691 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:34:50.792 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:34:50.825 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:50.859 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:34:50.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:50.862 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:34:50.963 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:51.038 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:51.085 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:51.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:51.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:51.194 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:51.256 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:51.351 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:51.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:51.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:51.464 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:34:51.502 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:51.526 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:34:51.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:51.534 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:34:51.635 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:34:51.669 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:51.694 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:34:51.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:51.697 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:34:51.798 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:34:51.867 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:51.903 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:34:51.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:51.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:34:51.908 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:51 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:34:51.908 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:51 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:34:51.908 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:51 smithi148 ceph-mon[44520]: pgmap v1173: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 6.1 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 795 op/s; 18850/200884 objects degraded (9.384%); 9.9 MiB/s, 159 objects/s recovering 2024-04-23T16:34:51.909 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:51 smithi148 ceph-mon[44520]: 1.4 scrub starts 2024-04-23T16:34:51.909 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:51 smithi148 ceph-mon[44520]: 1.4 scrub ok 2024-04-23T16:34:51.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:51 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:34:51.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:51 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:34:51.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:51 smithi188 ceph-mon[44101]: pgmap v1173: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 6.1 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 795 op/s; 18850/200884 objects degraded (9.384%); 9.9 MiB/s, 159 objects/s recovering 2024-04-23T16:34:51.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:51 smithi188 ceph-mon[44101]: 1.4 scrub starts 2024-04-23T16:34:51.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:51 smithi188 ceph-mon[44101]: 1.4 scrub ok 2024-04-23T16:34:52.010 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:34:52.042 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:52.079 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:34:52.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:52.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:34:52.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:51 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:34:52.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:51 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:34:52.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:51 smithi103 ceph-mon[39629]: pgmap v1173: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 6.1 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 795 op/s; 18850/200884 objects degraded (9.384%); 9.9 MiB/s, 159 objects/s recovering 2024-04-23T16:34:52.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:51 smithi103 ceph-mon[39629]: 1.4 scrub starts 2024-04-23T16:34:52.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:51 smithi103 ceph-mon[39629]: 1.4 scrub ok 2024-04-23T16:34:52.183 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:34:52.216 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:52.418 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:34:52.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:52.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:34:52.437 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-23T16:34:52.521 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:34:52.549 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:52.651 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:34:52.651 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:34:52.651 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:34:52.651 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:34:52.651 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:34:52.651 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:34:52.651 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:34:52.651 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:34:52.651 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:34:52.651 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:34:52.651 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:34:52.652 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:34:52.652 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:34:52.652 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:34:52.652 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:34:52.652 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:34:52.652 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:34:52.652 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:34:52.652 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:34:52.653 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:34:52.653 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:34:52.653 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:34:52.653 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:34:52.653 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:34:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:52.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:34:52.757 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:34:52.790 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:52.815 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:34:52.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:52.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:34:52.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:52 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:34:52.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:52 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:34:52.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:34:52.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:34:52.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:34:52.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:52 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/423551595' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:34:52.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:52.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:52.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:52.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:52.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:52.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:52.921 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:34:52.954 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:52 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:34:52.954 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:52 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:34:52.954 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:34:52.954 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:34:52.954 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:34:52.955 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:52 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/423551595' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:34:52.955 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:52.955 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:52.955 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:52.955 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:52.955 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:52.955 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:52.957 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:52.986 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:34:52.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:52.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:34:53.090 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:53.133 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:52 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:34:53.134 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:52 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:34:53.134 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:34:53.134 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:34:53.134 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:34:53.134 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:52 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/423551595' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:34:53.134 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.134 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.134 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.134 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.134 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.134 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.158 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:53.196 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:53.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:53.303 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:34:53.336 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:53.359 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:34:53.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:53.362 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:34:53.463 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:53.528 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:53.572 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:53.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:53.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:53.595 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:34:53.596 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:34:53.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:53 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:53 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:53 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:53 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:53 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:53 smithi188 ceph-mon[44101]: pgmap v1174: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 6.1 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 497 op/s; 18850/200014 objects degraded (9.424%); 5.0 MiB/s, 79 objects/s recovering 2024-04-23T16:34:53.681 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:34:53.751 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:53.812 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:34:53.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:53.817 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:53 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.817 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:53 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.817 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:53 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.818 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:53 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.818 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:53 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.818 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:53 smithi148 ceph-mon[44520]: pgmap v1174: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 6.1 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 497 op/s; 18850/200014 objects degraded (9.424%); 5.0 MiB/s, 79 objects/s recovering 2024-04-23T16:34:53.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:34:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:53 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:53 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:53 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:53 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:53 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:34:53.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:53 smithi103 ceph-mon[39629]: pgmap v1174: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 6.1 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 497 op/s; 18850/200014 objects degraded (9.424%); 5.0 MiB/s, 79 objects/s recovering 2024-04-23T16:34:53.919 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:34:53.949 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:53.967 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:53.975 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:34:53.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:53.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:34:54.079 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:34:54.129 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:54.180 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:34:54.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:54.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:34:54.288 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:34:54.322 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:54.342 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:34:54.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:54.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:34:54.446 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:34:54.475 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:54.496 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:34:54.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:54.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:34:54.601 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:34:54.631 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:54.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:54 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 18850/200014 objects degraded (9.424%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:55.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:54 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 18850/200014 objects degraded (9.424%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:55.105 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:55.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:54 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 18850/200014 objects degraded (9.424%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:34:55.562 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:34:55.565 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:34:55.590 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:34:55.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:55.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:34:55.596 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:34:55.696 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:34:55.772 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:55.808 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:34:55.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:55.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:34:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:55 smithi103 ceph-mon[39629]: pgmap v1175: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.9 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 666 op/s; 18249/194794 objects degraded (9.368%); 8.1 MiB/s, 129 objects/s recovering 2024-04-23T16:34:55.919 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:34:55.967 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:55 smithi148 ceph-mon[44520]: pgmap v1175: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.9 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 666 op/s; 18249/194794 objects degraded (9.368%); 8.1 MiB/s, 129 objects/s recovering 2024-04-23T16:34:55.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:55 smithi188 ceph-mon[44101]: pgmap v1175: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.9 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 666 op/s; 18249/194794 objects degraded (9.368%); 8.1 MiB/s, 129 objects/s recovering 2024-04-23T16:34:55.975 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:56.005 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:34:56.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:56.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:34:56.113 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:34:56.141 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:56.391 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:34:56.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:56.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:34:56.496 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:34:56.536 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:56.565 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:34:56.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:56.568 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:34:56.669 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:34:56.674 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:56 smithi103 ceph-mon[39629]: from='client.16035 v1:172.21.15.103:0/3315112081' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:34:56.728 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:56.760 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:34:56.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:56.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:34:56.869 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:34:56.924 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:56.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:56 smithi188 ceph-mon[44101]: from='client.16035 v1:172.21.15.103:0/3315112081' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:34:56.985 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:34:56.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:56.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:34:56.992 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:56 smithi148 ceph-mon[44520]: from='client.16035 v1:172.21.15.103:0/3315112081' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:34:57.091 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:34:57.131 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:57.161 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:34:57.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:57.173 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:34:57.274 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:34:57.303 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:57.906 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1175,"stamp":"2024-04-23T16:34:54.726472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6383403576,"num_objects":97397,"num_object_clones":0,"num_object_copies":194794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97397,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":209797,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10704,"num_bytes_recovered":701633056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162472,"ondisk_log_size":162472,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":14648848,"kb_used_data":11337608,"kb_used_omap":14,"kb_used_meta":3311153,"kb_avail":829057520,"statfs":{"total":863955320832,"available":848954900480,"internally_reserved":0,"allocated":11609710592,"data_stored":11606220489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3390621679},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,2,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-417333248,"num_objects":-6368,"num_object_clones":0,"num_object_copies":-12736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6368,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1237,"num_bytes_recovered":81068032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.548646"},"pg_stats":[{"pgid":"5.f","version":"415'13251","reported_seq":13476,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:54.348285+0000","last_change":"2024-04-23T16:33:17.526419+0000","last_active":"2024-04-23T16:34:54.348285+0000","last_peered":"2024-04-23T16:34:54.348285+0000","last_clean":"2024-04-23T16:34:54.348285+0000","last_became_active":"2024-04-23T16:31:29.140923+0000","last_became_peered":"2024-04-23T16:31:29.140923+0000","last_unstale":"2024-04-23T16:34:54.348285+0000","last_undegraded":"2024-04-23T16:34:54.348285+0000","last_fullsized":"2024-04-23T16:34:54.348285+0000","mapping_epoch":359,"log_start":"370'3200","ondisk_log_start":"370'3200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9683","last_scrub_stamp":"2024-04-23T16:33:17.526283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.526283+0000","objects_scrubbed":9683,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400752640,"num_objects":6115,"num_object_clones":0,"num_object_copies":12230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13251,"num_write_kb":619712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"415'13093","reported_seq":13390,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:54.041797+0000","last_change":"2024-04-23T16:34:02.531358+0000","last_active":"2024-04-23T16:34:54.041797+0000","last_peered":"2024-04-23T16:34:54.041797+0000","last_clean":"2024-04-23T16:34:54.041797+0000","last_became_active":"2024-04-23T16:31:29.137890+0000","last_became_peered":"2024-04-23T16:31:29.137890+0000","last_unstale":"2024-04-23T16:34:54.041797+0000","last_undegraded":"2024-04-23T16:34:54.041797+0000","last_fullsized":"2024-04-23T16:34:54.041797+0000","mapping_epoch":359,"log_start":"370'3000","ondisk_log_start":"370'3000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'10975","last_scrub_stamp":"2024-04-23T16:34:02.531162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:02.531162+0000","objects_scrubbed":8314,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:09.558502+0000","scrub_duration":1066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404422656,"num_objects":6171,"num_object_clones":0,"num_object_copies":12342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13093,"num_write_kb":616448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"415'13200","reported_seq":20380,"reported_epoch":415,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:34:53.054767+0000","last_change":"2024-04-23T16:33:26.432192+0000","last_active":"2024-04-23T16:34:53.054767+0000","last_peered":"2024-04-23T16:34:53.054767+0000","last_clean":"2024-04-23T16:32:59.749530+0000","last_became_active":"2024-04-23T16:33:26.338644+0000","last_became_peered":"2024-04-23T16:33:26.338644+0000","last_unstale":"2024-04-23T16:34:53.054767+0000","last_undegraded":"2024-04-23T16:33:25.874847+0000","last_fullsized":"2024-04-23T16:34:53.054767+0000","mapping_epoch":394,"log_start":"370'3100","ondisk_log_start":"370'3100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8711","last_scrub_stamp":"2024-04-23T16:32:43.928914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.928914+0000","objects_scrubbed":8680,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409075712,"num_objects":6242,"num_object_clones":0,"num_object_copies":12484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13200,"num_write_kb":622144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3480,"num_bytes_recovered":228065280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":["8"],"object_location_counts":[{"shards":"4,8","objects":10},{"shards":"8","objects":6232}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.9","version":"415'13099","reported_seq":13309,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:50.115002+0000","last_change":"2024-04-23T16:32:37.256944+0000","last_active":"2024-04-23T16:34:50.115002+0000","last_peered":"2024-04-23T16:34:50.115002+0000","last_clean":"2024-04-23T16:34:50.115002+0000","last_became_active":"2024-04-23T16:31:29.141551+0000","last_became_peered":"2024-04-23T16:31:29.141551+0000","last_unstale":"2024-04-23T16:34:50.115002+0000","last_undegraded":"2024-04-23T16:34:50.115002+0000","last_fullsized":"2024-04-23T16:34:50.115002+0000","mapping_epoch":359,"log_start":"370'3000","ondisk_log_start":"370'3000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8156","last_scrub_stamp":"2024-04-23T16:32:37.256787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:37.256787+0000","objects_scrubbed":8082,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407699456,"num_objects":6221,"num_object_clones":0,"num_object_copies":12442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13099,"num_write_kb":618240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"415'12747","reported_seq":13043,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:51.668412+0000","last_change":"2024-04-23T16:34:39.934502+0000","last_active":"2024-04-23T16:34:51.668412+0000","last_peered":"2024-04-23T16:34:51.668412+0000","last_clean":"2024-04-23T16:34:51.668412+0000","last_became_active":"2024-04-23T16:31:29.139045+0000","last_became_peered":"2024-04-23T16:31:29.139045+0000","last_unstale":"2024-04-23T16:34:51.668412+0000","last_undegraded":"2024-04-23T16:34:51.668412+0000","last_fullsized":"2024-04-23T16:34:51.668412+0000","mapping_epoch":359,"log_start":"370'2700","ondisk_log_start":"370'2700","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'12219","last_scrub_stamp":"2024-04-23T16:34:39.934190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:39.934190+0000","objects_scrubbed":6529,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:59.649413+0000","scrub_duration":810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392364032,"num_objects":5987,"num_object_clones":0,"num_object_copies":11974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12747,"num_write_kb":599488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"415'13954","reported_seq":14217,"reported_epoch":415,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-23T16:34:54.095340+0000","last_change":"2024-04-23T16:34:48.708083+0000","last_active":"2024-04-23T16:34:54.095340+0000","last_peered":"2024-04-23T16:34:54.095340+0000","last_clean":"2024-04-23T16:34:54.095340+0000","last_became_active":"2024-04-23T16:31:29.310906+0000","last_became_peered":"2024-04-23T16:31:29.310906+0000","last_unstale":"2024-04-23T16:34:54.095340+0000","last_undegraded":"2024-04-23T16:34:54.095340+0000","last_fullsized":"2024-04-23T16:34:54.095340+0000","mapping_epoch":359,"log_start":"371'3902","ondisk_log_start":"371'3902","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9559","last_scrub_stamp":"2024-04-23T16:33:22.102857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:22.102857+0000","objects_scrubbed":874,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"deep scrubbing for 5s","scrub_duration":758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395837440,"num_objects":6040,"num_object_clones":0,"num_object_copies":12080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13080,"num_write_kb":611840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1091,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:46.189977+0000","last_change":"2024-04-23T16:34:44.725424+0000","last_active":"2024-04-23T16:34:46.189977+0000","last_peered":"2024-04-23T16:34:46.189977+0000","last_clean":"2024-04-23T16:34:46.189977+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:34:46.189977+0000","last_undegraded":"2024-04-23T16:34:46.189977+0000","last_fullsized":"2024-04-23T16:34:46.189977+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:44.725324+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:44.725324+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:14.224827+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"415'13271","reported_seq":13482,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:54.339726+0000","last_change":"2024-04-23T16:32:38.103756+0000","last_active":"2024-04-23T16:34:54.339726+0000","last_peered":"2024-04-23T16:34:54.339726+0000","last_clean":"2024-04-23T16:34:54.339726+0000","last_became_active":"2024-04-23T16:31:29.141640+0000","last_became_peered":"2024-04-23T16:31:29.141640+0000","last_unstale":"2024-04-23T16:34:54.339726+0000","last_undegraded":"2024-04-23T16:34:54.339726+0000","last_fullsized":"2024-04-23T16:34:54.339726+0000","mapping_epoch":359,"log_start":"370'3200","ondisk_log_start":"370'3200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8300","last_scrub_stamp":"2024-04-23T16:32:38.103547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:38.103547+0000","objects_scrubbed":8231,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405340160,"num_objects":6185,"num_object_clones":0,"num_object_copies":12370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13271,"num_write_kb":622592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1085,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:46.195826+0000","last_change":"2024-04-23T16:33:25.851379+0000","last_active":"2024-04-23T16:34:46.195826+0000","last_peered":"2024-04-23T16:34:46.195826+0000","last_clean":"2024-04-23T16:34:46.195826+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:34:46.195826+0000","last_undegraded":"2024-04-23T16:34:46.195826+0000","last_fullsized":"2024-04-23T16:34:46.195826+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.498820+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.498820+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"415'12941","reported_seq":13241,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:51.602725+0000","last_change":"2024-04-23T16:34:41.375046+0000","last_active":"2024-04-23T16:34:51.602725+0000","last_peered":"2024-04-23T16:34:51.602725+0000","last_clean":"2024-04-23T16:34:51.602725+0000","last_became_active":"2024-04-23T16:31:29.140251+0000","last_became_peered":"2024-04-23T16:31:29.140251+0000","last_unstale":"2024-04-23T16:34:51.602725+0000","last_undegraded":"2024-04-23T16:34:51.602725+0000","last_fullsized":"2024-04-23T16:34:51.602725+0000","mapping_epoch":359,"log_start":"370'2900","ondisk_log_start":"370'2900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12511","last_scrub_stamp":"2024-04-23T16:34:41.374888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:41.374888+0000","objects_scrubbed":6643,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:07.291855+0000","scrub_duration":1271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405667864,"num_objects":6191,"num_object_clones":0,"num_object_copies":12382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6191,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12941,"num_write_kb":612161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1077,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:46.222520+0000","last_change":"2024-04-23T16:32:29.517947+0000","last_active":"2024-04-23T16:34:46.222520+0000","last_peered":"2024-04-23T16:34:46.222520+0000","last_clean":"2024-04-23T16:34:46.222520+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:34:46.222520+0000","last_undegraded":"2024-04-23T16:34:46.222520+0000","last_fullsized":"2024-04-23T16:34:46.222520+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:32:29.517800+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:32:29.517800+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"415'13305","reported_seq":20760,"reported_epoch":415,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:34:54.418356+0000","last_change":"2024-04-23T16:33:26.507399+0000","last_active":"2024-04-23T16:34:54.418356+0000","last_peered":"2024-04-23T16:34:54.418356+0000","last_clean":"2024-04-23T16:32:59.715575+0000","last_became_active":"2024-04-23T16:33:26.458611+0000","last_became_peered":"2024-04-23T16:33:26.458611+0000","last_unstale":"2024-04-23T16:34:54.418356+0000","last_undegraded":"2024-04-23T16:33:25.895472+0000","last_fullsized":"2024-04-23T16:34:54.418356+0000","mapping_epoch":394,"log_start":"370'3300","ondisk_log_start":"370'3300","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8607","last_scrub_stamp":"2024-04-23T16:32:43.130806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.130806+0000","objects_scrubbed":8577,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398131200,"num_objects":6075,"num_object_clones":0,"num_object_copies":12150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13305,"num_write_kb":620160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3618,"num_bytes_recovered":237109248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":6069},{"shards":"5,7","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1091,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:47.438436+0000","last_change":"2024-04-23T16:34:47.438492+0000","last_active":"2024-04-23T16:34:47.438436+0000","last_peered":"2024-04-23T16:34:47.438436+0000","last_clean":"2024-04-23T16:34:47.438436+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:34:47.438436+0000","last_undegraded":"2024-04-23T16:34:47.438436+0000","last_fullsized":"2024-04-23T16:34:47.438436+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:47.438356+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:47.438356+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:13.051838+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"415'12977","reported_seq":13200,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:54.434259+0000","last_change":"2024-04-23T16:33:18.576077+0000","last_active":"2024-04-23T16:34:54.434259+0000","last_peered":"2024-04-23T16:34:54.434259+0000","last_clean":"2024-04-23T16:34:54.434259+0000","last_became_active":"2024-04-23T16:31:29.139709+0000","last_became_peered":"2024-04-23T16:31:29.139709+0000","last_unstale":"2024-04-23T16:34:54.434259+0000","last_undegraded":"2024-04-23T16:34:54.434259+0000","last_fullsized":"2024-04-23T16:34:54.434259+0000","mapping_epoch":359,"log_start":"370'2900","ondisk_log_start":"370'2900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9472","last_scrub_stamp":"2024-04-23T16:33:18.575945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.575945+0000","objects_scrubbed":9472,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391053312,"num_objects":5967,"num_object_clones":0,"num_object_copies":11934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12977,"num_write_kb":606208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1084,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:46.200658+0000","last_change":"2024-04-23T16:34:02.399102+0000","last_active":"2024-04-23T16:34:46.200658+0000","last_peered":"2024-04-23T16:34:46.200658+0000","last_clean":"2024-04-23T16:34:46.200658+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:34:46.200658+0000","last_undegraded":"2024-04-23T16:34:46.200658+0000","last_fullsized":"2024-04-23T16:34:46.200658+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:02.398975+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:02.398975+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:23.945801+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"415'13175","reported_seq":13478,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:54.110103+0000","last_change":"2024-04-23T16:34:44.348696+0000","last_active":"2024-04-23T16:34:54.110103+0000","last_peered":"2024-04-23T16:34:54.110103+0000","last_clean":"2024-04-23T16:34:54.110103+0000","last_became_active":"2024-04-23T16:31:29.139606+0000","last_became_peered":"2024-04-23T16:31:29.139606+0000","last_unstale":"2024-04-23T16:34:54.110103+0000","last_undegraded":"2024-04-23T16:34:54.110103+0000","last_fullsized":"2024-04-23T16:34:54.110103+0000","mapping_epoch":359,"log_start":"370'3100","ondisk_log_start":"370'3100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12750","last_scrub_stamp":"2024-04-23T16:34:44.348550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:44.348550+0000","objects_scrubbed":6711,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:02.747435+0000","scrub_duration":1024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410583040,"num_objects":6265,"num_object_clones":0,"num_object_copies":12530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13175,"num_write_kb":622080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1091,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:51.848084+0000","last_change":"2024-04-23T16:34:51.848140+0000","last_active":"2024-04-23T16:34:51.848084+0000","last_peered":"2024-04-23T16:34:51.848084+0000","last_clean":"2024-04-23T16:34:51.848084+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:34:51.848084+0000","last_undegraded":"2024-04-23T16:34:51.848084+0000","last_fullsized":"2024-04-23T16:34:51.848084+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:51.847992+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:34:51.847992+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:18.910354+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.2","version":"415'13051","reported_seq":13353,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:54.537840+0000","last_change":"2024-04-23T16:33:57.879648+0000","last_active":"2024-04-23T16:34:54.537840+0000","last_peered":"2024-04-23T16:34:54.537840+0000","last_clean":"2024-04-23T16:34:54.537840+0000","last_became_active":"2024-04-23T16:31:29.138430+0000","last_became_peered":"2024-04-23T16:31:29.138430+0000","last_unstale":"2024-04-23T16:34:54.537840+0000","last_undegraded":"2024-04-23T16:34:54.537840+0000","last_fullsized":"2024-04-23T16:34:54.537840+0000","mapping_epoch":359,"log_start":"370'3000","ondisk_log_start":"370'3000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'10726","last_scrub_stamp":"2024-04-23T16:33:57.879516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:57.879516+0000","objects_scrubbed":8303,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:02.097116+0000","scrub_duration":1621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389611520,"num_objects":5945,"num_object_clones":0,"num_object_copies":11890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13051,"num_write_kb":607872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1091,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:46.193287+0000","last_change":"2024-04-23T16:34:37.693617+0000","last_active":"2024-04-23T16:34:46.193287+0000","last_peered":"2024-04-23T16:34:46.193287+0000","last_clean":"2024-04-23T16:34:46.193287+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:34:46.193287+0000","last_undegraded":"2024-04-23T16:34:46.193287+0000","last_fullsized":"2024-04-23T16:34:46.193287+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:37.693482+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:37.693482+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:49.920873+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"415'15370","reported_seq":15798,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:54.088070+0000","last_change":"2024-04-23T16:34:40.604268+0000","last_active":"2024-04-23T16:34:54.088070+0000","last_peered":"2024-04-23T16:34:54.088070+0000","last_clean":"2024-04-23T16:34:54.088070+0000","last_became_active":"2024-04-23T16:31:29.139398+0000","last_became_peered":"2024-04-23T16:31:29.139398+0000","last_unstale":"2024-04-23T16:34:54.088070+0000","last_undegraded":"2024-04-23T16:34:54.088070+0000","last_fullsized":"2024-04-23T16:34:54.088070+0000","mapping_epoch":359,"log_start":"374'5300","ondisk_log_start":"374'5300","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'14807","last_scrub_stamp":"2024-04-23T16:34:40.604136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:40.604136+0000","objects_scrubbed":6625,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:04.702557+0000","scrub_duration":1231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395313152,"num_objects":6032,"num_object_clones":0,"num_object_copies":12064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13096,"num_write_kb":612096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1084,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:46.198571+0000","last_change":"2024-04-23T16:34:04.444103+0000","last_active":"2024-04-23T16:34:46.198571+0000","last_peered":"2024-04-23T16:34:46.198571+0000","last_clean":"2024-04-23T16:34:46.198571+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:34:46.198571+0000","last_undegraded":"2024-04-23T16:34:46.198571+0000","last_fullsized":"2024-04-23T16:34:46.198571+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:04.443975+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:04.443975+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:05.109970+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"415'13019","reported_seq":13232,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:54.322321+0000","last_change":"2024-04-23T16:32:43.947374+0000","last_active":"2024-04-23T16:34:54.322321+0000","last_peered":"2024-04-23T16:34:54.322321+0000","last_clean":"2024-04-23T16:34:54.322321+0000","last_became_active":"2024-04-23T16:31:29.138968+0000","last_became_peered":"2024-04-23T16:31:29.138968+0000","last_unstale":"2024-04-23T16:34:54.322321+0000","last_undegraded":"2024-04-23T16:34:54.322321+0000","last_fullsized":"2024-04-23T16:34:54.322321+0000","mapping_epoch":359,"log_start":"370'3000","ondisk_log_start":"370'3000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8482","last_scrub_stamp":"2024-04-23T16:32:43.947177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.947177+0000","objects_scrubbed":8428,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385024000,"num_objects":5875,"num_object_clones":0,"num_object_copies":11750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13019,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":983,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:51.248851+0000","last_change":"2024-04-23T16:34:51.248895+0000","last_active":"2024-04-23T16:34:51.248851+0000","last_peered":"2024-04-23T16:34:51.248851+0000","last_clean":"2024-04-23T16:34:51.248851+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:34:51.248851+0000","last_undegraded":"2024-04-23T16:34:51.248851+0000","last_fullsized":"2024-04-23T16:34:51.248851+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:51.248783+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:51.248783+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:13.233910+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"415'13334","reported_seq":13546,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:54.298861+0000","last_change":"2024-04-23T16:32:39.103230+0000","last_active":"2024-04-23T16:34:54.298861+0000","last_peered":"2024-04-23T16:34:54.298861+0000","last_clean":"2024-04-23T16:34:54.298861+0000","last_became_active":"2024-04-23T16:31:29.310246+0000","last_became_peered":"2024-04-23T16:31:29.310246+0000","last_unstale":"2024-04-23T16:34:54.298861+0000","last_undegraded":"2024-04-23T16:34:54.298861+0000","last_fullsized":"2024-04-23T16:34:54.298861+0000","mapping_epoch":359,"log_start":"370'3300","ondisk_log_start":"370'3300","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8325","last_scrub_stamp":"2024-04-23T16:32:39.103059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:39.103059+0000","objects_scrubbed":8272,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401604608,"num_objects":6128,"num_object_clones":0,"num_object_copies":12256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13334,"num_write_kb":622784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":896,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:46.198385+0000","last_change":"2024-04-23T16:33:26.153079+0000","last_active":"2024-04-23T16:34:46.198385+0000","last_peered":"2024-04-23T16:34:46.198385+0000","last_clean":"2024-04-23T16:34:46.198385+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:34:46.198385+0000","last_undegraded":"2024-04-23T16:34:46.198385+0000","last_fullsized":"2024-04-23T16:34:46.198385+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:26.152962+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:33:26.152962+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:34:53.292007+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"415'13158","reported_seq":20583,"reported_epoch":415,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:34:54.428570+0000","last_change":"2024-04-23T16:33:26.132487+0000","last_active":"2024-04-23T16:34:54.428570+0000","last_peered":"2024-04-23T16:34:54.428570+0000","last_clean":"2024-04-23T16:32:59.695133+0000","last_became_active":"2024-04-23T16:33:26.078062+0000","last_became_peered":"2024-04-23T16:33:26.078062+0000","last_unstale":"2024-04-23T16:34:54.428570+0000","last_undegraded":"2024-04-23T16:33:25.892009+0000","last_fullsized":"2024-04-23T16:34:54.428570+0000","mapping_epoch":394,"log_start":"370'3100","ondisk_log_start":"370'3100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'7968","last_scrub_stamp":"2024-04-23T16:32:36.102122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:36.102122+0000","objects_scrubbed":7926,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390332416,"num_objects":5956,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13158,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3605,"num_bytes_recovered":236257280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":5948},{"shards":"5,8","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1081,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-04-23T16:34:46.222860+0000","last_change":"2024-04-23T16:33:17.354409+0000","last_active":"2024-04-23T16:34:46.222860+0000","last_peered":"2024-04-23T16:34:46.222860+0000","last_clean":"2024-04-23T16:34:46.222860+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:34:46.222860+0000","last_undegraded":"2024-04-23T16:34:46.222860+0000","last_fullsized":"2024-04-23T16:34:46.222860+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:17.354263+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.354263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":6382813208,"num_objects":97395,"num_object_clones":0,"num_object_copies":194790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97395,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":209797,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10703,"num_bytes_recovered":701431808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13443801088,"data_stored":13443792944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160943,"ondisk_log_size":160943,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056875,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1283708,"kb_used_data":783284,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":92461444,"statfs":{"total":95995035648,"available":94680518656,"internally_reserved":0,"allocated":802082816,"data_stored":801692649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":512424388},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351216,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612855,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1491192,"kb_used_data":1167088,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92253960,"statfs":{"total":95995035648,"available":94468055040,"internally_reserved":0,"allocated":1195098112,"data_stored":1194712017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":331872704},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841787,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":992760,"kb_used_data":794992,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":92752392,"statfs":{"total":95995035648,"available":94978449408,"internally_reserved":0,"allocated":814071808,"data_stored":813685713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168831,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":766072,"kb_used_data":401712,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":92979080,"statfs":{"total":95995035648,"available":95210577920,"internally_reserved":0,"allocated":411353088,"data_stored":410966993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":373094858},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463174,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1419704,"kb_used_data":1153904,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92325448,"statfs":{"total":95995035648,"available":94541258752,"internally_reserved":0,"allocated":1181597696,"data_stored":1181211601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":272169414},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137313,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738659,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3156984,"kb_used_data":2728752,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":90588168,"statfs":{"total":95995035648,"available":92762284032,"internally_reserved":0,"allocated":2794242048,"data_stored":2793855953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117267,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758720,"kb_used_data":402680,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92986432,"statfs":{"total":95995035648,"available":95218106368,"internally_reserved":0,"allocated":412344320,"data_stored":411954673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":364575175},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921038,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2452920,"kb_used_data":1970544,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":91292232,"statfs":{"total":95995035648,"available":93483245568,"internally_reserved":0,"allocated":2017837056,"data_stored":2017450961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":493943237},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757513,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2326788,"kb_used_data":1934652,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":91418364,"statfs":{"total":95995035648,"available":93612404736,"internally_reserved":0,"allocated":1981083648,"data_stored":1980689929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2779840512,"data_stored":2779840512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1886846976,"data_stored":1886846976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":410583040,"data_stored":410583040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2003632128,"data_stored":2003632128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1978011648,"data_stored":1978007576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1178730496,"data_stored":1178730496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":408158208,"data_stored":408158208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":807862272,"data_stored":807862272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1192099840,"data_stored":1192099840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":798035968,"data_stored":798031896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:34:57.908 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.a is not active+clean 2024-04-23T16:34:57.908 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.a', 'version': "415'13200", 'reported_seq': 20380, 'reported_epoch': 415, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:34:53.054767+0000', 'last_change': '2024-04-23T16:33:26.432192+0000', 'last_active': '2024-04-23T16:34:53.054767+0000', 'last_peered': '2024-04-23T16:34:53.054767+0000', 'last_clean': '2024-04-23T16:32:59.749530+0000', 'last_became_active': '2024-04-23T16:33:26.338644+0000', 'last_became_peered': '2024-04-23T16:33:26.338644+0000', 'last_unstale': '2024-04-23T16:34:53.054767+0000', 'last_undegraded': '2024-04-23T16:33:25.874847+0000', 'last_fullsized': '2024-04-23T16:34:53.054767+0000', 'mapping_epoch': 394, 'log_start': "370'3100", 'ondisk_log_start': "370'3100", 'created': 359, 'last_epoch_clean': 360, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "382'8711", 'last_scrub_stamp': '2024-04-23T16:32:43.928914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:31:28.121358+0000', 'last_clean_scrub_stamp': '2024-04-23T16:32:43.928914+0000', 'objects_scrubbed': 8680, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1273, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 409075712, 'num_objects': 6242, 'num_object_clones': 0, 'num_object_copies': 12484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 6232, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6242, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13200, 'num_write_kb': 622144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3480, 'num_bytes_recovered': 228065280, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [8, 4], 'acting': [8, 4], 'avail_no_missing': ['8'], 'object_location_counts': [{'shards': '4,8', 'objects': 10}, {'shards': '8', 'objects': 6232}], 'blocked_by': [], 'up_primary': 8, 'acting_primary': 8, 'purged_snaps': []} 2024-04-23T16:34:57.908 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.e is not active+clean 2024-04-23T16:34:57.908 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.e', 'version': "415'13954", 'reported_seq': 14217, 'reported_epoch': 415, 'state': 'active+clean+scrubbing+deep', 'last_fresh': '2024-04-23T16:34:54.095340+0000', 'last_change': '2024-04-23T16:34:48.708083+0000', 'last_active': '2024-04-23T16:34:54.095340+0000', 'last_peered': '2024-04-23T16:34:54.095340+0000', 'last_clean': '2024-04-23T16:34:54.095340+0000', 'last_became_active': '2024-04-23T16:31:29.310906+0000', 'last_became_peered': '2024-04-23T16:31:29.310906+0000', 'last_unstale': '2024-04-23T16:34:54.095340+0000', 'last_undegraded': '2024-04-23T16:34:54.095340+0000', 'last_fullsized': '2024-04-23T16:34:54.095340+0000', 'mapping_epoch': 359, 'log_start': "371'3902", 'ondisk_log_start': "371'3902", 'created': 359, 'last_epoch_clean': 360, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "387'9559", 'last_scrub_stamp': '2024-04-23T16:33:22.102857+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:31:28.121358+0000', 'last_clean_scrub_stamp': '2024-04-23T16:33:22.102857+0000', 'objects_scrubbed': 874, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'deep scrubbing for 5s', 'scrub_duration': 758, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 395837440, 'num_objects': 6040, 'num_object_clones': 0, 'num_object_copies': 12080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6040, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13080, 'num_write_kb': 611840, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9, 8], 'acting': [9, 8], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2024-04-23T16:34:57.908 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.6 is not active+clean 2024-04-23T16:34:57.908 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.6', 'version': "415'13305", 'reported_seq': 20760, 'reported_epoch': 415, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:34:54.418356+0000', 'last_change': '2024-04-23T16:33:26.507399+0000', 'last_active': '2024-04-23T16:34:54.418356+0000', 'last_peered': '2024-04-23T16:34:54.418356+0000', 'last_clean': '2024-04-23T16:32:59.715575+0000', 'last_became_active': '2024-04-23T16:33:26.458611+0000', 'last_became_peered': '2024-04-23T16:33:26.458611+0000', 'last_unstale': '2024-04-23T16:34:54.418356+0000', 'last_undegraded': '2024-04-23T16:33:25.895472+0000', 'last_fullsized': '2024-04-23T16:34:54.418356+0000', 'mapping_epoch': 394, 'log_start': "370'3300", 'ondisk_log_start': "370'3300", 'created': 359, 'last_epoch_clean': 360, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "382'8607", 'last_scrub_stamp': '2024-04-23T16:32:43.130806+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:31:28.121358+0000', 'last_clean_scrub_stamp': '2024-04-23T16:32:43.130806+0000', 'objects_scrubbed': 8577, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1484, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 398131200, 'num_objects': 6075, 'num_object_clones': 0, 'num_object_copies': 12150, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 6069, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6075, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13305, 'num_write_kb': 620160, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3618, 'num_bytes_recovered': 237109248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': ['5'], 'object_location_counts': [{'shards': '5', 'objects': 6069}, {'shards': '5,7', 'objects': 6}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-23T16:34:57.908 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.4 is not active+clean 2024-04-23T16:34:57.908 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.4', 'version': "415'13158", 'reported_seq': 20583, 'reported_epoch': 415, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:34:54.428570+0000', 'last_change': '2024-04-23T16:33:26.132487+0000', 'last_active': '2024-04-23T16:34:54.428570+0000', 'last_peered': '2024-04-23T16:34:54.428570+0000', 'last_clean': '2024-04-23T16:32:59.695133+0000', 'last_became_active': '2024-04-23T16:33:26.078062+0000', 'last_became_peered': '2024-04-23T16:33:26.078062+0000', 'last_unstale': '2024-04-23T16:34:54.428570+0000', 'last_undegraded': '2024-04-23T16:33:25.892009+0000', 'last_fullsized': '2024-04-23T16:34:54.428570+0000', 'mapping_epoch': 394, 'log_start': "370'3100", 'ondisk_log_start': "370'3100", 'created': 359, 'last_epoch_clean': 360, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "380'7968", 'last_scrub_stamp': '2024-04-23T16:32:36.102122+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:31:28.121358+0000', 'last_clean_scrub_stamp': '2024-04-23T16:32:36.102122+0000', 'objects_scrubbed': 7926, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1466, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 390332416, 'num_objects': 5956, 'num_object_clones': 0, 'num_object_copies': 11912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 5948, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13158, 'num_write_kb': 611648, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3605, 'num_bytes_recovered': 236257280, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 8], 'acting': [5, 8], 'avail_no_missing': ['5'], 'object_location_counts': [{'shards': '5', 'objects': 5948}, {'shards': '5,8', 'objects': 8}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-23T16:34:57.909 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-23T16:34:57.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:57 smithi188 ceph-mon[44101]: 1.5 scrub starts 2024-04-23T16:34:57.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:57 smithi188 ceph-mon[44101]: 1.5 scrub ok 2024-04-23T16:34:57.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:57 smithi188 ceph-mon[44101]: pgmap v1176: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.9 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 625 op/s; 18249/194352 objects degraded (9.390%); 7.3 MiB/s, 117 objects/s recovering 2024-04-23T16:34:57.974 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:57 smithi103 ceph-mon[39629]: 1.5 scrub starts 2024-04-23T16:34:57.974 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:57 smithi103 ceph-mon[39629]: 1.5 scrub ok 2024-04-23T16:34:57.974 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:57 smithi103 ceph-mon[39629]: pgmap v1176: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.9 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 625 op/s; 18249/194352 objects degraded (9.390%); 7.3 MiB/s, 117 objects/s recovering 2024-04-23T16:34:58.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:57 smithi148 ceph-mon[44520]: 1.5 scrub starts 2024-04-23T16:34:58.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:57 smithi148 ceph-mon[44520]: 1.5 scrub ok 2024-04-23T16:34:58.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:57 smithi148 ceph-mon[44520]: pgmap v1176: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.9 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 625 op/s; 18249/194352 objects degraded (9.390%); 7.3 MiB/s, 117 objects/s recovering 2024-04-23T16:34:58.785 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:34:58.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:58.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:34:58.893 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:34:58.922 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:59.178 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:34:59.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:59.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:34:59.282 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:34:59.316 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:59.473 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:34:59.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:59.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:34:59.577 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:34:59.609 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:34:59.618 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:34:59.618 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:34:59.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:34:59.619 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-23T16:34:59.619 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:34:59.859 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:59.860 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:34:59.865 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:34:59.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:34:59.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:34:59.970 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:35:00.005 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:00.036 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:35:00.037 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:59 smithi188 ceph-mon[44101]: pgmap v1177: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.9 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 598 op/s; 18035/193152 objects degraded (9.337%); 7.6 MiB/s, 120 objects/s recovering 2024-04-23T16:35:00.038 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:59 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 18035/193152 objects degraded (9.337%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:00.038 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:34:59 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3845107351' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:35:00.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:00.039 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:35:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:59 smithi103 ceph-mon[39629]: pgmap v1177: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.9 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 598 op/s; 18035/193152 objects degraded (9.337%); 7.6 MiB/s, 120 objects/s recovering 2024-04-23T16:35:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:59 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 18035/193152 objects degraded (9.337%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:34:59 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3845107351' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:35:00.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:59 smithi148 ceph-mon[44520]: pgmap v1177: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.9 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 598 op/s; 18035/193152 objects degraded (9.337%); 7.6 MiB/s, 120 objects/s recovering 2024-04-23T16:35:00.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:59 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 18035/193152 objects degraded (9.337%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:00.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:34:59 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3845107351' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:35:00.139 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:00.166 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:00.181 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:35:00.425 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-23T16:35:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:00 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:35:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:00 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3845107351' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:35:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:00 smithi103 ceph-mon[39629]: osdmap e416: 12 total, 11 up, 9 in 2024-04-23T16:35:01.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:00 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:35:01.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:00 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3845107351' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:35:01.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:00 smithi148 ceph-mon[44520]: osdmap e416: 12 total, 11 up, 9 in 2024-04-23T16:35:01.150 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:01.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:01.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:01.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:00 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:35:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:00 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3845107351' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:35:01.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:00 smithi188 ceph-mon[44101]: osdmap e416: 12 total, 11 up, 9 in 2024-04-23T16:35:01.254 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:35:01.301 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:01.336 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:35:01.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:01.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:35:01.444 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:35:01.493 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:01.533 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:35:01.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:01.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:35:01.642 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:35:01.672 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:01 smithi103 ceph-mon[39629]: pgmap v1179: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 671 op/s; 17630/187460 objects degraded (9.405%); 7.6 MiB/s, 121 objects/s recovering 2024-04-23T16:35:02.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:01 smithi148 ceph-mon[44520]: pgmap v1179: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 671 op/s; 17630/187460 objects degraded (9.405%); 7.6 MiB/s, 121 objects/s recovering 2024-04-23T16:35:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:01 smithi188 ceph-mon[44101]: pgmap v1179: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 671 op/s; 17630/187460 objects degraded (9.405%); 7.6 MiB/s, 121 objects/s recovering 2024-04-23T16:35:03.858 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:35:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:03 smithi103 ceph-mon[39629]: pgmap v1180: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 668 op/s; 17630/186640 objects degraded (9.446%); 7.6 MiB/s, 121 objects/s recovering 2024-04-23T16:35:04.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:03 smithi148 ceph-mon[44520]: pgmap v1180: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 668 op/s; 17630/186640 objects degraded (9.446%); 7.6 MiB/s, 121 objects/s recovering 2024-04-23T16:35:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:03 smithi188 ceph-mon[44101]: pgmap v1180: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 668 op/s; 17630/186640 objects degraded (9.446%); 7.6 MiB/s, 121 objects/s recovering 2024-04-23T16:35:05.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:04 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 17630/186640 objects degraded (9.446%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:05.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:04 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 17630/186640 objects degraded (9.446%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:04 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 17630/186640 objects degraded (9.446%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:05.764 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:35:05.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:05.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:35:05.869 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:35:05.953 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:06.009 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:35:06.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:06.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:35:06.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:05 smithi103 ceph-mon[39629]: pgmap v1181: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 681 op/s; 16989/181154 objects degraded (9.378%); 7.9 MiB/s, 125 objects/s recovering 2024-04-23T16:35:06.121 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:35:06.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:05 smithi148 ceph-mon[44520]: pgmap v1181: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 681 op/s; 16989/181154 objects degraded (9.378%); 7.9 MiB/s, 125 objects/s recovering 2024-04-23T16:35:06.158 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:06.195 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:35:06.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:06.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:35:06.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:05 smithi188 ceph-mon[44101]: pgmap v1181: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 681 op/s; 16989/181154 objects degraded (9.378%); 7.9 MiB/s, 125 objects/s recovering 2024-04-23T16:35:06.305 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:35:06.327 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:35:06.342 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:06.349 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:35:06.596 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:35:06.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:06.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:35:06.700 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:35:06.733 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:06.763 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:35:06.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:06.767 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:35:06.868 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:35:06.938 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:06.968 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:35:06.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:06.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:35:07.073 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:35:07.110 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:07.137 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:35:07.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:07.142 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:35:07.243 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:35:07.321 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:07.361 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:35:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:07.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:35:07.465 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:35:07.505 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:07.534 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:35:07.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:07.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:35:07.642 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:35:07.712 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:07.762 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:35:07.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:07.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:35:07.872 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:35:07.904 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:07.916 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:35:07.916 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16989/180720 objects degraded (9.401%), 3 pgs degraded","count":3},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1489,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":416,"num_osds":12,"num_up_osds":11,"osd_up_since":1713889979,"num_in_osds":9,"osd_in_since":1713890004,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":23},{"state_name":"active+recovering+degraded","count":3},{"state_name":"active+clean+scrubbing+deep","count":1}],"num_pgs":27,"num_pools":2,"num_objects":90360,"data_bytes":5922226744,"bytes_used":14220259328,"bytes_avail":849735061504,"bytes_total":863955320832,"degraded_objects":16989,"degraded_total":180720,"degraded_ratio":0.094007304116865864,"recovering_objects_per_sec":125,"recovering_bytes_per_sec":8255886,"recovering_keys_per_sec":0,"num_objects_recovered":1260,"num_bytes_recovered":82575360,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":681},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.103:8443/"}},"servicemap":{"epoch":32,"modified":"2024-04-23T16:32:56.701867+0000","services":{}},"progress_events":{}} 2024-04-23T16:35:07.937 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:35:07.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:07.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:35:08.042 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:35:08.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:07 smithi103 ceph-mon[39629]: pgmap v1182: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 681 op/s; 16989/180720 objects degraded (9.401%); 7.9 MiB/s, 125 objects/s recovering 2024-04-23T16:35:08.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:07 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1935056546' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:35:08.073 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:08.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:07 smithi148 ceph-mon[44520]: pgmap v1182: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 681 op/s; 16989/180720 objects degraded (9.401%); 7.9 MiB/s, 125 objects/s recovering 2024-04-23T16:35:08.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:07 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1935056546' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:35:08.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:07 smithi188 ceph-mon[44101]: pgmap v1182: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 681 op/s; 16989/180720 objects degraded (9.401%); 7.9 MiB/s, 125 objects/s recovering 2024-04-23T16:35:08.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:07 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1935056546' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:35:08.274 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:35:08.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:08.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:35:08.379 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:35:08.412 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:08.436 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:35:08.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:08.439 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:35:08.540 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:08.569 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:09.359 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-23T16:35:09.360 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:35:09.947 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:09 smithi103 ceph-mon[39629]: pgmap v1183: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 682 op/s; 16780/179492 objects degraded (9.349%); 7.8 MiB/s, 125 objects/s recovering 2024-04-23T16:35:09.947 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:09 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 16780/179492 objects degraded (9.349%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:10.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:09 smithi148 ceph-mon[44520]: pgmap v1183: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 682 op/s; 16780/179492 objects degraded (9.349%); 7.8 MiB/s, 125 objects/s recovering 2024-04-23T16:35:10.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:09 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 16780/179492 objects degraded (9.349%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:10.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:09 smithi188 ceph-mon[44101]: pgmap v1183: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 682 op/s; 16780/179492 objects degraded (9.349%); 7.8 MiB/s, 125 objects/s recovering 2024-04-23T16:35:10.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:09 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 16780/179492 objects degraded (9.349%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:10.521 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:10.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:10.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:10.626 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:35:10.661 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:10.696 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:35:10.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:10.699 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:35:10.800 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:35:10.833 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:10 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4273919190' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:35:10.833 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:10 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1855562453' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:35:10.834 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:10.838 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:35:10.848 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:35:10.858 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:35:10.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:10.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:35:10.867 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:35:10.893 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:35:10.931 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:35:10.957 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:35:10.963 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:35:10.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:10 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4273919190' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:35:10.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:10 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1855562453' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:35:10.981 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:35:10.997 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:11.004 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:35:11.022 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:35:11.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:11.025 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:35:11.026 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:35:11.049 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:35:11.070 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:35:11.093 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:35:11.094 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:35:11.094 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:35:11.094 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:35:11.094 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:35:11.094 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:35:11.094 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:35:11.094 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:35:11.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:35:11.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:35:11.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:35:11.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:35:11.095 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:35:11.126 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:35:11.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:10 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4273919190' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:35:11.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:10 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1855562453' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:35:11.203 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:11.239 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:35:11.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:11.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:35:11.345 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:11.376 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:12.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:11 smithi103 ceph-mon[39629]: pgmap v1184: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 646 op/s; 16343/173812 objects degraded (9.403%); 7.6 MiB/s, 121 objects/s recovering 2024-04-23T16:35:12.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:11 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:35:12.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:11 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4273919190' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:35:12.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:11 smithi103 ceph-mon[39629]: osdmap e417: 12 total, 11 up, 9 in 2024-04-23T16:35:12.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:11 smithi148 ceph-mon[44520]: pgmap v1184: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 646 op/s; 16343/173812 objects degraded (9.403%); 7.6 MiB/s, 121 objects/s recovering 2024-04-23T16:35:12.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:11 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:35:12.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:11 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4273919190' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:35:12.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:11 smithi148 ceph-mon[44520]: osdmap e417: 12 total, 11 up, 9 in 2024-04-23T16:35:12.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:11 smithi188 ceph-mon[44101]: pgmap v1184: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 646 op/s; 16343/173812 objects degraded (9.403%); 7.6 MiB/s, 121 objects/s recovering 2024-04-23T16:35:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:11 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:35:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:11 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4273919190' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:35:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:11 smithi188 ceph-mon[44101]: osdmap e417: 12 total, 11 up, 9 in 2024-04-23T16:35:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:12.888 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:35:12.889 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:35:13.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:13.127 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:13.128 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:35:13.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:13.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:13.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:13.232 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:35:13.291 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:13.330 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:35:13.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:13.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:35:13.436 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:35:13.466 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:13.667 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:35:13.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:13.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:35:13.771 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:35:13.803 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:13.829 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:35:13.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:13.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:35:13.933 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:35:13.970 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:13.998 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:35:14.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:14.005 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:35:14.106 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:35:14.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:13 smithi148 ceph-mon[44520]: pgmap v1186: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 688 op/s; 16343/172872 objects degraded (9.454%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:35:14.134 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:13 smithi188 ceph-mon[44101]: pgmap v1186: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 688 op/s; 16343/172872 objects degraded (9.454%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:35:14.136 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:14.158 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:35:14.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:14.164 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:35:14.195 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:13 smithi103 ceph-mon[39629]: pgmap v1186: 27 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+degraded, 23 active+clean; 5.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 688 op/s; 16343/172872 objects degraded (9.454%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:35:14.265 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:35:14.272 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:35:14.272 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:35:14.295 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:14.318 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:35:14.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:14.321 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:35:14.422 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:35:14.505 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:14.545 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:35:14.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:14.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:35:14.656 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:35:14.687 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:14.839 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:35:14.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:14.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:35:14.890 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:35:14.942 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:35:14.995 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:14 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:35:14.995 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:14 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:35:14.995 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:14 smithi103 ceph-mon[39629]: osdmap e418: 12 total, 11 up, 9 in 2024-04-23T16:35:14.995 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:14 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 16343/172872 objects degraded (9.454%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:15.010 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:15.047 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:35:15.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:15.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:35:15.153 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:35:15.206 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:14 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:35:15.206 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:14 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:35:15.206 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:14 smithi148 ceph-mon[44520]: osdmap e418: 12 total, 11 up, 9 in 2024-04-23T16:35:15.206 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:14 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 16343/172872 objects degraded (9.454%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:15.208 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:15.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:14 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:35:15.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:14 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:35:15.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:14 smithi188 ceph-mon[44101]: osdmap e418: 12 total, 11 up, 9 in 2024-04-23T16:35:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:14 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 16343/172872 objects degraded (9.454%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:15.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:15.270 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:35:15.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:35:15.371 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:35:15.400 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:15.427 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:35:15.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:15.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:35:15.533 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:35:15.610 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:15.632 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:35:15.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:15.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:35:15.680 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:35:15.736 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:35:15.766 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:15.785 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:35:15.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:15.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:35:15.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:15 smithi148 ceph-mon[44520]: pgmap v1188: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 5.1 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 861 op/s; 15651/166932 objects degraded (9.376%); 10 MiB/s, 167 objects/s recovering 2024-04-23T16:35:15.889 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:35:15.925 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:15.945 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:35:15.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:15.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:35:16.046 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:35:16.049 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:35:16.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:15 smithi188 ceph-mon[44101]: pgmap v1188: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 5.1 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 861 op/s; 15651/166932 objects degraded (9.376%); 10 MiB/s, 167 objects/s recovering 2024-04-23T16:35:16.078 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:16.084 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-23T16:35:16.097 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:35:16.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:16.100 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:35:16.201 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:35:16.256 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:16.298 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:35:16.302 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:15 smithi103 ceph-mon[39629]: pgmap v1188: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 5.1 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 861 op/s; 15651/166932 objects degraded (9.376%); 10 MiB/s, 167 objects/s recovering 2024-04-23T16:35:16.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:16.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:35:16.407 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:16.446 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:17.000 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:17.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:17.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:17.110 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:17.149 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:17.431 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:35:17.431 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:35:17.698 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:35:17.700 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:35:18.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:17 smithi103 ceph-mon[39629]: pgmap v1189: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 5.1 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 809 op/s; 15651/166536 objects degraded (9.398%); 8.8 MiB/s, 141 objects/s recovering 2024-04-23T16:35:18.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:17 smithi188 ceph-mon[44101]: pgmap v1189: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 5.1 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 809 op/s; 15651/166536 objects degraded (9.398%); 8.8 MiB/s, 141 objects/s recovering 2024-04-23T16:35:18.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:17 smithi148 ceph-mon[44520]: pgmap v1189: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 5.1 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 809 op/s; 15651/166536 objects degraded (9.398%); 8.8 MiB/s, 141 objects/s recovering 2024-04-23T16:35:19.149 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: from='client.25996 v1:172.21.15.103:0/3261209047' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:35:19.149 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=3192112479,config=2679774077,health=943580129,kv=665022891}) 2024-04-23T16:35:19.149 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {kv=9,logm=91} crc {kv=2030345658,logm=1756349974}) 2024-04-23T16:35:19.149 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3792922279}) 2024-04-23T16:35:19.149 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=329232663}) 2024-04-23T16:35:19.149 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1606548029}) 2024-04-23T16:35:19.149 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3174500840}) 2024-04-23T16:35:19.149 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2806377849}) 2024-04-23T16:35:19.150 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4207117881}) 2024-04-23T16:35:19.150 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=454428391}) 2024-04-23T16:35:19.150 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=795063541}) 2024-04-23T16:35:19.150 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3968081113}) 2024-04-23T16:35:19.150 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1119863315}) 2024-04-23T16:35:19.150 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2454627350}) 2024-04-23T16:35:19.150 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=962555964}) 2024-04-23T16:35:19.150 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2279827236}) 2024-04-23T16:35:19.150 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326754634}) 2024-04-23T16:35:19.150 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2843033377}) 2024-04-23T16:35:19.150 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2926511881}) 2024-04-23T16:35:19.150 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2153698373}) 2024-04-23T16:35:19.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=265830398}) 2024-04-23T16:35:19.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1008919019}) 2024-04-23T16:35:19.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1310019641}) 2024-04-23T16:35:19.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=545296216}) 2024-04-23T16:35:19.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2975333353}) 2024-04-23T16:35:19.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2270841425}) 2024-04-23T16:35:19.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3477933634}) 2024-04-23T16:35:19.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1049013322}) 2024-04-23T16:35:19.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4073793842}) 2024-04-23T16:35:19.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3395898877}) 2024-04-23T16:35:19.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1458337227}) 2024-04-23T16:35:19.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1507895768}) 2024-04-23T16:35:19.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1332924962}) 2024-04-23T16:35:19.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=113923232}) 2024-04-23T16:35:19.152 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2822164979}) 2024-04-23T16:35:19.152 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=168409992}) 2024-04-23T16:35:19.152 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=858226094}) 2024-04-23T16:35:19.152 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=64960910}) 2024-04-23T16:35:19.152 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1390187457}) 2024-04-23T16:35:19.152 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3815102280}) 2024-04-23T16:35:19.152 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3300026439}) 2024-04-23T16:35:19.152 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=182191104}) 2024-04-23T16:35:19.152 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1327308058}) 2024-04-23T16:35:19.152 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=368048965}) 2024-04-23T16:35:19.152 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4142331559}) 2024-04-23T16:35:19.152 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1054754199}) 2024-04-23T16:35:19.152 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2841210306}) 2024-04-23T16:35:19.153 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659913238}) 2024-04-23T16:35:19.153 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=484741841}) 2024-04-23T16:35:19.153 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3894074509}) 2024-04-23T16:35:19.153 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2965969305}) 2024-04-23T16:35:19.153 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921634651}) 2024-04-23T16:35:19.153 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2562255321}) 2024-04-23T16:35:19.153 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=547611368}) 2024-04-23T16:35:19.153 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3290647758}) 2024-04-23T16:35:19.153 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1006307700}) 2024-04-23T16:35:19.153 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=81,mdsmap=3,mgr=16} crc {logm=691619855,mdsmap=2840952586,mgr=3385795300}) 2024-04-23T16:35:19.153 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=77} crc {mgr=3426076356,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=111544576,mon_config_key=3994978317}) 2024-04-23T16:35:19.153 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3635558236}) 2024-04-23T16:35:19.153 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1103377695}) 2024-04-23T16:35:19.153 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=10,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=72} crc {mon_config_key=1811538665,monmap=2133572193,osd_metadata=3355936916,osd_pg_creating=3128308235,osdmap=266474419}) 2024-04-23T16:35:19.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=39} crc {osdmap=2967473235}) 2024-04-23T16:35:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: from='client.25996 v1:172.21.15.103:0/3261209047' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:35:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=3192112479,config=2679774077,health=943580129,kv=665022891}) 2024-04-23T16:35:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {kv=9,logm=91} crc {kv=2030345658,logm=1756349974}) 2024-04-23T16:35:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3792922279}) 2024-04-23T16:35:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=329232663}) 2024-04-23T16:35:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1606548029}) 2024-04-23T16:35:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3174500840}) 2024-04-23T16:35:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2806377849}) 2024-04-23T16:35:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4207117881}) 2024-04-23T16:35:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=454428391}) 2024-04-23T16:35:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=795063541}) 2024-04-23T16:35:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3968081113}) 2024-04-23T16:35:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1119863315}) 2024-04-23T16:35:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2454627350}) 2024-04-23T16:35:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=962555964}) 2024-04-23T16:35:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2279827236}) 2024-04-23T16:35:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326754634}) 2024-04-23T16:35:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2843033377}) 2024-04-23T16:35:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2926511881}) 2024-04-23T16:35:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2153698373}) 2024-04-23T16:35:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=265830398}) 2024-04-23T16:35:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1008919019}) 2024-04-23T16:35:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1310019641}) 2024-04-23T16:35:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=545296216}) 2024-04-23T16:35:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2975333353}) 2024-04-23T16:35:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2270841425}) 2024-04-23T16:35:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3477933634}) 2024-04-23T16:35:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1049013322}) 2024-04-23T16:35:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4073793842}) 2024-04-23T16:35:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3395898877}) 2024-04-23T16:35:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1458337227}) 2024-04-23T16:35:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1507895768}) 2024-04-23T16:35:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1332924962}) 2024-04-23T16:35:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=113923232}) 2024-04-23T16:35:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2822164979}) 2024-04-23T16:35:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=168409992}) 2024-04-23T16:35:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=858226094}) 2024-04-23T16:35:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=64960910}) 2024-04-23T16:35:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1390187457}) 2024-04-23T16:35:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3815102280}) 2024-04-23T16:35:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3300026439}) 2024-04-23T16:35:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=182191104}) 2024-04-23T16:35:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1327308058}) 2024-04-23T16:35:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=368048965}) 2024-04-23T16:35:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4142331559}) 2024-04-23T16:35:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1054754199}) 2024-04-23T16:35:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2841210306}) 2024-04-23T16:35:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659913238}) 2024-04-23T16:35:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=484741841}) 2024-04-23T16:35:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3894074509}) 2024-04-23T16:35:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2965969305}) 2024-04-23T16:35:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921634651}) 2024-04-23T16:35:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2562255321}) 2024-04-23T16:35:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=547611368}) 2024-04-23T16:35:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3290647758}) 2024-04-23T16:35:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1006307700}) 2024-04-23T16:35:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=81,mdsmap=3,mgr=16} crc {logm=691619855,mdsmap=2840952586,mgr=3385795300}) 2024-04-23T16:35:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=77} crc {mgr=3426076356,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=111544576,mon_config_key=3994978317}) 2024-04-23T16:35:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3635558236}) 2024-04-23T16:35:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1103377695}) 2024-04-23T16:35:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=10,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=72} crc {mon_config_key=1811538665,monmap=2133572193,osd_metadata=3355936916,osd_pg_creating=3128308235,osdmap=266474419}) 2024-04-23T16:35:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=39} crc {osdmap=2967473235}) 2024-04-23T16:35:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: from='client.25996 v1:172.21.15.103:0/3261209047' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:35:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=3192112479,config=2679774077,health=943580129,kv=665022891}) 2024-04-23T16:35:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {kv=9,logm=91} crc {kv=2030345658,logm=1756349974}) 2024-04-23T16:35:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3792922279}) 2024-04-23T16:35:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=329232663}) 2024-04-23T16:35:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1606548029}) 2024-04-23T16:35:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3174500840}) 2024-04-23T16:35:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2806377849}) 2024-04-23T16:35:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4207117881}) 2024-04-23T16:35:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=454428391}) 2024-04-23T16:35:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=795063541}) 2024-04-23T16:35:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3968081113}) 2024-04-23T16:35:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1119863315}) 2024-04-23T16:35:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2454627350}) 2024-04-23T16:35:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=962555964}) 2024-04-23T16:35:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2279827236}) 2024-04-23T16:35:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326754634}) 2024-04-23T16:35:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2843033377}) 2024-04-23T16:35:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2926511881}) 2024-04-23T16:35:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2153698373}) 2024-04-23T16:35:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=265830398}) 2024-04-23T16:35:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1008919019}) 2024-04-23T16:35:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1310019641}) 2024-04-23T16:35:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=545296216}) 2024-04-23T16:35:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2975333353}) 2024-04-23T16:35:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2270841425}) 2024-04-23T16:35:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3477933634}) 2024-04-23T16:35:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1049013322}) 2024-04-23T16:35:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4073793842}) 2024-04-23T16:35:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3395898877}) 2024-04-23T16:35:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1458337227}) 2024-04-23T16:35:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1507895768}) 2024-04-23T16:35:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1332924962}) 2024-04-23T16:35:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=113923232}) 2024-04-23T16:35:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2822164979}) 2024-04-23T16:35:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=168409992}) 2024-04-23T16:35:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=858226094}) 2024-04-23T16:35:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=64960910}) 2024-04-23T16:35:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1390187457}) 2024-04-23T16:35:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3815102280}) 2024-04-23T16:35:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3300026439}) 2024-04-23T16:35:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=182191104}) 2024-04-23T16:35:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1327308058}) 2024-04-23T16:35:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=368048965}) 2024-04-23T16:35:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4142331559}) 2024-04-23T16:35:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1054754199}) 2024-04-23T16:35:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2841210306}) 2024-04-23T16:35:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659913238}) 2024-04-23T16:35:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=484741841}) 2024-04-23T16:35:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3894074509}) 2024-04-23T16:35:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2965969305}) 2024-04-23T16:35:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921634651}) 2024-04-23T16:35:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2562255321}) 2024-04-23T16:35:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=547611368}) 2024-04-23T16:35:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3290647758}) 2024-04-23T16:35:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1006307700}) 2024-04-23T16:35:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=81,mdsmap=3,mgr=16} crc {logm=691619855,mdsmap=2840952586,mgr=3385795300}) 2024-04-23T16:35:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=77} crc {mgr=3426076356,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=111544576,mon_config_key=3994978317}) 2024-04-23T16:35:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3635558236}) 2024-04-23T16:35:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1103377695}) 2024-04-23T16:35:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=10,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=72} crc {mon_config_key=1811538665,monmap=2133572193,osd_metadata=3355936916,osd_pg_creating=3128308235,osdmap=266474419}) 2024-04-23T16:35:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=39} crc {osdmap=2967473235}) 2024-04-23T16:35:19.402 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:19.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:19.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:19.506 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:35:19.537 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:19.562 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:35:19.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:19.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:35:19.666 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:35:19.725 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:19.752 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:35:19.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:19.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:35:19.859 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:19.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:19 smithi188 ceph-mon[44101]: pgmap v1190: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 5.0 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 532 op/s; 15436/165292 objects degraded (9.339%); 7.1 MiB/s, 113 objects/s recovering 2024-04-23T16:35:19.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:19 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 15436/165292 objects degraded (9.339%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:19.890 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:19.910 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:19.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:19.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:20.030 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:35:20.072 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:19 smithi148 ceph-mon[44520]: pgmap v1190: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 5.0 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 532 op/s; 15436/165292 objects degraded (9.339%); 7.1 MiB/s, 113 objects/s recovering 2024-04-23T16:35:20.072 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:19 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 15436/165292 objects degraded (9.339%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:20.074 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:20.103 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:35:20.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:20.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:35:20.213 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:20.238 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:19 smithi103 ceph-mon[39629]: pgmap v1190: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 5.0 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 532 op/s; 15436/165292 objects degraded (9.339%); 7.1 MiB/s, 113 objects/s recovering 2024-04-23T16:35:20.238 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:19 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 15436/165292 objects degraded (9.339%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:20.240 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:20.991 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:20.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:20.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:21.096 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:35:21.150 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:21.203 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:35:21.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:21.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:35:21.308 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:35:21.309 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:35:21.334 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:35:21.342 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:21.357 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:35:21.388 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:35:21.422 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:35:21.444 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:35:21.473 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:35:21.494 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:35:21.520 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:35:21.548 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:35:21.570 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:35:21.570 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:35:21.570 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:35:21.570 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:35:21.570 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:35:21.570 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:35:21.570 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:35:21.570 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:35:21.570 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:35:21.570 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:35:21.571 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:35:21.571 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:35:21.571 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:35:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:21 smithi103 ceph-mon[39629]: pgmap v1191: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.9 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 735 op/s; 14939/159254 objects degraded (9.381%); 8.9 MiB/s, 141 objects/s recovering 2024-04-23T16:35:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:21 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1548979310' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:35:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:22.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:22.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:21 smithi188 ceph-mon[44101]: pgmap v1191: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.9 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 735 op/s; 14939/159254 objects degraded (9.381%); 8.9 MiB/s, 141 objects/s recovering 2024-04-23T16:35:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:21 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1548979310' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:35:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:22.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:21 smithi148 ceph-mon[44520]: pgmap v1191: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.9 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 735 op/s; 14939/159254 objects degraded (9.381%); 8.9 MiB/s, 141 objects/s recovering 2024-04-23T16:35:22.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:21 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1548979310' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:35:22.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:22.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:22.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:22.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:22.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:22.988 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1189,"stamp":"2024-04-23T16:35:16.730909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5457445432,"num_objects":83268,"num_object_clones":0,"num_object_copies":166536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15651,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83268,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":223926,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13302,"num_bytes_recovered":871895584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162386,"ondisk_log_size":162386,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":13159604,"kb_used_data":9714776,"kb_used_omap":14,"kb_used_meta":3444401,"kb_avail":830546764,"statfs":{"total":863955320832,"available":850479886336,"internally_reserved":0,"allocated":9947930624,"data_stored":9944850710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3527067631},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-424542208,"num_objects":-6478,"num_object_clones":0,"num_object_copies":-12956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6478,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1129,"num_bytes_recovered":73990144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001494"},"pg_stats":[{"pgid":"5.f","version":"418'14145","reported_seq":14376,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:14.642861+0000","last_change":"2024-04-23T16:33:17.526419+0000","last_active":"2024-04-23T16:35:14.642861+0000","last_peered":"2024-04-23T16:35:14.642861+0000","last_clean":"2024-04-23T16:35:14.642861+0000","last_became_active":"2024-04-23T16:31:29.140923+0000","last_became_peered":"2024-04-23T16:31:29.140923+0000","last_unstale":"2024-04-23T16:35:14.642861+0000","last_undegraded":"2024-04-23T16:35:14.642861+0000","last_fullsized":"2024-04-23T16:35:14.642861+0000","mapping_epoch":359,"log_start":"371'4100","ondisk_log_start":"371'4100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9683","last_scrub_stamp":"2024-04-23T16:33:17.526283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.526283+0000","objects_scrubbed":9683,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342163456,"num_objects":5221,"num_object_clones":0,"num_object_copies":10442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14145,"num_write_kb":619712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"418'13995","reported_seq":14298,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:14.266014+0000","last_change":"2024-04-23T16:34:02.531358+0000","last_active":"2024-04-23T16:35:14.266014+0000","last_peered":"2024-04-23T16:35:14.266014+0000","last_clean":"2024-04-23T16:35:14.266014+0000","last_became_active":"2024-04-23T16:31:29.137890+0000","last_became_peered":"2024-04-23T16:31:29.137890+0000","last_unstale":"2024-04-23T16:35:14.266014+0000","last_undegraded":"2024-04-23T16:35:14.266014+0000","last_fullsized":"2024-04-23T16:35:14.266014+0000","mapping_epoch":359,"log_start":"371'3900","ondisk_log_start":"371'3900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'10975","last_scrub_stamp":"2024-04-23T16:34:02.531162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:02.531162+0000","objects_scrubbed":8314,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:17.999122+0000","scrub_duration":1066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345309184,"num_objects":5269,"num_object_clones":0,"num_object_copies":10538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13995,"num_write_kb":616448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"417'14077","reported_seq":23012,"reported_epoch":417,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:35:13.127336+0000","last_change":"2024-04-23T16:33:26.432192+0000","last_active":"2024-04-23T16:35:13.127336+0000","last_peered":"2024-04-23T16:35:13.127336+0000","last_clean":"2024-04-23T16:32:59.749530+0000","last_became_active":"2024-04-23T16:33:26.338644+0000","last_became_peered":"2024-04-23T16:33:26.338644+0000","last_unstale":"2024-04-23T16:35:13.127336+0000","last_undegraded":"2024-04-23T16:33:25.874847+0000","last_fullsized":"2024-04-23T16:35:13.127336+0000","mapping_epoch":394,"log_start":"371'4000","ondisk_log_start":"371'4000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8711","last_scrub_stamp":"2024-04-23T16:32:43.928914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.928914+0000","objects_scrubbed":8680,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351600640,"num_objects":5365,"num_object_clones":0,"num_object_copies":10730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14077,"num_write_kb":622144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4356,"num_bytes_recovered":285474816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":["8"],"object_location_counts":[{"shards":"4,8","objects":9},{"shards":"8","objects":5356}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.9","version":"418'14184","reported_seq":14400,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:15.050807+0000","last_change":"2024-04-23T16:32:37.256944+0000","last_active":"2024-04-23T16:35:15.050807+0000","last_peered":"2024-04-23T16:35:15.050807+0000","last_clean":"2024-04-23T16:35:15.050807+0000","last_became_active":"2024-04-23T16:31:29.141551+0000","last_became_peered":"2024-04-23T16:31:29.141551+0000","last_unstale":"2024-04-23T16:35:15.050807+0000","last_undegraded":"2024-04-23T16:35:15.050807+0000","last_fullsized":"2024-04-23T16:35:15.050807+0000","mapping_epoch":359,"log_start":"371'4100","ondisk_log_start":"371'4100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8156","last_scrub_stamp":"2024-04-23T16:32:37.256787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:37.256787+0000","objects_scrubbed":8082,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336592896,"num_objects":5136,"num_object_clones":0,"num_object_copies":10272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14184,"num_write_kb":618240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"417'13556","reported_seq":13856,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-04-23T16:35:11.735397+0000","last_change":"2024-04-23T16:34:39.934502+0000","last_active":"2024-04-23T16:35:11.735397+0000","last_peered":"2024-04-23T16:35:11.735397+0000","last_clean":"2024-04-23T16:35:11.735397+0000","last_became_active":"2024-04-23T16:31:29.139045+0000","last_became_peered":"2024-04-23T16:31:29.139045+0000","last_unstale":"2024-04-23T16:35:11.735397+0000","last_undegraded":"2024-04-23T16:35:11.735397+0000","last_fullsized":"2024-04-23T16:35:11.735397+0000","mapping_epoch":359,"log_start":"370'3500","ondisk_log_start":"370'3500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'12219","last_scrub_stamp":"2024-04-23T16:34:39.934190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:39.934190+0000","objects_scrubbed":6529,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:59.649413+0000","scrub_duration":810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339345408,"num_objects":5178,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13556,"num_write_kb":599488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"418'17549","reported_seq":17928,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:14.276979+0000","last_change":"2024-04-23T16:35:10.892898+0000","last_active":"2024-04-23T16:35:14.276979+0000","last_peered":"2024-04-23T16:35:14.276979+0000","last_clean":"2024-04-23T16:35:14.276979+0000","last_became_active":"2024-04-23T16:31:29.310906+0000","last_became_peered":"2024-04-23T16:31:29.310906+0000","last_unstale":"2024-04-23T16:35:14.276979+0000","last_undegraded":"2024-04-23T16:35:14.276979+0000","last_fullsized":"2024-04-23T16:35:14.276979+0000","mapping_epoch":359,"log_start":"380'7504","ondisk_log_start":"380'7504","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9559","last_scrub_stamp":"2024-04-23T16:33:22.102857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:22.102857+0000","objects_scrubbed":3561,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:15.884411+0000","scrub_duration":758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336330752,"num_objects":5132,"num_object_clones":0,"num_object_copies":10264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13988,"num_write_kb":611840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1098,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:13.865701+0000","last_change":"2024-04-23T16:34:44.725424+0000","last_active":"2024-04-23T16:35:13.865701+0000","last_peered":"2024-04-23T16:35:13.865701+0000","last_clean":"2024-04-23T16:35:13.865701+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:35:13.865701+0000","last_undegraded":"2024-04-23T16:35:13.865701+0000","last_fullsized":"2024-04-23T16:35:13.865701+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:44.725324+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:44.725324+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:14.224827+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"418'14196","reported_seq":14413,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:14.395063+0000","last_change":"2024-04-23T16:32:38.103756+0000","last_active":"2024-04-23T16:35:14.395063+0000","last_peered":"2024-04-23T16:35:14.395063+0000","last_clean":"2024-04-23T16:35:14.395063+0000","last_became_active":"2024-04-23T16:31:29.141640+0000","last_became_peered":"2024-04-23T16:31:29.141640+0000","last_unstale":"2024-04-23T16:35:14.395063+0000","last_undegraded":"2024-04-23T16:35:14.395063+0000","last_fullsized":"2024-04-23T16:35:14.395063+0000","mapping_epoch":359,"log_start":"371'4100","ondisk_log_start":"371'4100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8300","last_scrub_stamp":"2024-04-23T16:32:38.103547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:38.103547+0000","objects_scrubbed":8231,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344719360,"num_objects":5260,"num_object_clones":0,"num_object_copies":10520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14196,"num_write_kb":622592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1092,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:13.868150+0000","last_change":"2024-04-23T16:33:25.851379+0000","last_active":"2024-04-23T16:35:13.868150+0000","last_peered":"2024-04-23T16:35:13.868150+0000","last_clean":"2024-04-23T16:35:13.868150+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:35:13.868150+0000","last_undegraded":"2024-04-23T16:35:13.868150+0000","last_fullsized":"2024-04-23T16:35:13.868150+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.498820+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.498820+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"417'13808","reported_seq":14112,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-04-23T16:35:11.752393+0000","last_change":"2024-04-23T16:34:41.375046+0000","last_active":"2024-04-23T16:35:11.752393+0000","last_peered":"2024-04-23T16:35:11.752393+0000","last_clean":"2024-04-23T16:35:11.752393+0000","last_became_active":"2024-04-23T16:31:29.140251+0000","last_became_peered":"2024-04-23T16:31:29.140251+0000","last_unstale":"2024-04-23T16:35:11.752393+0000","last_undegraded":"2024-04-23T16:35:11.752393+0000","last_fullsized":"2024-04-23T16:35:11.752393+0000","mapping_epoch":359,"log_start":"371'3800","ondisk_log_start":"371'3800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12511","last_scrub_stamp":"2024-04-23T16:34:41.374888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:41.374888+0000","objects_scrubbed":6643,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:07.291855+0000","scrub_duration":1271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348848152,"num_objects":5324,"num_object_clones":0,"num_object_copies":10648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5324,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13808,"num_write_kb":612161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1083,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:13.865674+0000","last_change":"2024-04-23T16:32:29.517947+0000","last_active":"2024-04-23T16:35:13.865674+0000","last_peered":"2024-04-23T16:35:13.865674+0000","last_clean":"2024-04-23T16:35:13.865674+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:35:13.865674+0000","last_undegraded":"2024-04-23T16:35:13.865674+0000","last_fullsized":"2024-04-23T16:35:13.865674+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:32:29.517800+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:32:29.517800+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"418'14193","reported_seq":23430,"reported_epoch":418,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:35:14.636653+0000","last_change":"2024-04-23T16:33:26.507399+0000","last_active":"2024-04-23T16:35:14.636653+0000","last_peered":"2024-04-23T16:35:14.636653+0000","last_clean":"2024-04-23T16:32:59.715575+0000","last_became_active":"2024-04-23T16:33:26.458611+0000","last_became_peered":"2024-04-23T16:33:26.458611+0000","last_unstale":"2024-04-23T16:35:14.636653+0000","last_undegraded":"2024-04-23T16:33:25.895472+0000","last_fullsized":"2024-04-23T16:35:14.636653+0000","mapping_epoch":394,"log_start":"371'4100","ondisk_log_start":"371'4100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8607","last_scrub_stamp":"2024-04-23T16:32:43.130806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.130806+0000","objects_scrubbed":8577,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339935232,"num_objects":5187,"num_object_clones":0,"num_object_copies":10374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14193,"num_write_kb":620160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4505,"num_bytes_recovered":295239680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":5182},{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1097,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:13.860639+0000","last_change":"2024-04-23T16:34:47.438492+0000","last_active":"2024-04-23T16:35:13.860639+0000","last_peered":"2024-04-23T16:35:13.860639+0000","last_clean":"2024-04-23T16:35:13.860639+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:35:13.860639+0000","last_undegraded":"2024-04-23T16:35:13.860639+0000","last_fullsized":"2024-04-23T16:35:13.860639+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:47.438356+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:47.438356+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:13.051838+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"418'13806","reported_seq":14035,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:14.613933+0000","last_change":"2024-04-23T16:33:18.576077+0000","last_active":"2024-04-23T16:35:14.613933+0000","last_peered":"2024-04-23T16:35:14.613933+0000","last_clean":"2024-04-23T16:35:14.613933+0000","last_became_active":"2024-04-23T16:31:29.139709+0000","last_became_peered":"2024-04-23T16:31:29.139709+0000","last_unstale":"2024-04-23T16:35:14.613933+0000","last_undegraded":"2024-04-23T16:35:14.613933+0000","last_fullsized":"2024-04-23T16:35:14.613933+0000","mapping_epoch":359,"log_start":"371'3800","ondisk_log_start":"371'3800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9472","last_scrub_stamp":"2024-04-23T16:33:18.575945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.575945+0000","objects_scrubbed":9472,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336723968,"num_objects":5138,"num_object_clones":0,"num_object_copies":10276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13806,"num_write_kb":606208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1090,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:13.866429+0000","last_change":"2024-04-23T16:34:02.399102+0000","last_active":"2024-04-23T16:35:13.866429+0000","last_peered":"2024-04-23T16:35:13.866429+0000","last_clean":"2024-04-23T16:35:13.866429+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:35:13.866429+0000","last_undegraded":"2024-04-23T16:35:13.866429+0000","last_fullsized":"2024-04-23T16:35:13.866429+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:02.398975+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:02.398975+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:23.945801+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"418'14057","reported_seq":14366,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:14.079236+0000","last_change":"2024-04-23T16:34:44.348696+0000","last_active":"2024-04-23T16:35:14.079236+0000","last_peered":"2024-04-23T16:35:14.079236+0000","last_clean":"2024-04-23T16:35:14.079236+0000","last_became_active":"2024-04-23T16:31:29.139606+0000","last_became_peered":"2024-04-23T16:31:29.139606+0000","last_unstale":"2024-04-23T16:35:14.079236+0000","last_undegraded":"2024-04-23T16:35:14.079236+0000","last_fullsized":"2024-04-23T16:35:14.079236+0000","mapping_epoch":359,"log_start":"371'4000","ondisk_log_start":"371'4000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12750","last_scrub_stamp":"2024-04-23T16:34:44.348550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:44.348550+0000","objects_scrubbed":6711,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:02.747435+0000","scrub_duration":1024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352780288,"num_objects":5383,"num_object_clones":0,"num_object_copies":10766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14057,"num_write_kb":622080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1097,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:13.865772+0000","last_change":"2024-04-23T16:34:51.848140+0000","last_active":"2024-04-23T16:35:13.865772+0000","last_peered":"2024-04-23T16:35:13.865772+0000","last_clean":"2024-04-23T16:35:13.865772+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:35:13.865772+0000","last_undegraded":"2024-04-23T16:35:13.865772+0000","last_fullsized":"2024-04-23T16:35:13.865772+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:51.847992+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:34:51.847992+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:18.910354+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.2","version":"418'13904","reported_seq":14212,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:14.563686+0000","last_change":"2024-04-23T16:33:57.879648+0000","last_active":"2024-04-23T16:35:14.563686+0000","last_peered":"2024-04-23T16:35:14.563686+0000","last_clean":"2024-04-23T16:35:14.563686+0000","last_became_active":"2024-04-23T16:31:29.138430+0000","last_became_peered":"2024-04-23T16:31:29.138430+0000","last_unstale":"2024-04-23T16:35:14.563686+0000","last_undegraded":"2024-04-23T16:35:14.563686+0000","last_fullsized":"2024-04-23T16:35:14.563686+0000","mapping_epoch":359,"log_start":"371'3900","ondisk_log_start":"371'3900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'10726","last_scrub_stamp":"2024-04-23T16:33:57.879516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:57.879516+0000","objects_scrubbed":8303,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:14.660441+0000","scrub_duration":1621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333709312,"num_objects":5092,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13904,"num_write_kb":607872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1098,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:13.865978+0000","last_change":"2024-04-23T16:34:37.693617+0000","last_active":"2024-04-23T16:35:13.865978+0000","last_peered":"2024-04-23T16:35:13.865978+0000","last_clean":"2024-04-23T16:35:13.865978+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:35:13.865978+0000","last_undegraded":"2024-04-23T16:35:13.865978+0000","last_fullsized":"2024-04-23T16:35:13.865978+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:37.693482+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:37.693482+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:49.920873+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"418'16226","reported_seq":16661,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:14.054593+0000","last_change":"2024-04-23T16:34:40.604268+0000","last_active":"2024-04-23T16:35:14.054593+0000","last_peered":"2024-04-23T16:35:14.054593+0000","last_clean":"2024-04-23T16:35:14.054593+0000","last_became_active":"2024-04-23T16:31:29.139398+0000","last_became_peered":"2024-04-23T16:31:29.139398+0000","last_unstale":"2024-04-23T16:35:14.054593+0000","last_undegraded":"2024-04-23T16:35:14.054593+0000","last_fullsized":"2024-04-23T16:35:14.054593+0000","mapping_epoch":359,"log_start":"376'6200","ondisk_log_start":"376'6200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'14807","last_scrub_stamp":"2024-04-23T16:34:40.604136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:40.604136+0000","objects_scrubbed":6625,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:04.702557+0000","scrub_duration":1231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339214336,"num_objects":5176,"num_object_clones":0,"num_object_copies":10352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13952,"num_write_kb":612096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1090,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:13.866504+0000","last_change":"2024-04-23T16:34:04.444103+0000","last_active":"2024-04-23T16:35:13.866504+0000","last_peered":"2024-04-23T16:35:13.866504+0000","last_clean":"2024-04-23T16:35:13.866504+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:35:13.866504+0000","last_undegraded":"2024-04-23T16:35:13.866504+0000","last_fullsized":"2024-04-23T16:35:13.866504+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:04.443975+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:04.443975+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:14.660492+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"418'13852","reported_seq":14071,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:14.423094+0000","last_change":"2024-04-23T16:32:43.947374+0000","last_active":"2024-04-23T16:35:14.423094+0000","last_peered":"2024-04-23T16:35:14.423094+0000","last_clean":"2024-04-23T16:35:14.423094+0000","last_became_active":"2024-04-23T16:31:29.138968+0000","last_became_peered":"2024-04-23T16:31:29.138968+0000","last_unstale":"2024-04-23T16:35:14.423094+0000","last_undegraded":"2024-04-23T16:35:14.423094+0000","last_fullsized":"2024-04-23T16:35:14.423094+0000","mapping_epoch":359,"log_start":"371'3800","ondisk_log_start":"371'3800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8482","last_scrub_stamp":"2024-04-23T16:32:43.947177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.947177+0000","objects_scrubbed":8428,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330432512,"num_objects":5042,"num_object_clones":0,"num_object_copies":10084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13852,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":987,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-04-23T16:35:10.849751+0000","last_change":"2024-04-23T16:34:51.248895+0000","last_active":"2024-04-23T16:35:10.849751+0000","last_peered":"2024-04-23T16:35:10.849751+0000","last_clean":"2024-04-23T16:35:10.849751+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:35:10.849751+0000","last_undegraded":"2024-04-23T16:35:10.849751+0000","last_fullsized":"2024-04-23T16:35:10.849751+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:51.248783+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:51.248783+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:13.233910+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"418'14217","reported_seq":14435,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:14.406224+0000","last_change":"2024-04-23T16:32:39.103230+0000","last_active":"2024-04-23T16:35:14.406224+0000","last_peered":"2024-04-23T16:35:14.406224+0000","last_clean":"2024-04-23T16:35:14.406224+0000","last_became_active":"2024-04-23T16:31:29.310246+0000","last_became_peered":"2024-04-23T16:31:29.310246+0000","last_unstale":"2024-04-23T16:35:14.406224+0000","last_undegraded":"2024-04-23T16:35:14.406224+0000","last_fullsized":"2024-04-23T16:35:14.406224+0000","mapping_epoch":359,"log_start":"371'4200","ondisk_log_start":"371'4200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8325","last_scrub_stamp":"2024-04-23T16:32:39.103059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:39.103059+0000","objects_scrubbed":8272,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343736320,"num_objects":5245,"num_object_clones":0,"num_object_copies":10490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14217,"num_write_kb":622784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":911,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:13.866442+0000","last_change":"2024-04-23T16:34:56.453746+0000","last_active":"2024-04-23T16:35:13.866442+0000","last_peered":"2024-04-23T16:35:13.866442+0000","last_clean":"2024-04-23T16:35:13.866442+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:35:13.866442+0000","last_undegraded":"2024-04-23T16:35:13.866442+0000","last_fullsized":"2024-04-23T16:35:13.866442+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:56.453617+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:34:56.453617+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:08.659133+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"418'13996","reported_seq":23097,"reported_epoch":418,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:35:14.660509+0000","last_change":"2024-04-23T16:33:26.132487+0000","last_active":"2024-04-23T16:35:14.660509+0000","last_peered":"2024-04-23T16:35:14.660509+0000","last_clean":"2024-04-23T16:32:59.695133+0000","last_became_active":"2024-04-23T16:33:26.078062+0000","last_became_peered":"2024-04-23T16:33:26.078062+0000","last_unstale":"2024-04-23T16:35:14.660509+0000","last_undegraded":"2024-04-23T16:33:25.892009+0000","last_fullsized":"2024-04-23T16:35:14.660509+0000","mapping_epoch":394,"log_start":"371'3900","ondisk_log_start":"371'3900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'7968","last_scrub_stamp":"2024-04-23T16:32:36.102122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:36.102122+0000","objects_scrubbed":7926,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335413248,"num_objects":5118,"num_object_clones":0,"num_object_copies":10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13996,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4440,"num_bytes_recovered":290979840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":5113},{"shards":"5,8","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1087,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-23T16:35:13.864128+0000","last_change":"2024-04-23T16:33:17.354409+0000","last_active":"2024-04-23T16:35:13.864128+0000","last_peered":"2024-04-23T16:35:13.864128+0000","last_clean":"2024-04-23T16:35:13.864128+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:35:13.864128+0000","last_undegraded":"2024-04-23T16:35:13.864128+0000","last_fullsized":"2024-04-23T16:35:13.864128+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:17.354263+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.354263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":5456855064,"num_objects":83266,"num_object_clones":0,"num_object_copies":166532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15651,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83266,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":223926,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13301,"num_bytes_recovered":871694336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11764572160,"data_stored":11764564016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160857,"ondisk_log_size":160857,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056879,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1184416,"kb_used_data":675668,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92560736,"statfs":{"total":95995035648,"available":94782193664,"internally_reserved":0,"allocated":691884032,"data_stored":691542110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":520944068},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351221,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612859,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1326824,"kb_used_data":998556,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92418328,"statfs":{"total":95995035648,"available":94636367872,"internally_reserved":0,"allocated":1022521344,"data_stored":1022177490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":336132544},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841791,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":948120,"kb_used_data":682320,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92797032,"statfs":{"total":95995035648,"available":95024160768,"internally_reserved":0,"allocated":698695680,"data_stored":698357830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168836,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":691880,"kb_used_data":331676,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93053272,"statfs":{"total":95995035648,"available":95286550528,"internally_reserved":0,"allocated":339636224,"data_stored":339292370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":368835018},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463178,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1286168,"kb_used_data":993936,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92458984,"statfs":{"total":95995035648,"available":94677999616,"internally_reserved":0,"allocated":1017790464,"data_stored":1017452614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":299170246},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137318,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738663,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2777624,"kb_used_data":2345232,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":90967528,"statfs":{"total":95995035648,"available":93150748672,"internally_reserved":0,"allocated":2401517568,"data_stored":2401179718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117272,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":694896,"kb_used_data":334692,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93050256,"statfs":{"total":95995035648,"available":95283462144,"internally_reserved":0,"allocated":342724608,"data_stored":342377202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":368835015},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921042,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2176856,"kb_used_data":1690320,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":91568296,"statfs":{"total":95995035648,"available":93765935104,"internally_reserved":0,"allocated":1730887680,"data_stored":1730549830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757517,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2072820,"kb_used_data":1662376,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":91672332,"statfs":{"total":95995035648,"available":93872467968,"internally_reserved":0,"allocated":1702273024,"data_stored":1701921546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2382692352,"data_stored":2382692352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1886846976,"data_stored":1886846976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":340590592,"data_stored":340590592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1714225152,"data_stored":1714225152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1694371840,"data_stored":1694367768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1008992256,"data_stored":1008992256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":337051648,"data_stored":337051648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":692453376,"data_stored":692453376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1019150336,"data_stored":1019150336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":688197632,"data_stored":688193560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:35:23.078 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:35:23.078 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:35:23.078 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-23T16:35:23.078 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-23T16:35:23.078 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:35:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:23.298 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:35:23.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:23.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:23.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:23.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:23.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:23.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:23.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:23.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:35:23.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:23.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:23.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:23.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:23.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:23.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:23.402 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:35:23.503 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:23.537 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:35:23.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:23.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:35:23.645 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:23.677 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:23.920 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:23 smithi103 ceph-mon[39629]: pgmap v1192: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.8 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 726 op/s; 14939/158332 objects degraded (9.435%); 8.8 MiB/s, 140 objects/s recovering 2024-04-23T16:35:24.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:23 smithi188 ceph-mon[44101]: pgmap v1192: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.8 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 726 op/s; 14939/158332 objects degraded (9.435%); 8.8 MiB/s, 140 objects/s recovering 2024-04-23T16:35:24.226 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:24.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:24.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:24.256 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:35:24.257 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:35:24.333 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:35:24.363 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:24.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:23 smithi148 ceph-mon[44520]: pgmap v1192: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.8 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 726 op/s; 14939/158332 objects degraded (9.435%); 8.8 MiB/s, 140 objects/s recovering 2024-04-23T16:35:24.385 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:35:24.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:24.388 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:35:24.489 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:35:24.518 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:24.542 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:35:24.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:24.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:35:24.646 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:35:24.717 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:24.776 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:35:24.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:24.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:35:24.887 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:35:24.919 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:24.935 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-23T16:35:24.937 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:35:24.947 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:35:24.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:24.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:35:25.050 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:35:25.107 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:24 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3299035802' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:35:25.107 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:24 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 14939/158332 objects degraded (9.435%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:25.117 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:25.163 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:35:25.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:25.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:35:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3299035802' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:35:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:24 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 14939/158332 objects degraded (9.435%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:25.270 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:35:25.297 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:25.326 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:35:25.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:25.328 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:35:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3299035802' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:35:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:24 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 14939/158332 objects degraded (9.435%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:25.430 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:35:25.493 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:25.533 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:35:25.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:25.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:35:25.644 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:35:25.671 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:25.698 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:35:25.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:25.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:35:25.802 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:35:25.858 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:35:25.867 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:25.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:25.913 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:35:25.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:35:25.926 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:25 smithi103 ceph-mon[39629]: pgmap v1193: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.7 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 845 op/s; 14693/154014 objects degraded (9.540%); 8.2 MiB/s, 130 objects/s recovering 2024-04-23T16:35:25.926 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:25 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:35:25.926 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:25 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3299035802' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:35:25.991 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:35:26.014 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:35:26.069 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:26.116 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:35:26.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:26.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:25 smithi148 ceph-mon[44520]: pgmap v1193: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.7 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 845 op/s; 14693/154014 objects degraded (9.540%); 8.2 MiB/s, 130 objects/s recovering 2024-04-23T16:35:26.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:25 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:35:26.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:25 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3299035802' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:35:26.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:25 smithi148 ceph-mon[44520]: osdmap e419: 12 total, 11 up, 9 in 2024-04-23T16:35:26.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:25 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:35:26.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:25 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:35:26.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:35:26.190 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:25 smithi103 ceph-mon[39629]: osdmap e419: 12 total, 11 up, 9 in 2024-04-23T16:35:26.190 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:25 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:35:26.190 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:25 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:35:26.226 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:35:26.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:25 smithi188 ceph-mon[44101]: pgmap v1193: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.7 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 845 op/s; 14693/154014 objects degraded (9.540%); 8.2 MiB/s, 130 objects/s recovering 2024-04-23T16:35:26.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:25 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:35:26.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:25 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3299035802' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:35:26.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:25 smithi188 ceph-mon[44101]: osdmap e419: 12 total, 11 up, 9 in 2024-04-23T16:35:26.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:25 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:35:26.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:25 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:35:26.258 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:35:26.267 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:26.471 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:35:26.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:26.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:35:26.579 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:35:26.607 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:26.631 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:35:26.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:26.635 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:35:26.736 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:35:26.766 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:26.789 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:35:26.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:26.792 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:35:26.865 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:35:26.893 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:26.924 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:27.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:26 smithi188 ceph-mon[44101]: 5.b scrub starts 2024-04-23T16:35:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:26 smithi188 ceph-mon[44101]: osdmap e420: 12 total, 11 up, 9 in 2024-04-23T16:35:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:26 smithi103 ceph-mon[39629]: 5.b scrub starts 2024-04-23T16:35:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:26 smithi103 ceph-mon[39629]: osdmap e420: 12 total, 11 up, 9 in 2024-04-23T16:35:27.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:26 smithi148 ceph-mon[44520]: 5.b scrub starts 2024-04-23T16:35:27.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:26 smithi148 ceph-mon[44520]: osdmap e420: 12 total, 11 up, 9 in 2024-04-23T16:35:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:35:27.733 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:35:27.943 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:27 smithi103 ceph-mon[39629]: 5.b scrub ok 2024-04-23T16:35:27.943 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:27 smithi103 ceph-mon[39629]: pgmap v1196: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.6 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 851 op/s; 14231/151672 objects degraded (9.383%); 9.4 MiB/s, 150 objects/s recovering 2024-04-23T16:35:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:27 smithi188 ceph-mon[44101]: 5.b scrub ok 2024-04-23T16:35:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:27 smithi188 ceph-mon[44101]: pgmap v1196: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.6 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 851 op/s; 14231/151672 objects degraded (9.383%); 9.4 MiB/s, 150 objects/s recovering 2024-04-23T16:35:28.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:27 smithi148 ceph-mon[44520]: 5.b scrub ok 2024-04-23T16:35:28.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:27 smithi148 ceph-mon[44520]: pgmap v1196: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.6 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 851 op/s; 14231/151672 objects degraded (9.383%); 9.4 MiB/s, 150 objects/s recovering 2024-04-23T16:35:28.915 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:35:28.916 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:35:28.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:35:28.916 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:35:28.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.916 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:283079 3.6 3:6de0a032:::benchmark_data_smithi188_89276_object114435:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:35:28.916 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:26:09.398245+0000", 2024-04-23T16:35:28.916 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 559.02268271299999, 2024-04-23T16:35:28.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.20610246900000001, 2024-04-23T16:35:28.916 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:28.916 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:28.916 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:28.916 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:28.916 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:35:28.917 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:35:28.917 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 283079 2024-04-23T16:35:28.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.917 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:28.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:28.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.398245+0000", 2024-04-23T16:35:28.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:28.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.398245+0000", 2024-04-23T16:35:28.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:28.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.398247+0000", 2024-04-23T16:35:28.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1270000000000001e-06 2024-04-23T16:35:28.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:28.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.398253+0000", 2024-04-23T16:35:28.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0699999999999997e-06 2024-04-23T16:35:28.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:28.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.398263+0000", 2024-04-23T16:35:28.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.092e-05 2024-04-23T16:35:28.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:28.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.398271+0000", 2024-04-23T16:35:28.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2910000000000002e-06 2024-04-23T16:35:28.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:28.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.398335+0000", 2024-04-23T16:35:28.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3757000000000006e-05 2024-04-23T16:35:28.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:28.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.398468+0000", 2024-04-23T16:35:28.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013329300000000001 2024-04-23T16:35:28.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:35:28.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.398581+0000", 2024-04-23T16:35:28.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011353699999999999 2024-04-23T16:35:28.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:28.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.402400+0000", 2024-04-23T16:35:28.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038187099999999999 2024-04-23T16:35:28.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:28.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.604235+0000", 2024-04-23T16:35:28.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.20183515699999999 2024-04-23T16:35:28.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:28.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.604297+0000", 2024-04-23T16:35:28.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1927999999999994e-05 2024-04-23T16:35:28.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:28.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.604348+0000", 2024-04-23T16:35:28.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0679e-05 2024-04-23T16:35:28.922 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.922 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:28.922 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.922 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:283080 3.1 3:8f8a07dc:::benchmark_data_smithi188_89276_object114436:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:35:28.922 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:26:09.405845+0000", 2024-04-23T16:35:28.922 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 559.01508352500002, 2024-04-23T16:35:28.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.214733119, 2024-04-23T16:35:28.922 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:28.923 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:28.923 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:28.923 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:28.923 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:35:28.923 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:35:28.923 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 283080 2024-04-23T16:35:28.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.923 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:28.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:28.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.405845+0000", 2024-04-23T16:35:28.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:28.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.405845+0000", 2024-04-23T16:35:28.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:28.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.405847+0000", 2024-04-23T16:35:28.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2010000000000002e-06 2024-04-23T16:35:28.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:28.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.405856+0000", 2024-04-23T16:35:28.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3659999999999998e-06 2024-04-23T16:35:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.405867+0000", 2024-04-23T16:35:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0502999999999999e-05 2024-04-23T16:35:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.405876+0000", 2024-04-23T16:35:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8160000000000005e-06 2024-04-23T16:35:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.405925+0000", 2024-04-23T16:35:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8241000000000001e-05 2024-04-23T16:35:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.406014+0000", 2024-04-23T16:35:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9475000000000006e-05 2024-04-23T16:35:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:35:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.406091+0000", 2024-04-23T16:35:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6990000000000007e-05 2024-04-23T16:35:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:28.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.409685+0000", 2024-04-23T16:35:28.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003593391 2024-04-23T16:35:28.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:28.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.620506+0000", 2024-04-23T16:35:28.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.21082145899999999 2024-04-23T16:35:28.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:28.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.620542+0000", 2024-04-23T16:35:28.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6276000000000003e-05 2024-04-23T16:35:28.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:28.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.620578+0000", 2024-04-23T16:35:28.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5401000000000002e-05 2024-04-23T16:35:28.928 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.928 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:28.928 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.928 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:283084 3.6 3:6829dcbb:::benchmark_data_smithi188_89276_object114440:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:35:28.928 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:26:09.409973+0000", 2024-04-23T16:35:28.928 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 559.01095477000001, 2024-04-23T16:35:28.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.19903251799999999, 2024-04-23T16:35:28.928 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:28.929 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:28.929 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:28.929 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:28.929 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:35:28.929 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:35:28.929 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 283084 2024-04-23T16:35:28.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.929 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:28.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:28.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.409973+0000", 2024-04-23T16:35:28.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:28.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.409973+0000", 2024-04-23T16:35:28.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:28.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.409976+0000", 2024-04-23T16:35:28.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0230000000000001e-06 2024-04-23T16:35:28.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:28.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.409987+0000", 2024-04-23T16:35:28.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0747000000000001e-05 2024-04-23T16:35:28.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:28.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.410007+0000", 2024-04-23T16:35:28.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9956e-05 2024-04-23T16:35:28.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:28.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.410027+0000", 2024-04-23T16:35:28.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9760000000000001e-05 2024-04-23T16:35:28.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:28.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.410150+0000", 2024-04-23T16:35:28.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012322799999999999 2024-04-23T16:35:28.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:28.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.410292+0000", 2024-04-23T16:35:28.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014207900000000001 2024-04-23T16:35:28.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:35:28.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.410364+0000", 2024-04-23T16:35:28.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1598000000000001e-05 2024-04-23T16:35:28.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:28.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.413904+0000", 2024-04-23T16:35:28.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035407429999999998 2024-04-23T16:35:28.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:28.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.608904+0000", 2024-04-23T16:35:28.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.19499992399999999 2024-04-23T16:35:28.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:28.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.608960+0000", 2024-04-23T16:35:28.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5779999999999998e-05 2024-04-23T16:35:28.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:28.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.609006+0000", 2024-04-23T16:35:28.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5680000000000003e-05 2024-04-23T16:35:28.934 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.934 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:28.934 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.934 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:283090 3.1 3:8f3df6ae:::benchmark_data_smithi188_89276_object114446:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:35:28.934 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:26:09.413270+0000", 2024-04-23T16:35:28.934 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 559.007658021, 2024-04-23T16:35:28.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.20745738799999999, 2024-04-23T16:35:28.934 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:28.935 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:28.935 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:28.935 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:28.935 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:35:28.935 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:35:28.935 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 283090 2024-04-23T16:35:28.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.935 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:28.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:28.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.413270+0000", 2024-04-23T16:35:28.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:28.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.413270+0000", 2024-04-23T16:35:28.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:28.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.413271+0000", 2024-04-23T16:35:28.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2139999999999999e-06 2024-04-23T16:35:28.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:28.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.413279+0000", 2024-04-23T16:35:28.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0499999999999992e-06 2024-04-23T16:35:28.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:28.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.413289+0000", 2024-04-23T16:35:28.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5249999999999994e-06 2024-04-23T16:35:28.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:28.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.413296+0000", 2024-04-23T16:35:28.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.221e-06 2024-04-23T16:35:28.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:28.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.413347+0000", 2024-04-23T16:35:28.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0491000000000002e-05 2024-04-23T16:35:28.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:28.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.413466+0000", 2024-04-23T16:35:28.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011993999999999999 2024-04-23T16:35:28.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:35:28.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.413583+0000", 2024-04-23T16:35:28.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116148 2024-04-23T16:35:28.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:28.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.417663+0000", 2024-04-23T16:35:28.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0040800860000000001 2024-04-23T16:35:28.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:28.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.620631+0000", 2024-04-23T16:35:28.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.20296804399999999 2024-04-23T16:35:28.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:28.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.620684+0000", 2024-04-23T16:35:28.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3229999999999997e-05 2024-04-23T16:35:28.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:28.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.620727+0000", 2024-04-23T16:35:28.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3439000000000002e-05 2024-04-23T16:35:28.940 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.940 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:28.940 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.940 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:283097 3.1 3:882c6406:::benchmark_data_smithi188_89276_object114453:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:35:28.940 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:26:09.421141+0000", 2024-04-23T16:35:28.940 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 558.99978724300001, 2024-04-23T16:35:28.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.19966055899999999, 2024-04-23T16:35:28.940 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:28.940 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:28.940 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:28.941 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:28.941 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:35:28.941 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:35:28.941 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 283097 2024-04-23T16:35:28.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.941 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:28.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:28.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.421141+0000", 2024-04-23T16:35:28.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:28.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.421141+0000", 2024-04-23T16:35:28.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:28.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.421143+0000", 2024-04-23T16:35:28.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.04e-06 2024-04-23T16:35:28.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:28.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.421150+0000", 2024-04-23T16:35:28.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3250000000000002e-06 2024-04-23T16:35:28.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:28.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.421162+0000", 2024-04-23T16:35:28.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1892999999999999e-05 2024-04-23T16:35:28.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:28.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.421172+0000", 2024-04-23T16:35:28.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.825e-06 2024-04-23T16:35:28.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:28.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.421228+0000", 2024-04-23T16:35:28.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5924999999999997e-05 2024-04-23T16:35:28.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:28.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.421324+0000", 2024-04-23T16:35:28.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6674000000000003e-05 2024-04-23T16:35:28.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:35:28.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.421400+0000", 2024-04-23T16:35:28.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5702000000000001e-05 2024-04-23T16:35:28.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:28.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.425085+0000", 2024-04-23T16:35:28.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003685093 2024-04-23T16:35:28.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:28.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.620754+0000", 2024-04-23T16:35:28.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.195668332 2024-04-23T16:35:28.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:28.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.620780+0000", 2024-04-23T16:35:28.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6723000000000001e-05 2024-04-23T16:35:28.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:28.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.620801+0000", 2024-04-23T16:35:28.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1027e-05 2024-04-23T16:35:28.946 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.946 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:28.946 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.946 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:283104 3.1 3:81e836ad:::benchmark_data_smithi188_89276_object114460:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:35:28.946 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:26:09.432672+0000", 2024-04-23T16:35:28.946 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 558.98825605399998, 2024-04-23T16:35:28.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.19185057699999999, 2024-04-23T16:35:28.946 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:28.946 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:28.947 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:28.947 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:28.947 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:35:28.947 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:35:28.947 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 283104 2024-04-23T16:35:28.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.947 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:28.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:28.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.432672+0000", 2024-04-23T16:35:28.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:28.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.432672+0000", 2024-04-23T16:35:28.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:28.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.432683+0000", 2024-04-23T16:35:28.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.06e-05 2024-04-23T16:35:28.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:28.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.432695+0000", 2024-04-23T16:35:28.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2063e-05 2024-04-23T16:35:28.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:28.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.432720+0000", 2024-04-23T16:35:28.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5760000000000001e-05 2024-04-23T16:35:28.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:28.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.432741+0000", 2024-04-23T16:35:28.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0795999999999999e-05 2024-04-23T16:35:28.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:28.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.432848+0000", 2024-04-23T16:35:28.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000106872 2024-04-23T16:35:28.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:28.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.432936+0000", 2024-04-23T16:35:28.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8239999999999995e-05 2024-04-23T16:35:28.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:35:28.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.433009+0000", 2024-04-23T16:35:28.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2947999999999993e-05 2024-04-23T16:35:28.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:28.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.436396+0000", 2024-04-23T16:35:28.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0033869759999999999 2024-04-23T16:35:28.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:28.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.624424+0000", 2024-04-23T16:35:28.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.18802730100000001 2024-04-23T16:35:28.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:28.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.624482+0000", 2024-04-23T16:35:28.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7948999999999997e-05 2024-04-23T16:35:28.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.952 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:28.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.624523+0000", 2024-04-23T16:35:28.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1072000000000002e-05 2024-04-23T16:35:28.952 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.952 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:28.952 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.953 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.15150.0:283106 3.1 3:8e187821:::benchmark_data_smithi188_89276_object114462:head [delete] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-04-23T16:35:28.953 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:26:09.436469+0000", 2024-04-23T16:35:28.953 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 558.984458765, 2024-04-23T16:35:28.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.18819633899999999, 2024-04-23T16:35:28.953 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:28.953 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:28.953 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:28.953 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:28.953 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.15150", 2024-04-23T16:35:28.953 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1530428419", 2024-04-23T16:35:28.953 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 283106 2024-04-23T16:35:28.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.953 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:28.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:28.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.436469+0000", 2024-04-23T16:35:28.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:28.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.436469+0000", 2024-04-23T16:35:28.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:28.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.436470+0000", 2024-04-23T16:35:28.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1909999999999999e-06 2024-04-23T16:35:28.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:28.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.436475+0000", 2024-04-23T16:35:28.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9620000000000001e-06 2024-04-23T16:35:28.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:28.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.436484+0000", 2024-04-23T16:35:28.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6759999999999996e-06 2024-04-23T16:35:28.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:28.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.436492+0000", 2024-04-23T16:35:28.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8809999999999996e-06 2024-04-23T16:35:28.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:28.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.436536+0000", 2024-04-23T16:35:28.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4400000000000002e-05 2024-04-23T16:35:28.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:28.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.436614+0000", 2024-04-23T16:35:28.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7688000000000006e-05 2024-04-23T16:35:28.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:35:28.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.436685+0000", 2024-04-23T16:35:28.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0912000000000001e-05 2024-04-23T16:35:28.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:28.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.439003+0000", 2024-04-23T16:35:28.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023179780000000001 2024-04-23T16:35:28.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:28.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.624582+0000", 2024-04-23T16:35:28.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.1855791 2024-04-23T16:35:28.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:28.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.624629+0000", 2024-04-23T16:35:28.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6928999999999997e-05 2024-04-23T16:35:28.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:28.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:26:09.624666+0000", 2024-04-23T16:35:28.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6622000000000003e-05 2024-04-23T16:35:28.958 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.958 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:28.958 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.958 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.25717.0:146410 5.3 5:c182460c:::benchmark_data_smithi188_113957_object146409:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-23T16:35:28.958 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:32:52.480505+0000", 2024-04-23T16:35:28.958 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 155.94042318000001, 2024-04-23T16:35:28.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.162363859, 2024-04-23T16:35:28.959 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:28.959 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:28.959 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:28.959 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:28.959 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.25717", 2024-04-23T16:35:28.959 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:35:28.959 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 146410 2024-04-23T16:35:28.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.959 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:28.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:28.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.480505+0000", 2024-04-23T16:35:28.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:28.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.480505+0000", 2024-04-23T16:35:28.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:28.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.480506+0000", 2024-04-23T16:35:28.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5400000000000001e-06 2024-04-23T16:35:28.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:28.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.480612+0000", 2024-04-23T16:35:28.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105188 2024-04-23T16:35:28.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:28.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.480625+0000", 2024-04-23T16:35:28.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3679000000000001e-05 2024-04-23T16:35:28.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:28.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.480631+0000", 2024-04-23T16:35:28.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8780000000000003e-06 2024-04-23T16:35:28.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:28.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.608432+0000", 2024-04-23T16:35:28.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.12780108400000001 2024-04-23T16:35:28.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:28.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.608560+0000", 2024-04-23T16:35:28.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012768299999999999 2024-04-23T16:35:28.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6", 2024-04-23T16:35:28.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.608702+0000", 2024-04-23T16:35:28.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014218499999999999 2024-04-23T16:35:28.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:28.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.617278+0000", 2024-04-23T16:35:28.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085761229999999997 2024-04-23T16:35:28.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:28.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.642773+0000", 2024-04-23T16:35:28.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025495203000000001 2024-04-23T16:35:28.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:28.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.642807+0000", 2024-04-23T16:35:28.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4003e-05 2024-04-23T16:35:28.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:28.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.642869+0000", 2024-04-23T16:35:28.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1292999999999997e-05 2024-04-23T16:35:28.963 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.964 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:28.964 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.964 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.25717.0:146626 5.3 5:cc430f87:::benchmark_data_smithi188_113957_object146625:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-23T16:35:28.964 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:32:52.741400+0000", 2024-04-23T16:35:28.964 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 155.679527716, 2024-04-23T16:35:28.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.172108119, 2024-04-23T16:35:28.964 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:28.964 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:28.964 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:28.964 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:28.964 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.25717", 2024-04-23T16:35:28.964 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:35:28.964 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 146626 2024-04-23T16:35:28.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.965 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:28.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:28.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.741400+0000", 2024-04-23T16:35:28.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:28.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.741400+0000", 2024-04-23T16:35:28.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:28.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.741403+0000", 2024-04-23T16:35:28.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2170000000000001e-06 2024-04-23T16:35:28.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:28.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.741447+0000", 2024-04-23T16:35:28.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4708e-05 2024-04-23T16:35:28.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:28.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.741463+0000", 2024-04-23T16:35:28.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5736e-05 2024-04-23T16:35:28.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:28.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.741471+0000", 2024-04-23T16:35:28.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7179999999999996e-06 2024-04-23T16:35:28.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:28.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.792848+0000", 2024-04-23T16:35:28.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.051377080999999998 2024-04-23T16:35:28.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:28.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.792989+0000", 2024-04-23T16:35:28.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000141168 2024-04-23T16:35:28.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6", 2024-04-23T16:35:28.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.793129+0000", 2024-04-23T16:35:28.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014046999999999999 2024-04-23T16:35:28.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:28.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.797726+0000", 2024-04-23T16:35:28.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004596887 2024-04-23T16:35:28.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:28.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.913384+0000", 2024-04-23T16:35:28.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.115658041 2024-04-23T16:35:28.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:28.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.913438+0000", 2024-04-23T16:35:28.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3953999999999999e-05 2024-04-23T16:35:28.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.969 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:28.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:52.913508+0000", 2024-04-23T16:35:28.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0139e-05 2024-04-23T16:35:28.969 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.969 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:28.969 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.969 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.25717.0:151461 5.3 5:ca10fa19:::benchmark_data_smithi188_113957_object151460:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-23T16:35:28.969 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:32:57.597668+0000", 2024-04-23T16:35:28.970 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 150.82325999700001, 2024-04-23T16:35:28.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.18513961900000001, 2024-04-23T16:35:28.970 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:28.970 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:28.970 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:28.970 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:28.970 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.25717", 2024-04-23T16:35:28.970 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:35:28.970 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 151461 2024-04-23T16:35:28.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.970 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:28.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:28.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.597668+0000", 2024-04-23T16:35:28.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:28.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.597668+0000", 2024-04-23T16:35:28.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:28.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.597670+0000", 2024-04-23T16:35:28.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.649e-06 2024-04-23T16:35:28.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:28.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.597724+0000", 2024-04-23T16:35:28.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4496000000000001e-05 2024-04-23T16:35:28.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:28.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.597733+0000", 2024-04-23T16:35:28.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2949999999999999e-06 2024-04-23T16:35:28.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:28.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.597740+0000", 2024-04-23T16:35:28.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4960000000000001e-06 2024-04-23T16:35:28.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.973 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:28.973 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.615635+0000", 2024-04-23T16:35:28.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017894541 2024-04-23T16:35:28.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.973 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:28.973 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.615734+0000", 2024-04-23T16:35:28.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9418000000000003e-05 2024-04-23T16:35:28.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.973 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6", 2024-04-23T16:35:28.973 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.615836+0000", 2024-04-23T16:35:28.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010213900000000001 2024-04-23T16:35:28.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:28.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.782705+0000", 2024-04-23T16:35:28.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.16686922600000001 2024-04-23T16:35:28.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:28.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.782716+0000", 2024-04-23T16:35:28.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0744000000000001e-05 2024-04-23T16:35:28.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:28.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.782752+0000", 2024-04-23T16:35:28.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6306000000000001e-05 2024-04-23T16:35:28.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:28.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.782808+0000", 2024-04-23T16:35:28.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5309000000000002e-05 2024-04-23T16:35:28.975 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.975 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:28.975 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.975 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.25717.0:151472 5.3 5:c2415992:::benchmark_data_smithi188_113957_object151471:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-23T16:35:28.975 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:32:57.606849+0000", 2024-04-23T16:35:28.975 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 150.81407859000001, 2024-04-23T16:35:28.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.17607524799999999, 2024-04-23T16:35:28.975 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:28.975 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:28.975 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:28.976 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:28.976 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.25717", 2024-04-23T16:35:28.976 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:35:28.976 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 151472 2024-04-23T16:35:28.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.976 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:28.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.976 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:28.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.606849+0000", 2024-04-23T16:35:28.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.976 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:28.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.606849+0000", 2024-04-23T16:35:28.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:28.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.606851+0000", 2024-04-23T16:35:28.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6530000000000001e-06 2024-04-23T16:35:28.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:28.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.606915+0000", 2024-04-23T16:35:28.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3578999999999996e-05 2024-04-23T16:35:28.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:28.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.606925+0000", 2024-04-23T16:35:28.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8789999999999998e-06 2024-04-23T16:35:28.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.978 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:28.978 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.606933+0000", 2024-04-23T16:35:28.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9589999999999995e-06 2024-04-23T16:35:28.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.978 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:28.978 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.619472+0000", 2024-04-23T16:35:28.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012538973 2024-04-23T16:35:28.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.978 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:28.978 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.619558+0000", 2024-04-23T16:35:28.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6487000000000001e-05 2024-04-23T16:35:28.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6", 2024-04-23T16:35:28.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.619670+0000", 2024-04-23T16:35:28.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112104 2024-04-23T16:35:28.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:28.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.782816+0000", 2024-04-23T16:35:28.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.16314540599999999 2024-04-23T16:35:28.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:28.980 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.782854+0000", 2024-04-23T16:35:28.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8235999999999998e-05 2024-04-23T16:35:28.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.980 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:28.980 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.782888+0000", 2024-04-23T16:35:28.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4238e-05 2024-04-23T16:35:28.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.980 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:28.980 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.782925+0000", 2024-04-23T16:35:28.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6733999999999998e-05 2024-04-23T16:35:28.980 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.980 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:28.981 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.981 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.981 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.25717.0:151474 5.3 5:ca671c76:::benchmark_data_smithi188_113957_object151473:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-23T16:35:28.981 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:32:57.607026+0000", 2024-04-23T16:35:28.981 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 150.81390236799999, 2024-04-23T16:35:28.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.176028726, 2024-04-23T16:35:28.981 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:28.981 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:28.981 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:28.981 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:28.981 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.25717", 2024-04-23T16:35:28.981 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:35:28.982 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 151474 2024-04-23T16:35:28.982 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.982 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:28.982 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.982 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:28.982 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.607026+0000", 2024-04-23T16:35:28.982 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.982 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.982 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.982 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:28.982 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.607026+0000", 2024-04-23T16:35:28.982 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.983 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:28.983 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.607027+0000", 2024-04-23T16:35:28.983 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7600000000000006e-07 2024-04-23T16:35:28.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.983 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:28.983 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.607104+0000", 2024-04-23T16:35:28.983 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7198000000000002e-05 2024-04-23T16:35:28.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.983 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:28.983 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.607120+0000", 2024-04-23T16:35:28.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6092000000000001e-05 2024-04-23T16:35:28.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.984 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.984 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:28.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.607127+0000", 2024-04-23T16:35:28.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0129999999999996e-06 2024-04-23T16:35:28.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.984 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.984 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:28.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.623156+0000", 2024-04-23T16:35:28.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016029451 2024-04-23T16:35:28.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.984 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.984 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:28.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.623242+0000", 2024-04-23T16:35:28.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6069999999999994e-05 2024-04-23T16:35:28.985 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.985 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.985 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6", 2024-04-23T16:35:28.985 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.623339+0000", 2024-04-23T16:35:28.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6638000000000005e-05 2024-04-23T16:35:28.985 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.985 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.985 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:28.985 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.782820+0000", 2024-04-23T16:35:28.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.15948044 2024-04-23T16:35:28.985 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.985 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.985 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:28.985 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.782960+0000", 2024-04-23T16:35:28.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014083800000000001 2024-04-23T16:35:28.986 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.986 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.986 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:28.986 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.782992+0000", 2024-04-23T16:35:28.986 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1692000000000001e-05 2024-04-23T16:35:28.986 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.986 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.986 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:28.986 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.783054+0000", 2024-04-23T16:35:28.986 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2317999999999995e-05 2024-04-23T16:35:28.986 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.986 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:28.986 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.987 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.25717.0:151483 5.3 5:c96424b6:::benchmark_data_smithi188_113957_object151482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-23T16:35:28.987 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:32:57.615904+0000", 2024-04-23T16:35:28.987 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 150.805024043, 2024-04-23T16:35:28.987 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.16724354299999999, 2024-04-23T16:35:28.987 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:28.987 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:28.987 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:28.987 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:28.987 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.25717", 2024-04-23T16:35:28.987 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:35:28.987 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 151483 2024-04-23T16:35:28.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.987 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:28.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:28.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.615904+0000", 2024-04-23T16:35:28.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:28.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.615904+0000", 2024-04-23T16:35:28.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:28.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.615906+0000", 2024-04-23T16:35:28.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.086e-06 2024-04-23T16:35:28.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:28.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.615956+0000", 2024-04-23T16:35:28.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0219000000000001e-05 2024-04-23T16:35:28.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:28.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.615967+0000", 2024-04-23T16:35:28.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0407e-05 2024-04-23T16:35:28.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:28.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.615974+0000", 2024-04-23T16:35:28.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5789999999999998e-06 2024-04-23T16:35:28.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:28.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.626411+0000", 2024-04-23T16:35:28.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01043635 2024-04-23T16:35:28.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:28.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.626519+0000", 2024-04-23T16:35:28.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108378 2024-04-23T16:35:28.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6", 2024-04-23T16:35:28.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.626617+0000", 2024-04-23T16:35:28.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8350999999999994e-05 2024-04-23T16:35:28.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.991 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:28.991 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.783072+0000", 2024-04-23T16:35:28.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.15645418899999999 2024-04-23T16:35:28.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.991 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:28.991 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.783091+0000", 2024-04-23T16:35:28.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9340000000000001e-05 2024-04-23T16:35:28.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.991 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:28.991 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.783120+0000", 2024-04-23T16:35:28.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9584999999999999e-05 2024-04-23T16:35:28.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:28.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.783148+0000", 2024-04-23T16:35:28.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7059e-05 2024-04-23T16:35:28.992 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.992 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:28.992 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.992 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.25717.0:151494 5.3 5:c022d8eb:::benchmark_data_smithi188_113957_object151493:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-23T16:35:28.992 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:32:57.629251+0000", 2024-04-23T16:35:28.992 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 150.791677357, 2024-04-23T16:35:28.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.183302307, 2024-04-23T16:35:28.992 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:28.992 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:28.992 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:28.993 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:28.993 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.25717", 2024-04-23T16:35:28.993 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:35:28.993 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 151494 2024-04-23T16:35:28.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.993 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:28.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:28.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.629251+0000", 2024-04-23T16:35:28.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:28.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.629251+0000", 2024-04-23T16:35:28.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:28.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.629253+0000", 2024-04-23T16:35:28.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0449999999999999e-06 2024-04-23T16:35:28.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:28.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.629278+0000", 2024-04-23T16:35:28.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5656999999999998e-05 2024-04-23T16:35:28.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:28.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.629288+0000", 2024-04-23T16:35:28.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0079e-05 2024-04-23T16:35:28.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.995 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:28.995 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.629294+0000", 2024-04-23T16:35:28.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4419999999999997e-06 2024-04-23T16:35:28.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.995 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:28.995 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.798886+0000", 2024-04-23T16:35:28.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.16959200799999999 2024-04-23T16:35:28.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.995 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:28.995 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.799025+0000", 2024-04-23T16:35:28.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139562 2024-04-23T16:35:28.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.995 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6", 2024-04-23T16:35:28.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.799195+0000", 2024-04-23T16:35:28.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000169581 2024-04-23T16:35:28.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:28.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.803709+0000", 2024-04-23T16:35:28.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004513665 2024-04-23T16:35:28.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:28.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.812470+0000", 2024-04-23T16:35:28.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0087613359999999998 2024-04-23T16:35:28.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:28.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.812507+0000", 2024-04-23T16:35:28.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7174e-05 2024-04-23T16:35:28.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.997 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.997 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:28.997 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.812553+0000", 2024-04-23T16:35:28.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5757999999999998e-05 2024-04-23T16:35:28.997 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.997 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:28.997 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:28.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.997 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.997 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.25717.0:151527 5.3 5:ce891cd9:::benchmark_data_smithi188_113957_object151526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-23T16:35:28.997 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:32:57.664005+0000", 2024-04-23T16:35:28.997 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 150.75692271099999, 2024-04-23T16:35:28.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.162601992, 2024-04-23T16:35:28.997 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:28.998 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:28.998 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:28.998 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:28.998 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.25717", 2024-04-23T16:35:28.998 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:35:28.998 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 151527 2024-04-23T16:35:28.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.998 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:28.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:28.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.664005+0000", 2024-04-23T16:35:28.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:28.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.664005+0000", 2024-04-23T16:35:28.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:28.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:28.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.664008+0000", 2024-04-23T16:35:28.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2869999999999999e-06 2024-04-23T16:35:28.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:28.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:28.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.664068+0000", 2024-04-23T16:35:28.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9944000000000001e-05 2024-04-23T16:35:28.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:28.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:29.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.664077+0000", 2024-04-23T16:35:29.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7510000000000003e-06 2024-04-23T16:35:29.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:29.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.664083+0000", 2024-04-23T16:35:29.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.728e-06 2024-04-23T16:35:29.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:29.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.807379+0000", 2024-04-23T16:35:29.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.14329619599999999 2024-04-23T16:35:29.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:29.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.807529+0000", 2024-04-23T16:35:29.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014959199999999999 2024-04-23T16:35:29.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6", 2024-04-23T16:35:29.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.807689+0000", 2024-04-23T16:35:29.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000159872 2024-04-23T16:35:29.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:29.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.812658+0000", 2024-04-23T16:35:29.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049690200000000002 2024-04-23T16:35:29.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:29.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.826519+0000", 2024-04-23T16:35:29.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013861210000000001 2024-04-23T16:35:29.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:29.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.826556+0000", 2024-04-23T16:35:29.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6634999999999997e-05 2024-04-23T16:35:29.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:29.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:32:57.826607+0000", 2024-04-23T16:35:29.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1756999999999999e-05 2024-04-23T16:35:29.003 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:29.003 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:29.003 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:29.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.003 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.25717.0:209653 5.5 5:a08b8def:::benchmark_data_smithi188_113957_object56055:head [delete] snapc 0=[] ondisk+write+known_if_redirected e415)", 2024-04-23T16:35:29.003 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:34:53.385025+0000", 2024-04-23T16:35:29.003 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 35.035903073999997, 2024-04-23T16:35:29.003 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.036842371999999998, 2024-04-23T16:35:29.003 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:29.003 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:29.003 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:29.003 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:29.003 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.25717", 2024-04-23T16:35:29.003 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:35:29.003 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 209653 2024-04-23T16:35:29.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.004 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:29.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:29.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.385025+0000", 2024-04-23T16:35:29.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:29.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:29.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.385025+0000", 2024-04-23T16:35:29.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:29.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:29.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.385027+0000", 2024-04-23T16:35:29.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1270000000000001e-06 2024-04-23T16:35:29.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:29.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.385033+0000", 2024-04-23T16:35:29.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.57e-06 2024-04-23T16:35:29.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:29.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.385044+0000", 2024-04-23T16:35:29.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0861e-05 2024-04-23T16:35:29.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:29.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.385051+0000", 2024-04-23T16:35:29.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5660000000000001e-06 2024-04-23T16:35:29.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:29.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.385102+0000", 2024-04-23T16:35:29.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0529999999999999e-05 2024-04-23T16:35:29.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:29.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.385211+0000", 2024-04-23T16:35:29.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109345 2024-04-23T16:35:29.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:35:29.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.385289+0000", 2024-04-23T16:35:29.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7991000000000006e-05 2024-04-23T16:35:29.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.007 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:29.007 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.389920+0000", 2024-04-23T16:35:29.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046313429999999996 2024-04-23T16:35:29.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.007 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:29.007 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.421798+0000", 2024-04-23T16:35:29.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.031877768000000001 2024-04-23T16:35:29.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.007 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:29.007 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.421831+0000", 2024-04-23T16:35:29.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3099999999999998e-05 2024-04-23T16:35:29.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.008 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:29.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.421867+0000", 2024-04-23T16:35:29.008 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6171000000000003e-05 2024-04-23T16:35:29.008 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:29.008 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:29.008 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:29.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.008 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.25717.0:217174 5.3 5:c7c61b6c:::benchmark_data_smithi188_113957_object63576:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-23T16:35:29.008 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:35:04.476386+0000", 2024-04-23T16:35:29.008 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.944541636, 2024-04-23T16:35:29.008 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.032969430000000001, 2024-04-23T16:35:29.009 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:29.009 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:29.009 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:29.009 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:29.009 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.25717", 2024-04-23T16:35:29.009 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:35:29.009 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 217174 2024-04-23T16:35:29.009 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.009 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:29.009 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.009 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:29.009 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:04.476386+0000", 2024-04-23T16:35:29.009 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:29.009 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.009 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.009 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:29.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:04.476386+0000", 2024-04-23T16:35:29.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:29.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.010 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:29.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:04.476390+0000", 2024-04-23T16:35:29.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2600000000000001e-06 2024-04-23T16:35:29.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.010 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:29.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:04.476401+0000", 2024-04-23T16:35:29.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0854e-05 2024-04-23T16:35:29.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:29.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:04.476422+0000", 2024-04-23T16:35:29.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.145e-05 2024-04-23T16:35:29.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:29.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:04.476436+0000", 2024-04-23T16:35:29.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4358999999999999e-05 2024-04-23T16:35:29.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:29.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:04.476539+0000", 2024-04-23T16:35:29.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103123 2024-04-23T16:35:29.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:29.012 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:04.499356+0000", 2024-04-23T16:35:29.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.022816757999999999 2024-04-23T16:35:29.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.012 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6", 2024-04-23T16:35:29.012 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:04.499485+0000", 2024-04-23T16:35:29.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012884800000000001 2024-04-23T16:35:29.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.012 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:29.012 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:04.503378+0000", 2024-04-23T16:35:29.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038929519999999999 2024-04-23T16:35:29.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.012 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:29.013 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:04.509268+0000", 2024-04-23T16:35:29.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058903540000000004 2024-04-23T16:35:29.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.013 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:29.013 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:04.509322+0000", 2024-04-23T16:35:29.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4035e-05 2024-04-23T16:35:29.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.013 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:29.013 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:04.509356+0000", 2024-04-23T16:35:29.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3436999999999999e-05 2024-04-23T16:35:29.013 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:29.013 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:29.013 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:29.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.014 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.014 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.25717.0:231910 5.5 5:a5d54f54:::benchmark_data_smithi188_113957_object78312:head [delete] snapc 0=[] ondisk+write+known_if_redirected e418)", 2024-04-23T16:35:29.014 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:35:24.823801+0000", 2024-04-23T16:35:29.014 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.5971273639999999, 2024-04-23T16:35:29.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.035021879999999998, 2024-04-23T16:35:29.014 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:29.014 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:29.014 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:29.014 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:29.014 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.25717", 2024-04-23T16:35:29.014 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:35:29.014 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 231910 2024-04-23T16:35:29.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.015 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:29.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.015 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:29.015 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.823801+0000", 2024-04-23T16:35:29.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:29.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.015 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:29.015 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.823801+0000", 2024-04-23T16:35:29.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:29.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.015 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:29.015 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.823802+0000", 2024-04-23T16:35:29.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0440000000000001e-06 2024-04-23T16:35:29.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:29.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.823805+0000", 2024-04-23T16:35:29.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5820000000000002e-06 2024-04-23T16:35:29.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:29.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.823814+0000", 2024-04-23T16:35:29.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0480000000000004e-06 2024-04-23T16:35:29.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:29.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.823822+0000", 2024-04-23T16:35:29.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1319999999999994e-06 2024-04-23T16:35:29.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:29.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.823871+0000", 2024-04-23T16:35:29.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8245999999999999e-05 2024-04-23T16:35:29.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:29.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.823993+0000", 2024-04-23T16:35:29.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000122607 2024-04-23T16:35:29.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:35:29.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.824111+0000", 2024-04-23T16:35:29.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011750499999999999 2024-04-23T16:35:29.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:29.018 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.831892+0000", 2024-04-23T16:35:29.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077809130000000004 2024-04-23T16:35:29.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:29.018 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.858710+0000", 2024-04-23T16:35:29.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.026817991999999999 2024-04-23T16:35:29.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:29.019 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.858769+0000", 2024-04-23T16:35:29.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9157999999999998e-05 2024-04-23T16:35:29.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.019 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:29.019 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.858823+0000", 2024-04-23T16:35:29.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3653000000000003e-05 2024-04-23T16:35:29.019 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:29.019 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:29.019 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:29.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.019 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.25717.0:231912 5.5 5:a18b369d:::benchmark_data_smithi188_113957_object78314:head [delete] snapc 0=[] ondisk+write+known_if_redirected e418)", 2024-04-23T16:35:29.019 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:35:24.824420+0000", 2024-04-23T16:35:29.020 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.596507779, 2024-04-23T16:35:29.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.034561268999999999, 2024-04-23T16:35:29.020 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:29.020 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:29.020 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:29.020 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:29.020 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.25717", 2024-04-23T16:35:29.020 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:35:29.020 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 231912 2024-04-23T16:35:29.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.020 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:29.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:29.020 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.824420+0000", 2024-04-23T16:35:29.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:29.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:29.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.824420+0000", 2024-04-23T16:35:29.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:29.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:29.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.824421+0000", 2024-04-23T16:35:29.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5799999999999998e-07 2024-04-23T16:35:29.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:29.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.824425+0000", 2024-04-23T16:35:29.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8279999999999996e-06 2024-04-23T16:35:29.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:29.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.824433+0000", 2024-04-23T16:35:29.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0309999999999998e-06 2024-04-23T16:35:29.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:29.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.824440+0000", 2024-04-23T16:35:29.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9269999999999999e-06 2024-04-23T16:35:29.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:29.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.828579+0000", 2024-04-23T16:35:29.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041393289999999998 2024-04-23T16:35:29.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:29.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.828664+0000", 2024-04-23T16:35:29.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4741999999999996e-05 2024-04-23T16:35:29.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:35:29.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.828739+0000", 2024-04-23T16:35:29.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4595e-05 2024-04-23T16:35:29.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.024 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:29.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.835276+0000", 2024-04-23T16:35:29.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006537653 2024-04-23T16:35:29.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.024 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:29.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.858881+0000", 2024-04-23T16:35:29.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.023605048 2024-04-23T16:35:29.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.024 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:29.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.858936+0000", 2024-04-23T16:35:29.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4585e-05 2024-04-23T16:35:29.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:29.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:24.858982+0000", 2024-04-23T16:35:29.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5572999999999999e-05 2024-04-23T16:35:29.025 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:29.025 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:29.025 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:29.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.025 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.25717.0:232545 5.5 5:a5e30749:::benchmark_data_smithi188_113957_object78947:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-23T16:35:29.025 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:35:25.666721+0000", 2024-04-23T16:35:29.025 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 2.7542073949999999, 2024-04-23T16:35:29.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.035309802000000001, 2024-04-23T16:35:29.025 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:35:29.026 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:35:29.026 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:35:29.026 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:35:29.026 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.25717", 2024-04-23T16:35:29.026 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:35:29.026 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 232545 2024-04-23T16:35:29.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.026 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:35:29.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:35:29.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:25.666721+0000", 2024-04-23T16:35:29.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:29.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:35:29.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:25.666721+0000", 2024-04-23T16:35:29.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:35:29.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:35:29.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:25.666722+0000", 2024-04-23T16:35:29.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.654e-06 2024-04-23T16:35:29.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:35:29.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:25.666728+0000", 2024-04-23T16:35:29.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4210000000000003e-06 2024-04-23T16:35:29.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:35:29.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:25.666739+0000", 2024-04-23T16:35:29.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1018e-05 2024-04-23T16:35:29.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:35:29.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:25.666747+0000", 2024-04-23T16:35:29.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5769999999999994e-06 2024-04-23T16:35:29.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:35:29.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:25.670343+0000", 2024-04-23T16:35:29.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035955570000000001 2024-04-23T16:35:29.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:35:29.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:25.670495+0000", 2024-04-23T16:35:29.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015188800000000001 2024-04-23T16:35:29.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:35:29.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:25.670638+0000", 2024-04-23T16:35:29.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014331000000000001 2024-04-23T16:35:29.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:35:29.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:25.674579+0000", 2024-04-23T16:35:29.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039410130000000002 2024-04-23T16:35:29.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:35:29.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:25.701954+0000", 2024-04-23T16:35:29.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.027375033999999999 2024-04-23T16:35:29.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:35:29.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:25.701993+0000", 2024-04-23T16:35:29.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9153000000000003e-05 2024-04-23T16:35:29.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:35:29.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:35:29.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:35:29.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:25.702030+0000", 2024-04-23T16:35:29.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7177e-05 2024-04-23T16:35:29.031 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:29.031 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:29.031 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:29.031 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:35:29.031 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:35:29.031 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:35:29.608 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:29.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:29.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:29.715 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:35:29.745 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:29.808 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:35:29.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:29.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:35:29.912 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:35:29.945 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:29.971 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:35:29.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:29.978 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:35:30.079 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:35:30.110 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:30.132 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:35:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:30.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:35:30.178 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:35:30.214 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-23T16:35:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:29 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:35:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:29 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:35:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:29 smithi188 ceph-mon[44101]: pgmap v1197: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.6 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 562 op/s; 13974/150254 objects degraded (9.300%); 7.5 MiB/s, 120 objects/s recovering 2024-04-23T16:35:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:29 smithi188 ceph-mon[44101]: osdmap e421: 12 total, 11 up, 9 in 2024-04-23T16:35:30.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:29 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 13974/150254 objects degraded (9.300%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:30.236 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:35:30.238 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:29 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:35:30.238 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:29 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:35:30.238 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:29 smithi103 ceph-mon[39629]: pgmap v1197: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.6 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 562 op/s; 13974/150254 objects degraded (9.300%); 7.5 MiB/s, 120 objects/s recovering 2024-04-23T16:35:30.239 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:29 smithi103 ceph-mon[39629]: osdmap e421: 12 total, 11 up, 9 in 2024-04-23T16:35:30.239 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:29 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 13974/150254 objects degraded (9.300%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:30.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:29 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:35:30.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:29 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:35:30.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:29 smithi148 ceph-mon[44520]: pgmap v1197: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.6 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 562 op/s; 13974/150254 objects degraded (9.300%); 7.5 MiB/s, 120 objects/s recovering 2024-04-23T16:35:30.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:29 smithi148 ceph-mon[44520]: osdmap e421: 12 total, 11 up, 9 in 2024-04-23T16:35:30.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:29 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 13974/150254 objects degraded (9.300%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:30.309 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:30.350 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:35:30.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:30.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:35:30.459 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:35:30.488 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:30.516 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:35:30.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:30.523 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:35:30.624 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:35:30.657 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:30.680 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:35:30.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:30.688 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:35:30.789 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:35:30.821 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:30.850 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:35:30.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:30.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:35:30.954 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:35:31.034 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:31.079 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:35:31.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:31.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:35:31.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:31 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:35:31.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:31 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:35:31.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:31 smithi188 ceph-mon[44101]: osdmap e422: 12 total, 11 up, 9 in 2024-04-23T16:35:31.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:31 smithi188 ceph-mon[44101]: pgmap v1200: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.4 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 702 op/s; 13522/143954 objects degraded (9.393%); 7.6 MiB/s, 122 objects/s recovering 2024-04-23T16:35:31.189 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:35:31.219 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:31.318 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:35:31.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:31.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:35:31.322 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:31 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:35:31.322 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:31 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:35:31.322 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:31 smithi103 ceph-mon[39629]: osdmap e422: 12 total, 11 up, 9 in 2024-04-23T16:35:31.322 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:31 smithi103 ceph-mon[39629]: pgmap v1200: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.4 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 702 op/s; 13522/143954 objects degraded (9.393%); 7.6 MiB/s, 122 objects/s recovering 2024-04-23T16:35:31.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:31 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:35:31.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:31 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:35:31.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:31 smithi148 ceph-mon[44520]: osdmap e422: 12 total, 11 up, 9 in 2024-04-23T16:35:31.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:31 smithi148 ceph-mon[44520]: pgmap v1200: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.4 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 702 op/s; 13522/143954 objects degraded (9.393%); 7.6 MiB/s, 122 objects/s recovering 2024-04-23T16:35:31.423 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:35:31.526 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:31.567 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:35:31.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:31.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:35:31.674 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:35:31.703 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:31.724 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:35:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:31.727 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:35:31.828 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:31.860 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:32.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:32 smithi188 ceph-mon[44101]: 5.2 scrub starts 2024-04-23T16:35:32.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:32 smithi188 ceph-mon[44101]: 5.e scrub starts 2024-04-23T16:35:32.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:32 smithi103 ceph-mon[39629]: 5.2 scrub starts 2024-04-23T16:35:32.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:32 smithi103 ceph-mon[39629]: 5.e scrub starts 2024-04-23T16:35:32.613 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:32.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:32.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:32.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:32 smithi148 ceph-mon[44520]: 5.2 scrub starts 2024-04-23T16:35:32.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:32 smithi148 ceph-mon[44520]: 5.e scrub starts 2024-04-23T16:35:32.719 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:35:32.720 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:35:32.728 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:35:32.756 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:33.106 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:35:33.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:33.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:35:33.218 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:35:33.293 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:33.334 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:35:33.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:33.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:35:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:33 smithi103 ceph-mon[39629]: 5.e scrub ok 2024-04-23T16:35:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:33 smithi103 ceph-mon[39629]: 5.2 scrub ok 2024-04-23T16:35:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:33 smithi103 ceph-mon[39629]: from='client.16083 v1:172.21.15.103:0/413226141' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:35:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:33 smithi103 ceph-mon[39629]: pgmap v1201: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.4 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 633 op/s; 13522/143056 objects degraded (9.452%); 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:35:33.441 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:33.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:33 smithi188 ceph-mon[44101]: 5.e scrub ok 2024-04-23T16:35:33.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:33 smithi188 ceph-mon[44101]: 5.2 scrub ok 2024-04-23T16:35:33.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:33 smithi188 ceph-mon[44101]: from='client.16083 v1:172.21.15.103:0/413226141' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:35:33.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:33 smithi188 ceph-mon[44101]: pgmap v1201: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.4 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 633 op/s; 13522/143056 objects degraded (9.452%); 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:35:33.476 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:33.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:33 smithi148 ceph-mon[44520]: 5.e scrub ok 2024-04-23T16:35:33.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:33 smithi148 ceph-mon[44520]: 5.2 scrub ok 2024-04-23T16:35:33.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:33 smithi148 ceph-mon[44520]: from='client.16083 v1:172.21.15.103:0/413226141' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:35:33.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:33 smithi148 ceph-mon[44520]: pgmap v1201: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.4 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 633 op/s; 13522/143056 objects degraded (9.452%); 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:35:34.229 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:34.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:34.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:34.334 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:35:34.361 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:34.385 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:35:34.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:34.390 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:35:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:34 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:35:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:34 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:35:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:34 smithi188 ceph-mon[44101]: osdmap e423: 12 total, 11 up, 9 in 2024-04-23T16:35:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:34 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 13522/143056 objects degraded (9.452%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:34.491 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:35:34.564 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:34.569 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:34 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:35:34.569 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:34 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:35:34.569 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:34 smithi148 ceph-mon[44520]: osdmap e423: 12 total, 11 up, 9 in 2024-04-23T16:35:34.570 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:34 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 13522/143056 objects degraded (9.452%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:34.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:34 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:35:34.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:34 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:35:34.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:34 smithi103 ceph-mon[39629]: osdmap e423: 12 total, 11 up, 9 in 2024-04-23T16:35:34.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:34 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 13522/143056 objects degraded (9.452%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:34.610 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:35:34.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:34.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:35:34.631 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:35:34.652 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:35:34.673 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:35:34.706 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:35:34.724 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:35:34.737 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:35:34.770 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:35:34.780 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:34.799 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:35:34.817 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:35:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:34.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:35:34.821 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:35:34.848 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:35:34.872 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:35:34.892 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:35:34.893 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:35:34.893 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:35:34.893 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:35:34.893 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:35:34.893 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:35:34.893 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:35:34.893 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:35:34.893 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:35:34.893 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:35:34.893 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:35:34.893 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:35:34.894 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:35:34.920 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:35:34.994 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:35.028 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:35:35.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:35.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:35:35.089 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1200,"stamp":"2024-04-23T16:35:30.733533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4717478456,"num_objects":71977,"num_object_clones":0,"num_object_copies":143954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71977,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":235217,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15431,"num_bytes_recovered":1011421728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162377,"ondisk_log_size":162377,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":11859108,"kb_used_data":8404356,"kb_used_omap":14,"kb_used_meta":3454129,"kb_avail":831847260,"statfs":{"total":863955320832,"available":851811594240,"internally_reserved":0,"allocated":8606060544,"data_stored":8602830080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3537029103},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,5,7,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-267255808,"num_objects":-4078,"num_object_clones":0,"num_object_copies":-8156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-709,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4078,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":709,"num_bytes_recovered":46465024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.802937"},"pg_stats":[{"pgid":"5.f","version":"421'14881","reported_seq":15118,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:29.889647+0000","last_change":"2024-04-23T16:33:17.526419+0000","last_active":"2024-04-23T16:35:29.889647+0000","last_peered":"2024-04-23T16:35:29.889647+0000","last_clean":"2024-04-23T16:35:29.889647+0000","last_became_active":"2024-04-23T16:31:29.140923+0000","last_became_peered":"2024-04-23T16:31:29.140923+0000","last_unstale":"2024-04-23T16:35:29.889647+0000","last_undegraded":"2024-04-23T16:35:29.889647+0000","last_fullsized":"2024-04-23T16:35:29.889647+0000","mapping_epoch":359,"log_start":"372'4800","ondisk_log_start":"372'4800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9683","last_scrub_stamp":"2024-04-23T16:33:17.526283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.526283+0000","objects_scrubbed":9683,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293928960,"num_objects":4485,"num_object_clones":0,"num_object_copies":8970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14881,"num_write_kb":619712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"421'14716","reported_seq":15079,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:29.140527+0000","last_change":"2024-04-23T16:35:26.407358+0000","last_active":"2024-04-23T16:35:29.140527+0000","last_peered":"2024-04-23T16:35:29.140527+0000","last_clean":"2024-04-23T16:35:29.140527+0000","last_became_active":"2024-04-23T16:31:29.137890+0000","last_became_peered":"2024-04-23T16:31:29.137890+0000","last_unstale":"2024-04-23T16:35:29.140527+0000","last_undegraded":"2024-04-23T16:35:29.140527+0000","last_fullsized":"2024-04-23T16:35:29.140527+0000","mapping_epoch":359,"log_start":"372'4700","ondisk_log_start":"372'4700","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'14582","last_scrub_stamp":"2024-04-23T16:35:26.407093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:26.407093+0000","objects_scrubbed":4692,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:38.641004+0000","scrub_duration":544,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298057728,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14716,"num_write_kb":616448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"420'14795","reported_seq":25176,"reported_epoch":420,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:35:28.165917+0000","last_change":"2024-04-23T16:33:26.432192+0000","last_active":"2024-04-23T16:35:28.165917+0000","last_peered":"2024-04-23T16:35:28.165917+0000","last_clean":"2024-04-23T16:32:59.749530+0000","last_became_active":"2024-04-23T16:33:26.338644+0000","last_became_peered":"2024-04-23T16:33:26.338644+0000","last_unstale":"2024-04-23T16:35:28.165917+0000","last_undegraded":"2024-04-23T16:33:25.874847+0000","last_fullsized":"2024-04-23T16:35:28.165917+0000","mapping_epoch":394,"log_start":"372'4700","ondisk_log_start":"372'4700","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8711","last_scrub_stamp":"2024-04-23T16:32:43.928914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.928914+0000","objects_scrubbed":8680,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304545792,"num_objects":4647,"num_object_clones":0,"num_object_copies":9294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4638,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14795,"num_write_kb":622144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5074,"num_bytes_recovered":332529664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":["8"],"object_location_counts":[{"shards":"4,8","objects":9},{"shards":"8","objects":4638}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.9","version":"421'14870","reported_seq":15092,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:30.130322+0000","last_change":"2024-04-23T16:32:37.256944+0000","last_active":"2024-04-23T16:35:30.130322+0000","last_peered":"2024-04-23T16:35:30.130322+0000","last_clean":"2024-04-23T16:35:30.130322+0000","last_became_active":"2024-04-23T16:31:29.141551+0000","last_became_peered":"2024-04-23T16:31:29.141551+0000","last_unstale":"2024-04-23T16:35:30.130322+0000","last_undegraded":"2024-04-23T16:35:30.130322+0000","last_fullsized":"2024-04-23T16:35:30.130322+0000","mapping_epoch":359,"log_start":"372'4800","ondisk_log_start":"372'4800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8156","last_scrub_stamp":"2024-04-23T16:32:37.256787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:37.256787+0000","objects_scrubbed":8082,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291635200,"num_objects":4450,"num_object_clones":0,"num_object_copies":8900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14870,"num_write_kb":618240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"420'14243","reported_seq":14549,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-23T16:35:26.728313+0000","last_change":"2024-04-23T16:34:39.934502+0000","last_active":"2024-04-23T16:35:26.728313+0000","last_peered":"2024-04-23T16:35:26.728313+0000","last_clean":"2024-04-23T16:35:26.728313+0000","last_became_active":"2024-04-23T16:31:29.139045+0000","last_became_peered":"2024-04-23T16:31:29.139045+0000","last_unstale":"2024-04-23T16:35:26.728313+0000","last_undegraded":"2024-04-23T16:35:26.728313+0000","last_fullsized":"2024-04-23T16:35:26.728313+0000","mapping_epoch":359,"log_start":"372'4200","ondisk_log_start":"372'4200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'12219","last_scrub_stamp":"2024-04-23T16:34:39.934190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:39.934190+0000","objects_scrubbed":6529,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:59.649413+0000","scrub_duration":810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294322176,"num_objects":4491,"num_object_clones":0,"num_object_copies":8982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14243,"num_write_kb":599488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"421'18278","reported_seq":18663,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:29.286207+0000","last_change":"2024-04-23T16:35:10.892898+0000","last_active":"2024-04-23T16:35:29.286207+0000","last_peered":"2024-04-23T16:35:29.286207+0000","last_clean":"2024-04-23T16:35:29.286207+0000","last_became_active":"2024-04-23T16:31:29.310906+0000","last_became_peered":"2024-04-23T16:31:29.310906+0000","last_unstale":"2024-04-23T16:35:29.286207+0000","last_undegraded":"2024-04-23T16:35:29.286207+0000","last_fullsized":"2024-04-23T16:35:29.286207+0000","mapping_epoch":359,"log_start":"381'8204","ondisk_log_start":"381'8204","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9559","last_scrub_stamp":"2024-04-23T16:33:22.102857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:22.102857+0000","objects_scrubbed":3561,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288555008,"num_objects":4403,"num_object_clones":0,"num_object_copies":8806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14717,"num_write_kb":611840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1104,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:28.956005+0000","last_change":"2024-04-23T16:34:44.725424+0000","last_active":"2024-04-23T16:35:28.956005+0000","last_peered":"2024-04-23T16:35:28.956005+0000","last_clean":"2024-04-23T16:35:28.956005+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:35:28.956005+0000","last_undegraded":"2024-04-23T16:35:28.956005+0000","last_fullsized":"2024-04-23T16:35:28.956005+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:44.725324+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:44.725324+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:14.224827+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"421'14957","reported_seq":15180,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:29.446667+0000","last_change":"2024-04-23T16:32:38.103756+0000","last_active":"2024-04-23T16:35:29.446667+0000","last_peered":"2024-04-23T16:35:29.446667+0000","last_clean":"2024-04-23T16:35:29.446667+0000","last_became_active":"2024-04-23T16:31:29.141640+0000","last_became_peered":"2024-04-23T16:31:29.141640+0000","last_unstale":"2024-04-23T16:35:29.446667+0000","last_undegraded":"2024-04-23T16:35:29.446667+0000","last_fullsized":"2024-04-23T16:35:29.446667+0000","mapping_epoch":359,"log_start":"372'4900","ondisk_log_start":"372'4900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8300","last_scrub_stamp":"2024-04-23T16:32:38.103547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:38.103547+0000","objects_scrubbed":8231,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294846464,"num_objects":4499,"num_object_clones":0,"num_object_copies":8998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14957,"num_write_kb":622592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1098,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:28.960792+0000","last_change":"2024-04-23T16:33:25.851379+0000","last_active":"2024-04-23T16:35:28.960792+0000","last_peered":"2024-04-23T16:35:28.960792+0000","last_clean":"2024-04-23T16:35:28.960792+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:35:28.960792+0000","last_undegraded":"2024-04-23T16:35:28.960792+0000","last_fullsized":"2024-04-23T16:35:28.960792+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.498820+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.498820+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"420'14444","reported_seq":14754,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-23T16:35:26.744339+0000","last_change":"2024-04-23T16:34:41.375046+0000","last_active":"2024-04-23T16:35:26.744339+0000","last_peered":"2024-04-23T16:35:26.744339+0000","last_clean":"2024-04-23T16:35:26.744339+0000","last_became_active":"2024-04-23T16:31:29.140251+0000","last_became_peered":"2024-04-23T16:31:29.140251+0000","last_unstale":"2024-04-23T16:35:26.744339+0000","last_undegraded":"2024-04-23T16:35:26.744339+0000","last_fullsized":"2024-04-23T16:35:26.744339+0000","mapping_epoch":359,"log_start":"372'4400","ondisk_log_start":"372'4400","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12511","last_scrub_stamp":"2024-04-23T16:34:41.374888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:41.374888+0000","objects_scrubbed":6643,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:07.291855+0000","scrub_duration":1271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307167256,"num_objects":4688,"num_object_clones":0,"num_object_copies":9376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4688,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14444,"num_write_kb":612161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1089,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:28.968590+0000","last_change":"2024-04-23T16:32:29.517947+0000","last_active":"2024-04-23T16:35:28.968590+0000","last_peered":"2024-04-23T16:35:28.968590+0000","last_clean":"2024-04-23T16:35:28.968590+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:35:28.968590+0000","last_undegraded":"2024-04-23T16:35:28.968590+0000","last_fullsized":"2024-04-23T16:35:28.968590+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:32:29.517800+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:32:29.517800+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"421'14895","reported_seq":25540,"reported_epoch":421,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:35:29.907982+0000","last_change":"2024-04-23T16:33:26.507399+0000","last_active":"2024-04-23T16:35:29.907982+0000","last_peered":"2024-04-23T16:35:29.907982+0000","last_clean":"2024-04-23T16:32:59.715575+0000","last_became_active":"2024-04-23T16:33:26.458611+0000","last_became_peered":"2024-04-23T16:33:26.458611+0000","last_unstale":"2024-04-23T16:35:29.907982+0000","last_undegraded":"2024-04-23T16:33:25.895472+0000","last_fullsized":"2024-04-23T16:35:29.907982+0000","mapping_epoch":394,"log_start":"372'4800","ondisk_log_start":"372'4800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8607","last_scrub_stamp":"2024-04-23T16:32:43.130806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.130806+0000","objects_scrubbed":8577,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293928960,"num_objects":4485,"num_object_clones":0,"num_object_copies":8970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4481,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14895,"num_write_kb":620160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5206,"num_bytes_recovered":341180416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":4481},{"shards":"5,7","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1103,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:28.959143+0000","last_change":"2024-04-23T16:34:47.438492+0000","last_active":"2024-04-23T16:35:28.959143+0000","last_peered":"2024-04-23T16:35:28.959143+0000","last_clean":"2024-04-23T16:35:28.959143+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:35:28.959143+0000","last_undegraded":"2024-04-23T16:35:28.959143+0000","last_fullsized":"2024-04-23T16:35:28.959143+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:47.438356+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:47.438356+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:13.051838+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"421'14518","reported_seq":14753,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:29.894513+0000","last_change":"2024-04-23T16:33:18.576077+0000","last_active":"2024-04-23T16:35:29.894513+0000","last_peered":"2024-04-23T16:35:29.894513+0000","last_clean":"2024-04-23T16:35:29.894513+0000","last_became_active":"2024-04-23T16:31:29.139709+0000","last_became_peered":"2024-04-23T16:31:29.139709+0000","last_unstale":"2024-04-23T16:35:29.894513+0000","last_undegraded":"2024-04-23T16:35:29.894513+0000","last_fullsized":"2024-04-23T16:35:29.894513+0000","mapping_epoch":359,"log_start":"372'4500","ondisk_log_start":"372'4500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9472","last_scrub_stamp":"2024-04-23T16:33:18.575945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.575945+0000","objects_scrubbed":9472,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290062336,"num_objects":4426,"num_object_clones":0,"num_object_copies":8852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14518,"num_write_kb":606208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1096,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:28.956995+0000","last_change":"2024-04-23T16:34:02.399102+0000","last_active":"2024-04-23T16:35:28.956995+0000","last_peered":"2024-04-23T16:35:28.956995+0000","last_clean":"2024-04-23T16:35:28.956995+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:35:28.956995+0000","last_undegraded":"2024-04-23T16:35:28.956995+0000","last_fullsized":"2024-04-23T16:35:28.956995+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:02.398975+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:02.398975+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"421'14810","reported_seq":15125,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:29.126505+0000","last_change":"2024-04-23T16:34:44.348696+0000","last_active":"2024-04-23T16:35:29.126505+0000","last_peered":"2024-04-23T16:35:29.126505+0000","last_clean":"2024-04-23T16:35:29.126505+0000","last_became_active":"2024-04-23T16:31:29.139606+0000","last_became_peered":"2024-04-23T16:31:29.139606+0000","last_unstale":"2024-04-23T16:35:29.126505+0000","last_undegraded":"2024-04-23T16:35:29.126505+0000","last_fullsized":"2024-04-23T16:35:29.126505+0000","mapping_epoch":359,"log_start":"373'4800","ondisk_log_start":"373'4800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12750","last_scrub_stamp":"2024-04-23T16:34:44.348550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:44.348550+0000","objects_scrubbed":6711,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:02.747435+0000","scrub_duration":1024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303431680,"num_objects":4630,"num_object_clones":0,"num_object_copies":9260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14810,"num_write_kb":622080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1103,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:28.956009+0000","last_change":"2024-04-23T16:34:51.848140+0000","last_active":"2024-04-23T16:35:28.956009+0000","last_peered":"2024-04-23T16:35:28.956009+0000","last_clean":"2024-04-23T16:35:28.956009+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:35:28.956009+0000","last_undegraded":"2024-04-23T16:35:28.956009+0000","last_fullsized":"2024-04-23T16:35:28.956009+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:51.847992+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:34:51.847992+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:18.910354+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.2","version":"421'14581","reported_seq":14895,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:29.528650+0000","last_change":"2024-04-23T16:33:57.879648+0000","last_active":"2024-04-23T16:35:29.528650+0000","last_peered":"2024-04-23T16:35:29.528650+0000","last_clean":"2024-04-23T16:35:29.528650+0000","last_became_active":"2024-04-23T16:31:29.138430+0000","last_became_peered":"2024-04-23T16:31:29.138430+0000","last_unstale":"2024-04-23T16:35:29.528650+0000","last_undegraded":"2024-04-23T16:35:29.528650+0000","last_fullsized":"2024-04-23T16:35:29.528650+0000","mapping_epoch":359,"log_start":"372'4500","ondisk_log_start":"372'4500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'10726","last_scrub_stamp":"2024-04-23T16:33:57.879516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:57.879516+0000","objects_scrubbed":8303,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289341440,"num_objects":4415,"num_object_clones":0,"num_object_copies":8830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14581,"num_write_kb":607872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1104,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:28.956123+0000","last_change":"2024-04-23T16:34:37.693617+0000","last_active":"2024-04-23T16:35:28.956123+0000","last_peered":"2024-04-23T16:35:28.956123+0000","last_clean":"2024-04-23T16:35:28.956123+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:35:28.956123+0000","last_undegraded":"2024-04-23T16:35:28.956123+0000","last_fullsized":"2024-04-23T16:35:28.956123+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:37.693482+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:37.693482+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:49.920873+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"421'16890","reported_seq":17331,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:29.098717+0000","last_change":"2024-04-23T16:34:40.604268+0000","last_active":"2024-04-23T16:35:29.098717+0000","last_peered":"2024-04-23T16:35:29.098717+0000","last_clean":"2024-04-23T16:35:29.098717+0000","last_became_active":"2024-04-23T16:31:29.139398+0000","last_became_peered":"2024-04-23T16:31:29.139398+0000","last_unstale":"2024-04-23T16:35:29.098717+0000","last_undegraded":"2024-04-23T16:35:29.098717+0000","last_fullsized":"2024-04-23T16:35:29.098717+0000","mapping_epoch":359,"log_start":"377'6800","ondisk_log_start":"377'6800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'14807","last_scrub_stamp":"2024-04-23T16:34:40.604136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:40.604136+0000","objects_scrubbed":6625,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:04.702557+0000","scrub_duration":1231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295698432,"num_objects":4512,"num_object_clones":0,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14616,"num_write_kb":612096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1096,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:28.956824+0000","last_change":"2024-04-23T16:34:04.444103+0000","last_active":"2024-04-23T16:35:28.956824+0000","last_peered":"2024-04-23T16:35:28.956824+0000","last_clean":"2024-04-23T16:35:28.956824+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:35:28.956824+0000","last_undegraded":"2024-04-23T16:35:28.956824+0000","last_fullsized":"2024-04-23T16:35:28.956824+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:04.443975+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:04.443975+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"421'14526","reported_seq":14752,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:29.436398+0000","last_change":"2024-04-23T16:32:43.947374+0000","last_active":"2024-04-23T16:35:29.436398+0000","last_peered":"2024-04-23T16:35:29.436398+0000","last_clean":"2024-04-23T16:35:29.436398+0000","last_became_active":"2024-04-23T16:31:29.138968+0000","last_became_peered":"2024-04-23T16:31:29.138968+0000","last_unstale":"2024-04-23T16:35:29.436398+0000","last_undegraded":"2024-04-23T16:35:29.436398+0000","last_fullsized":"2024-04-23T16:35:29.436398+0000","mapping_epoch":359,"log_start":"372'4500","ondisk_log_start":"372'4500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8482","last_scrub_stamp":"2024-04-23T16:32:43.947177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.947177+0000","objects_scrubbed":8428,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286261248,"num_objects":4368,"num_object_clones":0,"num_object_copies":8736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14526,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":993,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-23T16:35:25.938138+0000","last_change":"2024-04-23T16:34:51.248895+0000","last_active":"2024-04-23T16:35:25.938138+0000","last_peered":"2024-04-23T16:35:25.938138+0000","last_clean":"2024-04-23T16:35:25.938138+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:35:25.938138+0000","last_undegraded":"2024-04-23T16:35:25.938138+0000","last_fullsized":"2024-04-23T16:35:25.938138+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:51.248783+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:51.248783+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:13.233910+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"421'14943","reported_seq":15167,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:29.424313+0000","last_change":"2024-04-23T16:32:39.103230+0000","last_active":"2024-04-23T16:35:29.424313+0000","last_peered":"2024-04-23T16:35:29.424313+0000","last_clean":"2024-04-23T16:35:29.424313+0000","last_became_active":"2024-04-23T16:31:29.310246+0000","last_became_peered":"2024-04-23T16:31:29.310246+0000","last_unstale":"2024-04-23T16:35:29.424313+0000","last_undegraded":"2024-04-23T16:35:29.424313+0000","last_fullsized":"2024-04-23T16:35:29.424313+0000","mapping_epoch":359,"log_start":"373'4900","ondisk_log_start":"373'4900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8325","last_scrub_stamp":"2024-04-23T16:32:39.103059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:39.103059+0000","objects_scrubbed":8272,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296157184,"num_objects":4519,"num_object_clones":0,"num_object_copies":9038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14943,"num_write_kb":622784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":917,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:28.956701+0000","last_change":"2024-04-23T16:34:56.453746+0000","last_active":"2024-04-23T16:35:28.956701+0000","last_peered":"2024-04-23T16:35:28.956701+0000","last_clean":"2024-04-23T16:35:28.956701+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:35:28.956701+0000","last_undegraded":"2024-04-23T16:35:28.956701+0000","last_fullsized":"2024-04-23T16:35:28.956701+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:56.453617+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:34:56.453617+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:08.659133+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"421'14705","reported_seq":25233,"reported_epoch":421,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:35:29.844452+0000","last_change":"2024-04-23T16:33:26.132487+0000","last_active":"2024-04-23T16:35:29.844452+0000","last_peered":"2024-04-23T16:35:29.844452+0000","last_clean":"2024-04-23T16:32:59.695133+0000","last_became_active":"2024-04-23T16:33:26.078062+0000","last_became_peered":"2024-04-23T16:33:26.078062+0000","last_unstale":"2024-04-23T16:35:29.844452+0000","last_undegraded":"2024-04-23T16:33:25.892009+0000","last_fullsized":"2024-04-23T16:35:29.844452+0000","mapping_epoch":394,"log_start":"372'4700","ondisk_log_start":"372'4700","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'7968","last_scrub_stamp":"2024-04-23T16:32:36.102122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:36.102122+0000","objects_scrubbed":7926,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288948224,"num_objects":4409,"num_object_clones":0,"num_object_copies":8818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14705,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5150,"num_bytes_recovered":337510400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":4403},{"shards":"5,8","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1093,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-04-23T16:35:28.966354+0000","last_change":"2024-04-23T16:33:17.354409+0000","last_active":"2024-04-23T16:35:28.966354+0000","last_peered":"2024-04-23T16:35:28.966354+0000","last_clean":"2024-04-23T16:35:28.966354+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:35:28.966354+0000","last_undegraded":"2024-04-23T16:35:28.966354+0000","last_fullsized":"2024-04-23T16:35:28.966354+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:17.354263+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.354263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4716888088,"num_objects":71975,"num_object_clones":0,"num_object_copies":143950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71975,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":235217,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15430,"num_bytes_recovered":1011220480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10442055680,"data_stored":10442047536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056882,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1087112,"kb_used_data":590840,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":92658040,"statfs":{"total":95995035648,"available":94881832960,"internally_reserved":0,"allocated":605020160,"data_stored":604660176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":508164548},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351224,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612862,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1200124,"kb_used_data":862132,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92545028,"statfs":{"total":95995035648,"available":94766108672,"internally_reserved":0,"allocated":882823168,"data_stored":882467256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":346094016},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841794,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":863292,"kb_used_data":585012,"kb_used_omap":1,"kb_used_meta":278270,"kb_avail":92881860,"statfs":{"total":95995035648,"available":95111024640,"internally_reserved":0,"allocated":599052288,"data_stored":598696376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":284948933},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168839,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":656524,"kb_used_data":287872,"kb_used_omap":1,"kb_used_meta":368510,"kb_avail":93088628,"statfs":{"total":95995035648,"available":95322755072,"internally_reserved":0,"allocated":294780928,"data_stored":294419012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":377354698},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463181,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1148604,"kb_used_data":864692,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":92596548,"statfs":{"total":95995035648,"available":94818865152,"internally_reserved":0,"allocated":885444608,"data_stored":885088696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":290650566},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137320,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738666,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2448332,"kb_used_data":2011776,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":91296820,"statfs":{"total":95995035648,"available":93487943680,"internally_reserved":0,"allocated":2060058624,"data_stored":2059696708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117274,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663100,"kb_used_data":302896,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93082052,"statfs":{"total":95995035648,"available":95316021248,"internally_reserved":0,"allocated":310165504,"data_stored":309812044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":368835015},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921045,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1944444,"kb_used_data":1457588,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":91800708,"statfs":{"total":95995035648,"available":94003924992,"internally_reserved":0,"allocated":1492570112,"data_stored":1492214200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757520,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1847576,"kb_used_data":1441548,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":91897576,"statfs":{"total":95995035648,"available":94103117824,"internally_reserved":0,"allocated":1476145152,"data_stored":1475775612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2059272192,"data_stored":2059272192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1886846976,"data_stored":1886846976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":307494912,"data_stored":307494912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1476067328,"data_stored":1476067328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1465651200,"data_stored":1465647128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":876937216,"data_stored":876937216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":292028416,"data_stored":292028416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":597426176,"data_stored":597426176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":878837760,"data_stored":878837760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":601493504,"data_stored":601489432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:35:35.091 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:35:35.143 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:35:35.172 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:35.296 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:35:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:35.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:35:35.402 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:35:35.435 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:35.457 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:35:35.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:35.460 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:35:35.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:35 smithi188 ceph-mon[44101]: 1.7 scrub starts 2024-04-23T16:35:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:35 smithi188 ceph-mon[44101]: 1.7 scrub ok 2024-04-23T16:35:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:35 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2622147551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:35:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:35 smithi188 ceph-mon[44101]: pgmap v1203: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 970 op/s; 13267/138608 objects degraded (9.572%); 7.4 MiB/s, 117 objects/s recovering 2024-04-23T16:35:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:35.561 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:35.587 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:35 smithi103 ceph-mon[39629]: 1.7 scrub starts 2024-04-23T16:35:35.587 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:35 smithi103 ceph-mon[39629]: 1.7 scrub ok 2024-04-23T16:35:35.587 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:35 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2622147551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:35:35.587 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:35.588 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:35 smithi103 ceph-mon[39629]: pgmap v1203: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 970 op/s; 13267/138608 objects degraded (9.572%); 7.4 MiB/s, 117 objects/s recovering 2024-04-23T16:35:35.588 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:35.588 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:35.589 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:35.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:35 smithi148 ceph-mon[44520]: 1.7 scrub starts 2024-04-23T16:35:35.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:35 smithi148 ceph-mon[44520]: 1.7 scrub ok 2024-04-23T16:35:35.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:35 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2622147551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:35:35.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:35.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:35 smithi148 ceph-mon[44520]: pgmap v1203: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 970 op/s; 13267/138608 objects degraded (9.572%); 7.4 MiB/s, 117 objects/s recovering 2024-04-23T16:35:35.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:35.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:35.840 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:35.844 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:35:35.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:35.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:35.880 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:35:35.880 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:35:35.948 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:35:35.981 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:36.005 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:35:36.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:36.008 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:35:36.109 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:35:36.135 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:36.157 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:35:36.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:36.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:35:36.261 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:35:36.287 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.287 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.287 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.287 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.287 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:36 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:35:36.288 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.288 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.288 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.288 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:36 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:35:36.288 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.290 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:36.311 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:35:36.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:36.317 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:35:36.418 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:35:36.443 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.443 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.444 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.444 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.444 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:36 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:35:36.444 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.444 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.444 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.444 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:36 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:35:36.444 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.445 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:36.468 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:35:36.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:36.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:35:36.571 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:35:36.601 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:36.622 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:35:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:36 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:35:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:36 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:35:36.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:36.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:36.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:35:36.728 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:35:36.763 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:36.807 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:35:36.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:36.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:35:36.913 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:35:36.969 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:37.017 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:35:37.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:37.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:35:37.123 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:35:37.152 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:37.175 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:35:37.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:37.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:35:37.228 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:35:37.279 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:35:37.309 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:37 smithi148 ceph-mon[44520]: pgmap v1204: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 896 op/s; 12807/136296 objects degraded (9.396%); 9.4 MiB/s, 149 objects/s recovering 2024-04-23T16:35:37.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:37 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/103033314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:35:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:37 smithi188 ceph-mon[44101]: pgmap v1204: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 896 op/s; 12807/136296 objects degraded (9.396%); 9.4 MiB/s, 149 objects/s recovering 2024-04-23T16:35:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:37 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/103033314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:35:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:37 smithi103 ceph-mon[39629]: pgmap v1204: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 896 op/s; 12807/136296 objects degraded (9.396%); 9.4 MiB/s, 149 objects/s recovering 2024-04-23T16:35:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:37 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/103033314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:35:37.881 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:35:38.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:38 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:35:38.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:38 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/103033314' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:35:38.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:38 smithi103 ceph-mon[39629]: osdmap e424: 12 total, 11 up, 9 in 2024-04-23T16:35:38.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:38 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:35:38.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:38 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/103033314' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:35:38.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:38 smithi148 ceph-mon[44520]: osdmap e424: 12 total, 11 up, 9 in 2024-04-23T16:35:38.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:38 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:35:38.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:38 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/103033314' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:35:38.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:38 smithi188 ceph-mon[44101]: osdmap e424: 12 total, 11 up, 9 in 2024-04-23T16:35:39.261 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:35:39.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:39.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:35:39.366 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:39.395 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:39.520 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:39.524 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:39 smithi103 ceph-mon[39629]: pgmap v1206: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 570 op/s; 12553/134818 objects degraded (9.311%); 7.6 MiB/s, 121 objects/s recovering 2024-04-23T16:35:39.524 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:39 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 12807/136296 objects degraded (9.396%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:39.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:39.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:39.627 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:35:39.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:39 smithi148 ceph-mon[44520]: pgmap v1206: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 570 op/s; 12553/134818 objects degraded (9.311%); 7.6 MiB/s, 121 objects/s recovering 2024-04-23T16:35:39.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:39 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 12807/136296 objects degraded (9.396%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:39.655 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:39.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:39 smithi188 ceph-mon[44101]: pgmap v1206: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 4.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 570 op/s; 12553/134818 objects degraded (9.311%); 7.6 MiB/s, 121 objects/s recovering 2024-04-23T16:35:39.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:39 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 12807/136296 objects degraded (9.396%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:39.808 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:35:39.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:39.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:35:39.914 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:35:39.942 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:39.965 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:35:39.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:39.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:35:40.070 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:35:40.071 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:35:40.071 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:35:40.099 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:40.122 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:35:40.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:40.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:35:40.226 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:35:40.296 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:40.350 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:35:40.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:40.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:35:40.459 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:35:40.523 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:40.556 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:35:40.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:40.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:35:40.661 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:35:40.689 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:40.712 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:35:40.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:40.715 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:35:40.816 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:40.845 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:40.920 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:40.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:40.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:41.028 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:35:41.055 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:41.079 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:35:41.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:41.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:35:41.189 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:35:41.252 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:41.303 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:35:41.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:41.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:35:41.412 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:35:41.441 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:41.446 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:35:41.449 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:35:41.481 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:35:41.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:41.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:35:41.490 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-23T16:35:41.585 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:35:41.616 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:41.636 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:35:41.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:41.640 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:35:41.741 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:35:41.818 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:41.860 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:35:41.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:41.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:35:41.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:41 smithi188 ceph-mon[44101]: pgmap v1207: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.9 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 899 op/s; 12083/128664 objects degraded (9.391%); 11 MiB/s, 179 objects/s recovering 2024-04-23T16:35:41.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:41 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:35:41.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:41 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:35:41.985 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:35:42.017 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:42.040 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:35:42.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:42.043 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:35:42.064 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:41 smithi103 ceph-mon[39629]: pgmap v1207: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.9 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 899 op/s; 12083/128664 objects degraded (9.391%); 11 MiB/s, 179 objects/s recovering 2024-04-23T16:35:42.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:41 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:35:42.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:41 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:35:42.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:41 smithi148 ceph-mon[44520]: pgmap v1207: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.9 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 899 op/s; 12083/128664 objects degraded (9.391%); 11 MiB/s, 179 objects/s recovering 2024-04-23T16:35:42.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:41 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:35:42.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:41 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:35:42.143 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:35:42.171 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:42.196 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:35:42.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:42.201 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:35:42.302 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:35:42.332 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:42.360 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:35:42.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:42.367 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:35:42.468 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:35:42.500 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:42.526 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:35:42.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:42.531 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:35:42.632 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:35:42.661 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:42.686 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:35:42.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:42.689 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:35:42.790 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:35:42.819 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:42.843 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:35:42.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:42.846 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:35:42.938 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:35:42.946 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:35:42.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:42 smithi188 ceph-mon[44101]: osdmap e425: 12 total, 11 up, 9 in 2024-04-23T16:35:42.981 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:43.004 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:35:43.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:43.008 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:35:43.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:42 smithi103 ceph-mon[39629]: osdmap e425: 12 total, 11 up, 9 in 2024-04-23T16:35:43.109 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:35:43.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:42 smithi148 ceph-mon[44520]: osdmap e425: 12 total, 11 up, 9 in 2024-04-23T16:35:43.190 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:43.233 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:35:43.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:43.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:35:43.338 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:35:43.368 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:43.397 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:35:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:43.402 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:35:43.503 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:35:43.564 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:43.603 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:35:43.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:43.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:35:43.709 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:35:43.738 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:43.830 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:43 smithi148 ceph-mon[44520]: pgmap v1209: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.9 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 679 op/s; 12083/127734 objects degraded (9.460%); 9.2 MiB/s, 147 objects/s recovering 2024-04-23T16:35:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:43 smithi103 ceph-mon[39629]: pgmap v1209: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.9 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 679 op/s; 12083/127734 objects degraded (9.460%); 9.2 MiB/s, 147 objects/s recovering 2024-04-23T16:35:44.174 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:35:44.188 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:35:44.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:44.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:35:44.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:43 smithi188 ceph-mon[44101]: pgmap v1209: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.9 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 679 op/s; 12083/127734 objects degraded (9.460%); 9.2 MiB/s, 147 objects/s recovering 2024-04-23T16:35:44.292 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:35:44.332 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:44.366 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:35:44.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:44.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:35:44.475 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:35:44.508 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:44.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:44 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 12083/127734 objects degraded (9.460%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:45.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:44 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 12083/127734 objects degraded (9.460%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:45.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:44 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 12083/127734 objects degraded (9.460%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:45.484 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:35:45.485 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:35:45.708 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:35:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:45 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:35:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:45 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:35:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:45 smithi103 ceph-mon[39629]: pgmap v1210: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.8 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 813 op/s; 11814/123278 objects degraded (9.583%); 7.8 MiB/s, 124 objects/s recovering 2024-04-23T16:35:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:45 smithi103 ceph-mon[39629]: osdmap e426: 12 total, 11 up, 9 in 2024-04-23T16:35:46.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:45 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:35:46.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:45 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:35:46.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:45 smithi148 ceph-mon[44520]: pgmap v1210: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.8 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 813 op/s; 11814/123278 objects degraded (9.583%); 7.8 MiB/s, 124 objects/s recovering 2024-04-23T16:35:46.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:45 smithi148 ceph-mon[44520]: osdmap e426: 12 total, 11 up, 9 in 2024-04-23T16:35:46.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:45 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:35:46.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:45 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:35:46.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:45 smithi188 ceph-mon[44101]: pgmap v1210: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.8 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 813 op/s; 11814/123278 objects degraded (9.583%); 7.8 MiB/s, 124 objects/s recovering 2024-04-23T16:35:46.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:45 smithi188 ceph-mon[44101]: osdmap e426: 12 total, 11 up, 9 in 2024-04-23T16:35:46.302 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1211,"stamp":"2024-04-23T16:35:45.192019+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3959816760,"num_objects":60416,"num_object_clones":0,"num_object_copies":120832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60416,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":246778,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17639,"num_bytes_recovered":1156125216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162338,"ondisk_log_size":162338,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":10569788,"kb_used_data":7070896,"kb_used_omap":14,"kb_used_meta":3498545,"kb_avail":833136580,"statfs":{"total":863955320832,"available":853131857920,"internally_reserved":0,"allocated":7240597504,"data_stored":7237120426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3582511087},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-474939392,"num_objects":-7247,"num_object_clones":0,"num_object_copies":-14494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7247,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1493,"num_bytes_recovered":97845248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.967353"},"pg_stats":[{"pgid":"5.f","version":"425'15605","reported_seq":15850,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:44.969182+0000","last_change":"2024-04-23T16:33:17.526419+0000","last_active":"2024-04-23T16:35:44.969182+0000","last_peered":"2024-04-23T16:35:44.969182+0000","last_clean":"2024-04-23T16:35:44.969182+0000","last_became_active":"2024-04-23T16:31:29.140923+0000","last_became_peered":"2024-04-23T16:31:29.140923+0000","last_unstale":"2024-04-23T16:35:44.969182+0000","last_undegraded":"2024-04-23T16:35:44.969182+0000","last_fullsized":"2024-04-23T16:35:44.969182+0000","mapping_epoch":359,"log_start":"374'5600","ondisk_log_start":"374'5600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9683","last_scrub_stamp":"2024-04-23T16:33:17.526283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.526283+0000","objects_scrubbed":9683,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246480896,"num_objects":3761,"num_object_clones":0,"num_object_copies":7522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15605,"num_write_kb":619712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"425'15484","reported_seq":15855,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:44.292355+0000","last_change":"2024-04-23T16:35:26.407358+0000","last_active":"2024-04-23T16:35:44.292355+0000","last_peered":"2024-04-23T16:35:44.292355+0000","last_clean":"2024-04-23T16:35:44.292355+0000","last_became_active":"2024-04-23T16:31:29.137890+0000","last_became_peered":"2024-04-23T16:31:29.137890+0000","last_unstale":"2024-04-23T16:35:44.292355+0000","last_undegraded":"2024-04-23T16:35:44.292355+0000","last_fullsized":"2024-04-23T16:35:44.292355+0000","mapping_epoch":359,"log_start":"374'5400","ondisk_log_start":"374'5400","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'14582","last_scrub_stamp":"2024-04-23T16:35:26.407093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:26.407093+0000","objects_scrubbed":4692,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:38.641004+0000","scrub_duration":544,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247726080,"num_objects":3780,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15484,"num_write_kb":616448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"425'15572","reported_seq":27518,"reported_epoch":425,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:35:43.165217+0000","last_change":"2024-04-23T16:33:26.432192+0000","last_active":"2024-04-23T16:35:43.165217+0000","last_peered":"2024-04-23T16:35:43.165217+0000","last_clean":"2024-04-23T16:32:59.749530+0000","last_became_active":"2024-04-23T16:33:26.338644+0000","last_became_peered":"2024-04-23T16:33:26.338644+0000","last_unstale":"2024-04-23T16:35:43.165217+0000","last_undegraded":"2024-04-23T16:33:25.874847+0000","last_fullsized":"2024-04-23T16:35:43.165217+0000","mapping_epoch":394,"log_start":"374'5500","ondisk_log_start":"374'5500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8711","last_scrub_stamp":"2024-04-23T16:32:43.928914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.928914+0000","objects_scrubbed":8680,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253624320,"num_objects":3870,"num_object_clones":0,"num_object_copies":7740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15572,"num_write_kb":622144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5852,"num_bytes_recovered":383516672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":["8"],"object_location_counts":[{"shards":"4,8","objects":10},{"shards":"8","objects":3860}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.9","version":"425'15600","reported_seq":15830,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:45.154360+0000","last_change":"2024-04-23T16:32:37.256944+0000","last_active":"2024-04-23T16:35:45.154360+0000","last_peered":"2024-04-23T16:35:45.154360+0000","last_clean":"2024-04-23T16:35:45.154360+0000","last_became_active":"2024-04-23T16:31:29.141551+0000","last_became_peered":"2024-04-23T16:31:29.141551+0000","last_unstale":"2024-04-23T16:35:45.154360+0000","last_undegraded":"2024-04-23T16:35:45.154360+0000","last_fullsized":"2024-04-23T16:35:45.154360+0000","mapping_epoch":359,"log_start":"374'5500","ondisk_log_start":"374'5500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8156","last_scrub_stamp":"2024-04-23T16:32:37.256787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:37.256787+0000","objects_scrubbed":8082,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243793920,"num_objects":3720,"num_object_clones":0,"num_object_copies":7440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15600,"num_write_kb":618240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"424'14914","reported_seq":15228,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-23T16:35:41.715156+0000","last_change":"2024-04-23T16:34:39.934502+0000","last_active":"2024-04-23T16:35:41.715156+0000","last_peered":"2024-04-23T16:35:41.715156+0000","last_clean":"2024-04-23T16:35:41.715156+0000","last_became_active":"2024-04-23T16:31:29.139045+0000","last_became_peered":"2024-04-23T16:31:29.139045+0000","last_unstale":"2024-04-23T16:35:41.715156+0000","last_undegraded":"2024-04-23T16:35:41.715156+0000","last_fullsized":"2024-04-23T16:35:41.715156+0000","mapping_epoch":359,"log_start":"373'4900","ondisk_log_start":"373'4900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'12219","last_scrub_stamp":"2024-04-23T16:34:39.934190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:39.934190+0000","objects_scrubbed":6529,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:59.649413+0000","scrub_duration":810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250347520,"num_objects":3820,"num_object_clones":0,"num_object_copies":7640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14914,"num_write_kb":599488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"425'19002","reported_seq":19445,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:44.278278+0000","last_change":"2024-04-23T16:35:32.474057+0000","last_active":"2024-04-23T16:35:44.278278+0000","last_peered":"2024-04-23T16:35:44.278278+0000","last_clean":"2024-04-23T16:35:44.278278+0000","last_became_active":"2024-04-23T16:31:29.310906+0000","last_became_peered":"2024-04-23T16:31:29.310906+0000","last_unstale":"2024-04-23T16:35:44.278278+0000","last_undegraded":"2024-04-23T16:35:44.278278+0000","last_fullsized":"2024-04-23T16:35:44.278278+0000","mapping_epoch":359,"log_start":"384'8904","ondisk_log_start":"384'8904","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'18441","last_scrub_stamp":"2024-04-23T16:35:32.473828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:32.473828+0000","objects_scrubbed":4251,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:57.506974+0000","scrub_duration":548,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241106944,"num_objects":3679,"num_object_clones":0,"num_object_copies":7358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15441,"num_write_kb":611840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1112,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:41.813681+0000","last_change":"2024-04-23T16:34:44.725424+0000","last_active":"2024-04-23T16:35:41.813681+0000","last_peered":"2024-04-23T16:35:41.813681+0000","last_clean":"2024-04-23T16:35:41.813681+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:35:41.813681+0000","last_undegraded":"2024-04-23T16:35:41.813681+0000","last_fullsized":"2024-04-23T16:35:41.813681+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:44.725324+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:44.725324+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:14.224827+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"425'15660","reported_seq":15891,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:44.459525+0000","last_change":"2024-04-23T16:32:38.103756+0000","last_active":"2024-04-23T16:35:44.459525+0000","last_peered":"2024-04-23T16:35:44.459525+0000","last_clean":"2024-04-23T16:35:44.459525+0000","last_became_active":"2024-04-23T16:31:29.141640+0000","last_became_peered":"2024-04-23T16:31:29.141640+0000","last_unstale":"2024-04-23T16:35:44.459525+0000","last_undegraded":"2024-04-23T16:35:44.459525+0000","last_fullsized":"2024-04-23T16:35:44.459525+0000","mapping_epoch":359,"log_start":"374'5600","ondisk_log_start":"374'5600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8300","last_scrub_stamp":"2024-04-23T16:32:38.103547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:38.103547+0000","objects_scrubbed":8231,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248774656,"num_objects":3796,"num_object_clones":0,"num_object_copies":7592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15660,"num_write_kb":622592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1106,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:41.815623+0000","last_change":"2024-04-23T16:33:25.851379+0000","last_active":"2024-04-23T16:35:41.815623+0000","last_peered":"2024-04-23T16:35:41.815623+0000","last_clean":"2024-04-23T16:35:41.815623+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:35:41.815623+0000","last_undegraded":"2024-04-23T16:35:41.815623+0000","last_fullsized":"2024-04-23T16:35:41.815623+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.498820+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.498820+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"424'15172","reported_seq":15490,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-23T16:35:41.754709+0000","last_change":"2024-04-23T16:34:41.375046+0000","last_active":"2024-04-23T16:35:41.754709+0000","last_peered":"2024-04-23T16:35:41.754709+0000","last_clean":"2024-04-23T16:35:41.754709+0000","last_became_active":"2024-04-23T16:31:29.140251+0000","last_became_peered":"2024-04-23T16:31:29.140251+0000","last_unstale":"2024-04-23T16:35:41.754709+0000","last_undegraded":"2024-04-23T16:35:41.754709+0000","last_fullsized":"2024-04-23T16:35:41.754709+0000","mapping_epoch":359,"log_start":"373'5100","ondisk_log_start":"373'5100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12511","last_scrub_stamp":"2024-04-23T16:34:41.374888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:41.374888+0000","objects_scrubbed":6643,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:07.291855+0000","scrub_duration":1271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259457048,"num_objects":3960,"num_object_clones":0,"num_object_copies":7920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3960,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15172,"num_write_kb":612161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1097,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:41.843246+0000","last_change":"2024-04-23T16:32:29.517947+0000","last_active":"2024-04-23T16:35:41.843246+0000","last_peered":"2024-04-23T16:35:41.843246+0000","last_clean":"2024-04-23T16:35:41.843246+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:35:41.843246+0000","last_undegraded":"2024-04-23T16:35:41.843246+0000","last_fullsized":"2024-04-23T16:35:41.843246+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:32:29.517800+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:32:29.517800+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"425'15621","reported_seq":27732,"reported_epoch":425,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:35:44.969799+0000","last_change":"2024-04-23T16:33:26.507399+0000","last_active":"2024-04-23T16:35:44.969799+0000","last_peered":"2024-04-23T16:35:44.969799+0000","last_clean":"2024-04-23T16:32:59.715575+0000","last_became_active":"2024-04-23T16:33:26.458611+0000","last_became_peered":"2024-04-23T16:33:26.458611+0000","last_unstale":"2024-04-23T16:35:44.969799+0000","last_undegraded":"2024-04-23T16:33:25.895472+0000","last_fullsized":"2024-04-23T16:35:44.969799+0000","mapping_epoch":394,"log_start":"374'5600","ondisk_log_start":"374'5600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8607","last_scrub_stamp":"2024-04-23T16:32:43.130806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.130806+0000","objects_scrubbed":8577,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246349824,"num_objects":3759,"num_object_clones":0,"num_object_copies":7518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15621,"num_write_kb":620160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5935,"num_bytes_recovered":388956160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":3752},{"shards":"5,7","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1111,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:41.813358+0000","last_change":"2024-04-23T16:34:47.438492+0000","last_active":"2024-04-23T16:35:41.813358+0000","last_peered":"2024-04-23T16:35:41.813358+0000","last_clean":"2024-04-23T16:35:41.813358+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:35:41.813358+0000","last_undegraded":"2024-04-23T16:35:41.813358+0000","last_fullsized":"2024-04-23T16:35:41.813358+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:47.438356+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:47.438356+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:13.051838+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"425'15210","reported_seq":15454,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:44.941456+0000","last_change":"2024-04-23T16:33:18.576077+0000","last_active":"2024-04-23T16:35:44.941456+0000","last_peered":"2024-04-23T16:35:44.941456+0000","last_clean":"2024-04-23T16:35:44.941456+0000","last_became_active":"2024-04-23T16:31:29.139709+0000","last_became_peered":"2024-04-23T16:31:29.139709+0000","last_unstale":"2024-04-23T16:35:44.941456+0000","last_undegraded":"2024-04-23T16:35:44.941456+0000","last_fullsized":"2024-04-23T16:35:44.941456+0000","mapping_epoch":359,"log_start":"373'5200","ondisk_log_start":"373'5200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9472","last_scrub_stamp":"2024-04-23T16:33:18.575945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.575945+0000","objects_scrubbed":9472,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244711424,"num_objects":3734,"num_object_clones":0,"num_object_copies":7468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15210,"num_write_kb":606208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1112,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:41.824309+0000","last_change":"2024-04-23T16:35:34.878630+0000","last_active":"2024-04-23T16:35:41.824309+0000","last_peered":"2024-04-23T16:35:41.824309+0000","last_clean":"2024-04-23T16:35:41.824309+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:35:41.824309+0000","last_undegraded":"2024-04-23T16:35:41.824309+0000","last_fullsized":"2024-04-23T16:35:41.824309+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:35:34.878536+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:35:34.878536+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:52.503403+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"425'15587","reported_seq":15910,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:44.126119+0000","last_change":"2024-04-23T16:34:44.348696+0000","last_active":"2024-04-23T16:35:44.126119+0000","last_peered":"2024-04-23T16:35:44.126119+0000","last_clean":"2024-04-23T16:35:44.126119+0000","last_became_active":"2024-04-23T16:31:29.139606+0000","last_became_peered":"2024-04-23T16:31:29.139606+0000","last_unstale":"2024-04-23T16:35:44.126119+0000","last_undegraded":"2024-04-23T16:35:44.126119+0000","last_fullsized":"2024-04-23T16:35:44.126119+0000","mapping_epoch":359,"log_start":"374'5500","ondisk_log_start":"374'5500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12750","last_scrub_stamp":"2024-04-23T16:34:44.348550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:44.348550+0000","objects_scrubbed":6711,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:02.747435+0000","scrub_duration":1024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252510208,"num_objects":3853,"num_object_clones":0,"num_object_copies":7706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15587,"num_write_kb":622080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1111,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:41.813753+0000","last_change":"2024-04-23T16:34:51.848140+0000","last_active":"2024-04-23T16:35:41.813753+0000","last_peered":"2024-04-23T16:35:41.813753+0000","last_clean":"2024-04-23T16:35:41.813753+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:35:41.813753+0000","last_undegraded":"2024-04-23T16:35:41.813753+0000","last_fullsized":"2024-04-23T16:35:41.813753+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:51.847992+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:34:51.847992+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:18.910354+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.2","version":"425'15305","reported_seq":15679,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:44.566471+0000","last_change":"2024-04-23T16:35:32.569280+0000","last_active":"2024-04-23T16:35:44.566471+0000","last_peered":"2024-04-23T16:35:44.566471+0000","last_clean":"2024-04-23T16:35:44.566471+0000","last_became_active":"2024-04-23T16:31:29.138430+0000","last_became_peered":"2024-04-23T16:31:29.138430+0000","last_unstale":"2024-04-23T16:35:44.566471+0000","last_undegraded":"2024-04-23T16:35:44.566471+0000","last_fullsized":"2024-04-23T16:35:44.566471+0000","mapping_epoch":359,"log_start":"374'5300","ondisk_log_start":"374'5300","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'14723","last_scrub_stamp":"2024-04-23T16:35:32.569137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:32.569137+0000","objects_scrubbed":4294,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:34.926251+0000","scrub_duration":850,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241893376,"num_objects":3691,"num_object_clones":0,"num_object_copies":7382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15305,"num_write_kb":607872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1112,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:41.813861+0000","last_change":"2024-04-23T16:34:37.693617+0000","last_active":"2024-04-23T16:35:41.813861+0000","last_peered":"2024-04-23T16:35:41.813861+0000","last_clean":"2024-04-23T16:35:41.813861+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:35:41.813861+0000","last_undegraded":"2024-04-23T16:35:41.813861+0000","last_fullsized":"2024-04-23T16:35:41.813861+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:37.693482+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:37.693482+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:49.920873+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"425'17581","reported_seq":18030,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:44.111448+0000","last_change":"2024-04-23T16:34:40.604268+0000","last_active":"2024-04-23T16:35:44.111448+0000","last_peered":"2024-04-23T16:35:44.111448+0000","last_clean":"2024-04-23T16:35:44.111448+0000","last_became_active":"2024-04-23T16:31:29.139398+0000","last_became_peered":"2024-04-23T16:31:29.139398+0000","last_unstale":"2024-04-23T16:35:44.111448+0000","last_undegraded":"2024-04-23T16:35:44.111448+0000","last_fullsized":"2024-04-23T16:35:44.111448+0000","mapping_epoch":359,"log_start":"380'7500","ondisk_log_start":"380'7500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'14807","last_scrub_stamp":"2024-04-23T16:34:40.604136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:40.604136+0000","objects_scrubbed":6625,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:04.702557+0000","scrub_duration":1231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250413056,"num_objects":3821,"num_object_clones":0,"num_object_copies":7642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15307,"num_write_kb":612096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1112,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:41.824377+0000","last_change":"2024-04-23T16:35:33.875035+0000","last_active":"2024-04-23T16:35:41.824377+0000","last_peered":"2024-04-23T16:35:41.824377+0000","last_clean":"2024-04-23T16:35:41.824377+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:35:41.824377+0000","last_undegraded":"2024-04-23T16:35:41.824377+0000","last_fullsized":"2024-04-23T16:35:41.824377+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:35:33.874904+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:35:33.874904+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:59.611256+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"425'15234","reported_seq":15468,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:44.427497+0000","last_change":"2024-04-23T16:32:43.947374+0000","last_active":"2024-04-23T16:35:44.427497+0000","last_peered":"2024-04-23T16:35:44.427497+0000","last_clean":"2024-04-23T16:35:44.427497+0000","last_became_active":"2024-04-23T16:31:29.138968+0000","last_became_peered":"2024-04-23T16:31:29.138968+0000","last_unstale":"2024-04-23T16:35:44.427497+0000","last_undegraded":"2024-04-23T16:35:44.427497+0000","last_fullsized":"2024-04-23T16:35:44.427497+0000","mapping_epoch":359,"log_start":"373'5200","ondisk_log_start":"373'5200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8482","last_scrub_stamp":"2024-04-23T16:32:43.947177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.947177+0000","objects_scrubbed":8428,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239861760,"num_objects":3660,"num_object_clones":0,"num_object_copies":7320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15234,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1001,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-04-23T16:35:37.226536+0000","last_change":"2024-04-23T16:34:51.248895+0000","last_active":"2024-04-23T16:35:37.226536+0000","last_peered":"2024-04-23T16:35:37.226536+0000","last_clean":"2024-04-23T16:35:37.226536+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:35:37.226536+0000","last_undegraded":"2024-04-23T16:35:37.226536+0000","last_fullsized":"2024-04-23T16:35:37.226536+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:51.248783+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:51.248783+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:13.233910+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"425'15659","reported_seq":15892,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:44.427884+0000","last_change":"2024-04-23T16:32:39.103230+0000","last_active":"2024-04-23T16:35:44.427884+0000","last_peered":"2024-04-23T16:35:44.427884+0000","last_clean":"2024-04-23T16:35:44.427884+0000","last_became_active":"2024-04-23T16:31:29.310246+0000","last_became_peered":"2024-04-23T16:31:29.310246+0000","last_unstale":"2024-04-23T16:35:44.427884+0000","last_undegraded":"2024-04-23T16:35:44.427884+0000","last_fullsized":"2024-04-23T16:35:44.427884+0000","mapping_epoch":359,"log_start":"374'5600","ondisk_log_start":"374'5600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8325","last_scrub_stamp":"2024-04-23T16:32:39.103059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:39.103059+0000","objects_scrubbed":8272,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249233408,"num_objects":3803,"num_object_clones":0,"num_object_copies":7606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15659,"num_write_kb":622784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":925,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:41.824261+0000","last_change":"2024-04-23T16:34:56.453746+0000","last_active":"2024-04-23T16:35:41.824261+0000","last_peered":"2024-04-23T16:35:41.824261+0000","last_clean":"2024-04-23T16:35:41.824261+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:35:41.824261+0000","last_undegraded":"2024-04-23T16:35:41.824261+0000","last_fullsized":"2024-04-23T16:35:41.824261+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:56.453617+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:34:56.453617+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:08.659133+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"425'15407","reported_seq":27344,"reported_epoch":425,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:35:44.990946+0000","last_change":"2024-04-23T16:33:26.132487+0000","last_active":"2024-04-23T16:35:44.990946+0000","last_peered":"2024-04-23T16:35:44.990946+0000","last_clean":"2024-04-23T16:32:59.695133+0000","last_became_active":"2024-04-23T16:33:26.078062+0000","last_became_peered":"2024-04-23T16:33:26.078062+0000","last_unstale":"2024-04-23T16:35:44.990946+0000","last_undegraded":"2024-04-23T16:33:25.892009+0000","last_fullsized":"2024-04-23T16:35:44.990946+0000","mapping_epoch":394,"log_start":"374'5400","ondisk_log_start":"374'5400","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'7968","last_scrub_stamp":"2024-04-23T16:32:36.102122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:36.102122+0000","objects_scrubbed":7926,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242941952,"num_objects":3707,"num_object_clones":0,"num_object_copies":7414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3702,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15407,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5851,"num_bytes_recovered":383451136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":3702},{"shards":"5,8","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1101,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-23T16:35:41.843510+0000","last_change":"2024-04-23T16:33:17.354409+0000","last_active":"2024-04-23T16:35:41.843510+0000","last_peered":"2024-04-23T16:35:41.843510+0000","last_clean":"2024-04-23T16:35:41.843510+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:35:41.843510+0000","last_undegraded":"2024-04-23T16:35:41.843510+0000","last_fullsized":"2024-04-23T16:35:41.843510+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:17.354263+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.354263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3959226392,"num_objects":60414,"num_object_clones":0,"num_object_copies":120828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60414,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":246778,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17638,"num_bytes_recovered":1155923968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9069404160,"data_stored":9069396016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160809,"ondisk_log_size":160809,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056885,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1002032,"kb_used_data":501608,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":92743120,"statfs":{"total":95995035648,"available":94968954880,"internally_reserved":0,"allocated":513646592,"data_stored":513262542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":512424388},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351227,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612865,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1067640,"kb_used_data":721328,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92677512,"statfs":{"total":95995035648,"available":94901772288,"internally_reserved":0,"allocated":738639872,"data_stored":738253890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":354613696},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841797,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":763580,"kb_used_data":493616,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92981572,"statfs":{"total":95995035648,"available":95213129728,"internally_reserved":0,"allocated":505462784,"data_stored":505076802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168842,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":601976,"kb_used_data":241776,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93143176,"statfs":{"total":95995035648,"available":95378612224,"internally_reserved":0,"allocated":247578624,"data_stored":247192642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":368835018},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463184,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1021048,"kb_used_data":728880,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92724104,"statfs":{"total":95995035648,"available":94949482496,"internally_reserved":0,"allocated":746373120,"data_stored":745987138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":299170246},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137323,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738669,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2113336,"kb_used_data":1689200,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":91631816,"statfs":{"total":95995035648,"available":93830979584,"internally_reserved":0,"allocated":1729740800,"data_stored":1729354818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117277,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":623092,"kb_used_data":258732,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93122060,"statfs":{"total":95995035648,"available":95356989440,"internally_reserved":0,"allocated":264941568,"data_stored":264558038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921048,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1744824,"kb_used_data":1222192,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":92000328,"statfs":{"total":95995035648,"available":94208335872,"internally_reserved":0,"allocated":1251524608,"data_stored":1251138626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757523,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1632260,"kb_used_data":1213564,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92112892,"statfs":{"total":95995035648,"available":94323601408,"internally_reserved":0,"allocated":1242689536,"data_stored":1242295930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1727528960,"data_stored":1727528960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1886846976,"data_stored":1886846976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":262668288,"data_stored":262668288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1238368256,"data_stored":1238368256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1230049280,"data_stored":1230045208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":736362496,"data_stored":736362496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":244056064,"data_stored":244056064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":498139136,"data_stored":498139136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":735707136,"data_stored":735707136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":509677568,"data_stored":509673496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:35:46.304 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.a is not active+clean 2024-04-23T16:35:46.304 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.a', 'version': "425'15572", 'reported_seq': 27518, 'reported_epoch': 425, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:35:43.165217+0000', 'last_change': '2024-04-23T16:33:26.432192+0000', 'last_active': '2024-04-23T16:35:43.165217+0000', 'last_peered': '2024-04-23T16:35:43.165217+0000', 'last_clean': '2024-04-23T16:32:59.749530+0000', 'last_became_active': '2024-04-23T16:33:26.338644+0000', 'last_became_peered': '2024-04-23T16:33:26.338644+0000', 'last_unstale': '2024-04-23T16:35:43.165217+0000', 'last_undegraded': '2024-04-23T16:33:25.874847+0000', 'last_fullsized': '2024-04-23T16:35:43.165217+0000', 'mapping_epoch': 394, 'log_start': "374'5500", 'ondisk_log_start': "374'5500", 'created': 359, 'last_epoch_clean': 360, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "382'8711", 'last_scrub_stamp': '2024-04-23T16:32:43.928914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:31:28.121358+0000', 'last_clean_scrub_stamp': '2024-04-23T16:32:43.928914+0000', 'objects_scrubbed': 8680, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1273, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 253624320, 'num_objects': 3870, 'num_object_clones': 0, 'num_object_copies': 7740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3860, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3870, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15572, 'num_write_kb': 622144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5852, 'num_bytes_recovered': 383516672, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [8, 4], 'acting': [8, 4], 'avail_no_missing': ['8'], 'object_location_counts': [{'shards': '4,8', 'objects': 10}, {'shards': '8', 'objects': 3860}], 'blocked_by': [], 'up_primary': 8, 'acting_primary': 8, 'purged_snaps': []} 2024-04-23T16:35:46.304 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.6 is not active+clean 2024-04-23T16:35:46.304 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.6', 'version': "425'15621", 'reported_seq': 27732, 'reported_epoch': 425, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:35:44.969799+0000', 'last_change': '2024-04-23T16:33:26.507399+0000', 'last_active': '2024-04-23T16:35:44.969799+0000', 'last_peered': '2024-04-23T16:35:44.969799+0000', 'last_clean': '2024-04-23T16:32:59.715575+0000', 'last_became_active': '2024-04-23T16:33:26.458611+0000', 'last_became_peered': '2024-04-23T16:33:26.458611+0000', 'last_unstale': '2024-04-23T16:35:44.969799+0000', 'last_undegraded': '2024-04-23T16:33:25.895472+0000', 'last_fullsized': '2024-04-23T16:35:44.969799+0000', 'mapping_epoch': 394, 'log_start': "374'5600", 'ondisk_log_start': "374'5600", 'created': 359, 'last_epoch_clean': 360, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "382'8607", 'last_scrub_stamp': '2024-04-23T16:32:43.130806+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:31:28.121358+0000', 'last_clean_scrub_stamp': '2024-04-23T16:32:43.130806+0000', 'objects_scrubbed': 8577, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1484, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 246349824, 'num_objects': 3759, 'num_object_clones': 0, 'num_object_copies': 7518, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3752, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3759, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15621, 'num_write_kb': 620160, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5935, 'num_bytes_recovered': 388956160, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': ['5'], 'object_location_counts': [{'shards': '5', 'objects': 3752}, {'shards': '5,7', 'objects': 7}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-23T16:35:46.305 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.4 is not active+clean 2024-04-23T16:35:46.305 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.4', 'version': "425'15407", 'reported_seq': 27344, 'reported_epoch': 425, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:35:44.990946+0000', 'last_change': '2024-04-23T16:33:26.132487+0000', 'last_active': '2024-04-23T16:35:44.990946+0000', 'last_peered': '2024-04-23T16:35:44.990946+0000', 'last_clean': '2024-04-23T16:32:59.695133+0000', 'last_became_active': '2024-04-23T16:33:26.078062+0000', 'last_became_peered': '2024-04-23T16:33:26.078062+0000', 'last_unstale': '2024-04-23T16:35:44.990946+0000', 'last_undegraded': '2024-04-23T16:33:25.892009+0000', 'last_fullsized': '2024-04-23T16:35:44.990946+0000', 'mapping_epoch': 394, 'log_start': "374'5400", 'ondisk_log_start': "374'5400", 'created': 359, 'last_epoch_clean': 360, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "380'7968", 'last_scrub_stamp': '2024-04-23T16:32:36.102122+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:31:28.121358+0000', 'last_clean_scrub_stamp': '2024-04-23T16:32:36.102122+0000', 'objects_scrubbed': 7926, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1466, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 242941952, 'num_objects': 3707, 'num_object_clones': 0, 'num_object_copies': 7414, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3702, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3707, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15407, 'num_write_kb': 611648, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5851, 'num_bytes_recovered': 383451136, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 8], 'acting': [5, 8], 'avail_no_missing': ['5'], 'object_location_counts': [{'shards': '5', 'objects': 3702}, {'shards': '5,8', 'objects': 5}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-23T16:35:46.305 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-23T16:35:47.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:46 smithi103 ceph-mon[39629]: from='client.16101 v1:172.21.15.103:0/1918635028' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:35:47.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:46 smithi148 ceph-mon[44520]: from='client.16101 v1:172.21.15.103:0/1918635028' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:35:47.144 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:35:47.158 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:35:47.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:47.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:35:47.168 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:35:47.193 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:35:47.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:46 smithi188 ceph-mon[44101]: from='client.16101 v1:172.21.15.103:0/1918635028' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:35:47.229 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:35:47.261 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:35:47.266 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:35:47.288 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:35:47.298 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:47.315 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:35:47.319 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:35:47.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:47.321 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:35:47.336 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:35:47.360 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:35:47.459 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:35:47.461 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:35:47.461 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:35:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:35:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:35:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:35:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:35:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:35:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:35:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:35:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:35:47.462 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:35:47.462 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:35:47.462 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:35:47.462 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:35:47.518 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:47.549 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:35:47.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:47.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:35:47.663 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:35:47.692 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:47.842 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:47 smithi148 ceph-mon[44520]: pgmap v1212: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.7 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 873 op/s; 11314/120832 objects degraded (9.363%); 9.7 MiB/s, 154 objects/s recovering 2024-04-23T16:35:47.842 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:47 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2818996486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:35:47.842 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:47.842 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:47.842 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:47.843 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:47.843 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:47.843 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:47.843 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:47.843 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:47.843 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.022 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:47 smithi103 ceph-mon[39629]: pgmap v1212: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.7 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 873 op/s; 11314/120832 objects degraded (9.363%); 9.7 MiB/s, 154 objects/s recovering 2024-04-23T16:35:48.022 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:47 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2818996486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:35:48.022 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.022 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.022 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:47 smithi188 ceph-mon[44101]: pgmap v1212: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.7 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 873 op/s; 11314/120832 objects degraded (9.363%); 9.7 MiB/s, 154 objects/s recovering 2024-04-23T16:35:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:47 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2818996486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:35:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.642 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:35:48.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:48.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:35:48.747 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:35:48.833 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:48.845 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.845 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:48.888 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:35:48.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:48.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:35:48.997 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:35:49.054 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:49.101 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:35:49.104 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:49.104 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:49.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:49.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:35:49.206 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:35:49.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:49.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:35:49.270 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:49.322 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:35:49.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:49.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:35:49.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:35:49.468 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:49.526 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:35:49.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:49.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:35:49.638 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:35:49.695 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:49.737 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:35:49.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:49.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:35:49.845 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:35:49.874 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:49.895 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:35:49.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:49.897 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:35:49.999 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:35:50.028 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:35:50.028 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:35:50.039 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:49 smithi148 ceph-mon[44520]: pgmap v1213: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.6 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 581 op/s; 11088/119362 objects degraded (9.289%); 7.8 MiB/s, 124 objects/s recovering 2024-04-23T16:35:50.039 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:49 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 11088/119362 objects degraded (9.289%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:50.056 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:50.095 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:35:50.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:50.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:35:50.206 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:35:50.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:49 smithi188 ceph-mon[44101]: pgmap v1213: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.6 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 581 op/s; 11088/119362 objects degraded (9.289%); 7.8 MiB/s, 124 objects/s recovering 2024-04-23T16:35:50.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:49 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 11088/119362 objects degraded (9.289%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:50.281 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:50.319 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:35:50.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:50.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:35:50.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:49 smithi103 ceph-mon[39629]: pgmap v1213: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.6 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 581 op/s; 11088/119362 objects degraded (9.289%); 7.8 MiB/s, 124 objects/s recovering 2024-04-23T16:35:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:49 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 11088/119362 objects degraded (9.289%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:50.424 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:35:50.452 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:50.472 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:35:50.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:50.475 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:35:50.539 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:35:50.545 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:35:50.545 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:35:50.545 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-23T16:35:50.545 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-23T16:35:50.545 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:35:50.576 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:35:50.606 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:50.866 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:35:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:50 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3372223796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:35:51.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:50 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3372223796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:35:51.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:50 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3372223796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:35:51.356 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:35:51.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:51.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:35:51.460 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:35:51.489 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:51.510 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:35:51.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:51.513 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:35:51.614 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:35:51.686 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:51.713 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:35:51.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:51.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:35:51.825 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:51.855 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:51.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:51 smithi188 ceph-mon[44101]: pgmap v1214: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.5 GiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 820 op/s; 10557/113078 objects degraded (9.336%); 11 MiB/s, 170 objects/s recovering 2024-04-23T16:35:51.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:51 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:35:51.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:51 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3372223796' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:35:51.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:51 smithi188 ceph-mon[44101]: osdmap e427: 12 total, 11 up, 9 in 2024-04-23T16:35:51.875 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:51.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:51.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:51.979 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:35:52.030 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:35:52.056 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:51 smithi148 ceph-mon[44520]: pgmap v1214: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.5 GiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 820 op/s; 10557/113078 objects degraded (9.336%); 11 MiB/s, 170 objects/s recovering 2024-04-23T16:35:52.056 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:51 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:35:52.057 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:51 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3372223796' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:35:52.057 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:51 smithi148 ceph-mon[44520]: osdmap e427: 12 total, 11 up, 9 in 2024-04-23T16:35:52.059 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:52.099 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:35:52.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:52.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:35:52.216 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:52.253 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:51 smithi103 ceph-mon[39629]: pgmap v1214: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.5 GiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 820 op/s; 10557/113078 objects degraded (9.336%); 11 MiB/s, 170 objects/s recovering 2024-04-23T16:35:52.253 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:51 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:35:52.253 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:51 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3372223796' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:35:52.254 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:51 smithi103 ceph-mon[39629]: osdmap e427: 12 total, 11 up, 9 in 2024-04-23T16:35:52.255 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:53.012 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-23T16:35:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:52 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:35:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:52 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:35:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:35:53.204 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:52 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:35:53.205 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:52 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:35:53.205 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:35:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:52 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:35:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:52 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:35:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:35:53.546 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:53.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:53.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:53.650 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:35:53.701 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:53.735 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:35:53.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:53.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:35:53.843 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:53.874 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:53.977 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:53.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:53.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:54.082 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:54.109 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:54.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:53 smithi148 ceph-mon[44520]: pgmap v1216: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.4 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 697 op/s; 10557/112122 objects degraded (9.416%); 9.8 MiB/s, 157 objects/s recovering 2024-04-23T16:35:54.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:53 smithi148 ceph-mon[44520]: osdmap e428: 12 total, 11 up, 9 in 2024-04-23T16:35:54.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:53 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:35:54.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:53 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:35:54.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:53 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:35:54.170 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:35:54.170 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10557/112122 objects degraded (9.416%), 3 pgs degraded","count":3},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1536,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":428,"num_osds":12,"num_up_osds":11,"osd_up_since":1713889979,"num_in_osds":9,"osd_in_since":1713890004,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":24},{"state_name":"active+recovering+degraded","count":3}],"num_pgs":27,"num_pools":2,"num_objects":56061,"data_bytes":3674407480,"bytes_used":10330521600,"bytes_avail":853624799232,"bytes_total":863955320832,"degraded_objects":10557,"degraded_total":112122,"degraded_ratio":0.094156365387702681,"recovering_objects_per_sec":157,"recovering_bytes_per_sec":10295219,"recovering_keys_per_sec":0,"num_objects_recovered":1257,"num_bytes_recovered":82378752,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":697},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.103:8443/"}},"servicemap":{"epoch":34,"modified":"2024-04-23T16:35:30.733564+0000","services":{}},"progress_events":{}} 2024-04-23T16:35:54.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:53 smithi188 ceph-mon[44101]: pgmap v1216: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.4 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 697 op/s; 10557/112122 objects degraded (9.416%); 9.8 MiB/s, 157 objects/s recovering 2024-04-23T16:35:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:53 smithi188 ceph-mon[44101]: osdmap e428: 12 total, 11 up, 9 in 2024-04-23T16:35:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:53 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:35:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:53 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:35:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:53 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:35:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:53 smithi103 ceph-mon[39629]: pgmap v1216: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.4 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 697 op/s; 10557/112122 objects degraded (9.416%); 9.8 MiB/s, 157 objects/s recovering 2024-04-23T16:35:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:53 smithi103 ceph-mon[39629]: osdmap e428: 12 total, 11 up, 9 in 2024-04-23T16:35:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:35:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:35:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:53 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:35:54.362 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:54.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:54.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:54.467 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:35:54.541 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:54.561 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:35:54.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:54.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:35:54.671 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:35:54.737 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:54.774 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:35:54.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:54.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:35:54.876 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:35:54.928 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:54.974 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:35:54.996 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:35:55.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:55.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:35:55.012 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-23T16:35:55.012 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:35:55.106 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:35:55.174 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:54 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1282686371' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:35:55.175 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:54 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 10557/112122 objects degraded (9.416%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:55.175 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:54 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:35:55.175 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:54 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:35:55.175 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:55.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:54 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1282686371' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:35:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:54 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 10557/112122 objects degraded (9.416%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:54 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:35:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:54 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:35:55.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:55.226 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:35:55.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:35:55.277 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:54 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1282686371' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:35:55.277 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:54 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 10557/112122 objects degraded (9.416%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:35:55.277 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:54 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:35:55.277 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:54 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:35:55.327 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:35:55.358 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:55.381 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:35:55.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:55.385 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:35:55.486 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:35:55.533 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:55.578 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:35:55.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:55.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:35:55.692 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:35:55.775 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:55.807 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:35:55.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:55.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:35:55.875 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:35:55.914 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:35:55.915 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:35:55.980 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:55.991 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:35:56.033 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:35:56.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:56.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:35:56.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:55 smithi148 ceph-mon[44520]: pgmap v1218: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.3 GiB data, 9.3 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 818 op/s; 10285/107740 objects degraded (9.546%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:35:56.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:55 smithi148 ceph-mon[44520]: osdmap e429: 12 total, 11 up, 9 in 2024-04-23T16:35:56.146 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:56.179 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:55 smithi103 ceph-mon[39629]: pgmap v1218: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.3 GiB data, 9.3 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 818 op/s; 10285/107740 objects degraded (9.546%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:35:56.179 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:55 smithi103 ceph-mon[39629]: osdmap e429: 12 total, 11 up, 9 in 2024-04-23T16:35:56.180 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:56.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:55 smithi188 ceph-mon[44101]: pgmap v1218: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.3 GiB data, 9.3 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 818 op/s; 10285/107740 objects degraded (9.546%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:35:56.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:55 smithi188 ceph-mon[44101]: osdmap e429: 12 total, 11 up, 9 in 2024-04-23T16:35:56.445 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:56.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:56.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:56.549 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:35:56.608 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:56.661 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:35:56.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:56.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:35:56.780 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:35:56.809 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:56.829 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:35:56.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:56.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:35:56.937 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:35:56.964 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:56.987 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:35:56.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:56.992 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:35:57.093 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:35:57.193 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:57.232 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:35:57.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:57.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:35:57.342 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:57.371 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:57.722 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:57.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:57.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:57.826 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:35:57.854 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:57.875 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:35:57.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:57.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:35:57.982 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:58.006 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:57 smithi103 ceph-mon[39629]: pgmap v1220: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.2 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 650 op/s; 9799/105274 objects degraded (9.308%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:35:58.009 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:58.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:57 smithi188 ceph-mon[44101]: pgmap v1220: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.2 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 650 op/s; 9799/105274 objects degraded (9.308%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:35:58.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:57 smithi148 ceph-mon[44520]: pgmap v1220: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.2 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 650 op/s; 9799/105274 objects degraded (9.308%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:35:58.660 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:58.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:58.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:58.765 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:35:58.829 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:58.872 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:35:58.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:58.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:35:58.984 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:35:59.018 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:59.051 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:35:59.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:59.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:35:59.168 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:35:59.238 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:59.270 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:35:59.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:59.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:35:59.375 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:35:59.445 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:59.502 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:35:59.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:59.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:35:59.611 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:35:59.684 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:59.708 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:35:59.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:59.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:35:59.814 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:35:59.845 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:35:59.868 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:35:59.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:35:59.871 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:35:59.972 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:35:59.987 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:35:59.989 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:36:00.000 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:00.023 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:00.026 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:00.127 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:36:00.206 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:59 smithi148 ceph-mon[44520]: pgmap v1221: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.2 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 590 op/s; 9536/103776 objects degraded (9.189%); 8.1 MiB/s, 129 objects/s recovering 2024-04-23T16:36:00.206 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:35:59 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 9536/103776 objects degraded (9.189%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:00.216 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:00.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:59 smithi188 ceph-mon[44101]: pgmap v1221: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.2 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 590 op/s; 9536/103776 objects degraded (9.189%); 8.1 MiB/s, 129 objects/s recovering 2024-04-23T16:36:00.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:35:59 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 9536/103776 objects degraded (9.189%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:00.252 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:59 smithi103 ceph-mon[39629]: pgmap v1221: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.2 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 590 op/s; 9536/103776 objects degraded (9.189%); 8.1 MiB/s, 129 objects/s recovering 2024-04-23T16:36:00.252 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:35:59 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 9536/103776 objects degraded (9.189%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:00.258 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:36:00.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:00.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:36:00.369 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:36:00.397 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:00.418 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:00.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:00.425 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:00.526 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:36:00.556 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:01.107 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1221,"stamp":"2024-04-23T16:35:58.738784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3400925752,"num_objects":51888,"num_object_clones":0,"num_object_copies":103776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51888,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":255306,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19417,"num_bytes_recovered":1272648224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162466,"ondisk_log_size":162466,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":9616152,"kb_used_data":6105712,"kb_used_omap":14,"kb_used_meta":3509681,"kb_avail":834090216,"statfs":{"total":863955320832,"available":854108381184,"internally_reserved":0,"allocated":6252249088,"data_stored":6248579432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3593914351},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,6,4,9,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-304807936,"num_objects":-4651,"num_object_clones":0,"num_object_copies":-9302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4651,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1021,"num_bytes_recovered":66912256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.875540"},"pg_stats":[{"pgid":"5.f","version":"428'16090","reported_seq":16341,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:55.027655+0000","last_change":"2024-04-23T16:33:17.526419+0000","last_active":"2024-04-23T16:35:55.027655+0000","last_peered":"2024-04-23T16:35:55.027655+0000","last_clean":"2024-04-23T16:35:55.027655+0000","last_became_active":"2024-04-23T16:31:29.140923+0000","last_became_peered":"2024-04-23T16:31:29.140923+0000","last_unstale":"2024-04-23T16:35:55.027655+0000","last_undegraded":"2024-04-23T16:35:55.027655+0000","last_fullsized":"2024-04-23T16:35:55.027655+0000","mapping_epoch":359,"log_start":"376'6000","ondisk_log_start":"376'6000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9683","last_scrub_stamp":"2024-04-23T16:33:17.526283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.526283+0000","objects_scrubbed":9683,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214695936,"num_objects":3276,"num_object_clones":0,"num_object_copies":6552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16090,"num_write_kb":619712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"428'15988","reported_seq":16365,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:54.279564+0000","last_change":"2024-04-23T16:35:26.407358+0000","last_active":"2024-04-23T16:35:54.279564+0000","last_peered":"2024-04-23T16:35:54.279564+0000","last_clean":"2024-04-23T16:35:54.279564+0000","last_became_active":"2024-04-23T16:31:29.137890+0000","last_became_peered":"2024-04-23T16:31:29.137890+0000","last_unstale":"2024-04-23T16:35:54.279564+0000","last_undegraded":"2024-04-23T16:35:54.279564+0000","last_fullsized":"2024-04-23T16:35:54.279564+0000","mapping_epoch":359,"log_start":"375'5900","ondisk_log_start":"375'5900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'14582","last_scrub_stamp":"2024-04-23T16:35:26.407093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:26.407093+0000","objects_scrubbed":4692,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:38.641004+0000","scrub_duration":544,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214695936,"num_objects":3276,"num_object_clones":0,"num_object_copies":6552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15988,"num_write_kb":616448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"429'16333","reported_seq":29810,"reported_epoch":429,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:35:58.183976+0000","last_change":"2024-04-23T16:33:26.432192+0000","last_active":"2024-04-23T16:35:58.183976+0000","last_peered":"2024-04-23T16:35:58.183976+0000","last_clean":"2024-04-23T16:32:59.749530+0000","last_became_active":"2024-04-23T16:33:26.338644+0000","last_became_peered":"2024-04-23T16:33:26.338644+0000","last_unstale":"2024-04-23T16:35:58.183976+0000","last_undegraded":"2024-04-23T16:33:25.874847+0000","last_fullsized":"2024-04-23T16:35:58.183976+0000","mapping_epoch":394,"log_start":"376'6300","ondisk_log_start":"376'6300","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8711","last_scrub_stamp":"2024-04-23T16:32:43.928914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.928914+0000","objects_scrubbed":8680,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203751424,"num_objects":3109,"num_object_clones":0,"num_object_copies":6218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16333,"num_write_kb":622144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6613,"num_bytes_recovered":433389568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":["8"],"object_location_counts":[{"shards":"4,8","objects":10},{"shards":"8","objects":3099}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.9","version":"428'16075","reported_seq":16312,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:55.101720+0000","last_change":"2024-04-23T16:32:37.256944+0000","last_active":"2024-04-23T16:35:55.101720+0000","last_peered":"2024-04-23T16:35:55.101720+0000","last_clean":"2024-04-23T16:35:55.101720+0000","last_became_active":"2024-04-23T16:31:29.141551+0000","last_became_peered":"2024-04-23T16:31:29.141551+0000","last_unstale":"2024-04-23T16:35:55.101720+0000","last_undegraded":"2024-04-23T16:35:55.101720+0000","last_fullsized":"2024-04-23T16:35:55.101720+0000","mapping_epoch":359,"log_start":"376'6000","ondisk_log_start":"376'6000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8156","last_scrub_stamp":"2024-04-23T16:32:37.256787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:37.256787+0000","objects_scrubbed":8082,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212664320,"num_objects":3245,"num_object_clones":0,"num_object_copies":6490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16075,"num_write_kb":618240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"429'15643","reported_seq":15968,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-04-23T16:35:56.666061+0000","last_change":"2024-04-23T16:34:39.934502+0000","last_active":"2024-04-23T16:35:56.666061+0000","last_peered":"2024-04-23T16:35:56.666061+0000","last_clean":"2024-04-23T16:35:56.666061+0000","last_became_active":"2024-04-23T16:31:29.139045+0000","last_became_peered":"2024-04-23T16:31:29.139045+0000","last_unstale":"2024-04-23T16:35:56.666061+0000","last_undegraded":"2024-04-23T16:35:56.666061+0000","last_fullsized":"2024-04-23T16:35:56.666061+0000","mapping_epoch":359,"log_start":"375'5600","ondisk_log_start":"375'5600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'12219","last_scrub_stamp":"2024-04-23T16:34:39.934190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:39.934190+0000","objects_scrubbed":6529,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:59.649413+0000","scrub_duration":810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202571776,"num_objects":3091,"num_object_clones":0,"num_object_copies":6182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15643,"num_write_kb":599488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"428'19467","reported_seq":19916,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:54.279520+0000","last_change":"2024-04-23T16:35:32.474057+0000","last_active":"2024-04-23T16:35:54.279520+0000","last_peered":"2024-04-23T16:35:54.279520+0000","last_clean":"2024-04-23T16:35:54.279520+0000","last_became_active":"2024-04-23T16:31:29.310906+0000","last_became_peered":"2024-04-23T16:31:29.310906+0000","last_unstale":"2024-04-23T16:35:54.279520+0000","last_undegraded":"2024-04-23T16:35:54.279520+0000","last_fullsized":"2024-04-23T16:35:54.279520+0000","mapping_epoch":359,"log_start":"386'9404","ondisk_log_start":"386'9404","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'18441","last_scrub_stamp":"2024-04-23T16:35:32.473828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:32.473828+0000","objects_scrubbed":4251,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:57.506974+0000","scrub_duration":548,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210632704,"num_objects":3214,"num_object_clones":0,"num_object_copies":6428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15906,"num_write_kb":611840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1118,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:52.876835+0000","last_change":"2024-04-23T16:34:44.725424+0000","last_active":"2024-04-23T16:35:52.876835+0000","last_peered":"2024-04-23T16:35:52.876835+0000","last_clean":"2024-04-23T16:35:52.876835+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:35:52.876835+0000","last_undegraded":"2024-04-23T16:35:52.876835+0000","last_fullsized":"2024-04-23T16:35:52.876835+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:44.725324+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:44.725324+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:14.224827+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"428'16151","reported_seq":16388,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:54.488265+0000","last_change":"2024-04-23T16:32:38.103756+0000","last_active":"2024-04-23T16:35:54.488265+0000","last_peered":"2024-04-23T16:35:54.488265+0000","last_clean":"2024-04-23T16:35:54.488265+0000","last_became_active":"2024-04-23T16:31:29.141640+0000","last_became_peered":"2024-04-23T16:31:29.141640+0000","last_unstale":"2024-04-23T16:35:54.488265+0000","last_undegraded":"2024-04-23T16:35:54.488265+0000","last_fullsized":"2024-04-23T16:35:54.488265+0000","mapping_epoch":359,"log_start":"376'6100","ondisk_log_start":"376'6100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8300","last_scrub_stamp":"2024-04-23T16:32:38.103547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:38.103547+0000","objects_scrubbed":8231,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216596480,"num_objects":3305,"num_object_clones":0,"num_object_copies":6610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16151,"num_write_kb":622592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1112,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:52.874785+0000","last_change":"2024-04-23T16:33:25.851379+0000","last_active":"2024-04-23T16:35:52.874785+0000","last_peered":"2024-04-23T16:35:52.874785+0000","last_clean":"2024-04-23T16:35:52.874785+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:35:52.874785+0000","last_undegraded":"2024-04-23T16:35:52.874785+0000","last_fullsized":"2024-04-23T16:35:52.874785+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.498820+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.498820+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"429'15914","reported_seq":16242,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-04-23T16:35:56.788911+0000","last_change":"2024-04-23T16:34:41.375046+0000","last_active":"2024-04-23T16:35:56.788911+0000","last_peered":"2024-04-23T16:35:56.788911+0000","last_clean":"2024-04-23T16:35:56.788911+0000","last_became_active":"2024-04-23T16:31:29.140251+0000","last_became_peered":"2024-04-23T16:31:29.140251+0000","last_unstale":"2024-04-23T16:35:56.788911+0000","last_undegraded":"2024-04-23T16:35:56.788911+0000","last_fullsized":"2024-04-23T16:35:56.788911+0000","mapping_epoch":359,"log_start":"376'5900","ondisk_log_start":"376'5900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12511","last_scrub_stamp":"2024-04-23T16:34:41.374888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:41.374888+0000","objects_scrubbed":6643,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:07.291855+0000","scrub_duration":1271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210829336,"num_objects":3218,"num_object_clones":0,"num_object_copies":6436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3218,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15914,"num_write_kb":612161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1103,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:52.896684+0000","last_change":"2024-04-23T16:32:29.517947+0000","last_active":"2024-04-23T16:35:52.896684+0000","last_peered":"2024-04-23T16:35:52.896684+0000","last_clean":"2024-04-23T16:35:52.896684+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:35:52.896684+0000","last_undegraded":"2024-04-23T16:35:52.896684+0000","last_fullsized":"2024-04-23T16:35:52.896684+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:32:29.517800+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:32:29.517800+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"428'16119","reported_seq":29223,"reported_epoch":428,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:35:55.048981+0000","last_change":"2024-04-23T16:33:26.507399+0000","last_active":"2024-04-23T16:35:55.048981+0000","last_peered":"2024-04-23T16:35:55.048981+0000","last_clean":"2024-04-23T16:32:59.715575+0000","last_became_active":"2024-04-23T16:33:26.458611+0000","last_became_peered":"2024-04-23T16:33:26.458611+0000","last_unstale":"2024-04-23T16:35:55.048981+0000","last_undegraded":"2024-04-23T16:33:25.895472+0000","last_fullsized":"2024-04-23T16:35:55.048981+0000","mapping_epoch":394,"log_start":"376'6100","ondisk_log_start":"376'6100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8607","last_scrub_stamp":"2024-04-23T16:32:43.130806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.130806+0000","objects_scrubbed":8577,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213712896,"num_objects":3261,"num_object_clones":0,"num_object_copies":6522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16119,"num_write_kb":620160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6429,"num_bytes_recovered":421330944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":3258},{"shards":"5,7","objects":3}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1117,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:52.883854+0000","last_change":"2024-04-23T16:34:47.438492+0000","last_active":"2024-04-23T16:35:52.883854+0000","last_peered":"2024-04-23T16:35:52.883854+0000","last_clean":"2024-04-23T16:35:52.883854+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:35:52.883854+0000","last_undegraded":"2024-04-23T16:35:52.883854+0000","last_fullsized":"2024-04-23T16:35:52.883854+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:47.438356+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:47.438356+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:13.051838+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"428'15686","reported_seq":15936,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:55.009496+0000","last_change":"2024-04-23T16:33:18.576077+0000","last_active":"2024-04-23T16:35:55.009496+0000","last_peered":"2024-04-23T16:35:55.009496+0000","last_clean":"2024-04-23T16:35:55.009496+0000","last_became_active":"2024-04-23T16:31:29.139709+0000","last_became_peered":"2024-04-23T16:31:29.139709+0000","last_unstale":"2024-04-23T16:35:55.009496+0000","last_undegraded":"2024-04-23T16:35:55.009496+0000","last_fullsized":"2024-04-23T16:35:55.009496+0000","mapping_epoch":359,"log_start":"375'5600","ondisk_log_start":"375'5600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9472","last_scrub_stamp":"2024-04-23T16:33:18.575945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.575945+0000","objects_scrubbed":9472,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213516288,"num_objects":3258,"num_object_clones":0,"num_object_copies":6516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15686,"num_write_kb":606208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1118,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:52.876336+0000","last_change":"2024-04-23T16:35:34.878630+0000","last_active":"2024-04-23T16:35:52.876336+0000","last_peered":"2024-04-23T16:35:52.876336+0000","last_clean":"2024-04-23T16:35:52.876336+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:35:52.876336+0000","last_undegraded":"2024-04-23T16:35:52.876336+0000","last_fullsized":"2024-04-23T16:35:52.876336+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:35:34.878536+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:35:34.878536+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:52.503403+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"428'16090","reported_seq":16419,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:54.111013+0000","last_change":"2024-04-23T16:34:44.348696+0000","last_active":"2024-04-23T16:35:54.111013+0000","last_peered":"2024-04-23T16:35:54.111013+0000","last_clean":"2024-04-23T16:35:54.111013+0000","last_became_active":"2024-04-23T16:31:29.139606+0000","last_became_peered":"2024-04-23T16:31:29.139606+0000","last_unstale":"2024-04-23T16:35:54.111013+0000","last_undegraded":"2024-04-23T16:35:54.111013+0000","last_fullsized":"2024-04-23T16:35:54.111013+0000","mapping_epoch":359,"log_start":"376'6000","ondisk_log_start":"376'6000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12750","last_scrub_stamp":"2024-04-23T16:34:44.348550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:44.348550+0000","objects_scrubbed":6711,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:02.747435+0000","scrub_duration":1024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219545600,"num_objects":3350,"num_object_clones":0,"num_object_copies":6700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16090,"num_write_kb":622080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1117,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:52.876891+0000","last_change":"2024-04-23T16:34:51.848140+0000","last_active":"2024-04-23T16:35:52.876891+0000","last_peered":"2024-04-23T16:35:52.876891+0000","last_clean":"2024-04-23T16:35:52.876891+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:35:52.876891+0000","last_undegraded":"2024-04-23T16:35:52.876891+0000","last_fullsized":"2024-04-23T16:35:52.876891+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:51.847992+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:34:51.847992+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:18.910354+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.2","version":"428'15789","reported_seq":16169,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:54.569938+0000","last_change":"2024-04-23T16:35:32.569280+0000","last_active":"2024-04-23T16:35:54.569938+0000","last_peered":"2024-04-23T16:35:54.569938+0000","last_clean":"2024-04-23T16:35:54.569938+0000","last_became_active":"2024-04-23T16:31:29.138430+0000","last_became_peered":"2024-04-23T16:31:29.138430+0000","last_unstale":"2024-04-23T16:35:54.569938+0000","last_undegraded":"2024-04-23T16:35:54.569938+0000","last_fullsized":"2024-04-23T16:35:54.569938+0000","mapping_epoch":359,"log_start":"375'5700","ondisk_log_start":"375'5700","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'14723","last_scrub_stamp":"2024-04-23T16:35:32.569137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:32.569137+0000","objects_scrubbed":4294,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:34.926251+0000","scrub_duration":850,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210173952,"num_objects":3207,"num_object_clones":0,"num_object_copies":6414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15789,"num_write_kb":607872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1118,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:52.878284+0000","last_change":"2024-04-23T16:34:37.693617+0000","last_active":"2024-04-23T16:35:52.878284+0000","last_peered":"2024-04-23T16:35:52.878284+0000","last_clean":"2024-04-23T16:35:52.878284+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:35:52.878284+0000","last_undegraded":"2024-04-23T16:35:52.878284+0000","last_fullsized":"2024-04-23T16:35:52.878284+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:37.693482+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:37.693482+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"428'18076","reported_seq":18531,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:54.104993+0000","last_change":"2024-04-23T16:34:40.604268+0000","last_active":"2024-04-23T16:35:54.104993+0000","last_peered":"2024-04-23T16:35:54.104993+0000","last_clean":"2024-04-23T16:35:54.104993+0000","last_became_active":"2024-04-23T16:31:29.139398+0000","last_became_peered":"2024-04-23T16:31:29.139398+0000","last_unstale":"2024-04-23T16:35:54.104993+0000","last_undegraded":"2024-04-23T16:35:54.104993+0000","last_fullsized":"2024-04-23T16:35:54.104993+0000","mapping_epoch":359,"log_start":"381'8000","ondisk_log_start":"381'8000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'14807","last_scrub_stamp":"2024-04-23T16:34:40.604136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:40.604136+0000","objects_scrubbed":6625,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:04.702557+0000","scrub_duration":1231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217972736,"num_objects":3326,"num_object_clones":0,"num_object_copies":6652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15802,"num_write_kb":612096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1118,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:52.876143+0000","last_change":"2024-04-23T16:35:33.875035+0000","last_active":"2024-04-23T16:35:52.876143+0000","last_peered":"2024-04-23T16:35:52.876143+0000","last_clean":"2024-04-23T16:35:52.876143+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:35:52.876143+0000","last_undegraded":"2024-04-23T16:35:52.876143+0000","last_fullsized":"2024-04-23T16:35:52.876143+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:35:33.874904+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:35:33.874904+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:59.611256+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"428'15699","reported_seq":15939,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:54.455663+0000","last_change":"2024-04-23T16:32:43.947374+0000","last_active":"2024-04-23T16:35:54.455663+0000","last_peered":"2024-04-23T16:35:54.455663+0000","last_clean":"2024-04-23T16:35:54.455663+0000","last_became_active":"2024-04-23T16:31:29.138968+0000","last_became_peered":"2024-04-23T16:31:29.138968+0000","last_unstale":"2024-04-23T16:35:54.455663+0000","last_undegraded":"2024-04-23T16:35:54.455663+0000","last_fullsized":"2024-04-23T16:35:54.455663+0000","mapping_epoch":359,"log_start":"374'5600","ondisk_log_start":"374'5600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8482","last_scrub_stamp":"2024-04-23T16:32:43.947177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.947177+0000","objects_scrubbed":8428,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209387520,"num_objects":3195,"num_object_clones":0,"num_object_copies":6390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15699,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1011,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-04-23T16:35:55.222004+0000","last_change":"2024-04-23T16:34:51.248895+0000","last_active":"2024-04-23T16:35:55.222004+0000","last_peered":"2024-04-23T16:35:55.222004+0000","last_clean":"2024-04-23T16:35:55.222004+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:35:55.222004+0000","last_undegraded":"2024-04-23T16:35:55.222004+0000","last_fullsized":"2024-04-23T16:35:55.222004+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:51.248783+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:51.248783+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:13.233910+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"428'16110","reported_seq":16349,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:54.474976+0000","last_change":"2024-04-23T16:32:39.103230+0000","last_active":"2024-04-23T16:35:54.474976+0000","last_peered":"2024-04-23T16:35:54.474976+0000","last_clean":"2024-04-23T16:35:54.474976+0000","last_became_active":"2024-04-23T16:31:29.310246+0000","last_became_peered":"2024-04-23T16:31:29.310246+0000","last_unstale":"2024-04-23T16:35:54.474976+0000","last_undegraded":"2024-04-23T16:35:54.474976+0000","last_fullsized":"2024-04-23T16:35:54.474976+0000","mapping_epoch":359,"log_start":"376'6100","ondisk_log_start":"376'6100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8325","last_scrub_stamp":"2024-04-23T16:32:39.103059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:39.103059+0000","objects_scrubbed":8272,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219676672,"num_objects":3352,"num_object_clones":0,"num_object_copies":6704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16110,"num_write_kb":622784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":931,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:52.876113+0000","last_change":"2024-04-23T16:34:56.453746+0000","last_active":"2024-04-23T16:35:52.876113+0000","last_peered":"2024-04-23T16:35:52.876113+0000","last_clean":"2024-04-23T16:35:52.876113+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:35:52.876113+0000","last_undegraded":"2024-04-23T16:35:52.876113+0000","last_fullsized":"2024-04-23T16:35:52.876113+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:56.453617+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:34:56.453617+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:08.659133+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"428'15911","reported_seq":28901,"reported_epoch":428,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:35:55.006213+0000","last_change":"2024-04-23T16:33:26.132487+0000","last_active":"2024-04-23T16:35:55.006213+0000","last_peered":"2024-04-23T16:35:55.006213+0000","last_clean":"2024-04-23T16:32:59.695133+0000","last_became_active":"2024-04-23T16:33:26.078062+0000","last_became_peered":"2024-04-23T16:33:26.078062+0000","last_unstale":"2024-04-23T16:35:55.006213+0000","last_undegraded":"2024-04-23T16:33:25.892009+0000","last_fullsized":"2024-04-23T16:35:55.006213+0000","mapping_epoch":394,"log_start":"376'5900","ondisk_log_start":"376'5900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'7968","last_scrub_stamp":"2024-04-23T16:32:36.102122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:36.102122+0000","objects_scrubbed":7926,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209911808,"num_objects":3203,"num_object_clones":0,"num_object_copies":6406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15911,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6374,"num_bytes_recovered":417726464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":3179},{"shards":"5,8","objects":24}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1107,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-04-23T16:35:52.896558+0000","last_change":"2024-04-23T16:33:17.354409+0000","last_active":"2024-04-23T16:35:52.896558+0000","last_peered":"2024-04-23T16:35:52.896558+0000","last_clean":"2024-04-23T16:35:52.896558+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:35:52.896558+0000","last_undegraded":"2024-04-23T16:35:52.896558+0000","last_fullsized":"2024-04-23T16:35:52.896558+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:17.354263+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.354263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3400335384,"num_objects":51886,"num_object_clones":0,"num_object_copies":103772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51886,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":255306,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19416,"num_bytes_recovered":1272446976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8061329408,"data_stored":8061321264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160937,"ondisk_log_size":160937,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056888,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":916336,"kb_used_data":407588,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92828816,"statfs":{"total":95995035648,"available":95056707584,"internally_reserved":0,"allocated":417370112,"data_stored":416955992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":520944068},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351229,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612867,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":967008,"kb_used_data":629012,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92778144,"statfs":{"total":95995035648,"available":95004819456,"internally_reserved":0,"allocated":644108288,"data_stored":643704244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":346094016},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841800,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":666088,"kb_used_data":400288,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":93079064,"statfs":{"total":95995035648,"available":95312961536,"internally_reserved":0,"allocated":409894912,"data_stored":409484864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168844,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":574752,"kb_used_data":210388,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93170400,"statfs":{"total":95995035648,"available":95406489600,"internally_reserved":0,"allocated":215437312,"data_stored":215033268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":373094858},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463186,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912672,"kb_used_data":632916,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":92832480,"statfs":{"total":95995035648,"available":95060459520,"internally_reserved":0,"allocated":648105984,"data_stored":647701940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":286390726},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137326,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738671,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1924960,"kb_used_data":1488404,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":91820192,"statfs":{"total":95995035648,"available":94023876608,"internally_reserved":0,"allocated":1524125696,"data_stored":1523721652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117280,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":565044,"kb_used_data":213160,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93180108,"statfs":{"total":95995035648,"available":95416430592,"internally_reserved":0,"allocated":218275840,"data_stored":217862240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":360315335},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921050,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1623456,"kb_used_data":1068308,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92121696,"statfs":{"total":95995035648,"available":94332616704,"internally_reserved":0,"allocated":1093947392,"data_stored":1093543348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757525,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1465836,"kb_used_data":1055648,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92279316,"statfs":{"total":95995035648,"available":94494019584,"internally_reserved":0,"allocated":1080983552,"data_stored":1080571884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1501429760,"data_stored":1501429760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1886846976,"data_stored":1886846976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":215416832,"data_stored":215416832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1079836672,"data_stored":1079836672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1067716608,"data_stored":1067712536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":640679936,"data_stored":640679936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":212992000,"data_stored":212992000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":401932288,"data_stored":401932288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":641138688,"data_stored":641138688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":413339648,"data_stored":413335576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:36:01.245 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:36:01.245 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:36:01.245 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:36:01.246 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:36:01.246 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.246 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:159902 5.1 e396/359 5:87f7f7c8:::benchmark_data_smithi188_113957_object6304:head v 396'10127, mlcod=396'10125)", 2024-04-23T16:36:01.246 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:33:37.394669+0000", 2024-04-23T16:36:01.246 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 141.91053030099999, 2024-04-23T16:36:01.246 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.022806048999999998, 2024-04-23T16:36:01.246 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.246 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.246 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.246 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.246 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.246 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.247 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.394669+0000", 2024-04-23T16:36:01.247 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.247 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.247 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.247 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.394669+0000", 2024-04-23T16:36:01.247 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.247 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.247 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.247 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.394671+0000", 2024-04-23T16:36:01.247 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.871e-06 2024-04-23T16:36:01.247 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.247 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.248 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.394681+0000", 2024-04-23T16:36:01.248 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0195e-05 2024-04-23T16:36:01.248 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.248 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.248 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.248 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.394695+0000", 2024-04-23T16:36:01.248 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4098e-05 2024-04-23T16:36:01.248 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.248 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.248 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.248 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.394703+0000", 2024-04-23T16:36:01.248 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0099999999999995e-06 2024-04-23T16:36:01.248 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.248 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.248 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.249 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.394763+0000", 2024-04-23T16:36:01.249 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0038999999999999e-05 2024-04-23T16:36:01.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.249 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.249 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.249 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.394782+0000", 2024-04-23T16:36:01.249 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9304e-05 2024-04-23T16:36:01.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.249 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.249 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.249 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.417416+0000", 2024-04-23T16:36:01.249 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.022633863000000001 2024-04-23T16:36:01.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.250 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.250 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.250 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.417475+0000", 2024-04-23T16:36:01.250 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8668999999999997e-05 2024-04-23T16:36:01.250 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.250 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.250 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.250 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.250 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.250 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:159903 5.1 e396/359 5:8883dca1:::benchmark_data_smithi188_113957_object6305:head v 396'10128, mlcod=396'10125)", 2024-04-23T16:36:01.250 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:33:37.394719+0000", 2024-04-23T16:36:01.250 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 141.91048019300001, 2024-04-23T16:36:01.250 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.022782509999999999, 2024-04-23T16:36:01.250 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.251 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.251 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.251 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.251 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.251 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.251 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.394719+0000", 2024-04-23T16:36:01.251 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.251 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.251 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.251 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.251 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.394719+0000", 2024-04-23T16:36:01.251 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.251 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.252 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.252 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.252 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.394719+0000", 2024-04-23T16:36:01.252 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5600000000000001e-07 2024-04-23T16:36:01.252 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.252 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.252 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.252 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.394724+0000", 2024-04-23T16:36:01.252 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8289999999999998e-06 2024-04-23T16:36:01.252 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.252 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.252 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.252 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.394733+0000", 2024-04-23T16:36:01.253 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7260000000000008e-06 2024-04-23T16:36:01.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.253 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.253 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.253 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.394740+0000", 2024-04-23T16:36:01.253 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6320000000000002e-06 2024-04-23T16:36:01.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.253 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.253 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.253 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.398891+0000", 2024-04-23T16:36:01.253 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041518859999999996 2024-04-23T16:36:01.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.253 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.253 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.254 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.398913+0000", 2024-04-23T16:36:01.254 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1030999999999999e-05 2024-04-23T16:36:01.254 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.254 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.254 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.254 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.417478+0000", 2024-04-23T16:36:01.254 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018565505999999999 2024-04-23T16:36:01.254 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.254 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.254 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.254 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.417501+0000", 2024-04-23T16:36:01.254 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3444000000000001e-05 2024-04-23T16:36:01.254 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.254 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.255 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.255 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.255 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.255 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:159905 5.1 e396/359 5:853dd070:::benchmark_data_smithi188_113957_object6307:head v 396'10129, mlcod=396'10125)", 2024-04-23T16:36:01.255 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:33:37.396487+0000", 2024-04-23T16:36:01.255 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 141.90871241799999, 2024-04-23T16:36:01.255 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.021039476000000001, 2024-04-23T16:36:01.255 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.255 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.255 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.255 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.255 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.255 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.255 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.396487+0000", 2024-04-23T16:36:01.255 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.256 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.256 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.396487+0000", 2024-04-23T16:36:01.256 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.256 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.256 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.396488+0000", 2024-04-23T16:36:01.256 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0750000000000001e-06 2024-04-23T16:36:01.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.256 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.256 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.396494+0000", 2024-04-23T16:36:01.256 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5350000000000001e-06 2024-04-23T16:36:01.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.257 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.257 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.257 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.396504+0000", 2024-04-23T16:36:01.257 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7399999999999999e-06 2024-04-23T16:36:01.257 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.257 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.257 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.257 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.396510+0000", 2024-04-23T16:36:01.257 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.348e-06 2024-04-23T16:36:01.257 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.257 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.257 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.257 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.403103+0000", 2024-04-23T16:36:01.257 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0065921360000000002 2024-04-23T16:36:01.257 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.257 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.258 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.258 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.403127+0000", 2024-04-23T16:36:01.258 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4406e-05 2024-04-23T16:36:01.258 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.258 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.258 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.258 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.417506+0000", 2024-04-23T16:36:01.258 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014378996999999999 2024-04-23T16:36:01.258 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.258 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.258 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.258 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.417526+0000", 2024-04-23T16:36:01.258 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0239000000000001e-05 2024-04-23T16:36:01.258 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.258 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.259 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.259 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.259 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.259 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:159915 5.1 e396/359 5:89745345:::benchmark_data_smithi188_113957_object6317:head v 396'10130, mlcod=396'10125)", 2024-04-23T16:36:01.259 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:33:37.400384+0000", 2024-04-23T16:36:01.259 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 141.904815367, 2024-04-23T16:36:01.259 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017166252, 2024-04-23T16:36:01.259 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.259 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.259 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.259 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.259 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.259 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.259 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.400384+0000", 2024-04-23T16:36:01.260 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.260 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.260 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.260 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.260 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.400384+0000", 2024-04-23T16:36:01.260 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.260 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.260 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.260 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.260 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.400385+0000", 2024-04-23T16:36:01.260 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4890000000000001e-06 2024-04-23T16:36:01.260 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.260 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.260 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.261 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.400394+0000", 2024-04-23T16:36:01.261 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3189999999999995e-06 2024-04-23T16:36:01.261 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.261 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.261 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.261 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.400404+0000", 2024-04-23T16:36:01.261 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0249e-05 2024-04-23T16:36:01.261 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.261 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.261 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.261 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.400412+0000", 2024-04-23T16:36:01.261 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1899999999999995e-06 2024-04-23T16:36:01.261 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.261 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.261 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.261 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.407593+0000", 2024-04-23T16:36:01.262 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071812239999999999 2024-04-23T16:36:01.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.262 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.262 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.262 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.407620+0000", 2024-04-23T16:36:01.262 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6526e-05 2024-04-23T16:36:01.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.262 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.262 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.262 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.417528+0000", 2024-04-23T16:36:01.262 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099086980000000005 2024-04-23T16:36:01.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.262 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.262 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.263 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:37.417550+0000", 2024-04-23T16:36:01.263 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1557e-05 2024-04-23T16:36:01.263 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.263 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.263 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.263 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.263 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.263 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:160496 5.1 e396/359 5:8f58c58d:::benchmark_data_smithi188_113957_object6898:head v 396'10169, mlcod=396'10166)", 2024-04-23T16:36:01.263 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:33:38.337197+0000", 2024-04-23T16:36:01.263 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 140.968002394, 2024-04-23T16:36:01.263 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012031451, 2024-04-23T16:36:01.263 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.263 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.263 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.264 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.264 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:38.337197+0000", 2024-04-23T16:36:01.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.264 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:38.337197+0000", 2024-04-23T16:36:01.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.264 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:38.337198+0000", 2024-04-23T16:36:01.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8800000000000001e-07 2024-04-23T16:36:01.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.265 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:38.337203+0000", 2024-04-23T16:36:01.265 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.028e-06 2024-04-23T16:36:01.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.265 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:38.337210+0000", 2024-04-23T16:36:01.265 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7680000000000008e-06 2024-04-23T16:36:01.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.265 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:38.337215+0000", 2024-04-23T16:36:01.265 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6e-06 2024-04-23T16:36:01.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.266 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.266 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.266 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:38.340447+0000", 2024-04-23T16:36:01.266 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0032322689999999999 2024-04-23T16:36:01.266 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.266 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.266 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.266 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:38.340480+0000", 2024-04-23T16:36:01.266 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2285e-05 2024-04-23T16:36:01.266 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.266 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.266 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.266 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:38.349185+0000", 2024-04-23T16:36:01.266 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0087057679999999991 2024-04-23T16:36:01.266 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.266 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.267 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.267 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:38.349228+0000", 2024-04-23T16:36:01.267 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2845000000000001e-05 2024-04-23T16:36:01.267 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.267 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.267 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.267 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.267 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.267 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:167215 5.1 e399/359 5:8886789d:::benchmark_data_smithi188_113957_object13617:head v 399'10592, mlcod=399'10589)", 2024-04-23T16:36:01.267 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:33:49.230180+0000", 2024-04-23T16:36:01.267 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 130.07501930999999, 2024-04-23T16:36:01.267 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011822170999999999, 2024-04-23T16:36:01.267 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.267 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.268 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.268 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.268 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.268 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:49.230180+0000", 2024-04-23T16:36:01.268 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.268 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.268 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:49.230180+0000", 2024-04-23T16:36:01.268 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.268 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.268 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:49.230180+0000", 2024-04-23T16:36:01.269 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6400000000000002e-07 2024-04-23T16:36:01.269 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.269 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.269 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.269 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:49.230185+0000", 2024-04-23T16:36:01.269 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0660000000000003e-06 2024-04-23T16:36:01.269 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.269 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.269 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.269 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:49.230191+0000", 2024-04-23T16:36:01.269 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5470000000000004e-06 2024-04-23T16:36:01.270 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.270 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.270 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.270 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:49.230196+0000", 2024-04-23T16:36:01.270 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1420000000000001e-06 2024-04-23T16:36:01.270 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.270 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.270 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.270 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:49.235297+0000", 2024-04-23T16:36:01.270 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051008759999999998 2024-04-23T16:36:01.270 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.270 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.270 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.270 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:49.235312+0000", 2024-04-23T16:36:01.271 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5245999999999999e-05 2024-04-23T16:36:01.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.271 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.271 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.271 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:49.241959+0000", 2024-04-23T16:36:01.271 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066468710000000004 2024-04-23T16:36:01.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.271 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.271 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.271 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:33:49.242002+0000", 2024-04-23T16:36:01.271 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2858999999999997e-05 2024-04-23T16:36:01.271 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.271 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.271 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.271 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.272 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:197836 5.1 e411/359 5:80c5ba1e:::benchmark_data_smithi188_113957_object44238:head v 411'12548, mlcod=411'12544)", 2024-04-23T16:36:01.272 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:34:35.800796+0000", 2024-04-23T16:36:01.272 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 83.504403202000006, 2024-04-23T16:36:01.272 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012695046, 2024-04-23T16:36:01.272 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.272 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.272 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.272 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.272 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.272 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.272 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:35.800796+0000", 2024-04-23T16:36:01.272 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.272 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.272 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.272 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.273 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:35.800796+0000", 2024-04-23T16:36:01.273 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.273 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.273 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.273 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.273 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:35.800796+0000", 2024-04-23T16:36:01.273 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9399999999999995e-07 2024-04-23T16:36:01.273 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.273 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.273 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.273 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:35.800801+0000", 2024-04-23T16:36:01.273 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7029999999999997e-06 2024-04-23T16:36:01.273 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.273 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.274 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.274 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:35.800806+0000", 2024-04-23T16:36:01.274 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.916e-06 2024-04-23T16:36:01.274 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.274 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.274 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.274 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:35.800811+0000", 2024-04-23T16:36:01.274 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4800000000000003e-06 2024-04-23T16:36:01.274 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.274 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.274 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.274 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:35.805760+0000", 2024-04-23T16:36:01.274 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049496660000000001 2024-04-23T16:36:01.274 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.274 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.275 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.275 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:35.805782+0000", 2024-04-23T16:36:01.275 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2093e-05 2024-04-23T16:36:01.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.275 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.275 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:35.813441+0000", 2024-04-23T16:36:01.275 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076581990000000001 2024-04-23T16:36:01.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.275 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.275 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:35.813491+0000", 2024-04-23T16:36:01.275 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0494999999999997e-05 2024-04-23T16:36:01.275 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.275 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.275 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.276 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.276 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.276 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:210016 5.1 e415/359 5:820b2aa6:::benchmark_data_smithi188_113957_object56418:head v 415'13322, mlcod=415'13320)", 2024-04-23T16:36:01.276 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:34:53.970234+0000", 2024-04-23T16:36:01.276 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.334964921999998, 2024-04-23T16:36:01.276 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012066260000000001, 2024-04-23T16:36:01.276 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.276 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.276 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.276 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.276 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.276 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.276 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.970234+0000", 2024-04-23T16:36:01.276 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.276 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.277 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.277 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.277 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.970234+0000", 2024-04-23T16:36:01.277 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.277 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.277 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.277 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.277 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.970239+0000", 2024-04-23T16:36:01.277 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3209999999999998e-06 2024-04-23T16:36:01.277 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.277 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.277 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.277 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.970254+0000", 2024-04-23T16:36:01.277 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5461e-05 2024-04-23T16:36:01.277 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.277 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.278 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.278 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.970275+0000", 2024-04-23T16:36:01.278 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1171000000000001e-05 2024-04-23T16:36:01.278 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.278 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.278 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.278 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.970288+0000", 2024-04-23T16:36:01.278 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3067999999999999e-05 2024-04-23T16:36:01.278 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.278 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.278 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.278 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.970373+0000", 2024-04-23T16:36:01.278 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4573000000000005e-05 2024-04-23T16:36:01.278 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.278 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.279 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.970402+0000", 2024-04-23T16:36:01.279 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9023999999999999e-05 2024-04-23T16:36:01.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.279 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.982253+0000", 2024-04-23T16:36:01.279 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011851297 2024-04-23T16:36:01.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.279 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:34:53.982300+0000", 2024-04-23T16:36:01.279 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7345000000000002e-05 2024-04-23T16:36:01.279 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.279 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.280 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.280 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.280 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.280 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:218779 5.1 e416/359 5:8b8a3362:::benchmark_data_smithi188_113957_object65181:head v 416'13868, mlcod=416'13866)", 2024-04-23T16:36:01.280 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:35:06.868772+0000", 2024-04-23T16:36:01.280 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 52.436427051999999, 2024-04-23T16:36:01.280 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011899376, 2024-04-23T16:36:01.280 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.280 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.280 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.280 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.280 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.280 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.281 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:06.868772+0000", 2024-04-23T16:36:01.281 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.281 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.281 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.281 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:06.868772+0000", 2024-04-23T16:36:01.281 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.281 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.281 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.281 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:06.868774+0000", 2024-04-23T16:36:01.281 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.852e-06 2024-04-23T16:36:01.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.281 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.282 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.282 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:06.868783+0000", 2024-04-23T16:36:01.282 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5599999999999994e-06 2024-04-23T16:36:01.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.282 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.282 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.282 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:06.868794+0000", 2024-04-23T16:36:01.282 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1861999999999999e-05 2024-04-23T16:36:01.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.282 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.282 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.282 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:06.868801+0000", 2024-04-23T16:36:01.282 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2929999999999997e-06 2024-04-23T16:36:01.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.283 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.283 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.283 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:06.868868+0000", 2024-04-23T16:36:01.283 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7488000000000002e-05 2024-04-23T16:36:01.283 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.283 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.283 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.283 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:06.868884+0000", 2024-04-23T16:36:01.283 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5815999999999999e-05 2024-04-23T16:36:01.283 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.283 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.283 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.283 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:06.880634+0000", 2024-04-23T16:36:01.284 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011749536999999999 2024-04-23T16:36:01.284 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.284 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.284 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.284 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:06.880671+0000", 2024-04-23T16:36:01.284 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7968e-05 2024-04-23T16:36:01.284 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.284 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.284 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.284 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.284 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.284 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:221396 5.1 e416/359 5:8e32205f:::benchmark_data_smithi188_113957_object67798:head v 416'14038, mlcod=416'14036)", 2024-04-23T16:36:01.284 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:35:10.595157+0000", 2024-04-23T16:36:01.284 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.710042409000003, 2024-04-23T16:36:01.284 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012704116, 2024-04-23T16:36:01.285 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.285 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.285 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.285 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.285 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.285 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.595157+0000", 2024-04-23T16:36:01.285 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.285 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.285 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.595157+0000", 2024-04-23T16:36:01.285 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.285 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.286 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.595159+0000", 2024-04-23T16:36:01.286 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.835e-06 2024-04-23T16:36:01.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.286 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.286 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.286 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.595167+0000", 2024-04-23T16:36:01.286 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2039999999999997e-06 2024-04-23T16:36:01.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.286 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.286 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.286 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.595178+0000", 2024-04-23T16:36:01.286 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1024e-05 2024-04-23T16:36:01.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.287 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.287 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.287 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.595184+0000", 2024-04-23T16:36:01.287 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3969999999999999e-06 2024-04-23T16:36:01.287 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.287 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.287 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.287 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.595246+0000", 2024-04-23T16:36:01.287 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1362e-05 2024-04-23T16:36:01.287 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.287 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.287 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.287 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.595270+0000", 2024-04-23T16:36:01.287 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4851e-05 2024-04-23T16:36:01.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.288 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.288 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.288 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.607821+0000", 2024-04-23T16:36:01.288 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012550188 2024-04-23T16:36:01.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.288 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.288 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.288 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.607861+0000", 2024-04-23T16:36:01.288 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0255e-05 2024-04-23T16:36:01.288 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.288 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.288 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.289 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.289 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:221400 5.1 e416/359 5:8c380c56:::benchmark_data_smithi188_113957_object67802:head v 416'14039, mlcod=416'14036)", 2024-04-23T16:36:01.289 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:35:10.595199+0000", 2024-04-23T16:36:01.289 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.710000149000003, 2024-04-23T16:36:01.289 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012691186, 2024-04-23T16:36:01.289 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.289 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.289 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.289 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.289 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.289 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.289 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.595199+0000", 2024-04-23T16:36:01.289 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.289 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.289 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.290 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.290 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.595199+0000", 2024-04-23T16:36:01.290 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.290 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.290 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.290 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.290 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.595199+0000", 2024-04-23T16:36:01.290 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4499999999999998e-07 2024-04-23T16:36:01.290 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.290 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.290 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.290 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.595203+0000", 2024-04-23T16:36:01.290 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8800000000000001e-06 2024-04-23T16:36:01.291 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.291 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.291 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.291 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.595207+0000", 2024-04-23T16:36:01.291 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2080000000000002e-06 2024-04-23T16:36:01.291 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.291 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.291 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.291 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.595210+0000", 2024-04-23T16:36:01.291 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9699999999999999e-06 2024-04-23T16:36:01.291 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.291 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.291 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.291 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.600219+0000", 2024-04-23T16:36:01.291 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050087409999999997 2024-04-23T16:36:01.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.292 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.600241+0000", 2024-04-23T16:36:01.292 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1761999999999999e-05 2024-04-23T16:36:01.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.292 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.607865+0000", 2024-04-23T16:36:01.292 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076236109999999998 2024-04-23T16:36:01.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.292 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.607890+0000", 2024-04-23T16:36:01.292 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5669000000000001e-05 2024-04-23T16:36:01.292 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.293 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.293 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.293 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:221404 5.1 e416/359 5:8f488834:::benchmark_data_smithi188_113957_object67806:head v 416'14040, mlcod=416'14036)", 2024-04-23T16:36:01.293 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:35:10.599690+0000", 2024-04-23T16:36:01.293 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.705509573999997, 2024-04-23T16:36:01.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01980587, 2024-04-23T16:36:01.293 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.293 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.293 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.293 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.293 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.599690+0000", 2024-04-23T16:36:01.294 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.294 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.294 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.294 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.599690+0000", 2024-04-23T16:36:01.294 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.294 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.294 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.294 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.599691+0000", 2024-04-23T16:36:01.294 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.562e-06 2024-04-23T16:36:01.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.294 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.294 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.295 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.599698+0000", 2024-04-23T16:36:01.295 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.387e-06 2024-04-23T16:36:01.295 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.295 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.295 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.295 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.599707+0000", 2024-04-23T16:36:01.295 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8519999999999999e-06 2024-04-23T16:36:01.295 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.295 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.295 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.295 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.599715+0000", 2024-04-23T16:36:01.295 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1690000000000003e-06 2024-04-23T16:36:01.295 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.295 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.295 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.295 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.604764+0000", 2024-04-23T16:36:01.296 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050497060000000002 2024-04-23T16:36:01.296 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.296 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.296 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.296 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.604779+0000", 2024-04-23T16:36:01.296 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4504e-05 2024-04-23T16:36:01.296 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.296 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.296 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.296 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.619445+0000", 2024-04-23T16:36:01.296 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014666177000000001 2024-04-23T16:36:01.296 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.296 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.296 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.296 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.619495+0000", 2024-04-23T16:36:01.296 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0513000000000002e-05 2024-04-23T16:36:01.297 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.297 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.297 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.297 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.297 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:221407 5.1 e416/359 5:8b817ed3:::benchmark_data_smithi188_113957_object67809:head v 416'14041, mlcod=416'14036)", 2024-04-23T16:36:01.297 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:35:10.604402+0000", 2024-04-23T16:36:01.297 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.700797399000002, 2024-04-23T16:36:01.297 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015119521, 2024-04-23T16:36:01.297 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.297 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.297 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.297 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.297 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.297 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.604402+0000", 2024-04-23T16:36:01.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.298 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.298 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.604402+0000", 2024-04-23T16:36:01.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.298 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.298 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.604403+0000", 2024-04-23T16:36:01.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1930000000000001e-06 2024-04-23T16:36:01.298 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.298 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.604410+0000", 2024-04-23T16:36:01.299 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5939999999999999e-06 2024-04-23T16:36:01.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.299 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.299 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.299 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.604419+0000", 2024-04-23T16:36:01.299 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2930000000000006e-06 2024-04-23T16:36:01.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.299 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.299 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.299 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.604426+0000", 2024-04-23T16:36:01.299 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3239999999999997e-06 2024-04-23T16:36:01.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.299 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.299 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.300 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.607922+0000", 2024-04-23T16:36:01.300 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034960120000000002 2024-04-23T16:36:01.300 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.300 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.300 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.300 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.607945+0000", 2024-04-23T16:36:01.300 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3006999999999998e-05 2024-04-23T16:36:01.300 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.300 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.300 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.300 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.619500+0000", 2024-04-23T16:36:01.300 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011555159000000001 2024-04-23T16:36:01.300 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.300 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.300 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.301 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:10.619521+0000", 2024-04-23T16:36:01.301 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0939000000000001e-05 2024-04-23T16:36:01.301 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.301 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.301 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.301 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.301 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.301 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:236707 5.1 e422/359 5:864c493c:::benchmark_data_smithi188_113957_object83109:head v 422'15024, mlcod=422'15020)", 2024-04-23T16:36:01.301 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:35:31.079809+0000", 2024-04-23T16:36:01.301 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.225390467, 2024-04-23T16:36:01.301 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011919806, 2024-04-23T16:36:01.301 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.301 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.301 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.302 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.302 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:31.079809+0000", 2024-04-23T16:36:01.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.302 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:31.079809+0000", 2024-04-23T16:36:01.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.302 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:31.079809+0000", 2024-04-23T16:36:01.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6100000000000001e-07 2024-04-23T16:36:01.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:31.079814+0000", 2024-04-23T16:36:01.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8790000000000002e-06 2024-04-23T16:36:01.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:31.079819+0000", 2024-04-23T16:36:01.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8790000000000002e-06 2024-04-23T16:36:01.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:31.079824+0000", 2024-04-23T16:36:01.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2599999999999996e-06 2024-04-23T16:36:01.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:31.083838+0000", 2024-04-23T16:36:01.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0040140100000000001 2024-04-23T16:36:01.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:31.083854+0000", 2024-04-23T16:36:01.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5715000000000001e-05 2024-04-23T16:36:01.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.305 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:31.091681+0000", 2024-04-23T16:36:01.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078273170000000003 2024-04-23T16:36:01.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.305 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:31.091728+0000", 2024-04-23T16:36:01.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7185000000000003e-05 2024-04-23T16:36:01.305 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.305 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.305 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.305 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:239003 5.1 e423/359 5:88fbd4f4:::benchmark_data_smithi188_113957_object85405:head v 423'15156, mlcod=423'15153)", 2024-04-23T16:36:01.305 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:35:34.066565+0000", 2024-04-23T16:36:01.306 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 25.238634255000001, 2024-04-23T16:36:01.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015193809000000001, 2024-04-23T16:36:01.306 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.306 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.306 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.306 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.306 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.306 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.306 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:34.066565+0000", 2024-04-23T16:36:01.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.306 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.306 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.306 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.306 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:34.066565+0000", 2024-04-23T16:36:01.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:34.066565+0000", 2024-04-23T16:36:01.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6700000000000003e-07 2024-04-23T16:36:01.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:34.066569+0000", 2024-04-23T16:36:01.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.726e-06 2024-04-23T16:36:01.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:34.066575+0000", 2024-04-23T16:36:01.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8919999999999997e-06 2024-04-23T16:36:01.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:34.066579+0000", 2024-04-23T16:36:01.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5899999999999999e-06 2024-04-23T16:36:01.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:34.070999+0000", 2024-04-23T16:36:01.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004420513 2024-04-23T16:36:01.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:34.071016+0000", 2024-04-23T16:36:01.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7087000000000001e-05 2024-04-23T16:36:01.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.309 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:34.081720+0000", 2024-04-23T16:36:01.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010703294 2024-04-23T16:36:01.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.309 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:34.081759+0000", 2024-04-23T16:36:01.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9140000000000001e-05 2024-04-23T16:36:01.309 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.309 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.309 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.309 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:253762 5.1 e428/359 5:859d27fc:::benchmark_data_smithi188_113957_object100164:head v 428'16043, mlcod=428'16039)", 2024-04-23T16:36:01.310 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:35:53.150641+0000", 2024-04-23T16:36:01.310 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.1545577250000001, 2024-04-23T16:36:01.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012651124999999999, 2024-04-23T16:36:01.310 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.310 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.310 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.310 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.310 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.310 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.150641+0000", 2024-04-23T16:36:01.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.310 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.310 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.310 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.150641+0000", 2024-04-23T16:36:01.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.311 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.311 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.150642+0000", 2024-04-23T16:36:01.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7899999999999997e-07 2024-04-23T16:36:01.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.311 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.311 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.150648+0000", 2024-04-23T16:36:01.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5269999999999998e-06 2024-04-23T16:36:01.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.311 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.311 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.150656+0000", 2024-04-23T16:36:01.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5939999999999994e-06 2024-04-23T16:36:01.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.312 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.312 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.312 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.150663+0000", 2024-04-23T16:36:01.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0680000000000002e-06 2024-04-23T16:36:01.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.312 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.312 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.312 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.155633+0000", 2024-04-23T16:36:01.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00497052 2024-04-23T16:36:01.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.312 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.313 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.313 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.155659+0000", 2024-04-23T16:36:01.313 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5644e-05 2024-04-23T16:36:01.313 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.313 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.313 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.313 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.163264+0000", 2024-04-23T16:36:01.313 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076050429999999997 2024-04-23T16:36:01.313 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.313 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.313 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.313 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.163293+0000", 2024-04-23T16:36:01.313 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.885e-05 2024-04-23T16:36:01.313 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.314 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.314 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.314 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.314 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:254118 5.1 e428/359 5:82df52fb:::benchmark_data_smithi188_113957_object100520:head v 428'16068, mlcod=428'16064)", 2024-04-23T16:36:01.314 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:35:53.591848+0000", 2024-04-23T16:36:01.314 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 5.7133512780000002, 2024-04-23T16:36:01.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012498888, 2024-04-23T16:36:01.314 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.314 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.314 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.314 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.591848+0000", 2024-04-23T16:36:01.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.591848+0000", 2024-04-23T16:36:01.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.591850+0000", 2024-04-23T16:36:01.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.77e-06 2024-04-23T16:36:01.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.316 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.316 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.591858+0000", 2024-04-23T16:36:01.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1289999999999996e-06 2024-04-23T16:36:01.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.316 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.316 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.591870+0000", 2024-04-23T16:36:01.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2415000000000001e-05 2024-04-23T16:36:01.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.316 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.316 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.591877+0000", 2024-04-23T16:36:01.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.233e-06 2024-04-23T16:36:01.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.317 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.591934+0000", 2024-04-23T16:36:01.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6786999999999997e-05 2024-04-23T16:36:01.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.317 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.591951+0000", 2024-04-23T16:36:01.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6990000000000002e-05 2024-04-23T16:36:01.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.317 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.604297+0000", 2024-04-23T16:36:01.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012345988 2024-04-23T16:36:01.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.318 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.318 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:53.604347+0000", 2024-04-23T16:36:01.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9576000000000001e-05 2024-04-23T16:36:01.318 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.318 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.318 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.318 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:255348 5.1 e428/359 5:80db07b0:::benchmark_data_smithi188_113957_object101750:head v 428'16145, mlcod=428'16142)", 2024-04-23T16:36:01.318 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:35:55.149505+0000", 2024-04-23T16:36:01.319 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.1556940740000003, 2024-04-23T16:36:01.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011905321999999999, 2024-04-23T16:36:01.319 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.319 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.319 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.319 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.149505+0000", 2024-04-23T16:36:01.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.149505+0000", 2024-04-23T16:36:01.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.149506+0000", 2024-04-23T16:36:01.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9800000000000003e-07 2024-04-23T16:36:01.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.149511+0000", 2024-04-23T16:36:01.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2959999999999998e-06 2024-04-23T16:36:01.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.149517+0000", 2024-04-23T16:36:01.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4740000000000002e-06 2024-04-23T16:36:01.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.149523+0000", 2024-04-23T16:36:01.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2639999999999999e-06 2024-04-23T16:36:01.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.153193+0000", 2024-04-23T16:36:01.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0036703870000000002 2024-04-23T16:36:01.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.153208+0000", 2024-04-23T16:36:01.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5131e-05 2024-04-23T16:36:01.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.161368+0000", 2024-04-23T16:36:01.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0081597760000000005 2024-04-23T16:36:01.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.161410+0000", 2024-04-23T16:36:01.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2395999999999999e-05 2024-04-23T16:36:01.322 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.322 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.323 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.323 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:255553 5.1 e429/359 5:8c2dc45c:::benchmark_data_smithi188_113957_object101955:head v 429'16155, mlcod=429'16152)", 2024-04-23T16:36:01.323 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:35:55.430297+0000", 2024-04-23T16:36:01.323 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.8749019439999999, 2024-04-23T16:36:01.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012577580999999999, 2024-04-23T16:36:01.323 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.323 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.323 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.323 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.323 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.323 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.430297+0000", 2024-04-23T16:36:01.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.324 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.430297+0000", 2024-04-23T16:36:01.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.324 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.430298+0000", 2024-04-23T16:36:01.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6899999999999996e-07 2024-04-23T16:36:01.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.430303+0000", 2024-04-23T16:36:01.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.544e-06 2024-04-23T16:36:01.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.430308+0000", 2024-04-23T16:36:01.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5099999999999998e-06 2024-04-23T16:36:01.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.430312+0000", 2024-04-23T16:36:01.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9369999999999997e-06 2024-04-23T16:36:01.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.434708+0000", 2024-04-23T16:36:01.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043963149999999996 2024-04-23T16:36:01.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.434729+0000", 2024-04-23T16:36:01.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0658000000000001e-05 2024-04-23T16:36:01.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.442830+0000", 2024-04-23T16:36:01.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0081006180000000004 2024-04-23T16:36:01.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:55.442875+0000", 2024-04-23T16:36:01.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5229999999999999e-05 2024-04-23T16:36:01.327 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.327 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.327 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.327 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.25717.0:256057 5.1 e429/359 5:8807746b:::benchmark_data_smithi188_113957_object102459:head v 429'16192, mlcod=429'16190)", 2024-04-23T16:36:01.327 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:35:56.063121+0000", 2024-04-23T16:36:01.327 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.2420782840000002, 2024-04-23T16:36:01.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012639627000000001, 2024-04-23T16:36:01.327 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:36:01.327 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:36:01.327 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:01.327 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:36:01.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:36:01.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:56.063121+0000", 2024-04-23T16:36:01.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:36:01.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:56.063121+0000", 2024-04-23T16:36:01.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:36:01.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:36:01.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:56.063124+0000", 2024-04-23T16:36:01.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6749999999999998e-06 2024-04-23T16:36:01.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.329 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:36:01.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:56.063135+0000", 2024-04-23T16:36:01.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1336e-05 2024-04-23T16:36:01.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.329 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:36:01.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:56.063150+0000", 2024-04-23T16:36:01.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5245000000000001e-05 2024-04-23T16:36:01.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.329 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:36:01.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:56.063159+0000", 2024-04-23T16:36:01.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4190000000000002e-06 2024-04-23T16:36:01.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:36:01.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:56.063221+0000", 2024-04-23T16:36:01.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2403000000000005e-05 2024-04-23T16:36:01.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:36:01.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:56.063245+0000", 2024-04-23T16:36:01.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3621999999999998e-05 2024-04-23T16:36:01.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:36:01.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:56.075706+0000", 2024-04-23T16:36:01.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012461763000000001 2024-04-23T16:36:01.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:36:01.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:36:01.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:36:01.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:35:56.075761+0000", 2024-04-23T16:36:01.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4163999999999998e-05 2024-04-23T16:36:01.331 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.331 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.331 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.331 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:36:01.331 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:36:01.331 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:36:02.093 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-23T16:36:02.112 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:36:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:36:02.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:02.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:36:02.217 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:36:02.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:01 smithi188 ceph-mon[44101]: from='client.16125 v1:172.21.15.103:0/1596961891' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:36:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:01 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:36:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:01 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:36:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:01 smithi188 ceph-mon[44101]: pgmap v1222: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.0 GiB data, 8.8 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 936 op/s; 9026/97132 objects degraded (9.293%); 12 MiB/s, 191 objects/s recovering 2024-04-23T16:36:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:01 smithi188 ceph-mon[44101]: osdmap e430: 12 total, 11 up, 9 in 2024-04-23T16:36:02.274 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:01 smithi148 ceph-mon[44520]: from='client.16125 v1:172.21.15.103:0/1596961891' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:36:02.274 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:01 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:36:02.274 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:01 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:36:02.274 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:01 smithi148 ceph-mon[44520]: pgmap v1222: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.0 GiB data, 8.8 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 936 op/s; 9026/97132 objects degraded (9.293%); 12 MiB/s, 191 objects/s recovering 2024-04-23T16:36:02.274 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:01 smithi148 ceph-mon[44520]: osdmap e430: 12 total, 11 up, 9 in 2024-04-23T16:36:02.282 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:02.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:01 smithi103 ceph-mon[39629]: from='client.16125 v1:172.21.15.103:0/1596961891' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:36:02.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:01 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:36:02.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:01 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:36:02.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:01 smithi103 ceph-mon[39629]: pgmap v1222: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 3.0 GiB data, 8.8 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 936 op/s; 9026/97132 objects degraded (9.293%); 12 MiB/s, 191 objects/s recovering 2024-04-23T16:36:02.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:01 smithi103 ceph-mon[39629]: osdmap e430: 12 total, 11 up, 9 in 2024-04-23T16:36:02.330 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:36:02.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:02.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:36:02.439 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:36:02.469 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:02.594 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:36:02.621 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:36:02.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:02.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:36:02.724 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:36:02.752 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:02.775 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:36:02.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:02.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:36:02.883 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:36:02.916 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:02.939 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:02.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:02.942 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:03.043 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:36:03.088 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:03.121 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:02 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3611840171' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:03.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:02 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3611840171' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:03.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:02 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3611840171' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:03.868 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:36:03.869 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:36:03.869 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:36:03.869 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:36:03.869 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:36:03.869 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:36:03.869 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:36:03.869 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:36:03.869 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:36:03.869 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:36:03.869 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:36:03.869 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:36:03.869 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:36:03.870 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:36:03.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:36:03.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:36:03.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:36:03.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:36:03.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:36:03.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:36:03.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:36:03.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:36:03.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:36:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:03 smithi103 ceph-mon[39629]: pgmap v1224: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.9 GiB data, 8.8 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 716 op/s; 9026/96266 objects degraded (9.376%); 9.8 MiB/s, 157 objects/s recovering 2024-04-23T16:36:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.109 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:36:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:03 smithi188 ceph-mon[44101]: pgmap v1224: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.9 GiB data, 8.8 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 716 op/s; 9026/96266 objects degraded (9.376%); 9.8 MiB/s, 157 objects/s recovering 2024-04-23T16:36:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.348 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:03 smithi148 ceph-mon[44520]: pgmap v1224: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.9 GiB data, 8.8 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 716 op/s; 9026/96266 objects degraded (9.376%); 9.8 MiB/s, 157 objects/s recovering 2024-04-23T16:36:04.348 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.348 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.348 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.348 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.348 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:04.376 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:36:04.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:04.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:36:04.481 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:04.573 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:04.928 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:36:04.929 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:36:05.179 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:05.179 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:04 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 9026/96266 objects degraded (9.376%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:05.179 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:04 smithi103 ceph-mon[39629]: osdmap e431: 12 total, 11 up, 9 in 2024-04-23T16:36:05.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:04 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 9026/96266 objects degraded (9.376%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:04 smithi188 ceph-mon[44101]: osdmap e431: 12 total, 11 up, 9 in 2024-04-23T16:36:05.277 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:05.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:05.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:05.291 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:05.291 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:04 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 9026/96266 objects degraded (9.376%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:05.292 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:04 smithi148 ceph-mon[44520]: osdmap e431: 12 total, 11 up, 9 in 2024-04-23T16:36:05.390 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:36:05.420 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:05.458 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:36:05.461 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:36:05.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:05.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:36:05.568 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:05.670 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:05.932 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:05 smithi148 ceph-mon[44520]: pgmap v1226: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.8 GiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 838 op/s; 8770/91858 objects degraded (9.547%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:36:05.933 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:05 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/898750381' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:36:06.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:05 smithi103 ceph-mon[39629]: pgmap v1226: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.8 GiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 838 op/s; 8770/91858 objects degraded (9.547%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:36:06.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:05 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/898750381' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:36:06.208 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:36:06.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:05 smithi188 ceph-mon[44101]: pgmap v1226: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.8 GiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 838 op/s; 8770/91858 objects degraded (9.547%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:36:06.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:05 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/898750381' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:36:06.929 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:36:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:06 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:36:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:06 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:36:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:06 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:36:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:06 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/898750381' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:36:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:06 smithi103 ceph-mon[39629]: osdmap e432: 12 total, 11 up, 9 in 2024-04-23T16:36:07.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:06 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:36:07.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:06 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:36:07.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:06 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:36:07.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:06 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/898750381' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:36:07.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:06 smithi188 ceph-mon[44101]: osdmap e432: 12 total, 11 up, 9 in 2024-04-23T16:36:07.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:06 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:36:07.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:06 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:36:07.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:06 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:36:07.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:06 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/898750381' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:36:07.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:06 smithi148 ceph-mon[44520]: osdmap e432: 12 total, 11 up, 9 in 2024-04-23T16:36:07.956 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:07.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:07.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:07.970 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:07 smithi103 ceph-mon[39629]: pgmap v1228: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.7 GiB data, 8.4 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 653 op/s; 8261/89288 objects degraded (9.252%); 8.0 MiB/s, 127 objects/s recovering 2024-04-23T16:36:08.071 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:36:08.103 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:08.176 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:36:08.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:08.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:36:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:07 smithi188 ceph-mon[44101]: pgmap v1228: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.7 GiB data, 8.4 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 653 op/s; 8261/89288 objects degraded (9.252%); 8.0 MiB/s, 127 objects/s recovering 2024-04-23T16:36:08.280 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:36:08.282 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:08.347 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:07 smithi148 ceph-mon[44520]: pgmap v1228: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.7 GiB data, 8.4 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 653 op/s; 8261/89288 objects degraded (9.252%); 8.0 MiB/s, 127 objects/s recovering 2024-04-23T16:36:08.350 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:08.560 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:36:08.562 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:36:09.394 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:09.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:09.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:09.499 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:36:09.527 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:09.551 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:09.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:09.554 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:09.655 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:09.713 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:09.747 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:36:09.811 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:09.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:09.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:09.918 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:36:09.974 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:09 smithi148 ceph-mon[44520]: from='client.16137 v1:172.21.15.103:0/2752852409' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:36:09.974 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:09 smithi148 ceph-mon[44520]: pgmap v1229: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.7 GiB data, 8.3 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 594 op/s; 7996/87810 objects degraded (9.106%); 8.2 MiB/s, 131 objects/s recovering 2024-04-23T16:36:09.974 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:09 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 7996/87810 objects degraded (9.106%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:09.974 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:09 smithi148 ceph-mon[44520]: osdmap e433: 12 total, 11 up, 9 in 2024-04-23T16:36:09.988 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:10.080 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:36:10.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:10.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:36:10.190 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:36:10.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:09 smithi188 ceph-mon[44101]: from='client.16137 v1:172.21.15.103:0/2752852409' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:36:10.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:09 smithi188 ceph-mon[44101]: pgmap v1229: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.7 GiB data, 8.3 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 594 op/s; 7996/87810 objects degraded (9.106%); 8.2 MiB/s, 131 objects/s recovering 2024-04-23T16:36:10.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:09 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 7996/87810 objects degraded (9.106%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:10.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:09 smithi188 ceph-mon[44101]: osdmap e433: 12 total, 11 up, 9 in 2024-04-23T16:36:10.221 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:10.243 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:36:10.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:10.248 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:36:10.285 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:36:10.285 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:36:10.349 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:36:10.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:09 smithi103 ceph-mon[39629]: from='client.16137 v1:172.21.15.103:0/2752852409' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:36:10.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:10 smithi103 ceph-mon[39629]: pgmap v1229: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.7 GiB data, 8.3 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 594 op/s; 7996/87810 objects degraded (9.106%); 8.2 MiB/s, 131 objects/s recovering 2024-04-23T16:36:10.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:10 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 7996/87810 objects degraded (9.106%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:10.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:10 smithi103 ceph-mon[39629]: osdmap e433: 12 total, 11 up, 9 in 2024-04-23T16:36:10.378 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:10.400 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:36:10.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:10.404 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:36:10.504 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:36:10.534 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:10.562 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:36:10.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:10.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:36:10.666 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:36:10.715 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:10.940 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:36:10.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:10.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:36:10.991 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-23T16:36:11.052 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:36:11.086 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:11.109 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:11.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:11.112 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:11.213 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:36:11.243 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:11.554 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:36:11.594 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:36:11.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:11.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:36:11.622 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1228,"stamp":"2024-04-23T16:36:06.740259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2926182968,"num_objects":44644,"num_object_clones":0,"num_object_copies":89288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44644,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":262550,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20692,"num_bytes_recovered":1356206624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162310,"ondisk_log_size":162310,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":8797216,"kb_used_data":5252744,"kb_used_omap":14,"kb_used_meta":3544369,"kb_avail":834909152,"statfs":{"total":863955320832,"available":854946971648,"internally_reserved":0,"allocated":5378809856,"data_stored":5375031978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3629434863},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,2,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-257032192,"num_objects":-3922,"num_object_clones":0,"num_object_copies":-7844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-765,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3922,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":765,"num_bytes_recovered":50135040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001132"},"pg_stats":[{"pgid":"5.f","version":"431'16594","reported_seq":16852,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-04-23T16:36:05.089881+0000","last_change":"2024-04-23T16:33:17.526419+0000","last_active":"2024-04-23T16:36:05.089881+0000","last_peered":"2024-04-23T16:36:05.089881+0000","last_clean":"2024-04-23T16:36:05.089881+0000","last_became_active":"2024-04-23T16:31:29.140923+0000","last_became_peered":"2024-04-23T16:31:29.140923+0000","last_unstale":"2024-04-23T16:36:05.089881+0000","last_undegraded":"2024-04-23T16:36:05.089881+0000","last_fullsized":"2024-04-23T16:36:05.089881+0000","mapping_epoch":359,"log_start":"376'6500","ondisk_log_start":"376'6500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9683","last_scrub_stamp":"2024-04-23T16:33:17.526283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.526283+0000","objects_scrubbed":9683,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181665792,"num_objects":2772,"num_object_clones":0,"num_object_copies":5544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16594,"num_write_kb":619712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"431'16540","reported_seq":16924,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-04-23T16:36:04.291459+0000","last_change":"2024-04-23T16:35:26.407358+0000","last_active":"2024-04-23T16:36:04.291459+0000","last_peered":"2024-04-23T16:36:04.291459+0000","last_clean":"2024-04-23T16:36:04.291459+0000","last_became_active":"2024-04-23T16:31:29.137890+0000","last_became_peered":"2024-04-23T16:31:29.137890+0000","last_unstale":"2024-04-23T16:36:04.291459+0000","last_undegraded":"2024-04-23T16:36:04.291459+0000","last_fullsized":"2024-04-23T16:36:04.291459+0000","mapping_epoch":359,"log_start":"376'6500","ondisk_log_start":"376'6500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'14582","last_scrub_stamp":"2024-04-23T16:35:26.407093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:26.407093+0000","objects_scrubbed":4692,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:38.641004+0000","scrub_duration":544,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178520064,"num_objects":2724,"num_object_clones":0,"num_object_copies":5448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16540,"num_write_kb":616448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"430'16590","reported_seq":30579,"reported_epoch":430,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:36:03.193424+0000","last_change":"2024-04-23T16:33:26.432192+0000","last_active":"2024-04-23T16:36:03.193424+0000","last_peered":"2024-04-23T16:36:03.193424+0000","last_clean":"2024-04-23T16:32:59.749530+0000","last_became_active":"2024-04-23T16:33:26.338644+0000","last_became_peered":"2024-04-23T16:33:26.338644+0000","last_unstale":"2024-04-23T16:36:03.193424+0000","last_undegraded":"2024-04-23T16:33:25.874847+0000","last_fullsized":"2024-04-23T16:36:03.193424+0000","mapping_epoch":394,"log_start":"376'6500","ondisk_log_start":"376'6500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8711","last_scrub_stamp":"2024-04-23T16:32:43.928914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.928914+0000","objects_scrubbed":8680,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186908672,"num_objects":2852,"num_object_clones":0,"num_object_copies":5704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16590,"num_write_kb":622144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":["8"],"object_location_counts":[{"shards":"4,8","objects":9},{"shards":"8","objects":2843}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.9","version":"431'16582","reported_seq":16825,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-04-23T16:36:05.133073+0000","last_change":"2024-04-23T16:32:37.256944+0000","last_active":"2024-04-23T16:36:05.133073+0000","last_peered":"2024-04-23T16:36:05.133073+0000","last_clean":"2024-04-23T16:36:05.133073+0000","last_became_active":"2024-04-23T16:31:29.141551+0000","last_became_peered":"2024-04-23T16:31:29.141551+0000","last_unstale":"2024-04-23T16:36:05.133073+0000","last_undegraded":"2024-04-23T16:36:05.133073+0000","last_fullsized":"2024-04-23T16:36:05.133073+0000","mapping_epoch":359,"log_start":"376'6500","ondisk_log_start":"376'6500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8156","last_scrub_stamp":"2024-04-23T16:32:37.256787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:37.256787+0000","objects_scrubbed":8082,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179437568,"num_objects":2738,"num_object_clones":0,"num_object_copies":5476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16582,"num_write_kb":618240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"430'15863","reported_seq":16190,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-04-23T16:36:01.780300+0000","last_change":"2024-04-23T16:34:39.934502+0000","last_active":"2024-04-23T16:36:01.780300+0000","last_peered":"2024-04-23T16:36:01.780300+0000","last_clean":"2024-04-23T16:36:01.780300+0000","last_became_active":"2024-04-23T16:31:29.139045+0000","last_became_peered":"2024-04-23T16:31:29.139045+0000","last_unstale":"2024-04-23T16:36:01.780300+0000","last_undegraded":"2024-04-23T16:36:01.780300+0000","last_fullsized":"2024-04-23T16:36:01.780300+0000","mapping_epoch":359,"log_start":"376'5800","ondisk_log_start":"376'5800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'12219","last_scrub_stamp":"2024-04-23T16:34:39.934190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:39.934190+0000","objects_scrubbed":6529,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:06.252221+0000","scrub_duration":810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188153856,"num_objects":2871,"num_object_clones":0,"num_object_copies":5742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15863,"num_write_kb":599488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"431'19947","reported_seq":20402,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-04-23T16:36:04.302576+0000","last_change":"2024-04-23T16:35:32.474057+0000","last_active":"2024-04-23T16:36:04.302576+0000","last_peered":"2024-04-23T16:36:04.302576+0000","last_clean":"2024-04-23T16:36:04.302576+0000","last_became_active":"2024-04-23T16:31:29.310906+0000","last_became_peered":"2024-04-23T16:31:29.310906+0000","last_unstale":"2024-04-23T16:36:04.302576+0000","last_undegraded":"2024-04-23T16:36:04.302576+0000","last_fullsized":"2024-04-23T16:36:04.302576+0000","mapping_epoch":359,"log_start":"396'9904","ondisk_log_start":"396'9904","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'18441","last_scrub_stamp":"2024-04-23T16:35:32.473828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:32.473828+0000","objects_scrubbed":4251,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:57.506974+0000","scrub_duration":548,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179175424,"num_objects":2734,"num_object_clones":0,"num_object_copies":5468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16386,"num_write_kb":611840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1124,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-04-23T16:36:04.179576+0000","last_change":"2024-04-23T16:34:44.725424+0000","last_active":"2024-04-23T16:36:04.179576+0000","last_peered":"2024-04-23T16:36:04.179576+0000","last_clean":"2024-04-23T16:36:04.179576+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:36:04.179576+0000","last_undegraded":"2024-04-23T16:36:04.179576+0000","last_fullsized":"2024-04-23T16:36:04.179576+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:44.725324+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:44.725324+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:14.224827+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"431'16651","reported_seq":16895,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-04-23T16:36:04.456541+0000","last_change":"2024-04-23T16:32:38.103756+0000","last_active":"2024-04-23T16:36:04.456541+0000","last_peered":"2024-04-23T16:36:04.456541+0000","last_clean":"2024-04-23T16:36:04.456541+0000","last_became_active":"2024-04-23T16:31:29.141640+0000","last_became_peered":"2024-04-23T16:31:29.141640+0000","last_unstale":"2024-04-23T16:36:04.456541+0000","last_undegraded":"2024-04-23T16:36:04.456541+0000","last_fullsized":"2024-04-23T16:36:04.456541+0000","mapping_epoch":359,"log_start":"376'6600","ondisk_log_start":"376'6600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8300","last_scrub_stamp":"2024-04-23T16:32:38.103547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:38.103547+0000","objects_scrubbed":8231,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183828480,"num_objects":2805,"num_object_clones":0,"num_object_copies":5610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16651,"num_write_kb":622592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1118,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-04-23T16:36:04.189785+0000","last_change":"2024-04-23T16:33:25.851379+0000","last_active":"2024-04-23T16:36:04.189785+0000","last_peered":"2024-04-23T16:36:04.189785+0000","last_clean":"2024-04-23T16:36:04.189785+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:36:04.189785+0000","last_undegraded":"2024-04-23T16:36:04.189785+0000","last_fullsized":"2024-04-23T16:36:04.189785+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.498820+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.498820+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"430'16127","reported_seq":16458,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-04-23T16:36:01.776645+0000","last_change":"2024-04-23T16:34:41.375046+0000","last_active":"2024-04-23T16:36:01.776645+0000","last_peered":"2024-04-23T16:36:01.776645+0000","last_clean":"2024-04-23T16:36:01.776645+0000","last_became_active":"2024-04-23T16:31:29.140251+0000","last_became_peered":"2024-04-23T16:31:29.140251+0000","last_unstale":"2024-04-23T16:36:01.776645+0000","last_undegraded":"2024-04-23T16:36:01.776645+0000","last_fullsized":"2024-04-23T16:36:01.776645+0000","mapping_epoch":359,"log_start":"376'6100","ondisk_log_start":"376'6100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12511","last_scrub_stamp":"2024-04-23T16:34:41.374888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:41.374888+0000","objects_scrubbed":6643,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:07.291855+0000","scrub_duration":1271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196870168,"num_objects":3005,"num_object_clones":0,"num_object_copies":6010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3005,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16127,"num_write_kb":612161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1109,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-04-23T16:36:04.202552+0000","last_change":"2024-04-23T16:32:29.517947+0000","last_active":"2024-04-23T16:36:04.202552+0000","last_peered":"2024-04-23T16:36:04.202552+0000","last_clean":"2024-04-23T16:36:04.202552+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:36:04.202552+0000","last_undegraded":"2024-04-23T16:36:04.202552+0000","last_fullsized":"2024-04-23T16:36:04.202552+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:32:29.517800+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:32:29.517800+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"431'16652","reported_seq":30833,"reported_epoch":431,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:36:05.111840+0000","last_change":"2024-04-23T16:33:26.507399+0000","last_active":"2024-04-23T16:36:05.111840+0000","last_peered":"2024-04-23T16:36:05.111840+0000","last_clean":"2024-04-23T16:32:59.715575+0000","last_became_active":"2024-04-23T16:33:26.458611+0000","last_became_peered":"2024-04-23T16:33:26.458611+0000","last_unstale":"2024-04-23T16:36:05.111840+0000","last_undegraded":"2024-04-23T16:33:25.895472+0000","last_fullsized":"2024-04-23T16:36:05.111840+0000","mapping_epoch":394,"log_start":"376'6600","ondisk_log_start":"376'6600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8607","last_scrub_stamp":"2024-04-23T16:32:43.130806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.130806+0000","objects_scrubbed":8577,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178782208,"num_objects":2728,"num_object_clones":0,"num_object_copies":5456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2723,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16652,"num_write_kb":620160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6964,"num_bytes_recovered":456392704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":2723},{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1121,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-04-23T16:36:00.907311+0000","last_change":"2024-04-23T16:34:47.438492+0000","last_active":"2024-04-23T16:36:00.907311+0000","last_peered":"2024-04-23T16:36:00.907311+0000","last_clean":"2024-04-23T16:36:00.907311+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:36:00.907311+0000","last_undegraded":"2024-04-23T16:36:00.907311+0000","last_fullsized":"2024-04-23T16:36:00.907311+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:47.438356+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:47.438356+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:13.051838+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"431'16212","reported_seq":16468,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-04-23T16:36:05.065769+0000","last_change":"2024-04-23T16:33:18.576077+0000","last_active":"2024-04-23T16:36:05.065769+0000","last_peered":"2024-04-23T16:36:05.065769+0000","last_clean":"2024-04-23T16:36:05.065769+0000","last_became_active":"2024-04-23T16:31:29.139709+0000","last_became_peered":"2024-04-23T16:31:29.139709+0000","last_unstale":"2024-04-23T16:36:05.065769+0000","last_undegraded":"2024-04-23T16:36:05.065769+0000","last_fullsized":"2024-04-23T16:36:05.065769+0000","mapping_epoch":359,"log_start":"376'6200","ondisk_log_start":"376'6200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9472","last_scrub_stamp":"2024-04-23T16:33:18.575945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.575945+0000","objects_scrubbed":9472,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179044352,"num_objects":2732,"num_object_clones":0,"num_object_copies":5464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16212,"num_write_kb":606208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1124,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-04-23T16:36:04.188703+0000","last_change":"2024-04-23T16:35:34.878630+0000","last_active":"2024-04-23T16:36:04.188703+0000","last_peered":"2024-04-23T16:36:04.188703+0000","last_clean":"2024-04-23T16:36:04.188703+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:36:04.188703+0000","last_undegraded":"2024-04-23T16:36:04.188703+0000","last_fullsized":"2024-04-23T16:36:04.188703+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:35:34.878536+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:35:34.878536+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:52.503403+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"430'16614","reported_seq":16947,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-04-23T16:36:04.114009+0000","last_change":"2024-04-23T16:34:44.348696+0000","last_active":"2024-04-23T16:36:04.114009+0000","last_peered":"2024-04-23T16:36:04.114009+0000","last_clean":"2024-04-23T16:36:04.114009+0000","last_became_active":"2024-04-23T16:31:29.139606+0000","last_became_peered":"2024-04-23T16:31:29.139606+0000","last_unstale":"2024-04-23T16:36:04.114009+0000","last_undegraded":"2024-04-23T16:36:04.114009+0000","last_fullsized":"2024-04-23T16:36:04.114009+0000","mapping_epoch":359,"log_start":"376'6600","ondisk_log_start":"376'6600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12750","last_scrub_stamp":"2024-04-23T16:34:44.348550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:44.348550+0000","objects_scrubbed":6711,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185204736,"num_objects":2826,"num_object_clones":0,"num_object_copies":5652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16614,"num_write_kb":622080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1123,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-04-23T16:36:04.179458+0000","last_change":"2024-04-23T16:34:51.848140+0000","last_active":"2024-04-23T16:36:04.179458+0000","last_peered":"2024-04-23T16:36:04.179458+0000","last_clean":"2024-04-23T16:36:04.179458+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:36:04.179458+0000","last_undegraded":"2024-04-23T16:36:04.179458+0000","last_fullsized":"2024-04-23T16:36:04.179458+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:51.847992+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:34:51.847992+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:18.910354+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.2","version":"431'16250","reported_seq":16636,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-04-23T16:36:04.609566+0000","last_change":"2024-04-23T16:35:32.569280+0000","last_active":"2024-04-23T16:36:04.609566+0000","last_peered":"2024-04-23T16:36:04.609566+0000","last_clean":"2024-04-23T16:36:04.609566+0000","last_became_active":"2024-04-23T16:31:29.138430+0000","last_became_peered":"2024-04-23T16:31:29.138430+0000","last_unstale":"2024-04-23T16:36:04.609566+0000","last_undegraded":"2024-04-23T16:36:04.609566+0000","last_fullsized":"2024-04-23T16:36:04.609566+0000","mapping_epoch":359,"log_start":"376'6200","ondisk_log_start":"376'6200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'14723","last_scrub_stamp":"2024-04-23T16:35:32.569137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:32.569137+0000","objects_scrubbed":4294,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:34.926251+0000","scrub_duration":850,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179961856,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16250,"num_write_kb":607872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1124,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-04-23T16:36:04.179637+0000","last_change":"2024-04-23T16:34:37.693617+0000","last_active":"2024-04-23T16:36:04.179637+0000","last_peered":"2024-04-23T16:36:04.179637+0000","last_clean":"2024-04-23T16:36:04.179637+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:36:04.179637+0000","last_undegraded":"2024-04-23T16:36:04.179637+0000","last_fullsized":"2024-04-23T16:36:04.179637+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:37.693482+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:37.693482+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"430'18558","reported_seq":19017,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-04-23T16:36:04.156709+0000","last_change":"2024-04-23T16:34:40.604268+0000","last_active":"2024-04-23T16:36:04.156709+0000","last_peered":"2024-04-23T16:36:04.156709+0000","last_clean":"2024-04-23T16:36:04.156709+0000","last_became_active":"2024-04-23T16:31:29.139398+0000","last_became_peered":"2024-04-23T16:31:29.139398+0000","last_unstale":"2024-04-23T16:36:04.156709+0000","last_undegraded":"2024-04-23T16:36:04.156709+0000","last_fullsized":"2024-04-23T16:36:04.156709+0000","mapping_epoch":359,"log_start":"382'8500","ondisk_log_start":"382'8500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'14807","last_scrub_stamp":"2024-04-23T16:34:40.604136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:40.604136+0000","objects_scrubbed":6625,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:04.702557+0000","scrub_duration":1231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186384384,"num_objects":2844,"num_object_clones":0,"num_object_copies":5688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16284,"num_write_kb":612096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1124,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-04-23T16:36:04.188758+0000","last_change":"2024-04-23T16:35:33.875035+0000","last_active":"2024-04-23T16:36:04.188758+0000","last_peered":"2024-04-23T16:36:04.188758+0000","last_clean":"2024-04-23T16:36:04.188758+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:36:04.188758+0000","last_undegraded":"2024-04-23T16:36:04.188758+0000","last_fullsized":"2024-04-23T16:36:04.188758+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:35:33.874904+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:35:33.874904+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:59.611256+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"431'16141","reported_seq":16387,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-04-23T16:36:04.489588+0000","last_change":"2024-04-23T16:32:43.947374+0000","last_active":"2024-04-23T16:36:04.489588+0000","last_peered":"2024-04-23T16:36:04.489588+0000","last_clean":"2024-04-23T16:36:04.489588+0000","last_became_active":"2024-04-23T16:31:29.138968+0000","last_became_peered":"2024-04-23T16:31:29.138968+0000","last_unstale":"2024-04-23T16:36:04.489588+0000","last_undegraded":"2024-04-23T16:36:04.489588+0000","last_fullsized":"2024-04-23T16:36:04.489588+0000","mapping_epoch":359,"log_start":"376'6100","ondisk_log_start":"376'6100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8482","last_scrub_stamp":"2024-04-23T16:32:43.947177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.947177+0000","objects_scrubbed":8428,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180420608,"num_objects":2753,"num_object_clones":0,"num_object_copies":5506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16141,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1013,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-04-23T16:36:00.926773+0000","last_change":"2024-04-23T16:34:51.248895+0000","last_active":"2024-04-23T16:36:00.926773+0000","last_peered":"2024-04-23T16:36:00.926773+0000","last_clean":"2024-04-23T16:36:00.926773+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:36:00.926773+0000","last_undegraded":"2024-04-23T16:36:00.926773+0000","last_fullsized":"2024-04-23T16:36:00.926773+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:51.248783+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:51.248783+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:13.233910+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"431'16651","reported_seq":16896,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-04-23T16:36:04.475567+0000","last_change":"2024-04-23T16:32:39.103230+0000","last_active":"2024-04-23T16:36:04.475567+0000","last_peered":"2024-04-23T16:36:04.475567+0000","last_clean":"2024-04-23T16:36:04.475567+0000","last_became_active":"2024-04-23T16:31:29.310246+0000","last_became_peered":"2024-04-23T16:31:29.310246+0000","last_unstale":"2024-04-23T16:36:04.475567+0000","last_undegraded":"2024-04-23T16:36:04.475567+0000","last_fullsized":"2024-04-23T16:36:04.475567+0000","mapping_epoch":359,"log_start":"376'6600","ondisk_log_start":"376'6600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8325","last_scrub_stamp":"2024-04-23T16:32:39.103059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:39.103059+0000","objects_scrubbed":8272,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184221696,"num_objects":2811,"num_object_clones":0,"num_object_copies":5622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16651,"num_write_kb":622784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":937,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-04-23T16:36:04.188632+0000","last_change":"2024-04-23T16:34:56.453746+0000","last_active":"2024-04-23T16:36:04.188632+0000","last_peered":"2024-04-23T16:36:04.188632+0000","last_clean":"2024-04-23T16:36:04.188632+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:36:04.188632+0000","last_undegraded":"2024-04-23T16:36:04.188632+0000","last_fullsized":"2024-04-23T16:36:04.188632+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:56.453617+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:34:56.453617+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:08.659133+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"431'16413","reported_seq":30377,"reported_epoch":431,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:36:05.108137+0000","last_change":"2024-04-23T16:33:26.132487+0000","last_active":"2024-04-23T16:36:05.108137+0000","last_peered":"2024-04-23T16:36:05.108137+0000","last_clean":"2024-04-23T16:32:59.695133+0000","last_became_active":"2024-04-23T16:33:26.078062+0000","last_became_peered":"2024-04-23T16:33:26.078062+0000","last_unstale":"2024-04-23T16:36:05.108137+0000","last_undegraded":"2024-04-23T16:33:25.892009+0000","last_fullsized":"2024-04-23T16:36:05.108137+0000","mapping_epoch":394,"log_start":"376'6400","ondisk_log_start":"376'6400","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'7968","last_scrub_stamp":"2024-04-23T16:32:36.102122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:36.102122+0000","objects_scrubbed":7926,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177012736,"num_objects":2701,"num_object_clones":0,"num_object_copies":5402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16413,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6858,"num_bytes_recovered":449445888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":2695},{"shards":"5,8","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1114,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-04-23T16:36:04.200800+0000","last_change":"2024-04-23T16:33:17.354409+0000","last_active":"2024-04-23T16:36:04.200800+0000","last_peered":"2024-04-23T16:36:04.200800+0000","last_clean":"2024-04-23T16:36:04.200800+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:36:04.200800+0000","last_undegraded":"2024-04-23T16:36:04.200800+0000","last_fullsized":"2024-04-23T16:36:04.200800+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:17.354263+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.354263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":2925592600,"num_objects":44642,"num_object_clones":0,"num_object_copies":89284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44642,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":262550,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20691,"num_bytes_recovered":1356005376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7187079168,"data_stored":7187071024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160781,"ondisk_log_size":160781,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056889,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889916,"kb_used_data":381168,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92855236,"statfs":{"total":95995035648,"available":95083761664,"internally_reserved":0,"allocated":390316032,"data_stored":389895908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":520944068},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351231,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612869,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889972,"kb_used_data":529772,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92855180,"statfs":{"total":95995035648,"available":95083704320,"internally_reserved":0,"allocated":542486528,"data_stored":542070476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":368835008},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841801,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":639160,"kb_used_data":369196,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":93105992,"statfs":{"total":95995035648,"available":95340535808,"internally_reserved":0,"allocated":378056704,"data_stored":377640652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168846,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":530688,"kb_used_data":178808,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93214464,"statfs":{"total":95995035648,"available":95451611136,"internally_reserved":0,"allocated":183099392,"data_stored":182677286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":360315338},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463188,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":829300,"kb_used_data":541292,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":92915852,"statfs":{"total":95995035648,"available":95145832448,"internally_reserved":0,"allocated":554283008,"data_stored":553866956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":294910406},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137328,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738673,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1707448,"kb_used_data":1270892,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92037704,"statfs":{"total":95995035648,"available":94246608896,"internally_reserved":0,"allocated":1301393408,"data_stored":1300977356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117282,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":529224,"kb_used_data":177344,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93215928,"statfs":{"total":95995035648,"available":95453110272,"internally_reserved":0,"allocated":181600256,"data_stored":181174598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":360315335},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921052,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1471284,"kb_used_data":908396,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":92273868,"statfs":{"total":95995035648,"available":94488440832,"internally_reserved":0,"allocated":930197504,"data_stored":929781452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":576387525},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757527,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1310224,"kb_used_data":895876,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92434928,"statfs":{"total":95995035648,"available":94653366272,"internally_reserved":0,"allocated":917377024,"data_stored":916947294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1279131648,"data_stored":1279131648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1886846976,"data_stored":1886846976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":178913280,"data_stored":178913280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":917766144,"data_stored":917766144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":903614464,"data_stored":903610392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":546242560,"data_stored":546242560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":179896320,"data_stored":179896320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":370868224,"data_stored":370868224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":538771456,"data_stored":538771456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":385028096,"data_stored":385024024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:36:11.624 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:36:11.700 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:36:11.728 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:11.756 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:36:11.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:11.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:36:11.861 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:36:11.894 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:11.924 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:36:11.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:11.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:36:12.031 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:36:12.060 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:11 smithi188 ceph-mon[44101]: pgmap v1231: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.5 GiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s; 7492/81104 objects degraded (9.238%); 12 MiB/s, 194 objects/s recovering 2024-04-23T16:36:12.062 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:12.084 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:36:12.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:12.087 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:36:12.188 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:36:12.218 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:12.233 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:36:12.242 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:11 smithi148 ceph-mon[44520]: pgmap v1231: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.5 GiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s; 7492/81104 objects degraded (9.238%); 12 MiB/s, 194 objects/s recovering 2024-04-23T16:36:12.243 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:36:12.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:12.248 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:36:12.256 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:36:12.282 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:36:12.302 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:36:12.303 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:36:12.323 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:36:12.348 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:36:12.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:11 smithi103 ceph-mon[39629]: pgmap v1231: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.5 GiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s; 7492/81104 objects degraded (9.238%); 12 MiB/s, 194 objects/s recovering 2024-04-23T16:36:12.352 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:36:12.384 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:12.385 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:36:12.410 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:36:12.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:12.412 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:36:12.413 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:36:12.436 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:36:12.460 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:36:12.490 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:36:12.514 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:12.515 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:36:12.516 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:36:12.516 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:36:12.516 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:36:12.516 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:36:12.516 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:36:12.516 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:36:12.516 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:36:12.516 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:36:12.516 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:36:12.516 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:36:12.516 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:36:12.516 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:36:12.590 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:13.282 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:12 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/538662696' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:13.282 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.282 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.282 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.283 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.283 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.283 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.283 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.283 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.283 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:12 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/538662696' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:12 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/538662696' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:13.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:13.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:14.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:14.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:14.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:13 smithi103 ceph-mon[39629]: pgmap v1232: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.4 GiB data, 7.9 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 738 op/s; 7492/80036 objects degraded (9.361%); 10 MiB/s, 159 objects/s recovering 2024-04-23T16:36:14.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:14.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:14.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:13 smithi148 ceph-mon[44520]: pgmap v1232: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.4 GiB data, 7.9 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 738 op/s; 7492/80036 objects degraded (9.361%); 10 MiB/s, 159 objects/s recovering 2024-04-23T16:36:14.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:14.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:14.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:13 smithi188 ceph-mon[44101]: pgmap v1232: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.4 GiB data, 7.9 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 738 op/s; 7492/80036 objects degraded (9.361%); 10 MiB/s, 159 objects/s recovering 2024-04-23T16:36:15.080 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:15.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:15.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:15.188 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:15.231 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:14 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 7492/80036 objects degraded (9.361%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:15.231 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:14 smithi103 ceph-mon[39629]: osdmap e434: 12 total, 11 up, 9 in 2024-04-23T16:36:15.231 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:14 smithi103 ceph-mon[39629]: pgmap v1234: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.3 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 867 op/s; 7245/75408 objects degraded (9.608%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:36:15.259 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:36:15.259 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:36:15.260 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:14 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 7492/80036 objects degraded (9.361%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:15.260 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:14 smithi148 ceph-mon[44520]: osdmap e434: 12 total, 11 up, 9 in 2024-04-23T16:36:15.260 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:14 smithi148 ceph-mon[44520]: pgmap v1234: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.3 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 867 op/s; 7245/75408 objects degraded (9.608%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:36:15.260 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:15.443 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:15.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:15.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:15.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:14 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 7492/80036 objects degraded (9.361%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:14 smithi188 ceph-mon[44101]: osdmap e434: 12 total, 11 up, 9 in 2024-04-23T16:36:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:14 smithi188 ceph-mon[44101]: pgmap v1234: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.3 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 867 op/s; 7245/75408 objects degraded (9.608%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:36:15.549 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:36:15.578 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:15.604 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:36:15.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:15.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:36:15.710 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:36:15.739 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:15.759 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:36:15.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:15.763 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:36:15.864 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:36:15.901 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:15.909 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:36:15.910 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:36:15.910 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2024-04-23T16:36:15.910 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2024-04-23T16:36:15.910 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:36:16.562 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:36:16.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:16.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:36:16.669 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:36:16.703 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:16.786 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:36:16.810 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:36:16.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:16.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:36:16.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:16 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1070644771' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:36:16.915 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:36:16.947 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:16.972 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:16.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:16.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:17.067 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:36:17.069 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:36:17.077 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:36:17.110 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:17.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:16 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1070644771' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:36:17.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:16 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1070644771' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:36:17.260 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:36:17.472 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:36:17.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:17.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:36:17.578 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:36:17.616 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:17.639 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:17.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:17.645 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:17.720 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-23T16:36:17.746 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:36:17.774 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:17.794 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:17.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:17.796 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:17.897 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:36:17.926 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:17.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:17 smithi188 ceph-mon[44101]: pgmap v1235: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.2 GiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 939 op/s; 6731/72782 objects degraded (9.248%); 9.9 MiB/s, 158 objects/s recovering 2024-04-23T16:36:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:17 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:36:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:17 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1070644771' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:36:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:17 smithi188 ceph-mon[44101]: osdmap e435: 12 total, 11 up, 9 in 2024-04-23T16:36:18.088 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:17 smithi103 ceph-mon[39629]: pgmap v1235: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.2 GiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 939 op/s; 6731/72782 objects degraded (9.248%); 9.9 MiB/s, 158 objects/s recovering 2024-04-23T16:36:18.167 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:17 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:36:18.167 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:17 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1070644771' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:36:18.167 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:17 smithi103 ceph-mon[39629]: osdmap e435: 12 total, 11 up, 9 in 2024-04-23T16:36:18.168 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:17 smithi148 ceph-mon[44520]: pgmap v1235: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.2 GiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 939 op/s; 6731/72782 objects degraded (9.248%); 9.9 MiB/s, 158 objects/s recovering 2024-04-23T16:36:18.168 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:17 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:36:18.168 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:17 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1070644771' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:36:18.168 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:17 smithi148 ceph-mon[44520]: osdmap e435: 12 total, 11 up, 9 in 2024-04-23T16:36:18.169 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1236,"stamp":"2024-04-23T16:36:16.783175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2385314360,"num_objects":36391,"num_object_clones":0,"num_object_copies":72782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6731,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36391,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":270803,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22222,"num_bytes_recovered":1456476704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162263,"ondisk_log_size":162263,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":7897828,"kb_used_data":4289292,"kb_used_omap":14,"kb_used_meta":3608241,"kb_avail":835808540,"statfs":{"total":863955320832,"available":855867944960,"internally_reserved":0,"allocated":4392235008,"data_stored":4388294122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3694839791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-459276288,"num_objects":-7008,"num_object_clones":0,"num_object_copies":-14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7008,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1265,"num_bytes_recovered":82903040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.607598"},"pg_stats":[{"pgid":"5.f","version":"434'17143","reported_seq":17407,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-23T16:36:15.131136+0000","last_change":"2024-04-23T16:33:17.526419+0000","last_active":"2024-04-23T16:36:15.131136+0000","last_peered":"2024-04-23T16:36:15.131136+0000","last_clean":"2024-04-23T16:36:15.131136+0000","last_became_active":"2024-04-23T16:31:29.140923+0000","last_became_peered":"2024-04-23T16:31:29.140923+0000","last_unstale":"2024-04-23T16:36:15.131136+0000","last_undegraded":"2024-04-23T16:36:15.131136+0000","last_fullsized":"2024-04-23T16:36:15.131136+0000","mapping_epoch":359,"log_start":"378'7100","ondisk_log_start":"378'7100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9683","last_scrub_stamp":"2024-04-23T16:33:17.526283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.526283+0000","objects_scrubbed":9683,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145686528,"num_objects":2223,"num_object_clones":0,"num_object_copies":4446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17143,"num_write_kb":619712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"434'17014","reported_seq":17404,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-23T16:36:14.313742+0000","last_change":"2024-04-23T16:35:26.407358+0000","last_active":"2024-04-23T16:36:14.313742+0000","last_peered":"2024-04-23T16:36:14.313742+0000","last_clean":"2024-04-23T16:36:14.313742+0000","last_became_active":"2024-04-23T16:31:29.137890+0000","last_became_peered":"2024-04-23T16:31:29.137890+0000","last_unstale":"2024-04-23T16:36:14.313742+0000","last_undegraded":"2024-04-23T16:36:14.313742+0000","last_fullsized":"2024-04-23T16:36:14.313742+0000","mapping_epoch":359,"log_start":"378'7000","ondisk_log_start":"378'7000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'14582","last_scrub_stamp":"2024-04-23T16:35:26.407093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:26.407093+0000","objects_scrubbed":4692,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:38.641004+0000","scrub_duration":544,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147456000,"num_objects":2250,"num_object_clones":0,"num_object_copies":4500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17014,"num_write_kb":616448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"433'17101","reported_seq":32122,"reported_epoch":433,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:36:13.205140+0000","last_change":"2024-04-23T16:33:26.432192+0000","last_active":"2024-04-23T16:36:13.205140+0000","last_peered":"2024-04-23T16:36:13.205140+0000","last_clean":"2024-04-23T16:32:59.749530+0000","last_became_active":"2024-04-23T16:33:26.338644+0000","last_became_peered":"2024-04-23T16:33:26.338644+0000","last_unstale":"2024-04-23T16:36:13.205140+0000","last_undegraded":"2024-04-23T16:33:25.874847+0000","last_fullsized":"2024-04-23T16:36:13.205140+0000","mapping_epoch":394,"log_start":"378'7100","ondisk_log_start":"378'7100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8711","last_scrub_stamp":"2024-04-23T16:32:43.928914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.928914+0000","objects_scrubbed":8680,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153419776,"num_objects":2341,"num_object_clones":0,"num_object_copies":4682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17101,"num_write_kb":622144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7381,"num_bytes_recovered":483721216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":["8"],"object_location_counts":[{"shards":"4,8","objects":10},{"shards":"8","objects":2331}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.9","version":"434'17074","reported_seq":17323,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-23T16:36:15.163813+0000","last_change":"2024-04-23T16:32:37.256944+0000","last_active":"2024-04-23T16:36:15.163813+0000","last_peered":"2024-04-23T16:36:15.163813+0000","last_clean":"2024-04-23T16:36:15.163813+0000","last_became_active":"2024-04-23T16:31:29.141551+0000","last_became_peered":"2024-04-23T16:31:29.141551+0000","last_unstale":"2024-04-23T16:36:15.163813+0000","last_undegraded":"2024-04-23T16:36:15.163813+0000","last_fullsized":"2024-04-23T16:36:15.163813+0000","mapping_epoch":359,"log_start":"378'7000","ondisk_log_start":"378'7000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8156","last_scrub_stamp":"2024-04-23T16:32:37.256787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:37.256787+0000","objects_scrubbed":8082,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147193856,"num_objects":2246,"num_object_clones":0,"num_object_copies":4492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17074,"num_write_kb":618240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"433'16350","reported_seq":16683,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-04-23T16:36:11.690731+0000","last_change":"2024-04-23T16:34:39.934502+0000","last_active":"2024-04-23T16:36:11.690731+0000","last_peered":"2024-04-23T16:36:11.690731+0000","last_clean":"2024-04-23T16:36:11.690731+0000","last_became_active":"2024-04-23T16:31:29.139045+0000","last_became_peered":"2024-04-23T16:31:29.139045+0000","last_unstale":"2024-04-23T16:36:11.690731+0000","last_undegraded":"2024-04-23T16:36:11.690731+0000","last_fullsized":"2024-04-23T16:36:11.690731+0000","mapping_epoch":359,"log_start":"376'6300","ondisk_log_start":"376'6300","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'12219","last_scrub_stamp":"2024-04-23T16:34:39.934190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:39.934190+0000","objects_scrubbed":6529,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:13.238369+0000","scrub_duration":810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156237824,"num_objects":2384,"num_object_clones":0,"num_object_copies":4768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16350,"num_write_kb":599488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"434'20487","reported_seq":20948,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-23T16:36:14.286481+0000","last_change":"2024-04-23T16:35:32.474057+0000","last_active":"2024-04-23T16:36:14.286481+0000","last_peered":"2024-04-23T16:36:14.286481+0000","last_clean":"2024-04-23T16:36:14.286481+0000","last_became_active":"2024-04-23T16:31:29.310906+0000","last_became_peered":"2024-04-23T16:31:29.310906+0000","last_unstale":"2024-04-23T16:36:14.286481+0000","last_undegraded":"2024-04-23T16:36:14.286481+0000","last_fullsized":"2024-04-23T16:36:14.286481+0000","mapping_epoch":359,"log_start":"399'10404","ondisk_log_start":"399'10404","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'18441","last_scrub_stamp":"2024-04-23T16:35:32.473828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:32.473828+0000","objects_scrubbed":4251,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:57.506974+0000","scrub_duration":548,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143785984,"num_objects":2194,"num_object_clones":0,"num_object_copies":4388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16926,"num_write_kb":611840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1130,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-23T16:36:14.192889+0000","last_change":"2024-04-23T16:34:44.725424+0000","last_active":"2024-04-23T16:36:14.192889+0000","last_peered":"2024-04-23T16:36:14.192889+0000","last_clean":"2024-04-23T16:36:14.192889+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:36:14.192889+0000","last_undegraded":"2024-04-23T16:36:14.192889+0000","last_fullsized":"2024-04-23T16:36:14.192889+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:44.725324+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:44.725324+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:14.224827+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"434'17172","reported_seq":17422,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-23T16:36:14.506521+0000","last_change":"2024-04-23T16:32:38.103756+0000","last_active":"2024-04-23T16:36:14.506521+0000","last_peered":"2024-04-23T16:36:14.506521+0000","last_clean":"2024-04-23T16:36:14.506521+0000","last_became_active":"2024-04-23T16:31:29.141640+0000","last_became_peered":"2024-04-23T16:31:29.141640+0000","last_unstale":"2024-04-23T16:36:14.506521+0000","last_undegraded":"2024-04-23T16:36:14.506521+0000","last_fullsized":"2024-04-23T16:36:14.506521+0000","mapping_epoch":359,"log_start":"378'7100","ondisk_log_start":"378'7100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8300","last_scrub_stamp":"2024-04-23T16:32:38.103547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:38.103547+0000","objects_scrubbed":8231,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149684224,"num_objects":2284,"num_object_clones":0,"num_object_copies":4568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17172,"num_write_kb":622592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1124,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-23T16:36:14.191960+0000","last_change":"2024-04-23T16:33:25.851379+0000","last_active":"2024-04-23T16:36:14.191960+0000","last_peered":"2024-04-23T16:36:14.191960+0000","last_clean":"2024-04-23T16:36:14.191960+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:36:14.191960+0000","last_undegraded":"2024-04-23T16:36:14.191960+0000","last_fullsized":"2024-04-23T16:36:14.191960+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.498820+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.498820+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"433'16655","reported_seq":16991,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-04-23T16:36:11.794326+0000","last_change":"2024-04-23T16:34:41.375046+0000","last_active":"2024-04-23T16:36:11.794326+0000","last_peered":"2024-04-23T16:36:11.794326+0000","last_clean":"2024-04-23T16:36:11.794326+0000","last_became_active":"2024-04-23T16:31:29.140251+0000","last_became_peered":"2024-04-23T16:31:29.140251+0000","last_unstale":"2024-04-23T16:36:11.794326+0000","last_undegraded":"2024-04-23T16:36:11.794326+0000","last_fullsized":"2024-04-23T16:36:11.794326+0000","mapping_epoch":359,"log_start":"377'6600","ondisk_log_start":"377'6600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12511","last_scrub_stamp":"2024-04-23T16:34:41.374888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:41.374888+0000","objects_scrubbed":6643,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:13.238388+0000","scrub_duration":1271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162267160,"num_objects":2477,"num_object_clones":0,"num_object_copies":4954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16655,"num_write_kb":612161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1115,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-23T16:36:14.209949+0000","last_change":"2024-04-23T16:32:29.517947+0000","last_active":"2024-04-23T16:36:14.209949+0000","last_peered":"2024-04-23T16:36:14.209949+0000","last_clean":"2024-04-23T16:36:14.209949+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:36:14.209949+0000","last_undegraded":"2024-04-23T16:36:14.209949+0000","last_fullsized":"2024-04-23T16:36:14.209949+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:32:29.517800+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:32:29.517800+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"434'17167","reported_seq":32381,"reported_epoch":434,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:36:15.196676+0000","last_change":"2024-04-23T16:33:26.507399+0000","last_active":"2024-04-23T16:36:15.196676+0000","last_peered":"2024-04-23T16:36:15.196676+0000","last_clean":"2024-04-23T16:32:59.715575+0000","last_became_active":"2024-04-23T16:33:26.458611+0000","last_became_peered":"2024-04-23T16:33:26.458611+0000","last_unstale":"2024-04-23T16:36:15.196676+0000","last_undegraded":"2024-04-23T16:33:25.895472+0000","last_fullsized":"2024-04-23T16:36:15.196676+0000","mapping_epoch":394,"log_start":"378'7100","ondisk_log_start":"378'7100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8607","last_scrub_stamp":"2024-04-23T16:32:43.130806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.130806+0000","objects_scrubbed":8577,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145031168,"num_objects":2213,"num_object_clones":0,"num_object_copies":4426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17167,"num_write_kb":620160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7478,"num_bytes_recovered":490078208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":2209},{"shards":"5,7","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1127,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-04-23T16:36:09.177372+0000","last_change":"2024-04-23T16:34:47.438492+0000","last_active":"2024-04-23T16:36:09.177372+0000","last_peered":"2024-04-23T16:36:09.177372+0000","last_clean":"2024-04-23T16:36:09.177372+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:36:09.177372+0000","last_undegraded":"2024-04-23T16:36:09.177372+0000","last_fullsized":"2024-04-23T16:36:09.177372+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:47.438356+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:47.438356+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:13.051838+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"434'16723","reported_seq":16985,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-23T16:36:15.143260+0000","last_change":"2024-04-23T16:33:18.576077+0000","last_active":"2024-04-23T16:36:15.143260+0000","last_peered":"2024-04-23T16:36:15.143260+0000","last_clean":"2024-04-23T16:36:15.143260+0000","last_became_active":"2024-04-23T16:31:29.139709+0000","last_became_peered":"2024-04-23T16:31:29.139709+0000","last_unstale":"2024-04-23T16:36:15.143260+0000","last_undegraded":"2024-04-23T16:36:15.143260+0000","last_fullsized":"2024-04-23T16:36:15.143260+0000","mapping_epoch":359,"log_start":"377'6700","ondisk_log_start":"377'6700","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9472","last_scrub_stamp":"2024-04-23T16:33:18.575945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.575945+0000","objects_scrubbed":9472,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145555456,"num_objects":2221,"num_object_clones":0,"num_object_copies":4442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16723,"num_write_kb":606208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1130,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-23T16:36:14.188912+0000","last_change":"2024-04-23T16:35:34.878630+0000","last_active":"2024-04-23T16:36:14.188912+0000","last_peered":"2024-04-23T16:36:14.188912+0000","last_clean":"2024-04-23T16:36:14.188912+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:36:14.188912+0000","last_undegraded":"2024-04-23T16:36:14.188912+0000","last_fullsized":"2024-04-23T16:36:14.188912+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:35:34.878536+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:35:34.878536+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:52.503403+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"433'17110","reported_seq":17449,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-04-23T16:36:14.171768+0000","last_change":"2024-04-23T16:34:44.348696+0000","last_active":"2024-04-23T16:36:14.171768+0000","last_peered":"2024-04-23T16:36:14.171768+0000","last_clean":"2024-04-23T16:36:14.171768+0000","last_became_active":"2024-04-23T16:31:29.139606+0000","last_became_peered":"2024-04-23T16:31:29.139606+0000","last_unstale":"2024-04-23T16:36:14.171768+0000","last_undegraded":"2024-04-23T16:36:14.171768+0000","last_fullsized":"2024-04-23T16:36:14.171768+0000","mapping_epoch":359,"log_start":"378'7100","ondisk_log_start":"378'7100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12750","last_scrub_stamp":"2024-04-23T16:34:44.348550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:44.348550+0000","objects_scrubbed":6711,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:15.306361+0000","scrub_duration":1024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152698880,"num_objects":2330,"num_object_clones":0,"num_object_copies":4660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17110,"num_write_kb":622080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1129,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-23T16:36:14.192881+0000","last_change":"2024-04-23T16:34:51.848140+0000","last_active":"2024-04-23T16:36:14.192881+0000","last_peered":"2024-04-23T16:36:14.192881+0000","last_clean":"2024-04-23T16:36:14.192881+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:36:14.192881+0000","last_undegraded":"2024-04-23T16:36:14.192881+0000","last_fullsized":"2024-04-23T16:36:14.192881+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:51.847992+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:34:51.847992+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:18.910354+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.2","version":"434'16773","reported_seq":17165,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-23T16:36:14.610670+0000","last_change":"2024-04-23T16:35:32.569280+0000","last_active":"2024-04-23T16:36:14.610670+0000","last_peered":"2024-04-23T16:36:14.610670+0000","last_clean":"2024-04-23T16:36:14.610670+0000","last_became_active":"2024-04-23T16:31:29.138430+0000","last_became_peered":"2024-04-23T16:31:29.138430+0000","last_unstale":"2024-04-23T16:36:14.610670+0000","last_undegraded":"2024-04-23T16:36:14.610670+0000","last_fullsized":"2024-04-23T16:36:14.610670+0000","mapping_epoch":359,"log_start":"377'6700","ondisk_log_start":"377'6700","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'14723","last_scrub_stamp":"2024-04-23T16:35:32.569137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:32.569137+0000","objects_scrubbed":4294,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:34.926251+0000","scrub_duration":850,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145686528,"num_objects":2223,"num_object_clones":0,"num_object_copies":4446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16773,"num_write_kb":607872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1130,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-23T16:36:14.193015+0000","last_change":"2024-04-23T16:34:37.693617+0000","last_active":"2024-04-23T16:36:14.193015+0000","last_peered":"2024-04-23T16:36:14.193015+0000","last_clean":"2024-04-23T16:36:14.193015+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:36:14.193015+0000","last_undegraded":"2024-04-23T16:36:14.193015+0000","last_fullsized":"2024-04-23T16:36:14.193015+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:37.693482+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:37.693482+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:15.078772+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"433'19108","reported_seq":19573,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-04-23T16:36:14.148568+0000","last_change":"2024-04-23T16:34:40.604268+0000","last_active":"2024-04-23T16:36:14.148568+0000","last_peered":"2024-04-23T16:36:14.148568+0000","last_clean":"2024-04-23T16:36:14.148568+0000","last_became_active":"2024-04-23T16:31:29.139398+0000","last_became_peered":"2024-04-23T16:31:29.139398+0000","last_unstale":"2024-04-23T16:36:14.148568+0000","last_undegraded":"2024-04-23T16:36:14.148568+0000","last_fullsized":"2024-04-23T16:36:14.148568+0000","mapping_epoch":359,"log_start":"383'9100","ondisk_log_start":"383'9100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'14807","last_scrub_stamp":"2024-04-23T16:34:40.604136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:34:40.604136+0000","objects_scrubbed":6625,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:15.306395+0000","scrub_duration":1231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150339584,"num_objects":2294,"num_object_clones":0,"num_object_copies":4588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16834,"num_write_kb":612096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1130,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-23T16:36:14.191556+0000","last_change":"2024-04-23T16:35:33.875035+0000","last_active":"2024-04-23T16:36:14.191556+0000","last_peered":"2024-04-23T16:36:14.191556+0000","last_clean":"2024-04-23T16:36:14.191556+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:36:14.191556+0000","last_undegraded":"2024-04-23T16:36:14.191556+0000","last_fullsized":"2024-04-23T16:36:14.191556+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:35:33.874904+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:35:33.874904+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:59.611256+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"434'16660","reported_seq":16912,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-23T16:36:14.462840+0000","last_change":"2024-04-23T16:32:43.947374+0000","last_active":"2024-04-23T16:36:14.462840+0000","last_peered":"2024-04-23T16:36:14.462840+0000","last_clean":"2024-04-23T16:36:14.462840+0000","last_became_active":"2024-04-23T16:31:29.138968+0000","last_became_peered":"2024-04-23T16:31:29.138968+0000","last_unstale":"2024-04-23T16:36:14.462840+0000","last_undegraded":"2024-04-23T16:36:14.462840+0000","last_fullsized":"2024-04-23T16:36:14.462840+0000","mapping_epoch":359,"log_start":"377'6600","ondisk_log_start":"377'6600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8482","last_scrub_stamp":"2024-04-23T16:32:43.947177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.947177+0000","objects_scrubbed":8428,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146407424,"num_objects":2234,"num_object_clones":0,"num_object_copies":4468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16660,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1019,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-04-23T16:36:09.210716+0000","last_change":"2024-04-23T16:34:51.248895+0000","last_active":"2024-04-23T16:36:09.210716+0000","last_peered":"2024-04-23T16:36:09.210716+0000","last_clean":"2024-04-23T16:36:09.210716+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:36:09.210716+0000","last_undegraded":"2024-04-23T16:36:09.210716+0000","last_fullsized":"2024-04-23T16:36:09.210716+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:51.248783+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:51.248783+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:13.233910+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"434'17186","reported_seq":17436,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-23T16:36:14.513492+0000","last_change":"2024-04-23T16:32:39.103230+0000","last_active":"2024-04-23T16:36:14.513492+0000","last_peered":"2024-04-23T16:36:14.513492+0000","last_clean":"2024-04-23T16:36:14.513492+0000","last_became_active":"2024-04-23T16:31:29.310246+0000","last_became_peered":"2024-04-23T16:31:29.310246+0000","last_unstale":"2024-04-23T16:36:14.513492+0000","last_undegraded":"2024-04-23T16:36:14.513492+0000","last_fullsized":"2024-04-23T16:36:14.513492+0000","mapping_epoch":359,"log_start":"378'7100","ondisk_log_start":"378'7100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8325","last_scrub_stamp":"2024-04-23T16:32:39.103059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:39.103059+0000","objects_scrubbed":8272,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149159936,"num_objects":2276,"num_object_clones":0,"num_object_copies":4552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17186,"num_write_kb":622784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":943,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-23T16:36:14.188351+0000","last_change":"2024-04-23T16:34:56.453746+0000","last_active":"2024-04-23T16:36:14.188351+0000","last_peered":"2024-04-23T16:36:14.188351+0000","last_clean":"2024-04-23T16:36:14.188351+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:36:14.188351+0000","last_undegraded":"2024-04-23T16:36:14.188351+0000","last_fullsized":"2024-04-23T16:36:14.188351+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:56.453617+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:34:56.453617+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:14.873901+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"434'16915","reported_seq":31893,"reported_epoch":434,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:36:15.189339+0000","last_change":"2024-04-23T16:33:26.132487+0000","last_active":"2024-04-23T16:36:15.189339+0000","last_peered":"2024-04-23T16:36:15.189339+0000","last_clean":"2024-04-23T16:32:59.695133+0000","last_became_active":"2024-04-23T16:33:26.078062+0000","last_became_peered":"2024-04-23T16:33:26.078062+0000","last_unstale":"2024-04-23T16:36:15.189339+0000","last_undegraded":"2024-04-23T16:33:25.892009+0000","last_fullsized":"2024-04-23T16:36:15.189339+0000","mapping_epoch":394,"log_start":"378'6900","ondisk_log_start":"378'6900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'7968","last_scrub_stamp":"2024-04-23T16:32:36.102122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:36.102122+0000","objects_scrubbed":7926,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144113664,"num_objects":2199,"num_object_clones":0,"num_object_copies":4398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16915,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7362,"num_bytes_recovered":482476032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":2191},{"shards":"5,8","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1120,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-23T16:36:14.209837+0000","last_change":"2024-04-23T16:33:17.354409+0000","last_active":"2024-04-23T16:36:14.209837+0000","last_peered":"2024-04-23T16:36:14.209837+0000","last_clean":"2024-04-23T16:36:14.209837+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:36:14.209837+0000","last_undegraded":"2024-04-23T16:36:14.209837+0000","last_fullsized":"2024-04-23T16:36:14.209837+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:17.354263+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.354263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":2384723992,"num_objects":36389,"num_object_clones":0,"num_object_copies":72778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6731,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36389,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":270803,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22221,"num_bytes_recovered":1456275456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6202990592,"data_stored":6202982448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160734,"ondisk_log_size":160734,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056891,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":812192,"kb_used_data":315924,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":92932960,"statfs":{"total":95995035648,"available":95163351040,"internally_reserved":0,"allocated":323506176,"data_stored":323067940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":508164548},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351233,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612871,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":802456,"kb_used_data":433936,"kb_used_omap":1,"kb_used_meta":368510,"kb_avail":92942696,"statfs":{"total":95995035648,"available":95173320704,"internally_reserved":0,"allocated":444350464,"data_stored":443916300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":377354688},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841803,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":580952,"kb_used_data":301264,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":93164200,"statfs":{"total":95995035648,"available":95400140800,"internally_reserved":0,"allocated":308494336,"data_stored":308060172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168848,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":543208,"kb_used_data":146844,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93201944,"statfs":{"total":95995035648,"available":95438790656,"internally_reserved":0,"allocated":150368256,"data_stored":149928038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":405797322},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463190,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":733912,"kb_used_data":441680,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93011240,"statfs":{"total":95995035648,"available":95243509760,"internally_reserved":0,"allocated":452280320,"data_stored":451846156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":299170246},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137330,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738675,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1489880,"kb_used_data":1039376,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":92255272,"statfs":{"total":95995035648,"available":94469398528,"internally_reserved":0,"allocated":1064321024,"data_stored":1063886860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117284,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":499376,"kb_used_data":143332,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93245776,"statfs":{"total":95995035648,"available":95483674624,"internally_reserved":0,"allocated":146771968,"data_stored":146328198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":364575175},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921054,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1298008,"kb_used_data":739280,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92447144,"statfs":{"total":95995035648,"available":94665875456,"internally_reserved":0,"allocated":757022720,"data_stored":756588556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757529,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1137844,"kb_used_data":727656,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92607308,"statfs":{"total":95995035648,"available":94829883392,"internally_reserved":0,"allocated":745119744,"data_stored":744671902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1044512768,"data_stored":1044512768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1886846976,"data_stored":1886846976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":144769024,"data_stored":144769024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":745537536,"data_stored":745537536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":730140672,"data_stored":730136600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":443154432,"data_stored":443154432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":147390464,"data_stored":147390464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":302186496,"data_stored":302186496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":440008704,"data_stored":440008704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":318443520,"data_stored":318439448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:36:18.171 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.a is not active+clean 2024-04-23T16:36:18.171 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.a', 'version': "433'17101", 'reported_seq': 32122, 'reported_epoch': 433, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:36:13.205140+0000', 'last_change': '2024-04-23T16:33:26.432192+0000', 'last_active': '2024-04-23T16:36:13.205140+0000', 'last_peered': '2024-04-23T16:36:13.205140+0000', 'last_clean': '2024-04-23T16:32:59.749530+0000', 'last_became_active': '2024-04-23T16:33:26.338644+0000', 'last_became_peered': '2024-04-23T16:33:26.338644+0000', 'last_unstale': '2024-04-23T16:36:13.205140+0000', 'last_undegraded': '2024-04-23T16:33:25.874847+0000', 'last_fullsized': '2024-04-23T16:36:13.205140+0000', 'mapping_epoch': 394, 'log_start': "378'7100", 'ondisk_log_start': "378'7100", 'created': 359, 'last_epoch_clean': 360, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "382'8711", 'last_scrub_stamp': '2024-04-23T16:32:43.928914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:31:28.121358+0000', 'last_clean_scrub_stamp': '2024-04-23T16:32:43.928914+0000', 'objects_scrubbed': 8680, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1273, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 153419776, 'num_objects': 2341, 'num_object_clones': 0, 'num_object_copies': 4682, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2331, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2341, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17101, 'num_write_kb': 622144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7381, 'num_bytes_recovered': 483721216, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [8, 4], 'acting': [8, 4], 'avail_no_missing': ['8'], 'object_location_counts': [{'shards': '4,8', 'objects': 10}, {'shards': '8', 'objects': 2331}], 'blocked_by': [], 'up_primary': 8, 'acting_primary': 8, 'purged_snaps': []} 2024-04-23T16:36:18.171 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.6 is not active+clean 2024-04-23T16:36:18.171 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.6', 'version': "434'17167", 'reported_seq': 32381, 'reported_epoch': 434, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:36:15.196676+0000', 'last_change': '2024-04-23T16:33:26.507399+0000', 'last_active': '2024-04-23T16:36:15.196676+0000', 'last_peered': '2024-04-23T16:36:15.196676+0000', 'last_clean': '2024-04-23T16:32:59.715575+0000', 'last_became_active': '2024-04-23T16:33:26.458611+0000', 'last_became_peered': '2024-04-23T16:33:26.458611+0000', 'last_unstale': '2024-04-23T16:36:15.196676+0000', 'last_undegraded': '2024-04-23T16:33:25.895472+0000', 'last_fullsized': '2024-04-23T16:36:15.196676+0000', 'mapping_epoch': 394, 'log_start': "378'7100", 'ondisk_log_start': "378'7100", 'created': 359, 'last_epoch_clean': 360, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "382'8607", 'last_scrub_stamp': '2024-04-23T16:32:43.130806+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:31:28.121358+0000', 'last_clean_scrub_stamp': '2024-04-23T16:32:43.130806+0000', 'objects_scrubbed': 8577, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1484, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 145031168, 'num_objects': 2213, 'num_object_clones': 0, 'num_object_copies': 4426, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2209, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2213, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17167, 'num_write_kb': 620160, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7478, 'num_bytes_recovered': 490078208, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': ['5'], 'object_location_counts': [{'shards': '5', 'objects': 2209}, {'shards': '5,7', 'objects': 4}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-23T16:36:18.171 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.4 is not active+clean 2024-04-23T16:36:18.171 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.4', 'version': "434'16915", 'reported_seq': 31893, 'reported_epoch': 434, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:36:15.189339+0000', 'last_change': '2024-04-23T16:33:26.132487+0000', 'last_active': '2024-04-23T16:36:15.189339+0000', 'last_peered': '2024-04-23T16:36:15.189339+0000', 'last_clean': '2024-04-23T16:32:59.695133+0000', 'last_became_active': '2024-04-23T16:33:26.078062+0000', 'last_became_peered': '2024-04-23T16:33:26.078062+0000', 'last_unstale': '2024-04-23T16:36:15.189339+0000', 'last_undegraded': '2024-04-23T16:33:25.892009+0000', 'last_fullsized': '2024-04-23T16:36:15.189339+0000', 'mapping_epoch': 394, 'log_start': "378'6900", 'ondisk_log_start': "378'6900", 'created': 359, 'last_epoch_clean': 360, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "380'7968", 'last_scrub_stamp': '2024-04-23T16:32:36.102122+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:31:28.121358+0000', 'last_clean_scrub_stamp': '2024-04-23T16:32:36.102122+0000', 'objects_scrubbed': 7926, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1466, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 144113664, 'num_objects': 2199, 'num_object_clones': 0, 'num_object_copies': 4398, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2199, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16915, 'num_write_kb': 611648, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7362, 'num_bytes_recovered': 482476032, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 8], 'acting': [5, 8], 'avail_no_missing': ['5'], 'object_location_counts': [{'shards': '5', 'objects': 2191}, {'shards': '5,8', 'objects': 8}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-23T16:36:18.171 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-23T16:36:18.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:18.172 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:36:18.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:36:18.274 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:36:18.303 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:18.316 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:36:18.325 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:36:18.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:18.328 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:36:18.429 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:36:18.509 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:18.587 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:36:18.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:18.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:36:18.698 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:36:18.730 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:18.751 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:36:18.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:18.754 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:36:18.833 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:36:18.847 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:36:18.854 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:36:18.887 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:19.038 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:18 smithi148 ceph-mon[44520]: from='client.26089 v1:172.21.15.103:0/4265056621' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:36:19.092 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:36:19.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:19.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:36:19.098 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:18 smithi103 ceph-mon[39629]: from='client.26089 v1:172.21.15.103:0/4265056621' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:36:19.197 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:36:19.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:18 smithi188 ceph-mon[44101]: from='client.26089 v1:172.21.15.103:0/4265056621' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:36:19.288 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:19 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T16:36:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:19 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T16:36:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:19 smithi103 ceph-mon[39629]: pgmap v1237: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.2 GiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 618 op/s; 6479/71204 objects degraded (9.099%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:36:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:19 smithi103 ceph-mon[39629]: 1.5 scrub starts 2024-04-23T16:36:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:19 smithi103 ceph-mon[39629]: 1.5 scrub ok 2024-04-23T16:36:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:19 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 6479/71204 objects degraded (9.099%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:19 smithi103 ceph-mon[39629]: osdmap e436: 12 total, 11 up, 9 in 2024-04-23T16:36:20.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:19 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T16:36:20.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:19 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T16:36:20.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:19 smithi148 ceph-mon[44520]: pgmap v1237: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.2 GiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 618 op/s; 6479/71204 objects degraded (9.099%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:36:20.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:19 smithi148 ceph-mon[44520]: 1.5 scrub starts 2024-04-23T16:36:20.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:19 smithi148 ceph-mon[44520]: 1.5 scrub ok 2024-04-23T16:36:20.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:19 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 6479/71204 objects degraded (9.099%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:20.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:19 smithi148 ceph-mon[44520]: osdmap e436: 12 total, 11 up, 9 in 2024-04-23T16:36:20.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:19 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T16:36:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:19 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T16:36:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:19 smithi188 ceph-mon[44101]: pgmap v1237: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.2 GiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 618 op/s; 6479/71204 objects degraded (9.099%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:36:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:19 smithi188 ceph-mon[44101]: 1.5 scrub starts 2024-04-23T16:36:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:19 smithi188 ceph-mon[44101]: 1.5 scrub ok 2024-04-23T16:36:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:19 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 6479/71204 objects degraded (9.099%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:19 smithi188 ceph-mon[44101]: osdmap e436: 12 total, 11 up, 9 in 2024-04-23T16:36:20.727 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:36:20.776 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:36:20.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:20.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:36:20.884 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:36:20.926 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:21.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:20 smithi103 ceph-mon[39629]: 1.a scrub starts 2024-04-23T16:36:21.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:20 smithi103 ceph-mon[39629]: 1.a scrub ok 2024-04-23T16:36:21.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:20 smithi103 ceph-mon[39629]: 5.8 scrub starts 2024-04-23T16:36:21.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:20 smithi103 ceph-mon[39629]: 5.5 scrub starts 2024-04-23T16:36:21.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:20 smithi103 ceph-mon[39629]: 5.8 scrub ok 2024-04-23T16:36:21.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:20 smithi148 ceph-mon[44520]: 1.a scrub starts 2024-04-23T16:36:21.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:20 smithi148 ceph-mon[44520]: 1.a scrub ok 2024-04-23T16:36:21.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:20 smithi148 ceph-mon[44520]: 5.8 scrub starts 2024-04-23T16:36:21.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:20 smithi148 ceph-mon[44520]: 5.5 scrub starts 2024-04-23T16:36:21.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:20 smithi148 ceph-mon[44520]: 5.8 scrub ok 2024-04-23T16:36:21.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:20 smithi188 ceph-mon[44101]: 1.a scrub starts 2024-04-23T16:36:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:20 smithi188 ceph-mon[44101]: 1.a scrub ok 2024-04-23T16:36:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:20 smithi188 ceph-mon[44101]: 5.8 scrub starts 2024-04-23T16:36:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:20 smithi188 ceph-mon[44101]: 5.5 scrub starts 2024-04-23T16:36:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:20 smithi188 ceph-mon[44101]: 5.8 scrub ok 2024-04-23T16:36:21.316 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:36:21.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:21.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:36:21.420 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:36:21.455 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:21.582 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:21.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:21.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:21.685 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:36:21.754 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:21.898 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:36:21.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:21.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:36:22.007 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:36:22.038 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:22.098 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:36:22.098 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2024-04-23T16:36:22.098 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2024-04-23T16:36:22.098 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:36:22.098 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.098 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12658 4.d 4:be7ebf30:::benchmark_data_smithi188_101370_object12657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:36:22.099 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.230769+0000", 2024-04-23T16:36:22.099 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 541.85691439599998, 2024-04-23T16:36:22.099 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.204651206, 2024-04-23T16:36:22.099 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.099 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.099 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.099 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:22.099 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:36:22.099 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:36:22.099 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12658 2024-04-23T16:36:22.099 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.099 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.099 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.100 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.100 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.230769+0000", 2024-04-23T16:36:22.100 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.100 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.100 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.100 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.100 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.230769+0000", 2024-04-23T16:36:22.100 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.100 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.100 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.100 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.100 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.230770+0000", 2024-04-23T16:36:22.101 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.133e-06 2024-04-23T16:36:22.101 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.101 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.101 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.101 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.230846+0000", 2024-04-23T16:36:22.101 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5947999999999999e-05 2024-04-23T16:36:22.101 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.101 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.101 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.101 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.230853+0000", 2024-04-23T16:36:22.101 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7299999999999999e-06 2024-04-23T16:36:22.101 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.101 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.101 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.102 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.230858+0000", 2024-04-23T16:36:22.102 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3940000000000004e-06 2024-04-23T16:36:22.102 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.102 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.102 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.102 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.230908+0000", 2024-04-23T16:36:22.102 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9320999999999998e-05 2024-04-23T16:36:22.102 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.102 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.102 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.102 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.231012+0000", 2024-04-23T16:36:22.102 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000104196 2024-04-23T16:36:22.102 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.103 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.103 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:36:22.103 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.231122+0000", 2024-04-23T16:36:22.103 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000110728 2024-04-23T16:36:22.103 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.103 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.103 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:22.103 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.232467+0000", 2024-04-23T16:36:22.103 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0013445950000000001 2024-04-23T16:36:22.103 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.103 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.103 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:22.103 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.435342+0000", 2024-04-23T16:36:22.103 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20287460399999999 2024-04-23T16:36:22.104 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.104 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.104 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.104 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.435384+0000", 2024-04-23T16:36:22.104 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1915000000000001e-05 2024-04-23T16:36:22.104 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.104 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.104 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.104 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.435420+0000", 2024-04-23T16:36:22.104 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6641999999999999e-05 2024-04-23T16:36:22.104 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.104 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.104 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.104 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.105 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.105 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12698 4.d 4:bce91761:::benchmark_data_smithi188_101370_object12697:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:36:22.105 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.238200+0000", 2024-04-23T16:36:22.105 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 541.84948335499996, 2024-04-23T16:36:22.105 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19737084899999999, 2024-04-23T16:36:22.105 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.105 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.105 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.105 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:22.105 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:36:22.105 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:36:22.105 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12698 2024-04-23T16:36:22.106 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.106 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.106 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.106 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.106 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.238200+0000", 2024-04-23T16:36:22.106 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.106 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.106 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.106 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.106 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.238200+0000", 2024-04-23T16:36:22.106 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.106 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.106 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.106 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.107 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.238202+0000", 2024-04-23T16:36:22.107 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9910000000000001e-06 2024-04-23T16:36:22.107 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.107 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.107 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.107 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.238232+0000", 2024-04-23T16:36:22.107 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9935999999999999e-05 2024-04-23T16:36:22.107 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.107 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.107 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.107 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.238243+0000", 2024-04-23T16:36:22.107 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0747000000000001e-05 2024-04-23T16:36:22.107 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.107 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.108 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.108 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.238248+0000", 2024-04-23T16:36:22.108 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.716e-06 2024-04-23T16:36:22.108 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.108 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.108 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.108 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.238296+0000", 2024-04-23T16:36:22.108 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7821000000000002e-05 2024-04-23T16:36:22.108 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.108 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.108 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.108 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.238394+0000", 2024-04-23T16:36:22.108 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7775999999999994e-05 2024-04-23T16:36:22.109 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.109 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.109 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:36:22.109 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.238499+0000", 2024-04-23T16:36:22.109 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000104931 2024-04-23T16:36:22.109 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.109 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.109 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:22.109 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.239359+0000", 2024-04-23T16:36:22.109 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00086016700000000001 2024-04-23T16:36:22.109 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.109 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.109 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:22.109 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.435482+0000", 2024-04-23T16:36:22.110 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19612264500000001 2024-04-23T16:36:22.110 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.110 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.110 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.110 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.435542+0000", 2024-04-23T16:36:22.110 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0174999999999999e-05 2024-04-23T16:36:22.110 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.110 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.110 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.110 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.435571+0000", 2024-04-23T16:36:22.110 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8943999999999999e-05 2024-04-23T16:36:22.110 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.110 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.110 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.111 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.111 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.111 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12758 4.d 4:b109748d:::benchmark_data_smithi188_101370_object12757:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:36:22.111 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.251139+0000", 2024-04-23T16:36:22.111 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 541.83654405100003, 2024-04-23T16:36:22.111 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.185709438, 2024-04-23T16:36:22.111 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.111 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.111 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.111 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:22.111 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:36:22.111 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:36:22.111 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12758 2024-04-23T16:36:22.112 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.112 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.112 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.112 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.112 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251139+0000", 2024-04-23T16:36:22.112 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.112 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.112 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.112 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.112 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251139+0000", 2024-04-23T16:36:22.112 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.112 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.112 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.112 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.113 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251141+0000", 2024-04-23T16:36:22.113 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.951e-06 2024-04-23T16:36:22.113 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.113 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.113 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.113 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251179+0000", 2024-04-23T16:36:22.113 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7208e-05 2024-04-23T16:36:22.113 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.113 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.113 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.113 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251189+0000", 2024-04-23T16:36:22.113 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0556e-05 2024-04-23T16:36:22.113 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.113 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.114 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.114 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251197+0000", 2024-04-23T16:36:22.114 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7880000000000007e-06 2024-04-23T16:36:22.114 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.114 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.114 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.114 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251246+0000", 2024-04-23T16:36:22.114 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9379000000000003e-05 2024-04-23T16:36:22.114 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.114 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.114 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.114 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251329+0000", 2024-04-23T16:36:22.114 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2973000000000007e-05 2024-04-23T16:36:22.114 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.115 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.115 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:36:22.115 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251466+0000", 2024-04-23T16:36:22.115 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000137112 2024-04-23T16:36:22.115 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.115 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.115 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:22.115 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.252783+0000", 2024-04-23T16:36:22.115 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001316371 2024-04-23T16:36:22.115 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.115 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.115 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:22.115 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.436792+0000", 2024-04-23T16:36:22.116 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.18400925300000001 2024-04-23T16:36:22.116 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.116 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.116 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.116 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.436824+0000", 2024-04-23T16:36:22.116 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2264999999999997e-05 2024-04-23T16:36:22.116 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.116 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.116 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.116 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.436849+0000", 2024-04-23T16:36:22.116 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4581999999999999e-05 2024-04-23T16:36:22.116 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.116 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.116 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.117 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.117 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.117 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12759 4.d 4:bdc675b3:::benchmark_data_smithi188_101370_object12758:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:36:22.117 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.251342+0000", 2024-04-23T16:36:22.117 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 541.83634183599997, 2024-04-23T16:36:22.117 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.18565109299999999, 2024-04-23T16:36:22.117 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.117 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.117 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.117 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:22.117 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:36:22.117 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:36:22.117 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12759 2024-04-23T16:36:22.117 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.118 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.118 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.118 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.118 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251342+0000", 2024-04-23T16:36:22.118 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.118 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.118 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.118 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.118 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251342+0000", 2024-04-23T16:36:22.118 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.118 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.118 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.118 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.119 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251343+0000", 2024-04-23T16:36:22.119 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2529999999999999e-06 2024-04-23T16:36:22.119 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.119 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.119 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.119 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251484+0000", 2024-04-23T16:36:22.119 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014138099999999999 2024-04-23T16:36:22.119 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.119 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.119 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.119 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251492+0000", 2024-04-23T16:36:22.119 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2530000000000004e-06 2024-04-23T16:36:22.119 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.119 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.120 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.120 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.251500+0000", 2024-04-23T16:36:22.120 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0940000000000002e-06 2024-04-23T16:36:22.120 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.120 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.120 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.120 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.252084+0000", 2024-04-23T16:36:22.120 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00058474700000000004 2024-04-23T16:36:22.120 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.120 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.120 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.120 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.252195+0000", 2024-04-23T16:36:22.120 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011098 2024-04-23T16:36:22.120 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.121 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.121 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:36:22.121 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.252335+0000", 2024-04-23T16:36:22.121 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013999600000000001 2024-04-23T16:36:22.121 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.121 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.121 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:22.121 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.252972+0000", 2024-04-23T16:36:22.121 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00063641100000000003 2024-04-23T16:36:22.121 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.121 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.121 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:22.121 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.436896+0000", 2024-04-23T16:36:22.121 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.18392444599999999 2024-04-23T16:36:22.122 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.122 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.122 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.122 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.436950+0000", 2024-04-23T16:36:22.122 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4018999999999999e-05 2024-04-23T16:36:22.122 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.122 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.122 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.122 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.436993+0000", 2024-04-23T16:36:22.122 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2512999999999998e-05 2024-04-23T16:36:22.122 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.122 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.122 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.122 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.123 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.123 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12789 4.d 4:ba9dcfb2:::benchmark_data_smithi188_101370_object12788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:36:22.123 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.257550+0000", 2024-04-23T16:36:22.123 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 541.83013344899996, 2024-04-23T16:36:22.123 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.17956171100000001, 2024-04-23T16:36:22.123 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.123 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.123 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.123 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:22.123 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:36:22.123 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:36:22.123 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12789 2024-04-23T16:36:22.123 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.123 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.124 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.124 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.124 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.257550+0000", 2024-04-23T16:36:22.124 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.124 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.124 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.124 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.124 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.257550+0000", 2024-04-23T16:36:22.124 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.124 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.124 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.124 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.124 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.257551+0000", 2024-04-23T16:36:22.124 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.24e-06 2024-04-23T16:36:22.125 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.125 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.125 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.125 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.257612+0000", 2024-04-23T16:36:22.125 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0853999999999997e-05 2024-04-23T16:36:22.125 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.125 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.125 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.125 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.257621+0000", 2024-04-23T16:36:22.125 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1770000000000004e-06 2024-04-23T16:36:22.125 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.125 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.125 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.125 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.257627+0000", 2024-04-23T16:36:22.126 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0560000000000001e-06 2024-04-23T16:36:22.126 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.126 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.126 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.126 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.257667+0000", 2024-04-23T16:36:22.126 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9279999999999997e-05 2024-04-23T16:36:22.126 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.126 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.126 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.126 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.257774+0000", 2024-04-23T16:36:22.126 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107816 2024-04-23T16:36:22.126 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.126 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.127 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:36:22.127 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.257890+0000", 2024-04-23T16:36:22.127 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011557000000000001 2024-04-23T16:36:22.127 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.127 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.127 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:22.127 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.259226+0000", 2024-04-23T16:36:22.127 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0013363769999999999 2024-04-23T16:36:22.127 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.127 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.127 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:22.127 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437047+0000", 2024-04-23T16:36:22.127 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.17782076899999999 2024-04-23T16:36:22.128 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.128 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.128 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.128 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437079+0000", 2024-04-23T16:36:22.128 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1715999999999999e-05 2024-04-23T16:36:22.128 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.128 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.128 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.128 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437112+0000", 2024-04-23T16:36:22.128 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2855999999999998e-05 2024-04-23T16:36:22.128 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.128 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.128 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.129 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.129 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.129 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12849 4.d 4:b91fa107:::benchmark_data_smithi188_101370_object12848:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:36:22.129 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.271006+0000", 2024-04-23T16:36:22.129 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 541.81667697, 2024-04-23T16:36:22.129 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.16626290699999999, 2024-04-23T16:36:22.129 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.129 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.129 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.129 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:22.129 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:36:22.129 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:36:22.129 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12849 2024-04-23T16:36:22.129 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.130 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.130 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.130 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.130 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.271006+0000", 2024-04-23T16:36:22.130 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.130 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.130 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.130 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.130 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.271006+0000", 2024-04-23T16:36:22.130 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.130 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.130 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.130 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.131 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.271008+0000", 2024-04-23T16:36:22.131 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6249999999999999e-06 2024-04-23T16:36:22.131 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.131 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.131 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.131 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.271058+0000", 2024-04-23T16:36:22.131 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0324000000000001e-05 2024-04-23T16:36:22.131 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.131 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.131 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.131 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.271070+0000", 2024-04-23T16:36:22.131 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2112e-05 2024-04-23T16:36:22.131 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.132 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.132 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.132 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.271078+0000", 2024-04-23T16:36:22.132 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.9899999999999997e-06 2024-04-23T16:36:22.132 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.132 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.132 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.132 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.271124+0000", 2024-04-23T16:36:22.132 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5089000000000001e-05 2024-04-23T16:36:22.132 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.132 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.132 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.132 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.271208+0000", 2024-04-23T16:36:22.133 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4578000000000002e-05 2024-04-23T16:36:22.133 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.133 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.133 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:36:22.133 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.271422+0000", 2024-04-23T16:36:22.133 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000213463 2024-04-23T16:36:22.133 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.133 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.133 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:22.133 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.272355+0000", 2024-04-23T16:36:22.133 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00093360399999999999 2024-04-23T16:36:22.133 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.133 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.133 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:22.133 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437172+0000", 2024-04-23T16:36:22.134 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.164816616 2024-04-23T16:36:22.134 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.134 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.134 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.134 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437228+0000", 2024-04-23T16:36:22.134 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6047000000000001e-05 2024-04-23T16:36:22.134 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.134 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.134 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.134 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437269+0000", 2024-04-23T16:36:22.134 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1458999999999997e-05 2024-04-23T16:36:22.134 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.134 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.134 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.135 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.135 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.135 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12862 4.d 4:b829ec64:::benchmark_data_smithi188_101370_object12861:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:36:22.135 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.273139+0000", 2024-04-23T16:36:22.135 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 541.81454396599997, 2024-04-23T16:36:22.135 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.16464094500000001, 2024-04-23T16:36:22.135 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.135 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.135 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.135 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:22.135 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:36:22.135 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:36:22.135 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12862 2024-04-23T16:36:22.135 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.135 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.136 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.136 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.136 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.273139+0000", 2024-04-23T16:36:22.136 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.136 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.136 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.136 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.136 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.273139+0000", 2024-04-23T16:36:22.136 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.136 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.136 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.136 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.136 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.273141+0000", 2024-04-23T16:36:22.136 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6920000000000001e-06 2024-04-23T16:36:22.137 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.137 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.137 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.137 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.273241+0000", 2024-04-23T16:36:22.137 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000100157 2024-04-23T16:36:22.137 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.137 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.137 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.137 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.273252+0000", 2024-04-23T16:36:22.137 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1012000000000001e-05 2024-04-23T16:36:22.137 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.137 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.137 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.137 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.273260+0000", 2024-04-23T16:36:22.138 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.028e-06 2024-04-23T16:36:22.138 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.138 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.138 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.138 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.273309+0000", 2024-04-23T16:36:22.138 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8310000000000003e-05 2024-04-23T16:36:22.138 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.138 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.138 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.138 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.273427+0000", 2024-04-23T16:36:22.138 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000118149 2024-04-23T16:36:22.138 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.138 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.138 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:36:22.139 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.273629+0000", 2024-04-23T16:36:22.139 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00020209399999999999 2024-04-23T16:36:22.139 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.139 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.139 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:22.139 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.274363+0000", 2024-04-23T16:36:22.139 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00073423200000000003 2024-04-23T16:36:22.139 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.139 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.139 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:22.139 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437621+0000", 2024-04-23T16:36:22.139 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.163258073 2024-04-23T16:36:22.139 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.139 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.140 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.140 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437676+0000", 2024-04-23T16:36:22.140 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5061e-05 2024-04-23T16:36:22.140 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.140 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.140 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.140 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437780+0000", 2024-04-23T16:36:22.140 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000104137 2024-04-23T16:36:22.140 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.140 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.140 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.140 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.140 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.141 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12906 4.d 4:bb06cb71:::benchmark_data_smithi188_101370_object12905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:36:22.141 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.283740+0000", 2024-04-23T16:36:22.141 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 541.803943839, 2024-04-23T16:36:22.141 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.15417470999999999, 2024-04-23T16:36:22.141 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.141 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.141 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.141 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:22.141 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:36:22.141 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:36:22.141 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12906 2024-04-23T16:36:22.141 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.141 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.142 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.142 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.142 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.283740+0000", 2024-04-23T16:36:22.142 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.142 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.142 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.142 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.142 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.283740+0000", 2024-04-23T16:36:22.142 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.142 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.142 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.142 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.142 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.283741+0000", 2024-04-23T16:36:22.143 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.55e-06 2024-04-23T16:36:22.143 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.143 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.143 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.143 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.283821+0000", 2024-04-23T16:36:22.143 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0289000000000003e-05 2024-04-23T16:36:22.143 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.143 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.143 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.143 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.283834+0000", 2024-04-23T16:36:22.143 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2799000000000001e-05 2024-04-23T16:36:22.143 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.143 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.144 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.144 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.283842+0000", 2024-04-23T16:36:22.144 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0390000000000005e-06 2024-04-23T16:36:22.144 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.144 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.144 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.144 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.283896+0000", 2024-04-23T16:36:22.144 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3856000000000001e-05 2024-04-23T16:36:22.144 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.144 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.144 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.144 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.284029+0000", 2024-04-23T16:36:22.144 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000133079 2024-04-23T16:36:22.145 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.145 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.145 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:36:22.145 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.284168+0000", 2024-04-23T16:36:22.145 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013844000000000001 2024-04-23T16:36:22.145 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.145 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.145 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:22.145 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.284799+0000", 2024-04-23T16:36:22.145 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00063107599999999995 2024-04-23T16:36:22.145 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.145 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.145 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:22.146 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437820+0000", 2024-04-23T16:36:22.146 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.15302147299999999 2024-04-23T16:36:22.146 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.146 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.146 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.146 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437872+0000", 2024-04-23T16:36:22.146 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2173999999999999e-05 2024-04-23T16:36:22.146 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.146 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.146 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.146 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.437914+0000", 2024-04-23T16:36:22.146 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1934999999999997e-05 2024-04-23T16:36:22.146 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.146 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.147 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.147 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.147 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.147 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12952 4.d 4:b4108fc3:::benchmark_data_smithi188_101370_object12951:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:36:22.147 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.294705+0000", 2024-04-23T16:36:22.147 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 541.79297791399995, 2024-04-23T16:36:22.147 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.14389808100000001, 2024-04-23T16:36:22.147 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.147 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.147 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.147 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:22.147 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:36:22.147 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:36:22.148 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12952 2024-04-23T16:36:22.148 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.148 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.148 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.148 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.148 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294705+0000", 2024-04-23T16:36:22.148 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.148 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.148 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.148 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.148 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294705+0000", 2024-04-23T16:36:22.148 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.148 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.149 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.149 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.149 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294707+0000", 2024-04-23T16:36:22.149 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5400000000000001e-06 2024-04-23T16:36:22.149 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.149 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.149 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.149 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294786+0000", 2024-04-23T16:36:22.149 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.9336000000000001e-05 2024-04-23T16:36:22.149 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.149 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.149 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.149 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294797+0000", 2024-04-23T16:36:22.149 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0261999999999999e-05 2024-04-23T16:36:22.150 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.150 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.150 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.150 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294806+0000", 2024-04-23T16:36:22.150 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3640000000000005e-06 2024-04-23T16:36:22.150 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.150 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.150 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.150 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294850+0000", 2024-04-23T16:36:22.150 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4348999999999998e-05 2024-04-23T16:36:22.150 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.150 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.150 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.150 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294956+0000", 2024-04-23T16:36:22.151 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010563799999999999 2024-04-23T16:36:22.151 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.151 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.151 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:36:22.151 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.295084+0000", 2024-04-23T16:36:22.151 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012836999999999999 2024-04-23T16:36:22.151 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.151 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.151 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:22.151 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.296273+0000", 2024-04-23T16:36:22.151 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0011887899999999999 2024-04-23T16:36:22.151 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.151 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.151 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:22.151 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.438514+0000", 2024-04-23T16:36:22.152 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.142240754 2024-04-23T16:36:22.152 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.152 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.152 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.152 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.438566+0000", 2024-04-23T16:36:22.152 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.223e-05 2024-04-23T16:36:22.152 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.152 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.152 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.152 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.438604+0000", 2024-04-23T16:36:22.152 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7447999999999998e-05 2024-04-23T16:36:22.152 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.152 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.152 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.152 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.152 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.153 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:12954 4.d 4:b54e759f:::benchmark_data_smithi188_101370_object12953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:36:22.153 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.294975+0000", 2024-04-23T16:36:22.153 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 541.79270886100005, 2024-04-23T16:36:22.153 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.14378174599999999, 2024-04-23T16:36:22.153 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.153 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.153 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.153 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:22.153 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:36:22.153 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:36:22.153 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 12954 2024-04-23T16:36:22.153 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.153 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.153 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.153 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.153 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294975+0000", 2024-04-23T16:36:22.154 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.154 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.154 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.154 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.154 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294975+0000", 2024-04-23T16:36:22.154 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.154 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.154 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.154 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.154 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.294976+0000", 2024-04-23T16:36:22.154 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3740000000000001e-06 2024-04-23T16:36:22.154 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.154 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.155 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.155 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.295058+0000", 2024-04-23T16:36:22.155 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1865000000000003e-05 2024-04-23T16:36:22.155 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.155 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.155 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.155 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.295067+0000", 2024-04-23T16:36:22.155 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2410000000000001e-06 2024-04-23T16:36:22.155 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.155 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.155 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.155 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.295075+0000", 2024-04-23T16:36:22.155 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5700000000000004e-06 2024-04-23T16:36:22.155 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.155 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.156 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.156 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.295662+0000", 2024-04-23T16:36:22.156 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00058759600000000002 2024-04-23T16:36:22.156 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.156 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.156 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.156 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.295781+0000", 2024-04-23T16:36:22.156 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011877100000000001 2024-04-23T16:36:22.156 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.156 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.156 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:36:22.156 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.295921+0000", 2024-04-23T16:36:22.156 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013954200000000001 2024-04-23T16:36:22.156 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.156 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.156 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:22.157 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.296878+0000", 2024-04-23T16:36:22.157 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000957988 2024-04-23T16:36:22.157 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.157 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.157 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:22.157 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.438655+0000", 2024-04-23T16:36:22.157 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.141776021 2024-04-23T16:36:22.157 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.157 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.157 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.157 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.438716+0000", 2024-04-23T16:36:22.157 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1402000000000006e-05 2024-04-23T16:36:22.157 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.157 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.157 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.158 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.438756+0000", 2024-04-23T16:36:22.158 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0376000000000001e-05 2024-04-23T16:36:22.158 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.158 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.158 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.158 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.158 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.158 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:13024 4.d 4:b1add8c7:::benchmark_data_smithi188_101370_object13023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-04-23T16:36:22.158 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:20.315294+0000", 2024-04-23T16:36:22.158 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 541.77238918600005, 2024-04-23T16:36:22.158 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.124104795, 2024-04-23T16:36:22.158 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.158 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.159 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.159 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:22.159 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:36:22.159 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:36:22.159 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 13024 2024-04-23T16:36:22.159 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.159 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.159 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.159 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.159 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.315294+0000", 2024-04-23T16:36:22.159 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.159 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.159 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.159 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.160 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.315294+0000", 2024-04-23T16:36:22.160 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.160 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.160 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.160 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.160 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.315296+0000", 2024-04-23T16:36:22.160 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.418e-06 2024-04-23T16:36:22.160 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.160 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.160 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.160 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.315352+0000", 2024-04-23T16:36:22.160 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6385000000000003e-05 2024-04-23T16:36:22.160 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.160 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.161 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.161 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.315361+0000", 2024-04-23T16:36:22.161 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5120000000000008e-06 2024-04-23T16:36:22.161 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.161 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.161 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.161 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.315366+0000", 2024-04-23T16:36:22.161 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0730000000000004e-06 2024-04-23T16:36:22.161 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.161 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.161 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.161 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.315411+0000", 2024-04-23T16:36:22.161 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5886000000000001e-05 2024-04-23T16:36:22.161 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.161 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.162 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.162 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.315520+0000", 2024-04-23T16:36:22.162 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000108687 2024-04-23T16:36:22.162 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.162 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.162 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:36:22.162 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.315627+0000", 2024-04-23T16:36:22.162 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000106735 2024-04-23T16:36:22.162 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.162 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.162 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:22.162 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.316337+0000", 2024-04-23T16:36:22.162 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00071045099999999996 2024-04-23T16:36:22.162 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.162 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.163 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:22.163 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.439333+0000", 2024-04-23T16:36:22.163 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.122995585 2024-04-23T16:36:22.163 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.163 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.163 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.163 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.439371+0000", 2024-04-23T16:36:22.163 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8538999999999998e-05 2024-04-23T16:36:22.163 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.163 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.163 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.163 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:20.439399+0000", 2024-04-23T16:36:22.163 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7523999999999999e-05 2024-04-23T16:36:22.163 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.163 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.164 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.164 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.164 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.164 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:269570 5.a e433/394 5:5e8de167:::benchmark_data_smithi188_113957_object115972:head v 433'17069, mlcod=379'7347)", 2024-04-23T16:36:22.164 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:36:12.811628+0000", 2024-04-23T16:36:22.164 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.2760552769999993, 2024-04-23T16:36:22.164 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.051803180999999997, 2024-04-23T16:36:22.164 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.164 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.164 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.164 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.164 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.164 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.164 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.811628+0000", 2024-04-23T16:36:22.164 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.165 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.165 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.165 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.165 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.811628+0000", 2024-04-23T16:36:22.165 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.165 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.165 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.165 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.165 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.811629+0000", 2024-04-23T16:36:22.165 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8899999999999998e-07 2024-04-23T16:36:22.165 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.165 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.165 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.165 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.811635+0000", 2024-04-23T16:36:22.165 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8629999999999996e-06 2024-04-23T16:36:22.165 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.165 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.166 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.166 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.811644+0000", 2024-04-23T16:36:22.166 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8319999999999995e-06 2024-04-23T16:36:22.166 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.166 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.166 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.166 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.811651+0000", 2024-04-23T16:36:22.166 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9959999999999996e-06 2024-04-23T16:36:22.166 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.166 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.166 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.166 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.818397+0000", 2024-04-23T16:36:22.166 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0067466740000000002 2024-04-23T16:36:22.166 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.166 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.167 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.167 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.818428+0000", 2024-04-23T16:36:22.167 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0378e-05 2024-04-23T16:36:22.167 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.167 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.167 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.167 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.863392+0000", 2024-04-23T16:36:22.167 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.044964280000000002 2024-04-23T16:36:22.167 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.167 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.167 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.167 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.863431+0000", 2024-04-23T16:36:22.167 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9369000000000002e-05 2024-04-23T16:36:22.167 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.167 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.168 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.168 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.168 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.168 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:269574 5.a e433/394 5:547615da:::benchmark_data_smithi188_113957_object115976:head v 433'17070, mlcod=379'7347)", 2024-04-23T16:36:22.168 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:36:12.818481+0000", 2024-04-23T16:36:22.168 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.2692021330000003, 2024-04-23T16:36:22.168 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.044989987000000002, 2024-04-23T16:36:22.168 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.168 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.168 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.168 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.168 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.168 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.168 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.818481+0000", 2024-04-23T16:36:22.168 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.169 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.169 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.169 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.169 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.818481+0000", 2024-04-23T16:36:22.169 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.169 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.169 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.169 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.169 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.818483+0000", 2024-04-23T16:36:22.169 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5629999999999999e-06 2024-04-23T16:36:22.169 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.169 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.169 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.169 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.818491+0000", 2024-04-23T16:36:22.169 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2300000000000008e-06 2024-04-23T16:36:22.169 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.170 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.170 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.170 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.818504+0000", 2024-04-23T16:36:22.170 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3266e-05 2024-04-23T16:36:22.170 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.170 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.170 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.170 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.818512+0000", 2024-04-23T16:36:22.170 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.289e-06 2024-04-23T16:36:22.170 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.170 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.170 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.170 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.824406+0000", 2024-04-23T16:36:22.171 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0058943789999999999 2024-04-23T16:36:22.171 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.171 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.171 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.171 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.824430+0000", 2024-04-23T16:36:22.171 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3816e-05 2024-04-23T16:36:22.171 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.171 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.171 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.171 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.863435+0000", 2024-04-23T16:36:22.171 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.039005297000000001 2024-04-23T16:36:22.171 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.171 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.172 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.172 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:12.863471+0000", 2024-04-23T16:36:22.172 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6146999999999998e-05 2024-04-23T16:36:22.172 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.172 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.172 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.172 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.172 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.172 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:271924 5.a e434/394 5:561795e3:::benchmark_data_smithi188_113957_object118326:head v 434'17222, mlcod=379'7500)", 2024-04-23T16:36:22.172 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:36:15.643537+0000", 2024-04-23T16:36:22.172 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 6.4441459659999998, 2024-04-23T16:36:22.172 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.047711628999999998, 2024-04-23T16:36:22.172 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.172 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.173 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.173 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.173 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.173 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.173 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:15.643537+0000", 2024-04-23T16:36:22.173 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.173 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.173 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.173 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.173 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:15.643537+0000", 2024-04-23T16:36:22.173 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.173 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.173 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.174 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.174 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:15.643539+0000", 2024-04-23T16:36:22.174 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7910000000000001e-06 2024-04-23T16:36:22.174 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.174 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.174 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.174 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:15.643549+0000", 2024-04-23T16:36:22.174 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5400000000000001e-06 2024-04-23T16:36:22.174 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.174 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.174 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.174 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:15.643560+0000", 2024-04-23T16:36:22.175 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1296e-05 2024-04-23T16:36:22.175 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.175 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.175 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.175 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:15.643568+0000", 2024-04-23T16:36:22.175 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3699999999999995e-06 2024-04-23T16:36:22.175 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.175 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.175 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.175 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:15.644620+0000", 2024-04-23T16:36:22.175 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0010519349999999999 2024-04-23T16:36:22.175 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.175 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.175 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.176 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:15.644644+0000", 2024-04-23T16:36:22.176 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4085999999999999e-05 2024-04-23T16:36:22.176 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.176 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.176 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.176 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:15.691203+0000", 2024-04-23T16:36:22.176 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.046558169000000003 2024-04-23T16:36:22.176 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.176 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.176 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.176 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:15.691249+0000", 2024-04-23T16:36:22.176 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6442e-05 2024-04-23T16:36:22.176 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.176 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.176 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.176 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.177 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.177 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:272611 5.0 5:0187d8e4:::benchmark_data_smithi188_113957_object119013:head [delete] snapc 0=[] ondisk+write+known_if_redirected e434)", 2024-04-23T16:36:22.177 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:36:16.444523+0000", 2024-04-23T16:36:22.177 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.6431605099999995, 2024-04-23T16:36:22.177 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.05046494, 2024-04-23T16:36:22.177 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.177 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.177 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.177 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:22.177 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:36:22.177 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:36:22.177 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 272611 2024-04-23T16:36:22.177 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.177 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.177 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.178 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.178 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:16.444523+0000", 2024-04-23T16:36:22.178 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.178 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.178 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.178 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.178 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:16.444523+0000", 2024-04-23T16:36:22.178 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.178 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.178 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.178 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.178 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:16.444525+0000", 2024-04-23T16:36:22.178 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6610000000000001e-06 2024-04-23T16:36:22.178 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.178 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.179 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.179 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:16.444529+0000", 2024-04-23T16:36:22.179 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4349999999999999e-06 2024-04-23T16:36:22.179 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.179 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.179 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.179 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:16.482339+0000", 2024-04-23T16:36:22.179 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037809857000000002 2024-04-23T16:36:22.179 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.179 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.179 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.179 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:16.482355+0000", 2024-04-23T16:36:22.179 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6087999999999999e-05 2024-04-23T16:36:22.179 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.179 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.179 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.180 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:16.482413+0000", 2024-04-23T16:36:22.180 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7908999999999997e-05 2024-04-23T16:36:22.180 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.180 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.180 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.180 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:16.482549+0000", 2024-04-23T16:36:22.180 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000135852 2024-04-23T16:36:22.180 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.180 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.180 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:36:22.180 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:16.482654+0000", 2024-04-23T16:36:22.180 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010572 2024-04-23T16:36:22.180 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.180 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.180 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:22.181 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:16.492813+0000", 2024-04-23T16:36:22.181 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010158670999999999 2024-04-23T16:36:22.181 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.181 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.181 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:22.181 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:16.494888+0000", 2024-04-23T16:36:22.181 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0020751260000000001 2024-04-23T16:36:22.181 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.181 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.181 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.181 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:16.494946+0000", 2024-04-23T16:36:22.181 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7414999999999998e-05 2024-04-23T16:36:22.181 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.181 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.181 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.182 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:16.494988+0000", 2024-04-23T16:36:22.182 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2206000000000002e-05 2024-04-23T16:36:22.182 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.182 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.182 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.182 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.182 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.182 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:273114 5.c 5:377237ec:::benchmark_data_smithi188_113957_object119516:head [delete] snapc 0=[] ondisk+write+known_if_redirected e435)", 2024-04-23T16:36:22.182 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:36:17.082464+0000", 2024-04-23T16:36:22.182 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.005219748, 2024-04-23T16:36:22.182 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.040477341, 2024-04-23T16:36:22.182 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.182 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.182 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.182 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:22.183 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:36:22.183 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:36:22.183 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 273114 2024-04-23T16:36:22.183 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.183 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.183 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.183 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.183 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:17.082464+0000", 2024-04-23T16:36:22.183 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.183 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.183 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.183 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.183 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:17.082464+0000", 2024-04-23T16:36:22.183 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.183 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.183 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.184 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.184 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:17.082464+0000", 2024-04-23T16:36:22.184 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8699999999999995e-07 2024-04-23T16:36:22.184 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.184 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.184 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.184 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:17.082467+0000", 2024-04-23T16:36:22.184 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6730000000000001e-06 2024-04-23T16:36:22.184 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.184 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.184 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.184 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:17.082472+0000", 2024-04-23T16:36:22.184 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6550000000000003e-06 2024-04-23T16:36:22.184 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.185 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.185 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.185 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:17.082477+0000", 2024-04-23T16:36:22.185 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9289999999999997e-06 2024-04-23T16:36:22.185 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.185 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.185 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.185 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:17.088324+0000", 2024-04-23T16:36:22.185 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0058475100000000002 2024-04-23T16:36:22.185 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.185 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.185 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.185 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:17.088441+0000", 2024-04-23T16:36:22.185 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000117094 2024-04-23T16:36:22.185 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.185 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.186 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:36:22.186 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:17.088555+0000", 2024-04-23T16:36:22.186 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000114289 2024-04-23T16:36:22.186 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.186 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.186 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:22.186 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:17.107926+0000", 2024-04-23T16:36:22.186 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019371064 2024-04-23T16:36:22.186 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.186 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.186 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:22.186 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:17.122873+0000", 2024-04-23T16:36:22.186 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014946743 2024-04-23T16:36:22.186 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.187 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.187 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.187 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:17.122910+0000", 2024-04-23T16:36:22.187 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7184000000000002e-05 2024-04-23T16:36:22.187 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.187 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.187 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.187 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:17.122941+0000", 2024-04-23T16:36:22.187 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0613e-05 2024-04-23T16:36:22.187 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.187 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.187 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.187 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.187 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.188 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:275710 5.1 5:819b8b20:::benchmark_data_smithi188_113957_object122112:head [delete] snapc 0=[] ondisk+write+known_if_redirected e436)", 2024-04-23T16:36:22.188 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:36:20.123149+0000", 2024-04-23T16:36:22.188 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 1.964534641, 2024-04-23T16:36:22.188 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.042103254, 2024-04-23T16:36:22.188 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.188 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.188 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.188 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:22.188 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:36:22.188 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:36:22.188 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 275710 2024-04-23T16:36:22.188 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.188 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.189 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.189 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.189 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.123149+0000", 2024-04-23T16:36:22.189 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.189 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.189 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.189 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.189 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.123149+0000", 2024-04-23T16:36:22.189 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.189 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.189 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.189 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.189 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.123151+0000", 2024-04-23T16:36:22.189 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.08e-06 2024-04-23T16:36:22.190 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.190 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.190 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.190 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.123155+0000", 2024-04-23T16:36:22.190 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0740000000000003e-06 2024-04-23T16:36:22.190 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.190 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.190 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.190 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.123163+0000", 2024-04-23T16:36:22.190 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8120000000000008e-06 2024-04-23T16:36:22.190 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.190 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.190 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.191 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.123170+0000", 2024-04-23T16:36:22.191 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7889999999999997e-06 2024-04-23T16:36:22.191 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.191 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.191 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.191 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.134656+0000", 2024-04-23T16:36:22.191 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011486814999999999 2024-04-23T16:36:22.191 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.191 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.191 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.191 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.134790+0000", 2024-04-23T16:36:22.191 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000133337 2024-04-23T16:36:22.191 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.191 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.191 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:36:22.192 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.134912+0000", 2024-04-23T16:36:22.192 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012249599999999999 2024-04-23T16:36:22.192 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.192 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.192 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:22.192 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.148065+0000", 2024-04-23T16:36:22.192 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013152508 2024-04-23T16:36:22.192 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.192 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.192 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:22.192 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.165162+0000", 2024-04-23T16:36:22.192 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.017097668999999999 2024-04-23T16:36:22.192 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.192 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.193 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.193 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.165210+0000", 2024-04-23T16:36:22.193 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7484000000000001e-05 2024-04-23T16:36:22.193 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.193 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.193 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.193 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.165252+0000", 2024-04-23T16:36:22.193 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2190000000000001e-05 2024-04-23T16:36:22.193 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.193 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.193 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.193 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.193 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.193 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:276434 5.a e436/394 5:5a8be68d:::benchmark_data_smithi188_113957_object122836:head v 436'17505, mlcod=380'7786)", 2024-04-23T16:36:22.193 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:36:20.960584+0000", 2024-04-23T16:36:22.194 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 1.1270996389999999, 2024-04-23T16:36:22.194 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.045813908, 2024-04-23T16:36:22.194 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.194 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.194 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.194 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.194 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.194 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.194 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.960584+0000", 2024-04-23T16:36:22.194 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.194 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.194 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.194 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.194 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.960584+0000", 2024-04-23T16:36:22.194 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.195 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.195 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.195 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.195 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.960584+0000", 2024-04-23T16:36:22.195 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5499999999999998e-07 2024-04-23T16:36:22.195 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.195 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.195 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.195 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.960589+0000", 2024-04-23T16:36:22.195 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4789999999999998e-06 2024-04-23T16:36:22.195 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.195 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.195 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.195 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.960594+0000", 2024-04-23T16:36:22.195 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.451e-06 2024-04-23T16:36:22.196 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.196 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.196 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.196 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.960600+0000", 2024-04-23T16:36:22.196 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6699999999999999e-06 2024-04-23T16:36:22.196 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.196 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.196 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.196 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.970165+0000", 2024-04-23T16:36:22.196 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0095645629999999999 2024-04-23T16:36:22.196 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.196 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.196 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.196 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.970190+0000", 2024-04-23T16:36:22.196 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5499000000000001e-05 2024-04-23T16:36:22.196 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.197 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.197 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.197 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:21.006348+0000", 2024-04-23T16:36:22.197 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.036158388999999999 2024-04-23T16:36:22.197 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.197 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.197 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.197 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:21.006398+0000", 2024-04-23T16:36:22.197 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9202000000000001e-05 2024-04-23T16:36:22.197 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.197 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.197 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.197 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.198 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.198 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:276437 5.a e436/394 5:50a6b962:::benchmark_data_smithi188_113957_object122839:head v 436'17506, mlcod=380'7786)", 2024-04-23T16:36:22.198 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:36:20.965114+0000", 2024-04-23T16:36:22.198 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 1.1225689750000001, 2024-04-23T16:36:22.198 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.041311356, 2024-04-23T16:36:22.198 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.198 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.198 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.198 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.198 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.198 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.198 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.965114+0000", 2024-04-23T16:36:22.198 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.199 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.199 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.199 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.199 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.965114+0000", 2024-04-23T16:36:22.199 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.199 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.199 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.199 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.199 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.965117+0000", 2024-04-23T16:36:22.199 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4830000000000002e-06 2024-04-23T16:36:22.199 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.199 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.199 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.199 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.965127+0000", 2024-04-23T16:36:22.200 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0528e-05 2024-04-23T16:36:22.200 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.200 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.200 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.200 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.965140+0000", 2024-04-23T16:36:22.200 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2479e-05 2024-04-23T16:36:22.200 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.200 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.200 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.200 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.965149+0000", 2024-04-23T16:36:22.200 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6179999999999995e-06 2024-04-23T16:36:22.200 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.200 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.200 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.200 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.976269+0000", 2024-04-23T16:36:22.200 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011120543 2024-04-23T16:36:22.201 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.201 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.201 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.201 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.976290+0000", 2024-04-23T16:36:22.201 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0871999999999999e-05 2024-04-23T16:36:22.201 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.201 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.201 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.201 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:21.006401+0000", 2024-04-23T16:36:22.201 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.030110906999999999 2024-04-23T16:36:22.201 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.201 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.201 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.201 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:21.006426+0000", 2024-04-23T16:36:22.201 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4926000000000002e-05 2024-04-23T16:36:22.201 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.202 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.202 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.202 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.202 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.202 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:276449 5.a e436/394 5:5578887d:::benchmark_data_smithi188_113957_object122851:head v 436'17507, mlcod=380'7786)", 2024-04-23T16:36:22.202 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:36:20.965167+0000", 2024-04-23T16:36:22.202 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 1.1225165939999999, 2024-04-23T16:36:22.202 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.041291785999999997, 2024-04-23T16:36:22.202 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:22.202 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:22.202 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:22.202 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:22.203 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.203 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:22.203 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.965167+0000", 2024-04-23T16:36:22.203 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.203 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.203 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.203 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:22.203 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.965167+0000", 2024-04-23T16:36:22.203 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:22.203 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.203 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.203 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:22.203 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.965167+0000", 2024-04-23T16:36:22.203 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6699999999999999e-07 2024-04-23T16:36:22.203 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.203 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.204 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:22.204 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.965172+0000", 2024-04-23T16:36:22.204 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7260000000000002e-06 2024-04-23T16:36:22.204 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.204 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.204 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:22.204 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.965177+0000", 2024-04-23T16:36:22.204 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0839999999999999e-06 2024-04-23T16:36:22.204 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.204 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.204 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:22.204 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.965181+0000", 2024-04-23T16:36:22.204 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1829999999999996e-06 2024-04-23T16:36:22.204 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.204 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.205 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:22.205 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.982497+0000", 2024-04-23T16:36:22.205 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.017315898999999999 2024-04-23T16:36:22.205 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.205 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.205 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:22.205 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:20.982534+0000", 2024-04-23T16:36:22.205 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6866000000000002e-05 2024-04-23T16:36:22.205 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.205 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.205 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:22.205 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:21.006431+0000", 2024-04-23T16:36:22.205 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023896661 2024-04-23T16:36:22.205 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:22.205 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:22.206 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:22.206 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:21.006459+0000", 2024-04-23T16:36:22.206 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7900000000000001e-05 2024-04-23T16:36:22.206 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.206 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.206 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.206 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:22.206 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:22.206 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:36:22.207 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:21 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:36:22.207 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:21 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:36:22.207 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:21 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:36:22.207 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:21 smithi103 ceph-mon[39629]: 5.5 scrub ok 2024-04-23T16:36:22.207 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:21 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:36:22.207 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:21 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:36:22.208 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:21 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:36:22.208 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:21 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:36:22.208 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:21 smithi148 ceph-mon[44520]: 5.5 scrub ok 2024-04-23T16:36:22.208 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:21 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:36:22.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:22.208 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:36:22.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:36:22.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:21 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:36:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:21 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:36:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:21 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:36:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:21 smithi188 ceph-mon[44101]: 5.5 scrub ok 2024-04-23T16:36:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:21 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:36:22.310 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:36:22.343 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:22.367 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:36:22.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:22.371 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:36:22.472 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:36:22.501 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:22.526 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:36:22.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:22.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:36:22.630 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:36:22.698 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:22.755 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:36:22.755 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5961/64476 objects degraded (9.245%), 3 pgs degraded","count":3},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1564,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":436,"num_osds":12,"num_up_osds":11,"osd_up_since":1713889979,"num_in_osds":9,"osd_in_since":1713890004,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":24},{"state_name":"active+recovering+degraded","count":3}],"num_pgs":27,"num_pools":2,"num_objects":32238,"data_bytes":2113143352,"bytes_used":7607521280,"bytes_avail":856347799552,"bytes_total":863955320832,"degraded_objects":5961,"degraded_total":64476,"degraded_ratio":0.092453005769588689,"recovering_objects_per_sec":195,"recovering_bytes_per_sec":12820465,"recovering_keys_per_sec":0,"num_objects_recovered":1284,"num_bytes_recovered":84148224,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1064},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.103:8443/"}},"servicemap":{"epoch":34,"modified":"2024-04-23T16:35:30.733564+0000","services":{}},"progress_events":{}} 2024-04-23T16:36:22.783 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:36:22.798 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:36:22.826 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:36:22.848 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:36:22.880 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:36:22.916 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:36:22.949 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:36:22.984 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:36:22.991 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:36:22.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:22.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:36:23.009 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:36:23.030 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:36:23.054 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:36:23.079 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:36:23.080 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:36:23.080 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:36:23.080 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:36:23.080 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:36:23.080 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:36:23.080 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:36:23.080 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:36:23.080 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:36:23.080 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:36:23.080 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:36:23.080 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:36:23.080 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:36:23.098 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:36:23.128 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:23.156 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:36:23.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:23.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:36:23.260 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:36:23.296 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:23.332 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:36:23.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:23.339 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:36:23.441 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:36:23.471 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:23.491 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:23.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:23.495 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:23.596 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:36:23.629 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:23.712 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:36:23.712 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:36:23.712 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:36:23.713 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:36:23.713 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:36:23.713 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:23 smithi103 ceph-mon[39629]: pgmap v1240: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.9 GiB data, 7.0 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 750 op/s; 5961/63398 objects degraded (9.403%); 10 MiB/s, 160 objects/s recovering 2024-04-23T16:36:23.713 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:23 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3974816490' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:36:23.713 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:23 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1481870886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:23.713 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:23.713 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:23.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:36:23.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:36:23.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:36:23.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:36:23.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:36:23.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:23 smithi188 ceph-mon[44101]: pgmap v1240: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.9 GiB data, 7.0 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 750 op/s; 5961/63398 objects degraded (9.403%); 10 MiB/s, 160 objects/s recovering 2024-04-23T16:36:23.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:23 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3974816490' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:36:23.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:23 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1481870886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:23.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:23.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:23.844 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:36:23.844 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:36:23.844 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:36:23.844 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:36:23.844 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:36:23.844 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:23 smithi148 ceph-mon[44520]: pgmap v1240: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.9 GiB data, 7.0 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 750 op/s; 5961/63398 objects degraded (9.403%); 10 MiB/s, 160 objects/s recovering 2024-04-23T16:36:23.844 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:23 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3974816490' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:36:23.844 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:23 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1481870886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:23.844 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:23.845 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.223 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:36:24.224 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:36:24.302 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-23T16:36:24.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:24 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:36:24.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:24 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:36:24.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:24 smithi103 ceph-mon[39629]: 5.3 scrub starts 2024-04-23T16:36:24.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:24 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 5961/63398 objects degraded (9.403%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:24.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:24 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:36:24.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:24 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:36:24.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:24 smithi148 ceph-mon[44520]: 5.3 scrub starts 2024-04-23T16:36:24.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:24 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 5961/63398 objects degraded (9.403%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:24 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:36:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:24 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:36:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:24 smithi188 ceph-mon[44101]: 5.3 scrub starts 2024-04-23T16:36:24.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:24 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 5961/63398 objects degraded (9.403%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:24.780 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:36:24.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:24.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:36:24.862 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:36:24.883 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:36:24.967 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:25.034 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:36:25.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:25.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:36:25.145 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:36:25.179 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:25.199 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:36:25.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:25.203 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:36:25.304 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:25.362 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:25.400 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-23T16:36:25.400 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:36:25.462 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:25.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:25.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:25 smithi188 ceph-mon[44101]: 5.3 scrub ok 2024-04-23T16:36:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:25 smithi188 ceph-mon[44101]: 5.d scrub starts 2024-04-23T16:36:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:25 smithi188 ceph-mon[44101]: 5.d scrub ok 2024-04-23T16:36:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:25 smithi188 ceph-mon[44101]: pgmap v1241: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.8 GiB data, 6.7 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 890 op/s; 5685/58538 objects degraded (9.712%); 8.2 MiB/s, 130 objects/s recovering 2024-04-23T16:36:25.568 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:36:25.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:25 smithi148 ceph-mon[44520]: 5.3 scrub ok 2024-04-23T16:36:25.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:25 smithi148 ceph-mon[44520]: 5.d scrub starts 2024-04-23T16:36:25.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:25 smithi148 ceph-mon[44520]: 5.d scrub ok 2024-04-23T16:36:25.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:25 smithi148 ceph-mon[44520]: pgmap v1241: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.8 GiB data, 6.7 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 890 op/s; 5685/58538 objects degraded (9.712%); 8.2 MiB/s, 130 objects/s recovering 2024-04-23T16:36:25.652 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:25 smithi103 ceph-mon[39629]: 5.3 scrub ok 2024-04-23T16:36:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:25 smithi103 ceph-mon[39629]: 5.d scrub starts 2024-04-23T16:36:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:25 smithi103 ceph-mon[39629]: 5.d scrub ok 2024-04-23T16:36:25.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:25 smithi103 ceph-mon[39629]: pgmap v1241: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.8 GiB data, 6.7 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 890 op/s; 5685/58538 objects degraded (9.712%); 8.2 MiB/s, 130 objects/s recovering 2024-04-23T16:36:25.876 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:36:25.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:25.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:36:25.982 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:36:26.013 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:26.225 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:36:26.267 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:36:26.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:26.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:36:26.372 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:26.452 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:26.481 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:26.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:26.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:26.591 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:26.640 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:36:26.650 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:26.671 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:36:26.729 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:26.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:26.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:26.843 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:36:26.874 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:26.900 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:26.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:26.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:26.940 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:26 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:36:26.940 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:26 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:36:26.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:26 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:36:26.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:26 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:36:27.004 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:36:27.040 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:27.068 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:36:27.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:27.073 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:36:27.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:26 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:36:27.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:26 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:36:27.144 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:36:27.144 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:36:27.173 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:36:27.234 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:27.706 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:36:27.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:27 smithi188 ceph-mon[44101]: pgmap v1242: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.7 GiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 847 op/s; 5146/55890 objects degraded (9.207%); 9.9 MiB/s, 159 objects/s recovering 2024-04-23T16:36:27.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:27 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1623110324' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:36:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:27 smithi103 ceph-mon[39629]: pgmap v1242: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.7 GiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 847 op/s; 5146/55890 objects degraded (9.207%); 9.9 MiB/s, 159 objects/s recovering 2024-04-23T16:36:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:27 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1623110324' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:36:28.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:27 smithi148 ceph-mon[44520]: pgmap v1242: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.7 GiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 847 op/s; 5146/55890 objects degraded (9.207%); 9.9 MiB/s, 159 objects/s recovering 2024-04-23T16:36:28.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:27 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1623110324' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:36:28.529 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:36:28.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:28.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:36:28.638 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:36:28.677 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:28.701 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:36:28.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:28.708 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:36:28.809 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:36:28.840 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:28.871 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:36:28.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:28 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:36:28.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:28 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1623110324' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:36:28.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:28 smithi148 ceph-mon[44520]: osdmap e437: 12 total, 11 up, 9 in 2024-04-23T16:36:28.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:28.877 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:36:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:28 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:36:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:28 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1623110324' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:36:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:28 smithi188 ceph-mon[44101]: osdmap e437: 12 total, 11 up, 9 in 2024-04-23T16:36:28.978 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:36:29.009 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:29.034 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:36:29.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:29.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:36:29.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:28 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:36:29.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:28 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1623110324' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:36:29.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:28 smithi103 ceph-mon[39629]: osdmap e437: 12 total, 11 up, 9 in 2024-04-23T16:36:29.140 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:36:29.170 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:29.522 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:36:29.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:29.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:36:29.626 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:36:29.655 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:29.799 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:29 smithi103 ceph-mon[39629]: pgmap v1244: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.7 GiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 835 op/s; 4877/54244 objects degraded (8.991%); 10 MiB/s, 167 objects/s recovering 2024-04-23T16:36:29.800 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:29 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 5146/55890 objects degraded (9.207%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:29.800 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:29 smithi103 ceph-mon[39629]: osdmap e438: 12 total, 11 up, 9 in 2024-04-23T16:36:29.800 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:29 smithi103 ceph-mon[39629]: 1.4 scrub starts 2024-04-23T16:36:29.800 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:29 smithi103 ceph-mon[39629]: 1.4 scrub ok 2024-04-23T16:36:29.809 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:36:29.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:29.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:36:29.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:29 smithi148 ceph-mon[44520]: pgmap v1244: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.7 GiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 835 op/s; 4877/54244 objects degraded (8.991%); 10 MiB/s, 167 objects/s recovering 2024-04-23T16:36:29.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:29 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 5146/55890 objects degraded (9.207%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:29.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:29 smithi148 ceph-mon[44520]: osdmap e438: 12 total, 11 up, 9 in 2024-04-23T16:36:29.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:29 smithi148 ceph-mon[44520]: 1.4 scrub starts 2024-04-23T16:36:29.878 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:29 smithi148 ceph-mon[44520]: 1.4 scrub ok 2024-04-23T16:36:29.914 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:36:29.953 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-23T16:36:29.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:29 smithi188 ceph-mon[44101]: pgmap v1244: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.7 GiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 835 op/s; 4877/54244 objects degraded (8.991%); 10 MiB/s, 167 objects/s recovering 2024-04-23T16:36:29.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:29 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 5146/55890 objects degraded (9.207%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:29 smithi188 ceph-mon[44101]: osdmap e438: 12 total, 11 up, 9 in 2024-04-23T16:36:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:29 smithi188 ceph-mon[44101]: 1.4 scrub starts 2024-04-23T16:36:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:29 smithi188 ceph-mon[44101]: 1.4 scrub ok 2024-04-23T16:36:29.996 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:30.035 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:36:30.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:30.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:36:30.145 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:36:30.248 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:30.253 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:36:30.274 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:36:30.286 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:36:30.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:30.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:36:30.294 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:36:30.328 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:36:30.367 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:36:30.392 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:36:30.395 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:36:30.426 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:30.431 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:36:30.453 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:36:30.473 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:36:30.481 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:36:30.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:30.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:36:30.502 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:36:30.508 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:36:30.529 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:36:30.529 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:36:30.530 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:36:30.530 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:36:30.530 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:36:30.530 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:36:30.530 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:36:30.530 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:36:30.530 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:36:30.530 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:36:30.530 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:36:30.530 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:36:30.530 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:36:30.588 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:36:30.651 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:30.697 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:36:30.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:30.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:36:30.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:30 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2275537066' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:30.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:30.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:30.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:30.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:30.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:30.804 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:36:30.835 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:30.914 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:30 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2275537066' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:30.915 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:30.915 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:30.915 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:30.915 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:30.915 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:30.977 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:36:30.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:30.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:36:31.080 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:36:31.109 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:30 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2275537066' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:31.309 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:36:31.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:31.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:36:31.414 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:36:31.443 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:31.463 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:36:31.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:31.467 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:36:31.568 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:31.617 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:31.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:31.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:31.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:31.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:31.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:31 smithi148 ceph-mon[44520]: pgmap v1246: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.4 GiB data, 6.2 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 999 op/s; 4335/47410 objects degraded (9.144%); 13 MiB/s, 203 objects/s recovering 2024-04-23T16:36:31.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:31 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:36:31.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:31 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:36:31.929 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:36:31.930 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:36:31.984 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:31.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:31.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:32.088 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:36:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:31 smithi103 ceph-mon[39629]: pgmap v1246: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.4 GiB data, 6.2 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 999 op/s; 4335/47410 objects degraded (9.144%); 13 MiB/s, 203 objects/s recovering 2024-04-23T16:36:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:31 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:36:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:31 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:36:32.118 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:31 smithi188 ceph-mon[44101]: pgmap v1246: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.4 GiB data, 6.2 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 999 op/s; 4335/47410 objects degraded (9.144%); 13 MiB/s, 203 objects/s recovering 2024-04-23T16:36:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:31 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:36:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:31 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:36:32.243 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:36:32.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:32.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:36:32.347 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:36:32.375 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:32.395 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:36:32.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:32.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:36:32.476 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:36:32.479 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:36:32.500 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:36:32.558 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:32.587 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:36:32.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:32.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:36:32.695 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:36:32.726 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:32.755 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:36:32.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:32.759 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:36:32.780 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:36:32.781 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:36:32.781 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2024-04-23T16:36:32.781 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2024-04-23T16:36:32.781 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:36:32.859 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:36:32.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:32 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:36:32.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:32 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:36:32.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:32 smithi148 ceph-mon[44520]: osdmap e439: 12 total, 11 up, 9 in 2024-04-23T16:36:32.887 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:32.909 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:36:32.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:32.912 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:36:33.012 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:36:33.013 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:36:33.040 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:32 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:36:33.040 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:32 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:36:33.040 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:32 smithi103 ceph-mon[39629]: osdmap e439: 12 total, 11 up, 9 in 2024-04-23T16:36:33.102 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:32 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:36:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:32 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:36:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:32 smithi188 ceph-mon[44101]: osdmap e439: 12 total, 11 up, 9 in 2024-04-23T16:36:33.311 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1247,"stamp":"2024-04-23T16:36:31.732950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1553924664,"num_objects":23705,"num_object_clones":0,"num_object_copies":47410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23705,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":283489,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24618,"num_bytes_recovered":1613500960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162549,"ondisk_log_size":162549,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":6503684,"kb_used_data":2821540,"kb_used_omap":14,"kb_used_meta":3681777,"kb_avail":837202684,"statfs":{"total":863955320832,"available":857295548416,"internally_reserved":0,"allocated":2889256960,"data_stored":2885110238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3770140655},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-364642304,"num_objects":-5564,"num_object_clones":0,"num_object_copies":-11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1350,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5564,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1350,"num_bytes_recovered":88473600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.989219"},"pg_stats":[{"pgid":"5.f","version":"438'17947","reported_seq":18219,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-04-23T16:36:30.239688+0000","last_change":"2024-04-23T16:33:17.526419+0000","last_active":"2024-04-23T16:36:30.239688+0000","last_peered":"2024-04-23T16:36:30.239688+0000","last_clean":"2024-04-23T16:36:30.239688+0000","last_became_active":"2024-04-23T16:31:29.140923+0000","last_became_peered":"2024-04-23T16:31:29.140923+0000","last_unstale":"2024-04-23T16:36:30.239688+0000","last_undegraded":"2024-04-23T16:36:30.239688+0000","last_fullsized":"2024-04-23T16:36:30.239688+0000","mapping_epoch":359,"log_start":"380'7900","ondisk_log_start":"380'7900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9683","last_scrub_stamp":"2024-04-23T16:33:17.526283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.526283+0000","objects_scrubbed":9683,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92995584,"num_objects":1419,"num_object_clones":0,"num_object_copies":2838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17947,"num_write_kb":619712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"438'17787","reported_seq":18185,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-04-23T16:36:29.314344+0000","last_change":"2024-04-23T16:35:26.407358+0000","last_active":"2024-04-23T16:36:29.314344+0000","last_peered":"2024-04-23T16:36:29.314344+0000","last_clean":"2024-04-23T16:36:29.314344+0000","last_became_active":"2024-04-23T16:31:29.137890+0000","last_became_peered":"2024-04-23T16:31:29.137890+0000","last_unstale":"2024-04-23T16:36:29.314344+0000","last_undegraded":"2024-04-23T16:36:29.314344+0000","last_fullsized":"2024-04-23T16:36:29.314344+0000","mapping_epoch":359,"log_start":"380'7700","ondisk_log_start":"380'7700","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'14582","last_scrub_stamp":"2024-04-23T16:35:26.407093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:26.407093+0000","objects_scrubbed":4692,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:38.641004+0000","scrub_duration":544,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96796672,"num_objects":1477,"num_object_clones":0,"num_object_copies":2954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17787,"num_write_kb":616448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"437'17898","reported_seq":34521,"reported_epoch":437,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:36:28.216041+0000","last_change":"2024-04-23T16:33:26.432192+0000","last_active":"2024-04-23T16:36:28.216041+0000","last_peered":"2024-04-23T16:36:28.216041+0000","last_clean":"2024-04-23T16:32:59.749530+0000","last_became_active":"2024-04-23T16:33:26.338644+0000","last_became_peered":"2024-04-23T16:33:26.338644+0000","last_unstale":"2024-04-23T16:36:28.216041+0000","last_undegraded":"2024-04-23T16:33:25.874847+0000","last_fullsized":"2024-04-23T16:36:28.216041+0000","mapping_epoch":394,"log_start":"380'7800","ondisk_log_start":"380'7800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8711","last_scrub_stamp":"2024-04-23T16:32:43.928914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.928914+0000","objects_scrubbed":8680,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101187584,"num_objects":1544,"num_object_clones":0,"num_object_copies":3088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17898,"num_write_kb":622144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8178,"num_bytes_recovered":535953408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":["8"],"object_location_counts":[{"shards":"4,8","objects":10},{"shards":"8","objects":1534}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.9","version":"438'17880","reported_seq":18137,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-04-23T16:36:30.173238+0000","last_change":"2024-04-23T16:32:37.256944+0000","last_active":"2024-04-23T16:36:30.173238+0000","last_peered":"2024-04-23T16:36:30.173238+0000","last_clean":"2024-04-23T16:36:30.173238+0000","last_became_active":"2024-04-23T16:31:29.141551+0000","last_became_peered":"2024-04-23T16:31:29.141551+0000","last_unstale":"2024-04-23T16:36:30.173238+0000","last_undegraded":"2024-04-23T16:36:30.173238+0000","last_fullsized":"2024-04-23T16:36:30.173238+0000","mapping_epoch":359,"log_start":"380'7800","ondisk_log_start":"380'7800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8156","last_scrub_stamp":"2024-04-23T16:32:37.256787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:37.256787+0000","objects_scrubbed":8082,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94371840,"num_objects":1440,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17880,"num_write_kb":618240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"436'17147","reported_seq":17512,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-04-23T16:36:26.788663+0000","last_change":"2024-04-23T16:36:20.471641+0000","last_active":"2024-04-23T16:36:26.788663+0000","last_peered":"2024-04-23T16:36:26.788663+0000","last_clean":"2024-04-23T16:36:26.788663+0000","last_became_active":"2024-04-23T16:31:29.139045+0000","last_became_peered":"2024-04-23T16:31:29.139045+0000","last_unstale":"2024-04-23T16:36:26.788663+0000","last_undegraded":"2024-04-23T16:36:26.788663+0000","last_fullsized":"2024-04-23T16:36:26.788663+0000","mapping_epoch":359,"log_start":"379'7100","ondisk_log_start":"379'7100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'16781","last_scrub_stamp":"2024-04-23T16:36:20.471467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:36:20.471467+0000","objects_scrubbed":1958,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:48.916510+0000","scrub_duration":248,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104005632,"num_objects":1587,"num_object_clones":0,"num_object_copies":3174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":599488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"438'21295","reported_seq":21764,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-04-23T16:36:29.296384+0000","last_change":"2024-04-23T16:35:32.474057+0000","last_active":"2024-04-23T16:36:29.296384+0000","last_peered":"2024-04-23T16:36:29.296384+0000","last_clean":"2024-04-23T16:36:29.296384+0000","last_became_active":"2024-04-23T16:31:29.310906+0000","last_became_peered":"2024-04-23T16:31:29.310906+0000","last_unstale":"2024-04-23T16:36:29.296384+0000","last_undegraded":"2024-04-23T16:36:29.296384+0000","last_fullsized":"2024-04-23T16:36:29.296384+0000","mapping_epoch":359,"log_start":"405'11204","ondisk_log_start":"405'11204","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'18441","last_scrub_stamp":"2024-04-23T16:35:32.473828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:32.473828+0000","objects_scrubbed":4251,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:57.506974+0000","scrub_duration":548,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90832896,"num_objects":1386,"num_object_clones":0,"num_object_copies":2772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17734,"num_write_kb":611840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1146,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-04-23T16:36:29.184160+0000","last_change":"2024-04-23T16:36:19.132015+0000","last_active":"2024-04-23T16:36:29.184160+0000","last_peered":"2024-04-23T16:36:29.184160+0000","last_clean":"2024-04-23T16:36:29.184160+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:36:29.184160+0000","last_undegraded":"2024-04-23T16:36:29.184160+0000","last_fullsized":"2024-04-23T16:36:29.184160+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:19.131883+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:19.131883+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:41.022603+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"438'18012","reported_seq":18270,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-04-23T16:36:29.583174+0000","last_change":"2024-04-23T16:32:38.103756+0000","last_active":"2024-04-23T16:36:29.583174+0000","last_peered":"2024-04-23T16:36:29.583174+0000","last_clean":"2024-04-23T16:36:29.583174+0000","last_became_active":"2024-04-23T16:31:29.141640+0000","last_became_peered":"2024-04-23T16:31:29.141640+0000","last_unstale":"2024-04-23T16:36:29.583174+0000","last_undegraded":"2024-04-23T16:36:29.583174+0000","last_fullsized":"2024-04-23T16:36:29.583174+0000","mapping_epoch":359,"log_start":"380'8000","ondisk_log_start":"380'8000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8300","last_scrub_stamp":"2024-04-23T16:32:38.103547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:38.103547+0000","objects_scrubbed":8231,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94633984,"num_objects":1444,"num_object_clones":0,"num_object_copies":2888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18012,"num_write_kb":622592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1132,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-04-23T16:36:29.202126+0000","last_change":"2024-04-23T16:33:25.851379+0000","last_active":"2024-04-23T16:36:29.202126+0000","last_peered":"2024-04-23T16:36:29.202126+0000","last_clean":"2024-04-23T16:36:29.202126+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:36:29.202126+0000","last_undegraded":"2024-04-23T16:36:29.202126+0000","last_fullsized":"2024-04-23T16:36:29.202126+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.498820+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.498820+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"436'17484","reported_seq":17852,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-04-23T16:36:26.766191+0000","last_change":"2024-04-23T16:36:24.521917+0000","last_active":"2024-04-23T16:36:26.766191+0000","last_peered":"2024-04-23T16:36:26.766191+0000","last_clean":"2024-04-23T16:36:26.766191+0000","last_became_active":"2024-04-23T16:31:29.140251+0000","last_became_peered":"2024-04-23T16:31:29.140251+0000","last_unstale":"2024-04-23T16:36:26.766191+0000","last_undegraded":"2024-04-23T16:36:26.766191+0000","last_fullsized":"2024-04-23T16:36:26.766191+0000","mapping_epoch":359,"log_start":"380'7400","ondisk_log_start":"380'7400","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'17367","last_scrub_stamp":"2024-04-23T16:36:24.521728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:36:24.521728+0000","objects_scrubbed":1767,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:41.002308+0000","scrub_duration":288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107937816,"num_objects":1648,"num_object_clones":0,"num_object_copies":3296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17484,"num_write_kb":612161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1123,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-04-23T16:36:29.193725+0000","last_change":"2024-04-23T16:32:29.517947+0000","last_active":"2024-04-23T16:36:29.193725+0000","last_peered":"2024-04-23T16:36:29.193725+0000","last_clean":"2024-04-23T16:36:29.193725+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:36:29.193725+0000","last_undegraded":"2024-04-23T16:36:29.193725+0000","last_fullsized":"2024-04-23T16:36:29.193725+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:32:29.517800+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:32:29.517800+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"438'17913","reported_seq":34636,"reported_epoch":438,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:36:30.261270+0000","last_change":"2024-04-23T16:33:26.507399+0000","last_active":"2024-04-23T16:36:30.261270+0000","last_peered":"2024-04-23T16:36:30.261270+0000","last_clean":"2024-04-23T16:32:59.715575+0000","last_became_active":"2024-04-23T16:33:26.458611+0000","last_became_peered":"2024-04-23T16:33:26.458611+0000","last_unstale":"2024-04-23T16:36:30.261270+0000","last_undegraded":"2024-04-23T16:33:25.895472+0000","last_fullsized":"2024-04-23T16:36:30.261270+0000","mapping_epoch":394,"log_start":"380'7900","ondisk_log_start":"380'7900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8607","last_scrub_stamp":"2024-04-23T16:32:43.130806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.130806+0000","objects_scrubbed":8577,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96141312,"num_objects":1467,"num_object_clones":0,"num_object_copies":2934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17913,"num_write_kb":620160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8228,"num_bytes_recovered":539230208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1459},{"shards":"5,7","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1143,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-04-23T16:36:27.711249+0000","last_change":"2024-04-23T16:36:26.456131+0000","last_active":"2024-04-23T16:36:27.711249+0000","last_peered":"2024-04-23T16:36:27.711249+0000","last_clean":"2024-04-23T16:36:27.711249+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:36:27.711249+0000","last_undegraded":"2024-04-23T16:36:27.711249+0000","last_fullsized":"2024-04-23T16:36:27.711249+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:26.456003+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:26.456003+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:44.948237+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"438'17528","reported_seq":17798,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-04-23T16:36:30.259487+0000","last_change":"2024-04-23T16:33:18.576077+0000","last_active":"2024-04-23T16:36:30.259487+0000","last_peered":"2024-04-23T16:36:30.259487+0000","last_clean":"2024-04-23T16:36:30.259487+0000","last_became_active":"2024-04-23T16:31:29.139709+0000","last_became_peered":"2024-04-23T16:31:29.139709+0000","last_unstale":"2024-04-23T16:36:30.259487+0000","last_undegraded":"2024-04-23T16:36:30.259487+0000","last_fullsized":"2024-04-23T16:36:30.259487+0000","mapping_epoch":359,"log_start":"380'7500","ondisk_log_start":"380'7500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9472","last_scrub_stamp":"2024-04-23T16:33:18.575945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.575945+0000","objects_scrubbed":9472,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92798976,"num_objects":1416,"num_object_clones":0,"num_object_copies":2832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17528,"num_write_kb":606208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1138,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-04-23T16:36:29.185225+0000","last_change":"2024-04-23T16:35:34.878630+0000","last_active":"2024-04-23T16:36:29.185225+0000","last_peered":"2024-04-23T16:36:29.185225+0000","last_clean":"2024-04-23T16:36:29.185225+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:36:29.185225+0000","last_undegraded":"2024-04-23T16:36:29.185225+0000","last_fullsized":"2024-04-23T16:36:29.185225+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:35:34.878536+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:35:34.878536+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:52.503403+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"437'17889","reported_seq":18264,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-04-23T16:36:29.119149+0000","last_change":"2024-04-23T16:36:20.849660+0000","last_active":"2024-04-23T16:36:29.119149+0000","last_peered":"2024-04-23T16:36:29.119149+0000","last_clean":"2024-04-23T16:36:29.119149+0000","last_became_active":"2024-04-23T16:31:29.139606+0000","last_became_peered":"2024-04-23T16:31:29.139606+0000","last_unstale":"2024-04-23T16:36:29.119149+0000","last_undegraded":"2024-04-23T16:36:29.119149+0000","last_fullsized":"2024-04-23T16:36:29.119149+0000","mapping_epoch":359,"log_start":"380'7800","ondisk_log_start":"380'7800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'17441","last_scrub_stamp":"2024-04-23T16:36:20.849516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:36:20.849516+0000","objects_scrubbed":2009,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:50.602166+0000","scrub_duration":400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101646336,"num_objects":1551,"num_object_clones":0,"num_object_copies":3102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17889,"num_write_kb":622080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1145,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-04-23T16:36:29.184104+0000","last_change":"2024-04-23T16:36:23.239567+0000","last_active":"2024-04-23T16:36:29.184104+0000","last_peered":"2024-04-23T16:36:29.184104+0000","last_clean":"2024-04-23T16:36:29.184104+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:36:29.184104+0000","last_undegraded":"2024-04-23T16:36:29.184104+0000","last_fullsized":"2024-04-23T16:36:29.184104+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:23.239466+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:36:23.239466+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:27.031497+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.2","version":"438'17572","reported_seq":17971,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-04-23T16:36:29.653509+0000","last_change":"2024-04-23T16:35:32.569280+0000","last_active":"2024-04-23T16:36:29.653509+0000","last_peered":"2024-04-23T16:36:29.653509+0000","last_clean":"2024-04-23T16:36:29.653509+0000","last_became_active":"2024-04-23T16:31:29.138430+0000","last_became_peered":"2024-04-23T16:31:29.138430+0000","last_unstale":"2024-04-23T16:36:29.653509+0000","last_undegraded":"2024-04-23T16:36:29.653509+0000","last_fullsized":"2024-04-23T16:36:29.653509+0000","mapping_epoch":359,"log_start":"380'7500","ondisk_log_start":"380'7500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'14723","last_scrub_stamp":"2024-04-23T16:35:32.569137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:32.569137+0000","objects_scrubbed":4294,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:34.926251+0000","scrub_duration":850,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93323264,"num_objects":1424,"num_object_clones":0,"num_object_copies":2848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17572,"num_write_kb":607872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1146,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-04-23T16:36:29.187384+0000","last_change":"2024-04-23T16:36:18.085504+0000","last_active":"2024-04-23T16:36:29.187384+0000","last_peered":"2024-04-23T16:36:29.187384+0000","last_clean":"2024-04-23T16:36:29.187384+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:36:29.187384+0000","last_undegraded":"2024-04-23T16:36:29.187384+0000","last_fullsized":"2024-04-23T16:36:29.187384+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:18.085386+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:18.085386+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:25.354657+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"437'19896","reported_seq":20395,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-04-23T16:36:29.147326+0000","last_change":"2024-04-23T16:36:23.728326+0000","last_active":"2024-04-23T16:36:29.147326+0000","last_peered":"2024-04-23T16:36:29.147326+0000","last_clean":"2024-04-23T16:36:29.147326+0000","last_became_active":"2024-04-23T16:31:29.139398+0000","last_became_peered":"2024-04-23T16:31:29.139398+0000","last_unstale":"2024-04-23T16:36:29.147326+0000","last_undegraded":"2024-04-23T16:36:29.147326+0000","last_fullsized":"2024-04-23T16:36:29.147326+0000","mapping_epoch":359,"log_start":"385'9800","ondisk_log_start":"385'9800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'19588","last_scrub_stamp":"2024-04-23T16:36:23.728231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:36:23.728231+0000","objects_scrubbed":1817,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:47.414183+0000","scrub_duration":340,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98697216,"num_objects":1506,"num_object_clones":0,"num_object_copies":3012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17622,"num_write_kb":612096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1138,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-04-23T16:36:29.184400+0000","last_change":"2024-04-23T16:35:33.875035+0000","last_active":"2024-04-23T16:36:29.184400+0000","last_peered":"2024-04-23T16:36:29.184400+0000","last_clean":"2024-04-23T16:36:29.184400+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:36:29.184400+0000","last_undegraded":"2024-04-23T16:36:29.184400+0000","last_fullsized":"2024-04-23T16:36:29.184400+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:35:33.874904+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:35:33.874904+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:59.611256+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"438'17445","reported_seq":17704,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-04-23T16:36:29.594808+0000","last_change":"2024-04-23T16:32:43.947374+0000","last_active":"2024-04-23T16:36:29.594808+0000","last_peered":"2024-04-23T16:36:29.594808+0000","last_clean":"2024-04-23T16:36:29.594808+0000","last_became_active":"2024-04-23T16:31:29.138968+0000","last_became_peered":"2024-04-23T16:31:29.138968+0000","last_unstale":"2024-04-23T16:36:29.594808+0000","last_undegraded":"2024-04-23T16:36:29.594808+0000","last_fullsized":"2024-04-23T16:36:29.594808+0000","mapping_epoch":359,"log_start":"380'7400","ondisk_log_start":"380'7400","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8482","last_scrub_stamp":"2024-04-23T16:32:43.947177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.947177+0000","objects_scrubbed":8428,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94961664,"num_objects":1449,"num_object_clones":0,"num_object_copies":2898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17445,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1025,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-04-23T16:36:19.212864+0000","last_change":"2024-04-23T16:34:51.248895+0000","last_active":"2024-04-23T16:36:19.212864+0000","last_peered":"2024-04-23T16:36:19.212864+0000","last_clean":"2024-04-23T16:36:19.212864+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:36:19.212864+0000","last_undegraded":"2024-04-23T16:36:19.212864+0000","last_fullsized":"2024-04-23T16:36:19.212864+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:34:51.248783+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:34:51.248783+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"438'17960","reported_seq":18219,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-04-23T16:36:29.562673+0000","last_change":"2024-04-23T16:32:39.103230+0000","last_active":"2024-04-23T16:36:29.562673+0000","last_peered":"2024-04-23T16:36:29.562673+0000","last_clean":"2024-04-23T16:36:29.562673+0000","last_became_active":"2024-04-23T16:31:29.310246+0000","last_became_peered":"2024-04-23T16:31:29.310246+0000","last_unstale":"2024-04-23T16:36:29.562673+0000","last_undegraded":"2024-04-23T16:36:29.562673+0000","last_fullsized":"2024-04-23T16:36:29.562673+0000","mapping_epoch":359,"log_start":"380'7900","ondisk_log_start":"380'7900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8325","last_scrub_stamp":"2024-04-23T16:32:39.103059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:39.103059+0000","objects_scrubbed":8272,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98435072,"num_objects":1502,"num_object_clones":0,"num_object_copies":3004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17960,"num_write_kb":622784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":959,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-04-23T16:36:29.184220+0000","last_change":"2024-04-23T16:36:19.091622+0000","last_active":"2024-04-23T16:36:29.184220+0000","last_peered":"2024-04-23T16:36:29.184220+0000","last_clean":"2024-04-23T16:36:29.184220+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:36:29.184220+0000","last_undegraded":"2024-04-23T16:36:29.184220+0000","last_fullsized":"2024-04-23T16:36:29.184220+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:19.091490+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:36:19.091490+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:22.009913+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"438'17671","reported_seq":34355,"reported_epoch":438,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:36:30.265892+0000","last_change":"2024-04-23T16:33:26.132487+0000","last_active":"2024-04-23T16:36:30.265892+0000","last_peered":"2024-04-23T16:36:30.265892+0000","last_clean":"2024-04-23T16:32:59.695133+0000","last_became_active":"2024-04-23T16:33:26.078062+0000","last_became_peered":"2024-04-23T16:33:26.078062+0000","last_unstale":"2024-04-23T16:36:30.265892+0000","last_undegraded":"2024-04-23T16:33:25.892009+0000","last_fullsized":"2024-04-23T16:36:30.265892+0000","mapping_epoch":394,"log_start":"380'7600","ondisk_log_start":"380'7600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'7968","last_scrub_stamp":"2024-04-23T16:32:36.102122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:36.102122+0000","objects_scrubbed":7926,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94568448,"num_objects":1443,"num_object_clones":0,"num_object_copies":2886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17671,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8211,"num_bytes_recovered":538116096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1342},{"shards":"5,8","objects":101}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1128,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-04-23T16:36:29.193616+0000","last_change":"2024-04-23T16:33:17.354409+0000","last_active":"2024-04-23T16:36:29.193616+0000","last_peered":"2024-04-23T16:36:29.193616+0000","last_clean":"2024-04-23T16:36:29.193616+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:36:29.193616+0000","last_undegraded":"2024-04-23T16:36:29.193616+0000","last_fullsized":"2024-04-23T16:36:29.193616+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:17.354263+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.354263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":1553334296,"num_objects":23703,"num_object_clones":0,"num_object_copies":47406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23703,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":283489,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24617,"num_bytes_recovered":1613299712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4693630976,"data_stored":4693622832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161020,"ondisk_log_size":161020,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056894,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":716160,"kb_used_data":211576,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93028992,"statfs":{"total":95995035648,"available":95261687808,"internally_reserved":0,"allocated":216653824,"data_stored":216197426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":516684228},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351236,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612874,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":641288,"kb_used_data":281088,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93103864,"statfs":{"total":95995035648,"available":95338356736,"internally_reserved":0,"allocated":287834112,"data_stored":287375732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":368835008},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841806,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":511868,"kb_used_data":201652,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":93233284,"statfs":{"total":95995035648,"available":95470882816,"internally_reserved":0,"allocated":206491648,"data_stored":206039322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168851,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":500056,"kb_used_data":95308,"kb_used_omap":1,"kb_used_meta":404606,"kb_avail":93245096,"statfs":{"total":95995035648,"available":95482978304,"internally_reserved":0,"allocated":97595392,"data_stored":97130958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":414317002},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463193,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":569612,"kb_used_data":285760,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93175540,"statfs":{"total":95995035648,"available":95411752960,"internally_reserved":0,"allocated":292618240,"data_stored":292159860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":290650566},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137333,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738678,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1145672,"kb_used_data":686912,"kb_used_omap":1,"kb_used_meta":458750,"kb_avail":92599480,"statfs":{"total":95995035648,"available":94821867520,"internally_reserved":0,"allocated":703397888,"data_stored":702939508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":469760453},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117287,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":454240,"kb_used_data":94036,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93290912,"statfs":{"total":95995035648,"available":95529893888,"internally_reserved":0,"allocated":96292864,"data_stored":95824878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":368835015},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921057,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1046796,"kb_used_data":488000,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92698356,"statfs":{"total":95995035648,"available":94923116544,"internally_reserved":0,"allocated":499712000,"data_stored":499253620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757532,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917992,"kb_used_data":477208,"kb_used_omap":1,"kb_used_meta":440702,"kb_avail":92827160,"statfs":{"total":95995035648,"available":95055011840,"internally_reserved":0,"allocated":488660992,"data_stored":488188934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":451279301},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":679936000,"data_stored":679936000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1886846976,"data_stored":1886846976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94044160,"data_stored":94044160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":484245504,"data_stored":484245504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":471404544,"data_stored":471400472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":285802496,"data_stored":285802496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":94896128,"data_stored":94896128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":200212480,"data_stored":200212480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":284295168,"data_stored":284295168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":211947520,"data_stored":211943448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:36:33.348 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:36:33.604 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-23T16:36:33.606 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:36:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:33.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:36:33.713 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:36:33.747 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:33.770 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:36:33.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:33.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:36:33.875 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:36:33.878 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:33 smithi148 ceph-mon[44520]: from='client.16182 v1:172.21.15.103:0/2875361147' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:36:33.878 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:33 smithi148 ceph-mon[44520]: pgmap v1248: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.4 GiB data, 6.2 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 797 op/s; 4335/46324 objects degraded (9.358%); 8.4 MiB/s, 135 objects/s recovering 2024-04-23T16:36:33.903 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:33.925 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:36:33.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:33.928 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:36:33.930 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:33 smithi103 ceph-mon[39629]: from='client.16182 v1:172.21.15.103:0/2875361147' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:36:33.930 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:33 smithi103 ceph-mon[39629]: pgmap v1248: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.4 GiB data, 6.2 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 797 op/s; 4335/46324 objects degraded (9.358%); 8.4 MiB/s, 135 objects/s recovering 2024-04-23T16:36:33.930 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:36:34.028 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:36:34.103 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:36:34.141 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:34.174 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:36:34.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:34.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:36:34.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:33 smithi188 ceph-mon[44101]: from='client.16182 v1:172.21.15.103:0/2875361147' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:36:34.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:33 smithi188 ceph-mon[44101]: pgmap v1248: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.4 GiB data, 6.2 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 797 op/s; 4335/46324 objects degraded (9.358%); 8.4 MiB/s, 135 objects/s recovering 2024-04-23T16:36:34.282 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:36:34.312 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:34.331 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:34.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:34.334 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:34.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:36:34.463 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:34.912 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:36:34.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:34.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:36:35.015 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:35.076 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:34 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:36:35.076 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:34 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:36:35.076 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:34 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 4335/46324 objects degraded (9.358%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:35.076 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:34 smithi148 ceph-mon[44520]: osdmap e440: 12 total, 11 up, 9 in 2024-04-23T16:36:35.082 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:34 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:36:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:34 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:36:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:34 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 4335/46324 objects degraded (9.358%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:35.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:34 smithi103 ceph-mon[39629]: osdmap e440: 12 total, 11 up, 9 in 2024-04-23T16:36:35.116 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:35.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:35.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:35.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:34 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:36:35.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:34 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:36:35.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:34 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 4335/46324 objects degraded (9.358%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:35.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:34 smithi188 ceph-mon[44101]: osdmap e440: 12 total, 11 up, 9 in 2024-04-23T16:36:35.224 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:36:35.286 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:35.623 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:36:36.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:35 smithi103 ceph-mon[39629]: pgmap v1250: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.3 GiB data, 5.8 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 4060/41462 objects degraded (9.792%); 8.5 MiB/s, 136 objects/s recovering 2024-04-23T16:36:36.133 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:35 smithi148 ceph-mon[44520]: pgmap v1250: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.3 GiB data, 5.8 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 4060/41462 objects degraded (9.792%); 8.5 MiB/s, 136 objects/s recovering 2024-04-23T16:36:36.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:35 smithi188 ceph-mon[44101]: pgmap v1250: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.3 GiB data, 5.8 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 4060/41462 objects degraded (9.792%); 8.5 MiB/s, 136 objects/s recovering 2024-04-23T16:36:36.313 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:36:36.771 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:36:37.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1736488595' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:36:37.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1736488595' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:36:37.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1736488595' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:36:37.253 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:36:37.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:37.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:36:37.358 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:36:37.420 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:37.768 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:37 smithi148 ceph-mon[44520]: pgmap v1251: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.2 GiB data, 5.7 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 953 op/s; 3469/38738 objects degraded (8.955%); 12 MiB/s, 186 objects/s recovering 2024-04-23T16:36:37.769 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:37 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:36:37.769 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:37 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1736488595' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:36:37.769 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:37 smithi148 ceph-mon[44520]: osdmap e441: 12 total, 11 up, 9 in 2024-04-23T16:36:38.036 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:37 smithi103 ceph-mon[39629]: pgmap v1251: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.2 GiB data, 5.7 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 953 op/s; 3469/38738 objects degraded (8.955%); 12 MiB/s, 186 objects/s recovering 2024-04-23T16:36:38.036 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:37 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:36:38.036 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:37 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1736488595' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:36:38.036 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:37 smithi103 ceph-mon[39629]: osdmap e441: 12 total, 11 up, 9 in 2024-04-23T16:36:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:37 smithi188 ceph-mon[44101]: pgmap v1251: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.2 GiB data, 5.7 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 953 op/s; 3469/38738 objects degraded (8.955%); 12 MiB/s, 186 objects/s recovering 2024-04-23T16:36:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:37 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:36:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:37 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1736488595' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:36:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:37 smithi188 ceph-mon[44101]: osdmap e441: 12 total, 11 up, 9 in 2024-04-23T16:36:38.391 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:36:38.392 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:36:38.392 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:36:38.392 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:36:38.410 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:36:38.438 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:36:38.466 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:36:38.487 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:36:38.511 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:36:38.532 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:36:38.559 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:36:38.559 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:36:38.559 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:36:38.559 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:36:38.559 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:36:38.559 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:36:38.559 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:36:38.560 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:36:38.560 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:36:38.560 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:36:38.560 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:36:38.560 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:36:38.560 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:36:38.598 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:36:38.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:38.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:36:38.705 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:36:38.738 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:38.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:38 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:36:38.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:38 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:36:38.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:38 smithi103 ceph-mon[39629]: osdmap e442: 12 total, 11 up, 9 in 2024-04-23T16:36:38.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:38 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2893085334' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:38.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:38.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:38.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:38.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:38.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:38.879 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:38 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:36:38.880 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:38 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:36:38.880 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:38 smithi148 ceph-mon[44520]: osdmap e442: 12 total, 11 up, 9 in 2024-04-23T16:36:38.880 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:38 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2893085334' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:38.880 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:38.880 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:38.881 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:38.881 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:38.881 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:38.938 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:36:38.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:38.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:36:39.044 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:39.052 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:36:39.052 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:36:39.120 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:39.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:38 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:36:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:38 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:36:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:38 smithi188 ceph-mon[44101]: osdmap e442: 12 total, 11 up, 9 in 2024-04-23T16:36:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:38 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2893085334' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:39.292 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:39.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:39.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:39.396 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:39.455 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:39.703 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:36:39.704 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:36:39.723 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:36:39.782 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:39.782 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:40.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:40.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:40.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:40.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:40.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:40.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:40.029 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:39 smithi103 ceph-mon[39629]: pgmap v1254: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.1 GiB data, 5.6 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 766 op/s; 3187/37124 objects degraded (8.585%); 12 MiB/s, 191 objects/s recovering 2024-04-23T16:36:40.029 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:39 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 3187/37124 objects degraded (8.585%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:40.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:40.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:40.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:40.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:40.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:39 smithi148 ceph-mon[44520]: pgmap v1254: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.1 GiB data, 5.6 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 766 op/s; 3187/37124 objects degraded (8.585%); 12 MiB/s, 191 objects/s recovering 2024-04-23T16:36:40.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:39 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 3187/37124 objects degraded (8.585%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:39 smithi188 ceph-mon[44101]: pgmap v1254: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 1.1 GiB data, 5.6 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 766 op/s; 3187/37124 objects degraded (8.585%); 12 MiB/s, 191 objects/s recovering 2024-04-23T16:36:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:39 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 3187/37124 objects degraded (8.585%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:40.549 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:40.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:40.555 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-23T16:36:40.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:40.656 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:36:40.685 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:40.705 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:36:40.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:40.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:36:40.809 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:36:40.896 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:40.943 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:36:40.965 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:36:40.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:40.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:36:41.074 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:36:41.133 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:36:41.143 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:41.190 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:36:41.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:41.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:36:41.295 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:36:41.321 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:36:41.323 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:41.348 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:41.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:41.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:41.453 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:36:41.479 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:41.502 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:41.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:41.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:41.605 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:36:41.639 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:41.705 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:36:41.768 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:36:41.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:41.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:36:41.872 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:36:41.925 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:41 smithi103 ceph-mon[39629]: pgmap v1255: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 938 MiB data, 5.2 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s; 2550/30000 objects degraded (8.500%); 14 MiB/s, 230 objects/s recovering 2024-04-23T16:36:41.934 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:42.063 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:36:42.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:42.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:36:42.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:41 smithi148 ceph-mon[44520]: pgmap v1255: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 938 MiB data, 5.2 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s; 2550/30000 objects degraded (8.500%); 14 MiB/s, 230 objects/s recovering 2024-04-23T16:36:42.168 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:36:42.205 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:36:42.209 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:36:42.220 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:42.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:41 smithi188 ceph-mon[44101]: pgmap v1255: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 938 MiB data, 5.2 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s; 2550/30000 objects degraded (8.500%); 14 MiB/s, 230 objects/s recovering 2024-04-23T16:36:42.602 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:36:42.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:42.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:36:42.708 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:36:42.779 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:42.820 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:36:42.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:42.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:36:42.929 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:36:42.959 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:43.324 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:36:43.324 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:36:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:43 smithi103 ceph-mon[39629]: from='client.16197 v1:172.21.15.103:0/1693898061' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:36:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:43 smithi103 ceph-mon[39629]: pgmap v1256: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 905 MiB data, 5.2 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 782 op/s; 2550/28934 objects degraded (8.813%); 12 MiB/s, 188 objects/s recovering 2024-04-23T16:36:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:43 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:36:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:43 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:36:44.100 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:43 smithi148 ceph-mon[44520]: from='client.16197 v1:172.21.15.103:0/1693898061' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:36:44.101 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:43 smithi148 ceph-mon[44520]: pgmap v1256: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 905 MiB data, 5.2 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 782 op/s; 2550/28934 objects degraded (8.813%); 12 MiB/s, 188 objects/s recovering 2024-04-23T16:36:44.101 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:43 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:36:44.101 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:43 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:36:44.147 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-23T16:36:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:43 smithi188 ceph-mon[44101]: from='client.16197 v1:172.21.15.103:0/1693898061' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:36:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:43 smithi188 ceph-mon[44101]: pgmap v1256: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 905 MiB data, 5.2 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 782 op/s; 2550/28934 objects degraded (8.813%); 12 MiB/s, 188 objects/s recovering 2024-04-23T16:36:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:43 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:36:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:43 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:36:44.636 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:36:44.676 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1255,"stamp":"2024-04-23T16:36:40.746661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":983433784,"num_objects":15000,"num_object_clones":0,"num_object_copies":30000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15000,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":292194,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26403,"num_bytes_recovered":1730482720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162254,"ondisk_log_size":162254,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":5499276,"kb_used_data":1837880,"kb_used_omap":14,"kb_used_meta":3660977,"kb_avail":838207092,"statfs":{"total":863955320832,"available":858324062208,"internally_reserved":0,"allocated":1881989120,"data_stored":1877607942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3748841455},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,6,6,0,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-482213888,"num_objects":-7358,"num_object_clones":0,"num_object_copies":-14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1510,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7358,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1510,"num_bytes_recovered":98959360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.561502"},"pg_stats":[{"pgid":"5.f","version":"442'18501","reported_seq":18781,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:40.306116+0000","last_change":"2024-04-23T16:33:17.526419+0000","last_active":"2024-04-23T16:36:40.306116+0000","last_peered":"2024-04-23T16:36:40.306116+0000","last_clean":"2024-04-23T16:36:40.306116+0000","last_became_active":"2024-04-23T16:31:29.140923+0000","last_became_peered":"2024-04-23T16:31:29.140923+0000","last_unstale":"2024-04-23T16:36:40.306116+0000","last_undegraded":"2024-04-23T16:36:40.306116+0000","last_fullsized":"2024-04-23T16:36:40.306116+0000","mapping_epoch":359,"log_start":"382'8500","ondisk_log_start":"382'8500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9683","last_scrub_stamp":"2024-04-23T16:33:17.526283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.526283+0000","objects_scrubbed":9683,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56688640,"num_objects":865,"num_object_clones":0,"num_object_copies":1730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18501,"num_write_kb":619712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"442'18310","reported_seq":18716,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:39.309400+0000","last_change":"2024-04-23T16:35:26.407358+0000","last_active":"2024-04-23T16:36:39.309400+0000","last_peered":"2024-04-23T16:36:39.309400+0000","last_clean":"2024-04-23T16:36:39.309400+0000","last_became_active":"2024-04-23T16:31:29.137890+0000","last_became_peered":"2024-04-23T16:31:29.137890+0000","last_unstale":"2024-04-23T16:36:39.309400+0000","last_undegraded":"2024-04-23T16:36:39.309400+0000","last_fullsized":"2024-04-23T16:36:39.309400+0000","mapping_epoch":359,"log_start":"382'8300","ondisk_log_start":"382'8300","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'14582","last_scrub_stamp":"2024-04-23T16:35:26.407093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:26.407093+0000","objects_scrubbed":4692,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:44.093448+0000","scrub_duration":544,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62521344,"num_objects":954,"num_object_clones":0,"num_object_copies":1908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18310,"num_write_kb":616448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"442'18437","reported_seq":36184,"reported_epoch":442,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:36:38.235961+0000","last_change":"2024-04-23T16:33:26.432192+0000","last_active":"2024-04-23T16:36:38.235961+0000","last_peered":"2024-04-23T16:36:38.235961+0000","last_clean":"2024-04-23T16:32:59.749530+0000","last_became_active":"2024-04-23T16:33:26.338644+0000","last_became_peered":"2024-04-23T16:33:26.338644+0000","last_unstale":"2024-04-23T16:36:38.235961+0000","last_undegraded":"2024-04-23T16:33:25.874847+0000","last_fullsized":"2024-04-23T16:36:38.235961+0000","mapping_epoch":394,"log_start":"382'8400","ondisk_log_start":"382'8400","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8711","last_scrub_stamp":"2024-04-23T16:32:43.928914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.928914+0000","objects_scrubbed":8680,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65863680,"num_objects":1005,"num_object_clones":0,"num_object_copies":2010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18437,"num_write_kb":622144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8735,"num_bytes_recovered":572456960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":["8"],"object_location_counts":[{"shards":"4,8","objects":28},{"shards":"8","objects":977}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.9","version":"442'18421","reported_seq":18686,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:40.202705+0000","last_change":"2024-04-23T16:32:37.256944+0000","last_active":"2024-04-23T16:36:40.202705+0000","last_peered":"2024-04-23T16:36:40.202705+0000","last_clean":"2024-04-23T16:36:40.202705+0000","last_became_active":"2024-04-23T16:31:29.141551+0000","last_became_peered":"2024-04-23T16:31:29.141551+0000","last_unstale":"2024-04-23T16:36:40.202705+0000","last_undegraded":"2024-04-23T16:36:40.202705+0000","last_fullsized":"2024-04-23T16:36:40.202705+0000","mapping_epoch":359,"log_start":"382'8400","ondisk_log_start":"382'8400","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8156","last_scrub_stamp":"2024-04-23T16:32:37.256787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:37.256787+0000","objects_scrubbed":8082,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58916864,"num_objects":899,"num_object_clones":0,"num_object_copies":1798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18421,"num_write_kb":618240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"441'17669","reported_seq":18045,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-04-23T16:36:36.783763+0000","last_change":"2024-04-23T16:36:20.471641+0000","last_active":"2024-04-23T16:36:36.783763+0000","last_peered":"2024-04-23T16:36:36.783763+0000","last_clean":"2024-04-23T16:36:36.783763+0000","last_became_active":"2024-04-23T16:31:29.139045+0000","last_became_peered":"2024-04-23T16:31:29.139045+0000","last_unstale":"2024-04-23T16:36:36.783763+0000","last_undegraded":"2024-04-23T16:36:36.783763+0000","last_fullsized":"2024-04-23T16:36:36.783763+0000","mapping_epoch":359,"log_start":"380'7600","ondisk_log_start":"380'7600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'16781","last_scrub_stamp":"2024-04-23T16:36:20.471467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:36:20.471467+0000","objects_scrubbed":1958,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:48.916510+0000","scrub_duration":248,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69795840,"num_objects":1065,"num_object_clones":0,"num_object_copies":2130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17669,"num_write_kb":599488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"442'21804","reported_seq":22281,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:39.311891+0000","last_change":"2024-04-23T16:35:32.474057+0000","last_active":"2024-04-23T16:36:39.311891+0000","last_peered":"2024-04-23T16:36:39.311891+0000","last_clean":"2024-04-23T16:36:39.311891+0000","last_became_active":"2024-04-23T16:31:29.310906+0000","last_became_peered":"2024-04-23T16:31:29.310906+0000","last_unstale":"2024-04-23T16:36:39.311891+0000","last_undegraded":"2024-04-23T16:36:39.311891+0000","last_fullsized":"2024-04-23T16:36:39.311891+0000","mapping_epoch":359,"log_start":"406'11704","ondisk_log_start":"406'11704","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'18441","last_scrub_stamp":"2024-04-23T16:35:32.473828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:32.473828+0000","objects_scrubbed":4251,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:57.506974+0000","scrub_duration":548,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57475072,"num_objects":877,"num_object_clones":0,"num_object_copies":1754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18243,"num_write_kb":611840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1154,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:37.779209+0000","last_change":"2024-04-23T16:36:19.132015+0000","last_active":"2024-04-23T16:36:37.779209+0000","last_peered":"2024-04-23T16:36:37.779209+0000","last_clean":"2024-04-23T16:36:37.779209+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:36:37.779209+0000","last_undegraded":"2024-04-23T16:36:37.779209+0000","last_fullsized":"2024-04-23T16:36:37.779209+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:19.131883+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:19.131883+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:41.022603+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"442'18572","reported_seq":18838,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:39.645561+0000","last_change":"2024-04-23T16:32:38.103756+0000","last_active":"2024-04-23T16:36:39.645561+0000","last_peered":"2024-04-23T16:36:39.645561+0000","last_clean":"2024-04-23T16:36:39.645561+0000","last_became_active":"2024-04-23T16:31:29.141640+0000","last_became_peered":"2024-04-23T16:31:29.141640+0000","last_unstale":"2024-04-23T16:36:39.645561+0000","last_undegraded":"2024-04-23T16:36:39.645561+0000","last_fullsized":"2024-04-23T16:36:39.645561+0000","mapping_epoch":359,"log_start":"382'8500","ondisk_log_start":"382'8500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8300","last_scrub_stamp":"2024-04-23T16:32:38.103547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:38.103547+0000","objects_scrubbed":8231,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57933824,"num_objects":884,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18572,"num_write_kb":622592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1140,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:37.796559+0000","last_change":"2024-04-23T16:33:25.851379+0000","last_active":"2024-04-23T16:36:37.796559+0000","last_peered":"2024-04-23T16:36:37.796559+0000","last_clean":"2024-04-23T16:36:37.796559+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:36:37.796559+0000","last_undegraded":"2024-04-23T16:36:37.796559+0000","last_fullsized":"2024-04-23T16:36:37.796559+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.498820+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.498820+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"441'18047","reported_seq":18425,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-04-23T16:36:36.799959+0000","last_change":"2024-04-23T16:36:24.521917+0000","last_active":"2024-04-23T16:36:36.799959+0000","last_peered":"2024-04-23T16:36:36.799959+0000","last_clean":"2024-04-23T16:36:36.799959+0000","last_became_active":"2024-04-23T16:31:29.140251+0000","last_became_peered":"2024-04-23T16:31:29.140251+0000","last_unstale":"2024-04-23T16:36:36.799959+0000","last_undegraded":"2024-04-23T16:36:36.799959+0000","last_fullsized":"2024-04-23T16:36:36.799959+0000","mapping_epoch":359,"log_start":"381'8000","ondisk_log_start":"381'8000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'17367","last_scrub_stamp":"2024-04-23T16:36:24.521728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:36:24.521728+0000","objects_scrubbed":1767,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:41.002308+0000","scrub_duration":288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71041048,"num_objects":1085,"num_object_clones":0,"num_object_copies":2170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1085,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18047,"num_write_kb":612161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1131,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:37.783873+0000","last_change":"2024-04-23T16:32:29.517947+0000","last_active":"2024-04-23T16:36:37.783873+0000","last_peered":"2024-04-23T16:36:37.783873+0000","last_clean":"2024-04-23T16:36:37.783873+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:36:37.783873+0000","last_undegraded":"2024-04-23T16:36:37.783873+0000","last_fullsized":"2024-04-23T16:36:37.783873+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:32:29.517800+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:32:29.517800+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"442'18488","reported_seq":36369,"reported_epoch":442,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:36:40.328528+0000","last_change":"2024-04-23T16:33:26.507399+0000","last_active":"2024-04-23T16:36:40.328528+0000","last_peered":"2024-04-23T16:36:40.328528+0000","last_clean":"2024-04-23T16:32:59.715575+0000","last_became_active":"2024-04-23T16:33:26.458611+0000","last_became_peered":"2024-04-23T16:33:26.458611+0000","last_unstale":"2024-04-23T16:36:40.328528+0000","last_undegraded":"2024-04-23T16:33:25.895472+0000","last_fullsized":"2024-04-23T16:36:40.328528+0000","mapping_epoch":394,"log_start":"382'8400","ondisk_log_start":"382'8400","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8607","last_scrub_stamp":"2024-04-23T16:32:43.130806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.130806+0000","objects_scrubbed":8577,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58458112,"num_objects":892,"num_object_clones":0,"num_object_copies":1784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18488,"num_write_kb":620160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8803,"num_bytes_recovered":576913408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":884},{"shards":"5,7","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1153,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:37.779313+0000","last_change":"2024-04-23T16:36:26.456131+0000","last_active":"2024-04-23T16:36:37.779313+0000","last_peered":"2024-04-23T16:36:37.779313+0000","last_clean":"2024-04-23T16:36:37.779313+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:36:37.779313+0000","last_undegraded":"2024-04-23T16:36:37.779313+0000","last_fullsized":"2024-04-23T16:36:37.779313+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:26.456003+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:26.456003+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:44.948237+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"442'18070","reported_seq":18347,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:40.305941+0000","last_change":"2024-04-23T16:33:18.576077+0000","last_active":"2024-04-23T16:36:40.305941+0000","last_peered":"2024-04-23T16:36:40.305941+0000","last_clean":"2024-04-23T16:36:40.305941+0000","last_became_active":"2024-04-23T16:31:29.139709+0000","last_became_peered":"2024-04-23T16:31:29.139709+0000","last_unstale":"2024-04-23T16:36:40.305941+0000","last_undegraded":"2024-04-23T16:36:40.305941+0000","last_fullsized":"2024-04-23T16:36:40.305941+0000","mapping_epoch":359,"log_start":"381'8000","ondisk_log_start":"381'8000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9472","last_scrub_stamp":"2024-04-23T16:33:18.575945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.575945+0000","objects_scrubbed":9472,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57278464,"num_objects":874,"num_object_clones":0,"num_object_copies":1748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18070,"num_write_kb":606208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1146,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:37.780184+0000","last_change":"2024-04-23T16:35:34.878630+0000","last_active":"2024-04-23T16:36:37.780184+0000","last_peered":"2024-04-23T16:36:37.780184+0000","last_clean":"2024-04-23T16:36:37.780184+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:36:37.780184+0000","last_undegraded":"2024-04-23T16:36:37.780184+0000","last_fullsized":"2024-04-23T16:36:37.780184+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:35:34.878536+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:35:34.878536+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:52.503403+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"442'18430","reported_seq":18814,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:39.187329+0000","last_change":"2024-04-23T16:36:20.849660+0000","last_active":"2024-04-23T16:36:39.187329+0000","last_peered":"2024-04-23T16:36:39.187329+0000","last_clean":"2024-04-23T16:36:39.187329+0000","last_became_active":"2024-04-23T16:31:29.139606+0000","last_became_peered":"2024-04-23T16:31:29.139606+0000","last_unstale":"2024-04-23T16:36:39.187329+0000","last_undegraded":"2024-04-23T16:36:39.187329+0000","last_fullsized":"2024-04-23T16:36:39.187329+0000","mapping_epoch":359,"log_start":"382'8400","ondisk_log_start":"382'8400","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'17441","last_scrub_stamp":"2024-04-23T16:36:20.849516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:36:20.849516+0000","objects_scrubbed":2009,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:50.602166+0000","scrub_duration":400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66191360,"num_objects":1010,"num_object_clones":0,"num_object_copies":2020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18430,"num_write_kb":622080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1153,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:37.779210+0000","last_change":"2024-04-23T16:36:23.239567+0000","last_active":"2024-04-23T16:36:37.779210+0000","last_peered":"2024-04-23T16:36:37.779210+0000","last_clean":"2024-04-23T16:36:37.779210+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:36:37.779210+0000","last_undegraded":"2024-04-23T16:36:37.779210+0000","last_fullsized":"2024-04-23T16:36:37.779210+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:23.239466+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:36:23.239466+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:27.031497+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.2","version":"442'18093","reported_seq":18501,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:38.967624+0000","last_change":"2024-04-23T16:35:32.569280+0000","last_active":"2024-04-23T16:36:38.967624+0000","last_peered":"2024-04-23T16:36:38.967624+0000","last_clean":"2024-04-23T16:36:38.967624+0000","last_became_active":"2024-04-23T16:31:29.138430+0000","last_became_peered":"2024-04-23T16:31:29.138430+0000","last_unstale":"2024-04-23T16:36:38.967624+0000","last_undegraded":"2024-04-23T16:36:38.967624+0000","last_fullsized":"2024-04-23T16:36:38.967624+0000","mapping_epoch":359,"log_start":"381'8000","ondisk_log_start":"381'8000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'14723","last_scrub_stamp":"2024-04-23T16:35:32.569137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:32.569137+0000","objects_scrubbed":4294,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":850,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59179008,"num_objects":903,"num_object_clones":0,"num_object_copies":1806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18093,"num_write_kb":607872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1154,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:37.779239+0000","last_change":"2024-04-23T16:36:18.085504+0000","last_active":"2024-04-23T16:36:37.779239+0000","last_peered":"2024-04-23T16:36:37.779239+0000","last_clean":"2024-04-23T16:36:37.779239+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:36:37.779239+0000","last_undegraded":"2024-04-23T16:36:37.779239+0000","last_fullsized":"2024-04-23T16:36:37.779239+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:18.085386+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:18.085386+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:25.354657+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"442'20432","reported_seq":20941,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:39.173881+0000","last_change":"2024-04-23T16:36:23.728326+0000","last_active":"2024-04-23T16:36:39.173881+0000","last_peered":"2024-04-23T16:36:39.173881+0000","last_clean":"2024-04-23T16:36:39.173881+0000","last_became_active":"2024-04-23T16:31:29.139398+0000","last_became_peered":"2024-04-23T16:31:29.139398+0000","last_unstale":"2024-04-23T16:36:39.173881+0000","last_undegraded":"2024-04-23T16:36:39.173881+0000","last_fullsized":"2024-04-23T16:36:39.173881+0000","mapping_epoch":359,"log_start":"386'10400","ondisk_log_start":"386'10400","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'19588","last_scrub_stamp":"2024-04-23T16:36:23.728231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:36:23.728231+0000","objects_scrubbed":1817,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:47.414183+0000","scrub_duration":340,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63569920,"num_objects":970,"num_object_clones":0,"num_object_copies":1940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18158,"num_write_kb":612096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1146,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:37.782671+0000","last_change":"2024-04-23T16:35:33.875035+0000","last_active":"2024-04-23T16:36:37.782671+0000","last_peered":"2024-04-23T16:36:37.782671+0000","last_clean":"2024-04-23T16:36:37.782671+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:36:37.782671+0000","last_undegraded":"2024-04-23T16:36:37.782671+0000","last_fullsized":"2024-04-23T16:36:37.782671+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:35:33.874904+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:35:33.874904+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:59.611256+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"442'18010","reported_seq":18277,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:39.651597+0000","last_change":"2024-04-23T16:32:43.947374+0000","last_active":"2024-04-23T16:36:39.651597+0000","last_peered":"2024-04-23T16:36:39.651597+0000","last_clean":"2024-04-23T16:36:39.651597+0000","last_became_active":"2024-04-23T16:31:29.138968+0000","last_became_peered":"2024-04-23T16:31:29.138968+0000","last_unstale":"2024-04-23T16:36:39.651597+0000","last_undegraded":"2024-04-23T16:36:39.651597+0000","last_fullsized":"2024-04-23T16:36:39.651597+0000","mapping_epoch":359,"log_start":"381'8000","ondisk_log_start":"381'8000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8482","last_scrub_stamp":"2024-04-23T16:32:43.947177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.947177+0000","objects_scrubbed":8428,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57933824,"num_objects":884,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18010,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1043,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-04-23T16:36:36.777463+0000","last_change":"2024-04-23T16:36:29.341359+0000","last_active":"2024-04-23T16:36:36.777463+0000","last_peered":"2024-04-23T16:36:36.777463+0000","last_clean":"2024-04-23T16:36:36.777463+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:36:36.777463+0000","last_undegraded":"2024-04-23T16:36:36.777463+0000","last_fullsized":"2024-04-23T16:36:36.777463+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:29.341205+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:29.341205+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:50.871176+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"442'18525","reported_seq":18792,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:39.663509+0000","last_change":"2024-04-23T16:32:39.103230+0000","last_active":"2024-04-23T16:36:39.663509+0000","last_peered":"2024-04-23T16:36:39.663509+0000","last_clean":"2024-04-23T16:36:39.663509+0000","last_became_active":"2024-04-23T16:31:29.310246+0000","last_became_peered":"2024-04-23T16:31:29.310246+0000","last_unstale":"2024-04-23T16:36:39.663509+0000","last_undegraded":"2024-04-23T16:36:39.663509+0000","last_fullsized":"2024-04-23T16:36:39.663509+0000","mapping_epoch":359,"log_start":"382'8500","ondisk_log_start":"382'8500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8325","last_scrub_stamp":"2024-04-23T16:32:39.103059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:39.103059+0000","objects_scrubbed":8272,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61407232,"num_objects":937,"num_object_clones":0,"num_object_copies":1874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18525,"num_write_kb":622784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":967,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:37.780043+0000","last_change":"2024-04-23T16:36:19.091622+0000","last_active":"2024-04-23T16:36:37.780043+0000","last_peered":"2024-04-23T16:36:37.780043+0000","last_clean":"2024-04-23T16:36:37.780043+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:36:37.780043+0000","last_undegraded":"2024-04-23T16:36:37.780043+0000","last_fullsized":"2024-04-23T16:36:37.780043+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:19.091490+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:36:19.091490+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:22.009913+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"442'18220","reported_seq":36217,"reported_epoch":442,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:36:40.329282+0000","last_change":"2024-04-23T16:33:26.132487+0000","last_active":"2024-04-23T16:36:40.329282+0000","last_peered":"2024-04-23T16:36:40.329282+0000","last_clean":"2024-04-23T16:32:59.695133+0000","last_became_active":"2024-04-23T16:33:26.078062+0000","last_became_peered":"2024-04-23T16:33:26.078062+0000","last_unstale":"2024-04-23T16:36:40.329282+0000","last_undegraded":"2024-04-23T16:33:25.892009+0000","last_fullsized":"2024-04-23T16:36:40.329282+0000","mapping_epoch":394,"log_start":"382'8200","ondisk_log_start":"382'8200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'7968","last_scrub_stamp":"2024-04-23T16:32:36.102122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:36.102122+0000","objects_scrubbed":7926,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58589184,"num_objects":894,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18220,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8864,"num_bytes_recovered":580911104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":689},{"shards":"5,8","objects":205}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1136,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-04-23T16:36:37.783648+0000","last_change":"2024-04-23T16:33:17.354409+0000","last_active":"2024-04-23T16:36:37.783648+0000","last_peered":"2024-04-23T16:36:37.783648+0000","last_clean":"2024-04-23T16:36:37.783648+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:36:37.783648+0000","last_undegraded":"2024-04-23T16:36:37.783648+0000","last_fullsized":"2024-04-23T16:36:37.783648+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:17.354263+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.354263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":982843416,"num_objects":14998,"num_object_clones":0,"num_object_copies":29996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14998,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":292194,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26402,"num_bytes_recovered":1730281472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3692634112,"data_stored":3692625968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160725,"ondisk_log_size":160725,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056896,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":654704,"kb_used_data":145960,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93090448,"statfs":{"total":95995035648,"available":95324618752,"internally_reserved":0,"allocated":149463040,"data_stored":148982576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":520944068},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351238,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612876,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":543368,"kb_used_data":179004,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93201784,"statfs":{"total":95995035648,"available":95438626816,"internally_reserved":0,"allocated":183300096,"data_stored":182811646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":373094848},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841808,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":439356,"kb_used_data":137456,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93305796,"statfs":{"total":95995035648,"available":95545135104,"internally_reserved":0,"allocated":140754944,"data_stored":140272498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":453828,"kb_used_data":61628,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93291324,"statfs":{"total":95995035648,"available":95530315776,"internally_reserved":0,"allocated":63107072,"data_stored":62618622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":401537482},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463195,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":476612,"kb_used_data":184380,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93268540,"statfs":{"total":95995035648,"available":95506984960,"internally_reserved":0,"allocated":188805120,"data_stored":188316670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":299170246},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137334,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738680,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894280,"kb_used_data":439676,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":92850872,"statfs":{"total":95995035648,"available":95079292928,"internally_reserved":0,"allocated":450228224,"data_stored":449739774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117288,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":436724,"kb_used_data":76524,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93308428,"statfs":{"total":95995035648,"available":95547830272,"internally_reserved":0,"allocated":78360576,"data_stored":77880632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":368835015},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921059,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":868228,"kb_used_data":313532,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92876924,"statfs":{"total":95995035648,"available":95105970176,"internally_reserved":0,"allocated":321056768,"data_stored":320568318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757534,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":732176,"kb_used_data":299720,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93012976,"statfs":{"total":95995035648,"available":95245287424,"internally_reserved":0,"allocated":306913280,"data_stored":306417206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,6,0,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":432734208,"data_stored":432734208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1886846976,"data_stored":1886846976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":76349440,"data_stored":76349440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":303038464,"data_stored":303038464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":290263040,"data_stored":290258968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":187170816,"data_stored":187170816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":59441152,"data_stored":59441152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":135593984,"data_stored":135593984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":180420608,"data_stored":180420608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":140775424,"data_stored":140771352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:36:44.678 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:36:44.709 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:36:44.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:44.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:36:44.815 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:36:44.865 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:44.934 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:36:44.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:44.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:36:45.044 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:36:45.078 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:44 smithi188 ceph-mon[44101]: osdmap e443: 12 total, 11 up, 9 in 2024-04-23T16:36:45.078 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:44 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 2550/28934 objects degraded (8.813%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:45.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:44 smithi188 ceph-mon[44101]: osdmap e444: 12 total, 11 up, 9 in 2024-04-23T16:36:45.081 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:45.098 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:44 smithi148 ceph-mon[44520]: osdmap e443: 12 total, 11 up, 9 in 2024-04-23T16:36:45.099 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:44 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 2550/28934 objects degraded (8.813%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:45.099 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:44 smithi148 ceph-mon[44520]: osdmap e444: 12 total, 11 up, 9 in 2024-04-23T16:36:45.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:44 smithi103 ceph-mon[39629]: osdmap e443: 12 total, 11 up, 9 in 2024-04-23T16:36:45.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:44 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 2550/28934 objects degraded (8.813%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:45.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:44 smithi103 ceph-mon[39629]: osdmap e444: 12 total, 11 up, 9 in 2024-04-23T16:36:45.107 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:36:45.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:45.115 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:36:45.216 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:36:45.217 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:36:45.244 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:45.257 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:36:45.273 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:45.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:45.278 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:45.379 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:36:45.475 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:45.608 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:36:45.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:45.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:36:45.728 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:36:45.759 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:45.781 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:45.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:45.786 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:45.888 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:36:45.917 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:46.037 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:45 smithi103 ceph-mon[39629]: pgmap v1259: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 748 MiB data, 4.9 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 984 op/s; 2290/23916 objects degraded (9.575%); 11 MiB/s, 169 objects/s recovering 2024-04-23T16:36:46.132 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:45 smithi148 ceph-mon[44520]: pgmap v1259: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 748 MiB data, 4.9 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 984 op/s; 2290/23916 objects degraded (9.575%); 11 MiB/s, 169 objects/s recovering 2024-04-23T16:36:46.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:45 smithi188 ceph-mon[44101]: pgmap v1259: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 748 MiB data, 4.9 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 984 op/s; 2290/23916 objects degraded (9.575%); 11 MiB/s, 169 objects/s recovering 2024-04-23T16:36:47.127 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:36:47.127 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:36:47.472 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:36:47.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:47.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:36:47.579 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:36:47.607 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:47.627 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:36:47.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:47.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:36:47.732 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:36:47.762 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:47.787 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:36:47.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:47.790 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:36:47.834 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:47 smithi148 ceph-mon[44520]: pgmap v1260: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 661 MiB data, 4.7 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 998 op/s; 1618/21146 objects degraded (7.652%); 12 MiB/s, 196 objects/s recovering 2024-04-23T16:36:47.891 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:47.938 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-23T16:36:47.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:47 smithi188 ceph-mon[44101]: pgmap v1260: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 661 MiB data, 4.7 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 998 op/s; 1618/21146 objects degraded (7.652%); 12 MiB/s, 196 objects/s recovering 2024-04-23T16:36:47.986 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:48.026 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:48.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:48.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:47 smithi103 ceph-mon[39629]: pgmap v1260: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 661 MiB data, 4.7 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 998 op/s; 1618/21146 objects degraded (7.652%); 12 MiB/s, 196 objects/s recovering 2024-04-23T16:36:48.132 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:36:48.163 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:48.185 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:48.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:48.187 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:48.288 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:36:48.354 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:48.419 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:36:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:48.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:36:48.454 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:36:48.543 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:36:48.607 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:48.834 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:36:48.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:48.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:36:48.968 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:36:49.000 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:49.041 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:36:49.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:49.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:36:49.146 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:36:49.178 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:49.202 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:36:49.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:49.208 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:36:49.309 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:36:49.345 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:49.416 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:36:49.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:49.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:36:49.520 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:36:49.593 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:49.692 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:36:49.702 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:36:49.819 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:36:49.819 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:36:49.819 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:36:49.819 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:36:49.819 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:36:49.819 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:36:49.819 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:36:49.819 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:36:49.819 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:36:49.819 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:36:49.819 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:36:49.820 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:36:49.820 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:36:49.820 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:36:49.820 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:36:49.820 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:36:49.820 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:36:49.820 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:36:49.820 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:36:49.820 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:36:49.820 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:36:49.820 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:36:49.821 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:36:49.953 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:36:49.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:49.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:36:50.058 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:36:50.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:49 smithi103 ceph-mon[39629]: pgmap v1261: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 605 MiB data, 4.7 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 665 op/s; 1303/19352 objects degraded (6.733%); 9.7 MiB/s, 155 objects/s recovering 2024-04-23T16:36:50.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:49 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 1303/19352 objects degraded (6.733%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:50.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:49 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/833993738' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:50.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:50.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:50.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:50.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:50.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:50.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:50.088 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:50.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:49 smithi148 ceph-mon[44520]: pgmap v1261: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 605 MiB data, 4.7 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 665 op/s; 1303/19352 objects degraded (6.733%); 9.7 MiB/s, 155 objects/s recovering 2024-04-23T16:36:50.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:49 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 1303/19352 objects degraded (6.733%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:50.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:49 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/833993738' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:50.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:50.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:50.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:50.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:50.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:50.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:50.189 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:36:50.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:50.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:36:50.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:49 smithi188 ceph-mon[44101]: pgmap v1261: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 605 MiB data, 4.7 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 665 op/s; 1303/19352 objects degraded (6.733%); 9.7 MiB/s, 155 objects/s recovering 2024-04-23T16:36:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:49 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 1303/19352 objects degraded (6.733%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:49 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/833993738' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:50.292 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:50.348 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:50.390 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:50.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:50.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:50.503 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:50.564 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:50.722 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:50.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:50.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:50.756 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:36:50.756 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2024-04-23T16:36:50.756 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2024-04-23T16:36:50.756 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:36:50.756 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.756 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:70258 4.5 4:ae7ceeba:::benchmark_data_smithi188_101370_object70257:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e288)", 2024-04-23T16:36:50.757 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:38.898686+0000", 2024-04-23T16:36:50.757 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 551.85345897399998, 2024-04-23T16:36:50.757 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.074931969000000001, 2024-04-23T16:36:50.757 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.757 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.757 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:50.757 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:50.757 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:36:50.757 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:36:50.757 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 70258 2024-04-23T16:36:50.757 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.757 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.757 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.757 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.757 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.898686+0000", 2024-04-23T16:36:50.758 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.758 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.758 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.758 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.758 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.898686+0000", 2024-04-23T16:36:50.758 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.758 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.758 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.758 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.758 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.898688+0000", 2024-04-23T16:36:50.758 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.979e-06 2024-04-23T16:36:50.758 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.759 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.759 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.759 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.898740+0000", 2024-04-23T16:36:50.759 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2045000000000001e-05 2024-04-23T16:36:50.759 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.759 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.759 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.759 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.898749+0000", 2024-04-23T16:36:50.759 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7599999999999997e-06 2024-04-23T16:36:50.759 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.759 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.759 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.759 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.898757+0000", 2024-04-23T16:36:50.759 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4900000000000003e-06 2024-04-23T16:36:50.760 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.760 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.760 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.760 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.900642+0000", 2024-04-23T16:36:50.760 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0018855300000000001 2024-04-23T16:36:50.760 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.760 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.760 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.760 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.900757+0000", 2024-04-23T16:36:50.760 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000114824 2024-04-23T16:36:50.760 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.760 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.760 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:36:50.760 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.900890+0000", 2024-04-23T16:36:50.760 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013328099999999999 2024-04-23T16:36:50.761 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.761 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.761 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:50.761 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.906685+0000", 2024-04-23T16:36:50.761 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0057942200000000001 2024-04-23T16:36:50.761 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.761 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.761 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:50.761 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.973552+0000", 2024-04-23T16:36:50.761 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.066867001999999995 2024-04-23T16:36:50.761 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.761 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.761 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:50.761 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.973585+0000", 2024-04-23T16:36:50.761 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2818000000000003e-05 2024-04-23T16:36:50.762 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.762 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.762 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.762 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.973618+0000", 2024-04-23T16:36:50.762 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3019999999999999e-05 2024-04-23T16:36:50.762 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.762 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.762 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.762 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.762 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.762 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:70261 4.5 4:a9238a77:::benchmark_data_smithi188_101370_object70260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e288)", 2024-04-23T16:36:50.762 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:38.899453+0000", 2024-04-23T16:36:50.762 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 551.85269178299995, 2024-04-23T16:36:50.763 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.074320675000000003, 2024-04-23T16:36:50.763 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.763 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.763 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:50.763 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:50.763 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:36:50.763 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:36:50.763 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 70261 2024-04-23T16:36:50.763 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.763 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.763 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.763 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.764 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.899453+0000", 2024-04-23T16:36:50.764 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.764 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.764 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.764 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.764 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.899453+0000", 2024-04-23T16:36:50.764 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.764 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.764 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.764 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.764 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.899454+0000", 2024-04-23T16:36:50.764 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.437e-06 2024-04-23T16:36:50.764 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.765 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.765 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.765 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.899508+0000", 2024-04-23T16:36:50.765 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3582999999999998e-05 2024-04-23T16:36:50.765 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.765 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.765 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.765 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.899520+0000", 2024-04-23T16:36:50.765 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1950000000000001e-05 2024-04-23T16:36:50.765 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.765 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.765 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.765 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.899527+0000", 2024-04-23T16:36:50.765 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.396e-06 2024-04-23T16:36:50.765 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.766 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.766 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.766 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.902591+0000", 2024-04-23T16:36:50.766 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0030636909999999999 2024-04-23T16:36:50.766 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.766 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.766 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.766 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.902689+0000", 2024-04-23T16:36:50.766 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7959000000000002e-05 2024-04-23T16:36:50.766 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.766 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.766 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:36:50.766 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.902819+0000", 2024-04-23T16:36:50.766 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012998 2024-04-23T16:36:50.767 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.767 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.767 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:50.767 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.906689+0000", 2024-04-23T16:36:50.767 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0038702519999999998 2024-04-23T16:36:50.767 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.767 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.767 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:50.767 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.973673+0000", 2024-04-23T16:36:50.767 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.066984112999999998 2024-04-23T16:36:50.767 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.767 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.767 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:50.767 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.973731+0000", 2024-04-23T16:36:50.767 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7683000000000003e-05 2024-04-23T16:36:50.767 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.768 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.768 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.768 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.973773+0000", 2024-04-23T16:36:50.768 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2630999999999999e-05 2024-04-23T16:36:50.768 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.768 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.768 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.768 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.768 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.768 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:70264 4.5 4:a69a5e6e:::benchmark_data_smithi188_101370_object70263:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e288)", 2024-04-23T16:36:50.768 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:38.902021+0000", 2024-04-23T16:36:50.768 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 551.85012308099999, 2024-04-23T16:36:50.768 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.071839927999999997, 2024-04-23T16:36:50.768 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.769 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.769 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:50.769 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:50.769 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:36:50.769 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:36:50.769 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 70264 2024-04-23T16:36:50.769 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.769 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.769 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.769 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.769 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.902021+0000", 2024-04-23T16:36:50.769 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.769 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.769 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.769 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.770 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.902021+0000", 2024-04-23T16:36:50.770 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.770 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.770 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.770 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.770 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.902023+0000", 2024-04-23T16:36:50.770 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2729999999999999e-06 2024-04-23T16:36:50.770 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.770 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.770 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.770 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.902052+0000", 2024-04-23T16:36:50.770 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9053000000000001e-05 2024-04-23T16:36:50.770 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.771 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.771 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.771 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.902060+0000", 2024-04-23T16:36:50.771 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2500000000000006e-06 2024-04-23T16:36:50.771 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.771 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.771 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.771 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.902066+0000", 2024-04-23T16:36:50.771 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5790000000000003e-06 2024-04-23T16:36:50.771 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.771 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.771 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.771 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.904452+0000", 2024-04-23T16:36:50.772 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0023865090000000002 2024-04-23T16:36:50.772 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.772 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.772 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.772 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.904561+0000", 2024-04-23T16:36:50.772 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010875299999999999 2024-04-23T16:36:50.772 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.772 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.772 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:36:50.772 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.904694+0000", 2024-04-23T16:36:50.772 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000132743 2024-04-23T16:36:50.772 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.772 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.772 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:50.773 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.908497+0000", 2024-04-23T16:36:50.773 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0038033229999999999 2024-04-23T16:36:50.773 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.773 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.773 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:50.773 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.973808+0000", 2024-04-23T16:36:50.773 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.065311548999999997 2024-04-23T16:36:50.773 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.773 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.773 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:50.773 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.973837+0000", 2024-04-23T16:36:50.773 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8445999999999998e-05 2024-04-23T16:36:50.773 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.773 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.773 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.774 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.973861+0000", 2024-04-23T16:36:50.774 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4450000000000001e-05 2024-04-23T16:36:50.774 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.774 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.774 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.774 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.774 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.774 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:70270 4.5 4:a2c5ee34:::benchmark_data_smithi188_101370_object70269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e288)", 2024-04-23T16:36:50.774 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:38.905527+0000", 2024-04-23T16:36:50.774 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 551.846617606, 2024-04-23T16:36:50.774 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.072635020999999994, 2024-04-23T16:36:50.774 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.774 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.775 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:50.775 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:50.775 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:36:50.775 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:36:50.775 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 70270 2024-04-23T16:36:50.775 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.775 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.775 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.775 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.775 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.905527+0000", 2024-04-23T16:36:50.775 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.775 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.775 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.775 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.776 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.905527+0000", 2024-04-23T16:36:50.776 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.776 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.776 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.776 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.776 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.905528+0000", 2024-04-23T16:36:50.776 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4799999999999997e-07 2024-04-23T16:36:50.776 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.776 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.776 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.776 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.905597+0000", 2024-04-23T16:36:50.776 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9469999999999997e-05 2024-04-23T16:36:50.776 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.776 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.776 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.777 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.905605+0000", 2024-04-23T16:36:50.777 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7680000000000008e-06 2024-04-23T16:36:50.777 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.777 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.777 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.777 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.905615+0000", 2024-04-23T16:36:50.777 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9369999999999999e-06 2024-04-23T16:36:50.777 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.777 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.777 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.777 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.906709+0000", 2024-04-23T16:36:50.777 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0010942910000000001 2024-04-23T16:36:50.777 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.777 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.777 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.777 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.906783+0000", 2024-04-23T16:36:50.778 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3764999999999996e-05 2024-04-23T16:36:50.778 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.778 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.778 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:36:50.778 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.906860+0000", 2024-04-23T16:36:50.778 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6917999999999998e-05 2024-04-23T16:36:50.778 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.778 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.778 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:50.778 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.908591+0000", 2024-04-23T16:36:50.778 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0017309789999999999 2024-04-23T16:36:50.778 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.778 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.778 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:50.779 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.978065+0000", 2024-04-23T16:36:50.779 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.069474525999999995 2024-04-23T16:36:50.779 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.779 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.779 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:50.779 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.978121+0000", 2024-04-23T16:36:50.779 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5643000000000002e-05 2024-04-23T16:36:50.779 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.779 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.779 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.779 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:38.978162+0000", 2024-04-23T16:36:50.779 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0976000000000002e-05 2024-04-23T16:36:50.779 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.779 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.779 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.779 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.780 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.780 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.15486.0:72050 4.5 4:a40c3676:::benchmark_data_smithi188_101370_object72049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e289)", 2024-04-23T16:36:50.780 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:27:39.744552+0000", 2024-04-23T16:36:50.780 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 551.00759230599999, 2024-04-23T16:36:50.780 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.078171600999999993, 2024-04-23T16:36:50.780 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.780 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.780 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:50.780 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:50.780 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.15486", 2024-04-23T16:36:50.780 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3771379928", 2024-04-23T16:36:50.780 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 72050 2024-04-23T16:36:50.780 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.780 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.780 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.780 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.781 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.744552+0000", 2024-04-23T16:36:50.781 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.781 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.781 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.781 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.781 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.744552+0000", 2024-04-23T16:36:50.781 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.781 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.781 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.781 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.781 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.744554+0000", 2024-04-23T16:36:50.781 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7889999999999999e-06 2024-04-23T16:36:50.781 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.782 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.782 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.782 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.744598+0000", 2024-04-23T16:36:50.782 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4375000000000001e-05 2024-04-23T16:36:50.782 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.782 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.782 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.782 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.744608+0000", 2024-04-23T16:36:50.782 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9899999999999992e-06 2024-04-23T16:36:50.782 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.782 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.782 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.782 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.744614+0000", 2024-04-23T16:36:50.782 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6300000000000003e-06 2024-04-23T16:36:50.782 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.783 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.783 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.783 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.744661+0000", 2024-04-23T16:36:50.783 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6709000000000003e-05 2024-04-23T16:36:50.783 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.783 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.783 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.783 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.744757+0000", 2024-04-23T16:36:50.783 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6151999999999997e-05 2024-04-23T16:36:50.783 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.783 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.783 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:36:50.783 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.744855+0000", 2024-04-23T16:36:50.784 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7646999999999995e-05 2024-04-23T16:36:50.784 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.784 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.784 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:50.784 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.746687+0000", 2024-04-23T16:36:50.784 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0018320949999999999 2024-04-23T16:36:50.784 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.784 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.784 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:50.784 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.822629+0000", 2024-04-23T16:36:50.784 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.075942280000000001 2024-04-23T16:36:50.784 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.784 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.784 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:50.785 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.822686+0000", 2024-04-23T16:36:50.785 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6610999999999997e-05 2024-04-23T16:36:50.785 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.785 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.785 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.785 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:27:39.822724+0000", 2024-04-23T16:36:50.785 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8322999999999999e-05 2024-04-23T16:36:50.785 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.785 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.785 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.785 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.785 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.785 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.15486.0:105198 4.6 e295/293 4:623868bc:::benchmark_data_smithi188_101370_object105197:head v 295'6627, mlcod=281'131)", 2024-04-23T16:36:50.786 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:28:00.277692+0000", 2024-04-23T16:36:50.786 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 530.47445255900004, 2024-04-23T16:36:50.786 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.066640286000000007, 2024-04-23T16:36:50.786 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.786 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.786 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:50.786 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.786 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.786 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.786 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.277692+0000", 2024-04-23T16:36:50.786 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.786 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.786 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.786 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.787 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.277692+0000", 2024-04-23T16:36:50.787 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.787 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.787 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.787 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.787 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.277694+0000", 2024-04-23T16:36:50.787 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8109999999999999e-06 2024-04-23T16:36:50.787 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.787 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.787 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.787 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.277751+0000", 2024-04-23T16:36:50.787 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7343999999999998e-05 2024-04-23T16:36:50.787 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.787 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.788 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.788 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.277761+0000", 2024-04-23T16:36:50.788 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0267e-05 2024-04-23T16:36:50.788 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.788 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.788 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.788 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.277769+0000", 2024-04-23T16:36:50.788 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5229999999999998e-06 2024-04-23T16:36:50.788 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.788 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.788 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.788 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.324380+0000", 2024-04-23T16:36:50.788 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.046610780999999997 2024-04-23T16:36:50.788 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.788 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.789 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.789 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.324402+0000", 2024-04-23T16:36:50.789 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2171000000000002e-05 2024-04-23T16:36:50.789 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.789 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.789 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:50.789 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.344288+0000", 2024-04-23T16:36:50.789 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019885968 2024-04-23T16:36:50.789 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.789 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.789 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.789 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.344332+0000", 2024-04-23T16:36:50.789 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4421e-05 2024-04-23T16:36:50.789 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.789 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.790 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.790 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.790 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.790 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPush(4.6 295/293 [PushOp(4:6d6703c0:::benchmark_data_smithi188_101370_object2146:head, version: 281'134, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6d6703c0:::benchmark_data_smithi188_101370_object2146:head@281'134, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T16:36:50.790 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:28:00.311233+0000", 2024-04-23T16:36:50.790 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 530.44091113800005, 2024-04-23T16:36:50.790 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.111358305, 2024-04-23T16:36:50.790 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.790 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.790 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:36:50.790 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.790 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.790 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.790 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.311233+0000", 2024-04-23T16:36:50.790 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.790 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.791 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.791 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.791 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.311233+0000", 2024-04-23T16:36:50.791 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.791 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.791 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.791 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.791 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.311235+0000", 2024-04-23T16:36:50.791 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5009999999999999e-06 2024-04-23T16:36:50.791 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.791 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.791 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.791 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.311285+0000", 2024-04-23T16:36:50.791 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0590999999999997e-05 2024-04-23T16:36:50.791 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.791 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.792 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.792 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.311305+0000", 2024-04-23T16:36:50.792 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9247000000000001e-05 2024-04-23T16:36:50.792 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.792 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.792 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.792 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.311321+0000", 2024-04-23T16:36:50.792 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6240999999999999e-05 2024-04-23T16:36:50.792 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.792 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.792 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.792 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.420586+0000", 2024-04-23T16:36:50.792 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.10926459299999999 2024-04-23T16:36:50.792 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.792 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.792 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.793 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.420594+0000", 2024-04-23T16:36:50.793 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1070000000000005e-06 2024-04-23T16:36:50.793 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.793 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.793 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.793 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:00.422592+0000", 2024-04-23T16:36:50.793 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0019980250000000001 2024-04-23T16:36:50.793 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.793 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.793 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.793 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.793 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.793 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPush(4.6 297/293 [PushOp(4:6b111983:::benchmark_data_smithi188_101370_object4279:head, version: 281'273, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6b111983:::benchmark_data_smithi188_101370_object4279:head@281'273, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T16:36:50.793 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:28:05.437696+0000", 2024-04-23T16:36:50.793 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 525.31444866799995, 2024-04-23T16:36:50.793 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.068626678999999996, 2024-04-23T16:36:50.794 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.794 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.794 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:36:50.794 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.794 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.794 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.794 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:05.437696+0000", 2024-04-23T16:36:50.794 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.794 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.794 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.794 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.794 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:05.437696+0000", 2024-04-23T16:36:50.794 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.795 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.795 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.795 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.795 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:05.437697+0000", 2024-04-23T16:36:50.795 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7899999999999997e-07 2024-04-23T16:36:50.795 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.795 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.795 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.795 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:05.437749+0000", 2024-04-23T16:36:50.795 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1990999999999997e-05 2024-04-23T16:36:50.795 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.795 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.795 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.796 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:05.437769+0000", 2024-04-23T16:36:50.796 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0409000000000001e-05 2024-04-23T16:36:50.796 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.796 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.796 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.796 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:05.437787+0000", 2024-04-23T16:36:50.796 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8321e-05 2024-04-23T16:36:50.796 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.796 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.796 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.796 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:05.504146+0000", 2024-04-23T16:36:50.796 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.066358640999999996 2024-04-23T16:36:50.796 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.796 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.796 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.797 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:05.504164+0000", 2024-04-23T16:36:50.797 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.787e-05 2024-04-23T16:36:50.797 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.797 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.797 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.797 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:05.506323+0000", 2024-04-23T16:36:50.797 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021586679999999999 2024-04-23T16:36:50.797 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.797 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.797 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.797 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.797 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.797 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPush(4.6 298/293 [PushOp(4:64d5b26c:::benchmark_data_smithi188_101370_object5480:head, version: 281'356, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:64d5b26c:::benchmark_data_smithi188_101370_object5480:head@281'356, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T16:36:50.797 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:28:08.553825+0000", 2024-04-23T16:36:50.797 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 522.19831941400003, 2024-04-23T16:36:50.797 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.073229198999999995, 2024-04-23T16:36:50.798 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.798 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.798 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:36:50.798 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.798 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.798 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.798 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:08.553825+0000", 2024-04-23T16:36:50.798 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.798 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.798 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.798 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.798 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:08.553825+0000", 2024-04-23T16:36:50.798 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.798 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.799 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.799 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.799 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:08.553827+0000", 2024-04-23T16:36:50.799 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0420000000000001e-06 2024-04-23T16:36:50.799 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.799 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.799 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.799 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:08.553882+0000", 2024-04-23T16:36:50.799 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5133000000000002e-05 2024-04-23T16:36:50.799 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.799 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.799 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.799 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:08.553903+0000", 2024-04-23T16:36:50.799 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0823999999999999e-05 2024-04-23T16:36:50.800 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.800 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.800 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.800 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:08.553916+0000", 2024-04-23T16:36:50.800 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2645e-05 2024-04-23T16:36:50.800 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.800 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.800 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.800 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:08.624484+0000", 2024-04-23T16:36:50.800 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.070567932 2024-04-23T16:36:50.800 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.800 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.800 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.800 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:08.624505+0000", 2024-04-23T16:36:50.800 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1484999999999998e-05 2024-04-23T16:36:50.800 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.801 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.801 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.801 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:08.627054+0000", 2024-04-23T16:36:50.801 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025491379999999998 2024-04-23T16:36:50.801 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.801 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.801 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.801 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.801 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.801 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPush(4.6 311/293 [PushOp(4:6c219b0a:::benchmark_data_smithi188_101370_object16938:head, version: 282'1106, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6c219b0a:::benchmark_data_smithi188_101370_object16938:head@282'1106, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T16:36:50.801 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:28:34.443852+0000", 2024-04-23T16:36:50.801 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 496.30829302799998, 2024-04-23T16:36:50.801 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.10967393, 2024-04-23T16:36:50.801 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.801 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.801 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:36:50.801 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.802 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.802 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.802 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:34.443852+0000", 2024-04-23T16:36:50.802 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.802 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.802 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.802 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.802 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:34.443852+0000", 2024-04-23T16:36:50.802 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.802 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.802 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.802 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.802 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:34.443852+0000", 2024-04-23T16:36:50.802 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5300000000000002e-07 2024-04-23T16:36:50.803 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.803 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.803 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.803 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:34.443899+0000", 2024-04-23T16:36:50.803 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6904999999999999e-05 2024-04-23T16:36:50.803 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.803 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.803 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.803 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:34.443918+0000", 2024-04-23T16:36:50.803 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8258e-05 2024-04-23T16:36:50.803 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.803 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.803 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.803 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:34.443937+0000", 2024-04-23T16:36:50.804 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9046e-05 2024-04-23T16:36:50.804 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.804 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.804 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.804 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:34.538038+0000", 2024-04-23T16:36:50.804 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.094101406999999998 2024-04-23T16:36:50.804 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.804 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.804 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.804 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:34.538052+0000", 2024-04-23T16:36:50.804 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4122e-05 2024-04-23T16:36:50.804 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.804 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.804 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.804 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:34.553525+0000", 2024-04-23T16:36:50.804 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015473239 2024-04-23T16:36:50.805 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.805 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.805 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.805 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.805 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.805 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPush(4.6 314/293 [PushOp(4:6246f4d9:::benchmark_data_smithi188_101370_object20226:head, version: 283'1332, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6246f4d9:::benchmark_data_smithi188_101370_object20226:head@283'1332, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T16:36:50.805 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:28:41.952574+0000", 2024-04-23T16:36:50.805 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 488.799570389, 2024-04-23T16:36:50.805 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.097604073999999999, 2024-04-23T16:36:50.805 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.805 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.805 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:36:50.805 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.805 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.805 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.806 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:41.952574+0000", 2024-04-23T16:36:50.806 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.806 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.806 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.806 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.806 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:41.952574+0000", 2024-04-23T16:36:50.806 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.806 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.806 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.806 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.806 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:41.952575+0000", 2024-04-23T16:36:50.806 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.155e-06 2024-04-23T16:36:50.806 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.806 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.807 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.807 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:41.952622+0000", 2024-04-23T16:36:50.807 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6393000000000001e-05 2024-04-23T16:36:50.807 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.807 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.807 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.807 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:41.952643+0000", 2024-04-23T16:36:50.807 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1670000000000001e-05 2024-04-23T16:36:50.807 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.807 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.807 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.807 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:41.952659+0000", 2024-04-23T16:36:50.807 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5248e-05 2024-04-23T16:36:50.807 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.808 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.808 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.808 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.039224+0000", 2024-04-23T16:36:50.808 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.086565248999999997 2024-04-23T16:36:50.808 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.808 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.808 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.808 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.039237+0000", 2024-04-23T16:36:50.808 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2687000000000001e-05 2024-04-23T16:36:50.808 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.808 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.808 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.808 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.050178+0000", 2024-04-23T16:36:50.808 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010941672 2024-04-23T16:36:50.809 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.809 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.809 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.809 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.809 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.809 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPush(4.6 314/293 [PushOp(4:612d6aba:::benchmark_data_smithi188_101370_object20393:head, version: 283'1348, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:612d6aba:::benchmark_data_smithi188_101370_object20393:head@283'1348, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T16:36:50.809 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:28:42.683224+0000", 2024-04-23T16:36:50.809 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 488.06892102, 2024-04-23T16:36:50.809 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.072648131000000005, 2024-04-23T16:36:50.809 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.809 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.809 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:36:50.809 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.810 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.810 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.810 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.683224+0000", 2024-04-23T16:36:50.810 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.810 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.810 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.810 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.810 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.683224+0000", 2024-04-23T16:36:50.810 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.810 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.810 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.810 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.811 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.683224+0000", 2024-04-23T16:36:50.811 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8800000000000003e-07 2024-04-23T16:36:50.811 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.811 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.811 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.811 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.683284+0000", 2024-04-23T16:36:50.811 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0491000000000001e-05 2024-04-23T16:36:50.811 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.811 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.811 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.811 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.683301+0000", 2024-04-23T16:36:50.811 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6651999999999999e-05 2024-04-23T16:36:50.811 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.811 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.811 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.812 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.683315+0000", 2024-04-23T16:36:50.812 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3658e-05 2024-04-23T16:36:50.812 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.812 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.812 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.812 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.752838+0000", 2024-04-23T16:36:50.812 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.069522908999999994 2024-04-23T16:36:50.812 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.812 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.812 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.812 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.752851+0000", 2024-04-23T16:36:50.812 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3722e-05 2024-04-23T16:36:50.812 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.812 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.812 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.813 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:28:42.755872+0000", 2024-04-23T16:36:50.813 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0030202110000000001 2024-04-23T16:36:50.813 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.813 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.813 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.813 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.813 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.813 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:26106 5.2 5:461a31c8:::benchmark_data_smithi188_113957_object26105:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e369)", 2024-04-23T16:36:50.813 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:31:44.502808+0000", 2024-04-23T16:36:50.813 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 306.24933610300002, 2024-04-23T16:36:50.813 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.065193778999999993, 2024-04-23T16:36:50.813 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.813 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.813 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:50.813 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:50.814 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:36:50.814 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:36:50.814 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 26106 2024-04-23T16:36:50.814 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.814 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.814 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.814 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.814 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.502808+0000", 2024-04-23T16:36:50.814 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.814 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.814 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.814 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.814 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.502808+0000", 2024-04-23T16:36:50.815 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.815 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.815 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.815 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.815 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.502810+0000", 2024-04-23T16:36:50.815 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1739999999999999e-06 2024-04-23T16:36:50.815 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.815 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.815 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.815 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.503950+0000", 2024-04-23T16:36:50.815 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0011402719999999999 2024-04-23T16:36:50.815 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.815 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.815 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.815 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.503962+0000", 2024-04-23T16:36:50.816 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2483999999999999e-05 2024-04-23T16:36:50.816 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.816 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.816 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.816 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.503971+0000", 2024-04-23T16:36:50.816 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3950000000000001e-06 2024-04-23T16:36:50.816 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.816 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.816 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.816 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.504260+0000", 2024-04-23T16:36:50.816 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00028959999999999999 2024-04-23T16:36:50.816 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.816 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.816 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.817 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.504348+0000", 2024-04-23T16:36:50.817 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7793999999999993e-05 2024-04-23T16:36:50.817 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.817 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.817 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:36:50.817 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.504452+0000", 2024-04-23T16:36:50.817 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000104307 2024-04-23T16:36:50.817 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.817 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.817 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:50.817 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.507204+0000", 2024-04-23T16:36:50.817 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002751093 2024-04-23T16:36:50.817 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.817 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.817 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:50.817 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.567809+0000", 2024-04-23T16:36:50.818 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.060605277999999999 2024-04-23T16:36:50.818 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.818 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.818 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:50.818 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.567958+0000", 2024-04-23T16:36:50.818 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014895699999999999 2024-04-23T16:36:50.818 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.818 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.818 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.818 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.568002+0000", 2024-04-23T16:36:50.818 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4425000000000002e-05 2024-04-23T16:36:50.818 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.818 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.818 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.819 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.819 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.819 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:26117 5.2 5:47254ac5:::benchmark_data_smithi188_113957_object26116:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e369)", 2024-04-23T16:36:50.819 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:31:44.505370+0000", 2024-04-23T16:36:50.819 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 306.24677406400002, 2024-04-23T16:36:50.819 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.064646837999999998, 2024-04-23T16:36:50.819 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.819 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.819 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:50.819 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:50.819 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:36:50.819 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:36:50.819 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 26117 2024-04-23T16:36:50.819 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.820 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.820 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.820 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.820 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505370+0000", 2024-04-23T16:36:50.820 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.820 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.820 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.820 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.820 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505370+0000", 2024-04-23T16:36:50.820 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.820 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.820 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.820 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.820 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505372+0000", 2024-04-23T16:36:50.821 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4279999999999999e-06 2024-04-23T16:36:50.821 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.821 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.821 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.821 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505397+0000", 2024-04-23T16:36:50.821 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5253999999999999e-05 2024-04-23T16:36:50.821 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.821 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.821 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.821 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505405+0000", 2024-04-23T16:36:50.821 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1079999999999993e-06 2024-04-23T16:36:50.821 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.821 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.821 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.822 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505412+0000", 2024-04-23T16:36:50.822 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7819999999999996e-06 2024-04-23T16:36:50.822 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.822 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.822 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.822 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505472+0000", 2024-04-23T16:36:50.822 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0075000000000003e-05 2024-04-23T16:36:50.822 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.822 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.822 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.822 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505587+0000", 2024-04-23T16:36:50.822 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000115147 2024-04-23T16:36:50.822 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.823 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.823 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:36:50.823 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505706+0000", 2024-04-23T16:36:50.823 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000118366 2024-04-23T16:36:50.823 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.823 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.823 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:50.823 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.507946+0000", 2024-04-23T16:36:50.823 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0022402659999999999 2024-04-23T16:36:50.823 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.823 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.823 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:50.823 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.569916+0000", 2024-04-23T16:36:50.823 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.061970372000000003 2024-04-23T16:36:50.824 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.824 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.824 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:50.824 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.569976+0000", 2024-04-23T16:36:50.824 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9670000000000003e-05 2024-04-23T16:36:50.824 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.824 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.824 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.824 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.570017+0000", 2024-04-23T16:36:50.824 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1369999999999999e-05 2024-04-23T16:36:50.824 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.824 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.824 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.824 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.825 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.825 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:26124 5.2 5:4ffdf5ef:::benchmark_data_smithi188_113957_object26123:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e369)", 2024-04-23T16:36:50.825 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:31:44.505687+0000", 2024-04-23T16:36:50.825 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 306.246457405, 2024-04-23T16:36:50.825 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.064651243999999997, 2024-04-23T16:36:50.825 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.825 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.825 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:50.825 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:50.825 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:36:50.825 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:36:50.825 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 26124 2024-04-23T16:36:50.825 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.826 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.826 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.826 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.826 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505687+0000", 2024-04-23T16:36:50.826 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.826 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.826 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.826 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.826 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505687+0000", 2024-04-23T16:36:50.826 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.826 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.826 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.826 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.826 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505688+0000", 2024-04-23T16:36:50.827 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.341e-06 2024-04-23T16:36:50.827 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.827 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.827 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.827 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505779+0000", 2024-04-23T16:36:50.827 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0470000000000006e-05 2024-04-23T16:36:50.827 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.827 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.827 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.827 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505789+0000", 2024-04-23T16:36:50.827 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0052e-05 2024-04-23T16:36:50.827 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.827 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.827 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.827 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.505797+0000", 2024-04-23T16:36:50.828 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5870000000000004e-06 2024-04-23T16:36:50.828 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.828 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.828 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.828 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.506467+0000", 2024-04-23T16:36:50.828 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00067056699999999995 2024-04-23T16:36:50.828 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.828 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.828 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.828 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.506580+0000", 2024-04-23T16:36:50.828 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000113348 2024-04-23T16:36:50.828 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.828 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.828 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:36:50.828 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.506721+0000", 2024-04-23T16:36:50.829 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014030999999999999 2024-04-23T16:36:50.829 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.829 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.829 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:50.829 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.507959+0000", 2024-04-23T16:36:50.829 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012383299999999999 2024-04-23T16:36:50.829 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.829 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.829 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:50.829 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.570235+0000", 2024-04-23T16:36:50.829 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.062276146999999997 2024-04-23T16:36:50.829 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.829 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.829 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:50.830 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.570291+0000", 2024-04-23T16:36:50.830 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5427000000000003e-05 2024-04-23T16:36:50.830 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.830 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.830 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.830 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:31:44.570338+0000", 2024-04-23T16:36:50.830 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7664999999999998e-05 2024-04-23T16:36:50.830 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.830 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.830 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.830 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.830 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.830 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:145609 5.3 e385/359 5:ce32bf88:::benchmark_data_smithi188_113957_object145608:head v 385'10187, mlcod=385'10182)", 2024-04-23T16:36:50.831 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:32:51.714951+0000", 2024-04-23T16:36:50.831 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 239.037194027, 2024-04-23T16:36:50.831 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052001789, 2024-04-23T16:36:50.831 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.831 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.831 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:50.831 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.831 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.831 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.831 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.714951+0000", 2024-04-23T16:36:50.831 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.831 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.831 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.831 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.831 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.714951+0000", 2024-04-23T16:36:50.831 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.832 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.832 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.832 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.832 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.714952+0000", 2024-04-23T16:36:50.832 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7540000000000001e-06 2024-04-23T16:36:50.832 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.832 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.832 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.832 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.715057+0000", 2024-04-23T16:36:50.832 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000104467 2024-04-23T16:36:50.832 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.832 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.832 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.832 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.715068+0000", 2024-04-23T16:36:50.832 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1758e-05 2024-04-23T16:36:50.832 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.833 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.833 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.833 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.715076+0000", 2024-04-23T16:36:50.833 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.926e-06 2024-04-23T16:36:50.833 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.833 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.833 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.833 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.717231+0000", 2024-04-23T16:36:50.833 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021546489999999998 2024-04-23T16:36:50.833 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.833 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.833 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.833 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.717248+0000", 2024-04-23T16:36:50.833 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7365000000000001e-05 2024-04-23T16:36:50.833 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.833 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.834 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:50.834 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.766907+0000", 2024-04-23T16:36:50.834 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.049658901999999998 2024-04-23T16:36:50.834 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.834 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.834 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.834 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:51.766952+0000", 2024-04-23T16:36:50.834 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4968e-05 2024-04-23T16:36:50.834 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.834 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.834 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.834 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.834 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.834 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:146469 5.2 5:432c4c0b:::benchmark_data_smithi188_113957_object146468:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-23T16:36:50.834 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:32:52.553527+0000", 2024-04-23T16:36:50.835 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 238.19861795200001, 2024-04-23T16:36:50.835 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.051413853000000002, 2024-04-23T16:36:50.835 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.835 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.835 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:50.835 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:50.835 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:36:50.835 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:36:50.835 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 146469 2024-04-23T16:36:50.835 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.835 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.835 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.835 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.835 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.553527+0000", 2024-04-23T16:36:50.835 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.835 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.836 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.836 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.836 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.553527+0000", 2024-04-23T16:36:50.836 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.836 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.836 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.836 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.836 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.553530+0000", 2024-04-23T16:36:50.836 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0520000000000002e-06 2024-04-23T16:36:50.836 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.836 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.836 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.836 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.553606+0000", 2024-04-23T16:36:50.836 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6798000000000006e-05 2024-04-23T16:36:50.836 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.836 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.837 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.837 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.553618+0000", 2024-04-23T16:36:50.837 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1508999999999999e-05 2024-04-23T16:36:50.837 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.837 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.837 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.837 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.553627+0000", 2024-04-23T16:36:50.837 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0469999999999999e-06 2024-04-23T16:36:50.837 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.837 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.837 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.837 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.553708+0000", 2024-04-23T16:36:50.837 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0921999999999995e-05 2024-04-23T16:36:50.837 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.837 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.838 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.838 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.553879+0000", 2024-04-23T16:36:50.838 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000171423 2024-04-23T16:36:50.838 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.838 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.838 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:36:50.838 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.554038+0000", 2024-04-23T16:36:50.838 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015865000000000001 2024-04-23T16:36:50.838 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.838 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.838 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:50.838 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.558576+0000", 2024-04-23T16:36:50.838 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045381409999999999 2024-04-23T16:36:50.838 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.839 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.839 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:50.839 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.604819+0000", 2024-04-23T16:36:50.839 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.046243228999999997 2024-04-23T16:36:50.839 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.839 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.839 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:50.839 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.604889+0000", 2024-04-23T16:36:50.839 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9470999999999999e-05 2024-04-23T16:36:50.839 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.839 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.839 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.839 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:52.604940+0000", 2024-04-23T16:36:50.839 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1610999999999998e-05 2024-04-23T16:36:50.839 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.840 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.840 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.840 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.840 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.840 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:148177 5.3 e386/359 5:caa34e22:::benchmark_data_smithi188_113957_object148176:head v 386'10606, mlcod=386'10603)", 2024-04-23T16:36:50.840 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:32:54.408471+0000", 2024-04-23T16:36:50.840 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 236.343673647, 2024-04-23T16:36:50.840 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.051610369000000003, 2024-04-23T16:36:50.840 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.840 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.840 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:50.840 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.840 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.840 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.841 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.408471+0000", 2024-04-23T16:36:50.841 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.841 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.841 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.841 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.841 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.408471+0000", 2024-04-23T16:36:50.841 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.841 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.841 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.841 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.841 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.408473+0000", 2024-04-23T16:36:50.841 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1619999999999998e-06 2024-04-23T16:36:50.841 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.841 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.841 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.841 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.408504+0000", 2024-04-23T16:36:50.842 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1285999999999998e-05 2024-04-23T16:36:50.842 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.842 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.842 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.842 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.408515+0000", 2024-04-23T16:36:50.842 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0531e-05 2024-04-23T16:36:50.842 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.842 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.842 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.842 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.408523+0000", 2024-04-23T16:36:50.842 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.072e-06 2024-04-23T16:36:50.842 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.842 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.842 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.843 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.433140+0000", 2024-04-23T16:36:50.843 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.024616842 2024-04-23T16:36:50.843 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.843 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.843 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.843 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.433159+0000", 2024-04-23T16:36:50.843 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8978e-05 2024-04-23T16:36:50.843 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.843 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.843 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:50.843 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.460041+0000", 2024-04-23T16:36:50.843 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026882593 2024-04-23T16:36:50.843 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.843 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.843 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.843 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:54.460081+0000", 2024-04-23T16:36:50.844 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9904999999999998e-05 2024-04-23T16:36:50.844 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.844 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.844 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.844 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.844 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.844 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:151546 5.2 5:497e7a6e:::benchmark_data_smithi188_113957_object151545:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-23T16:36:50.844 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:32:57.686710+0000", 2024-04-23T16:36:50.844 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 233.06543429499999, 2024-04-23T16:36:50.844 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.067105949999999998, 2024-04-23T16:36:50.844 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.844 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.844 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:50.844 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:50.844 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:36:50.845 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:36:50.845 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 151546 2024-04-23T16:36:50.845 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.845 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.845 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.845 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.845 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.686710+0000", 2024-04-23T16:36:50.845 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.845 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.845 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.845 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.845 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.686710+0000", 2024-04-23T16:36:50.845 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.845 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.845 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.846 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.846 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.686713+0000", 2024-04-23T16:36:50.846 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3530000000000002e-06 2024-04-23T16:36:50.846 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.846 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.846 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.846 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.686768+0000", 2024-04-23T16:36:50.846 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5173999999999997e-05 2024-04-23T16:36:50.846 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.846 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.846 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.846 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.686780+0000", 2024-04-23T16:36:50.846 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2160999999999999e-05 2024-04-23T16:36:50.846 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.846 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.847 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.847 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.686788+0000", 2024-04-23T16:36:50.847 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7559999999999999e-06 2024-04-23T16:36:50.847 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.847 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.847 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.847 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.686846+0000", 2024-04-23T16:36:50.847 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8121000000000001e-05 2024-04-23T16:36:50.847 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.847 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.847 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.847 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.686974+0000", 2024-04-23T16:36:50.847 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012791199999999999 2024-04-23T16:36:50.847 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.847 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.848 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:36:50.848 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.687128+0000", 2024-04-23T16:36:50.848 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015439699999999999 2024-04-23T16:36:50.848 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.848 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.848 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:50.848 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.691955+0000", 2024-04-23T16:36:50.848 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0048266560000000003 2024-04-23T16:36:50.848 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.848 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.848 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:50.848 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.753708+0000", 2024-04-23T16:36:50.848 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.061752870000000001 2024-04-23T16:36:50.848 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.848 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.848 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:50.849 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.753766+0000", 2024-04-23T16:36:50.849 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8745000000000001e-05 2024-04-23T16:36:50.849 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.849 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.849 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.849 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:32:57.753816+0000", 2024-04-23T16:36:50.849 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9805000000000002e-05 2024-04-23T16:36:50.849 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.849 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.849 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.849 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.849 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.849 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:285806 5.2 5:4425f7dc:::benchmark_data_smithi188_113957_object132208:head [delete] snapc 0=[] ondisk+write+known_if_redirected e439)", 2024-04-23T16:36:50.849 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:36:31.960218+0000", 2024-04-23T16:36:50.849 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 18.791926384, 2024-04-23T16:36:50.849 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.047483161000000003, 2024-04-23T16:36:50.850 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:36:50.850 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:36:50.850 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:36:50.850 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:36:50.850 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:36:50.850 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:36:50.850 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 285806 2024-04-23T16:36:50.850 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.850 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:36:50.850 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.850 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:36:50.850 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:31.960218+0000", 2024-04-23T16:36:50.850 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.850 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.850 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.851 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:36:50.851 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:31.960218+0000", 2024-04-23T16:36:50.851 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:36:50.851 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.851 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.851 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:36:50.851 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:31.960219+0000", 2024-04-23T16:36:50.851 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1800000000000005e-07 2024-04-23T16:36:50.851 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.851 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.851 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:36:50.851 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:31.960222+0000", 2024-04-23T16:36:50.851 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9890000000000001e-06 2024-04-23T16:36:50.851 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.851 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.851 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:36:50.852 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:31.960228+0000", 2024-04-23T16:36:50.852 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7699999999999998e-06 2024-04-23T16:36:50.852 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.852 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.852 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:36:50.852 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:31.960233+0000", 2024-04-23T16:36:50.852 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2460000000000003e-06 2024-04-23T16:36:50.852 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.852 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.852 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:36:50.852 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:31.974276+0000", 2024-04-23T16:36:50.852 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01404296 2024-04-23T16:36:50.852 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.852 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.852 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:36:50.852 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:31.974380+0000", 2024-04-23T16:36:50.852 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000104632 2024-04-23T16:36:50.853 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.853 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.853 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:36:50.853 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:31.974502+0000", 2024-04-23T16:36:50.853 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000121971 2024-04-23T16:36:50.853 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.853 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.853 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:36:50.853 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:31.990907+0000", 2024-04-23T16:36:50.853 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016404161 2024-04-23T16:36:50.853 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.853 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.853 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:36:50.853 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:32.007639+0000", 2024-04-23T16:36:50.853 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016732710000000001 2024-04-23T16:36:50.854 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.854 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.854 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:36:50.854 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:32.007672+0000", 2024-04-23T16:36:50.854 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2886000000000003e-05 2024-04-23T16:36:50.854 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:36:50.854 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:36:50.854 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:36:50.854 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:32.007701+0000", 2024-04-23T16:36:50.854 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9118000000000001e-05 2024-04-23T16:36:50.854 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.854 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.854 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.854 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:36:50.855 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:36:50.855 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:36:50.855 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:50.855 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:36:50.915 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:51.027 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:51.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:51.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:51.035 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:51.036 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:51.036 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:51.036 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:51.036 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:51.036 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:50 smithi148 ceph-mon[44520]: from='client.35927 v1:172.21.15.103:0/1780373821' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:36:51.036 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:50 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2883177283' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:36:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:50 smithi103 ceph-mon[39629]: from='client.35927 v1:172.21.15.103:0/1780373821' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:36:51.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:50 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2883177283' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:36:51.136 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:36:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:50 smithi188 ceph-mon[44101]: from='client.35927 v1:172.21.15.103:0/1780373821' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:36:51.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:50 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2883177283' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:36:51.166 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:51.190 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:36:51.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:51.193 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:36:51.290 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:36:51.290 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:36:51.294 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:36:51.321 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:51.343 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:36:51.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:51.346 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:36:51.447 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:36:51.480 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:51.512 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:36:51.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:51.515 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:36:51.616 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:36:51.647 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:51.668 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:36:51.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:51.675 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:36:51.776 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:36:51.803 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:51.827 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:36:51.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:51.830 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:36:51.931 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:36:51.959 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:52.109 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:36:52.111 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:51 smithi103 ceph-mon[39629]: pgmap v1262: 27 pgs: 2 active+recovering+degraded, 25 active+clean; 370 MiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 714/11822 objects degraded (6.040%); 14 MiB/s, 229 objects/s recovering 2024-04-23T16:36:52.112 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:51 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:36:52.112 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:51 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2883177283' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:36:52.112 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:51 smithi103 ceph-mon[39629]: osdmap e445: 12 total, 11 up, 9 in 2024-04-23T16:36:52.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:52.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:36:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:51 smithi148 ceph-mon[44520]: pgmap v1262: 27 pgs: 2 active+recovering+degraded, 25 active+clean; 370 MiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 714/11822 objects degraded (6.040%); 14 MiB/s, 229 objects/s recovering 2024-04-23T16:36:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:51 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:36:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:51 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2883177283' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:36:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:51 smithi148 ceph-mon[44520]: osdmap e445: 12 total, 11 up, 9 in 2024-04-23T16:36:52.130 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1261,"stamp":"2024-04-23T16:36:48.748327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":634520120,"num_objects":9676,"num_object_clones":0,"num_object_copies":19352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9676,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":297518,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27650,"num_bytes_recovered":1812206112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162178,"ondisk_log_size":162178,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":4884960,"kb_used_data":1248836,"kb_used_omap":14,"kb_used_meta":3636017,"kb_avail":838821408,"statfs":{"total":863955320832,"available":858953121792,"internally_reserved":0,"allocated":1278808064,"data_stored":1274324468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3723282415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-348913664,"num_objects":-5324,"num_object_clones":0,"num_object_copies":-10648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5324,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1247,"num_bytes_recovered":81723392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001665"},"pg_stats":[{"pgid":"5.f","version":"444'18757","reported_seq":19041,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-04-23T16:36:45.293630+0000","last_change":"2024-04-23T16:33:17.526419+0000","last_active":"2024-04-23T16:36:45.293630+0000","last_peered":"2024-04-23T16:36:45.293630+0000","last_clean":"2024-04-23T16:36:45.293630+0000","last_became_active":"2024-04-23T16:31:29.140923+0000","last_became_peered":"2024-04-23T16:31:29.140923+0000","last_unstale":"2024-04-23T16:36:45.293630+0000","last_undegraded":"2024-04-23T16:36:45.293630+0000","last_fullsized":"2024-04-23T16:36:45.293630+0000","mapping_epoch":359,"log_start":"382'8700","ondisk_log_start":"382'8700","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9683","last_scrub_stamp":"2024-04-23T16:33:17.526283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.526283+0000","objects_scrubbed":9683,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39911424,"num_objects":609,"num_object_clones":0,"num_object_copies":1218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18757,"num_write_kb":619712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"444'18577","reported_seq":18987,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-04-23T16:36:44.320979+0000","last_change":"2024-04-23T16:35:26.407358+0000","last_active":"2024-04-23T16:36:44.320979+0000","last_peered":"2024-04-23T16:36:44.320979+0000","last_clean":"2024-04-23T16:36:44.320979+0000","last_became_active":"2024-04-23T16:31:29.137890+0000","last_became_peered":"2024-04-23T16:31:29.137890+0000","last_unstale":"2024-04-23T16:36:44.320979+0000","last_undegraded":"2024-04-23T16:36:44.320979+0000","last_fullsized":"2024-04-23T16:36:44.320979+0000","mapping_epoch":359,"log_start":"382'8500","ondisk_log_start":"382'8500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'14582","last_scrub_stamp":"2024-04-23T16:35:26.407093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:26.407093+0000","objects_scrubbed":4692,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":544,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45023232,"num_objects":687,"num_object_clones":0,"num_object_copies":1374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18577,"num_write_kb":616448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"444'19023","reported_seq":37924,"reported_epoch":444,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:36:48.241216+0000","last_change":"2024-04-23T16:33:26.432192+0000","last_active":"2024-04-23T16:36:48.241216+0000","last_peered":"2024-04-23T16:36:48.241216+0000","last_clean":"2024-04-23T16:32:59.749530+0000","last_became_active":"2024-04-23T16:33:26.338644+0000","last_became_peered":"2024-04-23T16:33:26.338644+0000","last_unstale":"2024-04-23T16:36:48.241216+0000","last_undegraded":"2024-04-23T16:33:25.874847+0000","last_fullsized":"2024-04-23T16:36:48.241216+0000","mapping_epoch":394,"log_start":"383'9000","ondisk_log_start":"383'9000","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8711","last_scrub_stamp":"2024-04-23T16:32:43.928914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.928914+0000","objects_scrubbed":8680,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27459584,"num_objects":419,"num_object_clones":0,"num_object_copies":838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19023,"num_write_kb":622144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9310,"num_bytes_recovered":610140160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":["8"],"object_location_counts":[{"shards":"4,8","objects":17},{"shards":"8","objects":402}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.9","version":"444'18701","reported_seq":18970,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-04-23T16:36:45.247576+0000","last_change":"2024-04-23T16:32:37.256944+0000","last_active":"2024-04-23T16:36:45.247576+0000","last_peered":"2024-04-23T16:36:45.247576+0000","last_clean":"2024-04-23T16:36:45.247576+0000","last_became_active":"2024-04-23T16:31:29.141551+0000","last_became_peered":"2024-04-23T16:31:29.141551+0000","last_unstale":"2024-04-23T16:36:45.247576+0000","last_undegraded":"2024-04-23T16:36:45.247576+0000","last_fullsized":"2024-04-23T16:36:45.247576+0000","mapping_epoch":359,"log_start":"382'8700","ondisk_log_start":"382'8700","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8156","last_scrub_stamp":"2024-04-23T16:32:37.256787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:37.256787+0000","objects_scrubbed":8082,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40566784,"num_objects":619,"num_object_clones":0,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18701,"num_write_kb":618240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"444'18228","reported_seq":18610,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-04-23T16:36:46.801062+0000","last_change":"2024-04-23T16:36:20.471641+0000","last_active":"2024-04-23T16:36:46.801062+0000","last_peered":"2024-04-23T16:36:46.801062+0000","last_clean":"2024-04-23T16:36:46.801062+0000","last_became_active":"2024-04-23T16:31:29.139045+0000","last_became_peered":"2024-04-23T16:31:29.139045+0000","last_unstale":"2024-04-23T16:36:46.801062+0000","last_undegraded":"2024-04-23T16:36:46.801062+0000","last_fullsized":"2024-04-23T16:36:46.801062+0000","mapping_epoch":359,"log_start":"382'8200","ondisk_log_start":"382'8200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'16781","last_scrub_stamp":"2024-04-23T16:36:20.471467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:36:20.471467+0000","objects_scrubbed":1958,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:48.916510+0000","scrub_duration":248,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33161216,"num_objects":506,"num_object_clones":0,"num_object_copies":1012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18228,"num_write_kb":599488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"444'22066","reported_seq":22547,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-04-23T16:36:44.328742+0000","last_change":"2024-04-23T16:35:32.474057+0000","last_active":"2024-04-23T16:36:44.328742+0000","last_peered":"2024-04-23T16:36:44.328742+0000","last_clean":"2024-04-23T16:36:44.328742+0000","last_became_active":"2024-04-23T16:31:29.310906+0000","last_became_peered":"2024-04-23T16:31:29.310906+0000","last_unstale":"2024-04-23T16:36:44.328742+0000","last_undegraded":"2024-04-23T16:36:44.328742+0000","last_fullsized":"2024-04-23T16:36:44.328742+0000","mapping_epoch":359,"log_start":"409'12004","ondisk_log_start":"409'12004","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'18441","last_scrub_stamp":"2024-04-23T16:35:32.473828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:32.473828+0000","objects_scrubbed":4251,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:57.506974+0000","scrub_duration":548,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40304640,"num_objects":615,"num_object_clones":0,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18505,"num_write_kb":611840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1158,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-04-23T16:36:44.217776+0000","last_change":"2024-04-23T16:36:19.132015+0000","last_active":"2024-04-23T16:36:44.217776+0000","last_peered":"2024-04-23T16:36:44.217776+0000","last_clean":"2024-04-23T16:36:44.217776+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:36:44.217776+0000","last_undegraded":"2024-04-23T16:36:44.217776+0000","last_fullsized":"2024-04-23T16:36:44.217776+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:19.131883+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:19.131883+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:41.022603+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"444'18853","reported_seq":19121,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-04-23T16:36:44.664913+0000","last_change":"2024-04-23T16:32:38.103756+0000","last_active":"2024-04-23T16:36:44.664913+0000","last_peered":"2024-04-23T16:36:44.664913+0000","last_clean":"2024-04-23T16:36:44.664913+0000","last_became_active":"2024-04-23T16:31:29.141640+0000","last_became_peered":"2024-04-23T16:31:29.141640+0000","last_unstale":"2024-04-23T16:36:44.664913+0000","last_undegraded":"2024-04-23T16:36:44.664913+0000","last_fullsized":"2024-04-23T16:36:44.664913+0000","mapping_epoch":359,"log_start":"382'8800","ondisk_log_start":"382'8800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8300","last_scrub_stamp":"2024-04-23T16:32:38.103547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:38.103547+0000","objects_scrubbed":8231,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39518208,"num_objects":603,"num_object_clones":0,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18853,"num_write_kb":622592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1144,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-04-23T16:36:44.215185+0000","last_change":"2024-04-23T16:33:25.851379+0000","last_active":"2024-04-23T16:36:44.215185+0000","last_peered":"2024-04-23T16:36:44.215185+0000","last_clean":"2024-04-23T16:36:44.215185+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:36:44.215185+0000","last_undegraded":"2024-04-23T16:36:44.215185+0000","last_fullsized":"2024-04-23T16:36:44.215185+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.498820+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.498820+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"444'18607","reported_seq":18991,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-04-23T16:36:46.806280+0000","last_change":"2024-04-23T16:36:24.521917+0000","last_active":"2024-04-23T16:36:46.806280+0000","last_peered":"2024-04-23T16:36:46.806280+0000","last_clean":"2024-04-23T16:36:46.806280+0000","last_became_active":"2024-04-23T16:31:29.140251+0000","last_became_peered":"2024-04-23T16:31:29.140251+0000","last_unstale":"2024-04-23T16:36:46.806280+0000","last_undegraded":"2024-04-23T16:36:46.806280+0000","last_fullsized":"2024-04-23T16:36:46.806280+0000","mapping_epoch":359,"log_start":"382'8600","ondisk_log_start":"382'8600","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'17367","last_scrub_stamp":"2024-04-23T16:36:24.521728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:36:24.521728+0000","objects_scrubbed":1767,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:41.002308+0000","scrub_duration":288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34340888,"num_objects":525,"num_object_clones":0,"num_object_copies":1050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":525,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18607,"num_write_kb":612161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1135,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-04-23T16:36:44.225313+0000","last_change":"2024-04-23T16:32:29.517947+0000","last_active":"2024-04-23T16:36:44.225313+0000","last_peered":"2024-04-23T16:36:44.225313+0000","last_clean":"2024-04-23T16:36:44.225313+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:36:44.225313+0000","last_undegraded":"2024-04-23T16:36:44.225313+0000","last_fullsized":"2024-04-23T16:36:44.225313+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:32:29.517800+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:32:29.517800+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"444'18777","reported_seq":37250,"reported_epoch":444,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:36:45.321108+0000","last_change":"2024-04-23T16:33:26.507399+0000","last_active":"2024-04-23T16:36:45.321108+0000","last_peered":"2024-04-23T16:36:45.321108+0000","last_clean":"2024-04-23T16:32:59.715575+0000","last_became_active":"2024-04-23T16:33:26.458611+0000","last_became_peered":"2024-04-23T16:33:26.458611+0000","last_unstale":"2024-04-23T16:36:45.321108+0000","last_undegraded":"2024-04-23T16:33:25.895472+0000","last_fullsized":"2024-04-23T16:36:45.321108+0000","mapping_epoch":394,"log_start":"382'8700","ondisk_log_start":"382'8700","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8607","last_scrub_stamp":"2024-04-23T16:32:43.130806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.130806+0000","objects_scrubbed":8577,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39518208,"num_objects":603,"num_object_clones":0,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18777,"num_write_kb":620160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9097,"num_bytes_recovered":596180992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":590},{"shards":"5,7","objects":13}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1155,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-04-23T16:36:43.811361+0000","last_change":"2024-04-23T16:36:26.456131+0000","last_active":"2024-04-23T16:36:43.811361+0000","last_peered":"2024-04-23T16:36:43.811361+0000","last_clean":"2024-04-23T16:36:43.811361+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:36:43.811361+0000","last_undegraded":"2024-04-23T16:36:43.811361+0000","last_fullsized":"2024-04-23T16:36:43.811361+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:26.456003+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:26.456003+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:44.948237+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"444'18333","reported_seq":18615,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-04-23T16:36:45.314837+0000","last_change":"2024-04-23T16:33:18.576077+0000","last_active":"2024-04-23T16:36:45.314837+0000","last_peered":"2024-04-23T16:36:45.314837+0000","last_clean":"2024-04-23T16:36:45.314837+0000","last_became_active":"2024-04-23T16:31:29.139709+0000","last_became_peered":"2024-04-23T16:31:29.139709+0000","last_unstale":"2024-04-23T16:36:45.314837+0000","last_undegraded":"2024-04-23T16:36:45.314837+0000","last_fullsized":"2024-04-23T16:36:45.314837+0000","mapping_epoch":359,"log_start":"382'8300","ondisk_log_start":"382'8300","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9472","last_scrub_stamp":"2024-04-23T16:33:18.575945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.575945+0000","objects_scrubbed":9472,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40042496,"num_objects":611,"num_object_clones":0,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18333,"num_write_kb":606208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1149,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-04-23T16:36:43.819415+0000","last_change":"2024-04-23T16:35:34.878630+0000","last_active":"2024-04-23T16:36:43.819415+0000","last_peered":"2024-04-23T16:36:43.819415+0000","last_clean":"2024-04-23T16:36:43.819415+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:36:43.819415+0000","last_undegraded":"2024-04-23T16:36:43.819415+0000","last_fullsized":"2024-04-23T16:36:43.819415+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:35:34.878536+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:35:34.878536+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:52.503403+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"443'18729","reported_seq":19116,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-04-23T16:36:44.166758+0000","last_change":"2024-04-23T16:36:20.849660+0000","last_active":"2024-04-23T16:36:44.166758+0000","last_peered":"2024-04-23T16:36:44.166758+0000","last_clean":"2024-04-23T16:36:44.166758+0000","last_became_active":"2024-04-23T16:31:29.139606+0000","last_became_peered":"2024-04-23T16:31:29.139606+0000","last_unstale":"2024-04-23T16:36:44.166758+0000","last_undegraded":"2024-04-23T16:36:44.166758+0000","last_fullsized":"2024-04-23T16:36:44.166758+0000","mapping_epoch":359,"log_start":"382'8700","ondisk_log_start":"382'8700","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'17441","last_scrub_stamp":"2024-04-23T16:36:20.849516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:36:20.849516+0000","objects_scrubbed":2009,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:50.602166+0000","scrub_duration":400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46596096,"num_objects":711,"num_object_clones":0,"num_object_copies":1422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18729,"num_write_kb":622080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1157,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-04-23T16:36:44.217548+0000","last_change":"2024-04-23T16:36:23.239567+0000","last_active":"2024-04-23T16:36:44.217548+0000","last_peered":"2024-04-23T16:36:44.217548+0000","last_clean":"2024-04-23T16:36:44.217548+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:36:44.217548+0000","last_undegraded":"2024-04-23T16:36:44.217548+0000","last_fullsized":"2024-04-23T16:36:44.217548+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:23.239466+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:36:23.239466+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:27.031497+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.2","version":"443'18360","reported_seq":18770,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-04-23T16:36:43.998424+0000","last_change":"2024-04-23T16:35:32.569280+0000","last_active":"2024-04-23T16:36:43.998424+0000","last_peered":"2024-04-23T16:36:43.998424+0000","last_clean":"2024-04-23T16:36:43.998424+0000","last_became_active":"2024-04-23T16:31:29.138430+0000","last_became_peered":"2024-04-23T16:31:29.138430+0000","last_unstale":"2024-04-23T16:36:43.998424+0000","last_undegraded":"2024-04-23T16:36:43.998424+0000","last_fullsized":"2024-04-23T16:36:43.998424+0000","mapping_epoch":359,"log_start":"382'8300","ondisk_log_start":"382'8300","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'14723","last_scrub_stamp":"2024-04-23T16:35:32.569137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:32.569137+0000","objects_scrubbed":4294,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:46.982993+0000","scrub_duration":850,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41680896,"num_objects":636,"num_object_clones":0,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18360,"num_write_kb":607872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1158,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-04-23T16:36:44.217707+0000","last_change":"2024-04-23T16:36:18.085504+0000","last_active":"2024-04-23T16:36:44.217707+0000","last_peered":"2024-04-23T16:36:44.217707+0000","last_clean":"2024-04-23T16:36:44.217707+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:36:44.217707+0000","last_undegraded":"2024-04-23T16:36:44.217707+0000","last_fullsized":"2024-04-23T16:36:44.217707+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:18.085386+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:18.085386+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:25.354657+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"443'20714","reported_seq":21225,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-04-23T16:36:44.126117+0000","last_change":"2024-04-23T16:36:23.728326+0000","last_active":"2024-04-23T16:36:44.126117+0000","last_peered":"2024-04-23T16:36:44.126117+0000","last_clean":"2024-04-23T16:36:44.126117+0000","last_became_active":"2024-04-23T16:31:29.139398+0000","last_became_peered":"2024-04-23T16:31:29.139398+0000","last_unstale":"2024-04-23T16:36:44.126117+0000","last_undegraded":"2024-04-23T16:36:44.126117+0000","last_fullsized":"2024-04-23T16:36:44.126117+0000","mapping_epoch":359,"log_start":"386'10700","ondisk_log_start":"386'10700","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'19588","last_scrub_stamp":"2024-04-23T16:36:23.728231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:36:23.728231+0000","objects_scrubbed":1817,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:47.414183+0000","scrub_duration":340,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45088768,"num_objects":688,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18440,"num_write_kb":612096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1148,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-04-23T16:36:43.819395+0000","last_change":"2024-04-23T16:35:33.875035+0000","last_active":"2024-04-23T16:36:43.819395+0000","last_peered":"2024-04-23T16:36:43.819395+0000","last_clean":"2024-04-23T16:36:43.819395+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:36:43.819395+0000","last_undegraded":"2024-04-23T16:36:43.819395+0000","last_fullsized":"2024-04-23T16:36:43.819395+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:35:33.874904+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:35:33.874904+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:36:59.611256+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"444'18289","reported_seq":18561,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-04-23T16:36:44.652591+0000","last_change":"2024-04-23T16:32:43.947374+0000","last_active":"2024-04-23T16:36:44.652591+0000","last_peered":"2024-04-23T16:36:44.652591+0000","last_clean":"2024-04-23T16:36:44.652591+0000","last_became_active":"2024-04-23T16:31:29.138968+0000","last_became_peered":"2024-04-23T16:31:29.138968+0000","last_unstale":"2024-04-23T16:36:44.652591+0000","last_undegraded":"2024-04-23T16:36:44.652591+0000","last_fullsized":"2024-04-23T16:36:44.652591+0000","mapping_epoch":359,"log_start":"382'8200","ondisk_log_start":"382'8200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8482","last_scrub_stamp":"2024-04-23T16:32:43.947177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.947177+0000","objects_scrubbed":8428,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39649280,"num_objects":605,"num_object_clones":0,"num_object_copies":1210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18289,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1050,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-04-23T16:36:44.217404+0000","last_change":"2024-04-23T16:36:29.341359+0000","last_active":"2024-04-23T16:36:44.217404+0000","last_peered":"2024-04-23T16:36:44.217404+0000","last_clean":"2024-04-23T16:36:44.217404+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:36:44.217404+0000","last_undegraded":"2024-04-23T16:36:44.217404+0000","last_fullsized":"2024-04-23T16:36:44.217404+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:29.341205+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:29.341205+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:50.871176+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"444'18822","reported_seq":19093,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-04-23T16:36:44.622715+0000","last_change":"2024-04-23T16:32:39.103230+0000","last_active":"2024-04-23T16:36:44.622715+0000","last_peered":"2024-04-23T16:36:44.622715+0000","last_clean":"2024-04-23T16:36:44.622715+0000","last_became_active":"2024-04-23T16:31:29.310246+0000","last_became_peered":"2024-04-23T16:31:29.310246+0000","last_unstale":"2024-04-23T16:36:44.622715+0000","last_undegraded":"2024-04-23T16:36:44.622715+0000","last_fullsized":"2024-04-23T16:36:44.622715+0000","mapping_epoch":359,"log_start":"383'8800","ondisk_log_start":"383'8800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8325","last_scrub_stamp":"2024-04-23T16:32:39.103059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:39.103059+0000","objects_scrubbed":8272,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":640,"num_object_clones":0,"num_object_copies":1280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18822,"num_write_kb":622784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":969,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-04-23T16:36:43.819264+0000","last_change":"2024-04-23T16:36:19.091622+0000","last_active":"2024-04-23T16:36:43.819264+0000","last_peered":"2024-04-23T16:36:43.819264+0000","last_clean":"2024-04-23T16:36:43.819264+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:36:43.819264+0000","last_undegraded":"2024-04-23T16:36:43.819264+0000","last_fullsized":"2024-04-23T16:36:43.819264+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:19.091490+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:36:19.091490+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:22.009913+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"444'18517","reported_seq":37275,"reported_epoch":444,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:36:45.333172+0000","last_change":"2024-04-23T16:33:26.132487+0000","last_active":"2024-04-23T16:36:45.333172+0000","last_peered":"2024-04-23T16:36:45.333172+0000","last_clean":"2024-04-23T16:32:59.695133+0000","last_became_active":"2024-04-23T16:33:26.078062+0000","last_became_peered":"2024-04-23T16:33:26.078062+0000","last_unstale":"2024-04-23T16:36:45.333172+0000","last_undegraded":"2024-04-23T16:33:25.892009+0000","last_fullsized":"2024-04-23T16:36:45.333172+0000","mapping_epoch":394,"log_start":"382'8500","ondisk_log_start":"382'8500","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'7968","last_scrub_stamp":"2024-04-23T16:32:36.102122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:36.102122+0000","objects_scrubbed":7926,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39124992,"num_objects":597,"num_object_clones":0,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18517,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9242,"num_bytes_recovered":605683712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":311},{"shards":"5,8","objects":286}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1140,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-04-23T16:36:44.224610+0000","last_change":"2024-04-23T16:33:17.354409+0000","last_active":"2024-04-23T16:36:44.224610+0000","last_peered":"2024-04-23T16:36:44.224610+0000","last_clean":"2024-04-23T16:36:44.224610+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:36:44.224610+0000","last_undegraded":"2024-04-23T16:36:44.224610+0000","last_fullsized":"2024-04-23T16:36:44.224610+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:17.354263+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.354263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":633929752,"num_objects":9674,"num_object_clones":0,"num_object_copies":19348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9674,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":297518,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27649,"num_bytes_recovered":1812004864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3086622720,"data_stored":3086614576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160649,"ondisk_log_size":160649,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":570912,"kb_used_data":74648,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93174240,"statfs":{"total":95995035648,"available":95410421760,"internally_reserved":0,"allocated":76439552,"data_stored":75934972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":508164548},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351239,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612877,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":480724,"kb_used_data":128840,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93264428,"statfs":{"total":95995035648,"available":95502774272,"internally_reserved":0,"allocated":131932160,"data_stored":131437706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":360315328},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841810,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":378012,"kb_used_data":80276,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":93367140,"statfs":{"total":95995035648,"available":95607951360,"internally_reserved":0,"allocated":82202624,"data_stored":81702116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":440796,"kb_used_data":44500,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93304356,"statfs":{"total":95995035648,"available":95543660544,"internally_reserved":0,"allocated":45568000,"data_stored":45067492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":405797322},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463196,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":422996,"kb_used_data":130824,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93322156,"statfs":{"total":95995035648,"available":95561887744,"internally_reserved":0,"allocated":133963776,"data_stored":133469322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":299170246},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137336,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738681,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":763076,"kb_used_data":316796,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":92982076,"statfs":{"total":95995035648,"available":95213645824,"internally_reserved":0,"allocated":324399104,"data_stored":323910654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117290,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":399208,"kb_used_data":39004,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93345944,"statfs":{"total":95995035648,"available":95586246656,"internally_reserved":0,"allocated":39940096,"data_stored":39436036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":368835015},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921060,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":781188,"kb_used_data":222460,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92963964,"statfs":{"total":95995035648,"available":95195099136,"internally_reserved":0,"allocated":227799040,"data_stored":227310590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757535,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":648048,"kb_used_data":211488,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93097104,"statfs":{"total":95995035648,"available":95331434496,"internally_reserved":0,"allocated":216563712,"data_stored":216055580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":305528832,"data_stored":305528832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1886846976,"data_stored":1886846976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38469632,"data_stored":38469632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":209256448,"data_stored":209256448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":199102464,"data_stored":199098392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":132382720,"data_stored":132382720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":41418752,"data_stored":41418752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":77201408,"data_stored":77201408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":128974848,"data_stored":128974848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":67440640,"data_stored":67436568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:36:52.132 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.a is not active+clean 2024-04-23T16:36:52.132 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.a', 'version': "444'19023", 'reported_seq': 37924, 'reported_epoch': 444, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:36:48.241216+0000', 'last_change': '2024-04-23T16:33:26.432192+0000', 'last_active': '2024-04-23T16:36:48.241216+0000', 'last_peered': '2024-04-23T16:36:48.241216+0000', 'last_clean': '2024-04-23T16:32:59.749530+0000', 'last_became_active': '2024-04-23T16:33:26.338644+0000', 'last_became_peered': '2024-04-23T16:33:26.338644+0000', 'last_unstale': '2024-04-23T16:36:48.241216+0000', 'last_undegraded': '2024-04-23T16:33:25.874847+0000', 'last_fullsized': '2024-04-23T16:36:48.241216+0000', 'mapping_epoch': 394, 'log_start': "383'9000", 'ondisk_log_start': "383'9000", 'created': 359, 'last_epoch_clean': 360, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "382'8711", 'last_scrub_stamp': '2024-04-23T16:32:43.928914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:31:28.121358+0000', 'last_clean_scrub_stamp': '2024-04-23T16:32:43.928914+0000', 'objects_scrubbed': 8680, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1273, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27459584, 'num_objects': 419, 'num_object_clones': 0, 'num_object_copies': 838, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 402, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 419, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19023, 'num_write_kb': 622144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9310, 'num_bytes_recovered': 610140160, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [8, 4], 'acting': [8, 4], 'avail_no_missing': ['8'], 'object_location_counts': [{'shards': '4,8', 'objects': 17}, {'shards': '8', 'objects': 402}], 'blocked_by': [], 'up_primary': 8, 'acting_primary': 8, 'purged_snaps': []} 2024-04-23T16:36:52.132 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.6 is not active+clean 2024-04-23T16:36:52.132 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.6', 'version': "444'18777", 'reported_seq': 37250, 'reported_epoch': 444, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:36:45.321108+0000', 'last_change': '2024-04-23T16:33:26.507399+0000', 'last_active': '2024-04-23T16:36:45.321108+0000', 'last_peered': '2024-04-23T16:36:45.321108+0000', 'last_clean': '2024-04-23T16:32:59.715575+0000', 'last_became_active': '2024-04-23T16:33:26.458611+0000', 'last_became_peered': '2024-04-23T16:33:26.458611+0000', 'last_unstale': '2024-04-23T16:36:45.321108+0000', 'last_undegraded': '2024-04-23T16:33:25.895472+0000', 'last_fullsized': '2024-04-23T16:36:45.321108+0000', 'mapping_epoch': 394, 'log_start': "382'8700", 'ondisk_log_start': "382'8700", 'created': 359, 'last_epoch_clean': 360, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "382'8607", 'last_scrub_stamp': '2024-04-23T16:32:43.130806+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:31:28.121358+0000', 'last_clean_scrub_stamp': '2024-04-23T16:32:43.130806+0000', 'objects_scrubbed': 8577, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1484, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39518208, 'num_objects': 603, 'num_object_clones': 0, 'num_object_copies': 1206, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 590, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 603, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18777, 'num_write_kb': 620160, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9097, 'num_bytes_recovered': 596180992, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': ['5'], 'object_location_counts': [{'shards': '5', 'objects': 590}, {'shards': '5,7', 'objects': 13}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-23T16:36:52.132 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.4 is not active+clean 2024-04-23T16:36:52.132 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.4', 'version': "444'18517", 'reported_seq': 37275, 'reported_epoch': 444, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:36:45.333172+0000', 'last_change': '2024-04-23T16:33:26.132487+0000', 'last_active': '2024-04-23T16:36:45.333172+0000', 'last_peered': '2024-04-23T16:36:45.333172+0000', 'last_clean': '2024-04-23T16:32:59.695133+0000', 'last_became_active': '2024-04-23T16:33:26.078062+0000', 'last_became_peered': '2024-04-23T16:33:26.078062+0000', 'last_unstale': '2024-04-23T16:36:45.333172+0000', 'last_undegraded': '2024-04-23T16:33:25.892009+0000', 'last_fullsized': '2024-04-23T16:36:45.333172+0000', 'mapping_epoch': 394, 'log_start': "382'8500", 'ondisk_log_start': "382'8500", 'created': 359, 'last_epoch_clean': 360, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "380'7968", 'last_scrub_stamp': '2024-04-23T16:32:36.102122+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:31:28.121358+0000', 'last_clean_scrub_stamp': '2024-04-23T16:32:36.102122+0000', 'objects_scrubbed': 7926, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1466, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39124992, 'num_objects': 597, 'num_object_clones': 0, 'num_object_copies': 1194, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 311, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18517, 'num_write_kb': 611648, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9242, 'num_bytes_recovered': 605683712, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 8], 'acting': [5, 8], 'avail_no_missing': ['5'], 'object_location_counts': [{'shards': '5', 'objects': 311}, {'shards': '5,8', 'objects': 286}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-23T16:36:52.133 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-23T16:36:52.213 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:36:52.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:51 smithi188 ceph-mon[44101]: pgmap v1262: 27 pgs: 2 active+recovering+degraded, 25 active+clean; 370 MiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 714/11822 objects degraded (6.040%); 14 MiB/s, 229 objects/s recovering 2024-04-23T16:36:52.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:51 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:36:52.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:51 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2883177283' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:36:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:51 smithi188 ceph-mon[44101]: osdmap e445: 12 total, 11 up, 9 in 2024-04-23T16:36:52.242 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:52.262 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:36:52.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:52.268 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:36:52.369 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:36:52.396 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:52.417 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:36:52.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:52.420 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:36:52.521 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:52.645 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:52.651 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:36:53.291 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:36:54.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:53 smithi103 ceph-mon[39629]: pgmap v1264: 27 pgs: 2 active+recovering+degraded, 25 active+clean; 332 MiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 714/10606 objects degraded (6.732%); 12 MiB/s, 184 objects/s recovering 2024-04-23T16:36:54.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:53 smithi148 ceph-mon[44520]: pgmap v1264: 27 pgs: 2 active+recovering+degraded, 25 active+clean; 332 MiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 714/10606 objects degraded (6.732%); 12 MiB/s, 184 objects/s recovering 2024-04-23T16:36:54.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:53 smithi188 ceph-mon[44101]: pgmap v1264: 27 pgs: 2 active+recovering+degraded, 25 active+clean; 332 MiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 714/10606 objects degraded (6.732%); 12 MiB/s, 184 objects/s recovering 2024-04-23T16:36:54.225 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-23T16:36:54.229 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:36:54.235 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:54.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:54.339 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:36:54.370 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:54.560 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:36:54.659 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:36:55.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:54 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 714/10606 objects degraded (6.732%), 2 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:55.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:54 smithi148 ceph-mon[44520]: osdmap e446: 12 total, 11 up, 9 in 2024-04-23T16:36:55.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:54 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:36:55.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:54 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:36:55.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:54 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 714/10606 objects degraded (6.732%), 2 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:54 smithi188 ceph-mon[44101]: osdmap e446: 12 total, 11 up, 9 in 2024-04-23T16:36:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:54 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:36:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:54 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:36:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:54 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 714/10606 objects degraded (6.732%), 2 pgs degraded (PG_DEGRADED) 2024-04-23T16:36:55.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:54 smithi103 ceph-mon[39629]: osdmap e446: 12 total, 11 up, 9 in 2024-04-23T16:36:55.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:54 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:36:55.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:54 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:36:55.453 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:36:55.453 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 399/5286 objects degraded (7.548%), 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1597,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":447,"num_osds":12,"num_up_osds":11,"osd_up_since":1713889979,"num_in_osds":9,"osd_in_since":1713890004,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":25},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":27,"num_pools":2,"num_objects":2643,"data_bytes":173605432,"bytes_used":4106813440,"bytes_avail":859848507392,"bytes_total":863955320832,"degraded_objects":399,"degraded_total":5286,"degraded_ratio":0.075482406356413165,"recovering_objects_per_sec":152,"recovering_bytes_per_sec":9984075,"recovering_keys_per_sec":0,"num_objects_recovered":1219,"num_bytes_recovered":79888384,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":991},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.103:8443/"}},"servicemap":{"epoch":34,"modified":"2024-04-23T16:35:30.733564+0000","services":{}},"progress_events":{}} 2024-04-23T16:36:55.484 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:36:55.496 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Removed 153595 objects 2024-04-23T16:36:55.496 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Clean up completed and total clean up time :208.207 2024-04-23T16:36:56.036 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:55 smithi148 ceph-mon[44520]: pgmap v1266: 27 pgs: 2 active+recovering+degraded, 25 active+clean; 166 MiB data, 3.8 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 991 op/s; 399/5286 objects degraded (7.548%); 9.5 MiB/s, 152 objects/s recovering 2024-04-23T16:36:56.036 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:55 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:36:56.037 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:55 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:36:56.037 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:55 smithi148 ceph-mon[44520]: osdmap e447: 12 total, 11 up, 9 in 2024-04-23T16:36:56.037 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:55 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2355205293' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:36:56.121 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:55 smithi103 ceph-mon[39629]: pgmap v1266: 27 pgs: 2 active+recovering+degraded, 25 active+clean; 166 MiB data, 3.8 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 991 op/s; 399/5286 objects degraded (7.548%); 9.5 MiB/s, 152 objects/s recovering 2024-04-23T16:36:56.121 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:55 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:36:56.121 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:55 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:36:56.121 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:55 smithi103 ceph-mon[39629]: osdmap e447: 12 total, 11 up, 9 in 2024-04-23T16:36:56.121 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:55 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2355205293' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:36:56.124 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:36:56.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:56.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:36:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:55 smithi188 ceph-mon[44101]: pgmap v1266: 27 pgs: 2 active+recovering+degraded, 25 active+clean; 166 MiB data, 3.8 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 991 op/s; 399/5286 objects degraded (7.548%); 9.5 MiB/s, 152 objects/s recovering 2024-04-23T16:36:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:55 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:36:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:55 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:36:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:55 smithi188 ceph-mon[44101]: osdmap e447: 12 total, 11 up, 9 in 2024-04-23T16:36:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:55 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2355205293' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:36:56.228 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:36:56.259 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:56.279 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:56.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:56.283 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:56.384 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:56.413 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:56.561 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-04-23T16:36:56.562 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-04-23T16:36:56.601 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:36:56.601 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:36:56.601 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2024-04-23T16:36:56.602 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:36:56.611 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:56.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:56.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:56.715 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:36:56.747 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:56.769 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:36:56.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:56.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:36:56.873 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:36:56.903 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:56.923 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:56.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:56.926 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:57.027 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:36:57.056 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:57.254 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:36:57.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:57.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:36:57.278 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-23T16:36:57.358 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:36:57.387 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:57.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:57.410 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:36:57.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:36:57.511 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:36:57.543 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:57.564 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:36:57.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:57.567 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:36:57.585 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:36:57.668 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:36:57.697 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:57.715 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-23T16:36:57.715 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:36:57.718 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:36:57.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:57.721 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:36:57.822 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:36:57.850 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:57.869 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:36:57.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:57.872 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:36:57.973 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:36:57.998 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:57 smithi103 ceph-mon[39629]: pgmap v1268: 27 pgs: 1 active+recovering+degraded, 26 active+clean; 71 MiB data, 3.7 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 795 op/s; 87/2268 objects degraded (3.836%); 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:36:58.001 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:58.123 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:36:58.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:57 smithi148 ceph-mon[44520]: pgmap v1268: 27 pgs: 1 active+recovering+degraded, 26 active+clean; 71 MiB data, 3.7 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 795 op/s; 87/2268 objects degraded (3.836%); 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:36:58.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:58.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:36:58.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:57 smithi188 ceph-mon[44101]: pgmap v1268: 27 pgs: 1 active+recovering+degraded, 26 active+clean; 71 MiB data, 3.7 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 795 op/s; 87/2268 objects degraded (3.836%); 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:36:58.227 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:36:58.255 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:58.274 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:36:58.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:58.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:36:58.378 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:36:58.404 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:58.408 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:36:58.417 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:36:58.424 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:36:58.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:58.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:36:58.438 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:36:58.459 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:36:58.478 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:36:58.497 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:36:58.516 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:36:58.528 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:36:58.536 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:36:58.554 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:36:58.557 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:36:58.587 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:36:58.608 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:36:58.608 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:36:58.608 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:36:58.608 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:36:58.608 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:36:58.608 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:36:58.608 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:36:58.608 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:36:58.608 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:36:58.608 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:36:58.609 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:36:58.609 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:36:58.609 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:36:59.132 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:58 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:36:59.133 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:58 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:36:59.133 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:58 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1500657160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:59.133 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:59.133 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:59.133 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:59.133 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:59.133 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:59.133 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:59.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:58 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:36:59.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:58 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:36:59.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:58 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1500657160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:59.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:59.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:59.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:59.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:59.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:59.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:59.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:58 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:36:59.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:58 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:36:59.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:58 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1500657160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:36:59.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:59.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:59.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:59.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:59.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:59.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:36:59.389 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:36:59.390 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:36:59.710 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:36:59.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:36:59.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:36:59.814 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:36:59.843 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:00.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:00.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:00.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:00.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:00.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:00.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:59 smithi148 ceph-mon[44520]: pgmap v1269: 27 pgs: 27 active+clean; 37 MiB data, 3.6 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 674 op/s; 5.6 MiB/s, 90 objects/s recovering 2024-04-23T16:37:00.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:59 smithi148 ceph-mon[44520]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 87/2268 objects degraded (3.836%), 1 pg degraded) 2024-04-23T16:37:00.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:36:59 smithi148 ceph-mon[44520]: osdmap e448: 12 total, 11 up, 9 in 2024-04-23T16:37:00.091 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-23T16:37:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:59 smithi103 ceph-mon[39629]: pgmap v1269: 27 pgs: 27 active+clean; 37 MiB data, 3.6 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 674 op/s; 5.6 MiB/s, 90 objects/s recovering 2024-04-23T16:37:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:59 smithi103 ceph-mon[39629]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 87/2268 objects degraded (3.836%), 1 pg degraded) 2024-04-23T16:37:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:36:59 smithi103 ceph-mon[39629]: osdmap e448: 12 total, 11 up, 9 in 2024-04-23T16:37:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:59 smithi188 ceph-mon[44101]: pgmap v1269: 27 pgs: 27 active+clean; 37 MiB data, 3.6 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 674 op/s; 5.6 MiB/s, 90 objects/s recovering 2024-04-23T16:37:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:59 smithi188 ceph-mon[44101]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 87/2268 objects degraded (3.836%), 1 pg degraded) 2024-04-23T16:37:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:36:59 smithi188 ceph-mon[44101]: osdmap e448: 12 total, 11 up, 9 in 2024-04-23T16:37:00.268 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:00.270 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:00.295 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:37:00.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:00.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:37:00.306 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:37:00.306 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:37:00.356 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:37:00.400 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:37:00.432 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:00.454 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:37:00.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:00.457 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:37:00.558 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:37:00.587 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:00.606 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:37:00.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:00.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:37:00.710 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:37:00.738 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:00.811 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:37:00.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:00.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:37:00.887 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:37:00.915 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:37:00.942 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:01.094 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:37:01.096 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:00 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/607955668' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:37:01.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:01.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:37:01.197 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:01.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:00 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/607955668' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:37:01.230 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:01.239 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:00 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/607955668' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:37:01.252 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:01.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:01.255 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:01.356 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:37:01.383 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:01.582 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:37:01.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:01.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:37:01.685 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:01.715 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:02.064 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:02.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:01 smithi103 ceph-mon[39629]: pgmap v1271: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 716 op/s; 3.8 MiB/s, 60 objects/s recovering 2024-04-23T16:37:02.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:01 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:37:02.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:01 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/607955668' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:37:02.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:01 smithi103 ceph-mon[39629]: osdmap e449: 12 total, 11 up, 9 in 2024-04-23T16:37:02.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:01 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:37:02.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:01 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:37:02.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:02.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:01 smithi148 ceph-mon[44520]: pgmap v1271: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 716 op/s; 3.8 MiB/s, 60 objects/s recovering 2024-04-23T16:37:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:01 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:37:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:01 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/607955668' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:37:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:01 smithi148 ceph-mon[44520]: osdmap e449: 12 total, 11 up, 9 in 2024-04-23T16:37:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:01 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:37:02.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:01 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:37:02.130 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:37:02.130 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:37:02.168 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:37:02.199 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:01 smithi188 ceph-mon[44101]: pgmap v1271: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 716 op/s; 3.8 MiB/s, 60 objects/s recovering 2024-04-23T16:37:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:01 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:37:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:01 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/607955668' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:37:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:01 smithi188 ceph-mon[44101]: osdmap e449: 12 total, 11 up, 9 in 2024-04-23T16:37:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:01 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:37:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:01 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:37:02.308 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:37:02.894 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:02 smithi103 ceph-mon[39629]: osdmap e450: 12 total, 11 up, 9 in 2024-04-23T16:37:02.895 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:02 smithi103 ceph-mon[39629]: 5.a scrub starts 2024-04-23T16:37:02.895 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:02 smithi103 ceph-mon[39629]: 5.a scrub ok 2024-04-23T16:37:02.895 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:02 smithi103 ceph-mon[39629]: 5.c scrub starts 2024-04-23T16:37:02.895 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:02 smithi103 ceph-mon[39629]: 5.c scrub ok 2024-04-23T16:37:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:02 smithi188 ceph-mon[44101]: osdmap e450: 12 total, 11 up, 9 in 2024-04-23T16:37:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:02 smithi188 ceph-mon[44101]: 5.a scrub starts 2024-04-23T16:37:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:02 smithi188 ceph-mon[44101]: 5.a scrub ok 2024-04-23T16:37:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:02 smithi188 ceph-mon[44101]: 5.c scrub starts 2024-04-23T16:37:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:02 smithi188 ceph-mon[44101]: 5.c scrub ok 2024-04-23T16:37:03.312 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:02 smithi148 ceph-mon[44520]: osdmap e450: 12 total, 11 up, 9 in 2024-04-23T16:37:03.312 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:02 smithi148 ceph-mon[44520]: 5.a scrub starts 2024-04-23T16:37:03.313 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:02 smithi148 ceph-mon[44520]: 5.a scrub ok 2024-04-23T16:37:03.313 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:02 smithi148 ceph-mon[44520]: 5.c scrub starts 2024-04-23T16:37:03.313 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:02 smithi148 ceph-mon[44520]: 5.c scrub ok 2024-04-23T16:37:03.650 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-23T16:37:03.678 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:37:03.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:03.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:37:03.782 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:37:03.811 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:03.919 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:37:03.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:03 smithi188 ceph-mon[44101]: pgmap v1274: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 188 op/s; 928 KiB/s, 14 objects/s recovering 2024-04-23T16:37:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:03 smithi188 ceph-mon[44101]: 5.2 scrub starts 2024-04-23T16:37:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:03 smithi188 ceph-mon[44101]: 5.2 scrub ok 2024-04-23T16:37:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:03 smithi103 ceph-mon[39629]: pgmap v1274: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 188 op/s; 928 KiB/s, 14 objects/s recovering 2024-04-23T16:37:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:03 smithi103 ceph-mon[39629]: 5.2 scrub starts 2024-04-23T16:37:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:03 smithi103 ceph-mon[39629]: 5.2 scrub ok 2024-04-23T16:37:04.172 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:03 smithi148 ceph-mon[44520]: pgmap v1274: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 188 op/s; 928 KiB/s, 14 objects/s recovering 2024-04-23T16:37:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:03 smithi148 ceph-mon[44520]: 5.2 scrub starts 2024-04-23T16:37:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:03 smithi148 ceph-mon[44520]: 5.2 scrub ok 2024-04-23T16:37:05.161 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:37:05.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:05.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:37:05.265 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:37:05.299 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:05.321 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:37:05.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:05.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:37:05.425 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:37:05.450 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:05 smithi188 ceph-mon[44101]: pgmap v1275: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 96 op/s 2024-04-23T16:37:05.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:05 smithi103 ceph-mon[39629]: pgmap v1275: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 96 op/s 2024-04-23T16:37:05.454 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:05.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:05 smithi148 ceph-mon[44520]: pgmap v1275: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 96 op/s 2024-04-23T16:37:05.653 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:37:05.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:05.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:37:05.744 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:37:05.744 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:37:05.756 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:05.784 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:05.880 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:37:05.882 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:37:06.203 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'unique_pool_3' removed 2024-04-23T16:37:06.204 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:37:06.234 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:06.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:06.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:06.338 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:37:06.364 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:06.365 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:06 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:37:06.365 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:06 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:37:06.365 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:06 smithi188 ceph-mon[44101]: 1.8 scrub starts 2024-04-23T16:37:06.365 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:06 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/602739026' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:37:06.365 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:06 smithi188 ceph-mon[44101]: 5.1 scrub starts 2024-04-23T16:37:06.365 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:06 smithi188 ceph-mon[44101]: 5.1 scrub ok 2024-04-23T16:37:06.365 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:06 smithi188 ceph-mon[44101]: from='client.16245 v1:172.21.15.103:0/2305205277' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:37:06.391 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-23T16:37:06.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:06 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:37:06.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:06 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:37:06.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:06 smithi103 ceph-mon[39629]: 1.8 scrub starts 2024-04-23T16:37:06.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:06 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/602739026' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:37:06.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:06 smithi103 ceph-mon[39629]: 5.1 scrub starts 2024-04-23T16:37:06.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:06 smithi103 ceph-mon[39629]: 5.1 scrub ok 2024-04-23T16:37:06.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:06 smithi103 ceph-mon[39629]: from='client.16245 v1:172.21.15.103:0/2305205277' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:37:06.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:06 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:37:06.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:06 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:37:06.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:06 smithi148 ceph-mon[44520]: 1.8 scrub starts 2024-04-23T16:37:06.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:06 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/602739026' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:37:06.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:06 smithi148 ceph-mon[44520]: 5.1 scrub starts 2024-04-23T16:37:06.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:06 smithi148 ceph-mon[44520]: 5.1 scrub ok 2024-04-23T16:37:06.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:06 smithi148 ceph-mon[44520]: from='client.16245 v1:172.21.15.103:0/2305205277' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:37:06.684 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:37:06.819 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:37:06.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:06.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:37:06.924 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:37:06.952 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:06.972 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:37:06.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:06.974 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:37:07.075 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:07.103 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:07.175 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:07.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:07.177 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:07.278 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:37:07.307 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:07.327 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:37:07.329 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-23T16:37:07.329 INFO:tasks.radosbench:Beginning radosbench... 2024-04-23T16:37:07.330 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-04-23T16:37:07.330 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-04-23T16:37:07.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:07.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:37:07.432 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:37:07.464 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:07.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:07 smithi188 ceph-mon[44101]: 1.9 scrub starts 2024-04-23T16:37:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:07 smithi188 ceph-mon[44101]: 1.8 scrub ok 2024-04-23T16:37:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:07 smithi188 ceph-mon[44101]: 1.9 scrub ok 2024-04-23T16:37:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:07 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/602739026' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:37:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:07 smithi188 ceph-mon[44101]: osdmap e451: 12 total, 11 up, 9 in 2024-04-23T16:37:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:07 smithi188 ceph-mon[44101]: 1.7 scrub starts 2024-04-23T16:37:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:07 smithi188 ceph-mon[44101]: 1.7 scrub ok 2024-04-23T16:37:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:07 smithi188 ceph-mon[44101]: pgmap v1277: 11 pgs: 11 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:07.485 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:37:07.487 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:07 smithi148 ceph-mon[44520]: 1.9 scrub starts 2024-04-23T16:37:07.487 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:07 smithi148 ceph-mon[44520]: 1.8 scrub ok 2024-04-23T16:37:07.487 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:07 smithi148 ceph-mon[44520]: 1.9 scrub ok 2024-04-23T16:37:07.487 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:07 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/602739026' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:37:07.487 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:07 smithi148 ceph-mon[44520]: osdmap e451: 12 total, 11 up, 9 in 2024-04-23T16:37:07.487 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:07 smithi148 ceph-mon[44520]: 1.7 scrub starts 2024-04-23T16:37:07.487 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:07 smithi148 ceph-mon[44520]: 1.7 scrub ok 2024-04-23T16:37:07.487 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:07 smithi148 ceph-mon[44520]: pgmap v1277: 11 pgs: 11 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:07.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:07.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:37:07.589 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:07.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:07 smithi103 ceph-mon[39629]: 1.9 scrub starts 2024-04-23T16:37:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:07 smithi103 ceph-mon[39629]: 1.8 scrub ok 2024-04-23T16:37:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:07 smithi103 ceph-mon[39629]: 1.9 scrub ok 2024-04-23T16:37:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:07 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/602739026' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:37:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:07 smithi103 ceph-mon[39629]: osdmap e451: 12 total, 11 up, 9 in 2024-04-23T16:37:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:07 smithi103 ceph-mon[39629]: 1.7 scrub starts 2024-04-23T16:37:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:07 smithi103 ceph-mon[39629]: 1.7 scrub ok 2024-04-23T16:37:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:07 smithi103 ceph-mon[39629]: pgmap v1277: 11 pgs: 11 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:07.616 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:07.636 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:07.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:07.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:07.742 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:37:07.771 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:07.791 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:37:07.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:07.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:37:07.895 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:07.924 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:07.961 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:07.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:07.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:08.065 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:37:08.090 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:37:08.093 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:08.109 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:37:08.113 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:37:08.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:08.117 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:37:08.128 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:37:08.148 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:37:08.170 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:37:08.189 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:37:08.209 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:37:08.217 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:37:08.231 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:37:08.245 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:08.253 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:37:08.273 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:37:08.293 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:37:08.293 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:37:08.293 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:37:08.293 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:37:08.294 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:37:08.294 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:37:08.294 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:37:08.294 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:37:08.294 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:37:08.294 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:37:08.294 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:37:08.294 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:37:08.294 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:37:08.415 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:37:08.415 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:37:08.415 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:37:08.415 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:37:08.415 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:37:08.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:08 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1518587766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:37:08.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:08 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1518587766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:37:08.614 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:08.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:08 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1518587766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:37:08.646 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1275,"stamp":"2024-04-23T16:37:04.751527+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":307192,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28953,"num_bytes_recovered":1897599520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162352,"ondisk_log_size":162352,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":3681260,"kb_used_data":10412,"kb_used_omap":14,"kb_used_meta":3670705,"kb_avail":840025108,"statfs":{"total":863955320832,"available":860185710592,"internally_reserved":0,"allocated":10661888,"data_stored":5873328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3758802927},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38076416,"num_objects":-581,"num_object_clones":0,"num_object_copies":-1162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":581,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001164"},"pg_stats":[{"pgid":"5.f","version":"447'19366","reported_seq":19658,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-23T16:36:59.184803+0000","last_change":"2024-04-23T16:33:17.526419+0000","last_active":"2024-04-23T16:36:59.184803+0000","last_peered":"2024-04-23T16:36:59.184803+0000","last_clean":"2024-04-23T16:36:59.184803+0000","last_became_active":"2024-04-23T16:31:29.140923+0000","last_became_peered":"2024-04-23T16:31:29.140923+0000","last_unstale":"2024-04-23T16:36:59.184803+0000","last_undegraded":"2024-04-23T16:36:59.184803+0000","last_fullsized":"2024-04-23T16:36:59.184803+0000","mapping_epoch":359,"log_start":"386'9300","ondisk_log_start":"386'9300","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9683","last_scrub_stamp":"2024-04-23T16:33:17.526283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.526283+0000","objects_scrubbed":9683,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19366,"num_write_kb":619712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"447'19264","reported_seq":19686,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-23T16:37:02.440174+0000","last_change":"2024-04-23T16:35:26.407358+0000","last_active":"2024-04-23T16:37:02.440174+0000","last_peered":"2024-04-23T16:37:02.440174+0000","last_clean":"2024-04-23T16:37:02.440174+0000","last_became_active":"2024-04-23T16:31:29.137890+0000","last_became_peered":"2024-04-23T16:31:29.137890+0000","last_unstale":"2024-04-23T16:37:02.440174+0000","last_undegraded":"2024-04-23T16:37:02.440174+0000","last_fullsized":"2024-04-23T16:37:02.440174+0000","mapping_epoch":359,"log_start":"386'9200","ondisk_log_start":"386'9200","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'14582","last_scrub_stamp":"2024-04-23T16:35:26.407093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:26.407093+0000","objects_scrubbed":4692,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:03.204402+0000","scrub_duration":544,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19264,"num_write_kb":616448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"447'19442","reported_seq":39170,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-23T16:37:02.591077+0000","last_change":"2024-04-23T16:37:02.591120+0000","last_active":"2024-04-23T16:37:02.591077+0000","last_peered":"2024-04-23T16:37:02.591077+0000","last_clean":"2024-04-23T16:37:02.591077+0000","last_became_active":"2024-04-23T16:33:26.338644+0000","last_became_peered":"2024-04-23T16:33:26.338644+0000","last_unstale":"2024-04-23T16:37:02.591077+0000","last_undegraded":"2024-04-23T16:37:02.591077+0000","last_fullsized":"2024-04-23T16:37:02.591077+0000","mapping_epoch":394,"log_start":"386'9400","ondisk_log_start":"386'9400","created":359,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"447'19442","last_scrub_stamp":"2024-04-23T16:37:02.590994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:37:02.590994+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:20.406888+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19442,"num_write_kb":622144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9712,"num_bytes_recovered":636485632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.9","version":"447'19320","reported_seq":19597,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-23T16:36:59.044968+0000","last_change":"2024-04-23T16:32:37.256944+0000","last_active":"2024-04-23T16:36:59.044968+0000","last_peered":"2024-04-23T16:36:59.044968+0000","last_clean":"2024-04-23T16:36:59.044968+0000","last_became_active":"2024-04-23T16:31:29.141551+0000","last_became_peered":"2024-04-23T16:31:29.141551+0000","last_unstale":"2024-04-23T16:36:59.044968+0000","last_undegraded":"2024-04-23T16:36:59.044968+0000","last_fullsized":"2024-04-23T16:36:59.044968+0000","mapping_epoch":359,"log_start":"386'9300","ondisk_log_start":"386'9300","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8156","last_scrub_stamp":"2024-04-23T16:32:37.256787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:37.256787+0000","objects_scrubbed":8082,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19320,"num_write_kb":618240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"447'18734","reported_seq":19126,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-23T16:37:01.748203+0000","last_change":"2024-04-23T16:36:20.471641+0000","last_active":"2024-04-23T16:37:01.748203+0000","last_peered":"2024-04-23T16:37:01.748203+0000","last_clean":"2024-04-23T16:37:01.748203+0000","last_became_active":"2024-04-23T16:31:29.139045+0000","last_became_peered":"2024-04-23T16:31:29.139045+0000","last_unstale":"2024-04-23T16:37:01.748203+0000","last_undegraded":"2024-04-23T16:37:01.748203+0000","last_fullsized":"2024-04-23T16:37:01.748203+0000","mapping_epoch":359,"log_start":"384'8700","ondisk_log_start":"384'8700","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'16781","last_scrub_stamp":"2024-04-23T16:36:20.471467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:36:20.471467+0000","objects_scrubbed":1958,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:48.916510+0000","scrub_duration":248,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18734,"num_write_kb":599488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"447'22681","reported_seq":23174,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-23T16:37:02.440181+0000","last_change":"2024-04-23T16:35:32.474057+0000","last_active":"2024-04-23T16:37:02.440181+0000","last_peered":"2024-04-23T16:37:02.440181+0000","last_clean":"2024-04-23T16:37:02.440181+0000","last_became_active":"2024-04-23T16:31:29.310906+0000","last_became_peered":"2024-04-23T16:31:29.310906+0000","last_unstale":"2024-04-23T16:37:02.440181+0000","last_undegraded":"2024-04-23T16:37:02.440181+0000","last_fullsized":"2024-04-23T16:37:02.440181+0000","mapping_epoch":359,"log_start":"413'12604","ondisk_log_start":"413'12604","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'18441","last_scrub_stamp":"2024-04-23T16:35:32.473828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:35:32.473828+0000","objects_scrubbed":4251,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:03.204423+0000","scrub_duration":548,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19120,"num_write_kb":611840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1170,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-23T16:37:02.440149+0000","last_change":"2024-04-23T16:36:19.132015+0000","last_active":"2024-04-23T16:37:02.440149+0000","last_peered":"2024-04-23T16:37:02.440149+0000","last_clean":"2024-04-23T16:37:02.440149+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:37:02.440149+0000","last_undegraded":"2024-04-23T16:37:02.440149+0000","last_fullsized":"2024-04-23T16:37:02.440149+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:19.131883+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:19.131883+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:41.022603+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"447'19456","reported_seq":19746,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-23T16:37:02.667090+0000","last_change":"2024-04-23T16:37:02.667116+0000","last_active":"2024-04-23T16:37:02.667090+0000","last_peered":"2024-04-23T16:37:02.667090+0000","last_clean":"2024-04-23T16:37:02.667090+0000","last_became_active":"2024-04-23T16:31:29.141640+0000","last_became_peered":"2024-04-23T16:31:29.141640+0000","last_unstale":"2024-04-23T16:37:02.667090+0000","last_undegraded":"2024-04-23T16:37:02.667090+0000","last_fullsized":"2024-04-23T16:37:02.667090+0000","mapping_epoch":359,"log_start":"386'9400","ondisk_log_start":"386'9400","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"447'19456","last_scrub_stamp":"2024-04-23T16:37:02.667016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:37:02.667016+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:31.207799+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19456,"num_write_kb":622592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1152,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-23T16:36:59.184602+0000","last_change":"2024-04-23T16:33:25.851379+0000","last_active":"2024-04-23T16:36:59.184602+0000","last_peered":"2024-04-23T16:36:59.184602+0000","last_clean":"2024-04-23T16:36:59.184602+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:36:59.184602+0000","last_undegraded":"2024-04-23T16:36:59.184602+0000","last_fullsized":"2024-04-23T16:36:59.184602+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:31:45.498820+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:31:45.498820+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"447'19132","reported_seq":19526,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-23T16:37:01.748203+0000","last_change":"2024-04-23T16:36:24.521917+0000","last_active":"2024-04-23T16:37:01.748203+0000","last_peered":"2024-04-23T16:37:01.748203+0000","last_clean":"2024-04-23T16:37:01.748203+0000","last_became_active":"2024-04-23T16:31:29.140251+0000","last_became_peered":"2024-04-23T16:31:29.140251+0000","last_unstale":"2024-04-23T16:37:01.748203+0000","last_undegraded":"2024-04-23T16:37:01.748203+0000","last_fullsized":"2024-04-23T16:37:01.748203+0000","mapping_epoch":359,"log_start":"386'9100","ondisk_log_start":"386'9100","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'17367","last_scrub_stamp":"2024-04-23T16:36:24.521728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:36:24.521728+0000","objects_scrubbed":1767,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:41.002308+0000","scrub_duration":288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":19132,"num_write_kb":612161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1144,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-23T16:36:59.045018+0000","last_change":"2024-04-23T16:32:29.517947+0000","last_active":"2024-04-23T16:36:59.045018+0000","last_peered":"2024-04-23T16:36:59.045018+0000","last_clean":"2024-04-23T16:36:59.045018+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:36:59.045018+0000","last_undegraded":"2024-04-23T16:36:59.045018+0000","last_fullsized":"2024-04-23T16:36:59.045018+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:32:29.517800+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:32:29.517800+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"447'19380","reported_seq":39044,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-23T16:36:59.184645+0000","last_change":"2024-04-23T16:36:55.340027+0000","last_active":"2024-04-23T16:36:59.184645+0000","last_peered":"2024-04-23T16:36:59.184645+0000","last_clean":"2024-04-23T16:36:59.184645+0000","last_became_active":"2024-04-23T16:33:26.458611+0000","last_became_peered":"2024-04-23T16:33:26.458611+0000","last_unstale":"2024-04-23T16:36:59.184645+0000","last_undegraded":"2024-04-23T16:36:59.184645+0000","last_fullsized":"2024-04-23T16:36:59.184645+0000","mapping_epoch":394,"log_start":"386'9300","ondisk_log_start":"386'9300","created":359,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8607","last_scrub_stamp":"2024-04-23T16:32:43.130806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.130806+0000","objects_scrubbed":8577,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19380,"num_write_kb":620160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9687,"num_bytes_recovered":634847232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1170,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-23T16:37:02.439665+0000","last_change":"2024-04-23T16:36:26.456131+0000","last_active":"2024-04-23T16:37:02.439665+0000","last_peered":"2024-04-23T16:37:02.439665+0000","last_clean":"2024-04-23T16:37:02.439665+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:37:02.439665+0000","last_undegraded":"2024-04-23T16:37:02.439665+0000","last_fullsized":"2024-04-23T16:37:02.439665+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:26.456003+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:26.456003+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:44.948237+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"447'18944","reported_seq":19234,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-23T16:36:59.184855+0000","last_change":"2024-04-23T16:33:18.576077+0000","last_active":"2024-04-23T16:36:59.184855+0000","last_peered":"2024-04-23T16:36:59.184855+0000","last_clean":"2024-04-23T16:36:59.184855+0000","last_became_active":"2024-04-23T16:31:29.139709+0000","last_became_peered":"2024-04-23T16:31:29.139709+0000","last_unstale":"2024-04-23T16:36:59.184855+0000","last_undegraded":"2024-04-23T16:36:59.184855+0000","last_fullsized":"2024-04-23T16:36:59.184855+0000","mapping_epoch":359,"log_start":"385'8900","ondisk_log_start":"385'8900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'9472","last_scrub_stamp":"2024-04-23T16:33:18.575945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:33:18.575945+0000","objects_scrubbed":9472,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18944,"num_write_kb":606208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1163,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-23T16:37:01.890270+0000","last_change":"2024-04-23T16:35:34.878630+0000","last_active":"2024-04-23T16:37:01.890270+0000","last_peered":"2024-04-23T16:37:01.890270+0000","last_clean":"2024-04-23T16:37:01.890270+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:37:01.890270+0000","last_undegraded":"2024-04-23T16:37:01.890270+0000","last_fullsized":"2024-04-23T16:37:01.890270+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:35:34.878536+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:35:34.878536+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:05.163611+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"447'19440","reported_seq":19842,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-23T16:37:02.439820+0000","last_change":"2024-04-23T16:36:20.849660+0000","last_active":"2024-04-23T16:37:02.439820+0000","last_peered":"2024-04-23T16:37:02.439820+0000","last_clean":"2024-04-23T16:37:02.439820+0000","last_became_active":"2024-04-23T16:31:29.139606+0000","last_became_peered":"2024-04-23T16:31:29.139606+0000","last_unstale":"2024-04-23T16:37:02.439820+0000","last_undegraded":"2024-04-23T16:37:02.439820+0000","last_fullsized":"2024-04-23T16:37:02.439820+0000","mapping_epoch":359,"log_start":"386'9400","ondisk_log_start":"386'9400","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'17441","last_scrub_stamp":"2024-04-23T16:36:20.849516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:36:20.849516+0000","objects_scrubbed":2009,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:50.602166+0000","scrub_duration":400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19440,"num_write_kb":622080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1170,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-23T16:37:02.440189+0000","last_change":"2024-04-23T16:36:23.239567+0000","last_active":"2024-04-23T16:37:02.440189+0000","last_peered":"2024-04-23T16:37:02.440189+0000","last_clean":"2024-04-23T16:37:02.440189+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:37:02.440189+0000","last_undegraded":"2024-04-23T16:37:02.440189+0000","last_fullsized":"2024-04-23T16:37:02.440189+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:23.239466+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:36:23.239466+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:27.031497+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.2","version":"447'18996","reported_seq":19428,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-23T16:37:03.210470+0000","last_change":"2024-04-23T16:37:03.210500+0000","last_active":"2024-04-23T16:37:03.210470+0000","last_peered":"2024-04-23T16:37:03.210470+0000","last_clean":"2024-04-23T16:37:03.210470+0000","last_became_active":"2024-04-23T16:31:29.138430+0000","last_became_peered":"2024-04-23T16:31:29.138430+0000","last_unstale":"2024-04-23T16:37:03.210470+0000","last_undegraded":"2024-04-23T16:37:03.210470+0000","last_fullsized":"2024-04-23T16:37:03.210470+0000","mapping_epoch":359,"log_start":"385'8900","ondisk_log_start":"385'8900","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"447'18996","last_scrub_stamp":"2024-04-23T16:37:03.210368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:37:03.210368+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:05.395128+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18996,"num_write_kb":607872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1170,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-23T16:37:02.440326+0000","last_change":"2024-04-23T16:36:18.085504+0000","last_active":"2024-04-23T16:37:02.440326+0000","last_peered":"2024-04-23T16:37:02.440326+0000","last_clean":"2024-04-23T16:37:02.440326+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:37:02.440326+0000","last_undegraded":"2024-04-23T16:37:02.440326+0000","last_fullsized":"2024-04-23T16:37:02.440326+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:18.085386+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:18.085386+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:25.354657+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"447'21402","reported_seq":21927,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-23T16:37:02.439789+0000","last_change":"2024-04-23T16:36:23.728326+0000","last_active":"2024-04-23T16:37:02.439789+0000","last_peered":"2024-04-23T16:37:02.439789+0000","last_clean":"2024-04-23T16:37:02.439789+0000","last_became_active":"2024-04-23T16:31:29.139398+0000","last_became_peered":"2024-04-23T16:31:29.139398+0000","last_unstale":"2024-04-23T16:37:02.439789+0000","last_undegraded":"2024-04-23T16:37:02.439789+0000","last_fullsized":"2024-04-23T16:37:02.439789+0000","mapping_epoch":359,"log_start":"386'11400","ondisk_log_start":"386'11400","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'19588","last_scrub_stamp":"2024-04-23T16:36:23.728231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:36:23.728231+0000","objects_scrubbed":1817,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:47.414183+0000","scrub_duration":340,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19128,"num_write_kb":612096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1163,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-23T16:37:01.890289+0000","last_change":"2024-04-23T16:35:33.875035+0000","last_active":"2024-04-23T16:37:01.890289+0000","last_peered":"2024-04-23T16:37:01.890289+0000","last_clean":"2024-04-23T16:37:01.890289+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:37:01.890289+0000","last_undegraded":"2024-04-23T16:37:01.890289+0000","last_fullsized":"2024-04-23T16:37:01.890289+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:35:33.874904+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:35:33.874904+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:05.163648+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"447'18894","reported_seq":19178,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-23T16:37:02.440351+0000","last_change":"2024-04-23T16:32:43.947374+0000","last_active":"2024-04-23T16:37:02.440351+0000","last_peered":"2024-04-23T16:37:02.440351+0000","last_clean":"2024-04-23T16:37:02.440351+0000","last_became_active":"2024-04-23T16:31:29.138968+0000","last_became_peered":"2024-04-23T16:31:29.138968+0000","last_unstale":"2024-04-23T16:37:02.440351+0000","last_undegraded":"2024-04-23T16:37:02.440351+0000","last_fullsized":"2024-04-23T16:37:02.440351+0000","mapping_epoch":359,"log_start":"384'8800","ondisk_log_start":"384'8800","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8482","last_scrub_stamp":"2024-04-23T16:32:43.947177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:43.947177+0000","objects_scrubbed":8428,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:02.578027+0000","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18894,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1060,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-23T16:37:01.748206+0000","last_change":"2024-04-23T16:36:29.341359+0000","last_active":"2024-04-23T16:37:01.748206+0000","last_peered":"2024-04-23T16:37:01.748206+0000","last_clean":"2024-04-23T16:37:01.748206+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:37:01.748206+0000","last_undegraded":"2024-04-23T16:37:01.748206+0000","last_fullsized":"2024-04-23T16:37:01.748206+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:29.341205+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:29.341205+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:50.871176+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"447'19462","reported_seq":19745,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-23T16:37:02.440419+0000","last_change":"2024-04-23T16:32:39.103230+0000","last_active":"2024-04-23T16:37:02.440419+0000","last_peered":"2024-04-23T16:37:02.440419+0000","last_clean":"2024-04-23T16:37:02.440419+0000","last_became_active":"2024-04-23T16:31:29.310246+0000","last_became_peered":"2024-04-23T16:31:29.310246+0000","last_unstale":"2024-04-23T16:37:02.440419+0000","last_undegraded":"2024-04-23T16:37:02.440419+0000","last_fullsized":"2024-04-23T16:37:02.440419+0000","mapping_epoch":359,"log_start":"386'9400","ondisk_log_start":"386'9400","created":359,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'8325","last_scrub_stamp":"2024-04-23T16:32:39.103059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:39.103059+0000","objects_scrubbed":8272,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:02.578010+0000","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19462,"num_write_kb":622784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":983,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-23T16:37:01.890209+0000","last_change":"2024-04-23T16:36:19.091622+0000","last_active":"2024-04-23T16:37:01.890209+0000","last_peered":"2024-04-23T16:37:01.890209+0000","last_clean":"2024-04-23T16:37:01.890209+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:37:01.890209+0000","last_undegraded":"2024-04-23T16:37:01.890209+0000","last_fullsized":"2024-04-23T16:37:01.890209+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:19.091490+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:36:19.091490+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:22.009913+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"447'19114","reported_seq":38505,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-23T16:36:59.184744+0000","last_change":"2024-04-23T16:36:48.984502+0000","last_active":"2024-04-23T16:36:59.184744+0000","last_peered":"2024-04-23T16:36:59.184744+0000","last_clean":"2024-04-23T16:36:59.184744+0000","last_became_active":"2024-04-23T16:33:26.078062+0000","last_became_peered":"2024-04-23T16:33:26.078062+0000","last_unstale":"2024-04-23T16:36:59.184744+0000","last_undegraded":"2024-04-23T16:36:59.184744+0000","last_fullsized":"2024-04-23T16:36:59.184744+0000","mapping_epoch":394,"log_start":"385'9100","ondisk_log_start":"385'9100","created":359,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'7968","last_scrub_stamp":"2024-04-23T16:32:36.102122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:31:28.121358+0000","last_clean_scrub_stamp":"2024-04-23T16:32:36.102122+0000","objects_scrubbed":7926,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19114,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9553,"num_bytes_recovered":626065408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1148,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-23T16:36:59.044964+0000","last_change":"2024-04-23T16:33:17.354409+0000","last_active":"2024-04-23T16:36:59.044964+0000","last_peered":"2024-04-23T16:36:59.044964+0000","last_clean":"2024-04-23T16:36:59.044964+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:36:59.044964+0000","last_undegraded":"2024-04-23T16:36:59.044964+0000","last_fullsized":"2024-04-23T16:36:59.044964+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:33:17.354263+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:33:17.354263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":307192,"num_write_kb":9830081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28952,"num_bytes_recovered":1897398272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1886846976,"data_stored":1886846976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160823,"ondisk_log_size":160823,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056901,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":501776,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93243376,"statfs":{"total":95995035648,"available":95481217024,"internally_reserved":0,"allocated":1380352,"data_stored":855778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":512424388},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351242,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612881,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":361260,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93383892,"statfs":{"total":95995035648,"available":95625105408,"internally_reserved":0,"allocated":1077248,"data_stored":540616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":368835008},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841813,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303276,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93441876,"statfs":{"total":95995035648,"available":95684481024,"internally_reserved":0,"allocated":1404928,"data_stored":868296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168857,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":405648,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":404606,"kb_avail":93339504,"statfs":{"total":95995035648,"available":95579652096,"internally_reserved":0,"allocated":1052672,"data_stored":528098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":414317002},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463200,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":284908,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93460244,"statfs":{"total":95995035648,"available":95703289856,"internally_reserved":0,"allocated":1077248,"data_stored":540616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":290650566},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137339,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738685,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":451500,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":93293652,"statfs":{"total":95995035648,"available":95532699648,"internally_reserved":0,"allocated":1077248,"data_stored":540616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117293,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":365592,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93379560,"statfs":{"total":95995035648,"available":95620669440,"internally_reserved":0,"allocated":1257472,"data_stored":729346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921064,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":577836,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93167316,"statfs":{"total":95995035648,"available":95403331584,"internally_reserved":0,"allocated":1077248,"data_stored":540616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757538,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":429464,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93315688,"statfs":{"total":95995035648,"available":95555264512,"internally_reserved":0,"allocated":1257472,"data_stored":729346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1886846976,"data_stored":1886846976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:37:08.993 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:37:08.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:08.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:37:09.063 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-23T16:37:09.099 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:37:09.133 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:09.153 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:37:09.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:09.157 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:37:09.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.256 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:09 smithi103 ceph-mon[39629]: pgmap v1278: 11 pgs: 11 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:09.256 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:09 smithi103 ceph-mon[39629]: osdmap e452: 12 total, 11 up, 9 in 2024-04-23T16:37:09.258 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:37:09.287 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:09.451 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:37:09.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:09.452 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:37:09.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:37:09.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:09 smithi188 ceph-mon[44101]: pgmap v1278: 11 pgs: 11 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:09.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:09 smithi188 ceph-mon[44101]: osdmap e452: 12 total, 11 up, 9 in 2024-04-23T16:37:09.553 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:37:09.583 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.583 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.583 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.583 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.583 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.583 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.583 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.583 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.583 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.583 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.583 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:09.583 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:09 smithi148 ceph-mon[44520]: pgmap v1278: 11 pgs: 11 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:09.584 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:09 smithi148 ceph-mon[44520]: osdmap e452: 12 total, 11 up, 9 in 2024-04-23T16:37:09.585 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:09.607 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:37:09.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:09.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:37:09.711 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:37:09.728 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:37:09.728 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:37:09.739 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:09.758 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:37:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:09.761 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:37:09.862 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:09.890 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:10.087 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:10.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:10.090 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:10.191 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:10.224 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:10.246 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:10.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:10.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:10.250 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:10.350 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:37:10.378 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:10.577 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:37:10.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:10.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:37:10.680 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:37:10.713 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:11.130 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:37:11.130 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-04-23T16:37:11.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-04-23T16:37:11.131 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:37:11.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.131 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:160761 5.a 5:50bf586a:::benchmark_data_smithi188_113957_object7163:head [delete] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-04-23T16:37:11.131 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:38.751162+0000", 2024-04-23T16:37:11.131 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 212.375341572, 2024-04-23T16:37:11.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17909995200000001, 2024-04-23T16:37:11.131 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.131 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.131 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.131 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.131 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.131 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.131 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 160761 2024-04-23T16:37:11.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.132 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.751162+0000", 2024-04-23T16:37:11.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.751162+0000", 2024-04-23T16:37:11.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.751163+0000", 2024-04-23T16:37:11.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.415e-06 2024-04-23T16:37:11.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.751168+0000", 2024-04-23T16:37:11.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6550000000000003e-06 2024-04-23T16:37:11.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.751177+0000", 2024-04-23T16:37:11.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3249999999999997e-06 2024-04-23T16:37:11.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.751185+0000", 2024-04-23T16:37:11.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8180000000000004e-06 2024-04-23T16:37:11.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.753098+0000", 2024-04-23T16:37:11.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00191292 2024-04-23T16:37:11.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.753383+0000", 2024-04-23T16:37:11.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00028443000000000001 2024-04-23T16:37:11.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.838181+0000", 2024-04-23T16:37:11.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.084798011000000006 2024-04-23T16:37:11.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.838272+0000", 2024-04-23T16:37:11.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1055000000000001e-05 2024-04-23T16:37:11.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.838402+0000", 2024-04-23T16:37:11.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013013799999999999 2024-04-23T16:37:11.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.864331+0000", 2024-04-23T16:37:11.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025929526000000001 2024-04-23T16:37:11.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.927147+0000", 2024-04-23T16:37:11.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062815864999999999 2024-04-23T16:37:11.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.927182+0000", 2024-04-23T16:37:11.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5064999999999997e-05 2024-04-23T16:37:11.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.930262+0000", 2024-04-23T16:37:11.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030797289999999998 2024-04-23T16:37:11.138 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.138 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.138 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.138 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:160764 5.a 5:5179712e:::benchmark_data_smithi188_113957_object7166:head [delete] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-04-23T16:37:11.138 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:38.756328+0000", 2024-04-23T16:37:11.138 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 212.37017544400001, 2024-04-23T16:37:11.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17681493700000001, 2024-04-23T16:37:11.138 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.138 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.138 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.139 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.139 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.139 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.139 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 160764 2024-04-23T16:37:11.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.139 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756328+0000", 2024-04-23T16:37:11.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756328+0000", 2024-04-23T16:37:11.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756330+0000", 2024-04-23T16:37:11.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.013e-06 2024-04-23T16:37:11.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756335+0000", 2024-04-23T16:37:11.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2024-04-23T16:37:11.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756345+0000", 2024-04-23T16:37:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0122000000000001e-05 2024-04-23T16:37:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756351+0000", 2024-04-23T16:37:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6740000000000002e-06 2024-04-23T16:37:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756393+0000", 2024-04-23T16:37:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2258000000000001e-05 2024-04-23T16:37:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756643+0000", 2024-04-23T16:37:11.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000249497 2024-04-23T16:37:11.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.847557+0000", 2024-04-23T16:37:11.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.090914377000000005 2024-04-23T16:37:11.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.847648+0000", 2024-04-23T16:37:11.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0999000000000001e-05 2024-04-23T16:37:11.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.847805+0000", 2024-04-23T16:37:11.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015637100000000001 2024-04-23T16:37:11.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.864339+0000", 2024-04-23T16:37:11.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016534505000000001 2024-04-23T16:37:11.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.930330+0000", 2024-04-23T16:37:11.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.065990368999999993 2024-04-23T16:37:11.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.930385+0000", 2024-04-23T16:37:11.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5630000000000001e-05 2024-04-23T16:37:11.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.933143+0000", 2024-04-23T16:37:11.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027578680000000001 2024-04-23T16:37:11.144 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.144 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.144 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.145 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:160766 5.a 5:588abbaa:::benchmark_data_smithi188_113957_object7168:head [delete] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-04-23T16:37:11.145 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:38.757349+0000", 2024-04-23T16:37:11.145 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 212.369154901, 2024-04-23T16:37:11.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17851373300000001, 2024-04-23T16:37:11.145 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.145 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.145 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.145 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.145 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.145 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.145 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 160766 2024-04-23T16:37:11.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.145 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757349+0000", 2024-04-23T16:37:11.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757349+0000", 2024-04-23T16:37:11.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757350+0000", 2024-04-23T16:37:11.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0419999999999999e-06 2024-04-23T16:37:11.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757353+0000", 2024-04-23T16:37:11.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7790000000000002e-06 2024-04-23T16:37:11.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757361+0000", 2024-04-23T16:37:11.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4769999999999997e-06 2024-04-23T16:37:11.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757366+0000", 2024-04-23T16:37:11.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5550000000000002e-06 2024-04-23T16:37:11.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757405+0000", 2024-04-23T16:37:11.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8531000000000001e-05 2024-04-23T16:37:11.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757654+0000", 2024-04-23T16:37:11.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000249132 2024-04-23T16:37:11.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.858043+0000", 2024-04-23T16:37:11.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.100388963 2024-04-23T16:37:11.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.858117+0000", 2024-04-23T16:37:11.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4305000000000001e-05 2024-04-23T16:37:11.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.858202+0000", 2024-04-23T16:37:11.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4151000000000001e-05 2024-04-23T16:37:11.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.868822+0000", 2024-04-23T16:37:11.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010620076000000001 2024-04-23T16:37:11.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.933212+0000", 2024-04-23T16:37:11.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.064390728999999994 2024-04-23T16:37:11.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.933272+0000", 2024-04-23T16:37:11.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9135000000000002e-05 2024-04-23T16:37:11.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.935862+0000", 2024-04-23T16:37:11.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025908580000000001 2024-04-23T16:37:11.151 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.151 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.151 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.151 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:183867 5.a 5:5ad9927f:::benchmark_data_smithi188_113957_object30269:head [delete] snapc 0=[] ondisk+write+known_if_redirected e405)", 2024-04-23T16:37:11.151 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:14.979043+0000", 2024-04-23T16:37:11.151 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 176.147460314, 2024-04-23T16:37:11.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.21871197000000001, 2024-04-23T16:37:11.151 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.151 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.152 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.152 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.152 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.152 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.152 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 183867 2024-04-23T16:37:11.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.152 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979043+0000", 2024-04-23T16:37:11.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979043+0000", 2024-04-23T16:37:11.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979047+0000", 2024-04-23T16:37:11.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.422e-06 2024-04-23T16:37:11.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979058+0000", 2024-04-23T16:37:11.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1155000000000001e-05 2024-04-23T16:37:11.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979074+0000", 2024-04-23T16:37:11.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6198999999999999e-05 2024-04-23T16:37:11.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979085+0000", 2024-04-23T16:37:11.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0954e-05 2024-04-23T16:37:11.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.057904+0000", 2024-04-23T16:37:11.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078818887000000004 2024-04-23T16:37:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.057945+0000", 2024-04-23T16:37:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1523999999999997e-05 2024-04-23T16:37:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.063469+0000", 2024-04-23T16:37:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055238930000000002 2024-04-23T16:37:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.063542+0000", 2024-04-23T16:37:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2451000000000001e-05 2024-04-23T16:37:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.063635+0000", 2024-04-23T16:37:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3566999999999994e-05 2024-04-23T16:37:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.080527+0000", 2024-04-23T16:37:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016891587 2024-04-23T16:37:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.195578+0000", 2024-04-23T16:37:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.11505106700000001 2024-04-23T16:37:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.195620+0000", 2024-04-23T16:37:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2385000000000002e-05 2024-04-23T16:37:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.197755+0000", 2024-04-23T16:37:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021348790000000001 2024-04-23T16:37:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.158 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:183868 5.a 5:54aa4085:::benchmark_data_smithi188_113957_object30270:head [delete] snapc 0=[] ondisk+write+known_if_redirected e405)", 2024-04-23T16:37:11.158 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:14.979944+0000", 2024-04-23T16:37:11.158 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 176.14655947899999, 2024-04-23T16:37:11.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.221022303, 2024-04-23T16:37:11.158 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.158 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.158 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.158 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.158 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.158 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.158 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 183868 2024-04-23T16:37:11.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.159 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979944+0000", 2024-04-23T16:37:11.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979944+0000", 2024-04-23T16:37:11.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979945+0000", 2024-04-23T16:37:11.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.373e-06 2024-04-23T16:37:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979951+0000", 2024-04-23T16:37:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4999999999999999e-06 2024-04-23T16:37:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979963+0000", 2024-04-23T16:37:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.182e-05 2024-04-23T16:37:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979970+0000", 2024-04-23T16:37:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6729999999999992e-06 2024-04-23T16:37:11.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.057974+0000", 2024-04-23T16:37:11.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078003804999999996 2024-04-23T16:37:11.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.058286+0000", 2024-04-23T16:37:11.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031136500000000002 2024-04-23T16:37:11.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.090326+0000", 2024-04-23T16:37:11.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032040043999999997 2024-04-23T16:37:11.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.090395+0000", 2024-04-23T16:37:11.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8946e-05 2024-04-23T16:37:11.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.090483+0000", 2024-04-23T16:37:11.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8832000000000006e-05 2024-04-23T16:37:11.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.109765+0000", 2024-04-23T16:37:11.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019281868000000001 2024-04-23T16:37:11.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.197826+0000", 2024-04-23T16:37:11.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.088060787000000001 2024-04-23T16:37:11.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.197882+0000", 2024-04-23T16:37:11.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5770999999999998e-05 2024-04-23T16:37:11.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.200966+0000", 2024-04-23T16:37:11.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003084519 2024-04-23T16:37:11.164 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.164 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.164 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.164 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:183875 5.a 5:56b558c0:::benchmark_data_smithi188_113957_object30277:head [delete] snapc 0=[] ondisk+write+known_if_redirected e405)", 2024-04-23T16:37:11.164 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:14.999858+0000", 2024-04-23T16:37:11.164 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 176.12664589400001, 2024-04-23T16:37:11.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.20427009400000001, 2024-04-23T16:37:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 183875 2024-04-23T16:37:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.999858+0000", 2024-04-23T16:37:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.999858+0000", 2024-04-23T16:37:11.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.999860+0000", 2024-04-23T16:37:11.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7460000000000001e-06 2024-04-23T16:37:11.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.999871+0000", 2024-04-23T16:37:11.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0336000000000001e-05 2024-04-23T16:37:11.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.999889+0000", 2024-04-23T16:37:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8607999999999999e-05 2024-04-23T16:37:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.999898+0000", 2024-04-23T16:37:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.276e-06 2024-04-23T16:37:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.058319+0000", 2024-04-23T16:37:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.058421147 2024-04-23T16:37:11.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.058550+0000", 2024-04-23T16:37:11.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00023164999999999999 2024-04-23T16:37:11.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.097650+0000", 2024-04-23T16:37:11.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.039099606000000002 2024-04-23T16:37:11.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.097751+0000", 2024-04-23T16:37:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010077300000000001 2024-04-23T16:37:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.097893+0000", 2024-04-23T16:37:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014181600000000001 2024-04-23T16:37:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.120527+0000", 2024-04-23T16:37:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022634399999999999 2024-04-23T16:37:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.201049+0000", 2024-04-23T16:37:11.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.080522257999999999 2024-04-23T16:37:11.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.201126+0000", 2024-04-23T16:37:11.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6848999999999996e-05 2024-04-23T16:37:11.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.204128+0000", 2024-04-23T16:37:11.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003001629 2024-04-23T16:37:11.170 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.171 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.171 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.171 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:183876 5.a 5:52b8ecf9:::benchmark_data_smithi188_113957_object30278:head [delete] snapc 0=[] ondisk+write+known_if_redirected e405)", 2024-04-23T16:37:11.171 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:15.000673+0000", 2024-04-23T16:37:11.171 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 176.12583080300001, 2024-04-23T16:37:11.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.21436903600000001, 2024-04-23T16:37:11.171 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.171 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.171 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.171 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.171 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.172 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.172 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 183876 2024-04-23T16:37:11.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.172 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.000673+0000", 2024-04-23T16:37:11.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.000673+0000", 2024-04-23T16:37:11.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.000674+0000", 2024-04-23T16:37:11.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7370000000000001e-06 2024-04-23T16:37:11.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.000681+0000", 2024-04-23T16:37:11.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7939999999999997e-06 2024-04-23T16:37:11.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.000693+0000", 2024-04-23T16:37:11.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1780000000000001e-05 2024-04-23T16:37:11.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.000703+0000", 2024-04-23T16:37:11.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5589999999999994e-06 2024-04-23T16:37:11.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.058583+0000", 2024-04-23T16:37:11.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.057880193000000003 2024-04-23T16:37:11.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.058799+0000", 2024-04-23T16:37:11.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021655599999999999 2024-04-23T16:37:11.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.106244+0000", 2024-04-23T16:37:11.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.047444457000000002 2024-04-23T16:37:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.106325+0000", 2024-04-23T16:37:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1497999999999998e-05 2024-04-23T16:37:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.106417+0000", 2024-04-23T16:37:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1558999999999995e-05 2024-04-23T16:37:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.124887+0000", 2024-04-23T16:37:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018470073 2024-04-23T16:37:11.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.212626+0000", 2024-04-23T16:37:11.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.087738732999999999 2024-04-23T16:37:11.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.212684+0000", 2024-04-23T16:37:11.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8588000000000002e-05 2024-04-23T16:37:11.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.215042+0000", 2024-04-23T16:37:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023575089999999998 2024-04-23T16:37:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:183879 5.a 5:530c738f:::benchmark_data_smithi188_113957_object30281:head [delete] snapc 0=[] ondisk+write+known_if_redirected e405)", 2024-04-23T16:37:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:15.015869+0000", 2024-04-23T16:37:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 176.11063434799999, 2024-04-23T16:37:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.202583191, 2024-04-23T16:37:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.179 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.179 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.179 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 183879 2024-04-23T16:37:11.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.179 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.015869+0000", 2024-04-23T16:37:11.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.015869+0000", 2024-04-23T16:37:11.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.015872+0000", 2024-04-23T16:37:11.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.689e-06 2024-04-23T16:37:11.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.015882+0000", 2024-04-23T16:37:11.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9159999999999996e-06 2024-04-23T16:37:11.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.015896+0000", 2024-04-23T16:37:11.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4431e-05 2024-04-23T16:37:11.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.015905+0000", 2024-04-23T16:37:11.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1989999999999995e-06 2024-04-23T16:37:11.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.058831+0000", 2024-04-23T16:37:11.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.042925824000000001 2024-04-23T16:37:11.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.059046+0000", 2024-04-23T16:37:11.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021425400000000001 2024-04-23T16:37:11.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.132876+0000", 2024-04-23T16:37:11.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.073830940999999997 2024-04-23T16:37:11.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.132936+0000", 2024-04-23T16:37:11.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9076999999999997e-05 2024-04-23T16:37:11.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.133011+0000", 2024-04-23T16:37:11.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5319000000000002e-05 2024-04-23T16:37:11.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.146665+0000", 2024-04-23T16:37:11.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013653748 2024-04-23T16:37:11.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.216832+0000", 2024-04-23T16:37:11.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.070167577999999994 2024-04-23T16:37:11.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.216886+0000", 2024-04-23T16:37:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3579999999999999e-05 2024-04-23T16:37:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.218452+0000", 2024-04-23T16:37:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001566635 2024-04-23T16:37:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:183883 5.a 5:561ddaeb:::benchmark_data_smithi188_113957_object30285:head [delete] snapc 0=[] ondisk+write+known_if_redirected e405)", 2024-04-23T16:37:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:15.030160+0000", 2024-04-23T16:37:11.185 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 176.096343612, 2024-04-23T16:37:11.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18660992900000001, 2024-04-23T16:37:11.185 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.185 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.185 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.185 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.185 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.185 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.185 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 183883 2024-04-23T16:37:11.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.185 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.030160+0000", 2024-04-23T16:37:11.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.030160+0000", 2024-04-23T16:37:11.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.030162+0000", 2024-04-23T16:37:11.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2220000000000001e-06 2024-04-23T16:37:11.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.030170+0000", 2024-04-23T16:37:11.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6890000000000004e-06 2024-04-23T16:37:11.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.030184+0000", 2024-04-23T16:37:11.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3862999999999999e-05 2024-04-23T16:37:11.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.030193+0000", 2024-04-23T16:37:11.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1800000000000002e-06 2024-04-23T16:37:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.059080+0000", 2024-04-23T16:37:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028887025 2024-04-23T16:37:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.059307+0000", 2024-04-23T16:37:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00022688600000000001 2024-04-23T16:37:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.129074+0000", 2024-04-23T16:37:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.069767423999999995 2024-04-23T16:37:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.129193+0000", 2024-04-23T16:37:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118529 2024-04-23T16:37:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.129348+0000", 2024-04-23T16:37:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000155284 2024-04-23T16:37:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.136011+0000", 2024-04-23T16:37:11.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0066625340000000003 2024-04-23T16:37:11.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.215122+0000", 2024-04-23T16:37:11.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.079111329999999994 2024-04-23T16:37:11.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.215179+0000", 2024-04-23T16:37:11.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7132999999999997e-05 2024-04-23T16:37:11.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.216770+0000", 2024-04-23T16:37:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0015908299999999999 2024-04-23T16:37:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:183888 5.a 5:5846d6fa:::benchmark_data_smithi188_113957_object30290:head [delete] snapc 0=[] ondisk+write+known_if_redirected e405)", 2024-04-23T16:37:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:15.045806+0000", 2024-04-23T16:37:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 176.080697953, 2024-04-23T16:37:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17485758000000001, 2024-04-23T16:37:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 183888 2024-04-23T16:37:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.045806+0000", 2024-04-23T16:37:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.045806+0000", 2024-04-23T16:37:11.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.045808+0000", 2024-04-23T16:37:11.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4980000000000001e-06 2024-04-23T16:37:11.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.045816+0000", 2024-04-23T16:37:11.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7360000000000001e-06 2024-04-23T16:37:11.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.045828+0000", 2024-04-23T16:37:11.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1824000000000001e-05 2024-04-23T16:37:11.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.045837+0000", 2024-04-23T16:37:11.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4520000000000003e-06 2024-04-23T16:37:11.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.059339+0000", 2024-04-23T16:37:11.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013501793 2024-04-23T16:37:11.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.059583+0000", 2024-04-23T16:37:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024406800000000001 2024-04-23T16:37:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.150992+0000", 2024-04-23T16:37:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.091408732000000006 2024-04-23T16:37:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.151067+0000", 2024-04-23T16:37:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5331000000000001e-05 2024-04-23T16:37:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.151169+0000", 2024-04-23T16:37:11.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101588 2024-04-23T16:37:11.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.164918+0000", 2024-04-23T16:37:11.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01374977 2024-04-23T16:37:11.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.218521+0000", 2024-04-23T16:37:11.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.053602859000000003 2024-04-23T16:37:11.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.218580+0000", 2024-04-23T16:37:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8273000000000003e-05 2024-04-23T16:37:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.220663+0000", 2024-04-23T16:37:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020836560000000001 2024-04-23T16:37:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:183892 5.a 5:5cedc568:::benchmark_data_smithi188_113957_object30294:head [delete] snapc 0=[] ondisk+write+known_if_redirected e405)", 2024-04-23T16:37:11.198 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:15.052916+0000", 2024-04-23T16:37:11.198 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 176.07358726699999, 2024-04-23T16:37:11.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17005552800000001, 2024-04-23T16:37:11.198 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.198 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.198 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.198 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.198 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.198 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.198 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 183892 2024-04-23T16:37:11.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.198 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.052916+0000", 2024-04-23T16:37:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.052916+0000", 2024-04-23T16:37:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.052919+0000", 2024-04-23T16:37:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.238e-06 2024-04-23T16:37:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.052924+0000", 2024-04-23T16:37:11.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4890000000000003e-06 2024-04-23T16:37:11.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.052936+0000", 2024-04-23T16:37:11.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1635e-05 2024-04-23T16:37:11.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.052945+0000", 2024-04-23T16:37:11.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3649999999999993e-06 2024-04-23T16:37:11.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.059618+0000", 2024-04-23T16:37:11.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0066725029999999998 2024-04-23T16:37:11.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.059833+0000", 2024-04-23T16:37:11.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021506800000000001 2024-04-23T16:37:11.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.170197+0000", 2024-04-23T16:37:11.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.110364221 2024-04-23T16:37:11.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.170272+0000", 2024-04-23T16:37:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5129999999999994e-05 2024-04-23T16:37:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.170371+0000", 2024-04-23T16:37:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9263000000000002e-05 2024-04-23T16:37:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.188683+0000", 2024-04-23T16:37:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018311874999999998 2024-04-23T16:37:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.220832+0000", 2024-04-23T16:37:11.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032148941 2024-04-23T16:37:11.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.220866+0000", 2024-04-23T16:37:11.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4357999999999999e-05 2024-04-23T16:37:11.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.222972+0000", 2024-04-23T16:37:11.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021054419999999999 2024-04-23T16:37:11.204 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.204 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.204 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.204 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:183896 5.a 5:56e6e14d:::benchmark_data_smithi188_113957_object30298:head [delete] snapc 0=[] ondisk+write+known_if_redirected e405)", 2024-04-23T16:37:11.204 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:15.061396+0000", 2024-04-23T16:37:11.204 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 176.065107474, 2024-04-23T16:37:11.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.21362087199999999, 2024-04-23T16:37:11.204 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.205 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.205 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.205 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.205 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.205 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.205 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 183896 2024-04-23T16:37:11.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.205 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.061396+0000", 2024-04-23T16:37:11.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.061396+0000", 2024-04-23T16:37:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.061398+0000", 2024-04-23T16:37:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6130000000000001e-06 2024-04-23T16:37:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.061403+0000", 2024-04-23T16:37:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1270000000000002e-06 2024-04-23T16:37:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.061411+0000", 2024-04-23T16:37:11.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3469999999999999e-06 2024-04-23T16:37:11.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.061418+0000", 2024-04-23T16:37:11.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8889999999999996e-06 2024-04-23T16:37:11.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.080478+0000", 2024-04-23T16:37:11.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019059646 2024-04-23T16:37:11.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.080517+0000", 2024-04-23T16:37:11.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8822999999999997e-05 2024-04-23T16:37:11.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.250424+0000", 2024-04-23T16:37:11.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.169907683 2024-04-23T16:37:11.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.250535+0000", 2024-04-23T16:37:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000111051 2024-04-23T16:37:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.250694+0000", 2024-04-23T16:37:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015892400000000001 2024-04-23T16:37:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.268933+0000", 2024-04-23T16:37:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018238800999999999 2024-04-23T16:37:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.272484+0000", 2024-04-23T16:37:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003551261 2024-04-23T16:37:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.272519+0000", 2024-04-23T16:37:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5141000000000001e-05 2024-04-23T16:37:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.275017+0000", 2024-04-23T16:37:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024975660000000001 2024-04-23T16:37:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:183942 5.a 5:53ac576d:::benchmark_data_smithi188_113957_object30344:head [delete] snapc 0=[] ondisk+write+known_if_redirected e405)", 2024-04-23T16:37:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:15.157233+0000", 2024-04-23T16:37:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 175.96927029400001, 2024-04-23T16:37:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.16180805400000001, 2024-04-23T16:37:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 183942 2024-04-23T16:37:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.157233+0000", 2024-04-23T16:37:11.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.157233+0000", 2024-04-23T16:37:11.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.157235+0000", 2024-04-23T16:37:11.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.294e-06 2024-04-23T16:37:11.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.157240+0000", 2024-04-23T16:37:11.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0499999999999999e-06 2024-04-23T16:37:11.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.157249+0000", 2024-04-23T16:37:11.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6930000000000002e-06 2024-04-23T16:37:11.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.214 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.157255+0000", 2024-04-23T16:37:11.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3349999999999997e-06 2024-04-23T16:37:11.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.214 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.231828+0000", 2024-04-23T16:37:11.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.074573766999999999 2024-04-23T16:37:11.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.214 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.231905+0000", 2024-04-23T16:37:11.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6188999999999998e-05 2024-04-23T16:37:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.283260+0000", 2024-04-23T16:37:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.051355158999999997 2024-04-23T16:37:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.283360+0000", 2024-04-23T16:37:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100479 2024-04-23T16:37:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.283516+0000", 2024-04-23T16:37:11.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015542099999999999 2024-04-23T16:37:11.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.304698+0000", 2024-04-23T16:37:11.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021181987999999999 2024-04-23T16:37:11.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.316307+0000", 2024-04-23T16:37:11.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011609408 2024-04-23T16:37:11.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.316373+0000", 2024-04-23T16:37:11.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6344e-05 2024-04-23T16:37:11.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.319041+0000", 2024-04-23T16:37:11.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026679270000000001 2024-04-23T16:37:11.217 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.217 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.217 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.217 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:214988 5.a 5:5a3f8f01:::benchmark_data_smithi188_113957_object61390:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-23T16:37:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:35:01.259340+0000", 2024-04-23T16:37:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 129.86716332500001, 2024-04-23T16:37:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.181216351, 2024-04-23T16:37:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 214988 2024-04-23T16:37:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.259340+0000", 2024-04-23T16:37:11.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.219 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.259340+0000", 2024-04-23T16:37:11.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.219 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.259342+0000", 2024-04-23T16:37:11.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4419999999999999e-06 2024-04-23T16:37:11.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.220 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.259348+0000", 2024-04-23T16:37:11.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8950000000000003e-06 2024-04-23T16:37:11.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.220 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.259357+0000", 2024-04-23T16:37:11.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3570000000000004e-06 2024-04-23T16:37:11.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.259365+0000", 2024-04-23T16:37:11.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7260000000000002e-06 2024-04-23T16:37:11.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.275330+0000", 2024-04-23T16:37:11.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01596537 2024-04-23T16:37:11.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.275768+0000", 2024-04-23T16:37:11.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043770799999999998 2024-04-23T16:37:11.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.347968+0000", 2024-04-23T16:37:11.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.072199812000000002 2024-04-23T16:37:11.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.348040+0000", 2024-04-23T16:37:11.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2589000000000006e-05 2024-04-23T16:37:11.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.348130+0000", 2024-04-23T16:37:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9439999999999997e-05 2024-04-23T16:37:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.364397+0000", 2024-04-23T16:37:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016267674999999999 2024-04-23T16:37:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.437459+0000", 2024-04-23T16:37:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.073061423 2024-04-23T16:37:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.224 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.437521+0000", 2024-04-23T16:37:11.224 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2776999999999999e-05 2024-04-23T16:37:11.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.224 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.224 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.440557+0000", 2024-04-23T16:37:11.224 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030351369999999998 2024-04-23T16:37:11.224 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.224 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.224 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.224 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:214994 5.a 5:5c3b6503:::benchmark_data_smithi188_113957_object61396:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-23T16:37:11.224 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:35:01.260876+0000", 2024-04-23T16:37:11.225 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 129.86562773200001, 2024-04-23T16:37:11.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18260874299999999, 2024-04-23T16:37:11.225 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.225 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.225 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.225 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.225 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.225 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.225 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 214994 2024-04-23T16:37:11.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.225 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.225 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.260876+0000", 2024-04-23T16:37:11.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.226 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.260876+0000", 2024-04-23T16:37:11.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.226 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.260877+0000", 2024-04-23T16:37:11.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4139999999999999e-06 2024-04-23T16:37:11.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.226 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.260882+0000", 2024-04-23T16:37:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0889999999999999e-06 2024-04-23T16:37:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.260891+0000", 2024-04-23T16:37:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7790000000000001e-06 2024-04-23T16:37:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.260898+0000", 2024-04-23T16:37:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1099999999999997e-06 2024-04-23T16:37:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.275819+0000", 2024-04-23T16:37:11.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014921228999999999 2024-04-23T16:37:11.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276334+0000", 2024-04-23T16:37:11.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00051455200000000004 2024-04-23T16:37:11.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.356496+0000", 2024-04-23T16:37:11.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.080162232999999999 2024-04-23T16:37:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.356587+0000", 2024-04-23T16:37:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0388000000000003e-05 2024-04-23T16:37:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.356707+0000", 2024-04-23T16:37:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000120189 2024-04-23T16:37:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.372935+0000", 2024-04-23T16:37:11.230 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016228604000000001 2024-04-23T16:37:11.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.230 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.230 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.440628+0000", 2024-04-23T16:37:11.230 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067692691999999999 2024-04-23T16:37:11.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.230 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.230 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.440689+0000", 2024-04-23T16:37:11.230 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0781e-05 2024-04-23T16:37:11.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.230 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.231 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.443485+0000", 2024-04-23T16:37:11.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027956830000000002 2024-04-23T16:37:11.231 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.231 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.234 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.234 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:214998 5.a 5:5aa06674:::benchmark_data_smithi188_113957_object61400:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-23T16:37:11.234 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:35:01.271851+0000", 2024-04-23T16:37:11.234 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 129.85465229799999, 2024-04-23T16:37:11.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17414512200000001, 2024-04-23T16:37:11.234 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.234 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.234 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.234 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.234 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.234 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.235 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 214998 2024-04-23T16:37:11.235 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.235 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.271851+0000", 2024-04-23T16:37:11.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.235 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.236 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.271851+0000", 2024-04-23T16:37:11.236 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.236 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.236 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.271853+0000", 2024-04-23T16:37:11.236 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2280000000000001e-06 2024-04-23T16:37:11.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.236 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.236 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.271860+0000", 2024-04-23T16:37:11.236 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8770000000000004e-06 2024-04-23T16:37:11.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.271871+0000", 2024-04-23T16:37:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0287e-05 2024-04-23T16:37:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.271881+0000", 2024-04-23T16:37:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0202e-05 2024-04-23T16:37:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276381+0000", 2024-04-23T16:37:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004499818 2024-04-23T16:37:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.238 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.238 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276905+0000", 2024-04-23T16:37:11.238 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00052421599999999998 2024-04-23T16:37:11.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.238 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.238 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.364455+0000", 2024-04-23T16:37:11.238 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.087549765000000002 2024-04-23T16:37:11.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.238 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.238 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.364549+0000", 2024-04-23T16:37:11.238 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4581999999999997e-05 2024-04-23T16:37:11.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.364655+0000", 2024-04-23T16:37:11.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105801 2024-04-23T16:37:11.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.380441+0000", 2024-04-23T16:37:11.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015786372 2024-04-23T16:37:11.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.443553+0000", 2024-04-23T16:37:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.063111492000000005 2024-04-23T16:37:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.443605+0000", 2024-04-23T16:37:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2460999999999998e-05 2024-04-23T16:37:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.445996+0000", 2024-04-23T16:37:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023910210000000001 2024-04-23T16:37:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.241 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:215000 5.a 5:5746befc:::benchmark_data_smithi188_113957_object61402:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-23T16:37:11.241 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:35:01.276226+0000", 2024-04-23T16:37:11.241 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 129.850277554, 2024-04-23T16:37:11.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17179708599999999, 2024-04-23T16:37:11.241 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.241 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.241 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.241 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.241 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.241 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.241 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 215000 2024-04-23T16:37:11.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.241 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276226+0000", 2024-04-23T16:37:11.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276226+0000", 2024-04-23T16:37:11.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276228+0000", 2024-04-23T16:37:11.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0389999999999999e-06 2024-04-23T16:37:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276234+0000", 2024-04-23T16:37:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.049e-06 2024-04-23T16:37:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276245+0000", 2024-04-23T16:37:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0448000000000001e-05 2024-04-23T16:37:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276252+0000", 2024-04-23T16:37:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4529999999999996e-06 2024-04-23T16:37:11.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276950+0000", 2024-04-23T16:37:11.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000698396 2024-04-23T16:37:11.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.277470+0000", 2024-04-23T16:37:11.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00051993799999999995 2024-04-23T16:37:11.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.372985+0000", 2024-04-23T16:37:11.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.095515172999999995 2024-04-23T16:37:11.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.373089+0000", 2024-04-23T16:37:11.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010385 2024-04-23T16:37:11.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.373186+0000", 2024-04-23T16:37:11.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6778000000000001e-05 2024-04-23T16:37:11.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.392441+0000", 2024-04-23T16:37:11.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019254513000000001 2024-04-23T16:37:11.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.246 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.446064+0000", 2024-04-23T16:37:11.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.053623404999999999 2024-04-23T16:37:11.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.246 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.446125+0000", 2024-04-23T16:37:11.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0590000000000001e-05 2024-04-23T16:37:11.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.448023+0000", 2024-04-23T16:37:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018984539999999999 2024-04-23T16:37:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:215010 5.a 5:5fcdf1a9:::benchmark_data_smithi188_113957_object61412:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-23T16:37:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:35:01.290493+0000", 2024-04-23T16:37:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 129.83601010800001, 2024-04-23T16:37:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.182786068, 2024-04-23T16:37:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.248 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.248 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.248 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.248 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 215010 2024-04-23T16:37:11.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.248 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.248 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.290493+0000", 2024-04-23T16:37:11.248 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.290493+0000", 2024-04-23T16:37:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.290495+0000", 2024-04-23T16:37:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9910000000000001e-06 2024-04-23T16:37:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.290502+0000", 2024-04-23T16:37:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3289999999999999e-06 2024-04-23T16:37:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.290511+0000", 2024-04-23T16:37:11.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5380000000000008e-06 2024-04-23T16:37:11.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.290518+0000", 2024-04-23T16:37:11.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.601e-06 2024-04-23T16:37:11.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.290572+0000", 2024-04-23T16:37:11.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4162000000000001e-05 2024-04-23T16:37:11.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.290615+0000", 2024-04-23T16:37:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2809000000000003e-05 2024-04-23T16:37:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.414270+0000", 2024-04-23T16:37:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.123654695 2024-04-23T16:37:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.414372+0000", 2024-04-23T16:37:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000102878 2024-04-23T16:37:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.414511+0000", 2024-04-23T16:37:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000138992 2024-04-23T16:37:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.429033+0000", 2024-04-23T16:37:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014521729000000001 2024-04-23T16:37:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.470520+0000", 2024-04-23T16:37:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.041487201000000001 2024-04-23T16:37:11.253 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.253 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.253 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.470578+0000", 2024-04-23T16:37:11.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8050999999999997e-05 2024-04-23T16:37:11.253 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.253 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.253 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.473280+0000", 2024-04-23T16:37:11.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002701092 2024-04-23T16:37:11.253 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.253 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.253 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.253 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.254 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:251104 5.a 5:56b6a247:::benchmark_data_smithi188_113957_object97506:head [delete] snapc 0=[] ondisk+write+known_if_redirected e426)", 2024-04-23T16:37:11.254 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:35:49.743007+0000", 2024-04-23T16:37:11.254 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 81.383496053000002, 2024-04-23T16:37:11.254 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18092238999999999, 2024-04-23T16:37:11.254 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.254 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.254 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.254 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.254 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.254 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.254 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 251104 2024-04-23T16:37:11.254 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.743007+0000", 2024-04-23T16:37:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.743007+0000", 2024-04-23T16:37:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.743009+0000", 2024-04-23T16:37:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9850000000000001e-06 2024-04-23T16:37:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.743015+0000", 2024-04-23T16:37:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6740000000000002e-06 2024-04-23T16:37:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.743027+0000", 2024-04-23T16:37:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1541e-05 2024-04-23T16:37:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.743037+0000", 2024-04-23T16:37:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0103e-05 2024-04-23T16:37:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.751004+0000", 2024-04-23T16:37:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0079672489999999992 2024-04-23T16:37:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.751074+0000", 2024-04-23T16:37:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9517000000000005e-05 2024-04-23T16:37:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.837979+0000", 2024-04-23T16:37:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.086905300000000005 2024-04-23T16:37:11.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.838074+0000", 2024-04-23T16:37:11.258 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5094999999999997e-05 2024-04-23T16:37:11.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.838233+0000", 2024-04-23T16:37:11.258 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0001593 2024-04-23T16:37:11.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.259 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.848903+0000", 2024-04-23T16:37:11.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010669621000000001 2024-04-23T16:37:11.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.259 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.922168+0000", 2024-04-23T16:37:11.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.073264971999999998 2024-04-23T16:37:11.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.259 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.922230+0000", 2024-04-23T16:37:11.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2403999999999994e-05 2024-04-23T16:37:11.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.923930+0000", 2024-04-23T16:37:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001699629 2024-04-23T16:37:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:251113 5.a 5:59bedb89:::benchmark_data_smithi188_113957_object97515:head [delete] snapc 0=[] ondisk+write+known_if_redirected e426)", 2024-04-23T16:37:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:35:49.753200+0000", 2024-04-23T16:37:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 81.373303652999994, 2024-04-23T16:37:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17198295899999999, 2024-04-23T16:37:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:11.261 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:11.261 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:11.261 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:11.261 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 251113 2024-04-23T16:37:11.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.261 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:11.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.261 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:11.261 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.753200+0000", 2024-04-23T16:37:11.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.261 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:11.261 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.753200+0000", 2024-04-23T16:37:11.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:11.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:11.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.753202+0000", 2024-04-23T16:37:11.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0870000000000001e-06 2024-04-23T16:37:11.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:11.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.753209+0000", 2024-04-23T16:37:11.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7830000000000001e-06 2024-04-23T16:37:11.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:11.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.753221+0000", 2024-04-23T16:37:11.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1764e-05 2024-04-23T16:37:11.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.263 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:11.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.753227+0000", 2024-04-23T16:37:11.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4520000000000002e-06 2024-04-23T16:37:11.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.263 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.756278+0000", 2024-04-23T16:37:11.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030510120000000001 2024-04-23T16:37:11.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:37:11.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.756316+0000", 2024-04-23T16:37:11.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8209e-05 2024-04-23T16:37:11.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:11.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.843902+0000", 2024-04-23T16:37:11.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.087585619000000003 2024-04-23T16:37:11.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:11.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.844005+0000", 2024-04-23T16:37:11.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000103303 2024-04-23T16:37:11.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:37:11.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.844170+0000", 2024-04-23T16:37:11.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016467000000000001 2024-04-23T16:37:11.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:11.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.855135+0000", 2024-04-23T16:37:11.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010965449 2024-04-23T16:37:11.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:11.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.923999+0000", 2024-04-23T16:37:11.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.068863419999999995 2024-04-23T16:37:11.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.266 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.266 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:11.266 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.924054+0000", 2024-04-23T16:37:11.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5470000000000003e-05 2024-04-23T16:37:11.266 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:11.266 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:11.266 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:11.266 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.925183+0000", 2024-04-23T16:37:11.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0011287210000000001 2024-04-23T16:37:11.266 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.266 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.266 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.266 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:11.266 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:11.267 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:37:11.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:11.268 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:37:11.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:37:11.369 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:37:11.398 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:11.649 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:37:11.729 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:37:11.793 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-23T16:37:11.846 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:37:11.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:11.849 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:37:11.950 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:37:11.979 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:12.081 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:37:12.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:11 smithi103 ceph-mon[39629]: pgmap v1280: 11 pgs: 11 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:12.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:12.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:37:12.093 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:37:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:11 smithi148 ceph-mon[44520]: pgmap v1280: 11 pgs: 11 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:12.184 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:12.216 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:11 smithi188 ceph-mon[44101]: pgmap v1280: 11 pgs: 11 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:12.237 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:12.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:12.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:12.341 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:37:12.370 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:12.391 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:37:12.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:12.394 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:37:12.495 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:37:12.521 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:12.574 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:37:12.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:12.577 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:37:12.678 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:12.708 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:12.729 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:12.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:12.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:12.817 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:37:12.834 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:37:12.861 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:12.881 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:37:12.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:12.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:37:12.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:12 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/450570397' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:37:12.985 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:13.012 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:13.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:12 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/450570397' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:37:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:12 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/450570397' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:37:13.661 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:13.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:13.671 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:13.772 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:13.804 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:13 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:37:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:13 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:37:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:13 smithi188 ceph-mon[44101]: pgmap v1281: 11 pgs: 11 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:13.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:13 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:37:13.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:13 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:37:13.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:13 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/450570397' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:37:13.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:13 smithi188 ceph-mon[44101]: osdmap e453: 12 total, 11 up, 9 in 2024-04-23T16:37:13.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:13 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:37:13.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:13 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:37:14.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:13 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:37:14.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:13 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:37:14.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:13 smithi103 ceph-mon[39629]: pgmap v1281: 11 pgs: 11 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:14.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:13 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:37:14.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:13 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:37:14.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:13 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/450570397' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:37:14.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:13 smithi103 ceph-mon[39629]: osdmap e453: 12 total, 11 up, 9 in 2024-04-23T16:37:14.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:13 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:37:14.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:13 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:37:14.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:13 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:37:14.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:13 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:37:14.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:13 smithi148 ceph-mon[44520]: pgmap v1281: 11 pgs: 11 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:14.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:13 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:37:14.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:13 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:37:14.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:13 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/450570397' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:37:14.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:13 smithi148 ceph-mon[44520]: osdmap e453: 12 total, 11 up, 9 in 2024-04-23T16:37:14.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:13 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:37:14.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:13 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:37:14.126 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:14.129 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:14.356 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:14.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:14.360 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:14.461 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:37:14.492 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:14.831 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'unique_pool_4' created 2024-04-23T16:37:14.888 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:37:14.889 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:37:14.889 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-04-23T16:37:14.889 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:37:14.910 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:14 smithi188 ceph-mon[44101]: osdmap e454: 12 total, 11 up, 9 in 2024-04-23T16:37:14.910 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:14 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1802245563' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-23T16:37:15.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:14 smithi103 ceph-mon[39629]: osdmap e454: 12 total, 11 up, 9 in 2024-04-23T16:37:15.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:14 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1802245563' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-23T16:37:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:14 smithi148 ceph-mon[44520]: osdmap e454: 12 total, 11 up, 9 in 2024-04-23T16:37:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:14 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1802245563' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-23T16:37:15.140 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:37:15.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:15.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:37:15.245 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:37:15.275 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:15.573 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-23T16:37:15.624 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:37:15.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:15.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:37:15.728 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:37:15.761 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:15.784 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:37:15.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:15.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:37:15.872 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:37:15.888 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:37:15.917 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:15.938 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:37:15.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:15.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:37:16.042 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:16.055 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:37:16.069 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:15 smithi148 ceph-mon[44520]: pgmap v1284: 11 pgs: 11 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:16.069 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:15 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1802245563' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-04-23T16:37:16.069 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:15 smithi148 ceph-mon[44520]: osdmap e455: 12 total, 11 up, 9 in 2024-04-23T16:37:16.071 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:16.087 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:15 smithi103 ceph-mon[39629]: pgmap v1284: 11 pgs: 11 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:16.087 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:15 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1802245563' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-04-23T16:37:16.087 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:15 smithi103 ceph-mon[39629]: osdmap e455: 12 total, 11 up, 9 in 2024-04-23T16:37:16.090 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:16.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:16.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:16.126 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:15 smithi188 ceph-mon[44101]: pgmap v1284: 11 pgs: 11 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:16.126 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:15 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1802245563' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-04-23T16:37:16.126 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:15 smithi188 ceph-mon[44101]: osdmap e455: 12 total, 11 up, 9 in 2024-04-23T16:37:16.194 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:37:16.222 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:16.241 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:37:16.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:16.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:37:16.345 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:16.373 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:16.395 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:16.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:16.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:16.505 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:37:16.533 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:16.630 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:37:16.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:16.633 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:37:16.734 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:16.762 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:16.782 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:16.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:16.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:16.885 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:37:16.912 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:17.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:16 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:37:17.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:16 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:37:17.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:16 smithi103 ceph-mon[39629]: osdmap e456: 12 total, 11 up, 9 in 2024-04-23T16:37:17.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:16 smithi103 ceph-mon[39629]: osdmap e457: 12 total, 11 up, 9 in 2024-04-23T16:37:17.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:16 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:37:17.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:16 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:37:17.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:16 smithi148 ceph-mon[44520]: osdmap e456: 12 total, 11 up, 9 in 2024-04-23T16:37:17.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:16 smithi148 ceph-mon[44520]: osdmap e457: 12 total, 11 up, 9 in 2024-04-23T16:37:17.154 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:16 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:37:17.154 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:16 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:37:17.154 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:16 smithi188 ceph-mon[44101]: osdmap e456: 12 total, 11 up, 9 in 2024-04-23T16:37:17.154 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:16 smithi188 ceph-mon[44101]: osdmap e457: 12 total, 11 up, 9 in 2024-04-23T16:37:17.160 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:37:17.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:17.164 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:37:17.265 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:17.295 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:17.315 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:17.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:17.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:17.420 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:37:17.448 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:17.468 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:37:17.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:17.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:37:17.516 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-04-23T16:37:17.572 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:37:17.599 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:17.622 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:37:17.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:17.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:37:17.701 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:37:17.701 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:37:17.701 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:37:17.702 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:37:17.702 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:37:17.728 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:17.757 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:17.776 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:17.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:17.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:17.880 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:17.909 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:17 smithi188 ceph-mon[44101]: pgmap v1287: 27 pgs: 9 creating+peering, 7 unknown, 11 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:17 smithi188 ceph-mon[44101]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:37:18.096 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:37:18.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:17 smithi103 ceph-mon[39629]: pgmap v1287: 27 pgs: 9 creating+peering, 7 unknown, 11 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:18.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:17 smithi103 ceph-mon[39629]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:37:18.106 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:18.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:18.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:18.116 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:37:18.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:17 smithi148 ceph-mon[44520]: pgmap v1287: 27 pgs: 9 creating+peering, 7 unknown, 11 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:17 smithi148 ceph-mon[44520]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:37:18.135 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:37:18.155 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:37:18.174 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:37:18.194 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:37:18.210 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:18.213 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:37:18.233 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:37:18.238 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:18.254 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:37:18.257 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:18.276 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:37:18.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:18.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:18.296 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:37:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:37:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:37:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:37:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:37:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:37:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:37:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:37:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:37:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:37:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:37:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:37:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:37:18.362 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-23T16:37:18.381 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:18.431 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:18.451 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:18.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:18.454 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:18.555 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:18.587 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:18.610 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:18.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:18.626 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:37:18.714 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:37:18.745 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:18.765 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:37:18.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:18.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:37:18.870 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:18.899 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:18.919 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:18.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:18.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:19.023 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:37:19.052 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:19.072 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:37:19.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:19.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:37:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:18 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1899473938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:37:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:18 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1899473938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:37:19.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:18 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1899473938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:37:19.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:19.176 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:37:19.206 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:19.456 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:37:19.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:19.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:37:19.560 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:37:19.586 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:19.639 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:37:19.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:19.650 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:37:19.751 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:37:19.783 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:19.806 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:37:19.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:19.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:37:19.811 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:37:19.812 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:37:19.910 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:19.936 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:19.942 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:37:19.942 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:37:19.955 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:19.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:19.958 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:20.059 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:20.089 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:20.089 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:20.089 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:20.089 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:19 smithi148 ceph-mon[44520]: pgmap v1289: 27 pgs: 9 creating+peering, 5 unknown, 13 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:20.091 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:19 smithi103 ceph-mon[39629]: pgmap v1289: 27 pgs: 9 creating+peering, 5 unknown, 13 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:20.113 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:20.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:20.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:20.139 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:20.174 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:20.174 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:20.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:20.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:19 smithi188 ceph-mon[44101]: pgmap v1289: 27 pgs: 9 creating+peering, 5 unknown, 13 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:20.217 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:37:20.245 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:20.369 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:37:20.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:20.373 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:37:20.474 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:37:20.498 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:37:20.498 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:37:20.504 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:20.523 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:37:20.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:20.526 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:37:20.605 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1289,"stamp":"2024-04-23T16:37:18.754099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":44,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":3681680,"kb_used_data":9824,"kb_used_omap":14,"kb_used_meta":3670705,"kb_avail":840024688,"statfs":{"total":863955320832,"available":860185280512,"internally_reserved":0,"allocated":10059776,"data_stored":5539975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3758802927},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.933591"},"pg_stats":[{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":455,"state":"creating+peering","last_fresh":"2024-04-23T16:37:14.832259+0000","last_change":"2024-04-23T16:37:14.832259+0000","last_active":"2024-04-23T16:37:14.820617+0000","last_peered":"2024-04-23T16:37:14.820617+0000","last_clean":"2024-04-23T16:37:14.820617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T16:37:14.832259+0000","last_undegraded":"2024-04-23T16:37:14.832259+0000","last_fullsized":"2024-04-23T16:37:14.832259+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":455,"state":"creating+peering","last_fresh":"2024-04-23T16:37:14.832299+0000","last_change":"2024-04-23T16:37:14.832299+0000","last_active":"2024-04-23T16:37:14.820617+0000","last_peered":"2024-04-23T16:37:14.820617+0000","last_clean":"2024-04-23T16:37:14.820617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T16:37:14.832299+0000","last_undegraded":"2024-04-23T16:37:14.832299+0000","last_fullsized":"2024-04-23T16:37:14.832299+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":455,"state":"creating+peering","last_fresh":"2024-04-23T16:37:14.834949+0000","last_change":"2024-04-23T16:37:14.834949+0000","last_active":"2024-04-23T16:37:14.820617+0000","last_peered":"2024-04-23T16:37:14.820617+0000","last_clean":"2024-04-23T16:37:14.820617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T16:37:14.834949+0000","last_undegraded":"2024-04-23T16:37:14.834949+0000","last_fullsized":"2024-04-23T16:37:14.834949+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-23T16:37:14.820617+0000","last_change":"2024-04-23T16:37:14.820617+0000","last_active":"2024-04-23T16:37:14.820617+0000","last_peered":"2024-04-23T16:37:14.820617+0000","last_clean":"2024-04-23T16:37:14.820617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T16:37:14.820617+0000","last_undegraded":"2024-04-23T16:37:14.820617+0000","last_fullsized":"2024-04-23T16:37:14.820617+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-23T16:37:15.840106+0000","last_change":"2024-04-23T16:37:15.839856+0000","last_active":"2024-04-23T16:37:15.840106+0000","last_peered":"2024-04-23T16:37:15.840106+0000","last_clean":"2024-04-23T16:37:15.840106+0000","last_became_active":"2024-04-23T16:37:15.839667+0000","last_became_peered":"2024-04-23T16:37:15.839667+0000","last_unstale":"2024-04-23T16:37:15.840106+0000","last_undegraded":"2024-04-23T16:37:15.840106+0000","last_fullsized":"2024-04-23T16:37:15.840106+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:24.719670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":455,"state":"creating+peering","last_fresh":"2024-04-23T16:37:14.835138+0000","last_change":"2024-04-23T16:37:14.835138+0000","last_active":"2024-04-23T16:37:14.820617+0000","last_peered":"2024-04-23T16:37:14.820617+0000","last_clean":"2024-04-23T16:37:14.820617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T16:37:14.835138+0000","last_undegraded":"2024-04-23T16:37:14.835138+0000","last_fullsized":"2024-04-23T16:37:14.835138+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1177,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-04-23T16:37:12.814187+0000","last_change":"2024-04-23T16:36:19.132015+0000","last_active":"2024-04-23T16:37:12.814187+0000","last_peered":"2024-04-23T16:37:12.814187+0000","last_clean":"2024-04-23T16:37:12.814187+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:37:12.814187+0000","last_undegraded":"2024-04-23T16:37:12.814187+0000","last_fullsized":"2024-04-23T16:37:12.814187+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:19.131883+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:19.131883+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:41.022603+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-23T16:37:14.820617+0000","last_change":"2024-04-23T16:37:14.820617+0000","last_active":"2024-04-23T16:37:14.820617+0000","last_peered":"2024-04-23T16:37:14.820617+0000","last_clean":"2024-04-23T16:37:14.820617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T16:37:14.820617+0000","last_undegraded":"2024-04-23T16:37:14.820617+0000","last_fullsized":"2024-04-23T16:37:14.820617+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1175,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-04-23T16:37:14.835957+0000","last_change":"2024-04-23T16:37:05.612328+0000","last_active":"2024-04-23T16:37:14.835957+0000","last_peered":"2024-04-23T16:37:14.835957+0000","last_clean":"2024-04-23T16:37:14.835957+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:37:14.835957+0000","last_undegraded":"2024-04-23T16:37:14.835957+0000","last_fullsized":"2024-04-23T16:37:14.835957+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:05.612182+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:37:05.612182+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:24.096518+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-23T16:37:14.820617+0000","last_change":"2024-04-23T16:37:14.820617+0000","last_active":"2024-04-23T16:37:14.820617+0000","last_peered":"2024-04-23T16:37:14.820617+0000","last_clean":"2024-04-23T16:37:14.820617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T16:37:14.820617+0000","last_undegraded":"2024-04-23T16:37:14.820617+0000","last_fullsized":"2024-04-23T16:37:14.820617+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1166,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-04-23T16:37:14.835360+0000","last_change":"2024-04-23T16:37:05.696710+0000","last_active":"2024-04-23T16:37:14.835360+0000","last_peered":"2024-04-23T16:37:14.835360+0000","last_clean":"2024-04-23T16:37:14.835360+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:37:14.835360+0000","last_undegraded":"2024-04-23T16:37:14.835360+0000","last_fullsized":"2024-04-23T16:37:14.835360+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:05.696583+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:37:05.696583+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:25.899474+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":455,"state":"creating+peering","last_fresh":"2024-04-23T16:37:14.834168+0000","last_change":"2024-04-23T16:37:14.834168+0000","last_active":"2024-04-23T16:37:14.820617+0000","last_peered":"2024-04-23T16:37:14.820617+0000","last_clean":"2024-04-23T16:37:14.820617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T16:37:14.834168+0000","last_undegraded":"2024-04-23T16:37:14.834168+0000","last_fullsized":"2024-04-23T16:37:14.834168+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1178,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-23T16:37:13.821388+0000","last_change":"2024-04-23T16:36:26.456131+0000","last_active":"2024-04-23T16:37:13.821388+0000","last_peered":"2024-04-23T16:37:13.821388+0000","last_clean":"2024-04-23T16:37:13.821388+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:37:13.821388+0000","last_undegraded":"2024-04-23T16:37:13.821388+0000","last_fullsized":"2024-04-23T16:37:13.821388+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:26.456003+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:26.456003+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:44.948237+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":455,"state":"creating+peering","last_fresh":"2024-04-23T16:37:14.834979+0000","last_change":"2024-04-23T16:37:14.834979+0000","last_active":"2024-04-23T16:37:14.820617+0000","last_peered":"2024-04-23T16:37:14.820617+0000","last_clean":"2024-04-23T16:37:14.820617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T16:37:14.834979+0000","last_undegraded":"2024-04-23T16:37:14.834979+0000","last_fullsized":"2024-04-23T16:37:14.834979+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1179,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-23T16:37:13.826654+0000","last_change":"2024-04-23T16:37:05.363014+0000","last_active":"2024-04-23T16:37:13.826654+0000","last_peered":"2024-04-23T16:37:13.826654+0000","last_clean":"2024-04-23T16:37:13.826654+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:37:13.826654+0000","last_undegraded":"2024-04-23T16:37:13.826654+0000","last_fullsized":"2024-04-23T16:37:13.826654+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:05.362908+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:37:05.362908+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:14.719978+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":455,"state":"creating+peering","last_fresh":"2024-04-23T16:37:14.832285+0000","last_change":"2024-04-23T16:37:14.832285+0000","last_active":"2024-04-23T16:37:14.820617+0000","last_peered":"2024-04-23T16:37:14.820617+0000","last_clean":"2024-04-23T16:37:14.820617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T16:37:14.832285+0000","last_undegraded":"2024-04-23T16:37:14.832285+0000","last_fullsized":"2024-04-23T16:37:14.832285+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1176,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-04-23T16:37:12.814256+0000","last_change":"2024-04-23T16:36:23.239567+0000","last_active":"2024-04-23T16:37:12.814256+0000","last_peered":"2024-04-23T16:37:12.814256+0000","last_clean":"2024-04-23T16:37:12.814256+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:37:12.814256+0000","last_undegraded":"2024-04-23T16:37:12.814256+0000","last_fullsized":"2024-04-23T16:37:12.814256+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:23.239466+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:36:23.239466+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:27.031497+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:16.950346+0000","last_change":"2024-04-23T16:37:15.834693+0000","last_active":"2024-04-23T16:37:16.950346+0000","last_peered":"2024-04-23T16:37:16.950346+0000","last_clean":"2024-04-23T16:37:16.950346+0000","last_became_active":"2024-04-23T16:37:15.834476+0000","last_became_peered":"2024-04-23T16:37:15.834476+0000","last_unstale":"2024-04-23T16:37:16.950346+0000","last_undegraded":"2024-04-23T16:37:16.950346+0000","last_fullsized":"2024-04-23T16:37:16.950346+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:41.203351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1177,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-04-23T16:37:12.814190+0000","last_change":"2024-04-23T16:36:18.085504+0000","last_active":"2024-04-23T16:37:12.814190+0000","last_peered":"2024-04-23T16:37:12.814190+0000","last_clean":"2024-04-23T16:37:12.814190+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:37:12.814190+0000","last_undegraded":"2024-04-23T16:37:12.814190+0000","last_fullsized":"2024-04-23T16:37:12.814190+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:18.085386+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:18.085386+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:25.354657+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-23T16:37:14.820617+0000","last_change":"2024-04-23T16:37:14.820617+0000","last_active":"2024-04-23T16:37:14.820617+0000","last_peered":"2024-04-23T16:37:14.820617+0000","last_clean":"2024-04-23T16:37:14.820617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T16:37:14.820617+0000","last_undegraded":"2024-04-23T16:37:14.820617+0000","last_fullsized":"2024-04-23T16:37:14.820617+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1179,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-23T16:37:13.826663+0000","last_change":"2024-04-23T16:37:06.338083+0000","last_active":"2024-04-23T16:37:13.826663+0000","last_peered":"2024-04-23T16:37:13.826663+0000","last_clean":"2024-04-23T16:37:13.826663+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:37:13.826663+0000","last_undegraded":"2024-04-23T16:37:13.826663+0000","last_fullsized":"2024-04-23T16:37:13.826663+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:06.337940+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:37:06.337940+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:22.707182+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":455,"state":"creating+peering","last_fresh":"2024-04-23T16:37:14.834171+0000","last_change":"2024-04-23T16:37:14.834171+0000","last_active":"2024-04-23T16:37:14.820617+0000","last_peered":"2024-04-23T16:37:14.820617+0000","last_clean":"2024-04-23T16:37:14.820617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T16:37:14.834171+0000","last_undegraded":"2024-04-23T16:37:14.834171+0000","last_fullsized":"2024-04-23T16:37:14.834171+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1074,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-23T16:37:15.833423+0000","last_change":"2024-04-23T16:36:29.341359+0000","last_active":"2024-04-23T16:37:15.833423+0000","last_peered":"2024-04-23T16:37:15.833423+0000","last_clean":"2024-04-23T16:37:15.833423+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:37:15.833423+0000","last_undegraded":"2024-04-23T16:37:15.833423+0000","last_fullsized":"2024-04-23T16:37:15.833423+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:29.341205+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:29.341205+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:50.871176+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-23T16:37:14.820617+0000","last_change":"2024-04-23T16:37:14.820617+0000","last_active":"2024-04-23T16:37:14.820617+0000","last_peered":"2024-04-23T16:37:14.820617+0000","last_clean":"2024-04-23T16:37:14.820617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T16:37:14.820617+0000","last_undegraded":"2024-04-23T16:37:14.820617+0000","last_fullsized":"2024-04-23T16:37:14.820617+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":991,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-23T16:37:13.826663+0000","last_change":"2024-04-23T16:36:19.091622+0000","last_active":"2024-04-23T16:37:13.826663+0000","last_peered":"2024-04-23T16:37:13.826663+0000","last_clean":"2024-04-23T16:37:13.826663+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:37:13.826663+0000","last_undegraded":"2024-04-23T16:37:13.826663+0000","last_fullsized":"2024-04-23T16:37:13.826663+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:19.091490+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:36:19.091490+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:22.009913+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":455,"state":"creating+peering","last_fresh":"2024-04-23T16:37:14.835100+0000","last_change":"2024-04-23T16:37:14.835100+0000","last_active":"2024-04-23T16:37:14.820617+0000","last_peered":"2024-04-23T16:37:14.820617+0000","last_clean":"2024-04-23T16:37:14.820617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T16:37:14.835100+0000","last_undegraded":"2024-04-23T16:37:14.835100+0000","last_fullsized":"2024-04-23T16:37:14.835100+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1170,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-04-23T16:37:14.835290+0000","last_change":"2024-04-23T16:37:12.716078+0000","last_active":"2024-04-23T16:37:14.835290+0000","last_peered":"2024-04-23T16:37:14.835290+0000","last_clean":"2024-04-23T16:37:14.835290+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:37:14.835290+0000","last_undegraded":"2024-04-23T16:37:14.835290+0000","last_fullsized":"2024-04-23T16:37:14.835290+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:12.715954+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:37:12.715954+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:27.717888+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":22,"acting":22,"num_store_stats":5},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056904,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":501880,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93243272,"statfs":{"total":95995035648,"available":95481110528,"internally_reserved":0,"allocated":1114112,"data_stored":717058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":512424388},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351245,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612883,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":361296,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93383856,"statfs":{"total":95995035648,"available":95625068544,"internally_reserved":0,"allocated":1118208,"data_stored":558596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":368835008},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303184,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93441968,"statfs":{"total":95995035648,"available":95684575232,"internally_reserved":0,"allocated":1015808,"data_stored":667126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168860,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":405728,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":404606,"kb_avail":93339424,"statfs":{"total":95995035648,"available":95579570176,"internally_reserved":0,"allocated":1130496,"data_stored":564515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":414317002},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463202,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":284944,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93460208,"statfs":{"total":95995035648,"available":95703252992,"internally_reserved":0,"allocated":1118208,"data_stored":558596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":290650566},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137343,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738687,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":451536,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":93293616,"statfs":{"total":95995035648,"available":95532662784,"internally_reserved":0,"allocated":1118208,"data_stored":558596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117296,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":365680,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93379472,"statfs":{"total":95995035648,"available":95620579328,"internally_reserved":0,"allocated":1163264,"data_stored":678446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921066,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":577872,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93167280,"statfs":{"total":95995035648,"available":95403294720,"internally_reserved":0,"allocated":1118208,"data_stored":558596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":429560,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93315592,"statfs":{"total":95995035648,"available":95555166208,"internally_reserved":0,"allocated":1163264,"data_stored":678446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:37:20.607 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:37:20.607 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:20.626 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:37:20.659 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:20.681 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:37:20.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:20.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:37:20.785 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:37:20.814 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:20.835 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:37:20.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:20.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:37:20.939 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:37:20.965 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:20 smithi148 ceph-mon[44520]: from='client.16275 v1:172.21.15.103:0/1241406778' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:37:20.967 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:20.987 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:37:20.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:20.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:37:21.091 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:21.118 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:21.136 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:21.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:21.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:21.150 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:20 smithi188 ceph-mon[44101]: from='client.16275 v1:172.21.15.103:0/1241406778' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:37:21.175 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-23T16:37:21.210 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:20 smithi103 ceph-mon[39629]: from='client.16275 v1:172.21.15.103:0/1241406778' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:37:21.240 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:37:21.268 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:21.433 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:37:21.465 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:37:21.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:21.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:37:21.569 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:21.597 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:21.617 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:21.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:21.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:21.721 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:37:21.748 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:21.766 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:37:21.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:21.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:37:21.813 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:37:21.870 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:21.899 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:21 smithi188 ceph-mon[44101]: pgmap v1290: 27 pgs: 3 creating+peering, 24 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:21 smithi103 ceph-mon[39629]: pgmap v1290: 27 pgs: 3 creating+peering, 24 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:22.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:21 smithi148 ceph-mon[44520]: pgmap v1290: 27 pgs: 3 creating+peering, 24 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:22.648 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:22.651 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:22.752 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:22.780 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:22.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:22 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:37:22.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:22 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:37:22.879 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:22.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:22.882 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:22.922 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:22.983 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:37:23.011 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:23.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:22 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:37:23.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:22 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:37:23.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:22 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:37:23.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:22 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:37:23.154 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:37:23.154 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:37:23.180 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:22 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:37:23.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:22 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:37:23.227 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:22 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:37:23.227 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:22 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:37:23.227 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:22 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:37:23.228 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:22 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:37:23.501 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:23.659 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:37:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:23.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:37:23.763 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:37:23.791 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:23.810 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:37:23.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:23.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:37:23.865 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-23T16:37:23.914 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:23.981 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:23.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:23.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:24.085 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:37:24.112 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:23 smithi148 ceph-mon[44520]: pgmap v1291: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:37:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:37:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:37:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:37:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:37:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:37:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:37:24.125 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:37:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:23 smithi188 ceph-mon[44101]: pgmap v1291: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:37:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:37:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:37:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:37:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:37:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:37:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:37:24.244 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:23 smithi103 ceph-mon[39629]: pgmap v1291: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:24.245 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:37:24.245 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:37:24.245 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:37:24.245 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:37:24.245 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:37:24.245 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:37:24.245 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:37:24.259 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:37:24.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:24.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:37:24.363 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:37:24.395 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:24.416 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:37:24.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:24.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:37:24.521 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:37:24.550 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:24.570 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:37:24.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:24.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:37:24.674 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:37:24.706 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:24.727 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:37:24.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:24.729 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:37:24.830 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:24.857 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:24.955 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:24.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:24.958 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:25.059 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:25.087 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:25.211 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:37:25.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:25 smithi188 ceph-mon[44101]: pgmap v1292: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:25.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:25 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/826714640' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:37:25.507 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:25 smithi103 ceph-mon[39629]: pgmap v1292: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:25.508 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:25 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/826714640' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:37:25.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:25 smithi148 ceph-mon[44520]: pgmap v1292: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:25.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:25 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/826714640' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:37:26.234 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:37:26.234 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:37:26.336 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:26.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:26.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:26.440 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:37:26.458 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:26 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:37:26.458 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:26 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/826714640' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:37:26.458 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:26 smithi103 ceph-mon[39629]: osdmap e458: 12 total, 11 up, 9 in 2024-04-23T16:37:26.470 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:26 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:37:26.470 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:26 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/826714640' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:37:26.470 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:26 smithi148 ceph-mon[44520]: osdmap e458: 12 total, 11 up, 9 in 2024-04-23T16:37:26.472 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:26.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:26 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:37:26.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:26 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/826714640' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:37:26.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:26 smithi188 ceph-mon[44101]: osdmap e458: 12 total, 11 up, 9 in 2024-04-23T16:37:26.494 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:37:26.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:26.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:37:26.598 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:26.627 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:26.647 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:26.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:26.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:26.752 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:37:26.782 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:26.856 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:37:26.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:26.859 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:37:26.885 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-23T16:37:26.960 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:37:26.994 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:27.012 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:37:27.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:27.015 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:37:27.116 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:27.148 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:27.170 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:27.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:27.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:27.189 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:37:27.210 INFO:teuthology.orchestra.run.smithi103.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-04-23T16:37:27.274 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:27.302 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:27 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2676310820' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:37:27.303 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:27 smithi103 ceph-mon[39629]: pgmap v1294: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:27.305 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:27.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:27 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2676310820' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:37:27.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:27 smithi188 ceph-mon[44101]: pgmap v1294: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:27.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:27 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2676310820' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:37:27.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:27 smithi148 ceph-mon[44520]: pgmap v1294: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:27.754 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:27.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:27.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:27.858 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:37:27.890 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:27.911 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:37:27.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:27.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:37:28.015 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:37:28.046 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:28.065 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:37:28.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:28.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:37:28.170 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:28.200 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:28.357 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:28 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2676310820' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:37:28.357 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:28 smithi188 ceph-mon[44101]: osdmap e459: 12 total, 11 up, 9 in 2024-04-23T16:37:28.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:28 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2676310820' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:37:28.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:28 smithi103 ceph-mon[39629]: osdmap e459: 12 total, 11 up, 9 in 2024-04-23T16:37:28.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:28 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2676310820' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:37:28.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:28 smithi148 ceph-mon[44520]: osdmap e459: 12 total, 11 up, 9 in 2024-04-23T16:37:28.644 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:37:28.646 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:37:28.651 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:28.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:28.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:28.665 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:37:28.686 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:37:28.705 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:37:28.726 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:37:28.747 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:37:28.756 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:37:28.768 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:37:28.786 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:28.788 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:37:28.805 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:37:28.809 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:37:28.829 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:37:28.850 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:37:28.870 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:37:28.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:37:28.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:37:28.870 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:37:28.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:37:28.870 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:37:28.871 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:37:28.871 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:37:28.871 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:37:28.871 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:37:28.871 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:37:28.871 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:37:28.871 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:37:28.909 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:37:28.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:28.912 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:37:28.922 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:37:28.922 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-04-23T16:37:28.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-04-23T16:37:28.922 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [] 2024-04-23T16:37:28.922 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:37:29.013 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:37:29.041 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:29.189 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:37:29.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:29.191 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:37:29.292 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:29.325 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:29.347 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:29.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:29.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:29.451 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:37:29.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:29 smithi188 ceph-mon[44101]: from='client.16293 v1:172.21.15.103:0/3702749999' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:37:29.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:29 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3310738307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:37:29.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:29.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:29 smithi188 ceph-mon[44101]: pgmap v1296: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:29.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:29.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:29.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:29.479 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:29 smithi148 ceph-mon[44520]: from='client.16293 v1:172.21.15.103:0/3702749999' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:37:29.480 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:29 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3310738307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:37:29.480 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:29.480 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:29 smithi148 ceph-mon[44520]: pgmap v1296: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:29.480 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:29.480 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:29.480 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:29.482 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:29.503 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:37:29.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:29.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:37:29.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:29 smithi103 ceph-mon[39629]: from='client.16293 v1:172.21.15.103:0/3702749999' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:37:29.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:29 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3310738307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:37:29.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:29.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:29 smithi103 ceph-mon[39629]: pgmap v1296: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:29.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:29.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:29.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:29.607 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:29.636 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:29.657 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-23T16:37:29.657 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:29.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:29.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:29.756 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:37:29.757 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:37:29.760 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:29.788 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:29.808 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:29.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:29.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:29.912 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:37:29.926 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:37:29.940 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:29.969 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:37:29.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:29.972 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:37:30.073 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:37:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:30.127 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:37:30.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:30.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:37:30.231 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:37:30.260 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:30.281 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:37:30.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:30.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:37:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:30 smithi148 ceph-mon[44520]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:37:30.385 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:30.414 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:30 smithi188 ceph-mon[44101]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:37:30.513 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.513 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.513 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.513 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.513 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.513 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.513 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:30.513 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:30 smithi103 ceph-mon[39629]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:37:30.912 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:30.914 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1295,"stamp":"2024-04-23T16:37:27.209360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":3679984,"kb_used_data":7112,"kb_used_omap":14,"kb_used_meta":3670705,"kb_avail":840026384,"statfs":{"total":863955320832,"available":860187017216,"internally_reserved":0,"allocated":7282688,"data_stored":4125312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3758802927},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.455260"},"pg_stats":[{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-04-23T16:37:25.210434+0000","last_change":"2024-04-23T16:37:15.833831+0000","last_active":"2024-04-23T16:37:25.210434+0000","last_peered":"2024-04-23T16:37:25.210434+0000","last_clean":"2024-04-23T16:37:25.210434+0000","last_became_active":"2024-04-23T16:37:15.833495+0000","last_became_peered":"2024-04-23T16:37:15.833495+0000","last_unstale":"2024-04-23T16:37:25.210434+0000","last_undegraded":"2024-04-23T16:37:25.210434+0000","last_fullsized":"2024-04-23T16:37:25.210434+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:22.491829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":14,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-04-23T16:37:25.210267+0000","last_change":"2024-04-23T16:37:15.838829+0000","last_active":"2024-04-23T16:37:25.210267+0000","last_peered":"2024-04-23T16:37:25.210267+0000","last_clean":"2024-04-23T16:37:25.210267+0000","last_became_active":"2024-04-23T16:37:15.838559+0000","last_became_peered":"2024-04-23T16:37:15.838559+0000","last_unstale":"2024-04-23T16:37:25.210267+0000","last_undegraded":"2024-04-23T16:37:25.210267+0000","last_fullsized":"2024-04-23T16:37:25.210267+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:22.231943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:17.089162+0000","last_change":"2024-04-23T16:37:15.837846+0000","last_active":"2024-04-23T16:37:17.089162+0000","last_peered":"2024-04-23T16:37:17.089162+0000","last_clean":"2024-04-23T16:37:17.089162+0000","last_became_active":"2024-04-23T16:37:15.837590+0000","last_became_peered":"2024-04-23T16:37:15.837590+0000","last_unstale":"2024-04-23T16:37:17.089162+0000","last_undegraded":"2024-04-23T16:37:17.089162+0000","last_fullsized":"2024-04-23T16:37:17.089162+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:35.063721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:17.088784+0000","last_change":"2024-04-23T16:37:15.834978+0000","last_active":"2024-04-23T16:37:17.088784+0000","last_peered":"2024-04-23T16:37:17.088784+0000","last_clean":"2024-04-23T16:37:17.088784+0000","last_became_active":"2024-04-23T16:37:15.834740+0000","last_became_peered":"2024-04-23T16:37:15.834740+0000","last_unstale":"2024-04-23T16:37:17.088784+0000","last_undegraded":"2024-04-23T16:37:17.088784+0000","last_fullsized":"2024-04-23T16:37:17.088784+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:33.557524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-04-23T16:37:26.256136+0000","last_change":"2024-04-23T16:37:15.839856+0000","last_active":"2024-04-23T16:37:26.256136+0000","last_peered":"2024-04-23T16:37:26.256136+0000","last_clean":"2024-04-23T16:37:26.256136+0000","last_became_active":"2024-04-23T16:37:15.839667+0000","last_became_peered":"2024-04-23T16:37:15.839667+0000","last_unstale":"2024-04-23T16:37:26.256136+0000","last_undegraded":"2024-04-23T16:37:26.256136+0000","last_fullsized":"2024-04-23T16:37:26.256136+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:28.602006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:17.089162+0000","last_change":"2024-04-23T16:37:15.836920+0000","last_active":"2024-04-23T16:37:17.089162+0000","last_peered":"2024-04-23T16:37:17.089162+0000","last_clean":"2024-04-23T16:37:17.089162+0000","last_became_active":"2024-04-23T16:37:15.836635+0000","last_became_peered":"2024-04-23T16:37:15.836635+0000","last_unstale":"2024-04-23T16:37:17.089162+0000","last_undegraded":"2024-04-23T16:37:17.089162+0000","last_fullsized":"2024-04-23T16:37:17.089162+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:41.459142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1184,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:17.089088+0000","last_change":"2024-04-23T16:36:19.132015+0000","last_active":"2024-04-23T16:37:17.089088+0000","last_peered":"2024-04-23T16:37:17.089088+0000","last_clean":"2024-04-23T16:37:17.089088+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:37:17.089088+0000","last_undegraded":"2024-04-23T16:37:17.089088+0000","last_fullsized":"2024-04-23T16:37:17.089088+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:19.131883+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:19.131883+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:41.022603+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:17.089196+0000","last_change":"2024-04-23T16:37:15.836051+0000","last_active":"2024-04-23T16:37:17.089196+0000","last_peered":"2024-04-23T16:37:17.089196+0000","last_clean":"2024-04-23T16:37:17.089196+0000","last_became_active":"2024-04-23T16:37:15.835794+0000","last_became_peered":"2024-04-23T16:37:15.835794+0000","last_unstale":"2024-04-23T16:37:17.089196+0000","last_undegraded":"2024-04-23T16:37:17.089196+0000","last_fullsized":"2024-04-23T16:37:17.089196+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:20.858859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1179,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:17.089063+0000","last_change":"2024-04-23T16:37:05.612328+0000","last_active":"2024-04-23T16:37:17.089063+0000","last_peered":"2024-04-23T16:37:17.089063+0000","last_clean":"2024-04-23T16:37:17.089063+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:37:17.089063+0000","last_undegraded":"2024-04-23T16:37:17.089063+0000","last_fullsized":"2024-04-23T16:37:17.089063+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:05.612182+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:37:05.612182+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:24.096518+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:17.089064+0000","last_change":"2024-04-23T16:37:15.837702+0000","last_active":"2024-04-23T16:37:17.089064+0000","last_peered":"2024-04-23T16:37:17.089064+0000","last_clean":"2024-04-23T16:37:17.089064+0000","last_became_active":"2024-04-23T16:37:15.837406+0000","last_became_peered":"2024-04-23T16:37:15.837406+0000","last_unstale":"2024-04-23T16:37:17.089064+0000","last_undegraded":"2024-04-23T16:37:17.089064+0000","last_fullsized":"2024-04-23T16:37:17.089064+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:42.237485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1170,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:17.089220+0000","last_change":"2024-04-23T16:37:05.696710+0000","last_active":"2024-04-23T16:37:17.089220+0000","last_peered":"2024-04-23T16:37:17.089220+0000","last_clean":"2024-04-23T16:37:17.089220+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:37:17.089220+0000","last_undegraded":"2024-04-23T16:37:17.089220+0000","last_fullsized":"2024-04-23T16:37:17.089220+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:05.696583+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:37:05.696583+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:25.899474+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:17.089092+0000","last_change":"2024-04-23T16:37:15.839946+0000","last_active":"2024-04-23T16:37:17.089092+0000","last_peered":"2024-04-23T16:37:17.089092+0000","last_clean":"2024-04-23T16:37:17.089092+0000","last_became_active":"2024-04-23T16:37:15.839669+0000","last_became_peered":"2024-04-23T16:37:15.839669+0000","last_unstale":"2024-04-23T16:37:17.089092+0000","last_undegraded":"2024-04-23T16:37:17.089092+0000","last_fullsized":"2024-04-23T16:37:17.089092+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:33.539181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1184,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:17.088693+0000","last_change":"2024-04-23T16:36:26.456131+0000","last_active":"2024-04-23T16:37:17.088693+0000","last_peered":"2024-04-23T16:37:17.088693+0000","last_clean":"2024-04-23T16:37:17.088693+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:37:17.088693+0000","last_undegraded":"2024-04-23T16:37:17.088693+0000","last_fullsized":"2024-04-23T16:37:17.088693+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:26.456003+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:26.456003+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:44.948237+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:17.089294+0000","last_change":"2024-04-23T16:37:15.837187+0000","last_active":"2024-04-23T16:37:17.089294+0000","last_peered":"2024-04-23T16:37:17.089294+0000","last_clean":"2024-04-23T16:37:17.089294+0000","last_became_active":"2024-04-23T16:37:15.836264+0000","last_became_peered":"2024-04-23T16:37:15.836264+0000","last_unstale":"2024-04-23T16:37:17.089294+0000","last_undegraded":"2024-04-23T16:37:17.089294+0000","last_fullsized":"2024-04-23T16:37:17.089294+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:30.550593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1185,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:16.834782+0000","last_change":"2024-04-23T16:37:05.363014+0000","last_active":"2024-04-23T16:37:16.834782+0000","last_peered":"2024-04-23T16:37:16.834782+0000","last_clean":"2024-04-23T16:37:16.834782+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:37:16.834782+0000","last_undegraded":"2024-04-23T16:37:16.834782+0000","last_fullsized":"2024-04-23T16:37:16.834782+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:05.362908+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:37:05.362908+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:14.719978+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":14,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-04-23T16:37:25.210551+0000","last_change":"2024-04-23T16:37:15.835369+0000","last_active":"2024-04-23T16:37:25.210551+0000","last_peered":"2024-04-23T16:37:25.210551+0000","last_clean":"2024-04-23T16:37:25.210551+0000","last_became_active":"2024-04-23T16:37:15.835131+0000","last_became_peered":"2024-04-23T16:37:15.835131+0000","last_unstale":"2024-04-23T16:37:25.210551+0000","last_undegraded":"2024-04-23T16:37:25.210551+0000","last_fullsized":"2024-04-23T16:37:25.210551+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:33.514168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1183,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:17.089045+0000","last_change":"2024-04-23T16:36:23.239567+0000","last_active":"2024-04-23T16:37:17.089045+0000","last_peered":"2024-04-23T16:37:17.089045+0000","last_clean":"2024-04-23T16:37:17.089045+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:37:17.089045+0000","last_undegraded":"2024-04-23T16:37:17.089045+0000","last_fullsized":"2024-04-23T16:37:17.089045+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:23.239466+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:36:23.239466+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:27.031497+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:16.950346+0000","last_change":"2024-04-23T16:37:15.834693+0000","last_active":"2024-04-23T16:37:16.950346+0000","last_peered":"2024-04-23T16:37:16.950346+0000","last_clean":"2024-04-23T16:37:16.950346+0000","last_became_active":"2024-04-23T16:37:15.834476+0000","last_became_peered":"2024-04-23T16:37:15.834476+0000","last_unstale":"2024-04-23T16:37:16.950346+0000","last_undegraded":"2024-04-23T16:37:16.950346+0000","last_fullsized":"2024-04-23T16:37:16.950346+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:41.203351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1184,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:17.089089+0000","last_change":"2024-04-23T16:36:18.085504+0000","last_active":"2024-04-23T16:37:17.089089+0000","last_peered":"2024-04-23T16:37:17.089089+0000","last_clean":"2024-04-23T16:37:17.089089+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:37:17.089089+0000","last_undegraded":"2024-04-23T16:37:17.089089+0000","last_fullsized":"2024-04-23T16:37:17.089089+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:18.085386+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:18.085386+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:25.354657+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:17.088768+0000","last_change":"2024-04-23T16:37:15.836339+0000","last_active":"2024-04-23T16:37:17.088768+0000","last_peered":"2024-04-23T16:37:17.088768+0000","last_clean":"2024-04-23T16:37:17.088768+0000","last_became_active":"2024-04-23T16:37:15.836027+0000","last_became_peered":"2024-04-23T16:37:15.836027+0000","last_unstale":"2024-04-23T16:37:17.088768+0000","last_undegraded":"2024-04-23T16:37:17.088768+0000","last_fullsized":"2024-04-23T16:37:17.088768+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:35.404279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1185,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:16.834856+0000","last_change":"2024-04-23T16:37:06.338083+0000","last_active":"2024-04-23T16:37:16.834856+0000","last_peered":"2024-04-23T16:37:16.834856+0000","last_clean":"2024-04-23T16:37:16.834856+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:37:16.834856+0000","last_undegraded":"2024-04-23T16:37:16.834856+0000","last_fullsized":"2024-04-23T16:37:16.834856+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:06.337940+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:37:06.337940+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:22.707182+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:17.089058+0000","last_change":"2024-04-23T16:37:15.839998+0000","last_active":"2024-04-23T16:37:17.089058+0000","last_peered":"2024-04-23T16:37:17.089058+0000","last_clean":"2024-04-23T16:37:17.089058+0000","last_became_active":"2024-04-23T16:37:15.839714+0000","last_became_peered":"2024-04-23T16:37:15.839714+0000","last_unstale":"2024-04-23T16:37:17.089058+0000","last_undegraded":"2024-04-23T16:37:17.089058+0000","last_fullsized":"2024-04-23T16:37:17.089058+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:27.241342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1078,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-04-23T16:37:26.256081+0000","last_change":"2024-04-23T16:36:29.341359+0000","last_active":"2024-04-23T16:37:26.256081+0000","last_peered":"2024-04-23T16:37:26.256081+0000","last_clean":"2024-04-23T16:37:26.256081+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:37:26.256081+0000","last_undegraded":"2024-04-23T16:37:26.256081+0000","last_fullsized":"2024-04-23T16:37:26.256081+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:29.341205+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:29.341205+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:50.871176+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:17.088963+0000","last_change":"2024-04-23T16:37:15.836315+0000","last_active":"2024-04-23T16:37:17.088963+0000","last_peered":"2024-04-23T16:37:17.088963+0000","last_clean":"2024-04-23T16:37:17.088963+0000","last_became_active":"2024-04-23T16:37:15.836050+0000","last_became_peered":"2024-04-23T16:37:15.836050+0000","last_unstale":"2024-04-23T16:37:17.088963+0000","last_undegraded":"2024-04-23T16:37:17.088963+0000","last_fullsized":"2024-04-23T16:37:17.088963+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:18.403217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":998,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:16.834901+0000","last_change":"2024-04-23T16:36:19.091622+0000","last_active":"2024-04-23T16:37:16.834901+0000","last_peered":"2024-04-23T16:37:16.834901+0000","last_clean":"2024-04-23T16:37:16.834901+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:37:16.834901+0000","last_undegraded":"2024-04-23T16:37:16.834901+0000","last_fullsized":"2024-04-23T16:37:16.834901+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:19.091490+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:36:19.091490+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:22.009913+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:17.089213+0000","last_change":"2024-04-23T16:37:15.836721+0000","last_active":"2024-04-23T16:37:17.089213+0000","last_peered":"2024-04-23T16:37:17.089213+0000","last_clean":"2024-04-23T16:37:17.089213+0000","last_became_active":"2024-04-23T16:37:15.836376+0000","last_became_peered":"2024-04-23T16:37:15.836376+0000","last_unstale":"2024-04-23T16:37:17.089213+0000","last_undegraded":"2024-04-23T16:37:17.089213+0000","last_fullsized":"2024-04-23T16:37:17.089213+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:32.971951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1174,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-23T16:37:17.088971+0000","last_change":"2024-04-23T16:37:12.716078+0000","last_active":"2024-04-23T16:37:17.088971+0000","last_peered":"2024-04-23T16:37:17.088971+0000","last_clean":"2024-04-23T16:37:17.088971+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:37:17.088971+0000","last_undegraded":"2024-04-23T16:37:17.088971+0000","last_fullsized":"2024-04-23T16:37:17.088971+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:12.715954+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:37:12.715954+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:27.717888+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056906,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":501432,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93243720,"statfs":{"total":95995035648,"available":95481569280,"internally_reserved":0,"allocated":1028096,"data_stored":673347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":512424388},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351247,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612885,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":360984,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93384168,"statfs":{"total":95995035648,"available":95625388032,"internally_reserved":0,"allocated":688128,"data_stored":339446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":368835008},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841817,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303184,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93441968,"statfs":{"total":95995035648,"available":95684575232,"internally_reserved":0,"allocated":1015808,"data_stored":667126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168862,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":405768,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":404606,"kb_avail":93339384,"statfs":{"total":95995035648,"available":95579529216,"internally_reserved":0,"allocated":688128,"data_stored":339446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":414317002},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463204,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":284816,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93460336,"statfs":{"total":95995035648,"available":95703384064,"internally_reserved":0,"allocated":688128,"data_stored":339446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":290650566},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137345,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738689,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":451224,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":93293928,"statfs":{"total":95995035648,"available":95532982272,"internally_reserved":0,"allocated":688128,"data_stored":339446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117298,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":365248,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93379904,"statfs":{"total":95995035648,"available":95621021696,"internally_reserved":0,"allocated":905216,"data_stored":546915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921068,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":577928,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93167224,"statfs":{"total":95995035648,"available":95403237376,"internally_reserved":0,"allocated":688128,"data_stored":339446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757543,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":429400,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93315752,"statfs":{"total":95995035648,"available":95555330048,"internally_reserved":0,"allocated":892928,"data_stored":540694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:37:30.916 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-23T16:37:31.264 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:31.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:31.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:31.354 DEBUG:teuthology.orchestra.run.smithi188:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-04-23T16:37:31.357 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-23T16:37:31.357 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-23T16:37:31.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:31 smithi188 ceph-mon[44101]: pgmap v1297: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:31.368 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:31.396 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:31.414 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:31.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:31.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:31.445 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:31 smithi103 ceph-mon[39629]: pgmap v1297: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:31.448 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:hints = 1 2024-04-23T16:37:31.449 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-23T16:37:31.449 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Object prefix: benchmark_data_smithi188_130221 2024-04-23T16:37:31.449 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:37:31.449 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 0 0 0 0 0 0 - 0 2024-04-23T16:37:31.517 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:31.562 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:31 smithi148 ceph-mon[44520]: pgmap v1297: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-04-23T16:37:31.564 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:31.597 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:31.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:31.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:31.674 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:37:31.675 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:37:31.675 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-04-23T16:37:31.675 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:37:31.702 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:37:31.744 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:31.758 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:37:31.812 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:31.950 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:37:31.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:31.954 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:37:32.055 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:37:32.090 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:32.415 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-23T16:37:32.442 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:37:32.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:32.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:37:32.449 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 1 16 5636 5620 351.231 351.25 0.00496137 0.00283901 2024-04-23T16:37:32.546 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:37:32.582 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:32.610 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:37:32.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:32.614 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:37:32.715 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:32.758 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:32.801 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:37:33.449 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 2 16 10669 10653 332.884 314.562 0.00172355 0.00300008 2024-04-23T16:37:33.813 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:33 smithi103 ceph-mon[39629]: pgmap v1298: 27 pgs: 27 active+clean; 9.5 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 1.1 MiB/s wr, 17 op/s 2024-04-23T16:37:34.115 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:33 smithi188 ceph-mon[44101]: pgmap v1298: 27 pgs: 27 active+clean; 9.5 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 1.1 MiB/s wr, 17 op/s 2024-04-23T16:37:34.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:33 smithi148 ceph-mon[44520]: pgmap v1298: 27 pgs: 27 active+clean; 9.5 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 1.1 MiB/s wr, 17 op/s 2024-04-23T16:37:34.421 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:34.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:34.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:34.449 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 3 16 15135 15119 314.957 279.125 0.00284475 0.0031725 2024-04-23T16:37:34.524 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:37:34.565 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:34.603 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:37:34.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:34.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:37:34.731 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:34.738 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:34.751 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:37:34.751 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:37:34.751 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:37:34.751 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:37:34.751 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:37:34.771 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:34.796 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:34.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:34.901 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:35.036 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:35.061 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:35.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:35.064 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:35.140 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:37:35.140 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1637,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":459,"num_osds":12,"num_up_osds":11,"osd_up_since":1713889979,"num_in_osds":9,"osd_in_since":1713890004,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":27}],"num_pgs":27,"num_pools":2,"num_objects":145,"data_bytes":9962016,"bytes_used":3771924480,"bytes_avail":860183396352,"bytes_total":863955320832,"write_bytes_sec":1171200,"read_op_per_sec":0,"write_op_per_sec":17},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.103:8443/"}},"servicemap":{"epoch":34,"modified":"2024-04-23T16:35:30.733564+0000","services":{}},"progress_events":{}} 2024-04-23T16:37:35.165 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:37:35.200 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:35.223 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:37:35.275 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:35 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3063876661' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:37:35.275 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:35 smithi103 ceph-mon[39629]: pgmap v1299: 27 pgs: 27 active+clean; 337 MiB data, 4.1 GiB used, 800 GiB / 805 GiB avail; 35 MiB/s wr, 563 op/s 2024-04-23T16:37:35.276 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:35 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1554739424' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:37:35.450 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 4 16 19108 19092 298.29 248.312 0.00451689 0.0033493 2024-04-23T16:37:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:35 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3063876661' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:37:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:35 smithi188 ceph-mon[44101]: pgmap v1299: 27 pgs: 27 active+clean; 337 MiB data, 4.1 GiB used, 800 GiB / 805 GiB avail; 35 MiB/s wr, 563 op/s 2024-04-23T16:37:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:35 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1554739424' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:37:35.554 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:37:35.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:35.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:37:35.574 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-23T16:37:35.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:35 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3063876661' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:37:35.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:35 smithi148 ceph-mon[44520]: pgmap v1299: 27 pgs: 27 active+clean; 337 MiB data, 4.1 GiB used, 800 GiB / 805 GiB avail; 35 MiB/s wr, 563 op/s 2024-04-23T16:37:35.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:35 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1554739424' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:37:35.660 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:37:35.703 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:35.760 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:37:35.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:35.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:37:35.865 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:37:35.913 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:35.943 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:37:35.944 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:37:35.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:35.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:37:36.048 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:37:36.097 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:37:36.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:36.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:37:36.240 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:37:36.277 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:36.302 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:37:36.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:36.305 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:37:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:36 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:37:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3063876661' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:37:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:36 smithi103 ceph-mon[39629]: osdmap e460: 12 total, 11 up, 9 in 2024-04-23T16:37:36.406 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:37:36.447 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:36.449 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 5 16 22767 22751 284.366 228.688 0.00733101 0.00351372 2024-04-23T16:37:36.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:36 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:37:36.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3063876661' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:37:36.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:36 smithi188 ceph-mon[44101]: osdmap e460: 12 total, 11 up, 9 in 2024-04-23T16:37:36.473 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-23T16:37:36.473 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:37:36.478 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:37:36.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:36.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:37:36.485 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:36 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:37:36.485 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3063876661' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:37:36.485 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:36 smithi148 ceph-mon[44520]: osdmap e460: 12 total, 11 up, 9 in 2024-04-23T16:37:36.583 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:37:36.616 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:37.268 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:37:37.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:37.271 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:37:37.373 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:37.403 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:37.449 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 6 16 26378 26362 274.584 225.688 0.00431382 0.0036403 2024-04-23T16:37:37.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:37 smithi188 ceph-mon[44101]: pgmap v1301: 27 pgs: 27 active+clean; 1020 MiB data, 5.3 GiB used, 799 GiB / 805 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-04-23T16:37:37.532 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:37.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:37.536 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:37 smithi103 ceph-mon[39629]: pgmap v1301: 27 pgs: 27 active+clean; 1020 MiB data, 5.3 GiB used, 799 GiB / 805 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-04-23T16:37:37.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:37 smithi148 ceph-mon[44520]: pgmap v1301: 27 pgs: 27 active+clean; 1020 MiB data, 5.3 GiB used, 799 GiB / 805 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-04-23T16:37:37.636 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:37.676 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:37.859 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:37:37.859 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-04-23T16:37:37.859 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-04-23T16:37:37.859 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:37:37.859 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.859 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:211670 5.d 5:b483b486:::benchmark_data_smithi188_113957_object58072:head [delete] snapc 0=[] ondisk+write+known_if_redirected e415)", 2024-04-23T16:37:37.859 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:56.380256+0000", 2024-04-23T16:37:37.859 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 161.47482931600001, 2024-04-23T16:37:37.860 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.063665811000000003, 2024-04-23T16:37:37.860 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.860 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.860 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.860 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.860 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.860 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.860 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 211670 2024-04-23T16:37:37.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.860 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.860 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.860 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.860 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.380256+0000", 2024-04-23T16:37:37.861 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.861 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.861 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.861 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.861 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.380256+0000", 2024-04-23T16:37:37.861 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.861 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.861 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.861 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.861 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.380259+0000", 2024-04-23T16:37:37.861 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1049999999999999e-06 2024-04-23T16:37:37.861 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.861 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.861 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.861 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.380270+0000", 2024-04-23T16:37:37.862 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1731e-05 2024-04-23T16:37:37.862 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.862 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.862 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.380289+0000", 2024-04-23T16:37:37.862 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8272999999999999e-05 2024-04-23T16:37:37.862 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.862 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.862 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.380298+0000", 2024-04-23T16:37:37.862 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8290000000000003e-06 2024-04-23T16:37:37.862 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.862 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.862 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.380355+0000", 2024-04-23T16:37:37.862 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6279000000000001e-05 2024-04-23T16:37:37.863 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.863 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.863 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.863 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.380460+0000", 2024-04-23T16:37:37.863 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105072 2024-04-23T16:37:37.863 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.863 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.863 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.863 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.380549+0000", 2024-04-23T16:37:37.863 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9381000000000004e-05 2024-04-23T16:37:37.863 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.863 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.863 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.863 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.393830+0000", 2024-04-23T16:37:37.863 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013281243 2024-04-23T16:37:37.864 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.864 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.864 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.864 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.443816+0000", 2024-04-23T16:37:37.864 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.049985578000000003 2024-04-23T16:37:37.864 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.864 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.864 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.864 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.443872+0000", 2024-04-23T16:37:37.864 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5858e-05 2024-04-23T16:37:37.864 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.864 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.865 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.865 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.443921+0000", 2024-04-23T16:37:37.865 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9462000000000002e-05 2024-04-23T16:37:37.865 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.865 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.865 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.865 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.865 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.865 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:211674 5.d 5:bd28a352:::benchmark_data_smithi188_113957_object58076:head [delete] snapc 0=[] ondisk+write+known_if_redirected e415)", 2024-04-23T16:37:37.865 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:56.389085+0000", 2024-04-23T16:37:37.865 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 161.465999724, 2024-04-23T16:37:37.865 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.054926060999999998, 2024-04-23T16:37:37.865 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.865 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.865 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.865 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.866 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.866 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.866 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 211674 2024-04-23T16:37:37.866 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.866 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.866 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.866 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.866 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.389085+0000", 2024-04-23T16:37:37.866 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.866 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.866 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.866 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.866 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.389085+0000", 2024-04-23T16:37:37.866 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.866 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.867 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.867 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.867 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.389088+0000", 2024-04-23T16:37:37.867 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8260000000000001e-06 2024-04-23T16:37:37.867 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.867 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.867 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.867 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.389102+0000", 2024-04-23T16:37:37.867 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4507e-05 2024-04-23T16:37:37.867 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.867 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.867 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.867 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.389120+0000", 2024-04-23T16:37:37.867 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7286999999999999e-05 2024-04-23T16:37:37.867 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.868 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.868 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.868 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.389129+0000", 2024-04-23T16:37:37.868 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0650000000000005e-06 2024-04-23T16:37:37.868 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.868 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.868 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.868 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.389183+0000", 2024-04-23T16:37:37.868 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.384e-05 2024-04-23T16:37:37.868 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.868 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.868 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.868 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.389301+0000", 2024-04-23T16:37:37.869 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011810499999999999 2024-04-23T16:37:37.869 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.869 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.869 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.869 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.389383+0000", 2024-04-23T16:37:37.869 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2389e-05 2024-04-23T16:37:37.869 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.869 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.869 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.869 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.399411+0000", 2024-04-23T16:37:37.869 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01002805 2024-04-23T16:37:37.869 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.869 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.869 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.869 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.443928+0000", 2024-04-23T16:37:37.870 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.044516941999999997 2024-04-23T16:37:37.870 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.870 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.870 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.870 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.443970+0000", 2024-04-23T16:37:37.870 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1795999999999998e-05 2024-04-23T16:37:37.870 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.870 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.870 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.870 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.444011+0000", 2024-04-23T16:37:37.870 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1254000000000002e-05 2024-04-23T16:37:37.870 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.870 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.870 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.870 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.871 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.871 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:253213 5.d 5:b46ae573:::benchmark_data_smithi188_113957_object99615:head [delete] snapc 0=[] ondisk+write+known_if_redirected e427)", 2024-04-23T16:37:37.871 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:35:52.421407+0000", 2024-04-23T16:37:37.871 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 105.433678082, 2024-04-23T16:37:37.871 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.061826733000000002, 2024-04-23T16:37:37.871 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.871 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.871 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.871 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.871 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.871 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.871 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 253213 2024-04-23T16:37:37.871 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.871 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.871 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.871 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.872 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.421407+0000", 2024-04-23T16:37:37.872 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.872 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.872 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.872 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.872 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.421407+0000", 2024-04-23T16:37:37.872 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.872 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.872 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.872 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.872 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.421410+0000", 2024-04-23T16:37:37.872 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9450000000000002e-06 2024-04-23T16:37:37.872 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.872 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.873 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.873 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.421418+0000", 2024-04-23T16:37:37.873 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3469999999999999e-06 2024-04-23T16:37:37.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.873 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.873 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.873 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.421433+0000", 2024-04-23T16:37:37.873 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4764e-05 2024-04-23T16:37:37.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.873 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.873 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.873 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.421442+0000", 2024-04-23T16:37:37.873 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3039999999999994e-06 2024-04-23T16:37:37.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.873 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.874 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.874 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.421507+0000", 2024-04-23T16:37:37.874 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4801000000000005e-05 2024-04-23T16:37:37.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.874 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.874 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.421605+0000", 2024-04-23T16:37:37.874 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7921e-05 2024-04-23T16:37:37.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.874 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.874 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.422311+0000", 2024-04-23T16:37:37.874 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070641900000000001 2024-04-23T16:37:37.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.875 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.426882+0000", 2024-04-23T16:37:37.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045707669999999999 2024-04-23T16:37:37.875 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.875 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.875 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.483117+0000", 2024-04-23T16:37:37.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.056235087000000003 2024-04-23T16:37:37.875 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.875 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.875 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.483181+0000", 2024-04-23T16:37:37.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3870999999999999e-05 2024-04-23T16:37:37.875 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.875 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.876 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.483233+0000", 2024-04-23T16:37:37.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2506999999999997e-05 2024-04-23T16:37:37.876 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.876 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.876 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.876 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.876 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:253215 5.d 5:b55ad2c8:::benchmark_data_smithi188_113957_object99617:head [delete] snapc 0=[] ondisk+write+known_if_redirected e427)", 2024-04-23T16:37:37.876 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:35:52.428319+0000", 2024-04-23T16:37:37.876 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 105.426766091, 2024-04-23T16:37:37.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.05504792, 2024-04-23T16:37:37.876 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.876 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.876 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.877 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.877 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.877 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.877 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 253215 2024-04-23T16:37:37.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.877 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.877 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.877 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.428319+0000", 2024-04-23T16:37:37.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.877 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.877 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.428319+0000", 2024-04-23T16:37:37.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.428321+0000", 2024-04-23T16:37:37.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3800000000000001e-06 2024-04-23T16:37:37.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.428329+0000", 2024-04-23T16:37:37.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5120000000000002e-06 2024-04-23T16:37:37.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.428340+0000", 2024-04-23T16:37:37.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1639e-05 2024-04-23T16:37:37.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.879 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.428349+0000", 2024-04-23T16:37:37.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8789999999999992e-06 2024-04-23T16:37:37.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.879 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.428407+0000", 2024-04-23T16:37:37.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8267000000000003e-05 2024-04-23T16:37:37.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.879 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.428530+0000", 2024-04-23T16:37:37.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000122442 2024-04-23T16:37:37.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.428629+0000", 2024-04-23T16:37:37.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9167999999999996e-05 2024-04-23T16:37:37.880 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.437178+0000", 2024-04-23T16:37:37.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085487089999999998 2024-04-23T16:37:37.880 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.483291+0000", 2024-04-23T16:37:37.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.046113287000000003 2024-04-23T16:37:37.880 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.483327+0000", 2024-04-23T16:37:37.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6157e-05 2024-04-23T16:37:37.881 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.881 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.483367+0000", 2024-04-23T16:37:37.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9480000000000001e-05 2024-04-23T16:37:37.881 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.881 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.881 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.881 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.881 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:253216 5.d 5:bbdd0550:::benchmark_data_smithi188_113957_object99618:head [delete] snapc 0=[] ondisk+write+known_if_redirected e427)", 2024-04-23T16:37:37.881 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:35:52.428377+0000", 2024-04-23T16:37:37.882 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 105.426708222, 2024-04-23T16:37:37.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.055157123000000002, 2024-04-23T16:37:37.882 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.882 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.882 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.882 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.882 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.882 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.882 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 253216 2024-04-23T16:37:37.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.882 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.882 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.428377+0000", 2024-04-23T16:37:37.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.883 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.428377+0000", 2024-04-23T16:37:37.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.883 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.428378+0000", 2024-04-23T16:37:37.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4399999999999998e-07 2024-04-23T16:37:37.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.883 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.428381+0000", 2024-04-23T16:37:37.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3579999999999999e-06 2024-04-23T16:37:37.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.428387+0000", 2024-04-23T16:37:37.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2009999999999996e-06 2024-04-23T16:37:37.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.428393+0000", 2024-04-23T16:37:37.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5160000000000002e-06 2024-04-23T16:37:37.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.432379+0000", 2024-04-23T16:37:37.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039864970000000003 2024-04-23T16:37:37.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.885 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.885 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.885 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.432490+0000", 2024-04-23T16:37:37.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110435 2024-04-23T16:37:37.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.885 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.885 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.885 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.432596+0000", 2024-04-23T16:37:37.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105952 2024-04-23T16:37:37.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.885 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.885 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.437333+0000", 2024-04-23T16:37:37.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047375949999999998 2024-04-23T16:37:37.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.886 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.483434+0000", 2024-04-23T16:37:37.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.046100973000000003 2024-04-23T16:37:37.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.886 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.483487+0000", 2024-04-23T16:37:37.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3301999999999999e-05 2024-04-23T16:37:37.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.886 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:52.483534+0000", 2024-04-23T16:37:37.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6350000000000002e-05 2024-04-23T16:37:37.887 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.887 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.887 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.887 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:255448 5.d 5:b007262b:::benchmark_data_smithi188_113957_object101850:head [delete] snapc 0=[] ondisk+write+known_if_redirected e429)", 2024-04-23T16:37:37.887 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:35:55.284209+0000", 2024-04-23T16:37:37.887 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 102.570875788, 2024-04-23T16:37:37.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.053437584000000003, 2024-04-23T16:37:37.887 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.887 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.887 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.887 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.887 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.887 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.888 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 255448 2024-04-23T16:37:37.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.888 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:55.284209+0000", 2024-04-23T16:37:37.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:55.284209+0000", 2024-04-23T16:37:37.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:55.284211+0000", 2024-04-23T16:37:37.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.962e-06 2024-04-23T16:37:37.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:55.284216+0000", 2024-04-23T16:37:37.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4879999999999998e-06 2024-04-23T16:37:37.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:55.284227+0000", 2024-04-23T16:37:37.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.083e-05 2024-04-23T16:37:37.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:55.284236+0000", 2024-04-23T16:37:37.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1890000000000007e-06 2024-04-23T16:37:37.890 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:55.284296+0000", 2024-04-23T16:37:37.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0956999999999999e-05 2024-04-23T16:37:37.890 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:55.284421+0000", 2024-04-23T16:37:37.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012496299999999999 2024-04-23T16:37:37.890 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.891 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:55.284545+0000", 2024-04-23T16:37:37.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000123258 2024-04-23T16:37:37.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.891 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:55.289017+0000", 2024-04-23T16:37:37.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044728040000000004 2024-04-23T16:37:37.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.891 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:55.337539+0000", 2024-04-23T16:37:37.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.048521094000000001 2024-04-23T16:37:37.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.892 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.892 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:55.337598+0000", 2024-04-23T16:37:37.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9132999999999998e-05 2024-04-23T16:37:37.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.892 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.892 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:55.337647+0000", 2024-04-23T16:37:37.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8906000000000002e-05 2024-04-23T16:37:37.892 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.892 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.892 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.893 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:271250 5.d 5:b9ac1130:::benchmark_data_smithi188_113957_object117652:head [delete] snapc 0=[] ondisk+write+known_if_redirected e434)", 2024-04-23T16:37:37.893 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:14.735375+0000", 2024-04-23T16:37:37.893 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 83.119709624999999, 2024-04-23T16:37:37.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.069947976999999995, 2024-04-23T16:37:37.893 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.893 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.893 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.893 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.893 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.893 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.893 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 271250 2024-04-23T16:37:37.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.893 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.893 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.735375+0000", 2024-04-23T16:37:37.894 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.894 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.735375+0000", 2024-04-23T16:37:37.894 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.894 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.735378+0000", 2024-04-23T16:37:37.894 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3870000000000002e-06 2024-04-23T16:37:37.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.735383+0000", 2024-04-23T16:37:37.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7320000000000003e-06 2024-04-23T16:37:37.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.735395+0000", 2024-04-23T16:37:37.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2053e-05 2024-04-23T16:37:37.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.735403+0000", 2024-04-23T16:37:37.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.926e-06 2024-04-23T16:37:37.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.735463+0000", 2024-04-23T16:37:37.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9247999999999999e-05 2024-04-23T16:37:37.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.735579+0000", 2024-04-23T16:37:37.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011688099999999999 2024-04-23T16:37:37.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.735686+0000", 2024-04-23T16:37:37.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010641099999999999 2024-04-23T16:37:37.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.740660+0000", 2024-04-23T16:37:37.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049736579999999997 2024-04-23T16:37:37.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.805247+0000", 2024-04-23T16:37:37.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.064587965999999997 2024-04-23T16:37:37.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.805286+0000", 2024-04-23T16:37:37.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8760000000000002e-05 2024-04-23T16:37:37.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.805323+0000", 2024-04-23T16:37:37.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6955000000000001e-05 2024-04-23T16:37:37.898 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.898 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.899 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.899 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:272578 5.d 5:b5d70dec:::benchmark_data_smithi188_113957_object118980:head [delete] snapc 0=[] ondisk+write+known_if_redirected e434)", 2024-04-23T16:37:37.899 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:16.413717+0000", 2024-04-23T16:37:37.899 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 81.441368276000006, 2024-04-23T16:37:37.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.060868555999999997, 2024-04-23T16:37:37.899 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.899 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.899 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.899 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.899 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.900 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.900 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 272578 2024-04-23T16:37:37.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.900 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.413717+0000", 2024-04-23T16:37:37.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.413717+0000", 2024-04-23T16:37:37.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.413720+0000", 2024-04-23T16:37:37.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0079999999999998e-06 2024-04-23T16:37:37.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.413730+0000", 2024-04-23T16:37:37.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0266e-05 2024-04-23T16:37:37.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.413749+0000", 2024-04-23T16:37:37.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9103999999999999e-05 2024-04-23T16:37:37.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.413760+0000", 2024-04-23T16:37:37.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0757000000000001e-05 2024-04-23T16:37:37.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.413818+0000", 2024-04-23T16:37:37.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8771999999999999e-05 2024-04-23T16:37:37.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.413919+0000", 2024-04-23T16:37:37.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100818 2024-04-23T16:37:37.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.413998+0000", 2024-04-23T16:37:37.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8497000000000004e-05 2024-04-23T16:37:37.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.418019+0000", 2024-04-23T16:37:37.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0040216879999999998 2024-04-23T16:37:37.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.474522+0000", 2024-04-23T16:37:37.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.056502961999999997 2024-04-23T16:37:37.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.474558+0000", 2024-04-23T16:37:37.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5812999999999997e-05 2024-04-23T16:37:37.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.905 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.905 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.474585+0000", 2024-04-23T16:37:37.905 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6871000000000001e-05 2024-04-23T16:37:37.905 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.905 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.905 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.905 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.905 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:272593 5.d 5:b22900a4:::benchmark_data_smithi188_113957_object118995:head [delete] snapc 0=[] ondisk+write+known_if_redirected e434)", 2024-04-23T16:37:37.905 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:16.422104+0000", 2024-04-23T16:37:37.905 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 81.432980420999996, 2024-04-23T16:37:37.905 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.052613634999999999, 2024-04-23T16:37:37.905 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.906 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.906 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.906 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.906 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.906 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.906 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 272593 2024-04-23T16:37:37.906 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.906 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.906 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.906 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.906 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.422104+0000", 2024-04-23T16:37:37.906 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.906 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.906 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.907 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.907 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.422104+0000", 2024-04-23T16:37:37.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.907 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.907 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.907 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.422106+0000", 2024-04-23T16:37:37.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.455e-06 2024-04-23T16:37:37.907 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.907 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.907 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.422110+0000", 2024-04-23T16:37:37.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2599999999999999e-06 2024-04-23T16:37:37.907 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.422119+0000", 2024-04-23T16:37:37.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2010000000000005e-06 2024-04-23T16:37:37.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.422125+0000", 2024-04-23T16:37:37.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2820000000000004e-06 2024-04-23T16:37:37.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.422168+0000", 2024-04-23T16:37:37.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3556000000000001e-05 2024-04-23T16:37:37.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.422257+0000", 2024-04-23T16:37:37.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9182000000000001e-05 2024-04-23T16:37:37.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.422331+0000", 2024-04-23T16:37:37.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3720000000000006e-05 2024-04-23T16:37:37.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.426025+0000", 2024-04-23T16:37:37.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036943179999999998 2024-04-23T16:37:37.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.474634+0000", 2024-04-23T16:37:37.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.048609012 2024-04-23T16:37:37.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.474688+0000", 2024-04-23T16:37:37.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3879999999999999e-05 2024-04-23T16:37:37.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.474718+0000", 2024-04-23T16:37:37.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9768999999999999e-05 2024-04-23T16:37:37.910 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.910 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.911 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.911 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:272851 5.d 5:b1ec686b:::benchmark_data_smithi188_113957_object119253:head [delete] snapc 0=[] ondisk+write+known_if_redirected e434)", 2024-04-23T16:37:37.911 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:16.728074+0000", 2024-04-23T16:37:37.911 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 81.127010635000005, 2024-04-23T16:37:37.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.066026524000000003, 2024-04-23T16:37:37.911 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.911 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.911 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.911 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.911 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.911 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.911 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 272851 2024-04-23T16:37:37.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.912 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.728074+0000", 2024-04-23T16:37:37.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.728074+0000", 2024-04-23T16:37:37.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.728077+0000", 2024-04-23T16:37:37.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7410000000000001e-06 2024-04-23T16:37:37.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.728084+0000", 2024-04-23T16:37:37.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2359999999999998e-06 2024-04-23T16:37:37.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.728109+0000", 2024-04-23T16:37:37.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4706999999999999e-05 2024-04-23T16:37:37.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.728118+0000", 2024-04-23T16:37:37.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5300000000000002e-06 2024-04-23T16:37:37.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.728181+0000", 2024-04-23T16:37:37.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2223000000000004e-05 2024-04-23T16:37:37.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.728312+0000", 2024-04-23T16:37:37.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000131261 2024-04-23T16:37:37.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.728453+0000", 2024-04-23T16:37:37.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000141123 2024-04-23T16:37:37.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.736532+0000", 2024-04-23T16:37:37.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080787619999999997 2024-04-23T16:37:37.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.793982+0000", 2024-04-23T16:37:37.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.057450169000000002 2024-04-23T16:37:37.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.794044+0000", 2024-04-23T16:37:37.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1577999999999999e-05 2024-04-23T16:37:37.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.794101+0000", 2024-04-23T16:37:37.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7194000000000002e-05 2024-04-23T16:37:37.916 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.916 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.916 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.916 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:272853 5.d 5:b671bc72:::benchmark_data_smithi188_113957_object119255:head [delete] snapc 0=[] ondisk+write+known_if_redirected e434)", 2024-04-23T16:37:37.917 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:16.731941+0000", 2024-04-23T16:37:37.917 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 81.123143585999998, 2024-04-23T16:37:37.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.064283106000000007, 2024-04-23T16:37:37.917 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.917 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.917 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.917 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.917 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.917 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.917 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 272853 2024-04-23T16:37:37.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.917 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.731941+0000", 2024-04-23T16:37:37.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.731941+0000", 2024-04-23T16:37:37.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.731943+0000", 2024-04-23T16:37:37.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.728e-06 2024-04-23T16:37:37.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.731948+0000", 2024-04-23T16:37:37.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3789999999999997e-06 2024-04-23T16:37:37.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.731960+0000", 2024-04-23T16:37:37.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1739000000000001e-05 2024-04-23T16:37:37.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.731967+0000", 2024-04-23T16:37:37.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7429999999999996e-06 2024-04-23T16:37:37.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.732700+0000", 2024-04-23T16:37:37.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00073352300000000003 2024-04-23T16:37:37.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.732801+0000", 2024-04-23T16:37:37.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010045899999999999 2024-04-23T16:37:37.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.732874+0000", 2024-04-23T16:37:37.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3347000000000001e-05 2024-04-23T16:37:37.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.736642+0000", 2024-04-23T16:37:37.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0037680639999999998 2024-04-23T16:37:37.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.796163+0000", 2024-04-23T16:37:37.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.059520380999999997 2024-04-23T16:37:37.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.796195+0000", 2024-04-23T16:37:37.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1894000000000003e-05 2024-04-23T16:37:37.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.796224+0000", 2024-04-23T16:37:37.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9848999999999998e-05 2024-04-23T16:37:37.921 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.921 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.922 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.922 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:274769 5.d 5:b1181e5a:::benchmark_data_smithi188_113957_object121171:head [delete] snapc 0=[] ondisk+write+known_if_redirected e435)", 2024-04-23T16:37:37.922 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:19.017664+0000", 2024-04-23T16:37:37.922 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 78.837421046000003, 2024-04-23T16:37:37.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.073474631999999998, 2024-04-23T16:37:37.922 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.922 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.922 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.922 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.922 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.922 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.922 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 274769 2024-04-23T16:37:37.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.923 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.017664+0000", 2024-04-23T16:37:37.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.017664+0000", 2024-04-23T16:37:37.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.017667+0000", 2024-04-23T16:37:37.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.942e-06 2024-04-23T16:37:37.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.017675+0000", 2024-04-23T16:37:37.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2109999999999998e-06 2024-04-23T16:37:37.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.017692+0000", 2024-04-23T16:37:37.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6748999999999998e-05 2024-04-23T16:37:37.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.017701+0000", 2024-04-23T16:37:37.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7699999999999996e-06 2024-04-23T16:37:37.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.017765+0000", 2024-04-23T16:37:37.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3190999999999999e-05 2024-04-23T16:37:37.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.017910+0000", 2024-04-23T16:37:37.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014524300000000001 2024-04-23T16:37:37.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.018033+0000", 2024-04-23T16:37:37.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012294199999999999 2024-04-23T16:37:37.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.023771+0000", 2024-04-23T16:37:37.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057386060000000003 2024-04-23T16:37:37.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.091034+0000", 2024-04-23T16:37:37.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067262418000000004 2024-04-23T16:37:37.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.091087+0000", 2024-04-23T16:37:37.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2689999999999999e-05 2024-04-23T16:37:37.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.091138+0000", 2024-04-23T16:37:37.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1870000000000003e-05 2024-04-23T16:37:37.927 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.927 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.928 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.928 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:274775 5.d 5:bef548fe:::benchmark_data_smithi188_113957_object121177:head [delete] snapc 0=[] ondisk+write+known_if_redirected e435)", 2024-04-23T16:37:37.928 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:19.024206+0000", 2024-04-23T16:37:37.928 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 78.830879147999994, 2024-04-23T16:37:37.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067015101999999993, 2024-04-23T16:37:37.928 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.928 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.928 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.928 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.928 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.928 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.929 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 274775 2024-04-23T16:37:37.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.929 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.929 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.024206+0000", 2024-04-23T16:37:37.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.929 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.024206+0000", 2024-04-23T16:37:37.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.024208+0000", 2024-04-23T16:37:37.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9870000000000002e-06 2024-04-23T16:37:37.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.024214+0000", 2024-04-23T16:37:37.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.64e-06 2024-04-23T16:37:37.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.024227+0000", 2024-04-23T16:37:37.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3124e-05 2024-04-23T16:37:37.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.024236+0000", 2024-04-23T16:37:37.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6130000000000004e-06 2024-04-23T16:37:37.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.024290+0000", 2024-04-23T16:37:37.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4021000000000003e-05 2024-04-23T16:37:37.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.024424+0000", 2024-04-23T16:37:37.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000133721 2024-04-23T16:37:37.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.024559+0000", 2024-04-23T16:37:37.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013498300000000001 2024-04-23T16:37:37.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.040733+0000", 2024-04-23T16:37:37.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01617416 2024-04-23T16:37:37.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.091146+0000", 2024-04-23T16:37:37.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.050413489999999998 2024-04-23T16:37:37.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.091187+0000", 2024-04-23T16:37:37.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0800000000000002e-05 2024-04-23T16:37:37.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.091221+0000", 2024-04-23T16:37:37.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3562999999999997e-05 2024-04-23T16:37:37.933 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.933 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.933 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.934 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:274783 5.d 5:b7ff44e5:::benchmark_data_smithi188_113957_object121185:head [delete] snapc 0=[] ondisk+write+known_if_redirected e435)", 2024-04-23T16:37:37.934 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:19.033760+0000", 2024-04-23T16:37:37.934 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 78.821325037999998, 2024-04-23T16:37:37.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.057534162, 2024-04-23T16:37:37.934 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.934 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.934 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.934 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.934 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.934 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.934 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 274783 2024-04-23T16:37:37.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.934 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.033760+0000", 2024-04-23T16:37:37.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.033760+0000", 2024-04-23T16:37:37.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.033762+0000", 2024-04-23T16:37:37.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4210000000000002e-06 2024-04-23T16:37:37.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.033770+0000", 2024-04-23T16:37:37.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1049999999999995e-06 2024-04-23T16:37:37.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.033786+0000", 2024-04-23T16:37:37.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5843000000000001e-05 2024-04-23T16:37:37.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.033797+0000", 2024-04-23T16:37:37.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1191e-05 2024-04-23T16:37:37.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.033863+0000", 2024-04-23T16:37:37.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.601e-05 2024-04-23T16:37:37.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.033999+0000", 2024-04-23T16:37:37.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000135833 2024-04-23T16:37:37.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.034129+0000", 2024-04-23T16:37:37.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000130147 2024-04-23T16:37:37.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.040757+0000", 2024-04-23T16:37:37.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0066271810000000002 2024-04-23T16:37:37.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.091226+0000", 2024-04-23T16:37:37.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.050469447000000001 2024-04-23T16:37:37.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.091262+0000", 2024-04-23T16:37:37.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6015e-05 2024-04-23T16:37:37.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.091294+0000", 2024-04-23T16:37:37.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1968999999999998e-05 2024-04-23T16:37:37.939 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.939 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.939 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.939 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:289596 5.d 5:bc0f71b6:::benchmark_data_smithi188_113957_object135998:head [delete] snapc 0=[] ondisk+write+known_if_redirected e440)", 2024-04-23T16:37:37.939 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:36.348411+0000", 2024-04-23T16:37:37.939 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 61.506673599999999, 2024-04-23T16:37:37.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062063621999999999, 2024-04-23T16:37:37.940 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.940 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.940 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.940 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.940 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.940 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.940 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 289596 2024-04-23T16:37:37.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.940 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:36.348411+0000", 2024-04-23T16:37:37.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:36.348411+0000", 2024-04-23T16:37:37.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:36.348414+0000", 2024-04-23T16:37:37.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6560000000000001e-06 2024-04-23T16:37:37.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:36.348420+0000", 2024-04-23T16:37:37.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9540000000000001e-06 2024-04-23T16:37:37.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:36.348433+0000", 2024-04-23T16:37:37.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3638000000000001e-05 2024-04-23T16:37:37.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:36.348442+0000", 2024-04-23T16:37:37.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6859999999999995e-06 2024-04-23T16:37:37.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:36.348500+0000", 2024-04-23T16:37:37.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7663e-05 2024-04-23T16:37:37.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:36.348618+0000", 2024-04-23T16:37:37.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118377 2024-04-23T16:37:37.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:36.348725+0000", 2024-04-23T16:37:37.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000106977 2024-04-23T16:37:37.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:36.353557+0000", 2024-04-23T16:37:37.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0048318939999999998 2024-04-23T16:37:37.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:36.410413+0000", 2024-04-23T16:37:37.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.056855743 2024-04-23T16:37:37.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:36.410446+0000", 2024-04-23T16:37:37.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3189999999999999e-05 2024-04-23T16:37:37.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:36.410475+0000", 2024-04-23T16:37:37.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8844e-05 2024-04-23T16:37:37.944 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.944 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.945 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.945 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:291106 5.d 5:ba6f9ce3:::benchmark_data_smithi188_113957_object137508:head [delete] snapc 0=[] ondisk+write+known_if_redirected e442)", 2024-04-23T16:37:37.945 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:38.048239+0000", 2024-04-23T16:37:37.945 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 59.806846135000001, 2024-04-23T16:37:37.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067926153000000003, 2024-04-23T16:37:37.945 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.945 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.945 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.945 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.945 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.945 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.945 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 291106 2024-04-23T16:37:37.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.946 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.048239+0000", 2024-04-23T16:37:37.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.048239+0000", 2024-04-23T16:37:37.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.048242+0000", 2024-04-23T16:37:37.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1370000000000002e-06 2024-04-23T16:37:37.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.048251+0000", 2024-04-23T16:37:37.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1139999999999995e-06 2024-04-23T16:37:37.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.048269+0000", 2024-04-23T16:37:37.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7688000000000001e-05 2024-04-23T16:37:37.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.048279+0000", 2024-04-23T16:37:37.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0335e-05 2024-04-23T16:37:37.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.048347+0000", 2024-04-23T16:37:37.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8032000000000002e-05 2024-04-23T16:37:37.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.048469+0000", 2024-04-23T16:37:37.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012229 2024-04-23T16:37:37.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.048575+0000", 2024-04-23T16:37:37.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010592800000000001 2024-04-23T16:37:37.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.053512+0000", 2024-04-23T16:37:37.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004936523 2024-04-23T16:37:37.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.116043+0000", 2024-04-23T16:37:37.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062531478000000001 2024-04-23T16:37:37.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.116114+0000", 2024-04-23T16:37:37.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0481999999999994e-05 2024-04-23T16:37:37.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.116165+0000", 2024-04-23T16:37:37.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1146000000000001e-05 2024-04-23T16:37:37.951 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.951 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.951 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.951 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:291110 5.d 5:b746b310:::benchmark_data_smithi188_113957_object137512:head [delete] snapc 0=[] ondisk+write+known_if_redirected e442)", 2024-04-23T16:37:37.951 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:38.053981+0000", 2024-04-23T16:37:37.951 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 59.801104100000003, 2024-04-23T16:37:37.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062318658999999998, 2024-04-23T16:37:37.951 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.951 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.951 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.951 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.951 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.951 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.952 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 291110 2024-04-23T16:37:37.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.952 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.053981+0000", 2024-04-23T16:37:37.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.053981+0000", 2024-04-23T16:37:37.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.053983+0000", 2024-04-23T16:37:37.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3949999999999999e-06 2024-04-23T16:37:37.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.053990+0000", 2024-04-23T16:37:37.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1149999999999996e-06 2024-04-23T16:37:37.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.054004+0000", 2024-04-23T16:37:37.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3305000000000001e-05 2024-04-23T16:37:37.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.054014+0000", 2024-04-23T16:37:37.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9699999999999994e-06 2024-04-23T16:37:37.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.054063+0000", 2024-04-23T16:37:37.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9614999999999998e-05 2024-04-23T16:37:37.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.054167+0000", 2024-04-23T16:37:37.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104028 2024-04-23T16:37:37.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.054259+0000", 2024-04-23T16:37:37.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1602999999999997e-05 2024-04-23T16:37:37.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.058066+0000", 2024-04-23T16:37:37.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0038074340000000002 2024-04-23T16:37:37.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.116214+0000", 2024-04-23T16:37:37.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.058147685999999997 2024-04-23T16:37:37.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.116261+0000", 2024-04-23T16:37:37.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7023e-05 2024-04-23T16:37:37.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.116299+0000", 2024-04-23T16:37:37.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8485000000000002e-05 2024-04-23T16:37:37.956 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.957 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.957 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.957 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:304250 5.d 5:bbcbb495:::benchmark_data_smithi188_113957_object150652:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-04-23T16:37:37.957 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:52.426086+0000", 2024-04-23T16:37:37.957 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 45.428998714000002, 2024-04-23T16:37:37.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.055118384999999999, 2024-04-23T16:37:37.957 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.957 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.957 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.957 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.957 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.957 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.958 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 304250 2024-04-23T16:37:37.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.958 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.426086+0000", 2024-04-23T16:37:37.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.426086+0000", 2024-04-23T16:37:37.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.959 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.959 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.426088+0000", 2024-04-23T16:37:37.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.426e-06 2024-04-23T16:37:37.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.959 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.959 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.426093+0000", 2024-04-23T16:37:37.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0629999999999996e-06 2024-04-23T16:37:37.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.959 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.959 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.426111+0000", 2024-04-23T16:37:37.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.808e-05 2024-04-23T16:37:37.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.960 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.426119+0000", 2024-04-23T16:37:37.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1280000000000008e-06 2024-04-23T16:37:37.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.960 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.426174+0000", 2024-04-23T16:37:37.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4824999999999998e-05 2024-04-23T16:37:37.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.960 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.426275+0000", 2024-04-23T16:37:37.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101622 2024-04-23T16:37:37.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.426381+0000", 2024-04-23T16:37:37.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105449 2024-04-23T16:37:37.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.430274+0000", 2024-04-23T16:37:37.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003893536 2024-04-23T16:37:37.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.481129+0000", 2024-04-23T16:37:37.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.050855150000000002 2024-04-23T16:37:37.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.481165+0000", 2024-04-23T16:37:37.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5735000000000003e-05 2024-04-23T16:37:37.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.481205+0000", 2024-04-23T16:37:37.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9371e-05 2024-04-23T16:37:37.962 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.963 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.963 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.963 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:304615 5.d 5:b8589301:::benchmark_data_smithi188_113957_object151017:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-04-23T16:37:37.963 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:52.820701+0000", 2024-04-23T16:37:37.963 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 45.034383974000001, 2024-04-23T16:37:37.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.059471313999999997, 2024-04-23T16:37:37.963 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.963 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.963 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.963 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.963 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.963 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.964 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 304615 2024-04-23T16:37:37.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.964 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.820701+0000", 2024-04-23T16:37:37.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.820701+0000", 2024-04-23T16:37:37.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.820703+0000", 2024-04-23T16:37:37.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.714e-06 2024-04-23T16:37:37.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.820708+0000", 2024-04-23T16:37:37.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.395e-06 2024-04-23T16:37:37.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.820718+0000", 2024-04-23T16:37:37.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0243e-05 2024-04-23T16:37:37.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.820727+0000", 2024-04-23T16:37:37.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2769999999999994e-06 2024-04-23T16:37:37.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.820785+0000", 2024-04-23T16:37:37.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7595e-05 2024-04-23T16:37:37.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.820920+0000", 2024-04-23T16:37:37.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000134788 2024-04-23T16:37:37.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.821068+0000", 2024-04-23T16:37:37.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014793200000000001 2024-04-23T16:37:37.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.826135+0000", 2024-04-23T16:37:37.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050673869999999996 2024-04-23T16:37:37.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.880103+0000", 2024-04-23T16:37:37.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.053967842000000002 2024-04-23T16:37:37.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.880142+0000", 2024-04-23T16:37:37.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8578999999999998e-05 2024-04-23T16:37:37.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.880172+0000", 2024-04-23T16:37:37.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0562000000000003e-05 2024-04-23T16:37:37.969 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.969 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.969 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.969 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:304628 5.d 5:b95cd153:::benchmark_data_smithi188_113957_object151030:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-04-23T16:37:37.969 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:52.827291+0000", 2024-04-23T16:37:37.969 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 45.027794049000001, 2024-04-23T16:37:37.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.052989912, 2024-04-23T16:37:37.970 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:37:37.970 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:37:37.970 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:37:37.970 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:37:37.970 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:37:37.970 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:37:37.970 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 304628 2024-04-23T16:37:37.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.970 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:37:37.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:37:37.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.827291+0000", 2024-04-23T16:37:37.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:37:37.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.827291+0000", 2024-04-23T16:37:37.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:37:37.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:37:37.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.827293+0000", 2024-04-23T16:37:37.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4600000000000002e-06 2024-04-23T16:37:37.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:37:37.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.827300+0000", 2024-04-23T16:37:37.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5189999999999998e-06 2024-04-23T16:37:37.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:37:37.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.827314+0000", 2024-04-23T16:37:37.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4121000000000001e-05 2024-04-23T16:37:37.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:37:37.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.827321+0000", 2024-04-23T16:37:37.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9789999999999996e-06 2024-04-23T16:37:37.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:37:37.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.827445+0000", 2024-04-23T16:37:37.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012383199999999999 2024-04-23T16:37:37.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:37:37.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.827562+0000", 2024-04-23T16:37:37.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000117576 2024-04-23T16:37:37.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:37:37.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.827645+0000", 2024-04-23T16:37:37.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3177999999999995e-05 2024-04-23T16:37:37.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:37:37.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.831571+0000", 2024-04-23T16:37:37.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039254260000000001 2024-04-23T16:37:37.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:37:37.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.880209+0000", 2024-04-23T16:37:37.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.048638065000000001 2024-04-23T16:37:37.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:37:37.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.880243+0000", 2024-04-23T16:37:37.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4490999999999999e-05 2024-04-23T16:37:37.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:37:37.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:37:37.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:37:37.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:52.880281+0000", 2024-04-23T16:37:37.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7265000000000003e-05 2024-04-23T16:37:37.975 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.975 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.975 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.975 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:37:37.975 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:37:37.975 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:37:38.137 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:38.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:38.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:38.242 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:37:38.287 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:38.327 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:37:38.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:38.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:37:38.432 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:37:38.449 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 7 16 29774 29758 265.676 212.25 0.0102646 0.00376062 2024-04-23T16:37:38.473 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:38.507 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:37:38.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:38.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:37:38.613 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:37:38.652 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:38.665 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-23T16:37:38.677 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:37:38.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:38.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:37:38.782 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:37:38.808 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:38.832 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:37:38.843 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:38.879 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:37:38.914 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:37:38.948 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:37:38.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:38 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/409931782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:37:38.985 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:37:39.014 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:37:39.040 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:37:39.065 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:37:39.093 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:37:39.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:38 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/409931782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:37:39.121 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:37:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:38 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/409931782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:37:39.150 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:37:39.150 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:37:39.150 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:37:39.150 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:37:39.150 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:37:39.151 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:37:39.151 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:37:39.151 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:37:39.151 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:37:39.151 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:37:39.151 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:37:39.151 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:37:39.151 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:37:39.407 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:37:39.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:39.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:37:39.450 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 8 16 32891 32875 256.816 194.812 0.00944163 0.00389097 2024-04-23T16:37:39.511 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:39.545 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:39.572 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:39.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:39.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:39.651 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:37:39.676 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:37:39.736 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:39.942 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:37:39.945 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:39 smithi103 ceph-mon[39629]: pgmap v1302: 27 pgs: 27 active+clean; 1.2 GiB data, 5.5 GiB used, 799 GiB / 805 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-04-23T16:37:39.945 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:39.945 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:39.945 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:39.945 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:39.945 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:39.945 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:39.945 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:39.945 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:39.946 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:39.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:39.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:37:40.048 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:40.099 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:39 smithi148 ceph-mon[44520]: pgmap v1302: 27 pgs: 27 active+clean; 1.2 GiB data, 5.5 GiB used, 799 GiB / 805 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-04-23T16:37:40.100 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:40.100 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:40.100 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:40.100 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:40.100 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:40.100 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:40.100 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:40.100 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:40.100 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:40.110 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:40.136 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:40.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:40.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:40.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:39 smithi188 ceph-mon[44101]: pgmap v1302: 27 pgs: 27 active+clean; 1.2 GiB data, 5.5 GiB used, 799 GiB / 805 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-04-23T16:37:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:40.241 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:37:40.287 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:40.325 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:37:40.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:40.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:37:40.430 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:37:40.450 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 9 16 35921 35905 249.321 189.375 0.00464152 0.00400916 2024-04-23T16:37:40.463 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:40.485 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:37:40.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:40.488 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:37:40.568 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:37:40.568 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:37:40.570 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:40.589 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:40.638 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:40.667 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:40.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:40.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:40.773 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:37:40.803 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:40.828 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:37:40.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:40.831 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:37:40.932 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:37:40.963 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:40.985 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:37:40.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:40.988 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:37:41.089 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:37:41.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:41.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:41.117 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:41.117 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:41.119 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:41.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:41.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:41.141 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:37:41.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:41.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:37:41.245 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:37:41.276 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:41.298 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:37:41.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:41.304 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:37:41.405 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:41.437 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:41.450 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 10 16 38820 38804 242.507 181.188 0.00433405 0.00412148 2024-04-23T16:37:41.458 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:41.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:41.461 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:41.562 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:37:41.604 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:41.629 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:37:41.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:41.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:37:41.733 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:41.770 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:41.805 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:41.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:41.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:41.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:41 smithi148 ceph-mon[44520]: pgmap v1303: 27 pgs: 27 active+clean; 1.9 GiB data, 7.2 GiB used, 797 GiB / 805 GiB avail; 193 MiB/s wr, 3.08k op/s 2024-04-23T16:37:41.912 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:37:41.953 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:41.984 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:37:41.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:41.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:37:42.092 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:42.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:41 smithi103 ceph-mon[39629]: pgmap v1303: 27 pgs: 27 active+clean; 1.9 GiB data, 7.2 GiB used, 797 GiB / 805 GiB avail; 193 MiB/s wr, 3.08k op/s 2024-04-23T16:37:42.128 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:42.139 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:42.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:41 smithi188 ceph-mon[44101]: pgmap v1303: 27 pgs: 27 active+clean; 1.9 GiB data, 7.2 GiB used, 797 GiB / 805 GiB avail; 193 MiB/s wr, 3.08k op/s 2024-04-23T16:37:42.287 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:42.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:42.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:42.396 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:42.426 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:42.449 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:42.450 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 11 16 41634 41618 236.448 175.875 0.00392813 0.0042273 2024-04-23T16:37:42.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:42.453 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:42.554 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:37:42.569 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:37:42.584 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:42.605 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:37:42.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:42.609 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:37:42.710 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:42.741 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:42.762 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:42.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:42.766 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:42.866 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:37:42.910 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:42.942 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:37:42.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:42.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:37:43.050 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:43.081 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:43.450 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 12 15 44272 44257 230.488 164.938 0.00924197 0.0043365 2024-04-23T16:37:44.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:43 smithi148 ceph-mon[44520]: pgmap v1304: 27 pgs: 27 active+clean; 2.1 GiB data, 7.7 GiB used, 797 GiB / 805 GiB avail; 217 MiB/s wr, 3.47k op/s 2024-04-23T16:37:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:43 smithi188 ceph-mon[44101]: pgmap v1304: 27 pgs: 27 active+clean; 2.1 GiB data, 7.7 GiB used, 797 GiB / 805 GiB avail; 217 MiB/s wr, 3.47k op/s 2024-04-23T16:37:44.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:43 smithi103 ceph-mon[39629]: pgmap v1304: 27 pgs: 27 active+clean; 2.1 GiB data, 7.7 GiB used, 797 GiB / 805 GiB avail; 217 MiB/s wr, 3.47k op/s 2024-04-23T16:37:44.450 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 13 16 46836 46820 225.079 160.188 0.00442904 0.00444134 2024-04-23T16:37:45.450 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 14 16 49333 49317 220.148 156.062 0.00169483 0.00453958 2024-04-23T16:37:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:45 smithi188 ceph-mon[44101]: pgmap v1305: 27 pgs: 27 active+clean; 2.4 GiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 217 MiB/s wr, 3.47k op/s 2024-04-23T16:37:45.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:45 smithi103 ceph-mon[39629]: pgmap v1305: 27 pgs: 27 active+clean; 2.4 GiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 217 MiB/s wr, 3.47k op/s 2024-04-23T16:37:45.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:45 smithi148 ceph-mon[44520]: pgmap v1305: 27 pgs: 27 active+clean; 2.4 GiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 217 MiB/s wr, 3.47k op/s 2024-04-23T16:37:46.167 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:37:46.168 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:37:46.450 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 15 16 51751 51735 215.546 151.125 0.0043034 0.00463731 2024-04-23T16:37:46.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:46 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:37:46.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:46 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:37:46.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:46 smithi188 ceph-mon[44101]: from='client.16332 v1:172.21.15.103:0/1342852411' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:37:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:46 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:37:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:46 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:37:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:46 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:37:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:46 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:37:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:46 smithi103 ceph-mon[39629]: from='client.16332 v1:172.21.15.103:0/1342852411' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:37:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:46 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:37:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:46 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:37:46.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:46 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:37:46.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:46 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:37:46.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:46 smithi148 ceph-mon[44520]: from='client.16332 v1:172.21.15.103:0/1342852411' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:37:46.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:46 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:37:46.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:46 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:37:47.452 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 16 16 54094 54078 211.208 146.438 0.00587767 0.00473278 2024-04-23T16:37:47.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:47 smithi103 ceph-mon[39629]: osdmap e461: 12 total, 11 up, 9 in 2024-04-23T16:37:47.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:47 smithi103 ceph-mon[39629]: pgmap v1307: 27 pgs: 27 active+clean; 2.9 GiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 196 MiB/s wr, 3.13k op/s 2024-04-23T16:37:47.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:47 smithi148 ceph-mon[44520]: osdmap e461: 12 total, 11 up, 9 in 2024-04-23T16:37:47.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:47 smithi148 ceph-mon[44520]: pgmap v1307: 27 pgs: 27 active+clean; 2.9 GiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 196 MiB/s wr, 3.13k op/s 2024-04-23T16:37:47.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:47 smithi188 ceph-mon[44101]: osdmap e461: 12 total, 11 up, 9 in 2024-04-23T16:37:47.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:47 smithi188 ceph-mon[44101]: pgmap v1307: 27 pgs: 27 active+clean; 2.9 GiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 196 MiB/s wr, 3.13k op/s 2024-04-23T16:37:48.452 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 17 16 56367 56351 207.14 142.062 0.00270353 0.00482564 2024-04-23T16:37:48.693 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1305,"stamp":"2024-04-23T16:37:44.759526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2627273248,"num_objects":40082,"num_object_clones":0,"num_object_copies":80164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40080,"num_write_kb":2565120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41609,"ondisk_log_size":41609,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":8880764,"kb_used_data":5083636,"kb_used_omap":14,"kb_used_meta":3797041,"kb_avail":834825604,"statfs":{"total":863955320832,"available":854861418496,"internally_reserved":0,"allocated":5205643264,"data_stored":5202337507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3888170991},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2273902592,"num_objects":34697,"num_object_clones":0,"num_object_copies":69394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34697,"num_write_kb":2220608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002001"},"pg_stats":[{"pgid":"6.c","version":"460'2224","reported_seq":2243,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:40.777870+0000","last_change":"2024-04-23T16:37:15.833831+0000","last_active":"2024-04-23T16:37:40.777870+0000","last_peered":"2024-04-23T16:37:40.777870+0000","last_clean":"2024-04-23T16:37:40.777870+0000","last_became_active":"2024-04-23T16:37:15.833495+0000","last_became_peered":"2024-04-23T16:37:15.833495+0000","last_unstale":"2024-04-23T16:37:40.777870+0000","last_undegraded":"2024-04-23T16:37:40.777870+0000","last_fullsized":"2024-04-23T16:37:40.777870+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":2224,"log_dups_size":0,"ondisk_log_size":2224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:29.352334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145752064,"num_objects":2224,"num_object_clones":0,"num_object_copies":4448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2224,"num_write_kb":142336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"460'2316","reported_seq":2335,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:40.777726+0000","last_change":"2024-04-23T16:37:15.838829+0000","last_active":"2024-04-23T16:37:40.777726+0000","last_peered":"2024-04-23T16:37:40.777726+0000","last_clean":"2024-04-23T16:37:40.777726+0000","last_became_active":"2024-04-23T16:37:15.838559+0000","last_became_peered":"2024-04-23T16:37:15.838559+0000","last_unstale":"2024-04-23T16:37:40.777726+0000","last_undegraded":"2024-04-23T16:37:40.777726+0000","last_fullsized":"2024-04-23T16:37:40.777726+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":2316,"log_dups_size":0,"ondisk_log_size":2316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:35.797306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151781376,"num_objects":2316,"num_object_clones":0,"num_object_copies":4632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2316,"num_write_kb":148224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"460'2248","reported_seq":2267,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:40.402906+0000","last_change":"2024-04-23T16:37:15.837846+0000","last_active":"2024-04-23T16:37:40.402906+0000","last_peered":"2024-04-23T16:37:40.402906+0000","last_clean":"2024-04-23T16:37:40.402906+0000","last_became_active":"2024-04-23T16:37:15.837590+0000","last_became_peered":"2024-04-23T16:37:15.837590+0000","last_unstale":"2024-04-23T16:37:40.402906+0000","last_undegraded":"2024-04-23T16:37:40.402906+0000","last_fullsized":"2024-04-23T16:37:40.402906+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":2248,"log_dups_size":0,"ondisk_log_size":2248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:20.622753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147324928,"num_objects":2248,"num_object_clones":0,"num_object_copies":4496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2248,"num_write_kb":143872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"460'2859","reported_seq":2878,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:44.167981+0000","last_change":"2024-04-23T16:37:15.834978+0000","last_active":"2024-04-23T16:37:44.167981+0000","last_peered":"2024-04-23T16:37:44.167981+0000","last_clean":"2024-04-23T16:37:44.167981+0000","last_became_active":"2024-04-23T16:37:15.834740+0000","last_became_peered":"2024-04-23T16:37:15.834740+0000","last_unstale":"2024-04-23T16:37:44.167981+0000","last_undegraded":"2024-04-23T16:37:44.167981+0000","last_fullsized":"2024-04-23T16:37:44.167981+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":2859,"log_dups_size":0,"ondisk_log_size":2859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:40.729580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187367424,"num_objects":2859,"num_object_clones":0,"num_object_copies":5718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":182976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"460'2491","reported_seq":2510,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:41.806558+0000","last_change":"2024-04-23T16:37:15.839856+0000","last_active":"2024-04-23T16:37:41.806558+0000","last_peered":"2024-04-23T16:37:41.806558+0000","last_clean":"2024-04-23T16:37:41.806558+0000","last_became_active":"2024-04-23T16:37:15.839667+0000","last_became_peered":"2024-04-23T16:37:15.839667+0000","last_unstale":"2024-04-23T16:37:41.806558+0000","last_undegraded":"2024-04-23T16:37:41.806558+0000","last_fullsized":"2024-04-23T16:37:41.806558+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":2491,"log_dups_size":0,"ondisk_log_size":2491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:33.221933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163250176,"num_objects":2491,"num_object_clones":0,"num_object_copies":4982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2491,"num_write_kb":159424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.d","version":"460'2198","reported_seq":2217,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:40.406166+0000","last_change":"2024-04-23T16:37:15.836920+0000","last_active":"2024-04-23T16:37:40.406166+0000","last_peered":"2024-04-23T16:37:40.406166+0000","last_clean":"2024-04-23T16:37:40.406166+0000","last_became_active":"2024-04-23T16:37:15.836635+0000","last_became_peered":"2024-04-23T16:37:15.836635+0000","last_unstale":"2024-04-23T16:37:40.406166+0000","last_undegraded":"2024-04-23T16:37:40.406166+0000","last_fullsized":"2024-04-23T16:37:40.406166+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":2198,"log_dups_size":0,"ondisk_log_size":2198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:28.500044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144048128,"num_objects":2198,"num_object_clones":0,"num_object_copies":4396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2198,"num_write_kb":140672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1190,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:35.220344+0000","last_change":"2024-04-23T16:36:19.132015+0000","last_active":"2024-04-23T16:37:35.220344+0000","last_peered":"2024-04-23T16:37:35.220344+0000","last_clean":"2024-04-23T16:37:35.220344+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:37:35.220344+0000","last_undegraded":"2024-04-23T16:37:35.220344+0000","last_fullsized":"2024-04-23T16:37:35.220344+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:19.131883+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:19.131883+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:41.022603+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.f","version":"460'2815","reported_seq":2834,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:44.326394+0000","last_change":"2024-04-23T16:37:15.836051+0000","last_active":"2024-04-23T16:37:44.326394+0000","last_peered":"2024-04-23T16:37:44.326394+0000","last_clean":"2024-04-23T16:37:44.326394+0000","last_became_active":"2024-04-23T16:37:15.835794+0000","last_became_peered":"2024-04-23T16:37:15.835794+0000","last_unstale":"2024-04-23T16:37:44.326394+0000","last_undegraded":"2024-04-23T16:37:44.326394+0000","last_fullsized":"2024-04-23T16:37:44.326394+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":2815,"log_dups_size":0,"ondisk_log_size":2815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:33.244802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184483840,"num_objects":2815,"num_object_clones":0,"num_object_copies":5630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2815,"num_write_kb":180160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1185,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:35.218345+0000","last_change":"2024-04-23T16:37:05.612328+0000","last_active":"2024-04-23T16:37:35.218345+0000","last_peered":"2024-04-23T16:37:35.218345+0000","last_clean":"2024-04-23T16:37:35.218345+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:37:35.218345+0000","last_undegraded":"2024-04-23T16:37:35.218345+0000","last_fullsized":"2024-04-23T16:37:35.218345+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:05.612182+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:37:05.612182+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:24.096518+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"460'2995","reported_seq":3014,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:44.705571+0000","last_change":"2024-04-23T16:37:15.837702+0000","last_active":"2024-04-23T16:37:44.705571+0000","last_peered":"2024-04-23T16:37:44.705571+0000","last_clean":"2024-04-23T16:37:44.705571+0000","last_became_active":"2024-04-23T16:37:15.837406+0000","last_became_peered":"2024-04-23T16:37:15.837406+0000","last_unstale":"2024-04-23T16:37:44.705571+0000","last_undegraded":"2024-04-23T16:37:44.705571+0000","last_fullsized":"2024-04-23T16:37:44.705571+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":2995,"log_dups_size":0,"ondisk_log_size":2995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:34.353376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196280320,"num_objects":2995,"num_object_clones":0,"num_object_copies":5990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2995,"num_write_kb":191680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1176,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:35.219950+0000","last_change":"2024-04-23T16:37:05.696710+0000","last_active":"2024-04-23T16:37:35.219950+0000","last_peered":"2024-04-23T16:37:35.219950+0000","last_clean":"2024-04-23T16:37:35.219950+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:37:35.219950+0000","last_undegraded":"2024-04-23T16:37:35.219950+0000","last_fullsized":"2024-04-23T16:37:35.219950+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:05.696583+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:37:05.696583+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:25.899474+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"460'2247","reported_seq":2266,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:40.282174+0000","last_change":"2024-04-23T16:37:15.839946+0000","last_active":"2024-04-23T16:37:40.282174+0000","last_peered":"2024-04-23T16:37:40.282174+0000","last_clean":"2024-04-23T16:37:40.282174+0000","last_became_active":"2024-04-23T16:37:15.839669+0000","last_became_peered":"2024-04-23T16:37:15.839669+0000","last_unstale":"2024-04-23T16:37:40.282174+0000","last_undegraded":"2024-04-23T16:37:40.282174+0000","last_fullsized":"2024-04-23T16:37:40.282174+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":2247,"log_dups_size":0,"ondisk_log_size":2247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:22.953490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147259392,"num_objects":2247,"num_object_clones":0,"num_object_copies":4494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2247,"num_write_kb":143808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1190,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:35.218426+0000","last_change":"2024-04-23T16:36:26.456131+0000","last_active":"2024-04-23T16:37:35.218426+0000","last_peered":"2024-04-23T16:37:35.218426+0000","last_clean":"2024-04-23T16:37:35.218426+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:37:35.218426+0000","last_undegraded":"2024-04-23T16:37:35.218426+0000","last_fullsized":"2024-04-23T16:37:35.218426+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:26.456003+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:26.456003+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:44.948237+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"460'2276","reported_seq":2294,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:40.409533+0000","last_change":"2024-04-23T16:37:15.837187+0000","last_active":"2024-04-23T16:37:40.409533+0000","last_peered":"2024-04-23T16:37:40.409533+0000","last_clean":"2024-04-23T16:37:40.409533+0000","last_became_active":"2024-04-23T16:37:15.836264+0000","last_became_peered":"2024-04-23T16:37:15.836264+0000","last_unstale":"2024-04-23T16:37:40.409533+0000","last_undegraded":"2024-04-23T16:37:40.409533+0000","last_fullsized":"2024-04-23T16:37:40.409533+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":2276,"log_dups_size":0,"ondisk_log_size":2276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:40.409891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149159936,"num_objects":2276,"num_object_clones":0,"num_object_copies":4552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2276,"num_write_kb":145664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1191,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:35.215875+0000","last_change":"2024-04-23T16:37:05.363014+0000","last_active":"2024-04-23T16:37:35.215875+0000","last_peered":"2024-04-23T16:37:35.215875+0000","last_clean":"2024-04-23T16:37:35.215875+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:37:35.215875+0000","last_undegraded":"2024-04-23T16:37:35.215875+0000","last_fullsized":"2024-04-23T16:37:35.215875+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:05.362908+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:37:05.362908+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:14.719978+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"460'2391","reported_seq":2410,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:40.781886+0000","last_change":"2024-04-23T16:37:15.835369+0000","last_active":"2024-04-23T16:37:40.781886+0000","last_peered":"2024-04-23T16:37:40.781886+0000","last_clean":"2024-04-23T16:37:40.781886+0000","last_became_active":"2024-04-23T16:37:15.835131+0000","last_became_peered":"2024-04-23T16:37:15.835131+0000","last_unstale":"2024-04-23T16:37:40.781886+0000","last_undegraded":"2024-04-23T16:37:40.781886+0000","last_fullsized":"2024-04-23T16:37:40.781886+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":2391,"log_dups_size":0,"ondisk_log_size":2391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:30.571812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156696576,"num_objects":2391,"num_object_clones":0,"num_object_copies":4782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2391,"num_write_kb":153024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1189,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:35.220354+0000","last_change":"2024-04-23T16:36:23.239567+0000","last_active":"2024-04-23T16:37:35.220354+0000","last_peered":"2024-04-23T16:37:35.220354+0000","last_clean":"2024-04-23T16:37:35.220354+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:37:35.220354+0000","last_undegraded":"2024-04-23T16:37:35.220354+0000","last_fullsized":"2024-04-23T16:37:35.220354+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:23.239466+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:36:23.239466+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:35.249326+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.1","version":"460'2706","reported_seq":2725,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:43.229876+0000","last_change":"2024-04-23T16:37:15.834693+0000","last_active":"2024-04-23T16:37:43.229876+0000","last_peered":"2024-04-23T16:37:43.229876+0000","last_clean":"2024-04-23T16:37:43.229876+0000","last_became_active":"2024-04-23T16:37:15.834476+0000","last_became_peered":"2024-04-23T16:37:15.834476+0000","last_unstale":"2024-04-23T16:37:43.229876+0000","last_undegraded":"2024-04-23T16:37:43.229876+0000","last_fullsized":"2024-04-23T16:37:43.229876+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":2706,"log_dups_size":0,"ondisk_log_size":2706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:18.272428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177340416,"num_objects":2706,"num_object_clones":0,"num_object_copies":5412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2706,"num_write_kb":173184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1190,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:35.220498+0000","last_change":"2024-04-23T16:36:18.085504+0000","last_active":"2024-04-23T16:37:35.220498+0000","last_peered":"2024-04-23T16:37:35.220498+0000","last_clean":"2024-04-23T16:37:35.220498+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:37:35.220498+0000","last_undegraded":"2024-04-23T16:37:35.220498+0000","last_fullsized":"2024-04-23T16:37:35.220498+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:18.085386+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:18.085386+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:35.249306+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.0","version":"460'2829","reported_seq":2848,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:44.190824+0000","last_change":"2024-04-23T16:37:15.836339+0000","last_active":"2024-04-23T16:37:44.190824+0000","last_peered":"2024-04-23T16:37:44.190824+0000","last_clean":"2024-04-23T16:37:44.190824+0000","last_became_active":"2024-04-23T16:37:15.836027+0000","last_became_peered":"2024-04-23T16:37:15.836027+0000","last_unstale":"2024-04-23T16:37:44.190824+0000","last_undegraded":"2024-04-23T16:37:44.190824+0000","last_fullsized":"2024-04-23T16:37:44.190824+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":2829,"log_dups_size":0,"ondisk_log_size":2829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:28.804087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185401344,"num_objects":2829,"num_object_clones":0,"num_object_copies":5658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2829,"num_write_kb":181056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1191,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:35.215727+0000","last_change":"2024-04-23T16:37:06.338083+0000","last_active":"2024-04-23T16:37:35.215727+0000","last_peered":"2024-04-23T16:37:35.215727+0000","last_clean":"2024-04-23T16:37:35.215727+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:37:35.215727+0000","last_undegraded":"2024-04-23T16:37:35.215727+0000","last_fullsized":"2024-04-23T16:37:35.215727+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:06.337940+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:37:06.337940+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:22.707182+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"460'2262","reported_seq":2281,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:40.276559+0000","last_change":"2024-04-23T16:37:15.839998+0000","last_active":"2024-04-23T16:37:40.276559+0000","last_peered":"2024-04-23T16:37:40.276559+0000","last_clean":"2024-04-23T16:37:40.276559+0000","last_became_active":"2024-04-23T16:37:15.839714+0000","last_became_peered":"2024-04-23T16:37:15.839714+0000","last_unstale":"2024-04-23T16:37:40.276559+0000","last_undegraded":"2024-04-23T16:37:40.276559+0000","last_fullsized":"2024-04-23T16:37:40.276559+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":2262,"log_dups_size":0,"ondisk_log_size":2262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:16.353176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148242432,"num_objects":2262,"num_object_clones":0,"num_object_copies":4524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2262,"num_write_kb":144768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1082,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:35.220331+0000","last_change":"2024-04-23T16:36:29.341359+0000","last_active":"2024-04-23T16:37:35.220331+0000","last_peered":"2024-04-23T16:37:35.220331+0000","last_clean":"2024-04-23T16:37:35.220331+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:37:35.220331+0000","last_undegraded":"2024-04-23T16:37:35.220331+0000","last_fullsized":"2024-04-23T16:37:35.220331+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:29.341205+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:29.341205+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:50.871176+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"460'2942","reported_seq":2961,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:44.687712+0000","last_change":"2024-04-23T16:37:15.836315+0000","last_active":"2024-04-23T16:37:44.687712+0000","last_peered":"2024-04-23T16:37:44.687712+0000","last_clean":"2024-04-23T16:37:44.687712+0000","last_became_active":"2024-04-23T16:37:15.836050+0000","last_became_peered":"2024-04-23T16:37:15.836050+0000","last_unstale":"2024-04-23T16:37:44.687712+0000","last_undegraded":"2024-04-23T16:37:44.687712+0000","last_fullsized":"2024-04-23T16:37:44.687712+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":2942,"log_dups_size":0,"ondisk_log_size":2942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:39.323249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192806912,"num_objects":2942,"num_object_clones":0,"num_object_copies":5884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2942,"num_write_kb":188288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1004,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:35.216915+0000","last_change":"2024-04-23T16:36:19.091622+0000","last_active":"2024-04-23T16:37:35.216915+0000","last_peered":"2024-04-23T16:37:35.216915+0000","last_clean":"2024-04-23T16:37:35.216915+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:37:35.216915+0000","last_undegraded":"2024-04-23T16:37:35.216915+0000","last_fullsized":"2024-04-23T16:37:35.216915+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:19.091490+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:36:19.091490+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:37.502395+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"460'2281","reported_seq":2300,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:40.415879+0000","last_change":"2024-04-23T16:37:15.836721+0000","last_active":"2024-04-23T16:37:40.415879+0000","last_peered":"2024-04-23T16:37:40.415879+0000","last_clean":"2024-04-23T16:37:40.415879+0000","last_became_active":"2024-04-23T16:37:15.836376+0000","last_became_peered":"2024-04-23T16:37:15.836376+0000","last_unstale":"2024-04-23T16:37:40.415879+0000","last_undegraded":"2024-04-23T16:37:40.415879+0000","last_fullsized":"2024-04-23T16:37:40.415879+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":2281,"log_dups_size":0,"ondisk_log_size":2281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:15.626663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149487616,"num_objects":2281,"num_object_clones":0,"num_object_copies":4562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2281,"num_write_kb":145984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1180,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-04-23T16:37:35.219996+0000","last_change":"2024-04-23T16:37:12.716078+0000","last_active":"2024-04-23T16:37:35.219996+0000","last_peered":"2024-04-23T16:37:35.219996+0000","last_clean":"2024-04-23T16:37:35.219996+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:37:35.219996+0000","last_undegraded":"2024-04-23T16:37:35.219996+0000","last_fullsized":"2024-04-23T16:37:35.219996+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:12.715954+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:37:12.715954+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:27.717888+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":2626682880,"num_objects":40080,"num_object_clones":0,"num_object_copies":80160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40080,"num_write_kb":2565120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5331288064,"data_stored":5331288064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40080,"ondisk_log_size":40080,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056909,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":673228,"kb_used_data":160324,"kb_used_omap":1,"kb_used_meta":512894,"kb_avail":93071924,"statfs":{"total":95995035648,"available":95305650176,"internally_reserved":0,"allocated":164171776,"data_stored":163805259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":525203908},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351250,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612889,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902668,"kb_used_data":542468,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92842484,"statfs":{"total":95995035648,"available":95070703616,"internally_reserved":0,"allocated":555487232,"data_stored":555120715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":368835008},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841821,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":462156,"kb_used_data":164420,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":93282996,"statfs":{"total":95995035648,"available":95521787904,"internally_reserved":0,"allocated":168366080,"data_stored":167999563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168865,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":947596,"kb_used_data":551300,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92797556,"statfs":{"total":95995035648,"available":95024697344,"internally_reserved":0,"allocated":564531200,"data_stored":564164683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":405797322},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463208,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1030796,"kb_used_data":706692,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92714356,"statfs":{"total":95995035648,"available":94939500544,"internally_reserved":0,"allocated":723652608,"data_stored":723286091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":331872710},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137348,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738693,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":993036,"kb_used_data":524548,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":92752116,"statfs":{"total":95995035648,"available":94978166784,"internally_reserved":0,"allocated":537137152,"data_stored":536770635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117301,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1418452,"kb_used_data":1026316,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92326700,"statfs":{"total":95995035648,"available":94542540800,"internally_reserved":0,"allocated":1050947584,"data_stored":1050577515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":401537479},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921072,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1127820,"kb_used_data":555204,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92617332,"statfs":{"total":95995035648,"available":94840147968,"internally_reserved":0,"allocated":568528896,"data_stored":568162379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757546,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1325012,"kb_used_data":852364,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92420140,"statfs":{"total":95995035648,"available":94638223360,"internally_reserved":0,"allocated":872820736,"data_stored":872450667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":561184768,"data_stored":561184768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1059913728,"data_stored":1059913728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":589692928,"data_stored":589692928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":879558656,"data_stored":879558656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":750059520,"data_stored":750059520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":588054528,"data_stored":588054528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":177340416,"data_stored":177340416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":562233344,"data_stored":562233344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":163250176,"data_stored":163250176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:37:49.085 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:49.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:49.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:49.189 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:37:49.245 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:49.275 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:37:49.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:49.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:37:49.380 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:49.411 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:49.435 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:49.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:49.438 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:49.452 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 18 16 58620 58604 203.455 140.812 0.00697396 0.00491349 2024-04-23T16:37:49.534 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:37:49.534 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:37:49.539 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:37:49.565 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:49 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:37:49.565 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:49 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:37:49.565 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:49 smithi188 ceph-mon[44101]: osdmap e462: 12 total, 11 up, 9 in 2024-04-23T16:37:49.565 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:49 smithi188 ceph-mon[44101]: pgmap v1309: 27 pgs: 27 active+clean; 3.0 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-04-23T16:37:49.567 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:49.595 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:37:49.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:49.599 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:37:49.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:49 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:37:49.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:49 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:37:49.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:49 smithi103 ceph-mon[39629]: osdmap e462: 12 total, 11 up, 9 in 2024-04-23T16:37:49.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:49 smithi103 ceph-mon[39629]: pgmap v1309: 27 pgs: 27 active+clean; 3.0 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-04-23T16:37:49.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:49 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:37:49.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:49 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:37:49.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:49 smithi148 ceph-mon[44520]: osdmap e462: 12 total, 11 up, 9 in 2024-04-23T16:37:49.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:49 smithi148 ceph-mon[44520]: pgmap v1309: 27 pgs: 27 active+clean; 3.0 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-04-23T16:37:49.699 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:49.727 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:49.748 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:49.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:49.752 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:49.853 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:37:49.890 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:50.346 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:37:50.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:50.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:37:50.362 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-23T16:37:50.450 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:37:50.452 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 19 16 60884 60868 200.194 141.5 0.00945418 0.00499352 2024-04-23T16:37:50.490 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:50.550 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:37:50.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:50.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:37:50.656 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:50.709 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:50.742 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:50.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:50.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:50.848 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:37:50.877 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:50.899 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:37:50.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:50.910 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:37:50.922 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:51.011 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:51.044 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:51.252 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:51.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:51.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:51.357 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:37:51.409 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:51.436 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:37:51.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:51.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:37:51.452 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:37:51.452099 min lat: 0.000989845 max lat: 0.0668246 avg lat: 0.00508292 2024-04-23T16:37:51.452 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:37:51.452 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 20 16 62952 62936 196.647 129.25 0.00584886 0.00508292 2024-04-23T16:37:51.543 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:37:51.589 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:51.615 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:37:51.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:51.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:37:51.696 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:37:51.724 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:51.780 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:51.801 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:51.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:51.804 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:51.821 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:51 smithi148 ceph-mon[44520]: pgmap v1310: 27 pgs: 27 active+clean; 3.5 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 174 MiB/s wr, 2.79k op/s 2024-04-23T16:37:51.904 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:37:51.933 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:51.957 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:37:51.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:51.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:37:52.062 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:37:52.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:51 smithi103 ceph-mon[39629]: pgmap v1310: 27 pgs: 27 active+clean; 3.5 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 174 MiB/s wr, 2.79k op/s 2024-04-23T16:37:52.116 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:52.143 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:37:52.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:52.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:37:52.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:51 smithi188 ceph-mon[44101]: pgmap v1310: 27 pgs: 27 active+clean; 3.5 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 174 MiB/s wr, 2.79k op/s 2024-04-23T16:37:52.251 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:37:52.283 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:52.306 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:37:52.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:52.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:37:52.409 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:37:52.439 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:52.452 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 21 15 65007 64992 193.401 128.5 0.0104299 0.00516899 2024-04-23T16:37:52.460 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:37:52.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:52.463 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:37:52.564 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:37:52.612 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:52.650 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:37:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:52.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:37:52.756 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:37:52.784 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:52.806 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:37:52.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:52.809 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:37:52.910 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:37:52.938 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:52.960 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:37:52.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:52.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:37:53.065 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:37:53.392 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:53.452 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 22 16 67000 66984 190.269 124.5 0.00735597 0.00525404 2024-04-23T16:37:53.833 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:37:53.895 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:53.955 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:37:53.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:53.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:37:54.060 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:54.096 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:53 smithi103 ceph-mon[39629]: pgmap v1311: 27 pgs: 27 active+clean; 3.7 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 156 MiB/s wr, 2.50k op/s 2024-04-23T16:37:54.097 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:53 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1306109661' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:37:54.100 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:54.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:53 smithi148 ceph-mon[44520]: pgmap v1311: 27 pgs: 27 active+clean; 3.7 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 156 MiB/s wr, 2.50k op/s 2024-04-23T16:37:54.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:53 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1306109661' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:37:54.162 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:54.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:54.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:54.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:53 smithi188 ceph-mon[44101]: pgmap v1311: 27 pgs: 27 active+clean; 3.7 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 156 MiB/s wr, 2.50k op/s 2024-04-23T16:37:54.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:53 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1306109661' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:37:54.265 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:37:54.320 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:54.355 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:37:54.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:54.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:37:54.452 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 23 16 68986 68970 187.393 124.125 0.00606045 0.00533466 2024-04-23T16:37:54.460 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:37:54.512 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:54.522 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:37:54.522 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:37:54.522 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:37:54.522 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:37:54.522 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:37:54.522 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:37:54.522 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:37:54.522 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:37:54.523 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:37:54.523 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:37:54.523 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:37:54.523 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:37:54.523 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:37:54.523 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:37:54.523 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:37:54.523 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:37:54.523 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:37:54.523 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:37:54.523 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:37:54.523 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:37:54.524 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:37:54.524 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:37:54.524 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:37:54.544 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:37:54.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:54.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:37:54.651 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:54.714 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:54.752 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:54.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:54.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:54.859 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:37:54.893 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:55.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:54 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:37:55.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:54 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1306109661' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:37:55.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:54 smithi103 ceph-mon[39629]: osdmap e463: 12 total, 11 up, 9 in 2024-04-23T16:37:55.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:54 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2014772097' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:37:55.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:54 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:37:55.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:54 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1306109661' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:37:55.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:54 smithi148 ceph-mon[44520]: osdmap e463: 12 total, 11 up, 9 in 2024-04-23T16:37:55.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:54 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2014772097' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:37:55.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:54 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:37:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:54 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1306109661' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:37:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:54 smithi188 ceph-mon[44101]: osdmap e463: 12 total, 11 up, 9 in 2024-04-23T16:37:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:54 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2014772097' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:37:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:55.300 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:37:55.300 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:37:55.452 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 24 16 70923 70907 184.629 121.062 0.00865831 0.00541476 2024-04-23T16:37:55.849 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:37:55.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:55.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:37:55.959 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:37:55.998 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:56.044 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:37:56.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:56.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:37:56.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:56.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:56.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:56.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:55 smithi103 ceph-mon[39629]: pgmap v1313: 27 pgs: 27 active+clean; 3.9 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-04-23T16:37:56.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:56.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:56.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:56.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:55 smithi148 ceph-mon[44520]: pgmap v1313: 27 pgs: 27 active+clean; 3.9 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-04-23T16:37:56.154 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:56.194 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:56.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:37:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:55 smithi188 ceph-mon[44101]: pgmap v1313: 27 pgs: 27 active+clean; 3.9 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-04-23T16:37:56.452 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 25 16 72878 72862 182.131 122.188 0.00822434 0.00548903 2024-04-23T16:37:56.466 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:56.853 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:56.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:56.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:37:56.958 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:37:56.995 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:57.302 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:37:57.452 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 26 16 74756 74740 179.64 117.375 0.0105981 0.00556527 2024-04-23T16:37:58.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:57 smithi148 ceph-mon[44520]: pgmap v1314: 27 pgs: 27 active+clean; 4.3 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 155 MiB/s wr, 2.49k op/s 2024-04-23T16:37:58.154 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:37:58.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:58.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:37:58.158 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:57 smithi103 ceph-mon[39629]: pgmap v1314: 27 pgs: 27 active+clean; 4.3 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 155 MiB/s wr, 2.49k op/s 2024-04-23T16:37:58.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:57 smithi188 ceph-mon[44101]: pgmap v1314: 27 pgs: 27 active+clean; 4.3 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 155 MiB/s wr, 2.49k op/s 2024-04-23T16:37:58.259 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:37:58.295 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:58.453 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 27 16 76634 76618 177.334 117.375 0.00319163 0.00563757 2024-04-23T16:37:58.511 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:37:59.204 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:37:59.212 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:37:59.248 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:37:59.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:59.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:37:59.353 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:37:59.406 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:59.453 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 28 16 78358 78342 174.849 107.75 0.00527165 0.00571736 2024-04-23T16:37:59.469 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:37:59.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:59.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:37:59.577 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:37:59.641 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:59.665 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:37:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:59.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:37:59.676 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:37:59.676 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:37:59.676 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-23T16:37:59.676 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-23T16:37:59.676 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:37:59.772 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:37:59.810 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:37:59.924 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:37:59.945 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:37:59.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:37:59.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:38:00.051 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:00.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:59 smithi103 ceph-mon[39629]: 1.5 scrub starts 2024-04-23T16:38:00.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:59 smithi103 ceph-mon[39629]: 1.5 scrub ok 2024-04-23T16:38:00.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:59 smithi103 ceph-mon[39629]: pgmap v1315: 27 pgs: 27 active+clean; 4.3 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 136 MiB/s wr, 2.17k op/s 2024-04-23T16:38:00.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:59 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:38:00.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:37:59 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:38:00.105 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:00.105 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:59 smithi148 ceph-mon[44520]: 1.5 scrub starts 2024-04-23T16:38:00.105 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:59 smithi148 ceph-mon[44520]: 1.5 scrub ok 2024-04-23T16:38:00.105 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:59 smithi148 ceph-mon[44520]: pgmap v1315: 27 pgs: 27 active+clean; 4.3 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 136 MiB/s wr, 2.17k op/s 2024-04-23T16:38:00.105 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:59 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:38:00.105 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:37:59 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:38:00.153 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:00.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:00.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:59 smithi188 ceph-mon[44101]: 1.5 scrub starts 2024-04-23T16:38:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:59 smithi188 ceph-mon[44101]: 1.5 scrub ok 2024-04-23T16:38:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:59 smithi188 ceph-mon[44101]: pgmap v1315: 27 pgs: 27 active+clean; 4.3 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 136 MiB/s wr, 2.17k op/s 2024-04-23T16:38:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:59 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:38:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:37:59 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:38:00.259 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:38:00.293 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:00.320 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:38:00.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:00.331 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:38:00.433 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:38:00.453 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 29 16 80027 80011 172.416 104.312 0.00305198 0.00579793 2024-04-23T16:38:00.462 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:00.484 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:38:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:00.488 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:38:00.589 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:38:00.648 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:00.674 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:38:00.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:00.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:38:00.782 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:38:00.808 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:00.828 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:38:00.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:00.831 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:38:00.932 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:00.975 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:00 smithi148 ceph-mon[44520]: from='client.26269 v1:172.21.15.103:0/3096258756' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:38:00.976 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:01.008 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:01.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:01.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:01.114 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:01.153 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:01.194 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:01.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:01.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:01.218 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:00 smithi103 ceph-mon[39629]: from='client.26269 v1:172.21.15.103:0/3096258756' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:38:01.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:00 smithi188 ceph-mon[44101]: from='client.26269 v1:172.21.15.103:0/3096258756' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:38:01.263 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-23T16:38:01.303 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:38:01.332 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:01.354 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:38:01.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:01.359 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:38:01.453 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 30 16 81660 81644 170.071 102.062 0.00804321 0.00587732 2024-04-23T16:38:01.460 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:01.510 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:01.533 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:01.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:01.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:01.642 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:01.703 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:01.749 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:01.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:01.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:01.857 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:38:01.865 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:38:01.901 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:01.921 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:38:01.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:01.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:38:02.026 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:38:02.079 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:01 smithi148 ceph-mon[44520]: pgmap v1316: 27 pgs: 27 active+clean; 4.7 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 125 MiB/s wr, 2k op/s 2024-04-23T16:38:02.080 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:02.115 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:38:02.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:02.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:38:02.221 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:38:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:01 smithi188 ceph-mon[44101]: pgmap v1316: 27 pgs: 27 active+clean; 4.7 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 125 MiB/s wr, 2k op/s 2024-04-23T16:38:02.251 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:02.271 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:38:02.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:02.274 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:38:02.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:01 smithi103 ceph-mon[39629]: pgmap v1316: 27 pgs: 27 active+clean; 4.7 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 125 MiB/s wr, 2k op/s 2024-04-23T16:38:02.375 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:38:02.404 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:02.424 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:38:02.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:02.428 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:38:02.453 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 31 15 83338 83323 167.97 104.938 0.0105901 0.00595201 2024-04-23T16:38:02.528 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:02.570 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:02.605 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:02.608 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1315,"stamp":"2024-04-23T16:37:58.762119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4647223840,"num_objects":70904,"num_object_clones":0,"num_object_copies":141808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70902,"num_write_kb":4537728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72431,"ondisk_log_size":72431,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":12831928,"kb_used_data":8920928,"kb_used_omap":14,"kb_used_meta":3910897,"kb_avail":830874440,"statfs":{"total":863955320832,"available":850815426560,"internally_reserved":0,"allocated":9135030272,"data_stored":9131574515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":4004759535},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,2,6],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1423245312,"num_objects":21717,"num_object_clones":0,"num_object_copies":43434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21717,"num_write_kb":1389888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001899"},"pg_stats":[{"pgid":"6.c","version":"463'4429","reported_seq":4454,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:55.785678+0000","last_change":"2024-04-23T16:37:15.833831+0000","last_active":"2024-04-23T16:37:55.785678+0000","last_peered":"2024-04-23T16:37:55.785678+0000","last_clean":"2024-04-23T16:37:55.785678+0000","last_became_active":"2024-04-23T16:37:15.833495+0000","last_became_peered":"2024-04-23T16:37:15.833495+0000","last_unstale":"2024-04-23T16:37:55.785678+0000","last_undegraded":"2024-04-23T16:37:55.785678+0000","last_fullsized":"2024-04-23T16:37:55.785678+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":4429,"log_dups_size":0,"ondisk_log_size":4429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:29.352334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290258944,"num_objects":4429,"num_object_clones":0,"num_object_copies":8858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4429,"num_write_kb":283456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"463'4528","reported_seq":4553,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:55.754542+0000","last_change":"2024-04-23T16:37:15.838829+0000","last_active":"2024-04-23T16:37:55.754542+0000","last_peered":"2024-04-23T16:37:55.754542+0000","last_clean":"2024-04-23T16:37:55.754542+0000","last_became_active":"2024-04-23T16:37:15.838559+0000","last_became_peered":"2024-04-23T16:37:15.838559+0000","last_unstale":"2024-04-23T16:37:55.754542+0000","last_undegraded":"2024-04-23T16:37:55.754542+0000","last_fullsized":"2024-04-23T16:37:55.754542+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":4528,"log_dups_size":0,"ondisk_log_size":4528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:35.797306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296747008,"num_objects":4528,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4528,"num_write_kb":289792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"463'4461","reported_seq":4486,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:55.416321+0000","last_change":"2024-04-23T16:37:15.837846+0000","last_active":"2024-04-23T16:37:55.416321+0000","last_peered":"2024-04-23T16:37:55.416321+0000","last_clean":"2024-04-23T16:37:55.416321+0000","last_became_active":"2024-04-23T16:37:15.837590+0000","last_became_peered":"2024-04-23T16:37:15.837590+0000","last_unstale":"2024-04-23T16:37:55.416321+0000","last_undegraded":"2024-04-23T16:37:55.416321+0000","last_fullsized":"2024-04-23T16:37:55.416321+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":4461,"log_dups_size":0,"ondisk_log_size":4461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:20.622753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292356096,"num_objects":4461,"num_object_clones":0,"num_object_copies":8922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4461,"num_write_kb":285504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"463'4255","reported_seq":4280,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:54.180055+0000","last_change":"2024-04-23T16:37:15.834978+0000","last_active":"2024-04-23T16:37:54.180055+0000","last_peered":"2024-04-23T16:37:54.180055+0000","last_clean":"2024-04-23T16:37:54.180055+0000","last_became_active":"2024-04-23T16:37:15.834740+0000","last_became_peered":"2024-04-23T16:37:15.834740+0000","last_unstale":"2024-04-23T16:37:54.180055+0000","last_undegraded":"2024-04-23T16:37:54.180055+0000","last_fullsized":"2024-04-23T16:37:54.180055+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":4255,"log_dups_size":0,"ondisk_log_size":4255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:40.729580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278855680,"num_objects":4255,"num_object_clones":0,"num_object_copies":8510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4255,"num_write_kb":272320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"463'4613","reported_seq":4638,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:56.796718+0000","last_change":"2024-04-23T16:37:15.839856+0000","last_active":"2024-04-23T16:37:56.796718+0000","last_peered":"2024-04-23T16:37:56.796718+0000","last_clean":"2024-04-23T16:37:56.796718+0000","last_became_active":"2024-04-23T16:37:15.839667+0000","last_became_peered":"2024-04-23T16:37:15.839667+0000","last_unstale":"2024-04-23T16:37:56.796718+0000","last_undegraded":"2024-04-23T16:37:56.796718+0000","last_fullsized":"2024-04-23T16:37:56.796718+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":4613,"log_dups_size":0,"ondisk_log_size":4613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:33.221933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302317568,"num_objects":4613,"num_object_clones":0,"num_object_copies":9226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4613,"num_write_kb":295232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.d","version":"463'4387","reported_seq":4412,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:55.409300+0000","last_change":"2024-04-23T16:37:15.836920+0000","last_active":"2024-04-23T16:37:55.409300+0000","last_peered":"2024-04-23T16:37:55.409300+0000","last_clean":"2024-04-23T16:37:55.409300+0000","last_became_active":"2024-04-23T16:37:15.836635+0000","last_became_peered":"2024-04-23T16:37:15.836635+0000","last_unstale":"2024-04-23T16:37:55.409300+0000","last_undegraded":"2024-04-23T16:37:55.409300+0000","last_fullsized":"2024-04-23T16:37:55.409300+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":4387,"log_dups_size":0,"ondisk_log_size":4387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:28.500044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287506432,"num_objects":4387,"num_object_clones":0,"num_object_copies":8774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4387,"num_write_kb":280768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1196,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:53.826499+0000","last_change":"2024-04-23T16:36:19.132015+0000","last_active":"2024-04-23T16:37:53.826499+0000","last_peered":"2024-04-23T16:37:53.826499+0000","last_clean":"2024-04-23T16:37:53.826499+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:37:53.826499+0000","last_undegraded":"2024-04-23T16:37:53.826499+0000","last_fullsized":"2024-04-23T16:37:53.826499+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:19.131883+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:19.131883+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:57.202307+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.f","version":"463'4247","reported_seq":4272,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:54.333533+0000","last_change":"2024-04-23T16:37:15.836051+0000","last_active":"2024-04-23T16:37:54.333533+0000","last_peered":"2024-04-23T16:37:54.333533+0000","last_clean":"2024-04-23T16:37:54.333533+0000","last_became_active":"2024-04-23T16:37:15.835794+0000","last_became_peered":"2024-04-23T16:37:15.835794+0000","last_unstale":"2024-04-23T16:37:54.333533+0000","last_undegraded":"2024-04-23T16:37:54.333533+0000","last_fullsized":"2024-04-23T16:37:54.333533+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":4247,"log_dups_size":0,"ondisk_log_size":4247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:33.244802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278331392,"num_objects":4247,"num_object_clones":0,"num_object_copies":8494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4247,"num_write_kb":271808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1191,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:53.835764+0000","last_change":"2024-04-23T16:37:05.612328+0000","last_active":"2024-04-23T16:37:53.835764+0000","last_peered":"2024-04-23T16:37:53.835764+0000","last_clean":"2024-04-23T16:37:53.835764+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:37:53.835764+0000","last_undegraded":"2024-04-23T16:37:53.835764+0000","last_fullsized":"2024-04-23T16:37:53.835764+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:05.612182+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:37:05.612182+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:24.096518+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"463'4353","reported_seq":4378,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:54.668139+0000","last_change":"2024-04-23T16:37:15.837702+0000","last_active":"2024-04-23T16:37:54.668139+0000","last_peered":"2024-04-23T16:37:54.668139+0000","last_clean":"2024-04-23T16:37:54.668139+0000","last_became_active":"2024-04-23T16:37:15.837406+0000","last_became_peered":"2024-04-23T16:37:15.837406+0000","last_unstale":"2024-04-23T16:37:54.668139+0000","last_undegraded":"2024-04-23T16:37:54.668139+0000","last_fullsized":"2024-04-23T16:37:54.668139+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":4353,"log_dups_size":0,"ondisk_log_size":4353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:34.353376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285278208,"num_objects":4353,"num_object_clones":0,"num_object_copies":8706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4353,"num_write_kb":278592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1182,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:53.830677+0000","last_change":"2024-04-23T16:37:05.696710+0000","last_active":"2024-04-23T16:37:53.830677+0000","last_peered":"2024-04-23T16:37:53.830677+0000","last_clean":"2024-04-23T16:37:53.830677+0000","last_became_active":"2024-04-23T16:28:22.061031+0000","last_became_peered":"2024-04-23T16:28:22.061031+0000","last_unstale":"2024-04-23T16:37:53.830677+0000","last_undegraded":"2024-04-23T16:37:53.830677+0000","last_fullsized":"2024-04-23T16:37:53.830677+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:05.696583+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:37:05.696583+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:25.899474+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"463'4464","reported_seq":4486,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:55.279034+0000","last_change":"2024-04-23T16:37:15.839946+0000","last_active":"2024-04-23T16:37:55.279034+0000","last_peered":"2024-04-23T16:37:55.279034+0000","last_clean":"2024-04-23T16:37:55.279034+0000","last_became_active":"2024-04-23T16:37:15.839669+0000","last_became_peered":"2024-04-23T16:37:15.839669+0000","last_unstale":"2024-04-23T16:37:55.279034+0000","last_undegraded":"2024-04-23T16:37:55.279034+0000","last_fullsized":"2024-04-23T16:37:55.279034+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":4464,"log_dups_size":0,"ondisk_log_size":4464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:22.953490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292552704,"num_objects":4464,"num_object_clones":0,"num_object_copies":8928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4464,"num_write_kb":285696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1196,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:53.823633+0000","last_change":"2024-04-23T16:36:26.456131+0000","last_active":"2024-04-23T16:37:53.823633+0000","last_peered":"2024-04-23T16:37:53.823633+0000","last_clean":"2024-04-23T16:37:53.823633+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:37:53.823633+0000","last_undegraded":"2024-04-23T16:37:53.823633+0000","last_fullsized":"2024-04-23T16:37:53.823633+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:26.456003+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:26.456003+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:58.365535+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"463'4439","reported_seq":4463,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:55.422502+0000","last_change":"2024-04-23T16:37:15.837187+0000","last_active":"2024-04-23T16:37:55.422502+0000","last_peered":"2024-04-23T16:37:55.422502+0000","last_clean":"2024-04-23T16:37:55.422502+0000","last_became_active":"2024-04-23T16:37:15.836264+0000","last_became_peered":"2024-04-23T16:37:15.836264+0000","last_unstale":"2024-04-23T16:37:55.422502+0000","last_undegraded":"2024-04-23T16:37:55.422502+0000","last_fullsized":"2024-04-23T16:37:55.422502+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":4439,"log_dups_size":0,"ondisk_log_size":4439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:40.409891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290914304,"num_objects":4439,"num_object_clones":0,"num_object_copies":8878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4439,"num_write_kb":284096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1197,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:53.829023+0000","last_change":"2024-04-23T16:37:05.363014+0000","last_active":"2024-04-23T16:37:53.829023+0000","last_peered":"2024-04-23T16:37:53.829023+0000","last_clean":"2024-04-23T16:37:53.829023+0000","last_became_active":"2024-04-23T16:28:25.051059+0000","last_became_peered":"2024-04-23T16:28:25.051059+0000","last_unstale":"2024-04-23T16:37:53.829023+0000","last_undegraded":"2024-04-23T16:37:53.829023+0000","last_fullsized":"2024-04-23T16:37:53.829023+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:05.362908+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:37:05.362908+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:14.719978+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"463'4536","reported_seq":4557,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:55.783318+0000","last_change":"2024-04-23T16:37:15.835369+0000","last_active":"2024-04-23T16:37:55.783318+0000","last_peered":"2024-04-23T16:37:55.783318+0000","last_clean":"2024-04-23T16:37:55.783318+0000","last_became_active":"2024-04-23T16:37:15.835131+0000","last_became_peered":"2024-04-23T16:37:15.835131+0000","last_unstale":"2024-04-23T16:37:55.783318+0000","last_undegraded":"2024-04-23T16:37:55.783318+0000","last_fullsized":"2024-04-23T16:37:55.783318+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":4536,"log_dups_size":0,"ondisk_log_size":4536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:30.571812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297271296,"num_objects":4536,"num_object_clones":0,"num_object_copies":9072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4536,"num_write_kb":290304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1195,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:53.826830+0000","last_change":"2024-04-23T16:36:23.239567+0000","last_active":"2024-04-23T16:37:53.826830+0000","last_peered":"2024-04-23T16:37:53.826830+0000","last_clean":"2024-04-23T16:37:53.826830+0000","last_became_active":"2024-04-23T16:28:25.049464+0000","last_became_peered":"2024-04-23T16:28:25.049464+0000","last_unstale":"2024-04-23T16:37:53.826830+0000","last_undegraded":"2024-04-23T16:37:53.826830+0000","last_fullsized":"2024-04-23T16:37:53.826830+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:23.239466+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:36:23.239466+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:57.202287+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.1","version":"463'4707","reported_seq":4732,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:58.255575+0000","last_change":"2024-04-23T16:37:15.834693+0000","last_active":"2024-04-23T16:37:58.255575+0000","last_peered":"2024-04-23T16:37:58.255575+0000","last_clean":"2024-04-23T16:37:58.255575+0000","last_became_active":"2024-04-23T16:37:15.834476+0000","last_became_peered":"2024-04-23T16:37:15.834476+0000","last_unstale":"2024-04-23T16:37:58.255575+0000","last_undegraded":"2024-04-23T16:37:58.255575+0000","last_fullsized":"2024-04-23T16:37:58.255575+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":4707,"log_dups_size":0,"ondisk_log_size":4707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:18.272428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308477952,"num_objects":4707,"num_object_clones":0,"num_object_copies":9414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4707,"num_write_kb":301248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1196,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:53.826522+0000","last_change":"2024-04-23T16:36:18.085504+0000","last_active":"2024-04-23T16:37:53.826522+0000","last_peered":"2024-04-23T16:37:53.826522+0000","last_clean":"2024-04-23T16:37:53.826522+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:37:53.826522+0000","last_undegraded":"2024-04-23T16:37:53.826522+0000","last_fullsized":"2024-04-23T16:37:53.826522+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:18.085386+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:18.085386+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:57.202227+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.0","version":"463'4260","reported_seq":4285,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:54.210017+0000","last_change":"2024-04-23T16:37:15.836339+0000","last_active":"2024-04-23T16:37:54.210017+0000","last_peered":"2024-04-23T16:37:54.210017+0000","last_clean":"2024-04-23T16:37:54.210017+0000","last_became_active":"2024-04-23T16:37:15.836027+0000","last_became_peered":"2024-04-23T16:37:15.836027+0000","last_unstale":"2024-04-23T16:37:54.210017+0000","last_undegraded":"2024-04-23T16:37:54.210017+0000","last_fullsized":"2024-04-23T16:37:54.210017+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:28.804087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279183360,"num_objects":4260,"num_object_clones":0,"num_object_copies":8520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4260,"num_write_kb":272640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1197,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:53.829072+0000","last_change":"2024-04-23T16:37:06.338083+0000","last_active":"2024-04-23T16:37:53.829072+0000","last_peered":"2024-04-23T16:37:53.829072+0000","last_clean":"2024-04-23T16:37:53.829072+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:37:53.829072+0000","last_undegraded":"2024-04-23T16:37:53.829072+0000","last_fullsized":"2024-04-23T16:37:53.829072+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:06.337940+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:37:06.337940+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:22.707182+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"463'4437","reported_seq":4462,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:55.272060+0000","last_change":"2024-04-23T16:37:15.839998+0000","last_active":"2024-04-23T16:37:55.272060+0000","last_peered":"2024-04-23T16:37:55.272060+0000","last_clean":"2024-04-23T16:37:55.272060+0000","last_became_active":"2024-04-23T16:37:15.839714+0000","last_became_peered":"2024-04-23T16:37:15.839714+0000","last_unstale":"2024-04-23T16:37:55.272060+0000","last_undegraded":"2024-04-23T16:37:55.272060+0000","last_fullsized":"2024-04-23T16:37:55.272060+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":4437,"log_dups_size":0,"ondisk_log_size":4437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:16.353176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290783232,"num_objects":4437,"num_object_clones":0,"num_object_copies":8874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4437,"num_write_kb":283968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1088,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:53.829607+0000","last_change":"2024-04-23T16:36:29.341359+0000","last_active":"2024-04-23T16:37:53.829607+0000","last_peered":"2024-04-23T16:37:53.829607+0000","last_clean":"2024-04-23T16:37:53.829607+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:37:53.829607+0000","last_undegraded":"2024-04-23T16:37:53.829607+0000","last_fullsized":"2024-04-23T16:37:53.829607+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:29.341205+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:36:29.341205+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:58.768397+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"463'4278","reported_seq":4303,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:54.709530+0000","last_change":"2024-04-23T16:37:15.836315+0000","last_active":"2024-04-23T16:37:54.709530+0000","last_peered":"2024-04-23T16:37:54.709530+0000","last_clean":"2024-04-23T16:37:54.709530+0000","last_became_active":"2024-04-23T16:37:15.836050+0000","last_became_peered":"2024-04-23T16:37:15.836050+0000","last_unstale":"2024-04-23T16:37:54.709530+0000","last_undegraded":"2024-04-23T16:37:54.709530+0000","last_fullsized":"2024-04-23T16:37:54.709530+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":4278,"log_dups_size":0,"ondisk_log_size":4278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:39.323249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280363008,"num_objects":4278,"num_object_clones":0,"num_object_copies":8556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4278,"num_write_kb":273792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1010,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:53.829074+0000","last_change":"2024-04-23T16:36:19.091622+0000","last_active":"2024-04-23T16:37:53.829074+0000","last_peered":"2024-04-23T16:37:53.829074+0000","last_clean":"2024-04-23T16:37:53.829074+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:37:53.829074+0000","last_undegraded":"2024-04-23T16:37:53.829074+0000","last_fullsized":"2024-04-23T16:37:53.829074+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:36:19.091490+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:51.684868+0000","last_clean_scrub_stamp":"2024-04-23T16:36:19.091490+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:55.491455+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"463'4508","reported_seq":4533,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:55.421712+0000","last_change":"2024-04-23T16:37:15.836721+0000","last_active":"2024-04-23T16:37:55.421712+0000","last_peered":"2024-04-23T16:37:55.421712+0000","last_clean":"2024-04-23T16:37:55.421712+0000","last_became_active":"2024-04-23T16:37:15.836376+0000","last_became_peered":"2024-04-23T16:37:15.836376+0000","last_unstale":"2024-04-23T16:37:55.421712+0000","last_undegraded":"2024-04-23T16:37:55.421712+0000","last_fullsized":"2024-04-23T16:37:55.421712+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:37:14.820617+0000","last_clean_scrub_stamp":"2024-04-23T16:37:14.820617+0000","objects_scrubbed":0,"log_size":4508,"log_dups_size":0,"ondisk_log_size":4508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:15.626663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295436288,"num_objects":4508,"num_object_clones":0,"num_object_copies":9016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4508,"num_write_kb":288512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1186,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-04-23T16:37:53.831579+0000","last_change":"2024-04-23T16:37:12.716078+0000","last_active":"2024-04-23T16:37:53.831579+0000","last_peered":"2024-04-23T16:37:53.831579+0000","last_clean":"2024-04-23T16:37:53.831579+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:37:53.831579+0000","last_undegraded":"2024-04-23T16:37:53.831579+0000","last_fullsized":"2024-04-23T16:37:53.831579+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:37:12.715954+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:37:12.715954+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:38:27.717888+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":4646633472,"num_objects":70902,"num_object_clones":0,"num_object_copies":141804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70902,"num_write_kb":4537728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9219211264,"data_stored":9219211264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70902,"ondisk_log_size":70902,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":800564,"kb_used_data":295976,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":92944588,"statfs":{"total":95995035648,"available":95175258112,"internally_reserved":0,"allocated":303079424,"data_stored":302694895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":516684228},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351253,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612891,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1194356,"kb_used_data":811944,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92550796,"statfs":{"total":95995035648,"available":94772015104,"internally_reserved":0,"allocated":831430656,"data_stored":831046127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":391576000},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":607600,"kb_used_data":301544,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":93137552,"statfs":{"total":95995035648,"available":95372853248,"internally_reserved":0,"allocated":308781056,"data_stored":308396527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168868,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1549104,"kb_used_data":1120872,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":92196048,"statfs":{"total":95995035648,"available":94408753152,"internally_reserved":0,"allocated":1147772928,"data_stored":1147388399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":438499786},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463210,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1397668,"kb_used_data":1073564,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92347484,"statfs":{"total":95995035648,"available":94563823616,"internally_reserved":0,"allocated":1099329536,"data_stored":1098951061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":331872710},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1163936137351,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738696,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1265252,"kb_used_data":796764,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":92479900,"statfs":{"total":95995035648,"available":94699417600,"internally_reserved":0,"allocated":815886336,"data_stored":815507861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117304,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2453180,"kb_used_data":2002736,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":91291972,"statfs":{"total":95995035648,"available":93482979328,"internally_reserved":0,"allocated":2050801664,"data_stored":2050413583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":461240775},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921074,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1407348,"kb_used_data":820840,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":92337804,"statfs":{"total":95995035648,"available":94553911296,"internally_reserved":0,"allocated":840540160,"data_stored":840155631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757549,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2156856,"kb_used_data":1696688,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":91588296,"statfs":{"total":95995035648,"available":93786415104,"internally_reserved":0,"allocated":1737408512,"data_stored":1737020431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":837353472,"data_stored":837353472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2056257536,"data_stored":2056257536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":854130688,"data_stored":854130688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1742798848,"data_stored":1742798848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1117913088,"data_stored":1117913088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1165230080,"data_stored":1165230080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":308477952,"data_stored":308477952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":834732032,"data_stored":834732032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":302317568,"data_stored":302317568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:38:02.609 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-23T16:38:02.610 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-04-23T16:38:02.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:02.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:02.711 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:38:02.755 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:02.818 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:38:02.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:02.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:38:02.922 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:38:02.977 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:02.998 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:38:03.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:03.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:38:03.102 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:38:03.130 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:03.154 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:38:03.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:03.158 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:38:03.259 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:38:03.314 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:03.453 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 32 16 84945 84929 165.857 100.375 0.0147 0.00602729 2024-04-23T16:38:03.637 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:38:03.782 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:38:03.782 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:38:03.782 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:38:03.782 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:38:03.782 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:38:03.782 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:38:03.782 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:38:03.782 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:38:03.782 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:38:03.782 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:38:03.783 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:38:03.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:38:03.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:38:03.783 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:38:03.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:38:03.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:38:03.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:38:03.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:38:03.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:38:03.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:38:03.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:38:03.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:38:03.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:38:04.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:03 smithi188 ceph-mon[44101]: pgmap v1317: 27 pgs: 27 active+clean; 4.8 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-04-23T16:38:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:03 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1578056406' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:38:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:03 smithi103 ceph-mon[39629]: pgmap v1317: 27 pgs: 27 active+clean; 4.8 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-04-23T16:38:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:03 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1578056406' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:38:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:03 smithi148 ceph-mon[44520]: pgmap v1317: 27 pgs: 27 active+clean; 4.8 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-04-23T16:38:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:03 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1578056406' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:38:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:04.453 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 33 15 86660 86645 164.081 107.25 0.0131583 0.00609256 2024-04-23T16:38:05.080 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:38:05.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:05.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:38:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:38:05.118 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:38:05.185 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:38:05.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:04 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:38:05.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:04 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:38:05.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:04 smithi188 ceph-mon[44101]: 1.4 scrub starts 2024-04-23T16:38:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:04 smithi188 ceph-mon[44101]: osdmap e464: 12 total, 11 up, 9 in 2024-04-23T16:38:05.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:04 smithi188 ceph-mon[44101]: 1.4 scrub ok 2024-04-23T16:38:05.232 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:04 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:38:05.232 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:04 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:38:05.232 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:05.232 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:05.232 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:05.232 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:05.232 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:05.233 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:04 smithi148 ceph-mon[44520]: 1.4 scrub starts 2024-04-23T16:38:05.233 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:04 smithi148 ceph-mon[44520]: osdmap e464: 12 total, 11 up, 9 in 2024-04-23T16:38:05.233 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:04 smithi148 ceph-mon[44520]: 1.4 scrub ok 2024-04-23T16:38:05.238 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:05.271 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:38:05.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:05.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:38:05.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:04 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:38:05.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:04 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:38:05.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:05.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:05.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:05.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:05.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:05.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:04 smithi103 ceph-mon[39629]: 1.4 scrub starts 2024-04-23T16:38:05.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:04 smithi103 ceph-mon[39629]: osdmap e464: 12 total, 11 up, 9 in 2024-04-23T16:38:05.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:04 smithi103 ceph-mon[39629]: 1.4 scrub ok 2024-04-23T16:38:05.380 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:38:05.433 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:05.453 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 34 16 88333 88317 162.329 104.5 0.00460603 0.00615887 2024-04-23T16:38:05.474 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:38:05.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:05.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:38:05.579 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:38:05.609 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:05.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:05.672 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:38:05.672 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:38:05.773 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:05.809 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:05.833 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:05.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:05.837 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:05.937 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:38:05.983 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:06.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:05 smithi188 ceph-mon[44101]: pgmap v1319: 27 pgs: 27 active+clean; 5.0 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-23T16:38:06.230 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:05 smithi103 ceph-mon[39629]: pgmap v1319: 27 pgs: 27 active+clean; 5.0 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-23T16:38:06.237 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:38:06.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:06.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:38:06.341 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:38:06.368 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:06.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:05 smithi148 ceph-mon[44520]: pgmap v1319: 27 pgs: 27 active+clean; 5.0 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-23T16:38:06.395 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:38:06.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:06.401 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:38:06.453 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 35 16 89990 89974 160.649 103.562 0.00562786 0.00622341 2024-04-23T16:38:06.502 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:38:06.547 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:06.592 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:38:06.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:06.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:38:06.699 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:38:06.752 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:06.788 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:38:06.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:06.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:38:06.896 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:38:06.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:06 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:38:06.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:06 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:38:06.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:06 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1809713522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:38:06.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:06 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:38:06.906 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:06 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:38:06.912 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:38:06.935 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:07.119 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:38:07.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:06 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:38:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:06 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:38:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:06 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1809713522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:38:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:06 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:38:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:06 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:38:07.193 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:38:07.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:07.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:38:07.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:06 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:38:07.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:06 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:38:07.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:06 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1809713522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:38:07.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:06 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:38:07.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:06 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:38:07.297 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:07.324 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:07.343 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:07.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:07.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:07.447 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:38:07.453 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 36 16 91374 91358 158.59 86.5 0.00666304 0.00630428 2024-04-23T16:38:07.495 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:08.091 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:07 smithi103 ceph-mon[39629]: pgmap v1320: 27 pgs: 27 active+clean; 5.3 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-04-23T16:38:08.091 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:07 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:38:08.091 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:07 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:38:08.091 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:07 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1809713522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:38:08.091 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:07 smithi103 ceph-mon[39629]: osdmap e465: 12 total, 11 up, 9 in 2024-04-23T16:38:08.153 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:38:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:08.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:38:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:07 smithi188 ceph-mon[44101]: pgmap v1320: 27 pgs: 27 active+clean; 5.3 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-04-23T16:38:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:07 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:38:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:07 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:38:08.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:07 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1809713522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:38:08.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:07 smithi188 ceph-mon[44101]: osdmap e465: 12 total, 11 up, 9 in 2024-04-23T16:38:08.258 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:38:08.287 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:08.308 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:38:08.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:08.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:38:08.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:07 smithi148 ceph-mon[44520]: pgmap v1320: 27 pgs: 27 active+clean; 5.3 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-04-23T16:38:08.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:07 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:38:08.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:07 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:38:08.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:07 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1809713522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:38:08.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:07 smithi148 ceph-mon[44520]: osdmap e465: 12 total, 11 up, 9 in 2024-04-23T16:38:08.412 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:38:08.453 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 37 16 92981 92965 157.018 100.438 0.00623362 0.00636708 2024-04-23T16:38:08.472 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:08.521 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:38:08.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:08.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:38:08.610 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:38:08.616 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:38:08.616 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:38:08.626 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:08.679 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:08.713 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:08.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:08.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:08.826 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:38:08.859 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:08.890 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:38:08.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:08.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:38:08.997 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:38:09.033 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:09.237 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:38:09.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:09.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:38:09.342 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:38:09.385 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:09.429 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:38:09.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:09.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:38:09.453 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 38 16 94474 94458 155.341 93.3125 0.00470764 0.00643637 2024-04-23T16:38:09.537 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:38:09.564 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:09.585 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:38:09.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:09.589 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:38:09.690 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:09.743 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:09.787 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:09.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:09.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:09.862 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-23T16:38:09.875 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:38:09.898 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:38:09.910 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:09 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T16:38:09.910 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:09 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T16:38:09.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:09 smithi188 ceph-mon[44101]: pgmap v1322: 27 pgs: 27 active+clean; 5.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-23T16:38:09.926 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:09.953 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:38:09.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:09.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:38:10.078 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:38:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:09 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T16:38:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:09 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T16:38:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:09 smithi103 ceph-mon[39629]: pgmap v1322: 27 pgs: 27 active+clean; 5.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-23T16:38:10.119 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:09 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T16:38:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:09 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T16:38:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:09 smithi148 ceph-mon[44520]: pgmap v1322: 27 pgs: 27 active+clean; 5.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-23T16:38:10.142 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:38:10.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:10.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:38:10.246 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:38:10.286 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:10.453 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 39 16 96052 96036 153.887 98.625 0.00513704 0.0064969 2024-04-23T16:38:10.750 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:38:10.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:10.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:38:10.788 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:38:10.793 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:38:10.865 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:38:10.901 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:11.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:10 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:38:11.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:10 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:38:11.160 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:38:11.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:11.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:38:11.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:10 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:38:11.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:10 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:38:11.269 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:38:11.310 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:11.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:10 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:38:11.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:10 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:38:11.442 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:38:11.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:11.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:38:11.453 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:38:11.453628 min lat: 0.000989845 max lat: 0.216385 avg lat: 0.00655618 2024-04-23T16:38:11.453 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:38:11.453 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 40 16 97621 97605 152.491 98.0625 0.00602561 0.00655618 2024-04-23T16:38:11.547 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:38:11.613 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:11.654 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:38:11.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:11.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:38:11.762 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:38:11.818 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:11.851 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:38:11.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:11.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:38:11.958 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:11.991 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:12.012 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:12.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:12.015 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:12.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:11 smithi103 ceph-mon[39629]: 1.a deep-scrub starts 2024-04-23T16:38:12.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:11 smithi103 ceph-mon[39629]: 1.a deep-scrub ok 2024-04-23T16:38:12.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:11 smithi103 ceph-mon[39629]: pgmap v1323: 27 pgs: 27 active+clean; 5.7 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-04-23T16:38:12.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:11 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3929673673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:38:12.116 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:11 smithi148 ceph-mon[44520]: 1.a deep-scrub starts 2024-04-23T16:38:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:11 smithi148 ceph-mon[44520]: 1.a deep-scrub ok 2024-04-23T16:38:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:11 smithi148 ceph-mon[44520]: pgmap v1323: 27 pgs: 27 active+clean; 5.7 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-04-23T16:38:12.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:11 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3929673673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:38:12.146 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:12.168 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:12.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:12.171 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:12.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:11 smithi188 ceph-mon[44101]: 1.a deep-scrub starts 2024-04-23T16:38:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:11 smithi188 ceph-mon[44101]: 1.a deep-scrub ok 2024-04-23T16:38:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:11 smithi188 ceph-mon[44101]: pgmap v1323: 27 pgs: 27 active+clean; 5.7 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-04-23T16:38:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:11 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3929673673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:38:12.272 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:38:12.300 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:12.320 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:38:12.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:12.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:38:12.424 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:12.450 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:12.454 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 41 16 99120 99104 151.057 93.6875 0.00965957 0.00661811 2024-04-23T16:38:12.470 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:12.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:12.473 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:12.476 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:38:12.476 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:38:12.477 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:38:12.477 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:38:12.477 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:38:12.477 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:38:12.477 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:38:12.477 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:38:12.477 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:38:12.477 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:38:12.477 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:38:12.477 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:38:12.477 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:38:12.478 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:38:12.478 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:38:12.478 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:38:12.478 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:38:12.478 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:38:12.478 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:38:12.478 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:38:12.478 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:38:12.478 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:38:12.478 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:38:12.574 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:12.635 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:12.670 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:12.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:12.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:12.778 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:38:12.807 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:12.830 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:38:12.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:12.835 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:38:12.936 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:38:12.976 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:13.008 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:38:13.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:13.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:38:13.112 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:38:13.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.141 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:13.163 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:38:13.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:13.166 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:38:13.267 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:38:13.296 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.296 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.296 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.297 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.297 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.297 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.297 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.297 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.297 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.297 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.299 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:13.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:13.453 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 42 16 100683 100667 149.786 97.6875 0.00864379 0.00667443 2024-04-23T16:38:14.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:13 smithi188 ceph-mon[44101]: pgmap v1324: 27 pgs: 27 active+clean; 5.8 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 110 MiB/s wr, 1.75k op/s 2024-04-23T16:38:14.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:13 smithi103 ceph-mon[39629]: pgmap v1324: 27 pgs: 27 active+clean; 5.8 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 110 MiB/s wr, 1.75k op/s 2024-04-23T16:38:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:14.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:13 smithi148 ceph-mon[44520]: pgmap v1324: 27 pgs: 27 active+clean; 5.8 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 110 MiB/s wr, 1.75k op/s 2024-04-23T16:38:14.454 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 43 16 102161 102145 148.451 92.375 0.0170444 0.00673435 2024-04-23T16:38:15.454 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 44 16 103566 103550 147.072 87.8125 0.0048697 0.0067977 2024-04-23T16:38:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:15 smithi188 ceph-mon[44101]: pgmap v1325: 27 pgs: 27 active+clean; 6.0 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-04-23T16:38:16.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:15 smithi103 ceph-mon[39629]: pgmap v1325: 27 pgs: 27 active+clean; 6.0 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-04-23T16:38:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:15 smithi148 ceph-mon[44520]: pgmap v1325: 27 pgs: 27 active+clean; 6.0 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-04-23T16:38:16.465 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 45 16 104966 104950 145.748 87.5 0.00376953 0.0068573 2024-04-23T16:38:17.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:16 smithi188 ceph-mon[44101]: 1.3 deep-scrub starts 2024-04-23T16:38:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:16 smithi188 ceph-mon[44101]: 6.7 deep-scrub starts 2024-04-23T16:38:17.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:16 smithi188 ceph-mon[44101]: 1.3 deep-scrub ok 2024-04-23T16:38:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:16 smithi103 ceph-mon[39629]: 1.3 deep-scrub starts 2024-04-23T16:38:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:17 smithi103 ceph-mon[39629]: 6.7 deep-scrub starts 2024-04-23T16:38:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:17 smithi103 ceph-mon[39629]: 1.3 deep-scrub ok 2024-04-23T16:38:17.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:16 smithi148 ceph-mon[44520]: 1.3 deep-scrub starts 2024-04-23T16:38:17.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:16 smithi148 ceph-mon[44520]: 6.7 deep-scrub starts 2024-04-23T16:38:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:16 smithi148 ceph-mon[44520]: 1.3 deep-scrub ok 2024-04-23T16:38:17.465 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 46 16 106114 106098 144.104 71.75 0.0283654 0.00693645 2024-04-23T16:38:18.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:17 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:38:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:17 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:38:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:17 smithi188 ceph-mon[44101]: pgmap v1326: 27 pgs: 27 active+clean; 6.3 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-04-23T16:38:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:17 smithi188 ceph-mon[44101]: osdmap e466: 12 total, 11 up, 9 in 2024-04-23T16:38:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:17 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:38:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:17 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:38:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:17 smithi103 ceph-mon[39629]: pgmap v1326: 27 pgs: 27 active+clean; 6.3 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-04-23T16:38:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:17 smithi103 ceph-mon[39629]: osdmap e466: 12 total, 11 up, 9 in 2024-04-23T16:38:18.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:17 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:38:18.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:17 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:38:18.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:17 smithi148 ceph-mon[44520]: pgmap v1326: 27 pgs: 27 active+clean; 6.3 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-04-23T16:38:18.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:17 smithi148 ceph-mon[44520]: osdmap e466: 12 total, 11 up, 9 in 2024-04-23T16:38:18.465 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 47 16 107372 107356 142.711 78.625 0.0132641 0.00700578 2024-04-23T16:38:19.466 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 48 16 108614 108598 141.355 77.625 0.00858839 0.00707109 2024-04-23T16:38:20.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:19 smithi188 ceph-mon[44101]: pgmap v1328: 27 pgs: 27 active+clean; 6.3 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-23T16:38:20.324 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:19 smithi103 ceph-mon[39629]: pgmap v1328: 27 pgs: 27 active+clean; 6.3 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-23T16:38:20.354 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:38:20.354 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:38:20.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:19 smithi148 ceph-mon[44520]: pgmap v1328: 27 pgs: 27 active+clean; 6.3 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-23T16:38:20.466 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 49 16 109871 109855 140.074 78.5625 0.0602413 0.00713611 2024-04-23T16:38:20.660 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:38:20.660 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:38:20.660 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-23T16:38:20.660 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:38:20.662 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:38:20.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:20.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:38:20.766 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:38:20.816 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:20.982 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:20 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:38:20.982 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:20 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:38:21.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:20 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:38:21.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:20 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:38:21.466 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 50 16 111089 111073 138.795 76.125 0.010749 0.00720367 2024-04-23T16:38:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:20 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:38:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:20 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:38:21.723 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-23T16:38:21.775 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:38:21.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:21.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:38:21.879 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:38:21.985 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:21 smithi103 ceph-mon[39629]: pgmap v1329: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-23T16:38:21.985 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:21 smithi103 ceph-mon[39629]: 6.1 scrub starts 2024-04-23T16:38:21.985 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:21 smithi103 ceph-mon[39629]: osdmap e467: 12 total, 11 up, 9 in 2024-04-23T16:38:21.985 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:21 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/615576795' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:38:21.985 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:21 smithi103 ceph-mon[39629]: from='client.26290 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:38:22.001 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:38:22.061 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:38:22.217 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:22.238 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:38:22.240 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:21 smithi188 ceph-mon[44101]: pgmap v1329: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-23T16:38:22.240 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:21 smithi188 ceph-mon[44101]: 6.1 scrub starts 2024-04-23T16:38:22.240 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:21 smithi188 ceph-mon[44101]: osdmap e467: 12 total, 11 up, 9 in 2024-04-23T16:38:22.240 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:21 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/615576795' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:38:22.240 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:21 smithi188 ceph-mon[44101]: from='client.26290 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:38:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:22.241 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:38:22.342 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:38:22.356 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:38:22.372 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:22.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:21 smithi148 ceph-mon[44520]: pgmap v1329: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-23T16:38:22.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:21 smithi148 ceph-mon[44520]: 6.1 scrub starts 2024-04-23T16:38:22.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:21 smithi148 ceph-mon[44520]: osdmap e467: 12 total, 11 up, 9 in 2024-04-23T16:38:22.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:21 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/615576795' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:38:22.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:21 smithi148 ceph-mon[44520]: from='client.26290 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:38:22.466 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 51 16 112215 112199 137.454 70.375 0.0093886 0.00727346 2024-04-23T16:38:22.937 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:38:22.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:22.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:38:23.045 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:23.072 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:23.093 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:23.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:23.096 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:23.197 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:23.198 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:22 smithi103 ceph-mon[39629]: 6.1 scrub ok 2024-04-23T16:38:23.198 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:22 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:38:23.199 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:22 smithi103 ceph-mon[39629]: from='client.26290 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:38:23.199 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:22 smithi103 ceph-mon[39629]: osdmap e468: 12 total, 11 up, 9 in 2024-04-23T16:38:23.199 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:22 smithi103 ceph-mon[39629]: pgmap v1332: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-23T16:38:23.225 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:23.247 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:23.249 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:22 smithi188 ceph-mon[44101]: 6.1 scrub ok 2024-04-23T16:38:23.249 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:22 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:38:23.249 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:22 smithi188 ceph-mon[44101]: from='client.26290 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:38:23.249 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:22 smithi188 ceph-mon[44101]: osdmap e468: 12 total, 11 up, 9 in 2024-04-23T16:38:23.249 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:22 smithi188 ceph-mon[44101]: pgmap v1332: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-23T16:38:23.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:23.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:23.351 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:23.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:22 smithi148 ceph-mon[44520]: 6.1 scrub ok 2024-04-23T16:38:23.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:22 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:38:23.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:22 smithi148 ceph-mon[44520]: from='client.26290 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:38:23.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:22 smithi148 ceph-mon[44520]: osdmap e468: 12 total, 11 up, 9 in 2024-04-23T16:38:23.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:22 smithi148 ceph-mon[44520]: pgmap v1332: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-23T16:38:23.405 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:23.440 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:23.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:23.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:23.466 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 52 16 113394 113378 136.228 73.6875 0.0355158 0.00733891 2024-04-23T16:38:23.545 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:23.597 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:23.638 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:23.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:23.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:23.749 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:38:23.777 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:23.802 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:38:23.804 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:38:23.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:23.805 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:38:23.906 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:38:23.932 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:23.952 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:38:23.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:23.956 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:38:24.057 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:24.114 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:24.147 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:24.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:24.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:24.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:23 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:38:24.256 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:38:24.285 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:23 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:38:24.288 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:24.373 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:38:24.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:23 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:38:24.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:24.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:38:24.466 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 53 16 114417 114401 134.864 63.9375 0.0689807 0.00741128 2024-04-23T16:38:24.481 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:38:24.548 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:24.593 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:38:24.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:24.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:38:24.700 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:24.726 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:24.746 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:24.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:24.850 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:38:24.904 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:24.930 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:38:24.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:24.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:38:24.960 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:38:25.038 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:38:25.066 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:25.086 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:38:25.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:25.089 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:38:25.190 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:38:25.219 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:25.239 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:38:25.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:25.242 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:38:25.343 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:38:25.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:25 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:38:25.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:25 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:38:25.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:38:25.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:38:25.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:25 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:38:25.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:25 smithi103 ceph-mon[39629]: pgmap v1333: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-23T16:38:25.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:25 smithi103 ceph-mon[39629]: osdmap e469: 12 total, 11 up, 9 in 2024-04-23T16:38:25.388 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:25.466 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 54 16 115523 115507 133.647 69.125 0.00822734 0.00747746 2024-04-23T16:38:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:25 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:38:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:25 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:38:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:38:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:38:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:25 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:38:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:25 smithi188 ceph-mon[44101]: pgmap v1333: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-23T16:38:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:25 smithi188 ceph-mon[44101]: osdmap e469: 12 total, 11 up, 9 in 2024-04-23T16:38:25.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:25 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:38:25.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:25 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:38:25.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:38:25.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:38:25.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:25 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:38:25.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:25 smithi148 ceph-mon[44520]: pgmap v1333: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-23T16:38:25.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:25 smithi148 ceph-mon[44520]: osdmap e469: 12 total, 11 up, 9 in 2024-04-23T16:38:26.059 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:38:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:26.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:38:26.163 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:38:26.206 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:26.233 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:38:26.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:26.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:38:26.338 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:38:26.375 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:26.466 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 55 16 116660 116644 132.509 71.0625 0.0748814 0.0075412 2024-04-23T16:38:26.546 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:38:26.550 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:38:26.550 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:38:26.550 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-23T16:38:26.550 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-23T16:38:26.550 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:38:26.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:26.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:38:26.652 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:38:26.702 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:26.732 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:38:26.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:26.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:38:26.836 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:26.881 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:26.918 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:26.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:26.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:26.999 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:38:26.999 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:38:27.000 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:38:27.000 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:38:27.000 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:38:27.000 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:38:27.009 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:38:27.013 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:38:27.026 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:27.045 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:38:27.059 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:26 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3547592562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:38:27.069 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:38:27.077 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:27.096 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:38:27.113 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:27.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:27.120 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:38:27.121 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:38:27.121 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:38:27.121 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:38:27.121 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:38:27.121 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:38:27.121 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:38:27.121 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:38:27.121 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:38:27.121 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:38:27.121 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:38:27.121 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:38:27.121 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:38:27.134 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:26 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3547592562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:38:27.217 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:27.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:26 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3547592562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:38:27.249 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:27.271 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:27.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:27.274 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:27.375 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:38:27.410 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:27.466 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 56 16 117760 117744 131.371 68.75 0.00463127 0.00761067 2024-04-23T16:38:27.664 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:38:27.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:27.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:38:27.769 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:27.809 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:27.835 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:27.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:27.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:27.940 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:38:27.988 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:28.024 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:38:28.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:28.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:38:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:27 smithi103 ceph-mon[39629]: pgmap v1335: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-23T16:38:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:27 smithi148 ceph-mon[44520]: pgmap v1335: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-23T16:38:28.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.130 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:28.166 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:28.196 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:28.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:28.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:27 smithi188 ceph-mon[44101]: pgmap v1335: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-23T16:38:28.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.233 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:38:28.233 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:38:28.302 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:28.352 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:28.378 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:28.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:28.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:28.466 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 57 16 118831 118815 130.24 66.9375 0.00463998 0.00767522 2024-04-23T16:38:28.483 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:28.539 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:28.570 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:28.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:28.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:28.674 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:38:28.702 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:28.721 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:38:28.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:28.725 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:38:28.826 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:38:28.893 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:28.922 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:38:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:28.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:38:28.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:28 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:38:28.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:28 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:38:29.028 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:38:29.060 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:29.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:29.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:29.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:28 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:38:29.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:28 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:38:29.195 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:38:29.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:29.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:38:29.232 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-23T16:38:29.299 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:29.325 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:29.344 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:29.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:29.347 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:28 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:38:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:28 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:38:29.448 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:29.466 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 58 16 119914 119898 129.162 67.6875 0.00895823 0.00774013 2024-04-23T16:38:29.475 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:29.496 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:29.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:29.499 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:29.600 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:38:29.654 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:29.682 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:38:29.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:29.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:38:29.787 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:38:29.814 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:29.836 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:38:29.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:29.838 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:38:29.939 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:29.975 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:29 smithi148 ceph-mon[44520]: pgmap v1336: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.1 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 52 MiB/s wr, 827 op/s 2024-04-23T16:38:29.976 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:29 smithi148 ceph-mon[44520]: osdmap e470: 12 total, 11 up, 9 in 2024-04-23T16:38:29.976 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:29 smithi148 ceph-mon[44520]: osdmap e471: 12 total, 11 up, 9 in 2024-04-23T16:38:29.978 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:30.013 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:30.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:30.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:30.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:29 smithi103 ceph-mon[39629]: pgmap v1336: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.1 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 52 MiB/s wr, 827 op/s 2024-04-23T16:38:30.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:29 smithi103 ceph-mon[39629]: osdmap e470: 12 total, 11 up, 9 in 2024-04-23T16:38:30.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:29 smithi103 ceph-mon[39629]: osdmap e471: 12 total, 11 up, 9 in 2024-04-23T16:38:30.117 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:38:30.156 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:30.215 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:38:30.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:30.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:38:30.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:29 smithi188 ceph-mon[44101]: pgmap v1336: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.1 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 52 MiB/s wr, 827 op/s 2024-04-23T16:38:30.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:29 smithi188 ceph-mon[44101]: osdmap e470: 12 total, 11 up, 9 in 2024-04-23T16:38:30.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:29 smithi188 ceph-mon[44101]: osdmap e471: 12 total, 11 up, 9 in 2024-04-23T16:38:30.235 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:38:30.322 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:30.347 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:38:30.362 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:30.399 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:30.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:30.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:30.466 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 59 16 120900 120884 128.017 61.625 0.00390883 0.00780444 2024-04-23T16:38:30.505 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:30.555 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:30.593 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:30.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:30.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:30.700 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:30.744 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:30.777 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:30.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:30.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:30.882 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:38:30.912 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:30 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:38:30.912 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:30 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:38:30.912 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:30 smithi103 ceph-mon[39629]: osdmap e472: 12 total, 11 up, 9 in 2024-04-23T16:38:30.912 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:30 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3619746967' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:38:30.913 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:30 smithi103 ceph-mon[39629]: from='client.26302 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:38:30.920 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:30 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:38:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:30 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:38:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:30 smithi148 ceph-mon[44520]: osdmap e472: 12 total, 11 up, 9 in 2024-04-23T16:38:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:30 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3619746967' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:38:31.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:30 smithi148 ceph-mon[44520]: from='client.26302 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:38:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:30 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:38:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:30 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:38:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:30 smithi188 ceph-mon[44101]: osdmap e472: 12 total, 11 up, 9 in 2024-04-23T16:38:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:30 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3619746967' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:38:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:30 smithi188 ceph-mon[44101]: from='client.26302 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:38:31.229 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:38:31.467 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:38:31.466697 min lat: 0.000989845 max lat: 0.216385 avg lat: 0.00787136 2024-04-23T16:38:31.467 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:38:31.467 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 60 16 121985 121969 127.014 67.8125 0.0163945 0.00787136 2024-04-23T16:38:32.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:31 smithi188 ceph-mon[44101]: pgmap v1340: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.3 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-23T16:38:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:31 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:38:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:31 smithi188 ceph-mon[44101]: from='client.26302 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:38:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:31 smithi188 ceph-mon[44101]: osdmap e473: 12 total, 11 up, 9 in 2024-04-23T16:38:32.243 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:31 smithi103 ceph-mon[39629]: pgmap v1340: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.3 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-23T16:38:32.243 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:31 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:38:32.243 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:31 smithi103 ceph-mon[39629]: from='client.26302 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:38:32.243 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:31 smithi103 ceph-mon[39629]: osdmap e473: 12 total, 11 up, 9 in 2024-04-23T16:38:32.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:31 smithi148 ceph-mon[44520]: pgmap v1340: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.3 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-23T16:38:32.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:31 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:38:32.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:31 smithi148 ceph-mon[44520]: from='client.26302 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:38:32.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:31 smithi148 ceph-mon[44520]: osdmap e473: 12 total, 11 up, 9 in 2024-04-23T16:38:32.467 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 61 16 123299 123283 126.278 82.125 0.0111383 0.00791681 2024-04-23T16:38:32.623 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:38:33.467 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 62 16 124571 124555 125.524 79.5 0.015106 0.00796483 2024-04-23T16:38:33.980 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:38:33.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:33.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:38:33.994 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:38:33.994 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:38:34.090 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:38:34.120 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:34.140 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:38:34.141 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:38:34.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:34.144 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:38:34.170 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:33 smithi103 ceph-mon[39629]: pgmap v1342: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-04-23T16:38:34.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:33 smithi188 ceph-mon[44101]: pgmap v1342: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-04-23T16:38:34.245 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:34.272 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:34.294 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:34.299 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:34.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:33 smithi148 ceph-mon[44520]: pgmap v1342: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-04-23T16:38:34.400 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:34.456 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:34.467 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 63 15 125809 125794 124.761 77.4375 0.0110005 0.00801369 2024-04-23T16:38:34.511 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:34.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:34.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:34.635 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:38:34.664 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:34.685 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:38:34.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:34.688 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:38:34.789 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:34.853 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:34.877 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:34.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:34.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:34.985 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:35.011 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:35.035 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:35.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:35.040 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:35.141 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:35.209 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:35.228 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:35 smithi103 ceph-mon[39629]: osdmap e474: 12 total, 11 up, 9 in 2024-04-23T16:38:35.229 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:35 smithi103 ceph-mon[39629]: pgmap v1344: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-23T16:38:35.243 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:35.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:35.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:35.354 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:35.399 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:35.430 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:35.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:35.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:35.467 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 64 15 127018 127003 123.992 75.5625 0.0285211 0.00806378 2024-04-23T16:38:35.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:35 smithi188 ceph-mon[44101]: osdmap e474: 12 total, 11 up, 9 in 2024-04-23T16:38:35.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:35 smithi188 ceph-mon[44101]: pgmap v1344: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-23T16:38:35.537 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:38:35.583 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:35 smithi148 ceph-mon[44520]: osdmap e474: 12 total, 11 up, 9 in 2024-04-23T16:38:35.583 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:35 smithi148 ceph-mon[44520]: pgmap v1344: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-23T16:38:35.591 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:35.592 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:38:35.614 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:38:35.631 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:38:35.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:35.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:38:35.643 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:38:35.684 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:38:35.711 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:38:35.740 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:35.750 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:38:35.781 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:38:35.808 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:38:35.815 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:35.832 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:38:35.852 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:35.854 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:38:35.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:35.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:35.878 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:38:35.878 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:38:35.878 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:38:35.878 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:38:35.878 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:38:35.878 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:38:35.878 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:38:35.879 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:38:35.879 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:38:35.879 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:38:35.879 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:38:35.879 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:38:35.879 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:38:35.961 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:38:36.021 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:36.056 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:38:36.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:36.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:38:36.165 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:36.173 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-23T16:38:36.195 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:36.218 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:36.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:36.221 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:36.322 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:38:36.375 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:36.414 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:38:36.414 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:38:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:36.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:38:36.467 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 65 16 128236 128220 123.255 76.0625 0.00531003 0.00811223 2024-04-23T16:38:36.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1567242087' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:38:36.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:36.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:36.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:36.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:36.516 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:38:36.549 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1567242087' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:38:36.549 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:36.549 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:36.549 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:36.549 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:36.550 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:36.552 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:36.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1567242087' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:38:36.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:36.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:36.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:36.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:36.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:36.689 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:38:36.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:36.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:38:36.795 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:36.852 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:36.889 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:36.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:36.992 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:38:37.028 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:37.386 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:38:37.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:37.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:38:37.426 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:38:37.426 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:38:37.467 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 66 16 129483 129467 122.568 77.9375 0.00681597 0.00815733 2024-04-23T16:38:37.495 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:38:37.540 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:37.540 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:37.540 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:37.540 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:37.540 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:37.540 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:37.540 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:37 smithi103 ceph-mon[39629]: pgmap v1345: 27 pgs: 27 active+clean; 7.7 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-23T16:38:37.542 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:37.589 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:38:37.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:37.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:38:37.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:37.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:37.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:37.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:37.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:37.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:37.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:37 smithi148 ceph-mon[44520]: pgmap v1345: 27 pgs: 27 active+clean; 7.7 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-23T16:38:37.696 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:37.720 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:37.721 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:37.721 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:37.721 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:37.721 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:37.721 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:37.721 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:37 smithi188 ceph-mon[44101]: pgmap v1345: 27 pgs: 27 active+clean; 7.7 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-23T16:38:37.722 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:37.747 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:37.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:37.750 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:37.851 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:37.896 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:37.940 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:37.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:37.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:38.046 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:38:38.082 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:38.123 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:38:38.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:38.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:38:38.228 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:38:38.255 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:38.277 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:38:38.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:38.279 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:38:38.380 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:38:38.417 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:38.467 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 67 16 130728 130712 121.9 77.8125 0.0126832 0.00820184 2024-04-23T16:38:38.486 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:38:38.496 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:38:38.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:38.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:38:38.605 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:38:38.663 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:38.696 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:38:38.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:38.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:38:38.804 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:38.861 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:38.900 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:38.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:38.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:39.008 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:38:39.077 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:39.106 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:38:39.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:39.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:38:39.216 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:38:39.243 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:39.269 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:38:39.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:39.274 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:38:39.375 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:39.403 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:39.424 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:39.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:39.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:39.427 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:38:39.467 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 68 16 131912 131896 121.196 74 0.0198695 0.00824892 2024-04-23T16:38:39.528 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:38:39.554 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:39.575 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:38:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:39.578 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:38:39.679 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:38:39.728 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:39.769 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:38:39.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:39.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:38:39.875 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:39.921 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:39.952 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:39.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:39.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:40.059 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:38:40.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:39 smithi103 ceph-mon[39629]: pgmap v1346: 27 pgs: 27 active+clean; 7.8 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 61 MiB/s wr, 974 op/s 2024-04-23T16:38:40.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:39 smithi103 ceph-mon[39629]: osdmap e475: 12 total, 11 up, 9 in 2024-04-23T16:38:40.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:39 smithi148 ceph-mon[44520]: pgmap v1346: 27 pgs: 27 active+clean; 7.8 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 61 MiB/s wr, 974 op/s 2024-04-23T16:38:40.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:39 smithi148 ceph-mon[44520]: osdmap e475: 12 total, 11 up, 9 in 2024-04-23T16:38:40.136 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:40.183 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:38:40.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:40.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:38:40.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:39 smithi188 ceph-mon[44101]: pgmap v1346: 27 pgs: 27 active+clean; 7.8 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 61 MiB/s wr, 974 op/s 2024-04-23T16:38:40.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:39 smithi188 ceph-mon[44101]: osdmap e475: 12 total, 11 up, 9 in 2024-04-23T16:38:40.288 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:40.329 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:40.385 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:40.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:40.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:40.467 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 69 16 133055 133039 120.475 71.4375 0.0135586 0.0082994 2024-04-23T16:38:40.500 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:38:40.535 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:40.801 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:38:40.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:40.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:38:40.846 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:38:40.853 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:38:40.906 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:38:40.935 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:40.956 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:38:40.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:40.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:38:41.060 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:41.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:40 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1559491232' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:38:41.102 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:40 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1559491232' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:38:41.105 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:41.142 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:41.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:41.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:41.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:40 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1559491232' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:38:41.251 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:38:41.291 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:41.336 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:38:41.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:41.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:38:41.441 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:41.468 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 70 16 134190 134174 119.767 70.9375 0.0163813 0.00834806 2024-04-23T16:38:41.498 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:41.531 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:41.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:41.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:41.639 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:41.684 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:41.713 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:41.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:41.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:41.823 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:41.847 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:41 smithi148 ceph-mon[44520]: pgmap v1348: 27 pgs: 27 active+clean; 8.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-23T16:38:41.847 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:41 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:38:41.848 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:41 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1559491232' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:38:41.848 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:41 smithi148 ceph-mon[44520]: osdmap e476: 12 total, 11 up, 9 in 2024-04-23T16:38:41.848 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:41 smithi148 ceph-mon[44520]: 6.b scrub starts 2024-04-23T16:38:41.848 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:41 smithi148 ceph-mon[44520]: 6.9 scrub starts 2024-04-23T16:38:41.882 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:41.922 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:41.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:41.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:42.033 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:38:42.069 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:42.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:41 smithi103 ceph-mon[39629]: pgmap v1348: 27 pgs: 27 active+clean; 8.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-23T16:38:42.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:41 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:38:42.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:41 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1559491232' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:38:42.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:41 smithi103 ceph-mon[39629]: osdmap e476: 12 total, 11 up, 9 in 2024-04-23T16:38:42.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:41 smithi103 ceph-mon[39629]: 6.b scrub starts 2024-04-23T16:38:42.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:41 smithi103 ceph-mon[39629]: 6.9 scrub starts 2024-04-23T16:38:42.103 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:38:42.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:42.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:38:42.208 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:38:42.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:41 smithi188 ceph-mon[44101]: pgmap v1348: 27 pgs: 27 active+clean; 8.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-23T16:38:42.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:41 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:38:42.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:41 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1559491232' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:38:42.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:41 smithi188 ceph-mon[44101]: osdmap e476: 12 total, 11 up, 9 in 2024-04-23T16:38:42.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:41 smithi188 ceph-mon[44101]: 6.b scrub starts 2024-04-23T16:38:42.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:41 smithi188 ceph-mon[44101]: 6.9 scrub starts 2024-04-23T16:38:42.254 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:42.278 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:38:42.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:42.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:38:42.329 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:38:42.329 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:38:42.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.329 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:135002 6.b 6:d992295f:::benchmark_data_smithi188_130221_object135001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e476)", 2024-04-23T16:38:42.329 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:38:42.233486+0000", 2024-04-23T16:38:42.329 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.015034356, 2024-04-23T16:38:42.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015059311000000001, 2024-04-23T16:38:42.329 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:38:42.330 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:38:42.330 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:38:42.330 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:38:42.330 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:38:42.330 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:38:42.330 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 135002 2024-04-23T16:38:42.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.330 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:38:42.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:38:42.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.233486+0000", 2024-04-23T16:38:42.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:38:42.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:38:42.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.233486+0000", 2024-04-23T16:38:42.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:38:42.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:38:42.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.233488+0000", 2024-04-23T16:38:42.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5170000000000002e-06 2024-04-23T16:38:42.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:38:42.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.233522+0000", 2024-04-23T16:38:42.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3717000000000003e-05 2024-04-23T16:38:42.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:38:42.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.233540+0000", 2024-04-23T16:38:42.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7975999999999999e-05 2024-04-23T16:38:42.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:38:42.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.233551+0000", 2024-04-23T16:38:42.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1034e-05 2024-04-23T16:38:42.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:38:42.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.233605+0000", 2024-04-23T16:38:42.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4240999999999998e-05 2024-04-23T16:38:42.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:38:42.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.233780+0000", 2024-04-23T16:38:42.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017450699999999999 2024-04-23T16:38:42.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 7", 2024-04-23T16:38:42.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.233951+0000", 2024-04-23T16:38:42.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017112799999999999 2024-04-23T16:38:42.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:38:42.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.243398+0000", 2024-04-23T16:38:42.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094472989999999993 2024-04-23T16:38:42.334 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:38:42.334 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:38:42.334 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:38:42.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.334 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:135010 6.b 6:d3d389a1:::benchmark_data_smithi188_130221_object135009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e476)", 2024-04-23T16:38:42.334 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:38:42.238562+0000", 2024-04-23T16:38:42.334 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.0099587070000000007, 2024-04-23T16:38:42.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010042198, 2024-04-23T16:38:42.334 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:38:42.334 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:38:42.335 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:38:42.335 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:38:42.335 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:38:42.335 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:38:42.335 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 135010 2024-04-23T16:38:42.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.335 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:38:42.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:38:42.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.238562+0000", 2024-04-23T16:38:42.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:38:42.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:38:42.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.238562+0000", 2024-04-23T16:38:42.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:38:42.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:38:42.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.238563+0000", 2024-04-23T16:38:42.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.455e-06 2024-04-23T16:38:42.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:38:42.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.238639+0000", 2024-04-23T16:38:42.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6477999999999995e-05 2024-04-23T16:38:42.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:38:42.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.238652+0000", 2024-04-23T16:38:42.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2858e-05 2024-04-23T16:38:42.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:38:42.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.238661+0000", 2024-04-23T16:38:42.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4470000000000006e-06 2024-04-23T16:38:42.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:38:42.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.238710+0000", 2024-04-23T16:38:42.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8983999999999997e-05 2024-04-23T16:38:42.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:38:42.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.238848+0000", 2024-04-23T16:38:42.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013802399999999999 2024-04-23T16:38:42.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 7", 2024-04-23T16:38:42.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.239004+0000", 2024-04-23T16:38:42.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000156317 2024-04-23T16:38:42.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:38:42.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.243406+0000", 2024-04-23T16:38:42.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044020969999999998 2024-04-23T16:38:42.339 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:38:42.339 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:38:42.339 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:38:42.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.339 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:135019 6.c 6:3856d882:::benchmark_data_smithi188_130221_object135018:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e476)", 2024-04-23T16:38:42.339 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:38:42.244043+0000", 2024-04-23T16:38:42.339 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.0044775400000000003, 2024-04-23T16:38:42.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046102929999999997, 2024-04-23T16:38:42.339 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:38:42.339 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:38:42.339 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:38:42.339 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:38:42.340 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:38:42.340 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:38:42.340 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 135019 2024-04-23T16:38:42.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.340 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:38:42.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:38:42.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.244043+0000", 2024-04-23T16:38:42.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:38:42.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:38:42.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.244043+0000", 2024-04-23T16:38:42.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:38:42.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:38:42.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.244045+0000", 2024-04-23T16:38:42.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.137e-06 2024-04-23T16:38:42.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:38:42.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.244078+0000", 2024-04-23T16:38:42.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3111999999999997e-05 2024-04-23T16:38:42.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:38:42.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.244094+0000", 2024-04-23T16:38:42.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5829999999999999e-05 2024-04-23T16:38:42.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:38:42.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.244103+0000", 2024-04-23T16:38:42.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4220000000000006e-06 2024-04-23T16:38:42.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:38:42.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.244164+0000", 2024-04-23T16:38:42.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1266000000000006e-05 2024-04-23T16:38:42.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:38:42.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.244299+0000", 2024-04-23T16:38:42.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000134425 2024-04-23T16:38:42.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:38:42.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:38:42.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 6", 2024-04-23T16:38:42.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:38:42.244435+0000", 2024-04-23T16:38:42.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013654100000000001 2024-04-23T16:38:42.343 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:38:42.343 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:38:42.343 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:38:42.343 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:38:42.343 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:38:42.343 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 3 2024-04-23T16:38:42.343 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:38:42.383 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:38:42.441 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:42.467 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 71 16 135256 135240 119.019 66.625 0.00476383 0.00839993 2024-04-23T16:38:42.553 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:38:42.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:42.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:38:42.611 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-04-23T16:38:42.611 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2024-04-23T16:38:42.611 DEBUG:teuthology.orchestra.run.smithi103:osd.2> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.2.service 2024-04-23T16:38:42.614 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl start ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.2 2024-04-23T16:38:42.649 INFO:journalctl@ceph.osd.2.smithi103.stdout:-- Logs begin at Tue 2024-04-23 15:57:24 UTC. -- 2024-04-23T16:38:42.659 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:38:42.745 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:42.850 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:42 smithi103 systemd[1]: Starting Ceph osd.2 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T16:38:43.468 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 72 16 136344 136328 118.31 68 0.00735347 0.00845089 2024-04-23T16:38:43.603 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:38:43.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:43.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:38:43.711 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:38:43.746 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:43.766 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:38:43.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:43.771 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:38:43.872 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:43.899 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:43.919 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:43.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:43.923 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:44.024 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:44.065 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:44.073 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-23T16:38:44.111 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:43 smithi103 ceph-mon[39629]: 6.b scrub ok 2024-04-23T16:38:44.111 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:43 smithi103 ceph-mon[39629]: 6.9 scrub ok 2024-04-23T16:38:44.111 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:43 smithi103 ceph-mon[39629]: pgmap v1350: 27 pgs: 27 active+clean; 8.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-04-23T16:38:44.111 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:43 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:38:44.111 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:43 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:38:44.112 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:43 smithi103 ceph-mon[39629]: 6.0 scrub starts 2024-04-23T16:38:44.125 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:44.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:43 smithi148 ceph-mon[44520]: 6.b scrub ok 2024-04-23T16:38:44.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:43 smithi148 ceph-mon[44520]: 6.9 scrub ok 2024-04-23T16:38:44.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:43 smithi148 ceph-mon[44520]: pgmap v1350: 27 pgs: 27 active+clean; 8.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-04-23T16:38:44.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:43 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:38:44.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:43 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:38:44.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:43 smithi148 ceph-mon[44520]: 6.0 scrub starts 2024-04-23T16:38:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:44.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:44.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:43 smithi188 ceph-mon[44101]: 6.b scrub ok 2024-04-23T16:38:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:43 smithi188 ceph-mon[44101]: 6.9 scrub ok 2024-04-23T16:38:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:43 smithi188 ceph-mon[44101]: pgmap v1350: 27 pgs: 27 active+clean; 8.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-04-23T16:38:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:43 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:38:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:43 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:38:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:43 smithi188 ceph-mon[44101]: 6.0 scrub starts 2024-04-23T16:38:44.234 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:38:44.261 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:44.281 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:38:44.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:44.286 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:38:44.387 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:38:44.442 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:44.467 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 73 16 137372 137356 117.57 64.25 0.0316854 0.00850377 2024-04-23T16:38:45.114 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:44 smithi103 ceph-mon[39629]: 6.2 scrub starts 2024-04-23T16:38:45.115 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:44 smithi103 ceph-mon[39629]: osdmap e477: 12 total, 11 up, 9 in 2024-04-23T16:38:45.115 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:44 smithi103 ceph-mon[39629]: 6.0 scrub ok 2024-04-23T16:38:45.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:44 smithi148 ceph-mon[44520]: 6.2 scrub starts 2024-04-23T16:38:45.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:44 smithi148 ceph-mon[44520]: osdmap e477: 12 total, 11 up, 9 in 2024-04-23T16:38:45.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:44 smithi148 ceph-mon[44520]: 6.0 scrub ok 2024-04-23T16:38:45.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:44 smithi188 ceph-mon[44101]: 6.2 scrub starts 2024-04-23T16:38:45.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:44 smithi188 ceph-mon[44101]: osdmap e477: 12 total, 11 up, 9 in 2024-04-23T16:38:45.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:44 smithi188 ceph-mon[44101]: 6.0 scrub ok 2024-04-23T16:38:45.425 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:38:45.427 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:38:45.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:45.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:38:45.467 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 74 16 138471 138455 116.909 68.6875 0.0138656 0.00855191 2024-04-23T16:38:45.474 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:38:45.525 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:38:45.534 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:45.556 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:38:45.589 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:45.591 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:38:45.620 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:45.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:45.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:45.628 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:38:45.659 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:38:45.708 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:38:45.729 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:45.741 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:38:45.774 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:38:45.785 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:45.808 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:38:45.833 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:45.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:45.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:45.846 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:38:45.846 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:38:45.846 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:38:45.846 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-23T16:38:45.847 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:38:45.847 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:38:45.847 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:38:45.847 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:38:45.847 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:38:45.847 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:38:45.847 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:38:45.847 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:38:45.847 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:38:45.935 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:38:45.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:45 smithi103 ceph-mon[39629]: pgmap v1352: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 8.2 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 80 MiB/s wr, 1.29k op/s 2024-04-23T16:38:45.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:45 smithi103 ceph-mon[39629]: 6.2 scrub ok 2024-04-23T16:38:45.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:45 smithi103 ceph-mon[39629]: 6.c scrub starts 2024-04-23T16:38:45.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:45 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3202922093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:38:45.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:45 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:45.981 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:45 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:45.985 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:46.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:45 smithi148 ceph-mon[44520]: pgmap v1352: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 8.2 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 80 MiB/s wr, 1.29k op/s 2024-04-23T16:38:46.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:45 smithi148 ceph-mon[44520]: 6.2 scrub ok 2024-04-23T16:38:46.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:45 smithi148 ceph-mon[44520]: 6.c scrub starts 2024-04-23T16:38:46.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:45 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3202922093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:38:46.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:45 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:46.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:45 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:46.145 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:38:46.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:46.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:38:46.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:45 smithi188 ceph-mon[44101]: pgmap v1352: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 8.2 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 80 MiB/s wr, 1.29k op/s 2024-04-23T16:38:46.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:45 smithi188 ceph-mon[44101]: 6.2 scrub ok 2024-04-23T16:38:46.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:45 smithi188 ceph-mon[44101]: 6.c scrub starts 2024-04-23T16:38:46.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:45 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3202922093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:38:46.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:45 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:46.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:45 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:46.251 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:38:46.291 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:46.319 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:38:46.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:46.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:38:46.423 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:38:46.453 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:46.467 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 75 16 139555 139539 116.254 67.75 0.00487615 0.00860022 2024-04-23T16:38:46.475 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:38:46.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:46.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:38:46.580 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:38:46.643 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:46.691 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:38:46.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:46.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:38:46.800 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:38:46.831 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:46.851 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:38:46.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:46.854 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:38:46.894 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:46 smithi148 ceph-mon[44520]: 1.7 scrub starts 2024-04-23T16:38:46.894 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:46.895 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:46 smithi148 ceph-mon[44520]: 1.7 scrub ok 2024-04-23T16:38:46.955 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:46.986 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:47.016 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:47.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:47.022 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:47.114 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:38:47.122 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:47.143 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:47 smithi103 bash[220457]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T16:38:47.143 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:47 smithi103 bash[220457]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-23T16:38:47.143 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:47 smithi103 bash[220457]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-23T16:38:47.143 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:47 smithi103 bash[220457]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-23T16:38:47.143 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:47 smithi103 bash[220457]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-23T16:38:47.144 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:47 smithi103 bash[220457]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-23T16:38:47.144 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:47 smithi103 bash[220457]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-23T16:38:47.144 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:47 smithi103 bash[220457]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-23T16:38:47.144 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:46 smithi103 ceph-mon[39629]: 1.7 scrub starts 2024-04-23T16:38:47.144 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.144 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:46 smithi103 ceph-mon[39629]: 1.7 scrub ok 2024-04-23T16:38:47.144 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.144 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.144 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.144 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.144 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.144 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.144 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.145 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.172 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.172 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.172 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.172 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.172 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.172 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.172 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.172 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.175 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:47.213 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:47.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:47.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:38:47.222 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:38:47.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:46 smithi188 ceph-mon[44101]: 1.7 scrub starts 2024-04-23T16:38:47.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:46 smithi188 ceph-mon[44101]: 1.7 scrub ok 2024-04-23T16:38:47.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:38:47.322 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:38:47.358 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:47.402 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:38:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:47.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:38:47.468 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 76 16 140551 140535 115.543 62.25 0.008839 0.00865258 2024-04-23T16:38:47.514 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:38:47.544 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:47.565 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:38:47.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:47.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:38:47.673 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:38:47.705 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:47.907 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:38:47.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:47.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:38:48.011 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:48.040 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:48.063 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:48.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:48.068 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:48.169 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:48.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:47 smithi188 ceph-mon[44101]: 6.f scrub starts 2024-04-23T16:38:48.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:47 smithi188 ceph-mon[44101]: 6.c scrub ok 2024-04-23T16:38:48.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:47 smithi188 ceph-mon[44101]: pgmap v1353: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 8.4 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-23T16:38:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:47 smithi188 ceph-mon[44101]: 6.8 scrub starts 2024-04-23T16:38:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:47 smithi188 ceph-mon[44101]: 6.3 scrub starts 2024-04-23T16:38:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:47 smithi188 ceph-mon[44101]: 6.e scrub starts 2024-04-23T16:38:48.237 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:47 smithi148 ceph-mon[44520]: 6.f scrub starts 2024-04-23T16:38:48.238 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:47 smithi148 ceph-mon[44520]: 6.c scrub ok 2024-04-23T16:38:48.238 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:47 smithi148 ceph-mon[44520]: pgmap v1353: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 8.4 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-23T16:38:48.238 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:47 smithi148 ceph-mon[44520]: 6.8 scrub starts 2024-04-23T16:38:48.238 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:47 smithi148 ceph-mon[44520]: 6.3 scrub starts 2024-04-23T16:38:48.238 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:47 smithi148 ceph-mon[44520]: 6.e scrub starts 2024-04-23T16:38:48.238 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:48.265 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:48.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:48.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:48.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:47 smithi103 ceph-mon[39629]: 6.f scrub starts 2024-04-23T16:38:48.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:47 smithi103 ceph-mon[39629]: 6.c scrub ok 2024-04-23T16:38:48.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:47 smithi103 ceph-mon[39629]: pgmap v1353: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 8.4 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-23T16:38:48.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:47 smithi103 ceph-mon[39629]: 6.8 scrub starts 2024-04-23T16:38:48.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:47 smithi103 ceph-mon[39629]: 6.3 scrub starts 2024-04-23T16:38:48.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:47 smithi103 ceph-mon[39629]: 6.e scrub starts 2024-04-23T16:38:48.372 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:38:48.399 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:48.419 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:38:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:48.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:38:48.468 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 77 16 141443 141427 114.767 55.75 0.0152673 0.00871155 2024-04-23T16:38:48.524 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:38:48.555 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:48.579 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:38:48.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:48.582 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:38:48.684 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:48.751 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:48.782 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:48.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:48.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:48.890 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:48.960 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:48.992 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:48.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:48.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:49.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:48 smithi103 ceph-mon[39629]: 6.f scrub ok 2024-04-23T16:38:49.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:48 smithi103 ceph-mon[39629]: 6.8 scrub ok 2024-04-23T16:38:49.096 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:38:49.117 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:38:49.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:48 smithi148 ceph-mon[44520]: 6.f scrub ok 2024-04-23T16:38:49.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:48 smithi148 ceph-mon[44520]: 6.8 scrub ok 2024-04-23T16:38:49.165 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:49.222 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:38:49.223 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:38:49.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:48 smithi188 ceph-mon[44101]: 6.f scrub ok 2024-04-23T16:38:49.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:48 smithi188 ceph-mon[44101]: 6.8 scrub ok 2024-04-23T16:38:49.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:49.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:38:49.333 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:38:49.362 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:49.390 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:38:49.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:49.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:38:49.468 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 78 16 142298 142282 113.98 53.4375 0.0233074 0.00877086 2024-04-23T16:38:49.507 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:49.580 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:49.606 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:49.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:49.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:49.710 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:49.738 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:49.757 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:49.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:49.762 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:49.863 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:38:49.891 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:49.911 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:38:49.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:49.914 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:38:50.015 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:38:50.045 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:49 smithi188 ceph-mon[44101]: 6.7 scrub starts 2024-04-23T16:38:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:49 smithi188 ceph-mon[44101]: pgmap v1354: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 8.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 57 MiB/s wr, 904 op/s 2024-04-23T16:38:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:49 smithi188 ceph-mon[44101]: 6.5 scrub starts 2024-04-23T16:38:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:49 smithi188 ceph-mon[44101]: 6.3 scrub ok 2024-04-23T16:38:50.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:49 smithi103 ceph-mon[39629]: 6.7 scrub starts 2024-04-23T16:38:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:49 smithi103 ceph-mon[39629]: pgmap v1354: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 8.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 57 MiB/s wr, 904 op/s 2024-04-23T16:38:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:49 smithi103 ceph-mon[39629]: 6.5 scrub starts 2024-04-23T16:38:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:49 smithi103 ceph-mon[39629]: 6.3 scrub ok 2024-04-23T16:38:50.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:49 smithi148 ceph-mon[44520]: 6.7 scrub starts 2024-04-23T16:38:50.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:49 smithi148 ceph-mon[44520]: pgmap v1354: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 8.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 57 MiB/s wr, 904 op/s 2024-04-23T16:38:50.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:49 smithi148 ceph-mon[44520]: 6.5 scrub starts 2024-04-23T16:38:50.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:49 smithi148 ceph-mon[44520]: 6.3 scrub ok 2024-04-23T16:38:50.468 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 79 16 143154 143138 113.215 53.5 0.0162952 0.00883106 2024-04-23T16:38:51.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:50 smithi188 ceph-mon[44101]: 6.a scrub starts 2024-04-23T16:38:51.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:50 smithi188 ceph-mon[44101]: 6.e scrub ok 2024-04-23T16:38:51.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:50 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:38:51.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:50 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:38:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:50 smithi103 ceph-mon[39629]: 6.a scrub starts 2024-04-23T16:38:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:50 smithi103 ceph-mon[39629]: 6.e scrub ok 2024-04-23T16:38:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:50 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:38:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:50 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:38:51.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:50 smithi148 ceph-mon[44520]: 6.a scrub starts 2024-04-23T16:38:51.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:50 smithi148 ceph-mon[44520]: 6.e scrub ok 2024-04-23T16:38:51.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:50 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:38:51.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:50 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:38:51.468 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:38:51.468255 min lat: 0.000989845 max lat: 0.216385 avg lat: 0.00889077 2024-04-23T16:38:51.468 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:38:51.468 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 80 16 143988 143972 112.451 52.125 0.00614719 0.00889077 2024-04-23T16:38:52.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:51 smithi188 ceph-mon[44101]: pgmap v1355: 27 pgs: 5 active+clean+scrubbing, 22 active+clean; 8.7 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 59 MiB/s wr, 951 op/s 2024-04-23T16:38:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:51 smithi188 ceph-mon[44101]: 6.6 scrub starts 2024-04-23T16:38:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:51 smithi188 ceph-mon[44101]: osdmap e478: 12 total, 11 up, 9 in 2024-04-23T16:38:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:51 smithi188 ceph-mon[44101]: 6.5 scrub ok 2024-04-23T16:38:52.290 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:51 smithi103 ceph-mon[39629]: pgmap v1355: 27 pgs: 5 active+clean+scrubbing, 22 active+clean; 8.7 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 59 MiB/s wr, 951 op/s 2024-04-23T16:38:52.290 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:51 smithi103 ceph-mon[39629]: 6.6 scrub starts 2024-04-23T16:38:52.291 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:51 smithi103 ceph-mon[39629]: osdmap e478: 12 total, 11 up, 9 in 2024-04-23T16:38:52.291 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:51 smithi103 ceph-mon[39629]: 6.5 scrub ok 2024-04-23T16:38:52.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:51 smithi148 ceph-mon[44520]: pgmap v1355: 27 pgs: 5 active+clean+scrubbing, 22 active+clean; 8.7 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 59 MiB/s wr, 951 op/s 2024-04-23T16:38:52.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:51 smithi148 ceph-mon[44520]: 6.6 scrub starts 2024-04-23T16:38:52.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:51 smithi148 ceph-mon[44520]: osdmap e478: 12 total, 11 up, 9 in 2024-04-23T16:38:52.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:51 smithi148 ceph-mon[44520]: 6.5 scrub ok 2024-04-23T16:38:52.468 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 81 16 145041 145025 111.876 65.8125 0.0106323 0.00893703 2024-04-23T16:38:52.497 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:38:52.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:52.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:38:52.604 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:38:52.644 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:52.860 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-23T16:38:52.942 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:52 smithi103 bash[220965]: f11627421acd60a5989dd32c203664ec222ccb72b6fd66e6f1eb2791d3cc6183 2024-04-23T16:38:52.942 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:52 smithi103 systemd[1]: Started Ceph osd.2 for 8e3594f2-018b-11ef-bc93-c7b262605968. 2024-04-23T16:38:53.299 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:38:53.306 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:53 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:53.169+0000 7f313cc487c0 -1 Falling back to public interface 2024-04-23T16:38:53.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:53.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:38:53.310 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:52 smithi103 ceph-mon[39629]: 6.7 scrub ok 2024-04-23T16:38:53.310 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:52 smithi103 ceph-mon[39629]: 6.a scrub ok 2024-04-23T16:38:53.310 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:52 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:38:53.310 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:52 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:38:53.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:52 smithi148 ceph-mon[44520]: 6.7 scrub ok 2024-04-23T16:38:53.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:52 smithi148 ceph-mon[44520]: 6.a scrub ok 2024-04-23T16:38:53.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:52 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:38:53.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:52 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:38:53.380 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:38:53.380 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:38:53.410 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:53.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:52 smithi188 ceph-mon[44101]: 6.7 scrub ok 2024-04-23T16:38:53.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:52 smithi188 ceph-mon[44101]: 6.a scrub ok 2024-04-23T16:38:53.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:52 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:38:53.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:52 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:38:53.436 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:53.456 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:53.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:53.459 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:53.468 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 82 16 146162 146146 111.366 70.0625 0.0148287 0.00897818 2024-04-23T16:38:53.560 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:38:53.596 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:53.642 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:38:53.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:53.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:38:53.745 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:38:53.772 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:53.796 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:38:53.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:53.800 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:38:53.901 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:38:53.930 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:53.949 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:38:53.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:53.955 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:38:54.056 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:38:54.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:53 smithi188 ceph-mon[44101]: pgmap v1357: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 8.7 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-23T16:38:54.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:53 smithi188 ceph-mon[44101]: 6.6 scrub ok 2024-04-23T16:38:54.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:53 smithi188 ceph-mon[44101]: osdmap e479: 12 total, 11 up, 9 in 2024-04-23T16:38:54.084 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:54.105 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:38:54.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:54.108 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:38:54.209 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:38:54.237 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:54.261 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:38:54.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:54.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:38:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:53 smithi103 ceph-mon[39629]: pgmap v1357: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 8.7 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-23T16:38:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:53 smithi103 ceph-mon[39629]: 6.6 scrub ok 2024-04-23T16:38:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:53 smithi103 ceph-mon[39629]: osdmap e479: 12 total, 11 up, 9 in 2024-04-23T16:38:54.367 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:38:54.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:53 smithi148 ceph-mon[44520]: pgmap v1357: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 8.7 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-23T16:38:54.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:53 smithi148 ceph-mon[44520]: 6.6 scrub ok 2024-04-23T16:38:54.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:53 smithi148 ceph-mon[44520]: osdmap e479: 12 total, 11 up, 9 in 2024-04-23T16:38:54.396 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:54.417 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:38:54.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:54.420 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:38:54.468 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 83 16 147275 147259 110.862 69.5625 0.0149853 0.00901888 2024-04-23T16:38:54.471 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-23T16:38:54.520 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:38:54.549 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:54.573 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:38:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:54.576 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:38:54.676 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:54.728 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:54.759 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:54.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:54.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:54.866 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:38:54.913 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:54.945 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:38:54.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:54.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:38:54.998 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:54 smithi148 ceph-mon[44520]: 1.8 scrub starts 2024-04-23T16:38:55.041 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:38:55.049 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:38:55.077 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:55.099 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:38:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:55.102 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:38:55.203 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:38:55.234 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:55.235 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:54 smithi103 ceph-mon[39629]: 1.8 scrub starts 2024-04-23T16:38:55.236 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:54 smithi103 ceph-mon[39629]: 1.8 scrub ok 2024-04-23T16:38:55.236 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:54 smithi103 ceph-mon[39629]: 1.9 scrub starts 2024-04-23T16:38:55.236 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:54 smithi103 ceph-mon[39629]: 1.9 scrub ok 2024-04-23T16:38:55.345 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:38:55.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:55.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:38:55.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:54 smithi148 ceph-mon[44520]: 1.8 scrub ok 2024-04-23T16:38:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:54 smithi148 ceph-mon[44520]: 1.9 scrub starts 2024-04-23T16:38:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:54 smithi148 ceph-mon[44520]: 1.9 scrub ok 2024-04-23T16:38:55.384 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:38:55.450 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:55.468 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 84 16 148344 148328 110.338 66.8125 0.00430249 0.00906188 2024-04-23T16:38:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:55 smithi188 ceph-mon[44101]: 1.8 scrub starts 2024-04-23T16:38:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:55 smithi188 ceph-mon[44101]: 1.8 scrub ok 2024-04-23T16:38:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:55 smithi188 ceph-mon[44101]: 1.9 scrub starts 2024-04-23T16:38:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:55 smithi188 ceph-mon[44101]: 1.9 scrub ok 2024-04-23T16:38:55.503 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:55.533 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:55.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:55.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:55.637 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:55.664 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:38:55.683 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:55.706 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:55.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:55.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:55.810 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:38:55.836 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:55.857 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:38:55.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:55.860 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:38:55.961 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:38:55.987 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:56.006 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:38:56.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:56.009 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:38:56.110 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:38:56.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:56 smithi188 ceph-mon[44101]: pgmap v1359: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 8.8 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 49 MiB/s wr, 785 op/s 2024-04-23T16:38:56.140 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:56.160 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:38:56.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:56.163 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:38:56.264 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:38:56.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:56 smithi103 ceph-mon[39629]: pgmap v1359: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 8.8 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 49 MiB/s wr, 785 op/s 2024-04-23T16:38:56.306 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:56.337 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:38:56.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:56.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:38:56.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:56 smithi148 ceph-mon[44520]: pgmap v1359: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 8.8 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 49 MiB/s wr, 785 op/s 2024-04-23T16:38:56.447 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:56.468 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 85 16 149388 149372 109.807 65.25 0.0109085 0.00910521 2024-04-23T16:38:56.485 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:56.515 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:56.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:56.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:56.619 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:38:56.672 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:56.708 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:38:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:56.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:38:56.812 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:56.878 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:56.912 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:56.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:56.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:57.019 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:56 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:56.760+0000 7f313cc487c0 -1 osd.2 386 log_to_monitors true 2024-04-23T16:38:57.023 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:38:57.042 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:38:57.089 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:57.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:57 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4147277748' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:38:57.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:57 smithi148 ceph-mon[44520]: from='client.36062 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:38:57.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:57 smithi148 ceph-mon[44520]: 6.d scrub starts 2024-04-23T16:38:57.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:57 smithi148 ceph-mon[44520]: from='osd.2 v1:172.21.15.103:6809/3324226049' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T16:38:57.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:57 smithi148 ceph-mon[44520]: pgmap v1360: 27 pgs: 27 active+clean; 9.0 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-23T16:38:57.143 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:38:57.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:57.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:38:57.258 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:38:57.287 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.029+0000 7f3129a2b700 -1 osd.2 386 failed to load OSD map for epoch 426, got 0 bytes 2024-04-23T16:38:57.287 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.034+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 387, got 0 bytes 2024-04-23T16:38:57.287 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.034+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 388, got 0 bytes 2024-04-23T16:38:57.287 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 389, got 0 bytes 2024-04-23T16:38:57.287 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 390, got 0 bytes 2024-04-23T16:38:57.287 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 391, got 0 bytes 2024-04-23T16:38:57.287 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 392, got 0 bytes 2024-04-23T16:38:57.287 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 393, got 0 bytes 2024-04-23T16:38:57.287 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 394, got 0 bytes 2024-04-23T16:38:57.287 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 395, got 0 bytes 2024-04-23T16:38:57.287 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 396, got 0 bytes 2024-04-23T16:38:57.287 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 397, got 0 bytes 2024-04-23T16:38:57.287 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 398, got 0 bytes 2024-04-23T16:38:57.287 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 399, got 0 bytes 2024-04-23T16:38:57.288 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 400, got 0 bytes 2024-04-23T16:38:57.288 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 401, got 0 bytes 2024-04-23T16:38:57.288 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 402, got 0 bytes 2024-04-23T16:38:57.288 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 403, got 0 bytes 2024-04-23T16:38:57.288 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 404, got 0 bytes 2024-04-23T16:38:57.288 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 405, got 0 bytes 2024-04-23T16:38:57.288 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 406, got 0 bytes 2024-04-23T16:38:57.288 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 407, got 0 bytes 2024-04-23T16:38:57.288 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 408, got 0 bytes 2024-04-23T16:38:57.288 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 409, got 0 bytes 2024-04-23T16:38:57.288 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 410, got 0 bytes 2024-04-23T16:38:57.288 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 411, got 0 bytes 2024-04-23T16:38:57.288 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 412, got 0 bytes 2024-04-23T16:38:57.288 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 413, got 0 bytes 2024-04-23T16:38:57.289 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 414, got 0 bytes 2024-04-23T16:38:57.289 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 415, got 0 bytes 2024-04-23T16:38:57.289 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 416, got 0 bytes 2024-04-23T16:38:57.289 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 417, got 0 bytes 2024-04-23T16:38:57.289 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 418, got 0 bytes 2024-04-23T16:38:57.289 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 419, got 0 bytes 2024-04-23T16:38:57.289 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 420, got 0 bytes 2024-04-23T16:38:57.289 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 421, got 0 bytes 2024-04-23T16:38:57.289 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 422, got 0 bytes 2024-04-23T16:38:57.289 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 423, got 0 bytes 2024-04-23T16:38:57.289 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 424, got 0 bytes 2024-04-23T16:38:57.289 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 425, got 0 bytes 2024-04-23T16:38:57.289 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f311c87d700 -1 osd.2 427 failed to load OSD map for epoch 426, got 0 bytes 2024-04-23T16:38:57.289 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 387, got 0 bytes 2024-04-23T16:38:57.289 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 388, got 0 bytes 2024-04-23T16:38:57.290 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 389, got 0 bytes 2024-04-23T16:38:57.290 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 390, got 0 bytes 2024-04-23T16:38:57.290 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 391, got 0 bytes 2024-04-23T16:38:57.290 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 392, got 0 bytes 2024-04-23T16:38:57.290 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 393, got 0 bytes 2024-04-23T16:38:57.290 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 394, got 0 bytes 2024-04-23T16:38:57.290 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 395, got 0 bytes 2024-04-23T16:38:57.291 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 396, got 0 bytes 2024-04-23T16:38:57.291 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 397, got 0 bytes 2024-04-23T16:38:57.291 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 398, got 0 bytes 2024-04-23T16:38:57.291 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 399, got 0 bytes 2024-04-23T16:38:57.291 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 400, got 0 bytes 2024-04-23T16:38:57.291 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 401, got 0 bytes 2024-04-23T16:38:57.291 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 402, got 0 bytes 2024-04-23T16:38:57.291 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 403, got 0 bytes 2024-04-23T16:38:57.291 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 404, got 0 bytes 2024-04-23T16:38:57.291 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 405, got 0 bytes 2024-04-23T16:38:57.291 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 406, got 0 bytes 2024-04-23T16:38:57.291 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 407, got 0 bytes 2024-04-23T16:38:57.291 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 408, got 0 bytes 2024-04-23T16:38:57.291 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 409, got 0 bytes 2024-04-23T16:38:57.291 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 410, got 0 bytes 2024-04-23T16:38:57.292 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 411, got 0 bytes 2024-04-23T16:38:57.292 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 412, got 0 bytes 2024-04-23T16:38:57.292 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 413, got 0 bytes 2024-04-23T16:38:57.292 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 414, got 0 bytes 2024-04-23T16:38:57.292 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 415, got 0 bytes 2024-04-23T16:38:57.292 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 416, got 0 bytes 2024-04-23T16:38:57.292 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 417, got 0 bytes 2024-04-23T16:38:57.292 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 418, got 0 bytes 2024-04-23T16:38:57.292 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 419, got 0 bytes 2024-04-23T16:38:57.292 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 420, got 0 bytes 2024-04-23T16:38:57.292 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 421, got 0 bytes 2024-04-23T16:38:57.292 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 422, got 0 bytes 2024-04-23T16:38:57.292 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 423, got 0 bytes 2024-04-23T16:38:57.292 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 424, got 0 bytes 2024-04-23T16:38:57.292 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 425, got 0 bytes 2024-04-23T16:38:57.293 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116070700 -1 osd.2 427 failed to load OSD map for epoch 426, got 0 bytes 2024-04-23T16:38:57.293 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 387, got 0 bytes 2024-04-23T16:38:57.293 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 388, got 0 bytes 2024-04-23T16:38:57.293 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 389, got 0 bytes 2024-04-23T16:38:57.293 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 390, got 0 bytes 2024-04-23T16:38:57.293 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 391, got 0 bytes 2024-04-23T16:38:57.293 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 392, got 0 bytes 2024-04-23T16:38:57.293 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4147277748' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:38:57.293 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-mon[39629]: from='client.36062 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:38:57.293 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-mon[39629]: 6.d scrub starts 2024-04-23T16:38:57.293 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-mon[39629]: from='osd.2 v1:172.21.15.103:6809/3324226049' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T16:38:57.293 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-mon[39629]: pgmap v1360: 27 pgs: 27 active+clean; 9.0 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-23T16:38:57.294 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 393, got 0 bytes 2024-04-23T16:38:57.294 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 394, got 0 bytes 2024-04-23T16:38:57.294 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 395, got 0 bytes 2024-04-23T16:38:57.294 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 396, got 0 bytes 2024-04-23T16:38:57.294 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 397, got 0 bytes 2024-04-23T16:38:57.294 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 398, got 0 bytes 2024-04-23T16:38:57.295 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 399, got 0 bytes 2024-04-23T16:38:57.295 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 400, got 0 bytes 2024-04-23T16:38:57.295 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 401, got 0 bytes 2024-04-23T16:38:57.295 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 402, got 0 bytes 2024-04-23T16:38:57.295 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 403, got 0 bytes 2024-04-23T16:38:57.295 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 404, got 0 bytes 2024-04-23T16:38:57.295 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 405, got 0 bytes 2024-04-23T16:38:57.295 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 406, got 0 bytes 2024-04-23T16:38:57.295 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 407, got 0 bytes 2024-04-23T16:38:57.295 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 408, got 0 bytes 2024-04-23T16:38:57.295 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 409, got 0 bytes 2024-04-23T16:38:57.295 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 410, got 0 bytes 2024-04-23T16:38:57.295 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 411, got 0 bytes 2024-04-23T16:38:57.295 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 412, got 0 bytes 2024-04-23T16:38:57.295 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 413, got 0 bytes 2024-04-23T16:38:57.296 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 414, got 0 bytes 2024-04-23T16:38:57.296 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 415, got 0 bytes 2024-04-23T16:38:57.296 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 416, got 0 bytes 2024-04-23T16:38:57.296 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 417, got 0 bytes 2024-04-23T16:38:57.296 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 418, got 0 bytes 2024-04-23T16:38:57.296 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 419, got 0 bytes 2024-04-23T16:38:57.296 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 420, got 0 bytes 2024-04-23T16:38:57.296 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 421, got 0 bytes 2024-04-23T16:38:57.296 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 422, got 0 bytes 2024-04-23T16:38:57.296 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 423, got 0 bytes 2024-04-23T16:38:57.296 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 424, got 0 bytes 2024-04-23T16:38:57.296 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 425, got 0 bytes 2024-04-23T16:38:57.296 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.035+0000 7f3116871700 -1 osd.2 427 failed to load OSD map for epoch 426, got 0 bytes 2024-04-23T16:38:57.296 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 387, got 0 bytes 2024-04-23T16:38:57.296 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 388, got 0 bytes 2024-04-23T16:38:57.297 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 389, got 0 bytes 2024-04-23T16:38:57.297 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 390, got 0 bytes 2024-04-23T16:38:57.297 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 391, got 0 bytes 2024-04-23T16:38:57.297 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 392, got 0 bytes 2024-04-23T16:38:57.297 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 393, got 0 bytes 2024-04-23T16:38:57.297 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 394, got 0 bytes 2024-04-23T16:38:57.297 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 395, got 0 bytes 2024-04-23T16:38:57.297 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 396, got 0 bytes 2024-04-23T16:38:57.297 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 397, got 0 bytes 2024-04-23T16:38:57.297 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 398, got 0 bytes 2024-04-23T16:38:57.297 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 399, got 0 bytes 2024-04-23T16:38:57.297 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 400, got 0 bytes 2024-04-23T16:38:57.297 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 401, got 0 bytes 2024-04-23T16:38:57.298 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 402, got 0 bytes 2024-04-23T16:38:57.298 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 403, got 0 bytes 2024-04-23T16:38:57.298 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 404, got 0 bytes 2024-04-23T16:38:57.298 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 405, got 0 bytes 2024-04-23T16:38:57.298 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 406, got 0 bytes 2024-04-23T16:38:57.298 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 407, got 0 bytes 2024-04-23T16:38:57.298 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 408, got 0 bytes 2024-04-23T16:38:57.298 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 409, got 0 bytes 2024-04-23T16:38:57.298 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 410, got 0 bytes 2024-04-23T16:38:57.298 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 411, got 0 bytes 2024-04-23T16:38:57.298 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 412, got 0 bytes 2024-04-23T16:38:57.298 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 413, got 0 bytes 2024-04-23T16:38:57.298 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 414, got 0 bytes 2024-04-23T16:38:57.299 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 415, got 0 bytes 2024-04-23T16:38:57.299 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 416, got 0 bytes 2024-04-23T16:38:57.299 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 417, got 0 bytes 2024-04-23T16:38:57.299 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 418, got 0 bytes 2024-04-23T16:38:57.299 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 419, got 0 bytes 2024-04-23T16:38:57.299 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 420, got 0 bytes 2024-04-23T16:38:57.299 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 421, got 0 bytes 2024-04-23T16:38:57.299 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 422, got 0 bytes 2024-04-23T16:38:57.299 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 423, got 0 bytes 2024-04-23T16:38:57.299 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 424, got 0 bytes 2024-04-23T16:38:57.299 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 425, got 0 bytes 2024-04-23T16:38:57.299 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.037+0000 7f311b87b700 -1 osd.2 427 failed to load OSD map for epoch 426, got 0 bytes 2024-04-23T16:38:57.299 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 387, got 0 bytes 2024-04-23T16:38:57.299 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 388, got 0 bytes 2024-04-23T16:38:57.299 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 389, got 0 bytes 2024-04-23T16:38:57.299 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 390, got 0 bytes 2024-04-23T16:38:57.300 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 391, got 0 bytes 2024-04-23T16:38:57.300 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 392, got 0 bytes 2024-04-23T16:38:57.300 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 393, got 0 bytes 2024-04-23T16:38:57.300 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 394, got 0 bytes 2024-04-23T16:38:57.300 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 395, got 0 bytes 2024-04-23T16:38:57.300 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 396, got 0 bytes 2024-04-23T16:38:57.300 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 397, got 0 bytes 2024-04-23T16:38:57.300 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 398, got 0 bytes 2024-04-23T16:38:57.300 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 399, got 0 bytes 2024-04-23T16:38:57.300 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 400, got 0 bytes 2024-04-23T16:38:57.300 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 401, got 0 bytes 2024-04-23T16:38:57.300 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 402, got 0 bytes 2024-04-23T16:38:57.300 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 403, got 0 bytes 2024-04-23T16:38:57.300 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 404, got 0 bytes 2024-04-23T16:38:57.300 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 405, got 0 bytes 2024-04-23T16:38:57.301 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 406, got 0 bytes 2024-04-23T16:38:57.301 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 407, got 0 bytes 2024-04-23T16:38:57.301 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 408, got 0 bytes 2024-04-23T16:38:57.301 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 409, got 0 bytes 2024-04-23T16:38:57.301 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 410, got 0 bytes 2024-04-23T16:38:57.301 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 411, got 0 bytes 2024-04-23T16:38:57.301 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 412, got 0 bytes 2024-04-23T16:38:57.301 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 413, got 0 bytes 2024-04-23T16:38:57.301 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 414, got 0 bytes 2024-04-23T16:38:57.301 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 415, got 0 bytes 2024-04-23T16:38:57.301 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 416, got 0 bytes 2024-04-23T16:38:57.301 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 417, got 0 bytes 2024-04-23T16:38:57.301 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 418, got 0 bytes 2024-04-23T16:38:57.301 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 419, got 0 bytes 2024-04-23T16:38:57.301 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 420, got 0 bytes 2024-04-23T16:38:57.301 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 421, got 0 bytes 2024-04-23T16:38:57.302 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 422, got 0 bytes 2024-04-23T16:38:57.302 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 423, got 0 bytes 2024-04-23T16:38:57.302 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 424, got 0 bytes 2024-04-23T16:38:57.302 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 425, got 0 bytes 2024-04-23T16:38:57.302 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.038+0000 7f311586f700 -1 osd.2 427 failed to load OSD map for epoch 426, got 0 bytes 2024-04-23T16:38:57.302 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:38:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:38:57.063+0000 7f312e0a0700 -1 osd.2 480 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:38:57.302 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:57.427 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:38:57.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:57.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:38:57.468 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 86 16 150392 150376 109.26 62.75 0.0103767 0.00915148 2024-04-23T16:38:57.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:57 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4147277748' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:38:57.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:57 smithi188 ceph-mon[44101]: from='client.36062 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:38:57.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:57 smithi188 ceph-mon[44101]: 6.d scrub starts 2024-04-23T16:38:57.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:57 smithi188 ceph-mon[44101]: from='osd.2 v1:172.21.15.103:6809/3324226049' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T16:38:57.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:57 smithi188 ceph-mon[44101]: pgmap v1360: 27 pgs: 27 active+clean; 9.0 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-23T16:38:57.536 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:38:57.565 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:57.585 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:38:57.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:57.589 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:38:57.690 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:38:57.772 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:57.803 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:38:57.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:57.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:38:57.916 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:38:57.944 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:57.970 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:38:57.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:38:57.973 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:38:58.045 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:58 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:38:58.045 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:58 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:38:58.045 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:58 smithi148 ceph-mon[44520]: from='client.36062 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:38:58.045 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:58 smithi148 ceph-mon[44520]: from='osd.2 v1:172.21.15.103:6809/3324226049' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T16:38:58.046 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:58 smithi148 ceph-mon[44520]: osdmap e480: 12 total, 11 up, 9 in 2024-04-23T16:38:58.046 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:58 smithi148 ceph-mon[44520]: from='osd.2 v1:172.21.15.103:6809/3324226049' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:38:58.046 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:58 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:38:58.046 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:58 smithi148 ceph-mon[44520]: 6.4 scrub starts 2024-04-23T16:38:58.074 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:38:58.108 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:38:58.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:58 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:38:58.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:58 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:38:58.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:58 smithi188 ceph-mon[44101]: from='client.36062 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:38:58.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:58 smithi188 ceph-mon[44101]: from='osd.2 v1:172.21.15.103:6809/3324226049' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T16:38:58.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:58 smithi188 ceph-mon[44101]: osdmap e480: 12 total, 11 up, 9 in 2024-04-23T16:38:58.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:58 smithi188 ceph-mon[44101]: from='osd.2 v1:172.21.15.103:6809/3324226049' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:38:58.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:58 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:38:58.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:58 smithi188 ceph-mon[44101]: 6.4 scrub starts 2024-04-23T16:38:58.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:58 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:38:58.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:58 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:38:58.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:58 smithi103 ceph-mon[39629]: from='client.36062 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:38:58.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:58 smithi103 ceph-mon[39629]: from='osd.2 v1:172.21.15.103:6809/3324226049' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T16:38:58.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:58 smithi103 ceph-mon[39629]: osdmap e480: 12 total, 11 up, 9 in 2024-04-23T16:38:58.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:58 smithi103 ceph-mon[39629]: from='osd.2 v1:172.21.15.103:6809/3324226049' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:38:58.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:58 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:38:58.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:58 smithi103 ceph-mon[39629]: 6.4 scrub starts 2024-04-23T16:38:58.469 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 87 16 151383 151367 108.716 61.9375 0.0177733 0.00919657 2024-04-23T16:38:59.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:59 smithi103 ceph-mon[39629]: osd.2 v1:172.21.15.103:6809/3324226049 boot 2024-04-23T16:38:59.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:59 smithi103 ceph-mon[39629]: osdmap e481: 12 total, 12 up, 9 in 2024-04-23T16:38:59.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:59 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:38:59.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:38:59 smithi103 ceph-mon[39629]: pgmap v1363: 27 pgs: 27 active+clean; 9.1 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 57 MiB/s wr, 911 op/s 2024-04-23T16:38:59.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:59 smithi148 ceph-mon[44520]: osd.2 v1:172.21.15.103:6809/3324226049 boot 2024-04-23T16:38:59.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:59 smithi148 ceph-mon[44520]: osdmap e481: 12 total, 12 up, 9 in 2024-04-23T16:38:59.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:59 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:38:59.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:38:59 smithi148 ceph-mon[44520]: pgmap v1363: 27 pgs: 27 active+clean; 9.1 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 57 MiB/s wr, 911 op/s 2024-04-23T16:38:59.469 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 88 16 152373 152357 108.184 61.875 0.0145024 0.00924189 2024-04-23T16:38:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:59 smithi188 ceph-mon[44101]: osd.2 v1:172.21.15.103:6809/3324226049 boot 2024-04-23T16:38:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:59 smithi188 ceph-mon[44101]: osdmap e481: 12 total, 12 up, 9 in 2024-04-23T16:38:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:59 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:38:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:38:59 smithi188 ceph-mon[44101]: pgmap v1363: 27 pgs: 27 active+clean; 9.1 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 57 MiB/s wr, 911 op/s 2024-04-23T16:38:59.993 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:39:00.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:00.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:39:00.106 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:39:00.167 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:00.204 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:39:00.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:00.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:39:00.286 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:00.286 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:00 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:39:00.286 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:00 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:39:00.286 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:00 smithi103 ceph-mon[39629]: 6.d scrub ok 2024-04-23T16:39:00.286 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:00 smithi103 ceph-mon[39629]: osdmap e482: 12 total, 12 up, 9 in 2024-04-23T16:39:00.313 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:39:00.368 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:00.368 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:00 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:39:00.368 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:00 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:39:00.368 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:00 smithi148 ceph-mon[44520]: 6.d scrub ok 2024-04-23T16:39:00.368 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:00 smithi148 ceph-mon[44520]: osdmap e482: 12 total, 12 up, 9 in 2024-04-23T16:39:00.370 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:00.402 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:39:00.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:00.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:39:00.469 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 89 16 153437 153421 107.716 66.5 0.0151232 0.00928238 2024-04-23T16:39:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:00 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:39:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:00 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:39:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:00 smithi188 ceph-mon[44101]: 6.d scrub ok 2024-04-23T16:39:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:00 smithi188 ceph-mon[44101]: osdmap e482: 12 total, 12 up, 9 in 2024-04-23T16:39:00.511 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:39:00.579 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:00.623 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:39:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:00.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:39:00.732 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:39:00.759 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:39:00.776 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:00.825 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:39:00.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:00.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:39:00.930 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:39:00.987 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:01.027 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:39:01.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:01.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:39:01.132 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:39:01.159 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:01.179 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:39:01.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:01.182 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:39:01.282 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:39:01.339 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:01 smithi148 ceph-mon[44520]: 6.4 scrub ok 2024-04-23T16:39:01.339 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:01 smithi148 ceph-mon[44520]: pgmap v1365: 27 pgs: 27 active+clean; 9.3 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-04-23T16:39:01.346 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:01.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:01 smithi103 ceph-mon[39629]: 6.4 scrub ok 2024-04-23T16:39:01.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:01 smithi103 ceph-mon[39629]: pgmap v1365: 27 pgs: 27 active+clean; 9.3 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-04-23T16:39:01.383 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:39:01.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:01.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:39:01.469 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Total time run: 90.0073 2024-04-23T16:39:01.469 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Total writes made: 154458 2024-04-23T16:39:01.470 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Write size: 65536 2024-04-23T16:39:01.470 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Object size: 65536 2024-04-23T16:39:01.470 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Bandwidth (MB/sec): 107.254 2024-04-23T16:39:01.470 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev Bandwidth: 58.7067 2024-04-23T16:39:01.470 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max bandwidth (MB/sec): 351.25 2024-04-23T16:39:01.470 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min bandwidth (MB/sec): 52.125 2024-04-23T16:39:01.470 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Average IOPS: 1716 2024-04-23T16:39:01.470 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev IOPS: 939.307 2024-04-23T16:39:01.470 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max IOPS: 5620 2024-04-23T16:39:01.470 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min IOPS: 834 2024-04-23T16:39:01.470 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Average Latency(s): 0.00932313 2024-04-23T16:39:01.470 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev Latency(s): 0.0085582 2024-04-23T16:39:01.470 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max latency(s): 0.216385 2024-04-23T16:39:01.471 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min latency(s): 0.000989845 2024-04-23T16:39:01.472 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:39:01.472 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:39:01.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:01 smithi188 ceph-mon[44101]: 6.4 scrub ok 2024-04-23T16:39:01.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:01 smithi188 ceph-mon[44101]: pgmap v1365: 27 pgs: 27 active+clean; 9.3 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-04-23T16:39:01.479 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Cleaning up (deleting benchmark objects) 2024-04-23T16:39:01.491 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:39:01.519 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:01.538 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:39:01.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:01.541 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:39:01.643 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:39:01.697 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:01.729 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:39:01.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:01.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:39:01.771 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:39:01.798 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:39:01.821 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:39:01.833 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:39:01.852 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:39:01.876 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:39:01.883 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:01.918 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:39:01.951 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:39:01.976 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:39:01.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:01.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:39:01.992 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:39:02.020 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:39:02.053 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:39:02.061 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:02 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1247078118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:39:02.062 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:02.083 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:39:02.086 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:39:02.107 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:39:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:39:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:39:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:39:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:39:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:39:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:39:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:39:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:39:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:39:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:39:02.113 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:39:02.114 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:39:02.135 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:02 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1247078118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:39:02.135 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:02.155 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:02.187 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:02 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1247078118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:39:02.187 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:02.204 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:39:02.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:02.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:39:02.308 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:39:02.337 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:02.359 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:39:02.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:02.363 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:39:02.464 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:39:02.492 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:02.512 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:39:02.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:02.515 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:39:02.616 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:39:02.680 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:02.720 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:39:02.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:02.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:39:02.821 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-23T16:39:02.826 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:39:02.902 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:02.933 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:39:02.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:02.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:39:03.042 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:39:03.100 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:03.172 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:39:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:03.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:39:03.279 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:39:03.302 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:39:03.307 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:39:03.307 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:39:03.307 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-23T16:39:03.307 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:39:03.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:03 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:39:03.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:03 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:39:03.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:03 smithi103 ceph-mon[39629]: pgmap v1366: 27 pgs: 27 active+clean; 9.3 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 56 MiB/s wr, 904 op/s 2024-04-23T16:39:03.341 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.341 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.341 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.341 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.341 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.341 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.341 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.341 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.342 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.342 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.342 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.342 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:03 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:39:03.342 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:03 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:39:03.342 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:03 smithi148 ceph-mon[44520]: pgmap v1366: 27 pgs: 27 active+clean; 9.3 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 56 MiB/s wr, 904 op/s 2024-04-23T16:39:03.343 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:03.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:03 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:39:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:03 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:39:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:03 smithi188 ceph-mon[44101]: pgmap v1366: 27 pgs: 27 active+clean; 9.3 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 56 MiB/s wr, 904 op/s 2024-04-23T16:39:03.631 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:39:03.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:03.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:39:03.746 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:39:03.772 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:03.795 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:39:03.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:03.798 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:39:03.899 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:39:03.935 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:03.960 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:39:03.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:03.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:39:04.065 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:39:04.101 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:04.179 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:39:04.180 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:39:04.183 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:04 smithi103 ceph-mon[39629]: osdmap e483: 12 total, 12 up, 9 in 2024-04-23T16:39:04.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:04.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:39:04.285 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:39:04.321 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:04.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:04 smithi148 ceph-mon[44520]: osdmap e483: 12 total, 12 up, 9 in 2024-04-23T16:39:04.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:04 smithi188 ceph-mon[44101]: osdmap e483: 12 total, 12 up, 9 in 2024-04-23T16:39:05.077 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:39:05.080 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:39:05.080 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-04-23T16:39:05.080 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-23T16:39:05.080 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-23T16:39:05.080 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-04-23T16:39:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:05.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:39:05.183 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:39:05.213 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:05.233 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:39:05.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:05.241 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:39:05.342 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:39:05.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:05 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:39:05.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:05 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:39:05.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:05 smithi103 ceph-mon[39629]: pgmap v1368: 27 pgs: 27 active+clean; 9.3 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 53 MiB/s wr, 955 op/s 2024-04-23T16:39:05.396 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:05.429 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:39:05.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:05 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:39:05.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:05 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:39:05.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:05 smithi188 ceph-mon[44101]: pgmap v1368: 27 pgs: 27 active+clean; 9.3 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 53 MiB/s wr, 955 op/s 2024-04-23T16:39:05.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:05 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:39:05.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:05 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:39:05.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:05 smithi148 ceph-mon[44520]: pgmap v1368: 27 pgs: 27 active+clean; 9.3 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 53 MiB/s wr, 955 op/s 2024-04-23T16:39:05.971 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:39:05.989 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:39:05.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:05.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:39:06.098 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:39:06.281 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:06.595 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:39:06.610 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:06 smithi188 ceph-mon[44101]: 1.4 scrub starts 2024-04-23T16:39:06.610 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:06 smithi188 ceph-mon[44101]: 1.4 scrub ok 2024-04-23T16:39:06.616 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:06 smithi148 ceph-mon[44520]: 1.4 scrub starts 2024-04-23T16:39:06.616 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:06 smithi148 ceph-mon[44520]: 1.4 scrub ok 2024-04-23T16:39:06.847 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:06 smithi103 ceph-mon[39629]: 1.4 scrub starts 2024-04-23T16:39:06.847 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:06 smithi103 ceph-mon[39629]: 1.4 scrub ok 2024-04-23T16:39:07.560 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:39:07.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:07.564 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:39:07.644 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:39:07.664 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:39:07.709 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:07.869 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:39:07.869 INFO:teuthology.orchestra.run.smithi188.stdout: "success": "" 2024-04-23T16:39:07.869 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:39:07.872 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:07 smithi148 ceph-mon[44520]: 1.5 scrub starts 2024-04-23T16:39:07.872 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:07 smithi148 ceph-mon[44520]: 1.5 scrub ok 2024-04-23T16:39:07.872 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:07 smithi148 ceph-mon[44520]: pgmap v1369: 27 pgs: 27 active+clean; 9.2 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 127 B/s rd, 47 MiB/s wr, 1.21k op/s 2024-04-23T16:39:07.872 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:07 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1915331612' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:39:07.872 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:07 smithi148 ceph-mon[44520]: from='client.26338 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:39:07.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:07 smithi188 ceph-mon[44101]: 1.5 scrub starts 2024-04-23T16:39:07.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:07 smithi188 ceph-mon[44101]: 1.5 scrub ok 2024-04-23T16:39:07.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:07 smithi188 ceph-mon[44101]: pgmap v1369: 27 pgs: 27 active+clean; 9.2 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 127 B/s rd, 47 MiB/s wr, 1.21k op/s 2024-04-23T16:39:07.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:07 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1915331612' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:39:07.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:07 smithi188 ceph-mon[44101]: from='client.26338 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:39:07.919 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:39:07.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:07.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:39:07.925 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:07 smithi103 ceph-mon[39629]: 1.5 scrub starts 2024-04-23T16:39:07.925 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:07 smithi103 ceph-mon[39629]: 1.5 scrub ok 2024-04-23T16:39:07.925 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:07 smithi103 ceph-mon[39629]: pgmap v1369: 27 pgs: 27 active+clean; 9.2 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 127 B/s rd, 47 MiB/s wr, 1.21k op/s 2024-04-23T16:39:07.925 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:07 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1915331612' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:39:07.925 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:07 smithi103 ceph-mon[39629]: from='client.26338 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:39:08.025 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:39:08.059 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:08.183 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:39:08.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:08.186 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:39:08.287 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:39:08.352 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:08.412 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:39:08.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:08.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:39:08.518 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:39:08.552 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:08.680 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:39:08.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:08.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:39:08.788 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:39:08.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:08 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:39:08.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:08 smithi103 ceph-mon[39629]: from='client.26338 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:39:08.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:08 smithi103 ceph-mon[39629]: osdmap e484: 12 total, 12 up, 9 in 2024-04-23T16:39:08.854 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:08.894 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:39:08.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:08.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:39:08.901 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:08 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:39:08.901 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:08 smithi148 ceph-mon[44520]: from='client.26338 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:39:08.901 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:08 smithi148 ceph-mon[44520]: osdmap e484: 12 total, 12 up, 9 in 2024-04-23T16:39:08.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:08 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:39:08.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:08 smithi188 ceph-mon[44101]: from='client.26338 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:39:08.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:08 smithi188 ceph-mon[44101]: osdmap e484: 12 total, 12 up, 9 in 2024-04-23T16:39:09.001 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:39:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:09.052 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:39:09.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:09.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:39:09.071 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:39:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:39:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:162370 6.2 6:4256984d:::benchmark_data_smithi188_130221_object7909:head [delete] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-23T16:39:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:39:09.060787+0000", 2024-04-23T16:39:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.0086042810000000001, 2024-04-23T16:39:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0086443270000000003, 2024-04-23T16:39:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:39:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:39:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:39:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 162370 2024-04-23T16:39:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.060787+0000", 2024-04-23T16:39:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.060787+0000", 2024-04-23T16:39:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.060789+0000", 2024-04-23T16:39:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6780000000000001e-06 2024-04-23T16:39:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.060794+0000", 2024-04-23T16:39:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8980000000000004e-06 2024-04-23T16:39:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.060804+0000", 2024-04-23T16:39:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0402e-05 2024-04-23T16:39:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.060810+0000", 2024-04-23T16:39:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8690000000000001e-06 2024-04-23T16:39:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.060858+0000", 2024-04-23T16:39:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8415000000000003e-05 2024-04-23T16:39:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.060970+0000", 2024-04-23T16:39:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011132 2024-04-23T16:39:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:39:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.061042+0000", 2024-04-23T16:39:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2241000000000002e-05 2024-04-23T16:39:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.067228+0000", 2024-04-23T16:39:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0061861559999999999 2024-04-23T16:39:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.16308.0:162373 6.6 e484/455 6:636b6e11:::benchmark_data_smithi188_130221_object7912:head v 484'10238, mlcod=484'10236)", 2024-04-23T16:39:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:39:09.062185+0000", 2024-04-23T16:39:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.0072064169999999997, 2024-04-23T16:39:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.007349062, 2024-04-23T16:39:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:39:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.062185+0000", 2024-04-23T16:39:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.062185+0000", 2024-04-23T16:39:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.062186+0000", 2024-04-23T16:39:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3400000000000001e-06 2024-04-23T16:39:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.062192+0000", 2024-04-23T16:39:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7830000000000004e-06 2024-04-23T16:39:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.062204+0000", 2024-04-23T16:39:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2289e-05 2024-04-23T16:39:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.062211+0000", 2024-04-23T16:39:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4400000000000002e-06 2024-04-23T16:39:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:09.079 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.062288+0000", 2024-04-23T16:39:09.079 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7244999999999996e-05 2024-04-23T16:39:09.079 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.079 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.079 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:09.079 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.062306+0000", 2024-04-23T16:39:09.079 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7898999999999999e-05 2024-04-23T16:39:09.079 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:09.079 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:09.079 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:09.079 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.079 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.079 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:162377 6.2 6:4357a68f:::benchmark_data_smithi188_130221_object7916:head [delete] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-23T16:39:09.079 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:39:09.068956+0000", 2024-04-23T16:39:09.079 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.00043571899999999999, 2024-04-23T16:39:09.080 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00064257500000000005, 2024-04-23T16:39:09.080 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:09.080 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:09.080 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:39:09.080 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:09.080 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:39:09.080 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:39:09.080 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 162377 2024-04-23T16:39:09.080 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.080 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:09.080 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.080 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:09.080 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.068956+0000", 2024-04-23T16:39:09.080 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:09.080 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.081 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:09.081 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.068956+0000", 2024-04-23T16:39:09.081 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:09.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.081 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:09.081 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.068958+0000", 2024-04-23T16:39:09.081 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.165e-06 2024-04-23T16:39:09.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.081 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:09.081 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.068963+0000", 2024-04-23T16:39:09.081 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6269999999999997e-06 2024-04-23T16:39:09.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.068974+0000", 2024-04-23T16:39:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0529e-05 2024-04-23T16:39:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.068980+0000", 2024-04-23T16:39:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2199999999999997e-06 2024-04-23T16:39:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.069021+0000", 2024-04-23T16:39:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0738999999999997e-05 2024-04-23T16:39:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.069113+0000", 2024-04-23T16:39:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2186000000000001e-05 2024-04-23T16:39:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:39:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:09.069194+0000", 2024-04-23T16:39:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1477999999999995e-05 2024-04-23T16:39:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-04-23T16:39:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 3 2024-04-23T16:39:09.084 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:39:09.156 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:39:09.186 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:09.345 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:39:09.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:09.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:39:09.458 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:39:09.487 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:09.624 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:39:09.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:09.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:39:09.668 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:39:09.695 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:39:09.720 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:39:09.733 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:39:09.747 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:39:09.765 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:09.779 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:39:09.788 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:39:09.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:09.791 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:39:09.816 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:39:09.853 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:39:09.882 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:39:09.892 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:39:09.905 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:39:09.928 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:39:09.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:09 smithi148 ceph-mon[44520]: pgmap v1371: 27 pgs: 27 active+clean; 9.2 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 127 B/s rd, 20 MiB/s wr, 858 op/s 2024-04-23T16:39:09.931 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:09.950 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:39:09.971 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:39:09.973 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:39:09.973 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:39:09.973 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:39:09.973 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:39:09.973 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:39:09.974 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:39:09.974 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:39:09.974 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:39:09.974 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:39:09.974 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:39:09.974 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:39:09.974 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:39:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:09 smithi188 ceph-mon[44101]: pgmap v1371: 27 pgs: 27 active+clean; 9.2 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 127 B/s rd, 20 MiB/s wr, 858 op/s 2024-04-23T16:39:09.996 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:09 smithi103 ceph-mon[39629]: pgmap v1371: 27 pgs: 27 active+clean; 9.2 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 127 B/s rd, 20 MiB/s wr, 858 op/s 2024-04-23T16:39:10.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:10 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:39:10.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:10 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:39:10.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:10 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3300870586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:39:10.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:10 smithi103 ceph-mon[39629]: osdmap e485: 12 total, 12 up, 9 in 2024-04-23T16:39:10.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:10.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:10.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:10.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:10.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:10.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:10.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:10.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:10.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:10 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:39:10.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:10 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:39:10.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:10 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3300870586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:39:10.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:10 smithi188 ceph-mon[44101]: osdmap e485: 12 total, 12 up, 9 in 2024-04-23T16:39:10.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:10.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:10.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:10.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:10.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:10.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:10.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:10.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.086 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:39:11.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:10 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:39:11.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:10 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:39:11.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:10 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3300870586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:39:11.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:10 smithi148 ceph-mon[44520]: osdmap e485: 12 total, 12 up, 9 in 2024-04-23T16:39:11.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.431 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:39:11.607 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:39:11.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:11.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:39:11.717 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:39:11.747 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:11.769 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:39:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:11.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:39:11.873 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:39:11.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:11 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:11 smithi148 ceph-mon[44520]: pgmap v1373: 27 pgs: 27 active+clean; 8.9 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 132 B/s rd, 11 MiB/s wr, 1.19k op/s 2024-04-23T16:39:11.927 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:11.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:11 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.950 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:11 smithi103 ceph-mon[39629]: pgmap v1373: 27 pgs: 27 active+clean; 8.9 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 132 B/s rd, 11 MiB/s wr, 1.19k op/s 2024-04-23T16:39:11.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:11 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:11 smithi188 ceph-mon[44101]: pgmap v1373: 27 pgs: 27 active+clean; 8.9 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 132 B/s rd, 11 MiB/s wr, 1.19k op/s 2024-04-23T16:39:13.030 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:39:13.842 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:13 smithi103 ceph-mon[39629]: pgmap v1374: 27 pgs: 27 active+clean; 8.9 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 127 B/s rd, 5.4 MiB/s wr, 1.14k op/s 2024-04-23T16:39:13.981 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:39:14.112 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:39:14.114 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:13 smithi148 ceph-mon[44520]: pgmap v1374: 27 pgs: 27 active+clean; 8.9 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 127 B/s rd, 5.4 MiB/s wr, 1.14k op/s 2024-04-23T16:39:14.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:14.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:39:14.159 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-23T16:39:14.206 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10, 1, 2] dead_osds: [] live_osds: [0, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 2] 2024-04-23T16:39:14.206 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:39:14.206 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-04-23T16:39:14.206 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd in 1 2024-04-23T16:39:14.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:13 smithi188 ceph-mon[44101]: pgmap v1374: 27 pgs: 27 active+clean; 8.9 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 127 B/s rd, 5.4 MiB/s wr, 1.14k op/s 2024-04-23T16:39:14.231 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:39:14.309 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:39:14.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:14.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:39:14.441 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:39:14.473 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:14.513 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:39:14.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:14.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:39:14.618 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:39:14.647 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:14.664 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:39:14.871 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:39:14.900 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:39:14.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:14.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:39:15.005 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:39:15.042 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:15.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:14 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1343300468' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:39:15.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:14 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1343300468' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:39:15.152 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:39:15.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:15.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:39:15.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:14 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1343300468' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:39:15.256 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:39:15.327 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:15 smithi103 ceph-mon[39629]: pgmap v1375: 27 pgs: 27 active+clean; 8.8 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 844 op/s 2024-04-23T16:39:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:15 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:39:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:15 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1343300468' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:39:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:15 smithi103 ceph-mon[39629]: osdmap e486: 12 total, 12 up, 9 in 2024-04-23T16:39:16.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:15 smithi148 ceph-mon[44520]: pgmap v1375: 27 pgs: 27 active+clean; 8.8 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 844 op/s 2024-04-23T16:39:16.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:15 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:39:16.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:15 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1343300468' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:39:16.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:15 smithi148 ceph-mon[44520]: osdmap e486: 12 total, 12 up, 9 in 2024-04-23T16:39:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:15 smithi188 ceph-mon[44101]: pgmap v1375: 27 pgs: 27 active+clean; 8.8 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 844 op/s 2024-04-23T16:39:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:15 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:39:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:15 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1343300468' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:39:16.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:15 smithi188 ceph-mon[44101]: osdmap e486: 12 total, 12 up, 9 in 2024-04-23T16:39:16.383 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:39:16.651 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:39:16.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:16.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:39:16.755 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:39:16.813 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:16.868 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:39:16.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:16.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:39:16.984 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:39:17.054 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:17.191 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:39:17.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:17.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:39:17.295 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:39:17.324 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:17.343 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:39:17.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:17.346 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:39:17.447 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:39:17.475 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:17.494 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:39:17.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:17.499 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:39:17.600 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:39:17.634 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:17.663 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:39:17.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:17.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:39:17.712 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:39:17.738 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:39:17.768 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:39:17.769 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:39:17.790 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:39:17.798 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:39:17.834 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:17.843 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:39:17.877 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:39:17.921 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:39:17.951 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:39:17.967 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:39:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:17.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:39:17.974 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:39:18.000 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:39:18.033 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:39:18.050 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:39:18.050 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:39:18.050 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2024-04-23T16:39:18.050 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2024-04-23T16:39:18.050 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:39:18.061 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:39:18.068 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:39:18.068 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:39:18.068 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:39:18.068 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:39:18.068 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:39:18.068 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:39:18.068 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:39:18.068 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:39:18.068 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:39:18.068 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:39:18.068 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:39:18.068 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:39:18.072 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:39:18.086 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:17 smithi103 ceph-mon[39629]: pgmap v1377: 27 pgs: 27 active+clean; 8.6 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:39:18.086 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:17 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3848637090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:39:18.123 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:18.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:17 smithi188 ceph-mon[44101]: pgmap v1377: 27 pgs: 27 active+clean; 8.6 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:39:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:17 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3848637090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:39:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:17 smithi148 ceph-mon[44520]: pgmap v1377: 27 pgs: 27 active+clean; 8.6 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:39:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:17 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3848637090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:39:18.909 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:39:18.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:18.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:39:19.019 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:39:19.073 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:19.166 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.166 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.166 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.166 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.166 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.166 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.166 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.166 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.166 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.166 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.167 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:18 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.174 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:39:19.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:19.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:39:19.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:18 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.265 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.265 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.265 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.266 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.266 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.266 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.266 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.266 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.266 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.266 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.266 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:18 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:19.290 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:39:19.328 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:19.358 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:39:19.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:19.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:39:19.464 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:39:19.513 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:19.556 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:39:19.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:19.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:39:19.599 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-23T16:39:19.663 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:39:19.768 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:19.806 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:39:19.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:19.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:39:19.905 INFO:teuthology.orchestra.run.smithi103.stderr:marked in osd.1. 2024-04-23T16:39:19.915 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:39:19.946 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:20.005 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:39:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:19 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T16:39:19.789+0000 7f135b1c7700 -1 mon.a@0(leader).osd e486 definitely_dead 0 2024-04-23T16:39:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:19 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:19 smithi103 ceph-mon[39629]: pgmap v1378: 27 pgs: 27 active+clean; 8.5 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-23T16:39:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:19 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3928481040' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-23T16:39:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:19 smithi103 ceph-mon[39629]: from='client.26356 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-23T16:39:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:19 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:19 smithi148 ceph-mon[44520]: pgmap v1378: 27 pgs: 27 active+clean; 8.5 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-23T16:39:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:19 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3928481040' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-23T16:39:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:19 smithi148 ceph-mon[44520]: from='client.26356 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-23T16:39:20.148 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:39:20.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:20.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:39:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:19 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:19 smithi188 ceph-mon[44101]: pgmap v1378: 27 pgs: 27 active+clean; 8.5 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-23T16:39:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:19 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3928481040' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-23T16:39:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:19 smithi188 ceph-mon[44101]: from='client.26356 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-23T16:39:20.251 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:39:20.279 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:20.730 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:39:20.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:20.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:39:20.736 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-04-23T16:39:20.737 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:39:20.834 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:39:20.863 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:21.033 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:20 smithi103 ceph-mon[39629]: from='client.26356 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-04-23T16:39:21.033 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:20 smithi103 ceph-mon[39629]: osdmap e487: 12 total, 12 up, 10 in 2024-04-23T16:39:21.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:20 smithi148 ceph-mon[44520]: from='client.26356 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-04-23T16:39:21.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:20 smithi148 ceph-mon[44520]: osdmap e487: 12 total, 12 up, 10 in 2024-04-23T16:39:21.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:20 smithi188 ceph-mon[44101]: from='client.26356 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-04-23T16:39:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:20 smithi188 ceph-mon[44101]: osdmap e487: 12 total, 12 up, 10 in 2024-04-23T16:39:21.313 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:39:21.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:21.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:39:21.417 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:39:21.445 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:21.643 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:39:21.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:21.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:39:21.748 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:39:21.778 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:21.837 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:39:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:21 smithi103 ceph-mon[39629]: pgmap v1380: 27 pgs: 1 peering, 26 active+clean; 8.4 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-23T16:39:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:21 smithi103 ceph-mon[39629]: osdmap e488: 12 total, 12 up, 10 in 2024-04-23T16:39:22.126 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:39:22.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:22.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:39:22.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:21 smithi188 ceph-mon[44101]: pgmap v1380: 27 pgs: 1 peering, 26 active+clean; 8.4 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-23T16:39:22.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:21 smithi188 ceph-mon[44101]: osdmap e488: 12 total, 12 up, 10 in 2024-04-23T16:39:22.230 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:39:22.280 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:21 smithi148 ceph-mon[44520]: pgmap v1380: 27 pgs: 1 peering, 26 active+clean; 8.4 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-23T16:39:22.280 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:21 smithi148 ceph-mon[44520]: osdmap e488: 12 total, 12 up, 10 in 2024-04-23T16:39:22.297 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:22.323 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:39:22.327 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:39:22.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:22.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:39:22.433 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:39:22.496 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:39:22.496 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2024-04-23T16:39:22.496 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2024-04-23T16:39:22.496 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:39:22.496 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.496 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:290055 5.1 5:89de282a:::benchmark_data_smithi188_113957_object136457:head [delete] snapc 0=[] ondisk+write+known_if_redirected e441)", 2024-04-23T16:39:22.496 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:36:36.841217+0000", 2024-04-23T16:39:22.496 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 165.64889048000001, 2024-04-23T16:39:22.496 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.065845413000000005, 2024-04-23T16:39:22.496 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.496 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.497 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.497 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:22.497 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:39:22.497 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:39:22.497 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 290055 2024-04-23T16:39:22.497 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.497 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.497 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.497 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.497 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.841217+0000", 2024-04-23T16:39:22.497 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.497 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.497 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.497 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.498 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.841217+0000", 2024-04-23T16:39:22.498 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.498 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.498 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.498 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.498 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.841219+0000", 2024-04-23T16:39:22.498 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.672e-06 2024-04-23T16:39:22.498 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.498 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.498 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.498 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.841225+0000", 2024-04-23T16:39:22.498 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9859999999999999e-06 2024-04-23T16:39:22.498 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.498 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.499 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.499 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.841234+0000", 2024-04-23T16:39:22.499 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.4800000000000007e-06 2024-04-23T16:39:22.499 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.499 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.499 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.499 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.841242+0000", 2024-04-23T16:39:22.499 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7230000000000004e-06 2024-04-23T16:39:22.499 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.499 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.499 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.499 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.846315+0000", 2024-04-23T16:39:22.499 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0050732609999999999 2024-04-23T16:39:22.499 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.499 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.500 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.500 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.846458+0000", 2024-04-23T16:39:22.500 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014247899999999999 2024-04-23T16:39:22.500 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.500 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.500 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:39:22.500 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.846612+0000", 2024-04-23T16:39:22.500 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015456199999999999 2024-04-23T16:39:22.500 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.500 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.500 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.500 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.857048+0000", 2024-04-23T16:39:22.500 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010435850999999999 2024-04-23T16:39:22.500 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.500 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.501 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:39:22.501 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.906943+0000", 2024-04-23T16:39:22.501 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.049895282999999999 2024-04-23T16:39:22.501 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.501 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.501 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.501 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.907011+0000", 2024-04-23T16:39:22.501 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7145999999999997e-05 2024-04-23T16:39:22.501 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.501 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.501 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.501 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.907063+0000", 2024-04-23T16:39:22.501 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1969999999999999e-05 2024-04-23T16:39:22.501 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.502 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.502 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.502 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.502 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.502 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:290060 5.c 5:35904e5e:::benchmark_data_smithi188_113957_object136462:head [delete] snapc 0=[] ondisk+write+known_if_redirected e441)", 2024-04-23T16:39:22.502 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:36:36.846971+0000", 2024-04-23T16:39:22.502 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 165.64313700299999, 2024-04-23T16:39:22.502 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.060108672000000002, 2024-04-23T16:39:22.502 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.502 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.502 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.502 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:22.502 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:39:22.503 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:39:22.503 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 290060 2024-04-23T16:39:22.503 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.503 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.503 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.503 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.503 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.846971+0000", 2024-04-23T16:39:22.503 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.503 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.503 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.503 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.503 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.846971+0000", 2024-04-23T16:39:22.503 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.503 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.503 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.503 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.504 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.846973+0000", 2024-04-23T16:39:22.504 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4399999999999999e-06 2024-04-23T16:39:22.504 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.504 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.504 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.504 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.846979+0000", 2024-04-23T16:39:22.504 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7270000000000004e-06 2024-04-23T16:39:22.504 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.504 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.504 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.504 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.846990+0000", 2024-04-23T16:39:22.504 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1522000000000001e-05 2024-04-23T16:39:22.504 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.504 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.504 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.504 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.846999+0000", 2024-04-23T16:39:22.505 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6670000000000002e-06 2024-04-23T16:39:22.505 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.505 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.505 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.505 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.847065+0000", 2024-04-23T16:39:22.505 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5716000000000006e-05 2024-04-23T16:39:22.505 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.505 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.505 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.505 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.847163+0000", 2024-04-23T16:39:22.505 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8642999999999997e-05 2024-04-23T16:39:22.505 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.505 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.505 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:39:22.505 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.847859+0000", 2024-04-23T16:39:22.506 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00069541899999999996 2024-04-23T16:39:22.506 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.506 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.506 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.506 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.857125+0000", 2024-04-23T16:39:22.506 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0092661159999999996 2024-04-23T16:39:22.506 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.506 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.506 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:39:22.506 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.906988+0000", 2024-04-23T16:39:22.506 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.049863166 2024-04-23T16:39:22.506 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.506 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.506 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.507 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.907037+0000", 2024-04-23T16:39:22.507 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8565e-05 2024-04-23T16:39:22.507 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.507 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.507 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.507 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.907079+0000", 2024-04-23T16:39:22.507 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2691000000000002e-05 2024-04-23T16:39:22.507 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.507 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.507 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.507 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.507 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.507 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:290062 5.1 5:84f36251:::benchmark_data_smithi188_113957_object136464:head [delete] snapc 0=[] ondisk+write+known_if_redirected e441)", 2024-04-23T16:39:22.507 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:36:36.847023+0000", 2024-04-23T16:39:22.507 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 165.643084335, 2024-04-23T16:39:22.508 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.068533091000000004, 2024-04-23T16:39:22.508 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.508 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.508 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.508 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:22.508 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:39:22.508 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:39:22.508 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 290062 2024-04-23T16:39:22.508 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.508 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.508 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.508 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.508 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.847023+0000", 2024-04-23T16:39:22.508 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.508 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.509 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.509 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.509 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.847023+0000", 2024-04-23T16:39:22.509 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.509 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.509 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.509 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.509 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.847024+0000", 2024-04-23T16:39:22.509 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2900000000000002e-07 2024-04-23T16:39:22.509 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.509 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.509 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.509 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.847028+0000", 2024-04-23T16:39:22.509 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7409999999999998e-06 2024-04-23T16:39:22.509 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.510 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.510 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.510 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.847047+0000", 2024-04-23T16:39:22.510 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9468000000000001e-05 2024-04-23T16:39:22.510 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.510 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.510 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.510 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.847055+0000", 2024-04-23T16:39:22.510 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.379e-06 2024-04-23T16:39:22.510 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.510 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.510 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.510 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.852177+0000", 2024-04-23T16:39:22.511 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0051222469999999999 2024-04-23T16:39:22.511 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.511 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.511 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.511 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.852294+0000", 2024-04-23T16:39:22.511 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011643599999999999 2024-04-23T16:39:22.511 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.511 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.511 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:39:22.511 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.852409+0000", 2024-04-23T16:39:22.511 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011517400000000001 2024-04-23T16:39:22.511 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.511 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.512 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.512 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.862335+0000", 2024-04-23T16:39:22.512 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0099266790000000008 2024-04-23T16:39:22.512 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.512 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.512 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:39:22.512 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.915481+0000", 2024-04-23T16:39:22.512 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.053145921999999998 2024-04-23T16:39:22.512 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.512 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.512 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.512 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.915520+0000", 2024-04-23T16:39:22.512 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8631999999999999e-05 2024-04-23T16:39:22.512 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.512 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.512 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.513 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.915556+0000", 2024-04-23T16:39:22.513 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6483999999999998e-05 2024-04-23T16:39:22.513 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.513 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.513 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.513 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.513 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.513 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:290064 5.0 5:0a4b04bc:::benchmark_data_smithi188_113957_object136466:head [delete] snapc 0=[] ondisk+write+known_if_redirected e441)", 2024-04-23T16:39:22.513 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:36:36.856601+0000", 2024-04-23T16:39:22.513 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 165.63350702100001, 2024-04-23T16:39:22.513 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.057030312, 2024-04-23T16:39:22.513 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.513 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.513 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.513 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:22.513 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:39:22.514 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:39:22.514 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 290064 2024-04-23T16:39:22.514 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.514 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.514 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.514 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.514 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.856601+0000", 2024-04-23T16:39:22.514 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.514 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.514 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.514 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.514 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.856601+0000", 2024-04-23T16:39:22.514 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.514 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.514 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.515 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.515 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.856603+0000", 2024-04-23T16:39:22.515 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5639999999999999e-06 2024-04-23T16:39:22.515 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.515 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.515 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.515 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.856612+0000", 2024-04-23T16:39:22.515 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6770000000000001e-06 2024-04-23T16:39:22.515 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.515 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.515 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.515 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.856635+0000", 2024-04-23T16:39:22.515 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3031999999999999e-05 2024-04-23T16:39:22.515 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.515 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.516 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.516 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.856645+0000", 2024-04-23T16:39:22.516 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0088000000000001e-05 2024-04-23T16:39:22.516 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.516 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.516 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.516 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.856710+0000", 2024-04-23T16:39:22.516 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4746000000000007e-05 2024-04-23T16:39:22.516 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.516 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.516 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.516 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.856797+0000", 2024-04-23T16:39:22.516 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7268999999999994e-05 2024-04-23T16:39:22.516 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.516 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.516 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:39:22.517 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.856873+0000", 2024-04-23T16:39:22.517 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5669999999999999e-05 2024-04-23T16:39:22.517 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.517 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.517 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.517 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.862388+0000", 2024-04-23T16:39:22.517 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0055157319999999998 2024-04-23T16:39:22.517 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.517 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.517 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:39:22.517 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.913542+0000", 2024-04-23T16:39:22.517 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.051153205 2024-04-23T16:39:22.517 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.517 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.517 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.518 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.913584+0000", 2024-04-23T16:39:22.518 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2530000000000001e-05 2024-04-23T16:39:22.518 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.518 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.518 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.518 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.913631+0000", 2024-04-23T16:39:22.518 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6798999999999997e-05 2024-04-23T16:39:22.518 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.518 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.518 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.518 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.518 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.518 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:290067 5.a e441/394 5:5f91bd63:::benchmark_data_smithi188_113957_object136469:head v 441'18370, mlcod=382'8648)", 2024-04-23T16:39:22.518 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:36:36.856918+0000", 2024-04-23T16:39:22.519 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 165.633189329, 2024-04-23T16:39:22.519 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.054621955999999999, 2024-04-23T16:39:22.519 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.519 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.519 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.519 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.519 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.519 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.519 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.856918+0000", 2024-04-23T16:39:22.519 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.519 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.519 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.519 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.520 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.856918+0000", 2024-04-23T16:39:22.520 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.520 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.520 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.520 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.520 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.856920+0000", 2024-04-23T16:39:22.520 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6670000000000001e-06 2024-04-23T16:39:22.520 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.520 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.520 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.520 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.856928+0000", 2024-04-23T16:39:22.520 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3089999999999996e-06 2024-04-23T16:39:22.520 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.520 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.520 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.521 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.856937+0000", 2024-04-23T16:39:22.521 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8980000000000002e-06 2024-04-23T16:39:22.521 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.521 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.521 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.521 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.856944+0000", 2024-04-23T16:39:22.521 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5010000000000001e-06 2024-04-23T16:39:22.521 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.521 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.521 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.521 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.857165+0000", 2024-04-23T16:39:22.521 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00022128999999999999 2024-04-23T16:39:22.521 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.521 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.522 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.522 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.857187+0000", 2024-04-23T16:39:22.522 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1566e-05 2024-04-23T16:39:22.522 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.522 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.522 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.522 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.911510+0000", 2024-04-23T16:39:22.522 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.054323241000000001 2024-04-23T16:39:22.522 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.522 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.522 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.522 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:36.911540+0000", 2024-04-23T16:39:22.522 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0484000000000002e-05 2024-04-23T16:39:22.523 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.523 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.523 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.523 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.523 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.523 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:290520 5.a e441/394 5:5d55b54a:::benchmark_data_smithi188_113957_object136922:head v 441'18402, mlcod=382'8686)", 2024-04-23T16:39:22.523 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:36:37.397582+0000", 2024-04-23T16:39:22.523 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 165.09252552699999, 2024-04-23T16:39:22.523 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.058738559000000003, 2024-04-23T16:39:22.523 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.523 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.523 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.523 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.523 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.524 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.524 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:37.397582+0000", 2024-04-23T16:39:22.524 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.524 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.524 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.524 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.524 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:37.397582+0000", 2024-04-23T16:39:22.524 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.524 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.524 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.524 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.524 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:37.397583+0000", 2024-04-23T16:39:22.524 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1260000000000001e-06 2024-04-23T16:39:22.525 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.525 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.525 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.525 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:37.397591+0000", 2024-04-23T16:39:22.525 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.9440000000000005e-06 2024-04-23T16:39:22.525 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.525 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.525 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.525 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:37.397600+0000", 2024-04-23T16:39:22.525 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6840000000000002e-06 2024-04-23T16:39:22.525 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.525 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.525 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.525 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:37.397607+0000", 2024-04-23T16:39:22.525 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8909999999999998e-06 2024-04-23T16:39:22.526 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.526 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.526 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.526 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:37.401995+0000", 2024-04-23T16:39:22.526 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0043883630000000002 2024-04-23T16:39:22.526 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.526 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.526 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.526 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:37.402017+0000", 2024-04-23T16:39:22.526 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2033999999999999e-05 2024-04-23T16:39:22.526 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.526 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.526 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.526 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:37.456289+0000", 2024-04-23T16:39:22.527 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.054272267999999999 2024-04-23T16:39:22.527 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.527 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.527 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.527 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:37.456321+0000", 2024-04-23T16:39:22.527 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1248999999999998e-05 2024-04-23T16:39:22.527 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.527 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.527 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.527 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.527 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.527 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:291099 5.c 5:308fc45a:::benchmark_data_smithi188_113957_object137501:head [delete] snapc 0=[] ondisk+write+known_if_redirected e442)", 2024-04-23T16:39:22.527 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:36:38.043885+0000", 2024-04-23T16:39:22.527 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 164.44622276600001, 2024-04-23T16:39:22.527 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.058961527999999999, 2024-04-23T16:39:22.528 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.528 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.528 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.528 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:22.528 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:39:22.528 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:39:22.528 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 291099 2024-04-23T16:39:22.528 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.528 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.528 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.528 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.528 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.043885+0000", 2024-04-23T16:39:22.528 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.528 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.528 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.529 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.529 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.043885+0000", 2024-04-23T16:39:22.529 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.529 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.529 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.529 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.529 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.043886+0000", 2024-04-23T16:39:22.529 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3499999999999996e-07 2024-04-23T16:39:22.529 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.529 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.529 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.529 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.043888+0000", 2024-04-23T16:39:22.529 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5600000000000001e-06 2024-04-23T16:39:22.529 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.529 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.530 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.530 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.043894+0000", 2024-04-23T16:39:22.530 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4559999999999999e-06 2024-04-23T16:39:22.530 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.530 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.530 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.530 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.043898+0000", 2024-04-23T16:39:22.530 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9089999999999999e-06 2024-04-23T16:39:22.530 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.530 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.530 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.530 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.046527+0000", 2024-04-23T16:39:22.530 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0026281429999999999 2024-04-23T16:39:22.530 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.531 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.531 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.531 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.046612+0000", 2024-04-23T16:39:22.531 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5641000000000002e-05 2024-04-23T16:39:22.531 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.531 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.531 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:39:22.531 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.046684+0000", 2024-04-23T16:39:22.531 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1396999999999994e-05 2024-04-23T16:39:22.531 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.531 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.531 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.532 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.054095+0000", 2024-04-23T16:39:22.532 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0074118409999999997 2024-04-23T16:39:22.532 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.532 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.532 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:39:22.532 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.102763+0000", 2024-04-23T16:39:22.532 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048667337999999997 2024-04-23T16:39:22.532 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.532 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.532 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.532 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.102806+0000", 2024-04-23T16:39:22.532 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3322000000000003e-05 2024-04-23T16:39:22.532 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.532 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.532 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.533 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.102846+0000", 2024-04-23T16:39:22.533 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0286e-05 2024-04-23T16:39:22.533 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.533 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.533 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.533 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.533 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.533 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.25717.0:291101 5.c 5:3fd41aa5:::benchmark_data_smithi188_113957_object137503:head [delete] snapc 0=[] ondisk+write+known_if_redirected e442)", 2024-04-23T16:39:22.533 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:36:38.044802+0000", 2024-04-23T16:39:22.533 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 164.44530597299999, 2024-04-23T16:39:22.533 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.058451008999999998, 2024-04-23T16:39:22.533 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.533 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.533 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.533 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:22.534 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.25717", 2024-04-23T16:39:22.534 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:39:22.534 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 291101 2024-04-23T16:39:22.534 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.534 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.534 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.534 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.534 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.044802+0000", 2024-04-23T16:39:22.534 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.534 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.534 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.534 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.534 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.044802+0000", 2024-04-23T16:39:22.535 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.535 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.535 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.535 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.535 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.044803+0000", 2024-04-23T16:39:22.535 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.328e-06 2024-04-23T16:39:22.535 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.535 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.535 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.535 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.044808+0000", 2024-04-23T16:39:22.535 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.561e-06 2024-04-23T16:39:22.535 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.535 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.535 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.536 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.044816+0000", 2024-04-23T16:39:22.536 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5839999999999995e-06 2024-04-23T16:39:22.536 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.536 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.536 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.536 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.044823+0000", 2024-04-23T16:39:22.536 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7830000000000001e-06 2024-04-23T16:39:22.536 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.536 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.536 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.536 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.050272+0000", 2024-04-23T16:39:22.536 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0054487299999999997 2024-04-23T16:39:22.536 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.537 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.537 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.537 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.050357+0000", 2024-04-23T16:39:22.537 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5677e-05 2024-04-23T16:39:22.537 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.537 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.537 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 0", 2024-04-23T16:39:22.537 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.050430+0000", 2024-04-23T16:39:22.537 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2248000000000004e-05 2024-04-23T16:39:22.537 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.537 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.537 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:39:22.537 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.102856+0000", 2024-04-23T16:39:22.537 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052426457000000003 2024-04-23T16:39:22.538 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.538 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.538 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.538 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.103167+0000", 2024-04-23T16:39:22.538 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00031110099999999999 2024-04-23T16:39:22.538 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.538 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.538 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.538 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.103214+0000", 2024-04-23T16:39:22.538 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6909000000000001e-05 2024-04-23T16:39:22.538 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.538 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.538 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.539 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.103253+0000", 2024-04-23T16:39:22.539 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8631000000000003e-05 2024-04-23T16:39:22.539 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.539 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.539 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.539 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.539 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.539 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.25717.0:291112 5.a e442/394 5:59252f95:::benchmark_data_smithi188_113957_object137514:head v 442'18429, mlcod=382'8726)", 2024-04-23T16:39:22.539 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:36:38.056784+0000", 2024-04-23T16:39:22.539 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 164.433323253, 2024-04-23T16:39:22.539 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.067959377000000001, 2024-04-23T16:39:22.539 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.539 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.539 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.539 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.539 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.540 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.540 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.056784+0000", 2024-04-23T16:39:22.540 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.540 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.540 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.540 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.540 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.056784+0000", 2024-04-23T16:39:22.540 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.540 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.540 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.540 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.540 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.056786+0000", 2024-04-23T16:39:22.540 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1510000000000001e-06 2024-04-23T16:39:22.540 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.540 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.541 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.541 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.056792+0000", 2024-04-23T16:39:22.541 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5339999999999996e-06 2024-04-23T16:39:22.541 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.541 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.541 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.541 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.102939+0000", 2024-04-23T16:39:22.541 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.046146752999999999 2024-04-23T16:39:22.541 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.541 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.541 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.541 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.102949+0000", 2024-04-23T16:39:22.541 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0407e-05 2024-04-23T16:39:22.541 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.541 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.542 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.542 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.107397+0000", 2024-04-23T16:39:22.542 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.004447482 2024-04-23T16:39:22.542 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.542 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.542 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.542 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.107420+0000", 2024-04-23T16:39:22.542 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3442e-05 2024-04-23T16:39:22.542 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.542 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.542 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.542 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.124710+0000", 2024-04-23T16:39:22.542 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.017290307000000001 2024-04-23T16:39:22.542 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.542 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.543 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.543 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:36:38.124744+0000", 2024-04-23T16:39:22.543 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3300999999999998e-05 2024-04-23T16:39:22.543 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.543 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.543 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.543 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.543 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.543 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:84217 6.e 6:7c4863cd:::benchmark_data_smithi188_130221_object84216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e463)", 2024-04-23T16:39:22.543 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:38:02.978188+0000", 2024-04-23T16:39:22.543 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 79.511919442999996, 2024-04-23T16:39:22.543 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.061388983000000001, 2024-04-23T16:39:22.543 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.543 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.543 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.543 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:22.544 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:39:22.544 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:39:22.544 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 84217 2024-04-23T16:39:22.544 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.544 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.544 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.544 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.544 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:02.978188+0000", 2024-04-23T16:39:22.544 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.544 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.544 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.544 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.544 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:02.978188+0000", 2024-04-23T16:39:22.544 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.544 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.544 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.545 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.545 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:02.978191+0000", 2024-04-23T16:39:22.545 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.79e-06 2024-04-23T16:39:22.545 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.545 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.545 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.545 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:02.978243+0000", 2024-04-23T16:39:22.545 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1717e-05 2024-04-23T16:39:22.545 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.545 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.545 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.545 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:02.978252+0000", 2024-04-23T16:39:22.545 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2529999999999993e-06 2024-04-23T16:39:22.545 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.545 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.545 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.546 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:02.978258+0000", 2024-04-23T16:39:22.546 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2709999999999998e-06 2024-04-23T16:39:22.546 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.546 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.546 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.546 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:02.978308+0000", 2024-04-23T16:39:22.546 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0195999999999998e-05 2024-04-23T16:39:22.546 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.546 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.546 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.546 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:02.978423+0000", 2024-04-23T16:39:22.546 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000114889 2024-04-23T16:39:22.546 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.546 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.546 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 5", 2024-04-23T16:39:22.547 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:02.978533+0000", 2024-04-23T16:39:22.547 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000109541 2024-04-23T16:39:22.547 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.547 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.547 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:39:22.547 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:02.980955+0000", 2024-04-23T16:39:22.547 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0024219519999999998 2024-04-23T16:39:22.547 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.547 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.547 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.547 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:03.039425+0000", 2024-04-23T16:39:22.547 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.058470138999999997 2024-04-23T16:39:22.547 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.547 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.547 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.548 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:03.039521+0000", 2024-04-23T16:39:22.548 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6100000000000005e-05 2024-04-23T16:39:22.548 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.548 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.548 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.548 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:03.039577+0000", 2024-04-23T16:39:22.548 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6134999999999997e-05 2024-04-23T16:39:22.548 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.548 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.548 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.548 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.548 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.548 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:102950 6.2 6:491887b1:::benchmark_data_smithi188_130221_object102949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e465)", 2024-04-23T16:39:22.548 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:38:14.997614+0000", 2024-04-23T16:39:22.549 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 67.492493177, 2024-04-23T16:39:22.549 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.056686131000000001, 2024-04-23T16:39:22.549 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.549 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.549 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.549 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:22.549 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:39:22.549 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:39:22.549 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 102950 2024-04-23T16:39:22.549 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.549 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.549 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.549 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.549 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:14.997614+0000", 2024-04-23T16:39:22.549 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.550 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.550 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.550 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.550 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:14.997614+0000", 2024-04-23T16:39:22.550 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.550 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.550 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.550 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.550 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:14.997617+0000", 2024-04-23T16:39:22.550 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0940000000000002e-06 2024-04-23T16:39:22.550 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.550 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.550 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.550 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:14.997684+0000", 2024-04-23T16:39:22.550 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7218999999999995e-05 2024-04-23T16:39:22.551 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.551 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.551 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.551 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:14.997692+0000", 2024-04-23T16:39:22.551 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1710000000000002e-06 2024-04-23T16:39:22.551 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.551 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.551 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.551 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:14.997697+0000", 2024-04-23T16:39:22.551 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.091e-06 2024-04-23T16:39:22.551 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.551 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.551 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.551 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:14.997739+0000", 2024-04-23T16:39:22.551 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2055999999999998e-05 2024-04-23T16:39:22.552 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.552 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.552 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.552 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:14.997839+0000", 2024-04-23T16:39:22.552 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9722999999999994e-05 2024-04-23T16:39:22.552 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.552 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.552 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:39:22.552 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:14.997946+0000", 2024-04-23T16:39:22.552 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107124 2024-04-23T16:39:22.552 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.552 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.552 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:39:22.552 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:15.001637+0000", 2024-04-23T16:39:22.552 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003690586 2024-04-23T16:39:22.552 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.553 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.553 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.553 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:15.054195+0000", 2024-04-23T16:39:22.553 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052558227999999999 2024-04-23T16:39:22.553 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.553 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.553 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.553 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:15.054254+0000", 2024-04-23T16:39:22.553 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9481999999999997e-05 2024-04-23T16:39:22.553 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.553 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.553 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.553 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:15.054301+0000", 2024-04-23T16:39:22.553 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6356999999999997e-05 2024-04-23T16:39:22.553 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.553 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.554 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.554 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.554 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.554 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:105312 6.2 6:4526a039:::benchmark_data_smithi188_130221_object105311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e465)", 2024-04-23T16:39:22.554 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:38:16.727614+0000", 2024-04-23T16:39:22.554 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 65.762493191999994, 2024-04-23T16:39:22.554 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.056919826999999999, 2024-04-23T16:39:22.554 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.554 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.554 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.554 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:22.554 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:39:22.554 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:39:22.554 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 105312 2024-04-23T16:39:22.555 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.555 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.555 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.555 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.555 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:16.727614+0000", 2024-04-23T16:39:22.555 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.555 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.555 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.555 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.555 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:16.727614+0000", 2024-04-23T16:39:22.555 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.555 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.555 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.555 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.555 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:16.727615+0000", 2024-04-23T16:39:22.555 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7800000000000001e-07 2024-04-23T16:39:22.555 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.556 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.556 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.556 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:16.727636+0000", 2024-04-23T16:39:22.556 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0842000000000001e-05 2024-04-23T16:39:22.556 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.556 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.556 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.556 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:16.727640+0000", 2024-04-23T16:39:22.556 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6679999999999999e-06 2024-04-23T16:39:22.556 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.556 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.556 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.556 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:16.727643+0000", 2024-04-23T16:39:22.556 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.332e-06 2024-04-23T16:39:22.556 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.557 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.557 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.557 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:16.733526+0000", 2024-04-23T16:39:22.557 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0058830009999999997 2024-04-23T16:39:22.557 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.557 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.557 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.557 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:16.733649+0000", 2024-04-23T16:39:22.557 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012301799999999999 2024-04-23T16:39:22.557 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.557 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.557 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:39:22.557 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:16.733740+0000", 2024-04-23T16:39:22.557 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1076999999999996e-05 2024-04-23T16:39:22.558 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.558 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.558 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:39:22.558 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:16.742502+0000", 2024-04-23T16:39:22.558 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0087620249999999997 2024-04-23T16:39:22.558 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.558 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.558 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.558 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:16.784448+0000", 2024-04-23T16:39:22.558 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.041945620000000003 2024-04-23T16:39:22.558 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.558 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.558 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.558 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:16.784496+0000", 2024-04-23T16:39:22.559 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8705000000000002e-05 2024-04-23T16:39:22.559 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.559 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.559 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.559 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:16.784534+0000", 2024-04-23T16:39:22.559 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7960999999999998e-05 2024-04-23T16:39:22.559 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.559 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.559 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.559 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.559 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.559 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:105998 6.2 6:4050af33:::benchmark_data_smithi188_130221_object105997:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e466)", 2024-04-23T16:39:22.559 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:38:17.284262+0000", 2024-04-23T16:39:22.559 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 65.205845358999994, 2024-04-23T16:39:22.559 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.067230376999999994, 2024-04-23T16:39:22.559 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.560 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.560 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.560 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:22.560 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:39:22.560 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:39:22.560 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 105998 2024-04-23T16:39:22.560 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.560 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.560 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.560 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.560 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:17.284262+0000", 2024-04-23T16:39:22.560 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.560 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.560 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.560 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.560 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:17.284262+0000", 2024-04-23T16:39:22.561 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.561 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.561 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.561 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.561 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:17.284264+0000", 2024-04-23T16:39:22.561 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.911e-06 2024-04-23T16:39:22.561 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.561 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.561 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.561 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:17.284334+0000", 2024-04-23T16:39:22.561 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9400999999999995e-05 2024-04-23T16:39:22.561 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.561 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.561 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.562 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:17.284345+0000", 2024-04-23T16:39:22.562 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1299e-05 2024-04-23T16:39:22.562 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.562 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.562 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.562 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:17.284353+0000", 2024-04-23T16:39:22.562 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0600000000000008e-06 2024-04-23T16:39:22.562 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.562 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.562 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.562 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:17.284407+0000", 2024-04-23T16:39:22.565 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3947999999999999e-05 2024-04-23T16:39:22.565 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.565 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.565 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.565 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:17.284525+0000", 2024-04-23T16:39:22.565 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000117787 2024-04-23T16:39:22.566 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.566 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.566 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 3", 2024-04-23T16:39:22.566 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:17.284625+0000", 2024-04-23T16:39:22.566 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010033400000000001 2024-04-23T16:39:22.566 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.566 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.566 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:39:22.566 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:17.287405+0000", 2024-04-23T16:39:22.566 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0027802 2024-04-23T16:39:22.566 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.566 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.566 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.567 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:17.351380+0000", 2024-04-23T16:39:22.567 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.063974302999999996 2024-04-23T16:39:22.567 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.567 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.567 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.567 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:17.351439+0000", 2024-04-23T16:39:22.567 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9954999999999998e-05 2024-04-23T16:39:22.567 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.567 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.567 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.567 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:38:17.351493+0000", 2024-04-23T16:39:22.567 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3179e-05 2024-04-23T16:39:22.567 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.568 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.568 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.568 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.568 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.568 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:156106 6.e 6:7c95864e:::benchmark_data_smithi188_130221_object1645:head [delete] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-23T16:39:22.568 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:39:03.004322+0000", 2024-04-23T16:39:22.568 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.485785303, 2024-04-23T16:39:22.568 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.065400229000000004, 2024-04-23T16:39:22.568 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.568 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.568 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.568 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:22.568 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:39:22.569 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:39:22.569 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 156106 2024-04-23T16:39:22.569 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.569 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.569 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.569 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.569 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.004322+0000", 2024-04-23T16:39:22.569 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.569 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.569 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.569 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.569 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.004322+0000", 2024-04-23T16:39:22.569 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.570 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.570 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.570 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.570 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.004325+0000", 2024-04-23T16:39:22.570 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4049999999999998e-06 2024-04-23T16:39:22.570 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.570 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.570 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.570 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.004331+0000", 2024-04-23T16:39:22.570 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8549999999999998e-06 2024-04-23T16:39:22.570 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.570 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.570 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.571 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.004341+0000", 2024-04-23T16:39:22.571 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9890000000000004e-06 2024-04-23T16:39:22.571 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.571 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.571 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.571 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.004349+0000", 2024-04-23T16:39:22.571 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4959999999999996e-06 2024-04-23T16:39:22.571 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.571 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.571 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.571 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.004411+0000", 2024-04-23T16:39:22.571 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.211e-05 2024-04-23T16:39:22.571 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.571 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.571 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.572 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.004536+0000", 2024-04-23T16:39:22.572 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012432 2024-04-23T16:39:22.572 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.572 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.572 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 5", 2024-04-23T16:39:22.572 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.004632+0000", 2024-04-23T16:39:22.572 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6742000000000003e-05 2024-04-23T16:39:22.572 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.572 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.572 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:39:22.572 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.019099+0000", 2024-04-23T16:39:22.572 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014466665 2024-04-23T16:39:22.572 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.572 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.572 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.573 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.069618+0000", 2024-04-23T16:39:22.573 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.050519214999999999 2024-04-23T16:39:22.573 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.573 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.573 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.573 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.069672+0000", 2024-04-23T16:39:22.573 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3597999999999997e-05 2024-04-23T16:39:22.573 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.573 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.573 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.573 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.069723+0000", 2024-04-23T16:39:22.573 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0834000000000002e-05 2024-04-23T16:39:22.573 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.573 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.574 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.574 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.574 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.574 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:156110 6.e 6:7b197472:::benchmark_data_smithi188_130221_object1649:head [delete] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-23T16:39:22.574 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:39:03.009467+0000", 2024-04-23T16:39:22.574 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.480640205, 2024-04-23T16:39:22.574 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.060918708000000002, 2024-04-23T16:39:22.574 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.574 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.574 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.574 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:22.574 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:39:22.574 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:39:22.575 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 156110 2024-04-23T16:39:22.575 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.575 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.575 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.575 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.575 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.009467+0000", 2024-04-23T16:39:22.575 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.575 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.575 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.575 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.575 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.009467+0000", 2024-04-23T16:39:22.575 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.575 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.575 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.575 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.576 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.009470+0000", 2024-04-23T16:39:22.576 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2110000000000001e-06 2024-04-23T16:39:22.576 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.576 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.576 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.576 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.009476+0000", 2024-04-23T16:39:22.576 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3350000000000003e-06 2024-04-23T16:39:22.576 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.576 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.576 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.576 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.009488+0000", 2024-04-23T16:39:22.576 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1970999999999999e-05 2024-04-23T16:39:22.576 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.576 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.576 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.577 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.009496+0000", 2024-04-23T16:39:22.577 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3680000000000002e-06 2024-04-23T16:39:22.577 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.577 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.577 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.577 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.013948+0000", 2024-04-23T16:39:22.577 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0044520009999999997 2024-04-23T16:39:22.577 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.577 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.577 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.577 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.014088+0000", 2024-04-23T16:39:22.577 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000139865 2024-04-23T16:39:22.577 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.577 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.577 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 5", 2024-04-23T16:39:22.578 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.014200+0000", 2024-04-23T16:39:22.578 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000111328 2024-04-23T16:39:22.578 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.578 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.578 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:39:22.578 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.019248+0000", 2024-04-23T16:39:22.578 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0050485690000000001 2024-04-23T16:39:22.578 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.578 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.578 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.578 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.070297+0000", 2024-04-23T16:39:22.578 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.051048431999999998 2024-04-23T16:39:22.578 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.579 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.579 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.579 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.070347+0000", 2024-04-23T16:39:22.579 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0667999999999997e-05 2024-04-23T16:39:22.579 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.579 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.579 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.579 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:03.070386+0000", 2024-04-23T16:39:22.579 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.896e-05 2024-04-23T16:39:22.579 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.579 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.579 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.579 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.579 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.579 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:173387 6.e 6:7cc917ba:::benchmark_data_smithi188_130221_object18926:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e489)", 2024-04-23T16:39:22.580 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:39:21.935652+0000", 2024-04-23T16:39:22.580 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.55445530899999995, 2024-04-23T16:39:22.580 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20711752999999999, 2024-04-23T16:39:22.580 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.580 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.580 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.580 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:22.580 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:39:22.580 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:39:22.580 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 173387 2024-04-23T16:39:22.580 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.580 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.580 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.580 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.580 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935652+0000", 2024-04-23T16:39:22.580 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.581 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.581 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.581 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.581 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935652+0000", 2024-04-23T16:39:22.581 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.581 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.581 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.581 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.581 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935655+0000", 2024-04-23T16:39:22.581 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2060000000000001e-06 2024-04-23T16:39:22.581 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.581 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.581 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.581 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935660+0000", 2024-04-23T16:39:22.582 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7960000000000001e-06 2024-04-23T16:39:22.582 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.582 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.582 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.582 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935673+0000", 2024-04-23T16:39:22.582 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2741999999999999e-05 2024-04-23T16:39:22.582 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.582 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.582 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.582 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935683+0000", 2024-04-23T16:39:22.582 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0002e-05 2024-04-23T16:39:22.582 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.582 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.582 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.583 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935735+0000", 2024-04-23T16:39:22.583 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2249000000000001e-05 2024-04-23T16:39:22.583 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.583 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.583 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:39:22.583 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935745+0000", 2024-04-23T16:39:22.583 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.73e-06 2024-04-23T16:39:22.583 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.583 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.583 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.583 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.128395+0000", 2024-04-23T16:39:22.583 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19265043400000001 2024-04-23T16:39:22.583 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.583 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.583 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.584 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.128503+0000", 2024-04-23T16:39:22.584 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107508 2024-04-23T16:39:22.584 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.584 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.584 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 1,5", 2024-04-23T16:39:22.584 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.128598+0000", 2024-04-23T16:39:22.584 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.4862000000000001e-05 2024-04-23T16:39:22.584 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.584 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.584 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.584 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.132780+0000", 2024-04-23T16:39:22.584 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0041825990000000004 2024-04-23T16:39:22.584 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.584 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.584 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:39:22.585 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142388+0000", 2024-04-23T16:39:22.585 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0096078719999999999 2024-04-23T16:39:22.585 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.585 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.585 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.585 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142628+0000", 2024-04-23T16:39:22.585 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000239886 2024-04-23T16:39:22.585 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.585 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.585 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.585 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142692+0000", 2024-04-23T16:39:22.585 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3414e-05 2024-04-23T16:39:22.585 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.586 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.586 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.586 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142770+0000", 2024-04-23T16:39:22.586 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8230000000000001e-05 2024-04-23T16:39:22.586 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.586 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.586 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.586 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.586 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.586 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:173393 6.e 6:73e3fd60:::benchmark_data_smithi188_130221_object18932:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e489)", 2024-04-23T16:39:22.586 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:39:21.935705+0000", 2024-04-23T16:39:22.586 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.55440246800000004, 2024-04-23T16:39:22.586 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20725584, 2024-04-23T16:39:22.586 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.587 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.587 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.587 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:22.587 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:39:22.587 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:39:22.587 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 173393 2024-04-23T16:39:22.587 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.587 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.587 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.587 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.587 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935705+0000", 2024-04-23T16:39:22.587 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.587 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.587 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.587 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.587 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935705+0000", 2024-04-23T16:39:22.588 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.588 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.588 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.588 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.588 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935706+0000", 2024-04-23T16:39:22.588 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6300000000000004e-07 2024-04-23T16:39:22.588 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.588 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.588 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.588 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935710+0000", 2024-04-23T16:39:22.588 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6270000000000002e-06 2024-04-23T16:39:22.588 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.588 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.588 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.589 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935716+0000", 2024-04-23T16:39:22.589 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6590000000000001e-06 2024-04-23T16:39:22.589 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.589 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.589 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.589 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935723+0000", 2024-04-23T16:39:22.589 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3829999999999997e-06 2024-04-23T16:39:22.589 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.589 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.589 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.589 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935789+0000", 2024-04-23T16:39:22.589 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6511000000000001e-05 2024-04-23T16:39:22.589 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.589 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.590 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:39:22.590 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935799+0000", 2024-04-23T16:39:22.590 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8560000000000002e-06 2024-04-23T16:39:22.590 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.590 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.590 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.590 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.132835+0000", 2024-04-23T16:39:22.590 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19703553500000001 2024-04-23T16:39:22.590 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.590 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.590 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.590 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.132984+0000", 2024-04-23T16:39:22.590 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000149319 2024-04-23T16:39:22.590 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.590 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.591 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 1,5", 2024-04-23T16:39:22.591 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.133139+0000", 2024-04-23T16:39:22.591 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015512800000000001 2024-04-23T16:39:22.591 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.591 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.591 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:39:22.591 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142778+0000", 2024-04-23T16:39:22.591 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.009639356 2024-04-23T16:39:22.591 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.591 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.591 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.591 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142816+0000", 2024-04-23T16:39:22.591 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7710999999999999e-05 2024-04-23T16:39:22.591 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.591 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.592 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.592 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142864+0000", 2024-04-23T16:39:22.592 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8038000000000003e-05 2024-04-23T16:39:22.592 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.592 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.592 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.592 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142895+0000", 2024-04-23T16:39:22.592 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1092e-05 2024-04-23T16:39:22.592 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.592 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.592 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.592 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142961+0000", 2024-04-23T16:39:22.592 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5761999999999998e-05 2024-04-23T16:39:22.592 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.592 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.593 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.593 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.593 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.593 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:173402 6.e 6:71c0b42e:::benchmark_data_smithi188_130221_object18941:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e489)", 2024-04-23T16:39:22.593 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:39:21.935731+0000", 2024-04-23T16:39:22.593 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.554377075, 2024-04-23T16:39:22.593 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.207581344, 2024-04-23T16:39:22.593 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.593 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.593 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.593 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:22.593 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:39:22.593 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:39:22.593 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 173402 2024-04-23T16:39:22.594 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.594 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.594 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.594 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.594 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935731+0000", 2024-04-23T16:39:22.594 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.594 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.594 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.594 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.594 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935731+0000", 2024-04-23T16:39:22.594 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.594 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.594 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.595 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.595 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935731+0000", 2024-04-23T16:39:22.595 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4600000000000005e-07 2024-04-23T16:39:22.595 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.595 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.595 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.595 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935734+0000", 2024-04-23T16:39:22.595 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7800000000000001e-06 2024-04-23T16:39:22.595 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.595 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.595 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.595 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935740+0000", 2024-04-23T16:39:22.595 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6840000000000001e-06 2024-04-23T16:39:22.596 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.596 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.596 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.596 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935746+0000", 2024-04-23T16:39:22.596 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0549999999999996e-06 2024-04-23T16:39:22.596 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.596 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.596 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.596 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935857+0000", 2024-04-23T16:39:22.596 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000111638 2024-04-23T16:39:22.596 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.596 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.596 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:39:22.596 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935867+0000", 2024-04-23T16:39:22.596 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8309999999999996e-06 2024-04-23T16:39:22.596 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.597 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.597 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.597 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.138168+0000", 2024-04-23T16:39:22.597 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20230051600000001 2024-04-23T16:39:22.597 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.597 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.597 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.597 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.138324+0000", 2024-04-23T16:39:22.597 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015627699999999999 2024-04-23T16:39:22.597 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.597 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.597 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 1,5", 2024-04-23T16:39:22.597 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.138498+0000", 2024-04-23T16:39:22.597 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000173616 2024-04-23T16:39:22.597 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.597 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.598 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:39:22.598 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142970+0000", 2024-04-23T16:39:22.598 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0044722119999999997 2024-04-23T16:39:22.598 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.598 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.598 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.598 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.143014+0000", 2024-04-23T16:39:22.598 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3940000000000003e-05 2024-04-23T16:39:22.598 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.598 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.598 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.598 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.143159+0000", 2024-04-23T16:39:22.598 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014494700000000001 2024-04-23T16:39:22.599 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.599 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.599 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.599 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.143216+0000", 2024-04-23T16:39:22.599 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7194000000000002e-05 2024-04-23T16:39:22.599 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.599 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.599 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.599 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.143312+0000", 2024-04-23T16:39:22.599 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6107999999999995e-05 2024-04-23T16:39:22.599 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.599 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.599 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.599 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.599 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.600 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:173404 6.e 6:767f53bf:::benchmark_data_smithi188_130221_object18943:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e489)", 2024-04-23T16:39:22.600 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:39:21.935780+0000", 2024-04-23T16:39:22.600 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.55432716599999998, 2024-04-23T16:39:22.600 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.22186477900000001, 2024-04-23T16:39:22.600 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.600 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.600 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.600 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:22.600 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:39:22.600 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:39:22.600 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 173404 2024-04-23T16:39:22.600 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.600 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.600 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.600 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.601 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935780+0000", 2024-04-23T16:39:22.601 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.601 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.601 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.601 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.601 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935780+0000", 2024-04-23T16:39:22.601 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.601 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.601 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.601 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.601 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935782+0000", 2024-04-23T16:39:22.601 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1599999999999999e-06 2024-04-23T16:39:22.601 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.601 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.601 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.601 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935786+0000", 2024-04-23T16:39:22.602 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7099999999999998e-06 2024-04-23T16:39:22.602 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.602 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.602 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.602 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935794+0000", 2024-04-23T16:39:22.602 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0460000000000006e-06 2024-04-23T16:39:22.602 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.602 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.602 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.602 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935800+0000", 2024-04-23T16:39:22.602 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1630000000000004e-06 2024-04-23T16:39:22.602 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.602 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.602 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.602 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935935+0000", 2024-04-23T16:39:22.603 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013570699999999999 2024-04-23T16:39:22.603 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.603 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.603 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:39:22.603 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935942+0000", 2024-04-23T16:39:22.603 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.866e-06 2024-04-23T16:39:22.603 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.603 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.603 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.603 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.143493+0000", 2024-04-23T16:39:22.603 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20755101000000001 2024-04-23T16:39:22.603 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.603 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.603 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.603 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.143625+0000", 2024-04-23T16:39:22.604 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013207200000000001 2024-04-23T16:39:22.604 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.604 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.604 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 1,5", 2024-04-23T16:39:22.604 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.143709+0000", 2024-04-23T16:39:22.604 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3735999999999999e-05 2024-04-23T16:39:22.604 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.605 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.605 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.605 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157306+0000", 2024-04-23T16:39:22.605 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013596924 2024-04-23T16:39:22.605 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.605 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.605 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.605 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157383+0000", 2024-04-23T16:39:22.605 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7142000000000001e-05 2024-04-23T16:39:22.605 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.605 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.605 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:39:22.605 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157471+0000", 2024-04-23T16:39:22.606 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8399000000000005e-05 2024-04-23T16:39:22.606 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.606 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.606 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.606 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157532+0000", 2024-04-23T16:39:22.606 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0378999999999999e-05 2024-04-23T16:39:22.606 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.606 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.606 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.606 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157645+0000", 2024-04-23T16:39:22.606 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000113465 2024-04-23T16:39:22.606 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.606 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.606 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.607 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.607 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.607 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:173424 6.e 6:721bacbd:::benchmark_data_smithi188_130221_object18963:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e489)", 2024-04-23T16:39:22.607 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:39:21.935810+0000", 2024-04-23T16:39:22.607 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.55429782900000002, 2024-04-23T16:39:22.607 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.221983126, 2024-04-23T16:39:22.607 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:22.607 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:22.607 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:39:22.607 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:22.607 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:39:22.607 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:39:22.607 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 173424 2024-04-23T16:39:22.607 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.607 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:22.607 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.608 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:22.608 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935810+0000", 2024-04-23T16:39:22.608 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.608 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.608 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.608 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:22.608 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935810+0000", 2024-04-23T16:39:22.608 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:22.608 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.608 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.608 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:22.608 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935810+0000", 2024-04-23T16:39:22.608 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.68e-07 2024-04-23T16:39:22.609 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.609 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.609 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:22.609 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935813+0000", 2024-04-23T16:39:22.609 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.559e-06 2024-04-23T16:39:22.609 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.609 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.609 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:22.609 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935817+0000", 2024-04-23T16:39:22.609 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.822e-06 2024-04-23T16:39:22.609 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.609 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.609 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:22.609 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935822+0000", 2024-04-23T16:39:22.610 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2089999999999996e-06 2024-04-23T16:39:22.610 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.610 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.610 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.610 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935964+0000", 2024-04-23T16:39:22.610 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014171 2024-04-23T16:39:22.610 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.610 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.610 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:39:22.610 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935970+0000", 2024-04-23T16:39:22.610 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6959999999999999e-06 2024-04-23T16:39:22.610 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.610 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.610 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:22.611 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.147821+0000", 2024-04-23T16:39:22.611 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.211850487 2024-04-23T16:39:22.611 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.611 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.611 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:22.611 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.147950+0000", 2024-04-23T16:39:22.611 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000129469 2024-04-23T16:39:22.611 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.611 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.611 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 1,5", 2024-04-23T16:39:22.611 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.148069+0000", 2024-04-23T16:39:22.611 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000118879 2024-04-23T16:39:22.611 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.611 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.611 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.612 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157455+0000", 2024-04-23T16:39:22.612 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00938604 2024-04-23T16:39:22.612 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.612 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.612 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:39:22.612 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157655+0000", 2024-04-23T16:39:22.612 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00019966600000000001 2024-04-23T16:39:22.612 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.612 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.612 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:22.612 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157690+0000", 2024-04-23T16:39:22.612 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4844e-05 2024-04-23T16:39:22.612 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.612 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.613 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:39:22.613 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157726+0000", 2024-04-23T16:39:22.613 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5672e-05 2024-04-23T16:39:22.613 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:22.613 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:22.613 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:39:22.613 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157793+0000", 2024-04-23T16:39:22.613 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7404999999999996e-05 2024-04-23T16:39:22.613 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.613 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.613 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.613 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:22.613 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:22.613 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:39:22.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:22.614 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:22.615 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:39:22.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:39:22.715 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:39:22.743 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:22.762 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:39:22.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:22.767 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:39:22.868 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:39:22.914 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:22.977 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:39:22.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:22.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:39:23.083 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:39:23.124 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:22 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:39:23.124 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:22 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:39:23.124 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:22 smithi103 ceph-mon[39629]: osdmap e489: 12 total, 12 up, 10 in 2024-04-23T16:39:23.125 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:22 smithi103 ceph-mon[39629]: 6.d starting backfill to osd.1 from (0'0,0'0] MAX to 486'10670 2024-04-23T16:39:23.125 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:22 smithi103 ceph-mon[39629]: 6.e starting backfill to osd.1 from (0'0,0'0] MAX to 486'10780 2024-04-23T16:39:23.125 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:22 smithi103 ceph-mon[39629]: 6.4 starting backfill to osd.1 from (0'0,0'0] MAX to 486'10847 2024-04-23T16:39:23.128 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:23.192 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:39:23.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:23.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:39:23.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:22 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:39:23.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:22 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:39:23.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:22 smithi188 ceph-mon[44101]: osdmap e489: 12 total, 12 up, 10 in 2024-04-23T16:39:23.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:22 smithi188 ceph-mon[44101]: 6.d starting backfill to osd.1 from (0'0,0'0] MAX to 486'10670 2024-04-23T16:39:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:22 smithi188 ceph-mon[44101]: 6.e starting backfill to osd.1 from (0'0,0'0] MAX to 486'10780 2024-04-23T16:39:23.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:22 smithi188 ceph-mon[44101]: 6.4 starting backfill to osd.1 from (0'0,0'0] MAX to 486'10847 2024-04-23T16:39:23.301 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:39:23.328 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:23.341 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:22 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:39:23.341 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:22 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:39:23.341 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:22 smithi148 ceph-mon[44520]: osdmap e489: 12 total, 12 up, 10 in 2024-04-23T16:39:23.341 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:22 smithi148 ceph-mon[44520]: 6.d starting backfill to osd.1 from (0'0,0'0] MAX to 486'10670 2024-04-23T16:39:23.341 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:22 smithi148 ceph-mon[44520]: 6.e starting backfill to osd.1 from (0'0,0'0] MAX to 486'10780 2024-04-23T16:39:23.342 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:22 smithi148 ceph-mon[44520]: 6.4 starting backfill to osd.1 from (0'0,0'0] MAX to 486'10847 2024-04-23T16:39:23.350 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:39:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:23.353 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:39:23.379 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-23T16:39:23.454 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:39:23.481 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:23.502 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:39:23.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:23.505 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:39:23.606 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:39:23.694 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:23.723 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:39:23.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:23.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:39:23.828 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:39:23.868 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:39:23.871 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:23.965 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:39:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:23 smithi103 ceph-mon[39629]: pgmap v1383: 27 pgs: 2 unknown, 1 peering, 24 active+clean; 7.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail 2024-04-23T16:39:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:23 smithi103 ceph-mon[39629]: osdmap e490: 12 total, 12 up, 10 in 2024-04-23T16:39:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:23 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:39:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:23 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:39:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:23 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1561750205' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:39:24.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:23 smithi103 ceph-mon[39629]: from='client.26362 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:39:24.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:23 smithi188 ceph-mon[44101]: pgmap v1383: 27 pgs: 2 unknown, 1 peering, 24 active+clean; 7.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail 2024-04-23T16:39:24.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:23 smithi188 ceph-mon[44101]: osdmap e490: 12 total, 12 up, 10 in 2024-04-23T16:39:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:23 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:39:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:23 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:39:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:23 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1561750205' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:39:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:23 smithi188 ceph-mon[44101]: from='client.26362 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:39:24.235 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:23 smithi148 ceph-mon[44520]: pgmap v1383: 27 pgs: 2 unknown, 1 peering, 24 active+clean; 7.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail 2024-04-23T16:39:24.235 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:23 smithi148 ceph-mon[44520]: osdmap e490: 12 total, 12 up, 10 in 2024-04-23T16:39:24.235 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:23 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:39:24.236 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:23 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:39:24.236 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:23 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1561750205' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:39:24.236 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:23 smithi148 ceph-mon[44520]: from='client.26362 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:39:25.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:24 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:39:25.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:24 smithi103 ceph-mon[39629]: from='client.26362 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:39:25.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:24 smithi103 ceph-mon[39629]: osdmap e491: 12 total, 12 up, 10 in 2024-04-23T16:39:25.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:24 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:39:25.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:24 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:39:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:24 smithi188 ceph-mon[44101]: from='client.26362 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:39:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:24 smithi188 ceph-mon[44101]: osdmap e491: 12 total, 12 up, 10 in 2024-04-23T16:39:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:24 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:39:25.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:24 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:39:25.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:24 smithi148 ceph-mon[44520]: from='client.26362 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:39:25.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:24 smithi148 ceph-mon[44520]: osdmap e491: 12 total, 12 up, 10 in 2024-04-23T16:39:25.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:24 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:39:25.443 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:39:25.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:25.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:39:25.549 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:39:25.600 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:25.653 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:39:25.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:25.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:39:25.737 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7, 8, 9, 11, 1] out_osds: [10, 2] dead_osds: [] live_osds: [0, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 2] 2024-04-23T16:39:25.737 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-23T16:39:25.759 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:39:25.878 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:25.910 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:39:25.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:25.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:39:26.023 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:39:26.090 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:26.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:25 smithi188 ceph-mon[44101]: pgmap v1386: 27 pgs: 2 unknown, 1 peering, 24 active+clean; 7.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail 2024-04-23T16:39:26.329 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:39:26.332 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:39:26.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:25 smithi103 ceph-mon[39629]: pgmap v1386: 27 pgs: 2 unknown, 1 peering, 24 active+clean; 7.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail 2024-04-23T16:39:26.365 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:39:26.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:25 smithi148 ceph-mon[44520]: pgmap v1386: 27 pgs: 2 unknown, 1 peering, 24 active+clean; 7.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail 2024-04-23T16:39:26.399 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:39:26.434 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:39:26.464 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:39:26.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:26.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:39:26.474 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:39:26.515 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:39:26.546 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:39:26.572 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:39:26.590 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:39:26.623 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:39:26.629 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:26.650 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:39:26.663 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:39:26.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:26.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:39:26.681 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:39:26.713 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:39:26.731 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:39:26.731 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:39:26.731 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:39:26.732 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:39:26.732 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:39:26.732 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:39:26.732 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:39:26.732 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:39:26.732 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:39:26.732 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:39:26.732 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:39:26.732 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:39:26.770 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:39:26.797 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:26.818 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:39:26.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:26.823 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:39:26.924 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:39:26.969 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:27.035 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:39:27.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:27.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:39:27.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:27 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:39:27.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:27 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:39:27.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:27 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3566073006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:39:27.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:27 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:27 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:39:27.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:27 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:39:27.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:27 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:39:27.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:27 smithi148 ceph-mon[44520]: pgmap v1387: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 8.1 GiB data, 21 GiB used, 874 GiB / 894 GiB avail; 1.7 KiB/s rd, 103 MiB/s wr, 2.36k op/s; 23236/265924 objects misplaced (8.738%); 15 MiB/s, 238 objects/s recovering 2024-04-23T16:39:27.141 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:39:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:27 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:39:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:27 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:39:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:27 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3566073006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:39:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:27 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:27 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:39:27.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:27 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:39:27.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:27 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:39:27.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:27 smithi188 ceph-mon[44101]: pgmap v1387: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 8.1 GiB data, 21 GiB used, 874 GiB / 894 GiB avail; 1.7 KiB/s rd, 103 MiB/s wr, 2.36k op/s; 23236/265924 objects misplaced (8.738%); 15 MiB/s, 238 objects/s recovering 2024-04-23T16:39:27.230 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:27.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:39:27.264 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:39:27.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:27.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:39:27.304 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:39:27.304 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:39:27.304 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:27 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3566073006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:39:27.304 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.304 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.304 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.304 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.304 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.304 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.305 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.305 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.305 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:27 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:27.305 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:39:27.305 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:39:27.305 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:27 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:39:27.305 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:27 smithi103 ceph-mon[39629]: pgmap v1387: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 8.1 GiB data, 21 GiB used, 874 GiB / 894 GiB avail; 1.7 KiB/s rd, 103 MiB/s wr, 2.36k op/s; 23236/265924 objects misplaced (8.738%); 15 MiB/s, 238 objects/s recovering 2024-04-23T16:39:27.371 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:39:27.399 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:27.420 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:39:27.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:27.424 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:39:27.510 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:39:27.510 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:39:27.524 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:39:27.572 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:28.321 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:28.321 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:28.321 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:28.321 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:28 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T16:39:28.321 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:28 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T16:39:28.322 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:28.322 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:28.322 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:28.322 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:28 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T16:39:28.322 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:28 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T16:39:28.332 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:39:28.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:28.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:39:28.352 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-23T16:39:28.437 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:39:28.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:28.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:28.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:28.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:28 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T16:39:28.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:28 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T16:39:28.513 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:28.547 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:39:28.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:28.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:39:28.661 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:39:28.688 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:28.708 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:39:28.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:28.711 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:39:28.812 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:39:28.813 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:39:28.839 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:28.859 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:39:28.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:28.863 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:39:28.965 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:39:28.974 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:39:28.994 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:29.014 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:39:29.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:29.017 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:39:29.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:29 smithi103 ceph-mon[39629]: pgmap v1388: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 8.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 1.5 KiB/s rd, 88 MiB/s wr, 2.07k op/s; 23236/264810 objects misplaced (8.775%); 13 MiB/s, 203 objects/s recovering 2024-04-23T16:39:29.118 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:39:29.176 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:29.215 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:39:29.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:29.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:39:29.322 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:39:29.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:29 smithi188 ceph-mon[44101]: pgmap v1388: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 8.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 1.5 KiB/s rd, 88 MiB/s wr, 2.07k op/s; 23236/264810 objects misplaced (8.775%); 13 MiB/s, 203 objects/s recovering 2024-04-23T16:39:29.349 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:29.369 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:39:29.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:29.373 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:39:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:29 smithi148 ceph-mon[44520]: pgmap v1388: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 8.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 1.5 KiB/s rd, 88 MiB/s wr, 2.07k op/s; 23236/264810 objects misplaced (8.775%); 13 MiB/s, 203 objects/s recovering 2024-04-23T16:39:29.474 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:39:29.540 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:29.602 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:39:29.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:29.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:39:29.710 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:39:29.756 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:29.871 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:39:29.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:29.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:39:29.981 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:39:30.026 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:30.174 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:39:30.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:30.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:39:30.275 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.0 to 1 (10000) 2024-04-23T16:39:30.286 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:39:30.315 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:30.324 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:39:30.338 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:39:30.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:30.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:39:30.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:30 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3570939667' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:39:30.442 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:39:30.467 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:30.487 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:39:30.489 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:30 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3570939667' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:39:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:30.491 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:39:30.592 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:39:30.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:30 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3570939667' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:39:30.661 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:30.778 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:39:30.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:30.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:39:30.885 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:39:30.938 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:30.976 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:39:30.977 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:39:30.993 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:39:30.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:30.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:39:31.100 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:39:31.147 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:31.239 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:39:31.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:31.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:39:31.344 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:39:31.370 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:31.392 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:39:31.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:31.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:39:31.499 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:39:31.539 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:31 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3570939667' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:39:31.539 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:31 smithi103 ceph-mon[39629]: osdmap e492: 12 total, 12 up, 10 in 2024-04-23T16:39:31.539 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:31 smithi103 ceph-mon[39629]: pgmap v1390: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 1.3 KiB/s rd, 77 MiB/s wr, 2.18k op/s; 21630/259094 objects misplaced (8.348%); 20 MiB/s, 312 objects/s recovering 2024-04-23T16:39:31.542 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:31.570 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:39:31.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:31.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:39:31.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:31 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3570939667' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:39:31.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:31 smithi148 ceph-mon[44520]: osdmap e492: 12 total, 12 up, 10 in 2024-04-23T16:39:31.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:31 smithi148 ceph-mon[44520]: pgmap v1390: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 1.3 KiB/s rd, 77 MiB/s wr, 2.18k op/s; 21630/259094 objects misplaced (8.348%); 20 MiB/s, 312 objects/s recovering 2024-04-23T16:39:31.678 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:39:31.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:31 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3570939667' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:39:31.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:31 smithi188 ceph-mon[44101]: osdmap e492: 12 total, 12 up, 10 in 2024-04-23T16:39:31.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:31 smithi188 ceph-mon[44101]: pgmap v1390: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 1.3 KiB/s rd, 77 MiB/s wr, 2.18k op/s; 21630/259094 objects misplaced (8.348%); 20 MiB/s, 312 objects/s recovering 2024-04-23T16:39:31.726 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:31.782 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:39:31.791 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-23T16:39:31.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:31.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:39:31.905 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:39:31.932 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:31.953 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:39:31.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:31.956 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:39:32.057 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:39:32.086 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:32.105 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:39:32.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:32.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:39:32.209 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:39:32.236 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:32.256 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:39:32.257 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:39:32.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:32.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:39:32.359 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:39:32.386 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:32.410 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:39:32.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:32.413 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:39:32.514 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:39:32.552 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:32 smithi148 ceph-mon[44520]: 1.a scrub starts 2024-04-23T16:39:32.552 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:32 smithi148 ceph-mon[44520]: 1.a scrub ok 2024-04-23T16:39:32.590 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:32.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:32 smithi103 ceph-mon[39629]: 1.a scrub starts 2024-04-23T16:39:32.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:32 smithi103 ceph-mon[39629]: 1.a scrub ok 2024-04-23T16:39:32.619 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:39:32.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:32.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:39:32.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:32 smithi188 ceph-mon[44101]: 1.a scrub starts 2024-04-23T16:39:32.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:32 smithi188 ceph-mon[44101]: 1.a scrub ok 2024-04-23T16:39:32.724 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:39:32.779 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:33.170 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:39:33.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:33.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:39:33.279 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:39:33.319 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:33.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:33 smithi103 ceph-mon[39629]: pgmap v1391: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 1.1 KiB/s rd, 68 MiB/s wr, 2.08k op/s; 20385/256246 objects misplaced (7.955%); 25 MiB/s, 392 objects/s recovering 2024-04-23T16:39:33.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:33 smithi148 ceph-mon[44520]: pgmap v1391: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 1.1 KiB/s rd, 68 MiB/s wr, 2.08k op/s; 20385/256246 objects misplaced (7.955%); 25 MiB/s, 392 objects/s recovering 2024-04-23T16:39:33.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:33 smithi188 ceph-mon[44101]: pgmap v1391: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 1.1 KiB/s rd, 68 MiB/s wr, 2.08k op/s; 20385/256246 objects misplaced (7.955%); 25 MiB/s, 392 objects/s recovering 2024-04-23T16:39:34.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:34 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:39:34.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:34 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:39:34.413 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:39:34.414 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:39:34.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:34 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:39:34.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:34 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:39:34.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:34 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:39:34.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:34 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:39:35.186 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:39:35.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:35.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:39:35.200 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-23T16:39:35.293 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:39:35.378 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:35.423 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:39:35.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:35.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:39:35.427 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:35 smithi148 ceph-mon[44520]: osdmap e493: 12 total, 12 up, 10 in 2024-04-23T16:39:35.427 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:35 smithi148 ceph-mon[44520]: pgmap v1393: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 747 op/s; 20385/253968 objects misplaced (8.027%); 16 MiB/s, 258 objects/s recovering 2024-04-23T16:39:35.528 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:39:35.580 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:35.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:35 smithi103 ceph-mon[39629]: osdmap e493: 12 total, 12 up, 10 in 2024-04-23T16:39:35.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:35 smithi103 ceph-mon[39629]: pgmap v1393: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 747 op/s; 20385/253968 objects misplaced (8.027%); 16 MiB/s, 258 objects/s recovering 2024-04-23T16:39:35.607 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:39:35.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:35.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:39:35.663 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:39:35.713 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:39:35.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:35 smithi188 ceph-mon[44101]: osdmap e493: 12 total, 12 up, 10 in 2024-04-23T16:39:35.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:35 smithi188 ceph-mon[44101]: pgmap v1393: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 747 op/s; 20385/253968 objects misplaced (8.027%); 16 MiB/s, 258 objects/s recovering 2024-04-23T16:39:35.752 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:37.823 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:39:37.910 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:39:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:37 smithi103 ceph-mon[39629]: pgmap v1394: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.08k op/s; 17306/247546 objects misplaced (6.991%); 35 MiB/s, 554 objects/s recovering 2024-04-23T16:39:38.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:37 smithi148 ceph-mon[44520]: pgmap v1394: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.08k op/s; 17306/247546 objects misplaced (6.991%); 35 MiB/s, 554 objects/s recovering 2024-04-23T16:39:38.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:37 smithi188 ceph-mon[44101]: pgmap v1394: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.08k op/s; 17306/247546 objects misplaced (6.991%); 35 MiB/s, 554 objects/s recovering 2024-04-23T16:39:38.595 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-23T16:39:38.871 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:38 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:39:38.871 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:38 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:39:39.040 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:39:39.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:38 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:39:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:38 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:39:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:38 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:39:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:38 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:39:39.515 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:39:39.519 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-23T16:39:39.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:39.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:39:39.626 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:39:39.655 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:39.674 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:39:39.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:39.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:39:39.779 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:39:39.828 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:39.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:39 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:39:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:39 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:39:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:39 smithi188 ceph-mon[44101]: pgmap v1395: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 911 op/s; 17306/246454 objects misplaced (7.022%); 29 MiB/s, 458 objects/s recovering 2024-04-23T16:39:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:39 smithi188 ceph-mon[44101]: osdmap e494: 12 total, 12 up, 10 in 2024-04-23T16:39:40.082 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:39 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:39:40.082 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:39 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:39:40.082 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:39 smithi148 ceph-mon[44520]: pgmap v1395: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 911 op/s; 17306/246454 objects misplaced (7.022%); 29 MiB/s, 458 objects/s recovering 2024-04-23T16:39:40.082 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:39 smithi148 ceph-mon[44520]: osdmap e494: 12 total, 12 up, 10 in 2024-04-23T16:39:40.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:39 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:39:40.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:39 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:39:40.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:39 smithi103 ceph-mon[39629]: pgmap v1395: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 911 op/s; 17306/246454 objects misplaced (7.022%); 29 MiB/s, 458 objects/s recovering 2024-04-23T16:39:40.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:39 smithi103 ceph-mon[39629]: osdmap e494: 12 total, 12 up, 10 in 2024-04-23T16:39:41.154 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:39:41.154 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:39:41.710 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:39:41.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:41.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:39:41.750 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:39:41.818 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:39:41.846 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:41.874 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:39:41.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:41.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:39:41.909 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-23T16:39:41.981 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:39:42.008 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:42.028 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:39:42.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:42.031 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:39:42.069 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:41 smithi103 ceph-mon[39629]: pgmap v1397: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 978 op/s; 15558/240582 objects misplaced (6.467%); 29 MiB/s, 458 objects/s recovering 2024-04-23T16:39:42.083 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:39:42.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:41 smithi148 ceph-mon[44520]: pgmap v1397: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 978 op/s; 15558/240582 objects misplaced (6.467%); 29 MiB/s, 458 objects/s recovering 2024-04-23T16:39:42.132 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:39:42.157 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:41 smithi188 ceph-mon[44101]: pgmap v1397: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 978 op/s; 15558/240582 objects misplaced (6.467%); 29 MiB/s, 458 objects/s recovering 2024-04-23T16:39:42.159 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:42.181 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:39:42.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:42.186 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:39:42.287 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:39:42.335 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:42.371 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:39:43.500 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:39:43.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:43.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:39:43.607 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:39:43.666 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:43.770 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:39:43.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:43.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:39:43.881 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:39:43.900 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:39:43.909 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:43.930 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:39:43.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:43.936 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:39:44.036 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:39:44.096 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:43 smithi148 ceph-mon[44520]: pgmap v1398: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s; 14263/237632 objects misplaced (6.002%); 35 MiB/s, 563 objects/s recovering 2024-04-23T16:39:44.096 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:43 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3544919185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:39:44.096 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:43 smithi148 ceph-mon[44520]: from='client.36077 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:39:44.096 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:43 smithi103 ceph-mon[39629]: pgmap v1398: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s; 14263/237632 objects misplaced (6.002%); 35 MiB/s, 563 objects/s recovering 2024-04-23T16:39:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:43 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3544919185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:39:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:43 smithi103 ceph-mon[39629]: from='client.36077 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:39:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:39:44.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:44.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:39:44.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:43 smithi188 ceph-mon[44101]: pgmap v1398: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s; 14263/237632 objects misplaced (6.002%); 35 MiB/s, 563 objects/s recovering 2024-04-23T16:39:44.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:43 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3544919185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:39:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:43 smithi188 ceph-mon[44101]: from='client.36077 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:39:44.239 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:39:44.305 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:44.433 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:39:44.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:44.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:39:44.497 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:39:44.498 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:39:44.545 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:39:44.587 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:44.952 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:39:44.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:44.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:39:45.062 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:39:45.093 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:45.118 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:39:45.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:45.122 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:39:45.223 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:39:45.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:44 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:39:45.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:44 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:39:45.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:44 smithi188 ceph-mon[44101]: from='client.36077 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:39:45.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:44 smithi188 ceph-mon[44101]: osdmap e495: 12 total, 12 up, 10 in 2024-04-23T16:39:45.262 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:44 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:39:45.263 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:44 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:39:45.263 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:44 smithi148 ceph-mon[44520]: from='client.36077 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:39:45.263 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:44 smithi148 ceph-mon[44520]: osdmap e495: 12 total, 12 up, 10 in 2024-04-23T16:39:45.293 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:45.308 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-23T16:39:45.327 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:39:45.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:45.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:39:45.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:44 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:39:45.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:44 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:39:45.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:44 smithi103 ceph-mon[39629]: from='client.36077 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:39:45.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:44 smithi103 ceph-mon[39629]: osdmap e495: 12 total, 12 up, 10 in 2024-04-23T16:39:45.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:39:45.476 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:45.506 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:39:45.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:45.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:39:45.616 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:39:45.649 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:45.757 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:39:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:45 smithi103 ceph-mon[39629]: pgmap v1400: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 767 op/s; 14263/235268 objects misplaced (6.062%); 19 MiB/s, 302 objects/s recovering 2024-04-23T16:39:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:45 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:39:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:45 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:39:46.107 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:39:46.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:46.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:39:46.211 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:39:46.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:45 smithi188 ceph-mon[44101]: pgmap v1400: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 767 op/s; 14263/235268 objects misplaced (6.062%); 19 MiB/s, 302 objects/s recovering 2024-04-23T16:39:46.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:45 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:39:46.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:45 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:39:46.242 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:45 smithi148 ceph-mon[44520]: pgmap v1400: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 767 op/s; 14263/235268 objects misplaced (6.062%); 19 MiB/s, 302 objects/s recovering 2024-04-23T16:39:46.243 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:45 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:39:46.243 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:45 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:39:46.246 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:46.315 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:39:46.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:46.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:39:46.424 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:39:46.460 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:46.656 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:39:46.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:46.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:39:46.761 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:39:46.803 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:46.927 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:46 smithi103 ceph-mon[39629]: osdmap e496: 12 total, 12 up, 10 in 2024-04-23T16:39:46.927 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:46 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:39:46.928 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:46 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:39:46.961 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:39:47.014 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:39:47.058 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:39:47.091 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:39:47.140 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:39:47.169 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:39:47.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:47.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:39:47.213 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:39:47.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:46 smithi188 ceph-mon[44101]: osdmap e496: 12 total, 12 up, 10 in 2024-04-23T16:39:47.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:46 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:39:47.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:46 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:39:47.257 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:39:47.299 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:39:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:39:47.339 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:46 smithi148 ceph-mon[44520]: osdmap e496: 12 total, 12 up, 10 in 2024-04-23T16:39:47.339 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:46 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:39:47.339 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:46 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:39:47.341 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:47.355 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:39:47.394 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:39:47.430 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:39:47.452 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:39:47.494 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:39:47.528 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:39:47.532 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:39:47.532 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:39:47.532 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:39:47.532 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:39:47.532 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:39:47.532 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:39:47.532 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:39:47.532 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:39:47.532 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:39:47.532 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:39:47.532 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:39:47.533 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:39:47.941 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.3 to 1 (10000) 2024-04-23T16:39:48.106 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:47 smithi103 ceph-mon[39629]: pgmap v1402: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.12k op/s; 11580/228702 objects misplaced (5.063%); 36 MiB/s, 578 objects/s recovering 2024-04-23T16:39:48.106 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:47 smithi103 ceph-mon[39629]: osdmap e497: 12 total, 12 up, 10 in 2024-04-23T16:39:48.106 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:47 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3872332213' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:39:48.106 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.106 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.106 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.106 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.107 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.107 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.107 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.107 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:47 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/420628561' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:39:48.107 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:47 smithi188 ceph-mon[44101]: pgmap v1402: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.12k op/s; 11580/228702 objects misplaced (5.063%); 36 MiB/s, 578 objects/s recovering 2024-04-23T16:39:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:47 smithi188 ceph-mon[44101]: osdmap e497: 12 total, 12 up, 10 in 2024-04-23T16:39:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:47 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3872332213' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:39:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:47 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/420628561' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:39:48.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:47 smithi148 ceph-mon[44520]: pgmap v1402: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 7.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.12k op/s; 11580/228702 objects misplaced (5.063%); 36 MiB/s, 578 objects/s recovering 2024-04-23T16:39:48.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:47 smithi148 ceph-mon[44520]: osdmap e497: 12 total, 12 up, 10 in 2024-04-23T16:39:48.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:47 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3872332213' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:39:48.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:48.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:47 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/420628561' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:39:48.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:49.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:49.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:49.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:49.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:48 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/420628561' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-23T16:39:49.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:48 smithi188 ceph-mon[44101]: osdmap e498: 12 total, 12 up, 10 in 2024-04-23T16:39:49.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:49.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:49.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:49.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:48 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/420628561' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-23T16:39:49.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:48 smithi103 ceph-mon[39629]: osdmap e498: 12 total, 12 up, 10 in 2024-04-23T16:39:49.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:49.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:49.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:49.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:48 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/420628561' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-23T16:39:49.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:48 smithi148 ceph-mon[44520]: osdmap e498: 12 total, 12 up, 10 in 2024-04-23T16:39:49.918 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-23T16:39:50.150 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:49 smithi103 ceph-mon[39629]: pgmap v1405: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 972 op/s; 11580/227466 objects misplaced (5.091%); 28 MiB/s, 441 objects/s recovering 2024-04-23T16:39:50.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:49 smithi188 ceph-mon[44101]: pgmap v1405: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 972 op/s; 11580/227466 objects misplaced (5.091%); 28 MiB/s, 441 objects/s recovering 2024-04-23T16:39:50.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:49 smithi148 ceph-mon[44520]: pgmap v1405: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 972 op/s; 11580/227466 objects misplaced (5.091%); 28 MiB/s, 441 objects/s recovering 2024-04-23T16:39:50.742 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:39:51.132 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:39:51.535 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:39:51.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:51.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:39:51.648 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:39:51.697 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:51.745 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:39:51.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:51.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:39:51.854 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:39:51.882 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:51.911 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:39:51.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:51.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:39:52.003 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:39:52.022 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:39:52.031 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:51 smithi103 ceph-mon[39629]: pgmap v1406: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.13k op/s; 10052/221674 objects misplaced (4.535%); 35 MiB/s, 554 objects/s recovering 2024-04-23T16:39:52.031 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:51 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3748694768' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:39:52.031 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:51 smithi103 ceph-mon[39629]: from='client.26398 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:39:52.079 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:52.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:51 smithi188 ceph-mon[44101]: pgmap v1406: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.13k op/s; 10052/221674 objects misplaced (4.535%); 35 MiB/s, 554 objects/s recovering 2024-04-23T16:39:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:51 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3748694768' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:39:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:51 smithi188 ceph-mon[44101]: from='client.26398 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:39:52.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:51 smithi148 ceph-mon[44520]: pgmap v1406: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.13k op/s; 10052/221674 objects misplaced (4.535%); 35 MiB/s, 554 objects/s recovering 2024-04-23T16:39:52.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:51 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3748694768' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:39:52.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:51 smithi148 ceph-mon[44520]: from='client.26398 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:39:53.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:52 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:39:53.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:52 smithi188 ceph-mon[44101]: from='client.26398 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:39:53.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:52 smithi188 ceph-mon[44101]: osdmap e499: 12 total, 12 up, 10 in 2024-04-23T16:39:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:52 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:39:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:52 smithi103 ceph-mon[39629]: from='client.26398 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:39:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:52 smithi103 ceph-mon[39629]: osdmap e499: 12 total, 12 up, 10 in 2024-04-23T16:39:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:52 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:39:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:52 smithi148 ceph-mon[44520]: from='client.26398 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:39:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:52 smithi148 ceph-mon[44520]: osdmap e499: 12 total, 12 up, 10 in 2024-04-23T16:39:53.532 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:39:53.668 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:39:53.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:53.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:39:53.775 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:39:53.838 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:53.880 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:39:53.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:53.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:39:53.988 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:39:54.015 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:54.039 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:39:54.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:54.041 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:39:54.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:54 smithi103 ceph-mon[39629]: pgmap v1408: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 833 op/s; 9144/218700 objects misplaced (4.181%); 21 MiB/s, 328 objects/s recovering 2024-04-23T16:39:54.143 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:39:54.170 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:54.190 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:39:54.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:54.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:39:54.294 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:39:54.359 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:54.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:54 smithi148 ceph-mon[44520]: pgmap v1408: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 833 op/s; 9144/218700 objects misplaced (4.181%); 21 MiB/s, 328 objects/s recovering 2024-04-23T16:39:54.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:54 smithi188 ceph-mon[44101]: pgmap v1408: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 833 op/s; 9144/218700 objects misplaced (4.181%); 21 MiB/s, 328 objects/s recovering 2024-04-23T16:39:54.521 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:39:54.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:54.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:39:54.560 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:39:54.560 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:39:54.560 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.560 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.16308.0:203517 6.b e500/455 6:d5b2c77a:::benchmark_data_smithi188_130221_object49056:head v 500'12721, mlcod=500'12718)", 2024-04-23T16:39:54.561 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:39:54.533052+0000", 2024-04-23T16:39:54.561 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.016897618, 2024-04-23T16:39:54.561 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016921535000000001, 2024-04-23T16:39:54.561 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:54.561 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:54.561 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:39:54.561 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:54.561 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.561 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:54.561 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.533052+0000", 2024-04-23T16:39:54.561 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:54.561 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.561 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.561 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:54.561 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.533052+0000", 2024-04-23T16:39:54.562 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:54.562 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.562 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.562 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:54.562 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.533054+0000", 2024-04-23T16:39:54.562 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.829e-06 2024-04-23T16:39:54.562 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.562 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.562 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:54.562 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.533063+0000", 2024-04-23T16:39:54.562 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7120000000000006e-06 2024-04-23T16:39:54.562 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.562 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.563 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:54.563 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.533076+0000", 2024-04-23T16:39:54.563 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3471e-05 2024-04-23T16:39:54.563 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.563 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.563 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:54.563 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.533084+0000", 2024-04-23T16:39:54.563 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.9650000000000008e-06 2024-04-23T16:39:54.563 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.563 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.563 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:54.563 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.533170+0000", 2024-04-23T16:39:54.563 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5186999999999996e-05 2024-04-23T16:39:54.563 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.563 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.564 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:54.564 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.533193+0000", 2024-04-23T16:39:54.564 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3105999999999999e-05 2024-04-23T16:39:54.564 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:54.564 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:54.564 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:54.564 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.564 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.564 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:203524 6.5 6:a3789fd8:::benchmark_data_smithi188_130221_object49063:head [delete] snapc 0=[] ondisk+write+known_if_redirected e500)", 2024-04-23T16:39:54.564 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:39:54.533700+0000", 2024-04-23T16:39:54.564 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.016250542, 2024-04-23T16:39:54.564 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016319159999999999, 2024-04-23T16:39:54.565 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:54.565 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:54.565 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:39:54.565 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:54.565 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:39:54.565 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:39:54.565 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 203524 2024-04-23T16:39:54.565 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.565 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:54.565 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.565 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:54.565 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.533700+0000", 2024-04-23T16:39:54.565 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:54.565 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.566 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.566 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:54.566 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.533700+0000", 2024-04-23T16:39:54.566 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:54.566 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.566 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.566 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:54.566 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.533701+0000", 2024-04-23T16:39:54.566 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9180000000000001e-06 2024-04-23T16:39:54.566 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.566 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.566 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:54.567 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.533708+0000", 2024-04-23T16:39:54.567 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0809999999999999e-06 2024-04-23T16:39:54.567 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.567 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.567 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:54.567 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.533721+0000", 2024-04-23T16:39:54.567 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3047000000000001e-05 2024-04-23T16:39:54.567 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.567 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.567 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:54.567 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.533728+0000", 2024-04-23T16:39:54.567 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0310000000000001e-06 2024-04-23T16:39:54.568 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.568 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.568 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:54.568 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.533775+0000", 2024-04-23T16:39:54.568 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6978999999999999e-05 2024-04-23T16:39:54.568 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.568 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.568 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:54.568 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.533899+0000", 2024-04-23T16:39:54.568 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000124256 2024-04-23T16:39:54.568 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.568 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.568 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:39:54.568 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.534012+0000", 2024-04-23T16:39:54.568 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000113164 2024-04-23T16:39:54.569 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.569 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.569 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:54.569 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.543764+0000", 2024-04-23T16:39:54.569 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.009752185 2024-04-23T16:39:54.569 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:54.569 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:54.569 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:54.569 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.569 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.569 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:203528 6.3 6:c60f1654:::benchmark_data_smithi188_130221_object49067:head [delete] snapc 0=[] ondisk+write+known_if_redirected e500)", 2024-04-23T16:39:54.569 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:39:54.537495+0000", 2024-04-23T16:39:54.569 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.012454611000000001, 2024-04-23T16:39:54.570 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012572913999999999, 2024-04-23T16:39:54.570 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:54.570 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:54.570 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:39:54.570 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:54.570 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:39:54.570 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:39:54.570 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 203528 2024-04-23T16:39:54.570 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.570 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:54.570 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.570 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:54.570 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.537495+0000", 2024-04-23T16:39:54.570 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:54.571 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.571 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.571 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:54.571 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.537495+0000", 2024-04-23T16:39:54.571 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:54.571 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.571 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.571 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:54.571 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.537497+0000", 2024-04-23T16:39:54.571 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.967e-06 2024-04-23T16:39:54.571 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.571 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.571 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:54.571 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.537504+0000", 2024-04-23T16:39:54.571 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7730000000000002e-06 2024-04-23T16:39:54.571 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.572 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.572 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:54.572 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.537520+0000", 2024-04-23T16:39:54.572 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6064000000000001e-05 2024-04-23T16:39:54.572 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.572 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.572 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:54.572 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.537529+0000", 2024-04-23T16:39:54.572 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5110000000000003e-06 2024-04-23T16:39:54.572 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.572 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.572 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:54.572 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.538276+0000", 2024-04-23T16:39:54.572 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00074702700000000004 2024-04-23T16:39:54.572 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.573 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.573 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:54.573 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.538424+0000", 2024-04-23T16:39:54.573 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014785800000000001 2024-04-23T16:39:54.573 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.573 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.573 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:39:54.573 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.538565+0000", 2024-04-23T16:39:54.573 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014166999999999999 2024-04-23T16:39:54.573 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.573 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.573 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:54.573 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.548321+0000", 2024-04-23T16:39:54.574 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0097557709999999999 2024-04-23T16:39:54.574 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:54.574 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:54.574 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:54.574 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.574 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.574 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:203531 6.5 6:aede353a:::benchmark_data_smithi188_130221_object49070:head [delete] snapc 0=[] ondisk+write+known_if_redirected e500)", 2024-04-23T16:39:54.574 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:39:54.538713+0000", 2024-04-23T16:39:54.574 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.011237166999999999, 2024-04-23T16:39:54.574 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011401638, 2024-04-23T16:39:54.574 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:39:54.574 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:39:54.574 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:39:54.575 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:39:54.575 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:39:54.575 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:39:54.575 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 203531 2024-04-23T16:39:54.575 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.575 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:39:54.575 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.575 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:39:54.575 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.538713+0000", 2024-04-23T16:39:54.575 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:54.575 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.575 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.575 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:39:54.575 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.538713+0000", 2024-04-23T16:39:54.575 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:39:54.575 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.576 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.576 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:39:54.576 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.538714+0000", 2024-04-23T16:39:54.576 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.167e-06 2024-04-23T16:39:54.576 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.576 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.576 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:39:54.576 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.538718+0000", 2024-04-23T16:39:54.576 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4009999999999999e-06 2024-04-23T16:39:54.576 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.576 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.576 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:39:54.576 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.538728+0000", 2024-04-23T16:39:54.576 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8519999999999999e-06 2024-04-23T16:39:54.576 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.576 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.577 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:39:54.577 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.538736+0000", 2024-04-23T16:39:54.577 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5039999999999996e-06 2024-04-23T16:39:54.577 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.577 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.577 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:39:54.577 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.538782+0000", 2024-04-23T16:39:54.577 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6134000000000002e-05 2024-04-23T16:39:54.577 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.577 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.577 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:39:54.577 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.538883+0000", 2024-04-23T16:39:54.577 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000100837 2024-04-23T16:39:54.577 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.577 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.577 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:39:54.578 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.538983+0000", 2024-04-23T16:39:54.578 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010048300000000001 2024-04-23T16:39:54.578 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:39:54.578 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:39:54.578 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:39:54.578 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:54.544189+0000", 2024-04-23T16:39:54.578 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0052057190000000001 2024-04-23T16:39:54.578 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:54.578 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:39:54.578 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:54.578 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:39:54.578 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-04-23T16:39:54.578 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 4 2024-04-23T16:39:54.578 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:39:54.626 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:39:54.654 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:54.675 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:39:54.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:54.677 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:39:54.778 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:39:54.841 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:55.040 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.4 to 1 (10000) 2024-04-23T16:39:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:55 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:39:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:55 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:39:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:55 smithi103 ceph-mon[39629]: osdmap e500: 12 total, 12 up, 10 in 2024-04-23T16:39:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:55 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/830570605' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-23T16:39:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:55 smithi103 ceph-mon[39629]: pgmap v1410: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 856 op/s; 9144/216358 objects misplaced (4.226%); 18 MiB/s, 287 objects/s recovering 2024-04-23T16:39:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:55 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:39:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:55 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:39:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:55 smithi148 ceph-mon[44520]: osdmap e500: 12 total, 12 up, 10 in 2024-04-23T16:39:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:55 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/830570605' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-23T16:39:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:55 smithi148 ceph-mon[44520]: pgmap v1410: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 856 op/s; 9144/216358 objects misplaced (4.226%); 18 MiB/s, 287 objects/s recovering 2024-04-23T16:39:55.398 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:39:55.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:55.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:39:55.442 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-23T16:39:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:55 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:39:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:55 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:39:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:55 smithi188 ceph-mon[44101]: osdmap e500: 12 total, 12 up, 10 in 2024-04-23T16:39:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:55 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/830570605' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-23T16:39:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:55 smithi188 ceph-mon[44101]: pgmap v1410: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 856 op/s; 9144/216358 objects misplaced (4.226%); 18 MiB/s, 287 objects/s recovering 2024-04-23T16:39:55.503 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:39:55.507 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:39:55.556 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:55.637 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:39:55.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:55.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:39:55.751 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:39:55.777 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:55.797 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:39:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:55.802 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:39:55.903 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:39:55.931 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:55.950 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:39:55.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:55.955 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:39:55.978 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:39:56.056 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:39:56.120 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:56.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:56 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/830570605' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-23T16:39:56.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:56 smithi103 ceph-mon[39629]: osdmap e501: 12 total, 12 up, 10 in 2024-04-23T16:39:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:56 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/830570605' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-23T16:39:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:56 smithi148 ceph-mon[44520]: osdmap e501: 12 total, 12 up, 10 in 2024-04-23T16:39:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:56 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/830570605' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-23T16:39:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:56 smithi188 ceph-mon[44101]: osdmap e501: 12 total, 12 up, 10 in 2024-04-23T16:39:56.603 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-23T16:39:56.785 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:39:56.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:56.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:39:56.894 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:39:56.949 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:57.016 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:39:57.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:57.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:39:57.127 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:39:57.214 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:57.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:57 smithi103 ceph-mon[39629]: pgmap v1412: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 981 op/s; 6745/209898 objects misplaced (3.213%); 30 MiB/s, 482 objects/s recovering 2024-04-23T16:39:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:57 smithi148 ceph-mon[44520]: pgmap v1412: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 981 op/s; 6745/209898 objects misplaced (3.213%); 30 MiB/s, 482 objects/s recovering 2024-04-23T16:39:57.393 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:39:57.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:57.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:39:57.403 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:39:57.453 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:39:57.454 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:39:57.470 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:39:57.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:57 smithi188 ceph-mon[44101]: pgmap v1412: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 981 op/s; 6745/209898 objects misplaced (3.213%); 30 MiB/s, 482 objects/s recovering 2024-04-23T16:39:57.499 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:39:57.508 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:39:57.525 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:57.546 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:39:57.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:57.549 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:39:57.551 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:39:57.591 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:39:57.623 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:39:57.650 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:39:57.657 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:39:57.696 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:39:57.701 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:57.741 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:39:57.758 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:39:57.778 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:39:57.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:57.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:39:57.790 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:39:57.821 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:39:57.827 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:39:57.827 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:39:57.827 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:39:57.827 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:39:57.827 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:39:57.828 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:39:57.828 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:39:57.828 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:39:57.828 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:39:57.828 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:39:57.828 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:39:57.828 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:39:57.885 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:39:57.934 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:57.967 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:39:57.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:57.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:39:58.078 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:39:58.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:58 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2250069769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:39:58.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.150 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:58.261 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:39:58.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:58.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:39:58.266 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:58 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2250069769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:39:58.266 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.266 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.266 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.266 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.266 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.266 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.266 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.266 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.267 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.366 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:39:58.419 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:58 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2250069769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:39:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:58.488 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:39:58.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:58.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:39:58.594 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:39:58.623 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:58.644 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:39:58.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:58.647 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:39:58.748 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:39:58.787 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:58.865 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:39:58.865 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:39:59.297 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:59.297 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:59.297 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:59.297 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:59 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:39:59.297 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:59 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:39:59.297 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:59 smithi148 ceph-mon[44520]: osdmap e502: 12 total, 12 up, 10 in 2024-04-23T16:39:59.297 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:39:59 smithi148 ceph-mon[44520]: pgmap v1414: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 825 op/s; 6745/208788 objects misplaced (3.231%); 22 MiB/s, 347 objects/s recovering 2024-04-23T16:39:59.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:59.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:59.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:59.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:59 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:39:59.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:59 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:39:59.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:59 smithi103 ceph-mon[39629]: osdmap e502: 12 total, 12 up, 10 in 2024-04-23T16:39:59.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:39:59 smithi103 ceph-mon[39629]: pgmap v1414: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 825 op/s; 6745/208788 objects misplaced (3.231%); 22 MiB/s, 347 objects/s recovering 2024-04-23T16:39:59.447 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:39:59.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:39:59.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:39:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:39:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:59 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:39:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:59 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:39:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:59 smithi188 ceph-mon[44101]: osdmap e502: 12 total, 12 up, 10 in 2024-04-23T16:39:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:39:59 smithi188 ceph-mon[44101]: pgmap v1414: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 825 op/s; 6745/208788 objects misplaced (3.231%); 22 MiB/s, 347 objects/s recovering 2024-04-23T16:39:59.550 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:39:59.579 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:39:59.631 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-23T16:39:59.948 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:40:00.353 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:00 smithi148 ceph-mon[44520]: osdmap e503: 12 total, 12 up, 10 in 2024-04-23T16:40:00.353 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:00 smithi148 ceph-mon[44520]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-04-23T16:40:00.353 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:00 smithi148 ceph-mon[44520]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-04-23T16:40:00.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:00 smithi103 ceph-mon[39629]: osdmap e503: 12 total, 12 up, 10 in 2024-04-23T16:40:00.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:00 smithi103 ceph-mon[39629]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-04-23T16:40:00.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:00 smithi103 ceph-mon[39629]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-04-23T16:40:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:00 smithi188 ceph-mon[44101]: osdmap e503: 12 total, 12 up, 10 in 2024-04-23T16:40:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:00 smithi188 ceph-mon[44101]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-04-23T16:40:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:00 smithi188 ceph-mon[44101]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-04-23T16:40:00.605 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:40:00.630 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:40:00.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:00.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:40:00.735 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:40:00.762 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:00.786 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:40:00.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:00.788 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:40:00.889 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:40:00.917 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:00.938 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:40:00.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:00.941 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:40:01.042 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:40:01.076 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:01.330 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:40:01.335 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:01 smithi103 ceph-mon[39629]: osdmap e504: 12 total, 12 up, 10 in 2024-04-23T16:40:01.336 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:01 smithi103 ceph-mon[39629]: pgmap v1417: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.2 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s; 4785/203416 objects misplaced (2.352%); 36 MiB/s, 570 objects/s recovering 2024-04-23T16:40:01.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:01.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:40:01.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:01 smithi148 ceph-mon[44520]: osdmap e504: 12 total, 12 up, 10 in 2024-04-23T16:40:01.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:01 smithi148 ceph-mon[44520]: pgmap v1417: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.2 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s; 4785/203416 objects misplaced (2.352%); 36 MiB/s, 570 objects/s recovering 2024-04-23T16:40:01.438 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:40:01.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:01 smithi188 ceph-mon[44101]: osdmap e504: 12 total, 12 up, 10 in 2024-04-23T16:40:01.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:01 smithi188 ceph-mon[44101]: pgmap v1417: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 6.2 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s; 4785/203416 objects misplaced (2.352%); 36 MiB/s, 570 objects/s recovering 2024-04-23T16:40:01.503 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:01.762 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:40:01.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:01.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:40:01.876 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:40:01.934 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:02.081 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:40:02.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:02.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:40:02.191 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:40:02.256 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:02.289 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.5 to 1 (10000) 2024-04-23T16:40:02.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:02 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/138741906' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-23T16:40:02.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:02 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/138741906' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-23T16:40:02.639 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:40:02.639 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:40:02.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:02 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/138741906' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-23T16:40:02.947 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:40:02.955 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:40:02.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:02.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:40:03.059 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:40:03.117 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:03.300 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:03 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/138741906' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-23T16:40:03.300 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:03 smithi148 ceph-mon[44520]: osdmap e505: 12 total, 12 up, 10 in 2024-04-23T16:40:03.300 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:03 smithi148 ceph-mon[44520]: pgmap v1419: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 6.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 896 op/s; 4334/200884 objects misplaced (2.157%); 29 MiB/s, 457 objects/s recovering 2024-04-23T16:40:03.301 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:03 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/245977864' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:40:03.329 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:40:03.497 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:03 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/138741906' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-23T16:40:03.498 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:03 smithi103 ceph-mon[39629]: osdmap e505: 12 total, 12 up, 10 in 2024-04-23T16:40:03.498 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:03 smithi103 ceph-mon[39629]: pgmap v1419: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 6.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 896 op/s; 4334/200884 objects misplaced (2.157%); 29 MiB/s, 457 objects/s recovering 2024-04-23T16:40:03.498 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:03 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/245977864' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:40:03.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:03 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/138741906' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-23T16:40:03.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:03 smithi188 ceph-mon[44101]: osdmap e505: 12 total, 12 up, 10 in 2024-04-23T16:40:03.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:03 smithi188 ceph-mon[44101]: pgmap v1419: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 6.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 896 op/s; 4334/200884 objects misplaced (2.157%); 29 MiB/s, 457 objects/s recovering 2024-04-23T16:40:03.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:03 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/245977864' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:40:04.296 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-23T16:40:04.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:04 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:40:04.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:04 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/245977864' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:40:04.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:04 smithi103 ceph-mon[39629]: osdmap e506: 12 total, 12 up, 10 in 2024-04-23T16:40:04.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:04 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:40:04.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:04 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:40:04.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:04 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:40:04.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:04 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:40:04.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:04 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:40:04.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:04 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:40:04.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:04 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:40:04.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:04 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/245977864' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:40:04.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:04 smithi148 ceph-mon[44520]: osdmap e506: 12 total, 12 up, 10 in 2024-04-23T16:40:04.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:04 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:40:04.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:04 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:40:04.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:04 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:40:04.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:04 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:40:04.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:04 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:40:04.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:04 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:40:04.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:04 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:40:04.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:04 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/245977864' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:40:04.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:04 smithi188 ceph-mon[44101]: osdmap e506: 12 total, 12 up, 10 in 2024-04-23T16:40:04.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:04 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:40:04.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:04 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:40:04.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:04 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:40:04.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:04 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:40:04.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:04 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:40:04.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:04 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:40:04.985 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-23T16:40:04.992 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:40:04.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:04.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:40:05.099 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:40:05.128 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:05.155 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:40:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:05.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:40:05.260 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:40:05.291 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:40:05.336 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:05.381 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:40:05.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:05.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:40:05.494 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:40:05.522 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:05.626 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:40:05.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:05.630 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:40:05.731 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:40:05.786 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:40:05.794 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:06.061 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:40:06.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:06.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:40:06.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:05 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:40:06.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:05 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:40:06.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:05 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:40:06.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:05 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:40:06.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:05 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:40:06.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:05 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:40:06.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:05 smithi103 ceph-mon[39629]: pgmap v1421: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 6.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 877 op/s; 4334/198794 objects misplaced (2.180%); 24 MiB/s, 378 objects/s recovering 2024-04-23T16:40:06.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:05 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:40:06.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:05 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:40:06.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:05 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:40:06.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:05 smithi103 ceph-mon[39629]: osdmap e507: 12 total, 12 up, 10 in 2024-04-23T16:40:06.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:05 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:40:06.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:05 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:40:06.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:05 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:40:06.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:05 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:40:06.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:05 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:40:06.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:05 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:40:06.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:05 smithi148 ceph-mon[44520]: pgmap v1421: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 6.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 877 op/s; 4334/198794 objects misplaced (2.180%); 24 MiB/s, 378 objects/s recovering 2024-04-23T16:40:06.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:05 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:40:06.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:05 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:40:06.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:05 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:40:06.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:05 smithi148 ceph-mon[44520]: osdmap e507: 12 total, 12 up, 10 in 2024-04-23T16:40:06.165 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:40:06.208 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:05 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:40:06.208 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:05 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:40:06.208 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:05 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:40:06.208 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:05 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:40:06.208 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:05 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:40:06.208 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:05 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:40:06.208 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:05 smithi188 ceph-mon[44101]: pgmap v1421: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 6.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 877 op/s; 4334/198794 objects misplaced (2.180%); 24 MiB/s, 378 objects/s recovering 2024-04-23T16:40:06.208 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:05 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:40:06.208 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:05 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:40:06.208 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:05 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:40:06.208 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:05 smithi188 ceph-mon[44101]: osdmap e507: 12 total, 12 up, 10 in 2024-04-23T16:40:06.227 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:06.278 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:40:06.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:06.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:40:06.382 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:40:06.403 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:40:06.431 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:06.435 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:40:06.469 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:40:06.502 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:40:06.528 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:40:06.551 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:40:06.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:06.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:40:06.577 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:40:06.610 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:40:06.654 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:40:06.656 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:40:06.680 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:40:06.711 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:40:06.729 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:40:06.730 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:06.757 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:40:06.761 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:40:06.761 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:40:06.761 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:40:06.761 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:40:06.761 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:40:06.762 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:40:06.762 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:40:06.762 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:40:06.762 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:40:06.762 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:40:06.762 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:40:06.762 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:40:06.762 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:40:06.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:06.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:40:06.870 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:40:06.919 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:06.963 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:06 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:40:06.963 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:06 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:40:06.963 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:06 smithi103 ceph-mon[39629]: osdmap e508: 12 total, 12 up, 10 in 2024-04-23T16:40:06.963 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:06 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/951634210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:40:06.963 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:06.963 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:06.963 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:06.963 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:06.963 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:06.963 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:06.963 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:06.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:06 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:40:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:06 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:40:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:06 smithi188 ceph-mon[44101]: osdmap e508: 12 total, 12 up, 10 in 2024-04-23T16:40:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:06 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/951634210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:40:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:06.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:06 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:40:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:06 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:40:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:06 smithi148 ceph-mon[44520]: osdmap e508: 12 total, 12 up, 10 in 2024-04-23T16:40:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:06 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/951634210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:40:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:07.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:07.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:07.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:07.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:07.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:07.761 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:40:07.761 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:40:07.762 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-04-23T16:40:07.762 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:40:07.770 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:40:08.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:07 smithi103 ceph-mon[39629]: pgmap v1424: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 5.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 967 op/s; 2816/192150 objects misplaced (1.466%); 19 MiB/s, 301 objects/s recovering 2024-04-23T16:40:08.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:08.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:08.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:08.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:08.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:08.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:07 smithi148 ceph-mon[44520]: pgmap v1424: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 5.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 967 op/s; 2816/192150 objects misplaced (1.466%); 19 MiB/s, 301 objects/s recovering 2024-04-23T16:40:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:08.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:08.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:07 smithi188 ceph-mon[44101]: pgmap v1424: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 5.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 967 op/s; 2816/192150 objects misplaced (1.466%); 19 MiB/s, 301 objects/s recovering 2024-04-23T16:40:09.906 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-23T16:40:10.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:09 smithi148 ceph-mon[44520]: pgmap v1425: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 5.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 833 op/s; 2816/190882 objects misplaced (1.475%); 14 MiB/s, 226 objects/s recovering 2024-04-23T16:40:10.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:09 smithi188 ceph-mon[44101]: pgmap v1425: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 5.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 833 op/s; 2816/190882 objects misplaced (1.475%); 14 MiB/s, 226 objects/s recovering 2024-04-23T16:40:10.261 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:40:10.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:09 smithi103 ceph-mon[39629]: pgmap v1425: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 5.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 833 op/s; 2816/190882 objects misplaced (1.475%); 14 MiB/s, 226 objects/s recovering 2024-04-23T16:40:12.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:11 smithi148 ceph-mon[44520]: pgmap v1426: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 5.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 959 op/s; 2188/186498 objects misplaced (1.173%); 16 MiB/s, 254 objects/s recovering 2024-04-23T16:40:12.148 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:40:12.149 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:40:12.149 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:40:12.149 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:40:12.149 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:40:12.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:11 smithi188 ceph-mon[44101]: pgmap v1426: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 5.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 959 op/s; 2188/186498 objects misplaced (1.173%); 16 MiB/s, 254 objects/s recovering 2024-04-23T16:40:12.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:11 smithi103 ceph-mon[39629]: pgmap v1426: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 5.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 959 op/s; 2188/186498 objects misplaced (1.173%); 16 MiB/s, 254 objects/s recovering 2024-04-23T16:40:12.956 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-23T16:40:13.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10740060744380098]}]: dispatch 2024-04-23T16:40:13.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10740060744380098]}]: dispatch 2024-04-23T16:40:13.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.0872955322265625]}]: dispatch 2024-04-23T16:40:13.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09871852241091887]}]: dispatch 2024-04-23T16:40:13.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.11318866413238907]}]: dispatch 2024-04-23T16:40:13.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.10161255075521292]}]: dispatch 2024-04-23T16:40:13.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.09871852241091887]}]: dispatch 2024-04-23T16:40:13.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.09995882027275918]}]: dispatch 2024-04-23T16:40:13.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.11318866413238907]}]: dispatch 2024-04-23T16:40:13.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.09995882027275918]}]: dispatch 2024-04-23T16:40:13.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0872955322265625]}]: dispatch 2024-04-23T16:40:13.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.10740060744380098]}]: dispatch 2024-04-23T16:40:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10740060744380098]}]: dispatch 2024-04-23T16:40:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10740060744380098]}]: dispatch 2024-04-23T16:40:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.0872955322265625]}]: dispatch 2024-04-23T16:40:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09871852241091887]}]: dispatch 2024-04-23T16:40:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.11318866413238907]}]: dispatch 2024-04-23T16:40:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.10161255075521292]}]: dispatch 2024-04-23T16:40:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.09871852241091887]}]: dispatch 2024-04-23T16:40:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.09995882027275918]}]: dispatch 2024-04-23T16:40:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.11318866413238907]}]: dispatch 2024-04-23T16:40:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.09995882027275918]}]: dispatch 2024-04-23T16:40:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0872955322265625]}]: dispatch 2024-04-23T16:40:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.10740060744380098]}]: dispatch 2024-04-23T16:40:13.260 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:40:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10740060744380098]}]: dispatch 2024-04-23T16:40:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10740060744380098]}]: dispatch 2024-04-23T16:40:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.0872955322265625]}]: dispatch 2024-04-23T16:40:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09871852241091887]}]: dispatch 2024-04-23T16:40:13.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.11318866413238907]}]: dispatch 2024-04-23T16:40:13.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.10161255075521292]}]: dispatch 2024-04-23T16:40:13.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.09871852241091887]}]: dispatch 2024-04-23T16:40:13.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.09995882027275918]}]: dispatch 2024-04-23T16:40:13.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.11318866413238907]}]: dispatch 2024-04-23T16:40:13.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.09995882027275918]}]: dispatch 2024-04-23T16:40:13.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0872955322265625]}]: dispatch 2024-04-23T16:40:13.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.10740060744380098]}]: dispatch 2024-04-23T16:40:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:13 smithi188 ceph-mon[44101]: pgmap v1427: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s; 1479/181388 objects misplaced (0.815%); 20 MiB/s, 322 objects/s recovering 2024-04-23T16:40:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10740060744380098]}]': finished 2024-04-23T16:40:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10740060744380098]}]': finished 2024-04-23T16:40:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.0872955322265625]}]': finished 2024-04-23T16:40:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09871852241091887]}]': finished 2024-04-23T16:40:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.11318866413238907]}]': finished 2024-04-23T16:40:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.10161255075521292]}]': finished 2024-04-23T16:40:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.09871852241091887]}]': finished 2024-04-23T16:40:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.09995882027275918]}]': finished 2024-04-23T16:40:14.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.11318866413238907]}]': finished 2024-04-23T16:40:14.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.09995882027275918]}]': finished 2024-04-23T16:40:14.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0872955322265625]}]': finished 2024-04-23T16:40:14.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.10740060744380098]}]': finished 2024-04-23T16:40:14.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:13 smithi188 ceph-mon[44101]: osdmap e509: 12 total, 12 up, 10 in 2024-04-23T16:40:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:13 smithi103 ceph-mon[39629]: pgmap v1427: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s; 1479/181388 objects misplaced (0.815%); 20 MiB/s, 322 objects/s recovering 2024-04-23T16:40:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10740060744380098]}]': finished 2024-04-23T16:40:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10740060744380098]}]': finished 2024-04-23T16:40:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.0872955322265625]}]': finished 2024-04-23T16:40:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09871852241091887]}]': finished 2024-04-23T16:40:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.11318866413238907]}]': finished 2024-04-23T16:40:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.10161255075521292]}]': finished 2024-04-23T16:40:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.09871852241091887]}]': finished 2024-04-23T16:40:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.09995882027275918]}]': finished 2024-04-23T16:40:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.11318866413238907]}]': finished 2024-04-23T16:40:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.09995882027275918]}]': finished 2024-04-23T16:40:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0872955322265625]}]': finished 2024-04-23T16:40:14.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.10740060744380098]}]': finished 2024-04-23T16:40:14.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:13 smithi103 ceph-mon[39629]: osdmap e509: 12 total, 12 up, 10 in 2024-04-23T16:40:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:13 smithi148 ceph-mon[44520]: pgmap v1427: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s; 1479/181388 objects misplaced (0.815%); 20 MiB/s, 322 objects/s recovering 2024-04-23T16:40:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10740060744380098]}]': finished 2024-04-23T16:40:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10740060744380098]}]': finished 2024-04-23T16:40:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.0872955322265625]}]': finished 2024-04-23T16:40:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09871852241091887]}]': finished 2024-04-23T16:40:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.11318866413238907]}]': finished 2024-04-23T16:40:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.10161255075521292]}]': finished 2024-04-23T16:40:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.09871852241091887]}]': finished 2024-04-23T16:40:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.09995882027275918]}]': finished 2024-04-23T16:40:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.11318866413238907]}]': finished 2024-04-23T16:40:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.09995882027275918]}]': finished 2024-04-23T16:40:14.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0872955322265625]}]': finished 2024-04-23T16:40:14.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.10740060744380098]}]': finished 2024-04-23T16:40:14.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:13 smithi148 ceph-mon[44520]: osdmap e509: 12 total, 12 up, 10 in 2024-04-23T16:40:15.082 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:40:15.082 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-04-23T16:40:15.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-04-23T16:40:15.082 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:40:15.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.082 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:160761 5.a 5:50bf586a:::benchmark_data_smithi188_113957_object7163:head [delete] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-04-23T16:40:15.082 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:38.751162+0000", 2024-04-23T16:40:15.082 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 396.32630748399998, 2024-04-23T16:40:15.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17909995200000001, 2024-04-23T16:40:15.082 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.082 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.083 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.083 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.083 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:15.083 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:15.083 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 160761 2024-04-23T16:40:15.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.083 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.751162+0000", 2024-04-23T16:40:15.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.751162+0000", 2024-04-23T16:40:15.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.751163+0000", 2024-04-23T16:40:15.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.415e-06 2024-04-23T16:40:15.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.751168+0000", 2024-04-23T16:40:15.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6550000000000003e-06 2024-04-23T16:40:15.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.751177+0000", 2024-04-23T16:40:15.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3249999999999997e-06 2024-04-23T16:40:15.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.751185+0000", 2024-04-23T16:40:15.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8180000000000004e-06 2024-04-23T16:40:15.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.753098+0000", 2024-04-23T16:40:15.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00191292 2024-04-23T16:40:15.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:40:15.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.753383+0000", 2024-04-23T16:40:15.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00028443000000000001 2024-04-23T16:40:15.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.838181+0000", 2024-04-23T16:40:15.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.084798011000000006 2024-04-23T16:40:15.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.838272+0000", 2024-04-23T16:40:15.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1055000000000001e-05 2024-04-23T16:40:15.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:40:15.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.838402+0000", 2024-04-23T16:40:15.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013013799999999999 2024-04-23T16:40:15.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.864331+0000", 2024-04-23T16:40:15.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025929526000000001 2024-04-23T16:40:15.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.927147+0000", 2024-04-23T16:40:15.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062815864999999999 2024-04-23T16:40:15.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.927182+0000", 2024-04-23T16:40:15.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5064999999999997e-05 2024-04-23T16:40:15.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.930262+0000", 2024-04-23T16:40:15.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030797289999999998 2024-04-23T16:40:15.088 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.088 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.088 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.088 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:160764 5.a 5:5179712e:::benchmark_data_smithi188_113957_object7166:head [delete] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-04-23T16:40:15.089 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:38.756328+0000", 2024-04-23T16:40:15.089 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 396.321141356, 2024-04-23T16:40:15.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17681493700000001, 2024-04-23T16:40:15.089 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.089 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.089 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.089 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.089 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:15.089 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:15.089 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 160764 2024-04-23T16:40:15.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.089 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756328+0000", 2024-04-23T16:40:15.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756328+0000", 2024-04-23T16:40:15.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756330+0000", 2024-04-23T16:40:15.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.013e-06 2024-04-23T16:40:15.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756335+0000", 2024-04-23T16:40:15.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2024-04-23T16:40:15.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756345+0000", 2024-04-23T16:40:15.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0122000000000001e-05 2024-04-23T16:40:15.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756351+0000", 2024-04-23T16:40:15.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6740000000000002e-06 2024-04-23T16:40:15.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756393+0000", 2024-04-23T16:40:15.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2258000000000001e-05 2024-04-23T16:40:15.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:40:15.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.756643+0000", 2024-04-23T16:40:15.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000249497 2024-04-23T16:40:15.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.847557+0000", 2024-04-23T16:40:15.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.090914377000000005 2024-04-23T16:40:15.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.847648+0000", 2024-04-23T16:40:15.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0999000000000001e-05 2024-04-23T16:40:15.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:40:15.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.847805+0000", 2024-04-23T16:40:15.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015637100000000001 2024-04-23T16:40:15.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.864339+0000", 2024-04-23T16:40:15.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016534505000000001 2024-04-23T16:40:15.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.930330+0000", 2024-04-23T16:40:15.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.065990368999999993 2024-04-23T16:40:15.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.930385+0000", 2024-04-23T16:40:15.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5630000000000001e-05 2024-04-23T16:40:15.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.933143+0000", 2024-04-23T16:40:15.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027578680000000001 2024-04-23T16:40:15.095 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.095 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.095 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.095 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:160766 5.a 5:588abbaa:::benchmark_data_smithi188_113957_object7168:head [delete] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-04-23T16:40:15.095 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:33:38.757349+0000", 2024-04-23T16:40:15.095 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 396.32012081300002, 2024-04-23T16:40:15.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17851373300000001, 2024-04-23T16:40:15.095 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.095 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.095 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.095 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.096 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:15.096 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:15.096 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 160766 2024-04-23T16:40:15.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.096 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757349+0000", 2024-04-23T16:40:15.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757349+0000", 2024-04-23T16:40:15.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757350+0000", 2024-04-23T16:40:15.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0419999999999999e-06 2024-04-23T16:40:15.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757353+0000", 2024-04-23T16:40:15.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7790000000000002e-06 2024-04-23T16:40:15.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757361+0000", 2024-04-23T16:40:15.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4769999999999997e-06 2024-04-23T16:40:15.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757366+0000", 2024-04-23T16:40:15.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5550000000000002e-06 2024-04-23T16:40:15.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757405+0000", 2024-04-23T16:40:15.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8531000000000001e-05 2024-04-23T16:40:15.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:40:15.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.757654+0000", 2024-04-23T16:40:15.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000249132 2024-04-23T16:40:15.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.858043+0000", 2024-04-23T16:40:15.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.100388963 2024-04-23T16:40:15.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.858117+0000", 2024-04-23T16:40:15.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4305000000000001e-05 2024-04-23T16:40:15.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:40:15.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.858202+0000", 2024-04-23T16:40:15.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4151000000000001e-05 2024-04-23T16:40:15.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.868822+0000", 2024-04-23T16:40:15.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010620076000000001 2024-04-23T16:40:15.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.933212+0000", 2024-04-23T16:40:15.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.064390728999999994 2024-04-23T16:40:15.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.933272+0000", 2024-04-23T16:40:15.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9135000000000002e-05 2024-04-23T16:40:15.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:33:38.935862+0000", 2024-04-23T16:40:15.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025908580000000001 2024-04-23T16:40:15.101 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.101 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.101 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.101 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:183867 5.a 5:5ad9927f:::benchmark_data_smithi188_113957_object30269:head [delete] snapc 0=[] ondisk+write+known_if_redirected e405)", 2024-04-23T16:40:15.101 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:14.979043+0000", 2024-04-23T16:40:15.101 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 360.09842622600002, 2024-04-23T16:40:15.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.21871197000000001, 2024-04-23T16:40:15.102 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.102 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.102 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.102 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.102 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:15.102 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:15.102 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 183867 2024-04-23T16:40:15.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.102 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979043+0000", 2024-04-23T16:40:15.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979043+0000", 2024-04-23T16:40:15.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979047+0000", 2024-04-23T16:40:15.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.422e-06 2024-04-23T16:40:15.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979058+0000", 2024-04-23T16:40:15.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1155000000000001e-05 2024-04-23T16:40:15.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979074+0000", 2024-04-23T16:40:15.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6198999999999999e-05 2024-04-23T16:40:15.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979085+0000", 2024-04-23T16:40:15.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0954e-05 2024-04-23T16:40:15.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.057904+0000", 2024-04-23T16:40:15.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078818887000000004 2024-04-23T16:40:15.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:40:15.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.057945+0000", 2024-04-23T16:40:15.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1523999999999997e-05 2024-04-23T16:40:15.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.063469+0000", 2024-04-23T16:40:15.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055238930000000002 2024-04-23T16:40:15.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.063542+0000", 2024-04-23T16:40:15.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2451000000000001e-05 2024-04-23T16:40:15.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:40:15.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.063635+0000", 2024-04-23T16:40:15.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3566999999999994e-05 2024-04-23T16:40:15.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.080527+0000", 2024-04-23T16:40:15.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016891587 2024-04-23T16:40:15.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.195578+0000", 2024-04-23T16:40:15.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.11505106700000001 2024-04-23T16:40:15.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.195620+0000", 2024-04-23T16:40:15.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2385000000000002e-05 2024-04-23T16:40:15.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.197755+0000", 2024-04-23T16:40:15.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021348790000000001 2024-04-23T16:40:15.107 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.107 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.107 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.107 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:183868 5.a 5:54aa4085:::benchmark_data_smithi188_113957_object30270:head [delete] snapc 0=[] ondisk+write+known_if_redirected e405)", 2024-04-23T16:40:15.107 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:14.979944+0000", 2024-04-23T16:40:15.107 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 360.09752539099998, 2024-04-23T16:40:15.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.221022303, 2024-04-23T16:40:15.107 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.107 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.108 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.108 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.108 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:15.108 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:15.108 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 183868 2024-04-23T16:40:15.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.108 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979944+0000", 2024-04-23T16:40:15.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979944+0000", 2024-04-23T16:40:15.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979945+0000", 2024-04-23T16:40:15.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.373e-06 2024-04-23T16:40:15.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979951+0000", 2024-04-23T16:40:15.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4999999999999999e-06 2024-04-23T16:40:15.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979963+0000", 2024-04-23T16:40:15.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.182e-05 2024-04-23T16:40:15.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.979970+0000", 2024-04-23T16:40:15.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6729999999999992e-06 2024-04-23T16:40:15.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.057974+0000", 2024-04-23T16:40:15.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078003804999999996 2024-04-23T16:40:15.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:40:15.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.058286+0000", 2024-04-23T16:40:15.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031136500000000002 2024-04-23T16:40:15.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.090326+0000", 2024-04-23T16:40:15.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032040043999999997 2024-04-23T16:40:15.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.090395+0000", 2024-04-23T16:40:15.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8946e-05 2024-04-23T16:40:15.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:40:15.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.090483+0000", 2024-04-23T16:40:15.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8832000000000006e-05 2024-04-23T16:40:15.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.109765+0000", 2024-04-23T16:40:15.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019281868000000001 2024-04-23T16:40:15.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.197826+0000", 2024-04-23T16:40:15.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.088060787000000001 2024-04-23T16:40:15.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.197882+0000", 2024-04-23T16:40:15.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5770999999999998e-05 2024-04-23T16:40:15.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.200966+0000", 2024-04-23T16:40:15.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003084519 2024-04-23T16:40:15.113 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.113 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.113 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.113 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:183875 5.a 5:56b558c0:::benchmark_data_smithi188_113957_object30277:head [delete] snapc 0=[] ondisk+write+known_if_redirected e405)", 2024-04-23T16:40:15.114 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:14.999858+0000", 2024-04-23T16:40:15.114 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 360.07761180599999, 2024-04-23T16:40:15.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.20427009400000001, 2024-04-23T16:40:15.114 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.114 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.114 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.114 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.114 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:15.114 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:15.114 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 183875 2024-04-23T16:40:15.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.114 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.999858+0000", 2024-04-23T16:40:15.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.999858+0000", 2024-04-23T16:40:15.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.999860+0000", 2024-04-23T16:40:15.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7460000000000001e-06 2024-04-23T16:40:15.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.999871+0000", 2024-04-23T16:40:15.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0336000000000001e-05 2024-04-23T16:40:15.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.999889+0000", 2024-04-23T16:40:15.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8607999999999999e-05 2024-04-23T16:40:15.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:14.999898+0000", 2024-04-23T16:40:15.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.276e-06 2024-04-23T16:40:15.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.058319+0000", 2024-04-23T16:40:15.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.058421147 2024-04-23T16:40:15.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:40:15.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.058550+0000", 2024-04-23T16:40:15.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00023164999999999999 2024-04-23T16:40:15.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.097650+0000", 2024-04-23T16:40:15.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.039099606000000002 2024-04-23T16:40:15.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.097751+0000", 2024-04-23T16:40:15.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010077300000000001 2024-04-23T16:40:15.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:40:15.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.097893+0000", 2024-04-23T16:40:15.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014181600000000001 2024-04-23T16:40:15.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.120527+0000", 2024-04-23T16:40:15.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022634399999999999 2024-04-23T16:40:15.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.201049+0000", 2024-04-23T16:40:15.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.080522257999999999 2024-04-23T16:40:15.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.201126+0000", 2024-04-23T16:40:15.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6848999999999996e-05 2024-04-23T16:40:15.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.204128+0000", 2024-04-23T16:40:15.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003001629 2024-04-23T16:40:15.120 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.120 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.120 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.120 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:183876 5.a 5:52b8ecf9:::benchmark_data_smithi188_113957_object30278:head [delete] snapc 0=[] ondisk+write+known_if_redirected e405)", 2024-04-23T16:40:15.120 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:15.000673+0000", 2024-04-23T16:40:15.120 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 360.076796715, 2024-04-23T16:40:15.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.21436903600000001, 2024-04-23T16:40:15.120 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.120 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.120 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.120 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.120 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:15.120 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:15.121 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 183876 2024-04-23T16:40:15.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.121 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.000673+0000", 2024-04-23T16:40:15.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.000673+0000", 2024-04-23T16:40:15.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.000674+0000", 2024-04-23T16:40:15.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7370000000000001e-06 2024-04-23T16:40:15.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.000681+0000", 2024-04-23T16:40:15.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7939999999999997e-06 2024-04-23T16:40:15.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.000693+0000", 2024-04-23T16:40:15.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1780000000000001e-05 2024-04-23T16:40:15.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.000703+0000", 2024-04-23T16:40:15.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5589999999999994e-06 2024-04-23T16:40:15.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.058583+0000", 2024-04-23T16:40:15.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.057880193000000003 2024-04-23T16:40:15.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:40:15.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.058799+0000", 2024-04-23T16:40:15.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021655599999999999 2024-04-23T16:40:15.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.106244+0000", 2024-04-23T16:40:15.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.047444457000000002 2024-04-23T16:40:15.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.106325+0000", 2024-04-23T16:40:15.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1497999999999998e-05 2024-04-23T16:40:15.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:40:15.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.106417+0000", 2024-04-23T16:40:15.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1558999999999995e-05 2024-04-23T16:40:15.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.124887+0000", 2024-04-23T16:40:15.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018470073 2024-04-23T16:40:15.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.212626+0000", 2024-04-23T16:40:15.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.087738732999999999 2024-04-23T16:40:15.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.212684+0000", 2024-04-23T16:40:15.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8588000000000002e-05 2024-04-23T16:40:15.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.215042+0000", 2024-04-23T16:40:15.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023575089999999998 2024-04-23T16:40:15.126 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.126 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.126 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.127 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:183879 5.a 5:530c738f:::benchmark_data_smithi188_113957_object30281:head [delete] snapc 0=[] ondisk+write+known_if_redirected e405)", 2024-04-23T16:40:15.127 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:15.015869+0000", 2024-04-23T16:40:15.127 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 360.06160025999998, 2024-04-23T16:40:15.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.202583191, 2024-04-23T16:40:15.127 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.127 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.127 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.127 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.127 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:15.127 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:15.127 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 183879 2024-04-23T16:40:15.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.127 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.015869+0000", 2024-04-23T16:40:15.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.015869+0000", 2024-04-23T16:40:15.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.015872+0000", 2024-04-23T16:40:15.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.689e-06 2024-04-23T16:40:15.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.015882+0000", 2024-04-23T16:40:15.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9159999999999996e-06 2024-04-23T16:40:15.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.015896+0000", 2024-04-23T16:40:15.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4431e-05 2024-04-23T16:40:15.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.015905+0000", 2024-04-23T16:40:15.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1989999999999995e-06 2024-04-23T16:40:15.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.058831+0000", 2024-04-23T16:40:15.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.042925824000000001 2024-04-23T16:40:15.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:40:15.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.059046+0000", 2024-04-23T16:40:15.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021425400000000001 2024-04-23T16:40:15.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.132876+0000", 2024-04-23T16:40:15.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.073830940999999997 2024-04-23T16:40:15.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.132936+0000", 2024-04-23T16:40:15.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9076999999999997e-05 2024-04-23T16:40:15.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:40:15.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.133011+0000", 2024-04-23T16:40:15.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5319000000000002e-05 2024-04-23T16:40:15.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.146665+0000", 2024-04-23T16:40:15.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013653748 2024-04-23T16:40:15.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.216832+0000", 2024-04-23T16:40:15.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.070167577999999994 2024-04-23T16:40:15.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.216886+0000", 2024-04-23T16:40:15.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3579999999999999e-05 2024-04-23T16:40:15.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.218452+0000", 2024-04-23T16:40:15.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001566635 2024-04-23T16:40:15.132 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.133 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.133 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.133 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:183883 5.a 5:561ddaeb:::benchmark_data_smithi188_113957_object30285:head [delete] snapc 0=[] ondisk+write+known_if_redirected e405)", 2024-04-23T16:40:15.133 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:15.030160+0000", 2024-04-23T16:40:15.133 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 360.04730952400001, 2024-04-23T16:40:15.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18660992900000001, 2024-04-23T16:40:15.133 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.133 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.133 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.133 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.133 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:15.133 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:15.133 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 183883 2024-04-23T16:40:15.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.134 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.030160+0000", 2024-04-23T16:40:15.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.030160+0000", 2024-04-23T16:40:15.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.030162+0000", 2024-04-23T16:40:15.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2220000000000001e-06 2024-04-23T16:40:15.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.030170+0000", 2024-04-23T16:40:15.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6890000000000004e-06 2024-04-23T16:40:15.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.030184+0000", 2024-04-23T16:40:15.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3862999999999999e-05 2024-04-23T16:40:15.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.030193+0000", 2024-04-23T16:40:15.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1800000000000002e-06 2024-04-23T16:40:15.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.059080+0000", 2024-04-23T16:40:15.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028887025 2024-04-23T16:40:15.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:40:15.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.059307+0000", 2024-04-23T16:40:15.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00022688600000000001 2024-04-23T16:40:15.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.129074+0000", 2024-04-23T16:40:15.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.069767423999999995 2024-04-23T16:40:15.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.129193+0000", 2024-04-23T16:40:15.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118529 2024-04-23T16:40:15.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:40:15.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.129348+0000", 2024-04-23T16:40:15.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000155284 2024-04-23T16:40:15.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.136011+0000", 2024-04-23T16:40:15.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0066625340000000003 2024-04-23T16:40:15.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.215122+0000", 2024-04-23T16:40:15.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.079111329999999994 2024-04-23T16:40:15.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.215179+0000", 2024-04-23T16:40:15.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7132999999999997e-05 2024-04-23T16:40:15.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.216770+0000", 2024-04-23T16:40:15.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0015908299999999999 2024-04-23T16:40:15.139 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.139 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.139 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.139 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:183888 5.a 5:5846d6fa:::benchmark_data_smithi188_113957_object30290:head [delete] snapc 0=[] ondisk+write+known_if_redirected e405)", 2024-04-23T16:40:15.139 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:15.045806+0000", 2024-04-23T16:40:15.139 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 360.03166386499998, 2024-04-23T16:40:15.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17485758000000001, 2024-04-23T16:40:15.139 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.139 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.140 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.140 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.140 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:15.140 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:15.140 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 183888 2024-04-23T16:40:15.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.140 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.045806+0000", 2024-04-23T16:40:15.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.045806+0000", 2024-04-23T16:40:15.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.045808+0000", 2024-04-23T16:40:15.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4980000000000001e-06 2024-04-23T16:40:15.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.045816+0000", 2024-04-23T16:40:15.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7360000000000001e-06 2024-04-23T16:40:15.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.045828+0000", 2024-04-23T16:40:15.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1824000000000001e-05 2024-04-23T16:40:15.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.045837+0000", 2024-04-23T16:40:15.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4520000000000003e-06 2024-04-23T16:40:15.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.059339+0000", 2024-04-23T16:40:15.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013501793 2024-04-23T16:40:15.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:40:15.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.059583+0000", 2024-04-23T16:40:15.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024406800000000001 2024-04-23T16:40:15.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.150992+0000", 2024-04-23T16:40:15.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.091408732000000006 2024-04-23T16:40:15.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.151067+0000", 2024-04-23T16:40:15.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5331000000000001e-05 2024-04-23T16:40:15.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:40:15.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.151169+0000", 2024-04-23T16:40:15.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101588 2024-04-23T16:40:15.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.164918+0000", 2024-04-23T16:40:15.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01374977 2024-04-23T16:40:15.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.218521+0000", 2024-04-23T16:40:15.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.053602859000000003 2024-04-23T16:40:15.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.218580+0000", 2024-04-23T16:40:15.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8273000000000003e-05 2024-04-23T16:40:15.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.220663+0000", 2024-04-23T16:40:15.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020836560000000001 2024-04-23T16:40:15.145 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.145 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.145 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.145 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:183892 5.a 5:5cedc568:::benchmark_data_smithi188_113957_object30294:head [delete] snapc 0=[] ondisk+write+known_if_redirected e405)", 2024-04-23T16:40:15.145 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:15.052916+0000", 2024-04-23T16:40:15.145 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 360.02455317900001, 2024-04-23T16:40:15.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17005552800000001, 2024-04-23T16:40:15.146 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.146 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.146 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.146 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.146 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:15.146 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:15.146 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 183892 2024-04-23T16:40:15.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.146 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.052916+0000", 2024-04-23T16:40:15.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.052916+0000", 2024-04-23T16:40:15.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.052919+0000", 2024-04-23T16:40:15.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.238e-06 2024-04-23T16:40:15.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.052924+0000", 2024-04-23T16:40:15.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4890000000000003e-06 2024-04-23T16:40:15.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.052936+0000", 2024-04-23T16:40:15.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1635e-05 2024-04-23T16:40:15.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.052945+0000", 2024-04-23T16:40:15.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3649999999999993e-06 2024-04-23T16:40:15.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.059618+0000", 2024-04-23T16:40:15.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0066725029999999998 2024-04-23T16:40:15.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:40:15.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.059833+0000", 2024-04-23T16:40:15.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021506800000000001 2024-04-23T16:40:15.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.170197+0000", 2024-04-23T16:40:15.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.110364221 2024-04-23T16:40:15.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.170272+0000", 2024-04-23T16:40:15.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5129999999999994e-05 2024-04-23T16:40:15.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:40:15.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.170371+0000", 2024-04-23T16:40:15.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9263000000000002e-05 2024-04-23T16:40:15.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.188683+0000", 2024-04-23T16:40:15.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018311874999999998 2024-04-23T16:40:15.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.220832+0000", 2024-04-23T16:40:15.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032148941 2024-04-23T16:40:15.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.220866+0000", 2024-04-23T16:40:15.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4357999999999999e-05 2024-04-23T16:40:15.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.222972+0000", 2024-04-23T16:40:15.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021054419999999999 2024-04-23T16:40:15.151 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.151 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.152 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.152 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:183896 5.a 5:56e6e14d:::benchmark_data_smithi188_113957_object30298:head [delete] snapc 0=[] ondisk+write+known_if_redirected e405)", 2024-04-23T16:40:15.152 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:15.061396+0000", 2024-04-23T16:40:15.152 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 360.01607338600002, 2024-04-23T16:40:15.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.21362087199999999, 2024-04-23T16:40:15.152 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.152 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.152 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.152 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.152 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:15.152 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:15.152 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 183896 2024-04-23T16:40:15.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.153 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.061396+0000", 2024-04-23T16:40:15.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.061396+0000", 2024-04-23T16:40:15.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.061398+0000", 2024-04-23T16:40:15.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6130000000000001e-06 2024-04-23T16:40:15.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.061403+0000", 2024-04-23T16:40:15.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1270000000000002e-06 2024-04-23T16:40:15.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.061411+0000", 2024-04-23T16:40:15.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3469999999999999e-06 2024-04-23T16:40:15.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.061418+0000", 2024-04-23T16:40:15.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8889999999999996e-06 2024-04-23T16:40:15.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.080478+0000", 2024-04-23T16:40:15.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019059646 2024-04-23T16:40:15.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:40:15.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.080517+0000", 2024-04-23T16:40:15.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8822999999999997e-05 2024-04-23T16:40:15.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.250424+0000", 2024-04-23T16:40:15.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.169907683 2024-04-23T16:40:15.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.250535+0000", 2024-04-23T16:40:15.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000111051 2024-04-23T16:40:15.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:40:15.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.250694+0000", 2024-04-23T16:40:15.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015892400000000001 2024-04-23T16:40:15.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.268933+0000", 2024-04-23T16:40:15.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018238800999999999 2024-04-23T16:40:15.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.272484+0000", 2024-04-23T16:40:15.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003551261 2024-04-23T16:40:15.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.272519+0000", 2024-04-23T16:40:15.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5141000000000001e-05 2024-04-23T16:40:15.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:15.275017+0000", 2024-04-23T16:40:15.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024975660000000001 2024-04-23T16:40:15.158 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.158 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.158 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.158 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:214988 5.a 5:5a3f8f01:::benchmark_data_smithi188_113957_object61390:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-23T16:40:15.158 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:35:01.259340+0000", 2024-04-23T16:40:15.158 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 313.81812923699999, 2024-04-23T16:40:15.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.181216351, 2024-04-23T16:40:15.158 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.158 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.158 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.159 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.159 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:15.159 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:15.159 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 214988 2024-04-23T16:40:15.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.159 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.259340+0000", 2024-04-23T16:40:15.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.259340+0000", 2024-04-23T16:40:15.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.259342+0000", 2024-04-23T16:40:15.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4419999999999999e-06 2024-04-23T16:40:15.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.259348+0000", 2024-04-23T16:40:15.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8950000000000003e-06 2024-04-23T16:40:15.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.259357+0000", 2024-04-23T16:40:15.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3570000000000004e-06 2024-04-23T16:40:15.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.259365+0000", 2024-04-23T16:40:15.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7260000000000002e-06 2024-04-23T16:40:15.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.275330+0000", 2024-04-23T16:40:15.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01596537 2024-04-23T16:40:15.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:40:15.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.275768+0000", 2024-04-23T16:40:15.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043770799999999998 2024-04-23T16:40:15.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.347968+0000", 2024-04-23T16:40:15.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.072199812000000002 2024-04-23T16:40:15.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.348040+0000", 2024-04-23T16:40:15.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2589000000000006e-05 2024-04-23T16:40:15.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:40:15.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.348130+0000", 2024-04-23T16:40:15.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9439999999999997e-05 2024-04-23T16:40:15.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.364397+0000", 2024-04-23T16:40:15.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016267674999999999 2024-04-23T16:40:15.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.437459+0000", 2024-04-23T16:40:15.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.073061423 2024-04-23T16:40:15.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.437521+0000", 2024-04-23T16:40:15.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2776999999999999e-05 2024-04-23T16:40:15.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.440557+0000", 2024-04-23T16:40:15.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030351369999999998 2024-04-23T16:40:15.164 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.164 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.164 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.164 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:214994 5.a 5:5c3b6503:::benchmark_data_smithi188_113957_object61396:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-23T16:40:15.164 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:35:01.260876+0000", 2024-04-23T16:40:15.164 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 313.81659364400002, 2024-04-23T16:40:15.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18260874299999999, 2024-04-23T16:40:15.165 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.165 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.165 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.165 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.165 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:15.165 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:15.165 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 214994 2024-04-23T16:40:15.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.165 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.260876+0000", 2024-04-23T16:40:15.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.260876+0000", 2024-04-23T16:40:15.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.260877+0000", 2024-04-23T16:40:15.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4139999999999999e-06 2024-04-23T16:40:15.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.260882+0000", 2024-04-23T16:40:15.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0889999999999999e-06 2024-04-23T16:40:15.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.260891+0000", 2024-04-23T16:40:15.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7790000000000001e-06 2024-04-23T16:40:15.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.260898+0000", 2024-04-23T16:40:15.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1099999999999997e-06 2024-04-23T16:40:15.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.275819+0000", 2024-04-23T16:40:15.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014921228999999999 2024-04-23T16:40:15.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:40:15.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276334+0000", 2024-04-23T16:40:15.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00051455200000000004 2024-04-23T16:40:15.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.356496+0000", 2024-04-23T16:40:15.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.080162232999999999 2024-04-23T16:40:15.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.356587+0000", 2024-04-23T16:40:15.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0388000000000003e-05 2024-04-23T16:40:15.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:40:15.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.356707+0000", 2024-04-23T16:40:15.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000120189 2024-04-23T16:40:15.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.372935+0000", 2024-04-23T16:40:15.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016228604000000001 2024-04-23T16:40:15.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.440628+0000", 2024-04-23T16:40:15.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067692691999999999 2024-04-23T16:40:15.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.440689+0000", 2024-04-23T16:40:15.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0781e-05 2024-04-23T16:40:15.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.443485+0000", 2024-04-23T16:40:15.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027956830000000002 2024-04-23T16:40:15.170 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.170 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.170 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.171 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:214998 5.a 5:5aa06674:::benchmark_data_smithi188_113957_object61400:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-23T16:40:15.171 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:35:01.271851+0000", 2024-04-23T16:40:15.171 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 313.80561820999998, 2024-04-23T16:40:15.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17414512200000001, 2024-04-23T16:40:15.171 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.171 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.171 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.171 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.171 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:15.171 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:15.171 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 214998 2024-04-23T16:40:15.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.171 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.271851+0000", 2024-04-23T16:40:15.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.271851+0000", 2024-04-23T16:40:15.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.271853+0000", 2024-04-23T16:40:15.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2280000000000001e-06 2024-04-23T16:40:15.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.271860+0000", 2024-04-23T16:40:15.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8770000000000004e-06 2024-04-23T16:40:15.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.271871+0000", 2024-04-23T16:40:15.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0287e-05 2024-04-23T16:40:15.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.271881+0000", 2024-04-23T16:40:15.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0202e-05 2024-04-23T16:40:15.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276381+0000", 2024-04-23T16:40:15.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004499818 2024-04-23T16:40:15.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:40:15.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276905+0000", 2024-04-23T16:40:15.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00052421599999999998 2024-04-23T16:40:15.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.364455+0000", 2024-04-23T16:40:15.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.087549765000000002 2024-04-23T16:40:15.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.364549+0000", 2024-04-23T16:40:15.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4581999999999997e-05 2024-04-23T16:40:15.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:40:15.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.364655+0000", 2024-04-23T16:40:15.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105801 2024-04-23T16:40:15.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.380441+0000", 2024-04-23T16:40:15.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015786372 2024-04-23T16:40:15.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.443553+0000", 2024-04-23T16:40:15.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.063111492000000005 2024-04-23T16:40:15.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.443605+0000", 2024-04-23T16:40:15.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2460999999999998e-05 2024-04-23T16:40:15.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.445996+0000", 2024-04-23T16:40:15.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023910210000000001 2024-04-23T16:40:15.177 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.177 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.177 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.177 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:215000 5.a 5:5746befc:::benchmark_data_smithi188_113957_object61402:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-23T16:40:15.177 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:35:01.276226+0000", 2024-04-23T16:40:15.177 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 313.80124346600002, 2024-04-23T16:40:15.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17179708599999999, 2024-04-23T16:40:15.177 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.177 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.177 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.177 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.177 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:15.178 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:15.178 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 215000 2024-04-23T16:40:15.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.178 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276226+0000", 2024-04-23T16:40:15.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276226+0000", 2024-04-23T16:40:15.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276228+0000", 2024-04-23T16:40:15.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0389999999999999e-06 2024-04-23T16:40:15.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276234+0000", 2024-04-23T16:40:15.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.049e-06 2024-04-23T16:40:15.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276245+0000", 2024-04-23T16:40:15.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0448000000000001e-05 2024-04-23T16:40:15.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276252+0000", 2024-04-23T16:40:15.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4529999999999996e-06 2024-04-23T16:40:15.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.276950+0000", 2024-04-23T16:40:15.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000698396 2024-04-23T16:40:15.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:40:15.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.277470+0000", 2024-04-23T16:40:15.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00051993799999999995 2024-04-23T16:40:15.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.372985+0000", 2024-04-23T16:40:15.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.095515172999999995 2024-04-23T16:40:15.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.373089+0000", 2024-04-23T16:40:15.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010385 2024-04-23T16:40:15.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:40:15.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.373186+0000", 2024-04-23T16:40:15.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6778000000000001e-05 2024-04-23T16:40:15.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.392441+0000", 2024-04-23T16:40:15.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019254513000000001 2024-04-23T16:40:15.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.446064+0000", 2024-04-23T16:40:15.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.053623404999999999 2024-04-23T16:40:15.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.446125+0000", 2024-04-23T16:40:15.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0590000000000001e-05 2024-04-23T16:40:15.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.448023+0000", 2024-04-23T16:40:15.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018984539999999999 2024-04-23T16:40:15.183 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.183 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.183 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.183 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:215010 5.a 5:5fcdf1a9:::benchmark_data_smithi188_113957_object61412:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-23T16:40:15.183 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:35:01.290493+0000", 2024-04-23T16:40:15.183 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 313.78697602, 2024-04-23T16:40:15.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.182786068, 2024-04-23T16:40:15.184 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.184 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.184 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.184 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.184 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:15.184 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:15.184 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 215010 2024-04-23T16:40:15.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.184 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.290493+0000", 2024-04-23T16:40:15.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.290493+0000", 2024-04-23T16:40:15.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.290495+0000", 2024-04-23T16:40:15.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9910000000000001e-06 2024-04-23T16:40:15.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.290502+0000", 2024-04-23T16:40:15.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3289999999999999e-06 2024-04-23T16:40:15.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.290511+0000", 2024-04-23T16:40:15.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5380000000000008e-06 2024-04-23T16:40:15.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.290518+0000", 2024-04-23T16:40:15.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.601e-06 2024-04-23T16:40:15.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.290572+0000", 2024-04-23T16:40:15.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4162000000000001e-05 2024-04-23T16:40:15.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:40:15.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.290615+0000", 2024-04-23T16:40:15.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2809000000000003e-05 2024-04-23T16:40:15.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.414270+0000", 2024-04-23T16:40:15.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.123654695 2024-04-23T16:40:15.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.414372+0000", 2024-04-23T16:40:15.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000102878 2024-04-23T16:40:15.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:40:15.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.414511+0000", 2024-04-23T16:40:15.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000138992 2024-04-23T16:40:15.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.429033+0000", 2024-04-23T16:40:15.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014521729000000001 2024-04-23T16:40:15.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.470520+0000", 2024-04-23T16:40:15.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.041487201000000001 2024-04-23T16:40:15.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.470578+0000", 2024-04-23T16:40:15.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8050999999999997e-05 2024-04-23T16:40:15.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:01.473280+0000", 2024-04-23T16:40:15.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002701092 2024-04-23T16:40:15.189 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.189 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.189 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.189 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:251104 5.a 5:56b6a247:::benchmark_data_smithi188_113957_object97506:head [delete] snapc 0=[] ondisk+write+known_if_redirected e426)", 2024-04-23T16:40:15.189 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:35:49.743007+0000", 2024-04-23T16:40:15.189 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 265.334461965, 2024-04-23T16:40:15.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18092238999999999, 2024-04-23T16:40:15.189 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.190 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.190 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.190 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.190 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:15.190 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:15.190 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 251104 2024-04-23T16:40:15.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.190 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.743007+0000", 2024-04-23T16:40:15.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.743007+0000", 2024-04-23T16:40:15.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.743009+0000", 2024-04-23T16:40:15.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9850000000000001e-06 2024-04-23T16:40:15.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.743015+0000", 2024-04-23T16:40:15.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6740000000000002e-06 2024-04-23T16:40:15.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.743027+0000", 2024-04-23T16:40:15.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1541e-05 2024-04-23T16:40:15.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.743037+0000", 2024-04-23T16:40:15.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0103e-05 2024-04-23T16:40:15.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.751004+0000", 2024-04-23T16:40:15.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0079672489999999992 2024-04-23T16:40:15.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:40:15.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.751074+0000", 2024-04-23T16:40:15.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9517000000000005e-05 2024-04-23T16:40:15.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.837979+0000", 2024-04-23T16:40:15.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.086905300000000005 2024-04-23T16:40:15.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.838074+0000", 2024-04-23T16:40:15.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5094999999999997e-05 2024-04-23T16:40:15.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:40:15.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.838233+0000", 2024-04-23T16:40:15.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0001593 2024-04-23T16:40:15.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.848903+0000", 2024-04-23T16:40:15.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010669621000000001 2024-04-23T16:40:15.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.922168+0000", 2024-04-23T16:40:15.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.073264971999999998 2024-04-23T16:40:15.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.922230+0000", 2024-04-23T16:40:15.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2403999999999994e-05 2024-04-23T16:40:15.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.923930+0000", 2024-04-23T16:40:15.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001699629 2024-04-23T16:40:15.195 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.195 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.195 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.195 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:251113 5.a 5:59bedb89:::benchmark_data_smithi188_113957_object97515:head [delete] snapc 0=[] ondisk+write+known_if_redirected e426)", 2024-04-23T16:40:15.195 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:35:49.753200+0000", 2024-04-23T16:40:15.195 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 265.32426956500001, 2024-04-23T16:40:15.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.17198295899999999, 2024-04-23T16:40:15.196 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.196 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.196 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.196 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.196 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:15.196 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:15.196 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 251113 2024-04-23T16:40:15.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.196 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.753200+0000", 2024-04-23T16:40:15.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.753200+0000", 2024-04-23T16:40:15.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.753202+0000", 2024-04-23T16:40:15.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0870000000000001e-06 2024-04-23T16:40:15.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.753209+0000", 2024-04-23T16:40:15.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7830000000000001e-06 2024-04-23T16:40:15.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.753221+0000", 2024-04-23T16:40:15.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1764e-05 2024-04-23T16:40:15.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.753227+0000", 2024-04-23T16:40:15.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4520000000000002e-06 2024-04-23T16:40:15.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.756278+0000", 2024-04-23T16:40:15.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030510120000000001 2024-04-23T16:40:15.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for degraded object", 2024-04-23T16:40:15.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.756316+0000", 2024-04-23T16:40:15.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8209e-05 2024-04-23T16:40:15.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.843902+0000", 2024-04-23T16:40:15.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.087585619000000003 2024-04-23T16:40:15.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.844005+0000", 2024-04-23T16:40:15.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000103303 2024-04-23T16:40:15.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:40:15.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.844170+0000", 2024-04-23T16:40:15.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016467000000000001 2024-04-23T16:40:15.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.855135+0000", 2024-04-23T16:40:15.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010965449 2024-04-23T16:40:15.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.923999+0000", 2024-04-23T16:40:15.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.068863419999999995 2024-04-23T16:40:15.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.924054+0000", 2024-04-23T16:40:15.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5470000000000003e-05 2024-04-23T16:40:15.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:35:49.925183+0000", 2024-04-23T16:40:15.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0011287210000000001 2024-04-23T16:40:15.201 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.201 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.201 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.201 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16308.0:161912 6.1 6:846915cb:::benchmark_data_smithi188_130221_object7451:head [delete] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-23T16:40:15.201 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:39:08.519057+0000", 2024-04-23T16:40:15.201 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 66.558412739999994, 2024-04-23T16:40:15.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.16532101099999999, 2024-04-23T16:40:15.202 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:15.202 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:15.202 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:15.202 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:15.202 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16308", 2024-04-23T16:40:15.202 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:40:15.202 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 161912 2024-04-23T16:40:15.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.202 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:15.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:15.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:08.519057+0000", 2024-04-23T16:40:15.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:15.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:08.519057+0000", 2024-04-23T16:40:15.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:15.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:15.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:08.519058+0000", 2024-04-23T16:40:15.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5570000000000001e-06 2024-04-23T16:40:15.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:15.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:08.519063+0000", 2024-04-23T16:40:15.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9810000000000003e-06 2024-04-23T16:40:15.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:15.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:08.519074+0000", 2024-04-23T16:40:15.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.044e-05 2024-04-23T16:40:15.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:15.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:08.519079+0000", 2024-04-23T16:40:15.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6389999999999997e-06 2024-04-23T16:40:15.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:15.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:08.519131+0000", 2024-04-23T16:40:15.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1419999999999999e-05 2024-04-23T16:40:15.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:15.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:08.519214+0000", 2024-04-23T16:40:15.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3269999999999994e-05 2024-04-23T16:40:15.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:40:15.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:08.519282+0000", 2024-04-23T16:40:15.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8347000000000001e-05 2024-04-23T16:40:15.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:15.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:08.522602+0000", 2024-04-23T16:40:15.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033195109999999998 2024-04-23T16:40:15.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:15.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:08.684273+0000", 2024-04-23T16:40:15.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.16167064 2024-04-23T16:40:15.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:15.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:08.684329+0000", 2024-04-23T16:40:15.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6576000000000002e-05 2024-04-23T16:40:15.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:15.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:15.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:15.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:08.684378+0000", 2024-04-23T16:40:15.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.863e-05 2024-04-23T16:40:15.206 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.206 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.206 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.206 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:15.207 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:15.207 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:40:15.207 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:40:15.208 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:14 smithi188 ceph-mon[44101]: osdmap e510: 12 total, 12 up, 10 in 2024-04-23T16:40:15.208 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:14 smithi103 ceph-mon[39629]: osdmap e510: 12 total, 12 up, 10 in 2024-04-23T16:40:15.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:15.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:40:15.309 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:40:15.351 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:14 smithi148 ceph-mon[44520]: osdmap e510: 12 total, 12 up, 10 in 2024-04-23T16:40:15.353 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:15.385 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:40:15.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:15.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:40:15.496 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:40:15.523 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:15.543 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:40:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:15.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:40:15.647 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:40:15.673 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:15.692 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:40:15.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:15.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:40:15.796 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:40:15.797 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-23T16:40:15.906 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:16.033 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:40:16.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:16.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:40:16.084 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:40:16.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:15 smithi148 ceph-mon[44520]: pgmap v1430: 27 pgs: 1 remapped+peering, 2 active+remapped+backfilling, 24 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 827 op/s; 1479/178964 objects misplaced (0.826%); 9.5 MiB/s, 152 objects/s recovering 2024-04-23T16:40:16.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:15 smithi148 ceph-mon[44520]: osdmap e511: 12 total, 12 up, 10 in 2024-04-23T16:40:16.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:15 smithi148 ceph-mon[44520]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-23T16:40:16.147 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:40:16.186 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:15 smithi188 ceph-mon[44101]: pgmap v1430: 27 pgs: 1 remapped+peering, 2 active+remapped+backfilling, 24 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 827 op/s; 1479/178964 objects misplaced (0.826%); 9.5 MiB/s, 152 objects/s recovering 2024-04-23T16:40:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:15 smithi188 ceph-mon[44101]: osdmap e511: 12 total, 12 up, 10 in 2024-04-23T16:40:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:15 smithi188 ceph-mon[44101]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-23T16:40:16.227 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:40:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:16.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:40:16.334 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:40:16.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:15 smithi103 ceph-mon[39629]: pgmap v1430: 27 pgs: 1 remapped+peering, 2 active+remapped+backfilling, 24 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 827 op/s; 1479/178964 objects misplaced (0.826%); 9.5 MiB/s, 152 objects/s recovering 2024-04-23T16:40:16.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:15 smithi103 ceph-mon[39629]: osdmap e511: 12 total, 12 up, 10 in 2024-04-23T16:40:16.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:15 smithi103 ceph-mon[39629]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-23T16:40:16.365 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:16.528 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:40:16.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:16.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:40:16.555 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:40:16.633 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:40:16.700 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:16.742 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:40:16.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:16.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:40:16.851 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:40:16.890 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:16.922 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:40:16.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:16.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:40:17.028 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:40:17.071 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:17.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:16 smithi103 ceph-mon[39629]: osdmap e512: 12 total, 12 up, 10 in 2024-04-23T16:40:17.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:16 smithi148 ceph-mon[44520]: osdmap e512: 12 total, 12 up, 10 in 2024-04-23T16:40:17.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:16 smithi188 ceph-mon[44101]: osdmap e512: 12 total, 12 up, 10 in 2024-04-23T16:40:17.628 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:40:17.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:17.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:40:17.734 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:40:17.789 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:17.832 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:40:17.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:17.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:40:17.937 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:40:18.011 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:18.044 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:40:18.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:18.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:40:18.057 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:17 smithi148 ceph-mon[44520]: pgmap v1433: 27 pgs: 1 remapped+peering, 1 active+clean+remapped, 25 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 2.5 KiB/s rd, 67 KiB/s wr, 953 op/s; 23 MiB/s, 360 objects/s recovering 2024-04-23T16:40:18.057 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:17 smithi148 ceph-mon[44520]: osdmap e513: 12 total, 12 up, 10 in 2024-04-23T16:40:18.157 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:40:18.204 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:18.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:17 smithi188 ceph-mon[44101]: pgmap v1433: 27 pgs: 1 remapped+peering, 1 active+clean+remapped, 25 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 2.5 KiB/s rd, 67 KiB/s wr, 953 op/s; 23 MiB/s, 360 objects/s recovering 2024-04-23T16:40:18.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:17 smithi188 ceph-mon[44101]: osdmap e513: 12 total, 12 up, 10 in 2024-04-23T16:40:18.245 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:40:18.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:18.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:40:18.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:17 smithi103 ceph-mon[39629]: pgmap v1433: 27 pgs: 1 remapped+peering, 1 active+clean+remapped, 25 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 2.5 KiB/s rd, 67 KiB/s wr, 953 op/s; 23 MiB/s, 360 objects/s recovering 2024-04-23T16:40:18.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:17 smithi103 ceph-mon[39629]: osdmap e513: 12 total, 12 up, 10 in 2024-04-23T16:40:18.351 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:40:18.399 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:18.431 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:40:18.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:18.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:40:18.544 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:40:18.555 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:40:18.593 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:18.631 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:40:18.631 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:40:18.631 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.631 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16308.0:225229 6.f 6:f46470fc:::benchmark_data_smithi188_130221_object70768:head [delete] snapc 0=[] ondisk+write+known_if_redirected e513)", 2024-04-23T16:40:18.631 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:40:18.621251+0000", 2024-04-23T16:40:18.631 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0094678590000000003, 2024-04-23T16:40:18.632 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0094889699999999993, 2024-04-23T16:40:18.632 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:18.632 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:18.632 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:40:18.632 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:18.632 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16308", 2024-04-23T16:40:18.632 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:40:18.632 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 225229 2024-04-23T16:40:18.632 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:18.632 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:18.632 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.632 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:18.632 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:40:18.621251+0000", 2024-04-23T16:40:18.632 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:18.632 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:18.632 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.633 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:18.633 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:40:18.621251+0000", 2024-04-23T16:40:18.633 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:18.633 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:18.633 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.633 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:18.633 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:40:18.621252+0000", 2024-04-23T16:40:18.633 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.685e-06 2024-04-23T16:40:18.633 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:18.633 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.633 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:18.633 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:40:18.621256+0000", 2024-04-23T16:40:18.633 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9700000000000001e-06 2024-04-23T16:40:18.633 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:18.633 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.633 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:18.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:40:18.621266+0000", 2024-04-23T16:40:18.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3379999999999994e-06 2024-04-23T16:40:18.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:18.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.634 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:18.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:40:18.621271+0000", 2024-04-23T16:40:18.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4580000000000001e-06 2024-04-23T16:40:18.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:18.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.634 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:18.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:40:18.621309+0000", 2024-04-23T16:40:18.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8040999999999997e-05 2024-04-23T16:40:18.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:18.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.635 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:18.635 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:40:18.621392+0000", 2024-04-23T16:40:18.635 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2628999999999998e-05 2024-04-23T16:40:18.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:18.635 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.635 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:40:18.635 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:40:18.621460+0000", 2024-04-23T16:40:18.635 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8399999999999996e-05 2024-04-23T16:40:18.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:18.635 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.635 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:18.635 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:40:18.625198+0000", 2024-04-23T16:40:18.635 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003737616 2024-04-23T16:40:18.635 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:18.635 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:18.635 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:18.636 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:18.636 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.636 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.16308.0:225234 6.9 e513/455 6:915a2227:::benchmark_data_smithi188_130221_object70773:head v 513'14024, mlcod=513'14022)", 2024-04-23T16:40:18.636 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:40:18.630092+0000", 2024-04-23T16:40:18.636 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.00062610599999999997, 2024-04-23T16:40:18.636 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070147899999999997, 2024-04-23T16:40:18.636 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:18.636 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:18.636 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-04-23T16:40:18.636 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:18.636 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.636 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:18.636 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:40:18.630092+0000", 2024-04-23T16:40:18.636 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:18.636 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:18.636 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.637 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:18.637 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:40:18.630092+0000", 2024-04-23T16:40:18.637 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:18.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:18.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.637 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:18.637 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:40:18.630099+0000", 2024-04-23T16:40:18.637 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5150000000000003e-06 2024-04-23T16:40:18.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:18.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.637 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:18.637 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:40:18.630107+0000", 2024-04-23T16:40:18.637 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1720000000000007e-06 2024-04-23T16:40:18.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:18.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.638 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:18.638 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:40:18.630117+0000", 2024-04-23T16:40:18.638 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8239999999999995e-06 2024-04-23T16:40:18.638 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:18.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.638 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:18.638 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:40:18.630123+0000", 2024-04-23T16:40:18.638 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7379999999999999e-06 2024-04-23T16:40:18.638 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:18.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.638 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:18.638 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:40:18.630167+0000", 2024-04-23T16:40:18.638 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4088999999999997e-05 2024-04-23T16:40:18.638 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:18.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:18.639 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:18.639 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:40:18.630181+0000", 2024-04-23T16:40:18.639 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4525000000000001e-05 2024-04-23T16:40:18.639 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:18.639 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:18.639 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:18.639 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:18.639 INFO:teuthology.orchestra.run.smithi188.stdout: ], 2024-04-23T16:40:18.639 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 2 2024-04-23T16:40:18.639 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:40:18.855 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:40:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:18.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:40:18.953 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.6 to 1 (10000) 2024-04-23T16:40:18.963 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:40:19.004 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:19.026 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:40:19.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:19.030 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:40:19.131 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:40:19.183 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/827843282' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:40:19.184 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {auth=35,config=2,health=12,kv=51} crc {auth=691748081,config=2679774077,health=3230865504,kv=2096937162}) 2024-04-23T16:40:19.184 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=2009567873,logm=1638915236}) 2024-04-23T16:40:19.184 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=702018235}) 2024-04-23T16:40:19.184 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=173251056}) 2024-04-23T16:40:19.184 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1587372624}) 2024-04-23T16:40:19.184 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1524818448}) 2024-04-23T16:40:19.184 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1038944430}) 2024-04-23T16:40:19.184 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3551981901}) 2024-04-23T16:40:19.184 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=815781885}) 2024-04-23T16:40:19.184 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4175066348}) 2024-04-23T16:40:19.184 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3432507998}) 2024-04-23T16:40:19.184 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=294478811}) 2024-04-23T16:40:19.184 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2927467537}) 2024-04-23T16:40:19.184 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1006197168}) 2024-04-23T16:40:19.185 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2269490736}) 2024-04-23T16:40:19.185 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3300394305}) 2024-04-23T16:40:19.185 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2039753178}) 2024-04-23T16:40:19.185 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936457999}) 2024-04-23T16:40:19.185 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2462907123}) 2024-04-23T16:40:19.185 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=297777979}) 2024-04-23T16:40:19.185 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3884827595}) 2024-04-23T16:40:19.185 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=925948782}) 2024-04-23T16:40:19.185 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4087077356}) 2024-04-23T16:40:19.185 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1708629641}) 2024-04-23T16:40:19.185 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4277549604}) 2024-04-23T16:40:19.185 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3894017690}) 2024-04-23T16:40:19.185 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1964382640}) 2024-04-23T16:40:19.185 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3604205921}) 2024-04-23T16:40:19.185 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=424419643}) 2024-04-23T16:40:19.186 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2050266225}) 2024-04-23T16:40:19.186 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4110267170}) 2024-04-23T16:40:19.186 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2083218364}) 2024-04-23T16:40:19.186 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1088742049}) 2024-04-23T16:40:19.186 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2976777457}) 2024-04-23T16:40:19.186 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3333445736}) 2024-04-23T16:40:19.186 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3096402485}) 2024-04-23T16:40:19.186 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3989572212}) 2024-04-23T16:40:19.187 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231338298}) 2024-04-23T16:40:19.187 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=959351190}) 2024-04-23T16:40:19.187 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2132669668}) 2024-04-23T16:40:19.187 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=585629724}) 2024-04-23T16:40:19.187 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2084629441}) 2024-04-23T16:40:19.187 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2641242670}) 2024-04-23T16:40:19.187 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1997876616}) 2024-04-23T16:40:19.187 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1906603013}) 2024-04-23T16:40:19.187 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1985028378}) 2024-04-23T16:40:19.187 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1838907771}) 2024-04-23T16:40:19.187 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=324219988}) 2024-04-23T16:40:19.187 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1367112687}) 2024-04-23T16:40:19.187 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4222110393}) 2024-04-23T16:40:19.187 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=807902138}) 2024-04-23T16:40:19.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4219119351}) 2024-04-23T16:40:19.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=123926849}) 2024-04-23T16:40:19.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=686534606}) 2024-04-23T16:40:19.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=131147719}) 2024-04-23T16:40:19.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2858583838}) 2024-04-23T16:40:19.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2249011576}) 2024-04-23T16:40:19.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3094938919}) 2024-04-23T16:40:19.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2824720286}) 2024-04-23T16:40:19.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=971687370}) 2024-04-23T16:40:19.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2531681425}) 2024-04-23T16:40:19.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2180719270}) 2024-04-23T16:40:19.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=937665479}) 2024-04-23T16:40:19.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=918165132}) 2024-04-23T16:40:19.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=322860531}) 2024-04-23T16:40:19.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3801891149}) 2024-04-23T16:40:19.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=48,mdsmap=3,mgr=19,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=12} crc {logm=3009694577,mdsmap=2840952586,mgr=1765502803,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=3890172374,mon_config_key=2335023530}) 2024-04-23T16:40:19.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=174379660}) 2024-04-23T16:40:19.189 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1877809579}) 2024-04-23T16:40:19.189 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=75,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=7} crc {mon_config_key=480456263,monmap=2133572193,osd_metadata=1962451961,osd_pg_creating=1313992078,osdmap=2269969647}) 2024-04-23T16:40:19.189 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1464215148}) 2024-04-23T16:40:19.189 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:18 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=18} crc {osdmap=1073405249}) 2024-04-23T16:40:19.189 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/827843282' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:40:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {auth=35,config=2,health=12,kv=51} crc {auth=691748081,config=2679774077,health=3230865504,kv=2096937162}) 2024-04-23T16:40:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=2009567873,logm=1638915236}) 2024-04-23T16:40:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=702018235}) 2024-04-23T16:40:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=173251056}) 2024-04-23T16:40:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1587372624}) 2024-04-23T16:40:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1524818448}) 2024-04-23T16:40:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1038944430}) 2024-04-23T16:40:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3551981901}) 2024-04-23T16:40:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=815781885}) 2024-04-23T16:40:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4175066348}) 2024-04-23T16:40:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3432507998}) 2024-04-23T16:40:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=294478811}) 2024-04-23T16:40:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2927467537}) 2024-04-23T16:40:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1006197168}) 2024-04-23T16:40:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2269490736}) 2024-04-23T16:40:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3300394305}) 2024-04-23T16:40:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2039753178}) 2024-04-23T16:40:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936457999}) 2024-04-23T16:40:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2462907123}) 2024-04-23T16:40:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=297777979}) 2024-04-23T16:40:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3884827595}) 2024-04-23T16:40:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=925948782}) 2024-04-23T16:40:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4087077356}) 2024-04-23T16:40:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1708629641}) 2024-04-23T16:40:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4277549604}) 2024-04-23T16:40:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3894017690}) 2024-04-23T16:40:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1964382640}) 2024-04-23T16:40:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3604205921}) 2024-04-23T16:40:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=424419643}) 2024-04-23T16:40:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2050266225}) 2024-04-23T16:40:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4110267170}) 2024-04-23T16:40:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2083218364}) 2024-04-23T16:40:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1088742049}) 2024-04-23T16:40:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2976777457}) 2024-04-23T16:40:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3333445736}) 2024-04-23T16:40:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3096402485}) 2024-04-23T16:40:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3989572212}) 2024-04-23T16:40:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231338298}) 2024-04-23T16:40:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=959351190}) 2024-04-23T16:40:19.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2132669668}) 2024-04-23T16:40:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=585629724}) 2024-04-23T16:40:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2084629441}) 2024-04-23T16:40:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2641242670}) 2024-04-23T16:40:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1997876616}) 2024-04-23T16:40:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1906603013}) 2024-04-23T16:40:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1985028378}) 2024-04-23T16:40:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1838907771}) 2024-04-23T16:40:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=324219988}) 2024-04-23T16:40:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1367112687}) 2024-04-23T16:40:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4222110393}) 2024-04-23T16:40:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=807902138}) 2024-04-23T16:40:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4219119351}) 2024-04-23T16:40:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=123926849}) 2024-04-23T16:40:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=686534606}) 2024-04-23T16:40:19.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=131147719}) 2024-04-23T16:40:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2858583838}) 2024-04-23T16:40:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2249011576}) 2024-04-23T16:40:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3094938919}) 2024-04-23T16:40:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2824720286}) 2024-04-23T16:40:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=971687370}) 2024-04-23T16:40:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2531681425}) 2024-04-23T16:40:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2180719270}) 2024-04-23T16:40:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=937665479}) 2024-04-23T16:40:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=918165132}) 2024-04-23T16:40:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=322860531}) 2024-04-23T16:40:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3801891149}) 2024-04-23T16:40:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=48,mdsmap=3,mgr=19,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=12} crc {logm=3009694577,mdsmap=2840952586,mgr=1765502803,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=3890172374,mon_config_key=2335023530}) 2024-04-23T16:40:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=174379660}) 2024-04-23T16:40:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1877809579}) 2024-04-23T16:40:19.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=75,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=7} crc {mon_config_key=480456263,monmap=2133572193,osd_metadata=1962451961,osd_pg_creating=1313992078,osdmap=2269969647}) 2024-04-23T16:40:19.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1464215148}) 2024-04-23T16:40:19.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:18 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=18} crc {osdmap=1073405249}) 2024-04-23T16:40:19.372 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-23T16:40:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/827843282' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:40:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {auth=35,config=2,health=12,kv=51} crc {auth=691748081,config=2679774077,health=3230865504,kv=2096937162}) 2024-04-23T16:40:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=2009567873,logm=1638915236}) 2024-04-23T16:40:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=702018235}) 2024-04-23T16:40:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=173251056}) 2024-04-23T16:40:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1587372624}) 2024-04-23T16:40:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1524818448}) 2024-04-23T16:40:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1038944430}) 2024-04-23T16:40:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3551981901}) 2024-04-23T16:40:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=815781885}) 2024-04-23T16:40:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4175066348}) 2024-04-23T16:40:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3432507998}) 2024-04-23T16:40:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=294478811}) 2024-04-23T16:40:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2927467537}) 2024-04-23T16:40:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1006197168}) 2024-04-23T16:40:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2269490736}) 2024-04-23T16:40:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3300394305}) 2024-04-23T16:40:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2039753178}) 2024-04-23T16:40:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936457999}) 2024-04-23T16:40:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2462907123}) 2024-04-23T16:40:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=297777979}) 2024-04-23T16:40:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3884827595}) 2024-04-23T16:40:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=925948782}) 2024-04-23T16:40:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4087077356}) 2024-04-23T16:40:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1708629641}) 2024-04-23T16:40:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4277549604}) 2024-04-23T16:40:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3894017690}) 2024-04-23T16:40:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1964382640}) 2024-04-23T16:40:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3604205921}) 2024-04-23T16:40:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=424419643}) 2024-04-23T16:40:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2050266225}) 2024-04-23T16:40:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4110267170}) 2024-04-23T16:40:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2083218364}) 2024-04-23T16:40:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1088742049}) 2024-04-23T16:40:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2976777457}) 2024-04-23T16:40:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3333445736}) 2024-04-23T16:40:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3096402485}) 2024-04-23T16:40:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3989572212}) 2024-04-23T16:40:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231338298}) 2024-04-23T16:40:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=959351190}) 2024-04-23T16:40:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2132669668}) 2024-04-23T16:40:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=585629724}) 2024-04-23T16:40:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2084629441}) 2024-04-23T16:40:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2641242670}) 2024-04-23T16:40:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1997876616}) 2024-04-23T16:40:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1906603013}) 2024-04-23T16:40:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1985028378}) 2024-04-23T16:40:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1838907771}) 2024-04-23T16:40:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=324219988}) 2024-04-23T16:40:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1367112687}) 2024-04-23T16:40:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4222110393}) 2024-04-23T16:40:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=807902138}) 2024-04-23T16:40:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4219119351}) 2024-04-23T16:40:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=123926849}) 2024-04-23T16:40:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=686534606}) 2024-04-23T16:40:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=131147719}) 2024-04-23T16:40:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2858583838}) 2024-04-23T16:40:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2249011576}) 2024-04-23T16:40:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3094938919}) 2024-04-23T16:40:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2824720286}) 2024-04-23T16:40:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=971687370}) 2024-04-23T16:40:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2531681425}) 2024-04-23T16:40:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2180719270}) 2024-04-23T16:40:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=937665479}) 2024-04-23T16:40:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=918165132}) 2024-04-23T16:40:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=322860531}) 2024-04-23T16:40:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3801891149}) 2024-04-23T16:40:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=48,mdsmap=3,mgr=19,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=12} crc {logm=3009694577,mdsmap=2840952586,mgr=1765502803,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=3890172374,mon_config_key=2335023530}) 2024-04-23T16:40:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=174379660}) 2024-04-23T16:40:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1877809579}) 2024-04-23T16:40:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=75,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=7} crc {mon_config_key=480456263,monmap=2133572193,osd_metadata=1962451961,osd_pg_creating=1313992078,osdmap=2269969647}) 2024-04-23T16:40:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1464215148}) 2024-04-23T16:40:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:18 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=18} crc {osdmap=1073405249}) 2024-04-23T16:40:19.728 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:40:20.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:19 smithi188 ceph-mon[44101]: pgmap v1435: 27 pgs: 1 active+clean+remapped, 26 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 2.0 KiB/s rd, 55 KiB/s wr, 815 op/s; 18 MiB/s, 294 objects/s recovering 2024-04-23T16:40:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:19 smithi188 ceph-mon[44101]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-23T16:40:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:19 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/827843282' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:40:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:19 smithi188 ceph-mon[44101]: osdmap e514: 12 total, 12 up, 10 in 2024-04-23T16:40:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:19 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3233006895' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:40:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:19 smithi188 ceph-mon[44101]: from='client.26434 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:40:20.293 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:19 smithi103 ceph-mon[39629]: pgmap v1435: 27 pgs: 1 active+clean+remapped, 26 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 2.0 KiB/s rd, 55 KiB/s wr, 815 op/s; 18 MiB/s, 294 objects/s recovering 2024-04-23T16:40:20.293 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:19 smithi103 ceph-mon[39629]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-23T16:40:20.293 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:19 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/827843282' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:40:20.293 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:19 smithi103 ceph-mon[39629]: osdmap e514: 12 total, 12 up, 10 in 2024-04-23T16:40:20.293 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:19 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3233006895' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:40:20.294 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:19 smithi103 ceph-mon[39629]: from='client.26434 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:40:20.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:19 smithi148 ceph-mon[44520]: pgmap v1435: 27 pgs: 1 active+clean+remapped, 26 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 2.0 KiB/s rd, 55 KiB/s wr, 815 op/s; 18 MiB/s, 294 objects/s recovering 2024-04-23T16:40:20.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:19 smithi148 ceph-mon[44520]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-23T16:40:20.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:19 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/827843282' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:40:20.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:19 smithi148 ceph-mon[44520]: osdmap e514: 12 total, 12 up, 10 in 2024-04-23T16:40:20.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:19 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3233006895' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:40:20.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:19 smithi148 ceph-mon[44520]: from='client.26434 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:40:20.494 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:40:21.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:20 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:40:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:20 smithi188 ceph-mon[44101]: from='client.26434 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:40:21.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:20 smithi188 ceph-mon[44101]: osdmap e515: 12 total, 12 up, 10 in 2024-04-23T16:40:21.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:20 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:40:21.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:20 smithi148 ceph-mon[44520]: from='client.26434 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:40:21.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:20 smithi148 ceph-mon[44520]: osdmap e515: 12 total, 12 up, 10 in 2024-04-23T16:40:21.381 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:20 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:40:21.382 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:20 smithi103 ceph-mon[39629]: from='client.26434 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:40:21.382 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:20 smithi103 ceph-mon[39629]: osdmap e515: 12 total, 12 up, 10 in 2024-04-23T16:40:21.851 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:40:21.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:21.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:40:21.880 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-23T16:40:21.957 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:40:22.010 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:22.047 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:40:22.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:22.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:40:22.079 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:40:22.080 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:40:22.080 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:40:22.080 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:40:22.080 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:40:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:21 smithi103 ceph-mon[39629]: 6.6 scrub starts 2024-04-23T16:40:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:21 smithi103 ceph-mon[39629]: pgmap v1438: 27 pgs: 27 active+clean; 5.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 633 op/s 2024-04-23T16:40:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:21 smithi103 ceph-mon[39629]: 6.1 scrub starts 2024-04-23T16:40:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:21 smithi103 ceph-mon[39629]: 1.8 scrub starts 2024-04-23T16:40:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:21 smithi103 ceph-mon[39629]: 1.8 scrub ok 2024-04-23T16:40:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:21 smithi103 ceph-mon[39629]: 6.1 scrub ok 2024-04-23T16:40:22.152 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:40:22.185 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:21 smithi188 ceph-mon[44101]: 6.6 scrub starts 2024-04-23T16:40:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:21 smithi188 ceph-mon[44101]: pgmap v1438: 27 pgs: 27 active+clean; 5.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 633 op/s 2024-04-23T16:40:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:21 smithi188 ceph-mon[44101]: 6.1 scrub starts 2024-04-23T16:40:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:21 smithi188 ceph-mon[44101]: 1.8 scrub starts 2024-04-23T16:40:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:21 smithi188 ceph-mon[44101]: 1.8 scrub ok 2024-04-23T16:40:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:21 smithi188 ceph-mon[44101]: 6.1 scrub ok 2024-04-23T16:40:22.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:21 smithi148 ceph-mon[44520]: 6.6 scrub starts 2024-04-23T16:40:22.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:21 smithi148 ceph-mon[44520]: pgmap v1438: 27 pgs: 27 active+clean; 5.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 633 op/s 2024-04-23T16:40:22.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:21 smithi148 ceph-mon[44520]: 6.1 scrub starts 2024-04-23T16:40:22.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:21 smithi148 ceph-mon[44520]: 1.8 scrub starts 2024-04-23T16:40:22.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:21 smithi148 ceph-mon[44520]: 1.8 scrub ok 2024-04-23T16:40:22.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:21 smithi148 ceph-mon[44520]: 6.1 scrub ok 2024-04-23T16:40:22.489 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:40:23.038 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:40:23.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:23.044 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:40:23.083 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:22 smithi103 ceph-mon[39629]: 6.6 scrub ok 2024-04-23T16:40:23.083 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:22 smithi103 ceph-mon[39629]: 6.7 scrub starts 2024-04-23T16:40:23.144 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:40:23.183 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:23.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:22 smithi148 ceph-mon[44520]: 6.6 scrub ok 2024-04-23T16:40:23.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:22 smithi148 ceph-mon[44520]: 6.7 scrub starts 2024-04-23T16:40:23.407 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:40:23.439 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:40:23.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:23.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:40:23.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:22 smithi188 ceph-mon[44101]: 6.6 scrub ok 2024-04-23T16:40:23.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:22 smithi188 ceph-mon[44101]: 6.7 scrub starts 2024-04-23T16:40:23.514 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-23T16:40:23.542 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:40:23.577 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:23.690 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:40:23.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:23.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:40:23.799 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:40:23.848 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:23.880 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:40:23.914 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:40:23.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:23.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:40:23.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:23 smithi103 ceph-mon[39629]: pgmap v1439: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 854 op/s 2024-04-23T16:40:23.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:23 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:40:23.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:23 smithi103 ceph-mon[39629]: 6.8 scrub starts 2024-04-23T16:40:23.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:23 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:40:23.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:23 smithi103 ceph-mon[39629]: 6.f scrub starts 2024-04-23T16:40:23.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:23 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:40:23.993 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:23 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:40:24.018 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:40:24.058 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:24.268 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:23 smithi188 ceph-mon[44101]: pgmap v1439: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 854 op/s 2024-04-23T16:40:24.268 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:23 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:40:24.268 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:23 smithi188 ceph-mon[44101]: 6.8 scrub starts 2024-04-23T16:40:24.268 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:23 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:40:24.268 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:23 smithi188 ceph-mon[44101]: 6.f scrub starts 2024-04-23T16:40:24.269 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:23 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:40:24.269 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:23 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:40:24.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:24 smithi148 ceph-mon[44520]: pgmap v1439: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 854 op/s 2024-04-23T16:40:24.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:24 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:40:24.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:24 smithi148 ceph-mon[44520]: 6.8 scrub starts 2024-04-23T16:40:24.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:24 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:40:24.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:24 smithi148 ceph-mon[44520]: 6.f scrub starts 2024-04-23T16:40:24.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:24 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:40:24.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:24 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:40:24.424 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:40:24.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:24.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:40:24.460 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:40:24.530 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:40:24.558 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:24 smithi103 ceph-mon[39629]: 1.7 scrub starts 2024-04-23T16:40:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:24 smithi103 ceph-mon[39629]: 1.7 scrub ok 2024-04-23T16:40:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:24 smithi103 ceph-mon[39629]: 6.7 scrub ok 2024-04-23T16:40:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:24 smithi103 ceph-mon[39629]: 6.8 scrub ok 2024-04-23T16:40:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:24 smithi103 ceph-mon[39629]: osdmap e516: 12 total, 12 up, 10 in 2024-04-23T16:40:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:24 smithi103 ceph-mon[39629]: 6.f scrub ok 2024-04-23T16:40:25.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:24 smithi148 ceph-mon[44520]: 1.7 scrub starts 2024-04-23T16:40:25.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:24 smithi148 ceph-mon[44520]: 1.7 scrub ok 2024-04-23T16:40:25.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:24 smithi148 ceph-mon[44520]: 6.7 scrub ok 2024-04-23T16:40:25.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:24 smithi148 ceph-mon[44520]: 6.8 scrub ok 2024-04-23T16:40:25.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:24 smithi148 ceph-mon[44520]: osdmap e516: 12 total, 12 up, 10 in 2024-04-23T16:40:25.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:24 smithi148 ceph-mon[44520]: 6.f scrub ok 2024-04-23T16:40:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:24 smithi188 ceph-mon[44101]: 1.7 scrub starts 2024-04-23T16:40:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:24 smithi188 ceph-mon[44101]: 1.7 scrub ok 2024-04-23T16:40:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:24 smithi188 ceph-mon[44101]: 6.7 scrub ok 2024-04-23T16:40:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:24 smithi188 ceph-mon[44101]: 6.8 scrub ok 2024-04-23T16:40:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:24 smithi188 ceph-mon[44101]: osdmap e516: 12 total, 12 up, 10 in 2024-04-23T16:40:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:24 smithi188 ceph-mon[44101]: 6.f scrub ok 2024-04-23T16:40:26.009 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:40:26.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:26.016 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:40:26.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:26 smithi103 ceph-mon[39629]: pgmap v1441: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.00k op/s 2024-04-23T16:40:26.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:26 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:40:26.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:26 smithi103 ceph-mon[39629]: 6.2 scrub starts 2024-04-23T16:40:26.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:26 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:40:26.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:26 smithi103 ceph-mon[39629]: 6.a scrub starts 2024-04-23T16:40:26.116 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:40:26.145 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:26.346 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:40:26.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:26 smithi188 ceph-mon[44101]: pgmap v1441: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.00k op/s 2024-04-23T16:40:26.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:26 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:40:26.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:26 smithi188 ceph-mon[44101]: 6.2 scrub starts 2024-04-23T16:40:26.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:26 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:40:26.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:26 smithi188 ceph-mon[44101]: 6.a scrub starts 2024-04-23T16:40:26.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:26.354 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:40:26.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:26 smithi148 ceph-mon[44520]: pgmap v1441: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.00k op/s 2024-04-23T16:40:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:26 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:40:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:26 smithi148 ceph-mon[44520]: 6.2 scrub starts 2024-04-23T16:40:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:26 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:40:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:26 smithi148 ceph-mon[44520]: 6.a scrub starts 2024-04-23T16:40:26.454 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:40:26.512 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:26.553 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:40:26.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:26.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:40:26.667 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:40:26.672 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:40:26.673 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-04-23T16:40:26.673 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-04-23T16:40:26.673 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:40:26.673 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.673 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76742 5.e 5:735ec33a:::benchmark_data_smithi188_113957_object76741:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:40:26.673 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.360842+0000", 2024-04-23T16:40:26.673 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.308898994, 2024-04-23T16:40:26.673 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8321158799999999, 2024-04-23T16:40:26.673 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.673 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.673 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:26.673 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.673 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.674 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.674 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76742 2024-04-23T16:40:26.674 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.674 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.674 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.674 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.674 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360842+0000", 2024-04-23T16:40:26.674 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.674 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.674 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.674 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.674 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360842+0000", 2024-04-23T16:40:26.675 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.675 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.675 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.675 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.675 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360845+0000", 2024-04-23T16:40:26.675 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4940000000000002e-06 2024-04-23T16:40:26.675 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.675 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.675 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.675 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360880+0000", 2024-04-23T16:40:26.675 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4926999999999999e-05 2024-04-23T16:40:26.675 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.676 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.676 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.676 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360895+0000", 2024-04-23T16:40:26.676 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5508000000000001e-05 2024-04-23T16:40:26.676 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.676 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.676 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.676 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360904+0000", 2024-04-23T16:40:26.676 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1430000000000004e-06 2024-04-23T16:40:26.676 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.676 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.676 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.677 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.361056+0000", 2024-04-23T16:40:26.677 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015130699999999999 2024-04-23T16:40:26.677 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.677 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.677 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:26.677 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.361166+0000", 2024-04-23T16:40:26.677 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110179 2024-04-23T16:40:26.677 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.677 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.677 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:40:26.677 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.361311+0000", 2024-04-23T16:40:26.678 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014554300000000001 2024-04-23T16:40:26.678 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.678 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.678 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:26.678 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.368636+0000", 2024-04-23T16:40:26.678 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073243010000000001 2024-04-23T16:40:26.678 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.678 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.678 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:26.678 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.192829+0000", 2024-04-23T16:40:26.678 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8241933100000001 2024-04-23T16:40:26.678 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.679 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.679 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:26.679 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.192891+0000", 2024-04-23T16:40:26.679 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1778000000000004e-05 2024-04-23T16:40:26.679 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.679 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.679 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.679 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.192958+0000", 2024-04-23T16:40:26.679 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7390000000000004e-05 2024-04-23T16:40:26.679 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.679 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.680 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.680 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.680 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.680 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76743 5.e 5:7e57457a:::benchmark_data_smithi188_113957_object76742:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:40:26.680 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.360918+0000", 2024-04-23T16:40:26.680 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.30882289800002, 2024-04-23T16:40:26.680 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.832188693, 2024-04-23T16:40:26.680 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.680 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.680 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:26.680 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.680 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.681 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.681 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76743 2024-04-23T16:40:26.681 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.681 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.681 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.681 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.681 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360918+0000", 2024-04-23T16:40:26.681 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.681 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.681 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.681 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.681 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360918+0000", 2024-04-23T16:40:26.681 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.682 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.682 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.682 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.682 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360919+0000", 2024-04-23T16:40:26.682 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5000000000000001e-07 2024-04-23T16:40:26.682 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.682 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.682 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.682 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360941+0000", 2024-04-23T16:40:26.682 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1345999999999999e-05 2024-04-23T16:40:26.682 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.682 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.682 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.683 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360947+0000", 2024-04-23T16:40:26.683 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6379999999999998e-06 2024-04-23T16:40:26.683 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.683 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.683 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.683 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.360953+0000", 2024-04-23T16:40:26.683 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9449999999999998e-06 2024-04-23T16:40:26.683 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.683 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.683 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.683 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.363820+0000", 2024-04-23T16:40:26.683 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028672939999999998 2024-04-23T16:40:26.683 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.684 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.684 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:26.684 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.363957+0000", 2024-04-23T16:40:26.684 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013641200000000001 2024-04-23T16:40:26.684 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.684 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.684 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:40:26.684 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.364114+0000", 2024-04-23T16:40:26.684 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015690300000000001 2024-04-23T16:40:26.684 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.684 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.684 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:26.684 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.368640+0000", 2024-04-23T16:40:26.685 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045266120000000002 2024-04-23T16:40:26.685 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.685 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.685 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:26.685 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.193003+0000", 2024-04-23T16:40:26.685 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.82436227 2024-04-23T16:40:26.685 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.685 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.685 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:26.685 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.193053+0000", 2024-04-23T16:40:26.685 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9811000000000001e-05 2024-04-23T16:40:26.685 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.686 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.686 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.686 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.193107+0000", 2024-04-23T16:40:26.686 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4611999999999998e-05 2024-04-23T16:40:26.686 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.686 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.686 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.686 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.686 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.686 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76746 5.e 5:71b55959:::benchmark_data_smithi188_113957_object76745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:40:26.686 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.364139+0000", 2024-04-23T16:40:26.686 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.30560275400001, 2024-04-23T16:40:26.687 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.83183684, 2024-04-23T16:40:26.687 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.687 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.687 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:26.687 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.687 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.687 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.687 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76746 2024-04-23T16:40:26.687 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.687 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.687 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.687 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.687 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.364139+0000", 2024-04-23T16:40:26.688 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.688 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.688 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.688 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.688 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.364139+0000", 2024-04-23T16:40:26.688 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.688 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.688 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.688 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.688 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.364140+0000", 2024-04-23T16:40:26.688 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7489999999999999e-06 2024-04-23T16:40:26.688 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.689 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.689 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.689 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.364164+0000", 2024-04-23T16:40:26.689 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.355e-05 2024-04-23T16:40:26.689 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.689 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.689 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.689 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.364173+0000", 2024-04-23T16:40:26.689 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8170000000000005e-06 2024-04-23T16:40:26.689 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.689 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.689 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.689 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.364180+0000", 2024-04-23T16:40:26.690 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7330000000000003e-06 2024-04-23T16:40:26.690 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.690 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.690 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.690 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.366059+0000", 2024-04-23T16:40:26.690 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018783630000000001 2024-04-23T16:40:26.690 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.690 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.690 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:26.690 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.366204+0000", 2024-04-23T16:40:26.690 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014560899999999999 2024-04-23T16:40:26.690 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.690 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.691 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:40:26.691 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.366354+0000", 2024-04-23T16:40:26.691 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000149982 2024-04-23T16:40:26.691 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.691 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.691 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:26.691 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.372229+0000", 2024-04-23T16:40:26.691 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005874893 2024-04-23T16:40:26.691 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.691 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.691 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:26.691 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.195882+0000", 2024-04-23T16:40:26.692 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.823652869 2024-04-23T16:40:26.692 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.692 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.692 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:26.692 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.195918+0000", 2024-04-23T16:40:26.692 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5555000000000001e-05 2024-04-23T16:40:26.692 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.692 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.692 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.692 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.195975+0000", 2024-04-23T16:40:26.692 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7720000000000003e-05 2024-04-23T16:40:26.692 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.693 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.693 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.693 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.693 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.693 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76759 5.e 5:79bad6f2:::benchmark_data_smithi188_113957_object76758:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:40:26.693 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.369738+0000", 2024-04-23T16:40:26.693 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.300003162, 2024-04-23T16:40:26.693 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.826393253, 2024-04-23T16:40:26.693 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.693 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.693 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:26.693 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.693 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.694 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.694 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76759 2024-04-23T16:40:26.694 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.694 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.694 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.694 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.694 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.369738+0000", 2024-04-23T16:40:26.694 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.694 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.694 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.694 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.694 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.369738+0000", 2024-04-23T16:40:26.694 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.695 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.695 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.695 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.695 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.369740+0000", 2024-04-23T16:40:26.695 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.863e-06 2024-04-23T16:40:26.695 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.695 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.695 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.695 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.369761+0000", 2024-04-23T16:40:26.695 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1452999999999999e-05 2024-04-23T16:40:26.695 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.695 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.695 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.696 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.369772+0000", 2024-04-23T16:40:26.696 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0964999999999999e-05 2024-04-23T16:40:26.696 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.696 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.696 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.696 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.369780+0000", 2024-04-23T16:40:26.696 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3429999999999998e-06 2024-04-23T16:40:26.696 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.696 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.696 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.696 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.369834+0000", 2024-04-23T16:40:26.696 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4416999999999998e-05 2024-04-23T16:40:26.696 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.697 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.697 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:26.697 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.369968+0000", 2024-04-23T16:40:26.697 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013420700000000001 2024-04-23T16:40:26.697 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.697 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.697 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:40:26.697 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.370114+0000", 2024-04-23T16:40:26.697 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014593599999999999 2024-04-23T16:40:26.697 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.697 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.697 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:26.697 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.372383+0000", 2024-04-23T16:40:26.698 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022682470000000001 2024-04-23T16:40:26.698 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.698 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.698 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:26.698 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.196026+0000", 2024-04-23T16:40:26.698 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8236435479999999 2024-04-23T16:40:26.698 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.698 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.698 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:26.699 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.196076+0000", 2024-04-23T16:40:26.699 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9665e-05 2024-04-23T16:40:26.699 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.699 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.699 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.699 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.196131+0000", 2024-04-23T16:40:26.699 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5609000000000002e-05 2024-04-23T16:40:26.699 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.699 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.699 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.699 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.699 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.699 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76778 5.e 5:7fc0c275:::benchmark_data_smithi188_113957_object76777:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:40:26.700 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.380005+0000", 2024-04-23T16:40:26.700 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.28973649300002, 2024-04-23T16:40:26.700 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8170255630000001, 2024-04-23T16:40:26.700 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.700 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.700 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:26.700 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.700 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.700 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.700 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76778 2024-04-23T16:40:26.700 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.700 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.701 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.701 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.701 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.380005+0000", 2024-04-23T16:40:26.701 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.701 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.701 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.701 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.701 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.380005+0000", 2024-04-23T16:40:26.701 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.701 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.701 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.701 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.702 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.380007+0000", 2024-04-23T16:40:26.702 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0839999999999999e-06 2024-04-23T16:40:26.702 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.702 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.702 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.702 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.380031+0000", 2024-04-23T16:40:26.702 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.438e-05 2024-04-23T16:40:26.702 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.702 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.702 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.702 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.380043+0000", 2024-04-23T16:40:26.702 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1313e-05 2024-04-23T16:40:26.702 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.703 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.703 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.703 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.380051+0000", 2024-04-23T16:40:26.703 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2190000000000004e-06 2024-04-23T16:40:26.703 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.703 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.703 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.703 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.380115+0000", 2024-04-23T16:40:26.703 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4021000000000002e-05 2024-04-23T16:40:26.703 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.703 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.703 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:26.704 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.380230+0000", 2024-04-23T16:40:26.704 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011500700000000001 2024-04-23T16:40:26.704 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.704 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.704 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:40:26.704 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.380351+0000", 2024-04-23T16:40:26.704 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000121514 2024-04-23T16:40:26.704 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.704 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.704 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:26.704 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.382807+0000", 2024-04-23T16:40:26.704 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002455942 2024-04-23T16:40:26.704 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.705 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.705 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:26.705 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.196947+0000", 2024-04-23T16:40:26.705 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8141396109999999 2024-04-23T16:40:26.705 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.705 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.705 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:26.705 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.196979+0000", 2024-04-23T16:40:26.705 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1775e-05 2024-04-23T16:40:26.705 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.705 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.705 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.705 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.197030+0000", 2024-04-23T16:40:26.706 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1697000000000003e-05 2024-04-23T16:40:26.706 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.706 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.706 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.706 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.706 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.706 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76851 5.e 5:7bc22d79:::benchmark_data_smithi188_113957_object76850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:40:26.706 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.419329+0000", 2024-04-23T16:40:26.706 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.25041203500001, 2024-04-23T16:40:26.706 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.777843713, 2024-04-23T16:40:26.706 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.706 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.707 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:26.707 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.707 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.707 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.707 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76851 2024-04-23T16:40:26.707 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.707 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.707 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.707 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.707 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.419329+0000", 2024-04-23T16:40:26.707 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.707 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.707 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.708 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.708 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.419329+0000", 2024-04-23T16:40:26.708 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.708 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.708 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.708 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.708 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.419331+0000", 2024-04-23T16:40:26.708 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6649999999999999e-06 2024-04-23T16:40:26.708 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.708 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.708 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.708 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.419352+0000", 2024-04-23T16:40:26.708 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0752999999999999e-05 2024-04-23T16:40:26.709 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.709 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.709 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.709 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.419363+0000", 2024-04-23T16:40:26.709 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0893000000000001e-05 2024-04-23T16:40:26.709 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.709 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.709 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.709 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.419369+0000", 2024-04-23T16:40:26.709 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2879999999999998e-06 2024-04-23T16:40:26.709 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.709 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.710 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.710 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.419416+0000", 2024-04-23T16:40:26.710 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6740999999999999e-05 2024-04-23T16:40:26.710 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.710 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.710 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:26.710 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.419522+0000", 2024-04-23T16:40:26.710 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010632399999999999 2024-04-23T16:40:26.710 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.710 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.710 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:40:26.710 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.419612+0000", 2024-04-23T16:40:26.710 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0290999999999993e-05 2024-04-23T16:40:26.711 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.711 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.711 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:26.711 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.421998+0000", 2024-04-23T16:40:26.711 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002385617 2024-04-23T16:40:26.711 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.711 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.711 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:26.711 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.197076+0000", 2024-04-23T16:40:26.711 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7750786189999999 2024-04-23T16:40:26.711 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.711 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.711 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:26.711 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.197123+0000", 2024-04-23T16:40:26.711 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6780999999999998e-05 2024-04-23T16:40:26.712 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.712 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.712 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.712 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.197173+0000", 2024-04-23T16:40:26.712 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9740999999999997e-05 2024-04-23T16:40:26.712 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.712 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.712 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.712 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.712 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.712 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76874 5.e 5:70e21036:::benchmark_data_smithi188_113957_object76873:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:40:26.712 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.430647+0000", 2024-04-23T16:40:26.712 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.23909425800002, 2024-04-23T16:40:26.712 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7694722189999998, 2024-04-23T16:40:26.712 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.713 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.713 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:26.713 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.713 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.713 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.713 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76874 2024-04-23T16:40:26.713 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.713 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.713 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.713 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.713 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.430647+0000", 2024-04-23T16:40:26.713 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.713 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.713 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.713 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.714 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.430647+0000", 2024-04-23T16:40:26.714 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.714 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.714 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.714 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.714 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.430649+0000", 2024-04-23T16:40:26.714 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5200000000000001e-06 2024-04-23T16:40:26.714 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.714 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.714 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.714 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.430669+0000", 2024-04-23T16:40:26.715 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0089e-05 2024-04-23T16:40:26.715 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.715 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.715 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.715 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.430679+0000", 2024-04-23T16:40:26.715 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0278999999999999e-05 2024-04-23T16:40:26.715 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.715 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.715 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.715 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.430686+0000", 2024-04-23T16:40:26.715 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3189999999999997e-06 2024-04-23T16:40:26.715 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.715 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.715 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.715 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.430745+0000", 2024-04-23T16:40:26.716 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8516e-05 2024-04-23T16:40:26.716 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.716 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.716 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:26.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.430892+0000", 2024-04-23T16:40:26.716 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014720099999999999 2024-04-23T16:40:26.716 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.716 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.716 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:40:26.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.430982+0000", 2024-04-23T16:40:26.716 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9639e-05 2024-04-23T16:40:26.716 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.716 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.716 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:26.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.432989+0000", 2024-04-23T16:40:26.717 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002007146 2024-04-23T16:40:26.717 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.717 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.717 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:26.717 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.200001+0000", 2024-04-23T16:40:26.717 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7670122859999999 2024-04-23T16:40:26.717 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.717 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.717 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:26.717 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.200050+0000", 2024-04-23T16:40:26.717 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9403000000000001e-05 2024-04-23T16:40:26.717 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.718 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.718 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.718 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.200119+0000", 2024-04-23T16:40:26.718 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8820999999999997e-05 2024-04-23T16:40:26.718 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.718 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.718 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.718 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.718 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.718 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76885 5.e 5:728c5bad:::benchmark_data_smithi188_113957_object76884:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:40:26.718 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.437272+0000", 2024-04-23T16:40:26.718 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.23246935999998, 2024-04-23T16:40:26.718 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.763021298, 2024-04-23T16:40:26.718 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.719 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.719 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:26.719 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.719 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.719 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.719 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76885 2024-04-23T16:40:26.719 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.719 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.719 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.719 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.719 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.437272+0000", 2024-04-23T16:40:26.719 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.719 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.719 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.720 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.720 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.437272+0000", 2024-04-23T16:40:26.720 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.720 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.720 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.720 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.720 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.437273+0000", 2024-04-23T16:40:26.720 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1230000000000001e-06 2024-04-23T16:40:26.720 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.720 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.720 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.720 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.437300+0000", 2024-04-23T16:40:26.720 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6961000000000002e-05 2024-04-23T16:40:26.720 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.721 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.721 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.721 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.437311+0000", 2024-04-23T16:40:26.721 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.115e-05 2024-04-23T16:40:26.721 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.721 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.721 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.721 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.437318+0000", 2024-04-23T16:40:26.721 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1040000000000001e-06 2024-04-23T16:40:26.721 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.721 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.721 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.721 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.437368+0000", 2024-04-23T16:40:26.721 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9421e-05 2024-04-23T16:40:26.721 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.722 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.722 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:26.722 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.437492+0000", 2024-04-23T16:40:26.722 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000123786 2024-04-23T16:40:26.722 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.722 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.722 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:40:26.722 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.437602+0000", 2024-04-23T16:40:26.722 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011039099999999999 2024-04-23T16:40:26.722 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.722 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.722 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:26.722 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.439385+0000", 2024-04-23T16:40:26.722 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001783527 2024-04-23T16:40:26.722 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.723 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.723 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:26.723 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.200172+0000", 2024-04-23T16:40:26.723 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7607869630000001 2024-04-23T16:40:26.723 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.723 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.723 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:26.723 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.200225+0000", 2024-04-23T16:40:26.723 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2633999999999998e-05 2024-04-23T16:40:26.723 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.723 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.723 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.723 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.200293+0000", 2024-04-23T16:40:26.723 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8238000000000006e-05 2024-04-23T16:40:26.724 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.724 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.724 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.724 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.724 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.724 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76889 5.e 5:7ef64d28:::benchmark_data_smithi188_113957_object76888:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:40:26.724 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.439666+0000", 2024-04-23T16:40:26.724 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.23007489899999, 2024-04-23T16:40:26.724 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7632851199999999, 2024-04-23T16:40:26.724 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.724 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.724 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:26.724 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.724 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.724 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.725 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76889 2024-04-23T16:40:26.725 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.725 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.725 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.725 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.439666+0000", 2024-04-23T16:40:26.725 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.725 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.725 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.725 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.439666+0000", 2024-04-23T16:40:26.725 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.725 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.725 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.726 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.439668+0000", 2024-04-23T16:40:26.726 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5319999999999999e-06 2024-04-23T16:40:26.726 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.726 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.726 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.726 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.439687+0000", 2024-04-23T16:40:26.726 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9388000000000001e-05 2024-04-23T16:40:26.726 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.726 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.726 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.726 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.439697+0000", 2024-04-23T16:40:26.726 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9310000000000003e-06 2024-04-23T16:40:26.726 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.726 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.726 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.727 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.439705+0000", 2024-04-23T16:40:26.727 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6690000000000006e-06 2024-04-23T16:40:26.727 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.727 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.727 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.727 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.439753+0000", 2024-04-23T16:40:26.727 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8050000000000002e-05 2024-04-23T16:40:26.727 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.727 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.727 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:26.727 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.439880+0000", 2024-04-23T16:40:26.727 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012687700000000001 2024-04-23T16:40:26.727 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.727 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.728 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:40:26.728 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.440021+0000", 2024-04-23T16:40:26.728 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014075200000000001 2024-04-23T16:40:26.728 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.728 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.728 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:26.728 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.442122+0000", 2024-04-23T16:40:26.728 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021011659999999998 2024-04-23T16:40:26.728 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.728 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.728 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:26.728 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.202835+0000", 2024-04-23T16:40:26.728 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7607130579999999 2024-04-23T16:40:26.728 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.728 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.729 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:26.729 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.202889+0000", 2024-04-23T16:40:26.729 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3702000000000002e-05 2024-04-23T16:40:26.729 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.729 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.729 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.729 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.202952+0000", 2024-04-23T16:40:26.729 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2995000000000002e-05 2024-04-23T16:40:26.729 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.729 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.729 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.729 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.730 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.730 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76900 5.e 5:74fb6a1f:::benchmark_data_smithi188_113957_object76899:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-04-23T16:40:26.730 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.444361+0000", 2024-04-23T16:40:26.730 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.225380465, 2024-04-23T16:40:26.730 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.758767958, 2024-04-23T16:40:26.730 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.730 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.730 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:26.730 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.730 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.730 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.730 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76900 2024-04-23T16:40:26.730 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.730 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.731 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.731 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.444361+0000", 2024-04-23T16:40:26.731 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.731 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.731 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.444361+0000", 2024-04-23T16:40:26.731 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.731 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.731 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.444362+0000", 2024-04-23T16:40:26.731 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2099999999999995e-07 2024-04-23T16:40:26.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.444399+0000", 2024-04-23T16:40:26.732 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7076000000000002e-05 2024-04-23T16:40:26.732 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.444407+0000", 2024-04-23T16:40:26.732 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8059999999999995e-06 2024-04-23T16:40:26.732 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.444411+0000", 2024-04-23T16:40:26.732 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6840000000000004e-06 2024-04-23T16:40:26.733 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.733 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.733 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.733 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.444455+0000", 2024-04-23T16:40:26.733 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3414999999999997e-05 2024-04-23T16:40:26.733 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.733 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.733 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:26.733 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.444572+0000", 2024-04-23T16:40:26.733 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011748600000000001 2024-04-23T16:40:26.733 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.733 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.733 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 8", 2024-04-23T16:40:26.733 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.444706+0000", 2024-04-23T16:40:26.733 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013363700000000001 2024-04-23T16:40:26.733 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.734 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.734 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:26.734 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.446856+0000", 2024-04-23T16:40:26.734 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021506070000000001 2024-04-23T16:40:26.734 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.734 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.734 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:26.734 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.203002+0000", 2024-04-23T16:40:26.734 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7561458669999999 2024-04-23T16:40:26.734 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.734 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.734 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:26.734 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.203064+0000", 2024-04-23T16:40:26.734 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2135000000000001e-05 2024-04-23T16:40:26.735 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.735 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.735 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.735 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.203129+0000", 2024-04-23T16:40:26.735 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4424000000000005e-05 2024-04-23T16:40:26.735 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.735 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.735 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.735 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.735 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.735 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76742 5.e 5:735ec33a:::benchmark_data_smithi188_113957_object76741:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:40:26.735 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.463775+0000", 2024-04-23T16:40:26.735 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.20596665900001, 2024-04-23T16:40:26.735 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.729131014, 2024-04-23T16:40:26.735 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.735 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.736 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:40:26.736 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.736 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.736 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.736 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76742 2024-04-23T16:40:26.736 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.736 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.736 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.736 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.736 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463775+0000", 2024-04-23T16:40:26.736 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.736 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.736 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.736 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.737 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463775+0000", 2024-04-23T16:40:26.737 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.737 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.737 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.737 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.737 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463777+0000", 2024-04-23T16:40:26.737 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9750000000000001e-06 2024-04-23T16:40:26.737 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.737 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.737 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.737 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463831+0000", 2024-04-23T16:40:26.737 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4188999999999999e-05 2024-04-23T16:40:26.737 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.738 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.738 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.738 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463842+0000", 2024-04-23T16:40:26.738 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1535000000000001e-05 2024-04-23T16:40:26.738 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.738 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.738 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.738 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463854+0000", 2024-04-23T16:40:26.738 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1851000000000001e-05 2024-04-23T16:40:26.738 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.738 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.738 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.738 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463921+0000", 2024-04-23T16:40:26.738 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6563999999999995e-05 2024-04-23T16:40:26.739 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.739 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.739 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:40:26.739 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463991+0000", 2024-04-23T16:40:26.739 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0281e-05 2024-04-23T16:40:26.739 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.739 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.739 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.739 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.192906+0000", 2024-04-23T16:40:26.739 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.728914619 2024-04-23T16:40:26.739 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.739 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.739 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.739 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.739 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.740 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76743 5.e 5:7e57457a:::benchmark_data_smithi188_113957_object76742:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:40:26.740 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.463873+0000", 2024-04-23T16:40:26.740 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.20586834400001, 2024-04-23T16:40:26.740 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7291898450000001, 2024-04-23T16:40:26.740 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.740 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.740 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:40:26.740 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.740 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.740 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.740 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76743 2024-04-23T16:40:26.740 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.740 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.740 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.740 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.741 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463873+0000", 2024-04-23T16:40:26.741 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.741 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.741 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.741 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.741 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463873+0000", 2024-04-23T16:40:26.741 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.741 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.741 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.741 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.741 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463875+0000", 2024-04-23T16:40:26.741 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.703e-06 2024-04-23T16:40:26.741 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.741 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.741 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.742 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463903+0000", 2024-04-23T16:40:26.742 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8789000000000002e-05 2024-04-23T16:40:26.742 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.742 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.742 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.742 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463915+0000", 2024-04-23T16:40:26.742 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1966e-05 2024-04-23T16:40:26.742 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.742 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.742 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.742 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463923+0000", 2024-04-23T16:40:26.742 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.79e-06 2024-04-23T16:40:26.742 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.742 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.743 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.743 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464029+0000", 2024-04-23T16:40:26.743 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105728 2024-04-23T16:40:26.743 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.743 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.743 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:40:26.743 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464079+0000", 2024-04-23T16:40:26.743 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0535999999999999e-05 2024-04-23T16:40:26.743 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.743 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.743 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.743 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.193063+0000", 2024-04-23T16:40:26.744 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.728983333 2024-04-23T16:40:26.744 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.744 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.744 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.744 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.744 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.744 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76746 5.e 5:71b55959:::benchmark_data_smithi188_113957_object76745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:40:26.744 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.463936+0000", 2024-04-23T16:40:26.744 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.20580561600002, 2024-04-23T16:40:26.744 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7319912259999999, 2024-04-23T16:40:26.744 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.744 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.744 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:40:26.744 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.744 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.745 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.745 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76746 2024-04-23T16:40:26.745 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.745 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.745 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.745 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463936+0000", 2024-04-23T16:40:26.745 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.745 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.745 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.745 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463936+0000", 2024-04-23T16:40:26.745 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.745 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.746 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.746 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.463936+0000", 2024-04-23T16:40:26.746 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3499999999999995e-07 2024-04-23T16:40:26.746 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.746 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.746 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.746 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464009+0000", 2024-04-23T16:40:26.746 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2646999999999997e-05 2024-04-23T16:40:26.746 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.746 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.746 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.746 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464017+0000", 2024-04-23T16:40:26.746 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6070000000000002e-06 2024-04-23T16:40:26.746 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.747 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.747 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.747 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464032+0000", 2024-04-23T16:40:26.747 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5053e-05 2024-04-23T16:40:26.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.747 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.747 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.747 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464125+0000", 2024-04-23T16:40:26.747 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3388999999999996e-05 2024-04-23T16:40:26.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.747 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.747 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:40:26.747 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464191+0000", 2024-04-23T16:40:26.747 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6356e-05 2024-04-23T16:40:26.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.748 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.748 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.748 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.195927+0000", 2024-04-23T16:40:26.748 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7317354389999999 2024-04-23T16:40:26.748 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.748 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.748 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.748 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.748 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.748 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76759 5.e 5:79bad6f2:::benchmark_data_smithi188_113957_object76758:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:40:26.748 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.464041+0000", 2024-04-23T16:40:26.748 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.205700218, 2024-04-23T16:40:26.748 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.732046242, 2024-04-23T16:40:26.748 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.749 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.749 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:40:26.749 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.749 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.749 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.749 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76759 2024-04-23T16:40:26.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.749 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.749 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.749 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.749 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464041+0000", 2024-04-23T16:40:26.749 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.750 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.750 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464041+0000", 2024-04-23T16:40:26.750 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.750 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.750 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464042+0000", 2024-04-23T16:40:26.750 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7100000000000002e-07 2024-04-23T16:40:26.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.750 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.750 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464074+0000", 2024-04-23T16:40:26.750 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2184000000000002e-05 2024-04-23T16:40:26.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.751 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.751 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.751 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464080+0000", 2024-04-23T16:40:26.751 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8000000000000004e-06 2024-04-23T16:40:26.751 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.751 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.751 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.751 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464083+0000", 2024-04-23T16:40:26.751 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7469999999999999e-06 2024-04-23T16:40:26.751 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.751 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.751 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.751 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464238+0000", 2024-04-23T16:40:26.751 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015478099999999999 2024-04-23T16:40:26.751 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.751 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.752 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:40:26.752 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464289+0000", 2024-04-23T16:40:26.752 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0766999999999997e-05 2024-04-23T16:40:26.752 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.752 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.752 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.752 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.196087+0000", 2024-04-23T16:40:26.752 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.731798392 2024-04-23T16:40:26.752 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.752 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.752 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.752 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.752 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.752 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76778 5.e 5:7fc0c275:::benchmark_data_smithi188_113957_object76777:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:40:26.752 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.464091+0000", 2024-04-23T16:40:26.752 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.20565037799997, 2024-04-23T16:40:26.753 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7328962859999999, 2024-04-23T16:40:26.753 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.753 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.753 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:40:26.753 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.753 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.753 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.753 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76778 2024-04-23T16:40:26.753 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.753 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.753 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.753 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.753 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464091+0000", 2024-04-23T16:40:26.753 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.753 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.754 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.754 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.754 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464091+0000", 2024-04-23T16:40:26.754 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.754 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.754 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.754 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.754 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464091+0000", 2024-04-23T16:40:26.754 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8299999999999998e-07 2024-04-23T16:40:26.754 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.754 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.754 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.754 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464142+0000", 2024-04-23T16:40:26.754 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0362000000000003e-05 2024-04-23T16:40:26.754 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.754 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.755 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.755 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464148+0000", 2024-04-23T16:40:26.755 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8050000000000001e-06 2024-04-23T16:40:26.755 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.755 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.755 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.755 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464155+0000", 2024-04-23T16:40:26.755 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.241e-06 2024-04-23T16:40:26.755 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.755 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.755 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.755 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464329+0000", 2024-04-23T16:40:26.755 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017406400000000001 2024-04-23T16:40:26.755 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.755 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.755 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:40:26.756 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464391+0000", 2024-04-23T16:40:26.756 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1785999999999994e-05 2024-04-23T16:40:26.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.756 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.756 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.196987+0000", 2024-04-23T16:40:26.756 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7325966450000001 2024-04-23T16:40:26.756 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.756 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.756 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.756 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.756 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76851 5.e 5:7bc22d79:::benchmark_data_smithi188_113957_object76850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:40:26.756 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.464163+0000", 2024-04-23T16:40:26.756 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.20557852600001, 2024-04-23T16:40:26.757 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.732972556, 2024-04-23T16:40:26.757 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.757 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.757 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:40:26.757 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.757 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.757 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.757 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76851 2024-04-23T16:40:26.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.757 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.757 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.757 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.757 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464163+0000", 2024-04-23T16:40:26.757 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.757 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.758 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464163+0000", 2024-04-23T16:40:26.758 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.758 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.758 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464164+0000", 2024-04-23T16:40:26.758 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5799999999999998e-07 2024-04-23T16:40:26.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.758 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.758 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464203+0000", 2024-04-23T16:40:26.758 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9759999999999999e-05 2024-04-23T16:40:26.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.759 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.759 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.759 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464210+0000", 2024-04-23T16:40:26.759 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5610000000000004e-06 2024-04-23T16:40:26.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.759 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.759 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.759 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464216+0000", 2024-04-23T16:40:26.759 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9830000000000001e-06 2024-04-23T16:40:26.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.759 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.759 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.759 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464427+0000", 2024-04-23T16:40:26.759 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021098199999999999 2024-04-23T16:40:26.760 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.760 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.760 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:40:26.760 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464479+0000", 2024-04-23T16:40:26.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2076000000000001e-05 2024-04-23T16:40:26.760 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.760 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.760 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.760 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.197135+0000", 2024-04-23T16:40:26.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7326564360000001 2024-04-23T16:40:26.760 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.760 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.760 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.761 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.761 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.761 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76874 5.e 5:70e21036:::benchmark_data_smithi188_113957_object76873:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:40:26.761 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.464233+0000", 2024-04-23T16:40:26.761 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.20550841599999, 2024-04-23T16:40:26.761 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7358331840000001, 2024-04-23T16:40:26.761 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.761 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.761 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:40:26.761 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.761 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.761 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.761 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76874 2024-04-23T16:40:26.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.762 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.762 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.762 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.762 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464233+0000", 2024-04-23T16:40:26.762 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.762 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.762 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.762 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464233+0000", 2024-04-23T16:40:26.762 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.763 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.763 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.763 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.763 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464234+0000", 2024-04-23T16:40:26.763 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.85e-07 2024-04-23T16:40:26.763 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.763 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.763 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.763 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464272+0000", 2024-04-23T16:40:26.763 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8439999999999998e-05 2024-04-23T16:40:26.763 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.763 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.763 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.763 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464279+0000", 2024-04-23T16:40:26.763 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5599999999999999e-06 2024-04-23T16:40:26.763 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.764 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.764 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.764 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464284+0000", 2024-04-23T16:40:26.764 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.575e-06 2024-04-23T16:40:26.764 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.764 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.764 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.764 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464515+0000", 2024-04-23T16:40:26.764 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000231091 2024-04-23T16:40:26.764 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.764 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.764 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:40:26.764 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464574+0000", 2024-04-23T16:40:26.764 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9131e-05 2024-04-23T16:40:26.764 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.764 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.765 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.765 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.200066+0000", 2024-04-23T16:40:26.765 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.735491602 2024-04-23T16:40:26.765 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.765 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.765 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.765 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.765 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76885 5.e 5:728c5bad:::benchmark_data_smithi188_113957_object76884:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:40:26.765 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.464292+0000", 2024-04-23T16:40:26.765 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.20544938699999, 2024-04-23T16:40:26.765 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.735947431, 2024-04-23T16:40:26.765 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.765 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.765 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:40:26.765 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.766 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.766 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.766 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76885 2024-04-23T16:40:26.766 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.766 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.766 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.766 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.766 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464292+0000", 2024-04-23T16:40:26.766 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.766 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.766 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.766 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.766 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464292+0000", 2024-04-23T16:40:26.766 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.767 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.767 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.767 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.767 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464293+0000", 2024-04-23T16:40:26.767 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2699999999999999e-07 2024-04-23T16:40:26.767 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.767 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.767 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.767 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464411+0000", 2024-04-23T16:40:26.767 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011827 2024-04-23T16:40:26.767 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.767 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.767 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.767 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464417+0000", 2024-04-23T16:40:26.767 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3210000000000001e-06 2024-04-23T16:40:26.767 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.767 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.768 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.768 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464423+0000", 2024-04-23T16:40:26.768 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5740000000000003e-06 2024-04-23T16:40:26.768 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.768 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.768 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.768 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464611+0000", 2024-04-23T16:40:26.768 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000188553 2024-04-23T16:40:26.768 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.768 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.768 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:40:26.768 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464664+0000", 2024-04-23T16:40:26.768 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2893999999999999e-05 2024-04-23T16:40:26.768 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.768 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.768 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.769 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.200239+0000", 2024-04-23T16:40:26.769 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7355750919999999 2024-04-23T16:40:26.769 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.769 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.769 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.769 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.769 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.769 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76889 5.e 5:7ef64d28:::benchmark_data_smithi188_113957_object76888:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:40:26.769 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.464430+0000", 2024-04-23T16:40:26.769 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.20531097499997, 2024-04-23T16:40:26.769 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7384693900000001, 2024-04-23T16:40:26.769 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.769 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.769 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:40:26.769 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.769 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.770 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.770 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76889 2024-04-23T16:40:26.770 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.770 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.770 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.770 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.770 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464430+0000", 2024-04-23T16:40:26.770 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.770 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.770 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.770 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.770 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464430+0000", 2024-04-23T16:40:26.770 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.771 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.771 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.771 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.771 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464431+0000", 2024-04-23T16:40:26.771 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5199999999999996e-07 2024-04-23T16:40:26.771 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.771 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.771 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.771 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464470+0000", 2024-04-23T16:40:26.771 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9023000000000002e-05 2024-04-23T16:40:26.771 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.771 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.771 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.771 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464476+0000", 2024-04-23T16:40:26.771 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8250000000000001e-06 2024-04-23T16:40:26.771 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.772 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.772 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.772 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464482+0000", 2024-04-23T16:40:26.772 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7660000000000004e-06 2024-04-23T16:40:26.772 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.772 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.772 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.772 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464701+0000", 2024-04-23T16:40:26.772 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00021927399999999999 2024-04-23T16:40:26.772 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.772 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.772 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:40:26.772 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464761+0000", 2024-04-23T16:40:26.772 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9636999999999998e-05 2024-04-23T16:40:26.773 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.773 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.773 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.773 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.202900+0000", 2024-04-23T16:40:26.773 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.738139213 2024-04-23T16:40:26.773 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.773 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.773 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.773 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.773 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.773 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:76900 5.e 5:74fb6a1f:::benchmark_data_smithi188_113957_object76899:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e372)", 2024-04-23T16:40:26.773 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:32:03.464509+0000", 2024-04-23T16:40:26.773 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 503.20523216300001, 2024-04-23T16:40:26.773 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7385653969999999, 2024-04-23T16:40:26.773 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:26.773 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:26.774 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for ondisk", 2024-04-23T16:40:26.774 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:26.774 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:26.774 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:26.774 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 76900 2024-04-23T16:40:26.774 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.774 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:26.774 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.774 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:26.774 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464509+0000", 2024-04-23T16:40:26.774 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.774 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.774 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.774 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:26.774 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464509+0000", 2024-04-23T16:40:26.774 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:26.775 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.775 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.775 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:26.775 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464510+0000", 2024-04-23T16:40:26.775 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7999999999999995e-07 2024-04-23T16:40:26.775 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.775 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.775 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:26.775 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464548+0000", 2024-04-23T16:40:26.775 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8022999999999998e-05 2024-04-23T16:40:26.775 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.775 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.775 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:26.775 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464554+0000", 2024-04-23T16:40:26.775 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9270000000000002e-06 2024-04-23T16:40:26.775 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.775 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.776 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:26.776 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464559+0000", 2024-04-23T16:40:26.776 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0819999999999998e-06 2024-04-23T16:40:26.776 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.776 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.776 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:26.776 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464786+0000", 2024-04-23T16:40:26.776 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00022706500000000001 2024-04-23T16:40:26.776 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.776 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.776 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for ondisk", 2024-04-23T16:40:26.776 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:03.464839+0000", 2024-04-23T16:40:26.776 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3056999999999997e-05 2024-04-23T16:40:26.776 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:26.776 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:26.776 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:26.777 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:32:05.203075+0000", 2024-04-23T16:40:26.777 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7382355629999999 2024-04-23T16:40:26.777 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.777 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.777 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.777 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:26.777 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:26.777 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:40:26.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:26.778 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:26.778 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:40:26.779 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:40:26.879 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:40:27.264 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:27.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:27 smithi103 ceph-mon[39629]: 1.5 scrub starts 2024-04-23T16:40:27.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:27 smithi103 ceph-mon[39629]: 1.5 scrub ok 2024-04-23T16:40:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:27 smithi103 ceph-mon[39629]: 1.4 scrub starts 2024-04-23T16:40:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:27 smithi103 ceph-mon[39629]: 6.2 scrub ok 2024-04-23T16:40:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:27 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:40:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:27 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:40:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:27 smithi103 ceph-mon[39629]: 1.4 scrub ok 2024-04-23T16:40:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:27 smithi103 ceph-mon[39629]: 6.a scrub ok 2024-04-23T16:40:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:27 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3087974799' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:40:27.374 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.7 to 1 (10000) 2024-04-23T16:40:27.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:27 smithi148 ceph-mon[44520]: 1.5 scrub starts 2024-04-23T16:40:27.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:27 smithi148 ceph-mon[44520]: 1.5 scrub ok 2024-04-23T16:40:27.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:27 smithi148 ceph-mon[44520]: 1.4 scrub starts 2024-04-23T16:40:27.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:27 smithi148 ceph-mon[44520]: 6.2 scrub ok 2024-04-23T16:40:27.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:27 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:40:27.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:27 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:40:27.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:27 smithi148 ceph-mon[44520]: 1.4 scrub ok 2024-04-23T16:40:27.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:27 smithi148 ceph-mon[44520]: 6.a scrub ok 2024-04-23T16:40:27.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:27 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3087974799' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:40:27.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:27 smithi188 ceph-mon[44101]: 1.5 scrub starts 2024-04-23T16:40:27.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:27 smithi188 ceph-mon[44101]: 1.5 scrub ok 2024-04-23T16:40:27.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:27 smithi188 ceph-mon[44101]: 1.4 scrub starts 2024-04-23T16:40:27.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:27 smithi188 ceph-mon[44101]: 6.2 scrub ok 2024-04-23T16:40:27.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:27 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:40:27.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:27 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:40:27.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:27 smithi188 ceph-mon[44101]: 1.4 scrub ok 2024-04-23T16:40:27.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:27 smithi188 ceph-mon[44101]: 6.a scrub ok 2024-04-23T16:40:27.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:27 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3087974799' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:40:27.464 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-23T16:40:27.764 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:40:28.221 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:40:28.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:28.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:40:28.289 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:28 smithi188 ceph-mon[44101]: pgmap v1442: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.7 GiB data, 15 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:40:28.289 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:28 smithi188 ceph-mon[44101]: 6.e scrub starts 2024-04-23T16:40:28.289 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:28 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3087974799' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:40:28.289 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:28 smithi188 ceph-mon[44101]: osdmap e517: 12 total, 12 up, 10 in 2024-04-23T16:40:28.289 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:28 smithi188 ceph-mon[44101]: 6.0 scrub starts 2024-04-23T16:40:28.332 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:40:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:28 smithi103 ceph-mon[39629]: pgmap v1442: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.7 GiB data, 15 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:40:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:28 smithi103 ceph-mon[39629]: 6.e scrub starts 2024-04-23T16:40:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:28 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3087974799' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:40:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:28 smithi103 ceph-mon[39629]: osdmap e517: 12 total, 12 up, 10 in 2024-04-23T16:40:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:28 smithi103 ceph-mon[39629]: 6.0 scrub starts 2024-04-23T16:40:28.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:28 smithi148 ceph-mon[44520]: pgmap v1442: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.7 GiB data, 15 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:40:28.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:28 smithi148 ceph-mon[44520]: 6.e scrub starts 2024-04-23T16:40:28.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:28 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3087974799' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:40:28.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:28 smithi148 ceph-mon[44520]: osdmap e517: 12 total, 12 up, 10 in 2024-04-23T16:40:28.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:28 smithi148 ceph-mon[44520]: 6.0 scrub starts 2024-04-23T16:40:28.383 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:28.425 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:40:28.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:28.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:40:28.532 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:40:29.026 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:29 smithi103 ceph-mon[39629]: 6.e scrub ok 2024-04-23T16:40:29.133 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:29 smithi188 ceph-mon[44101]: 6.e scrub ok 2024-04-23T16:40:29.133 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:29 smithi188 ceph-mon[44101]: 6.0 scrub ok 2024-04-23T16:40:29.133 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:29 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2101785766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:40:29.133 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:29 smithi188 ceph-mon[44101]: 6.d scrub starts 2024-04-23T16:40:29.134 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.134 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.134 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.134 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.134 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.134 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.134 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.134 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:40:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:40:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:40:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:40:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:40:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:40:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:40:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:40:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:40:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:40:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:40:29.205 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:40:29.205 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:40:29.205 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:40:29.205 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:40:29.205 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:40:29.205 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:40:29.205 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:40:29.205 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:40:29.205 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:40:29.205 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:40:29.205 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:40:29.205 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:40:29.205 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:40:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:29 smithi103 ceph-mon[39629]: 6.0 scrub ok 2024-04-23T16:40:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:29 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2101785766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:40:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:29 smithi103 ceph-mon[39629]: 6.d scrub starts 2024-04-23T16:40:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:29 smithi148 ceph-mon[44520]: 6.e scrub ok 2024-04-23T16:40:29.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:29 smithi148 ceph-mon[44520]: 6.0 scrub ok 2024-04-23T16:40:29.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:29 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2101785766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:40:29.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:29 smithi148 ceph-mon[44520]: 6.d scrub starts 2024-04-23T16:40:29.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:29.484 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:29.741 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:40:29.741 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:40:30.037 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:30.037 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:30.037 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:30.038 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:30.038 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:30 smithi188 ceph-mon[44101]: pgmap v1444: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-23T16:40:30.038 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:30 smithi188 ceph-mon[44101]: 6.4 scrub starts 2024-04-23T16:40:30.038 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:30 smithi188 ceph-mon[44101]: 6.5 scrub starts 2024-04-23T16:40:30.038 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:30 smithi188 ceph-mon[44101]: 6.d scrub ok 2024-04-23T16:40:30.070 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:40:30.084 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:40:30.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:30.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:40:30.247 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:40:30.291 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:30.297 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:30.297 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:30.297 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:30.297 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:30.297 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:30 smithi148 ceph-mon[44520]: pgmap v1444: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-23T16:40:30.298 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:30 smithi148 ceph-mon[44520]: 6.4 scrub starts 2024-04-23T16:40:30.298 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:30 smithi148 ceph-mon[44520]: 6.5 scrub starts 2024-04-23T16:40:30.298 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:30 smithi148 ceph-mon[44520]: 6.d scrub ok 2024-04-23T16:40:30.334 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:40:30.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:30.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:40:30.361 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:30.361 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:30.361 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:30.361 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:30.361 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:30 smithi103 ceph-mon[39629]: pgmap v1444: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-23T16:40:30.361 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:30 smithi103 ceph-mon[39629]: 6.4 scrub starts 2024-04-23T16:40:30.361 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:30 smithi103 ceph-mon[39629]: 6.5 scrub starts 2024-04-23T16:40:30.361 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:30 smithi103 ceph-mon[39629]: 6.d scrub ok 2024-04-23T16:40:30.441 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:40:30.474 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:30.498 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:40:30.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:30.503 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:40:30.531 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-23T16:40:30.604 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:40:30.629 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:30.648 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:40:30.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:30.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:40:30.753 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:40:30.812 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:30.849 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:40:30.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:30.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:40:30.851 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:40:30.951 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:40:30.978 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:31.006 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:40:31.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:31.009 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:40:31.110 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:40:31.212 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:31 smithi103 ceph-mon[39629]: 6.b scrub starts 2024-04-23T16:40:31.212 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:31 smithi103 ceph-mon[39629]: 6.4 scrub ok 2024-04-23T16:40:31.212 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:31 smithi103 ceph-mon[39629]: 6.5 scrub ok 2024-04-23T16:40:31.212 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:31 smithi103 ceph-mon[39629]: pgmap v1445: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 4.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 976 op/s 2024-04-23T16:40:31.214 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:31.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:31 smithi148 ceph-mon[44520]: 6.b scrub starts 2024-04-23T16:40:31.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:31 smithi148 ceph-mon[44520]: 6.4 scrub ok 2024-04-23T16:40:31.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:31 smithi148 ceph-mon[44520]: 6.5 scrub ok 2024-04-23T16:40:31.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:31 smithi148 ceph-mon[44520]: pgmap v1445: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 4.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 976 op/s 2024-04-23T16:40:31.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:31 smithi188 ceph-mon[44101]: 6.b scrub starts 2024-04-23T16:40:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:31 smithi188 ceph-mon[44101]: 6.4 scrub ok 2024-04-23T16:40:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:31 smithi188 ceph-mon[44101]: 6.5 scrub ok 2024-04-23T16:40:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:31 smithi188 ceph-mon[44101]: pgmap v1445: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 4.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 976 op/s 2024-04-23T16:40:32.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:32 smithi188 ceph-mon[44101]: 6.b scrub ok 2024-04-23T16:40:32.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:32 smithi188 ceph-mon[44101]: 6.3 scrub starts 2024-04-23T16:40:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:32 smithi188 ceph-mon[44101]: 6.9 scrub starts 2024-04-23T16:40:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:32 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T16:40:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:32 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T16:40:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:32 smithi103 ceph-mon[39629]: 6.b scrub ok 2024-04-23T16:40:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:32 smithi103 ceph-mon[39629]: 6.3 scrub starts 2024-04-23T16:40:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:32 smithi103 ceph-mon[39629]: 6.9 scrub starts 2024-04-23T16:40:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:32 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T16:40:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:32 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T16:40:32.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:32 smithi148 ceph-mon[44520]: 6.b scrub ok 2024-04-23T16:40:32.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:32 smithi148 ceph-mon[44520]: 6.3 scrub starts 2024-04-23T16:40:32.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:32 smithi148 ceph-mon[44520]: 6.9 scrub starts 2024-04-23T16:40:32.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:32 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T16:40:32.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:32 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T16:40:32.815 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:40:32.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:32.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:40:32.857 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:40:32.857 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:40:32.918 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:40:33.166 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:33.207 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:33 smithi188 ceph-mon[44101]: 6.9 scrub ok 2024-04-23T16:40:33.207 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:33 smithi188 ceph-mon[44101]: 6.3 scrub ok 2024-04-23T16:40:33.207 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:33 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:40:33.207 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:33 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:40:33.207 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:33 smithi188 ceph-mon[44101]: pgmap v1446: 27 pgs: 27 active+clean; 4.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-23T16:40:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:33 smithi103 ceph-mon[39629]: 6.9 scrub ok 2024-04-23T16:40:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:33 smithi103 ceph-mon[39629]: 6.3 scrub ok 2024-04-23T16:40:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:33 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:40:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:33 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:40:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:33 smithi103 ceph-mon[39629]: pgmap v1446: 27 pgs: 27 active+clean; 4.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-23T16:40:33.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:33 smithi148 ceph-mon[44520]: 6.9 scrub ok 2024-04-23T16:40:33.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:33 smithi148 ceph-mon[44520]: 6.3 scrub ok 2024-04-23T16:40:33.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:33 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:40:33.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:33 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:40:33.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:33 smithi148 ceph-mon[44520]: pgmap v1446: 27 pgs: 27 active+clean; 4.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-23T16:40:33.581 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-23T16:40:33.616 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:40:33.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:33.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:40:33.654 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 8 1 2024-04-23T16:40:33.724 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:40:33.781 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:33.817 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:40:33.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:33.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:40:33.887 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:40:33.931 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:40:33.975 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:34.245 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:40:34.253 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:34 smithi103 ceph-mon[39629]: 1.9 scrub starts 2024-04-23T16:40:34.253 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:34 smithi103 ceph-mon[39629]: 1.9 scrub ok 2024-04-23T16:40:34.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:34.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:40:34.356 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:40:34.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:34 smithi148 ceph-mon[44520]: 1.9 scrub starts 2024-04-23T16:40:34.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:34 smithi148 ceph-mon[44520]: 1.9 scrub ok 2024-04-23T16:40:34.404 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:34.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:34 smithi188 ceph-mon[44101]: 1.9 scrub starts 2024-04-23T16:40:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:34 smithi188 ceph-mon[44101]: 1.9 scrub ok 2024-04-23T16:40:34.617 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:40:34.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:34.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:40:34.726 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:40:34.773 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:34.806 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:40:34.815 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:40:34.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:34.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:40:34.921 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:40:34.963 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:35.008 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:40:35.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:35.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:40:35.037 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:40:35.112 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:40:35.153 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:35 smithi188 ceph-mon[44101]: 6.c scrub starts 2024-04-23T16:40:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:35 smithi188 ceph-mon[44101]: pgmap v1447: 27 pgs: 27 active+clean; 4.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-23T16:40:35.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:35 smithi103 ceph-mon[39629]: 6.c scrub starts 2024-04-23T16:40:35.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:35 smithi103 ceph-mon[39629]: pgmap v1447: 27 pgs: 27 active+clean; 4.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-23T16:40:35.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:35 smithi148 ceph-mon[44520]: 6.c scrub starts 2024-04-23T16:40:35.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:35 smithi148 ceph-mon[44520]: pgmap v1447: 27 pgs: 27 active+clean; 4.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-23T16:40:36.005 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:40:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:36.012 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:40:36.113 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:40:36.286 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:40:36.521 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:40:36.522 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:40:36.544 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:36 smithi188 ceph-mon[44101]: 6.c scrub ok 2024-04-23T16:40:36.544 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:36 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:40:36.544 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:36 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:40:36.544 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1733935361' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:40:36.544 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:36 smithi188 ceph-mon[44101]: from='client.36119 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:40:36.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:36 smithi103 ceph-mon[39629]: 6.c scrub ok 2024-04-23T16:40:36.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:36 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:40:36.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:36 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:40:36.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1733935361' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:40:36.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:36 smithi103 ceph-mon[39629]: from='client.36119 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:40:36.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:36 smithi148 ceph-mon[44520]: 6.c scrub ok 2024-04-23T16:40:36.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:36 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:40:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:36 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:40:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1733935361' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:40:36.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:36 smithi148 ceph-mon[44520]: from='client.36119 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:40:36.643 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:36.797 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:40:36.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:36.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:40:36.902 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:40:36.934 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:37.146 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:40:37.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:37.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:40:37.251 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:40:37.294 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:37.330 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-23T16:40:37.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:37.346 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:40:37.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:40:37.447 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:40:37.510 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:37.542 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:40:37.550 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:37 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:40:37.551 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:37 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:40:37.551 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:37 smithi148 ceph-mon[44520]: from='client.36119 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:40:37.551 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:37 smithi148 ceph-mon[44520]: osdmap e518: 12 total, 12 up, 10 in 2024-04-23T16:40:37.551 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:37 smithi148 ceph-mon[44520]: pgmap v1449: 27 pgs: 27 active+clean; 4.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-23T16:40:37.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:37.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:40:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:37 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:40:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:37 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:40:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:37 smithi103 ceph-mon[39629]: from='client.36119 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:40:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:37 smithi103 ceph-mon[39629]: osdmap e518: 12 total, 12 up, 10 in 2024-04-23T16:40:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:37 smithi103 ceph-mon[39629]: pgmap v1449: 27 pgs: 27 active+clean; 4.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-23T16:40:37.627 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:40:37.653 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:40:37.721 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:37.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:37 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:40:37.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:37 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:40:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:37 smithi188 ceph-mon[44101]: from='client.36119 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:40:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:37 smithi188 ceph-mon[44101]: osdmap e518: 12 total, 12 up, 10 in 2024-04-23T16:40:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:37 smithi188 ceph-mon[44101]: pgmap v1449: 27 pgs: 27 active+clean; 4.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-23T16:40:37.752 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:40:37.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:37.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:40:37.863 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:40:37.900 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:38.359 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:40:38.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:38.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:40:38.433 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:40:38.464 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:40:38.514 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:38.731 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:40:38.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:38.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:40:38.835 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:40:38.865 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:38.892 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:40:38.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:38.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:40:38.996 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:40:39.037 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:39.072 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:40:39.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:39.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:40:39.176 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:40:39.211 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:39.248 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:40:39.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:39.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:40:39.352 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:40:39.380 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:39.579 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:40:39.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:39.583 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:40:39.684 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:40:39.728 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:39.752 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:40:39.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:39.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:40:39.793 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:40:39.872 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:40:39.884 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.8 to 1 (10000) 2024-04-23T16:40:39.898 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:39 smithi188 ceph-mon[44101]: pgmap v1450: 27 pgs: 27 active+clean; 4.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-23T16:40:39.898 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:39 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:40:39.898 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:39 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:40:39.898 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:39 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3930322713' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:40:39.901 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:39.906 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:40:39.906 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:40:39.923 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:40:39.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:39.929 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:40:40.030 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:40:40.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:39 smithi103 ceph-mon[39629]: pgmap v1450: 27 pgs: 27 active+clean; 4.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-23T16:40:40.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:39 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:40:40.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:39 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:40:40.103 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:39 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3930322713' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:40:40.105 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:39 smithi148 ceph-mon[44520]: pgmap v1450: 27 pgs: 27 active+clean; 4.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-23T16:40:40.105 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:39 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:40:40.105 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:39 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:40:40.105 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:39 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3930322713' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:40:40.106 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:40.136 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:40:40.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:40.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:40:40.242 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:40:40.270 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:40.292 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:40:40.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:40.305 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:40:40.406 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:40:40.460 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:40.504 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:40:40.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:40.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:40:40.611 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:40:40.641 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:40.651 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-23T16:40:40.662 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:40:40.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:40.667 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:40:40.768 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:40:40.799 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:40.819 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:40:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:40.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:40:40.925 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:40:40.961 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:40.985 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:40:40.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:40.990 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:40:41.002 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:40:41.092 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:40:41.120 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:41.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:40 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3930322713' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:40:41.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:40 smithi188 ceph-mon[44101]: osdmap e519: 12 total, 12 up, 10 in 2024-04-23T16:40:41.246 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:40 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3930322713' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:40:41.246 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:40 smithi103 ceph-mon[39629]: osdmap e519: 12 total, 12 up, 10 in 2024-04-23T16:40:41.269 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:40:41.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:41.274 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:40:41.374 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:40:41.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:40 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3930322713' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:40:41.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:40 smithi148 ceph-mon[44520]: osdmap e519: 12 total, 12 up, 10 in 2024-04-23T16:40:41.409 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:42.063 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 9 1 2024-04-23T16:40:42.065 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:40:42.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:42.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:40:42.169 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:40:42.197 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:41 smithi188 ceph-mon[44101]: pgmap v1452: 27 pgs: 27 active+clean; 3.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-23T16:40:42.200 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:42.217 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:41 smithi103 ceph-mon[39629]: pgmap v1452: 27 pgs: 27 active+clean; 3.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-23T16:40:42.352 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:40:42.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:42.359 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:40:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:41 smithi148 ceph-mon[44520]: pgmap v1452: 27 pgs: 27 active+clean; 3.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-23T16:40:42.460 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:40:42.492 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:42.746 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:40:42.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:42.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:40:42.766 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:40:42.850 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:40:42.926 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:42.955 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:40:42.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:42.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:40:42.988 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:40:42.988 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:40:43.062 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:40:43.093 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:43.245 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:40:43.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:43.248 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:40:43.349 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:40:43.380 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:43.406 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:40:43.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:43.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:40:43.515 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:40:43.547 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:43.653 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:40:43.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:43.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:40:43.759 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:40:43.766 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-23T16:40:43.816 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:43.841 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:40:43.876 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:40:43.896 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:40:43.926 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:40:43.964 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:40:43.997 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:40:44.035 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:40:44.066 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:40:44.068 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:40:44.094 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:40:44.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:43 smithi103 ceph-mon[39629]: pgmap v1453: 27 pgs: 27 active+clean; 3.7 GiB data, 13 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:40:44.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:43 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:40:44.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:43 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:40:44.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:43 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1679312925' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:40:44.133 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:40:44.151 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:40:44.174 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:40:44.186 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:40:44.186 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:40:44.186 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:40:44.186 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:40:44.186 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:40:44.186 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:40:44.186 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:40:44.186 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:40:44.186 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:40:44.187 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:40:44.187 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:40:44.187 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:40:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:43 smithi188 ceph-mon[44101]: pgmap v1453: 27 pgs: 27 active+clean; 3.7 GiB data, 13 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:40:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:43 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:40:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:43 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:40:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:43 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1679312925' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:40:44.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:43 smithi148 ceph-mon[44520]: pgmap v1453: 27 pgs: 27 active+clean; 3.7 GiB data, 13 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-23T16:40:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:43 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:40:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:43 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:40:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:43 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1679312925' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:40:44.975 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:40:44.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:44.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:40:45.083 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:40:45.121 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:45.153 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:40:45.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:45.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:40:45.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:44 smithi188 ceph-mon[44101]: osdmap e520: 12 total, 12 up, 10 in 2024-04-23T16:40:45.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.259 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:40:45.288 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:44 smithi103 ceph-mon[39629]: osdmap e520: 12 total, 12 up, 10 in 2024-04-23T16:40:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:44 smithi148 ceph-mon[44520]: osdmap e520: 12 total, 12 up, 10 in 2024-04-23T16:40:45.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:45.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:46.019 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:40:46.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:46.023 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:40:46.124 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:40:46.152 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:46.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:45 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:46.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:45 smithi188 ceph-mon[44101]: pgmap v1455: 27 pgs: 27 active+clean; 3.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 912 op/s 2024-04-23T16:40:46.228 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:45 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:46.228 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:45 smithi103 ceph-mon[39629]: pgmap v1455: 27 pgs: 27 active+clean; 3.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 912 op/s 2024-04-23T16:40:46.297 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:40:46.353 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:40:46.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:46.356 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:40:46.383 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:45 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:46.383 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:45 smithi148 ceph-mon[44520]: pgmap v1455: 27 pgs: 27 active+clean; 3.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 912 op/s 2024-04-23T16:40:46.457 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:40:46.487 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:46.689 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:40:46.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:46.692 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:40:46.702 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:40:46.702 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-04-23T16:40:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-04-23T16:40:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:40:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:211670 5.d 5:b483b486:::benchmark_data_smithi188_113957_object58072:head [delete] snapc 0=[] ondisk+write+known_if_redirected e415)", 2024-04-23T16:40:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:34:56.380256+0000", 2024-04-23T16:40:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 350.318286554, 2024-04-23T16:40:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.063665811000000003, 2024-04-23T16:40:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.703 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 211670 2024-04-23T16:40:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.380256+0000", 2024-04-23T16:40:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.704 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.380256+0000", 2024-04-23T16:40:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.380259+0000", 2024-04-23T16:40:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1049999999999999e-06 2024-04-23T16:40:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.705 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.380270+0000", 2024-04-23T16:40:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1731e-05 2024-04-23T16:40:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.380289+0000", 2024-04-23T16:40:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8272999999999999e-05 2024-04-23T16:40:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.706 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.380298+0000", 2024-04-23T16:40:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8290000000000003e-06 2024-04-23T16:40:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.380355+0000", 2024-04-23T16:40:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6279000000000001e-05 2024-04-23T16:40:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.707 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.380460+0000", 2024-04-23T16:40:46.708 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105072 2024-04-23T16:40:46.708 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.708 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.708 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:40:46.708 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.380549+0000", 2024-04-23T16:40:46.708 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9381000000000004e-05 2024-04-23T16:40:46.708 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.708 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.708 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.708 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.393830+0000", 2024-04-23T16:40:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013281243 2024-04-23T16:40:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.443816+0000", 2024-04-23T16:40:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.049985578000000003 2024-04-23T16:40:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.709 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.443872+0000", 2024-04-23T16:40:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5858e-05 2024-04-23T16:40:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:34:56.443921+0000", 2024-04-23T16:40:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9462000000000002e-05 2024-04-23T16:40:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.710 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:271250 5.d 5:b9ac1130:::benchmark_data_smithi188_113957_object117652:head [delete] snapc 0=[] ondisk+write+known_if_redirected e434)", 2024-04-23T16:40:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:14.735375+0000", 2024-04-23T16:40:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 271.96316686300003, 2024-04-23T16:40:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.069947976999999995, 2024-04-23T16:40:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.711 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 271250 2024-04-23T16:40:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.735375+0000", 2024-04-23T16:40:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.712 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.735375+0000", 2024-04-23T16:40:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.735378+0000", 2024-04-23T16:40:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3870000000000002e-06 2024-04-23T16:40:46.713 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.714 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.714 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.714 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.735383+0000", 2024-04-23T16:40:46.714 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7320000000000003e-06 2024-04-23T16:40:46.714 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.714 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.714 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.714 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.735395+0000", 2024-04-23T16:40:46.714 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2053e-05 2024-04-23T16:40:46.715 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.715 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.715 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.715 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.735403+0000", 2024-04-23T16:40:46.715 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.926e-06 2024-04-23T16:40:46.715 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.715 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.715 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.715 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.735463+0000", 2024-04-23T16:40:46.715 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9247999999999999e-05 2024-04-23T16:40:46.715 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.716 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.716 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.735579+0000", 2024-04-23T16:40:46.716 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011688099999999999 2024-04-23T16:40:46.716 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.716 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.716 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:40:46.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.735686+0000", 2024-04-23T16:40:46.716 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010641099999999999 2024-04-23T16:40:46.716 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.716 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.716 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.740660+0000", 2024-04-23T16:40:46.716 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049736579999999997 2024-04-23T16:40:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.805247+0000", 2024-04-23T16:40:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.064587965999999997 2024-04-23T16:40:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.805286+0000", 2024-04-23T16:40:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8760000000000002e-05 2024-04-23T16:40:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:14.805323+0000", 2024-04-23T16:40:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6955000000000001e-05 2024-04-23T16:40:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:272851 5.d 5:b1ec686b:::benchmark_data_smithi188_113957_object119253:head [delete] snapc 0=[] ondisk+write+known_if_redirected e434)", 2024-04-23T16:40:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:16.728074+0000", 2024-04-23T16:40:46.718 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 269.97046787300002, 2024-04-23T16:40:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.066026524000000003, 2024-04-23T16:40:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 272851 2024-04-23T16:40:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.719 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.728074+0000", 2024-04-23T16:40:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.728074+0000", 2024-04-23T16:40:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.720 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.728077+0000", 2024-04-23T16:40:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7410000000000001e-06 2024-04-23T16:40:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.728084+0000", 2024-04-23T16:40:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2359999999999998e-06 2024-04-23T16:40:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.728109+0000", 2024-04-23T16:40:46.721 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4706999999999999e-05 2024-04-23T16:40:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.728118+0000", 2024-04-23T16:40:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5300000000000002e-06 2024-04-23T16:40:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.722 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.723 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.728181+0000", 2024-04-23T16:40:46.723 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2223000000000004e-05 2024-04-23T16:40:46.723 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.723 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.723 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.723 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.728312+0000", 2024-04-23T16:40:46.723 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000131261 2024-04-23T16:40:46.723 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.723 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.723 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:40:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.728453+0000", 2024-04-23T16:40:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000141123 2024-04-23T16:40:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.736532+0000", 2024-04-23T16:40:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080787619999999997 2024-04-23T16:40:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.724 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.793982+0000", 2024-04-23T16:40:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.057450169000000002 2024-04-23T16:40:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.794044+0000", 2024-04-23T16:40:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1577999999999999e-05 2024-04-23T16:40:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.725 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.794101+0000", 2024-04-23T16:40:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7194000000000002e-05 2024-04-23T16:40:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:272853 5.d 5:b671bc72:::benchmark_data_smithi188_113957_object119255:head [delete] snapc 0=[] ondisk+write+known_if_redirected e434)", 2024-04-23T16:40:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:16.731941+0000", 2024-04-23T16:40:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 269.96660082400001, 2024-04-23T16:40:46.726 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.064283106000000007, 2024-04-23T16:40:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 272853 2024-04-23T16:40:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.727 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.731941+0000", 2024-04-23T16:40:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.731941+0000", 2024-04-23T16:40:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.728 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.731943+0000", 2024-04-23T16:40:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.728e-06 2024-04-23T16:40:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.731948+0000", 2024-04-23T16:40:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3789999999999997e-06 2024-04-23T16:40:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.731960+0000", 2024-04-23T16:40:46.729 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1739000000000001e-05 2024-04-23T16:40:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.731967+0000", 2024-04-23T16:40:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7429999999999996e-06 2024-04-23T16:40:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.730 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.732700+0000", 2024-04-23T16:40:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00073352300000000003 2024-04-23T16:40:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.732801+0000", 2024-04-23T16:40:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010045899999999999 2024-04-23T16:40:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:40:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.732874+0000", 2024-04-23T16:40:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3347000000000001e-05 2024-04-23T16:40:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.736642+0000", 2024-04-23T16:40:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0037680639999999998 2024-04-23T16:40:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.796163+0000", 2024-04-23T16:40:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.059520380999999997 2024-04-23T16:40:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.796195+0000", 2024-04-23T16:40:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1894000000000003e-05 2024-04-23T16:40:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:16.796224+0000", 2024-04-23T16:40:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9848999999999998e-05 2024-04-23T16:40:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.733 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:274769 5.d 5:b1181e5a:::benchmark_data_smithi188_113957_object121171:head [delete] snapc 0=[] ondisk+write+known_if_redirected e435)", 2024-04-23T16:40:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:19.017664+0000", 2024-04-23T16:40:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 267.68087828400002, 2024-04-23T16:40:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.073474631999999998, 2024-04-23T16:40:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.734 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 274769 2024-04-23T16:40:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.017664+0000", 2024-04-23T16:40:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.735 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.017664+0000", 2024-04-23T16:40:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.017667+0000", 2024-04-23T16:40:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.942e-06 2024-04-23T16:40:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.017675+0000", 2024-04-23T16:40:46.736 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2109999999999998e-06 2024-04-23T16:40:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.017692+0000", 2024-04-23T16:40:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6748999999999998e-05 2024-04-23T16:40:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.017701+0000", 2024-04-23T16:40:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7699999999999996e-06 2024-04-23T16:40:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.737 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.017765+0000", 2024-04-23T16:40:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3190999999999999e-05 2024-04-23T16:40:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.017910+0000", 2024-04-23T16:40:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014524300000000001 2024-04-23T16:40:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.738 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:40:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.018033+0000", 2024-04-23T16:40:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012294199999999999 2024-04-23T16:40:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.023771+0000", 2024-04-23T16:40:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057386060000000003 2024-04-23T16:40:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.091034+0000", 2024-04-23T16:40:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067262418000000004 2024-04-23T16:40:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.739 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.091087+0000", 2024-04-23T16:40:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2689999999999999e-05 2024-04-23T16:40:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.091138+0000", 2024-04-23T16:40:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1870000000000003e-05 2024-04-23T16:40:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.740 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:274775 5.d 5:bef548fe:::benchmark_data_smithi188_113957_object121177:head [delete] snapc 0=[] ondisk+write+known_if_redirected e435)", 2024-04-23T16:40:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:19.024206+0000", 2024-04-23T16:40:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 267.67433638599999, 2024-04-23T16:40:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067015101999999993, 2024-04-23T16:40:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 274775 2024-04-23T16:40:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.741 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.024206+0000", 2024-04-23T16:40:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.024206+0000", 2024-04-23T16:40:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.742 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.024208+0000", 2024-04-23T16:40:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9870000000000002e-06 2024-04-23T16:40:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.024214+0000", 2024-04-23T16:40:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.64e-06 2024-04-23T16:40:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.024227+0000", 2024-04-23T16:40:46.743 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3124e-05 2024-04-23T16:40:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.024236+0000", 2024-04-23T16:40:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6130000000000004e-06 2024-04-23T16:40:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.024290+0000", 2024-04-23T16:40:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4021000000000003e-05 2024-04-23T16:40:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.744 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.024424+0000", 2024-04-23T16:40:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000133721 2024-04-23T16:40:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:40:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.024559+0000", 2024-04-23T16:40:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013498300000000001 2024-04-23T16:40:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.040733+0000", 2024-04-23T16:40:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01617416 2024-04-23T16:40:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.091146+0000", 2024-04-23T16:40:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.050413489999999998 2024-04-23T16:40:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.091187+0000", 2024-04-23T16:40:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0800000000000002e-05 2024-04-23T16:40:46.746 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:19.091221+0000", 2024-04-23T16:40:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3562999999999997e-05 2024-04-23T16:40:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:291106 5.d 5:ba6f9ce3:::benchmark_data_smithi188_113957_object137508:head [delete] snapc 0=[] ondisk+write+known_if_redirected e442)", 2024-04-23T16:40:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:38.048239+0000", 2024-04-23T16:40:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 248.65030337300001, 2024-04-23T16:40:46.747 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067926153000000003, 2024-04-23T16:40:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 291106 2024-04-23T16:40:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.048239+0000", 2024-04-23T16:40:46.748 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.048239+0000", 2024-04-23T16:40:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.048242+0000", 2024-04-23T16:40:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1370000000000002e-06 2024-04-23T16:40:46.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.048251+0000", 2024-04-23T16:40:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1139999999999995e-06 2024-04-23T16:40:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.048269+0000", 2024-04-23T16:40:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7688000000000001e-05 2024-04-23T16:40:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.048279+0000", 2024-04-23T16:40:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0335e-05 2024-04-23T16:40:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.048347+0000", 2024-04-23T16:40:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8032000000000002e-05 2024-04-23T16:40:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.751 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.048469+0000", 2024-04-23T16:40:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012229 2024-04-23T16:40:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:40:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.048575+0000", 2024-04-23T16:40:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010592800000000001 2024-04-23T16:40:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.053512+0000", 2024-04-23T16:40:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004936523 2024-04-23T16:40:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.752 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.116043+0000", 2024-04-23T16:40:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062531478000000001 2024-04-23T16:40:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.116114+0000", 2024-04-23T16:40:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0481999999999994e-05 2024-04-23T16:40:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.753 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.116165+0000", 2024-04-23T16:40:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1146000000000001e-05 2024-04-23T16:40:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.25717.0:291110 5.d 5:b746b310:::benchmark_data_smithi188_113957_object137512:head [delete] snapc 0=[] ondisk+write+known_if_redirected e442)", 2024-04-23T16:40:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:36:38.053981+0000", 2024-04-23T16:40:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 248.64456133799999, 2024-04-23T16:40:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062318658999999998, 2024-04-23T16:40:46.754 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.25717", 2024-04-23T16:40:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/3223835157", 2024-04-23T16:40:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 291110 2024-04-23T16:40:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.053981+0000", 2024-04-23T16:40:46.755 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.053981+0000", 2024-04-23T16:40:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.053983+0000", 2024-04-23T16:40:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3949999999999999e-06 2024-04-23T16:40:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.053990+0000", 2024-04-23T16:40:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1149999999999996e-06 2024-04-23T16:40:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.054004+0000", 2024-04-23T16:40:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3305000000000001e-05 2024-04-23T16:40:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.054014+0000", 2024-04-23T16:40:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9699999999999994e-06 2024-04-23T16:40:46.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.054063+0000", 2024-04-23T16:40:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9614999999999998e-05 2024-04-23T16:40:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.054167+0000", 2024-04-23T16:40:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104028 2024-04-23T16:40:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.758 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 5", 2024-04-23T16:40:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.054259+0000", 2024-04-23T16:40:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1602999999999997e-05 2024-04-23T16:40:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.058066+0000", 2024-04-23T16:40:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0038074340000000002 2024-04-23T16:40:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.759 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.116214+0000", 2024-04-23T16:40:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.058147685999999997 2024-04-23T16:40:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.116261+0000", 2024-04-23T16:40:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7023e-05 2024-04-23T16:40:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:36:38.116299+0000", 2024-04-23T16:40:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8485000000000002e-05 2024-04-23T16:40:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.760 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16308.0:135950 6.8 6:18870227:::benchmark_data_smithi188_130221_object135949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e476)", 2024-04-23T16:40:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:38:43.071615+0000", 2024-04-23T16:40:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 123.626927427, 2024-04-23T16:40:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.10452592400000001, 2024-04-23T16:40:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16308", 2024-04-23T16:40:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:40:46.761 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 135950 2024-04-23T16:40:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.071615+0000", 2024-04-23T16:40:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.071615+0000", 2024-04-23T16:40:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.762 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.763 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.763 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.071616+0000", 2024-04-23T16:40:46.763 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.505e-06 2024-04-23T16:40:46.763 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.763 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.763 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.763 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.071637+0000", 2024-04-23T16:40:46.763 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0877999999999999e-05 2024-04-23T16:40:46.763 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.763 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.763 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.763 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.071650+0000", 2024-04-23T16:40:46.764 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2866e-05 2024-04-23T16:40:46.764 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.764 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.764 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.764 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.071658+0000", 2024-04-23T16:40:46.764 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5799999999999992e-06 2024-04-23T16:40:46.764 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.764 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.764 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.764 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.075666+0000", 2024-04-23T16:40:46.764 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0040079959999999998 2024-04-23T16:40:46.764 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.764 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.765 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.765 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.075809+0000", 2024-04-23T16:40:46.765 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000142211 2024-04-23T16:40:46.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.765 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.765 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 7", 2024-04-23T16:40:46.765 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.075917+0000", 2024-04-23T16:40:46.765 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010791299999999999 2024-04-23T16:40:46.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.765 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.765 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.766 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.079272+0000", 2024-04-23T16:40:46.766 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033552209999999998 2024-04-23T16:40:46.766 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.766 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.766 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.766 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.176071+0000", 2024-04-23T16:40:46.766 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.096799607999999995 2024-04-23T16:40:46.766 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.766 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.766 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.766 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.176111+0000", 2024-04-23T16:40:46.766 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9978000000000002e-05 2024-04-23T16:40:46.767 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.767 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.767 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.767 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.176141+0000", 2024-04-23T16:40:46.767 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9167999999999999e-05 2024-04-23T16:40:46.767 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.767 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.767 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.767 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.767 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.767 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16308.0:135956 6.8 6:164d9b48:::benchmark_data_smithi188_130221_object135955:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e476)", 2024-04-23T16:40:46.767 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:38:43.079475+0000", 2024-04-23T16:40:46.767 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 123.619066997, 2024-04-23T16:40:46.767 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.096768011000000001, 2024-04-23T16:40:46.768 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.768 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.768 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.768 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.768 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16308", 2024-04-23T16:40:46.768 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:40:46.768 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 135956 2024-04-23T16:40:46.768 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.768 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.768 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.768 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.769 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.079475+0000", 2024-04-23T16:40:46.769 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.769 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.769 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.769 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.769 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.079475+0000", 2024-04-23T16:40:46.769 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.769 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.769 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.770 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.770 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.079477+0000", 2024-04-23T16:40:46.770 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.759e-06 2024-04-23T16:40:46.770 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.770 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.770 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.770 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.079498+0000", 2024-04-23T16:40:46.770 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1407999999999999e-05 2024-04-23T16:40:46.770 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.770 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.770 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.770 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.079509+0000", 2024-04-23T16:40:46.770 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0924000000000001e-05 2024-04-23T16:40:46.770 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.771 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.771 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.771 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.079516+0000", 2024-04-23T16:40:46.771 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1520000000000003e-06 2024-04-23T16:40:46.771 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.771 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.771 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.771 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.079561+0000", 2024-04-23T16:40:46.771 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5073e-05 2024-04-23T16:40:46.771 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.771 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.771 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.771 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.079668+0000", 2024-04-23T16:40:46.772 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000106925 2024-04-23T16:40:46.772 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.772 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.772 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 7", 2024-04-23T16:40:46.772 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.079786+0000", 2024-04-23T16:40:46.772 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000117873 2024-04-23T16:40:46.772 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.772 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.772 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.772 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.083156+0000", 2024-04-23T16:40:46.772 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003370035 2024-04-23T16:40:46.772 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.772 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.772 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.773 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.176173+0000", 2024-04-23T16:40:46.773 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.093016952999999999 2024-04-23T16:40:46.773 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.773 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.773 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.773 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.176204+0000", 2024-04-23T16:40:46.773 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1272000000000001e-05 2024-04-23T16:40:46.773 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.773 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.773 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.774 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.176243+0000", 2024-04-23T16:40:46.774 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8637000000000003e-05 2024-04-23T16:40:46.774 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.774 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.774 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.774 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.774 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.774 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16308.0:135965 6.8 6:1c706532:::benchmark_data_smithi188_130221_object135964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e476)", 2024-04-23T16:40:46.774 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:38:43.089310+0000", 2024-04-23T16:40:46.774 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 123.60923203199999, 2024-04-23T16:40:46.774 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.089574450999999999, 2024-04-23T16:40:46.774 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.774 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.775 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.775 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.775 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16308", 2024-04-23T16:40:46.775 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:40:46.775 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 135965 2024-04-23T16:40:46.775 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.775 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.775 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.775 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.775 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.089310+0000", 2024-04-23T16:40:46.775 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.776 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.776 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.776 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.776 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.089310+0000", 2024-04-23T16:40:46.776 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.776 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.776 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.776 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.776 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.089312+0000", 2024-04-23T16:40:46.777 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.95e-06 2024-04-23T16:40:46.777 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.777 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.777 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.777 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.089337+0000", 2024-04-23T16:40:46.777 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5143999999999998e-05 2024-04-23T16:40:46.777 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.777 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.777 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.777 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.089350+0000", 2024-04-23T16:40:46.777 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2865e-05 2024-04-23T16:40:46.778 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.778 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.778 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.778 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.089357+0000", 2024-04-23T16:40:46.778 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6440000000000003e-06 2024-04-23T16:40:46.778 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.778 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.778 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.778 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.089400+0000", 2024-04-23T16:40:46.778 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3225e-05 2024-04-23T16:40:46.778 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.779 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.779 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.779 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.089506+0000", 2024-04-23T16:40:46.779 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000106505 2024-04-23T16:40:46.779 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.779 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.779 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 7", 2024-04-23T16:40:46.779 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.089595+0000", 2024-04-23T16:40:46.779 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9090000000000003e-05 2024-04-23T16:40:46.779 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.779 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.779 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.780 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.094179+0000", 2024-04-23T16:40:46.780 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045833369999999998 2024-04-23T16:40:46.780 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.780 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.780 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.780 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.178830+0000", 2024-04-23T16:40:46.780 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.084651405999999998 2024-04-23T16:40:46.780 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.780 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.780 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.780 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.178861+0000", 2024-04-23T16:40:46.780 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1087000000000002e-05 2024-04-23T16:40:46.780 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.781 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.781 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.781 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.178884+0000", 2024-04-23T16:40:46.781 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3198e-05 2024-04-23T16:40:46.781 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.781 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.781 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.781 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.781 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.781 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16308.0:135966 6.8 6:12541106:::benchmark_data_smithi188_130221_object135965:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e476)", 2024-04-23T16:40:46.781 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:38:43.089372+0000", 2024-04-23T16:40:46.781 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 123.609170444, 2024-04-23T16:40:46.782 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.102001679, 2024-04-23T16:40:46.782 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.782 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.782 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.782 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.782 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16308", 2024-04-23T16:40:46.782 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:40:46.782 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 135966 2024-04-23T16:40:46.782 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.782 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.783 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.783 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.783 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.089372+0000", 2024-04-23T16:40:46.783 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.783 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.783 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.783 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.783 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.089372+0000", 2024-04-23T16:40:46.783 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.783 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.783 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.783 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.783 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.089372+0000", 2024-04-23T16:40:46.784 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7700000000000004e-07 2024-04-23T16:40:46.784 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.784 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.784 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.784 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.089388+0000", 2024-04-23T16:40:46.784 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5409000000000001e-05 2024-04-23T16:40:46.784 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.784 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.784 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.784 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.089392+0000", 2024-04-23T16:40:46.784 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2080000000000002e-06 2024-04-23T16:40:46.784 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.785 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.785 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.785 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.089396+0000", 2024-04-23T16:40:46.785 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.703e-06 2024-04-23T16:40:46.785 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.785 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.785 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.785 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.092051+0000", 2024-04-23T16:40:46.785 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026551280000000001 2024-04-23T16:40:46.785 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.785 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.785 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.785 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.092182+0000", 2024-04-23T16:40:46.786 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000131185 2024-04-23T16:40:46.786 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.786 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.786 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 7", 2024-04-23T16:40:46.786 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.092272+0000", 2024-04-23T16:40:46.786 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9932999999999994e-05 2024-04-23T16:40:46.786 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.786 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.786 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.786 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.094314+0000", 2024-04-23T16:40:46.786 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020425769999999998 2024-04-23T16:40:46.786 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.787 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.787 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.787 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.191315+0000", 2024-04-23T16:40:46.787 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.097000535999999998 2024-04-23T16:40:46.787 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.787 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.787 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.787 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.191347+0000", 2024-04-23T16:40:46.787 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2512999999999999e-05 2024-04-23T16:40:46.787 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.787 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.787 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.788 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.191373+0000", 2024-04-23T16:40:46.788 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5809999999999999e-05 2024-04-23T16:40:46.788 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.788 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.788 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.788 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.788 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.788 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16308.0:135980 6.8 6:123d50d1:::benchmark_data_smithi188_130221_object135979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e476)", 2024-04-23T16:40:46.788 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:38:43.110813+0000", 2024-04-23T16:40:46.788 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 123.587729192, 2024-04-23T16:40:46.788 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.080683851000000001, 2024-04-23T16:40:46.788 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.789 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.789 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.789 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.789 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16308", 2024-04-23T16:40:46.789 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:40:46.789 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 135980 2024-04-23T16:40:46.789 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.789 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.789 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.789 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.789 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.110813+0000", 2024-04-23T16:40:46.790 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.790 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.790 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.790 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.790 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.110813+0000", 2024-04-23T16:40:46.790 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.790 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.790 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.790 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.790 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.110815+0000", 2024-04-23T16:40:46.790 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.869e-06 2024-04-23T16:40:46.790 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.791 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.791 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.791 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.110837+0000", 2024-04-23T16:40:46.791 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1911000000000001e-05 2024-04-23T16:40:46.791 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.791 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.791 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.791 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.110852+0000", 2024-04-23T16:40:46.791 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5138e-05 2024-04-23T16:40:46.791 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.791 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.791 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.791 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.110861+0000", 2024-04-23T16:40:46.791 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3640000000000005e-06 2024-04-23T16:40:46.792 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.792 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.792 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.792 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.110920+0000", 2024-04-23T16:40:46.792 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.838e-05 2024-04-23T16:40:46.792 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.792 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.792 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.792 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.111050+0000", 2024-04-23T16:40:46.792 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013077499999999999 2024-04-23T16:40:46.792 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.792 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.792 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 7", 2024-04-23T16:40:46.793 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.111174+0000", 2024-04-23T16:40:46.793 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000123738 2024-04-23T16:40:46.793 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.793 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.793 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.793 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.120084+0000", 2024-04-23T16:40:46.793 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0089099160000000004 2024-04-23T16:40:46.793 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.793 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.793 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.793 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.191412+0000", 2024-04-23T16:40:46.793 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.071327893000000003 2024-04-23T16:40:46.794 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.794 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.794 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.794 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.191458+0000", 2024-04-23T16:40:46.794 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5762e-05 2024-04-23T16:40:46.794 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.794 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.794 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.794 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.191497+0000", 2024-04-23T16:40:46.794 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9104999999999999e-05 2024-04-23T16:40:46.794 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.794 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.795 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.795 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.795 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.795 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16308.0:135982 6.8 6:19ca0610:::benchmark_data_smithi188_130221_object135981:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e476)", 2024-04-23T16:40:46.795 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:38:43.114031+0000", 2024-04-23T16:40:46.795 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 123.584510786, 2024-04-23T16:40:46.795 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.077636831000000003, 2024-04-23T16:40:46.795 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.795 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.795 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.795 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.795 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16308", 2024-04-23T16:40:46.796 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:40:46.796 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 135982 2024-04-23T16:40:46.796 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.796 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.796 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.796 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.796 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.114031+0000", 2024-04-23T16:40:46.796 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.796 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.796 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.796 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.796 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.114031+0000", 2024-04-23T16:40:46.796 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.796 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.797 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.797 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.797 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.114033+0000", 2024-04-23T16:40:46.797 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7540000000000001e-06 2024-04-23T16:40:46.797 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.797 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.797 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.797 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.114073+0000", 2024-04-23T16:40:46.797 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0117000000000002e-05 2024-04-23T16:40:46.797 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.797 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.798 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.798 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.114085+0000", 2024-04-23T16:40:46.798 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1858000000000001e-05 2024-04-23T16:40:46.798 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.798 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.798 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.798 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.114101+0000", 2024-04-23T16:40:46.798 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5641000000000002e-05 2024-04-23T16:40:46.798 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.798 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.798 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.799 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.114158+0000", 2024-04-23T16:40:46.799 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7151999999999998e-05 2024-04-23T16:40:46.799 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.799 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.799 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.799 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.114290+0000", 2024-04-23T16:40:46.799 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013201599999999999 2024-04-23T16:40:46.799 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.799 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.800 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 7", 2024-04-23T16:40:46.800 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.114435+0000", 2024-04-23T16:40:46.800 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014487699999999999 2024-04-23T16:40:46.800 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.800 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.800 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.800 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.120087+0000", 2024-04-23T16:40:46.800 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0056526199999999997 2024-04-23T16:40:46.800 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.801 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.801 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.801 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.191562+0000", 2024-04-23T16:40:46.801 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.071474788999999997 2024-04-23T16:40:46.801 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.801 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.801 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.801 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.191628+0000", 2024-04-23T16:40:46.801 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5587999999999996e-05 2024-04-23T16:40:46.801 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.801 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.801 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.802 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.191668+0000", 2024-04-23T16:40:46.802 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0419e-05 2024-04-23T16:40:46.802 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.802 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.802 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.802 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.802 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.802 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16308.0:135983 6.8 6:1fe4fb73:::benchmark_data_smithi188_130221_object135982:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e476)", 2024-04-23T16:40:46.802 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:38:43.114118+0000", 2024-04-23T16:40:46.802 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 123.584423867, 2024-04-23T16:40:46.802 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.077671467999999994, 2024-04-23T16:40:46.802 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.803 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.803 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.803 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.803 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16308", 2024-04-23T16:40:46.803 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:40:46.803 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 135983 2024-04-23T16:40:46.803 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.803 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.803 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.803 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.803 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.114118+0000", 2024-04-23T16:40:46.803 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.803 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.803 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.804 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.804 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.114118+0000", 2024-04-23T16:40:46.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.804 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.804 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.804 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.804 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.114119+0000", 2024-04-23T16:40:46.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6300000000000004e-07 2024-04-23T16:40:46.804 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.804 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.804 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.804 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.114142+0000", 2024-04-23T16:40:46.805 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3397e-05 2024-04-23T16:40:46.805 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.805 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.805 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.805 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.114150+0000", 2024-04-23T16:40:46.805 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5689999999999999e-06 2024-04-23T16:40:46.805 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.805 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.805 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.805 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.114157+0000", 2024-04-23T16:40:46.805 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.435e-06 2024-04-23T16:40:46.805 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.805 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.806 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.806 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.117710+0000", 2024-04-23T16:40:46.806 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0035527409999999999 2024-04-23T16:40:46.806 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.806 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.806 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.806 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.117833+0000", 2024-04-23T16:40:46.806 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000122457 2024-04-23T16:40:46.806 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.806 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.806 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 7", 2024-04-23T16:40:46.806 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.117925+0000", 2024-04-23T16:40:46.806 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2122999999999999e-05 2024-04-23T16:40:46.807 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.807 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.807 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.807 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.120218+0000", 2024-04-23T16:40:46.807 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022936789999999999 2024-04-23T16:40:46.807 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.807 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.807 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.807 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.191715+0000", 2024-04-23T16:40:46.807 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.071496692000000001 2024-04-23T16:40:46.807 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.808 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.808 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.808 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.191767+0000", 2024-04-23T16:40:46.808 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2219999999999998e-05 2024-04-23T16:40:46.808 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.808 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.808 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.808 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:38:43.191790+0000", 2024-04-23T16:40:46.808 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2291999999999999e-05 2024-04-23T16:40:46.808 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.808 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.808 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.809 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.809 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.809 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16308.0:171883 6.8 6:1a659558:::benchmark_data_smithi188_130221_object17422:head [delete] snapc 0=[] ondisk+write+known_if_redirected e486)", 2024-04-23T16:40:46.809 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:39:18.332865+0000", 2024-04-23T16:40:46.809 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 88.365676859000004, 2024-04-23T16:40:46.809 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.087669823999999993, 2024-04-23T16:40:46.809 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.809 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.809 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.809 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.809 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16308", 2024-04-23T16:40:46.809 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:40:46.809 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 171883 2024-04-23T16:40:46.810 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.810 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.810 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.810 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.810 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.332865+0000", 2024-04-23T16:40:46.810 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.810 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.810 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.810 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.810 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.332865+0000", 2024-04-23T16:40:46.811 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.811 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.811 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.811 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.811 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.332868+0000", 2024-04-23T16:40:46.811 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4389999999999999e-06 2024-04-23T16:40:46.811 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.811 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.811 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.811 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.332875+0000", 2024-04-23T16:40:46.811 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4000000000000003e-06 2024-04-23T16:40:46.811 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.811 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.812 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.812 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.332892+0000", 2024-04-23T16:40:46.812 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7377e-05 2024-04-23T16:40:46.812 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.812 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.812 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.812 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.332903+0000", 2024-04-23T16:40:46.812 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0827e-05 2024-04-23T16:40:46.812 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.812 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.812 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.812 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.332959+0000", 2024-04-23T16:40:46.812 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5336e-05 2024-04-23T16:40:46.813 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.813 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.813 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.813 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.333048+0000", 2024-04-23T16:40:46.813 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9016000000000003e-05 2024-04-23T16:40:46.813 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.813 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.813 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 7", 2024-04-23T16:40:46.813 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.333136+0000", 2024-04-23T16:40:46.813 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8097999999999996e-05 2024-04-23T16:40:46.813 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.813 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.813 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.814 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.340369+0000", 2024-04-23T16:40:46.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007232835 2024-04-23T16:40:46.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.814 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.814 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.814 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.420454+0000", 2024-04-23T16:40:46.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.080085208000000005 2024-04-23T16:40:46.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.814 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.814 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.814 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.420488+0000", 2024-04-23T16:40:46.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4081000000000001e-05 2024-04-23T16:40:46.815 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.815 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.815 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.815 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.420535+0000", 2024-04-23T16:40:46.815 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7206999999999997e-05 2024-04-23T16:40:46.815 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.815 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.815 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.815 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.815 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.815 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16308.0:171885 6.8 6:12ce8859:::benchmark_data_smithi188_130221_object17424:head [delete] snapc 0=[] ondisk+write+known_if_redirected e486)", 2024-04-23T16:40:46.815 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:39:18.332925+0000", 2024-04-23T16:40:46.816 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 88.365617553999996, 2024-04-23T16:40:46.816 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.087749643000000002, 2024-04-23T16:40:46.816 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.816 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.816 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.816 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.816 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16308", 2024-04-23T16:40:46.816 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:40:46.816 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 171885 2024-04-23T16:40:46.816 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.816 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.816 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.816 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.817 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.332925+0000", 2024-04-23T16:40:46.817 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.817 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.817 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.817 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.817 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.332925+0000", 2024-04-23T16:40:46.817 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.817 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.817 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.817 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.817 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.332925+0000", 2024-04-23T16:40:46.817 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8400000000000004e-07 2024-04-23T16:40:46.817 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.817 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.818 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.818 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.332928+0000", 2024-04-23T16:40:46.818 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.818e-06 2024-04-23T16:40:46.818 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.818 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.818 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.818 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.332933+0000", 2024-04-23T16:40:46.818 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4450000000000004e-06 2024-04-23T16:40:46.818 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.818 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.818 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.818 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.332937+0000", 2024-04-23T16:40:46.819 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8809999999999998e-06 2024-04-23T16:40:46.819 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.819 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.819 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.819 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.336894+0000", 2024-04-23T16:40:46.819 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039564400000000003 2024-04-23T16:40:46.819 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.819 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.819 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.819 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.336988+0000", 2024-04-23T16:40:46.819 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4550999999999996e-05 2024-04-23T16:40:46.819 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.820 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.820 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 7", 2024-04-23T16:40:46.820 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.337087+0000", 2024-04-23T16:40:46.820 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8444999999999997e-05 2024-04-23T16:40:46.820 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.820 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.820 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.820 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.340537+0000", 2024-04-23T16:40:46.820 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034503139999999999 2024-04-23T16:40:46.820 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.820 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.820 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.821 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.420580+0000", 2024-04-23T16:40:46.821 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.080042749999999996 2024-04-23T16:40:46.821 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.821 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.821 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.821 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.420632+0000", 2024-04-23T16:40:46.821 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2676000000000002e-05 2024-04-23T16:40:46.821 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.821 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.821 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.821 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.420674+0000", 2024-04-23T16:40:46.821 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1739000000000001e-05 2024-04-23T16:40:46.822 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.822 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.822 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.822 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.822 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.822 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16308.0:171896 6.8 6:18d37b77:::benchmark_data_smithi188_130221_object17435:head [delete] snapc 0=[] ondisk+write+known_if_redirected e486)", 2024-04-23T16:40:46.822 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:39:18.343192+0000", 2024-04-23T16:40:46.822 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 88.355350212000005, 2024-04-23T16:40:46.822 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.095203370999999995, 2024-04-23T16:40:46.822 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.822 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.823 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.823 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.823 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16308", 2024-04-23T16:40:46.823 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:40:46.823 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 171896 2024-04-23T16:40:46.823 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.823 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.823 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.823 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.823 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.343192+0000", 2024-04-23T16:40:46.823 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.824 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.824 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.824 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.824 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.343192+0000", 2024-04-23T16:40:46.824 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.824 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.824 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.824 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.824 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.343194+0000", 2024-04-23T16:40:46.825 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9850000000000001e-06 2024-04-23T16:40:46.825 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.825 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.825 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.825 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.343200+0000", 2024-04-23T16:40:46.825 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0870000000000003e-06 2024-04-23T16:40:46.825 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.825 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.825 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.825 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.343212+0000", 2024-04-23T16:40:46.825 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2407e-05 2024-04-23T16:40:46.826 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.826 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.826 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.826 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.343221+0000", 2024-04-23T16:40:46.826 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5920000000000001e-06 2024-04-23T16:40:46.826 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.826 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.826 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.826 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.343275+0000", 2024-04-23T16:40:46.826 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3981999999999999e-05 2024-04-23T16:40:46.826 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.826 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.827 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.827 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.343360+0000", 2024-04-23T16:40:46.827 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4649999999999998e-05 2024-04-23T16:40:46.827 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.827 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.827 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 7", 2024-04-23T16:40:46.827 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.343434+0000", 2024-04-23T16:40:46.827 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4039e-05 2024-04-23T16:40:46.827 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.827 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.827 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.827 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.346821+0000", 2024-04-23T16:40:46.827 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033873309999999999 2024-04-23T16:40:46.828 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.828 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.828 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.828 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.438287+0000", 2024-04-23T16:40:46.828 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.091466140000000001 2024-04-23T16:40:46.828 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.828 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.828 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.828 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.438346+0000", 2024-04-23T16:40:46.828 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9227000000000001e-05 2024-04-23T16:40:46.828 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.829 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.829 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.829 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.438395+0000", 2024-04-23T16:40:46.829 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8931000000000003e-05 2024-04-23T16:40:46.829 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.829 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.829 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.829 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.829 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.829 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16308.0:171903 6.8 6:14026a5b:::benchmark_data_smithi188_130221_object17442:head [delete] snapc 0=[] ondisk+write+known_if_redirected e486)", 2024-04-23T16:40:46.829 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:39:18.349466+0000", 2024-04-23T16:40:46.829 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 88.349075647999996, 2024-04-23T16:40:46.829 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.089092357999999996, 2024-04-23T16:40:46.830 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.830 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.830 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.830 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.830 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16308", 2024-04-23T16:40:46.830 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:40:46.830 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 171903 2024-04-23T16:40:46.830 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.830 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.830 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.830 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.830 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.349466+0000", 2024-04-23T16:40:46.831 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.831 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.831 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.831 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.831 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.349466+0000", 2024-04-23T16:40:46.831 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.831 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.831 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.831 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.831 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.349468+0000", 2024-04-23T16:40:46.831 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4479999999999999e-06 2024-04-23T16:40:46.832 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.832 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.832 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.832 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.349473+0000", 2024-04-23T16:40:46.832 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0690000000000001e-06 2024-04-23T16:40:46.832 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.832 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.832 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.832 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.349484+0000", 2024-04-23T16:40:46.832 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0662000000000001e-05 2024-04-23T16:40:46.832 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.832 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.833 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.833 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.349492+0000", 2024-04-23T16:40:46.833 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1389999999999995e-06 2024-04-23T16:40:46.833 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.833 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.833 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.833 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.349539+0000", 2024-04-23T16:40:46.833 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7348999999999997e-05 2024-04-23T16:40:46.833 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.833 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.833 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.833 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.349617+0000", 2024-04-23T16:40:46.834 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8238999999999994e-05 2024-04-23T16:40:46.834 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.834 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.834 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 7", 2024-04-23T16:40:46.834 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.349687+0000", 2024-04-23T16:40:46.834 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0036999999999994e-05 2024-04-23T16:40:46.834 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.834 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.834 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.834 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.352435+0000", 2024-04-23T16:40:46.835 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027478149999999998 2024-04-23T16:40:46.835 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.835 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.835 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.835 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.438454+0000", 2024-04-23T16:40:46.835 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.086018654999999999 2024-04-23T16:40:46.835 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.835 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.835 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.836 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.438513+0000", 2024-04-23T16:40:46.836 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8943999999999997e-05 2024-04-23T16:40:46.836 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.836 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.836 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.836 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.438559+0000", 2024-04-23T16:40:46.836 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6001000000000002e-05 2024-04-23T16:40:46.836 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.836 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.836 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.836 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.836 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.837 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16308.0:171919 6.8 6:1dd3b9a1:::benchmark_data_smithi188_130221_object17458:head [delete] snapc 0=[] ondisk+write+known_if_redirected e486)", 2024-04-23T16:40:46.837 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:39:18.371727+0000", 2024-04-23T16:40:46.837 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 88.326814713000005, 2024-04-23T16:40:46.837 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.066995974999999999, 2024-04-23T16:40:46.837 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:40:46.837 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:40:46.837 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:40:46.837 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:40:46.837 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16308", 2024-04-23T16:40:46.837 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:40:46.837 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 171919 2024-04-23T16:40:46.837 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.838 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:40:46.838 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.838 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:40:46.838 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.371727+0000", 2024-04-23T16:40:46.838 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.838 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.838 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.838 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:40:46.838 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.371727+0000", 2024-04-23T16:40:46.838 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:40:46.838 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.839 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.839 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:40:46.839 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.371730+0000", 2024-04-23T16:40:46.839 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4420000000000001e-06 2024-04-23T16:40:46.839 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.839 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.839 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:40:46.839 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.371738+0000", 2024-04-23T16:40:46.839 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7919999999999993e-06 2024-04-23T16:40:46.839 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.839 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.839 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:40:46.839 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.371755+0000", 2024-04-23T16:40:46.839 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7598e-05 2024-04-23T16:40:46.840 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.840 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.840 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:40:46.840 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.371764+0000", 2024-04-23T16:40:46.840 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.738e-06 2024-04-23T16:40:46.840 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.840 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.840 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:40:46.840 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.371824+0000", 2024-04-23T16:40:46.840 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0219000000000001e-05 2024-04-23T16:40:46.840 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.840 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.840 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:40:46.841 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.371934+0000", 2024-04-23T16:40:46.841 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010997699999999999 2024-04-23T16:40:46.841 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.841 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.841 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 7", 2024-04-23T16:40:46.841 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.372010+0000", 2024-04-23T16:40:46.841 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5402999999999996e-05 2024-04-23T16:40:46.841 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.841 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.841 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:40:46.841 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.375834+0000", 2024-04-23T16:40:46.841 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0038244400000000001 2024-04-23T16:40:46.841 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.841 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.842 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:40:46.842 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.438622+0000", 2024-04-23T16:40:46.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062787929000000006 2024-04-23T16:40:46.842 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.842 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.842 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:40:46.842 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.438680+0000", 2024-04-23T16:40:46.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8182e-05 2024-04-23T16:40:46.842 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:40:46.842 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:40:46.842 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:40:46.842 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:39:18.438723+0000", 2024-04-23T16:40:46.843 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3254999999999998e-05 2024-04-23T16:40:46.843 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.843 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.843 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.843 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:40:46.843 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:40:46.843 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:40:46.843 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:40:46.882 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:46.965 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:40:46.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:46.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:40:47.069 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:40:47.104 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:47.238 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:40:47.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:47.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:40:47.346 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:40:47.375 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:47.450 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:40:47.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:47.456 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:40:47.476 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-23T16:40:47.557 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:40:47.605 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:47.634 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:40:47.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:47.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:40:47.651 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:40:47.747 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:40:47.800 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:47.947 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:40:48.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:47 smithi103 ceph-mon[39629]: pgmap v1456: 27 pgs: 27 active+clean; 3.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-23T16:40:48.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:47 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/809730017' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:40:48.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:47 smithi188 ceph-mon[44101]: pgmap v1456: 27 pgs: 27 active+clean; 3.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-23T16:40:48.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:47 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/809730017' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:40:48.363 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:40:48.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:48.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:40:48.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:47 smithi148 ceph-mon[44520]: pgmap v1456: 27 pgs: 27 active+clean; 3.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-23T16:40:48.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:47 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/809730017' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:40:48.466 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:40:48.544 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:48.586 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:40:48.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:48.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:40:48.701 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:40:48.736 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:48 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:40:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:48 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/809730017' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:40:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:48 smithi103 ceph-mon[39629]: osdmap e521: 12 total, 12 up, 10 in 2024-04-23T16:40:49.187 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:40:49.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:49.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:40:49.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:48 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:40:49.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:48 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/809730017' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:40:49.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:48 smithi188 ceph-mon[44101]: osdmap e521: 12 total, 12 up, 10 in 2024-04-23T16:40:49.301 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:40:49.354 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:48 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:40:49.354 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:48 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/809730017' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:40:49.354 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:48 smithi148 ceph-mon[44520]: osdmap e521: 12 total, 12 up, 10 in 2024-04-23T16:40:49.366 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:49.395 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:40:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:49.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:40:49.512 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:40:49.570 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:49.617 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:40:49.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:49.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:40:49.722 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:40:49.755 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:50.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:49 smithi103 ceph-mon[39629]: pgmap v1458: 27 pgs: 27 active+clean; 3.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-23T16:40:50.223 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:40:50.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:49 smithi188 ceph-mon[44101]: pgmap v1458: 27 pgs: 27 active+clean; 3.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-23T16:40:50.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:50.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:40:50.295 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:40:50.327 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:40:50.356 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:40:50.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:49 smithi148 ceph-mon[44520]: pgmap v1458: 27 pgs: 27 active+clean; 3.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-23T16:40:50.382 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:50.411 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:40:50.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:50.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:40:50.518 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:40:50.568 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:50.625 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:40:50.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:50.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:40:50.735 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:40:50.764 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:50.786 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:40:50.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:50.791 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:40:50.892 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:40:50.922 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:50.941 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:40:50.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:50.944 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:40:51.045 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:40:51.091 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:51.729 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:40:52.158 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:40:52.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:52.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:40:52.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:51 smithi188 ceph-mon[44101]: pgmap v1459: 27 pgs: 27 active+clean; 3.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-23T16:40:52.269 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:40:52.304 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:51 smithi148 ceph-mon[44520]: pgmap v1459: 27 pgs: 27 active+clean; 3.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-23T16:40:52.309 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:52.340 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:40:52.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:52.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:40:52.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:51 smithi103 ceph-mon[39629]: pgmap v1459: 27 pgs: 27 active+clean; 3.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-23T16:40:52.448 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:40:52.485 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:53.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:52 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2640952883' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:40:53.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:52 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2640952883' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:40:53.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:52 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2640952883' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:40:53.547 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:40:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:53.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:40:53.652 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:40:53.704 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:53.718 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.9 to 1 (10000) 2024-04-23T16:40:53.734 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:40:53.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:53.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:40:53.846 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:40:53.874 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:53.898 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:40:53.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:53.902 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:40:53.958 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:53 smithi148 ceph-mon[44520]: pgmap v1460: 27 pgs: 27 active+clean; 3.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:40:53.958 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:53 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2640952883' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:40:53.958 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:53 smithi148 ceph-mon[44520]: osdmap e522: 12 total, 12 up, 10 in 2024-04-23T16:40:54.003 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:40:54.044 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:54.097 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:40:54.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:54.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:40:54.216 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:40:54.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:53 smithi188 ceph-mon[44101]: pgmap v1460: 27 pgs: 27 active+clean; 3.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:40:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:53 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2640952883' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:40:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:53 smithi188 ceph-mon[44101]: osdmap e522: 12 total, 12 up, 10 in 2024-04-23T16:40:54.253 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:53 smithi103 ceph-mon[39629]: pgmap v1460: 27 pgs: 27 active+clean; 3.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:40:54.253 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:53 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2640952883' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:40:54.253 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:53 smithi103 ceph-mon[39629]: osdmap e522: 12 total, 12 up, 10 in 2024-04-23T16:40:54.255 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:54.711 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:40:54.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:54.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:40:54.748 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 11 1 2024-04-23T16:40:54.829 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:40:54.873 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:55.141 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:40:55.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:55.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:40:55.245 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:40:55.273 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:55.294 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:40:55.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:55.297 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:40:55.398 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:40:55.429 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:55.794 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:40:55.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:55.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:40:55.907 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:40:55.935 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:55.956 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:40:55.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:55.962 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:40:56.039 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:55 smithi103 ceph-mon[39629]: pgmap v1462: 27 pgs: 27 active+clean; 2.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 918 op/s 2024-04-23T16:40:56.063 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:40:56.104 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:56.135 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:40:56.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:56.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:40:56.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:55 smithi188 ceph-mon[44101]: pgmap v1462: 27 pgs: 27 active+clean; 2.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 918 op/s 2024-04-23T16:40:56.243 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:40:56.291 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:56.294 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:55 smithi148 ceph-mon[44520]: pgmap v1462: 27 pgs: 27 active+clean; 2.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 918 op/s 2024-04-23T16:40:56.327 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:40:56.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:56.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:40:56.430 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:40:56.475 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:56.588 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:40:56.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:56.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:40:56.701 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:40:56.764 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:40:56.764 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:40:56.764 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:56.798 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:40:56.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:56.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:40:56.905 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:40:56.905 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:40:56.905 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:40:56.905 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:40:56.905 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:40:56.905 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:40:56.905 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:40:56.905 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:40:56.907 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:40:56.919 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:40:56.919 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:40:56.921 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:40:56.955 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:40:56.972 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:56.991 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:40:57.037 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:40:57.048 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:40:57.048 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:40:57.048 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:40:57.049 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:40:57.049 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:40:57.049 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:40:57.049 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:40:57.049 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:40:57.049 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:40:57.049 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:40:57.049 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:40:57.049 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:40:57.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:56 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1850158160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:40:57.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:57.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:57.268 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:56 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1850158160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:40:57.269 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:57.269 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:57.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:56 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1850158160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:40:57.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:57.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:57.747 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-23T16:40:57.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:57.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:57.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:57.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:57 smithi103 ceph-mon[39629]: pgmap v1463: 27 pgs: 27 active+clean; 2.7 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:40:57.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:57.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:57.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:57.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:57.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:57.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:57.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:57.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:57 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:40:57.990 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:57 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:40:58.336 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:40:58.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:58.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:40:58.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:57 smithi148 ceph-mon[44520]: pgmap v1463: 27 pgs: 27 active+clean; 2.7 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:40:58.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:57 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:40:58.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:57 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:40:58.444 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:40:58.470 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.470 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:57 smithi188 ceph-mon[44101]: pgmap v1463: 27 pgs: 27 active+clean; 2.7 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-23T16:40:58.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:40:58.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:57 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:40:58.471 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:57 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:40:58.472 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:58.497 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:40:58.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:58.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:40:58.601 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:40:58.656 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:58.691 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:40:58.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:40:58.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:40:58.795 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:40:58.834 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:40:59.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:40:58 smithi103 ceph-mon[39629]: osdmap e523: 12 total, 12 up, 10 in 2024-04-23T16:40:59.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:40:58 smithi148 ceph-mon[44520]: osdmap e523: 12 total, 12 up, 10 in 2024-04-23T16:40:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:40:58 smithi188 ceph-mon[44101]: osdmap e523: 12 total, 12 up, 10 in 2024-04-23T16:41:00.012 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:00 smithi103 ceph-mon[39629]: pgmap v1465: 27 pgs: 27 active+clean; 2.6 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-23T16:41:00.012 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:00 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:41:00.012 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:00 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:41:00.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:00 smithi148 ceph-mon[44520]: pgmap v1465: 27 pgs: 27 active+clean; 2.6 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-23T16:41:00.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:00 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:41:00.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:00 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:41:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:00 smithi188 ceph-mon[44101]: pgmap v1465: 27 pgs: 27 active+clean; 2.6 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-23T16:41:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:00 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:41:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:00 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:41:01.022 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:01 smithi103 ceph-mon[39629]: osdmap e524: 12 total, 12 up, 10 in 2024-04-23T16:41:01.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:01 smithi148 ceph-mon[44520]: osdmap e524: 12 total, 12 up, 10 in 2024-04-23T16:41:01.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:01 smithi188 ceph-mon[44101]: osdmap e524: 12 total, 12 up, 10 in 2024-04-23T16:41:01.845 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:41:02.036 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:02 smithi103 ceph-mon[39629]: pgmap v1467: 27 pgs: 27 active+clean; 2.5 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-23T16:41:02.087 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:41:02.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:02.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:41:02.194 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:41:02.237 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:02.284 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:41:02.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:02.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:41:02.293 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:02 smithi148 ceph-mon[44520]: pgmap v1467: 27 pgs: 27 active+clean; 2.5 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-23T16:41:02.392 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:41:02.455 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:02.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:02 smithi188 ceph-mon[44101]: pgmap v1467: 27 pgs: 27 active+clean; 2.5 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-23T16:41:02.483 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:41:02.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:02.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:41:02.594 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:41:02.638 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:02.657 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:41:02.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:02.664 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:41:02.765 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:41:02.800 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:03.054 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:41:03.078 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:03 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/542472898' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:41:03.078 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:03 smithi103 ceph-mon[39629]: pgmap v1468: 27 pgs: 27 active+clean; 2.3 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-23T16:41:03.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:03 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/542472898' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:41:03.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:03 smithi148 ceph-mon[44520]: pgmap v1468: 27 pgs: 27 active+clean; 2.3 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-23T16:41:03.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:03 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/542472898' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:41:03.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:03 smithi188 ceph-mon[44101]: pgmap v1468: 27 pgs: 27 active+clean; 2.3 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-23T16:41:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:04 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:41:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:04 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/542472898' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:41:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:04 smithi103 ceph-mon[39629]: osdmap e525: 12 total, 12 up, 10 in 2024-04-23T16:41:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:04 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:41:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:04 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/542472898' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:41:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:04 smithi148 ceph-mon[44520]: osdmap e525: 12 total, 12 up, 10 in 2024-04-23T16:41:04.455 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:41:04.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:04.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:41:04.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:04 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:41:04.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:04 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/542472898' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:41:04.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:04 smithi188 ceph-mon[44101]: osdmap e525: 12 total, 12 up, 10 in 2024-04-23T16:41:04.560 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:41:04.600 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:04.667 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:41:04.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:04.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:41:04.776 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:41:04.805 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:04.827 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:41:04.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:04.830 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:41:04.931 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:41:04.958 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:04.981 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:41:04.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:04.986 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:41:05.087 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:41:05.145 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:05.188 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:41:05.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:05.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:41:05.246 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:41:05.294 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:41:05.358 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:05.411 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:41:05.417 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:05 smithi148 ceph-mon[44520]: pgmap v1470: 27 pgs: 27 active+clean; 2.2 GiB data, 9.5 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-23T16:41:05.418 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:05 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:41:05.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:05.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:41:05.524 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:41:05.581 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:05.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:05 smithi103 ceph-mon[39629]: pgmap v1470: 27 pgs: 27 active+clean; 2.2 GiB data, 9.5 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-23T16:41:05.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:05 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:41:05.618 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:41:05.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:05.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:41:05.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:05 smithi188 ceph-mon[44101]: pgmap v1470: 27 pgs: 27 active+clean; 2.2 GiB data, 9.5 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-23T16:41:05.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:05 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:41:05.725 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:41:05.782 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:05.824 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:41:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:05.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:41:05.939 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:41:05.966 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:05.987 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:41:05.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:05.990 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:41:06.091 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:41:06.119 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:06.144 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:41:06.148 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:41:06.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:06.149 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:41:06.250 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:41:06.310 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:06.321 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.11 to 1 (10000) 2024-04-23T16:41:06.355 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:41:06.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:06.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:41:06.466 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:41:06.517 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:06 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:41:06.517 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:06 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:41:06.517 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:06 smithi103 ceph-mon[39629]: osdmap e526: 12 total, 12 up, 10 in 2024-04-23T16:41:06.517 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:06 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2686791913' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:41:06.517 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:06 smithi103 ceph-mon[39629]: from='client.26485 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:41:06.517 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:06 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:41:06.517 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:06 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:41:06.521 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:06.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:06 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:41:06.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:06 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:41:06.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:06 smithi148 ceph-mon[44520]: osdmap e526: 12 total, 12 up, 10 in 2024-04-23T16:41:06.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:06 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2686791913' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:41:06.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:06 smithi148 ceph-mon[44520]: from='client.26485 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:41:06.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:06 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:41:06.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:06 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:41:06.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:06 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:41:06.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:06 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:41:06.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:06 smithi188 ceph-mon[44101]: osdmap e526: 12 total, 12 up, 10 in 2024-04-23T16:41:06.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:06 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2686791913' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:41:06.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:06 smithi188 ceph-mon[44101]: from='client.26485 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:41:06.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:06 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:41:06.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:06 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:41:07.326 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:07 smithi103 ceph-mon[39629]: from='client.26485 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:41:07.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:07 smithi103 ceph-mon[39629]: osdmap e527: 12 total, 12 up, 10 in 2024-04-23T16:41:07.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:07 smithi103 ceph-mon[39629]: pgmap v1473: 27 pgs: 27 active+clean; 2.0 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-23T16:41:07.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:41:07.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:41:07.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:07 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:41:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:07 smithi148 ceph-mon[44520]: from='client.26485 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:41:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:07 smithi148 ceph-mon[44520]: osdmap e527: 12 total, 12 up, 10 in 2024-04-23T16:41:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:07 smithi148 ceph-mon[44520]: pgmap v1473: 27 pgs: 27 active+clean; 2.0 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-23T16:41:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:07 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:41:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:07 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:41:07.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:07 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:41:07.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:07 smithi188 ceph-mon[44101]: from='client.26485 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:41:07.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:07 smithi188 ceph-mon[44101]: osdmap e527: 12 total, 12 up, 10 in 2024-04-23T16:41:07.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:07 smithi188 ceph-mon[44101]: pgmap v1473: 27 pgs: 27 active+clean; 2.0 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-23T16:41:07.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:41:07.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:41:07.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:07 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:41:07.798 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:41:07.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:07.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:41:07.909 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:41:07.987 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:08.037 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:41:08.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:08.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:41:08.149 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:41:08.176 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:41:08.189 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:08.213 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:41:08.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:08.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:41:08.320 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:41:08.373 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:08.398 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:41:08.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:08.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:41:08.411 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-23T16:41:08.506 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:41:08.511 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:08 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:41:08.511 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:08 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:41:08.554 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:08.593 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:41:08.596 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:08 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:41:08.596 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:08 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:41:08.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:08.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:41:08.697 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:41:08.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:08 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:41:08.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:08 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:41:08.730 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:08.940 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:41:08.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:08.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:41:09.048 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:41:09.087 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:09.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:09 smithi148 ceph-mon[44520]: osdmap e528: 12 total, 12 up, 10 in 2024-04-23T16:41:09.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:09 smithi148 ceph-mon[44520]: pgmap v1475: 27 pgs: 27 active+clean; 2.0 GiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 986 op/s 2024-04-23T16:41:09.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:09 smithi188 ceph-mon[44101]: osdmap e528: 12 total, 12 up, 10 in 2024-04-23T16:41:09.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:09 smithi188 ceph-mon[44101]: pgmap v1475: 27 pgs: 27 active+clean; 2.0 GiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 986 op/s 2024-04-23T16:41:09.854 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:09 smithi103 ceph-mon[39629]: osdmap e528: 12 total, 12 up, 10 in 2024-04-23T16:41:09.854 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:09 smithi103 ceph-mon[39629]: pgmap v1475: 27 pgs: 27 active+clean; 2.0 GiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 986 op/s 2024-04-23T16:41:09.943 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:41:09.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:09.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:41:09.977 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:41:09.977 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:41:10.057 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:41:10.084 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:10.106 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:41:10.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:10.109 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:41:10.210 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:41:10.266 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:10.310 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:41:10.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:10.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:41:10.421 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:41:10.464 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:10.503 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:41:10.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:10.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:41:10.607 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:41:10.641 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:10.799 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:41:10.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:10.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:41:10.903 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:41:10.930 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:10.950 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:41:10.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:10.954 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:41:11.055 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:41:11.093 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:12.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:11 smithi188 ceph-mon[44101]: pgmap v1476: 27 pgs: 27 active+clean; 1.8 GiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-23T16:41:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:11 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1694100922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:41:12.298 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:11 smithi103 ceph-mon[39629]: pgmap v1476: 27 pgs: 27 active+clean; 1.8 GiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-23T16:41:12.298 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:11 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1694100922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:41:12.329 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:41:12.329 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:41:12.330 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:41:12.330 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:41:12.330 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:41:12.330 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:41:12.330 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:41:12.330 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:41:12.330 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:41:12.330 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:41:12.330 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:41:12.330 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:41:12.330 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:41:12.331 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:41:12.331 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:41:12.331 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:41:12.331 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:41:12.331 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:41:12.331 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:41:12.331 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:41:12.331 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:41:12.331 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:41:12.331 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:41:12.331 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:41:12.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:11 smithi148 ceph-mon[44520]: pgmap v1476: 27 pgs: 27 active+clean; 1.8 GiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-23T16:41:12.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:11 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1694100922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:41:13.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:12 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:12 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.326 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:41:13.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:13.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:41:13.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:12 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:13.431 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:41:13.467 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:14.134 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:41:14.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:14.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:41:14.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:13 smithi188 ceph-mon[44101]: pgmap v1477: 27 pgs: 27 active+clean; 1.7 GiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:41:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10967137641766242]}]: dispatch 2024-04-23T16:41:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10967137641766242]}]: dispatch 2024-04-23T16:41:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.0872955322265625]}]: dispatch 2024-04-23T16:41:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09462550240506312]}]: dispatch 2024-04-23T16:41:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.12046780206160382]}]: dispatch 2024-04-23T16:41:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09949522549054476]}]: dispatch 2024-04-23T16:41:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.09462550240506312]}]: dispatch 2024-04-23T16:41:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.09668802856976086]}]: dispatch 2024-04-23T16:41:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.11365602416564116]}]: dispatch 2024-04-23T16:41:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.09897417236808581]}]: dispatch 2024-04-23T16:41:14.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0872955322265625]}]: dispatch 2024-04-23T16:41:14.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:13 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.10967137641766242]}]: dispatch 2024-04-23T16:41:14.240 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:41:14.274 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:14.275 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:14.275 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:14.275 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:13 smithi103 ceph-mon[39629]: pgmap v1477: 27 pgs: 27 active+clean; 1.7 GiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:41:14.275 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10967137641766242]}]: dispatch 2024-04-23T16:41:14.275 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10967137641766242]}]: dispatch 2024-04-23T16:41:14.275 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.0872955322265625]}]: dispatch 2024-04-23T16:41:14.275 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09462550240506312]}]: dispatch 2024-04-23T16:41:14.275 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.12046780206160382]}]: dispatch 2024-04-23T16:41:14.275 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09949522549054476]}]: dispatch 2024-04-23T16:41:14.275 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.09462550240506312]}]: dispatch 2024-04-23T16:41:14.275 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.09668802856976086]}]: dispatch 2024-04-23T16:41:14.275 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.11365602416564116]}]: dispatch 2024-04-23T16:41:14.275 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.09897417236808581]}]: dispatch 2024-04-23T16:41:14.275 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0872955322265625]}]: dispatch 2024-04-23T16:41:14.276 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:13 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.10967137641766242]}]: dispatch 2024-04-23T16:41:14.276 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:41:14.276 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:41:14.276 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:14.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:14.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:14.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:14.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:13 smithi148 ceph-mon[44520]: pgmap v1477: 27 pgs: 27 active+clean; 1.7 GiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-23T16:41:14.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10967137641766242]}]: dispatch 2024-04-23T16:41:14.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10967137641766242]}]: dispatch 2024-04-23T16:41:14.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.0872955322265625]}]: dispatch 2024-04-23T16:41:14.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09462550240506312]}]: dispatch 2024-04-23T16:41:14.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.12046780206160382]}]: dispatch 2024-04-23T16:41:14.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09949522549054476]}]: dispatch 2024-04-23T16:41:14.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.09462550240506312]}]: dispatch 2024-04-23T16:41:14.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.09668802856976086]}]: dispatch 2024-04-23T16:41:14.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.11365602416564116]}]: dispatch 2024-04-23T16:41:14.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.09897417236808581]}]: dispatch 2024-04-23T16:41:14.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0872955322265625]}]: dispatch 2024-04-23T16:41:14.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:13 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.10967137641766242]}]: dispatch 2024-04-23T16:41:14.487 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:41:14.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:14.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:41:14.596 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:41:14.647 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:14.682 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:41:14.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:14.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:41:14.786 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:41:14.814 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:14.835 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:41:14.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:14.841 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:41:14.927 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10967137641766242]}]': finished 2024-04-23T16:41:14.927 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10967137641766242]}]': finished 2024-04-23T16:41:14.927 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.0872955322265625]}]': finished 2024-04-23T16:41:14.927 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09462550240506312]}]': finished 2024-04-23T16:41:14.928 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.12046780206160382]}]': finished 2024-04-23T16:41:14.928 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09949522549054476]}]': finished 2024-04-23T16:41:14.928 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.09462550240506312]}]': finished 2024-04-23T16:41:14.928 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.09668802856976086]}]': finished 2024-04-23T16:41:14.928 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.11365602416564116]}]': finished 2024-04-23T16:41:14.928 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.09897417236808581]}]': finished 2024-04-23T16:41:14.928 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0872955322265625]}]': finished 2024-04-23T16:41:14.928 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:14 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.10967137641766242]}]': finished 2024-04-23T16:41:14.928 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:14 smithi103 ceph-mon[39629]: osdmap e529: 12 total, 12 up, 10 in 2024-04-23T16:41:14.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10967137641766242]}]': finished 2024-04-23T16:41:14.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10967137641766242]}]': finished 2024-04-23T16:41:14.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.0872955322265625]}]': finished 2024-04-23T16:41:14.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09462550240506312]}]': finished 2024-04-23T16:41:14.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.12046780206160382]}]': finished 2024-04-23T16:41:14.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09949522549054476]}]': finished 2024-04-23T16:41:14.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.09462550240506312]}]': finished 2024-04-23T16:41:14.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.09668802856976086]}]': finished 2024-04-23T16:41:14.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.11365602416564116]}]': finished 2024-04-23T16:41:14.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.09897417236808581]}]': finished 2024-04-23T16:41:14.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0872955322265625]}]': finished 2024-04-23T16:41:14.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:14 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.10967137641766242]}]': finished 2024-04-23T16:41:14.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:14 smithi148 ceph-mon[44520]: osdmap e529: 12 total, 12 up, 10 in 2024-04-23T16:41:14.942 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:41:14.971 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:14.992 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:41:14.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:14.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:41:15.036 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-23T16:41:15.097 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:41:15.137 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:15.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10967137641766242]}]': finished 2024-04-23T16:41:15.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10967137641766242]}]': finished 2024-04-23T16:41:15.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.0872955322265625]}]': finished 2024-04-23T16:41:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09462550240506312]}]': finished 2024-04-23T16:41:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.12046780206160382]}]': finished 2024-04-23T16:41:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09949522549054476]}]': finished 2024-04-23T16:41:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.09462550240506312]}]': finished 2024-04-23T16:41:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.09668802856976086]}]': finished 2024-04-23T16:41:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.11365602416564116]}]': finished 2024-04-23T16:41:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.09897417236808581]}]': finished 2024-04-23T16:41:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0872955322265625]}]': finished 2024-04-23T16:41:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:14 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.10967137641766242]}]': finished 2024-04-23T16:41:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:14 smithi188 ceph-mon[44101]: osdmap e529: 12 total, 12 up, 10 in 2024-04-23T16:41:15.518 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:41:15.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:15.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:41:15.626 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:41:15.654 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:15.674 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:41:15.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:15.677 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:41:15.778 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:41:15.829 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:15.884 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:41:15.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:15.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:41:15.989 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:41:16.023 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:16.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:15 smithi148 ceph-mon[44520]: pgmap v1479: 27 pgs: 27 active+clean; 1.6 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 902 op/s 2024-04-23T16:41:16.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:15 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:41:16.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:15 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:41:16.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:15 smithi148 ceph-mon[44520]: osdmap e530: 12 total, 12 up, 10 in 2024-04-23T16:41:16.151 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:41:16.179 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:15 smithi103 ceph-mon[39629]: pgmap v1479: 27 pgs: 27 active+clean; 1.6 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 902 op/s 2024-04-23T16:41:16.179 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:15 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:41:16.179 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:15 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:41:16.179 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:15 smithi103 ceph-mon[39629]: osdmap e530: 12 total, 12 up, 10 in 2024-04-23T16:41:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:15 smithi188 ceph-mon[44101]: pgmap v1479: 27 pgs: 27 active+clean; 1.6 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 902 op/s 2024-04-23T16:41:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:15 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:41:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:15 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:41:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:15 smithi188 ceph-mon[44101]: osdmap e530: 12 total, 12 up, 10 in 2024-04-23T16:41:16.477 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:41:16.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:16.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:41:16.590 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:41:16.635 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:16.677 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:41:16.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:16.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:41:16.788 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:41:16.814 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:16.835 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:41:16.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:16.838 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:41:16.939 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:41:16.980 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:17.084 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:41:17.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:17.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:41:17.191 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:41:17.234 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:17.258 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:41:17.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:17.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:41:17.363 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:41:17.407 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:17.457 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:41:17.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:17.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:41:17.563 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:41:17.610 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:17.639 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:41:17.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:17.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:41:17.656 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:41:17.665 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:41:17.743 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:41:17.769 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:17.789 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:41:17.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:17.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:41:17.893 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:41:17.920 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:17.940 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:41:17.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:17.943 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:41:17.984 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.1 to 1 (10000) 2024-04-23T16:41:18.044 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:41:18.077 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:18.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:17 smithi188 ceph-mon[44101]: pgmap v1481: 27 pgs: 27 active+clean; 1.3 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-23T16:41:18.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:17 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2921831952' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:41:18.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:17 smithi188 ceph-mon[44101]: from='client.26497 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:41:18.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:17 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:41:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:17 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:41:18.245 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:17 smithi103 ceph-mon[39629]: pgmap v1481: 27 pgs: 27 active+clean; 1.3 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-23T16:41:18.245 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:17 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2921831952' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:41:18.245 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:17 smithi103 ceph-mon[39629]: from='client.26497 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:41:18.245 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:17 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:41:18.245 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:17 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:41:18.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:17 smithi148 ceph-mon[44520]: pgmap v1481: 27 pgs: 27 active+clean; 1.3 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-23T16:41:18.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:17 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2921831952' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:41:18.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:17 smithi148 ceph-mon[44520]: from='client.26497 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:41:18.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:17 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:41:18.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:17 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:41:18.744 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:41:18.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:18.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:41:18.849 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:41:18.892 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:18.986 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:41:19.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:18 smithi188 ceph-mon[44101]: from='client.26497 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-23T16:41:19.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:18 smithi188 ceph-mon[44101]: osdmap e531: 12 total, 12 up, 10 in 2024-04-23T16:41:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:18 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2899037605' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:41:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:18 smithi188 ceph-mon[44101]: from='client.36137 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:41:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:18 smithi103 ceph-mon[39629]: from='client.26497 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-23T16:41:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:18 smithi103 ceph-mon[39629]: osdmap e531: 12 total, 12 up, 10 in 2024-04-23T16:41:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:18 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2899037605' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:41:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:18 smithi103 ceph-mon[39629]: from='client.36137 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:41:19.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:18 smithi148 ceph-mon[44520]: from='client.26497 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-23T16:41:19.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:18 smithi148 ceph-mon[44520]: osdmap e531: 12 total, 12 up, 10 in 2024-04-23T16:41:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:18 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2899037605' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:41:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:18 smithi148 ceph-mon[44520]: from='client.36137 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:41:20.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:19 smithi103 ceph-mon[39629]: pgmap v1483: 27 pgs: 27 active+clean; 1.3 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 992 op/s 2024-04-23T16:41:20.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:19 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:41:20.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:19 smithi103 ceph-mon[39629]: from='client.36137 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:41:20.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:19 smithi103 ceph-mon[39629]: osdmap e532: 12 total, 12 up, 10 in 2024-04-23T16:41:20.366 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:41:20.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:20.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:41:20.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:19 smithi148 ceph-mon[44520]: pgmap v1483: 27 pgs: 27 active+clean; 1.3 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 992 op/s 2024-04-23T16:41:20.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:19 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:41:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:19 smithi148 ceph-mon[44520]: from='client.36137 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:41:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:19 smithi148 ceph-mon[44520]: osdmap e532: 12 total, 12 up, 10 in 2024-04-23T16:41:20.472 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:41:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:19 smithi188 ceph-mon[44101]: pgmap v1483: 27 pgs: 27 active+clean; 1.3 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 992 op/s 2024-04-23T16:41:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:19 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:41:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:19 smithi188 ceph-mon[44101]: from='client.36137 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:41:20.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:19 smithi188 ceph-mon[44101]: osdmap e532: 12 total, 12 up, 10 in 2024-04-23T16:41:20.501 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:20.525 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:41:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:20.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:41:20.631 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:41:20.679 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:20.982 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:20 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:41:20.982 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:20 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:41:20.982 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:20 smithi103 ceph-mon[39629]: osdmap e533: 12 total, 12 up, 10 in 2024-04-23T16:41:21.240 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:41:21.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:21.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:41:21.344 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:41:21.378 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:21.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:20 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:41:21.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:20 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:41:21.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:20 smithi148 ceph-mon[44520]: osdmap e533: 12 total, 12 up, 10 in 2024-04-23T16:41:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:20 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:41:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:20 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:41:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:20 smithi188 ceph-mon[44101]: osdmap e533: 12 total, 12 up, 10 in 2024-04-23T16:41:22.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:21 smithi103 ceph-mon[39629]: pgmap v1486: 27 pgs: 27 active+clean; 1.2 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:41:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:21 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:41:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:21 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:41:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:21 smithi103 ceph-mon[39629]: 1.a scrub starts 2024-04-23T16:41:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:21 smithi103 ceph-mon[39629]: osdmap e534: 12 total, 12 up, 10 in 2024-04-23T16:41:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:21 smithi103 ceph-mon[39629]: 1.a scrub ok 2024-04-23T16:41:22.237 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:41:22.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:22.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:41:22.344 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:41:22.369 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:21 smithi188 ceph-mon[44101]: pgmap v1486: 27 pgs: 27 active+clean; 1.2 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:41:22.369 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:21 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:41:22.369 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:21 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:41:22.369 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:21 smithi188 ceph-mon[44101]: 1.a scrub starts 2024-04-23T16:41:22.369 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:21 smithi188 ceph-mon[44101]: osdmap e534: 12 total, 12 up, 10 in 2024-04-23T16:41:22.369 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:21 smithi188 ceph-mon[44101]: 1.a scrub ok 2024-04-23T16:41:22.371 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:22.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:21 smithi148 ceph-mon[44520]: pgmap v1486: 27 pgs: 27 active+clean; 1.2 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:41:22.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:21 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:41:22.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:21 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:41:22.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:21 smithi148 ceph-mon[44520]: 1.a scrub starts 2024-04-23T16:41:22.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:21 smithi148 ceph-mon[44520]: osdmap e534: 12 total, 12 up, 10 in 2024-04-23T16:41:22.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:21 smithi148 ceph-mon[44520]: 1.a scrub ok 2024-04-23T16:41:22.390 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:41:22.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:22.393 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:41:22.495 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:41:22.544 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:22.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:22.590 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:41:22.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:41:22.691 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:41:22.720 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:22.740 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:41:22.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:22.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:41:22.843 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:41:22.883 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:23.343 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:41:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:23.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:41:23.358 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-04-23T16:41:23.358 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-04-23T16:41:23.358 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-23T16:41:23.358 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T16:41:23.448 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:41:23.477 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:23.498 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:41:23.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:23.506 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:41:23.607 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:41:23.656 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:23.690 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:41:23.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:23.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:41:23.798 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:41:23.836 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:23.977 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:41:23.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:23.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:41:24.082 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:41:24.110 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:24.238 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:41:24.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:24.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:41:24.343 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:41:24.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:23 smithi103 ceph-mon[39629]: pgmap v1488: 27 pgs: 27 active+clean; 1013 MiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-23T16:41:24.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:23 smithi148 ceph-mon[44520]: pgmap v1488: 27 pgs: 27 active+clean; 1013 MiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-23T16:41:24.382 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:24.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:23 smithi188 ceph-mon[44101]: pgmap v1488: 27 pgs: 27 active+clean; 1013 MiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-23T16:41:25.019 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:41:25.020 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:41:25.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:41:25.020 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:41:25.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.020 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(6.d 508/488 [PushOp(6:be026432:::benchmark_data_smithi188_130221_object72841:head, version: 463'4508, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:be026432:::benchmark_data_smithi188_130221_object72841:head@463'4508, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T16:41:25.020 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:40:08.779001+0000", 2024-04-23T16:41:25.020 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 75.246499995999997, 2024-04-23T16:41:25.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.039921781000000003, 2024-04-23T16:41:25.020 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.020 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.020 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:41:25.020 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:08.779001+0000", 2024-04-23T16:41:25.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:08.779001+0000", 2024-04-23T16:41:25.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:08.779001+0000", 2024-04-23T16:41:25.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2899999999999999e-07 2024-04-23T16:41:25.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:08.779031+0000", 2024-04-23T16:41:25.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9411e-05 2024-04-23T16:41:25.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:08.814205+0000", 2024-04-23T16:41:25.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.035173757999999999 2024-04-23T16:41:25.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:08.814233+0000", 2024-04-23T16:41:25.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8303e-05 2024-04-23T16:41:25.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:08.818688+0000", 2024-04-23T16:41:25.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044551260000000002 2024-04-23T16:41:25.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:08.818701+0000", 2024-04-23T16:41:25.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2707e-05 2024-04-23T16:41:25.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:08.818923+0000", 2024-04-23T16:41:25.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00022214700000000001 2024-04-23T16:41:25.024 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.024 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.024 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.024 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:218086 6.4 6:24d2a9ae:::benchmark_data_smithi188_130221_object63625:head [delete] snapc 0=[] ondisk+write+known_if_redirected e508)", 2024-04-23T16:41:25.024 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:40:10.178966+0000", 2024-04-23T16:41:25.024 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 73.846534575000007, 2024-04-23T16:41:25.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.041378409999999997, 2024-04-23T16:41:25.024 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.024 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.025 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:41:25.025 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:41:25.025 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:41:25.025 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:41:25.025 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 218086 2024-04-23T16:41:25.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.025 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.178966+0000", 2024-04-23T16:41:25.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.178966+0000", 2024-04-23T16:41:25.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.178967+0000", 2024-04-23T16:41:25.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8999999999999995e-07 2024-04-23T16:41:25.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.178970+0000", 2024-04-23T16:41:25.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5909999999999998e-06 2024-04-23T16:41:25.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.178976+0000", 2024-04-23T16:41:25.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4609999999999996e-06 2024-04-23T16:41:25.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.178982+0000", 2024-04-23T16:41:25.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6019999999999999e-06 2024-04-23T16:41:25.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.182517+0000", 2024-04-23T16:41:25.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00353543 2024-04-23T16:41:25.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.182635+0000", 2024-04-23T16:41:25.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117945 2024-04-23T16:41:25.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 5", 2024-04-23T16:41:25.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.182743+0000", 2024-04-23T16:41:25.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010779499999999999 2024-04-23T16:41:25.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:41:25.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.188716+0000", 2024-04-23T16:41:25.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0059731999999999997 2024-04-23T16:41:25.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:41:25.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.194526+0000", 2024-04-23T16:41:25.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058094599999999998 2024-04-23T16:41:25.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:41:25.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.194575+0000", 2024-04-23T16:41:25.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.888e-05 2024-04-23T16:41:25.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.220345+0000", 2024-04-23T16:41:25.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025770255999999998 2024-04-23T16:41:25.030 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.030 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.030 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.030 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:218087 6.4 6:2b30b250:::benchmark_data_smithi188_130221_object63626:head [delete] snapc 0=[] ondisk+write+known_if_redirected e508)", 2024-04-23T16:41:25.030 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:40:10.179127+0000", 2024-04-23T16:41:25.030 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 73.846374104999995, 2024-04-23T16:41:25.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045514222999999999, 2024-04-23T16:41:25.030 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.030 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.030 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:41:25.030 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:41:25.031 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:41:25.031 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:41:25.031 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 218087 2024-04-23T16:41:25.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.031 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.179127+0000", 2024-04-23T16:41:25.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.179127+0000", 2024-04-23T16:41:25.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.179128+0000", 2024-04-23T16:41:25.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3600000000000002e-07 2024-04-23T16:41:25.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.179131+0000", 2024-04-23T16:41:25.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9560000000000002e-06 2024-04-23T16:41:25.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.179136+0000", 2024-04-23T16:41:25.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7150000000000003e-06 2024-04-23T16:41:25.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.179141+0000", 2024-04-23T16:41:25.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5329999999999996e-06 2024-04-23T16:41:25.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.186298+0000", 2024-04-23T16:41:25.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071568689999999997 2024-04-23T16:41:25.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.186393+0000", 2024-04-23T16:41:25.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4888000000000004e-05 2024-04-23T16:41:25.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 5", 2024-04-23T16:41:25.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.186466+0000", 2024-04-23T16:41:25.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3379000000000003e-05 2024-04-23T16:41:25.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:41:25.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.188846+0000", 2024-04-23T16:41:25.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023798740000000001 2024-04-23T16:41:25.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:41:25.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.224544+0000", 2024-04-23T16:41:25.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.035698415999999997 2024-04-23T16:41:25.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:41:25.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.224596+0000", 2024-04-23T16:41:25.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2048e-05 2024-04-23T16:41:25.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:10.224641+0000", 2024-04-23T16:41:25.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4709000000000002e-05 2024-04-23T16:41:25.035 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.035 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.035 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.035 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:219374 6.4 6:2a7d594f:::benchmark_data_smithi188_130221_object64913:head [delete] snapc 0=[] ondisk+write+known_if_redirected e508)", 2024-04-23T16:41:25.035 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:40:11.466360+0000", 2024-04-23T16:41:25.036 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 72.559140756000005, 2024-04-23T16:41:25.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.039727881999999999, 2024-04-23T16:41:25.036 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.036 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.036 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:41:25.036 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:41:25.036 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:41:25.036 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:41:25.036 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 219374 2024-04-23T16:41:25.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.036 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.036 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.036 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:11.466360+0000", 2024-04-23T16:41:25.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:11.466360+0000", 2024-04-23T16:41:25.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:11.466363+0000", 2024-04-23T16:41:25.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6730000000000001e-06 2024-04-23T16:41:25.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:11.466369+0000", 2024-04-23T16:41:25.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9619999999999998e-06 2024-04-23T16:41:25.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:11.466380+0000", 2024-04-23T16:41:25.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1326e-05 2024-04-23T16:41:25.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:11.466388+0000", 2024-04-23T16:41:25.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.729e-06 2024-04-23T16:41:25.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:11.466438+0000", 2024-04-23T16:41:25.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9682999999999998e-05 2024-04-23T16:41:25.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:11.466524+0000", 2024-04-23T16:41:25.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6489999999999994e-05 2024-04-23T16:41:25.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 5", 2024-04-23T16:41:25.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:11.466593+0000", 2024-04-23T16:41:25.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8929000000000004e-05 2024-04-23T16:41:25.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:41:25.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:11.469248+0000", 2024-04-23T16:41:25.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026552210000000001 2024-04-23T16:41:25.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:41:25.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:11.505986+0000", 2024-04-23T16:41:25.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.036737977999999998 2024-04-23T16:41:25.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:41:25.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:11.506039+0000", 2024-04-23T16:41:25.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2587000000000003e-05 2024-04-23T16:41:25.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:11.506088+0000", 2024-04-23T16:41:25.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9304000000000001e-05 2024-04-23T16:41:25.041 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.041 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.041 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.041 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:232631 6.4 6:2565f9c8:::benchmark_data_smithi188_130221_object78170:head [delete] snapc 0=[] ondisk+write+known_if_redirected e516)", 2024-04-23T16:41:25.041 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:40:25.487273+0000", 2024-04-23T16:41:25.041 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 58.538227966000001, 2024-04-23T16:41:25.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.041809200999999997, 2024-04-23T16:41:25.042 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.042 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.042 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:41:25.042 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:41:25.042 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:41:25.042 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:41:25.042 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 232631 2024-04-23T16:41:25.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.042 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.042 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.042 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.487273+0000", 2024-04-23T16:41:25.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.487273+0000", 2024-04-23T16:41:25.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.487275+0000", 2024-04-23T16:41:25.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.835e-06 2024-04-23T16:41:25.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.487280+0000", 2024-04-23T16:41:25.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2469999999999999e-06 2024-04-23T16:41:25.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.044 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.044 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.487290+0000", 2024-04-23T16:41:25.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0101e-05 2024-04-23T16:41:25.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.044 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.044 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.487296+0000", 2024-04-23T16:41:25.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9020000000000004e-06 2024-04-23T16:41:25.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.044 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.044 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.490267+0000", 2024-04-23T16:41:25.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0029713040000000001 2024-04-23T16:41:25.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.490385+0000", 2024-04-23T16:41:25.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117562 2024-04-23T16:41:25.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 5", 2024-04-23T16:41:25.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.490503+0000", 2024-04-23T16:41:25.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118317 2024-04-23T16:41:25.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:41:25.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.497645+0000", 2024-04-23T16:41:25.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071420570000000003 2024-04-23T16:41:25.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:41:25.046 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.528978+0000", 2024-04-23T16:41:25.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.031332896999999998 2024-04-23T16:41:25.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:41:25.046 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.529037+0000", 2024-04-23T16:41:25.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8272e-05 2024-04-23T16:41:25.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.047 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.529082+0000", 2024-04-23T16:41:25.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5707000000000001e-05 2024-04-23T16:41:25.047 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.047 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.047 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.047 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:232638 6.4 6:24847e63:::benchmark_data_smithi188_130221_object78177:head [delete] snapc 0=[] ondisk+write+known_if_redirected e516)", 2024-04-23T16:41:25.047 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:40:25.488316+0000", 2024-04-23T16:41:25.047 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 58.537184668999998, 2024-04-23T16:41:25.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.04089367, 2024-04-23T16:41:25.047 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.047 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.047 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:41:25.048 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:41:25.048 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:41:25.048 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:41:25.048 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 232638 2024-04-23T16:41:25.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.048 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.488316+0000", 2024-04-23T16:41:25.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.488316+0000", 2024-04-23T16:41:25.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.488318+0000", 2024-04-23T16:41:25.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5880000000000001e-06 2024-04-23T16:41:25.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.488331+0000", 2024-04-23T16:41:25.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3021e-05 2024-04-23T16:41:25.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.050 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.488346+0000", 2024-04-23T16:41:25.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5199e-05 2024-04-23T16:41:25.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.050 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.488354+0000", 2024-04-23T16:41:25.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0409999999999998e-06 2024-04-23T16:41:25.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.494188+0000", 2024-04-23T16:41:25.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058335390000000004 2024-04-23T16:41:25.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.494265+0000", 2024-04-23T16:41:25.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7600000000000002e-05 2024-04-23T16:41:25.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 5", 2024-04-23T16:41:25.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.494378+0000", 2024-04-23T16:41:25.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000113056 2024-04-23T16:41:25.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.052 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:41:25.052 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.497797+0000", 2024-04-23T16:41:25.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034190599999999998 2024-04-23T16:41:25.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.052 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:41:25.052 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.529128+0000", 2024-04-23T16:41:25.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.031330961999999997 2024-04-23T16:41:25.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.052 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:41:25.053 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.529174+0000", 2024-04-23T16:41:25.053 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5674999999999999e-05 2024-04-23T16:41:25.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.053 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.053 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:25.529210+0000", 2024-04-23T16:41:25.053 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5929000000000001e-05 2024-04-23T16:41:25.053 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.053 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.053 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.053 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:234619 6.4 6:280e4ab7:::benchmark_data_smithi188_130221_object80158:head [delete] snapc 0=[] ondisk+write+known_if_redirected e517)", 2024-04-23T16:41:25.053 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:40:27.453417+0000", 2024-04-23T16:41:25.053 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 56.572084101999998, 2024-04-23T16:41:25.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.051152977000000002, 2024-04-23T16:41:25.054 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.054 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.054 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:41:25.054 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:41:25.054 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:41:25.054 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:41:25.054 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 234619 2024-04-23T16:41:25.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.054 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.453417+0000", 2024-04-23T16:41:25.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.453417+0000", 2024-04-23T16:41:25.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.453419+0000", 2024-04-23T16:41:25.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0190000000000001e-06 2024-04-23T16:41:25.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.453425+0000", 2024-04-23T16:41:25.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1700000000000002e-06 2024-04-23T16:41:25.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.453436+0000", 2024-04-23T16:41:25.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1061e-05 2024-04-23T16:41:25.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.453444+0000", 2024-04-23T16:41:25.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3919999999999997e-06 2024-04-23T16:41:25.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.453562+0000", 2024-04-23T16:41:25.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118329 2024-04-23T16:41:25.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.453683+0000", 2024-04-23T16:41:25.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000121438 2024-04-23T16:41:25.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 5", 2024-04-23T16:41:25.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.453791+0000", 2024-04-23T16:41:25.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107642 2024-04-23T16:41:25.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:41:25.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.456577+0000", 2024-04-23T16:41:25.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027864669999999999 2024-04-23T16:41:25.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:41:25.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.504478+0000", 2024-04-23T16:41:25.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047900814 2024-04-23T16:41:25.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:41:25.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.504529+0000", 2024-04-23T16:41:25.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0584000000000002e-05 2024-04-23T16:41:25.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.504570+0000", 2024-04-23T16:41:25.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1060999999999999e-05 2024-04-23T16:41:25.059 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.059 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.059 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.059 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:234626 6.4 6:2906facc:::benchmark_data_smithi188_130221_object80165:head [delete] snapc 0=[] ondisk+write+known_if_redirected e517)", 2024-04-23T16:41:25.059 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:40:27.463427+0000", 2024-04-23T16:41:25.059 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 56.562074228, 2024-04-23T16:41:25.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.041256497000000003, 2024-04-23T16:41:25.060 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.060 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.060 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:41:25.060 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:41:25.060 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:41:25.060 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:41:25.060 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 234626 2024-04-23T16:41:25.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.060 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.463427+0000", 2024-04-23T16:41:25.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.463427+0000", 2024-04-23T16:41:25.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.463429+0000", 2024-04-23T16:41:25.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.136e-06 2024-04-23T16:41:25.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.463434+0000", 2024-04-23T16:41:25.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5589999999999996e-06 2024-04-23T16:41:25.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.463446+0000", 2024-04-23T16:41:25.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1144999999999999e-05 2024-04-23T16:41:25.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.463464+0000", 2024-04-23T16:41:25.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8151e-05 2024-04-23T16:41:25.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.463518+0000", 2024-04-23T16:41:25.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3863000000000003e-05 2024-04-23T16:41:25.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.463627+0000", 2024-04-23T16:41:25.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109159 2024-04-23T16:41:25.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 5", 2024-04-23T16:41:25.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.463731+0000", 2024-04-23T16:41:25.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103847 2024-04-23T16:41:25.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:41:25.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.467287+0000", 2024-04-23T16:41:25.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035564490000000002 2024-04-23T16:41:25.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.065 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:41:25.065 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.504608+0000", 2024-04-23T16:41:25.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.037321355000000001 2024-04-23T16:41:25.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.065 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:41:25.065 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.504650+0000", 2024-04-23T16:41:25.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2058000000000003e-05 2024-04-23T16:41:25.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.066 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.504683+0000", 2024-04-23T16:41:25.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2774999999999997e-05 2024-04-23T16:41:25.066 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.066 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.066 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.066 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:234635 6.4 6:25bd9020:::benchmark_data_smithi188_130221_object80174:head [delete] snapc 0=[] ondisk+write+known_if_redirected e517)", 2024-04-23T16:41:25.066 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:40:27.472800+0000", 2024-04-23T16:41:25.067 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 56.552700838, 2024-04-23T16:41:25.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.042526516, 2024-04-23T16:41:25.067 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.067 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.067 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:41:25.067 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:41:25.067 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:41:25.067 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:41:25.067 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 234635 2024-04-23T16:41:25.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.067 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.472800+0000", 2024-04-23T16:41:25.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.472800+0000", 2024-04-23T16:41:25.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.472802+0000", 2024-04-23T16:41:25.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3829999999999999e-06 2024-04-23T16:41:25.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.472806+0000", 2024-04-23T16:41:25.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1119999999999998e-06 2024-04-23T16:41:25.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.505299+0000", 2024-04-23T16:41:25.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.032493693999999997 2024-04-23T16:41:25.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.505309+0000", 2024-04-23T16:41:25.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7149999999999993e-06 2024-04-23T16:41:25.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.506706+0000", 2024-04-23T16:41:25.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001396607 2024-04-23T16:41:25.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.506812+0000", 2024-04-23T16:41:25.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010610700000000001 2024-04-23T16:41:25.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 5", 2024-04-23T16:41:25.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.506911+0000", 2024-04-23T16:41:25.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9448000000000001e-05 2024-04-23T16:41:25.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:41:25.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.513568+0000", 2024-04-23T16:41:25.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066565979999999997 2024-04-23T16:41:25.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:41:25.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.515229+0000", 2024-04-23T16:41:25.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016607449999999999 2024-04-23T16:41:25.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:41:25.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.515281+0000", 2024-04-23T16:41:25.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2049999999999998e-05 2024-04-23T16:41:25.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:27.515327+0000", 2024-04-23T16:41:25.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6057000000000003e-05 2024-04-23T16:41:25.073 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.073 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.073 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.074 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:262194 6.4 6:23aa18e1:::benchmark_data_smithi188_130221_object107733:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-23T16:41:25.074 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:40:52.563900+0000", 2024-04-23T16:41:25.074 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.461601124000001, 2024-04-23T16:41:25.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.064303605, 2024-04-23T16:41:25.074 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.074 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.074 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:41:25.074 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:41:25.074 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:41:25.075 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:41:25.075 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 262194 2024-04-23T16:41:25.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.075 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.563900+0000", 2024-04-23T16:41:25.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.563900+0000", 2024-04-23T16:41:25.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.563902+0000", 2024-04-23T16:41:25.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0080000000000001e-06 2024-04-23T16:41:25.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.563908+0000", 2024-04-23T16:41:25.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5520000000000001e-06 2024-04-23T16:41:25.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.563919+0000", 2024-04-23T16:41:25.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0393e-05 2024-04-23T16:41:25.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.563926+0000", 2024-04-23T16:41:25.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.447e-06 2024-04-23T16:41:25.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.563981+0000", 2024-04-23T16:41:25.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4870000000000002e-05 2024-04-23T16:41:25.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.564073+0000", 2024-04-23T16:41:25.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1667000000000002e-05 2024-04-23T16:41:25.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 5", 2024-04-23T16:41:25.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.564143+0000", 2024-04-23T16:41:25.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9870999999999996e-05 2024-04-23T16:41:25.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:41:25.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.571245+0000", 2024-04-23T16:41:25.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071025929999999999 2024-04-23T16:41:25.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:41:25.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.580100+0000", 2024-04-23T16:41:25.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088549900000000001 2024-04-23T16:41:25.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:41:25.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628161+0000", 2024-04-23T16:41:25.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.048061039 2024-04-23T16:41:25.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628203+0000", 2024-04-23T16:41:25.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2175000000000002e-05 2024-04-23T16:41:25.080 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.080 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.080 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.080 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:262201 6.4 6:27eefc91:::benchmark_data_smithi188_130221_object107740:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-23T16:41:25.080 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:40:52.564624+0000", 2024-04-23T16:41:25.080 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.46087691, 2024-04-23T16:41:25.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.063732075999999999, 2024-04-23T16:41:25.081 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.081 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.081 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:41:25.081 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:41:25.081 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:41:25.081 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:41:25.081 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 262201 2024-04-23T16:41:25.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.081 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.564624+0000", 2024-04-23T16:41:25.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.564624+0000", 2024-04-23T16:41:25.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.564626+0000", 2024-04-23T16:41:25.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.646e-06 2024-04-23T16:41:25.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.564631+0000", 2024-04-23T16:41:25.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4190000000000001e-06 2024-04-23T16:41:25.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.564643+0000", 2024-04-23T16:41:25.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1535000000000001e-05 2024-04-23T16:41:25.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.564651+0000", 2024-04-23T16:41:25.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3960000000000006e-06 2024-04-23T16:41:25.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.568131+0000", 2024-04-23T16:41:25.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003480351 2024-04-23T16:41:25.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.568211+0000", 2024-04-23T16:41:25.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9142999999999998e-05 2024-04-23T16:41:25.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 5", 2024-04-23T16:41:25.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.568288+0000", 2024-04-23T16:41:25.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6978999999999996e-05 2024-04-23T16:41:25.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:41:25.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.574334+0000", 2024-04-23T16:41:25.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060461170000000002 2024-04-23T16:41:25.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:41:25.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628268+0000", 2024-04-23T16:41:25.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.053934425000000001 2024-04-23T16:41:25.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:41:25.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628316+0000", 2024-04-23T16:41:25.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8235000000000002e-05 2024-04-23T16:41:25.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628356+0000", 2024-04-23T16:41:25.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9830000000000003e-05 2024-04-23T16:41:25.086 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.086 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.086 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.087 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:262204 6.4 6:2193abc7:::benchmark_data_smithi188_130221_object107743:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-23T16:41:25.087 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:40:52.569467+0000", 2024-04-23T16:41:25.087 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.456033952999999, 2024-04-23T16:41:25.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.059027589999999998, 2024-04-23T16:41:25.087 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.087 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.087 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:41:25.087 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:41:25.087 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:41:25.087 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:41:25.087 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 262204 2024-04-23T16:41:25.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.088 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.569467+0000", 2024-04-23T16:41:25.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.569467+0000", 2024-04-23T16:41:25.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.569469+0000", 2024-04-23T16:41:25.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9350000000000001e-06 2024-04-23T16:41:25.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.569475+0000", 2024-04-23T16:41:25.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8000000000000004e-06 2024-04-23T16:41:25.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.569486+0000", 2024-04-23T16:41:25.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1127e-05 2024-04-23T16:41:25.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.569494+0000", 2024-04-23T16:41:25.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2940000000000006e-06 2024-04-23T16:41:25.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.571271+0000", 2024-04-23T16:41:25.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0017771849999999999 2024-04-23T16:41:25.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.571348+0000", 2024-04-23T16:41:25.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6470000000000005e-05 2024-04-23T16:41:25.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 5", 2024-04-23T16:41:25.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.571418+0000", 2024-04-23T16:41:25.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9659000000000005e-05 2024-04-23T16:41:25.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:41:25.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.574476+0000", 2024-04-23T16:41:25.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030587570000000001 2024-04-23T16:41:25.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:41:25.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628402+0000", 2024-04-23T16:41:25.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.053925858 2024-04-23T16:41:25.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:41:25.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628444+0000", 2024-04-23T16:41:25.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1665999999999997e-05 2024-04-23T16:41:25.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628495+0000", 2024-04-23T16:41:25.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0838999999999999e-05 2024-04-23T16:41:25.093 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.093 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.093 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.093 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop_reply(client.16308.0:262194 6.4 e521/503 ondisk, result = 0)", 2024-04-23T16:41:25.093 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:40:52.580024+0000", 2024-04-23T16:41:25.093 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.445477071999999, 2024-04-23T16:41:25.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.048187992999999998, 2024-04-23T16:41:25.093 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.093 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.093 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:41:25.094 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.580024+0000", 2024-04-23T16:41:25.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.580024+0000", 2024-04-23T16:41:25.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.580026+0000", 2024-04-23T16:41:25.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.782e-06 2024-04-23T16:41:25.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.580034+0000", 2024-04-23T16:41:25.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7620000000000001e-06 2024-04-23T16:41:25.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.580043+0000", 2024-04-23T16:41:25.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9760000000000001e-06 2024-04-23T16:41:25.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.580049+0000", 2024-04-23T16:41:25.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5539999999999997e-06 2024-04-23T16:41:25.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.580091+0000", 2024-04-23T16:41:25.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2366000000000001e-05 2024-04-23T16:41:25.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.580097+0000", 2024-04-23T16:41:25.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2789999999999998e-06 2024-04-23T16:41:25.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628212+0000", 2024-04-23T16:41:25.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.048115274 2024-04-23T16:41:25.097 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.097 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.097 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.098 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop_reply(client.16308.0:262201 6.4 e521/503 ondisk, result = 0)", 2024-04-23T16:41:25.098 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:40:52.580063+0000", 2024-04-23T16:41:25.098 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.445438189000001, 2024-04-23T16:41:25.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.048301819000000003, 2024-04-23T16:41:25.098 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.098 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.098 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:41:25.098 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.580063+0000", 2024-04-23T16:41:25.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.580063+0000", 2024-04-23T16:41:25.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.580063+0000", 2024-04-23T16:41:25.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8999999999999997e-07 2024-04-23T16:41:25.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.580066+0000", 2024-04-23T16:41:25.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.869e-06 2024-04-23T16:41:25.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.580071+0000", 2024-04-23T16:41:25.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9869999999999999e-06 2024-04-23T16:41:25.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.580077+0000", 2024-04-23T16:41:25.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.609e-06 2024-04-23T16:41:25.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628254+0000", 2024-04-23T16:41:25.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.048177109000000003 2024-04-23T16:41:25.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628262+0000", 2024-04-23T16:41:25.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1040000000000007e-06 2024-04-23T16:41:25.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628365+0000", 2024-04-23T16:41:25.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000102651 2024-04-23T16:41:25.102 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.103 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.103 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.103 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop_reply(client.16308.0:262204 6.4 e521/503 ondisk, result = 0)", 2024-04-23T16:41:25.103 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:40:52.581530+0000", 2024-04-23T16:41:25.103 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.443971086000001, 2024-04-23T16:41:25.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046968803000000003, 2024-04-23T16:41:25.103 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.103 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.104 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:41:25.104 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.581530+0000", 2024-04-23T16:41:25.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.581530+0000", 2024-04-23T16:41:25.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.581531+0000", 2024-04-23T16:41:25.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.175e-06 2024-04-23T16:41:25.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.581536+0000", 2024-04-23T16:41:25.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9699999999999998e-06 2024-04-23T16:41:25.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628103+0000", 2024-04-23T16:41:25.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046566593000000003 2024-04-23T16:41:25.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628115+0000", 2024-04-23T16:41:25.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2391999999999999e-05 2024-04-23T16:41:25.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628392+0000", 2024-04-23T16:41:25.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00027649499999999997 2024-04-23T16:41:25.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628397+0000", 2024-04-23T16:41:25.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7370000000000003e-06 2024-04-23T16:41:25.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.107 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628499+0000", 2024-04-23T16:41:25.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010144100000000001 2024-04-23T16:41:25.107 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.107 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.107 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.107 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:262229 6.4 6:290f76ea:::benchmark_data_smithi188_130221_object107768:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-23T16:41:25.107 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:40:52.592297+0000", 2024-04-23T16:41:25.107 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.433204195999998, 2024-04-23T16:41:25.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.054965180000000002, 2024-04-23T16:41:25.108 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.108 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.108 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:41:25.108 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:41:25.108 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:41:25.108 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:41:25.108 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 262229 2024-04-23T16:41:25.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.108 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.592297+0000", 2024-04-23T16:41:25.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.592297+0000", 2024-04-23T16:41:25.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.592299+0000", 2024-04-23T16:41:25.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3819999999999998e-06 2024-04-23T16:41:25.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.592306+0000", 2024-04-23T16:41:25.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9179999999999997e-06 2024-04-23T16:41:25.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628100+0000", 2024-04-23T16:41:25.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.035794356999999999 2024-04-23T16:41:25.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628117+0000", 2024-04-23T16:41:25.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6269999999999998e-05 2024-04-23T16:41:25.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628524+0000", 2024-04-23T16:41:25.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040728200000000002 2024-04-23T16:41:25.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628629+0000", 2024-04-23T16:41:25.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010476000000000001 2024-04-23T16:41:25.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 5", 2024-04-23T16:41:25.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.628730+0000", 2024-04-23T16:41:25.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101369 2024-04-23T16:41:25.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:41:25.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.640287+0000", 2024-04-23T16:41:25.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011556741000000001 2024-04-23T16:41:25.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:41:25.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.647196+0000", 2024-04-23T16:41:25.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069093360000000003 2024-04-23T16:41:25.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:41:25.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.647228+0000", 2024-04-23T16:41:25.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2221999999999997e-05 2024-04-23T16:41:25.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.113 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:40:52.647262+0000", 2024-04-23T16:41:25.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3543000000000001e-05 2024-04-23T16:41:25.113 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.113 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.113 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.113 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:291492 6.4 6:2161f0f3:::benchmark_data_smithi188_130221_object137031:head [delete] snapc 0=[] ondisk+write+known_if_redirected e532)", 2024-04-23T16:41:25.113 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:41:19.288911+0000", 2024-04-23T16:41:25.113 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.7365904499999996, 2024-04-23T16:41:25.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.10719031499999999, 2024-04-23T16:41:25.113 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.113 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.113 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:41:25.114 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:41:25.114 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:41:25.114 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:41:25.114 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 291492 2024-04-23T16:41:25.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.114 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.114 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.288911+0000", 2024-04-23T16:41:25.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.114 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.288911+0000", 2024-04-23T16:41:25.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.288912+0000", 2024-04-23T16:41:25.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.891e-06 2024-04-23T16:41:25.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.288918+0000", 2024-04-23T16:41:25.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6359999999999999e-06 2024-04-23T16:41:25.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.288928+0000", 2024-04-23T16:41:25.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.431e-06 2024-04-23T16:41:25.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.288934+0000", 2024-04-23T16:41:25.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.489e-06 2024-04-23T16:41:25.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.288985+0000", 2024-04-23T16:41:25.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0929e-05 2024-04-23T16:41:25.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.289064+0000", 2024-04-23T16:41:25.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9529000000000004e-05 2024-04-23T16:41:25.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 5", 2024-04-23T16:41:25.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.289129+0000", 2024-04-23T16:41:25.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4898000000000001e-05 2024-04-23T16:41:25.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:41:25.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.292319+0000", 2024-04-23T16:41:25.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0031894779999999999 2024-04-23T16:41:25.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:41:25.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.396033+0000", 2024-04-23T16:41:25.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.10371372299999999 2024-04-23T16:41:25.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:41:25.119 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.396068+0000", 2024-04-23T16:41:25.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4986e-05 2024-04-23T16:41:25.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.119 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.119 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.396101+0000", 2024-04-23T16:41:25.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3324999999999997e-05 2024-04-23T16:41:25.119 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.119 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.119 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.120 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:291496 6.4 6:2e9d1974:::benchmark_data_smithi188_130221_object137035:head [delete] snapc 0=[] ondisk+write+known_if_redirected e532)", 2024-04-23T16:41:25.120 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:41:19.293393+0000", 2024-04-23T16:41:25.120 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.732107729, 2024-04-23T16:41:25.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.102865559, 2024-04-23T16:41:25.120 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.120 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.120 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:41:25.120 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:41:25.120 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:41:25.120 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:41:25.120 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 291496 2024-04-23T16:41:25.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.120 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.293393+0000", 2024-04-23T16:41:25.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.293393+0000", 2024-04-23T16:41:25.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.293395+0000", 2024-04-23T16:41:25.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.392e-06 2024-04-23T16:41:25.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.293399+0000", 2024-04-23T16:41:25.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9169999999999999e-06 2024-04-23T16:41:25.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.293406+0000", 2024-04-23T16:41:25.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9809999999999997e-06 2024-04-23T16:41:25.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.293410+0000", 2024-04-23T16:41:25.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7249999999999997e-06 2024-04-23T16:41:25.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.293470+0000", 2024-04-23T16:41:25.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9361000000000003e-05 2024-04-23T16:41:25.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.389392+0000", 2024-04-23T16:41:25.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.095922293000000006 2024-04-23T16:41:25.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 5", 2024-04-23T16:41:25.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.389507+0000", 2024-04-23T16:41:25.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011497699999999999 2024-04-23T16:41:25.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:41:25.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.396142+0000", 2024-04-23T16:41:25.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066347419999999999 2024-04-23T16:41:25.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:41:25.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.396203+0000", 2024-04-23T16:41:25.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1032000000000001e-05 2024-04-23T16:41:25.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:41:25.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.396230+0000", 2024-04-23T16:41:25.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6826e-05 2024-04-23T16:41:25.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.396259+0000", 2024-04-23T16:41:25.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9312999999999999e-05 2024-04-23T16:41:25.126 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.126 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.126 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.126 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.16308.0:291497 6.d e532/510 6:bcda3e19:::benchmark_data_smithi188_130221_object137036:head v 532'17913, mlcod=532'17911)", 2024-04-23T16:41:25.126 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:41:19.296543+0000", 2024-04-23T16:41:25.126 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.7289578739999998, 2024-04-23T16:41:25.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.1058099, 2024-04-23T16:41:25.127 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.127 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.127 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:41:25.127 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.127 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.296543+0000", 2024-04-23T16:41:25.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.296543+0000", 2024-04-23T16:41:25.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.128 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.296546+0000", 2024-04-23T16:41:25.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6929999999999999e-06 2024-04-23T16:41:25.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.128 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.296559+0000", 2024-04-23T16:41:25.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3107e-05 2024-04-23T16:41:25.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.389339+0000", 2024-04-23T16:41:25.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.092780043000000006 2024-04-23T16:41:25.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.389351+0000", 2024-04-23T16:41:25.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1911e-05 2024-04-23T16:41:25.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.389406+0000", 2024-04-23T16:41:25.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4818000000000003e-05 2024-04-23T16:41:25.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.389422+0000", 2024-04-23T16:41:25.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5993000000000001e-05 2024-04-23T16:41:25.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:41:25.131 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.402303+0000", 2024-04-23T16:41:25.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012881737000000001 2024-04-23T16:41:25.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.131 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.402353+0000", 2024-04-23T16:41:25.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9598000000000002e-05 2024-04-23T16:41:25.131 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.131 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.131 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.132 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:291536 6.4 6:2025fb99:::benchmark_data_smithi188_130221_object137075:head [delete] snapc 0=[] ondisk+write+known_if_redirected e532)", 2024-04-23T16:41:25.132 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:41:19.345256+0000", 2024-04-23T16:41:25.132 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.6802445080000004, 2024-04-23T16:41:25.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.056776678999999997, 2024-04-23T16:41:25.132 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:41:25.132 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:41:25.132 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:41:25.132 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:41:25.132 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:41:25.132 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:41:25.132 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 291536 2024-04-23T16:41:25.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.133 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:41:25.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:41:25.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.345256+0000", 2024-04-23T16:41:25.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:41:25.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.345256+0000", 2024-04-23T16:41:25.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:41:25.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:41:25.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.345259+0000", 2024-04-23T16:41:25.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3779999999999999e-06 2024-04-23T16:41:25.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:41:25.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.345265+0000", 2024-04-23T16:41:25.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2029999999999997e-06 2024-04-23T16:41:25.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:41:25.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.389340+0000", 2024-04-23T16:41:25.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.04407486 2024-04-23T16:41:25.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:41:25.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.389354+0000", 2024-04-23T16:41:25.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4054e-05 2024-04-23T16:41:25.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:41:25.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.392926+0000", 2024-04-23T16:41:25.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00357208 2024-04-23T16:41:25.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:41:25.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.393052+0000", 2024-04-23T16:41:25.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000126346 2024-04-23T16:41:25.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 5", 2024-04-23T16:41:25.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.393133+0000", 2024-04-23T16:41:25.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0106999999999997e-05 2024-04-23T16:41:25.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:41:25.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.396194+0000", 2024-04-23T16:41:25.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030616319999999999 2024-04-23T16:41:25.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:41:25.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.401942+0000", 2024-04-23T16:41:25.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0057478110000000002 2024-04-23T16:41:25.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:41:25.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.401988+0000", 2024-04-23T16:41:25.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6437000000000003e-05 2024-04-23T16:41:25.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:41:25.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:41:25.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:41:25.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.402033+0000", 2024-04-23T16:41:25.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4771000000000002e-05 2024-04-23T16:41:25.137 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.137 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.137 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.138 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:41:25.138 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:41:25.138 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:41:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:24 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:41:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2775573471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:41:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:24 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:41:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.383 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:25 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:41:25.383 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:25 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2775573471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:41:25.383 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:25 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:41:25.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:24 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:41:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2775573471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:41:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:24 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:41:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:25.846 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:41:25.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:25.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:41:25.953 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:41:25.989 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:26.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:26.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:26.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:26.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:26 smithi103 ceph-mon[39629]: pgmap v1489: 27 pgs: 27 active+clean; 929 MiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-23T16:41:26.132 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:41:26.132 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:41:26.133 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:41:26.133 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:41:26.133 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:41:26.133 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:41:26.133 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:41:26.133 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:41:26.133 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:41:26.133 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:41:26.133 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:41:26.133 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:41:26.133 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:41:26.133 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:41:26.134 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:41:26.134 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:41:26.134 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:41:26.134 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:41:26.134 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:41:26.134 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:41:26.134 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:41:26.134 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:41:26.134 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:41:26.134 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:41:26.345 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:41:26.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:26.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:41:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:26 smithi148 ceph-mon[44520]: pgmap v1489: 27 pgs: 27 active+clean; 929 MiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-23T16:41:26.450 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:41:26.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:26.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:26.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:26.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:26 smithi188 ceph-mon[44101]: pgmap v1489: 27 pgs: 27 active+clean; 929 MiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-23T16:41:26.509 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:26.544 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:41:26.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:26.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:41:26.649 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:41:26.706 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:26.739 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:41:26.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:26.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:41:26.847 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:41:26.873 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:26.897 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:41:26.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:26.902 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:41:27.003 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:41:27.044 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:27.069 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:41:27.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:27.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:41:27.192 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:41:27.220 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:27.239 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:41:27.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:27.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:41:27.343 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:41:27.375 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:27.574 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:41:27.579 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:41:27.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:27.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:41:27.613 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-23T16:41:27.694 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:41:27.734 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:27.773 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:41:27.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:27.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:41:27.854 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:41:27.878 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:41:27.917 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:27.952 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:41:27.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:27.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:41:28.019 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:28 smithi103 ceph-mon[39629]: pgmap v1490: 27 pgs: 27 active+clean; 691 MiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-23T16:41:28.021 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:28 smithi148 ceph-mon[44520]: pgmap v1490: 27 pgs: 27 active+clean; 691 MiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-23T16:41:28.071 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:41:28.098 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:28.117 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:41:28.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:28.120 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:41:28.221 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:41:28.268 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:28.306 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:41:28.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:28.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:41:28.413 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:41:28.451 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:28.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:28 smithi188 ceph-mon[44101]: pgmap v1490: 27 pgs: 27 active+clean; 691 MiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-23T16:41:28.477 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:41:28.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:28.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:41:28.555 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:41:28.583 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:41:28.627 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:28.715 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:41:28.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:28.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:41:28.840 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:41:28.876 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:28.905 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:41:30.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:30 smithi103 ceph-mon[39629]: pgmap v1491: 27 pgs: 27 active+clean; 650 MiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:41:30.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:30 smithi148 ceph-mon[44520]: pgmap v1491: 27 pgs: 27 active+clean; 650 MiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:41:30.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:30 smithi188 ceph-mon[44101]: pgmap v1491: 27 pgs: 27 active+clean; 650 MiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-23T16:41:31.139 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:41:31.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:31.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:41:31.247 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:41:31.298 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:31.340 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:41:31.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:31.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:41:31.453 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:41:31.518 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:32 smithi103 ceph-mon[39629]: pgmap v1492: 27 pgs: 27 active+clean; 498 MiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-23T16:41:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:32 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:41:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:32 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:41:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:32 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1790052352' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:41:32.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:32 smithi103 ceph-mon[39629]: 6.7 scrub starts 2024-04-23T16:41:32.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:32 smithi148 ceph-mon[44520]: pgmap v1492: 27 pgs: 27 active+clean; 498 MiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-23T16:41:32.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:32 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:41:32.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:32 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:41:32.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:32 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1790052352' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:41:32.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:32 smithi148 ceph-mon[44520]: 6.7 scrub starts 2024-04-23T16:41:32.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:32 smithi188 ceph-mon[44101]: pgmap v1492: 27 pgs: 27 active+clean; 498 MiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-23T16:41:32.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:32 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:41:32.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:32 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:41:32.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:32 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1790052352' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:41:32.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:32 smithi188 ceph-mon[44101]: 6.7 scrub starts 2024-04-23T16:41:32.523 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:41:32.524 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":534,"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","created":"2024-04-23T16:08:13.477590+0000","modified":"2024-04-23T16:41:21.271699+0000","last_up_change":"2024-04-23T16:38:58.018521+0000","last_in_change":"2024-04-23T16:39:19.891409+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":31,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T16:11:22.416777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"303","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"301","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.190000057220459,"score_stable":2.190000057220459,"optimal_score":0.75,"raw_score_acting":1.6399999856948853,"raw_score_stable":1.6399999856948853,"primary_affinity_weighted":0.98181778192520142,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}},{"pool":6,"pool_name":"unique_pool_4","create_time":"2024-04-23T16:37:13.886119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"459","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.2699999809265137,"score_stable":2.2699999809265137,"optimal_score":0.82999998331069946,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.98181778192520142,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}}],"osds":[{"osd":0,"uuid":"4c084047-8f01-4b86-a60c-5592735b5862","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":511,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6801","nonce":4274040097}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6802","nonce":4274040097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6804","nonce":4274040097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6803","nonce":4274040097}]},"public_addr":"172.21.15.103:6801/4274040097","cluster_addr":"172.21.15.103:6802/4274040097","heartbeat_back_addr":"172.21.15.103:6804/4274040097","heartbeat_front_addr":"172.21.15.103:6803/4274040097","state":["exists","up"]},{"osd":1,"uuid":"3438f2bd-5fb7-432c-b383-29e18c442f20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":232,"up_from":271,"up_thru":503,"down_at":233,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6805","nonce":859282187}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6806","nonce":859282187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6808","nonce":859282187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6807","nonce":859282187}]},"public_addr":"172.21.15.103:6805/859282187","cluster_addr":"172.21.15.103:6806/859282187","heartbeat_back_addr":"172.21.15.103:6808/859282187","heartbeat_front_addr":"172.21.15.103:6807/859282187","state":["exists","up"]},{"osd":2,"uuid":"c17fffb4-8d17-4546-ba24-e5951a2a37fe","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":386,"up_from":481,"up_thru":359,"down_at":387,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6809","nonce":3324226049}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6810","nonce":3324226049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6812","nonce":3324226049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6811","nonce":3324226049}]},"public_addr":"172.21.15.103:6809/3324226049","cluster_addr":"172.21.15.103:6810/3324226049","heartbeat_back_addr":"172.21.15.103:6812/3324226049","heartbeat_front_addr":"172.21.15.103:6811/3324226049","state":["exists","up"]},{"osd":3,"uuid":"cee91a3d-55dc-4e89-b568-86821632bb1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":502,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6813","nonce":1895183074}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6814","nonce":1895183074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6816","nonce":1895183074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6815","nonce":1895183074}]},"public_addr":"172.21.15.103:6813/1895183074","cluster_addr":"172.21.15.103:6814/1895183074","heartbeat_back_addr":"172.21.15.103:6816/1895183074","heartbeat_front_addr":"172.21.15.103:6815/1895183074","state":["exists","up"]},{"osd":4,"uuid":"61d5385b-2c82-42f8-93c3-8f1baf71c65e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":511,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6800","nonce":794945071}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6801","nonce":794945071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6803","nonce":794945071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6802","nonce":794945071}]},"public_addr":"172.21.15.148:6800/794945071","cluster_addr":"172.21.15.148:6801/794945071","heartbeat_back_addr":"172.21.15.148:6803/794945071","heartbeat_front_addr":"172.21.15.148:6802/794945071","state":["exists","up"]},{"osd":5,"uuid":"5d492189-c3cd-47ad-a12a-ed4d275bcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":510,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6804","nonce":3547200711}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6805","nonce":3547200711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6807","nonce":3547200711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6806","nonce":3547200711}]},"public_addr":"172.21.15.148:6804/3547200711","cluster_addr":"172.21.15.148:6805/3547200711","heartbeat_back_addr":"172.21.15.148:6807/3547200711","heartbeat_front_addr":"172.21.15.148:6806/3547200711","state":["exists","up"]},{"osd":6,"uuid":"89f2379e-cf14-4e69-8747-7535db81b70b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":394,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6808","nonce":366919758}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6809","nonce":366919758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6811","nonce":366919758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6810","nonce":366919758}]},"public_addr":"172.21.15.148:6808/366919758","cluster_addr":"172.21.15.148:6809/366919758","heartbeat_back_addr":"172.21.15.148:6811/366919758","heartbeat_front_addr":"172.21.15.148:6810/366919758","state":["exists","up"]},{"osd":7,"uuid":"f27b035a-558f-4528-80ea-2a00d03f6539","up":1,"in":1,"weight":1,"primary_affinity":0.7818145751953125,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":529,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6812","nonce":3719550888}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6813","nonce":3719550888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6815","nonce":3719550888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6814","nonce":3719550888}]},"public_addr":"172.21.15.148:6812/3719550888","cluster_addr":"172.21.15.148:6813/3719550888","heartbeat_back_addr":"172.21.15.148:6815/3719550888","heartbeat_front_addr":"172.21.15.148:6814/3719550888","state":["exists","up"]},{"osd":8,"uuid":"a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":512,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6800","nonce":1151408595}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6801","nonce":1151408595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6803","nonce":1151408595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6802","nonce":1151408595}]},"public_addr":"172.21.15.188:6800/1151408595","cluster_addr":"172.21.15.188:6801/1151408595","heartbeat_back_addr":"172.21.15.188:6803/1151408595","heartbeat_front_addr":"172.21.15.188:6802/1151408595","state":["exists","up"]},{"osd":9,"uuid":"124d9b05-11eb-4622-b6a2-c4d5bd60b7e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":455,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6804","nonce":306501862}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6805","nonce":306501862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6807","nonce":306501862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6806","nonce":306501862}]},"public_addr":"172.21.15.188:6804/306501862","cluster_addr":"172.21.15.188:6805/306501862","heartbeat_back_addr":"172.21.15.188:6807/306501862","heartbeat_front_addr":"172.21.15.188:6806/306501862","state":["exists","up"]},{"osd":10,"uuid":"3dfb9070-e534-4ea4-a25e-ee1c51297447","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6808","nonce":1231263138}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6809","nonce":1231263138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6811","nonce":1231263138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6810","nonce":1231263138}]},"public_addr":"172.21.15.188:6808/1231263138","cluster_addr":"172.21.15.188:6809/1231263138","heartbeat_back_addr":"172.21.15.188:6811/1231263138","heartbeat_front_addr":"172.21.15.188:6810/1231263138","state":["exists","up"]},{"osd":11,"uuid":"bc163925-f623-4f63-ab89-35e2d511a730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":455,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6812","nonce":457992661}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6813","nonce":457992661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6815","nonce":457992661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6814","nonce":457992661}]},"public_addr":"172.21.15.188:6812/457992661","cluster_addr":"172.21.15.188:6813/457992661","heartbeat_back_addr":"172.21.15.188:6815/457992661","heartbeat_front_addr":"172.21.15.188:6814/457992661","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:40:23.396020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:41:20.920306+0000","dead_epoch":232},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-23T16:41:17.780803+0000","dead_epoch":386},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:41:04.562526+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:40:42.895780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:40:57.635873+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:41:19.505260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:41:14.874297+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:40:26.063810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:40:38.933052+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-23T16:41:07.853088+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:40:59.478809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6800/3535470771":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1188058303":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1510553721":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/281182801":"2024-04-24T16:08:44.450632+0000","172.21.15.103:6800/2049124518":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/466241963":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/2726838698":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/596230573":"2024-04-24T16:09:14.324448+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T16:41:32.884 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:41:32.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:32.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:41:32.993 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:41:33.024 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:33.260 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:33 smithi103 ceph-mon[39629]: osdmap e535: 12 total, 12 up, 10 in 2024-04-23T16:41:33.260 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:33 smithi103 ceph-mon[39629]: 6.7 scrub ok 2024-04-23T16:41:33.281 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:41:33.292 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:41:33.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:33.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:41:33.304 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-23T16:41:33.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:33 smithi148 ceph-mon[44520]: osdmap e535: 12 total, 12 up, 10 in 2024-04-23T16:41:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:33 smithi148 ceph-mon[44520]: 6.7 scrub ok 2024-04-23T16:41:33.398 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:41:33.443 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:33.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:33 smithi188 ceph-mon[44101]: osdmap e535: 12 total, 12 up, 10 in 2024-04-23T16:41:33.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:33 smithi188 ceph-mon[44101]: 6.7 scrub ok 2024-04-23T16:41:33.659 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:41:33.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:33.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:41:33.766 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:41:33.793 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:33.813 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:41:33.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:33.816 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:41:33.917 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:41:33.973 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:34.011 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:41:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:34.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:41:34.124 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:41:34.167 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:34.197 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:41:34.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:34.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:41:34.243 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:34 smithi103 ceph-mon[39629]: pgmap v1494: 27 pgs: 27 active+clean; 328 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-23T16:41:34.243 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:34 smithi103 ceph-mon[39629]: 6.f scrub starts 2024-04-23T16:41:34.243 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:34 smithi103 ceph-mon[39629]: 6.f scrub ok 2024-04-23T16:41:34.306 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:41:34.338 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:34.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:34 smithi148 ceph-mon[44520]: pgmap v1494: 27 pgs: 27 active+clean; 328 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-23T16:41:34.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:34 smithi148 ceph-mon[44520]: 6.f scrub starts 2024-04-23T16:41:34.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:34 smithi148 ceph-mon[44520]: 6.f scrub ok 2024-04-23T16:41:34.467 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:41:34.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:34.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:41:34.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:34 smithi188 ceph-mon[44101]: pgmap v1494: 27 pgs: 27 active+clean; 328 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-23T16:41:34.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:34 smithi188 ceph-mon[44101]: 6.f scrub starts 2024-04-23T16:41:34.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:34 smithi188 ceph-mon[44101]: 6.f scrub ok 2024-04-23T16:41:34.572 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:41:34.594 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:41:34.594 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:41:34.608 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:35.294 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:35 smithi103 ceph-mon[39629]: pgmap v1495: 27 pgs: 27 active+clean; 244 MiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-23T16:41:35.466 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Removed 154458 objects 2024-04-23T16:41:35.466 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Clean up completed and total clean up time :153.987 2024-04-23T16:41:35.554 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-04-23T16:41:35.583 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-04-23T16:41:35.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:35 smithi148 ceph-mon[44520]: pgmap v1495: 27 pgs: 27 active+clean; 244 MiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-23T16:41:35.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:35 smithi188 ceph-mon[44101]: pgmap v1495: 27 pgs: 27 active+clean; 244 MiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-23T16:41:35.864 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:41:35.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:35.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:41:35.894 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-23T16:41:35.900 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:41:35.968 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:41:35.994 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:36.842 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:36 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:41:36.843 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:36 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:41:36.844 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:41:36.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:36.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:41:36.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:36 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:41:36.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:36 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:41:36.949 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:41:36.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:36 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:41:36.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:36 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:41:36.981 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:37.002 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:41:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:37.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:41:37.107 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:41:37.139 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:37.159 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:41:37.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:37.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:41:37.263 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:41:37.292 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:37.843 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:41:37.845 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:37 smithi103 ceph-mon[39629]: 6.2 scrub starts 2024-04-23T16:41:37.845 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:37 smithi103 ceph-mon[39629]: 6.2 scrub ok 2024-04-23T16:41:37.845 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:37 smithi103 ceph-mon[39629]: pgmap v1496: 27 pgs: 27 active+clean; 42 MiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-23T16:41:37.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:37.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:41:37.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:37 smithi148 ceph-mon[44520]: 6.2 scrub starts 2024-04-23T16:41:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:37 smithi148 ceph-mon[44520]: 6.2 scrub ok 2024-04-23T16:41:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:37 smithi148 ceph-mon[44520]: pgmap v1496: 27 pgs: 27 active+clean; 42 MiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-23T16:41:37.947 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:41:37.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:37 smithi188 ceph-mon[44101]: 6.2 scrub starts 2024-04-23T16:41:37.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:37 smithi188 ceph-mon[44101]: 6.2 scrub ok 2024-04-23T16:41:37.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:37 smithi188 ceph-mon[44101]: pgmap v1496: 27 pgs: 27 active+clean; 42 MiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-23T16:41:37.979 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:38.000 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:41:38.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:38.003 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:41:38.104 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:41:38.133 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:38.283 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:41:38.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:38.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:41:38.387 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:41:38.420 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:38.441 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:41:38.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:38.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:41:38.545 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:41:38.576 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:38.590 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:41:38.599 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:41:38.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:38.602 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:41:38.703 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:41:38.732 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:38 smithi148 ceph-mon[44520]: 6.0 scrub starts 2024-04-23T16:41:38.732 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:38 smithi148 ceph-mon[44520]: 6.0 scrub ok 2024-04-23T16:41:38.732 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:38 smithi148 ceph-mon[44520]: 6.e scrub starts 2024-04-23T16:41:38.732 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:38 smithi148 ceph-mon[44520]: 6.e scrub ok 2024-04-23T16:41:38.733 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:38 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2837051552' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:41:38.733 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:38 smithi148 ceph-mon[44520]: 6.8 scrub starts 2024-04-23T16:41:38.733 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:38 smithi148 ceph-mon[44520]: 6.8 scrub ok 2024-04-23T16:41:38.735 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:38.757 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:41:38.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:38.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:41:38.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:38 smithi103 ceph-mon[39629]: 6.0 scrub starts 2024-04-23T16:41:38.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:38 smithi103 ceph-mon[39629]: 6.0 scrub ok 2024-04-23T16:41:38.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:38 smithi103 ceph-mon[39629]: 6.e scrub starts 2024-04-23T16:41:38.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:38 smithi103 ceph-mon[39629]: 6.e scrub ok 2024-04-23T16:41:38.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:38 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2837051552' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:41:38.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:38 smithi103 ceph-mon[39629]: 6.8 scrub starts 2024-04-23T16:41:38.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:38 smithi103 ceph-mon[39629]: 6.8 scrub ok 2024-04-23T16:41:38.861 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:41:38.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:38 smithi188 ceph-mon[44101]: 6.0 scrub starts 2024-04-23T16:41:38.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:38 smithi188 ceph-mon[44101]: 6.0 scrub ok 2024-04-23T16:41:38.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:38 smithi188 ceph-mon[44101]: 6.e scrub starts 2024-04-23T16:41:38.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:38 smithi188 ceph-mon[44101]: 6.e scrub ok 2024-04-23T16:41:38.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:38 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2837051552' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:41:38.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:38 smithi188 ceph-mon[44101]: 6.8 scrub starts 2024-04-23T16:41:38.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:38 smithi188 ceph-mon[44101]: 6.8 scrub ok 2024-04-23T16:41:38.893 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:38.914 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:41:38.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:38.917 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:41:39.018 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:41:39.047 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:39.068 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:41:39.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:39.071 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:41:39.171 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:41:39.200 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:39.219 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:41:39.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:39.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:41:39.323 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:41:39.350 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:39.370 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:41:39.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:39.373 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:41:39.474 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:41:39.500 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:39.519 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:41:39.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:39.522 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:41:39.614 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:41:39.614 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:41:39.623 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:41:39.653 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:39.675 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:41:39.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:39.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:41:39.779 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:41:39.808 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:39.828 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:41:39.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:39.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:41:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:39 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:41:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:39 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:41:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:39 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2837051552' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:41:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:39 smithi103 ceph-mon[39629]: osdmap e536: 12 total, 12 up, 10 in 2024-04-23T16:41:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:39 smithi103 ceph-mon[39629]: pgmap v1498: 27 pgs: 27 active+clean; 20 MiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 956 op/s 2024-04-23T16:41:39.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:39 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2059277149' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:41:39.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:39 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:41:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:39 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:41:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:39 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2837051552' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:41:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:39 smithi148 ceph-mon[44520]: osdmap e536: 12 total, 12 up, 10 in 2024-04-23T16:41:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:39 smithi148 ceph-mon[44520]: pgmap v1498: 27 pgs: 27 active+clean; 20 MiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 956 op/s 2024-04-23T16:41:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:39 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2059277149' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:41:39.932 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:41:39.959 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:39.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:39 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:41:39.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:39 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:41:39.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:39 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2837051552' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:41:39.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:39 smithi188 ceph-mon[44101]: osdmap e536: 12 total, 12 up, 10 in 2024-04-23T16:41:39.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:39 smithi188 ceph-mon[44101]: pgmap v1498: 27 pgs: 27 active+clean; 20 MiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 956 op/s 2024-04-23T16:41:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:39 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2059277149' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:41:39.979 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:41:39.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:39.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:41:40.084 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:41:40.112 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:40.131 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:41:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:40.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:41:40.231 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:41:40.231 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:41:40.231 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:41:40.231 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:41:40.231 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:41:40.232 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:41:40.232 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:41:40.232 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:41:40.232 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:41:40.232 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:41:40.232 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:41:40.232 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:41:40.232 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:41:40.232 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:41:40.232 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:41:40.232 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:41:40.233 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:41:40.233 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:41:40.233 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:41:40.233 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:41:40.233 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:41:40.233 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:41:40.233 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:41:40.233 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:41:40.234 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:41:40.263 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:40.512 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:41:40.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:40.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:41:40.616 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:41:40.643 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:40.696 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:41:40.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:40.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:41:40.804 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:41:40.838 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:40 smithi103 ceph-mon[39629]: 6.3 scrub starts 2024-04-23T16:41:40.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:40 smithi103 ceph-mon[39629]: 6.3 scrub ok 2024-04-23T16:41:40.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.859 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:41:40.861 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.862 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.862 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.862 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.862 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.862 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.862 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.862 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.862 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.862 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:40 smithi148 ceph-mon[44520]: 6.3 scrub starts 2024-04-23T16:41:40.862 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:40 smithi148 ceph-mon[44520]: 6.3 scrub ok 2024-04-23T16:41:40.862 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.862 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.862 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:40.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:41:40.964 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:41:40.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:40 smithi188 ceph-mon[44101]: 6.3 scrub starts 2024-04-23T16:41:40.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:40 smithi188 ceph-mon[44101]: 6.3 scrub ok 2024-04-23T16:41:40.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:40.993 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:41.241 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:41:41.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:41.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:41:41.345 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:41:41.378 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:41.399 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:41:41.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:41.402 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:41:41.503 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:41:41.535 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:41.556 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:41:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:41.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:41:41.661 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:41:41.690 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:41.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:41 smithi188 ceph-mon[44101]: pgmap v1499: 27 pgs: 27 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 595 op/s 2024-04-23T16:41:41.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:41 smithi148 ceph-mon[44520]: pgmap v1499: 27 pgs: 27 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 595 op/s 2024-04-23T16:41:42.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:41 smithi103 ceph-mon[39629]: pgmap v1499: 27 pgs: 27 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 595 op/s 2024-04-23T16:41:42.139 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:41:42.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:42.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:41:42.152 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:41:42.243 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:41:42.276 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:42.297 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:41:42.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:42.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:41:42.402 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:41:42.429 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:42.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:42 smithi103 ceph-mon[39629]: 6.1 scrub starts 2024-04-23T16:41:42.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:42 smithi103 ceph-mon[39629]: 6.1 scrub ok 2024-04-23T16:41:42.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:42 smithi103 ceph-mon[39629]: 6.5 deep-scrub starts 2024-04-23T16:41:42.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:42 smithi103 ceph-mon[39629]: 6.5 deep-scrub ok 2024-04-23T16:41:42.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:42 smithi148 ceph-mon[44520]: 6.1 scrub starts 2024-04-23T16:41:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:42 smithi148 ceph-mon[44520]: 6.1 scrub ok 2024-04-23T16:41:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:42 smithi148 ceph-mon[44520]: 6.5 deep-scrub starts 2024-04-23T16:41:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:42 smithi148 ceph-mon[44520]: 6.5 deep-scrub ok 2024-04-23T16:41:42.878 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:41:42.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:42.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:41:42.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:42 smithi188 ceph-mon[44101]: 6.1 scrub starts 2024-04-23T16:41:42.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:42 smithi188 ceph-mon[44101]: 6.1 scrub ok 2024-04-23T16:41:42.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:42 smithi188 ceph-mon[44101]: 6.5 deep-scrub starts 2024-04-23T16:41:42.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:42 smithi188 ceph-mon[44101]: 6.5 deep-scrub ok 2024-04-23T16:41:42.982 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:41:43.009 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:43.357 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:41:43.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:43.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:41:43.461 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:41:43.490 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:43.638 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:41:43.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:43.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:41:43.742 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:41:43.775 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:43.796 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:41:43.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:43.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:41:43.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:43 smithi148 ceph-mon[44520]: 6.a scrub starts 2024-04-23T16:41:43.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:43 smithi148 ceph-mon[44520]: 6.a scrub ok 2024-04-23T16:41:43.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:43 smithi148 ceph-mon[44520]: pgmap v1500: 27 pgs: 27 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 524 op/s 2024-04-23T16:41:43.888 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:41:43.900 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:41:43.928 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:43 smithi103 ceph-mon[39629]: 6.a scrub starts 2024-04-23T16:41:43.928 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:43 smithi103 ceph-mon[39629]: 6.a scrub ok 2024-04-23T16:41:43.928 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:43 smithi103 ceph-mon[39629]: pgmap v1500: 27 pgs: 27 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 524 op/s 2024-04-23T16:41:43.930 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:43.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:43 smithi188 ceph-mon[44101]: 6.a scrub starts 2024-04-23T16:41:43.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:43 smithi188 ceph-mon[44101]: 6.a scrub ok 2024-04-23T16:41:43.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:43 smithi188 ceph-mon[44101]: pgmap v1500: 27 pgs: 27 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 524 op/s 2024-04-23T16:41:44.079 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:41:44.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:44.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:41:44.194 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:41:44.220 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:44.569 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:41:44.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:44.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:41:44.624 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'unique_pool_4' removed 2024-04-23T16:41:44.673 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:41:44.700 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:44.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:44 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T16:41:44.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:44 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T16:41:44.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:44 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2347969289' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:41:44.903 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:41:44.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:44 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T16:41:44.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:44 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T16:41:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:44 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2347969289' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:41:45.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:44 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T16:41:45.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:44 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T16:41:45.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:44 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2347969289' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:41:45.906 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:45 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2347969289' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:41:45.906 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:45 smithi103 ceph-mon[39629]: osdmap e537: 12 total, 12 up, 10 in 2024-04-23T16:41:45.907 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:45 smithi103 ceph-mon[39629]: pgmap v1502: 11 pgs: 11 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:45.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:45 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2347969289' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:41:45.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:45 smithi188 ceph-mon[44101]: osdmap e537: 12 total, 12 up, 10 in 2024-04-23T16:41:45.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:45 smithi188 ceph-mon[44101]: pgmap v1502: 11 pgs: 11 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:46.055 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:41:46.055 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:41:46.056 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-23T16:41:46.056 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-23T16:41:46.056 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:41:46.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:45 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2347969289' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:41:46.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:45 smithi148 ceph-mon[44520]: osdmap e537: 12 total, 12 up, 10 in 2024-04-23T16:41:46.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:45 smithi148 ceph-mon[44520]: pgmap v1502: 11 pgs: 11 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:46.842 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-23T16:41:46.850 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:41:46.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:46.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:41:46.954 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:41:46.981 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:47.229 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:41:47.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:47.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:41:47.333 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:41:47.366 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:47.388 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:41:47.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:47.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:41:47.492 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:41:47.524 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:47.546 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:41:47.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:47.549 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:41:47.650 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:41:47.682 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:47.704 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:41:47.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:47.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:41:47.808 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:41:47.838 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:47.858 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:41:47.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:47.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:41:47.962 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:41:47.979 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-23T16:41:47.980 INFO:tasks.radosbench:Beginning radosbench... 2024-04-23T16:41:47.980 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-04-23T16:41:47.980 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-04-23T16:41:47.989 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:47 smithi148 ceph-mon[44520]: pgmap v1503: 11 pgs: 11 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:47.991 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:48.011 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:41:48.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:48.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:41:48.025 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:41:48.037 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:41:48.115 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:41:48.139 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:47 smithi103 ceph-mon[39629]: pgmap v1503: 11 pgs: 11 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:48.141 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:48.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:47 smithi188 ceph-mon[44101]: pgmap v1503: 11 pgs: 11 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:48.362 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:41:48.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:48.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:41:48.467 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:41:48.495 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:48.843 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:41:48.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:48.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:41:48.947 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:41:48.975 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:48 smithi103 ceph-mon[39629]: 1.4 deep-scrub starts 2024-04-23T16:41:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:48 smithi103 ceph-mon[39629]: 1.4 deep-scrub ok 2024-04-23T16:41:49.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:48 smithi188 ceph-mon[44101]: 1.4 deep-scrub starts 2024-04-23T16:41:49.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:48 smithi188 ceph-mon[44101]: 1.4 deep-scrub ok 2024-04-23T16:41:49.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:48 smithi148 ceph-mon[44520]: 1.4 deep-scrub starts 2024-04-23T16:41:49.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:48 smithi148 ceph-mon[44520]: 1.4 deep-scrub ok 2024-04-23T16:41:49.631 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:41:49.889 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:41:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:49 smithi103 ceph-mon[39629]: pgmap v1504: 11 pgs: 11 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:49 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3858281497' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:41:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:49 smithi103 ceph-mon[39629]: 1.8 scrub starts 2024-04-23T16:41:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:49 smithi103 ceph-mon[39629]: 1.8 scrub ok 2024-04-23T16:41:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:49 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:41:50.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:49 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3858281497' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:41:50.124 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:41:50.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:50.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:41:50.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:49 smithi188 ceph-mon[44101]: pgmap v1504: 11 pgs: 11 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:49 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3858281497' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:41:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:49 smithi188 ceph-mon[44101]: 1.8 scrub starts 2024-04-23T16:41:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:49 smithi188 ceph-mon[44101]: 1.8 scrub ok 2024-04-23T16:41:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:49 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:41:50.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:49 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3858281497' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:41:50.228 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:41:50.257 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:50.309 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:41:50.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:50.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:41:50.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:49 smithi148 ceph-mon[44520]: pgmap v1504: 11 pgs: 11 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:50.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:49 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3858281497' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:41:50.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:49 smithi148 ceph-mon[44520]: 1.8 scrub starts 2024-04-23T16:41:50.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:49 smithi148 ceph-mon[44520]: 1.8 scrub ok 2024-04-23T16:41:50.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:49 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:41:50.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:49 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3858281497' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:41:50.413 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:41:50.446 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:50.467 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:41:50.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:50.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:41:50.571 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:41:50.603 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:50.624 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:41:50.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:50.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:41:50.728 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:41:50.755 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:51.173 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:50 smithi103 ceph-mon[39629]: 1.5 deep-scrub starts 2024-04-23T16:41:51.173 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:50 smithi103 ceph-mon[39629]: 1.5 deep-scrub ok 2024-04-23T16:41:51.174 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:50 smithi103 ceph-mon[39629]: osdmap e538: 12 total, 12 up, 10 in 2024-04-23T16:41:51.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:50 smithi188 ceph-mon[44101]: 1.5 deep-scrub starts 2024-04-23T16:41:51.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:50 smithi188 ceph-mon[44101]: 1.5 deep-scrub ok 2024-04-23T16:41:51.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:50 smithi188 ceph-mon[44101]: osdmap e538: 12 total, 12 up, 10 in 2024-04-23T16:41:51.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:50 smithi148 ceph-mon[44520]: 1.5 deep-scrub starts 2024-04-23T16:41:51.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:50 smithi148 ceph-mon[44520]: 1.5 deep-scrub ok 2024-04-23T16:41:51.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:50 smithi148 ceph-mon[44520]: osdmap e538: 12 total, 12 up, 10 in 2024-04-23T16:41:51.535 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:41:51.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:51.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:41:51.638 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:41:51.670 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:51.692 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:41:51.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:51.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:41:51.797 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:41:51.825 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:51.853 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:41:51.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:51.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:41:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:41:51.886 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:41:51.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:41:51.886 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-23T16:41:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:41:51.957 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:41:51.989 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:52.010 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:41:52.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:52.014 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:41:52.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:51 smithi103 ceph-mon[39629]: pgmap v1506: 11 pgs: 11 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:52.115 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:41:52.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:51 smithi148 ceph-mon[44520]: pgmap v1506: 11 pgs: 11 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:52.143 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:52.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:51 smithi188 ceph-mon[44101]: pgmap v1506: 11 pgs: 11 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:52.393 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:41:52.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:52.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:41:52.497 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:41:52.529 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:52.550 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:41:52.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:52.553 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:41:52.654 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:41:52.687 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:52.709 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:41:52.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:52.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:41:52.813 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:41:52.845 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:52.866 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:41:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:52.869 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:41:52.970 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:41:52.997 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:53.305 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-23T16:41:53.305 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:41:53.678 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-23T16:41:53.962 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:53 smithi103 ceph-mon[39629]: pgmap v1507: 11 pgs: 11 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:54.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:53 smithi188 ceph-mon[44101]: pgmap v1507: 11 pgs: 11 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:54.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:53 smithi148 ceph-mon[44520]: pgmap v1507: 11 pgs: 11 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:54.418 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:41:54.749 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:41:54.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:54.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:41:54.853 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:41:54.886 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:54.907 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:41:54.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:54.911 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:41:55.012 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:41:55.044 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:55.066 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:41:55.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:55.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:41:55.170 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:41:55.199 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:55.285 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:55 smithi103 ceph-mon[39629]: pgmap v1508: 11 pgs: 11 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:55.286 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:55 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3808516731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:41:55.291 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:41:55.299 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:41:55.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:55.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:41:55.321 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:41:55.340 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:41:55.359 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:41:55.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:55 smithi148 ceph-mon[44520]: pgmap v1508: 11 pgs: 11 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:55.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:55 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3808516731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:41:55.379 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:41:55.399 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:41:55.404 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:41:55.420 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:41:55.437 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:55.441 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:41:55.459 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:41:55.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:55.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:41:55.463 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:41:55.482 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:41:55.502 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:41:55.523 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:41:55.524 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:41:55.524 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:41:55.524 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:41:55.524 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:41:55.524 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:41:55.524 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:41:55.525 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:41:55.525 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:41:55.525 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:41:55.525 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:41:55.525 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:41:55.525 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:41:55.564 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:41:55.594 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:55.614 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:41:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:55.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:41:55.719 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:41:55.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:55 smithi188 ceph-mon[44101]: pgmap v1508: 11 pgs: 11 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:55.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:55 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3808516731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:41:55.747 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:55.801 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:41:55.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:55.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:41:55.904 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:41:55.937 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:55.958 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:41:55.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:55.961 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:41:56.062 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:41:56.094 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:56.115 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:41:56.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:56.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:41:56.220 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:41:56.248 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:56.299 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'unique_pool_5' created 2024-04-23T16:41:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:56 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:41:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:56 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:41:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:56 smithi148 ceph-mon[44520]: osdmap e539: 12 total, 12 up, 10 in 2024-04-23T16:41:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:56 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2675733792' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-04-23T16:41:56.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:56 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:41:56.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:56 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:41:56.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:56 smithi103 ceph-mon[39629]: osdmap e539: 12 total, 12 up, 10 in 2024-04-23T16:41:56.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:56 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2675733792' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-04-23T16:41:56.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:56 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:41:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:56 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:41:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:56 smithi188 ceph-mon[44101]: osdmap e539: 12 total, 12 up, 10 in 2024-04-23T16:41:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:56.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:56 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2675733792' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-04-23T16:41:57.176 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-04-23T16:41:57.179 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:41:57.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:57.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:41:57.192 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:41:57.192 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:41:57.194 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:41:57.284 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:41:57.319 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:57.340 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:41:57.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:57.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:41:57.445 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:41:57.472 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:57.472 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:57 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2675733792' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-04-23T16:41:57.472 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:57 smithi103 ceph-mon[39629]: osdmap e540: 12 total, 12 up, 10 in 2024-04-23T16:41:57.473 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:57 smithi103 ceph-mon[39629]: pgmap v1511: 27 pgs: 16 unknown, 11 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:57.477 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:57.498 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:41:57.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:57.502 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:41:57.603 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:41:57.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:57.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:57 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2675733792' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-04-23T16:41:57.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:57 smithi148 ceph-mon[44520]: osdmap e540: 12 total, 12 up, 10 in 2024-04-23T16:41:57.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:57 smithi148 ceph-mon[44520]: pgmap v1511: 27 pgs: 16 unknown, 11 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:57.635 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:57.657 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:41:57.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:41:57.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:41:57.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:41:57.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:57 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2675733792' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-04-23T16:41:57.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:57 smithi188 ceph-mon[44101]: osdmap e540: 12 total, 12 up, 10 in 2024-04-23T16:41:57.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:57 smithi188 ceph-mon[44101]: pgmap v1511: 27 pgs: 16 unknown, 11 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:57.762 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:41:57.789 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:41:58.496 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:41:58.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:58 smithi103 ceph-mon[39629]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:41:58.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:58 smithi103 ceph-mon[39629]: osdmap e541: 12 total, 12 up, 10 in 2024-04-23T16:41:58.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:58 smithi148 ceph-mon[44520]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:41:58.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:58 smithi148 ceph-mon[44520]: osdmap e541: 12 total, 12 up, 10 in 2024-04-23T16:41:58.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:58 smithi188 ceph-mon[44101]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:41:58.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:58 smithi188 ceph-mon[44101]: osdmap e541: 12 total, 12 up, 10 in 2024-04-23T16:41:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:59 smithi103 ceph-mon[39629]: osdmap e542: 12 total, 12 up, 10 in 2024-04-23T16:41:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:41:59 smithi103 ceph-mon[39629]: pgmap v1514: 27 pgs: 1 creating+peering, 15 unknown, 11 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:59.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:59 smithi148 ceph-mon[44520]: osdmap e542: 12 total, 12 up, 10 in 2024-04-23T16:41:59.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:41:59 smithi148 ceph-mon[44520]: pgmap v1514: 27 pgs: 1 creating+peering, 15 unknown, 11 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:41:59.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:59 smithi188 ceph-mon[44101]: osdmap e542: 12 total, 12 up, 10 in 2024-04-23T16:41:59.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:41:59 smithi188 ceph-mon[44101]: pgmap v1514: 27 pgs: 1 creating+peering, 15 unknown, 11 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:42:01.124 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:42:01.124 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:42:01.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:01 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:42:01.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:01 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:42:01.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:01 smithi103 ceph-mon[39629]: osdmap e543: 12 total, 12 up, 10 in 2024-04-23T16:42:01.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:01 smithi103 ceph-mon[39629]: pgmap v1516: 27 pgs: 1 creating+peering, 4 unknown, 22 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:42:01.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:01 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:42:01.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:01 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:42:01.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:01 smithi148 ceph-mon[44520]: osdmap e543: 12 total, 12 up, 10 in 2024-04-23T16:42:01.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:01 smithi148 ceph-mon[44520]: pgmap v1516: 27 pgs: 1 creating+peering, 4 unknown, 22 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:42:01.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:01 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:42:01.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:01 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:42:01.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:01 smithi188 ceph-mon[44101]: osdmap e543: 12 total, 12 up, 10 in 2024-04-23T16:42:01.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:01 smithi188 ceph-mon[44101]: pgmap v1516: 27 pgs: 1 creating+peering, 4 unknown, 22 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:42:01.949 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:42:01.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:01.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:42:02.057 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:42:02.090 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:02.112 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:42:02.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:02.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:42:02.216 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:42:02.247 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:02.321 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:42:02.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:02.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:42:02.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:02 smithi148 ceph-mon[44520]: from='client.16596 v1:172.21.15.103:0/2278390501' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:42:02.426 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:42:02.426 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:42:02.437 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:42:02.466 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:02.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:02 smithi103 ceph-mon[39629]: from='client.16596 v1:172.21.15.103:0/2278390501' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:42:02.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:02 smithi188 ceph-mon[44101]: from='client.16596 v1:172.21.15.103:0/2278390501' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:42:02.814 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:42:02.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:02.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:42:02.842 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1516,"stamp":"2024-04-23T16:42:00.831873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":46,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":5083952,"kb_used_data":342052,"kb_used_omap":245,"kb_used_meta":4741002,"kb_avail":932367568,"statfs":{"total":959950356480,"available":954744389632,"internally_reserved":0,"allocated":350261248,"data_stored":346637744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":251009,"internal_metadata":4854786943},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.540259"},"pg_stats":[{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-23T16:41:58.300776+0000","last_change":"2024-04-23T16:41:57.310289+0000","last_active":"2024-04-23T16:41:58.300776+0000","last_peered":"2024-04-23T16:41:58.300776+0000","last_clean":"2024-04-23T16:41:58.300776+0000","last_became_active":"2024-04-23T16:41:57.310025+0000","last_became_peered":"2024-04-23T16:41:57.310025+0000","last_unstale":"2024-04-23T16:41:58.300776+0000","last_undegraded":"2024-04-23T16:41:58.300776+0000","last_fullsized":"2024-04-23T16:41:58.300776+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:11.505533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-23T16:41:56.284274+0000","last_change":"2024-04-23T16:41:56.284274+0000","last_active":"2024-04-23T16:41:56.284274+0000","last_peered":"2024-04-23T16:41:56.284274+0000","last_clean":"2024-04-23T16:41:56.284274+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T16:41:56.284274+0000","last_undegraded":"2024-04-23T16:41:56.284274+0000","last_fullsized":"2024-04-23T16:41:56.284274+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-23T16:41:58.339129+0000","last_change":"2024-04-23T16:41:57.310470+0000","last_active":"2024-04-23T16:41:58.339129+0000","last_peered":"2024-04-23T16:41:58.339129+0000","last_clean":"2024-04-23T16:41:58.339129+0000","last_became_active":"2024-04-23T16:41:57.310189+0000","last_became_peered":"2024-04-23T16:41:57.310189+0000","last_unstale":"2024-04-23T16:41:58.339129+0000","last_undegraded":"2024-04-23T16:41:58.339129+0000","last_fullsized":"2024-04-23T16:41:58.339129+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:07.365157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-23T16:41:58.338655+0000","last_change":"2024-04-23T16:41:57.309821+0000","last_active":"2024-04-23T16:41:58.338655+0000","last_peered":"2024-04-23T16:41:58.338655+0000","last_clean":"2024-04-23T16:41:58.338655+0000","last_became_active":"2024-04-23T16:41:57.309569+0000","last_became_peered":"2024-04-23T16:41:57.309569+0000","last_unstale":"2024-04-23T16:41:58.338655+0000","last_undegraded":"2024-04-23T16:41:58.338655+0000","last_fullsized":"2024-04-23T16:41:58.338655+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:04.107658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-23T16:41:58.463583+0000","last_change":"2024-04-23T16:41:57.310144+0000","last_active":"2024-04-23T16:41:58.463583+0000","last_peered":"2024-04-23T16:41:58.463583+0000","last_clean":"2024-04-23T16:41:58.463583+0000","last_became_active":"2024-04-23T16:41:57.309841+0000","last_became_peered":"2024-04-23T16:41:57.309841+0000","last_unstale":"2024-04-23T16:41:58.463583+0000","last_undegraded":"2024-04-23T16:41:58.463583+0000","last_fullsized":"2024-04-23T16:41:58.463583+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:02.352775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-23T16:41:58.300668+0000","last_change":"2024-04-23T16:41:57.304298+0000","last_active":"2024-04-23T16:41:58.300668+0000","last_peered":"2024-04-23T16:41:58.300668+0000","last_clean":"2024-04-23T16:41:58.300668+0000","last_became_active":"2024-04-23T16:41:57.304044+0000","last_became_peered":"2024-04-23T16:41:57.304044+0000","last_unstale":"2024-04-23T16:41:58.300668+0000","last_undegraded":"2024-04-23T16:41:58.300668+0000","last_fullsized":"2024-04-23T16:41:58.300668+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:16.798762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1380,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-23T16:41:58.339079+0000","last_change":"2024-04-23T16:41:21.296533+0000","last_active":"2024-04-23T16:41:58.339079+0000","last_peered":"2024-04-23T16:41:58.339079+0000","last_clean":"2024-04-23T16:41:58.339079+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:41:58.339079+0000","last_undegraded":"2024-04-23T16:41:58.339079+0000","last_fullsized":"2024-04-23T16:41:58.339079+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:21.296440+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:41:21.296440+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:42:37.084905+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-23T16:41:58.339134+0000","last_change":"2024-04-23T16:41:57.312048+0000","last_active":"2024-04-23T16:41:58.339134+0000","last_peered":"2024-04-23T16:41:58.339134+0000","last_clean":"2024-04-23T16:41:58.339134+0000","last_became_active":"2024-04-23T16:41:57.311753+0000","last_became_peered":"2024-04-23T16:41:57.311753+0000","last_unstale":"2024-04-23T16:41:58.339134+0000","last_undegraded":"2024-04-23T16:41:58.339134+0000","last_fullsized":"2024-04-23T16:41:58.339134+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:26.029714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1367,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-23T16:41:49.886183+0000","last_change":"2024-04-23T16:41:49.795200+0000","last_active":"2024-04-23T16:41:49.886183+0000","last_peered":"2024-04-23T16:41:49.886183+0000","last_clean":"2024-04-23T16:41:49.886183+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:41:49.886183+0000","last_undegraded":"2024-04-23T16:41:49.886183+0000","last_fullsized":"2024-04-23T16:41:49.886183+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:49.795083+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:41:49.795083+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:18.848065+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-23T16:41:58.338642+0000","last_change":"2024-04-23T16:41:57.306175+0000","last_active":"2024-04-23T16:41:58.338642+0000","last_peered":"2024-04-23T16:41:58.338642+0000","last_clean":"2024-04-23T16:41:58.338642+0000","last_became_active":"2024-04-23T16:41:57.305944+0000","last_became_peered":"2024-04-23T16:41:57.305944+0000","last_unstale":"2024-04-23T16:41:58.338642+0000","last_undegraded":"2024-04-23T16:41:58.338642+0000","last_fullsized":"2024-04-23T16:41:58.338642+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:01.624032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1363,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-23T16:41:58.463494+0000","last_change":"2024-04-23T16:41:14.928958+0000","last_active":"2024-04-23T16:41:58.463494+0000","last_peered":"2024-04-23T16:41:58.463494+0000","last_clean":"2024-04-23T16:41:58.463494+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:41:58.463494+0000","last_undegraded":"2024-04-23T16:41:58.463494+0000","last_fullsized":"2024-04-23T16:41:58.463494+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:40:33.224420+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:40:33.224420+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:42:00.198792+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-23T16:41:58.338639+0000","last_change":"2024-04-23T16:41:57.310022+0000","last_active":"2024-04-23T16:41:58.338639+0000","last_peered":"2024-04-23T16:41:58.338639+0000","last_clean":"2024-04-23T16:41:58.338639+0000","last_became_active":"2024-04-23T16:41:57.309858+0000","last_became_peered":"2024-04-23T16:41:57.309858+0000","last_unstale":"2024-04-23T16:41:58.338639+0000","last_undegraded":"2024-04-23T16:41:58.338639+0000","last_fullsized":"2024-04-23T16:41:58.338639+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:18.961985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1380,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-23T16:41:58.338511+0000","last_change":"2024-04-23T16:40:32.634479+0000","last_active":"2024-04-23T16:41:58.338511+0000","last_peered":"2024-04-23T16:41:58.338511+0000","last_clean":"2024-04-23T16:41:58.338511+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:41:58.338511+0000","last_undegraded":"2024-04-23T16:41:58.338511+0000","last_fullsized":"2024-04-23T16:41:58.338511+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:40:32.634336+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:40:32.634336+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:42:02.102624+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":540,"state":"creating+peering","last_fresh":"2024-04-23T16:41:56.296803+0000","last_change":"2024-04-23T16:41:56.296803+0000","last_active":"2024-04-23T16:41:56.284274+0000","last_peered":"2024-04-23T16:41:56.284274+0000","last_clean":"2024-04-23T16:41:56.284274+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T16:41:56.296803+0000","last_undegraded":"2024-04-23T16:41:56.296803+0000","last_fullsized":"2024-04-23T16:41:56.296803+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":945,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-23T16:41:51.445801+0000","last_change":"2024-04-23T16:41:35.951699+0000","last_active":"2024-04-23T16:41:51.445801+0000","last_peered":"2024-04-23T16:41:51.445801+0000","last_clean":"2024-04-23T16:41:51.445801+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:41:51.445801+0000","last_undegraded":"2024-04-23T16:41:51.445801+0000","last_fullsized":"2024-04-23T16:41:51.445801+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:35.951597+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:41:35.951597+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:42:51.648114+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-23T16:41:58.300710+0000","last_change":"2024-04-23T16:41:57.306526+0000","last_active":"2024-04-23T16:41:58.300710+0000","last_peered":"2024-04-23T16:41:58.300710+0000","last_clean":"2024-04-23T16:41:58.300710+0000","last_became_active":"2024-04-23T16:41:57.306229+0000","last_became_peered":"2024-04-23T16:41:57.306229+0000","last_unstale":"2024-04-23T16:41:58.300710+0000","last_undegraded":"2024-04-23T16:41:58.300710+0000","last_fullsized":"2024-04-23T16:41:58.300710+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:01.861172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":905,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-04-23T16:41:57.304899+0000","last_change":"2024-04-23T16:41:24.262727+0000","last_active":"2024-04-23T16:41:57.304899+0000","last_peered":"2024-04-23T16:41:57.304899+0000","last_clean":"2024-04-23T16:41:57.304899+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:41:57.304899+0000","last_undegraded":"2024-04-23T16:41:57.304899+0000","last_fullsized":"2024-04-23T16:41:57.304899+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:24.262646+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:41:24.262646+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:42:32.402472+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-23T16:41:56.284274+0000","last_change":"2024-04-23T16:41:56.284274+0000","last_active":"2024-04-23T16:41:56.284274+0000","last_peered":"2024-04-23T16:41:56.284274+0000","last_clean":"2024-04-23T16:41:56.284274+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T16:41:56.284274+0000","last_undegraded":"2024-04-23T16:41:56.284274+0000","last_fullsized":"2024-04-23T16:41:56.284274+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1388,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-23T16:41:58.339067+0000","last_change":"2024-04-23T16:41:43.486498+0000","last_active":"2024-04-23T16:41:58.339067+0000","last_peered":"2024-04-23T16:41:58.339067+0000","last_clean":"2024-04-23T16:41:58.339067+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:41:58.339067+0000","last_undegraded":"2024-04-23T16:41:58.339067+0000","last_fullsized":"2024-04-23T16:41:58.339067+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:43.486391+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:41:43.486391+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:42:47.286490+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-23T16:41:58.338913+0000","last_change":"2024-04-23T16:41:57.310232+0000","last_active":"2024-04-23T16:41:58.338913+0000","last_peered":"2024-04-23T16:41:58.338913+0000","last_clean":"2024-04-23T16:41:58.338913+0000","last_became_active":"2024-04-23T16:41:57.309953+0000","last_became_peered":"2024-04-23T16:41:57.309953+0000","last_unstale":"2024-04-23T16:41:58.338913+0000","last_undegraded":"2024-04-23T16:41:58.338913+0000","last_fullsized":"2024-04-23T16:41:58.338913+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:14.320456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1373,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-23T16:41:58.300664+0000","last_change":"2024-04-23T16:40:23.268930+0000","last_active":"2024-04-23T16:41:58.300664+0000","last_peered":"2024-04-23T16:41:58.300664+0000","last_clean":"2024-04-23T16:41:58.300664+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:41:58.300664+0000","last_undegraded":"2024-04-23T16:41:58.300664+0000","last_fullsized":"2024-04-23T16:41:58.300664+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:40:23.268803+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:40:23.268803+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-23T16:41:56.284274+0000","last_change":"2024-04-23T16:41:56.284274+0000","last_active":"2024-04-23T16:41:56.284274+0000","last_peered":"2024-04-23T16:41:56.284274+0000","last_clean":"2024-04-23T16:41:56.284274+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T16:41:56.284274+0000","last_undegraded":"2024-04-23T16:41:56.284274+0000","last_fullsized":"2024-04-23T16:41:56.284274+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1276,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-04-23T16:41:56.297071+0000","last_change":"2024-04-23T16:41:47.722866+0000","last_active":"2024-04-23T16:41:56.297071+0000","last_peered":"2024-04-23T16:41:56.297071+0000","last_clean":"2024-04-23T16:41:56.297071+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:41:56.297071+0000","last_undegraded":"2024-04-23T16:41:56.297071+0000","last_fullsized":"2024-04-23T16:41:56.297071+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:41:47.722762+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:03.039468+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-23T16:41:56.284274+0000","last_change":"2024-04-23T16:41:56.284274+0000","last_active":"2024-04-23T16:41:56.284274+0000","last_peered":"2024-04-23T16:41:56.284274+0000","last_clean":"2024-04-23T16:41:56.284274+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T16:41:56.284274+0000","last_undegraded":"2024-04-23T16:41:56.284274+0000","last_fullsized":"2024-04-23T16:41:56.284274+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1202,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-23T16:41:58.300681+0000","last_change":"2024-04-23T16:41:49.835121+0000","last_active":"2024-04-23T16:41:58.300681+0000","last_peered":"2024-04-23T16:41:58.300681+0000","last_clean":"2024-04-23T16:41:58.300681+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:41:58.300681+0000","last_undegraded":"2024-04-23T16:41:58.300681+0000","last_fullsized":"2024-04-23T16:41:58.300681+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:41:49.834374+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:42:56.109043+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-23T16:41:58.338650+0000","last_change":"2024-04-23T16:41:57.309579+0000","last_active":"2024-04-23T16:41:58.338650+0000","last_peered":"2024-04-23T16:41:58.338650+0000","last_clean":"2024-04-23T16:41:58.338650+0000","last_became_active":"2024-04-23T16:41:57.309349+0000","last_became_peered":"2024-04-23T16:41:57.309349+0000","last_unstale":"2024-04-23T16:41:58.338650+0000","last_undegraded":"2024-04-23T16:41:58.338650+0000","last_fullsized":"2024-04-23T16:41:58.338650+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:42:57.356837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1363,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-23T16:41:58.463574+0000","last_change":"2024-04-23T16:40:35.156593+0000","last_active":"2024-04-23T16:41:58.463574+0000","last_peered":"2024-04-23T16:41:58.463574+0000","last_clean":"2024-04-23T16:41:58.463574+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:41:58.463574+0000","last_undegraded":"2024-04-23T16:41:58.463574+0000","last_fullsized":"2024-04-23T16:41:58.463574+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:40:35.156484+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:40:35.156484+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:42:02.492809+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1269760,"data_stored":1259104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056960,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":533472,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93211680,"statfs":{"total":95995035648,"available":95448760320,"internally_reserved":0,"allocated":1007616,"data_stored":666357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":545126852},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351302,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612940,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":437376,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93307776,"statfs":{"total":95995035648,"available":95547162624,"internally_reserved":0,"allocated":704512,"data_stored":351518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":447019456},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841872,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":334960,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93410192,"statfs":{"total":95995035648,"available":95652036608,"internally_reserved":0,"allocated":1019904,"data_stored":672650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168917,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545540,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199612,"statfs":{"total":95995035648,"available":95436402688,"internally_reserved":0,"allocated":704512,"data_stored":351518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":557906378},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":374788,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93370364,"statfs":{"total":95995035648,"available":95611252736,"internally_reserved":0,"allocated":704512,"data_stored":351518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":383056326},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137399,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":195268,"kb_used_data":760,"kb_used_omap":231,"kb_used_meta":194264,"kb_avail":93549884,"statfs":{"total":95995035648,"available":95795081216,"internally_reserved":0,"allocated":778240,"data_stored":390601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":198927248},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738745,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":509444,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93235708,"statfs":{"total":95995035648,"available":95473364992,"internally_reserved":0,"allocated":704512,"data_stored":351518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269415,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117352,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":736584,"kb_used_data":182016,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":93008568,"statfs":{"total":95995035648,"available":95240773632,"internally_reserved":0,"allocated":186384384,"data_stored":185993193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":567867847},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921123,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653952,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93091200,"statfs":{"total":95995035648,"available":95325388800,"internally_reserved":0,"allocated":704512,"data_stored":351518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757597,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":762568,"kb_used_data":153856,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":92982584,"statfs":{"total":95995035648,"available":95214166016,"internally_reserved":0,"allocated":157548544,"data_stored":157157353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:42:02.843 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:42:02.917 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:42:02.951 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:03.441 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:03 smithi103 ceph-mon[39629]: pgmap v1517: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:42:03.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:03 smithi148 ceph-mon[44520]: pgmap v1517: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:42:03.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:03 smithi188 ceph-mon[44101]: pgmap v1517: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:42:03.827 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:42:03.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:03.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:42:03.843 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-23T16:42:03.932 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:42:03.964 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:03.985 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:42:03.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:03.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:42:04.068 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:42:04.068 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:42:04.089 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:42:04.129 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:42:04.130 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:04.384 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:04 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:42:04.384 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:04 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:42:04.384 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:04 smithi103 ceph-mon[39629]: osdmap e544: 12 total, 12 up, 10 in 2024-04-23T16:42:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:04 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:42:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:04 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:42:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:04 smithi148 ceph-mon[44520]: osdmap e544: 12 total, 12 up, 10 in 2024-04-23T16:42:04.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:04 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:42:04.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:04 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:42:04.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:04 smithi188 ceph-mon[44101]: osdmap e544: 12 total, 12 up, 10 in 2024-04-23T16:42:05.339 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:42:05.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:05 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:42:05.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:05 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:42:05.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:05 smithi103 ceph-mon[39629]: osdmap e545: 12 total, 12 up, 10 in 2024-04-23T16:42:05.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:05 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2434980756' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:42:05.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:05 smithi103 ceph-mon[39629]: from='client.26563 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:42:05.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:05 smithi103 ceph-mon[39629]: pgmap v1520: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:42:05.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:05 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:42:05.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:05 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:42:05.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:05 smithi148 ceph-mon[44520]: osdmap e545: 12 total, 12 up, 10 in 2024-04-23T16:42:05.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:05 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2434980756' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:42:05.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:05 smithi148 ceph-mon[44520]: from='client.26563 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:42:05.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:05 smithi148 ceph-mon[44520]: pgmap v1520: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:42:05.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:05 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:42:05.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:05 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:42:05.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:05 smithi188 ceph-mon[44101]: osdmap e545: 12 total, 12 up, 10 in 2024-04-23T16:42:05.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:05 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2434980756' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:42:05.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:05 smithi188 ceph-mon[44101]: from='client.26563 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:42:05.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:05 smithi188 ceph-mon[44101]: pgmap v1520: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:42:06.283 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:42:06.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:06.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:42:06.388 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:42:06.418 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:06.457 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:42:06.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:06.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:42:06.560 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:42:06.591 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:06 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:42:06.591 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:06 smithi148 ceph-mon[44520]: from='client.26563 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:42:06.591 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:06 smithi148 ceph-mon[44520]: osdmap e546: 12 total, 12 up, 10 in 2024-04-23T16:42:06.593 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:06.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:06 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:42:06.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:06 smithi103 ceph-mon[39629]: from='client.26563 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:42:06.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:06 smithi103 ceph-mon[39629]: osdmap e546: 12 total, 12 up, 10 in 2024-04-23T16:42:06.615 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:42:06.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:06.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:42:06.719 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:42:06.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:06 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:42:06.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:06 smithi188 ceph-mon[44101]: from='client.26563 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:42:06.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:06 smithi188 ceph-mon[44101]: osdmap e546: 12 total, 12 up, 10 in 2024-04-23T16:42:06.751 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:06.772 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:42:06.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:06.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:42:06.877 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:42:06.907 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:06.945 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:42:06.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:06.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:42:07.049 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:42:07.080 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:07.101 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:42:07.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:07.104 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:42:07.205 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:42:07.238 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:07.260 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:42:07.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:07.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:42:07.365 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:42:07.396 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:07.417 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:42:07.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:07.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:42:07.458 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:07 smithi103 ceph-mon[39629]: pgmap v1522: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:42:07.520 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:42:07.546 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:07 smithi148 ceph-mon[44520]: pgmap v1522: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:42:07.548 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:07.568 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:42:07.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:07.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:42:07.673 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:42:07.698 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:07 smithi188 ceph-mon[44101]: pgmap v1522: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:42:07.700 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:07.719 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:42:07.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:07.722 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:42:07.823 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:42:07.850 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:07.872 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:42:07.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:07.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:42:07.976 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:42:08.001 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:08.021 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:42:08.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:08.025 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:42:08.126 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:42:08.157 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:08.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:08 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:42:08.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:08 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:42:08.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:08 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:42:08.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:08 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:42:08.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:08 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:42:08.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:08 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/45506949' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:42:08.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:08.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:08 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:42:08.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:08 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:42:08.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:08 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:42:08.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:08 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:42:08.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:08 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:42:08.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:08 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/45506949' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:42:08.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:08 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:08.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:08 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:42:08.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:08 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:42:08.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:08 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:42:08.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:08 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:42:08.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:08 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:42:08.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:08 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/45506949' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:42:08.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:08 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:08.807 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:42:08.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:08.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:42:08.911 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:42:08.942 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:08.964 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:42:08.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:08.967 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:42:09.068 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:42:09.096 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:09.359 INFO:teuthology.orchestra.run.smithi103.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-04-23T16:42:09.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:09 smithi103 ceph-mon[39629]: pgmap v1523: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:42:09.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:09 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/77566076' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:42:09.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:09 smithi103 ceph-mon[39629]: from='client.26575 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:42:09.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:09 smithi148 ceph-mon[44520]: pgmap v1523: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:42:09.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:09 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/77566076' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:42:09.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:09 smithi148 ceph-mon[44520]: from='client.26575 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:42:09.652 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:42:09.652 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:42:09.653 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:42:09.653 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:42:09.653 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:42:09.653 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:42:09.653 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:42:09.653 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:42:09.653 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:42:09.653 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:42:09.653 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:42:09.653 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:42:09.653 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:42:09.653 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:42:09.653 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:42:09.653 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:42:09.654 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:42:09.654 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:42:09.654 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:42:09.654 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:42:09.654 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:42:09.654 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:42:09.654 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:42:09.654 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:42:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:09.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:09 smithi188 ceph-mon[44101]: pgmap v1523: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:42:09.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:09 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/77566076' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:42:09.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:09 smithi188 ceph-mon[44101]: from='client.26575 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:42:10.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:10 smithi103 ceph-mon[39629]: from='client.26575 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:42:10.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:10 smithi103 ceph-mon[39629]: osdmap e547: 12 total, 12 up, 10 in 2024-04-23T16:42:10.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:10 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:42:10.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:10 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:42:10.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:10 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:42:10.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:10 smithi148 ceph-mon[44520]: from='client.26575 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:42:10.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:10 smithi148 ceph-mon[44520]: osdmap e547: 12 total, 12 up, 10 in 2024-04-23T16:42:10.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:10 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:42:10.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:10 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:42:10.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:10 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:42:10.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:10 smithi188 ceph-mon[44101]: from='client.26575 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:42:10.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:10 smithi188 ceph-mon[44101]: osdmap e547: 12 total, 12 up, 10 in 2024-04-23T16:42:10.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:10 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:42:10.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:10 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:42:10.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:10 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:42:11.347 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:42:11.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:11.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:42:11.359 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:42:11.379 DEBUG:teuthology.orchestra.run.smithi188:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-04-23T16:42:11.381 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-23T16:42:11.382 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-23T16:42:11.412 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:42:11.453 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:42:11.483 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:11.527 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:hints = 1 2024-04-23T16:42:11.527 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-23T16:42:11.527 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Object prefix: benchmark_data_smithi188_141483 2024-04-23T16:42:11.527 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:42:11.527 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 0 0 0 0 0 0 - 0 2024-04-23T16:42:11.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:11 smithi103 ceph-mon[39629]: pgmap v1525: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:42:11.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:11 smithi148 ceph-mon[44520]: pgmap v1525: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:42:11.633 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:42:11.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:11.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:42:11.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:11 smithi188 ceph-mon[44101]: pgmap v1525: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:42:11.744 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:42:11.783 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:11.813 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:42:11.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:11.817 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:42:11.918 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:42:11.966 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:12.082 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:42:12.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:12.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:42:12.188 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:42:12.226 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:12.254 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:42:12.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:12.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:42:12.359 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:42:12.400 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:12.431 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:42:12.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:12.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:42:12.527 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 1 16 6047 6031 376.918 376.938 0.0017901 0.00264718 2024-04-23T16:42:12.537 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:42:12.592 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:12.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:12 smithi148 ceph-mon[44520]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:42:12.649 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:42:12.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:12.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:42:12.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:12 smithi103 ceph-mon[39629]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:42:12.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:12 smithi188 ceph-mon[44101]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:42:12.755 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:42:12.793 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:12.850 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:42:12.899 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:42:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:12.906 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:42:13.007 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:42:13.046 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:13.076 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:42:13.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:13.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:42:13.111 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:42:13.128 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:42:13.181 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:42:13.216 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:13.247 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:42:13.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:13.250 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:42:13.351 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:42:13.368 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:13 smithi148 ceph-mon[44520]: pgmap v1526: 27 pgs: 27 active+clean; 9.5 MiB data, 4.8 GiB used, 889 GiB / 894 GiB avail; 1.1 MiB/s wr, 17 op/s 2024-04-23T16:42:13.389 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:13.415 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:42:13.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:13.418 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:42:13.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:13 smithi188 ceph-mon[44101]: pgmap v1526: 27 pgs: 27 active+clean; 9.5 MiB data, 4.8 GiB used, 889 GiB / 894 GiB avail; 1.1 MiB/s wr, 17 op/s 2024-04-23T16:42:13.520 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:42:13.527 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 2 16 11156 11140 348.101 319.312 0.00137667 0.00286763 2024-04-23T16:42:13.555 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:13.582 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:42:13.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:13.586 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:42:13.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:13 smithi103 ceph-mon[39629]: pgmap v1526: 27 pgs: 27 active+clean; 9.5 MiB data, 4.8 GiB used, 889 GiB / 894 GiB avail; 1.1 MiB/s wr, 17 op/s 2024-04-23T16:42:13.687 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:42:13.729 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:13.763 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:42:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:13.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:42:13.868 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:42:13.912 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:14.527 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 3 16 16258 16242 338.351 318.875 0.00193254 0.00295271 2024-04-23T16:42:14.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:14 smithi188 ceph-mon[44101]: from='client.36215 v1:172.21.15.103:0/4202723613' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:42:14.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:14 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:42:14.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:14 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:42:14.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:14 smithi103 ceph-mon[39629]: from='client.36215 v1:172.21.15.103:0/4202723613' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:42:14.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:14 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:42:14.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:14 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:42:14.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:14 smithi148 ceph-mon[44520]: from='client.36215 v1:172.21.15.103:0/4202723613' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:42:14.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:14 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:42:14.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:14 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:42:15.279 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:42:15.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:15.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:42:15.385 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:42:15.426 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:15.528 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 4 16 21051 21035 328.647 299.562 0.00684311 0.00304096 2024-04-23T16:42:15.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:15 smithi103 ceph-mon[39629]: osdmap e548: 12 total, 12 up, 10 in 2024-04-23T16:42:15.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:15 smithi103 ceph-mon[39629]: pgmap v1528: 27 pgs: 27 active+clean; 555 MiB data, 5.5 GiB used, 889 GiB / 894 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-04-23T16:42:15.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:15 smithi188 ceph-mon[44101]: osdmap e548: 12 total, 12 up, 10 in 2024-04-23T16:42:15.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:15 smithi188 ceph-mon[44101]: pgmap v1528: 27 pgs: 27 active+clean; 555 MiB data, 5.5 GiB used, 889 GiB / 894 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-04-23T16:42:15.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:15 smithi148 ceph-mon[44520]: osdmap e548: 12 total, 12 up, 10 in 2024-04-23T16:42:15.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:15 smithi148 ceph-mon[44520]: pgmap v1528: 27 pgs: 27 active+clean; 555 MiB data, 5.5 GiB used, 889 GiB / 894 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-04-23T16:42:16.390 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:42:16.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:16.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:42:16.462 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:42:16.495 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:42:16.528 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 5 16 25025 25009 312.545 248.375 0.00467055 0.003197 2024-04-23T16:42:16.535 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:16.562 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:42:16.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:16.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:42:16.668 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:42:16.715 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:16.750 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:42:16.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:16.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:42:16.866 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:42:16.907 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:16.939 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:42:16.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:16.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:42:17.046 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:42:17.093 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:17.118 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:42:17.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:17.122 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:42:17.223 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:42:17.262 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:17.528 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 6 16 28844 28828 300.233 238.688 0.00242819 0.00332914 2024-04-23T16:42:18.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:17 smithi188 ceph-mon[44101]: pgmap v1529: 27 pgs: 27 active+clean; 1.0 GiB data, 6.5 GiB used, 887 GiB / 894 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-04-23T16:42:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:17 smithi103 ceph-mon[39629]: pgmap v1529: 27 pgs: 27 active+clean; 1.0 GiB data, 6.5 GiB used, 887 GiB / 894 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-04-23T16:42:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:17 smithi148 ceph-mon[44520]: pgmap v1529: 27 pgs: 27 active+clean; 1.0 GiB data, 6.5 GiB used, 887 GiB / 894 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-04-23T16:42:18.528 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 7 16 32716 32700 291.913 242 0.00198908 0.00342411 2024-04-23T16:42:18.900 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:18 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:42:18.900 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:18 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:42:18.900 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:18 smithi103 ceph-mon[39629]: osdmap e549: 12 total, 12 up, 10 in 2024-04-23T16:42:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:18 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:42:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:18 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:42:19.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:18 smithi188 ceph-mon[44101]: osdmap e549: 12 total, 12 up, 10 in 2024-04-23T16:42:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:18 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:42:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:18 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:42:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:18 smithi148 ceph-mon[44520]: osdmap e549: 12 total, 12 up, 10 in 2024-04-23T16:42:19.529 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 8 16 35710 35694 278.78 187.125 0.00383391 0.00358503 2024-04-23T16:42:20.007 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1526,"stamp":"2024-04-23T16:42:12.834498+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9962016,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":143,"num_write_kb":9152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1672,"ondisk_log_size":1672,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":5026764,"kb_used_data":285388,"kb_used_omap":245,"kb_used_meta":4741002,"kb_avail":932424756,"statfs":{"total":959950356480,"available":954802950144,"internally_reserved":0,"allocated":292237312,"data_stored":288430518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":251009,"internal_metadata":4854786943},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9371648,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":143,"num_write_kb":9152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001869"},"pg_stats":[{"pgid":"7.d","version":"0'0","reported_seq":20,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-04-23T16:42:05.337889+0000","last_change":"2024-04-23T16:41:57.310289+0000","last_active":"2024-04-23T16:42:05.337889+0000","last_peered":"2024-04-23T16:42:05.337889+0000","last_clean":"2024-04-23T16:42:05.337889+0000","last_became_active":"2024-04-23T16:41:57.310025+0000","last_became_peered":"2024-04-23T16:41:57.310025+0000","last_unstale":"2024-04-23T16:42:05.337889+0000","last_undegraded":"2024-04-23T16:42:05.337889+0000","last_fullsized":"2024-04-23T16:42:05.337889+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:11.505533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":23,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-23T16:42:09.539187+0000","last_change":"2024-04-23T16:41:57.310978+0000","last_active":"2024-04-23T16:42:09.539187+0000","last_peered":"2024-04-23T16:42:09.539187+0000","last_clean":"2024-04-23T16:42:09.539187+0000","last_became_active":"2024-04-23T16:41:57.310754+0000","last_became_peered":"2024-04-23T16:41:57.310754+0000","last_unstale":"2024-04-23T16:42:09.539187+0000","last_undegraded":"2024-04-23T16:42:09.539187+0000","last_fullsized":"2024-04-23T16:42:09.539187+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:00.884563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":20,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-04-23T16:42:05.874206+0000","last_change":"2024-04-23T16:41:57.310470+0000","last_active":"2024-04-23T16:42:05.874206+0000","last_peered":"2024-04-23T16:42:05.874206+0000","last_clean":"2024-04-23T16:42:05.874206+0000","last_became_active":"2024-04-23T16:41:57.310189+0000","last_became_peered":"2024-04-23T16:41:57.310189+0000","last_unstale":"2024-04-23T16:42:05.874206+0000","last_undegraded":"2024-04-23T16:42:05.874206+0000","last_fullsized":"2024-04-23T16:42:05.874206+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:07.365157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":22,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-23T16:42:09.348265+0000","last_change":"2024-04-23T16:41:57.309821+0000","last_active":"2024-04-23T16:42:09.348265+0000","last_peered":"2024-04-23T16:42:09.348265+0000","last_clean":"2024-04-23T16:42:09.348265+0000","last_became_active":"2024-04-23T16:41:57.309569+0000","last_became_peered":"2024-04-23T16:41:57.309569+0000","last_unstale":"2024-04-23T16:42:09.348265+0000","last_undegraded":"2024-04-23T16:42:09.348265+0000","last_fullsized":"2024-04-23T16:42:09.348265+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:07.958497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":23,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-23T16:42:09.449425+0000","last_change":"2024-04-23T16:41:57.310144+0000","last_active":"2024-04-23T16:42:09.449425+0000","last_peered":"2024-04-23T16:42:09.449425+0000","last_clean":"2024-04-23T16:42:09.449425+0000","last_became_active":"2024-04-23T16:41:57.309841+0000","last_became_peered":"2024-04-23T16:41:57.309841+0000","last_unstale":"2024-04-23T16:42:09.449425+0000","last_undegraded":"2024-04-23T16:42:09.449425+0000","last_fullsized":"2024-04-23T16:42:09.449425+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:04.197064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":20,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-04-23T16:42:05.337873+0000","last_change":"2024-04-23T16:41:57.304298+0000","last_active":"2024-04-23T16:42:05.337873+0000","last_peered":"2024-04-23T16:42:05.337873+0000","last_clean":"2024-04-23T16:42:05.337873+0000","last_became_active":"2024-04-23T16:41:57.304044+0000","last_became_peered":"2024-04-23T16:41:57.304044+0000","last_unstale":"2024-04-23T16:42:05.337873+0000","last_undegraded":"2024-04-23T16:42:05.337873+0000","last_fullsized":"2024-04-23T16:42:05.337873+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:16.798762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1388,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-04-23T16:42:05.874250+0000","last_change":"2024-04-23T16:41:21.296533+0000","last_active":"2024-04-23T16:42:05.874250+0000","last_peered":"2024-04-23T16:42:05.874250+0000","last_clean":"2024-04-23T16:42:05.874250+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:42:05.874250+0000","last_undegraded":"2024-04-23T16:42:05.874250+0000","last_fullsized":"2024-04-23T16:42:05.874250+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:21.296440+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:41:21.296440+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:42:37.084905+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":20,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-04-23T16:42:05.874013+0000","last_change":"2024-04-23T16:41:57.312048+0000","last_active":"2024-04-23T16:42:05.874013+0000","last_peered":"2024-04-23T16:42:05.874013+0000","last_clean":"2024-04-23T16:42:05.874013+0000","last_became_active":"2024-04-23T16:41:57.311753+0000","last_became_peered":"2024-04-23T16:41:57.311753+0000","last_unstale":"2024-04-23T16:42:05.874013+0000","last_undegraded":"2024-04-23T16:42:05.874013+0000","last_fullsized":"2024-04-23T16:42:05.874013+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:26.029714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1385,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-23T16:42:09.352869+0000","last_change":"2024-04-23T16:41:49.795200+0000","last_active":"2024-04-23T16:42:09.352869+0000","last_peered":"2024-04-23T16:42:09.352869+0000","last_clean":"2024-04-23T16:42:09.352869+0000","last_became_active":"2024-04-23T16:33:25.845152+0000","last_became_peered":"2024-04-23T16:33:25.845152+0000","last_unstale":"2024-04-23T16:42:09.352869+0000","last_undegraded":"2024-04-23T16:42:09.352869+0000","last_fullsized":"2024-04-23T16:42:09.352869+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:49.795083+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:41:49.795083+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:18.848065+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":22,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-23T16:42:09.348219+0000","last_change":"2024-04-23T16:41:57.306175+0000","last_active":"2024-04-23T16:42:09.348219+0000","last_peered":"2024-04-23T16:42:09.348219+0000","last_clean":"2024-04-23T16:42:09.348219+0000","last_became_active":"2024-04-23T16:41:57.305944+0000","last_became_peered":"2024-04-23T16:41:57.305944+0000","last_unstale":"2024-04-23T16:42:09.348219+0000","last_undegraded":"2024-04-23T16:42:09.348219+0000","last_fullsized":"2024-04-23T16:42:09.348219+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:15.109223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1373,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-23T16:42:09.449391+0000","last_change":"2024-04-23T16:41:14.928958+0000","last_active":"2024-04-23T16:42:09.449391+0000","last_peered":"2024-04-23T16:42:09.449391+0000","last_clean":"2024-04-23T16:42:09.449391+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:42:09.449391+0000","last_undegraded":"2024-04-23T16:42:09.449391+0000","last_fullsized":"2024-04-23T16:42:09.449391+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:40:33.224420+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:40:33.224420+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:42:14.369869+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":22,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-23T16:42:09.348180+0000","last_change":"2024-04-23T16:41:57.310022+0000","last_active":"2024-04-23T16:42:09.348180+0000","last_peered":"2024-04-23T16:42:09.348180+0000","last_clean":"2024-04-23T16:42:09.348180+0000","last_became_active":"2024-04-23T16:41:57.309858+0000","last_became_peered":"2024-04-23T16:41:57.309858+0000","last_unstale":"2024-04-23T16:42:09.348180+0000","last_undegraded":"2024-04-23T16:42:09.348180+0000","last_fullsized":"2024-04-23T16:42:09.348180+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:03.255644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1389,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-23T16:42:09.348134+0000","last_change":"2024-04-23T16:40:32.634479+0000","last_active":"2024-04-23T16:42:09.348134+0000","last_peered":"2024-04-23T16:42:09.348134+0000","last_clean":"2024-04-23T16:42:09.348134+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:42:09.348134+0000","last_undegraded":"2024-04-23T16:42:09.348134+0000","last_fullsized":"2024-04-23T16:42:09.348134+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:40:32.634336+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:40:32.634336+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"547'143","reported_seq":166,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-23T16:42:11.873504+0000","last_change":"2024-04-23T16:41:57.310695+0000","last_active":"2024-04-23T16:42:11.873504+0000","last_peered":"2024-04-23T16:42:11.873504+0000","last_clean":"2024-04-23T16:42:11.873504+0000","last_became_active":"2024-04-23T16:41:57.310400+0000","last_became_peered":"2024-04-23T16:41:57.310400+0000","last_unstale":"2024-04-23T16:42:11.873504+0000","last_undegraded":"2024-04-23T16:42:11.873504+0000","last_fullsized":"2024-04-23T16:42:11.873504+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:20.482863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9371648,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":143,"num_write_kb":9152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":963,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-23T16:42:09.539123+0000","last_change":"2024-04-23T16:41:35.951699+0000","last_active":"2024-04-23T16:42:09.539123+0000","last_peered":"2024-04-23T16:42:09.539123+0000","last_clean":"2024-04-23T16:42:09.539123+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:42:09.539123+0000","last_undegraded":"2024-04-23T16:42:09.539123+0000","last_fullsized":"2024-04-23T16:42:09.539123+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:35.951597+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:41:35.951597+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:42:51.648114+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":20,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-04-23T16:42:05.337938+0000","last_change":"2024-04-23T16:41:57.306526+0000","last_active":"2024-04-23T16:42:05.337938+0000","last_peered":"2024-04-23T16:42:05.337938+0000","last_clean":"2024-04-23T16:42:05.337938+0000","last_became_active":"2024-04-23T16:41:57.306229+0000","last_became_peered":"2024-04-23T16:41:57.306229+0000","last_unstale":"2024-04-23T16:42:05.337938+0000","last_undegraded":"2024-04-23T16:42:05.337938+0000","last_fullsized":"2024-04-23T16:42:05.337938+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:01.861172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":915,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-04-23T16:42:05.333062+0000","last_change":"2024-04-23T16:41:24.262727+0000","last_active":"2024-04-23T16:42:05.333062+0000","last_peered":"2024-04-23T16:42:05.333062+0000","last_clean":"2024-04-23T16:42:05.333062+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:42:05.333062+0000","last_undegraded":"2024-04-23T16:42:05.333062+0000","last_fullsized":"2024-04-23T16:42:05.333062+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:24.262646+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:41:24.262646+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:42:32.402472+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":23,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-23T16:42:09.353047+0000","last_change":"2024-04-23T16:41:57.311773+0000","last_active":"2024-04-23T16:42:09.353047+0000","last_peered":"2024-04-23T16:42:09.353047+0000","last_clean":"2024-04-23T16:42:09.353047+0000","last_became_active":"2024-04-23T16:41:57.311479+0000","last_became_peered":"2024-04-23T16:41:57.311479+0000","last_unstale":"2024-04-23T16:42:09.353047+0000","last_undegraded":"2024-04-23T16:42:09.353047+0000","last_fullsized":"2024-04-23T16:42:09.353047+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:25.544513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1396,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-04-23T16:42:05.874117+0000","last_change":"2024-04-23T16:41:43.486498+0000","last_active":"2024-04-23T16:42:05.874117+0000","last_peered":"2024-04-23T16:42:05.874117+0000","last_clean":"2024-04-23T16:42:05.874117+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:42:05.874117+0000","last_undegraded":"2024-04-23T16:42:05.874117+0000","last_fullsized":"2024-04-23T16:42:05.874117+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:43.486391+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:41:43.486391+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:42:47.286490+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":22,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-23T16:42:09.973869+0000","last_change":"2024-04-23T16:41:57.310232+0000","last_active":"2024-04-23T16:42:09.973869+0000","last_peered":"2024-04-23T16:42:09.973869+0000","last_clean":"2024-04-23T16:42:09.973869+0000","last_became_active":"2024-04-23T16:41:57.309953+0000","last_became_peered":"2024-04-23T16:41:57.309953+0000","last_unstale":"2024-04-23T16:42:09.973869+0000","last_undegraded":"2024-04-23T16:42:09.973869+0000","last_fullsized":"2024-04-23T16:42:09.973869+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:08.270051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1381,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-04-23T16:42:05.338014+0000","last_change":"2024-04-23T16:40:23.268930+0000","last_active":"2024-04-23T16:42:05.338014+0000","last_peered":"2024-04-23T16:42:05.338014+0000","last_clean":"2024-04-23T16:42:05.338014+0000","last_became_active":"2024-04-23T16:28:25.051797+0000","last_became_peered":"2024-04-23T16:28:25.051797+0000","last_unstale":"2024-04-23T16:42:05.338014+0000","last_undegraded":"2024-04-23T16:42:05.338014+0000","last_fullsized":"2024-04-23T16:42:05.338014+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:40:23.268803+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:40:23.268803+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:42:06.652250+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":23,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-23T16:42:09.973531+0000","last_change":"2024-04-23T16:41:57.304749+0000","last_active":"2024-04-23T16:42:09.973531+0000","last_peered":"2024-04-23T16:42:09.973531+0000","last_clean":"2024-04-23T16:42:09.973531+0000","last_became_active":"2024-04-23T16:41:57.304523+0000","last_became_peered":"2024-04-23T16:41:57.304523+0000","last_unstale":"2024-04-23T16:42:09.973531+0000","last_undegraded":"2024-04-23T16:42:09.973531+0000","last_fullsized":"2024-04-23T16:42:09.973531+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:21.499311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1290,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-23T16:42:09.974230+0000","last_change":"2024-04-23T16:41:47.722866+0000","last_active":"2024-04-23T16:42:09.974230+0000","last_peered":"2024-04-23T16:42:09.974230+0000","last_clean":"2024-04-23T16:42:09.974230+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:42:09.974230+0000","last_undegraded":"2024-04-23T16:42:09.974230+0000","last_fullsized":"2024-04-23T16:42:09.974230+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:41:47.722762+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:03.039468+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":23,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-23T16:42:09.973543+0000","last_change":"2024-04-23T16:41:57.311521+0000","last_active":"2024-04-23T16:42:09.973543+0000","last_peered":"2024-04-23T16:42:09.973543+0000","last_clean":"2024-04-23T16:42:09.973543+0000","last_became_active":"2024-04-23T16:41:57.311302+0000","last_became_peered":"2024-04-23T16:41:57.311302+0000","last_unstale":"2024-04-23T16:42:09.973543+0000","last_undegraded":"2024-04-23T16:42:09.973543+0000","last_fullsized":"2024-04-23T16:42:09.973543+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:12.501144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1210,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-04-23T16:42:05.338012+0000","last_change":"2024-04-23T16:41:49.835121+0000","last_active":"2024-04-23T16:42:05.338012+0000","last_peered":"2024-04-23T16:42:05.338012+0000","last_clean":"2024-04-23T16:42:05.338012+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:42:05.338012+0000","last_undegraded":"2024-04-23T16:42:05.338012+0000","last_fullsized":"2024-04-23T16:42:05.338012+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:41:49.834374+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:42:56.109043+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":22,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-23T16:42:09.348229+0000","last_change":"2024-04-23T16:41:57.309579+0000","last_active":"2024-04-23T16:42:09.348229+0000","last_peered":"2024-04-23T16:42:09.348229+0000","last_clean":"2024-04-23T16:42:09.348229+0000","last_became_active":"2024-04-23T16:41:57.309349+0000","last_became_peered":"2024-04-23T16:41:57.309349+0000","last_unstale":"2024-04-23T16:42:09.348229+0000","last_undegraded":"2024-04-23T16:42:09.348229+0000","last_fullsized":"2024-04-23T16:42:09.348229+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:05.299815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1373,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-23T16:42:09.449373+0000","last_change":"2024-04-23T16:40:35.156593+0000","last_active":"2024-04-23T16:42:09.449373+0000","last_peered":"2024-04-23T16:42:09.449373+0000","last_clean":"2024-04-23T16:42:09.449373+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:42:09.449373+0000","last_undegraded":"2024-04-23T16:42:09.449373+0000","last_fullsized":"2024-04-23T16:42:09.449373+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:40:35.156484+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:40:35.156484+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:42:14.369891+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":9371648,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":143,"num_write_kb":9152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26411008,"data_stored":26411008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1269760,"data_stored":1259104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056963,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":534008,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93211144,"statfs":{"total":95995035648,"available":95448211456,"internally_reserved":0,"allocated":1683456,"data_stored":1300630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":545126852},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351304,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612942,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":437296,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93307856,"statfs":{"total":95995035648,"available":95547244544,"internally_reserved":0,"allocated":753664,"data_stored":376552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":447019456},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841874,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":334896,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93410256,"statfs":{"total":95995035648,"available":95652102144,"internally_reserved":0,"allocated":1081344,"data_stored":704232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168919,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545592,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199560,"statfs":{"total":95995035648,"available":95436349440,"internally_reserved":0,"allocated":765952,"data_stored":383126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":557906378},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463261,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":374828,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93370324,"statfs":{"total":95995035648,"available":95611211776,"internally_reserved":0,"allocated":753664,"data_stored":376552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":383056326},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137402,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":195256,"kb_used_data":748,"kb_used_omap":231,"kb_used_meta":194264,"kb_avail":93549896,"statfs":{"total":95995035648,"available":95795093504,"internally_reserved":0,"allocated":765952,"data_stored":383126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":198927248},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738747,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":509484,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93235668,"statfs":{"total":95995035648,"available":95473324032,"internally_reserved":0,"allocated":753664,"data_stored":376552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269417,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117355,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":707776,"kb_used_data":153204,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":93037376,"statfs":{"total":95995035648,"available":95270273024,"internally_reserved":0,"allocated":156880896,"data_stored":156494518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":567867847},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921125,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653868,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93091284,"statfs":{"total":95995035648,"available":95325474816,"internally_reserved":0,"allocated":753664,"data_stored":376552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757600,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":733760,"kb_used_data":125044,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93011392,"statfs":{"total":95995035648,"available":95243665408,"internally_reserved":0,"allocated":128045056,"data_stored":127658678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":26411008,"data_stored":26411008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:42:20.009 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-23T16:42:20.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:19 smithi188 ceph-mon[44101]: pgmap v1531: 27 pgs: 27 active+clean; 1.1 GiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 142 MiB/s wr, 2.28k op/s 2024-04-23T16:42:20.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:19 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:42:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:19 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:42:20.329 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:42:20.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:19 smithi103 ceph-mon[39629]: pgmap v1531: 27 pgs: 27 active+clean; 1.1 GiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 142 MiB/s wr, 2.28k op/s 2024-04-23T16:42:20.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:19 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:42:20.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:19 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:42:20.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:20.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:42:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:19 smithi148 ceph-mon[44520]: pgmap v1531: 27 pgs: 27 active+clean; 1.1 GiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 142 MiB/s wr, 2.28k op/s 2024-04-23T16:42:20.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:19 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:42:20.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:19 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:42:20.433 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:42:20.433 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:42:20.433 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-23T16:42:20.434 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-23T16:42:20.434 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:42:20.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:42:20.481 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:20.530 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 9 16 39198 39182 272.026 218 0.00366526 0.00367415 2024-04-23T16:42:20.687 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:42:20.692 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:42:20.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:20.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:42:20.797 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:42:20.848 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:20.869 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:42:20.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:20.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:42:20.959 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:20 smithi103 ceph-mon[39629]: osdmap e550: 12 total, 12 up, 10 in 2024-04-23T16:42:20.975 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:42:21.012 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:20 smithi148 ceph-mon[44520]: osdmap e550: 12 total, 12 up, 10 in 2024-04-23T16:42:21.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:20 smithi188 ceph-mon[44101]: osdmap e550: 12 total, 12 up, 10 in 2024-04-23T16:42:21.530 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 10 16 42252 42236 263.91 190.875 0.00211896 0.00376493 2024-04-23T16:42:21.774 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:42:21.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:21.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:42:21.890 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:42:21.928 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:21 smithi103 ceph-mon[39629]: pgmap v1533: 27 pgs: 27 active+clean; 2.1 GiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 298 MiB/s wr, 4.77k op/s 2024-04-23T16:42:22.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:21 smithi188 ceph-mon[44101]: pgmap v1533: 27 pgs: 27 active+clean; 2.1 GiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 298 MiB/s wr, 4.77k op/s 2024-04-23T16:42:22.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:21 smithi148 ceph-mon[44520]: pgmap v1533: 27 pgs: 27 active+clean; 2.1 GiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 298 MiB/s wr, 4.77k op/s 2024-04-23T16:42:22.531 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 11 16 45027 45011 255.667 173.438 0.00322501 0.00390968 2024-04-23T16:42:23.531 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 12 16 48018 48002 249.939 186.938 0.00271958 0.00399976 2024-04-23T16:42:23.585 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:42:23.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:23.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:42:23.612 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-23T16:42:23.692 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:42:23.738 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:23.771 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:42:23.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:23.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:42:23.877 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:42:23.912 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:23.936 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:42:23.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:23.940 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:42:24.041 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:42:24.076 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:24.101 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:42:24.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:24.105 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:42:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:23 smithi148 ceph-mon[44520]: pgmap v1534: 27 pgs: 27 active+clean; 2.3 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 221 MiB/s wr, 3.54k op/s 2024-04-23T16:42:24.207 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:42:24.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:23 smithi188 ceph-mon[44101]: pgmap v1534: 27 pgs: 27 active+clean; 2.3 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 221 MiB/s wr, 3.54k op/s 2024-04-23T16:42:24.242 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:24.267 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:42:24.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:24.271 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:42:24.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:23 smithi103 ceph-mon[39629]: pgmap v1534: 27 pgs: 27 active+clean; 2.3 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 221 MiB/s wr, 3.54k op/s 2024-04-23T16:42:24.372 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:42:24.420 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:24.531 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 13 16 50851 50835 244.334 177.062 0.00816235 0.00409104 2024-04-23T16:42:25.531 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 14 16 53549 53533 238.926 168.625 0.00376645 0.00418369 2024-04-23T16:42:25.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:25 smithi103 ceph-mon[39629]: pgmap v1535: 27 pgs: 27 active+clean; 2.8 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 227 MiB/s wr, 3.64k op/s 2024-04-23T16:42:25.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:25 smithi148 ceph-mon[44520]: pgmap v1535: 27 pgs: 27 active+clean; 2.8 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 227 MiB/s wr, 3.64k op/s 2024-04-23T16:42:25.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:25 smithi188 ceph-mon[44101]: pgmap v1535: 27 pgs: 27 active+clean; 2.8 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 227 MiB/s wr, 3.64k op/s 2024-04-23T16:42:26.010 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:42:26.015 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:42:26.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:26.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:42:26.132 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:42:26.178 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:26.205 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:42:26.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:26.208 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:42:26.304 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:26 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:42:26.304 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:26 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:42:26.304 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:26 smithi103 ceph-mon[39629]: osdmap e551: 12 total, 12 up, 10 in 2024-04-23T16:42:26.308 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:42:26.340 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:26.371 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:42:26.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:26.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:42:26.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:26 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:42:26.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:26 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:42:26.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:26 smithi188 ceph-mon[44101]: osdmap e551: 12 total, 12 up, 10 in 2024-04-23T16:42:26.476 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:42:26.509 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:26.531 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 15 16 56136 56120 233.777 161.688 0.00731209 0.00427597 2024-04-23T16:42:26.532 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:42:26.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:26.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:42:26.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:26 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:42:26.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:26 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:42:26.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:26 smithi148 ceph-mon[44520]: osdmap e551: 12 total, 12 up, 10 in 2024-04-23T16:42:26.637 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:42:26.685 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:26.715 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:42:26.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:26.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:42:26.822 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:42:26.862 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:26.896 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:42:26.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:26.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:42:27.001 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:42:27.034 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:27.063 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:42:27.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:27.067 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:42:27.168 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:42:27.219 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:27.247 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:42:27.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:27.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:42:27.323 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:27 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:42:27.324 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:27 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:42:27.324 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:27 smithi188 ceph-mon[44101]: osdmap e552: 12 total, 12 up, 10 in 2024-04-23T16:42:27.324 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:27 smithi188 ceph-mon[44101]: pgmap v1538: 27 pgs: 27 active+clean; 3.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 198 MiB/s wr, 3.17k op/s 2024-04-23T16:42:27.324 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:27 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3128713374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:42:27.324 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:27 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:42:27.324 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:27 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:42:27.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:27 smithi103 ceph-mon[39629]: osdmap e552: 12 total, 12 up, 10 in 2024-04-23T16:42:27.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:27 smithi103 ceph-mon[39629]: pgmap v1538: 27 pgs: 27 active+clean; 3.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 198 MiB/s wr, 3.17k op/s 2024-04-23T16:42:27.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:27 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3128713374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:42:27.353 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:42:27.450 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:27.531 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 16 16 58536 58520 228.541 150 0.00275252 0.00437343 2024-04-23T16:42:27.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:27 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:42:27.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:27 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:42:27.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:27 smithi148 ceph-mon[44520]: osdmap e552: 12 total, 12 up, 10 in 2024-04-23T16:42:27.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:27 smithi148 ceph-mon[44520]: pgmap v1538: 27 pgs: 27 active+clean; 3.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 198 MiB/s wr, 3.17k op/s 2024-04-23T16:42:27.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:27 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3128713374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:42:28.277 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:42:28.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:28.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:42:28.384 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:42:28.397 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:42:28.441 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:28.531 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 17 16 61005 60989 224.174 154.312 0.00405978 0.00445943 2024-04-23T16:42:28.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:28 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3798958728' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:42:28.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:28 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3798958728' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:42:28.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:28.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:28 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3798958728' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:42:29.299 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:42:29.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:29.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:42:29.406 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:42:29.454 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:29.486 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:42:29.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:29.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:42:29.531 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 18 16 63400 63384 220.036 149.688 0.00531826 0.00454304 2024-04-23T16:42:29.588 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:42:29.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:29.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:29.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:29 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:42:29.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:29 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:42:29.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:29 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:42:29.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:29 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3798958728' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:42:29.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:29 smithi103 ceph-mon[39629]: osdmap e553: 12 total, 12 up, 10 in 2024-04-23T16:42:29.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:29 smithi103 ceph-mon[39629]: pgmap v1540: 27 pgs: 27 active+clean; 3.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-04-23T16:42:29.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:29 smithi103 ceph-mon[39629]: osdmap e554: 12 total, 12 up, 10 in 2024-04-23T16:42:29.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:29.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:29.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:29 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:42:29.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:29 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:42:29.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:29 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:42:29.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:29 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3798958728' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:42:29.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:29 smithi148 ceph-mon[44520]: osdmap e553: 12 total, 12 up, 10 in 2024-04-23T16:42:29.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:29 smithi148 ceph-mon[44520]: pgmap v1540: 27 pgs: 27 active+clean; 3.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-04-23T16:42:29.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:29 smithi148 ceph-mon[44520]: osdmap e554: 12 total, 12 up, 10 in 2024-04-23T16:42:29.626 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:29.653 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:42:29.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:29.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:42:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:29 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:42:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:29 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:42:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:29 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:42:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:29 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3798958728' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:42:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:29 smithi188 ceph-mon[44101]: osdmap e553: 12 total, 12 up, 10 in 2024-04-23T16:42:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:29 smithi188 ceph-mon[44101]: pgmap v1540: 27 pgs: 27 active+clean; 3.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-04-23T16:42:29.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:29 smithi188 ceph-mon[44101]: osdmap e554: 12 total, 12 up, 10 in 2024-04-23T16:42:29.763 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:42:29.796 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:29.822 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:42:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:29.827 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:42:29.853 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:42:29.853 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:42:29.853 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:42:29.853 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:42:29.853 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:42:29.853 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:42:29.853 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:42:29.853 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:42:29.853 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:42:29.854 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:42:29.854 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:42:29.854 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:42:29.854 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:42:29.854 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:42:29.854 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:42:29.854 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:42:29.854 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:42:29.854 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:42:29.854 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:42:29.854 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:42:29.855 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:42:29.855 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:42:29.855 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:42:29.855 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:42:29.927 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:42:29.964 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:29.998 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:42:30.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:30.002 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:42:30.103 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:42:30.137 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:30.531 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 19 16 65831 65815 216.452 151.938 0.00539307 0.00461883 2024-04-23T16:42:31.531 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:42:31.531659 min lat: 0.000969795 max lat: 0.106105 avg lat: 0.00469798 2024-04-23T16:42:31.532 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:42:31.532 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 20 16 68130 68114 212.813 143.688 0.0106006 0.00469798 2024-04-23T16:42:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:31 smithi103 ceph-mon[39629]: pgmap v1542: 27 pgs: 27 active+clean; 3.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 174 MiB/s wr, 2.78k op/s 2024-04-23T16:42:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:31 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:42:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:31 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:42:32.195 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:42:32.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:32.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:42:32.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:31 smithi188 ceph-mon[44101]: pgmap v1542: 27 pgs: 27 active+clean; 3.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 174 MiB/s wr, 2.78k op/s 2024-04-23T16:42:32.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:31 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:42:32.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:31 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:42:32.299 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:42:32.346 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:31 smithi148 ceph-mon[44520]: pgmap v1542: 27 pgs: 27 active+clean; 3.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 174 MiB/s wr, 2.78k op/s 2024-04-23T16:42:32.346 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:31 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:42:32.346 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:31 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:42:32.352 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:32.388 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:42:32.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:32.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:42:32.494 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:42:32.531 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 21 16 70467 70451 209.635 146.062 0.0143675 0.00476844 2024-04-23T16:42:32.554 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:32.591 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:42:32.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:32.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:42:32.697 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:42:32.737 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:32.766 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:42:32.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:32.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:42:32.873 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:42:32.911 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:32.943 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:42:32.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:32.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:42:33.049 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:42:33.092 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:32 smithi148 ceph-mon[44520]: 1.9 scrub starts 2024-04-23T16:42:33.092 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:32 smithi148 ceph-mon[44520]: 1.9 scrub ok 2024-04-23T16:42:33.095 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:33.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:32 smithi103 ceph-mon[39629]: 1.9 scrub starts 2024-04-23T16:42:33.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:32 smithi103 ceph-mon[39629]: 1.9 scrub ok 2024-04-23T16:42:33.131 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:42:33.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:33.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:42:33.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:32 smithi188 ceph-mon[44101]: 1.9 scrub starts 2024-04-23T16:42:33.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:32 smithi188 ceph-mon[44101]: 1.9 scrub ok 2024-04-23T16:42:33.241 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:42:33.272 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:33.299 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:42:33.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:33.304 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:42:33.405 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:42:33.451 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:33.484 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:42:33.484 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:42:33.489 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:42:33.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:33.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:42:33.531 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 22 16 72598 72582 206.16 133.188 0.00740301 0.00484971 2024-04-23T16:42:33.596 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:42:33.631 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:33.653 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:42:33.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:33.656 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:42:33.691 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:42:33.757 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:42:33.823 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:33.909 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:33 smithi188 ceph-mon[44101]: pgmap v1543: 27 pgs: 27 active+clean; 3.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-04-23T16:42:33.909 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:33 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:42:33.910 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:33 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:42:33.910 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:33 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:42:33.910 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:33 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:42:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:33 smithi103 ceph-mon[39629]: pgmap v1543: 27 pgs: 27 active+clean; 3.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-04-23T16:42:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:33 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:42:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:33 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:42:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:33 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:42:34.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:33 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:42:34.159 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:42:34.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:33 smithi148 ceph-mon[44520]: pgmap v1543: 27 pgs: 27 active+clean; 3.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-04-23T16:42:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:33 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:42:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:33 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:42:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:33 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:42:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:33 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:42:34.532 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 23 15 74739 74724 203.017 133.875 0.00761294 0.0049241 2024-04-23T16:42:35.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:34 smithi188 ceph-mon[44101]: 1.7 scrub starts 2024-04-23T16:42:35.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:34 smithi188 ceph-mon[44101]: 1.7 scrub ok 2024-04-23T16:42:35.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:34 smithi188 ceph-mon[44101]: osdmap e555: 12 total, 12 up, 10 in 2024-04-23T16:42:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:34 smithi103 ceph-mon[39629]: 1.7 scrub starts 2024-04-23T16:42:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:34 smithi103 ceph-mon[39629]: 1.7 scrub ok 2024-04-23T16:42:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:34 smithi103 ceph-mon[39629]: osdmap e555: 12 total, 12 up, 10 in 2024-04-23T16:42:35.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:34 smithi148 ceph-mon[44520]: 1.7 scrub starts 2024-04-23T16:42:35.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:34 smithi148 ceph-mon[44520]: 1.7 scrub ok 2024-04-23T16:42:35.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:34 smithi148 ceph-mon[44520]: osdmap e555: 12 total, 12 up, 10 in 2024-04-23T16:42:35.520 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:42:35.531 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 24 16 76786 76770 199.886 127.875 0.00802662 0.00500135 2024-04-23T16:42:35.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:35.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:42:35.732 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:42:35.778 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:35.811 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:42:35.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:35.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:42:35.918 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:42:35.946 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-23T16:42:35.960 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:35.969 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:42:35.969 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:42:35.984 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:42:35.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:35.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:42:36.090 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:42:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:35 smithi148 ceph-mon[44520]: pgmap v1545: 27 pgs: 27 active+clean; 4.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 181 MiB/s wr, 2.89k op/s 2024-04-23T16:42:36.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:35 smithi103 ceph-mon[39629]: pgmap v1545: 27 pgs: 27 active+clean; 4.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 181 MiB/s wr, 2.89k op/s 2024-04-23T16:42:36.160 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:36.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:35 smithi188 ceph-mon[44101]: pgmap v1545: 27 pgs: 27 active+clean; 4.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 181 MiB/s wr, 2.89k op/s 2024-04-23T16:42:36.283 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:42:36.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:36.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:42:36.390 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:42:36.422 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:36.446 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:42:36.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:36.454 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:42:36.532 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 25 16 78811 78795 196.952 126.562 0.00572796 0.00507596 2024-04-23T16:42:36.555 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:42:36.608 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:37.532 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 26 16 80858 80842 194.298 127.938 0.0083304 0.00514521 2024-04-23T16:42:37.569 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:42:37.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:37.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:42:37.675 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:42:37.720 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:38.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:37 smithi188 ceph-mon[44101]: pgmap v1546: 27 pgs: 27 active+clean; 4.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 179 MiB/s wr, 2.86k op/s 2024-04-23T16:42:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:37 smithi103 ceph-mon[39629]: pgmap v1546: 27 pgs: 27 active+clean; 4.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 179 MiB/s wr, 2.86k op/s 2024-04-23T16:42:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:37 smithi148 ceph-mon[44520]: pgmap v1546: 27 pgs: 27 active+clean; 4.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 179 MiB/s wr, 2.86k op/s 2024-04-23T16:42:38.532 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 27 16 82708 82692 191.384 115.625 0.00686304 0.00522344 2024-04-23T16:42:39.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:38 smithi188 ceph-mon[44101]: 1.a scrub starts 2024-04-23T16:42:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:38 smithi188 ceph-mon[44101]: 1.a scrub ok 2024-04-23T16:42:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:38 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:42:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:38 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:42:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:38 smithi188 ceph-mon[44101]: osdmap e556: 12 total, 12 up, 10 in 2024-04-23T16:42:39.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:38 smithi103 ceph-mon[39629]: 1.a scrub starts 2024-04-23T16:42:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:38 smithi103 ceph-mon[39629]: 1.a scrub ok 2024-04-23T16:42:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:38 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:42:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:38 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:42:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:38 smithi103 ceph-mon[39629]: osdmap e556: 12 total, 12 up, 10 in 2024-04-23T16:42:39.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:38 smithi148 ceph-mon[44520]: 1.a scrub starts 2024-04-23T16:42:39.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:38 smithi148 ceph-mon[44520]: 1.a scrub ok 2024-04-23T16:42:39.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:38 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:42:39.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:38 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:42:39.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:38 smithi148 ceph-mon[44520]: osdmap e556: 12 total, 12 up, 10 in 2024-04-23T16:42:39.532 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 28 16 84528 84512 188.611 113.75 0.0102661 0.00530067 2024-04-23T16:42:40.009 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-04-23T16:42:40.010 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 2] 2024-04-23T16:42:40.010 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-23T16:42:40.010 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.3 2024-04-23T16:42:40.093 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:42:40.105 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:42:40.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:40.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:42:40.214 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:42:40.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:39 smithi188 ceph-mon[44101]: pgmap v1548: 27 pgs: 27 active+clean; 4.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-04-23T16:42:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:39 smithi188 ceph-mon[44101]: osdmap e557: 12 total, 12 up, 10 in 2024-04-23T16:42:40.249 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:40.275 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:42:40.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:40.278 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:42:40.350 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:42:40 smithi103 systemd[1]: Stopping Ceph osd.3 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T16:42:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:39 smithi103 ceph-mon[39629]: pgmap v1548: 27 pgs: 27 active+clean; 4.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-04-23T16:42:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:39 smithi103 ceph-mon[39629]: osdmap e557: 12 total, 12 up, 10 in 2024-04-23T16:42:40.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:39 smithi148 ceph-mon[44520]: pgmap v1548: 27 pgs: 27 active+clean; 4.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-04-23T16:42:40.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:39 smithi148 ceph-mon[44520]: osdmap e557: 12 total, 12 up, 10 in 2024-04-23T16:42:40.379 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:42:40.417 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:40.446 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:42:40.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:40.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:42:40.532 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 29 15 86465 86450 186.284 121.125 0.00663212 0.00536677 2024-04-23T16:42:40.551 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:42:40.600 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:40.626 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:42:40.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:40.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:42:40.735 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:42:40.779 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:40.809 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:42:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:40.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:42:40.917 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:42:40.950 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:40.973 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:42:40.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:40.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:42:41.078 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:42:41.111 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:41.136 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:42:41.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:41.140 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:42:41.241 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:42:41.271 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:41.300 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:42:41.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:41.303 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:42:41.404 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:42:41.441 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:41.464 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:42:41.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:41.469 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:42:41.532 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 30 16 88335 88319 183.968 116.812 0.00583473 0.00543442 2024-04-23T16:42:41.569 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:42:41.770 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:41.971 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:42:42.036 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:41 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:42:42.036 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:41 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:42:42.036 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:41 smithi103 ceph-mon[39629]: pgmap v1550: 27 pgs: 27 active+clean; 5.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 161 MiB/s wr, 2.58k op/s 2024-04-23T16:42:42.036 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:41 smithi103 ceph-mon[39629]: osdmap e558: 12 total, 12 up, 10 in 2024-04-23T16:42:42.036 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:41 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3735996673' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:42:42.036 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:41 smithi103 ceph-mon[39629]: from='client.26605 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:42:42.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:41 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:42:42.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:41 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:42:42.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:41 smithi188 ceph-mon[44101]: pgmap v1550: 27 pgs: 27 active+clean; 5.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 161 MiB/s wr, 2.58k op/s 2024-04-23T16:42:42.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:41 smithi188 ceph-mon[44101]: osdmap e558: 12 total, 12 up, 10 in 2024-04-23T16:42:42.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:41 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3735996673' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:42:42.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:41 smithi188 ceph-mon[44101]: from='client.26605 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:42:42.349 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:42:42 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-3[80767]: 2024-04-23T16:42:42.034+0000 7f23a962d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T16:42:42.350 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:42:42 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-3[80767]: 2024-04-23T16:42:42.034+0000 7f23a962d700 -1 osd.3 559 *** Got signal Terminated *** 2024-04-23T16:42:42.350 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:42:42 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-3[80767]: 2024-04-23T16:42:42.034+0000 7f23a962d700 -1 osd.3 559 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T16:42:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:41 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:42:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:41 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:42:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:41 smithi148 ceph-mon[44520]: pgmap v1550: 27 pgs: 27 active+clean; 5.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 161 MiB/s wr, 2.58k op/s 2024-04-23T16:42:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:41 smithi148 ceph-mon[44520]: osdmap e558: 12 total, 12 up, 10 in 2024-04-23T16:42:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:41 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3735996673' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:42:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:41 smithi148 ceph-mon[44520]: from='client.26605 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:42:42.532 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 31 16 90122 90106 181.636 111.688 0.00589847 0.00550413 2024-04-23T16:42:42.566 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:42:42.573 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:42:42.598 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:42:42.624 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:42:42.657 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:42:42.690 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:42:42.718 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:42:42.748 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:42:42.777 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:42:42.809 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:42:42.832 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:42:42.872 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:42:42.872 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:42:42.872 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:42:42.872 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:42:42.872 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:42:42.872 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:42:42.872 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:42:42.872 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:42:42.872 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:42:42.873 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:42:42.873 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:42:42.873 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:42:42.873 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:42:43.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:42 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:42:43.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:42 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:42:43.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:42 smithi188 ceph-mon[44101]: from='client.26605 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:42:43.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:42 smithi188 ceph-mon[44101]: osdmap e559: 12 total, 12 up, 10 in 2024-04-23T16:42:43.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:42 smithi188 ceph-mon[44101]: osd.3 marked itself down and dead 2024-04-23T16:42:43.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:42 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3945674913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:42:43.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:43.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:43.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:43.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:43.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:43.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:42 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:42:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:42 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:42:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:42 smithi103 ceph-mon[39629]: from='client.26605 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:42:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:42 smithi103 ceph-mon[39629]: osdmap e559: 12 total, 12 up, 10 in 2024-04-23T16:42:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:42 smithi103 ceph-mon[39629]: osd.3 marked itself down and dead 2024-04-23T16:42:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:42 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3945674913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:42:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:43.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:43.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:43.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:43.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:43.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:42 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:42:43.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:42 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:42:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:42 smithi148 ceph-mon[44520]: from='client.26605 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:42:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:42 smithi148 ceph-mon[44520]: osdmap e559: 12 total, 12 up, 10 in 2024-04-23T16:42:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:42 smithi148 ceph-mon[44520]: osd.3 marked itself down and dead 2024-04-23T16:42:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:42 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3945674913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:42:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:43.532 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 32 16 90985 90969 177.646 53.9375 0.00325615 0.00553317 2024-04-23T16:42:43.533 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:42:43.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:43.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:42:43.638 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:42:43.667 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:43.687 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:42:43.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:43.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:42:43.791 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:42:43.821 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:43 smithi103 ceph-mon[39629]: pgmap v1553: 27 pgs: 27 active+clean; 5.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 122 MiB/s wr, 1.96k op/s 2024-04-23T16:42:44.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:44.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:43 smithi103 ceph-mon[39629]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T16:42:44.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:43 smithi103 ceph-mon[39629]: osdmap e560: 12 total, 11 up, 10 in 2024-04-23T16:42:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:44.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:43 smithi188 ceph-mon[44101]: pgmap v1553: 27 pgs: 27 active+clean; 5.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 122 MiB/s wr, 1.96k op/s 2024-04-23T16:42:44.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:44.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:43 smithi188 ceph-mon[44101]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T16:42:44.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:43 smithi188 ceph-mon[44101]: osdmap e560: 12 total, 11 up, 10 in 2024-04-23T16:42:44.271 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:42:44.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:44.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:42:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:43 smithi148 ceph-mon[44520]: pgmap v1553: 27 pgs: 27 active+clean; 5.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 122 MiB/s wr, 1.96k op/s 2024-04-23T16:42:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:43 smithi148 ceph-mon[44520]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T16:42:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:43 smithi148 ceph-mon[44520]: osdmap e560: 12 total, 11 up, 10 in 2024-04-23T16:42:44.375 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:42:44.406 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:44.426 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:42:44.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:44.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:42:44.530 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:42:44.533 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 33 16 90985 90969 172.262 0 - 0.00553317 2024-04-23T16:42:44.558 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:44.813 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:42:44.816 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:42:44 smithi103 bash[240902]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-3 2024-04-23T16:42:44.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:44.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:42:44.918 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:42:44.950 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:44.972 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:42:44.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:44.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:42:45.069 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:44 smithi103 ceph-mon[39629]: osdmap e561: 12 total, 11 up, 10 in 2024-04-23T16:42:45.075 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:42:45.106 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:45.205 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:42:45.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:45.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:42:45.310 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:42:45.311 INFO:teuthology.orchestra.run.smithi103.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-23T16:42:45.340 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:45.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:44 smithi148 ceph-mon[44520]: osdmap e561: 12 total, 11 up, 10 in 2024-04-23T16:42:45.464 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:42:45.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:45.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:42:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:44 smithi188 ceph-mon[44101]: osdmap e561: 12 total, 11 up, 10 in 2024-04-23T16:42:45.533 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 34 16 90985 90969 167.196 0 - 0.00553317 2024-04-23T16:42:45.568 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:42:45.600 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:45.622 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:42:45.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:45.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:42:45.727 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:42:45.754 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:45 smithi103 ceph-mon[39629]: pgmap v1556: 27 pgs: 2 undersized+degraded+peered, 2 peering, 1 undersized+peered, 22 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 104 MiB/s wr, 1.66k op/s; 11451/180218 objects degraded (6.354%) 2024-04-23T16:42:46.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:45 smithi103 ceph-mon[39629]: Health check failed: Degraded data redundancy: 11451/180218 objects degraded (6.354%), 2 pgs degraded (PG_DEGRADED) 2024-04-23T16:42:46.200 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T16:42:46.201 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-23T16:42:46.203 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:42:46.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:46.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:42:46.307 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:42:46.337 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:45 smithi148 ceph-mon[44520]: pgmap v1556: 27 pgs: 2 undersized+degraded+peered, 2 peering, 1 undersized+peered, 22 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 104 MiB/s wr, 1.66k op/s; 11451/180218 objects degraded (6.354%) 2024-04-23T16:42:46.338 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:45 smithi148 ceph-mon[44520]: Health check failed: Degraded data redundancy: 11451/180218 objects degraded (6.354%), 2 pgs degraded (PG_DEGRADED) 2024-04-23T16:42:46.340 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:46.361 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:42:46.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:46.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:42:46.465 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:42:46.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:45 smithi188 ceph-mon[44101]: pgmap v1556: 27 pgs: 2 undersized+degraded+peered, 2 peering, 1 undersized+peered, 22 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 104 MiB/s wr, 1.66k op/s; 11451/180218 objects degraded (6.354%) 2024-04-23T16:42:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:45 smithi188 ceph-mon[44101]: Health check failed: Degraded data redundancy: 11451/180218 objects degraded (6.354%), 2 pgs degraded (PG_DEGRADED) 2024-04-23T16:42:46.495 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:46.515 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:42:46.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:46.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:42:46.533 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 35 16 90985 90969 162.419 0 - 0.00553317 2024-04-23T16:42:46.619 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:42:46.646 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:46.769 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:42:46.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:46.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:42:46.873 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:42:46.905 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:46.926 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:42:46.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:46.929 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:42:47.030 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:42:47.062 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:47.079 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:42:47.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:47.082 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:42:47.183 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:42:47.215 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:47.237 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:42:47.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:47.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:42:47.316 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:42:47.341 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:42:47.373 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:47.394 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:42:47.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:47.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:42:47.498 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:42:47.527 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:47.533 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 36 16 90985 90969 157.907 0 - 0.00553317 2024-04-23T16:42:47.547 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:42:47.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:47.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:42:47.651 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:42:47.679 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:47.698 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:42:47.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:47.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:42:47.802 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:42:47.834 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:47.855 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:42:47.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:47.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:42:47.959 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:42:47.991 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:48.019 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:42:48.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:48.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:42:48.122 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:42:48.150 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:47 smithi188 ceph-mon[44101]: pgmap v1557: 27 pgs: 3 undersized+degraded+peered, 2 peering, 1 undersized+peered, 21 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 72 MiB/s wr, 1.15k op/s; 11452/181722 objects degraded (6.302%) 2024-04-23T16:42:48.152 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:48.173 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:42:48.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:48.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:42:48.277 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:42:48.305 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:48.325 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:42:48.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:48.328 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:42:48.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:47 smithi103 ceph-mon[39629]: pgmap v1557: 27 pgs: 3 undersized+degraded+peered, 2 peering, 1 undersized+peered, 21 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 72 MiB/s wr, 1.15k op/s; 11452/181722 objects degraded (6.302%) 2024-04-23T16:42:48.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:47 smithi148 ceph-mon[44520]: pgmap v1557: 27 pgs: 3 undersized+degraded+peered, 2 peering, 1 undersized+peered, 21 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 72 MiB/s wr, 1.15k op/s; 11452/181722 objects degraded (6.302%) 2024-04-23T16:42:48.428 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:42:48.456 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:48.474 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:42:48.475 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:42:48.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:48.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:42:48.533 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 37 16 90985 90969 153.64 0 - 0.00553317 2024-04-23T16:42:48.579 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:42:48.612 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:48.633 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:42:48.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:48.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:42:48.737 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:42:48.765 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:49.344 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:42:49.514 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:42:49.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:49.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:42:49.533 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 38 16 90985 90969 149.597 0 - 0.00553317 2024-04-23T16:42:49.623 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:42:49.649 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:49.873 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:42:49.876 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.3.service' 2024-04-23T16:42:49.899 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:42:49.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:49.905 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:42:49 smithi103 systemd[1]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.3.service: Succeeded. 2024-04-23T16:42:49.905 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:42:49 smithi103 systemd[1]: Stopped Ceph osd.3 for 8e3594f2-018b-11ef-bc93-c7b262605968. 2024-04-23T16:42:49.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:42:49.937 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T16:42:49.938 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-23T16:42:49.938 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd down 3 2024-04-23T16:42:50.007 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:42:50.040 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:50.061 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:42:50.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:50.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:42:50.164 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:42:50.166 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:49 smithi103 ceph-mon[39629]: pgmap v1558: 27 pgs: 3 undersized+degraded+peered, 2 peering, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 52 MiB/s wr, 832 op/s; 11452/181950 objects degraded (6.294%) 2024-04-23T16:42:50.194 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:50.214 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:42:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:50.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:42:50.318 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:42:50.350 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:50.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:49 smithi148 ceph-mon[44520]: pgmap v1558: 27 pgs: 3 undersized+degraded+peered, 2 peering, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 52 MiB/s wr, 832 op/s; 11452/181950 objects degraded (6.294%) 2024-04-23T16:42:50.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:49 smithi188 ceph-mon[44101]: pgmap v1558: 27 pgs: 3 undersized+degraded+peered, 2 peering, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 52 MiB/s wr, 832 op/s; 11452/181950 objects degraded (6.294%) 2024-04-23T16:42:50.533 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 39 16 90985 90969 145.761 0 - 0.00553317 2024-04-23T16:42:50.874 INFO:teuthology.orchestra.run.smithi103.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-23T16:42:51.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:51 smithi103 ceph-mon[39629]: 1.6 deep-scrub starts 2024-04-23T16:42:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:51 smithi103 ceph-mon[39629]: 1.6 deep-scrub ok 2024-04-23T16:42:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:51 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 22696/181950 objects degraded (12.474%), 5 pgs degraded (PG_DEGRADED) 2024-04-23T16:42:51.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:51 smithi148 ceph-mon[44520]: 1.6 deep-scrub starts 2024-04-23T16:42:51.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:51 smithi148 ceph-mon[44520]: 1.6 deep-scrub ok 2024-04-23T16:42:51.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:51 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 22696/181950 objects degraded (12.474%), 5 pgs degraded (PG_DEGRADED) 2024-04-23T16:42:51.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:51 smithi188 ceph-mon[44101]: 1.6 deep-scrub starts 2024-04-23T16:42:51.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:51 smithi188 ceph-mon[44101]: 1.6 deep-scrub ok 2024-04-23T16:42:51.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:51 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 22696/181950 objects degraded (12.474%), 5 pgs degraded (PG_DEGRADED) 2024-04-23T16:42:51.534 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:42:51.533864 min lat: 0.000969795 max lat: 0.209921 avg lat: 0.00553317 2024-04-23T16:42:51.534 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:42:51.534 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 40 16 90985 90969 142.117 0 - 0.00553317 2024-04-23T16:42:51.889 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:42:51.893 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:42:51.899 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:42:51.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:51.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:42:52.005 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:42:52.035 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:52.055 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:42:52.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:52.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:42:52.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:52 smithi103 ceph-mon[39629]: pgmap v1559: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 45 MiB/s wr, 715 op/s; 22696/181950 objects degraded (12.474%) 2024-04-23T16:42:52.159 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:42:52.191 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:52.211 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:42:52.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:52.214 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:42:52.315 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:42:52.345 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:52 smithi148 ceph-mon[44520]: pgmap v1559: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 45 MiB/s wr, 715 op/s; 22696/181950 objects degraded (12.474%) 2024-04-23T16:42:52.347 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:52.369 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:42:52.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:52.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:42:52.473 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:42:52.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:52 smithi188 ceph-mon[44101]: pgmap v1559: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 45 MiB/s wr, 715 op/s; 22696/181950 objects degraded (12.474%) 2024-04-23T16:42:52.503 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:52.534 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 41 16 90985 90969 138.651 0 - 0.00553317 2024-04-23T16:42:52.627 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:42:52.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:52.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:42:52.649 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T16:42:52.650 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-23T16:42:52.730 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:42:52.762 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:52.783 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:42:52.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:52.787 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:42:52.887 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:42:52.918 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:52.939 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:42:52.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:52.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:42:53.043 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:42:53.072 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:53.092 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:42:53.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:53.095 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:42:53.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:53 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:42:53.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:53 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:42:53.196 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:42:53.226 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:53 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:42:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:53 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:42:53.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:53 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:42:53.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:53 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:42:53.474 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:42:53.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:53.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:42:53.534 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 42 16 90985 90969 135.35 0 - 0.00553317 2024-04-23T16:42:53.578 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:42:53.610 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:53.630 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:42:53.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:53.633 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:42:53.734 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:42:53.767 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:53.789 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:42:53.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:53.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:42:53.893 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:42:53.925 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:53.945 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:42:53.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:53.948 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:42:54.049 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:42:54.082 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:54.102 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:42:54.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:54.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:42:54.206 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:42:54.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:54 smithi188 ceph-mon[44101]: pgmap v1560: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 36 MiB/s wr, 580 op/s; 22696/181950 objects degraded (12.474%) 2024-04-23T16:42:54.236 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:54.258 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:42:54.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:54.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:42:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:54 smithi103 ceph-mon[39629]: pgmap v1560: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 36 MiB/s wr, 580 op/s; 22696/181950 objects degraded (12.474%) 2024-04-23T16:42:54.361 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:42:54.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:54 smithi148 ceph-mon[44520]: pgmap v1560: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 36 MiB/s wr, 580 op/s; 22696/181950 objects degraded (12.474%) 2024-04-23T16:42:54.393 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:54.414 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:42:54.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:54.417 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:42:54.518 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:42:54.534 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 43 16 90985 90969 132.202 0 - 0.00553317 2024-04-23T16:42:54.551 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:54.572 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:42:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:54.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:42:54.676 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:42:54.706 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:55.047 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:55 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1288760776' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:42:55.060 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:42:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:55 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1288760776' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:42:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:55 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1288760776' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:42:55.534 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 44 16 90985 90969 129.197 0 - 0.00553317 2024-04-23T16:42:56.056 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:42:56.058 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:56 smithi103 ceph-mon[39629]: pgmap v1561: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 29 MiB/s wr, 456 op/s; 22696/181950 objects degraded (12.474%) 2024-04-23T16:42:56.058 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:56 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:42:56.058 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:56 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1288760776' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:42:56.059 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:56 smithi103 ceph-mon[39629]: osdmap e562: 12 total, 11 up, 10 in 2024-04-23T16:42:56.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:56.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:42:56.161 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:42:56.193 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:56.215 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:42:56.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:56.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:42:56.318 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:42:56.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:56 smithi188 ceph-mon[44101]: pgmap v1561: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 29 MiB/s wr, 456 op/s; 22696/181950 objects degraded (12.474%) 2024-04-23T16:42:56.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:56 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:42:56.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:56 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1288760776' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:42:56.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:56 smithi188 ceph-mon[44101]: osdmap e562: 12 total, 11 up, 10 in 2024-04-23T16:42:56.350 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:56.372 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:42:56.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:56 smithi148 ceph-mon[44520]: pgmap v1561: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 29 MiB/s wr, 456 op/s; 22696/181950 objects degraded (12.474%) 2024-04-23T16:42:56.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:56 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:42:56.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:56 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1288760776' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:42:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:56 smithi148 ceph-mon[44520]: osdmap e562: 12 total, 11 up, 10 in 2024-04-23T16:42:56.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:56.375 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:42:56.475 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:42:56.505 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:56.526 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:42:56.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:56.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:42:56.534 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 45 16 90985 90969 126.326 0 - 0.00553317 2024-04-23T16:42:56.630 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:42:56.658 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:56.678 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:42:56.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:56.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:42:56.682 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:42:56.700 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:42:56.719 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:42:56.739 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:42:56.758 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:42:56.778 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:42:56.782 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:42:56.797 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:42:56.814 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:56.817 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:42:56.835 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:42:56.837 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:42:56.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:56.839 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:42:56.857 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:42:56.877 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:42:56.878 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:42:56.878 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:42:56.878 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:42:56.878 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:42:56.878 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:42:56.878 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:42:56.878 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:42:56.878 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:42:56.878 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:42:56.878 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:42:56.879 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:42:56.879 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:42:56.939 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:42:56.967 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:56.986 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:42:56.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:56.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:42:57.090 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:42:57.117 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:57.136 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:42:57.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:57.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:42:57.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:57 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1899137029' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:42:57.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:57.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:57.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:57.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:57.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:57.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:57.239 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:42:57.254 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:57 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1899137029' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:42:57.254 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:57.254 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:57.254 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:57.254 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:57.254 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:57.254 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:57.272 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:57.293 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:42:57.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:57.297 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:42:57.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:57 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1899137029' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:42:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:57.397 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:42:57.428 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:57.534 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 46 16 90985 90969 123.58 0 - 0.00553317 2024-04-23T16:42:57.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:57 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T16:42:57.251+0000 7f135b1c7700 -1 mon.a@0(leader).osd e562 definitely_dead 0 2024-04-23T16:42:57.733 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:42:57.733 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:42:58.164 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3 is already down. 2024-04-23T16:42:58.346 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:42:58.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:58.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:58.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:58.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:58 smithi103 ceph-mon[39629]: pgmap v1563: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 729 KiB/s wr, 11 op/s; 22696/181950 objects degraded (12.474%) 2024-04-23T16:42:58.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:58.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:58.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:58 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1403082892' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["3"]}]: dispatch 2024-04-23T16:42:58.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:58.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:58.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:58.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:58 smithi148 ceph-mon[44520]: pgmap v1563: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 729 KiB/s wr, 11 op/s; 22696/181950 objects degraded (12.474%) 2024-04-23T16:42:58.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:58.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:58.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:58 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1403082892' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["3"]}]: dispatch 2024-04-23T16:42:58.376 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:42:58.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:58.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:42:58.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:58.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:58 smithi188 ceph-mon[44101]: pgmap v1563: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 729 KiB/s wr, 11 op/s; 22696/181950 objects degraded (12.474%) 2024-04-23T16:42:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:42:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:58 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1403082892' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["3"]}]: dispatch 2024-04-23T16:42:58.483 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:42:58.510 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:58.535 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 47 16 90985 90969 120.951 0 - 0.00553317 2024-04-23T16:42:58.658 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:42:58.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:58.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:42:58.762 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:42:58.794 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:58.815 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:42:58.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:58.818 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:42:58.919 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:42:58.951 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:58.972 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:42:58.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:58.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:42:59.077 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:42:59.109 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:59.130 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:42:59.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:59.133 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:42:59.234 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:42:59.264 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:59.284 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:42:59.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:59.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:42:59.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:42:59 smithi103 ceph-mon[39629]: pgmap v1564: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 22696/181950 objects degraded (12.474%) 2024-04-23T16:42:59.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:42:59 smithi148 ceph-mon[44520]: pgmap v1564: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 22696/181950 objects degraded (12.474%) 2024-04-23T16:42:59.388 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:42:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:42:59 smithi188 ceph-mon[44101]: pgmap v1564: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 22696/181950 objects degraded (12.474%) 2024-04-23T16:42:59.417 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:59.437 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:42:59.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:59.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:42:59.535 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 48 16 90985 90969 118.431 0 - 0.00553317 2024-04-23T16:42:59.541 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:42:59.570 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:59.589 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:42:59.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:59.592 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:42:59.693 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:42:59.720 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:59.734 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:42:59.739 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:42:59.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:59.741 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:42:59.843 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:42:59.871 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 3, 4, 5, 6, 7, 8, 9, 11, 1] 2024-04-23T16:42:59.871 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd out 3 2024-04-23T16:42:59.875 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:42:59.896 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:42:59.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:42:59.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:43:00.001 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:43:00.028 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:43:00.031 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:00.053 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:43:00.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:00.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:43:00.157 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:43:00.183 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:00.531 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:43:00.535 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 49 16 90985 90969 116.014 0 - 0.00553317 2024-04-23T16:43:00.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:00.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:43:00.555 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:43:00.638 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:43:00.671 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:00.693 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:43:00.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:00.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:43:00.797 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:43:00.829 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:00.850 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:43:00.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:00.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:43:00.909 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:43:00.954 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:43:00.986 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:01.007 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:43:01.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:01.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:43:01.112 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:43:01.143 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:01.165 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:43:01.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:01.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:43:01.270 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:43:01.300 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:01.535 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 50 16 90985 90969 113.694 0 - 0.00553317 2024-04-23T16:43:02.051 INFO:teuthology.orchestra.run.smithi103.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-23T16:43:02.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:01 smithi188 ceph-mon[44101]: pgmap v1565: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 22696/181950 objects degraded (12.474%) 2024-04-23T16:43:02.321 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:01 smithi103 ceph-mon[39629]: pgmap v1565: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 22696/181950 objects degraded (12.474%) 2024-04-23T16:43:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:01 smithi148 ceph-mon[44520]: pgmap v1565: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 22696/181950 objects degraded (12.474%) 2024-04-23T16:43:02.535 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 51 16 90985 90969 111.465 0 - 0.00553317 2024-04-23T16:43:02.611 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T16:43:02.612 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-23T16:43:02.652 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:43:02.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:02.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:43:02.756 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:43:02.789 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:02.810 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:43:02.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:02.813 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:43:02.888 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:43:02.913 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:43:02.939 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:03.188 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:43:03.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:03.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:43:03.293 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:43:03.321 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:03.341 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:43:03.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:03.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:43:03.445 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:43:03.471 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:03.535 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 52 16 90985 90969 109.321 0 - 0.00553317 2024-04-23T16:43:03.619 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:43:03.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:03.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:43:03.723 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:43:03.727 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:43:03.745 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:43:03.751 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:03.764 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:43:03.785 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:43:03.804 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:43:03.823 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:43:03.842 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:43:03.862 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:43:03.875 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:43:03.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:03.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:43:03.882 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:43:03.902 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:43:03.922 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:43:03.923 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:43:03.923 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:43:03.923 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:43:03.923 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:43:03.923 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:43:03.923 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:43:03.923 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:43:03.923 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:43:03.923 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:43:03.923 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:43:03.923 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:43:03.924 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:43:03.978 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:43:04.003 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:03 smithi148 ceph-mon[44520]: pgmap v1566: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 22696/181950 objects degraded (12.474%) 2024-04-23T16:43:04.004 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:03 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3608142750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:43:04.004 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:04.004 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:04.004 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:04.005 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:04.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:03 smithi188 ceph-mon[44101]: pgmap v1566: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 22696/181950 objects degraded (12.474%) 2024-04-23T16:43:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:03 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3608142750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:43:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:04.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:04.291 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:03 smithi103 ceph-mon[39629]: pgmap v1566: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 22696/181950 objects degraded (12.474%) 2024-04-23T16:43:04.291 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:03 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3608142750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:43:04.291 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:04.292 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:04.292 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:04.388 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:43:04.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:04.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:43:04.492 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:43:04.520 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:04.536 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 53 16 90985 90969 107.258 0 - 0.00553317 2024-04-23T16:43:04.620 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:43:04.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:04.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:43:04.724 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:43:04.748 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:43:04.757 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:04.778 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:43:04.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:04.781 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:43:04.781 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:43:04.782 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:43:04.882 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:43:04.914 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:04.935 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:43:04.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:04.938 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:43:05.010 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.010 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.011 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.011 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.011 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.011 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.011 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.011 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.039 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:43:05.067 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.067 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.067 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.067 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.067 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.067 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.067 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.067 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.069 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:05.090 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:43:05.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:05.093 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:43:05.194 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:43:05.225 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:05.238 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:43:05.238 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:43:05.238 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2024-04-23T16:43:05.238 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:43:05.262 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.263 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.263 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.263 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.263 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.263 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.263 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.263 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:05.536 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 54 16 90985 90969 105.272 0 - 0.00553317 2024-04-23T16:43:05.929 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-23T16:43:06.193 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:43:06.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:05 smithi188 ceph-mon[44101]: pgmap v1567: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 22696/181950 objects degraded (12.474%) 2024-04-23T16:43:06.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:05 smithi103 ceph-mon[39629]: pgmap v1567: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 22696/181950 objects degraded (12.474%) 2024-04-23T16:43:06.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:05 smithi148 ceph-mon[44520]: pgmap v1567: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 22696/181950 objects degraded (12.474%) 2024-04-23T16:43:06.536 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 55 16 90985 90969 103.358 0 - 0.00553317 2024-04-23T16:43:06.574 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:43:06.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:06.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:43:06.678 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:43:06.708 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:06.783 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:43:06.913 INFO:teuthology.orchestra.run.smithi103.stderr:marked out osd.3. 2024-04-23T16:43:06.956 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:43:06.958 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:06 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/294035061' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-23T16:43:06.958 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:06 smithi148 ceph-mon[44520]: Client client.admin marked osd.3 out, after it was down for 23 seconds 2024-04-23T16:43:06.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:06.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:43:07.060 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:43:07.086 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:06 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T16:43:06.804+0000 7f135b1c7700 -1 mon.a@0(leader).osd e562 definitely_dead 0 2024-04-23T16:43:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:06 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/294035061' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-23T16:43:07.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:06 smithi103 ceph-mon[39629]: Client client.admin marked osd.3 out, after it was down for 23 seconds 2024-04-23T16:43:07.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:06 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/294035061' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-23T16:43:07.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:06 smithi188 ceph-mon[44101]: Client client.admin marked osd.3 out, after it was down for 23 seconds 2024-04-23T16:43:07.435 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:43:07.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:07.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:43:07.536 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 56 16 90985 90969 101.512 0 - 0.00553317 2024-04-23T16:43:07.541 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:43:07.569 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:07.672 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:43:07.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:07.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:43:07.776 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:43:07.782 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2024-04-23T16:43:07.782 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi103.front.sepia.ceph.com 2024-04-23T16:43:07.782 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.3 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2024-04-23T16:43:07.808 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:07.830 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:43:07.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:07.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:43:07.933 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:43:07.957 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:43:07.957 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:43:07.957 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2024-04-23T16:43:07.957 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2024-04-23T16:43:07.957 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:43:07.959 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:07.980 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:07 smithi148 ceph-mon[44520]: pgmap v1568: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 22696/181950 objects degraded (12.474%) 2024-04-23T16:43:07.980 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:07 smithi148 ceph-mon[44520]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T16:43:07.980 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:07 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/294035061' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-04-23T16:43:07.980 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:07 smithi148 ceph-mon[44520]: osdmap e563: 12 total, 11 up, 9 in 2024-04-23T16:43:08.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:07 smithi103 ceph-mon[39629]: pgmap v1568: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 22696/181950 objects degraded (12.474%) 2024-04-23T16:43:08.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:07 smithi103 ceph-mon[39629]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T16:43:08.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:07 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/294035061' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-04-23T16:43:08.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:07 smithi103 ceph-mon[39629]: osdmap e563: 12 total, 11 up, 9 in 2024-04-23T16:43:08.015 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:43:08.016 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.3/config 2024-04-23T16:43:08.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:08.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:43:08.120 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:43:08.148 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:07 smithi188 ceph-mon[44101]: pgmap v1568: 27 pgs: 5 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 22696/181950 objects degraded (12.474%) 2024-04-23T16:43:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:07 smithi188 ceph-mon[44101]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T16:43:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:07 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/294035061' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-04-23T16:43:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:07 smithi188 ceph-mon[44101]: osdmap e563: 12 total, 11 up, 9 in 2024-04-23T16:43:08.298 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:43:08.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:08.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:43:08.402 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:43:08.470 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:08.536 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 57 16 91173 91157 99.9377 0.47 0.0422377 0.0100023 2024-04-23T16:43:08.809 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-23T16:43:08.849 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:43:08.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:08.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:43:08.967 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:43:08.993 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:43:08.996 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:09.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:08 smithi148 ceph-mon[44520]: osdmap e564: 12 total, 11 up, 9 in 2024-04-23T16:43:09.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:08 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1049399578' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:43:09.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:08 smithi188 ceph-mon[44101]: osdmap e564: 12 total, 11 up, 9 in 2024-04-23T16:43:09.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:08 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1049399578' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:43:09.304 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:43:09.340 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:43:09.345 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:43:09.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:08 smithi103 ceph-mon[39629]: osdmap e564: 12 total, 11 up, 9 in 2024-04-23T16:43:09.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:08 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1049399578' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:43:09.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:09.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:43:09.450 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:43:09.479 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:09.500 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:43:09.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:09.503 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:43:09.536 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 58 16 91995 91979 99.1004 51.375 0.00363381 0.0100865 2024-04-23T16:43:09.604 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:43:09.632 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:09.654 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:43:09.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:09.657 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:43:09.758 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:43:09.786 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:09.812 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:43:09.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:09.815 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:43:09.916 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:43:09.936 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:09 smithi148 ceph-mon[44520]: pgmap v1571: 27 pgs: 1 peering, 4 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 22695/181950 objects degraded (12.473%) 2024-04-23T16:43:09.936 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:09 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 22695/181950 objects degraded (12.473%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:09.936 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:09 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:43:09.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:09 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1049399578' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:43:09.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:09 smithi148 ceph-mon[44520]: osdmap e565: 12 total, 11 up, 9 in 2024-04-23T16:43:10.012 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:10.041 INFO:teuthology.orchestra.run.smithi103.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:43:10.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:09 smithi188 ceph-mon[44101]: pgmap v1571: 27 pgs: 1 peering, 4 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 22695/181950 objects degraded (12.473%) 2024-04-23T16:43:10.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:09 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 22695/181950 objects degraded (12.473%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:10.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:09 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:43:10.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:09 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1049399578' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:43:10.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:09 smithi188 ceph-mon[44101]: osdmap e565: 12 total, 11 up, 9 in 2024-04-23T16:43:10.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:09 smithi103 ceph-mon[39629]: pgmap v1571: 27 pgs: 1 peering, 4 undersized+degraded+peered, 1 undersized+peered, 21 active+clean; 5.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 22695/181950 objects degraded (12.473%) 2024-04-23T16:43:10.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:09 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 22695/181950 objects degraded (12.473%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:10.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:09 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:43:10.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:09 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1049399578' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:43:10.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:09 smithi103 ceph-mon[39629]: osdmap e565: 12 total, 11 up, 9 in 2024-04-23T16:43:10.402 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:43:10.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:10.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:43:10.511 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:43:10.536 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 59 16 92840 92824 98.3158 52.8125 0.0562248 0.0101665 2024-04-23T16:43:10.577 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:11.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:10 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:43:11.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:10 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:43:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:10 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:43:11.536 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:43:11.536546 min lat: 0.000969795 max lat: 25.4619 avg lat: 0.010247 2024-04-23T16:43:11.536 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:43:11.536 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 60 16 93693 93677 97.5657 53.3125 0.0177684 0.010247 2024-04-23T16:43:12.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:11 smithi188 ceph-mon[44101]: pgmap v1573: 27 pgs: 4 active+recovering+degraded, 1 peering, 22 active+clean; 5.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 7.6 MiB/s wr, 121 op/s; 22581/183404 objects degraded (12.312%); 1.2 MiB/s, 18 objects/s recovering 2024-04-23T16:43:12.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:11 smithi103 ceph-mon[39629]: pgmap v1573: 27 pgs: 4 active+recovering+degraded, 1 peering, 22 active+clean; 5.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 7.6 MiB/s wr, 121 op/s; 22581/183404 objects degraded (12.312%); 1.2 MiB/s, 18 objects/s recovering 2024-04-23T16:43:12.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:11 smithi148 ceph-mon[44520]: pgmap v1573: 27 pgs: 4 active+recovering+degraded, 1 peering, 22 active+clean; 5.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 7.6 MiB/s wr, 121 op/s; 22581/183404 objects degraded (12.312%); 1.2 MiB/s, 18 objects/s recovering 2024-04-23T16:43:12.536 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 61 16 94596 94580 96.8915 56.4375 0.00597433 0.0103184 2024-04-23T16:43:12.996 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:43:12.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:12.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:43:13.015 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:43:13.100 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:43:13.131 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:13.192 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:43:13.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:13.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:43:13.304 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:43:13.337 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:13.360 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:43:13.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:13.363 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:43:13.370 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:43:13.395 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:43:13.417 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:43:13.442 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:43:13.464 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:43:13.486 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:43:13.495 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:13.520 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:43:13.536 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 62 16 95460 95444 96.1996 54 0.00485224 0.0103927 2024-04-23T16:43:13.544 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:43:13.565 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:43:13.587 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:43:13.611 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:43:13.636 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:43:13.636 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:43:13.636 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:43:13.636 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:43:13.636 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:43:13.636 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:43:13.636 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:43:13.636 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:43:13.636 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:43:13.636 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:43:13.637 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:43:13.637 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:43:13.637 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:43:13.649 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:43:13.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:13.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:43:13.753 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:43:13.787 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:13.813 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:43:13.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:13.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:43:13.918 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:43:13.997 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:14.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:13 smithi103 ceph-mon[39629]: pgmap v1574: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.6 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 13 MiB/s wr, 207 op/s; 22581/184442 objects degraded (12.243%); 1.2 MiB/s, 19 objects/s recovering 2024-04-23T16:43:14.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:13 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3772124232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:43:14.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:13 smithi188 ceph-mon[44101]: pgmap v1574: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.6 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 13 MiB/s wr, 207 op/s; 22581/184442 objects degraded (12.243%); 1.2 MiB/s, 19 objects/s recovering 2024-04-23T16:43:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:13 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3772124232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:43:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.361 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:13 smithi148 ceph-mon[44520]: pgmap v1574: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.6 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 13 MiB/s wr, 207 op/s; 22581/184442 objects degraded (12.243%); 1.2 MiB/s, 19 objects/s recovering 2024-04-23T16:43:14.362 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:13 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3772124232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:43:14.362 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.362 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.362 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.362 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.362 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.362 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.362 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:14.536 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 63 16 96298 96282 95.504 52.375 0.00701542 0.0104668 2024-04-23T16:43:14.860 INFO:teuthology.orchestra.run.smithi103.stdout:7.c 2024-04-23T16:43:14.861 INFO:teuthology.orchestra.run.smithi103.stdout:1.8 2024-04-23T16:43:14.861 INFO:teuthology.orchestra.run.smithi103.stdout:1.7 2024-04-23T16:43:14.861 INFO:teuthology.orchestra.run.smithi103.stdout:6.4 2024-04-23T16:43:14.861 INFO:teuthology.orchestra.run.smithi103.stdout:7.3 2024-04-23T16:43:14.861 INFO:teuthology.orchestra.run.smithi103.stdout:7.1 2024-04-23T16:43:14.861 INFO:teuthology.orchestra.run.smithi103.stdout:7.2 2024-04-23T16:43:15.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:15.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:15.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:15.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:14 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 22581/184442 objects degraded (12.243%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:15.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:15.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:15.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:15.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:15.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:14 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 22581/184442 objects degraded (12.243%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:15.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:15.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:15.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:15.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:15.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:14 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 22581/184442 objects degraded (12.243%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:15.536 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 64 16 97207 97191 94.8994 56.8125 0.00388597 0.0105354 2024-04-23T16:43:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:15 smithi103 ceph-mon[39629]: pgmap v1575: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 36 MiB/s wr, 568 op/s; 22085/190982 objects degraded (11.564%); 4.8 MiB/s, 76 objects/s recovering 2024-04-23T16:43:16.183 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.3 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op export --pgid 7.3 --file /var/log/ceph/exp.7.3.3 2024-04-23T16:43:16.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:15 smithi188 ceph-mon[44101]: pgmap v1575: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 36 MiB/s wr, 568 op/s; 22085/190982 objects degraded (11.564%); 4.8 MiB/s, 76 objects/s recovering 2024-04-23T16:43:16.238 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:43:16.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:16.246 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:15 smithi148 ceph-mon[44520]: pgmap v1575: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 36 MiB/s wr, 568 op/s; 22085/190982 objects degraded (11.564%); 4.8 MiB/s, 76 objects/s recovering 2024-04-23T16:43:16.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:43:16.348 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:43:16.380 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:16.454 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.3/config 2024-04-23T16:43:16.455 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:43:16.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:16.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:43:16.537 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 65 16 98099 98083 94.297 55.75 0.0080487 0.0106018 2024-04-23T16:43:16.563 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:43:16.593 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:16.616 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:43:16.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:16.620 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:43:16.721 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:43:16.826 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:16.898 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:43:16.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:16.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:43:17.006 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:43:17.037 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:17.393 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:43:17.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:17.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:43:17.497 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:43:17.531 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:17.537 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 66 16 98911 98895 93.6372 50.75 0.00753749 0.0106768 2024-04-23T16:43:17.885 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:43:17.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:17.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:43:17.909 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:43:17.930 INFO:teuthology.orchestra.run.smithi103.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:43:17.989 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:43:18.072 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:18.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:17 smithi103 ceph-mon[39629]: pgmap v1576: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 37 MiB/s wr, 597 op/s; 22085/192624 objects degraded (11.465%); 4.3 MiB/s, 68 objects/s recovering 2024-04-23T16:43:18.109 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:43:18.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:18.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:43:18.148 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:43:18.148 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:43:18.223 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:43:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:17 smithi188 ceph-mon[44101]: pgmap v1576: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 37 MiB/s wr, 597 op/s; 22085/192624 objects degraded (11.465%); 4.3 MiB/s, 68 objects/s recovering 2024-04-23T16:43:18.300 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:17 smithi148 ceph-mon[44520]: pgmap v1576: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 37 MiB/s wr, 597 op/s; 22085/192624 objects degraded (11.465%); 4.3 MiB/s, 68 objects/s recovering 2024-04-23T16:43:18.306 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:18.450 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:43:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:18.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:43:18.537 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 67 16 99755 99739 93.0269 52.75 0.00371397 0.0107455 2024-04-23T16:43:18.560 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:43:18.612 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:19.537 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 68 16 100678 100662 92.5072 57.6875 0.0106127 0.010807 2024-04-23T16:43:19.769 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:43:19.769 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:43:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:19 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:43:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:19 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:43:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:19 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:43:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:19 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:43:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:19 smithi103 ceph-mon[39629]: pgmap v1577: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 35 MiB/s wr, 560 op/s; 22085/193154 objects degraded (11.434%); 3.8 MiB/s, 61 objects/s recovering 2024-04-23T16:43:20.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:19 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:43:20.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:19 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:43:20.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:19 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:43:20.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:19 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 22085/192624 objects degraded (11.465%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:20.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:19 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:43:20.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:19 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:43:20.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:19 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:43:20.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:19 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:43:20.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:19 smithi148 ceph-mon[44520]: pgmap v1577: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 35 MiB/s wr, 560 op/s; 22085/193154 objects degraded (11.434%); 3.8 MiB/s, 61 objects/s recovering 2024-04-23T16:43:20.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:19 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:43:20.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:19 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:43:20.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:19 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:43:20.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:19 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 22085/192624 objects degraded (11.465%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:20.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:19 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:43:20.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:19 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:43:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:19 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:43:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:19 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:43:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:19 smithi188 ceph-mon[44101]: pgmap v1577: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 35 MiB/s wr, 560 op/s; 22085/193154 objects degraded (11.434%); 3.8 MiB/s, 61 objects/s recovering 2024-04-23T16:43:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:19 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:43:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:19 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:43:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:19 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:43:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:19 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 22085/192624 objects degraded (11.465%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:20.537 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 69 16 101534 101518 91.9418 53.5 0.00569328 0.0108743 2024-04-23T16:43:21.131 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:20 smithi148 ceph-mon[44520]: osdmap e566: 12 total, 11 up, 9 in 2024-04-23T16:43:21.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:20 smithi188 ceph-mon[44101]: osdmap e566: 12 total, 11 up, 9 in 2024-04-23T16:43:21.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:20 smithi103 ceph-mon[39629]: osdmap e566: 12 total, 11 up, 9 in 2024-04-23T16:43:21.537 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 70 16 102426 102410 91.4248 55.75 0.0145614 0.0109353 2024-04-23T16:43:21.770 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:43:22.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:21 smithi103 ceph-mon[39629]: pgmap v1579: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 53 MiB/s wr, 842 op/s; 21574/200266 objects degraded (10.773%); 6.3 MiB/s, 100 objects/s recovering 2024-04-23T16:43:22.155 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:43:22.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:22.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:43:22.167 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:21 smithi148 ceph-mon[44520]: pgmap v1579: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 53 MiB/s wr, 842 op/s; 21574/200266 objects degraded (10.773%); 6.3 MiB/s, 100 objects/s recovering 2024-04-23T16:43:22.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:21 smithi188 ceph-mon[44101]: pgmap v1579: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 53 MiB/s wr, 842 op/s; 21574/200266 objects degraded (10.773%); 6.3 MiB/s, 100 objects/s recovering 2024-04-23T16:43:22.264 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:43:22.281 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:43:22.291 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:22.315 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:43:22.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:22.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:43:22.424 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:43:22.484 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:22.522 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:43:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:22.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:43:22.537 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 71 16 103239 103223 90.8528 50.8125 0.0410554 0.0110015 2024-04-23T16:43:22.582 INFO:teuthology.orchestra.run.smithi103.stderr:Exporting 7.3 info 7.3( v 559'5656 (0'0,559'5656] local-lis/les=540/541 n=5656 ec=540/540 lis/c=540/540 les/c/f=541/541/0 sis=540) 2024-04-23T16:43:22.607 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0014ee7:::benchmark_data_smithi188_141483_object816:head# 2024-04-23T16:43:22.607 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c001b0c0:::benchmark_data_smithi188_141483_object89236:head# 2024-04-23T16:43:22.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0033545:::benchmark_data_smithi188_141483_object36957:head# 2024-04-23T16:43:22.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0036501:::benchmark_data_smithi188_141483_object50421:head# 2024-04-23T16:43:22.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0041547:::benchmark_data_smithi188_141483_object54456:head# 2024-04-23T16:43:22.609 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0048666:::benchmark_data_smithi188_141483_object56914:head# 2024-04-23T16:43:22.609 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c005bcc5:::benchmark_data_smithi188_141483_object60436:head# 2024-04-23T16:43:22.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c005d4c6:::benchmark_data_smithi188_141483_object53279:head# 2024-04-23T16:43:22.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0062d2b:::benchmark_data_smithi188_141483_object26071:head# 2024-04-23T16:43:22.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c006e672:::benchmark_data_smithi188_141483_object90167:head# 2024-04-23T16:43:22.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00716a6:::benchmark_data_smithi188_141483_object6904:head# 2024-04-23T16:43:22.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c008041d:::benchmark_data_smithi188_141483_object53053:head# 2024-04-23T16:43:22.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0087c6d:::benchmark_data_smithi188_141483_object48895:head# 2024-04-23T16:43:22.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00931eb:::benchmark_data_smithi188_141483_object1342:head# 2024-04-23T16:43:22.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c009f20e:::benchmark_data_smithi188_141483_object60741:head# 2024-04-23T16:43:22.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00a06be:::benchmark_data_smithi188_141483_object19916:head# 2024-04-23T16:43:22.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00a4f7f:::benchmark_data_smithi188_141483_object9991:head# 2024-04-23T16:43:22.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00a8c59:::benchmark_data_smithi188_141483_object885:head# 2024-04-23T16:43:22.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00aa466:::benchmark_data_smithi188_141483_object42263:head# 2024-04-23T16:43:22.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00b28d4:::benchmark_data_smithi188_141483_object22519:head# 2024-04-23T16:43:22.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00b4f6f:::benchmark_data_smithi188_141483_object47832:head# 2024-04-23T16:43:22.616 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00c7679:::benchmark_data_smithi188_141483_object20097:head# 2024-04-23T16:43:22.616 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00c9ad1:::benchmark_data_smithi188_141483_object88619:head# 2024-04-23T16:43:22.616 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00ceb65:::benchmark_data_smithi188_141483_object16034:head# 2024-04-23T16:43:22.617 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00e8da9:::benchmark_data_smithi188_141483_object84174:head# 2024-04-23T16:43:22.618 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00eb4d6:::benchmark_data_smithi188_141483_object59957:head# 2024-04-23T16:43:22.618 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00ec1b7:::benchmark_data_smithi188_141483_object25997:head# 2024-04-23T16:43:22.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00ef2df:::benchmark_data_smithi188_141483_object88755:head# 2024-04-23T16:43:22.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00f058a:::benchmark_data_smithi188_141483_object83989:head# 2024-04-23T16:43:22.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00f45c9:::benchmark_data_smithi188_141483_object50021:head# 2024-04-23T16:43:22.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00f9cdb:::benchmark_data_smithi188_141483_object79579:head# 2024-04-23T16:43:22.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00fa02c:::benchmark_data_smithi188_141483_object16661:head# 2024-04-23T16:43:22.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c00fb59a:::benchmark_data_smithi188_141483_object63835:head# 2024-04-23T16:43:22.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c010a585:::benchmark_data_smithi188_141483_object59823:head# 2024-04-23T16:43:22.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c011c7ba:::benchmark_data_smithi188_141483_object23061:head# 2024-04-23T16:43:22.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c012213b:::benchmark_data_smithi188_141483_object76057:head# 2024-04-23T16:43:22.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0122edb:::benchmark_data_smithi188_141483_object38664:head# 2024-04-23T16:43:22.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c012ef9b:::benchmark_data_smithi188_141483_object51942:head# 2024-04-23T16:43:22.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0145c4c:::benchmark_data_smithi188_141483_object32073:head# 2024-04-23T16:43:22.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0150384:::benchmark_data_smithi188_141483_object89490:head# 2024-04-23T16:43:22.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c01561b3:::benchmark_data_smithi188_141483_object20612:head# 2024-04-23T16:43:22.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c016d3f7:::benchmark_data_smithi188_141483_object90106:head# 2024-04-23T16:43:22.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c016f811:::benchmark_data_smithi188_141483_object13777:head# 2024-04-23T16:43:22.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0175f44:::benchmark_data_smithi188_141483_object50618:head# 2024-04-23T16:43:22.626 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:43:22.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c017c52a:::benchmark_data_smithi188_141483_object58694:head# 2024-04-23T16:43:22.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0181bec:::benchmark_data_smithi188_141483_object50566:head# 2024-04-23T16:43:22.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c01881ae:::benchmark_data_smithi188_141483_object90342:head# 2024-04-23T16:43:22.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c019ce59:::benchmark_data_smithi188_141483_object53057:head# 2024-04-23T16:43:22.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c01a38e1:::benchmark_data_smithi188_141483_object43109:head# 2024-04-23T16:43:22.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c01a635b:::benchmark_data_smithi188_141483_object15854:head# 2024-04-23T16:43:22.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c01b982e:::benchmark_data_smithi188_141483_object37970:head# 2024-04-23T16:43:22.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c01c57db:::benchmark_data_smithi188_141483_object13382:head# 2024-04-23T16:43:22.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c01ca240:::benchmark_data_smithi188_141483_object78388:head# 2024-04-23T16:43:22.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c01d5060:::benchmark_data_smithi188_141483_object33286:head# 2024-04-23T16:43:22.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c01e3d1f:::benchmark_data_smithi188_141483_object35911:head# 2024-04-23T16:43:22.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c01f1311:::benchmark_data_smithi188_141483_object44452:head# 2024-04-23T16:43:22.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0209bf4:::benchmark_data_smithi188_141483_object42646:head# 2024-04-23T16:43:22.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0223c40:::benchmark_data_smithi188_141483_object77983:head# 2024-04-23T16:43:22.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c02292b2:::benchmark_data_smithi188_141483_object5815:head# 2024-04-23T16:43:22.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0239cfb:::benchmark_data_smithi188_141483_object79035:head# 2024-04-23T16:43:22.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c023cb72:::benchmark_data_smithi188_141483_object32964:head# 2024-04-23T16:43:22.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c024a404:::benchmark_data_smithi188_141483_object76484:head# 2024-04-23T16:43:22.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c024bb47:::benchmark_data_smithi188_141483_object63114:head# 2024-04-23T16:43:22.635 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0256804:::benchmark_data_smithi188_141483_object87735:head# 2024-04-23T16:43:22.635 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c02685d8:::benchmark_data_smithi188_141483_object27330:head# 2024-04-23T16:43:22.635 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c026ec27:::benchmark_data_smithi188_141483_object78865:head# 2024-04-23T16:43:22.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c02873a3:::benchmark_data_smithi188_141483_object25384:head# 2024-04-23T16:43:22.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c029116b:::benchmark_data_smithi188_141483_object13159:head# 2024-04-23T16:43:22.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c02a1e8b:::benchmark_data_smithi188_141483_object89628:head# 2024-04-23T16:43:22.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c02a5de6:::benchmark_data_smithi188_141483_object90005:head# 2024-04-23T16:43:22.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c02afe67:::benchmark_data_smithi188_141483_object12697:head# 2024-04-23T16:43:22.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c02b4e9b:::benchmark_data_smithi188_141483_object5588:head# 2024-04-23T16:43:22.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c02c8542:::benchmark_data_smithi188_141483_object2479:head# 2024-04-23T16:43:22.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c02f2ced:::benchmark_data_smithi188_141483_object64039:head# 2024-04-23T16:43:22.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c032562d:::benchmark_data_smithi188_141483_object41945:head# 2024-04-23T16:43:22.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c032fd40:::benchmark_data_smithi188_141483_object63298:head# 2024-04-23T16:43:22.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c034824f:::benchmark_data_smithi188_141483_object64967:head# 2024-04-23T16:43:22.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0369b19:::benchmark_data_smithi188_141483_object18649:head# 2024-04-23T16:43:22.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c036cab1:::benchmark_data_smithi188_141483_object58447:head# 2024-04-23T16:43:22.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0392dd6:::benchmark_data_smithi188_141483_object88410:head# 2024-04-23T16:43:22.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c039ef4d:::benchmark_data_smithi188_141483_object17709:head# 2024-04-23T16:43:22.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c03a4f85:::benchmark_data_smithi188_141483_object23491:head# 2024-04-23T16:43:22.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c03aa5ff:::benchmark_data_smithi188_141483_object57901:head# 2024-04-23T16:43:22.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c03b0876:::benchmark_data_smithi188_141483_object52147:head# 2024-04-23T16:43:22.645 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c03b9aba:::benchmark_data_smithi188_141483_object76821:head# 2024-04-23T16:43:22.645 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c03cd1e6:::benchmark_data_smithi188_141483_object889:head# 2024-04-23T16:43:22.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c03d05f7:::benchmark_data_smithi188_141483_object56817:head# 2024-04-23T16:43:22.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c03dc722:::benchmark_data_smithi188_141483_object3976:head# 2024-04-23T16:43:22.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c03e23e6:::benchmark_data_smithi188_141483_object74088:head# 2024-04-23T16:43:22.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c03e532d:::benchmark_data_smithi188_141483_object26531:head# 2024-04-23T16:43:22.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c03e9250:::benchmark_data_smithi188_141483_object73594:head# 2024-04-23T16:43:22.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c03f2f2d:::benchmark_data_smithi188_141483_object31746:head# 2024-04-23T16:43:22.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c03f31b6:::benchmark_data_smithi188_141483_object19820:head# 2024-04-23T16:43:22.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0401505:::benchmark_data_smithi188_141483_object17711:head# 2024-04-23T16:43:22.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c04176a2:::benchmark_data_smithi188_141483_object68561:head# 2024-04-23T16:43:22.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c041fa77:::benchmark_data_smithi188_141483_object29463:head# 2024-04-23T16:43:22.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0432a6f:::benchmark_data_smithi188_141483_object56377:head# 2024-04-23T16:43:22.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0439760:::benchmark_data_smithi188_141483_object11310:head# 2024-04-23T16:43:22.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0439b99:::benchmark_data_smithi188_141483_object75614:head# 2024-04-23T16:43:22.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c04506c1:::benchmark_data_smithi188_141483_object90410:head# 2024-04-23T16:43:22.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c045725c:::benchmark_data_smithi188_141483_object59916:head# 2024-04-23T16:43:22.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c04583c4:::benchmark_data_smithi188_141483_object2948:head# 2024-04-23T16:43:22.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c04608d2:::benchmark_data_smithi188_141483_object24869:head# 2024-04-23T16:43:22.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c046d7ba:::benchmark_data_smithi188_141483_object12405:head# 2024-04-23T16:43:22.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0474111:::benchmark_data_smithi188_141483_object70776:head# 2024-04-23T16:43:22.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c047f59f:::benchmark_data_smithi188_141483_object21363:head# 2024-04-23T16:43:22.659 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:22.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0482d24:::benchmark_data_smithi188_141483_object79392:head# 2024-04-23T16:43:22.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0497010:::benchmark_data_smithi188_141483_object70081:head# 2024-04-23T16:43:22.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c049ac74:::benchmark_data_smithi188_141483_object71426:head# 2024-04-23T16:43:22.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c04c5efd:::benchmark_data_smithi188_141483_object89359:head# 2024-04-23T16:43:22.661 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c04c9df7:::benchmark_data_smithi188_141483_object52693:head# 2024-04-23T16:43:22.661 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c04f44e0:::benchmark_data_smithi188_141483_object46655:head# 2024-04-23T16:43:22.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c050727c:::benchmark_data_smithi188_141483_object34693:head# 2024-04-23T16:43:22.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c050a065:::benchmark_data_smithi188_141483_object269:head# 2024-04-23T16:43:22.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c05193fc:::benchmark_data_smithi188_141483_object944:head# 2024-04-23T16:43:22.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0528ad3:::benchmark_data_smithi188_141483_object41158:head# 2024-04-23T16:43:22.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c052979b:::benchmark_data_smithi188_141483_object29501:head# 2024-04-23T16:43:22.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c052e86b:::benchmark_data_smithi188_141483_object72714:head# 2024-04-23T16:43:22.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c053c7cf:::benchmark_data_smithi188_141483_object79940:head# 2024-04-23T16:43:22.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c053deb9:::benchmark_data_smithi188_141483_object26345:head# 2024-04-23T16:43:22.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c054aa4a:::benchmark_data_smithi188_141483_object2822:head# 2024-04-23T16:43:22.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c055468c:::benchmark_data_smithi188_141483_object36886:head# 2024-04-23T16:43:22.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c056494a:::benchmark_data_smithi188_141483_object36204:head# 2024-04-23T16:43:22.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c057c8b0:::benchmark_data_smithi188_141483_object79670:head# 2024-04-23T16:43:22.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c057f42a:::benchmark_data_smithi188_141483_object57078:head# 2024-04-23T16:43:22.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0586bff:::benchmark_data_smithi188_141483_object50138:head# 2024-04-23T16:43:22.668 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c058efb6:::benchmark_data_smithi188_141483_object81565:head# 2024-04-23T16:43:22.668 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c05b2ec3:::benchmark_data_smithi188_141483_object71260:head# 2024-04-23T16:43:22.669 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c05b4f3a:::benchmark_data_smithi188_141483_object17587:head# 2024-04-23T16:43:22.669 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c05c0fae:::benchmark_data_smithi188_141483_object45802:head# 2024-04-23T16:43:22.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c05eb487:::benchmark_data_smithi188_141483_object2839:head# 2024-04-23T16:43:22.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c05ed09b:::benchmark_data_smithi188_141483_object14627:head# 2024-04-23T16:43:22.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c05f42af:::benchmark_data_smithi188_141483_object31993:head# 2024-04-23T16:43:22.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c05fb9a0:::benchmark_data_smithi188_141483_object6696:head# 2024-04-23T16:43:22.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0607451:::benchmark_data_smithi188_141483_object89949:head# 2024-04-23T16:43:22.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c06271cf:::benchmark_data_smithi188_141483_object1881:head# 2024-04-23T16:43:22.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c062a81b:::benchmark_data_smithi188_141483_object76672:head# 2024-04-23T16:43:22.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c062f33e:::benchmark_data_smithi188_141483_object40352:head# 2024-04-23T16:43:22.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c063561c:::benchmark_data_smithi188_141483_object57914:head# 2024-04-23T16:43:22.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0635977:::benchmark_data_smithi188_141483_object76226:head# 2024-04-23T16:43:22.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c064b36b:::benchmark_data_smithi188_141483_object66097:head# 2024-04-23T16:43:22.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c06517ad:::benchmark_data_smithi188_141483_object1796:head# 2024-04-23T16:43:22.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0654bb3:::benchmark_data_smithi188_141483_object15344:head# 2024-04-23T16:43:22.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0662ef4:::benchmark_data_smithi188_141483_object59576:head# 2024-04-23T16:43:22.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0665859:::benchmark_data_smithi188_141483_object52570:head# 2024-04-23T16:43:22.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c066f2eb:::benchmark_data_smithi188_141483_object13212:head# 2024-04-23T16:43:22.676 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c069713e:::benchmark_data_smithi188_141483_object60999:head# 2024-04-23T16:43:22.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c069ed6b:::benchmark_data_smithi188_141483_object87529:head# 2024-04-23T16:43:22.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c06a326f:::benchmark_data_smithi188_141483_object36505:head# 2024-04-23T16:43:22.678 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c06a7de7:::benchmark_data_smithi188_141483_object57374:head# 2024-04-23T16:43:22.678 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c06adb94:::benchmark_data_smithi188_141483_object66030:head# 2024-04-23T16:43:22.679 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c06b881b:::benchmark_data_smithi188_141483_object64778:head# 2024-04-23T16:43:22.679 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c06c6204:::benchmark_data_smithi188_141483_object13892:head# 2024-04-23T16:43:22.679 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c06c6934:::benchmark_data_smithi188_141483_object6259:head# 2024-04-23T16:43:22.680 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c06cb494:::benchmark_data_smithi188_141483_object3999:head# 2024-04-23T16:43:22.680 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c06cef9a:::benchmark_data_smithi188_141483_object28666:head# 2024-04-23T16:43:22.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c06cf5ce:::benchmark_data_smithi188_141483_object41369:head# 2024-04-23T16:43:22.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c06df36f:::benchmark_data_smithi188_141483_object4830:head# 2024-04-23T16:43:22.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c06e36e3:::benchmark_data_smithi188_141483_object40978:head# 2024-04-23T16:43:22.683 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c06e41c0:::benchmark_data_smithi188_141483_object35214:head# 2024-04-23T16:43:22.683 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c072f733:::benchmark_data_smithi188_141483_object54824:head# 2024-04-23T16:43:22.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0734453:::benchmark_data_smithi188_141483_object32134:head# 2024-04-23T16:43:22.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0739af1:::benchmark_data_smithi188_141483_object29313:head# 2024-04-23T16:43:22.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0744510:::benchmark_data_smithi188_141483_object28053:head# 2024-04-23T16:43:22.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0746b13:::benchmark_data_smithi188_141483_object7348:head# 2024-04-23T16:43:22.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0753543:::benchmark_data_smithi188_141483_object69783:head# 2024-04-23T16:43:22.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0755efd:::benchmark_data_smithi188_141483_object50029:head# 2024-04-23T16:43:22.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0756248:::benchmark_data_smithi188_141483_object87163:head# 2024-04-23T16:43:22.689 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0779fa7:::benchmark_data_smithi188_141483_object5181:head# 2024-04-23T16:43:22.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c079e013:::benchmark_data_smithi188_141483_object1867:head# 2024-04-23T16:43:22.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c07ac1b7:::benchmark_data_smithi188_141483_object69837:head# 2024-04-23T16:43:22.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c07b3c2e:::benchmark_data_smithi188_141483_object591:head# 2024-04-23T16:43:22.691 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c07b7c6e:::benchmark_data_smithi188_141483_object68719:head# 2024-04-23T16:43:22.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c07c9ada:::benchmark_data_smithi188_141483_object27789:head# 2024-04-23T16:43:22.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c07cdb06:::benchmark_data_smithi188_141483_object53877:head# 2024-04-23T16:43:22.692 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c07cf00e:::benchmark_data_smithi188_141483_object50161:head# 2024-04-23T16:43:22.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c07d416f:::benchmark_data_smithi188_141483_object23504:head# 2024-04-23T16:43:22.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c07d444f:::benchmark_data_smithi188_141483_object53788:head# 2024-04-23T16:43:22.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c07d4998:::benchmark_data_smithi188_141483_object50246:head# 2024-04-23T16:43:22.694 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c07d7fab:::benchmark_data_smithi188_141483_object65227:head# 2024-04-23T16:43:22.694 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c07dc50c:::benchmark_data_smithi188_141483_object40692:head# 2024-04-23T16:43:22.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c07e5645:::benchmark_data_smithi188_141483_object46680:head# 2024-04-23T16:43:22.695 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c07f6007:::benchmark_data_smithi188_141483_object11307:head# 2024-04-23T16:43:22.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c07fafe8:::benchmark_data_smithi188_141483_object76617:head# 2024-04-23T16:43:22.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c07fb53b:::benchmark_data_smithi188_141483_object37325:head# 2024-04-23T16:43:22.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c08063e0:::benchmark_data_smithi188_141483_object50440:head# 2024-04-23T16:43:22.697 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c080652f:::benchmark_data_smithi188_141483_object89207:head# 2024-04-23T16:43:22.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0818950:::benchmark_data_smithi188_141483_object49487:head# 2024-04-23T16:43:22.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c08297df:::benchmark_data_smithi188_141483_object12763:head# 2024-04-23T16:43:22.699 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c082fc04:::benchmark_data_smithi188_141483_object78059:head# 2024-04-23T16:43:22.699 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c083bafb:::benchmark_data_smithi188_141483_object72159:head# 2024-04-23T16:43:22.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c08430ab:::benchmark_data_smithi188_141483_object88875:head# 2024-04-23T16:43:22.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0843f00:::benchmark_data_smithi188_141483_object57764:head# 2024-04-23T16:43:22.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c08576ea:::benchmark_data_smithi188_141483_object61376:head# 2024-04-23T16:43:22.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c086d349:::benchmark_data_smithi188_141483_object86220:head# 2024-04-23T16:43:22.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c086fb7e:::benchmark_data_smithi188_141483_object70069:head# 2024-04-23T16:43:22.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c088b032:::benchmark_data_smithi188_141483_object44379:head# 2024-04-23T16:43:22.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c088c04b:::benchmark_data_smithi188_141483_object21334:head# 2024-04-23T16:43:22.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c08af0ee:::benchmark_data_smithi188_141483_object81745:head# 2024-04-23T16:43:22.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c08bf463:::benchmark_data_smithi188_141483_object10779:head# 2024-04-23T16:43:22.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c08d5eb6:::benchmark_data_smithi188_141483_object42346:head# 2024-04-23T16:43:22.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c08d7aa0:::benchmark_data_smithi188_141483_object67216:head# 2024-04-23T16:43:22.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c08ffdb1:::benchmark_data_smithi188_141483_object66747:head# 2024-04-23T16:43:22.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c09124af:::benchmark_data_smithi188_141483_object72302:head# 2024-04-23T16:43:22.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c09162fb:::benchmark_data_smithi188_141483_object52468:head# 2024-04-23T16:43:22.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c091b91b:::benchmark_data_smithi188_141483_object480:head# 2024-04-23T16:43:22.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0921630:::benchmark_data_smithi188_141483_object4433:head# 2024-04-23T16:43:22.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c094d356:::benchmark_data_smithi188_141483_object1970:head# 2024-04-23T16:43:22.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c094dc23:::benchmark_data_smithi188_141483_object28220:head# 2024-04-23T16:43:22.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c097471e:::benchmark_data_smithi188_141483_object68882:head# 2024-04-23T16:43:22.709 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c097c3bd:::benchmark_data_smithi188_141483_object54825:head# 2024-04-23T16:43:22.709 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0981073:::benchmark_data_smithi188_141483_object14474:head# 2024-04-23T16:43:22.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c098b0a0:::benchmark_data_smithi188_141483_object20064:head# 2024-04-23T16:43:22.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c098c22a:::benchmark_data_smithi188_141483_object19543:head# 2024-04-23T16:43:22.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c09963a1:::benchmark_data_smithi188_141483_object22591:head# 2024-04-23T16:43:22.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0999d24:::benchmark_data_smithi188_141483_object24882:head# 2024-04-23T16:43:22.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c099fa2b:::benchmark_data_smithi188_141483_object32620:head# 2024-04-23T16:43:22.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c09a1107:::benchmark_data_smithi188_141483_object3510:head# 2024-04-23T16:43:22.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c09a82ea:::benchmark_data_smithi188_141483_object70942:head# 2024-04-23T16:43:22.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c09aed36:::benchmark_data_smithi188_141483_object8969:head# 2024-04-23T16:43:22.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c09bb1fb:::benchmark_data_smithi188_141483_object60794:head# 2024-04-23T16:43:22.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c09bc32a:::benchmark_data_smithi188_141483_object82714:head# 2024-04-23T16:43:22.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c09cdb81:::benchmark_data_smithi188_141483_object22749:head# 2024-04-23T16:43:22.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c09dbef9:::benchmark_data_smithi188_141483_object81866:head# 2024-04-23T16:43:22.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c09ecc2b:::benchmark_data_smithi188_141483_object54264:head# 2024-04-23T16:43:22.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c09f2f5c:::benchmark_data_smithi188_141483_object33196:head# 2024-04-23T16:43:22.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c09f8f64:::benchmark_data_smithi188_141483_object1828:head# 2024-04-23T16:43:22.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c09ffa03:::benchmark_data_smithi188_141483_object59177:head# 2024-04-23T16:43:22.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0a05eb3:::benchmark_data_smithi188_141483_object59104:head# 2024-04-23T16:43:22.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0a07dda:::benchmark_data_smithi188_141483_object13834:head# 2024-04-23T16:43:22.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0a0ed19:::benchmark_data_smithi188_141483_object5451:head# 2024-04-23T16:43:22.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0a165d5:::benchmark_data_smithi188_141483_object65443:head# 2024-04-23T16:43:22.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0a1b974:::benchmark_data_smithi188_141483_object2756:head# 2024-04-23T16:43:22.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0a1e452:::benchmark_data_smithi188_141483_object56780:head# 2024-04-23T16:43:22.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0a20f3a:::benchmark_data_smithi188_141483_object3334:head# 2024-04-23T16:43:22.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0a2df9f:::benchmark_data_smithi188_141483_object35627:head# 2024-04-23T16:43:22.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0a4189a:::benchmark_data_smithi188_141483_object62497:head# 2024-04-23T16:43:22.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0a4e7fd:::benchmark_data_smithi188_141483_object84501:head# 2024-04-23T16:43:22.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0a5818e:::benchmark_data_smithi188_141483_object26647:head# 2024-04-23T16:43:22.725 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0a72ccf:::benchmark_data_smithi188_141483_object57053:head# 2024-04-23T16:43:22.725 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0a763b5:::benchmark_data_smithi188_141483_object60700:head# 2024-04-23T16:43:22.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0a8abd5:::benchmark_data_smithi188_141483_object81593:head# 2024-04-23T16:43:22.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0a8f36a:::benchmark_data_smithi188_141483_object687:head# 2024-04-23T16:43:22.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0aa46b5:::benchmark_data_smithi188_141483_object64388:head# 2024-04-23T16:43:22.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0ab3133:::benchmark_data_smithi188_141483_object66636:head# 2024-04-23T16:43:22.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0ab86bf:::benchmark_data_smithi188_141483_object89968:head# 2024-04-23T16:43:22.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0ac9ba6:::benchmark_data_smithi188_141483_object18576:head# 2024-04-23T16:43:22.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0adec1f:::benchmark_data_smithi188_141483_object54494:head# 2024-04-23T16:43:22.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0aed50b:::benchmark_data_smithi188_141483_object55218:head# 2024-04-23T16:43:22.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0b06925:::benchmark_data_smithi188_141483_object88546:head# 2024-04-23T16:43:22.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0b13903:::benchmark_data_smithi188_141483_object27493:head# 2024-04-23T16:43:22.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0b239cf:::benchmark_data_smithi188_141483_object4641:head# 2024-04-23T16:43:22.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0b26f3d:::benchmark_data_smithi188_141483_object74333:head# 2024-04-23T16:43:22.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0b407fe:::benchmark_data_smithi188_141483_object51933:head# 2024-04-23T16:43:22.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0b5070d:::benchmark_data_smithi188_141483_object88153:head# 2024-04-23T16:43:22.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0b5a267:::benchmark_data_smithi188_141483_object84567:head# 2024-04-23T16:43:22.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0b5b6e7:::benchmark_data_smithi188_141483_object55292:head# 2024-04-23T16:43:22.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0b5cf1b:::benchmark_data_smithi188_141483_object55042:head# 2024-04-23T16:43:22.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0b6b2e4:::benchmark_data_smithi188_141483_object47126:head# 2024-04-23T16:43:22.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0b71856:::benchmark_data_smithi188_141483_object74609:head# 2024-04-23T16:43:22.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0b8235c:::benchmark_data_smithi188_141483_object70332:head# 2024-04-23T16:43:22.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0b9e265:::benchmark_data_smithi188_141483_object14814:head# 2024-04-23T16:43:22.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0ba6064:::benchmark_data_smithi188_141483_object77448:head# 2024-04-23T16:43:22.737 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0bac161:::benchmark_data_smithi188_141483_object21121:head# 2024-04-23T16:43:22.737 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0bbbe20:::benchmark_data_smithi188_141483_object54929:head# 2024-04-23T16:43:22.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0bc67c5:::benchmark_data_smithi188_141483_object34718:head# 2024-04-23T16:43:22.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0bd8610:::benchmark_data_smithi188_141483_object56964:head# 2024-04-23T16:43:22.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0be33bd:::benchmark_data_smithi188_141483_object10942:head# 2024-04-23T16:43:22.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0be4027:::benchmark_data_smithi188_141483_object67270:head# 2024-04-23T16:43:22.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0be497c:::benchmark_data_smithi188_141483_object57928:head# 2024-04-23T16:43:22.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0be6115:::benchmark_data_smithi188_141483_object65400:head# 2024-04-23T16:43:22.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0befb7d:::benchmark_data_smithi188_141483_object56164:head# 2024-04-23T16:43:22.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0bf8553:::benchmark_data_smithi188_141483_object43556:head# 2024-04-23T16:43:22.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0c0106b:::benchmark_data_smithi188_141483_object81581:head# 2024-04-23T16:43:22.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0c20a18:::benchmark_data_smithi188_141483_object29716:head# 2024-04-23T16:43:22.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0c4f9bf:::benchmark_data_smithi188_141483_object70672:head# 2024-04-23T16:43:22.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0c51140:::benchmark_data_smithi188_141483_object51945:head# 2024-04-23T16:43:22.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0c7d1cb:::benchmark_data_smithi188_141483_object37573:head# 2024-04-23T16:43:22.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0c7da3b:::benchmark_data_smithi188_141483_object41049:head# 2024-04-23T16:43:22.744 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0c7dbbf:::benchmark_data_smithi188_141483_object54588:head# 2024-04-23T16:43:22.744 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0c836c7:::benchmark_data_smithi188_141483_object10060:head# 2024-04-23T16:43:22.745 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0c99494:::benchmark_data_smithi188_141483_object51556:head# 2024-04-23T16:43:22.745 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0c9b448:::benchmark_data_smithi188_141483_object59080:head# 2024-04-23T16:43:22.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0c9d886:::benchmark_data_smithi188_141483_object67822:head# 2024-04-23T16:43:22.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0ca264f:::benchmark_data_smithi188_141483_object52309:head# 2024-04-23T16:43:22.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0cb84a2:::benchmark_data_smithi188_141483_object10231:head# 2024-04-23T16:43:22.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0ce1a48:::benchmark_data_smithi188_141483_object10447:head# 2024-04-23T16:43:22.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0cf786b:::benchmark_data_smithi188_141483_object34325:head# 2024-04-23T16:43:22.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0d0633d:::benchmark_data_smithi188_141483_object7366:head# 2024-04-23T16:43:22.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0d0d8c0:::benchmark_data_smithi188_141483_object40639:head# 2024-04-23T16:43:22.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0d218d7:::benchmark_data_smithi188_141483_object73716:head# 2024-04-23T16:43:22.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0d2968b:::benchmark_data_smithi188_141483_object12687:head# 2024-04-23T16:43:22.750 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0d44730:::benchmark_data_smithi188_141483_object12312:head# 2024-04-23T16:43:22.750 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0d5172e:::benchmark_data_smithi188_141483_object13470:head# 2024-04-23T16:43:22.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0d78545:::benchmark_data_smithi188_141483_object23704:head# 2024-04-23T16:43:22.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0d7fd39:::benchmark_data_smithi188_141483_object48139:head# 2024-04-23T16:43:22.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0d824d1:::benchmark_data_smithi188_141483_object88121:head# 2024-04-23T16:43:22.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0d85b9e:::benchmark_data_smithi188_141483_object40202:head# 2024-04-23T16:43:22.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0d8d18c:::benchmark_data_smithi188_141483_object71829:head# 2024-04-23T16:43:22.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0da9b78:::benchmark_data_smithi188_141483_object86020:head# 2024-04-23T16:43:22.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0dae813:::benchmark_data_smithi188_141483_object31400:head# 2024-04-23T16:43:22.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0db5094:::benchmark_data_smithi188_141483_object86674:head# 2024-04-23T16:43:22.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0dcdb6d:::benchmark_data_smithi188_141483_object45313:head# 2024-04-23T16:43:22.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0dd42ee:::benchmark_data_smithi188_141483_object50172:head# 2024-04-23T16:43:22.756 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0dda241:::benchmark_data_smithi188_141483_object32118:head# 2024-04-23T16:43:22.756 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0de4b73:::benchmark_data_smithi188_141483_object44730:head# 2024-04-23T16:43:22.757 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0dec315:::benchmark_data_smithi188_141483_object27554:head# 2024-04-23T16:43:22.757 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0deca29:::benchmark_data_smithi188_141483_object74635:head# 2024-04-23T16:43:22.757 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0df33ed:::benchmark_data_smithi188_141483_object63247:head# 2024-04-23T16:43:22.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e02ba9:::benchmark_data_smithi188_141483_object40719:head# 2024-04-23T16:43:22.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e09ac8:::benchmark_data_smithi188_141483_object36172:head# 2024-04-23T16:43:22.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e11bd6:::benchmark_data_smithi188_141483_object34522:head# 2024-04-23T16:43:22.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e165b0:::benchmark_data_smithi188_141483_object102:head# 2024-04-23T16:43:22.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e1f076:::benchmark_data_smithi188_141483_object31721:head# 2024-04-23T16:43:22.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e200cd:::benchmark_data_smithi188_141483_object63324:head# 2024-04-23T16:43:22.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e2051f:::benchmark_data_smithi188_141483_object23379:head# 2024-04-23T16:43:22.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e2e1f8:::benchmark_data_smithi188_141483_object11433:head# 2024-04-23T16:43:22.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e31f9e:::benchmark_data_smithi188_141483_object56855:head# 2024-04-23T16:43:22.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e3263c:::benchmark_data_smithi188_141483_object21715:head# 2024-04-23T16:43:22.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e37dd5:::benchmark_data_smithi188_141483_object62063:head# 2024-04-23T16:43:22.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e3b2a4:::benchmark_data_smithi188_141483_object82043:head# 2024-04-23T16:43:22.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e3dc27:::benchmark_data_smithi188_141483_object5361:head# 2024-04-23T16:43:22.764 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e4e185:::benchmark_data_smithi188_141483_object74173:head# 2024-04-23T16:43:22.764 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e5886e:::benchmark_data_smithi188_141483_object47500:head# 2024-04-23T16:43:22.764 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e626e6:::benchmark_data_smithi188_141483_object1771:head# 2024-04-23T16:43:22.765 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e65380:::benchmark_data_smithi188_141483_object17547:head# 2024-04-23T16:43:22.765 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e6a23a:::benchmark_data_smithi188_141483_object50759:head# 2024-04-23T16:43:22.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e6ff60:::benchmark_data_smithi188_141483_object36183:head# 2024-04-23T16:43:22.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e741d5:::benchmark_data_smithi188_141483_object85117:head# 2024-04-23T16:43:22.767 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e762cb:::benchmark_data_smithi188_141483_object78817:head# 2024-04-23T16:43:22.767 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e7c173:::benchmark_data_smithi188_141483_object77861:head# 2024-04-23T16:43:22.767 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0e835d5:::benchmark_data_smithi188_141483_object45999:head# 2024-04-23T16:43:22.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0ea7a1c:::benchmark_data_smithi188_141483_object47646:head# 2024-04-23T16:43:22.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0ea9abb:::benchmark_data_smithi188_141483_object7865:head# 2024-04-23T16:43:22.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0eb4c90:::benchmark_data_smithi188_141483_object62556:head# 2024-04-23T16:43:22.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0ebbe20:::benchmark_data_smithi188_141483_object45565:head# 2024-04-23T16:43:22.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0eece46:::benchmark_data_smithi188_141483_object85115:head# 2024-04-23T16:43:22.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0ef3b12:::benchmark_data_smithi188_141483_object69492:head# 2024-04-23T16:43:22.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0ef4ee0:::benchmark_data_smithi188_141483_object5535:head# 2024-04-23T16:43:22.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0ef5b71:::benchmark_data_smithi188_141483_object57762:head# 2024-04-23T16:43:22.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0f0114b:::benchmark_data_smithi188_141483_object61492:head# 2024-04-23T16:43:22.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0f0e021:::benchmark_data_smithi188_141483_object54906:head# 2024-04-23T16:43:22.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0f114ad:::benchmark_data_smithi188_141483_object63626:head# 2024-04-23T16:43:22.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0f1e478:::benchmark_data_smithi188_141483_object21032:head# 2024-04-23T16:43:22.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0f251df:::benchmark_data_smithi188_141483_object22292:head# 2024-04-23T16:43:22.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0f2cfab:::benchmark_data_smithi188_141483_object30597:head# 2024-04-23T16:43:22.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0f345e1:::benchmark_data_smithi188_141483_object88462:head# 2024-04-23T16:43:22.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0f44f99:::benchmark_data_smithi188_141483_object79711:head# 2024-04-23T16:43:22.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0f6791c:::benchmark_data_smithi188_141483_object3656:head# 2024-04-23T16:43:22.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0f8e099:::benchmark_data_smithi188_141483_object23744:head# 2024-04-23T16:43:22.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0f94892:::benchmark_data_smithi188_141483_object2089:head# 2024-04-23T16:43:22.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0f94f73:::benchmark_data_smithi188_141483_object32035:head# 2024-04-23T16:43:22.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0fa6c2f:::benchmark_data_smithi188_141483_object20983:head# 2024-04-23T16:43:22.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0fc8bdb:::benchmark_data_smithi188_141483_object57598:head# 2024-04-23T16:43:22.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c0fe06db:::benchmark_data_smithi188_141483_object25187:head# 2024-04-23T16:43:22.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1012f91:::benchmark_data_smithi188_141483_object28975:head# 2024-04-23T16:43:22.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c102b083:::benchmark_data_smithi188_141483_object51376:head# 2024-04-23T16:43:22.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1050c65:::benchmark_data_smithi188_141483_object24375:head# 2024-04-23T16:43:22.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1055e13:::benchmark_data_smithi188_141483_object85555:head# 2024-04-23T16:43:22.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c105abde:::benchmark_data_smithi188_141483_object19580:head# 2024-04-23T16:43:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c105fcef:::benchmark_data_smithi188_141483_object67184:head# 2024-04-23T16:43:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1083332:::benchmark_data_smithi188_141483_object83871:head# 2024-04-23T16:43:22.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1085113:::benchmark_data_smithi188_141483_object49815:head# 2024-04-23T16:43:22.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c108bec4:::benchmark_data_smithi188_141483_object67787:head# 2024-04-23T16:43:22.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c10acab2:::benchmark_data_smithi188_141483_object58897:head# 2024-04-23T16:43:22.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c10c02ff:::benchmark_data_smithi188_141483_object18410:head# 2024-04-23T16:43:22.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c10cda7a:::benchmark_data_smithi188_141483_object24423:head# 2024-04-23T16:43:22.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c10cf739:::benchmark_data_smithi188_141483_object19835:head# 2024-04-23T16:43:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c10d5a46:::benchmark_data_smithi188_141483_object62142:head# 2024-04-23T16:43:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c10d9f70:::benchmark_data_smithi188_141483_object90674:head# 2024-04-23T16:43:22.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c10de4f6:::benchmark_data_smithi188_141483_object70649:head# 2024-04-23T16:43:22.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c10ebac9:::benchmark_data_smithi188_141483_object36298:head# 2024-04-23T16:43:22.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c10f00be:::benchmark_data_smithi188_141483_object63284:head# 2024-04-23T16:43:22.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c10f693e:::benchmark_data_smithi188_141483_object70247:head# 2024-04-23T16:43:22.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1103b3a:::benchmark_data_smithi188_141483_object65577:head# 2024-04-23T16:43:22.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1121a93:::benchmark_data_smithi188_141483_object66109:head# 2024-04-23T16:43:22.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c112bfff:::benchmark_data_smithi188_141483_object291:head# 2024-04-23T16:43:22.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c113008a:::benchmark_data_smithi188_141483_object76767:head# 2024-04-23T16:43:22.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1132e76:::benchmark_data_smithi188_141483_object11656:head# 2024-04-23T16:43:22.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1134c72:::benchmark_data_smithi188_141483_object51019:head# 2024-04-23T16:43:22.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1137e4c:::benchmark_data_smithi188_141483_object47286:head# 2024-04-23T16:43:22.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c113a14e:::benchmark_data_smithi188_141483_object83231:head# 2024-04-23T16:43:22.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1143c7b:::benchmark_data_smithi188_141483_object11883:head# 2024-04-23T16:43:22.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1145c0f:::benchmark_data_smithi188_141483_object90341:head# 2024-04-23T16:43:22.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c11494f0:::benchmark_data_smithi188_141483_object53465:head# 2024-04-23T16:43:22.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c114ca25:::benchmark_data_smithi188_141483_object3862:head# 2024-04-23T16:43:22.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c114d44d:::benchmark_data_smithi188_141483_object86978:head# 2024-04-23T16:43:22.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1153f01:::benchmark_data_smithi188_141483_object14747:head# 2024-04-23T16:43:22.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1154c14:::benchmark_data_smithi188_141483_object65667:head# 2024-04-23T16:43:22.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c115a02f:::benchmark_data_smithi188_141483_object4887:head# 2024-04-23T16:43:22.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c115d8e6:::benchmark_data_smithi188_141483_object51729:head# 2024-04-23T16:43:22.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1161d54:::benchmark_data_smithi188_141483_object52531:head# 2024-04-23T16:43:22.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c11802b4:::benchmark_data_smithi188_141483_object90317:head# 2024-04-23T16:43:22.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c11cae2a:::benchmark_data_smithi188_141483_object54878:head# 2024-04-23T16:43:22.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c11cf9fb:::benchmark_data_smithi188_141483_object48527:head# 2024-04-23T16:43:22.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c11d6a97:::benchmark_data_smithi188_141483_object47218:head# 2024-04-23T16:43:22.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c11d93ce:::benchmark_data_smithi188_141483_object55460:head# 2024-04-23T16:43:22.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c11e8bb4:::benchmark_data_smithi188_141483_object16880:head# 2024-04-23T16:43:22.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c11ef7f7:::benchmark_data_smithi188_141483_object64137:head# 2024-04-23T16:43:22.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c122b90e:::benchmark_data_smithi188_141483_object31828:head# 2024-04-23T16:43:22.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1230133:::benchmark_data_smithi188_141483_object17377:head# 2024-04-23T16:43:22.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c12378a3:::benchmark_data_smithi188_141483_object42110:head# 2024-04-23T16:43:22.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c123abcc:::benchmark_data_smithi188_141483_object46451:head# 2024-04-23T16:43:22.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1246490:::benchmark_data_smithi188_141483_object23486:head# 2024-04-23T16:43:22.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c124a0fe:::benchmark_data_smithi188_141483_object31338:head# 2024-04-23T16:43:22.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c125088a:::benchmark_data_smithi188_141483_object70234:head# 2024-04-23T16:43:22.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c125396b:::benchmark_data_smithi188_141483_object83203:head# 2024-04-23T16:43:22.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c125451d:::benchmark_data_smithi188_141483_object57460:head# 2024-04-23T16:43:22.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c125952d:::benchmark_data_smithi188_141483_object40390:head# 2024-04-23T16:43:22.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c127be1f:::benchmark_data_smithi188_141483_object53688:head# 2024-04-23T16:43:22.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c128c2da:::benchmark_data_smithi188_141483_object48063:head# 2024-04-23T16:43:22.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c128e669:::benchmark_data_smithi188_141483_object5722:head# 2024-04-23T16:43:22.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1299037:::benchmark_data_smithi188_141483_object2727:head# 2024-04-23T16:43:22.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c12b373a:::benchmark_data_smithi188_141483_object10400:head# 2024-04-23T16:43:22.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c12b660d:::benchmark_data_smithi188_141483_object29580:head# 2024-04-23T16:43:22.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c12b7761:::benchmark_data_smithi188_141483_object54522:head# 2024-04-23T16:43:22.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c12c0a58:::benchmark_data_smithi188_141483_object48552:head# 2024-04-23T16:43:22.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1305885:::benchmark_data_smithi188_141483_object52663:head# 2024-04-23T16:43:22.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1306a72:::benchmark_data_smithi188_141483_object15338:head# 2024-04-23T16:43:22.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1309e39:::benchmark_data_smithi188_141483_object50042:head# 2024-04-23T16:43:22.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1311430:::benchmark_data_smithi188_141483_object17411:head# 2024-04-23T16:43:22.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c131af80:::benchmark_data_smithi188_141483_object2907:head# 2024-04-23T16:43:22.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c134c8d3:::benchmark_data_smithi188_141483_object3538:head# 2024-04-23T16:43:22.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c13540a4:::benchmark_data_smithi188_141483_object2990:head# 2024-04-23T16:43:22.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c135ea1a:::benchmark_data_smithi188_141483_object75223:head# 2024-04-23T16:43:22.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c13763ee:::benchmark_data_smithi188_141483_object49196:head# 2024-04-23T16:43:22.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c137b535:::benchmark_data_smithi188_141483_object11237:head# 2024-04-23T16:43:22.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c137b93c:::benchmark_data_smithi188_141483_object45609:head# 2024-04-23T16:43:22.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1387782:::benchmark_data_smithi188_141483_object6091:head# 2024-04-23T16:43:22.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c138919c:::benchmark_data_smithi188_141483_object60148:head# 2024-04-23T16:43:22.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c138f0c6:::benchmark_data_smithi188_141483_object76418:head# 2024-04-23T16:43:22.814 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c13945ac:::benchmark_data_smithi188_141483_object68153:head# 2024-04-23T16:43:22.814 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1396309:::benchmark_data_smithi188_141483_object41810:head# 2024-04-23T16:43:22.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c13a8c75:::benchmark_data_smithi188_141483_object16979:head# 2024-04-23T16:43:22.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c13b9f6c:::benchmark_data_smithi188_141483_object86435:head# 2024-04-23T16:43:22.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c13c56aa:::benchmark_data_smithi188_141483_object86901:head# 2024-04-23T16:43:22.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c13cac0d:::benchmark_data_smithi188_141483_object64123:head# 2024-04-23T16:43:22.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c13d8eed:::benchmark_data_smithi188_141483_object69344:head# 2024-04-23T16:43:22.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c13e2fff:::benchmark_data_smithi188_141483_object60758:head# 2024-04-23T16:43:22.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c13f8007:::benchmark_data_smithi188_141483_object41173:head# 2024-04-23T16:43:22.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c13f91a7:::benchmark_data_smithi188_141483_object43242:head# 2024-04-23T16:43:22.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c140f547:::benchmark_data_smithi188_141483_object44060:head# 2024-04-23T16:43:22.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1413051:::benchmark_data_smithi188_141483_object32722:head# 2024-04-23T16:43:22.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c141c324:::benchmark_data_smithi188_141483_object50340:head# 2024-04-23T16:43:22.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c142c7dc:::benchmark_data_smithi188_141483_object88618:head# 2024-04-23T16:43:22.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1447214:::benchmark_data_smithi188_141483_object15635:head# 2024-04-23T16:43:22.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1457917:::benchmark_data_smithi188_141483_object29557:head# 2024-04-23T16:43:22.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c145d348:::benchmark_data_smithi188_141483_object14542:head# 2024-04-23T16:43:22.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c145ddab:::benchmark_data_smithi188_141483_object77320:head# 2024-04-23T16:43:22.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c146132e:::benchmark_data_smithi188_141483_object57720:head# 2024-04-23T16:43:22.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1464f83:::benchmark_data_smithi188_141483_object52125:head# 2024-04-23T16:43:22.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1482cae:::benchmark_data_smithi188_141483_object29647:head# 2024-04-23T16:43:22.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1483f3d:::benchmark_data_smithi188_141483_object78510:head# 2024-04-23T16:43:22.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c148c32b:::benchmark_data_smithi188_141483_object75027:head# 2024-04-23T16:43:22.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c149399f:::benchmark_data_smithi188_141483_object79176:head# 2024-04-23T16:43:22.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1495057:::benchmark_data_smithi188_141483_object49478:head# 2024-04-23T16:43:22.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c149993c:::benchmark_data_smithi188_141483_object23630:head# 2024-04-23T16:43:22.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c14a2355:::benchmark_data_smithi188_141483_object60624:head# 2024-04-23T16:43:22.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c14ae7e7:::benchmark_data_smithi188_141483_object31076:head# 2024-04-23T16:43:22.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c14b93ed:::benchmark_data_smithi188_141483_object7111:head# 2024-04-23T16:43:22.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c14d3402:::benchmark_data_smithi188_141483_object51528:head# 2024-04-23T16:43:22.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c14dea65:::benchmark_data_smithi188_141483_object22786:head# 2024-04-23T16:43:22.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c14debb3:::benchmark_data_smithi188_141483_object64669:head# 2024-04-23T16:43:22.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c14e8dc2:::benchmark_data_smithi188_141483_object85163:head# 2024-04-23T16:43:22.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c14eb0dc:::benchmark_data_smithi188_141483_object44006:head# 2024-04-23T16:43:22.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c14ee0aa:::benchmark_data_smithi188_141483_object24493:head# 2024-04-23T16:43:22.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c14ee6fc:::benchmark_data_smithi188_141483_object55944:head# 2024-04-23T16:43:22.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c14f0a2e:::benchmark_data_smithi188_141483_object29621:head# 2024-04-23T16:43:22.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c14fd073:::benchmark_data_smithi188_141483_object77498:head# 2024-04-23T16:43:22.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c14fde02:::benchmark_data_smithi188_141483_object85697:head# 2024-04-23T16:43:22.832 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c14fecf0:::benchmark_data_smithi188_141483_object89850:head# 2024-04-23T16:43:22.832 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c150c4d5:::benchmark_data_smithi188_141483_object6820:head# 2024-04-23T16:43:22.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1517679:::benchmark_data_smithi188_141483_object23690:head# 2024-04-23T16:43:22.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c151be64:::benchmark_data_smithi188_141483_object31933:head# 2024-04-23T16:43:22.834 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1522e99:::benchmark_data_smithi188_141483_object8329:head# 2024-04-23T16:43:22.834 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1529807:::benchmark_data_smithi188_141483_object66326:head# 2024-04-23T16:43:22.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1530758:::benchmark_data_smithi188_141483_object69925:head# 2024-04-23T16:43:22.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c15375df:::benchmark_data_smithi188_141483_object16647:head# 2024-04-23T16:43:22.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1541afc:::benchmark_data_smithi188_141483_object57588:head# 2024-04-23T16:43:22.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c15571d0:::benchmark_data_smithi188_141483_object9289:head# 2024-04-23T16:43:22.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c155a5f6:::benchmark_data_smithi188_141483_object85216:head# 2024-04-23T16:43:22.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1561f8b:::benchmark_data_smithi188_141483_object66964:head# 2024-04-23T16:43:22.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c156d203:::benchmark_data_smithi188_141483_object17905:head# 2024-04-23T16:43:22.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c156eb92:::benchmark_data_smithi188_141483_object29997:head# 2024-04-23T16:43:22.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c157301f:::benchmark_data_smithi188_141483_object12886:head# 2024-04-23T16:43:22.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1576fff:::benchmark_data_smithi188_141483_object37940:head# 2024-04-23T16:43:22.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c15877ee:::benchmark_data_smithi188_141483_object71052:head# 2024-04-23T16:43:22.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1588d17:::benchmark_data_smithi188_141483_object3734:head# 2024-04-23T16:43:22.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c158c8d7:::benchmark_data_smithi188_141483_object22824:head# 2024-04-23T16:43:22.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1594b4d:::benchmark_data_smithi188_141483_object87606:head# 2024-04-23T16:43:22.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c15a0ec6:::benchmark_data_smithi188_141483_object83538:head# 2024-04-23T16:43:22.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c15a6742:::benchmark_data_smithi188_141483_object10936:head# 2024-04-23T16:43:22.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c15b81b1:::benchmark_data_smithi188_141483_object16010:head# 2024-04-23T16:43:22.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c15c62ec:::benchmark_data_smithi188_141483_object69146:head# 2024-04-23T16:43:22.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c15dd701:::benchmark_data_smithi188_141483_object77922:head# 2024-04-23T16:43:22.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c15e6bfd:::benchmark_data_smithi188_141483_object26272:head# 2024-04-23T16:43:22.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c15eb67a:::benchmark_data_smithi188_141483_object85149:head# 2024-04-23T16:43:22.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c15f4966:::benchmark_data_smithi188_141483_object90404:head# 2024-04-23T16:43:22.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c15fbd5c:::benchmark_data_smithi188_141483_object51790:head# 2024-04-23T16:43:22.845 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c15ff79d:::benchmark_data_smithi188_141483_object77558:head# 2024-04-23T16:43:22.845 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c161559e:::benchmark_data_smithi188_141483_object13814:head# 2024-04-23T16:43:22.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1616545:::benchmark_data_smithi188_141483_object79871:head# 2024-04-23T16:43:22.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c161d78e:::benchmark_data_smithi188_141483_object49945:head# 2024-04-23T16:43:22.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c162182a:::benchmark_data_smithi188_141483_object15760:head# 2024-04-23T16:43:22.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c163d22d:::benchmark_data_smithi188_141483_object56742:head# 2024-04-23T16:43:22.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1644d4c:::benchmark_data_smithi188_141483_object44568:head# 2024-04-23T16:43:22.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1647379:::benchmark_data_smithi188_141483_object56334:head# 2024-04-23T16:43:22.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c165f7e7:::benchmark_data_smithi188_141483_object37848:head# 2024-04-23T16:43:22.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1662c9f:::benchmark_data_smithi188_141483_object29931:head# 2024-04-23T16:43:22.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c166b650:::benchmark_data_smithi188_141483_object26532:head# 2024-04-23T16:43:22.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c166e44a:::benchmark_data_smithi188_141483_object68995:head# 2024-04-23T16:43:22.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1679cfb:::benchmark_data_smithi188_141483_object29760:head# 2024-04-23T16:43:22.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c167b61d:::benchmark_data_smithi188_141483_object16934:head# 2024-04-23T16:43:22.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c167de91:::benchmark_data_smithi188_141483_object76671:head# 2024-04-23T16:43:22.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c168d799:::benchmark_data_smithi188_141483_object84516:head# 2024-04-23T16:43:22.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1690ec3:::benchmark_data_smithi188_141483_object4795:head# 2024-04-23T16:43:22.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c16aa1d3:::benchmark_data_smithi188_141483_object8687:head# 2024-04-23T16:43:22.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c16b4db4:::benchmark_data_smithi188_141483_object3248:head# 2024-04-23T16:43:22.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c16cf5a4:::benchmark_data_smithi188_141483_object62958:head# 2024-04-23T16:43:22.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c16d32eb:::benchmark_data_smithi188_141483_object57984:head# 2024-04-23T16:43:22.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c16d7d9e:::benchmark_data_smithi188_141483_object56156:head# 2024-04-23T16:43:22.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c16de6b9:::benchmark_data_smithi188_141483_object57079:head# 2024-04-23T16:43:22.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c16e3c29:::benchmark_data_smithi188_141483_object90191:head# 2024-04-23T16:43:22.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c16f781f:::benchmark_data_smithi188_141483_object23881:head# 2024-04-23T16:43:22.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1708fd5:::benchmark_data_smithi188_141483_object86647:head# 2024-04-23T16:43:22.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c171525a:::benchmark_data_smithi188_141483_object55734:head# 2024-04-23T16:43:22.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c171efda:::benchmark_data_smithi188_141483_object50404:head# 2024-04-23T16:43:22.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1726f5e:::benchmark_data_smithi188_141483_object2188:head# 2024-04-23T16:43:22.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c17272ec:::benchmark_data_smithi188_141483_object85237:head# 2024-04-23T16:43:22.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c172c51a:::benchmark_data_smithi188_141483_object25126:head# 2024-04-23T16:43:22.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1733f09:::benchmark_data_smithi188_141483_object61220:head# 2024-04-23T16:43:22.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c174746d:::benchmark_data_smithi188_141483_object28833:head# 2024-04-23T16:43:22.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c177b02f:::benchmark_data_smithi188_141483_object8079:head# 2024-04-23T16:43:22.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1782fb2:::benchmark_data_smithi188_141483_object77228:head# 2024-04-23T16:43:22.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1798804:::benchmark_data_smithi188_141483_object10772:head# 2024-04-23T16:43:22.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c17a3d06:::benchmark_data_smithi188_141483_object64273:head# 2024-04-23T16:43:22.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c17aa7a6:::benchmark_data_smithi188_141483_object81488:head# 2024-04-23T16:43:22.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c17aa8f6:::benchmark_data_smithi188_141483_object42691:head# 2024-04-23T16:43:22.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c17ae7b5:::benchmark_data_smithi188_141483_object75673:head# 2024-04-23T16:43:22.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c17c90a4:::benchmark_data_smithi188_141483_object39848:head# 2024-04-23T16:43:22.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c17de52c:::benchmark_data_smithi188_141483_object41491:head# 2024-04-23T16:43:22.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c17ee4c3:::benchmark_data_smithi188_141483_object23272:head# 2024-04-23T16:43:22.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c17ef543:::benchmark_data_smithi188_141483_object12407:head# 2024-04-23T16:43:22.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c17f28aa:::benchmark_data_smithi188_141483_object89737:head# 2024-04-23T16:43:22.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c17fa53f:::benchmark_data_smithi188_141483_object26281:head# 2024-04-23T16:43:22.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1806a62:::benchmark_data_smithi188_141483_object49831:head# 2024-04-23T16:43:22.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c180ac96:::benchmark_data_smithi188_141483_object6140:head# 2024-04-23T16:43:22.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1812323:::benchmark_data_smithi188_141483_object43093:head# 2024-04-23T16:43:22.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1815471:::benchmark_data_smithi188_141483_object7340:head# 2024-04-23T16:43:22.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c18174e3:::benchmark_data_smithi188_141483_object90780:head# 2024-04-23T16:43:22.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c18226a3:::benchmark_data_smithi188_141483_object53201:head# 2024-04-23T16:43:22.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c183399f:::benchmark_data_smithi188_141483_object38251:head# 2024-04-23T16:43:22.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c18343d6:::benchmark_data_smithi188_141483_object49186:head# 2024-04-23T16:43:22.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c186161c:::benchmark_data_smithi188_141483_object60208:head# 2024-04-23T16:43:22.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c186bf96:::benchmark_data_smithi188_141483_object6378:head# 2024-04-23T16:43:22.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1872c6f:::benchmark_data_smithi188_141483_object43984:head# 2024-04-23T16:43:22.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1873440:::benchmark_data_smithi188_141483_object6432:head# 2024-04-23T16:43:22.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1874d54:::benchmark_data_smithi188_141483_object3021:head# 2024-04-23T16:43:22.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c18802e8:::benchmark_data_smithi188_141483_object9740:head# 2024-04-23T16:43:22.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c18922bf:::benchmark_data_smithi188_141483_object2931:head# 2024-04-23T16:43:22.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c18940f1:::benchmark_data_smithi188_141483_object43775:head# 2024-04-23T16:43:22.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c189c999:::benchmark_data_smithi188_141483_object67934:head# 2024-04-23T16:43:22.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c18ac14c:::benchmark_data_smithi188_141483_object6660:head# 2024-04-23T16:43:22.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c18b9fdf:::benchmark_data_smithi188_141483_object77858:head# 2024-04-23T16:43:22.875 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c18bbcc8:::benchmark_data_smithi188_141483_object81767:head# 2024-04-23T16:43:22.875 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c18c291e:::benchmark_data_smithi188_141483_object75650:head# 2024-04-23T16:43:22.876 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c18d30bd:::benchmark_data_smithi188_141483_object42891:head# 2024-04-23T16:43:22.876 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c18d3d7f:::benchmark_data_smithi188_141483_object18214:head# 2024-04-23T16:43:22.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c18fd477:::benchmark_data_smithi188_141483_object69495:head# 2024-04-23T16:43:22.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c190be0b:::benchmark_data_smithi188_141483_object1644:head# 2024-04-23T16:43:22.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1910507:::benchmark_data_smithi188_141483_object849:head# 2024-04-23T16:43:22.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1914840:::benchmark_data_smithi188_141483_object72080:head# 2024-04-23T16:43:22.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c191900f:::benchmark_data_smithi188_141483_object25428:head# 2024-04-23T16:43:22.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1929f85:::benchmark_data_smithi188_141483_object22452:head# 2024-04-23T16:43:22.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c192e660:::benchmark_data_smithi188_141483_object52980:head# 2024-04-23T16:43:22.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1942670:::benchmark_data_smithi188_141483_object52795:head# 2024-04-23T16:43:22.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c19512e4:::benchmark_data_smithi188_141483_object78741:head# 2024-04-23T16:43:22.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1972651:::benchmark_data_smithi188_141483_object46661:head# 2024-04-23T16:43:22.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1974cf8:::benchmark_data_smithi188_141483_object42107:head# 2024-04-23T16:43:22.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c197f56e:::benchmark_data_smithi188_141483_object34932:head# 2024-04-23T16:43:22.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c19887a5:::benchmark_data_smithi188_141483_object3645:head# 2024-04-23T16:43:22.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1991793:::benchmark_data_smithi188_141483_object18876:head# 2024-04-23T16:43:22.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c19b83af:::benchmark_data_smithi188_141483_object61803:head# 2024-04-23T16:43:22.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c19bad05:::benchmark_data_smithi188_141483_object86260:head# 2024-04-23T16:43:22.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c19caafe:::benchmark_data_smithi188_141483_object89103:head# 2024-04-23T16:43:22.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c19dcd04:::benchmark_data_smithi188_141483_object46424:head# 2024-04-23T16:43:22.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c19e8b26:::benchmark_data_smithi188_141483_object36010:head# 2024-04-23T16:43:22.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c19ef1ea:::benchmark_data_smithi188_141483_object8139:head# 2024-04-23T16:43:22.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c19f536b:::benchmark_data_smithi188_141483_object21631:head# 2024-04-23T16:43:22.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c19f91e0:::benchmark_data_smithi188_141483_object50240:head# 2024-04-23T16:43:22.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c19fa921:::benchmark_data_smithi188_141483_object38501:head# 2024-04-23T16:43:22.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c19fb6b5:::benchmark_data_smithi188_141483_object76589:head# 2024-04-23T16:43:22.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c19fd4cd:::benchmark_data_smithi188_141483_object11965:head# 2024-04-23T16:43:22.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1a00e68:::benchmark_data_smithi188_141483_object10440:head# 2024-04-23T16:43:22.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1a040aa:::benchmark_data_smithi188_141483_object76283:head# 2024-04-23T16:43:22.893 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1a12da3:::benchmark_data_smithi188_141483_object9430:head# 2024-04-23T16:43:22.893 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1a21882:::benchmark_data_smithi188_141483_object35058:head# 2024-04-23T16:43:22.893 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1a2badc:::benchmark_data_smithi188_141483_object28736:head# 2024-04-23T16:43:22.893 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1a618c3:::benchmark_data_smithi188_141483_object12468:head# 2024-04-23T16:43:22.893 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:43:22.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1a6f792:::benchmark_data_smithi188_141483_object604:head# 2024-04-23T16:43:22.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1a7f9a8:::benchmark_data_smithi188_141483_object82098:head# 2024-04-23T16:43:22.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1a85010:::benchmark_data_smithi188_141483_object89398:head# 2024-04-23T16:43:22.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1a990eb:::benchmark_data_smithi188_141483_object65744:head# 2024-04-23T16:43:22.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1a9fb06:::benchmark_data_smithi188_141483_object20311:head# 2024-04-23T16:43:22.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1aa6326:::benchmark_data_smithi188_141483_object60419:head# 2024-04-23T16:43:22.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ab1845:::benchmark_data_smithi188_141483_object61530:head# 2024-04-23T16:43:22.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ab899f:::benchmark_data_smithi188_141483_object32010:head# 2024-04-23T16:43:22.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1aba843:::benchmark_data_smithi188_141483_object49377:head# 2024-04-23T16:43:22.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ad557e:::benchmark_data_smithi188_141483_object31244:head# 2024-04-23T16:43:22.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1adc927:::benchmark_data_smithi188_141483_object11685:head# 2024-04-23T16:43:22.899 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ae02a2:::benchmark_data_smithi188_141483_object72746:head# 2024-04-23T16:43:22.899 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ae1867:::benchmark_data_smithi188_141483_object50772:head# 2024-04-23T16:43:22.899 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ae5295:::benchmark_data_smithi188_141483_object27482:head# 2024-04-23T16:43:22.900 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1af1937:::benchmark_data_smithi188_141483_object75947:head# 2024-04-23T16:43:22.900 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1af5999:::benchmark_data_smithi188_141483_object67156:head# 2024-04-23T16:43:22.900 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1af7171:::benchmark_data_smithi188_141483_object84041:head# 2024-04-23T16:43:22.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1afd138:::benchmark_data_smithi188_141483_object23232:head# 2024-04-23T16:43:22.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1b25b07:::benchmark_data_smithi188_141483_object81133:head# 2024-04-23T16:43:22.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1b2f7b5:::benchmark_data_smithi188_141483_object85362:head# 2024-04-23T16:43:22.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1b2fb4c:::benchmark_data_smithi188_141483_object84200:head# 2024-04-23T16:43:22.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1b32aeb:::benchmark_data_smithi188_141483_object87072:head# 2024-04-23T16:43:22.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1b3a42d:::benchmark_data_smithi188_141483_object27910:head# 2024-04-23T16:43:22.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1b56580:::benchmark_data_smithi188_141483_object8144:head# 2024-04-23T16:43:22.904 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1b5a822:::benchmark_data_smithi188_141483_object9196:head# 2024-04-23T16:43:22.904 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1b66896:::benchmark_data_smithi188_141483_object72045:head# 2024-04-23T16:43:22.904 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1b87420:::benchmark_data_smithi188_141483_object39606:head# 2024-04-23T16:43:22.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1b88948:::benchmark_data_smithi188_141483_object79184:head# 2024-04-23T16:43:22.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1b929c3:::benchmark_data_smithi188_141483_object5438:head# 2024-04-23T16:43:22.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1b9339c:::benchmark_data_smithi188_141483_object80142:head# 2024-04-23T16:43:22.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1b98bdf:::benchmark_data_smithi188_141483_object51203:head# 2024-04-23T16:43:22.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ba2d9f:::benchmark_data_smithi188_141483_object41439:head# 2024-04-23T16:43:22.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1bb00ce:::benchmark_data_smithi188_141483_object44919:head# 2024-04-23T16:43:22.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1bb880c:::benchmark_data_smithi188_141483_object47426:head# 2024-04-23T16:43:22.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1bc6440:::benchmark_data_smithi188_141483_object79290:head# 2024-04-23T16:43:22.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1bcf0ff:::benchmark_data_smithi188_141483_object4847:head# 2024-04-23T16:43:22.909 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1bda696:::benchmark_data_smithi188_141483_object62535:head# 2024-04-23T16:43:22.909 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1be5363:::benchmark_data_smithi188_141483_object2821:head# 2024-04-23T16:43:22.909 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1be93d0:::benchmark_data_smithi188_141483_object5865:head# 2024-04-23T16:43:22.910 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1be9cd1:::benchmark_data_smithi188_141483_object73554:head# 2024-04-23T16:43:22.910 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1bf4b84:::benchmark_data_smithi188_141483_object73523:head# 2024-04-23T16:43:22.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1c06ddb:::benchmark_data_smithi188_141483_object31946:head# 2024-04-23T16:43:22.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1c11774:::benchmark_data_smithi188_141483_object11161:head# 2024-04-23T16:43:22.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1c118b5:::benchmark_data_smithi188_141483_object66714:head# 2024-04-23T16:43:22.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1c2c830:::benchmark_data_smithi188_141483_object41637:head# 2024-04-23T16:43:22.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1c4878c:::benchmark_data_smithi188_141483_object7192:head# 2024-04-23T16:43:22.914 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1c5b25c:::benchmark_data_smithi188_141483_object24565:head# 2024-04-23T16:43:22.914 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1c6bc07:::benchmark_data_smithi188_141483_object84616:head# 2024-04-23T16:43:22.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1c79fab:::benchmark_data_smithi188_141483_object19412:head# 2024-04-23T16:43:22.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1c7dbe0:::benchmark_data_smithi188_141483_object2466:head# 2024-04-23T16:43:22.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1c82ae3:::benchmark_data_smithi188_141483_object84292:head# 2024-04-23T16:43:22.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1c88f61:::benchmark_data_smithi188_141483_object22966:head# 2024-04-23T16:43:22.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1c911ab:::benchmark_data_smithi188_141483_object78825:head# 2024-04-23T16:43:22.917 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1c9133e:::benchmark_data_smithi188_141483_object28729:head# 2024-04-23T16:43:22.917 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1c9730f:::benchmark_data_smithi188_141483_object32321:head# 2024-04-23T16:43:22.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1c9df6a:::benchmark_data_smithi188_141483_object67340:head# 2024-04-23T16:43:22.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ca7364:::benchmark_data_smithi188_141483_object20421:head# 2024-04-23T16:43:22.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ca9a50:::benchmark_data_smithi188_141483_object90170:head# 2024-04-23T16:43:22.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1cac71f:::benchmark_data_smithi188_141483_object21822:head# 2024-04-23T16:43:22.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1cb1372:::benchmark_data_smithi188_141483_object9291:head# 2024-04-23T16:43:22.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1cbcd40:::benchmark_data_smithi188_141483_object54340:head# 2024-04-23T16:43:22.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1cc60f9:::benchmark_data_smithi188_141483_object8064:head# 2024-04-23T16:43:22.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1cc6fd0:::benchmark_data_smithi188_141483_object22461:head# 2024-04-23T16:43:22.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1cc7c55:::benchmark_data_smithi188_141483_object20661:head# 2024-04-23T16:43:22.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1cce7fc:::benchmark_data_smithi188_141483_object10198:head# 2024-04-23T16:43:22.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1cd26c5:::benchmark_data_smithi188_141483_object75117:head# 2024-04-23T16:43:22.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1cd7e97:::benchmark_data_smithi188_141483_object62852:head# 2024-04-23T16:43:22.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1cdaee1:::benchmark_data_smithi188_141483_object66990:head# 2024-04-23T16:43:22.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ce1bbb:::benchmark_data_smithi188_141483_object16802:head# 2024-04-23T16:43:22.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ce98d4:::benchmark_data_smithi188_141483_object35229:head# 2024-04-23T16:43:22.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1cf9306:::benchmark_data_smithi188_141483_object89652:head# 2024-04-23T16:43:22.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1d08307:::benchmark_data_smithi188_141483_object75909:head# 2024-04-23T16:43:22.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1d1a7d9:::benchmark_data_smithi188_141483_object87028:head# 2024-04-23T16:43:22.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1d1ab1f:::benchmark_data_smithi188_141483_object62213:head# 2024-04-23T16:43:22.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1d28930:::benchmark_data_smithi188_141483_object81050:head# 2024-04-23T16:43:22.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1d43be9:::benchmark_data_smithi188_141483_object43165:head# 2024-04-23T16:43:22.930 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1d513cd:::benchmark_data_smithi188_141483_object60170:head# 2024-04-23T16:43:22.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1d5e42c:::benchmark_data_smithi188_141483_object13608:head# 2024-04-23T16:43:22.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1d6cec8:::benchmark_data_smithi188_141483_object64387:head# 2024-04-23T16:43:22.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1d6e2ab:::benchmark_data_smithi188_141483_object14495:head# 2024-04-23T16:43:22.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1d814c0:::benchmark_data_smithi188_141483_object72241:head# 2024-04-23T16:43:22.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1d8259e:::benchmark_data_smithi188_141483_object44044:head# 2024-04-23T16:43:22.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1d8ea10:::benchmark_data_smithi188_141483_object50131:head# 2024-04-23T16:43:22.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1da01f7:::benchmark_data_smithi188_141483_object78661:head# 2024-04-23T16:43:22.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1db2d30:::benchmark_data_smithi188_141483_object77124:head# 2024-04-23T16:43:22.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1dbbd28:::benchmark_data_smithi188_141483_object78550:head# 2024-04-23T16:43:22.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1dbd66c:::benchmark_data_smithi188_141483_object90349:head# 2024-04-23T16:43:22.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1dc1bdb:::benchmark_data_smithi188_141483_object49387:head# 2024-04-23T16:43:22.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1de5255:::benchmark_data_smithi188_141483_object74037:head# 2024-04-23T16:43:22.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1de65f3:::benchmark_data_smithi188_141483_object84481:head# 2024-04-23T16:43:22.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1dfb196:::benchmark_data_smithi188_141483_object31272:head# 2024-04-23T16:43:22.939 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1dfd3b0:::benchmark_data_smithi188_141483_object83774:head# 2024-04-23T16:43:22.940 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1e044fd:::benchmark_data_smithi188_141483_object79401:head# 2024-04-23T16:43:22.940 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1e2bacd:::benchmark_data_smithi188_141483_object52554:head# 2024-04-23T16:43:22.940 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1e3e12e:::benchmark_data_smithi188_141483_object43919:head# 2024-04-23T16:43:22.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1e49781:::benchmark_data_smithi188_141483_object38730:head# 2024-04-23T16:43:22.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1e5b10f:::benchmark_data_smithi188_141483_object1489:head# 2024-04-23T16:43:22.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1e6c964:::benchmark_data_smithi188_141483_object24479:head# 2024-04-23T16:43:22.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1e6cfc5:::benchmark_data_smithi188_141483_object5496:head# 2024-04-23T16:43:22.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1e86086:::benchmark_data_smithi188_141483_object55675:head# 2024-04-23T16:43:22.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1e8be6b:::benchmark_data_smithi188_141483_object88726:head# 2024-04-23T16:43:22.945 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1e8fee3:::benchmark_data_smithi188_141483_object41653:head# 2024-04-23T16:43:22.946 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1e9b7d6:::benchmark_data_smithi188_141483_object23114:head# 2024-04-23T16:43:22.946 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1e9f5d3:::benchmark_data_smithi188_141483_object23643:head# 2024-04-23T16:43:22.946 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1e9ff23:::benchmark_data_smithi188_141483_object54247:head# 2024-04-23T16:43:22.947 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1eb0a7e:::benchmark_data_smithi188_141483_object67166:head# 2024-04-23T16:43:22.947 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1eb44bd:::benchmark_data_smithi188_141483_object79277:head# 2024-04-23T16:43:22.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1eb5470:::benchmark_data_smithi188_141483_object34239:head# 2024-04-23T16:43:22.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ebfcda:::benchmark_data_smithi188_141483_object78236:head# 2024-04-23T16:43:22.949 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ec3134:::benchmark_data_smithi188_141483_object8695:head# 2024-04-23T16:43:22.949 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ec313f:::benchmark_data_smithi188_141483_object80995:head# 2024-04-23T16:43:22.950 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ec4149:::benchmark_data_smithi188_141483_object58193:head# 2024-04-23T16:43:22.950 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1eedb97:::benchmark_data_smithi188_141483_object83689:head# 2024-04-23T16:43:22.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1eedde8:::benchmark_data_smithi188_141483_object13284:head# 2024-04-23T16:43:22.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1eee471:::benchmark_data_smithi188_141483_object59877:head# 2024-04-23T16:43:22.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ef1355:::benchmark_data_smithi188_141483_object60427:head# 2024-04-23T16:43:22.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ef466b:::benchmark_data_smithi188_141483_object21724:head# 2024-04-23T16:43:22.953 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1f07544:::benchmark_data_smithi188_141483_object29268:head# 2024-04-23T16:43:22.953 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1f0984e:::benchmark_data_smithi188_141483_object86558:head# 2024-04-23T16:43:22.953 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1f20ff1:::benchmark_data_smithi188_141483_object83271:head# 2024-04-23T16:43:22.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1f22c57:::benchmark_data_smithi188_141483_object7056:head# 2024-04-23T16:43:22.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1f33b45:::benchmark_data_smithi188_141483_object33805:head# 2024-04-23T16:43:22.955 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1f509af:::benchmark_data_smithi188_141483_object88580:head# 2024-04-23T16:43:22.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1f6894b:::benchmark_data_smithi188_141483_object32306:head# 2024-04-23T16:43:22.960 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1f76273:::benchmark_data_smithi188_141483_object25182:head# 2024-04-23T16:43:22.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1f78063:::benchmark_data_smithi188_141483_object19147:head# 2024-04-23T16:43:22.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1f7a72a:::benchmark_data_smithi188_141483_object13193:head# 2024-04-23T16:43:22.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1f87e40:::benchmark_data_smithi188_141483_object3922:head# 2024-04-23T16:43:22.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1f8ccc2:::benchmark_data_smithi188_141483_object86331:head# 2024-04-23T16:43:22.965 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1f90442:::benchmark_data_smithi188_141483_object66515:head# 2024-04-23T16:43:22.965 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1f93b1a:::benchmark_data_smithi188_141483_object11138:head# 2024-04-23T16:43:22.965 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1f9624f:::benchmark_data_smithi188_141483_object23848:head# 2024-04-23T16:43:22.966 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1f9c32e:::benchmark_data_smithi188_141483_object90424:head# 2024-04-23T16:43:22.966 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1fbe5b2:::benchmark_data_smithi188_141483_object84003:head# 2024-04-23T16:43:22.967 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1fc06ed:::benchmark_data_smithi188_141483_object1415:head# 2024-04-23T16:43:22.967 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1fc74fa:::benchmark_data_smithi188_141483_object51904:head# 2024-04-23T16:43:22.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1fccb74:::benchmark_data_smithi188_141483_object84855:head# 2024-04-23T16:43:22.969 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1fd09cd:::benchmark_data_smithi188_141483_object27850:head# 2024-04-23T16:43:22.969 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1fd63fe:::benchmark_data_smithi188_141483_object34819:head# 2024-04-23T16:43:22.969 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1fd6d34:::benchmark_data_smithi188_141483_object80039:head# 2024-04-23T16:43:22.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1fd8ecb:::benchmark_data_smithi188_141483_object47422:head# 2024-04-23T16:43:22.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1fe08a3:::benchmark_data_smithi188_141483_object45100:head# 2024-04-23T16:43:22.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ff254f:::benchmark_data_smithi188_141483_object8069:head# 2024-04-23T16:43:22.973 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ff2fed:::benchmark_data_smithi188_141483_object90446:head# 2024-04-23T16:43:22.974 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c1ff74f5:::benchmark_data_smithi188_141483_object2567:head# 2024-04-23T16:43:22.974 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2007d59:::benchmark_data_smithi188_141483_object36875:head# 2024-04-23T16:43:22.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2025d59:::benchmark_data_smithi188_141483_object90282:head# 2024-04-23T16:43:22.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c202f063:::benchmark_data_smithi188_141483_object20914:head# 2024-04-23T16:43:22.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c20312ec:::benchmark_data_smithi188_141483_object78566:head# 2024-04-23T16:43:22.977 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2032baf:::benchmark_data_smithi188_141483_object2836:head# 2024-04-23T16:43:22.977 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2040587:::benchmark_data_smithi188_141483_object12310:head# 2024-04-23T16:43:22.977 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c20594be:::benchmark_data_smithi188_141483_object89023:head# 2024-04-23T16:43:22.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c205b2d5:::benchmark_data_smithi188_141483_object82868:head# 2024-04-23T16:43:22.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c20624d9:::benchmark_data_smithi188_141483_object6743:head# 2024-04-23T16:43:22.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2065729:::benchmark_data_smithi188_141483_object23596:head# 2024-04-23T16:43:22.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c206d9e9:::benchmark_data_smithi188_141483_object20156:head# 2024-04-23T16:43:22.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c208863e:::benchmark_data_smithi188_141483_object83392:head# 2024-04-23T16:43:22.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c208aac4:::benchmark_data_smithi188_141483_object17674:head# 2024-04-23T16:43:22.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2094419:::benchmark_data_smithi188_141483_object323:head# 2024-04-23T16:43:22.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c209b317:::benchmark_data_smithi188_141483_object55088:head# 2024-04-23T16:43:22.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c209bfa1:::benchmark_data_smithi188_141483_object25613:head# 2024-04-23T16:43:22.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c20a3e73:::benchmark_data_smithi188_141483_object34498:head# 2024-04-23T16:43:22.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c20b32f1:::benchmark_data_smithi188_141483_object90554:head# 2024-04-23T16:43:22.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c20b58e2:::benchmark_data_smithi188_141483_object16138:head# 2024-04-23T16:43:22.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c20bb296:::benchmark_data_smithi188_141483_object75396:head# 2024-04-23T16:43:22.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c20e5d4d:::benchmark_data_smithi188_141483_object87896:head# 2024-04-23T16:43:22.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c20fbe73:::benchmark_data_smithi188_141483_object75023:head# 2024-04-23T16:43:22.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2103074:::benchmark_data_smithi188_141483_object67065:head# 2024-04-23T16:43:22.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c214a30d:::benchmark_data_smithi188_141483_object65904:head# 2024-04-23T16:43:22.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2154164:::benchmark_data_smithi188_141483_object26506:head# 2024-04-23T16:43:22.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2163f3b:::benchmark_data_smithi188_141483_object73083:head# 2024-04-23T16:43:22.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2165540:::benchmark_data_smithi188_141483_object40036:head# 2024-04-23T16:43:22.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c216651b:::benchmark_data_smithi188_141483_object37996:head# 2024-04-23T16:43:22.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2184987:::benchmark_data_smithi188_141483_object4532:head# 2024-04-23T16:43:22.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c218a3af:::benchmark_data_smithi188_141483_object69302:head# 2024-04-23T16:43:22.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c218b3de:::benchmark_data_smithi188_141483_object50437:head# 2024-04-23T16:43:22.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c219c377:::benchmark_data_smithi188_141483_object12744:head# 2024-04-23T16:43:22.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c219d2a0:::benchmark_data_smithi188_141483_object33078:head# 2024-04-23T16:43:22.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c21a625d:::benchmark_data_smithi188_141483_object41057:head# 2024-04-23T16:43:22.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c21b24b4:::benchmark_data_smithi188_141483_object12458:head# 2024-04-23T16:43:22.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c21c42be:::benchmark_data_smithi188_141483_object38363:head# 2024-04-23T16:43:22.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c21deb2d:::benchmark_data_smithi188_141483_object49700:head# 2024-04-23T16:43:22.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c21e0921:::benchmark_data_smithi188_141483_object24552:head# 2024-04-23T16:43:22.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c21e5f1b:::benchmark_data_smithi188_141483_object36520:head# 2024-04-23T16:43:22.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c21ff243:::benchmark_data_smithi188_141483_object54232:head# 2024-04-23T16:43:22.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c220df18:::benchmark_data_smithi188_141483_object18598:head# 2024-04-23T16:43:22.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c22188a3:::benchmark_data_smithi188_141483_object40612:head# 2024-04-23T16:43:22.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c221b3e3:::benchmark_data_smithi188_141483_object6471:head# 2024-04-23T16:43:22.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2240fbb:::benchmark_data_smithi188_141483_object13445:head# 2024-04-23T16:43:22.997 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2247288:::benchmark_data_smithi188_141483_object46741:head# 2024-04-23T16:43:22.997 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2261b1e:::benchmark_data_smithi188_141483_object46260:head# 2024-04-23T16:43:22.997 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2261b87:::benchmark_data_smithi188_141483_object32542:head# 2024-04-23T16:43:22.998 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c226810c:::benchmark_data_smithi188_141483_object77491:head# 2024-04-23T16:43:22.998 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c226dbec:::benchmark_data_smithi188_141483_object75567:head# 2024-04-23T16:43:22.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c226e422:::benchmark_data_smithi188_141483_object78767:head# 2024-04-23T16:43:23.001 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c228f8c4:::benchmark_data_smithi188_141483_object53030:head# 2024-04-23T16:43:23.001 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c22904d9:::benchmark_data_smithi188_141483_object55992:head# 2024-04-23T16:43:23.001 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c22a5f6b:::benchmark_data_smithi188_141483_object27103:head# 2024-04-23T16:43:23.002 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c22abf90:::benchmark_data_smithi188_141483_object41476:head# 2024-04-23T16:43:23.004 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c22aff87:::benchmark_data_smithi188_141483_object14905:head# 2024-04-23T16:43:23.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c22ba3e6:::benchmark_data_smithi188_141483_object52922:head# 2024-04-23T16:43:23.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c22c2414:::benchmark_data_smithi188_141483_object12970:head# 2024-04-23T16:43:23.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c22c6f2d:::benchmark_data_smithi188_141483_object80846:head# 2024-04-23T16:43:23.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c22cb842:::benchmark_data_smithi188_141483_object51128:head# 2024-04-23T16:43:23.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c22e3cc9:::benchmark_data_smithi188_141483_object3880:head# 2024-04-23T16:43:23.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c22f3d1e:::benchmark_data_smithi188_141483_object72992:head# 2024-04-23T16:43:23.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c22f756f:::benchmark_data_smithi188_141483_object34364:head# 2024-04-23T16:43:23.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c22fbc1d:::benchmark_data_smithi188_141483_object46531:head# 2024-04-23T16:43:23.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c22fbd43:::benchmark_data_smithi188_141483_object43474:head# 2024-04-23T16:43:23.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c22fc710:::benchmark_data_smithi188_141483_object56995:head# 2024-04-23T16:43:23.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2303909:::benchmark_data_smithi188_141483_object88625:head# 2024-04-23T16:43:23.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2304991:::benchmark_data_smithi188_141483_object55955:head# 2024-04-23T16:43:23.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c230908d:::benchmark_data_smithi188_141483_object58871:head# 2024-04-23T16:43:23.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2309b63:::benchmark_data_smithi188_141483_object25101:head# 2024-04-23T16:43:23.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2314981:::benchmark_data_smithi188_141483_object37387:head# 2024-04-23T16:43:23.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2323377:::benchmark_data_smithi188_141483_object39893:head# 2024-04-23T16:43:23.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2325ca3:::benchmark_data_smithi188_141483_object8482:head# 2024-04-23T16:43:23.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c233b4d2:::benchmark_data_smithi188_141483_object21686:head# 2024-04-23T16:43:23.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c233cb59:::benchmark_data_smithi188_141483_object70311:head# 2024-04-23T16:43:23.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2343531:::benchmark_data_smithi188_141483_object782:head# 2024-04-23T16:43:23.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c235340f:::benchmark_data_smithi188_141483_object35754:head# 2024-04-23T16:43:23.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2355f1b:::benchmark_data_smithi188_141483_object84903:head# 2024-04-23T16:43:23.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2365067:::benchmark_data_smithi188_141483_object33837:head# 2024-04-23T16:43:23.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2378d43:::benchmark_data_smithi188_141483_object86058:head# 2024-04-23T16:43:23.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c23809bd:::benchmark_data_smithi188_141483_object83365:head# 2024-04-23T16:43:23.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2389289:::benchmark_data_smithi188_141483_object23802:head# 2024-04-23T16:43:23.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c238c540:::benchmark_data_smithi188_141483_object70786:head# 2024-04-23T16:43:23.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c23949d1:::benchmark_data_smithi188_141483_object87847:head# 2024-04-23T16:43:23.019 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c23c03a9:::benchmark_data_smithi188_141483_object83749:head# 2024-04-23T16:43:23.019 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c23c4277:::benchmark_data_smithi188_141483_object67995:head# 2024-04-23T16:43:23.019 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c23d5857:::benchmark_data_smithi188_141483_object29456:head# 2024-04-23T16:43:23.020 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c23e3484:::benchmark_data_smithi188_141483_object72346:head# 2024-04-23T16:43:23.020 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c23efee4:::benchmark_data_smithi188_141483_object58303:head# 2024-04-23T16:43:23.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c23f9cd0:::benchmark_data_smithi188_141483_object81640:head# 2024-04-23T16:43:23.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c23fb30c:::benchmark_data_smithi188_141483_object79969:head# 2024-04-23T16:43:23.022 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2410071:::benchmark_data_smithi188_141483_object90644:head# 2024-04-23T16:43:23.023 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2410e37:::benchmark_data_smithi188_141483_object66835:head# 2024-04-23T16:43:23.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2420857:::benchmark_data_smithi188_141483_object21460:head# 2024-04-23T16:43:23.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2429788:::benchmark_data_smithi188_141483_object60266:head# 2024-04-23T16:43:23.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c242ee05:::benchmark_data_smithi188_141483_object37950:head# 2024-04-23T16:43:23.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2435724:::benchmark_data_smithi188_141483_object73281:head# 2024-04-23T16:43:23.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c24357bd:::benchmark_data_smithi188_141483_object52578:head# 2024-04-23T16:43:23.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2440089:::benchmark_data_smithi188_141483_object30871:head# 2024-04-23T16:43:23.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2440d43:::benchmark_data_smithi188_141483_object45854:head# 2024-04-23T16:43:23.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2440e8f:::benchmark_data_smithi188_141483_object80956:head# 2024-04-23T16:43:23.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2441044:::benchmark_data_smithi188_141483_object14107:head# 2024-04-23T16:43:23.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2452cbf:::benchmark_data_smithi188_141483_object30113:head# 2024-04-23T16:43:23.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2459f66:::benchmark_data_smithi188_141483_object41924:head# 2024-04-23T16:43:23.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c245a3c8:::benchmark_data_smithi188_141483_object79380:head# 2024-04-23T16:43:23.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2463141:::benchmark_data_smithi188_141483_object66697:head# 2024-04-23T16:43:23.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c24670c5:::benchmark_data_smithi188_141483_object68577:head# 2024-04-23T16:43:23.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c246d00c:::benchmark_data_smithi188_141483_object1995:head# 2024-04-23T16:43:23.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c24839ab:::benchmark_data_smithi188_141483_object20639:head# 2024-04-23T16:43:23.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c24845b1:::benchmark_data_smithi188_141483_object34986:head# 2024-04-23T16:43:23.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c248b803:::benchmark_data_smithi188_141483_object16017:head# 2024-04-23T16:43:23.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c248fab8:::benchmark_data_smithi188_141483_object11375:head# 2024-04-23T16:43:23.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2498d9b:::benchmark_data_smithi188_141483_object3157:head# 2024-04-23T16:43:23.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c24a3180:::benchmark_data_smithi188_141483_object4580:head# 2024-04-23T16:43:23.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c24a5bc8:::benchmark_data_smithi188_141483_object86641:head# 2024-04-23T16:43:23.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c24b89bb:::benchmark_data_smithi188_141483_object48260:head# 2024-04-23T16:43:23.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c24c27c8:::benchmark_data_smithi188_141483_object78279:head# 2024-04-23T16:43:23.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c24c699b:::benchmark_data_smithi188_141483_object56659:head# 2024-04-23T16:43:23.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c24cee3f:::benchmark_data_smithi188_141483_object14782:head# 2024-04-23T16:43:23.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c24d041b:::benchmark_data_smithi188_141483_object39604:head# 2024-04-23T16:43:23.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c24d967a:::benchmark_data_smithi188_141483_object41625:head# 2024-04-23T16:43:23.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c24e2dc7:::benchmark_data_smithi188_141483_object31665:head# 2024-04-23T16:43:23.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c24e341a:::benchmark_data_smithi188_141483_object77759:head# 2024-04-23T16:43:23.040 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c24eb17e:::benchmark_data_smithi188_141483_object16645:head# 2024-04-23T16:43:23.040 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c250ca60:::benchmark_data_smithi188_141483_object7392:head# 2024-04-23T16:43:23.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c252cfb1:::benchmark_data_smithi188_141483_object43090:head# 2024-04-23T16:43:23.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2530c51:::benchmark_data_smithi188_141483_object87608:head# 2024-04-23T16:43:23.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2536e60:::benchmark_data_smithi188_141483_object11842:head# 2024-04-23T16:43:23.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c254b18c:::benchmark_data_smithi188_141483_object31926:head# 2024-04-23T16:43:23.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2553e35:::benchmark_data_smithi188_141483_object33870:head# 2024-04-23T16:43:23.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c25543db:::benchmark_data_smithi188_141483_object3533:head# 2024-04-23T16:43:23.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2564ba8:::benchmark_data_smithi188_141483_object84512:head# 2024-04-23T16:43:23.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c256cd74:::benchmark_data_smithi188_141483_object56133:head# 2024-04-23T16:43:23.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2597c67:::benchmark_data_smithi188_141483_object37815:head# 2024-04-23T16:43:23.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c259aac9:::benchmark_data_smithi188_141483_object8119:head# 2024-04-23T16:43:23.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c259b334:::benchmark_data_smithi188_141483_object45582:head# 2024-04-23T16:43:23.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c25cd88c:::benchmark_data_smithi188_141483_object75662:head# 2024-04-23T16:43:23.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c25e388f:::benchmark_data_smithi188_141483_object38841:head# 2024-04-23T16:43:23.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c25eaa10:::benchmark_data_smithi188_141483_object64006:head# 2024-04-23T16:43:23.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c25f2313:::benchmark_data_smithi188_141483_object39283:head# 2024-04-23T16:43:23.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c25f8488:::benchmark_data_smithi188_141483_object4770:head# 2024-04-23T16:43:23.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2610d01:::benchmark_data_smithi188_141483_object39017:head# 2024-04-23T16:43:23.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2617e1f:::benchmark_data_smithi188_141483_object12005:head# 2024-04-23T16:43:23.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2618689:::benchmark_data_smithi188_141483_object25894:head# 2024-04-23T16:43:23.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c261e619:::benchmark_data_smithi188_141483_object54409:head# 2024-04-23T16:43:23.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2621db0:::benchmark_data_smithi188_141483_object48449:head# 2024-04-23T16:43:23.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c262ce01:::benchmark_data_smithi188_141483_object19089:head# 2024-04-23T16:43:23.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c262f67e:::benchmark_data_smithi188_141483_object63774:head# 2024-04-23T16:43:23.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2632925:::benchmark_data_smithi188_141483_object49429:head# 2024-04-23T16:43:23.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c26555f3:::benchmark_data_smithi188_141483_object13548:head# 2024-04-23T16:43:23.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2658ce3:::benchmark_data_smithi188_141483_object53079:head# 2024-04-23T16:43:23.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c265b9ad:::benchmark_data_smithi188_141483_object25648:head# 2024-04-23T16:43:23.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2664009:::benchmark_data_smithi188_141483_object73076:head# 2024-04-23T16:43:23.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c267d6a3:::benchmark_data_smithi188_141483_object18735:head# 2024-04-23T16:43:23.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2693df3:::benchmark_data_smithi188_141483_object20808:head# 2024-04-23T16:43:23.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c26a4941:::benchmark_data_smithi188_141483_object88568:head# 2024-04-23T16:43:23.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c26a570e:::benchmark_data_smithi188_141483_object85663:head# 2024-04-23T16:43:23.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c26d009b:::benchmark_data_smithi188_141483_object49594:head# 2024-04-23T16:43:23.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c26d0cf0:::benchmark_data_smithi188_141483_object66969:head# 2024-04-23T16:43:23.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c26d171c:::benchmark_data_smithi188_141483_object84926:head# 2024-04-23T16:43:23.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c26d5e9b:::benchmark_data_smithi188_141483_object9550:head# 2024-04-23T16:43:23.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c26da2a5:::benchmark_data_smithi188_141483_object73540:head# 2024-04-23T16:43:23.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c26f43b3:::benchmark_data_smithi188_141483_object4693:head# 2024-04-23T16:43:23.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c26f58ac:::benchmark_data_smithi188_141483_object90735:head# 2024-04-23T16:43:23.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c26fff12:::benchmark_data_smithi188_141483_object76853:head# 2024-04-23T16:43:23.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c270e816:::benchmark_data_smithi188_141483_object995:head# 2024-04-23T16:43:23.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c271cbcf:::benchmark_data_smithi188_141483_object38139:head# 2024-04-23T16:43:23.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c272f75c:::benchmark_data_smithi188_141483_object54407:head# 2024-04-23T16:43:23.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2731899:::benchmark_data_smithi188_141483_object58805:head# 2024-04-23T16:43:23.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c275298b:::benchmark_data_smithi188_141483_object37081:head# 2024-04-23T16:43:23.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c275b85a:::benchmark_data_smithi188_141483_object4891:head# 2024-04-23T16:43:23.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2769340:::benchmark_data_smithi188_141483_object1106:head# 2024-04-23T16:43:23.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c276d3e1:::benchmark_data_smithi188_141483_object21250:head# 2024-04-23T16:43:23.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2770cde:::benchmark_data_smithi188_141483_object10990:head# 2024-04-23T16:43:23.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c27711e2:::benchmark_data_smithi188_141483_object18915:head# 2024-04-23T16:43:23.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c278a796:::benchmark_data_smithi188_141483_object6597:head# 2024-04-23T16:43:23.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c279cfe2:::benchmark_data_smithi188_141483_object50591:head# 2024-04-23T16:43:23.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c27ad5a2:::benchmark_data_smithi188_141483_object17201:head# 2024-04-23T16:43:23.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c27b3264:::benchmark_data_smithi188_141483_object29158:head# 2024-04-23T16:43:23.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c27b8873:::benchmark_data_smithi188_141483_object50398:head# 2024-04-23T16:43:23.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c27c1ebc:::benchmark_data_smithi188_141483_object50328:head# 2024-04-23T16:43:23.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c27cb739:::benchmark_data_smithi188_141483_object29620:head# 2024-04-23T16:43:23.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c27cffb2:::benchmark_data_smithi188_141483_object50171:head# 2024-04-23T16:43:23.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c27da79a:::benchmark_data_smithi188_141483_object28866:head# 2024-04-23T16:43:23.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c27e6621:::benchmark_data_smithi188_141483_object67993:head# 2024-04-23T16:43:23.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c27eb0fa:::benchmark_data_smithi188_141483_object16557:head# 2024-04-23T16:43:23.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c27f0689:::benchmark_data_smithi188_141483_object26870:head# 2024-04-23T16:43:23.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c27f07ea:::benchmark_data_smithi188_141483_object4321:head# 2024-04-23T16:43:23.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c280466d:::benchmark_data_smithi188_141483_object7915:head# 2024-04-23T16:43:23.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c280ddd5:::benchmark_data_smithi188_141483_object46892:head# 2024-04-23T16:43:23.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2810035:::benchmark_data_smithi188_141483_object48968:head# 2024-04-23T16:43:23.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c283c656:::benchmark_data_smithi188_141483_object22319:head# 2024-04-23T16:43:23.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c285345e:::benchmark_data_smithi188_141483_object35521:head# 2024-04-23T16:43:23.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c285846f:::benchmark_data_smithi188_141483_object6889:head# 2024-04-23T16:43:23.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c285b1e4:::benchmark_data_smithi188_141483_object39484:head# 2024-04-23T16:43:23.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2885b7a:::benchmark_data_smithi188_141483_object49986:head# 2024-04-23T16:43:23.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c288dc64:::benchmark_data_smithi188_141483_object49060:head# 2024-04-23T16:43:23.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c289ca12:::benchmark_data_smithi188_141483_object50552:head# 2024-04-23T16:43:23.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c289f4e5:::benchmark_data_smithi188_141483_object9602:head# 2024-04-23T16:43:23.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c28a4de5:::benchmark_data_smithi188_141483_object82184:head# 2024-04-23T16:43:23.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c28aad13:::benchmark_data_smithi188_141483_object38648:head# 2024-04-23T16:43:23.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c28b96de:::benchmark_data_smithi188_141483_object67243:head# 2024-04-23T16:43:23.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c28cb9e6:::benchmark_data_smithi188_141483_object6797:head# 2024-04-23T16:43:23.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c28cd325:::benchmark_data_smithi188_141483_object4521:head# 2024-04-23T16:43:23.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c28d0aa5:::benchmark_data_smithi188_141483_object27409:head# 2024-04-23T16:43:23.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c28dd6cf:::benchmark_data_smithi188_141483_object28104:head# 2024-04-23T16:43:23.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c28e8e0c:::benchmark_data_smithi188_141483_object46557:head# 2024-04-23T16:43:23.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c28ef9d0:::benchmark_data_smithi188_141483_object78901:head# 2024-04-23T16:43:23.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c28f3816:::benchmark_data_smithi188_141483_object69090:head# 2024-04-23T16:43:23.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c290eae0:::benchmark_data_smithi188_141483_object43161:head# 2024-04-23T16:43:23.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c290ff4c:::benchmark_data_smithi188_141483_object29791:head# 2024-04-23T16:43:23.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2914353:::benchmark_data_smithi188_141483_object15070:head# 2024-04-23T16:43:23.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2928ed8:::benchmark_data_smithi188_141483_object6907:head# 2024-04-23T16:43:23.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c29331e6:::benchmark_data_smithi188_141483_object8232:head# 2024-04-23T16:43:23.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c293a0a9:::benchmark_data_smithi188_141483_object1360:head# 2024-04-23T16:43:23.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c29406ee:::benchmark_data_smithi188_141483_object68144:head# 2024-04-23T16:43:23.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c29413b2:::benchmark_data_smithi188_141483_object35223:head# 2024-04-23T16:43:23.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2942d85:::benchmark_data_smithi188_141483_object60936:head# 2024-04-23T16:43:23.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2946eca:::benchmark_data_smithi188_141483_object65773:head# 2024-04-23T16:43:23.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c294bac4:::benchmark_data_smithi188_141483_object44909:head# 2024-04-23T16:43:23.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2966e40:::benchmark_data_smithi188_141483_object58181:head# 2024-04-23T16:43:23.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2996314:::benchmark_data_smithi188_141483_object57033:head# 2024-04-23T16:43:23.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c29c3db9:::benchmark_data_smithi188_141483_object12422:head# 2024-04-23T16:43:23.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c29c6cc8:::benchmark_data_smithi188_141483_object27488:head# 2024-04-23T16:43:23.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c29c8285:::benchmark_data_smithi188_141483_object23069:head# 2024-04-23T16:43:23.086 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c29ca852:::benchmark_data_smithi188_141483_object14945:head# 2024-04-23T16:43:23.087 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c29cce7c:::benchmark_data_smithi188_141483_object60259:head# 2024-04-23T16:43:23.087 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c29cf8c7:::benchmark_data_smithi188_141483_object39369:head# 2024-04-23T16:43:23.087 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c29d2bd4:::benchmark_data_smithi188_141483_object37716:head# 2024-04-23T16:43:23.088 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c29d5d1a:::benchmark_data_smithi188_141483_object11167:head# 2024-04-23T16:43:23.088 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2a147a6:::benchmark_data_smithi188_141483_object51640:head# 2024-04-23T16:43:23.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2a2b041:::benchmark_data_smithi188_141483_object46539:head# 2024-04-23T16:43:23.089 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2a3960b:::benchmark_data_smithi188_141483_object35136:head# 2024-04-23T16:43:23.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2a503ac:::benchmark_data_smithi188_141483_object60091:head# 2024-04-23T16:43:23.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2a618d3:::benchmark_data_smithi188_141483_object44915:head# 2024-04-23T16:43:23.090 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2a78b5f:::benchmark_data_smithi188_141483_object86324:head# 2024-04-23T16:43:23.091 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2a7afe2:::benchmark_data_smithi188_141483_object28257:head# 2024-04-23T16:43:23.091 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2aa5329:::benchmark_data_smithi188_141483_object83253:head# 2024-04-23T16:43:23.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2aa74f3:::benchmark_data_smithi188_141483_object72603:head# 2024-04-23T16:43:23.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2ab8a46:::benchmark_data_smithi188_141483_object56904:head# 2024-04-23T16:43:23.092 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2ab92db:::benchmark_data_smithi188_141483_object63810:head# 2024-04-23T16:43:23.093 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2abc893:::benchmark_data_smithi188_141483_object1201:head# 2024-04-23T16:43:23.093 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2abe566:::benchmark_data_smithi188_141483_object3006:head# 2024-04-23T16:43:23.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2ac58c3:::benchmark_data_smithi188_141483_object4243:head# 2024-04-23T16:43:23.094 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2ac771a:::benchmark_data_smithi188_141483_object9843:head# 2024-04-23T16:43:23.095 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2acfea2:::benchmark_data_smithi188_141483_object76321:head# 2024-04-23T16:43:23.095 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2adaa0a:::benchmark_data_smithi188_141483_object14327:head# 2024-04-23T16:43:23.095 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2aefa4a:::benchmark_data_smithi188_141483_object42231:head# 2024-04-23T16:43:23.096 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2af9c7a:::benchmark_data_smithi188_141483_object48672:head# 2024-04-23T16:43:23.096 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2afcba4:::benchmark_data_smithi188_141483_object73988:head# 2024-04-23T16:43:23.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2b0b695:::benchmark_data_smithi188_141483_object34793:head# 2024-04-23T16:43:23.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2b23289:::benchmark_data_smithi188_141483_object64167:head# 2024-04-23T16:43:23.097 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2b2b544:::benchmark_data_smithi188_141483_object82237:head# 2024-04-23T16:43:23.098 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2b3959a:::benchmark_data_smithi188_141483_object61507:head# 2024-04-23T16:43:23.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2b457cf:::benchmark_data_smithi188_141483_object85039:head# 2024-04-23T16:43:23.101 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2b549f1:::benchmark_data_smithi188_141483_object34570:head# 2024-04-23T16:43:23.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:22 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3550752204' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:43:23.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2b571e9:::benchmark_data_smithi188_141483_object60403:head# 2024-04-23T16:43:23.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2b5fad6:::benchmark_data_smithi188_141483_object6003:head# 2024-04-23T16:43:23.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2b6ce4d:::benchmark_data_smithi188_141483_object7961:head# 2024-04-23T16:43:23.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2b74e12:::benchmark_data_smithi188_141483_object81126:head# 2024-04-23T16:43:23.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2b75cea:::benchmark_data_smithi188_141483_object48301:head# 2024-04-23T16:43:23.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2b83195:::benchmark_data_smithi188_141483_object73751:head# 2024-04-23T16:43:23.103 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2b85784:::benchmark_data_smithi188_141483_object37994:head# 2024-04-23T16:43:23.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2b99c58:::benchmark_data_smithi188_141483_object33932:head# 2024-04-23T16:43:23.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2b9f6dd:::benchmark_data_smithi188_141483_object80320:head# 2024-04-23T16:43:23.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2baec1c:::benchmark_data_smithi188_141483_object26985:head# 2024-04-23T16:43:23.104 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2bb2b80:::benchmark_data_smithi188_141483_object33342:head# 2024-04-23T16:43:23.105 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2bba694:::benchmark_data_smithi188_141483_object60677:head# 2024-04-23T16:43:23.105 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2bcee68:::benchmark_data_smithi188_141483_object58439:head# 2024-04-23T16:43:23.105 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2be7989:::benchmark_data_smithi188_141483_object71550:head# 2024-04-23T16:43:23.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2beef3b:::benchmark_data_smithi188_141483_object9437:head# 2024-04-23T16:43:23.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2bf857d:::benchmark_data_smithi188_141483_object66205:head# 2024-04-23T16:43:23.106 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2c37270:::benchmark_data_smithi188_141483_object30818:head# 2024-04-23T16:43:23.107 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2c440f3:::benchmark_data_smithi188_141483_object56438:head# 2024-04-23T16:43:23.107 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2c5c400:::benchmark_data_smithi188_141483_object64615:head# 2024-04-23T16:43:23.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2c637a5:::benchmark_data_smithi188_141483_object88196:head# 2024-04-23T16:43:23.108 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2c66a97:::benchmark_data_smithi188_141483_object34816:head# 2024-04-23T16:43:23.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2c69d7c:::benchmark_data_smithi188_141483_object62545:head# 2024-04-23T16:43:23.109 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2c7b946:::benchmark_data_smithi188_141483_object49110:head# 2024-04-23T16:43:23.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2c7c3cb:::benchmark_data_smithi188_141483_object32301:head# 2024-04-23T16:43:23.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2ca5c1c:::benchmark_data_smithi188_141483_object34965:head# 2024-04-23T16:43:23.110 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2ca7694:::benchmark_data_smithi188_141483_object62266:head# 2024-04-23T16:43:23.111 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2cacebf:::benchmark_data_smithi188_141483_object34250:head# 2024-04-23T16:43:23.111 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2cafe21:::benchmark_data_smithi188_141483_object31918:head# 2024-04-23T16:43:23.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2cc2b61:::benchmark_data_smithi188_141483_object73634:head# 2024-04-23T16:43:23.112 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2cc53cf:::benchmark_data_smithi188_141483_object7752:head# 2024-04-23T16:43:23.113 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2cd738c:::benchmark_data_smithi188_141483_object29585:head# 2024-04-23T16:43:23.113 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2cddfc7:::benchmark_data_smithi188_141483_object88100:head# 2024-04-23T16:43:23.113 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2ce3d25:::benchmark_data_smithi188_141483_object30632:head# 2024-04-23T16:43:23.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2cf1fc9:::benchmark_data_smithi188_141483_object53760:head# 2024-04-23T16:43:23.114 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2d1b796:::benchmark_data_smithi188_141483_object19831:head# 2024-04-23T16:43:23.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2d1ca81:::benchmark_data_smithi188_141483_object21485:head# 2024-04-23T16:43:23.115 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2d299e3:::benchmark_data_smithi188_141483_object26040:head# 2024-04-23T16:43:23.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2d2cd0d:::benchmark_data_smithi188_141483_object53540:head# 2024-04-23T16:43:23.116 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2d3b833:::benchmark_data_smithi188_141483_object11599:head# 2024-04-23T16:43:23.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2d492a1:::benchmark_data_smithi188_141483_object44876:head# 2024-04-23T16:43:23.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2d49501:::benchmark_data_smithi188_141483_object17071:head# 2024-04-23T16:43:23.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2d4ab4f:::benchmark_data_smithi188_141483_object23155:head# 2024-04-23T16:43:23.117 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2d59fe3:::benchmark_data_smithi188_141483_object87919:head# 2024-04-23T16:43:23.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2d5c1b5:::benchmark_data_smithi188_141483_object64319:head# 2024-04-23T16:43:23.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2d5d8a2:::benchmark_data_smithi188_141483_object80873:head# 2024-04-23T16:43:23.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2d6d155:::benchmark_data_smithi188_141483_object27659:head# 2024-04-23T16:43:23.119 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2d922cf:::benchmark_data_smithi188_141483_object16318:head# 2024-04-23T16:43:23.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2d9789e:::benchmark_data_smithi188_141483_object48322:head# 2024-04-23T16:43:23.120 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2d9dd63:::benchmark_data_smithi188_141483_object65574:head# 2024-04-23T16:43:23.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2da122a:::benchmark_data_smithi188_141483_object35742:head# 2024-04-23T16:43:23.121 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2da78fc:::benchmark_data_smithi188_141483_object57138:head# 2024-04-23T16:43:23.122 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2daab0b:::benchmark_data_smithi188_141483_object27354:head# 2024-04-23T16:43:23.122 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2db34da:::benchmark_data_smithi188_141483_object62874:head# 2024-04-23T16:43:23.122 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2dc06d2:::benchmark_data_smithi188_141483_object49464:head# 2024-04-23T16:43:23.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2dcd2fc:::benchmark_data_smithi188_141483_object56364:head# 2024-04-23T16:43:23.123 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2dcf439:::benchmark_data_smithi188_141483_object88812:head# 2024-04-23T16:43:23.124 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2dcf857:::benchmark_data_smithi188_141483_object80839:head# 2024-04-23T16:43:23.124 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2ddb271:::benchmark_data_smithi188_141483_object10876:head# 2024-04-23T16:43:23.124 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2de4cd6:::benchmark_data_smithi188_141483_object15360:head# 2024-04-23T16:43:23.125 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2e0868f:::benchmark_data_smithi188_141483_object38413:head# 2024-04-23T16:43:23.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2e0e397:::benchmark_data_smithi188_141483_object22216:head# 2024-04-23T16:43:23.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2e23d3d:::benchmark_data_smithi188_141483_object32695:head# 2024-04-23T16:43:23.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2e273c3:::benchmark_data_smithi188_141483_object60153:head# 2024-04-23T16:43:23.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2e386f6:::benchmark_data_smithi188_141483_object16678:head# 2024-04-23T16:43:23.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2e47a92:::benchmark_data_smithi188_141483_object2107:head# 2024-04-23T16:43:23.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2e81807:::benchmark_data_smithi188_141483_object16911:head# 2024-04-23T16:43:23.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2e858de:::benchmark_data_smithi188_141483_object16974:head# 2024-04-23T16:43:23.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2e93415:::benchmark_data_smithi188_141483_object767:head# 2024-04-23T16:43:23.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2ea17a1:::benchmark_data_smithi188_141483_object7537:head# 2024-04-23T16:43:23.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2ea1e24:::benchmark_data_smithi188_141483_object74329:head# 2024-04-23T16:43:23.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2eaaed5:::benchmark_data_smithi188_141483_object83611:head# 2024-04-23T16:43:23.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2eb6377:::benchmark_data_smithi188_141483_object58306:head# 2024-04-23T16:43:23.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2ed49b3:::benchmark_data_smithi188_141483_object318:head# 2024-04-23T16:43:23.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2eeb37b:::benchmark_data_smithi188_141483_object36630:head# 2024-04-23T16:43:23.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2eee0d5:::benchmark_data_smithi188_141483_object10081:head# 2024-04-23T16:43:23.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2f00f97:::benchmark_data_smithi188_141483_object391:head# 2024-04-23T16:43:23.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2f0e481:::benchmark_data_smithi188_141483_object2386:head# 2024-04-23T16:43:23.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2f118c4:::benchmark_data_smithi188_141483_object12306:head# 2024-04-23T16:43:23.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2f49a51:::benchmark_data_smithi188_141483_object21861:head# 2024-04-23T16:43:23.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2f4fa9b:::benchmark_data_smithi188_141483_object85349:head# 2024-04-23T16:43:23.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2f51a3d:::benchmark_data_smithi188_141483_object60033:head# 2024-04-23T16:43:23.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2f59492:::benchmark_data_smithi188_141483_object71348:head# 2024-04-23T16:43:23.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2f75052:::benchmark_data_smithi188_141483_object67724:head# 2024-04-23T16:43:23.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2f75706:::benchmark_data_smithi188_141483_object83394:head# 2024-04-23T16:43:23.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2f7b1ff:::benchmark_data_smithi188_141483_object35043:head# 2024-04-23T16:43:23.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2f7d872:::benchmark_data_smithi188_141483_object36383:head# 2024-04-23T16:43:23.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2f7e713:::benchmark_data_smithi188_141483_object29779:head# 2024-04-23T16:43:23.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2f7fb6e:::benchmark_data_smithi188_141483_object51290:head# 2024-04-23T16:43:23.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2f830c1:::benchmark_data_smithi188_141483_object20348:head# 2024-04-23T16:43:23.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2f90901:::benchmark_data_smithi188_141483_object33863:head# 2024-04-23T16:43:23.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2f92e52:::benchmark_data_smithi188_141483_object88918:head# 2024-04-23T16:43:23.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2f9b440:::benchmark_data_smithi188_141483_object19616:head# 2024-04-23T16:43:23.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2fb56e4:::benchmark_data_smithi188_141483_object89732:head# 2024-04-23T16:43:23.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2fbb792:::benchmark_data_smithi188_141483_object85156:head# 2024-04-23T16:43:23.142 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2fc17af:::benchmark_data_smithi188_141483_object73358:head# 2024-04-23T16:43:23.142 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2fc1eb0:::benchmark_data_smithi188_141483_object12501:head# 2024-04-23T16:43:23.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2fd49f4:::benchmark_data_smithi188_141483_object4626:head# 2024-04-23T16:43:23.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c2ffea61:::benchmark_data_smithi188_141483_object32345:head# 2024-04-23T16:43:23.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3000f01:::benchmark_data_smithi188_141483_object45816:head# 2024-04-23T16:43:23.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3013c24:::benchmark_data_smithi188_141483_object1652:head# 2024-04-23T16:43:23.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c301dc38:::benchmark_data_smithi188_141483_object72438:head# 2024-04-23T16:43:23.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3026a26:::benchmark_data_smithi188_141483_object9198:head# 2024-04-23T16:43:23.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c302b331:::benchmark_data_smithi188_141483_object37663:head# 2024-04-23T16:43:23.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c306eb3a:::benchmark_data_smithi188_141483_object18770:head# 2024-04-23T16:43:23.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c306f804:::benchmark_data_smithi188_141483_object34154:head# 2024-04-23T16:43:23.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c308c808:::benchmark_data_smithi188_141483_object84817:head# 2024-04-23T16:43:23.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3092596:::benchmark_data_smithi188_141483_object5171:head# 2024-04-23T16:43:23.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c30a1aa5:::benchmark_data_smithi188_141483_object16406:head# 2024-04-23T16:43:23.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c30abba7:::benchmark_data_smithi188_141483_object79005:head# 2024-04-23T16:43:23.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c30d21be:::benchmark_data_smithi188_141483_object34063:head# 2024-04-23T16:43:23.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c30e1df4:::benchmark_data_smithi188_141483_object40729:head# 2024-04-23T16:43:23.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c30ee06d:::benchmark_data_smithi188_141483_object27120:head# 2024-04-23T16:43:23.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c30fab69:::benchmark_data_smithi188_141483_object1128:head# 2024-04-23T16:43:23.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c30fc6a0:::benchmark_data_smithi188_141483_object12674:head# 2024-04-23T16:43:23.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c314784b:::benchmark_data_smithi188_141483_object67267:head# 2024-04-23T16:43:23.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c31678ed:::benchmark_data_smithi188_141483_object23591:head# 2024-04-23T16:43:23.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c31734d5:::benchmark_data_smithi188_141483_object39904:head# 2024-04-23T16:43:23.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3180f6a:::benchmark_data_smithi188_141483_object78895:head# 2024-04-23T16:43:23.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c31894cd:::benchmark_data_smithi188_141483_object17134:head# 2024-04-23T16:43:23.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c318de5b:::benchmark_data_smithi188_141483_object12039:head# 2024-04-23T16:43:23.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c31933ba:::benchmark_data_smithi188_141483_object33252:head# 2024-04-23T16:43:23.155 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3194b9a:::benchmark_data_smithi188_141483_object20490:head# 2024-04-23T16:43:23.155 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c319cecd:::benchmark_data_smithi188_141483_object17511:head# 2024-04-23T16:43:23.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c31a6474:::benchmark_data_smithi188_141483_object57351:head# 2024-04-23T16:43:23.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c31b4374:::benchmark_data_smithi188_141483_object76866:head# 2024-04-23T16:43:23.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c31cc01e:::benchmark_data_smithi188_141483_object84075:head# 2024-04-23T16:43:23.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c31e5132:::benchmark_data_smithi188_141483_object18357:head# 2024-04-23T16:43:23.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c31e608f:::benchmark_data_smithi188_141483_object35492:head# 2024-04-23T16:43:23.158 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c31ed402:::benchmark_data_smithi188_141483_object43732:head# 2024-04-23T16:43:23.158 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c31f1f3b:::benchmark_data_smithi188_141483_object49434:head# 2024-04-23T16:43:23.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3200ff3:::benchmark_data_smithi188_141483_object46620:head# 2024-04-23T16:43:23.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32067ab:::benchmark_data_smithi188_141483_object47209:head# 2024-04-23T16:43:23.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c320760f:::benchmark_data_smithi188_141483_object25213:head# 2024-04-23T16:43:23.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c320897d:::benchmark_data_smithi188_141483_object88334:head# 2024-04-23T16:43:23.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c322e1a4:::benchmark_data_smithi188_141483_object30426:head# 2024-04-23T16:43:23.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3233b6c:::benchmark_data_smithi188_141483_object63224:head# 2024-04-23T16:43:23.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32396c5:::benchmark_data_smithi188_141483_object37103:head# 2024-04-23T16:43:23.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3240ae0:::benchmark_data_smithi188_141483_object60702:head# 2024-04-23T16:43:23.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32534f4:::benchmark_data_smithi188_141483_object11748:head# 2024-04-23T16:43:23.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32592c0:::benchmark_data_smithi188_141483_object3605:head# 2024-04-23T16:43:23.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c325db1d:::benchmark_data_smithi188_141483_object39301:head# 2024-04-23T16:43:23.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c325dfb6:::benchmark_data_smithi188_141483_object8377:head# 2024-04-23T16:43:23.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3267586:::benchmark_data_smithi188_141483_object75481:head# 2024-04-23T16:43:23.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32a5d75:::benchmark_data_smithi188_141483_object65551:head# 2024-04-23T16:43:23.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32ad96c:::benchmark_data_smithi188_141483_object8210:head# 2024-04-23T16:43:23.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32adbaf:::benchmark_data_smithi188_141483_object14578:head# 2024-04-23T16:43:23.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32ba3eb:::benchmark_data_smithi188_141483_object77403:head# 2024-04-23T16:43:23.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32bb56d:::benchmark_data_smithi188_141483_object68188:head# 2024-04-23T16:43:23.167 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32c2263:::benchmark_data_smithi188_141483_object38307:head# 2024-04-23T16:43:23.167 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32c3285:::benchmark_data_smithi188_141483_object46829:head# 2024-04-23T16:43:23.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32c33f3:::benchmark_data_smithi188_141483_object5668:head# 2024-04-23T16:43:23.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32c5eb0:::benchmark_data_smithi188_141483_object15575:head# 2024-04-23T16:43:23.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32cccb2:::benchmark_data_smithi188_141483_object78724:head# 2024-04-23T16:43:23.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32d0ed6:::benchmark_data_smithi188_141483_object80923:head# 2024-04-23T16:43:23.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32d4e8c:::benchmark_data_smithi188_141483_object84175:head# 2024-04-23T16:43:23.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32d5532:::benchmark_data_smithi188_141483_object38241:head# 2024-04-23T16:43:23.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32e59da:::benchmark_data_smithi188_141483_object66182:head# 2024-04-23T16:43:23.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32f2dae:::benchmark_data_smithi188_141483_object82446:head# 2024-04-23T16:43:23.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c32f6254:::benchmark_data_smithi188_141483_object47870:head# 2024-04-23T16:43:23.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3300544:::benchmark_data_smithi188_141483_object72442:head# 2024-04-23T16:43:23.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3306170:::benchmark_data_smithi188_141483_object22112:head# 2024-04-23T16:43:23.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c33066f4:::benchmark_data_smithi188_141483_object40935:head# 2024-04-23T16:43:23.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c330efbc:::benchmark_data_smithi188_141483_object43311:head# 2024-04-23T16:43:23.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3317c2c:::benchmark_data_smithi188_141483_object59117:head# 2024-04-23T16:43:23.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3322b09:::benchmark_data_smithi188_141483_object56352:head# 2024-04-23T16:43:23.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c33307e1:::benchmark_data_smithi188_141483_object80348:head# 2024-04-23T16:43:23.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c334bf68:::benchmark_data_smithi188_141483_object15783:head# 2024-04-23T16:43:23.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c337196d:::benchmark_data_smithi188_141483_object33062:head# 2024-04-23T16:43:23.176 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c33886ef:::benchmark_data_smithi188_141483_object49517:head# 2024-04-23T16:43:23.176 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c338f48e:::benchmark_data_smithi188_141483_object78687:head# 2024-04-23T16:43:23.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c33a2ed8:::benchmark_data_smithi188_141483_object76758:head# 2024-04-23T16:43:23.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c33aae49:::benchmark_data_smithi188_141483_object37620:head# 2024-04-23T16:43:23.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c33ab8b1:::benchmark_data_smithi188_141483_object39434:head# 2024-04-23T16:43:23.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c33b0621:::benchmark_data_smithi188_141483_object9351:head# 2024-04-23T16:43:23.181 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c33b0ef2:::benchmark_data_smithi188_141483_object56918:head# 2024-04-23T16:43:23.181 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c33cc72f:::benchmark_data_smithi188_141483_object69077:head# 2024-04-23T16:43:23.181 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c33d892f:::benchmark_data_smithi188_141483_object68908:head# 2024-04-23T16:43:23.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c33dcf03:::benchmark_data_smithi188_141483_object14717:head# 2024-04-23T16:43:23.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c33f045f:::benchmark_data_smithi188_141483_object77707:head# 2024-04-23T16:43:23.183 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3408da7:::benchmark_data_smithi188_141483_object12190:head# 2024-04-23T16:43:23.183 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3416685:::benchmark_data_smithi188_141483_object39524:head# 2024-04-23T16:43:23.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c341b2b8:::benchmark_data_smithi188_141483_object18185:head# 2024-04-23T16:43:23.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c341d003:::benchmark_data_smithi188_141483_object28581:head# 2024-04-23T16:43:23.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c342b937:::benchmark_data_smithi188_141483_object70885:head# 2024-04-23T16:43:23.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c342d78b:::benchmark_data_smithi188_141483_object48653:head# 2024-04-23T16:43:23.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c343005a:::benchmark_data_smithi188_141483_object68996:head# 2024-04-23T16:43:23.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c34333b4:::benchmark_data_smithi188_141483_object69628:head# 2024-04-23T16:43:23.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c343a147:::benchmark_data_smithi188_141483_object43540:head# 2024-04-23T16:43:23.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c344c611:::benchmark_data_smithi188_141483_object85774:head# 2024-04-23T16:43:23.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c345a56c:::benchmark_data_smithi188_141483_object6590:head# 2024-04-23T16:43:23.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c34688fd:::benchmark_data_smithi188_141483_object5343:head# 2024-04-23T16:43:23.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c347f31c:::benchmark_data_smithi188_141483_object20220:head# 2024-04-23T16:43:23.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c348c6ca:::benchmark_data_smithi188_141483_object29175:head# 2024-04-23T16:43:23.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c34a0442:::benchmark_data_smithi188_141483_object7197:head# 2024-04-23T16:43:23.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c34a0e0d:::benchmark_data_smithi188_141483_object26921:head# 2024-04-23T16:43:23.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c34afb3b:::benchmark_data_smithi188_141483_object70377:head# 2024-04-23T16:43:23.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c34bcfff:::benchmark_data_smithi188_141483_object77577:head# 2024-04-23T16:43:23.192 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c34bfe82:::benchmark_data_smithi188_141483_object45155:head# 2024-04-23T16:43:23.192 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c34ca44f:::benchmark_data_smithi188_141483_object7279:head# 2024-04-23T16:43:23.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c34ca846:::benchmark_data_smithi188_141483_object39581:head# 2024-04-23T16:43:23.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c34fe3f4:::benchmark_data_smithi188_141483_object61152:head# 2024-04-23T16:43:23.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3516d0e:::benchmark_data_smithi188_141483_object18673:head# 2024-04-23T16:43:23.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c35316ad:::benchmark_data_smithi188_141483_object59144:head# 2024-04-23T16:43:23.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c354406b:::benchmark_data_smithi188_141483_object52642:head# 2024-04-23T16:43:23.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3550791:::benchmark_data_smithi188_141483_object37905:head# 2024-04-23T16:43:23.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c35637d9:::benchmark_data_smithi188_141483_object44383:head# 2024-04-23T16:43:23.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c357d6bf:::benchmark_data_smithi188_141483_object11374:head# 2024-04-23T16:43:23.197 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3583b47:::benchmark_data_smithi188_141483_object34617:head# 2024-04-23T16:43:23.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3594a91:::benchmark_data_smithi188_141483_object11953:head# 2024-04-23T16:43:23.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c35a6e60:::benchmark_data_smithi188_141483_object7471:head# 2024-04-23T16:43:23.199 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c35a8101:::benchmark_data_smithi188_141483_object3490:head# 2024-04-23T16:43:23.199 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c35c7245:::benchmark_data_smithi188_141483_object55911:head# 2024-04-23T16:43:23.199 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c35cb96a:::benchmark_data_smithi188_141483_object72971:head# 2024-04-23T16:43:23.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c35d4f9d:::benchmark_data_smithi188_141483_object54764:head# 2024-04-23T16:43:23.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c35d7f38:::benchmark_data_smithi188_141483_object55284:head# 2024-04-23T16:43:23.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c35dad34:::benchmark_data_smithi188_141483_object23565:head# 2024-04-23T16:43:23.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c35def0a:::benchmark_data_smithi188_141483_object86912:head# 2024-04-23T16:43:23.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c35ed003:::benchmark_data_smithi188_141483_object67819:head# 2024-04-23T16:43:23.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c35f497a:::benchmark_data_smithi188_141483_object69417:head# 2024-04-23T16:43:23.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c35f4f6d:::benchmark_data_smithi188_141483_object15193:head# 2024-04-23T16:43:23.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c35f8ffb:::benchmark_data_smithi188_141483_object27160:head# 2024-04-23T16:43:23.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3608491:::benchmark_data_smithi188_141483_object83080:head# 2024-04-23T16:43:23.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3615654:::benchmark_data_smithi188_141483_object54616:head# 2024-04-23T16:43:23.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3626fa8:::benchmark_data_smithi188_141483_object15000:head# 2024-04-23T16:43:23.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c36299ff:::benchmark_data_smithi188_141483_object58404:head# 2024-04-23T16:43:23.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c362b1fd:::benchmark_data_smithi188_141483_object22816:head# 2024-04-23T16:43:23.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3649c52:::benchmark_data_smithi188_141483_object35495:head# 2024-04-23T16:43:23.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3654180:::benchmark_data_smithi188_141483_object39232:head# 2024-04-23T16:43:23.206 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c365b64d:::benchmark_data_smithi188_141483_object51877:head# 2024-04-23T16:43:23.206 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c365c261:::benchmark_data_smithi188_141483_object72446:head# 2024-04-23T16:43:23.207 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c365dcce:::benchmark_data_smithi188_141483_object46388:head# 2024-04-23T16:43:23.207 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3665dec:::benchmark_data_smithi188_141483_object16218:head# 2024-04-23T16:43:23.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c36687c6:::benchmark_data_smithi188_141483_object44166:head# 2024-04-23T16:43:23.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c367069f:::benchmark_data_smithi188_141483_object11840:head# 2024-04-23T16:43:23.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c36766ae:::benchmark_data_smithi188_141483_object4789:head# 2024-04-23T16:43:23.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c36842aa:::benchmark_data_smithi188_141483_object60457:head# 2024-04-23T16:43:23.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3693573:::benchmark_data_smithi188_141483_object62622:head# 2024-04-23T16:43:23.212 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3695a90:::benchmark_data_smithi188_141483_object74017:head# 2024-04-23T16:43:23.212 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c36a5b23:::benchmark_data_smithi188_141483_object54742:head# 2024-04-23T16:43:23.213 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c36a77de:::benchmark_data_smithi188_141483_object54812:head# 2024-04-23T16:43:23.213 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c36b941f:::benchmark_data_smithi188_141483_object88958:head# 2024-04-23T16:43:23.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c36bf692:::benchmark_data_smithi188_141483_object5006:head# 2024-04-23T16:43:23.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c36c1cd1:::benchmark_data_smithi188_141483_object38440:head# 2024-04-23T16:43:23.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c36cb4d8:::benchmark_data_smithi188_141483_object42435:head# 2024-04-23T16:43:23.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c36cca75:::benchmark_data_smithi188_141483_object23602:head# 2024-04-23T16:43:23.216 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c36d459e:::benchmark_data_smithi188_141483_object71659:head# 2024-04-23T16:43:23.216 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c36d8263:::benchmark_data_smithi188_141483_object76925:head# 2024-04-23T16:43:23.217 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c36fc3c6:::benchmark_data_smithi188_141483_object58769:head# 2024-04-23T16:43:23.217 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3702b0e:::benchmark_data_smithi188_141483_object28137:head# 2024-04-23T16:43:23.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c371a80a:::benchmark_data_smithi188_141483_object4883:head# 2024-04-23T16:43:23.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3720826:::benchmark_data_smithi188_141483_object26240:head# 2024-04-23T16:43:23.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c373d501:::benchmark_data_smithi188_141483_object54105:head# 2024-04-23T16:43:23.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c373e2f6:::benchmark_data_smithi188_141483_object46754:head# 2024-04-23T16:43:23.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c375706f:::benchmark_data_smithi188_141483_object71377:head# 2024-04-23T16:43:23.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c378775c:::benchmark_data_smithi188_141483_object61584:head# 2024-04-23T16:43:23.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c379e20e:::benchmark_data_smithi188_141483_object4294:head# 2024-04-23T16:43:23.221 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c37a5ae5:::benchmark_data_smithi188_141483_object17396:head# 2024-04-23T16:43:23.221 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c37b5b68:::benchmark_data_smithi188_141483_object82037:head# 2024-04-23T16:43:23.222 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c37c605c:::benchmark_data_smithi188_141483_object90698:head# 2024-04-23T16:43:23.222 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c37cb484:::benchmark_data_smithi188_141483_object83904:head# 2024-04-23T16:43:23.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c37cf2fc:::benchmark_data_smithi188_141483_object3140:head# 2024-04-23T16:43:23.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c37d1f31:::benchmark_data_smithi188_141483_object76216:head# 2024-04-23T16:43:23.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:22 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3550752204' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:43:23.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c380a4a0:::benchmark_data_smithi188_141483_object34481:head# 2024-04-23T16:43:23.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3810c59:::benchmark_data_smithi188_141483_object59013:head# 2024-04-23T16:43:23.231 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3820fc1:::benchmark_data_smithi188_141483_object37354:head# 2024-04-23T16:43:23.231 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3834e1b:::benchmark_data_smithi188_141483_object4082:head# 2024-04-23T16:43:23.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c38390b4:::benchmark_data_smithi188_141483_object23737:head# 2024-04-23T16:43:23.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3840756:::benchmark_data_smithi188_141483_object88505:head# 2024-04-23T16:43:23.235 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3845551:::benchmark_data_smithi188_141483_object54405:head# 2024-04-23T16:43:23.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c386b118:::benchmark_data_smithi188_141483_object41942:head# 2024-04-23T16:43:23.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3870e14:::benchmark_data_smithi188_141483_object22294:head# 2024-04-23T16:43:23.237 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c387ebb7:::benchmark_data_smithi188_141483_object7613:head# 2024-04-23T16:43:23.237 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c388042d:::benchmark_data_smithi188_141483_object3030:head# 2024-04-23T16:43:23.237 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c388ba60:::benchmark_data_smithi188_141483_object31450:head# 2024-04-23T16:43:23.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c388f97c:::benchmark_data_smithi188_141483_object72899:head# 2024-04-23T16:43:23.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c38cab74:::benchmark_data_smithi188_141483_object83643:head# 2024-04-23T16:43:23.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c38d04a5:::benchmark_data_smithi188_141483_object90901:head# 2024-04-23T16:43:23.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c38de385:::benchmark_data_smithi188_141483_object76582:head# 2024-04-23T16:43:23.240 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c38e1d3d:::benchmark_data_smithi188_141483_object24655:head# 2024-04-23T16:43:23.240 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3904301:::benchmark_data_smithi188_141483_object88423:head# 2024-04-23T16:43:23.240 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c39067a7:::benchmark_data_smithi188_141483_object87376:head# 2024-04-23T16:43:23.241 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3914ba5:::benchmark_data_smithi188_141483_object90478:head# 2024-04-23T16:43:23.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c39160f1:::benchmark_data_smithi188_141483_object51505:head# 2024-04-23T16:43:23.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c392d981:::benchmark_data_smithi188_141483_object78242:head# 2024-04-23T16:43:23.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3930e65:::benchmark_data_smithi188_141483_object10027:head# 2024-04-23T16:43:23.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3932e6a:::benchmark_data_smithi188_141483_object52527:head# 2024-04-23T16:43:23.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c39427ab:::benchmark_data_smithi188_141483_object80050:head# 2024-04-23T16:43:23.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c39562c1:::benchmark_data_smithi188_141483_object12063:head# 2024-04-23T16:43:23.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c395d6af:::benchmark_data_smithi188_141483_object64062:head# 2024-04-23T16:43:23.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c39644d2:::benchmark_data_smithi188_141483_object22072:head# 2024-04-23T16:43:23.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3965ab7:::benchmark_data_smithi188_141483_object3114:head# 2024-04-23T16:43:23.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3965c70:::benchmark_data_smithi188_141483_object66653:head# 2024-04-23T16:43:23.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3969af7:::benchmark_data_smithi188_141483_object84859:head# 2024-04-23T16:43:23.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c396a6a4:::benchmark_data_smithi188_141483_object52479:head# 2024-04-23T16:43:23.248 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3978648:::benchmark_data_smithi188_141483_object84955:head# 2024-04-23T16:43:23.248 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c397e9a6:::benchmark_data_smithi188_141483_object57317:head# 2024-04-23T16:43:23.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3984374:::benchmark_data_smithi188_141483_object18093:head# 2024-04-23T16:43:23.251 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-23T16:43:23.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c398dbd2:::benchmark_data_smithi188_141483_object41683:head# 2024-04-23T16:43:23.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c399af63:::benchmark_data_smithi188_141483_object36351:head# 2024-04-23T16:43:23.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c39a4614:::benchmark_data_smithi188_141483_object22628:head# 2024-04-23T16:43:23.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c39a735c:::benchmark_data_smithi188_141483_object49036:head# 2024-04-23T16:43:23.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c39ae074:::benchmark_data_smithi188_141483_object45939:head# 2024-04-23T16:43:23.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c39b2d0a:::benchmark_data_smithi188_141483_object17979:head# 2024-04-23T16:43:23.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c39b954d:::benchmark_data_smithi188_141483_object65861:head# 2024-04-23T16:43:23.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c39bbd6b:::benchmark_data_smithi188_141483_object6902:head# 2024-04-23T16:43:23.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c39e6ea2:::benchmark_data_smithi188_141483_object46894:head# 2024-04-23T16:43:23.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3a181a4:::benchmark_data_smithi188_141483_object37460:head# 2024-04-23T16:43:23.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3a1fb06:::benchmark_data_smithi188_141483_object77404:head# 2024-04-23T16:43:23.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3a2198f:::benchmark_data_smithi188_141483_object10127:head# 2024-04-23T16:43:23.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3a2d752:::benchmark_data_smithi188_141483_object58070:head# 2024-04-23T16:43:23.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3a2e343:::benchmark_data_smithi188_141483_object16220:head# 2024-04-23T16:43:23.258 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3a2ebb0:::benchmark_data_smithi188_141483_object16103:head# 2024-04-23T16:43:23.258 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3a35207:::benchmark_data_smithi188_141483_object12023:head# 2024-04-23T16:43:23.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3a37e5f:::benchmark_data_smithi188_141483_object58730:head# 2024-04-23T16:43:23.260 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:22 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3550752204' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:43:23.262 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3a3c8cc:::benchmark_data_smithi188_141483_object50728:head# 2024-04-23T16:43:23.262 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3a4618c:::benchmark_data_smithi188_141483_object73140:head# 2024-04-23T16:43:23.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3a5082b:::benchmark_data_smithi188_141483_object46746:head# 2024-04-23T16:43:23.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3a68cdf:::benchmark_data_smithi188_141483_object39644:head# 2024-04-23T16:43:23.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3a70bb6:::benchmark_data_smithi188_141483_object39404:head# 2024-04-23T16:43:23.266 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3aa8592:::benchmark_data_smithi188_141483_object31167:head# 2024-04-23T16:43:23.266 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3ad5332:::benchmark_data_smithi188_141483_object65491:head# 2024-04-23T16:43:23.267 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3ae1cad:::benchmark_data_smithi188_141483_object54762:head# 2024-04-23T16:43:23.267 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3aec645:::benchmark_data_smithi188_141483_object72228:head# 2024-04-23T16:43:23.268 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3af80ec:::benchmark_data_smithi188_141483_object36683:head# 2024-04-23T16:43:23.268 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3b07139:::benchmark_data_smithi188_141483_object34764:head# 2024-04-23T16:43:23.268 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3b17a43:::benchmark_data_smithi188_141483_object30252:head# 2024-04-23T16:43:23.269 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3b2a2fc:::benchmark_data_smithi188_141483_object66388:head# 2024-04-23T16:43:23.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3b42545:::benchmark_data_smithi188_141483_object10252:head# 2024-04-23T16:43:23.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3b4cffa:::benchmark_data_smithi188_141483_object46757:head# 2024-04-23T16:43:23.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3b4e84f:::benchmark_data_smithi188_141483_object8912:head# 2024-04-23T16:43:23.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3b50f72:::benchmark_data_smithi188_141483_object58165:head# 2024-04-23T16:43:23.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3b56528:::benchmark_data_smithi188_141483_object4996:head# 2024-04-23T16:43:23.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3b68c2c:::benchmark_data_smithi188_141483_object75317:head# 2024-04-23T16:43:23.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3b6c6a1:::benchmark_data_smithi188_141483_object46494:head# 2024-04-23T16:43:23.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3b7045d:::benchmark_data_smithi188_141483_object16522:head# 2024-04-23T16:43:23.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3b763e6:::benchmark_data_smithi188_141483_object64175:head# 2024-04-23T16:43:23.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3b8a3df:::benchmark_data_smithi188_141483_object37858:head# 2024-04-23T16:43:23.277 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3b8dd71:::benchmark_data_smithi188_141483_object52165:head# 2024-04-23T16:43:23.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3b956e0:::benchmark_data_smithi188_141483_object25140:head# 2024-04-23T16:43:23.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3ba3e51:::benchmark_data_smithi188_141483_object27241:head# 2024-04-23T16:43:23.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3ba4c7f:::benchmark_data_smithi188_141483_object85331:head# 2024-04-23T16:43:23.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3ba5016:::benchmark_data_smithi188_141483_object73957:head# 2024-04-23T16:43:23.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3bb516f:::benchmark_data_smithi188_141483_object15361:head# 2024-04-23T16:43:23.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3bb85ed:::benchmark_data_smithi188_141483_object30571:head# 2024-04-23T16:43:23.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3bbd632:::benchmark_data_smithi188_141483_object78260:head# 2024-04-23T16:43:23.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3bd6de5:::benchmark_data_smithi188_141483_object68504:head# 2024-04-23T16:43:23.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3bd85fc:::benchmark_data_smithi188_141483_object43352:head# 2024-04-23T16:43:23.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3bde610:::benchmark_data_smithi188_141483_object19462:head# 2024-04-23T16:43:23.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3be846e:::benchmark_data_smithi188_141483_object51336:head# 2024-04-23T16:43:23.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3c02923:::benchmark_data_smithi188_141483_object50857:head# 2024-04-23T16:43:23.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3c1834e:::benchmark_data_smithi188_141483_object22090:head# 2024-04-23T16:43:23.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3c187eb:::benchmark_data_smithi188_141483_object79409:head# 2024-04-23T16:43:23.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3c21d94:::benchmark_data_smithi188_141483_object43795:head# 2024-04-23T16:43:23.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3c438f8:::benchmark_data_smithi188_141483_object53780:head# 2024-04-23T16:43:23.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3c47246:::benchmark_data_smithi188_141483_object23345:head# 2024-04-23T16:43:23.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3c4d998:::benchmark_data_smithi188_141483_object48739:head# 2024-04-23T16:43:23.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3c4ea9a:::benchmark_data_smithi188_141483_object26461:head# 2024-04-23T16:43:23.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3c76040:::benchmark_data_smithi188_141483_object47486:head# 2024-04-23T16:43:23.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3c7b781:::benchmark_data_smithi188_141483_object82739:head# 2024-04-23T16:43:23.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3c86823:::benchmark_data_smithi188_141483_object349:head# 2024-04-23T16:43:23.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3c86bef:::benchmark_data_smithi188_141483_object32805:head# 2024-04-23T16:43:23.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3c880c7:::benchmark_data_smithi188_141483_object77616:head# 2024-04-23T16:43:23.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3c89d62:::benchmark_data_smithi188_141483_object34167:head# 2024-04-23T16:43:23.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3c8eb6a:::benchmark_data_smithi188_141483_object62981:head# 2024-04-23T16:43:23.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3c938dc:::benchmark_data_smithi188_141483_object32392:head# 2024-04-23T16:43:23.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3caed10:::benchmark_data_smithi188_141483_object14655:head# 2024-04-23T16:43:23.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3cc69ae:::benchmark_data_smithi188_141483_object58312:head# 2024-04-23T16:43:23.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3cca222:::benchmark_data_smithi188_141483_object80382:head# 2024-04-23T16:43:23.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3cd296b:::benchmark_data_smithi188_141483_object4630:head# 2024-04-23T16:43:23.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3cdb9fc:::benchmark_data_smithi188_141483_object84326:head# 2024-04-23T16:43:23.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3d038ee:::benchmark_data_smithi188_141483_object86195:head# 2024-04-23T16:43:23.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3d0a52e:::benchmark_data_smithi188_141483_object40921:head# 2024-04-23T16:43:23.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3d17fdc:::benchmark_data_smithi188_141483_object13335:head# 2024-04-23T16:43:23.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3d2422a:::benchmark_data_smithi188_141483_object89309:head# 2024-04-23T16:43:23.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3d2a6b0:::benchmark_data_smithi188_141483_object46530:head# 2024-04-23T16:43:23.295 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3d2c76f:::benchmark_data_smithi188_141483_object113:head# 2024-04-23T16:43:23.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3d3a352:::benchmark_data_smithi188_141483_object23159:head# 2024-04-23T16:43:23.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3d52029:::benchmark_data_smithi188_141483_object29937:head# 2024-04-23T16:43:23.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3d5cce0:::benchmark_data_smithi188_141483_object11734:head# 2024-04-23T16:43:23.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3d5eedf:::benchmark_data_smithi188_141483_object36978:head# 2024-04-23T16:43:23.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3d62e84:::benchmark_data_smithi188_141483_object35718:head# 2024-04-23T16:43:23.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3d752a9:::benchmark_data_smithi188_141483_object46113:head# 2024-04-23T16:43:23.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3d7b1de:::benchmark_data_smithi188_141483_object26536:head# 2024-04-23T16:43:23.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3d92efe:::benchmark_data_smithi188_141483_object10047:head# 2024-04-23T16:43:23.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3d9ef42:::benchmark_data_smithi188_141483_object40024:head# 2024-04-23T16:43:23.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3db0529:::benchmark_data_smithi188_141483_object12551:head# 2024-04-23T16:43:23.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3dbdcf9:::benchmark_data_smithi188_141483_object59292:head# 2024-04-23T16:43:23.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3dbfa66:::benchmark_data_smithi188_141483_object66588:head# 2024-04-23T16:43:23.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3dc7bbe:::benchmark_data_smithi188_141483_object5111:head# 2024-04-23T16:43:23.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3de4c69:::benchmark_data_smithi188_141483_object45843:head# 2024-04-23T16:43:23.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3e067a3:::benchmark_data_smithi188_141483_object78988:head# 2024-04-23T16:43:23.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3e0e303:::benchmark_data_smithi188_141483_object50072:head# 2024-04-23T16:43:23.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3e1ce51:::benchmark_data_smithi188_141483_object46534:head# 2024-04-23T16:43:23.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3e23775:::benchmark_data_smithi188_141483_object45925:head# 2024-04-23T16:43:23.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3e2c194:::benchmark_data_smithi188_141483_object34051:head# 2024-04-23T16:43:23.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3e3299d:::benchmark_data_smithi188_141483_object87859:head# 2024-04-23T16:43:23.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3e3a00e:::benchmark_data_smithi188_141483_object33617:head# 2024-04-23T16:43:23.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3e3e073:::benchmark_data_smithi188_141483_object28054:head# 2024-04-23T16:43:23.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3e704e9:::benchmark_data_smithi188_141483_object15445:head# 2024-04-23T16:43:23.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3e773f7:::benchmark_data_smithi188_141483_object74710:head# 2024-04-23T16:43:23.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3e7a19b:::benchmark_data_smithi188_141483_object48206:head# 2024-04-23T16:43:23.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3e7a959:::benchmark_data_smithi188_141483_object87270:head# 2024-04-23T16:43:23.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3e7bb4d:::benchmark_data_smithi188_141483_object6485:head# 2024-04-23T16:43:23.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3e8a154:::benchmark_data_smithi188_141483_object76319:head# 2024-04-23T16:43:23.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3e8af2d:::benchmark_data_smithi188_141483_object23952:head# 2024-04-23T16:43:23.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3e979a5:::benchmark_data_smithi188_141483_object6537:head# 2024-04-23T16:43:23.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3e97c92:::benchmark_data_smithi188_141483_object10978:head# 2024-04-23T16:43:23.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3eabd40:::benchmark_data_smithi188_141483_object81140:head# 2024-04-23T16:43:23.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3ebad5e:::benchmark_data_smithi188_141483_object63813:head# 2024-04-23T16:43:23.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3ec110c:::benchmark_data_smithi188_141483_object41925:head# 2024-04-23T16:43:23.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3ed4d89:::benchmark_data_smithi188_141483_object90535:head# 2024-04-23T16:43:23.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3ed54cd:::benchmark_data_smithi188_141483_object34255:head# 2024-04-23T16:43:23.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3edd47f:::benchmark_data_smithi188_141483_object17651:head# 2024-04-23T16:43:23.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3f2af37:::benchmark_data_smithi188_141483_object62567:head# 2024-04-23T16:43:23.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3f37d34:::benchmark_data_smithi188_141483_object79327:head# 2024-04-23T16:43:23.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3f3bdb2:::benchmark_data_smithi188_141483_object62033:head# 2024-04-23T16:43:23.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3f4953b:::benchmark_data_smithi188_141483_object84006:head# 2024-04-23T16:43:23.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3f50e8b:::benchmark_data_smithi188_141483_object67579:head# 2024-04-23T16:43:23.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3f51e22:::benchmark_data_smithi188_141483_object37064:head# 2024-04-23T16:43:23.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3f5b7a5:::benchmark_data_smithi188_141483_object47110:head# 2024-04-23T16:43:23.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3f5decc:::benchmark_data_smithi188_141483_object82248:head# 2024-04-23T16:43:23.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3f6d520:::benchmark_data_smithi188_141483_object70868:head# 2024-04-23T16:43:23.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3f87664:::benchmark_data_smithi188_141483_object14849:head# 2024-04-23T16:43:23.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3f8cd49:::benchmark_data_smithi188_141483_object29545:head# 2024-04-23T16:43:23.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3f974f2:::benchmark_data_smithi188_141483_object85067:head# 2024-04-23T16:43:23.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3f99797:::benchmark_data_smithi188_141483_object84818:head# 2024-04-23T16:43:23.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3fbbfbd:::benchmark_data_smithi188_141483_object25769:head# 2024-04-23T16:43:23.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3fbf624:::benchmark_data_smithi188_141483_object50976:head# 2024-04-23T16:43:23.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3fce587:::benchmark_data_smithi188_141483_object5178:head# 2024-04-23T16:43:23.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3ff1302:::benchmark_data_smithi188_141483_object64773:head# 2024-04-23T16:43:23.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3ff62a2:::benchmark_data_smithi188_141483_object83349:head# 2024-04-23T16:43:23.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c3ffe7c7:::benchmark_data_smithi188_141483_object46397:head# 2024-04-23T16:43:23.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4028b5f:::benchmark_data_smithi188_141483_object71337:head# 2024-04-23T16:43:23.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c402a715:::benchmark_data_smithi188_141483_object73885:head# 2024-04-23T16:43:23.326 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4035379:::benchmark_data_smithi188_141483_object78598:head# 2024-04-23T16:43:23.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c403f6ce:::benchmark_data_smithi188_141483_object43186:head# 2024-04-23T16:43:23.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4049ba0:::benchmark_data_smithi188_141483_object85140:head# 2024-04-23T16:43:23.328 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c404f763:::benchmark_data_smithi188_141483_object52091:head# 2024-04-23T16:43:23.328 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c40568e3:::benchmark_data_smithi188_141483_object74647:head# 2024-04-23T16:43:23.329 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c405c0cd:::benchmark_data_smithi188_141483_object6303:head# 2024-04-23T16:43:23.329 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4073f75:::benchmark_data_smithi188_141483_object56069:head# 2024-04-23T16:43:23.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4076182:::benchmark_data_smithi188_141483_object70382:head# 2024-04-23T16:43:23.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c40764d6:::benchmark_data_smithi188_141483_object86907:head# 2024-04-23T16:43:23.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4088bc5:::benchmark_data_smithi188_141483_object52052:head# 2024-04-23T16:43:23.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4096754:::benchmark_data_smithi188_141483_object73078:head# 2024-04-23T16:43:23.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c409a246:::benchmark_data_smithi188_141483_object89634:head# 2024-04-23T16:43:23.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c40adb85:::benchmark_data_smithi188_141483_object24919:head# 2024-04-23T16:43:23.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c40aecf1:::benchmark_data_smithi188_141483_object19839:head# 2024-04-23T16:43:23.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c40c2d53:::benchmark_data_smithi188_141483_object33:head# 2024-04-23T16:43:23.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c40cb868:::benchmark_data_smithi188_141483_object59182:head# 2024-04-23T16:43:23.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c40cbe7b:::benchmark_data_smithi188_141483_object84063:head# 2024-04-23T16:43:23.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c40e346b:::benchmark_data_smithi188_141483_object59020:head# 2024-04-23T16:43:23.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c40e493f:::benchmark_data_smithi188_141483_object42912:head# 2024-04-23T16:43:23.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4103773:::benchmark_data_smithi188_141483_object43749:head# 2024-04-23T16:43:23.337 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c410747e:::benchmark_data_smithi188_141483_object16191:head# 2024-04-23T16:43:23.337 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4107e5d:::benchmark_data_smithi188_141483_object5113:head# 2024-04-23T16:43:23.338 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c410e771:::benchmark_data_smithi188_141483_object20172:head# 2024-04-23T16:43:23.338 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4111abd:::benchmark_data_smithi188_141483_object30473:head# 2024-04-23T16:43:23.338 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4113bce:::benchmark_data_smithi188_141483_object23284:head# 2024-04-23T16:43:23.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4116987:::benchmark_data_smithi188_141483_object33775:head# 2024-04-23T16:43:23.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c412de6c:::benchmark_data_smithi188_141483_object14795:head# 2024-04-23T16:43:23.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c41334b6:::benchmark_data_smithi188_141483_object62756:head# 2024-04-23T16:43:23.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c413c315:::benchmark_data_smithi188_141483_object18641:head# 2024-04-23T16:43:23.341 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4141340:::benchmark_data_smithi188_141483_object48403:head# 2024-04-23T16:43:23.342 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4143f09:::benchmark_data_smithi188_141483_object32355:head# 2024-04-23T16:43:23.342 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c416c5aa:::benchmark_data_smithi188_141483_object39466:head# 2024-04-23T16:43:23.342 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4199e95:::benchmark_data_smithi188_141483_object61438:head# 2024-04-23T16:43:23.342 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c41a6c08:::benchmark_data_smithi188_141483_object48639:head# 2024-04-23T16:43:23.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c41aa74e:::benchmark_data_smithi188_141483_object49641:head# 2024-04-23T16:43:23.343 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c41b9397:::benchmark_data_smithi188_141483_object26854:head# 2024-04-23T16:43:23.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c41be3ba:::benchmark_data_smithi188_141483_object65322:head# 2024-04-23T16:43:23.344 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c41c8992:::benchmark_data_smithi188_141483_object21366:head# 2024-04-23T16:43:23.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c41f2248:::benchmark_data_smithi188_141483_object22739:head# 2024-04-23T16:43:23.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c420ac75:::benchmark_data_smithi188_141483_object69812:head# 2024-04-23T16:43:23.345 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c42115cb:::benchmark_data_smithi188_141483_object77679:head# 2024-04-23T16:43:23.346 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4220126:::benchmark_data_smithi188_141483_object28239:head# 2024-04-23T16:43:23.346 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c42528d1:::benchmark_data_smithi188_141483_object83177:head# 2024-04-23T16:43:23.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4259e3d:::benchmark_data_smithi188_141483_object90226:head# 2024-04-23T16:43:23.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c426c8c2:::benchmark_data_smithi188_141483_object23282:head# 2024-04-23T16:43:23.348 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4273465:::benchmark_data_smithi188_141483_object32783:head# 2024-04-23T16:43:23.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c427505c:::benchmark_data_smithi188_141483_object73208:head# 2024-04-23T16:43:23.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4277787:::benchmark_data_smithi188_141483_object39788:head# 2024-04-23T16:43:23.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c42824a6:::benchmark_data_smithi188_141483_object62588:head# 2024-04-23T16:43:23.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4287170:::benchmark_data_smithi188_141483_object1486:head# 2024-04-23T16:43:23.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c42a3b6f:::benchmark_data_smithi188_141483_object35610:head# 2024-04-23T16:43:23.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c42a3f7f:::benchmark_data_smithi188_141483_object43651:head# 2024-04-23T16:43:23.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c42a9d69:::benchmark_data_smithi188_141483_object53171:head# 2024-04-23T16:43:23.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c42b3c11:::benchmark_data_smithi188_141483_object9033:head# 2024-04-23T16:43:23.353 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c42badaa:::benchmark_data_smithi188_141483_object7930:head# 2024-04-23T16:43:23.353 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c42c96fe:::benchmark_data_smithi188_141483_object47062:head# 2024-04-23T16:43:23.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c42cc22e:::benchmark_data_smithi188_141483_object39558:head# 2024-04-23T16:43:23.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c42d0cc7:::benchmark_data_smithi188_141483_object50892:head# 2024-04-23T16:43:23.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c42e0cdb:::benchmark_data_smithi188_141483_object35753:head# 2024-04-23T16:43:23.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c42ec403:::benchmark_data_smithi188_141483_object14019:head# 2024-04-23T16:43:23.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c42f47d5:::benchmark_data_smithi188_141483_object16112:head# 2024-04-23T16:43:23.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c430966d:::benchmark_data_smithi188_141483_object71627:head# 2024-04-23T16:43:23.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c430f677:::benchmark_data_smithi188_141483_object28148:head# 2024-04-23T16:43:23.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4321541:::benchmark_data_smithi188_141483_object72021:head# 2024-04-23T16:43:23.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4323c1a:::benchmark_data_smithi188_141483_object88620:head# 2024-04-23T16:43:23.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c433e10f:::benchmark_data_smithi188_141483_object79469:head# 2024-04-23T16:43:23.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c435875f:::benchmark_data_smithi188_141483_object48770:head# 2024-04-23T16:43:23.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c435de49:::benchmark_data_smithi188_141483_object62939:head# 2024-04-23T16:43:23.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c437512e:::benchmark_data_smithi188_141483_object2425:head# 2024-04-23T16:43:23.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c43aa704:::benchmark_data_smithi188_141483_object34226:head# 2024-04-23T16:43:23.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c43b1956:::benchmark_data_smithi188_141483_object78052:head# 2024-04-23T16:43:23.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c43c5e0a:::benchmark_data_smithi188_141483_object24463:head# 2024-04-23T16:43:23.361 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c43ce775:::benchmark_data_smithi188_141483_object74697:head# 2024-04-23T16:43:23.361 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c43d0293:::benchmark_data_smithi188_141483_object42459:head# 2024-04-23T16:43:23.362 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c43e5ccd:::benchmark_data_smithi188_141483_object44780:head# 2024-04-23T16:43:23.362 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c442849c:::benchmark_data_smithi188_141483_object44858:head# 2024-04-23T16:43:23.362 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c442d022:::benchmark_data_smithi188_141483_object87829:head# 2024-04-23T16:43:23.363 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c443427f:::benchmark_data_smithi188_141483_object52362:head# 2024-04-23T16:43:23.363 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c44431ac:::benchmark_data_smithi188_141483_object46803:head# 2024-04-23T16:43:23.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c444c7ee:::benchmark_data_smithi188_141483_object22983:head# 2024-04-23T16:43:23.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c444f055:::benchmark_data_smithi188_141483_object23031:head# 2024-04-23T16:43:23.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4456738:::benchmark_data_smithi188_141483_object52826:head# 2024-04-23T16:43:23.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c445caae:::benchmark_data_smithi188_141483_object36346:head# 2024-04-23T16:43:23.366 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c445dacd:::benchmark_data_smithi188_141483_object53700:head# 2024-04-23T16:43:23.366 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c445e975:::benchmark_data_smithi188_141483_object3894:head# 2024-04-23T16:43:23.367 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c44670f4:::benchmark_data_smithi188_141483_object90371:head# 2024-04-23T16:43:23.367 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c446bfeb:::benchmark_data_smithi188_141483_object56183:head# 2024-04-23T16:43:23.367 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c446ea4e:::benchmark_data_smithi188_141483_object76370:head# 2024-04-23T16:43:23.368 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4470821:::benchmark_data_smithi188_141483_object22884:head# 2024-04-23T16:43:23.368 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4470e1d:::benchmark_data_smithi188_141483_object7898:head# 2024-04-23T16:43:23.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4477323:::benchmark_data_smithi188_141483_object9025:head# 2024-04-23T16:43:23.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4493b96:::benchmark_data_smithi188_141483_object58656:head# 2024-04-23T16:43:23.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4496681:::benchmark_data_smithi188_141483_object65729:head# 2024-04-23T16:43:23.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c44972e6:::benchmark_data_smithi188_141483_object70872:head# 2024-04-23T16:43:23.371 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c44982ed:::benchmark_data_smithi188_141483_object49230:head# 2024-04-23T16:43:23.371 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c449cf76:::benchmark_data_smithi188_141483_object63722:head# 2024-04-23T16:43:23.372 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c449d87f:::benchmark_data_smithi188_141483_object4562:head# 2024-04-23T16:43:23.372 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c44a80f8:::benchmark_data_smithi188_141483_object36060:head# 2024-04-23T16:43:23.372 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c44c2276:::benchmark_data_smithi188_141483_object51864:head# 2024-04-23T16:43:23.373 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c44c974d:::benchmark_data_smithi188_141483_object2003:head# 2024-04-23T16:43:23.373 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c44cb7f4:::benchmark_data_smithi188_141483_object43442:head# 2024-04-23T16:43:23.373 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c44e2e97:::benchmark_data_smithi188_141483_object75204:head# 2024-04-23T16:43:23.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c44e595a:::benchmark_data_smithi188_141483_object10331:head# 2024-04-23T16:43:23.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c44f07c2:::benchmark_data_smithi188_141483_object86319:head# 2024-04-23T16:43:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c44fedae:::benchmark_data_smithi188_141483_object5867:head# 2024-04-23T16:43:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c450341d:::benchmark_data_smithi188_141483_object12300:head# 2024-04-23T16:43:23.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4504bcc:::benchmark_data_smithi188_141483_object20587:head# 2024-04-23T16:43:23.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c450db4b:::benchmark_data_smithi188_141483_object43248:head# 2024-04-23T16:43:23.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c450efdf:::benchmark_data_smithi188_141483_object75292:head# 2024-04-23T16:43:23.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c451040c:::benchmark_data_smithi188_141483_object74573:head# 2024-04-23T16:43:23.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4511361:::benchmark_data_smithi188_141483_object37952:head# 2024-04-23T16:43:23.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4516aaa:::benchmark_data_smithi188_141483_object47595:head# 2024-04-23T16:43:23.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4518f3a:::benchmark_data_smithi188_141483_object71508:head# 2024-04-23T16:43:23.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c451d39b:::benchmark_data_smithi188_141483_object62698:head# 2024-04-23T16:43:23.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c451f213:::benchmark_data_smithi188_141483_object27152:head# 2024-04-23T16:43:23.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4520542:::benchmark_data_smithi188_141483_object26839:head# 2024-04-23T16:43:23.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4532961:::benchmark_data_smithi188_141483_object75783:head# 2024-04-23T16:43:23.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c454eb04:::benchmark_data_smithi188_141483_object86175:head# 2024-04-23T16:43:23.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c457360d:::benchmark_data_smithi188_141483_object37851:head# 2024-04-23T16:43:23.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c457cd38:::benchmark_data_smithi188_141483_object17219:head# 2024-04-23T16:43:23.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c45985da:::benchmark_data_smithi188_141483_object21539:head# 2024-04-23T16:43:23.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c459c2ff:::benchmark_data_smithi188_141483_object41536:head# 2024-04-23T16:43:23.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c459cba7:::benchmark_data_smithi188_141483_object39421:head# 2024-04-23T16:43:23.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c45a8136:::benchmark_data_smithi188_141483_object84268:head# 2024-04-23T16:43:23.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c45b572e:::benchmark_data_smithi188_141483_object12441:head# 2024-04-23T16:43:23.384 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c45c22ea:::benchmark_data_smithi188_141483_object47682:head# 2024-04-23T16:43:23.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c45c9258:::benchmark_data_smithi188_141483_object31760:head# 2024-04-23T16:43:23.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c45cb3c1:::benchmark_data_smithi188_141483_object13168:head# 2024-04-23T16:43:23.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c45d1eff:::benchmark_data_smithi188_141483_object27652:head# 2024-04-23T16:43:23.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c45d79a9:::benchmark_data_smithi188_141483_object35049:head# 2024-04-23T16:43:23.387 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c45f0c69:::benchmark_data_smithi188_141483_object45206:head# 2024-04-23T16:43:23.387 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c45f102e:::benchmark_data_smithi188_141483_object82850:head# 2024-04-23T16:43:23.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c45f494c:::benchmark_data_smithi188_141483_object65223:head# 2024-04-23T16:43:23.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c45fc2a1:::benchmark_data_smithi188_141483_object56631:head# 2024-04-23T16:43:23.389 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c460e95b:::benchmark_data_smithi188_141483_object163:head# 2024-04-23T16:43:23.389 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4618290:::benchmark_data_smithi188_141483_object19056:head# 2024-04-23T16:43:23.389 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c461d98f:::benchmark_data_smithi188_141483_object3844:head# 2024-04-23T16:43:23.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c46214f7:::benchmark_data_smithi188_141483_object74105:head# 2024-04-23T16:43:23.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4628e64:::benchmark_data_smithi188_141483_object676:head# 2024-04-23T16:43:23.391 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c463a0d3:::benchmark_data_smithi188_141483_object67205:head# 2024-04-23T16:43:23.391 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c463e4d1:::benchmark_data_smithi188_141483_object51798:head# 2024-04-23T16:43:23.391 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4644306:::benchmark_data_smithi188_141483_object67491:head# 2024-04-23T16:43:23.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4646ccc:::benchmark_data_smithi188_141483_object74825:head# 2024-04-23T16:43:23.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4654464:::benchmark_data_smithi188_141483_object59440:head# 2024-04-23T16:43:23.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c46552c7:::benchmark_data_smithi188_141483_object30701:head# 2024-04-23T16:43:23.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4658f41:::benchmark_data_smithi188_141483_object40897:head# 2024-04-23T16:43:23.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c465b4aa:::benchmark_data_smithi188_141483_object18131:head# 2024-04-23T16:43:23.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c465bbe7:::benchmark_data_smithi188_141483_object39030:head# 2024-04-23T16:43:23.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4666f1b:::benchmark_data_smithi188_141483_object49912:head# 2024-04-23T16:43:23.395 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4690d32:::benchmark_data_smithi188_141483_object51180:head# 2024-04-23T16:43:23.395 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c469474e:::benchmark_data_smithi188_141483_object58913:head# 2024-04-23T16:43:23.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4694d59:::benchmark_data_smithi188_141483_object14347:head# 2024-04-23T16:43:23.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c469c2f3:::benchmark_data_smithi188_141483_object50461:head# 2024-04-23T16:43:23.397 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c46a4c62:::benchmark_data_smithi188_141483_object3986:head# 2024-04-23T16:43:23.397 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c46a68a9:::benchmark_data_smithi188_141483_object55551:head# 2024-04-23T16:43:23.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c46adcb1:::benchmark_data_smithi188_141483_object10700:head# 2024-04-23T16:43:23.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c46b9a5f:::benchmark_data_smithi188_141483_object37688:head# 2024-04-23T16:43:23.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c46bfd4e:::benchmark_data_smithi188_141483_object58113:head# 2024-04-23T16:43:23.399 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c46cf085:::benchmark_data_smithi188_141483_object53594:head# 2024-04-23T16:43:23.399 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c46f4ec4:::benchmark_data_smithi188_141483_object2515:head# 2024-04-23T16:43:23.399 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c46fb2ff:::benchmark_data_smithi188_141483_object60462:head# 2024-04-23T16:43:23.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c470f162:::benchmark_data_smithi188_141483_object9716:head# 2024-04-23T16:43:23.400 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c471e5fa:::benchmark_data_smithi188_141483_object42411:head# 2024-04-23T16:43:23.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4724e77:::benchmark_data_smithi188_141483_object86282:head# 2024-04-23T16:43:23.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c472a3eb:::benchmark_data_smithi188_141483_object40005:head# 2024-04-23T16:43:23.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c472dbdf:::benchmark_data_smithi188_141483_object70127:head# 2024-04-23T16:43:23.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4730252:::benchmark_data_smithi188_141483_object68033:head# 2024-04-23T16:43:23.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4733b60:::benchmark_data_smithi188_141483_object49632:head# 2024-04-23T16:43:23.403 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4736e58:::benchmark_data_smithi188_141483_object5286:head# 2024-04-23T16:43:23.403 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c473ae75:::benchmark_data_smithi188_141483_object72460:head# 2024-04-23T16:43:23.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c476cdb4:::benchmark_data_smithi188_141483_object7486:head# 2024-04-23T16:43:23.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4777f5c:::benchmark_data_smithi188_141483_object15433:head# 2024-04-23T16:43:23.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4783b3a:::benchmark_data_smithi188_141483_object45595:head# 2024-04-23T16:43:23.405 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c47b6c3e:::benchmark_data_smithi188_141483_object68646:head# 2024-04-23T16:43:23.405 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c47ba9ae:::benchmark_data_smithi188_141483_object77698:head# 2024-04-23T16:43:23.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c47cb59e:::benchmark_data_smithi188_141483_object3132:head# 2024-04-23T16:43:23.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c47cebc9:::benchmark_data_smithi188_141483_object84580:head# 2024-04-23T16:43:23.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c47d2a7b:::benchmark_data_smithi188_141483_object3580:head# 2024-04-23T16:43:23.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4800ccf:::benchmark_data_smithi188_141483_object37741:head# 2024-04-23T16:43:23.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c480694f:::benchmark_data_smithi188_141483_object49874:head# 2024-04-23T16:43:23.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4808545:::benchmark_data_smithi188_141483_object59798:head# 2024-04-23T16:43:23.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c481ccb4:::benchmark_data_smithi188_141483_object41267:head# 2024-04-23T16:43:23.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c48237c4:::benchmark_data_smithi188_141483_object75940:head# 2024-04-23T16:43:23.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c482427c:::benchmark_data_smithi188_141483_object16216:head# 2024-04-23T16:43:23.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c48272ed:::benchmark_data_smithi188_141483_object49816:head# 2024-04-23T16:43:23.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4833b22:::benchmark_data_smithi188_141483_object7747:head# 2024-04-23T16:43:23.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c484078f:::benchmark_data_smithi188_141483_object49973:head# 2024-04-23T16:43:23.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4864de2:::benchmark_data_smithi188_141483_object41747:head# 2024-04-23T16:43:23.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c487017d:::benchmark_data_smithi188_141483_object27475:head# 2024-04-23T16:43:23.412 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4877096:::benchmark_data_smithi188_141483_object56857:head# 2024-04-23T16:43:23.412 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c48772f0:::benchmark_data_smithi188_141483_object63886:head# 2024-04-23T16:43:23.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4888e53:::benchmark_data_smithi188_141483_object44703:head# 2024-04-23T16:43:23.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c48909df:::benchmark_data_smithi188_141483_object17271:head# 2024-04-23T16:43:23.414 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c489e201:::benchmark_data_smithi188_141483_object78136:head# 2024-04-23T16:43:23.414 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c48a9eef:::benchmark_data_smithi188_141483_object2293:head# 2024-04-23T16:43:23.414 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c48b0097:::benchmark_data_smithi188_141483_object90061:head# 2024-04-23T16:43:23.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c48b8815:::benchmark_data_smithi188_141483_object22706:head# 2024-04-23T16:43:23.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c48b981d:::benchmark_data_smithi188_141483_object11212:head# 2024-04-23T16:43:23.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c48bb68e:::benchmark_data_smithi188_141483_object86013:head# 2024-04-23T16:43:23.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c48bef0e:::benchmark_data_smithi188_141483_object22635:head# 2024-04-23T16:43:23.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c48d7759:::benchmark_data_smithi188_141483_object8495:head# 2024-04-23T16:43:23.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c48d9079:::benchmark_data_smithi188_141483_object25516:head# 2024-04-23T16:43:23.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c48e5ecd:::benchmark_data_smithi188_141483_object66172:head# 2024-04-23T16:43:23.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c48e98a2:::benchmark_data_smithi188_141483_object63069:head# 2024-04-23T16:43:23.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c48eb084:::benchmark_data_smithi188_141483_object77424:head# 2024-04-23T16:43:23.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c490b491:::benchmark_data_smithi188_141483_object33272:head# 2024-04-23T16:43:23.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c490be84:::benchmark_data_smithi188_141483_object9915:head# 2024-04-23T16:43:23.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c49178f2:::benchmark_data_smithi188_141483_object6774:head# 2024-04-23T16:43:23.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4925e8d:::benchmark_data_smithi188_141483_object57834:head# 2024-04-23T16:43:23.421 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4939af7:::benchmark_data_smithi188_141483_object69293:head# 2024-04-23T16:43:23.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4940373:::benchmark_data_smithi188_141483_object2796:head# 2024-04-23T16:43:23.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c49497e5:::benchmark_data_smithi188_141483_object82812:head# 2024-04-23T16:43:23.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c495218b:::benchmark_data_smithi188_141483_object46801:head# 2024-04-23T16:43:23.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c495a953:::benchmark_data_smithi188_141483_object29147:head# 2024-04-23T16:43:23.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c495f68f:::benchmark_data_smithi188_141483_object49047:head# 2024-04-23T16:43:23.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c49b13a8:::benchmark_data_smithi188_141483_object50918:head# 2024-04-23T16:43:23.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c49fcc74:::benchmark_data_smithi188_141483_object15200:head# 2024-04-23T16:43:23.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4a06083:::benchmark_data_smithi188_141483_object3170:head# 2024-04-23T16:43:23.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4a0c21e:::benchmark_data_smithi188_141483_object66912:head# 2024-04-23T16:43:23.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4a1db9d:::benchmark_data_smithi188_141483_object63480:head# 2024-04-23T16:43:23.426 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4a210d1:::benchmark_data_smithi188_141483_object82442:head# 2024-04-23T16:43:23.426 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4a2b8e7:::benchmark_data_smithi188_141483_object66837:head# 2024-04-23T16:43:23.426 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4a35d79:::benchmark_data_smithi188_141483_object58576:head# 2024-04-23T16:43:23.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4a46889:::benchmark_data_smithi188_141483_object44399:head# 2024-04-23T16:43:23.427 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4a4f14c:::benchmark_data_smithi188_141483_object15815:head# 2024-04-23T16:43:23.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4a5cb6e:::benchmark_data_smithi188_141483_object72405:head# 2024-04-23T16:43:23.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4a73253:::benchmark_data_smithi188_141483_object8499:head# 2024-04-23T16:43:23.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4a77cb7:::benchmark_data_smithi188_141483_object7724:head# 2024-04-23T16:43:23.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4a82520:::benchmark_data_smithi188_141483_object65486:head# 2024-04-23T16:43:23.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4a8ede7:::benchmark_data_smithi188_141483_object60669:head# 2024-04-23T16:43:23.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4a91610:::benchmark_data_smithi188_141483_object31297:head# 2024-04-23T16:43:23.431 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4a93157:::benchmark_data_smithi188_141483_object39507:head# 2024-04-23T16:43:23.431 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4a932f5:::benchmark_data_smithi188_141483_object27134:head# 2024-04-23T16:43:23.432 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4a9797e:::benchmark_data_smithi188_141483_object77348:head# 2024-04-23T16:43:23.432 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4ab629d:::benchmark_data_smithi188_141483_object87371:head# 2024-04-23T16:43:23.432 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4ad2aed:::benchmark_data_smithi188_141483_object59041:head# 2024-04-23T16:43:23.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4ada67f:::benchmark_data_smithi188_141483_object35717:head# 2024-04-23T16:43:23.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4adfd6c:::benchmark_data_smithi188_141483_object2866:head# 2024-04-23T16:43:23.434 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4af2926:::benchmark_data_smithi188_141483_object41550:head# 2024-04-23T16:43:23.434 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4af3927:::benchmark_data_smithi188_141483_object64726:head# 2024-04-23T16:43:23.435 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4b008cc:::benchmark_data_smithi188_141483_object2125:head# 2024-04-23T16:43:23.435 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4b077a6:::benchmark_data_smithi188_141483_object2036:head# 2024-04-23T16:43:23.435 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4b0d0bf:::benchmark_data_smithi188_141483_object8584:head# 2024-04-23T16:43:23.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4b23034:::benchmark_data_smithi188_141483_object52643:head# 2024-04-23T16:43:23.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4b3b4a6:::benchmark_data_smithi188_141483_object83941:head# 2024-04-23T16:43:23.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4b4bc95:::benchmark_data_smithi188_141483_object80322:head# 2024-04-23T16:43:23.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4b5d8e8:::benchmark_data_smithi188_141483_object18370:head# 2024-04-23T16:43:23.438 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4b66cd4:::benchmark_data_smithi188_141483_object925:head# 2024-04-23T16:43:23.438 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4b6c96b:::benchmark_data_smithi188_141483_object64619:head# 2024-04-23T16:43:23.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4b759ad:::benchmark_data_smithi188_141483_object81723:head# 2024-04-23T16:43:23.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4b88afd:::benchmark_data_smithi188_141483_object43552:head# 2024-04-23T16:43:23.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4b8cb1a:::benchmark_data_smithi188_141483_object54510:head# 2024-04-23T16:43:23.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4b93ab3:::benchmark_data_smithi188_141483_object59083:head# 2024-04-23T16:43:23.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4b9a49b:::benchmark_data_smithi188_141483_object56170:head# 2024-04-23T16:43:23.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4ba4153:::benchmark_data_smithi188_141483_object50024:head# 2024-04-23T16:43:23.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4baf159:::benchmark_data_smithi188_141483_object79393:head# 2024-04-23T16:43:23.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4bb48e9:::benchmark_data_smithi188_141483_object114:head# 2024-04-23T16:43:23.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4bb7d80:::benchmark_data_smithi188_141483_object72904:head# 2024-04-23T16:43:23.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4bc3828:::benchmark_data_smithi188_141483_object86884:head# 2024-04-23T16:43:23.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4bf2bbe:::benchmark_data_smithi188_141483_object41167:head# 2024-04-23T16:43:23.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4bf410a:::benchmark_data_smithi188_141483_object55625:head# 2024-04-23T16:43:23.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4bf8ecb:::benchmark_data_smithi188_141483_object43133:head# 2024-04-23T16:43:23.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4c12bba:::benchmark_data_smithi188_141483_object72687:head# 2024-04-23T16:43:23.446 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4c25208:::benchmark_data_smithi188_141483_object57398:head# 2024-04-23T16:43:23.446 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4c353e5:::benchmark_data_smithi188_141483_object31619:head# 2024-04-23T16:43:23.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4c372d1:::benchmark_data_smithi188_141483_object39518:head# 2024-04-23T16:43:23.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4c38390:::benchmark_data_smithi188_141483_object43099:head# 2024-04-23T16:43:23.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4c4b651:::benchmark_data_smithi188_141483_object41219:head# 2024-04-23T16:43:23.448 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4c5b272:::benchmark_data_smithi188_141483_object44456:head# 2024-04-23T16:43:23.448 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4c5b33b:::benchmark_data_smithi188_141483_object58483:head# 2024-04-23T16:43:23.449 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4c62f53:::benchmark_data_smithi188_141483_object18682:head# 2024-04-23T16:43:23.449 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4c6a5fe:::benchmark_data_smithi188_141483_object49544:head# 2024-04-23T16:43:23.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4c78ad2:::benchmark_data_smithi188_141483_object81264:head# 2024-04-23T16:43:23.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4c7e7ed:::benchmark_data_smithi188_141483_object82414:head# 2024-04-23T16:43:23.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4c91aa8:::benchmark_data_smithi188_141483_object50423:head# 2024-04-23T16:43:23.451 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4c92c59:::benchmark_data_smithi188_141483_object30497:head# 2024-04-23T16:43:23.451 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4c9bbb3:::benchmark_data_smithi188_141483_object40319:head# 2024-04-23T16:43:23.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4c9bd0c:::benchmark_data_smithi188_141483_object90632:head# 2024-04-23T16:43:23.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4ca8442:::benchmark_data_smithi188_141483_object53326:head# 2024-04-23T16:43:23.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4cb3117:::benchmark_data_smithi188_141483_object75298:head# 2024-04-23T16:43:23.454 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4cc694e:::benchmark_data_smithi188_141483_object28089:head# 2024-04-23T16:43:23.454 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4ccc781:::benchmark_data_smithi188_141483_object10000:head# 2024-04-23T16:43:23.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4cd85ca:::benchmark_data_smithi188_141483_object45173:head# 2024-04-23T16:43:23.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4ce04b8:::benchmark_data_smithi188_141483_object71274:head# 2024-04-23T16:43:23.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4ce4a74:::benchmark_data_smithi188_141483_object19558:head# 2024-04-23T16:43:23.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4cf0c41:::benchmark_data_smithi188_141483_object54994:head# 2024-04-23T16:43:23.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4cf2874:::benchmark_data_smithi188_141483_object46775:head# 2024-04-23T16:43:23.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d01b0b:::benchmark_data_smithi188_141483_object81424:head# 2024-04-23T16:43:23.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d02191:::benchmark_data_smithi188_141483_object83362:head# 2024-04-23T16:43:23.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d0b0d5:::benchmark_data_smithi188_141483_object73181:head# 2024-04-23T16:43:23.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d0bc5b:::benchmark_data_smithi188_141483_object3296:head# 2024-04-23T16:43:23.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d0e72f:::benchmark_data_smithi188_141483_object40904:head# 2024-04-23T16:43:23.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d10df1:::benchmark_data_smithi188_141483_object59993:head# 2024-04-23T16:43:23.460 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d20b43:::benchmark_data_smithi188_141483_object17408:head# 2024-04-23T16:43:23.460 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d24615:::benchmark_data_smithi188_141483_object17923:head# 2024-04-23T16:43:23.461 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d2ef00:::benchmark_data_smithi188_141483_object86865:head# 2024-04-23T16:43:23.461 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d32f3c:::benchmark_data_smithi188_141483_object77207:head# 2024-04-23T16:43:23.462 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d35d91:::benchmark_data_smithi188_141483_object32639:head# 2024-04-23T16:43:23.462 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d4c022:::benchmark_data_smithi188_141483_object32925:head# 2024-04-23T16:43:23.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d4c047:::benchmark_data_smithi188_141483_object68822:head# 2024-04-23T16:43:23.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d55816:::benchmark_data_smithi188_141483_object35904:head# 2024-04-23T16:43:23.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d5ce8d:::benchmark_data_smithi188_141483_object2330:head# 2024-04-23T16:43:23.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d610ed:::benchmark_data_smithi188_141483_object20744:head# 2024-04-23T16:43:23.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d7524c:::benchmark_data_smithi188_141483_object37016:head# 2024-04-23T16:43:23.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d87b9d:::benchmark_data_smithi188_141483_object14097:head# 2024-04-23T16:43:23.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d8aaff:::benchmark_data_smithi188_141483_object24425:head# 2024-04-23T16:43:23.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4d8f61d:::benchmark_data_smithi188_141483_object71351:head# 2024-04-23T16:43:23.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4da2d7d:::benchmark_data_smithi188_141483_object42879:head# 2024-04-23T16:43:23.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4da9092:::benchmark_data_smithi188_141483_object21523:head# 2024-04-23T16:43:23.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4db6338:::benchmark_data_smithi188_141483_object71706:head# 2024-04-23T16:43:23.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4dbd9dc:::benchmark_data_smithi188_141483_object75081:head# 2024-04-23T16:43:23.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4dc4a9d:::benchmark_data_smithi188_141483_object58194:head# 2024-04-23T16:43:23.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4df837a:::benchmark_data_smithi188_141483_object56994:head# 2024-04-23T16:43:23.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4dfa4f6:::benchmark_data_smithi188_141483_object36969:head# 2024-04-23T16:43:23.469 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4e030c5:::benchmark_data_smithi188_141483_object50222:head# 2024-04-23T16:43:23.469 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4e048de:::benchmark_data_smithi188_141483_object40327:head# 2024-04-23T16:43:23.470 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4e05b24:::benchmark_data_smithi188_141483_object65695:head# 2024-04-23T16:43:23.470 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4e09ace:::benchmark_data_smithi188_141483_object85897:head# 2024-04-23T16:43:23.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4e5e41d:::benchmark_data_smithi188_141483_object64498:head# 2024-04-23T16:43:23.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4e5e7ed:::benchmark_data_smithi188_141483_object53930:head# 2024-04-23T16:43:23.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4e63843:::benchmark_data_smithi188_141483_object41277:head# 2024-04-23T16:43:23.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4e69416:::benchmark_data_smithi188_141483_object84867:head# 2024-04-23T16:43:23.473 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4e697e0:::benchmark_data_smithi188_141483_object66056:head# 2024-04-23T16:43:23.473 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4e6d867:::benchmark_data_smithi188_141483_object29288:head# 2024-04-23T16:43:23.474 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4e8b823:::benchmark_data_smithi188_141483_object6998:head# 2024-04-23T16:43:23.474 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4e903f5:::benchmark_data_smithi188_141483_object63890:head# 2024-04-23T16:43:23.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4eb6ad8:::benchmark_data_smithi188_141483_object46093:head# 2024-04-23T16:43:23.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4ecb9a1:::benchmark_data_smithi188_141483_object64948:head# 2024-04-23T16:43:23.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4ed48f5:::benchmark_data_smithi188_141483_object18870:head# 2024-04-23T16:43:23.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4ee1bbe:::benchmark_data_smithi188_141483_object69037:head# 2024-04-23T16:43:23.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4eef215:::benchmark_data_smithi188_141483_object68687:head# 2024-04-23T16:43:23.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4ef04f0:::benchmark_data_smithi188_141483_object89718:head# 2024-04-23T16:43:23.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4efbcda:::benchmark_data_smithi188_141483_object40878:head# 2024-04-23T16:43:23.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f00c6a:::benchmark_data_smithi188_141483_object19813:head# 2024-04-23T16:43:23.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f01fc6:::benchmark_data_smithi188_141483_object42152:head# 2024-04-23T16:43:23.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f08789:::benchmark_data_smithi188_141483_object87566:head# 2024-04-23T16:43:23.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f0b339:::benchmark_data_smithi188_141483_object81328:head# 2024-04-23T16:43:23.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f10476:::benchmark_data_smithi188_141483_object64781:head# 2024-04-23T16:43:23.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f22da6:::benchmark_data_smithi188_141483_object78552:head# 2024-04-23T16:43:23.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f2a64b:::benchmark_data_smithi188_141483_object44761:head# 2024-04-23T16:43:23.481 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f2d43e:::benchmark_data_smithi188_141483_object71673:head# 2024-04-23T16:43:23.481 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f44dde:::benchmark_data_smithi188_141483_object86940:head# 2024-04-23T16:43:23.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f4b352:::benchmark_data_smithi188_141483_object44463:head# 2024-04-23T16:43:23.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f4d2e1:::benchmark_data_smithi188_141483_object32691:head# 2024-04-23T16:43:23.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f4eb67:::benchmark_data_smithi188_141483_object41453:head# 2024-04-23T16:43:23.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f5082d:::benchmark_data_smithi188_141483_object22224:head# 2024-04-23T16:43:23.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f56fbe:::benchmark_data_smithi188_141483_object13996:head# 2024-04-23T16:43:23.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f5a43a:::benchmark_data_smithi188_141483_object65976:head# 2024-04-23T16:43:23.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f5ced9:::benchmark_data_smithi188_141483_object34637:head# 2024-04-23T16:43:23.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f5fceb:::benchmark_data_smithi188_141483_object85890:head# 2024-04-23T16:43:23.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f6e2bd:::benchmark_data_smithi188_141483_object60725:head# 2024-04-23T16:43:23.486 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f7a54d:::benchmark_data_smithi188_141483_object73066:head# 2024-04-23T16:43:23.486 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f7e8f9:::benchmark_data_smithi188_141483_object40235:head# 2024-04-23T16:43:23.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f7ffcc:::benchmark_data_smithi188_141483_object79608:head# 2024-04-23T16:43:23.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4f910f9:::benchmark_data_smithi188_141483_object54663:head# 2024-04-23T16:43:23.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4fa169c:::benchmark_data_smithi188_141483_object5631:head# 2024-04-23T16:43:23.488 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4fa3b5d:::benchmark_data_smithi188_141483_object23112:head# 2024-04-23T16:43:23.488 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4fa6f53:::benchmark_data_smithi188_141483_object58328:head# 2024-04-23T16:43:23.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4fb93b6:::benchmark_data_smithi188_141483_object87918:head# 2024-04-23T16:43:23.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4fd86e4:::benchmark_data_smithi188_141483_object55679:head# 2024-04-23T16:43:23.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4fe6762:::benchmark_data_smithi188_141483_object26589:head# 2024-04-23T16:43:23.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4feb849:::benchmark_data_smithi188_141483_object7768:head# 2024-04-23T16:43:23.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4fef8e3:::benchmark_data_smithi188_141483_object49101:head# 2024-04-23T16:43:23.491 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4ff15c4:::benchmark_data_smithi188_141483_object86385:head# 2024-04-23T16:43:23.492 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c4ff3e6a:::benchmark_data_smithi188_141483_object6491:head# 2024-04-23T16:43:23.492 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c500a853:::benchmark_data_smithi188_141483_object25237:head# 2024-04-23T16:43:23.493 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c500be36:::benchmark_data_smithi188_141483_object9984:head# 2024-04-23T16:43:23.493 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c501261b:::benchmark_data_smithi188_141483_object21838:head# 2024-04-23T16:43:23.494 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5013328:::benchmark_data_smithi188_141483_object26885:head# 2024-04-23T16:43:23.494 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c501d6e1:::benchmark_data_smithi188_141483_object33275:head# 2024-04-23T16:43:23.494 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c501e4fa:::benchmark_data_smithi188_141483_object27477:head# 2024-04-23T16:43:23.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5028934:::benchmark_data_smithi188_141483_object71216:head# 2024-04-23T16:43:23.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c50333fe:::benchmark_data_smithi188_141483_object81298:head# 2024-04-23T16:43:23.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5037437:::benchmark_data_smithi188_141483_object62817:head# 2024-04-23T16:43:23.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5043e6f:::benchmark_data_smithi188_141483_object32977:head# 2024-04-23T16:43:23.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c504a3e3:::benchmark_data_smithi188_141483_object89267:head# 2024-04-23T16:43:23.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5050a69:::benchmark_data_smithi188_141483_object15071:head# 2024-04-23T16:43:23.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5057cf3:::benchmark_data_smithi188_141483_object21419:head# 2024-04-23T16:43:23.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c505f4d3:::benchmark_data_smithi188_141483_object54687:head# 2024-04-23T16:43:23.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5074321:::benchmark_data_smithi188_141483_object77294:head# 2024-04-23T16:43:23.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c50747d6:::benchmark_data_smithi188_141483_object41719:head# 2024-04-23T16:43:23.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c50866d8:::benchmark_data_smithi188_141483_object36491:head# 2024-04-23T16:43:23.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5091071:::benchmark_data_smithi188_141483_object61363:head# 2024-04-23T16:43:23.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c50af078:::benchmark_data_smithi188_141483_object58640:head# 2024-04-23T16:43:23.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c50c38b5:::benchmark_data_smithi188_141483_object77383:head# 2024-04-23T16:43:23.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c50d8f88:::benchmark_data_smithi188_141483_object29747:head# 2024-04-23T16:43:23.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c50e6cfc:::benchmark_data_smithi188_141483_object66154:head# 2024-04-23T16:43:23.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c50e780b:::benchmark_data_smithi188_141483_object37909:head# 2024-04-23T16:43:23.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c50ef5ac:::benchmark_data_smithi188_141483_object10349:head# 2024-04-23T16:43:23.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c50fc0d7:::benchmark_data_smithi188_141483_object29934:head# 2024-04-23T16:43:23.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c51085ad:::benchmark_data_smithi188_141483_object22188:head# 2024-04-23T16:43:23.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c511b3d0:::benchmark_data_smithi188_141483_object42570:head# 2024-04-23T16:43:23.504 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c51218bb:::benchmark_data_smithi188_141483_object49254:head# 2024-04-23T16:43:23.504 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5121e19:::benchmark_data_smithi188_141483_object44014:head# 2024-04-23T16:43:23.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c512473a:::benchmark_data_smithi188_141483_object17770:head# 2024-04-23T16:43:23.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c512e31e:::benchmark_data_smithi188_141483_object48778:head# 2024-04-23T16:43:23.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5137055:::benchmark_data_smithi188_141483_object78008:head# 2024-04-23T16:43:23.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c514098d:::benchmark_data_smithi188_141483_object35405:head# 2024-04-23T16:43:23.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c51467c3:::benchmark_data_smithi188_141483_object73710:head# 2024-04-23T16:43:23.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c514cf34:::benchmark_data_smithi188_141483_object65528:head# 2024-04-23T16:43:23.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c515054b:::benchmark_data_smithi188_141483_object14520:head# 2024-04-23T16:43:23.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5153110:::benchmark_data_smithi188_141483_object24531:head# 2024-04-23T16:43:23.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5172baf:::benchmark_data_smithi188_141483_object30648:head# 2024-04-23T16:43:23.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c519b757:::benchmark_data_smithi188_141483_object13586:head# 2024-04-23T16:43:23.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c51b8efd:::benchmark_data_smithi188_141483_object81149:head# 2024-04-23T16:43:23.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c51bb739:::benchmark_data_smithi188_141483_object22085:head# 2024-04-23T16:43:23.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c51bce8a:::benchmark_data_smithi188_141483_object31494:head# 2024-04-23T16:43:23.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c51d9014:::benchmark_data_smithi188_141483_object73515:head# 2024-04-23T16:43:23.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c51fbfec:::benchmark_data_smithi188_141483_object71474:head# 2024-04-23T16:43:23.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5206288:::benchmark_data_smithi188_141483_object53317:head# 2024-04-23T16:43:23.512 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:43:23.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5219446:::benchmark_data_smithi188_141483_object42738:head# 2024-04-23T16:43:23.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5244f48:::benchmark_data_smithi188_141483_object14670:head# 2024-04-23T16:43:23.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:23.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:43:23.516 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c524bfc4:::benchmark_data_smithi188_141483_object30998:head# 2024-04-23T16:43:23.516 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c524d488:::benchmark_data_smithi188_141483_object10203:head# 2024-04-23T16:43:23.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5257d24:::benchmark_data_smithi188_141483_object52812:head# 2024-04-23T16:43:23.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c52688f1:::benchmark_data_smithi188_141483_object62466:head# 2024-04-23T16:43:23.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c526f215:::benchmark_data_smithi188_141483_object57520:head# 2024-04-23T16:43:23.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c527753b:::benchmark_data_smithi188_141483_object6795:head# 2024-04-23T16:43:23.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5281195:::benchmark_data_smithi188_141483_object51416:head# 2024-04-23T16:43:23.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c529d72f:::benchmark_data_smithi188_141483_object12231:head# 2024-04-23T16:43:23.517 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c529e9c6:::benchmark_data_smithi188_141483_object33615:head# 2024-04-23T16:43:23.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c52be50c:::benchmark_data_smithi188_141483_object36717:head# 2024-04-23T16:43:23.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c52cc463:::benchmark_data_smithi188_141483_object8820:head# 2024-04-23T16:43:23.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c52cf1f1:::benchmark_data_smithi188_141483_object18337:head# 2024-04-23T16:43:23.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c52dc676:::benchmark_data_smithi188_141483_object40911:head# 2024-04-23T16:43:23.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c52df010:::benchmark_data_smithi188_141483_object17456:head# 2024-04-23T16:43:23.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c52e07e7:::benchmark_data_smithi188_141483_object22380:head# 2024-04-23T16:43:23.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c52e4d38:::benchmark_data_smithi188_141483_object36569:head# 2024-04-23T16:43:23.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c52ea147:::benchmark_data_smithi188_141483_object6780:head# 2024-04-23T16:43:23.523 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c52f9038:::benchmark_data_smithi188_141483_object60574:head# 2024-04-23T16:43:23.523 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5301fe2:::benchmark_data_smithi188_141483_object17524:head# 2024-04-23T16:43:23.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c530b048:::benchmark_data_smithi188_141483_object71793:head# 2024-04-23T16:43:23.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5318705:::benchmark_data_smithi188_141483_object67113:head# 2024-04-23T16:43:23.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c532a3ee:::benchmark_data_smithi188_141483_object54058:head# 2024-04-23T16:43:23.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c53444cd:::benchmark_data_smithi188_141483_object36851:head# 2024-04-23T16:43:23.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5358a08:::benchmark_data_smithi188_141483_object12259:head# 2024-04-23T16:43:23.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c535f752:::benchmark_data_smithi188_141483_object24244:head# 2024-04-23T16:43:23.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c536635f:::benchmark_data_smithi188_141483_object16716:head# 2024-04-23T16:43:23.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5385057:::benchmark_data_smithi188_141483_object6078:head# 2024-04-23T16:43:23.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5388965:::benchmark_data_smithi188_141483_object49440:head# 2024-04-23T16:43:23.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c53a2dc8:::benchmark_data_smithi188_141483_object4199:head# 2024-04-23T16:43:23.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c53aae6f:::benchmark_data_smithi188_141483_object24001:head# 2024-04-23T16:43:23.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c53acdf0:::benchmark_data_smithi188_141483_object56121:head# 2024-04-23T16:43:23.531 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c53b129d:::benchmark_data_smithi188_141483_object12655:head# 2024-04-23T16:43:23.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c53b60fb:::benchmark_data_smithi188_141483_object79376:head# 2024-04-23T16:43:23.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c53c108f:::benchmark_data_smithi188_141483_object46938:head# 2024-04-23T16:43:23.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c53c4f85:::benchmark_data_smithi188_141483_object27649:head# 2024-04-23T16:43:23.536 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c53d888b:::benchmark_data_smithi188_141483_object38441:head# 2024-04-23T16:43:23.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c53dc7cd:::benchmark_data_smithi188_141483_object56755:head# 2024-04-23T16:43:23.537 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 72 16 104034 104018 90.281 49.6875 0.0155735 0.0110737 2024-04-23T16:43:23.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c53e6b4a:::benchmark_data_smithi188_141483_object58186:head# 2024-04-23T16:43:23.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c53e747d:::benchmark_data_smithi188_141483_object49851:head# 2024-04-23T16:43:23.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c53eb717:::benchmark_data_smithi188_141483_object73815:head# 2024-04-23T16:43:23.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c53f282e:::benchmark_data_smithi188_141483_object38246:head# 2024-04-23T16:43:23.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c53f8b37:::benchmark_data_smithi188_141483_object36079:head# 2024-04-23T16:43:23.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c53fd94e:::benchmark_data_smithi188_141483_object21346:head# 2024-04-23T16:43:23.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c53fdde7:::benchmark_data_smithi188_141483_object16566:head# 2024-04-23T16:43:23.579 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5400a95:::benchmark_data_smithi188_141483_object48780:head# 2024-04-23T16:43:23.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c540874a:::benchmark_data_smithi188_141483_object61326:head# 2024-04-23T16:43:23.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5410166:::benchmark_data_smithi188_141483_object70760:head# 2024-04-23T16:43:23.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5416e56:::benchmark_data_smithi188_141483_object66527:head# 2024-04-23T16:43:23.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5430959:::benchmark_data_smithi188_141483_object20943:head# 2024-04-23T16:43:23.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5436cdc:::benchmark_data_smithi188_141483_object3988:head# 2024-04-23T16:43:23.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5457626:::benchmark_data_smithi188_141483_object59591:head# 2024-04-23T16:43:23.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5460787:::benchmark_data_smithi188_141483_object37248:head# 2024-04-23T16:43:23.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c546e938:::benchmark_data_smithi188_141483_object34737:head# 2024-04-23T16:43:23.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5482b4b:::benchmark_data_smithi188_141483_object75778:head# 2024-04-23T16:43:23.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5483a2a:::benchmark_data_smithi188_141483_object47520:head# 2024-04-23T16:43:23.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c54a44d0:::benchmark_data_smithi188_141483_object15269:head# 2024-04-23T16:43:23.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c54aa2ef:::benchmark_data_smithi188_141483_object47575:head# 2024-04-23T16:43:23.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c54aceed:::benchmark_data_smithi188_141483_object54321:head# 2024-04-23T16:43:23.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c54c4f38:::benchmark_data_smithi188_141483_object4114:head# 2024-04-23T16:43:23.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c54c5b78:::benchmark_data_smithi188_141483_object61211:head# 2024-04-23T16:43:23.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c54c9dc8:::benchmark_data_smithi188_141483_object28561:head# 2024-04-23T16:43:23.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c54e29a1:::benchmark_data_smithi188_141483_object29594:head# 2024-04-23T16:43:23.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c54ebc9f:::benchmark_data_smithi188_141483_object3057:head# 2024-04-23T16:43:23.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c54fab02:::benchmark_data_smithi188_141483_object72543:head# 2024-04-23T16:43:23.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c550511c:::benchmark_data_smithi188_141483_object5825:head# 2024-04-23T16:43:23.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5515be6:::benchmark_data_smithi188_141483_object25566:head# 2024-04-23T16:43:23.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c551ac90:::benchmark_data_smithi188_141483_object78805:head# 2024-04-23T16:43:23.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c55250b9:::benchmark_data_smithi188_141483_object40053:head# 2024-04-23T16:43:23.585 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c55296ea:::benchmark_data_smithi188_141483_object11127:head# 2024-04-23T16:43:23.585 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c552f02f:::benchmark_data_smithi188_141483_object88976:head# 2024-04-23T16:43:23.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c553f329:::benchmark_data_smithi188_141483_object50724:head# 2024-04-23T16:43:23.586 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c55462b7:::benchmark_data_smithi188_141483_object32713:head# 2024-04-23T16:43:23.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5546730:::benchmark_data_smithi188_141483_object50913:head# 2024-04-23T16:43:23.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5556e80:::benchmark_data_smithi188_141483_object15137:head# 2024-04-23T16:43:23.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5565403:::benchmark_data_smithi188_141483_object67484:head# 2024-04-23T16:43:23.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5569ff4:::benchmark_data_smithi188_141483_object56104:head# 2024-04-23T16:43:23.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c556a403:::benchmark_data_smithi188_141483_object46340:head# 2024-04-23T16:43:23.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c55751a5:::benchmark_data_smithi188_141483_object85409:head# 2024-04-23T16:43:23.592 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c557c78f:::benchmark_data_smithi188_141483_object7748:head# 2024-04-23T16:43:23.593 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5591105:::benchmark_data_smithi188_141483_object60065:head# 2024-04-23T16:43:23.593 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5593ad9:::benchmark_data_smithi188_141483_object21204:head# 2024-04-23T16:43:23.593 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5598413:::benchmark_data_smithi188_141483_object37847:head# 2024-04-23T16:43:23.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c559ef11:::benchmark_data_smithi188_141483_object84093:head# 2024-04-23T16:43:23.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c55aaa94:::benchmark_data_smithi188_141483_object83044:head# 2024-04-23T16:43:23.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c55acc6f:::benchmark_data_smithi188_141483_object34962:head# 2024-04-23T16:43:23.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c55aed46:::benchmark_data_smithi188_141483_object35150:head# 2024-04-23T16:43:23.598 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c55b6cec:::benchmark_data_smithi188_141483_object63503:head# 2024-04-23T16:43:23.598 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c55c9f8a:::benchmark_data_smithi188_141483_object42918:head# 2024-04-23T16:43:23.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c55da707:::benchmark_data_smithi188_141483_object32701:head# 2024-04-23T16:43:23.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c55fc793:::benchmark_data_smithi188_141483_object86505:head# 2024-04-23T16:43:23.599 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c55ff838:::benchmark_data_smithi188_141483_object82253:head# 2024-04-23T16:43:23.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c560164f:::benchmark_data_smithi188_141483_object57882:head# 2024-04-23T16:43:23.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5602abd:::benchmark_data_smithi188_141483_object25062:head# 2024-04-23T16:43:23.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c560c98d:::benchmark_data_smithi188_141483_object2628:head# 2024-04-23T16:43:23.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c561a178:::benchmark_data_smithi188_141483_object52454:head# 2024-04-23T16:43:23.602 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c561d988:::benchmark_data_smithi188_141483_object5900:head# 2024-04-23T16:43:23.602 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c562469a:::benchmark_data_smithi188_141483_object38130:head# 2024-04-23T16:43:23.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c562a3d8:::benchmark_data_smithi188_141483_object50283:head# 2024-04-23T16:43:23.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c564787e:::benchmark_data_smithi188_141483_object28263:head# 2024-04-23T16:43:23.605 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c566b0a3:::benchmark_data_smithi188_141483_object80900:head# 2024-04-23T16:43:23.605 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c56792f6:::benchmark_data_smithi188_141483_object87062:head# 2024-04-23T16:43:23.607 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c56889c9:::benchmark_data_smithi188_141483_object29917:head# 2024-04-23T16:43:23.607 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5699b1e:::benchmark_data_smithi188_141483_object4435:head# 2024-04-23T16:43:23.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c569cb0d:::benchmark_data_smithi188_141483_object80404:head# 2024-04-23T16:43:23.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c56a53eb:::benchmark_data_smithi188_141483_object48775:head# 2024-04-23T16:43:23.609 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c56b0c23:::benchmark_data_smithi188_141483_object47716:head# 2024-04-23T16:43:23.609 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c56d534b:::benchmark_data_smithi188_141483_object89468:head# 2024-04-23T16:43:23.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c56e2705:::benchmark_data_smithi188_141483_object78871:head# 2024-04-23T16:43:23.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c56e2dea:::benchmark_data_smithi188_141483_object27136:head# 2024-04-23T16:43:23.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c571b4fc:::benchmark_data_smithi188_141483_object20825:head# 2024-04-23T16:43:23.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c571b958:::benchmark_data_smithi188_141483_object75645:head# 2024-04-23T16:43:23.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c571d335:::benchmark_data_smithi188_141483_object23728:head# 2024-04-23T16:43:23.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c572985c:::benchmark_data_smithi188_141483_object5209:head# 2024-04-23T16:43:23.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c573314d:::benchmark_data_smithi188_141483_object83399:head# 2024-04-23T16:43:23.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5744e84:::benchmark_data_smithi188_141483_object49259:head# 2024-04-23T16:43:23.616 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:43:23.616 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c574b339:::benchmark_data_smithi188_141483_object5692:head# 2024-04-23T16:43:23.617 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c576375c:::benchmark_data_smithi188_141483_object77434:head# 2024-04-23T16:43:23.617 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5767eeb:::benchmark_data_smithi188_141483_object64894:head# 2024-04-23T16:43:23.617 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5794460:::benchmark_data_smithi188_141483_object86184:head# 2024-04-23T16:43:23.618 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c579b555:::benchmark_data_smithi188_141483_object9636:head# 2024-04-23T16:43:23.618 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c57b367e:::benchmark_data_smithi188_141483_object14274:head# 2024-04-23T16:43:23.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c57b71ae:::benchmark_data_smithi188_141483_object82657:head# 2024-04-23T16:43:23.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c57c9ffd:::benchmark_data_smithi188_141483_object49587:head# 2024-04-23T16:43:23.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c57cb50f:::benchmark_data_smithi188_141483_object55884:head# 2024-04-23T16:43:23.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c57e0ff2:::benchmark_data_smithi188_141483_object22830:head# 2024-04-23T16:43:23.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c57eca76:::benchmark_data_smithi188_141483_object47590:head# 2024-04-23T16:43:23.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c57fdc16:::benchmark_data_smithi188_141483_object7796:head# 2024-04-23T16:43:23.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c57fe4a7:::benchmark_data_smithi188_141483_object20708:head# 2024-04-23T16:43:23.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c57fe748:::benchmark_data_smithi188_141483_object71658:head# 2024-04-23T16:43:23.624 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5810e52:::benchmark_data_smithi188_141483_object26339:head# 2024-04-23T16:43:23.624 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5811008:::benchmark_data_smithi188_141483_object81504:head# 2024-04-23T16:43:23.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5834eda:::benchmark_data_smithi188_141483_object75860:head# 2024-04-23T16:43:23.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5835189:::benchmark_data_smithi188_141483_object64757:head# 2024-04-23T16:43:23.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c583eead:::benchmark_data_smithi188_141483_object13222:head# 2024-04-23T16:43:23.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5870cfd:::benchmark_data_smithi188_141483_object13425:head# 2024-04-23T16:43:23.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c587ba5d:::benchmark_data_smithi188_141483_object88627:head# 2024-04-23T16:43:23.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5897a19:::benchmark_data_smithi188_141483_object89202:head# 2024-04-23T16:43:23.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c58d46c3:::benchmark_data_smithi188_141483_object49240:head# 2024-04-23T16:43:23.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c58deeb3:::benchmark_data_smithi188_141483_object65472:head# 2024-04-23T16:43:23.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c58e1704:::benchmark_data_smithi188_141483_object3330:head# 2024-04-23T16:43:23.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c58e8f56:::benchmark_data_smithi188_141483_object64206:head# 2024-04-23T16:43:23.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c58f78ae:::benchmark_data_smithi188_141483_object15179:head# 2024-04-23T16:43:23.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c59146d8:::benchmark_data_smithi188_141483_object82498:head# 2024-04-23T16:43:23.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5920238:::benchmark_data_smithi188_141483_object38430:head# 2024-04-23T16:43:23.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c592112f:::benchmark_data_smithi188_141483_object208:head# 2024-04-23T16:43:23.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5925f38:::benchmark_data_smithi188_141483_object54983:head# 2024-04-23T16:43:23.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c592733d:::benchmark_data_smithi188_141483_object84707:head# 2024-04-23T16:43:23.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c592a276:::benchmark_data_smithi188_141483_object86188:head# 2024-04-23T16:43:23.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c592c7c8:::benchmark_data_smithi188_141483_object5427:head# 2024-04-23T16:43:23.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c593b1e0:::benchmark_data_smithi188_141483_object64602:head# 2024-04-23T16:43:23.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5940134:::benchmark_data_smithi188_141483_object47920:head# 2024-04-23T16:43:23.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5944eb0:::benchmark_data_smithi188_141483_object37542:head# 2024-04-23T16:43:23.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5960220:::benchmark_data_smithi188_141483_object64947:head# 2024-04-23T16:43:23.635 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5978b0f:::benchmark_data_smithi188_141483_object7793:head# 2024-04-23T16:43:23.635 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5983f98:::benchmark_data_smithi188_141483_object79250:head# 2024-04-23T16:43:23.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c59960cc:::benchmark_data_smithi188_141483_object24817:head# 2024-04-23T16:43:23.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c599666d:::benchmark_data_smithi188_141483_object81092:head# 2024-04-23T16:43:23.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c59a39fc:::benchmark_data_smithi188_141483_object5922:head# 2024-04-23T16:43:23.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c59be33c:::benchmark_data_smithi188_141483_object57517:head# 2024-04-23T16:43:23.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c59c32b4:::benchmark_data_smithi188_141483_object17087:head# 2024-04-23T16:43:23.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c59d7710:::benchmark_data_smithi188_141483_object29247:head# 2024-04-23T16:43:23.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c59f2771:::benchmark_data_smithi188_141483_object54784:head# 2024-04-23T16:43:23.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c59fe2d0:::benchmark_data_smithi188_141483_object35195:head# 2024-04-23T16:43:23.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5a032cc:::benchmark_data_smithi188_141483_object85411:head# 2024-04-23T16:43:23.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5a182c4:::benchmark_data_smithi188_141483_object37589:head# 2024-04-23T16:43:23.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5a1f4bb:::benchmark_data_smithi188_141483_object27626:head# 2024-04-23T16:43:23.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5a2a4a6:::benchmark_data_smithi188_141483_object3681:head# 2024-04-23T16:43:23.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5a33be2:::benchmark_data_smithi188_141483_object73112:head# 2024-04-23T16:43:23.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5a3eaf2:::benchmark_data_smithi188_141483_object67750:head# 2024-04-23T16:43:23.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5a50575:::benchmark_data_smithi188_141483_object60135:head# 2024-04-23T16:43:23.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5a51a52:::benchmark_data_smithi188_141483_object6002:head# 2024-04-23T16:43:23.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5a7e19c:::benchmark_data_smithi188_141483_object14801:head# 2024-04-23T16:43:23.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5a7fe8a:::benchmark_data_smithi188_141483_object45023:head# 2024-04-23T16:43:23.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5a84d3e:::benchmark_data_smithi188_141483_object35418:head# 2024-04-23T16:43:23.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5a91f25:::benchmark_data_smithi188_141483_object17662:head# 2024-04-23T16:43:23.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5a97749:::benchmark_data_smithi188_141483_object36666:head# 2024-04-23T16:43:23.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5aa1550:::benchmark_data_smithi188_141483_object3954:head# 2024-04-23T16:43:23.644 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5aab806:::benchmark_data_smithi188_141483_object87288:head# 2024-04-23T16:43:23.645 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5abd72f:::benchmark_data_smithi188_141483_object16879:head# 2024-04-23T16:43:23.645 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5abe49e:::benchmark_data_smithi188_141483_object58902:head# 2024-04-23T16:43:23.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5ae3c57:::benchmark_data_smithi188_141483_object2624:head# 2024-04-23T16:43:23.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5b0dfef:::benchmark_data_smithi188_141483_object52322:head# 2024-04-23T16:43:23.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5b14e13:::benchmark_data_smithi188_141483_object38955:head# 2024-04-23T16:43:23.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5b2897c:::benchmark_data_smithi188_141483_object3583:head# 2024-04-23T16:43:23.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5b2a268:::benchmark_data_smithi188_141483_object14064:head# 2024-04-23T16:43:23.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5b4501c:::benchmark_data_smithi188_141483_object76746:head# 2024-04-23T16:43:23.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5b53778:::benchmark_data_smithi188_141483_object25843:head# 2024-04-23T16:43:23.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5b5c685:::benchmark_data_smithi188_141483_object68203:head# 2024-04-23T16:43:23.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5b6dd55:::benchmark_data_smithi188_141483_object70415:head# 2024-04-23T16:43:23.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5b76e8c:::benchmark_data_smithi188_141483_object33473:head# 2024-04-23T16:43:23.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5b7ce9d:::benchmark_data_smithi188_141483_object85619:head# 2024-04-23T16:43:23.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5b89efa:::benchmark_data_smithi188_141483_object31839:head# 2024-04-23T16:43:23.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5b9c745:::benchmark_data_smithi188_141483_object10969:head# 2024-04-23T16:43:23.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5ba155e:::benchmark_data_smithi188_141483_object22181:head# 2024-04-23T16:43:23.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5ba8eff:::benchmark_data_smithi188_141483_object1806:head# 2024-04-23T16:43:23.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5bbbf5e:::benchmark_data_smithi188_141483_object39648:head# 2024-04-23T16:43:23.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5bc1ff7:::benchmark_data_smithi188_141483_object42998:head# 2024-04-23T16:43:23.655 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:23.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5bd4d45:::benchmark_data_smithi188_141483_object81873:head# 2024-04-23T16:43:23.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5bd520c:::benchmark_data_smithi188_141483_object52916:head# 2024-04-23T16:43:23.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5bd92b9:::benchmark_data_smithi188_141483_object69911:head# 2024-04-23T16:43:23.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5bf23b8:::benchmark_data_smithi188_141483_object82449:head# 2024-04-23T16:43:23.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5c01bde:::benchmark_data_smithi188_141483_object89893:head# 2024-04-23T16:43:23.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5c1ed8f:::benchmark_data_smithi188_141483_object85190:head# 2024-04-23T16:43:23.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5c3411c:::benchmark_data_smithi188_141483_object59073:head# 2024-04-23T16:43:23.657 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5c3afd7:::benchmark_data_smithi188_141483_object34339:head# 2024-04-23T16:43:23.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5c4d8bf:::benchmark_data_smithi188_141483_object62277:head# 2024-04-23T16:43:23.658 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5c4dd39:::benchmark_data_smithi188_141483_object36853:head# 2024-04-23T16:43:23.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5c5d0be:::benchmark_data_smithi188_141483_object6169:head# 2024-04-23T16:43:23.659 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5c60a45:::benchmark_data_smithi188_141483_object49715:head# 2024-04-23T16:43:23.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5c63f4f:::benchmark_data_smithi188_141483_object72741:head# 2024-04-23T16:43:23.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5c6a3e8:::benchmark_data_smithi188_141483_object4824:head# 2024-04-23T16:43:23.660 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5c6b676:::benchmark_data_smithi188_141483_object67416:head# 2024-04-23T16:43:23.661 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5c78389:::benchmark_data_smithi188_141483_object53951:head# 2024-04-23T16:43:23.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5c7b25d:::benchmark_data_smithi188_141483_object61420:head# 2024-04-23T16:43:23.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5c7de02:::benchmark_data_smithi188_141483_object2619:head# 2024-04-23T16:43:23.662 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5c7f330:::benchmark_data_smithi188_141483_object64812:head# 2024-04-23T16:43:23.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5c9e37c:::benchmark_data_smithi188_141483_object9420:head# 2024-04-23T16:43:23.663 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5ca1e56:::benchmark_data_smithi188_141483_object86261:head# 2024-04-23T16:43:23.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5ca9197:::benchmark_data_smithi188_141483_object69907:head# 2024-04-23T16:43:23.664 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5cb7a52:::benchmark_data_smithi188_141483_object59560:head# 2024-04-23T16:43:23.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5cc7d28:::benchmark_data_smithi188_141483_object5645:head# 2024-04-23T16:43:23.665 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5ce483d:::benchmark_data_smithi188_141483_object20865:head# 2024-04-23T16:43:23.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5ce5a4d:::benchmark_data_smithi188_141483_object60873:head# 2024-04-23T16:43:23.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5cf4978:::benchmark_data_smithi188_141483_object62129:head# 2024-04-23T16:43:23.666 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5cf8b91:::benchmark_data_smithi188_141483_object71832:head# 2024-04-23T16:43:23.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5d0c26a:::benchmark_data_smithi188_141483_object72022:head# 2024-04-23T16:43:23.667 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5d0c3ff:::benchmark_data_smithi188_141483_object75276:head# 2024-04-23T16:43:23.668 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5d0d56b:::benchmark_data_smithi188_141483_object50737:head# 2024-04-23T16:43:23.668 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5d12ef4:::benchmark_data_smithi188_141483_object85488:head# 2024-04-23T16:43:23.669 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5d18f02:::benchmark_data_smithi188_141483_object86812:head# 2024-04-23T16:43:23.669 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5d1ea17:::benchmark_data_smithi188_141483_object36030:head# 2024-04-23T16:43:23.669 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5d42b90:::benchmark_data_smithi188_141483_object80667:head# 2024-04-23T16:43:23.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5d44ce5:::benchmark_data_smithi188_141483_object32348:head# 2024-04-23T16:43:23.670 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5d4f52d:::benchmark_data_smithi188_141483_object57103:head# 2024-04-23T16:43:23.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5d862c8:::benchmark_data_smithi188_141483_object84974:head# 2024-04-23T16:43:23.671 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5d8973d:::benchmark_data_smithi188_141483_object35185:head# 2024-04-23T16:43:23.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5da12b2:::benchmark_data_smithi188_141483_object74734:head# 2024-04-23T16:43:23.672 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5da6be4:::benchmark_data_smithi188_141483_object87982:head# 2024-04-23T16:43:23.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5da7e88:::benchmark_data_smithi188_141483_object80167:head# 2024-04-23T16:43:23.673 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5db76a0:::benchmark_data_smithi188_141483_object40880:head# 2024-04-23T16:43:23.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5dd7b82:::benchmark_data_smithi188_141483_object29536:head# 2024-04-23T16:43:23.674 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5ddda4e:::benchmark_data_smithi188_141483_object74126:head# 2024-04-23T16:43:23.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5deea2d:::benchmark_data_smithi188_141483_object48747:head# 2024-04-23T16:43:23.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e0551a:::benchmark_data_smithi188_141483_object85640:head# 2024-04-23T16:43:23.675 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e0dc4d:::benchmark_data_smithi188_141483_object63949:head# 2024-04-23T16:43:23.677 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e0df13:::benchmark_data_smithi188_141483_object67168:head# 2024-04-23T16:43:23.678 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e11927:::benchmark_data_smithi188_141483_object58453:head# 2024-04-23T16:43:23.678 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e17753:::benchmark_data_smithi188_141483_object51155:head# 2024-04-23T16:43:23.679 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e18684:::benchmark_data_smithi188_141483_object27928:head# 2024-04-23T16:43:23.679 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e1b2a4:::benchmark_data_smithi188_141483_object5931:head# 2024-04-23T16:43:23.680 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e1e63a:::benchmark_data_smithi188_141483_object69545:head# 2024-04-23T16:43:23.680 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e26607:::benchmark_data_smithi188_141483_object15764:head# 2024-04-23T16:43:23.680 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e36d97:::benchmark_data_smithi188_141483_object78167:head# 2024-04-23T16:43:23.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e3e238:::benchmark_data_smithi188_141483_object63400:head# 2024-04-23T16:43:23.681 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e445b0:::benchmark_data_smithi188_141483_object77582:head# 2024-04-23T16:43:23.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e47496:::benchmark_data_smithi188_141483_object66576:head# 2024-04-23T16:43:23.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e59043:::benchmark_data_smithi188_141483_object35118:head# 2024-04-23T16:43:23.682 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e7282e:::benchmark_data_smithi188_141483_object83845:head# 2024-04-23T16:43:23.683 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e7a503:::benchmark_data_smithi188_141483_object48574:head# 2024-04-23T16:43:23.683 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e807c6:::benchmark_data_smithi188_141483_object34449:head# 2024-04-23T16:43:23.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e8c405:::benchmark_data_smithi188_141483_object66382:head# 2024-04-23T16:43:23.684 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e960a0:::benchmark_data_smithi188_141483_object52259:head# 2024-04-23T16:43:23.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e98137:::benchmark_data_smithi188_141483_object33343:head# 2024-04-23T16:43:23.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5e99f53:::benchmark_data_smithi188_141483_object85525:head# 2024-04-23T16:43:23.685 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5eaf30e:::benchmark_data_smithi188_141483_object10929:head# 2024-04-23T16:43:23.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5ecc7a4:::benchmark_data_smithi188_141483_object80668:head# 2024-04-23T16:43:23.686 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5ed0d10:::benchmark_data_smithi188_141483_object1225:head# 2024-04-23T16:43:23.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5ed8d23:::benchmark_data_smithi188_141483_object60106:head# 2024-04-23T16:43:23.687 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5f00816:::benchmark_data_smithi188_141483_object72237:head# 2024-04-23T16:43:23.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5f01e9b:::benchmark_data_smithi188_141483_object87443:head# 2024-04-23T16:43:23.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5f2bd43:::benchmark_data_smithi188_141483_object719:head# 2024-04-23T16:43:23.688 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5f46e0d:::benchmark_data_smithi188_141483_object89415:head# 2024-04-23T16:43:23.689 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5f47221:::benchmark_data_smithi188_141483_object77779:head# 2024-04-23T16:43:23.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5f51b00:::benchmark_data_smithi188_141483_object21495:head# 2024-04-23T16:43:23.690 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5f64bd1:::benchmark_data_smithi188_141483_object27606:head# 2024-04-23T16:43:23.691 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5f6d625:::benchmark_data_smithi188_141483_object90307:head# 2024-04-23T16:43:23.691 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5f7302b:::benchmark_data_smithi188_141483_object30740:head# 2024-04-23T16:43:23.691 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5f7607c:::benchmark_data_smithi188_141483_object66081:head# 2024-04-23T16:43:23.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5f7bfae:::benchmark_data_smithi188_141483_object45226:head# 2024-04-23T16:43:23.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5f92b01:::benchmark_data_smithi188_141483_object62413:head# 2024-04-23T16:43:23.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5f94c35:::benchmark_data_smithi188_141483_object36741:head# 2024-04-23T16:43:23.693 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5f9ae1d:::benchmark_data_smithi188_141483_object62309:head# 2024-04-23T16:43:23.694 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5fb03b4:::benchmark_data_smithi188_141483_object63812:head# 2024-04-23T16:43:23.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5fb193c:::benchmark_data_smithi188_141483_object14356:head# 2024-04-23T16:43:23.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5fb1c28:::benchmark_data_smithi188_141483_object16190:head# 2024-04-23T16:43:23.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5fb79cb:::benchmark_data_smithi188_141483_object14205:head# 2024-04-23T16:43:23.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c5fc116f:::benchmark_data_smithi188_141483_object33125:head# 2024-04-23T16:43:23.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6001557:::benchmark_data_smithi188_141483_object5089:head# 2024-04-23T16:43:23.696 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c600d158:::benchmark_data_smithi188_141483_object25524:head# 2024-04-23T16:43:23.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c601530a:::benchmark_data_smithi188_141483_object11727:head# 2024-04-23T16:43:23.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c602261c:::benchmark_data_smithi188_141483_object39587:head# 2024-04-23T16:43:23.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c602f538:::benchmark_data_smithi188_141483_object59921:head# 2024-04-23T16:43:23.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c602fa1b:::benchmark_data_smithi188_141483_object5780:head# 2024-04-23T16:43:23.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c60398fc:::benchmark_data_smithi188_141483_object86036:head# 2024-04-23T16:43:23.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c603c971:::benchmark_data_smithi188_141483_object79625:head# 2024-04-23T16:43:23.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6042328:::benchmark_data_smithi188_141483_object23968:head# 2024-04-23T16:43:23.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6042bbb:::benchmark_data_smithi188_141483_object18283:head# 2024-04-23T16:43:23.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c60491da:::benchmark_data_smithi188_141483_object16679:head# 2024-04-23T16:43:23.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6049bda:::benchmark_data_smithi188_141483_object90200:head# 2024-04-23T16:43:23.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6073cc0:::benchmark_data_smithi188_141483_object88773:head# 2024-04-23T16:43:23.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c60744db:::benchmark_data_smithi188_141483_object72540:head# 2024-04-23T16:43:23.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c608b26c:::benchmark_data_smithi188_141483_object9332:head# 2024-04-23T16:43:23.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c609016a:::benchmark_data_smithi188_141483_object77238:head# 2024-04-23T16:43:23.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c609421e:::benchmark_data_smithi188_141483_object39191:head# 2024-04-23T16:43:23.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c60976d2:::benchmark_data_smithi188_141483_object77264:head# 2024-04-23T16:43:23.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6097b68:::benchmark_data_smithi188_141483_object8689:head# 2024-04-23T16:43:23.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c609fdbe:::benchmark_data_smithi188_141483_object31383:head# 2024-04-23T16:43:23.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c60aecaa:::benchmark_data_smithi188_141483_object25184:head# 2024-04-23T16:43:23.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c60c049f:::benchmark_data_smithi188_141483_object30108:head# 2024-04-23T16:43:23.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c60c21db:::benchmark_data_smithi188_141483_object75402:head# 2024-04-23T16:43:23.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c60c5a3c:::benchmark_data_smithi188_141483_object48979:head# 2024-04-23T16:43:23.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c60d5eed:::benchmark_data_smithi188_141483_object71528:head# 2024-04-23T16:43:23.706 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6105582:::benchmark_data_smithi188_141483_object25795:head# 2024-04-23T16:43:23.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c610972d:::benchmark_data_smithi188_141483_object41966:head# 2024-04-23T16:43:23.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c610b0b5:::benchmark_data_smithi188_141483_object53139:head# 2024-04-23T16:43:23.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c61132bb:::benchmark_data_smithi188_141483_object29319:head# 2024-04-23T16:43:23.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6115be8:::benchmark_data_smithi188_141483_object88070:head# 2024-04-23T16:43:23.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c611f385:::benchmark_data_smithi188_141483_object85219:head# 2024-04-23T16:43:23.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c612035b:::benchmark_data_smithi188_141483_object39773:head# 2024-04-23T16:43:23.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6120fcd:::benchmark_data_smithi188_141483_object77362:head# 2024-04-23T16:43:23.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c613785a:::benchmark_data_smithi188_141483_object46872:head# 2024-04-23T16:43:23.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6141067:::benchmark_data_smithi188_141483_object13635:head# 2024-04-23T16:43:23.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6141076:::benchmark_data_smithi188_141483_object77163:head# 2024-04-23T16:43:23.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c61420c4:::benchmark_data_smithi188_141483_object36242:head# 2024-04-23T16:43:23.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6163dc2:::benchmark_data_smithi188_141483_object32082:head# 2024-04-23T16:43:23.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6176d91:::benchmark_data_smithi188_141483_object41636:head# 2024-04-23T16:43:23.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c618e1a2:::benchmark_data_smithi188_141483_object89009:head# 2024-04-23T16:43:23.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c618e30e:::benchmark_data_smithi188_141483_object65159:head# 2024-04-23T16:43:23.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c61a7cb0:::benchmark_data_smithi188_141483_object76771:head# 2024-04-23T16:43:23.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c61a8467:::benchmark_data_smithi188_141483_object35759:head# 2024-04-23T16:43:23.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c61d2b94:::benchmark_data_smithi188_141483_object26568:head# 2024-04-23T16:43:23.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c61d3922:::benchmark_data_smithi188_141483_object12292:head# 2024-04-23T16:43:23.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c61dda3a:::benchmark_data_smithi188_141483_object47483:head# 2024-04-23T16:43:23.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c61e4528:::benchmark_data_smithi188_141483_object31446:head# 2024-04-23T16:43:23.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c61e6963:::benchmark_data_smithi188_141483_object71656:head# 2024-04-23T16:43:23.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6210399:::benchmark_data_smithi188_141483_object7527:head# 2024-04-23T16:43:23.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6211340:::benchmark_data_smithi188_141483_object81799:head# 2024-04-23T16:43:23.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c62152b7:::benchmark_data_smithi188_141483_object42183:head# 2024-04-23T16:43:23.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6216d4b:::benchmark_data_smithi188_141483_object57474:head# 2024-04-23T16:43:23.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6251542:::benchmark_data_smithi188_141483_object41699:head# 2024-04-23T16:43:23.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6260eb6:::benchmark_data_smithi188_141483_object61630:head# 2024-04-23T16:43:23.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6266353:::benchmark_data_smithi188_141483_object24894:head# 2024-04-23T16:43:23.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c626f10f:::benchmark_data_smithi188_141483_object7360:head# 2024-04-23T16:43:23.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6270d47:::benchmark_data_smithi188_141483_object45629:head# 2024-04-23T16:43:23.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6277819:::benchmark_data_smithi188_141483_object22862:head# 2024-04-23T16:43:23.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c627eb78:::benchmark_data_smithi188_141483_object12509:head# 2024-04-23T16:43:23.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c62aebbb:::benchmark_data_smithi188_141483_object30041:head# 2024-04-23T16:43:23.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c62bc11d:::benchmark_data_smithi188_141483_object65891:head# 2024-04-23T16:43:23.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c62c647e:::benchmark_data_smithi188_141483_object45374:head# 2024-04-23T16:43:23.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c62ca1f5:::benchmark_data_smithi188_141483_object48849:head# 2024-04-23T16:43:23.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c62e33cd:::benchmark_data_smithi188_141483_object35602:head# 2024-04-23T16:43:23.724 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c62e3c04:::benchmark_data_smithi188_141483_object21015:head# 2024-04-23T16:43:23.725 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c62e5188:::benchmark_data_smithi188_141483_object18797:head# 2024-04-23T16:43:23.725 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c62e6982:::benchmark_data_smithi188_141483_object74987:head# 2024-04-23T16:43:23.725 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c62eec54:::benchmark_data_smithi188_141483_object31885:head# 2024-04-23T16:43:23.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c62fa7e7:::benchmark_data_smithi188_141483_object36236:head# 2024-04-23T16:43:23.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c63115ea:::benchmark_data_smithi188_141483_object14150:head# 2024-04-23T16:43:23.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6314847:::benchmark_data_smithi188_141483_object21722:head# 2024-04-23T16:43:23.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c63175e1:::benchmark_data_smithi188_141483_object8106:head# 2024-04-23T16:43:23.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c63179f1:::benchmark_data_smithi188_141483_object29367:head# 2024-04-23T16:43:23.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c633dfe5:::benchmark_data_smithi188_141483_object15628:head# 2024-04-23T16:43:23.728 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c634c9ae:::benchmark_data_smithi188_141483_object66292:head# 2024-04-23T16:43:23.729 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c634d877:::benchmark_data_smithi188_141483_object4593:head# 2024-04-23T16:43:23.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c635561c:::benchmark_data_smithi188_141483_object66076:head# 2024-04-23T16:43:23.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6369ec9:::benchmark_data_smithi188_141483_object71980:head# 2024-04-23T16:43:23.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c638136a:::benchmark_data_smithi188_141483_object46088:head# 2024-04-23T16:43:23.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c63834a6:::benchmark_data_smithi188_141483_object41880:head# 2024-04-23T16:43:23.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c63a4702:::benchmark_data_smithi188_141483_object64591:head# 2024-04-23T16:43:23.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c63a600f:::benchmark_data_smithi188_141483_object7737:head# 2024-04-23T16:43:23.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c63ac4b8:::benchmark_data_smithi188_141483_object49922:head# 2024-04-23T16:43:23.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c63bf066:::benchmark_data_smithi188_141483_object62513:head# 2024-04-23T16:43:23.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c63c7a31:::benchmark_data_smithi188_141483_object50095:head# 2024-04-23T16:43:23.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c63ce6c2:::benchmark_data_smithi188_141483_object60270:head# 2024-04-23T16:43:23.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c63d75ae:::benchmark_data_smithi188_141483_object72330:head# 2024-04-23T16:43:23.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c63df69f:::benchmark_data_smithi188_141483_object86503:head# 2024-04-23T16:43:23.735 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c63e6f31:::benchmark_data_smithi188_141483_object83906:head# 2024-04-23T16:43:23.735 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c63f136f:::benchmark_data_smithi188_141483_object21039:head# 2024-04-23T16:43:23.735 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c63f32b1:::benchmark_data_smithi188_141483_object10004:head# 2024-04-23T16:43:23.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c640296f:::benchmark_data_smithi188_141483_object87950:head# 2024-04-23T16:43:23.737 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6413fe1:::benchmark_data_smithi188_141483_object15505:head# 2024-04-23T16:43:23.737 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c641d1b9:::benchmark_data_smithi188_141483_object51381:head# 2024-04-23T16:43:23.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c64218f0:::benchmark_data_smithi188_141483_object27630:head# 2024-04-23T16:43:23.739 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:43:23.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c642545c:::benchmark_data_smithi188_141483_object69074:head# 2024-04-23T16:43:23.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6425cc5:::benchmark_data_smithi188_141483_object69289:head# 2024-04-23T16:43:23.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c643292e:::benchmark_data_smithi188_141483_object81205:head# 2024-04-23T16:43:23.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6455c9c:::benchmark_data_smithi188_141483_object18752:head# 2024-04-23T16:43:23.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6465b6d:::benchmark_data_smithi188_141483_object73527:head# 2024-04-23T16:43:23.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c64769dc:::benchmark_data_smithi188_141483_object63999:head# 2024-04-23T16:43:23.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6477044:::benchmark_data_smithi188_141483_object3634:head# 2024-04-23T16:43:23.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c647c701:::benchmark_data_smithi188_141483_object53093:head# 2024-04-23T16:43:23.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c649bb6b:::benchmark_data_smithi188_141483_object37489:head# 2024-04-23T16:43:23.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c64a6778:::benchmark_data_smithi188_141483_object55180:head# 2024-04-23T16:43:23.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c64aa44b:::benchmark_data_smithi188_141483_object47598:head# 2024-04-23T16:43:23.744 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c64b58ce:::benchmark_data_smithi188_141483_object43717:head# 2024-04-23T16:43:23.744 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c64d2f10:::benchmark_data_smithi188_141483_object24428:head# 2024-04-23T16:43:23.745 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c64dbd61:::benchmark_data_smithi188_141483_object30471:head# 2024-04-23T16:43:23.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c64e91e3:::benchmark_data_smithi188_141483_object76020:head# 2024-04-23T16:43:23.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c65005bc:::benchmark_data_smithi188_141483_object57039:head# 2024-04-23T16:43:23.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c650db3a:::benchmark_data_smithi188_141483_object76347:head# 2024-04-23T16:43:23.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6512883:::benchmark_data_smithi188_141483_object62254:head# 2024-04-23T16:43:23.747 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c653c612:::benchmark_data_smithi188_141483_object76023:head# 2024-04-23T16:43:23.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c65446ef:::benchmark_data_smithi188_141483_object73383:head# 2024-04-23T16:43:23.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c65558c7:::benchmark_data_smithi188_141483_object76961:head# 2024-04-23T16:43:23.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6559512:::benchmark_data_smithi188_141483_object22774:head# 2024-04-23T16:43:23.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6563e2e:::benchmark_data_smithi188_141483_object48768:head# 2024-04-23T16:43:23.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c65697ca:::benchmark_data_smithi188_141483_object32389:head# 2024-04-23T16:43:23.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c658159e:::benchmark_data_smithi188_141483_object72336:head# 2024-04-23T16:43:23.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c65925c5:::benchmark_data_smithi188_141483_object35473:head# 2024-04-23T16:43:23.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c65a4cc2:::benchmark_data_smithi188_141483_object86026:head# 2024-04-23T16:43:23.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c65ac111:::benchmark_data_smithi188_141483_object49202:head# 2024-04-23T16:43:23.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c65b5a6a:::benchmark_data_smithi188_141483_object15727:head# 2024-04-23T16:43:23.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c65b7bf7:::benchmark_data_smithi188_141483_object975:head# 2024-04-23T16:43:23.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c65ba40e:::benchmark_data_smithi188_141483_object45808:head# 2024-04-23T16:43:23.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c65c1971:::benchmark_data_smithi188_141483_object28665:head# 2024-04-23T16:43:23.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c65dc094:::benchmark_data_smithi188_141483_object62147:head# 2024-04-23T16:43:23.753 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c65e6433:::benchmark_data_smithi188_141483_object84866:head# 2024-04-23T16:43:23.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:23.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:43:23.757 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c65ead8d:::benchmark_data_smithi188_141483_object67664:head# 2024-04-23T16:43:23.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c65eb043:::benchmark_data_smithi188_141483_object62347:head# 2024-04-23T16:43:23.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c65ff60f:::benchmark_data_smithi188_141483_object63611:head# 2024-04-23T16:43:23.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6608086:::benchmark_data_smithi188_141483_object25882:head# 2024-04-23T16:43:23.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c660ea3e:::benchmark_data_smithi188_141483_object87100:head# 2024-04-23T16:43:23.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6610039:::benchmark_data_smithi188_141483_object68689:head# 2024-04-23T16:43:23.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c661a3d2:::benchmark_data_smithi188_141483_object44927:head# 2024-04-23T16:43:23.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6643967:::benchmark_data_smithi188_141483_object78001:head# 2024-04-23T16:43:23.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6649f4b:::benchmark_data_smithi188_141483_object11370:head# 2024-04-23T16:43:23.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6653348:::benchmark_data_smithi188_141483_object69981:head# 2024-04-23T16:43:23.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c66533bf:::benchmark_data_smithi188_141483_object10672:head# 2024-04-23T16:43:23.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c665c159:::benchmark_data_smithi188_141483_object87450:head# 2024-04-23T16:43:23.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c666095f:::benchmark_data_smithi188_141483_object61745:head# 2024-04-23T16:43:23.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6664852:::benchmark_data_smithi188_141483_object37358:head# 2024-04-23T16:43:23.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6666867:::benchmark_data_smithi188_141483_object31111:head# 2024-04-23T16:43:23.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6668d37:::benchmark_data_smithi188_141483_object71212:head# 2024-04-23T16:43:23.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c666e742:::benchmark_data_smithi188_141483_object11865:head# 2024-04-23T16:43:23.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6670be7:::benchmark_data_smithi188_141483_object32147:head# 2024-04-23T16:43:23.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6671f3b:::benchmark_data_smithi188_141483_object11352:head# 2024-04-23T16:43:23.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c667b046:::benchmark_data_smithi188_141483_object71203:head# 2024-04-23T16:43:23.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c667c61d:::benchmark_data_smithi188_141483_object50670:head# 2024-04-23T16:43:23.765 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c668d7ca:::benchmark_data_smithi188_141483_object8712:head# 2024-04-23T16:43:23.765 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6698d93:::benchmark_data_smithi188_141483_object41980:head# 2024-04-23T16:43:23.765 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c669cbd1:::benchmark_data_smithi188_141483_object39985:head# 2024-04-23T16:43:23.765 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c66a38bb:::benchmark_data_smithi188_141483_object3495:head# 2024-04-23T16:43:23.766 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c66b370c:::benchmark_data_smithi188_141483_object83324:head# 2024-04-23T16:43:23.767 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c66dc4e3:::benchmark_data_smithi188_141483_object28153:head# 2024-04-23T16:43:23.767 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c66e2936:::benchmark_data_smithi188_141483_object5880:head# 2024-04-23T16:43:23.767 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c66e4be3:::benchmark_data_smithi188_141483_object34355:head# 2024-04-23T16:43:23.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c66f0b91:::benchmark_data_smithi188_141483_object63092:head# 2024-04-23T16:43:23.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c66f6595:::benchmark_data_smithi188_141483_object85085:head# 2024-04-23T16:43:23.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c66ff211:::benchmark_data_smithi188_141483_object34661:head# 2024-04-23T16:43:23.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6708cf3:::benchmark_data_smithi188_141483_object77419:head# 2024-04-23T16:43:23.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c670c094:::benchmark_data_smithi188_141483_object19174:head# 2024-04-23T16:43:23.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c670f0d6:::benchmark_data_smithi188_141483_object82405:head# 2024-04-23T16:43:23.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c672d1f5:::benchmark_data_smithi188_141483_object60291:head# 2024-04-23T16:43:23.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c67403d1:::benchmark_data_smithi188_141483_object73187:head# 2024-04-23T16:43:23.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6740c9f:::benchmark_data_smithi188_141483_object27738:head# 2024-04-23T16:43:23.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c675584b:::benchmark_data_smithi188_141483_object77753:head# 2024-04-23T16:43:23.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c67618f4:::benchmark_data_smithi188_141483_object85869:head# 2024-04-23T16:43:23.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c676b5cf:::benchmark_data_smithi188_141483_object36922:head# 2024-04-23T16:43:23.772 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c676d1c1:::benchmark_data_smithi188_141483_object54292:head# 2024-04-23T16:43:23.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6788388:::benchmark_data_smithi188_141483_object13856:head# 2024-04-23T16:43:23.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c67917f9:::benchmark_data_smithi188_141483_object35752:head# 2024-04-23T16:43:23.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6791bfc:::benchmark_data_smithi188_141483_object70750:head# 2024-04-23T16:43:23.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c67a3dea:::benchmark_data_smithi188_141483_object37795:head# 2024-04-23T16:43:23.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c67b05c5:::benchmark_data_smithi188_141483_object49936:head# 2024-04-23T16:43:23.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c67b7256:::benchmark_data_smithi188_141483_object82053:head# 2024-04-23T16:43:23.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c67b91c4:::benchmark_data_smithi188_141483_object42639:head# 2024-04-23T16:43:23.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c67db449:::benchmark_data_smithi188_141483_object71030:head# 2024-04-23T16:43:23.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c67f6071:::benchmark_data_smithi188_141483_object84963:head# 2024-04-23T16:43:23.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c67fce44:::benchmark_data_smithi188_141483_object18570:head# 2024-04-23T16:43:23.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c68048ac:::benchmark_data_smithi188_141483_object67939:head# 2024-04-23T16:43:23.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6812ef1:::benchmark_data_smithi188_141483_object90300:head# 2024-04-23T16:43:23.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c68163f4:::benchmark_data_smithi188_141483_object59990:head# 2024-04-23T16:43:23.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c68167eb:::benchmark_data_smithi188_141483_object21643:head# 2024-04-23T16:43:23.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6823a30:::benchmark_data_smithi188_141483_object37130:head# 2024-04-23T16:43:23.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c68281b1:::benchmark_data_smithi188_141483_object75185:head# 2024-04-23T16:43:23.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6828601:::benchmark_data_smithi188_141483_object62384:head# 2024-04-23T16:43:23.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c682dbf8:::benchmark_data_smithi188_141483_object48246:head# 2024-04-23T16:43:23.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6840391:::benchmark_data_smithi188_141483_object74919:head# 2024-04-23T16:43:23.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c68410d5:::benchmark_data_smithi188_141483_object28897:head# 2024-04-23T16:43:23.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6843d52:::benchmark_data_smithi188_141483_object11438:head# 2024-04-23T16:43:23.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6848d2d:::benchmark_data_smithi188_141483_object54355:head# 2024-04-23T16:43:23.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6865ec2:::benchmark_data_smithi188_141483_object81079:head# 2024-04-23T16:43:23.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6866f8f:::benchmark_data_smithi188_141483_object20958:head# 2024-04-23T16:43:23.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c68700bb:::benchmark_data_smithi188_141483_object49235:head# 2024-04-23T16:43:23.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6879c0c:::benchmark_data_smithi188_141483_object74311:head# 2024-04-23T16:43:23.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6896734:::benchmark_data_smithi188_141483_object86064:head# 2024-04-23T16:43:23.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c689819c:::benchmark_data_smithi188_141483_object67637:head# 2024-04-23T16:43:23.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c68c4f93:::benchmark_data_smithi188_141483_object17214:head# 2024-04-23T16:43:23.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c68ccf24:::benchmark_data_smithi188_141483_object28317:head# 2024-04-23T16:43:23.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c68cfebe:::benchmark_data_smithi188_141483_object8596:head# 2024-04-23T16:43:23.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c68e0fc7:::benchmark_data_smithi188_141483_object73475:head# 2024-04-23T16:43:23.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c68e20de:::benchmark_data_smithi188_141483_object23553:head# 2024-04-23T16:43:23.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c68e8006:::benchmark_data_smithi188_141483_object15745:head# 2024-04-23T16:43:23.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c68eb587:::benchmark_data_smithi188_141483_object32843:head# 2024-04-23T16:43:23.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c68f2646:::benchmark_data_smithi188_141483_object61834:head# 2024-04-23T16:43:23.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c68f2ba2:::benchmark_data_smithi188_141483_object51375:head# 2024-04-23T16:43:23.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c68fff23:::benchmark_data_smithi188_141483_object13069:head# 2024-04-23T16:43:23.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6910e49:::benchmark_data_smithi188_141483_object10845:head# 2024-04-23T16:43:23.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6913dc2:::benchmark_data_smithi188_141483_object46762:head# 2024-04-23T16:43:23.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c692499c:::benchmark_data_smithi188_141483_object5149:head# 2024-04-23T16:43:23.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c692de60:::benchmark_data_smithi188_141483_object18824:head# 2024-04-23T16:43:23.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6945d6e:::benchmark_data_smithi188_141483_object60335:head# 2024-04-23T16:43:23.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c695f4f7:::benchmark_data_smithi188_141483_object10193:head# 2024-04-23T16:43:23.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6963eec:::benchmark_data_smithi188_141483_object41995:head# 2024-04-23T16:43:23.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c697b54d:::benchmark_data_smithi188_141483_object22474:head# 2024-04-23T16:43:23.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c697ea23:::benchmark_data_smithi188_141483_object31997:head# 2024-04-23T16:43:23.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c699b01e:::benchmark_data_smithi188_141483_object3774:head# 2024-04-23T16:43:23.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c69a037e:::benchmark_data_smithi188_141483_object31991:head# 2024-04-23T16:43:23.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c69a37e6:::benchmark_data_smithi188_141483_object44916:head# 2024-04-23T16:43:23.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c69b3d16:::benchmark_data_smithi188_141483_object14665:head# 2024-04-23T16:43:23.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c69b7d83:::benchmark_data_smithi188_141483_object11816:head# 2024-04-23T16:43:23.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c69bfdea:::benchmark_data_smithi188_141483_object30256:head# 2024-04-23T16:43:23.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c69cab32:::benchmark_data_smithi188_141483_object19058:head# 2024-04-23T16:43:23.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c69cd7b4:::benchmark_data_smithi188_141483_object77256:head# 2024-04-23T16:43:23.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c69d2a73:::benchmark_data_smithi188_141483_object56355:head# 2024-04-23T16:43:23.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c69d3228:::benchmark_data_smithi188_141483_object86128:head# 2024-04-23T16:43:23.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c69e31a8:::benchmark_data_smithi188_141483_object11620:head# 2024-04-23T16:43:23.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c69ed580:::benchmark_data_smithi188_141483_object45788:head# 2024-04-23T16:43:23.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c69f11d4:::benchmark_data_smithi188_141483_object88353:head# 2024-04-23T16:43:23.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c69fa509:::benchmark_data_smithi188_141483_object88913:head# 2024-04-23T16:43:23.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6a129fa:::benchmark_data_smithi188_141483_object10012:head# 2024-04-23T16:43:23.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6a1be37:::benchmark_data_smithi188_141483_object23166:head# 2024-04-23T16:43:23.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6a261e4:::benchmark_data_smithi188_141483_object63844:head# 2024-04-23T16:43:23.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6a311d8:::benchmark_data_smithi188_141483_object7683:head# 2024-04-23T16:43:23.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6a3f433:::benchmark_data_smithi188_141483_object7533:head# 2024-04-23T16:43:23.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6a558eb:::benchmark_data_smithi188_141483_object77230:head# 2024-04-23T16:43:23.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6a78bd3:::benchmark_data_smithi188_141483_object61017:head# 2024-04-23T16:43:23.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6a84b07:::benchmark_data_smithi188_141483_object52612:head# 2024-04-23T16:43:23.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6aaaba9:::benchmark_data_smithi188_141483_object7514:head# 2024-04-23T16:43:23.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6ab3820:::benchmark_data_smithi188_141483_object47012:head# 2024-04-23T16:43:23.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6ac027c:::benchmark_data_smithi188_141483_object15970:head# 2024-04-23T16:43:23.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6ac8fce:::benchmark_data_smithi188_141483_object54223:head# 2024-04-23T16:43:23.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6aca995:::benchmark_data_smithi188_141483_object2326:head# 2024-04-23T16:43:23.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6ad9c8c:::benchmark_data_smithi188_141483_object39156:head# 2024-04-23T16:43:23.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6ae36e7:::benchmark_data_smithi188_141483_object6001:head# 2024-04-23T16:43:23.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6af727d:::benchmark_data_smithi188_141483_object2870:head# 2024-04-23T16:43:23.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6afca1a:::benchmark_data_smithi188_141483_object2791:head# 2024-04-23T16:43:23.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6afeaed:::benchmark_data_smithi188_141483_object19341:head# 2024-04-23T16:43:23.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6b075fd:::benchmark_data_smithi188_141483_object45097:head# 2024-04-23T16:43:23.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6b11182:::benchmark_data_smithi188_141483_object36793:head# 2024-04-23T16:43:23.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6b153c2:::benchmark_data_smithi188_141483_object77258:head# 2024-04-23T16:43:23.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6b2e1f6:::benchmark_data_smithi188_141483_object56920:head# 2024-04-23T16:43:23.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6b3a4f0:::benchmark_data_smithi188_141483_object79309:head# 2024-04-23T16:43:23.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6b3b2f0:::benchmark_data_smithi188_141483_object82474:head# 2024-04-23T16:43:23.809 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6b42abf:::benchmark_data_smithi188_141483_object19092:head# 2024-04-23T16:43:23.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6b4f4e5:::benchmark_data_smithi188_141483_object75464:head# 2024-04-23T16:43:23.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6b6a820:::benchmark_data_smithi188_141483_object75469:head# 2024-04-23T16:43:23.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6b911a2:::benchmark_data_smithi188_141483_object49541:head# 2024-04-23T16:43:23.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6b94027:::benchmark_data_smithi188_141483_object43546:head# 2024-04-23T16:43:23.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6b99aba:::benchmark_data_smithi188_141483_object46431:head# 2024-04-23T16:43:23.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6ba831f:::benchmark_data_smithi188_141483_object3431:head# 2024-04-23T16:43:23.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6bad97a:::benchmark_data_smithi188_141483_object1962:head# 2024-04-23T16:43:23.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6bbf3dd:::benchmark_data_smithi188_141483_object25033:head# 2024-04-23T16:43:23.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6bd3c46:::benchmark_data_smithi188_141483_object28021:head# 2024-04-23T16:43:23.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6bf4fb3:::benchmark_data_smithi188_141483_object41284:head# 2024-04-23T16:43:23.814 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6c0a6b3:::benchmark_data_smithi188_141483_object3749:head# 2024-04-23T16:43:23.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6c17f36:::benchmark_data_smithi188_141483_object58938:head# 2024-04-23T16:43:23.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6c18de8:::benchmark_data_smithi188_141483_object6669:head# 2024-04-23T16:43:23.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6c4755e:::benchmark_data_smithi188_141483_object63100:head# 2024-04-23T16:43:23.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6c48f0f:::benchmark_data_smithi188_141483_object42543:head# 2024-04-23T16:43:23.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6c4af17:::benchmark_data_smithi188_141483_object69452:head# 2024-04-23T16:43:23.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6c4fe40:::benchmark_data_smithi188_141483_object28360:head# 2024-04-23T16:43:23.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6c578ac:::benchmark_data_smithi188_141483_object63107:head# 2024-04-23T16:43:23.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6c61f37:::benchmark_data_smithi188_141483_object72999:head# 2024-04-23T16:43:23.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6c6cd5e:::benchmark_data_smithi188_141483_object85440:head# 2024-04-23T16:43:23.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6c700f0:::benchmark_data_smithi188_141483_object13913:head# 2024-04-23T16:43:23.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6c7d7db:::benchmark_data_smithi188_141483_object50822:head# 2024-04-23T16:43:23.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6c91fd9:::benchmark_data_smithi188_141483_object12629:head# 2024-04-23T16:43:23.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6c92fe7:::benchmark_data_smithi188_141483_object6534:head# 2024-04-23T16:43:23.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6cc6cf6:::benchmark_data_smithi188_141483_object35137:head# 2024-04-23T16:43:23.821 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6ccde12:::benchmark_data_smithi188_141483_object76097:head# 2024-04-23T16:43:23.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6cd5a6b:::benchmark_data_smithi188_141483_object10186:head# 2024-04-23T16:43:23.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6cd656b:::benchmark_data_smithi188_141483_object68364:head# 2024-04-23T16:43:23.822 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6cdfdc8:::benchmark_data_smithi188_141483_object76352:head# 2024-04-23T16:43:23.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6d3871c:::benchmark_data_smithi188_141483_object75429:head# 2024-04-23T16:43:23.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6d45a8c:::benchmark_data_smithi188_141483_object26938:head# 2024-04-23T16:43:23.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6d4f47d:::benchmark_data_smithi188_141483_object84419:head# 2024-04-23T16:43:23.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6d4ff4f:::benchmark_data_smithi188_141483_object82952:head# 2024-04-23T16:43:23.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6d5939c:::benchmark_data_smithi188_141483_object2896:head# 2024-04-23T16:43:23.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6d5950b:::benchmark_data_smithi188_141483_object10922:head# 2024-04-23T16:43:23.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6d5ed3d:::benchmark_data_smithi188_141483_object68383:head# 2024-04-23T16:43:23.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6d6929f:::benchmark_data_smithi188_141483_object39225:head# 2024-04-23T16:43:23.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6d84d77:::benchmark_data_smithi188_141483_object55271:head# 2024-04-23T16:43:23.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6d96d2e:::benchmark_data_smithi188_141483_object55745:head# 2024-04-23T16:43:23.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6db4d5f:::benchmark_data_smithi188_141483_object56063:head# 2024-04-23T16:43:23.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6dc0069:::benchmark_data_smithi188_141483_object37763:head# 2024-04-23T16:43:23.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6dcd8bb:::benchmark_data_smithi188_141483_object88624:head# 2024-04-23T16:43:23.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6dd23f7:::benchmark_data_smithi188_141483_object71811:head# 2024-04-23T16:43:23.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6dd6340:::benchmark_data_smithi188_141483_object74969:head# 2024-04-23T16:43:23.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6defdc5:::benchmark_data_smithi188_141483_object5092:head# 2024-04-23T16:43:23.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6df47f3:::benchmark_data_smithi188_141483_object32232:head# 2024-04-23T16:43:23.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6df78c7:::benchmark_data_smithi188_141483_object80632:head# 2024-04-23T16:43:23.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6e12a53:::benchmark_data_smithi188_141483_object88012:head# 2024-04-23T16:43:23.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6e1b072:::benchmark_data_smithi188_141483_object41928:head# 2024-04-23T16:43:23.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6e1d956:::benchmark_data_smithi188_141483_object67604:head# 2024-04-23T16:43:23.832 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6e2e262:::benchmark_data_smithi188_141483_object79156:head# 2024-04-23T16:43:23.832 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6e30ce0:::benchmark_data_smithi188_141483_object9665:head# 2024-04-23T16:43:23.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6e35ac1:::benchmark_data_smithi188_141483_object45099:head# 2024-04-23T16:43:23.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6e3eb9d:::benchmark_data_smithi188_141483_object78542:head# 2024-04-23T16:43:23.834 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6e4632a:::benchmark_data_smithi188_141483_object38428:head# 2024-04-23T16:43:23.834 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6e473c6:::benchmark_data_smithi188_141483_object44462:head# 2024-04-23T16:43:23.834 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6e499d5:::benchmark_data_smithi188_141483_object55931:head# 2024-04-23T16:43:23.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6e4b184:::benchmark_data_smithi188_141483_object22427:head# 2024-04-23T16:43:23.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6e57795:::benchmark_data_smithi188_141483_object3588:head# 2024-04-23T16:43:23.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6e5c44f:::benchmark_data_smithi188_141483_object22269:head# 2024-04-23T16:43:23.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6e5c689:::benchmark_data_smithi188_141483_object15637:head# 2024-04-23T16:43:23.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6e6f87d:::benchmark_data_smithi188_141483_object46317:head# 2024-04-23T16:43:23.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6e79ffb:::benchmark_data_smithi188_141483_object78226:head# 2024-04-23T16:43:23.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6e8ba5e:::benchmark_data_smithi188_141483_object22641:head# 2024-04-23T16:43:23.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6e8cdf2:::benchmark_data_smithi188_141483_object22945:head# 2024-04-23T16:43:23.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6e8e3a6:::benchmark_data_smithi188_141483_object4235:head# 2024-04-23T16:43:23.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6eb2fdb:::benchmark_data_smithi188_141483_object63603:head# 2024-04-23T16:43:23.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6ecc6c2:::benchmark_data_smithi188_141483_object86969:head# 2024-04-23T16:43:23.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6ed614e:::benchmark_data_smithi188_141483_object23496:head# 2024-04-23T16:43:23.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6ee0c65:::benchmark_data_smithi188_141483_object10007:head# 2024-04-23T16:43:23.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6ee22a8:::benchmark_data_smithi188_141483_object88427:head# 2024-04-23T16:43:23.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6eebf83:::benchmark_data_smithi188_141483_object30630:head# 2024-04-23T16:43:23.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6ef7f0e:::benchmark_data_smithi188_141483_object69741:head# 2024-04-23T16:43:23.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6f1a7bd:::benchmark_data_smithi188_141483_object32076:head# 2024-04-23T16:43:23.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6f1ba08:::benchmark_data_smithi188_141483_object50193:head# 2024-04-23T16:43:23.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6f1e5d9:::benchmark_data_smithi188_141483_object77875:head# 2024-04-23T16:43:23.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6f24ea6:::benchmark_data_smithi188_141483_object19131:head# 2024-04-23T16:43:23.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6f29646:::benchmark_data_smithi188_141483_object49216:head# 2024-04-23T16:43:23.845 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6f2b710:::benchmark_data_smithi188_141483_object31569:head# 2024-04-23T16:43:23.845 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6f38762:::benchmark_data_smithi188_141483_object84928:head# 2024-04-23T16:43:23.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6f5e909:::benchmark_data_smithi188_141483_object21233:head# 2024-04-23T16:43:23.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6f83434:::benchmark_data_smithi188_141483_object66122:head# 2024-04-23T16:43:23.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6f94a89:::benchmark_data_smithi188_141483_object62644:head# 2024-04-23T16:43:23.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6faa3c6:::benchmark_data_smithi188_141483_object45148:head# 2024-04-23T16:43:23.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6fad590:::benchmark_data_smithi188_141483_object50589:head# 2024-04-23T16:43:23.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6fc9c2a:::benchmark_data_smithi188_141483_object64443:head# 2024-04-23T16:43:23.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6fe4b0e:::benchmark_data_smithi188_141483_object27490:head# 2024-04-23T16:43:23.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6feb09c:::benchmark_data_smithi188_141483_object62748:head# 2024-04-23T16:43:23.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c6ffae52:::benchmark_data_smithi188_141483_object87637:head# 2024-04-23T16:43:23.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7000c8d:::benchmark_data_smithi188_141483_object41588:head# 2024-04-23T16:43:23.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7001b79:::benchmark_data_smithi188_141483_object69546:head# 2024-04-23T16:43:23.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7011fa5:::benchmark_data_smithi188_141483_object53491:head# 2024-04-23T16:43:23.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c70136e6:::benchmark_data_smithi188_141483_object70283:head# 2024-04-23T16:43:23.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c70175a4:::benchmark_data_smithi188_141483_object1886:head# 2024-04-23T16:43:23.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7017d92:::benchmark_data_smithi188_141483_object24275:head# 2024-04-23T16:43:23.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c701e69e:::benchmark_data_smithi188_141483_object82677:head# 2024-04-23T16:43:23.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c702695d:::benchmark_data_smithi188_141483_object19071:head# 2024-04-23T16:43:23.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c703553f:::benchmark_data_smithi188_141483_object8991:head# 2024-04-23T16:43:23.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7040ad4:::benchmark_data_smithi188_141483_object14845:head# 2024-04-23T16:43:23.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c704bcfe:::benchmark_data_smithi188_141483_object4287:head# 2024-04-23T16:43:23.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7067a88:::benchmark_data_smithi188_141483_object26835:head# 2024-04-23T16:43:23.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c706b0db:::benchmark_data_smithi188_141483_object20970:head# 2024-04-23T16:43:23.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c70709f3:::benchmark_data_smithi188_141483_object63724:head# 2024-04-23T16:43:23.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c707e918:::benchmark_data_smithi188_141483_object41855:head# 2024-04-23T16:43:23.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7093d9d:::benchmark_data_smithi188_141483_object40619:head# 2024-04-23T16:43:23.857 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:43:23.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c709e961:::benchmark_data_smithi188_141483_object27982:head# 2024-04-23T16:43:23.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c70a340c:::benchmark_data_smithi188_141483_object6006:head# 2024-04-23T16:43:23.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c70a369b:::benchmark_data_smithi188_141483_object26798:head# 2024-04-23T16:43:23.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c70ba8bf:::benchmark_data_smithi188_141483_object48844:head# 2024-04-23T16:43:23.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c70c2752:::benchmark_data_smithi188_141483_object84008:head# 2024-04-23T16:43:23.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c70c4658:::benchmark_data_smithi188_141483_object50165:head# 2024-04-23T16:43:23.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c70ce765:::benchmark_data_smithi188_141483_object32965:head# 2024-04-23T16:43:23.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c70d185f:::benchmark_data_smithi188_141483_object75066:head# 2024-04-23T16:43:23.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c70eabe1:::benchmark_data_smithi188_141483_object71312:head# 2024-04-23T16:43:23.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c70f71f0:::benchmark_data_smithi188_141483_object39925:head# 2024-04-23T16:43:23.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c70fc798:::benchmark_data_smithi188_141483_object43154:head# 2024-04-23T16:43:23.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c710ed58:::benchmark_data_smithi188_141483_object70662:head# 2024-04-23T16:43:23.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c712fc53:::benchmark_data_smithi188_141483_object83815:head# 2024-04-23T16:43:23.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7147172:::benchmark_data_smithi188_141483_object55848:head# 2024-04-23T16:43:23.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c714cff0:::benchmark_data_smithi188_141483_object25831:head# 2024-04-23T16:43:23.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c71538ae:::benchmark_data_smithi188_141483_object5760:head# 2024-04-23T16:43:23.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c715695d:::benchmark_data_smithi188_141483_object80141:head# 2024-04-23T16:43:23.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c715aac9:::benchmark_data_smithi188_141483_object35157:head# 2024-04-23T16:43:23.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c715ee4a:::benchmark_data_smithi188_141483_object39829:head# 2024-04-23T16:43:23.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c71647e2:::benchmark_data_smithi188_141483_object87199:head# 2024-04-23T16:43:23.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c718180e:::benchmark_data_smithi188_141483_object18542:head# 2024-04-23T16:43:23.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7184b00:::benchmark_data_smithi188_141483_object67838:head# 2024-04-23T16:43:23.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c71989be:::benchmark_data_smithi188_141483_object36538:head# 2024-04-23T16:43:23.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c71aa744:::benchmark_data_smithi188_141483_object45728:head# 2024-04-23T16:43:23.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c71acaa6:::benchmark_data_smithi188_141483_object88984:head# 2024-04-23T16:43:23.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c71b4b0b:::benchmark_data_smithi188_141483_object80928:head# 2024-04-23T16:43:23.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c71bcb6c:::benchmark_data_smithi188_141483_object48105:head# 2024-04-23T16:43:23.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c71c6ff8:::benchmark_data_smithi188_141483_object22715:head# 2024-04-23T16:43:23.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c71ceea0:::benchmark_data_smithi188_141483_object43148:head# 2024-04-23T16:43:23.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c71d557f:::benchmark_data_smithi188_141483_object81479:head# 2024-04-23T16:43:23.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c71d6f32:::benchmark_data_smithi188_141483_object16221:head# 2024-04-23T16:43:23.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c71e35b4:::benchmark_data_smithi188_141483_object6950:head# 2024-04-23T16:43:23.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c71e472c:::benchmark_data_smithi188_141483_object71732:head# 2024-04-23T16:43:23.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c71e6995:::benchmark_data_smithi188_141483_object6321:head# 2024-04-23T16:43:23.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c71f0d4c:::benchmark_data_smithi188_141483_object34258:head# 2024-04-23T16:43:23.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c71feac8:::benchmark_data_smithi188_141483_object17203:head# 2024-04-23T16:43:23.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7209155:::benchmark_data_smithi188_141483_object2113:head# 2024-04-23T16:43:23.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7209982:::benchmark_data_smithi188_141483_object1923:head# 2024-04-23T16:43:23.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c720b182:::benchmark_data_smithi188_141483_object66680:head# 2024-04-23T16:43:23.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c720ebb4:::benchmark_data_smithi188_141483_object67692:head# 2024-04-23T16:43:23.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c72182b9:::benchmark_data_smithi188_141483_object63952:head# 2024-04-23T16:43:23.876 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7223e38:::benchmark_data_smithi188_141483_object47584:head# 2024-04-23T16:43:23.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7240b6d:::benchmark_data_smithi188_141483_object86474:head# 2024-04-23T16:43:23.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7256797:::benchmark_data_smithi188_141483_object88926:head# 2024-04-23T16:43:23.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7258357:::benchmark_data_smithi188_141483_object40769:head# 2024-04-23T16:43:23.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7259f3b:::benchmark_data_smithi188_141483_object86147:head# 2024-04-23T16:43:23.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c725ebfc:::benchmark_data_smithi188_141483_object2838:head# 2024-04-23T16:43:23.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c726197f:::benchmark_data_smithi188_141483_object65371:head# 2024-04-23T16:43:23.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7273ff3:::benchmark_data_smithi188_141483_object22574:head# 2024-04-23T16:43:23.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c727e9e4:::benchmark_data_smithi188_141483_object66820:head# 2024-04-23T16:43:23.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7288544:::benchmark_data_smithi188_141483_object14949:head# 2024-04-23T16:43:23.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c72cf9cc:::benchmark_data_smithi188_141483_object73510:head# 2024-04-23T16:43:23.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c72dc52d:::benchmark_data_smithi188_141483_object40870:head# 2024-04-23T16:43:23.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c72ea501:::benchmark_data_smithi188_141483_object89782:head# 2024-04-23T16:43:23.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c72faec6:::benchmark_data_smithi188_141483_object47533:head# 2024-04-23T16:43:23.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c72ff30d:::benchmark_data_smithi188_141483_object3236:head# 2024-04-23T16:43:23.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c730a27f:::benchmark_data_smithi188_141483_object69738:head# 2024-04-23T16:43:23.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c73227c4:::benchmark_data_smithi188_141483_object88368:head# 2024-04-23T16:43:23.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7325d47:::benchmark_data_smithi188_141483_object47641:head# 2024-04-23T16:43:23.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7346b15:::benchmark_data_smithi188_141483_object46184:head# 2024-04-23T16:43:23.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c734d5b5:::benchmark_data_smithi188_141483_object20677:head# 2024-04-23T16:43:23.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7363004:::benchmark_data_smithi188_141483_object66359:head# 2024-04-23T16:43:23.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7365699:::benchmark_data_smithi188_141483_object53467:head# 2024-04-23T16:43:23.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c738668e:::benchmark_data_smithi188_141483_object87860:head# 2024-04-23T16:43:23.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c738d9df:::benchmark_data_smithi188_141483_object44511:head# 2024-04-23T16:43:23.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7396b7c:::benchmark_data_smithi188_141483_object27503:head# 2024-04-23T16:43:23.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c73ae625:::benchmark_data_smithi188_141483_object78747:head# 2024-04-23T16:43:23.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c73af15f:::benchmark_data_smithi188_141483_object31424:head# 2024-04-23T16:43:23.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c73c26c2:::benchmark_data_smithi188_141483_object44373:head# 2024-04-23T16:43:23.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c73ce2a1:::benchmark_data_smithi188_141483_object13958:head# 2024-04-23T16:43:23.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c73e5a98:::benchmark_data_smithi188_141483_object55893:head# 2024-04-23T16:43:23.891 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c73f3bd5:::benchmark_data_smithi188_141483_object39237:head# 2024-04-23T16:43:23.891 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c73fabf8:::benchmark_data_smithi188_141483_object31525:head# 2024-04-23T16:43:23.892 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7407793:::benchmark_data_smithi188_141483_object85195:head# 2024-04-23T16:43:23.892 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7418539:::benchmark_data_smithi188_141483_object26455:head# 2024-04-23T16:43:23.892 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c74295a1:::benchmark_data_smithi188_141483_object32324:head# 2024-04-23T16:43:23.893 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c743160b:::benchmark_data_smithi188_141483_object74578:head# 2024-04-23T16:43:23.893 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7431d47:::benchmark_data_smithi188_141483_object17186:head# 2024-04-23T16:43:23.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c743b64c:::benchmark_data_smithi188_141483_object21254:head# 2024-04-23T16:43:23.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7443188:::benchmark_data_smithi188_141483_object42673:head# 2024-04-23T16:43:23.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7446ffd:::benchmark_data_smithi188_141483_object74961:head# 2024-04-23T16:43:23.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7448baa:::benchmark_data_smithi188_141483_object60840:head# 2024-04-23T16:43:23.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7474c94:::benchmark_data_smithi188_141483_object83155:head# 2024-04-23T16:43:23.896 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7475a13:::benchmark_data_smithi188_141483_object5080:head# 2024-04-23T16:43:23.896 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c74858ee:::benchmark_data_smithi188_141483_object47324:head# 2024-04-23T16:43:23.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c748a296:::benchmark_data_smithi188_141483_object25001:head# 2024-04-23T16:43:23.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c74a3e45:::benchmark_data_smithi188_141483_object20538:head# 2024-04-23T16:43:23.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c74ab857:::benchmark_data_smithi188_141483_object39954:head# 2024-04-23T16:43:23.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c74af0e5:::benchmark_data_smithi188_141483_object40627:head# 2024-04-23T16:43:23.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c74b1539:::benchmark_data_smithi188_141483_object33019:head# 2024-04-23T16:43:23.899 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c74b1f9a:::benchmark_data_smithi188_141483_object71767:head# 2024-04-23T16:43:23.899 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c74b3a39:::benchmark_data_smithi188_141483_object87248:head# 2024-04-23T16:43:23.900 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c74c49fc:::benchmark_data_smithi188_141483_object45269:head# 2024-04-23T16:43:23.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c74c894e:::benchmark_data_smithi188_141483_object86299:head# 2024-04-23T16:43:23.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c74dff3c:::benchmark_data_smithi188_141483_object74115:head# 2024-04-23T16:43:23.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c74fabf4:::benchmark_data_smithi188_141483_object1868:head# 2024-04-23T16:43:23.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7504372:::benchmark_data_smithi188_141483_object4488:head# 2024-04-23T16:43:23.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c75075fc:::benchmark_data_smithi188_141483_object38750:head# 2024-04-23T16:43:23.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c750bcab:::benchmark_data_smithi188_141483_object59772:head# 2024-04-23T16:43:23.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7518c53:::benchmark_data_smithi188_141483_object37776:head# 2024-04-23T16:43:23.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c751de20:::benchmark_data_smithi188_141483_object72090:head# 2024-04-23T16:43:23.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7522314:::benchmark_data_smithi188_141483_object10387:head# 2024-04-23T16:43:23.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7532edd:::benchmark_data_smithi188_141483_object61349:head# 2024-04-23T16:43:23.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7534a48:::benchmark_data_smithi188_141483_object82585:head# 2024-04-23T16:43:23.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c754d403:::benchmark_data_smithi188_141483_object68802:head# 2024-04-23T16:43:23.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c75500f2:::benchmark_data_smithi188_141483_object35895:head# 2024-04-23T16:43:23.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c756a1d1:::benchmark_data_smithi188_141483_object10516:head# 2024-04-23T16:43:23.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7572d0e:::benchmark_data_smithi188_141483_object40862:head# 2024-04-23T16:43:23.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c757f2e2:::benchmark_data_smithi188_141483_object82272:head# 2024-04-23T16:43:23.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c75829da:::benchmark_data_smithi188_141483_object49892:head# 2024-04-23T16:43:23.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c759dfe2:::benchmark_data_smithi188_141483_object33872:head# 2024-04-23T16:43:23.908 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:23 smithi148 ceph-mon[44520]: pgmap v1580: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 54 MiB/s wr, 869 op/s; 21574/201838 objects degraded (10.689%); 6.3 MiB/s, 100 objects/s recovering 2024-04-23T16:43:23.909 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:23 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:43:23.909 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:23 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3550752204' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:43:23.909 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:23 smithi148 ceph-mon[44520]: osdmap e567: 12 total, 11 up, 9 in 2024-04-23T16:43:23.909 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:23.910 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c75aeba5:::benchmark_data_smithi188_141483_object43064:head# 2024-04-23T16:43:23.910 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c75b9f77:::benchmark_data_smithi188_141483_object39234:head# 2024-04-23T16:43:23.910 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c75c7d2d:::benchmark_data_smithi188_141483_object56788:head# 2024-04-23T16:43:23.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c75dc036:::benchmark_data_smithi188_141483_object22321:head# 2024-04-23T16:43:23.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c75de009:::benchmark_data_smithi188_141483_object69932:head# 2024-04-23T16:43:23.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c75e12d8:::benchmark_data_smithi188_141483_object60635:head# 2024-04-23T16:43:23.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c76011aa:::benchmark_data_smithi188_141483_object56809:head# 2024-04-23T16:43:23.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c761da2f:::benchmark_data_smithi188_141483_object65689:head# 2024-04-23T16:43:23.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c763167d:::benchmark_data_smithi188_141483_object61943:head# 2024-04-23T16:43:23.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7632ded:::benchmark_data_smithi188_141483_object4702:head# 2024-04-23T16:43:23.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7634caa:::benchmark_data_smithi188_141483_object10085:head# 2024-04-23T16:43:23.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7643050:::benchmark_data_smithi188_141483_object2215:head# 2024-04-23T16:43:23.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7648662:::benchmark_data_smithi188_141483_object32982:head# 2024-04-23T16:43:23.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7649ac8:::benchmark_data_smithi188_141483_object60478:head# 2024-04-23T16:43:23.914 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c764a306:::benchmark_data_smithi188_141483_object2291:head# 2024-04-23T16:43:23.914 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c764b461:::benchmark_data_smithi188_141483_object13878:head# 2024-04-23T16:43:23.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c76506cc:::benchmark_data_smithi188_141483_object49428:head# 2024-04-23T16:43:23.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7661ab8:::benchmark_data_smithi188_141483_object69396:head# 2024-04-23T16:43:23.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7662155:::benchmark_data_smithi188_141483_object43009:head# 2024-04-23T16:43:23.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c766bd2b:::benchmark_data_smithi188_141483_object62745:head# 2024-04-23T16:43:23.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c76733e8:::benchmark_data_smithi188_141483_object85565:head# 2024-04-23T16:43:23.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c76769a6:::benchmark_data_smithi188_141483_object14146:head# 2024-04-23T16:43:23.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7679495:::benchmark_data_smithi188_141483_object55178:head# 2024-04-23T16:43:23.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c768155a:::benchmark_data_smithi188_141483_object9019:head# 2024-04-23T16:43:23.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7681b6e:::benchmark_data_smithi188_141483_object72210:head# 2024-04-23T16:43:23.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c768b50e:::benchmark_data_smithi188_141483_object33982:head# 2024-04-23T16:43:23.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c769a12f:::benchmark_data_smithi188_141483_object36938:head# 2024-04-23T16:43:23.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c769fdb2:::benchmark_data_smithi188_141483_object34163:head# 2024-04-23T16:43:23.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c76a8403:::benchmark_data_smithi188_141483_object5299:head# 2024-04-23T16:43:23.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c76af0d8:::benchmark_data_smithi188_141483_object2014:head# 2024-04-23T16:43:23.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c76c70c5:::benchmark_data_smithi188_141483_object84906:head# 2024-04-23T16:43:23.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c76cbb98:::benchmark_data_smithi188_141483_object30804:head# 2024-04-23T16:43:23.923 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c76d166b:::benchmark_data_smithi188_141483_object63627:head# 2024-04-23T16:43:23.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c76efd19:::benchmark_data_smithi188_141483_object46593:head# 2024-04-23T16:43:23.924 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c76f2ce0:::benchmark_data_smithi188_141483_object58207:head# 2024-04-23T16:43:23.925 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c76feabc:::benchmark_data_smithi188_141483_object3209:head# 2024-04-23T16:43:23.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c77026bf:::benchmark_data_smithi188_141483_object77386:head# 2024-04-23T16:43:23.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7704dd8:::benchmark_data_smithi188_141483_object77188:head# 2024-04-23T16:43:23.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7708272:::benchmark_data_smithi188_141483_object41641:head# 2024-04-23T16:43:23.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c771e562:::benchmark_data_smithi188_141483_object29261:head# 2024-04-23T16:43:23.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c774c243:::benchmark_data_smithi188_141483_object58256:head# 2024-04-23T16:43:23.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c774e3db:::benchmark_data_smithi188_141483_object29628:head# 2024-04-23T16:43:23.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c77629c0:::benchmark_data_smithi188_141483_object77776:head# 2024-04-23T16:43:23.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7763b82:::benchmark_data_smithi188_141483_object61973:head# 2024-04-23T16:43:23.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7769835:::benchmark_data_smithi188_141483_object39958:head# 2024-04-23T16:43:23.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c77702bd:::benchmark_data_smithi188_141483_object89575:head# 2024-04-23T16:43:23.930 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c77888e5:::benchmark_data_smithi188_141483_object6675:head# 2024-04-23T16:43:23.930 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7789240:::benchmark_data_smithi188_141483_object4983:head# 2024-04-23T16:43:23.930 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c778ac46:::benchmark_data_smithi188_141483_object5848:head# 2024-04-23T16:43:23.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c778faa2:::benchmark_data_smithi188_141483_object10884:head# 2024-04-23T16:43:23.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c779a828:::benchmark_data_smithi188_141483_object57222:head# 2024-04-23T16:43:23.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c779ea05:::benchmark_data_smithi188_141483_object84384:head# 2024-04-23T16:43:23.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c77a02d9:::benchmark_data_smithi188_141483_object28225:head# 2024-04-23T16:43:23.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c77b3c25:::benchmark_data_smithi188_141483_object47714:head# 2024-04-23T16:43:23.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c77b47ec:::benchmark_data_smithi188_141483_object12925:head# 2024-04-23T16:43:23.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c77b9eff:::benchmark_data_smithi188_141483_object43091:head# 2024-04-23T16:43:23.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c77c4954:::benchmark_data_smithi188_141483_object22370:head# 2024-04-23T16:43:23.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c77c8cf9:::benchmark_data_smithi188_141483_object84047:head# 2024-04-23T16:43:23.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c77d5e27:::benchmark_data_smithi188_141483_object15908:head# 2024-04-23T16:43:23.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c77d8445:::benchmark_data_smithi188_141483_object21981:head# 2024-04-23T16:43:23.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c77ddb1b:::benchmark_data_smithi188_141483_object81156:head# 2024-04-23T16:43:23.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c77ed632:::benchmark_data_smithi188_141483_object28052:head# 2024-04-23T16:43:23.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c77fbe29:::benchmark_data_smithi188_141483_object62284:head# 2024-04-23T16:43:23.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c780fab5:::benchmark_data_smithi188_141483_object45037:head# 2024-04-23T16:43:23.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7814854:::benchmark_data_smithi188_141483_object81979:head# 2024-04-23T16:43:23.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c781f629:::benchmark_data_smithi188_141483_object75136:head# 2024-04-23T16:43:23.939 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c782cdaf:::benchmark_data_smithi188_141483_object23240:head# 2024-04-23T16:43:23.939 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7850836:::benchmark_data_smithi188_141483_object89031:head# 2024-04-23T16:43:23.940 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7857286:::benchmark_data_smithi188_141483_object10667:head# 2024-04-23T16:43:23.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c785ab30:::benchmark_data_smithi188_141483_object74418:head# 2024-04-23T16:43:23.941 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7862589:::benchmark_data_smithi188_141483_object14762:head# 2024-04-23T16:43:23.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c78635ae:::benchmark_data_smithi188_141483_object7586:head# 2024-04-23T16:43:23.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c786d48d:::benchmark_data_smithi188_141483_object4238:head# 2024-04-23T16:43:23.943 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7879c1b:::benchmark_data_smithi188_141483_object45798:head# 2024-04-23T16:43:23.943 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c787b173:::benchmark_data_smithi188_141483_object12690:head# 2024-04-23T16:43:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c789ba1e:::benchmark_data_smithi188_141483_object12084:head# 2024-04-23T16:43:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c78a70ec:::benchmark_data_smithi188_141483_object59499:head# 2024-04-23T16:43:23.945 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c78a7fcc:::benchmark_data_smithi188_141483_object44717:head# 2024-04-23T16:43:23.946 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c78b1e2a:::benchmark_data_smithi188_141483_object27206:head# 2024-04-23T16:43:23.946 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c78b4de2:::benchmark_data_smithi188_141483_object33545:head# 2024-04-23T16:43:23.946 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c78c0d57:::benchmark_data_smithi188_141483_object16292:head# 2024-04-23T16:43:23.947 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c78d9d41:::benchmark_data_smithi188_141483_object56675:head# 2024-04-23T16:43:23.947 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c78fdaf8:::benchmark_data_smithi188_141483_object37869:head# 2024-04-23T16:43:23.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7935540:::benchmark_data_smithi188_141483_object20152:head# 2024-04-23T16:43:23.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7935ebd:::benchmark_data_smithi188_141483_object59593:head# 2024-04-23T16:43:23.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c79431d8:::benchmark_data_smithi188_141483_object63008:head# 2024-04-23T16:43:23.949 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7944e1f:::benchmark_data_smithi188_141483_object54103:head# 2024-04-23T16:43:23.949 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c794566a:::benchmark_data_smithi188_141483_object26762:head# 2024-04-23T16:43:23.950 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c794a24c:::benchmark_data_smithi188_141483_object27225:head# 2024-04-23T16:43:23.950 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7959840:::benchmark_data_smithi188_141483_object59767:head# 2024-04-23T16:43:23.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c796b753:::benchmark_data_smithi188_141483_object90352:head# 2024-04-23T16:43:23.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c798ed1c:::benchmark_data_smithi188_141483_object61167:head# 2024-04-23T16:43:23.953 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7993dd4:::benchmark_data_smithi188_141483_object51361:head# 2024-04-23T16:43:23.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c79943d5:::benchmark_data_smithi188_141483_object86982:head# 2024-04-23T16:43:23.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c79a2031:::benchmark_data_smithi188_141483_object25535:head# 2024-04-23T16:43:23.955 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c79b8b7f:::benchmark_data_smithi188_141483_object24631:head# 2024-04-23T16:43:23.955 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c79c4267:::benchmark_data_smithi188_141483_object16022:head# 2024-04-23T16:43:23.956 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c79c7d59:::benchmark_data_smithi188_141483_object88792:head# 2024-04-23T16:43:23.956 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a035db:::benchmark_data_smithi188_141483_object65042:head# 2024-04-23T16:43:23.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a08d54:::benchmark_data_smithi188_141483_object75361:head# 2024-04-23T16:43:23.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a17c82:::benchmark_data_smithi188_141483_object41510:head# 2024-04-23T16:43:23.958 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a1aae1:::benchmark_data_smithi188_141483_object2091:head# 2024-04-23T16:43:23.959 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a2671d:::benchmark_data_smithi188_141483_object41561:head# 2024-04-23T16:43:23.959 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a28bf6:::benchmark_data_smithi188_141483_object22283:head# 2024-04-23T16:43:23.959 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a30ab9:::benchmark_data_smithi188_141483_object71366:head# 2024-04-23T16:43:23.960 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a341a1:::benchmark_data_smithi188_141483_object77334:head# 2024-04-23T16:43:23.960 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a42b6e:::benchmark_data_smithi188_141483_object10304:head# 2024-04-23T16:43:23.960 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a4c5a9:::benchmark_data_smithi188_141483_object10551:head# 2024-04-23T16:43:23.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a598c2:::benchmark_data_smithi188_141483_object35956:head# 2024-04-23T16:43:23.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a5ed46:::benchmark_data_smithi188_141483_object35357:head# 2024-04-23T16:43:23.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a64fd1:::benchmark_data_smithi188_141483_object49042:head# 2024-04-23T16:43:23.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a6c60b:::benchmark_data_smithi188_141483_object24157:head# 2024-04-23T16:43:23.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a724de:::benchmark_data_smithi188_141483_object54477:head# 2024-04-23T16:43:23.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a7fb09:::benchmark_data_smithi188_141483_object60546:head# 2024-04-23T16:43:23.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a877eb:::benchmark_data_smithi188_141483_object66372:head# 2024-04-23T16:43:23.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a8fdc3:::benchmark_data_smithi188_141483_object28948:head# 2024-04-23T16:43:23.965 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:43:23.966 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a92fab:::benchmark_data_smithi188_141483_object64359:head# 2024-04-23T16:43:23.967 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a9e653:::benchmark_data_smithi188_141483_object67499:head# 2024-04-23T16:43:23.967 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7a9fc2c:::benchmark_data_smithi188_141483_object65277:head# 2024-04-23T16:43:23.967 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7aacbe0:::benchmark_data_smithi188_141483_object87161:head# 2024-04-23T16:43:23.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7ab9c89:::benchmark_data_smithi188_141483_object68864:head# 2024-04-23T16:43:23.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7ad771a:::benchmark_data_smithi188_141483_object54158:head# 2024-04-23T16:43:23.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7ad8150:::benchmark_data_smithi188_141483_object70456:head# 2024-04-23T16:43:23.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7ad830c:::benchmark_data_smithi188_141483_object85222:head# 2024-04-23T16:43:23.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7ae2d80:::benchmark_data_smithi188_141483_object11950:head# 2024-04-23T16:43:23.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7af0ed6:::benchmark_data_smithi188_141483_object20419:head# 2024-04-23T16:43:23.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7af86db:::benchmark_data_smithi188_141483_object52992:head# 2024-04-23T16:43:23.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7b03675:::benchmark_data_smithi188_141483_object18596:head# 2024-04-23T16:43:23.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7b04c03:::benchmark_data_smithi188_141483_object21390:head# 2024-04-23T16:43:23.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7b21cc7:::benchmark_data_smithi188_141483_object36588:head# 2024-04-23T16:43:23.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7b33d00:::benchmark_data_smithi188_141483_object84502:head# 2024-04-23T16:43:23.974 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7b346f4:::benchmark_data_smithi188_141483_object69222:head# 2024-04-23T16:43:23.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7b35a32:::benchmark_data_smithi188_141483_object25677:head# 2024-04-23T16:43:23.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7b414f7:::benchmark_data_smithi188_141483_object14293:head# 2024-04-23T16:43:23.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7b51029:::benchmark_data_smithi188_141483_object33001:head# 2024-04-23T16:43:23.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7b52187:::benchmark_data_smithi188_141483_object51787:head# 2024-04-23T16:43:23.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7b59f97:::benchmark_data_smithi188_141483_object17970:head# 2024-04-23T16:43:23.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7b6bdc3:::benchmark_data_smithi188_141483_object32204:head# 2024-04-23T16:43:23.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7b73331:::benchmark_data_smithi188_141483_object19040:head# 2024-04-23T16:43:23.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7b8d483:::benchmark_data_smithi188_141483_object63914:head# 2024-04-23T16:43:23.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7b8f9f1:::benchmark_data_smithi188_141483_object75989:head# 2024-04-23T16:43:23.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7ba2682:::benchmark_data_smithi188_141483_object2680:head# 2024-04-23T16:43:23.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7bb6e7e:::benchmark_data_smithi188_141483_object5387:head# 2024-04-23T16:43:23.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7bc1ec1:::benchmark_data_smithi188_141483_object3826:head# 2024-04-23T16:43:23.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7bc3cc1:::benchmark_data_smithi188_141483_object10891:head# 2024-04-23T16:43:23.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7bc6ca1:::benchmark_data_smithi188_141483_object36328:head# 2024-04-23T16:43:23.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:23.980 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7bcd278:::benchmark_data_smithi188_141483_object37668:head# 2024-04-23T16:43:23.980 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7bd6f05:::benchmark_data_smithi188_141483_object62203:head# 2024-04-23T16:43:23.980 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7bd7fbe:::benchmark_data_smithi188_141483_object47251:head# 2024-04-23T16:43:23.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7bd9009:::benchmark_data_smithi188_141483_object23032:head# 2024-04-23T16:43:23.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:43:23.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7bf7a42:::benchmark_data_smithi188_141483_object25148:head# 2024-04-23T16:43:23.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7bf90de:::benchmark_data_smithi188_141483_object69016:head# 2024-04-23T16:43:23.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7c02908:::benchmark_data_smithi188_141483_object5699:head# 2024-04-23T16:43:23.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7c06dcb:::benchmark_data_smithi188_141483_object14784:head# 2024-04-23T16:43:23.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7c08443:::benchmark_data_smithi188_141483_object59143:head# 2024-04-23T16:43:23.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7c105d3:::benchmark_data_smithi188_141483_object41129:head# 2024-04-23T16:43:23.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7c13e9a:::benchmark_data_smithi188_141483_object12344:head# 2024-04-23T16:43:23.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7c4a794:::benchmark_data_smithi188_141483_object67198:head# 2024-04-23T16:43:23.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7c72f93:::benchmark_data_smithi188_141483_object26332:head# 2024-04-23T16:43:23.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7c79f95:::benchmark_data_smithi188_141483_object19542:head# 2024-04-23T16:43:23.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7c830ca:::benchmark_data_smithi188_141483_object90751:head# 2024-04-23T16:43:23.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7c8f5fd:::benchmark_data_smithi188_141483_object75163:head# 2024-04-23T16:43:23.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7c8ff79:::benchmark_data_smithi188_141483_object28953:head# 2024-04-23T16:43:23.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7c9bc00:::benchmark_data_smithi188_141483_object19364:head# 2024-04-23T16:43:23.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7c9f559:::benchmark_data_smithi188_141483_object64603:head# 2024-04-23T16:43:23.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7ca3a87:::benchmark_data_smithi188_141483_object75443:head# 2024-04-23T16:43:23.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7ca8b1d:::benchmark_data_smithi188_141483_object38619:head# 2024-04-23T16:43:23.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7cb528e:::benchmark_data_smithi188_141483_object7969:head# 2024-04-23T16:43:23.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7cb61b5:::benchmark_data_smithi188_141483_object16972:head# 2024-04-23T16:43:23.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7cbd3b9:::benchmark_data_smithi188_141483_object1217:head# 2024-04-23T16:43:23.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7cd1d9b:::benchmark_data_smithi188_141483_object34802:head# 2024-04-23T16:43:23.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7cd8ce7:::benchmark_data_smithi188_141483_object39496:head# 2024-04-23T16:43:23.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7ce0dbe:::benchmark_data_smithi188_141483_object21405:head# 2024-04-23T16:43:23.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7ce5509:::benchmark_data_smithi188_141483_object56255:head# 2024-04-23T16:43:23.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7cfcd48:::benchmark_data_smithi188_141483_object62240:head# 2024-04-23T16:43:23.997 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7cff05d:::benchmark_data_smithi188_141483_object45070:head# 2024-04-23T16:43:23.997 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7d0d116:::benchmark_data_smithi188_141483_object84254:head# 2024-04-23T16:43:23.998 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7d0f1aa:::benchmark_data_smithi188_141483_object28564:head# 2024-04-23T16:43:23.998 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7d1359e:::benchmark_data_smithi188_141483_object55177:head# 2024-04-23T16:43:23.999 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7d17501:::benchmark_data_smithi188_141483_object81957:head# 2024-04-23T16:43:24.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7d1a86e:::benchmark_data_smithi188_141483_object5596:head# 2024-04-23T16:43:24.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7d201aa:::benchmark_data_smithi188_141483_object22259:head# 2024-04-23T16:43:24.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7d4134d:::benchmark_data_smithi188_141483_object32529:head# 2024-04-23T16:43:24.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7d4d7be:::benchmark_data_smithi188_141483_object75367:head# 2024-04-23T16:43:24.002 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7d51535:::benchmark_data_smithi188_141483_object67594:head# 2024-04-23T16:43:24.002 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7d5302b:::benchmark_data_smithi188_141483_object41999:head# 2024-04-23T16:43:24.002 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7d5f9d4:::benchmark_data_smithi188_141483_object26487:head# 2024-04-23T16:43:24.002 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7d8a2e2:::benchmark_data_smithi188_141483_object83607:head# 2024-04-23T16:43:24.003 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7d9a729:::benchmark_data_smithi188_141483_object19746:head# 2024-04-23T16:43:24.003 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7dd368c:::benchmark_data_smithi188_141483_object68867:head# 2024-04-23T16:43:24.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7de4d65:::benchmark_data_smithi188_141483_object67064:head# 2024-04-23T16:43:24.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7df7d8d:::benchmark_data_smithi188_141483_object2380:head# 2024-04-23T16:43:24.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7df8867:::benchmark_data_smithi188_141483_object31646:head# 2024-04-23T16:43:24.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7dfeee5:::benchmark_data_smithi188_141483_object5355:head# 2024-04-23T16:43:24.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7e0ee36:::benchmark_data_smithi188_141483_object84679:head# 2024-04-23T16:43:24.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7e110e7:::benchmark_data_smithi188_141483_object14913:head# 2024-04-23T16:43:24.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7e14ecc:::benchmark_data_smithi188_141483_object89843:head# 2024-04-23T16:43:24.007 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7e24d42:::benchmark_data_smithi188_141483_object79979:head# 2024-04-23T16:43:24.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7e26aef:::benchmark_data_smithi188_141483_object74068:head# 2024-04-23T16:43:24.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7e3106c:::benchmark_data_smithi188_141483_object14944:head# 2024-04-23T16:43:24.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7e33b45:::benchmark_data_smithi188_141483_object87375:head# 2024-04-23T16:43:24.008 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7e35803:::benchmark_data_smithi188_141483_object10476:head# 2024-04-23T16:43:24.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7e38a80:::benchmark_data_smithi188_141483_object28902:head# 2024-04-23T16:43:24.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7e4b1a0:::benchmark_data_smithi188_141483_object58151:head# 2024-04-23T16:43:24.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7e5b250:::benchmark_data_smithi188_141483_object46144:head# 2024-04-23T16:43:24.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7e5dfbd:::benchmark_data_smithi188_141483_object53603:head# 2024-04-23T16:43:24.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7e5f139:::benchmark_data_smithi188_141483_object71128:head# 2024-04-23T16:43:24.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7e74c03:::benchmark_data_smithi188_141483_object68937:head# 2024-04-23T16:43:24.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7e78adf:::benchmark_data_smithi188_141483_object28494:head# 2024-04-23T16:43:24.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7e7c901:::benchmark_data_smithi188_141483_object85248:head# 2024-04-23T16:43:24.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7e7f3b0:::benchmark_data_smithi188_141483_object19530:head# 2024-04-23T16:43:24.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7e8ac97:::benchmark_data_smithi188_141483_object31772:head# 2024-04-23T16:43:24.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7ea46af:::benchmark_data_smithi188_141483_object78570:head# 2024-04-23T16:43:24.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7eb6551:::benchmark_data_smithi188_141483_object16107:head# 2024-04-23T16:43:24.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7ec1e99:::benchmark_data_smithi188_141483_object49673:head# 2024-04-23T16:43:24.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7ecf04e:::benchmark_data_smithi188_141483_object90353:head# 2024-04-23T16:43:24.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7edcae3:::benchmark_data_smithi188_141483_object86170:head# 2024-04-23T16:43:24.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7edd1a2:::benchmark_data_smithi188_141483_object17656:head# 2024-04-23T16:43:24.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7ef52b1:::benchmark_data_smithi188_141483_object37964:head# 2024-04-23T16:43:24.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7efb3b8:::benchmark_data_smithi188_141483_object4880:head# 2024-04-23T16:43:24.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7efbf1a:::benchmark_data_smithi188_141483_object78523:head# 2024-04-23T16:43:24.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7efcbd8:::benchmark_data_smithi188_141483_object2748:head# 2024-04-23T16:43:24.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7efdfa6:::benchmark_data_smithi188_141483_object38717:head# 2024-04-23T16:43:24.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7f042b7:::benchmark_data_smithi188_141483_object30793:head# 2024-04-23T16:43:24.020 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7f05978:::benchmark_data_smithi188_141483_object90890:head# 2024-04-23T16:43:24.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7f0ccfb:::benchmark_data_smithi188_141483_object80020:head# 2024-04-23T16:43:24.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7f39481:::benchmark_data_smithi188_141483_object62895:head# 2024-04-23T16:43:24.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7f40bd3:::benchmark_data_smithi188_141483_object33484:head# 2024-04-23T16:43:24.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7f43568:::benchmark_data_smithi188_141483_object78061:head# 2024-04-23T16:43:24.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7f518cf:::benchmark_data_smithi188_141483_object85517:head# 2024-04-23T16:43:24.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7f58e81:::benchmark_data_smithi188_141483_object34246:head# 2024-04-23T16:43:24.022 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7f6f7f0:::benchmark_data_smithi188_141483_object83233:head# 2024-04-23T16:43:24.022 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7f79f5f:::benchmark_data_smithi188_141483_object53912:head# 2024-04-23T16:43:24.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7f7d127:::benchmark_data_smithi188_141483_object45867:head# 2024-04-23T16:43:24.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7f87454:::benchmark_data_smithi188_141483_object65685:head# 2024-04-23T16:43:24.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7fa6f13:::benchmark_data_smithi188_141483_object10862:head# 2024-04-23T16:43:24.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7fa8586:::benchmark_data_smithi188_141483_object58122:head# 2024-04-23T16:43:24.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7fab197:::benchmark_data_smithi188_141483_object57441:head# 2024-04-23T16:43:24.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7fb7e3a:::benchmark_data_smithi188_141483_object80647:head# 2024-04-23T16:43:24.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7fb81a6:::benchmark_data_smithi188_141483_object27038:head# 2024-04-23T16:43:24.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7fc6ad6:::benchmark_data_smithi188_141483_object52889:head# 2024-04-23T16:43:24.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7fcbffd:::benchmark_data_smithi188_141483_object3670:head# 2024-04-23T16:43:24.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c7fe98a2:::benchmark_data_smithi188_141483_object13247:head# 2024-04-23T16:43:24.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c800b8dd:::benchmark_data_smithi188_141483_object7252:head# 2024-04-23T16:43:24.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c802ec7e:::benchmark_data_smithi188_141483_object40029:head# 2024-04-23T16:43:24.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c80321d8:::benchmark_data_smithi188_141483_object86372:head# 2024-04-23T16:43:24.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8032c59:::benchmark_data_smithi188_141483_object34873:head# 2024-04-23T16:43:24.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c804f7af:::benchmark_data_smithi188_141483_object81484:head# 2024-04-23T16:43:24.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8059fa7:::benchmark_data_smithi188_141483_object47890:head# 2024-04-23T16:43:24.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8064f3c:::benchmark_data_smithi188_141483_object73395:head# 2024-04-23T16:43:24.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c806568e:::benchmark_data_smithi188_141483_object34475:head# 2024-04-23T16:43:24.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c807654f:::benchmark_data_smithi188_141483_object84225:head# 2024-04-23T16:43:24.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c807c529:::benchmark_data_smithi188_141483_object50205:head# 2024-04-23T16:43:24.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8083f78:::benchmark_data_smithi188_141483_object62709:head# 2024-04-23T16:43:24.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8084fe5:::benchmark_data_smithi188_141483_object80854:head# 2024-04-23T16:43:24.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8086fda:::benchmark_data_smithi188_141483_object49122:head# 2024-04-23T16:43:24.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8087470:::benchmark_data_smithi188_141483_object3073:head# 2024-04-23T16:43:24.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c80940a5:::benchmark_data_smithi188_141483_object51414:head# 2024-04-23T16:43:24.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c80981cb:::benchmark_data_smithi188_141483_object50376:head# 2024-04-23T16:43:24.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c809f3bc:::benchmark_data_smithi188_141483_object55589:head# 2024-04-23T16:43:24.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c80aa60f:::benchmark_data_smithi188_141483_object34147:head# 2024-04-23T16:43:24.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c80bdac1:::benchmark_data_smithi188_141483_object35493:head# 2024-04-23T16:43:24.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c80bfbdd:::benchmark_data_smithi188_141483_object73250:head# 2024-04-23T16:43:24.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c80ca5d1:::benchmark_data_smithi188_141483_object59665:head# 2024-04-23T16:43:24.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c80dcae9:::benchmark_data_smithi188_141483_object16244:head# 2024-04-23T16:43:24.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c80dcfd7:::benchmark_data_smithi188_141483_object14909:head# 2024-04-23T16:43:24.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c80f8beb:::benchmark_data_smithi188_141483_object26374:head# 2024-04-23T16:43:24.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c80fab79:::benchmark_data_smithi188_141483_object57763:head# 2024-04-23T16:43:24.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c80fd62b:::benchmark_data_smithi188_141483_object30689:head# 2024-04-23T16:43:24.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c81038c9:::benchmark_data_smithi188_141483_object6953:head# 2024-04-23T16:43:24.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8113a34:::benchmark_data_smithi188_141483_object4832:head# 2024-04-23T16:43:24.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8114f9c:::benchmark_data_smithi188_141483_object88298:head# 2024-04-23T16:43:24.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c811d2cb:::benchmark_data_smithi188_141483_object6506:head# 2024-04-23T16:43:24.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8127e70:::benchmark_data_smithi188_141483_object30989:head# 2024-04-23T16:43:24.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c812f69c:::benchmark_data_smithi188_141483_object23685:head# 2024-04-23T16:43:24.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c814c9f6:::benchmark_data_smithi188_141483_object79856:head# 2024-04-23T16:43:24.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c81557fb:::benchmark_data_smithi188_141483_object70726:head# 2024-04-23T16:43:24.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c81564a5:::benchmark_data_smithi188_141483_object46755:head# 2024-04-23T16:43:24.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8159de4:::benchmark_data_smithi188_141483_object67290:head# 2024-04-23T16:43:24.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8160143:::benchmark_data_smithi188_141483_object8030:head# 2024-04-23T16:43:24.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8163204:::benchmark_data_smithi188_141483_object7774:head# 2024-04-23T16:43:24.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c816c344:::benchmark_data_smithi188_141483_object11921:head# 2024-04-23T16:43:24.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c816ec31:::benchmark_data_smithi188_141483_object67524:head# 2024-04-23T16:43:24.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c817c137:::benchmark_data_smithi188_141483_object26884:head# 2024-04-23T16:43:24.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c817f692:::benchmark_data_smithi188_141483_object55342:head# 2024-04-23T16:43:24.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8194d88:::benchmark_data_smithi188_141483_object17419:head# 2024-04-23T16:43:24.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8195ef6:::benchmark_data_smithi188_141483_object67407:head# 2024-04-23T16:43:24.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c819ce09:::benchmark_data_smithi188_141483_object61280:head# 2024-04-23T16:43:24.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c81ab53e:::benchmark_data_smithi188_141483_object90230:head# 2024-04-23T16:43:24.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c81babf3:::benchmark_data_smithi188_141483_object85355:head# 2024-04-23T16:43:24.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c81c2ecb:::benchmark_data_smithi188_141483_object10262:head# 2024-04-23T16:43:24.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c81c7242:::benchmark_data_smithi188_141483_object79504:head# 2024-04-23T16:43:24.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c81cf7f3:::benchmark_data_smithi188_141483_object52325:head# 2024-04-23T16:43:24.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c81d132b:::benchmark_data_smithi188_141483_object15298:head# 2024-04-23T16:43:24.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c81df3b3:::benchmark_data_smithi188_141483_object35154:head# 2024-04-23T16:43:24.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c81e8a30:::benchmark_data_smithi188_141483_object85501:head# 2024-04-23T16:43:24.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c820b4f2:::benchmark_data_smithi188_141483_object36672:head# 2024-04-23T16:43:24.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c82118d0:::benchmark_data_smithi188_141483_object89442:head# 2024-04-23T16:43:24.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8216bb7:::benchmark_data_smithi188_141483_object79272:head# 2024-04-23T16:43:24.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c822ca58:::benchmark_data_smithi188_141483_object71193:head# 2024-04-23T16:43:24.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8240e47:::benchmark_data_smithi188_141483_object61090:head# 2024-04-23T16:43:24.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8261fcb:::benchmark_data_smithi188_141483_object55220:head# 2024-04-23T16:43:24.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c826cd16:::benchmark_data_smithi188_141483_object78793:head# 2024-04-23T16:43:24.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c826e43b:::benchmark_data_smithi188_141483_object5696:head# 2024-04-23T16:43:24.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c82784c1:::benchmark_data_smithi188_141483_object85299:head# 2024-04-23T16:43:24.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c828373a:::benchmark_data_smithi188_141483_object90438:head# 2024-04-23T16:43:24.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c828ea27:::benchmark_data_smithi188_141483_object81621:head# 2024-04-23T16:43:24.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c829fb73:::benchmark_data_smithi188_141483_object87283:head# 2024-04-23T16:43:24.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c82a2683:::benchmark_data_smithi188_141483_object25356:head# 2024-04-23T16:43:24.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c82a534b:::benchmark_data_smithi188_141483_object52489:head# 2024-04-23T16:43:24.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c82b9271:::benchmark_data_smithi188_141483_object4686:head# 2024-04-23T16:43:24.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c82c9c2d:::benchmark_data_smithi188_141483_object70914:head# 2024-04-23T16:43:24.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c82caa74:::benchmark_data_smithi188_141483_object55036:head# 2024-04-23T16:43:24.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c82cd0d0:::benchmark_data_smithi188_141483_object86645:head# 2024-04-23T16:43:24.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c82ce4cf:::benchmark_data_smithi188_141483_object47262:head# 2024-04-23T16:43:24.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c82e4393:::benchmark_data_smithi188_141483_object90344:head# 2024-04-23T16:43:24.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c83005a2:::benchmark_data_smithi188_141483_object88080:head# 2024-04-23T16:43:24.065 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8314294:::benchmark_data_smithi188_141483_object82366:head# 2024-04-23T16:43:24.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c83267fb:::benchmark_data_smithi188_141483_object14748:head# 2024-04-23T16:43:24.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c83582ab:::benchmark_data_smithi188_141483_object29395:head# 2024-04-23T16:43:24.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c835b411:::benchmark_data_smithi188_141483_object73769:head# 2024-04-23T16:43:24.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c838ea82:::benchmark_data_smithi188_141483_object27528:head# 2024-04-23T16:43:24.067 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c83b55c8:::benchmark_data_smithi188_141483_object17368:head# 2024-04-23T16:43:24.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c83bb0d8:::benchmark_data_smithi188_141483_object29697:head# 2024-04-23T16:43:24.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c83d8011:::benchmark_data_smithi188_141483_object46018:head# 2024-04-23T16:43:24.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c83e7290:::benchmark_data_smithi188_141483_object40441:head# 2024-04-23T16:43:24.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8408d23:::benchmark_data_smithi188_141483_object198:head# 2024-04-23T16:43:24.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c840c300:::benchmark_data_smithi188_141483_object78512:head# 2024-04-23T16:43:24.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c842a76d:::benchmark_data_smithi188_141483_object25562:head# 2024-04-23T16:43:24.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c84369a5:::benchmark_data_smithi188_141483_object83320:head# 2024-04-23T16:43:24.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c84405c1:::benchmark_data_smithi188_141483_object84552:head# 2024-04-23T16:43:24.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8444d7e:::benchmark_data_smithi188_141483_object70503:head# 2024-04-23T16:43:24.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8464f15:::benchmark_data_smithi188_141483_object87428:head# 2024-04-23T16:43:24.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c847a971:::benchmark_data_smithi188_141483_object39707:head# 2024-04-23T16:43:24.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c848e27b:::benchmark_data_smithi188_141483_object81838:head# 2024-04-23T16:43:24.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c84a9ab0:::benchmark_data_smithi188_141483_object70089:head# 2024-04-23T16:43:24.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c84ae123:::benchmark_data_smithi188_141483_object10567:head# 2024-04-23T16:43:24.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c84c4fba:::benchmark_data_smithi188_141483_object73889:head# 2024-04-23T16:43:24.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c84c62fc:::benchmark_data_smithi188_141483_object47358:head# 2024-04-23T16:43:24.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c84cf37d:::benchmark_data_smithi188_141483_object78746:head# 2024-04-23T16:43:24.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c84e6467:::benchmark_data_smithi188_141483_object69744:head# 2024-04-23T16:43:24.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c84fa6fd:::benchmark_data_smithi188_141483_object45036:head# 2024-04-23T16:43:24.077 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c84fd906:::benchmark_data_smithi188_141483_object69168:head# 2024-04-23T16:43:24.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c84fe8cc:::benchmark_data_smithi188_141483_object34670:head# 2024-04-23T16:43:24.078 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8506a59:::benchmark_data_smithi188_141483_object55505:head# 2024-04-23T16:43:24.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c850b74a:::benchmark_data_smithi188_141483_object73372:head# 2024-04-23T16:43:24.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8516f83:::benchmark_data_smithi188_141483_object57909:head# 2024-04-23T16:43:24.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c851fe02:::benchmark_data_smithi188_141483_object58146:head# 2024-04-23T16:43:24.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8524bdd:::benchmark_data_smithi188_141483_object9413:head# 2024-04-23T16:43:24.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c85268da:::benchmark_data_smithi188_141483_object52450:head# 2024-04-23T16:43:24.081 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:43:24.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8526efb:::benchmark_data_smithi188_141483_object27205:head# 2024-04-23T16:43:24.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8527109:::benchmark_data_smithi188_141483_object67265:head# 2024-04-23T16:43:24.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c853483d:::benchmark_data_smithi188_141483_object65197:head# 2024-04-23T16:43:24.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c85424ff:::benchmark_data_smithi188_141483_object51520:head# 2024-04-23T16:43:24.083 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c85452f7:::benchmark_data_smithi188_141483_object11445:head# 2024-04-23T16:43:24.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c855890b:::benchmark_data_smithi188_141483_object36219:head# 2024-04-23T16:43:24.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8563d4b:::benchmark_data_smithi188_141483_object55207:head# 2024-04-23T16:43:24.084 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c85705df:::benchmark_data_smithi188_141483_object84459:head# 2024-04-23T16:43:24.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c857f573:::benchmark_data_smithi188_141483_object24406:head# 2024-04-23T16:43:24.085 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c859b47e:::benchmark_data_smithi188_141483_object1153:head# 2024-04-23T16:43:24.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c85aab0d:::benchmark_data_smithi188_141483_object14157:head# 2024-04-23T16:43:24.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c85ae457:::benchmark_data_smithi188_141483_object25122:head# 2024-04-23T16:43:24.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c85c3ed5:::benchmark_data_smithi188_141483_object1208:head# 2024-04-23T16:43:24.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c85d7a42:::benchmark_data_smithi188_141483_object62104:head# 2024-04-23T16:43:24.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c85d7bf7:::benchmark_data_smithi188_141483_object49889:head# 2024-04-23T16:43:24.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c85e5027:::benchmark_data_smithi188_141483_object56259:head# 2024-04-23T16:43:24.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8611204:::benchmark_data_smithi188_141483_object49040:head# 2024-04-23T16:43:24.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8612813:::benchmark_data_smithi188_141483_object16962:head# 2024-04-23T16:43:24.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8612b60:::benchmark_data_smithi188_141483_object22861:head# 2024-04-23T16:43:24.131 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:24.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8646559:::benchmark_data_smithi188_141483_object16893:head# 2024-04-23T16:43:24.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c865805b:::benchmark_data_smithi188_141483_object50776:head# 2024-04-23T16:43:24.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c867dbe1:::benchmark_data_smithi188_141483_object82221:head# 2024-04-23T16:43:24.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8681de3:::benchmark_data_smithi188_141483_object69574:head# 2024-04-23T16:43:24.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8687ca0:::benchmark_data_smithi188_141483_object78559:head# 2024-04-23T16:43:24.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8696f39:::benchmark_data_smithi188_141483_object31773:head# 2024-04-23T16:43:24.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c86976c6:::benchmark_data_smithi188_141483_object88296:head# 2024-04-23T16:43:24.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8698508:::benchmark_data_smithi188_141483_object9238:head# 2024-04-23T16:43:24.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c86a3658:::benchmark_data_smithi188_141483_object21633:head# 2024-04-23T16:43:24.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c86b9a55:::benchmark_data_smithi188_141483_object71309:head# 2024-04-23T16:43:24.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c86c052f:::benchmark_data_smithi188_141483_object7904:head# 2024-04-23T16:43:24.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c86c4d05:::benchmark_data_smithi188_141483_object26874:head# 2024-04-23T16:43:24.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c86cb88d:::benchmark_data_smithi188_141483_object76437:head# 2024-04-23T16:43:24.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c86d8ec2:::benchmark_data_smithi188_141483_object78346:head# 2024-04-23T16:43:24.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c86dd760:::benchmark_data_smithi188_141483_object31756:head# 2024-04-23T16:43:24.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c86eb279:::benchmark_data_smithi188_141483_object27933:head# 2024-04-23T16:43:24.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c86ee14e:::benchmark_data_smithi188_141483_object74363:head# 2024-04-23T16:43:24.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c86f42ec:::benchmark_data_smithi188_141483_object46151:head# 2024-04-23T16:43:24.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8709dd2:::benchmark_data_smithi188_141483_object53437:head# 2024-04-23T16:43:24.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c870bb18:::benchmark_data_smithi188_141483_object39500:head# 2024-04-23T16:43:24.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c87389c7:::benchmark_data_smithi188_141483_object13739:head# 2024-04-23T16:43:24.140 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c874a1b4:::benchmark_data_smithi188_141483_object33859:head# 2024-04-23T16:43:24.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c874b71b:::benchmark_data_smithi188_141483_object56060:head# 2024-04-23T16:43:24.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c874d338:::benchmark_data_smithi188_141483_object84149:head# 2024-04-23T16:43:24.141 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c874f43a:::benchmark_data_smithi188_141483_object53251:head# 2024-04-23T16:43:24.142 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c876d707:::benchmark_data_smithi188_141483_object63877:head# 2024-04-23T16:43:24.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c877269f:::benchmark_data_smithi188_141483_object51390:head# 2024-04-23T16:43:24.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c877784f:::benchmark_data_smithi188_141483_object54160:head# 2024-04-23T16:43:24.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c87780b5:::benchmark_data_smithi188_141483_object3859:head# 2024-04-23T16:43:24.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c877867c:::benchmark_data_smithi188_141483_object15212:head# 2024-04-23T16:43:24.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8784b2e:::benchmark_data_smithi188_141483_object37289:head# 2024-04-23T16:43:24.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8787202:::benchmark_data_smithi188_141483_object2902:head# 2024-04-23T16:43:24.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8790ce7:::benchmark_data_smithi188_141483_object51054:head# 2024-04-23T16:43:24.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8796a75:::benchmark_data_smithi188_141483_object50409:head# 2024-04-23T16:43:24.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c879b1ba:::benchmark_data_smithi188_141483_object22854:head# 2024-04-23T16:43:24.147 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c879d609:::benchmark_data_smithi188_141483_object59414:head# 2024-04-23T16:43:24.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c87b7843:::benchmark_data_smithi188_141483_object80119:head# 2024-04-23T16:43:24.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c87c5580:::benchmark_data_smithi188_141483_object42698:head# 2024-04-23T16:43:24.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c87d8143:::benchmark_data_smithi188_141483_object74577:head# 2024-04-23T16:43:24.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c87e41ac:::benchmark_data_smithi188_141483_object44938:head# 2024-04-23T16:43:24.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c87fd055:::benchmark_data_smithi188_141483_object62845:head# 2024-04-23T16:43:24.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8800b0b:::benchmark_data_smithi188_141483_object22126:head# 2024-04-23T16:43:24.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c880f196:::benchmark_data_smithi188_141483_object79217:head# 2024-04-23T16:43:24.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c881588b:::benchmark_data_smithi188_141483_object84142:head# 2024-04-23T16:43:24.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c881cc0c:::benchmark_data_smithi188_141483_object41275:head# 2024-04-23T16:43:24.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c882a598:::benchmark_data_smithi188_141483_object27046:head# 2024-04-23T16:43:24.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c882e130:::benchmark_data_smithi188_141483_object3204:head# 2024-04-23T16:43:24.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c885249c:::benchmark_data_smithi188_141483_object51476:head# 2024-04-23T16:43:24.153 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c886932b:::benchmark_data_smithi188_141483_object29515:head# 2024-04-23T16:43:24.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c888a580:::benchmark_data_smithi188_141483_object42303:head# 2024-04-23T16:43:24.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c888ed7f:::benchmark_data_smithi188_141483_object13525:head# 2024-04-23T16:43:24.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8892cd1:::benchmark_data_smithi188_141483_object39881:head# 2024-04-23T16:43:24.155 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c889ea8e:::benchmark_data_smithi188_141483_object76567:head# 2024-04-23T16:43:24.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c88a3a9f:::benchmark_data_smithi188_141483_object67279:head# 2024-04-23T16:43:24.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c88a898e:::benchmark_data_smithi188_141483_object83234:head# 2024-04-23T16:43:24.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c88ab58b:::benchmark_data_smithi188_141483_object20724:head# 2024-04-23T16:43:24.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c88d37d6:::benchmark_data_smithi188_141483_object3078:head# 2024-04-23T16:43:24.158 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c88e9b37:::benchmark_data_smithi188_141483_object75003:head# 2024-04-23T16:43:24.158 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c88ec25a:::benchmark_data_smithi188_141483_object24676:head# 2024-04-23T16:43:24.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c88f709c:::benchmark_data_smithi188_141483_object88519:head# 2024-04-23T16:43:24.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8917022:::benchmark_data_smithi188_141483_object15591:head# 2024-04-23T16:43:24.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c89375df:::benchmark_data_smithi188_141483_object10950:head# 2024-04-23T16:43:24.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c89384b6:::benchmark_data_smithi188_141483_object20686:head# 2024-04-23T16:43:24.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c89447c6:::benchmark_data_smithi188_141483_object51192:head# 2024-04-23T16:43:24.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c894e060:::benchmark_data_smithi188_141483_object42625:head# 2024-04-23T16:43:24.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c894e947:::benchmark_data_smithi188_141483_object30643:head# 2024-04-23T16:43:24.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c89537e5:::benchmark_data_smithi188_141483_object90262:head# 2024-04-23T16:43:24.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8954c07:::benchmark_data_smithi188_141483_object71752:head# 2024-04-23T16:43:24.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8958722:::benchmark_data_smithi188_141483_object15440:head# 2024-04-23T16:43:24.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8959b75:::benchmark_data_smithi188_141483_object90032:head# 2024-04-23T16:43:24.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c897bcd9:::benchmark_data_smithi188_141483_object82261:head# 2024-04-23T16:43:24.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c89a7160:::benchmark_data_smithi188_141483_object7981:head# 2024-04-23T16:43:24.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c89adac8:::benchmark_data_smithi188_141483_object18413:head# 2024-04-23T16:43:24.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c89aea8c:::benchmark_data_smithi188_141483_object89783:head# 2024-04-23T16:43:24.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c89aecd4:::benchmark_data_smithi188_141483_object82966:head# 2024-04-23T16:43:24.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c89be4b0:::benchmark_data_smithi188_141483_object65765:head# 2024-04-23T16:43:24.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c89bef84:::benchmark_data_smithi188_141483_object27644:head# 2024-04-23T16:43:24.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c89c45d3:::benchmark_data_smithi188_141483_object81929:head# 2024-04-23T16:43:24.167 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c89c5dea:::benchmark_data_smithi188_141483_object46251:head# 2024-04-23T16:43:24.167 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c89d33d8:::benchmark_data_smithi188_141483_object79668:head# 2024-04-23T16:43:24.167 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c89e1f65:::benchmark_data_smithi188_141483_object59762:head# 2024-04-23T16:43:24.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c89e23b7:::benchmark_data_smithi188_141483_object3868:head# 2024-04-23T16:43:24.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8a03376:::benchmark_data_smithi188_141483_object19915:head# 2024-04-23T16:43:24.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8a07c6b:::benchmark_data_smithi188_141483_object83030:head# 2024-04-23T16:43:24.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8a0ffc4:::benchmark_data_smithi188_141483_object57202:head# 2024-04-23T16:43:24.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8a14472:::benchmark_data_smithi188_141483_object66902:head# 2024-04-23T16:43:24.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8a1fa6e:::benchmark_data_smithi188_141483_object32047:head# 2024-04-23T16:43:24.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8a31243:::benchmark_data_smithi188_141483_object78651:head# 2024-04-23T16:43:24.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8a3abfc:::benchmark_data_smithi188_141483_object41830:head# 2024-04-23T16:43:24.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8a3e976:::benchmark_data_smithi188_141483_object18317:head# 2024-04-23T16:43:24.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8a42448:::benchmark_data_smithi188_141483_object44352:head# 2024-04-23T16:43:24.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8a4627b:::benchmark_data_smithi188_141483_object42952:head# 2024-04-23T16:43:24.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8a5541d:::benchmark_data_smithi188_141483_object21302:head# 2024-04-23T16:43:24.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8a59bb5:::benchmark_data_smithi188_141483_object67616:head# 2024-04-23T16:43:24.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8a65c37:::benchmark_data_smithi188_141483_object51619:head# 2024-04-23T16:43:24.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8a677d6:::benchmark_data_smithi188_141483_object70242:head# 2024-04-23T16:43:24.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8a6c8a3:::benchmark_data_smithi188_141483_object53921:head# 2024-04-23T16:43:24.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8a75afe:::benchmark_data_smithi188_141483_object77235:head# 2024-04-23T16:43:24.176 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8a80049:::benchmark_data_smithi188_141483_object26448:head# 2024-04-23T16:43:24.177 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8a823b8:::benchmark_data_smithi188_141483_object71416:head# 2024-04-23T16:43:24.177 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8a9e57c:::benchmark_data_smithi188_141483_object120:head# 2024-04-23T16:43:24.177 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8aae6b0:::benchmark_data_smithi188_141483_object81497:head# 2024-04-23T16:43:24.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8ad7534:::benchmark_data_smithi188_141483_object6420:head# 2024-04-23T16:43:24.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8af34a4:::benchmark_data_smithi188_141483_object5768:head# 2024-04-23T16:43:24.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8b0e2bb:::benchmark_data_smithi188_141483_object9142:head# 2024-04-23T16:43:24.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8b3d364:::benchmark_data_smithi188_141483_object89192:head# 2024-04-23T16:43:24.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8b41db0:::benchmark_data_smithi188_141483_object43709:head# 2024-04-23T16:43:24.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8b4c95e:::benchmark_data_smithi188_141483_object47940:head# 2024-04-23T16:43:24.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8b52b9b:::benchmark_data_smithi188_141483_object61231:head# 2024-04-23T16:43:24.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8b5e0aa:::benchmark_data_smithi188_141483_object56315:head# 2024-04-23T16:43:24.181 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8b6df54:::benchmark_data_smithi188_141483_object38693:head# 2024-04-23T16:43:24.181 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8b70577:::benchmark_data_smithi188_141483_object45288:head# 2024-04-23T16:43:24.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8b70d0d:::benchmark_data_smithi188_141483_object44589:head# 2024-04-23T16:43:24.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8b86b22:::benchmark_data_smithi188_141483_object89145:head# 2024-04-23T16:43:24.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8b9afc6:::benchmark_data_smithi188_141483_object19945:head# 2024-04-23T16:43:24.183 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8bb8910:::benchmark_data_smithi188_141483_object21971:head# 2024-04-23T16:43:24.183 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8bbafe9:::benchmark_data_smithi188_141483_object9325:head# 2024-04-23T16:43:24.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8bc4099:::benchmark_data_smithi188_141483_object76396:head# 2024-04-23T16:43:24.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8bc69c5:::benchmark_data_smithi188_141483_object957:head# 2024-04-23T16:43:24.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8bcd7f4:::benchmark_data_smithi188_141483_object19382:head# 2024-04-23T16:43:24.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8bd9f2a:::benchmark_data_smithi188_141483_object44950:head# 2024-04-23T16:43:24.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8bf4efa:::benchmark_data_smithi188_141483_object57285:head# 2024-04-23T16:43:24.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8c11142:::benchmark_data_smithi188_141483_object85961:head# 2024-04-23T16:43:24.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8c16a3b:::benchmark_data_smithi188_141483_object41755:head# 2024-04-23T16:43:24.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8c1ed85:::benchmark_data_smithi188_141483_object25273:head# 2024-04-23T16:43:24.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8c2171c:::benchmark_data_smithi188_141483_object28469:head# 2024-04-23T16:43:24.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8c22722:::benchmark_data_smithi188_141483_object9790:head# 2024-04-23T16:43:24.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8c22e27:::benchmark_data_smithi188_141483_object2784:head# 2024-04-23T16:43:24.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8c330d5:::benchmark_data_smithi188_141483_object43251:head# 2024-04-23T16:43:24.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8c36794:::benchmark_data_smithi188_141483_object39278:head# 2024-04-23T16:43:24.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8c396ea:::benchmark_data_smithi188_141483_object45785:head# 2024-04-23T16:43:24.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8c529b1:::benchmark_data_smithi188_141483_object59233:head# 2024-04-23T16:43:24.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8c6d14f:::benchmark_data_smithi188_141483_object59261:head# 2024-04-23T16:43:24.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8c7e5c2:::benchmark_data_smithi188_141483_object90861:head# 2024-04-23T16:43:24.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8c8088f:::benchmark_data_smithi188_141483_object49234:head# 2024-04-23T16:43:24.192 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8c8c6d6:::benchmark_data_smithi188_141483_object49505:head# 2024-04-23T16:43:24.192 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8c9ab1c:::benchmark_data_smithi188_141483_object54047:head# 2024-04-23T16:43:24.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8cab903:::benchmark_data_smithi188_141483_object6144:head# 2024-04-23T16:43:24.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8cb5193:::benchmark_data_smithi188_141483_object22001:head# 2024-04-23T16:43:24.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8cb72ed:::benchmark_data_smithi188_141483_object53464:head# 2024-04-23T16:43:24.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8cbc6fe:::benchmark_data_smithi188_141483_object26251:head# 2024-04-23T16:43:24.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8cd108c:::benchmark_data_smithi188_141483_object21463:head# 2024-04-23T16:43:24.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8cd2d85:::benchmark_data_smithi188_141483_object29218:head# 2024-04-23T16:43:24.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8cd3962:::benchmark_data_smithi188_141483_object31353:head# 2024-04-23T16:43:24.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8cfdfe5:::benchmark_data_smithi188_141483_object75848:head# 2024-04-23T16:43:24.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8d4fef3:::benchmark_data_smithi188_141483_object4177:head# 2024-04-23T16:43:24.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8d6d958:::benchmark_data_smithi188_141483_object654:head# 2024-04-23T16:43:24.197 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8d706de:::benchmark_data_smithi188_141483_object79509:head# 2024-04-23T16:43:24.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8d7f605:::benchmark_data_smithi188_141483_object36799:head# 2024-04-23T16:43:24.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8d8d6df:::benchmark_data_smithi188_141483_object2939:head# 2024-04-23T16:43:24.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8d93881:::benchmark_data_smithi188_141483_object54674:head# 2024-04-23T16:43:24.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8db685f:::benchmark_data_smithi188_141483_object25598:head# 2024-04-23T16:43:24.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8db9cce:::benchmark_data_smithi188_141483_object76848:head# 2024-04-23T16:43:24.199 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8dd2fbb:::benchmark_data_smithi188_141483_object73718:head# 2024-04-23T16:43:24.199 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8df0901:::benchmark_data_smithi188_141483_object51853:head# 2024-04-23T16:43:24.199 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8dfd06e:::benchmark_data_smithi188_141483_object51572:head# 2024-04-23T16:43:24.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8e018ad:::benchmark_data_smithi188_141483_object39964:head# 2024-04-23T16:43:24.200 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:43:24.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8e0a9d4:::benchmark_data_smithi188_141483_object53521:head# 2024-04-23T16:43:24.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8e22c25:::benchmark_data_smithi188_141483_object88884:head# 2024-04-23T16:43:24.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8e2ce7c:::benchmark_data_smithi188_141483_object4398:head# 2024-04-23T16:43:24.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8e33667:::benchmark_data_smithi188_141483_object66805:head# 2024-04-23T16:43:24.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8e3cd9d:::benchmark_data_smithi188_141483_object58504:head# 2024-04-23T16:43:24.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8e46104:::benchmark_data_smithi188_141483_object60743:head# 2024-04-23T16:43:24.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8e670ab:::benchmark_data_smithi188_141483_object49317:head# 2024-04-23T16:43:24.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8e77960:::benchmark_data_smithi188_141483_object36493:head# 2024-04-23T16:43:24.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8e7a5fc:::benchmark_data_smithi188_141483_object89075:head# 2024-04-23T16:43:24.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8e80f1a:::benchmark_data_smithi188_141483_object73657:head# 2024-04-23T16:43:24.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8e9b18f:::benchmark_data_smithi188_141483_object5557:head# 2024-04-23T16:43:24.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8ead7df:::benchmark_data_smithi188_141483_object82147:head# 2024-04-23T16:43:24.206 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8eaea58:::benchmark_data_smithi188_141483_object83332:head# 2024-04-23T16:43:24.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:24.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:43:24.207 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8ec5237:::benchmark_data_smithi188_141483_object30783:head# 2024-04-23T16:43:24.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8ece313:::benchmark_data_smithi188_141483_object22711:head# 2024-04-23T16:43:24.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8edb96b:::benchmark_data_smithi188_141483_object79364:head# 2024-04-23T16:43:24.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8edcd62:::benchmark_data_smithi188_141483_object66409:head# 2024-04-23T16:43:24.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8eddcc3:::benchmark_data_smithi188_141483_object84546:head# 2024-04-23T16:43:24.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8ee2f77:::benchmark_data_smithi188_141483_object84314:head# 2024-04-23T16:43:24.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8ef5c1d:::benchmark_data_smithi188_141483_object27930:head# 2024-04-23T16:43:24.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8ef9a3e:::benchmark_data_smithi188_141483_object67813:head# 2024-04-23T16:43:24.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8f142e6:::benchmark_data_smithi188_141483_object73748:head# 2024-04-23T16:43:24.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8f1e5c3:::benchmark_data_smithi188_141483_object82949:head# 2024-04-23T16:43:24.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8f2b517:::benchmark_data_smithi188_141483_object77166:head# 2024-04-23T16:43:24.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8f2f76c:::benchmark_data_smithi188_141483_object43991:head# 2024-04-23T16:43:24.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8f3acca:::benchmark_data_smithi188_141483_object64649:head# 2024-04-23T16:43:24.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8f3af08:::benchmark_data_smithi188_141483_object16938:head# 2024-04-23T16:43:24.212 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8f4b691:::benchmark_data_smithi188_141483_object81240:head# 2024-04-23T16:43:24.212 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8f4e267:::benchmark_data_smithi188_141483_object3810:head# 2024-04-23T16:43:24.213 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8f602ee:::benchmark_data_smithi188_141483_object67711:head# 2024-04-23T16:43:24.213 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8f620be:::benchmark_data_smithi188_141483_object58369:head# 2024-04-23T16:43:24.213 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8f6ce5e:::benchmark_data_smithi188_141483_object64817:head# 2024-04-23T16:43:24.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8f7e417:::benchmark_data_smithi188_141483_object48817:head# 2024-04-23T16:43:24.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8f991d9:::benchmark_data_smithi188_141483_object31929:head# 2024-04-23T16:43:24.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8fa8db7:::benchmark_data_smithi188_141483_object53532:head# 2024-04-23T16:43:24.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8fb6587:::benchmark_data_smithi188_141483_object81730:head# 2024-04-23T16:43:24.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8fcc87c:::benchmark_data_smithi188_141483_object57961:head# 2024-04-23T16:43:24.216 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8fd3de7:::benchmark_data_smithi188_141483_object79850:head# 2024-04-23T16:43:24.216 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8fe49e1:::benchmark_data_smithi188_141483_object31193:head# 2024-04-23T16:43:24.217 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c8ff5f49:::benchmark_data_smithi188_141483_object58632:head# 2024-04-23T16:43:24.217 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c901bfec:::benchmark_data_smithi188_141483_object15375:head# 2024-04-23T16:43:24.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c901f8a2:::benchmark_data_smithi188_141483_object76088:head# 2024-04-23T16:43:24.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c903fb6b:::benchmark_data_smithi188_141483_object1405:head# 2024-04-23T16:43:24.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9048ae5:::benchmark_data_smithi188_141483_object16141:head# 2024-04-23T16:43:24.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c90527e0:::benchmark_data_smithi188_141483_object62959:head# 2024-04-23T16:43:24.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9058bc4:::benchmark_data_smithi188_141483_object33025:head# 2024-04-23T16:43:24.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c905fb3a:::benchmark_data_smithi188_141483_object42293:head# 2024-04-23T16:43:24.220 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c906d4cd:::benchmark_data_smithi188_141483_object1216:head# 2024-04-23T16:43:24.221 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c906d6d2:::benchmark_data_smithi188_141483_object46849:head# 2024-04-23T16:43:24.221 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c907c3f6:::benchmark_data_smithi188_141483_object84614:head# 2024-04-23T16:43:24.221 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c907ff77:::benchmark_data_smithi188_141483_object72773:head# 2024-04-23T16:43:24.222 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9083d3c:::benchmark_data_smithi188_141483_object37616:head# 2024-04-23T16:43:24.222 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c908a946:::benchmark_data_smithi188_141483_object28314:head# 2024-04-23T16:43:24.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c908f480:::benchmark_data_smithi188_141483_object24241:head# 2024-04-23T16:43:24.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9094d04:::benchmark_data_smithi188_141483_object47526:head# 2024-04-23T16:43:24.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:23 smithi188 ceph-mon[44101]: pgmap v1580: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 54 MiB/s wr, 869 op/s; 21574/201838 objects degraded (10.689%); 6.3 MiB/s, 100 objects/s recovering 2024-04-23T16:43:24.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:23 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:43:24.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:23 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3550752204' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:43:24.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:23 smithi188 ceph-mon[44101]: osdmap e567: 12 total, 11 up, 9 in 2024-04-23T16:43:24.224 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9095759:::benchmark_data_smithi188_141483_object66085:head# 2024-04-23T16:43:24.224 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9095f44:::benchmark_data_smithi188_141483_object32069:head# 2024-04-23T16:43:24.224 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c90a3a60:::benchmark_data_smithi188_141483_object23499:head# 2024-04-23T16:43:24.225 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c90ae359:::benchmark_data_smithi188_141483_object50786:head# 2024-04-23T16:43:24.225 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c90fd00d:::benchmark_data_smithi188_141483_object81537:head# 2024-04-23T16:43:24.225 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c910c9d6:::benchmark_data_smithi188_141483_object48995:head# 2024-04-23T16:43:24.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c910df08:::benchmark_data_smithi188_141483_object73605:head# 2024-04-23T16:43:24.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9121318:::benchmark_data_smithi188_141483_object73344:head# 2024-04-23T16:43:24.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c912e5c6:::benchmark_data_smithi188_141483_object36566:head# 2024-04-23T16:43:24.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9140f4b:::benchmark_data_smithi188_141483_object47708:head# 2024-04-23T16:43:24.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c91454b7:::benchmark_data_smithi188_141483_object43271:head# 2024-04-23T16:43:24.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c915bb83:::benchmark_data_smithi188_141483_object47132:head# 2024-04-23T16:43:24.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c916f0b9:::benchmark_data_smithi188_141483_object41289:head# 2024-04-23T16:43:24.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c917a96e:::benchmark_data_smithi188_141483_object90332:head# 2024-04-23T16:43:24.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c917c75b:::benchmark_data_smithi188_141483_object54490:head# 2024-04-23T16:43:24.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9184509:::benchmark_data_smithi188_141483_object68260:head# 2024-04-23T16:43:24.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9186f5c:::benchmark_data_smithi188_141483_object57917:head# 2024-04-23T16:43:24.231 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c91b5218:::benchmark_data_smithi188_141483_object50303:head# 2024-04-23T16:43:24.231 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c91c68ce:::benchmark_data_smithi188_141483_object60768:head# 2024-04-23T16:43:24.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c91c8c37:::benchmark_data_smithi188_141483_object77561:head# 2024-04-23T16:43:24.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c91d235c:::benchmark_data_smithi188_141483_object30941:head# 2024-04-23T16:43:24.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c91da6da:::benchmark_data_smithi188_141483_object82451:head# 2024-04-23T16:43:24.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c91db5eb:::benchmark_data_smithi188_141483_object11312:head# 2024-04-23T16:43:24.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c91e7aa2:::benchmark_data_smithi188_141483_object66876:head# 2024-04-23T16:43:24.234 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c91ebda9:::benchmark_data_smithi188_141483_object56326:head# 2024-04-23T16:43:24.234 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c91fd725:::benchmark_data_smithi188_141483_object83252:head# 2024-04-23T16:43:24.235 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c92075c3:::benchmark_data_smithi188_141483_object75412:head# 2024-04-23T16:43:24.235 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c922567e:::benchmark_data_smithi188_141483_object65741:head# 2024-04-23T16:43:24.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9225a6b:::benchmark_data_smithi188_141483_object32532:head# 2024-04-23T16:43:24.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9228d56:::benchmark_data_smithi188_141483_object81666:head# 2024-04-23T16:43:24.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9248dcf:::benchmark_data_smithi188_141483_object14153:head# 2024-04-23T16:43:24.237 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c92493fa:::benchmark_data_smithi188_141483_object62238:head# 2024-04-23T16:43:24.237 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9249fe3:::benchmark_data_smithi188_141483_object73508:head# 2024-04-23T16:43:24.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c924ba27:::benchmark_data_smithi188_141483_object26315:head# 2024-04-23T16:43:24.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c925dae7:::benchmark_data_smithi188_141483_object13142:head# 2024-04-23T16:43:24.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9263951:::benchmark_data_smithi188_141483_object40129:head# 2024-04-23T16:43:24.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9271b7e:::benchmark_data_smithi188_141483_object28653:head# 2024-04-23T16:43:24.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c927233f:::benchmark_data_smithi188_141483_object72477:head# 2024-04-23T16:43:24.240 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9281537:::benchmark_data_smithi188_141483_object60526:head# 2024-04-23T16:43:24.240 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9286f19:::benchmark_data_smithi188_141483_object20608:head# 2024-04-23T16:43:24.241 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c928803f:::benchmark_data_smithi188_141483_object88917:head# 2024-04-23T16:43:24.241 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9289201:::benchmark_data_smithi188_141483_object68248:head# 2024-04-23T16:43:24.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c92aa7b5:::benchmark_data_smithi188_141483_object64928:head# 2024-04-23T16:43:24.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c92b00b2:::benchmark_data_smithi188_141483_object6012:head# 2024-04-23T16:43:24.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c92b0f69:::benchmark_data_smithi188_141483_object64618:head# 2024-04-23T16:43:24.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c92b8a67:::benchmark_data_smithi188_141483_object10515:head# 2024-04-23T16:43:24.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c92ca342:::benchmark_data_smithi188_141483_object78010:head# 2024-04-23T16:43:24.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c92cb73b:::benchmark_data_smithi188_141483_object54883:head# 2024-04-23T16:43:24.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c92d58a2:::benchmark_data_smithi188_141483_object74871:head# 2024-04-23T16:43:24.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c92d7d23:::benchmark_data_smithi188_141483_object75300:head# 2024-04-23T16:43:24.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c92da16c:::benchmark_data_smithi188_141483_object69364:head# 2024-04-23T16:43:24.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c92e6284:::benchmark_data_smithi188_141483_object83017:head# 2024-04-23T16:43:24.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c92ee629:::benchmark_data_smithi188_141483_object77811:head# 2024-04-23T16:43:24.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c92fca43:::benchmark_data_smithi188_141483_object83617:head# 2024-04-23T16:43:24.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c92ffcf1:::benchmark_data_smithi188_141483_object73125:head# 2024-04-23T16:43:24.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c930428f:::benchmark_data_smithi188_141483_object49063:head# 2024-04-23T16:43:24.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c930475b:::benchmark_data_smithi188_141483_object80368:head# 2024-04-23T16:43:24.248 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9309c20:::benchmark_data_smithi188_141483_object25450:head# 2024-04-23T16:43:24.248 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c932c303:::benchmark_data_smithi188_141483_object61261:head# 2024-04-23T16:43:24.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c933db67:::benchmark_data_smithi188_141483_object79137:head# 2024-04-23T16:43:24.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c933e265:::benchmark_data_smithi188_141483_object21946:head# 2024-04-23T16:43:24.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c933e265:::benchmark_data_smithi188_141483_object27861:head# 2024-04-23T16:43:24.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c934634b:::benchmark_data_smithi188_141483_object87253:head# 2024-04-23T16:43:24.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c934d465:::benchmark_data_smithi188_141483_object46293:head# 2024-04-23T16:43:24.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9367654:::benchmark_data_smithi188_141483_object17488:head# 2024-04-23T16:43:24.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9373c92:::benchmark_data_smithi188_141483_object12462:head# 2024-04-23T16:43:24.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c93843c1:::benchmark_data_smithi188_141483_object3945:head# 2024-04-23T16:43:24.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c938b06a:::benchmark_data_smithi188_141483_object59245:head# 2024-04-23T16:43:24.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c938c240:::benchmark_data_smithi188_141483_object43243:head# 2024-04-23T16:43:24.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c939adf1:::benchmark_data_smithi188_141483_object54892:head# 2024-04-23T16:43:24.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c939fe6b:::benchmark_data_smithi188_141483_object22097:head# 2024-04-23T16:43:24.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c93a0b9e:::benchmark_data_smithi188_141483_object41570:head# 2024-04-23T16:43:24.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c93ace50:::benchmark_data_smithi188_141483_object60665:head# 2024-04-23T16:43:24.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c93c636a:::benchmark_data_smithi188_141483_object54296:head# 2024-04-23T16:43:24.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c93d0376:::benchmark_data_smithi188_141483_object49293:head# 2024-04-23T16:43:24.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c93de28e:::benchmark_data_smithi188_141483_object76079:head# 2024-04-23T16:43:24.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c93e7718:::benchmark_data_smithi188_141483_object63051:head# 2024-04-23T16:43:24.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c93f6471:::benchmark_data_smithi188_141483_object87771:head# 2024-04-23T16:43:24.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9412151:::benchmark_data_smithi188_141483_object40485:head# 2024-04-23T16:43:24.258 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9416649:::benchmark_data_smithi188_141483_object17684:head# 2024-04-23T16:43:24.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c94225fe:::benchmark_data_smithi188_141483_object76333:head# 2024-04-23T16:43:24.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9422c68:::benchmark_data_smithi188_141483_object72112:head# 2024-04-23T16:43:24.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9429dc1:::benchmark_data_smithi188_141483_object8331:head# 2024-04-23T16:43:24.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c943763f:::benchmark_data_smithi188_141483_object3818:head# 2024-04-23T16:43:24.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c944691a:::benchmark_data_smithi188_141483_object54790:head# 2024-04-23T16:43:24.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c944f1ed:::benchmark_data_smithi188_141483_object82215:head# 2024-04-23T16:43:24.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c944f5e1:::benchmark_data_smithi188_141483_object35698:head# 2024-04-23T16:43:24.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c94556c7:::benchmark_data_smithi188_141483_object88260:head# 2024-04-23T16:43:24.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c94585ad:::benchmark_data_smithi188_141483_object74052:head# 2024-04-23T16:43:24.262 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c945aa0a:::benchmark_data_smithi188_141483_object21943:head# 2024-04-23T16:43:24.262 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c945d072:::benchmark_data_smithi188_141483_object89717:head# 2024-04-23T16:43:24.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c945e4f3:::benchmark_data_smithi188_141483_object76301:head# 2024-04-23T16:43:24.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c946c5c7:::benchmark_data_smithi188_141483_object14756:head# 2024-04-23T16:43:24.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c947755a:::benchmark_data_smithi188_141483_object6857:head# 2024-04-23T16:43:24.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9479023:::benchmark_data_smithi188_141483_object55612:head# 2024-04-23T16:43:24.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c947e7c4:::benchmark_data_smithi188_141483_object77394:head# 2024-04-23T16:43:24.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9494c83:::benchmark_data_smithi188_141483_object61189:head# 2024-04-23T16:43:24.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c94b39a3:::benchmark_data_smithi188_141483_object11626:head# 2024-04-23T16:43:24.266 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c94bb81b:::benchmark_data_smithi188_141483_object57813:head# 2024-04-23T16:43:24.266 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c94bbb9c:::benchmark_data_smithi188_141483_object7339:head# 2024-04-23T16:43:24.268 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c94cc07d:::benchmark_data_smithi188_141483_object14429:head# 2024-04-23T16:43:24.269 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c94d6bc2:::benchmark_data_smithi188_141483_object80795:head# 2024-04-23T16:43:24.269 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c94e94f3:::benchmark_data_smithi188_141483_object19509:head# 2024-04-23T16:43:24.269 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c94ed035:::benchmark_data_smithi188_141483_object11973:head# 2024-04-23T16:43:24.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c94ee9da:::benchmark_data_smithi188_141483_object53647:head# 2024-04-23T16:43:24.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c94fc22b:::benchmark_data_smithi188_141483_object13451:head# 2024-04-23T16:43:24.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c95121da:::benchmark_data_smithi188_141483_object53356:head# 2024-04-23T16:43:24.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c953eedb:::benchmark_data_smithi188_141483_object9886:head# 2024-04-23T16:43:24.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c954e413:::benchmark_data_smithi188_141483_object66270:head# 2024-04-23T16:43:24.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c955bcad:::benchmark_data_smithi188_141483_object43189:head# 2024-04-23T16:43:24.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c955ff29:::benchmark_data_smithi188_141483_object29137:head# 2024-04-23T16:43:24.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9560ad1:::benchmark_data_smithi188_141483_object27431:head# 2024-04-23T16:43:24.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9572cab:::benchmark_data_smithi188_141483_object87611:head# 2024-04-23T16:43:24.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c958480c:::benchmark_data_smithi188_141483_object8330:head# 2024-04-23T16:43:24.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c959f61c:::benchmark_data_smithi188_141483_object72850:head# 2024-04-23T16:43:24.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c95a69ac:::benchmark_data_smithi188_141483_object72601:head# 2024-04-23T16:43:24.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c95c4c59:::benchmark_data_smithi188_141483_object7368:head# 2024-04-23T16:43:24.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c95c953a:::benchmark_data_smithi188_141483_object4518:head# 2024-04-23T16:43:24.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c95ca063:::benchmark_data_smithi188_141483_object66043:head# 2024-04-23T16:43:24.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c95ccf76:::benchmark_data_smithi188_141483_object33821:head# 2024-04-23T16:43:24.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c95cd70a:::benchmark_data_smithi188_141483_object80168:head# 2024-04-23T16:43:24.277 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c95dcfc1:::benchmark_data_smithi188_141483_object75154:head# 2024-04-23T16:43:24.277 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c95e9d94:::benchmark_data_smithi188_141483_object87704:head# 2024-04-23T16:43:24.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c95fd091:::benchmark_data_smithi188_141483_object89258:head# 2024-04-23T16:43:24.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c960c1eb:::benchmark_data_smithi188_141483_object47545:head# 2024-04-23T16:43:24.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9614c9f:::benchmark_data_smithi188_141483_object5182:head# 2024-04-23T16:43:24.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9632580:::benchmark_data_smithi188_141483_object31166:head# 2024-04-23T16:43:24.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c963902e:::benchmark_data_smithi188_141483_object45002:head# 2024-04-23T16:43:24.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9642d1e:::benchmark_data_smithi188_141483_object24682:head# 2024-04-23T16:43:24.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c96584c2:::benchmark_data_smithi188_141483_object50344:head# 2024-04-23T16:43:24.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9659acc:::benchmark_data_smithi188_141483_object65353:head# 2024-04-23T16:43:24.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c966a873:::benchmark_data_smithi188_141483_object68209:head# 2024-04-23T16:43:24.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c966b5ee:::benchmark_data_smithi188_141483_object6495:head# 2024-04-23T16:43:24.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c966bd80:::benchmark_data_smithi188_141483_object25750:head# 2024-04-23T16:43:24.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c96728e4:::benchmark_data_smithi188_141483_object80960:head# 2024-04-23T16:43:24.287 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9674ad3:::benchmark_data_smithi188_141483_object28760:head# 2024-04-23T16:43:24.287 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c968393d:::benchmark_data_smithi188_141483_object43683:head# 2024-04-23T16:43:24.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9685a03:::benchmark_data_smithi188_141483_object11623:head# 2024-04-23T16:43:24.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c968d078:::benchmark_data_smithi188_141483_object24544:head# 2024-04-23T16:43:24.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c96900cf:::benchmark_data_smithi188_141483_object3748:head# 2024-04-23T16:43:24.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9699a13:::benchmark_data_smithi188_141483_object42776:head# 2024-04-23T16:43:24.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c969f7f5:::benchmark_data_smithi188_141483_object21041:head# 2024-04-23T16:43:24.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c96a794f:::benchmark_data_smithi188_141483_object86849:head# 2024-04-23T16:43:24.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c96bf966:::benchmark_data_smithi188_141483_object2406:head# 2024-04-23T16:43:24.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c96cd061:::benchmark_data_smithi188_141483_object45424:head# 2024-04-23T16:43:24.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c96d839e:::benchmark_data_smithi188_141483_object29404:head# 2024-04-23T16:43:24.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c96dcfb0:::benchmark_data_smithi188_141483_object38236:head# 2024-04-23T16:43:24.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c96e1616:::benchmark_data_smithi188_141483_object71886:head# 2024-04-23T16:43:24.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c96e9951:::benchmark_data_smithi188_141483_object28201:head# 2024-04-23T16:43:24.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c96ecbba:::benchmark_data_smithi188_141483_object55838:head# 2024-04-23T16:43:24.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c96efa90:::benchmark_data_smithi188_141483_object20303:head# 2024-04-23T16:43:24.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c96f1e98:::benchmark_data_smithi188_141483_object39593:head# 2024-04-23T16:43:24.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c96f5341:::benchmark_data_smithi188_141483_object55359:head# 2024-04-23T16:43:24.295 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9701105:::benchmark_data_smithi188_141483_object68376:head# 2024-04-23T16:43:24.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c97109e0:::benchmark_data_smithi188_141483_object13941:head# 2024-04-23T16:43:24.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c971c70a:::benchmark_data_smithi188_141483_object27259:head# 2024-04-23T16:43:24.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c97204f4:::benchmark_data_smithi188_141483_object6827:head# 2024-04-23T16:43:24.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c973fd3f:::benchmark_data_smithi188_141483_object8216:head# 2024-04-23T16:43:24.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c973fd7f:::benchmark_data_smithi188_141483_object79445:head# 2024-04-23T16:43:24.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c974a700:::benchmark_data_smithi188_141483_object62078:head# 2024-04-23T16:43:24.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c974e437:::benchmark_data_smithi188_141483_object90205:head# 2024-04-23T16:43:24.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9758c29:::benchmark_data_smithi188_141483_object9813:head# 2024-04-23T16:43:24.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c977717f:::benchmark_data_smithi188_141483_object33895:head# 2024-04-23T16:43:24.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c978c6b5:::benchmark_data_smithi188_141483_object8323:head# 2024-04-23T16:43:24.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c979eb93:::benchmark_data_smithi188_141483_object992:head# 2024-04-23T16:43:24.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c97a9f5b:::benchmark_data_smithi188_141483_object38752:head# 2024-04-23T16:43:24.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c97b4e6b:::benchmark_data_smithi188_141483_object5965:head# 2024-04-23T16:43:24.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c97be262:::benchmark_data_smithi188_141483_object83317:head# 2024-04-23T16:43:24.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c97c074d:::benchmark_data_smithi188_141483_object34112:head# 2024-04-23T16:43:24.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c97c0ca9:::benchmark_data_smithi188_141483_object77862:head# 2024-04-23T16:43:24.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c97c2286:::benchmark_data_smithi188_141483_object58629:head# 2024-04-23T16:43:24.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c97c78f6:::benchmark_data_smithi188_141483_object69232:head# 2024-04-23T16:43:24.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c97dacd9:::benchmark_data_smithi188_141483_object66869:head# 2024-04-23T16:43:24.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c97e88dd:::benchmark_data_smithi188_141483_object64856:head# 2024-04-23T16:43:24.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c97f4deb:::benchmark_data_smithi188_141483_object29212:head# 2024-04-23T16:43:24.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c97fa253:::benchmark_data_smithi188_141483_object25913:head# 2024-04-23T16:43:24.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c980298b:::benchmark_data_smithi188_141483_object80902:head# 2024-04-23T16:43:24.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9811777:::benchmark_data_smithi188_141483_object73652:head# 2024-04-23T16:43:24.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9828771:::benchmark_data_smithi188_141483_object18827:head# 2024-04-23T16:43:24.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9829932:::benchmark_data_smithi188_141483_object60774:head# 2024-04-23T16:43:24.306 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c983aa67:::benchmark_data_smithi188_141483_object67162:head# 2024-04-23T16:43:24.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9847523:::benchmark_data_smithi188_141483_object3578:head# 2024-04-23T16:43:24.307 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:43:24.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c98475e9:::benchmark_data_smithi188_141483_object29896:head# 2024-04-23T16:43:24.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c985f0de:::benchmark_data_smithi188_141483_object40345:head# 2024-04-23T16:43:24.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9874689:::benchmark_data_smithi188_141483_object21672:head# 2024-04-23T16:43:24.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9875464:::benchmark_data_smithi188_141483_object40599:head# 2024-04-23T16:43:24.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c987fd60:::benchmark_data_smithi188_141483_object66285:head# 2024-04-23T16:43:24.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c989e5f9:::benchmark_data_smithi188_141483_object80261:head# 2024-04-23T16:43:24.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c98a2ce5:::benchmark_data_smithi188_141483_object72826:head# 2024-04-23T16:43:24.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c98aab63:::benchmark_data_smithi188_141483_object9280:head# 2024-04-23T16:43:24.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c98c5f35:::benchmark_data_smithi188_141483_object33676:head# 2024-04-23T16:43:24.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c98c9328:::benchmark_data_smithi188_141483_object45505:head# 2024-04-23T16:43:24.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c98cf09d:::benchmark_data_smithi188_141483_object21630:head# 2024-04-23T16:43:24.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c98d8299:::benchmark_data_smithi188_141483_object48732:head# 2024-04-23T16:43:24.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c98e77fc:::benchmark_data_smithi188_141483_object7819:head# 2024-04-23T16:43:24.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c990c715:::benchmark_data_smithi188_141483_object69773:head# 2024-04-23T16:43:24.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c99133ad:::benchmark_data_smithi188_141483_object36615:head# 2024-04-23T16:43:24.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9935e92:::benchmark_data_smithi188_141483_object83089:head# 2024-04-23T16:43:24.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9945382:::benchmark_data_smithi188_141483_object73287:head# 2024-04-23T16:43:24.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9949211:::benchmark_data_smithi188_141483_object7518:head# 2024-04-23T16:43:24.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c994992c:::benchmark_data_smithi188_141483_object58751:head# 2024-04-23T16:43:24.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c994c3f1:::benchmark_data_smithi188_141483_object66281:head# 2024-04-23T16:43:24.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c995e1cf:::benchmark_data_smithi188_141483_object75930:head# 2024-04-23T16:43:24.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9983bf6:::benchmark_data_smithi188_141483_object71998:head# 2024-04-23T16:43:24.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c998f8bb:::benchmark_data_smithi188_141483_object5999:head# 2024-04-23T16:43:24.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9992910:::benchmark_data_smithi188_141483_object52534:head# 2024-04-23T16:43:24.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c99a0e0c:::benchmark_data_smithi188_141483_object64130:head# 2024-04-23T16:43:24.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c99a80e4:::benchmark_data_smithi188_141483_object50560:head# 2024-04-23T16:43:24.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c99b338d:::benchmark_data_smithi188_141483_object58790:head# 2024-04-23T16:43:24.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c99d1c8d:::benchmark_data_smithi188_141483_object10065:head# 2024-04-23T16:43:24.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c99d2a9d:::benchmark_data_smithi188_141483_object54855:head# 2024-04-23T16:43:24.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c99dea22:::benchmark_data_smithi188_141483_object21909:head# 2024-04-23T16:43:24.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c99e1b55:::benchmark_data_smithi188_141483_object51123:head# 2024-04-23T16:43:24.326 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c99fe3c7:::benchmark_data_smithi188_141483_object40946:head# 2024-04-23T16:43:24.326 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9a00fb4:::benchmark_data_smithi188_141483_object20723:head# 2024-04-23T16:43:24.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9a06c2d:::benchmark_data_smithi188_141483_object36685:head# 2024-04-23T16:43:24.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9a0e518:::benchmark_data_smithi188_141483_object30082:head# 2024-04-23T16:43:24.328 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9a10dad:::benchmark_data_smithi188_141483_object47801:head# 2024-04-23T16:43:24.328 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9a22c3b:::benchmark_data_smithi188_141483_object56117:head# 2024-04-23T16:43:24.328 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9a3ed1b:::benchmark_data_smithi188_141483_object47299:head# 2024-04-23T16:43:24.329 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9a4f431:::benchmark_data_smithi188_141483_object43549:head# 2024-04-23T16:43:24.329 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9a5e2a4:::benchmark_data_smithi188_141483_object75536:head# 2024-04-23T16:43:24.329 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9a64628:::benchmark_data_smithi188_141483_object77700:head# 2024-04-23T16:43:24.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9a78d33:::benchmark_data_smithi188_141483_object21307:head# 2024-04-23T16:43:24.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9a8130d:::benchmark_data_smithi188_141483_object58836:head# 2024-04-23T16:43:24.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9a8cce5:::benchmark_data_smithi188_141483_object78210:head# 2024-04-23T16:43:24.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9a9a0ba:::benchmark_data_smithi188_141483_object69304:head# 2024-04-23T16:43:24.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9aa863a:::benchmark_data_smithi188_141483_object55065:head# 2024-04-23T16:43:24.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9ab9743:::benchmark_data_smithi188_141483_object20445:head# 2024-04-23T16:43:24.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:23 smithi103 ceph-mon[39629]: pgmap v1580: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 54 MiB/s wr, 869 op/s; 21574/201838 objects degraded (10.689%); 6.3 MiB/s, 100 objects/s recovering 2024-04-23T16:43:24.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:23 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:43:24.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:23 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3550752204' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:43:24.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:23 smithi103 ceph-mon[39629]: osdmap e567: 12 total, 11 up, 9 in 2024-04-23T16:43:24.354 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:43:24.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9abd6c7:::benchmark_data_smithi188_141483_object11889:head# 2024-04-23T16:43:24.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9ac7145:::benchmark_data_smithi188_141483_object65115:head# 2024-04-23T16:43:24.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9acb2db:::benchmark_data_smithi188_141483_object48145:head# 2024-04-23T16:43:24.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9ad09a6:::benchmark_data_smithi188_141483_object40860:head# 2024-04-23T16:43:24.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9ad0e32:::benchmark_data_smithi188_141483_object167:head# 2024-04-23T16:43:24.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9adf079:::benchmark_data_smithi188_141483_object32395:head# 2024-04-23T16:43:24.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9ae0d50:::benchmark_data_smithi188_141483_object88622:head# 2024-04-23T16:43:24.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9ae76e5:::benchmark_data_smithi188_141483_object60705:head# 2024-04-23T16:43:24.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9af6664:::benchmark_data_smithi188_141483_object32278:head# 2024-04-23T16:43:24.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9b03970:::benchmark_data_smithi188_141483_object71834:head# 2024-04-23T16:43:24.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9b0d2da:::benchmark_data_smithi188_141483_object80940:head# 2024-04-23T16:43:24.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9b42a1a:::benchmark_data_smithi188_141483_object58618:head# 2024-04-23T16:43:24.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9b49a55:::benchmark_data_smithi188_141483_object51135:head# 2024-04-23T16:43:24.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9b4d75b:::benchmark_data_smithi188_141483_object84806:head# 2024-04-23T16:43:24.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9b53900:::benchmark_data_smithi188_141483_object3642:head# 2024-04-23T16:43:24.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9b7275a:::benchmark_data_smithi188_141483_object56873:head# 2024-04-23T16:43:24.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9b73164:::benchmark_data_smithi188_141483_object47305:head# 2024-04-23T16:43:24.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9b923be:::benchmark_data_smithi188_141483_object64893:head# 2024-04-23T16:43:24.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9b96c27:::benchmark_data_smithi188_141483_object39976:head# 2024-04-23T16:43:24.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9ba5e3e:::benchmark_data_smithi188_141483_object63174:head# 2024-04-23T16:43:24.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9bb9ae2:::benchmark_data_smithi188_141483_object18119:head# 2024-04-23T16:43:24.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9bba1f8:::benchmark_data_smithi188_141483_object45676:head# 2024-04-23T16:43:24.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9bbd631:::benchmark_data_smithi188_141483_object52718:head# 2024-04-23T16:43:24.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9bcc5e4:::benchmark_data_smithi188_141483_object7255:head# 2024-04-23T16:43:24.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9bd9063:::benchmark_data_smithi188_141483_object48691:head# 2024-04-23T16:43:24.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9bf3c65:::benchmark_data_smithi188_141483_object30305:head# 2024-04-23T16:43:24.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9bf9db1:::benchmark_data_smithi188_141483_object78016:head# 2024-04-23T16:43:24.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9c05948:::benchmark_data_smithi188_141483_object478:head# 2024-04-23T16:43:24.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9c54278:::benchmark_data_smithi188_141483_object10213:head# 2024-04-23T16:43:24.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9c5f44e:::benchmark_data_smithi188_141483_object49769:head# 2024-04-23T16:43:24.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9c613f2:::benchmark_data_smithi188_141483_object20366:head# 2024-04-23T16:43:24.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9c75f8a:::benchmark_data_smithi188_141483_object31497:head# 2024-04-23T16:43:24.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9c764b6:::benchmark_data_smithi188_141483_object57877:head# 2024-04-23T16:43:24.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9c8586e:::benchmark_data_smithi188_141483_object48603:head# 2024-04-23T16:43:24.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9c89e0e:::benchmark_data_smithi188_141483_object37453:head# 2024-04-23T16:43:24.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9c9591b:::benchmark_data_smithi188_141483_object79442:head# 2024-04-23T16:43:24.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9c9e3bd:::benchmark_data_smithi188_141483_object1491:head# 2024-04-23T16:43:24.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9ca5905:::benchmark_data_smithi188_141483_object38181:head# 2024-04-23T16:43:24.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9caeea0:::benchmark_data_smithi188_141483_object65780:head# 2024-04-23T16:43:24.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9cc5ff0:::benchmark_data_smithi188_141483_object66261:head# 2024-04-23T16:43:24.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9ccbc1d:::benchmark_data_smithi188_141483_object80009:head# 2024-04-23T16:43:24.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9cda5f5:::benchmark_data_smithi188_141483_object7186:head# 2024-04-23T16:43:24.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9cdb9d1:::benchmark_data_smithi188_141483_object32531:head# 2024-04-23T16:43:24.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9cdfcdb:::benchmark_data_smithi188_141483_object58549:head# 2024-04-23T16:43:24.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9ce49d3:::benchmark_data_smithi188_141483_object2786:head# 2024-04-23T16:43:24.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9d2b07d:::benchmark_data_smithi188_141483_object81280:head# 2024-04-23T16:43:24.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9d31ecf:::benchmark_data_smithi188_141483_object29479:head# 2024-04-23T16:43:24.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9d441e7:::benchmark_data_smithi188_141483_object31194:head# 2024-04-23T16:43:24.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9d47d05:::benchmark_data_smithi188_141483_object36241:head# 2024-04-23T16:43:24.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9d706b1:::benchmark_data_smithi188_141483_object7439:head# 2024-04-23T16:43:24.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9d7df8a:::benchmark_data_smithi188_141483_object85398:head# 2024-04-23T16:43:24.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9d85851:::benchmark_data_smithi188_141483_object81152:head# 2024-04-23T16:43:24.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9d90c3b:::benchmark_data_smithi188_141483_object57686:head# 2024-04-23T16:43:24.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9d9351b:::benchmark_data_smithi188_141483_object38041:head# 2024-04-23T16:43:24.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9d9d23f:::benchmark_data_smithi188_141483_object105:head# 2024-04-23T16:43:24.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9d9f765:::benchmark_data_smithi188_141483_object26343:head# 2024-04-23T16:43:24.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9da04f3:::benchmark_data_smithi188_141483_object23931:head# 2024-04-23T16:43:24.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9daf654:::benchmark_data_smithi188_141483_object42309:head# 2024-04-23T16:43:24.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9dbe89a:::benchmark_data_smithi188_141483_object61314:head# 2024-04-23T16:43:24.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9dc8884:::benchmark_data_smithi188_141483_object62159:head# 2024-04-23T16:43:24.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9dc9238:::benchmark_data_smithi188_141483_object21705:head# 2024-04-23T16:43:24.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9de6873:::benchmark_data_smithi188_141483_object87816:head# 2024-04-23T16:43:24.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9dee16f:::benchmark_data_smithi188_141483_object7682:head# 2024-04-23T16:43:24.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9dfb165:::benchmark_data_smithi188_141483_object45033:head# 2024-04-23T16:43:24.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9dfe441:::benchmark_data_smithi188_141483_object16363:head# 2024-04-23T16:43:24.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9e0981a:::benchmark_data_smithi188_141483_object71263:head# 2024-04-23T16:43:24.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9e113dc:::benchmark_data_smithi188_141483_object13895:head# 2024-04-23T16:43:24.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9e1786a:::benchmark_data_smithi188_141483_object66705:head# 2024-04-23T16:43:24.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9e206ae:::benchmark_data_smithi188_141483_object2059:head# 2024-04-23T16:43:24.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9e2f151:::benchmark_data_smithi188_141483_object89597:head# 2024-04-23T16:43:24.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9e3431a:::benchmark_data_smithi188_141483_object42009:head# 2024-04-23T16:43:24.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9e496ba:::benchmark_data_smithi188_141483_object13565:head# 2024-04-23T16:43:24.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9e4cfd3:::benchmark_data_smithi188_141483_object81389:head# 2024-04-23T16:43:24.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9e569cc:::benchmark_data_smithi188_141483_object80410:head# 2024-04-23T16:43:24.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9e6105b:::benchmark_data_smithi188_141483_object50520:head# 2024-04-23T16:43:24.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9e6aad0:::benchmark_data_smithi188_141483_object59937:head# 2024-04-23T16:43:24.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9e727a5:::benchmark_data_smithi188_141483_object38099:head# 2024-04-23T16:43:24.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9e770ad:::benchmark_data_smithi188_141483_object9077:head# 2024-04-23T16:43:24.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9e8c0a6:::benchmark_data_smithi188_141483_object83290:head# 2024-04-23T16:43:24.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9e94194:::benchmark_data_smithi188_141483_object34342:head# 2024-04-23T16:43:24.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9e9bdf0:::benchmark_data_smithi188_141483_object4023:head# 2024-04-23T16:43:24.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9e9cb41:::benchmark_data_smithi188_141483_object33563:head# 2024-04-23T16:43:24.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9ebbeab:::benchmark_data_smithi188_141483_object47124:head# 2024-04-23T16:43:24.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9ec4541:::benchmark_data_smithi188_141483_object86832:head# 2024-04-23T16:43:24.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9ee4c53:::benchmark_data_smithi188_141483_object16160:head# 2024-04-23T16:43:24.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9ef0ffd:::benchmark_data_smithi188_141483_object29565:head# 2024-04-23T16:43:24.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9f0212a:::benchmark_data_smithi188_141483_object63881:head# 2024-04-23T16:43:24.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9f046c6:::benchmark_data_smithi188_141483_object89984:head# 2024-04-23T16:43:24.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9f3bccd:::benchmark_data_smithi188_141483_object49367:head# 2024-04-23T16:43:24.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9f3efa7:::benchmark_data_smithi188_141483_object2590:head# 2024-04-23T16:43:24.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9f49e7a:::benchmark_data_smithi188_141483_object52347:head# 2024-04-23T16:43:24.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9f4ce8b:::benchmark_data_smithi188_141483_object5227:head# 2024-04-23T16:43:24.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9f64511:::benchmark_data_smithi188_141483_object9758:head# 2024-04-23T16:43:24.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9f6874f:::benchmark_data_smithi188_141483_object31980:head# 2024-04-23T16:43:24.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9f6a5bf:::benchmark_data_smithi188_141483_object44738:head# 2024-04-23T16:43:24.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9f7a398:::benchmark_data_smithi188_141483_object24858:head# 2024-04-23T16:43:24.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9f824c3:::benchmark_data_smithi188_141483_object19031:head# 2024-04-23T16:43:24.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9f901cf:::benchmark_data_smithi188_141483_object43913:head# 2024-04-23T16:43:24.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9f9ac8d:::benchmark_data_smithi188_141483_object37968:head# 2024-04-23T16:43:24.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9fa0a91:::benchmark_data_smithi188_141483_object70610:head# 2024-04-23T16:43:24.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9fa642e:::benchmark_data_smithi188_141483_object66335:head# 2024-04-23T16:43:24.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9fc0b2a:::benchmark_data_smithi188_141483_object59183:head# 2024-04-23T16:43:24.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9fc0e09:::benchmark_data_smithi188_141483_object88920:head# 2024-04-23T16:43:24.388 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:24.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9fc6dee:::benchmark_data_smithi188_141483_object44267:head# 2024-04-23T16:43:24.388 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9fcf21c:::benchmark_data_smithi188_141483_object30850:head# 2024-04-23T16:43:24.389 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:c9fd67ae:::benchmark_data_smithi188_141483_object78544:head# 2024-04-23T16:43:24.389 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca00232f:::benchmark_data_smithi188_141483_object58990:head# 2024-04-23T16:43:24.389 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca0105a9:::benchmark_data_smithi188_141483_object27121:head# 2024-04-23T16:43:24.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca0166c1:::benchmark_data_smithi188_141483_object65498:head# 2024-04-23T16:43:24.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca0205e7:::benchmark_data_smithi188_141483_object12546:head# 2024-04-23T16:43:24.391 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca02aa7c:::benchmark_data_smithi188_141483_object65049:head# 2024-04-23T16:43:24.391 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca0338f2:::benchmark_data_smithi188_141483_object80520:head# 2024-04-23T16:43:24.391 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca03a82c:::benchmark_data_smithi188_141483_object90366:head# 2024-04-23T16:43:24.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca060100:::benchmark_data_smithi188_141483_object25992:head# 2024-04-23T16:43:24.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca065757:::benchmark_data_smithi188_141483_object217:head# 2024-04-23T16:43:24.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca07d0bd:::benchmark_data_smithi188_141483_object34424:head# 2024-04-23T16:43:24.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca081362:::benchmark_data_smithi188_141483_object53019:head# 2024-04-23T16:43:24.393 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca08398e:::benchmark_data_smithi188_141483_object32179:head# 2024-04-23T16:43:24.396 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca0a62d6:::benchmark_data_smithi188_141483_object30770:head# 2024-04-23T16:43:24.397 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca0afa4d:::benchmark_data_smithi188_141483_object30843:head# 2024-04-23T16:43:24.397 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca0b8907:::benchmark_data_smithi188_141483_object90501:head# 2024-04-23T16:43:24.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca0d4591:::benchmark_data_smithi188_141483_object44030:head# 2024-04-23T16:43:24.398 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca0d75bc:::benchmark_data_smithi188_141483_object83725:head# 2024-04-23T16:43:24.401 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca0e833b:::benchmark_data_smithi188_141483_object63272:head# 2024-04-23T16:43:24.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca0f6e02:::benchmark_data_smithi188_141483_object10948:head# 2024-04-23T16:43:24.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca0fc5d8:::benchmark_data_smithi188_141483_object22543:head# 2024-04-23T16:43:24.402 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca10b778:::benchmark_data_smithi188_141483_object44088:head# 2024-04-23T16:43:24.403 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca13c6ee:::benchmark_data_smithi188_141483_object80788:head# 2024-04-23T16:43:24.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca14e26c:::benchmark_data_smithi188_141483_object72213:head# 2024-04-23T16:43:24.404 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca14e367:::benchmark_data_smithi188_141483_object87680:head# 2024-04-23T16:43:24.405 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca1692f5:::benchmark_data_smithi188_141483_object3606:head# 2024-04-23T16:43:24.405 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca16fe07:::benchmark_data_smithi188_141483_object41669:head# 2024-04-23T16:43:24.405 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca175769:::benchmark_data_smithi188_141483_object4942:head# 2024-04-23T16:43:24.405 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca17e45b:::benchmark_data_smithi188_141483_object37942:head# 2024-04-23T16:43:24.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca184d04:::benchmark_data_smithi188_141483_object9624:head# 2024-04-23T16:43:24.406 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca19b0d9:::benchmark_data_smithi188_141483_object38016:head# 2024-04-23T16:43:24.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca1aedfb:::benchmark_data_smithi188_141483_object37168:head# 2024-04-23T16:43:24.407 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca1b90c8:::benchmark_data_smithi188_141483_object80040:head# 2024-04-23T16:43:24.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca1ba3c1:::benchmark_data_smithi188_141483_object6205:head# 2024-04-23T16:43:24.408 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca1bcb6f:::benchmark_data_smithi188_141483_object78400:head# 2024-04-23T16:43:24.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca1bd19f:::benchmark_data_smithi188_141483_object63007:head# 2024-04-23T16:43:24.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca1c61fb:::benchmark_data_smithi188_141483_object29344:head# 2024-04-23T16:43:24.409 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca1cc926:::benchmark_data_smithi188_141483_object80855:head# 2024-04-23T16:43:24.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca1d0e13:::benchmark_data_smithi188_141483_object33977:head# 2024-04-23T16:43:24.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca1d55e9:::benchmark_data_smithi188_141483_object78635:head# 2024-04-23T16:43:24.410 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca1dfb83:::benchmark_data_smithi188_141483_object76294:head# 2024-04-23T16:43:24.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca1eb706:::benchmark_data_smithi188_141483_object33980:head# 2024-04-23T16:43:24.411 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca1f74da:::benchmark_data_smithi188_141483_object7321:head# 2024-04-23T16:43:24.412 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca1f8dba:::benchmark_data_smithi188_141483_object88066:head# 2024-04-23T16:43:24.412 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca1fc133:::benchmark_data_smithi188_141483_object43679:head# 2024-04-23T16:43:24.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca1ff8d9:::benchmark_data_smithi188_141483_object71014:head# 2024-04-23T16:43:24.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca205286:::benchmark_data_smithi188_141483_object32196:head# 2024-04-23T16:43:24.413 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca241aa6:::benchmark_data_smithi188_141483_object40959:head# 2024-04-23T16:43:24.415 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca2506cc:::benchmark_data_smithi188_141483_object49864:head# 2024-04-23T16:43:24.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca25a9d8:::benchmark_data_smithi188_141483_object51808:head# 2024-04-23T16:43:24.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca263e1a:::benchmark_data_smithi188_141483_object80444:head# 2024-04-23T16:43:24.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca268611:::benchmark_data_smithi188_141483_object81881:head# 2024-04-23T16:43:24.416 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca26c2cf:::benchmark_data_smithi188_141483_object68889:head# 2024-04-23T16:43:24.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca26cfad:::benchmark_data_smithi188_141483_object58726:head# 2024-04-23T16:43:24.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca26fae0:::benchmark_data_smithi188_141483_object46230:head# 2024-04-23T16:43:24.417 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca2702b5:::benchmark_data_smithi188_141483_object27587:head# 2024-04-23T16:43:24.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca27213c:::benchmark_data_smithi188_141483_object25037:head# 2024-04-23T16:43:24.418 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca276ea1:::benchmark_data_smithi188_141483_object61505:head# 2024-04-23T16:43:24.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca2776fe:::benchmark_data_smithi188_141483_object83901:head# 2024-04-23T16:43:24.419 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca2934a4:::benchmark_data_smithi188_141483_object89955:head# 2024-04-23T16:43:24.420 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca29c220:::benchmark_data_smithi188_141483_object79014:head# 2024-04-23T16:43:24.422 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca2a34bc:::benchmark_data_smithi188_141483_object40920:head# 2024-04-23T16:43:24.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca2a5a74:::benchmark_data_smithi188_141483_object73072:head# 2024-04-23T16:43:24.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca2b991c:::benchmark_data_smithi188_141483_object73661:head# 2024-04-23T16:43:24.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca2d6917:::benchmark_data_smithi188_141483_object11631:head# 2024-04-23T16:43:24.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca2e0d8c:::benchmark_data_smithi188_141483_object36274:head# 2024-04-23T16:43:24.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca2ec2ee:::benchmark_data_smithi188_141483_object25036:head# 2024-04-23T16:43:24.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca2f72a4:::benchmark_data_smithi188_141483_object38320:head# 2024-04-23T16:43:24.423 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca2feba8:::benchmark_data_smithi188_141483_object83953:head# 2024-04-23T16:43:24.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca320072:::benchmark_data_smithi188_141483_object67017:head# 2024-04-23T16:43:24.424 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca33ffc2:::benchmark_data_smithi188_141483_object1640:head# 2024-04-23T16:43:24.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca351fdc:::benchmark_data_smithi188_141483_object24935:head# 2024-04-23T16:43:24.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca359fb7:::benchmark_data_smithi188_141483_object89711:head# 2024-04-23T16:43:24.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca36992b:::benchmark_data_smithi188_141483_object57625:head# 2024-04-23T16:43:24.425 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca384e6e:::benchmark_data_smithi188_141483_object24938:head# 2024-04-23T16:43:24.427 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:43:24.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca38532d:::benchmark_data_smithi188_141483_object87683:head# 2024-04-23T16:43:24.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca38596f:::benchmark_data_smithi188_141483_object30429:head# 2024-04-23T16:43:24.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca3904e7:::benchmark_data_smithi188_141483_object82117:head# 2024-04-23T16:43:24.428 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca39e05f:::benchmark_data_smithi188_141483_object62343:head# 2024-04-23T16:43:24.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca3b93db:::benchmark_data_smithi188_141483_object28447:head# 2024-04-23T16:43:24.429 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca3bf97b:::benchmark_data_smithi188_141483_object65754:head# 2024-04-23T16:43:24.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca3c0f5d:::benchmark_data_smithi188_141483_object56450:head# 2024-04-23T16:43:24.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca3c16d5:::benchmark_data_smithi188_141483_object53266:head# 2024-04-23T16:43:24.430 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca3cbfd6:::benchmark_data_smithi188_141483_object17504:head# 2024-04-23T16:43:24.431 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca3d6882:::benchmark_data_smithi188_141483_object39821:head# 2024-04-23T16:43:24.431 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca3e0ef1:::benchmark_data_smithi188_141483_object1891:head# 2024-04-23T16:43:24.432 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca3e335d:::benchmark_data_smithi188_141483_object76971:head# 2024-04-23T16:43:24.432 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca3ef224:::benchmark_data_smithi188_141483_object83242:head# 2024-04-23T16:43:24.432 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca417aa7:::benchmark_data_smithi188_141483_object70237:head# 2024-04-23T16:43:24.432 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca41e1de:::benchmark_data_smithi188_141483_object70009:head# 2024-04-23T16:43:24.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca42091c:::benchmark_data_smithi188_141483_object58027:head# 2024-04-23T16:43:24.433 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca429ba0:::benchmark_data_smithi188_141483_object24771:head# 2024-04-23T16:43:24.434 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca42d8da:::benchmark_data_smithi188_141483_object12976:head# 2024-04-23T16:43:24.435 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca43e22d:::benchmark_data_smithi188_141483_object75222:head# 2024-04-23T16:43:24.435 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca4439ab:::benchmark_data_smithi188_141483_object45124:head# 2024-04-23T16:43:24.435 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca448ce4:::benchmark_data_smithi188_141483_object16819:head# 2024-04-23T16:43:24.435 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca4767f9:::benchmark_data_smithi188_141483_object26180:head# 2024-04-23T16:43:24.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca48fc7b:::benchmark_data_smithi188_141483_object73302:head# 2024-04-23T16:43:24.436 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca4be2ab:::benchmark_data_smithi188_141483_object72133:head# 2024-04-23T16:43:24.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca4c5fa5:::benchmark_data_smithi188_141483_object89084:head# 2024-04-23T16:43:24.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca4c885a:::benchmark_data_smithi188_141483_object15461:head# 2024-04-23T16:43:24.437 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca4de8ef:::benchmark_data_smithi188_141483_object9520:head# 2024-04-23T16:43:24.438 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca4e664f:::benchmark_data_smithi188_141483_object24773:head# 2024-04-23T16:43:24.438 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca4effa0:::benchmark_data_smithi188_141483_object36454:head# 2024-04-23T16:43:24.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca4f03b0:::benchmark_data_smithi188_141483_object73266:head# 2024-04-23T16:43:24.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca4f403a:::benchmark_data_smithi188_141483_object31154:head# 2024-04-23T16:43:24.439 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca5016a2:::benchmark_data_smithi188_141483_object32293:head# 2024-04-23T16:43:24.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca505ef3:::benchmark_data_smithi188_141483_object72268:head# 2024-04-23T16:43:24.440 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca510f74:::benchmark_data_smithi188_141483_object68104:head# 2024-04-23T16:43:24.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca513363:::benchmark_data_smithi188_141483_object54358:head# 2024-04-23T16:43:24.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca519712:::benchmark_data_smithi188_141483_object25881:head# 2024-04-23T16:43:24.441 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca51e3e2:::benchmark_data_smithi188_141483_object2984:head# 2024-04-23T16:43:24.442 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca531f25:::benchmark_data_smithi188_141483_object22729:head# 2024-04-23T16:43:24.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca533739:::benchmark_data_smithi188_141483_object44122:head# 2024-04-23T16:43:24.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca540fa8:::benchmark_data_smithi188_141483_object56567:head# 2024-04-23T16:43:24.443 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca54a502:::benchmark_data_smithi188_141483_object52906:head# 2024-04-23T16:43:24.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca54aba1:::benchmark_data_smithi188_141483_object47357:head# 2024-04-23T16:43:24.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca555c7c:::benchmark_data_smithi188_141483_object71986:head# 2024-04-23T16:43:24.444 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca5654e6:::benchmark_data_smithi188_141483_object3737:head# 2024-04-23T16:43:24.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca5679bd:::benchmark_data_smithi188_141483_object79033:head# 2024-04-23T16:43:24.445 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca57afa7:::benchmark_data_smithi188_141483_object32176:head# 2024-04-23T16:43:24.446 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca58aa37:::benchmark_data_smithi188_141483_object67514:head# 2024-04-23T16:43:24.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca597179:::benchmark_data_smithi188_141483_object28776:head# 2024-04-23T16:43:24.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca59e5ac:::benchmark_data_smithi188_141483_object89825:head# 2024-04-23T16:43:24.447 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca5a04bb:::benchmark_data_smithi188_141483_object66178:head# 2024-04-23T16:43:24.448 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca5b2e81:::benchmark_data_smithi188_141483_object11615:head# 2024-04-23T16:43:24.448 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca5ce0d0:::benchmark_data_smithi188_141483_object53067:head# 2024-04-23T16:43:24.449 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca5d339f:::benchmark_data_smithi188_141483_object58762:head# 2024-04-23T16:43:24.449 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca5f34e6:::benchmark_data_smithi188_141483_object47792:head# 2024-04-23T16:43:24.449 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca5f77c0:::benchmark_data_smithi188_141483_object49132:head# 2024-04-23T16:43:24.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca60abe6:::benchmark_data_smithi188_141483_object64581:head# 2024-04-23T16:43:24.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca61d094:::benchmark_data_smithi188_141483_object6232:head# 2024-04-23T16:43:24.450 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca6312ad:::benchmark_data_smithi188_141483_object7230:head# 2024-04-23T16:43:24.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:24.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:43:24.452 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca634da0:::benchmark_data_smithi188_141483_object78806:head# 2024-04-23T16:43:24.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca635917:::benchmark_data_smithi188_141483_object2434:head# 2024-04-23T16:43:24.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca652789:::benchmark_data_smithi188_141483_object89412:head# 2024-04-23T16:43:24.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca65ae09:::benchmark_data_smithi188_141483_object18588:head# 2024-04-23T16:43:24.453 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca664207:::benchmark_data_smithi188_141483_object9657:head# 2024-04-23T16:43:24.454 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca66ca59:::benchmark_data_smithi188_141483_object69239:head# 2024-04-23T16:43:24.454 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca670edd:::benchmark_data_smithi188_141483_object56924:head# 2024-04-23T16:43:24.454 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca6731d9:::benchmark_data_smithi188_141483_object48611:head# 2024-04-23T16:43:24.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca688f06:::benchmark_data_smithi188_141483_object33792:head# 2024-04-23T16:43:24.455 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca6944c5:::benchmark_data_smithi188_141483_object8051:head# 2024-04-23T16:43:24.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca694a4e:::benchmark_data_smithi188_141483_object82951:head# 2024-04-23T16:43:24.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca6ab5e7:::benchmark_data_smithi188_141483_object37256:head# 2024-04-23T16:43:24.456 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca6b83b3:::benchmark_data_smithi188_141483_object7712:head# 2024-04-23T16:43:24.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca6c20cf:::benchmark_data_smithi188_141483_object3700:head# 2024-04-23T16:43:24.457 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca6ce1d5:::benchmark_data_smithi188_141483_object76825:head# 2024-04-23T16:43:24.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca6d5b58:::benchmark_data_smithi188_141483_object22329:head# 2024-04-23T16:43:24.458 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca6da80d:::benchmark_data_smithi188_141483_object86290:head# 2024-04-23T16:43:24.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca6df7ab:::benchmark_data_smithi188_141483_object14642:head# 2024-04-23T16:43:24.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca6e4bbb:::benchmark_data_smithi188_141483_object53469:head# 2024-04-23T16:43:24.459 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca6e6673:::benchmark_data_smithi188_141483_object80252:head# 2024-04-23T16:43:24.460 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca6eaff7:::benchmark_data_smithi188_141483_object10124:head# 2024-04-23T16:43:24.461 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca6f5e80:::benchmark_data_smithi188_141483_object72431:head# 2024-04-23T16:43:24.461 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca6f6528:::benchmark_data_smithi188_141483_object11786:head# 2024-04-23T16:43:24.461 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca6f8531:::benchmark_data_smithi188_141483_object11234:head# 2024-04-23T16:43:24.461 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca6fecbb:::benchmark_data_smithi188_141483_object17195:head# 2024-04-23T16:43:24.462 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca6ff5fe:::benchmark_data_smithi188_141483_object32070:head# 2024-04-23T16:43:24.462 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca6ffdd6:::benchmark_data_smithi188_141483_object73995:head# 2024-04-23T16:43:24.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca707c14:::benchmark_data_smithi188_141483_object70883:head# 2024-04-23T16:43:24.463 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca70ddc3:::benchmark_data_smithi188_141483_object64410:head# 2024-04-23T16:43:24.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca70fda2:::benchmark_data_smithi188_141483_object86488:head# 2024-04-23T16:43:24.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca7258e5:::benchmark_data_smithi188_141483_object87343:head# 2024-04-23T16:43:24.464 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca72d371:::benchmark_data_smithi188_141483_object9709:head# 2024-04-23T16:43:24.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca738856:::benchmark_data_smithi188_141483_object41745:head# 2024-04-23T16:43:24.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca742ffb:::benchmark_data_smithi188_141483_object80545:head# 2024-04-23T16:43:24.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca7482c5:::benchmark_data_smithi188_141483_object59971:head# 2024-04-23T16:43:24.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca750f53:::benchmark_data_smithi188_141483_object26207:head# 2024-04-23T16:43:24.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca752563:::benchmark_data_smithi188_141483_object77458:head# 2024-04-23T16:43:24.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca753635:::benchmark_data_smithi188_141483_object69258:head# 2024-04-23T16:43:24.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca76526b:::benchmark_data_smithi188_141483_object52594:head# 2024-04-23T16:43:24.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca76f8dd:::benchmark_data_smithi188_141483_object27829:head# 2024-04-23T16:43:24.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca77e32d:::benchmark_data_smithi188_141483_object28769:head# 2024-04-23T16:43:24.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca786d35:::benchmark_data_smithi188_141483_object63349:head# 2024-04-23T16:43:24.469 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca790e21:::benchmark_data_smithi188_141483_object80841:head# 2024-04-23T16:43:24.470 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca793df3:::benchmark_data_smithi188_141483_object34081:head# 2024-04-23T16:43:24.470 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca7b76b4:::benchmark_data_smithi188_141483_object51058:head# 2024-04-23T16:43:24.470 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca7c940e:::benchmark_data_smithi188_141483_object65358:head# 2024-04-23T16:43:24.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca7d923d:::benchmark_data_smithi188_141483_object80505:head# 2024-04-23T16:43:24.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca7db83c:::benchmark_data_smithi188_141483_object39765:head# 2024-04-23T16:43:24.471 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca7de7ff:::benchmark_data_smithi188_141483_object67935:head# 2024-04-23T16:43:24.472 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca802e09:::benchmark_data_smithi188_141483_object59902:head# 2024-04-23T16:43:24.473 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca80ead3:::benchmark_data_smithi188_141483_object81321:head# 2024-04-23T16:43:24.473 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca81b7c1:::benchmark_data_smithi188_141483_object24007:head# 2024-04-23T16:43:24.473 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca81c50d:::benchmark_data_smithi188_141483_object81182:head# 2024-04-23T16:43:24.473 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca820022:::benchmark_data_smithi188_141483_object20888:head# 2024-04-23T16:43:24.474 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca82198a:::benchmark_data_smithi188_141483_object21197:head# 2024-04-23T16:43:24.474 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca832489:::benchmark_data_smithi188_141483_object12333:head# 2024-04-23T16:43:24.474 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca83ceb4:::benchmark_data_smithi188_141483_object29762:head# 2024-04-23T16:43:24.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca844920:::benchmark_data_smithi188_141483_object31558:head# 2024-04-23T16:43:24.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca84986e:::benchmark_data_smithi188_141483_object29702:head# 2024-04-23T16:43:24.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca8613b1:::benchmark_data_smithi188_141483_object80016:head# 2024-04-23T16:43:24.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca896053:::benchmark_data_smithi188_141483_object8419:head# 2024-04-23T16:43:24.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca89735e:::benchmark_data_smithi188_141483_object90420:head# 2024-04-23T16:43:24.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca8ba1fb:::benchmark_data_smithi188_141483_object30265:head# 2024-04-23T16:43:24.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca8c5dba:::benchmark_data_smithi188_141483_object31784:head# 2024-04-23T16:43:24.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca8dc4e9:::benchmark_data_smithi188_141483_object26055:head# 2024-04-23T16:43:24.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca8dd9db:::benchmark_data_smithi188_141483_object80959:head# 2024-04-23T16:43:24.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca8e8855:::benchmark_data_smithi188_141483_object32503:head# 2024-04-23T16:43:24.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca8ec499:::benchmark_data_smithi188_141483_object45607:head# 2024-04-23T16:43:24.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca8f06db:::benchmark_data_smithi188_141483_object74582:head# 2024-04-23T16:43:24.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca8f3c9b:::benchmark_data_smithi188_141483_object30939:head# 2024-04-23T16:43:24.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca8f8c61:::benchmark_data_smithi188_141483_object16271:head# 2024-04-23T16:43:24.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca91de2f:::benchmark_data_smithi188_141483_object67307:head# 2024-04-23T16:43:24.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca935b3b:::benchmark_data_smithi188_141483_object22737:head# 2024-04-23T16:43:24.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca944f84:::benchmark_data_smithi188_141483_object80399:head# 2024-04-23T16:43:24.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca95d508:::benchmark_data_smithi188_141483_object42604:head# 2024-04-23T16:43:24.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca97039a:::benchmark_data_smithi188_141483_object41592:head# 2024-04-23T16:43:24.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca977180:::benchmark_data_smithi188_141483_object7691:head# 2024-04-23T16:43:24.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca979d3e:::benchmark_data_smithi188_141483_object82887:head# 2024-04-23T16:43:24.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca99e79f:::benchmark_data_smithi188_141483_object69517:head# 2024-04-23T16:43:24.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca9a50ae:::benchmark_data_smithi188_141483_object28488:head# 2024-04-23T16:43:24.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca9a9738:::benchmark_data_smithi188_141483_object52419:head# 2024-04-23T16:43:24.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca9a9a7e:::benchmark_data_smithi188_141483_object40283:head# 2024-04-23T16:43:24.486 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca9b192d:::benchmark_data_smithi188_141483_object3394:head# 2024-04-23T16:43:24.486 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca9bacfb:::benchmark_data_smithi188_141483_object88477:head# 2024-04-23T16:43:24.486 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca9bb29b:::benchmark_data_smithi188_141483_object24135:head# 2024-04-23T16:43:24.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca9c93e7:::benchmark_data_smithi188_141483_object60434:head# 2024-04-23T16:43:24.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca9da072:::benchmark_data_smithi188_141483_object25328:head# 2024-04-23T16:43:24.488 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ca9f6d5e:::benchmark_data_smithi188_141483_object14668:head# 2024-04-23T16:43:24.488 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caa0069a:::benchmark_data_smithi188_141483_object16313:head# 2024-04-23T16:43:24.488 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caa0202c:::benchmark_data_smithi188_141483_object20580:head# 2024-04-23T16:43:24.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caa09dfe:::benchmark_data_smithi188_141483_object49887:head# 2024-04-23T16:43:24.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caa13070:::benchmark_data_smithi188_141483_object71531:head# 2024-04-23T16:43:24.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caa33816:::benchmark_data_smithi188_141483_object3288:head# 2024-04-23T16:43:24.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caa3b5ca:::benchmark_data_smithi188_141483_object59943:head# 2024-04-23T16:43:24.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caa47ecb:::benchmark_data_smithi188_141483_object2000:head# 2024-04-23T16:43:24.491 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caa591ef:::benchmark_data_smithi188_141483_object10182:head# 2024-04-23T16:43:24.491 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caa60996:::benchmark_data_smithi188_141483_object54436:head# 2024-04-23T16:43:24.492 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caa65e42:::benchmark_data_smithi188_141483_object59997:head# 2024-04-23T16:43:24.492 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caa74f93:::benchmark_data_smithi188_141483_object51896:head# 2024-04-23T16:43:24.493 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caa83738:::benchmark_data_smithi188_141483_object78530:head# 2024-04-23T16:43:24.493 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caa9191b:::benchmark_data_smithi188_141483_object34485:head# 2024-04-23T16:43:24.494 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caa988d5:::benchmark_data_smithi188_141483_object23487:head# 2024-04-23T16:43:24.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caaa5ffb:::benchmark_data_smithi188_141483_object2855:head# 2024-04-23T16:43:24.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caab930f:::benchmark_data_smithi188_141483_object5313:head# 2024-04-23T16:43:24.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caabf2d4:::benchmark_data_smithi188_141483_object7511:head# 2024-04-23T16:43:24.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caac2454:::benchmark_data_smithi188_141483_object82379:head# 2024-04-23T16:43:24.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caac54cc:::benchmark_data_smithi188_141483_object2206:head# 2024-04-23T16:43:24.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caac63eb:::benchmark_data_smithi188_141483_object1043:head# 2024-04-23T16:43:24.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caac95ef:::benchmark_data_smithi188_141483_object14594:head# 2024-04-23T16:43:24.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caaced08:::benchmark_data_smithi188_141483_object43837:head# 2024-04-23T16:43:24.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caacf2f3:::benchmark_data_smithi188_141483_object39592:head# 2024-04-23T16:43:24.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caae69a2:::benchmark_data_smithi188_141483_object58907:head# 2024-04-23T16:43:24.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cab0a8df:::benchmark_data_smithi188_141483_object7952:head# 2024-04-23T16:43:24.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cab11c61:::benchmark_data_smithi188_141483_object89642:head# 2024-04-23T16:43:24.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cab14ff2:::benchmark_data_smithi188_141483_object32994:head# 2024-04-23T16:43:24.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cab1d376:::benchmark_data_smithi188_141483_object21101:head# 2024-04-23T16:43:24.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cab1ec04:::benchmark_data_smithi188_141483_object59231:head# 2024-04-23T16:43:24.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cab2091f:::benchmark_data_smithi188_141483_object9608:head# 2024-04-23T16:43:24.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cab22741:::benchmark_data_smithi188_141483_object17536:head# 2024-04-23T16:43:24.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cab2323c:::benchmark_data_smithi188_141483_object60179:head# 2024-04-23T16:43:24.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cab641f6:::benchmark_data_smithi188_141483_object68831:head# 2024-04-23T16:43:24.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cab890c7:::benchmark_data_smithi188_141483_object57432:head# 2024-04-23T16:43:24.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cab95452:::benchmark_data_smithi188_141483_object37864:head# 2024-04-23T16:43:24.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caba1a92:::benchmark_data_smithi188_141483_object56616:head# 2024-04-23T16:43:24.504 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caba7a16:::benchmark_data_smithi188_141483_object77024:head# 2024-04-23T16:43:24.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cabc55bf:::benchmark_data_smithi188_141483_object76056:head# 2024-04-23T16:43:24.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cabc6ab5:::benchmark_data_smithi188_141483_object46519:head# 2024-04-23T16:43:24.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cabd773b:::benchmark_data_smithi188_141483_object81779:head# 2024-04-23T16:43:24.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cabe52fe:::benchmark_data_smithi188_141483_object15121:head# 2024-04-23T16:43:24.507 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cabea079:::benchmark_data_smithi188_141483_object12121:head# 2024-04-23T16:43:24.507 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cabf5241:::benchmark_data_smithi188_141483_object87377:head# 2024-04-23T16:43:24.507 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cabf5525:::benchmark_data_smithi188_141483_object49189:head# 2024-04-23T16:43:24.509 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cabfb846:::benchmark_data_smithi188_141483_object29732:head# 2024-04-23T16:43:24.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cabfbe88:::benchmark_data_smithi188_141483_object14392:head# 2024-04-23T16:43:24.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cac06381:::benchmark_data_smithi188_141483_object67778:head# 2024-04-23T16:43:24.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cac2ede8:::benchmark_data_smithi188_141483_object85019:head# 2024-04-23T16:43:24.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cac314db:::benchmark_data_smithi188_141483_object88775:head# 2024-04-23T16:43:24.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cac36b2b:::benchmark_data_smithi188_141483_object49346:head# 2024-04-23T16:43:24.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cac3ad93:::benchmark_data_smithi188_141483_object85979:head# 2024-04-23T16:43:24.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cac54a3d:::benchmark_data_smithi188_141483_object2308:head# 2024-04-23T16:43:24.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cac55a7c:::benchmark_data_smithi188_141483_object52140:head# 2024-04-23T16:43:24.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cac5a9ad:::benchmark_data_smithi188_141483_object42286:head# 2024-04-23T16:43:24.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cac5ed0f:::benchmark_data_smithi188_141483_object19317:head# 2024-04-23T16:43:24.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cac66097:::benchmark_data_smithi188_141483_object35066:head# 2024-04-23T16:43:24.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cac6ca31:::benchmark_data_smithi188_141483_object23362:head# 2024-04-23T16:43:24.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cac7e210:::benchmark_data_smithi188_141483_object82747:head# 2024-04-23T16:43:24.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cac8c4cc:::benchmark_data_smithi188_141483_object63104:head# 2024-04-23T16:43:24.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cac9e0cb:::benchmark_data_smithi188_141483_object29314:head# 2024-04-23T16:43:24.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caca314e:::benchmark_data_smithi188_141483_object79264:head# 2024-04-23T16:43:24.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caca56fb:::benchmark_data_smithi188_141483_object38454:head# 2024-04-23T16:43:24.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caca6ca5:::benchmark_data_smithi188_141483_object32369:head# 2024-04-23T16:43:24.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cacb2887:::benchmark_data_smithi188_141483_object24265:head# 2024-04-23T16:43:24.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cacb5076:::benchmark_data_smithi188_141483_object230:head# 2024-04-23T16:43:24.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cacb840d:::benchmark_data_smithi188_141483_object53611:head# 2024-04-23T16:43:24.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cacc1adc:::benchmark_data_smithi188_141483_object39761:head# 2024-04-23T16:43:24.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caccd5d0:::benchmark_data_smithi188_141483_object58499:head# 2024-04-23T16:43:24.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cacce554:::benchmark_data_smithi188_141483_object4959:head# 2024-04-23T16:43:24.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cace05c2:::benchmark_data_smithi188_141483_object80253:head# 2024-04-23T16:43:24.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cacea436:::benchmark_data_smithi188_141483_object10537:head# 2024-04-23T16:43:24.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cad19fec:::benchmark_data_smithi188_141483_object38345:head# 2024-04-23T16:43:24.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cad2242e:::benchmark_data_smithi188_141483_object37821:head# 2024-04-23T16:43:24.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cad2bf9f:::benchmark_data_smithi188_141483_object29222:head# 2024-04-23T16:43:24.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cad2df02:::benchmark_data_smithi188_141483_object1420:head# 2024-04-23T16:43:24.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cad5dd7c:::benchmark_data_smithi188_141483_object54601:head# 2024-04-23T16:43:24.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cad632fc:::benchmark_data_smithi188_141483_object2163:head# 2024-04-23T16:43:24.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cad70171:::benchmark_data_smithi188_141483_object82783:head# 2024-04-23T16:43:24.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cad71559:::benchmark_data_smithi188_141483_object85367:head# 2024-04-23T16:43:24.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cad88422:::benchmark_data_smithi188_141483_object54743:head# 2024-04-23T16:43:24.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cad8f14f:::benchmark_data_smithi188_141483_object40033:head# 2024-04-23T16:43:24.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cad968d9:::benchmark_data_smithi188_141483_object81325:head# 2024-04-23T16:43:24.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cad96ee4:::benchmark_data_smithi188_141483_object74678:head# 2024-04-23T16:43:24.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cad9bd0e:::benchmark_data_smithi188_141483_object70539:head# 2024-04-23T16:43:24.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cad9ed18:::benchmark_data_smithi188_141483_object63771:head# 2024-04-23T16:43:24.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cada8fa8:::benchmark_data_smithi188_141483_object89519:head# 2024-04-23T16:43:24.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cadb78bb:::benchmark_data_smithi188_141483_object24934:head# 2024-04-23T16:43:24.536 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cadc5909:::benchmark_data_smithi188_141483_object42268:head# 2024-04-23T16:43:24.536 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cadde2c1:::benchmark_data_smithi188_141483_object22579:head# 2024-04-23T16:43:24.536 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cade7af7:::benchmark_data_smithi188_141483_object48766:head# 2024-04-23T16:43:24.536 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cadec3e4:::benchmark_data_smithi188_141483_object84535:head# 2024-04-23T16:43:24.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cadfaea4:::benchmark_data_smithi188_141483_object71529:head# 2024-04-23T16:43:24.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cae05d00:::benchmark_data_smithi188_141483_object36374:head# 2024-04-23T16:43:24.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cae0c71d:::benchmark_data_smithi188_141483_object14666:head# 2024-04-23T16:43:24.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cae217ec:::benchmark_data_smithi188_141483_object17670:head# 2024-04-23T16:43:24.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cae3e638:::benchmark_data_smithi188_141483_object80783:head# 2024-04-23T16:43:24.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cae424f9:::benchmark_data_smithi188_141483_object83454:head# 2024-04-23T16:43:24.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cae4ac7c:::benchmark_data_smithi188_141483_object48558:head# 2024-04-23T16:43:24.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cae5328d:::benchmark_data_smithi188_141483_object46657:head# 2024-04-23T16:43:24.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cae5994a:::benchmark_data_smithi188_141483_object65614:head# 2024-04-23T16:43:24.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cae62f3d:::benchmark_data_smithi188_141483_object82415:head# 2024-04-23T16:43:24.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cae69a95:::benchmark_data_smithi188_141483_object72537:head# 2024-04-23T16:43:24.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cae7f1f7:::benchmark_data_smithi188_141483_object87973:head# 2024-04-23T16:43:24.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cae8ce77:::benchmark_data_smithi188_141483_object64573:head# 2024-04-23T16:43:24.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cae91ed9:::benchmark_data_smithi188_141483_object12882:head# 2024-04-23T16:43:24.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caeb55e0:::benchmark_data_smithi188_141483_object2653:head# 2024-04-23T16:43:24.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caedfee0:::benchmark_data_smithi188_141483_object87300:head# 2024-04-23T16:43:24.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caefe464:::benchmark_data_smithi188_141483_object84185:head# 2024-04-23T16:43:24.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caf05e93:::benchmark_data_smithi188_141483_object5462:head# 2024-04-23T16:43:24.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caf123b9:::benchmark_data_smithi188_141483_object50502:head# 2024-04-23T16:43:24.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caf25bc8:::benchmark_data_smithi188_141483_object69620:head# 2024-04-23T16:43:24.541 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 73 16 104827 104811 89.7232 49.5625 0.00412192 0.0111425 2024-04-23T16:43:24.548 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caf40bfd:::benchmark_data_smithi188_141483_object89965:head# 2024-04-23T16:43:24.548 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caf541d7:::benchmark_data_smithi188_141483_object16821:head# 2024-04-23T16:43:24.548 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caf54248:::benchmark_data_smithi188_141483_object31292:head# 2024-04-23T16:43:24.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caf5db99:::benchmark_data_smithi188_141483_object86661:head# 2024-04-23T16:43:24.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caf6009b:::benchmark_data_smithi188_141483_object32600:head# 2024-04-23T16:43:24.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caf63dd9:::benchmark_data_smithi188_141483_object10825:head# 2024-04-23T16:43:24.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caf734ae:::benchmark_data_smithi188_141483_object60012:head# 2024-04-23T16:43:24.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caf8ced1:::benchmark_data_smithi188_141483_object25411:head# 2024-04-23T16:43:24.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caf91f46:::benchmark_data_smithi188_141483_object77589:head# 2024-04-23T16:43:24.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caf92c19:::benchmark_data_smithi188_141483_object8788:head# 2024-04-23T16:43:24.549 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cafa5f82:::benchmark_data_smithi188_141483_object17312:head# 2024-04-23T16:43:24.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cafe2a1a:::benchmark_data_smithi188_141483_object47190:head# 2024-04-23T16:43:24.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cafe9a99:::benchmark_data_smithi188_141483_object46363:head# 2024-04-23T16:43:24.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:caffc1a9:::benchmark_data_smithi188_141483_object9655:head# 2024-04-23T16:43:24.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb00830c:::benchmark_data_smithi188_141483_object57956:head# 2024-04-23T16:43:24.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb02058b:::benchmark_data_smithi188_141483_object52011:head# 2024-04-23T16:43:24.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb024483:::benchmark_data_smithi188_141483_object88397:head# 2024-04-23T16:43:24.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb036f55:::benchmark_data_smithi188_141483_object44270:head# 2024-04-23T16:43:24.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb04397b:::benchmark_data_smithi188_141483_object89769:head# 2024-04-23T16:43:24.550 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb04798c:::benchmark_data_smithi188_141483_object38412:head# 2024-04-23T16:43:24.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb0591c0:::benchmark_data_smithi188_141483_object19786:head# 2024-04-23T16:43:24.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb078920:::benchmark_data_smithi188_141483_object21737:head# 2024-04-23T16:43:24.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb07bb64:::benchmark_data_smithi188_141483_object19021:head# 2024-04-23T16:43:24.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb080d41:::benchmark_data_smithi188_141483_object62864:head# 2024-04-23T16:43:24.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb0a07c0:::benchmark_data_smithi188_141483_object16747:head# 2024-04-23T16:43:24.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb0a36a4:::benchmark_data_smithi188_141483_object47312:head# 2024-04-23T16:43:24.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb0dbbd2:::benchmark_data_smithi188_141483_object72587:head# 2024-04-23T16:43:24.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb0ddad2:::benchmark_data_smithi188_141483_object89080:head# 2024-04-23T16:43:24.551 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb0de72f:::benchmark_data_smithi188_141483_object17612:head# 2024-04-23T16:43:24.555 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb0e3fb5:::benchmark_data_smithi188_141483_object54322:head# 2024-04-23T16:43:24.555 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb0f5819:::benchmark_data_smithi188_141483_object59028:head# 2024-04-23T16:43:24.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb0fa0e8:::benchmark_data_smithi188_141483_object39160:head# 2024-04-23T16:43:24.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb115208:::benchmark_data_smithi188_141483_object4981:head# 2024-04-23T16:43:24.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb116afc:::benchmark_data_smithi188_141483_object83121:head# 2024-04-23T16:43:24.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb1197ad:::benchmark_data_smithi188_141483_object3452:head# 2024-04-23T16:43:24.556 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb1199d2:::benchmark_data_smithi188_141483_object87762:head# 2024-04-23T16:43:24.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb12af73:::benchmark_data_smithi188_141483_object29263:head# 2024-04-23T16:43:24.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb12c109:::benchmark_data_smithi188_141483_object15770:head# 2024-04-23T16:43:24.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb13089a:::benchmark_data_smithi188_141483_object52110:head# 2024-04-23T16:43:24.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb1355f3:::benchmark_data_smithi188_141483_object6610:head# 2024-04-23T16:43:24.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb136adb:::benchmark_data_smithi188_141483_object80501:head# 2024-04-23T16:43:24.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb137268:::benchmark_data_smithi188_141483_object84829:head# 2024-04-23T16:43:24.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb1384bb:::benchmark_data_smithi188_141483_object25103:head# 2024-04-23T16:43:24.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb138dbb:::benchmark_data_smithi188_141483_object22552:head# 2024-04-23T16:43:24.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb15043c:::benchmark_data_smithi188_141483_object81185:head# 2024-04-23T16:43:24.557 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb167907:::benchmark_data_smithi188_141483_object17257:head# 2024-04-23T16:43:24.557 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:43:24.561 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb16e223:::benchmark_data_smithi188_141483_object66929:head# 2024-04-23T16:43:24.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb17114c:::benchmark_data_smithi188_141483_object11085:head# 2024-04-23T16:43:24.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb185323:::benchmark_data_smithi188_141483_object17148:head# 2024-04-23T16:43:24.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb198fbc:::benchmark_data_smithi188_141483_object87826:head# 2024-04-23T16:43:24.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb1990f1:::benchmark_data_smithi188_141483_object17463:head# 2024-04-23T16:43:24.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb1999f3:::benchmark_data_smithi188_141483_object35442:head# 2024-04-23T16:43:24.562 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb199f55:::benchmark_data_smithi188_141483_object69268:head# 2024-04-23T16:43:24.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb19aab2:::benchmark_data_smithi188_141483_object54791:head# 2024-04-23T16:43:24.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb1a888d:::benchmark_data_smithi188_141483_object33807:head# 2024-04-23T16:43:24.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb1c1923:::benchmark_data_smithi188_141483_object8342:head# 2024-04-23T16:43:24.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb1cdb02:::benchmark_data_smithi188_141483_object77274:head# 2024-04-23T16:43:24.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb1ec734:::benchmark_data_smithi188_141483_object65669:head# 2024-04-23T16:43:24.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb1f4f74:::benchmark_data_smithi188_141483_object83553:head# 2024-04-23T16:43:24.563 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb206b86:::benchmark_data_smithi188_141483_object17854:head# 2024-04-23T16:43:24.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb2474e1:::benchmark_data_smithi188_141483_object85518:head# 2024-04-23T16:43:24.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb26a56b:::benchmark_data_smithi188_141483_object52615:head# 2024-04-23T16:43:24.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb27c1ba:::benchmark_data_smithi188_141483_object32212:head# 2024-04-23T16:43:24.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb296486:::benchmark_data_smithi188_141483_object71457:head# 2024-04-23T16:43:24.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb29786b:::benchmark_data_smithi188_141483_object32999:head# 2024-04-23T16:43:24.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb29f2a6:::benchmark_data_smithi188_141483_object90574:head# 2024-04-23T16:43:24.567 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb2b64f8:::benchmark_data_smithi188_141483_object34396:head# 2024-04-23T16:43:24.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb2c68e8:::benchmark_data_smithi188_141483_object11052:head# 2024-04-23T16:43:24.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb2ce697:::benchmark_data_smithi188_141483_object29471:head# 2024-04-23T16:43:24.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb2e9b33:::benchmark_data_smithi188_141483_object85776:head# 2024-04-23T16:43:24.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb2ea74f:::benchmark_data_smithi188_141483_object18494:head# 2024-04-23T16:43:24.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb2f6077:::benchmark_data_smithi188_141483_object21047:head# 2024-04-23T16:43:24.568 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb2f9ea5:::benchmark_data_smithi188_141483_object36342:head# 2024-04-23T16:43:24.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb2fa7b7:::benchmark_data_smithi188_141483_object16602:head# 2024-04-23T16:43:24.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb3094ec:::benchmark_data_smithi188_141483_object87670:head# 2024-04-23T16:43:24.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb30db32:::benchmark_data_smithi188_141483_object37321:head# 2024-04-23T16:43:24.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb313e20:::benchmark_data_smithi188_141483_object52055:head# 2024-04-23T16:43:24.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb3165ee:::benchmark_data_smithi188_141483_object43268:head# 2024-04-23T16:43:24.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb31d897:::benchmark_data_smithi188_141483_object11775:head# 2024-04-23T16:43:24.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb31df53:::benchmark_data_smithi188_141483_object50964:head# 2024-04-23T16:43:24.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb34b8d3:::benchmark_data_smithi188_141483_object88386:head# 2024-04-23T16:43:24.572 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb35bb9d:::benchmark_data_smithi188_141483_object70592:head# 2024-04-23T16:43:24.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb35efeb:::benchmark_data_smithi188_141483_object7405:head# 2024-04-23T16:43:24.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb36c6f2:::benchmark_data_smithi188_141483_object31643:head# 2024-04-23T16:43:24.573 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb3762f4:::benchmark_data_smithi188_141483_object70606:head# 2024-04-23T16:43:24.574 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb37fefc:::benchmark_data_smithi188_141483_object5269:head# 2024-04-23T16:43:24.574 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb384cc9:::benchmark_data_smithi188_141483_object12975:head# 2024-04-23T16:43:24.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb38c2bf:::benchmark_data_smithi188_141483_object15168:head# 2024-04-23T16:43:24.575 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb3d911d:::benchmark_data_smithi188_141483_object10733:head# 2024-04-23T16:43:24.576 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb3dd4ee:::benchmark_data_smithi188_141483_object26172:head# 2024-04-23T16:43:24.576 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb3e8ed8:::benchmark_data_smithi188_141483_object44517:head# 2024-04-23T16:43:24.576 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb3fcbd1:::benchmark_data_smithi188_141483_object54462:head# 2024-04-23T16:43:24.577 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb40d654:::benchmark_data_smithi188_141483_object61631:head# 2024-04-23T16:43:24.577 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb414997:::benchmark_data_smithi188_141483_object44339:head# 2024-04-23T16:43:24.577 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb41bc9b:::benchmark_data_smithi188_141483_object48155:head# 2024-04-23T16:43:24.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb43217d:::benchmark_data_smithi188_141483_object46996:head# 2024-04-23T16:43:24.578 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb4366ae:::benchmark_data_smithi188_141483_object59568:head# 2024-04-23T16:43:24.579 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb441eec:::benchmark_data_smithi188_141483_object82052:head# 2024-04-23T16:43:24.579 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb46d109:::benchmark_data_smithi188_141483_object1801:head# 2024-04-23T16:43:24.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb470643:::benchmark_data_smithi188_141483_object70527:head# 2024-04-23T16:43:24.580 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb474e5e:::benchmark_data_smithi188_141483_object66686:head# 2024-04-23T16:43:24.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb47b81b:::benchmark_data_smithi188_141483_object49792:head# 2024-04-23T16:43:24.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb47fe86:::benchmark_data_smithi188_141483_object78846:head# 2024-04-23T16:43:24.581 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb4821b3:::benchmark_data_smithi188_141483_object33605:head# 2024-04-23T16:43:24.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb4a032a:::benchmark_data_smithi188_141483_object71600:head# 2024-04-23T16:43:24.582 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb4a0824:::benchmark_data_smithi188_141483_object36941:head# 2024-04-23T16:43:24.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb4b746a:::benchmark_data_smithi188_141483_object73326:head# 2024-04-23T16:43:24.583 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb4c9af0:::benchmark_data_smithi188_141483_object60231:head# 2024-04-23T16:43:24.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb4d42a1:::benchmark_data_smithi188_141483_object38727:head# 2024-04-23T16:43:24.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb4dbcfc:::benchmark_data_smithi188_141483_object24690:head# 2024-04-23T16:43:24.584 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb4dedf0:::benchmark_data_smithi188_141483_object15428:head# 2024-04-23T16:43:24.585 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb50042f:::benchmark_data_smithi188_141483_object2543:head# 2024-04-23T16:43:24.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb501d42:::benchmark_data_smithi188_141483_object70058:head# 2024-04-23T16:43:24.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb503e55:::benchmark_data_smithi188_141483_object79835:head# 2024-04-23T16:43:24.587 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb506349:::benchmark_data_smithi188_141483_object40835:head# 2024-04-23T16:43:24.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb5067aa:::benchmark_data_smithi188_141483_object3925:head# 2024-04-23T16:43:24.589 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb5073c3:::benchmark_data_smithi188_141483_object44935:head# 2024-04-23T16:43:24.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb50eb48:::benchmark_data_smithi188_141483_object50364:head# 2024-04-23T16:43:24.590 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb5144f9:::benchmark_data_smithi188_141483_object64680:head# 2024-04-23T16:43:24.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb5189eb:::benchmark_data_smithi188_141483_object35629:head# 2024-04-23T16:43:24.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb52a88d:::benchmark_data_smithi188_141483_object26946:head# 2024-04-23T16:43:24.591 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb531162:::benchmark_data_smithi188_141483_object45983:head# 2024-04-23T16:43:24.592 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb534e6e:::benchmark_data_smithi188_141483_object666:head# 2024-04-23T16:43:24.592 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb551962:::benchmark_data_smithi188_141483_object73939:head# 2024-04-23T16:43:24.593 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb553710:::benchmark_data_smithi188_141483_object11324:head# 2024-04-23T16:43:24.593 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb555699:::benchmark_data_smithi188_141483_object20632:head# 2024-04-23T16:43:24.593 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb576be7:::benchmark_data_smithi188_141483_object86814:head# 2024-04-23T16:43:24.594 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb57a623:::benchmark_data_smithi188_141483_object45178:head# 2024-04-23T16:43:24.594 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb57b28f:::benchmark_data_smithi188_141483_object40937:head# 2024-04-23T16:43:24.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb58c819:::benchmark_data_smithi188_141483_object70600:head# 2024-04-23T16:43:24.595 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb5945cf:::benchmark_data_smithi188_141483_object6447:head# 2024-04-23T16:43:24.597 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb59a4b0:::benchmark_data_smithi188_141483_object28507:head# 2024-04-23T16:43:24.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb5a28d4:::benchmark_data_smithi188_141483_object58092:head# 2024-04-23T16:43:24.600 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb5a3e2d:::benchmark_data_smithi188_141483_object11492:head# 2024-04-23T16:43:24.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb5aa2e8:::benchmark_data_smithi188_141483_object43868:head# 2024-04-23T16:43:24.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb5bbb5c:::benchmark_data_smithi188_141483_object7244:head# 2024-04-23T16:43:24.601 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb5be4b0:::benchmark_data_smithi188_141483_object18453:head# 2024-04-23T16:43:24.602 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb5bff6f:::benchmark_data_smithi188_141483_object78040:head# 2024-04-23T16:43:24.602 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb5ce1cd:::benchmark_data_smithi188_141483_object41286:head# 2024-04-23T16:43:24.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb5e6a7f:::benchmark_data_smithi188_141483_object86310:head# 2024-04-23T16:43:24.603 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb5e79ca:::benchmark_data_smithi188_141483_object81384:head# 2024-04-23T16:43:24.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb5ebbe5:::benchmark_data_smithi188_141483_object9597:head# 2024-04-23T16:43:24.604 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb60ccba:::benchmark_data_smithi188_141483_object50937:head# 2024-04-23T16:43:24.605 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb613e6b:::benchmark_data_smithi188_141483_object35456:head# 2024-04-23T16:43:24.605 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb61c5b0:::benchmark_data_smithi188_141483_object82361:head# 2024-04-23T16:43:24.605 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb61c85d:::benchmark_data_smithi188_141483_object74065:head# 2024-04-23T16:43:24.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb61f620:::benchmark_data_smithi188_141483_object59420:head# 2024-04-23T16:43:24.606 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb62121c:::benchmark_data_smithi188_141483_object7812:head# 2024-04-23T16:43:24.608 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb62ccf5:::benchmark_data_smithi188_141483_object19718:head# 2024-04-23T16:43:24.609 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb635c38:::benchmark_data_smithi188_141483_object90799:head# 2024-04-23T16:43:24.609 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb63b20c:::benchmark_data_smithi188_141483_object70146:head# 2024-04-23T16:43:24.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb642e62:::benchmark_data_smithi188_141483_object81676:head# 2024-04-23T16:43:24.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb64e394:::benchmark_data_smithi188_141483_object2225:head# 2024-04-23T16:43:24.610 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb659c9d:::benchmark_data_smithi188_141483_object30814:head# 2024-04-23T16:43:24.611 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:24.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb65e314:::benchmark_data_smithi188_141483_object15804:head# 2024-04-23T16:43:24.611 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb66e697:::benchmark_data_smithi188_141483_object44793:head# 2024-04-23T16:43:24.612 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb670519:::benchmark_data_smithi188_141483_object58533:head# 2024-04-23T16:43:24.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb67bd58:::benchmark_data_smithi188_141483_object67649:head# 2024-04-23T16:43:24.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb67e68b:::benchmark_data_smithi188_141483_object14344:head# 2024-04-23T16:43:24.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb6863cc:::benchmark_data_smithi188_141483_object59717:head# 2024-04-23T16:43:24.613 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb690850:::benchmark_data_smithi188_141483_object71612:head# 2024-04-23T16:43:24.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb699b66:::benchmark_data_smithi188_141483_object41326:head# 2024-04-23T16:43:24.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb69ac3b:::benchmark_data_smithi188_141483_object65848:head# 2024-04-23T16:43:24.614 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb6a7bd3:::benchmark_data_smithi188_141483_object20780:head# 2024-04-23T16:43:24.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb6ad54d:::benchmark_data_smithi188_141483_object14122:head# 2024-04-23T16:43:24.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb6c2599:::benchmark_data_smithi188_141483_object27096:head# 2024-04-23T16:43:24.615 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb6c7685:::benchmark_data_smithi188_141483_object17333:head# 2024-04-23T16:43:24.616 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb6c83ce:::benchmark_data_smithi188_141483_object75225:head# 2024-04-23T16:43:24.616 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb6de878:::benchmark_data_smithi188_141483_object76773:head# 2024-04-23T16:43:24.617 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb6e5191:::benchmark_data_smithi188_141483_object67823:head# 2024-04-23T16:43:24.619 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb6e8144:::benchmark_data_smithi188_141483_object76632:head# 2024-04-23T16:43:24.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb6e834c:::benchmark_data_smithi188_141483_object33123:head# 2024-04-23T16:43:24.620 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb6f2110:::benchmark_data_smithi188_141483_object31793:head# 2024-04-23T16:43:24.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb70f624:::benchmark_data_smithi188_141483_object78845:head# 2024-04-23T16:43:24.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb714c82:::benchmark_data_smithi188_141483_object42373:head# 2024-04-23T16:43:24.621 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb749e45:::benchmark_data_smithi188_141483_object65244:head# 2024-04-23T16:43:24.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb74b2b4:::benchmark_data_smithi188_141483_object46292:head# 2024-04-23T16:43:24.622 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb762942:::benchmark_data_smithi188_141483_object28998:head# 2024-04-23T16:43:24.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb76db66:::benchmark_data_smithi188_141483_object29080:head# 2024-04-23T16:43:24.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb771574:::benchmark_data_smithi188_141483_object18607:head# 2024-04-23T16:43:24.623 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb7903d7:::benchmark_data_smithi188_141483_object62532:head# 2024-04-23T16:43:24.624 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb797de5:::benchmark_data_smithi188_141483_object30877:head# 2024-04-23T16:43:24.624 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb79a3dd:::benchmark_data_smithi188_141483_object27061:head# 2024-04-23T16:43:24.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb79f938:::benchmark_data_smithi188_141483_object39428:head# 2024-04-23T16:43:24.625 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb7ac580:::benchmark_data_smithi188_141483_object9311:head# 2024-04-23T16:43:24.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb7b7bfc:::benchmark_data_smithi188_141483_object16773:head# 2024-04-23T16:43:24.626 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb7ba191:::benchmark_data_smithi188_141483_object24444:head# 2024-04-23T16:43:24.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb7c5ecf:::benchmark_data_smithi188_141483_object71350:head# 2024-04-23T16:43:24.627 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb7ce6a2:::benchmark_data_smithi188_141483_object27717:head# 2024-04-23T16:43:24.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb7de4fa:::benchmark_data_smithi188_141483_object4482:head# 2024-04-23T16:43:24.628 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb7ed74f:::benchmark_data_smithi188_141483_object23186:head# 2024-04-23T16:43:24.629 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb7ee1b6:::benchmark_data_smithi188_141483_object76991:head# 2024-04-23T16:43:24.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb7f3474:::benchmark_data_smithi188_141483_object84596:head# 2024-04-23T16:43:24.630 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb7f866f:::benchmark_data_smithi188_141483_object83430:head# 2024-04-23T16:43:24.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb7fb9fa:::benchmark_data_smithi188_141483_object17117:head# 2024-04-23T16:43:24.631 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb8021ec:::benchmark_data_smithi188_141483_object83795:head# 2024-04-23T16:43:24.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb80584a:::benchmark_data_smithi188_141483_object28611:head# 2024-04-23T16:43:24.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb80e1aa:::benchmark_data_smithi188_141483_object76575:head# 2024-04-23T16:43:24.632 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb80faac:::benchmark_data_smithi188_141483_object49220:head# 2024-04-23T16:43:24.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb8171a8:::benchmark_data_smithi188_141483_object87257:head# 2024-04-23T16:43:24.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb828114:::benchmark_data_smithi188_141483_object34341:head# 2024-04-23T16:43:24.633 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb82f845:::benchmark_data_smithi188_141483_object32930:head# 2024-04-23T16:43:24.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb834b5e:::benchmark_data_smithi188_141483_object69019:head# 2024-04-23T16:43:24.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb8673cd:::benchmark_data_smithi188_141483_object42323:head# 2024-04-23T16:43:24.634 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb8673fa:::benchmark_data_smithi188_141483_object30178:head# 2024-04-23T16:43:24.635 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb867422:::benchmark_data_smithi188_141483_object68527:head# 2024-04-23T16:43:24.635 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb869d5e:::benchmark_data_smithi188_141483_object40494:head# 2024-04-23T16:43:24.635 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb86bc79:::benchmark_data_smithi188_141483_object6745:head# 2024-04-23T16:43:24.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb880bf7:::benchmark_data_smithi188_141483_object60750:head# 2024-04-23T16:43:24.636 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb88ae96:::benchmark_data_smithi188_141483_object40473:head# 2024-04-23T16:43:24.637 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb895dc2:::benchmark_data_smithi188_141483_object18636:head# 2024-04-23T16:43:24.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb895f81:::benchmark_data_smithi188_141483_object28585:head# 2024-04-23T16:43:24.638 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb8a6a66:::benchmark_data_smithi188_141483_object63760:head# 2024-04-23T16:43:24.639 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:43:24.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb8ac78e:::benchmark_data_smithi188_141483_object2971:head# 2024-04-23T16:43:24.639 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb8c413b:::benchmark_data_smithi188_141483_object5591:head# 2024-04-23T16:43:24.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb8c7229:::benchmark_data_smithi188_141483_object16398:head# 2024-04-23T16:43:24.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb8cd64c:::benchmark_data_smithi188_141483_object75759:head# 2024-04-23T16:43:24.640 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb8dd59c:::benchmark_data_smithi188_141483_object48362:head# 2024-04-23T16:43:24.641 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb8e2b40:::benchmark_data_smithi188_141483_object41866:head# 2024-04-23T16:43:24.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb8e8560:::benchmark_data_smithi188_141483_object77723:head# 2024-04-23T16:43:24.642 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb8ef08c:::benchmark_data_smithi188_141483_object70575:head# 2024-04-23T16:43:24.643 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb927ebe:::benchmark_data_smithi188_141483_object21329:head# 2024-04-23T16:43:24.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:24.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:43:24.645 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb946159:::benchmark_data_smithi188_141483_object43266:head# 2024-04-23T16:43:24.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb94923b:::benchmark_data_smithi188_141483_object5040:head# 2024-04-23T16:43:24.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb958a44:::benchmark_data_smithi188_141483_object48495:head# 2024-04-23T16:43:24.646 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb966ba2:::benchmark_data_smithi188_141483_object47658:head# 2024-04-23T16:43:24.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb96c5d6:::benchmark_data_smithi188_141483_object79981:head# 2024-04-23T16:43:24.647 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb971336:::benchmark_data_smithi188_141483_object57939:head# 2024-04-23T16:43:24.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb97b677:::benchmark_data_smithi188_141483_object62014:head# 2024-04-23T16:43:24.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb991ba0:::benchmark_data_smithi188_141483_object4356:head# 2024-04-23T16:43:24.648 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb99a930:::benchmark_data_smithi188_141483_object50980:head# 2024-04-23T16:43:24.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb9ab669:::benchmark_data_smithi188_141483_object43445:head# 2024-04-23T16:43:24.649 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb9bc0da:::benchmark_data_smithi188_141483_object28536:head# 2024-04-23T16:43:24.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb9df3ce:::benchmark_data_smithi188_141483_object39696:head# 2024-04-23T16:43:24.650 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cb9fc84a:::benchmark_data_smithi188_141483_object44398:head# 2024-04-23T16:43:24.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cba07133:::benchmark_data_smithi188_141483_object77211:head# 2024-04-23T16:43:24.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cba1512e:::benchmark_data_smithi188_141483_object71266:head# 2024-04-23T16:43:24.651 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cba1b25a:::benchmark_data_smithi188_141483_object75619:head# 2024-04-23T16:43:24.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cba2fb49:::benchmark_data_smithi188_141483_object14312:head# 2024-04-23T16:43:24.652 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cba32fac:::benchmark_data_smithi188_141483_object87287:head# 2024-04-23T16:43:24.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cba546a6:::benchmark_data_smithi188_141483_object29838:head# 2024-04-23T16:43:24.653 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cba6723a:::benchmark_data_smithi188_141483_object38017:head# 2024-04-23T16:43:24.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cba6bb9e:::benchmark_data_smithi188_141483_object52245:head# 2024-04-23T16:43:24.654 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cba79ce3:::benchmark_data_smithi188_141483_object28248:head# 2024-04-23T16:43:24.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cba7fc43:::benchmark_data_smithi188_141483_object28083:head# 2024-04-23T16:43:24.655 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cba80da1:::benchmark_data_smithi188_141483_object81391:head# 2024-04-23T16:43:24.656 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cba84312:::benchmark_data_smithi188_141483_object77669:head# 2024-04-23T16:43:24.698 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbaa1e97:::benchmark_data_smithi188_141483_object42614:head# 2024-04-23T16:43:24.699 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbaa6a6d:::benchmark_data_smithi188_141483_object31936:head# 2024-04-23T16:43:24.699 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbab2985:::benchmark_data_smithi188_141483_object42150:head# 2024-04-23T16:43:24.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbab72de:::benchmark_data_smithi188_141483_object11033:head# 2024-04-23T16:43:24.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbabf723:::benchmark_data_smithi188_141483_object70743:head# 2024-04-23T16:43:24.700 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbac2f27:::benchmark_data_smithi188_141483_object80975:head# 2024-04-23T16:43:24.701 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbacc067:::benchmark_data_smithi188_141483_object5032:head# 2024-04-23T16:43:24.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbad426b:::benchmark_data_smithi188_141483_object5096:head# 2024-04-23T16:43:24.702 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbad7363:::benchmark_data_smithi188_141483_object52577:head# 2024-04-23T16:43:24.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbae2279:::benchmark_data_smithi188_141483_object60652:head# 2024-04-23T16:43:24.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbae5c24:::benchmark_data_smithi188_141483_object77140:head# 2024-04-23T16:43:24.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbaeb170:::benchmark_data_smithi188_141483_object8468:head# 2024-04-23T16:43:24.703 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbaed4b0:::benchmark_data_smithi188_141483_object83245:head# 2024-04-23T16:43:24.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbafb361:::benchmark_data_smithi188_141483_object80822:head# 2024-04-23T16:43:24.704 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbb03f06:::benchmark_data_smithi188_141483_object57336:head# 2024-04-23T16:43:24.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbb0f575:::benchmark_data_smithi188_141483_object50886:head# 2024-04-23T16:43:24.705 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbb261fe:::benchmark_data_smithi188_141483_object55593:head# 2024-04-23T16:43:24.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbb39ce2:::benchmark_data_smithi188_141483_object6713:head# 2024-04-23T16:43:24.707 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbb41e0b:::benchmark_data_smithi188_141483_object62202:head# 2024-04-23T16:43:24.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbb4f465:::benchmark_data_smithi188_141483_object71763:head# 2024-04-23T16:43:24.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbb4fde0:::benchmark_data_smithi188_141483_object15397:head# 2024-04-23T16:43:24.708 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbb5f2d7:::benchmark_data_smithi188_141483_object75506:head# 2024-04-23T16:43:24.709 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbb6e876:::benchmark_data_smithi188_141483_object68286:head# 2024-04-23T16:43:24.709 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbb7542f:::benchmark_data_smithi188_141483_object89014:head# 2024-04-23T16:43:24.710 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbb81621:::benchmark_data_smithi188_141483_object49691:head# 2024-04-23T16:43:24.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbb90655:::benchmark_data_smithi188_141483_object1651:head# 2024-04-23T16:43:24.711 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbb9879e:::benchmark_data_smithi188_141483_object35803:head# 2024-04-23T16:43:24.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbbb418e:::benchmark_data_smithi188_141483_object66955:head# 2024-04-23T16:43:24.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbbb469f:::benchmark_data_smithi188_141483_object65972:head# 2024-04-23T16:43:24.712 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbbc5067:::benchmark_data_smithi188_141483_object61804:head# 2024-04-23T16:43:24.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbbc74ed:::benchmark_data_smithi188_141483_object80214:head# 2024-04-23T16:43:24.713 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbbd9ec2:::benchmark_data_smithi188_141483_object46180:head# 2024-04-23T16:43:24.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbbf7978:::benchmark_data_smithi188_141483_object84479:head# 2024-04-23T16:43:24.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbbf7da9:::benchmark_data_smithi188_141483_object4421:head# 2024-04-23T16:43:24.714 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbc15e31:::benchmark_data_smithi188_141483_object42029:head# 2024-04-23T16:43:24.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbc1dd80:::benchmark_data_smithi188_141483_object78745:head# 2024-04-23T16:43:24.715 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbc2bf15:::benchmark_data_smithi188_141483_object81158:head# 2024-04-23T16:43:24.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbc30bd5:::benchmark_data_smithi188_141483_object29294:head# 2024-04-23T16:43:24.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbc4b60a:::benchmark_data_smithi188_141483_object34654:head# 2024-04-23T16:43:24.716 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbc4e8cf:::benchmark_data_smithi188_141483_object16676:head# 2024-04-23T16:43:24.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbc4f342:::benchmark_data_smithi188_141483_object36366:head# 2024-04-23T16:43:24.717 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbc57e54:::benchmark_data_smithi188_141483_object47711:head# 2024-04-23T16:43:24.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbc57ebd:::benchmark_data_smithi188_141483_object47566:head# 2024-04-23T16:43:24.718 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbc5b934:::benchmark_data_smithi188_141483_object43587:head# 2024-04-23T16:43:24.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbc8c252:::benchmark_data_smithi188_141483_object27869:head# 2024-04-23T16:43:24.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbca2ab3:::benchmark_data_smithi188_141483_object18796:head# 2024-04-23T16:43:24.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbca4b3c:::benchmark_data_smithi188_141483_object30795:head# 2024-04-23T16:43:24.719 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbcaad70:::benchmark_data_smithi188_141483_object50221:head# 2024-04-23T16:43:24.720 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbcb196a:::benchmark_data_smithi188_141483_object57256:head# 2024-04-23T16:43:24.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbcc3d47:::benchmark_data_smithi188_141483_object66432:head# 2024-04-23T16:43:24.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbcc50eb:::benchmark_data_smithi188_141483_object466:head# 2024-04-23T16:43:24.721 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbce1113:::benchmark_data_smithi188_141483_object31185:head# 2024-04-23T16:43:24.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbce3102:::benchmark_data_smithi188_141483_object42651:head# 2024-04-23T16:43:24.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbcfac94:::benchmark_data_smithi188_141483_object10943:head# 2024-04-23T16:43:24.722 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbd11b3b:::benchmark_data_smithi188_141483_object14903:head# 2024-04-23T16:43:24.723 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbd182d2:::benchmark_data_smithi188_141483_object65660:head# 2024-04-23T16:43:24.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbd1b0b7:::benchmark_data_smithi188_141483_object3194:head# 2024-04-23T16:43:24.726 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbd38391:::benchmark_data_smithi188_141483_object67704:head# 2024-04-23T16:43:24.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbd40f9c:::benchmark_data_smithi188_141483_object21314:head# 2024-04-23T16:43:24.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbd42cae:::benchmark_data_smithi188_141483_object65819:head# 2024-04-23T16:43:24.727 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbd661d1:::benchmark_data_smithi188_141483_object72230:head# 2024-04-23T16:43:24.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbd85051:::benchmark_data_smithi188_141483_object64849:head# 2024-04-23T16:43:24.730 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbd851f8:::benchmark_data_smithi188_141483_object26106:head# 2024-04-23T16:43:24.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbd8f363:::benchmark_data_smithi188_141483_object83518:head# 2024-04-23T16:43:24.731 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbd9df2e:::benchmark_data_smithi188_141483_object82064:head# 2024-04-23T16:43:24.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbda0683:::benchmark_data_smithi188_141483_object13813:head# 2024-04-23T16:43:24.732 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbdb01d7:::benchmark_data_smithi188_141483_object86585:head# 2024-04-23T16:43:24.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbdc67c1:::benchmark_data_smithi188_141483_object27657:head# 2024-04-23T16:43:24.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbdd55b2:::benchmark_data_smithi188_141483_object24403:head# 2024-04-23T16:43:24.733 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbddbb9c:::benchmark_data_smithi188_141483_object34666:head# 2024-04-23T16:43:24.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbddc446:::benchmark_data_smithi188_141483_object44958:head# 2024-04-23T16:43:24.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbde75ab:::benchmark_data_smithi188_141483_object55861:head# 2024-04-23T16:43:24.734 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbdfe7aa:::benchmark_data_smithi188_141483_object79938:head# 2024-04-23T16:43:24.736 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbe15a12:::benchmark_data_smithi188_141483_object39256:head# 2024-04-23T16:43:24.737 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbe3749d:::benchmark_data_smithi188_141483_object85540:head# 2024-04-23T16:43:24.737 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbe4130a:::benchmark_data_smithi188_141483_object73004:head# 2024-04-23T16:43:24.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbe57a61:::benchmark_data_smithi188_141483_object75922:head# 2024-04-23T16:43:24.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbe589ca:::benchmark_data_smithi188_141483_object84721:head# 2024-04-23T16:43:24.738 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbe7da8c:::benchmark_data_smithi188_141483_object38680:head# 2024-04-23T16:43:24.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbe7e7da:::benchmark_data_smithi188_141483_object11398:head# 2024-04-23T16:43:24.739 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbe83ae3:::benchmark_data_smithi188_141483_object5413:head# 2024-04-23T16:43:24.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbe9d565:::benchmark_data_smithi188_141483_object18044:head# 2024-04-23T16:43:24.740 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbe9f733:::benchmark_data_smithi188_141483_object57767:head# 2024-04-23T16:43:24.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbebba28:::benchmark_data_smithi188_141483_object47601:head# 2024-04-23T16:43:24.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbed135f:::benchmark_data_smithi188_141483_object90908:head# 2024-04-23T16:43:24.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbed42ce:::benchmark_data_smithi188_141483_object83895:head# 2024-04-23T16:43:24.741 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbedc41e:::benchmark_data_smithi188_141483_object78680:head# 2024-04-23T16:43:24.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbee5b9b:::benchmark_data_smithi188_141483_object40451:head# 2024-04-23T16:43:24.742 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbee8894:::benchmark_data_smithi188_141483_object42704:head# 2024-04-23T16:43:24.743 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbeebc7d:::benchmark_data_smithi188_141483_object3813:head# 2024-04-23T16:43:24.744 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbeed14d:::benchmark_data_smithi188_141483_object88531:head# 2024-04-23T16:43:24.744 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:43:24.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbeeda11:::benchmark_data_smithi188_141483_object26873:head# 2024-04-23T16:43:24.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbef8360:::benchmark_data_smithi188_141483_object41632:head# 2024-04-23T16:43:24.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbef900a:::benchmark_data_smithi188_141483_object14749:head# 2024-04-23T16:43:24.746 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbf033b5:::benchmark_data_smithi188_141483_object66855:head# 2024-04-23T16:43:24.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbf135e9:::benchmark_data_smithi188_141483_object43250:head# 2024-04-23T16:43:24.748 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbf15c88:::benchmark_data_smithi188_141483_object38813:head# 2024-04-23T16:43:24.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbf30683:::benchmark_data_smithi188_141483_object72767:head# 2024-04-23T16:43:24.749 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbf38dd1:::benchmark_data_smithi188_141483_object37799:head# 2024-04-23T16:43:24.750 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbf3a708:::benchmark_data_smithi188_141483_object38963:head# 2024-04-23T16:43:24.750 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbf3e31b:::benchmark_data_smithi188_141483_object27696:head# 2024-04-23T16:43:24.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbf51c33:::benchmark_data_smithi188_141483_object54018:head# 2024-04-23T16:43:24.751 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbf594db:::benchmark_data_smithi188_141483_object47022:head# 2024-04-23T16:43:24.752 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbf6c072:::benchmark_data_smithi188_141483_object77161:head# 2024-04-23T16:43:24.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbf6c921:::benchmark_data_smithi188_141483_object35396:head# 2024-04-23T16:43:24.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbf803fb:::benchmark_data_smithi188_141483_object17397:head# 2024-04-23T16:43:24.754 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbf84aa3:::benchmark_data_smithi188_141483_object67012:head# 2024-04-23T16:43:24.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbf84fcc:::benchmark_data_smithi188_141483_object245:head# 2024-04-23T16:43:24.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbf857c6:::benchmark_data_smithi188_141483_object61956:head# 2024-04-23T16:43:24.755 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbf99b72:::benchmark_data_smithi188_141483_object70206:head# 2024-04-23T16:43:24.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbfb2929:::benchmark_data_smithi188_141483_object9436:head# 2024-04-23T16:43:24.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbfb40ae:::benchmark_data_smithi188_141483_object80773:head# 2024-04-23T16:43:24.758 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbfba78a:::benchmark_data_smithi188_141483_object38863:head# 2024-04-23T16:43:24.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbfc92bc:::benchmark_data_smithi188_141483_object3523:head# 2024-04-23T16:43:24.759 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbfd9d6d:::benchmark_data_smithi188_141483_object33875:head# 2024-04-23T16:43:24.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbff88d6:::benchmark_data_smithi188_141483_object68520:head# 2024-04-23T16:43:24.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cbffa79f:::benchmark_data_smithi188_141483_object4946:head# 2024-04-23T16:43:24.760 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc000043:::benchmark_data_smithi188_141483_object69992:head# 2024-04-23T16:43:24.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc00d388:::benchmark_data_smithi188_141483_object44739:head# 2024-04-23T16:43:24.761 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc035781:::benchmark_data_smithi188_141483_object17789:head# 2024-04-23T16:43:24.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc03779f:::benchmark_data_smithi188_141483_object15935:head# 2024-04-23T16:43:24.762 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc0394fd:::benchmark_data_smithi188_141483_object42343:head# 2024-04-23T16:43:24.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc046573:::benchmark_data_smithi188_141483_object59304:head# 2024-04-23T16:43:24.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc04c559:::benchmark_data_smithi188_141483_object63166:head# 2024-04-23T16:43:24.763 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc063a13:::benchmark_data_smithi188_141483_object70392:head# 2024-04-23T16:43:24.764 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc06d210:::benchmark_data_smithi188_141483_object15527:head# 2024-04-23T16:43:24.764 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc06f056:::benchmark_data_smithi188_141483_object80640:head# 2024-04-23T16:43:24.765 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc080489:::benchmark_data_smithi188_141483_object82659:head# 2024-04-23T16:43:24.767 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc0a605d:::benchmark_data_smithi188_141483_object23550:head# 2024-04-23T16:43:24.768 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc0b67be:::benchmark_data_smithi188_141483_object27085:head# 2024-04-23T16:43:24.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc0bcd98:::benchmark_data_smithi188_141483_object15240:head# 2024-04-23T16:43:24.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc0c9364:::benchmark_data_smithi188_141483_object66198:head# 2024-04-23T16:43:24.769 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc0dbebc:::benchmark_data_smithi188_141483_object44095:head# 2024-04-23T16:43:24.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc0e83f6:::benchmark_data_smithi188_141483_object18794:head# 2024-04-23T16:43:24.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc0fa3e2:::benchmark_data_smithi188_141483_object36707:head# 2024-04-23T16:43:24.770 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc0ff34d:::benchmark_data_smithi188_141483_object23627:head# 2024-04-23T16:43:24.771 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc108cd7:::benchmark_data_smithi188_141483_object85463:head# 2024-04-23T16:43:24.773 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc149edd:::benchmark_data_smithi188_141483_object46493:head# 2024-04-23T16:43:24.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc159ef1:::benchmark_data_smithi188_141483_object79299:head# 2024-04-23T16:43:24.774 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc15b167:::benchmark_data_smithi188_141483_object5024:head# 2024-04-23T16:43:24.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc15d948:::benchmark_data_smithi188_141483_object5578:head# 2024-04-23T16:43:24.775 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc15daca:::benchmark_data_smithi188_141483_object37373:head# 2024-04-23T16:43:24.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc16184c:::benchmark_data_smithi188_141483_object34267:head# 2024-04-23T16:43:24.776 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:24.776 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc165d75:::benchmark_data_smithi188_141483_object56847:head# 2024-04-23T16:43:24.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc177093:::benchmark_data_smithi188_141483_object84369:head# 2024-04-23T16:43:24.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc178ac3:::benchmark_data_smithi188_141483_object16192:head# 2024-04-23T16:43:24.777 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc1839eb:::benchmark_data_smithi188_141483_object32792:head# 2024-04-23T16:43:24.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc18d99b:::benchmark_data_smithi188_141483_object22947:head# 2024-04-23T16:43:24.778 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc18df96:::benchmark_data_smithi188_141483_object30797:head# 2024-04-23T16:43:24.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc18f464:::benchmark_data_smithi188_141483_object92:head# 2024-04-23T16:43:24.779 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc190215:::benchmark_data_smithi188_141483_object23724:head# 2024-04-23T16:43:24.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc1a9da2:::benchmark_data_smithi188_141483_object27197:head# 2024-04-23T16:43:24.780 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc1b28e6:::benchmark_data_smithi188_141483_object64568:head# 2024-04-23T16:43:24.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc1c9b45:::benchmark_data_smithi188_141483_object28634:head# 2024-04-23T16:43:24.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc1ce218:::benchmark_data_smithi188_141483_object18546:head# 2024-04-23T16:43:24.781 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc1db456:::benchmark_data_smithi188_141483_object88212:head# 2024-04-23T16:43:24.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc1e1ddc:::benchmark_data_smithi188_141483_object61224:head# 2024-04-23T16:43:24.782 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc1e8b4b:::benchmark_data_smithi188_141483_object69875:head# 2024-04-23T16:43:24.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc1f35f0:::benchmark_data_smithi188_141483_object89189:head# 2024-04-23T16:43:24.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc1ffa0c:::benchmark_data_smithi188_141483_object48800:head# 2024-04-23T16:43:24.783 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc218a24:::benchmark_data_smithi188_141483_object86442:head# 2024-04-23T16:43:24.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc218bb5:::benchmark_data_smithi188_141483_object39248:head# 2024-04-23T16:43:24.784 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc223189:::benchmark_data_smithi188_141483_object53252:head# 2024-04-23T16:43:24.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc23c169:::benchmark_data_smithi188_141483_object14408:head# 2024-04-23T16:43:24.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc23e332:::benchmark_data_smithi188_141483_object75362:head# 2024-04-23T16:43:24.785 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc240654:::benchmark_data_smithi188_141483_object56345:head# 2024-04-23T16:43:24.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc250b4c:::benchmark_data_smithi188_141483_object59639:head# 2024-04-23T16:43:24.786 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc251f80:::benchmark_data_smithi188_141483_object60098:head# 2024-04-23T16:43:24.787 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc268d13:::benchmark_data_smithi188_141483_object30862:head# 2024-04-23T16:43:24.788 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc26e621:::benchmark_data_smithi188_141483_object36532:head# 2024-04-23T16:43:24.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc2780ee:::benchmark_data_smithi188_141483_object49666:head# 2024-04-23T16:43:24.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc279247:::benchmark_data_smithi188_141483_object10223:head# 2024-04-23T16:43:24.789 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc27aefd:::benchmark_data_smithi188_141483_object17592:head# 2024-04-23T16:43:24.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc28831a:::benchmark_data_smithi188_141483_object22962:head# 2024-04-23T16:43:24.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc2a13dc:::benchmark_data_smithi188_141483_object61692:head# 2024-04-23T16:43:24.790 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc2c7d64:::benchmark_data_smithi188_141483_object72173:head# 2024-04-23T16:43:24.791 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc2c971b:::benchmark_data_smithi188_141483_object58900:head# 2024-04-23T16:43:24.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc2e99b7:::benchmark_data_smithi188_141483_object10806:head# 2024-04-23T16:43:24.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc2f21d0:::benchmark_data_smithi188_141483_object36164:head# 2024-04-23T16:43:24.792 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc30320d:::benchmark_data_smithi188_141483_object41772:head# 2024-04-23T16:43:24.793 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc309816:::benchmark_data_smithi188_141483_object22082:head# 2024-04-23T16:43:24.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc337673:::benchmark_data_smithi188_141483_object68486:head# 2024-04-23T16:43:24.794 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc34072f:::benchmark_data_smithi188_141483_object27340:head# 2024-04-23T16:43:24.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc35dae0:::benchmark_data_smithi188_141483_object9621:head# 2024-04-23T16:43:24.795 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc368a25:::benchmark_data_smithi188_141483_object21167:head# 2024-04-23T16:43:24.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc397dc2:::benchmark_data_smithi188_141483_object78826:head# 2024-04-23T16:43:24.796 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc39ef48:::benchmark_data_smithi188_141483_object76842:head# 2024-04-23T16:43:24.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc3a2cb3:::benchmark_data_smithi188_141483_object58018:head# 2024-04-23T16:43:24.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc3ab5a2:::benchmark_data_smithi188_141483_object40526:head# 2024-04-23T16:43:24.797 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc3ba101:::benchmark_data_smithi188_141483_object32615:head# 2024-04-23T16:43:24.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc3c8521:::benchmark_data_smithi188_141483_object32747:head# 2024-04-23T16:43:24.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc3cc6f6:::benchmark_data_smithi188_141483_object20072:head# 2024-04-23T16:43:24.798 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc3eb343:::benchmark_data_smithi188_141483_object13821:head# 2024-04-23T16:43:24.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc3edb3e:::benchmark_data_smithi188_141483_object17308:head# 2024-04-23T16:43:24.799 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc3fe47d:::benchmark_data_smithi188_141483_object32937:head# 2024-04-23T16:43:24.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc407f07:::benchmark_data_smithi188_141483_object81251:head# 2024-04-23T16:43:24.800 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc416bcd:::benchmark_data_smithi188_141483_object14780:head# 2024-04-23T16:43:24.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc41aad9:::benchmark_data_smithi188_141483_object7427:head# 2024-04-23T16:43:24.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc41c919:::benchmark_data_smithi188_141483_object27994:head# 2024-04-23T16:43:24.801 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc41d97b:::benchmark_data_smithi188_141483_object5993:head# 2024-04-23T16:43:24.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc42ffe8:::benchmark_data_smithi188_141483_object56187:head# 2024-04-23T16:43:24.802 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc45cc0c:::benchmark_data_smithi188_141483_object25946:head# 2024-04-23T16:43:24.803 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc465510:::benchmark_data_smithi188_141483_object89163:head# 2024-04-23T16:43:24.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc46772e:::benchmark_data_smithi188_141483_object9:head# 2024-04-23T16:43:24.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc46cc9a:::benchmark_data_smithi188_141483_object39516:head# 2024-04-23T16:43:24.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc481a84:::benchmark_data_smithi188_141483_object75766:head# 2024-04-23T16:43:24.804 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc489b47:::benchmark_data_smithi188_141483_object24426:head# 2024-04-23T16:43:24.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc48c69f:::benchmark_data_smithi188_141483_object45990:head# 2024-04-23T16:43:24.805 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc48ddaf:::benchmark_data_smithi188_141483_object56112:head# 2024-04-23T16:43:24.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc48eb18:::benchmark_data_smithi188_141483_object28474:head# 2024-04-23T16:43:24.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc4a6308:::benchmark_data_smithi188_141483_object17624:head# 2024-04-23T16:43:24.806 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc4ab0b9:::benchmark_data_smithi188_141483_object6070:head# 2024-04-23T16:43:24.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc4b5fd4:::benchmark_data_smithi188_141483_object84122:head# 2024-04-23T16:43:24.807 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc4b8d4f:::benchmark_data_smithi188_141483_object57054:head# 2024-04-23T16:43:24.808 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc4bf8f0:::benchmark_data_smithi188_141483_object26094:head# 2024-04-23T16:43:24.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc4cc7bd:::benchmark_data_smithi188_141483_object51859:head# 2024-04-23T16:43:24.810 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc4e1166:::benchmark_data_smithi188_141483_object54940:head# 2024-04-23T16:43:24.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc4e4347:::benchmark_data_smithi188_141483_object84134:head# 2024-04-23T16:43:24.811 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc4f076c:::benchmark_data_smithi188_141483_object35542:head# 2024-04-23T16:43:24.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc507305:::benchmark_data_smithi188_141483_object75678:head# 2024-04-23T16:43:24.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc5232e6:::benchmark_data_smithi188_141483_object51984:head# 2024-04-23T16:43:24.812 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc525720:::benchmark_data_smithi188_141483_object66452:head# 2024-04-23T16:43:24.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc525adc:::benchmark_data_smithi188_141483_object29932:head# 2024-04-23T16:43:24.813 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc525d60:::benchmark_data_smithi188_141483_object47150:head# 2024-04-23T16:43:24.814 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc52d5ba:::benchmark_data_smithi188_141483_object1595:head# 2024-04-23T16:43:24.814 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc52e987:::benchmark_data_smithi188_141483_object80434:head# 2024-04-23T16:43:24.814 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc53a9ab:::benchmark_data_smithi188_141483_object11264:head# 2024-04-23T16:43:24.815 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc5697a8:::benchmark_data_smithi188_141483_object45174:head# 2024-04-23T16:43:24.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc586103:::benchmark_data_smithi188_141483_object76858:head# 2024-04-23T16:43:24.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc5934ca:::benchmark_data_smithi188_141483_object76544:head# 2024-04-23T16:43:24.816 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc5be4d2:::benchmark_data_smithi188_141483_object5198:head# 2024-04-23T16:43:24.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc5c1bc0:::benchmark_data_smithi188_141483_object33371:head# 2024-04-23T16:43:24.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc5c4038:::benchmark_data_smithi188_141483_object72443:head# 2024-04-23T16:43:24.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc5cbbcb:::benchmark_data_smithi188_141483_object40834:head# 2024-04-23T16:43:24.817 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc5d7252:::benchmark_data_smithi188_141483_object20570:head# 2024-04-23T16:43:24.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc5e3f79:::benchmark_data_smithi188_141483_object84123:head# 2024-04-23T16:43:24.818 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc5ef56c:::benchmark_data_smithi188_141483_object43403:head# 2024-04-23T16:43:24.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc5f1cfb:::benchmark_data_smithi188_141483_object28663:head# 2024-04-23T16:43:24.819 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc5fea70:::benchmark_data_smithi188_141483_object25555:head# 2024-04-23T16:43:24.820 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc619780:::benchmark_data_smithi188_141483_object15460:head# 2024-04-23T16:43:24.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc627a01:::benchmark_data_smithi188_141483_object41891:head# 2024-04-23T16:43:24.823 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc628290:::benchmark_data_smithi188_141483_object35374:head# 2024-04-23T16:43:24.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc631e16:::benchmark_data_smithi188_141483_object23363:head# 2024-04-23T16:43:24.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc643251:::benchmark_data_smithi188_141483_object28090:head# 2024-04-23T16:43:24.824 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc64b94e:::benchmark_data_smithi188_141483_object58198:head# 2024-04-23T16:43:24.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc64f091:::benchmark_data_smithi188_141483_object79720:head# 2024-04-23T16:43:24.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc67b773:::benchmark_data_smithi188_141483_object52345:head# 2024-04-23T16:43:24.825 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc68bd34:::benchmark_data_smithi188_141483_object50556:head# 2024-04-23T16:43:24.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc68c104:::benchmark_data_smithi188_141483_object47094:head# 2024-04-23T16:43:24.826 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc693886:::benchmark_data_smithi188_141483_object49794:head# 2024-04-23T16:43:24.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc6e23c1:::benchmark_data_smithi188_141483_object84656:head# 2024-04-23T16:43:24.827 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc6f4d29:::benchmark_data_smithi188_141483_object10166:head# 2024-04-23T16:43:24.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc70c5b3:::benchmark_data_smithi188_141483_object44479:head# 2024-04-23T16:43:24.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc718ea7:::benchmark_data_smithi188_141483_object17609:head# 2024-04-23T16:43:24.828 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc71be99:::benchmark_data_smithi188_141483_object86109:head# 2024-04-23T16:43:24.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc71bebf:::benchmark_data_smithi188_141483_object48146:head# 2024-04-23T16:43:24.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc723aed:::benchmark_data_smithi188_141483_object39972:head# 2024-04-23T16:43:24.829 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc7289a8:::benchmark_data_smithi188_141483_object83402:head# 2024-04-23T16:43:24.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc72ad9a:::benchmark_data_smithi188_141483_object45753:head# 2024-04-23T16:43:24.830 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc72d5fc:::benchmark_data_smithi188_141483_object81136:head# 2024-04-23T16:43:24.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc74d1a5:::benchmark_data_smithi188_141483_object89524:head# 2024-04-23T16:43:24.831 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc7602c1:::benchmark_data_smithi188_141483_object46760:head# 2024-04-23T16:43:24.832 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc7658a1:::benchmark_data_smithi188_141483_object11246:head# 2024-04-23T16:43:24.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc785628:::benchmark_data_smithi188_141483_object85921:head# 2024-04-23T16:43:24.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc78caaa:::benchmark_data_smithi188_141483_object37738:head# 2024-04-23T16:43:24.833 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc7aad5e:::benchmark_data_smithi188_141483_object45367:head# 2024-04-23T16:43:24.834 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc7b95d0:::benchmark_data_smithi188_141483_object66819:head# 2024-04-23T16:43:24.834 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc7b994d:::benchmark_data_smithi188_141483_object75692:head# 2024-04-23T16:43:24.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc7c34b3:::benchmark_data_smithi188_141483_object88562:head# 2024-04-23T16:43:24.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc7cd0fd:::benchmark_data_smithi188_141483_object19218:head# 2024-04-23T16:43:24.835 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc7e8fa4:::benchmark_data_smithi188_141483_object45531:head# 2024-04-23T16:43:24.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc7f9ff2:::benchmark_data_smithi188_141483_object21748:head# 2024-04-23T16:43:24.836 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc808eb7:::benchmark_data_smithi188_141483_object80472:head# 2024-04-23T16:43:24.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc80bde1:::benchmark_data_smithi188_141483_object34956:head# 2024-04-23T16:43:24.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc811d70:::benchmark_data_smithi188_141483_object72824:head# 2024-04-23T16:43:24.837 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc84e712:::benchmark_data_smithi188_141483_object72174:head# 2024-04-23T16:43:24.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc84fc71:::benchmark_data_smithi188_141483_object74660:head# 2024-04-23T16:43:24.838 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc8590a6:::benchmark_data_smithi188_141483_object27865:head# 2024-04-23T16:43:24.839 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc861e12:::benchmark_data_smithi188_141483_object39723:head# 2024-04-23T16:43:24.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc863ee0:::benchmark_data_smithi188_141483_object7980:head# 2024-04-23T16:43:24.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc87156b:::benchmark_data_smithi188_141483_object11474:head# 2024-04-23T16:43:24.840 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc873c6a:::benchmark_data_smithi188_141483_object46173:head# 2024-04-23T16:43:24.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc88d6c5:::benchmark_data_smithi188_141483_object57146:head# 2024-04-23T16:43:24.841 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc89181a:::benchmark_data_smithi188_141483_object82362:head# 2024-04-23T16:43:24.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc89b1f0:::benchmark_data_smithi188_141483_object21944:head# 2024-04-23T16:43:24.842 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc8a1fd2:::benchmark_data_smithi188_141483_object71315:head# 2024-04-23T16:43:24.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc8aea60:::benchmark_data_smithi188_141483_object59430:head# 2024-04-23T16:43:24.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc8af06e:::benchmark_data_smithi188_141483_object28301:head# 2024-04-23T16:43:24.843 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc8d1290:::benchmark_data_smithi188_141483_object5654:head# 2024-04-23T16:43:24.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc8e18f0:::benchmark_data_smithi188_141483_object4594:head# 2024-04-23T16:43:24.844 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc8e3215:::benchmark_data_smithi188_141483_object17361:head# 2024-04-23T16:43:24.845 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc8f0357:::benchmark_data_smithi188_141483_object32717:head# 2024-04-23T16:43:24.845 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc90eb74:::benchmark_data_smithi188_141483_object61448:head# 2024-04-23T16:43:24.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc92259c:::benchmark_data_smithi188_141483_object5912:head# 2024-04-23T16:43:24.846 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc9232cc:::benchmark_data_smithi188_141483_object80327:head# 2024-04-23T16:43:24.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc939047:::benchmark_data_smithi188_141483_object10572:head# 2024-04-23T16:43:24.847 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc93ee07:::benchmark_data_smithi188_141483_object87365:head# 2024-04-23T16:43:24.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc94465f:::benchmark_data_smithi188_141483_object38716:head# 2024-04-23T16:43:24.848 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc951f60:::benchmark_data_smithi188_141483_object63850:head# 2024-04-23T16:43:24.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc96075c:::benchmark_data_smithi188_141483_object90839:head# 2024-04-23T16:43:24.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc9621c8:::benchmark_data_smithi188_141483_object54868:head# 2024-04-23T16:43:24.849 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc962430:::benchmark_data_smithi188_141483_object13005:head# 2024-04-23T16:43:24.850 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc9671df:::benchmark_data_smithi188_141483_object45434:head# 2024-04-23T16:43:24.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc977a6f:::benchmark_data_smithi188_141483_object46902:head# 2024-04-23T16:43:24.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc97f1f3:::benchmark_data_smithi188_141483_object36459:head# 2024-04-23T16:43:24.851 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc98dbe3:::benchmark_data_smithi188_141483_object51521:head# 2024-04-23T16:43:24.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc99fc74:::benchmark_data_smithi188_141483_object60480:head# 2024-04-23T16:43:24.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc9a0ab4:::benchmark_data_smithi188_141483_object51197:head# 2024-04-23T16:43:24.852 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc9a2b1c:::benchmark_data_smithi188_141483_object41098:head# 2024-04-23T16:43:24.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc9a5c09:::benchmark_data_smithi188_141483_object62703:head# 2024-04-23T16:43:24.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc9b8da5:::benchmark_data_smithi188_141483_object1018:head# 2024-04-23T16:43:24.853 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc9c71e4:::benchmark_data_smithi188_141483_object69989:head# 2024-04-23T16:43:24.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc9c8ca5:::benchmark_data_smithi188_141483_object53457:head# 2024-04-23T16:43:24.854 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc9d21bc:::benchmark_data_smithi188_141483_object22860:head# 2024-04-23T16:43:24.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc9d5a70:::benchmark_data_smithi188_141483_object10289:head# 2024-04-23T16:43:24.855 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc9f6a50:::benchmark_data_smithi188_141483_object22725:head# 2024-04-23T16:43:24.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cc9fe68e:::benchmark_data_smithi188_141483_object67386:head# 2024-04-23T16:43:24.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cca08b2c:::benchmark_data_smithi188_141483_object80120:head# 2024-04-23T16:43:24.856 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cca09af5:::benchmark_data_smithi188_141483_object66157:head# 2024-04-23T16:43:24.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cca0a9f6:::benchmark_data_smithi188_141483_object24024:head# 2024-04-23T16:43:24.857 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cca0de95:::benchmark_data_smithi188_141483_object9072:head# 2024-04-23T16:43:24.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cca1100c:::benchmark_data_smithi188_141483_object38104:head# 2024-04-23T16:43:24.858 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cca14c8b:::benchmark_data_smithi188_141483_object60184:head# 2024-04-23T16:43:24.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cca167d0:::benchmark_data_smithi188_141483_object30330:head# 2024-04-23T16:43:24.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cca1ed24:::benchmark_data_smithi188_141483_object31391:head# 2024-04-23T16:43:24.859 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cca2ee1f:::benchmark_data_smithi188_141483_object47783:head# 2024-04-23T16:43:24.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cca3a75d:::benchmark_data_smithi188_141483_object61967:head# 2024-04-23T16:43:24.860 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cca4a5a8:::benchmark_data_smithi188_141483_object49139:head# 2024-04-23T16:43:24.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cca521f8:::benchmark_data_smithi188_141483_object8148:head# 2024-04-23T16:43:24.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cca5abd4:::benchmark_data_smithi188_141483_object79459:head# 2024-04-23T16:43:24.861 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cca766d7:::benchmark_data_smithi188_141483_object51306:head# 2024-04-23T16:43:24.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cca7fbdf:::benchmark_data_smithi188_141483_object65380:head# 2024-04-23T16:43:24.862 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cca82aee:::benchmark_data_smithi188_141483_object27798:head# 2024-04-23T16:43:24.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cca9554f:::benchmark_data_smithi188_141483_object47263:head# 2024-04-23T16:43:24.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccac91f0:::benchmark_data_smithi188_141483_object1893:head# 2024-04-23T16:43:24.863 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccacc758:::benchmark_data_smithi188_141483_object21263:head# 2024-04-23T16:43:24.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccacc9e3:::benchmark_data_smithi188_141483_object35914:head# 2024-04-23T16:43:24.864 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccad3a7c:::benchmark_data_smithi188_141483_object86548:head# 2024-04-23T16:43:24.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccad521e:::benchmark_data_smithi188_141483_object19006:head# 2024-04-23T16:43:24.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccad7aa0:::benchmark_data_smithi188_141483_object19425:head# 2024-04-23T16:43:24.865 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccad9623:::benchmark_data_smithi188_141483_object43001:head# 2024-04-23T16:43:24.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccadcc31:::benchmark_data_smithi188_141483_object79023:head# 2024-04-23T16:43:24.866 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccb150c3:::benchmark_data_smithi188_141483_object60653:head# 2024-04-23T16:43:24.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccb1c9cd:::benchmark_data_smithi188_141483_object47549:head# 2024-04-23T16:43:24.867 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccb200fe:::benchmark_data_smithi188_141483_object41287:head# 2024-04-23T16:43:24.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccb22639:::benchmark_data_smithi188_141483_object14680:head# 2024-04-23T16:43:24.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccb357ca:::benchmark_data_smithi188_141483_object88867:head# 2024-04-23T16:43:24.868 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccb427dc:::benchmark_data_smithi188_141483_object5525:head# 2024-04-23T16:43:24.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccb4bd48:::benchmark_data_smithi188_141483_object41795:head# 2024-04-23T16:43:24.869 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccb5706b:::benchmark_data_smithi188_141483_object71480:head# 2024-04-23T16:43:24.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccb67e50:::benchmark_data_smithi188_141483_object9675:head# 2024-04-23T16:43:24.870 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccb7594f:::benchmark_data_smithi188_141483_object28167:head# 2024-04-23T16:43:24.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccb76f6b:::benchmark_data_smithi188_141483_object24803:head# 2024-04-23T16:43:24.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccb7b578:::benchmark_data_smithi188_141483_object86931:head# 2024-04-23T16:43:24.871 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccb84380:::benchmark_data_smithi188_141483_object89907:head# 2024-04-23T16:43:24.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccb86e00:::benchmark_data_smithi188_141483_object45541:head# 2024-04-23T16:43:24.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccb98dcb:::benchmark_data_smithi188_141483_object61946:head# 2024-04-23T16:43:24.872 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccb99a20:::benchmark_data_smithi188_141483_object40616:head# 2024-04-23T16:43:24.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccb99b05:::benchmark_data_smithi188_141483_object9817:head# 2024-04-23T16:43:24.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccbbd337:::benchmark_data_smithi188_141483_object2442:head# 2024-04-23T16:43:24.873 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccbbdaed:::benchmark_data_smithi188_141483_object54268:head# 2024-04-23T16:43:24.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccbc1c79:::benchmark_data_smithi188_141483_object38031:head# 2024-04-23T16:43:24.874 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccbc4e50:::benchmark_data_smithi188_141483_object67138:head# 2024-04-23T16:43:24.875 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccbc6d84:::benchmark_data_smithi188_141483_object54912:head# 2024-04-23T16:43:24.875 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccbc6d9d:::benchmark_data_smithi188_141483_object75634:head# 2024-04-23T16:43:24.876 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccbd3223:::benchmark_data_smithi188_141483_object20961:head# 2024-04-23T16:43:24.876 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccbedfff:::benchmark_data_smithi188_141483_object85541:head# 2024-04-23T16:43:24.876 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccc129b1:::benchmark_data_smithi188_141483_object6700:head# 2024-04-23T16:43:24.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccc1a3da:::benchmark_data_smithi188_141483_object88237:head# 2024-04-23T16:43:24.877 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccc29613:::benchmark_data_smithi188_141483_object81330:head# 2024-04-23T16:43:24.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccc389c3:::benchmark_data_smithi188_141483_object90750:head# 2024-04-23T16:43:24.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccc5f035:::benchmark_data_smithi188_141483_object14031:head# 2024-04-23T16:43:24.878 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccc6dce6:::benchmark_data_smithi188_141483_object69965:head# 2024-04-23T16:43:24.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccc8e559:::benchmark_data_smithi188_141483_object69569:head# 2024-04-23T16:43:24.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccc9c19c:::benchmark_data_smithi188_141483_object25642:head# 2024-04-23T16:43:24.879 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccc9edc4:::benchmark_data_smithi188_141483_object400:head# 2024-04-23T16:43:24.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cccacf13:::benchmark_data_smithi188_141483_object3181:head# 2024-04-23T16:43:24.880 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cccb2434:::benchmark_data_smithi188_141483_object72749:head# 2024-04-23T16:43:24.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cccb8a0d:::benchmark_data_smithi188_141483_object19159:head# 2024-04-23T16:43:24.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccccb203:::benchmark_data_smithi188_141483_object15958:head# 2024-04-23T16:43:24.881 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccccb94d:::benchmark_data_smithi188_141483_object43204:head# 2024-04-23T16:43:24.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cccd01f1:::benchmark_data_smithi188_141483_object42357:head# 2024-04-23T16:43:24.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cccd6936:::benchmark_data_smithi188_141483_object62227:head# 2024-04-23T16:43:24.882 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cccd7c76:::benchmark_data_smithi188_141483_object28623:head# 2024-04-23T16:43:24.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccce0f4a:::benchmark_data_smithi188_141483_object40241:head# 2024-04-23T16:43:24.883 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccce1d60:::benchmark_data_smithi188_141483_object60643:head# 2024-04-23T16:43:24.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccce5b00:::benchmark_data_smithi188_141483_object69939:head# 2024-04-23T16:43:24.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccce8f8f:::benchmark_data_smithi188_141483_object86771:head# 2024-04-23T16:43:24.884 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccd0aa20:::benchmark_data_smithi188_141483_object2635:head# 2024-04-23T16:43:24.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccd167d2:::benchmark_data_smithi188_141483_object45008:head# 2024-04-23T16:43:24.885 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccd20f83:::benchmark_data_smithi188_141483_object8402:head# 2024-04-23T16:43:24.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccd24501:::benchmark_data_smithi188_141483_object12633:head# 2024-04-23T16:43:24.886 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccd266c3:::benchmark_data_smithi188_141483_object87828:head# 2024-04-23T16:43:24.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccd2a87a:::benchmark_data_smithi188_141483_object47399:head# 2024-04-23T16:43:24.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccd3498c:::benchmark_data_smithi188_141483_object341:head# 2024-04-23T16:43:24.887 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccd3d5ea:::benchmark_data_smithi188_141483_object89355:head# 2024-04-23T16:43:24.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccd58b56:::benchmark_data_smithi188_141483_object89693:head# 2024-04-23T16:43:24.888 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccd69bc0:::benchmark_data_smithi188_141483_object39987:head# 2024-04-23T16:43:24.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccd6f369:::benchmark_data_smithi188_141483_object17602:head# 2024-04-23T16:43:24.889 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccd76633:::benchmark_data_smithi188_141483_object17268:head# 2024-04-23T16:43:24.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccd7a0c7:::benchmark_data_smithi188_141483_object47671:head# 2024-04-23T16:43:24.890 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccd8506c:::benchmark_data_smithi188_141483_object7689:head# 2024-04-23T16:43:24.891 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccda3729:::benchmark_data_smithi188_141483_object56949:head# 2024-04-23T16:43:24.891 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccda86e3:::benchmark_data_smithi188_141483_object78920:head# 2024-04-23T16:43:24.891 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccdaccd7:::benchmark_data_smithi188_141483_object75213:head# 2024-04-23T16:43:24.892 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccdbabff:::benchmark_data_smithi188_141483_object6163:head# 2024-04-23T16:43:24.893 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccdbe64e:::benchmark_data_smithi188_141483_object47191:head# 2024-04-23T16:43:24.893 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccddc03b:::benchmark_data_smithi188_141483_object60103:head# 2024-04-23T16:43:24.893 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccde0318:::benchmark_data_smithi188_141483_object73154:head# 2024-04-23T16:43:24.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccdeb878:::benchmark_data_smithi188_141483_object1246:head# 2024-04-23T16:43:24.894 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccdffd7b:::benchmark_data_smithi188_141483_object64594:head# 2024-04-23T16:43:24.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cce003b0:::benchmark_data_smithi188_141483_object43033:head# 2024-04-23T16:43:24.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cce0b005:::benchmark_data_smithi188_141483_object60041:head# 2024-04-23T16:43:24.895 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cce10bc8:::benchmark_data_smithi188_141483_object45442:head# 2024-04-23T16:43:24.896 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cce37508:::benchmark_data_smithi188_141483_object9949:head# 2024-04-23T16:43:24.896 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cce49691:::benchmark_data_smithi188_141483_object33276:head# 2024-04-23T16:43:24.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cce4be7f:::benchmark_data_smithi188_141483_object63500:head# 2024-04-23T16:43:24.897 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cce627ba:::benchmark_data_smithi188_141483_object41469:head# 2024-04-23T16:43:24.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cce82105:::benchmark_data_smithi188_141483_object15519:head# 2024-04-23T16:43:24.898 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cce93280:::benchmark_data_smithi188_141483_object7090:head# 2024-04-23T16:43:24.899 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cce9b3b0:::benchmark_data_smithi188_141483_object33076:head# 2024-04-23T16:43:24.899 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccea0193:::benchmark_data_smithi188_141483_object80968:head# 2024-04-23T16:43:24.899 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cceb77b1:::benchmark_data_smithi188_141483_object55970:head# 2024-04-23T16:43:24.900 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cceb9101:::benchmark_data_smithi188_141483_object30616:head# 2024-04-23T16:43:24.900 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccee7ef6:::benchmark_data_smithi188_141483_object54881:head# 2024-04-23T16:43:24.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccef665d:::benchmark_data_smithi188_141483_object65442:head# 2024-04-23T16:43:24.901 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccefa7d3:::benchmark_data_smithi188_141483_object69481:head# 2024-04-23T16:43:24.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccf02d51:::benchmark_data_smithi188_141483_object56098:head# 2024-04-23T16:43:24.902 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccf0bf68:::benchmark_data_smithi188_141483_object66896:head# 2024-04-23T16:43:24.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccf11cf3:::benchmark_data_smithi188_141483_object78355:head# 2024-04-23T16:43:24.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccf16214:::benchmark_data_smithi188_141483_object81093:head# 2024-04-23T16:43:24.903 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccf1ae0a:::benchmark_data_smithi188_141483_object32522:head# 2024-04-23T16:43:24.904 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccf24132:::benchmark_data_smithi188_141483_object76349:head# 2024-04-23T16:43:24.904 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccf25d26:::benchmark_data_smithi188_141483_object69895:head# 2024-04-23T16:43:24.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccf3000e:::benchmark_data_smithi188_141483_object23426:head# 2024-04-23T16:43:24.905 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccf3d657:::benchmark_data_smithi188_141483_object49114:head# 2024-04-23T16:43:24.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccf43bc6:::benchmark_data_smithi188_141483_object21305:head# 2024-04-23T16:43:24.906 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccf4ee38:::benchmark_data_smithi188_141483_object29461:head# 2024-04-23T16:43:24.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccf5cce4:::benchmark_data_smithi188_141483_object32291:head# 2024-04-23T16:43:24.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccf5e90d:::benchmark_data_smithi188_141483_object8851:head# 2024-04-23T16:43:24.907 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccf61f4f:::benchmark_data_smithi188_141483_object40355:head# 2024-04-23T16:43:24.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccf67e4f:::benchmark_data_smithi188_141483_object28867:head# 2024-04-23T16:43:24.908 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccf6d6bb:::benchmark_data_smithi188_141483_object13784:head# 2024-04-23T16:43:24.909 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccf99b78:::benchmark_data_smithi188_141483_object70947:head# 2024-04-23T16:43:24.909 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccfa187f:::benchmark_data_smithi188_141483_object25143:head# 2024-04-23T16:43:24.910 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccfb5a1e:::benchmark_data_smithi188_141483_object24373:head# 2024-04-23T16:43:24.910 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccfb9948:::benchmark_data_smithi188_141483_object9150:head# 2024-04-23T16:43:24.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccfbc4fc:::benchmark_data_smithi188_141483_object89555:head# 2024-04-23T16:43:24.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccfd158f:::benchmark_data_smithi188_141483_object60600:head# 2024-04-23T16:43:24.911 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccfe0b0d:::benchmark_data_smithi188_141483_object26711:head# 2024-04-23T16:43:24.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccfe2c7c:::benchmark_data_smithi188_141483_object75156:head# 2024-04-23T16:43:24.912 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccfe2cf3:::benchmark_data_smithi188_141483_object63884:head# 2024-04-23T16:43:24.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccfe73f8:::benchmark_data_smithi188_141483_object90207:head# 2024-04-23T16:43:24.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ccff302a:::benchmark_data_smithi188_141483_object69139:head# 2024-04-23T16:43:24.913 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd006fac:::benchmark_data_smithi188_141483_object55554:head# 2024-04-23T16:43:24.914 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd00ee04:::benchmark_data_smithi188_141483_object12966:head# 2024-04-23T16:43:24.914 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd013edb:::benchmark_data_smithi188_141483_object34221:head# 2024-04-23T16:43:24.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd01a6f8:::benchmark_data_smithi188_141483_object44529:head# 2024-04-23T16:43:24.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd01de4b:::benchmark_data_smithi188_141483_object2294:head# 2024-04-23T16:43:24.915 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd0210c8:::benchmark_data_smithi188_141483_object74555:head# 2024-04-23T16:43:24.916 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd023424:::benchmark_data_smithi188_141483_object90727:head# 2024-04-23T16:43:24.917 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd04f78e:::benchmark_data_smithi188_141483_object55530:head# 2024-04-23T16:43:24.917 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd058115:::benchmark_data_smithi188_141483_object56858:head# 2024-04-23T16:43:24.917 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd06ad67:::benchmark_data_smithi188_141483_object77898:head# 2024-04-23T16:43:24.917 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd081090:::benchmark_data_smithi188_141483_object79432:head# 2024-04-23T16:43:24.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd088003:::benchmark_data_smithi188_141483_object70480:head# 2024-04-23T16:43:24.918 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd097237:::benchmark_data_smithi188_141483_object36016:head# 2024-04-23T16:43:24.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd09fa63:::benchmark_data_smithi188_141483_object83574:head# 2024-04-23T16:43:24.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd0a2d7c:::benchmark_data_smithi188_141483_object72674:head# 2024-04-23T16:43:24.919 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd0a4652:::benchmark_data_smithi188_141483_object80328:head# 2024-04-23T16:43:24.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd0ad9f0:::benchmark_data_smithi188_141483_object57260:head# 2024-04-23T16:43:24.920 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd0b2273:::benchmark_data_smithi188_141483_object2212:head# 2024-04-23T16:43:24.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd0b838d:::benchmark_data_smithi188_141483_object76078:head# 2024-04-23T16:43:24.921 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd0bd515:::benchmark_data_smithi188_141483_object66903:head# 2024-04-23T16:43:24.922 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd0cbf95:::benchmark_data_smithi188_141483_object65336:head# 2024-04-23T16:43:24.925 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd0ced33:::benchmark_data_smithi188_141483_object5036:head# 2024-04-23T16:43:24.925 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd0e5329:::benchmark_data_smithi188_141483_object6077:head# 2024-04-23T16:43:24.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd0ec880:::benchmark_data_smithi188_141483_object34100:head# 2024-04-23T16:43:24.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd10245c:::benchmark_data_smithi188_141483_object2571:head# 2024-04-23T16:43:24.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd10465c:::benchmark_data_smithi188_141483_object77820:head# 2024-04-23T16:43:24.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd110d63:::benchmark_data_smithi188_141483_object8725:head# 2024-04-23T16:43:24.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd116b05:::benchmark_data_smithi188_141483_object22603:head# 2024-04-23T16:43:24.926 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd143143:::benchmark_data_smithi188_141483_object59498:head# 2024-04-23T16:43:24.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd14e73f:::benchmark_data_smithi188_141483_object13078:head# 2024-04-23T16:43:24.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd14f740:::benchmark_data_smithi188_141483_object47731:head# 2024-04-23T16:43:24.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd15a533:::benchmark_data_smithi188_141483_object5560:head# 2024-04-23T16:43:24.927 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd17a860:::benchmark_data_smithi188_141483_object33130:head# 2024-04-23T16:43:24.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd18675e:::benchmark_data_smithi188_141483_object57870:head# 2024-04-23T16:43:24.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd19be3d:::benchmark_data_smithi188_141483_object41954:head# 2024-04-23T16:43:24.928 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd1a7e31:::benchmark_data_smithi188_141483_object61898:head# 2024-04-23T16:43:24.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd1af2d8:::benchmark_data_smithi188_141483_object84230:head# 2024-04-23T16:43:24.929 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd1b0701:::benchmark_data_smithi188_141483_object33709:head# 2024-04-23T16:43:24.930 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd1bd572:::benchmark_data_smithi188_141483_object33839:head# 2024-04-23T16:43:24.930 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd1c2c07:::benchmark_data_smithi188_141483_object29483:head# 2024-04-23T16:43:24.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd1c7609:::benchmark_data_smithi188_141483_object87305:head# 2024-04-23T16:43:24.931 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd1d7f9f:::benchmark_data_smithi188_141483_object44787:head# 2024-04-23T16:43:24.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd1d959f:::benchmark_data_smithi188_141483_object28060:head# 2024-04-23T16:43:24.932 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd1df672:::benchmark_data_smithi188_141483_object59151:head# 2024-04-23T16:43:24.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd1e68f4:::benchmark_data_smithi188_141483_object20664:head# 2024-04-23T16:43:24.933 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd209d71:::benchmark_data_smithi188_141483_object40944:head# 2024-04-23T16:43:24.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd23206d:::benchmark_data_smithi188_141483_object11658:head# 2024-04-23T16:43:24.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd25791c:::benchmark_data_smithi188_141483_object32684:head# 2024-04-23T16:43:24.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd26d503:::benchmark_data_smithi188_141483_object41146:head# 2024-04-23T16:43:24.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd26ef71:::benchmark_data_smithi188_141483_object56892:head# 2024-04-23T16:43:24.934 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd2712b9:::benchmark_data_smithi188_141483_object90130:head# 2024-04-23T16:43:24.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd27fff4:::benchmark_data_smithi188_141483_object40981:head# 2024-04-23T16:43:24.935 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd2a75da:::benchmark_data_smithi188_141483_object38338:head# 2024-04-23T16:43:24.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd2b3383:::benchmark_data_smithi188_141483_object53423:head# 2024-04-23T16:43:24.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd2cc849:::benchmark_data_smithi188_141483_object11166:head# 2024-04-23T16:43:24.936 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd2eeae5:::benchmark_data_smithi188_141483_object38424:head# 2024-04-23T16:43:24.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd2f8e20:::benchmark_data_smithi188_141483_object38802:head# 2024-04-23T16:43:24.937 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd317c14:::benchmark_data_smithi188_141483_object25803:head# 2024-04-23T16:43:24.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd32df3f:::benchmark_data_smithi188_141483_object77501:head# 2024-04-23T16:43:24.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd33becf:::benchmark_data_smithi188_141483_object17022:head# 2024-04-23T16:43:24.938 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd34f7cc:::benchmark_data_smithi188_141483_object45985:head# 2024-04-23T16:43:24.939 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd354924:::benchmark_data_smithi188_141483_object53481:head# 2024-04-23T16:43:24.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd36c998:::benchmark_data_smithi188_141483_object78949:head# 2024-04-23T16:43:24.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd36feb8:::benchmark_data_smithi188_141483_object77099:head# 2024-04-23T16:43:24.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd37c168:::benchmark_data_smithi188_141483_object24572:head# 2024-04-23T16:43:24.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd37e315:::benchmark_data_smithi188_141483_object332:head# 2024-04-23T16:43:24.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd37f8c4:::benchmark_data_smithi188_141483_object30788:head# 2024-04-23T16:43:24.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd387c88:::benchmark_data_smithi188_141483_object66651:head# 2024-04-23T16:43:24.942 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd3c1874:::benchmark_data_smithi188_141483_object63278:head# 2024-04-23T16:43:24.943 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd3c8295:::benchmark_data_smithi188_141483_object76770:head# 2024-04-23T16:43:24.943 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd3ca140:::benchmark_data_smithi188_141483_object74030:head# 2024-04-23T16:43:24.943 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd3dca25:::benchmark_data_smithi188_141483_object70904:head# 2024-04-23T16:43:24.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd3dcb87:::benchmark_data_smithi188_141483_object66857:head# 2024-04-23T16:43:24.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd3fbbbd:::benchmark_data_smithi188_141483_object63666:head# 2024-04-23T16:43:24.944 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd400521:::benchmark_data_smithi188_141483_object22511:head# 2024-04-23T16:43:24.945 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd400b1b:::benchmark_data_smithi188_141483_object53495:head# 2024-04-23T16:43:24.945 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd40d5f4:::benchmark_data_smithi188_141483_object74330:head# 2024-04-23T16:43:24.946 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd438d66:::benchmark_data_smithi188_141483_object58930:head# 2024-04-23T16:43:24.946 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd44d019:::benchmark_data_smithi188_141483_object78518:head# 2024-04-23T16:43:24.946 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd457e5d:::benchmark_data_smithi188_141483_object32949:head# 2024-04-23T16:43:24.947 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd45fe6e:::benchmark_data_smithi188_141483_object19496:head# 2024-04-23T16:43:24.947 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd4643e3:::benchmark_data_smithi188_141483_object61868:head# 2024-04-23T16:43:24.947 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd4674d2:::benchmark_data_smithi188_141483_object88253:head# 2024-04-23T16:43:24.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd469790:::benchmark_data_smithi188_141483_object79135:head# 2024-04-23T16:43:24.948 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd480b7f:::benchmark_data_smithi188_141483_object49997:head# 2024-04-23T16:43:24.949 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd481452:::benchmark_data_smithi188_141483_object57602:head# 2024-04-23T16:43:24.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd482a07:::benchmark_data_smithi188_141483_object23924:head# 2024-04-23T16:43:24.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd48e5ff:::benchmark_data_smithi188_141483_object598:head# 2024-04-23T16:43:24.951 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd4926a2:::benchmark_data_smithi188_141483_object90368:head# 2024-04-23T16:43:24.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd49727a:::benchmark_data_smithi188_141483_object12092:head# 2024-04-23T16:43:24.952 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd4acd95:::benchmark_data_smithi188_141483_object32629:head# 2024-04-23T16:43:24.953 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd4cb7ff:::benchmark_data_smithi188_141483_object76344:head# 2024-04-23T16:43:24.953 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd4ce68f:::benchmark_data_smithi188_141483_object69013:head# 2024-04-23T16:43:24.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd4cef32:::benchmark_data_smithi188_141483_object64528:head# 2024-04-23T16:43:24.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd4dff7d:::benchmark_data_smithi188_141483_object68124:head# 2024-04-23T16:43:24.954 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd4eef1c:::benchmark_data_smithi188_141483_object49073:head# 2024-04-23T16:43:24.955 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd4f3023:::benchmark_data_smithi188_141483_object89367:head# 2024-04-23T16:43:24.955 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd4fa643:::benchmark_data_smithi188_141483_object6621:head# 2024-04-23T16:43:24.956 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd4fb201:::benchmark_data_smithi188_141483_object46121:head# 2024-04-23T16:43:24.956 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd51569e:::benchmark_data_smithi188_141483_object55288:head# 2024-04-23T16:43:24.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd51d62d:::benchmark_data_smithi188_141483_object57359:head# 2024-04-23T16:43:24.957 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd52b0a9:::benchmark_data_smithi188_141483_object26726:head# 2024-04-23T16:43:24.959 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd52bba8:::benchmark_data_smithi188_141483_object47951:head# 2024-04-23T16:43:24.959 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd52e7eb:::benchmark_data_smithi188_141483_object15449:head# 2024-04-23T16:43:24.960 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd52e973:::benchmark_data_smithi188_141483_object27758:head# 2024-04-23T16:43:24.960 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd5309ad:::benchmark_data_smithi188_141483_object66121:head# 2024-04-23T16:43:24.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd55ff62:::benchmark_data_smithi188_141483_object88916:head# 2024-04-23T16:43:24.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd560772:::benchmark_data_smithi188_141483_object11466:head# 2024-04-23T16:43:24.961 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd56ad9b:::benchmark_data_smithi188_141483_object21605:head# 2024-04-23T16:43:24.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd56b655:::benchmark_data_smithi188_141483_object16263:head# 2024-04-23T16:43:24.962 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd56dfd7:::benchmark_data_smithi188_141483_object21203:head# 2024-04-23T16:43:24.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd571180:::benchmark_data_smithi188_141483_object76913:head# 2024-04-23T16:43:24.963 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd57c2b6:::benchmark_data_smithi188_141483_object88674:head# 2024-04-23T16:43:24.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd58d998:::benchmark_data_smithi188_141483_object80456:head# 2024-04-23T16:43:24.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd5915aa:::benchmark_data_smithi188_141483_object40343:head# 2024-04-23T16:43:24.964 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd598e6b:::benchmark_data_smithi188_141483_object50645:head# 2024-04-23T16:43:24.965 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd5a8607:::benchmark_data_smithi188_141483_object35675:head# 2024-04-23T16:43:24.965 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd5c23b4:::benchmark_data_smithi188_141483_object77331:head# 2024-04-23T16:43:24.966 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd5c713c:::benchmark_data_smithi188_141483_object63356:head# 2024-04-23T16:43:24.966 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd5dc1e7:::benchmark_data_smithi188_141483_object416:head# 2024-04-23T16:43:24.966 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd5ec1ce:::benchmark_data_smithi188_141483_object1939:head# 2024-04-23T16:43:24.967 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd623dc3:::benchmark_data_smithi188_141483_object44212:head# 2024-04-23T16:43:24.967 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd6287f0:::benchmark_data_smithi188_141483_object85407:head# 2024-04-23T16:43:24.968 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd63f048:::benchmark_data_smithi188_141483_object42074:head# 2024-04-23T16:43:24.969 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd64643b:::benchmark_data_smithi188_141483_object30786:head# 2024-04-23T16:43:24.969 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd65098f:::benchmark_data_smithi188_141483_object8893:head# 2024-04-23T16:43:24.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd6518d9:::benchmark_data_smithi188_141483_object61223:head# 2024-04-23T16:43:24.970 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd65ec1c:::benchmark_data_smithi188_141483_object76783:head# 2024-04-23T16:43:24.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd664433:::benchmark_data_smithi188_141483_object44227:head# 2024-04-23T16:43:24.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd673493:::benchmark_data_smithi188_141483_object72480:head# 2024-04-23T16:43:24.971 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd6812a9:::benchmark_data_smithi188_141483_object3782:head# 2024-04-23T16:43:24.972 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd687c78:::benchmark_data_smithi188_141483_object24656:head# 2024-04-23T16:43:24.973 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd69006a:::benchmark_data_smithi188_141483_object61779:head# 2024-04-23T16:43:24.974 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd693968:::benchmark_data_smithi188_141483_object4377:head# 2024-04-23T16:43:24.974 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd6b7de5:::benchmark_data_smithi188_141483_object41607:head# 2024-04-23T16:43:24.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd6cb1eb:::benchmark_data_smithi188_141483_object46152:head# 2024-04-23T16:43:24.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd6d28c6:::benchmark_data_smithi188_141483_object70308:head# 2024-04-23T16:43:24.975 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd6d55a8:::benchmark_data_smithi188_141483_object28198:head# 2024-04-23T16:43:24.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd6f4d56:::benchmark_data_smithi188_141483_object67558:head# 2024-04-23T16:43:24.976 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd6feb8b:::benchmark_data_smithi188_141483_object52770:head# 2024-04-23T16:43:24.977 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd715e81:::benchmark_data_smithi188_141483_object1282:head# 2024-04-23T16:43:24.977 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd71997a:::benchmark_data_smithi188_141483_object39177:head# 2024-04-23T16:43:24.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd71d353:::benchmark_data_smithi188_141483_object72667:head# 2024-04-23T16:43:24.978 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd722968:::benchmark_data_smithi188_141483_object46468:head# 2024-04-23T16:43:24.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd734f18:::benchmark_data_smithi188_141483_object36738:head# 2024-04-23T16:43:24.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd737abd:::benchmark_data_smithi188_141483_object12748:head# 2024-04-23T16:43:24.979 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd739b1a:::benchmark_data_smithi188_141483_object4923:head# 2024-04-23T16:43:24.980 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd745d5a:::benchmark_data_smithi188_141483_object76752:head# 2024-04-23T16:43:24.980 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd76272e:::benchmark_data_smithi188_141483_object32714:head# 2024-04-23T16:43:24.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd7631f8:::benchmark_data_smithi188_141483_object26655:head# 2024-04-23T16:43:24.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd764805:::benchmark_data_smithi188_141483_object26441:head# 2024-04-23T16:43:24.981 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd76604d:::benchmark_data_smithi188_141483_object22489:head# 2024-04-23T16:43:24.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd7668e4:::benchmark_data_smithi188_141483_object57809:head# 2024-04-23T16:43:24.982 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd76bb05:::benchmark_data_smithi188_141483_object48861:head# 2024-04-23T16:43:24.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd76d6da:::benchmark_data_smithi188_141483_object62612:head# 2024-04-23T16:43:24.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd790437:::benchmark_data_smithi188_141483_object90761:head# 2024-04-23T16:43:24.983 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd7b7aaa:::benchmark_data_smithi188_141483_object34128:head# 2024-04-23T16:43:24.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd7c5683:::benchmark_data_smithi188_141483_object67840:head# 2024-04-23T16:43:24.984 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd7d382a:::benchmark_data_smithi188_141483_object48278:head# 2024-04-23T16:43:24.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd7e457d:::benchmark_data_smithi188_141483_object61708:head# 2024-04-23T16:43:24.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd7ee199:::benchmark_data_smithi188_141483_object23762:head# 2024-04-23T16:43:24.985 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd7fe6f1:::benchmark_data_smithi188_141483_object5417:head# 2024-04-23T16:43:24.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd8024ac:::benchmark_data_smithi188_141483_object87806:head# 2024-04-23T16:43:24.986 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd807f68:::benchmark_data_smithi188_141483_object6563:head# 2024-04-23T16:43:24.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd80af08:::benchmark_data_smithi188_141483_object23508:head# 2024-04-23T16:43:24.987 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd80dde8:::benchmark_data_smithi188_141483_object88059:head# 2024-04-23T16:43:24.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd81000a:::benchmark_data_smithi188_141483_object80669:head# 2024-04-23T16:43:24.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd81a217:::benchmark_data_smithi188_141483_object21626:head# 2024-04-23T16:43:24.988 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd821f2d:::benchmark_data_smithi188_141483_object441:head# 2024-04-23T16:43:24.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd83b024:::benchmark_data_smithi188_141483_object51463:head# 2024-04-23T16:43:24.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd843edb:::benchmark_data_smithi188_141483_object70900:head# 2024-04-23T16:43:24.989 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd854dee:::benchmark_data_smithi188_141483_object22621:head# 2024-04-23T16:43:24.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd859f64:::benchmark_data_smithi188_141483_object13764:head# 2024-04-23T16:43:24.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd85ef31:::benchmark_data_smithi188_141483_object3673:head# 2024-04-23T16:43:24.990 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd86f806:::benchmark_data_smithi188_141483_object30362:head# 2024-04-23T16:43:24.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd870d6c:::benchmark_data_smithi188_141483_object22198:head# 2024-04-23T16:43:24.991 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd87e509:::benchmark_data_smithi188_141483_object20144:head# 2024-04-23T16:43:24.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd887629:::benchmark_data_smithi188_141483_object16825:head# 2024-04-23T16:43:24.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd889b20:::benchmark_data_smithi188_141483_object17442:head# 2024-04-23T16:43:24.992 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd88b54a:::benchmark_data_smithi188_141483_object47039:head# 2024-04-23T16:43:24.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd896c06:::benchmark_data_smithi188_141483_object11852:head# 2024-04-23T16:43:24.993 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd8a5ea7:::benchmark_data_smithi188_141483_object85185:head# 2024-04-23T16:43:24.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd8b120d:::benchmark_data_smithi188_141483_object6120:head# 2024-04-23T16:43:24.994 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd8b6bdc:::benchmark_data_smithi188_141483_object28463:head# 2024-04-23T16:43:24.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd8d143e:::benchmark_data_smithi188_141483_object32662:head# 2024-04-23T16:43:24.995 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd8d7fd1:::benchmark_data_smithi188_141483_object89234:head# 2024-04-23T16:43:24.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd8e0824:::benchmark_data_smithi188_141483_object47356:head# 2024-04-23T16:43:24.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd8e8d4a:::benchmark_data_smithi188_141483_object48666:head# 2024-04-23T16:43:24.996 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd9034d8:::benchmark_data_smithi188_141483_object66181:head# 2024-04-23T16:43:24.997 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd90d53d:::benchmark_data_smithi188_141483_object64748:head# 2024-04-23T16:43:25.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd910513:::benchmark_data_smithi188_141483_object652:head# 2024-04-23T16:43:25.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd910547:::benchmark_data_smithi188_141483_object59610:head# 2024-04-23T16:43:25.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd914457:::benchmark_data_smithi188_141483_object73368:head# 2024-04-23T16:43:25.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd931595:::benchmark_data_smithi188_141483_object14340:head# 2024-04-23T16:43:25.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd9333d6:::benchmark_data_smithi188_141483_object33538:head# 2024-04-23T16:43:25.000 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd9370cb:::benchmark_data_smithi188_141483_object25547:head# 2024-04-23T16:43:25.002 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd93e4fd:::benchmark_data_smithi188_141483_object26717:head# 2024-04-23T16:43:25.002 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd940c56:::benchmark_data_smithi188_141483_object90431:head# 2024-04-23T16:43:25.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd9440ef:::benchmark_data_smithi188_141483_object36425:head# 2024-04-23T16:43:25.005 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd94fc31:::benchmark_data_smithi188_141483_object41940:head# 2024-04-23T16:43:25.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd95117d:::benchmark_data_smithi188_141483_object34641:head# 2024-04-23T16:43:25.006 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd95ad06:::benchmark_data_smithi188_141483_object40183:head# 2024-04-23T16:43:25.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd963d33:::benchmark_data_smithi188_141483_object63420:head# 2024-04-23T16:43:25.009 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd96f06b:::benchmark_data_smithi188_141483_object7055:head# 2024-04-23T16:43:25.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd972142:::benchmark_data_smithi188_141483_object52920:head# 2024-04-23T16:43:25.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd974879:::benchmark_data_smithi188_141483_object39729:head# 2024-04-23T16:43:25.010 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd975bce:::benchmark_data_smithi188_141483_object81016:head# 2024-04-23T16:43:25.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd97f156:::benchmark_data_smithi188_141483_object87139:head# 2024-04-23T16:43:25.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd980ce7:::benchmark_data_smithi188_141483_object116:head# 2024-04-23T16:43:25.011 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd98d207:::benchmark_data_smithi188_141483_object23645:head# 2024-04-23T16:43:25.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd98d693:::benchmark_data_smithi188_141483_object1135:head# 2024-04-23T16:43:25.012 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd99331c:::benchmark_data_smithi188_141483_object6488:head# 2024-04-23T16:43:25.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd9a5f1b:::benchmark_data_smithi188_141483_object39024:head# 2024-04-23T16:43:25.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd9a7b9a:::benchmark_data_smithi188_141483_object17941:head# 2024-04-23T16:43:25.013 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd9b91f7:::benchmark_data_smithi188_141483_object69228:head# 2024-04-23T16:43:25.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd9c22e7:::benchmark_data_smithi188_141483_object17775:head# 2024-04-23T16:43:25.014 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd9c5632:::benchmark_data_smithi188_141483_object68311:head# 2024-04-23T16:43:25.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd9d3941:::benchmark_data_smithi188_141483_object10216:head# 2024-04-23T16:43:25.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd9d7847:::benchmark_data_smithi188_141483_object22217:head# 2024-04-23T16:43:25.015 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd9db443:::benchmark_data_smithi188_141483_object67784:head# 2024-04-23T16:43:25.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cd9f5f86:::benchmark_data_smithi188_141483_object46824:head# 2024-04-23T16:43:25.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cda1a3ac:::benchmark_data_smithi188_141483_object32406:head# 2024-04-23T16:43:25.016 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cda1b8af:::benchmark_data_smithi188_141483_object75789:head# 2024-04-23T16:43:25.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cda1e1ef:::benchmark_data_smithi188_141483_object85592:head# 2024-04-23T16:43:25.017 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cda20cb6:::benchmark_data_smithi188_141483_object64217:head# 2024-04-23T16:43:25.018 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cda2132b:::benchmark_data_smithi188_141483_object18732:head# 2024-04-23T16:43:25.018 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cda23652:::benchmark_data_smithi188_141483_object66297:head# 2024-04-23T16:43:25.018 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cda326d6:::benchmark_data_smithi188_141483_object52304:head# 2024-04-23T16:43:25.019 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cda3f2d8:::benchmark_data_smithi188_141483_object9339:head# 2024-04-23T16:43:25.019 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cda62fac:::benchmark_data_smithi188_141483_object37430:head# 2024-04-23T16:43:25.021 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cda6532d:::benchmark_data_smithi188_141483_object76812:head# 2024-04-23T16:43:25.023 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cda66e19:::benchmark_data_smithi188_141483_object49923:head# 2024-04-23T16:43:25.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cda698e8:::benchmark_data_smithi188_141483_object85494:head# 2024-04-23T16:43:25.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cda6e665:::benchmark_data_smithi188_141483_object66006:head# 2024-04-23T16:43:25.024 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cda8efb8:::benchmark_data_smithi188_141483_object58593:head# 2024-04-23T16:43:25.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cda91703:::benchmark_data_smithi188_141483_object66144:head# 2024-04-23T16:43:25.025 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cda9a635:::benchmark_data_smithi188_141483_object22309:head# 2024-04-23T16:43:25.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdaa22e2:::benchmark_data_smithi188_141483_object24376:head# 2024-04-23T16:43:25.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdaa4b1a:::benchmark_data_smithi188_141483_object35740:head# 2024-04-23T16:43:25.026 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdac519b:::benchmark_data_smithi188_141483_object11534:head# 2024-04-23T16:43:25.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdad2eea:::benchmark_data_smithi188_141483_object81761:head# 2024-04-23T16:43:25.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdaf7157:::benchmark_data_smithi188_141483_object25864:head# 2024-04-23T16:43:25.027 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb012b1:::benchmark_data_smithi188_141483_object37827:head# 2024-04-23T16:43:25.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb04746:::benchmark_data_smithi188_141483_object59027:head# 2024-04-23T16:43:25.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb0d088:::benchmark_data_smithi188_141483_object3351:head# 2024-04-23T16:43:25.028 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb1b9f3:::benchmark_data_smithi188_141483_object24270:head# 2024-04-23T16:43:25.029 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb1e8ea:::benchmark_data_smithi188_141483_object67163:head# 2024-04-23T16:43:25.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb2237d:::benchmark_data_smithi188_141483_object61213:head# 2024-04-23T16:43:25.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb30857:::benchmark_data_smithi188_141483_object66597:head# 2024-04-23T16:43:25.030 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb33906:::benchmark_data_smithi188_141483_object28328:head# 2024-04-23T16:43:25.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb36579:::benchmark_data_smithi188_141483_object18380:head# 2024-04-23T16:43:25.031 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb39c59:::benchmark_data_smithi188_141483_object28396:head# 2024-04-23T16:43:25.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb3a986:::benchmark_data_smithi188_141483_object84214:head# 2024-04-23T16:43:25.032 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb3ca6d:::benchmark_data_smithi188_141483_object36228:head# 2024-04-23T16:43:25.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb3d49a:::benchmark_data_smithi188_141483_object30623:head# 2024-04-23T16:43:25.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb41c23:::benchmark_data_smithi188_141483_object37740:head# 2024-04-23T16:43:25.033 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb46fbb:::benchmark_data_smithi188_141483_object24800:head# 2024-04-23T16:43:25.034 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb4c04e:::benchmark_data_smithi188_141483_object50371:head# 2024-04-23T16:43:25.034 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb4ff57:::benchmark_data_smithi188_141483_object53839:head# 2024-04-23T16:43:25.034 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb5029b:::benchmark_data_smithi188_141483_object90466:head# 2024-04-23T16:43:25.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb5a8bc:::benchmark_data_smithi188_141483_object24834:head# 2024-04-23T16:43:25.035 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb5b0d0:::benchmark_data_smithi188_141483_object51509:head# 2024-04-23T16:43:25.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb6b515:::benchmark_data_smithi188_141483_object40362:head# 2024-04-23T16:43:25.036 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb6be99:::benchmark_data_smithi188_141483_object87340:head# 2024-04-23T16:43:25.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb72262:::benchmark_data_smithi188_141483_object47735:head# 2024-04-23T16:43:25.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb77c00:::benchmark_data_smithi188_141483_object50681:head# 2024-04-23T16:43:25.037 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdb998f7:::benchmark_data_smithi188_141483_object12659:head# 2024-04-23T16:43:25.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdba7158:::benchmark_data_smithi188_141483_object30946:head# 2024-04-23T16:43:25.038 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdbb0291:::benchmark_data_smithi188_141483_object20530:head# 2024-04-23T16:43:25.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdbb10ab:::benchmark_data_smithi188_141483_object33651:head# 2024-04-23T16:43:25.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdbb8455:::benchmark_data_smithi188_141483_object8172:head# 2024-04-23T16:43:25.039 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdbbc668:::benchmark_data_smithi188_141483_object73733:head# 2024-04-23T16:43:25.040 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdbd4484:::benchmark_data_smithi188_141483_object10644:head# 2024-04-23T16:43:25.040 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdc092f7:::benchmark_data_smithi188_141483_object72579:head# 2024-04-23T16:43:25.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdc09b2b:::benchmark_data_smithi188_141483_object85131:head# 2024-04-23T16:43:25.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdc12583:::benchmark_data_smithi188_141483_object81363:head# 2024-04-23T16:43:25.041 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdc173a6:::benchmark_data_smithi188_141483_object81390:head# 2024-04-23T16:43:25.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdc22deb:::benchmark_data_smithi188_141483_object55529:head# 2024-04-23T16:43:25.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdc304de:::benchmark_data_smithi188_141483_object17846:head# 2024-04-23T16:43:25.042 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdc3cc71:::benchmark_data_smithi188_141483_object15945:head# 2024-04-23T16:43:25.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdc4a6f3:::benchmark_data_smithi188_141483_object37449:head# 2024-04-23T16:43:25.043 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdc656a4:::benchmark_data_smithi188_141483_object5975:head# 2024-04-23T16:43:25.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdc82a8a:::benchmark_data_smithi188_141483_object66174:head# 2024-04-23T16:43:25.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdc85029:::benchmark_data_smithi188_141483_object60150:head# 2024-04-23T16:43:25.044 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdc86e57:::benchmark_data_smithi188_141483_object79451:head# 2024-04-23T16:43:25.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdc91c88:::benchmark_data_smithi188_141483_object77453:head# 2024-04-23T16:43:25.045 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdc98a47:::benchmark_data_smithi188_141483_object18374:head# 2024-04-23T16:43:25.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdc99f12:::benchmark_data_smithi188_141483_object32604:head# 2024-04-23T16:43:25.046 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdc9a86a:::benchmark_data_smithi188_141483_object26625:head# 2024-04-23T16:43:25.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdc9f9a3:::benchmark_data_smithi188_141483_object79972:head# 2024-04-23T16:43:25.047 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdca39ef:::benchmark_data_smithi188_141483_object80883:head# 2024-04-23T16:43:25.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdca7e4f:::benchmark_data_smithi188_141483_object14726:head# 2024-04-23T16:43:25.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdcaa0ee:::benchmark_data_smithi188_141483_object84807:head# 2024-04-23T16:43:25.048 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdcfe04d:::benchmark_data_smithi188_141483_object79970:head# 2024-04-23T16:43:25.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdd0dcae:::benchmark_data_smithi188_141483_object23457:head# 2024-04-23T16:43:25.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdd1da8d:::benchmark_data_smithi188_141483_object68257:head# 2024-04-23T16:43:25.049 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdd1dd76:::benchmark_data_smithi188_141483_object6749:head# 2024-04-23T16:43:25.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdd1e30e:::benchmark_data_smithi188_141483_object23755:head# 2024-04-23T16:43:25.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdd381c5:::benchmark_data_smithi188_141483_object55182:head# 2024-04-23T16:43:25.050 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdd3f864:::benchmark_data_smithi188_141483_object69557:head# 2024-04-23T16:43:25.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdd580b1:::benchmark_data_smithi188_141483_object49549:head# 2024-04-23T16:43:25.051 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdd6debb:::benchmark_data_smithi188_141483_object65206:head# 2024-04-23T16:43:25.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdd6e152:::benchmark_data_smithi188_141483_object87774:head# 2024-04-23T16:43:25.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdd7134c:::benchmark_data_smithi188_141483_object37000:head# 2024-04-23T16:43:25.052 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdd78ac6:::benchmark_data_smithi188_141483_object89056:head# 2024-04-23T16:43:25.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdd78b9b:::benchmark_data_smithi188_141483_object51876:head# 2024-04-23T16:43:25.053 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdd86196:::benchmark_data_smithi188_141483_object49178:head# 2024-04-23T16:43:25.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdd8e081:::benchmark_data_smithi188_141483_object83686:head# 2024-04-23T16:43:25.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdd93313:::benchmark_data_smithi188_141483_object82409:head# 2024-04-23T16:43:25.054 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdd9802b:::benchmark_data_smithi188_141483_object79740:head# 2024-04-23T16:43:25.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdd9b5e7:::benchmark_data_smithi188_141483_object83483:head# 2024-04-23T16:43:25.055 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdda4486:::benchmark_data_smithi188_141483_object61666:head# 2024-04-23T16:43:25.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdda6baa:::benchmark_data_smithi188_141483_object22016:head# 2024-04-23T16:43:25.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cddb14e6:::benchmark_data_smithi188_141483_object12225:head# 2024-04-23T16:43:25.056 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cddb3365:::benchmark_data_smithi188_141483_object7106:head# 2024-04-23T16:43:25.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cddcef69:::benchmark_data_smithi188_141483_object5751:head# 2024-04-23T16:43:25.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdddf1b3:::benchmark_data_smithi188_141483_object79640:head# 2024-04-23T16:43:25.057 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cddeade0:::benchmark_data_smithi188_141483_object47948:head# 2024-04-23T16:43:25.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cde0123f:::benchmark_data_smithi188_141483_object4669:head# 2024-04-23T16:43:25.058 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cde10a25:::benchmark_data_smithi188_141483_object20733:head# 2024-04-23T16:43:25.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cde11056:::benchmark_data_smithi188_141483_object51330:head# 2024-04-23T16:43:25.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cde2f84f:::benchmark_data_smithi188_141483_object55217:head# 2024-04-23T16:43:25.059 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cde3b3e6:::benchmark_data_smithi188_141483_object22671:head# 2024-04-23T16:43:25.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cde5196a:::benchmark_data_smithi188_141483_object12799:head# 2024-04-23T16:43:25.060 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cde52b47:::benchmark_data_smithi188_141483_object41159:head# 2024-04-23T16:43:25.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cde5a88e:::benchmark_data_smithi188_141483_object28894:head# 2024-04-23T16:43:25.061 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cde5d933:::benchmark_data_smithi188_141483_object36132:head# 2024-04-23T16:43:25.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cde60739:::benchmark_data_smithi188_141483_object52258:head# 2024-04-23T16:43:25.062 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cde6876a:::benchmark_data_smithi188_141483_object1287:head# 2024-04-23T16:43:25.063 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cde8009f:::benchmark_data_smithi188_141483_object17299:head# 2024-04-23T16:43:25.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cde99672:::benchmark_data_smithi188_141483_object40307:head# 2024-04-23T16:43:25.064 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdec0f79:::benchmark_data_smithi188_141483_object89241:head# 2024-04-23T16:43:25.066 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdec724b:::benchmark_data_smithi188_141483_object90094:head# 2024-04-23T16:43:25.068 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cded206a:::benchmark_data_smithi188_141483_object23218:head# 2024-04-23T16:43:25.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cded236d:::benchmark_data_smithi188_141483_object77912:head# 2024-04-23T16:43:25.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cded4776:::benchmark_data_smithi188_141483_object88892:head# 2024-04-23T16:43:25.069 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdedae96:::benchmark_data_smithi188_141483_object65108:head# 2024-04-23T16:43:25.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdee50e3:::benchmark_data_smithi188_141483_object29110:head# 2024-04-23T16:43:25.070 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdef3b05:::benchmark_data_smithi188_141483_object73915:head# 2024-04-23T16:43:25.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdef8fe6:::benchmark_data_smithi188_141483_object86485:head# 2024-04-23T16:43:25.071 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdefd105:::benchmark_data_smithi188_141483_object79854:head# 2024-04-23T16:43:25.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdf101e6:::benchmark_data_smithi188_141483_object76786:head# 2024-04-23T16:43:25.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdf257dc:::benchmark_data_smithi188_141483_object3920:head# 2024-04-23T16:43:25.072 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdf308be:::benchmark_data_smithi188_141483_object68249:head# 2024-04-23T16:43:25.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdf38d68:::benchmark_data_smithi188_141483_object60050:head# 2024-04-23T16:43:25.073 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdf3cf9f:::benchmark_data_smithi188_141483_object7115:head# 2024-04-23T16:43:25.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdf44758:::benchmark_data_smithi188_141483_object44299:head# 2024-04-23T16:43:25.074 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdf462ce:::benchmark_data_smithi188_141483_object49832:head# 2024-04-23T16:43:25.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdf47924:::benchmark_data_smithi188_141483_object82207:head# 2024-04-23T16:43:25.075 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdf4c7d9:::benchmark_data_smithi188_141483_object79947:head# 2024-04-23T16:43:25.076 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdf50712:::benchmark_data_smithi188_141483_object74271:head# 2024-04-23T16:43:25.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdf58e97:::benchmark_data_smithi188_141483_object71731:head# 2024-04-23T16:43:25.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdf62bbe:::benchmark_data_smithi188_141483_object7598:head# 2024-04-23T16:43:25.079 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdf7c7a7:::benchmark_data_smithi188_141483_object31723:head# 2024-04-23T16:43:25.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdf7d478:::benchmark_data_smithi188_141483_object4540:head# 2024-04-23T16:43:25.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdf90288:::benchmark_data_smithi188_141483_object5271:head# 2024-04-23T16:43:25.080 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdf906c4:::benchmark_data_smithi188_141483_object28003:head# 2024-04-23T16:43:25.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdf93663:::benchmark_data_smithi188_141483_object55999:head# 2024-04-23T16:43:25.081 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdfaabac:::benchmark_data_smithi188_141483_object51971:head# 2024-04-23T16:43:25.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdfc03f3:::benchmark_data_smithi188_141483_object58817:head# 2024-04-23T16:43:25.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdfe65c3:::benchmark_data_smithi188_141483_object29184:head# 2024-04-23T16:43:25.082 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cdff2e69:::benchmark_data_smithi188_141483_object83115:head# 2024-04-23T16:43:25.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:24 smithi103 ceph-mon[39629]: 7.4 scrub starts 2024-04-23T16:43:25.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:24 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:43:25.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:24 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:43:25.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:24 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 21574/201838 objects degraded (10.689%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:25.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce00ada1:::benchmark_data_smithi188_141483_object18094:head# 2024-04-23T16:43:25.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce00e886:::benchmark_data_smithi188_141483_object15493:head# 2024-04-23T16:43:25.126 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce01115e:::benchmark_data_smithi188_141483_object52999:head# 2024-04-23T16:43:25.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce01335c:::benchmark_data_smithi188_141483_object25217:head# 2024-04-23T16:43:25.127 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce035e3c:::benchmark_data_smithi188_141483_object35994:head# 2024-04-23T16:43:25.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce039016:::benchmark_data_smithi188_141483_object54667:head# 2024-04-23T16:43:25.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce043de1:::benchmark_data_smithi188_141483_object81317:head# 2024-04-23T16:43:25.128 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce046d64:::benchmark_data_smithi188_141483_object60323:head# 2024-04-23T16:43:25.129 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce05ab39:::benchmark_data_smithi188_141483_object12502:head# 2024-04-23T16:43:25.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:24 smithi148 ceph-mon[44520]: 7.4 scrub starts 2024-04-23T16:43:25.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:24 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:43:25.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:24 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:43:25.130 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:24 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 21574/201838 objects degraded (10.689%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:25.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce060289:::benchmark_data_smithi188_141483_object24447:head# 2024-04-23T16:43:25.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce06548b:::benchmark_data_smithi188_141483_object35954:head# 2024-04-23T16:43:25.130 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce06602f:::benchmark_data_smithi188_141483_object13852:head# 2024-04-23T16:43:25.131 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce07f3d1:::benchmark_data_smithi188_141483_object39101:head# 2024-04-23T16:43:25.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce0834f4:::benchmark_data_smithi188_141483_object8856:head# 2024-04-23T16:43:25.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce097b2d:::benchmark_data_smithi188_141483_object2093:head# 2024-04-23T16:43:25.132 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce09ea93:::benchmark_data_smithi188_141483_object15603:head# 2024-04-23T16:43:25.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce0ac640:::benchmark_data_smithi188_141483_object2309:head# 2024-04-23T16:43:25.133 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce0b020b:::benchmark_data_smithi188_141483_object68557:head# 2024-04-23T16:43:25.134 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce0b62f6:::benchmark_data_smithi188_141483_object62729:head# 2024-04-23T16:43:25.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce0b6b8d:::benchmark_data_smithi188_141483_object14108:head# 2024-04-23T16:43:25.135 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce0b8401:::benchmark_data_smithi188_141483_object62383:head# 2024-04-23T16:43:25.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce0ba633:::benchmark_data_smithi188_141483_object55855:head# 2024-04-23T16:43:25.136 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce0bfff1:::benchmark_data_smithi188_141483_object576:head# 2024-04-23T16:43:25.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce0c7f33:::benchmark_data_smithi188_141483_object12824:head# 2024-04-23T16:43:25.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce0e740f:::benchmark_data_smithi188_141483_object37910:head# 2024-04-23T16:43:25.137 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce0f5faa:::benchmark_data_smithi188_141483_object31716:head# 2024-04-23T16:43:25.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce0fee8a:::benchmark_data_smithi188_141483_object78534:head# 2024-04-23T16:43:25.138 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce1064c3:::benchmark_data_smithi188_141483_object47562:head# 2024-04-23T16:43:25.139 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce1099ab:::benchmark_data_smithi188_141483_object27958:head# 2024-04-23T16:43:25.142 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce10badb:::benchmark_data_smithi188_141483_object4357:head# 2024-04-23T16:43:25.142 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce11313f:::benchmark_data_smithi188_141483_object72960:head# 2024-04-23T16:43:25.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce1181ff:::benchmark_data_smithi188_141483_object45934:head# 2024-04-23T16:43:25.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce11d510:::benchmark_data_smithi188_141483_object68022:head# 2024-04-23T16:43:25.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce128ea6:::benchmark_data_smithi188_141483_object44370:head# 2024-04-23T16:43:25.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce12dd9b:::benchmark_data_smithi188_141483_object41037:head# 2024-04-23T16:43:25.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce131a27:::benchmark_data_smithi188_141483_object61271:head# 2024-04-23T16:43:25.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce138f1d:::benchmark_data_smithi188_141483_object6210:head# 2024-04-23T16:43:25.143 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce144390:::benchmark_data_smithi188_141483_object17441:head# 2024-04-23T16:43:25.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce151497:::benchmark_data_smithi188_141483_object83996:head# 2024-04-23T16:43:25.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce155420:::benchmark_data_smithi188_141483_object18061:head# 2024-04-23T16:43:25.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce15da52:::benchmark_data_smithi188_141483_object53218:head# 2024-04-23T16:43:25.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce15dc3e:::benchmark_data_smithi188_141483_object8034:head# 2024-04-23T16:43:25.144 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce15edda:::benchmark_data_smithi188_141483_object25369:head# 2024-04-23T16:43:25.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce165f0e:::benchmark_data_smithi188_141483_object85967:head# 2024-04-23T16:43:25.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce171f80:::benchmark_data_smithi188_141483_object72211:head# 2024-04-23T16:43:25.145 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce175225:::benchmark_data_smithi188_141483_object42845:head# 2024-04-23T16:43:25.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce185011:::benchmark_data_smithi188_141483_object72923:head# 2024-04-23T16:43:25.146 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce18b5ad:::benchmark_data_smithi188_141483_object52493:head# 2024-04-23T16:43:25.148 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce18b773:::benchmark_data_smithi188_141483_object10395:head# 2024-04-23T16:43:25.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce18cbfd:::benchmark_data_smithi188_141483_object8460:head# 2024-04-23T16:43:25.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce1952f5:::benchmark_data_smithi188_141483_object21415:head# 2024-04-23T16:43:25.149 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce1978d8:::benchmark_data_smithi188_141483_object80620:head# 2024-04-23T16:43:25.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce1b06c1:::benchmark_data_smithi188_141483_object21048:head# 2024-04-23T16:43:25.150 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce1b203f:::benchmark_data_smithi188_141483_object16585:head# 2024-04-23T16:43:25.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce1b35e1:::benchmark_data_smithi188_141483_object26634:head# 2024-04-23T16:43:25.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce1bc60a:::benchmark_data_smithi188_141483_object19653:head# 2024-04-23T16:43:25.151 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce1c084b:::benchmark_data_smithi188_141483_object13267:head# 2024-04-23T16:43:25.152 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce1d2320:::benchmark_data_smithi188_141483_object12278:head# 2024-04-23T16:43:25.154 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce1e767d:::benchmark_data_smithi188_141483_object75519:head# 2024-04-23T16:43:25.155 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce1ec97d:::benchmark_data_smithi188_141483_object3284:head# 2024-04-23T16:43:25.155 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce1fba05:::benchmark_data_smithi188_141483_object46101:head# 2024-04-23T16:43:25.155 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce21f2df:::benchmark_data_smithi188_141483_object14400:head# 2024-04-23T16:43:25.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce280432:::benchmark_data_smithi188_141483_object1744:head# 2024-04-23T16:43:25.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce284ca6:::benchmark_data_smithi188_141483_object21999:head# 2024-04-23T16:43:25.156 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce295f04:::benchmark_data_smithi188_141483_object48413:head# 2024-04-23T16:43:25.157 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce298864:::benchmark_data_smithi188_141483_object76822:head# 2024-04-23T16:43:25.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce2a2c8d:::benchmark_data_smithi188_141483_object50999:head# 2024-04-23T16:43:25.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce2a8827:::benchmark_data_smithi188_141483_object12476:head# 2024-04-23T16:43:25.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce2b28ce:::benchmark_data_smithi188_141483_object59449:head# 2024-04-23T16:43:25.159 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce2c38bd:::benchmark_data_smithi188_141483_object41377:head# 2024-04-23T16:43:25.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce2d23cf:::benchmark_data_smithi188_141483_object66726:head# 2024-04-23T16:43:25.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce2dbe27:::benchmark_data_smithi188_141483_object2237:head# 2024-04-23T16:43:25.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce2dfbf9:::benchmark_data_smithi188_141483_object27943:head# 2024-04-23T16:43:25.160 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce2efa36:::benchmark_data_smithi188_141483_object8807:head# 2024-04-23T16:43:25.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce2fad0c:::benchmark_data_smithi188_141483_object75565:head# 2024-04-23T16:43:25.161 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce2fb9ab:::benchmark_data_smithi188_141483_object84253:head# 2024-04-23T16:43:25.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce2fd219:::benchmark_data_smithi188_141483_object61118:head# 2024-04-23T16:43:25.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce304799:::benchmark_data_smithi188_141483_object29673:head# 2024-04-23T16:43:25.162 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce309cdd:::benchmark_data_smithi188_141483_object89755:head# 2024-04-23T16:43:25.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce30e52e:::benchmark_data_smithi188_141483_object70566:head# 2024-04-23T16:43:25.163 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce3205ff:::benchmark_data_smithi188_141483_object21990:head# 2024-04-23T16:43:25.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce32c1ea:::benchmark_data_smithi188_141483_object82476:head# 2024-04-23T16:43:25.164 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce32ff0e:::benchmark_data_smithi188_141483_object7182:head# 2024-04-23T16:43:25.165 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce357234:::benchmark_data_smithi188_141483_object9958:head# 2024-04-23T16:43:25.166 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce36d58f:::benchmark_data_smithi188_141483_object14118:head# 2024-04-23T16:43:25.167 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce37016d:::benchmark_data_smithi188_141483_object13040:head# 2024-04-23T16:43:25.167 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce379094:::benchmark_data_smithi188_141483_object4765:head# 2024-04-23T16:43:25.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce37983a:::benchmark_data_smithi188_141483_object11270:head# 2024-04-23T16:43:25.168 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce38537f:::benchmark_data_smithi188_141483_object74327:head# 2024-04-23T16:43:25.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce386453:::benchmark_data_smithi188_141483_object33109:head# 2024-04-23T16:43:25.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce386457:::benchmark_data_smithi188_141483_object34173:head# 2024-04-23T16:43:25.169 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce396806:::benchmark_data_smithi188_141483_object51172:head# 2024-04-23T16:43:25.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce39827d:::benchmark_data_smithi188_141483_object68233:head# 2024-04-23T16:43:25.170 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce39ed25:::benchmark_data_smithi188_141483_object10650:head# 2024-04-23T16:43:25.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce3a2530:::benchmark_data_smithi188_141483_object82211:head# 2024-04-23T16:43:25.171 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce3a879d:::benchmark_data_smithi188_141483_object17328:head# 2024-04-23T16:43:25.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce3b8093:::benchmark_data_smithi188_141483_object6229:head# 2024-04-23T16:43:25.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce3e21c0:::benchmark_data_smithi188_141483_object3850:head# 2024-04-23T16:43:25.172 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce3f4983:::benchmark_data_smithi188_141483_object58683:head# 2024-04-23T16:43:25.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce3f5708:::benchmark_data_smithi188_141483_object35598:head# 2024-04-23T16:43:25.173 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce407e1d:::benchmark_data_smithi188_141483_object32534:head# 2024-04-23T16:43:25.174 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce40f11f:::benchmark_data_smithi188_141483_object56400:head# 2024-04-23T16:43:25.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce40fd9b:::benchmark_data_smithi188_141483_object78689:head# 2024-04-23T16:43:25.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce41dcb6:::benchmark_data_smithi188_141483_object76491:head# 2024-04-23T16:43:25.175 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce42a5a0:::benchmark_data_smithi188_141483_object60211:head# 2024-04-23T16:43:25.176 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce462d3b:::benchmark_data_smithi188_141483_object76072:head# 2024-04-23T16:43:25.176 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce489d9d:::benchmark_data_smithi188_141483_object88547:head# 2024-04-23T16:43:25.176 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce48b079:::benchmark_data_smithi188_141483_object86998:head# 2024-04-23T16:43:25.177 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce4b6517:::benchmark_data_smithi188_141483_object12909:head# 2024-04-23T16:43:25.177 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce4c48c7:::benchmark_data_smithi188_141483_object18017:head# 2024-04-23T16:43:25.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce4d686d:::benchmark_data_smithi188_141483_object50066:head# 2024-04-23T16:43:25.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce4efbdf:::benchmark_data_smithi188_141483_object9574:head# 2024-04-23T16:43:25.178 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce4f8258:::benchmark_data_smithi188_141483_object88371:head# 2024-04-23T16:43:25.179 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce4f9f91:::benchmark_data_smithi188_141483_object50108:head# 2024-04-23T16:43:25.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce501117:::benchmark_data_smithi188_141483_object86496:head# 2024-04-23T16:43:25.180 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce507e33:::benchmark_data_smithi188_141483_object35564:head# 2024-04-23T16:43:25.181 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce50f790:::benchmark_data_smithi188_141483_object15511:head# 2024-04-23T16:43:25.181 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce510a81:::benchmark_data_smithi188_141483_object26961:head# 2024-04-23T16:43:25.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce5223c6:::benchmark_data_smithi188_141483_object68272:head# 2024-04-23T16:43:25.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce52a351:::benchmark_data_smithi188_141483_object2045:head# 2024-04-23T16:43:25.182 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce5320eb:::benchmark_data_smithi188_141483_object83112:head# 2024-04-23T16:43:25.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce53261d:::benchmark_data_smithi188_141483_object36790:head# 2024-04-23T16:43:25.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce55d040:::benchmark_data_smithi188_141483_object23560:head# 2024-04-23T16:43:25.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce55dfe9:::benchmark_data_smithi188_141483_object5038:head# 2024-04-23T16:43:25.184 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce582a25:::benchmark_data_smithi188_141483_object432:head# 2024-04-23T16:43:25.185 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce586ccf:::benchmark_data_smithi188_141483_object78369:head# 2024-04-23T16:43:25.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce58e9d2:::benchmark_data_smithi188_141483_object1818:head# 2024-04-23T16:43:25.186 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce5afce0:::benchmark_data_smithi188_141483_object5729:head# 2024-04-23T16:43:25.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce5c1bc3:::benchmark_data_smithi188_141483_object7125:head# 2024-04-23T16:43:25.187 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce5c4125:::benchmark_data_smithi188_141483_object15507:head# 2024-04-23T16:43:25.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce5c730b:::benchmark_data_smithi188_141483_object79833:head# 2024-04-23T16:43:25.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce5c87d4:::benchmark_data_smithi188_141483_object31458:head# 2024-04-23T16:43:25.188 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce5d20a0:::benchmark_data_smithi188_141483_object40411:head# 2024-04-23T16:43:25.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce5d251b:::benchmark_data_smithi188_141483_object78612:head# 2024-04-23T16:43:25.189 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce5d8107:::benchmark_data_smithi188_141483_object20281:head# 2024-04-23T16:43:25.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce5df78c:::benchmark_data_smithi188_141483_object72704:head# 2024-04-23T16:43:25.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce5e5288:::benchmark_data_smithi188_141483_object5672:head# 2024-04-23T16:43:25.190 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce61b337:::benchmark_data_smithi188_141483_object14023:head# 2024-04-23T16:43:25.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce627702:::benchmark_data_smithi188_141483_object68219:head# 2024-04-23T16:43:25.191 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce628fb1:::benchmark_data_smithi188_141483_object21321:head# 2024-04-23T16:43:25.192 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce6327e3:::benchmark_data_smithi188_141483_object53077:head# 2024-04-23T16:43:25.192 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce636422:::benchmark_data_smithi188_141483_object35975:head# 2024-04-23T16:43:25.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce6415c3:::benchmark_data_smithi188_141483_object23096:head# 2024-04-23T16:43:25.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce641cfc:::benchmark_data_smithi188_141483_object54195:head# 2024-04-23T16:43:25.193 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce6428f1:::benchmark_data_smithi188_141483_object39319:head# 2024-04-23T16:43:25.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce650b8b:::benchmark_data_smithi188_141483_object32518:head# 2024-04-23T16:43:25.194 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce658ad8:::benchmark_data_smithi188_141483_object23500:head# 2024-04-23T16:43:25.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce659ef3:::benchmark_data_smithi188_141483_object16736:head# 2024-04-23T16:43:25.195 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce661437:::benchmark_data_smithi188_141483_object1150:head# 2024-04-23T16:43:25.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce665134:::benchmark_data_smithi188_141483_object81047:head# 2024-04-23T16:43:25.196 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce6678b8:::benchmark_data_smithi188_141483_object5969:head# 2024-04-23T16:43:25.197 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce66ddaf:::benchmark_data_smithi188_141483_object24441:head# 2024-04-23T16:43:25.197 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce671999:::benchmark_data_smithi188_141483_object6274:head# 2024-04-23T16:43:25.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce675513:::benchmark_data_smithi188_141483_object27399:head# 2024-04-23T16:43:25.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce68dccd:::benchmark_data_smithi188_141483_object86338:head# 2024-04-23T16:43:25.198 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce69afa4:::benchmark_data_smithi188_141483_object81973:head# 2024-04-23T16:43:25.199 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce6a106f:::benchmark_data_smithi188_141483_object32036:head# 2024-04-23T16:43:25.199 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce6ab202:::benchmark_data_smithi188_141483_object60165:head# 2024-04-23T16:43:25.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce6b03a2:::benchmark_data_smithi188_141483_object83973:head# 2024-04-23T16:43:25.200 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce6c125a:::benchmark_data_smithi188_141483_object60839:head# 2024-04-23T16:43:25.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce6c1bbf:::benchmark_data_smithi188_141483_object81254:head# 2024-04-23T16:43:25.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce6cf0e4:::benchmark_data_smithi188_141483_object37505:head# 2024-04-23T16:43:25.201 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce6d6923:::benchmark_data_smithi188_141483_object85479:head# 2024-04-23T16:43:25.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce6e3af1:::benchmark_data_smithi188_141483_object66603:head# 2024-04-23T16:43:25.202 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce6f2b56:::benchmark_data_smithi188_141483_object43172:head# 2024-04-23T16:43:25.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce710be8:::benchmark_data_smithi188_141483_object67246:head# 2024-04-23T16:43:25.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce713708:::benchmark_data_smithi188_141483_object61726:head# 2024-04-23T16:43:25.203 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce714c48:::benchmark_data_smithi188_141483_object43869:head# 2024-04-23T16:43:25.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce71c911:::benchmark_data_smithi188_141483_object41433:head# 2024-04-23T16:43:25.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce723227:::benchmark_data_smithi188_141483_object13771:head# 2024-04-23T16:43:25.204 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce72ab4a:::benchmark_data_smithi188_141483_object22939:head# 2024-04-23T16:43:25.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce73f200:::benchmark_data_smithi188_141483_object84405:head# 2024-04-23T16:43:25.205 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce73f7d5:::benchmark_data_smithi188_141483_object60139:head# 2024-04-23T16:43:25.206 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce7467d7:::benchmark_data_smithi188_141483_object34096:head# 2024-04-23T16:43:25.206 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce74c6cb:::benchmark_data_smithi188_141483_object39798:head# 2024-04-23T16:43:25.206 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce76edc3:::benchmark_data_smithi188_141483_object77533:head# 2024-04-23T16:43:25.207 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce773d82:::benchmark_data_smithi188_141483_object70248:head# 2024-04-23T16:43:25.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce777b88:::benchmark_data_smithi188_141483_object42239:head# 2024-04-23T16:43:25.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce78ca18:::benchmark_data_smithi188_141483_object89948:head# 2024-04-23T16:43:25.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce79005e:::benchmark_data_smithi188_141483_object53332:head# 2024-04-23T16:43:25.208 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce799014:::benchmark_data_smithi188_141483_object17163:head# 2024-04-23T16:43:25.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce7a48b4:::benchmark_data_smithi188_141483_object65704:head# 2024-04-23T16:43:25.209 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce7acbf1:::benchmark_data_smithi188_141483_object53145:head# 2024-04-23T16:43:25.210 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce7b7978:::benchmark_data_smithi188_141483_object59684:head# 2024-04-23T16:43:25.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce7bd735:::benchmark_data_smithi188_141483_object71800:head# 2024-04-23T16:43:25.211 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce7d0178:::benchmark_data_smithi188_141483_object20701:head# 2024-04-23T16:43:25.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce7f6686:::benchmark_data_smithi188_141483_object90884:head# 2024-04-23T16:43:25.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce7f85b7:::benchmark_data_smithi188_141483_object34937:head# 2024-04-23T16:43:25.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce804367:::benchmark_data_smithi188_141483_object68208:head# 2024-04-23T16:43:25.214 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce80a16c:::benchmark_data_smithi188_141483_object66660:head# 2024-04-23T16:43:25.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce80e25d:::benchmark_data_smithi188_141483_object57641:head# 2024-04-23T16:43:25.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce81dd7b:::benchmark_data_smithi188_141483_object46763:head# 2024-04-23T16:43:25.215 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce838a4e:::benchmark_data_smithi188_141483_object5624:head# 2024-04-23T16:43:25.216 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce847427:::benchmark_data_smithi188_141483_object68641:head# 2024-04-23T16:43:25.218 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce84b502:::benchmark_data_smithi188_141483_object1165:head# 2024-04-23T16:43:25.219 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce84e2dc:::benchmark_data_smithi188_141483_object51563:head# 2024-04-23T16:43:25.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce85c12f:::benchmark_data_smithi188_141483_object57637:head# 2024-04-23T16:43:25.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce85d3aa:::benchmark_data_smithi188_141483_object19704:head# 2024-04-23T16:43:25.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce85e564:::benchmark_data_smithi188_141483_object24055:head# 2024-04-23T16:43:25.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce86f332:::benchmark_data_smithi188_141483_object7974:head# 2024-04-23T16:43:25.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce879fc7:::benchmark_data_smithi188_141483_object53127:head# 2024-04-23T16:43:25.223 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce87db45:::benchmark_data_smithi188_141483_object77925:head# 2024-04-23T16:43:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:24 smithi188 ceph-mon[44101]: 7.4 scrub starts 2024-04-23T16:43:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:24 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:43:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:24 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:43:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:24 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 21574/201838 objects degraded (10.689%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:25.225 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce88329d:::benchmark_data_smithi188_141483_object47587:head# 2024-04-23T16:43:25.225 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce88a3a2:::benchmark_data_smithi188_141483_object30955:head# 2024-04-23T16:43:25.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce88e2bf:::benchmark_data_smithi188_141483_object47428:head# 2024-04-23T16:43:25.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce8942a3:::benchmark_data_smithi188_141483_object22944:head# 2024-04-23T16:43:25.226 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce8942ed:::benchmark_data_smithi188_141483_object4849:head# 2024-04-23T16:43:25.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce8948c6:::benchmark_data_smithi188_141483_object17791:head# 2024-04-23T16:43:25.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce89ee84:::benchmark_data_smithi188_141483_object68354:head# 2024-04-23T16:43:25.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce8a2d6a:::benchmark_data_smithi188_141483_object88395:head# 2024-04-23T16:43:25.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce8ad47c:::benchmark_data_smithi188_141483_object21381:head# 2024-04-23T16:43:25.227 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce8b1351:::benchmark_data_smithi188_141483_object78995:head# 2024-04-23T16:43:25.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce8b6efe:::benchmark_data_smithi188_141483_object34113:head# 2024-04-23T16:43:25.228 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce8bb98a:::benchmark_data_smithi188_141483_object15006:head# 2024-04-23T16:43:25.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce8c0792:::benchmark_data_smithi188_141483_object83119:head# 2024-04-23T16:43:25.229 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce8c6e86:::benchmark_data_smithi188_141483_object30509:head# 2024-04-23T16:43:25.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce8dde34:::benchmark_data_smithi188_141483_object89431:head# 2024-04-23T16:43:25.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce8f5368:::benchmark_data_smithi188_141483_object20558:head# 2024-04-23T16:43:25.230 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce8f5df2:::benchmark_data_smithi188_141483_object18261:head# 2024-04-23T16:43:25.231 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce8f66d6:::benchmark_data_smithi188_141483_object68151:head# 2024-04-23T16:43:25.231 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce910a24:::benchmark_data_smithi188_141483_object18982:head# 2024-04-23T16:43:25.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce917958:::benchmark_data_smithi188_141483_object51466:head# 2024-04-23T16:43:25.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce91f1bd:::benchmark_data_smithi188_141483_object57330:head# 2024-04-23T16:43:25.232 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce9266e3:::benchmark_data_smithi188_141483_object35805:head# 2024-04-23T16:43:25.233 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce929ff2:::benchmark_data_smithi188_141483_object23128:head# 2024-04-23T16:43:25.235 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce93335f:::benchmark_data_smithi188_141483_object43290:head# 2024-04-23T16:43:25.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce93f51c:::benchmark_data_smithi188_141483_object36108:head# 2024-04-23T16:43:25.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce94c529:::benchmark_data_smithi188_141483_object21730:head# 2024-04-23T16:43:25.236 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce963831:::benchmark_data_smithi188_141483_object34558:head# 2024-04-23T16:43:25.237 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce971600:::benchmark_data_smithi188_141483_object65124:head# 2024-04-23T16:43:25.237 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce979be5:::benchmark_data_smithi188_141483_object51405:head# 2024-04-23T16:43:25.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce980428:::benchmark_data_smithi188_141483_object18553:head# 2024-04-23T16:43:25.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce985a59:::benchmark_data_smithi188_141483_object9205:head# 2024-04-23T16:43:25.238 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce99913e:::benchmark_data_smithi188_141483_object80831:head# 2024-04-23T16:43:25.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce9a2c71:::benchmark_data_smithi188_141483_object6372:head# 2024-04-23T16:43:25.239 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce9a4b74:::benchmark_data_smithi188_141483_object90657:head# 2024-04-23T16:43:25.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce9a878f:::benchmark_data_smithi188_141483_object53548:head# 2024-04-23T16:43:25.242 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce9a978d:::benchmark_data_smithi188_141483_object81233:head# 2024-04-23T16:43:25.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce9ba3a0:::benchmark_data_smithi188_141483_object38186:head# 2024-04-23T16:43:25.243 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce9bdcff:::benchmark_data_smithi188_141483_object51166:head# 2024-04-23T16:43:25.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce9c215b:::benchmark_data_smithi188_141483_object9319:head# 2024-04-23T16:43:25.244 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce9cb3ba:::benchmark_data_smithi188_141483_object28363:head# 2024-04-23T16:43:25.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce9cb91d:::benchmark_data_smithi188_141483_object77367:head# 2024-04-23T16:43:25.245 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce9efc47:::benchmark_data_smithi188_141483_object62990:head# 2024-04-23T16:43:25.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ce9f9f3c:::benchmark_data_smithi188_141483_object65352:head# 2024-04-23T16:43:25.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cea05587:::benchmark_data_smithi188_141483_object58815:head# 2024-04-23T16:43:25.246 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cea072ee:::benchmark_data_smithi188_141483_object55836:head# 2024-04-23T16:43:25.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cea1387e:::benchmark_data_smithi188_141483_object89766:head# 2024-04-23T16:43:25.247 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cea203a9:::benchmark_data_smithi188_141483_object76408:head# 2024-04-23T16:43:25.248 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cea23646:::benchmark_data_smithi188_141483_object37050:head# 2024-04-23T16:43:25.248 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cea30394:::benchmark_data_smithi188_141483_object37921:head# 2024-04-23T16:43:25.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cea41d23:::benchmark_data_smithi188_141483_object22756:head# 2024-04-23T16:43:25.249 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cea432e5:::benchmark_data_smithi188_141483_object73688:head# 2024-04-23T16:43:25.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cea46f77:::benchmark_data_smithi188_141483_object2487:head# 2024-04-23T16:43:25.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cea5458c:::benchmark_data_smithi188_141483_object26083:head# 2024-04-23T16:43:25.250 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cea648fd:::benchmark_data_smithi188_141483_object15666:head# 2024-04-23T16:43:25.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cea76907:::benchmark_data_smithi188_141483_object30072:head# 2024-04-23T16:43:25.251 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cea79259:::benchmark_data_smithi188_141483_object2950:head# 2024-04-23T16:43:25.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cea92b4f:::benchmark_data_smithi188_141483_object58397:head# 2024-04-23T16:43:25.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cea98cf8:::benchmark_data_smithi188_141483_object70344:head# 2024-04-23T16:43:25.252 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceaa02e7:::benchmark_data_smithi188_141483_object58133:head# 2024-04-23T16:43:25.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceaafbfa:::benchmark_data_smithi188_141483_object41628:head# 2024-04-23T16:43:25.253 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceab2f14:::benchmark_data_smithi188_141483_object71126:head# 2024-04-23T16:43:25.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceac177a:::benchmark_data_smithi188_141483_object35924:head# 2024-04-23T16:43:25.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cead2ffb:::benchmark_data_smithi188_141483_object90379:head# 2024-04-23T16:43:25.254 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cead8f1f:::benchmark_data_smithi188_141483_object62511:head# 2024-04-23T16:43:25.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceae733f:::benchmark_data_smithi188_141483_object46263:head# 2024-04-23T16:43:25.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceaeb864:::benchmark_data_smithi188_141483_object13301:head# 2024-04-23T16:43:25.255 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceaef81b:::benchmark_data_smithi188_141483_object18062:head# 2024-04-23T16:43:25.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceaf5721:::benchmark_data_smithi188_141483_object45469:head# 2024-04-23T16:43:25.256 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceaf9cde:::benchmark_data_smithi188_141483_object44686:head# 2024-04-23T16:43:25.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb05afa:::benchmark_data_smithi188_141483_object72049:head# 2024-04-23T16:43:25.257 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb05d63:::benchmark_data_smithi188_141483_object61345:head# 2024-04-23T16:43:25.258 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb0cc52:::benchmark_data_smithi188_141483_object51010:head# 2024-04-23T16:43:25.258 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb110b8:::benchmark_data_smithi188_141483_object90627:head# 2024-04-23T16:43:25.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb2190c:::benchmark_data_smithi188_141483_object1118:head# 2024-04-23T16:43:25.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb24baf:::benchmark_data_smithi188_141483_object76780:head# 2024-04-23T16:43:25.259 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb2cd76:::benchmark_data_smithi188_141483_object24961:head# 2024-04-23T16:43:25.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb30821:::benchmark_data_smithi188_141483_object28241:head# 2024-04-23T16:43:25.260 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb3acaf:::benchmark_data_smithi188_141483_object16075:head# 2024-04-23T16:43:25.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb3db8d:::benchmark_data_smithi188_141483_object88375:head# 2024-04-23T16:43:25.261 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb43a4a:::benchmark_data_smithi188_141483_object18760:head# 2024-04-23T16:43:25.262 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb552f6:::benchmark_data_smithi188_141483_object21850:head# 2024-04-23T16:43:25.262 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb6690b:::benchmark_data_smithi188_141483_object26028:head# 2024-04-23T16:43:25.262 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb69f88:::benchmark_data_smithi188_141483_object10025:head# 2024-04-23T16:43:25.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb6ac95:::benchmark_data_smithi188_141483_object77054:head# 2024-04-23T16:43:25.263 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb7d312:::benchmark_data_smithi188_141483_object25135:head# 2024-04-23T16:43:25.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb80b1c:::benchmark_data_smithi188_141483_object83050:head# 2024-04-23T16:43:25.264 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb80e2d:::benchmark_data_smithi188_141483_object73970:head# 2024-04-23T16:43:25.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb86f41:::benchmark_data_smithi188_141483_object43560:head# 2024-04-23T16:43:25.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceb9ebe3:::benchmark_data_smithi188_141483_object41305:head# 2024-04-23T16:43:25.265 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceba1a4d:::benchmark_data_smithi188_141483_object43526:head# 2024-04-23T16:43:25.266 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cebac2ad:::benchmark_data_smithi188_141483_object29446:head# 2024-04-23T16:43:25.266 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cebc804a:::benchmark_data_smithi188_141483_object88412:head# 2024-04-23T16:43:25.267 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cebcb243:::benchmark_data_smithi188_141483_object81516:head# 2024-04-23T16:43:25.267 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cebee301:::benchmark_data_smithi188_141483_object58580:head# 2024-04-23T16:43:25.267 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cebf47ec:::benchmark_data_smithi188_141483_object44748:head# 2024-04-23T16:43:25.268 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cebf9643:::benchmark_data_smithi188_141483_object42145:head# 2024-04-23T16:43:25.268 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cebfe7ae:::benchmark_data_smithi188_141483_object8977:head# 2024-04-23T16:43:25.268 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec0b8ad:::benchmark_data_smithi188_141483_object12504:head# 2024-04-23T16:43:25.269 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec0fcf6:::benchmark_data_smithi188_141483_object67899:head# 2024-04-23T16:43:25.269 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec15535:::benchmark_data_smithi188_141483_object14984:head# 2024-04-23T16:43:25.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec173f4:::benchmark_data_smithi188_141483_object22029:head# 2024-04-23T16:43:25.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec1bc08:::benchmark_data_smithi188_141483_object13221:head# 2024-04-23T16:43:25.270 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec2320d:::benchmark_data_smithi188_141483_object23481:head# 2024-04-23T16:43:25.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec29c5b:::benchmark_data_smithi188_141483_object81983:head# 2024-04-23T16:43:25.271 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec2de8f:::benchmark_data_smithi188_141483_object3567:head# 2024-04-23T16:43:25.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec30aea:::benchmark_data_smithi188_141483_object52792:head# 2024-04-23T16:43:25.272 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec36fd5:::benchmark_data_smithi188_141483_object57365:head# 2024-04-23T16:43:25.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec37762:::benchmark_data_smithi188_141483_object79931:head# 2024-04-23T16:43:25.273 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec37949:::benchmark_data_smithi188_141483_object12821:head# 2024-04-23T16:43:25.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec47afe:::benchmark_data_smithi188_141483_object89952:head# 2024-04-23T16:43:25.274 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec5245d:::benchmark_data_smithi188_141483_object66562:head# 2024-04-23T16:43:25.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec577a5:::benchmark_data_smithi188_141483_object70000:head# 2024-04-23T16:43:25.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec5c440:::benchmark_data_smithi188_141483_object32850:head# 2024-04-23T16:43:25.275 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec60be1:::benchmark_data_smithi188_141483_object4827:head# 2024-04-23T16:43:25.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec61d22:::benchmark_data_smithi188_141483_object16689:head# 2024-04-23T16:43:25.276 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec69c88:::benchmark_data_smithi188_141483_object42340:head# 2024-04-23T16:43:25.277 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec6fc09:::benchmark_data_smithi188_141483_object458:head# 2024-04-23T16:43:25.277 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec7d85f:::benchmark_data_smithi188_141483_object4004:head# 2024-04-23T16:43:25.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec823a5:::benchmark_data_smithi188_141483_object54100:head# 2024-04-23T16:43:25.278 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cec8f1c9:::benchmark_data_smithi188_141483_object21293:head# 2024-04-23T16:43:25.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceca6c14:::benchmark_data_smithi188_141483_object19909:head# 2024-04-23T16:43:25.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cecb17ec:::benchmark_data_smithi188_141483_object22720:head# 2024-04-23T16:43:25.279 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cecb981f:::benchmark_data_smithi188_141483_object6433:head# 2024-04-23T16:43:25.280 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cecc087c:::benchmark_data_smithi188_141483_object37544:head# 2024-04-23T16:43:25.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cecd5586:::benchmark_data_smithi188_141483_object84064:head# 2024-04-23T16:43:25.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cecd7d73:::benchmark_data_smithi188_141483_object79784:head# 2024-04-23T16:43:25.281 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cece022a:::benchmark_data_smithi188_141483_object57339:head# 2024-04-23T16:43:25.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cecef57a:::benchmark_data_smithi188_141483_object4045:head# 2024-04-23T16:43:25.282 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cecf474e:::benchmark_data_smithi188_141483_object23742:head# 2024-04-23T16:43:25.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cecfa06a:::benchmark_data_smithi188_141483_object59910:head# 2024-04-23T16:43:25.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ced0403b:::benchmark_data_smithi188_141483_object31604:head# 2024-04-23T16:43:25.283 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ced09dfe:::benchmark_data_smithi188_141483_object62216:head# 2024-04-23T16:43:25.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ced1ebb6:::benchmark_data_smithi188_141483_object82246:head# 2024-04-23T16:43:25.284 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ced2122e:::benchmark_data_smithi188_141483_object7880:head# 2024-04-23T16:43:25.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ced38f46:::benchmark_data_smithi188_141483_object50989:head# 2024-04-23T16:43:25.285 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ced405b0:::benchmark_data_smithi188_141483_object75615:head# 2024-04-23T16:43:25.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ced60e86:::benchmark_data_smithi188_141483_object38405:head# 2024-04-23T16:43:25.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ced81ea9:::benchmark_data_smithi188_141483_object16092:head# 2024-04-23T16:43:25.286 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ced89613:::benchmark_data_smithi188_141483_object87719:head# 2024-04-23T16:43:25.287 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ced8cc0b:::benchmark_data_smithi188_141483_object31399:head# 2024-04-23T16:43:25.287 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceda6708:::benchmark_data_smithi188_141483_object62031:head# 2024-04-23T16:43:25.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cedb942d:::benchmark_data_smithi188_141483_object7757:head# 2024-04-23T16:43:25.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cedc6ded:::benchmark_data_smithi188_141483_object88003:head# 2024-04-23T16:43:25.288 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cedca4bf:::benchmark_data_smithi188_141483_object71485:head# 2024-04-23T16:43:25.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cedd99e2:::benchmark_data_smithi188_141483_object21978:head# 2024-04-23T16:43:25.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cede0a0e:::benchmark_data_smithi188_141483_object11053:head# 2024-04-23T16:43:25.289 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cedee90a:::benchmark_data_smithi188_141483_object78056:head# 2024-04-23T16:43:25.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cedfce81:::benchmark_data_smithi188_141483_object83145:head# 2024-04-23T16:43:25.290 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cee192f2:::benchmark_data_smithi188_141483_object59617:head# 2024-04-23T16:43:25.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cee1dcb6:::benchmark_data_smithi188_141483_object80741:head# 2024-04-23T16:43:25.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cee2f8a1:::benchmark_data_smithi188_141483_object16621:head# 2024-04-23T16:43:25.291 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cee39187:::benchmark_data_smithi188_141483_object20624:head# 2024-04-23T16:43:25.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cee49aee:::benchmark_data_smithi188_141483_object4091:head# 2024-04-23T16:43:25.292 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cee5ca0b:::benchmark_data_smithi188_141483_object40058:head# 2024-04-23T16:43:25.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cee6f198:::benchmark_data_smithi188_141483_object36837:head# 2024-04-23T16:43:25.293 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cee76e98:::benchmark_data_smithi188_141483_object9044:head# 2024-04-23T16:43:25.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cee925f9:::benchmark_data_smithi188_141483_object86562:head# 2024-04-23T16:43:25.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cee9695d:::benchmark_data_smithi188_141483_object25153:head# 2024-04-23T16:43:25.294 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cee98b80:::benchmark_data_smithi188_141483_object79146:head# 2024-04-23T16:43:25.295 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceeada16:::benchmark_data_smithi188_141483_object69985:head# 2024-04-23T16:43:25.295 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceeb9e79:::benchmark_data_smithi188_141483_object83885:head# 2024-04-23T16:43:25.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceec89f0:::benchmark_data_smithi188_141483_object74964:head# 2024-04-23T16:43:25.296 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceedfb6e:::benchmark_data_smithi188_141483_object76880:head# 2024-04-23T16:43:25.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceee5bae:::benchmark_data_smithi188_141483_object57552:head# 2024-04-23T16:43:25.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceef0967:::benchmark_data_smithi188_141483_object58272:head# 2024-04-23T16:43:25.297 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceef5272:::benchmark_data_smithi188_141483_object4228:head# 2024-04-23T16:43:25.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceef59f6:::benchmark_data_smithi188_141483_object87962:head# 2024-04-23T16:43:25.298 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cef0e475:::benchmark_data_smithi188_141483_object21289:head# 2024-04-23T16:43:25.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cef2b49c:::benchmark_data_smithi188_141483_object43607:head# 2024-04-23T16:43:25.299 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cef35a32:::benchmark_data_smithi188_141483_object88089:head# 2024-04-23T16:43:25.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cef42549:::benchmark_data_smithi188_141483_object50650:head# 2024-04-23T16:43:25.300 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cef46710:::benchmark_data_smithi188_141483_object46600:head# 2024-04-23T16:43:25.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cef5d4ab:::benchmark_data_smithi188_141483_object22551:head# 2024-04-23T16:43:25.301 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cef607a2:::benchmark_data_smithi188_141483_object38696:head# 2024-04-23T16:43:25.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cef615f3:::benchmark_data_smithi188_141483_object86273:head# 2024-04-23T16:43:25.302 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cef6c3e2:::benchmark_data_smithi188_141483_object87246:head# 2024-04-23T16:43:25.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cef6e2a2:::benchmark_data_smithi188_141483_object69409:head# 2024-04-23T16:43:25.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cef70a51:::benchmark_data_smithi188_141483_object77153:head# 2024-04-23T16:43:25.303 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cef9b042:::benchmark_data_smithi188_141483_object26899:head# 2024-04-23T16:43:25.304 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cefa3c6d:::benchmark_data_smithi188_141483_object58868:head# 2024-04-23T16:43:25.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cefa512b:::benchmark_data_smithi188_141483_object19429:head# 2024-04-23T16:43:25.305 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cefa8b77:::benchmark_data_smithi188_141483_object80796:head# 2024-04-23T16:43:25.307 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cefd8f11:::benchmark_data_smithi188_141483_object54241:head# 2024-04-23T16:43:25.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cefe0940:::benchmark_data_smithi188_141483_object30366:head# 2024-04-23T16:43:25.308 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceff16ff:::benchmark_data_smithi188_141483_object51793:head# 2024-04-23T16:43:25.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceff1be4:::benchmark_data_smithi188_141483_object9888:head# 2024-04-23T16:43:25.309 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:ceff92c9:::benchmark_data_smithi188_141483_object45848:head# 2024-04-23T16:43:25.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf00d345:::benchmark_data_smithi188_141483_object60506:head# 2024-04-23T16:43:25.310 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf010fee:::benchmark_data_smithi188_141483_object47211:head# 2024-04-23T16:43:25.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf03a1fd:::benchmark_data_smithi188_141483_object51768:head# 2024-04-23T16:43:25.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf041ad5:::benchmark_data_smithi188_141483_object87415:head# 2024-04-23T16:43:25.311 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf053d5b:::benchmark_data_smithi188_141483_object70131:head# 2024-04-23T16:43:25.312 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf056383:::benchmark_data_smithi188_141483_object26387:head# 2024-04-23T16:43:25.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf05c08b:::benchmark_data_smithi188_141483_object26279:head# 2024-04-23T16:43:25.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf0624f9:::benchmark_data_smithi188_141483_object32973:head# 2024-04-23T16:43:25.313 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf063c29:::benchmark_data_smithi188_141483_object6061:head# 2024-04-23T16:43:25.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf065e93:::benchmark_data_smithi188_141483_object29597:head# 2024-04-23T16:43:25.314 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf066498:::benchmark_data_smithi188_141483_object9561:head# 2024-04-23T16:43:25.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf07215c:::benchmark_data_smithi188_141483_object19365:head# 2024-04-23T16:43:25.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf07c4d7:::benchmark_data_smithi188_141483_object71063:head# 2024-04-23T16:43:25.315 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf083b4b:::benchmark_data_smithi188_141483_object49414:head# 2024-04-23T16:43:25.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf083f84:::benchmark_data_smithi188_141483_object70230:head# 2024-04-23T16:43:25.316 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf092412:::benchmark_data_smithi188_141483_object24729:head# 2024-04-23T16:43:25.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf09e104:::benchmark_data_smithi188_141483_object22541:head# 2024-04-23T16:43:25.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf09e91b:::benchmark_data_smithi188_141483_object35652:head# 2024-04-23T16:43:25.317 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf0a4824:::benchmark_data_smithi188_141483_object59553:head# 2024-04-23T16:43:25.318 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf0a8361:::benchmark_data_smithi188_141483_object11851:head# 2024-04-23T16:43:25.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf0b2678:::benchmark_data_smithi188_141483_object22024:head# 2024-04-23T16:43:25.319 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf0b697b:::benchmark_data_smithi188_141483_object13687:head# 2024-04-23T16:43:25.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf0bf7bb:::benchmark_data_smithi188_141483_object31169:head# 2024-04-23T16:43:25.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf0c37aa:::benchmark_data_smithi188_141483_object26253:head# 2024-04-23T16:43:25.320 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf0c7386:::benchmark_data_smithi188_141483_object58912:head# 2024-04-23T16:43:25.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf0c990a:::benchmark_data_smithi188_141483_object15206:head# 2024-04-23T16:43:25.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf0cbc7f:::benchmark_data_smithi188_141483_object47233:head# 2024-04-23T16:43:25.321 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf0d4907:::benchmark_data_smithi188_141483_object13481:head# 2024-04-23T16:43:25.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf0d9c2b:::benchmark_data_smithi188_141483_object20251:head# 2024-04-23T16:43:25.322 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf0e3f7b:::benchmark_data_smithi188_141483_object88951:head# 2024-04-23T16:43:25.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf0eceb5:::benchmark_data_smithi188_141483_object81909:head# 2024-04-23T16:43:25.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf0f2903:::benchmark_data_smithi188_141483_object34002:head# 2024-04-23T16:43:25.323 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf110e09:::benchmark_data_smithi188_141483_object6373:head# 2024-04-23T16:43:25.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf1193cf:::benchmark_data_smithi188_141483_object79354:head# 2024-04-23T16:43:25.324 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf11b90b:::benchmark_data_smithi188_141483_object9011:head# 2024-04-23T16:43:25.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf133ba5:::benchmark_data_smithi188_141483_object5015:head# 2024-04-23T16:43:25.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf153e8a:::benchmark_data_smithi188_141483_object36692:head# 2024-04-23T16:43:25.325 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf158d8e:::benchmark_data_smithi188_141483_object3621:head# 2024-04-23T16:43:25.326 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf16e201:::benchmark_data_smithi188_141483_object40240:head# 2024-04-23T16:43:25.326 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf174e80:::benchmark_data_smithi188_141483_object17437:head# 2024-04-23T16:43:25.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf1c549f:::benchmark_data_smithi188_141483_object63214:head# 2024-04-23T16:43:25.327 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf1cdd02:::benchmark_data_smithi188_141483_object14504:head# 2024-04-23T16:43:25.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf1d4548:::benchmark_data_smithi188_141483_object55038:head# 2024-04-23T16:43:25.330 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf1d570c:::benchmark_data_smithi188_141483_object80441:head# 2024-04-23T16:43:25.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf1e4be6:::benchmark_data_smithi188_141483_object69647:head# 2024-04-23T16:43:25.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf1ee73d:::benchmark_data_smithi188_141483_object14329:head# 2024-04-23T16:43:25.331 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf1f0a20:::benchmark_data_smithi188_141483_object21111:head# 2024-04-23T16:43:25.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf20522a:::benchmark_data_smithi188_141483_object13065:head# 2024-04-23T16:43:25.332 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf20f812:::benchmark_data_smithi188_141483_object42631:head# 2024-04-23T16:43:25.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf214f14:::benchmark_data_smithi188_141483_object16691:head# 2024-04-23T16:43:25.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf21ec4b:::benchmark_data_smithi188_141483_object33981:head# 2024-04-23T16:43:25.333 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf235ed3:::benchmark_data_smithi188_141483_object46005:head# 2024-04-23T16:43:25.334 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf23e7b2:::benchmark_data_smithi188_141483_object12011:head# 2024-04-23T16:43:25.334 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf250221:::benchmark_data_smithi188_141483_object21514:head# 2024-04-23T16:43:25.334 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf25774a:::benchmark_data_smithi188_141483_object5830:head# 2024-04-23T16:43:25.335 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf25a4fe:::benchmark_data_smithi188_141483_object1049:head# 2024-04-23T16:43:25.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf2687cf:::benchmark_data_smithi188_141483_object26300:head# 2024-04-23T16:43:25.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf2706a8:::benchmark_data_smithi188_141483_object76183:head# 2024-04-23T16:43:25.336 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf27ba45:::benchmark_data_smithi188_141483_object77106:head# 2024-04-23T16:43:25.337 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf2802db:::benchmark_data_smithi188_141483_object78073:head# 2024-04-23T16:43:25.337 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf288dee:::benchmark_data_smithi188_141483_object46274:head# 2024-04-23T16:43:25.338 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf28a44f:::benchmark_data_smithi188_141483_object45063:head# 2024-04-23T16:43:25.338 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf28d711:::benchmark_data_smithi188_141483_object72496:head# 2024-04-23T16:43:25.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf29795a:::benchmark_data_smithi188_141483_object24617:head# 2024-04-23T16:43:25.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf2b45eb:::benchmark_data_smithi188_141483_object80457:head# 2024-04-23T16:43:25.339 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf2bb96a:::benchmark_data_smithi188_141483_object27083:head# 2024-04-23T16:43:25.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf2c341d:::benchmark_data_smithi188_141483_object75879:head# 2024-04-23T16:43:25.340 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf2cd6a5:::benchmark_data_smithi188_141483_object47107:head# 2024-04-23T16:43:25.341 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf2d6e10:::benchmark_data_smithi188_141483_object47741:head# 2024-04-23T16:43:25.341 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf2d9925:::benchmark_data_smithi188_141483_object61115:head# 2024-04-23T16:43:25.342 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf3029ea:::benchmark_data_smithi188_141483_object70088:head# 2024-04-23T16:43:25.342 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf30be71:::benchmark_data_smithi188_141483_object78368:head# 2024-04-23T16:43:25.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf30f3f3:::benchmark_data_smithi188_141483_object44917:head# 2024-04-23T16:43:25.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf317574:::benchmark_data_smithi188_141483_object5100:head# 2024-04-23T16:43:25.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf31cd63:::benchmark_data_smithi188_141483_object85809:head# 2024-04-23T16:43:25.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf329131:::benchmark_data_smithi188_141483_object29100:head# 2024-04-23T16:43:25.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf32a895:::benchmark_data_smithi188_141483_object11692:head# 2024-04-23T16:43:25.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf33c93b:::benchmark_data_smithi188_141483_object72656:head# 2024-04-23T16:43:25.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf33dc1f:::benchmark_data_smithi188_141483_object86841:head# 2024-04-23T16:43:25.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf3415d3:::benchmark_data_smithi188_141483_object24836:head# 2024-04-23T16:43:25.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf341ea1:::benchmark_data_smithi188_141483_object28353:head# 2024-04-23T16:43:25.347 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf3480a6:::benchmark_data_smithi188_141483_object43662:head# 2024-04-23T16:43:25.348 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf348a11:::benchmark_data_smithi188_141483_object34776:head# 2024-04-23T16:43:25.348 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf34b255:::benchmark_data_smithi188_141483_object10430:head# 2024-04-23T16:43:25.348 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf34d4dd:::benchmark_data_smithi188_141483_object6649:head# 2024-04-23T16:43:25.348 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf355013:::benchmark_data_smithi188_141483_object37404:head# 2024-04-23T16:43:25.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf3696f9:::benchmark_data_smithi188_141483_object9804:head# 2024-04-23T16:43:25.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf3921e6:::benchmark_data_smithi188_141483_object27733:head# 2024-04-23T16:43:25.349 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf398e3d:::benchmark_data_smithi188_141483_object59966:head# 2024-04-23T16:43:25.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf3be3de:::benchmark_data_smithi188_141483_object65016:head# 2024-04-23T16:43:25.350 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf3bf962:::benchmark_data_smithi188_141483_object38613:head# 2024-04-23T16:43:25.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf3c619a:::benchmark_data_smithi188_141483_object25674:head# 2024-04-23T16:43:25.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf3e1b9e:::benchmark_data_smithi188_141483_object33655:head# 2024-04-23T16:43:25.351 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf3e78f9:::benchmark_data_smithi188_141483_object53764:head# 2024-04-23T16:43:25.352 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf3ea086:::benchmark_data_smithi188_141483_object60861:head# 2024-04-23T16:43:25.353 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf3eb51a:::benchmark_data_smithi188_141483_object70613:head# 2024-04-23T16:43:25.353 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf4168c8:::benchmark_data_smithi188_141483_object2020:head# 2024-04-23T16:43:25.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf42101b:::benchmark_data_smithi188_141483_object33294:head# 2024-04-23T16:43:25.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf42707f:::benchmark_data_smithi188_141483_object46461:head# 2024-04-23T16:43:25.354 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf42a50e:::benchmark_data_smithi188_141483_object5067:head# 2024-04-23T16:43:25.355 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf42bc32:::benchmark_data_smithi188_141483_object6258:head# 2024-04-23T16:43:25.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf42d4e4:::benchmark_data_smithi188_141483_object71671:head# 2024-04-23T16:43:25.356 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf4518b7:::benchmark_data_smithi188_141483_object87401:head# 2024-04-23T16:43:25.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf453114:::benchmark_data_smithi188_141483_object39465:head# 2024-04-23T16:43:25.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf45cf24:::benchmark_data_smithi188_141483_object79397:head# 2024-04-23T16:43:25.357 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf4657c8:::benchmark_data_smithi188_141483_object88473:head# 2024-04-23T16:43:25.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf472dd1:::benchmark_data_smithi188_141483_object52862:head# 2024-04-23T16:43:25.358 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf47b0d5:::benchmark_data_smithi188_141483_object81198:head# 2024-04-23T16:43:25.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf47b4aa:::benchmark_data_smithi188_141483_object23637:head# 2024-04-23T16:43:25.359 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf4899e6:::benchmark_data_smithi188_141483_object75172:head# 2024-04-23T16:43:25.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf49a805:::benchmark_data_smithi188_141483_object9374:head# 2024-04-23T16:43:25.360 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf4bbeb3:::benchmark_data_smithi188_141483_object3035:head# 2024-04-23T16:43:25.361 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf4ca282:::benchmark_data_smithi188_141483_object53517:head# 2024-04-23T16:43:25.361 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf4e8804:::benchmark_data_smithi188_141483_object45956:head# 2024-04-23T16:43:25.362 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf4eb321:::benchmark_data_smithi188_141483_object68308:head# 2024-04-23T16:43:25.362 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf50b699:::benchmark_data_smithi188_141483_object41531:head# 2024-04-23T16:43:25.363 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf50bf81:::benchmark_data_smithi188_141483_object3752:head# 2024-04-23T16:43:25.363 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf51d19f:::benchmark_data_smithi188_141483_object86052:head# 2024-04-23T16:43:25.363 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf531ab8:::benchmark_data_smithi188_141483_object12472:head# 2024-04-23T16:43:25.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf53262a:::benchmark_data_smithi188_141483_object31663:head# 2024-04-23T16:43:25.364 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf53370e:::benchmark_data_smithi188_141483_object55818:head# 2024-04-23T16:43:25.365 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf55ad03:::benchmark_data_smithi188_141483_object30893:head# 2024-04-23T16:43:25.367 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf56fe7b:::benchmark_data_smithi188_141483_object19017:head# 2024-04-23T16:43:25.368 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf573552:::benchmark_data_smithi188_141483_object11884:head# 2024-04-23T16:43:25.368 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf58664b:::benchmark_data_smithi188_141483_object83296:head# 2024-04-23T16:43:25.368 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf58c71d:::benchmark_data_smithi188_141483_object5486:head# 2024-04-23T16:43:25.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf5ae2ce:::benchmark_data_smithi188_141483_object16323:head# 2024-04-23T16:43:25.369 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf5b0491:::benchmark_data_smithi188_141483_object31637:head# 2024-04-23T16:43:25.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf5c275a:::benchmark_data_smithi188_141483_object20574:head# 2024-04-23T16:43:25.370 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf5cdd1a:::benchmark_data_smithi188_141483_object62048:head# 2024-04-23T16:43:25.371 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf5cf6fd:::benchmark_data_smithi188_141483_object34401:head# 2024-04-23T16:43:25.371 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf5ec86e:::benchmark_data_smithi188_141483_object56078:head# 2024-04-23T16:43:25.372 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf5f5100:::benchmark_data_smithi188_141483_object74027:head# 2024-04-23T16:43:25.372 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf609652:::benchmark_data_smithi188_141483_object63723:head# 2024-04-23T16:43:25.373 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf638401:::benchmark_data_smithi188_141483_object5148:head# 2024-04-23T16:43:25.373 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf638eba:::benchmark_data_smithi188_141483_object73947:head# 2024-04-23T16:43:25.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf6555fc:::benchmark_data_smithi188_141483_object28913:head# 2024-04-23T16:43:25.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf65a922:::benchmark_data_smithi188_141483_object31304:head# 2024-04-23T16:43:25.374 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf66527e:::benchmark_data_smithi188_141483_object45547:head# 2024-04-23T16:43:25.375 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf668a71:::benchmark_data_smithi188_141483_object37432:head# 2024-04-23T16:43:25.375 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf694e4c:::benchmark_data_smithi188_141483_object80073:head# 2024-04-23T16:43:25.375 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf69b5af:::benchmark_data_smithi188_141483_object7002:head# 2024-04-23T16:43:25.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf6bb165:::benchmark_data_smithi188_141483_object75266:head# 2024-04-23T16:43:25.376 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf6bd706:::benchmark_data_smithi188_141483_object59959:head# 2024-04-23T16:43:25.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf6d1fec:::benchmark_data_smithi188_141483_object46364:head# 2024-04-23T16:43:25.377 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf6d4ab7:::benchmark_data_smithi188_141483_object14959:head# 2024-04-23T16:43:25.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf6dd896:::benchmark_data_smithi188_141483_object82216:head# 2024-04-23T16:43:25.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf6f4713:::benchmark_data_smithi188_141483_object63304:head# 2024-04-23T16:43:25.378 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf6f54db:::benchmark_data_smithi188_141483_object20766:head# 2024-04-23T16:43:25.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf6f7a89:::benchmark_data_smithi188_141483_object24371:head# 2024-04-23T16:43:25.379 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf6fa71f:::benchmark_data_smithi188_141483_object71754:head# 2024-04-23T16:43:25.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf708f8b:::benchmark_data_smithi188_141483_object1294:head# 2024-04-23T16:43:25.380 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf710a03:::benchmark_data_smithi188_141483_object66385:head# 2024-04-23T16:43:25.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf712a54:::benchmark_data_smithi188_141483_object32337:head# 2024-04-23T16:43:25.381 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf723278:::benchmark_data_smithi188_141483_object33201:head# 2024-04-23T16:43:25.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf729634:::benchmark_data_smithi188_141483_object31874:head# 2024-04-23T16:43:25.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf7468ab:::benchmark_data_smithi188_141483_object89339:head# 2024-04-23T16:43:25.382 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf747e99:::benchmark_data_smithi188_141483_object11550:head# 2024-04-23T16:43:25.383 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf7485ec:::benchmark_data_smithi188_141483_object28376:head# 2024-04-23T16:43:25.385 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf751141:::benchmark_data_smithi188_141483_object80164:head# 2024-04-23T16:43:25.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf7533a5:::benchmark_data_smithi188_141483_object11481:head# 2024-04-23T16:43:25.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf75cf14:::benchmark_data_smithi188_141483_object14486:head# 2024-04-23T16:43:25.386 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf75d27f:::benchmark_data_smithi188_141483_object8564:head# 2024-04-23T16:43:25.387 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf7620f2:::benchmark_data_smithi188_141483_object35974:head# 2024-04-23T16:43:25.387 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf77ac75:::benchmark_data_smithi188_141483_object58563:head# 2024-04-23T16:43:25.389 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf79322e:::benchmark_data_smithi188_141483_object1025:head# 2024-04-23T16:43:25.389 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf796e13:::benchmark_data_smithi188_141483_object62912:head# 2024-04-23T16:43:25.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf7a7fae:::benchmark_data_smithi188_141483_object81500:head# 2024-04-23T16:43:25.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf7a8e64:::benchmark_data_smithi188_141483_object33139:head# 2024-04-23T16:43:25.390 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf7a989c:::benchmark_data_smithi188_141483_object53153:head# 2024-04-23T16:43:25.391 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf7beee3:::benchmark_data_smithi188_141483_object7770:head# 2024-04-23T16:43:25.391 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf7c07cf:::benchmark_data_smithi188_141483_object80459:head# 2024-04-23T16:43:25.392 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf7c55b6:::benchmark_data_smithi188_141483_object29364:head# 2024-04-23T16:43:25.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf7d5464:::benchmark_data_smithi188_141483_object71932:head# 2024-04-23T16:43:25.394 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf7e2a91:::benchmark_data_smithi188_141483_object62495:head# 2024-04-23T16:43:25.395 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf7f0f57:::benchmark_data_smithi188_141483_object68871:head# 2024-04-23T16:43:25.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf7f4166:::benchmark_data_smithi188_141483_object45822:head# 2024-04-23T16:43:25.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf80d57b:::benchmark_data_smithi188_141483_object51186:head# 2024-04-23T16:43:25.465 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf80fe18:::benchmark_data_smithi188_141483_object9581:head# 2024-04-23T16:43:25.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf8100ec:::benchmark_data_smithi188_141483_object56729:head# 2024-04-23T16:43:25.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf826ffa:::benchmark_data_smithi188_141483_object56295:head# 2024-04-23T16:43:25.466 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf827bcb:::benchmark_data_smithi188_141483_object22091:head# 2024-04-23T16:43:25.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf82a824:::benchmark_data_smithi188_141483_object48736:head# 2024-04-23T16:43:25.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf832544:::benchmark_data_smithi188_141483_object72353:head# 2024-04-23T16:43:25.467 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf83b832:::benchmark_data_smithi188_141483_object18344:head# 2024-04-23T16:43:25.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf8781da:::benchmark_data_smithi188_141483_object31437:head# 2024-04-23T16:43:25.468 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf8825ce:::benchmark_data_smithi188_141483_object77977:head# 2024-04-23T16:43:25.469 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf8bc134:::benchmark_data_smithi188_141483_object56872:head# 2024-04-23T16:43:25.469 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf8c1c7f:::benchmark_data_smithi188_141483_object66594:head# 2024-04-23T16:43:25.470 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf9005a7:::benchmark_data_smithi188_141483_object62690:head# 2024-04-23T16:43:25.470 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf91811d:::benchmark_data_smithi188_141483_object55456:head# 2024-04-23T16:43:25.473 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf91a02b:::benchmark_data_smithi188_141483_object36556:head# 2024-04-23T16:43:25.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf920ae5:::benchmark_data_smithi188_141483_object4953:head# 2024-04-23T16:43:25.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf929066:::benchmark_data_smithi188_141483_object49876:head# 2024-04-23T16:43:25.475 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf92b231:::benchmark_data_smithi188_141483_object85833:head# 2024-04-23T16:43:25.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf92bfac:::benchmark_data_smithi188_141483_object33100:head# 2024-04-23T16:43:25.476 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf930018:::benchmark_data_smithi188_141483_object80053:head# 2024-04-23T16:43:25.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf9366b8:::benchmark_data_smithi188_141483_object27166:head# 2024-04-23T16:43:25.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf93e543:::benchmark_data_smithi188_141483_object42403:head# 2024-04-23T16:43:25.477 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf9690dd:::benchmark_data_smithi188_141483_object12936:head# 2024-04-23T16:43:25.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf96cd4e:::benchmark_data_smithi188_141483_object1035:head# 2024-04-23T16:43:25.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf97477a:::benchmark_data_smithi188_141483_object46414:head# 2024-04-23T16:43:25.478 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf975545:::benchmark_data_smithi188_141483_object57371:head# 2024-04-23T16:43:25.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf98eabd:::benchmark_data_smithi188_141483_object29220:head# 2024-04-23T16:43:25.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf99bdb4:::benchmark_data_smithi188_141483_object84060:head# 2024-04-23T16:43:25.479 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf9a14ad:::benchmark_data_smithi188_141483_object86339:head# 2024-04-23T16:43:25.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf9a2166:::benchmark_data_smithi188_141483_object9293:head# 2024-04-23T16:43:25.480 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf9ad631:::benchmark_data_smithi188_141483_object86751:head# 2024-04-23T16:43:25.481 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf9d1ebd:::benchmark_data_smithi188_141483_object55328:head# 2024-04-23T16:43:25.481 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf9d8195:::benchmark_data_smithi188_141483_object40752:head# 2024-04-23T16:43:25.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf9daf4a:::benchmark_data_smithi188_141483_object50032:head# 2024-04-23T16:43:25.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf9de71e:::benchmark_data_smithi188_141483_object17057:head# 2024-04-23T16:43:25.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf9e03af:::benchmark_data_smithi188_141483_object81454:head# 2024-04-23T16:43:25.482 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf9eb69c:::benchmark_data_smithi188_141483_object46747:head# 2024-04-23T16:43:25.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf9ede8b:::benchmark_data_smithi188_141483_object42220:head# 2024-04-23T16:43:25.483 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf9f0df7:::benchmark_data_smithi188_141483_object37100:head# 2024-04-23T16:43:25.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf9f3454:::benchmark_data_smithi188_141483_object59241:head# 2024-04-23T16:43:25.484 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf9f9692:::benchmark_data_smithi188_141483_object30925:head# 2024-04-23T16:43:25.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf9f97ea:::benchmark_data_smithi188_141483_object67392:head# 2024-04-23T16:43:25.485 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cf9feed8:::benchmark_data_smithi188_141483_object47164:head# 2024-04-23T16:43:25.486 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfa00c30:::benchmark_data_smithi188_141483_object21164:head# 2024-04-23T16:43:25.486 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfa0b875:::benchmark_data_smithi188_141483_object20984:head# 2024-04-23T16:43:25.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfa11b9a:::benchmark_data_smithi188_141483_object29558:head# 2024-04-23T16:43:25.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfa1aae9:::benchmark_data_smithi188_141483_object23512:head# 2024-04-23T16:43:25.487 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfa1b3e5:::benchmark_data_smithi188_141483_object81788:head# 2024-04-23T16:43:25.488 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfa26f3f:::benchmark_data_smithi188_141483_object74538:head# 2024-04-23T16:43:25.488 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfa38553:::benchmark_data_smithi188_141483_object71560:head# 2024-04-23T16:43:25.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfa42084:::benchmark_data_smithi188_141483_object36107:head# 2024-04-23T16:43:25.489 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfa43adc:::benchmark_data_smithi188_141483_object27804:head# 2024-04-23T16:43:25.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfa4521e:::benchmark_data_smithi188_141483_object57989:head# 2024-04-23T16:43:25.490 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfa7d365:::benchmark_data_smithi188_141483_object76940:head# 2024-04-23T16:43:25.491 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfa9905c:::benchmark_data_smithi188_141483_object327:head# 2024-04-23T16:43:25.491 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfa9db6b:::benchmark_data_smithi188_141483_object2693:head# 2024-04-23T16:43:25.491 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfaaea07:::benchmark_data_smithi188_141483_object65966:head# 2024-04-23T16:43:25.492 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfab3b4d:::benchmark_data_smithi188_141483_object72833:head# 2024-04-23T16:43:25.493 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfac7e8e:::benchmark_data_smithi188_141483_object67569:head# 2024-04-23T16:43:25.493 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfacb396:::benchmark_data_smithi188_141483_object83560:head# 2024-04-23T16:43:25.493 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfad085e:::benchmark_data_smithi188_141483_object5913:head# 2024-04-23T16:43:25.494 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfad1580:::benchmark_data_smithi188_141483_object40730:head# 2024-04-23T16:43:25.494 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfad55f9:::benchmark_data_smithi188_141483_object87134:head# 2024-04-23T16:43:25.494 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfaec77f:::benchmark_data_smithi188_141483_object51338:head# 2024-04-23T16:43:25.495 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfb108e6:::benchmark_data_smithi188_141483_object73813:head# 2024-04-23T16:43:25.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfb25db5:::benchmark_data_smithi188_141483_object40426:head# 2024-04-23T16:43:25.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfb2f362:::benchmark_data_smithi188_141483_object79994:head# 2024-04-23T16:43:25.496 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfb4a434:::benchmark_data_smithi188_141483_object13210:head# 2024-04-23T16:43:25.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfb4fafd:::benchmark_data_smithi188_141483_object27748:head# 2024-04-23T16:43:25.497 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfb87e0c:::benchmark_data_smithi188_141483_object87052:head# 2024-04-23T16:43:25.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfb90fdd:::benchmark_data_smithi188_141483_object75474:head# 2024-04-23T16:43:25.498 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfb95f91:::benchmark_data_smithi188_141483_object36146:head# 2024-04-23T16:43:25.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfb98692:::benchmark_data_smithi188_141483_object79901:head# 2024-04-23T16:43:25.499 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfba829e:::benchmark_data_smithi188_141483_object30445:head# 2024-04-23T16:43:25.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfba9193:::benchmark_data_smithi188_141483_object70652:head# 2024-04-23T16:43:25.500 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfbc8b9a:::benchmark_data_smithi188_141483_object63466:head# 2024-04-23T16:43:25.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfbcb5f1:::benchmark_data_smithi188_141483_object63399:head# 2024-04-23T16:43:25.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfbce2d9:::benchmark_data_smithi188_141483_object53300:head# 2024-04-23T16:43:25.501 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfbd0609:::benchmark_data_smithi188_141483_object1555:head# 2024-04-23T16:43:25.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfbd49ee:::benchmark_data_smithi188_141483_object1061:head# 2024-04-23T16:43:25.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfbd83f9:::benchmark_data_smithi188_141483_object81870:head# 2024-04-23T16:43:25.502 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfbef77c:::benchmark_data_smithi188_141483_object69551:head# 2024-04-23T16:43:25.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfbf3cbb:::benchmark_data_smithi188_141483_object19082:head# 2024-04-23T16:43:25.503 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfbf4ea6:::benchmark_data_smithi188_141483_object6159:head# 2024-04-23T16:43:25.504 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfbfa053:::benchmark_data_smithi188_141483_object7224:head# 2024-04-23T16:43:25.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfbfa6b0:::benchmark_data_smithi188_141483_object54907:head# 2024-04-23T16:43:25.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfbfec0f:::benchmark_data_smithi188_141483_object53305:head# 2024-04-23T16:43:25.505 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfc08971:::benchmark_data_smithi188_141483_object22601:head# 2024-04-23T16:43:25.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfc0fb19:::benchmark_data_smithi188_141483_object58377:head# 2024-04-23T16:43:25.506 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfc2633f:::benchmark_data_smithi188_141483_object84573:head# 2024-04-23T16:43:25.507 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfc2a399:::benchmark_data_smithi188_141483_object49080:head# 2024-04-23T16:43:25.507 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfc2c8f4:::benchmark_data_smithi188_141483_object42939:head# 2024-04-23T16:43:25.507 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfc4fbac:::benchmark_data_smithi188_141483_object55304:head# 2024-04-23T16:43:25.508 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfc55eea:::benchmark_data_smithi188_141483_object64238:head# 2024-04-23T16:43:25.508 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfc6bee9:::benchmark_data_smithi188_141483_object75821:head# 2024-04-23T16:43:25.509 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfc6d78a:::benchmark_data_smithi188_141483_object63443:head# 2024-04-23T16:43:25.509 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfc739b6:::benchmark_data_smithi188_141483_object799:head# 2024-04-23T16:43:25.509 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfc74abd:::benchmark_data_smithi188_141483_object41915:head# 2024-04-23T16:43:25.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfc8f499:::benchmark_data_smithi188_141483_object82821:head# 2024-04-23T16:43:25.510 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfca15ce:::benchmark_data_smithi188_141483_object72911:head# 2024-04-23T16:43:25.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfca41fa:::benchmark_data_smithi188_141483_object67695:head# 2024-04-23T16:43:25.511 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfcbb4d2:::benchmark_data_smithi188_141483_object2027:head# 2024-04-23T16:43:25.512 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfcc73b3:::benchmark_data_smithi188_141483_object78411:head# 2024-04-23T16:43:25.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfccb77e:::benchmark_data_smithi188_141483_object74793:head# 2024-04-23T16:43:25.513 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfcd4ea4:::benchmark_data_smithi188_141483_object16801:head# 2024-04-23T16:43:25.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfce626d:::benchmark_data_smithi188_141483_object88493:head# 2024-04-23T16:43:25.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfcf0233:::benchmark_data_smithi188_141483_object46820:head# 2024-04-23T16:43:25.514 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfcf3549:::benchmark_data_smithi188_141483_object31119:head# 2024-04-23T16:43:25.515 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfd033c9:::benchmark_data_smithi188_141483_object26185:head# 2024-04-23T16:43:25.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfd04c34:::benchmark_data_smithi188_141483_object67390:head# 2024-04-23T16:43:25.518 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfd0cda8:::benchmark_data_smithi188_141483_object16266:head# 2024-04-23T16:43:25.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfd15720:::benchmark_data_smithi188_141483_object71714:head# 2024-04-23T16:43:25.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfd23922:::benchmark_data_smithi188_141483_object75763:head# 2024-04-23T16:43:25.519 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfd2865a:::benchmark_data_smithi188_141483_object43357:head# 2024-04-23T16:43:25.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfd37a32:::benchmark_data_smithi188_141483_object34431:head# 2024-04-23T16:43:25.520 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfd4bf01:::benchmark_data_smithi188_141483_object41903:head# 2024-04-23T16:43:25.521 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfd4ca64:::benchmark_data_smithi188_141483_object52050:head# 2024-04-23T16:43:25.521 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfd4e056:::benchmark_data_smithi188_141483_object57759:head# 2024-04-23T16:43:25.521 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfd4f19f:::benchmark_data_smithi188_141483_object90009:head# 2024-04-23T16:43:25.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfd5efbf:::benchmark_data_smithi188_141483_object79416:head# 2024-04-23T16:43:25.522 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfd64e8d:::benchmark_data_smithi188_141483_object43636:head# 2024-04-23T16:43:25.523 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfd6988d:::benchmark_data_smithi188_141483_object73165:head# 2024-04-23T16:43:25.523 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfd69da1:::benchmark_data_smithi188_141483_object83170:head# 2024-04-23T16:43:25.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfd76f94:::benchmark_data_smithi188_141483_object64683:head# 2024-04-23T16:43:25.524 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfd897b7:::benchmark_data_smithi188_141483_object23872:head# 2024-04-23T16:43:25.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfd92fac:::benchmark_data_smithi188_141483_object61006:head# 2024-04-23T16:43:25.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfda1a54:::benchmark_data_smithi188_141483_object5851:head# 2024-04-23T16:43:25.525 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfda3fa1:::benchmark_data_smithi188_141483_object29323:head# 2024-04-23T16:43:25.526 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfdb07e6:::benchmark_data_smithi188_141483_object48220:head# 2024-04-23T16:43:25.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfdc3559:::benchmark_data_smithi188_141483_object69173:head# 2024-04-23T16:43:25.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfdcfce3:::benchmark_data_smithi188_141483_object18868:head# 2024-04-23T16:43:25.527 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfde57b4:::benchmark_data_smithi188_141483_object80267:head# 2024-04-23T16:43:25.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfe05239:::benchmark_data_smithi188_141483_object64688:head# 2024-04-23T16:43:25.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfe0d6a3:::benchmark_data_smithi188_141483_object5520:head# 2024-04-23T16:43:25.528 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfe10f94:::benchmark_data_smithi188_141483_object78043:head# 2024-04-23T16:43:25.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfe13ad0:::benchmark_data_smithi188_141483_object45725:head# 2024-04-23T16:43:25.529 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfe17bbe:::benchmark_data_smithi188_141483_object14683:head# 2024-04-23T16:43:25.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfe3d455:::benchmark_data_smithi188_141483_object42325:head# 2024-04-23T16:43:25.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfe41cae:::benchmark_data_smithi188_141483_object25166:head# 2024-04-23T16:43:25.530 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfe58ce0:::benchmark_data_smithi188_141483_object82386:head# 2024-04-23T16:43:25.531 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfe610f1:::benchmark_data_smithi188_141483_object62128:head# 2024-04-23T16:43:25.531 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfe76701:::benchmark_data_smithi188_141483_object22424:head# 2024-04-23T16:43:25.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfe7a0a9:::benchmark_data_smithi188_141483_object9613:head# 2024-04-23T16:43:25.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfe937db:::benchmark_data_smithi188_141483_object70061:head# 2024-04-23T16:43:25.532 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfea1c9d:::benchmark_data_smithi188_141483_object32364:head# 2024-04-23T16:43:25.533 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfeb43d5:::benchmark_data_smithi188_141483_object34391:head# 2024-04-23T16:43:25.533 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfeb48c2:::benchmark_data_smithi188_141483_object48538:head# 2024-04-23T16:43:25.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfeb5451:::benchmark_data_smithi188_141483_object89416:head# 2024-04-23T16:43:25.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfed84d5:::benchmark_data_smithi188_141483_object80996:head# 2024-04-23T16:43:25.534 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfee23b1:::benchmark_data_smithi188_141483_object74827:head# 2024-04-23T16:43:25.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfee9a17:::benchmark_data_smithi188_141483_object86560:head# 2024-04-23T16:43:25.535 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfeec085:::benchmark_data_smithi188_141483_object78719:head# 2024-04-23T16:43:25.536 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfefe2b6:::benchmark_data_smithi188_141483_object37301:head# 2024-04-23T16:43:25.536 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cfeff2ae:::benchmark_data_smithi188_141483_object88249:head# 2024-04-23T16:43:25.536 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cff0baf4:::benchmark_data_smithi188_141483_object6151:head# 2024-04-23T16:43:25.537 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cff0d3bf:::benchmark_data_smithi188_141483_object9458:head# 2024-04-23T16:43:25.537 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 74 16 105621 105605 89.1813 49.625 0.01457 0.0112098 2024-04-23T16:43:25.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cff0dd8e:::benchmark_data_smithi188_141483_object33118:head# 2024-04-23T16:43:25.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cff0ff27:::benchmark_data_smithi188_141483_object40158:head# 2024-04-23T16:43:25.538 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cff1aff9:::benchmark_data_smithi188_141483_object21092:head# 2024-04-23T16:43:25.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cff32909:::benchmark_data_smithi188_141483_object31044:head# 2024-04-23T16:43:25.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cff35204:::benchmark_data_smithi188_141483_object69818:head# 2024-04-23T16:43:25.539 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cff49d40:::benchmark_data_smithi188_141483_object29148:head# 2024-04-23T16:43:25.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cff4ea0e:::benchmark_data_smithi188_141483_object69041:head# 2024-04-23T16:43:25.540 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cff6c6c0:::benchmark_data_smithi188_141483_object15443:head# 2024-04-23T16:43:25.541 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cff6f90b:::benchmark_data_smithi188_141483_object1019:head# 2024-04-23T16:43:25.541 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cff842f5:::benchmark_data_smithi188_141483_object61773:head# 2024-04-23T16:43:25.541 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cff94b93:::benchmark_data_smithi188_141483_object38496:head# 2024-04-23T16:43:25.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cff97b75:::benchmark_data_smithi188_141483_object86753:head# 2024-04-23T16:43:25.542 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cffc2673:::benchmark_data_smithi188_141483_object4400:head# 2024-04-23T16:43:25.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cffc3dd8:::benchmark_data_smithi188_141483_object42195:head# 2024-04-23T16:43:25.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cffc4393:::benchmark_data_smithi188_141483_object87708:head# 2024-04-23T16:43:25.543 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cffc9c76:::benchmark_data_smithi188_141483_object21812:head# 2024-04-23T16:43:25.544 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cffd7c8d:::benchmark_data_smithi188_141483_object36827:head# 2024-04-23T16:43:25.544 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cffd8180:::benchmark_data_smithi188_141483_object29041:head# 2024-04-23T16:43:25.544 INFO:teuthology.orchestra.run.smithi103.stderr:Read #7:cffed9b0:::benchmark_data_smithi188_141483_object35463:head# 2024-04-23T16:43:25.547 INFO:teuthology.orchestra.run.smithi103.stderr:Export successful 2024-04-23T16:43:26.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:25 smithi188 ceph-mon[44101]: pgmap v1582: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 22 active+clean; 6.4 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 60 MiB/s wr, 967 op/s; 21045/208100 objects degraded (10.113%); 8.1 MiB/s, 129 objects/s recovering 2024-04-23T16:43:26.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:25 smithi188 ceph-mon[44101]: osdmap e568: 12 total, 11 up, 9 in 2024-04-23T16:43:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:25 smithi188 ceph-mon[44101]: 7.6 scrub starts 2024-04-23T16:43:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:25 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:43:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:25 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:43:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:25 smithi188 ceph-mon[44101]: 7.4 scrub ok 2024-04-23T16:43:26.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:25 smithi103 ceph-mon[39629]: pgmap v1582: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 22 active+clean; 6.4 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 60 MiB/s wr, 967 op/s; 21045/208100 objects degraded (10.113%); 8.1 MiB/s, 129 objects/s recovering 2024-04-23T16:43:26.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:25 smithi103 ceph-mon[39629]: osdmap e568: 12 total, 11 up, 9 in 2024-04-23T16:43:26.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:25 smithi103 ceph-mon[39629]: 7.6 scrub starts 2024-04-23T16:43:26.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:25 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:43:26.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:25 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:43:26.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:25 smithi103 ceph-mon[39629]: 7.4 scrub ok 2024-04-23T16:43:26.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:25 smithi148 ceph-mon[44520]: pgmap v1582: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 22 active+clean; 6.4 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 60 MiB/s wr, 967 op/s; 21045/208100 objects degraded (10.113%); 8.1 MiB/s, 129 objects/s recovering 2024-04-23T16:43:26.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:25 smithi148 ceph-mon[44520]: osdmap e568: 12 total, 11 up, 9 in 2024-04-23T16:43:26.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:25 smithi148 ceph-mon[44520]: 7.6 scrub starts 2024-04-23T16:43:26.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:25 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:43:26.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:25 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:43:26.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:25 smithi148 ceph-mon[44520]: 7.4 scrub ok 2024-04-23T16:43:26.537 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 75 16 106413 106397 88.6522 49.5 0.00438429 0.0112764 2024-04-23T16:43:27.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:26 smithi188 ceph-mon[44101]: 7.a scrub starts 2024-04-23T16:43:27.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:26 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:43:27.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:26 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:43:27.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:26 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:43:27.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:26 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:43:27.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:26 smithi188 ceph-mon[44101]: 7.9 scrub starts 2024-04-23T16:43:27.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:26 smithi188 ceph-mon[44101]: osdmap e569: 12 total, 11 up, 9 in 2024-04-23T16:43:27.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:26 smithi188 ceph-mon[44101]: 7.6 scrub ok 2024-04-23T16:43:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:26 smithi103 ceph-mon[39629]: 7.a scrub starts 2024-04-23T16:43:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:26 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:43:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:26 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:43:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:26 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:43:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:26 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:43:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:26 smithi103 ceph-mon[39629]: 7.9 scrub starts 2024-04-23T16:43:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:26 smithi103 ceph-mon[39629]: osdmap e569: 12 total, 11 up, 9 in 2024-04-23T16:43:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:26 smithi103 ceph-mon[39629]: 7.6 scrub ok 2024-04-23T16:43:27.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:26 smithi148 ceph-mon[44520]: 7.a scrub starts 2024-04-23T16:43:27.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:26 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:43:27.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:26 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:43:27.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:26 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:43:27.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:26 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:43:27.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:26 smithi148 ceph-mon[44520]: 7.9 scrub starts 2024-04-23T16:43:27.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:26 smithi148 ceph-mon[44520]: osdmap e569: 12 total, 11 up, 9 in 2024-04-23T16:43:27.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:26 smithi148 ceph-mon[44520]: 7.6 scrub ok 2024-04-23T16:43:27.537 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 76 16 107145 107129 88.0877 45.75 0.00660407 0.0113469 2024-04-23T16:43:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:27 smithi188 ceph-mon[44101]: 7.a scrub ok 2024-04-23T16:43:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:27 smithi188 ceph-mon[44101]: pgmap v1585: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 22 active+clean; 6.4 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 49 MiB/s wr, 782 op/s; 21045/209662 objects degraded (10.038%); 5.5 MiB/s, 88 objects/s recovering 2024-04-23T16:43:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:27 smithi188 ceph-mon[44101]: 7.9 scrub ok 2024-04-23T16:43:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:27 smithi188 ceph-mon[44101]: 7.f scrub starts 2024-04-23T16:43:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:27 smithi188 ceph-mon[44101]: 1.8 scrub starts 2024-04-23T16:43:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:27 smithi188 ceph-mon[44101]: 1.8 scrub ok 2024-04-23T16:43:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:27 smithi103 ceph-mon[39629]: 7.a scrub ok 2024-04-23T16:43:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:27 smithi103 ceph-mon[39629]: pgmap v1585: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 22 active+clean; 6.4 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 49 MiB/s wr, 782 op/s; 21045/209662 objects degraded (10.038%); 5.5 MiB/s, 88 objects/s recovering 2024-04-23T16:43:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:27 smithi103 ceph-mon[39629]: 7.9 scrub ok 2024-04-23T16:43:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:27 smithi103 ceph-mon[39629]: 7.f scrub starts 2024-04-23T16:43:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:27 smithi103 ceph-mon[39629]: 1.8 scrub starts 2024-04-23T16:43:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:27 smithi103 ceph-mon[39629]: 1.8 scrub ok 2024-04-23T16:43:28.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:27 smithi148 ceph-mon[44520]: 7.a scrub ok 2024-04-23T16:43:28.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:27 smithi148 ceph-mon[44520]: pgmap v1585: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 22 active+clean; 6.4 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 49 MiB/s wr, 782 op/s; 21045/209662 objects degraded (10.038%); 5.5 MiB/s, 88 objects/s recovering 2024-04-23T16:43:28.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:27 smithi148 ceph-mon[44520]: 7.9 scrub ok 2024-04-23T16:43:28.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:27 smithi148 ceph-mon[44520]: 7.f scrub starts 2024-04-23T16:43:28.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:27 smithi148 ceph-mon[44520]: 1.8 scrub starts 2024-04-23T16:43:28.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:27 smithi148 ceph-mon[44520]: 1.8 scrub ok 2024-04-23T16:43:28.538 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 77 16 108007 107991 87.6433 53.875 0.00466643 0.0114064 2024-04-23T16:43:29.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:28 smithi148 ceph-mon[44520]: 7.e scrub starts 2024-04-23T16:43:29.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:28 smithi148 ceph-mon[44520]: 7.f scrub ok 2024-04-23T16:43:29.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:28 smithi148 ceph-mon[44520]: 7.b scrub starts 2024-04-23T16:43:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:28 smithi188 ceph-mon[44101]: 7.e scrub starts 2024-04-23T16:43:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:28 smithi188 ceph-mon[44101]: 7.f scrub ok 2024-04-23T16:43:29.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:28 smithi188 ceph-mon[44101]: 7.b scrub starts 2024-04-23T16:43:29.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:28 smithi103 ceph-mon[39629]: 7.e scrub starts 2024-04-23T16:43:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:28 smithi103 ceph-mon[39629]: 7.f scrub ok 2024-04-23T16:43:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:28 smithi103 ceph-mon[39629]: 7.b scrub starts 2024-04-23T16:43:29.538 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 78 15 108823 108808 87.1743 51.0625 0.0140871 0.011469 2024-04-23T16:43:29.830 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:43:29.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:29.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:43:29.934 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:43:29.964 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:30.002 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:43:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:30.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:43:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:29 smithi103 ceph-mon[39629]: 7.e scrub ok 2024-04-23T16:43:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:29 smithi103 ceph-mon[39629]: pgmap v1586: 27 pgs: 2 active+clean+scrubbing, 4 active+recovering+degraded, 21 active+clean; 6.4 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 43 MiB/s wr, 693 op/s; 21045/210160 objects degraded (10.014%); 5.5 MiB/s, 88 objects/s recovering 2024-04-23T16:43:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:29 smithi103 ceph-mon[39629]: 7.b scrub ok 2024-04-23T16:43:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:29 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 21045/210160 objects degraded (10.014%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:29 smithi103 ceph-mon[39629]: 7.0 scrub starts 2024-04-23T16:43:30.106 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:43:30.185 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:29 smithi148 ceph-mon[44520]: 7.e scrub ok 2024-04-23T16:43:30.185 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:29 smithi148 ceph-mon[44520]: pgmap v1586: 27 pgs: 2 active+clean+scrubbing, 4 active+recovering+degraded, 21 active+clean; 6.4 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 43 MiB/s wr, 693 op/s; 21045/210160 objects degraded (10.014%); 5.5 MiB/s, 88 objects/s recovering 2024-04-23T16:43:30.185 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:29 smithi148 ceph-mon[44520]: 7.b scrub ok 2024-04-23T16:43:30.185 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:29 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 21045/210160 objects degraded (10.014%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:30.186 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:29 smithi148 ceph-mon[44520]: 7.0 scrub starts 2024-04-23T16:43:30.192 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:30.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:29 smithi188 ceph-mon[44101]: 7.e scrub ok 2024-04-23T16:43:30.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:29 smithi188 ceph-mon[44101]: pgmap v1586: 27 pgs: 2 active+clean+scrubbing, 4 active+recovering+degraded, 21 active+clean; 6.4 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 43 MiB/s wr, 693 op/s; 21045/210160 objects degraded (10.014%); 5.5 MiB/s, 88 objects/s recovering 2024-04-23T16:43:30.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:29 smithi188 ceph-mon[44101]: 7.b scrub ok 2024-04-23T16:43:30.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:29 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 21045/210160 objects degraded (10.014%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:30.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:29 smithi188 ceph-mon[44101]: 7.0 scrub starts 2024-04-23T16:43:30.431 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:43:30.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:30.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:43:30.538 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 79 16 109571 109555 86.6618 46.6875 0.0141559 0.0115358 2024-04-23T16:43:30.549 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:43:30.655 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:30.730 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:43:30.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:30.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:43:30.836 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:43:30.878 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:30.912 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:43:30.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:30.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:43:31.016 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:43:31.043 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:30 smithi103 ceph-mon[39629]: 7.0 scrub ok 2024-04-23T16:43:31.095 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:30 smithi148 ceph-mon[44520]: 7.0 scrub ok 2024-04-23T16:43:31.099 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:31.147 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:43:31.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:31.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:43:31.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:30 smithi188 ceph-mon[44101]: 7.0 scrub ok 2024-04-23T16:43:31.252 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:43:31.282 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:31.384 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:43:31.384 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:43:31.538 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:43:31.538157 min lat: 0.000969795 max lat: 25.4619 avg lat: 0.0116015 2024-04-23T16:43:31.538 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:43:31.538 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 80 16 110339 110323 86.1785 48 0.0106333 0.0116015 2024-04-23T16:43:31.815 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.3 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --force --op remove --pgid 7.3 2024-04-23T16:43:31.982 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:43:31.982 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:43:31.982 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:43:31.982 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:43:31.982 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:43:31.982 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:43:31.983 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:43:31.983 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:43:31.983 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:43:31.983 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:43:31.983 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:43:31.983 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:43:31.983 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:43:31.984 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:43:31.984 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:43:31.984 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:43:31.984 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:43:31.984 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:43:31.984 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:43:31.984 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:43:31.984 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:43:31.984 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:43:31.984 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:43:32.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:31 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:43:32.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:31 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:43:32.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:31 smithi148 ceph-mon[44520]: pgmap v1587: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 22 active+clean; 6.6 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 58 MiB/s wr, 927 op/s; 20516/216606 objects degraded (9.472%); 8.3 MiB/s, 132 objects/s recovering 2024-04-23T16:43:32.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:31 smithi148 ceph-mon[44520]: osdmap e570: 12 total, 11 up, 9 in 2024-04-23T16:43:32.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:31 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4214515627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:43:32.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:31 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:43:32.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:31 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:43:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:31 smithi188 ceph-mon[44101]: pgmap v1587: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 22 active+clean; 6.6 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 58 MiB/s wr, 927 op/s; 20516/216606 objects degraded (9.472%); 8.3 MiB/s, 132 objects/s recovering 2024-04-23T16:43:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:31 smithi188 ceph-mon[44101]: osdmap e570: 12 total, 11 up, 9 in 2024-04-23T16:43:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:31 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4214515627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:43:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.229 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-23T16:43:32.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:31 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:43:32.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:31 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:43:32.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:31 smithi103 ceph-mon[39629]: pgmap v1587: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 22 active+clean; 6.6 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 58 MiB/s wr, 927 op/s; 20516/216606 objects degraded (9.472%); 8.3 MiB/s, 132 objects/s recovering 2024-04-23T16:43:32.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:31 smithi103 ceph-mon[39629]: osdmap e570: 12 total, 11 up, 9 in 2024-04-23T16:43:32.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:31 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4214515627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:43:32.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.320 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.320 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.320 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.320 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.320 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.320 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:32.538 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 81 16 111209 111193 85.7858 54.375 0.0303865 0.0116529 2024-04-23T16:43:32.720 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.3/config 2024-04-23T16:43:32.727 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:43:32.727 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:43:32.732 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:43:32.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:32.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:43:32.756 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:43:32.769 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:43:32.837 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:43:32.873 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:32.897 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:43:32.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:32.900 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:43:33.001 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:43:33.041 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:33.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:32 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:33.244 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:33.244 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:33.244 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:33.245 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:32 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:33.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:33.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:33.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:33.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:32 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:33.538 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 82 16 112006 111990 85.3471 49.8125 0.00389744 0.0117135 2024-04-23T16:43:33.592 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:43:34.167 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:33 smithi103 ceph-mon[39629]: pgmap v1589: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.7 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 39 MiB/s wr, 630 op/s; 20516/218118 objects degraded (9.406%); 4.2 MiB/s, 66 objects/s recovering 2024-04-23T16:43:34.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:33 smithi188 ceph-mon[44101]: pgmap v1589: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.7 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 39 MiB/s wr, 630 op/s; 20516/218118 objects degraded (9.406%); 4.2 MiB/s, 66 objects/s recovering 2024-04-23T16:43:34.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:33 smithi148 ceph-mon[44520]: pgmap v1589: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.7 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 39 MiB/s wr, 630 op/s; 20516/218118 objects degraded (9.406%); 4.2 MiB/s, 66 objects/s recovering 2024-04-23T16:43:34.538 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 83 16 112869 112853 84.9686 53.9375 0.00569496 0.0117656 2024-04-23T16:43:34.709 INFO:teuthology.orchestra.run.smithi103.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:43:34.728 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:43:35.135 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:43:35.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:35.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:43:35.242 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:43:35.332 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:34 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 20516/218118 objects degraded (9.406%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:35.342 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:35.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:34 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 20516/218118 objects degraded (9.406%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:35.383 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:43:35.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:35.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:43:35.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:34 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 20516/218118 objects degraded (9.406%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:35.488 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:43:35.538 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 84 16 113749 113733 84.6118 55 0.00874583 0.0118169 2024-04-23T16:43:35.562 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:35.601 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:43:35.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:35.607 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:43:35.669 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:43:35.708 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:43:35.805 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:35.947 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:43:35.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:35.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:43:36.052 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:43:36.110 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:35 smithi148 ceph-mon[44520]: pgmap v1590: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.8 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 55 MiB/s wr, 885 op/s; 19996/224268 objects degraded (8.916%); 7.7 MiB/s, 123 objects/s recovering 2024-04-23T16:43:36.111 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:36.194 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:35 smithi103 ceph-mon[39629]: pgmap v1590: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.8 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 55 MiB/s wr, 885 op/s; 19996/224268 objects degraded (8.916%); 7.7 MiB/s, 123 objects/s recovering 2024-04-23T16:43:36.225 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:43:36.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:36.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:43:36.330 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:43:36.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:35 smithi188 ceph-mon[44101]: pgmap v1590: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.8 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 55 MiB/s wr, 885 op/s; 19996/224268 objects degraded (8.916%); 7.7 MiB/s, 123 objects/s recovering 2024-04-23T16:43:36.362 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:36.384 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:43:36.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:36.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:43:36.488 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:43:36.516 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:36.538 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 85 16 114577 114561 84.2252 51.75 0.022004 0.0118699 2024-04-23T16:43:36.539 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:43:36.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:36.544 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:43:36.645 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:43:36.679 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:36.996 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:43:37.129 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3671204576' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:43:37.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3671204576' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:43:37.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3671204576' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:43:37.538 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 86 16 115378 115362 83.8279 50.0625 0.0047807 0.0119258 2024-04-23T16:43:37.630 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:43:37.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:37.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:43:37.734 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:43:37.766 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:37.792 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:43:37.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:37.795 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:43:37.896 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:43:37.984 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:43:37.985 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:43:37.985 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2024-04-23T16:43:37.985 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2024-04-23T16:43:37.985 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:43:38.003 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:38.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:37 smithi188 ceph-mon[44101]: pgmap v1591: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 50 MiB/s wr, 807 op/s; 19996/225812 objects degraded (8.855%); 6.6 MiB/s, 104 objects/s recovering 2024-04-23T16:43:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:37 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:43:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:37 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3671204576' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:43:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:37 smithi188 ceph-mon[44101]: osdmap e571: 12 total, 11 up, 9 in 2024-04-23T16:43:38.270 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:43:38.289 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:43:38.313 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:43:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:37 smithi103 ceph-mon[39629]: pgmap v1591: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 50 MiB/s wr, 807 op/s; 19996/225812 objects degraded (8.855%); 6.6 MiB/s, 104 objects/s recovering 2024-04-23T16:43:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:37 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:43:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:37 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3671204576' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:43:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:37 smithi103 ceph-mon[39629]: osdmap e571: 12 total, 11 up, 9 in 2024-04-23T16:43:38.358 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:43:38.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:37 smithi148 ceph-mon[44520]: pgmap v1591: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 50 MiB/s wr, 807 op/s; 19996/225812 objects degraded (8.855%); 6.6 MiB/s, 104 objects/s recovering 2024-04-23T16:43:38.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:37 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:43:38.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:37 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3671204576' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:43:38.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:37 smithi148 ceph-mon[44520]: osdmap e571: 12 total, 11 up, 9 in 2024-04-23T16:43:38.388 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:43:38.421 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:43:38.474 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:43:38.474 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:43:38.492 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:43:38.514 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:43:38.538 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 87 16 116168 116152 83.4319 49.375 0.012765 0.0119836 2024-04-23T16:43:38.540 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:43:38.540 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:43:38.541 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:43:38.541 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:43:38.541 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:43:38.541 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:43:38.541 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:43:38.541 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:43:38.541 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:43:38.541 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:43:38.541 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:43:38.541 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:43:38.541 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:43:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:38 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1601438502' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:43:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:38 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1601438502' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:43:39.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:38 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1601438502' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:43:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:39.538 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 88 16 117026 117010 83.0931 53.625 0.00440545 0.012031 2024-04-23T16:43:39.739 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:43:39.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:39.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:43:39.844 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:43:39.877 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:39.904 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:43:39.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:39.907 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:43:40.008 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:43:40.101 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:40.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:40 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:40 smithi188 ceph-mon[44101]: pgmap v1593: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 38 MiB/s wr, 604 op/s; 19996/226284 objects degraded (8.837%); 4.1 MiB/s, 64 objects/s recovering 2024-04-23T16:43:40.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:40 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 19996/226284 objects degraded (8.837%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:40 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:40 smithi103 ceph-mon[39629]: pgmap v1593: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 38 MiB/s wr, 604 op/s; 19996/226284 objects degraded (8.837%); 4.1 MiB/s, 64 objects/s recovering 2024-04-23T16:43:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:40 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 19996/226284 objects degraded (8.837%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:40.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:40.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:40.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:40 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:40.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:40 smithi148 ceph-mon[44520]: pgmap v1593: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 38 MiB/s wr, 604 op/s; 19996/226284 objects degraded (8.837%); 4.1 MiB/s, 64 objects/s recovering 2024-04-23T16:43:40.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:40 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 19996/226284 objects degraded (8.837%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:40.518 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:43:40.519 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:43:40.538 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 89 16 117857 117841 82.7431 51.9375 0.00751847 0.0120835 2024-04-23T16:43:41.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:41 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:43:41.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:41 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:43:41.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:41 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:43:41.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:41 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:43:41.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:41 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:43:41.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:41 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:43:41.539 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 90 16 118678 118662 82.3938 51.3125 0.0248605 0.0121329 2024-04-23T16:43:41.634 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:43:41.905 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:43:42.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:42 smithi103 ceph-mon[39629]: pgmap v1594: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 7.1 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 50 MiB/s wr, 807 op/s; 19461/233098 objects degraded (8.349%); 6.7 MiB/s, 106 objects/s recovering 2024-04-23T16:43:42.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:42 smithi148 ceph-mon[44520]: pgmap v1594: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 7.1 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 50 MiB/s wr, 807 op/s; 19461/233098 objects degraded (8.349%); 6.7 MiB/s, 106 objects/s recovering 2024-04-23T16:43:42.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:42 smithi188 ceph-mon[44101]: pgmap v1594: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 7.1 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 50 MiB/s wr, 807 op/s; 19461/233098 objects degraded (8.349%); 6.7 MiB/s, 106 objects/s recovering 2024-04-23T16:43:42.520 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:43:42.539 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Total time run: 90.0284 2024-04-23T16:43:42.539 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Total writes made: 118678 2024-04-23T16:43:42.539 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Write size: 65536 2024-04-23T16:43:42.539 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Object size: 65536 2024-04-23T16:43:42.539 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Bandwidth (MB/sec): 82.3893 2024-04-23T16:43:42.539 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev Bandwidth: 85.5477 2024-04-23T16:43:42.539 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max bandwidth (MB/sec): 376.938 2024-04-23T16:43:42.539 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min bandwidth (MB/sec): 0 2024-04-23T16:43:42.539 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Average IOPS: 1318 2024-04-23T16:43:42.540 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev IOPS: 1368.77 2024-04-23T16:43:42.540 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max IOPS: 6031 2024-04-23T16:43:42.540 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min IOPS: 0 2024-04-23T16:43:42.540 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Average Latency(s): 0.0121364 2024-04-23T16:43:42.540 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev Latency(s): 0.293613 2024-04-23T16:43:42.540 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max latency(s): 25.4619 2024-04-23T16:43:42.540 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min latency(s): 0.000969795 2024-04-23T16:43:42.545 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Cleaning up (deleting benchmark objects) 2024-04-23T16:43:44.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:44 smithi103 ceph-mon[39629]: pgmap v1595: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 7.2 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 51 MiB/s wr, 819 op/s; 19461/234494 objects degraded (8.299%); 6.6 MiB/s, 105 objects/s recovering 2024-04-23T16:43:44.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:44 smithi148 ceph-mon[44520]: pgmap v1595: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 7.2 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 51 MiB/s wr, 819 op/s; 19461/234494 objects degraded (8.299%); 6.6 MiB/s, 105 objects/s recovering 2024-04-23T16:43:44.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:44 smithi188 ceph-mon[44101]: pgmap v1595: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 7.2 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 51 MiB/s wr, 819 op/s; 19461/234494 objects degraded (8.299%); 6.6 MiB/s, 105 objects/s recovering 2024-04-23T16:43:44.509 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:43:44.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:44.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:43:44.536 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-23T16:43:44.623 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:43:44.652 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:44.673 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:43:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:44.679 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:43:44.753 INFO:teuthology.orchestra.run.smithi103.stdout: marking collection for removal 2024-04-23T16:43:44.753 INFO:teuthology.orchestra.run.smithi103.stdout:setting '_remove' omap key 2024-04-23T16:43:44.773 INFO:teuthology.orchestra.run.smithi103.stdout:finish_remove_pgs 7.3_head removing 7.3 2024-04-23T16:43:44.780 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:43:44.811 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:44.832 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:43:44.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:44.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:43:44.858 INFO:teuthology.orchestra.run.smithi103.stdout:Remove successful 2024-04-23T16:43:44.939 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:43:45.054 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:45.097 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:43:45.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:45.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:43:45.206 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:43:45.244 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:43:45.301 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:45 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 19461/234494 objects degraded (8.299%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:45.301 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:45 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:43:45.301 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:45 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:43:45.312 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:45.314 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:45 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 19461/234494 objects degraded (8.299%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:45.314 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:45 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:43:45.314 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:45 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:43:45.368 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:43:45.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:45.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:43:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:45 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 19461/234494 objects degraded (8.299%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:45 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:43:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:45 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:43:45.480 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:43:45.510 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:45.641 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:43:45.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:45.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:43:45.667 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:43:45.745 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:43:45.774 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:45.974 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:43:45.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:45.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:43:46.078 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:43:46.130 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:46.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:46 smithi103 ceph-mon[39629]: pgmap v1596: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 7.2 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 102 B/s rd, 40 MiB/s wr, 765 op/s; 18744/234814 objects degraded (7.982%); 7.8 MiB/s, 125 objects/s recovering 2024-04-23T16:43:46.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:46 smithi103 ceph-mon[39629]: osdmap e572: 12 total, 11 up, 9 in 2024-04-23T16:43:46.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:46 smithi148 ceph-mon[44520]: pgmap v1596: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 7.2 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 102 B/s rd, 40 MiB/s wr, 765 op/s; 18744/234814 objects degraded (7.982%); 7.8 MiB/s, 125 objects/s recovering 2024-04-23T16:43:46.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:46 smithi148 ceph-mon[44520]: osdmap e572: 12 total, 11 up, 9 in 2024-04-23T16:43:46.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:46 smithi188 ceph-mon[44101]: pgmap v1596: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 7.2 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 102 B/s rd, 40 MiB/s wr, 765 op/s; 18744/234814 objects degraded (7.982%); 7.8 MiB/s, 125 objects/s recovering 2024-04-23T16:43:46.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:46 smithi188 ceph-mon[44101]: osdmap e572: 12 total, 11 up, 9 in 2024-04-23T16:43:46.730 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.3 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.7.3.3 2024-04-23T16:43:47.080 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:43:47.090 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.3/config 2024-04-23T16:43:47.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:47 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2301998118' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:43:47.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:47 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2301998118' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:43:47.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:47 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2301998118' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:43:47.714 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:43:47.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:47.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:43:47.823 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:43:47.853 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:47.874 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:43:47.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:47.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:43:47.979 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:43:48.010 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:48.038 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:43:48.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:48.043 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:43:48.144 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:43:48.226 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:48.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:48 smithi103 ceph-mon[39629]: pgmap v1598: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 7.1 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 103 B/s rd, 37 MiB/s wr, 761 op/s; 18744/233884 objects degraded (8.014%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:43:48.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:48 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:43:48.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:48 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2301998118' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:43:48.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:48 smithi103 ceph-mon[39629]: osdmap e573: 12 total, 11 up, 9 in 2024-04-23T16:43:48.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:48 smithi148 ceph-mon[44520]: pgmap v1598: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 7.1 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 103 B/s rd, 37 MiB/s wr, 761 op/s; 18744/233884 objects degraded (8.014%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:43:48.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:48 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:43:48.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:48 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2301998118' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:43:48.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:48 smithi148 ceph-mon[44520]: osdmap e573: 12 total, 11 up, 9 in 2024-04-23T16:43:48.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:48 smithi188 ceph-mon[44101]: pgmap v1598: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 7.1 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 103 B/s rd, 37 MiB/s wr, 761 op/s; 18744/233884 objects degraded (8.014%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:43:48.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:48 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:43:48.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:48 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2301998118' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:43:48.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:48 smithi188 ceph-mon[44101]: osdmap e573: 12 total, 11 up, 9 in 2024-04-23T16:43:49.233 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:43:49.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:49.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:43:49.341 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:43:49.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:49 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:43:49.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:49 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:43:49.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:49 smithi103 ceph-mon[39629]: osdmap e574: 12 total, 11 up, 9 in 2024-04-23T16:43:49.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:49 smithi103 ceph-mon[39629]: pgmap v1601: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 7.1 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 170 B/s rd, 15 MiB/s wr, 573 op/s; 18744/233328 objects degraded (8.033%); 7.5 MiB/s, 119 objects/s recovering 2024-04-23T16:43:49.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:49 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:43:49.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:49 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:43:49.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:49 smithi188 ceph-mon[44101]: osdmap e574: 12 total, 11 up, 9 in 2024-04-23T16:43:49.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:49 smithi188 ceph-mon[44101]: pgmap v1601: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 7.1 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 170 B/s rd, 15 MiB/s wr, 573 op/s; 18744/233328 objects degraded (8.033%); 7.5 MiB/s, 119 objects/s recovering 2024-04-23T16:43:49.374 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:49.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:49 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:43:49.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:49 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:43:49.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:49 smithi148 ceph-mon[44520]: osdmap e574: 12 total, 11 up, 9 in 2024-04-23T16:43:49.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:49 smithi148 ceph-mon[44520]: pgmap v1601: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 7.1 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 170 B/s rd, 15 MiB/s wr, 573 op/s; 18744/233328 objects degraded (8.033%); 7.5 MiB/s, 119 objects/s recovering 2024-04-23T16:43:49.397 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:43:49.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:49.400 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:43:49.501 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:43:49.592 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:50.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:50 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 18744/233328 objects degraded (8.033%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:50.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:50 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 18744/233328 objects degraded (8.033%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:50.527 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:50 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 18744/233328 objects degraded (8.033%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:50.582 INFO:teuthology.orchestra.run.smithi103.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:43:50.678 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:43:50.678 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:43:50.678 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:43:50.682 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:43:50.704 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:43:50.749 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:43:50.773 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:43:50.796 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:43:50.818 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:43:50.841 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:43:50.865 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:43:50.866 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:43:50.866 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:43:50.866 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:43:50.866 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:43:50.866 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:43:50.866 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:43:50.866 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:43:50.866 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:43:50.866 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:43:50.866 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:43:50.866 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:43:50.867 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:43:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:51 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2357525175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:43:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:51 smithi103 ceph-mon[39629]: pgmap v1602: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.9 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 906 KiB/s wr, 723 op/s; 18198/226476 objects degraded (8.035%); 5.7 MiB/s, 90 objects/s recovering 2024-04-23T16:43:51.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:51 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2357525175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:43:51.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:51.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:51.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:51.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:51.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:51.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:51.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:51 smithi188 ceph-mon[44101]: pgmap v1602: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.9 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 906 KiB/s wr, 723 op/s; 18198/226476 objects degraded (8.035%); 5.7 MiB/s, 90 objects/s recovering 2024-04-23T16:43:51.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:51 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2357525175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:43:51.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:51.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:51.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:51.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:51.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:51.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:51.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:51 smithi148 ceph-mon[44520]: pgmap v1602: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.9 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 906 KiB/s wr, 723 op/s; 18198/226476 objects degraded (8.035%); 5.7 MiB/s, 90 objects/s recovering 2024-04-23T16:43:51.646 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:43:51.667 INFO:teuthology.orchestra.run.smithi103.stdout:get_pg_num_history pg_num_history pg_num_history(e559 pg_nums {1={301=11},6={455=16},7={540=16}} deleted_pools 537,6) 2024-04-23T16:43:51.667 INFO:teuthology.orchestra.run.smithi103.stdout:Importing pgid 7.3 2024-04-23T16:43:51.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0014ee7:::benchmark_data_smithi188_141483_object816:head# 2024-04-23T16:43:51.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c001b0c0:::benchmark_data_smithi188_141483_object89236:head# 2024-04-23T16:43:51.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0033545:::benchmark_data_smithi188_141483_object36957:head# 2024-04-23T16:43:51.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0036501:::benchmark_data_smithi188_141483_object50421:head# 2024-04-23T16:43:51.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.690 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0041547:::benchmark_data_smithi188_141483_object54456:head# 2024-04-23T16:43:51.690 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.691 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0048666:::benchmark_data_smithi188_141483_object56914:head# 2024-04-23T16:43:51.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c005bcc5:::benchmark_data_smithi188_141483_object60436:head# 2024-04-23T16:43:51.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.692 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c005d4c6:::benchmark_data_smithi188_141483_object53279:head# 2024-04-23T16:43:51.692 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.694 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0062d2b:::benchmark_data_smithi188_141483_object26071:head# 2024-04-23T16:43:51.694 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c006e672:::benchmark_data_smithi188_141483_object90167:head# 2024-04-23T16:43:51.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.695 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00716a6:::benchmark_data_smithi188_141483_object6904:head# 2024-04-23T16:43:51.695 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.696 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c008041d:::benchmark_data_smithi188_141483_object53053:head# 2024-04-23T16:43:51.696 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.696 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0087c6d:::benchmark_data_smithi188_141483_object48895:head# 2024-04-23T16:43:51.696 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.696 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00931eb:::benchmark_data_smithi188_141483_object1342:head# 2024-04-23T16:43:51.696 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.697 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c009f20e:::benchmark_data_smithi188_141483_object60741:head# 2024-04-23T16:43:51.697 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00a06be:::benchmark_data_smithi188_141483_object19916:head# 2024-04-23T16:43:51.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00a4f7f:::benchmark_data_smithi188_141483_object9991:head# 2024-04-23T16:43:51.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00a8c59:::benchmark_data_smithi188_141483_object885:head# 2024-04-23T16:43:51.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00aa466:::benchmark_data_smithi188_141483_object42263:head# 2024-04-23T16:43:51.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00b28d4:::benchmark_data_smithi188_141483_object22519:head# 2024-04-23T16:43:51.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.698 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00b4f6f:::benchmark_data_smithi188_141483_object47832:head# 2024-04-23T16:43:51.698 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00c7679:::benchmark_data_smithi188_141483_object20097:head# 2024-04-23T16:43:51.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00c9ad1:::benchmark_data_smithi188_141483_object88619:head# 2024-04-23T16:43:51.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00ceb65:::benchmark_data_smithi188_141483_object16034:head# 2024-04-23T16:43:51.701 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.701 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00e8da9:::benchmark_data_smithi188_141483_object84174:head# 2024-04-23T16:43:51.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00eb4d6:::benchmark_data_smithi188_141483_object59957:head# 2024-04-23T16:43:51.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00ec1b7:::benchmark_data_smithi188_141483_object25997:head# 2024-04-23T16:43:51.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00ef2df:::benchmark_data_smithi188_141483_object88755:head# 2024-04-23T16:43:51.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00f058a:::benchmark_data_smithi188_141483_object83989:head# 2024-04-23T16:43:51.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00f45c9:::benchmark_data_smithi188_141483_object50021:head# 2024-04-23T16:43:51.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.702 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00f9cdb:::benchmark_data_smithi188_141483_object79579:head# 2024-04-23T16:43:51.702 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00fa02c:::benchmark_data_smithi188_141483_object16661:head# 2024-04-23T16:43:51.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c00fb59a:::benchmark_data_smithi188_141483_object63835:head# 2024-04-23T16:43:51.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c010a585:::benchmark_data_smithi188_141483_object59823:head# 2024-04-23T16:43:51.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c011c7ba:::benchmark_data_smithi188_141483_object23061:head# 2024-04-23T16:43:51.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.703 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c012213b:::benchmark_data_smithi188_141483_object76057:head# 2024-04-23T16:43:51.703 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0122edb:::benchmark_data_smithi188_141483_object38664:head# 2024-04-23T16:43:51.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c012ef9b:::benchmark_data_smithi188_141483_object51942:head# 2024-04-23T16:43:51.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0145c4c:::benchmark_data_smithi188_141483_object32073:head# 2024-04-23T16:43:51.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0150384:::benchmark_data_smithi188_141483_object89490:head# 2024-04-23T16:43:51.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c01561b3:::benchmark_data_smithi188_141483_object20612:head# 2024-04-23T16:43:51.706 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.706 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c016d3f7:::benchmark_data_smithi188_141483_object90106:head# 2024-04-23T16:43:51.707 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.707 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c016f811:::benchmark_data_smithi188_141483_object13777:head# 2024-04-23T16:43:51.707 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.707 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0175f44:::benchmark_data_smithi188_141483_object50618:head# 2024-04-23T16:43:51.707 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.707 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c017c52a:::benchmark_data_smithi188_141483_object58694:head# 2024-04-23T16:43:51.707 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.707 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0181bec:::benchmark_data_smithi188_141483_object50566:head# 2024-04-23T16:43:51.707 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.707 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c01881ae:::benchmark_data_smithi188_141483_object90342:head# 2024-04-23T16:43:51.707 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.707 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c019ce59:::benchmark_data_smithi188_141483_object53057:head# 2024-04-23T16:43:51.707 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.707 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c01a38e1:::benchmark_data_smithi188_141483_object43109:head# 2024-04-23T16:43:51.708 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.708 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c01a635b:::benchmark_data_smithi188_141483_object15854:head# 2024-04-23T16:43:51.708 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.708 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c01b982e:::benchmark_data_smithi188_141483_object37970:head# 2024-04-23T16:43:51.708 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.708 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c01c57db:::benchmark_data_smithi188_141483_object13382:head# 2024-04-23T16:43:51.708 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.708 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c01ca240:::benchmark_data_smithi188_141483_object78388:head# 2024-04-23T16:43:51.708 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c01d5060:::benchmark_data_smithi188_141483_object33286:head# 2024-04-23T16:43:51.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.711 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c01e3d1f:::benchmark_data_smithi188_141483_object35911:head# 2024-04-23T16:43:51.711 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c01f1311:::benchmark_data_smithi188_141483_object44452:head# 2024-04-23T16:43:51.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0209bf4:::benchmark_data_smithi188_141483_object42646:head# 2024-04-23T16:43:51.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0223c40:::benchmark_data_smithi188_141483_object77983:head# 2024-04-23T16:43:51.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c02292b2:::benchmark_data_smithi188_141483_object5815:head# 2024-04-23T16:43:51.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0239cfb:::benchmark_data_smithi188_141483_object79035:head# 2024-04-23T16:43:51.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c023cb72:::benchmark_data_smithi188_141483_object32964:head# 2024-04-23T16:43:51.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c024a404:::benchmark_data_smithi188_141483_object76484:head# 2024-04-23T16:43:51.712 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.712 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c024bb47:::benchmark_data_smithi188_141483_object63114:head# 2024-04-23T16:43:51.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0256804:::benchmark_data_smithi188_141483_object87735:head# 2024-04-23T16:43:51.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c02685d8:::benchmark_data_smithi188_141483_object27330:head# 2024-04-23T16:43:51.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c026ec27:::benchmark_data_smithi188_141483_object78865:head# 2024-04-23T16:43:51.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c02873a3:::benchmark_data_smithi188_141483_object25384:head# 2024-04-23T16:43:51.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c029116b:::benchmark_data_smithi188_141483_object13159:head# 2024-04-23T16:43:51.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c02a1e8b:::benchmark_data_smithi188_141483_object89628:head# 2024-04-23T16:43:51.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.713 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c02a5de6:::benchmark_data_smithi188_141483_object90005:head# 2024-04-23T16:43:51.713 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.714 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c02afe67:::benchmark_data_smithi188_141483_object12697:head# 2024-04-23T16:43:51.714 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.714 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c02b4e9b:::benchmark_data_smithi188_141483_object5588:head# 2024-04-23T16:43:51.714 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.714 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c02c8542:::benchmark_data_smithi188_141483_object2479:head# 2024-04-23T16:43:51.714 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c02f2ced:::benchmark_data_smithi188_141483_object64039:head# 2024-04-23T16:43:51.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c032562d:::benchmark_data_smithi188_141483_object41945:head# 2024-04-23T16:43:51.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c032fd40:::benchmark_data_smithi188_141483_object63298:head# 2024-04-23T16:43:51.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c034824f:::benchmark_data_smithi188_141483_object64967:head# 2024-04-23T16:43:51.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.717 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0369b19:::benchmark_data_smithi188_141483_object18649:head# 2024-04-23T16:43:51.717 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c036cab1:::benchmark_data_smithi188_141483_object58447:head# 2024-04-23T16:43:51.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0392dd6:::benchmark_data_smithi188_141483_object88410:head# 2024-04-23T16:43:51.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c039ef4d:::benchmark_data_smithi188_141483_object17709:head# 2024-04-23T16:43:51.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c03a4f85:::benchmark_data_smithi188_141483_object23491:head# 2024-04-23T16:43:51.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c03aa5ff:::benchmark_data_smithi188_141483_object57901:head# 2024-04-23T16:43:51.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c03b0876:::benchmark_data_smithi188_141483_object52147:head# 2024-04-23T16:43:51.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.718 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c03b9aba:::benchmark_data_smithi188_141483_object76821:head# 2024-04-23T16:43:51.718 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c03cd1e6:::benchmark_data_smithi188_141483_object889:head# 2024-04-23T16:43:51.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c03d05f7:::benchmark_data_smithi188_141483_object56817:head# 2024-04-23T16:43:51.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c03dc722:::benchmark_data_smithi188_141483_object3976:head# 2024-04-23T16:43:51.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c03e23e6:::benchmark_data_smithi188_141483_object74088:head# 2024-04-23T16:43:51.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c03e532d:::benchmark_data_smithi188_141483_object26531:head# 2024-04-23T16:43:51.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c03e9250:::benchmark_data_smithi188_141483_object73594:head# 2024-04-23T16:43:51.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c03f2f2d:::benchmark_data_smithi188_141483_object31746:head# 2024-04-23T16:43:51.719 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.719 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c03f31b6:::benchmark_data_smithi188_141483_object19820:head# 2024-04-23T16:43:51.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0401505:::benchmark_data_smithi188_141483_object17711:head# 2024-04-23T16:43:51.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c04176a2:::benchmark_data_smithi188_141483_object68561:head# 2024-04-23T16:43:51.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.720 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c041fa77:::benchmark_data_smithi188_141483_object29463:head# 2024-04-23T16:43:51.720 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0432a6f:::benchmark_data_smithi188_141483_object56377:head# 2024-04-23T16:43:51.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0439760:::benchmark_data_smithi188_141483_object11310:head# 2024-04-23T16:43:51.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0439b99:::benchmark_data_smithi188_141483_object75614:head# 2024-04-23T16:43:51.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c04506c1:::benchmark_data_smithi188_141483_object90410:head# 2024-04-23T16:43:51.722 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.722 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c045725c:::benchmark_data_smithi188_141483_object59916:head# 2024-04-23T16:43:51.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c04583c4:::benchmark_data_smithi188_141483_object2948:head# 2024-04-23T16:43:51.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c04608d2:::benchmark_data_smithi188_141483_object24869:head# 2024-04-23T16:43:51.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c046d7ba:::benchmark_data_smithi188_141483_object12405:head# 2024-04-23T16:43:51.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0474111:::benchmark_data_smithi188_141483_object70776:head# 2024-04-23T16:43:51.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c047f59f:::benchmark_data_smithi188_141483_object21363:head# 2024-04-23T16:43:51.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0482d24:::benchmark_data_smithi188_141483_object79392:head# 2024-04-23T16:43:51.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.723 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0497010:::benchmark_data_smithi188_141483_object70081:head# 2024-04-23T16:43:51.723 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c049ac74:::benchmark_data_smithi188_141483_object71426:head# 2024-04-23T16:43:51.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.724 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c04c5efd:::benchmark_data_smithi188_141483_object89359:head# 2024-04-23T16:43:51.724 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c04c9df7:::benchmark_data_smithi188_141483_object52693:head# 2024-04-23T16:43:51.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c04f44e0:::benchmark_data_smithi188_141483_object46655:head# 2024-04-23T16:43:51.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c050727c:::benchmark_data_smithi188_141483_object34693:head# 2024-04-23T16:43:51.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c050a065:::benchmark_data_smithi188_141483_object269:head# 2024-04-23T16:43:51.725 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.725 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c05193fc:::benchmark_data_smithi188_141483_object944:head# 2024-04-23T16:43:51.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0528ad3:::benchmark_data_smithi188_141483_object41158:head# 2024-04-23T16:43:51.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c052979b:::benchmark_data_smithi188_141483_object29501:head# 2024-04-23T16:43:51.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.726 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c052e86b:::benchmark_data_smithi188_141483_object72714:head# 2024-04-23T16:43:51.726 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c053c7cf:::benchmark_data_smithi188_141483_object79940:head# 2024-04-23T16:43:51.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c053deb9:::benchmark_data_smithi188_141483_object26345:head# 2024-04-23T16:43:51.730 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.730 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c054aa4a:::benchmark_data_smithi188_141483_object2822:head# 2024-04-23T16:43:51.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c055468c:::benchmark_data_smithi188_141483_object36886:head# 2024-04-23T16:43:51.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c056494a:::benchmark_data_smithi188_141483_object36204:head# 2024-04-23T16:43:51.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c057c8b0:::benchmark_data_smithi188_141483_object79670:head# 2024-04-23T16:43:51.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c057f42a:::benchmark_data_smithi188_141483_object57078:head# 2024-04-23T16:43:51.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0586bff:::benchmark_data_smithi188_141483_object50138:head# 2024-04-23T16:43:51.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.731 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c058efb6:::benchmark_data_smithi188_141483_object81565:head# 2024-04-23T16:43:51.731 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c05b2ec3:::benchmark_data_smithi188_141483_object71260:head# 2024-04-23T16:43:51.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.732 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c05b4f3a:::benchmark_data_smithi188_141483_object17587:head# 2024-04-23T16:43:51.732 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c05c0fae:::benchmark_data_smithi188_141483_object45802:head# 2024-04-23T16:43:51.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c05eb487:::benchmark_data_smithi188_141483_object2839:head# 2024-04-23T16:43:51.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c05ed09b:::benchmark_data_smithi188_141483_object14627:head# 2024-04-23T16:43:51.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c05f42af:::benchmark_data_smithi188_141483_object31993:head# 2024-04-23T16:43:51.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.733 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c05fb9a0:::benchmark_data_smithi188_141483_object6696:head# 2024-04-23T16:43:51.733 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0607451:::benchmark_data_smithi188_141483_object89949:head# 2024-04-23T16:43:51.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.734 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c06271cf:::benchmark_data_smithi188_141483_object1881:head# 2024-04-23T16:43:51.734 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c062a81b:::benchmark_data_smithi188_141483_object76672:head# 2024-04-23T16:43:51.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c062f33e:::benchmark_data_smithi188_141483_object40352:head# 2024-04-23T16:43:51.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c063561c:::benchmark_data_smithi188_141483_object57914:head# 2024-04-23T16:43:51.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.735 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0635977:::benchmark_data_smithi188_141483_object76226:head# 2024-04-23T16:43:51.735 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c064b36b:::benchmark_data_smithi188_141483_object66097:head# 2024-04-23T16:43:51.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c06517ad:::benchmark_data_smithi188_141483_object1796:head# 2024-04-23T16:43:51.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0654bb3:::benchmark_data_smithi188_141483_object15344:head# 2024-04-23T16:43:51.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.736 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0662ef4:::benchmark_data_smithi188_141483_object59576:head# 2024-04-23T16:43:51.736 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.743 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0665859:::benchmark_data_smithi188_141483_object52570:head# 2024-04-23T16:43:51.743 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.743 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c066f2eb:::benchmark_data_smithi188_141483_object13212:head# 2024-04-23T16:43:51.743 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.744 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c069713e:::benchmark_data_smithi188_141483_object60999:head# 2024-04-23T16:43:51.744 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.744 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c069ed6b:::benchmark_data_smithi188_141483_object87529:head# 2024-04-23T16:43:51.744 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.744 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c06a326f:::benchmark_data_smithi188_141483_object36505:head# 2024-04-23T16:43:51.744 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.744 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c06a7de7:::benchmark_data_smithi188_141483_object57374:head# 2024-04-23T16:43:51.744 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.744 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c06adb94:::benchmark_data_smithi188_141483_object66030:head# 2024-04-23T16:43:51.744 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.744 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c06b881b:::benchmark_data_smithi188_141483_object64778:head# 2024-04-23T16:43:51.744 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.744 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c06c6204:::benchmark_data_smithi188_141483_object13892:head# 2024-04-23T16:43:51.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c06c6934:::benchmark_data_smithi188_141483_object6259:head# 2024-04-23T16:43:51.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c06cb494:::benchmark_data_smithi188_141483_object3999:head# 2024-04-23T16:43:51.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c06cef9a:::benchmark_data_smithi188_141483_object28666:head# 2024-04-23T16:43:51.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c06cf5ce:::benchmark_data_smithi188_141483_object41369:head# 2024-04-23T16:43:51.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c06df36f:::benchmark_data_smithi188_141483_object4830:head# 2024-04-23T16:43:51.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c06e36e3:::benchmark_data_smithi188_141483_object40978:head# 2024-04-23T16:43:51.745 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.745 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c06e41c0:::benchmark_data_smithi188_141483_object35214:head# 2024-04-23T16:43:51.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.746 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c072f733:::benchmark_data_smithi188_141483_object54824:head# 2024-04-23T16:43:51.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.746 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0734453:::benchmark_data_smithi188_141483_object32134:head# 2024-04-23T16:43:51.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.746 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0739af1:::benchmark_data_smithi188_141483_object29313:head# 2024-04-23T16:43:51.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.746 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0744510:::benchmark_data_smithi188_141483_object28053:head# 2024-04-23T16:43:51.746 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0746b13:::benchmark_data_smithi188_141483_object7348:head# 2024-04-23T16:43:51.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0753543:::benchmark_data_smithi188_141483_object69783:head# 2024-04-23T16:43:51.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0755efd:::benchmark_data_smithi188_141483_object50029:head# 2024-04-23T16:43:51.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.749 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0756248:::benchmark_data_smithi188_141483_object87163:head# 2024-04-23T16:43:51.749 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.750 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0779fa7:::benchmark_data_smithi188_141483_object5181:head# 2024-04-23T16:43:51.750 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.750 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c079e013:::benchmark_data_smithi188_141483_object1867:head# 2024-04-23T16:43:51.750 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.750 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c07ac1b7:::benchmark_data_smithi188_141483_object69837:head# 2024-04-23T16:43:51.750 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.750 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c07b3c2e:::benchmark_data_smithi188_141483_object591:head# 2024-04-23T16:43:51.750 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.750 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c07b7c6e:::benchmark_data_smithi188_141483_object68719:head# 2024-04-23T16:43:51.750 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.750 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c07c9ada:::benchmark_data_smithi188_141483_object27789:head# 2024-04-23T16:43:51.750 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.750 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c07cdb06:::benchmark_data_smithi188_141483_object53877:head# 2024-04-23T16:43:51.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c07cf00e:::benchmark_data_smithi188_141483_object50161:head# 2024-04-23T16:43:51.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c07d416f:::benchmark_data_smithi188_141483_object23504:head# 2024-04-23T16:43:51.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c07d444f:::benchmark_data_smithi188_141483_object53788:head# 2024-04-23T16:43:51.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c07d4998:::benchmark_data_smithi188_141483_object50246:head# 2024-04-23T16:43:51.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c07d7fab:::benchmark_data_smithi188_141483_object65227:head# 2024-04-23T16:43:51.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c07dc50c:::benchmark_data_smithi188_141483_object40692:head# 2024-04-23T16:43:51.751 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.751 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c07e5645:::benchmark_data_smithi188_141483_object46680:head# 2024-04-23T16:43:51.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.752 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c07f6007:::benchmark_data_smithi188_141483_object11307:head# 2024-04-23T16:43:51.752 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.753 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c07fafe8:::benchmark_data_smithi188_141483_object76617:head# 2024-04-23T16:43:51.753 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.753 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c07fb53b:::benchmark_data_smithi188_141483_object37325:head# 2024-04-23T16:43:51.753 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.753 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c08063e0:::benchmark_data_smithi188_141483_object50440:head# 2024-04-23T16:43:51.753 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.753 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c080652f:::benchmark_data_smithi188_141483_object89207:head# 2024-04-23T16:43:51.753 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.753 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0818950:::benchmark_data_smithi188_141483_object49487:head# 2024-04-23T16:43:51.754 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.754 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c08297df:::benchmark_data_smithi188_141483_object12763:head# 2024-04-23T16:43:51.754 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.755 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c082fc04:::benchmark_data_smithi188_141483_object78059:head# 2024-04-23T16:43:51.755 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.755 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c083bafb:::benchmark_data_smithi188_141483_object72159:head# 2024-04-23T16:43:51.755 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.755 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c08430ab:::benchmark_data_smithi188_141483_object88875:head# 2024-04-23T16:43:51.755 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.755 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0843f00:::benchmark_data_smithi188_141483_object57764:head# 2024-04-23T16:43:51.755 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.755 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c08576ea:::benchmark_data_smithi188_141483_object61376:head# 2024-04-23T16:43:51.755 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.756 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c086d349:::benchmark_data_smithi188_141483_object86220:head# 2024-04-23T16:43:51.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.756 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c086fb7e:::benchmark_data_smithi188_141483_object70069:head# 2024-04-23T16:43:51.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.756 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c088b032:::benchmark_data_smithi188_141483_object44379:head# 2024-04-23T16:43:51.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.756 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c088c04b:::benchmark_data_smithi188_141483_object21334:head# 2024-04-23T16:43:51.756 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.759 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c08af0ee:::benchmark_data_smithi188_141483_object81745:head# 2024-04-23T16:43:51.759 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.759 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c08bf463:::benchmark_data_smithi188_141483_object10779:head# 2024-04-23T16:43:51.759 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.759 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c08d5eb6:::benchmark_data_smithi188_141483_object42346:head# 2024-04-23T16:43:51.759 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.759 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c08d7aa0:::benchmark_data_smithi188_141483_object67216:head# 2024-04-23T16:43:51.759 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.759 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c08ffdb1:::benchmark_data_smithi188_141483_object66747:head# 2024-04-23T16:43:51.760 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c09124af:::benchmark_data_smithi188_141483_object72302:head# 2024-04-23T16:43:51.760 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c09162fb:::benchmark_data_smithi188_141483_object52468:head# 2024-04-23T16:43:51.760 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c091b91b:::benchmark_data_smithi188_141483_object480:head# 2024-04-23T16:43:51.760 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0921630:::benchmark_data_smithi188_141483_object4433:head# 2024-04-23T16:43:51.760 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c094d356:::benchmark_data_smithi188_141483_object1970:head# 2024-04-23T16:43:51.760 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c094dc23:::benchmark_data_smithi188_141483_object28220:head# 2024-04-23T16:43:51.760 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.760 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c097471e:::benchmark_data_smithi188_141483_object68882:head# 2024-04-23T16:43:51.761 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.761 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c097c3bd:::benchmark_data_smithi188_141483_object54825:head# 2024-04-23T16:43:51.761 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.761 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0981073:::benchmark_data_smithi188_141483_object14474:head# 2024-04-23T16:43:51.761 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.761 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c098b0a0:::benchmark_data_smithi188_141483_object20064:head# 2024-04-23T16:43:51.761 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.761 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c098c22a:::benchmark_data_smithi188_141483_object19543:head# 2024-04-23T16:43:51.761 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c09963a1:::benchmark_data_smithi188_141483_object22591:head# 2024-04-23T16:43:51.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0999d24:::benchmark_data_smithi188_141483_object24882:head# 2024-04-23T16:43:51.764 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.764 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c099fa2b:::benchmark_data_smithi188_141483_object32620:head# 2024-04-23T16:43:51.765 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.765 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c09a1107:::benchmark_data_smithi188_141483_object3510:head# 2024-04-23T16:43:51.765 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.765 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c09a82ea:::benchmark_data_smithi188_141483_object70942:head# 2024-04-23T16:43:51.765 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.765 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c09aed36:::benchmark_data_smithi188_141483_object8969:head# 2024-04-23T16:43:51.765 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.765 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c09bb1fb:::benchmark_data_smithi188_141483_object60794:head# 2024-04-23T16:43:51.765 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.765 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c09bc32a:::benchmark_data_smithi188_141483_object82714:head# 2024-04-23T16:43:51.765 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.765 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c09cdb81:::benchmark_data_smithi188_141483_object22749:head# 2024-04-23T16:43:51.765 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.765 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c09dbef9:::benchmark_data_smithi188_141483_object81866:head# 2024-04-23T16:43:51.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c09ecc2b:::benchmark_data_smithi188_141483_object54264:head# 2024-04-23T16:43:51.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c09f2f5c:::benchmark_data_smithi188_141483_object33196:head# 2024-04-23T16:43:51.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c09f8f64:::benchmark_data_smithi188_141483_object1828:head# 2024-04-23T16:43:51.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c09ffa03:::benchmark_data_smithi188_141483_object59177:head# 2024-04-23T16:43:51.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0a05eb3:::benchmark_data_smithi188_141483_object59104:head# 2024-04-23T16:43:51.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.766 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0a07dda:::benchmark_data_smithi188_141483_object13834:head# 2024-04-23T16:43:51.766 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.767 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0a0ed19:::benchmark_data_smithi188_141483_object5451:head# 2024-04-23T16:43:51.767 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.767 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0a165d5:::benchmark_data_smithi188_141483_object65443:head# 2024-04-23T16:43:51.767 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.767 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0a1b974:::benchmark_data_smithi188_141483_object2756:head# 2024-04-23T16:43:51.768 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.768 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0a1e452:::benchmark_data_smithi188_141483_object56780:head# 2024-04-23T16:43:51.768 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.768 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0a20f3a:::benchmark_data_smithi188_141483_object3334:head# 2024-04-23T16:43:51.768 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.768 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0a2df9f:::benchmark_data_smithi188_141483_object35627:head# 2024-04-23T16:43:51.768 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.769 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0a4189a:::benchmark_data_smithi188_141483_object62497:head# 2024-04-23T16:43:51.769 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.770 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0a4e7fd:::benchmark_data_smithi188_141483_object84501:head# 2024-04-23T16:43:51.770 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.770 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0a5818e:::benchmark_data_smithi188_141483_object26647:head# 2024-04-23T16:43:51.770 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.770 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0a72ccf:::benchmark_data_smithi188_141483_object57053:head# 2024-04-23T16:43:51.770 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.770 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0a763b5:::benchmark_data_smithi188_141483_object60700:head# 2024-04-23T16:43:51.770 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.770 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0a8abd5:::benchmark_data_smithi188_141483_object81593:head# 2024-04-23T16:43:51.770 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.770 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0a8f36a:::benchmark_data_smithi188_141483_object687:head# 2024-04-23T16:43:51.770 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.770 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0aa46b5:::benchmark_data_smithi188_141483_object64388:head# 2024-04-23T16:43:51.771 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0ab3133:::benchmark_data_smithi188_141483_object66636:head# 2024-04-23T16:43:51.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0ab86bf:::benchmark_data_smithi188_141483_object89968:head# 2024-04-23T16:43:51.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0ac9ba6:::benchmark_data_smithi188_141483_object18576:head# 2024-04-23T16:43:51.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0adec1f:::benchmark_data_smithi188_141483_object54494:head# 2024-04-23T16:43:51.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.773 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0aed50b:::benchmark_data_smithi188_141483_object55218:head# 2024-04-23T16:43:51.773 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.774 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0b06925:::benchmark_data_smithi188_141483_object88546:head# 2024-04-23T16:43:51.774 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.774 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0b13903:::benchmark_data_smithi188_141483_object27493:head# 2024-04-23T16:43:51.774 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.774 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0b239cf:::benchmark_data_smithi188_141483_object4641:head# 2024-04-23T16:43:51.774 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.774 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0b26f3d:::benchmark_data_smithi188_141483_object74333:head# 2024-04-23T16:43:51.774 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.774 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0b407fe:::benchmark_data_smithi188_141483_object51933:head# 2024-04-23T16:43:51.774 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.774 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0b5070d:::benchmark_data_smithi188_141483_object88153:head# 2024-04-23T16:43:51.774 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.774 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0b5a267:::benchmark_data_smithi188_141483_object84567:head# 2024-04-23T16:43:51.775 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.775 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0b5b6e7:::benchmark_data_smithi188_141483_object55292:head# 2024-04-23T16:43:51.775 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.778 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0b5cf1b:::benchmark_data_smithi188_141483_object55042:head# 2024-04-23T16:43:51.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0b6b2e4:::benchmark_data_smithi188_141483_object47126:head# 2024-04-23T16:43:51.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0b71856:::benchmark_data_smithi188_141483_object74609:head# 2024-04-23T16:43:51.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0b8235c:::benchmark_data_smithi188_141483_object70332:head# 2024-04-23T16:43:51.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0b9e265:::benchmark_data_smithi188_141483_object14814:head# 2024-04-23T16:43:51.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0ba6064:::benchmark_data_smithi188_141483_object77448:head# 2024-04-23T16:43:51.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0bac161:::benchmark_data_smithi188_141483_object21121:head# 2024-04-23T16:43:51.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0bbbe20:::benchmark_data_smithi188_141483_object54929:head# 2024-04-23T16:43:51.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0bc67c5:::benchmark_data_smithi188_141483_object34718:head# 2024-04-23T16:43:51.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0bd8610:::benchmark_data_smithi188_141483_object56964:head# 2024-04-23T16:43:51.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0be33bd:::benchmark_data_smithi188_141483_object10942:head# 2024-04-23T16:43:51.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0be4027:::benchmark_data_smithi188_141483_object67270:head# 2024-04-23T16:43:51.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0be497c:::benchmark_data_smithi188_141483_object57928:head# 2024-04-23T16:43:51.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0be6115:::benchmark_data_smithi188_141483_object65400:head# 2024-04-23T16:43:51.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0befb7d:::benchmark_data_smithi188_141483_object56164:head# 2024-04-23T16:43:51.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0bf8553:::benchmark_data_smithi188_141483_object43556:head# 2024-04-23T16:43:51.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0c0106b:::benchmark_data_smithi188_141483_object81581:head# 2024-04-23T16:43:51.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0c20a18:::benchmark_data_smithi188_141483_object29716:head# 2024-04-23T16:43:51.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0c4f9bf:::benchmark_data_smithi188_141483_object70672:head# 2024-04-23T16:43:51.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0c51140:::benchmark_data_smithi188_141483_object51945:head# 2024-04-23T16:43:51.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0c7d1cb:::benchmark_data_smithi188_141483_object37573:head# 2024-04-23T16:43:51.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0c7da3b:::benchmark_data_smithi188_141483_object41049:head# 2024-04-23T16:43:51.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0c7dbbf:::benchmark_data_smithi188_141483_object54588:head# 2024-04-23T16:43:51.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0c836c7:::benchmark_data_smithi188_141483_object10060:head# 2024-04-23T16:43:51.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0c99494:::benchmark_data_smithi188_141483_object51556:head# 2024-04-23T16:43:51.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0c9b448:::benchmark_data_smithi188_141483_object59080:head# 2024-04-23T16:43:51.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0c9d886:::benchmark_data_smithi188_141483_object67822:head# 2024-04-23T16:43:51.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0ca264f:::benchmark_data_smithi188_141483_object52309:head# 2024-04-23T16:43:51.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0cb84a2:::benchmark_data_smithi188_141483_object10231:head# 2024-04-23T16:43:51.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0ce1a48:::benchmark_data_smithi188_141483_object10447:head# 2024-04-23T16:43:51.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0cf786b:::benchmark_data_smithi188_141483_object34325:head# 2024-04-23T16:43:51.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0d0633d:::benchmark_data_smithi188_141483_object7366:head# 2024-04-23T16:43:51.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0d0d8c0:::benchmark_data_smithi188_141483_object40639:head# 2024-04-23T16:43:51.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0d218d7:::benchmark_data_smithi188_141483_object73716:head# 2024-04-23T16:43:51.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0d2968b:::benchmark_data_smithi188_141483_object12687:head# 2024-04-23T16:43:51.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0d44730:::benchmark_data_smithi188_141483_object12312:head# 2024-04-23T16:43:51.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0d5172e:::benchmark_data_smithi188_141483_object13470:head# 2024-04-23T16:43:51.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0d78545:::benchmark_data_smithi188_141483_object23704:head# 2024-04-23T16:43:51.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0d7fd39:::benchmark_data_smithi188_141483_object48139:head# 2024-04-23T16:43:51.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0d824d1:::benchmark_data_smithi188_141483_object88121:head# 2024-04-23T16:43:51.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0d85b9e:::benchmark_data_smithi188_141483_object40202:head# 2024-04-23T16:43:51.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0d8d18c:::benchmark_data_smithi188_141483_object71829:head# 2024-04-23T16:43:51.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0da9b78:::benchmark_data_smithi188_141483_object86020:head# 2024-04-23T16:43:51.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0dae813:::benchmark_data_smithi188_141483_object31400:head# 2024-04-23T16:43:51.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0db5094:::benchmark_data_smithi188_141483_object86674:head# 2024-04-23T16:43:51.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0dcdb6d:::benchmark_data_smithi188_141483_object45313:head# 2024-04-23T16:43:51.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0dd42ee:::benchmark_data_smithi188_141483_object50172:head# 2024-04-23T16:43:51.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0dda241:::benchmark_data_smithi188_141483_object32118:head# 2024-04-23T16:43:51.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0de4b73:::benchmark_data_smithi188_141483_object44730:head# 2024-04-23T16:43:51.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0dec315:::benchmark_data_smithi188_141483_object27554:head# 2024-04-23T16:43:51.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0deca29:::benchmark_data_smithi188_141483_object74635:head# 2024-04-23T16:43:51.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0df33ed:::benchmark_data_smithi188_141483_object63247:head# 2024-04-23T16:43:51.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.791 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e02ba9:::benchmark_data_smithi188_141483_object40719:head# 2024-04-23T16:43:51.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.791 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e09ac8:::benchmark_data_smithi188_141483_object36172:head# 2024-04-23T16:43:51.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.791 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e11bd6:::benchmark_data_smithi188_141483_object34522:head# 2024-04-23T16:43:51.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e165b0:::benchmark_data_smithi188_141483_object102:head# 2024-04-23T16:43:51.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e1f076:::benchmark_data_smithi188_141483_object31721:head# 2024-04-23T16:43:51.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e200cd:::benchmark_data_smithi188_141483_object63324:head# 2024-04-23T16:43:51.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e2051f:::benchmark_data_smithi188_141483_object23379:head# 2024-04-23T16:43:51.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e2e1f8:::benchmark_data_smithi188_141483_object11433:head# 2024-04-23T16:43:51.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e31f9e:::benchmark_data_smithi188_141483_object56855:head# 2024-04-23T16:43:51.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e3263c:::benchmark_data_smithi188_141483_object21715:head# 2024-04-23T16:43:51.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e37dd5:::benchmark_data_smithi188_141483_object62063:head# 2024-04-23T16:43:51.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e3b2a4:::benchmark_data_smithi188_141483_object82043:head# 2024-04-23T16:43:51.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e3dc27:::benchmark_data_smithi188_141483_object5361:head# 2024-04-23T16:43:51.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e4e185:::benchmark_data_smithi188_141483_object74173:head# 2024-04-23T16:43:51.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e5886e:::benchmark_data_smithi188_141483_object47500:head# 2024-04-23T16:43:51.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e626e6:::benchmark_data_smithi188_141483_object1771:head# 2024-04-23T16:43:51.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e65380:::benchmark_data_smithi188_141483_object17547:head# 2024-04-23T16:43:51.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e6a23a:::benchmark_data_smithi188_141483_object50759:head# 2024-04-23T16:43:51.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e6ff60:::benchmark_data_smithi188_141483_object36183:head# 2024-04-23T16:43:51.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.799 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e741d5:::benchmark_data_smithi188_141483_object85117:head# 2024-04-23T16:43:51.799 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.799 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e762cb:::benchmark_data_smithi188_141483_object78817:head# 2024-04-23T16:43:51.799 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.799 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e7c173:::benchmark_data_smithi188_141483_object77861:head# 2024-04-23T16:43:51.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0e835d5:::benchmark_data_smithi188_141483_object45999:head# 2024-04-23T16:43:51.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0ea7a1c:::benchmark_data_smithi188_141483_object47646:head# 2024-04-23T16:43:51.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0ea9abb:::benchmark_data_smithi188_141483_object7865:head# 2024-04-23T16:43:51.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0eb4c90:::benchmark_data_smithi188_141483_object62556:head# 2024-04-23T16:43:51.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0ebbe20:::benchmark_data_smithi188_141483_object45565:head# 2024-04-23T16:43:51.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0eece46:::benchmark_data_smithi188_141483_object85115:head# 2024-04-23T16:43:51.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0ef3b12:::benchmark_data_smithi188_141483_object69492:head# 2024-04-23T16:43:51.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0ef4ee0:::benchmark_data_smithi188_141483_object5535:head# 2024-04-23T16:43:51.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0ef5b71:::benchmark_data_smithi188_141483_object57762:head# 2024-04-23T16:43:51.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0f0114b:::benchmark_data_smithi188_141483_object61492:head# 2024-04-23T16:43:51.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0f0e021:::benchmark_data_smithi188_141483_object54906:head# 2024-04-23T16:43:51.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0f114ad:::benchmark_data_smithi188_141483_object63626:head# 2024-04-23T16:43:51.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0f1e478:::benchmark_data_smithi188_141483_object21032:head# 2024-04-23T16:43:51.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0f251df:::benchmark_data_smithi188_141483_object22292:head# 2024-04-23T16:43:51.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0f2cfab:::benchmark_data_smithi188_141483_object30597:head# 2024-04-23T16:43:51.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0f345e1:::benchmark_data_smithi188_141483_object88462:head# 2024-04-23T16:43:51.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0f44f99:::benchmark_data_smithi188_141483_object79711:head# 2024-04-23T16:43:51.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0f6791c:::benchmark_data_smithi188_141483_object3656:head# 2024-04-23T16:43:51.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0f8e099:::benchmark_data_smithi188_141483_object23744:head# 2024-04-23T16:43:51.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0f94892:::benchmark_data_smithi188_141483_object2089:head# 2024-04-23T16:43:51.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.803 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0f94f73:::benchmark_data_smithi188_141483_object32035:head# 2024-04-23T16:43:51.803 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.803 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0fa6c2f:::benchmark_data_smithi188_141483_object20983:head# 2024-04-23T16:43:51.803 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0fc8bdb:::benchmark_data_smithi188_141483_object57598:head# 2024-04-23T16:43:51.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c0fe06db:::benchmark_data_smithi188_141483_object25187:head# 2024-04-23T16:43:51.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1012f91:::benchmark_data_smithi188_141483_object28975:head# 2024-04-23T16:43:51.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c102b083:::benchmark_data_smithi188_141483_object51376:head# 2024-04-23T16:43:51.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1050c65:::benchmark_data_smithi188_141483_object24375:head# 2024-04-23T16:43:51.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1055e13:::benchmark_data_smithi188_141483_object85555:head# 2024-04-23T16:43:51.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c105abde:::benchmark_data_smithi188_141483_object19580:head# 2024-04-23T16:43:51.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c105fcef:::benchmark_data_smithi188_141483_object67184:head# 2024-04-23T16:43:51.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1083332:::benchmark_data_smithi188_141483_object83871:head# 2024-04-23T16:43:51.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1085113:::benchmark_data_smithi188_141483_object49815:head# 2024-04-23T16:43:51.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c108bec4:::benchmark_data_smithi188_141483_object67787:head# 2024-04-23T16:43:51.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c10acab2:::benchmark_data_smithi188_141483_object58897:head# 2024-04-23T16:43:51.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c10c02ff:::benchmark_data_smithi188_141483_object18410:head# 2024-04-23T16:43:51.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c10cda7a:::benchmark_data_smithi188_141483_object24423:head# 2024-04-23T16:43:51.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c10cf739:::benchmark_data_smithi188_141483_object19835:head# 2024-04-23T16:43:51.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c10d5a46:::benchmark_data_smithi188_141483_object62142:head# 2024-04-23T16:43:51.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c10d9f70:::benchmark_data_smithi188_141483_object90674:head# 2024-04-23T16:43:51.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c10de4f6:::benchmark_data_smithi188_141483_object70649:head# 2024-04-23T16:43:51.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c10ebac9:::benchmark_data_smithi188_141483_object36298:head# 2024-04-23T16:43:51.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c10f00be:::benchmark_data_smithi188_141483_object63284:head# 2024-04-23T16:43:51.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c10f693e:::benchmark_data_smithi188_141483_object70247:head# 2024-04-23T16:43:51.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1103b3a:::benchmark_data_smithi188_141483_object65577:head# 2024-04-23T16:43:51.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1121a93:::benchmark_data_smithi188_141483_object66109:head# 2024-04-23T16:43:51.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c112bfff:::benchmark_data_smithi188_141483_object291:head# 2024-04-23T16:43:51.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c113008a:::benchmark_data_smithi188_141483_object76767:head# 2024-04-23T16:43:51.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1132e76:::benchmark_data_smithi188_141483_object11656:head# 2024-04-23T16:43:51.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1134c72:::benchmark_data_smithi188_141483_object51019:head# 2024-04-23T16:43:51.814 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1137e4c:::benchmark_data_smithi188_141483_object47286:head# 2024-04-23T16:43:51.814 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c113a14e:::benchmark_data_smithi188_141483_object83231:head# 2024-04-23T16:43:51.814 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1143c7b:::benchmark_data_smithi188_141483_object11883:head# 2024-04-23T16:43:51.814 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1145c0f:::benchmark_data_smithi188_141483_object90341:head# 2024-04-23T16:43:51.814 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c11494f0:::benchmark_data_smithi188_141483_object53465:head# 2024-04-23T16:43:51.814 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c114ca25:::benchmark_data_smithi188_141483_object3862:head# 2024-04-23T16:43:51.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c114d44d:::benchmark_data_smithi188_141483_object86978:head# 2024-04-23T16:43:51.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1153f01:::benchmark_data_smithi188_141483_object14747:head# 2024-04-23T16:43:51.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1154c14:::benchmark_data_smithi188_141483_object65667:head# 2024-04-23T16:43:51.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c115a02f:::benchmark_data_smithi188_141483_object4887:head# 2024-04-23T16:43:51.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c115d8e6:::benchmark_data_smithi188_141483_object51729:head# 2024-04-23T16:43:51.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1161d54:::benchmark_data_smithi188_141483_object52531:head# 2024-04-23T16:43:51.820 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c11802b4:::benchmark_data_smithi188_141483_object90317:head# 2024-04-23T16:43:51.820 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c11cae2a:::benchmark_data_smithi188_141483_object54878:head# 2024-04-23T16:43:51.820 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c11cf9fb:::benchmark_data_smithi188_141483_object48527:head# 2024-04-23T16:43:51.820 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c11d6a97:::benchmark_data_smithi188_141483_object47218:head# 2024-04-23T16:43:51.820 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c11d93ce:::benchmark_data_smithi188_141483_object55460:head# 2024-04-23T16:43:51.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c11e8bb4:::benchmark_data_smithi188_141483_object16880:head# 2024-04-23T16:43:51.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c11ef7f7:::benchmark_data_smithi188_141483_object64137:head# 2024-04-23T16:43:51.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c122b90e:::benchmark_data_smithi188_141483_object31828:head# 2024-04-23T16:43:51.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1230133:::benchmark_data_smithi188_141483_object17377:head# 2024-04-23T16:43:51.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c12378a3:::benchmark_data_smithi188_141483_object42110:head# 2024-04-23T16:43:51.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c123abcc:::benchmark_data_smithi188_141483_object46451:head# 2024-04-23T16:43:51.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1246490:::benchmark_data_smithi188_141483_object23486:head# 2024-04-23T16:43:51.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c124a0fe:::benchmark_data_smithi188_141483_object31338:head# 2024-04-23T16:43:51.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c125088a:::benchmark_data_smithi188_141483_object70234:head# 2024-04-23T16:43:51.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c125396b:::benchmark_data_smithi188_141483_object83203:head# 2024-04-23T16:43:51.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c125451d:::benchmark_data_smithi188_141483_object57460:head# 2024-04-23T16:43:51.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c125952d:::benchmark_data_smithi188_141483_object40390:head# 2024-04-23T16:43:51.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c127be1f:::benchmark_data_smithi188_141483_object53688:head# 2024-04-23T16:43:51.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c128c2da:::benchmark_data_smithi188_141483_object48063:head# 2024-04-23T16:43:51.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c128e669:::benchmark_data_smithi188_141483_object5722:head# 2024-04-23T16:43:51.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1299037:::benchmark_data_smithi188_141483_object2727:head# 2024-04-23T16:43:51.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c12b373a:::benchmark_data_smithi188_141483_object10400:head# 2024-04-23T16:43:51.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c12b660d:::benchmark_data_smithi188_141483_object29580:head# 2024-04-23T16:43:51.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c12b7761:::benchmark_data_smithi188_141483_object54522:head# 2024-04-23T16:43:51.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c12c0a58:::benchmark_data_smithi188_141483_object48552:head# 2024-04-23T16:43:51.824 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1305885:::benchmark_data_smithi188_141483_object52663:head# 2024-04-23T16:43:51.824 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1306a72:::benchmark_data_smithi188_141483_object15338:head# 2024-04-23T16:43:51.824 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1309e39:::benchmark_data_smithi188_141483_object50042:head# 2024-04-23T16:43:51.824 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1311430:::benchmark_data_smithi188_141483_object17411:head# 2024-04-23T16:43:51.824 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c131af80:::benchmark_data_smithi188_141483_object2907:head# 2024-04-23T16:43:51.824 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c134c8d3:::benchmark_data_smithi188_141483_object3538:head# 2024-04-23T16:43:51.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c13540a4:::benchmark_data_smithi188_141483_object2990:head# 2024-04-23T16:43:51.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c135ea1a:::benchmark_data_smithi188_141483_object75223:head# 2024-04-23T16:43:51.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c13763ee:::benchmark_data_smithi188_141483_object49196:head# 2024-04-23T16:43:51.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c137b535:::benchmark_data_smithi188_141483_object11237:head# 2024-04-23T16:43:51.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c137b93c:::benchmark_data_smithi188_141483_object45609:head# 2024-04-23T16:43:51.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1387782:::benchmark_data_smithi188_141483_object6091:head# 2024-04-23T16:43:51.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c138919c:::benchmark_data_smithi188_141483_object60148:head# 2024-04-23T16:43:51.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c138f0c6:::benchmark_data_smithi188_141483_object76418:head# 2024-04-23T16:43:51.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c13945ac:::benchmark_data_smithi188_141483_object68153:head# 2024-04-23T16:43:51.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.828 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1396309:::benchmark_data_smithi188_141483_object41810:head# 2024-04-23T16:43:51.828 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.828 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c13a8c75:::benchmark_data_smithi188_141483_object16979:head# 2024-04-23T16:43:51.828 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.828 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c13b9f6c:::benchmark_data_smithi188_141483_object86435:head# 2024-04-23T16:43:51.828 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.828 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c13c56aa:::benchmark_data_smithi188_141483_object86901:head# 2024-04-23T16:43:51.828 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.828 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c13cac0d:::benchmark_data_smithi188_141483_object64123:head# 2024-04-23T16:43:51.828 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.828 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c13d8eed:::benchmark_data_smithi188_141483_object69344:head# 2024-04-23T16:43:51.828 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.828 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c13e2fff:::benchmark_data_smithi188_141483_object60758:head# 2024-04-23T16:43:51.828 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c13f8007:::benchmark_data_smithi188_141483_object41173:head# 2024-04-23T16:43:51.829 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c13f91a7:::benchmark_data_smithi188_141483_object43242:head# 2024-04-23T16:43:51.829 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c140f547:::benchmark_data_smithi188_141483_object44060:head# 2024-04-23T16:43:51.829 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1413051:::benchmark_data_smithi188_141483_object32722:head# 2024-04-23T16:43:51.829 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c141c324:::benchmark_data_smithi188_141483_object50340:head# 2024-04-23T16:43:51.829 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c142c7dc:::benchmark_data_smithi188_141483_object88618:head# 2024-04-23T16:43:51.829 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1447214:::benchmark_data_smithi188_141483_object15635:head# 2024-04-23T16:43:51.829 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1457917:::benchmark_data_smithi188_141483_object29557:head# 2024-04-23T16:43:51.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.830 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c145d348:::benchmark_data_smithi188_141483_object14542:head# 2024-04-23T16:43:51.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.830 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c145ddab:::benchmark_data_smithi188_141483_object77320:head# 2024-04-23T16:43:51.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.830 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c146132e:::benchmark_data_smithi188_141483_object57720:head# 2024-04-23T16:43:51.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1464f83:::benchmark_data_smithi188_141483_object52125:head# 2024-04-23T16:43:51.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1482cae:::benchmark_data_smithi188_141483_object29647:head# 2024-04-23T16:43:51.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1483f3d:::benchmark_data_smithi188_141483_object78510:head# 2024-04-23T16:43:51.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c148c32b:::benchmark_data_smithi188_141483_object75027:head# 2024-04-23T16:43:51.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c149399f:::benchmark_data_smithi188_141483_object79176:head# 2024-04-23T16:43:51.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1495057:::benchmark_data_smithi188_141483_object49478:head# 2024-04-23T16:43:51.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c149993c:::benchmark_data_smithi188_141483_object23630:head# 2024-04-23T16:43:51.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c14a2355:::benchmark_data_smithi188_141483_object60624:head# 2024-04-23T16:43:51.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c14ae7e7:::benchmark_data_smithi188_141483_object31076:head# 2024-04-23T16:43:51.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c14b93ed:::benchmark_data_smithi188_141483_object7111:head# 2024-04-23T16:43:51.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c14d3402:::benchmark_data_smithi188_141483_object51528:head# 2024-04-23T16:43:51.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.833 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c14dea65:::benchmark_data_smithi188_141483_object22786:head# 2024-04-23T16:43:51.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.833 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c14debb3:::benchmark_data_smithi188_141483_object64669:head# 2024-04-23T16:43:51.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.833 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c14e8dc2:::benchmark_data_smithi188_141483_object85163:head# 2024-04-23T16:43:51.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.833 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c14eb0dc:::benchmark_data_smithi188_141483_object44006:head# 2024-04-23T16:43:51.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.833 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c14ee0aa:::benchmark_data_smithi188_141483_object24493:head# 2024-04-23T16:43:51.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.833 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c14ee6fc:::benchmark_data_smithi188_141483_object55944:head# 2024-04-23T16:43:51.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.833 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c14f0a2e:::benchmark_data_smithi188_141483_object29621:head# 2024-04-23T16:43:51.834 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.834 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c14fd073:::benchmark_data_smithi188_141483_object77498:head# 2024-04-23T16:43:51.834 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.834 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c14fde02:::benchmark_data_smithi188_141483_object85697:head# 2024-04-23T16:43:51.834 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.834 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c14fecf0:::benchmark_data_smithi188_141483_object89850:head# 2024-04-23T16:43:51.834 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.836 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c150c4d5:::benchmark_data_smithi188_141483_object6820:head# 2024-04-23T16:43:51.836 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.836 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1517679:::benchmark_data_smithi188_141483_object23690:head# 2024-04-23T16:43:51.836 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.836 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c151be64:::benchmark_data_smithi188_141483_object31933:head# 2024-04-23T16:43:51.836 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.836 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1522e99:::benchmark_data_smithi188_141483_object8329:head# 2024-04-23T16:43:51.837 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.837 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1529807:::benchmark_data_smithi188_141483_object66326:head# 2024-04-23T16:43:51.837 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.837 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1530758:::benchmark_data_smithi188_141483_object69925:head# 2024-04-23T16:43:51.837 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.837 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c15375df:::benchmark_data_smithi188_141483_object16647:head# 2024-04-23T16:43:51.837 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.837 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1541afc:::benchmark_data_smithi188_141483_object57588:head# 2024-04-23T16:43:51.837 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.837 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c15571d0:::benchmark_data_smithi188_141483_object9289:head# 2024-04-23T16:43:51.837 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.837 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c155a5f6:::benchmark_data_smithi188_141483_object85216:head# 2024-04-23T16:43:51.837 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.837 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1561f8b:::benchmark_data_smithi188_141483_object66964:head# 2024-04-23T16:43:51.837 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.838 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c156d203:::benchmark_data_smithi188_141483_object17905:head# 2024-04-23T16:43:51.838 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.838 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c156eb92:::benchmark_data_smithi188_141483_object29997:head# 2024-04-23T16:43:51.838 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.838 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c157301f:::benchmark_data_smithi188_141483_object12886:head# 2024-04-23T16:43:51.838 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.838 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1576fff:::benchmark_data_smithi188_141483_object37940:head# 2024-04-23T16:43:51.838 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.838 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c15877ee:::benchmark_data_smithi188_141483_object71052:head# 2024-04-23T16:43:51.838 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.838 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1588d17:::benchmark_data_smithi188_141483_object3734:head# 2024-04-23T16:43:51.838 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.841 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c158c8d7:::benchmark_data_smithi188_141483_object22824:head# 2024-04-23T16:43:51.841 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.841 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1594b4d:::benchmark_data_smithi188_141483_object87606:head# 2024-04-23T16:43:51.841 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.841 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c15a0ec6:::benchmark_data_smithi188_141483_object83538:head# 2024-04-23T16:43:51.841 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.841 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c15a6742:::benchmark_data_smithi188_141483_object10936:head# 2024-04-23T16:43:51.841 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.841 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c15b81b1:::benchmark_data_smithi188_141483_object16010:head# 2024-04-23T16:43:51.841 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.842 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c15c62ec:::benchmark_data_smithi188_141483_object69146:head# 2024-04-23T16:43:51.842 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.842 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c15dd701:::benchmark_data_smithi188_141483_object77922:head# 2024-04-23T16:43:51.842 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.842 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c15e6bfd:::benchmark_data_smithi188_141483_object26272:head# 2024-04-23T16:43:51.842 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.842 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c15eb67a:::benchmark_data_smithi188_141483_object85149:head# 2024-04-23T16:43:51.842 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.842 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c15f4966:::benchmark_data_smithi188_141483_object90404:head# 2024-04-23T16:43:51.842 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.842 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c15fbd5c:::benchmark_data_smithi188_141483_object51790:head# 2024-04-23T16:43:51.842 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.842 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c15ff79d:::benchmark_data_smithi188_141483_object77558:head# 2024-04-23T16:43:51.842 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.843 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c161559e:::benchmark_data_smithi188_141483_object13814:head# 2024-04-23T16:43:51.843 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.843 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1616545:::benchmark_data_smithi188_141483_object79871:head# 2024-04-23T16:43:51.843 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c161d78e:::benchmark_data_smithi188_141483_object49945:head# 2024-04-23T16:43:51.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c162182a:::benchmark_data_smithi188_141483_object15760:head# 2024-04-23T16:43:51.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c163d22d:::benchmark_data_smithi188_141483_object56742:head# 2024-04-23T16:43:51.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1644d4c:::benchmark_data_smithi188_141483_object44568:head# 2024-04-23T16:43:51.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1647379:::benchmark_data_smithi188_141483_object56334:head# 2024-04-23T16:43:51.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c165f7e7:::benchmark_data_smithi188_141483_object37848:head# 2024-04-23T16:43:51.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1662c9f:::benchmark_data_smithi188_141483_object29931:head# 2024-04-23T16:43:51.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c166b650:::benchmark_data_smithi188_141483_object26532:head# 2024-04-23T16:43:51.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c166e44a:::benchmark_data_smithi188_141483_object68995:head# 2024-04-23T16:43:51.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1679cfb:::benchmark_data_smithi188_141483_object29760:head# 2024-04-23T16:43:51.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c167b61d:::benchmark_data_smithi188_141483_object16934:head# 2024-04-23T16:43:51.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c167de91:::benchmark_data_smithi188_141483_object76671:head# 2024-04-23T16:43:51.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c168d799:::benchmark_data_smithi188_141483_object84516:head# 2024-04-23T16:43:51.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1690ec3:::benchmark_data_smithi188_141483_object4795:head# 2024-04-23T16:43:51.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c16aa1d3:::benchmark_data_smithi188_141483_object8687:head# 2024-04-23T16:43:51.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c16b4db4:::benchmark_data_smithi188_141483_object3248:head# 2024-04-23T16:43:51.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c16cf5a4:::benchmark_data_smithi188_141483_object62958:head# 2024-04-23T16:43:51.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c16d32eb:::benchmark_data_smithi188_141483_object57984:head# 2024-04-23T16:43:51.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c16d7d9e:::benchmark_data_smithi188_141483_object56156:head# 2024-04-23T16:43:51.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c16de6b9:::benchmark_data_smithi188_141483_object57079:head# 2024-04-23T16:43:51.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c16e3c29:::benchmark_data_smithi188_141483_object90191:head# 2024-04-23T16:43:51.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c16f781f:::benchmark_data_smithi188_141483_object23881:head# 2024-04-23T16:43:51.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1708fd5:::benchmark_data_smithi188_141483_object86647:head# 2024-04-23T16:43:51.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c171525a:::benchmark_data_smithi188_141483_object55734:head# 2024-04-23T16:43:51.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c171efda:::benchmark_data_smithi188_141483_object50404:head# 2024-04-23T16:43:51.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1726f5e:::benchmark_data_smithi188_141483_object2188:head# 2024-04-23T16:43:51.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c17272ec:::benchmark_data_smithi188_141483_object85237:head# 2024-04-23T16:43:51.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c172c51a:::benchmark_data_smithi188_141483_object25126:head# 2024-04-23T16:43:51.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1733f09:::benchmark_data_smithi188_141483_object61220:head# 2024-04-23T16:43:51.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c174746d:::benchmark_data_smithi188_141483_object28833:head# 2024-04-23T16:43:51.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c177b02f:::benchmark_data_smithi188_141483_object8079:head# 2024-04-23T16:43:51.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1782fb2:::benchmark_data_smithi188_141483_object77228:head# 2024-04-23T16:43:51.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1798804:::benchmark_data_smithi188_141483_object10772:head# 2024-04-23T16:43:51.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c17a3d06:::benchmark_data_smithi188_141483_object64273:head# 2024-04-23T16:43:51.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.857 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c17aa7a6:::benchmark_data_smithi188_141483_object81488:head# 2024-04-23T16:43:51.857 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.858 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c17aa8f6:::benchmark_data_smithi188_141483_object42691:head# 2024-04-23T16:43:51.858 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.858 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c17ae7b5:::benchmark_data_smithi188_141483_object75673:head# 2024-04-23T16:43:51.858 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.858 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c17c90a4:::benchmark_data_smithi188_141483_object39848:head# 2024-04-23T16:43:51.859 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c17de52c:::benchmark_data_smithi188_141483_object41491:head# 2024-04-23T16:43:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c17ee4c3:::benchmark_data_smithi188_141483_object23272:head# 2024-04-23T16:43:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c17ef543:::benchmark_data_smithi188_141483_object12407:head# 2024-04-23T16:43:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c17f28aa:::benchmark_data_smithi188_141483_object89737:head# 2024-04-23T16:43:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c17fa53f:::benchmark_data_smithi188_141483_object26281:head# 2024-04-23T16:43:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1806a62:::benchmark_data_smithi188_141483_object49831:head# 2024-04-23T16:43:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c180ac96:::benchmark_data_smithi188_141483_object6140:head# 2024-04-23T16:43:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1812323:::benchmark_data_smithi188_141483_object43093:head# 2024-04-23T16:43:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1815471:::benchmark_data_smithi188_141483_object7340:head# 2024-04-23T16:43:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c18174e3:::benchmark_data_smithi188_141483_object90780:head# 2024-04-23T16:43:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c18226a3:::benchmark_data_smithi188_141483_object53201:head# 2024-04-23T16:43:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c183399f:::benchmark_data_smithi188_141483_object38251:head# 2024-04-23T16:43:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c18343d6:::benchmark_data_smithi188_141483_object49186:head# 2024-04-23T16:43:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c186161c:::benchmark_data_smithi188_141483_object60208:head# 2024-04-23T16:43:51.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c186bf96:::benchmark_data_smithi188_141483_object6378:head# 2024-04-23T16:43:51.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1872c6f:::benchmark_data_smithi188_141483_object43984:head# 2024-04-23T16:43:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1873440:::benchmark_data_smithi188_141483_object6432:head# 2024-04-23T16:43:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1874d54:::benchmark_data_smithi188_141483_object3021:head# 2024-04-23T16:43:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c18802e8:::benchmark_data_smithi188_141483_object9740:head# 2024-04-23T16:43:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c18922bf:::benchmark_data_smithi188_141483_object2931:head# 2024-04-23T16:43:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c18940f1:::benchmark_data_smithi188_141483_object43775:head# 2024-04-23T16:43:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c189c999:::benchmark_data_smithi188_141483_object67934:head# 2024-04-23T16:43:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c18ac14c:::benchmark_data_smithi188_141483_object6660:head# 2024-04-23T16:43:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c18b9fdf:::benchmark_data_smithi188_141483_object77858:head# 2024-04-23T16:43:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c18bbcc8:::benchmark_data_smithi188_141483_object81767:head# 2024-04-23T16:43:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c18c291e:::benchmark_data_smithi188_141483_object75650:head# 2024-04-23T16:43:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c18d30bd:::benchmark_data_smithi188_141483_object42891:head# 2024-04-23T16:43:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c18d3d7f:::benchmark_data_smithi188_141483_object18214:head# 2024-04-23T16:43:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c18fd477:::benchmark_data_smithi188_141483_object69495:head# 2024-04-23T16:43:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c190be0b:::benchmark_data_smithi188_141483_object1644:head# 2024-04-23T16:43:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1910507:::benchmark_data_smithi188_141483_object849:head# 2024-04-23T16:43:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1914840:::benchmark_data_smithi188_141483_object72080:head# 2024-04-23T16:43:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c191900f:::benchmark_data_smithi188_141483_object25428:head# 2024-04-23T16:43:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1929f85:::benchmark_data_smithi188_141483_object22452:head# 2024-04-23T16:43:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c192e660:::benchmark_data_smithi188_141483_object52980:head# 2024-04-23T16:43:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1942670:::benchmark_data_smithi188_141483_object52795:head# 2024-04-23T16:43:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c19512e4:::benchmark_data_smithi188_141483_object78741:head# 2024-04-23T16:43:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1972651:::benchmark_data_smithi188_141483_object46661:head# 2024-04-23T16:43:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1974cf8:::benchmark_data_smithi188_141483_object42107:head# 2024-04-23T16:43:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c197f56e:::benchmark_data_smithi188_141483_object34932:head# 2024-04-23T16:43:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c19887a5:::benchmark_data_smithi188_141483_object3645:head# 2024-04-23T16:43:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1991793:::benchmark_data_smithi188_141483_object18876:head# 2024-04-23T16:43:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c19b83af:::benchmark_data_smithi188_141483_object61803:head# 2024-04-23T16:43:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c19bad05:::benchmark_data_smithi188_141483_object86260:head# 2024-04-23T16:43:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c19caafe:::benchmark_data_smithi188_141483_object89103:head# 2024-04-23T16:43:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c19dcd04:::benchmark_data_smithi188_141483_object46424:head# 2024-04-23T16:43:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c19e8b26:::benchmark_data_smithi188_141483_object36010:head# 2024-04-23T16:43:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c19ef1ea:::benchmark_data_smithi188_141483_object8139:head# 2024-04-23T16:43:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c19f536b:::benchmark_data_smithi188_141483_object21631:head# 2024-04-23T16:43:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c19f91e0:::benchmark_data_smithi188_141483_object50240:head# 2024-04-23T16:43:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c19fa921:::benchmark_data_smithi188_141483_object38501:head# 2024-04-23T16:43:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c19fb6b5:::benchmark_data_smithi188_141483_object76589:head# 2024-04-23T16:43:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c19fd4cd:::benchmark_data_smithi188_141483_object11965:head# 2024-04-23T16:43:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1a00e68:::benchmark_data_smithi188_141483_object10440:head# 2024-04-23T16:43:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1a040aa:::benchmark_data_smithi188_141483_object76283:head# 2024-04-23T16:43:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1a12da3:::benchmark_data_smithi188_141483_object9430:head# 2024-04-23T16:43:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1a21882:::benchmark_data_smithi188_141483_object35058:head# 2024-04-23T16:43:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1a2badc:::benchmark_data_smithi188_141483_object28736:head# 2024-04-23T16:43:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1a618c3:::benchmark_data_smithi188_141483_object12468:head# 2024-04-23T16:43:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1a6f792:::benchmark_data_smithi188_141483_object604:head# 2024-04-23T16:43:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1a7f9a8:::benchmark_data_smithi188_141483_object82098:head# 2024-04-23T16:43:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1a85010:::benchmark_data_smithi188_141483_object89398:head# 2024-04-23T16:43:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1a990eb:::benchmark_data_smithi188_141483_object65744:head# 2024-04-23T16:43:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1a9fb06:::benchmark_data_smithi188_141483_object20311:head# 2024-04-23T16:43:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1aa6326:::benchmark_data_smithi188_141483_object60419:head# 2024-04-23T16:43:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ab1845:::benchmark_data_smithi188_141483_object61530:head# 2024-04-23T16:43:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ab899f:::benchmark_data_smithi188_141483_object32010:head# 2024-04-23T16:43:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1aba843:::benchmark_data_smithi188_141483_object49377:head# 2024-04-23T16:43:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ad557e:::benchmark_data_smithi188_141483_object31244:head# 2024-04-23T16:43:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1adc927:::benchmark_data_smithi188_141483_object11685:head# 2024-04-23T16:43:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ae02a2:::benchmark_data_smithi188_141483_object72746:head# 2024-04-23T16:43:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ae1867:::benchmark_data_smithi188_141483_object50772:head# 2024-04-23T16:43:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ae5295:::benchmark_data_smithi188_141483_object27482:head# 2024-04-23T16:43:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1af1937:::benchmark_data_smithi188_141483_object75947:head# 2024-04-23T16:43:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1af5999:::benchmark_data_smithi188_141483_object67156:head# 2024-04-23T16:43:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1af7171:::benchmark_data_smithi188_141483_object84041:head# 2024-04-23T16:43:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1afd138:::benchmark_data_smithi188_141483_object23232:head# 2024-04-23T16:43:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1b25b07:::benchmark_data_smithi188_141483_object81133:head# 2024-04-23T16:43:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1b2f7b5:::benchmark_data_smithi188_141483_object85362:head# 2024-04-23T16:43:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1b2fb4c:::benchmark_data_smithi188_141483_object84200:head# 2024-04-23T16:43:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1b32aeb:::benchmark_data_smithi188_141483_object87072:head# 2024-04-23T16:43:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1b3a42d:::benchmark_data_smithi188_141483_object27910:head# 2024-04-23T16:43:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1b56580:::benchmark_data_smithi188_141483_object8144:head# 2024-04-23T16:43:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1b5a822:::benchmark_data_smithi188_141483_object9196:head# 2024-04-23T16:43:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1b66896:::benchmark_data_smithi188_141483_object72045:head# 2024-04-23T16:43:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1b87420:::benchmark_data_smithi188_141483_object39606:head# 2024-04-23T16:43:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1b88948:::benchmark_data_smithi188_141483_object79184:head# 2024-04-23T16:43:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1b929c3:::benchmark_data_smithi188_141483_object5438:head# 2024-04-23T16:43:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1b9339c:::benchmark_data_smithi188_141483_object80142:head# 2024-04-23T16:43:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1b98bdf:::benchmark_data_smithi188_141483_object51203:head# 2024-04-23T16:43:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ba2d9f:::benchmark_data_smithi188_141483_object41439:head# 2024-04-23T16:43:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1bb00ce:::benchmark_data_smithi188_141483_object44919:head# 2024-04-23T16:43:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1bb880c:::benchmark_data_smithi188_141483_object47426:head# 2024-04-23T16:43:51.892 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.892 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1bc6440:::benchmark_data_smithi188_141483_object79290:head# 2024-04-23T16:43:51.892 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.892 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1bcf0ff:::benchmark_data_smithi188_141483_object4847:head# 2024-04-23T16:43:51.892 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1bda696:::benchmark_data_smithi188_141483_object62535:head# 2024-04-23T16:43:51.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1be5363:::benchmark_data_smithi188_141483_object2821:head# 2024-04-23T16:43:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1be93d0:::benchmark_data_smithi188_141483_object5865:head# 2024-04-23T16:43:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1be9cd1:::benchmark_data_smithi188_141483_object73554:head# 2024-04-23T16:43:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1bf4b84:::benchmark_data_smithi188_141483_object73523:head# 2024-04-23T16:43:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1c06ddb:::benchmark_data_smithi188_141483_object31946:head# 2024-04-23T16:43:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1c11774:::benchmark_data_smithi188_141483_object11161:head# 2024-04-23T16:43:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1c118b5:::benchmark_data_smithi188_141483_object66714:head# 2024-04-23T16:43:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1c2c830:::benchmark_data_smithi188_141483_object41637:head# 2024-04-23T16:43:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1c4878c:::benchmark_data_smithi188_141483_object7192:head# 2024-04-23T16:43:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1c5b25c:::benchmark_data_smithi188_141483_object24565:head# 2024-04-23T16:43:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1c6bc07:::benchmark_data_smithi188_141483_object84616:head# 2024-04-23T16:43:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1c79fab:::benchmark_data_smithi188_141483_object19412:head# 2024-04-23T16:43:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1c7dbe0:::benchmark_data_smithi188_141483_object2466:head# 2024-04-23T16:43:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1c82ae3:::benchmark_data_smithi188_141483_object84292:head# 2024-04-23T16:43:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1c88f61:::benchmark_data_smithi188_141483_object22966:head# 2024-04-23T16:43:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1c911ab:::benchmark_data_smithi188_141483_object78825:head# 2024-04-23T16:43:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1c9133e:::benchmark_data_smithi188_141483_object28729:head# 2024-04-23T16:43:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1c9730f:::benchmark_data_smithi188_141483_object32321:head# 2024-04-23T16:43:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1c9df6a:::benchmark_data_smithi188_141483_object67340:head# 2024-04-23T16:43:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ca7364:::benchmark_data_smithi188_141483_object20421:head# 2024-04-23T16:43:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ca9a50:::benchmark_data_smithi188_141483_object90170:head# 2024-04-23T16:43:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.899 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1cac71f:::benchmark_data_smithi188_141483_object21822:head# 2024-04-23T16:43:51.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1cb1372:::benchmark_data_smithi188_141483_object9291:head# 2024-04-23T16:43:51.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1cbcd40:::benchmark_data_smithi188_141483_object54340:head# 2024-04-23T16:43:51.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1cc60f9:::benchmark_data_smithi188_141483_object8064:head# 2024-04-23T16:43:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1cc6fd0:::benchmark_data_smithi188_141483_object22461:head# 2024-04-23T16:43:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1cc7c55:::benchmark_data_smithi188_141483_object20661:head# 2024-04-23T16:43:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1cce7fc:::benchmark_data_smithi188_141483_object10198:head# 2024-04-23T16:43:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1cd26c5:::benchmark_data_smithi188_141483_object75117:head# 2024-04-23T16:43:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1cd7e97:::benchmark_data_smithi188_141483_object62852:head# 2024-04-23T16:43:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1cdaee1:::benchmark_data_smithi188_141483_object66990:head# 2024-04-23T16:43:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ce1bbb:::benchmark_data_smithi188_141483_object16802:head# 2024-04-23T16:43:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ce98d4:::benchmark_data_smithi188_141483_object35229:head# 2024-04-23T16:43:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1cf9306:::benchmark_data_smithi188_141483_object89652:head# 2024-04-23T16:43:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1d08307:::benchmark_data_smithi188_141483_object75909:head# 2024-04-23T16:43:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1d1a7d9:::benchmark_data_smithi188_141483_object87028:head# 2024-04-23T16:43:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1d1ab1f:::benchmark_data_smithi188_141483_object62213:head# 2024-04-23T16:43:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1d28930:::benchmark_data_smithi188_141483_object81050:head# 2024-04-23T16:43:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1d43be9:::benchmark_data_smithi188_141483_object43165:head# 2024-04-23T16:43:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1d513cd:::benchmark_data_smithi188_141483_object60170:head# 2024-04-23T16:43:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1d5e42c:::benchmark_data_smithi188_141483_object13608:head# 2024-04-23T16:43:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1d6cec8:::benchmark_data_smithi188_141483_object64387:head# 2024-04-23T16:43:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1d6e2ab:::benchmark_data_smithi188_141483_object14495:head# 2024-04-23T16:43:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1d814c0:::benchmark_data_smithi188_141483_object72241:head# 2024-04-23T16:43:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1d8259e:::benchmark_data_smithi188_141483_object44044:head# 2024-04-23T16:43:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1d8ea10:::benchmark_data_smithi188_141483_object50131:head# 2024-04-23T16:43:51.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1da01f7:::benchmark_data_smithi188_141483_object78661:head# 2024-04-23T16:43:51.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1db2d30:::benchmark_data_smithi188_141483_object77124:head# 2024-04-23T16:43:51.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1dbbd28:::benchmark_data_smithi188_141483_object78550:head# 2024-04-23T16:43:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1dbd66c:::benchmark_data_smithi188_141483_object90349:head# 2024-04-23T16:43:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1dc1bdb:::benchmark_data_smithi188_141483_object49387:head# 2024-04-23T16:43:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1de5255:::benchmark_data_smithi188_141483_object74037:head# 2024-04-23T16:43:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1de65f3:::benchmark_data_smithi188_141483_object84481:head# 2024-04-23T16:43:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1dfb196:::benchmark_data_smithi188_141483_object31272:head# 2024-04-23T16:43:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1dfd3b0:::benchmark_data_smithi188_141483_object83774:head# 2024-04-23T16:43:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1e044fd:::benchmark_data_smithi188_141483_object79401:head# 2024-04-23T16:43:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1e2bacd:::benchmark_data_smithi188_141483_object52554:head# 2024-04-23T16:43:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1e3e12e:::benchmark_data_smithi188_141483_object43919:head# 2024-04-23T16:43:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1e49781:::benchmark_data_smithi188_141483_object38730:head# 2024-04-23T16:43:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1e5b10f:::benchmark_data_smithi188_141483_object1489:head# 2024-04-23T16:43:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1e6c964:::benchmark_data_smithi188_141483_object24479:head# 2024-04-23T16:43:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1e6cfc5:::benchmark_data_smithi188_141483_object5496:head# 2024-04-23T16:43:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1e86086:::benchmark_data_smithi188_141483_object55675:head# 2024-04-23T16:43:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1e8be6b:::benchmark_data_smithi188_141483_object88726:head# 2024-04-23T16:43:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1e8fee3:::benchmark_data_smithi188_141483_object41653:head# 2024-04-23T16:43:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1e9b7d6:::benchmark_data_smithi188_141483_object23114:head# 2024-04-23T16:43:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1e9f5d3:::benchmark_data_smithi188_141483_object23643:head# 2024-04-23T16:43:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1e9ff23:::benchmark_data_smithi188_141483_object54247:head# 2024-04-23T16:43:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1eb0a7e:::benchmark_data_smithi188_141483_object67166:head# 2024-04-23T16:43:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1eb44bd:::benchmark_data_smithi188_141483_object79277:head# 2024-04-23T16:43:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1eb5470:::benchmark_data_smithi188_141483_object34239:head# 2024-04-23T16:43:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ebfcda:::benchmark_data_smithi188_141483_object78236:head# 2024-04-23T16:43:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ec3134:::benchmark_data_smithi188_141483_object8695:head# 2024-04-23T16:43:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ec313f:::benchmark_data_smithi188_141483_object80995:head# 2024-04-23T16:43:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ec4149:::benchmark_data_smithi188_141483_object58193:head# 2024-04-23T16:43:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.914 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1eedb97:::benchmark_data_smithi188_141483_object83689:head# 2024-04-23T16:43:51.914 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.914 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1eedde8:::benchmark_data_smithi188_141483_object13284:head# 2024-04-23T16:43:51.914 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1eee471:::benchmark_data_smithi188_141483_object59877:head# 2024-04-23T16:43:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ef1355:::benchmark_data_smithi188_141483_object60427:head# 2024-04-23T16:43:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ef466b:::benchmark_data_smithi188_141483_object21724:head# 2024-04-23T16:43:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1f07544:::benchmark_data_smithi188_141483_object29268:head# 2024-04-23T16:43:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1f0984e:::benchmark_data_smithi188_141483_object86558:head# 2024-04-23T16:43:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1f20ff1:::benchmark_data_smithi188_141483_object83271:head# 2024-04-23T16:43:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1f22c57:::benchmark_data_smithi188_141483_object7056:head# 2024-04-23T16:43:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1f33b45:::benchmark_data_smithi188_141483_object33805:head# 2024-04-23T16:43:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1f509af:::benchmark_data_smithi188_141483_object88580:head# 2024-04-23T16:43:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1f6894b:::benchmark_data_smithi188_141483_object32306:head# 2024-04-23T16:43:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1f76273:::benchmark_data_smithi188_141483_object25182:head# 2024-04-23T16:43:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1f78063:::benchmark_data_smithi188_141483_object19147:head# 2024-04-23T16:43:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1f7a72a:::benchmark_data_smithi188_141483_object13193:head# 2024-04-23T16:43:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1f87e40:::benchmark_data_smithi188_141483_object3922:head# 2024-04-23T16:43:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1f8ccc2:::benchmark_data_smithi188_141483_object86331:head# 2024-04-23T16:43:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1f90442:::benchmark_data_smithi188_141483_object66515:head# 2024-04-23T16:43:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1f93b1a:::benchmark_data_smithi188_141483_object11138:head# 2024-04-23T16:43:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1f9624f:::benchmark_data_smithi188_141483_object23848:head# 2024-04-23T16:43:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1f9c32e:::benchmark_data_smithi188_141483_object90424:head# 2024-04-23T16:43:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1fbe5b2:::benchmark_data_smithi188_141483_object84003:head# 2024-04-23T16:43:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1fc06ed:::benchmark_data_smithi188_141483_object1415:head# 2024-04-23T16:43:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1fc74fa:::benchmark_data_smithi188_141483_object51904:head# 2024-04-23T16:43:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1fccb74:::benchmark_data_smithi188_141483_object84855:head# 2024-04-23T16:43:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1fd09cd:::benchmark_data_smithi188_141483_object27850:head# 2024-04-23T16:43:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1fd63fe:::benchmark_data_smithi188_141483_object34819:head# 2024-04-23T16:43:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1fd6d34:::benchmark_data_smithi188_141483_object80039:head# 2024-04-23T16:43:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1fd8ecb:::benchmark_data_smithi188_141483_object47422:head# 2024-04-23T16:43:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1fe08a3:::benchmark_data_smithi188_141483_object45100:head# 2024-04-23T16:43:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ff254f:::benchmark_data_smithi188_141483_object8069:head# 2024-04-23T16:43:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ff2fed:::benchmark_data_smithi188_141483_object90446:head# 2024-04-23T16:43:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c1ff74f5:::benchmark_data_smithi188_141483_object2567:head# 2024-04-23T16:43:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2007d59:::benchmark_data_smithi188_141483_object36875:head# 2024-04-23T16:43:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2025d59:::benchmark_data_smithi188_141483_object90282:head# 2024-04-23T16:43:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c202f063:::benchmark_data_smithi188_141483_object20914:head# 2024-04-23T16:43:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c20312ec:::benchmark_data_smithi188_141483_object78566:head# 2024-04-23T16:43:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2032baf:::benchmark_data_smithi188_141483_object2836:head# 2024-04-23T16:43:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2040587:::benchmark_data_smithi188_141483_object12310:head# 2024-04-23T16:43:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c20594be:::benchmark_data_smithi188_141483_object89023:head# 2024-04-23T16:43:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c205b2d5:::benchmark_data_smithi188_141483_object82868:head# 2024-04-23T16:43:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c20624d9:::benchmark_data_smithi188_141483_object6743:head# 2024-04-23T16:43:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2065729:::benchmark_data_smithi188_141483_object23596:head# 2024-04-23T16:43:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c206d9e9:::benchmark_data_smithi188_141483_object20156:head# 2024-04-23T16:43:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c208863e:::benchmark_data_smithi188_141483_object83392:head# 2024-04-23T16:43:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c208aac4:::benchmark_data_smithi188_141483_object17674:head# 2024-04-23T16:43:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2094419:::benchmark_data_smithi188_141483_object323:head# 2024-04-23T16:43:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c209b317:::benchmark_data_smithi188_141483_object55088:head# 2024-04-23T16:43:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c209bfa1:::benchmark_data_smithi188_141483_object25613:head# 2024-04-23T16:43:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c20a3e73:::benchmark_data_smithi188_141483_object34498:head# 2024-04-23T16:43:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c20b32f1:::benchmark_data_smithi188_141483_object90554:head# 2024-04-23T16:43:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c20b58e2:::benchmark_data_smithi188_141483_object16138:head# 2024-04-23T16:43:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c20bb296:::benchmark_data_smithi188_141483_object75396:head# 2024-04-23T16:43:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c20e5d4d:::benchmark_data_smithi188_141483_object87896:head# 2024-04-23T16:43:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c20fbe73:::benchmark_data_smithi188_141483_object75023:head# 2024-04-23T16:43:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2103074:::benchmark_data_smithi188_141483_object67065:head# 2024-04-23T16:43:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c214a30d:::benchmark_data_smithi188_141483_object65904:head# 2024-04-23T16:43:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2154164:::benchmark_data_smithi188_141483_object26506:head# 2024-04-23T16:43:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2163f3b:::benchmark_data_smithi188_141483_object73083:head# 2024-04-23T16:43:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2165540:::benchmark_data_smithi188_141483_object40036:head# 2024-04-23T16:43:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c216651b:::benchmark_data_smithi188_141483_object37996:head# 2024-04-23T16:43:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2184987:::benchmark_data_smithi188_141483_object4532:head# 2024-04-23T16:43:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c218a3af:::benchmark_data_smithi188_141483_object69302:head# 2024-04-23T16:43:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c218b3de:::benchmark_data_smithi188_141483_object50437:head# 2024-04-23T16:43:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c219c377:::benchmark_data_smithi188_141483_object12744:head# 2024-04-23T16:43:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c219d2a0:::benchmark_data_smithi188_141483_object33078:head# 2024-04-23T16:43:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c21a625d:::benchmark_data_smithi188_141483_object41057:head# 2024-04-23T16:43:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c21b24b4:::benchmark_data_smithi188_141483_object12458:head# 2024-04-23T16:43:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c21c42be:::benchmark_data_smithi188_141483_object38363:head# 2024-04-23T16:43:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c21deb2d:::benchmark_data_smithi188_141483_object49700:head# 2024-04-23T16:43:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c21e0921:::benchmark_data_smithi188_141483_object24552:head# 2024-04-23T16:43:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c21e5f1b:::benchmark_data_smithi188_141483_object36520:head# 2024-04-23T16:43:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c21ff243:::benchmark_data_smithi188_141483_object54232:head# 2024-04-23T16:43:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c220df18:::benchmark_data_smithi188_141483_object18598:head# 2024-04-23T16:43:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c22188a3:::benchmark_data_smithi188_141483_object40612:head# 2024-04-23T16:43:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c221b3e3:::benchmark_data_smithi188_141483_object6471:head# 2024-04-23T16:43:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2240fbb:::benchmark_data_smithi188_141483_object13445:head# 2024-04-23T16:43:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2247288:::benchmark_data_smithi188_141483_object46741:head# 2024-04-23T16:43:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2261b1e:::benchmark_data_smithi188_141483_object46260:head# 2024-04-23T16:43:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2261b87:::benchmark_data_smithi188_141483_object32542:head# 2024-04-23T16:43:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c226810c:::benchmark_data_smithi188_141483_object77491:head# 2024-04-23T16:43:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c226dbec:::benchmark_data_smithi188_141483_object75567:head# 2024-04-23T16:43:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c226e422:::benchmark_data_smithi188_141483_object78767:head# 2024-04-23T16:43:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c228f8c4:::benchmark_data_smithi188_141483_object53030:head# 2024-04-23T16:43:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c22904d9:::benchmark_data_smithi188_141483_object55992:head# 2024-04-23T16:43:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c22a5f6b:::benchmark_data_smithi188_141483_object27103:head# 2024-04-23T16:43:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c22abf90:::benchmark_data_smithi188_141483_object41476:head# 2024-04-23T16:43:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c22aff87:::benchmark_data_smithi188_141483_object14905:head# 2024-04-23T16:43:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c22ba3e6:::benchmark_data_smithi188_141483_object52922:head# 2024-04-23T16:43:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c22c2414:::benchmark_data_smithi188_141483_object12970:head# 2024-04-23T16:43:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c22c6f2d:::benchmark_data_smithi188_141483_object80846:head# 2024-04-23T16:43:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c22cb842:::benchmark_data_smithi188_141483_object51128:head# 2024-04-23T16:43:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c22e3cc9:::benchmark_data_smithi188_141483_object3880:head# 2024-04-23T16:43:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c22f3d1e:::benchmark_data_smithi188_141483_object72992:head# 2024-04-23T16:43:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c22f756f:::benchmark_data_smithi188_141483_object34364:head# 2024-04-23T16:43:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c22fbc1d:::benchmark_data_smithi188_141483_object46531:head# 2024-04-23T16:43:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c22fbd43:::benchmark_data_smithi188_141483_object43474:head# 2024-04-23T16:43:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c22fc710:::benchmark_data_smithi188_141483_object56995:head# 2024-04-23T16:43:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2303909:::benchmark_data_smithi188_141483_object88625:head# 2024-04-23T16:43:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2304991:::benchmark_data_smithi188_141483_object55955:head# 2024-04-23T16:43:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c230908d:::benchmark_data_smithi188_141483_object58871:head# 2024-04-23T16:43:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2309b63:::benchmark_data_smithi188_141483_object25101:head# 2024-04-23T16:43:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2314981:::benchmark_data_smithi188_141483_object37387:head# 2024-04-23T16:43:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2323377:::benchmark_data_smithi188_141483_object39893:head# 2024-04-23T16:43:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2325ca3:::benchmark_data_smithi188_141483_object8482:head# 2024-04-23T16:43:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c233b4d2:::benchmark_data_smithi188_141483_object21686:head# 2024-04-23T16:43:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c233cb59:::benchmark_data_smithi188_141483_object70311:head# 2024-04-23T16:43:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2343531:::benchmark_data_smithi188_141483_object782:head# 2024-04-23T16:43:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c235340f:::benchmark_data_smithi188_141483_object35754:head# 2024-04-23T16:43:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2355f1b:::benchmark_data_smithi188_141483_object84903:head# 2024-04-23T16:43:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2365067:::benchmark_data_smithi188_141483_object33837:head# 2024-04-23T16:43:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2378d43:::benchmark_data_smithi188_141483_object86058:head# 2024-04-23T16:43:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c23809bd:::benchmark_data_smithi188_141483_object83365:head# 2024-04-23T16:43:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2389289:::benchmark_data_smithi188_141483_object23802:head# 2024-04-23T16:43:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c238c540:::benchmark_data_smithi188_141483_object70786:head# 2024-04-23T16:43:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c23949d1:::benchmark_data_smithi188_141483_object87847:head# 2024-04-23T16:43:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c23c03a9:::benchmark_data_smithi188_141483_object83749:head# 2024-04-23T16:43:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c23c4277:::benchmark_data_smithi188_141483_object67995:head# 2024-04-23T16:43:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c23d5857:::benchmark_data_smithi188_141483_object29456:head# 2024-04-23T16:43:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c23e3484:::benchmark_data_smithi188_141483_object72346:head# 2024-04-23T16:43:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c23efee4:::benchmark_data_smithi188_141483_object58303:head# 2024-04-23T16:43:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c23f9cd0:::benchmark_data_smithi188_141483_object81640:head# 2024-04-23T16:43:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c23fb30c:::benchmark_data_smithi188_141483_object79969:head# 2024-04-23T16:43:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2410071:::benchmark_data_smithi188_141483_object90644:head# 2024-04-23T16:43:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2410e37:::benchmark_data_smithi188_141483_object66835:head# 2024-04-23T16:43:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2420857:::benchmark_data_smithi188_141483_object21460:head# 2024-04-23T16:43:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2429788:::benchmark_data_smithi188_141483_object60266:head# 2024-04-23T16:43:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c242ee05:::benchmark_data_smithi188_141483_object37950:head# 2024-04-23T16:43:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2435724:::benchmark_data_smithi188_141483_object73281:head# 2024-04-23T16:43:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c24357bd:::benchmark_data_smithi188_141483_object52578:head# 2024-04-23T16:43:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2440089:::benchmark_data_smithi188_141483_object30871:head# 2024-04-23T16:43:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2440d43:::benchmark_data_smithi188_141483_object45854:head# 2024-04-23T16:43:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2440e8f:::benchmark_data_smithi188_141483_object80956:head# 2024-04-23T16:43:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2441044:::benchmark_data_smithi188_141483_object14107:head# 2024-04-23T16:43:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2452cbf:::benchmark_data_smithi188_141483_object30113:head# 2024-04-23T16:43:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2459f66:::benchmark_data_smithi188_141483_object41924:head# 2024-04-23T16:43:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c245a3c8:::benchmark_data_smithi188_141483_object79380:head# 2024-04-23T16:43:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2463141:::benchmark_data_smithi188_141483_object66697:head# 2024-04-23T16:43:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c24670c5:::benchmark_data_smithi188_141483_object68577:head# 2024-04-23T16:43:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c246d00c:::benchmark_data_smithi188_141483_object1995:head# 2024-04-23T16:43:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c24839ab:::benchmark_data_smithi188_141483_object20639:head# 2024-04-23T16:43:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c24845b1:::benchmark_data_smithi188_141483_object34986:head# 2024-04-23T16:43:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c248b803:::benchmark_data_smithi188_141483_object16017:head# 2024-04-23T16:43:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c248fab8:::benchmark_data_smithi188_141483_object11375:head# 2024-04-23T16:43:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2498d9b:::benchmark_data_smithi188_141483_object3157:head# 2024-04-23T16:43:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c24a3180:::benchmark_data_smithi188_141483_object4580:head# 2024-04-23T16:43:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c24a5bc8:::benchmark_data_smithi188_141483_object86641:head# 2024-04-23T16:43:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c24b89bb:::benchmark_data_smithi188_141483_object48260:head# 2024-04-23T16:43:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c24c27c8:::benchmark_data_smithi188_141483_object78279:head# 2024-04-23T16:43:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c24c699b:::benchmark_data_smithi188_141483_object56659:head# 2024-04-23T16:43:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c24cee3f:::benchmark_data_smithi188_141483_object14782:head# 2024-04-23T16:43:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c24d041b:::benchmark_data_smithi188_141483_object39604:head# 2024-04-23T16:43:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c24d967a:::benchmark_data_smithi188_141483_object41625:head# 2024-04-23T16:43:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c24e2dc7:::benchmark_data_smithi188_141483_object31665:head# 2024-04-23T16:43:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c24e341a:::benchmark_data_smithi188_141483_object77759:head# 2024-04-23T16:43:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c24eb17e:::benchmark_data_smithi188_141483_object16645:head# 2024-04-23T16:43:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c250ca60:::benchmark_data_smithi188_141483_object7392:head# 2024-04-23T16:43:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c252cfb1:::benchmark_data_smithi188_141483_object43090:head# 2024-04-23T16:43:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2530c51:::benchmark_data_smithi188_141483_object87608:head# 2024-04-23T16:43:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2536e60:::benchmark_data_smithi188_141483_object11842:head# 2024-04-23T16:43:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c254b18c:::benchmark_data_smithi188_141483_object31926:head# 2024-04-23T16:43:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2553e35:::benchmark_data_smithi188_141483_object33870:head# 2024-04-23T16:43:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c25543db:::benchmark_data_smithi188_141483_object3533:head# 2024-04-23T16:43:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2564ba8:::benchmark_data_smithi188_141483_object84512:head# 2024-04-23T16:43:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c256cd74:::benchmark_data_smithi188_141483_object56133:head# 2024-04-23T16:43:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2597c67:::benchmark_data_smithi188_141483_object37815:head# 2024-04-23T16:43:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c259aac9:::benchmark_data_smithi188_141483_object8119:head# 2024-04-23T16:43:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c259b334:::benchmark_data_smithi188_141483_object45582:head# 2024-04-23T16:43:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c25cd88c:::benchmark_data_smithi188_141483_object75662:head# 2024-04-23T16:43:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c25e388f:::benchmark_data_smithi188_141483_object38841:head# 2024-04-23T16:43:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c25eaa10:::benchmark_data_smithi188_141483_object64006:head# 2024-04-23T16:43:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c25f2313:::benchmark_data_smithi188_141483_object39283:head# 2024-04-23T16:43:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c25f8488:::benchmark_data_smithi188_141483_object4770:head# 2024-04-23T16:43:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2610d01:::benchmark_data_smithi188_141483_object39017:head# 2024-04-23T16:43:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2617e1f:::benchmark_data_smithi188_141483_object12005:head# 2024-04-23T16:43:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2618689:::benchmark_data_smithi188_141483_object25894:head# 2024-04-23T16:43:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c261e619:::benchmark_data_smithi188_141483_object54409:head# 2024-04-23T16:43:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2621db0:::benchmark_data_smithi188_141483_object48449:head# 2024-04-23T16:43:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c262ce01:::benchmark_data_smithi188_141483_object19089:head# 2024-04-23T16:43:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c262f67e:::benchmark_data_smithi188_141483_object63774:head# 2024-04-23T16:43:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2632925:::benchmark_data_smithi188_141483_object49429:head# 2024-04-23T16:43:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c26555f3:::benchmark_data_smithi188_141483_object13548:head# 2024-04-23T16:43:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2658ce3:::benchmark_data_smithi188_141483_object53079:head# 2024-04-23T16:43:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c265b9ad:::benchmark_data_smithi188_141483_object25648:head# 2024-04-23T16:43:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2664009:::benchmark_data_smithi188_141483_object73076:head# 2024-04-23T16:43:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c267d6a3:::benchmark_data_smithi188_141483_object18735:head# 2024-04-23T16:43:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2693df3:::benchmark_data_smithi188_141483_object20808:head# 2024-04-23T16:43:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c26a4941:::benchmark_data_smithi188_141483_object88568:head# 2024-04-23T16:43:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c26a570e:::benchmark_data_smithi188_141483_object85663:head# 2024-04-23T16:43:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c26d009b:::benchmark_data_smithi188_141483_object49594:head# 2024-04-23T16:43:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c26d0cf0:::benchmark_data_smithi188_141483_object66969:head# 2024-04-23T16:43:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c26d171c:::benchmark_data_smithi188_141483_object84926:head# 2024-04-23T16:43:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c26d5e9b:::benchmark_data_smithi188_141483_object9550:head# 2024-04-23T16:43:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c26da2a5:::benchmark_data_smithi188_141483_object73540:head# 2024-04-23T16:43:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c26f43b3:::benchmark_data_smithi188_141483_object4693:head# 2024-04-23T16:43:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c26f58ac:::benchmark_data_smithi188_141483_object90735:head# 2024-04-23T16:43:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c26fff12:::benchmark_data_smithi188_141483_object76853:head# 2024-04-23T16:43:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c270e816:::benchmark_data_smithi188_141483_object995:head# 2024-04-23T16:43:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c271cbcf:::benchmark_data_smithi188_141483_object38139:head# 2024-04-23T16:43:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c272f75c:::benchmark_data_smithi188_141483_object54407:head# 2024-04-23T16:43:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2731899:::benchmark_data_smithi188_141483_object58805:head# 2024-04-23T16:43:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c275298b:::benchmark_data_smithi188_141483_object37081:head# 2024-04-23T16:43:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c275b85a:::benchmark_data_smithi188_141483_object4891:head# 2024-04-23T16:43:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2769340:::benchmark_data_smithi188_141483_object1106:head# 2024-04-23T16:43:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c276d3e1:::benchmark_data_smithi188_141483_object21250:head# 2024-04-23T16:43:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2770cde:::benchmark_data_smithi188_141483_object10990:head# 2024-04-23T16:43:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c27711e2:::benchmark_data_smithi188_141483_object18915:head# 2024-04-23T16:43:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c278a796:::benchmark_data_smithi188_141483_object6597:head# 2024-04-23T16:43:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c279cfe2:::benchmark_data_smithi188_141483_object50591:head# 2024-04-23T16:43:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c27ad5a2:::benchmark_data_smithi188_141483_object17201:head# 2024-04-23T16:43:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c27b3264:::benchmark_data_smithi188_141483_object29158:head# 2024-04-23T16:43:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c27b8873:::benchmark_data_smithi188_141483_object50398:head# 2024-04-23T16:43:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c27c1ebc:::benchmark_data_smithi188_141483_object50328:head# 2024-04-23T16:43:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c27cb739:::benchmark_data_smithi188_141483_object29620:head# 2024-04-23T16:43:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c27cffb2:::benchmark_data_smithi188_141483_object50171:head# 2024-04-23T16:43:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.006 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c27da79a:::benchmark_data_smithi188_141483_object28866:head# 2024-04-23T16:43:52.006 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.006 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c27e6621:::benchmark_data_smithi188_141483_object67993:head# 2024-04-23T16:43:52.006 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.006 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c27eb0fa:::benchmark_data_smithi188_141483_object16557:head# 2024-04-23T16:43:52.007 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.007 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c27f0689:::benchmark_data_smithi188_141483_object26870:head# 2024-04-23T16:43:52.007 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.007 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c27f07ea:::benchmark_data_smithi188_141483_object4321:head# 2024-04-23T16:43:52.007 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.007 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c280466d:::benchmark_data_smithi188_141483_object7915:head# 2024-04-23T16:43:52.007 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.007 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c280ddd5:::benchmark_data_smithi188_141483_object46892:head# 2024-04-23T16:43:52.007 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.007 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2810035:::benchmark_data_smithi188_141483_object48968:head# 2024-04-23T16:43:52.007 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.007 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c283c656:::benchmark_data_smithi188_141483_object22319:head# 2024-04-23T16:43:52.007 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.007 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c285345e:::benchmark_data_smithi188_141483_object35521:head# 2024-04-23T16:43:52.008 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.008 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c285846f:::benchmark_data_smithi188_141483_object6889:head# 2024-04-23T16:43:52.008 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.008 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c285b1e4:::benchmark_data_smithi188_141483_object39484:head# 2024-04-23T16:43:52.008 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.008 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2885b7a:::benchmark_data_smithi188_141483_object49986:head# 2024-04-23T16:43:52.008 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.008 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c288dc64:::benchmark_data_smithi188_141483_object49060:head# 2024-04-23T16:43:52.008 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.008 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c289ca12:::benchmark_data_smithi188_141483_object50552:head# 2024-04-23T16:43:52.008 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.008 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c289f4e5:::benchmark_data_smithi188_141483_object9602:head# 2024-04-23T16:43:52.008 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.008 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c28a4de5:::benchmark_data_smithi188_141483_object82184:head# 2024-04-23T16:43:52.009 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.009 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c28aad13:::benchmark_data_smithi188_141483_object38648:head# 2024-04-23T16:43:52.009 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.009 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c28b96de:::benchmark_data_smithi188_141483_object67243:head# 2024-04-23T16:43:52.009 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.009 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c28cb9e6:::benchmark_data_smithi188_141483_object6797:head# 2024-04-23T16:43:52.009 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.009 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c28cd325:::benchmark_data_smithi188_141483_object4521:head# 2024-04-23T16:43:52.009 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.009 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c28d0aa5:::benchmark_data_smithi188_141483_object27409:head# 2024-04-23T16:43:52.009 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.009 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c28dd6cf:::benchmark_data_smithi188_141483_object28104:head# 2024-04-23T16:43:52.009 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.010 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c28e8e0c:::benchmark_data_smithi188_141483_object46557:head# 2024-04-23T16:43:52.010 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.010 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c28ef9d0:::benchmark_data_smithi188_141483_object78901:head# 2024-04-23T16:43:52.010 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.010 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c28f3816:::benchmark_data_smithi188_141483_object69090:head# 2024-04-23T16:43:52.010 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.010 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c290eae0:::benchmark_data_smithi188_141483_object43161:head# 2024-04-23T16:43:52.010 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.010 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c290ff4c:::benchmark_data_smithi188_141483_object29791:head# 2024-04-23T16:43:52.010 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.010 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2914353:::benchmark_data_smithi188_141483_object15070:head# 2024-04-23T16:43:52.010 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.010 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2928ed8:::benchmark_data_smithi188_141483_object6907:head# 2024-04-23T16:43:52.011 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.011 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c29331e6:::benchmark_data_smithi188_141483_object8232:head# 2024-04-23T16:43:52.011 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.011 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c293a0a9:::benchmark_data_smithi188_141483_object1360:head# 2024-04-23T16:43:52.011 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.011 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c29406ee:::benchmark_data_smithi188_141483_object68144:head# 2024-04-23T16:43:52.011 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.011 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c29413b2:::benchmark_data_smithi188_141483_object35223:head# 2024-04-23T16:43:52.011 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.011 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2942d85:::benchmark_data_smithi188_141483_object60936:head# 2024-04-23T16:43:52.011 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.011 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2946eca:::benchmark_data_smithi188_141483_object65773:head# 2024-04-23T16:43:52.011 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.011 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c294bac4:::benchmark_data_smithi188_141483_object44909:head# 2024-04-23T16:43:52.012 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.012 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2966e40:::benchmark_data_smithi188_141483_object58181:head# 2024-04-23T16:43:52.012 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.012 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2996314:::benchmark_data_smithi188_141483_object57033:head# 2024-04-23T16:43:52.012 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.012 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c29c3db9:::benchmark_data_smithi188_141483_object12422:head# 2024-04-23T16:43:52.012 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.012 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c29c6cc8:::benchmark_data_smithi188_141483_object27488:head# 2024-04-23T16:43:52.012 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.012 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c29c8285:::benchmark_data_smithi188_141483_object23069:head# 2024-04-23T16:43:52.012 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.012 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c29ca852:::benchmark_data_smithi188_141483_object14945:head# 2024-04-23T16:43:52.012 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.013 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c29cce7c:::benchmark_data_smithi188_141483_object60259:head# 2024-04-23T16:43:52.013 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.013 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c29cf8c7:::benchmark_data_smithi188_141483_object39369:head# 2024-04-23T16:43:52.013 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.013 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c29d2bd4:::benchmark_data_smithi188_141483_object37716:head# 2024-04-23T16:43:52.013 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.013 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c29d5d1a:::benchmark_data_smithi188_141483_object11167:head# 2024-04-23T16:43:52.013 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.013 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2a147a6:::benchmark_data_smithi188_141483_object51640:head# 2024-04-23T16:43:52.013 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.013 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2a2b041:::benchmark_data_smithi188_141483_object46539:head# 2024-04-23T16:43:52.013 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.013 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2a3960b:::benchmark_data_smithi188_141483_object35136:head# 2024-04-23T16:43:52.013 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.014 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2a503ac:::benchmark_data_smithi188_141483_object60091:head# 2024-04-23T16:43:52.014 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.014 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2a618d3:::benchmark_data_smithi188_141483_object44915:head# 2024-04-23T16:43:52.014 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.014 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2a78b5f:::benchmark_data_smithi188_141483_object86324:head# 2024-04-23T16:43:52.014 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.014 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2a7afe2:::benchmark_data_smithi188_141483_object28257:head# 2024-04-23T16:43:52.014 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.014 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2aa5329:::benchmark_data_smithi188_141483_object83253:head# 2024-04-23T16:43:52.014 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.014 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2aa74f3:::benchmark_data_smithi188_141483_object72603:head# 2024-04-23T16:43:52.014 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.014 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2ab8a46:::benchmark_data_smithi188_141483_object56904:head# 2024-04-23T16:43:52.014 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2ab92db:::benchmark_data_smithi188_141483_object63810:head# 2024-04-23T16:43:52.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2abc893:::benchmark_data_smithi188_141483_object1201:head# 2024-04-23T16:43:52.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2abe566:::benchmark_data_smithi188_141483_object3006:head# 2024-04-23T16:43:52.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2ac58c3:::benchmark_data_smithi188_141483_object4243:head# 2024-04-23T16:43:52.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2ac771a:::benchmark_data_smithi188_141483_object9843:head# 2024-04-23T16:43:52.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2acfea2:::benchmark_data_smithi188_141483_object76321:head# 2024-04-23T16:43:52.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.015 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2adaa0a:::benchmark_data_smithi188_141483_object14327:head# 2024-04-23T16:43:52.015 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2aefa4a:::benchmark_data_smithi188_141483_object42231:head# 2024-04-23T16:43:52.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2af9c7a:::benchmark_data_smithi188_141483_object48672:head# 2024-04-23T16:43:52.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2afcba4:::benchmark_data_smithi188_141483_object73988:head# 2024-04-23T16:43:52.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2b0b695:::benchmark_data_smithi188_141483_object34793:head# 2024-04-23T16:43:52.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2b23289:::benchmark_data_smithi188_141483_object64167:head# 2024-04-23T16:43:52.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2b2b544:::benchmark_data_smithi188_141483_object82237:head# 2024-04-23T16:43:52.016 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.016 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2b3959a:::benchmark_data_smithi188_141483_object61507:head# 2024-04-23T16:43:52.017 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.017 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2b457cf:::benchmark_data_smithi188_141483_object85039:head# 2024-04-23T16:43:52.017 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.017 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2b549f1:::benchmark_data_smithi188_141483_object34570:head# 2024-04-23T16:43:52.017 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.017 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2b571e9:::benchmark_data_smithi188_141483_object60403:head# 2024-04-23T16:43:52.017 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.017 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2b5fad6:::benchmark_data_smithi188_141483_object6003:head# 2024-04-23T16:43:52.017 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.017 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2b6ce4d:::benchmark_data_smithi188_141483_object7961:head# 2024-04-23T16:43:52.017 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.017 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2b74e12:::benchmark_data_smithi188_141483_object81126:head# 2024-04-23T16:43:52.017 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.017 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2b75cea:::benchmark_data_smithi188_141483_object48301:head# 2024-04-23T16:43:52.018 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.018 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2b83195:::benchmark_data_smithi188_141483_object73751:head# 2024-04-23T16:43:52.018 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.018 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2b85784:::benchmark_data_smithi188_141483_object37994:head# 2024-04-23T16:43:52.018 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.018 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2b99c58:::benchmark_data_smithi188_141483_object33932:head# 2024-04-23T16:43:52.018 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.018 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2b9f6dd:::benchmark_data_smithi188_141483_object80320:head# 2024-04-23T16:43:52.018 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.018 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2baec1c:::benchmark_data_smithi188_141483_object26985:head# 2024-04-23T16:43:52.018 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.018 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2bb2b80:::benchmark_data_smithi188_141483_object33342:head# 2024-04-23T16:43:52.018 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.018 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2bba694:::benchmark_data_smithi188_141483_object60677:head# 2024-04-23T16:43:52.019 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.019 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2bcee68:::benchmark_data_smithi188_141483_object58439:head# 2024-04-23T16:43:52.019 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.019 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2be7989:::benchmark_data_smithi188_141483_object71550:head# 2024-04-23T16:43:52.019 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.019 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2beef3b:::benchmark_data_smithi188_141483_object9437:head# 2024-04-23T16:43:52.019 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.019 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2bf857d:::benchmark_data_smithi188_141483_object66205:head# 2024-04-23T16:43:52.019 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.019 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2c37270:::benchmark_data_smithi188_141483_object30818:head# 2024-04-23T16:43:52.019 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.019 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2c440f3:::benchmark_data_smithi188_141483_object56438:head# 2024-04-23T16:43:52.019 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.019 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2c5c400:::benchmark_data_smithi188_141483_object64615:head# 2024-04-23T16:43:52.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2c637a5:::benchmark_data_smithi188_141483_object88196:head# 2024-04-23T16:43:52.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2c66a97:::benchmark_data_smithi188_141483_object34816:head# 2024-04-23T16:43:52.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2c69d7c:::benchmark_data_smithi188_141483_object62545:head# 2024-04-23T16:43:52.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2c7b946:::benchmark_data_smithi188_141483_object49110:head# 2024-04-23T16:43:52.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2c7c3cb:::benchmark_data_smithi188_141483_object32301:head# 2024-04-23T16:43:52.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2ca5c1c:::benchmark_data_smithi188_141483_object34965:head# 2024-04-23T16:43:52.020 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.020 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2ca7694:::benchmark_data_smithi188_141483_object62266:head# 2024-04-23T16:43:52.021 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2cacebf:::benchmark_data_smithi188_141483_object34250:head# 2024-04-23T16:43:52.021 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2cafe21:::benchmark_data_smithi188_141483_object31918:head# 2024-04-23T16:43:52.021 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2cc2b61:::benchmark_data_smithi188_141483_object73634:head# 2024-04-23T16:43:52.021 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2cc53cf:::benchmark_data_smithi188_141483_object7752:head# 2024-04-23T16:43:52.021 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2cd738c:::benchmark_data_smithi188_141483_object29585:head# 2024-04-23T16:43:52.021 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2cddfc7:::benchmark_data_smithi188_141483_object88100:head# 2024-04-23T16:43:52.021 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.021 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2ce3d25:::benchmark_data_smithi188_141483_object30632:head# 2024-04-23T16:43:52.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2cf1fc9:::benchmark_data_smithi188_141483_object53760:head# 2024-04-23T16:43:52.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2d1b796:::benchmark_data_smithi188_141483_object19831:head# 2024-04-23T16:43:52.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2d1ca81:::benchmark_data_smithi188_141483_object21485:head# 2024-04-23T16:43:52.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2d299e3:::benchmark_data_smithi188_141483_object26040:head# 2024-04-23T16:43:52.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2d2cd0d:::benchmark_data_smithi188_141483_object53540:head# 2024-04-23T16:43:52.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.022 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2d3b833:::benchmark_data_smithi188_141483_object11599:head# 2024-04-23T16:43:52.022 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2d492a1:::benchmark_data_smithi188_141483_object44876:head# 2024-04-23T16:43:52.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2d49501:::benchmark_data_smithi188_141483_object17071:head# 2024-04-23T16:43:52.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2d4ab4f:::benchmark_data_smithi188_141483_object23155:head# 2024-04-23T16:43:52.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2d59fe3:::benchmark_data_smithi188_141483_object87919:head# 2024-04-23T16:43:52.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2d5c1b5:::benchmark_data_smithi188_141483_object64319:head# 2024-04-23T16:43:52.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2d5d8a2:::benchmark_data_smithi188_141483_object80873:head# 2024-04-23T16:43:52.023 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.023 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2d6d155:::benchmark_data_smithi188_141483_object27659:head# 2024-04-23T16:43:52.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2d922cf:::benchmark_data_smithi188_141483_object16318:head# 2024-04-23T16:43:52.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2d9789e:::benchmark_data_smithi188_141483_object48322:head# 2024-04-23T16:43:52.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2d9dd63:::benchmark_data_smithi188_141483_object65574:head# 2024-04-23T16:43:52.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2da122a:::benchmark_data_smithi188_141483_object35742:head# 2024-04-23T16:43:52.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2da78fc:::benchmark_data_smithi188_141483_object57138:head# 2024-04-23T16:43:52.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2daab0b:::benchmark_data_smithi188_141483_object27354:head# 2024-04-23T16:43:52.024 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.024 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2db34da:::benchmark_data_smithi188_141483_object62874:head# 2024-04-23T16:43:52.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2dc06d2:::benchmark_data_smithi188_141483_object49464:head# 2024-04-23T16:43:52.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2dcd2fc:::benchmark_data_smithi188_141483_object56364:head# 2024-04-23T16:43:52.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2dcf439:::benchmark_data_smithi188_141483_object88812:head# 2024-04-23T16:43:52.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2dcf857:::benchmark_data_smithi188_141483_object80839:head# 2024-04-23T16:43:52.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2ddb271:::benchmark_data_smithi188_141483_object10876:head# 2024-04-23T16:43:52.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2de4cd6:::benchmark_data_smithi188_141483_object15360:head# 2024-04-23T16:43:52.025 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.025 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2e0868f:::benchmark_data_smithi188_141483_object38413:head# 2024-04-23T16:43:52.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2e0e397:::benchmark_data_smithi188_141483_object22216:head# 2024-04-23T16:43:52.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2e23d3d:::benchmark_data_smithi188_141483_object32695:head# 2024-04-23T16:43:52.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2e273c3:::benchmark_data_smithi188_141483_object60153:head# 2024-04-23T16:43:52.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2e386f6:::benchmark_data_smithi188_141483_object16678:head# 2024-04-23T16:43:52.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2e47a92:::benchmark_data_smithi188_141483_object2107:head# 2024-04-23T16:43:52.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.026 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2e81807:::benchmark_data_smithi188_141483_object16911:head# 2024-04-23T16:43:52.026 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2e858de:::benchmark_data_smithi188_141483_object16974:head# 2024-04-23T16:43:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2e93415:::benchmark_data_smithi188_141483_object767:head# 2024-04-23T16:43:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2ea17a1:::benchmark_data_smithi188_141483_object7537:head# 2024-04-23T16:43:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2ea1e24:::benchmark_data_smithi188_141483_object74329:head# 2024-04-23T16:43:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2eaaed5:::benchmark_data_smithi188_141483_object83611:head# 2024-04-23T16:43:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2eb6377:::benchmark_data_smithi188_141483_object58306:head# 2024-04-23T16:43:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2ed49b3:::benchmark_data_smithi188_141483_object318:head# 2024-04-23T16:43:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2eeb37b:::benchmark_data_smithi188_141483_object36630:head# 2024-04-23T16:43:52.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2eee0d5:::benchmark_data_smithi188_141483_object10081:head# 2024-04-23T16:43:52.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2f00f97:::benchmark_data_smithi188_141483_object391:head# 2024-04-23T16:43:52.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2f0e481:::benchmark_data_smithi188_141483_object2386:head# 2024-04-23T16:43:52.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2f118c4:::benchmark_data_smithi188_141483_object12306:head# 2024-04-23T16:43:52.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2f49a51:::benchmark_data_smithi188_141483_object21861:head# 2024-04-23T16:43:52.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2f4fa9b:::benchmark_data_smithi188_141483_object85349:head# 2024-04-23T16:43:52.028 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.028 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2f51a3d:::benchmark_data_smithi188_141483_object60033:head# 2024-04-23T16:43:52.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2f59492:::benchmark_data_smithi188_141483_object71348:head# 2024-04-23T16:43:52.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2f75052:::benchmark_data_smithi188_141483_object67724:head# 2024-04-23T16:43:52.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2f75706:::benchmark_data_smithi188_141483_object83394:head# 2024-04-23T16:43:52.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2f7b1ff:::benchmark_data_smithi188_141483_object35043:head# 2024-04-23T16:43:52.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2f7d872:::benchmark_data_smithi188_141483_object36383:head# 2024-04-23T16:43:52.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2f7e713:::benchmark_data_smithi188_141483_object29779:head# 2024-04-23T16:43:52.029 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.029 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2f7fb6e:::benchmark_data_smithi188_141483_object51290:head# 2024-04-23T16:43:52.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2f830c1:::benchmark_data_smithi188_141483_object20348:head# 2024-04-23T16:43:52.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2f90901:::benchmark_data_smithi188_141483_object33863:head# 2024-04-23T16:43:52.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2f92e52:::benchmark_data_smithi188_141483_object88918:head# 2024-04-23T16:43:52.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2f9b440:::benchmark_data_smithi188_141483_object19616:head# 2024-04-23T16:43:52.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2fb56e4:::benchmark_data_smithi188_141483_object89732:head# 2024-04-23T16:43:52.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2fbb792:::benchmark_data_smithi188_141483_object85156:head# 2024-04-23T16:43:52.030 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.030 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2fc17af:::benchmark_data_smithi188_141483_object73358:head# 2024-04-23T16:43:52.031 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.073 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2fc1eb0:::benchmark_data_smithi188_141483_object12501:head# 2024-04-23T16:43:52.073 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.073 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2fd49f4:::benchmark_data_smithi188_141483_object4626:head# 2024-04-23T16:43:52.073 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.073 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c2ffea61:::benchmark_data_smithi188_141483_object32345:head# 2024-04-23T16:43:52.073 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.073 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3000f01:::benchmark_data_smithi188_141483_object45816:head# 2024-04-23T16:43:52.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3013c24:::benchmark_data_smithi188_141483_object1652:head# 2024-04-23T16:43:52.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c301dc38:::benchmark_data_smithi188_141483_object72438:head# 2024-04-23T16:43:52.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3026a26:::benchmark_data_smithi188_141483_object9198:head# 2024-04-23T16:43:52.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c302b331:::benchmark_data_smithi188_141483_object37663:head# 2024-04-23T16:43:52.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c306eb3a:::benchmark_data_smithi188_141483_object18770:head# 2024-04-23T16:43:52.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.074 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c306f804:::benchmark_data_smithi188_141483_object34154:head# 2024-04-23T16:43:52.074 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c308c808:::benchmark_data_smithi188_141483_object84817:head# 2024-04-23T16:43:52.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3092596:::benchmark_data_smithi188_141483_object5171:head# 2024-04-23T16:43:52.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c30a1aa5:::benchmark_data_smithi188_141483_object16406:head# 2024-04-23T16:43:52.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c30abba7:::benchmark_data_smithi188_141483_object79005:head# 2024-04-23T16:43:52.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c30d21be:::benchmark_data_smithi188_141483_object34063:head# 2024-04-23T16:43:52.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c30e1df4:::benchmark_data_smithi188_141483_object40729:head# 2024-04-23T16:43:52.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.075 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c30ee06d:::benchmark_data_smithi188_141483_object27120:head# 2024-04-23T16:43:52.075 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c30fab69:::benchmark_data_smithi188_141483_object1128:head# 2024-04-23T16:43:52.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c30fc6a0:::benchmark_data_smithi188_141483_object12674:head# 2024-04-23T16:43:52.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c314784b:::benchmark_data_smithi188_141483_object67267:head# 2024-04-23T16:43:52.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c31678ed:::benchmark_data_smithi188_141483_object23591:head# 2024-04-23T16:43:52.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c31734d5:::benchmark_data_smithi188_141483_object39904:head# 2024-04-23T16:43:52.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3180f6a:::benchmark_data_smithi188_141483_object78895:head# 2024-04-23T16:43:52.076 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.076 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c31894cd:::benchmark_data_smithi188_141483_object17134:head# 2024-04-23T16:43:52.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c318de5b:::benchmark_data_smithi188_141483_object12039:head# 2024-04-23T16:43:52.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c31933ba:::benchmark_data_smithi188_141483_object33252:head# 2024-04-23T16:43:52.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3194b9a:::benchmark_data_smithi188_141483_object20490:head# 2024-04-23T16:43:52.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c319cecd:::benchmark_data_smithi188_141483_object17511:head# 2024-04-23T16:43:52.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.077 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c31a6474:::benchmark_data_smithi188_141483_object57351:head# 2024-04-23T16:43:52.077 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.078 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c31b4374:::benchmark_data_smithi188_141483_object76866:head# 2024-04-23T16:43:52.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.078 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c31cc01e:::benchmark_data_smithi188_141483_object84075:head# 2024-04-23T16:43:52.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.078 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c31e5132:::benchmark_data_smithi188_141483_object18357:head# 2024-04-23T16:43:52.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.078 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c31e608f:::benchmark_data_smithi188_141483_object35492:head# 2024-04-23T16:43:52.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.078 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c31ed402:::benchmark_data_smithi188_141483_object43732:head# 2024-04-23T16:43:52.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.078 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c31f1f3b:::benchmark_data_smithi188_141483_object49434:head# 2024-04-23T16:43:52.078 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.079 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3200ff3:::benchmark_data_smithi188_141483_object46620:head# 2024-04-23T16:43:52.079 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.079 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32067ab:::benchmark_data_smithi188_141483_object47209:head# 2024-04-23T16:43:52.079 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.079 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c320760f:::benchmark_data_smithi188_141483_object25213:head# 2024-04-23T16:43:52.079 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.079 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c320897d:::benchmark_data_smithi188_141483_object88334:head# 2024-04-23T16:43:52.079 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.079 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c322e1a4:::benchmark_data_smithi188_141483_object30426:head# 2024-04-23T16:43:52.079 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.079 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3233b6c:::benchmark_data_smithi188_141483_object63224:head# 2024-04-23T16:43:52.079 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.079 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32396c5:::benchmark_data_smithi188_141483_object37103:head# 2024-04-23T16:43:52.080 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.080 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3240ae0:::benchmark_data_smithi188_141483_object60702:head# 2024-04-23T16:43:52.080 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.080 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32534f4:::benchmark_data_smithi188_141483_object11748:head# 2024-04-23T16:43:52.080 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.080 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32592c0:::benchmark_data_smithi188_141483_object3605:head# 2024-04-23T16:43:52.080 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.080 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c325db1d:::benchmark_data_smithi188_141483_object39301:head# 2024-04-23T16:43:52.080 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.080 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c325dfb6:::benchmark_data_smithi188_141483_object8377:head# 2024-04-23T16:43:52.080 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.080 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3267586:::benchmark_data_smithi188_141483_object75481:head# 2024-04-23T16:43:52.080 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.081 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32a5d75:::benchmark_data_smithi188_141483_object65551:head# 2024-04-23T16:43:52.081 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.081 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32ad96c:::benchmark_data_smithi188_141483_object8210:head# 2024-04-23T16:43:52.081 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.081 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32adbaf:::benchmark_data_smithi188_141483_object14578:head# 2024-04-23T16:43:52.081 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.081 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32ba3eb:::benchmark_data_smithi188_141483_object77403:head# 2024-04-23T16:43:52.081 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.081 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32bb56d:::benchmark_data_smithi188_141483_object68188:head# 2024-04-23T16:43:52.081 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.081 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32c2263:::benchmark_data_smithi188_141483_object38307:head# 2024-04-23T16:43:52.081 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.081 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32c3285:::benchmark_data_smithi188_141483_object46829:head# 2024-04-23T16:43:52.082 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.082 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32c33f3:::benchmark_data_smithi188_141483_object5668:head# 2024-04-23T16:43:52.082 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.082 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32c5eb0:::benchmark_data_smithi188_141483_object15575:head# 2024-04-23T16:43:52.082 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.082 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32cccb2:::benchmark_data_smithi188_141483_object78724:head# 2024-04-23T16:43:52.082 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.082 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32d0ed6:::benchmark_data_smithi188_141483_object80923:head# 2024-04-23T16:43:52.082 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.082 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32d4e8c:::benchmark_data_smithi188_141483_object84175:head# 2024-04-23T16:43:52.082 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.082 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32d5532:::benchmark_data_smithi188_141483_object38241:head# 2024-04-23T16:43:52.082 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.083 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32e59da:::benchmark_data_smithi188_141483_object66182:head# 2024-04-23T16:43:52.083 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.083 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32f2dae:::benchmark_data_smithi188_141483_object82446:head# 2024-04-23T16:43:52.083 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.083 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c32f6254:::benchmark_data_smithi188_141483_object47870:head# 2024-04-23T16:43:52.083 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.083 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3300544:::benchmark_data_smithi188_141483_object72442:head# 2024-04-23T16:43:52.083 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.083 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3306170:::benchmark_data_smithi188_141483_object22112:head# 2024-04-23T16:43:52.083 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.083 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c33066f4:::benchmark_data_smithi188_141483_object40935:head# 2024-04-23T16:43:52.083 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.083 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c330efbc:::benchmark_data_smithi188_141483_object43311:head# 2024-04-23T16:43:52.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3317c2c:::benchmark_data_smithi188_141483_object59117:head# 2024-04-23T16:43:52.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3322b09:::benchmark_data_smithi188_141483_object56352:head# 2024-04-23T16:43:52.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c33307e1:::benchmark_data_smithi188_141483_object80348:head# 2024-04-23T16:43:52.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c334bf68:::benchmark_data_smithi188_141483_object15783:head# 2024-04-23T16:43:52.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c337196d:::benchmark_data_smithi188_141483_object33062:head# 2024-04-23T16:43:52.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c33886ef:::benchmark_data_smithi188_141483_object49517:head# 2024-04-23T16:43:52.084 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.084 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c338f48e:::benchmark_data_smithi188_141483_object78687:head# 2024-04-23T16:43:52.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c33a2ed8:::benchmark_data_smithi188_141483_object76758:head# 2024-04-23T16:43:52.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c33aae49:::benchmark_data_smithi188_141483_object37620:head# 2024-04-23T16:43:52.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c33ab8b1:::benchmark_data_smithi188_141483_object39434:head# 2024-04-23T16:43:52.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c33b0621:::benchmark_data_smithi188_141483_object9351:head# 2024-04-23T16:43:52.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c33b0ef2:::benchmark_data_smithi188_141483_object56918:head# 2024-04-23T16:43:52.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c33cc72f:::benchmark_data_smithi188_141483_object69077:head# 2024-04-23T16:43:52.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.085 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c33d892f:::benchmark_data_smithi188_141483_object68908:head# 2024-04-23T16:43:52.085 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.086 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c33dcf03:::benchmark_data_smithi188_141483_object14717:head# 2024-04-23T16:43:52.086 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.086 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c33f045f:::benchmark_data_smithi188_141483_object77707:head# 2024-04-23T16:43:52.086 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.086 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3408da7:::benchmark_data_smithi188_141483_object12190:head# 2024-04-23T16:43:52.086 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.086 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3416685:::benchmark_data_smithi188_141483_object39524:head# 2024-04-23T16:43:52.086 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.086 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c341b2b8:::benchmark_data_smithi188_141483_object18185:head# 2024-04-23T16:43:52.086 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.086 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c341d003:::benchmark_data_smithi188_141483_object28581:head# 2024-04-23T16:43:52.086 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.086 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c342b937:::benchmark_data_smithi188_141483_object70885:head# 2024-04-23T16:43:52.087 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.087 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c342d78b:::benchmark_data_smithi188_141483_object48653:head# 2024-04-23T16:43:52.087 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.087 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c343005a:::benchmark_data_smithi188_141483_object68996:head# 2024-04-23T16:43:52.087 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.087 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c34333b4:::benchmark_data_smithi188_141483_object69628:head# 2024-04-23T16:43:52.087 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.087 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c343a147:::benchmark_data_smithi188_141483_object43540:head# 2024-04-23T16:43:52.087 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.087 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c344c611:::benchmark_data_smithi188_141483_object85774:head# 2024-04-23T16:43:52.087 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.087 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c345a56c:::benchmark_data_smithi188_141483_object6590:head# 2024-04-23T16:43:52.087 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.087 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c34688fd:::benchmark_data_smithi188_141483_object5343:head# 2024-04-23T16:43:52.088 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.088 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c347f31c:::benchmark_data_smithi188_141483_object20220:head# 2024-04-23T16:43:52.088 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.088 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c348c6ca:::benchmark_data_smithi188_141483_object29175:head# 2024-04-23T16:43:52.088 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.088 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c34a0442:::benchmark_data_smithi188_141483_object7197:head# 2024-04-23T16:43:52.088 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.088 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c34a0e0d:::benchmark_data_smithi188_141483_object26921:head# 2024-04-23T16:43:52.088 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.088 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c34afb3b:::benchmark_data_smithi188_141483_object70377:head# 2024-04-23T16:43:52.088 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.088 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c34bcfff:::benchmark_data_smithi188_141483_object77577:head# 2024-04-23T16:43:52.088 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.089 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c34bfe82:::benchmark_data_smithi188_141483_object45155:head# 2024-04-23T16:43:52.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.089 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c34ca44f:::benchmark_data_smithi188_141483_object7279:head# 2024-04-23T16:43:52.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.089 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c34ca846:::benchmark_data_smithi188_141483_object39581:head# 2024-04-23T16:43:52.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.089 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c34fe3f4:::benchmark_data_smithi188_141483_object61152:head# 2024-04-23T16:43:52.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.089 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3516d0e:::benchmark_data_smithi188_141483_object18673:head# 2024-04-23T16:43:52.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.089 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c35316ad:::benchmark_data_smithi188_141483_object59144:head# 2024-04-23T16:43:52.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.089 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c354406b:::benchmark_data_smithi188_141483_object52642:head# 2024-04-23T16:43:52.089 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3550791:::benchmark_data_smithi188_141483_object37905:head# 2024-04-23T16:43:52.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c35637d9:::benchmark_data_smithi188_141483_object44383:head# 2024-04-23T16:43:52.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c357d6bf:::benchmark_data_smithi188_141483_object11374:head# 2024-04-23T16:43:52.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3583b47:::benchmark_data_smithi188_141483_object34617:head# 2024-04-23T16:43:52.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3594a91:::benchmark_data_smithi188_141483_object11953:head# 2024-04-23T16:43:52.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c35a6e60:::benchmark_data_smithi188_141483_object7471:head# 2024-04-23T16:43:52.090 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.090 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c35a8101:::benchmark_data_smithi188_141483_object3490:head# 2024-04-23T16:43:52.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c35c7245:::benchmark_data_smithi188_141483_object55911:head# 2024-04-23T16:43:52.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c35cb96a:::benchmark_data_smithi188_141483_object72971:head# 2024-04-23T16:43:52.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c35d4f9d:::benchmark_data_smithi188_141483_object54764:head# 2024-04-23T16:43:52.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c35d7f38:::benchmark_data_smithi188_141483_object55284:head# 2024-04-23T16:43:52.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c35dad34:::benchmark_data_smithi188_141483_object23565:head# 2024-04-23T16:43:52.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c35def0a:::benchmark_data_smithi188_141483_object86912:head# 2024-04-23T16:43:52.091 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.091 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c35ed003:::benchmark_data_smithi188_141483_object67819:head# 2024-04-23T16:43:52.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c35f497a:::benchmark_data_smithi188_141483_object69417:head# 2024-04-23T16:43:52.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c35f4f6d:::benchmark_data_smithi188_141483_object15193:head# 2024-04-23T16:43:52.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c35f8ffb:::benchmark_data_smithi188_141483_object27160:head# 2024-04-23T16:43:52.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3608491:::benchmark_data_smithi188_141483_object83080:head# 2024-04-23T16:43:52.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3615654:::benchmark_data_smithi188_141483_object54616:head# 2024-04-23T16:43:52.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3626fa8:::benchmark_data_smithi188_141483_object15000:head# 2024-04-23T16:43:52.092 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.092 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c36299ff:::benchmark_data_smithi188_141483_object58404:head# 2024-04-23T16:43:52.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c362b1fd:::benchmark_data_smithi188_141483_object22816:head# 2024-04-23T16:43:52.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3649c52:::benchmark_data_smithi188_141483_object35495:head# 2024-04-23T16:43:52.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3654180:::benchmark_data_smithi188_141483_object39232:head# 2024-04-23T16:43:52.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c365b64d:::benchmark_data_smithi188_141483_object51877:head# 2024-04-23T16:43:52.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c365c261:::benchmark_data_smithi188_141483_object72446:head# 2024-04-23T16:43:52.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.093 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c365dcce:::benchmark_data_smithi188_141483_object46388:head# 2024-04-23T16:43:52.093 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3665dec:::benchmark_data_smithi188_141483_object16218:head# 2024-04-23T16:43:52.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c36687c6:::benchmark_data_smithi188_141483_object44166:head# 2024-04-23T16:43:52.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c367069f:::benchmark_data_smithi188_141483_object11840:head# 2024-04-23T16:43:52.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c36766ae:::benchmark_data_smithi188_141483_object4789:head# 2024-04-23T16:43:52.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c36842aa:::benchmark_data_smithi188_141483_object60457:head# 2024-04-23T16:43:52.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3693573:::benchmark_data_smithi188_141483_object62622:head# 2024-04-23T16:43:52.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.094 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3695a90:::benchmark_data_smithi188_141483_object74017:head# 2024-04-23T16:43:52.094 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c36a5b23:::benchmark_data_smithi188_141483_object54742:head# 2024-04-23T16:43:52.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c36a77de:::benchmark_data_smithi188_141483_object54812:head# 2024-04-23T16:43:52.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c36b941f:::benchmark_data_smithi188_141483_object88958:head# 2024-04-23T16:43:52.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c36bf692:::benchmark_data_smithi188_141483_object5006:head# 2024-04-23T16:43:52.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c36c1cd1:::benchmark_data_smithi188_141483_object38440:head# 2024-04-23T16:43:52.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c36cb4d8:::benchmark_data_smithi188_141483_object42435:head# 2024-04-23T16:43:52.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.095 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c36cca75:::benchmark_data_smithi188_141483_object23602:head# 2024-04-23T16:43:52.095 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c36d459e:::benchmark_data_smithi188_141483_object71659:head# 2024-04-23T16:43:52.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c36d8263:::benchmark_data_smithi188_141483_object76925:head# 2024-04-23T16:43:52.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c36fc3c6:::benchmark_data_smithi188_141483_object58769:head# 2024-04-23T16:43:52.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3702b0e:::benchmark_data_smithi188_141483_object28137:head# 2024-04-23T16:43:52.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c371a80a:::benchmark_data_smithi188_141483_object4883:head# 2024-04-23T16:43:52.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3720826:::benchmark_data_smithi188_141483_object26240:head# 2024-04-23T16:43:52.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.096 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c373d501:::benchmark_data_smithi188_141483_object54105:head# 2024-04-23T16:43:52.096 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c373e2f6:::benchmark_data_smithi188_141483_object46754:head# 2024-04-23T16:43:52.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c375706f:::benchmark_data_smithi188_141483_object71377:head# 2024-04-23T16:43:52.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c378775c:::benchmark_data_smithi188_141483_object61584:head# 2024-04-23T16:43:52.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c379e20e:::benchmark_data_smithi188_141483_object4294:head# 2024-04-23T16:43:52.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c37a5ae5:::benchmark_data_smithi188_141483_object17396:head# 2024-04-23T16:43:52.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c37b5b68:::benchmark_data_smithi188_141483_object82037:head# 2024-04-23T16:43:52.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.097 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c37c605c:::benchmark_data_smithi188_141483_object90698:head# 2024-04-23T16:43:52.097 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c37cb484:::benchmark_data_smithi188_141483_object83904:head# 2024-04-23T16:43:52.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c37cf2fc:::benchmark_data_smithi188_141483_object3140:head# 2024-04-23T16:43:52.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c37d1f31:::benchmark_data_smithi188_141483_object76216:head# 2024-04-23T16:43:52.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c380a4a0:::benchmark_data_smithi188_141483_object34481:head# 2024-04-23T16:43:52.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3810c59:::benchmark_data_smithi188_141483_object59013:head# 2024-04-23T16:43:52.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3820fc1:::benchmark_data_smithi188_141483_object37354:head# 2024-04-23T16:43:52.098 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.098 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3834e1b:::benchmark_data_smithi188_141483_object4082:head# 2024-04-23T16:43:52.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c38390b4:::benchmark_data_smithi188_141483_object23737:head# 2024-04-23T16:43:52.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3840756:::benchmark_data_smithi188_141483_object88505:head# 2024-04-23T16:43:52.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3845551:::benchmark_data_smithi188_141483_object54405:head# 2024-04-23T16:43:52.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c386b118:::benchmark_data_smithi188_141483_object41942:head# 2024-04-23T16:43:52.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3870e14:::benchmark_data_smithi188_141483_object22294:head# 2024-04-23T16:43:52.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c387ebb7:::benchmark_data_smithi188_141483_object7613:head# 2024-04-23T16:43:52.099 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.099 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c388042d:::benchmark_data_smithi188_141483_object3030:head# 2024-04-23T16:43:52.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c388ba60:::benchmark_data_smithi188_141483_object31450:head# 2024-04-23T16:43:52.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c388f97c:::benchmark_data_smithi188_141483_object72899:head# 2024-04-23T16:43:52.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c38cab74:::benchmark_data_smithi188_141483_object83643:head# 2024-04-23T16:43:52.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c38d04a5:::benchmark_data_smithi188_141483_object90901:head# 2024-04-23T16:43:52.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c38de385:::benchmark_data_smithi188_141483_object76582:head# 2024-04-23T16:43:52.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c38e1d3d:::benchmark_data_smithi188_141483_object24655:head# 2024-04-23T16:43:52.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.100 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3904301:::benchmark_data_smithi188_141483_object88423:head# 2024-04-23T16:43:52.100 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c39067a7:::benchmark_data_smithi188_141483_object87376:head# 2024-04-23T16:43:52.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3914ba5:::benchmark_data_smithi188_141483_object90478:head# 2024-04-23T16:43:52.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c39160f1:::benchmark_data_smithi188_141483_object51505:head# 2024-04-23T16:43:52.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c392d981:::benchmark_data_smithi188_141483_object78242:head# 2024-04-23T16:43:52.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3930e65:::benchmark_data_smithi188_141483_object10027:head# 2024-04-23T16:43:52.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3932e6a:::benchmark_data_smithi188_141483_object52527:head# 2024-04-23T16:43:52.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.101 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c39427ab:::benchmark_data_smithi188_141483_object80050:head# 2024-04-23T16:43:52.101 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c39562c1:::benchmark_data_smithi188_141483_object12063:head# 2024-04-23T16:43:52.102 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c395d6af:::benchmark_data_smithi188_141483_object64062:head# 2024-04-23T16:43:52.102 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c39644d2:::benchmark_data_smithi188_141483_object22072:head# 2024-04-23T16:43:52.102 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3965ab7:::benchmark_data_smithi188_141483_object3114:head# 2024-04-23T16:43:52.102 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3965c70:::benchmark_data_smithi188_141483_object66653:head# 2024-04-23T16:43:52.102 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3969af7:::benchmark_data_smithi188_141483_object84859:head# 2024-04-23T16:43:52.102 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.102 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c396a6a4:::benchmark_data_smithi188_141483_object52479:head# 2024-04-23T16:43:52.103 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.103 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3978648:::benchmark_data_smithi188_141483_object84955:head# 2024-04-23T16:43:52.103 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.103 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c397e9a6:::benchmark_data_smithi188_141483_object57317:head# 2024-04-23T16:43:52.103 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.103 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3984374:::benchmark_data_smithi188_141483_object18093:head# 2024-04-23T16:43:52.103 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.103 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c398dbd2:::benchmark_data_smithi188_141483_object41683:head# 2024-04-23T16:43:52.103 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.103 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c399af63:::benchmark_data_smithi188_141483_object36351:head# 2024-04-23T16:43:52.103 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.103 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c39a4614:::benchmark_data_smithi188_141483_object22628:head# 2024-04-23T16:43:52.103 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.103 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c39a735c:::benchmark_data_smithi188_141483_object49036:head# 2024-04-23T16:43:52.104 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.104 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c39ae074:::benchmark_data_smithi188_141483_object45939:head# 2024-04-23T16:43:52.104 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.104 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c39b2d0a:::benchmark_data_smithi188_141483_object17979:head# 2024-04-23T16:43:52.104 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.104 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c39b954d:::benchmark_data_smithi188_141483_object65861:head# 2024-04-23T16:43:52.104 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.104 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c39bbd6b:::benchmark_data_smithi188_141483_object6902:head# 2024-04-23T16:43:52.104 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.104 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c39e6ea2:::benchmark_data_smithi188_141483_object46894:head# 2024-04-23T16:43:52.104 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.104 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3a181a4:::benchmark_data_smithi188_141483_object37460:head# 2024-04-23T16:43:52.104 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.104 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3a1fb06:::benchmark_data_smithi188_141483_object77404:head# 2024-04-23T16:43:52.105 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.105 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3a2198f:::benchmark_data_smithi188_141483_object10127:head# 2024-04-23T16:43:52.105 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.105 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3a2d752:::benchmark_data_smithi188_141483_object58070:head# 2024-04-23T16:43:52.105 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.105 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3a2e343:::benchmark_data_smithi188_141483_object16220:head# 2024-04-23T16:43:52.105 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.105 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3a2ebb0:::benchmark_data_smithi188_141483_object16103:head# 2024-04-23T16:43:52.105 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.105 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3a35207:::benchmark_data_smithi188_141483_object12023:head# 2024-04-23T16:43:52.105 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.105 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3a37e5f:::benchmark_data_smithi188_141483_object58730:head# 2024-04-23T16:43:52.105 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.106 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3a3c8cc:::benchmark_data_smithi188_141483_object50728:head# 2024-04-23T16:43:52.106 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.106 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3a4618c:::benchmark_data_smithi188_141483_object73140:head# 2024-04-23T16:43:52.106 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.106 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3a5082b:::benchmark_data_smithi188_141483_object46746:head# 2024-04-23T16:43:52.106 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.106 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3a68cdf:::benchmark_data_smithi188_141483_object39644:head# 2024-04-23T16:43:52.106 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.106 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3a70bb6:::benchmark_data_smithi188_141483_object39404:head# 2024-04-23T16:43:52.106 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.106 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3aa8592:::benchmark_data_smithi188_141483_object31167:head# 2024-04-23T16:43:52.106 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.106 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3ad5332:::benchmark_data_smithi188_141483_object65491:head# 2024-04-23T16:43:52.107 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.107 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3ae1cad:::benchmark_data_smithi188_141483_object54762:head# 2024-04-23T16:43:52.107 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.107 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3aec645:::benchmark_data_smithi188_141483_object72228:head# 2024-04-23T16:43:52.107 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.107 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3af80ec:::benchmark_data_smithi188_141483_object36683:head# 2024-04-23T16:43:52.107 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.107 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3b07139:::benchmark_data_smithi188_141483_object34764:head# 2024-04-23T16:43:52.107 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.107 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3b17a43:::benchmark_data_smithi188_141483_object30252:head# 2024-04-23T16:43:52.107 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.107 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3b2a2fc:::benchmark_data_smithi188_141483_object66388:head# 2024-04-23T16:43:52.107 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3b42545:::benchmark_data_smithi188_141483_object10252:head# 2024-04-23T16:43:52.108 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3b4cffa:::benchmark_data_smithi188_141483_object46757:head# 2024-04-23T16:43:52.108 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3b4e84f:::benchmark_data_smithi188_141483_object8912:head# 2024-04-23T16:43:52.108 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3b50f72:::benchmark_data_smithi188_141483_object58165:head# 2024-04-23T16:43:52.108 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3b56528:::benchmark_data_smithi188_141483_object4996:head# 2024-04-23T16:43:52.108 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3b68c2c:::benchmark_data_smithi188_141483_object75317:head# 2024-04-23T16:43:52.108 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.108 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3b6c6a1:::benchmark_data_smithi188_141483_object46494:head# 2024-04-23T16:43:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3b7045d:::benchmark_data_smithi188_141483_object16522:head# 2024-04-23T16:43:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3b763e6:::benchmark_data_smithi188_141483_object64175:head# 2024-04-23T16:43:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3b8a3df:::benchmark_data_smithi188_141483_object37858:head# 2024-04-23T16:43:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3b8dd71:::benchmark_data_smithi188_141483_object52165:head# 2024-04-23T16:43:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3b956e0:::benchmark_data_smithi188_141483_object25140:head# 2024-04-23T16:43:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3ba3e51:::benchmark_data_smithi188_141483_object27241:head# 2024-04-23T16:43:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3ba4c7f:::benchmark_data_smithi188_141483_object85331:head# 2024-04-23T16:43:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3ba5016:::benchmark_data_smithi188_141483_object73957:head# 2024-04-23T16:43:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3bb516f:::benchmark_data_smithi188_141483_object15361:head# 2024-04-23T16:43:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3bb85ed:::benchmark_data_smithi188_141483_object30571:head# 2024-04-23T16:43:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3bbd632:::benchmark_data_smithi188_141483_object78260:head# 2024-04-23T16:43:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3bd6de5:::benchmark_data_smithi188_141483_object68504:head# 2024-04-23T16:43:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3bd85fc:::benchmark_data_smithi188_141483_object43352:head# 2024-04-23T16:43:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3bde610:::benchmark_data_smithi188_141483_object19462:head# 2024-04-23T16:43:52.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3be846e:::benchmark_data_smithi188_141483_object51336:head# 2024-04-23T16:43:52.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3c02923:::benchmark_data_smithi188_141483_object50857:head# 2024-04-23T16:43:52.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3c1834e:::benchmark_data_smithi188_141483_object22090:head# 2024-04-23T16:43:52.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3c187eb:::benchmark_data_smithi188_141483_object79409:head# 2024-04-23T16:43:52.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3c21d94:::benchmark_data_smithi188_141483_object43795:head# 2024-04-23T16:43:52.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3c438f8:::benchmark_data_smithi188_141483_object53780:head# 2024-04-23T16:43:52.111 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.111 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3c47246:::benchmark_data_smithi188_141483_object23345:head# 2024-04-23T16:43:52.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3c4d998:::benchmark_data_smithi188_141483_object48739:head# 2024-04-23T16:43:52.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3c4ea9a:::benchmark_data_smithi188_141483_object26461:head# 2024-04-23T16:43:52.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3c76040:::benchmark_data_smithi188_141483_object47486:head# 2024-04-23T16:43:52.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3c7b781:::benchmark_data_smithi188_141483_object82739:head# 2024-04-23T16:43:52.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3c86823:::benchmark_data_smithi188_141483_object349:head# 2024-04-23T16:43:52.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3c86bef:::benchmark_data_smithi188_141483_object32805:head# 2024-04-23T16:43:52.112 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.112 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3c880c7:::benchmark_data_smithi188_141483_object77616:head# 2024-04-23T16:43:52.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3c89d62:::benchmark_data_smithi188_141483_object34167:head# 2024-04-23T16:43:52.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3c8eb6a:::benchmark_data_smithi188_141483_object62981:head# 2024-04-23T16:43:52.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3c938dc:::benchmark_data_smithi188_141483_object32392:head# 2024-04-23T16:43:52.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3caed10:::benchmark_data_smithi188_141483_object14655:head# 2024-04-23T16:43:52.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3cc69ae:::benchmark_data_smithi188_141483_object58312:head# 2024-04-23T16:43:52.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3cca222:::benchmark_data_smithi188_141483_object80382:head# 2024-04-23T16:43:52.113 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.113 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3cd296b:::benchmark_data_smithi188_141483_object4630:head# 2024-04-23T16:43:52.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3cdb9fc:::benchmark_data_smithi188_141483_object84326:head# 2024-04-23T16:43:52.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3d038ee:::benchmark_data_smithi188_141483_object86195:head# 2024-04-23T16:43:52.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3d0a52e:::benchmark_data_smithi188_141483_object40921:head# 2024-04-23T16:43:52.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3d17fdc:::benchmark_data_smithi188_141483_object13335:head# 2024-04-23T16:43:52.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3d2422a:::benchmark_data_smithi188_141483_object89309:head# 2024-04-23T16:43:52.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3d2a6b0:::benchmark_data_smithi188_141483_object46530:head# 2024-04-23T16:43:52.114 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.114 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3d2c76f:::benchmark_data_smithi188_141483_object113:head# 2024-04-23T16:43:52.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3d3a352:::benchmark_data_smithi188_141483_object23159:head# 2024-04-23T16:43:52.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3d52029:::benchmark_data_smithi188_141483_object29937:head# 2024-04-23T16:43:52.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3d5cce0:::benchmark_data_smithi188_141483_object11734:head# 2024-04-23T16:43:52.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3d5eedf:::benchmark_data_smithi188_141483_object36978:head# 2024-04-23T16:43:52.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3d62e84:::benchmark_data_smithi188_141483_object35718:head# 2024-04-23T16:43:52.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.115 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3d752a9:::benchmark_data_smithi188_141483_object46113:head# 2024-04-23T16:43:52.115 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.159 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3d7b1de:::benchmark_data_smithi188_141483_object26536:head# 2024-04-23T16:43:52.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3d92efe:::benchmark_data_smithi188_141483_object10047:head# 2024-04-23T16:43:52.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3d9ef42:::benchmark_data_smithi188_141483_object40024:head# 2024-04-23T16:43:52.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3db0529:::benchmark_data_smithi188_141483_object12551:head# 2024-04-23T16:43:52.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3dbdcf9:::benchmark_data_smithi188_141483_object59292:head# 2024-04-23T16:43:52.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3dbfa66:::benchmark_data_smithi188_141483_object66588:head# 2024-04-23T16:43:52.160 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.160 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3dc7bbe:::benchmark_data_smithi188_141483_object5111:head# 2024-04-23T16:43:52.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3de4c69:::benchmark_data_smithi188_141483_object45843:head# 2024-04-23T16:43:52.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3e067a3:::benchmark_data_smithi188_141483_object78988:head# 2024-04-23T16:43:52.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3e0e303:::benchmark_data_smithi188_141483_object50072:head# 2024-04-23T16:43:52.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3e1ce51:::benchmark_data_smithi188_141483_object46534:head# 2024-04-23T16:43:52.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3e23775:::benchmark_data_smithi188_141483_object45925:head# 2024-04-23T16:43:52.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.161 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3e2c194:::benchmark_data_smithi188_141483_object34051:head# 2024-04-23T16:43:52.161 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3e3299d:::benchmark_data_smithi188_141483_object87859:head# 2024-04-23T16:43:52.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3e3a00e:::benchmark_data_smithi188_141483_object33617:head# 2024-04-23T16:43:52.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3e3e073:::benchmark_data_smithi188_141483_object28054:head# 2024-04-23T16:43:52.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3e704e9:::benchmark_data_smithi188_141483_object15445:head# 2024-04-23T16:43:52.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3e773f7:::benchmark_data_smithi188_141483_object74710:head# 2024-04-23T16:43:52.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3e7a19b:::benchmark_data_smithi188_141483_object48206:head# 2024-04-23T16:43:52.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.162 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3e7a959:::benchmark_data_smithi188_141483_object87270:head# 2024-04-23T16:43:52.162 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3e7bb4d:::benchmark_data_smithi188_141483_object6485:head# 2024-04-23T16:43:52.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3e8a154:::benchmark_data_smithi188_141483_object76319:head# 2024-04-23T16:43:52.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3e8af2d:::benchmark_data_smithi188_141483_object23952:head# 2024-04-23T16:43:52.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3e979a5:::benchmark_data_smithi188_141483_object6537:head# 2024-04-23T16:43:52.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3e97c92:::benchmark_data_smithi188_141483_object10978:head# 2024-04-23T16:43:52.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3eabd40:::benchmark_data_smithi188_141483_object81140:head# 2024-04-23T16:43:52.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.163 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3ebad5e:::benchmark_data_smithi188_141483_object63813:head# 2024-04-23T16:43:52.163 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3ec110c:::benchmark_data_smithi188_141483_object41925:head# 2024-04-23T16:43:52.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3ed4d89:::benchmark_data_smithi188_141483_object90535:head# 2024-04-23T16:43:52.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3ed54cd:::benchmark_data_smithi188_141483_object34255:head# 2024-04-23T16:43:52.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3edd47f:::benchmark_data_smithi188_141483_object17651:head# 2024-04-23T16:43:52.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3f2af37:::benchmark_data_smithi188_141483_object62567:head# 2024-04-23T16:43:52.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3f37d34:::benchmark_data_smithi188_141483_object79327:head# 2024-04-23T16:43:52.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.164 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3f3bdb2:::benchmark_data_smithi188_141483_object62033:head# 2024-04-23T16:43:52.164 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3f4953b:::benchmark_data_smithi188_141483_object84006:head# 2024-04-23T16:43:52.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3f50e8b:::benchmark_data_smithi188_141483_object67579:head# 2024-04-23T16:43:52.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3f51e22:::benchmark_data_smithi188_141483_object37064:head# 2024-04-23T16:43:52.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3f5b7a5:::benchmark_data_smithi188_141483_object47110:head# 2024-04-23T16:43:52.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3f5decc:::benchmark_data_smithi188_141483_object82248:head# 2024-04-23T16:43:52.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3f6d520:::benchmark_data_smithi188_141483_object70868:head# 2024-04-23T16:43:52.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.165 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3f87664:::benchmark_data_smithi188_141483_object14849:head# 2024-04-23T16:43:52.165 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.166 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3f8cd49:::benchmark_data_smithi188_141483_object29545:head# 2024-04-23T16:43:52.166 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.166 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3f974f2:::benchmark_data_smithi188_141483_object85067:head# 2024-04-23T16:43:52.166 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.166 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3f99797:::benchmark_data_smithi188_141483_object84818:head# 2024-04-23T16:43:52.166 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.166 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3fbbfbd:::benchmark_data_smithi188_141483_object25769:head# 2024-04-23T16:43:52.166 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.166 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3fbf624:::benchmark_data_smithi188_141483_object50976:head# 2024-04-23T16:43:52.166 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.166 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3fce587:::benchmark_data_smithi188_141483_object5178:head# 2024-04-23T16:43:52.166 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.166 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3ff1302:::benchmark_data_smithi188_141483_object64773:head# 2024-04-23T16:43:52.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3ff62a2:::benchmark_data_smithi188_141483_object83349:head# 2024-04-23T16:43:52.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c3ffe7c7:::benchmark_data_smithi188_141483_object46397:head# 2024-04-23T16:43:52.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4028b5f:::benchmark_data_smithi188_141483_object71337:head# 2024-04-23T16:43:52.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c402a715:::benchmark_data_smithi188_141483_object73885:head# 2024-04-23T16:43:52.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4035379:::benchmark_data_smithi188_141483_object78598:head# 2024-04-23T16:43:52.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c403f6ce:::benchmark_data_smithi188_141483_object43186:head# 2024-04-23T16:43:52.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.167 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4049ba0:::benchmark_data_smithi188_141483_object85140:head# 2024-04-23T16:43:52.167 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c404f763:::benchmark_data_smithi188_141483_object52091:head# 2024-04-23T16:43:52.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c40568e3:::benchmark_data_smithi188_141483_object74647:head# 2024-04-23T16:43:52.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c405c0cd:::benchmark_data_smithi188_141483_object6303:head# 2024-04-23T16:43:52.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4073f75:::benchmark_data_smithi188_141483_object56069:head# 2024-04-23T16:43:52.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4076182:::benchmark_data_smithi188_141483_object70382:head# 2024-04-23T16:43:52.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c40764d6:::benchmark_data_smithi188_141483_object86907:head# 2024-04-23T16:43:52.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.168 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4088bc5:::benchmark_data_smithi188_141483_object52052:head# 2024-04-23T16:43:52.168 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4096754:::benchmark_data_smithi188_141483_object73078:head# 2024-04-23T16:43:52.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c409a246:::benchmark_data_smithi188_141483_object89634:head# 2024-04-23T16:43:52.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c40adb85:::benchmark_data_smithi188_141483_object24919:head# 2024-04-23T16:43:52.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c40aecf1:::benchmark_data_smithi188_141483_object19839:head# 2024-04-23T16:43:52.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c40c2d53:::benchmark_data_smithi188_141483_object33:head# 2024-04-23T16:43:52.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c40cb868:::benchmark_data_smithi188_141483_object59182:head# 2024-04-23T16:43:52.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.169 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c40cbe7b:::benchmark_data_smithi188_141483_object84063:head# 2024-04-23T16:43:52.169 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c40e346b:::benchmark_data_smithi188_141483_object59020:head# 2024-04-23T16:43:52.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c40e493f:::benchmark_data_smithi188_141483_object42912:head# 2024-04-23T16:43:52.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4103773:::benchmark_data_smithi188_141483_object43749:head# 2024-04-23T16:43:52.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c410747e:::benchmark_data_smithi188_141483_object16191:head# 2024-04-23T16:43:52.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4107e5d:::benchmark_data_smithi188_141483_object5113:head# 2024-04-23T16:43:52.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c410e771:::benchmark_data_smithi188_141483_object20172:head# 2024-04-23T16:43:52.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.170 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4111abd:::benchmark_data_smithi188_141483_object30473:head# 2024-04-23T16:43:52.170 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4113bce:::benchmark_data_smithi188_141483_object23284:head# 2024-04-23T16:43:52.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4116987:::benchmark_data_smithi188_141483_object33775:head# 2024-04-23T16:43:52.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c412de6c:::benchmark_data_smithi188_141483_object14795:head# 2024-04-23T16:43:52.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c41334b6:::benchmark_data_smithi188_141483_object62756:head# 2024-04-23T16:43:52.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c413c315:::benchmark_data_smithi188_141483_object18641:head# 2024-04-23T16:43:52.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4141340:::benchmark_data_smithi188_141483_object48403:head# 2024-04-23T16:43:52.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4143f09:::benchmark_data_smithi188_141483_object32355:head# 2024-04-23T16:43:52.171 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.171 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c416c5aa:::benchmark_data_smithi188_141483_object39466:head# 2024-04-23T16:43:52.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4199e95:::benchmark_data_smithi188_141483_object61438:head# 2024-04-23T16:43:52.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c41a6c08:::benchmark_data_smithi188_141483_object48639:head# 2024-04-23T16:43:52.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c41aa74e:::benchmark_data_smithi188_141483_object49641:head# 2024-04-23T16:43:52.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c41b9397:::benchmark_data_smithi188_141483_object26854:head# 2024-04-23T16:43:52.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c41be3ba:::benchmark_data_smithi188_141483_object65322:head# 2024-04-23T16:43:52.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c41c8992:::benchmark_data_smithi188_141483_object21366:head# 2024-04-23T16:43:52.172 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.172 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c41f2248:::benchmark_data_smithi188_141483_object22739:head# 2024-04-23T16:43:52.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c420ac75:::benchmark_data_smithi188_141483_object69812:head# 2024-04-23T16:43:52.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c42115cb:::benchmark_data_smithi188_141483_object77679:head# 2024-04-23T16:43:52.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4220126:::benchmark_data_smithi188_141483_object28239:head# 2024-04-23T16:43:52.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c42528d1:::benchmark_data_smithi188_141483_object83177:head# 2024-04-23T16:43:52.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4259e3d:::benchmark_data_smithi188_141483_object90226:head# 2024-04-23T16:43:52.173 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.173 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c426c8c2:::benchmark_data_smithi188_141483_object23282:head# 2024-04-23T16:43:52.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4273465:::benchmark_data_smithi188_141483_object32783:head# 2024-04-23T16:43:52.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c427505c:::benchmark_data_smithi188_141483_object73208:head# 2024-04-23T16:43:52.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4277787:::benchmark_data_smithi188_141483_object39788:head# 2024-04-23T16:43:52.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c42824a6:::benchmark_data_smithi188_141483_object62588:head# 2024-04-23T16:43:52.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4287170:::benchmark_data_smithi188_141483_object1486:head# 2024-04-23T16:43:52.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.174 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c42a3b6f:::benchmark_data_smithi188_141483_object35610:head# 2024-04-23T16:43:52.174 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c42a3f7f:::benchmark_data_smithi188_141483_object43651:head# 2024-04-23T16:43:52.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c42a9d69:::benchmark_data_smithi188_141483_object53171:head# 2024-04-23T16:43:52.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c42b3c11:::benchmark_data_smithi188_141483_object9033:head# 2024-04-23T16:43:52.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c42badaa:::benchmark_data_smithi188_141483_object7930:head# 2024-04-23T16:43:52.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c42c96fe:::benchmark_data_smithi188_141483_object47062:head# 2024-04-23T16:43:52.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c42cc22e:::benchmark_data_smithi188_141483_object39558:head# 2024-04-23T16:43:52.175 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.175 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c42d0cc7:::benchmark_data_smithi188_141483_object50892:head# 2024-04-23T16:43:52.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c42e0cdb:::benchmark_data_smithi188_141483_object35753:head# 2024-04-23T16:43:52.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c42ec403:::benchmark_data_smithi188_141483_object14019:head# 2024-04-23T16:43:52.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c42f47d5:::benchmark_data_smithi188_141483_object16112:head# 2024-04-23T16:43:52.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c430966d:::benchmark_data_smithi188_141483_object71627:head# 2024-04-23T16:43:52.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c430f677:::benchmark_data_smithi188_141483_object28148:head# 2024-04-23T16:43:52.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4321541:::benchmark_data_smithi188_141483_object72021:head# 2024-04-23T16:43:52.176 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.176 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4323c1a:::benchmark_data_smithi188_141483_object88620:head# 2024-04-23T16:43:52.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c433e10f:::benchmark_data_smithi188_141483_object79469:head# 2024-04-23T16:43:52.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c435875f:::benchmark_data_smithi188_141483_object48770:head# 2024-04-23T16:43:52.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c435de49:::benchmark_data_smithi188_141483_object62939:head# 2024-04-23T16:43:52.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c437512e:::benchmark_data_smithi188_141483_object2425:head# 2024-04-23T16:43:52.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c43aa704:::benchmark_data_smithi188_141483_object34226:head# 2024-04-23T16:43:52.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c43b1956:::benchmark_data_smithi188_141483_object78052:head# 2024-04-23T16:43:52.177 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.177 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c43c5e0a:::benchmark_data_smithi188_141483_object24463:head# 2024-04-23T16:43:52.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c43ce775:::benchmark_data_smithi188_141483_object74697:head# 2024-04-23T16:43:52.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c43d0293:::benchmark_data_smithi188_141483_object42459:head# 2024-04-23T16:43:52.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c43e5ccd:::benchmark_data_smithi188_141483_object44780:head# 2024-04-23T16:43:52.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c442849c:::benchmark_data_smithi188_141483_object44858:head# 2024-04-23T16:43:52.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c442d022:::benchmark_data_smithi188_141483_object87829:head# 2024-04-23T16:43:52.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c443427f:::benchmark_data_smithi188_141483_object52362:head# 2024-04-23T16:43:52.178 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.178 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c44431ac:::benchmark_data_smithi188_141483_object46803:head# 2024-04-23T16:43:52.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c444c7ee:::benchmark_data_smithi188_141483_object22983:head# 2024-04-23T16:43:52.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c444f055:::benchmark_data_smithi188_141483_object23031:head# 2024-04-23T16:43:52.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4456738:::benchmark_data_smithi188_141483_object52826:head# 2024-04-23T16:43:52.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c445caae:::benchmark_data_smithi188_141483_object36346:head# 2024-04-23T16:43:52.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c445dacd:::benchmark_data_smithi188_141483_object53700:head# 2024-04-23T16:43:52.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c445e975:::benchmark_data_smithi188_141483_object3894:head# 2024-04-23T16:43:52.179 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.179 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c44670f4:::benchmark_data_smithi188_141483_object90371:head# 2024-04-23T16:43:52.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c446bfeb:::benchmark_data_smithi188_141483_object56183:head# 2024-04-23T16:43:52.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c446ea4e:::benchmark_data_smithi188_141483_object76370:head# 2024-04-23T16:43:52.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4470821:::benchmark_data_smithi188_141483_object22884:head# 2024-04-23T16:43:52.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4470e1d:::benchmark_data_smithi188_141483_object7898:head# 2024-04-23T16:43:52.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4477323:::benchmark_data_smithi188_141483_object9025:head# 2024-04-23T16:43:52.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4493b96:::benchmark_data_smithi188_141483_object58656:head# 2024-04-23T16:43:52.180 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.180 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4496681:::benchmark_data_smithi188_141483_object65729:head# 2024-04-23T16:43:52.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c44972e6:::benchmark_data_smithi188_141483_object70872:head# 2024-04-23T16:43:52.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c44982ed:::benchmark_data_smithi188_141483_object49230:head# 2024-04-23T16:43:52.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c449cf76:::benchmark_data_smithi188_141483_object63722:head# 2024-04-23T16:43:52.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c449d87f:::benchmark_data_smithi188_141483_object4562:head# 2024-04-23T16:43:52.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c44a80f8:::benchmark_data_smithi188_141483_object36060:head# 2024-04-23T16:43:52.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c44c2276:::benchmark_data_smithi188_141483_object51864:head# 2024-04-23T16:43:52.181 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.181 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c44c974d:::benchmark_data_smithi188_141483_object2003:head# 2024-04-23T16:43:52.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c44cb7f4:::benchmark_data_smithi188_141483_object43442:head# 2024-04-23T16:43:52.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c44e2e97:::benchmark_data_smithi188_141483_object75204:head# 2024-04-23T16:43:52.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c44e595a:::benchmark_data_smithi188_141483_object10331:head# 2024-04-23T16:43:52.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c44f07c2:::benchmark_data_smithi188_141483_object86319:head# 2024-04-23T16:43:52.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c44fedae:::benchmark_data_smithi188_141483_object5867:head# 2024-04-23T16:43:52.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c450341d:::benchmark_data_smithi188_141483_object12300:head# 2024-04-23T16:43:52.182 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.182 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4504bcc:::benchmark_data_smithi188_141483_object20587:head# 2024-04-23T16:43:52.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c450db4b:::benchmark_data_smithi188_141483_object43248:head# 2024-04-23T16:43:52.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c450efdf:::benchmark_data_smithi188_141483_object75292:head# 2024-04-23T16:43:52.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c451040c:::benchmark_data_smithi188_141483_object74573:head# 2024-04-23T16:43:52.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4511361:::benchmark_data_smithi188_141483_object37952:head# 2024-04-23T16:43:52.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4516aaa:::benchmark_data_smithi188_141483_object47595:head# 2024-04-23T16:43:52.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4518f3a:::benchmark_data_smithi188_141483_object71508:head# 2024-04-23T16:43:52.183 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.183 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c451d39b:::benchmark_data_smithi188_141483_object62698:head# 2024-04-23T16:43:52.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c451f213:::benchmark_data_smithi188_141483_object27152:head# 2024-04-23T16:43:52.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4520542:::benchmark_data_smithi188_141483_object26839:head# 2024-04-23T16:43:52.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4532961:::benchmark_data_smithi188_141483_object75783:head# 2024-04-23T16:43:52.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c454eb04:::benchmark_data_smithi188_141483_object86175:head# 2024-04-23T16:43:52.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c457360d:::benchmark_data_smithi188_141483_object37851:head# 2024-04-23T16:43:52.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c457cd38:::benchmark_data_smithi188_141483_object17219:head# 2024-04-23T16:43:52.184 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.184 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c45985da:::benchmark_data_smithi188_141483_object21539:head# 2024-04-23T16:43:52.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c459c2ff:::benchmark_data_smithi188_141483_object41536:head# 2024-04-23T16:43:52.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c459cba7:::benchmark_data_smithi188_141483_object39421:head# 2024-04-23T16:43:52.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c45a8136:::benchmark_data_smithi188_141483_object84268:head# 2024-04-23T16:43:52.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c45b572e:::benchmark_data_smithi188_141483_object12441:head# 2024-04-23T16:43:52.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c45c22ea:::benchmark_data_smithi188_141483_object47682:head# 2024-04-23T16:43:52.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c45c9258:::benchmark_data_smithi188_141483_object31760:head# 2024-04-23T16:43:52.185 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.185 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c45cb3c1:::benchmark_data_smithi188_141483_object13168:head# 2024-04-23T16:43:52.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c45d1eff:::benchmark_data_smithi188_141483_object27652:head# 2024-04-23T16:43:52.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c45d79a9:::benchmark_data_smithi188_141483_object35049:head# 2024-04-23T16:43:52.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c45f0c69:::benchmark_data_smithi188_141483_object45206:head# 2024-04-23T16:43:52.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c45f102e:::benchmark_data_smithi188_141483_object82850:head# 2024-04-23T16:43:52.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c45f494c:::benchmark_data_smithi188_141483_object65223:head# 2024-04-23T16:43:52.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c45fc2a1:::benchmark_data_smithi188_141483_object56631:head# 2024-04-23T16:43:52.186 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.186 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c460e95b:::benchmark_data_smithi188_141483_object163:head# 2024-04-23T16:43:52.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4618290:::benchmark_data_smithi188_141483_object19056:head# 2024-04-23T16:43:52.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c461d98f:::benchmark_data_smithi188_141483_object3844:head# 2024-04-23T16:43:52.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c46214f7:::benchmark_data_smithi188_141483_object74105:head# 2024-04-23T16:43:52.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4628e64:::benchmark_data_smithi188_141483_object676:head# 2024-04-23T16:43:52.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c463a0d3:::benchmark_data_smithi188_141483_object67205:head# 2024-04-23T16:43:52.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.187 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c463e4d1:::benchmark_data_smithi188_141483_object51798:head# 2024-04-23T16:43:52.187 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4644306:::benchmark_data_smithi188_141483_object67491:head# 2024-04-23T16:43:52.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4646ccc:::benchmark_data_smithi188_141483_object74825:head# 2024-04-23T16:43:52.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4654464:::benchmark_data_smithi188_141483_object59440:head# 2024-04-23T16:43:52.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c46552c7:::benchmark_data_smithi188_141483_object30701:head# 2024-04-23T16:43:52.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4658f41:::benchmark_data_smithi188_141483_object40897:head# 2024-04-23T16:43:52.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c465b4aa:::benchmark_data_smithi188_141483_object18131:head# 2024-04-23T16:43:52.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.188 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c465bbe7:::benchmark_data_smithi188_141483_object39030:head# 2024-04-23T16:43:52.188 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4666f1b:::benchmark_data_smithi188_141483_object49912:head# 2024-04-23T16:43:52.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4690d32:::benchmark_data_smithi188_141483_object51180:head# 2024-04-23T16:43:52.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c469474e:::benchmark_data_smithi188_141483_object58913:head# 2024-04-23T16:43:52.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4694d59:::benchmark_data_smithi188_141483_object14347:head# 2024-04-23T16:43:52.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c469c2f3:::benchmark_data_smithi188_141483_object50461:head# 2024-04-23T16:43:52.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c46a4c62:::benchmark_data_smithi188_141483_object3986:head# 2024-04-23T16:43:52.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.189 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c46a68a9:::benchmark_data_smithi188_141483_object55551:head# 2024-04-23T16:43:52.189 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c46adcb1:::benchmark_data_smithi188_141483_object10700:head# 2024-04-23T16:43:52.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c46b9a5f:::benchmark_data_smithi188_141483_object37688:head# 2024-04-23T16:43:52.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c46bfd4e:::benchmark_data_smithi188_141483_object58113:head# 2024-04-23T16:43:52.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c46cf085:::benchmark_data_smithi188_141483_object53594:head# 2024-04-23T16:43:52.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c46f4ec4:::benchmark_data_smithi188_141483_object2515:head# 2024-04-23T16:43:52.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c46fb2ff:::benchmark_data_smithi188_141483_object60462:head# 2024-04-23T16:43:52.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.190 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c470f162:::benchmark_data_smithi188_141483_object9716:head# 2024-04-23T16:43:52.190 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c471e5fa:::benchmark_data_smithi188_141483_object42411:head# 2024-04-23T16:43:52.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4724e77:::benchmark_data_smithi188_141483_object86282:head# 2024-04-23T16:43:52.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c472a3eb:::benchmark_data_smithi188_141483_object40005:head# 2024-04-23T16:43:52.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c472dbdf:::benchmark_data_smithi188_141483_object70127:head# 2024-04-23T16:43:52.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4730252:::benchmark_data_smithi188_141483_object68033:head# 2024-04-23T16:43:52.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4733b60:::benchmark_data_smithi188_141483_object49632:head# 2024-04-23T16:43:52.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.191 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4736e58:::benchmark_data_smithi188_141483_object5286:head# 2024-04-23T16:43:52.191 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c473ae75:::benchmark_data_smithi188_141483_object72460:head# 2024-04-23T16:43:52.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c476cdb4:::benchmark_data_smithi188_141483_object7486:head# 2024-04-23T16:43:52.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4777f5c:::benchmark_data_smithi188_141483_object15433:head# 2024-04-23T16:43:52.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4783b3a:::benchmark_data_smithi188_141483_object45595:head# 2024-04-23T16:43:52.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c47b6c3e:::benchmark_data_smithi188_141483_object68646:head# 2024-04-23T16:43:52.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c47ba9ae:::benchmark_data_smithi188_141483_object77698:head# 2024-04-23T16:43:52.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.192 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c47cb59e:::benchmark_data_smithi188_141483_object3132:head# 2024-04-23T16:43:52.192 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c47cebc9:::benchmark_data_smithi188_141483_object84580:head# 2024-04-23T16:43:52.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c47d2a7b:::benchmark_data_smithi188_141483_object3580:head# 2024-04-23T16:43:52.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4800ccf:::benchmark_data_smithi188_141483_object37741:head# 2024-04-23T16:43:52.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c480694f:::benchmark_data_smithi188_141483_object49874:head# 2024-04-23T16:43:52.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4808545:::benchmark_data_smithi188_141483_object59798:head# 2024-04-23T16:43:52.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c481ccb4:::benchmark_data_smithi188_141483_object41267:head# 2024-04-23T16:43:52.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.193 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c48237c4:::benchmark_data_smithi188_141483_object75940:head# 2024-04-23T16:43:52.193 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c482427c:::benchmark_data_smithi188_141483_object16216:head# 2024-04-23T16:43:52.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c48272ed:::benchmark_data_smithi188_141483_object49816:head# 2024-04-23T16:43:52.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4833b22:::benchmark_data_smithi188_141483_object7747:head# 2024-04-23T16:43:52.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c484078f:::benchmark_data_smithi188_141483_object49973:head# 2024-04-23T16:43:52.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4864de2:::benchmark_data_smithi188_141483_object41747:head# 2024-04-23T16:43:52.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c487017d:::benchmark_data_smithi188_141483_object27475:head# 2024-04-23T16:43:52.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4877096:::benchmark_data_smithi188_141483_object56857:head# 2024-04-23T16:43:52.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c48772f0:::benchmark_data_smithi188_141483_object63886:head# 2024-04-23T16:43:52.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4888e53:::benchmark_data_smithi188_141483_object44703:head# 2024-04-23T16:43:52.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c48909df:::benchmark_data_smithi188_141483_object17271:head# 2024-04-23T16:43:52.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c489e201:::benchmark_data_smithi188_141483_object78136:head# 2024-04-23T16:43:52.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c48a9eef:::benchmark_data_smithi188_141483_object2293:head# 2024-04-23T16:43:52.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c48b0097:::benchmark_data_smithi188_141483_object90061:head# 2024-04-23T16:43:52.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c48b8815:::benchmark_data_smithi188_141483_object22706:head# 2024-04-23T16:43:52.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c48b981d:::benchmark_data_smithi188_141483_object11212:head# 2024-04-23T16:43:52.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c48bb68e:::benchmark_data_smithi188_141483_object86013:head# 2024-04-23T16:43:52.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c48bef0e:::benchmark_data_smithi188_141483_object22635:head# 2024-04-23T16:43:52.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c48d7759:::benchmark_data_smithi188_141483_object8495:head# 2024-04-23T16:43:52.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c48d9079:::benchmark_data_smithi188_141483_object25516:head# 2024-04-23T16:43:52.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c48e5ecd:::benchmark_data_smithi188_141483_object66172:head# 2024-04-23T16:43:52.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c48e98a2:::benchmark_data_smithi188_141483_object63069:head# 2024-04-23T16:43:52.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c48eb084:::benchmark_data_smithi188_141483_object77424:head# 2024-04-23T16:43:52.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c490b491:::benchmark_data_smithi188_141483_object33272:head# 2024-04-23T16:43:52.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c490be84:::benchmark_data_smithi188_141483_object9915:head# 2024-04-23T16:43:52.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c49178f2:::benchmark_data_smithi188_141483_object6774:head# 2024-04-23T16:43:52.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4925e8d:::benchmark_data_smithi188_141483_object57834:head# 2024-04-23T16:43:52.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4939af7:::benchmark_data_smithi188_141483_object69293:head# 2024-04-23T16:43:52.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4940373:::benchmark_data_smithi188_141483_object2796:head# 2024-04-23T16:43:52.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c49497e5:::benchmark_data_smithi188_141483_object82812:head# 2024-04-23T16:43:52.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c495218b:::benchmark_data_smithi188_141483_object46801:head# 2024-04-23T16:43:52.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c495a953:::benchmark_data_smithi188_141483_object29147:head# 2024-04-23T16:43:52.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c495f68f:::benchmark_data_smithi188_141483_object49047:head# 2024-04-23T16:43:52.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c49b13a8:::benchmark_data_smithi188_141483_object50918:head# 2024-04-23T16:43:52.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c49fcc74:::benchmark_data_smithi188_141483_object15200:head# 2024-04-23T16:43:52.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4a06083:::benchmark_data_smithi188_141483_object3170:head# 2024-04-23T16:43:52.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4a0c21e:::benchmark_data_smithi188_141483_object66912:head# 2024-04-23T16:43:52.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4a1db9d:::benchmark_data_smithi188_141483_object63480:head# 2024-04-23T16:43:52.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4a210d1:::benchmark_data_smithi188_141483_object82442:head# 2024-04-23T16:43:52.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4a2b8e7:::benchmark_data_smithi188_141483_object66837:head# 2024-04-23T16:43:52.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4a35d79:::benchmark_data_smithi188_141483_object58576:head# 2024-04-23T16:43:52.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4a46889:::benchmark_data_smithi188_141483_object44399:head# 2024-04-23T16:43:52.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4a4f14c:::benchmark_data_smithi188_141483_object15815:head# 2024-04-23T16:43:52.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4a5cb6e:::benchmark_data_smithi188_141483_object72405:head# 2024-04-23T16:43:52.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4a73253:::benchmark_data_smithi188_141483_object8499:head# 2024-04-23T16:43:52.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4a77cb7:::benchmark_data_smithi188_141483_object7724:head# 2024-04-23T16:43:52.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4a82520:::benchmark_data_smithi188_141483_object65486:head# 2024-04-23T16:43:52.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4a8ede7:::benchmark_data_smithi188_141483_object60669:head# 2024-04-23T16:43:52.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4a91610:::benchmark_data_smithi188_141483_object31297:head# 2024-04-23T16:43:52.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4a93157:::benchmark_data_smithi188_141483_object39507:head# 2024-04-23T16:43:52.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4a932f5:::benchmark_data_smithi188_141483_object27134:head# 2024-04-23T16:43:52.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4a9797e:::benchmark_data_smithi188_141483_object77348:head# 2024-04-23T16:43:52.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4ab629d:::benchmark_data_smithi188_141483_object87371:head# 2024-04-23T16:43:52.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4ad2aed:::benchmark_data_smithi188_141483_object59041:head# 2024-04-23T16:43:52.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4ada67f:::benchmark_data_smithi188_141483_object35717:head# 2024-04-23T16:43:52.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4adfd6c:::benchmark_data_smithi188_141483_object2866:head# 2024-04-23T16:43:52.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4af2926:::benchmark_data_smithi188_141483_object41550:head# 2024-04-23T16:43:52.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4af3927:::benchmark_data_smithi188_141483_object64726:head# 2024-04-23T16:43:52.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4b008cc:::benchmark_data_smithi188_141483_object2125:head# 2024-04-23T16:43:52.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4b077a6:::benchmark_data_smithi188_141483_object2036:head# 2024-04-23T16:43:52.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4b0d0bf:::benchmark_data_smithi188_141483_object8584:head# 2024-04-23T16:43:52.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4b23034:::benchmark_data_smithi188_141483_object52643:head# 2024-04-23T16:43:52.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4b3b4a6:::benchmark_data_smithi188_141483_object83941:head# 2024-04-23T16:43:52.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4b4bc95:::benchmark_data_smithi188_141483_object80322:head# 2024-04-23T16:43:52.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4b5d8e8:::benchmark_data_smithi188_141483_object18370:head# 2024-04-23T16:43:52.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4b66cd4:::benchmark_data_smithi188_141483_object925:head# 2024-04-23T16:43:52.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4b6c96b:::benchmark_data_smithi188_141483_object64619:head# 2024-04-23T16:43:52.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.203 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:43:52.204 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:43:52.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4b759ad:::benchmark_data_smithi188_141483_object81723:head# 2024-04-23T16:43:52.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4b88afd:::benchmark_data_smithi188_141483_object43552:head# 2024-04-23T16:43:52.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4b8cb1a:::benchmark_data_smithi188_141483_object54510:head# 2024-04-23T16:43:52.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4b93ab3:::benchmark_data_smithi188_141483_object59083:head# 2024-04-23T16:43:52.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4b9a49b:::benchmark_data_smithi188_141483_object56170:head# 2024-04-23T16:43:52.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4ba4153:::benchmark_data_smithi188_141483_object50024:head# 2024-04-23T16:43:52.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4baf159:::benchmark_data_smithi188_141483_object79393:head# 2024-04-23T16:43:52.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4bb48e9:::benchmark_data_smithi188_141483_object114:head# 2024-04-23T16:43:52.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4bb7d80:::benchmark_data_smithi188_141483_object72904:head# 2024-04-23T16:43:52.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4bc3828:::benchmark_data_smithi188_141483_object86884:head# 2024-04-23T16:43:52.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4bf2bbe:::benchmark_data_smithi188_141483_object41167:head# 2024-04-23T16:43:52.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4bf410a:::benchmark_data_smithi188_141483_object55625:head# 2024-04-23T16:43:52.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4bf8ecb:::benchmark_data_smithi188_141483_object43133:head# 2024-04-23T16:43:52.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4c12bba:::benchmark_data_smithi188_141483_object72687:head# 2024-04-23T16:43:52.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4c25208:::benchmark_data_smithi188_141483_object57398:head# 2024-04-23T16:43:52.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4c353e5:::benchmark_data_smithi188_141483_object31619:head# 2024-04-23T16:43:52.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4c372d1:::benchmark_data_smithi188_141483_object39518:head# 2024-04-23T16:43:52.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4c38390:::benchmark_data_smithi188_141483_object43099:head# 2024-04-23T16:43:52.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4c4b651:::benchmark_data_smithi188_141483_object41219:head# 2024-04-23T16:43:52.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4c5b272:::benchmark_data_smithi188_141483_object44456:head# 2024-04-23T16:43:52.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4c5b33b:::benchmark_data_smithi188_141483_object58483:head# 2024-04-23T16:43:52.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4c62f53:::benchmark_data_smithi188_141483_object18682:head# 2024-04-23T16:43:52.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4c6a5fe:::benchmark_data_smithi188_141483_object49544:head# 2024-04-23T16:43:52.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4c78ad2:::benchmark_data_smithi188_141483_object81264:head# 2024-04-23T16:43:52.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4c7e7ed:::benchmark_data_smithi188_141483_object82414:head# 2024-04-23T16:43:52.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4c91aa8:::benchmark_data_smithi188_141483_object50423:head# 2024-04-23T16:43:52.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4c92c59:::benchmark_data_smithi188_141483_object30497:head# 2024-04-23T16:43:52.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4c9bbb3:::benchmark_data_smithi188_141483_object40319:head# 2024-04-23T16:43:52.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4c9bd0c:::benchmark_data_smithi188_141483_object90632:head# 2024-04-23T16:43:52.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4ca8442:::benchmark_data_smithi188_141483_object53326:head# 2024-04-23T16:43:52.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4cb3117:::benchmark_data_smithi188_141483_object75298:head# 2024-04-23T16:43:52.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4cc694e:::benchmark_data_smithi188_141483_object28089:head# 2024-04-23T16:43:52.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4ccc781:::benchmark_data_smithi188_141483_object10000:head# 2024-04-23T16:43:52.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4cd85ca:::benchmark_data_smithi188_141483_object45173:head# 2024-04-23T16:43:52.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4ce04b8:::benchmark_data_smithi188_141483_object71274:head# 2024-04-23T16:43:52.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4ce4a74:::benchmark_data_smithi188_141483_object19558:head# 2024-04-23T16:43:52.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4cf0c41:::benchmark_data_smithi188_141483_object54994:head# 2024-04-23T16:43:52.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4cf2874:::benchmark_data_smithi188_141483_object46775:head# 2024-04-23T16:43:52.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d01b0b:::benchmark_data_smithi188_141483_object81424:head# 2024-04-23T16:43:52.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d02191:::benchmark_data_smithi188_141483_object83362:head# 2024-04-23T16:43:52.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d0b0d5:::benchmark_data_smithi188_141483_object73181:head# 2024-04-23T16:43:52.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d0bc5b:::benchmark_data_smithi188_141483_object3296:head# 2024-04-23T16:43:52.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d0e72f:::benchmark_data_smithi188_141483_object40904:head# 2024-04-23T16:43:52.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d10df1:::benchmark_data_smithi188_141483_object59993:head# 2024-04-23T16:43:52.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d20b43:::benchmark_data_smithi188_141483_object17408:head# 2024-04-23T16:43:52.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d24615:::benchmark_data_smithi188_141483_object17923:head# 2024-04-23T16:43:52.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d2ef00:::benchmark_data_smithi188_141483_object86865:head# 2024-04-23T16:43:52.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d32f3c:::benchmark_data_smithi188_141483_object77207:head# 2024-04-23T16:43:52.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d35d91:::benchmark_data_smithi188_141483_object32639:head# 2024-04-23T16:43:52.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d4c022:::benchmark_data_smithi188_141483_object32925:head# 2024-04-23T16:43:52.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d4c047:::benchmark_data_smithi188_141483_object68822:head# 2024-04-23T16:43:52.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d55816:::benchmark_data_smithi188_141483_object35904:head# 2024-04-23T16:43:52.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d5ce8d:::benchmark_data_smithi188_141483_object2330:head# 2024-04-23T16:43:52.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d610ed:::benchmark_data_smithi188_141483_object20744:head# 2024-04-23T16:43:52.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d7524c:::benchmark_data_smithi188_141483_object37016:head# 2024-04-23T16:43:52.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d87b9d:::benchmark_data_smithi188_141483_object14097:head# 2024-04-23T16:43:52.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d8aaff:::benchmark_data_smithi188_141483_object24425:head# 2024-04-23T16:43:52.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4d8f61d:::benchmark_data_smithi188_141483_object71351:head# 2024-04-23T16:43:52.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4da2d7d:::benchmark_data_smithi188_141483_object42879:head# 2024-04-23T16:43:52.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4da9092:::benchmark_data_smithi188_141483_object21523:head# 2024-04-23T16:43:52.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4db6338:::benchmark_data_smithi188_141483_object71706:head# 2024-04-23T16:43:52.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4dbd9dc:::benchmark_data_smithi188_141483_object75081:head# 2024-04-23T16:43:52.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4dc4a9d:::benchmark_data_smithi188_141483_object58194:head# 2024-04-23T16:43:52.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4df837a:::benchmark_data_smithi188_141483_object56994:head# 2024-04-23T16:43:52.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4dfa4f6:::benchmark_data_smithi188_141483_object36969:head# 2024-04-23T16:43:52.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4e030c5:::benchmark_data_smithi188_141483_object50222:head# 2024-04-23T16:43:52.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4e048de:::benchmark_data_smithi188_141483_object40327:head# 2024-04-23T16:43:52.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4e05b24:::benchmark_data_smithi188_141483_object65695:head# 2024-04-23T16:43:52.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4e09ace:::benchmark_data_smithi188_141483_object85897:head# 2024-04-23T16:43:52.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4e5e41d:::benchmark_data_smithi188_141483_object64498:head# 2024-04-23T16:43:52.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4e5e7ed:::benchmark_data_smithi188_141483_object53930:head# 2024-04-23T16:43:52.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4e63843:::benchmark_data_smithi188_141483_object41277:head# 2024-04-23T16:43:52.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4e69416:::benchmark_data_smithi188_141483_object84867:head# 2024-04-23T16:43:52.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4e697e0:::benchmark_data_smithi188_141483_object66056:head# 2024-04-23T16:43:52.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4e6d867:::benchmark_data_smithi188_141483_object29288:head# 2024-04-23T16:43:52.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4e8b823:::benchmark_data_smithi188_141483_object6998:head# 2024-04-23T16:43:52.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4e903f5:::benchmark_data_smithi188_141483_object63890:head# 2024-04-23T16:43:52.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4eb6ad8:::benchmark_data_smithi188_141483_object46093:head# 2024-04-23T16:43:52.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4ecb9a1:::benchmark_data_smithi188_141483_object64948:head# 2024-04-23T16:43:52.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4ed48f5:::benchmark_data_smithi188_141483_object18870:head# 2024-04-23T16:43:52.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4ee1bbe:::benchmark_data_smithi188_141483_object69037:head# 2024-04-23T16:43:52.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4eef215:::benchmark_data_smithi188_141483_object68687:head# 2024-04-23T16:43:52.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4ef04f0:::benchmark_data_smithi188_141483_object89718:head# 2024-04-23T16:43:52.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4efbcda:::benchmark_data_smithi188_141483_object40878:head# 2024-04-23T16:43:52.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f00c6a:::benchmark_data_smithi188_141483_object19813:head# 2024-04-23T16:43:52.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f01fc6:::benchmark_data_smithi188_141483_object42152:head# 2024-04-23T16:43:52.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f08789:::benchmark_data_smithi188_141483_object87566:head# 2024-04-23T16:43:52.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f0b339:::benchmark_data_smithi188_141483_object81328:head# 2024-04-23T16:43:52.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f10476:::benchmark_data_smithi188_141483_object64781:head# 2024-04-23T16:43:52.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f22da6:::benchmark_data_smithi188_141483_object78552:head# 2024-04-23T16:43:52.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f2a64b:::benchmark_data_smithi188_141483_object44761:head# 2024-04-23T16:43:52.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f2d43e:::benchmark_data_smithi188_141483_object71673:head# 2024-04-23T16:43:52.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f44dde:::benchmark_data_smithi188_141483_object86940:head# 2024-04-23T16:43:52.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f4b352:::benchmark_data_smithi188_141483_object44463:head# 2024-04-23T16:43:52.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f4d2e1:::benchmark_data_smithi188_141483_object32691:head# 2024-04-23T16:43:52.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f4eb67:::benchmark_data_smithi188_141483_object41453:head# 2024-04-23T16:43:52.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f5082d:::benchmark_data_smithi188_141483_object22224:head# 2024-04-23T16:43:52.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f56fbe:::benchmark_data_smithi188_141483_object13996:head# 2024-04-23T16:43:52.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f5a43a:::benchmark_data_smithi188_141483_object65976:head# 2024-04-23T16:43:52.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f5ced9:::benchmark_data_smithi188_141483_object34637:head# 2024-04-23T16:43:52.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f5fceb:::benchmark_data_smithi188_141483_object85890:head# 2024-04-23T16:43:52.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f6e2bd:::benchmark_data_smithi188_141483_object60725:head# 2024-04-23T16:43:52.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f7a54d:::benchmark_data_smithi188_141483_object73066:head# 2024-04-23T16:43:52.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f7e8f9:::benchmark_data_smithi188_141483_object40235:head# 2024-04-23T16:43:52.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f7ffcc:::benchmark_data_smithi188_141483_object79608:head# 2024-04-23T16:43:52.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4f910f9:::benchmark_data_smithi188_141483_object54663:head# 2024-04-23T16:43:52.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4fa169c:::benchmark_data_smithi188_141483_object5631:head# 2024-04-23T16:43:52.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4fa3b5d:::benchmark_data_smithi188_141483_object23112:head# 2024-04-23T16:43:52.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4fa6f53:::benchmark_data_smithi188_141483_object58328:head# 2024-04-23T16:43:52.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4fb93b6:::benchmark_data_smithi188_141483_object87918:head# 2024-04-23T16:43:52.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4fd86e4:::benchmark_data_smithi188_141483_object55679:head# 2024-04-23T16:43:52.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4fe6762:::benchmark_data_smithi188_141483_object26589:head# 2024-04-23T16:43:52.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4feb849:::benchmark_data_smithi188_141483_object7768:head# 2024-04-23T16:43:52.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4fef8e3:::benchmark_data_smithi188_141483_object49101:head# 2024-04-23T16:43:52.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4ff15c4:::benchmark_data_smithi188_141483_object86385:head# 2024-04-23T16:43:52.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c4ff3e6a:::benchmark_data_smithi188_141483_object6491:head# 2024-04-23T16:43:52.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c500a853:::benchmark_data_smithi188_141483_object25237:head# 2024-04-23T16:43:52.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c500be36:::benchmark_data_smithi188_141483_object9984:head# 2024-04-23T16:43:52.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c501261b:::benchmark_data_smithi188_141483_object21838:head# 2024-04-23T16:43:52.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5013328:::benchmark_data_smithi188_141483_object26885:head# 2024-04-23T16:43:52.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c501d6e1:::benchmark_data_smithi188_141483_object33275:head# 2024-04-23T16:43:52.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c501e4fa:::benchmark_data_smithi188_141483_object27477:head# 2024-04-23T16:43:52.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5028934:::benchmark_data_smithi188_141483_object71216:head# 2024-04-23T16:43:52.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c50333fe:::benchmark_data_smithi188_141483_object81298:head# 2024-04-23T16:43:52.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5037437:::benchmark_data_smithi188_141483_object62817:head# 2024-04-23T16:43:52.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5043e6f:::benchmark_data_smithi188_141483_object32977:head# 2024-04-23T16:43:52.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c504a3e3:::benchmark_data_smithi188_141483_object89267:head# 2024-04-23T16:43:52.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5050a69:::benchmark_data_smithi188_141483_object15071:head# 2024-04-23T16:43:52.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5057cf3:::benchmark_data_smithi188_141483_object21419:head# 2024-04-23T16:43:52.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c505f4d3:::benchmark_data_smithi188_141483_object54687:head# 2024-04-23T16:43:52.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5074321:::benchmark_data_smithi188_141483_object77294:head# 2024-04-23T16:43:52.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c50747d6:::benchmark_data_smithi188_141483_object41719:head# 2024-04-23T16:43:52.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c50866d8:::benchmark_data_smithi188_141483_object36491:head# 2024-04-23T16:43:52.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5091071:::benchmark_data_smithi188_141483_object61363:head# 2024-04-23T16:43:52.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c50af078:::benchmark_data_smithi188_141483_object58640:head# 2024-04-23T16:43:52.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c50c38b5:::benchmark_data_smithi188_141483_object77383:head# 2024-04-23T16:43:52.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c50d8f88:::benchmark_data_smithi188_141483_object29747:head# 2024-04-23T16:43:52.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c50e6cfc:::benchmark_data_smithi188_141483_object66154:head# 2024-04-23T16:43:52.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c50e780b:::benchmark_data_smithi188_141483_object37909:head# 2024-04-23T16:43:52.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c50ef5ac:::benchmark_data_smithi188_141483_object10349:head# 2024-04-23T16:43:52.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c50fc0d7:::benchmark_data_smithi188_141483_object29934:head# 2024-04-23T16:43:52.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c51085ad:::benchmark_data_smithi188_141483_object22188:head# 2024-04-23T16:43:52.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c511b3d0:::benchmark_data_smithi188_141483_object42570:head# 2024-04-23T16:43:52.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c51218bb:::benchmark_data_smithi188_141483_object49254:head# 2024-04-23T16:43:52.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5121e19:::benchmark_data_smithi188_141483_object44014:head# 2024-04-23T16:43:52.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c512473a:::benchmark_data_smithi188_141483_object17770:head# 2024-04-23T16:43:52.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c512e31e:::benchmark_data_smithi188_141483_object48778:head# 2024-04-23T16:43:52.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5137055:::benchmark_data_smithi188_141483_object78008:head# 2024-04-23T16:43:52.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c514098d:::benchmark_data_smithi188_141483_object35405:head# 2024-04-23T16:43:52.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c51467c3:::benchmark_data_smithi188_141483_object73710:head# 2024-04-23T16:43:52.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c514cf34:::benchmark_data_smithi188_141483_object65528:head# 2024-04-23T16:43:52.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c515054b:::benchmark_data_smithi188_141483_object14520:head# 2024-04-23T16:43:52.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5153110:::benchmark_data_smithi188_141483_object24531:head# 2024-04-23T16:43:52.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5172baf:::benchmark_data_smithi188_141483_object30648:head# 2024-04-23T16:43:52.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c519b757:::benchmark_data_smithi188_141483_object13586:head# 2024-04-23T16:43:52.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c51b8efd:::benchmark_data_smithi188_141483_object81149:head# 2024-04-23T16:43:52.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c51bb739:::benchmark_data_smithi188_141483_object22085:head# 2024-04-23T16:43:52.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c51bce8a:::benchmark_data_smithi188_141483_object31494:head# 2024-04-23T16:43:52.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c51d9014:::benchmark_data_smithi188_141483_object73515:head# 2024-04-23T16:43:52.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c51fbfec:::benchmark_data_smithi188_141483_object71474:head# 2024-04-23T16:43:52.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5206288:::benchmark_data_smithi188_141483_object53317:head# 2024-04-23T16:43:52.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5219446:::benchmark_data_smithi188_141483_object42738:head# 2024-04-23T16:43:52.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5244f48:::benchmark_data_smithi188_141483_object14670:head# 2024-04-23T16:43:52.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c524bfc4:::benchmark_data_smithi188_141483_object30998:head# 2024-04-23T16:43:52.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c524d488:::benchmark_data_smithi188_141483_object10203:head# 2024-04-23T16:43:52.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5257d24:::benchmark_data_smithi188_141483_object52812:head# 2024-04-23T16:43:52.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c52688f1:::benchmark_data_smithi188_141483_object62466:head# 2024-04-23T16:43:52.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c526f215:::benchmark_data_smithi188_141483_object57520:head# 2024-04-23T16:43:52.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c527753b:::benchmark_data_smithi188_141483_object6795:head# 2024-04-23T16:43:52.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5281195:::benchmark_data_smithi188_141483_object51416:head# 2024-04-23T16:43:52.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c529d72f:::benchmark_data_smithi188_141483_object12231:head# 2024-04-23T16:43:52.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c529e9c6:::benchmark_data_smithi188_141483_object33615:head# 2024-04-23T16:43:52.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c52be50c:::benchmark_data_smithi188_141483_object36717:head# 2024-04-23T16:43:52.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c52cc463:::benchmark_data_smithi188_141483_object8820:head# 2024-04-23T16:43:52.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c52cf1f1:::benchmark_data_smithi188_141483_object18337:head# 2024-04-23T16:43:52.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c52dc676:::benchmark_data_smithi188_141483_object40911:head# 2024-04-23T16:43:52.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c52df010:::benchmark_data_smithi188_141483_object17456:head# 2024-04-23T16:43:52.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c52e07e7:::benchmark_data_smithi188_141483_object22380:head# 2024-04-23T16:43:52.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c52e4d38:::benchmark_data_smithi188_141483_object36569:head# 2024-04-23T16:43:52.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c52ea147:::benchmark_data_smithi188_141483_object6780:head# 2024-04-23T16:43:52.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c52f9038:::benchmark_data_smithi188_141483_object60574:head# 2024-04-23T16:43:52.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5301fe2:::benchmark_data_smithi188_141483_object17524:head# 2024-04-23T16:43:52.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c530b048:::benchmark_data_smithi188_141483_object71793:head# 2024-04-23T16:43:52.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5318705:::benchmark_data_smithi188_141483_object67113:head# 2024-04-23T16:43:52.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c532a3ee:::benchmark_data_smithi188_141483_object54058:head# 2024-04-23T16:43:52.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c53444cd:::benchmark_data_smithi188_141483_object36851:head# 2024-04-23T16:43:52.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5358a08:::benchmark_data_smithi188_141483_object12259:head# 2024-04-23T16:43:52.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c535f752:::benchmark_data_smithi188_141483_object24244:head# 2024-04-23T16:43:52.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c536635f:::benchmark_data_smithi188_141483_object16716:head# 2024-04-23T16:43:52.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5385057:::benchmark_data_smithi188_141483_object6078:head# 2024-04-23T16:43:52.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5388965:::benchmark_data_smithi188_141483_object49440:head# 2024-04-23T16:43:52.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c53a2dc8:::benchmark_data_smithi188_141483_object4199:head# 2024-04-23T16:43:52.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c53aae6f:::benchmark_data_smithi188_141483_object24001:head# 2024-04-23T16:43:52.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c53acdf0:::benchmark_data_smithi188_141483_object56121:head# 2024-04-23T16:43:52.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c53b129d:::benchmark_data_smithi188_141483_object12655:head# 2024-04-23T16:43:52.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c53b60fb:::benchmark_data_smithi188_141483_object79376:head# 2024-04-23T16:43:52.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c53c108f:::benchmark_data_smithi188_141483_object46938:head# 2024-04-23T16:43:52.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c53c4f85:::benchmark_data_smithi188_141483_object27649:head# 2024-04-23T16:43:52.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c53d888b:::benchmark_data_smithi188_141483_object38441:head# 2024-04-23T16:43:52.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c53dc7cd:::benchmark_data_smithi188_141483_object56755:head# 2024-04-23T16:43:52.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c53e6b4a:::benchmark_data_smithi188_141483_object58186:head# 2024-04-23T16:43:52.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c53e747d:::benchmark_data_smithi188_141483_object49851:head# 2024-04-23T16:43:52.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c53eb717:::benchmark_data_smithi188_141483_object73815:head# 2024-04-23T16:43:52.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c53f282e:::benchmark_data_smithi188_141483_object38246:head# 2024-04-23T16:43:52.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c53f8b37:::benchmark_data_smithi188_141483_object36079:head# 2024-04-23T16:43:52.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c53fd94e:::benchmark_data_smithi188_141483_object21346:head# 2024-04-23T16:43:52.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c53fdde7:::benchmark_data_smithi188_141483_object16566:head# 2024-04-23T16:43:52.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5400a95:::benchmark_data_smithi188_141483_object48780:head# 2024-04-23T16:43:52.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c540874a:::benchmark_data_smithi188_141483_object61326:head# 2024-04-23T16:43:52.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5410166:::benchmark_data_smithi188_141483_object70760:head# 2024-04-23T16:43:52.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5416e56:::benchmark_data_smithi188_141483_object66527:head# 2024-04-23T16:43:52.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5430959:::benchmark_data_smithi188_141483_object20943:head# 2024-04-23T16:43:52.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5436cdc:::benchmark_data_smithi188_141483_object3988:head# 2024-04-23T16:43:52.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5457626:::benchmark_data_smithi188_141483_object59591:head# 2024-04-23T16:43:52.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5460787:::benchmark_data_smithi188_141483_object37248:head# 2024-04-23T16:43:52.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c546e938:::benchmark_data_smithi188_141483_object34737:head# 2024-04-23T16:43:52.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5482b4b:::benchmark_data_smithi188_141483_object75778:head# 2024-04-23T16:43:52.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5483a2a:::benchmark_data_smithi188_141483_object47520:head# 2024-04-23T16:43:52.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c54a44d0:::benchmark_data_smithi188_141483_object15269:head# 2024-04-23T16:43:52.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c54aa2ef:::benchmark_data_smithi188_141483_object47575:head# 2024-04-23T16:43:52.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c54aceed:::benchmark_data_smithi188_141483_object54321:head# 2024-04-23T16:43:52.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c54c4f38:::benchmark_data_smithi188_141483_object4114:head# 2024-04-23T16:43:52.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c54c5b78:::benchmark_data_smithi188_141483_object61211:head# 2024-04-23T16:43:52.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c54c9dc8:::benchmark_data_smithi188_141483_object28561:head# 2024-04-23T16:43:52.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c54e29a1:::benchmark_data_smithi188_141483_object29594:head# 2024-04-23T16:43:52.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c54ebc9f:::benchmark_data_smithi188_141483_object3057:head# 2024-04-23T16:43:52.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c54fab02:::benchmark_data_smithi188_141483_object72543:head# 2024-04-23T16:43:52.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c550511c:::benchmark_data_smithi188_141483_object5825:head# 2024-04-23T16:43:52.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.296 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:43:52.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5515be6:::benchmark_data_smithi188_141483_object25566:head# 2024-04-23T16:43:52.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c551ac90:::benchmark_data_smithi188_141483_object78805:head# 2024-04-23T16:43:52.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c55250b9:::benchmark_data_smithi188_141483_object40053:head# 2024-04-23T16:43:52.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c55296ea:::benchmark_data_smithi188_141483_object11127:head# 2024-04-23T16:43:52.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c552f02f:::benchmark_data_smithi188_141483_object88976:head# 2024-04-23T16:43:52.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c553f329:::benchmark_data_smithi188_141483_object50724:head# 2024-04-23T16:43:52.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c55462b7:::benchmark_data_smithi188_141483_object32713:head# 2024-04-23T16:43:52.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5546730:::benchmark_data_smithi188_141483_object50913:head# 2024-04-23T16:43:52.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5556e80:::benchmark_data_smithi188_141483_object15137:head# 2024-04-23T16:43:52.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5565403:::benchmark_data_smithi188_141483_object67484:head# 2024-04-23T16:43:52.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5569ff4:::benchmark_data_smithi188_141483_object56104:head# 2024-04-23T16:43:52.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c556a403:::benchmark_data_smithi188_141483_object46340:head# 2024-04-23T16:43:52.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c55751a5:::benchmark_data_smithi188_141483_object85409:head# 2024-04-23T16:43:52.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c557c78f:::benchmark_data_smithi188_141483_object7748:head# 2024-04-23T16:43:52.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5591105:::benchmark_data_smithi188_141483_object60065:head# 2024-04-23T16:43:52.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5593ad9:::benchmark_data_smithi188_141483_object21204:head# 2024-04-23T16:43:52.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5598413:::benchmark_data_smithi188_141483_object37847:head# 2024-04-23T16:43:52.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c559ef11:::benchmark_data_smithi188_141483_object84093:head# 2024-04-23T16:43:52.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c55aaa94:::benchmark_data_smithi188_141483_object83044:head# 2024-04-23T16:43:52.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c55acc6f:::benchmark_data_smithi188_141483_object34962:head# 2024-04-23T16:43:52.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c55aed46:::benchmark_data_smithi188_141483_object35150:head# 2024-04-23T16:43:52.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c55b6cec:::benchmark_data_smithi188_141483_object63503:head# 2024-04-23T16:43:52.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c55c9f8a:::benchmark_data_smithi188_141483_object42918:head# 2024-04-23T16:43:52.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c55da707:::benchmark_data_smithi188_141483_object32701:head# 2024-04-23T16:43:52.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c55fc793:::benchmark_data_smithi188_141483_object86505:head# 2024-04-23T16:43:52.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c55ff838:::benchmark_data_smithi188_141483_object82253:head# 2024-04-23T16:43:52.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c560164f:::benchmark_data_smithi188_141483_object57882:head# 2024-04-23T16:43:52.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5602abd:::benchmark_data_smithi188_141483_object25062:head# 2024-04-23T16:43:52.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c560c98d:::benchmark_data_smithi188_141483_object2628:head# 2024-04-23T16:43:52.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c561a178:::benchmark_data_smithi188_141483_object52454:head# 2024-04-23T16:43:52.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c561d988:::benchmark_data_smithi188_141483_object5900:head# 2024-04-23T16:43:52.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c562469a:::benchmark_data_smithi188_141483_object38130:head# 2024-04-23T16:43:52.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c562a3d8:::benchmark_data_smithi188_141483_object50283:head# 2024-04-23T16:43:52.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c564787e:::benchmark_data_smithi188_141483_object28263:head# 2024-04-23T16:43:52.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c566b0a3:::benchmark_data_smithi188_141483_object80900:head# 2024-04-23T16:43:52.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c56792f6:::benchmark_data_smithi188_141483_object87062:head# 2024-04-23T16:43:52.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c56889c9:::benchmark_data_smithi188_141483_object29917:head# 2024-04-23T16:43:52.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5699b1e:::benchmark_data_smithi188_141483_object4435:head# 2024-04-23T16:43:52.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c569cb0d:::benchmark_data_smithi188_141483_object80404:head# 2024-04-23T16:43:52.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c56a53eb:::benchmark_data_smithi188_141483_object48775:head# 2024-04-23T16:43:52.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c56b0c23:::benchmark_data_smithi188_141483_object47716:head# 2024-04-23T16:43:52.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c56d534b:::benchmark_data_smithi188_141483_object89468:head# 2024-04-23T16:43:52.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c56e2705:::benchmark_data_smithi188_141483_object78871:head# 2024-04-23T16:43:52.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c56e2dea:::benchmark_data_smithi188_141483_object27136:head# 2024-04-23T16:43:52.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c571b4fc:::benchmark_data_smithi188_141483_object20825:head# 2024-04-23T16:43:52.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c571b958:::benchmark_data_smithi188_141483_object75645:head# 2024-04-23T16:43:52.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c571d335:::benchmark_data_smithi188_141483_object23728:head# 2024-04-23T16:43:52.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c572985c:::benchmark_data_smithi188_141483_object5209:head# 2024-04-23T16:43:52.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c573314d:::benchmark_data_smithi188_141483_object83399:head# 2024-04-23T16:43:52.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5744e84:::benchmark_data_smithi188_141483_object49259:head# 2024-04-23T16:43:52.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c574b339:::benchmark_data_smithi188_141483_object5692:head# 2024-04-23T16:43:52.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c576375c:::benchmark_data_smithi188_141483_object77434:head# 2024-04-23T16:43:52.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5767eeb:::benchmark_data_smithi188_141483_object64894:head# 2024-04-23T16:43:52.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5794460:::benchmark_data_smithi188_141483_object86184:head# 2024-04-23T16:43:52.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c579b555:::benchmark_data_smithi188_141483_object9636:head# 2024-04-23T16:43:52.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c57b367e:::benchmark_data_smithi188_141483_object14274:head# 2024-04-23T16:43:52.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c57b71ae:::benchmark_data_smithi188_141483_object82657:head# 2024-04-23T16:43:52.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c57c9ffd:::benchmark_data_smithi188_141483_object49587:head# 2024-04-23T16:43:52.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c57cb50f:::benchmark_data_smithi188_141483_object55884:head# 2024-04-23T16:43:52.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c57e0ff2:::benchmark_data_smithi188_141483_object22830:head# 2024-04-23T16:43:52.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c57eca76:::benchmark_data_smithi188_141483_object47590:head# 2024-04-23T16:43:52.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c57fdc16:::benchmark_data_smithi188_141483_object7796:head# 2024-04-23T16:43:52.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c57fe4a7:::benchmark_data_smithi188_141483_object20708:head# 2024-04-23T16:43:52.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c57fe748:::benchmark_data_smithi188_141483_object71658:head# 2024-04-23T16:43:52.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5810e52:::benchmark_data_smithi188_141483_object26339:head# 2024-04-23T16:43:52.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5811008:::benchmark_data_smithi188_141483_object81504:head# 2024-04-23T16:43:52.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5834eda:::benchmark_data_smithi188_141483_object75860:head# 2024-04-23T16:43:52.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5835189:::benchmark_data_smithi188_141483_object64757:head# 2024-04-23T16:43:52.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c583eead:::benchmark_data_smithi188_141483_object13222:head# 2024-04-23T16:43:52.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5870cfd:::benchmark_data_smithi188_141483_object13425:head# 2024-04-23T16:43:52.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c587ba5d:::benchmark_data_smithi188_141483_object88627:head# 2024-04-23T16:43:52.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5897a19:::benchmark_data_smithi188_141483_object89202:head# 2024-04-23T16:43:52.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c58d46c3:::benchmark_data_smithi188_141483_object49240:head# 2024-04-23T16:43:52.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c58deeb3:::benchmark_data_smithi188_141483_object65472:head# 2024-04-23T16:43:52.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c58e1704:::benchmark_data_smithi188_141483_object3330:head# 2024-04-23T16:43:52.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c58e8f56:::benchmark_data_smithi188_141483_object64206:head# 2024-04-23T16:43:52.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c58f78ae:::benchmark_data_smithi188_141483_object15179:head# 2024-04-23T16:43:52.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c59146d8:::benchmark_data_smithi188_141483_object82498:head# 2024-04-23T16:43:52.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5920238:::benchmark_data_smithi188_141483_object38430:head# 2024-04-23T16:43:52.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c592112f:::benchmark_data_smithi188_141483_object208:head# 2024-04-23T16:43:52.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5925f38:::benchmark_data_smithi188_141483_object54983:head# 2024-04-23T16:43:52.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c592733d:::benchmark_data_smithi188_141483_object84707:head# 2024-04-23T16:43:52.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c592a276:::benchmark_data_smithi188_141483_object86188:head# 2024-04-23T16:43:52.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c592c7c8:::benchmark_data_smithi188_141483_object5427:head# 2024-04-23T16:43:52.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c593b1e0:::benchmark_data_smithi188_141483_object64602:head# 2024-04-23T16:43:52.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5940134:::benchmark_data_smithi188_141483_object47920:head# 2024-04-23T16:43:52.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5944eb0:::benchmark_data_smithi188_141483_object37542:head# 2024-04-23T16:43:52.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5960220:::benchmark_data_smithi188_141483_object64947:head# 2024-04-23T16:43:52.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5978b0f:::benchmark_data_smithi188_141483_object7793:head# 2024-04-23T16:43:52.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5983f98:::benchmark_data_smithi188_141483_object79250:head# 2024-04-23T16:43:52.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c59960cc:::benchmark_data_smithi188_141483_object24817:head# 2024-04-23T16:43:52.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c599666d:::benchmark_data_smithi188_141483_object81092:head# 2024-04-23T16:43:52.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c59a39fc:::benchmark_data_smithi188_141483_object5922:head# 2024-04-23T16:43:52.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c59be33c:::benchmark_data_smithi188_141483_object57517:head# 2024-04-23T16:43:52.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c59c32b4:::benchmark_data_smithi188_141483_object17087:head# 2024-04-23T16:43:52.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c59d7710:::benchmark_data_smithi188_141483_object29247:head# 2024-04-23T16:43:52.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c59f2771:::benchmark_data_smithi188_141483_object54784:head# 2024-04-23T16:43:52.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c59fe2d0:::benchmark_data_smithi188_141483_object35195:head# 2024-04-23T16:43:52.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5a032cc:::benchmark_data_smithi188_141483_object85411:head# 2024-04-23T16:43:52.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5a182c4:::benchmark_data_smithi188_141483_object37589:head# 2024-04-23T16:43:52.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5a1f4bb:::benchmark_data_smithi188_141483_object27626:head# 2024-04-23T16:43:52.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5a2a4a6:::benchmark_data_smithi188_141483_object3681:head# 2024-04-23T16:43:52.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5a33be2:::benchmark_data_smithi188_141483_object73112:head# 2024-04-23T16:43:52.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5a3eaf2:::benchmark_data_smithi188_141483_object67750:head# 2024-04-23T16:43:52.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5a50575:::benchmark_data_smithi188_141483_object60135:head# 2024-04-23T16:43:52.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5a51a52:::benchmark_data_smithi188_141483_object6002:head# 2024-04-23T16:43:52.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5a7e19c:::benchmark_data_smithi188_141483_object14801:head# 2024-04-23T16:43:52.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5a7fe8a:::benchmark_data_smithi188_141483_object45023:head# 2024-04-23T16:43:52.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5a84d3e:::benchmark_data_smithi188_141483_object35418:head# 2024-04-23T16:43:52.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5a91f25:::benchmark_data_smithi188_141483_object17662:head# 2024-04-23T16:43:52.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5a97749:::benchmark_data_smithi188_141483_object36666:head# 2024-04-23T16:43:52.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5aa1550:::benchmark_data_smithi188_141483_object3954:head# 2024-04-23T16:43:52.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5aab806:::benchmark_data_smithi188_141483_object87288:head# 2024-04-23T16:43:52.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5abd72f:::benchmark_data_smithi188_141483_object16879:head# 2024-04-23T16:43:52.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5abe49e:::benchmark_data_smithi188_141483_object58902:head# 2024-04-23T16:43:52.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5ae3c57:::benchmark_data_smithi188_141483_object2624:head# 2024-04-23T16:43:52.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5b0dfef:::benchmark_data_smithi188_141483_object52322:head# 2024-04-23T16:43:52.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5b14e13:::benchmark_data_smithi188_141483_object38955:head# 2024-04-23T16:43:52.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5b2897c:::benchmark_data_smithi188_141483_object3583:head# 2024-04-23T16:43:52.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5b2a268:::benchmark_data_smithi188_141483_object14064:head# 2024-04-23T16:43:52.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5b4501c:::benchmark_data_smithi188_141483_object76746:head# 2024-04-23T16:43:52.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5b53778:::benchmark_data_smithi188_141483_object25843:head# 2024-04-23T16:43:52.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5b5c685:::benchmark_data_smithi188_141483_object68203:head# 2024-04-23T16:43:52.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5b6dd55:::benchmark_data_smithi188_141483_object70415:head# 2024-04-23T16:43:52.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5b76e8c:::benchmark_data_smithi188_141483_object33473:head# 2024-04-23T16:43:52.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5b7ce9d:::benchmark_data_smithi188_141483_object85619:head# 2024-04-23T16:43:52.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5b89efa:::benchmark_data_smithi188_141483_object31839:head# 2024-04-23T16:43:52.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5b9c745:::benchmark_data_smithi188_141483_object10969:head# 2024-04-23T16:43:52.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5ba155e:::benchmark_data_smithi188_141483_object22181:head# 2024-04-23T16:43:52.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5ba8eff:::benchmark_data_smithi188_141483_object1806:head# 2024-04-23T16:43:52.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5bbbf5e:::benchmark_data_smithi188_141483_object39648:head# 2024-04-23T16:43:52.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5bc1ff7:::benchmark_data_smithi188_141483_object42998:head# 2024-04-23T16:43:52.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5bd4d45:::benchmark_data_smithi188_141483_object81873:head# 2024-04-23T16:43:52.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5bd520c:::benchmark_data_smithi188_141483_object52916:head# 2024-04-23T16:43:52.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5bd92b9:::benchmark_data_smithi188_141483_object69911:head# 2024-04-23T16:43:52.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5bf23b8:::benchmark_data_smithi188_141483_object82449:head# 2024-04-23T16:43:52.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5c01bde:::benchmark_data_smithi188_141483_object89893:head# 2024-04-23T16:43:52.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5c1ed8f:::benchmark_data_smithi188_141483_object85190:head# 2024-04-23T16:43:52.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5c3411c:::benchmark_data_smithi188_141483_object59073:head# 2024-04-23T16:43:52.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5c3afd7:::benchmark_data_smithi188_141483_object34339:head# 2024-04-23T16:43:52.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5c4d8bf:::benchmark_data_smithi188_141483_object62277:head# 2024-04-23T16:43:52.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5c4dd39:::benchmark_data_smithi188_141483_object36853:head# 2024-04-23T16:43:52.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5c5d0be:::benchmark_data_smithi188_141483_object6169:head# 2024-04-23T16:43:52.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5c60a45:::benchmark_data_smithi188_141483_object49715:head# 2024-04-23T16:43:52.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5c63f4f:::benchmark_data_smithi188_141483_object72741:head# 2024-04-23T16:43:52.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5c6a3e8:::benchmark_data_smithi188_141483_object4824:head# 2024-04-23T16:43:52.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5c6b676:::benchmark_data_smithi188_141483_object67416:head# 2024-04-23T16:43:52.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5c78389:::benchmark_data_smithi188_141483_object53951:head# 2024-04-23T16:43:52.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5c7b25d:::benchmark_data_smithi188_141483_object61420:head# 2024-04-23T16:43:52.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.432 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5c7de02:::benchmark_data_smithi188_141483_object2619:head# 2024-04-23T16:43:52.432 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.432 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5c7f330:::benchmark_data_smithi188_141483_object64812:head# 2024-04-23T16:43:52.432 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.432 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5c9e37c:::benchmark_data_smithi188_141483_object9420:head# 2024-04-23T16:43:52.432 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.432 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5ca1e56:::benchmark_data_smithi188_141483_object86261:head# 2024-04-23T16:43:52.432 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.432 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5ca9197:::benchmark_data_smithi188_141483_object69907:head# 2024-04-23T16:43:52.432 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.432 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5cb7a52:::benchmark_data_smithi188_141483_object59560:head# 2024-04-23T16:43:52.432 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.432 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5cc7d28:::benchmark_data_smithi188_141483_object5645:head# 2024-04-23T16:43:52.432 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5ce483d:::benchmark_data_smithi188_141483_object20865:head# 2024-04-23T16:43:52.433 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5ce5a4d:::benchmark_data_smithi188_141483_object60873:head# 2024-04-23T16:43:52.433 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5cf4978:::benchmark_data_smithi188_141483_object62129:head# 2024-04-23T16:43:52.433 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5cf8b91:::benchmark_data_smithi188_141483_object71832:head# 2024-04-23T16:43:52.433 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5d0c26a:::benchmark_data_smithi188_141483_object72022:head# 2024-04-23T16:43:52.433 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5d0c3ff:::benchmark_data_smithi188_141483_object75276:head# 2024-04-23T16:43:52.433 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.433 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5d0d56b:::benchmark_data_smithi188_141483_object50737:head# 2024-04-23T16:43:52.434 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.434 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5d12ef4:::benchmark_data_smithi188_141483_object85488:head# 2024-04-23T16:43:52.434 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.434 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5d18f02:::benchmark_data_smithi188_141483_object86812:head# 2024-04-23T16:43:52.434 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.434 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5d1ea17:::benchmark_data_smithi188_141483_object36030:head# 2024-04-23T16:43:52.434 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.434 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5d42b90:::benchmark_data_smithi188_141483_object80667:head# 2024-04-23T16:43:52.434 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.434 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5d44ce5:::benchmark_data_smithi188_141483_object32348:head# 2024-04-23T16:43:52.434 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.434 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5d4f52d:::benchmark_data_smithi188_141483_object57103:head# 2024-04-23T16:43:52.435 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.435 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5d862c8:::benchmark_data_smithi188_141483_object84974:head# 2024-04-23T16:43:52.435 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.435 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5d8973d:::benchmark_data_smithi188_141483_object35185:head# 2024-04-23T16:43:52.435 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.435 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5da12b2:::benchmark_data_smithi188_141483_object74734:head# 2024-04-23T16:43:52.435 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.435 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5da6be4:::benchmark_data_smithi188_141483_object87982:head# 2024-04-23T16:43:52.435 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.435 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5da7e88:::benchmark_data_smithi188_141483_object80167:head# 2024-04-23T16:43:52.435 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.435 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5db76a0:::benchmark_data_smithi188_141483_object40880:head# 2024-04-23T16:43:52.435 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5dd7b82:::benchmark_data_smithi188_141483_object29536:head# 2024-04-23T16:43:52.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5ddda4e:::benchmark_data_smithi188_141483_object74126:head# 2024-04-23T16:43:52.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5deea2d:::benchmark_data_smithi188_141483_object48747:head# 2024-04-23T16:43:52.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e0551a:::benchmark_data_smithi188_141483_object85640:head# 2024-04-23T16:43:52.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e0dc4d:::benchmark_data_smithi188_141483_object63949:head# 2024-04-23T16:43:52.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e0df13:::benchmark_data_smithi188_141483_object67168:head# 2024-04-23T16:43:52.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.436 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e11927:::benchmark_data_smithi188_141483_object58453:head# 2024-04-23T16:43:52.436 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e17753:::benchmark_data_smithi188_141483_object51155:head# 2024-04-23T16:43:52.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e18684:::benchmark_data_smithi188_141483_object27928:head# 2024-04-23T16:43:52.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e1b2a4:::benchmark_data_smithi188_141483_object5931:head# 2024-04-23T16:43:52.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e1e63a:::benchmark_data_smithi188_141483_object69545:head# 2024-04-23T16:43:52.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e26607:::benchmark_data_smithi188_141483_object15764:head# 2024-04-23T16:43:52.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e36d97:::benchmark_data_smithi188_141483_object78167:head# 2024-04-23T16:43:52.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.437 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e3e238:::benchmark_data_smithi188_141483_object63400:head# 2024-04-23T16:43:52.437 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.438 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e445b0:::benchmark_data_smithi188_141483_object77582:head# 2024-04-23T16:43:52.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.438 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e47496:::benchmark_data_smithi188_141483_object66576:head# 2024-04-23T16:43:52.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.438 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e59043:::benchmark_data_smithi188_141483_object35118:head# 2024-04-23T16:43:52.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.438 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e7282e:::benchmark_data_smithi188_141483_object83845:head# 2024-04-23T16:43:52.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.438 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e7a503:::benchmark_data_smithi188_141483_object48574:head# 2024-04-23T16:43:52.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.438 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e807c6:::benchmark_data_smithi188_141483_object34449:head# 2024-04-23T16:43:52.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.438 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e8c405:::benchmark_data_smithi188_141483_object66382:head# 2024-04-23T16:43:52.438 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e960a0:::benchmark_data_smithi188_141483_object52259:head# 2024-04-23T16:43:52.439 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e98137:::benchmark_data_smithi188_141483_object33343:head# 2024-04-23T16:43:52.439 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5e99f53:::benchmark_data_smithi188_141483_object85525:head# 2024-04-23T16:43:52.439 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5eaf30e:::benchmark_data_smithi188_141483_object10929:head# 2024-04-23T16:43:52.439 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5ecc7a4:::benchmark_data_smithi188_141483_object80668:head# 2024-04-23T16:43:52.439 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5ed0d10:::benchmark_data_smithi188_141483_object1225:head# 2024-04-23T16:43:52.439 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.439 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5ed8d23:::benchmark_data_smithi188_141483_object60106:head# 2024-04-23T16:43:52.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5f00816:::benchmark_data_smithi188_141483_object72237:head# 2024-04-23T16:43:52.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5f01e9b:::benchmark_data_smithi188_141483_object87443:head# 2024-04-23T16:43:52.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5f2bd43:::benchmark_data_smithi188_141483_object719:head# 2024-04-23T16:43:52.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5f46e0d:::benchmark_data_smithi188_141483_object89415:head# 2024-04-23T16:43:52.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5f47221:::benchmark_data_smithi188_141483_object77779:head# 2024-04-23T16:43:52.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.440 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5f51b00:::benchmark_data_smithi188_141483_object21495:head# 2024-04-23T16:43:52.440 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5f64bd1:::benchmark_data_smithi188_141483_object27606:head# 2024-04-23T16:43:52.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5f6d625:::benchmark_data_smithi188_141483_object90307:head# 2024-04-23T16:43:52.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5f7302b:::benchmark_data_smithi188_141483_object30740:head# 2024-04-23T16:43:52.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5f7607c:::benchmark_data_smithi188_141483_object66081:head# 2024-04-23T16:43:52.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5f7bfae:::benchmark_data_smithi188_141483_object45226:head# 2024-04-23T16:43:52.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5f92b01:::benchmark_data_smithi188_141483_object62413:head# 2024-04-23T16:43:52.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.441 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5f94c35:::benchmark_data_smithi188_141483_object36741:head# 2024-04-23T16:43:52.441 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5f9ae1d:::benchmark_data_smithi188_141483_object62309:head# 2024-04-23T16:43:52.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5fb03b4:::benchmark_data_smithi188_141483_object63812:head# 2024-04-23T16:43:52.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5fb193c:::benchmark_data_smithi188_141483_object14356:head# 2024-04-23T16:43:52.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5fb1c28:::benchmark_data_smithi188_141483_object16190:head# 2024-04-23T16:43:52.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5fb79cb:::benchmark_data_smithi188_141483_object14205:head# 2024-04-23T16:43:52.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c5fc116f:::benchmark_data_smithi188_141483_object33125:head# 2024-04-23T16:43:52.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.442 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6001557:::benchmark_data_smithi188_141483_object5089:head# 2024-04-23T16:43:52.442 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c600d158:::benchmark_data_smithi188_141483_object25524:head# 2024-04-23T16:43:52.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c601530a:::benchmark_data_smithi188_141483_object11727:head# 2024-04-23T16:43:52.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c602261c:::benchmark_data_smithi188_141483_object39587:head# 2024-04-23T16:43:52.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c602f538:::benchmark_data_smithi188_141483_object59921:head# 2024-04-23T16:43:52.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c602fa1b:::benchmark_data_smithi188_141483_object5780:head# 2024-04-23T16:43:52.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c60398fc:::benchmark_data_smithi188_141483_object86036:head# 2024-04-23T16:43:52.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.443 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c603c971:::benchmark_data_smithi188_141483_object79625:head# 2024-04-23T16:43:52.443 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6042328:::benchmark_data_smithi188_141483_object23968:head# 2024-04-23T16:43:52.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6042bbb:::benchmark_data_smithi188_141483_object18283:head# 2024-04-23T16:43:52.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c60491da:::benchmark_data_smithi188_141483_object16679:head# 2024-04-23T16:43:52.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6049bda:::benchmark_data_smithi188_141483_object90200:head# 2024-04-23T16:43:52.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6073cc0:::benchmark_data_smithi188_141483_object88773:head# 2024-04-23T16:43:52.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c60744db:::benchmark_data_smithi188_141483_object72540:head# 2024-04-23T16:43:52.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.444 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c608b26c:::benchmark_data_smithi188_141483_object9332:head# 2024-04-23T16:43:52.444 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.445 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c609016a:::benchmark_data_smithi188_141483_object77238:head# 2024-04-23T16:43:52.445 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.445 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c609421e:::benchmark_data_smithi188_141483_object39191:head# 2024-04-23T16:43:52.445 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.445 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c60976d2:::benchmark_data_smithi188_141483_object77264:head# 2024-04-23T16:43:52.445 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.445 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6097b68:::benchmark_data_smithi188_141483_object8689:head# 2024-04-23T16:43:52.445 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.445 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c609fdbe:::benchmark_data_smithi188_141483_object31383:head# 2024-04-23T16:43:52.445 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.445 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c60aecaa:::benchmark_data_smithi188_141483_object25184:head# 2024-04-23T16:43:52.445 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.445 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c60c049f:::benchmark_data_smithi188_141483_object30108:head# 2024-04-23T16:43:52.445 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.446 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c60c21db:::benchmark_data_smithi188_141483_object75402:head# 2024-04-23T16:43:52.446 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.446 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c60c5a3c:::benchmark_data_smithi188_141483_object48979:head# 2024-04-23T16:43:52.446 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.446 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c60d5eed:::benchmark_data_smithi188_141483_object71528:head# 2024-04-23T16:43:52.446 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.446 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6105582:::benchmark_data_smithi188_141483_object25795:head# 2024-04-23T16:43:52.446 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.446 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c610972d:::benchmark_data_smithi188_141483_object41966:head# 2024-04-23T16:43:52.446 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.446 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c610b0b5:::benchmark_data_smithi188_141483_object53139:head# 2024-04-23T16:43:52.446 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.446 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c61132bb:::benchmark_data_smithi188_141483_object29319:head# 2024-04-23T16:43:52.447 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.447 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6115be8:::benchmark_data_smithi188_141483_object88070:head# 2024-04-23T16:43:52.447 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.447 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c611f385:::benchmark_data_smithi188_141483_object85219:head# 2024-04-23T16:43:52.447 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.447 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c612035b:::benchmark_data_smithi188_141483_object39773:head# 2024-04-23T16:43:52.447 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.447 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6120fcd:::benchmark_data_smithi188_141483_object77362:head# 2024-04-23T16:43:52.447 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.447 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c613785a:::benchmark_data_smithi188_141483_object46872:head# 2024-04-23T16:43:52.447 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.447 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6141067:::benchmark_data_smithi188_141483_object13635:head# 2024-04-23T16:43:52.447 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6141076:::benchmark_data_smithi188_141483_object77163:head# 2024-04-23T16:43:52.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c61420c4:::benchmark_data_smithi188_141483_object36242:head# 2024-04-23T16:43:52.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6163dc2:::benchmark_data_smithi188_141483_object32082:head# 2024-04-23T16:43:52.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6176d91:::benchmark_data_smithi188_141483_object41636:head# 2024-04-23T16:43:52.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c618e1a2:::benchmark_data_smithi188_141483_object89009:head# 2024-04-23T16:43:52.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c618e30e:::benchmark_data_smithi188_141483_object65159:head# 2024-04-23T16:43:52.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.448 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c61a7cb0:::benchmark_data_smithi188_141483_object76771:head# 2024-04-23T16:43:52.448 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c61a8467:::benchmark_data_smithi188_141483_object35759:head# 2024-04-23T16:43:52.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c61d2b94:::benchmark_data_smithi188_141483_object26568:head# 2024-04-23T16:43:52.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c61d3922:::benchmark_data_smithi188_141483_object12292:head# 2024-04-23T16:43:52.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c61dda3a:::benchmark_data_smithi188_141483_object47483:head# 2024-04-23T16:43:52.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c61e4528:::benchmark_data_smithi188_141483_object31446:head# 2024-04-23T16:43:52.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c61e6963:::benchmark_data_smithi188_141483_object71656:head# 2024-04-23T16:43:52.449 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.449 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6210399:::benchmark_data_smithi188_141483_object7527:head# 2024-04-23T16:43:52.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6211340:::benchmark_data_smithi188_141483_object81799:head# 2024-04-23T16:43:52.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c62152b7:::benchmark_data_smithi188_141483_object42183:head# 2024-04-23T16:43:52.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6216d4b:::benchmark_data_smithi188_141483_object57474:head# 2024-04-23T16:43:52.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6251542:::benchmark_data_smithi188_141483_object41699:head# 2024-04-23T16:43:52.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6260eb6:::benchmark_data_smithi188_141483_object61630:head# 2024-04-23T16:43:52.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6266353:::benchmark_data_smithi188_141483_object24894:head# 2024-04-23T16:43:52.450 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.450 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c626f10f:::benchmark_data_smithi188_141483_object7360:head# 2024-04-23T16:43:52.451 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.451 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6270d47:::benchmark_data_smithi188_141483_object45629:head# 2024-04-23T16:43:52.451 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.451 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6277819:::benchmark_data_smithi188_141483_object22862:head# 2024-04-23T16:43:52.451 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.451 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c627eb78:::benchmark_data_smithi188_141483_object12509:head# 2024-04-23T16:43:52.451 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.451 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c62aebbb:::benchmark_data_smithi188_141483_object30041:head# 2024-04-23T16:43:52.451 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.451 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c62bc11d:::benchmark_data_smithi188_141483_object65891:head# 2024-04-23T16:43:52.451 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.451 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c62c647e:::benchmark_data_smithi188_141483_object45374:head# 2024-04-23T16:43:52.451 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.452 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c62ca1f5:::benchmark_data_smithi188_141483_object48849:head# 2024-04-23T16:43:52.452 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.452 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c62e33cd:::benchmark_data_smithi188_141483_object35602:head# 2024-04-23T16:43:52.452 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.452 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c62e3c04:::benchmark_data_smithi188_141483_object21015:head# 2024-04-23T16:43:52.452 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.452 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c62e5188:::benchmark_data_smithi188_141483_object18797:head# 2024-04-23T16:43:52.452 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.452 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c62e6982:::benchmark_data_smithi188_141483_object74987:head# 2024-04-23T16:43:52.452 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.452 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c62eec54:::benchmark_data_smithi188_141483_object31885:head# 2024-04-23T16:43:52.452 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.452 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c62fa7e7:::benchmark_data_smithi188_141483_object36236:head# 2024-04-23T16:43:52.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c63115ea:::benchmark_data_smithi188_141483_object14150:head# 2024-04-23T16:43:52.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6314847:::benchmark_data_smithi188_141483_object21722:head# 2024-04-23T16:43:52.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c63175e1:::benchmark_data_smithi188_141483_object8106:head# 2024-04-23T16:43:52.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c63179f1:::benchmark_data_smithi188_141483_object29367:head# 2024-04-23T16:43:52.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c633dfe5:::benchmark_data_smithi188_141483_object15628:head# 2024-04-23T16:43:52.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.453 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c634c9ae:::benchmark_data_smithi188_141483_object66292:head# 2024-04-23T16:43:52.453 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.454 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c634d877:::benchmark_data_smithi188_141483_object4593:head# 2024-04-23T16:43:52.454 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.454 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c635561c:::benchmark_data_smithi188_141483_object66076:head# 2024-04-23T16:43:52.454 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.454 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6369ec9:::benchmark_data_smithi188_141483_object71980:head# 2024-04-23T16:43:52.454 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.454 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c638136a:::benchmark_data_smithi188_141483_object46088:head# 2024-04-23T16:43:52.454 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.454 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c63834a6:::benchmark_data_smithi188_141483_object41880:head# 2024-04-23T16:43:52.454 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.454 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c63a4702:::benchmark_data_smithi188_141483_object64591:head# 2024-04-23T16:43:52.454 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.454 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c63a600f:::benchmark_data_smithi188_141483_object7737:head# 2024-04-23T16:43:52.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c63ac4b8:::benchmark_data_smithi188_141483_object49922:head# 2024-04-23T16:43:52.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c63bf066:::benchmark_data_smithi188_141483_object62513:head# 2024-04-23T16:43:52.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c63c7a31:::benchmark_data_smithi188_141483_object50095:head# 2024-04-23T16:43:52.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c63ce6c2:::benchmark_data_smithi188_141483_object60270:head# 2024-04-23T16:43:52.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c63d75ae:::benchmark_data_smithi188_141483_object72330:head# 2024-04-23T16:43:52.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.455 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c63df69f:::benchmark_data_smithi188_141483_object86503:head# 2024-04-23T16:43:52.455 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c63e6f31:::benchmark_data_smithi188_141483_object83906:head# 2024-04-23T16:43:52.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c63f136f:::benchmark_data_smithi188_141483_object21039:head# 2024-04-23T16:43:52.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c63f32b1:::benchmark_data_smithi188_141483_object10004:head# 2024-04-23T16:43:52.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c640296f:::benchmark_data_smithi188_141483_object87950:head# 2024-04-23T16:43:52.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6413fe1:::benchmark_data_smithi188_141483_object15505:head# 2024-04-23T16:43:52.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c641d1b9:::benchmark_data_smithi188_141483_object51381:head# 2024-04-23T16:43:52.456 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.456 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c64218f0:::benchmark_data_smithi188_141483_object27630:head# 2024-04-23T16:43:52.457 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.457 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c642545c:::benchmark_data_smithi188_141483_object69074:head# 2024-04-23T16:43:52.457 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.457 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6425cc5:::benchmark_data_smithi188_141483_object69289:head# 2024-04-23T16:43:52.457 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.457 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c643292e:::benchmark_data_smithi188_141483_object81205:head# 2024-04-23T16:43:52.457 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.457 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6455c9c:::benchmark_data_smithi188_141483_object18752:head# 2024-04-23T16:43:52.457 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.457 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6465b6d:::benchmark_data_smithi188_141483_object73527:head# 2024-04-23T16:43:52.457 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.457 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c64769dc:::benchmark_data_smithi188_141483_object63999:head# 2024-04-23T16:43:52.457 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.457 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6477044:::benchmark_data_smithi188_141483_object3634:head# 2024-04-23T16:43:52.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c647c701:::benchmark_data_smithi188_141483_object53093:head# 2024-04-23T16:43:52.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c649bb6b:::benchmark_data_smithi188_141483_object37489:head# 2024-04-23T16:43:52.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c64a6778:::benchmark_data_smithi188_141483_object55180:head# 2024-04-23T16:43:52.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c64aa44b:::benchmark_data_smithi188_141483_object47598:head# 2024-04-23T16:43:52.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c64b58ce:::benchmark_data_smithi188_141483_object43717:head# 2024-04-23T16:43:52.458 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.458 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c64d2f10:::benchmark_data_smithi188_141483_object24428:head# 2024-04-23T16:43:52.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c64dbd61:::benchmark_data_smithi188_141483_object30471:head# 2024-04-23T16:43:52.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c64e91e3:::benchmark_data_smithi188_141483_object76020:head# 2024-04-23T16:43:52.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c65005bc:::benchmark_data_smithi188_141483_object57039:head# 2024-04-23T16:43:52.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c650db3a:::benchmark_data_smithi188_141483_object76347:head# 2024-04-23T16:43:52.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6512883:::benchmark_data_smithi188_141483_object62254:head# 2024-04-23T16:43:52.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c653c612:::benchmark_data_smithi188_141483_object76023:head# 2024-04-23T16:43:52.459 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.459 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c65446ef:::benchmark_data_smithi188_141483_object73383:head# 2024-04-23T16:43:52.460 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.460 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c65558c7:::benchmark_data_smithi188_141483_object76961:head# 2024-04-23T16:43:52.460 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.460 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6559512:::benchmark_data_smithi188_141483_object22774:head# 2024-04-23T16:43:52.460 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.460 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6563e2e:::benchmark_data_smithi188_141483_object48768:head# 2024-04-23T16:43:52.460 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.460 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c65697ca:::benchmark_data_smithi188_141483_object32389:head# 2024-04-23T16:43:52.460 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.460 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c658159e:::benchmark_data_smithi188_141483_object72336:head# 2024-04-23T16:43:52.460 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.460 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c65925c5:::benchmark_data_smithi188_141483_object35473:head# 2024-04-23T16:43:52.460 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.460 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c65a4cc2:::benchmark_data_smithi188_141483_object86026:head# 2024-04-23T16:43:52.461 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.461 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c65ac111:::benchmark_data_smithi188_141483_object49202:head# 2024-04-23T16:43:52.461 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.461 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c65b5a6a:::benchmark_data_smithi188_141483_object15727:head# 2024-04-23T16:43:52.461 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.461 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c65b7bf7:::benchmark_data_smithi188_141483_object975:head# 2024-04-23T16:43:52.461 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.461 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c65ba40e:::benchmark_data_smithi188_141483_object45808:head# 2024-04-23T16:43:52.461 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.461 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c65c1971:::benchmark_data_smithi188_141483_object28665:head# 2024-04-23T16:43:52.461 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.461 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c65dc094:::benchmark_data_smithi188_141483_object62147:head# 2024-04-23T16:43:52.461 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c65e6433:::benchmark_data_smithi188_141483_object84866:head# 2024-04-23T16:43:52.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c65ead8d:::benchmark_data_smithi188_141483_object67664:head# 2024-04-23T16:43:52.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c65eb043:::benchmark_data_smithi188_141483_object62347:head# 2024-04-23T16:43:52.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c65ff60f:::benchmark_data_smithi188_141483_object63611:head# 2024-04-23T16:43:52.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6608086:::benchmark_data_smithi188_141483_object25882:head# 2024-04-23T16:43:52.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c660ea3e:::benchmark_data_smithi188_141483_object87100:head# 2024-04-23T16:43:52.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.462 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6610039:::benchmark_data_smithi188_141483_object68689:head# 2024-04-23T16:43:52.462 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c661a3d2:::benchmark_data_smithi188_141483_object44927:head# 2024-04-23T16:43:52.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6643967:::benchmark_data_smithi188_141483_object78001:head# 2024-04-23T16:43:52.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6649f4b:::benchmark_data_smithi188_141483_object11370:head# 2024-04-23T16:43:52.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6653348:::benchmark_data_smithi188_141483_object69981:head# 2024-04-23T16:43:52.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c66533bf:::benchmark_data_smithi188_141483_object10672:head# 2024-04-23T16:43:52.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c665c159:::benchmark_data_smithi188_141483_object87450:head# 2024-04-23T16:43:52.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.463 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c666095f:::benchmark_data_smithi188_141483_object61745:head# 2024-04-23T16:43:52.463 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.464 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6664852:::benchmark_data_smithi188_141483_object37358:head# 2024-04-23T16:43:52.464 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.464 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6666867:::benchmark_data_smithi188_141483_object31111:head# 2024-04-23T16:43:52.464 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.464 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6668d37:::benchmark_data_smithi188_141483_object71212:head# 2024-04-23T16:43:52.464 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.464 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c666e742:::benchmark_data_smithi188_141483_object11865:head# 2024-04-23T16:43:52.464 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.464 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6670be7:::benchmark_data_smithi188_141483_object32147:head# 2024-04-23T16:43:52.464 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.464 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6671f3b:::benchmark_data_smithi188_141483_object11352:head# 2024-04-23T16:43:52.464 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.464 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c667b046:::benchmark_data_smithi188_141483_object71203:head# 2024-04-23T16:43:52.464 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c667c61d:::benchmark_data_smithi188_141483_object50670:head# 2024-04-23T16:43:52.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c668d7ca:::benchmark_data_smithi188_141483_object8712:head# 2024-04-23T16:43:52.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6698d93:::benchmark_data_smithi188_141483_object41980:head# 2024-04-23T16:43:52.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c669cbd1:::benchmark_data_smithi188_141483_object39985:head# 2024-04-23T16:43:52.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c66a38bb:::benchmark_data_smithi188_141483_object3495:head# 2024-04-23T16:43:52.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c66b370c:::benchmark_data_smithi188_141483_object83324:head# 2024-04-23T16:43:52.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c66dc4e3:::benchmark_data_smithi188_141483_object28153:head# 2024-04-23T16:43:52.465 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.465 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c66e2936:::benchmark_data_smithi188_141483_object5880:head# 2024-04-23T16:43:52.466 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.466 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c66e4be3:::benchmark_data_smithi188_141483_object34355:head# 2024-04-23T16:43:52.466 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.466 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c66f0b91:::benchmark_data_smithi188_141483_object63092:head# 2024-04-23T16:43:52.466 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.466 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c66f6595:::benchmark_data_smithi188_141483_object85085:head# 2024-04-23T16:43:52.466 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.466 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c66ff211:::benchmark_data_smithi188_141483_object34661:head# 2024-04-23T16:43:52.466 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.466 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6708cf3:::benchmark_data_smithi188_141483_object77419:head# 2024-04-23T16:43:52.466 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.467 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c670c094:::benchmark_data_smithi188_141483_object19174:head# 2024-04-23T16:43:52.467 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.467 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c670f0d6:::benchmark_data_smithi188_141483_object82405:head# 2024-04-23T16:43:52.467 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.467 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c672d1f5:::benchmark_data_smithi188_141483_object60291:head# 2024-04-23T16:43:52.467 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.467 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c67403d1:::benchmark_data_smithi188_141483_object73187:head# 2024-04-23T16:43:52.467 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.467 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6740c9f:::benchmark_data_smithi188_141483_object27738:head# 2024-04-23T16:43:52.467 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.467 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c675584b:::benchmark_data_smithi188_141483_object77753:head# 2024-04-23T16:43:52.467 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.467 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c67618f4:::benchmark_data_smithi188_141483_object85869:head# 2024-04-23T16:43:52.467 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.468 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c676b5cf:::benchmark_data_smithi188_141483_object36922:head# 2024-04-23T16:43:52.468 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.468 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c676d1c1:::benchmark_data_smithi188_141483_object54292:head# 2024-04-23T16:43:52.468 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.468 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6788388:::benchmark_data_smithi188_141483_object13856:head# 2024-04-23T16:43:52.468 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.468 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c67917f9:::benchmark_data_smithi188_141483_object35752:head# 2024-04-23T16:43:52.468 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.468 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6791bfc:::benchmark_data_smithi188_141483_object70750:head# 2024-04-23T16:43:52.468 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.468 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c67a3dea:::benchmark_data_smithi188_141483_object37795:head# 2024-04-23T16:43:52.468 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.468 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c67b05c5:::benchmark_data_smithi188_141483_object49936:head# 2024-04-23T16:43:52.468 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.469 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c67b7256:::benchmark_data_smithi188_141483_object82053:head# 2024-04-23T16:43:52.469 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.469 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c67b91c4:::benchmark_data_smithi188_141483_object42639:head# 2024-04-23T16:43:52.469 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.469 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c67db449:::benchmark_data_smithi188_141483_object71030:head# 2024-04-23T16:43:52.469 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.469 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c67f6071:::benchmark_data_smithi188_141483_object84963:head# 2024-04-23T16:43:52.469 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.469 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c67fce44:::benchmark_data_smithi188_141483_object18570:head# 2024-04-23T16:43:52.469 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.469 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c68048ac:::benchmark_data_smithi188_141483_object67939:head# 2024-04-23T16:43:52.469 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.469 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6812ef1:::benchmark_data_smithi188_141483_object90300:head# 2024-04-23T16:43:52.469 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.470 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c68163f4:::benchmark_data_smithi188_141483_object59990:head# 2024-04-23T16:43:52.470 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.470 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c68167eb:::benchmark_data_smithi188_141483_object21643:head# 2024-04-23T16:43:52.470 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.470 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6823a30:::benchmark_data_smithi188_141483_object37130:head# 2024-04-23T16:43:52.470 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.470 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c68281b1:::benchmark_data_smithi188_141483_object75185:head# 2024-04-23T16:43:52.470 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.470 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6828601:::benchmark_data_smithi188_141483_object62384:head# 2024-04-23T16:43:52.470 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.470 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c682dbf8:::benchmark_data_smithi188_141483_object48246:head# 2024-04-23T16:43:52.470 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.470 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6840391:::benchmark_data_smithi188_141483_object74919:head# 2024-04-23T16:43:52.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c68410d5:::benchmark_data_smithi188_141483_object28897:head# 2024-04-23T16:43:52.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6843d52:::benchmark_data_smithi188_141483_object11438:head# 2024-04-23T16:43:52.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6848d2d:::benchmark_data_smithi188_141483_object54355:head# 2024-04-23T16:43:52.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6865ec2:::benchmark_data_smithi188_141483_object81079:head# 2024-04-23T16:43:52.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6866f8f:::benchmark_data_smithi188_141483_object20958:head# 2024-04-23T16:43:52.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c68700bb:::benchmark_data_smithi188_141483_object49235:head# 2024-04-23T16:43:52.471 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.471 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6879c0c:::benchmark_data_smithi188_141483_object74311:head# 2024-04-23T16:43:52.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6896734:::benchmark_data_smithi188_141483_object86064:head# 2024-04-23T16:43:52.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c689819c:::benchmark_data_smithi188_141483_object67637:head# 2024-04-23T16:43:52.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c68c4f93:::benchmark_data_smithi188_141483_object17214:head# 2024-04-23T16:43:52.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c68ccf24:::benchmark_data_smithi188_141483_object28317:head# 2024-04-23T16:43:52.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c68cfebe:::benchmark_data_smithi188_141483_object8596:head# 2024-04-23T16:43:52.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.472 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c68e0fc7:::benchmark_data_smithi188_141483_object73475:head# 2024-04-23T16:43:52.472 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c68e20de:::benchmark_data_smithi188_141483_object23553:head# 2024-04-23T16:43:52.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c68e8006:::benchmark_data_smithi188_141483_object15745:head# 2024-04-23T16:43:52.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c68eb587:::benchmark_data_smithi188_141483_object32843:head# 2024-04-23T16:43:52.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c68f2646:::benchmark_data_smithi188_141483_object61834:head# 2024-04-23T16:43:52.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c68f2ba2:::benchmark_data_smithi188_141483_object51375:head# 2024-04-23T16:43:52.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c68fff23:::benchmark_data_smithi188_141483_object13069:head# 2024-04-23T16:43:52.473 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.473 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6910e49:::benchmark_data_smithi188_141483_object10845:head# 2024-04-23T16:43:52.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6913dc2:::benchmark_data_smithi188_141483_object46762:head# 2024-04-23T16:43:52.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c692499c:::benchmark_data_smithi188_141483_object5149:head# 2024-04-23T16:43:52.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c692de60:::benchmark_data_smithi188_141483_object18824:head# 2024-04-23T16:43:52.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6945d6e:::benchmark_data_smithi188_141483_object60335:head# 2024-04-23T16:43:52.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c695f4f7:::benchmark_data_smithi188_141483_object10193:head# 2024-04-23T16:43:52.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.474 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6963eec:::benchmark_data_smithi188_141483_object41995:head# 2024-04-23T16:43:52.474 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c697b54d:::benchmark_data_smithi188_141483_object22474:head# 2024-04-23T16:43:52.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c697ea23:::benchmark_data_smithi188_141483_object31997:head# 2024-04-23T16:43:52.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c699b01e:::benchmark_data_smithi188_141483_object3774:head# 2024-04-23T16:43:52.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c69a037e:::benchmark_data_smithi188_141483_object31991:head# 2024-04-23T16:43:52.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c69a37e6:::benchmark_data_smithi188_141483_object44916:head# 2024-04-23T16:43:52.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c69b3d16:::benchmark_data_smithi188_141483_object14665:head# 2024-04-23T16:43:52.475 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.475 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c69b7d83:::benchmark_data_smithi188_141483_object11816:head# 2024-04-23T16:43:52.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c69bfdea:::benchmark_data_smithi188_141483_object30256:head# 2024-04-23T16:43:52.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c69cab32:::benchmark_data_smithi188_141483_object19058:head# 2024-04-23T16:43:52.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c69cd7b4:::benchmark_data_smithi188_141483_object77256:head# 2024-04-23T16:43:52.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c69d2a73:::benchmark_data_smithi188_141483_object56355:head# 2024-04-23T16:43:52.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c69d3228:::benchmark_data_smithi188_141483_object86128:head# 2024-04-23T16:43:52.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c69e31a8:::benchmark_data_smithi188_141483_object11620:head# 2024-04-23T16:43:52.476 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.476 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c69ed580:::benchmark_data_smithi188_141483_object45788:head# 2024-04-23T16:43:52.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c69f11d4:::benchmark_data_smithi188_141483_object88353:head# 2024-04-23T16:43:52.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c69fa509:::benchmark_data_smithi188_141483_object88913:head# 2024-04-23T16:43:52.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6a129fa:::benchmark_data_smithi188_141483_object10012:head# 2024-04-23T16:43:52.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6a1be37:::benchmark_data_smithi188_141483_object23166:head# 2024-04-23T16:43:52.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6a261e4:::benchmark_data_smithi188_141483_object63844:head# 2024-04-23T16:43:52.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6a311d8:::benchmark_data_smithi188_141483_object7683:head# 2024-04-23T16:43:52.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.477 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6a3f433:::benchmark_data_smithi188_141483_object7533:head# 2024-04-23T16:43:52.477 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.478 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6a558eb:::benchmark_data_smithi188_141483_object77230:head# 2024-04-23T16:43:52.478 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.478 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6a78bd3:::benchmark_data_smithi188_141483_object61017:head# 2024-04-23T16:43:52.478 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.478 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6a84b07:::benchmark_data_smithi188_141483_object52612:head# 2024-04-23T16:43:52.478 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.478 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6aaaba9:::benchmark_data_smithi188_141483_object7514:head# 2024-04-23T16:43:52.478 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.478 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6ab3820:::benchmark_data_smithi188_141483_object47012:head# 2024-04-23T16:43:52.478 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.478 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6ac027c:::benchmark_data_smithi188_141483_object15970:head# 2024-04-23T16:43:52.478 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.478 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6ac8fce:::benchmark_data_smithi188_141483_object54223:head# 2024-04-23T16:43:52.478 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6aca995:::benchmark_data_smithi188_141483_object2326:head# 2024-04-23T16:43:52.479 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6ad9c8c:::benchmark_data_smithi188_141483_object39156:head# 2024-04-23T16:43:52.479 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6ae36e7:::benchmark_data_smithi188_141483_object6001:head# 2024-04-23T16:43:52.479 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6af727d:::benchmark_data_smithi188_141483_object2870:head# 2024-04-23T16:43:52.479 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6afca1a:::benchmark_data_smithi188_141483_object2791:head# 2024-04-23T16:43:52.479 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.479 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6afeaed:::benchmark_data_smithi188_141483_object19341:head# 2024-04-23T16:43:52.480 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.480 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6b075fd:::benchmark_data_smithi188_141483_object45097:head# 2024-04-23T16:43:52.480 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.480 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6b11182:::benchmark_data_smithi188_141483_object36793:head# 2024-04-23T16:43:52.480 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.480 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6b153c2:::benchmark_data_smithi188_141483_object77258:head# 2024-04-23T16:43:52.480 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.480 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6b2e1f6:::benchmark_data_smithi188_141483_object56920:head# 2024-04-23T16:43:52.480 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.480 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6b3a4f0:::benchmark_data_smithi188_141483_object79309:head# 2024-04-23T16:43:52.480 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.481 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6b3b2f0:::benchmark_data_smithi188_141483_object82474:head# 2024-04-23T16:43:52.481 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.481 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6b42abf:::benchmark_data_smithi188_141483_object19092:head# 2024-04-23T16:43:52.481 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.481 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6b4f4e5:::benchmark_data_smithi188_141483_object75464:head# 2024-04-23T16:43:52.481 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.481 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6b6a820:::benchmark_data_smithi188_141483_object75469:head# 2024-04-23T16:43:52.481 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.481 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6b911a2:::benchmark_data_smithi188_141483_object49541:head# 2024-04-23T16:43:52.481 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.481 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6b94027:::benchmark_data_smithi188_141483_object43546:head# 2024-04-23T16:43:52.481 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.481 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6b99aba:::benchmark_data_smithi188_141483_object46431:head# 2024-04-23T16:43:52.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6ba831f:::benchmark_data_smithi188_141483_object3431:head# 2024-04-23T16:43:52.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6bad97a:::benchmark_data_smithi188_141483_object1962:head# 2024-04-23T16:43:52.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6bbf3dd:::benchmark_data_smithi188_141483_object25033:head# 2024-04-23T16:43:52.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6bd3c46:::benchmark_data_smithi188_141483_object28021:head# 2024-04-23T16:43:52.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6bf4fb3:::benchmark_data_smithi188_141483_object41284:head# 2024-04-23T16:43:52.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.482 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6c0a6b3:::benchmark_data_smithi188_141483_object3749:head# 2024-04-23T16:43:52.482 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6c17f36:::benchmark_data_smithi188_141483_object58938:head# 2024-04-23T16:43:52.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6c18de8:::benchmark_data_smithi188_141483_object6669:head# 2024-04-23T16:43:52.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6c4755e:::benchmark_data_smithi188_141483_object63100:head# 2024-04-23T16:43:52.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6c48f0f:::benchmark_data_smithi188_141483_object42543:head# 2024-04-23T16:43:52.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6c4af17:::benchmark_data_smithi188_141483_object69452:head# 2024-04-23T16:43:52.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6c4fe40:::benchmark_data_smithi188_141483_object28360:head# 2024-04-23T16:43:52.483 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.483 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6c578ac:::benchmark_data_smithi188_141483_object63107:head# 2024-04-23T16:43:52.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6c61f37:::benchmark_data_smithi188_141483_object72999:head# 2024-04-23T16:43:52.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6c6cd5e:::benchmark_data_smithi188_141483_object85440:head# 2024-04-23T16:43:52.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6c700f0:::benchmark_data_smithi188_141483_object13913:head# 2024-04-23T16:43:52.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6c7d7db:::benchmark_data_smithi188_141483_object50822:head# 2024-04-23T16:43:52.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6c91fd9:::benchmark_data_smithi188_141483_object12629:head# 2024-04-23T16:43:52.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6c92fe7:::benchmark_data_smithi188_141483_object6534:head# 2024-04-23T16:43:52.484 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.484 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6cc6cf6:::benchmark_data_smithi188_141483_object35137:head# 2024-04-23T16:43:52.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6ccde12:::benchmark_data_smithi188_141483_object76097:head# 2024-04-23T16:43:52.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6cd5a6b:::benchmark_data_smithi188_141483_object10186:head# 2024-04-23T16:43:52.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6cd656b:::benchmark_data_smithi188_141483_object68364:head# 2024-04-23T16:43:52.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6cdfdc8:::benchmark_data_smithi188_141483_object76352:head# 2024-04-23T16:43:52.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6d3871c:::benchmark_data_smithi188_141483_object75429:head# 2024-04-23T16:43:52.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.485 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6d45a8c:::benchmark_data_smithi188_141483_object26938:head# 2024-04-23T16:43:52.485 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.486 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6d4f47d:::benchmark_data_smithi188_141483_object84419:head# 2024-04-23T16:43:52.486 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.486 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6d4ff4f:::benchmark_data_smithi188_141483_object82952:head# 2024-04-23T16:43:52.486 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.486 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6d5939c:::benchmark_data_smithi188_141483_object2896:head# 2024-04-23T16:43:52.486 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.486 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6d5950b:::benchmark_data_smithi188_141483_object10922:head# 2024-04-23T16:43:52.486 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.486 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6d5ed3d:::benchmark_data_smithi188_141483_object68383:head# 2024-04-23T16:43:52.486 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.486 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6d6929f:::benchmark_data_smithi188_141483_object39225:head# 2024-04-23T16:43:52.486 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6d84d77:::benchmark_data_smithi188_141483_object55271:head# 2024-04-23T16:43:52.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6d96d2e:::benchmark_data_smithi188_141483_object55745:head# 2024-04-23T16:43:52.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6db4d5f:::benchmark_data_smithi188_141483_object56063:head# 2024-04-23T16:43:52.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6dc0069:::benchmark_data_smithi188_141483_object37763:head# 2024-04-23T16:43:52.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6dcd8bb:::benchmark_data_smithi188_141483_object88624:head# 2024-04-23T16:43:52.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6dd23f7:::benchmark_data_smithi188_141483_object71811:head# 2024-04-23T16:43:52.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.487 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6dd6340:::benchmark_data_smithi188_141483_object74969:head# 2024-04-23T16:43:52.487 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.488 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6defdc5:::benchmark_data_smithi188_141483_object5092:head# 2024-04-23T16:43:52.488 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.488 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6df47f3:::benchmark_data_smithi188_141483_object32232:head# 2024-04-23T16:43:52.488 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.488 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6df78c7:::benchmark_data_smithi188_141483_object80632:head# 2024-04-23T16:43:52.488 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.488 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6e12a53:::benchmark_data_smithi188_141483_object88012:head# 2024-04-23T16:43:52.488 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.488 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6e1b072:::benchmark_data_smithi188_141483_object41928:head# 2024-04-23T16:43:52.488 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.488 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6e1d956:::benchmark_data_smithi188_141483_object67604:head# 2024-04-23T16:43:52.488 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.488 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6e2e262:::benchmark_data_smithi188_141483_object79156:head# 2024-04-23T16:43:52.489 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.489 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6e30ce0:::benchmark_data_smithi188_141483_object9665:head# 2024-04-23T16:43:52.489 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.489 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6e35ac1:::benchmark_data_smithi188_141483_object45099:head# 2024-04-23T16:43:52.489 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.489 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6e3eb9d:::benchmark_data_smithi188_141483_object78542:head# 2024-04-23T16:43:52.489 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.489 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6e4632a:::benchmark_data_smithi188_141483_object38428:head# 2024-04-23T16:43:52.489 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.489 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6e473c6:::benchmark_data_smithi188_141483_object44462:head# 2024-04-23T16:43:52.489 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.489 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6e499d5:::benchmark_data_smithi188_141483_object55931:head# 2024-04-23T16:43:52.489 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.489 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6e4b184:::benchmark_data_smithi188_141483_object22427:head# 2024-04-23T16:43:52.490 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.490 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6e57795:::benchmark_data_smithi188_141483_object3588:head# 2024-04-23T16:43:52.490 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.490 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6e5c44f:::benchmark_data_smithi188_141483_object22269:head# 2024-04-23T16:43:52.490 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.490 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6e5c689:::benchmark_data_smithi188_141483_object15637:head# 2024-04-23T16:43:52.490 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.490 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6e6f87d:::benchmark_data_smithi188_141483_object46317:head# 2024-04-23T16:43:52.490 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.490 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6e79ffb:::benchmark_data_smithi188_141483_object78226:head# 2024-04-23T16:43:52.490 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.490 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6e8ba5e:::benchmark_data_smithi188_141483_object22641:head# 2024-04-23T16:43:52.490 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.491 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6e8cdf2:::benchmark_data_smithi188_141483_object22945:head# 2024-04-23T16:43:52.491 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.491 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6e8e3a6:::benchmark_data_smithi188_141483_object4235:head# 2024-04-23T16:43:52.491 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.491 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6eb2fdb:::benchmark_data_smithi188_141483_object63603:head# 2024-04-23T16:43:52.491 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.491 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6ecc6c2:::benchmark_data_smithi188_141483_object86969:head# 2024-04-23T16:43:52.491 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.491 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6ed614e:::benchmark_data_smithi188_141483_object23496:head# 2024-04-23T16:43:52.491 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.491 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6ee0c65:::benchmark_data_smithi188_141483_object10007:head# 2024-04-23T16:43:52.491 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.491 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6ee22a8:::benchmark_data_smithi188_141483_object88427:head# 2024-04-23T16:43:52.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6eebf83:::benchmark_data_smithi188_141483_object30630:head# 2024-04-23T16:43:52.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6ef7f0e:::benchmark_data_smithi188_141483_object69741:head# 2024-04-23T16:43:52.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6f1a7bd:::benchmark_data_smithi188_141483_object32076:head# 2024-04-23T16:43:52.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6f1ba08:::benchmark_data_smithi188_141483_object50193:head# 2024-04-23T16:43:52.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6f1e5d9:::benchmark_data_smithi188_141483_object77875:head# 2024-04-23T16:43:52.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6f24ea6:::benchmark_data_smithi188_141483_object19131:head# 2024-04-23T16:43:52.492 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.492 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6f29646:::benchmark_data_smithi188_141483_object49216:head# 2024-04-23T16:43:52.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6f2b710:::benchmark_data_smithi188_141483_object31569:head# 2024-04-23T16:43:52.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6f38762:::benchmark_data_smithi188_141483_object84928:head# 2024-04-23T16:43:52.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6f5e909:::benchmark_data_smithi188_141483_object21233:head# 2024-04-23T16:43:52.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6f83434:::benchmark_data_smithi188_141483_object66122:head# 2024-04-23T16:43:52.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6f94a89:::benchmark_data_smithi188_141483_object62644:head# 2024-04-23T16:43:52.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6faa3c6:::benchmark_data_smithi188_141483_object45148:head# 2024-04-23T16:43:52.493 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.493 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6fad590:::benchmark_data_smithi188_141483_object50589:head# 2024-04-23T16:43:52.494 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.494 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6fc9c2a:::benchmark_data_smithi188_141483_object64443:head# 2024-04-23T16:43:52.494 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.494 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6fe4b0e:::benchmark_data_smithi188_141483_object27490:head# 2024-04-23T16:43:52.494 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.494 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6feb09c:::benchmark_data_smithi188_141483_object62748:head# 2024-04-23T16:43:52.494 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.494 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c6ffae52:::benchmark_data_smithi188_141483_object87637:head# 2024-04-23T16:43:52.494 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.494 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7000c8d:::benchmark_data_smithi188_141483_object41588:head# 2024-04-23T16:43:52.494 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.494 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7001b79:::benchmark_data_smithi188_141483_object69546:head# 2024-04-23T16:43:52.494 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.495 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7011fa5:::benchmark_data_smithi188_141483_object53491:head# 2024-04-23T16:43:52.495 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.495 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c70136e6:::benchmark_data_smithi188_141483_object70283:head# 2024-04-23T16:43:52.495 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.495 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c70175a4:::benchmark_data_smithi188_141483_object1886:head# 2024-04-23T16:43:52.495 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.495 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7017d92:::benchmark_data_smithi188_141483_object24275:head# 2024-04-23T16:43:52.495 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.495 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c701e69e:::benchmark_data_smithi188_141483_object82677:head# 2024-04-23T16:43:52.495 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.495 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c702695d:::benchmark_data_smithi188_141483_object19071:head# 2024-04-23T16:43:52.495 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.495 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c703553f:::benchmark_data_smithi188_141483_object8991:head# 2024-04-23T16:43:52.495 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.496 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7040ad4:::benchmark_data_smithi188_141483_object14845:head# 2024-04-23T16:43:52.496 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.496 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c704bcfe:::benchmark_data_smithi188_141483_object4287:head# 2024-04-23T16:43:52.496 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.496 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7067a88:::benchmark_data_smithi188_141483_object26835:head# 2024-04-23T16:43:52.496 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.496 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c706b0db:::benchmark_data_smithi188_141483_object20970:head# 2024-04-23T16:43:52.496 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.496 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c70709f3:::benchmark_data_smithi188_141483_object63724:head# 2024-04-23T16:43:52.496 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.496 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c707e918:::benchmark_data_smithi188_141483_object41855:head# 2024-04-23T16:43:52.496 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.496 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7093d9d:::benchmark_data_smithi188_141483_object40619:head# 2024-04-23T16:43:52.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c709e961:::benchmark_data_smithi188_141483_object27982:head# 2024-04-23T16:43:52.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c70a340c:::benchmark_data_smithi188_141483_object6006:head# 2024-04-23T16:43:52.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c70a369b:::benchmark_data_smithi188_141483_object26798:head# 2024-04-23T16:43:52.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c70ba8bf:::benchmark_data_smithi188_141483_object48844:head# 2024-04-23T16:43:52.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c70c2752:::benchmark_data_smithi188_141483_object84008:head# 2024-04-23T16:43:52.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.497 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c70c4658:::benchmark_data_smithi188_141483_object50165:head# 2024-04-23T16:43:52.497 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c70ce765:::benchmark_data_smithi188_141483_object32965:head# 2024-04-23T16:43:52.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c70d185f:::benchmark_data_smithi188_141483_object75066:head# 2024-04-23T16:43:52.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c70eabe1:::benchmark_data_smithi188_141483_object71312:head# 2024-04-23T16:43:52.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c70f71f0:::benchmark_data_smithi188_141483_object39925:head# 2024-04-23T16:43:52.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c70fc798:::benchmark_data_smithi188_141483_object43154:head# 2024-04-23T16:43:52.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c710ed58:::benchmark_data_smithi188_141483_object70662:head# 2024-04-23T16:43:52.498 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.498 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c712fc53:::benchmark_data_smithi188_141483_object83815:head# 2024-04-23T16:43:52.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7147172:::benchmark_data_smithi188_141483_object55848:head# 2024-04-23T16:43:52.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c714cff0:::benchmark_data_smithi188_141483_object25831:head# 2024-04-23T16:43:52.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c71538ae:::benchmark_data_smithi188_141483_object5760:head# 2024-04-23T16:43:52.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c715695d:::benchmark_data_smithi188_141483_object80141:head# 2024-04-23T16:43:52.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c715aac9:::benchmark_data_smithi188_141483_object35157:head# 2024-04-23T16:43:52.499 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.499 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c715ee4a:::benchmark_data_smithi188_141483_object39829:head# 2024-04-23T16:43:52.500 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.500 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c71647e2:::benchmark_data_smithi188_141483_object87199:head# 2024-04-23T16:43:52.500 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.500 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c718180e:::benchmark_data_smithi188_141483_object18542:head# 2024-04-23T16:43:52.500 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.500 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7184b00:::benchmark_data_smithi188_141483_object67838:head# 2024-04-23T16:43:52.500 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.500 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c71989be:::benchmark_data_smithi188_141483_object36538:head# 2024-04-23T16:43:52.500 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.500 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c71aa744:::benchmark_data_smithi188_141483_object45728:head# 2024-04-23T16:43:52.500 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.500 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c71acaa6:::benchmark_data_smithi188_141483_object88984:head# 2024-04-23T16:43:52.500 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.501 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c71b4b0b:::benchmark_data_smithi188_141483_object80928:head# 2024-04-23T16:43:52.501 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.501 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c71bcb6c:::benchmark_data_smithi188_141483_object48105:head# 2024-04-23T16:43:52.501 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.501 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c71c6ff8:::benchmark_data_smithi188_141483_object22715:head# 2024-04-23T16:43:52.501 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.501 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c71ceea0:::benchmark_data_smithi188_141483_object43148:head# 2024-04-23T16:43:52.501 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.501 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c71d557f:::benchmark_data_smithi188_141483_object81479:head# 2024-04-23T16:43:52.501 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.501 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c71d6f32:::benchmark_data_smithi188_141483_object16221:head# 2024-04-23T16:43:52.501 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.501 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c71e35b4:::benchmark_data_smithi188_141483_object6950:head# 2024-04-23T16:43:52.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.502 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c71e472c:::benchmark_data_smithi188_141483_object71732:head# 2024-04-23T16:43:52.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.502 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c71e6995:::benchmark_data_smithi188_141483_object6321:head# 2024-04-23T16:43:52.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.502 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c71f0d4c:::benchmark_data_smithi188_141483_object34258:head# 2024-04-23T16:43:52.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.502 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c71feac8:::benchmark_data_smithi188_141483_object17203:head# 2024-04-23T16:43:52.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.502 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7209155:::benchmark_data_smithi188_141483_object2113:head# 2024-04-23T16:43:52.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.502 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7209982:::benchmark_data_smithi188_141483_object1923:head# 2024-04-23T16:43:52.502 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c720b182:::benchmark_data_smithi188_141483_object66680:head# 2024-04-23T16:43:52.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c720ebb4:::benchmark_data_smithi188_141483_object67692:head# 2024-04-23T16:43:52.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c72182b9:::benchmark_data_smithi188_141483_object63952:head# 2024-04-23T16:43:52.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7223e38:::benchmark_data_smithi188_141483_object47584:head# 2024-04-23T16:43:52.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7240b6d:::benchmark_data_smithi188_141483_object86474:head# 2024-04-23T16:43:52.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7256797:::benchmark_data_smithi188_141483_object88926:head# 2024-04-23T16:43:52.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.503 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7258357:::benchmark_data_smithi188_141483_object40769:head# 2024-04-23T16:43:52.503 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7259f3b:::benchmark_data_smithi188_141483_object86147:head# 2024-04-23T16:43:52.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c725ebfc:::benchmark_data_smithi188_141483_object2838:head# 2024-04-23T16:43:52.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c726197f:::benchmark_data_smithi188_141483_object65371:head# 2024-04-23T16:43:52.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7273ff3:::benchmark_data_smithi188_141483_object22574:head# 2024-04-23T16:43:52.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c727e9e4:::benchmark_data_smithi188_141483_object66820:head# 2024-04-23T16:43:52.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7288544:::benchmark_data_smithi188_141483_object14949:head# 2024-04-23T16:43:52.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.504 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c72cf9cc:::benchmark_data_smithi188_141483_object73510:head# 2024-04-23T16:43:52.504 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.505 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c72dc52d:::benchmark_data_smithi188_141483_object40870:head# 2024-04-23T16:43:52.505 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.505 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c72ea501:::benchmark_data_smithi188_141483_object89782:head# 2024-04-23T16:43:52.505 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.505 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c72faec6:::benchmark_data_smithi188_141483_object47533:head# 2024-04-23T16:43:52.505 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.505 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c72ff30d:::benchmark_data_smithi188_141483_object3236:head# 2024-04-23T16:43:52.505 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.505 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c730a27f:::benchmark_data_smithi188_141483_object69738:head# 2024-04-23T16:43:52.505 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.505 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c73227c4:::benchmark_data_smithi188_141483_object88368:head# 2024-04-23T16:43:52.505 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.505 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7325d47:::benchmark_data_smithi188_141483_object47641:head# 2024-04-23T16:43:52.505 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.506 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7346b15:::benchmark_data_smithi188_141483_object46184:head# 2024-04-23T16:43:52.506 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.506 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c734d5b5:::benchmark_data_smithi188_141483_object20677:head# 2024-04-23T16:43:52.506 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.506 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7363004:::benchmark_data_smithi188_141483_object66359:head# 2024-04-23T16:43:52.506 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.506 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7365699:::benchmark_data_smithi188_141483_object53467:head# 2024-04-23T16:43:52.506 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.506 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c738668e:::benchmark_data_smithi188_141483_object87860:head# 2024-04-23T16:43:52.506 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.506 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c738d9df:::benchmark_data_smithi188_141483_object44511:head# 2024-04-23T16:43:52.506 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.506 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7396b7c:::benchmark_data_smithi188_141483_object27503:head# 2024-04-23T16:43:52.506 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c73ae625:::benchmark_data_smithi188_141483_object78747:head# 2024-04-23T16:43:52.507 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c73af15f:::benchmark_data_smithi188_141483_object31424:head# 2024-04-23T16:43:52.507 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c73c26c2:::benchmark_data_smithi188_141483_object44373:head# 2024-04-23T16:43:52.507 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c73ce2a1:::benchmark_data_smithi188_141483_object13958:head# 2024-04-23T16:43:52.507 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c73e5a98:::benchmark_data_smithi188_141483_object55893:head# 2024-04-23T16:43:52.507 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c73f3bd5:::benchmark_data_smithi188_141483_object39237:head# 2024-04-23T16:43:52.507 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.507 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c73fabf8:::benchmark_data_smithi188_141483_object31525:head# 2024-04-23T16:43:52.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7407793:::benchmark_data_smithi188_141483_object85195:head# 2024-04-23T16:43:52.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7418539:::benchmark_data_smithi188_141483_object26455:head# 2024-04-23T16:43:52.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c74295a1:::benchmark_data_smithi188_141483_object32324:head# 2024-04-23T16:43:52.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c743160b:::benchmark_data_smithi188_141483_object74578:head# 2024-04-23T16:43:52.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7431d47:::benchmark_data_smithi188_141483_object17186:head# 2024-04-23T16:43:52.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c743b64c:::benchmark_data_smithi188_141483_object21254:head# 2024-04-23T16:43:52.508 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.508 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7443188:::benchmark_data_smithi188_141483_object42673:head# 2024-04-23T16:43:52.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7446ffd:::benchmark_data_smithi188_141483_object74961:head# 2024-04-23T16:43:52.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7448baa:::benchmark_data_smithi188_141483_object60840:head# 2024-04-23T16:43:52.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7474c94:::benchmark_data_smithi188_141483_object83155:head# 2024-04-23T16:43:52.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7475a13:::benchmark_data_smithi188_141483_object5080:head# 2024-04-23T16:43:52.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c74858ee:::benchmark_data_smithi188_141483_object47324:head# 2024-04-23T16:43:52.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.509 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c748a296:::benchmark_data_smithi188_141483_object25001:head# 2024-04-23T16:43:52.509 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.510 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c74a3e45:::benchmark_data_smithi188_141483_object20538:head# 2024-04-23T16:43:52.510 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.510 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c74ab857:::benchmark_data_smithi188_141483_object39954:head# 2024-04-23T16:43:52.510 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.510 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c74af0e5:::benchmark_data_smithi188_141483_object40627:head# 2024-04-23T16:43:52.510 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.510 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c74b1539:::benchmark_data_smithi188_141483_object33019:head# 2024-04-23T16:43:52.510 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.510 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c74b1f9a:::benchmark_data_smithi188_141483_object71767:head# 2024-04-23T16:43:52.510 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.510 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c74b3a39:::benchmark_data_smithi188_141483_object87248:head# 2024-04-23T16:43:52.511 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.511 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c74c49fc:::benchmark_data_smithi188_141483_object45269:head# 2024-04-23T16:43:52.511 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.511 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c74c894e:::benchmark_data_smithi188_141483_object86299:head# 2024-04-23T16:43:52.511 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.511 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c74dff3c:::benchmark_data_smithi188_141483_object74115:head# 2024-04-23T16:43:52.511 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.511 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c74fabf4:::benchmark_data_smithi188_141483_object1868:head# 2024-04-23T16:43:52.511 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.511 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7504372:::benchmark_data_smithi188_141483_object4488:head# 2024-04-23T16:43:52.511 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.511 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c75075fc:::benchmark_data_smithi188_141483_object38750:head# 2024-04-23T16:43:52.511 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.511 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c750bcab:::benchmark_data_smithi188_141483_object59772:head# 2024-04-23T16:43:52.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7518c53:::benchmark_data_smithi188_141483_object37776:head# 2024-04-23T16:43:52.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c751de20:::benchmark_data_smithi188_141483_object72090:head# 2024-04-23T16:43:52.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7522314:::benchmark_data_smithi188_141483_object10387:head# 2024-04-23T16:43:52.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7532edd:::benchmark_data_smithi188_141483_object61349:head# 2024-04-23T16:43:52.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7534a48:::benchmark_data_smithi188_141483_object82585:head# 2024-04-23T16:43:52.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.512 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c754d403:::benchmark_data_smithi188_141483_object68802:head# 2024-04-23T16:43:52.512 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c75500f2:::benchmark_data_smithi188_141483_object35895:head# 2024-04-23T16:43:52.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c756a1d1:::benchmark_data_smithi188_141483_object10516:head# 2024-04-23T16:43:52.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7572d0e:::benchmark_data_smithi188_141483_object40862:head# 2024-04-23T16:43:52.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c757f2e2:::benchmark_data_smithi188_141483_object82272:head# 2024-04-23T16:43:52.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c75829da:::benchmark_data_smithi188_141483_object49892:head# 2024-04-23T16:43:52.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.513 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c759dfe2:::benchmark_data_smithi188_141483_object33872:head# 2024-04-23T16:43:52.513 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c75aeba5:::benchmark_data_smithi188_141483_object43064:head# 2024-04-23T16:43:52.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c75b9f77:::benchmark_data_smithi188_141483_object39234:head# 2024-04-23T16:43:52.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c75c7d2d:::benchmark_data_smithi188_141483_object56788:head# 2024-04-23T16:43:52.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c75dc036:::benchmark_data_smithi188_141483_object22321:head# 2024-04-23T16:43:52.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c75de009:::benchmark_data_smithi188_141483_object69932:head# 2024-04-23T16:43:52.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c75e12d8:::benchmark_data_smithi188_141483_object60635:head# 2024-04-23T16:43:52.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.514 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c76011aa:::benchmark_data_smithi188_141483_object56809:head# 2024-04-23T16:43:52.514 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.515 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c761da2f:::benchmark_data_smithi188_141483_object65689:head# 2024-04-23T16:43:52.515 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.515 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c763167d:::benchmark_data_smithi188_141483_object61943:head# 2024-04-23T16:43:52.515 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.515 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7632ded:::benchmark_data_smithi188_141483_object4702:head# 2024-04-23T16:43:52.515 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.515 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7634caa:::benchmark_data_smithi188_141483_object10085:head# 2024-04-23T16:43:52.515 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.515 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7643050:::benchmark_data_smithi188_141483_object2215:head# 2024-04-23T16:43:52.515 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.515 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7648662:::benchmark_data_smithi188_141483_object32982:head# 2024-04-23T16:43:52.515 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.515 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7649ac8:::benchmark_data_smithi188_141483_object60478:head# 2024-04-23T16:43:52.515 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c764a306:::benchmark_data_smithi188_141483_object2291:head# 2024-04-23T16:43:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c764b461:::benchmark_data_smithi188_141483_object13878:head# 2024-04-23T16:43:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c76506cc:::benchmark_data_smithi188_141483_object49428:head# 2024-04-23T16:43:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7661ab8:::benchmark_data_smithi188_141483_object69396:head# 2024-04-23T16:43:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7662155:::benchmark_data_smithi188_141483_object43009:head# 2024-04-23T16:43:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c766bd2b:::benchmark_data_smithi188_141483_object62745:head# 2024-04-23T16:43:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c76733e8:::benchmark_data_smithi188_141483_object85565:head# 2024-04-23T16:43:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c76769a6:::benchmark_data_smithi188_141483_object14146:head# 2024-04-23T16:43:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7679495:::benchmark_data_smithi188_141483_object55178:head# 2024-04-23T16:43:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c768155a:::benchmark_data_smithi188_141483_object9019:head# 2024-04-23T16:43:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7681b6e:::benchmark_data_smithi188_141483_object72210:head# 2024-04-23T16:43:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c768b50e:::benchmark_data_smithi188_141483_object33982:head# 2024-04-23T16:43:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c769a12f:::benchmark_data_smithi188_141483_object36938:head# 2024-04-23T16:43:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c769fdb2:::benchmark_data_smithi188_141483_object34163:head# 2024-04-23T16:43:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c76a8403:::benchmark_data_smithi188_141483_object5299:head# 2024-04-23T16:43:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c76af0d8:::benchmark_data_smithi188_141483_object2014:head# 2024-04-23T16:43:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c76c70c5:::benchmark_data_smithi188_141483_object84906:head# 2024-04-23T16:43:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c76cbb98:::benchmark_data_smithi188_141483_object30804:head# 2024-04-23T16:43:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c76d166b:::benchmark_data_smithi188_141483_object63627:head# 2024-04-23T16:43:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c76efd19:::benchmark_data_smithi188_141483_object46593:head# 2024-04-23T16:43:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c76f2ce0:::benchmark_data_smithi188_141483_object58207:head# 2024-04-23T16:43:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c76feabc:::benchmark_data_smithi188_141483_object3209:head# 2024-04-23T16:43:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c77026bf:::benchmark_data_smithi188_141483_object77386:head# 2024-04-23T16:43:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7704dd8:::benchmark_data_smithi188_141483_object77188:head# 2024-04-23T16:43:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7708272:::benchmark_data_smithi188_141483_object41641:head# 2024-04-23T16:43:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c771e562:::benchmark_data_smithi188_141483_object29261:head# 2024-04-23T16:43:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c774c243:::benchmark_data_smithi188_141483_object58256:head# 2024-04-23T16:43:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c774e3db:::benchmark_data_smithi188_141483_object29628:head# 2024-04-23T16:43:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c77629c0:::benchmark_data_smithi188_141483_object77776:head# 2024-04-23T16:43:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7763b82:::benchmark_data_smithi188_141483_object61973:head# 2024-04-23T16:43:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7769835:::benchmark_data_smithi188_141483_object39958:head# 2024-04-23T16:43:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c77702bd:::benchmark_data_smithi188_141483_object89575:head# 2024-04-23T16:43:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c77888e5:::benchmark_data_smithi188_141483_object6675:head# 2024-04-23T16:43:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7789240:::benchmark_data_smithi188_141483_object4983:head# 2024-04-23T16:43:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c778ac46:::benchmark_data_smithi188_141483_object5848:head# 2024-04-23T16:43:52.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c778faa2:::benchmark_data_smithi188_141483_object10884:head# 2024-04-23T16:43:52.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c779a828:::benchmark_data_smithi188_141483_object57222:head# 2024-04-23T16:43:52.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c779ea05:::benchmark_data_smithi188_141483_object84384:head# 2024-04-23T16:43:52.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c77a02d9:::benchmark_data_smithi188_141483_object28225:head# 2024-04-23T16:43:52.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c77b3c25:::benchmark_data_smithi188_141483_object47714:head# 2024-04-23T16:43:52.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c77b47ec:::benchmark_data_smithi188_141483_object12925:head# 2024-04-23T16:43:52.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.521 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c77b9eff:::benchmark_data_smithi188_141483_object43091:head# 2024-04-23T16:43:52.521 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.522 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c77c4954:::benchmark_data_smithi188_141483_object22370:head# 2024-04-23T16:43:52.522 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.522 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c77c8cf9:::benchmark_data_smithi188_141483_object84047:head# 2024-04-23T16:43:52.522 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.522 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c77d5e27:::benchmark_data_smithi188_141483_object15908:head# 2024-04-23T16:43:52.522 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:52.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:52.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:52.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:52.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:52.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:52 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:43:52.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:52 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:43:52.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:52 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:43:52.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:52 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:43:52.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c77d8445:::benchmark_data_smithi188_141483_object21981:head# 2024-04-23T16:43:52.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c77ddb1b:::benchmark_data_smithi188_141483_object81156:head# 2024-04-23T16:43:52.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c77ed632:::benchmark_data_smithi188_141483_object28052:head# 2024-04-23T16:43:52.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c77fbe29:::benchmark_data_smithi188_141483_object62284:head# 2024-04-23T16:43:52.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.572 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c780fab5:::benchmark_data_smithi188_141483_object45037:head# 2024-04-23T16:43:52.572 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7814854:::benchmark_data_smithi188_141483_object81979:head# 2024-04-23T16:43:52.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c781f629:::benchmark_data_smithi188_141483_object75136:head# 2024-04-23T16:43:52.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c782cdaf:::benchmark_data_smithi188_141483_object23240:head# 2024-04-23T16:43:52.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7850836:::benchmark_data_smithi188_141483_object89031:head# 2024-04-23T16:43:52.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7857286:::benchmark_data_smithi188_141483_object10667:head# 2024-04-23T16:43:52.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c785ab30:::benchmark_data_smithi188_141483_object74418:head# 2024-04-23T16:43:52.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.573 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7862589:::benchmark_data_smithi188_141483_object14762:head# 2024-04-23T16:43:52.573 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c78635ae:::benchmark_data_smithi188_141483_object7586:head# 2024-04-23T16:43:52.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c786d48d:::benchmark_data_smithi188_141483_object4238:head# 2024-04-23T16:43:52.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7879c1b:::benchmark_data_smithi188_141483_object45798:head# 2024-04-23T16:43:52.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c787b173:::benchmark_data_smithi188_141483_object12690:head# 2024-04-23T16:43:52.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c789ba1e:::benchmark_data_smithi188_141483_object12084:head# 2024-04-23T16:43:52.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c78a70ec:::benchmark_data_smithi188_141483_object59499:head# 2024-04-23T16:43:52.574 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.574 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c78a7fcc:::benchmark_data_smithi188_141483_object44717:head# 2024-04-23T16:43:52.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c78b1e2a:::benchmark_data_smithi188_141483_object27206:head# 2024-04-23T16:43:52.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c78b4de2:::benchmark_data_smithi188_141483_object33545:head# 2024-04-23T16:43:52.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c78c0d57:::benchmark_data_smithi188_141483_object16292:head# 2024-04-23T16:43:52.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c78d9d41:::benchmark_data_smithi188_141483_object56675:head# 2024-04-23T16:43:52.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c78fdaf8:::benchmark_data_smithi188_141483_object37869:head# 2024-04-23T16:43:52.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7935540:::benchmark_data_smithi188_141483_object20152:head# 2024-04-23T16:43:52.575 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.575 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7935ebd:::benchmark_data_smithi188_141483_object59593:head# 2024-04-23T16:43:52.576 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.576 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c79431d8:::benchmark_data_smithi188_141483_object63008:head# 2024-04-23T16:43:52.576 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.576 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7944e1f:::benchmark_data_smithi188_141483_object54103:head# 2024-04-23T16:43:52.576 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.576 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c794566a:::benchmark_data_smithi188_141483_object26762:head# 2024-04-23T16:43:52.576 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.576 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c794a24c:::benchmark_data_smithi188_141483_object27225:head# 2024-04-23T16:43:52.576 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.576 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7959840:::benchmark_data_smithi188_141483_object59767:head# 2024-04-23T16:43:52.576 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.576 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c796b753:::benchmark_data_smithi188_141483_object90352:head# 2024-04-23T16:43:52.576 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.576 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c798ed1c:::benchmark_data_smithi188_141483_object61167:head# 2024-04-23T16:43:52.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7993dd4:::benchmark_data_smithi188_141483_object51361:head# 2024-04-23T16:43:52.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c79943d5:::benchmark_data_smithi188_141483_object86982:head# 2024-04-23T16:43:52.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c79a2031:::benchmark_data_smithi188_141483_object25535:head# 2024-04-23T16:43:52.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c79b8b7f:::benchmark_data_smithi188_141483_object24631:head# 2024-04-23T16:43:52.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c79c4267:::benchmark_data_smithi188_141483_object16022:head# 2024-04-23T16:43:52.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c79c7d59:::benchmark_data_smithi188_141483_object88792:head# 2024-04-23T16:43:52.577 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.577 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a035db:::benchmark_data_smithi188_141483_object65042:head# 2024-04-23T16:43:52.578 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.578 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a08d54:::benchmark_data_smithi188_141483_object75361:head# 2024-04-23T16:43:52.578 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.578 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a17c82:::benchmark_data_smithi188_141483_object41510:head# 2024-04-23T16:43:52.578 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.578 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a1aae1:::benchmark_data_smithi188_141483_object2091:head# 2024-04-23T16:43:52.578 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.578 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a2671d:::benchmark_data_smithi188_141483_object41561:head# 2024-04-23T16:43:52.578 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.578 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a28bf6:::benchmark_data_smithi188_141483_object22283:head# 2024-04-23T16:43:52.578 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.578 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a30ab9:::benchmark_data_smithi188_141483_object71366:head# 2024-04-23T16:43:52.578 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a341a1:::benchmark_data_smithi188_141483_object77334:head# 2024-04-23T16:43:52.579 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a42b6e:::benchmark_data_smithi188_141483_object10304:head# 2024-04-23T16:43:52.579 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a4c5a9:::benchmark_data_smithi188_141483_object10551:head# 2024-04-23T16:43:52.579 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a598c2:::benchmark_data_smithi188_141483_object35956:head# 2024-04-23T16:43:52.579 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a5ed46:::benchmark_data_smithi188_141483_object35357:head# 2024-04-23T16:43:52.579 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a64fd1:::benchmark_data_smithi188_141483_object49042:head# 2024-04-23T16:43:52.579 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.579 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a6c60b:::benchmark_data_smithi188_141483_object24157:head# 2024-04-23T16:43:52.579 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.580 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a724de:::benchmark_data_smithi188_141483_object54477:head# 2024-04-23T16:43:52.580 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.580 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a7fb09:::benchmark_data_smithi188_141483_object60546:head# 2024-04-23T16:43:52.580 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.580 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a877eb:::benchmark_data_smithi188_141483_object66372:head# 2024-04-23T16:43:52.580 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.580 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a8fdc3:::benchmark_data_smithi188_141483_object28948:head# 2024-04-23T16:43:52.580 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.580 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a92fab:::benchmark_data_smithi188_141483_object64359:head# 2024-04-23T16:43:52.580 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.580 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a9e653:::benchmark_data_smithi188_141483_object67499:head# 2024-04-23T16:43:52.580 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.580 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7a9fc2c:::benchmark_data_smithi188_141483_object65277:head# 2024-04-23T16:43:52.580 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.581 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7aacbe0:::benchmark_data_smithi188_141483_object87161:head# 2024-04-23T16:43:52.581 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.581 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7ab9c89:::benchmark_data_smithi188_141483_object68864:head# 2024-04-23T16:43:52.581 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.581 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7ad771a:::benchmark_data_smithi188_141483_object54158:head# 2024-04-23T16:43:52.581 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.581 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7ad8150:::benchmark_data_smithi188_141483_object70456:head# 2024-04-23T16:43:52.581 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.581 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7ad830c:::benchmark_data_smithi188_141483_object85222:head# 2024-04-23T16:43:52.581 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.581 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7ae2d80:::benchmark_data_smithi188_141483_object11950:head# 2024-04-23T16:43:52.581 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.581 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7af0ed6:::benchmark_data_smithi188_141483_object20419:head# 2024-04-23T16:43:52.581 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.581 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7af86db:::benchmark_data_smithi188_141483_object52992:head# 2024-04-23T16:43:52.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7b03675:::benchmark_data_smithi188_141483_object18596:head# 2024-04-23T16:43:52.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7b04c03:::benchmark_data_smithi188_141483_object21390:head# 2024-04-23T16:43:52.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7b21cc7:::benchmark_data_smithi188_141483_object36588:head# 2024-04-23T16:43:52.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7b33d00:::benchmark_data_smithi188_141483_object84502:head# 2024-04-23T16:43:52.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7b346f4:::benchmark_data_smithi188_141483_object69222:head# 2024-04-23T16:43:52.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.582 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7b35a32:::benchmark_data_smithi188_141483_object25677:head# 2024-04-23T16:43:52.582 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7b414f7:::benchmark_data_smithi188_141483_object14293:head# 2024-04-23T16:43:52.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7b51029:::benchmark_data_smithi188_141483_object33001:head# 2024-04-23T16:43:52.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7b52187:::benchmark_data_smithi188_141483_object51787:head# 2024-04-23T16:43:52.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7b59f97:::benchmark_data_smithi188_141483_object17970:head# 2024-04-23T16:43:52.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7b6bdc3:::benchmark_data_smithi188_141483_object32204:head# 2024-04-23T16:43:52.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7b73331:::benchmark_data_smithi188_141483_object19040:head# 2024-04-23T16:43:52.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7b8d483:::benchmark_data_smithi188_141483_object63914:head# 2024-04-23T16:43:52.583 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.583 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7b8f9f1:::benchmark_data_smithi188_141483_object75989:head# 2024-04-23T16:43:52.584 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.584 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7ba2682:::benchmark_data_smithi188_141483_object2680:head# 2024-04-23T16:43:52.584 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.584 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7bb6e7e:::benchmark_data_smithi188_141483_object5387:head# 2024-04-23T16:43:52.584 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.584 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7bc1ec1:::benchmark_data_smithi188_141483_object3826:head# 2024-04-23T16:43:52.584 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.584 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7bc3cc1:::benchmark_data_smithi188_141483_object10891:head# 2024-04-23T16:43:52.584 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.584 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7bc6ca1:::benchmark_data_smithi188_141483_object36328:head# 2024-04-23T16:43:52.584 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.584 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7bcd278:::benchmark_data_smithi188_141483_object37668:head# 2024-04-23T16:43:52.584 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.584 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7bd6f05:::benchmark_data_smithi188_141483_object62203:head# 2024-04-23T16:43:52.585 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.585 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7bd7fbe:::benchmark_data_smithi188_141483_object47251:head# 2024-04-23T16:43:52.585 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.585 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7bd9009:::benchmark_data_smithi188_141483_object23032:head# 2024-04-23T16:43:52.585 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.585 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7bf7a42:::benchmark_data_smithi188_141483_object25148:head# 2024-04-23T16:43:52.585 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.585 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7bf90de:::benchmark_data_smithi188_141483_object69016:head# 2024-04-23T16:43:52.585 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.585 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7c02908:::benchmark_data_smithi188_141483_object5699:head# 2024-04-23T16:43:52.585 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.585 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7c06dcb:::benchmark_data_smithi188_141483_object14784:head# 2024-04-23T16:43:52.585 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.585 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7c08443:::benchmark_data_smithi188_141483_object59143:head# 2024-04-23T16:43:52.586 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.586 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7c105d3:::benchmark_data_smithi188_141483_object41129:head# 2024-04-23T16:43:52.586 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.586 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7c13e9a:::benchmark_data_smithi188_141483_object12344:head# 2024-04-23T16:43:52.586 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.586 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7c4a794:::benchmark_data_smithi188_141483_object67198:head# 2024-04-23T16:43:52.586 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.586 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7c72f93:::benchmark_data_smithi188_141483_object26332:head# 2024-04-23T16:43:52.586 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.586 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7c79f95:::benchmark_data_smithi188_141483_object19542:head# 2024-04-23T16:43:52.586 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.586 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7c830ca:::benchmark_data_smithi188_141483_object90751:head# 2024-04-23T16:43:52.586 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.587 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7c8f5fd:::benchmark_data_smithi188_141483_object75163:head# 2024-04-23T16:43:52.587 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.587 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7c8ff79:::benchmark_data_smithi188_141483_object28953:head# 2024-04-23T16:43:52.587 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.587 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7c9bc00:::benchmark_data_smithi188_141483_object19364:head# 2024-04-23T16:43:52.587 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.587 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7c9f559:::benchmark_data_smithi188_141483_object64603:head# 2024-04-23T16:43:52.587 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.587 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7ca3a87:::benchmark_data_smithi188_141483_object75443:head# 2024-04-23T16:43:52.587 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.587 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7ca8b1d:::benchmark_data_smithi188_141483_object38619:head# 2024-04-23T16:43:52.587 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.587 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7cb528e:::benchmark_data_smithi188_141483_object7969:head# 2024-04-23T16:43:52.587 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.588 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7cb61b5:::benchmark_data_smithi188_141483_object16972:head# 2024-04-23T16:43:52.588 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.588 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7cbd3b9:::benchmark_data_smithi188_141483_object1217:head# 2024-04-23T16:43:52.588 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.588 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7cd1d9b:::benchmark_data_smithi188_141483_object34802:head# 2024-04-23T16:43:52.588 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.588 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7cd8ce7:::benchmark_data_smithi188_141483_object39496:head# 2024-04-23T16:43:52.588 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.588 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7ce0dbe:::benchmark_data_smithi188_141483_object21405:head# 2024-04-23T16:43:52.588 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.588 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7ce5509:::benchmark_data_smithi188_141483_object56255:head# 2024-04-23T16:43:52.588 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.588 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7cfcd48:::benchmark_data_smithi188_141483_object62240:head# 2024-04-23T16:43:52.588 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.589 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7cff05d:::benchmark_data_smithi188_141483_object45070:head# 2024-04-23T16:43:52.589 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.589 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7d0d116:::benchmark_data_smithi188_141483_object84254:head# 2024-04-23T16:43:52.589 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.589 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7d0f1aa:::benchmark_data_smithi188_141483_object28564:head# 2024-04-23T16:43:52.589 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.589 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7d1359e:::benchmark_data_smithi188_141483_object55177:head# 2024-04-23T16:43:52.589 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.589 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7d17501:::benchmark_data_smithi188_141483_object81957:head# 2024-04-23T16:43:52.589 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.589 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7d1a86e:::benchmark_data_smithi188_141483_object5596:head# 2024-04-23T16:43:52.589 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.589 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7d201aa:::benchmark_data_smithi188_141483_object22259:head# 2024-04-23T16:43:52.589 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.590 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7d4134d:::benchmark_data_smithi188_141483_object32529:head# 2024-04-23T16:43:52.590 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.590 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7d4d7be:::benchmark_data_smithi188_141483_object75367:head# 2024-04-23T16:43:52.590 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.590 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7d51535:::benchmark_data_smithi188_141483_object67594:head# 2024-04-23T16:43:52.590 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.590 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7d5302b:::benchmark_data_smithi188_141483_object41999:head# 2024-04-23T16:43:52.590 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.590 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7d5f9d4:::benchmark_data_smithi188_141483_object26487:head# 2024-04-23T16:43:52.590 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.590 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7d8a2e2:::benchmark_data_smithi188_141483_object83607:head# 2024-04-23T16:43:52.590 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.590 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7d9a729:::benchmark_data_smithi188_141483_object19746:head# 2024-04-23T16:43:52.591 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.591 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7dd368c:::benchmark_data_smithi188_141483_object68867:head# 2024-04-23T16:43:52.591 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.591 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7de4d65:::benchmark_data_smithi188_141483_object67064:head# 2024-04-23T16:43:52.591 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.591 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7df7d8d:::benchmark_data_smithi188_141483_object2380:head# 2024-04-23T16:43:52.591 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.591 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7df8867:::benchmark_data_smithi188_141483_object31646:head# 2024-04-23T16:43:52.591 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.591 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7dfeee5:::benchmark_data_smithi188_141483_object5355:head# 2024-04-23T16:43:52.591 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.591 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7e0ee36:::benchmark_data_smithi188_141483_object84679:head# 2024-04-23T16:43:52.591 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.591 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7e110e7:::benchmark_data_smithi188_141483_object14913:head# 2024-04-23T16:43:52.592 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.592 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7e14ecc:::benchmark_data_smithi188_141483_object89843:head# 2024-04-23T16:43:52.592 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.592 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7e24d42:::benchmark_data_smithi188_141483_object79979:head# 2024-04-23T16:43:52.592 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.592 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7e26aef:::benchmark_data_smithi188_141483_object74068:head# 2024-04-23T16:43:52.592 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.592 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7e3106c:::benchmark_data_smithi188_141483_object14944:head# 2024-04-23T16:43:52.592 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.592 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7e33b45:::benchmark_data_smithi188_141483_object87375:head# 2024-04-23T16:43:52.592 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.592 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7e35803:::benchmark_data_smithi188_141483_object10476:head# 2024-04-23T16:43:52.592 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.592 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7e38a80:::benchmark_data_smithi188_141483_object28902:head# 2024-04-23T16:43:52.593 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.593 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7e4b1a0:::benchmark_data_smithi188_141483_object58151:head# 2024-04-23T16:43:52.593 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.593 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7e5b250:::benchmark_data_smithi188_141483_object46144:head# 2024-04-23T16:43:52.593 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.593 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7e5dfbd:::benchmark_data_smithi188_141483_object53603:head# 2024-04-23T16:43:52.593 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.593 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7e5f139:::benchmark_data_smithi188_141483_object71128:head# 2024-04-23T16:43:52.593 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.593 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7e74c03:::benchmark_data_smithi188_141483_object68937:head# 2024-04-23T16:43:52.593 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.593 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7e78adf:::benchmark_data_smithi188_141483_object28494:head# 2024-04-23T16:43:52.593 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.593 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7e7c901:::benchmark_data_smithi188_141483_object85248:head# 2024-04-23T16:43:52.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.594 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7e7f3b0:::benchmark_data_smithi188_141483_object19530:head# 2024-04-23T16:43:52.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.594 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7e8ac97:::benchmark_data_smithi188_141483_object31772:head# 2024-04-23T16:43:52.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.594 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7ea46af:::benchmark_data_smithi188_141483_object78570:head# 2024-04-23T16:43:52.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.594 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7eb6551:::benchmark_data_smithi188_141483_object16107:head# 2024-04-23T16:43:52.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.594 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7ec1e99:::benchmark_data_smithi188_141483_object49673:head# 2024-04-23T16:43:52.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.594 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7ecf04e:::benchmark_data_smithi188_141483_object90353:head# 2024-04-23T16:43:52.594 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7edcae3:::benchmark_data_smithi188_141483_object86170:head# 2024-04-23T16:43:52.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7edd1a2:::benchmark_data_smithi188_141483_object17656:head# 2024-04-23T16:43:52.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7ef52b1:::benchmark_data_smithi188_141483_object37964:head# 2024-04-23T16:43:52.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7efb3b8:::benchmark_data_smithi188_141483_object4880:head# 2024-04-23T16:43:52.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7efbf1a:::benchmark_data_smithi188_141483_object78523:head# 2024-04-23T16:43:52.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7efcbd8:::benchmark_data_smithi188_141483_object2748:head# 2024-04-23T16:43:52.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.595 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7efdfa6:::benchmark_data_smithi188_141483_object38717:head# 2024-04-23T16:43:52.595 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7f042b7:::benchmark_data_smithi188_141483_object30793:head# 2024-04-23T16:43:52.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7f05978:::benchmark_data_smithi188_141483_object90890:head# 2024-04-23T16:43:52.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7f0ccfb:::benchmark_data_smithi188_141483_object80020:head# 2024-04-23T16:43:52.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7f39481:::benchmark_data_smithi188_141483_object62895:head# 2024-04-23T16:43:52.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7f40bd3:::benchmark_data_smithi188_141483_object33484:head# 2024-04-23T16:43:52.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7f43568:::benchmark_data_smithi188_141483_object78061:head# 2024-04-23T16:43:52.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7f518cf:::benchmark_data_smithi188_141483_object85517:head# 2024-04-23T16:43:52.596 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.596 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7f58e81:::benchmark_data_smithi188_141483_object34246:head# 2024-04-23T16:43:52.597 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.597 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7f6f7f0:::benchmark_data_smithi188_141483_object83233:head# 2024-04-23T16:43:52.597 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.597 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7f79f5f:::benchmark_data_smithi188_141483_object53912:head# 2024-04-23T16:43:52.597 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.597 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7f7d127:::benchmark_data_smithi188_141483_object45867:head# 2024-04-23T16:43:52.597 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.597 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7f87454:::benchmark_data_smithi188_141483_object65685:head# 2024-04-23T16:43:52.597 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.597 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7fa6f13:::benchmark_data_smithi188_141483_object10862:head# 2024-04-23T16:43:52.597 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.597 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7fa8586:::benchmark_data_smithi188_141483_object58122:head# 2024-04-23T16:43:52.597 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.597 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7fab197:::benchmark_data_smithi188_141483_object57441:head# 2024-04-23T16:43:52.598 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.598 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7fb7e3a:::benchmark_data_smithi188_141483_object80647:head# 2024-04-23T16:43:52.598 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.598 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7fb81a6:::benchmark_data_smithi188_141483_object27038:head# 2024-04-23T16:43:52.598 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.598 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7fc6ad6:::benchmark_data_smithi188_141483_object52889:head# 2024-04-23T16:43:52.598 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.598 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7fcbffd:::benchmark_data_smithi188_141483_object3670:head# 2024-04-23T16:43:52.598 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.598 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c7fe98a2:::benchmark_data_smithi188_141483_object13247:head# 2024-04-23T16:43:52.598 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.598 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c800b8dd:::benchmark_data_smithi188_141483_object7252:head# 2024-04-23T16:43:52.598 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.598 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c802ec7e:::benchmark_data_smithi188_141483_object40029:head# 2024-04-23T16:43:52.599 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.599 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c80321d8:::benchmark_data_smithi188_141483_object86372:head# 2024-04-23T16:43:52.599 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.599 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8032c59:::benchmark_data_smithi188_141483_object34873:head# 2024-04-23T16:43:52.599 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.599 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c804f7af:::benchmark_data_smithi188_141483_object81484:head# 2024-04-23T16:43:52.599 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.599 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8059fa7:::benchmark_data_smithi188_141483_object47890:head# 2024-04-23T16:43:52.599 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.599 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8064f3c:::benchmark_data_smithi188_141483_object73395:head# 2024-04-23T16:43:52.599 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.599 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c806568e:::benchmark_data_smithi188_141483_object34475:head# 2024-04-23T16:43:52.599 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.599 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c807654f:::benchmark_data_smithi188_141483_object84225:head# 2024-04-23T16:43:52.600 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.600 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c807c529:::benchmark_data_smithi188_141483_object50205:head# 2024-04-23T16:43:52.600 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.600 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8083f78:::benchmark_data_smithi188_141483_object62709:head# 2024-04-23T16:43:52.600 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.600 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8084fe5:::benchmark_data_smithi188_141483_object80854:head# 2024-04-23T16:43:52.600 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.600 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8086fda:::benchmark_data_smithi188_141483_object49122:head# 2024-04-23T16:43:52.600 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.600 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8087470:::benchmark_data_smithi188_141483_object3073:head# 2024-04-23T16:43:52.600 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.600 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c80940a5:::benchmark_data_smithi188_141483_object51414:head# 2024-04-23T16:43:52.600 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.600 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c80981cb:::benchmark_data_smithi188_141483_object50376:head# 2024-04-23T16:43:52.601 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c809f3bc:::benchmark_data_smithi188_141483_object55589:head# 2024-04-23T16:43:52.601 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c80aa60f:::benchmark_data_smithi188_141483_object34147:head# 2024-04-23T16:43:52.601 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c80bdac1:::benchmark_data_smithi188_141483_object35493:head# 2024-04-23T16:43:52.601 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c80bfbdd:::benchmark_data_smithi188_141483_object73250:head# 2024-04-23T16:43:52.601 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c80ca5d1:::benchmark_data_smithi188_141483_object59665:head# 2024-04-23T16:43:52.601 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c80dcae9:::benchmark_data_smithi188_141483_object16244:head# 2024-04-23T16:43:52.601 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.601 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c80dcfd7:::benchmark_data_smithi188_141483_object14909:head# 2024-04-23T16:43:52.602 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.602 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c80f8beb:::benchmark_data_smithi188_141483_object26374:head# 2024-04-23T16:43:52.602 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.602 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c80fab79:::benchmark_data_smithi188_141483_object57763:head# 2024-04-23T16:43:52.602 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.602 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c80fd62b:::benchmark_data_smithi188_141483_object30689:head# 2024-04-23T16:43:52.602 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.602 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c81038c9:::benchmark_data_smithi188_141483_object6953:head# 2024-04-23T16:43:52.602 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.602 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8113a34:::benchmark_data_smithi188_141483_object4832:head# 2024-04-23T16:43:52.602 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.602 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8114f9c:::benchmark_data_smithi188_141483_object88298:head# 2024-04-23T16:43:52.602 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c811d2cb:::benchmark_data_smithi188_141483_object6506:head# 2024-04-23T16:43:52.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8127e70:::benchmark_data_smithi188_141483_object30989:head# 2024-04-23T16:43:52.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c812f69c:::benchmark_data_smithi188_141483_object23685:head# 2024-04-23T16:43:52.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c814c9f6:::benchmark_data_smithi188_141483_object79856:head# 2024-04-23T16:43:52.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c81557fb:::benchmark_data_smithi188_141483_object70726:head# 2024-04-23T16:43:52.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c81564a5:::benchmark_data_smithi188_141483_object46755:head# 2024-04-23T16:43:52.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.603 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8159de4:::benchmark_data_smithi188_141483_object67290:head# 2024-04-23T16:43:52.603 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8160143:::benchmark_data_smithi188_141483_object8030:head# 2024-04-23T16:43:52.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8163204:::benchmark_data_smithi188_141483_object7774:head# 2024-04-23T16:43:52.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c816c344:::benchmark_data_smithi188_141483_object11921:head# 2024-04-23T16:43:52.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c816ec31:::benchmark_data_smithi188_141483_object67524:head# 2024-04-23T16:43:52.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c817c137:::benchmark_data_smithi188_141483_object26884:head# 2024-04-23T16:43:52.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c817f692:::benchmark_data_smithi188_141483_object55342:head# 2024-04-23T16:43:52.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.604 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8194d88:::benchmark_data_smithi188_141483_object17419:head# 2024-04-23T16:43:52.604 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.605 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8195ef6:::benchmark_data_smithi188_141483_object67407:head# 2024-04-23T16:43:52.605 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.605 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c819ce09:::benchmark_data_smithi188_141483_object61280:head# 2024-04-23T16:43:52.605 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.605 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c81ab53e:::benchmark_data_smithi188_141483_object90230:head# 2024-04-23T16:43:52.605 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.605 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c81babf3:::benchmark_data_smithi188_141483_object85355:head# 2024-04-23T16:43:52.605 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.605 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c81c2ecb:::benchmark_data_smithi188_141483_object10262:head# 2024-04-23T16:43:52.605 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.605 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c81c7242:::benchmark_data_smithi188_141483_object79504:head# 2024-04-23T16:43:52.605 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.605 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c81cf7f3:::benchmark_data_smithi188_141483_object52325:head# 2024-04-23T16:43:52.605 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.606 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c81d132b:::benchmark_data_smithi188_141483_object15298:head# 2024-04-23T16:43:52.606 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.606 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c81df3b3:::benchmark_data_smithi188_141483_object35154:head# 2024-04-23T16:43:52.606 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.606 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c81e8a30:::benchmark_data_smithi188_141483_object85501:head# 2024-04-23T16:43:52.606 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.606 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c820b4f2:::benchmark_data_smithi188_141483_object36672:head# 2024-04-23T16:43:52.606 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.606 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c82118d0:::benchmark_data_smithi188_141483_object89442:head# 2024-04-23T16:43:52.606 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.606 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8216bb7:::benchmark_data_smithi188_141483_object79272:head# 2024-04-23T16:43:52.606 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.606 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c822ca58:::benchmark_data_smithi188_141483_object71193:head# 2024-04-23T16:43:52.607 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.607 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8240e47:::benchmark_data_smithi188_141483_object61090:head# 2024-04-23T16:43:52.607 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.607 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8261fcb:::benchmark_data_smithi188_141483_object55220:head# 2024-04-23T16:43:52.607 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.607 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c826cd16:::benchmark_data_smithi188_141483_object78793:head# 2024-04-23T16:43:52.607 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.607 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c826e43b:::benchmark_data_smithi188_141483_object5696:head# 2024-04-23T16:43:52.607 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.607 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c82784c1:::benchmark_data_smithi188_141483_object85299:head# 2024-04-23T16:43:52.607 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.607 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c828373a:::benchmark_data_smithi188_141483_object90438:head# 2024-04-23T16:43:52.607 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.607 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c828ea27:::benchmark_data_smithi188_141483_object81621:head# 2024-04-23T16:43:52.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c829fb73:::benchmark_data_smithi188_141483_object87283:head# 2024-04-23T16:43:52.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c82a2683:::benchmark_data_smithi188_141483_object25356:head# 2024-04-23T16:43:52.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c82a534b:::benchmark_data_smithi188_141483_object52489:head# 2024-04-23T16:43:52.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c82b9271:::benchmark_data_smithi188_141483_object4686:head# 2024-04-23T16:43:52.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c82c9c2d:::benchmark_data_smithi188_141483_object70914:head# 2024-04-23T16:43:52.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c82caa74:::benchmark_data_smithi188_141483_object55036:head# 2024-04-23T16:43:52.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.608 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c82cd0d0:::benchmark_data_smithi188_141483_object86645:head# 2024-04-23T16:43:52.608 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c82ce4cf:::benchmark_data_smithi188_141483_object47262:head# 2024-04-23T16:43:52.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c82e4393:::benchmark_data_smithi188_141483_object90344:head# 2024-04-23T16:43:52.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c83005a2:::benchmark_data_smithi188_141483_object88080:head# 2024-04-23T16:43:52.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8314294:::benchmark_data_smithi188_141483_object82366:head# 2024-04-23T16:43:52.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c83267fb:::benchmark_data_smithi188_141483_object14748:head# 2024-04-23T16:43:52.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c83582ab:::benchmark_data_smithi188_141483_object29395:head# 2024-04-23T16:43:52.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.609 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c835b411:::benchmark_data_smithi188_141483_object73769:head# 2024-04-23T16:43:52.609 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c838ea82:::benchmark_data_smithi188_141483_object27528:head# 2024-04-23T16:43:52.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c83b55c8:::benchmark_data_smithi188_141483_object17368:head# 2024-04-23T16:43:52.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c83bb0d8:::benchmark_data_smithi188_141483_object29697:head# 2024-04-23T16:43:52.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c83d8011:::benchmark_data_smithi188_141483_object46018:head# 2024-04-23T16:43:52.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c83e7290:::benchmark_data_smithi188_141483_object40441:head# 2024-04-23T16:43:52.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8408d23:::benchmark_data_smithi188_141483_object198:head# 2024-04-23T16:43:52.610 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.610 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c840c300:::benchmark_data_smithi188_141483_object78512:head# 2024-04-23T16:43:52.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c842a76d:::benchmark_data_smithi188_141483_object25562:head# 2024-04-23T16:43:52.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c84369a5:::benchmark_data_smithi188_141483_object83320:head# 2024-04-23T16:43:52.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c84405c1:::benchmark_data_smithi188_141483_object84552:head# 2024-04-23T16:43:52.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8444d7e:::benchmark_data_smithi188_141483_object70503:head# 2024-04-23T16:43:52.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8464f15:::benchmark_data_smithi188_141483_object87428:head# 2024-04-23T16:43:52.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c847a971:::benchmark_data_smithi188_141483_object39707:head# 2024-04-23T16:43:52.611 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.611 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c848e27b:::benchmark_data_smithi188_141483_object81838:head# 2024-04-23T16:43:52.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c84a9ab0:::benchmark_data_smithi188_141483_object70089:head# 2024-04-23T16:43:52.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c84ae123:::benchmark_data_smithi188_141483_object10567:head# 2024-04-23T16:43:52.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c84c4fba:::benchmark_data_smithi188_141483_object73889:head# 2024-04-23T16:43:52.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c84c62fc:::benchmark_data_smithi188_141483_object47358:head# 2024-04-23T16:43:52.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c84cf37d:::benchmark_data_smithi188_141483_object78746:head# 2024-04-23T16:43:52.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c84e6467:::benchmark_data_smithi188_141483_object69744:head# 2024-04-23T16:43:52.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.612 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c84fa6fd:::benchmark_data_smithi188_141483_object45036:head# 2024-04-23T16:43:52.612 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c84fd906:::benchmark_data_smithi188_141483_object69168:head# 2024-04-23T16:43:52.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c84fe8cc:::benchmark_data_smithi188_141483_object34670:head# 2024-04-23T16:43:52.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8506a59:::benchmark_data_smithi188_141483_object55505:head# 2024-04-23T16:43:52.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c850b74a:::benchmark_data_smithi188_141483_object73372:head# 2024-04-23T16:43:52.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8516f83:::benchmark_data_smithi188_141483_object57909:head# 2024-04-23T16:43:52.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c851fe02:::benchmark_data_smithi188_141483_object58146:head# 2024-04-23T16:43:52.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.613 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8524bdd:::benchmark_data_smithi188_141483_object9413:head# 2024-04-23T16:43:52.613 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c85268da:::benchmark_data_smithi188_141483_object52450:head# 2024-04-23T16:43:52.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8526efb:::benchmark_data_smithi188_141483_object27205:head# 2024-04-23T16:43:52.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8527109:::benchmark_data_smithi188_141483_object67265:head# 2024-04-23T16:43:52.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c853483d:::benchmark_data_smithi188_141483_object65197:head# 2024-04-23T16:43:52.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c85424ff:::benchmark_data_smithi188_141483_object51520:head# 2024-04-23T16:43:52.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c85452f7:::benchmark_data_smithi188_141483_object11445:head# 2024-04-23T16:43:52.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.614 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c855890b:::benchmark_data_smithi188_141483_object36219:head# 2024-04-23T16:43:52.614 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8563d4b:::benchmark_data_smithi188_141483_object55207:head# 2024-04-23T16:43:52.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c85705df:::benchmark_data_smithi188_141483_object84459:head# 2024-04-23T16:43:52.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c857f573:::benchmark_data_smithi188_141483_object24406:head# 2024-04-23T16:43:52.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c859b47e:::benchmark_data_smithi188_141483_object1153:head# 2024-04-23T16:43:52.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c85aab0d:::benchmark_data_smithi188_141483_object14157:head# 2024-04-23T16:43:52.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c85ae457:::benchmark_data_smithi188_141483_object25122:head# 2024-04-23T16:43:52.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.615 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c85c3ed5:::benchmark_data_smithi188_141483_object1208:head# 2024-04-23T16:43:52.615 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c85d7a42:::benchmark_data_smithi188_141483_object62104:head# 2024-04-23T16:43:52.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c85d7bf7:::benchmark_data_smithi188_141483_object49889:head# 2024-04-23T16:43:52.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c85e5027:::benchmark_data_smithi188_141483_object56259:head# 2024-04-23T16:43:52.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8611204:::benchmark_data_smithi188_141483_object49040:head# 2024-04-23T16:43:52.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8612813:::benchmark_data_smithi188_141483_object16962:head# 2024-04-23T16:43:52.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8612b60:::benchmark_data_smithi188_141483_object22861:head# 2024-04-23T16:43:52.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.616 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8646559:::benchmark_data_smithi188_141483_object16893:head# 2024-04-23T16:43:52.616 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c865805b:::benchmark_data_smithi188_141483_object50776:head# 2024-04-23T16:43:52.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c867dbe1:::benchmark_data_smithi188_141483_object82221:head# 2024-04-23T16:43:52.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8681de3:::benchmark_data_smithi188_141483_object69574:head# 2024-04-23T16:43:52.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8687ca0:::benchmark_data_smithi188_141483_object78559:head# 2024-04-23T16:43:52.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8696f39:::benchmark_data_smithi188_141483_object31773:head# 2024-04-23T16:43:52.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c86976c6:::benchmark_data_smithi188_141483_object88296:head# 2024-04-23T16:43:52.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.617 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8698508:::benchmark_data_smithi188_141483_object9238:head# 2024-04-23T16:43:52.617 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c86a3658:::benchmark_data_smithi188_141483_object21633:head# 2024-04-23T16:43:52.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c86b9a55:::benchmark_data_smithi188_141483_object71309:head# 2024-04-23T16:43:52.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c86c052f:::benchmark_data_smithi188_141483_object7904:head# 2024-04-23T16:43:52.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c86c4d05:::benchmark_data_smithi188_141483_object26874:head# 2024-04-23T16:43:52.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c86cb88d:::benchmark_data_smithi188_141483_object76437:head# 2024-04-23T16:43:52.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c86d8ec2:::benchmark_data_smithi188_141483_object78346:head# 2024-04-23T16:43:52.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.618 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c86dd760:::benchmark_data_smithi188_141483_object31756:head# 2024-04-23T16:43:52.618 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c86eb279:::benchmark_data_smithi188_141483_object27933:head# 2024-04-23T16:43:52.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c86ee14e:::benchmark_data_smithi188_141483_object74363:head# 2024-04-23T16:43:52.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c86f42ec:::benchmark_data_smithi188_141483_object46151:head# 2024-04-23T16:43:52.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8709dd2:::benchmark_data_smithi188_141483_object53437:head# 2024-04-23T16:43:52.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c870bb18:::benchmark_data_smithi188_141483_object39500:head# 2024-04-23T16:43:52.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c87389c7:::benchmark_data_smithi188_141483_object13739:head# 2024-04-23T16:43:52.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.619 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c874a1b4:::benchmark_data_smithi188_141483_object33859:head# 2024-04-23T16:43:52.619 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c874b71b:::benchmark_data_smithi188_141483_object56060:head# 2024-04-23T16:43:52.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c874d338:::benchmark_data_smithi188_141483_object84149:head# 2024-04-23T16:43:52.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c874f43a:::benchmark_data_smithi188_141483_object53251:head# 2024-04-23T16:43:52.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c876d707:::benchmark_data_smithi188_141483_object63877:head# 2024-04-23T16:43:52.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c877269f:::benchmark_data_smithi188_141483_object51390:head# 2024-04-23T16:43:52.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.620 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c877784f:::benchmark_data_smithi188_141483_object54160:head# 2024-04-23T16:43:52.620 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.621 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:52.621 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:52.621 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:52.621 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:52.621 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:52.621 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:52 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:43:52.621 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:52 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:43:52.621 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:52 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:43:52.621 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:52 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:43:52.624 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:43:52.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:52.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:52.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:52.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:52.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:52.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:52 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:43:52.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:52 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:43:52.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:52 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:43:52.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:52 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:43:52.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:52.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:43:52.732 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:43:52.778 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:52.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c87780b5:::benchmark_data_smithi188_141483_object3859:head# 2024-04-23T16:43:52.779 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.779 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c877867c:::benchmark_data_smithi188_141483_object15212:head# 2024-04-23T16:43:52.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8784b2e:::benchmark_data_smithi188_141483_object37289:head# 2024-04-23T16:43:52.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8787202:::benchmark_data_smithi188_141483_object2902:head# 2024-04-23T16:43:52.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8790ce7:::benchmark_data_smithi188_141483_object51054:head# 2024-04-23T16:43:52.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8796a75:::benchmark_data_smithi188_141483_object50409:head# 2024-04-23T16:43:52.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.780 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c879b1ba:::benchmark_data_smithi188_141483_object22854:head# 2024-04-23T16:43:52.780 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c879d609:::benchmark_data_smithi188_141483_object59414:head# 2024-04-23T16:43:52.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c87b7843:::benchmark_data_smithi188_141483_object80119:head# 2024-04-23T16:43:52.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c87c5580:::benchmark_data_smithi188_141483_object42698:head# 2024-04-23T16:43:52.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c87d8143:::benchmark_data_smithi188_141483_object74577:head# 2024-04-23T16:43:52.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c87e41ac:::benchmark_data_smithi188_141483_object44938:head# 2024-04-23T16:43:52.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c87fd055:::benchmark_data_smithi188_141483_object62845:head# 2024-04-23T16:43:52.781 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.781 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8800b0b:::benchmark_data_smithi188_141483_object22126:head# 2024-04-23T16:43:52.782 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.782 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c880f196:::benchmark_data_smithi188_141483_object79217:head# 2024-04-23T16:43:52.782 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.782 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c881588b:::benchmark_data_smithi188_141483_object84142:head# 2024-04-23T16:43:52.782 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.782 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c881cc0c:::benchmark_data_smithi188_141483_object41275:head# 2024-04-23T16:43:52.782 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.782 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c882a598:::benchmark_data_smithi188_141483_object27046:head# 2024-04-23T16:43:52.782 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.782 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c882e130:::benchmark_data_smithi188_141483_object3204:head# 2024-04-23T16:43:52.782 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.782 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c885249c:::benchmark_data_smithi188_141483_object51476:head# 2024-04-23T16:43:52.782 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.782 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c886932b:::benchmark_data_smithi188_141483_object29515:head# 2024-04-23T16:43:52.783 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.783 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c888a580:::benchmark_data_smithi188_141483_object42303:head# 2024-04-23T16:43:52.783 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.783 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c888ed7f:::benchmark_data_smithi188_141483_object13525:head# 2024-04-23T16:43:52.783 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.783 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8892cd1:::benchmark_data_smithi188_141483_object39881:head# 2024-04-23T16:43:52.783 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.783 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c889ea8e:::benchmark_data_smithi188_141483_object76567:head# 2024-04-23T16:43:52.783 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.783 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c88a3a9f:::benchmark_data_smithi188_141483_object67279:head# 2024-04-23T16:43:52.783 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.783 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c88a898e:::benchmark_data_smithi188_141483_object83234:head# 2024-04-23T16:43:52.783 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c88ab58b:::benchmark_data_smithi188_141483_object20724:head# 2024-04-23T16:43:52.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c88d37d6:::benchmark_data_smithi188_141483_object3078:head# 2024-04-23T16:43:52.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c88e9b37:::benchmark_data_smithi188_141483_object75003:head# 2024-04-23T16:43:52.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c88ec25a:::benchmark_data_smithi188_141483_object24676:head# 2024-04-23T16:43:52.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c88f709c:::benchmark_data_smithi188_141483_object88519:head# 2024-04-23T16:43:52.784 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.784 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8917022:::benchmark_data_smithi188_141483_object15591:head# 2024-04-23T16:43:52.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c89375df:::benchmark_data_smithi188_141483_object10950:head# 2024-04-23T16:43:52.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c89384b6:::benchmark_data_smithi188_141483_object20686:head# 2024-04-23T16:43:52.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c89447c6:::benchmark_data_smithi188_141483_object51192:head# 2024-04-23T16:43:52.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c894e060:::benchmark_data_smithi188_141483_object42625:head# 2024-04-23T16:43:52.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c894e947:::benchmark_data_smithi188_141483_object30643:head# 2024-04-23T16:43:52.785 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.785 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c89537e5:::benchmark_data_smithi188_141483_object90262:head# 2024-04-23T16:43:52.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8954c07:::benchmark_data_smithi188_141483_object71752:head# 2024-04-23T16:43:52.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8958722:::benchmark_data_smithi188_141483_object15440:head# 2024-04-23T16:43:52.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8959b75:::benchmark_data_smithi188_141483_object90032:head# 2024-04-23T16:43:52.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c897bcd9:::benchmark_data_smithi188_141483_object82261:head# 2024-04-23T16:43:52.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c89a7160:::benchmark_data_smithi188_141483_object7981:head# 2024-04-23T16:43:52.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c89adac8:::benchmark_data_smithi188_141483_object18413:head# 2024-04-23T16:43:52.786 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.786 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c89aea8c:::benchmark_data_smithi188_141483_object89783:head# 2024-04-23T16:43:52.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c89aecd4:::benchmark_data_smithi188_141483_object82966:head# 2024-04-23T16:43:52.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c89be4b0:::benchmark_data_smithi188_141483_object65765:head# 2024-04-23T16:43:52.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c89bef84:::benchmark_data_smithi188_141483_object27644:head# 2024-04-23T16:43:52.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c89c45d3:::benchmark_data_smithi188_141483_object81929:head# 2024-04-23T16:43:52.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c89c5dea:::benchmark_data_smithi188_141483_object46251:head# 2024-04-23T16:43:52.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c89d33d8:::benchmark_data_smithi188_141483_object79668:head# 2024-04-23T16:43:52.787 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.787 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c89e1f65:::benchmark_data_smithi188_141483_object59762:head# 2024-04-23T16:43:52.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c89e23b7:::benchmark_data_smithi188_141483_object3868:head# 2024-04-23T16:43:52.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8a03376:::benchmark_data_smithi188_141483_object19915:head# 2024-04-23T16:43:52.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8a07c6b:::benchmark_data_smithi188_141483_object83030:head# 2024-04-23T16:43:52.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8a0ffc4:::benchmark_data_smithi188_141483_object57202:head# 2024-04-23T16:43:52.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8a14472:::benchmark_data_smithi188_141483_object66902:head# 2024-04-23T16:43:52.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8a1fa6e:::benchmark_data_smithi188_141483_object32047:head# 2024-04-23T16:43:52.788 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.788 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8a31243:::benchmark_data_smithi188_141483_object78651:head# 2024-04-23T16:43:52.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8a3abfc:::benchmark_data_smithi188_141483_object41830:head# 2024-04-23T16:43:52.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8a3e976:::benchmark_data_smithi188_141483_object18317:head# 2024-04-23T16:43:52.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8a42448:::benchmark_data_smithi188_141483_object44352:head# 2024-04-23T16:43:52.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8a4627b:::benchmark_data_smithi188_141483_object42952:head# 2024-04-23T16:43:52.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8a5541d:::benchmark_data_smithi188_141483_object21302:head# 2024-04-23T16:43:52.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8a59bb5:::benchmark_data_smithi188_141483_object67616:head# 2024-04-23T16:43:52.789 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.789 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8a65c37:::benchmark_data_smithi188_141483_object51619:head# 2024-04-23T16:43:52.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8a677d6:::benchmark_data_smithi188_141483_object70242:head# 2024-04-23T16:43:52.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8a6c8a3:::benchmark_data_smithi188_141483_object53921:head# 2024-04-23T16:43:52.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8a75afe:::benchmark_data_smithi188_141483_object77235:head# 2024-04-23T16:43:52.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8a80049:::benchmark_data_smithi188_141483_object26448:head# 2024-04-23T16:43:52.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8a823b8:::benchmark_data_smithi188_141483_object71416:head# 2024-04-23T16:43:52.790 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.790 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8a9e57c:::benchmark_data_smithi188_141483_object120:head# 2024-04-23T16:43:52.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.791 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8aae6b0:::benchmark_data_smithi188_141483_object81497:head# 2024-04-23T16:43:52.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.791 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8ad7534:::benchmark_data_smithi188_141483_object6420:head# 2024-04-23T16:43:52.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.791 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8af34a4:::benchmark_data_smithi188_141483_object5768:head# 2024-04-23T16:43:52.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.791 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8b0e2bb:::benchmark_data_smithi188_141483_object9142:head# 2024-04-23T16:43:52.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.791 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8b3d364:::benchmark_data_smithi188_141483_object89192:head# 2024-04-23T16:43:52.791 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.791 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8b41db0:::benchmark_data_smithi188_141483_object43709:head# 2024-04-23T16:43:52.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8b4c95e:::benchmark_data_smithi188_141483_object47940:head# 2024-04-23T16:43:52.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8b52b9b:::benchmark_data_smithi188_141483_object61231:head# 2024-04-23T16:43:52.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8b5e0aa:::benchmark_data_smithi188_141483_object56315:head# 2024-04-23T16:43:52.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8b6df54:::benchmark_data_smithi188_141483_object38693:head# 2024-04-23T16:43:52.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8b70577:::benchmark_data_smithi188_141483_object45288:head# 2024-04-23T16:43:52.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.792 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8b70d0d:::benchmark_data_smithi188_141483_object44589:head# 2024-04-23T16:43:52.792 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8b86b22:::benchmark_data_smithi188_141483_object89145:head# 2024-04-23T16:43:52.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8b9afc6:::benchmark_data_smithi188_141483_object19945:head# 2024-04-23T16:43:52.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8bb8910:::benchmark_data_smithi188_141483_object21971:head# 2024-04-23T16:43:52.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8bbafe9:::benchmark_data_smithi188_141483_object9325:head# 2024-04-23T16:43:52.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8bc4099:::benchmark_data_smithi188_141483_object76396:head# 2024-04-23T16:43:52.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8bc69c5:::benchmark_data_smithi188_141483_object957:head# 2024-04-23T16:43:52.793 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.793 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8bcd7f4:::benchmark_data_smithi188_141483_object19382:head# 2024-04-23T16:43:52.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8bd9f2a:::benchmark_data_smithi188_141483_object44950:head# 2024-04-23T16:43:52.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8bf4efa:::benchmark_data_smithi188_141483_object57285:head# 2024-04-23T16:43:52.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8c11142:::benchmark_data_smithi188_141483_object85961:head# 2024-04-23T16:43:52.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.794 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8c16a3b:::benchmark_data_smithi188_141483_object41755:head# 2024-04-23T16:43:52.794 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8c1ed85:::benchmark_data_smithi188_141483_object25273:head# 2024-04-23T16:43:52.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8c2171c:::benchmark_data_smithi188_141483_object28469:head# 2024-04-23T16:43:52.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8c22722:::benchmark_data_smithi188_141483_object9790:head# 2024-04-23T16:43:52.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8c22e27:::benchmark_data_smithi188_141483_object2784:head# 2024-04-23T16:43:52.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8c330d5:::benchmark_data_smithi188_141483_object43251:head# 2024-04-23T16:43:52.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.795 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8c36794:::benchmark_data_smithi188_141483_object39278:head# 2024-04-23T16:43:52.795 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.796 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8c396ea:::benchmark_data_smithi188_141483_object45785:head# 2024-04-23T16:43:52.796 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.796 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8c529b1:::benchmark_data_smithi188_141483_object59233:head# 2024-04-23T16:43:52.796 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.796 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8c6d14f:::benchmark_data_smithi188_141483_object59261:head# 2024-04-23T16:43:52.796 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.796 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8c7e5c2:::benchmark_data_smithi188_141483_object90861:head# 2024-04-23T16:43:52.796 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.796 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8c8088f:::benchmark_data_smithi188_141483_object49234:head# 2024-04-23T16:43:52.796 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.796 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8c8c6d6:::benchmark_data_smithi188_141483_object49505:head# 2024-04-23T16:43:52.796 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8c9ab1c:::benchmark_data_smithi188_141483_object54047:head# 2024-04-23T16:43:52.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8cab903:::benchmark_data_smithi188_141483_object6144:head# 2024-04-23T16:43:52.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8cb5193:::benchmark_data_smithi188_141483_object22001:head# 2024-04-23T16:43:52.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8cb72ed:::benchmark_data_smithi188_141483_object53464:head# 2024-04-23T16:43:52.797 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.797 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8cbc6fe:::benchmark_data_smithi188_141483_object26251:head# 2024-04-23T16:43:52.798 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.798 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8cd108c:::benchmark_data_smithi188_141483_object21463:head# 2024-04-23T16:43:52.798 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.798 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8cd2d85:::benchmark_data_smithi188_141483_object29218:head# 2024-04-23T16:43:52.798 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.798 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8cd3962:::benchmark_data_smithi188_141483_object31353:head# 2024-04-23T16:43:52.798 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.798 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8cfdfe5:::benchmark_data_smithi188_141483_object75848:head# 2024-04-23T16:43:52.798 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.798 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8d4fef3:::benchmark_data_smithi188_141483_object4177:head# 2024-04-23T16:43:52.798 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.798 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8d6d958:::benchmark_data_smithi188_141483_object654:head# 2024-04-23T16:43:52.799 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.799 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8d706de:::benchmark_data_smithi188_141483_object79509:head# 2024-04-23T16:43:52.799 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.799 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8d7f605:::benchmark_data_smithi188_141483_object36799:head# 2024-04-23T16:43:52.799 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.799 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8d8d6df:::benchmark_data_smithi188_141483_object2939:head# 2024-04-23T16:43:52.799 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.799 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8d93881:::benchmark_data_smithi188_141483_object54674:head# 2024-04-23T16:43:52.799 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.799 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8db685f:::benchmark_data_smithi188_141483_object25598:head# 2024-04-23T16:43:52.799 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8db9cce:::benchmark_data_smithi188_141483_object76848:head# 2024-04-23T16:43:52.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8dd2fbb:::benchmark_data_smithi188_141483_object73718:head# 2024-04-23T16:43:52.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8df0901:::benchmark_data_smithi188_141483_object51853:head# 2024-04-23T16:43:52.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8dfd06e:::benchmark_data_smithi188_141483_object51572:head# 2024-04-23T16:43:52.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8e018ad:::benchmark_data_smithi188_141483_object39964:head# 2024-04-23T16:43:52.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8e0a9d4:::benchmark_data_smithi188_141483_object53521:head# 2024-04-23T16:43:52.800 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.800 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8e22c25:::benchmark_data_smithi188_141483_object88884:head# 2024-04-23T16:43:52.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8e2ce7c:::benchmark_data_smithi188_141483_object4398:head# 2024-04-23T16:43:52.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8e33667:::benchmark_data_smithi188_141483_object66805:head# 2024-04-23T16:43:52.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8e3cd9d:::benchmark_data_smithi188_141483_object58504:head# 2024-04-23T16:43:52.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8e46104:::benchmark_data_smithi188_141483_object60743:head# 2024-04-23T16:43:52.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8e670ab:::benchmark_data_smithi188_141483_object49317:head# 2024-04-23T16:43:52.801 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.801 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8e77960:::benchmark_data_smithi188_141483_object36493:head# 2024-04-23T16:43:52.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8e7a5fc:::benchmark_data_smithi188_141483_object89075:head# 2024-04-23T16:43:52.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8e80f1a:::benchmark_data_smithi188_141483_object73657:head# 2024-04-23T16:43:52.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8e9b18f:::benchmark_data_smithi188_141483_object5557:head# 2024-04-23T16:43:52.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8ead7df:::benchmark_data_smithi188_141483_object82147:head# 2024-04-23T16:43:52.802 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.802 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8eaea58:::benchmark_data_smithi188_141483_object83332:head# 2024-04-23T16:43:52.803 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.803 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8ec5237:::benchmark_data_smithi188_141483_object30783:head# 2024-04-23T16:43:52.803 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.803 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8ece313:::benchmark_data_smithi188_141483_object22711:head# 2024-04-23T16:43:52.803 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.803 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8edb96b:::benchmark_data_smithi188_141483_object79364:head# 2024-04-23T16:43:52.803 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.803 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8edcd62:::benchmark_data_smithi188_141483_object66409:head# 2024-04-23T16:43:52.803 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.803 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8eddcc3:::benchmark_data_smithi188_141483_object84546:head# 2024-04-23T16:43:52.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8ee2f77:::benchmark_data_smithi188_141483_object84314:head# 2024-04-23T16:43:52.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8ef5c1d:::benchmark_data_smithi188_141483_object27930:head# 2024-04-23T16:43:52.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8ef9a3e:::benchmark_data_smithi188_141483_object67813:head# 2024-04-23T16:43:52.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8f142e6:::benchmark_data_smithi188_141483_object73748:head# 2024-04-23T16:43:52.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8f1e5c3:::benchmark_data_smithi188_141483_object82949:head# 2024-04-23T16:43:52.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.804 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8f2b517:::benchmark_data_smithi188_141483_object77166:head# 2024-04-23T16:43:52.804 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8f2f76c:::benchmark_data_smithi188_141483_object43991:head# 2024-04-23T16:43:52.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8f3acca:::benchmark_data_smithi188_141483_object64649:head# 2024-04-23T16:43:52.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8f3af08:::benchmark_data_smithi188_141483_object16938:head# 2024-04-23T16:43:52.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8f4b691:::benchmark_data_smithi188_141483_object81240:head# 2024-04-23T16:43:52.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8f4e267:::benchmark_data_smithi188_141483_object3810:head# 2024-04-23T16:43:52.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8f602ee:::benchmark_data_smithi188_141483_object67711:head# 2024-04-23T16:43:52.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.805 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8f620be:::benchmark_data_smithi188_141483_object58369:head# 2024-04-23T16:43:52.805 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8f6ce5e:::benchmark_data_smithi188_141483_object64817:head# 2024-04-23T16:43:52.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8f7e417:::benchmark_data_smithi188_141483_object48817:head# 2024-04-23T16:43:52.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8f991d9:::benchmark_data_smithi188_141483_object31929:head# 2024-04-23T16:43:52.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8fa8db7:::benchmark_data_smithi188_141483_object53532:head# 2024-04-23T16:43:52.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8fb6587:::benchmark_data_smithi188_141483_object81730:head# 2024-04-23T16:43:52.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8fcc87c:::benchmark_data_smithi188_141483_object57961:head# 2024-04-23T16:43:52.806 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.806 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8fd3de7:::benchmark_data_smithi188_141483_object79850:head# 2024-04-23T16:43:52.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8fe49e1:::benchmark_data_smithi188_141483_object31193:head# 2024-04-23T16:43:52.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c8ff5f49:::benchmark_data_smithi188_141483_object58632:head# 2024-04-23T16:43:52.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c901bfec:::benchmark_data_smithi188_141483_object15375:head# 2024-04-23T16:43:52.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c901f8a2:::benchmark_data_smithi188_141483_object76088:head# 2024-04-23T16:43:52.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c903fb6b:::benchmark_data_smithi188_141483_object1405:head# 2024-04-23T16:43:52.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.807 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9048ae5:::benchmark_data_smithi188_141483_object16141:head# 2024-04-23T16:43:52.807 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c90527e0:::benchmark_data_smithi188_141483_object62959:head# 2024-04-23T16:43:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9058bc4:::benchmark_data_smithi188_141483_object33025:head# 2024-04-23T16:43:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c905fb3a:::benchmark_data_smithi188_141483_object42293:head# 2024-04-23T16:43:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c906d4cd:::benchmark_data_smithi188_141483_object1216:head# 2024-04-23T16:43:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c906d6d2:::benchmark_data_smithi188_141483_object46849:head# 2024-04-23T16:43:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c907c3f6:::benchmark_data_smithi188_141483_object84614:head# 2024-04-23T16:43:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c907ff77:::benchmark_data_smithi188_141483_object72773:head# 2024-04-23T16:43:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9083d3c:::benchmark_data_smithi188_141483_object37616:head# 2024-04-23T16:43:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c908a946:::benchmark_data_smithi188_141483_object28314:head# 2024-04-23T16:43:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c908f480:::benchmark_data_smithi188_141483_object24241:head# 2024-04-23T16:43:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9094d04:::benchmark_data_smithi188_141483_object47526:head# 2024-04-23T16:43:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9095759:::benchmark_data_smithi188_141483_object66085:head# 2024-04-23T16:43:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9095f44:::benchmark_data_smithi188_141483_object32069:head# 2024-04-23T16:43:52.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c90a3a60:::benchmark_data_smithi188_141483_object23499:head# 2024-04-23T16:43:52.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c90ae359:::benchmark_data_smithi188_141483_object50786:head# 2024-04-23T16:43:52.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c90fd00d:::benchmark_data_smithi188_141483_object81537:head# 2024-04-23T16:43:52.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c910c9d6:::benchmark_data_smithi188_141483_object48995:head# 2024-04-23T16:43:52.810 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.810 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c910df08:::benchmark_data_smithi188_141483_object73605:head# 2024-04-23T16:43:52.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9121318:::benchmark_data_smithi188_141483_object73344:head# 2024-04-23T16:43:52.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c912e5c6:::benchmark_data_smithi188_141483_object36566:head# 2024-04-23T16:43:52.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9140f4b:::benchmark_data_smithi188_141483_object47708:head# 2024-04-23T16:43:52.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c91454b7:::benchmark_data_smithi188_141483_object43271:head# 2024-04-23T16:43:52.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c915bb83:::benchmark_data_smithi188_141483_object47132:head# 2024-04-23T16:43:52.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.811 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c916f0b9:::benchmark_data_smithi188_141483_object41289:head# 2024-04-23T16:43:52.811 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c917a96e:::benchmark_data_smithi188_141483_object90332:head# 2024-04-23T16:43:52.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c917c75b:::benchmark_data_smithi188_141483_object54490:head# 2024-04-23T16:43:52.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9184509:::benchmark_data_smithi188_141483_object68260:head# 2024-04-23T16:43:52.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9186f5c:::benchmark_data_smithi188_141483_object57917:head# 2024-04-23T16:43:52.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.812 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c91b5218:::benchmark_data_smithi188_141483_object50303:head# 2024-04-23T16:43:52.812 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c91c68ce:::benchmark_data_smithi188_141483_object60768:head# 2024-04-23T16:43:52.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c91c8c37:::benchmark_data_smithi188_141483_object77561:head# 2024-04-23T16:43:52.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c91d235c:::benchmark_data_smithi188_141483_object30941:head# 2024-04-23T16:43:52.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c91da6da:::benchmark_data_smithi188_141483_object82451:head# 2024-04-23T16:43:52.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c91db5eb:::benchmark_data_smithi188_141483_object11312:head# 2024-04-23T16:43:52.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.813 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c91e7aa2:::benchmark_data_smithi188_141483_object66876:head# 2024-04-23T16:43:52.813 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c91ebda9:::benchmark_data_smithi188_141483_object56326:head# 2024-04-23T16:43:52.814 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c91fd725:::benchmark_data_smithi188_141483_object83252:head# 2024-04-23T16:43:52.814 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c92075c3:::benchmark_data_smithi188_141483_object75412:head# 2024-04-23T16:43:52.814 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c922567e:::benchmark_data_smithi188_141483_object65741:head# 2024-04-23T16:43:52.814 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9225a6b:::benchmark_data_smithi188_141483_object32532:head# 2024-04-23T16:43:52.814 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9228d56:::benchmark_data_smithi188_141483_object81666:head# 2024-04-23T16:43:52.814 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.814 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9248dcf:::benchmark_data_smithi188_141483_object14153:head# 2024-04-23T16:43:52.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c92493fa:::benchmark_data_smithi188_141483_object62238:head# 2024-04-23T16:43:52.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9249fe3:::benchmark_data_smithi188_141483_object73508:head# 2024-04-23T16:43:52.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c924ba27:::benchmark_data_smithi188_141483_object26315:head# 2024-04-23T16:43:52.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c925dae7:::benchmark_data_smithi188_141483_object13142:head# 2024-04-23T16:43:52.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.815 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9263951:::benchmark_data_smithi188_141483_object40129:head# 2024-04-23T16:43:52.815 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.816 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9271b7e:::benchmark_data_smithi188_141483_object28653:head# 2024-04-23T16:43:52.816 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.816 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c927233f:::benchmark_data_smithi188_141483_object72477:head# 2024-04-23T16:43:52.816 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.816 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9281537:::benchmark_data_smithi188_141483_object60526:head# 2024-04-23T16:43:52.816 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.816 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9286f19:::benchmark_data_smithi188_141483_object20608:head# 2024-04-23T16:43:52.816 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.816 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c928803f:::benchmark_data_smithi188_141483_object88917:head# 2024-04-23T16:43:52.816 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.816 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9289201:::benchmark_data_smithi188_141483_object68248:head# 2024-04-23T16:43:52.816 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.817 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c92aa7b5:::benchmark_data_smithi188_141483_object64928:head# 2024-04-23T16:43:52.817 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.817 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c92b00b2:::benchmark_data_smithi188_141483_object6012:head# 2024-04-23T16:43:52.817 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.817 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c92b0f69:::benchmark_data_smithi188_141483_object64618:head# 2024-04-23T16:43:52.817 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.817 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c92b8a67:::benchmark_data_smithi188_141483_object10515:head# 2024-04-23T16:43:52.817 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.817 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c92ca342:::benchmark_data_smithi188_141483_object78010:head# 2024-04-23T16:43:52.817 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.817 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c92cb73b:::benchmark_data_smithi188_141483_object54883:head# 2024-04-23T16:43:52.817 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.817 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c92d58a2:::benchmark_data_smithi188_141483_object74871:head# 2024-04-23T16:43:52.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c92d7d23:::benchmark_data_smithi188_141483_object75300:head# 2024-04-23T16:43:52.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c92da16c:::benchmark_data_smithi188_141483_object69364:head# 2024-04-23T16:43:52.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c92e6284:::benchmark_data_smithi188_141483_object83017:head# 2024-04-23T16:43:52.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c92ee629:::benchmark_data_smithi188_141483_object77811:head# 2024-04-23T16:43:52.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c92fca43:::benchmark_data_smithi188_141483_object83617:head# 2024-04-23T16:43:52.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.818 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c92ffcf1:::benchmark_data_smithi188_141483_object73125:head# 2024-04-23T16:43:52.818 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.819 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c930428f:::benchmark_data_smithi188_141483_object49063:head# 2024-04-23T16:43:52.819 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.819 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c930475b:::benchmark_data_smithi188_141483_object80368:head# 2024-04-23T16:43:52.819 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.819 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9309c20:::benchmark_data_smithi188_141483_object25450:head# 2024-04-23T16:43:52.819 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.819 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c932c303:::benchmark_data_smithi188_141483_object61261:head# 2024-04-23T16:43:52.819 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.819 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c933db67:::benchmark_data_smithi188_141483_object79137:head# 2024-04-23T16:43:52.819 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.819 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c933e265:::benchmark_data_smithi188_141483_object21946:head# 2024-04-23T16:43:52.819 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c933e265:::benchmark_data_smithi188_141483_object27861:head# 2024-04-23T16:43:52.820 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c934634b:::benchmark_data_smithi188_141483_object87253:head# 2024-04-23T16:43:52.820 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c934d465:::benchmark_data_smithi188_141483_object46293:head# 2024-04-23T16:43:52.820 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9367654:::benchmark_data_smithi188_141483_object17488:head# 2024-04-23T16:43:52.820 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9373c92:::benchmark_data_smithi188_141483_object12462:head# 2024-04-23T16:43:52.820 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c93843c1:::benchmark_data_smithi188_141483_object3945:head# 2024-04-23T16:43:52.820 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.820 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c938b06a:::benchmark_data_smithi188_141483_object59245:head# 2024-04-23T16:43:52.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c938c240:::benchmark_data_smithi188_141483_object43243:head# 2024-04-23T16:43:52.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c939adf1:::benchmark_data_smithi188_141483_object54892:head# 2024-04-23T16:43:52.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c939fe6b:::benchmark_data_smithi188_141483_object22097:head# 2024-04-23T16:43:52.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c93a0b9e:::benchmark_data_smithi188_141483_object41570:head# 2024-04-23T16:43:52.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c93ace50:::benchmark_data_smithi188_141483_object60665:head# 2024-04-23T16:43:52.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.821 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c93c636a:::benchmark_data_smithi188_141483_object54296:head# 2024-04-23T16:43:52.821 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c93d0376:::benchmark_data_smithi188_141483_object49293:head# 2024-04-23T16:43:52.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c93de28e:::benchmark_data_smithi188_141483_object76079:head# 2024-04-23T16:43:52.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c93e7718:::benchmark_data_smithi188_141483_object63051:head# 2024-04-23T16:43:52.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c93f6471:::benchmark_data_smithi188_141483_object87771:head# 2024-04-23T16:43:52.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9412151:::benchmark_data_smithi188_141483_object40485:head# 2024-04-23T16:43:52.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9416649:::benchmark_data_smithi188_141483_object17684:head# 2024-04-23T16:43:52.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.822 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c94225fe:::benchmark_data_smithi188_141483_object76333:head# 2024-04-23T16:43:52.822 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9422c68:::benchmark_data_smithi188_141483_object72112:head# 2024-04-23T16:43:52.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9429dc1:::benchmark_data_smithi188_141483_object8331:head# 2024-04-23T16:43:52.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c943763f:::benchmark_data_smithi188_141483_object3818:head# 2024-04-23T16:43:52.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c944691a:::benchmark_data_smithi188_141483_object54790:head# 2024-04-23T16:43:52.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c944f1ed:::benchmark_data_smithi188_141483_object82215:head# 2024-04-23T16:43:52.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c944f5e1:::benchmark_data_smithi188_141483_object35698:head# 2024-04-23T16:43:52.823 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.823 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c94556c7:::benchmark_data_smithi188_141483_object88260:head# 2024-04-23T16:43:52.824 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c94585ad:::benchmark_data_smithi188_141483_object74052:head# 2024-04-23T16:43:52.824 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c945aa0a:::benchmark_data_smithi188_141483_object21943:head# 2024-04-23T16:43:52.824 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c945d072:::benchmark_data_smithi188_141483_object89717:head# 2024-04-23T16:43:52.824 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c945e4f3:::benchmark_data_smithi188_141483_object76301:head# 2024-04-23T16:43:52.824 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c946c5c7:::benchmark_data_smithi188_141483_object14756:head# 2024-04-23T16:43:52.824 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c947755a:::benchmark_data_smithi188_141483_object6857:head# 2024-04-23T16:43:52.824 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.824 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9479023:::benchmark_data_smithi188_141483_object55612:head# 2024-04-23T16:43:52.825 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.825 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c947e7c4:::benchmark_data_smithi188_141483_object77394:head# 2024-04-23T16:43:52.825 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.825 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9494c83:::benchmark_data_smithi188_141483_object61189:head# 2024-04-23T16:43:52.825 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.825 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c94b39a3:::benchmark_data_smithi188_141483_object11626:head# 2024-04-23T16:43:52.825 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.825 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c94bb81b:::benchmark_data_smithi188_141483_object57813:head# 2024-04-23T16:43:52.825 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.825 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c94bbb9c:::benchmark_data_smithi188_141483_object7339:head# 2024-04-23T16:43:52.825 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.825 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c94cc07d:::benchmark_data_smithi188_141483_object14429:head# 2024-04-23T16:43:52.825 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c94d6bc2:::benchmark_data_smithi188_141483_object80795:head# 2024-04-23T16:43:52.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c94e94f3:::benchmark_data_smithi188_141483_object19509:head# 2024-04-23T16:43:52.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c94ed035:::benchmark_data_smithi188_141483_object11973:head# 2024-04-23T16:43:52.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c94ee9da:::benchmark_data_smithi188_141483_object53647:head# 2024-04-23T16:43:52.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c94fc22b:::benchmark_data_smithi188_141483_object13451:head# 2024-04-23T16:43:52.826 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.826 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c95121da:::benchmark_data_smithi188_141483_object53356:head# 2024-04-23T16:43:52.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c953eedb:::benchmark_data_smithi188_141483_object9886:head# 2024-04-23T16:43:52.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c954e413:::benchmark_data_smithi188_141483_object66270:head# 2024-04-23T16:43:52.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c955bcad:::benchmark_data_smithi188_141483_object43189:head# 2024-04-23T16:43:52.827 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.827 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c955ff29:::benchmark_data_smithi188_141483_object29137:head# 2024-04-23T16:43:52.828 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.828 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9560ad1:::benchmark_data_smithi188_141483_object27431:head# 2024-04-23T16:43:52.828 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.828 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9572cab:::benchmark_data_smithi188_141483_object87611:head# 2024-04-23T16:43:52.828 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.828 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c958480c:::benchmark_data_smithi188_141483_object8330:head# 2024-04-23T16:43:52.828 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.828 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c959f61c:::benchmark_data_smithi188_141483_object72850:head# 2024-04-23T16:43:52.828 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.828 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c95a69ac:::benchmark_data_smithi188_141483_object72601:head# 2024-04-23T16:43:52.828 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c95c4c59:::benchmark_data_smithi188_141483_object7368:head# 2024-04-23T16:43:52.829 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c95c953a:::benchmark_data_smithi188_141483_object4518:head# 2024-04-23T16:43:52.829 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c95ca063:::benchmark_data_smithi188_141483_object66043:head# 2024-04-23T16:43:52.829 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c95ccf76:::benchmark_data_smithi188_141483_object33821:head# 2024-04-23T16:43:52.829 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c95cd70a:::benchmark_data_smithi188_141483_object80168:head# 2024-04-23T16:43:52.829 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.829 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c95dcfc1:::benchmark_data_smithi188_141483_object75154:head# 2024-04-23T16:43:52.829 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.830 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c95e9d94:::benchmark_data_smithi188_141483_object87704:head# 2024-04-23T16:43:52.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.830 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c95fd091:::benchmark_data_smithi188_141483_object89258:head# 2024-04-23T16:43:52.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.830 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c960c1eb:::benchmark_data_smithi188_141483_object47545:head# 2024-04-23T16:43:52.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.830 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9614c9f:::benchmark_data_smithi188_141483_object5182:head# 2024-04-23T16:43:52.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.830 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9632580:::benchmark_data_smithi188_141483_object31166:head# 2024-04-23T16:43:52.830 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.830 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c963902e:::benchmark_data_smithi188_141483_object45002:head# 2024-04-23T16:43:52.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9642d1e:::benchmark_data_smithi188_141483_object24682:head# 2024-04-23T16:43:52.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c96584c2:::benchmark_data_smithi188_141483_object50344:head# 2024-04-23T16:43:52.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9659acc:::benchmark_data_smithi188_141483_object65353:head# 2024-04-23T16:43:52.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c966a873:::benchmark_data_smithi188_141483_object68209:head# 2024-04-23T16:43:52.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c966b5ee:::benchmark_data_smithi188_141483_object6495:head# 2024-04-23T16:43:52.831 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.831 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c966bd80:::benchmark_data_smithi188_141483_object25750:head# 2024-04-23T16:43:52.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c96728e4:::benchmark_data_smithi188_141483_object80960:head# 2024-04-23T16:43:52.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9674ad3:::benchmark_data_smithi188_141483_object28760:head# 2024-04-23T16:43:52.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c968393d:::benchmark_data_smithi188_141483_object43683:head# 2024-04-23T16:43:52.832 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.832 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9685a03:::benchmark_data_smithi188_141483_object11623:head# 2024-04-23T16:43:52.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.833 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c968d078:::benchmark_data_smithi188_141483_object24544:head# 2024-04-23T16:43:52.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.833 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c96900cf:::benchmark_data_smithi188_141483_object3748:head# 2024-04-23T16:43:52.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.833 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9699a13:::benchmark_data_smithi188_141483_object42776:head# 2024-04-23T16:43:52.833 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.834 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c969f7f5:::benchmark_data_smithi188_141483_object21041:head# 2024-04-23T16:43:52.834 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.834 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c96a794f:::benchmark_data_smithi188_141483_object86849:head# 2024-04-23T16:43:52.834 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.834 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c96bf966:::benchmark_data_smithi188_141483_object2406:head# 2024-04-23T16:43:52.834 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.835 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c96cd061:::benchmark_data_smithi188_141483_object45424:head# 2024-04-23T16:43:52.835 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.835 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c96d839e:::benchmark_data_smithi188_141483_object29404:head# 2024-04-23T16:43:52.835 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.835 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c96dcfb0:::benchmark_data_smithi188_141483_object38236:head# 2024-04-23T16:43:52.835 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.835 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c96e1616:::benchmark_data_smithi188_141483_object71886:head# 2024-04-23T16:43:52.836 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.836 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c96e9951:::benchmark_data_smithi188_141483_object28201:head# 2024-04-23T16:43:52.836 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.836 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c96ecbba:::benchmark_data_smithi188_141483_object55838:head# 2024-04-23T16:43:52.836 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.836 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c96efa90:::benchmark_data_smithi188_141483_object20303:head# 2024-04-23T16:43:52.837 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.837 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c96f1e98:::benchmark_data_smithi188_141483_object39593:head# 2024-04-23T16:43:52.837 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.837 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c96f5341:::benchmark_data_smithi188_141483_object55359:head# 2024-04-23T16:43:52.837 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.837 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9701105:::benchmark_data_smithi188_141483_object68376:head# 2024-04-23T16:43:52.838 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.838 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c97109e0:::benchmark_data_smithi188_141483_object13941:head# 2024-04-23T16:43:52.838 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.838 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c971c70a:::benchmark_data_smithi188_141483_object27259:head# 2024-04-23T16:43:52.838 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.838 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c97204f4:::benchmark_data_smithi188_141483_object6827:head# 2024-04-23T16:43:52.838 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.839 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c973fd3f:::benchmark_data_smithi188_141483_object8216:head# 2024-04-23T16:43:52.839 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.839 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c973fd7f:::benchmark_data_smithi188_141483_object79445:head# 2024-04-23T16:43:52.839 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.839 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c974a700:::benchmark_data_smithi188_141483_object62078:head# 2024-04-23T16:43:52.839 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.839 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c974e437:::benchmark_data_smithi188_141483_object90205:head# 2024-04-23T16:43:52.839 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.840 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9758c29:::benchmark_data_smithi188_141483_object9813:head# 2024-04-23T16:43:52.840 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.840 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c977717f:::benchmark_data_smithi188_141483_object33895:head# 2024-04-23T16:43:52.840 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.840 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c978c6b5:::benchmark_data_smithi188_141483_object8323:head# 2024-04-23T16:43:52.840 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.840 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c979eb93:::benchmark_data_smithi188_141483_object992:head# 2024-04-23T16:43:52.841 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.841 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c97a9f5b:::benchmark_data_smithi188_141483_object38752:head# 2024-04-23T16:43:52.841 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.841 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c97b4e6b:::benchmark_data_smithi188_141483_object5965:head# 2024-04-23T16:43:52.841 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.842 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c97be262:::benchmark_data_smithi188_141483_object83317:head# 2024-04-23T16:43:52.842 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.842 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c97c074d:::benchmark_data_smithi188_141483_object34112:head# 2024-04-23T16:43:52.842 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.842 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c97c0ca9:::benchmark_data_smithi188_141483_object77862:head# 2024-04-23T16:43:52.842 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.842 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c97c2286:::benchmark_data_smithi188_141483_object58629:head# 2024-04-23T16:43:52.842 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.842 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c97c78f6:::benchmark_data_smithi188_141483_object69232:head# 2024-04-23T16:43:52.843 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.843 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c97dacd9:::benchmark_data_smithi188_141483_object66869:head# 2024-04-23T16:43:52.843 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.843 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c97e88dd:::benchmark_data_smithi188_141483_object64856:head# 2024-04-23T16:43:52.843 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.843 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c97f4deb:::benchmark_data_smithi188_141483_object29212:head# 2024-04-23T16:43:52.843 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.843 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c97fa253:::benchmark_data_smithi188_141483_object25913:head# 2024-04-23T16:43:52.843 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.843 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c980298b:::benchmark_data_smithi188_141483_object80902:head# 2024-04-23T16:43:52.843 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.843 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9811777:::benchmark_data_smithi188_141483_object73652:head# 2024-04-23T16:43:52.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9828771:::benchmark_data_smithi188_141483_object18827:head# 2024-04-23T16:43:52.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9829932:::benchmark_data_smithi188_141483_object60774:head# 2024-04-23T16:43:52.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c983aa67:::benchmark_data_smithi188_141483_object67162:head# 2024-04-23T16:43:52.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9847523:::benchmark_data_smithi188_141483_object3578:head# 2024-04-23T16:43:52.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c98475e9:::benchmark_data_smithi188_141483_object29896:head# 2024-04-23T16:43:52.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c985f0de:::benchmark_data_smithi188_141483_object40345:head# 2024-04-23T16:43:52.844 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.844 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9874689:::benchmark_data_smithi188_141483_object21672:head# 2024-04-23T16:43:52.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9875464:::benchmark_data_smithi188_141483_object40599:head# 2024-04-23T16:43:52.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c987fd60:::benchmark_data_smithi188_141483_object66285:head# 2024-04-23T16:43:52.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c989e5f9:::benchmark_data_smithi188_141483_object80261:head# 2024-04-23T16:43:52.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c98a2ce5:::benchmark_data_smithi188_141483_object72826:head# 2024-04-23T16:43:52.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c98aab63:::benchmark_data_smithi188_141483_object9280:head# 2024-04-23T16:43:52.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.845 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c98c5f35:::benchmark_data_smithi188_141483_object33676:head# 2024-04-23T16:43:52.845 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c98c9328:::benchmark_data_smithi188_141483_object45505:head# 2024-04-23T16:43:52.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c98cf09d:::benchmark_data_smithi188_141483_object21630:head# 2024-04-23T16:43:52.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c98d8299:::benchmark_data_smithi188_141483_object48732:head# 2024-04-23T16:43:52.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c98e77fc:::benchmark_data_smithi188_141483_object7819:head# 2024-04-23T16:43:52.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c990c715:::benchmark_data_smithi188_141483_object69773:head# 2024-04-23T16:43:52.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c99133ad:::benchmark_data_smithi188_141483_object36615:head# 2024-04-23T16:43:52.846 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.846 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9935e92:::benchmark_data_smithi188_141483_object83089:head# 2024-04-23T16:43:52.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9945382:::benchmark_data_smithi188_141483_object73287:head# 2024-04-23T16:43:52.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9949211:::benchmark_data_smithi188_141483_object7518:head# 2024-04-23T16:43:52.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c994992c:::benchmark_data_smithi188_141483_object58751:head# 2024-04-23T16:43:52.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c994c3f1:::benchmark_data_smithi188_141483_object66281:head# 2024-04-23T16:43:52.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c995e1cf:::benchmark_data_smithi188_141483_object75930:head# 2024-04-23T16:43:52.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.847 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9983bf6:::benchmark_data_smithi188_141483_object71998:head# 2024-04-23T16:43:52.847 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c998f8bb:::benchmark_data_smithi188_141483_object5999:head# 2024-04-23T16:43:52.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9992910:::benchmark_data_smithi188_141483_object52534:head# 2024-04-23T16:43:52.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c99a0e0c:::benchmark_data_smithi188_141483_object64130:head# 2024-04-23T16:43:52.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c99a80e4:::benchmark_data_smithi188_141483_object50560:head# 2024-04-23T16:43:52.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c99b338d:::benchmark_data_smithi188_141483_object58790:head# 2024-04-23T16:43:52.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.848 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c99d1c8d:::benchmark_data_smithi188_141483_object10065:head# 2024-04-23T16:43:52.848 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c99d2a9d:::benchmark_data_smithi188_141483_object54855:head# 2024-04-23T16:43:52.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c99dea22:::benchmark_data_smithi188_141483_object21909:head# 2024-04-23T16:43:52.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c99e1b55:::benchmark_data_smithi188_141483_object51123:head# 2024-04-23T16:43:52.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c99fe3c7:::benchmark_data_smithi188_141483_object40946:head# 2024-04-23T16:43:52.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9a00fb4:::benchmark_data_smithi188_141483_object20723:head# 2024-04-23T16:43:52.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9a06c2d:::benchmark_data_smithi188_141483_object36685:head# 2024-04-23T16:43:52.849 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.849 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9a0e518:::benchmark_data_smithi188_141483_object30082:head# 2024-04-23T16:43:52.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9a10dad:::benchmark_data_smithi188_141483_object47801:head# 2024-04-23T16:43:52.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9a22c3b:::benchmark_data_smithi188_141483_object56117:head# 2024-04-23T16:43:52.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9a3ed1b:::benchmark_data_smithi188_141483_object47299:head# 2024-04-23T16:43:52.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9a4f431:::benchmark_data_smithi188_141483_object43549:head# 2024-04-23T16:43:52.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9a5e2a4:::benchmark_data_smithi188_141483_object75536:head# 2024-04-23T16:43:52.850 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.850 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9a64628:::benchmark_data_smithi188_141483_object77700:head# 2024-04-23T16:43:52.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9a78d33:::benchmark_data_smithi188_141483_object21307:head# 2024-04-23T16:43:52.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9a8130d:::benchmark_data_smithi188_141483_object58836:head# 2024-04-23T16:43:52.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9a8cce5:::benchmark_data_smithi188_141483_object78210:head# 2024-04-23T16:43:52.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9a9a0ba:::benchmark_data_smithi188_141483_object69304:head# 2024-04-23T16:43:52.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9aa863a:::benchmark_data_smithi188_141483_object55065:head# 2024-04-23T16:43:52.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.851 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9ab9743:::benchmark_data_smithi188_141483_object20445:head# 2024-04-23T16:43:52.851 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9abd6c7:::benchmark_data_smithi188_141483_object11889:head# 2024-04-23T16:43:52.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9ac7145:::benchmark_data_smithi188_141483_object65115:head# 2024-04-23T16:43:52.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9acb2db:::benchmark_data_smithi188_141483_object48145:head# 2024-04-23T16:43:52.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9ad09a6:::benchmark_data_smithi188_141483_object40860:head# 2024-04-23T16:43:52.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9ad0e32:::benchmark_data_smithi188_141483_object167:head# 2024-04-23T16:43:52.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9adf079:::benchmark_data_smithi188_141483_object32395:head# 2024-04-23T16:43:52.852 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.852 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9ae0d50:::benchmark_data_smithi188_141483_object88622:head# 2024-04-23T16:43:52.853 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.853 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9ae76e5:::benchmark_data_smithi188_141483_object60705:head# 2024-04-23T16:43:52.853 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.853 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9af6664:::benchmark_data_smithi188_141483_object32278:head# 2024-04-23T16:43:52.853 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.853 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9b03970:::benchmark_data_smithi188_141483_object71834:head# 2024-04-23T16:43:52.853 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.853 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9b0d2da:::benchmark_data_smithi188_141483_object80940:head# 2024-04-23T16:43:52.853 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.853 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9b42a1a:::benchmark_data_smithi188_141483_object58618:head# 2024-04-23T16:43:52.853 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.853 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9b49a55:::benchmark_data_smithi188_141483_object51135:head# 2024-04-23T16:43:52.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9b4d75b:::benchmark_data_smithi188_141483_object84806:head# 2024-04-23T16:43:52.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9b53900:::benchmark_data_smithi188_141483_object3642:head# 2024-04-23T16:43:52.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9b7275a:::benchmark_data_smithi188_141483_object56873:head# 2024-04-23T16:43:52.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9b73164:::benchmark_data_smithi188_141483_object47305:head# 2024-04-23T16:43:52.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9b923be:::benchmark_data_smithi188_141483_object64893:head# 2024-04-23T16:43:52.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.854 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9b96c27:::benchmark_data_smithi188_141483_object39976:head# 2024-04-23T16:43:52.854 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9ba5e3e:::benchmark_data_smithi188_141483_object63174:head# 2024-04-23T16:43:52.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9bb9ae2:::benchmark_data_smithi188_141483_object18119:head# 2024-04-23T16:43:52.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9bba1f8:::benchmark_data_smithi188_141483_object45676:head# 2024-04-23T16:43:52.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9bbd631:::benchmark_data_smithi188_141483_object52718:head# 2024-04-23T16:43:52.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9bcc5e4:::benchmark_data_smithi188_141483_object7255:head# 2024-04-23T16:43:52.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9bd9063:::benchmark_data_smithi188_141483_object48691:head# 2024-04-23T16:43:52.855 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.855 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9bf3c65:::benchmark_data_smithi188_141483_object30305:head# 2024-04-23T16:43:52.856 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.856 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9bf9db1:::benchmark_data_smithi188_141483_object78016:head# 2024-04-23T16:43:52.856 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.856 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9c05948:::benchmark_data_smithi188_141483_object478:head# 2024-04-23T16:43:52.856 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.856 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9c54278:::benchmark_data_smithi188_141483_object10213:head# 2024-04-23T16:43:52.856 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.856 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9c5f44e:::benchmark_data_smithi188_141483_object49769:head# 2024-04-23T16:43:52.856 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.856 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9c613f2:::benchmark_data_smithi188_141483_object20366:head# 2024-04-23T16:43:52.856 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.856 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9c75f8a:::benchmark_data_smithi188_141483_object31497:head# 2024-04-23T16:43:52.856 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.856 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9c764b6:::benchmark_data_smithi188_141483_object57877:head# 2024-04-23T16:43:52.857 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.857 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9c8586e:::benchmark_data_smithi188_141483_object48603:head# 2024-04-23T16:43:52.857 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.857 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9c89e0e:::benchmark_data_smithi188_141483_object37453:head# 2024-04-23T16:43:52.857 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.857 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9c9591b:::benchmark_data_smithi188_141483_object79442:head# 2024-04-23T16:43:52.857 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.857 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9c9e3bd:::benchmark_data_smithi188_141483_object1491:head# 2024-04-23T16:43:52.857 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.857 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9ca5905:::benchmark_data_smithi188_141483_object38181:head# 2024-04-23T16:43:52.857 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.857 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9caeea0:::benchmark_data_smithi188_141483_object65780:head# 2024-04-23T16:43:52.857 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.858 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9cc5ff0:::benchmark_data_smithi188_141483_object66261:head# 2024-04-23T16:43:52.858 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.858 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9ccbc1d:::benchmark_data_smithi188_141483_object80009:head# 2024-04-23T16:43:52.858 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.858 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9cda5f5:::benchmark_data_smithi188_141483_object7186:head# 2024-04-23T16:43:52.858 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.858 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9cdb9d1:::benchmark_data_smithi188_141483_object32531:head# 2024-04-23T16:43:52.858 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.858 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9cdfcdb:::benchmark_data_smithi188_141483_object58549:head# 2024-04-23T16:43:52.858 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.858 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9ce49d3:::benchmark_data_smithi188_141483_object2786:head# 2024-04-23T16:43:52.858 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.858 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9d2b07d:::benchmark_data_smithi188_141483_object81280:head# 2024-04-23T16:43:52.859 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.859 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9d31ecf:::benchmark_data_smithi188_141483_object29479:head# 2024-04-23T16:43:52.859 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.859 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9d441e7:::benchmark_data_smithi188_141483_object31194:head# 2024-04-23T16:43:52.859 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.859 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9d47d05:::benchmark_data_smithi188_141483_object36241:head# 2024-04-23T16:43:52.859 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.859 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9d706b1:::benchmark_data_smithi188_141483_object7439:head# 2024-04-23T16:43:52.859 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.859 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9d7df8a:::benchmark_data_smithi188_141483_object85398:head# 2024-04-23T16:43:52.859 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.859 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9d85851:::benchmark_data_smithi188_141483_object81152:head# 2024-04-23T16:43:52.859 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.859 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9d90c3b:::benchmark_data_smithi188_141483_object57686:head# 2024-04-23T16:43:52.860 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.860 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9d9351b:::benchmark_data_smithi188_141483_object38041:head# 2024-04-23T16:43:52.860 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.860 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9d9d23f:::benchmark_data_smithi188_141483_object105:head# 2024-04-23T16:43:52.860 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.860 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9d9f765:::benchmark_data_smithi188_141483_object26343:head# 2024-04-23T16:43:52.860 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.860 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9da04f3:::benchmark_data_smithi188_141483_object23931:head# 2024-04-23T16:43:52.860 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.860 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9daf654:::benchmark_data_smithi188_141483_object42309:head# 2024-04-23T16:43:52.860 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.860 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9dbe89a:::benchmark_data_smithi188_141483_object61314:head# 2024-04-23T16:43:52.860 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.860 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9dc8884:::benchmark_data_smithi188_141483_object62159:head# 2024-04-23T16:43:52.861 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.861 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9dc9238:::benchmark_data_smithi188_141483_object21705:head# 2024-04-23T16:43:52.861 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.861 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9de6873:::benchmark_data_smithi188_141483_object87816:head# 2024-04-23T16:43:52.861 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.861 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9dee16f:::benchmark_data_smithi188_141483_object7682:head# 2024-04-23T16:43:52.861 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.861 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9dfb165:::benchmark_data_smithi188_141483_object45033:head# 2024-04-23T16:43:52.861 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.861 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9dfe441:::benchmark_data_smithi188_141483_object16363:head# 2024-04-23T16:43:52.861 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.861 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9e0981a:::benchmark_data_smithi188_141483_object71263:head# 2024-04-23T16:43:52.861 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9e113dc:::benchmark_data_smithi188_141483_object13895:head# 2024-04-23T16:43:52.862 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9e1786a:::benchmark_data_smithi188_141483_object66705:head# 2024-04-23T16:43:52.862 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9e206ae:::benchmark_data_smithi188_141483_object2059:head# 2024-04-23T16:43:52.862 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9e2f151:::benchmark_data_smithi188_141483_object89597:head# 2024-04-23T16:43:52.862 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9e3431a:::benchmark_data_smithi188_141483_object42009:head# 2024-04-23T16:43:52.862 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9e496ba:::benchmark_data_smithi188_141483_object13565:head# 2024-04-23T16:43:52.862 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.862 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9e4cfd3:::benchmark_data_smithi188_141483_object81389:head# 2024-04-23T16:43:52.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9e569cc:::benchmark_data_smithi188_141483_object80410:head# 2024-04-23T16:43:52.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9e6105b:::benchmark_data_smithi188_141483_object50520:head# 2024-04-23T16:43:52.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9e6aad0:::benchmark_data_smithi188_141483_object59937:head# 2024-04-23T16:43:52.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9e727a5:::benchmark_data_smithi188_141483_object38099:head# 2024-04-23T16:43:52.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9e770ad:::benchmark_data_smithi188_141483_object9077:head# 2024-04-23T16:43:52.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9e8c0a6:::benchmark_data_smithi188_141483_object83290:head# 2024-04-23T16:43:52.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.863 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9e94194:::benchmark_data_smithi188_141483_object34342:head# 2024-04-23T16:43:52.863 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9e9bdf0:::benchmark_data_smithi188_141483_object4023:head# 2024-04-23T16:43:52.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9e9cb41:::benchmark_data_smithi188_141483_object33563:head# 2024-04-23T16:43:52.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9ebbeab:::benchmark_data_smithi188_141483_object47124:head# 2024-04-23T16:43:52.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9ec4541:::benchmark_data_smithi188_141483_object86832:head# 2024-04-23T16:43:52.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9ee4c53:::benchmark_data_smithi188_141483_object16160:head# 2024-04-23T16:43:52.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9ef0ffd:::benchmark_data_smithi188_141483_object29565:head# 2024-04-23T16:43:52.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9f0212a:::benchmark_data_smithi188_141483_object63881:head# 2024-04-23T16:43:52.864 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.864 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9f046c6:::benchmark_data_smithi188_141483_object89984:head# 2024-04-23T16:43:52.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9f3bccd:::benchmark_data_smithi188_141483_object49367:head# 2024-04-23T16:43:52.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9f3efa7:::benchmark_data_smithi188_141483_object2590:head# 2024-04-23T16:43:52.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9f49e7a:::benchmark_data_smithi188_141483_object52347:head# 2024-04-23T16:43:52.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9f4ce8b:::benchmark_data_smithi188_141483_object5227:head# 2024-04-23T16:43:52.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9f64511:::benchmark_data_smithi188_141483_object9758:head# 2024-04-23T16:43:52.865 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.865 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9f6874f:::benchmark_data_smithi188_141483_object31980:head# 2024-04-23T16:43:52.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9f6a5bf:::benchmark_data_smithi188_141483_object44738:head# 2024-04-23T16:43:52.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9f7a398:::benchmark_data_smithi188_141483_object24858:head# 2024-04-23T16:43:52.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9f824c3:::benchmark_data_smithi188_141483_object19031:head# 2024-04-23T16:43:52.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9f901cf:::benchmark_data_smithi188_141483_object43913:head# 2024-04-23T16:43:52.866 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.866 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9f9ac8d:::benchmark_data_smithi188_141483_object37968:head# 2024-04-23T16:43:52.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9fa0a91:::benchmark_data_smithi188_141483_object70610:head# 2024-04-23T16:43:52.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9fa642e:::benchmark_data_smithi188_141483_object66335:head# 2024-04-23T16:43:52.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9fc0b2a:::benchmark_data_smithi188_141483_object59183:head# 2024-04-23T16:43:52.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9fc0e09:::benchmark_data_smithi188_141483_object88920:head# 2024-04-23T16:43:52.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9fc6dee:::benchmark_data_smithi188_141483_object44267:head# 2024-04-23T16:43:52.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.867 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9fcf21c:::benchmark_data_smithi188_141483_object30850:head# 2024-04-23T16:43:52.867 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:c9fd67ae:::benchmark_data_smithi188_141483_object78544:head# 2024-04-23T16:43:52.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca00232f:::benchmark_data_smithi188_141483_object58990:head# 2024-04-23T16:43:52.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca0105a9:::benchmark_data_smithi188_141483_object27121:head# 2024-04-23T16:43:52.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca0166c1:::benchmark_data_smithi188_141483_object65498:head# 2024-04-23T16:43:52.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca0205e7:::benchmark_data_smithi188_141483_object12546:head# 2024-04-23T16:43:52.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca02aa7c:::benchmark_data_smithi188_141483_object65049:head# 2024-04-23T16:43:52.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.868 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca0338f2:::benchmark_data_smithi188_141483_object80520:head# 2024-04-23T16:43:52.868 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca03a82c:::benchmark_data_smithi188_141483_object90366:head# 2024-04-23T16:43:52.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca060100:::benchmark_data_smithi188_141483_object25992:head# 2024-04-23T16:43:52.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca065757:::benchmark_data_smithi188_141483_object217:head# 2024-04-23T16:43:52.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca07d0bd:::benchmark_data_smithi188_141483_object34424:head# 2024-04-23T16:43:52.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca081362:::benchmark_data_smithi188_141483_object53019:head# 2024-04-23T16:43:52.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca08398e:::benchmark_data_smithi188_141483_object32179:head# 2024-04-23T16:43:52.869 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.869 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca0a62d6:::benchmark_data_smithi188_141483_object30770:head# 2024-04-23T16:43:52.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca0afa4d:::benchmark_data_smithi188_141483_object30843:head# 2024-04-23T16:43:52.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca0b8907:::benchmark_data_smithi188_141483_object90501:head# 2024-04-23T16:43:52.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca0d4591:::benchmark_data_smithi188_141483_object44030:head# 2024-04-23T16:43:52.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca0d75bc:::benchmark_data_smithi188_141483_object83725:head# 2024-04-23T16:43:52.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca0e833b:::benchmark_data_smithi188_141483_object63272:head# 2024-04-23T16:43:52.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.870 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca0f6e02:::benchmark_data_smithi188_141483_object10948:head# 2024-04-23T16:43:52.870 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca0fc5d8:::benchmark_data_smithi188_141483_object22543:head# 2024-04-23T16:43:52.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca10b778:::benchmark_data_smithi188_141483_object44088:head# 2024-04-23T16:43:52.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca13c6ee:::benchmark_data_smithi188_141483_object80788:head# 2024-04-23T16:43:52.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca14e26c:::benchmark_data_smithi188_141483_object72213:head# 2024-04-23T16:43:52.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca14e367:::benchmark_data_smithi188_141483_object87680:head# 2024-04-23T16:43:52.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca1692f5:::benchmark_data_smithi188_141483_object3606:head# 2024-04-23T16:43:52.871 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.871 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca16fe07:::benchmark_data_smithi188_141483_object41669:head# 2024-04-23T16:43:52.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca175769:::benchmark_data_smithi188_141483_object4942:head# 2024-04-23T16:43:52.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca17e45b:::benchmark_data_smithi188_141483_object37942:head# 2024-04-23T16:43:52.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca184d04:::benchmark_data_smithi188_141483_object9624:head# 2024-04-23T16:43:52.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca19b0d9:::benchmark_data_smithi188_141483_object38016:head# 2024-04-23T16:43:52.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca1aedfb:::benchmark_data_smithi188_141483_object37168:head# 2024-04-23T16:43:52.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca1b90c8:::benchmark_data_smithi188_141483_object80040:head# 2024-04-23T16:43:52.872 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.872 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca1ba3c1:::benchmark_data_smithi188_141483_object6205:head# 2024-04-23T16:43:52.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca1bcb6f:::benchmark_data_smithi188_141483_object78400:head# 2024-04-23T16:43:52.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca1bd19f:::benchmark_data_smithi188_141483_object63007:head# 2024-04-23T16:43:52.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca1c61fb:::benchmark_data_smithi188_141483_object29344:head# 2024-04-23T16:43:52.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca1cc926:::benchmark_data_smithi188_141483_object80855:head# 2024-04-23T16:43:52.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca1d0e13:::benchmark_data_smithi188_141483_object33977:head# 2024-04-23T16:43:52.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.873 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca1d55e9:::benchmark_data_smithi188_141483_object78635:head# 2024-04-23T16:43:52.873 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca1dfb83:::benchmark_data_smithi188_141483_object76294:head# 2024-04-23T16:43:52.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca1eb706:::benchmark_data_smithi188_141483_object33980:head# 2024-04-23T16:43:52.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca1f74da:::benchmark_data_smithi188_141483_object7321:head# 2024-04-23T16:43:52.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca1f8dba:::benchmark_data_smithi188_141483_object88066:head# 2024-04-23T16:43:52.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca1fc133:::benchmark_data_smithi188_141483_object43679:head# 2024-04-23T16:43:52.874 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.874 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca1ff8d9:::benchmark_data_smithi188_141483_object71014:head# 2024-04-23T16:43:52.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca205286:::benchmark_data_smithi188_141483_object32196:head# 2024-04-23T16:43:52.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca241aa6:::benchmark_data_smithi188_141483_object40959:head# 2024-04-23T16:43:52.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca2506cc:::benchmark_data_smithi188_141483_object49864:head# 2024-04-23T16:43:52.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca25a9d8:::benchmark_data_smithi188_141483_object51808:head# 2024-04-23T16:43:52.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca263e1a:::benchmark_data_smithi188_141483_object80444:head# 2024-04-23T16:43:52.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.875 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca268611:::benchmark_data_smithi188_141483_object81881:head# 2024-04-23T16:43:52.875 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca26c2cf:::benchmark_data_smithi188_141483_object68889:head# 2024-04-23T16:43:52.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca26cfad:::benchmark_data_smithi188_141483_object58726:head# 2024-04-23T16:43:52.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca26fae0:::benchmark_data_smithi188_141483_object46230:head# 2024-04-23T16:43:52.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca2702b5:::benchmark_data_smithi188_141483_object27587:head# 2024-04-23T16:43:52.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca27213c:::benchmark_data_smithi188_141483_object25037:head# 2024-04-23T16:43:52.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca276ea1:::benchmark_data_smithi188_141483_object61505:head# 2024-04-23T16:43:52.876 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.876 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca2776fe:::benchmark_data_smithi188_141483_object83901:head# 2024-04-23T16:43:52.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca2934a4:::benchmark_data_smithi188_141483_object89955:head# 2024-04-23T16:43:52.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca29c220:::benchmark_data_smithi188_141483_object79014:head# 2024-04-23T16:43:52.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca2a34bc:::benchmark_data_smithi188_141483_object40920:head# 2024-04-23T16:43:52.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca2a5a74:::benchmark_data_smithi188_141483_object73072:head# 2024-04-23T16:43:52.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca2b991c:::benchmark_data_smithi188_141483_object73661:head# 2024-04-23T16:43:52.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.877 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca2d6917:::benchmark_data_smithi188_141483_object11631:head# 2024-04-23T16:43:52.877 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca2e0d8c:::benchmark_data_smithi188_141483_object36274:head# 2024-04-23T16:43:52.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca2ec2ee:::benchmark_data_smithi188_141483_object25036:head# 2024-04-23T16:43:52.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca2f72a4:::benchmark_data_smithi188_141483_object38320:head# 2024-04-23T16:43:52.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca2feba8:::benchmark_data_smithi188_141483_object83953:head# 2024-04-23T16:43:52.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.878 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca320072:::benchmark_data_smithi188_141483_object67017:head# 2024-04-23T16:43:52.878 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca33ffc2:::benchmark_data_smithi188_141483_object1640:head# 2024-04-23T16:43:52.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca351fdc:::benchmark_data_smithi188_141483_object24935:head# 2024-04-23T16:43:52.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca359fb7:::benchmark_data_smithi188_141483_object89711:head# 2024-04-23T16:43:52.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca36992b:::benchmark_data_smithi188_141483_object57625:head# 2024-04-23T16:43:52.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca384e6e:::benchmark_data_smithi188_141483_object24938:head# 2024-04-23T16:43:52.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.879 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca38532d:::benchmark_data_smithi188_141483_object87683:head# 2024-04-23T16:43:52.879 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca38596f:::benchmark_data_smithi188_141483_object30429:head# 2024-04-23T16:43:52.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca3904e7:::benchmark_data_smithi188_141483_object82117:head# 2024-04-23T16:43:52.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca39e05f:::benchmark_data_smithi188_141483_object62343:head# 2024-04-23T16:43:52.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca3b93db:::benchmark_data_smithi188_141483_object28447:head# 2024-04-23T16:43:52.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca3bf97b:::benchmark_data_smithi188_141483_object65754:head# 2024-04-23T16:43:52.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.880 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca3c0f5d:::benchmark_data_smithi188_141483_object56450:head# 2024-04-23T16:43:52.880 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca3c16d5:::benchmark_data_smithi188_141483_object53266:head# 2024-04-23T16:43:52.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca3cbfd6:::benchmark_data_smithi188_141483_object17504:head# 2024-04-23T16:43:52.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca3d6882:::benchmark_data_smithi188_141483_object39821:head# 2024-04-23T16:43:52.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca3e0ef1:::benchmark_data_smithi188_141483_object1891:head# 2024-04-23T16:43:52.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca3e335d:::benchmark_data_smithi188_141483_object76971:head# 2024-04-23T16:43:52.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca3ef224:::benchmark_data_smithi188_141483_object83242:head# 2024-04-23T16:43:52.881 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.881 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca417aa7:::benchmark_data_smithi188_141483_object70237:head# 2024-04-23T16:43:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca41e1de:::benchmark_data_smithi188_141483_object70009:head# 2024-04-23T16:43:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca42091c:::benchmark_data_smithi188_141483_object58027:head# 2024-04-23T16:43:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca429ba0:::benchmark_data_smithi188_141483_object24771:head# 2024-04-23T16:43:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca42d8da:::benchmark_data_smithi188_141483_object12976:head# 2024-04-23T16:43:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca43e22d:::benchmark_data_smithi188_141483_object75222:head# 2024-04-23T16:43:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca4439ab:::benchmark_data_smithi188_141483_object45124:head# 2024-04-23T16:43:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca448ce4:::benchmark_data_smithi188_141483_object16819:head# 2024-04-23T16:43:52.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca4767f9:::benchmark_data_smithi188_141483_object26180:head# 2024-04-23T16:43:52.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca48fc7b:::benchmark_data_smithi188_141483_object73302:head# 2024-04-23T16:43:52.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca4be2ab:::benchmark_data_smithi188_141483_object72133:head# 2024-04-23T16:43:52.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca4c5fa5:::benchmark_data_smithi188_141483_object89084:head# 2024-04-23T16:43:52.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.883 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca4c885a:::benchmark_data_smithi188_141483_object15461:head# 2024-04-23T16:43:52.883 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca4de8ef:::benchmark_data_smithi188_141483_object9520:head# 2024-04-23T16:43:52.884 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca4e664f:::benchmark_data_smithi188_141483_object24773:head# 2024-04-23T16:43:52.884 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca4effa0:::benchmark_data_smithi188_141483_object36454:head# 2024-04-23T16:43:52.884 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca4f03b0:::benchmark_data_smithi188_141483_object73266:head# 2024-04-23T16:43:52.884 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca4f403a:::benchmark_data_smithi188_141483_object31154:head# 2024-04-23T16:43:52.884 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca5016a2:::benchmark_data_smithi188_141483_object32293:head# 2024-04-23T16:43:52.884 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.884 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca505ef3:::benchmark_data_smithi188_141483_object72268:head# 2024-04-23T16:43:52.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca510f74:::benchmark_data_smithi188_141483_object68104:head# 2024-04-23T16:43:52.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca513363:::benchmark_data_smithi188_141483_object54358:head# 2024-04-23T16:43:52.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca519712:::benchmark_data_smithi188_141483_object25881:head# 2024-04-23T16:43:52.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca51e3e2:::benchmark_data_smithi188_141483_object2984:head# 2024-04-23T16:43:52.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca531f25:::benchmark_data_smithi188_141483_object22729:head# 2024-04-23T16:43:52.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.885 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca533739:::benchmark_data_smithi188_141483_object44122:head# 2024-04-23T16:43:52.885 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca540fa8:::benchmark_data_smithi188_141483_object56567:head# 2024-04-23T16:43:52.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca54a502:::benchmark_data_smithi188_141483_object52906:head# 2024-04-23T16:43:52.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca54aba1:::benchmark_data_smithi188_141483_object47357:head# 2024-04-23T16:43:52.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca555c7c:::benchmark_data_smithi188_141483_object71986:head# 2024-04-23T16:43:52.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca5654e6:::benchmark_data_smithi188_141483_object3737:head# 2024-04-23T16:43:52.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca5679bd:::benchmark_data_smithi188_141483_object79033:head# 2024-04-23T16:43:52.886 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.886 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca57afa7:::benchmark_data_smithi188_141483_object32176:head# 2024-04-23T16:43:52.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca58aa37:::benchmark_data_smithi188_141483_object67514:head# 2024-04-23T16:43:52.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca597179:::benchmark_data_smithi188_141483_object28776:head# 2024-04-23T16:43:52.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca59e5ac:::benchmark_data_smithi188_141483_object89825:head# 2024-04-23T16:43:52.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca5a04bb:::benchmark_data_smithi188_141483_object66178:head# 2024-04-23T16:43:52.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca5b2e81:::benchmark_data_smithi188_141483_object11615:head# 2024-04-23T16:43:52.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.887 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca5ce0d0:::benchmark_data_smithi188_141483_object53067:head# 2024-04-23T16:43:52.887 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca5d339f:::benchmark_data_smithi188_141483_object58762:head# 2024-04-23T16:43:52.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca5f34e6:::benchmark_data_smithi188_141483_object47792:head# 2024-04-23T16:43:52.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca5f77c0:::benchmark_data_smithi188_141483_object49132:head# 2024-04-23T16:43:52.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca60abe6:::benchmark_data_smithi188_141483_object64581:head# 2024-04-23T16:43:52.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca61d094:::benchmark_data_smithi188_141483_object6232:head# 2024-04-23T16:43:52.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca6312ad:::benchmark_data_smithi188_141483_object7230:head# 2024-04-23T16:43:52.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.888 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca634da0:::benchmark_data_smithi188_141483_object78806:head# 2024-04-23T16:43:52.888 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca635917:::benchmark_data_smithi188_141483_object2434:head# 2024-04-23T16:43:52.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca652789:::benchmark_data_smithi188_141483_object89412:head# 2024-04-23T16:43:52.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca65ae09:::benchmark_data_smithi188_141483_object18588:head# 2024-04-23T16:43:52.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca664207:::benchmark_data_smithi188_141483_object9657:head# 2024-04-23T16:43:52.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca66ca59:::benchmark_data_smithi188_141483_object69239:head# 2024-04-23T16:43:52.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca670edd:::benchmark_data_smithi188_141483_object56924:head# 2024-04-23T16:43:52.889 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.889 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca6731d9:::benchmark_data_smithi188_141483_object48611:head# 2024-04-23T16:43:52.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca688f06:::benchmark_data_smithi188_141483_object33792:head# 2024-04-23T16:43:52.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca6944c5:::benchmark_data_smithi188_141483_object8051:head# 2024-04-23T16:43:52.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca694a4e:::benchmark_data_smithi188_141483_object82951:head# 2024-04-23T16:43:52.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca6ab5e7:::benchmark_data_smithi188_141483_object37256:head# 2024-04-23T16:43:52.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca6b83b3:::benchmark_data_smithi188_141483_object7712:head# 2024-04-23T16:43:52.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca6c20cf:::benchmark_data_smithi188_141483_object3700:head# 2024-04-23T16:43:52.890 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.890 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca6ce1d5:::benchmark_data_smithi188_141483_object76825:head# 2024-04-23T16:43:52.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca6d5b58:::benchmark_data_smithi188_141483_object22329:head# 2024-04-23T16:43:52.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca6da80d:::benchmark_data_smithi188_141483_object86290:head# 2024-04-23T16:43:52.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca6df7ab:::benchmark_data_smithi188_141483_object14642:head# 2024-04-23T16:43:52.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca6e4bbb:::benchmark_data_smithi188_141483_object53469:head# 2024-04-23T16:43:52.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca6e6673:::benchmark_data_smithi188_141483_object80252:head# 2024-04-23T16:43:52.891 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.891 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca6eaff7:::benchmark_data_smithi188_141483_object10124:head# 2024-04-23T16:43:52.892 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.892 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca6f5e80:::benchmark_data_smithi188_141483_object72431:head# 2024-04-23T16:43:52.892 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.892 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca6f6528:::benchmark_data_smithi188_141483_object11786:head# 2024-04-23T16:43:52.892 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.892 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca6f8531:::benchmark_data_smithi188_141483_object11234:head# 2024-04-23T16:43:52.892 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.892 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca6fecbb:::benchmark_data_smithi188_141483_object17195:head# 2024-04-23T16:43:52.892 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.892 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca6ff5fe:::benchmark_data_smithi188_141483_object32070:head# 2024-04-23T16:43:52.892 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.892 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca6ffdd6:::benchmark_data_smithi188_141483_object73995:head# 2024-04-23T16:43:52.892 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.892 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca707c14:::benchmark_data_smithi188_141483_object70883:head# 2024-04-23T16:43:52.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.893 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca70ddc3:::benchmark_data_smithi188_141483_object64410:head# 2024-04-23T16:43:52.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.893 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca70fda2:::benchmark_data_smithi188_141483_object86488:head# 2024-04-23T16:43:52.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.893 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca7258e5:::benchmark_data_smithi188_141483_object87343:head# 2024-04-23T16:43:52.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.893 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca72d371:::benchmark_data_smithi188_141483_object9709:head# 2024-04-23T16:43:52.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.893 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca738856:::benchmark_data_smithi188_141483_object41745:head# 2024-04-23T16:43:52.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.893 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca742ffb:::benchmark_data_smithi188_141483_object80545:head# 2024-04-23T16:43:52.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.893 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca7482c5:::benchmark_data_smithi188_141483_object59971:head# 2024-04-23T16:43:52.893 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca750f53:::benchmark_data_smithi188_141483_object26207:head# 2024-04-23T16:43:52.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca752563:::benchmark_data_smithi188_141483_object77458:head# 2024-04-23T16:43:52.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca753635:::benchmark_data_smithi188_141483_object69258:head# 2024-04-23T16:43:52.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca76526b:::benchmark_data_smithi188_141483_object52594:head# 2024-04-23T16:43:52.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca76f8dd:::benchmark_data_smithi188_141483_object27829:head# 2024-04-23T16:43:52.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.894 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca77e32d:::benchmark_data_smithi188_141483_object28769:head# 2024-04-23T16:43:52.894 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca786d35:::benchmark_data_smithi188_141483_object63349:head# 2024-04-23T16:43:52.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca790e21:::benchmark_data_smithi188_141483_object80841:head# 2024-04-23T16:43:52.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca793df3:::benchmark_data_smithi188_141483_object34081:head# 2024-04-23T16:43:52.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca7b76b4:::benchmark_data_smithi188_141483_object51058:head# 2024-04-23T16:43:52.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca7c940e:::benchmark_data_smithi188_141483_object65358:head# 2024-04-23T16:43:52.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca7d923d:::benchmark_data_smithi188_141483_object80505:head# 2024-04-23T16:43:52.895 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.895 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca7db83c:::benchmark_data_smithi188_141483_object39765:head# 2024-04-23T16:43:52.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca7de7ff:::benchmark_data_smithi188_141483_object67935:head# 2024-04-23T16:43:52.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca802e09:::benchmark_data_smithi188_141483_object59902:head# 2024-04-23T16:43:52.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca80ead3:::benchmark_data_smithi188_141483_object81321:head# 2024-04-23T16:43:52.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca81b7c1:::benchmark_data_smithi188_141483_object24007:head# 2024-04-23T16:43:52.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca81c50d:::benchmark_data_smithi188_141483_object81182:head# 2024-04-23T16:43:52.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca820022:::benchmark_data_smithi188_141483_object20888:head# 2024-04-23T16:43:52.896 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.896 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca82198a:::benchmark_data_smithi188_141483_object21197:head# 2024-04-23T16:43:52.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca832489:::benchmark_data_smithi188_141483_object12333:head# 2024-04-23T16:43:52.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca83ceb4:::benchmark_data_smithi188_141483_object29762:head# 2024-04-23T16:43:52.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca844920:::benchmark_data_smithi188_141483_object31558:head# 2024-04-23T16:43:52.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca84986e:::benchmark_data_smithi188_141483_object29702:head# 2024-04-23T16:43:52.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca8613b1:::benchmark_data_smithi188_141483_object80016:head# 2024-04-23T16:43:52.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.897 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca896053:::benchmark_data_smithi188_141483_object8419:head# 2024-04-23T16:43:52.897 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca89735e:::benchmark_data_smithi188_141483_object90420:head# 2024-04-23T16:43:52.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca8ba1fb:::benchmark_data_smithi188_141483_object30265:head# 2024-04-23T16:43:52.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca8c5dba:::benchmark_data_smithi188_141483_object31784:head# 2024-04-23T16:43:52.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca8dc4e9:::benchmark_data_smithi188_141483_object26055:head# 2024-04-23T16:43:52.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca8dd9db:::benchmark_data_smithi188_141483_object80959:head# 2024-04-23T16:43:52.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca8e8855:::benchmark_data_smithi188_141483_object32503:head# 2024-04-23T16:43:52.898 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.898 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca8ec499:::benchmark_data_smithi188_141483_object45607:head# 2024-04-23T16:43:52.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.899 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca8f06db:::benchmark_data_smithi188_141483_object74582:head# 2024-04-23T16:43:52.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.899 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca8f3c9b:::benchmark_data_smithi188_141483_object30939:head# 2024-04-23T16:43:52.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.899 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca8f8c61:::benchmark_data_smithi188_141483_object16271:head# 2024-04-23T16:43:52.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.899 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca91de2f:::benchmark_data_smithi188_141483_object67307:head# 2024-04-23T16:43:52.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.899 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca935b3b:::benchmark_data_smithi188_141483_object22737:head# 2024-04-23T16:43:52.899 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca944f84:::benchmark_data_smithi188_141483_object80399:head# 2024-04-23T16:43:52.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca95d508:::benchmark_data_smithi188_141483_object42604:head# 2024-04-23T16:43:52.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca97039a:::benchmark_data_smithi188_141483_object41592:head# 2024-04-23T16:43:52.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca977180:::benchmark_data_smithi188_141483_object7691:head# 2024-04-23T16:43:52.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca979d3e:::benchmark_data_smithi188_141483_object82887:head# 2024-04-23T16:43:52.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca99e79f:::benchmark_data_smithi188_141483_object69517:head# 2024-04-23T16:43:52.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.900 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca9a50ae:::benchmark_data_smithi188_141483_object28488:head# 2024-04-23T16:43:52.900 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.901 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca9a9738:::benchmark_data_smithi188_141483_object52419:head# 2024-04-23T16:43:52.901 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.901 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca9a9a7e:::benchmark_data_smithi188_141483_object40283:head# 2024-04-23T16:43:52.901 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.901 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca9b192d:::benchmark_data_smithi188_141483_object3394:head# 2024-04-23T16:43:52.901 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.901 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca9bacfb:::benchmark_data_smithi188_141483_object88477:head# 2024-04-23T16:43:52.901 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.901 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca9bb29b:::benchmark_data_smithi188_141483_object24135:head# 2024-04-23T16:43:52.901 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.901 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca9c93e7:::benchmark_data_smithi188_141483_object60434:head# 2024-04-23T16:43:52.901 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.901 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca9da072:::benchmark_data_smithi188_141483_object25328:head# 2024-04-23T16:43:52.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ca9f6d5e:::benchmark_data_smithi188_141483_object14668:head# 2024-04-23T16:43:52.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caa0069a:::benchmark_data_smithi188_141483_object16313:head# 2024-04-23T16:43:52.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caa0202c:::benchmark_data_smithi188_141483_object20580:head# 2024-04-23T16:43:52.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caa09dfe:::benchmark_data_smithi188_141483_object49887:head# 2024-04-23T16:43:52.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caa13070:::benchmark_data_smithi188_141483_object71531:head# 2024-04-23T16:43:52.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.902 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caa33816:::benchmark_data_smithi188_141483_object3288:head# 2024-04-23T16:43:52.902 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caa3b5ca:::benchmark_data_smithi188_141483_object59943:head# 2024-04-23T16:43:52.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caa47ecb:::benchmark_data_smithi188_141483_object2000:head# 2024-04-23T16:43:52.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caa591ef:::benchmark_data_smithi188_141483_object10182:head# 2024-04-23T16:43:52.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caa60996:::benchmark_data_smithi188_141483_object54436:head# 2024-04-23T16:43:52.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caa65e42:::benchmark_data_smithi188_141483_object59997:head# 2024-04-23T16:43:52.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caa74f93:::benchmark_data_smithi188_141483_object51896:head# 2024-04-23T16:43:52.903 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.903 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caa83738:::benchmark_data_smithi188_141483_object78530:head# 2024-04-23T16:43:52.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caa9191b:::benchmark_data_smithi188_141483_object34485:head# 2024-04-23T16:43:52.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caa988d5:::benchmark_data_smithi188_141483_object23487:head# 2024-04-23T16:43:52.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caaa5ffb:::benchmark_data_smithi188_141483_object2855:head# 2024-04-23T16:43:52.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caab930f:::benchmark_data_smithi188_141483_object5313:head# 2024-04-23T16:43:52.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caabf2d4:::benchmark_data_smithi188_141483_object7511:head# 2024-04-23T16:43:52.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.904 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caac2454:::benchmark_data_smithi188_141483_object82379:head# 2024-04-23T16:43:52.904 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caac54cc:::benchmark_data_smithi188_141483_object2206:head# 2024-04-23T16:43:52.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caac63eb:::benchmark_data_smithi188_141483_object1043:head# 2024-04-23T16:43:52.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caac95ef:::benchmark_data_smithi188_141483_object14594:head# 2024-04-23T16:43:52.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caaced08:::benchmark_data_smithi188_141483_object43837:head# 2024-04-23T16:43:52.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caacf2f3:::benchmark_data_smithi188_141483_object39592:head# 2024-04-23T16:43:52.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.905 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caae69a2:::benchmark_data_smithi188_141483_object58907:head# 2024-04-23T16:43:52.905 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cab0a8df:::benchmark_data_smithi188_141483_object7952:head# 2024-04-23T16:43:52.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cab11c61:::benchmark_data_smithi188_141483_object89642:head# 2024-04-23T16:43:52.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cab14ff2:::benchmark_data_smithi188_141483_object32994:head# 2024-04-23T16:43:52.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cab1d376:::benchmark_data_smithi188_141483_object21101:head# 2024-04-23T16:43:52.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cab1ec04:::benchmark_data_smithi188_141483_object59231:head# 2024-04-23T16:43:52.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.906 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cab2091f:::benchmark_data_smithi188_141483_object9608:head# 2024-04-23T16:43:52.906 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cab22741:::benchmark_data_smithi188_141483_object17536:head# 2024-04-23T16:43:52.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cab2323c:::benchmark_data_smithi188_141483_object60179:head# 2024-04-23T16:43:52.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cab641f6:::benchmark_data_smithi188_141483_object68831:head# 2024-04-23T16:43:52.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cab890c7:::benchmark_data_smithi188_141483_object57432:head# 2024-04-23T16:43:52.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cab95452:::benchmark_data_smithi188_141483_object37864:head# 2024-04-23T16:43:52.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caba1a92:::benchmark_data_smithi188_141483_object56616:head# 2024-04-23T16:43:52.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.907 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caba7a16:::benchmark_data_smithi188_141483_object77024:head# 2024-04-23T16:43:52.907 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cabc55bf:::benchmark_data_smithi188_141483_object76056:head# 2024-04-23T16:43:52.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cabc6ab5:::benchmark_data_smithi188_141483_object46519:head# 2024-04-23T16:43:52.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cabd773b:::benchmark_data_smithi188_141483_object81779:head# 2024-04-23T16:43:52.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cabe52fe:::benchmark_data_smithi188_141483_object15121:head# 2024-04-23T16:43:52.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cabea079:::benchmark_data_smithi188_141483_object12121:head# 2024-04-23T16:43:52.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cabf5241:::benchmark_data_smithi188_141483_object87377:head# 2024-04-23T16:43:52.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.908 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cabf5525:::benchmark_data_smithi188_141483_object49189:head# 2024-04-23T16:43:52.908 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.909 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cabfb846:::benchmark_data_smithi188_141483_object29732:head# 2024-04-23T16:43:52.909 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.909 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cabfbe88:::benchmark_data_smithi188_141483_object14392:head# 2024-04-23T16:43:52.909 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.909 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cac06381:::benchmark_data_smithi188_141483_object67778:head# 2024-04-23T16:43:52.909 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.909 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cac2ede8:::benchmark_data_smithi188_141483_object85019:head# 2024-04-23T16:43:52.909 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.909 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cac314db:::benchmark_data_smithi188_141483_object88775:head# 2024-04-23T16:43:52.909 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.909 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cac36b2b:::benchmark_data_smithi188_141483_object49346:head# 2024-04-23T16:43:52.909 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.909 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cac3ad93:::benchmark_data_smithi188_141483_object85979:head# 2024-04-23T16:43:52.909 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.909 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cac54a3d:::benchmark_data_smithi188_141483_object2308:head# 2024-04-23T16:43:52.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cac55a7c:::benchmark_data_smithi188_141483_object52140:head# 2024-04-23T16:43:52.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cac5a9ad:::benchmark_data_smithi188_141483_object42286:head# 2024-04-23T16:43:52.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cac5ed0f:::benchmark_data_smithi188_141483_object19317:head# 2024-04-23T16:43:52.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cac66097:::benchmark_data_smithi188_141483_object35066:head# 2024-04-23T16:43:52.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cac6ca31:::benchmark_data_smithi188_141483_object23362:head# 2024-04-23T16:43:52.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.910 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cac7e210:::benchmark_data_smithi188_141483_object82747:head# 2024-04-23T16:43:52.910 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cac8c4cc:::benchmark_data_smithi188_141483_object63104:head# 2024-04-23T16:43:52.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cac9e0cb:::benchmark_data_smithi188_141483_object29314:head# 2024-04-23T16:43:52.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caca314e:::benchmark_data_smithi188_141483_object79264:head# 2024-04-23T16:43:52.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caca56fb:::benchmark_data_smithi188_141483_object38454:head# 2024-04-23T16:43:52.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caca6ca5:::benchmark_data_smithi188_141483_object32369:head# 2024-04-23T16:43:52.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cacb2887:::benchmark_data_smithi188_141483_object24265:head# 2024-04-23T16:43:52.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.911 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cacb5076:::benchmark_data_smithi188_141483_object230:head# 2024-04-23T16:43:52.911 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cacb840d:::benchmark_data_smithi188_141483_object53611:head# 2024-04-23T16:43:52.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cacc1adc:::benchmark_data_smithi188_141483_object39761:head# 2024-04-23T16:43:52.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caccd5d0:::benchmark_data_smithi188_141483_object58499:head# 2024-04-23T16:43:52.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cacce554:::benchmark_data_smithi188_141483_object4959:head# 2024-04-23T16:43:52.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cace05c2:::benchmark_data_smithi188_141483_object80253:head# 2024-04-23T16:43:52.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.912 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cacea436:::benchmark_data_smithi188_141483_object10537:head# 2024-04-23T16:43:52.912 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cad19fec:::benchmark_data_smithi188_141483_object38345:head# 2024-04-23T16:43:52.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cad2242e:::benchmark_data_smithi188_141483_object37821:head# 2024-04-23T16:43:52.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cad2bf9f:::benchmark_data_smithi188_141483_object29222:head# 2024-04-23T16:43:52.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cad2df02:::benchmark_data_smithi188_141483_object1420:head# 2024-04-23T16:43:52.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cad5dd7c:::benchmark_data_smithi188_141483_object54601:head# 2024-04-23T16:43:52.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cad632fc:::benchmark_data_smithi188_141483_object2163:head# 2024-04-23T16:43:52.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cad70171:::benchmark_data_smithi188_141483_object82783:head# 2024-04-23T16:43:52.913 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.913 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cad71559:::benchmark_data_smithi188_141483_object85367:head# 2024-04-23T16:43:52.914 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.914 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cad88422:::benchmark_data_smithi188_141483_object54743:head# 2024-04-23T16:43:52.914 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.914 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cad8f14f:::benchmark_data_smithi188_141483_object40033:head# 2024-04-23T16:43:52.914 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.914 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cad968d9:::benchmark_data_smithi188_141483_object81325:head# 2024-04-23T16:43:52.914 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.914 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cad96ee4:::benchmark_data_smithi188_141483_object74678:head# 2024-04-23T16:43:52.914 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.914 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cad9bd0e:::benchmark_data_smithi188_141483_object70539:head# 2024-04-23T16:43:52.914 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.914 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cad9ed18:::benchmark_data_smithi188_141483_object63771:head# 2024-04-23T16:43:52.914 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.915 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cada8fa8:::benchmark_data_smithi188_141483_object89519:head# 2024-04-23T16:43:52.915 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.915 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cadb78bb:::benchmark_data_smithi188_141483_object24934:head# 2024-04-23T16:43:52.915 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.915 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cadc5909:::benchmark_data_smithi188_141483_object42268:head# 2024-04-23T16:43:52.915 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.915 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cadde2c1:::benchmark_data_smithi188_141483_object22579:head# 2024-04-23T16:43:52.915 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.915 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cade7af7:::benchmark_data_smithi188_141483_object48766:head# 2024-04-23T16:43:52.915 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.915 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cadec3e4:::benchmark_data_smithi188_141483_object84535:head# 2024-04-23T16:43:52.915 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.915 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cadfaea4:::benchmark_data_smithi188_141483_object71529:head# 2024-04-23T16:43:52.915 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.916 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cae05d00:::benchmark_data_smithi188_141483_object36374:head# 2024-04-23T16:43:52.916 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.916 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cae0c71d:::benchmark_data_smithi188_141483_object14666:head# 2024-04-23T16:43:52.916 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.916 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cae217ec:::benchmark_data_smithi188_141483_object17670:head# 2024-04-23T16:43:52.916 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.916 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cae3e638:::benchmark_data_smithi188_141483_object80783:head# 2024-04-23T16:43:52.916 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.916 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cae424f9:::benchmark_data_smithi188_141483_object83454:head# 2024-04-23T16:43:52.916 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.916 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cae4ac7c:::benchmark_data_smithi188_141483_object48558:head# 2024-04-23T16:43:52.916 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.916 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cae5328d:::benchmark_data_smithi188_141483_object46657:head# 2024-04-23T16:43:52.916 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.916 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cae5994a:::benchmark_data_smithi188_141483_object65614:head# 2024-04-23T16:43:52.916 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cae62f3d:::benchmark_data_smithi188_141483_object82415:head# 2024-04-23T16:43:52.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cae69a95:::benchmark_data_smithi188_141483_object72537:head# 2024-04-23T16:43:52.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cae7f1f7:::benchmark_data_smithi188_141483_object87973:head# 2024-04-23T16:43:52.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cae8ce77:::benchmark_data_smithi188_141483_object64573:head# 2024-04-23T16:43:52.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cae91ed9:::benchmark_data_smithi188_141483_object12882:head# 2024-04-23T16:43:52.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caeb55e0:::benchmark_data_smithi188_141483_object2653:head# 2024-04-23T16:43:52.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caedfee0:::benchmark_data_smithi188_141483_object87300:head# 2024-04-23T16:43:52.917 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.917 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caefe464:::benchmark_data_smithi188_141483_object84185:head# 2024-04-23T16:43:52.918 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.918 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caf05e93:::benchmark_data_smithi188_141483_object5462:head# 2024-04-23T16:43:52.918 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.918 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caf123b9:::benchmark_data_smithi188_141483_object50502:head# 2024-04-23T16:43:52.918 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.918 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caf25bc8:::benchmark_data_smithi188_141483_object69620:head# 2024-04-23T16:43:52.918 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.918 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caf40bfd:::benchmark_data_smithi188_141483_object89965:head# 2024-04-23T16:43:52.918 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.918 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caf541d7:::benchmark_data_smithi188_141483_object16821:head# 2024-04-23T16:43:52.918 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.918 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caf54248:::benchmark_data_smithi188_141483_object31292:head# 2024-04-23T16:43:52.918 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.919 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caf5db99:::benchmark_data_smithi188_141483_object86661:head# 2024-04-23T16:43:52.919 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.919 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caf6009b:::benchmark_data_smithi188_141483_object32600:head# 2024-04-23T16:43:52.919 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.919 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caf63dd9:::benchmark_data_smithi188_141483_object10825:head# 2024-04-23T16:43:52.919 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.919 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caf734ae:::benchmark_data_smithi188_141483_object60012:head# 2024-04-23T16:43:52.919 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.919 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caf8ced1:::benchmark_data_smithi188_141483_object25411:head# 2024-04-23T16:43:52.919 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.919 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caf91f46:::benchmark_data_smithi188_141483_object77589:head# 2024-04-23T16:43:52.919 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.920 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caf92c19:::benchmark_data_smithi188_141483_object8788:head# 2024-04-23T16:43:52.920 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.920 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cafa5f82:::benchmark_data_smithi188_141483_object17312:head# 2024-04-23T16:43:52.920 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.920 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cafe2a1a:::benchmark_data_smithi188_141483_object47190:head# 2024-04-23T16:43:52.920 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.920 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cafe9a99:::benchmark_data_smithi188_141483_object46363:head# 2024-04-23T16:43:52.920 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.920 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:caffc1a9:::benchmark_data_smithi188_141483_object9655:head# 2024-04-23T16:43:52.920 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.920 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb00830c:::benchmark_data_smithi188_141483_object57956:head# 2024-04-23T16:43:52.920 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.920 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb02058b:::benchmark_data_smithi188_141483_object52011:head# 2024-04-23T16:43:52.920 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.921 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb024483:::benchmark_data_smithi188_141483_object88397:head# 2024-04-23T16:43:52.921 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.921 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb036f55:::benchmark_data_smithi188_141483_object44270:head# 2024-04-23T16:43:52.921 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.921 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb04397b:::benchmark_data_smithi188_141483_object89769:head# 2024-04-23T16:43:52.921 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.921 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb04798c:::benchmark_data_smithi188_141483_object38412:head# 2024-04-23T16:43:52.921 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.921 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb0591c0:::benchmark_data_smithi188_141483_object19786:head# 2024-04-23T16:43:52.921 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.921 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb078920:::benchmark_data_smithi188_141483_object21737:head# 2024-04-23T16:43:52.921 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.921 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb07bb64:::benchmark_data_smithi188_141483_object19021:head# 2024-04-23T16:43:52.921 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.921 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb080d41:::benchmark_data_smithi188_141483_object62864:head# 2024-04-23T16:43:52.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.922 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb0a07c0:::benchmark_data_smithi188_141483_object16747:head# 2024-04-23T16:43:52.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.922 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb0a36a4:::benchmark_data_smithi188_141483_object47312:head# 2024-04-23T16:43:52.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.922 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb0dbbd2:::benchmark_data_smithi188_141483_object72587:head# 2024-04-23T16:43:52.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.922 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb0ddad2:::benchmark_data_smithi188_141483_object89080:head# 2024-04-23T16:43:52.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.922 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb0de72f:::benchmark_data_smithi188_141483_object17612:head# 2024-04-23T16:43:52.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.922 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb0e3fb5:::benchmark_data_smithi188_141483_object54322:head# 2024-04-23T16:43:52.922 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb0f5819:::benchmark_data_smithi188_141483_object59028:head# 2024-04-23T16:43:52.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb0fa0e8:::benchmark_data_smithi188_141483_object39160:head# 2024-04-23T16:43:52.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb115208:::benchmark_data_smithi188_141483_object4981:head# 2024-04-23T16:43:52.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb116afc:::benchmark_data_smithi188_141483_object83121:head# 2024-04-23T16:43:52.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb1197ad:::benchmark_data_smithi188_141483_object3452:head# 2024-04-23T16:43:52.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb1199d2:::benchmark_data_smithi188_141483_object87762:head# 2024-04-23T16:43:52.923 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.923 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb12af73:::benchmark_data_smithi188_141483_object29263:head# 2024-04-23T16:43:52.924 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb12c109:::benchmark_data_smithi188_141483_object15770:head# 2024-04-23T16:43:52.924 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb13089a:::benchmark_data_smithi188_141483_object52110:head# 2024-04-23T16:43:52.924 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb1355f3:::benchmark_data_smithi188_141483_object6610:head# 2024-04-23T16:43:52.924 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb136adb:::benchmark_data_smithi188_141483_object80501:head# 2024-04-23T16:43:52.924 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb137268:::benchmark_data_smithi188_141483_object84829:head# 2024-04-23T16:43:52.924 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb1384bb:::benchmark_data_smithi188_141483_object25103:head# 2024-04-23T16:43:52.924 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.924 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb138dbb:::benchmark_data_smithi188_141483_object22552:head# 2024-04-23T16:43:52.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb15043c:::benchmark_data_smithi188_141483_object81185:head# 2024-04-23T16:43:52.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb167907:::benchmark_data_smithi188_141483_object17257:head# 2024-04-23T16:43:52.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb16e223:::benchmark_data_smithi188_141483_object66929:head# 2024-04-23T16:43:52.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb17114c:::benchmark_data_smithi188_141483_object11085:head# 2024-04-23T16:43:52.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb185323:::benchmark_data_smithi188_141483_object17148:head# 2024-04-23T16:43:52.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb198fbc:::benchmark_data_smithi188_141483_object87826:head# 2024-04-23T16:43:52.925 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.925 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb1990f1:::benchmark_data_smithi188_141483_object17463:head# 2024-04-23T16:43:52.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb1999f3:::benchmark_data_smithi188_141483_object35442:head# 2024-04-23T16:43:52.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb199f55:::benchmark_data_smithi188_141483_object69268:head# 2024-04-23T16:43:52.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb19aab2:::benchmark_data_smithi188_141483_object54791:head# 2024-04-23T16:43:52.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb1a888d:::benchmark_data_smithi188_141483_object33807:head# 2024-04-23T16:43:52.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb1c1923:::benchmark_data_smithi188_141483_object8342:head# 2024-04-23T16:43:52.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.926 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb1cdb02:::benchmark_data_smithi188_141483_object77274:head# 2024-04-23T16:43:52.926 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb1ec734:::benchmark_data_smithi188_141483_object65669:head# 2024-04-23T16:43:52.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb1f4f74:::benchmark_data_smithi188_141483_object83553:head# 2024-04-23T16:43:52.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb206b86:::benchmark_data_smithi188_141483_object17854:head# 2024-04-23T16:43:52.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb2474e1:::benchmark_data_smithi188_141483_object85518:head# 2024-04-23T16:43:52.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb26a56b:::benchmark_data_smithi188_141483_object52615:head# 2024-04-23T16:43:52.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb27c1ba:::benchmark_data_smithi188_141483_object32212:head# 2024-04-23T16:43:52.927 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.927 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb296486:::benchmark_data_smithi188_141483_object71457:head# 2024-04-23T16:43:52.928 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.928 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb29786b:::benchmark_data_smithi188_141483_object32999:head# 2024-04-23T16:43:52.928 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.928 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb29f2a6:::benchmark_data_smithi188_141483_object90574:head# 2024-04-23T16:43:52.928 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.928 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb2b64f8:::benchmark_data_smithi188_141483_object34396:head# 2024-04-23T16:43:52.928 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.928 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb2c68e8:::benchmark_data_smithi188_141483_object11052:head# 2024-04-23T16:43:52.928 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.928 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb2ce697:::benchmark_data_smithi188_141483_object29471:head# 2024-04-23T16:43:52.928 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.928 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb2e9b33:::benchmark_data_smithi188_141483_object85776:head# 2024-04-23T16:43:52.928 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.929 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb2ea74f:::benchmark_data_smithi188_141483_object18494:head# 2024-04-23T16:43:52.929 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.929 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb2f6077:::benchmark_data_smithi188_141483_object21047:head# 2024-04-23T16:43:52.929 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.929 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb2f9ea5:::benchmark_data_smithi188_141483_object36342:head# 2024-04-23T16:43:52.929 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.929 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb2fa7b7:::benchmark_data_smithi188_141483_object16602:head# 2024-04-23T16:43:52.929 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.929 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb3094ec:::benchmark_data_smithi188_141483_object87670:head# 2024-04-23T16:43:52.929 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.929 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb30db32:::benchmark_data_smithi188_141483_object37321:head# 2024-04-23T16:43:52.929 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.929 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb313e20:::benchmark_data_smithi188_141483_object52055:head# 2024-04-23T16:43:52.929 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb3165ee:::benchmark_data_smithi188_141483_object43268:head# 2024-04-23T16:43:52.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb31d897:::benchmark_data_smithi188_141483_object11775:head# 2024-04-23T16:43:52.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb31df53:::benchmark_data_smithi188_141483_object50964:head# 2024-04-23T16:43:52.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb34b8d3:::benchmark_data_smithi188_141483_object88386:head# 2024-04-23T16:43:52.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb35bb9d:::benchmark_data_smithi188_141483_object70592:head# 2024-04-23T16:43:52.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb35efeb:::benchmark_data_smithi188_141483_object7405:head# 2024-04-23T16:43:52.930 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.930 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb36c6f2:::benchmark_data_smithi188_141483_object31643:head# 2024-04-23T16:43:52.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb3762f4:::benchmark_data_smithi188_141483_object70606:head# 2024-04-23T16:43:52.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb37fefc:::benchmark_data_smithi188_141483_object5269:head# 2024-04-23T16:43:52.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb384cc9:::benchmark_data_smithi188_141483_object12975:head# 2024-04-23T16:43:52.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb38c2bf:::benchmark_data_smithi188_141483_object15168:head# 2024-04-23T16:43:52.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb3d911d:::benchmark_data_smithi188_141483_object10733:head# 2024-04-23T16:43:52.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb3dd4ee:::benchmark_data_smithi188_141483_object26172:head# 2024-04-23T16:43:52.931 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.931 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb3e8ed8:::benchmark_data_smithi188_141483_object44517:head# 2024-04-23T16:43:52.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb3fcbd1:::benchmark_data_smithi188_141483_object54462:head# 2024-04-23T16:43:52.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb40d654:::benchmark_data_smithi188_141483_object61631:head# 2024-04-23T16:43:52.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb414997:::benchmark_data_smithi188_141483_object44339:head# 2024-04-23T16:43:52.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb41bc9b:::benchmark_data_smithi188_141483_object48155:head# 2024-04-23T16:43:52.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb43217d:::benchmark_data_smithi188_141483_object46996:head# 2024-04-23T16:43:52.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb4366ae:::benchmark_data_smithi188_141483_object59568:head# 2024-04-23T16:43:52.932 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.932 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb441eec:::benchmark_data_smithi188_141483_object82052:head# 2024-04-23T16:43:52.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb46d109:::benchmark_data_smithi188_141483_object1801:head# 2024-04-23T16:43:52.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb470643:::benchmark_data_smithi188_141483_object70527:head# 2024-04-23T16:43:52.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb474e5e:::benchmark_data_smithi188_141483_object66686:head# 2024-04-23T16:43:52.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb47b81b:::benchmark_data_smithi188_141483_object49792:head# 2024-04-23T16:43:52.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb47fe86:::benchmark_data_smithi188_141483_object78846:head# 2024-04-23T16:43:52.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb4821b3:::benchmark_data_smithi188_141483_object33605:head# 2024-04-23T16:43:52.933 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.933 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb4a032a:::benchmark_data_smithi188_141483_object71600:head# 2024-04-23T16:43:52.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb4a0824:::benchmark_data_smithi188_141483_object36941:head# 2024-04-23T16:43:52.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb4b746a:::benchmark_data_smithi188_141483_object73326:head# 2024-04-23T16:43:52.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb4c9af0:::benchmark_data_smithi188_141483_object60231:head# 2024-04-23T16:43:52.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb4d42a1:::benchmark_data_smithi188_141483_object38727:head# 2024-04-23T16:43:52.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb4dbcfc:::benchmark_data_smithi188_141483_object24690:head# 2024-04-23T16:43:52.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb4dedf0:::benchmark_data_smithi188_141483_object15428:head# 2024-04-23T16:43:52.934 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.934 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb50042f:::benchmark_data_smithi188_141483_object2543:head# 2024-04-23T16:43:52.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb501d42:::benchmark_data_smithi188_141483_object70058:head# 2024-04-23T16:43:52.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb503e55:::benchmark_data_smithi188_141483_object79835:head# 2024-04-23T16:43:52.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb506349:::benchmark_data_smithi188_141483_object40835:head# 2024-04-23T16:43:52.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb5067aa:::benchmark_data_smithi188_141483_object3925:head# 2024-04-23T16:43:52.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb5073c3:::benchmark_data_smithi188_141483_object44935:head# 2024-04-23T16:43:52.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb50eb48:::benchmark_data_smithi188_141483_object50364:head# 2024-04-23T16:43:52.935 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.935 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb5144f9:::benchmark_data_smithi188_141483_object64680:head# 2024-04-23T16:43:52.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb5189eb:::benchmark_data_smithi188_141483_object35629:head# 2024-04-23T16:43:52.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb52a88d:::benchmark_data_smithi188_141483_object26946:head# 2024-04-23T16:43:52.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb531162:::benchmark_data_smithi188_141483_object45983:head# 2024-04-23T16:43:52.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb534e6e:::benchmark_data_smithi188_141483_object666:head# 2024-04-23T16:43:52.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb551962:::benchmark_data_smithi188_141483_object73939:head# 2024-04-23T16:43:52.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb553710:::benchmark_data_smithi188_141483_object11324:head# 2024-04-23T16:43:52.936 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.936 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb555699:::benchmark_data_smithi188_141483_object20632:head# 2024-04-23T16:43:52.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb576be7:::benchmark_data_smithi188_141483_object86814:head# 2024-04-23T16:43:52.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb57a623:::benchmark_data_smithi188_141483_object45178:head# 2024-04-23T16:43:52.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb57b28f:::benchmark_data_smithi188_141483_object40937:head# 2024-04-23T16:43:52.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb58c819:::benchmark_data_smithi188_141483_object70600:head# 2024-04-23T16:43:52.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.937 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb5945cf:::benchmark_data_smithi188_141483_object6447:head# 2024-04-23T16:43:52.937 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb59a4b0:::benchmark_data_smithi188_141483_object28507:head# 2024-04-23T16:43:52.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb5a28d4:::benchmark_data_smithi188_141483_object58092:head# 2024-04-23T16:43:52.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb5a3e2d:::benchmark_data_smithi188_141483_object11492:head# 2024-04-23T16:43:52.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb5aa2e8:::benchmark_data_smithi188_141483_object43868:head# 2024-04-23T16:43:52.938 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.938 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb5bbb5c:::benchmark_data_smithi188_141483_object7244:head# 2024-04-23T16:43:52.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb5be4b0:::benchmark_data_smithi188_141483_object18453:head# 2024-04-23T16:43:52.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb5bff6f:::benchmark_data_smithi188_141483_object78040:head# 2024-04-23T16:43:52.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb5ce1cd:::benchmark_data_smithi188_141483_object41286:head# 2024-04-23T16:43:52.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb5e6a7f:::benchmark_data_smithi188_141483_object86310:head# 2024-04-23T16:43:52.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb5e79ca:::benchmark_data_smithi188_141483_object81384:head# 2024-04-23T16:43:52.939 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.939 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb5ebbe5:::benchmark_data_smithi188_141483_object9597:head# 2024-04-23T16:43:52.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb60ccba:::benchmark_data_smithi188_141483_object50937:head# 2024-04-23T16:43:52.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb613e6b:::benchmark_data_smithi188_141483_object35456:head# 2024-04-23T16:43:52.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb61c5b0:::benchmark_data_smithi188_141483_object82361:head# 2024-04-23T16:43:52.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb61c85d:::benchmark_data_smithi188_141483_object74065:head# 2024-04-23T16:43:52.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb61f620:::benchmark_data_smithi188_141483_object59420:head# 2024-04-23T16:43:52.940 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.940 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb62121c:::benchmark_data_smithi188_141483_object7812:head# 2024-04-23T16:43:52.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb62ccf5:::benchmark_data_smithi188_141483_object19718:head# 2024-04-23T16:43:52.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb635c38:::benchmark_data_smithi188_141483_object90799:head# 2024-04-23T16:43:52.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb63b20c:::benchmark_data_smithi188_141483_object70146:head# 2024-04-23T16:43:52.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb642e62:::benchmark_data_smithi188_141483_object81676:head# 2024-04-23T16:43:52.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb64e394:::benchmark_data_smithi188_141483_object2225:head# 2024-04-23T16:43:52.941 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.941 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb659c9d:::benchmark_data_smithi188_141483_object30814:head# 2024-04-23T16:43:52.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb65e314:::benchmark_data_smithi188_141483_object15804:head# 2024-04-23T16:43:52.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb66e697:::benchmark_data_smithi188_141483_object44793:head# 2024-04-23T16:43:52.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb670519:::benchmark_data_smithi188_141483_object58533:head# 2024-04-23T16:43:52.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb67bd58:::benchmark_data_smithi188_141483_object67649:head# 2024-04-23T16:43:52.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb67e68b:::benchmark_data_smithi188_141483_object14344:head# 2024-04-23T16:43:52.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.942 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb6863cc:::benchmark_data_smithi188_141483_object59717:head# 2024-04-23T16:43:52.942 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb690850:::benchmark_data_smithi188_141483_object71612:head# 2024-04-23T16:43:52.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb699b66:::benchmark_data_smithi188_141483_object41326:head# 2024-04-23T16:43:52.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb69ac3b:::benchmark_data_smithi188_141483_object65848:head# 2024-04-23T16:43:52.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb6a7bd3:::benchmark_data_smithi188_141483_object20780:head# 2024-04-23T16:43:52.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb6ad54d:::benchmark_data_smithi188_141483_object14122:head# 2024-04-23T16:43:52.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb6c2599:::benchmark_data_smithi188_141483_object27096:head# 2024-04-23T16:43:52.943 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.943 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb6c7685:::benchmark_data_smithi188_141483_object17333:head# 2024-04-23T16:43:52.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb6c83ce:::benchmark_data_smithi188_141483_object75225:head# 2024-04-23T16:43:52.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb6de878:::benchmark_data_smithi188_141483_object76773:head# 2024-04-23T16:43:52.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb6e5191:::benchmark_data_smithi188_141483_object67823:head# 2024-04-23T16:43:52.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb6e8144:::benchmark_data_smithi188_141483_object76632:head# 2024-04-23T16:43:52.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb6e834c:::benchmark_data_smithi188_141483_object33123:head# 2024-04-23T16:43:52.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb6f2110:::benchmark_data_smithi188_141483_object31793:head# 2024-04-23T16:43:52.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.944 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb70f624:::benchmark_data_smithi188_141483_object78845:head# 2024-04-23T16:43:52.944 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb714c82:::benchmark_data_smithi188_141483_object42373:head# 2024-04-23T16:43:52.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb749e45:::benchmark_data_smithi188_141483_object65244:head# 2024-04-23T16:43:52.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb74b2b4:::benchmark_data_smithi188_141483_object46292:head# 2024-04-23T16:43:52.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb762942:::benchmark_data_smithi188_141483_object28998:head# 2024-04-23T16:43:52.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb76db66:::benchmark_data_smithi188_141483_object29080:head# 2024-04-23T16:43:52.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb771574:::benchmark_data_smithi188_141483_object18607:head# 2024-04-23T16:43:52.945 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.945 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb7903d7:::benchmark_data_smithi188_141483_object62532:head# 2024-04-23T16:43:52.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb797de5:::benchmark_data_smithi188_141483_object30877:head# 2024-04-23T16:43:52.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb79a3dd:::benchmark_data_smithi188_141483_object27061:head# 2024-04-23T16:43:52.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb79f938:::benchmark_data_smithi188_141483_object39428:head# 2024-04-23T16:43:52.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb7ac580:::benchmark_data_smithi188_141483_object9311:head# 2024-04-23T16:43:52.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb7b7bfc:::benchmark_data_smithi188_141483_object16773:head# 2024-04-23T16:43:52.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.946 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb7ba191:::benchmark_data_smithi188_141483_object24444:head# 2024-04-23T16:43:52.946 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb7c5ecf:::benchmark_data_smithi188_141483_object71350:head# 2024-04-23T16:43:52.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb7ce6a2:::benchmark_data_smithi188_141483_object27717:head# 2024-04-23T16:43:52.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb7de4fa:::benchmark_data_smithi188_141483_object4482:head# 2024-04-23T16:43:52.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb7ed74f:::benchmark_data_smithi188_141483_object23186:head# 2024-04-23T16:43:52.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb7ee1b6:::benchmark_data_smithi188_141483_object76991:head# 2024-04-23T16:43:52.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb7f3474:::benchmark_data_smithi188_141483_object84596:head# 2024-04-23T16:43:52.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb7f866f:::benchmark_data_smithi188_141483_object83430:head# 2024-04-23T16:43:52.947 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.947 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb7fb9fa:::benchmark_data_smithi188_141483_object17117:head# 2024-04-23T16:43:52.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb8021ec:::benchmark_data_smithi188_141483_object83795:head# 2024-04-23T16:43:52.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb80584a:::benchmark_data_smithi188_141483_object28611:head# 2024-04-23T16:43:52.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb80e1aa:::benchmark_data_smithi188_141483_object76575:head# 2024-04-23T16:43:52.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb80faac:::benchmark_data_smithi188_141483_object49220:head# 2024-04-23T16:43:52.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb8171a8:::benchmark_data_smithi188_141483_object87257:head# 2024-04-23T16:43:52.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.948 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb828114:::benchmark_data_smithi188_141483_object34341:head# 2024-04-23T16:43:52.948 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb82f845:::benchmark_data_smithi188_141483_object32930:head# 2024-04-23T16:43:52.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb834b5e:::benchmark_data_smithi188_141483_object69019:head# 2024-04-23T16:43:52.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb8673cd:::benchmark_data_smithi188_141483_object42323:head# 2024-04-23T16:43:52.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb8673fa:::benchmark_data_smithi188_141483_object30178:head# 2024-04-23T16:43:52.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb867422:::benchmark_data_smithi188_141483_object68527:head# 2024-04-23T16:43:52.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb869d5e:::benchmark_data_smithi188_141483_object40494:head# 2024-04-23T16:43:52.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb86bc79:::benchmark_data_smithi188_141483_object6745:head# 2024-04-23T16:43:52.949 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.949 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb880bf7:::benchmark_data_smithi188_141483_object60750:head# 2024-04-23T16:43:52.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb88ae96:::benchmark_data_smithi188_141483_object40473:head# 2024-04-23T16:43:52.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb895dc2:::benchmark_data_smithi188_141483_object18636:head# 2024-04-23T16:43:52.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb895f81:::benchmark_data_smithi188_141483_object28585:head# 2024-04-23T16:43:52.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb8a6a66:::benchmark_data_smithi188_141483_object63760:head# 2024-04-23T16:43:52.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb8ac78e:::benchmark_data_smithi188_141483_object2971:head# 2024-04-23T16:43:52.950 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.950 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb8c413b:::benchmark_data_smithi188_141483_object5591:head# 2024-04-23T16:43:52.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb8c7229:::benchmark_data_smithi188_141483_object16398:head# 2024-04-23T16:43:52.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb8cd64c:::benchmark_data_smithi188_141483_object75759:head# 2024-04-23T16:43:52.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb8dd59c:::benchmark_data_smithi188_141483_object48362:head# 2024-04-23T16:43:52.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb8e2b40:::benchmark_data_smithi188_141483_object41866:head# 2024-04-23T16:43:52.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb8e8560:::benchmark_data_smithi188_141483_object77723:head# 2024-04-23T16:43:52.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.951 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb8ef08c:::benchmark_data_smithi188_141483_object70575:head# 2024-04-23T16:43:52.951 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb927ebe:::benchmark_data_smithi188_141483_object21329:head# 2024-04-23T16:43:52.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb946159:::benchmark_data_smithi188_141483_object43266:head# 2024-04-23T16:43:52.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb94923b:::benchmark_data_smithi188_141483_object5040:head# 2024-04-23T16:43:52.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb958a44:::benchmark_data_smithi188_141483_object48495:head# 2024-04-23T16:43:52.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb966ba2:::benchmark_data_smithi188_141483_object47658:head# 2024-04-23T16:43:52.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb96c5d6:::benchmark_data_smithi188_141483_object79981:head# 2024-04-23T16:43:52.952 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.952 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb971336:::benchmark_data_smithi188_141483_object57939:head# 2024-04-23T16:43:52.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb97b677:::benchmark_data_smithi188_141483_object62014:head# 2024-04-23T16:43:52.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb991ba0:::benchmark_data_smithi188_141483_object4356:head# 2024-04-23T16:43:52.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb99a930:::benchmark_data_smithi188_141483_object50980:head# 2024-04-23T16:43:52.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb9ab669:::benchmark_data_smithi188_141483_object43445:head# 2024-04-23T16:43:52.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb9bc0da:::benchmark_data_smithi188_141483_object28536:head# 2024-04-23T16:43:52.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb9df3ce:::benchmark_data_smithi188_141483_object39696:head# 2024-04-23T16:43:52.953 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.953 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cb9fc84a:::benchmark_data_smithi188_141483_object44398:head# 2024-04-23T16:43:52.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cba07133:::benchmark_data_smithi188_141483_object77211:head# 2024-04-23T16:43:52.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cba1512e:::benchmark_data_smithi188_141483_object71266:head# 2024-04-23T16:43:52.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cba1b25a:::benchmark_data_smithi188_141483_object75619:head# 2024-04-23T16:43:52.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cba2fb49:::benchmark_data_smithi188_141483_object14312:head# 2024-04-23T16:43:52.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cba32fac:::benchmark_data_smithi188_141483_object87287:head# 2024-04-23T16:43:52.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cba546a6:::benchmark_data_smithi188_141483_object29838:head# 2024-04-23T16:43:52.954 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.954 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cba6723a:::benchmark_data_smithi188_141483_object38017:head# 2024-04-23T16:43:52.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cba6bb9e:::benchmark_data_smithi188_141483_object52245:head# 2024-04-23T16:43:52.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cba79ce3:::benchmark_data_smithi188_141483_object28248:head# 2024-04-23T16:43:52.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cba7fc43:::benchmark_data_smithi188_141483_object28083:head# 2024-04-23T16:43:52.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cba80da1:::benchmark_data_smithi188_141483_object81391:head# 2024-04-23T16:43:52.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cba84312:::benchmark_data_smithi188_141483_object77669:head# 2024-04-23T16:43:52.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbaa1e97:::benchmark_data_smithi188_141483_object42614:head# 2024-04-23T16:43:52.955 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.955 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbaa6a6d:::benchmark_data_smithi188_141483_object31936:head# 2024-04-23T16:43:52.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbab2985:::benchmark_data_smithi188_141483_object42150:head# 2024-04-23T16:43:52.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbab72de:::benchmark_data_smithi188_141483_object11033:head# 2024-04-23T16:43:52.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbabf723:::benchmark_data_smithi188_141483_object70743:head# 2024-04-23T16:43:52.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbac2f27:::benchmark_data_smithi188_141483_object80975:head# 2024-04-23T16:43:52.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbacc067:::benchmark_data_smithi188_141483_object5032:head# 2024-04-23T16:43:52.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbad426b:::benchmark_data_smithi188_141483_object5096:head# 2024-04-23T16:43:52.956 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.956 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbad7363:::benchmark_data_smithi188_141483_object52577:head# 2024-04-23T16:43:52.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbae2279:::benchmark_data_smithi188_141483_object60652:head# 2024-04-23T16:43:52.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbae5c24:::benchmark_data_smithi188_141483_object77140:head# 2024-04-23T16:43:52.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbaeb170:::benchmark_data_smithi188_141483_object8468:head# 2024-04-23T16:43:52.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbaed4b0:::benchmark_data_smithi188_141483_object83245:head# 2024-04-23T16:43:52.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbafb361:::benchmark_data_smithi188_141483_object80822:head# 2024-04-23T16:43:52.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbb03f06:::benchmark_data_smithi188_141483_object57336:head# 2024-04-23T16:43:52.957 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.957 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbb0f575:::benchmark_data_smithi188_141483_object50886:head# 2024-04-23T16:43:52.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbb261fe:::benchmark_data_smithi188_141483_object55593:head# 2024-04-23T16:43:52.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbb39ce2:::benchmark_data_smithi188_141483_object6713:head# 2024-04-23T16:43:52.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbb41e0b:::benchmark_data_smithi188_141483_object62202:head# 2024-04-23T16:43:52.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbb4f465:::benchmark_data_smithi188_141483_object71763:head# 2024-04-23T16:43:52.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbb4fde0:::benchmark_data_smithi188_141483_object15397:head# 2024-04-23T16:43:52.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.958 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbb5f2d7:::benchmark_data_smithi188_141483_object75506:head# 2024-04-23T16:43:52.958 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbb6e876:::benchmark_data_smithi188_141483_object68286:head# 2024-04-23T16:43:52.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbb7542f:::benchmark_data_smithi188_141483_object89014:head# 2024-04-23T16:43:52.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbb81621:::benchmark_data_smithi188_141483_object49691:head# 2024-04-23T16:43:52.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbb90655:::benchmark_data_smithi188_141483_object1651:head# 2024-04-23T16:43:52.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbb9879e:::benchmark_data_smithi188_141483_object35803:head# 2024-04-23T16:43:52.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbbb418e:::benchmark_data_smithi188_141483_object66955:head# 2024-04-23T16:43:52.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.959 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbbb469f:::benchmark_data_smithi188_141483_object65972:head# 2024-04-23T16:43:52.959 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbbc5067:::benchmark_data_smithi188_141483_object61804:head# 2024-04-23T16:43:52.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbbc74ed:::benchmark_data_smithi188_141483_object80214:head# 2024-04-23T16:43:52.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbbd9ec2:::benchmark_data_smithi188_141483_object46180:head# 2024-04-23T16:43:52.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.960 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbbf7978:::benchmark_data_smithi188_141483_object84479:head# 2024-04-23T16:43:52.960 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:52.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:52.962 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:43:52.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:43:53.065 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:43:53.142 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:53.194 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbbf7da9:::benchmark_data_smithi188_141483_object4421:head# 2024-04-23T16:43:53.194 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbc15e31:::benchmark_data_smithi188_141483_object42029:head# 2024-04-23T16:43:53.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbc1dd80:::benchmark_data_smithi188_141483_object78745:head# 2024-04-23T16:43:53.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbc2bf15:::benchmark_data_smithi188_141483_object81158:head# 2024-04-23T16:43:53.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbc30bd5:::benchmark_data_smithi188_141483_object29294:head# 2024-04-23T16:43:53.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbc4b60a:::benchmark_data_smithi188_141483_object34654:head# 2024-04-23T16:43:53.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbc4e8cf:::benchmark_data_smithi188_141483_object16676:head# 2024-04-23T16:43:53.195 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.195 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbc4f342:::benchmark_data_smithi188_141483_object36366:head# 2024-04-23T16:43:53.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbc57e54:::benchmark_data_smithi188_141483_object47711:head# 2024-04-23T16:43:53.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbc57ebd:::benchmark_data_smithi188_141483_object47566:head# 2024-04-23T16:43:53.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbc5b934:::benchmark_data_smithi188_141483_object43587:head# 2024-04-23T16:43:53.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbc8c252:::benchmark_data_smithi188_141483_object27869:head# 2024-04-23T16:43:53.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbca2ab3:::benchmark_data_smithi188_141483_object18796:head# 2024-04-23T16:43:53.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbca4b3c:::benchmark_data_smithi188_141483_object30795:head# 2024-04-23T16:43:53.196 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.196 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbcaad70:::benchmark_data_smithi188_141483_object50221:head# 2024-04-23T16:43:53.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbcb196a:::benchmark_data_smithi188_141483_object57256:head# 2024-04-23T16:43:53.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbcc3d47:::benchmark_data_smithi188_141483_object66432:head# 2024-04-23T16:43:53.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbcc50eb:::benchmark_data_smithi188_141483_object466:head# 2024-04-23T16:43:53.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbce1113:::benchmark_data_smithi188_141483_object31185:head# 2024-04-23T16:43:53.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbce3102:::benchmark_data_smithi188_141483_object42651:head# 2024-04-23T16:43:53.197 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.197 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbcfac94:::benchmark_data_smithi188_141483_object10943:head# 2024-04-23T16:43:53.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbd11b3b:::benchmark_data_smithi188_141483_object14903:head# 2024-04-23T16:43:53.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbd182d2:::benchmark_data_smithi188_141483_object65660:head# 2024-04-23T16:43:53.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbd1b0b7:::benchmark_data_smithi188_141483_object3194:head# 2024-04-23T16:43:53.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbd38391:::benchmark_data_smithi188_141483_object67704:head# 2024-04-23T16:43:53.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbd40f9c:::benchmark_data_smithi188_141483_object21314:head# 2024-04-23T16:43:53.198 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.198 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbd42cae:::benchmark_data_smithi188_141483_object65819:head# 2024-04-23T16:43:53.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbd661d1:::benchmark_data_smithi188_141483_object72230:head# 2024-04-23T16:43:53.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbd85051:::benchmark_data_smithi188_141483_object64849:head# 2024-04-23T16:43:53.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbd851f8:::benchmark_data_smithi188_141483_object26106:head# 2024-04-23T16:43:53.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbd8f363:::benchmark_data_smithi188_141483_object83518:head# 2024-04-23T16:43:53.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbd9df2e:::benchmark_data_smithi188_141483_object82064:head# 2024-04-23T16:43:53.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbda0683:::benchmark_data_smithi188_141483_object13813:head# 2024-04-23T16:43:53.199 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.199 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbdb01d7:::benchmark_data_smithi188_141483_object86585:head# 2024-04-23T16:43:53.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbdc67c1:::benchmark_data_smithi188_141483_object27657:head# 2024-04-23T16:43:53.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbdd55b2:::benchmark_data_smithi188_141483_object24403:head# 2024-04-23T16:43:53.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbddbb9c:::benchmark_data_smithi188_141483_object34666:head# 2024-04-23T16:43:53.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbddc446:::benchmark_data_smithi188_141483_object44958:head# 2024-04-23T16:43:53.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbde75ab:::benchmark_data_smithi188_141483_object55861:head# 2024-04-23T16:43:53.200 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.200 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbdfe7aa:::benchmark_data_smithi188_141483_object79938:head# 2024-04-23T16:43:53.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbe15a12:::benchmark_data_smithi188_141483_object39256:head# 2024-04-23T16:43:53.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbe3749d:::benchmark_data_smithi188_141483_object85540:head# 2024-04-23T16:43:53.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbe4130a:::benchmark_data_smithi188_141483_object73004:head# 2024-04-23T16:43:53.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbe57a61:::benchmark_data_smithi188_141483_object75922:head# 2024-04-23T16:43:53.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbe589ca:::benchmark_data_smithi188_141483_object84721:head# 2024-04-23T16:43:53.201 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.201 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbe7da8c:::benchmark_data_smithi188_141483_object38680:head# 2024-04-23T16:43:53.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbe7e7da:::benchmark_data_smithi188_141483_object11398:head# 2024-04-23T16:43:53.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbe83ae3:::benchmark_data_smithi188_141483_object5413:head# 2024-04-23T16:43:53.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbe9d565:::benchmark_data_smithi188_141483_object18044:head# 2024-04-23T16:43:53.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbe9f733:::benchmark_data_smithi188_141483_object57767:head# 2024-04-23T16:43:53.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbebba28:::benchmark_data_smithi188_141483_object47601:head# 2024-04-23T16:43:53.202 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.202 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbed135f:::benchmark_data_smithi188_141483_object90908:head# 2024-04-23T16:43:53.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbed42ce:::benchmark_data_smithi188_141483_object83895:head# 2024-04-23T16:43:53.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbedc41e:::benchmark_data_smithi188_141483_object78680:head# 2024-04-23T16:43:53.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbee5b9b:::benchmark_data_smithi188_141483_object40451:head# 2024-04-23T16:43:53.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbee8894:::benchmark_data_smithi188_141483_object42704:head# 2024-04-23T16:43:53.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbeebc7d:::benchmark_data_smithi188_141483_object3813:head# 2024-04-23T16:43:53.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbeed14d:::benchmark_data_smithi188_141483_object88531:head# 2024-04-23T16:43:53.203 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.203 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbeeda11:::benchmark_data_smithi188_141483_object26873:head# 2024-04-23T16:43:53.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbef8360:::benchmark_data_smithi188_141483_object41632:head# 2024-04-23T16:43:53.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbef900a:::benchmark_data_smithi188_141483_object14749:head# 2024-04-23T16:43:53.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbf033b5:::benchmark_data_smithi188_141483_object66855:head# 2024-04-23T16:43:53.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbf135e9:::benchmark_data_smithi188_141483_object43250:head# 2024-04-23T16:43:53.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.204 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbf15c88:::benchmark_data_smithi188_141483_object38813:head# 2024-04-23T16:43:53.204 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbf30683:::benchmark_data_smithi188_141483_object72767:head# 2024-04-23T16:43:53.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbf38dd1:::benchmark_data_smithi188_141483_object37799:head# 2024-04-23T16:43:53.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbf3a708:::benchmark_data_smithi188_141483_object38963:head# 2024-04-23T16:43:53.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbf3e31b:::benchmark_data_smithi188_141483_object27696:head# 2024-04-23T16:43:53.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbf51c33:::benchmark_data_smithi188_141483_object54018:head# 2024-04-23T16:43:53.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbf594db:::benchmark_data_smithi188_141483_object47022:head# 2024-04-23T16:43:53.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbf6c072:::benchmark_data_smithi188_141483_object77161:head# 2024-04-23T16:43:53.205 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.205 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbf6c921:::benchmark_data_smithi188_141483_object35396:head# 2024-04-23T16:43:53.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbf803fb:::benchmark_data_smithi188_141483_object17397:head# 2024-04-23T16:43:53.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbf84aa3:::benchmark_data_smithi188_141483_object67012:head# 2024-04-23T16:43:53.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbf84fcc:::benchmark_data_smithi188_141483_object245:head# 2024-04-23T16:43:53.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbf857c6:::benchmark_data_smithi188_141483_object61956:head# 2024-04-23T16:43:53.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbf99b72:::benchmark_data_smithi188_141483_object70206:head# 2024-04-23T16:43:53.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbfb2929:::benchmark_data_smithi188_141483_object9436:head# 2024-04-23T16:43:53.206 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.206 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbfb40ae:::benchmark_data_smithi188_141483_object80773:head# 2024-04-23T16:43:53.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbfba78a:::benchmark_data_smithi188_141483_object38863:head# 2024-04-23T16:43:53.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbfc92bc:::benchmark_data_smithi188_141483_object3523:head# 2024-04-23T16:43:53.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbfd9d6d:::benchmark_data_smithi188_141483_object33875:head# 2024-04-23T16:43:53.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbff88d6:::benchmark_data_smithi188_141483_object68520:head# 2024-04-23T16:43:53.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cbffa79f:::benchmark_data_smithi188_141483_object4946:head# 2024-04-23T16:43:53.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.207 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc000043:::benchmark_data_smithi188_141483_object69992:head# 2024-04-23T16:43:53.207 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc00d388:::benchmark_data_smithi188_141483_object44739:head# 2024-04-23T16:43:53.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc035781:::benchmark_data_smithi188_141483_object17789:head# 2024-04-23T16:43:53.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc03779f:::benchmark_data_smithi188_141483_object15935:head# 2024-04-23T16:43:53.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc0394fd:::benchmark_data_smithi188_141483_object42343:head# 2024-04-23T16:43:53.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc046573:::benchmark_data_smithi188_141483_object59304:head# 2024-04-23T16:43:53.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc04c559:::benchmark_data_smithi188_141483_object63166:head# 2024-04-23T16:43:53.208 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.208 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc063a13:::benchmark_data_smithi188_141483_object70392:head# 2024-04-23T16:43:53.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc06d210:::benchmark_data_smithi188_141483_object15527:head# 2024-04-23T16:43:53.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc06f056:::benchmark_data_smithi188_141483_object80640:head# 2024-04-23T16:43:53.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc080489:::benchmark_data_smithi188_141483_object82659:head# 2024-04-23T16:43:53.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc0a605d:::benchmark_data_smithi188_141483_object23550:head# 2024-04-23T16:43:53.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc0b67be:::benchmark_data_smithi188_141483_object27085:head# 2024-04-23T16:43:53.209 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.209 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc0bcd98:::benchmark_data_smithi188_141483_object15240:head# 2024-04-23T16:43:53.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc0c9364:::benchmark_data_smithi188_141483_object66198:head# 2024-04-23T16:43:53.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc0dbebc:::benchmark_data_smithi188_141483_object44095:head# 2024-04-23T16:43:53.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc0e83f6:::benchmark_data_smithi188_141483_object18794:head# 2024-04-23T16:43:53.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc0fa3e2:::benchmark_data_smithi188_141483_object36707:head# 2024-04-23T16:43:53.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc0ff34d:::benchmark_data_smithi188_141483_object23627:head# 2024-04-23T16:43:53.210 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.210 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc108cd7:::benchmark_data_smithi188_141483_object85463:head# 2024-04-23T16:43:53.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc149edd:::benchmark_data_smithi188_141483_object46493:head# 2024-04-23T16:43:53.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc159ef1:::benchmark_data_smithi188_141483_object79299:head# 2024-04-23T16:43:53.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc15b167:::benchmark_data_smithi188_141483_object5024:head# 2024-04-23T16:43:53.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc15d948:::benchmark_data_smithi188_141483_object5578:head# 2024-04-23T16:43:53.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc15daca:::benchmark_data_smithi188_141483_object37373:head# 2024-04-23T16:43:53.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.211 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc16184c:::benchmark_data_smithi188_141483_object34267:head# 2024-04-23T16:43:53.211 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc165d75:::benchmark_data_smithi188_141483_object56847:head# 2024-04-23T16:43:53.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc177093:::benchmark_data_smithi188_141483_object84369:head# 2024-04-23T16:43:53.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc178ac3:::benchmark_data_smithi188_141483_object16192:head# 2024-04-23T16:43:53.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc1839eb:::benchmark_data_smithi188_141483_object32792:head# 2024-04-23T16:43:53.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc18d99b:::benchmark_data_smithi188_141483_object22947:head# 2024-04-23T16:43:53.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc18df96:::benchmark_data_smithi188_141483_object30797:head# 2024-04-23T16:43:53.212 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.212 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc18f464:::benchmark_data_smithi188_141483_object92:head# 2024-04-23T16:43:53.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc190215:::benchmark_data_smithi188_141483_object23724:head# 2024-04-23T16:43:53.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc1a9da2:::benchmark_data_smithi188_141483_object27197:head# 2024-04-23T16:43:53.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc1b28e6:::benchmark_data_smithi188_141483_object64568:head# 2024-04-23T16:43:53.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc1c9b45:::benchmark_data_smithi188_141483_object28634:head# 2024-04-23T16:43:53.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc1ce218:::benchmark_data_smithi188_141483_object18546:head# 2024-04-23T16:43:53.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc1db456:::benchmark_data_smithi188_141483_object88212:head# 2024-04-23T16:43:53.213 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.213 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc1e1ddc:::benchmark_data_smithi188_141483_object61224:head# 2024-04-23T16:43:53.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc1e8b4b:::benchmark_data_smithi188_141483_object69875:head# 2024-04-23T16:43:53.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc1f35f0:::benchmark_data_smithi188_141483_object89189:head# 2024-04-23T16:43:53.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc1ffa0c:::benchmark_data_smithi188_141483_object48800:head# 2024-04-23T16:43:53.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc218a24:::benchmark_data_smithi188_141483_object86442:head# 2024-04-23T16:43:53.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc218bb5:::benchmark_data_smithi188_141483_object39248:head# 2024-04-23T16:43:53.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.214 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc223189:::benchmark_data_smithi188_141483_object53252:head# 2024-04-23T16:43:53.214 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc23c169:::benchmark_data_smithi188_141483_object14408:head# 2024-04-23T16:43:53.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc23e332:::benchmark_data_smithi188_141483_object75362:head# 2024-04-23T16:43:53.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc240654:::benchmark_data_smithi188_141483_object56345:head# 2024-04-23T16:43:53.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc250b4c:::benchmark_data_smithi188_141483_object59639:head# 2024-04-23T16:43:53.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc251f80:::benchmark_data_smithi188_141483_object60098:head# 2024-04-23T16:43:53.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc268d13:::benchmark_data_smithi188_141483_object30862:head# 2024-04-23T16:43:53.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.215 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc26e621:::benchmark_data_smithi188_141483_object36532:head# 2024-04-23T16:43:53.215 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc2780ee:::benchmark_data_smithi188_141483_object49666:head# 2024-04-23T16:43:53.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc279247:::benchmark_data_smithi188_141483_object10223:head# 2024-04-23T16:43:53.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc27aefd:::benchmark_data_smithi188_141483_object17592:head# 2024-04-23T16:43:53.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc28831a:::benchmark_data_smithi188_141483_object22962:head# 2024-04-23T16:43:53.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc2a13dc:::benchmark_data_smithi188_141483_object61692:head# 2024-04-23T16:43:53.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc2c7d64:::benchmark_data_smithi188_141483_object72173:head# 2024-04-23T16:43:53.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.216 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc2c971b:::benchmark_data_smithi188_141483_object58900:head# 2024-04-23T16:43:53.216 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc2e99b7:::benchmark_data_smithi188_141483_object10806:head# 2024-04-23T16:43:53.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc2f21d0:::benchmark_data_smithi188_141483_object36164:head# 2024-04-23T16:43:53.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc30320d:::benchmark_data_smithi188_141483_object41772:head# 2024-04-23T16:43:53.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc309816:::benchmark_data_smithi188_141483_object22082:head# 2024-04-23T16:43:53.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc337673:::benchmark_data_smithi188_141483_object68486:head# 2024-04-23T16:43:53.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc34072f:::benchmark_data_smithi188_141483_object27340:head# 2024-04-23T16:43:53.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.217 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc35dae0:::benchmark_data_smithi188_141483_object9621:head# 2024-04-23T16:43:53.217 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc368a25:::benchmark_data_smithi188_141483_object21167:head# 2024-04-23T16:43:53.218 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc397dc2:::benchmark_data_smithi188_141483_object78826:head# 2024-04-23T16:43:53.218 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc39ef48:::benchmark_data_smithi188_141483_object76842:head# 2024-04-23T16:43:53.218 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc3a2cb3:::benchmark_data_smithi188_141483_object58018:head# 2024-04-23T16:43:53.218 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc3ab5a2:::benchmark_data_smithi188_141483_object40526:head# 2024-04-23T16:43:53.218 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.218 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc3ba101:::benchmark_data_smithi188_141483_object32615:head# 2024-04-23T16:43:53.218 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc3c8521:::benchmark_data_smithi188_141483_object32747:head# 2024-04-23T16:43:53.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc3cc6f6:::benchmark_data_smithi188_141483_object20072:head# 2024-04-23T16:43:53.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc3eb343:::benchmark_data_smithi188_141483_object13821:head# 2024-04-23T16:43:53.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc3edb3e:::benchmark_data_smithi188_141483_object17308:head# 2024-04-23T16:43:53.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc3fe47d:::benchmark_data_smithi188_141483_object32937:head# 2024-04-23T16:43:53.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc407f07:::benchmark_data_smithi188_141483_object81251:head# 2024-04-23T16:43:53.219 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.219 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc416bcd:::benchmark_data_smithi188_141483_object14780:head# 2024-04-23T16:43:53.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc41aad9:::benchmark_data_smithi188_141483_object7427:head# 2024-04-23T16:43:53.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc41c919:::benchmark_data_smithi188_141483_object27994:head# 2024-04-23T16:43:53.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc41d97b:::benchmark_data_smithi188_141483_object5993:head# 2024-04-23T16:43:53.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc42ffe8:::benchmark_data_smithi188_141483_object56187:head# 2024-04-23T16:43:53.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc45cc0c:::benchmark_data_smithi188_141483_object25946:head# 2024-04-23T16:43:53.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc465510:::benchmark_data_smithi188_141483_object89163:head# 2024-04-23T16:43:53.220 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.220 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc46772e:::benchmark_data_smithi188_141483_object9:head# 2024-04-23T16:43:53.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc46cc9a:::benchmark_data_smithi188_141483_object39516:head# 2024-04-23T16:43:53.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc481a84:::benchmark_data_smithi188_141483_object75766:head# 2024-04-23T16:43:53.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc489b47:::benchmark_data_smithi188_141483_object24426:head# 2024-04-23T16:43:53.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc48c69f:::benchmark_data_smithi188_141483_object45990:head# 2024-04-23T16:43:53.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc48ddaf:::benchmark_data_smithi188_141483_object56112:head# 2024-04-23T16:43:53.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc48eb18:::benchmark_data_smithi188_141483_object28474:head# 2024-04-23T16:43:53.221 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.221 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc4a6308:::benchmark_data_smithi188_141483_object17624:head# 2024-04-23T16:43:53.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc4ab0b9:::benchmark_data_smithi188_141483_object6070:head# 2024-04-23T16:43:53.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc4b5fd4:::benchmark_data_smithi188_141483_object84122:head# 2024-04-23T16:43:53.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc4b8d4f:::benchmark_data_smithi188_141483_object57054:head# 2024-04-23T16:43:53.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc4bf8f0:::benchmark_data_smithi188_141483_object26094:head# 2024-04-23T16:43:53.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc4cc7bd:::benchmark_data_smithi188_141483_object51859:head# 2024-04-23T16:43:53.222 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.222 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc4e1166:::benchmark_data_smithi188_141483_object54940:head# 2024-04-23T16:43:53.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc4e4347:::benchmark_data_smithi188_141483_object84134:head# 2024-04-23T16:43:53.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc4f076c:::benchmark_data_smithi188_141483_object35542:head# 2024-04-23T16:43:53.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc507305:::benchmark_data_smithi188_141483_object75678:head# 2024-04-23T16:43:53.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc5232e6:::benchmark_data_smithi188_141483_object51984:head# 2024-04-23T16:43:53.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc525720:::benchmark_data_smithi188_141483_object66452:head# 2024-04-23T16:43:53.223 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.223 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc525adc:::benchmark_data_smithi188_141483_object29932:head# 2024-04-23T16:43:53.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc525d60:::benchmark_data_smithi188_141483_object47150:head# 2024-04-23T16:43:53.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc52d5ba:::benchmark_data_smithi188_141483_object1595:head# 2024-04-23T16:43:53.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc52e987:::benchmark_data_smithi188_141483_object80434:head# 2024-04-23T16:43:53.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc53a9ab:::benchmark_data_smithi188_141483_object11264:head# 2024-04-23T16:43:53.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.224 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc5697a8:::benchmark_data_smithi188_141483_object45174:head# 2024-04-23T16:43:53.224 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc586103:::benchmark_data_smithi188_141483_object76858:head# 2024-04-23T16:43:53.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc5934ca:::benchmark_data_smithi188_141483_object76544:head# 2024-04-23T16:43:53.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc5be4d2:::benchmark_data_smithi188_141483_object5198:head# 2024-04-23T16:43:53.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc5c1bc0:::benchmark_data_smithi188_141483_object33371:head# 2024-04-23T16:43:53.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc5c4038:::benchmark_data_smithi188_141483_object72443:head# 2024-04-23T16:43:53.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc5cbbcb:::benchmark_data_smithi188_141483_object40834:head# 2024-04-23T16:43:53.225 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.225 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc5d7252:::benchmark_data_smithi188_141483_object20570:head# 2024-04-23T16:43:53.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc5e3f79:::benchmark_data_smithi188_141483_object84123:head# 2024-04-23T16:43:53.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc5ef56c:::benchmark_data_smithi188_141483_object43403:head# 2024-04-23T16:43:53.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc5f1cfb:::benchmark_data_smithi188_141483_object28663:head# 2024-04-23T16:43:53.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc5fea70:::benchmark_data_smithi188_141483_object25555:head# 2024-04-23T16:43:53.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc619780:::benchmark_data_smithi188_141483_object15460:head# 2024-04-23T16:43:53.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.226 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc627a01:::benchmark_data_smithi188_141483_object41891:head# 2024-04-23T16:43:53.226 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc628290:::benchmark_data_smithi188_141483_object35374:head# 2024-04-23T16:43:53.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc631e16:::benchmark_data_smithi188_141483_object23363:head# 2024-04-23T16:43:53.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc643251:::benchmark_data_smithi188_141483_object28090:head# 2024-04-23T16:43:53.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc64b94e:::benchmark_data_smithi188_141483_object58198:head# 2024-04-23T16:43:53.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc64f091:::benchmark_data_smithi188_141483_object79720:head# 2024-04-23T16:43:53.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc67b773:::benchmark_data_smithi188_141483_object52345:head# 2024-04-23T16:43:53.227 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.227 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc68bd34:::benchmark_data_smithi188_141483_object50556:head# 2024-04-23T16:43:53.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc68c104:::benchmark_data_smithi188_141483_object47094:head# 2024-04-23T16:43:53.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc693886:::benchmark_data_smithi188_141483_object49794:head# 2024-04-23T16:43:53.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc6e23c1:::benchmark_data_smithi188_141483_object84656:head# 2024-04-23T16:43:53.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc6f4d29:::benchmark_data_smithi188_141483_object10166:head# 2024-04-23T16:43:53.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.228 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc70c5b3:::benchmark_data_smithi188_141483_object44479:head# 2024-04-23T16:43:53.228 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc718ea7:::benchmark_data_smithi188_141483_object17609:head# 2024-04-23T16:43:53.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc71be99:::benchmark_data_smithi188_141483_object86109:head# 2024-04-23T16:43:53.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc71bebf:::benchmark_data_smithi188_141483_object48146:head# 2024-04-23T16:43:53.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc723aed:::benchmark_data_smithi188_141483_object39972:head# 2024-04-23T16:43:53.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc7289a8:::benchmark_data_smithi188_141483_object83402:head# 2024-04-23T16:43:53.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc72ad9a:::benchmark_data_smithi188_141483_object45753:head# 2024-04-23T16:43:53.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.229 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc72d5fc:::benchmark_data_smithi188_141483_object81136:head# 2024-04-23T16:43:53.229 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc74d1a5:::benchmark_data_smithi188_141483_object89524:head# 2024-04-23T16:43:53.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc7602c1:::benchmark_data_smithi188_141483_object46760:head# 2024-04-23T16:43:53.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc7658a1:::benchmark_data_smithi188_141483_object11246:head# 2024-04-23T16:43:53.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc785628:::benchmark_data_smithi188_141483_object85921:head# 2024-04-23T16:43:53.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc78caaa:::benchmark_data_smithi188_141483_object37738:head# 2024-04-23T16:43:53.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc7aad5e:::benchmark_data_smithi188_141483_object45367:head# 2024-04-23T16:43:53.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.230 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc7b95d0:::benchmark_data_smithi188_141483_object66819:head# 2024-04-23T16:43:53.230 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc7b994d:::benchmark_data_smithi188_141483_object75692:head# 2024-04-23T16:43:53.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc7c34b3:::benchmark_data_smithi188_141483_object88562:head# 2024-04-23T16:43:53.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc7cd0fd:::benchmark_data_smithi188_141483_object19218:head# 2024-04-23T16:43:53.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc7e8fa4:::benchmark_data_smithi188_141483_object45531:head# 2024-04-23T16:43:53.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc7f9ff2:::benchmark_data_smithi188_141483_object21748:head# 2024-04-23T16:43:53.231 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.231 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc808eb7:::benchmark_data_smithi188_141483_object80472:head# 2024-04-23T16:43:53.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc80bde1:::benchmark_data_smithi188_141483_object34956:head# 2024-04-23T16:43:53.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc811d70:::benchmark_data_smithi188_141483_object72824:head# 2024-04-23T16:43:53.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc84e712:::benchmark_data_smithi188_141483_object72174:head# 2024-04-23T16:43:53.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc84fc71:::benchmark_data_smithi188_141483_object74660:head# 2024-04-23T16:43:53.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.232 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc8590a6:::benchmark_data_smithi188_141483_object27865:head# 2024-04-23T16:43:53.232 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc861e12:::benchmark_data_smithi188_141483_object39723:head# 2024-04-23T16:43:53.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc863ee0:::benchmark_data_smithi188_141483_object7980:head# 2024-04-23T16:43:53.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc87156b:::benchmark_data_smithi188_141483_object11474:head# 2024-04-23T16:43:53.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc873c6a:::benchmark_data_smithi188_141483_object46173:head# 2024-04-23T16:43:53.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc88d6c5:::benchmark_data_smithi188_141483_object57146:head# 2024-04-23T16:43:53.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc89181a:::benchmark_data_smithi188_141483_object82362:head# 2024-04-23T16:43:53.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.233 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc89b1f0:::benchmark_data_smithi188_141483_object21944:head# 2024-04-23T16:43:53.233 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc8a1fd2:::benchmark_data_smithi188_141483_object71315:head# 2024-04-23T16:43:53.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc8aea60:::benchmark_data_smithi188_141483_object59430:head# 2024-04-23T16:43:53.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc8af06e:::benchmark_data_smithi188_141483_object28301:head# 2024-04-23T16:43:53.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc8d1290:::benchmark_data_smithi188_141483_object5654:head# 2024-04-23T16:43:53.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc8e18f0:::benchmark_data_smithi188_141483_object4594:head# 2024-04-23T16:43:53.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc8e3215:::benchmark_data_smithi188_141483_object17361:head# 2024-04-23T16:43:53.234 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.234 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc8f0357:::benchmark_data_smithi188_141483_object32717:head# 2024-04-23T16:43:53.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc90eb74:::benchmark_data_smithi188_141483_object61448:head# 2024-04-23T16:43:53.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc92259c:::benchmark_data_smithi188_141483_object5912:head# 2024-04-23T16:43:53.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc9232cc:::benchmark_data_smithi188_141483_object80327:head# 2024-04-23T16:43:53.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc939047:::benchmark_data_smithi188_141483_object10572:head# 2024-04-23T16:43:53.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc93ee07:::benchmark_data_smithi188_141483_object87365:head# 2024-04-23T16:43:53.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.235 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc94465f:::benchmark_data_smithi188_141483_object38716:head# 2024-04-23T16:43:53.235 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc951f60:::benchmark_data_smithi188_141483_object63850:head# 2024-04-23T16:43:53.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc96075c:::benchmark_data_smithi188_141483_object90839:head# 2024-04-23T16:43:53.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc9621c8:::benchmark_data_smithi188_141483_object54868:head# 2024-04-23T16:43:53.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc962430:::benchmark_data_smithi188_141483_object13005:head# 2024-04-23T16:43:53.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc9671df:::benchmark_data_smithi188_141483_object45434:head# 2024-04-23T16:43:53.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc977a6f:::benchmark_data_smithi188_141483_object46902:head# 2024-04-23T16:43:53.236 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.236 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc97f1f3:::benchmark_data_smithi188_141483_object36459:head# 2024-04-23T16:43:53.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc98dbe3:::benchmark_data_smithi188_141483_object51521:head# 2024-04-23T16:43:53.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc99fc74:::benchmark_data_smithi188_141483_object60480:head# 2024-04-23T16:43:53.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc9a0ab4:::benchmark_data_smithi188_141483_object51197:head# 2024-04-23T16:43:53.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc9a2b1c:::benchmark_data_smithi188_141483_object41098:head# 2024-04-23T16:43:53.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc9a5c09:::benchmark_data_smithi188_141483_object62703:head# 2024-04-23T16:43:53.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc9b8da5:::benchmark_data_smithi188_141483_object1018:head# 2024-04-23T16:43:53.237 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.237 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc9c71e4:::benchmark_data_smithi188_141483_object69989:head# 2024-04-23T16:43:53.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc9c8ca5:::benchmark_data_smithi188_141483_object53457:head# 2024-04-23T16:43:53.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc9d21bc:::benchmark_data_smithi188_141483_object22860:head# 2024-04-23T16:43:53.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc9d5a70:::benchmark_data_smithi188_141483_object10289:head# 2024-04-23T16:43:53.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc9f6a50:::benchmark_data_smithi188_141483_object22725:head# 2024-04-23T16:43:53.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cc9fe68e:::benchmark_data_smithi188_141483_object67386:head# 2024-04-23T16:43:53.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cca08b2c:::benchmark_data_smithi188_141483_object80120:head# 2024-04-23T16:43:53.238 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.238 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cca09af5:::benchmark_data_smithi188_141483_object66157:head# 2024-04-23T16:43:53.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cca0a9f6:::benchmark_data_smithi188_141483_object24024:head# 2024-04-23T16:43:53.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cca0de95:::benchmark_data_smithi188_141483_object9072:head# 2024-04-23T16:43:53.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cca1100c:::benchmark_data_smithi188_141483_object38104:head# 2024-04-23T16:43:53.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cca14c8b:::benchmark_data_smithi188_141483_object60184:head# 2024-04-23T16:43:53.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.239 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cca167d0:::benchmark_data_smithi188_141483_object30330:head# 2024-04-23T16:43:53.239 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cca1ed24:::benchmark_data_smithi188_141483_object31391:head# 2024-04-23T16:43:53.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cca2ee1f:::benchmark_data_smithi188_141483_object47783:head# 2024-04-23T16:43:53.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cca3a75d:::benchmark_data_smithi188_141483_object61967:head# 2024-04-23T16:43:53.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cca4a5a8:::benchmark_data_smithi188_141483_object49139:head# 2024-04-23T16:43:53.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cca521f8:::benchmark_data_smithi188_141483_object8148:head# 2024-04-23T16:43:53.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cca5abd4:::benchmark_data_smithi188_141483_object79459:head# 2024-04-23T16:43:53.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.240 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cca766d7:::benchmark_data_smithi188_141483_object51306:head# 2024-04-23T16:43:53.240 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cca7fbdf:::benchmark_data_smithi188_141483_object65380:head# 2024-04-23T16:43:53.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cca82aee:::benchmark_data_smithi188_141483_object27798:head# 2024-04-23T16:43:53.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cca9554f:::benchmark_data_smithi188_141483_object47263:head# 2024-04-23T16:43:53.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccac91f0:::benchmark_data_smithi188_141483_object1893:head# 2024-04-23T16:43:53.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccacc758:::benchmark_data_smithi188_141483_object21263:head# 2024-04-23T16:43:53.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccacc9e3:::benchmark_data_smithi188_141483_object35914:head# 2024-04-23T16:43:53.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.241 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccad3a7c:::benchmark_data_smithi188_141483_object86548:head# 2024-04-23T16:43:53.241 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccad521e:::benchmark_data_smithi188_141483_object19006:head# 2024-04-23T16:43:53.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccad7aa0:::benchmark_data_smithi188_141483_object19425:head# 2024-04-23T16:43:53.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccad9623:::benchmark_data_smithi188_141483_object43001:head# 2024-04-23T16:43:53.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccadcc31:::benchmark_data_smithi188_141483_object79023:head# 2024-04-23T16:43:53.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccb150c3:::benchmark_data_smithi188_141483_object60653:head# 2024-04-23T16:43:53.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccb1c9cd:::benchmark_data_smithi188_141483_object47549:head# 2024-04-23T16:43:53.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.242 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccb200fe:::benchmark_data_smithi188_141483_object41287:head# 2024-04-23T16:43:53.242 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccb22639:::benchmark_data_smithi188_141483_object14680:head# 2024-04-23T16:43:53.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccb357ca:::benchmark_data_smithi188_141483_object88867:head# 2024-04-23T16:43:53.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccb427dc:::benchmark_data_smithi188_141483_object5525:head# 2024-04-23T16:43:53.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccb4bd48:::benchmark_data_smithi188_141483_object41795:head# 2024-04-23T16:43:53.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccb5706b:::benchmark_data_smithi188_141483_object71480:head# 2024-04-23T16:43:53.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccb67e50:::benchmark_data_smithi188_141483_object9675:head# 2024-04-23T16:43:53.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.243 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccb7594f:::benchmark_data_smithi188_141483_object28167:head# 2024-04-23T16:43:53.243 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccb76f6b:::benchmark_data_smithi188_141483_object24803:head# 2024-04-23T16:43:53.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccb7b578:::benchmark_data_smithi188_141483_object86931:head# 2024-04-23T16:43:53.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccb84380:::benchmark_data_smithi188_141483_object89907:head# 2024-04-23T16:43:53.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccb86e00:::benchmark_data_smithi188_141483_object45541:head# 2024-04-23T16:43:53.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccb98dcb:::benchmark_data_smithi188_141483_object61946:head# 2024-04-23T16:43:53.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccb99a20:::benchmark_data_smithi188_141483_object40616:head# 2024-04-23T16:43:53.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.244 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccb99b05:::benchmark_data_smithi188_141483_object9817:head# 2024-04-23T16:43:53.244 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccbbd337:::benchmark_data_smithi188_141483_object2442:head# 2024-04-23T16:43:53.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccbbdaed:::benchmark_data_smithi188_141483_object54268:head# 2024-04-23T16:43:53.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccbc1c79:::benchmark_data_smithi188_141483_object38031:head# 2024-04-23T16:43:53.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccbc4e50:::benchmark_data_smithi188_141483_object67138:head# 2024-04-23T16:43:53.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccbc6d84:::benchmark_data_smithi188_141483_object54912:head# 2024-04-23T16:43:53.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccbc6d9d:::benchmark_data_smithi188_141483_object75634:head# 2024-04-23T16:43:53.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.245 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccbd3223:::benchmark_data_smithi188_141483_object20961:head# 2024-04-23T16:43:53.245 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccbedfff:::benchmark_data_smithi188_141483_object85541:head# 2024-04-23T16:43:53.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccc129b1:::benchmark_data_smithi188_141483_object6700:head# 2024-04-23T16:43:53.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccc1a3da:::benchmark_data_smithi188_141483_object88237:head# 2024-04-23T16:43:53.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccc29613:::benchmark_data_smithi188_141483_object81330:head# 2024-04-23T16:43:53.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccc389c3:::benchmark_data_smithi188_141483_object90750:head# 2024-04-23T16:43:53.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccc5f035:::benchmark_data_smithi188_141483_object14031:head# 2024-04-23T16:43:53.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.246 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccc6dce6:::benchmark_data_smithi188_141483_object69965:head# 2024-04-23T16:43:53.246 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccc8e559:::benchmark_data_smithi188_141483_object69569:head# 2024-04-23T16:43:53.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccc9c19c:::benchmark_data_smithi188_141483_object25642:head# 2024-04-23T16:43:53.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccc9edc4:::benchmark_data_smithi188_141483_object400:head# 2024-04-23T16:43:53.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cccacf13:::benchmark_data_smithi188_141483_object3181:head# 2024-04-23T16:43:53.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cccb2434:::benchmark_data_smithi188_141483_object72749:head# 2024-04-23T16:43:53.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cccb8a0d:::benchmark_data_smithi188_141483_object19159:head# 2024-04-23T16:43:53.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccccb203:::benchmark_data_smithi188_141483_object15958:head# 2024-04-23T16:43:53.247 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.247 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccccb94d:::benchmark_data_smithi188_141483_object43204:head# 2024-04-23T16:43:53.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cccd01f1:::benchmark_data_smithi188_141483_object42357:head# 2024-04-23T16:43:53.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cccd6936:::benchmark_data_smithi188_141483_object62227:head# 2024-04-23T16:43:53.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cccd7c76:::benchmark_data_smithi188_141483_object28623:head# 2024-04-23T16:43:53.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccce0f4a:::benchmark_data_smithi188_141483_object40241:head# 2024-04-23T16:43:53.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccce1d60:::benchmark_data_smithi188_141483_object60643:head# 2024-04-23T16:43:53.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.248 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccce5b00:::benchmark_data_smithi188_141483_object69939:head# 2024-04-23T16:43:53.248 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccce8f8f:::benchmark_data_smithi188_141483_object86771:head# 2024-04-23T16:43:53.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccd0aa20:::benchmark_data_smithi188_141483_object2635:head# 2024-04-23T16:43:53.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccd167d2:::benchmark_data_smithi188_141483_object45008:head# 2024-04-23T16:43:53.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccd20f83:::benchmark_data_smithi188_141483_object8402:head# 2024-04-23T16:43:53.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccd24501:::benchmark_data_smithi188_141483_object12633:head# 2024-04-23T16:43:53.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccd266c3:::benchmark_data_smithi188_141483_object87828:head# 2024-04-23T16:43:53.249 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.249 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccd2a87a:::benchmark_data_smithi188_141483_object47399:head# 2024-04-23T16:43:53.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccd3498c:::benchmark_data_smithi188_141483_object341:head# 2024-04-23T16:43:53.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccd3d5ea:::benchmark_data_smithi188_141483_object89355:head# 2024-04-23T16:43:53.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccd58b56:::benchmark_data_smithi188_141483_object89693:head# 2024-04-23T16:43:53.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccd69bc0:::benchmark_data_smithi188_141483_object39987:head# 2024-04-23T16:43:53.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccd6f369:::benchmark_data_smithi188_141483_object17602:head# 2024-04-23T16:43:53.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccd76633:::benchmark_data_smithi188_141483_object17268:head# 2024-04-23T16:43:53.250 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.250 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccd7a0c7:::benchmark_data_smithi188_141483_object47671:head# 2024-04-23T16:43:53.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccd8506c:::benchmark_data_smithi188_141483_object7689:head# 2024-04-23T16:43:53.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccda3729:::benchmark_data_smithi188_141483_object56949:head# 2024-04-23T16:43:53.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccda86e3:::benchmark_data_smithi188_141483_object78920:head# 2024-04-23T16:43:53.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccdaccd7:::benchmark_data_smithi188_141483_object75213:head# 2024-04-23T16:43:53.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccdbabff:::benchmark_data_smithi188_141483_object6163:head# 2024-04-23T16:43:53.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.251 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccdbe64e:::benchmark_data_smithi188_141483_object47191:head# 2024-04-23T16:43:53.251 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccddc03b:::benchmark_data_smithi188_141483_object60103:head# 2024-04-23T16:43:53.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccde0318:::benchmark_data_smithi188_141483_object73154:head# 2024-04-23T16:43:53.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccdeb878:::benchmark_data_smithi188_141483_object1246:head# 2024-04-23T16:43:53.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccdffd7b:::benchmark_data_smithi188_141483_object64594:head# 2024-04-23T16:43:53.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cce003b0:::benchmark_data_smithi188_141483_object43033:head# 2024-04-23T16:43:53.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cce0b005:::benchmark_data_smithi188_141483_object60041:head# 2024-04-23T16:43:53.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.252 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cce10bc8:::benchmark_data_smithi188_141483_object45442:head# 2024-04-23T16:43:53.252 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cce37508:::benchmark_data_smithi188_141483_object9949:head# 2024-04-23T16:43:53.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cce49691:::benchmark_data_smithi188_141483_object33276:head# 2024-04-23T16:43:53.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cce4be7f:::benchmark_data_smithi188_141483_object63500:head# 2024-04-23T16:43:53.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cce627ba:::benchmark_data_smithi188_141483_object41469:head# 2024-04-23T16:43:53.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cce82105:::benchmark_data_smithi188_141483_object15519:head# 2024-04-23T16:43:53.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.253 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cce93280:::benchmark_data_smithi188_141483_object7090:head# 2024-04-23T16:43:53.253 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cce9b3b0:::benchmark_data_smithi188_141483_object33076:head# 2024-04-23T16:43:53.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccea0193:::benchmark_data_smithi188_141483_object80968:head# 2024-04-23T16:43:53.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cceb77b1:::benchmark_data_smithi188_141483_object55970:head# 2024-04-23T16:43:53.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cceb9101:::benchmark_data_smithi188_141483_object30616:head# 2024-04-23T16:43:53.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccee7ef6:::benchmark_data_smithi188_141483_object54881:head# 2024-04-23T16:43:53.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.254 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccef665d:::benchmark_data_smithi188_141483_object65442:head# 2024-04-23T16:43:53.254 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccefa7d3:::benchmark_data_smithi188_141483_object69481:head# 2024-04-23T16:43:53.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccf02d51:::benchmark_data_smithi188_141483_object56098:head# 2024-04-23T16:43:53.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccf0bf68:::benchmark_data_smithi188_141483_object66896:head# 2024-04-23T16:43:53.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccf11cf3:::benchmark_data_smithi188_141483_object78355:head# 2024-04-23T16:43:53.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccf16214:::benchmark_data_smithi188_141483_object81093:head# 2024-04-23T16:43:53.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.255 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccf1ae0a:::benchmark_data_smithi188_141483_object32522:head# 2024-04-23T16:43:53.255 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccf24132:::benchmark_data_smithi188_141483_object76349:head# 2024-04-23T16:43:53.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccf25d26:::benchmark_data_smithi188_141483_object69895:head# 2024-04-23T16:43:53.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccf3000e:::benchmark_data_smithi188_141483_object23426:head# 2024-04-23T16:43:53.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccf3d657:::benchmark_data_smithi188_141483_object49114:head# 2024-04-23T16:43:53.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccf43bc6:::benchmark_data_smithi188_141483_object21305:head# 2024-04-23T16:43:53.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccf4ee38:::benchmark_data_smithi188_141483_object29461:head# 2024-04-23T16:43:53.256 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.256 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccf5cce4:::benchmark_data_smithi188_141483_object32291:head# 2024-04-23T16:43:53.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccf5e90d:::benchmark_data_smithi188_141483_object8851:head# 2024-04-23T16:43:53.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccf61f4f:::benchmark_data_smithi188_141483_object40355:head# 2024-04-23T16:43:53.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccf67e4f:::benchmark_data_smithi188_141483_object28867:head# 2024-04-23T16:43:53.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccf6d6bb:::benchmark_data_smithi188_141483_object13784:head# 2024-04-23T16:43:53.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccf99b78:::benchmark_data_smithi188_141483_object70947:head# 2024-04-23T16:43:53.257 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.257 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccfa187f:::benchmark_data_smithi188_141483_object25143:head# 2024-04-23T16:43:53.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccfb5a1e:::benchmark_data_smithi188_141483_object24373:head# 2024-04-23T16:43:53.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccfb9948:::benchmark_data_smithi188_141483_object9150:head# 2024-04-23T16:43:53.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccfbc4fc:::benchmark_data_smithi188_141483_object89555:head# 2024-04-23T16:43:53.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccfd158f:::benchmark_data_smithi188_141483_object60600:head# 2024-04-23T16:43:53.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccfe0b0d:::benchmark_data_smithi188_141483_object26711:head# 2024-04-23T16:43:53.258 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.258 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccfe2c7c:::benchmark_data_smithi188_141483_object75156:head# 2024-04-23T16:43:53.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccfe2cf3:::benchmark_data_smithi188_141483_object63884:head# 2024-04-23T16:43:53.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccfe73f8:::benchmark_data_smithi188_141483_object90207:head# 2024-04-23T16:43:53.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ccff302a:::benchmark_data_smithi188_141483_object69139:head# 2024-04-23T16:43:53.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd006fac:::benchmark_data_smithi188_141483_object55554:head# 2024-04-23T16:43:53.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd00ee04:::benchmark_data_smithi188_141483_object12966:head# 2024-04-23T16:43:53.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.259 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd013edb:::benchmark_data_smithi188_141483_object34221:head# 2024-04-23T16:43:53.259 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd01a6f8:::benchmark_data_smithi188_141483_object44529:head# 2024-04-23T16:43:53.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd01de4b:::benchmark_data_smithi188_141483_object2294:head# 2024-04-23T16:43:53.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd0210c8:::benchmark_data_smithi188_141483_object74555:head# 2024-04-23T16:43:53.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd023424:::benchmark_data_smithi188_141483_object90727:head# 2024-04-23T16:43:53.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd04f78e:::benchmark_data_smithi188_141483_object55530:head# 2024-04-23T16:43:53.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.260 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd058115:::benchmark_data_smithi188_141483_object56858:head# 2024-04-23T16:43:53.260 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd06ad67:::benchmark_data_smithi188_141483_object77898:head# 2024-04-23T16:43:53.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd081090:::benchmark_data_smithi188_141483_object79432:head# 2024-04-23T16:43:53.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd088003:::benchmark_data_smithi188_141483_object70480:head# 2024-04-23T16:43:53.261 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.261 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd097237:::benchmark_data_smithi188_141483_object36016:head# 2024-04-23T16:43:53.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd09fa63:::benchmark_data_smithi188_141483_object83574:head# 2024-04-23T16:43:53.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd0a2d7c:::benchmark_data_smithi188_141483_object72674:head# 2024-04-23T16:43:53.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.262 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd0a4652:::benchmark_data_smithi188_141483_object80328:head# 2024-04-23T16:43:53.262 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd0ad9f0:::benchmark_data_smithi188_141483_object57260:head# 2024-04-23T16:43:53.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd0b2273:::benchmark_data_smithi188_141483_object2212:head# 2024-04-23T16:43:53.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd0b838d:::benchmark_data_smithi188_141483_object76078:head# 2024-04-23T16:43:53.263 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.263 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd0bd515:::benchmark_data_smithi188_141483_object66903:head# 2024-04-23T16:43:53.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd0cbf95:::benchmark_data_smithi188_141483_object65336:head# 2024-04-23T16:43:53.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd0ced33:::benchmark_data_smithi188_141483_object5036:head# 2024-04-23T16:43:53.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd0e5329:::benchmark_data_smithi188_141483_object6077:head# 2024-04-23T16:43:53.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd0ec880:::benchmark_data_smithi188_141483_object34100:head# 2024-04-23T16:43:53.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd10245c:::benchmark_data_smithi188_141483_object2571:head# 2024-04-23T16:43:53.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.264 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd10465c:::benchmark_data_smithi188_141483_object77820:head# 2024-04-23T16:43:53.264 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd110d63:::benchmark_data_smithi188_141483_object8725:head# 2024-04-23T16:43:53.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd116b05:::benchmark_data_smithi188_141483_object22603:head# 2024-04-23T16:43:53.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd143143:::benchmark_data_smithi188_141483_object59498:head# 2024-04-23T16:43:53.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd14e73f:::benchmark_data_smithi188_141483_object13078:head# 2024-04-23T16:43:53.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd14f740:::benchmark_data_smithi188_141483_object47731:head# 2024-04-23T16:43:53.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.265 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd15a533:::benchmark_data_smithi188_141483_object5560:head# 2024-04-23T16:43:53.265 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd17a860:::benchmark_data_smithi188_141483_object33130:head# 2024-04-23T16:43:53.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd18675e:::benchmark_data_smithi188_141483_object57870:head# 2024-04-23T16:43:53.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd19be3d:::benchmark_data_smithi188_141483_object41954:head# 2024-04-23T16:43:53.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd1a7e31:::benchmark_data_smithi188_141483_object61898:head# 2024-04-23T16:43:53.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd1af2d8:::benchmark_data_smithi188_141483_object84230:head# 2024-04-23T16:43:53.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.266 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd1b0701:::benchmark_data_smithi188_141483_object33709:head# 2024-04-23T16:43:53.266 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd1bd572:::benchmark_data_smithi188_141483_object33839:head# 2024-04-23T16:43:53.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd1c2c07:::benchmark_data_smithi188_141483_object29483:head# 2024-04-23T16:43:53.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd1c7609:::benchmark_data_smithi188_141483_object87305:head# 2024-04-23T16:43:53.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd1d7f9f:::benchmark_data_smithi188_141483_object44787:head# 2024-04-23T16:43:53.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd1d959f:::benchmark_data_smithi188_141483_object28060:head# 2024-04-23T16:43:53.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.267 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd1df672:::benchmark_data_smithi188_141483_object59151:head# 2024-04-23T16:43:53.267 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd1e68f4:::benchmark_data_smithi188_141483_object20664:head# 2024-04-23T16:43:53.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd209d71:::benchmark_data_smithi188_141483_object40944:head# 2024-04-23T16:43:53.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd23206d:::benchmark_data_smithi188_141483_object11658:head# 2024-04-23T16:43:53.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd25791c:::benchmark_data_smithi188_141483_object32684:head# 2024-04-23T16:43:53.268 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.268 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd26d503:::benchmark_data_smithi188_141483_object41146:head# 2024-04-23T16:43:53.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd26ef71:::benchmark_data_smithi188_141483_object56892:head# 2024-04-23T16:43:53.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd2712b9:::benchmark_data_smithi188_141483_object90130:head# 2024-04-23T16:43:53.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd27fff4:::benchmark_data_smithi188_141483_object40981:head# 2024-04-23T16:43:53.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd2a75da:::benchmark_data_smithi188_141483_object38338:head# 2024-04-23T16:43:53.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd2b3383:::benchmark_data_smithi188_141483_object53423:head# 2024-04-23T16:43:53.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.269 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd2cc849:::benchmark_data_smithi188_141483_object11166:head# 2024-04-23T16:43:53.269 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd2eeae5:::benchmark_data_smithi188_141483_object38424:head# 2024-04-23T16:43:53.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd2f8e20:::benchmark_data_smithi188_141483_object38802:head# 2024-04-23T16:43:53.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd317c14:::benchmark_data_smithi188_141483_object25803:head# 2024-04-23T16:43:53.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd32df3f:::benchmark_data_smithi188_141483_object77501:head# 2024-04-23T16:43:53.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd33becf:::benchmark_data_smithi188_141483_object17022:head# 2024-04-23T16:43:53.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.270 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd34f7cc:::benchmark_data_smithi188_141483_object45985:head# 2024-04-23T16:43:53.270 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd354924:::benchmark_data_smithi188_141483_object53481:head# 2024-04-23T16:43:53.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd36c998:::benchmark_data_smithi188_141483_object78949:head# 2024-04-23T16:43:53.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd36feb8:::benchmark_data_smithi188_141483_object77099:head# 2024-04-23T16:43:53.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd37c168:::benchmark_data_smithi188_141483_object24572:head# 2024-04-23T16:43:53.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd37e315:::benchmark_data_smithi188_141483_object332:head# 2024-04-23T16:43:53.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd37f8c4:::benchmark_data_smithi188_141483_object30788:head# 2024-04-23T16:43:53.271 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.271 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd387c88:::benchmark_data_smithi188_141483_object66651:head# 2024-04-23T16:43:53.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd3c1874:::benchmark_data_smithi188_141483_object63278:head# 2024-04-23T16:43:53.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd3c8295:::benchmark_data_smithi188_141483_object76770:head# 2024-04-23T16:43:53.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd3ca140:::benchmark_data_smithi188_141483_object74030:head# 2024-04-23T16:43:53.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd3dca25:::benchmark_data_smithi188_141483_object70904:head# 2024-04-23T16:43:53.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd3dcb87:::benchmark_data_smithi188_141483_object66857:head# 2024-04-23T16:43:53.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.272 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd3fbbbd:::benchmark_data_smithi188_141483_object63666:head# 2024-04-23T16:43:53.272 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd400521:::benchmark_data_smithi188_141483_object22511:head# 2024-04-23T16:43:53.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd400b1b:::benchmark_data_smithi188_141483_object53495:head# 2024-04-23T16:43:53.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd40d5f4:::benchmark_data_smithi188_141483_object74330:head# 2024-04-23T16:43:53.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd438d66:::benchmark_data_smithi188_141483_object58930:head# 2024-04-23T16:43:53.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd44d019:::benchmark_data_smithi188_141483_object78518:head# 2024-04-23T16:43:53.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd457e5d:::benchmark_data_smithi188_141483_object32949:head# 2024-04-23T16:43:53.273 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.273 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd45fe6e:::benchmark_data_smithi188_141483_object19496:head# 2024-04-23T16:43:53.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd4643e3:::benchmark_data_smithi188_141483_object61868:head# 2024-04-23T16:43:53.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd4674d2:::benchmark_data_smithi188_141483_object88253:head# 2024-04-23T16:43:53.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd469790:::benchmark_data_smithi188_141483_object79135:head# 2024-04-23T16:43:53.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd480b7f:::benchmark_data_smithi188_141483_object49997:head# 2024-04-23T16:43:53.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.274 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd481452:::benchmark_data_smithi188_141483_object57602:head# 2024-04-23T16:43:53.274 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd482a07:::benchmark_data_smithi188_141483_object23924:head# 2024-04-23T16:43:53.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd48e5ff:::benchmark_data_smithi188_141483_object598:head# 2024-04-23T16:43:53.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd4926a2:::benchmark_data_smithi188_141483_object90368:head# 2024-04-23T16:43:53.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd49727a:::benchmark_data_smithi188_141483_object12092:head# 2024-04-23T16:43:53.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd4acd95:::benchmark_data_smithi188_141483_object32629:head# 2024-04-23T16:43:53.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd4cb7ff:::benchmark_data_smithi188_141483_object76344:head# 2024-04-23T16:43:53.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.275 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd4ce68f:::benchmark_data_smithi188_141483_object69013:head# 2024-04-23T16:43:53.275 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd4cef32:::benchmark_data_smithi188_141483_object64528:head# 2024-04-23T16:43:53.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd4dff7d:::benchmark_data_smithi188_141483_object68124:head# 2024-04-23T16:43:53.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd4eef1c:::benchmark_data_smithi188_141483_object49073:head# 2024-04-23T16:43:53.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd4f3023:::benchmark_data_smithi188_141483_object89367:head# 2024-04-23T16:43:53.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd4fa643:::benchmark_data_smithi188_141483_object6621:head# 2024-04-23T16:43:53.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd4fb201:::benchmark_data_smithi188_141483_object46121:head# 2024-04-23T16:43:53.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.276 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd51569e:::benchmark_data_smithi188_141483_object55288:head# 2024-04-23T16:43:53.276 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd51d62d:::benchmark_data_smithi188_141483_object57359:head# 2024-04-23T16:43:53.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd52b0a9:::benchmark_data_smithi188_141483_object26726:head# 2024-04-23T16:43:53.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd52bba8:::benchmark_data_smithi188_141483_object47951:head# 2024-04-23T16:43:53.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd52e7eb:::benchmark_data_smithi188_141483_object15449:head# 2024-04-23T16:43:53.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd52e973:::benchmark_data_smithi188_141483_object27758:head# 2024-04-23T16:43:53.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd5309ad:::benchmark_data_smithi188_141483_object66121:head# 2024-04-23T16:43:53.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.277 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd55ff62:::benchmark_data_smithi188_141483_object88916:head# 2024-04-23T16:43:53.277 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd560772:::benchmark_data_smithi188_141483_object11466:head# 2024-04-23T16:43:53.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd56ad9b:::benchmark_data_smithi188_141483_object21605:head# 2024-04-23T16:43:53.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd56b655:::benchmark_data_smithi188_141483_object16263:head# 2024-04-23T16:43:53.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd56dfd7:::benchmark_data_smithi188_141483_object21203:head# 2024-04-23T16:43:53.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd571180:::benchmark_data_smithi188_141483_object76913:head# 2024-04-23T16:43:53.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd57c2b6:::benchmark_data_smithi188_141483_object88674:head# 2024-04-23T16:43:53.278 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.278 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd58d998:::benchmark_data_smithi188_141483_object80456:head# 2024-04-23T16:43:53.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd5915aa:::benchmark_data_smithi188_141483_object40343:head# 2024-04-23T16:43:53.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd598e6b:::benchmark_data_smithi188_141483_object50645:head# 2024-04-23T16:43:53.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd5a8607:::benchmark_data_smithi188_141483_object35675:head# 2024-04-23T16:43:53.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd5c23b4:::benchmark_data_smithi188_141483_object77331:head# 2024-04-23T16:43:53.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd5c713c:::benchmark_data_smithi188_141483_object63356:head# 2024-04-23T16:43:53.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd5dc1e7:::benchmark_data_smithi188_141483_object416:head# 2024-04-23T16:43:53.279 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.279 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd5ec1ce:::benchmark_data_smithi188_141483_object1939:head# 2024-04-23T16:43:53.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd623dc3:::benchmark_data_smithi188_141483_object44212:head# 2024-04-23T16:43:53.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd6287f0:::benchmark_data_smithi188_141483_object85407:head# 2024-04-23T16:43:53.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd63f048:::benchmark_data_smithi188_141483_object42074:head# 2024-04-23T16:43:53.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd64643b:::benchmark_data_smithi188_141483_object30786:head# 2024-04-23T16:43:53.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd65098f:::benchmark_data_smithi188_141483_object8893:head# 2024-04-23T16:43:53.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd6518d9:::benchmark_data_smithi188_141483_object61223:head# 2024-04-23T16:43:53.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.280 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd65ec1c:::benchmark_data_smithi188_141483_object76783:head# 2024-04-23T16:43:53.280 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd664433:::benchmark_data_smithi188_141483_object44227:head# 2024-04-23T16:43:53.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd673493:::benchmark_data_smithi188_141483_object72480:head# 2024-04-23T16:43:53.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd6812a9:::benchmark_data_smithi188_141483_object3782:head# 2024-04-23T16:43:53.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd687c78:::benchmark_data_smithi188_141483_object24656:head# 2024-04-23T16:43:53.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd69006a:::benchmark_data_smithi188_141483_object61779:head# 2024-04-23T16:43:53.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.281 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd693968:::benchmark_data_smithi188_141483_object4377:head# 2024-04-23T16:43:53.281 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd6b7de5:::benchmark_data_smithi188_141483_object41607:head# 2024-04-23T16:43:53.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd6cb1eb:::benchmark_data_smithi188_141483_object46152:head# 2024-04-23T16:43:53.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd6d28c6:::benchmark_data_smithi188_141483_object70308:head# 2024-04-23T16:43:53.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd6d55a8:::benchmark_data_smithi188_141483_object28198:head# 2024-04-23T16:43:53.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd6f4d56:::benchmark_data_smithi188_141483_object67558:head# 2024-04-23T16:43:53.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd6feb8b:::benchmark_data_smithi188_141483_object52770:head# 2024-04-23T16:43:53.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.282 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd715e81:::benchmark_data_smithi188_141483_object1282:head# 2024-04-23T16:43:53.282 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd71997a:::benchmark_data_smithi188_141483_object39177:head# 2024-04-23T16:43:53.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd71d353:::benchmark_data_smithi188_141483_object72667:head# 2024-04-23T16:43:53.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd722968:::benchmark_data_smithi188_141483_object46468:head# 2024-04-23T16:43:53.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd734f18:::benchmark_data_smithi188_141483_object36738:head# 2024-04-23T16:43:53.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd737abd:::benchmark_data_smithi188_141483_object12748:head# 2024-04-23T16:43:53.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd739b1a:::benchmark_data_smithi188_141483_object4923:head# 2024-04-23T16:43:53.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.283 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd745d5a:::benchmark_data_smithi188_141483_object76752:head# 2024-04-23T16:43:53.283 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd76272e:::benchmark_data_smithi188_141483_object32714:head# 2024-04-23T16:43:53.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd7631f8:::benchmark_data_smithi188_141483_object26655:head# 2024-04-23T16:43:53.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd764805:::benchmark_data_smithi188_141483_object26441:head# 2024-04-23T16:43:53.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd76604d:::benchmark_data_smithi188_141483_object22489:head# 2024-04-23T16:43:53.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd7668e4:::benchmark_data_smithi188_141483_object57809:head# 2024-04-23T16:43:53.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd76bb05:::benchmark_data_smithi188_141483_object48861:head# 2024-04-23T16:43:53.284 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.284 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd76d6da:::benchmark_data_smithi188_141483_object62612:head# 2024-04-23T16:43:53.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd790437:::benchmark_data_smithi188_141483_object90761:head# 2024-04-23T16:43:53.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd7b7aaa:::benchmark_data_smithi188_141483_object34128:head# 2024-04-23T16:43:53.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd7c5683:::benchmark_data_smithi188_141483_object67840:head# 2024-04-23T16:43:53.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd7d382a:::benchmark_data_smithi188_141483_object48278:head# 2024-04-23T16:43:53.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd7e457d:::benchmark_data_smithi188_141483_object61708:head# 2024-04-23T16:43:53.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.285 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd7ee199:::benchmark_data_smithi188_141483_object23762:head# 2024-04-23T16:43:53.285 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd7fe6f1:::benchmark_data_smithi188_141483_object5417:head# 2024-04-23T16:43:53.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd8024ac:::benchmark_data_smithi188_141483_object87806:head# 2024-04-23T16:43:53.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd807f68:::benchmark_data_smithi188_141483_object6563:head# 2024-04-23T16:43:53.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd80af08:::benchmark_data_smithi188_141483_object23508:head# 2024-04-23T16:43:53.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd80dde8:::benchmark_data_smithi188_141483_object88059:head# 2024-04-23T16:43:53.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd81000a:::benchmark_data_smithi188_141483_object80669:head# 2024-04-23T16:43:53.286 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.286 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd81a217:::benchmark_data_smithi188_141483_object21626:head# 2024-04-23T16:43:53.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd821f2d:::benchmark_data_smithi188_141483_object441:head# 2024-04-23T16:43:53.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd83b024:::benchmark_data_smithi188_141483_object51463:head# 2024-04-23T16:43:53.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd843edb:::benchmark_data_smithi188_141483_object70900:head# 2024-04-23T16:43:53.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd854dee:::benchmark_data_smithi188_141483_object22621:head# 2024-04-23T16:43:53.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd859f64:::benchmark_data_smithi188_141483_object13764:head# 2024-04-23T16:43:53.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.287 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd85ef31:::benchmark_data_smithi188_141483_object3673:head# 2024-04-23T16:43:53.287 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd86f806:::benchmark_data_smithi188_141483_object30362:head# 2024-04-23T16:43:53.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd870d6c:::benchmark_data_smithi188_141483_object22198:head# 2024-04-23T16:43:53.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd87e509:::benchmark_data_smithi188_141483_object20144:head# 2024-04-23T16:43:53.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd887629:::benchmark_data_smithi188_141483_object16825:head# 2024-04-23T16:43:53.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd889b20:::benchmark_data_smithi188_141483_object17442:head# 2024-04-23T16:43:53.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd88b54a:::benchmark_data_smithi188_141483_object47039:head# 2024-04-23T16:43:53.288 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.288 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd896c06:::benchmark_data_smithi188_141483_object11852:head# 2024-04-23T16:43:53.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd8a5ea7:::benchmark_data_smithi188_141483_object85185:head# 2024-04-23T16:43:53.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd8b120d:::benchmark_data_smithi188_141483_object6120:head# 2024-04-23T16:43:53.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd8b6bdc:::benchmark_data_smithi188_141483_object28463:head# 2024-04-23T16:43:53.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd8d143e:::benchmark_data_smithi188_141483_object32662:head# 2024-04-23T16:43:53.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd8d7fd1:::benchmark_data_smithi188_141483_object89234:head# 2024-04-23T16:43:53.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.289 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd8e0824:::benchmark_data_smithi188_141483_object47356:head# 2024-04-23T16:43:53.289 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd8e8d4a:::benchmark_data_smithi188_141483_object48666:head# 2024-04-23T16:43:53.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd9034d8:::benchmark_data_smithi188_141483_object66181:head# 2024-04-23T16:43:53.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd90d53d:::benchmark_data_smithi188_141483_object64748:head# 2024-04-23T16:43:53.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd910513:::benchmark_data_smithi188_141483_object652:head# 2024-04-23T16:43:53.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd910547:::benchmark_data_smithi188_141483_object59610:head# 2024-04-23T16:43:53.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.290 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd914457:::benchmark_data_smithi188_141483_object73368:head# 2024-04-23T16:43:53.290 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd931595:::benchmark_data_smithi188_141483_object14340:head# 2024-04-23T16:43:53.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd9333d6:::benchmark_data_smithi188_141483_object33538:head# 2024-04-23T16:43:53.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd9370cb:::benchmark_data_smithi188_141483_object25547:head# 2024-04-23T16:43:53.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd93e4fd:::benchmark_data_smithi188_141483_object26717:head# 2024-04-23T16:43:53.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd940c56:::benchmark_data_smithi188_141483_object90431:head# 2024-04-23T16:43:53.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd9440ef:::benchmark_data_smithi188_141483_object36425:head# 2024-04-23T16:43:53.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.291 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd94fc31:::benchmark_data_smithi188_141483_object41940:head# 2024-04-23T16:43:53.291 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd95117d:::benchmark_data_smithi188_141483_object34641:head# 2024-04-23T16:43:53.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd95ad06:::benchmark_data_smithi188_141483_object40183:head# 2024-04-23T16:43:53.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd963d33:::benchmark_data_smithi188_141483_object63420:head# 2024-04-23T16:43:53.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd96f06b:::benchmark_data_smithi188_141483_object7055:head# 2024-04-23T16:43:53.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd972142:::benchmark_data_smithi188_141483_object52920:head# 2024-04-23T16:43:53.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.292 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd974879:::benchmark_data_smithi188_141483_object39729:head# 2024-04-23T16:43:53.292 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd975bce:::benchmark_data_smithi188_141483_object81016:head# 2024-04-23T16:43:53.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd97f156:::benchmark_data_smithi188_141483_object87139:head# 2024-04-23T16:43:53.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd980ce7:::benchmark_data_smithi188_141483_object116:head# 2024-04-23T16:43:53.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd98d207:::benchmark_data_smithi188_141483_object23645:head# 2024-04-23T16:43:53.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd98d693:::benchmark_data_smithi188_141483_object1135:head# 2024-04-23T16:43:53.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd99331c:::benchmark_data_smithi188_141483_object6488:head# 2024-04-23T16:43:53.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.293 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd9a5f1b:::benchmark_data_smithi188_141483_object39024:head# 2024-04-23T16:43:53.293 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd9a7b9a:::benchmark_data_smithi188_141483_object17941:head# 2024-04-23T16:43:53.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd9b91f7:::benchmark_data_smithi188_141483_object69228:head# 2024-04-23T16:43:53.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd9c22e7:::benchmark_data_smithi188_141483_object17775:head# 2024-04-23T16:43:53.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd9c5632:::benchmark_data_smithi188_141483_object68311:head# 2024-04-23T16:43:53.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd9d3941:::benchmark_data_smithi188_141483_object10216:head# 2024-04-23T16:43:53.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.294 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd9d7847:::benchmark_data_smithi188_141483_object22217:head# 2024-04-23T16:43:53.294 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd9db443:::benchmark_data_smithi188_141483_object67784:head# 2024-04-23T16:43:53.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cd9f5f86:::benchmark_data_smithi188_141483_object46824:head# 2024-04-23T16:43:53.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cda1a3ac:::benchmark_data_smithi188_141483_object32406:head# 2024-04-23T16:43:53.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cda1b8af:::benchmark_data_smithi188_141483_object75789:head# 2024-04-23T16:43:53.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cda1e1ef:::benchmark_data_smithi188_141483_object85592:head# 2024-04-23T16:43:53.295 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.295 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cda20cb6:::benchmark_data_smithi188_141483_object64217:head# 2024-04-23T16:43:53.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cda2132b:::benchmark_data_smithi188_141483_object18732:head# 2024-04-23T16:43:53.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cda23652:::benchmark_data_smithi188_141483_object66297:head# 2024-04-23T16:43:53.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cda326d6:::benchmark_data_smithi188_141483_object52304:head# 2024-04-23T16:43:53.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cda3f2d8:::benchmark_data_smithi188_141483_object9339:head# 2024-04-23T16:43:53.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cda62fac:::benchmark_data_smithi188_141483_object37430:head# 2024-04-23T16:43:53.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cda6532d:::benchmark_data_smithi188_141483_object76812:head# 2024-04-23T16:43:53.296 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.296 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cda66e19:::benchmark_data_smithi188_141483_object49923:head# 2024-04-23T16:43:53.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cda698e8:::benchmark_data_smithi188_141483_object85494:head# 2024-04-23T16:43:53.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cda6e665:::benchmark_data_smithi188_141483_object66006:head# 2024-04-23T16:43:53.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cda8efb8:::benchmark_data_smithi188_141483_object58593:head# 2024-04-23T16:43:53.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cda91703:::benchmark_data_smithi188_141483_object66144:head# 2024-04-23T16:43:53.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cda9a635:::benchmark_data_smithi188_141483_object22309:head# 2024-04-23T16:43:53.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdaa22e2:::benchmark_data_smithi188_141483_object24376:head# 2024-04-23T16:43:53.297 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.297 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdaa4b1a:::benchmark_data_smithi188_141483_object35740:head# 2024-04-23T16:43:53.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdac519b:::benchmark_data_smithi188_141483_object11534:head# 2024-04-23T16:43:53.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdad2eea:::benchmark_data_smithi188_141483_object81761:head# 2024-04-23T16:43:53.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdaf7157:::benchmark_data_smithi188_141483_object25864:head# 2024-04-23T16:43:53.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb012b1:::benchmark_data_smithi188_141483_object37827:head# 2024-04-23T16:43:53.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb04746:::benchmark_data_smithi188_141483_object59027:head# 2024-04-23T16:43:53.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.298 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb0d088:::benchmark_data_smithi188_141483_object3351:head# 2024-04-23T16:43:53.298 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb1b9f3:::benchmark_data_smithi188_141483_object24270:head# 2024-04-23T16:43:53.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb1e8ea:::benchmark_data_smithi188_141483_object67163:head# 2024-04-23T16:43:53.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb2237d:::benchmark_data_smithi188_141483_object61213:head# 2024-04-23T16:43:53.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb30857:::benchmark_data_smithi188_141483_object66597:head# 2024-04-23T16:43:53.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb33906:::benchmark_data_smithi188_141483_object28328:head# 2024-04-23T16:43:53.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.299 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb36579:::benchmark_data_smithi188_141483_object18380:head# 2024-04-23T16:43:53.299 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb39c59:::benchmark_data_smithi188_141483_object28396:head# 2024-04-23T16:43:53.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb3a986:::benchmark_data_smithi188_141483_object84214:head# 2024-04-23T16:43:53.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb3ca6d:::benchmark_data_smithi188_141483_object36228:head# 2024-04-23T16:43:53.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb3d49a:::benchmark_data_smithi188_141483_object30623:head# 2024-04-23T16:43:53.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb41c23:::benchmark_data_smithi188_141483_object37740:head# 2024-04-23T16:43:53.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb46fbb:::benchmark_data_smithi188_141483_object24800:head# 2024-04-23T16:43:53.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.300 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb4c04e:::benchmark_data_smithi188_141483_object50371:head# 2024-04-23T16:43:53.300 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb4ff57:::benchmark_data_smithi188_141483_object53839:head# 2024-04-23T16:43:53.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb5029b:::benchmark_data_smithi188_141483_object90466:head# 2024-04-23T16:43:53.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb5a8bc:::benchmark_data_smithi188_141483_object24834:head# 2024-04-23T16:43:53.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb5b0d0:::benchmark_data_smithi188_141483_object51509:head# 2024-04-23T16:43:53.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb6b515:::benchmark_data_smithi188_141483_object40362:head# 2024-04-23T16:43:53.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb6be99:::benchmark_data_smithi188_141483_object87340:head# 2024-04-23T16:43:53.301 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.301 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb72262:::benchmark_data_smithi188_141483_object47735:head# 2024-04-23T16:43:53.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb77c00:::benchmark_data_smithi188_141483_object50681:head# 2024-04-23T16:43:53.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdb998f7:::benchmark_data_smithi188_141483_object12659:head# 2024-04-23T16:43:53.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdba7158:::benchmark_data_smithi188_141483_object30946:head# 2024-04-23T16:43:53.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdbb0291:::benchmark_data_smithi188_141483_object20530:head# 2024-04-23T16:43:53.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.302 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdbb10ab:::benchmark_data_smithi188_141483_object33651:head# 2024-04-23T16:43:53.302 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdbb8455:::benchmark_data_smithi188_141483_object8172:head# 2024-04-23T16:43:53.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdbbc668:::benchmark_data_smithi188_141483_object73733:head# 2024-04-23T16:43:53.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdbd4484:::benchmark_data_smithi188_141483_object10644:head# 2024-04-23T16:43:53.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdc092f7:::benchmark_data_smithi188_141483_object72579:head# 2024-04-23T16:43:53.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdc09b2b:::benchmark_data_smithi188_141483_object85131:head# 2024-04-23T16:43:53.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdc12583:::benchmark_data_smithi188_141483_object81363:head# 2024-04-23T16:43:53.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.303 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdc173a6:::benchmark_data_smithi188_141483_object81390:head# 2024-04-23T16:43:53.303 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdc22deb:::benchmark_data_smithi188_141483_object55529:head# 2024-04-23T16:43:53.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdc304de:::benchmark_data_smithi188_141483_object17846:head# 2024-04-23T16:43:53.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdc3cc71:::benchmark_data_smithi188_141483_object15945:head# 2024-04-23T16:43:53.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdc4a6f3:::benchmark_data_smithi188_141483_object37449:head# 2024-04-23T16:43:53.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdc656a4:::benchmark_data_smithi188_141483_object5975:head# 2024-04-23T16:43:53.304 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.304 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdc82a8a:::benchmark_data_smithi188_141483_object66174:head# 2024-04-23T16:43:53.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdc85029:::benchmark_data_smithi188_141483_object60150:head# 2024-04-23T16:43:53.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdc86e57:::benchmark_data_smithi188_141483_object79451:head# 2024-04-23T16:43:53.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdc91c88:::benchmark_data_smithi188_141483_object77453:head# 2024-04-23T16:43:53.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdc98a47:::benchmark_data_smithi188_141483_object18374:head# 2024-04-23T16:43:53.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdc99f12:::benchmark_data_smithi188_141483_object32604:head# 2024-04-23T16:43:53.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.305 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdc9a86a:::benchmark_data_smithi188_141483_object26625:head# 2024-04-23T16:43:53.305 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdc9f9a3:::benchmark_data_smithi188_141483_object79972:head# 2024-04-23T16:43:53.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdca39ef:::benchmark_data_smithi188_141483_object80883:head# 2024-04-23T16:43:53.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdca7e4f:::benchmark_data_smithi188_141483_object14726:head# 2024-04-23T16:43:53.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdcaa0ee:::benchmark_data_smithi188_141483_object84807:head# 2024-04-23T16:43:53.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdcfe04d:::benchmark_data_smithi188_141483_object79970:head# 2024-04-23T16:43:53.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdd0dcae:::benchmark_data_smithi188_141483_object23457:head# 2024-04-23T16:43:53.306 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.306 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdd1da8d:::benchmark_data_smithi188_141483_object68257:head# 2024-04-23T16:43:53.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdd1dd76:::benchmark_data_smithi188_141483_object6749:head# 2024-04-23T16:43:53.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdd1e30e:::benchmark_data_smithi188_141483_object23755:head# 2024-04-23T16:43:53.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdd381c5:::benchmark_data_smithi188_141483_object55182:head# 2024-04-23T16:43:53.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdd3f864:::benchmark_data_smithi188_141483_object69557:head# 2024-04-23T16:43:53.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdd580b1:::benchmark_data_smithi188_141483_object49549:head# 2024-04-23T16:43:53.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.307 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdd6debb:::benchmark_data_smithi188_141483_object65206:head# 2024-04-23T16:43:53.307 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdd6e152:::benchmark_data_smithi188_141483_object87774:head# 2024-04-23T16:43:53.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdd7134c:::benchmark_data_smithi188_141483_object37000:head# 2024-04-23T16:43:53.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdd78ac6:::benchmark_data_smithi188_141483_object89056:head# 2024-04-23T16:43:53.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdd78b9b:::benchmark_data_smithi188_141483_object51876:head# 2024-04-23T16:43:53.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdd86196:::benchmark_data_smithi188_141483_object49178:head# 2024-04-23T16:43:53.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdd8e081:::benchmark_data_smithi188_141483_object83686:head# 2024-04-23T16:43:53.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdd93313:::benchmark_data_smithi188_141483_object82409:head# 2024-04-23T16:43:53.308 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.308 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdd9802b:::benchmark_data_smithi188_141483_object79740:head# 2024-04-23T16:43:53.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdd9b5e7:::benchmark_data_smithi188_141483_object83483:head# 2024-04-23T16:43:53.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdda4486:::benchmark_data_smithi188_141483_object61666:head# 2024-04-23T16:43:53.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdda6baa:::benchmark_data_smithi188_141483_object22016:head# 2024-04-23T16:43:53.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cddb14e6:::benchmark_data_smithi188_141483_object12225:head# 2024-04-23T16:43:53.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cddb3365:::benchmark_data_smithi188_141483_object7106:head# 2024-04-23T16:43:53.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cddcef69:::benchmark_data_smithi188_141483_object5751:head# 2024-04-23T16:43:53.309 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.309 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdddf1b3:::benchmark_data_smithi188_141483_object79640:head# 2024-04-23T16:43:53.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cddeade0:::benchmark_data_smithi188_141483_object47948:head# 2024-04-23T16:43:53.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cde0123f:::benchmark_data_smithi188_141483_object4669:head# 2024-04-23T16:43:53.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cde10a25:::benchmark_data_smithi188_141483_object20733:head# 2024-04-23T16:43:53.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cde11056:::benchmark_data_smithi188_141483_object51330:head# 2024-04-23T16:43:53.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cde2f84f:::benchmark_data_smithi188_141483_object55217:head# 2024-04-23T16:43:53.310 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.310 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cde3b3e6:::benchmark_data_smithi188_141483_object22671:head# 2024-04-23T16:43:53.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cde5196a:::benchmark_data_smithi188_141483_object12799:head# 2024-04-23T16:43:53.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cde52b47:::benchmark_data_smithi188_141483_object41159:head# 2024-04-23T16:43:53.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cde5a88e:::benchmark_data_smithi188_141483_object28894:head# 2024-04-23T16:43:53.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cde5d933:::benchmark_data_smithi188_141483_object36132:head# 2024-04-23T16:43:53.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cde60739:::benchmark_data_smithi188_141483_object52258:head# 2024-04-23T16:43:53.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cde6876a:::benchmark_data_smithi188_141483_object1287:head# 2024-04-23T16:43:53.311 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.311 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cde8009f:::benchmark_data_smithi188_141483_object17299:head# 2024-04-23T16:43:53.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cde99672:::benchmark_data_smithi188_141483_object40307:head# 2024-04-23T16:43:53.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdec0f79:::benchmark_data_smithi188_141483_object89241:head# 2024-04-23T16:43:53.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdec724b:::benchmark_data_smithi188_141483_object90094:head# 2024-04-23T16:43:53.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cded206a:::benchmark_data_smithi188_141483_object23218:head# 2024-04-23T16:43:53.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cded236d:::benchmark_data_smithi188_141483_object77912:head# 2024-04-23T16:43:53.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cded4776:::benchmark_data_smithi188_141483_object88892:head# 2024-04-23T16:43:53.312 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.312 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdedae96:::benchmark_data_smithi188_141483_object65108:head# 2024-04-23T16:43:53.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdee50e3:::benchmark_data_smithi188_141483_object29110:head# 2024-04-23T16:43:53.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdef3b05:::benchmark_data_smithi188_141483_object73915:head# 2024-04-23T16:43:53.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdef8fe6:::benchmark_data_smithi188_141483_object86485:head# 2024-04-23T16:43:53.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdefd105:::benchmark_data_smithi188_141483_object79854:head# 2024-04-23T16:43:53.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.313 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdf101e6:::benchmark_data_smithi188_141483_object76786:head# 2024-04-23T16:43:53.313 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdf257dc:::benchmark_data_smithi188_141483_object3920:head# 2024-04-23T16:43:53.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdf308be:::benchmark_data_smithi188_141483_object68249:head# 2024-04-23T16:43:53.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdf38d68:::benchmark_data_smithi188_141483_object60050:head# 2024-04-23T16:43:53.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdf3cf9f:::benchmark_data_smithi188_141483_object7115:head# 2024-04-23T16:43:53.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdf44758:::benchmark_data_smithi188_141483_object44299:head# 2024-04-23T16:43:53.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdf462ce:::benchmark_data_smithi188_141483_object49832:head# 2024-04-23T16:43:53.314 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.314 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdf47924:::benchmark_data_smithi188_141483_object82207:head# 2024-04-23T16:43:53.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdf4c7d9:::benchmark_data_smithi188_141483_object79947:head# 2024-04-23T16:43:53.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdf50712:::benchmark_data_smithi188_141483_object74271:head# 2024-04-23T16:43:53.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdf58e97:::benchmark_data_smithi188_141483_object71731:head# 2024-04-23T16:43:53.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdf62bbe:::benchmark_data_smithi188_141483_object7598:head# 2024-04-23T16:43:53.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.315 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdf7c7a7:::benchmark_data_smithi188_141483_object31723:head# 2024-04-23T16:43:53.315 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdf7d478:::benchmark_data_smithi188_141483_object4540:head# 2024-04-23T16:43:53.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdf90288:::benchmark_data_smithi188_141483_object5271:head# 2024-04-23T16:43:53.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdf906c4:::benchmark_data_smithi188_141483_object28003:head# 2024-04-23T16:43:53.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdf93663:::benchmark_data_smithi188_141483_object55999:head# 2024-04-23T16:43:53.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdfaabac:::benchmark_data_smithi188_141483_object51971:head# 2024-04-23T16:43:53.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdfc03f3:::benchmark_data_smithi188_141483_object58817:head# 2024-04-23T16:43:53.316 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.316 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdfe65c3:::benchmark_data_smithi188_141483_object29184:head# 2024-04-23T16:43:53.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cdff2e69:::benchmark_data_smithi188_141483_object83115:head# 2024-04-23T16:43:53.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce00ada1:::benchmark_data_smithi188_141483_object18094:head# 2024-04-23T16:43:53.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce00e886:::benchmark_data_smithi188_141483_object15493:head# 2024-04-23T16:43:53.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce01115e:::benchmark_data_smithi188_141483_object52999:head# 2024-04-23T16:43:53.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce01335c:::benchmark_data_smithi188_141483_object25217:head# 2024-04-23T16:43:53.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.317 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce035e3c:::benchmark_data_smithi188_141483_object35994:head# 2024-04-23T16:43:53.317 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce039016:::benchmark_data_smithi188_141483_object54667:head# 2024-04-23T16:43:53.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce043de1:::benchmark_data_smithi188_141483_object81317:head# 2024-04-23T16:43:53.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce046d64:::benchmark_data_smithi188_141483_object60323:head# 2024-04-23T16:43:53.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce05ab39:::benchmark_data_smithi188_141483_object12502:head# 2024-04-23T16:43:53.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce060289:::benchmark_data_smithi188_141483_object24447:head# 2024-04-23T16:43:53.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce06548b:::benchmark_data_smithi188_141483_object35954:head# 2024-04-23T16:43:53.318 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.318 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce06602f:::benchmark_data_smithi188_141483_object13852:head# 2024-04-23T16:43:53.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce07f3d1:::benchmark_data_smithi188_141483_object39101:head# 2024-04-23T16:43:53.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce0834f4:::benchmark_data_smithi188_141483_object8856:head# 2024-04-23T16:43:53.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce097b2d:::benchmark_data_smithi188_141483_object2093:head# 2024-04-23T16:43:53.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce09ea93:::benchmark_data_smithi188_141483_object15603:head# 2024-04-23T16:43:53.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce0ac640:::benchmark_data_smithi188_141483_object2309:head# 2024-04-23T16:43:53.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce0b020b:::benchmark_data_smithi188_141483_object68557:head# 2024-04-23T16:43:53.319 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.319 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce0b62f6:::benchmark_data_smithi188_141483_object62729:head# 2024-04-23T16:43:53.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce0b6b8d:::benchmark_data_smithi188_141483_object14108:head# 2024-04-23T16:43:53.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce0b8401:::benchmark_data_smithi188_141483_object62383:head# 2024-04-23T16:43:53.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce0ba633:::benchmark_data_smithi188_141483_object55855:head# 2024-04-23T16:43:53.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce0bfff1:::benchmark_data_smithi188_141483_object576:head# 2024-04-23T16:43:53.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce0c7f33:::benchmark_data_smithi188_141483_object12824:head# 2024-04-23T16:43:53.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.320 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce0e740f:::benchmark_data_smithi188_141483_object37910:head# 2024-04-23T16:43:53.320 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce0f5faa:::benchmark_data_smithi188_141483_object31716:head# 2024-04-23T16:43:53.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce0fee8a:::benchmark_data_smithi188_141483_object78534:head# 2024-04-23T16:43:53.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce1064c3:::benchmark_data_smithi188_141483_object47562:head# 2024-04-23T16:43:53.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce1099ab:::benchmark_data_smithi188_141483_object27958:head# 2024-04-23T16:43:53.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce10badb:::benchmark_data_smithi188_141483_object4357:head# 2024-04-23T16:43:53.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.321 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce11313f:::benchmark_data_smithi188_141483_object72960:head# 2024-04-23T16:43:53.321 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce1181ff:::benchmark_data_smithi188_141483_object45934:head# 2024-04-23T16:43:53.322 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce11d510:::benchmark_data_smithi188_141483_object68022:head# 2024-04-23T16:43:53.322 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce128ea6:::benchmark_data_smithi188_141483_object44370:head# 2024-04-23T16:43:53.322 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce12dd9b:::benchmark_data_smithi188_141483_object41037:head# 2024-04-23T16:43:53.322 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce131a27:::benchmark_data_smithi188_141483_object61271:head# 2024-04-23T16:43:53.322 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce138f1d:::benchmark_data_smithi188_141483_object6210:head# 2024-04-23T16:43:53.322 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.322 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce144390:::benchmark_data_smithi188_141483_object17441:head# 2024-04-23T16:43:53.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce151497:::benchmark_data_smithi188_141483_object83996:head# 2024-04-23T16:43:53.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce155420:::benchmark_data_smithi188_141483_object18061:head# 2024-04-23T16:43:53.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce15da52:::benchmark_data_smithi188_141483_object53218:head# 2024-04-23T16:43:53.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce15dc3e:::benchmark_data_smithi188_141483_object8034:head# 2024-04-23T16:43:53.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce15edda:::benchmark_data_smithi188_141483_object25369:head# 2024-04-23T16:43:53.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.323 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce165f0e:::benchmark_data_smithi188_141483_object85967:head# 2024-04-23T16:43:53.323 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.324 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce171f80:::benchmark_data_smithi188_141483_object72211:head# 2024-04-23T16:43:53.324 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.324 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce175225:::benchmark_data_smithi188_141483_object42845:head# 2024-04-23T16:43:53.324 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.324 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce185011:::benchmark_data_smithi188_141483_object72923:head# 2024-04-23T16:43:53.324 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.324 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce18b5ad:::benchmark_data_smithi188_141483_object52493:head# 2024-04-23T16:43:53.324 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.324 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce18b773:::benchmark_data_smithi188_141483_object10395:head# 2024-04-23T16:43:53.324 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.324 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce18cbfd:::benchmark_data_smithi188_141483_object8460:head# 2024-04-23T16:43:53.324 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.324 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce1952f5:::benchmark_data_smithi188_141483_object21415:head# 2024-04-23T16:43:53.325 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.325 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce1978d8:::benchmark_data_smithi188_141483_object80620:head# 2024-04-23T16:43:53.325 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.325 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce1b06c1:::benchmark_data_smithi188_141483_object21048:head# 2024-04-23T16:43:53.325 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.325 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce1b203f:::benchmark_data_smithi188_141483_object16585:head# 2024-04-23T16:43:53.325 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.325 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce1b35e1:::benchmark_data_smithi188_141483_object26634:head# 2024-04-23T16:43:53.325 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.325 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce1bc60a:::benchmark_data_smithi188_141483_object19653:head# 2024-04-23T16:43:53.325 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.325 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce1c084b:::benchmark_data_smithi188_141483_object13267:head# 2024-04-23T16:43:53.326 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.326 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce1d2320:::benchmark_data_smithi188_141483_object12278:head# 2024-04-23T16:43:53.326 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.326 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce1e767d:::benchmark_data_smithi188_141483_object75519:head# 2024-04-23T16:43:53.326 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.326 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce1ec97d:::benchmark_data_smithi188_141483_object3284:head# 2024-04-23T16:43:53.326 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.326 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce1fba05:::benchmark_data_smithi188_141483_object46101:head# 2024-04-23T16:43:53.326 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.326 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce21f2df:::benchmark_data_smithi188_141483_object14400:head# 2024-04-23T16:43:53.326 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.326 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce280432:::benchmark_data_smithi188_141483_object1744:head# 2024-04-23T16:43:53.326 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.326 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce284ca6:::benchmark_data_smithi188_141483_object21999:head# 2024-04-23T16:43:53.327 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.327 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce295f04:::benchmark_data_smithi188_141483_object48413:head# 2024-04-23T16:43:53.327 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.327 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce298864:::benchmark_data_smithi188_141483_object76822:head# 2024-04-23T16:43:53.327 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.327 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce2a2c8d:::benchmark_data_smithi188_141483_object50999:head# 2024-04-23T16:43:53.327 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.327 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce2a8827:::benchmark_data_smithi188_141483_object12476:head# 2024-04-23T16:43:53.327 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.327 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce2b28ce:::benchmark_data_smithi188_141483_object59449:head# 2024-04-23T16:43:53.327 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.327 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce2c38bd:::benchmark_data_smithi188_141483_object41377:head# 2024-04-23T16:43:53.327 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.328 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce2d23cf:::benchmark_data_smithi188_141483_object66726:head# 2024-04-23T16:43:53.328 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.328 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce2dbe27:::benchmark_data_smithi188_141483_object2237:head# 2024-04-23T16:43:53.328 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.328 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce2dfbf9:::benchmark_data_smithi188_141483_object27943:head# 2024-04-23T16:43:53.328 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.328 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce2efa36:::benchmark_data_smithi188_141483_object8807:head# 2024-04-23T16:43:53.328 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.328 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce2fad0c:::benchmark_data_smithi188_141483_object75565:head# 2024-04-23T16:43:53.328 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.328 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce2fb9ab:::benchmark_data_smithi188_141483_object84253:head# 2024-04-23T16:43:53.328 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.328 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce2fd219:::benchmark_data_smithi188_141483_object61118:head# 2024-04-23T16:43:53.328 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce304799:::benchmark_data_smithi188_141483_object29673:head# 2024-04-23T16:43:53.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce309cdd:::benchmark_data_smithi188_141483_object89755:head# 2024-04-23T16:43:53.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce30e52e:::benchmark_data_smithi188_141483_object70566:head# 2024-04-23T16:43:53.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce3205ff:::benchmark_data_smithi188_141483_object21990:head# 2024-04-23T16:43:53.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce32c1ea:::benchmark_data_smithi188_141483_object82476:head# 2024-04-23T16:43:53.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce32ff0e:::benchmark_data_smithi188_141483_object7182:head# 2024-04-23T16:43:53.329 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.329 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce357234:::benchmark_data_smithi188_141483_object9958:head# 2024-04-23T16:43:53.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce36d58f:::benchmark_data_smithi188_141483_object14118:head# 2024-04-23T16:43:53.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce37016d:::benchmark_data_smithi188_141483_object13040:head# 2024-04-23T16:43:53.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce379094:::benchmark_data_smithi188_141483_object4765:head# 2024-04-23T16:43:53.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce37983a:::benchmark_data_smithi188_141483_object11270:head# 2024-04-23T16:43:53.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce38537f:::benchmark_data_smithi188_141483_object74327:head# 2024-04-23T16:43:53.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce386453:::benchmark_data_smithi188_141483_object33109:head# 2024-04-23T16:43:53.330 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.330 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce386457:::benchmark_data_smithi188_141483_object34173:head# 2024-04-23T16:43:53.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce396806:::benchmark_data_smithi188_141483_object51172:head# 2024-04-23T16:43:53.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce39827d:::benchmark_data_smithi188_141483_object68233:head# 2024-04-23T16:43:53.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce39ed25:::benchmark_data_smithi188_141483_object10650:head# 2024-04-23T16:43:53.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce3a2530:::benchmark_data_smithi188_141483_object82211:head# 2024-04-23T16:43:53.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce3a879d:::benchmark_data_smithi188_141483_object17328:head# 2024-04-23T16:43:53.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce3b8093:::benchmark_data_smithi188_141483_object6229:head# 2024-04-23T16:43:53.331 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.331 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce3e21c0:::benchmark_data_smithi188_141483_object3850:head# 2024-04-23T16:43:53.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce3f4983:::benchmark_data_smithi188_141483_object58683:head# 2024-04-23T16:43:53.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce3f5708:::benchmark_data_smithi188_141483_object35598:head# 2024-04-23T16:43:53.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce407e1d:::benchmark_data_smithi188_141483_object32534:head# 2024-04-23T16:43:53.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce40f11f:::benchmark_data_smithi188_141483_object56400:head# 2024-04-23T16:43:53.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce40fd9b:::benchmark_data_smithi188_141483_object78689:head# 2024-04-23T16:43:53.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce41dcb6:::benchmark_data_smithi188_141483_object76491:head# 2024-04-23T16:43:53.332 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.332 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce42a5a0:::benchmark_data_smithi188_141483_object60211:head# 2024-04-23T16:43:53.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce462d3b:::benchmark_data_smithi188_141483_object76072:head# 2024-04-23T16:43:53.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce489d9d:::benchmark_data_smithi188_141483_object88547:head# 2024-04-23T16:43:53.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce48b079:::benchmark_data_smithi188_141483_object86998:head# 2024-04-23T16:43:53.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce4b6517:::benchmark_data_smithi188_141483_object12909:head# 2024-04-23T16:43:53.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce4c48c7:::benchmark_data_smithi188_141483_object18017:head# 2024-04-23T16:43:53.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.333 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce4d686d:::benchmark_data_smithi188_141483_object50066:head# 2024-04-23T16:43:53.333 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.334 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce4efbdf:::benchmark_data_smithi188_141483_object9574:head# 2024-04-23T16:43:53.334 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.334 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce4f8258:::benchmark_data_smithi188_141483_object88371:head# 2024-04-23T16:43:53.334 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.334 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce4f9f91:::benchmark_data_smithi188_141483_object50108:head# 2024-04-23T16:43:53.334 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.334 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce501117:::benchmark_data_smithi188_141483_object86496:head# 2024-04-23T16:43:53.334 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.334 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce507e33:::benchmark_data_smithi188_141483_object35564:head# 2024-04-23T16:43:53.334 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.334 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce50f790:::benchmark_data_smithi188_141483_object15511:head# 2024-04-23T16:43:53.334 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.334 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce510a81:::benchmark_data_smithi188_141483_object26961:head# 2024-04-23T16:43:53.335 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.335 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce5223c6:::benchmark_data_smithi188_141483_object68272:head# 2024-04-23T16:43:53.335 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.335 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce52a351:::benchmark_data_smithi188_141483_object2045:head# 2024-04-23T16:43:53.335 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.335 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce5320eb:::benchmark_data_smithi188_141483_object83112:head# 2024-04-23T16:43:53.335 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.335 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce53261d:::benchmark_data_smithi188_141483_object36790:head# 2024-04-23T16:43:53.335 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.335 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce55d040:::benchmark_data_smithi188_141483_object23560:head# 2024-04-23T16:43:53.335 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.335 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce55dfe9:::benchmark_data_smithi188_141483_object5038:head# 2024-04-23T16:43:53.335 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.336 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce582a25:::benchmark_data_smithi188_141483_object432:head# 2024-04-23T16:43:53.336 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.336 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce586ccf:::benchmark_data_smithi188_141483_object78369:head# 2024-04-23T16:43:53.336 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.336 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce58e9d2:::benchmark_data_smithi188_141483_object1818:head# 2024-04-23T16:43:53.336 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.336 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce5afce0:::benchmark_data_smithi188_141483_object5729:head# 2024-04-23T16:43:53.336 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.336 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce5c1bc3:::benchmark_data_smithi188_141483_object7125:head# 2024-04-23T16:43:53.336 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.336 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce5c4125:::benchmark_data_smithi188_141483_object15507:head# 2024-04-23T16:43:53.336 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.337 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce5c730b:::benchmark_data_smithi188_141483_object79833:head# 2024-04-23T16:43:53.337 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.337 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce5c87d4:::benchmark_data_smithi188_141483_object31458:head# 2024-04-23T16:43:53.337 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.337 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce5d20a0:::benchmark_data_smithi188_141483_object40411:head# 2024-04-23T16:43:53.337 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.337 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce5d251b:::benchmark_data_smithi188_141483_object78612:head# 2024-04-23T16:43:53.337 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.337 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce5d8107:::benchmark_data_smithi188_141483_object20281:head# 2024-04-23T16:43:53.337 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.337 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce5df78c:::benchmark_data_smithi188_141483_object72704:head# 2024-04-23T16:43:53.337 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.338 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce5e5288:::benchmark_data_smithi188_141483_object5672:head# 2024-04-23T16:43:53.338 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.338 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce61b337:::benchmark_data_smithi188_141483_object14023:head# 2024-04-23T16:43:53.338 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.338 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce627702:::benchmark_data_smithi188_141483_object68219:head# 2024-04-23T16:43:53.338 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.338 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce628fb1:::benchmark_data_smithi188_141483_object21321:head# 2024-04-23T16:43:53.338 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.338 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce6327e3:::benchmark_data_smithi188_141483_object53077:head# 2024-04-23T16:43:53.338 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.338 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce636422:::benchmark_data_smithi188_141483_object35975:head# 2024-04-23T16:43:53.339 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.339 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce6415c3:::benchmark_data_smithi188_141483_object23096:head# 2024-04-23T16:43:53.339 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.339 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce641cfc:::benchmark_data_smithi188_141483_object54195:head# 2024-04-23T16:43:53.339 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.339 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce6428f1:::benchmark_data_smithi188_141483_object39319:head# 2024-04-23T16:43:53.339 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.339 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce650b8b:::benchmark_data_smithi188_141483_object32518:head# 2024-04-23T16:43:53.339 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.339 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce658ad8:::benchmark_data_smithi188_141483_object23500:head# 2024-04-23T16:43:53.339 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.339 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce659ef3:::benchmark_data_smithi188_141483_object16736:head# 2024-04-23T16:43:53.339 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce661437:::benchmark_data_smithi188_141483_object1150:head# 2024-04-23T16:43:53.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce665134:::benchmark_data_smithi188_141483_object81047:head# 2024-04-23T16:43:53.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce6678b8:::benchmark_data_smithi188_141483_object5969:head# 2024-04-23T16:43:53.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce66ddaf:::benchmark_data_smithi188_141483_object24441:head# 2024-04-23T16:43:53.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce671999:::benchmark_data_smithi188_141483_object6274:head# 2024-04-23T16:43:53.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce675513:::benchmark_data_smithi188_141483_object27399:head# 2024-04-23T16:43:53.340 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.340 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce68dccd:::benchmark_data_smithi188_141483_object86338:head# 2024-04-23T16:43:53.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.341 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce69afa4:::benchmark_data_smithi188_141483_object81973:head# 2024-04-23T16:43:53.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.341 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce6a106f:::benchmark_data_smithi188_141483_object32036:head# 2024-04-23T16:43:53.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.341 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce6ab202:::benchmark_data_smithi188_141483_object60165:head# 2024-04-23T16:43:53.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.341 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce6b03a2:::benchmark_data_smithi188_141483_object83973:head# 2024-04-23T16:43:53.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.341 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce6c125a:::benchmark_data_smithi188_141483_object60839:head# 2024-04-23T16:43:53.341 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.341 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce6c1bbf:::benchmark_data_smithi188_141483_object81254:head# 2024-04-23T16:43:53.342 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.342 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce6cf0e4:::benchmark_data_smithi188_141483_object37505:head# 2024-04-23T16:43:53.342 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.342 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce6d6923:::benchmark_data_smithi188_141483_object85479:head# 2024-04-23T16:43:53.342 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.342 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce6e3af1:::benchmark_data_smithi188_141483_object66603:head# 2024-04-23T16:43:53.342 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.342 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce6f2b56:::benchmark_data_smithi188_141483_object43172:head# 2024-04-23T16:43:53.342 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.342 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce710be8:::benchmark_data_smithi188_141483_object67246:head# 2024-04-23T16:43:53.342 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.342 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce713708:::benchmark_data_smithi188_141483_object61726:head# 2024-04-23T16:43:53.343 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.343 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce714c48:::benchmark_data_smithi188_141483_object43869:head# 2024-04-23T16:43:53.343 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.343 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce71c911:::benchmark_data_smithi188_141483_object41433:head# 2024-04-23T16:43:53.343 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.343 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce723227:::benchmark_data_smithi188_141483_object13771:head# 2024-04-23T16:43:53.343 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.343 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce72ab4a:::benchmark_data_smithi188_141483_object22939:head# 2024-04-23T16:43:53.343 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.343 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce73f200:::benchmark_data_smithi188_141483_object84405:head# 2024-04-23T16:43:53.343 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.343 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce73f7d5:::benchmark_data_smithi188_141483_object60139:head# 2024-04-23T16:43:53.343 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.344 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce7467d7:::benchmark_data_smithi188_141483_object34096:head# 2024-04-23T16:43:53.344 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.344 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce74c6cb:::benchmark_data_smithi188_141483_object39798:head# 2024-04-23T16:43:53.344 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.344 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce76edc3:::benchmark_data_smithi188_141483_object77533:head# 2024-04-23T16:43:53.344 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.344 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce773d82:::benchmark_data_smithi188_141483_object70248:head# 2024-04-23T16:43:53.344 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.344 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce777b88:::benchmark_data_smithi188_141483_object42239:head# 2024-04-23T16:43:53.344 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.344 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce78ca18:::benchmark_data_smithi188_141483_object89948:head# 2024-04-23T16:43:53.344 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.344 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce79005e:::benchmark_data_smithi188_141483_object53332:head# 2024-04-23T16:43:53.344 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.345 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce799014:::benchmark_data_smithi188_141483_object17163:head# 2024-04-23T16:43:53.345 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.345 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce7a48b4:::benchmark_data_smithi188_141483_object65704:head# 2024-04-23T16:43:53.345 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.345 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce7acbf1:::benchmark_data_smithi188_141483_object53145:head# 2024-04-23T16:43:53.345 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.345 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce7b7978:::benchmark_data_smithi188_141483_object59684:head# 2024-04-23T16:43:53.345 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.345 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce7bd735:::benchmark_data_smithi188_141483_object71800:head# 2024-04-23T16:43:53.345 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.345 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce7d0178:::benchmark_data_smithi188_141483_object20701:head# 2024-04-23T16:43:53.345 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.345 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce7f6686:::benchmark_data_smithi188_141483_object90884:head# 2024-04-23T16:43:53.346 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.346 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce7f85b7:::benchmark_data_smithi188_141483_object34937:head# 2024-04-23T16:43:53.346 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.346 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce804367:::benchmark_data_smithi188_141483_object68208:head# 2024-04-23T16:43:53.346 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.346 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce80a16c:::benchmark_data_smithi188_141483_object66660:head# 2024-04-23T16:43:53.346 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.346 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce80e25d:::benchmark_data_smithi188_141483_object57641:head# 2024-04-23T16:43:53.346 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.346 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce81dd7b:::benchmark_data_smithi188_141483_object46763:head# 2024-04-23T16:43:53.346 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.346 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce838a4e:::benchmark_data_smithi188_141483_object5624:head# 2024-04-23T16:43:53.347 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.347 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce847427:::benchmark_data_smithi188_141483_object68641:head# 2024-04-23T16:43:53.347 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.347 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce84b502:::benchmark_data_smithi188_141483_object1165:head# 2024-04-23T16:43:53.347 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.347 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce84e2dc:::benchmark_data_smithi188_141483_object51563:head# 2024-04-23T16:43:53.347 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.347 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce85c12f:::benchmark_data_smithi188_141483_object57637:head# 2024-04-23T16:43:53.347 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.347 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce85d3aa:::benchmark_data_smithi188_141483_object19704:head# 2024-04-23T16:43:53.347 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.347 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce85e564:::benchmark_data_smithi188_141483_object24055:head# 2024-04-23T16:43:53.347 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.347 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce86f332:::benchmark_data_smithi188_141483_object7974:head# 2024-04-23T16:43:53.347 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.348 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce879fc7:::benchmark_data_smithi188_141483_object53127:head# 2024-04-23T16:43:53.348 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.348 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce87db45:::benchmark_data_smithi188_141483_object77925:head# 2024-04-23T16:43:53.348 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.348 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce88329d:::benchmark_data_smithi188_141483_object47587:head# 2024-04-23T16:43:53.348 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.348 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce88a3a2:::benchmark_data_smithi188_141483_object30955:head# 2024-04-23T16:43:53.348 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.348 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce88e2bf:::benchmark_data_smithi188_141483_object47428:head# 2024-04-23T16:43:53.348 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.348 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce8942a3:::benchmark_data_smithi188_141483_object22944:head# 2024-04-23T16:43:53.348 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.348 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce8942ed:::benchmark_data_smithi188_141483_object4849:head# 2024-04-23T16:43:53.349 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.349 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce8948c6:::benchmark_data_smithi188_141483_object17791:head# 2024-04-23T16:43:53.349 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.349 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce89ee84:::benchmark_data_smithi188_141483_object68354:head# 2024-04-23T16:43:53.349 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.349 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce8a2d6a:::benchmark_data_smithi188_141483_object88395:head# 2024-04-23T16:43:53.349 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.349 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce8ad47c:::benchmark_data_smithi188_141483_object21381:head# 2024-04-23T16:43:53.349 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.349 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce8b1351:::benchmark_data_smithi188_141483_object78995:head# 2024-04-23T16:43:53.349 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.349 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce8b6efe:::benchmark_data_smithi188_141483_object34113:head# 2024-04-23T16:43:53.349 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.349 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce8bb98a:::benchmark_data_smithi188_141483_object15006:head# 2024-04-23T16:43:53.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce8c0792:::benchmark_data_smithi188_141483_object83119:head# 2024-04-23T16:43:53.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce8c6e86:::benchmark_data_smithi188_141483_object30509:head# 2024-04-23T16:43:53.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce8dde34:::benchmark_data_smithi188_141483_object89431:head# 2024-04-23T16:43:53.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce8f5368:::benchmark_data_smithi188_141483_object20558:head# 2024-04-23T16:43:53.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce8f5df2:::benchmark_data_smithi188_141483_object18261:head# 2024-04-23T16:43:53.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce8f66d6:::benchmark_data_smithi188_141483_object68151:head# 2024-04-23T16:43:53.350 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.350 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce910a24:::benchmark_data_smithi188_141483_object18982:head# 2024-04-23T16:43:53.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce917958:::benchmark_data_smithi188_141483_object51466:head# 2024-04-23T16:43:53.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce91f1bd:::benchmark_data_smithi188_141483_object57330:head# 2024-04-23T16:43:53.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce9266e3:::benchmark_data_smithi188_141483_object35805:head# 2024-04-23T16:43:53.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce929ff2:::benchmark_data_smithi188_141483_object23128:head# 2024-04-23T16:43:53.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce93335f:::benchmark_data_smithi188_141483_object43290:head# 2024-04-23T16:43:53.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce93f51c:::benchmark_data_smithi188_141483_object36108:head# 2024-04-23T16:43:53.351 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.351 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce94c529:::benchmark_data_smithi188_141483_object21730:head# 2024-04-23T16:43:53.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce963831:::benchmark_data_smithi188_141483_object34558:head# 2024-04-23T16:43:53.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce971600:::benchmark_data_smithi188_141483_object65124:head# 2024-04-23T16:43:53.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce979be5:::benchmark_data_smithi188_141483_object51405:head# 2024-04-23T16:43:53.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce980428:::benchmark_data_smithi188_141483_object18553:head# 2024-04-23T16:43:53.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce985a59:::benchmark_data_smithi188_141483_object9205:head# 2024-04-23T16:43:53.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce99913e:::benchmark_data_smithi188_141483_object80831:head# 2024-04-23T16:43:53.352 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.352 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce9a2c71:::benchmark_data_smithi188_141483_object6372:head# 2024-04-23T16:43:53.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce9a4b74:::benchmark_data_smithi188_141483_object90657:head# 2024-04-23T16:43:53.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce9a878f:::benchmark_data_smithi188_141483_object53548:head# 2024-04-23T16:43:53.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce9a978d:::benchmark_data_smithi188_141483_object81233:head# 2024-04-23T16:43:53.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce9ba3a0:::benchmark_data_smithi188_141483_object38186:head# 2024-04-23T16:43:53.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce9bdcff:::benchmark_data_smithi188_141483_object51166:head# 2024-04-23T16:43:53.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce9c215b:::benchmark_data_smithi188_141483_object9319:head# 2024-04-23T16:43:53.353 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.353 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce9cb3ba:::benchmark_data_smithi188_141483_object28363:head# 2024-04-23T16:43:53.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce9cb91d:::benchmark_data_smithi188_141483_object77367:head# 2024-04-23T16:43:53.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce9efc47:::benchmark_data_smithi188_141483_object62990:head# 2024-04-23T16:43:53.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ce9f9f3c:::benchmark_data_smithi188_141483_object65352:head# 2024-04-23T16:43:53.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cea05587:::benchmark_data_smithi188_141483_object58815:head# 2024-04-23T16:43:53.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cea072ee:::benchmark_data_smithi188_141483_object55836:head# 2024-04-23T16:43:53.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.354 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cea1387e:::benchmark_data_smithi188_141483_object89766:head# 2024-04-23T16:43:53.354 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cea203a9:::benchmark_data_smithi188_141483_object76408:head# 2024-04-23T16:43:53.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cea23646:::benchmark_data_smithi188_141483_object37050:head# 2024-04-23T16:43:53.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cea30394:::benchmark_data_smithi188_141483_object37921:head# 2024-04-23T16:43:53.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cea41d23:::benchmark_data_smithi188_141483_object22756:head# 2024-04-23T16:43:53.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cea432e5:::benchmark_data_smithi188_141483_object73688:head# 2024-04-23T16:43:53.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cea46f77:::benchmark_data_smithi188_141483_object2487:head# 2024-04-23T16:43:53.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.355 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cea5458c:::benchmark_data_smithi188_141483_object26083:head# 2024-04-23T16:43:53.355 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cea648fd:::benchmark_data_smithi188_141483_object15666:head# 2024-04-23T16:43:53.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cea76907:::benchmark_data_smithi188_141483_object30072:head# 2024-04-23T16:43:53.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cea79259:::benchmark_data_smithi188_141483_object2950:head# 2024-04-23T16:43:53.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cea92b4f:::benchmark_data_smithi188_141483_object58397:head# 2024-04-23T16:43:53.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cea98cf8:::benchmark_data_smithi188_141483_object70344:head# 2024-04-23T16:43:53.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.356 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceaa02e7:::benchmark_data_smithi188_141483_object58133:head# 2024-04-23T16:43:53.356 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceaafbfa:::benchmark_data_smithi188_141483_object41628:head# 2024-04-23T16:43:53.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceab2f14:::benchmark_data_smithi188_141483_object71126:head# 2024-04-23T16:43:53.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceac177a:::benchmark_data_smithi188_141483_object35924:head# 2024-04-23T16:43:53.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cead2ffb:::benchmark_data_smithi188_141483_object90379:head# 2024-04-23T16:43:53.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cead8f1f:::benchmark_data_smithi188_141483_object62511:head# 2024-04-23T16:43:53.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceae733f:::benchmark_data_smithi188_141483_object46263:head# 2024-04-23T16:43:53.357 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.357 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceaeb864:::benchmark_data_smithi188_141483_object13301:head# 2024-04-23T16:43:53.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceaef81b:::benchmark_data_smithi188_141483_object18062:head# 2024-04-23T16:43:53.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceaf5721:::benchmark_data_smithi188_141483_object45469:head# 2024-04-23T16:43:53.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceaf9cde:::benchmark_data_smithi188_141483_object44686:head# 2024-04-23T16:43:53.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb05afa:::benchmark_data_smithi188_141483_object72049:head# 2024-04-23T16:43:53.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb05d63:::benchmark_data_smithi188_141483_object61345:head# 2024-04-23T16:43:53.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.358 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb0cc52:::benchmark_data_smithi188_141483_object51010:head# 2024-04-23T16:43:53.358 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb110b8:::benchmark_data_smithi188_141483_object90627:head# 2024-04-23T16:43:53.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb2190c:::benchmark_data_smithi188_141483_object1118:head# 2024-04-23T16:43:53.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb24baf:::benchmark_data_smithi188_141483_object76780:head# 2024-04-23T16:43:53.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb2cd76:::benchmark_data_smithi188_141483_object24961:head# 2024-04-23T16:43:53.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb30821:::benchmark_data_smithi188_141483_object28241:head# 2024-04-23T16:43:53.359 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.359 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb3acaf:::benchmark_data_smithi188_141483_object16075:head# 2024-04-23T16:43:53.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb3db8d:::benchmark_data_smithi188_141483_object88375:head# 2024-04-23T16:43:53.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb43a4a:::benchmark_data_smithi188_141483_object18760:head# 2024-04-23T16:43:53.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb552f6:::benchmark_data_smithi188_141483_object21850:head# 2024-04-23T16:43:53.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb6690b:::benchmark_data_smithi188_141483_object26028:head# 2024-04-23T16:43:53.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.360 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb69f88:::benchmark_data_smithi188_141483_object10025:head# 2024-04-23T16:43:53.360 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb6ac95:::benchmark_data_smithi188_141483_object77054:head# 2024-04-23T16:43:53.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb7d312:::benchmark_data_smithi188_141483_object25135:head# 2024-04-23T16:43:53.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb80b1c:::benchmark_data_smithi188_141483_object83050:head# 2024-04-23T16:43:53.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb80e2d:::benchmark_data_smithi188_141483_object73970:head# 2024-04-23T16:43:53.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb86f41:::benchmark_data_smithi188_141483_object43560:head# 2024-04-23T16:43:53.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceb9ebe3:::benchmark_data_smithi188_141483_object41305:head# 2024-04-23T16:43:53.361 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.361 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceba1a4d:::benchmark_data_smithi188_141483_object43526:head# 2024-04-23T16:43:53.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cebac2ad:::benchmark_data_smithi188_141483_object29446:head# 2024-04-23T16:43:53.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cebc804a:::benchmark_data_smithi188_141483_object88412:head# 2024-04-23T16:43:53.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cebcb243:::benchmark_data_smithi188_141483_object81516:head# 2024-04-23T16:43:53.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cebee301:::benchmark_data_smithi188_141483_object58580:head# 2024-04-23T16:43:53.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cebf47ec:::benchmark_data_smithi188_141483_object44748:head# 2024-04-23T16:43:53.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.362 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cebf9643:::benchmark_data_smithi188_141483_object42145:head# 2024-04-23T16:43:53.362 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cebfe7ae:::benchmark_data_smithi188_141483_object8977:head# 2024-04-23T16:43:53.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec0b8ad:::benchmark_data_smithi188_141483_object12504:head# 2024-04-23T16:43:53.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec0fcf6:::benchmark_data_smithi188_141483_object67899:head# 2024-04-23T16:43:53.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec15535:::benchmark_data_smithi188_141483_object14984:head# 2024-04-23T16:43:53.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec173f4:::benchmark_data_smithi188_141483_object22029:head# 2024-04-23T16:43:53.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec1bc08:::benchmark_data_smithi188_141483_object13221:head# 2024-04-23T16:43:53.363 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.363 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec2320d:::benchmark_data_smithi188_141483_object23481:head# 2024-04-23T16:43:53.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec29c5b:::benchmark_data_smithi188_141483_object81983:head# 2024-04-23T16:43:53.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec2de8f:::benchmark_data_smithi188_141483_object3567:head# 2024-04-23T16:43:53.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec30aea:::benchmark_data_smithi188_141483_object52792:head# 2024-04-23T16:43:53.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec36fd5:::benchmark_data_smithi188_141483_object57365:head# 2024-04-23T16:43:53.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec37762:::benchmark_data_smithi188_141483_object79931:head# 2024-04-23T16:43:53.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec37949:::benchmark_data_smithi188_141483_object12821:head# 2024-04-23T16:43:53.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.364 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec47afe:::benchmark_data_smithi188_141483_object89952:head# 2024-04-23T16:43:53.364 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec5245d:::benchmark_data_smithi188_141483_object66562:head# 2024-04-23T16:43:53.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec577a5:::benchmark_data_smithi188_141483_object70000:head# 2024-04-23T16:43:53.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec5c440:::benchmark_data_smithi188_141483_object32850:head# 2024-04-23T16:43:53.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec60be1:::benchmark_data_smithi188_141483_object4827:head# 2024-04-23T16:43:53.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec61d22:::benchmark_data_smithi188_141483_object16689:head# 2024-04-23T16:43:53.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec69c88:::benchmark_data_smithi188_141483_object42340:head# 2024-04-23T16:43:53.365 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.365 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec6fc09:::benchmark_data_smithi188_141483_object458:head# 2024-04-23T16:43:53.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec7d85f:::benchmark_data_smithi188_141483_object4004:head# 2024-04-23T16:43:53.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec823a5:::benchmark_data_smithi188_141483_object54100:head# 2024-04-23T16:43:53.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cec8f1c9:::benchmark_data_smithi188_141483_object21293:head# 2024-04-23T16:43:53.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceca6c14:::benchmark_data_smithi188_141483_object19909:head# 2024-04-23T16:43:53.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cecb17ec:::benchmark_data_smithi188_141483_object22720:head# 2024-04-23T16:43:53.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.366 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cecb981f:::benchmark_data_smithi188_141483_object6433:head# 2024-04-23T16:43:53.366 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cecc087c:::benchmark_data_smithi188_141483_object37544:head# 2024-04-23T16:43:53.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cecd5586:::benchmark_data_smithi188_141483_object84064:head# 2024-04-23T16:43:53.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cecd7d73:::benchmark_data_smithi188_141483_object79784:head# 2024-04-23T16:43:53.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cece022a:::benchmark_data_smithi188_141483_object57339:head# 2024-04-23T16:43:53.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cecef57a:::benchmark_data_smithi188_141483_object4045:head# 2024-04-23T16:43:53.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cecf474e:::benchmark_data_smithi188_141483_object23742:head# 2024-04-23T16:43:53.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.367 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cecfa06a:::benchmark_data_smithi188_141483_object59910:head# 2024-04-23T16:43:53.367 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ced0403b:::benchmark_data_smithi188_141483_object31604:head# 2024-04-23T16:43:53.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ced09dfe:::benchmark_data_smithi188_141483_object62216:head# 2024-04-23T16:43:53.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ced1ebb6:::benchmark_data_smithi188_141483_object82246:head# 2024-04-23T16:43:53.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ced2122e:::benchmark_data_smithi188_141483_object7880:head# 2024-04-23T16:43:53.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ced38f46:::benchmark_data_smithi188_141483_object50989:head# 2024-04-23T16:43:53.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ced405b0:::benchmark_data_smithi188_141483_object75615:head# 2024-04-23T16:43:53.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.368 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ced60e86:::benchmark_data_smithi188_141483_object38405:head# 2024-04-23T16:43:53.368 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ced81ea9:::benchmark_data_smithi188_141483_object16092:head# 2024-04-23T16:43:53.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ced89613:::benchmark_data_smithi188_141483_object87719:head# 2024-04-23T16:43:53.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ced8cc0b:::benchmark_data_smithi188_141483_object31399:head# 2024-04-23T16:43:53.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceda6708:::benchmark_data_smithi188_141483_object62031:head# 2024-04-23T16:43:53.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cedb942d:::benchmark_data_smithi188_141483_object7757:head# 2024-04-23T16:43:53.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cedc6ded:::benchmark_data_smithi188_141483_object88003:head# 2024-04-23T16:43:53.369 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.369 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cedca4bf:::benchmark_data_smithi188_141483_object71485:head# 2024-04-23T16:43:53.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cedd99e2:::benchmark_data_smithi188_141483_object21978:head# 2024-04-23T16:43:53.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cede0a0e:::benchmark_data_smithi188_141483_object11053:head# 2024-04-23T16:43:53.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cedee90a:::benchmark_data_smithi188_141483_object78056:head# 2024-04-23T16:43:53.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cedfce81:::benchmark_data_smithi188_141483_object83145:head# 2024-04-23T16:43:53.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cee192f2:::benchmark_data_smithi188_141483_object59617:head# 2024-04-23T16:43:53.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cee1dcb6:::benchmark_data_smithi188_141483_object80741:head# 2024-04-23T16:43:53.370 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.370 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cee2f8a1:::benchmark_data_smithi188_141483_object16621:head# 2024-04-23T16:43:53.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cee39187:::benchmark_data_smithi188_141483_object20624:head# 2024-04-23T16:43:53.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cee49aee:::benchmark_data_smithi188_141483_object4091:head# 2024-04-23T16:43:53.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cee5ca0b:::benchmark_data_smithi188_141483_object40058:head# 2024-04-23T16:43:53.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cee6f198:::benchmark_data_smithi188_141483_object36837:head# 2024-04-23T16:43:53.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cee76e98:::benchmark_data_smithi188_141483_object9044:head# 2024-04-23T16:43:53.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cee925f9:::benchmark_data_smithi188_141483_object86562:head# 2024-04-23T16:43:53.371 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.371 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cee9695d:::benchmark_data_smithi188_141483_object25153:head# 2024-04-23T16:43:53.372 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.372 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cee98b80:::benchmark_data_smithi188_141483_object79146:head# 2024-04-23T16:43:53.372 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.372 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceeada16:::benchmark_data_smithi188_141483_object69985:head# 2024-04-23T16:43:53.372 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.372 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceeb9e79:::benchmark_data_smithi188_141483_object83885:head# 2024-04-23T16:43:53.372 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.372 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceec89f0:::benchmark_data_smithi188_141483_object74964:head# 2024-04-23T16:43:53.372 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.372 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceedfb6e:::benchmark_data_smithi188_141483_object76880:head# 2024-04-23T16:43:53.372 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.372 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceee5bae:::benchmark_data_smithi188_141483_object57552:head# 2024-04-23T16:43:53.372 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.372 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceef0967:::benchmark_data_smithi188_141483_object58272:head# 2024-04-23T16:43:53.373 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.373 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceef5272:::benchmark_data_smithi188_141483_object4228:head# 2024-04-23T16:43:53.373 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.373 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceef59f6:::benchmark_data_smithi188_141483_object87962:head# 2024-04-23T16:43:53.373 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.373 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cef0e475:::benchmark_data_smithi188_141483_object21289:head# 2024-04-23T16:43:53.373 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.373 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cef2b49c:::benchmark_data_smithi188_141483_object43607:head# 2024-04-23T16:43:53.373 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.373 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cef35a32:::benchmark_data_smithi188_141483_object88089:head# 2024-04-23T16:43:53.373 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.373 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cef42549:::benchmark_data_smithi188_141483_object50650:head# 2024-04-23T16:43:53.373 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.373 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cef46710:::benchmark_data_smithi188_141483_object46600:head# 2024-04-23T16:43:53.374 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.374 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cef5d4ab:::benchmark_data_smithi188_141483_object22551:head# 2024-04-23T16:43:53.374 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.374 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cef607a2:::benchmark_data_smithi188_141483_object38696:head# 2024-04-23T16:43:53.374 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.374 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cef615f3:::benchmark_data_smithi188_141483_object86273:head# 2024-04-23T16:43:53.374 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.374 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cef6c3e2:::benchmark_data_smithi188_141483_object87246:head# 2024-04-23T16:43:53.374 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.374 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cef6e2a2:::benchmark_data_smithi188_141483_object69409:head# 2024-04-23T16:43:53.374 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.374 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cef70a51:::benchmark_data_smithi188_141483_object77153:head# 2024-04-23T16:43:53.374 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cef9b042:::benchmark_data_smithi188_141483_object26899:head# 2024-04-23T16:43:53.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cefa3c6d:::benchmark_data_smithi188_141483_object58868:head# 2024-04-23T16:43:53.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cefa512b:::benchmark_data_smithi188_141483_object19429:head# 2024-04-23T16:43:53.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cefa8b77:::benchmark_data_smithi188_141483_object80796:head# 2024-04-23T16:43:53.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cefd8f11:::benchmark_data_smithi188_141483_object54241:head# 2024-04-23T16:43:53.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.375 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cefe0940:::benchmark_data_smithi188_141483_object30366:head# 2024-04-23T16:43:53.375 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceff16ff:::benchmark_data_smithi188_141483_object51793:head# 2024-04-23T16:43:53.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceff1be4:::benchmark_data_smithi188_141483_object9888:head# 2024-04-23T16:43:53.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:ceff92c9:::benchmark_data_smithi188_141483_object45848:head# 2024-04-23T16:43:53.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf00d345:::benchmark_data_smithi188_141483_object60506:head# 2024-04-23T16:43:53.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf010fee:::benchmark_data_smithi188_141483_object47211:head# 2024-04-23T16:43:53.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf03a1fd:::benchmark_data_smithi188_141483_object51768:head# 2024-04-23T16:43:53.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.376 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf041ad5:::benchmark_data_smithi188_141483_object87415:head# 2024-04-23T16:43:53.376 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf053d5b:::benchmark_data_smithi188_141483_object70131:head# 2024-04-23T16:43:53.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf056383:::benchmark_data_smithi188_141483_object26387:head# 2024-04-23T16:43:53.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf05c08b:::benchmark_data_smithi188_141483_object26279:head# 2024-04-23T16:43:53.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf0624f9:::benchmark_data_smithi188_141483_object32973:head# 2024-04-23T16:43:53.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf063c29:::benchmark_data_smithi188_141483_object6061:head# 2024-04-23T16:43:53.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf065e93:::benchmark_data_smithi188_141483_object29597:head# 2024-04-23T16:43:53.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.377 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf066498:::benchmark_data_smithi188_141483_object9561:head# 2024-04-23T16:43:53.377 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.378 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf07215c:::benchmark_data_smithi188_141483_object19365:head# 2024-04-23T16:43:53.378 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.378 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf07c4d7:::benchmark_data_smithi188_141483_object71063:head# 2024-04-23T16:43:53.378 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.378 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf083b4b:::benchmark_data_smithi188_141483_object49414:head# 2024-04-23T16:43:53.378 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.378 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf083f84:::benchmark_data_smithi188_141483_object70230:head# 2024-04-23T16:43:53.378 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.378 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf092412:::benchmark_data_smithi188_141483_object24729:head# 2024-04-23T16:43:53.378 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.378 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf09e104:::benchmark_data_smithi188_141483_object22541:head# 2024-04-23T16:43:53.378 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.378 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf09e91b:::benchmark_data_smithi188_141483_object35652:head# 2024-04-23T16:43:53.379 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf0a4824:::benchmark_data_smithi188_141483_object59553:head# 2024-04-23T16:43:53.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf0a8361:::benchmark_data_smithi188_141483_object11851:head# 2024-04-23T16:43:53.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf0b2678:::benchmark_data_smithi188_141483_object22024:head# 2024-04-23T16:43:53.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf0b697b:::benchmark_data_smithi188_141483_object13687:head# 2024-04-23T16:43:53.381 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.381 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf0bf7bb:::benchmark_data_smithi188_141483_object31169:head# 2024-04-23T16:43:53.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf0c37aa:::benchmark_data_smithi188_141483_object26253:head# 2024-04-23T16:43:53.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf0c7386:::benchmark_data_smithi188_141483_object58912:head# 2024-04-23T16:43:53.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf0c990a:::benchmark_data_smithi188_141483_object15206:head# 2024-04-23T16:43:53.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf0cbc7f:::benchmark_data_smithi188_141483_object47233:head# 2024-04-23T16:43:53.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf0d4907:::benchmark_data_smithi188_141483_object13481:head# 2024-04-23T16:43:53.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.382 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf0d9c2b:::benchmark_data_smithi188_141483_object20251:head# 2024-04-23T16:43:53.382 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf0e3f7b:::benchmark_data_smithi188_141483_object88951:head# 2024-04-23T16:43:53.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf0eceb5:::benchmark_data_smithi188_141483_object81909:head# 2024-04-23T16:43:53.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf0f2903:::benchmark_data_smithi188_141483_object34002:head# 2024-04-23T16:43:53.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf110e09:::benchmark_data_smithi188_141483_object6373:head# 2024-04-23T16:43:53.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf1193cf:::benchmark_data_smithi188_141483_object79354:head# 2024-04-23T16:43:53.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf11b90b:::benchmark_data_smithi188_141483_object9011:head# 2024-04-23T16:43:53.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.383 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf133ba5:::benchmark_data_smithi188_141483_object5015:head# 2024-04-23T16:43:53.383 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf153e8a:::benchmark_data_smithi188_141483_object36692:head# 2024-04-23T16:43:53.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf158d8e:::benchmark_data_smithi188_141483_object3621:head# 2024-04-23T16:43:53.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf16e201:::benchmark_data_smithi188_141483_object40240:head# 2024-04-23T16:43:53.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf174e80:::benchmark_data_smithi188_141483_object17437:head# 2024-04-23T16:43:53.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf1c549f:::benchmark_data_smithi188_141483_object63214:head# 2024-04-23T16:43:53.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf1cdd02:::benchmark_data_smithi188_141483_object14504:head# 2024-04-23T16:43:53.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf1d4548:::benchmark_data_smithi188_141483_object55038:head# 2024-04-23T16:43:53.384 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.384 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf1d570c:::benchmark_data_smithi188_141483_object80441:head# 2024-04-23T16:43:53.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf1e4be6:::benchmark_data_smithi188_141483_object69647:head# 2024-04-23T16:43:53.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf1ee73d:::benchmark_data_smithi188_141483_object14329:head# 2024-04-23T16:43:53.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf1f0a20:::benchmark_data_smithi188_141483_object21111:head# 2024-04-23T16:43:53.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf20522a:::benchmark_data_smithi188_141483_object13065:head# 2024-04-23T16:43:53.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.385 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf20f812:::benchmark_data_smithi188_141483_object42631:head# 2024-04-23T16:43:53.385 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf214f14:::benchmark_data_smithi188_141483_object16691:head# 2024-04-23T16:43:53.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf21ec4b:::benchmark_data_smithi188_141483_object33981:head# 2024-04-23T16:43:53.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf235ed3:::benchmark_data_smithi188_141483_object46005:head# 2024-04-23T16:43:53.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf23e7b2:::benchmark_data_smithi188_141483_object12011:head# 2024-04-23T16:43:53.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf250221:::benchmark_data_smithi188_141483_object21514:head# 2024-04-23T16:43:53.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf25774a:::benchmark_data_smithi188_141483_object5830:head# 2024-04-23T16:43:53.386 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.386 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf25a4fe:::benchmark_data_smithi188_141483_object1049:head# 2024-04-23T16:43:53.387 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.387 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf2687cf:::benchmark_data_smithi188_141483_object26300:head# 2024-04-23T16:43:53.387 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.387 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf2706a8:::benchmark_data_smithi188_141483_object76183:head# 2024-04-23T16:43:53.387 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.387 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf27ba45:::benchmark_data_smithi188_141483_object77106:head# 2024-04-23T16:43:53.387 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.387 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf2802db:::benchmark_data_smithi188_141483_object78073:head# 2024-04-23T16:43:53.387 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.387 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf288dee:::benchmark_data_smithi188_141483_object46274:head# 2024-04-23T16:43:53.387 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.387 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf28a44f:::benchmark_data_smithi188_141483_object45063:head# 2024-04-23T16:43:53.387 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.387 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf28d711:::benchmark_data_smithi188_141483_object72496:head# 2024-04-23T16:43:53.387 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf29795a:::benchmark_data_smithi188_141483_object24617:head# 2024-04-23T16:43:53.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf2b45eb:::benchmark_data_smithi188_141483_object80457:head# 2024-04-23T16:43:53.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf2bb96a:::benchmark_data_smithi188_141483_object27083:head# 2024-04-23T16:43:53.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf2c341d:::benchmark_data_smithi188_141483_object75879:head# 2024-04-23T16:43:53.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf2cd6a5:::benchmark_data_smithi188_141483_object47107:head# 2024-04-23T16:43:53.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf2d6e10:::benchmark_data_smithi188_141483_object47741:head# 2024-04-23T16:43:53.388 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.388 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf2d9925:::benchmark_data_smithi188_141483_object61115:head# 2024-04-23T16:43:53.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf3029ea:::benchmark_data_smithi188_141483_object70088:head# 2024-04-23T16:43:53.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf30be71:::benchmark_data_smithi188_141483_object78368:head# 2024-04-23T16:43:53.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf30f3f3:::benchmark_data_smithi188_141483_object44917:head# 2024-04-23T16:43:53.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf317574:::benchmark_data_smithi188_141483_object5100:head# 2024-04-23T16:43:53.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf31cd63:::benchmark_data_smithi188_141483_object85809:head# 2024-04-23T16:43:53.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf329131:::benchmark_data_smithi188_141483_object29100:head# 2024-04-23T16:43:53.389 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.389 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf32a895:::benchmark_data_smithi188_141483_object11692:head# 2024-04-23T16:43:53.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf33c93b:::benchmark_data_smithi188_141483_object72656:head# 2024-04-23T16:43:53.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf33dc1f:::benchmark_data_smithi188_141483_object86841:head# 2024-04-23T16:43:53.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf3415d3:::benchmark_data_smithi188_141483_object24836:head# 2024-04-23T16:43:53.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf341ea1:::benchmark_data_smithi188_141483_object28353:head# 2024-04-23T16:43:53.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf3480a6:::benchmark_data_smithi188_141483_object43662:head# 2024-04-23T16:43:53.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.390 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf348a11:::benchmark_data_smithi188_141483_object34776:head# 2024-04-23T16:43:53.390 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf34b255:::benchmark_data_smithi188_141483_object10430:head# 2024-04-23T16:43:53.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf34d4dd:::benchmark_data_smithi188_141483_object6649:head# 2024-04-23T16:43:53.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf355013:::benchmark_data_smithi188_141483_object37404:head# 2024-04-23T16:43:53.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf3696f9:::benchmark_data_smithi188_141483_object9804:head# 2024-04-23T16:43:53.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf3921e6:::benchmark_data_smithi188_141483_object27733:head# 2024-04-23T16:43:53.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf398e3d:::benchmark_data_smithi188_141483_object59966:head# 2024-04-23T16:43:53.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.391 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf3be3de:::benchmark_data_smithi188_141483_object65016:head# 2024-04-23T16:43:53.391 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf3bf962:::benchmark_data_smithi188_141483_object38613:head# 2024-04-23T16:43:53.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf3c619a:::benchmark_data_smithi188_141483_object25674:head# 2024-04-23T16:43:53.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf3e1b9e:::benchmark_data_smithi188_141483_object33655:head# 2024-04-23T16:43:53.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf3e78f9:::benchmark_data_smithi188_141483_object53764:head# 2024-04-23T16:43:53.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf3ea086:::benchmark_data_smithi188_141483_object60861:head# 2024-04-23T16:43:53.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf3eb51a:::benchmark_data_smithi188_141483_object70613:head# 2024-04-23T16:43:53.392 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.392 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf4168c8:::benchmark_data_smithi188_141483_object2020:head# 2024-04-23T16:43:53.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf42101b:::benchmark_data_smithi188_141483_object33294:head# 2024-04-23T16:43:53.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf42707f:::benchmark_data_smithi188_141483_object46461:head# 2024-04-23T16:43:53.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf42a50e:::benchmark_data_smithi188_141483_object5067:head# 2024-04-23T16:43:53.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf42bc32:::benchmark_data_smithi188_141483_object6258:head# 2024-04-23T16:43:53.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf42d4e4:::benchmark_data_smithi188_141483_object71671:head# 2024-04-23T16:43:53.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.393 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf4518b7:::benchmark_data_smithi188_141483_object87401:head# 2024-04-23T16:43:53.393 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf453114:::benchmark_data_smithi188_141483_object39465:head# 2024-04-23T16:43:53.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf45cf24:::benchmark_data_smithi188_141483_object79397:head# 2024-04-23T16:43:53.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf4657c8:::benchmark_data_smithi188_141483_object88473:head# 2024-04-23T16:43:53.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf472dd1:::benchmark_data_smithi188_141483_object52862:head# 2024-04-23T16:43:53.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf47b0d5:::benchmark_data_smithi188_141483_object81198:head# 2024-04-23T16:43:53.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.394 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf47b4aa:::benchmark_data_smithi188_141483_object23637:head# 2024-04-23T16:43:53.394 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf4899e6:::benchmark_data_smithi188_141483_object75172:head# 2024-04-23T16:43:53.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf49a805:::benchmark_data_smithi188_141483_object9374:head# 2024-04-23T16:43:53.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf4bbeb3:::benchmark_data_smithi188_141483_object3035:head# 2024-04-23T16:43:53.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf4ca282:::benchmark_data_smithi188_141483_object53517:head# 2024-04-23T16:43:53.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf4e8804:::benchmark_data_smithi188_141483_object45956:head# 2024-04-23T16:43:53.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf4eb321:::benchmark_data_smithi188_141483_object68308:head# 2024-04-23T16:43:53.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.395 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf50b699:::benchmark_data_smithi188_141483_object41531:head# 2024-04-23T16:43:53.395 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf50bf81:::benchmark_data_smithi188_141483_object3752:head# 2024-04-23T16:43:53.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf51d19f:::benchmark_data_smithi188_141483_object86052:head# 2024-04-23T16:43:53.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf531ab8:::benchmark_data_smithi188_141483_object12472:head# 2024-04-23T16:43:53.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf53262a:::benchmark_data_smithi188_141483_object31663:head# 2024-04-23T16:43:53.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf53370e:::benchmark_data_smithi188_141483_object55818:head# 2024-04-23T16:43:53.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf55ad03:::benchmark_data_smithi188_141483_object30893:head# 2024-04-23T16:43:53.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.396 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf56fe7b:::benchmark_data_smithi188_141483_object19017:head# 2024-04-23T16:43:53.396 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf573552:::benchmark_data_smithi188_141483_object11884:head# 2024-04-23T16:43:53.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf58664b:::benchmark_data_smithi188_141483_object83296:head# 2024-04-23T16:43:53.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf58c71d:::benchmark_data_smithi188_141483_object5486:head# 2024-04-23T16:43:53.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf5ae2ce:::benchmark_data_smithi188_141483_object16323:head# 2024-04-23T16:43:53.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf5b0491:::benchmark_data_smithi188_141483_object31637:head# 2024-04-23T16:43:53.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf5c275a:::benchmark_data_smithi188_141483_object20574:head# 2024-04-23T16:43:53.397 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.397 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf5cdd1a:::benchmark_data_smithi188_141483_object62048:head# 2024-04-23T16:43:53.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf5cf6fd:::benchmark_data_smithi188_141483_object34401:head# 2024-04-23T16:43:53.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf5ec86e:::benchmark_data_smithi188_141483_object56078:head# 2024-04-23T16:43:53.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf5f5100:::benchmark_data_smithi188_141483_object74027:head# 2024-04-23T16:43:53.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf609652:::benchmark_data_smithi188_141483_object63723:head# 2024-04-23T16:43:53.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf638401:::benchmark_data_smithi188_141483_object5148:head# 2024-04-23T16:43:53.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.398 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf638eba:::benchmark_data_smithi188_141483_object73947:head# 2024-04-23T16:43:53.398 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf6555fc:::benchmark_data_smithi188_141483_object28913:head# 2024-04-23T16:43:53.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf65a922:::benchmark_data_smithi188_141483_object31304:head# 2024-04-23T16:43:53.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf66527e:::benchmark_data_smithi188_141483_object45547:head# 2024-04-23T16:43:53.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf668a71:::benchmark_data_smithi188_141483_object37432:head# 2024-04-23T16:43:53.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf694e4c:::benchmark_data_smithi188_141483_object80073:head# 2024-04-23T16:43:53.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf69b5af:::benchmark_data_smithi188_141483_object7002:head# 2024-04-23T16:43:53.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.399 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf6bb165:::benchmark_data_smithi188_141483_object75266:head# 2024-04-23T16:43:53.399 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf6bd706:::benchmark_data_smithi188_141483_object59959:head# 2024-04-23T16:43:53.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf6d1fec:::benchmark_data_smithi188_141483_object46364:head# 2024-04-23T16:43:53.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf6d4ab7:::benchmark_data_smithi188_141483_object14959:head# 2024-04-23T16:43:53.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf6dd896:::benchmark_data_smithi188_141483_object82216:head# 2024-04-23T16:43:53.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf6f4713:::benchmark_data_smithi188_141483_object63304:head# 2024-04-23T16:43:53.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf6f54db:::benchmark_data_smithi188_141483_object20766:head# 2024-04-23T16:43:53.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf6f7a89:::benchmark_data_smithi188_141483_object24371:head# 2024-04-23T16:43:53.400 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.400 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf6fa71f:::benchmark_data_smithi188_141483_object71754:head# 2024-04-23T16:43:53.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf708f8b:::benchmark_data_smithi188_141483_object1294:head# 2024-04-23T16:43:53.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf710a03:::benchmark_data_smithi188_141483_object66385:head# 2024-04-23T16:43:53.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf712a54:::benchmark_data_smithi188_141483_object32337:head# 2024-04-23T16:43:53.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf723278:::benchmark_data_smithi188_141483_object33201:head# 2024-04-23T16:43:53.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf729634:::benchmark_data_smithi188_141483_object31874:head# 2024-04-23T16:43:53.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf7468ab:::benchmark_data_smithi188_141483_object89339:head# 2024-04-23T16:43:53.401 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.401 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf747e99:::benchmark_data_smithi188_141483_object11550:head# 2024-04-23T16:43:53.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf7485ec:::benchmark_data_smithi188_141483_object28376:head# 2024-04-23T16:43:53.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf751141:::benchmark_data_smithi188_141483_object80164:head# 2024-04-23T16:43:53.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf7533a5:::benchmark_data_smithi188_141483_object11481:head# 2024-04-23T16:43:53.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf75cf14:::benchmark_data_smithi188_141483_object14486:head# 2024-04-23T16:43:53.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf75d27f:::benchmark_data_smithi188_141483_object8564:head# 2024-04-23T16:43:53.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.402 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf7620f2:::benchmark_data_smithi188_141483_object35974:head# 2024-04-23T16:43:53.402 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf77ac75:::benchmark_data_smithi188_141483_object58563:head# 2024-04-23T16:43:53.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf79322e:::benchmark_data_smithi188_141483_object1025:head# 2024-04-23T16:43:53.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf796e13:::benchmark_data_smithi188_141483_object62912:head# 2024-04-23T16:43:53.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf7a7fae:::benchmark_data_smithi188_141483_object81500:head# 2024-04-23T16:43:53.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf7a8e64:::benchmark_data_smithi188_141483_object33139:head# 2024-04-23T16:43:53.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf7a989c:::benchmark_data_smithi188_141483_object53153:head# 2024-04-23T16:43:53.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.403 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf7beee3:::benchmark_data_smithi188_141483_object7770:head# 2024-04-23T16:43:53.403 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf7c07cf:::benchmark_data_smithi188_141483_object80459:head# 2024-04-23T16:43:53.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf7c55b6:::benchmark_data_smithi188_141483_object29364:head# 2024-04-23T16:43:53.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf7d5464:::benchmark_data_smithi188_141483_object71932:head# 2024-04-23T16:43:53.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf7e2a91:::benchmark_data_smithi188_141483_object62495:head# 2024-04-23T16:43:53.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf7f0f57:::benchmark_data_smithi188_141483_object68871:head# 2024-04-23T16:43:53.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf7f4166:::benchmark_data_smithi188_141483_object45822:head# 2024-04-23T16:43:53.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.404 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf80d57b:::benchmark_data_smithi188_141483_object51186:head# 2024-04-23T16:43:53.404 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf80fe18:::benchmark_data_smithi188_141483_object9581:head# 2024-04-23T16:43:53.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf8100ec:::benchmark_data_smithi188_141483_object56729:head# 2024-04-23T16:43:53.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf826ffa:::benchmark_data_smithi188_141483_object56295:head# 2024-04-23T16:43:53.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf827bcb:::benchmark_data_smithi188_141483_object22091:head# 2024-04-23T16:43:53.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf82a824:::benchmark_data_smithi188_141483_object48736:head# 2024-04-23T16:43:53.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf832544:::benchmark_data_smithi188_141483_object72353:head# 2024-04-23T16:43:53.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf83b832:::benchmark_data_smithi188_141483_object18344:head# 2024-04-23T16:43:53.405 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.405 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf8781da:::benchmark_data_smithi188_141483_object31437:head# 2024-04-23T16:43:53.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf8825ce:::benchmark_data_smithi188_141483_object77977:head# 2024-04-23T16:43:53.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf8bc134:::benchmark_data_smithi188_141483_object56872:head# 2024-04-23T16:43:53.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf8c1c7f:::benchmark_data_smithi188_141483_object66594:head# 2024-04-23T16:43:53.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf9005a7:::benchmark_data_smithi188_141483_object62690:head# 2024-04-23T16:43:53.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf91811d:::benchmark_data_smithi188_141483_object55456:head# 2024-04-23T16:43:53.406 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.406 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf91a02b:::benchmark_data_smithi188_141483_object36556:head# 2024-04-23T16:43:53.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf920ae5:::benchmark_data_smithi188_141483_object4953:head# 2024-04-23T16:43:53.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf929066:::benchmark_data_smithi188_141483_object49876:head# 2024-04-23T16:43:53.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf92b231:::benchmark_data_smithi188_141483_object85833:head# 2024-04-23T16:43:53.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf92bfac:::benchmark_data_smithi188_141483_object33100:head# 2024-04-23T16:43:53.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf930018:::benchmark_data_smithi188_141483_object80053:head# 2024-04-23T16:43:53.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf9366b8:::benchmark_data_smithi188_141483_object27166:head# 2024-04-23T16:43:53.407 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.407 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf93e543:::benchmark_data_smithi188_141483_object42403:head# 2024-04-23T16:43:53.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf9690dd:::benchmark_data_smithi188_141483_object12936:head# 2024-04-23T16:43:53.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf96cd4e:::benchmark_data_smithi188_141483_object1035:head# 2024-04-23T16:43:53.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf97477a:::benchmark_data_smithi188_141483_object46414:head# 2024-04-23T16:43:53.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf975545:::benchmark_data_smithi188_141483_object57371:head# 2024-04-23T16:43:53.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf98eabd:::benchmark_data_smithi188_141483_object29220:head# 2024-04-23T16:43:53.408 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.408 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf99bdb4:::benchmark_data_smithi188_141483_object84060:head# 2024-04-23T16:43:53.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf9a14ad:::benchmark_data_smithi188_141483_object86339:head# 2024-04-23T16:43:53.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf9a2166:::benchmark_data_smithi188_141483_object9293:head# 2024-04-23T16:43:53.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf9ad631:::benchmark_data_smithi188_141483_object86751:head# 2024-04-23T16:43:53.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf9d1ebd:::benchmark_data_smithi188_141483_object55328:head# 2024-04-23T16:43:53.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf9d8195:::benchmark_data_smithi188_141483_object40752:head# 2024-04-23T16:43:53.409 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.409 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf9daf4a:::benchmark_data_smithi188_141483_object50032:head# 2024-04-23T16:43:53.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf9de71e:::benchmark_data_smithi188_141483_object17057:head# 2024-04-23T16:43:53.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf9e03af:::benchmark_data_smithi188_141483_object81454:head# 2024-04-23T16:43:53.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf9eb69c:::benchmark_data_smithi188_141483_object46747:head# 2024-04-23T16:43:53.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf9ede8b:::benchmark_data_smithi188_141483_object42220:head# 2024-04-23T16:43:53.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf9f0df7:::benchmark_data_smithi188_141483_object37100:head# 2024-04-23T16:43:53.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.410 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf9f3454:::benchmark_data_smithi188_141483_object59241:head# 2024-04-23T16:43:53.410 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf9f9692:::benchmark_data_smithi188_141483_object30925:head# 2024-04-23T16:43:53.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf9f97ea:::benchmark_data_smithi188_141483_object67392:head# 2024-04-23T16:43:53.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cf9feed8:::benchmark_data_smithi188_141483_object47164:head# 2024-04-23T16:43:53.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfa00c30:::benchmark_data_smithi188_141483_object21164:head# 2024-04-23T16:43:53.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfa0b875:::benchmark_data_smithi188_141483_object20984:head# 2024-04-23T16:43:53.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfa11b9a:::benchmark_data_smithi188_141483_object29558:head# 2024-04-23T16:43:53.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.411 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfa1aae9:::benchmark_data_smithi188_141483_object23512:head# 2024-04-23T16:43:53.411 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfa1b3e5:::benchmark_data_smithi188_141483_object81788:head# 2024-04-23T16:43:53.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfa26f3f:::benchmark_data_smithi188_141483_object74538:head# 2024-04-23T16:43:53.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfa38553:::benchmark_data_smithi188_141483_object71560:head# 2024-04-23T16:43:53.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfa42084:::benchmark_data_smithi188_141483_object36107:head# 2024-04-23T16:43:53.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfa43adc:::benchmark_data_smithi188_141483_object27804:head# 2024-04-23T16:43:53.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfa4521e:::benchmark_data_smithi188_141483_object57989:head# 2024-04-23T16:43:53.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.412 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfa7d365:::benchmark_data_smithi188_141483_object76940:head# 2024-04-23T16:43:53.412 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfa9905c:::benchmark_data_smithi188_141483_object327:head# 2024-04-23T16:43:53.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfa9db6b:::benchmark_data_smithi188_141483_object2693:head# 2024-04-23T16:43:53.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfaaea07:::benchmark_data_smithi188_141483_object65966:head# 2024-04-23T16:43:53.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfab3b4d:::benchmark_data_smithi188_141483_object72833:head# 2024-04-23T16:43:53.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfac7e8e:::benchmark_data_smithi188_141483_object67569:head# 2024-04-23T16:43:53.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfacb396:::benchmark_data_smithi188_141483_object83560:head# 2024-04-23T16:43:53.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.413 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfad085e:::benchmark_data_smithi188_141483_object5913:head# 2024-04-23T16:43:53.413 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfad1580:::benchmark_data_smithi188_141483_object40730:head# 2024-04-23T16:43:53.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfad55f9:::benchmark_data_smithi188_141483_object87134:head# 2024-04-23T16:43:53.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfaec77f:::benchmark_data_smithi188_141483_object51338:head# 2024-04-23T16:43:53.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfb108e6:::benchmark_data_smithi188_141483_object73813:head# 2024-04-23T16:43:53.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfb25db5:::benchmark_data_smithi188_141483_object40426:head# 2024-04-23T16:43:53.414 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.414 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfb2f362:::benchmark_data_smithi188_141483_object79994:head# 2024-04-23T16:43:53.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfb4a434:::benchmark_data_smithi188_141483_object13210:head# 2024-04-23T16:43:53.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfb4fafd:::benchmark_data_smithi188_141483_object27748:head# 2024-04-23T16:43:53.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfb87e0c:::benchmark_data_smithi188_141483_object87052:head# 2024-04-23T16:43:53.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfb90fdd:::benchmark_data_smithi188_141483_object75474:head# 2024-04-23T16:43:53.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfb95f91:::benchmark_data_smithi188_141483_object36146:head# 2024-04-23T16:43:53.415 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.415 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfb98692:::benchmark_data_smithi188_141483_object79901:head# 2024-04-23T16:43:53.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfba829e:::benchmark_data_smithi188_141483_object30445:head# 2024-04-23T16:43:53.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfba9193:::benchmark_data_smithi188_141483_object70652:head# 2024-04-23T16:43:53.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfbc8b9a:::benchmark_data_smithi188_141483_object63466:head# 2024-04-23T16:43:53.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfbcb5f1:::benchmark_data_smithi188_141483_object63399:head# 2024-04-23T16:43:53.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfbce2d9:::benchmark_data_smithi188_141483_object53300:head# 2024-04-23T16:43:53.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfbd0609:::benchmark_data_smithi188_141483_object1555:head# 2024-04-23T16:43:53.416 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.416 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfbd49ee:::benchmark_data_smithi188_141483_object1061:head# 2024-04-23T16:43:53.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfbd83f9:::benchmark_data_smithi188_141483_object81870:head# 2024-04-23T16:43:53.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfbef77c:::benchmark_data_smithi188_141483_object69551:head# 2024-04-23T16:43:53.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfbf3cbb:::benchmark_data_smithi188_141483_object19082:head# 2024-04-23T16:43:53.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfbf4ea6:::benchmark_data_smithi188_141483_object6159:head# 2024-04-23T16:43:53.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfbfa053:::benchmark_data_smithi188_141483_object7224:head# 2024-04-23T16:43:53.417 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.417 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfbfa6b0:::benchmark_data_smithi188_141483_object54907:head# 2024-04-23T16:43:53.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfbfec0f:::benchmark_data_smithi188_141483_object53305:head# 2024-04-23T16:43:53.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfc08971:::benchmark_data_smithi188_141483_object22601:head# 2024-04-23T16:43:53.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfc0fb19:::benchmark_data_smithi188_141483_object58377:head# 2024-04-23T16:43:53.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfc2633f:::benchmark_data_smithi188_141483_object84573:head# 2024-04-23T16:43:53.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfc2a399:::benchmark_data_smithi188_141483_object49080:head# 2024-04-23T16:43:53.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.418 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfc2c8f4:::benchmark_data_smithi188_141483_object42939:head# 2024-04-23T16:43:53.418 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfc4fbac:::benchmark_data_smithi188_141483_object55304:head# 2024-04-23T16:43:53.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfc55eea:::benchmark_data_smithi188_141483_object64238:head# 2024-04-23T16:43:53.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfc6bee9:::benchmark_data_smithi188_141483_object75821:head# 2024-04-23T16:43:53.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfc6d78a:::benchmark_data_smithi188_141483_object63443:head# 2024-04-23T16:43:53.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfc739b6:::benchmark_data_smithi188_141483_object799:head# 2024-04-23T16:43:53.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfc74abd:::benchmark_data_smithi188_141483_object41915:head# 2024-04-23T16:43:53.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.419 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfc8f499:::benchmark_data_smithi188_141483_object82821:head# 2024-04-23T16:43:53.419 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfca15ce:::benchmark_data_smithi188_141483_object72911:head# 2024-04-23T16:43:53.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfca41fa:::benchmark_data_smithi188_141483_object67695:head# 2024-04-23T16:43:53.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfcbb4d2:::benchmark_data_smithi188_141483_object2027:head# 2024-04-23T16:43:53.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfcc73b3:::benchmark_data_smithi188_141483_object78411:head# 2024-04-23T16:43:53.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfccb77e:::benchmark_data_smithi188_141483_object74793:head# 2024-04-23T16:43:53.420 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.420 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfcd4ea4:::benchmark_data_smithi188_141483_object16801:head# 2024-04-23T16:43:53.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfce626d:::benchmark_data_smithi188_141483_object88493:head# 2024-04-23T16:43:53.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfcf0233:::benchmark_data_smithi188_141483_object46820:head# 2024-04-23T16:43:53.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfcf3549:::benchmark_data_smithi188_141483_object31119:head# 2024-04-23T16:43:53.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfd033c9:::benchmark_data_smithi188_141483_object26185:head# 2024-04-23T16:43:53.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfd04c34:::benchmark_data_smithi188_141483_object67390:head# 2024-04-23T16:43:53.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.421 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfd0cda8:::benchmark_data_smithi188_141483_object16266:head# 2024-04-23T16:43:53.421 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfd15720:::benchmark_data_smithi188_141483_object71714:head# 2024-04-23T16:43:53.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfd23922:::benchmark_data_smithi188_141483_object75763:head# 2024-04-23T16:43:53.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfd2865a:::benchmark_data_smithi188_141483_object43357:head# 2024-04-23T16:43:53.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfd37a32:::benchmark_data_smithi188_141483_object34431:head# 2024-04-23T16:43:53.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfd4bf01:::benchmark_data_smithi188_141483_object41903:head# 2024-04-23T16:43:53.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfd4ca64:::benchmark_data_smithi188_141483_object52050:head# 2024-04-23T16:43:53.422 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.422 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfd4e056:::benchmark_data_smithi188_141483_object57759:head# 2024-04-23T16:43:53.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfd4f19f:::benchmark_data_smithi188_141483_object90009:head# 2024-04-23T16:43:53.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfd5efbf:::benchmark_data_smithi188_141483_object79416:head# 2024-04-23T16:43:53.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfd64e8d:::benchmark_data_smithi188_141483_object43636:head# 2024-04-23T16:43:53.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfd6988d:::benchmark_data_smithi188_141483_object73165:head# 2024-04-23T16:43:53.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfd69da1:::benchmark_data_smithi188_141483_object83170:head# 2024-04-23T16:43:53.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfd76f94:::benchmark_data_smithi188_141483_object64683:head# 2024-04-23T16:43:53.423 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.423 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfd897b7:::benchmark_data_smithi188_141483_object23872:head# 2024-04-23T16:43:53.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfd92fac:::benchmark_data_smithi188_141483_object61006:head# 2024-04-23T16:43:53.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfda1a54:::benchmark_data_smithi188_141483_object5851:head# 2024-04-23T16:43:53.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfda3fa1:::benchmark_data_smithi188_141483_object29323:head# 2024-04-23T16:43:53.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfdb07e6:::benchmark_data_smithi188_141483_object48220:head# 2024-04-23T16:43:53.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfdc3559:::benchmark_data_smithi188_141483_object69173:head# 2024-04-23T16:43:53.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.424 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfdcfce3:::benchmark_data_smithi188_141483_object18868:head# 2024-04-23T16:43:53.424 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfde57b4:::benchmark_data_smithi188_141483_object80267:head# 2024-04-23T16:43:53.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfe05239:::benchmark_data_smithi188_141483_object64688:head# 2024-04-23T16:43:53.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfe0d6a3:::benchmark_data_smithi188_141483_object5520:head# 2024-04-23T16:43:53.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfe10f94:::benchmark_data_smithi188_141483_object78043:head# 2024-04-23T16:43:53.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfe13ad0:::benchmark_data_smithi188_141483_object45725:head# 2024-04-23T16:43:53.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfe17bbe:::benchmark_data_smithi188_141483_object14683:head# 2024-04-23T16:43:53.425 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.425 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfe3d455:::benchmark_data_smithi188_141483_object42325:head# 2024-04-23T16:43:53.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfe41cae:::benchmark_data_smithi188_141483_object25166:head# 2024-04-23T16:43:53.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfe58ce0:::benchmark_data_smithi188_141483_object82386:head# 2024-04-23T16:43:53.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfe610f1:::benchmark_data_smithi188_141483_object62128:head# 2024-04-23T16:43:53.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfe76701:::benchmark_data_smithi188_141483_object22424:head# 2024-04-23T16:43:53.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfe7a0a9:::benchmark_data_smithi188_141483_object9613:head# 2024-04-23T16:43:53.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfe937db:::benchmark_data_smithi188_141483_object70061:head# 2024-04-23T16:43:53.426 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.426 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfea1c9d:::benchmark_data_smithi188_141483_object32364:head# 2024-04-23T16:43:53.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfeb43d5:::benchmark_data_smithi188_141483_object34391:head# 2024-04-23T16:43:53.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfeb48c2:::benchmark_data_smithi188_141483_object48538:head# 2024-04-23T16:43:53.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfeb5451:::benchmark_data_smithi188_141483_object89416:head# 2024-04-23T16:43:53.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfed84d5:::benchmark_data_smithi188_141483_object80996:head# 2024-04-23T16:43:53.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfee23b1:::benchmark_data_smithi188_141483_object74827:head# 2024-04-23T16:43:53.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfee9a17:::benchmark_data_smithi188_141483_object86560:head# 2024-04-23T16:43:53.427 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.427 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfeec085:::benchmark_data_smithi188_141483_object78719:head# 2024-04-23T16:43:53.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfefe2b6:::benchmark_data_smithi188_141483_object37301:head# 2024-04-23T16:43:53.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cfeff2ae:::benchmark_data_smithi188_141483_object88249:head# 2024-04-23T16:43:53.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cff0baf4:::benchmark_data_smithi188_141483_object6151:head# 2024-04-23T16:43:53.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cff0d3bf:::benchmark_data_smithi188_141483_object9458:head# 2024-04-23T16:43:53.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cff0dd8e:::benchmark_data_smithi188_141483_object33118:head# 2024-04-23T16:43:53.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cff0ff27:::benchmark_data_smithi188_141483_object40158:head# 2024-04-23T16:43:53.428 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.428 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cff1aff9:::benchmark_data_smithi188_141483_object21092:head# 2024-04-23T16:43:53.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cff32909:::benchmark_data_smithi188_141483_object31044:head# 2024-04-23T16:43:53.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cff35204:::benchmark_data_smithi188_141483_object69818:head# 2024-04-23T16:43:53.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cff49d40:::benchmark_data_smithi188_141483_object29148:head# 2024-04-23T16:43:53.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cff4ea0e:::benchmark_data_smithi188_141483_object69041:head# 2024-04-23T16:43:53.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cff6c6c0:::benchmark_data_smithi188_141483_object15443:head# 2024-04-23T16:43:53.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cff6f90b:::benchmark_data_smithi188_141483_object1019:head# 2024-04-23T16:43:53.429 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.429 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cff842f5:::benchmark_data_smithi188_141483_object61773:head# 2024-04-23T16:43:53.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cff94b93:::benchmark_data_smithi188_141483_object38496:head# 2024-04-23T16:43:53.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cff97b75:::benchmark_data_smithi188_141483_object86753:head# 2024-04-23T16:43:53.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cffc2673:::benchmark_data_smithi188_141483_object4400:head# 2024-04-23T16:43:53.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cffc3dd8:::benchmark_data_smithi188_141483_object42195:head# 2024-04-23T16:43:53.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cffc4393:::benchmark_data_smithi188_141483_object87708:head# 2024-04-23T16:43:53.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.430 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cffc9c76:::benchmark_data_smithi188_141483_object21812:head# 2024-04-23T16:43:53.430 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cffd7c8d:::benchmark_data_smithi188_141483_object36827:head# 2024-04-23T16:43:53.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cffd8180:::benchmark_data_smithi188_141483_object29041:head# 2024-04-23T16:43:53.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.431 INFO:teuthology.orchestra.run.smithi103.stdout:Write #7:cffed9b0:::benchmark_data_smithi188_141483_object35463:head# 2024-04-23T16:43:53.431 INFO:teuthology.orchestra.run.smithi103.stdout:snapset 0=[]:{} 2024-04-23T16:43:53.431 INFO:teuthology.orchestra.run.smithi103.stdout:write_pg epoch 559 info 7.3( v 559'5656 (0'0,559'5656] local-lis/les=540/541 n=5656 ec=540/540 lis/c=540/540 les/c/f=541/541/0 sis=540) 2024-04-23T16:43:53.431 INFO:teuthology.orchestra.run.smithi103.stdout:Import successful 2024-04-23T16:43:53.432 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:53 smithi148 ceph-mon[44520]: osdmap e575: 12 total, 11 up, 9 in 2024-04-23T16:43:53.432 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:53 smithi148 ceph-mon[44520]: pgmap v1604: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 749 op/s; 18198/224884 objects degraded (8.092%); 5.7 MiB/s, 90 objects/s recovering 2024-04-23T16:43:53.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:53 smithi188 ceph-mon[44101]: osdmap e575: 12 total, 11 up, 9 in 2024-04-23T16:43:53.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:53 smithi188 ceph-mon[44101]: pgmap v1604: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 749 op/s; 18198/224884 objects degraded (8.092%); 5.7 MiB/s, 90 objects/s recovering 2024-04-23T16:43:53.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:53 smithi103 ceph-mon[39629]: osdmap e575: 12 total, 11 up, 9 in 2024-04-23T16:43:53.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:53 smithi103 ceph-mon[39629]: pgmap v1604: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 749 op/s; 18198/224884 objects degraded (8.092%); 5.7 MiB/s, 90 objects/s recovering 2024-04-23T16:43:54.205 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:43:54.940 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/exp.7.3.3 2024-04-23T16:43:54.969 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:43:54.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:54.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:43:55.025 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-04-23T16:43:55.025 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-23T16:43:55.025 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:43:55.078 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:43:55.107 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:55.132 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:43:55.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:55.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:43:55.173 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:43:55.220 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:55 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:43:55.220 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:55 smithi103 ceph-mon[39629]: osdmap e576: 12 total, 11 up, 9 in 2024-04-23T16:43:55.221 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:55 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 18198/224884 objects degraded (8.092%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:55.221 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:55 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2078383868' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:43:55.221 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:55 smithi103 ceph-mon[39629]: pgmap v1606: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.7 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 17638/218352 objects degraded (8.078%); 10 MiB/s, 163 objects/s recovering 2024-04-23T16:43:55.240 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:43:55.269 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:55.326 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:43:55.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:55 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:43:55.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:55 smithi148 ceph-mon[44520]: osdmap e576: 12 total, 11 up, 9 in 2024-04-23T16:43:55.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:55 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 18198/224884 objects degraded (8.092%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:55.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:55 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2078383868' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:43:55.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:55 smithi148 ceph-mon[44520]: pgmap v1606: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.7 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 17638/218352 objects degraded (8.078%); 10 MiB/s, 163 objects/s recovering 2024-04-23T16:43:55.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:55 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:43:55.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:55 smithi188 ceph-mon[44101]: osdmap e576: 12 total, 11 up, 9 in 2024-04-23T16:43:55.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:55 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 18198/224884 objects degraded (8.092%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:55.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:55 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2078383868' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:43:55.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:55 smithi188 ceph-mon[44101]: pgmap v1606: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.7 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 17638/218352 objects degraded (8.078%); 10 MiB/s, 163 objects/s recovering 2024-04-23T16:43:55.521 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:43:55.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:55.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:43:55.625 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:43:55.656 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:55.754 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:43:55.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:55.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:43:55.858 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:43:55.894 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:55.918 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:43:55.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:55.922 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:43:56.022 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:43:56.050 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:56.300 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:43:56.301 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:56 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:43:56.301 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:56 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2078383868' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:43:56.302 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:56 smithi103 ceph-mon[39629]: osdmap e577: 12 total, 11 up, 9 in 2024-04-23T16:43:56.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:56.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:43:56.404 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:43:56.467 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:56 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:43:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:56 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2078383868' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:43:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:56 smithi188 ceph-mon[44101]: osdmap e577: 12 total, 11 up, 9 in 2024-04-23T16:43:56.596 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:43:56.603 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:56 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:43:56.603 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:56 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2078383868' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:43:56.603 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:56 smithi148 ceph-mon[44520]: osdmap e577: 12 total, 11 up, 9 in 2024-04-23T16:43:56.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:56.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:43:56.706 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:43:56.735 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:56.983 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:43:56.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:56.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:43:57.087 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:43:57.116 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:57.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:57 smithi103 ceph-mon[39629]: pgmap v1608: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 809 op/s; 17638/216764 objects degraded (8.137%); 5.8 MiB/s, 93 objects/s recovering 2024-04-23T16:43:57.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:57 smithi148 ceph-mon[44520]: pgmap v1608: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 809 op/s; 17638/216764 objects degraded (8.137%); 5.8 MiB/s, 93 objects/s recovering 2024-04-23T16:43:57.664 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:43:57.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:57 smithi188 ceph-mon[44101]: pgmap v1608: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 809 op/s; 17638/216764 objects degraded (8.137%); 5.8 MiB/s, 93 objects/s recovering 2024-04-23T16:43:58.279 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:43:58.295 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:43:58.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:58.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:43:58.302 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:43:58.322 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:43:58.353 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:43:58.389 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:43:58.400 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:43:58.419 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:43:58.419 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:43:58.419 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:43:58.459 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:43:58.481 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:43:58.500 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:58.501 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:43:58.522 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:43:58.542 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:43:58.543 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:43:58.543 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:43:58.543 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:43:58.543 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:43:58.543 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:43:58.543 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:43:58.543 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:43:58.543 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:43:58.543 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:43:58.543 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:43:58.543 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:43:58.544 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:43:58.558 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:58 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/737112622' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:43:58.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:58 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/737112622' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:43:58.720 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:43:58.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:58 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/737112622' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:43:58.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:58.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:43:58.834 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:43:58.907 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:58.979 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:43:58.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:58.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:43:59.090 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:43:59.120 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:59.222 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:43:59.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:59.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:43:59.326 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:43:59.327 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.327 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.327 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.327 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.327 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.328 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.328 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.328 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.328 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.328 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.328 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.328 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:59 smithi148 ceph-mon[44520]: pgmap v1609: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 682 op/s; 17638/216262 objects degraded (8.156%); 5.2 MiB/s, 83 objects/s recovering 2024-04-23T16:43:59.328 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:43:59 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 17638/216764 objects degraded (8.137%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:59.359 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:59.383 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:43:59.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:43:59.387 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:43:59.488 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:43:59.573 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:43:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:59 smithi103 ceph-mon[39629]: pgmap v1609: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 682 op/s; 17638/216262 objects degraded (8.156%); 5.2 MiB/s, 83 objects/s recovering 2024-04-23T16:43:59.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:43:59 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 17638/216764 objects degraded (8.137%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:43:59.603 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:43:59.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:43:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:59 smithi188 ceph-mon[44101]: pgmap v1609: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 682 op/s; 17638/216262 objects degraded (8.156%); 5.2 MiB/s, 83 objects/s recovering 2024-04-23T16:43:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:43:59 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 17638/216764 objects degraded (8.137%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:00.253 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:44:00.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:00.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:44:00.393 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:44:00.397 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:44:00.399 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:44:00.399 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:44:00.432 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:00.456 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:44:00.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:00.459 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:44:00.560 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:44:00.641 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:01.541 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:44:01.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:01.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:44:01.601 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-23T16:44:01.657 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:44:01.690 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:01.714 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:44:01.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:01.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:44:01.818 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:44:01.899 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:01.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:01 smithi188 ceph-mon[44101]: pgmap v1610: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.4 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 953 op/s; 17080/209628 objects degraded (8.148%); 8.7 MiB/s, 139 objects/s recovering 2024-04-23T16:44:01.982 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:44:01.984 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:44:01.984 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:44:01.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:01.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:44:02.095 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:44:02.175 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:01 smithi148 ceph-mon[44520]: pgmap v1610: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.4 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 953 op/s; 17080/209628 objects degraded (8.148%); 8.7 MiB/s, 139 objects/s recovering 2024-04-23T16:44:02.175 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:02.261 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:44:02.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:02.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:44:02.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:01 smithi103 ceph-mon[39629]: pgmap v1610: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.4 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 953 op/s; 17080/209628 objects degraded (8.148%); 8.7 MiB/s, 139 objects/s recovering 2024-04-23T16:44:02.375 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:44:02.401 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:44:02.480 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:44:02.481 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:02.546 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:44:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:02.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:44:02.657 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:44:02.686 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:02.725 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:44:02.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:02.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:44:02.828 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:44:02.856 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:03.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:02 smithi148 ceph-mon[44520]: from='client.16746 v1:172.21.15.103:0/2135105448' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:44:03.195 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:02 smithi103 ceph-mon[39629]: from='client.16746 v1:172.21.15.103:0/2135105448' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:44:03.204 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:44:03.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:03.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:44:03.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:02 smithi188 ceph-mon[44101]: from='client.16746 v1:172.21.15.103:0/2135105448' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:44:03.235 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1610,"stamp":"2024-04-23T16:44:00.858011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6869484088,"num_objects":104814,"num_object_clones":0,"num_object_copies":209628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104814,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132546,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5617,"num_bytes_recovered":368387136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134075,"ondisk_log_size":134075,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":17049596,"kb_used_data":12380960,"kb_used_omap":243,"kb_used_meta":4668172,"kb_avail":826656772,"statfs":{"total":863955320832,"available":846496534528,"internally_reserved":0,"allocated":12678103040,"data_stored":12674776089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":249416,"internal_metadata":4780208568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,4,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-499908608,"num_objects":-7628,"num_object_clones":0,"num_object_copies":-15256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7628,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1118,"num_bytes_recovered":73269248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001804"},"pg_stats":[{"pgid":"7.d","version":"577'8416","reported_seq":8500,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:59.536491+0000","last_change":"2024-04-23T16:41:57.310289+0000","last_active":"2024-04-23T16:43:59.536491+0000","last_peered":"2024-04-23T16:43:59.536491+0000","last_clean":"2024-04-23T16:43:59.536491+0000","last_became_active":"2024-04-23T16:41:57.310025+0000","last_became_peered":"2024-04-23T16:41:57.310025+0000","last_unstale":"2024-04-23T16:43:59.536491+0000","last_undegraded":"2024-04-23T16:43:59.536491+0000","last_fullsized":"2024-04-23T16:43:59.536491+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8416,"log_dups_size":0,"ondisk_log_size":8416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436011032,"num_objects":6654,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6654,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8416,"num_write_kb":482177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"577'8347","reported_seq":8507,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:55.858622+0000","last_change":"2024-04-23T16:43:29.012370+0000","last_active":"2024-04-23T16:43:55.858622+0000","last_peered":"2024-04-23T16:43:55.858622+0000","last_clean":"2024-04-23T16:43:55.858622+0000","last_became_active":"2024-04-23T16:41:57.310754+0000","last_became_peered":"2024-04-23T16:41:57.310754+0000","last_unstale":"2024-04-23T16:43:55.858622+0000","last_undegraded":"2024-04-23T16:43:55.858622+0000","last_fullsized":"2024-04-23T16:43:55.858622+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6945","last_scrub_stamp":"2024-04-23T16:43:29.012285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:29.012285+0000","objects_scrubbed":6912,"log_size":8347,"log_dups_size":0,"ondisk_log_size":8347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:58.968073+0000","scrub_duration":1082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445448192,"num_objects":6797,"num_object_clones":0,"num_object_copies":13594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8347,"num_write_kb":484608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"577'8351","reported_seq":8508,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:59.400464+0000","last_change":"2024-04-23T16:43:25.997269+0000","last_active":"2024-04-23T16:43:59.400464+0000","last_peered":"2024-04-23T16:43:59.400464+0000","last_clean":"2024-04-23T16:43:59.400464+0000","last_became_active":"2024-04-23T16:41:57.310189+0000","last_became_peered":"2024-04-23T16:41:57.310189+0000","last_unstale":"2024-04-23T16:43:59.400464+0000","last_undegraded":"2024-04-23T16:43:59.400464+0000","last_fullsized":"2024-04-23T16:43:59.400464+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'6616","last_scrub_stamp":"2024-04-23T16:43:25.997150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:25.997150+0000","objects_scrubbed":6600,"log_size":8351,"log_dups_size":0,"ondisk_log_size":8351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:37.694305+0000","scrub_duration":806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429326336,"num_objects":6551,"num_object_clones":0,"num_object_copies":13102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8351,"num_write_kb":476864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"577'8339","reported_seq":8495,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:59.330963+0000","last_change":"2024-04-23T16:43:26.892913+0000","last_active":"2024-04-23T16:43:59.330963+0000","last_peered":"2024-04-23T16:43:59.330963+0000","last_clean":"2024-04-23T16:43:59.330963+0000","last_became_active":"2024-04-23T16:41:57.309569+0000","last_became_peered":"2024-04-23T16:41:57.309569+0000","last_unstale":"2024-04-23T16:43:59.330963+0000","last_undegraded":"2024-04-23T16:43:59.330963+0000","last_fullsized":"2024-04-23T16:43:59.330963+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6695","last_scrub_stamp":"2024-04-23T16:43:26.892785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:26.892785+0000","objects_scrubbed":6673,"log_size":8339,"log_dups_size":0,"ondisk_log_size":8339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:56.021559+0000","scrub_duration":739,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429457408,"num_objects":6553,"num_object_clones":0,"num_object_copies":13106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8339,"num_write_kb":476544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"577'8166","reported_seq":8249,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:44:00.636364+0000","last_change":"2024-04-23T16:41:57.310144+0000","last_active":"2024-04-23T16:44:00.636364+0000","last_peered":"2024-04-23T16:44:00.636364+0000","last_clean":"2024-04-23T16:44:00.636364+0000","last_became_active":"2024-04-23T16:41:57.309841+0000","last_became_peered":"2024-04-23T16:41:57.309841+0000","last_unstale":"2024-04-23T16:44:00.636364+0000","last_undegraded":"2024-04-23T16:44:00.636364+0000","last_fullsized":"2024-04-23T16:44:00.636364+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8166,"log_dups_size":0,"ondisk_log_size":8166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417726464,"num_objects":6374,"num_object_clones":0,"num_object_copies":12748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8166,"num_write_kb":465280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"577'8319","reported_seq":11398,"reported_epoch":577,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:43:59.608470+0000","last_change":"2024-04-23T16:43:08.145899+0000","last_active":"2024-04-23T16:43:59.608470+0000","last_peered":"2024-04-23T16:43:59.608470+0000","last_clean":"2024-04-23T16:42:42.970788+0000","last_became_active":"2024-04-23T16:43:08.106076+0000","last_became_peered":"2024-04-23T16:43:08.106076+0000","last_unstale":"2024-04-23T16:43:59.608470+0000","last_undegraded":"2024-04-23T16:43:07.951894+0000","last_fullsized":"2024-04-23T16:43:59.608470+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8319,"log_dups_size":0,"ondisk_log_size":8319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421199872,"num_objects":6427,"num_object_clones":0,"num_object_copies":12854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8319,"num_write_kb":471872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1495,"num_bytes_recovered":97976320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":4219},{"shards":"6,11","objects":2208}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1459,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:55.331279+0000","last_change":"2024-04-23T16:42:37.344941+0000","last_active":"2024-04-23T16:43:55.331279+0000","last_peered":"2024-04-23T16:43:55.331279+0000","last_clean":"2024-04-23T16:43:55.331279+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:43:55.331279+0000","last_undegraded":"2024-04-23T16:43:55.331279+0000","last_fullsized":"2024-04-23T16:43:55.331279+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:37.344822+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:42:37.344822+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:56.281436+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"577'8311","reported_seq":8469,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:59.452622+0000","last_change":"2024-04-23T16:43:28.303396+0000","last_active":"2024-04-23T16:43:59.452622+0000","last_peered":"2024-04-23T16:43:59.452622+0000","last_clean":"2024-04-23T16:43:59.452622+0000","last_became_active":"2024-04-23T16:41:57.311753+0000","last_became_peered":"2024-04-23T16:41:57.311753+0000","last_unstale":"2024-04-23T16:43:59.452622+0000","last_undegraded":"2024-04-23T16:43:59.452622+0000","last_fullsized":"2024-04-23T16:43:59.452622+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6763","last_scrub_stamp":"2024-04-23T16:43:28.303250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:28.303250+0000","objects_scrubbed":6733,"log_size":8311,"log_dups_size":0,"ondisk_log_size":8311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:51.576731+0000","scrub_duration":1201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433520640,"num_objects":6615,"num_object_clones":0,"num_object_copies":13230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8311,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1465,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:55.334216+0000","last_change":"2024-04-23T16:43:27.668916+0000","last_active":"2024-04-23T16:43:55.334216+0000","last_peered":"2024-04-23T16:43:55.334216+0000","last_clean":"2024-04-23T16:43:55.334216+0000","last_became_active":"2024-04-23T16:43:07.962092+0000","last_became_peered":"2024-04-23T16:43:07.962092+0000","last_unstale":"2024-04-23T16:43:55.334216+0000","last_undegraded":"2024-04-23T16:43:55.334216+0000","last_fullsized":"2024-04-23T16:43:55.334216+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:43:27.668793+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:43:27.668793+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:45.595521+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"577'8289","reported_seq":8446,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:59.447536+0000","last_change":"2024-04-23T16:43:27.928311+0000","last_active":"2024-04-23T16:43:59.447536+0000","last_peered":"2024-04-23T16:43:59.447536+0000","last_clean":"2024-04-23T16:43:59.447536+0000","last_became_active":"2024-04-23T16:41:57.305944+0000","last_became_peered":"2024-04-23T16:41:57.305944+0000","last_unstale":"2024-04-23T16:43:59.447536+0000","last_undegraded":"2024-04-23T16:43:59.447536+0000","last_fullsized":"2024-04-23T16:43:59.447536+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6718","last_scrub_stamp":"2024-04-23T16:43:27.928174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:27.928174+0000","objects_scrubbed":6703,"log_size":8289,"log_dups_size":0,"ondisk_log_size":8289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:34.856929+0000","scrub_duration":750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429850624,"num_objects":6559,"num_object_clones":0,"num_object_copies":13118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8289,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1442,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:55.334509+0000","last_change":"2024-04-23T16:42:32.662620+0000","last_active":"2024-04-23T16:43:55.334509+0000","last_peered":"2024-04-23T16:43:55.334509+0000","last_clean":"2024-04-23T16:43:55.334509+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:43:55.334509+0000","last_undegraded":"2024-04-23T16:43:55.334509+0000","last_fullsized":"2024-04-23T16:43:55.334509+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:32.662476+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:42:32.662476+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"577'8537","reported_seq":8695,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:59.400917+0000","last_change":"2024-04-23T16:43:25.389006+0000","last_active":"2024-04-23T16:43:59.400917+0000","last_peered":"2024-04-23T16:43:59.400917+0000","last_clean":"2024-04-23T16:43:59.400917+0000","last_became_active":"2024-04-23T16:41:57.309858+0000","last_became_peered":"2024-04-23T16:41:57.309858+0000","last_unstale":"2024-04-23T16:43:59.400917+0000","last_undegraded":"2024-04-23T16:43:59.400917+0000","last_fullsized":"2024-04-23T16:43:59.400917+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'6776","last_scrub_stamp":"2024-04-23T16:43:25.388898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:25.388898+0000","objects_scrubbed":6743,"log_size":8537,"log_dups_size":0,"ondisk_log_size":8537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:39.155605+0000","scrub_duration":1233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434438144,"num_objects":6629,"num_object_clones":0,"num_object_copies":13258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8537,"num_write_kb":485312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1466,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:55.327970+0000","last_change":"2024-04-23T16:43:40.258384+0000","last_active":"2024-04-23T16:43:55.327970+0000","last_peered":"2024-04-23T16:43:55.327970+0000","last_clean":"2024-04-23T16:43:55.327970+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:43:55.327970+0000","last_undegraded":"2024-04-23T16:43:55.327970+0000","last_fullsized":"2024-04-23T16:43:55.327970+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:43:40.258213+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:43:40.258213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:07.779820+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"577'8074","reported_seq":8157,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:56.494638+0000","last_change":"2024-04-23T16:41:57.310695+0000","last_active":"2024-04-23T16:43:56.494638+0000","last_peered":"2024-04-23T16:43:56.494638+0000","last_clean":"2024-04-23T16:43:56.494638+0000","last_became_active":"2024-04-23T16:41:57.310400+0000","last_became_peered":"2024-04-23T16:41:57.310400+0000","last_unstale":"2024-04-23T16:43:56.494638+0000","last_undegraded":"2024-04-23T16:43:56.494638+0000","last_fullsized":"2024-04-23T16:43:56.494638+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8074,"log_dups_size":0,"ondisk_log_size":8074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434110464,"num_objects":6624,"num_object_clones":0,"num_object_copies":13248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8074,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1032,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:55.331057+0000","last_change":"2024-04-23T16:42:51.989882+0000","last_active":"2024-04-23T16:43:55.331057+0000","last_peered":"2024-04-23T16:43:55.331057+0000","last_clean":"2024-04-23T16:43:55.331057+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:43:55.331057+0000","last_undegraded":"2024-04-23T16:43:55.331057+0000","last_fullsized":"2024-04-23T16:43:55.331057+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:51.989679+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:42:51.989679+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:55.664919+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"577'8264","reported_seq":8347,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:59.607626+0000","last_change":"2024-04-23T16:41:57.306526+0000","last_active":"2024-04-23T16:43:59.607626+0000","last_peered":"2024-04-23T16:43:59.607626+0000","last_clean":"2024-04-23T16:43:59.607626+0000","last_became_active":"2024-04-23T16:41:57.306229+0000","last_became_peered":"2024-04-23T16:41:57.306229+0000","last_unstale":"2024-04-23T16:43:59.607626+0000","last_undegraded":"2024-04-23T16:43:59.607626+0000","last_fullsized":"2024-04-23T16:43:59.607626+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8264,"log_dups_size":0,"ondisk_log_size":8264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422051840,"num_objects":6440,"num_object_clones":0,"num_object_copies":12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8264,"num_write_kb":470528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":986,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:55.346017+0000","last_change":"2024-04-23T16:42:32.970258+0000","last_active":"2024-04-23T16:43:55.346017+0000","last_peered":"2024-04-23T16:43:55.346017+0000","last_clean":"2024-04-23T16:43:55.346017+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:43:55.346017+0000","last_undegraded":"2024-04-23T16:43:55.346017+0000","last_fullsized":"2024-04-23T16:43:55.346017+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:32.970112+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:42:32.970112+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.0","version":"577'8139","reported_seq":8299,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:57.670547+0000","last_change":"2024-04-23T16:43:30.757721+0000","last_active":"2024-04-23T16:43:57.670547+0000","last_peered":"2024-04-23T16:43:57.670547+0000","last_clean":"2024-04-23T16:43:57.670547+0000","last_became_active":"2024-04-23T16:41:57.311479+0000","last_became_peered":"2024-04-23T16:41:57.311479+0000","last_unstale":"2024-04-23T16:43:57.670547+0000","last_undegraded":"2024-04-23T16:43:57.670547+0000","last_fullsized":"2024-04-23T16:43:57.670547+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6823","last_scrub_stamp":"2024-04-23T16:43:30.757555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:30.757555+0000","objects_scrubbed":6802,"log_size":8139,"log_dups_size":0,"ondisk_log_size":8139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:59.054459+0000","scrub_duration":1354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429981696,"num_objects":6561,"num_object_clones":0,"num_object_copies":13122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8139,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1467,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:55.331370+0000","last_change":"2024-04-23T16:42:50.245835+0000","last_active":"2024-04-23T16:43:55.331370+0000","last_peered":"2024-04-23T16:43:55.331370+0000","last_clean":"2024-04-23T16:43:55.331370+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:43:55.331370+0000","last_undegraded":"2024-04-23T16:43:55.331370+0000","last_fullsized":"2024-04-23T16:43:55.331370+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:42:50.245710+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:10.789326+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.1","version":"577'8157","reported_seq":10602,"reported_epoch":577,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:43:59.239429+0000","last_change":"2024-04-23T16:43:08.081567+0000","last_active":"2024-04-23T16:43:59.239429+0000","last_peered":"2024-04-23T16:43:59.239429+0000","last_clean":"2024-04-23T16:42:42.976600+0000","last_became_active":"2024-04-23T16:43:08.067887+0000","last_became_peered":"2024-04-23T16:43:08.067887+0000","last_unstale":"2024-04-23T16:43:59.239429+0000","last_undegraded":"2024-04-23T16:43:07.950381+0000","last_fullsized":"2024-04-23T16:43:59.239429+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8157,"log_dups_size":0,"ondisk_log_size":8157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424607744,"num_objects":6479,"num_object_clones":0,"num_object_copies":12958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8157,"num_write_kb":468352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1179,"num_bytes_recovered":77266944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":4409},{"shards":"4,6","objects":2070}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1459,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:55.336091+0000","last_change":"2024-04-23T16:43:07.918302+0000","last_active":"2024-04-23T16:43:55.336091+0000","last_peered":"2024-04-23T16:43:55.336091+0000","last_clean":"2024-04-23T16:43:55.336091+0000","last_became_active":"2024-04-23T16:43:07.918042+0000","last_became_peered":"2024-04-23T16:43:07.918042+0000","last_unstale":"2024-04-23T16:43:55.336091+0000","last_undegraded":"2024-04-23T16:43:55.336091+0000","last_fullsized":"2024-04-23T16:43:55.336091+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:33.716343+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:42:33.716343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"577'8298","reported_seq":11012,"reported_epoch":577,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:43:59.597851+0000","last_change":"2024-04-23T16:43:08.363306+0000","last_active":"2024-04-23T16:43:59.597851+0000","last_peered":"2024-04-23T16:43:59.597851+0000","last_clean":"2024-04-23T16:42:42.950766+0000","last_became_active":"2024-04-23T16:43:08.333793+0000","last_became_peered":"2024-04-23T16:43:08.333793+0000","last_unstale":"2024-04-23T16:43:59.597851+0000","last_undegraded":"2024-04-23T16:43:07.950356+0000","last_fullsized":"2024-04-23T16:43:59.597851+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8298,"log_dups_size":0,"ondisk_log_size":8298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427294720,"num_objects":6520,"num_object_clones":0,"num_object_copies":13040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8298,"num_write_kb":474176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1314,"num_bytes_recovered":86114304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":4423},{"shards":"6,7","objects":2097}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1351,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:55.327084+0000","last_change":"2024-04-23T16:41:47.722866+0000","last_active":"2024-04-23T16:43:55.327084+0000","last_peered":"2024-04-23T16:43:55.327084+0000","last_clean":"2024-04-23T16:43:55.327084+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:43:55.327084+0000","last_undegraded":"2024-04-23T16:43:55.327084+0000","last_fullsized":"2024-04-23T16:43:55.327084+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:41:47.722762+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"577'8357","reported_seq":11700,"reported_epoch":577,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:43:59.258424+0000","last_change":"2024-04-23T16:43:08.136028+0000","last_active":"2024-04-23T16:43:59.258424+0000","last_peered":"2024-04-23T16:43:59.258424+0000","last_clean":"2024-04-23T16:42:42.962639+0000","last_became_active":"2024-04-23T16:43:08.109240+0000","last_became_peered":"2024-04-23T16:43:08.109240+0000","last_unstale":"2024-04-23T16:43:59.258424+0000","last_undegraded":"2024-04-23T16:43:07.955254+0000","last_fullsized":"2024-04-23T16:43:59.258424+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8357,"log_dups_size":0,"ondisk_log_size":8357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430374912,"num_objects":6567,"num_object_clones":0,"num_object_copies":13134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8357,"num_write_kb":477568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1627,"num_bytes_recovered":106627072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":4029},{"shards":"4,11","objects":2538}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1273,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:55.336267+0000","last_change":"2024-04-23T16:41:49.835121+0000","last_active":"2024-04-23T16:43:55.336267+0000","last_peered":"2024-04-23T16:43:55.336267+0000","last_clean":"2024-04-23T16:43:55.336267+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:43:55.336267+0000","last_undegraded":"2024-04-23T16:43:55.336267+0000","last_fullsized":"2024-04-23T16:43:55.336267+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:41:49.834374+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"577'8182","reported_seq":8338,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:59.269001+0000","last_change":"2024-04-23T16:43:26.589353+0000","last_active":"2024-04-23T16:43:59.269001+0000","last_peered":"2024-04-23T16:43:59.269001+0000","last_clean":"2024-04-23T16:43:59.269001+0000","last_became_active":"2024-04-23T16:41:57.309349+0000","last_became_peered":"2024-04-23T16:41:57.309349+0000","last_unstale":"2024-04-23T16:43:59.269001+0000","last_undegraded":"2024-04-23T16:43:59.269001+0000","last_fullsized":"2024-04-23T16:43:59.269001+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6582","last_scrub_stamp":"2024-04-23T16:43:26.589248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:26.589248+0000","objects_scrubbed":6550,"log_size":8182,"log_dups_size":0,"ondisk_log_size":8182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:27.742741+0000","scrub_duration":1397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423493632,"num_objects":6462,"num_object_clones":0,"num_object_copies":12924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8182,"num_write_kb":468608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1442,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:55.334118+0000","last_change":"2024-04-23T16:42:33.628340+0000","last_active":"2024-04-23T16:43:55.334118+0000","last_peered":"2024-04-23T16:43:55.334118+0000","last_clean":"2024-04-23T16:43:55.334118+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:43:55.334118+0000","last_undegraded":"2024-04-23T16:43:55.334118+0000","last_fullsized":"2024-04-23T16:43:55.334118+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:33.628217+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:42:33.628217+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6868893720,"num_objects":104812,"num_object_clones":0,"num_object_copies":209624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104812,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132546,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5615,"num_bytes_recovered":367984640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14062329856,"data_stored":14062321712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132546,"ondisk_log_size":132546,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1474560,"data_stored":1460352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056984,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2219284,"kb_used_data":1602184,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":91525868,"statfs":{"total":95995035648,"available":93722488832,"internally_reserved":0,"allocated":1640636416,"data_stored":1640268449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":631830980},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351326,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612964,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2193808,"kb_used_data":1671176,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":91551344,"statfs":{"total":95995035648,"available":93748576256,"internally_reserved":0,"allocated":1711284224,"data_stored":1710916257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":535165376},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841896,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1195740,"kb_used_data":843856,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":92549412,"statfs":{"total":95995035648,"available":94770597888,"internally_reserved":0,"allocated":864108544,"data_stored":863737025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168942,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1117968,"kb_used_data":541128,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92627184,"statfs":{"total":95995035648,"available":94850236416,"internally_reserved":0,"allocated":554115072,"data_stored":553747105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":590608842},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463283,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2250452,"kb_used_data":1800012,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":91494700,"statfs":{"total":95995035648,"available":93690572800,"internally_reserved":0,"allocated":1843212288,"data_stored":1842840249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":461240774},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137423,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1107292,"kb_used_data":876684,"kb_used_omap":231,"kb_used_meta":230360,"kb_avail":92637860,"statfs":{"total":95995035648,"available":94861168640,"internally_reserved":0,"allocated":897724416,"data_stored":897352377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":235889552},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738769,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2233808,"kb_used_data":1679240,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":91511344,"statfs":{"total":95995035648,"available":93707616256,"internally_reserved":0,"allocated":1719541760,"data_stored":1719173793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269439,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921147,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2816400,"kb_used_data":2086920,"kb_used_omap":1,"kb_used_meta":729470,"kb_avail":90928752,"statfs":{"total":95995035648,"available":93111042048,"internally_reserved":0,"allocated":2137006080,"data_stored":2136638113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":746977733},"hb_peers":[0,1,2,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757622,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1914844,"kb_used_data":1279760,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":91830308,"statfs":{"total":95995035648,"available":94034235392,"internally_reserved":0,"allocated":1310474240,"data_stored":1310102721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1717108736,"data_stored":1717108736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":894701568,"data_stored":894697496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1411842048,"data_stored":1411842048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2135687168,"data_stored":2135687168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1307967488,"data_stored":1307967488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1841238016,"data_stored":1841233944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":553385984,"data_stored":553385984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":861011968,"data_stored":861011968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1710030848,"data_stored":1710030848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1629356032,"data_stored":1629356032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:44:03.237 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:44:03.310 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:44:03.347 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:03.495 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:44:03.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:03.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:44:03.603 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:44:03.684 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:03.868 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:44:03.911 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:44:03.954 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:03 smithi148 ceph-mon[44520]: pgmap v1611: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.4 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 972 op/s; 17080/208068 objects degraded (8.209%); 8.1 MiB/s, 129 objects/s recovering 2024-04-23T16:44:04.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:03 smithi188 ceph-mon[44101]: pgmap v1611: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.4 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 972 op/s; 17080/208068 objects degraded (8.209%); 8.1 MiB/s, 129 objects/s recovering 2024-04-23T16:44:04.226 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:03 smithi103 ceph-mon[39629]: pgmap v1611: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.4 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 972 op/s; 17080/208068 objects degraded (8.209%); 8.1 MiB/s, 129 objects/s recovering 2024-04-23T16:44:04.308 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:44:04.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:04.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:44:04.412 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:44:04.494 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:04.614 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:44:04.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:04.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:44:04.731 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:44:04.762 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:04.787 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:44:04.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:04.790 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:44:04.891 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:44:04.925 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:04.976 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:44:05.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:04 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2816928379' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:44:05.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:04 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 17080/208068 objects degraded (8.209%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:05.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:04 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2816928379' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:44:05.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:04 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 17080/208068 objects degraded (8.209%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:05.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:04 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2816928379' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:44:05.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:04 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 17080/208068 objects degraded (8.209%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:06.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:05 smithi188 ceph-mon[44101]: pgmap v1612: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 822 op/s; 16517/201902 objects degraded (8.181%); 7.0 MiB/s, 112 objects/s recovering 2024-04-23T16:44:06.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:05 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:44:06.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:05 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2816928379' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:44:06.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:05 smithi188 ceph-mon[44101]: osdmap e578: 12 total, 11 up, 9 in 2024-04-23T16:44:06.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:05 smithi103 ceph-mon[39629]: pgmap v1612: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 822 op/s; 16517/201902 objects degraded (8.181%); 7.0 MiB/s, 112 objects/s recovering 2024-04-23T16:44:06.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:05 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:44:06.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:05 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2816928379' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:44:06.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:05 smithi103 ceph-mon[39629]: osdmap e578: 12 total, 11 up, 9 in 2024-04-23T16:44:06.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:05 smithi148 ceph-mon[44520]: pgmap v1612: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 822 op/s; 16517/201902 objects degraded (8.181%); 7.0 MiB/s, 112 objects/s recovering 2024-04-23T16:44:06.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:05 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:44:06.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:05 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2816928379' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:44:06.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:05 smithi148 ceph-mon[44520]: osdmap e578: 12 total, 11 up, 9 in 2024-04-23T16:44:06.476 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:44:06.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:06.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:44:06.581 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:44:06.651 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:06.712 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:44:06.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:06.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:44:06.817 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:44:06.847 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:07.262 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:44:07.262 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:44:07.270 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:44:07.272 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:44:07.627 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:44:07.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:07.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:44:07.731 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:44:07.759 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:08.222 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:07 smithi103 ceph-mon[39629]: pgmap v1614: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.1 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 824 op/s; 16517/200278 objects degraded (8.247%); 7.0 MiB/s, 112 objects/s recovering 2024-04-23T16:44:08.222 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:07 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:44:08.223 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:07 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:44:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:07 smithi188 ceph-mon[44101]: pgmap v1614: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.1 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 824 op/s; 16517/200278 objects degraded (8.247%); 7.0 MiB/s, 112 objects/s recovering 2024-04-23T16:44:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:07 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:44:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:07 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:44:08.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:07 smithi148 ceph-mon[44520]: pgmap v1614: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.1 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 824 op/s; 16517/200278 objects degraded (8.247%); 7.0 MiB/s, 112 objects/s recovering 2024-04-23T16:44:08.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:07 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:44:08.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:07 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:44:08.822 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:44:08.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:08.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:44:08.831 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:44:08.852 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:44:08.872 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:44:08.899 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:44:08.920 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:44:08.927 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:44:08.951 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:44:08.961 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:08.979 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:44:08.990 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:44:08.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:08.993 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:44:09.013 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:44:09.036 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:44:09.056 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:44:09.082 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:44:09.083 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:44:09.083 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:44:09.083 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:44:09.083 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:44:09.083 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:44:09.083 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:44:09.083 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:44:09.083 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:44:09.083 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:44:09.083 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:44:09.083 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:44:09.083 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:44:09.094 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:44:09.134 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:09.154 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:44:09.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:09.159 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:44:09.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:08 smithi188 ceph-mon[44101]: from='client.16758 v1:172.21.15.103:0/888921497' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:44:09.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:08 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3374368422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:44:09.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:08 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:09.260 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:44:09.290 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:09.317 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:44:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:09.325 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:44:09.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:08 smithi103 ceph-mon[39629]: from='client.16758 v1:172.21.15.103:0/888921497' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:44:09.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:08 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3374368422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:44:09.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:09.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:08 smithi148 ceph-mon[44520]: from='client.16758 v1:172.21.15.103:0/888921497' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:44:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:08 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3374368422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:44:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:08 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:09.426 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:44:09.430 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1614,"stamp":"2024-04-23T16:44:06.859338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6563103288,"num_objects":100139,"num_object_clones":0,"num_object_copies":200278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100139,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137221,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6180,"num_bytes_recovered":405283904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138750,"ondisk_log_size":138750,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":16537688,"kb_used_data":11824452,"kb_used_omap":243,"kb_used_meta":4712588,"kb_avail":827168680,"statfs":{"total":863955320832,"available":847020728320,"internally_reserved":0,"allocated":12108238848,"data_stored":12104893735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":249416,"internal_metadata":4825690552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-540213248,"num_objects":-8243,"num_object_clones":0,"num_object_copies":-16486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8243,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1121,"num_bytes_recovered":73465856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002142"},"pg_stats":[{"pgid":"7.d","version":"577'8661","reported_seq":8745,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.619492+0000","last_change":"2024-04-23T16:41:57.310289+0000","last_active":"2024-04-23T16:44:04.619492+0000","last_peered":"2024-04-23T16:44:04.619492+0000","last_clean":"2024-04-23T16:44:04.619492+0000","last_became_active":"2024-04-23T16:41:57.310025+0000","last_became_peered":"2024-04-23T16:41:57.310025+0000","last_unstale":"2024-04-23T16:44:04.619492+0000","last_undegraded":"2024-04-23T16:44:04.619492+0000","last_fullsized":"2024-04-23T16:44:04.619492+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8661,"log_dups_size":0,"ondisk_log_size":8661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419954712,"num_objects":6409,"num_object_clones":0,"num_object_copies":12818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6409,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8661,"num_write_kb":482177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"578'8893","reported_seq":9055,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:05.844516+0000","last_change":"2024-04-23T16:43:29.012370+0000","last_active":"2024-04-23T16:44:05.844516+0000","last_peered":"2024-04-23T16:44:05.844516+0000","last_clean":"2024-04-23T16:44:05.844516+0000","last_became_active":"2024-04-23T16:41:57.310754+0000","last_became_peered":"2024-04-23T16:41:57.310754+0000","last_unstale":"2024-04-23T16:44:05.844516+0000","last_undegraded":"2024-04-23T16:44:05.844516+0000","last_fullsized":"2024-04-23T16:44:05.844516+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6945","last_scrub_stamp":"2024-04-23T16:43:29.012285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:29.012285+0000","objects_scrubbed":6912,"log_size":8893,"log_dups_size":0,"ondisk_log_size":8893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:58.968073+0000","scrub_duration":1082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409665536,"num_objects":6251,"num_object_clones":0,"num_object_copies":12502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8893,"num_write_kb":484608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"577'8603","reported_seq":8760,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.385888+0000","last_change":"2024-04-23T16:43:25.997269+0000","last_active":"2024-04-23T16:44:04.385888+0000","last_peered":"2024-04-23T16:44:04.385888+0000","last_clean":"2024-04-23T16:44:04.385888+0000","last_became_active":"2024-04-23T16:41:57.310189+0000","last_became_peered":"2024-04-23T16:41:57.310189+0000","last_unstale":"2024-04-23T16:44:04.385888+0000","last_undegraded":"2024-04-23T16:44:04.385888+0000","last_fullsized":"2024-04-23T16:44:04.385888+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'6616","last_scrub_stamp":"2024-04-23T16:43:25.997150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:25.997150+0000","objects_scrubbed":6600,"log_size":8603,"log_dups_size":0,"ondisk_log_size":8603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:37.694305+0000","scrub_duration":806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412811264,"num_objects":6299,"num_object_clones":0,"num_object_copies":12598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8603,"num_write_kb":476864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"577'8622","reported_seq":8778,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.440121+0000","last_change":"2024-04-23T16:43:26.892913+0000","last_active":"2024-04-23T16:44:04.440121+0000","last_peered":"2024-04-23T16:44:04.440121+0000","last_clean":"2024-04-23T16:44:04.440121+0000","last_became_active":"2024-04-23T16:41:57.309569+0000","last_became_peered":"2024-04-23T16:41:57.309569+0000","last_unstale":"2024-04-23T16:44:04.440121+0000","last_undegraded":"2024-04-23T16:44:04.440121+0000","last_fullsized":"2024-04-23T16:44:04.440121+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6695","last_scrub_stamp":"2024-04-23T16:43:26.892785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:26.892785+0000","objects_scrubbed":6673,"log_size":8622,"log_dups_size":0,"ondisk_log_size":8622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:56.021559+0000","scrub_duration":739,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410910720,"num_objects":6270,"num_object_clones":0,"num_object_copies":12540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8622,"num_write_kb":476544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"578'8430","reported_seq":8515,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:05.614960+0000","last_change":"2024-04-23T16:41:57.310144+0000","last_active":"2024-04-23T16:44:05.614960+0000","last_peered":"2024-04-23T16:44:05.614960+0000","last_clean":"2024-04-23T16:44:05.614960+0000","last_became_active":"2024-04-23T16:41:57.309841+0000","last_became_peered":"2024-04-23T16:41:57.309841+0000","last_unstale":"2024-04-23T16:44:05.614960+0000","last_undegraded":"2024-04-23T16:44:05.614960+0000","last_fullsized":"2024-04-23T16:44:05.614960+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8430,"log_dups_size":0,"ondisk_log_size":8430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400424960,"num_objects":6110,"num_object_clones":0,"num_object_copies":12220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8430,"num_write_kb":465280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"577'8561","reported_seq":11964,"reported_epoch":577,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:04.646986+0000","last_change":"2024-04-23T16:43:08.145899+0000","last_active":"2024-04-23T16:44:04.646986+0000","last_peered":"2024-04-23T16:44:04.646986+0000","last_clean":"2024-04-23T16:42:42.970788+0000","last_became_active":"2024-04-23T16:43:08.106076+0000","last_became_peered":"2024-04-23T16:43:08.106076+0000","last_unstale":"2024-04-23T16:44:04.646986+0000","last_undegraded":"2024-04-23T16:43:07.951894+0000","last_fullsized":"2024-04-23T16:44:04.646986+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8561,"log_dups_size":0,"ondisk_log_size":8561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405340160,"num_objects":6185,"num_object_clones":0,"num_object_copies":12370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8561,"num_write_kb":471872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1657,"num_bytes_recovered":108593152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":4057},{"shards":"6,11","objects":2128}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1459,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:55.331279+0000","last_change":"2024-04-23T16:42:37.344941+0000","last_active":"2024-04-23T16:43:55.331279+0000","last_peered":"2024-04-23T16:43:55.331279+0000","last_clean":"2024-04-23T16:43:55.331279+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:43:55.331279+0000","last_undegraded":"2024-04-23T16:43:55.331279+0000","last_fullsized":"2024-04-23T16:43:55.331279+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:37.344822+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:42:37.344822+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:43:56.281436+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"577'8537","reported_seq":8695,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.452389+0000","last_change":"2024-04-23T16:43:28.303396+0000","last_active":"2024-04-23T16:44:04.452389+0000","last_peered":"2024-04-23T16:44:04.452389+0000","last_clean":"2024-04-23T16:44:04.452389+0000","last_became_active":"2024-04-23T16:41:57.311753+0000","last_became_peered":"2024-04-23T16:41:57.311753+0000","last_unstale":"2024-04-23T16:44:04.452389+0000","last_undegraded":"2024-04-23T16:44:04.452389+0000","last_fullsized":"2024-04-23T16:44:04.452389+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6763","last_scrub_stamp":"2024-04-23T16:43:28.303250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:28.303250+0000","objects_scrubbed":6733,"log_size":8537,"log_dups_size":0,"ondisk_log_size":8537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:51.576731+0000","scrub_duration":1201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418709504,"num_objects":6389,"num_object_clones":0,"num_object_copies":12778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8537,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1465,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:55.334216+0000","last_change":"2024-04-23T16:43:27.668916+0000","last_active":"2024-04-23T16:43:55.334216+0000","last_peered":"2024-04-23T16:43:55.334216+0000","last_clean":"2024-04-23T16:43:55.334216+0000","last_became_active":"2024-04-23T16:43:07.962092+0000","last_became_peered":"2024-04-23T16:43:07.962092+0000","last_unstale":"2024-04-23T16:43:55.334216+0000","last_undegraded":"2024-04-23T16:43:55.334216+0000","last_fullsized":"2024-04-23T16:43:55.334216+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:43:27.668793+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:43:27.668793+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:45.595521+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"577'8563","reported_seq":8720,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.418927+0000","last_change":"2024-04-23T16:43:27.928311+0000","last_active":"2024-04-23T16:44:04.418927+0000","last_peered":"2024-04-23T16:44:04.418927+0000","last_clean":"2024-04-23T16:44:04.418927+0000","last_became_active":"2024-04-23T16:41:57.305944+0000","last_became_peered":"2024-04-23T16:41:57.305944+0000","last_unstale":"2024-04-23T16:44:04.418927+0000","last_undegraded":"2024-04-23T16:44:04.418927+0000","last_fullsized":"2024-04-23T16:44:04.418927+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6718","last_scrub_stamp":"2024-04-23T16:43:27.928174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:27.928174+0000","objects_scrubbed":6703,"log_size":8563,"log_dups_size":0,"ondisk_log_size":8563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:34.856929+0000","scrub_duration":750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411893760,"num_objects":6285,"num_object_clones":0,"num_object_copies":12570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8563,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1444,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.992192+0000","last_change":"2024-04-23T16:42:32.662620+0000","last_active":"2024-04-23T16:44:04.992192+0000","last_peered":"2024-04-23T16:44:04.992192+0000","last_clean":"2024-04-23T16:44:04.992192+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:44:04.992192+0000","last_undegraded":"2024-04-23T16:44:04.992192+0000","last_fullsized":"2024-04-23T16:44:04.992192+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:32.662476+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:42:32.662476+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"577'8803","reported_seq":8961,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.442316+0000","last_change":"2024-04-23T16:43:25.389006+0000","last_active":"2024-04-23T16:44:04.442316+0000","last_peered":"2024-04-23T16:44:04.442316+0000","last_clean":"2024-04-23T16:44:04.442316+0000","last_became_active":"2024-04-23T16:41:57.309858+0000","last_became_peered":"2024-04-23T16:41:57.309858+0000","last_unstale":"2024-04-23T16:44:04.442316+0000","last_undegraded":"2024-04-23T16:44:04.442316+0000","last_fullsized":"2024-04-23T16:44:04.442316+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'6776","last_scrub_stamp":"2024-04-23T16:43:25.388898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:25.388898+0000","objects_scrubbed":6743,"log_size":8803,"log_dups_size":0,"ondisk_log_size":8803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:39.155605+0000","scrub_duration":1233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417005568,"num_objects":6363,"num_object_clones":0,"num_object_copies":12726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8803,"num_write_kb":485312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1466,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:55.327970+0000","last_change":"2024-04-23T16:43:40.258384+0000","last_active":"2024-04-23T16:43:55.327970+0000","last_peered":"2024-04-23T16:43:55.327970+0000","last_clean":"2024-04-23T16:43:55.327970+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:43:55.327970+0000","last_undegraded":"2024-04-23T16:43:55.327970+0000","last_fullsized":"2024-04-23T16:43:55.327970+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:43:40.258213+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:43:40.258213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:07.779820+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"578'8600","reported_seq":8684,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:06.511015+0000","last_change":"2024-04-23T16:41:57.310695+0000","last_active":"2024-04-23T16:44:06.511015+0000","last_peered":"2024-04-23T16:44:06.511015+0000","last_clean":"2024-04-23T16:44:06.511015+0000","last_became_active":"2024-04-23T16:41:57.310400+0000","last_became_peered":"2024-04-23T16:41:57.310400+0000","last_unstale":"2024-04-23T16:44:06.511015+0000","last_undegraded":"2024-04-23T16:44:06.511015+0000","last_fullsized":"2024-04-23T16:44:06.511015+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8600,"log_dups_size":0,"ondisk_log_size":8600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399638528,"num_objects":6098,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8600,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1034,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.974159+0000","last_change":"2024-04-23T16:42:51.989882+0000","last_active":"2024-04-23T16:44:04.974159+0000","last_peered":"2024-04-23T16:44:04.974159+0000","last_clean":"2024-04-23T16:44:04.974159+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:44:04.974159+0000","last_undegraded":"2024-04-23T16:44:04.974159+0000","last_fullsized":"2024-04-23T16:44:04.974159+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:51.989679+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:42:51.989679+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:07.606374+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"577'8531","reported_seq":8614,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.647411+0000","last_change":"2024-04-23T16:41:57.306526+0000","last_active":"2024-04-23T16:44:04.647411+0000","last_peered":"2024-04-23T16:44:04.647411+0000","last_clean":"2024-04-23T16:44:04.647411+0000","last_became_active":"2024-04-23T16:41:57.306229+0000","last_became_peered":"2024-04-23T16:41:57.306229+0000","last_unstale":"2024-04-23T16:44:04.647411+0000","last_undegraded":"2024-04-23T16:44:04.647411+0000","last_fullsized":"2024-04-23T16:44:04.647411+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8531,"log_dups_size":0,"ondisk_log_size":8531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404553728,"num_objects":6173,"num_object_clones":0,"num_object_copies":12346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8531,"num_write_kb":470528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":986,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:55.346017+0000","last_change":"2024-04-23T16:42:32.970258+0000","last_active":"2024-04-23T16:43:55.346017+0000","last_peered":"2024-04-23T16:43:55.346017+0000","last_clean":"2024-04-23T16:43:55.346017+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:43:55.346017+0000","last_undegraded":"2024-04-23T16:43:55.346017+0000","last_fullsized":"2024-04-23T16:43:55.346017+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:32.970112+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:42:32.970112+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.0","version":"577'8395","reported_seq":8556,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:44:02.661055+0000","last_change":"2024-04-23T16:43:30.757721+0000","last_active":"2024-04-23T16:44:02.661055+0000","last_peered":"2024-04-23T16:44:02.661055+0000","last_clean":"2024-04-23T16:44:02.661055+0000","last_became_active":"2024-04-23T16:41:57.311479+0000","last_became_peered":"2024-04-23T16:41:57.311479+0000","last_unstale":"2024-04-23T16:44:02.661055+0000","last_undegraded":"2024-04-23T16:44:02.661055+0000","last_fullsized":"2024-04-23T16:44:02.661055+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6823","last_scrub_stamp":"2024-04-23T16:43:30.757555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:30.757555+0000","objects_scrubbed":6802,"log_size":8395,"log_dups_size":0,"ondisk_log_size":8395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:59.054459+0000","scrub_duration":1354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413204480,"num_objects":6305,"num_object_clones":0,"num_object_copies":12610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8395,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1467,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:55.331370+0000","last_change":"2024-04-23T16:42:50.245835+0000","last_active":"2024-04-23T16:43:55.331370+0000","last_peered":"2024-04-23T16:43:55.331370+0000","last_clean":"2024-04-23T16:43:55.331370+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:43:55.331370+0000","last_undegraded":"2024-04-23T16:43:55.331370+0000","last_fullsized":"2024-04-23T16:43:55.331370+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:42:50.245710+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:10.789326+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.1","version":"577'8421","reported_seq":11087,"reported_epoch":577,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:04.269087+0000","last_change":"2024-04-23T16:43:08.081567+0000","last_active":"2024-04-23T16:44:04.269087+0000","last_peered":"2024-04-23T16:44:04.269087+0000","last_clean":"2024-04-23T16:42:42.976600+0000","last_became_active":"2024-04-23T16:43:08.067887+0000","last_became_peered":"2024-04-23T16:43:08.067887+0000","last_unstale":"2024-04-23T16:44:04.269087+0000","last_undegraded":"2024-04-23T16:43:07.950381+0000","last_fullsized":"2024-04-23T16:44:04.269087+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8421,"log_dups_size":0,"ondisk_log_size":8421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407306240,"num_objects":6215,"num_object_clones":0,"num_object_copies":12430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8421,"num_write_kb":468352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1289,"num_bytes_recovered":84475904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":4299},{"shards":"4,6","objects":1916}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1459,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:55.336091+0000","last_change":"2024-04-23T16:43:07.918302+0000","last_active":"2024-04-23T16:43:55.336091+0000","last_peered":"2024-04-23T16:43:55.336091+0000","last_clean":"2024-04-23T16:43:55.336091+0000","last_became_active":"2024-04-23T16:43:07.918042+0000","last_became_peered":"2024-04-23T16:43:07.918042+0000","last_unstale":"2024-04-23T16:43:55.336091+0000","last_undegraded":"2024-04-23T16:43:55.336091+0000","last_fullsized":"2024-04-23T16:43:55.336091+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:33.716343+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:42:33.716343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"577'8547","reported_seq":11515,"reported_epoch":577,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:04.617300+0000","last_change":"2024-04-23T16:43:08.363306+0000","last_active":"2024-04-23T16:44:04.617300+0000","last_peered":"2024-04-23T16:44:04.617300+0000","last_clean":"2024-04-23T16:42:42.950766+0000","last_became_active":"2024-04-23T16:43:08.333793+0000","last_became_peered":"2024-04-23T16:43:08.333793+0000","last_unstale":"2024-04-23T16:44:04.617300+0000","last_undegraded":"2024-04-23T16:43:07.950356+0000","last_fullsized":"2024-04-23T16:44:04.617300+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8547,"log_dups_size":0,"ondisk_log_size":8547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410976256,"num_objects":6271,"num_object_clones":0,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8547,"num_write_kb":474176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1441,"num_bytes_recovered":94437376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":4296},{"shards":"6,7","objects":1975}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1353,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.975850+0000","last_change":"2024-04-23T16:41:47.722866+0000","last_active":"2024-04-23T16:44:04.975850+0000","last_peered":"2024-04-23T16:44:04.975850+0000","last_clean":"2024-04-23T16:44:04.975850+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:44:04.975850+0000","last_undegraded":"2024-04-23T16:44:04.975850+0000","last_fullsized":"2024-04-23T16:44:04.975850+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:41:47.722762+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"577'8614","reported_seq":12285,"reported_epoch":577,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:04.249100+0000","last_change":"2024-04-23T16:43:08.136028+0000","last_active":"2024-04-23T16:44:04.249100+0000","last_peered":"2024-04-23T16:44:04.249100+0000","last_clean":"2024-04-23T16:42:42.962639+0000","last_became_active":"2024-04-23T16:43:08.109240+0000","last_became_peered":"2024-04-23T16:43:08.109240+0000","last_unstale":"2024-04-23T16:44:04.249100+0000","last_undegraded":"2024-04-23T16:43:07.955254+0000","last_fullsized":"2024-04-23T16:44:04.249100+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8614,"log_dups_size":0,"ondisk_log_size":8614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413532160,"num_objects":6310,"num_object_clones":0,"num_object_copies":12620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8614,"num_write_kb":477568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1791,"num_bytes_recovered":117374976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":3865},{"shards":"4,11","objects":2445}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1273,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:43:55.336267+0000","last_change":"2024-04-23T16:41:49.835121+0000","last_active":"2024-04-23T16:43:55.336267+0000","last_peered":"2024-04-23T16:43:55.336267+0000","last_clean":"2024-04-23T16:43:55.336267+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:43:55.336267+0000","last_undegraded":"2024-04-23T16:43:55.336267+0000","last_fullsized":"2024-04-23T16:43:55.336267+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:41:49.834374+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"577'8440","reported_seq":8596,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.425881+0000","last_change":"2024-04-23T16:43:26.589353+0000","last_active":"2024-04-23T16:44:04.425881+0000","last_peered":"2024-04-23T16:44:04.425881+0000","last_clean":"2024-04-23T16:44:04.425881+0000","last_became_active":"2024-04-23T16:41:57.309349+0000","last_became_peered":"2024-04-23T16:41:57.309349+0000","last_unstale":"2024-04-23T16:44:04.425881+0000","last_undegraded":"2024-04-23T16:44:04.425881+0000","last_fullsized":"2024-04-23T16:44:04.425881+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6582","last_scrub_stamp":"2024-04-23T16:43:26.589248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:26.589248+0000","objects_scrubbed":6550,"log_size":8440,"log_dups_size":0,"ondisk_log_size":8440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:27.742741+0000","scrub_duration":1397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406585344,"num_objects":6204,"num_object_clones":0,"num_object_copies":12408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8440,"num_write_kb":468608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1444,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.989793+0000","last_change":"2024-04-23T16:42:33.628340+0000","last_active":"2024-04-23T16:44:04.989793+0000","last_peered":"2024-04-23T16:44:04.989793+0000","last_clean":"2024-04-23T16:44:04.989793+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:44:04.989793+0000","last_undegraded":"2024-04-23T16:44:04.989793+0000","last_fullsized":"2024-04-23T16:44:04.989793+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:33.628217+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:42:33.628217+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6562512920,"num_objects":100137,"num_object_clones":0,"num_object_copies":200274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100137,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137221,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6178,"num_bytes_recovered":404881408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13459070976,"data_stored":13459062832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137221,"ondisk_log_size":137221,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1474560,"data_stored":1460352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056986,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2094816,"kb_used_data":1481940,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":91650336,"statfs":{"total":95995035648,"available":93849944064,"internally_reserved":0,"allocated":1517506560,"data_stored":1517132539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":627571140},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351327,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612965,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2136784,"kb_used_data":1609992,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":91608368,"statfs":{"total":95995035648,"available":93806968832,"internally_reserved":0,"allocated":1648631808,"data_stored":1648263841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":539425216},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841897,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1164120,"kb_used_data":808080,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92581032,"statfs":{"total":95995035648,"available":94802976768,"internally_reserved":0,"allocated":827473920,"data_stored":827102401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168943,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1092896,"kb_used_data":516116,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92652256,"statfs":{"total":95995035648,"available":94875910144,"internally_reserved":0,"allocated":528502784,"data_stored":528128763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":590608842},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463284,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2212372,"kb_used_data":1739660,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":91532780,"statfs":{"total":95995035648,"available":93729566720,"internally_reserved":0,"allocated":1781411840,"data_stored":1781039801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":483981766},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137425,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1044388,"kb_used_data":809624,"kb_used_omap":231,"kb_used_meta":234520,"kb_avail":92700764,"statfs":{"total":95995035648,"available":94925582336,"internally_reserved":0,"allocated":829054976,"data_stored":828676883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":240149392},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738770,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2186448,"kb_used_data":1609672,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":91558704,"statfs":{"total":95995035648,"available":93756112896,"internally_reserved":0,"allocated":1648304128,"data_stored":1647936161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269440,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921148,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2736336,"kb_used_data":2019080,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":91008816,"statfs":{"total":95995035648,"available":93193027584,"internally_reserved":0,"allocated":2067537920,"data_stored":2067169953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[0,1,2,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757623,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1869528,"kb_used_data":1230288,"kb_used_omap":1,"kb_used_meta":639230,"kb_avail":91875624,"statfs":{"total":95995035648,"available":94080638976,"internally_reserved":0,"allocated":1259814912,"data_stored":1259443393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":654571973},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1646395392,"data_stored":1646395392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":825102336,"data_stored":825098264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1411842048,"data_stored":1411842048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2050424832,"data_stored":2050424832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1257439232,"data_stored":1257439232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1765412864,"data_stored":1765408792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":527433728,"data_stored":527433728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":824180736,"data_stored":824180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1645346816,"data_stored":1645346816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1505492992,"data_stored":1505492992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:44:09.432 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:44:09.453 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:09.472 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:44:09.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:09.475 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:44:09.577 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:44:09.606 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:09.627 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:44:09.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:09.630 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:44:09.731 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:44:09.757 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:09 smithi103 ceph-mon[39629]: pgmap v1615: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.1 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 823 op/s; 16517/199790 objects degraded (8.267%); 7.0 MiB/s, 112 objects/s recovering 2024-04-23T16:44:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:09 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 16517/199790 objects degraded (8.267%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:10.218 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:44:10.219 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:44:10.242 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:44:10.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:10.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:44:10.291 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:44:10.349 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:44:10.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:09 smithi148 ceph-mon[44520]: pgmap v1615: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.1 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 823 op/s; 16517/199790 objects degraded (8.267%); 7.0 MiB/s, 112 objects/s recovering 2024-04-23T16:44:10.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:09 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 16517/199790 objects degraded (8.267%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:10.377 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:10.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:09 smithi188 ceph-mon[44101]: pgmap v1615: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 6.1 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 823 op/s; 16517/199790 objects degraded (8.267%); 7.0 MiB/s, 112 objects/s recovering 2024-04-23T16:44:10.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:10.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:09 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 16517/199790 objects degraded (8.267%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:10.576 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:44:10.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:10.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:44:10.609 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:44:10.679 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:44:10.709 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:10.730 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:44:10.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:10.733 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:44:10.834 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:44:10.867 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:11.046 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-23T16:44:11.217 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:44:11.219 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:11.219 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:10 smithi103 ceph-mon[39629]: 1.a scrub starts 2024-04-23T16:44:11.219 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:10 smithi103 ceph-mon[39629]: 1.a scrub ok 2024-04-23T16:44:11.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:11.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:44:11.321 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:44:11.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:11.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:10 smithi188 ceph-mon[44101]: 1.a scrub starts 2024-04-23T16:44:11.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:10 smithi188 ceph-mon[44101]: 1.a scrub ok 2024-04-23T16:44:11.351 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:11.374 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:44:11.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:11.378 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:44:11.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:11.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:10 smithi148 ceph-mon[44520]: 1.a scrub starts 2024-04-23T16:44:11.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:10 smithi148 ceph-mon[44520]: 1.a scrub ok 2024-04-23T16:44:11.478 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:44:11.570 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:11.624 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:44:11.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:11.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:44:11.645 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:44:11.735 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:44:11.816 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:11.852 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:44:11.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:11.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:44:11.948 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:44:11.967 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:44:12.005 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:12.049 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:44:12.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:12.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:44:12.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:12 smithi148 ceph-mon[44520]: pgmap v1616: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.9 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 830 op/s; 15935/193020 objects degraded (8.256%); 7.2 MiB/s, 114 objects/s recovering 2024-04-23T16:44:12.167 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:44:12.196 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:11 smithi103 ceph-mon[39629]: pgmap v1616: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.9 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 830 op/s; 15935/193020 objects degraded (8.256%); 7.2 MiB/s, 114 objects/s recovering 2024-04-23T16:44:12.199 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:12.220 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:44:12.274 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:44:12.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:12.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:44:12.383 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:44:12.413 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:12.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:11 smithi188 ceph-mon[44101]: pgmap v1616: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.9 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 830 op/s; 15935/193020 objects degraded (8.256%); 7.2 MiB/s, 114 objects/s recovering 2024-04-23T16:44:12.594 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:44:12.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:12.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:44:12.698 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:44:12.817 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:12.840 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:44:12.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:12.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:44:12.955 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:44:12.985 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:13.006 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:44:13.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:13.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:44:13.114 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:44:13.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:13 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T16:44:13.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:13 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T16:44:13.177 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:13.288 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:44:13.290 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:44:13.317 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:13 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T16:44:13.317 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:13 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T16:44:13.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:13 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T16:44:13.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:13 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T16:44:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:14 smithi103 ceph-mon[39629]: pgmap v1617: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.8 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 830 op/s; 15935/191452 objects degraded (8.323%); 7.2 MiB/s, 114 objects/s recovering 2024-04-23T16:44:14.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:14 smithi148 ceph-mon[44520]: pgmap v1617: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.8 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 830 op/s; 15935/191452 objects degraded (8.323%); 7.2 MiB/s, 114 objects/s recovering 2024-04-23T16:44:14.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:14 smithi188 ceph-mon[44101]: pgmap v1617: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.8 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 830 op/s; 15935/191452 objects degraded (8.323%); 7.2 MiB/s, 114 objects/s recovering 2024-04-23T16:44:15.030 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1617,"stamp":"2024-04-23T16:44:12.860450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6273892920,"num_objects":95726,"num_object_clones":0,"num_object_copies":191452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95726,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141634,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6762,"num_bytes_recovered":443425856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143163,"ondisk_log_size":143163,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":15975124,"kb_used_data":11269132,"kb_used_omap":243,"kb_used_meta":4705676,"kb_avail":827731244,"statfs":{"total":863955320832,"available":847596793856,"internally_reserved":0,"allocated":11539591168,"data_stored":11536209731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":249416,"internal_metadata":4818612664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-544473088,"num_objects":-8308,"num_object_clones":0,"num_object_copies":-16616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8308,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1145,"num_bytes_recovered":75038720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002016"},"pg_stats":[{"pgid":"7.d","version":"578'8926","reported_seq":9012,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:09.763255+0000","last_change":"2024-04-23T16:41:57.310289+0000","last_active":"2024-04-23T16:44:09.763255+0000","last_peered":"2024-04-23T16:44:09.763255+0000","last_clean":"2024-04-23T16:44:09.763255+0000","last_became_active":"2024-04-23T16:41:57.310025+0000","last_became_peered":"2024-04-23T16:41:57.310025+0000","last_unstale":"2024-04-23T16:44:09.763255+0000","last_undegraded":"2024-04-23T16:44:09.763255+0000","last_fullsized":"2024-04-23T16:44:09.763255+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8926,"log_dups_size":0,"ondisk_log_size":8926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402587672,"num_objects":6144,"num_object_clones":0,"num_object_copies":12288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6144,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8926,"num_write_kb":482177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"578'9162","reported_seq":9323,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:10.869565+0000","last_change":"2024-04-23T16:43:29.012370+0000","last_active":"2024-04-23T16:44:10.869565+0000","last_peered":"2024-04-23T16:44:10.869565+0000","last_clean":"2024-04-23T16:44:10.869565+0000","last_became_active":"2024-04-23T16:41:57.310754+0000","last_became_peered":"2024-04-23T16:41:57.310754+0000","last_unstale":"2024-04-23T16:44:10.869565+0000","last_undegraded":"2024-04-23T16:44:10.869565+0000","last_fullsized":"2024-04-23T16:44:10.869565+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6945","last_scrub_stamp":"2024-04-23T16:43:29.012285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:29.012285+0000","objects_scrubbed":6912,"log_size":9162,"log_dups_size":0,"ondisk_log_size":9162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:58.968073+0000","scrub_duration":1082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392036352,"num_objects":5982,"num_object_clones":0,"num_object_copies":11964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9162,"num_write_kb":484608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"578'8847","reported_seq":9007,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:09.450160+0000","last_change":"2024-04-23T16:43:25.997269+0000","last_active":"2024-04-23T16:44:09.450160+0000","last_peered":"2024-04-23T16:44:09.450160+0000","last_clean":"2024-04-23T16:44:09.450160+0000","last_became_active":"2024-04-23T16:41:57.310189+0000","last_became_peered":"2024-04-23T16:41:57.310189+0000","last_unstale":"2024-04-23T16:44:09.450160+0000","last_undegraded":"2024-04-23T16:44:09.450160+0000","last_fullsized":"2024-04-23T16:44:09.450160+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'6616","last_scrub_stamp":"2024-04-23T16:43:25.997150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:25.997150+0000","objects_scrubbed":6600,"log_size":8847,"log_dups_size":0,"ondisk_log_size":8847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:37.694305+0000","scrub_duration":806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396820480,"num_objects":6055,"num_object_clones":0,"num_object_copies":12110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8847,"num_write_kb":476864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"578'8872","reported_seq":9031,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:09.430040+0000","last_change":"2024-04-23T16:43:26.892913+0000","last_active":"2024-04-23T16:44:09.430040+0000","last_peered":"2024-04-23T16:44:09.430040+0000","last_clean":"2024-04-23T16:44:09.430040+0000","last_became_active":"2024-04-23T16:41:57.309569+0000","last_became_peered":"2024-04-23T16:41:57.309569+0000","last_unstale":"2024-04-23T16:44:09.430040+0000","last_undegraded":"2024-04-23T16:44:09.430040+0000","last_fullsized":"2024-04-23T16:44:09.430040+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6695","last_scrub_stamp":"2024-04-23T16:43:26.892785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:26.892785+0000","objects_scrubbed":6673,"log_size":8872,"log_dups_size":0,"ondisk_log_size":8872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:56.021559+0000","scrub_duration":739,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394526720,"num_objects":6020,"num_object_clones":0,"num_object_copies":12040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8872,"num_write_kb":476544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"578'8689","reported_seq":8774,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:10.699429+0000","last_change":"2024-04-23T16:41:57.310144+0000","last_active":"2024-04-23T16:44:10.699429+0000","last_peered":"2024-04-23T16:44:10.699429+0000","last_clean":"2024-04-23T16:44:10.699429+0000","last_became_active":"2024-04-23T16:41:57.309841+0000","last_became_peered":"2024-04-23T16:41:57.309841+0000","last_unstale":"2024-04-23T16:44:10.699429+0000","last_undegraded":"2024-04-23T16:44:10.699429+0000","last_fullsized":"2024-04-23T16:44:10.699429+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8689,"log_dups_size":0,"ondisk_log_size":8689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383451136,"num_objects":5851,"num_object_clones":0,"num_object_copies":11702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8689,"num_write_kb":465280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"578'8845","reported_seq":12562,"reported_epoch":578,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:09.752900+0000","last_change":"2024-04-23T16:43:08.145899+0000","last_active":"2024-04-23T16:44:09.752900+0000","last_peered":"2024-04-23T16:44:09.752900+0000","last_clean":"2024-04-23T16:42:42.970788+0000","last_became_active":"2024-04-23T16:43:08.106076+0000","last_became_peered":"2024-04-23T16:43:08.106076+0000","last_unstale":"2024-04-23T16:44:09.752900+0000","last_undegraded":"2024-04-23T16:43:07.951894+0000","last_fullsized":"2024-04-23T16:44:09.752900+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8845,"log_dups_size":0,"ondisk_log_size":8845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386727936,"num_objects":5901,"num_object_clones":0,"num_object_copies":11802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3901,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8845,"num_write_kb":471872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1813,"num_bytes_recovered":118816768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":3901},{"shards":"6,11","objects":2000}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1469,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:09.435288+0000","last_change":"2024-04-23T16:44:09.435333+0000","last_active":"2024-04-23T16:44:09.435288+0000","last_peered":"2024-04-23T16:44:09.435288+0000","last_clean":"2024-04-23T16:44:09.435288+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:44:09.435288+0000","last_undegraded":"2024-04-23T16:44:09.435288+0000","last_fullsized":"2024-04-23T16:44:09.435288+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:09.435212+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:44:09.435212+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:17.709965+0000","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"578'8827","reported_seq":8987,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:09.406896+0000","last_change":"2024-04-23T16:43:28.303396+0000","last_active":"2024-04-23T16:44:09.406896+0000","last_peered":"2024-04-23T16:44:09.406896+0000","last_clean":"2024-04-23T16:44:09.406896+0000","last_became_active":"2024-04-23T16:41:57.311753+0000","last_became_peered":"2024-04-23T16:41:57.311753+0000","last_unstale":"2024-04-23T16:44:09.406896+0000","last_undegraded":"2024-04-23T16:44:09.406896+0000","last_fullsized":"2024-04-23T16:44:09.406896+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6763","last_scrub_stamp":"2024-04-23T16:43:28.303250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:28.303250+0000","objects_scrubbed":6733,"log_size":8827,"log_dups_size":0,"ondisk_log_size":8827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:51.576731+0000","scrub_duration":1201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399704064,"num_objects":6099,"num_object_clones":0,"num_object_copies":12198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8827,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1467,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.976338+0000","last_change":"2024-04-23T16:43:27.668916+0000","last_active":"2024-04-23T16:44:04.976338+0000","last_peered":"2024-04-23T16:44:04.976338+0000","last_clean":"2024-04-23T16:44:04.976338+0000","last_became_active":"2024-04-23T16:43:07.962092+0000","last_became_peered":"2024-04-23T16:43:07.962092+0000","last_unstale":"2024-04-23T16:44:04.976338+0000","last_undegraded":"2024-04-23T16:44:04.976338+0000","last_fullsized":"2024-04-23T16:44:04.976338+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:43:27.668793+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:43:27.668793+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:45.595521+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"578'8840","reported_seq":8999,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:09.370070+0000","last_change":"2024-04-23T16:43:27.928311+0000","last_active":"2024-04-23T16:44:09.370070+0000","last_peered":"2024-04-23T16:44:09.370070+0000","last_clean":"2024-04-23T16:44:09.370070+0000","last_became_active":"2024-04-23T16:41:57.305944+0000","last_became_peered":"2024-04-23T16:41:57.305944+0000","last_unstale":"2024-04-23T16:44:09.370070+0000","last_undegraded":"2024-04-23T16:44:09.370070+0000","last_fullsized":"2024-04-23T16:44:09.370070+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6718","last_scrub_stamp":"2024-04-23T16:43:27.928174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:27.928174+0000","objects_scrubbed":6703,"log_size":8840,"log_dups_size":0,"ondisk_log_size":8840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:34.856929+0000","scrub_duration":750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393740288,"num_objects":6008,"num_object_clones":0,"num_object_copies":12016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8840,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1444,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.992192+0000","last_change":"2024-04-23T16:42:32.662620+0000","last_active":"2024-04-23T16:44:04.992192+0000","last_peered":"2024-04-23T16:44:04.992192+0000","last_clean":"2024-04-23T16:44:04.992192+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:44:04.992192+0000","last_undegraded":"2024-04-23T16:44:04.992192+0000","last_fullsized":"2024-04-23T16:44:04.992192+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:32.662476+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:42:32.662476+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"578'9051","reported_seq":9211,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:09.390334+0000","last_change":"2024-04-23T16:43:25.389006+0000","last_active":"2024-04-23T16:44:09.390334+0000","last_peered":"2024-04-23T16:44:09.390334+0000","last_clean":"2024-04-23T16:44:09.390334+0000","last_became_active":"2024-04-23T16:41:57.309858+0000","last_became_peered":"2024-04-23T16:41:57.309858+0000","last_unstale":"2024-04-23T16:44:09.390334+0000","last_undegraded":"2024-04-23T16:44:09.390334+0000","last_fullsized":"2024-04-23T16:44:09.390334+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'6776","last_scrub_stamp":"2024-04-23T16:43:25.388898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:25.388898+0000","objects_scrubbed":6743,"log_size":9051,"log_dups_size":0,"ondisk_log_size":9051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:39.155605+0000","scrub_duration":1233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400752640,"num_objects":6115,"num_object_clones":0,"num_object_copies":12230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9051,"num_write_kb":485312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1468,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.976350+0000","last_change":"2024-04-23T16:43:40.258384+0000","last_active":"2024-04-23T16:44:04.976350+0000","last_peered":"2024-04-23T16:44:04.976350+0000","last_clean":"2024-04-23T16:44:04.976350+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:44:04.976350+0000","last_undegraded":"2024-04-23T16:44:04.976350+0000","last_fullsized":"2024-04-23T16:44:04.976350+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:43:40.258213+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:43:40.258213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:07.779820+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"578'8867","reported_seq":8953,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:11.514594+0000","last_change":"2024-04-23T16:41:57.310695+0000","last_active":"2024-04-23T16:44:11.514594+0000","last_peered":"2024-04-23T16:44:11.514594+0000","last_clean":"2024-04-23T16:44:11.514594+0000","last_became_active":"2024-04-23T16:41:57.310400+0000","last_became_peered":"2024-04-23T16:41:57.310400+0000","last_unstale":"2024-04-23T16:44:11.514594+0000","last_undegraded":"2024-04-23T16:44:11.514594+0000","last_fullsized":"2024-04-23T16:44:11.514594+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8867,"log_dups_size":0,"ondisk_log_size":8867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382140416,"num_objects":5831,"num_object_clones":0,"num_object_copies":11662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8867,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1034,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.974159+0000","last_change":"2024-04-23T16:42:51.989882+0000","last_active":"2024-04-23T16:44:04.974159+0000","last_peered":"2024-04-23T16:44:04.974159+0000","last_clean":"2024-04-23T16:44:04.974159+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:44:04.974159+0000","last_undegraded":"2024-04-23T16:44:04.974159+0000","last_fullsized":"2024-04-23T16:44:04.974159+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:51.989679+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:42:51.989679+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:07.606374+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"578'8782","reported_seq":8865,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:09.766639+0000","last_change":"2024-04-23T16:41:57.306526+0000","last_active":"2024-04-23T16:44:09.766639+0000","last_peered":"2024-04-23T16:44:09.766639+0000","last_clean":"2024-04-23T16:44:09.766639+0000","last_became_active":"2024-04-23T16:41:57.306229+0000","last_became_peered":"2024-04-23T16:41:57.306229+0000","last_unstale":"2024-04-23T16:44:09.766639+0000","last_undegraded":"2024-04-23T16:44:09.766639+0000","last_fullsized":"2024-04-23T16:44:09.766639+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8782,"log_dups_size":0,"ondisk_log_size":8782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388104192,"num_objects":5922,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8782,"num_write_kb":470528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":996,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:07.230736+0000","last_change":"2024-04-23T16:44:07.230780+0000","last_active":"2024-04-23T16:44:07.230736+0000","last_peered":"2024-04-23T16:44:07.230736+0000","last_clean":"2024-04-23T16:44:07.230736+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:44:07.230736+0000","last_undegraded":"2024-04-23T16:44:07.230736+0000","last_fullsized":"2024-04-23T16:44:07.230736+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:07.230640+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:44:07.230640+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:12.044599+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.0","version":"578'8887","reported_seq":9050,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:12.579794+0000","last_change":"2024-04-23T16:43:30.757721+0000","last_active":"2024-04-23T16:44:12.579794+0000","last_peered":"2024-04-23T16:44:12.579794+0000","last_clean":"2024-04-23T16:44:12.579794+0000","last_became_active":"2024-04-23T16:41:57.311479+0000","last_became_peered":"2024-04-23T16:41:57.311479+0000","last_unstale":"2024-04-23T16:44:12.579794+0000","last_undegraded":"2024-04-23T16:44:12.579794+0000","last_fullsized":"2024-04-23T16:44:12.579794+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6823","last_scrub_stamp":"2024-04-23T16:43:30.757555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:30.757555+0000","objects_scrubbed":6802,"log_size":8887,"log_dups_size":0,"ondisk_log_size":8887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:59.054459+0000","scrub_duration":1354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380960768,"num_objects":5813,"num_object_clones":0,"num_object_copies":11626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8887,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1469,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.977038+0000","last_change":"2024-04-23T16:42:50.245835+0000","last_active":"2024-04-23T16:44:04.977038+0000","last_peered":"2024-04-23T16:44:04.977038+0000","last_clean":"2024-04-23T16:44:04.977038+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:44:04.977038+0000","last_undegraded":"2024-04-23T16:44:04.977038+0000","last_fullsized":"2024-04-23T16:44:04.977038+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:42:50.245710+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:10.789326+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.1","version":"578'8670","reported_seq":11564,"reported_epoch":578,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:09.276296+0000","last_change":"2024-04-23T16:43:08.081567+0000","last_active":"2024-04-23T16:44:09.276296+0000","last_peered":"2024-04-23T16:44:09.276296+0000","last_clean":"2024-04-23T16:42:42.976600+0000","last_became_active":"2024-04-23T16:43:08.067887+0000","last_became_peered":"2024-04-23T16:43:08.067887+0000","last_unstale":"2024-04-23T16:44:09.276296+0000","last_undegraded":"2024-04-23T16:43:07.950381+0000","last_fullsized":"2024-04-23T16:44:09.276296+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8670,"log_dups_size":0,"ondisk_log_size":8670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390987776,"num_objects":5966,"num_object_clones":0,"num_object_copies":11932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8670,"num_write_kb":468352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1402,"num_bytes_recovered":91881472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":4186},{"shards":"4,6","objects":1780}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1461,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.975852+0000","last_change":"2024-04-23T16:43:07.918302+0000","last_active":"2024-04-23T16:44:04.975852+0000","last_peered":"2024-04-23T16:44:04.975852+0000","last_clean":"2024-04-23T16:44:04.975852+0000","last_became_active":"2024-04-23T16:43:07.918042+0000","last_became_peered":"2024-04-23T16:43:07.918042+0000","last_unstale":"2024-04-23T16:44:04.975852+0000","last_undegraded":"2024-04-23T16:44:04.975852+0000","last_fullsized":"2024-04-23T16:44:04.975852+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:33.716343+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:42:33.716343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"578'8815","reported_seq":12037,"reported_epoch":578,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:09.649757+0000","last_change":"2024-04-23T16:43:08.363306+0000","last_active":"2024-04-23T16:44:09.649757+0000","last_peered":"2024-04-23T16:44:09.649757+0000","last_clean":"2024-04-23T16:42:42.950766+0000","last_became_active":"2024-04-23T16:43:08.333793+0000","last_became_peered":"2024-04-23T16:43:08.333793+0000","last_unstale":"2024-04-23T16:44:09.649757+0000","last_undegraded":"2024-04-23T16:43:07.950356+0000","last_fullsized":"2024-04-23T16:44:09.649757+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8815,"log_dups_size":0,"ondisk_log_size":8815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393412608,"num_objects":6003,"num_object_clones":0,"num_object_copies":12006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8815,"num_write_kb":474176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1567,"num_bytes_recovered":102694912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":4170},{"shards":"6,7","objects":1833}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1353,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.975850+0000","last_change":"2024-04-23T16:41:47.722866+0000","last_active":"2024-04-23T16:44:04.975850+0000","last_peered":"2024-04-23T16:44:04.975850+0000","last_clean":"2024-04-23T16:44:04.975850+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:44:04.975850+0000","last_undegraded":"2024-04-23T16:44:04.975850+0000","last_fullsized":"2024-04-23T16:44:04.975850+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:41:47.722762+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"578'8856","reported_seq":12903,"reported_epoch":578,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:09.255429+0000","last_change":"2024-04-23T16:43:08.136028+0000","last_active":"2024-04-23T16:44:09.255429+0000","last_peered":"2024-04-23T16:44:09.255429+0000","last_clean":"2024-04-23T16:42:42.962639+0000","last_became_active":"2024-04-23T16:43:08.109240+0000","last_became_peered":"2024-04-23T16:43:08.109240+0000","last_unstale":"2024-04-23T16:44:09.255429+0000","last_undegraded":"2024-04-23T16:43:07.955254+0000","last_fullsized":"2024-04-23T16:44:09.255429+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":8856,"log_dups_size":0,"ondisk_log_size":8856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397672448,"num_objects":6068,"num_object_clones":0,"num_object_copies":12136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3678,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8856,"num_write_kb":477568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1978,"num_bytes_recovered":129630208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":3678},{"shards":"4,11","objects":2390}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1275,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.980602+0000","last_change":"2024-04-23T16:41:49.835121+0000","last_active":"2024-04-23T16:44:04.980602+0000","last_peered":"2024-04-23T16:44:04.980602+0000","last_clean":"2024-04-23T16:44:04.980602+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:44:04.980602+0000","last_undegraded":"2024-04-23T16:44:04.980602+0000","last_fullsized":"2024-04-23T16:44:04.980602+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:41:49.834374+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"578'8698","reported_seq":8856,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:09.429188+0000","last_change":"2024-04-23T16:43:26.589353+0000","last_active":"2024-04-23T16:44:09.429188+0000","last_peered":"2024-04-23T16:44:09.429188+0000","last_clean":"2024-04-23T16:44:09.429188+0000","last_became_active":"2024-04-23T16:41:57.309349+0000","last_became_peered":"2024-04-23T16:41:57.309349+0000","last_unstale":"2024-04-23T16:44:09.429188+0000","last_undegraded":"2024-04-23T16:44:09.429188+0000","last_fullsized":"2024-04-23T16:44:09.429188+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6582","last_scrub_stamp":"2024-04-23T16:43:26.589248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:26.589248+0000","objects_scrubbed":6550,"log_size":8698,"log_dups_size":0,"ondisk_log_size":8698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:27.742741+0000","scrub_duration":1397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389677056,"num_objects":5946,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8698,"num_write_kb":468608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1444,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-23T16:44:04.989793+0000","last_change":"2024-04-23T16:42:33.628340+0000","last_active":"2024-04-23T16:44:04.989793+0000","last_peered":"2024-04-23T16:44:04.989793+0000","last_clean":"2024-04-23T16:44:04.989793+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:44:04.989793+0000","last_undegraded":"2024-04-23T16:44:04.989793+0000","last_fullsized":"2024-04-23T16:44:04.989793+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:33.628217+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:42:33.628217+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6273302552,"num_objects":95724,"num_object_clones":0,"num_object_copies":191448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95724,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141634,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6760,"num_bytes_recovered":443023360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12906602496,"data_stored":12906594352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141634,"ondisk_log_size":141634,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1474560,"data_stored":1460352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056987,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2038748,"kb_used_data":1421716,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":91706404,"statfs":{"total":95995035648,"available":93907357696,"internally_reserved":0,"allocated":1455837184,"data_stored":1455463163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":631830980},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351328,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612966,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2058396,"kb_used_data":1544084,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":91686756,"statfs":{"total":95995035648,"available":93887238144,"internally_reserved":0,"allocated":1581142016,"data_stored":1580767995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":526645696},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841898,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1130532,"kb_used_data":774492,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92614620,"statfs":{"total":95995035648,"available":94837370880,"internally_reserved":0,"allocated":793079808,"data_stored":792702235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168944,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1072284,"kb_used_data":491348,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92672868,"statfs":{"total":95995035648,"available":94897016832,"internally_reserved":0,"allocated":503140352,"data_stored":502766331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":594868682},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463285,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2127264,"kb_used_data":1649048,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":91617888,"statfs":{"total":95995035648,"available":93816717312,"internally_reserved":0,"allocated":1688625152,"data_stored":1688247059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":489683398},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137426,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1010336,"kb_used_data":775576,"kb_used_omap":231,"kb_used_meta":234520,"kb_avail":92734816,"statfs":{"total":95995035648,"available":94960451584,"internally_reserved":0,"allocated":794189824,"data_stored":793811731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":240149392},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738771,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2124636,"kb_used_data":1543700,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":91620516,"statfs":{"total":95995035648,"available":93819408384,"internally_reserved":0,"allocated":1580748800,"data_stored":1580374779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269441,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921149,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2656476,"kb_used_data":1939476,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":91088676,"statfs":{"total":95995035648,"available":93274804224,"internally_reserved":0,"allocated":1986023424,"data_stored":1985649403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[0,1,2,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757625,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1756452,"kb_used_data":1129692,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":91988700,"statfs":{"total":95995035648,"available":94196428800,"internally_reserved":0,"allocated":1156804608,"data_stored":1156427035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1578631168,"data_stored":1578631168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":790892544,"data_stored":790888472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1411842048,"data_stored":1411842048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1969160192,"data_stored":1969160192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1155661824,"data_stored":1155661824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1686638592,"data_stored":1686634520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":501022720,"data_stored":501022720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":790691840,"data_stored":790691840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1577910272,"data_stored":1577910272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1444151296,"data_stored":1444151296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:44:15.032 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.c is not active+clean 2024-04-23T16:44:15.032 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.c', 'version': "578'8845", 'reported_seq': 12562, 'reported_epoch': 578, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:44:09.752900+0000', 'last_change': '2024-04-23T16:43:08.145899+0000', 'last_active': '2024-04-23T16:44:09.752900+0000', 'last_peered': '2024-04-23T16:44:09.752900+0000', 'last_clean': '2024-04-23T16:42:42.970788+0000', 'last_became_active': '2024-04-23T16:43:08.106076+0000', 'last_became_peered': '2024-04-23T16:43:08.106076+0000', 'last_unstale': '2024-04-23T16:44:09.752900+0000', 'last_undegraded': '2024-04-23T16:43:07.951894+0000', 'last_fullsized': '2024-04-23T16:44:09.752900+0000', 'mapping_epoch': 563, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 540, 'last_epoch_clean': 541, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_clean_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'objects_scrubbed': 0, 'log_size': 8845, 'log_dups_size': 0, 'ondisk_log_size': 8845, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 386727936, 'num_objects': 5901, 'num_object_clones': 0, 'num_object_copies': 11802, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3901, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5901, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8845, 'num_write_kb': 471872, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1813, 'num_bytes_recovered': 118816768, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 11], 'acting': [6, 11], 'avail_no_missing': ['6'], 'object_location_counts': [{'shards': '6', 'objects': 3901}, {'shards': '6,11', 'objects': 2000}], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-23T16:44:15.032 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.1 is not active+clean 2024-04-23T16:44:15.032 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.1', 'version': "578'8670", 'reported_seq': 11564, 'reported_epoch': 578, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:44:09.276296+0000', 'last_change': '2024-04-23T16:43:08.081567+0000', 'last_active': '2024-04-23T16:44:09.276296+0000', 'last_peered': '2024-04-23T16:44:09.276296+0000', 'last_clean': '2024-04-23T16:42:42.976600+0000', 'last_became_active': '2024-04-23T16:43:08.067887+0000', 'last_became_peered': '2024-04-23T16:43:08.067887+0000', 'last_unstale': '2024-04-23T16:44:09.276296+0000', 'last_undegraded': '2024-04-23T16:43:07.950381+0000', 'last_fullsized': '2024-04-23T16:44:09.276296+0000', 'mapping_epoch': 563, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 540, 'last_epoch_clean': 541, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_clean_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'objects_scrubbed': 0, 'log_size': 8670, 'log_dups_size': 0, 'ondisk_log_size': 8670, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 390987776, 'num_objects': 5966, 'num_object_clones': 0, 'num_object_copies': 11932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5966, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8670, 'num_write_kb': 468352, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1402, 'num_bytes_recovered': 91881472, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': ['4'], 'object_location_counts': [{'shards': '4', 'objects': 4186}, {'shards': '4,6', 'objects': 1780}], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-23T16:44:15.032 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.2 is not active+clean 2024-04-23T16:44:15.033 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.2', 'version': "578'8815", 'reported_seq': 12037, 'reported_epoch': 578, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:44:09.649757+0000', 'last_change': '2024-04-23T16:43:08.363306+0000', 'last_active': '2024-04-23T16:44:09.649757+0000', 'last_peered': '2024-04-23T16:44:09.649757+0000', 'last_clean': '2024-04-23T16:42:42.950766+0000', 'last_became_active': '2024-04-23T16:43:08.333793+0000', 'last_became_peered': '2024-04-23T16:43:08.333793+0000', 'last_unstale': '2024-04-23T16:44:09.649757+0000', 'last_undegraded': '2024-04-23T16:43:07.950356+0000', 'last_fullsized': '2024-04-23T16:44:09.649757+0000', 'mapping_epoch': 563, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 540, 'last_epoch_clean': 541, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_clean_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'objects_scrubbed': 0, 'log_size': 8815, 'log_dups_size': 0, 'ondisk_log_size': 8815, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 393412608, 'num_objects': 6003, 'num_object_clones': 0, 'num_object_copies': 12006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4170, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6003, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8815, 'num_write_kb': 474176, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1567, 'num_bytes_recovered': 102694912, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': ['6'], 'object_location_counts': [{'shards': '6', 'objects': 4170}, {'shards': '6,7', 'objects': 1833}], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-23T16:44:15.033 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.3 is not active+clean 2024-04-23T16:44:15.033 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.3', 'version': "578'8856", 'reported_seq': 12903, 'reported_epoch': 578, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:44:09.255429+0000', 'last_change': '2024-04-23T16:43:08.136028+0000', 'last_active': '2024-04-23T16:44:09.255429+0000', 'last_peered': '2024-04-23T16:44:09.255429+0000', 'last_clean': '2024-04-23T16:42:42.962639+0000', 'last_became_active': '2024-04-23T16:43:08.109240+0000', 'last_became_peered': '2024-04-23T16:43:08.109240+0000', 'last_unstale': '2024-04-23T16:44:09.255429+0000', 'last_undegraded': '2024-04-23T16:43:07.955254+0000', 'last_fullsized': '2024-04-23T16:44:09.255429+0000', 'mapping_epoch': 563, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 540, 'last_epoch_clean': 541, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_clean_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'objects_scrubbed': 0, 'log_size': 8856, 'log_dups_size': 0, 'ondisk_log_size': 8856, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 397672448, 'num_objects': 6068, 'num_object_clones': 0, 'num_object_copies': 12136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3678, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8856, 'num_write_kb': 477568, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1978, 'num_bytes_recovered': 129630208, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 11], 'acting': [4, 11], 'avail_no_missing': ['4'], 'object_location_counts': [{'shards': '4', 'objects': 3678}, {'shards': '4,11', 'objects': 2390}], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-23T16:44:15.033 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-23T16:44:15.182 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:44:15.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:15.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:44:15.266 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:15 smithi103 ceph-mon[39629]: from='client.16770 v1:172.21.15.103:0/194458308' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:44:15.266 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:15 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 15935/191452 objects degraded (8.323%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:15.295 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:44:15.350 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:15 smithi148 ceph-mon[44520]: from='client.16770 v1:172.21.15.103:0/194458308' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:44:15.350 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:15 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 15935/191452 objects degraded (8.323%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:15.356 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:15.433 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:44:15.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:15 smithi188 ceph-mon[44101]: from='client.16770 v1:172.21.15.103:0/194458308' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:44:15.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:15 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 15935/191452 objects degraded (8.323%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:16.337 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:44:16.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:16 smithi103 ceph-mon[39629]: pgmap v1618: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.7 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 802 op/s; 15253/185850 objects degraded (8.207%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:44:16.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:16 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1876313318' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:44:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:16 smithi148 ceph-mon[44520]: pgmap v1618: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.7 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 802 op/s; 15253/185850 objects degraded (8.207%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:44:16.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:16 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1876313318' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:44:16.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:16 smithi188 ceph-mon[44101]: pgmap v1618: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.7 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 802 op/s; 15253/185850 objects degraded (8.207%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:44:16.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:16 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1876313318' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:44:17.137 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:44:17.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:17 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:44:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:17 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1876313318' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:44:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:17 smithi103 ceph-mon[39629]: osdmap e579: 12 total, 11 up, 9 in 2024-04-23T16:44:17.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:17 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:44:17.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:17 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1876313318' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:44:17.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:17 smithi148 ceph-mon[44520]: osdmap e579: 12 total, 11 up, 9 in 2024-04-23T16:44:17.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:17 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:44:17.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:17 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1876313318' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:44:17.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:17 smithi188 ceph-mon[44101]: osdmap e579: 12 total, 11 up, 9 in 2024-04-23T16:44:17.556 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:44:17.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:17.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:44:17.667 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:44:17.766 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:17.955 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:44:17.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:17.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:44:18.069 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:44:18.098 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:18.126 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:44:18.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:18.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:44:18.231 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:44:18.261 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:18 smithi103 ceph-mon[39629]: pgmap v1620: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.6 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 788 op/s; 15253/184514 objects degraded (8.267%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:44:18.264 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:18.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:18 smithi148 ceph-mon[44520]: pgmap v1620: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.6 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 788 op/s; 15253/184514 objects degraded (8.267%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:44:18.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:18 smithi188 ceph-mon[44101]: pgmap v1620: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.6 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 788 op/s; 15253/184514 objects degraded (8.267%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:44:18.572 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:44:18.572 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:44:18.572 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2024-04-23T16:44:18.572 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2024-04-23T16:44:18.572 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:44:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:19 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:44:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:19 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:44:19.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:19 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:44:19.599 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-23T16:44:19.670 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:44:20.117 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:44:20.120 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:44:20.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:20.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:44:20.224 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:44:20.252 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:20.273 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:44:20.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:20.278 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:44:20.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:20 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:44:20.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:20 smithi103 ceph-mon[39629]: pgmap v1621: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.6 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 789 op/s; 15253/184004 objects degraded (8.289%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:44:20.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:20 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:44:20.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:20 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 15253/184004 objects degraded (8.289%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:20.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:20 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:44:20.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:20 smithi148 ceph-mon[44520]: pgmap v1621: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.6 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 789 op/s; 15253/184004 objects degraded (8.289%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:44:20.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:20 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:44:20.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:20 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 15253/184004 objects degraded (8.289%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:20.379 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:44:20.472 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:20 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:44:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:20 smithi188 ceph-mon[44101]: pgmap v1621: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.6 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 789 op/s; 15253/184004 objects degraded (8.289%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:44:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:20 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:44:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:20 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 15253/184004 objects degraded (8.289%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:20.523 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:44:20.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:20.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:44:20.634 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:44:20.668 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:20.693 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:44:20.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:20.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:44:20.797 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:44:20.848 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:44:20.901 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:21.108 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:44:21.108 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14500/177964 objects degraded (8.148%), 4 pgs degraded","count":4},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2043,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":579,"num_osds":12,"num_up_osds":11,"osd_up_since":1713890562,"num_in_osds":9,"osd_in_since":1713890586,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":23},{"state_name":"active+recovering+degraded","count":4}],"num_pgs":27,"num_pools":2,"num_objects":88982,"data_bytes":5831918136,"bytes_used":15625531392,"bytes_avail":848329789440,"bytes_total":863955320832,"degraded_objects":14500,"degraded_total":177964,"degraded_ratio":0.081477152682565016,"recovering_objects_per_sec":143,"recovering_bytes_per_sec":9402591,"recovering_keys_per_sec":0,"num_objects_recovered":1435,"num_bytes_recovered":94044160,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":752},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.103:8443/"}},"servicemap":{"epoch":46,"modified":"2024-04-23T16:44:02.858545+0000","services":{}},"progress_events":{}} 2024-04-23T16:44:21.297 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:44:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:21.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:44:21.304 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:21 smithi148 ceph-mon[44520]: pgmap v1622: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 752 op/s; 14500/177964 objects degraded (8.148%); 9.0 MiB/s, 143 objects/s recovering 2024-04-23T16:44:21.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:21 smithi103 ceph-mon[39629]: pgmap v1622: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 752 op/s; 14500/177964 objects degraded (8.148%); 9.0 MiB/s, 143 objects/s recovering 2024-04-23T16:44:21.404 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:44:21.436 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:21.436 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:21 smithi188 ceph-mon[44101]: pgmap v1622: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 752 op/s; 14500/177964 objects degraded (8.148%); 9.0 MiB/s, 143 objects/s recovering 2024-04-23T16:44:21.461 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:44:21.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:21.465 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:44:21.567 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:44:21.599 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:21.622 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:44:21.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:21.628 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:44:21.729 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:44:21.760 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:22.109 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:44:22.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:22.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:44:22.216 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:44:22.244 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:22.263 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:44:22.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:22.266 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:44:22.367 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:44:22.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:22 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2689720237' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:44:22.438 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:22.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:22 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2689720237' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:44:22.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:22 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2689720237' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:44:23.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:23 smithi103 ceph-mon[39629]: pgmap v1623: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 739 op/s; 14500/176668 objects degraded (8.207%); 9.0 MiB/s, 143 objects/s recovering 2024-04-23T16:44:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:23 smithi188 ceph-mon[44101]: pgmap v1623: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 739 op/s; 14500/176668 objects degraded (8.207%); 9.0 MiB/s, 143 objects/s recovering 2024-04-23T16:44:23.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:23 smithi148 ceph-mon[44520]: pgmap v1623: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 739 op/s; 14500/176668 objects degraded (8.207%); 9.0 MiB/s, 143 objects/s recovering 2024-04-23T16:44:23.816 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-23T16:44:23.816 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:44:23.953 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:44:23.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:23.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:44:24.080 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:44:24.109 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:24.232 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:44:24.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:24.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:44:24.337 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:44:24.367 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:24.381 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:44:24.394 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:44:24.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:24.403 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:44:24.504 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:44:24.534 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:24.561 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:44:24.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:24.565 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:44:24.633 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:24 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 14500/176668 objects degraded (8.207%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:24.665 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:44:24.673 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:24 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 14500/176668 objects degraded (8.207%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:24.698 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:24.720 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:44:24.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:24 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 14500/176668 objects degraded (8.207%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:24.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:24.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:44:24.825 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:44:24.901 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:24.952 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:44:24.977 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:44:25.000 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:44:25.055 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:44:25.089 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:44:25.109 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:44:25.139 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:44:25.165 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:44:25.187 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:44:25.212 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:44:25.235 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:44:25.235 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:44:25.235 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:44:25.236 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:44:25.236 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:44:25.236 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:44:25.236 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:44:25.236 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:44:25.236 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:44:25.236 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:44:25.236 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:44:25.236 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:44:25.236 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:44:25.410 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:44:25.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:25 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2955039146' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:44:25.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:25 smithi103 ceph-mon[39629]: pgmap v1624: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.3 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 633 op/s; 14110/173176 objects degraded (8.148%); 7.1 MiB/s, 114 objects/s recovering 2024-04-23T16:44:25.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:25 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3862907826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:44:25.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:25.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:25 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2955039146' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:44:25.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:25 smithi188 ceph-mon[44101]: pgmap v1624: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.3 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 633 op/s; 14110/173176 objects degraded (8.148%); 7.1 MiB/s, 114 objects/s recovering 2024-04-23T16:44:25.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:25 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3862907826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:44:25.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:25.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:25 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2955039146' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:44:25.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:25 smithi148 ceph-mon[44520]: pgmap v1624: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.3 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 633 op/s; 14110/173176 objects degraded (8.148%); 7.1 MiB/s, 114 objects/s recovering 2024-04-23T16:44:25.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:25 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3862907826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:44:25.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.190 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:44:26.191 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:44:26.511 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:44:26.514 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.514 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.514 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.514 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.514 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.514 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.515 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.515 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.515 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.515 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.515 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:26 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:44:26.515 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:26 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2955039146' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:44:26.515 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:26 smithi148 ceph-mon[44520]: osdmap e580: 12 total, 11 up, 9 in 2024-04-23T16:44:26.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:26.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:44:26.617 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:44:26.695 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:26.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:26 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:44:26.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:26 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2955039146' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:44:26.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:26 smithi188 ceph-mon[44101]: osdmap e580: 12 total, 11 up, 9 in 2024-04-23T16:44:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:26.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:26 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:44:26.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:26 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2955039146' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:44:26.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:26 smithi103 ceph-mon[39629]: osdmap e580: 12 total, 11 up, 9 in 2024-04-23T16:44:26.892 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:44:26.893 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:44:26.998 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:44:27.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:27.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:44:27.058 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-23T16:44:27.126 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:44:27.198 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:27.302 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:44:27.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:27.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:44:27.375 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:44:27.416 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:44:27.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:27 smithi188 ceph-mon[44101]: pgmap v1626: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 732 op/s; 13705/169870 objects degraded (8.068%); 9.7 MiB/s, 154 objects/s recovering 2024-04-23T16:44:27.445 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:27.569 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:44:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:27.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:44:27.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:27 smithi148 ceph-mon[44520]: pgmap v1626: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 732 op/s; 13705/169870 objects degraded (8.068%); 9.7 MiB/s, 154 objects/s recovering 2024-04-23T16:44:27.672 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:44:27.732 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:27 smithi103 ceph-mon[39629]: pgmap v1626: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 732 op/s; 13705/169870 objects degraded (8.068%); 9.7 MiB/s, 154 objects/s recovering 2024-04-23T16:44:27.760 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:27.810 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:44:27.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:27.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:44:27.919 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:44:27.951 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:27.974 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:44:27.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:27.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:44:28.080 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:44:28.111 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:28.319 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:44:28.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:28.324 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:44:28.425 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:44:28.458 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:28.709 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:44:28.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:28.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:44:28.813 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:44:28.844 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:28.869 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:44:28.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:28.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:44:28.894 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:44:28.973 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:44:28.985 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:44:29.006 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:44:29.007 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:29.031 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:44:29.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:29.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:44:29.135 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:44:29.222 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:29.317 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:44:29.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:29.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:44:29.424 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:44:29.486 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:29.539 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:44:29.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:29.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:44:29.584 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:44:29.624 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:44:29.624 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:44:29.646 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:44:29.686 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:29.804 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:44:29.804 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:44:29.804 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:29 smithi148 ceph-mon[44520]: pgmap v1627: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 730 op/s; 13705/169388 objects degraded (8.091%); 9.7 MiB/s, 154 objects/s recovering 2024-04-23T16:44:29.804 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:29 smithi148 ceph-mon[44520]: from='client.26746 v1:172.21.15.103:0/3850150092' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:44:29.804 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:29 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 13705/169870 objects degraded (8.068%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:29.895 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:44:29.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:29.901 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:44:29.934 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1627,"stamp":"2024-04-23T16:44:28.863593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5550899768,"num_objects":84694,"num_object_clones":0,"num_object_copies":169388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13705,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84694,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152666,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8992,"num_bytes_recovered":589571136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154195,"ondisk_log_size":154195,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":14715564,"kb_used_data":9969244,"kb_used_omap":243,"kb_used_meta":4745932,"kb_avail":828990804,"statfs":{"total":863955320832,"available":848886583296,"internally_reserved":0,"allocated":10208505856,"data_stored":10205051771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":249416,"internal_metadata":4859834808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,4,3,5,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-478937088,"num_objects":-7308,"num_object_clones":0,"num_object_copies":-14616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7308,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1548,"num_bytes_recovered":101449728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002019"},"pg_stats":[{"pgid":"7.d","version":"579'9666","reported_seq":9755,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-04-23T16:44:24.916765+0000","last_change":"2024-04-23T16:41:57.310289+0000","last_active":"2024-04-23T16:44:24.916765+0000","last_peered":"2024-04-23T16:44:24.916765+0000","last_clean":"2024-04-23T16:44:24.916765+0000","last_became_active":"2024-04-23T16:41:57.310025+0000","last_became_peered":"2024-04-23T16:41:57.310025+0000","last_unstale":"2024-04-23T16:44:24.916765+0000","last_undegraded":"2024-04-23T16:44:24.916765+0000","last_fullsized":"2024-04-23T16:44:24.916765+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":9666,"log_dups_size":0,"ondisk_log_size":9666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354091032,"num_objects":5404,"num_object_clones":0,"num_object_copies":10808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5404,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9666,"num_write_kb":482177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"580'9857","reported_seq":10024,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-04-23T16:44:25.840852+0000","last_change":"2024-04-23T16:43:29.012370+0000","last_active":"2024-04-23T16:44:25.840852+0000","last_peered":"2024-04-23T16:44:25.840852+0000","last_clean":"2024-04-23T16:44:25.840852+0000","last_became_active":"2024-04-23T16:41:57.310754+0000","last_became_peered":"2024-04-23T16:41:57.310754+0000","last_unstale":"2024-04-23T16:44:25.840852+0000","last_undegraded":"2024-04-23T16:44:25.840852+0000","last_fullsized":"2024-04-23T16:44:25.840852+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6945","last_scrub_stamp":"2024-04-23T16:43:29.012285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:29.012285+0000","objects_scrubbed":6912,"log_size":9857,"log_dups_size":0,"ondisk_log_size":9857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:58.968073+0000","scrub_duration":1082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346488832,"num_objects":5287,"num_object_clones":0,"num_object_copies":10574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9857,"num_write_kb":484608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"579'9480","reported_seq":9642,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-04-23T16:44:24.398509+0000","last_change":"2024-04-23T16:43:25.997269+0000","last_active":"2024-04-23T16:44:24.398509+0000","last_peered":"2024-04-23T16:44:24.398509+0000","last_clean":"2024-04-23T16:44:24.398509+0000","last_became_active":"2024-04-23T16:41:57.310189+0000","last_became_peered":"2024-04-23T16:41:57.310189+0000","last_unstale":"2024-04-23T16:44:24.398509+0000","last_undegraded":"2024-04-23T16:44:24.398509+0000","last_fullsized":"2024-04-23T16:44:24.398509+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'6616","last_scrub_stamp":"2024-04-23T16:43:25.997150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:25.997150+0000","objects_scrubbed":6600,"log_size":9480,"log_dups_size":0,"ondisk_log_size":9480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:37.694305+0000","scrub_duration":806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355336192,"num_objects":5422,"num_object_clones":0,"num_object_copies":10844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9480,"num_write_kb":476864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"579'9574","reported_seq":9735,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-04-23T16:44:24.440358+0000","last_change":"2024-04-23T16:43:26.892913+0000","last_active":"2024-04-23T16:44:24.440358+0000","last_peered":"2024-04-23T16:44:24.440358+0000","last_clean":"2024-04-23T16:44:24.440358+0000","last_became_active":"2024-04-23T16:41:57.309569+0000","last_became_peered":"2024-04-23T16:41:57.309569+0000","last_unstale":"2024-04-23T16:44:24.440358+0000","last_undegraded":"2024-04-23T16:44:24.440358+0000","last_fullsized":"2024-04-23T16:44:24.440358+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6695","last_scrub_stamp":"2024-04-23T16:43:26.892785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:26.892785+0000","objects_scrubbed":6673,"log_size":9574,"log_dups_size":0,"ondisk_log_size":9574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:56.021559+0000","scrub_duration":739,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348520448,"num_objects":5318,"num_object_clones":0,"num_object_copies":10636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9574,"num_write_kb":476544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"580'9368","reported_seq":9457,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-04-23T16:44:25.715097+0000","last_change":"2024-04-23T16:41:57.310144+0000","last_active":"2024-04-23T16:44:25.715097+0000","last_peered":"2024-04-23T16:44:25.715097+0000","last_clean":"2024-04-23T16:44:25.715097+0000","last_became_active":"2024-04-23T16:41:57.309841+0000","last_became_peered":"2024-04-23T16:41:57.309841+0000","last_unstale":"2024-04-23T16:44:25.715097+0000","last_undegraded":"2024-04-23T16:44:25.715097+0000","last_fullsized":"2024-04-23T16:44:25.715097+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":9368,"log_dups_size":0,"ondisk_log_size":9368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338952192,"num_objects":5172,"num_object_clones":0,"num_object_copies":10344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9368,"num_write_kb":465280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"579'9550","reported_seq":14320,"reported_epoch":579,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:24.933454+0000","last_change":"2024-04-23T16:43:08.145899+0000","last_active":"2024-04-23T16:44:24.933454+0000","last_peered":"2024-04-23T16:44:24.933454+0000","last_clean":"2024-04-23T16:42:42.970788+0000","last_became_active":"2024-04-23T16:43:08.106076+0000","last_became_peered":"2024-04-23T16:43:08.106076+0000","last_unstale":"2024-04-23T16:44:24.933454+0000","last_undegraded":"2024-04-23T16:43:07.951894+0000","last_fullsized":"2024-04-23T16:44:24.933454+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":9550,"log_dups_size":0,"ondisk_log_size":9550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340525056,"num_objects":5196,"num_object_clones":0,"num_object_copies":10392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9550,"num_write_kb":471872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":153223168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":3376},{"shards":"6,11","objects":1820}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1471,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-04-23T16:44:16.336782+0000","last_change":"2024-04-23T16:44:09.435333+0000","last_active":"2024-04-23T16:44:16.336782+0000","last_peered":"2024-04-23T16:44:16.336782+0000","last_clean":"2024-04-23T16:44:16.336782+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:44:16.336782+0000","last_undegraded":"2024-04-23T16:44:16.336782+0000","last_fullsized":"2024-04-23T16:44:16.336782+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:09.435212+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:44:09.435212+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:17.709965+0000","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"579'9534","reported_seq":9697,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-04-23T16:44:24.457143+0000","last_change":"2024-04-23T16:43:28.303396+0000","last_active":"2024-04-23T16:44:24.457143+0000","last_peered":"2024-04-23T16:44:24.457143+0000","last_clean":"2024-04-23T16:44:24.457143+0000","last_became_active":"2024-04-23T16:41:57.311753+0000","last_became_peered":"2024-04-23T16:41:57.311753+0000","last_unstale":"2024-04-23T16:44:24.457143+0000","last_undegraded":"2024-04-23T16:44:24.457143+0000","last_fullsized":"2024-04-23T16:44:24.457143+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6763","last_scrub_stamp":"2024-04-23T16:43:28.303250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:28.303250+0000","objects_scrubbed":6733,"log_size":9534,"log_dups_size":0,"ondisk_log_size":9534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:51.576731+0000","scrub_duration":1201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353370112,"num_objects":5392,"num_object_clones":0,"num_object_copies":10784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9534,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1471,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-04-23T16:44:25.436564+0000","last_change":"2024-04-23T16:43:27.668916+0000","last_active":"2024-04-23T16:44:25.436564+0000","last_peered":"2024-04-23T16:44:25.436564+0000","last_clean":"2024-04-23T16:44:25.436564+0000","last_became_active":"2024-04-23T16:43:07.962092+0000","last_became_peered":"2024-04-23T16:43:07.962092+0000","last_unstale":"2024-04-23T16:44:25.436564+0000","last_undegraded":"2024-04-23T16:44:25.436564+0000","last_fullsized":"2024-04-23T16:44:25.436564+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:43:27.668793+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:43:27.668793+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:45.595521+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"579'9483","reported_seq":9645,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-04-23T16:44:24.418560+0000","last_change":"2024-04-23T16:43:27.928311+0000","last_active":"2024-04-23T16:44:24.418560+0000","last_peered":"2024-04-23T16:44:24.418560+0000","last_clean":"2024-04-23T16:44:24.418560+0000","last_became_active":"2024-04-23T16:41:57.305944+0000","last_became_peered":"2024-04-23T16:41:57.305944+0000","last_unstale":"2024-04-23T16:44:24.418560+0000","last_undegraded":"2024-04-23T16:44:24.418560+0000","last_fullsized":"2024-04-23T16:44:24.418560+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6718","last_scrub_stamp":"2024-04-23T16:43:27.928174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:27.928174+0000","objects_scrubbed":6703,"log_size":9483,"log_dups_size":0,"ondisk_log_size":9483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:34.856929+0000","scrub_duration":750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351600640,"num_objects":5365,"num_object_clones":0,"num_object_copies":10730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9483,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1448,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-04-23T16:44:25.481832+0000","last_change":"2024-04-23T16:42:32.662620+0000","last_active":"2024-04-23T16:44:25.481832+0000","last_peered":"2024-04-23T16:44:25.481832+0000","last_clean":"2024-04-23T16:44:25.481832+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:44:25.481832+0000","last_undegraded":"2024-04-23T16:44:25.481832+0000","last_fullsized":"2024-04-23T16:44:25.481832+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:32.662476+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:42:32.662476+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"579'9752","reported_seq":9914,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-04-23T16:44:24.452031+0000","last_change":"2024-04-23T16:43:25.389006+0000","last_active":"2024-04-23T16:44:24.452031+0000","last_peered":"2024-04-23T16:44:24.452031+0000","last_clean":"2024-04-23T16:44:24.452031+0000","last_became_active":"2024-04-23T16:41:57.309858+0000","last_became_peered":"2024-04-23T16:41:57.309858+0000","last_unstale":"2024-04-23T16:44:24.452031+0000","last_undegraded":"2024-04-23T16:44:24.452031+0000","last_fullsized":"2024-04-23T16:44:24.452031+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'6776","last_scrub_stamp":"2024-04-23T16:43:25.388898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:25.388898+0000","objects_scrubbed":6743,"log_size":9752,"log_dups_size":0,"ondisk_log_size":9752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:39.155605+0000","scrub_duration":1233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354811904,"num_objects":5414,"num_object_clones":0,"num_object_copies":10828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9752,"num_write_kb":485312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1470,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-04-23T16:44:16.364902+0000","last_change":"2024-04-23T16:43:40.258384+0000","last_active":"2024-04-23T16:44:16.364902+0000","last_peered":"2024-04-23T16:44:16.364902+0000","last_clean":"2024-04-23T16:44:16.364902+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:44:16.364902+0000","last_undegraded":"2024-04-23T16:44:16.364902+0000","last_fullsized":"2024-04-23T16:44:16.364902+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:43:40.258213+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:43:40.258213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:07.779820+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"580'9545","reported_seq":9635,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-04-23T16:44:26.524890+0000","last_change":"2024-04-23T16:41:57.310695+0000","last_active":"2024-04-23T16:44:26.524890+0000","last_peered":"2024-04-23T16:44:26.524890+0000","last_clean":"2024-04-23T16:44:26.524890+0000","last_became_active":"2024-04-23T16:41:57.310400+0000","last_became_peered":"2024-04-23T16:41:57.310400+0000","last_unstale":"2024-04-23T16:44:26.524890+0000","last_undegraded":"2024-04-23T16:44:26.524890+0000","last_fullsized":"2024-04-23T16:44:26.524890+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":9545,"log_dups_size":0,"ondisk_log_size":9545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337707008,"num_objects":5153,"num_object_clones":0,"num_object_copies":10306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9545,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1046,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-04-23T16:44:25.406358+0000","last_change":"2024-04-23T16:44:18.555667+0000","last_active":"2024-04-23T16:44:25.406358+0000","last_peered":"2024-04-23T16:44:25.406358+0000","last_clean":"2024-04-23T16:44:25.406358+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:44:25.406358+0000","last_undegraded":"2024-04-23T16:44:25.406358+0000","last_fullsized":"2024-04-23T16:44:25.406358+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:18.555538+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:44:18.555538+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:35.829709+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"579'9469","reported_seq":9556,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-04-23T16:44:24.944793+0000","last_change":"2024-04-23T16:41:57.306526+0000","last_active":"2024-04-23T16:44:24.944793+0000","last_peered":"2024-04-23T16:44:24.944793+0000","last_clean":"2024-04-23T16:44:24.944793+0000","last_became_active":"2024-04-23T16:41:57.306229+0000","last_became_peered":"2024-04-23T16:41:57.306229+0000","last_unstale":"2024-04-23T16:44:24.944793+0000","last_undegraded":"2024-04-23T16:44:24.944793+0000","last_fullsized":"2024-04-23T16:44:24.944793+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":9469,"log_dups_size":0,"ondisk_log_size":9469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343080960,"num_objects":5235,"num_object_clones":0,"num_object_copies":10470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9469,"num_write_kb":470528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1001,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-04-23T16:44:25.415390+0000","last_change":"2024-04-23T16:44:07.230780+0000","last_active":"2024-04-23T16:44:25.415390+0000","last_peered":"2024-04-23T16:44:25.415390+0000","last_clean":"2024-04-23T16:44:25.415390+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:44:25.415390+0000","last_undegraded":"2024-04-23T16:44:25.415390+0000","last_fullsized":"2024-04-23T16:44:25.415390+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:07.230640+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:44:07.230640+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:12.044599+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.0","version":"580'9589","reported_seq":9756,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-04-23T16:44:27.672926+0000","last_change":"2024-04-23T16:43:30.757721+0000","last_active":"2024-04-23T16:44:27.672926+0000","last_peered":"2024-04-23T16:44:27.672926+0000","last_clean":"2024-04-23T16:44:27.672926+0000","last_became_active":"2024-04-23T16:41:57.311479+0000","last_became_peered":"2024-04-23T16:41:57.311479+0000","last_unstale":"2024-04-23T16:44:27.672926+0000","last_undegraded":"2024-04-23T16:44:27.672926+0000","last_fullsized":"2024-04-23T16:44:27.672926+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6823","last_scrub_stamp":"2024-04-23T16:43:30.757555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:30.757555+0000","objects_scrubbed":6802,"log_size":9589,"log_dups_size":0,"ondisk_log_size":9589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:59.054459+0000","scrub_duration":1354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334954496,"num_objects":5111,"num_object_clones":0,"num_object_copies":10222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9589,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1479,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-04-23T16:44:16.338264+0000","last_change":"2024-04-23T16:44:11.349248+0000","last_active":"2024-04-23T16:44:16.338264+0000","last_peered":"2024-04-23T16:44:16.338264+0000","last_clean":"2024-04-23T16:44:16.338264+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:44:16.338264+0000","last_undegraded":"2024-04-23T16:44:16.338264+0000","last_fullsized":"2024-04-23T16:44:16.338264+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:11.348079+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:44:11.348079+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:21.183735+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.1","version":"579'9316","reported_seq":13407,"reported_epoch":579,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:24.328084+0000","last_change":"2024-04-23T16:43:08.081567+0000","last_active":"2024-04-23T16:44:24.328084+0000","last_peered":"2024-04-23T16:44:24.328084+0000","last_clean":"2024-04-23T16:42:42.976600+0000","last_became_active":"2024-04-23T16:43:08.067887+0000","last_became_peered":"2024-04-23T16:43:08.067887+0000","last_unstale":"2024-04-23T16:44:24.328084+0000","last_undegraded":"2024-04-23T16:43:07.950381+0000","last_fullsized":"2024-04-23T16:44:24.328084+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":9316,"log_dups_size":0,"ondisk_log_size":9316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348651520,"num_objects":5320,"num_object_clones":0,"num_object_copies":10640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9316,"num_write_kb":468352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1999,"num_bytes_recovered":131006464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":3589},{"shards":"4,6","objects":1731}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1463,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-04-23T16:44:16.351656+0000","last_change":"2024-04-23T16:43:07.918302+0000","last_active":"2024-04-23T16:44:16.351656+0000","last_peered":"2024-04-23T16:44:16.351656+0000","last_clean":"2024-04-23T16:44:16.351656+0000","last_became_active":"2024-04-23T16:43:07.918042+0000","last_became_peered":"2024-04-23T16:43:07.918042+0000","last_unstale":"2024-04-23T16:44:16.351656+0000","last_undegraded":"2024-04-23T16:44:16.351656+0000","last_fullsized":"2024-04-23T16:44:16.351656+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:33.716343+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:42:33.716343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"579'9512","reported_seq":13824,"reported_epoch":579,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:24.936704+0000","last_change":"2024-04-23T16:43:08.363306+0000","last_active":"2024-04-23T16:44:24.936704+0000","last_peered":"2024-04-23T16:44:24.936704+0000","last_clean":"2024-04-23T16:42:42.950766+0000","last_became_active":"2024-04-23T16:43:08.333793+0000","last_became_peered":"2024-04-23T16:43:08.333793+0000","last_unstale":"2024-04-23T16:44:24.936704+0000","last_undegraded":"2024-04-23T16:43:07.950356+0000","last_fullsized":"2024-04-23T16:44:24.936704+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":9512,"log_dups_size":0,"ondisk_log_size":9512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347734016,"num_objects":5306,"num_object_clones":0,"num_object_copies":10612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9512,"num_write_kb":474176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":138346496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":3626},{"shards":"6,7","objects":1680}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1357,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-04-23T16:44:25.417838+0000","last_change":"2024-04-23T16:41:47.722866+0000","last_active":"2024-04-23T16:44:25.417838+0000","last_peered":"2024-04-23T16:44:25.417838+0000","last_clean":"2024-04-23T16:44:25.417838+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:44:25.417838+0000","last_undegraded":"2024-04-23T16:44:25.417838+0000","last_fullsized":"2024-04-23T16:44:25.417838+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:41:47.722762+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"579'9560","reported_seq":14737,"reported_epoch":579,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:24.313385+0000","last_change":"2024-04-23T16:43:08.136028+0000","last_active":"2024-04-23T16:44:24.313385+0000","last_peered":"2024-04-23T16:44:24.313385+0000","last_clean":"2024-04-23T16:42:42.962639+0000","last_became_active":"2024-04-23T16:43:08.109240+0000","last_became_peered":"2024-04-23T16:43:08.109240+0000","last_unstale":"2024-04-23T16:44:24.313385+0000","last_undegraded":"2024-04-23T16:43:07.955254+0000","last_fullsized":"2024-04-23T16:44:24.313385+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":9560,"log_dups_size":0,"ondisk_log_size":9560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351535104,"num_objects":5364,"num_object_clones":0,"num_object_copies":10728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9560,"num_write_kb":477568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2542,"num_bytes_recovered":166592512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":3114},{"shards":"4,11","objects":2250}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1277,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-04-23T16:44:16.351794+0000","last_change":"2024-04-23T16:41:49.835121+0000","last_active":"2024-04-23T16:44:16.351794+0000","last_peered":"2024-04-23T16:44:16.351794+0000","last_clean":"2024-04-23T16:44:16.351794+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:44:16.351794+0000","last_undegraded":"2024-04-23T16:44:16.351794+0000","last_fullsized":"2024-04-23T16:44:16.351794+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:41:49.834374+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"579'9411","reported_seq":9572,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-04-23T16:44:24.434747+0000","last_change":"2024-04-23T16:43:26.589353+0000","last_active":"2024-04-23T16:44:24.434747+0000","last_peered":"2024-04-23T16:44:24.434747+0000","last_clean":"2024-04-23T16:44:24.434747+0000","last_became_active":"2024-04-23T16:41:57.309349+0000","last_became_peered":"2024-04-23T16:41:57.309349+0000","last_unstale":"2024-04-23T16:44:24.434747+0000","last_undegraded":"2024-04-23T16:44:24.434747+0000","last_fullsized":"2024-04-23T16:44:24.434747+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6582","last_scrub_stamp":"2024-04-23T16:43:26.589248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:26.589248+0000","objects_scrubbed":6550,"log_size":9411,"log_dups_size":0,"ondisk_log_size":9411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:27.742741+0000","scrub_duration":1397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342949888,"num_objects":5233,"num_object_clones":0,"num_object_copies":10466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9411,"num_write_kb":468608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1448,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-04-23T16:44:25.486202+0000","last_change":"2024-04-23T16:42:33.628340+0000","last_active":"2024-04-23T16:44:25.486202+0000","last_peered":"2024-04-23T16:44:25.486202+0000","last_clean":"2024-04-23T16:44:25.486202+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:44:25.486202+0000","last_undegraded":"2024-04-23T16:44:25.486202+0000","last_fullsized":"2024-04-23T16:44:25.486202+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:33.628217+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:42:33.628217+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5550309400,"num_objects":84692,"num_object_clones":0,"num_object_copies":169384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13705,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84692,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152666,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8990,"num_bytes_recovered":589168640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11584348160,"data_stored":11584340016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152666,"ondisk_log_size":152666,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1474560,"data_stored":1460352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056990,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1878584,"kb_used_data":1274028,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":91866568,"statfs":{"total":95995035648,"available":94071365632,"internally_reserved":0,"allocated":1304604672,"data_stored":1304218543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":619051460},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351331,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612969,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1896424,"kb_used_data":1369632,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":91848728,"statfs":{"total":95995035648,"available":94053097472,"internally_reserved":0,"allocated":1402503168,"data_stored":1402123093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":539425216},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841902,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1043584,"kb_used_data":661172,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92701568,"statfs":{"total":95995035648,"available":94926405632,"internally_reserved":0,"allocated":677040128,"data_stored":676650447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168947,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1013864,"kb_used_data":441248,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92731288,"statfs":{"total":95995035648,"available":94956838912,"internally_reserved":0,"allocated":451837952,"data_stored":451457877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":586349002},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463288,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1977580,"kb_used_data":1468772,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":91767572,"statfs":{"total":95995035648,"available":93969993728,"internally_reserved":0,"allocated":1504022528,"data_stored":1503638381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":520944070},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137429,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912428,"kb_used_data":685988,"kb_used_omap":231,"kb_used_meta":226200,"kb_avail":92832724,"statfs":{"total":95995035648,"available":95060709376,"internally_reserved":0,"allocated":702451712,"data_stored":702067565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":231629712},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738774,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1944360,"kb_used_data":1367584,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":91800792,"statfs":{"total":95995035648,"available":94004011008,"internally_reserved":0,"allocated":1400406016,"data_stored":1400025941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269445,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921152,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2414568,"kb_used_data":1701728,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":91330584,"statfs":{"total":95995035648,"available":93522518016,"internally_reserved":0,"allocated":1742569472,"data_stored":1742189397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,2,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757628,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1634172,"kb_used_data":999092,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92110980,"statfs":{"total":95995035648,"available":94321643520,"internally_reserved":0,"allocated":1023070208,"data_stored":1022680527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1397817344,"data_stored":1397817344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":697372672,"data_stored":697368600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1411842048,"data_stored":1411842048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1741094912,"data_stored":1741094912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1021509632,"data_stored":1021509632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1498877952,"data_stored":1498873880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":449904640,"data_stored":449904640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":672595968,"data_stored":672595968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1397882880,"data_stored":1397882880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1295450112,"data_stored":1295450112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:44:29.964 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:44:29.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:44:29.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:44:29.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:29 smithi188 ceph-mon[44101]: pgmap v1627: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 730 op/s; 13705/169388 objects degraded (8.091%); 9.7 MiB/s, 154 objects/s recovering 2024-04-23T16:44:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:29 smithi188 ceph-mon[44101]: from='client.26746 v1:172.21.15.103:0/3850150092' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:44:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:29 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 13705/169870 objects degraded (8.068%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:30.002 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:44:30.088 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:30.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:44:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:44:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:29 smithi103 ceph-mon[39629]: pgmap v1627: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 730 op/s; 13705/169388 objects degraded (8.091%); 9.7 MiB/s, 154 objects/s recovering 2024-04-23T16:44:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:29 smithi103 ceph-mon[39629]: from='client.26746 v1:172.21.15.103:0/3850150092' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:44:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:29 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 13705/169870 objects degraded (8.068%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:44:30.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:30.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:44:30.219 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:44:30.250 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:30.400 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:44:30.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:30.407 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:44:30.421 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-23T16:44:30.508 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:44:30.536 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:30.590 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:44:30.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:30.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:44:30.693 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:44:30.782 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:30.817 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:44:30.835 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:44:30.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:30.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:44:30.946 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:44:30.953 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:44:30.976 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:31.827 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:44:31.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:31.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:44:31.931 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:44:31.962 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:31 smithi103 ceph-mon[39629]: pgmap v1628: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.0 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 716 op/s; 12875/163630 objects degraded (7.868%); 10 MiB/s, 162 objects/s recovering 2024-04-23T16:44:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:31 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:44:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:31 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:44:32.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:31 smithi188 ceph-mon[44101]: pgmap v1628: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.0 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 716 op/s; 12875/163630 objects degraded (7.868%); 10 MiB/s, 162 objects/s recovering 2024-04-23T16:44:32.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:31 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:44:32.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:31 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:44:32.149 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:44:32.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:32.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:44:32.256 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:44:32.291 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:31 smithi148 ceph-mon[44520]: pgmap v1628: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.0 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 716 op/s; 12875/163630 objects degraded (7.868%); 10 MiB/s, 162 objects/s recovering 2024-04-23T16:44:32.291 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:31 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:44:32.291 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:31 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:44:32.298 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:32.775 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:44:32.775 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:44:32.938 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:44:33.203 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:32 smithi188 ceph-mon[44101]: osdmap e581: 12 total, 11 up, 9 in 2024-04-23T16:44:33.271 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:32 smithi103 ceph-mon[39629]: osdmap e581: 12 total, 11 up, 9 in 2024-04-23T16:44:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:32 smithi148 ceph-mon[44520]: osdmap e581: 12 total, 11 up, 9 in 2024-04-23T16:44:33.570 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:44:33.577 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-23T16:44:33.903 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:44:33.968 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:44:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:33 smithi103 ceph-mon[39629]: pgmap v1630: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.0 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 680 op/s; 12875/162278 objects degraded (7.934%); 9.6 MiB/s, 154 objects/s recovering 2024-04-23T16:44:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:33 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4238712899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:44:34.135 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:44:34.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:34.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:44:34.244 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:44:34.335 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:33 smithi148 ceph-mon[44520]: pgmap v1630: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.0 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 680 op/s; 12875/162278 objects degraded (7.934%); 9.6 MiB/s, 154 objects/s recovering 2024-04-23T16:44:34.335 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:33 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4238712899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:44:34.351 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:34.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:33 smithi188 ceph-mon[44101]: pgmap v1630: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 5.0 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 680 op/s; 12875/162278 objects degraded (7.934%); 9.6 MiB/s, 154 objects/s recovering 2024-04-23T16:44:34.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:33 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4238712899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:44:35.207 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:34 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:44:35.207 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:34 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4238712899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:44:35.207 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:34 smithi188 ceph-mon[44101]: osdmap e582: 12 total, 11 up, 9 in 2024-04-23T16:44:35.207 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:34 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 12875/162278 objects degraded (7.934%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:35.339 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:34 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:44:35.339 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:34 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4238712899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:44:35.339 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:34 smithi148 ceph-mon[44520]: osdmap e582: 12 total, 11 up, 9 in 2024-04-23T16:44:35.340 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:34 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 12875/162278 objects degraded (7.934%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:35.347 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:34 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:44:35.347 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:34 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4238712899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:44:35.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:34 smithi103 ceph-mon[39629]: osdmap e582: 12 total, 11 up, 9 in 2024-04-23T16:44:35.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:34 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 12875/162278 objects degraded (7.934%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:35.837 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:44:35.837 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:44:35.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:35 smithi103 ceph-mon[39629]: pgmap v1632: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.8 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 699 op/s; 12427/158670 objects degraded (7.832%); 10 MiB/s, 159 objects/s recovering 2024-04-23T16:44:35.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:35 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/55709886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:44:35.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:35.992 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:36.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:35 smithi188 ceph-mon[44101]: pgmap v1632: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.8 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 699 op/s; 12427/158670 objects degraded (7.832%); 10 MiB/s, 159 objects/s recovering 2024-04-23T16:44:36.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:35 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/55709886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:44:36.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:36.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:36.289 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:44:36.289 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:44:36.289 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:44:36.289 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:44:36.289 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:44:36.289 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:44:36.289 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:44:36.289 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:44:36.289 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:44:36.290 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:44:36.290 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:44:36.290 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:44:36.290 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:44:36.290 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:44:36.290 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:44:36.290 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:44:36.290 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:44:36.290 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:44:36.290 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:44:36.290 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:44:36.290 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:44:36.291 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:44:36.291 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:44:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:35 smithi148 ceph-mon[44520]: pgmap v1632: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.8 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 699 op/s; 12427/158670 objects degraded (7.832%); 10 MiB/s, 159 objects/s recovering 2024-04-23T16:44:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:35 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/55709886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:44:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:36.614 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-23T16:44:36.957 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:44:36.979 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:44:36.980 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:44:37.273 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.273 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.273 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.273 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.274 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.274 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.274 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.274 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.274 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.307 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.307 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.307 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.373 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:44:37.383 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.383 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.383 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.383 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.383 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.383 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:37.691 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:44:37.843 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:44:37.844 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:44:38.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:38 smithi188 ceph-mon[44101]: pgmap v1633: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.8 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 858 op/s; 11989/155652 objects degraded (7.702%); 13 MiB/s, 214 objects/s recovering 2024-04-23T16:44:38.299 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:37 smithi103 ceph-mon[39629]: pgmap v1633: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.8 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 858 op/s; 11989/155652 objects degraded (7.702%); 13 MiB/s, 214 objects/s recovering 2024-04-23T16:44:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:38 smithi148 ceph-mon[44520]: pgmap v1633: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.8 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 858 op/s; 11989/155652 objects degraded (7.702%); 13 MiB/s, 214 objects/s recovering 2024-04-23T16:44:38.872 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:44:38.872 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:44:38.873 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-04-23T16:44:38.873 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:44:38.981 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:44:39.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:39 smithi103 ceph-mon[39629]: from='client.26761 v1:172.21.15.103:0/2713135151' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:44:39.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:39 smithi188 ceph-mon[44101]: from='client.26761 v1:172.21.15.103:0/2713135151' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:44:39.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:39 smithi148 ceph-mon[44520]: from='client.26761 v1:172.21.15.103:0/2713135151' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:44:39.720 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:44:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:39.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:44:39.781 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:44:39.782 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1633,"stamp":"2024-04-23T16:44:36.865338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5100798520,"num_objects":77826,"num_object_clones":0,"num_object_copies":155652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77826,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159534,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10708,"num_bytes_recovered":702030912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160563,"ondisk_log_size":160563,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":14004812,"kb_used_data":9241508,"kb_used_omap":243,"kb_used_meta":4762572,"kb_avail":829701556,"statfs":{"total":863955320832,"available":849614393344,"internally_reserved":0,"allocated":9463304192,"data_stored":9459717369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":249416,"internal_metadata":4876874168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-450101248,"num_objects":-6868,"num_object_clones":0,"num_object_copies":-13736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6868,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1716,"num_bytes_recovered":112459776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001744"},"pg_stats":[{"pgid":"7.d","version":"582'10107","reported_seq":10201,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:35.003432+0000","last_change":"2024-04-23T16:41:57.310289+0000","last_active":"2024-04-23T16:44:35.003432+0000","last_peered":"2024-04-23T16:44:35.003432+0000","last_clean":"2024-04-23T16:44:35.003432+0000","last_became_active":"2024-04-23T16:41:57.310025+0000","last_became_peered":"2024-04-23T16:41:57.310025+0000","last_unstale":"2024-04-23T16:44:35.003432+0000","last_undegraded":"2024-04-23T16:44:35.003432+0000","last_fullsized":"2024-04-23T16:44:35.003432+0000","mapping_epoch":540,"log_start":"547'100","ondisk_log_start":"547'100","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325189656,"num_objects":4963,"num_object_clones":0,"num_object_copies":9926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4963,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10107,"num_write_kb":482177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"582'10292","reported_seq":10463,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:35.879671+0000","last_change":"2024-04-23T16:43:29.012370+0000","last_active":"2024-04-23T16:44:35.879671+0000","last_peered":"2024-04-23T16:44:35.879671+0000","last_clean":"2024-04-23T16:44:35.879671+0000","last_became_active":"2024-04-23T16:41:57.310754+0000","last_became_peered":"2024-04-23T16:41:57.310754+0000","last_unstale":"2024-04-23T16:44:35.879671+0000","last_undegraded":"2024-04-23T16:44:35.879671+0000","last_fullsized":"2024-04-23T16:44:35.879671+0000","mapping_epoch":540,"log_start":"547'200","ondisk_log_start":"547'200","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6945","last_scrub_stamp":"2024-04-23T16:43:29.012285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:29.012285+0000","objects_scrubbed":6912,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:58.968073+0000","scrub_duration":1082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317980672,"num_objects":4852,"num_object_clones":0,"num_object_copies":9704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10292,"num_write_kb":484608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"582'9942","reported_seq":10110,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:34.453798+0000","last_change":"2024-04-23T16:43:25.997269+0000","last_active":"2024-04-23T16:44:34.453798+0000","last_peered":"2024-04-23T16:44:34.453798+0000","last_clean":"2024-04-23T16:44:34.453798+0000","last_became_active":"2024-04-23T16:41:57.310189+0000","last_became_peered":"2024-04-23T16:41:57.310189+0000","last_unstale":"2024-04-23T16:44:34.453798+0000","last_undegraded":"2024-04-23T16:44:34.453798+0000","last_fullsized":"2024-04-23T16:44:34.453798+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'6616","last_scrub_stamp":"2024-04-23T16:43:25.997150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:25.997150+0000","objects_scrubbed":6600,"log_size":9942,"log_dups_size":0,"ondisk_log_size":9942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:37.694305+0000","scrub_duration":806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325058560,"num_objects":4960,"num_object_clones":0,"num_object_copies":9920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9942,"num_write_kb":476864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"582'9996","reported_seq":10163,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:34.451557+0000","last_change":"2024-04-23T16:43:26.892913+0000","last_active":"2024-04-23T16:44:34.451557+0000","last_peered":"2024-04-23T16:44:34.451557+0000","last_clean":"2024-04-23T16:44:34.451557+0000","last_became_active":"2024-04-23T16:41:57.309569+0000","last_became_peered":"2024-04-23T16:41:57.309569+0000","last_unstale":"2024-04-23T16:44:34.451557+0000","last_undegraded":"2024-04-23T16:44:34.451557+0000","last_fullsized":"2024-04-23T16:44:34.451557+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6695","last_scrub_stamp":"2024-04-23T16:43:26.892785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:26.892785+0000","objects_scrubbed":6673,"log_size":9996,"log_dups_size":0,"ondisk_log_size":9996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:56.021559+0000","scrub_duration":739,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320864256,"num_objects":4896,"num_object_clones":0,"num_object_copies":9792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9996,"num_write_kb":476544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"582'9810","reported_seq":9904,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:35.696154+0000","last_change":"2024-04-23T16:41:57.310144+0000","last_active":"2024-04-23T16:44:35.696154+0000","last_peered":"2024-04-23T16:44:35.696154+0000","last_clean":"2024-04-23T16:44:35.696154+0000","last_became_active":"2024-04-23T16:41:57.309841+0000","last_became_peered":"2024-04-23T16:41:57.309841+0000","last_unstale":"2024-04-23T16:44:35.696154+0000","last_undegraded":"2024-04-23T16:44:35.696154+0000","last_fullsized":"2024-04-23T16:44:35.696154+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":9810,"log_dups_size":0,"ondisk_log_size":9810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309985280,"num_objects":4730,"num_object_clones":0,"num_object_copies":9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9810,"num_write_kb":465280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"582'9991","reported_seq":15542,"reported_epoch":582,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:35.022667+0000","last_change":"2024-04-23T16:43:08.145899+0000","last_active":"2024-04-23T16:44:35.022667+0000","last_peered":"2024-04-23T16:44:35.022667+0000","last_clean":"2024-04-23T16:42:42.970788+0000","last_became_active":"2024-04-23T16:43:08.106076+0000","last_became_peered":"2024-04-23T16:43:08.106076+0000","last_unstale":"2024-04-23T16:44:35.022667+0000","last_undegraded":"2024-04-23T16:43:07.951894+0000","last_fullsized":"2024-04-23T16:44:35.022667+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":9991,"log_dups_size":0,"ondisk_log_size":9991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311623680,"num_objects":4755,"num_object_clones":0,"num_object_copies":9510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2988,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9991,"num_write_kb":471872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2726,"num_bytes_recovered":178651136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":2988},{"shards":"6,11","objects":1767}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1477,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:33.980230+0000","last_change":"2024-04-23T16:44:09.435333+0000","last_active":"2024-04-23T16:44:33.980230+0000","last_peered":"2024-04-23T16:44:33.980230+0000","last_clean":"2024-04-23T16:44:33.980230+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:44:33.980230+0000","last_undegraded":"2024-04-23T16:44:33.980230+0000","last_fullsized":"2024-04-23T16:44:33.980230+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:09.435212+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:44:09.435212+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:17.709965+0000","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"582'9955","reported_seq":10124,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:34.417870+0000","last_change":"2024-04-23T16:43:28.303396+0000","last_active":"2024-04-23T16:44:34.417870+0000","last_peered":"2024-04-23T16:44:34.417870+0000","last_clean":"2024-04-23T16:44:34.417870+0000","last_became_active":"2024-04-23T16:41:57.311753+0000","last_became_peered":"2024-04-23T16:41:57.311753+0000","last_unstale":"2024-04-23T16:44:34.417870+0000","last_undegraded":"2024-04-23T16:44:34.417870+0000","last_fullsized":"2024-04-23T16:44:34.417870+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6763","last_scrub_stamp":"2024-04-23T16:43:28.303250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:28.303250+0000","objects_scrubbed":6733,"log_size":9955,"log_dups_size":0,"ondisk_log_size":9955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:51.576731+0000","scrub_duration":1201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325779456,"num_objects":4971,"num_object_clones":0,"num_object_copies":9942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9955,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1473,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-04-23T16:44:31.940944+0000","last_change":"2024-04-23T16:43:27.668916+0000","last_active":"2024-04-23T16:44:31.940944+0000","last_peered":"2024-04-23T16:44:31.940944+0000","last_clean":"2024-04-23T16:44:31.940944+0000","last_became_active":"2024-04-23T16:43:07.962092+0000","last_became_peered":"2024-04-23T16:43:07.962092+0000","last_unstale":"2024-04-23T16:44:31.940944+0000","last_undegraded":"2024-04-23T16:44:31.940944+0000","last_fullsized":"2024-04-23T16:44:31.940944+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:43:27.668793+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:43:27.668793+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:45.595521+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"582'9948","reported_seq":10116,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:34.434171+0000","last_change":"2024-04-23T16:43:27.928311+0000","last_active":"2024-04-23T16:44:34.434171+0000","last_peered":"2024-04-23T16:44:34.434171+0000","last_clean":"2024-04-23T16:44:34.434171+0000","last_became_active":"2024-04-23T16:41:57.305944+0000","last_became_peered":"2024-04-23T16:41:57.305944+0000","last_unstale":"2024-04-23T16:44:34.434171+0000","last_undegraded":"2024-04-23T16:44:34.434171+0000","last_fullsized":"2024-04-23T16:44:34.434171+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6718","last_scrub_stamp":"2024-04-23T16:43:27.928174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:27.928174+0000","objects_scrubbed":6703,"log_size":9948,"log_dups_size":0,"ondisk_log_size":9948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:34.856929+0000","scrub_duration":750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321126400,"num_objects":4900,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9948,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1452,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:33.981857+0000","last_change":"2024-04-23T16:42:32.662620+0000","last_active":"2024-04-23T16:44:33.981857+0000","last_peered":"2024-04-23T16:44:33.981857+0000","last_clean":"2024-04-23T16:44:33.981857+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:44:33.981857+0000","last_undegraded":"2024-04-23T16:44:33.981857+0000","last_fullsized":"2024-04-23T16:44:33.981857+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:32.662476+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:42:32.662476+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"582'10226","reported_seq":10394,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:34.418571+0000","last_change":"2024-04-23T16:43:25.389006+0000","last_active":"2024-04-23T16:44:34.418571+0000","last_peered":"2024-04-23T16:44:34.418571+0000","last_clean":"2024-04-23T16:44:34.418571+0000","last_became_active":"2024-04-23T16:41:57.309858+0000","last_became_peered":"2024-04-23T16:41:57.309858+0000","last_unstale":"2024-04-23T16:44:34.418571+0000","last_undegraded":"2024-04-23T16:44:34.418571+0000","last_fullsized":"2024-04-23T16:44:34.418571+0000","mapping_epoch":540,"log_start":"547'200","ondisk_log_start":"547'200","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'6776","last_scrub_stamp":"2024-04-23T16:43:25.388898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:25.388898+0000","objects_scrubbed":6743,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:39.155605+0000","scrub_duration":1233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323747840,"num_objects":4940,"num_object_clones":0,"num_object_copies":9880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10226,"num_write_kb":485312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1476,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:33.975482+0000","last_change":"2024-04-23T16:43:40.258384+0000","last_active":"2024-04-23T16:44:33.975482+0000","last_peered":"2024-04-23T16:44:33.975482+0000","last_clean":"2024-04-23T16:44:33.975482+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:44:33.975482+0000","last_undegraded":"2024-04-23T16:44:33.975482+0000","last_fullsized":"2024-04-23T16:44:33.975482+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:43:40.258213+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:43:40.258213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:07.779820+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"582'9985","reported_seq":10079,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:36.426608+0000","last_change":"2024-04-23T16:41:57.310695+0000","last_active":"2024-04-23T16:44:36.426608+0000","last_peered":"2024-04-23T16:44:36.426608+0000","last_clean":"2024-04-23T16:44:36.426608+0000","last_became_active":"2024-04-23T16:41:57.310400+0000","last_became_peered":"2024-04-23T16:41:57.310400+0000","last_unstale":"2024-04-23T16:44:36.426608+0000","last_undegraded":"2024-04-23T16:44:36.426608+0000","last_fullsized":"2024-04-23T16:44:36.426608+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":9985,"log_dups_size":0,"ondisk_log_size":9985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308871168,"num_objects":4713,"num_object_clones":0,"num_object_copies":9426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9985,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1050,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:33.964790+0000","last_change":"2024-04-23T16:44:18.555667+0000","last_active":"2024-04-23T16:44:33.964790+0000","last_peered":"2024-04-23T16:44:33.964790+0000","last_clean":"2024-04-23T16:44:33.964790+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:44:33.964790+0000","last_undegraded":"2024-04-23T16:44:33.964790+0000","last_fullsized":"2024-04-23T16:44:33.964790+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:18.555538+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:44:18.555538+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:35.829709+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"582'9917","reported_seq":10011,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:34.991153+0000","last_change":"2024-04-23T16:41:57.306526+0000","last_active":"2024-04-23T16:44:34.991153+0000","last_peered":"2024-04-23T16:44:34.991153+0000","last_clean":"2024-04-23T16:44:34.991153+0000","last_became_active":"2024-04-23T16:41:57.306229+0000","last_became_peered":"2024-04-23T16:41:57.306229+0000","last_unstale":"2024-04-23T16:44:34.991153+0000","last_undegraded":"2024-04-23T16:44:34.991153+0000","last_fullsized":"2024-04-23T16:44:34.991153+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":9917,"log_dups_size":0,"ondisk_log_size":9917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313720832,"num_objects":4787,"num_object_clones":0,"num_object_copies":9574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9917,"num_write_kb":470528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1003,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-04-23T16:44:31.963778+0000","last_change":"2024-04-23T16:44:07.230780+0000","last_active":"2024-04-23T16:44:31.963778+0000","last_peered":"2024-04-23T16:44:31.963778+0000","last_clean":"2024-04-23T16:44:31.963778+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:44:31.963778+0000","last_undegraded":"2024-04-23T16:44:31.963778+0000","last_fullsized":"2024-04-23T16:44:31.963778+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:07.230640+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:44:07.230640+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:12.044599+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.0","version":"581'9814","reported_seq":9983,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-04-23T16:44:32.730592+0000","last_change":"2024-04-23T16:43:30.757721+0000","last_active":"2024-04-23T16:44:32.730592+0000","last_peered":"2024-04-23T16:44:32.730592+0000","last_clean":"2024-04-23T16:44:32.730592+0000","last_became_active":"2024-04-23T16:41:57.311479+0000","last_became_peered":"2024-04-23T16:41:57.311479+0000","last_unstale":"2024-04-23T16:44:32.730592+0000","last_undegraded":"2024-04-23T16:44:32.730592+0000","last_fullsized":"2024-04-23T16:44:32.730592+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6823","last_scrub_stamp":"2024-04-23T16:43:30.757555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:30.757555+0000","objects_scrubbed":6802,"log_size":9814,"log_dups_size":0,"ondisk_log_size":9814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:59.054459+0000","scrub_duration":1354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320208896,"num_objects":4886,"num_object_clones":0,"num_object_copies":9772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9814,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1485,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:33.980148+0000","last_change":"2024-04-23T16:44:11.349248+0000","last_active":"2024-04-23T16:44:33.980148+0000","last_peered":"2024-04-23T16:44:33.980148+0000","last_clean":"2024-04-23T16:44:33.980148+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:44:33.980148+0000","last_undegraded":"2024-04-23T16:44:33.980148+0000","last_fullsized":"2024-04-23T16:44:33.980148+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:11.348079+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:44:11.348079+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:21.183735+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.1","version":"582'9757","reported_seq":14735,"reported_epoch":582,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:34.399332+0000","last_change":"2024-04-23T16:43:08.081567+0000","last_active":"2024-04-23T16:44:34.399332+0000","last_peered":"2024-04-23T16:44:34.399332+0000","last_clean":"2024-04-23T16:42:42.976600+0000","last_became_active":"2024-04-23T16:43:08.067887+0000","last_became_peered":"2024-04-23T16:43:08.067887+0000","last_unstale":"2024-04-23T16:44:34.399332+0000","last_undegraded":"2024-04-23T16:43:07.950381+0000","last_fullsized":"2024-04-23T16:44:34.399332+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":9757,"log_dups_size":0,"ondisk_log_size":9757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319750144,"num_objects":4879,"num_object_clones":0,"num_object_copies":9758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9757,"num_write_kb":468352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2440,"num_bytes_recovered":159907840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":3148},{"shards":"4,6","objects":1731}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1469,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:33.966795+0000","last_change":"2024-04-23T16:43:07.918302+0000","last_active":"2024-04-23T16:44:33.966795+0000","last_peered":"2024-04-23T16:44:33.966795+0000","last_clean":"2024-04-23T16:44:33.966795+0000","last_became_active":"2024-04-23T16:43:07.918042+0000","last_became_peered":"2024-04-23T16:43:07.918042+0000","last_unstale":"2024-04-23T16:44:33.966795+0000","last_undegraded":"2024-04-23T16:44:33.966795+0000","last_fullsized":"2024-04-23T16:44:33.966795+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:33.716343+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:42:33.716343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"582'9969","reported_seq":15195,"reported_epoch":582,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:34.990980+0000","last_change":"2024-04-23T16:43:08.363306+0000","last_active":"2024-04-23T16:44:34.990980+0000","last_peered":"2024-04-23T16:44:34.990980+0000","last_clean":"2024-04-23T16:42:42.950766+0000","last_became_active":"2024-04-23T16:43:08.333793+0000","last_became_peered":"2024-04-23T16:43:08.333793+0000","last_unstale":"2024-04-23T16:44:34.990980+0000","last_undegraded":"2024-04-23T16:43:07.950356+0000","last_fullsized":"2024-04-23T16:44:34.990980+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":9969,"log_dups_size":0,"ondisk_log_size":9969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317784064,"num_objects":4849,"num_object_clones":0,"num_object_copies":9698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9969,"num_write_kb":474176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2566,"num_bytes_recovered":168165376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":3171},{"shards":"6,7","objects":1678}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1361,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:33.972223+0000","last_change":"2024-04-23T16:41:47.722866+0000","last_active":"2024-04-23T16:44:33.972223+0000","last_peered":"2024-04-23T16:44:33.972223+0000","last_clean":"2024-04-23T16:44:33.972223+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:44:33.972223+0000","last_undegraded":"2024-04-23T16:44:33.972223+0000","last_fullsized":"2024-04-23T16:44:33.972223+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:41:47.722762+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"582'9989","reported_seq":16036,"reported_epoch":582,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:34.408936+0000","last_change":"2024-04-23T16:43:08.136028+0000","last_active":"2024-04-23T16:44:34.408936+0000","last_peered":"2024-04-23T16:44:34.408936+0000","last_clean":"2024-04-23T16:42:42.962639+0000","last_became_active":"2024-04-23T16:43:08.109240+0000","last_became_peered":"2024-04-23T16:43:08.109240+0000","last_unstale":"2024-04-23T16:44:34.408936+0000","last_undegraded":"2024-04-23T16:43:07.955254+0000","last_fullsized":"2024-04-23T16:44:34.408936+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":9989,"log_dups_size":0,"ondisk_log_size":9989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323420160,"num_objects":4935,"num_object_clones":0,"num_object_copies":9870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9989,"num_write_kb":477568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2974,"num_bytes_recovered":194904064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":2682},{"shards":"4,11","objects":2253}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1283,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:33.967630+0000","last_change":"2024-04-23T16:41:49.835121+0000","last_active":"2024-04-23T16:44:33.967630+0000","last_peered":"2024-04-23T16:44:33.967630+0000","last_clean":"2024-04-23T16:44:33.967630+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:44:33.967630+0000","last_undegraded":"2024-04-23T16:44:33.967630+0000","last_fullsized":"2024-04-23T16:44:33.967630+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:41:49.834374+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"582'9836","reported_seq":10003,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:34.418518+0000","last_change":"2024-04-23T16:43:26.589353+0000","last_active":"2024-04-23T16:44:34.418518+0000","last_peered":"2024-04-23T16:44:34.418518+0000","last_clean":"2024-04-23T16:44:34.418518+0000","last_became_active":"2024-04-23T16:41:57.309349+0000","last_became_peered":"2024-04-23T16:41:57.309349+0000","last_unstale":"2024-04-23T16:44:34.418518+0000","last_undegraded":"2024-04-23T16:44:34.418518+0000","last_fullsized":"2024-04-23T16:44:34.418518+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6582","last_scrub_stamp":"2024-04-23T16:43:26.589248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:26.589248+0000","objects_scrubbed":6550,"log_size":9836,"log_dups_size":0,"ondisk_log_size":9836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:37.240413+0000","scrub_duration":1397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315097088,"num_objects":4808,"num_object_clones":0,"num_object_copies":9616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9836,"num_write_kb":468608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1452,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-04-23T16:44:33.982408+0000","last_change":"2024-04-23T16:42:33.628340+0000","last_active":"2024-04-23T16:44:33.982408+0000","last_peered":"2024-04-23T16:44:33.982408+0000","last_clean":"2024-04-23T16:44:33.982408+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:44:33.982408+0000","last_undegraded":"2024-04-23T16:44:33.982408+0000","last_fullsized":"2024-04-23T16:44:33.982408+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:33.628217+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:42:33.628217+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5100208152,"num_objects":77824,"num_object_clones":0,"num_object_copies":155648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77824,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159534,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10706,"num_bytes_recovered":701628416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10836254720,"data_stored":10836246576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159034,"ondisk_log_size":159034,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1474560,"data_stored":1460352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056992,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1803344,"kb_used_data":1186244,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":91941808,"statfs":{"total":95995035648,"available":94148411392,"internally_reserved":0,"allocated":1214713856,"data_stored":1214315669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":631830980},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351333,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612971,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1769744,"kb_used_data":1255428,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":91975408,"statfs":{"total":95995035648,"available":94182817792,"internally_reserved":0,"allocated":1285558272,"data_stored":1285160085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":526645696},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841903,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1016716,"kb_used_data":634304,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92728436,"statfs":{"total":95995035648,"available":94953918464,"internally_reserved":0,"allocated":649527296,"data_stored":649131611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168949,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":988364,"kb_used_data":411588,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92756788,"statfs":{"total":95995035648,"available":94982950912,"internally_reserved":0,"allocated":421466112,"data_stored":421067925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":590608842},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463290,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1854292,"kb_used_data":1353672,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":91890860,"statfs":{"total":95995035648,"available":94096240640,"internally_reserved":0,"allocated":1386160128,"data_stored":1385757869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":512424390},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137431,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":859220,"kb_used_data":628616,"kb_used_omap":231,"kb_used_meta":230360,"kb_avail":92885932,"statfs":{"total":95995035648,"available":95115194368,"internally_reserved":0,"allocated":643702784,"data_stored":643300525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":235889552},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738776,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1840720,"kb_used_data":1255620,"kb_used_omap":1,"kb_used_meta":585086,"kb_avail":91904432,"statfs":{"total":95995035648,"available":94110138368,"internally_reserved":0,"allocated":1285754880,"data_stored":1285356693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":599128517},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269446,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921154,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2280660,"kb_used_data":1559428,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":91464492,"statfs":{"total":95995035648,"available":93659639808,"internally_reserved":0,"allocated":1596854272,"data_stored":1596456085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,2,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757629,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1591752,"kb_used_data":956608,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92153400,"statfs":{"total":95995035648,"available":94365081600,"internally_reserved":0,"allocated":979566592,"data_stored":979170907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1280770048,"data_stored":1280770048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":640552960,"data_stored":640548888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1411842048,"data_stored":1411842048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1595539456,"data_stored":1595539456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":977666048,"data_stored":977666048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1381699584,"data_stored":1381695512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":420085760,"data_stored":420085760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":644022272,"data_stored":644022272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1281032192,"data_stored":1281032192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1203044352,"data_stored":1203044352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:44:39.784 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:44:39.831 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:44:39.860 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:40.017 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:40 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:44:40.017 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:40 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:44:40.018 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:40 smithi148 ceph-mon[44520]: pgmap v1634: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.7 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 526 op/s; 11989/155204 objects degraded (7.725%); 6.9 MiB/s, 110 objects/s recovering 2024-04-23T16:44:40.018 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:40 smithi148 ceph-mon[44520]: osdmap e583: 12 total, 11 up, 9 in 2024-04-23T16:44:40.018 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:40 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 11989/155204 objects degraded (7.725%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:40.018 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:40 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:44:40.018 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:40 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:44:40.018 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:40 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:44:40.018 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:40 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:44:40.018 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:40 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:44:40.018 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:40 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:44:40.019 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:40 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:44:40.019 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:40 smithi103 ceph-mon[39629]: pgmap v1634: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.7 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 526 op/s; 11989/155204 objects degraded (7.725%); 6.9 MiB/s, 110 objects/s recovering 2024-04-23T16:44:40.019 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:40 smithi103 ceph-mon[39629]: osdmap e583: 12 total, 11 up, 9 in 2024-04-23T16:44:40.019 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:40 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 11989/155204 objects degraded (7.725%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:40.019 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:40 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:44:40.019 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:40 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:44:40.019 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:40 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:44:40.019 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:40 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:44:40.019 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:40 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:44:40.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:40 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:44:40.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:40 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:44:40.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:40 smithi188 ceph-mon[44101]: pgmap v1634: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.7 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 526 op/s; 11989/155204 objects degraded (7.725%); 6.9 MiB/s, 110 objects/s recovering 2024-04-23T16:44:40.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:40 smithi188 ceph-mon[44101]: osdmap e583: 12 total, 11 up, 9 in 2024-04-23T16:44:40.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:40 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 11989/155204 objects degraded (7.725%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:40.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:40 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:44:40.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:40 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:44:40.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:40 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:44:40.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:40 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:44:40.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:40 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:44:40.515 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:44:40.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:40.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:44:40.620 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:44:40.688 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:40.726 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:44:40.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:40.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:44:40.836 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:44:40.871 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:40.929 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-23T16:44:41.262 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:44:41.331 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:44:41.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:41.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:44:41.337 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:44:41.436 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:44:41.465 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:41.522 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:44:41.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:41.531 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:44:41.632 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:44:41.663 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:41.766 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:44:41.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:41.769 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:44:41.870 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:44:41.905 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:42.021 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:42 smithi188 ceph-mon[44101]: pgmap v1636: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 824 op/s; 11092/149082 objects degraded (7.440%); 14 MiB/s, 222 objects/s recovering 2024-04-23T16:44:42.021 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:42 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1568896764' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:44:42.021 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:42 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:44:42.021 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:42 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:44:42.040 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:44:42.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:42 smithi103 ceph-mon[39629]: pgmap v1636: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 824 op/s; 11092/149082 objects degraded (7.440%); 14 MiB/s, 222 objects/s recovering 2024-04-23T16:44:42.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:42 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1568896764' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:44:42.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:42 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:44:42.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:42 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:44:42.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:42 smithi148 ceph-mon[44520]: pgmap v1636: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 824 op/s; 11092/149082 objects degraded (7.440%); 14 MiB/s, 222 objects/s recovering 2024-04-23T16:44:42.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:42 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1568896764' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:44:42.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:42 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:44:42.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:42 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:44:43.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:43 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:44:43.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:43 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1568896764' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:44:43.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:43 smithi188 ceph-mon[44101]: osdmap e584: 12 total, 11 up, 9 in 2024-04-23T16:44:43.261 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:44:43.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:43.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:44:43.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:43 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:44:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:43 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1568896764' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:44:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:43 smithi103 ceph-mon[39629]: osdmap e584: 12 total, 11 up, 9 in 2024-04-23T16:44:43.365 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:44:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:43 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:44:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:43 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1568896764' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:44:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:43 smithi148 ceph-mon[44520]: osdmap e584: 12 total, 11 up, 9 in 2024-04-23T16:44:43.408 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:43.514 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:44:43.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:43.518 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:44:43.566 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:44:43.566 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:44:43.567 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:44:43.567 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:44:43.567 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:44:43.618 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:44:43.623 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:44:43.623 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:44:43.624 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:44:43.624 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:44:43.646 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:44:43.669 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:44:43.695 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:44:43.719 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:44:43.736 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:43.741 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:44:43.762 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:44:43.777 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:44:43.782 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:44:43.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:44:43.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:44:43.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:44:43.783 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:44:43.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:44:43.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:44:43.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:44:43.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:44:43.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:44:43.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:44:43.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:44:43.783 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:44:43.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:43.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:44:43.887 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:44:43.954 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:43.989 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:44:44.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:44.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:44:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:44 smithi103 ceph-mon[39629]: pgmap v1638: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 684 op/s; 11092/147714 objects degraded (7.509%); 10 MiB/s, 166 objects/s recovering 2024-04-23T16:44:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:44 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:44:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:44 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:44:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:44 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3991328431' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:44:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:44.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:44.105 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:44:44.196 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:44.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:44 smithi188 ceph-mon[44101]: pgmap v1638: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 684 op/s; 11092/147714 objects degraded (7.509%); 10 MiB/s, 166 objects/s recovering 2024-04-23T16:44:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:44 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:44:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:44 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:44:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:44 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3991328431' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:44:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:44.229 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:44:44.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:44.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:44:44.287 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-23T16:44:44.337 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:44:44.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:44 smithi148 ceph-mon[44520]: pgmap v1638: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 684 op/s; 11092/147714 objects degraded (7.509%); 10 MiB/s, 166 objects/s recovering 2024-04-23T16:44:44.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:44 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:44:44.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:44 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:44:44.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:44 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3991328431' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:44:44.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:44.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:44.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:44.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:44.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:44.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:44.420 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:44.472 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:44:44.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:44.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:44:44.579 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:44:44.609 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:44.657 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:44:45.059 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:45 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:45.060 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:45 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:45.060 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:45 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:45.060 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:45 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:45.060 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:45 smithi103 ceph-mon[39629]: osdmap e585: 12 total, 11 up, 9 in 2024-04-23T16:44:45.060 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:45 smithi103 ceph-mon[39629]: 7.f scrub starts 2024-04-23T16:44:45.060 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:45 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 11092/147714 objects degraded (7.509%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:45.060 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:45 smithi103 ceph-mon[39629]: 7.f scrub ok 2024-04-23T16:44:45.199 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:45 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:45.199 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:45 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:45.199 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:45 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:45.200 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:45 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:45.200 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:45 smithi188 ceph-mon[44101]: osdmap e585: 12 total, 11 up, 9 in 2024-04-23T16:44:45.200 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:45 smithi188 ceph-mon[44101]: 7.f scrub starts 2024-04-23T16:44:45.200 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:45 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 11092/147714 objects degraded (7.509%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:45.200 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:45 smithi188 ceph-mon[44101]: 7.f scrub ok 2024-04-23T16:44:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:45 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:45 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:45 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:45 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:45 smithi148 ceph-mon[44520]: osdmap e585: 12 total, 11 up, 9 in 2024-04-23T16:44:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:45 smithi148 ceph-mon[44520]: 7.f scrub starts 2024-04-23T16:44:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:45 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 11092/147714 objects degraded (7.509%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:45 smithi148 ceph-mon[44520]: 7.f scrub ok 2024-04-23T16:44:45.628 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:44:45.629 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:44:45.990 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:44:45.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:45.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:44:46.094 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:44:46.125 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:46.147 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:44:46.258 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:44:46.261 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:46 smithi103 ceph-mon[39629]: pgmap v1640: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 22 active+clean; 4.4 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 933 op/s; 10622/143994 objects degraded (7.377%); 14 MiB/s, 227 objects/s recovering 2024-04-23T16:44:46.261 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:46 smithi103 ceph-mon[39629]: 7.6 scrub starts 2024-04-23T16:44:46.261 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:46 smithi103 ceph-mon[39629]: 7.a scrub starts 2024-04-23T16:44:46.261 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:46 smithi103 ceph-mon[39629]: 1.8 scrub starts 2024-04-23T16:44:46.261 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:46 smithi103 ceph-mon[39629]: 7.a scrub ok 2024-04-23T16:44:46.262 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:46 smithi103 ceph-mon[39629]: 1.8 scrub ok 2024-04-23T16:44:46.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:46.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:44:46.364 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:44:46.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:46 smithi148 ceph-mon[44520]: pgmap v1640: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 22 active+clean; 4.4 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 933 op/s; 10622/143994 objects degraded (7.377%); 14 MiB/s, 227 objects/s recovering 2024-04-23T16:44:46.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:46 smithi148 ceph-mon[44520]: 7.6 scrub starts 2024-04-23T16:44:46.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:46 smithi148 ceph-mon[44520]: 7.a scrub starts 2024-04-23T16:44:46.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:46 smithi148 ceph-mon[44520]: 1.8 scrub starts 2024-04-23T16:44:46.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:46 smithi148 ceph-mon[44520]: 7.a scrub ok 2024-04-23T16:44:46.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:46 smithi148 ceph-mon[44520]: 1.8 scrub ok 2024-04-23T16:44:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:46 smithi188 ceph-mon[44101]: pgmap v1640: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 22 active+clean; 4.4 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 933 op/s; 10622/143994 objects degraded (7.377%); 14 MiB/s, 227 objects/s recovering 2024-04-23T16:44:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:46 smithi188 ceph-mon[44101]: 7.6 scrub starts 2024-04-23T16:44:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:46 smithi188 ceph-mon[44101]: 7.a scrub starts 2024-04-23T16:44:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:46 smithi188 ceph-mon[44101]: 1.8 scrub starts 2024-04-23T16:44:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:46 smithi188 ceph-mon[44101]: 7.a scrub ok 2024-04-23T16:44:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:46 smithi188 ceph-mon[44101]: 1.8 scrub ok 2024-04-23T16:44:46.414 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:46.472 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:44:46.487 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:44:46.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:46.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:44:46.517 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:44:46.520 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:44:46.594 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:44:46.629 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:47.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:47 smithi103 ceph-mon[39629]: 7.6 scrub ok 2024-04-23T16:44:47.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:47 smithi103 ceph-mon[39629]: 7.4 scrub starts 2024-04-23T16:44:47.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:47 smithi103 ceph-mon[39629]: from='client.16836 v1:172.21.15.103:0/120195390' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:44:47.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:47 smithi103 ceph-mon[39629]: pgmap v1641: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 22 active+clean; 4.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 921 op/s; 10170/140736 objects degraded (7.226%); 14 MiB/s, 231 objects/s recovering 2024-04-23T16:44:47.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:47 smithi148 ceph-mon[44520]: 7.6 scrub ok 2024-04-23T16:44:47.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:47 smithi148 ceph-mon[44520]: 7.4 scrub starts 2024-04-23T16:44:47.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:47 smithi148 ceph-mon[44520]: from='client.16836 v1:172.21.15.103:0/120195390' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:44:47.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:47 smithi148 ceph-mon[44520]: pgmap v1641: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 22 active+clean; 4.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 921 op/s; 10170/140736 objects degraded (7.226%); 14 MiB/s, 231 objects/s recovering 2024-04-23T16:44:47.383 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:44:47.383 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:44:47.408 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:47 smithi188 ceph-mon[44101]: 7.6 scrub ok 2024-04-23T16:44:47.408 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:47 smithi188 ceph-mon[44101]: 7.4 scrub starts 2024-04-23T16:44:47.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:47 smithi188 ceph-mon[44101]: from='client.16836 v1:172.21.15.103:0/120195390' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:44:47.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:47 smithi188 ceph-mon[44101]: pgmap v1641: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 22 active+clean; 4.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 921 op/s; 10170/140736 objects degraded (7.226%); 14 MiB/s, 231 objects/s recovering 2024-04-23T16:44:47.498 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:44:47.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:47.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:44:47.602 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:44:47.634 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:44:47.637 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:47.661 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:44:47.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:47.664 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:44:47.765 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:44:47.846 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:47.890 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:44:47.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:47.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:44:47.999 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:44:48.067 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:48.097 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:48 smithi148 ceph-mon[44520]: 7.4 scrub ok 2024-04-23T16:44:48.097 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:48 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:44:48.097 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:48 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:44:48.117 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:44:48.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:48.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:44:48.220 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:44:48.273 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:48.310 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:44:48.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:48.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:44:48.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:48 smithi103 ceph-mon[39629]: 7.4 scrub ok 2024-04-23T16:44:48.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:48 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:44:48.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:48 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:44:48.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:48 smithi188 ceph-mon[44101]: 7.4 scrub ok 2024-04-23T16:44:48.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:48 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:44:48.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:48 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:44:48.429 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:44:48.460 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-23T16:44:48.508 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:48.547 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:44:48.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:48.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:44:48.655 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:44:48.733 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:48.779 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:44:48.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:48.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:44:48.828 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:44:48.886 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:44:48.916 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:49.105 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:49 smithi148 ceph-mon[44520]: osdmap e586: 12 total, 11 up, 9 in 2024-04-23T16:44:49.106 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:49 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:44:49.106 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:49 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:44:49.106 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:49 smithi148 ceph-mon[44520]: pgmap v1643: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 22 active+clean; 4.3 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 639 op/s; 10170/139456 objects degraded (7.293%); 8.4 MiB/s, 134 objects/s recovering 2024-04-23T16:44:49.122 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:44:49.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:49.126 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:44:49.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:49 smithi188 ceph-mon[44101]: osdmap e586: 12 total, 11 up, 9 in 2024-04-23T16:44:49.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:49 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:44:49.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:49 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:44:49.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:49 smithi188 ceph-mon[44101]: pgmap v1643: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 22 active+clean; 4.3 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 639 op/s; 10170/139456 objects degraded (7.293%); 8.4 MiB/s, 134 objects/s recovering 2024-04-23T16:44:49.227 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:44:49.257 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:49.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:49 smithi103 ceph-mon[39629]: osdmap e586: 12 total, 11 up, 9 in 2024-04-23T16:44:49.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:49 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:44:49.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:49 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:44:49.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:49 smithi103 ceph-mon[39629]: pgmap v1643: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 22 active+clean; 4.3 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 639 op/s; 10170/139456 objects degraded (7.293%); 8.4 MiB/s, 134 objects/s recovering 2024-04-23T16:44:49.711 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:44:49.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:49.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:44:49.818 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:44:49.849 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:50.202 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:44:50.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:50.205 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:44:50.307 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:44:50.342 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:50.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:50 smithi103 ceph-mon[39629]: osdmap e587: 12 total, 11 up, 9 in 2024-04-23T16:44:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:50 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 10170/139456 objects degraded (7.293%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:50.367 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:44:50.370 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:50 smithi188 ceph-mon[44101]: osdmap e587: 12 total, 11 up, 9 in 2024-04-23T16:44:50.370 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:50 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 10170/139456 objects degraded (7.293%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:50.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:50.371 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:44:50.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:50 smithi148 ceph-mon[44520]: osdmap e587: 12 total, 11 up, 9 in 2024-04-23T16:44:50.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:50 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 10170/139456 objects degraded (7.293%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:50.472 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:44:50.519 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:50.679 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:44:50.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:50.684 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:44:50.786 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:44:50.841 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:44:50.841 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:44:50.853 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:50.892 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:44:50.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:50.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:44:50.997 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:44:51.027 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:51.118 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:51 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:44:51.118 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:51 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:44:51.119 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:51 smithi148 ceph-mon[44520]: pgmap v1645: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.1 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 994 op/s; 9192/134146 objects degraded (6.852%); 17 MiB/s, 278 objects/s recovering 2024-04-23T16:44:51.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:51 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:44:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:51 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:44:51.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:51 smithi103 ceph-mon[39629]: pgmap v1645: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.1 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 994 op/s; 9192/134146 objects degraded (6.852%); 17 MiB/s, 278 objects/s recovering 2024-04-23T16:44:51.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:51 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:44:51.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:51 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:44:51.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:51 smithi188 ceph-mon[44101]: pgmap v1645: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.1 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 994 op/s; 9192/134146 objects degraded (6.852%); 17 MiB/s, 278 objects/s recovering 2024-04-23T16:44:51.513 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-23T16:44:51.861 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:44:52.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:52 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:44:52.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:52 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:44:52.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:52 smithi148 ceph-mon[44520]: osdmap e588: 12 total, 11 up, 9 in 2024-04-23T16:44:52.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:52 smithi148 ceph-mon[44520]: 7.e scrub starts 2024-04-23T16:44:52.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:52 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:44:52.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:52 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:44:52.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:52 smithi188 ceph-mon[44101]: osdmap e588: 12 total, 11 up, 9 in 2024-04-23T16:44:52.395 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:52 smithi188 ceph-mon[44101]: 7.e scrub starts 2024-04-23T16:44:52.485 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:52 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:44:52.485 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:52 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:44:52.485 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:52 smithi103 ceph-mon[39629]: osdmap e588: 12 total, 11 up, 9 in 2024-04-23T16:44:52.485 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:52 smithi103 ceph-mon[39629]: 7.e scrub starts 2024-04-23T16:44:52.513 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1640,"stamp":"2024-04-23T16:44:44.867117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4718789176,"num_objects":71997,"num_object_clones":0,"num_object_copies":143994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10622,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71997,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165363,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12075,"num_bytes_recovered":791618624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162292,"ondisk_log_size":162292,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":13321504,"kb_used_data":8559524,"kb_used_omap":243,"kb_used_meta":4761228,"kb_avail":830384864,"statfs":{"total":863955320832,"available":850314100736,"internally_reserved":0,"allocated":8764952576,"data_stored":8761834552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":249416,"internal_metadata":4875497912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,4,5,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-367329280,"num_objects":-5605,"num_object_clones":0,"num_object_copies":-11210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1367,"num_bytes_recovered":89587712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001311"},"pg_stats":[{"pgid":"7.d","version":"583'10373","reported_seq":10470,"reported_epoch":583,"state":"active+clean","last_fresh":"2024-04-23T16:44:40.102385+0000","last_change":"2024-04-23T16:41:57.310289+0000","last_active":"2024-04-23T16:44:40.102385+0000","last_peered":"2024-04-23T16:44:40.102385+0000","last_clean":"2024-04-23T16:44:40.102385+0000","last_became_active":"2024-04-23T16:41:57.310025+0000","last_became_peered":"2024-04-23T16:41:57.310025+0000","last_unstale":"2024-04-23T16:44:40.102385+0000","last_undegraded":"2024-04-23T16:44:40.102385+0000","last_fullsized":"2024-04-23T16:44:40.102385+0000","mapping_epoch":540,"log_start":"547'300","ondisk_log_start":"547'300","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307757080,"num_objects":4697,"num_object_clones":0,"num_object_copies":9394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10373,"num_write_kb":482177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"583'10518","reported_seq":10691,"reported_epoch":583,"state":"active+clean","last_fresh":"2024-04-23T16:44:40.773322+0000","last_change":"2024-04-23T16:43:29.012370+0000","last_active":"2024-04-23T16:44:40.773322+0000","last_peered":"2024-04-23T16:44:40.773322+0000","last_clean":"2024-04-23T16:44:40.773322+0000","last_became_active":"2024-04-23T16:41:57.310754+0000","last_became_peered":"2024-04-23T16:41:57.310754+0000","last_unstale":"2024-04-23T16:44:40.773322+0000","last_undegraded":"2024-04-23T16:44:40.773322+0000","last_fullsized":"2024-04-23T16:44:40.773322+0000","mapping_epoch":540,"log_start":"547'500","ondisk_log_start":"547'500","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6945","last_scrub_stamp":"2024-04-23T16:43:29.012285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:29.012285+0000","objects_scrubbed":6912,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:58.968073+0000","scrub_duration":1082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303169536,"num_objects":4626,"num_object_clones":0,"num_object_copies":9252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10518,"num_write_kb":484608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"585'10379","reported_seq":10553,"reported_epoch":585,"state":"active+clean","last_fresh":"2024-04-23T16:44:44.423148+0000","last_change":"2024-04-23T16:43:25.997269+0000","last_active":"2024-04-23T16:44:44.423148+0000","last_peered":"2024-04-23T16:44:44.423148+0000","last_clean":"2024-04-23T16:44:44.423148+0000","last_became_active":"2024-04-23T16:41:57.310189+0000","last_became_peered":"2024-04-23T16:41:57.310189+0000","last_unstale":"2024-04-23T16:44:44.423148+0000","last_undegraded":"2024-04-23T16:44:44.423148+0000","last_fullsized":"2024-04-23T16:44:44.423148+0000","mapping_epoch":540,"log_start":"547'300","ondisk_log_start":"547'300","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'6616","last_scrub_stamp":"2024-04-23T16:43:25.997150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:25.997150+0000","objects_scrubbed":6600,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296419328,"num_objects":4523,"num_object_clones":0,"num_object_copies":9046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10379,"num_write_kb":476864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"585'10454","reported_seq":10627,"reported_epoch":585,"state":"active+clean","last_fresh":"2024-04-23T16:44:44.416063+0000","last_change":"2024-04-23T16:43:26.892913+0000","last_active":"2024-04-23T16:44:44.416063+0000","last_peered":"2024-04-23T16:44:44.416063+0000","last_clean":"2024-04-23T16:44:44.416063+0000","last_became_active":"2024-04-23T16:41:57.309569+0000","last_became_peered":"2024-04-23T16:41:57.309569+0000","last_unstale":"2024-04-23T16:44:44.416063+0000","last_undegraded":"2024-04-23T16:44:44.416063+0000","last_fullsized":"2024-04-23T16:44:44.416063+0000","mapping_epoch":540,"log_start":"547'400","ondisk_log_start":"547'400","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6695","last_scrub_stamp":"2024-04-23T16:43:26.892785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:26.892785+0000","objects_scrubbed":6673,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:56.021559+0000","scrub_duration":739,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290848768,"num_objects":4438,"num_object_clones":0,"num_object_copies":8876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10454,"num_write_kb":476544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"583'10044","reported_seq":10140,"reported_epoch":583,"state":"active+clean","last_fresh":"2024-04-23T16:44:40.748079+0000","last_change":"2024-04-23T16:41:57.310144+0000","last_active":"2024-04-23T16:44:40.748079+0000","last_peered":"2024-04-23T16:44:40.748079+0000","last_clean":"2024-04-23T16:44:40.748079+0000","last_became_active":"2024-04-23T16:41:57.309841+0000","last_became_peered":"2024-04-23T16:41:57.309841+0000","last_unstale":"2024-04-23T16:44:40.748079+0000","last_undegraded":"2024-04-23T16:44:40.748079+0000","last_fullsized":"2024-04-23T16:44:40.748079+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294649856,"num_objects":4496,"num_object_clones":0,"num_object_copies":8992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10044,"num_write_kb":465280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"583'10226","reported_seq":16188,"reported_epoch":583,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:40.139328+0000","last_change":"2024-04-23T16:43:08.145899+0000","last_active":"2024-04-23T16:44:40.139328+0000","last_peered":"2024-04-23T16:44:40.139328+0000","last_clean":"2024-04-23T16:42:42.970788+0000","last_became_active":"2024-04-23T16:43:08.106076+0000","last_became_peered":"2024-04-23T16:43:08.106076+0000","last_unstale":"2024-04-23T16:44:40.139328+0000","last_undegraded":"2024-04-23T16:43:07.951894+0000","last_fullsized":"2024-04-23T16:44:40.139328+0000","mapping_epoch":563,"log_start":"547'200","ondisk_log_start":"547'200","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296222720,"num_objects":4520,"num_object_clones":0,"num_object_copies":9040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2784,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10226,"num_write_kb":471872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2930,"num_bytes_recovered":192020480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":2784},{"shards":"6,11","objects":1736}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1483,"reported_epoch":585,"state":"active+clean","last_fresh":"2024-04-23T16:44:44.054804+0000","last_change":"2024-04-23T16:44:09.435333+0000","last_active":"2024-04-23T16:44:44.054804+0000","last_peered":"2024-04-23T16:44:44.054804+0000","last_clean":"2024-04-23T16:44:44.054804+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:44:44.054804+0000","last_undegraded":"2024-04-23T16:44:44.054804+0000","last_fullsized":"2024-04-23T16:44:44.054804+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:09.435212+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:44:09.435212+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:17.709965+0000","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"585'10414","reported_seq":10589,"reported_epoch":585,"state":"active+clean","last_fresh":"2024-04-23T16:44:44.463615+0000","last_change":"2024-04-23T16:43:28.303396+0000","last_active":"2024-04-23T16:44:44.463615+0000","last_peered":"2024-04-23T16:44:44.463615+0000","last_clean":"2024-04-23T16:44:44.463615+0000","last_became_active":"2024-04-23T16:41:57.311753+0000","last_became_peered":"2024-04-23T16:41:57.311753+0000","last_unstale":"2024-04-23T16:44:44.463615+0000","last_undegraded":"2024-04-23T16:44:44.463615+0000","last_fullsized":"2024-04-23T16:44:44.463615+0000","mapping_epoch":540,"log_start":"547'400","ondisk_log_start":"547'400","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6763","last_scrub_stamp":"2024-04-23T16:43:28.303250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:28.303250+0000","objects_scrubbed":6733,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:51.576731+0000","scrub_duration":1201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295698432,"num_objects":4512,"num_object_clones":0,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10414,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1479,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-04-23T16:44:42.047173+0000","last_change":"2024-04-23T16:43:27.668916+0000","last_active":"2024-04-23T16:44:42.047173+0000","last_peered":"2024-04-23T16:44:42.047173+0000","last_clean":"2024-04-23T16:44:42.047173+0000","last_became_active":"2024-04-23T16:43:07.962092+0000","last_became_peered":"2024-04-23T16:43:07.962092+0000","last_unstale":"2024-04-23T16:44:42.047173+0000","last_undegraded":"2024-04-23T16:44:42.047173+0000","last_fullsized":"2024-04-23T16:44:42.047173+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:43:27.668793+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:43:27.668793+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:45.595521+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"585'10403","reported_seq":10616,"reported_epoch":585,"state":"active+clean+scrubbing","last_fresh":"2024-04-23T16:44:44.466991+0000","last_change":"2024-04-23T16:44:44.072140+0000","last_active":"2024-04-23T16:44:44.466991+0000","last_peered":"2024-04-23T16:44:44.466991+0000","last_clean":"2024-04-23T16:44:44.466991+0000","last_became_active":"2024-04-23T16:41:57.305944+0000","last_became_peered":"2024-04-23T16:41:57.305944+0000","last_unstale":"2024-04-23T16:44:44.466991+0000","last_undegraded":"2024-04-23T16:44:44.466991+0000","last_fullsized":"2024-04-23T16:44:44.466991+0000","mapping_epoch":540,"log_start":"547'400","ondisk_log_start":"547'400","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6718","last_scrub_stamp":"2024-04-23T16:43:27.928174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:27.928174+0000","objects_scrubbed":3599,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291307520,"num_objects":4445,"num_object_clones":0,"num_object_copies":8890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10403,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1454,"reported_epoch":583,"state":"active+clean","last_fresh":"2024-04-23T16:44:39.046119+0000","last_change":"2024-04-23T16:42:32.662620+0000","last_active":"2024-04-23T16:44:39.046119+0000","last_peered":"2024-04-23T16:44:39.046119+0000","last_clean":"2024-04-23T16:44:39.046119+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:44:39.046119+0000","last_undegraded":"2024-04-23T16:44:39.046119+0000","last_fullsized":"2024-04-23T16:44:39.046119+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:32.662476+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:42:32.662476+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"585'10696","reported_seq":10870,"reported_epoch":585,"state":"active+clean","last_fresh":"2024-04-23T16:44:44.448330+0000","last_change":"2024-04-23T16:43:25.389006+0000","last_active":"2024-04-23T16:44:44.448330+0000","last_peered":"2024-04-23T16:44:44.448330+0000","last_clean":"2024-04-23T16:44:44.448330+0000","last_became_active":"2024-04-23T16:41:57.309858+0000","last_became_peered":"2024-04-23T16:41:57.309858+0000","last_unstale":"2024-04-23T16:44:44.448330+0000","last_undegraded":"2024-04-23T16:44:44.448330+0000","last_fullsized":"2024-04-23T16:44:44.448330+0000","mapping_epoch":540,"log_start":"547'600","ondisk_log_start":"547'600","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'6776","last_scrub_stamp":"2024-04-23T16:43:25.388898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:25.388898+0000","objects_scrubbed":6743,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292945920,"num_objects":4470,"num_object_clones":0,"num_object_copies":8940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10696,"num_write_kb":485312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1482,"reported_epoch":585,"state":"active+clean","last_fresh":"2024-04-23T16:44:44.056587+0000","last_change":"2024-04-23T16:43:40.258384+0000","last_active":"2024-04-23T16:44:44.056587+0000","last_peered":"2024-04-23T16:44:44.056587+0000","last_clean":"2024-04-23T16:44:44.056587+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:44:44.056587+0000","last_undegraded":"2024-04-23T16:44:44.056587+0000","last_fullsized":"2024-04-23T16:44:44.056587+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:43:40.258213+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:43:40.258213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:07.779820+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"583'10205","reported_seq":10301,"reported_epoch":583,"state":"active+clean","last_fresh":"2024-04-23T16:44:41.531192+0000","last_change":"2024-04-23T16:41:57.310695+0000","last_active":"2024-04-23T16:44:41.531192+0000","last_peered":"2024-04-23T16:44:41.531192+0000","last_clean":"2024-04-23T16:44:41.531192+0000","last_became_active":"2024-04-23T16:41:57.310400+0000","last_became_peered":"2024-04-23T16:41:57.310400+0000","last_unstale":"2024-04-23T16:44:41.531192+0000","last_undegraded":"2024-04-23T16:44:41.531192+0000","last_fullsized":"2024-04-23T16:44:41.531192+0000","mapping_epoch":540,"log_start":"547'200","ondisk_log_start":"547'200","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294453248,"num_objects":4493,"num_object_clones":0,"num_object_copies":8986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10205,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1052,"reported_epoch":583,"state":"active+clean","last_fresh":"2024-04-23T16:44:39.066056+0000","last_change":"2024-04-23T16:44:18.555667+0000","last_active":"2024-04-23T16:44:39.066056+0000","last_peered":"2024-04-23T16:44:39.066056+0000","last_clean":"2024-04-23T16:44:39.066056+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:44:39.066056+0000","last_undegraded":"2024-04-23T16:44:39.066056+0000","last_fullsized":"2024-04-23T16:44:39.066056+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:18.555538+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:44:18.555538+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:35.829709+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"583'10161","reported_seq":10257,"reported_epoch":583,"state":"active+clean","last_fresh":"2024-04-23T16:44:40.117535+0000","last_change":"2024-04-23T16:41:57.306526+0000","last_active":"2024-04-23T16:44:40.117535+0000","last_peered":"2024-04-23T16:44:40.117535+0000","last_clean":"2024-04-23T16:44:40.117535+0000","last_became_active":"2024-04-23T16:41:57.306229+0000","last_became_peered":"2024-04-23T16:41:57.306229+0000","last_unstale":"2024-04-23T16:44:40.117535+0000","last_undegraded":"2024-04-23T16:44:40.117535+0000","last_fullsized":"2024-04-23T16:44:40.117535+0000","mapping_epoch":540,"log_start":"547'100","ondisk_log_start":"547'100","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297730048,"num_objects":4543,"num_object_clones":0,"num_object_copies":9086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10161,"num_write_kb":470528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1009,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-04-23T16:44:42.041326+0000","last_change":"2024-04-23T16:44:07.230780+0000","last_active":"2024-04-23T16:44:42.041326+0000","last_peered":"2024-04-23T16:44:42.041326+0000","last_clean":"2024-04-23T16:44:42.041326+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:44:42.041326+0000","last_undegraded":"2024-04-23T16:44:42.041326+0000","last_fullsized":"2024-04-23T16:44:42.041326+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:07.230640+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:44:07.230640+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:12.044599+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.0","version":"584'10276","reported_seq":10451,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-04-23T16:44:42.697730+0000","last_change":"2024-04-23T16:43:30.757721+0000","last_active":"2024-04-23T16:44:42.697730+0000","last_peered":"2024-04-23T16:44:42.697730+0000","last_clean":"2024-04-23T16:44:42.697730+0000","last_became_active":"2024-04-23T16:41:57.311479+0000","last_became_peered":"2024-04-23T16:41:57.311479+0000","last_unstale":"2024-04-23T16:44:42.697730+0000","last_undegraded":"2024-04-23T16:44:42.697730+0000","last_fullsized":"2024-04-23T16:44:42.697730+0000","mapping_epoch":540,"log_start":"547'200","ondisk_log_start":"547'200","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6823","last_scrub_stamp":"2024-04-23T16:43:30.757555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:30.757555+0000","objects_scrubbed":6802,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:59.054459+0000","scrub_duration":1354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289931264,"num_objects":4424,"num_object_clones":0,"num_object_copies":8848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10276,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1491,"reported_epoch":585,"state":"active+clean","last_fresh":"2024-04-23T16:44:44.054856+0000","last_change":"2024-04-23T16:44:11.349248+0000","last_active":"2024-04-23T16:44:44.054856+0000","last_peered":"2024-04-23T16:44:44.054856+0000","last_clean":"2024-04-23T16:44:44.054856+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:44:44.054856+0000","last_undegraded":"2024-04-23T16:44:44.054856+0000","last_fullsized":"2024-04-23T16:44:44.054856+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:11.348079+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:44:11.348079+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:21.183735+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.1","version":"585'10240","reported_seq":16189,"reported_epoch":585,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:44.434044+0000","last_change":"2024-04-23T16:43:08.081567+0000","last_active":"2024-04-23T16:44:44.434044+0000","last_peered":"2024-04-23T16:44:44.434044+0000","last_clean":"2024-04-23T16:42:42.976600+0000","last_became_active":"2024-04-23T16:43:08.067887+0000","last_became_peered":"2024-04-23T16:43:08.067887+0000","last_unstale":"2024-04-23T16:44:44.434044+0000","last_undegraded":"2024-04-23T16:43:07.950381+0000","last_fullsized":"2024-04-23T16:44:44.434044+0000","mapping_epoch":563,"log_start":"547'200","ondisk_log_start":"547'200","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288096256,"num_objects":4396,"num_object_clones":0,"num_object_copies":8792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10240,"num_write_kb":468352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2923,"num_bytes_recovered":191561728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":2665},{"shards":"4,6","objects":1731}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1471,"reported_epoch":583,"state":"active+clean","last_fresh":"2024-04-23T16:44:39.025489+0000","last_change":"2024-04-23T16:43:07.918302+0000","last_active":"2024-04-23T16:44:39.025489+0000","last_peered":"2024-04-23T16:44:39.025489+0000","last_clean":"2024-04-23T16:44:39.025489+0000","last_became_active":"2024-04-23T16:43:07.918042+0000","last_became_peered":"2024-04-23T16:43:07.918042+0000","last_unstale":"2024-04-23T16:44:39.025489+0000","last_undegraded":"2024-04-23T16:44:39.025489+0000","last_fullsized":"2024-04-23T16:44:39.025489+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:33.716343+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:42:33.716343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"583'10185","reported_seq":15842,"reported_epoch":583,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:40.128466+0000","last_change":"2024-04-23T16:43:08.363306+0000","last_active":"2024-04-23T16:44:40.128466+0000","last_peered":"2024-04-23T16:44:40.128466+0000","last_clean":"2024-04-23T16:42:42.950766+0000","last_became_active":"2024-04-23T16:43:08.333793+0000","last_became_peered":"2024-04-23T16:43:08.333793+0000","last_unstale":"2024-04-23T16:44:40.128466+0000","last_undegraded":"2024-04-23T16:43:07.950356+0000","last_fullsized":"2024-04-23T16:44:40.128466+0000","mapping_epoch":563,"log_start":"547'100","ondisk_log_start":"547'100","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303628288,"num_objects":4633,"num_object_clones":0,"num_object_copies":9266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10185,"num_write_kb":474176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2780,"num_bytes_recovered":182190080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":2957},{"shards":"6,7","objects":1676}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1363,"reported_epoch":583,"state":"active+clean","last_fresh":"2024-04-23T16:44:39.034197+0000","last_change":"2024-04-23T16:41:47.722866+0000","last_active":"2024-04-23T16:44:39.034197+0000","last_peered":"2024-04-23T16:44:39.034197+0000","last_clean":"2024-04-23T16:44:39.034197+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:44:39.034197+0000","last_undegraded":"2024-04-23T16:44:39.034197+0000","last_fullsized":"2024-04-23T16:44:39.034197+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:41:47.722762+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"585'10463","reported_seq":17448,"reported_epoch":585,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:44:44.428400+0000","last_change":"2024-04-23T16:43:08.136028+0000","last_active":"2024-04-23T16:44:44.428400+0000","last_peered":"2024-04-23T16:44:44.428400+0000","last_clean":"2024-04-23T16:42:42.962639+0000","last_became_active":"2024-04-23T16:43:08.109240+0000","last_became_peered":"2024-04-23T16:43:08.109240+0000","last_unstale":"2024-04-23T16:44:44.428400+0000","last_undegraded":"2024-04-23T16:43:07.955254+0000","last_fullsized":"2024-04-23T16:44:44.428400+0000","mapping_epoch":563,"log_start":"547'400","ondisk_log_start":"547'400","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292356096,"num_objects":4461,"num_object_clones":0,"num_object_copies":8922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10463,"num_write_kb":477568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3440,"num_bytes_recovered":225443840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":2216},{"shards":"4,11","objects":2245}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1285,"reported_epoch":583,"state":"active+clean","last_fresh":"2024-04-23T16:44:39.025465+0000","last_change":"2024-04-23T16:41:49.835121+0000","last_active":"2024-04-23T16:44:39.025465+0000","last_peered":"2024-04-23T16:44:39.025465+0000","last_clean":"2024-04-23T16:44:39.025465+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:44:39.025465+0000","last_undegraded":"2024-04-23T16:44:39.025465+0000","last_fullsized":"2024-04-23T16:44:39.025465+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:41:49.834374+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"585'10326","reported_seq":10499,"reported_epoch":585,"state":"active+clean","last_fresh":"2024-04-23T16:44:44.448326+0000","last_change":"2024-04-23T16:43:26.589353+0000","last_active":"2024-04-23T16:44:44.448326+0000","last_peered":"2024-04-23T16:44:44.448326+0000","last_clean":"2024-04-23T16:44:44.448326+0000","last_became_active":"2024-04-23T16:41:57.309349+0000","last_became_peered":"2024-04-23T16:41:57.309349+0000","last_unstale":"2024-04-23T16:44:44.448326+0000","last_undegraded":"2024-04-23T16:44:44.448326+0000","last_fullsized":"2024-04-23T16:44:44.448326+0000","mapping_epoch":540,"log_start":"547'300","ondisk_log_start":"547'300","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6582","last_scrub_stamp":"2024-04-23T16:43:26.589248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:26.589248+0000","objects_scrubbed":6550,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282984448,"num_objects":4318,"num_object_clones":0,"num_object_copies":8636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10326,"num_write_kb":468608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1454,"reported_epoch":583,"state":"active+clean","last_fresh":"2024-04-23T16:44:39.046099+0000","last_change":"2024-04-23T16:42:33.628340+0000","last_active":"2024-04-23T16:44:39.046099+0000","last_peered":"2024-04-23T16:44:39.046099+0000","last_clean":"2024-04-23T16:44:39.046099+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:44:39.046099+0000","last_undegraded":"2024-04-23T16:44:39.046099+0000","last_fullsized":"2024-04-23T16:44:39.046099+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:33.628217+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:42:33.628217+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4718198808,"num_objects":71995,"num_object_clones":0,"num_object_copies":143990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10622,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71995,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165363,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12073,"num_bytes_recovered":791216128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10136854528,"data_stored":10136846384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160763,"ondisk_log_size":160763,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1474560,"data_stored":1460352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056993,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1765972,"kb_used_data":1143372,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":91979180,"statfs":{"total":95995035648,"available":94186680320,"internally_reserved":0,"allocated":1170812928,"data_stored":1170473512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":637532612},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351335,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612973,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1663668,"kb_used_data":1141028,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":92081484,"statfs":{"total":95995035648,"available":94291439616,"internally_reserved":0,"allocated":1168412672,"data_stored":1168061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":535165376},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841905,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":942572,"kb_used_data":572640,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92802580,"statfs":{"total":95995035648,"available":95029841920,"internally_reserved":0,"allocated":586383360,"data_stored":586034388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168950,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":973532,"kb_used_data":396620,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92771620,"statfs":{"total":95995035648,"available":94998138880,"internally_reserved":0,"allocated":406138880,"data_stored":405799464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":590608842},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463291,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1797728,"kb_used_data":1293008,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":91947424,"statfs":{"total":95995035648,"available":94154162176,"internally_reserved":0,"allocated":1324040192,"data_stored":1323696704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":516684230},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137432,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":828384,"kb_used_data":597648,"kb_used_omap":231,"kb_used_meta":230360,"kb_avail":92916768,"statfs":{"total":95995035648,"available":95146770432,"internally_reserved":0,"allocated":611991552,"data_stored":611648064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":235889552},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738778,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1713460,"kb_used_data":1136676,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92031692,"statfs":{"total":95995035648,"available":94240452608,"internally_reserved":0,"allocated":1163956224,"data_stored":1163604800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269448,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921156,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2121136,"kb_used_data":1408292,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":91624016,"statfs":{"total":95995035648,"available":93822992384,"internally_reserved":0,"allocated":1442091008,"data_stored":1441739584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,2,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757631,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1515052,"kb_used_data":870240,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":92230100,"statfs":{"total":95995035648,"available":94443622400,"internally_reserved":0,"allocated":891125760,"data_stored":890776788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1158086656,"data_stored":1158086656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":608374784,"data_stored":608370712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1411842048,"data_stored":1411842048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1439498240,"data_stored":1439498240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":889454592,"data_stored":889454592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1318785024,"data_stored":1318780952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":404488192,"data_stored":404488192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":582287360,"data_stored":582287360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1163984896,"data_stored":1163984896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1160052736,"data_stored":1160052736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:44:52.515 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.c is not active+clean 2024-04-23T16:44:52.515 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.c', 'version': "583'10226", 'reported_seq': 16188, 'reported_epoch': 583, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:44:40.139328+0000', 'last_change': '2024-04-23T16:43:08.145899+0000', 'last_active': '2024-04-23T16:44:40.139328+0000', 'last_peered': '2024-04-23T16:44:40.139328+0000', 'last_clean': '2024-04-23T16:42:42.970788+0000', 'last_became_active': '2024-04-23T16:43:08.106076+0000', 'last_became_peered': '2024-04-23T16:43:08.106076+0000', 'last_unstale': '2024-04-23T16:44:40.139328+0000', 'last_undegraded': '2024-04-23T16:43:07.951894+0000', 'last_fullsized': '2024-04-23T16:44:40.139328+0000', 'mapping_epoch': 563, 'log_start': "547'200", 'ondisk_log_start': "547'200", 'created': 540, 'last_epoch_clean': 541, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_clean_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 296222720, 'num_objects': 4520, 'num_object_clones': 0, 'num_object_copies': 9040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2784, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4520, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10226, 'num_write_kb': 471872, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2930, 'num_bytes_recovered': 192020480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 11], 'acting': [6, 11], 'avail_no_missing': ['6'], 'object_location_counts': [{'shards': '6', 'objects': 2784}, {'shards': '6,11', 'objects': 1736}], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-23T16:44:52.515 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.f is not active+clean 2024-04-23T16:44:52.515 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.f', 'version': "585'10403", 'reported_seq': 10616, 'reported_epoch': 585, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-23T16:44:44.466991+0000', 'last_change': '2024-04-23T16:44:44.072140+0000', 'last_active': '2024-04-23T16:44:44.466991+0000', 'last_peered': '2024-04-23T16:44:44.466991+0000', 'last_clean': '2024-04-23T16:44:44.466991+0000', 'last_became_active': '2024-04-23T16:41:57.305944+0000', 'last_became_peered': '2024-04-23T16:41:57.305944+0000', 'last_unstale': '2024-04-23T16:44:44.466991+0000', 'last_undegraded': '2024-04-23T16:44:44.466991+0000', 'last_fullsized': '2024-04-23T16:44:44.466991+0000', 'mapping_epoch': 540, 'log_start': "547'400", 'ondisk_log_start': "547'400", 'created': 540, 'last_epoch_clean': 541, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "569'6718", 'last_scrub_stamp': '2024-04-23T16:43:27.928174+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_clean_scrub_stamp': '2024-04-23T16:43:27.928174+0000', 'objects_scrubbed': 3599, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'scrubbing for 0s', 'scrub_duration': 750, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 291307520, 'num_objects': 4445, 'num_object_clones': 0, 'num_object_copies': 8890, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4445, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10403, 'num_write_kb': 475136, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 8], 'acting': [0, 8], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-23T16:44:52.515 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.1 is not active+clean 2024-04-23T16:44:52.515 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.1', 'version': "585'10240", 'reported_seq': 16189, 'reported_epoch': 585, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:44:44.434044+0000', 'last_change': '2024-04-23T16:43:08.081567+0000', 'last_active': '2024-04-23T16:44:44.434044+0000', 'last_peered': '2024-04-23T16:44:44.434044+0000', 'last_clean': '2024-04-23T16:42:42.976600+0000', 'last_became_active': '2024-04-23T16:43:08.067887+0000', 'last_became_peered': '2024-04-23T16:43:08.067887+0000', 'last_unstale': '2024-04-23T16:44:44.434044+0000', 'last_undegraded': '2024-04-23T16:43:07.950381+0000', 'last_fullsized': '2024-04-23T16:44:44.434044+0000', 'mapping_epoch': 563, 'log_start': "547'200", 'ondisk_log_start': "547'200", 'created': 540, 'last_epoch_clean': 541, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_clean_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 288096256, 'num_objects': 4396, 'num_object_clones': 0, 'num_object_copies': 8792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2665, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4396, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10240, 'num_write_kb': 468352, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2923, 'num_bytes_recovered': 191561728, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': ['4'], 'object_location_counts': [{'shards': '4', 'objects': 2665}, {'shards': '4,6', 'objects': 1731}], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-23T16:44:52.515 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.2 is not active+clean 2024-04-23T16:44:52.515 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.2', 'version': "583'10185", 'reported_seq': 15842, 'reported_epoch': 583, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:44:40.128466+0000', 'last_change': '2024-04-23T16:43:08.363306+0000', 'last_active': '2024-04-23T16:44:40.128466+0000', 'last_peered': '2024-04-23T16:44:40.128466+0000', 'last_clean': '2024-04-23T16:42:42.950766+0000', 'last_became_active': '2024-04-23T16:43:08.333793+0000', 'last_became_peered': '2024-04-23T16:43:08.333793+0000', 'last_unstale': '2024-04-23T16:44:40.128466+0000', 'last_undegraded': '2024-04-23T16:43:07.950356+0000', 'last_fullsized': '2024-04-23T16:44:40.128466+0000', 'mapping_epoch': 563, 'log_start': "547'100", 'ondisk_log_start': "547'100", 'created': 540, 'last_epoch_clean': 541, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_clean_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 303628288, 'num_objects': 4633, 'num_object_clones': 0, 'num_object_copies': 9266, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2957, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4633, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10185, 'num_write_kb': 474176, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2780, 'num_bytes_recovered': 182190080, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': ['6'], 'object_location_counts': [{'shards': '6', 'objects': 2957}, {'shards': '6,7', 'objects': 1676}], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-23T16:44:52.516 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.3 is not active+clean 2024-04-23T16:44:52.516 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.3', 'version': "585'10463", 'reported_seq': 17448, 'reported_epoch': 585, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:44:44.428400+0000', 'last_change': '2024-04-23T16:43:08.136028+0000', 'last_active': '2024-04-23T16:44:44.428400+0000', 'last_peered': '2024-04-23T16:44:44.428400+0000', 'last_clean': '2024-04-23T16:42:42.962639+0000', 'last_became_active': '2024-04-23T16:43:08.109240+0000', 'last_became_peered': '2024-04-23T16:43:08.109240+0000', 'last_unstale': '2024-04-23T16:44:44.428400+0000', 'last_undegraded': '2024-04-23T16:43:07.955254+0000', 'last_fullsized': '2024-04-23T16:44:44.428400+0000', 'mapping_epoch': 563, 'log_start': "547'400", 'ondisk_log_start': "547'400", 'created': 540, 'last_epoch_clean': 541, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_clean_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 292356096, 'num_objects': 4461, 'num_object_clones': 0, 'num_object_copies': 8922, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2216, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4461, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10463, 'num_write_kb': 477568, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3440, 'num_bytes_recovered': 225443840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 11], 'acting': [4, 11], 'avail_no_missing': ['4'], 'object_location_counts': [{'shards': '4', 'objects': 2216}, {'shards': '4,11', 'objects': 2245}], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-23T16:44:52.516 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-23T16:44:52.984 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:44:52.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:52.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:44:52.991 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:44:53.090 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:44:53.163 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:53.250 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:44:53.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:53.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:44:53.310 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:53 smithi103 ceph-mon[39629]: 7.e scrub ok 2024-04-23T16:44:53.310 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:53 smithi103 ceph-mon[39629]: pgmap v1647: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.0 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 670 op/s; 9192/132688 objects degraded (6.928%); 10 MiB/s, 162 objects/s recovering 2024-04-23T16:44:53.359 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:44:53.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:53 smithi188 ceph-mon[44101]: 7.e scrub ok 2024-04-23T16:44:53.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:53 smithi188 ceph-mon[44101]: pgmap v1647: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.0 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 670 op/s; 9192/132688 objects degraded (6.928%); 10 MiB/s, 162 objects/s recovering 2024-04-23T16:44:53.391 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:53.469 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:44:53.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:53.476 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:44:53.577 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:44:53.607 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:53.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:53 smithi148 ceph-mon[44520]: 7.e scrub ok 2024-04-23T16:44:53.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:53 smithi148 ceph-mon[44520]: pgmap v1647: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 4.0 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 670 op/s; 9192/132688 objects degraded (6.928%); 10 MiB/s, 162 objects/s recovering 2024-04-23T16:44:54.156 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:44:54.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:54.161 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:44:54.263 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:44:54.295 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:54.463 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:44:54.778 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:44:54.778 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:44:55.246 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:44:55.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:55 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:44:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:55 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:44:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:55 smithi103 ceph-mon[39629]: osdmap e589: 12 total, 11 up, 9 in 2024-04-23T16:44:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:55 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 9192/132688 objects degraded (6.928%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:55 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3017516392' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:44:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:55 smithi103 ceph-mon[39629]: pgmap v1649: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.9 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 899 op/s; 8670/128660 objects degraded (6.739%); 16 MiB/s, 249 objects/s recovering 2024-04-23T16:44:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:55 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:44:55.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:55 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:44:55.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:55 smithi188 ceph-mon[44101]: osdmap e589: 12 total, 11 up, 9 in 2024-04-23T16:44:55.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:55 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 9192/132688 objects degraded (6.928%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:55.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:55 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3017516392' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:44:55.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:55 smithi188 ceph-mon[44101]: pgmap v1649: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.9 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 899 op/s; 8670/128660 objects degraded (6.739%); 16 MiB/s, 249 objects/s recovering 2024-04-23T16:44:55.558 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-23T16:44:55.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:55 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:44:55.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:55 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:44:55.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:55 smithi148 ceph-mon[44520]: osdmap e589: 12 total, 11 up, 9 in 2024-04-23T16:44:55.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:55 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 9192/132688 objects degraded (6.928%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:44:55.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:55 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3017516392' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:44:55.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:55 smithi148 ceph-mon[44520]: pgmap v1649: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.9 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 899 op/s; 8670/128660 objects degraded (6.739%); 16 MiB/s, 249 objects/s recovering 2024-04-23T16:44:55.855 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:44:55.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:55.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:44:55.941 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:44:55.960 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:44:55.991 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:56.140 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:44:56.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:56.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:44:56.244 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:44:56.280 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:56.305 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:44:56.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:56.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:44:56.410 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:44:56.444 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:56 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:44:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:56 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3017516392' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:44:56.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:56 smithi188 ceph-mon[44101]: osdmap e590: 12 total, 11 up, 9 in 2024-04-23T16:44:56.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:56 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:44:56.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:56 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3017516392' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:44:56.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:56 smithi103 ceph-mon[39629]: osdmap e590: 12 total, 11 up, 9 in 2024-04-23T16:44:56.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:56 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:44:56.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:56 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3017516392' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:44:56.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:56 smithi148 ceph-mon[44520]: osdmap e590: 12 total, 11 up, 9 in 2024-04-23T16:44:56.903 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:44:56.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:56.906 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:44:57.007 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:44:57.038 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:57.246 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:44:57.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:57.256 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:44:57.357 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:44:57.392 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:57.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:57 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:44:57.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:57 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:44:57.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:57 smithi103 ceph-mon[39629]: osdmap e591: 12 total, 11 up, 9 in 2024-04-23T16:44:57.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:57 smithi103 ceph-mon[39629]: pgmap v1652: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.8 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 734 op/s; 8140/125188 objects degraded (6.502%); 11 MiB/s, 182 objects/s recovering 2024-04-23T16:44:57.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:57 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:44:57.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:57 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:44:57.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:57 smithi148 ceph-mon[44520]: osdmap e591: 12 total, 11 up, 9 in 2024-04-23T16:44:57.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:57 smithi148 ceph-mon[44520]: pgmap v1652: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.8 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 734 op/s; 8140/125188 objects degraded (6.502%); 11 MiB/s, 182 objects/s recovering 2024-04-23T16:44:57.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:57 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:44:57.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:57 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:44:57.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:57 smithi188 ceph-mon[44101]: osdmap e591: 12 total, 11 up, 9 in 2024-04-23T16:44:57.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:57 smithi188 ceph-mon[44101]: pgmap v1652: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.8 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 734 op/s; 8140/125188 objects degraded (6.502%); 11 MiB/s, 182 objects/s recovering 2024-04-23T16:44:57.766 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:44:57.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:57.774 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:44:57.875 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:44:57.910 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:58.048 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:44:58.074 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:44:58.093 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:44:58.118 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:44:58.139 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:44:58.164 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:44:58.171 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:44:58.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:58.175 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:44:58.193 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:44:58.221 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:44:58.244 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:44:58.269 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:44:58.276 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:44:58.294 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:44:58.295 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:44:58.295 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:44:58.295 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:44:58.295 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:44:58.295 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:44:58.295 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:44:58.295 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:44:58.295 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:44:58.295 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:44:58.295 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:44:58.296 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:44:58.296 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:44:58.318 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:58.423 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:44:58.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:58.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:44:58.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:58 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3626250386' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:44:58.486 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:44:58.486 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-04-23T16:44:58.486 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-04-23T16:44:58.486 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [] 2024-04-23T16:44:58.486 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:44:58.534 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:44:58.565 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:58 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3626250386' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:44:58.570 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:58.594 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:44:58.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:58.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:44:58.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:58 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3626250386' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:44:58.658 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:44:58.702 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:44:58.734 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:58.988 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:44:58.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:58.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:44:59.093 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:44:59.126 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:59.283 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.283 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.284 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.284 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.284 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.284 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.284 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.284 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.284 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.284 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.284 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:59 smithi103 ceph-mon[39629]: 7.9 scrub starts 2024-04-23T16:44:59.284 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.284 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:44:59 smithi103 ceph-mon[39629]: pgmap v1653: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.8 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 761 op/s; 8140/123548 objects degraded (6.589%); 11 MiB/s, 175 objects/s recovering 2024-04-23T16:44:59.321 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:44:59.321 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8140/123548 objects degraded (6.589%), 4 pgs degraded","count":4},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2081,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":591,"num_osds":12,"num_up_osds":11,"osd_up_since":1713890562,"num_in_osds":9,"osd_in_since":1713890586,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":23},{"state_name":"active+recovering+degraded","count":4}],"num_pgs":27,"num_pools":2,"num_objects":61774,"data_bytes":4048814648,"bytes_used":12534706176,"bytes_avail":851420614656,"bytes_total":863955320832,"degraded_objects":8140,"degraded_total":123548,"degraded_ratio":0.065885323922685923,"recovering_objects_per_sec":175,"recovering_bytes_per_sec":11488484,"recovering_keys_per_sec":0,"num_objects_recovered":1052,"num_bytes_recovered":68943872,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":761},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.103:8443/"}},"servicemap":{"epoch":46,"modified":"2024-04-23T16:44:02.858545+0000","services":{}},"progress_events":{}} 2024-04-23T16:44:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:59 smithi188 ceph-mon[44101]: 7.9 scrub starts 2024-04-23T16:44:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:44:59 smithi188 ceph-mon[44101]: pgmap v1653: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.8 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 761 op/s; 8140/123548 objects degraded (6.589%); 11 MiB/s, 175 objects/s recovering 2024-04-23T16:44:59.485 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:44:59.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:59.490 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:44:59.520 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-23T16:44:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:44:59.537 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:44:59.593 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:44:59.620 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:59.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:59 smithi148 ceph-mon[44520]: 7.9 scrub starts 2024-04-23T16:44:59.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:44:59.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:44:59 smithi148 ceph-mon[44520]: pgmap v1653: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.8 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 761 op/s; 8140/123548 objects degraded (6.589%); 11 MiB/s, 175 objects/s recovering 2024-04-23T16:44:59.769 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:44:59.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:59.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:44:59.803 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:44:59.872 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:44:59.906 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:44:59.931 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:44:59.968 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:44:59.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:44:59.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:45:00.079 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:45:00.107 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:00.127 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:45:00.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:00.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:45:00.231 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:45:00.259 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:00.288 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:45:00.291 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:00 smithi148 ceph-mon[44520]: 7.9 scrub ok 2024-04-23T16:45:00.291 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:00 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/41059063' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:45:00.291 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:00 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 8140/123548 objects degraded (6.589%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:00.291 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:00 smithi148 ceph-mon[44520]: 7.0 scrub starts 2024-04-23T16:45:00.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:00.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:45:00.401 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:00.447 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:00 smithi188 ceph-mon[44101]: 7.9 scrub ok 2024-04-23T16:45:00.447 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:00 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/41059063' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:45:00.447 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:00 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 8140/123548 objects degraded (6.589%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:00.448 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:00 smithi188 ceph-mon[44101]: 7.0 scrub starts 2024-04-23T16:45:00.471 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:00 smithi103 ceph-mon[39629]: 7.9 scrub ok 2024-04-23T16:45:00.472 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:00 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/41059063' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:45:00.472 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:00 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 8140/123548 objects degraded (6.589%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:00.472 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:00 smithi103 ceph-mon[39629]: 7.0 scrub starts 2024-04-23T16:45:00.490 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:00.531 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:00.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:00.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:00.636 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:45:00.715 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:00.749 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:45:00.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:00.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:45:00.855 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:45:00.892 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:01.283 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:45:01.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:01.291 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:45:01.392 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:45:01.421 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:01.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:01 smithi188 ceph-mon[44101]: 7.0 scrub ok 2024-04-23T16:45:01.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:01 smithi188 ceph-mon[44101]: pgmap v1654: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.6 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s; 7104/117892 objects degraded (6.026%); 20 MiB/s, 314 objects/s recovering 2024-04-23T16:45:01.538 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:45:01.563 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:01 smithi103 ceph-mon[39629]: 7.0 scrub ok 2024-04-23T16:45:01.563 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:01 smithi103 ceph-mon[39629]: pgmap v1654: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.6 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s; 7104/117892 objects degraded (6.026%); 20 MiB/s, 314 objects/s recovering 2024-04-23T16:45:01.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:01 smithi148 ceph-mon[44520]: 7.0 scrub ok 2024-04-23T16:45:01.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:01 smithi148 ceph-mon[44520]: pgmap v1654: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.6 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s; 7104/117892 objects degraded (6.026%); 20 MiB/s, 314 objects/s recovering 2024-04-23T16:45:01.823 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:45:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:45:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-04-23T16:45:01.823 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:45:01.824 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-23T16:45:01.824 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:45:02.488 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:45:02.496 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:45:02.500 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:45:02.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:02.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:45:02.597 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-23T16:45:02.608 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:45:02.675 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:02.702 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:45:02.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:02.705 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:45:02.769 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:45:02.779 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:02 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2184261519' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:45:02.806 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:45:02.861 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:02.908 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:45:02.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:02.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:45:02.973 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:45:02.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:02 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2184261519' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:45:03.014 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:45:03.044 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:03.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:02 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2184261519' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:45:03.494 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:45:03.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:03.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:45:03.598 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:45:03.631 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:04.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:03 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:45:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:03 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2184261519' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:45:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:03 smithi103 ceph-mon[39629]: osdmap e592: 12 total, 11 up, 9 in 2024-04-23T16:45:04.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:03 smithi103 ceph-mon[39629]: pgmap v1656: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.6 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 672 op/s; 7104/116400 objects degraded (6.103%); 8.5 MiB/s, 135 objects/s recovering 2024-04-23T16:45:04.132 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:03 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:45:04.132 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:03 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2184261519' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:45:04.132 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:03 smithi148 ceph-mon[44520]: osdmap e592: 12 total, 11 up, 9 in 2024-04-23T16:45:04.132 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:03 smithi148 ceph-mon[44520]: pgmap v1656: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.6 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 672 op/s; 7104/116400 objects degraded (6.103%); 8.5 MiB/s, 135 objects/s recovering 2024-04-23T16:45:04.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:03 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:45:04.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:03 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2184261519' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:45:04.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:03 smithi188 ceph-mon[44101]: osdmap e592: 12 total, 11 up, 9 in 2024-04-23T16:45:04.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:03 smithi188 ceph-mon[44101]: pgmap v1656: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.6 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 672 op/s; 7104/116400 objects degraded (6.103%); 8.5 MiB/s, 135 objects/s recovering 2024-04-23T16:45:04.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:04 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 7104/116400 objects degraded (6.103%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:04.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:04 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 7104/116400 objects degraded (6.103%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:04.991 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:45:05.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:05.003 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:45:05.104 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:05.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:04 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 7104/116400 objects degraded (6.103%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:05.158 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:05.196 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:05.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:05.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:05.303 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:45:05.355 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:05.376 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:45:05.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:05.380 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:45:05.481 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:45:05.529 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:05.667 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:45:05.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:05.673 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:45:05.681 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:45:05.681 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:45:05.681 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:45:05.681 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:45:05.682 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:45:05.773 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:45:05.806 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:05.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:05 smithi188 ceph-mon[44101]: pgmap v1657: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.4 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 769 op/s; 6540/112468 objects degraded (5.815%); 12 MiB/s, 186 objects/s recovering 2024-04-23T16:45:06.057 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:45:06.059 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:05 smithi103 ceph-mon[39629]: pgmap v1657: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.4 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 769 op/s; 6540/112468 objects degraded (5.815%); 12 MiB/s, 186 objects/s recovering 2024-04-23T16:45:06.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:06.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:45:06.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:05 smithi148 ceph-mon[44520]: pgmap v1657: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.4 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 769 op/s; 6540/112468 objects degraded (5.815%); 12 MiB/s, 186 objects/s recovering 2024-04-23T16:45:06.161 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:45:06.191 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:45:06.250 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:06.287 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:45:06.289 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:45:06.297 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:45:06.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:06.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:45:06.403 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:45:06.436 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:06.465 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:45:06.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:06.473 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:45:06.505 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-23T16:45:06.574 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:45:06.601 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:06.640 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:45:06.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:06.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:45:06.745 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:45:06.780 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:06.808 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:45:06.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:06.813 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:45:06.913 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:45:06.914 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:45:06.953 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:07.213 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:45:07.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:07.216 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:45:07.317 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:45:07.346 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:07.509 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:45:07.544 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1657,"stamp":"2024-04-23T16:45:04.871027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3685745208,"num_objects":56234,"num_object_clones":0,"num_object_copies":112468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181126,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16157,"num_bytes_recovered":1059136576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162355,"ondisk_log_size":162355,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":11666728,"kb_used_data":6800916,"kb_used_omap":243,"kb_used_meta":4865292,"kb_avail":832039640,"statfs":{"total":863955320832,"available":852008591360,"internally_reserved":0,"allocated":6964137984,"data_stored":6960611488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":249416,"internal_metadata":4982059448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,6,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-433782784,"num_objects":-6619,"num_object_clones":0,"num_object_copies":-13238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6619,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1600,"num_bytes_recovered":104857600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.598248"},"pg_stats":[{"pgid":"7.d","version":"591'11400","reported_seq":11513,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-04-23T16:45:00.161329+0000","last_change":"2024-04-23T16:41:57.310289+0000","last_active":"2024-04-23T16:45:00.161329+0000","last_peered":"2024-04-23T16:45:00.161329+0000","last_clean":"2024-04-23T16:45:00.161329+0000","last_became_active":"2024-04-23T16:41:57.310025+0000","last_became_peered":"2024-04-23T16:41:57.310025+0000","last_unstale":"2024-04-23T16:45:00.161329+0000","last_undegraded":"2024-04-23T16:45:00.161329+0000","last_fullsized":"2024-04-23T16:45:00.161329+0000","mapping_epoch":540,"log_start":"548'1300","ondisk_log_start":"548'1300","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240451608,"num_objects":3670,"num_object_clones":0,"num_object_copies":7340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3670,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11400,"num_write_kb":482177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"591'11502","reported_seq":11691,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-04-23T16:45:00.889789+0000","last_change":"2024-04-23T16:43:29.012370+0000","last_active":"2024-04-23T16:45:00.889789+0000","last_peered":"2024-04-23T16:45:00.889789+0000","last_clean":"2024-04-23T16:45:00.889789+0000","last_became_active":"2024-04-23T16:41:57.310754+0000","last_became_peered":"2024-04-23T16:41:57.310754+0000","last_unstale":"2024-04-23T16:45:00.889789+0000","last_undegraded":"2024-04-23T16:45:00.889789+0000","last_fullsized":"2024-04-23T16:45:00.889789+0000","mapping_epoch":540,"log_start":"548'1500","ondisk_log_start":"548'1500","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6945","last_scrub_stamp":"2024-04-23T16:43:29.012285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:29.012285+0000","objects_scrubbed":6912,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238682112,"num_objects":3642,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11502,"num_write_kb":484608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"592'11353","reported_seq":11594,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:03.597462+0000","last_change":"2024-04-23T16:44:45.745602+0000","last_active":"2024-04-23T16:45:03.597462+0000","last_peered":"2024-04-23T16:45:03.597462+0000","last_clean":"2024-04-23T16:45:03.597462+0000","last_became_active":"2024-04-23T16:41:57.310189+0000","last_became_peered":"2024-04-23T16:41:57.310189+0000","last_unstale":"2024-04-23T16:45:03.597462+0000","last_undegraded":"2024-04-23T16:45:03.597462+0000","last_fullsized":"2024-04-23T16:45:03.597462+0000","mapping_epoch":540,"log_start":"548'1300","ondisk_log_start":"548'1300","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'10439","last_scrub_stamp":"2024-04-23T16:44:45.745464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:45.745464+0000","objects_scrubbed":4477,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:10.632547+0000","scrub_duration":680,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232587264,"num_objects":3549,"num_object_clones":0,"num_object_copies":7098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11353,"num_write_kb":476864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"592'11428","reported_seq":11661,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:04.437187+0000","last_change":"2024-04-23T16:44:58.743309+0000","last_active":"2024-04-23T16:45:04.437187+0000","last_peered":"2024-04-23T16:45:04.437187+0000","last_clean":"2024-04-23T16:45:04.437187+0000","last_became_active":"2024-04-23T16:41:57.309569+0000","last_became_peered":"2024-04-23T16:41:57.309569+0000","last_unstale":"2024-04-23T16:45:04.437187+0000","last_undegraded":"2024-04-23T16:45:04.437187+0000","last_fullsized":"2024-04-23T16:45:04.437187+0000","mapping_epoch":540,"log_start":"548'1400","ondisk_log_start":"548'1400","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"591'11141","last_scrub_stamp":"2024-04-23T16:44:58.743181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:58.743181+0000","objects_scrubbed":3762,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:00.499252+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227016704,"num_objects":3464,"num_object_clones":0,"num_object_copies":6928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11428,"num_write_kb":476544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"591'10998","reported_seq":11110,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-04-23T16:45:00.731031+0000","last_change":"2024-04-23T16:41:57.310144+0000","last_active":"2024-04-23T16:45:00.731031+0000","last_peered":"2024-04-23T16:45:00.731031+0000","last_clean":"2024-04-23T16:45:00.731031+0000","last_became_active":"2024-04-23T16:41:57.309841+0000","last_became_peered":"2024-04-23T16:41:57.309841+0000","last_unstale":"2024-04-23T16:45:00.731031+0000","last_undegraded":"2024-04-23T16:45:00.731031+0000","last_fullsized":"2024-04-23T16:45:00.731031+0000","mapping_epoch":540,"log_start":"547'900","ondisk_log_start":"547'900","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232128512,"num_objects":3542,"num_object_clones":0,"num_object_copies":7084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10998,"num_write_kb":465280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"591'11191","reported_seq":19029,"reported_epoch":591,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:45:00.214245+0000","last_change":"2024-04-23T16:43:08.145899+0000","last_active":"2024-04-23T16:45:00.214245+0000","last_peered":"2024-04-23T16:45:00.214245+0000","last_clean":"2024-04-23T16:42:42.970788+0000","last_became_active":"2024-04-23T16:43:08.106076+0000","last_became_peered":"2024-04-23T16:43:08.106076+0000","last_unstale":"2024-04-23T16:45:00.214245+0000","last_undegraded":"2024-04-23T16:43:07.951894+0000","last_fullsized":"2024-04-23T16:45:00.214245+0000","mapping_epoch":563,"log_start":"548'1100","ondisk_log_start":"548'1100","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232980480,"num_objects":3555,"num_object_clones":0,"num_object_copies":7110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1854,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11191,"num_write_kb":471872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3860,"num_bytes_recovered":252968960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":1854},{"shards":"6,11","objects":1701}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1497,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:02.771247+0000","last_change":"2024-04-23T16:44:09.435333+0000","last_active":"2024-04-23T16:45:02.771247+0000","last_peered":"2024-04-23T16:45:02.771247+0000","last_clean":"2024-04-23T16:45:02.771247+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:45:02.771247+0000","last_undegraded":"2024-04-23T16:45:02.771247+0000","last_fullsized":"2024-04-23T16:45:02.771247+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:09.435212+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:44:09.435212+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:17.709965+0000","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"592'11423","reported_seq":11662,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:03.554356+0000","last_change":"2024-04-23T16:44:52.727725+0000","last_active":"2024-04-23T16:45:03.554356+0000","last_peered":"2024-04-23T16:45:03.554356+0000","last_clean":"2024-04-23T16:45:03.554356+0000","last_became_active":"2024-04-23T16:41:57.311753+0000","last_became_peered":"2024-04-23T16:41:57.311753+0000","last_unstale":"2024-04-23T16:45:03.554356+0000","last_undegraded":"2024-04-23T16:45:03.554356+0000","last_fullsized":"2024-04-23T16:45:03.554356+0000","mapping_epoch":540,"log_start":"548'1400","ondisk_log_start":"548'1400","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"588'10838","last_scrub_stamp":"2024-04-23T16:44:52.727632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:52.727632+0000","objects_scrubbed":4109,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:52.880867+0000","scrub_duration":730,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229572608,"num_objects":3503,"num_object_clones":0,"num_object_copies":7006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11423,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1501,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-04-23T16:44:56.289686+0000","last_change":"2024-04-23T16:44:45.850797+0000","last_active":"2024-04-23T16:44:56.289686+0000","last_peered":"2024-04-23T16:44:56.289686+0000","last_clean":"2024-04-23T16:44:56.289686+0000","last_became_active":"2024-04-23T16:43:07.962092+0000","last_became_peered":"2024-04-23T16:43:07.962092+0000","last_unstale":"2024-04-23T16:44:56.289686+0000","last_undegraded":"2024-04-23T16:44:56.289686+0000","last_fullsized":"2024-04-23T16:44:56.289686+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:45.850682+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:44:45.850682+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:12.889710+0000","scrub_duration":234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"592'11425","reported_seq":11666,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:04.428641+0000","last_change":"2024-04-23T16:44:44.578218+0000","last_active":"2024-04-23T16:45:04.428641+0000","last_peered":"2024-04-23T16:45:04.428641+0000","last_clean":"2024-04-23T16:45:04.428641+0000","last_became_active":"2024-04-23T16:41:57.305944+0000","last_became_peered":"2024-04-23T16:41:57.305944+0000","last_unstale":"2024-04-23T16:45:04.428641+0000","last_undegraded":"2024-04-23T16:45:04.428641+0000","last_fullsized":"2024-04-23T16:45:04.428641+0000","mapping_epoch":540,"log_start":"548'1400","ondisk_log_start":"548'1400","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'10405","last_scrub_stamp":"2024-04-23T16:44:44.578065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:44.578065+0000","objects_scrubbed":4454,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:01.634216+0000","scrub_duration":505,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224329728,"num_objects":3423,"num_object_clones":0,"num_object_copies":6846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11425,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1470,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-04-23T16:44:56.301844+0000","last_change":"2024-04-23T16:42:32.662620+0000","last_active":"2024-04-23T16:44:56.301844+0000","last_peered":"2024-04-23T16:44:56.301844+0000","last_clean":"2024-04-23T16:44:56.301844+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:44:56.301844+0000","last_undegraded":"2024-04-23T16:44:56.301844+0000","last_fullsized":"2024-04-23T16:44:56.301844+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:32.662476+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:42:32.662476+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"592'11759","reported_seq":11998,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:04.458173+0000","last_change":"2024-04-23T16:44:46.996868+0000","last_active":"2024-04-23T16:45:04.458173+0000","last_peered":"2024-04-23T16:45:04.458173+0000","last_clean":"2024-04-23T16:45:04.458173+0000","last_became_active":"2024-04-23T16:41:57.309858+0000","last_became_peered":"2024-04-23T16:41:57.309858+0000","last_unstale":"2024-04-23T16:45:04.458173+0000","last_undegraded":"2024-04-23T16:45:04.458173+0000","last_fullsized":"2024-04-23T16:45:04.458173+0000","mapping_epoch":540,"log_start":"548'1700","ondisk_log_start":"548'1700","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'10834","last_scrub_stamp":"2024-04-23T16:44:46.996775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:46.996775+0000","objects_scrubbed":4359,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:53.498504+0000","scrub_duration":887,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223281152,"num_objects":3407,"num_object_clones":0,"num_object_copies":6814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11759,"num_write_kb":485312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1496,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:02.771632+0000","last_change":"2024-04-23T16:43:40.258384+0000","last_active":"2024-04-23T16:45:02.771632+0000","last_peered":"2024-04-23T16:45:02.771632+0000","last_clean":"2024-04-23T16:45:02.771632+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:45:02.771632+0000","last_undegraded":"2024-04-23T16:45:02.771632+0000","last_fullsized":"2024-04-23T16:45:02.771632+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:43:40.258213+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:43:40.258213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:07.779820+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"591'11144","reported_seq":11256,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-04-23T16:45:01.463485+0000","last_change":"2024-04-23T16:41:57.310695+0000","last_active":"2024-04-23T16:45:01.463485+0000","last_peered":"2024-04-23T16:45:01.463485+0000","last_clean":"2024-04-23T16:45:01.463485+0000","last_became_active":"2024-04-23T16:41:57.310400+0000","last_became_peered":"2024-04-23T16:41:57.310400+0000","last_unstale":"2024-04-23T16:45:01.463485+0000","last_undegraded":"2024-04-23T16:45:01.463485+0000","last_fullsized":"2024-04-23T16:45:01.463485+0000","mapping_epoch":540,"log_start":"548'1100","ondisk_log_start":"548'1100","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232914944,"num_objects":3554,"num_object_clones":0,"num_object_copies":7108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11144,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1068,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-04-23T16:44:56.278585+0000","last_change":"2024-04-23T16:44:18.555667+0000","last_active":"2024-04-23T16:44:56.278585+0000","last_peered":"2024-04-23T16:44:56.278585+0000","last_clean":"2024-04-23T16:44:56.278585+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:44:56.278585+0000","last_undegraded":"2024-04-23T16:44:56.278585+0000","last_fullsized":"2024-04-23T16:44:56.278585+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:18.555538+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:44:18.555538+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:35.829709+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"591'11094","reported_seq":11206,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-04-23T16:45:00.189004+0000","last_change":"2024-04-23T16:41:57.306526+0000","last_active":"2024-04-23T16:45:00.189004+0000","last_peered":"2024-04-23T16:45:00.189004+0000","last_clean":"2024-04-23T16:45:00.189004+0000","last_became_active":"2024-04-23T16:41:57.306229+0000","last_became_peered":"2024-04-23T16:41:57.306229+0000","last_unstale":"2024-04-23T16:45:00.189004+0000","last_undegraded":"2024-04-23T16:45:00.189004+0000","last_fullsized":"2024-04-23T16:45:00.189004+0000","mapping_epoch":540,"log_start":"548'1000","ondisk_log_start":"548'1000","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236584960,"num_objects":3610,"num_object_clones":0,"num_object_copies":7220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11094,"num_write_kb":470528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1025,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:02.810482+0000","last_change":"2024-04-23T16:44:07.230780+0000","last_active":"2024-04-23T16:45:02.810482+0000","last_peered":"2024-04-23T16:45:02.810482+0000","last_clean":"2024-04-23T16:45:02.810482+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:45:02.810482+0000","last_undegraded":"2024-04-23T16:45:02.810482+0000","last_fullsized":"2024-04-23T16:45:02.810482+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:07.230640+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:44:07.230640+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:12.044599+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.0","version":"591'11257","reported_seq":11489,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-04-23T16:45:02.667272+0000","last_change":"2024-04-23T16:45:00.368292+0000","last_active":"2024-04-23T16:45:02.667272+0000","last_peered":"2024-04-23T16:45:02.667272+0000","last_clean":"2024-04-23T16:45:02.667272+0000","last_became_active":"2024-04-23T16:41:57.311479+0000","last_became_peered":"2024-04-23T16:41:57.311479+0000","last_unstale":"2024-04-23T16:45:02.667272+0000","last_undegraded":"2024-04-23T16:45:02.667272+0000","last_fullsized":"2024-04-23T16:45:02.667272+0000","mapping_epoch":540,"log_start":"548'1200","ondisk_log_start":"548'1200","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"591'11135","last_scrub_stamp":"2024-04-23T16:45:00.368005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:00.368005+0000","objects_scrubbed":3585,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:06.315750+0000","scrub_duration":778,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225640448,"num_objects":3443,"num_object_clones":0,"num_object_copies":6886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11257,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1505,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:02.771187+0000","last_change":"2024-04-23T16:44:11.349248+0000","last_active":"2024-04-23T16:45:02.771187+0000","last_peered":"2024-04-23T16:45:02.771187+0000","last_clean":"2024-04-23T16:45:02.771187+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:45:02.771187+0000","last_undegraded":"2024-04-23T16:45:02.771187+0000","last_fullsized":"2024-04-23T16:45:02.771187+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:11.348079+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:44:11.348079+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:21.183735+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.1","version":"592'11219","reported_seq":19148,"reported_epoch":592,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:45:04.555745+0000","last_change":"2024-04-23T16:43:08.081567+0000","last_active":"2024-04-23T16:45:04.555745+0000","last_peered":"2024-04-23T16:45:04.555745+0000","last_clean":"2024-04-23T16:42:42.976600+0000","last_became_active":"2024-04-23T16:43:08.067887+0000","last_became_peered":"2024-04-23T16:43:08.067887+0000","last_unstale":"2024-04-23T16:45:04.555745+0000","last_undegraded":"2024-04-23T16:43:07.950381+0000","last_fullsized":"2024-04-23T16:45:04.555745+0000","mapping_epoch":563,"log_start":"548'1200","ondisk_log_start":"548'1200","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223936512,"num_objects":3417,"num_object_clones":0,"num_object_copies":6834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1683,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11219,"num_write_kb":468352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3905,"num_bytes_recovered":255918080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":1683},{"shards":"4,6","objects":1734}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1487,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-04-23T16:44:56.272184+0000","last_change":"2024-04-23T16:43:07.918302+0000","last_active":"2024-04-23T16:44:56.272184+0000","last_peered":"2024-04-23T16:44:56.272184+0000","last_clean":"2024-04-23T16:44:56.272184+0000","last_became_active":"2024-04-23T16:43:07.918042+0000","last_became_peered":"2024-04-23T16:43:07.918042+0000","last_unstale":"2024-04-23T16:44:56.272184+0000","last_undegraded":"2024-04-23T16:44:56.272184+0000","last_fullsized":"2024-04-23T16:44:56.272184+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:33.716343+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:42:33.716343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"591'11196","reported_seq":18889,"reported_epoch":591,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:45:00.198082+0000","last_change":"2024-04-23T16:43:08.363306+0000","last_active":"2024-04-23T16:45:00.198082+0000","last_peered":"2024-04-23T16:45:00.198082+0000","last_clean":"2024-04-23T16:42:42.950766+0000","last_became_active":"2024-04-23T16:43:08.333793+0000","last_became_peered":"2024-04-23T16:43:08.333793+0000","last_unstale":"2024-04-23T16:45:00.198082+0000","last_undegraded":"2024-04-23T16:43:07.950356+0000","last_fullsized":"2024-04-23T16:45:00.198082+0000","mapping_epoch":563,"log_start":"548'1100","ondisk_log_start":"548'1100","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237371392,"num_objects":3622,"num_object_clones":0,"num_object_copies":7244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11196,"num_write_kb":474176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3792,"num_bytes_recovered":248512512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":1945},{"shards":"6,7","objects":1677}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1380,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-04-23T16:44:56.288769+0000","last_change":"2024-04-23T16:41:47.722866+0000","last_active":"2024-04-23T16:44:56.288769+0000","last_peered":"2024-04-23T16:44:56.288769+0000","last_clean":"2024-04-23T16:44:56.288769+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:44:56.288769+0000","last_undegraded":"2024-04-23T16:44:56.288769+0000","last_fullsized":"2024-04-23T16:44:56.288769+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:41:47.722762+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"592'11426","reported_seq":20742,"reported_epoch":592,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:45:04.555196+0000","last_change":"2024-04-23T16:43:08.136028+0000","last_active":"2024-04-23T16:45:04.555196+0000","last_peered":"2024-04-23T16:45:04.555196+0000","last_clean":"2024-04-23T16:42:42.962639+0000","last_became_active":"2024-04-23T16:43:08.109240+0000","last_became_peered":"2024-04-23T16:43:08.109240+0000","last_unstale":"2024-04-23T16:45:04.555196+0000","last_undegraded":"2024-04-23T16:43:07.955254+0000","last_fullsized":"2024-04-23T16:45:04.555196+0000","mapping_epoch":563,"log_start":"548'1400","ondisk_log_start":"548'1400","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229244928,"num_objects":3498,"num_object_clones":0,"num_object_copies":6996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11426,"num_write_kb":477568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4598,"num_bytes_recovered":301334528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":1058},{"shards":"4,11","objects":2440}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1301,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-04-23T16:44:56.275961+0000","last_change":"2024-04-23T16:41:49.835121+0000","last_active":"2024-04-23T16:44:56.275961+0000","last_peered":"2024-04-23T16:44:56.275961+0000","last_clean":"2024-04-23T16:44:56.275961+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:44:56.275961+0000","last_undegraded":"2024-04-23T16:44:56.275961+0000","last_fullsized":"2024-04-23T16:44:56.275961+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:41:49.834374+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"592'11311","reported_seq":11548,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:04.477584+0000","last_change":"2024-04-23T16:44:45.847313+0000","last_active":"2024-04-23T16:45:04.477584+0000","last_peered":"2024-04-23T16:45:04.477584+0000","last_clean":"2024-04-23T16:45:04.477584+0000","last_became_active":"2024-04-23T16:41:57.309349+0000","last_became_peered":"2024-04-23T16:41:57.309349+0000","last_unstale":"2024-04-23T16:45:04.477584+0000","last_undegraded":"2024-04-23T16:45:04.477584+0000","last_fullsized":"2024-04-23T16:45:04.477584+0000","mapping_epoch":540,"log_start":"548'1300","ondisk_log_start":"548'1300","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'10393","last_scrub_stamp":"2024-04-23T16:44:45.847160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:45.847160+0000","objects_scrubbed":4276,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:47.079172+0000","scrub_duration":795,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218431488,"num_objects":3333,"num_object_clones":0,"num_object_copies":6666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11311,"num_write_kb":468608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1470,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-04-23T16:44:56.301881+0000","last_change":"2024-04-23T16:42:33.628340+0000","last_active":"2024-04-23T16:44:56.301881+0000","last_peered":"2024-04-23T16:44:56.301881+0000","last_clean":"2024-04-23T16:44:56.301881+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:44:56.301881+0000","last_undegraded":"2024-04-23T16:44:56.301881+0000","last_fullsized":"2024-04-23T16:44:56.301881+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:33.628217+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:42:33.628217+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3685154840,"num_objects":56232,"num_object_clones":0,"num_object_copies":112464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56232,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181126,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16155,"num_bytes_recovered":1058734080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8332976128,"data_stored":8332967984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160826,"ondisk_log_size":160826,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1474560,"data_stored":1460352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056997,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1615220,"kb_used_data":962092,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":92129932,"statfs":{"total":95995035648,"available":94341050368,"internally_reserved":0,"allocated":985182208,"data_stored":984794760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":668793284},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351339,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612977,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1438852,"kb_used_data":898168,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92306300,"statfs":{"total":95995035648,"available":94521651200,"internally_reserved":0,"allocated":919724032,"data_stored":919330530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":553646528},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841909,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":828236,"kb_used_data":449984,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92916916,"statfs":{"total":95995035648,"available":95146921984,"internally_reserved":0,"allocated":460783616,"data_stored":460386562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168954,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":908532,"kb_used_data":335916,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92836620,"statfs":{"total":95995035648,"available":95064698880,"internally_reserved":0,"allocated":343977984,"data_stored":343590536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":586349002},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463295,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1569336,"kb_used_data":1042480,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":92175816,"statfs":{"total":95995035648,"available":94388035584,"internally_reserved":0,"allocated":1067499520,"data_stored":1067108000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":539425222},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137436,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":707960,"kb_used_data":469040,"kb_used_omap":231,"kb_used_meta":238680,"kb_avail":93037192,"statfs":{"total":95995035648,"available":95270084608,"internally_reserved":0,"allocated":480296960,"data_stored":479905440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":244409232},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738782,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1448960,"kb_used_data":876344,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92296192,"statfs":{"total":95995035648,"available":94511300608,"internally_reserved":0,"allocated":897376256,"data_stored":896982754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269452,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921160,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1819076,"kb_used_data":1089592,"kb_used_omap":1,"kb_used_meta":729470,"kb_avail":91926076,"statfs":{"total":95995035648,"available":94132301824,"internally_reserved":0,"allocated":1115742208,"data_stored":1115348706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":746977733},"hb_peers":[0,1,2,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757635,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1330556,"kb_used_data":677300,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":92414596,"statfs":{"total":95995035648,"available":94632546304,"internally_reserved":0,"allocated":693555200,"data_stored":693164200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":892993536,"data_stored":892993536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":477761536,"data_stored":477757464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1411842048,"data_stored":1411842048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1113325568,"data_stored":1113325568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":690814976,"data_stored":690814976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1060769792,"data_stored":1060765720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":342163456,"data_stored":342163456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":457179136,"data_stored":457179136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":909377536,"data_stored":909377536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":976748544,"data_stored":976748544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:45:07.546 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:45:07.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:07.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:45:07.650 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:45:07.724 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:07.776 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:45:07.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:07.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:45:07.885 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:45:07.949 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:08.006 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:45:08.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:08.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:45:08.008 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:07 smithi148 ceph-mon[44520]: from='client.16866 v1:172.21.15.103:0/647896861' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:45:08.008 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:07 smithi148 ceph-mon[44520]: pgmap v1658: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 819 op/s; 5972/108786 objects degraded (5.490%); 14 MiB/s, 216 objects/s recovering 2024-04-23T16:45:08.108 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:45:08.143 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:07 smithi188 ceph-mon[44101]: from='client.16866 v1:172.21.15.103:0/647896861' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:45:08.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:07 smithi188 ceph-mon[44101]: pgmap v1658: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 819 op/s; 5972/108786 objects degraded (5.490%); 14 MiB/s, 216 objects/s recovering 2024-04-23T16:45:08.280 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:45:08.301 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:45:08.322 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:45:08.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:07 smithi103 ceph-mon[39629]: from='client.16866 v1:172.21.15.103:0/647896861' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:45:08.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:07 smithi103 ceph-mon[39629]: pgmap v1658: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 819 op/s; 5972/108786 objects degraded (5.490%); 14 MiB/s, 216 objects/s recovering 2024-04-23T16:45:08.359 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:45:08.385 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:45:08.415 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:45:08.447 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:45:08.470 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:45:08.492 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:45:08.517 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:45:08.541 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:45:08.541 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:45:08.541 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:45:08.541 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:45:08.541 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:45:08.541 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:45:08.541 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:45:08.542 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:45:08.542 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:45:08.542 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:45:08.542 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:45:08.542 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:45:08.542 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:45:09.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:08 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3508549813' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:45:09.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:09.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:09.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:09.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:09.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:09.103 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:45:09.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:09.107 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:45:09.209 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:45:09.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:08 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3508549813' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:45:09.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:08 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:09.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:08 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:09.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:08 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:09.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:08 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:09.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:08 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:09.279 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:08 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3508549813' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:45:09.279 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:08 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:09.279 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:08 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:09.279 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:08 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:09.279 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:08 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:09.279 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:08 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:09.283 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:09.334 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:45:09.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:09.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:45:09.446 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:45:09.479 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:09.737 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:45:09.737 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:45:09.830 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:45:09.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:09.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:45:09.853 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:45:09.853 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:45:09.935 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:45:09.985 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:10.010 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:45:10.012 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:10.012 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:10.012 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:10.012 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:10.012 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:10.012 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:10.012 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:09 smithi188 ceph-mon[44101]: pgmap v1659: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 822 op/s; 5972/107086 objects degraded (5.577%); 14 MiB/s, 216 objects/s recovering 2024-04-23T16:45:10.012 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:09 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 5972/107086 objects degraded (5.577%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:10.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:10.013 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:45:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:09 smithi103 ceph-mon[39629]: pgmap v1659: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 822 op/s; 5972/107086 objects degraded (5.577%); 14 MiB/s, 216 objects/s recovering 2024-04-23T16:45:10.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:09 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 5972/107086 objects degraded (5.577%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:10.114 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:45:10.147 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:10.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:10.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:10.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:10.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:10.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:10.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:10.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:09 smithi148 ceph-mon[44520]: pgmap v1659: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 822 op/s; 5972/107086 objects degraded (5.577%); 14 MiB/s, 216 objects/s recovering 2024-04-23T16:45:10.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:09 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 5972/107086 objects degraded (5.577%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:10.547 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:45:10.612 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:45:10.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:10.616 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:45:10.621 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-23T16:45:10.717 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:45:10.790 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:10.830 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:45:10.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:10.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:45:10.940 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:10.995 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:11.051 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:11.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:11.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:11.156 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:45:11.166 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:45:11.177 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:45:11.207 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:11.231 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:45:11.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:11.238 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:45:11.339 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:45:11.370 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:11.396 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:45:11.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:11.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:45:11.504 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:11.572 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:11.608 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:11.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:11.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:11.720 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:45:11.756 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:11.855 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:45:11.977 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:45:11.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:11 smithi103 ceph-mon[39629]: pgmap v1660: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 839 op/s; 4822/101092 objects degraded (4.770%); 14 MiB/s, 228 objects/s recovering 2024-04-23T16:45:11.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:11 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1416033463' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:45:12.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:11 smithi148 ceph-mon[44520]: pgmap v1660: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 839 op/s; 4822/101092 objects degraded (4.770%); 14 MiB/s, 228 objects/s recovering 2024-04-23T16:45:12.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:11 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1416033463' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:45:12.408 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:45:12.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:12.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:45:12.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:11 smithi188 ceph-mon[44101]: pgmap v1660: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 839 op/s; 4822/101092 objects degraded (4.770%); 14 MiB/s, 228 objects/s recovering 2024-04-23T16:45:12.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:11 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1416033463' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:45:12.512 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:45:12.545 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:12.578 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:45:12.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:12.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:45:12.685 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:45:12.714 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:12.983 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:12 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:45:12.983 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:12 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1416033463' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:45:12.984 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:12 smithi103 ceph-mon[39629]: osdmap e593: 12 total, 11 up, 9 in 2024-04-23T16:45:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:12 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:45:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:12 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1416033463' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:45:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:12 smithi148 ceph-mon[44520]: osdmap e593: 12 total, 11 up, 9 in 2024-04-23T16:45:13.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:12 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:45:13.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:12 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1416033463' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:45:13.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:12 smithi188 ceph-mon[44101]: osdmap e593: 12 total, 11 up, 9 in 2024-04-23T16:45:14.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:13 smithi103 ceph-mon[39629]: pgmap v1662: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 845 op/s; 4822/99494 objects degraded (4.847%); 14 MiB/s, 228 objects/s recovering 2024-04-23T16:45:14.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:13 smithi148 ceph-mon[44520]: pgmap v1662: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 845 op/s; 4822/99494 objects degraded (4.847%); 14 MiB/s, 228 objects/s recovering 2024-04-23T16:45:14.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:13 smithi188 ceph-mon[44101]: pgmap v1662: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 3.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 845 op/s; 4822/99494 objects degraded (4.847%); 14 MiB/s, 228 objects/s recovering 2024-04-23T16:45:15.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:14 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 4822/99494 objects degraded (4.847%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:15.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:14 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 4822/99494 objects degraded (4.847%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:15.445 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:45:15.446 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:45:15.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:14 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 4822/99494 objects degraded (4.847%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:15.693 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:45:15.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:15.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:45:15.799 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:45:15.883 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:15.914 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:45:15.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:15.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:45:16.023 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:45:16.058 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:16.093 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:45:16.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:16.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:45:16.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:16 smithi103 ceph-mon[39629]: pgmap v1663: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 2.9 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 863 op/s; 4182/95186 objects degraded (4.394%); 15 MiB/s, 235 objects/s recovering 2024-04-23T16:45:16.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:16 smithi148 ceph-mon[44520]: pgmap v1663: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 2.9 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 863 op/s; 4182/95186 objects degraded (4.394%); 15 MiB/s, 235 objects/s recovering 2024-04-23T16:45:16.199 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:45:16.202 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:45:16.235 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:16.239 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-23T16:45:16.315 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:45:16.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:16.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:45:16.424 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:45:16.467 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:16.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:16 smithi188 ceph-mon[44101]: pgmap v1663: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 2.9 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 863 op/s; 4182/95186 objects degraded (4.394%); 15 MiB/s, 235 objects/s recovering 2024-04-23T16:45:16.500 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:45:16.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:16.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:45:16.515 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:45:16.604 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:45:16.667 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:16.726 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:45:16.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:16.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:45:16.747 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:45:16.749 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:45:16.831 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:45:16.863 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:17.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:17 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:45:17.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:17 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:45:17.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:17 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:45:17.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:17 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:45:17.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:17 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:45:17.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:17 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:45:18.112 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:45:18.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:18.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:45:18.119 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1663,"stamp":"2024-04-23T16:45:14.872962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3119448632,"num_objects":47593,"num_object_clones":0,"num_object_copies":95186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47593,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189767,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18515,"num_bytes_recovered":1213670464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162396,"ondisk_log_size":162396,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":10698480,"kb_used_data":5843508,"kb_used_omap":243,"kb_used_meta":4854220,"kb_avail":833007888,"statfs":{"total":863955320832,"available":853000077312,"internally_reserved":0,"allocated":5983752192,"data_stored":5980470141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":249416,"internal_metadata":4970721720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-566296576,"num_objects":-8641,"num_object_clones":0,"num_object_copies":-17282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8641,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2358,"num_bytes_recovered":154533888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001934"},"pg_stats":[{"pgid":"7.d","version":"592'11913","reported_seq":12028,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:10.278039+0000","last_change":"2024-04-23T16:41:57.310289+0000","last_active":"2024-04-23T16:45:10.278039+0000","last_peered":"2024-04-23T16:45:10.278039+0000","last_clean":"2024-04-23T16:45:10.278039+0000","last_became_active":"2024-04-23T16:41:57.310025+0000","last_became_peered":"2024-04-23T16:41:57.310025+0000","last_unstale":"2024-04-23T16:45:10.278039+0000","last_undegraded":"2024-04-23T16:45:10.278039+0000","last_fullsized":"2024-04-23T16:45:10.278039+0000","mapping_epoch":540,"log_start":"549'1900","ondisk_log_start":"549'1900","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206831640,"num_objects":3157,"num_object_clones":0,"num_object_copies":6314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3157,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11913,"num_write_kb":482177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"592'12031","reported_seq":12222,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:10.855376+0000","last_change":"2024-04-23T16:43:29.012370+0000","last_active":"2024-04-23T16:45:10.855376+0000","last_peered":"2024-04-23T16:45:10.855376+0000","last_clean":"2024-04-23T16:45:10.855376+0000","last_became_active":"2024-04-23T16:41:57.310754+0000","last_became_peered":"2024-04-23T16:41:57.310754+0000","last_unstale":"2024-04-23T16:45:10.855376+0000","last_undegraded":"2024-04-23T16:45:10.855376+0000","last_fullsized":"2024-04-23T16:45:10.855376+0000","mapping_epoch":540,"log_start":"548'2000","ondisk_log_start":"548'2000","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6945","last_scrub_stamp":"2024-04-23T16:43:29.012285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:29.012285+0000","objects_scrubbed":6912,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:13.741098+0000","scrub_duration":1082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204013568,"num_objects":3113,"num_object_clones":0,"num_object_copies":6226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12031,"num_write_kb":484608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"593'11908","reported_seq":12151,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-04-23T16:45:13.519247+0000","last_change":"2024-04-23T16:44:45.745602+0000","last_active":"2024-04-23T16:45:13.519247+0000","last_peered":"2024-04-23T16:45:13.519247+0000","last_clean":"2024-04-23T16:45:13.519247+0000","last_became_active":"2024-04-23T16:41:57.310189+0000","last_became_peered":"2024-04-23T16:41:57.310189+0000","last_unstale":"2024-04-23T16:45:13.519247+0000","last_undegraded":"2024-04-23T16:45:13.519247+0000","last_fullsized":"2024-04-23T16:45:13.519247+0000","mapping_epoch":540,"log_start":"549'1900","ondisk_log_start":"549'1900","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'10439","last_scrub_stamp":"2024-04-23T16:44:45.745464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:45.745464+0000","objects_scrubbed":4477,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:10.632547+0000","scrub_duration":680,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196214784,"num_objects":2994,"num_object_clones":0,"num_object_copies":5988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11908,"num_write_kb":476864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"593'11974","reported_seq":12209,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-04-23T16:45:14.412041+0000","last_change":"2024-04-23T16:44:58.743309+0000","last_active":"2024-04-23T16:45:14.412041+0000","last_peered":"2024-04-23T16:45:14.412041+0000","last_clean":"2024-04-23T16:45:14.412041+0000","last_became_active":"2024-04-23T16:41:57.309569+0000","last_became_peered":"2024-04-23T16:41:57.309569+0000","last_unstale":"2024-04-23T16:45:14.412041+0000","last_undegraded":"2024-04-23T16:45:14.412041+0000","last_fullsized":"2024-04-23T16:45:14.412041+0000","mapping_epoch":540,"log_start":"548'1900","ondisk_log_start":"548'1900","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"591'11141","last_scrub_stamp":"2024-04-23T16:44:58.743181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:58.743181+0000","objects_scrubbed":3762,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:00.499252+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191234048,"num_objects":2918,"num_object_clones":0,"num_object_copies":5836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11974,"num_write_kb":476544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"592'11534","reported_seq":11645,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:10.713896+0000","last_change":"2024-04-23T16:41:57.310144+0000","last_active":"2024-04-23T16:45:10.713896+0000","last_peered":"2024-04-23T16:45:10.713896+0000","last_clean":"2024-04-23T16:45:10.713896+0000","last_became_active":"2024-04-23T16:41:57.309841+0000","last_became_peered":"2024-04-23T16:41:57.309841+0000","last_unstale":"2024-04-23T16:45:10.713896+0000","last_undegraded":"2024-04-23T16:45:10.713896+0000","last_fullsized":"2024-04-23T16:45:10.713896+0000","mapping_epoch":540,"log_start":"548'1500","ondisk_log_start":"548'1500","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197001216,"num_objects":3006,"num_object_clones":0,"num_object_copies":6012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11534,"num_write_kb":465280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"592'11761","reported_seq":20731,"reported_epoch":592,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:45:10.304166+0000","last_change":"2024-04-23T16:43:08.145899+0000","last_active":"2024-04-23T16:45:10.304166+0000","last_peered":"2024-04-23T16:45:10.304166+0000","last_clean":"2024-04-23T16:42:42.970788+0000","last_became_active":"2024-04-23T16:43:08.106076+0000","last_became_peered":"2024-04-23T16:43:08.106076+0000","last_unstale":"2024-04-23T16:45:10.304166+0000","last_undegraded":"2024-04-23T16:43:07.951894+0000","last_fullsized":"2024-04-23T16:45:10.304166+0000","mapping_epoch":563,"log_start":"548'1700","ondisk_log_start":"548'1700","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195624960,"num_objects":2985,"num_object_clones":0,"num_object_copies":5970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11761,"num_write_kb":471872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4425,"num_bytes_recovered":289996800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":1289},{"shards":"6,11","objects":1696}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1499,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-04-23T16:45:11.983272+0000","last_change":"2024-04-23T16:44:09.435333+0000","last_active":"2024-04-23T16:45:11.983272+0000","last_peered":"2024-04-23T16:45:11.983272+0000","last_clean":"2024-04-23T16:45:11.983272+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:45:11.983272+0000","last_undegraded":"2024-04-23T16:45:11.983272+0000","last_fullsized":"2024-04-23T16:45:11.983272+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:09.435212+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:44:09.435212+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:17.709965+0000","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"593'11966","reported_seq":12207,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-04-23T16:45:13.610695+0000","last_change":"2024-04-23T16:44:52.727725+0000","last_active":"2024-04-23T16:45:13.610695+0000","last_peered":"2024-04-23T16:45:13.610695+0000","last_clean":"2024-04-23T16:45:13.610695+0000","last_became_active":"2024-04-23T16:41:57.311753+0000","last_became_peered":"2024-04-23T16:41:57.311753+0000","last_unstale":"2024-04-23T16:45:13.610695+0000","last_undegraded":"2024-04-23T16:45:13.610695+0000","last_fullsized":"2024-04-23T16:45:13.610695+0000","mapping_epoch":540,"log_start":"549'1900","ondisk_log_start":"549'1900","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"588'10838","last_scrub_stamp":"2024-04-23T16:44:52.727632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:52.727632+0000","objects_scrubbed":4109,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:52.880867+0000","scrub_duration":730,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193986560,"num_objects":2960,"num_object_clones":0,"num_object_copies":5920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11966,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1505,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-04-23T16:45:11.995968+0000","last_change":"2024-04-23T16:44:45.850797+0000","last_active":"2024-04-23T16:45:11.995968+0000","last_peered":"2024-04-23T16:45:11.995968+0000","last_clean":"2024-04-23T16:45:11.995968+0000","last_became_active":"2024-04-23T16:43:07.962092+0000","last_became_peered":"2024-04-23T16:43:07.962092+0000","last_unstale":"2024-04-23T16:45:11.995968+0000","last_undegraded":"2024-04-23T16:45:11.995968+0000","last_fullsized":"2024-04-23T16:45:11.995968+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:45.850682+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:44:45.850682+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:12.889710+0000","scrub_duration":234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"593'11993","reported_seq":12236,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-04-23T16:45:14.475331+0000","last_change":"2024-04-23T16:44:44.578218+0000","last_active":"2024-04-23T16:45:14.475331+0000","last_peered":"2024-04-23T16:45:14.475331+0000","last_clean":"2024-04-23T16:45:14.475331+0000","last_became_active":"2024-04-23T16:41:57.305944+0000","last_became_peered":"2024-04-23T16:41:57.305944+0000","last_unstale":"2024-04-23T16:45:14.475331+0000","last_undegraded":"2024-04-23T16:45:14.475331+0000","last_fullsized":"2024-04-23T16:45:14.475331+0000","mapping_epoch":540,"log_start":"549'1900","ondisk_log_start":"549'1900","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'10405","last_scrub_stamp":"2024-04-23T16:44:44.578065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:44.578065+0000","objects_scrubbed":4454,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:01.634216+0000","scrub_duration":505,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187105280,"num_objects":2855,"num_object_clones":0,"num_object_copies":5710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11993,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1472,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:02.776595+0000","last_change":"2024-04-23T16:42:32.662620+0000","last_active":"2024-04-23T16:45:02.776595+0000","last_peered":"2024-04-23T16:45:02.776595+0000","last_clean":"2024-04-23T16:45:02.776595+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:45:02.776595+0000","last_undegraded":"2024-04-23T16:45:02.776595+0000","last_fullsized":"2024-04-23T16:45:02.776595+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:32.662476+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:42:32.662476+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"593'12297","reported_seq":12539,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-04-23T16:45:14.434884+0000","last_change":"2024-04-23T16:44:46.996868+0000","last_active":"2024-04-23T16:45:14.434884+0000","last_peered":"2024-04-23T16:45:14.434884+0000","last_clean":"2024-04-23T16:45:14.434884+0000","last_became_active":"2024-04-23T16:41:57.309858+0000","last_became_peered":"2024-04-23T16:41:57.309858+0000","last_unstale":"2024-04-23T16:45:14.434884+0000","last_undegraded":"2024-04-23T16:45:14.434884+0000","last_fullsized":"2024-04-23T16:45:14.434884+0000","mapping_epoch":540,"log_start":"549'2200","ondisk_log_start":"549'2200","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'10834","last_scrub_stamp":"2024-04-23T16:44:46.996775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:46.996775+0000","objects_scrubbed":4359,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:53.498504+0000","scrub_duration":887,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188022784,"num_objects":2869,"num_object_clones":0,"num_object_copies":5738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12297,"num_write_kb":485312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1499,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-04-23T16:45:11.972147+0000","last_change":"2024-04-23T16:43:40.258384+0000","last_active":"2024-04-23T16:45:11.972147+0000","last_peered":"2024-04-23T16:45:11.972147+0000","last_clean":"2024-04-23T16:45:11.972147+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:45:11.972147+0000","last_undegraded":"2024-04-23T16:45:11.972147+0000","last_fullsized":"2024-04-23T16:45:11.972147+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:43:40.258213+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:43:40.258213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:14.287643+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"592'11681","reported_seq":11795,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:11.541679+0000","last_change":"2024-04-23T16:41:57.310695+0000","last_active":"2024-04-23T16:45:11.541679+0000","last_peered":"2024-04-23T16:45:11.541679+0000","last_clean":"2024-04-23T16:45:11.541679+0000","last_became_active":"2024-04-23T16:41:57.310400+0000","last_became_peered":"2024-04-23T16:41:57.310400+0000","last_unstale":"2024-04-23T16:45:11.541679+0000","last_undegraded":"2024-04-23T16:45:11.541679+0000","last_fullsized":"2024-04-23T16:45:11.541679+0000","mapping_epoch":540,"log_start":"548'1600","ondisk_log_start":"548'1600","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197722112,"num_objects":3017,"num_object_clones":0,"num_object_copies":6034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11681,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1070,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:02.780643+0000","last_change":"2024-04-23T16:44:18.555667+0000","last_active":"2024-04-23T16:45:02.780643+0000","last_peered":"2024-04-23T16:45:02.780643+0000","last_clean":"2024-04-23T16:45:02.780643+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:45:02.780643+0000","last_undegraded":"2024-04-23T16:45:02.780643+0000","last_fullsized":"2024-04-23T16:45:02.780643+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:18.555538+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:44:18.555538+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:35.829709+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"592'11599","reported_seq":11712,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:10.284299+0000","last_change":"2024-04-23T16:41:57.306526+0000","last_active":"2024-04-23T16:45:10.284299+0000","last_peered":"2024-04-23T16:45:10.284299+0000","last_clean":"2024-04-23T16:45:10.284299+0000","last_became_active":"2024-04-23T16:41:57.306229+0000","last_became_peered":"2024-04-23T16:41:57.306229+0000","last_unstale":"2024-04-23T16:45:10.284299+0000","last_undegraded":"2024-04-23T16:45:10.284299+0000","last_fullsized":"2024-04-23T16:45:10.284299+0000","mapping_epoch":540,"log_start":"548'1500","ondisk_log_start":"548'1500","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203489280,"num_objects":3105,"num_object_clones":0,"num_object_copies":6210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11599,"num_write_kb":470528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1027,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-04-23T16:45:11.972467+0000","last_change":"2024-04-23T16:44:07.230780+0000","last_active":"2024-04-23T16:45:11.972467+0000","last_peered":"2024-04-23T16:45:11.972467+0000","last_clean":"2024-04-23T16:45:11.972467+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:45:11.972467+0000","last_undegraded":"2024-04-23T16:45:11.972467+0000","last_fullsized":"2024-04-23T16:45:11.972467+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:07.230640+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:44:07.230640+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:12.044599+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.0","version":"593'11811","reported_seq":12047,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-04-23T16:45:12.754402+0000","last_change":"2024-04-23T16:45:00.368292+0000","last_active":"2024-04-23T16:45:12.754402+0000","last_peered":"2024-04-23T16:45:12.754402+0000","last_clean":"2024-04-23T16:45:12.754402+0000","last_became_active":"2024-04-23T16:41:57.311479+0000","last_became_peered":"2024-04-23T16:41:57.311479+0000","last_unstale":"2024-04-23T16:45:12.754402+0000","last_undegraded":"2024-04-23T16:45:12.754402+0000","last_fullsized":"2024-04-23T16:45:12.754402+0000","mapping_epoch":540,"log_start":"548'1800","ondisk_log_start":"548'1800","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"591'11135","last_scrub_stamp":"2024-04-23T16:45:00.368005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:00.368005+0000","objects_scrubbed":3585,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:06.315750+0000","scrub_duration":778,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189333504,"num_objects":2889,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11811,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1507,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-04-23T16:45:11.982325+0000","last_change":"2024-04-23T16:44:11.349248+0000","last_active":"2024-04-23T16:45:11.982325+0000","last_peered":"2024-04-23T16:45:11.982325+0000","last_clean":"2024-04-23T16:45:11.982325+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:45:11.982325+0000","last_undegraded":"2024-04-23T16:45:11.982325+0000","last_fullsized":"2024-04-23T16:45:11.982325+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:11.348079+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:44:11.348079+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:21.183735+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.1","version":"593'11739","reported_seq":20712,"reported_epoch":593,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:45:14.562991+0000","last_change":"2024-04-23T16:43:08.081567+0000","last_active":"2024-04-23T16:45:14.562991+0000","last_peered":"2024-04-23T16:45:14.562991+0000","last_clean":"2024-04-23T16:42:42.976600+0000","last_became_active":"2024-04-23T16:43:08.067887+0000","last_became_peered":"2024-04-23T16:43:08.067887+0000","last_unstale":"2024-04-23T16:45:14.562991+0000","last_undegraded":"2024-04-23T16:43:07.950381+0000","last_fullsized":"2024-04-23T16:45:14.562991+0000","mapping_epoch":563,"log_start":"548'1700","ondisk_log_start":"548'1700","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189857792,"num_objects":2897,"num_object_clones":0,"num_object_copies":5794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11739,"num_write_kb":468352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4426,"num_bytes_recovered":290062336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":1162},{"shards":"4,6","objects":1735}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1489,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:02.778019+0000","last_change":"2024-04-23T16:43:07.918302+0000","last_active":"2024-04-23T16:45:02.778019+0000","last_peered":"2024-04-23T16:45:02.778019+0000","last_clean":"2024-04-23T16:45:02.778019+0000","last_became_active":"2024-04-23T16:43:07.918042+0000","last_became_peered":"2024-04-23T16:43:07.918042+0000","last_unstale":"2024-04-23T16:45:02.778019+0000","last_undegraded":"2024-04-23T16:45:02.778019+0000","last_fullsized":"2024-04-23T16:45:02.778019+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:33.716343+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:42:33.716343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"592'11767","reported_seq":20610,"reported_epoch":592,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:45:10.301244+0000","last_change":"2024-04-23T16:43:08.363306+0000","last_active":"2024-04-23T16:45:10.301244+0000","last_peered":"2024-04-23T16:45:10.301244+0000","last_clean":"2024-04-23T16:42:42.950766+0000","last_became_active":"2024-04-23T16:43:08.333793+0000","last_became_peered":"2024-04-23T16:43:08.333793+0000","last_unstale":"2024-04-23T16:45:10.301244+0000","last_undegraded":"2024-04-23T16:43:07.950356+0000","last_fullsized":"2024-04-23T16:45:10.301244+0000","mapping_epoch":563,"log_start":"548'1700","ondisk_log_start":"548'1700","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199950336,"num_objects":3051,"num_object_clones":0,"num_object_copies":6102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1374,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11767,"num_write_kb":474176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4363,"num_bytes_recovered":285933568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":1374},{"shards":"6,7","objects":1677}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1382,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:02.769717+0000","last_change":"2024-04-23T16:41:47.722866+0000","last_active":"2024-04-23T16:45:02.769717+0000","last_peered":"2024-04-23T16:45:02.769717+0000","last_clean":"2024-04-23T16:45:02.769717+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:45:02.769717+0000","last_undegraded":"2024-04-23T16:45:02.769717+0000","last_fullsized":"2024-04-23T16:45:02.769717+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:41:47.722762+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"593'11952","reported_seq":22672,"reported_epoch":593,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:45:14.577177+0000","last_change":"2024-04-23T16:43:08.136028+0000","last_active":"2024-04-23T16:45:14.577177+0000","last_peered":"2024-04-23T16:45:14.577177+0000","last_clean":"2024-04-23T16:42:42.962639+0000","last_became_active":"2024-04-23T16:43:08.109240+0000","last_became_peered":"2024-04-23T16:43:08.109240+0000","last_unstale":"2024-04-23T16:45:14.577177+0000","last_undegraded":"2024-04-23T16:43:07.955254+0000","last_fullsized":"2024-04-23T16:45:14.577177+0000","mapping_epoch":563,"log_start":"549'1900","ondisk_log_start":"549'1900","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194772992,"num_objects":2972,"num_object_clones":0,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11952,"num_write_kb":477568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5299,"num_bytes_recovered":347275264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":357},{"shards":"4,11","objects":2615}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1303,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:02.777987+0000","last_change":"2024-04-23T16:41:49.835121+0000","last_active":"2024-04-23T16:45:02.777987+0000","last_peered":"2024-04-23T16:45:02.777987+0000","last_clean":"2024-04-23T16:45:02.777987+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:45:02.777987+0000","last_undegraded":"2024-04-23T16:45:02.777987+0000","last_fullsized":"2024-04-23T16:45:02.777987+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:41:49.834374+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"593'11841","reported_seq":12081,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-04-23T16:45:14.359197+0000","last_change":"2024-04-23T16:44:45.847313+0000","last_active":"2024-04-23T16:45:14.359197+0000","last_peered":"2024-04-23T16:45:14.359197+0000","last_clean":"2024-04-23T16:45:14.359197+0000","last_became_active":"2024-04-23T16:41:57.309349+0000","last_became_peered":"2024-04-23T16:41:57.309349+0000","last_unstale":"2024-04-23T16:45:14.359197+0000","last_undegraded":"2024-04-23T16:45:14.359197+0000","last_fullsized":"2024-04-23T16:45:14.359197+0000","mapping_epoch":540,"log_start":"548'1800","ondisk_log_start":"548'1800","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'10393","last_scrub_stamp":"2024-04-23T16:44:45.847160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:45.847160+0000","objects_scrubbed":4276,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:47.079172+0000","scrub_duration":795,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183697408,"num_objects":2803,"num_object_clones":0,"num_object_copies":5606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11841,"num_write_kb":468608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1472,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-04-23T16:45:02.776573+0000","last_change":"2024-04-23T16:42:33.628340+0000","last_active":"2024-04-23T16:45:02.776573+0000","last_peered":"2024-04-23T16:45:02.776573+0000","last_clean":"2024-04-23T16:45:02.776573+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:45:02.776573+0000","last_undegraded":"2024-04-23T16:45:02.776573+0000","last_fullsized":"2024-04-23T16:45:02.776573+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:33.628217+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:42:33.628217+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3118858264,"num_objects":47591,"num_object_clones":0,"num_object_copies":95182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47591,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189767,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18513,"num_bytes_recovered":1213267968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7352360960,"data_stored":7352352816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160867,"ondisk_log_size":160867,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1474560,"data_stored":1460352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056999,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1519552,"kb_used_data":870584,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92225600,"statfs":{"total":95995035648,"available":94439014400,"internally_reserved":0,"allocated":891478016,"data_stored":891084514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":664533444},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351341,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612979,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1289680,"kb_used_data":757196,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":92455472,"statfs":{"total":95995035648,"available":94674403328,"internally_reserved":0,"allocated":775368704,"data_stored":775030145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":545126848},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841911,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":762456,"kb_used_data":379924,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92982696,"statfs":{"total":95995035648,"available":95214280704,"internally_reserved":0,"allocated":389042176,"data_stored":388700065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168956,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":878784,"kb_used_data":301944,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92866368,"statfs":{"total":95995035648,"available":95095160832,"internally_reserved":0,"allocated":309190656,"data_stored":308797154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":590608842},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463297,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1422212,"kb_used_data":903740,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":92322940,"statfs":{"total":95995035648,"available":94538690560,"internally_reserved":0,"allocated":925429760,"data_stored":925032186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":530905542},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137438,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":627396,"kb_used_data":400956,"kb_used_omap":231,"kb_used_meta":226200,"kb_avail":93117756,"statfs":{"total":95995035648,"available":95352582144,"internally_reserved":0,"allocated":410578944,"data_stored":410181370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":231629712},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738784,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1322004,"kb_used_data":736908,"kb_used_omap":1,"kb_used_meta":585086,"kb_avail":92423148,"statfs":{"total":95995035648,"available":94641303552,"internally_reserved":0,"allocated":754593792,"data_stored":754255233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":599128517},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269454,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921162,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1660048,"kb_used_data":920716,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":92085104,"statfs":{"total":95995035648,"available":94295146496,"internally_reserved":0,"allocated":942813184,"data_stored":942474625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756939205},"hb_peers":[0,1,2,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757637,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1216348,"kb_used_data":571540,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":92528804,"statfs":{"total":95995035648,"available":94749495296,"internally_reserved":0,"allocated":585256960,"data_stored":584914849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":749928448,"data_stored":749928448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":409014272,"data_stored":409010200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1411842048,"data_stored":1411842048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":938606592,"data_stored":938606592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":583270400,"data_stored":583270400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":919932928,"data_stored":919928856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":307167232,"data_stored":307167232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":385941504,"data_stored":385941504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":766246912,"data_stored":766246912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":880410624,"data_stored":880410624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:45:18.121 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:45:18.145 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:45:18.216 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:45:18.246 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:18.269 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:45:18.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:18.275 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:45:18.297 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:18 smithi103 ceph-mon[39629]: from='client.16881 v1:172.21.15.103:0/2358742248' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:45:18.297 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:18 smithi103 ceph-mon[39629]: pgmap v1664: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 2.8 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 876 op/s; 3566/91258 objects degraded (3.908%); 15 MiB/s, 240 objects/s recovering 2024-04-23T16:45:18.298 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:18 smithi103 ceph-mon[39629]: osdmap e594: 12 total, 11 up, 9 in 2024-04-23T16:45:18.376 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:45:18.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:18 smithi148 ceph-mon[44520]: from='client.16881 v1:172.21.15.103:0/2358742248' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:45:18.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:18 smithi148 ceph-mon[44520]: pgmap v1664: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 2.8 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 876 op/s; 3566/91258 objects degraded (3.908%); 15 MiB/s, 240 objects/s recovering 2024-04-23T16:45:18.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:18 smithi148 ceph-mon[44520]: osdmap e594: 12 total, 11 up, 9 in 2024-04-23T16:45:18.410 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:18.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:18 smithi188 ceph-mon[44101]: from='client.16881 v1:172.21.15.103:0/2358742248' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:45:18.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:18 smithi188 ceph-mon[44101]: pgmap v1664: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 2.8 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 876 op/s; 3566/91258 objects degraded (3.908%); 15 MiB/s, 240 objects/s recovering 2024-04-23T16:45:18.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:18 smithi188 ceph-mon[44101]: osdmap e594: 12 total, 11 up, 9 in 2024-04-23T16:45:18.695 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:45:18.960 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:45:18.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:18.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:45:19.073 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:45:19.107 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:19.212 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=3212989789,config=2679774077,health=4115056671,kv=1660059694}) 2024-04-23T16:45:19.212 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=1194111669,logm=2998910825}) 2024-04-23T16:45:19.212 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3895610374}) 2024-04-23T16:45:19.212 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1516904209}) 2024-04-23T16:45:19.212 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3119619694}) 2024-04-23T16:45:19.212 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=207327141}) 2024-04-23T16:45:19.212 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=99817758}) 2024-04-23T16:45:19.212 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=163409682}) 2024-04-23T16:45:19.213 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3031676889}) 2024-04-23T16:45:19.213 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131987484}) 2024-04-23T16:45:19.213 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339945755}) 2024-04-23T16:45:19.213 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2548970789}) 2024-04-23T16:45:19.213 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3751315787}) 2024-04-23T16:45:19.213 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2171580365}) 2024-04-23T16:45:19.213 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4110102592}) 2024-04-23T16:45:19.213 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3575208581}) 2024-04-23T16:45:19.213 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=530857639}) 2024-04-23T16:45:19.213 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2302035919}) 2024-04-23T16:45:19.213 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2927334366}) 2024-04-23T16:45:19.213 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1197232988}) 2024-04-23T16:45:19.213 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3875159588}) 2024-04-23T16:45:19.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3694708224}) 2024-04-23T16:45:19.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3114824281}) 2024-04-23T16:45:19.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3462776889}) 2024-04-23T16:45:19.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1311439992}) 2024-04-23T16:45:19.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1580201295}) 2024-04-23T16:45:19.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=369156003}) 2024-04-23T16:45:19.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3433316739}) 2024-04-23T16:45:19.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=160941900}) 2024-04-23T16:45:19.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1063047065}) 2024-04-23T16:45:19.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1096943401}) 2024-04-23T16:45:19.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2342713296}) 2024-04-23T16:45:19.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=80800282}) 2024-04-23T16:45:19.215 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1200434783}) 2024-04-23T16:45:19.215 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2949891798}) 2024-04-23T16:45:19.215 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=833178607}) 2024-04-23T16:45:19.215 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1779819818}) 2024-04-23T16:45:19.215 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3418568891}) 2024-04-23T16:45:19.215 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3612662711}) 2024-04-23T16:45:19.215 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3151095334}) 2024-04-23T16:45:19.215 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2816814230}) 2024-04-23T16:45:19.215 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1470352017}) 2024-04-23T16:45:19.215 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2447872828}) 2024-04-23T16:45:19.215 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1012957773}) 2024-04-23T16:45:19.215 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=809551641}) 2024-04-23T16:45:19.215 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1225223168}) 2024-04-23T16:45:19.215 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3942949286}) 2024-04-23T16:45:19.216 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1603376748}) 2024-04-23T16:45:19.216 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3885851126}) 2024-04-23T16:45:19.216 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3256686771}) 2024-04-23T16:45:19.216 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3038288440}) 2024-04-23T16:45:19.216 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3904526390}) 2024-04-23T16:45:19.216 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3782649746}) 2024-04-23T16:45:19.216 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3850559544}) 2024-04-23T16:45:19.216 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692307399}) 2024-04-23T16:45:19.216 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=911498674}) 2024-04-23T16:45:19.216 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2086095196}) 2024-04-23T16:45:19.216 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2597830701}) 2024-04-23T16:45:19.216 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1358353531}) 2024-04-23T16:45:19.216 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1460018385}) 2024-04-23T16:45:19.216 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1337603612}) 2024-04-23T16:45:19.217 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3765500450}) 2024-04-23T16:45:19.217 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3851056258}) 2024-04-23T16:45:19.217 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3364380191}) 2024-04-23T16:45:19.217 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3962392136}) 2024-04-23T16:45:19.217 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2171929238}) 2024-04-23T16:45:19.217 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3580419075}) 2024-04-23T16:45:19.217 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2508080607}) 2024-04-23T16:45:19.217 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2926300092}) 2024-04-23T16:45:19.217 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2356686234}) 2024-04-23T16:45:19.217 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=467308763}) 2024-04-23T16:45:19.217 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2760782781}) 2024-04-23T16:45:19.217 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1314181277}) 2024-04-23T16:45:19.217 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3718344968}) 2024-04-23T16:45:19.217 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2340668058}) 2024-04-23T16:45:19.218 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=342770345}) 2024-04-23T16:45:19.218 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=84,mdsmap=3,mgr=13} crc {logm=2609652647,mdsmap=2840952586,mgr=1322674327}) 2024-04-23T16:45:19.218 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mgr=6,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=76} crc {mgr=3879362914,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=2248419553,mon_config_key=320736623}) 2024-04-23T16:45:19.218 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1738979286}) 2024-04-23T16:45:19.218 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3843184555}) 2024-04-23T16:45:19.218 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=11,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=71} crc {mon_config_key=3320863617,monmap=2133572193,osd_metadata=1962451961,osd_pg_creating=3381508937,osdmap=1583554060}) 2024-04-23T16:45:19.218 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=38} crc {osdmap=744622700}) 2024-04-23T16:45:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=3212989789,config=2679774077,health=4115056671,kv=1660059694}) 2024-04-23T16:45:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=1194111669,logm=2998910825}) 2024-04-23T16:45:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3895610374}) 2024-04-23T16:45:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1516904209}) 2024-04-23T16:45:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3119619694}) 2024-04-23T16:45:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=207327141}) 2024-04-23T16:45:19.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=99817758}) 2024-04-23T16:45:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=163409682}) 2024-04-23T16:45:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3031676889}) 2024-04-23T16:45:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131987484}) 2024-04-23T16:45:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339945755}) 2024-04-23T16:45:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2548970789}) 2024-04-23T16:45:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3751315787}) 2024-04-23T16:45:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2171580365}) 2024-04-23T16:45:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4110102592}) 2024-04-23T16:45:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3575208581}) 2024-04-23T16:45:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=530857639}) 2024-04-23T16:45:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2302035919}) 2024-04-23T16:45:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2927334366}) 2024-04-23T16:45:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1197232988}) 2024-04-23T16:45:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3875159588}) 2024-04-23T16:45:19.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3694708224}) 2024-04-23T16:45:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3114824281}) 2024-04-23T16:45:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3462776889}) 2024-04-23T16:45:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1311439992}) 2024-04-23T16:45:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1580201295}) 2024-04-23T16:45:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=369156003}) 2024-04-23T16:45:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3433316739}) 2024-04-23T16:45:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=160941900}) 2024-04-23T16:45:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1063047065}) 2024-04-23T16:45:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1096943401}) 2024-04-23T16:45:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2342713296}) 2024-04-23T16:45:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=80800282}) 2024-04-23T16:45:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1200434783}) 2024-04-23T16:45:19.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2949891798}) 2024-04-23T16:45:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=833178607}) 2024-04-23T16:45:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1779819818}) 2024-04-23T16:45:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3418568891}) 2024-04-23T16:45:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3612662711}) 2024-04-23T16:45:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3151095334}) 2024-04-23T16:45:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2816814230}) 2024-04-23T16:45:19.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1470352017}) 2024-04-23T16:45:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2447872828}) 2024-04-23T16:45:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1012957773}) 2024-04-23T16:45:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=809551641}) 2024-04-23T16:45:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1225223168}) 2024-04-23T16:45:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3942949286}) 2024-04-23T16:45:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1603376748}) 2024-04-23T16:45:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3885851126}) 2024-04-23T16:45:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3256686771}) 2024-04-23T16:45:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3038288440}) 2024-04-23T16:45:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3904526390}) 2024-04-23T16:45:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3782649746}) 2024-04-23T16:45:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3850559544}) 2024-04-23T16:45:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692307399}) 2024-04-23T16:45:19.379 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=911498674}) 2024-04-23T16:45:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2086095196}) 2024-04-23T16:45:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2597830701}) 2024-04-23T16:45:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1358353531}) 2024-04-23T16:45:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1460018385}) 2024-04-23T16:45:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1337603612}) 2024-04-23T16:45:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3765500450}) 2024-04-23T16:45:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3851056258}) 2024-04-23T16:45:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3364380191}) 2024-04-23T16:45:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3962392136}) 2024-04-23T16:45:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2171929238}) 2024-04-23T16:45:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3580419075}) 2024-04-23T16:45:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2508080607}) 2024-04-23T16:45:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2926300092}) 2024-04-23T16:45:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2356686234}) 2024-04-23T16:45:19.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=467308763}) 2024-04-23T16:45:19.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2760782781}) 2024-04-23T16:45:19.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1314181277}) 2024-04-23T16:45:19.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3718344968}) 2024-04-23T16:45:19.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2340668058}) 2024-04-23T16:45:19.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=342770345}) 2024-04-23T16:45:19.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=84,mdsmap=3,mgr=13} crc {logm=2609652647,mdsmap=2840952586,mgr=1322674327}) 2024-04-23T16:45:19.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mgr=6,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=76} crc {mgr=3879362914,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=2248419553,mon_config_key=320736623}) 2024-04-23T16:45:19.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1738979286}) 2024-04-23T16:45:19.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3843184555}) 2024-04-23T16:45:19.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=11,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=71} crc {mon_config_key=3320863617,monmap=2133572193,osd_metadata=1962451961,osd_pg_creating=3381508937,osdmap=1583554060}) 2024-04-23T16:45:19.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=38} crc {osdmap=744622700}) 2024-04-23T16:45:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=3212989789,config=2679774077,health=4115056671,kv=1660059694}) 2024-04-23T16:45:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=1194111669,logm=2998910825}) 2024-04-23T16:45:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3895610374}) 2024-04-23T16:45:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1516904209}) 2024-04-23T16:45:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3119619694}) 2024-04-23T16:45:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=207327141}) 2024-04-23T16:45:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=99817758}) 2024-04-23T16:45:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=163409682}) 2024-04-23T16:45:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3031676889}) 2024-04-23T16:45:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131987484}) 2024-04-23T16:45:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339945755}) 2024-04-23T16:45:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2548970789}) 2024-04-23T16:45:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3751315787}) 2024-04-23T16:45:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2171580365}) 2024-04-23T16:45:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4110102592}) 2024-04-23T16:45:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3575208581}) 2024-04-23T16:45:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=530857639}) 2024-04-23T16:45:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2302035919}) 2024-04-23T16:45:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2927334366}) 2024-04-23T16:45:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1197232988}) 2024-04-23T16:45:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3875159588}) 2024-04-23T16:45:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3694708224}) 2024-04-23T16:45:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3114824281}) 2024-04-23T16:45:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3462776889}) 2024-04-23T16:45:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1311439992}) 2024-04-23T16:45:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1580201295}) 2024-04-23T16:45:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=369156003}) 2024-04-23T16:45:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3433316739}) 2024-04-23T16:45:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=160941900}) 2024-04-23T16:45:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1063047065}) 2024-04-23T16:45:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1096943401}) 2024-04-23T16:45:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2342713296}) 2024-04-23T16:45:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=80800282}) 2024-04-23T16:45:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1200434783}) 2024-04-23T16:45:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2949891798}) 2024-04-23T16:45:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=833178607}) 2024-04-23T16:45:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1779819818}) 2024-04-23T16:45:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3418568891}) 2024-04-23T16:45:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3612662711}) 2024-04-23T16:45:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3151095334}) 2024-04-23T16:45:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2816814230}) 2024-04-23T16:45:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1470352017}) 2024-04-23T16:45:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2447872828}) 2024-04-23T16:45:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1012957773}) 2024-04-23T16:45:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=809551641}) 2024-04-23T16:45:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1225223168}) 2024-04-23T16:45:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3942949286}) 2024-04-23T16:45:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1603376748}) 2024-04-23T16:45:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3885851126}) 2024-04-23T16:45:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3256686771}) 2024-04-23T16:45:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3038288440}) 2024-04-23T16:45:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3904526390}) 2024-04-23T16:45:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3782649746}) 2024-04-23T16:45:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3850559544}) 2024-04-23T16:45:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692307399}) 2024-04-23T16:45:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=911498674}) 2024-04-23T16:45:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2086095196}) 2024-04-23T16:45:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2597830701}) 2024-04-23T16:45:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1358353531}) 2024-04-23T16:45:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1460018385}) 2024-04-23T16:45:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1337603612}) 2024-04-23T16:45:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3765500450}) 2024-04-23T16:45:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3851056258}) 2024-04-23T16:45:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3364380191}) 2024-04-23T16:45:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3962392136}) 2024-04-23T16:45:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2171929238}) 2024-04-23T16:45:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3580419075}) 2024-04-23T16:45:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2508080607}) 2024-04-23T16:45:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2926300092}) 2024-04-23T16:45:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2356686234}) 2024-04-23T16:45:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=467308763}) 2024-04-23T16:45:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2760782781}) 2024-04-23T16:45:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1314181277}) 2024-04-23T16:45:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3718344968}) 2024-04-23T16:45:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2340668058}) 2024-04-23T16:45:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=342770345}) 2024-04-23T16:45:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=84,mdsmap=3,mgr=13} crc {logm=2609652647,mdsmap=2840952586,mgr=1322674327}) 2024-04-23T16:45:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mgr=6,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=76} crc {mgr=3879362914,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=2248419553,mon_config_key=320736623}) 2024-04-23T16:45:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1738979286}) 2024-04-23T16:45:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3843184555}) 2024-04-23T16:45:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=11,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=71} crc {mon_config_key=3320863617,monmap=2133572193,osd_metadata=1962451961,osd_pg_creating=3381508937,osdmap=1583554060}) 2024-04-23T16:45:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=38} crc {osdmap=744622700}) 2024-04-23T16:45:20.044 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:45:20.069 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:45:20.089 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:45:20.115 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:45:20.143 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:45:20.160 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:45:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:20.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:45:20.180 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:45:20.206 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:45:20.229 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:45:20.250 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:45:20.273 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:45:20.277 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:45:20.296 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:45:20.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:45:20.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:45:20.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:45:20.297 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:45:20.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:45:20.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:45:20.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:45:20.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:45:20.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:45:20.297 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:45:20.298 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:45:20.298 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:45:20.309 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:20 smithi103 ceph-mon[39629]: pgmap v1666: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 2.7 GiB data, 9.9 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 726 op/s; 3566/89458 objects degraded (3.986%); 9.8 MiB/s, 156 objects/s recovering 2024-04-23T16:45:20.309 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:20 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 3566/89458 objects degraded (3.986%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:20.309 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:20 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3475853235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:45:20.309 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:20.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:20 smithi148 ceph-mon[44520]: pgmap v1666: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 2.7 GiB data, 9.9 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 726 op/s; 3566/89458 objects degraded (3.986%); 9.8 MiB/s, 156 objects/s recovering 2024-04-23T16:45:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:20 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 3566/89458 objects degraded (3.986%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:20 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3475853235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:45:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:20 smithi188 ceph-mon[44101]: pgmap v1666: 27 pgs: 4 active+recovering+degraded, 23 active+clean; 2.7 GiB data, 9.9 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 726 op/s; 3566/89458 objects degraded (3.986%); 9.8 MiB/s, 156 objects/s recovering 2024-04-23T16:45:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:20 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 3566/89458 objects degraded (3.986%), 4 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:20 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3475853235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:45:20.661 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:45:20.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:20.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:45:20.766 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:45:20.796 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:20.870 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:45:20.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:20.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:45:20.976 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:45:21.049 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:21.087 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:45:21.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:21.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:45:21.200 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:45:21.251 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:21.298 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:45:21.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:21.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:45:21.309 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.309 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.309 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.309 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.309 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.309 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.309 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.408 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:45:21.431 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:45:21.432 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:45:21.457 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:21.498 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:45:21.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:21.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:45:21.606 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:45:21.635 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:21.673 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:45:21.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:21.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:45:21.687 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:45:21.778 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:45:21.838 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:21.908 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:45:21.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:21.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:45:22.027 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:45:22.098 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:22.098 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:22.098 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:22.098 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:22.098 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:22 smithi148 ceph-mon[44520]: pgmap v1667: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.5 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 949 op/s; 2259/83090 objects degraded (2.719%); 18 MiB/s, 287 objects/s recovering 2024-04-23T16:45:22.110 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:22.137 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:45:22.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:22.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:45:22.225 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:22.226 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:22.226 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:22.226 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:22.226 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:22 smithi103 ceph-mon[39629]: pgmap v1667: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.5 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 949 op/s; 2259/83090 objects degraded (2.719%); 18 MiB/s, 287 objects/s recovering 2024-04-23T16:45:22.248 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:45:22.276 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:22.301 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:45:22.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:22.307 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:45:22.408 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:45:22.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:22.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:22.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:22.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:22.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:22 smithi188 ceph-mon[44101]: pgmap v1667: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.5 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 949 op/s; 2259/83090 objects degraded (2.719%); 18 MiB/s, 287 objects/s recovering 2024-04-23T16:45:22.443 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:22.465 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:45:22.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:22.470 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:45:22.572 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:45:22.602 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:23.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:23 smithi148 ceph-mon[44520]: pgmap v1668: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.5 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 906 op/s; 2259/81358 objects degraded (2.777%); 16 MiB/s, 256 objects/s recovering 2024-04-23T16:45:23.433 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:45:23.458 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:23 smithi103 ceph-mon[39629]: pgmap v1668: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.5 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 906 op/s; 2259/81358 objects degraded (2.777%); 16 MiB/s, 256 objects/s recovering 2024-04-23T16:45:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:23 smithi188 ceph-mon[44101]: pgmap v1668: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.5 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 906 op/s; 2259/81358 objects degraded (2.777%); 16 MiB/s, 256 objects/s recovering 2024-04-23T16:45:24.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:24 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 2259/81358 objects degraded (2.777%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:24.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:24 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 2259/81358 objects degraded (2.777%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:24.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:24 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 2259/81358 objects degraded (2.777%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:25.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:25 smithi188 ceph-mon[44101]: pgmap v1669: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.3 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 938 op/s; 1923/76408 objects degraded (2.517%); 14 MiB/s, 225 objects/s recovering 2024-04-23T16:45:25.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:25 smithi103 ceph-mon[39629]: pgmap v1669: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.3 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 938 op/s; 1923/76408 objects degraded (2.517%); 14 MiB/s, 225 objects/s recovering 2024-04-23T16:45:25.878 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:25 smithi148 ceph-mon[44520]: pgmap v1669: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.3 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 938 op/s; 1923/76408 objects degraded (2.517%); 14 MiB/s, 225 objects/s recovering 2024-04-23T16:45:26.357 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:45:26.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:26.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:45:26.461 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:26.514 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:26.546 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:26.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:26.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:26.659 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:45:26.729 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:26.765 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:45:26.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:26.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:45:26.798 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:45:26.873 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:45:26.907 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:26.935 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:45:26.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:26.940 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:45:27.041 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:45:27.076 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:27.077 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:26 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1086031855' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:45:27.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:26 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1086031855' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:45:27.101 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:45:27.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:27.107 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:45:27.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:26 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1086031855' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:45:27.207 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:45:27.266 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:27.297 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:45:27.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:27.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:45:27.406 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:45:27.440 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:27.465 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:45:27.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:27.468 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:45:27.569 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:45:27.610 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:27.636 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:45:27.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:27.640 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:45:27.741 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:45:27.811 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:27.852 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:45:27.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:27.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:45:27.958 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:45:28.017 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:28.058 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:45:28.065 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:27 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:45:28.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:27 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1086031855' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:45:28.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:27 smithi148 ceph-mon[44520]: osdmap e595: 12 total, 11 up, 9 in 2024-04-23T16:45:28.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:27 smithi148 ceph-mon[44520]: pgmap v1671: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.2 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 965 op/s; 1252/72248 objects degraded (1.733%); 15 MiB/s, 234 objects/s recovering 2024-04-23T16:45:28.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:28.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:45:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:27 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:45:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:27 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1086031855' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:45:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:27 smithi103 ceph-mon[39629]: osdmap e595: 12 total, 11 up, 9 in 2024-04-23T16:45:28.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:27 smithi103 ceph-mon[39629]: pgmap v1671: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.2 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 965 op/s; 1252/72248 objects degraded (1.733%); 15 MiB/s, 234 objects/s recovering 2024-04-23T16:45:28.168 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:45:28.195 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:27 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:45:28.195 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:27 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1086031855' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:45:28.195 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:27 smithi188 ceph-mon[44101]: osdmap e595: 12 total, 11 up, 9 in 2024-04-23T16:45:28.195 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:27 smithi188 ceph-mon[44101]: pgmap v1671: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.2 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 965 op/s; 1252/72248 objects degraded (1.733%); 15 MiB/s, 234 objects/s recovering 2024-04-23T16:45:28.198 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:28.228 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:45:28.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:28.230 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:45:28.331 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:45:28.373 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:28.401 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:45:28.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:28.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:45:28.508 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:45:28.541 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:28.568 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:45:28.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:28.572 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:45:28.672 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:28.738 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:28.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:28.785 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:28.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:28.886 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:45:28.924 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:28.948 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:45:28.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:28.951 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:45:29.052 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:45:29.106 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:29.144 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:45:29.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:29.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:45:29.248 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:45:29.279 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:29.335 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:45:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:29.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:45:29.441 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:45:29.475 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:29.496 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:45:29.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:29.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:45:29.602 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:45:29.634 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:29.661 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:45:29.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:29.671 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:45:29.729 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:45:29.732 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:45:29.756 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:45:29.767 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:45:29.767 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:45:29.767 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-23T16:45:29.767 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-23T16:45:29.767 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:45:29.772 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:45:29.812 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:29.828 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:45:29.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:29.833 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:45:29.934 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:30.005 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:30.048 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:29 smithi103 ceph-mon[39629]: 1.4 scrub starts 2024-04-23T16:45:30.048 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:29 smithi103 ceph-mon[39629]: pgmap v1672: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.1 GiB data, 8.9 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 951 op/s; 1252/70424 objects degraded (1.778%); 14 MiB/s, 231 objects/s recovering 2024-04-23T16:45:30.048 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:29 smithi103 ceph-mon[39629]: 1.4 scrub ok 2024-04-23T16:45:30.048 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:29 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 1252/70424 objects degraded (1.778%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:30.051 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:30.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:30.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:30.167 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:45:30.194 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:29 smithi188 ceph-mon[44101]: 1.4 scrub starts 2024-04-23T16:45:30.194 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:29 smithi188 ceph-mon[44101]: pgmap v1672: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.1 GiB data, 8.9 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 951 op/s; 1252/70424 objects degraded (1.778%); 14 MiB/s, 231 objects/s recovering 2024-04-23T16:45:30.195 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:29 smithi188 ceph-mon[44101]: 1.4 scrub ok 2024-04-23T16:45:30.195 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:29 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 1252/70424 objects degraded (1.778%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:30.196 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:30.220 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:45:30.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:30.227 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:45:30.327 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:45:30.354 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:30.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:29 smithi148 ceph-mon[44520]: 1.4 scrub starts 2024-04-23T16:45:30.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:29 smithi148 ceph-mon[44520]: pgmap v1672: 27 pgs: 3 active+recovering+degraded, 24 active+clean; 2.1 GiB data, 8.9 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 951 op/s; 1252/70424 objects degraded (1.778%); 14 MiB/s, 231 objects/s recovering 2024-04-23T16:45:30.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:29 smithi148 ceph-mon[44520]: 1.4 scrub ok 2024-04-23T16:45:30.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:29 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 1252/70424 objects degraded (1.778%), 3 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:30.384 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:45:30.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:30.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:45:30.488 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:45:30.535 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:30.575 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:45:30.584 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:45:30.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:30.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:45:30.692 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:45:30.726 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:30.754 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:45:30.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:30.757 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:45:30.858 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:45:30.888 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:31.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:30 smithi188 ceph-mon[44101]: from='client.26839 v1:172.21.15.103:0/2563462438' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:45:31.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:30 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:45:31.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:30 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:45:31.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:30 smithi103 ceph-mon[39629]: from='client.26839 v1:172.21.15.103:0/2563462438' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:45:31.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:30 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:45:31.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:30 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:45:31.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:30 smithi148 ceph-mon[44520]: from='client.26839 v1:172.21.15.103:0/2563462438' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:45:31.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:30 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:45:31.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:30 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:45:31.917 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-23T16:45:32.140 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:45:32.146 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:31 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T16:45:32.146 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:31 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T16:45:32.146 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:31 smithi103 ceph-mon[39629]: pgmap v1673: 27 pgs: 2 active+recovering+degraded, 25 active+clean; 1.9 GiB data, 8.6 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 975 op/s; 371/63580 objects degraded (0.584%); 12 MiB/s, 188 objects/s recovering 2024-04-23T16:45:32.146 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:31 smithi103 ceph-mon[39629]: 7.b scrub starts 2024-04-23T16:45:32.146 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:31 smithi103 ceph-mon[39629]: 7.b scrub ok 2024-04-23T16:45:32.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:32.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:45:32.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:31 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T16:45:32.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:31 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T16:45:32.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:31 smithi188 ceph-mon[44101]: pgmap v1673: 27 pgs: 2 active+recovering+degraded, 25 active+clean; 1.9 GiB data, 8.6 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 975 op/s; 371/63580 objects degraded (0.584%); 12 MiB/s, 188 objects/s recovering 2024-04-23T16:45:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:31 smithi188 ceph-mon[44101]: 7.b scrub starts 2024-04-23T16:45:32.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:31 smithi188 ceph-mon[44101]: 7.b scrub ok 2024-04-23T16:45:32.248 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:45:32.307 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:31 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T16:45:32.307 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:31 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T16:45:32.307 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:31 smithi148 ceph-mon[44520]: pgmap v1673: 27 pgs: 2 active+recovering+degraded, 25 active+clean; 1.9 GiB data, 8.6 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 975 op/s; 371/63580 objects degraded (0.584%); 12 MiB/s, 188 objects/s recovering 2024-04-23T16:45:32.307 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:31 smithi148 ceph-mon[44520]: 7.b scrub starts 2024-04-23T16:45:32.307 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:31 smithi148 ceph-mon[44520]: 7.b scrub ok 2024-04-23T16:45:32.314 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:32.350 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:45:32.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:32.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:45:32.455 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:45:32.483 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:32.503 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:45:32.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:32.508 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:45:32.609 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:45:32.673 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:32.735 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:45:32.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:32.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:45:32.839 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:45:32.869 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:32.890 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:45:32.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:32.896 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:45:32.997 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:45:33.025 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:33.178 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:45:33.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:33.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:45:33.284 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:45:33.324 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:33.348 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:45:33.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:33.351 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:45:33.452 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:45:33.480 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:33.729 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:45:33.731 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:45:33.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:33.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:45:33.772 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1672,"stamp":"2024-04-23T16:45:28.875650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2308047416,"num_objects":35212,"num_object_clones":0,"num_object_copies":70424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35212,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":202148,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21445,"num_bytes_recovered":1405690944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162377,"ondisk_log_size":162377,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":9280616,"kb_used_data":4434004,"kb_used_omap":243,"kb_used_meta":4845900,"kb_avail":834425752,"statfs":{"total":863955320832,"available":854451970048,"internally_reserved":0,"allocated":4540420096,"data_stored":4537285619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":249416,"internal_metadata":4962202040},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,4,5,5,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-623706112,"num_objects":-9517,"num_object_clones":0,"num_object_copies":-19034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9517,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2314,"num_bytes_recovered":151650304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001906"},"pg_stats":[{"pgid":"7.d","version":"594'12761","reported_seq":12880,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-04-23T16:45:25.378960+0000","last_change":"2024-04-23T16:41:57.310289+0000","last_active":"2024-04-23T16:45:25.378960+0000","last_peered":"2024-04-23T16:45:25.378960+0000","last_clean":"2024-04-23T16:45:25.378960+0000","last_became_active":"2024-04-23T16:41:57.310025+0000","last_became_peered":"2024-04-23T16:41:57.310025+0000","last_unstale":"2024-04-23T16:45:25.378960+0000","last_undegraded":"2024-04-23T16:45:25.378960+0000","last_fullsized":"2024-04-23T16:45:25.378960+0000","mapping_epoch":540,"log_start":"550'2700","ondisk_log_start":"550'2700","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151257112,"num_objects":2309,"num_object_clones":0,"num_object_copies":4618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2309,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12761,"num_write_kb":482177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"594'12933","reported_seq":13128,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-04-23T16:45:25.769440+0000","last_change":"2024-04-23T16:43:29.012370+0000","last_active":"2024-04-23T16:45:25.769440+0000","last_peered":"2024-04-23T16:45:25.769440+0000","last_clean":"2024-04-23T16:45:25.769440+0000","last_became_active":"2024-04-23T16:41:57.310754+0000","last_became_peered":"2024-04-23T16:41:57.310754+0000","last_unstale":"2024-04-23T16:45:25.769440+0000","last_undegraded":"2024-04-23T16:45:25.769440+0000","last_fullsized":"2024-04-23T16:45:25.769440+0000","mapping_epoch":540,"log_start":"550'2900","ondisk_log_start":"550'2900","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'6945","last_scrub_stamp":"2024-04-23T16:43:29.012285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:43:29.012285+0000","objects_scrubbed":6912,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:29.994785+0000","scrub_duration":1082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144900096,"num_objects":2211,"num_object_clones":0,"num_object_copies":4422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12933,"num_write_kb":484608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"595'12824","reported_seq":13071,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-04-23T16:45:28.638210+0000","last_change":"2024-04-23T16:44:45.745602+0000","last_active":"2024-04-23T16:45:28.638210+0000","last_peered":"2024-04-23T16:45:28.638210+0000","last_clean":"2024-04-23T16:45:28.638210+0000","last_became_active":"2024-04-23T16:41:57.310189+0000","last_became_peered":"2024-04-23T16:41:57.310189+0000","last_unstale":"2024-04-23T16:45:28.638210+0000","last_undegraded":"2024-04-23T16:45:28.638210+0000","last_fullsized":"2024-04-23T16:45:28.638210+0000","mapping_epoch":540,"log_start":"550'2800","ondisk_log_start":"550'2800","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'10439","last_scrub_stamp":"2024-04-23T16:44:45.745464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:45.745464+0000","objects_scrubbed":4477,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:10.632547+0000","scrub_duration":680,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136183808,"num_objects":2078,"num_object_clones":0,"num_object_copies":4156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12824,"num_write_kb":476864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"594'12591","reported_seq":12828,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-04-23T16:45:24.472815+0000","last_change":"2024-04-23T16:44:58.743309+0000","last_active":"2024-04-23T16:45:24.472815+0000","last_peered":"2024-04-23T16:45:24.472815+0000","last_clean":"2024-04-23T16:45:24.472815+0000","last_became_active":"2024-04-23T16:41:57.309569+0000","last_became_peered":"2024-04-23T16:41:57.309569+0000","last_unstale":"2024-04-23T16:45:24.472815+0000","last_undegraded":"2024-04-23T16:45:24.472815+0000","last_fullsized":"2024-04-23T16:45:24.472815+0000","mapping_epoch":540,"log_start":"550'2500","ondisk_log_start":"550'2500","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"591'11141","last_scrub_stamp":"2024-04-23T16:44:58.743181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:58.743181+0000","objects_scrubbed":3762,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:00.499252+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150798336,"num_objects":2301,"num_object_clones":0,"num_object_copies":4602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":476544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"594'12393","reported_seq":12511,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-04-23T16:45:25.800364+0000","last_change":"2024-04-23T16:41:57.310144+0000","last_active":"2024-04-23T16:45:25.800364+0000","last_peered":"2024-04-23T16:45:25.800364+0000","last_clean":"2024-04-23T16:45:25.800364+0000","last_became_active":"2024-04-23T16:41:57.309841+0000","last_became_peered":"2024-04-23T16:41:57.309841+0000","last_unstale":"2024-04-23T16:45:25.800364+0000","last_undegraded":"2024-04-23T16:45:25.800364+0000","last_fullsized":"2024-04-23T16:45:25.800364+0000","mapping_epoch":540,"log_start":"550'2300","ondisk_log_start":"550'2300","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140705792,"num_objects":2147,"num_object_clones":0,"num_object_copies":4294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12393,"num_write_kb":465280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"594'12627","reported_seq":23713,"reported_epoch":594,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:45:25.372764+0000","last_change":"2024-04-23T16:43:08.145899+0000","last_active":"2024-04-23T16:45:25.372764+0000","last_peered":"2024-04-23T16:45:25.372764+0000","last_clean":"2024-04-23T16:42:42.970788+0000","last_became_active":"2024-04-23T16:43:08.106076+0000","last_became_peered":"2024-04-23T16:43:08.106076+0000","last_unstale":"2024-04-23T16:45:25.372764+0000","last_undegraded":"2024-04-23T16:43:07.951894+0000","last_fullsized":"2024-04-23T16:45:25.372764+0000","mapping_epoch":563,"log_start":"550'2600","ondisk_log_start":"550'2600","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138870784,"num_objects":2119,"num_object_clones":0,"num_object_copies":4238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12627,"num_write_kb":471872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5481,"num_bytes_recovered":359202816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":233},{"shards":"6,11","objects":1886}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1503,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-04-23T16:45:26.801144+0000","last_change":"2024-04-23T16:44:09.435333+0000","last_active":"2024-04-23T16:45:26.801144+0000","last_peered":"2024-04-23T16:45:26.801144+0000","last_clean":"2024-04-23T16:45:26.801144+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:45:26.801144+0000","last_undegraded":"2024-04-23T16:45:26.801144+0000","last_fullsized":"2024-04-23T16:45:26.801144+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:09.435212+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:44:09.435212+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:30.133580+0000","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"595'12865","reported_seq":13110,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-04-23T16:45:28.645758+0000","last_change":"2024-04-23T16:44:52.727725+0000","last_active":"2024-04-23T16:45:28.645758+0000","last_peered":"2024-04-23T16:45:28.645758+0000","last_clean":"2024-04-23T16:45:28.645758+0000","last_became_active":"2024-04-23T16:41:57.311753+0000","last_became_peered":"2024-04-23T16:41:57.311753+0000","last_unstale":"2024-04-23T16:45:28.645758+0000","last_undegraded":"2024-04-23T16:45:28.645758+0000","last_fullsized":"2024-04-23T16:45:28.645758+0000","mapping_epoch":540,"log_start":"550'2800","ondisk_log_start":"550'2800","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"588'10838","last_scrub_stamp":"2024-04-23T16:44:52.727632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:52.727632+0000","objects_scrubbed":4109,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:52.880867+0000","scrub_duration":730,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135069696,"num_objects":2061,"num_object_clones":0,"num_object_copies":4122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12865,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1509,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-04-23T16:45:26.807406+0000","last_change":"2024-04-23T16:44:45.850797+0000","last_active":"2024-04-23T16:45:26.807406+0000","last_peered":"2024-04-23T16:45:26.807406+0000","last_clean":"2024-04-23T16:45:26.807406+0000","last_became_active":"2024-04-23T16:43:07.962092+0000","last_became_peered":"2024-04-23T16:43:07.962092+0000","last_unstale":"2024-04-23T16:45:26.807406+0000","last_undegraded":"2024-04-23T16:45:26.807406+0000","last_fullsized":"2024-04-23T16:45:26.807406+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:45.850682+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:44:45.850682+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:12.889710+0000","scrub_duration":234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"594'12562","reported_seq":12807,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-04-23T16:45:24.423400+0000","last_change":"2024-04-23T16:44:44.578218+0000","last_active":"2024-04-23T16:45:24.423400+0000","last_peered":"2024-04-23T16:45:24.423400+0000","last_clean":"2024-04-23T16:45:24.423400+0000","last_became_active":"2024-04-23T16:41:57.305944+0000","last_became_peered":"2024-04-23T16:41:57.305944+0000","last_unstale":"2024-04-23T16:45:24.423400+0000","last_undegraded":"2024-04-23T16:45:24.423400+0000","last_fullsized":"2024-04-23T16:45:24.423400+0000","mapping_epoch":540,"log_start":"550'2500","ondisk_log_start":"550'2500","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'10405","last_scrub_stamp":"2024-04-23T16:44:44.578065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:44.578065+0000","objects_scrubbed":4454,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:01.634216+0000","scrub_duration":505,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149815296,"num_objects":2286,"num_object_clones":0,"num_object_copies":4572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12562,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1476,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-04-23T16:45:17.043752+0000","last_change":"2024-04-23T16:42:32.662620+0000","last_active":"2024-04-23T16:45:17.043752+0000","last_peered":"2024-04-23T16:45:17.043752+0000","last_clean":"2024-04-23T16:45:17.043752+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:45:17.043752+0000","last_undegraded":"2024-04-23T16:45:17.043752+0000","last_fullsized":"2024-04-23T16:45:17.043752+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:32.662476+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:42:32.662476+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"594'12913","reported_seq":13157,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-04-23T16:45:24.490353+0000","last_change":"2024-04-23T16:44:46.996868+0000","last_active":"2024-04-23T16:45:24.490353+0000","last_peered":"2024-04-23T16:45:24.490353+0000","last_clean":"2024-04-23T16:45:24.490353+0000","last_became_active":"2024-04-23T16:41:57.309858+0000","last_became_peered":"2024-04-23T16:41:57.309858+0000","last_unstale":"2024-04-23T16:45:24.490353+0000","last_undegraded":"2024-04-23T16:45:24.490353+0000","last_fullsized":"2024-04-23T16:45:24.490353+0000","mapping_epoch":540,"log_start":"550'2900","ondisk_log_start":"550'2900","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'10834","last_scrub_stamp":"2024-04-23T16:44:46.996775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:46.996775+0000","objects_scrubbed":4359,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:53.498504+0000","scrub_duration":887,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147652608,"num_objects":2253,"num_object_clones":0,"num_object_copies":4506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12913,"num_write_kb":485312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1501,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-04-23T16:45:17.041026+0000","last_change":"2024-04-23T16:43:40.258384+0000","last_active":"2024-04-23T16:45:17.041026+0000","last_peered":"2024-04-23T16:45:17.041026+0000","last_clean":"2024-04-23T16:45:17.041026+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:45:17.041026+0000","last_undegraded":"2024-04-23T16:45:17.041026+0000","last_fullsized":"2024-04-23T16:45:17.041026+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:43:40.258213+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:43:40.258213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:21.268238+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"594'12567","reported_seq":12685,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-04-23T16:45:26.462249+0000","last_change":"2024-04-23T16:41:57.310695+0000","last_active":"2024-04-23T16:45:26.462249+0000","last_peered":"2024-04-23T16:45:26.462249+0000","last_clean":"2024-04-23T16:45:26.462249+0000","last_became_active":"2024-04-23T16:41:57.310400+0000","last_became_peered":"2024-04-23T16:41:57.310400+0000","last_unstale":"2024-04-23T16:45:26.462249+0000","last_undegraded":"2024-04-23T16:45:26.462249+0000","last_fullsized":"2024-04-23T16:45:26.462249+0000","mapping_epoch":540,"log_start":"550'2500","ondisk_log_start":"550'2500","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139657216,"num_objects":2131,"num_object_clones":0,"num_object_copies":4262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12567,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1074,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-04-23T16:45:17.038376+0000","last_change":"2024-04-23T16:44:18.555667+0000","last_active":"2024-04-23T16:45:17.038376+0000","last_peered":"2024-04-23T16:45:17.038376+0000","last_clean":"2024-04-23T16:45:17.038376+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:45:17.038376+0000","last_undegraded":"2024-04-23T16:45:17.038376+0000","last_fullsized":"2024-04-23T16:45:17.038376+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:18.555538+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:44:18.555538+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:35.829709+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"594'12450","reported_seq":12568,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-04-23T16:45:25.384429+0000","last_change":"2024-04-23T16:41:57.306526+0000","last_active":"2024-04-23T16:45:25.384429+0000","last_peered":"2024-04-23T16:45:25.384429+0000","last_clean":"2024-04-23T16:45:25.384429+0000","last_became_active":"2024-04-23T16:41:57.306229+0000","last_became_peered":"2024-04-23T16:41:57.306229+0000","last_unstale":"2024-04-23T16:45:25.384429+0000","last_undegraded":"2024-04-23T16:45:25.384429+0000","last_fullsized":"2024-04-23T16:45:25.384429+0000","mapping_epoch":540,"log_start":"550'2400","ondisk_log_start":"550'2400","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147718144,"num_objects":2254,"num_object_clones":0,"num_object_copies":4508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12450,"num_write_kb":470528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1031,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-04-23T16:45:26.807707+0000","last_change":"2024-04-23T16:44:07.230780+0000","last_active":"2024-04-23T16:45:26.807707+0000","last_peered":"2024-04-23T16:45:26.807707+0000","last_clean":"2024-04-23T16:45:26.807707+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:45:26.807707+0000","last_undegraded":"2024-04-23T16:45:26.807707+0000","last_fullsized":"2024-04-23T16:45:26.807707+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:07.230640+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:44:07.230640+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:31.040757+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.0","version":"595'12667","reported_seq":12907,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-04-23T16:45:27.759562+0000","last_change":"2024-04-23T16:45:00.368292+0000","last_active":"2024-04-23T16:45:27.759562+0000","last_peered":"2024-04-23T16:45:27.759562+0000","last_clean":"2024-04-23T16:45:27.759562+0000","last_became_active":"2024-04-23T16:41:57.311479+0000","last_became_peered":"2024-04-23T16:41:57.311479+0000","last_unstale":"2024-04-23T16:45:27.759562+0000","last_undegraded":"2024-04-23T16:45:27.759562+0000","last_fullsized":"2024-04-23T16:45:27.759562+0000","mapping_epoch":540,"log_start":"550'2600","ondisk_log_start":"550'2600","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"591'11135","last_scrub_stamp":"2024-04-23T16:45:00.368005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:00.368005+0000","objects_scrubbed":3585,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:06.315750+0000","scrub_duration":778,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133234688,"num_objects":2033,"num_object_clones":0,"num_object_copies":4066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12667,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1511,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-04-23T16:45:26.801313+0000","last_change":"2024-04-23T16:44:11.349248+0000","last_active":"2024-04-23T16:45:26.801313+0000","last_peered":"2024-04-23T16:45:26.801313+0000","last_clean":"2024-04-23T16:45:26.801313+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:45:26.801313+0000","last_undegraded":"2024-04-23T16:45:26.801313+0000","last_fullsized":"2024-04-23T16:45:26.801313+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:11.348079+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:44:11.348079+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:28.153743+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.1","version":"594'12378","reported_seq":22625,"reported_epoch":594,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:45:24.577237+0000","last_change":"2024-04-23T16:43:08.081567+0000","last_active":"2024-04-23T16:45:24.577237+0000","last_peered":"2024-04-23T16:45:24.577237+0000","last_clean":"2024-04-23T16:42:42.976600+0000","last_became_active":"2024-04-23T16:43:08.067887+0000","last_became_peered":"2024-04-23T16:43:08.067887+0000","last_unstale":"2024-04-23T16:45:24.577237+0000","last_undegraded":"2024-04-23T16:43:07.950381+0000","last_fullsized":"2024-04-23T16:45:24.577237+0000","mapping_epoch":563,"log_start":"550'2300","ondisk_log_start":"550'2300","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147980288,"num_objects":2258,"num_object_clones":0,"num_object_copies":4516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12378,"num_write_kb":468352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5062,"num_bytes_recovered":331743232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":526},{"shards":"4,6","objects":1732}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1493,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-04-23T16:45:17.053639+0000","last_change":"2024-04-23T16:43:07.918302+0000","last_active":"2024-04-23T16:45:17.053639+0000","last_peered":"2024-04-23T16:45:17.053639+0000","last_clean":"2024-04-23T16:45:17.053639+0000","last_became_active":"2024-04-23T16:43:07.918042+0000","last_became_peered":"2024-04-23T16:43:07.918042+0000","last_unstale":"2024-04-23T16:45:17.053639+0000","last_undegraded":"2024-04-23T16:45:17.053639+0000","last_fullsized":"2024-04-23T16:45:17.053639+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:33.716343+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:42:33.716343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"594'12647","reported_seq":23256,"reported_epoch":594,"state":"active+recovering+degraded","last_fresh":"2024-04-23T16:45:25.383755+0000","last_change":"2024-04-23T16:43:08.363306+0000","last_active":"2024-04-23T16:45:25.383755+0000","last_peered":"2024-04-23T16:45:25.383755+0000","last_clean":"2024-04-23T16:42:42.950766+0000","last_became_active":"2024-04-23T16:43:08.333793+0000","last_became_peered":"2024-04-23T16:43:08.333793+0000","last_unstale":"2024-04-23T16:45:25.383755+0000","last_undegraded":"2024-04-23T16:43:07.950356+0000","last_fullsized":"2024-04-23T16:45:25.383755+0000","mapping_epoch":563,"log_start":"550'2600","ondisk_log_start":"550'2600","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142278656,"num_objects":2171,"num_object_clones":0,"num_object_copies":4342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":474176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5244,"num_bytes_recovered":343670784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":493},{"shards":"6,7","objects":1678}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1386,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-04-23T16:45:17.037260+0000","last_change":"2024-04-23T16:41:47.722866+0000","last_active":"2024-04-23T16:45:17.037260+0000","last_peered":"2024-04-23T16:45:17.037260+0000","last_clean":"2024-04-23T16:45:17.037260+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:45:17.037260+0000","last_undegraded":"2024-04-23T16:45:17.037260+0000","last_fullsized":"2024-04-23T16:45:17.037260+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:41:47.722762+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"594'12564","reported_seq":24003,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-04-23T16:45:24.550394+0000","last_change":"2024-04-23T16:45:18.915577+0000","last_active":"2024-04-23T16:45:24.550394+0000","last_peered":"2024-04-23T16:45:24.550394+0000","last_clean":"2024-04-23T16:45:24.550394+0000","last_became_active":"2024-04-23T16:43:08.109240+0000","last_became_peered":"2024-04-23T16:43:08.109240+0000","last_unstale":"2024-04-23T16:45:24.550394+0000","last_undegraded":"2024-04-23T16:45:24.550394+0000","last_fullsized":"2024-04-23T16:45:24.550394+0000","mapping_epoch":563,"log_start":"550'2500","ondisk_log_start":"550'2500","created":540,"last_epoch_clean":594,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:41:56.284274+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154664960,"num_objects":2360,"num_object_clones":0,"num_object_copies":4720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12564,"num_write_kb":477568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5656,"num_bytes_recovered":370671616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1307,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-04-23T16:45:17.045702+0000","last_change":"2024-04-23T16:41:49.835121+0000","last_active":"2024-04-23T16:45:17.045702+0000","last_peered":"2024-04-23T16:45:17.045702+0000","last_clean":"2024-04-23T16:45:17.045702+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:45:17.045702+0000","last_undegraded":"2024-04-23T16:45:17.045702+0000","last_fullsized":"2024-04-23T16:45:17.045702+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:41:49.834374+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"594'12406","reported_seq":12648,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-04-23T16:45:24.486039+0000","last_change":"2024-04-23T16:44:45.847313+0000","last_active":"2024-04-23T16:45:24.486039+0000","last_peered":"2024-04-23T16:45:24.486039+0000","last_clean":"2024-04-23T16:45:24.486039+0000","last_became_active":"2024-04-23T16:41:57.309349+0000","last_became_peered":"2024-04-23T16:41:57.309349+0000","last_unstale":"2024-04-23T16:45:24.486039+0000","last_undegraded":"2024-04-23T16:45:24.486039+0000","last_fullsized":"2024-04-23T16:45:24.486039+0000","mapping_epoch":540,"log_start":"550'2400","ondisk_log_start":"550'2400","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'10393","last_scrub_stamp":"2024-04-23T16:44:45.847160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:45.847160+0000","objects_scrubbed":4276,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:45:47.079172+0000","scrub_duration":795,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146669568,"num_objects":2238,"num_object_clones":0,"num_object_copies":4476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12406,"num_write_kb":468608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1476,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-04-23T16:45:17.042549+0000","last_change":"2024-04-23T16:42:33.628340+0000","last_active":"2024-04-23T16:45:17.042549+0000","last_peered":"2024-04-23T16:45:17.042549+0000","last_clean":"2024-04-23T16:45:17.042549+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:45:17.042549+0000","last_undegraded":"2024-04-23T16:45:17.042549+0000","last_fullsized":"2024-04-23T16:45:17.042549+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:42:33.628217+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:42:33.628217+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2307457048,"num_objects":35210,"num_object_clones":0,"num_object_copies":70420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35210,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":202148,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21443,"num_bytes_recovered":1405288448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5895430144,"data_stored":5895422000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1474560,"data_stored":1460352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057002,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1335712,"kb_used_data":695064,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92409440,"statfs":{"total":95995035648,"available":94627266560,"internally_reserved":0,"allocated":711745536,"data_stored":711400973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":656013764},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351344,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612982,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1074992,"kb_used_data":530148,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":92670160,"statfs":{"total":95995035648,"available":94894243840,"internally_reserved":0,"allocated":542871552,"data_stored":542520935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":557906368},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841914,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":640504,"kb_used_data":266412,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93104648,"statfs":{"total":95995035648,"available":95339159552,"internally_reserved":0,"allocated":272805888,"data_stored":272451719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168959,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":819424,"kb_used_data":246808,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92925728,"statfs":{"total":95995035648,"available":95155945472,"internally_reserved":0,"allocated":252731392,"data_stored":252386829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":586349002},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463300,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1204772,"kb_used_data":686044,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":92540380,"statfs":{"total":95995035648,"available":94761349120,"internally_reserved":0,"allocated":702509056,"data_stored":702160421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":530905542},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,4,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137441,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":529636,"kb_used_data":294876,"kb_used_omap":231,"kb_used_meta":234520,"kb_avail":93215516,"statfs":{"total":95995035648,"available":95452688384,"internally_reserved":0,"allocated":301953024,"data_stored":301604389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":240149392},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738786,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1164768,"kb_used_data":587992,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92580384,"statfs":{"total":95995035648,"available":94802313216,"internally_reserved":0,"allocated":602103808,"data_stored":601759245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269457,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921164,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1456288,"kb_used_data":725272,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":92288864,"statfs":{"total":95995035648,"available":94503796736,"internally_reserved":0,"allocated":742678528,"data_stored":742333965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,2,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757640,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1054520,"kb_used_data":401388,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":92690632,"statfs":{"total":95995035648,"available":94915207168,"internally_reserved":0,"allocated":411021312,"data_stored":410667143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":594935808,"data_stored":594935808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":293474304,"data_stored":293470232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1411842048,"data_stored":1411842048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":741933056,"data_stored":741933056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":408748032,"data_stored":408748032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":693768192,"data_stored":693764120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":250937344,"data_stored":250937344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":268828672,"data_stored":268828672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532414464,"data_stored":532414464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":698548224,"data_stored":698548224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:45:33.774 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.c is not active+clean 2024-04-23T16:45:33.774 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.c', 'version': "594'12627", 'reported_seq': 23713, 'reported_epoch': 594, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:45:25.372764+0000', 'last_change': '2024-04-23T16:43:08.145899+0000', 'last_active': '2024-04-23T16:45:25.372764+0000', 'last_peered': '2024-04-23T16:45:25.372764+0000', 'last_clean': '2024-04-23T16:42:42.970788+0000', 'last_became_active': '2024-04-23T16:43:08.106076+0000', 'last_became_peered': '2024-04-23T16:43:08.106076+0000', 'last_unstale': '2024-04-23T16:45:25.372764+0000', 'last_undegraded': '2024-04-23T16:43:07.951894+0000', 'last_fullsized': '2024-04-23T16:45:25.372764+0000', 'mapping_epoch': 563, 'log_start': "550'2600", 'ondisk_log_start': "550'2600", 'created': 540, 'last_epoch_clean': 541, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_clean_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 138870784, 'num_objects': 2119, 'num_object_clones': 0, 'num_object_copies': 4238, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12627, 'num_write_kb': 471872, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5481, 'num_bytes_recovered': 359202816, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 11], 'acting': [6, 11], 'avail_no_missing': ['6'], 'object_location_counts': [{'shards': '6', 'objects': 233}, {'shards': '6,11', 'objects': 1886}], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-23T16:45:33.774 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.1 is not active+clean 2024-04-23T16:45:33.774 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.1', 'version': "594'12378", 'reported_seq': 22625, 'reported_epoch': 594, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:45:24.577237+0000', 'last_change': '2024-04-23T16:43:08.081567+0000', 'last_active': '2024-04-23T16:45:24.577237+0000', 'last_peered': '2024-04-23T16:45:24.577237+0000', 'last_clean': '2024-04-23T16:42:42.976600+0000', 'last_became_active': '2024-04-23T16:43:08.067887+0000', 'last_became_peered': '2024-04-23T16:43:08.067887+0000', 'last_unstale': '2024-04-23T16:45:24.577237+0000', 'last_undegraded': '2024-04-23T16:43:07.950381+0000', 'last_fullsized': '2024-04-23T16:45:24.577237+0000', 'mapping_epoch': 563, 'log_start': "550'2300", 'ondisk_log_start': "550'2300", 'created': 540, 'last_epoch_clean': 541, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_clean_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 147980288, 'num_objects': 2258, 'num_object_clones': 0, 'num_object_copies': 4516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2258, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12378, 'num_write_kb': 468352, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5062, 'num_bytes_recovered': 331743232, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': ['4'], 'object_location_counts': [{'shards': '4', 'objects': 526}, {'shards': '4,6', 'objects': 1732}], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-23T16:45:33.775 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.2 is not active+clean 2024-04-23T16:45:33.775 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.2', 'version': "594'12647", 'reported_seq': 23256, 'reported_epoch': 594, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-23T16:45:25.383755+0000', 'last_change': '2024-04-23T16:43:08.363306+0000', 'last_active': '2024-04-23T16:45:25.383755+0000', 'last_peered': '2024-04-23T16:45:25.383755+0000', 'last_clean': '2024-04-23T16:42:42.950766+0000', 'last_became_active': '2024-04-23T16:43:08.333793+0000', 'last_became_peered': '2024-04-23T16:43:08.333793+0000', 'last_unstale': '2024-04-23T16:45:25.383755+0000', 'last_undegraded': '2024-04-23T16:43:07.950356+0000', 'last_fullsized': '2024-04-23T16:45:25.383755+0000', 'mapping_epoch': 563, 'log_start': "550'2600", 'ondisk_log_start': "550'2600", 'created': 540, 'last_epoch_clean': 541, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'last_clean_scrub_stamp': '2024-04-23T16:41:56.284274+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 142278656, 'num_objects': 2171, 'num_object_clones': 0, 'num_object_copies': 4342, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 493, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2171, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12647, 'num_write_kb': 474176, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5244, 'num_bytes_recovered': 343670784, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': ['6'], 'object_location_counts': [{'shards': '6', 'objects': 493}, {'shards': '6,7', 'objects': 1678}], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-23T16:45:33.775 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-23T16:45:33.782 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:45:33.838 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:33.883 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:33.922 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:33.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:33.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:34.029 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:45:34.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:33 smithi103 ceph-mon[39629]: pgmap v1674: 27 pgs: 2 active+recovering+degraded, 25 active+clean; 1.9 GiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 982 op/s; 371/61696 objects degraded (0.601%); 12 MiB/s, 188 objects/s recovering 2024-04-23T16:45:34.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:33 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:45:34.051 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:33 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:45:34.065 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:34.088 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:45:34.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:34.093 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:45:34.194 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:45:34.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:33 smithi188 ceph-mon[44101]: pgmap v1674: 27 pgs: 2 active+recovering+degraded, 25 active+clean; 1.9 GiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 982 op/s; 371/61696 objects degraded (0.601%); 12 MiB/s, 188 objects/s recovering 2024-04-23T16:45:34.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:33 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:45:34.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:33 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:45:34.229 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:34.256 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:45:34.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:34.261 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:45:34.362 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:45:34.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:33 smithi148 ceph-mon[44520]: pgmap v1674: 27 pgs: 2 active+recovering+degraded, 25 active+clean; 1.9 GiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 982 op/s; 371/61696 objects degraded (0.601%); 12 MiB/s, 188 objects/s recovering 2024-04-23T16:45:34.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:33 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:45:34.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:33 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:45:34.414 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:34.448 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:45:34.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:34.451 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:45:34.552 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:45:34.595 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:34.625 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:45:34.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:34.628 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:45:34.729 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:45:34.760 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:34.952 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:34 smithi188 ceph-mon[44101]: 1.a scrub starts 2024-04-23T16:45:34.953 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:34 smithi188 ceph-mon[44101]: 1.a scrub ok 2024-04-23T16:45:34.953 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:34 smithi188 ceph-mon[44101]: 1.5 scrub starts 2024-04-23T16:45:34.953 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:34 smithi188 ceph-mon[44101]: 7.1 scrub starts 2024-04-23T16:45:34.953 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:34 smithi188 ceph-mon[44101]: 1.5 scrub ok 2024-04-23T16:45:34.953 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:34 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 371/61696 objects degraded (0.601%), 2 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:35.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:34 smithi103 ceph-mon[39629]: 1.a scrub starts 2024-04-23T16:45:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:34 smithi103 ceph-mon[39629]: 1.a scrub ok 2024-04-23T16:45:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:34 smithi103 ceph-mon[39629]: 1.5 scrub starts 2024-04-23T16:45:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:34 smithi103 ceph-mon[39629]: 7.1 scrub starts 2024-04-23T16:45:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:34 smithi103 ceph-mon[39629]: 1.5 scrub ok 2024-04-23T16:45:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:34 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 371/61696 objects degraded (0.601%), 2 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:35.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:34 smithi148 ceph-mon[44520]: 1.a scrub starts 2024-04-23T16:45:35.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:34 smithi148 ceph-mon[44520]: 1.a scrub ok 2024-04-23T16:45:35.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:34 smithi148 ceph-mon[44520]: 1.5 scrub starts 2024-04-23T16:45:35.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:34 smithi148 ceph-mon[44520]: 7.1 scrub starts 2024-04-23T16:45:35.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:34 smithi148 ceph-mon[44520]: 1.5 scrub ok 2024-04-23T16:45:35.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:34 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 371/61696 objects degraded (0.601%), 2 pgs degraded (PG_DEGRADED) 2024-04-23T16:45:36.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:35 smithi188 ceph-mon[44101]: 7.1 scrub ok 2024-04-23T16:45:36.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:35 smithi188 ceph-mon[44101]: pgmap v1675: 27 pgs: 1 active+recovering+degraded, 26 active+clean; 1.7 GiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 996 op/s; 175/56468 objects degraded (0.310%); 11 MiB/s, 174 objects/s recovering 2024-04-23T16:45:36.303 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:35 smithi103 ceph-mon[39629]: 7.1 scrub ok 2024-04-23T16:45:36.303 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:35 smithi103 ceph-mon[39629]: pgmap v1675: 27 pgs: 1 active+recovering+degraded, 26 active+clean; 1.7 GiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 996 op/s; 175/56468 objects degraded (0.310%); 11 MiB/s, 174 objects/s recovering 2024-04-23T16:45:36.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:35 smithi148 ceph-mon[44520]: 7.1 scrub ok 2024-04-23T16:45:36.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:35 smithi148 ceph-mon[44520]: pgmap v1675: 27 pgs: 1 active+recovering+degraded, 26 active+clean; 1.7 GiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 996 op/s; 175/56468 objects degraded (0.310%); 11 MiB/s, 174 objects/s recovering 2024-04-23T16:45:36.618 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:45:36.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:36.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:45:36.721 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:45:36.754 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:37.257 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:36 smithi103 ceph-mon[39629]: 7.8 scrub starts 2024-04-23T16:45:37.257 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:36 smithi103 ceph-mon[39629]: 7.3 scrub starts 2024-04-23T16:45:37.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:36 smithi103 ceph-mon[39629]: 7.5 scrub starts 2024-04-23T16:45:37.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:36 smithi103 ceph-mon[39629]: 7.8 scrub ok 2024-04-23T16:45:37.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:36 smithi103 ceph-mon[39629]: 7.3 scrub ok 2024-04-23T16:45:37.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:36 smithi103 ceph-mon[39629]: 7.5 scrub ok 2024-04-23T16:45:37.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4043655799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:45:37.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:36 smithi148 ceph-mon[44520]: 7.8 scrub starts 2024-04-23T16:45:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:36 smithi148 ceph-mon[44520]: 7.3 scrub starts 2024-04-23T16:45:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:36 smithi148 ceph-mon[44520]: 7.5 scrub starts 2024-04-23T16:45:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:36 smithi148 ceph-mon[44520]: 7.8 scrub ok 2024-04-23T16:45:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:36 smithi148 ceph-mon[44520]: 7.3 scrub ok 2024-04-23T16:45:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:36 smithi148 ceph-mon[44520]: 7.5 scrub ok 2024-04-23T16:45:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4043655799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:45:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:36 smithi188 ceph-mon[44101]: 7.8 scrub starts 2024-04-23T16:45:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:36 smithi188 ceph-mon[44101]: 7.3 scrub starts 2024-04-23T16:45:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:36 smithi188 ceph-mon[44101]: 7.5 scrub starts 2024-04-23T16:45:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:36 smithi188 ceph-mon[44101]: 7.8 scrub ok 2024-04-23T16:45:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:36 smithi188 ceph-mon[44101]: 7.3 scrub ok 2024-04-23T16:45:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:36 smithi188 ceph-mon[44101]: 7.5 scrub ok 2024-04-23T16:45:37.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4043655799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:45:37.807 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:45:37.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:37.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:45:37.911 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:45:37.941 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:37.965 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:45:37.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:37.970 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:45:38.071 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:45:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:37 smithi103 ceph-mon[39629]: 7.c scrub starts 2024-04-23T16:45:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:37 smithi103 ceph-mon[39629]: pgmap v1676: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.6 GiB data, 7.9 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 7.8 MiB/s, 124 objects/s recovering 2024-04-23T16:45:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:37 smithi103 ceph-mon[39629]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 175/56468 objects degraded (0.310%), 1 pg degraded) 2024-04-23T16:45:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:37 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:45:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:37 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:45:38.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:37 smithi103 ceph-mon[39629]: 7.c scrub ok 2024-04-23T16:45:38.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.107 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:38.132 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:45:38.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:38.141 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:45:38.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:37 smithi188 ceph-mon[44101]: 7.c scrub starts 2024-04-23T16:45:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:37 smithi188 ceph-mon[44101]: pgmap v1676: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.6 GiB data, 7.9 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 7.8 MiB/s, 124 objects/s recovering 2024-04-23T16:45:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:37 smithi188 ceph-mon[44101]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 175/56468 objects degraded (0.310%), 1 pg degraded) 2024-04-23T16:45:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:37 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:45:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:37 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:45:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:37 smithi188 ceph-mon[44101]: 7.c scrub ok 2024-04-23T16:45:38.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.242 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:45:38.277 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:38.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:37 smithi148 ceph-mon[44520]: 7.c scrub starts 2024-04-23T16:45:38.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:37 smithi148 ceph-mon[44520]: pgmap v1676: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.6 GiB data, 7.9 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 7.8 MiB/s, 124 objects/s recovering 2024-04-23T16:45:38.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:37 smithi148 ceph-mon[44520]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 175/56468 objects degraded (0.310%), 1 pg degraded) 2024-04-23T16:45:38.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:37 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:45:38.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:37 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:45:38.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:37 smithi148 ceph-mon[44520]: 7.c scrub ok 2024-04-23T16:45:38.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:38.380 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:45:38.381 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:45:38.381 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:45:38.381 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:45:38.381 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:45:38.381 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:45:38.381 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:45:38.381 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:45:38.381 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:45:38.381 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:45:38.381 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:45:38.381 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:45:38.382 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:45:38.382 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:45:38.382 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:45:38.382 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:45:38.382 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:45:38.382 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:45:38.382 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:45:38.382 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:45:38.382 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:45:38.382 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:45:38.382 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:45:38.611 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:45:38.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:38.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:45:38.714 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:45:38.747 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:38.778 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:45:38.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:38.784 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:45:38.886 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:45:38.921 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:38.959 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:45:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:38.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:45:39.066 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:45:39.100 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:39.197 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:45:39.197 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:45:39.202 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:45:39.204 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:39.204 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:39.204 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:39.204 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:38 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:45:39.204 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:38 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:45:39.204 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:38 smithi103 ceph-mon[39629]: 7.2 scrub starts 2024-04-23T16:45:39.204 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:38 smithi103 ceph-mon[39629]: 7.2 scrub ok 2024-04-23T16:45:39.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:39.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:45:39.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:39.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:39.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:39.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:38 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:45:39.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:38 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:45:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:38 smithi188 ceph-mon[44101]: 7.2 scrub starts 2024-04-23T16:45:39.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:38 smithi188 ceph-mon[44101]: 7.2 scrub ok 2024-04-23T16:45:39.306 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:39.357 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:39.357 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:39.357 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:39.357 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:38 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:45:39.357 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:38 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:45:39.357 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:38 smithi148 ceph-mon[44520]: 7.2 scrub starts 2024-04-23T16:45:39.357 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:38 smithi148 ceph-mon[44520]: 7.2 scrub ok 2024-04-23T16:45:39.360 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:39.409 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:39.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:39.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:39.521 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:45:39.551 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:39.577 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:45:39.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:39.583 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:45:39.684 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:45:39.723 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:39.747 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:45:39.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:39.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:45:39.850 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:45:39.883 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:39.914 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:45:39.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:39.917 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:45:40.019 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:45:40.050 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:40.071 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:45:40.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:40.074 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:45:40.121 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:45:40.165 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:45:40.165 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:45:40.175 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:45:40.204 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:39 smithi188 ceph-mon[44101]: 1.9 scrub starts 2024-04-23T16:45:40.204 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:39 smithi188 ceph-mon[44101]: 1.9 scrub ok 2024-04-23T16:45:40.204 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:39 smithi188 ceph-mon[44101]: pgmap v1677: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.5 GiB data, 7.7 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 940 op/s; 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:45:40.211 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:40.242 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:45:40.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:40.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:45:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:39 smithi103 ceph-mon[39629]: 1.9 scrub starts 2024-04-23T16:45:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:39 smithi103 ceph-mon[39629]: 1.9 scrub ok 2024-04-23T16:45:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:39 smithi103 ceph-mon[39629]: pgmap v1677: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.5 GiB data, 7.7 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 940 op/s; 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:45:40.350 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:45:40.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:39 smithi148 ceph-mon[44520]: 1.9 scrub starts 2024-04-23T16:45:40.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:39 smithi148 ceph-mon[44520]: 1.9 scrub ok 2024-04-23T16:45:40.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:39 smithi148 ceph-mon[44520]: pgmap v1677: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.5 GiB data, 7.7 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 940 op/s; 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:45:40.394 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:40.439 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:45:40.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:40.443 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:45:40.544 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:40.599 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:40.628 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:40.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:40.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:40.738 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:45:40.797 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:41.004 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:41 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:45:41.068 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:45:41.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:41.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:45:41.175 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:45:41.212 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:41.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:41 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:45:41.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:41 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:45:41.719 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-23T16:45:42.017 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:45:42.166 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:45:42.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:42 smithi103 ceph-mon[39629]: pgmap v1678: 27 pgs: 27 active+clean; 1.3 GiB data, 7.4 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.19k op/s; 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:45:42.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:42 smithi103 ceph-mon[39629]: 7.d scrub starts 2024-04-23T16:45:42.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:42 smithi103 ceph-mon[39629]: 7.d scrub ok 2024-04-23T16:45:42.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:42 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3342600498' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:45:42.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:42 smithi148 ceph-mon[44520]: pgmap v1678: 27 pgs: 27 active+clean; 1.3 GiB data, 7.4 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.19k op/s; 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:45:42.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:42 smithi148 ceph-mon[44520]: 7.d scrub starts 2024-04-23T16:45:42.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:42 smithi148 ceph-mon[44520]: 7.d scrub ok 2024-04-23T16:45:42.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:42 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3342600498' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:45:42.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:42 smithi188 ceph-mon[44101]: pgmap v1678: 27 pgs: 27 active+clean; 1.3 GiB data, 7.4 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.19k op/s; 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:45:42.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:42 smithi188 ceph-mon[44101]: 7.d scrub starts 2024-04-23T16:45:42.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:42 smithi188 ceph-mon[44101]: 7.d scrub ok 2024-04-23T16:45:42.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:42 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3342600498' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:45:43.168 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:45:43.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:43.178 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:45:43.279 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:45:43.326 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:43 smithi148 ceph-mon[44520]: 7.7 scrub starts 2024-04-23T16:45:43.326 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:43 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:45:43.326 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:43 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:45:43.326 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:43 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3342600498' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:45:43.326 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:43 smithi148 ceph-mon[44520]: osdmap e596: 12 total, 11 up, 9 in 2024-04-23T16:45:43.326 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:43 smithi148 ceph-mon[44520]: 7.7 scrub ok 2024-04-23T16:45:43.326 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:43 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:45:43.326 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:43 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:45:43.327 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:43.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:43 smithi103 ceph-mon[39629]: 7.7 scrub starts 2024-04-23T16:45:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:43 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:45:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:43 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:45:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:43 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3342600498' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:45:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:43 smithi103 ceph-mon[39629]: osdmap e596: 12 total, 11 up, 9 in 2024-04-23T16:45:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:43 smithi103 ceph-mon[39629]: 7.7 scrub ok 2024-04-23T16:45:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:43 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:45:43.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:43 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:45:43.374 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:45:43.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:43.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:45:43.397 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:43 smithi188 ceph-mon[44101]: 7.7 scrub starts 2024-04-23T16:45:43.397 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:43 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:45:43.397 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:43 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:45:43.398 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:43 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3342600498' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:45:43.398 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:43 smithi188 ceph-mon[44101]: osdmap e596: 12 total, 11 up, 9 in 2024-04-23T16:45:43.398 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:43 smithi188 ceph-mon[44101]: 7.7 scrub ok 2024-04-23T16:45:43.398 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:43 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:45:43.398 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:43 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:45:43.479 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:43.519 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:43.568 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:43.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:43.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:43.674 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:45:43.675 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:45:43.712 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:44.167 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:44 smithi103 ceph-mon[39629]: pgmap v1680: 27 pgs: 27 active+clean; 1.2 GiB data, 7.2 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s; 2.3 MiB/s, 37 objects/s recovering 2024-04-23T16:45:44.274 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:44 smithi188 ceph-mon[44101]: pgmap v1680: 27 pgs: 27 active+clean; 1.2 GiB data, 7.2 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s; 2.3 MiB/s, 37 objects/s recovering 2024-04-23T16:45:44.276 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:45:44.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:44.281 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:45:44.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:44 smithi148 ceph-mon[44520]: pgmap v1680: 27 pgs: 27 active+clean; 1.2 GiB data, 7.2 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s; 2.3 MiB/s, 37 objects/s recovering 2024-04-23T16:45:44.381 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:45:44.414 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:45.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:45 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/927401693' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:45:45.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:45 smithi103 ceph-mon[39629]: pgmap v1681: 27 pgs: 27 active+clean; 1.1 GiB data, 6.9 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s; 1.1 MiB/s, 17 objects/s recovering 2024-04-23T16:45:45.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:45 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/927401693' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:45:45.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:45 smithi148 ceph-mon[44520]: pgmap v1681: 27 pgs: 27 active+clean; 1.1 GiB data, 6.9 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s; 1.1 MiB/s, 17 objects/s recovering 2024-04-23T16:45:45.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:45 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/927401693' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:45:45.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:45 smithi188 ceph-mon[44101]: pgmap v1681: 27 pgs: 27 active+clean; 1.1 GiB data, 6.9 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s; 1.1 MiB/s, 17 objects/s recovering 2024-04-23T16:45:45.756 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:45:45.757 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2126,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":596,"num_osds":12,"num_up_osds":11,"osd_up_since":1713890562,"num_in_osds":9,"osd_in_since":1713890586,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":27}],"num_pgs":27,"num_pools":2,"num_objects":19938,"data_bytes":1307050552,"bytes_used":7771049984,"bytes_avail":856184270848,"bytes_total":863955320832,"recovering_objects_per_sec":37,"recovering_bytes_per_sec":2430901,"recovering_keys_per_sec":0,"num_objects_recovered":371,"num_bytes_recovered":24313856,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1090},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.103:8443/"}},"servicemap":{"epoch":46,"modified":"2024-04-23T16:44:02.858545+0000","services":{}},"progress_events":{}} 2024-04-23T16:45:46.070 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:45:46.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:46.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:45:46.174 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:45:46.217 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:46.263 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:45:46.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:46.269 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:45:46.369 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:46.431 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:46.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:46 smithi188 ceph-mon[44101]: 1.7 scrub starts 2024-04-23T16:45:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:46 smithi188 ceph-mon[44101]: 1.7 scrub ok 2024-04-23T16:45:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:46 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:45:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:46 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:45:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:46 smithi188 ceph-mon[44101]: osdmap e597: 12 total, 11 up, 9 in 2024-04-23T16:45:46.493 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:46.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:46.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:46.609 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:45:46.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:46 smithi148 ceph-mon[44520]: 1.7 scrub starts 2024-04-23T16:45:46.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:46 smithi148 ceph-mon[44520]: 1.7 scrub ok 2024-04-23T16:45:46.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:46 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:45:46.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:46 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:45:46.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:46 smithi148 ceph-mon[44520]: osdmap e597: 12 total, 11 up, 9 in 2024-04-23T16:45:46.667 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:46.774 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:46 smithi103 ceph-mon[39629]: 1.7 scrub starts 2024-04-23T16:45:46.775 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:46 smithi103 ceph-mon[39629]: 1.7 scrub ok 2024-04-23T16:45:46.775 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:46 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:45:46.775 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:46 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:45:46.775 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:46 smithi103 ceph-mon[39629]: osdmap e597: 12 total, 11 up, 9 in 2024-04-23T16:45:46.807 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-23T16:45:46.807 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:45:47.256 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:45:47.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:47.263 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:45:47.365 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:47.381 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:45:47.416 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:47.456 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:47.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:47.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:47.563 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:45:47.593 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:47.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:47 smithi103 ceph-mon[39629]: pgmap v1683: 27 pgs: 27 active+clean; 929 MiB data, 6.7 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:45:47.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:47 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:45:47.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:47 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:45:47.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:47 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:45:47.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:47 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:45:47.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:47 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:45:47.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:47 smithi188 ceph-mon[44101]: pgmap v1683: 27 pgs: 27 active+clean; 929 MiB data, 6.7 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:45:47.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:47 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:45:47.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:47 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:45:47.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:47 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:45:47.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:47 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:45:47.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:47 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:45:47.724 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:45:47.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:47.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:45:47.829 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:45:47.857 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:47.876 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:47 smithi148 ceph-mon[44520]: pgmap v1683: 27 pgs: 27 active+clean; 929 MiB data, 6.7 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-23T16:45:47.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:47 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:45:47.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:47 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:45:47.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:47 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:45:47.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:47 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:45:47.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:47 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:45:47.881 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:45:47.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:47.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:45:47.987 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:45:48.041 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:48.075 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:45:48.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:48.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:45:48.187 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:48.232 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:48.266 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:48.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:48.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:48.289 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:45:48.376 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:45:48.411 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:48.441 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:45:48.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:48.445 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:45:48.538 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:45:48.538 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:45:48.546 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:45:48.584 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:48.606 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:45:48.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:48.610 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:45:48.710 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:45:48.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:48 smithi188 ceph-mon[44101]: 7.6 deep-scrub starts 2024-04-23T16:45:48.756 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:48.764 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:48 smithi148 ceph-mon[44520]: 7.6 deep-scrub starts 2024-04-23T16:45:48.792 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:45:48.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:48.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:45:48.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:48 smithi103 ceph-mon[39629]: 7.6 deep-scrub starts 2024-04-23T16:45:48.903 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:48.953 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:48.983 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:48.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:48.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:49.091 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:45:49.120 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:49.124 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:45:49.140 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:45:49.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:49.143 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:45:49.166 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:45:49.244 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:49.305 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:49.347 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:49.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:49.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:49.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:49 smithi188 ceph-mon[44101]: pgmap v1684: 27 pgs: 27 active+clean; 866 MiB data, 6.5 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 889 op/s 2024-04-23T16:45:49.458 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:45:49.486 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:49.510 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:45:49.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:49.514 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:45:49.615 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:45:49.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:49 smithi148 ceph-mon[44520]: pgmap v1684: 27 pgs: 27 active+clean; 866 MiB data, 6.5 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 889 op/s 2024-04-23T16:45:49.672 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:49 smithi103 ceph-mon[39629]: pgmap v1684: 27 pgs: 27 active+clean; 866 MiB data, 6.5 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 889 op/s 2024-04-23T16:45:49.684 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:49.726 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:45:49.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:49.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:45:49.831 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:45:49.862 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:50.215 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:45:50.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:50.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:45:50.326 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:50.376 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:50.403 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:50.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:50.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:50.511 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:45:50.539 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:50.615 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:45:50.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:50.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:45:50.719 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:50.769 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:50.804 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:50.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:50.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:50.912 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:45:50.957 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:51.006 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:45:51.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:51.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:45:51.113 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:45:51.142 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:51.171 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:45:51.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:51.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:45:51.277 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:45:51.338 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:51.378 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:45:51.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:51.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:45:51.487 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:45:51.521 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:52.172 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:45:52.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:52.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:45:52.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:51 smithi188 ceph-mon[44101]: pgmap v1685: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 650 MiB data, 6.1 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-23T16:45:52.276 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:45:52.303 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:51 smithi103 ceph-mon[39629]: pgmap v1685: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 650 MiB data, 6.1 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-23T16:45:52.304 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:52.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:51 smithi148 ceph-mon[44520]: pgmap v1685: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 650 MiB data, 6.1 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-23T16:45:52.786 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:45:52.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:52.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:45:52.819 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-23T16:45:52.893 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:45:52.926 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:52.952 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:45:52.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:52.960 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:45:53.061 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:45:53.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:52 smithi103 ceph-mon[39629]: 7.6 deep-scrub ok 2024-04-23T16:45:53.120 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:53.160 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:45:53.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:53.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:45:53.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:52 smithi188 ceph-mon[44101]: 7.6 deep-scrub ok 2024-04-23T16:45:53.267 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:53.329 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:52 smithi148 ceph-mon[44520]: 7.6 deep-scrub ok 2024-04-23T16:45:53.332 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:53.362 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:53.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:53.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:53.472 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:45:53.508 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:53.529 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:45:53.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:53.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:45:53.638 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:45:53.670 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:53.699 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:45:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:53.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:45:53.807 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:45:53.836 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:53.862 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:45:53.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:53.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:45:53.973 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:45:54.005 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:54.037 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:45:54.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:54.041 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:45:54.089 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:53 smithi103 ceph-mon[39629]: pgmap v1686: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 591 MiB data, 5.9 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-23T16:45:54.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:53 smithi148 ceph-mon[44520]: pgmap v1686: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 591 MiB data, 5.9 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-23T16:45:54.142 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:45:54.173 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:54.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:53 smithi188 ceph-mon[44101]: pgmap v1686: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 591 MiB data, 5.9 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-23T16:45:54.427 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:45:54.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:54.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:45:54.449 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:45:54.470 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:45:54.473 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:45:54.502 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:45:54.530 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:45:54.534 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:45:54.562 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:45:54.572 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:54.589 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:45:54.620 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:45:54.646 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:45:54.674 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:45:54.702 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:45:54.702 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:45:54.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:54.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:45:54.729 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:45:54.730 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:45:54.730 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:45:54.730 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:45:54.730 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:45:54.730 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:45:54.730 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:45:54.730 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:45:54.730 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:45:54.730 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:45:54.730 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:45:54.731 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:45:54.731 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:45:54.814 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:45:54.844 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:54.957 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:54 smithi103 ceph-mon[39629]: 7.4 scrub starts 2024-04-23T16:45:54.958 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:54 smithi103 ceph-mon[39629]: 7.4 scrub ok 2024-04-23T16:45:54.958 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:54 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4139533963' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:45:54.958 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:54.958 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:54.958 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:54.958 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:55.047 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:45:55.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:55.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:45:55.158 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:45:55.191 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:55.216 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:45:55.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:55.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:45:55.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:54 smithi188 ceph-mon[44101]: 7.4 scrub starts 2024-04-23T16:45:55.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:54 smithi188 ceph-mon[44101]: 7.4 scrub ok 2024-04-23T16:45:55.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:54 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4139533963' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:45:55.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:55.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:55.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:55.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:55.321 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:45:55.369 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:54 smithi148 ceph-mon[44520]: 7.4 scrub starts 2024-04-23T16:45:55.369 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:54 smithi148 ceph-mon[44520]: 7.4 scrub ok 2024-04-23T16:45:55.369 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:54 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4139533963' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:45:55.370 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:55.370 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:55.370 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:55.370 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:55.371 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:55.418 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:45:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:55.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:45:55.526 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:45:55.557 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:55.929 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:45:55.929 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:45:56.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:55 smithi188 ceph-mon[44101]: 7.e scrub starts 2024-04-23T16:45:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:55 smithi188 ceph-mon[44101]: 7.e scrub ok 2024-04-23T16:45:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:55 smithi188 ceph-mon[44101]: pgmap v1687: 27 pgs: 27 active+clean; 418 MiB data, 5.6 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-23T16:45:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:55 smithi103 ceph-mon[39629]: 7.e scrub starts 2024-04-23T16:45:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:55 smithi103 ceph-mon[39629]: 7.e scrub ok 2024-04-23T16:45:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:55 smithi103 ceph-mon[39629]: pgmap v1687: 27 pgs: 27 active+clean; 418 MiB data, 5.6 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-23T16:45:56.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:55 smithi148 ceph-mon[44520]: 7.e scrub starts 2024-04-23T16:45:56.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:55 smithi148 ceph-mon[44520]: 7.e scrub ok 2024-04-23T16:45:56.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:45:56.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:55 smithi148 ceph-mon[44520]: pgmap v1687: 27 pgs: 27 active+clean; 418 MiB data, 5.6 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-23T16:45:56.411 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:45:56.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:56.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:45:56.515 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:45:56.546 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:56.603 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:45:56.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:56.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:45:56.707 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:45:56.768 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:56.809 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:45:56.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:56.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:45:56.918 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:45:56.967 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:56 smithi103 ceph-mon[39629]: from='client.16926 v1:172.21.15.103:0/2751803821' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:45:56.968 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:56 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:45:56.968 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:56 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:45:56.971 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:57.002 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:45:57.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:57.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:45:57.109 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:57.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:56 smithi148 ceph-mon[44520]: from='client.16926 v1:172.21.15.103:0/2751803821' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:45:57.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:56 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:45:57.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:56 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:45:57.170 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:57.197 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:57.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:57.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:57.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:56 smithi188 ceph-mon[44101]: from='client.16926 v1:172.21.15.103:0/2751803821' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:45:57.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:56 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:45:57.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:56 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:45:57.306 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:45:57.335 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:57.649 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:45:57.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:57.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:45:57.683 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1687,"stamp":"2024-04-23T16:45:54.882569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":438108728,"num_objects":6679,"num_object_clones":0,"num_object_copies":13358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6679,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":230681,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22697,"num_bytes_recovered":1487742016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162213,"ondisk_log_size":162213,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":5896700,"kb_used_data":944756,"kb_used_omap":243,"kb_used_meta":4951436,"kb_avail":837809668,"statfs":{"total":863955320832,"available":857917100032,"internally_reserved":0,"allocated":967430144,"data_stored":964150821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":249416,"internal_metadata":5070270904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-690356224,"num_objects":-10534,"num_object_clones":0,"num_object_copies":-21068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10534,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003799"},"pg_stats":[{"pgid":"7.d","version":"597'14409","reported_seq":14555,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:50.397713+0000","last_change":"2024-04-23T16:45:41.166043+0000","last_active":"2024-04-23T16:45:50.397713+0000","last_peered":"2024-04-23T16:45:50.397713+0000","last_clean":"2024-04-23T16:45:50.397713+0000","last_became_active":"2024-04-23T16:41:57.310025+0000","last_became_peered":"2024-04-23T16:41:57.310025+0000","last_unstale":"2024-04-23T16:45:50.397713+0000","last_undegraded":"2024-04-23T16:45:50.397713+0000","last_fullsized":"2024-04-23T16:45:50.397713+0000","mapping_epoch":540,"log_start":"554'4400","ondisk_log_start":"554'4400","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'13803","last_scrub_stamp":"2024-04-23T16:45:41.165863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:41.165863+0000","objects_scrubbed":1276,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:54.268892+0000","scrub_duration":278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43253784,"num_objects":661,"num_object_clones":0,"num_object_copies":1322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":661,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14409,"num_write_kb":482177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"597'14608","reported_seq":14835,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:50.836666+0000","last_change":"2024-04-23T16:45:31.421258+0000","last_active":"2024-04-23T16:45:50.836666+0000","last_peered":"2024-04-23T16:45:50.836666+0000","last_clean":"2024-04-23T16:45:50.836666+0000","last_became_active":"2024-04-23T16:41:57.310754+0000","last_became_peered":"2024-04-23T16:41:57.310754+0000","last_unstale":"2024-04-23T16:45:50.836666+0000","last_undegraded":"2024-04-23T16:45:50.836666+0000","last_fullsized":"2024-04-23T16:45:50.836666+0000","mapping_epoch":540,"log_start":"554'4600","ondisk_log_start":"554'4600","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'13307","last_scrub_stamp":"2024-04-23T16:45:31.421125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:31.421125+0000","objects_scrubbed":1846,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:42.053633+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35127296,"num_objects":536,"num_object_clones":0,"num_object_copies":1072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14608,"num_write_kb":484608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"597'14554","reported_seq":14805,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:53.641562+0000","last_change":"2024-04-23T16:44:45.745602+0000","last_active":"2024-04-23T16:45:53.641562+0000","last_peered":"2024-04-23T16:45:53.641562+0000","last_clean":"2024-04-23T16:45:53.641562+0000","last_became_active":"2024-04-23T16:41:57.310189+0000","last_became_peered":"2024-04-23T16:41:57.310189+0000","last_unstale":"2024-04-23T16:45:53.641562+0000","last_undegraded":"2024-04-23T16:45:53.641562+0000","last_fullsized":"2024-04-23T16:45:53.641562+0000","mapping_epoch":540,"log_start":"554'4500","ondisk_log_start":"554'4500","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'10439","last_scrub_stamp":"2024-04-23T16:44:45.745464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:45.745464+0000","objects_scrubbed":4477,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:10.632547+0000","scrub_duration":680,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22806528,"num_objects":348,"num_object_clones":0,"num_object_copies":696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14554,"num_write_kb":476864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"597'14612","reported_seq":14855,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:54.491816+0000","last_change":"2024-04-23T16:44:58.743309+0000","last_active":"2024-04-23T16:45:54.491816+0000","last_peered":"2024-04-23T16:45:54.491816+0000","last_clean":"2024-04-23T16:45:54.491816+0000","last_became_active":"2024-04-23T16:41:57.309569+0000","last_became_peered":"2024-04-23T16:41:57.309569+0000","last_unstale":"2024-04-23T16:45:54.491816+0000","last_undegraded":"2024-04-23T16:45:54.491816+0000","last_fullsized":"2024-04-23T16:45:54.491816+0000","mapping_epoch":540,"log_start":"554'4600","ondisk_log_start":"554'4600","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"591'11141","last_scrub_stamp":"2024-04-23T16:44:58.743181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:58.743181+0000","objects_scrubbed":3762,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:00.499252+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18350080,"num_objects":280,"num_object_clones":0,"num_object_copies":560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14612,"num_write_kb":476544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"597'14013","reported_seq":14159,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:50.809275+0000","last_change":"2024-04-23T16:45:35.841405+0000","last_active":"2024-04-23T16:45:50.809275+0000","last_peered":"2024-04-23T16:45:50.809275+0000","last_clean":"2024-04-23T16:45:50.809275+0000","last_became_active":"2024-04-23T16:41:57.309841+0000","last_became_peered":"2024-04-23T16:41:57.309841+0000","last_unstale":"2024-04-23T16:45:50.809275+0000","last_undegraded":"2024-04-23T16:45:50.809275+0000","last_fullsized":"2024-04-23T16:45:50.809275+0000","mapping_epoch":540,"log_start":"554'4000","ondisk_log_start":"554'4000","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'13051","last_scrub_stamp":"2024-04-23T16:45:35.841096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:35.841096+0000","objects_scrubbed":1495,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:51.669940+0000","scrub_duration":354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34537472,"num_objects":527,"num_object_clones":0,"num_object_copies":1054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14013,"num_write_kb":465280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"597'14218","reported_seq":25801,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:50.418040+0000","last_change":"2024-04-23T16:45:37.206907+0000","last_active":"2024-04-23T16:45:50.418040+0000","last_peered":"2024-04-23T16:45:50.418040+0000","last_clean":"2024-04-23T16:45:50.418040+0000","last_became_active":"2024-04-23T16:43:08.106076+0000","last_became_peered":"2024-04-23T16:43:08.106076+0000","last_unstale":"2024-04-23T16:45:50.418040+0000","last_undegraded":"2024-04-23T16:45:50.418040+0000","last_fullsized":"2024-04-23T16:45:50.418040+0000","mapping_epoch":563,"log_start":"554'4200","ondisk_log_start":"554'4200","created":540,"last_epoch_clean":595,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'13379","last_scrub_stamp":"2024-04-23T16:45:37.206679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:37.206679+0000","objects_scrubbed":1381,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:49.712976+0000","scrub_duration":350,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34603008,"num_objects":528,"num_object_clones":0,"num_object_copies":1056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14218,"num_write_kb":471872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5714,"num_bytes_recovered":374472704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1516,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:45.368106+0000","last_change":"2024-04-23T16:45:33.395526+0000","last_active":"2024-04-23T16:45:45.368106+0000","last_peered":"2024-04-23T16:45:45.368106+0000","last_clean":"2024-04-23T16:45:45.368106+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:45:45.368106+0000","last_undegraded":"2024-04-23T16:45:45.368106+0000","last_fullsized":"2024-04-23T16:45:45.368106+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:33.395406+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:45:33.395406+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:01.585504+0000","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"597'14583","reported_seq":14832,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:53.647176+0000","last_change":"2024-04-23T16:44:52.727725+0000","last_active":"2024-04-23T16:45:53.647176+0000","last_peered":"2024-04-23T16:45:53.647176+0000","last_clean":"2024-04-23T16:45:53.647176+0000","last_became_active":"2024-04-23T16:41:57.311753+0000","last_became_peered":"2024-04-23T16:41:57.311753+0000","last_unstale":"2024-04-23T16:45:53.647176+0000","last_undegraded":"2024-04-23T16:45:53.647176+0000","last_fullsized":"2024-04-23T16:45:53.647176+0000","mapping_epoch":540,"log_start":"554'4500","ondisk_log_start":"554'4500","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"588'10838","last_scrub_stamp":"2024-04-23T16:44:52.727632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:52.727632+0000","objects_scrubbed":4109,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":730,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22478848,"num_objects":343,"num_object_clones":0,"num_object_copies":686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14583,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1514,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:45.378946+0000","last_change":"2024-04-23T16:44:45.850797+0000","last_active":"2024-04-23T16:45:45.378946+0000","last_peered":"2024-04-23T16:45:45.378946+0000","last_clean":"2024-04-23T16:45:45.378946+0000","last_became_active":"2024-04-23T16:43:07.962092+0000","last_became_peered":"2024-04-23T16:43:07.962092+0000","last_unstale":"2024-04-23T16:45:45.378946+0000","last_undegraded":"2024-04-23T16:45:45.378946+0000","last_fullsized":"2024-04-23T16:45:45.378946+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:45.850682+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:44:45.850682+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:12.889710+0000","scrub_duration":234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"597'14538","reported_seq":14789,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:54.523211+0000","last_change":"2024-04-23T16:44:44.578218+0000","last_active":"2024-04-23T16:45:54.523211+0000","last_peered":"2024-04-23T16:45:54.523211+0000","last_clean":"2024-04-23T16:45:54.523211+0000","last_became_active":"2024-04-23T16:41:57.305944+0000","last_became_peered":"2024-04-23T16:41:57.305944+0000","last_unstale":"2024-04-23T16:45:54.523211+0000","last_undegraded":"2024-04-23T16:45:54.523211+0000","last_fullsized":"2024-04-23T16:45:54.523211+0000","mapping_epoch":540,"log_start":"554'4500","ondisk_log_start":"554'4500","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'10405","last_scrub_stamp":"2024-04-23T16:44:44.578065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:44.578065+0000","objects_scrubbed":4454,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:01.634216+0000","scrub_duration":505,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20316160,"num_objects":310,"num_object_clones":0,"num_object_copies":620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14538,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1491,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:45.370798+0000","last_change":"2024-04-23T16:45:38.633704+0000","last_active":"2024-04-23T16:45:45.370798+0000","last_peered":"2024-04-23T16:45:45.370798+0000","last_clean":"2024-04-23T16:45:45.370798+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:45:45.370798+0000","last_undegraded":"2024-04-23T16:45:45.370798+0000","last_fullsized":"2024-04-23T16:45:45.370798+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:38.633554+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:45:38.633554+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:50.365278+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"597'14874","reported_seq":15135,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:54.552018+0000","last_change":"2024-04-23T16:45:54.360285+0000","last_active":"2024-04-23T16:45:54.552018+0000","last_peered":"2024-04-23T16:45:54.552018+0000","last_clean":"2024-04-23T16:45:54.552018+0000","last_became_active":"2024-04-23T16:41:57.309858+0000","last_became_peered":"2024-04-23T16:41:57.309858+0000","last_unstale":"2024-04-23T16:45:54.552018+0000","last_undegraded":"2024-04-23T16:45:54.552018+0000","last_fullsized":"2024-04-23T16:45:54.552018+0000","mapping_epoch":540,"log_start":"554'4800","ondisk_log_start":"554'4800","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"597'14858","last_scrub_stamp":"2024-04-23T16:45:54.360119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:54.360119+0000","objects_scrubbed":315,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:55.562395+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19136512,"num_objects":292,"num_object_clones":0,"num_object_copies":584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14874,"num_write_kb":485312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1515,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:45.367765+0000","last_change":"2024-04-23T16:45:30.314968+0000","last_active":"2024-04-23T16:45:45.367765+0000","last_peered":"2024-04-23T16:45:45.367765+0000","last_clean":"2024-04-23T16:45:45.367765+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:45:45.367765+0000","last_undegraded":"2024-04-23T16:45:45.367765+0000","last_fullsized":"2024-04-23T16:45:45.367765+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:30.314881+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:45:30.314881+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:54.455465+0000","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"597'14200","reported_seq":14348,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:51.586237+0000","last_change":"2024-04-23T16:45:36.077763+0000","last_active":"2024-04-23T16:45:51.586237+0000","last_peered":"2024-04-23T16:45:51.586237+0000","last_clean":"2024-04-23T16:45:51.586237+0000","last_became_active":"2024-04-23T16:41:57.310400+0000","last_became_peered":"2024-04-23T16:41:57.310400+0000","last_unstale":"2024-04-23T16:45:51.586237+0000","last_undegraded":"2024-04-23T16:45:51.586237+0000","last_fullsized":"2024-04-23T16:45:51.586237+0000","mapping_epoch":540,"log_start":"554'4100","ondisk_log_start":"554'4100","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'13167","last_scrub_stamp":"2024-04-23T16:45:36.077623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:36.077623+0000","objects_scrubbed":1539,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:59.580518+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32636928,"num_objects":498,"num_object_clones":0,"num_object_copies":996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14200,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1089,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:45.375070+0000","last_change":"2024-04-23T16:45:37.184028+0000","last_active":"2024-04-23T16:45:45.375070+0000","last_peered":"2024-04-23T16:45:45.375070+0000","last_clean":"2024-04-23T16:45:45.375070+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:45:45.375070+0000","last_undegraded":"2024-04-23T16:45:45.375070+0000","last_fullsized":"2024-04-23T16:45:45.375070+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:37.183866+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:45:37.183866+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:04.176221+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"597'14175","reported_seq":14318,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:50.423071+0000","last_change":"2024-04-23T16:45:42.270579+0000","last_active":"2024-04-23T16:45:50.423071+0000","last_peered":"2024-04-23T16:45:50.423071+0000","last_clean":"2024-04-23T16:45:50.423071+0000","last_became_active":"2024-04-23T16:41:57.306229+0000","last_became_peered":"2024-04-23T16:41:57.306229+0000","last_unstale":"2024-04-23T16:45:50.423071+0000","last_undegraded":"2024-04-23T16:45:50.423071+0000","last_fullsized":"2024-04-23T16:45:50.423071+0000","mapping_epoch":540,"log_start":"554'4100","ondisk_log_start":"554'4100","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'13614","last_scrub_stamp":"2024-04-23T16:45:42.270426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:42.270426+0000","objects_scrubbed":1112,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:53.144483+0000","scrub_duration":405,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34668544,"num_objects":529,"num_object_clones":0,"num_object_copies":1058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14175,"num_write_kb":470528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1043,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:45.369639+0000","last_change":"2024-04-23T16:45:33.205702+0000","last_active":"2024-04-23T16:45:45.369639+0000","last_peered":"2024-04-23T16:45:45.369639+0000","last_clean":"2024-04-23T16:45:45.369639+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:45:45.369639+0000","last_undegraded":"2024-04-23T16:45:45.369639+0000","last_fullsized":"2024-04-23T16:45:45.369639+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:33.205593+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:45:33.205593+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:58.254203+0000","scrub_duration":121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.0","version":"597'14319","reported_seq":14562,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:52.793581+0000","last_change":"2024-04-23T16:45:00.368292+0000","last_active":"2024-04-23T16:45:52.793581+0000","last_peered":"2024-04-23T16:45:52.793581+0000","last_clean":"2024-04-23T16:45:52.793581+0000","last_became_active":"2024-04-23T16:41:57.311479+0000","last_became_peered":"2024-04-23T16:41:57.311479+0000","last_unstale":"2024-04-23T16:45:52.793581+0000","last_undegraded":"2024-04-23T16:45:52.793581+0000","last_fullsized":"2024-04-23T16:45:52.793581+0000","mapping_epoch":540,"log_start":"554'4300","ondisk_log_start":"554'4300","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"591'11135","last_scrub_stamp":"2024-04-23T16:45:00.368005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:00.368005+0000","objects_scrubbed":3585,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:06.315750+0000","scrub_duration":778,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24969216,"num_objects":381,"num_object_clones":0,"num_object_copies":762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14319,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1524,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:45.368271+0000","last_change":"2024-04-23T16:45:30.258925+0000","last_active":"2024-04-23T16:45:45.368271+0000","last_peered":"2024-04-23T16:45:45.368271+0000","last_clean":"2024-04-23T16:45:45.368271+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:45:45.368271+0000","last_undegraded":"2024-04-23T16:45:45.368271+0000","last_fullsized":"2024-04-23T16:45:45.368271+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:30.258804+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:45:30.258804+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:51.589576+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.1","version":"597'14360","reported_seq":25693,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:54.634808+0000","last_change":"2024-04-23T16:45:34.213681+0000","last_active":"2024-04-23T16:45:54.634808+0000","last_peered":"2024-04-23T16:45:54.634808+0000","last_clean":"2024-04-23T16:45:54.634808+0000","last_became_active":"2024-04-23T16:43:08.067887+0000","last_became_peered":"2024-04-23T16:43:08.067887+0000","last_unstale":"2024-04-23T16:45:54.634808+0000","last_undegraded":"2024-04-23T16:45:54.634808+0000","last_fullsized":"2024-04-23T16:45:54.634808+0000","mapping_epoch":563,"log_start":"554'4300","ondisk_log_start":"554'4300","created":540,"last_epoch_clean":595,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'12978","last_scrub_stamp":"2024-04-23T16:45:34.213502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:34.213502+0000","objects_scrubbed":1669,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:40.877992+0000","scrub_duration":431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18087936,"num_objects":276,"num_object_clones":0,"num_object_copies":552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14360,"num_write_kb":468352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5588,"num_bytes_recovered":366215168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1508,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:45.370894+0000","last_change":"2024-04-23T16:45:45.059609+0000","last_active":"2024-04-23T16:45:45.370894+0000","last_peered":"2024-04-23T16:45:45.370894+0000","last_clean":"2024-04-23T16:45:45.370894+0000","last_became_active":"2024-04-23T16:43:07.918042+0000","last_became_peered":"2024-04-23T16:43:07.918042+0000","last_unstale":"2024-04-23T16:45:45.370894+0000","last_undegraded":"2024-04-23T16:45:45.370894+0000","last_fullsized":"2024-04-23T16:45:45.370894+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:45.059503+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:45:45.059503+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:59.782775+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"597'14260","reported_seq":25886,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:50.375004+0000","last_change":"2024-04-23T16:45:38.202729+0000","last_active":"2024-04-23T16:45:50.375004+0000","last_peered":"2024-04-23T16:45:50.375004+0000","last_clean":"2024-04-23T16:45:50.375004+0000","last_became_active":"2024-04-23T16:43:08.333793+0000","last_became_peered":"2024-04-23T16:43:08.333793+0000","last_unstale":"2024-04-23T16:45:50.375004+0000","last_undegraded":"2024-04-23T16:45:50.375004+0000","last_fullsized":"2024-04-23T16:45:50.375004+0000","mapping_epoch":563,"log_start":"554'4200","ondisk_log_start":"554'4200","created":540,"last_epoch_clean":595,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'13496","last_scrub_stamp":"2024-04-23T16:45:38.202648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:38.202648+0000","objects_scrubbed":1328,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:04.203602+0000","scrub_duration":339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36569088,"num_objects":558,"num_object_clones":0,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14260,"num_write_kb":474176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5737,"num_bytes_recovered":375980032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1400,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:45.369192+0000","last_change":"2024-04-23T16:45:28.948302+0000","last_active":"2024-04-23T16:45:45.369192+0000","last_peered":"2024-04-23T16:45:45.369192+0000","last_clean":"2024-04-23T16:45:45.369192+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:45:45.369192+0000","last_undegraded":"2024-04-23T16:45:45.369192+0000","last_fullsized":"2024-04-23T16:45:45.369192+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:28.948185+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:45:28.948185+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:44.902931+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"597'14611","reported_seq":26080,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:54.640088+0000","last_change":"2024-04-23T16:45:36.018343+0000","last_active":"2024-04-23T16:45:54.640088+0000","last_peered":"2024-04-23T16:45:54.640088+0000","last_clean":"2024-04-23T16:45:54.640088+0000","last_became_active":"2024-04-23T16:43:08.109240+0000","last_became_peered":"2024-04-23T16:43:08.109240+0000","last_unstale":"2024-04-23T16:45:54.640088+0000","last_undegraded":"2024-04-23T16:45:54.640088+0000","last_fullsized":"2024-04-23T16:45:54.640088+0000","mapping_epoch":563,"log_start":"555'4600","ondisk_log_start":"555'4600","created":540,"last_epoch_clean":594,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'13296","last_scrub_stamp":"2024-04-23T16:45:36.018091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:36.018091+0000","objects_scrubbed":1632,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:36.440846+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20512768,"num_objects":313,"num_object_clones":0,"num_object_copies":626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14611,"num_write_kb":477568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5656,"num_bytes_recovered":370671616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1322,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:45.370733+0000","last_change":"2024-04-23T16:45:33.917246+0000","last_active":"2024-04-23T16:45:45.370733+0000","last_peered":"2024-04-23T16:45:45.370733+0000","last_clean":"2024-04-23T16:45:45.370733+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:45:45.370733+0000","last_undegraded":"2024-04-23T16:45:45.370733+0000","last_fullsized":"2024-04-23T16:45:45.370733+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:33.917127+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:45:33.917127+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:57.447552+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"597'14951","reported_seq":15235,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:54.532437+0000","last_change":"2024-04-23T16:45:51.785542+0000","last_active":"2024-04-23T16:45:54.532437+0000","last_peered":"2024-04-23T16:45:54.532437+0000","last_clean":"2024-04-23T16:45:54.532437+0000","last_became_active":"2024-04-23T16:41:57.309349+0000","last_became_peered":"2024-04-23T16:41:57.309349+0000","last_unstale":"2024-04-23T16:45:54.532437+0000","last_undegraded":"2024-04-23T16:45:54.532437+0000","last_fullsized":"2024-04-23T16:45:54.532437+0000","mapping_epoch":540,"log_start":"555'4901","ondisk_log_start":"555'4901","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"597'14772","last_scrub_stamp":"2024-04-23T16:45:51.785406+0000","last_deep_scrub":"597'14772","last_deep_scrub_stamp":"2024-04-23T16:45:51.785406+0000","last_clean_scrub_stamp":"2024-04-23T16:45:51.785406+0000","objects_scrubbed":604,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:51.799605+0000","scrub_duration":4584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19464192,"num_objects":297,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14347,"num_write_kb":468608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1490,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-23T16:45:45.370759+0000","last_change":"2024-04-23T16:45:37.584810+0000","last_active":"2024-04-23T16:45:45.370759+0000","last_peered":"2024-04-23T16:45:45.370759+0000","last_clean":"2024-04-23T16:45:45.370759+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:45:45.370759+0000","last_undegraded":"2024-04-23T16:45:45.370759+0000","last_fullsized":"2024-04-23T16:45:45.370759+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:37.584692+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:45:37.584692+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:05.236607+0000","scrub_duration":121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":437518360,"num_objects":6677,"num_object_clones":0,"num_object_copies":13354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6677,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":230681,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22695,"num_bytes_recovered":1487339520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2295472128,"data_stored":2295463984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160684,"ondisk_log_size":160684,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1474560,"data_stored":1460352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057007,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888964,"kb_used_data":176124,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":92856188,"statfs":{"total":95995035648,"available":95084736512,"internally_reserved":0,"allocated":180350976,"data_stored":179988301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":729938372},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351350,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612987,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":636420,"kb_used_data":95740,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":93108732,"statfs":{"total":95995035648,"available":95343341568,"internally_reserved":0,"allocated":98037760,"data_stored":97675085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":553646528},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841919,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":420812,"kb_used_data":50884,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93324340,"statfs":{"total":95995035648,"available":95564124160,"internally_reserved":0,"allocated":52105216,"data_stored":51738989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168964,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":675460,"kb_used_data":70908,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93069692,"statfs":{"total":95995035648,"available":95303364608,"internally_reserved":0,"allocated":72609792,"data_stored":72247117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":619051466},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463305,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":723336,"kb_used_data":190848,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93021816,"statfs":{"total":95995035648,"available":95254339584,"internally_reserved":0,"allocated":195428352,"data_stored":195061605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":545126854},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137446,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312264,"kb_used_data":81664,"kb_used_omap":231,"kb_used_meta":230360,"kb_avail":93432888,"statfs":{"total":95995035648,"available":95675277312,"internally_reserved":0,"allocated":83623936,"data_stored":83257189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":235889552},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738792,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":660228,"kb_used_data":83388,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93084924,"statfs":{"total":95995035648,"available":95318962176,"internally_reserved":0,"allocated":85389312,"data_stored":85026637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269462,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921170,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":846404,"kb_used_data":115516,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":92898748,"statfs":{"total":95995035648,"available":95128317952,"internally_reserved":0,"allocated":118288384,"data_stored":117925709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,2,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757645,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":732812,"kb_used_data":79684,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93012340,"statfs":{"total":95995035648,"available":95244636160,"internally_reserved":0,"allocated":81596416,"data_stored":81230189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":77201408,"data_stored":77201408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75960320,"data_stored":75956248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1411842048,"data_stored":1411842048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":94633984,"data_stored":94633984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":78643200,"data_stored":78643200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":185274368,"data_stored":185270296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":69140480,"data_stored":69140480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":48758784,"data_stored":48758784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":90243072,"data_stored":90243072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":163774464,"data_stored":163774464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:45:57.754 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:45:57.817 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:57.858 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:45:57.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:57.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:45:57.962 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:45:57.978 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:45:58.005 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:58.029 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:45:58.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:58.033 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:45:58.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:57 smithi148 ceph-mon[44520]: pgmap v1688: 27 pgs: 27 active+clean; 269 MiB data, 5.4 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-23T16:45:58.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:57 smithi148 ceph-mon[44520]: osdmap e598: 12 total, 11 up, 9 in 2024-04-23T16:45:58.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:57 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/679022495' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:45:58.134 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:45:58.162 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:58.186 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:45:58.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:58.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:45:58.233 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:57 smithi103 ceph-mon[39629]: pgmap v1688: 27 pgs: 27 active+clean; 269 MiB data, 5.4 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-23T16:45:58.233 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:57 smithi103 ceph-mon[39629]: osdmap e598: 12 total, 11 up, 9 in 2024-04-23T16:45:58.233 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:57 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/679022495' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:45:58.291 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:45:58.321 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:58.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:57 smithi188 ceph-mon[44101]: pgmap v1688: 27 pgs: 27 active+clean; 269 MiB data, 5.4 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-23T16:45:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:57 smithi188 ceph-mon[44101]: osdmap e598: 12 total, 11 up, 9 in 2024-04-23T16:45:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:57 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/679022495' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:45:58.618 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Removed 118678 objects 2024-04-23T16:45:58.618 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Clean up completed and total clean up time :136.073 2024-04-23T16:45:58.888 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:45:58.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:58.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:45:58.992 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:45:59.022 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:59.043 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:45:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:59.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:45:59.147 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:45:59.173 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:58 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:45:59.173 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:58 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:45:59.173 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:58 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:45:59.173 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:58 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/679022495' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:45:59.173 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:58 smithi103 ceph-mon[39629]: osdmap e599: 12 total, 11 up, 9 in 2024-04-23T16:45:59.175 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:59.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:58 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:45:59.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:58 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:45:59.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:58 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:45:59.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:58 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/679022495' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:45:59.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:58 smithi148 ceph-mon[44520]: osdmap e599: 12 total, 11 up, 9 in 2024-04-23T16:45:59.424 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:45:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:45:59.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:45:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:58 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:45:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:58 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:45:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:58 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:45:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:58 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/679022495' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:45:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:58 smithi188 ceph-mon[44101]: osdmap e599: 12 total, 11 up, 9 in 2024-04-23T16:45:59.528 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:45:59.555 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:45:59.856 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:45:59.857 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:46:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:45:59 smithi103 ceph-mon[39629]: pgmap v1691: 27 pgs: 27 active+clean; 205 MiB data, 5.2 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 890 op/s 2024-04-23T16:46:00.104 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:00.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:00.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:00.208 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:00.238 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:45:59 smithi148 ceph-mon[44520]: pgmap v1691: 27 pgs: 27 active+clean; 205 MiB data, 5.2 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 890 op/s 2024-04-23T16:46:00.240 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:00.262 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:00.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:00.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:00.367 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:46:00.397 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:00.418 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:46:00.418 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:46:00.418 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-23T16:46:00.418 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-23T16:46:00.418 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:46:00.419 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:46:00.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:00.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:46:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:45:59 smithi188 ceph-mon[44101]: pgmap v1691: 27 pgs: 27 active+clean; 205 MiB data, 5.2 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 890 op/s 2024-04-23T16:46:00.523 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:46:00.553 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:00.626 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:00.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:00.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:00.686 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:46:00.730 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:46:00.762 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:00.783 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:46:00.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:00.787 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:46:00.888 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:46:00.913 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:01.311 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-23T16:46:01.361 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:46:01.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:01.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:46:01.465 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:46:01.491 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:01.840 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:46:01.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:01.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:46:01.858 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:46:01.928 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:01.945 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:46:01.964 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:01.978 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:02.000 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:46:02.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:02.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:46:02.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:01 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:46:02.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:01 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:46:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:01 smithi103 ceph-mon[39629]: pgmap v1692: 27 pgs: 27 active+clean; 24 MiB data, 4.9 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-23T16:46:02.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:01 smithi103 ceph-mon[39629]: osdmap e600: 12 total, 11 up, 9 in 2024-04-23T16:46:02.104 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:46:02.131 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:02.206 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:46:02.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:02.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:46:02.310 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:46:02.340 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:01 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:46:02.340 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:01 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:46:02.340 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:01 smithi188 ceph-mon[44101]: pgmap v1692: 27 pgs: 27 active+clean; 24 MiB data, 4.9 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-23T16:46:02.340 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:01 smithi188 ceph-mon[44101]: osdmap e600: 12 total, 11 up, 9 in 2024-04-23T16:46:02.342 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:02.363 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:46:02.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:02.366 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:46:02.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:01 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:46:02.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:01 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:46:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:01 smithi148 ceph-mon[44520]: pgmap v1692: 27 pgs: 27 active+clean; 24 MiB data, 4.9 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-23T16:46:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:01 smithi148 ceph-mon[44520]: osdmap e600: 12 total, 11 up, 9 in 2024-04-23T16:46:02.467 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:46:02.497 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:02.518 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:46:02.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:02.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:46:02.622 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:46:02.651 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:02.900 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:46:02.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:02.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:46:03.006 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:46:03.038 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:03.058 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:46:03.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:03.063 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:46:03.163 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:46:03.195 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:03.216 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:46:03.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:03.219 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:46:03.235 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-04-23T16:46:03.235 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-04-23T16:46:03.262 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:03 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:46:03.262 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:03 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:46:03.262 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:03 smithi103 ceph-mon[39629]: osdmap e601: 12 total, 11 up, 9 in 2024-04-23T16:46:03.320 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:46:03.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:03 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:46:03.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:03 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:46:03.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:03 smithi188 ceph-mon[44101]: osdmap e601: 12 total, 11 up, 9 in 2024-04-23T16:46:03.349 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:03.369 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:46:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:03.372 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:46:03.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:03 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:46:03.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:03 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:46:03.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:03 smithi148 ceph-mon[44520]: osdmap e601: 12 total, 11 up, 9 in 2024-04-23T16:46:03.472 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:46:03.505 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:03.528 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:46:03.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:03.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:46:03.632 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:46:03.659 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:03.867 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:46:03.925 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:04 smithi103 ceph-mon[39629]: pgmap v1695: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 726 op/s 2024-04-23T16:46:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:04 smithi148 ceph-mon[44520]: pgmap v1695: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 726 op/s 2024-04-23T16:46:04.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:04 smithi188 ceph-mon[44101]: pgmap v1695: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 726 op/s 2024-04-23T16:46:05.445 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:05 smithi103 ceph-mon[39629]: pgmap v1696: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 573 op/s 2024-04-23T16:46:05.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:05 smithi148 ceph-mon[44520]: pgmap v1696: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 573 op/s 2024-04-23T16:46:05.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:05 smithi188 ceph-mon[44101]: pgmap v1696: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 573 op/s 2024-04-23T16:46:06.205 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:46:06.206 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:46:06.438 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:46:06.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:06.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:46:06.543 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:06.576 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:06.598 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:06.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:06 smithi103 ceph-mon[39629]: from='client.16938 v1:172.21.15.103:0/755345775' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:46:06.600 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:06 smithi148 ceph-mon[44520]: from='client.16938 v1:172.21.15.103:0/755345775' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:46:06.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:06.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:06.701 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:06.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:06 smithi188 ceph-mon[44101]: from='client.16938 v1:172.21.15.103:0/755345775' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:46:06.733 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:06.754 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:06.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:06.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:06.859 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:46:06.891 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:06.912 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:46:06.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:06.915 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:46:07.016 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:46:07.046 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:07.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:07 smithi148 ceph-mon[44520]: pgmap v1697: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 408 op/s 2024-04-23T16:46:07.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:07 smithi188 ceph-mon[44101]: pgmap v1697: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 408 op/s 2024-04-23T16:46:07.819 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:07 smithi103 ceph-mon[39629]: pgmap v1697: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 408 op/s 2024-04-23T16:46:08.358 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1696,"stamp":"2024-04-23T16:46:04.884360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":237358,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22697,"num_bytes_recovered":1487742016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162290,"ondisk_log_size":162290,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":5001076,"kb_used_data":7676,"kb_used_omap":243,"kb_used_meta":4993036,"kb_avail":838705292,"statfs":{"total":863955320832,"available":858834219008,"internally_reserved":0,"allocated":7860224,"data_stored":4420933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":249416,"internal_metadata":5112869304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-259522584,"num_objects":-3961,"num_object_clones":0,"num_object_copies":-7922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3961,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.910735"},"pg_stats":[{"pgid":"7.d","version":"599'15070","reported_seq":15220,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-04-23T16:45:58.618226+0000","last_change":"2024-04-23T16:45:41.166043+0000","last_active":"2024-04-23T16:45:58.618226+0000","last_peered":"2024-04-23T16:45:58.618226+0000","last_clean":"2024-04-23T16:45:58.618226+0000","last_became_active":"2024-04-23T16:41:57.310025+0000","last_became_peered":"2024-04-23T16:41:57.310025+0000","last_unstale":"2024-04-23T16:45:58.618226+0000","last_undegraded":"2024-04-23T16:45:58.618226+0000","last_fullsized":"2024-04-23T16:45:58.618226+0000","mapping_epoch":540,"log_start":"555'5000","ondisk_log_start":"555'5000","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'13803","last_scrub_stamp":"2024-04-23T16:45:41.165863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:41.165863+0000","objects_scrubbed":1276,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:54.268892+0000","scrub_duration":278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15070,"num_write_kb":482177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"599'15144","reported_seq":15375,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-04-23T16:45:58.545642+0000","last_change":"2024-04-23T16:45:31.421258+0000","last_active":"2024-04-23T16:45:58.545642+0000","last_peered":"2024-04-23T16:45:58.545642+0000","last_clean":"2024-04-23T16:45:58.545642+0000","last_became_active":"2024-04-23T16:41:57.310754+0000","last_became_peered":"2024-04-23T16:41:57.310754+0000","last_unstale":"2024-04-23T16:45:58.545642+0000","last_undegraded":"2024-04-23T16:45:58.545642+0000","last_fullsized":"2024-04-23T16:45:58.545642+0000","mapping_epoch":540,"log_start":"555'5100","ondisk_log_start":"555'5100","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'13307","last_scrub_stamp":"2024-04-23T16:45:31.421125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:31.421125+0000","objects_scrubbed":1846,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:42.053633+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15144,"num_write_kb":484608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"599'14902","reported_seq":15161,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-04-23T16:46:02.142652+0000","last_change":"2024-04-23T16:44:45.745602+0000","last_active":"2024-04-23T16:46:02.142652+0000","last_peered":"2024-04-23T16:46:02.142652+0000","last_clean":"2024-04-23T16:46:02.142652+0000","last_became_active":"2024-04-23T16:41:57.310189+0000","last_became_peered":"2024-04-23T16:41:57.310189+0000","last_unstale":"2024-04-23T16:46:02.142652+0000","last_undegraded":"2024-04-23T16:46:02.142652+0000","last_fullsized":"2024-04-23T16:46:02.142652+0000","mapping_epoch":540,"log_start":"555'4900","ondisk_log_start":"555'4900","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'10439","last_scrub_stamp":"2024-04-23T16:44:45.745464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:45.745464+0000","objects_scrubbed":4477,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:10.632547+0000","scrub_duration":680,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14902,"num_write_kb":476864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"599'14892","reported_seq":15143,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-04-23T16:46:02.142205+0000","last_change":"2024-04-23T16:44:58.743309+0000","last_active":"2024-04-23T16:46:02.142205+0000","last_peered":"2024-04-23T16:46:02.142205+0000","last_clean":"2024-04-23T16:46:02.142205+0000","last_became_active":"2024-04-23T16:41:57.309569+0000","last_became_peered":"2024-04-23T16:41:57.309569+0000","last_unstale":"2024-04-23T16:46:02.142205+0000","last_undegraded":"2024-04-23T16:46:02.142205+0000","last_fullsized":"2024-04-23T16:46:02.142205+0000","mapping_epoch":540,"log_start":"555'4800","ondisk_log_start":"555'4800","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"591'11141","last_scrub_stamp":"2024-04-23T16:44:58.743181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:58.743181+0000","objects_scrubbed":3762,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14892,"num_write_kb":476544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"599'14540","reported_seq":14691,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-04-23T16:45:58.605691+0000","last_change":"2024-04-23T16:45:35.841405+0000","last_active":"2024-04-23T16:45:58.605691+0000","last_peered":"2024-04-23T16:45:58.605691+0000","last_clean":"2024-04-23T16:45:58.605691+0000","last_became_active":"2024-04-23T16:41:57.309841+0000","last_became_peered":"2024-04-23T16:41:57.309841+0000","last_unstale":"2024-04-23T16:45:58.605691+0000","last_undegraded":"2024-04-23T16:45:58.605691+0000","last_fullsized":"2024-04-23T16:45:58.605691+0000","mapping_epoch":540,"log_start":"554'4500","ondisk_log_start":"554'4500","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'13051","last_scrub_stamp":"2024-04-23T16:45:35.841096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:35.841096+0000","objects_scrubbed":1495,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:51.669940+0000","scrub_duration":354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14540,"num_write_kb":465280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"599'14746","reported_seq":26333,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-04-23T16:45:58.580030+0000","last_change":"2024-04-23T16:45:37.206907+0000","last_active":"2024-04-23T16:45:58.580030+0000","last_peered":"2024-04-23T16:45:58.580030+0000","last_clean":"2024-04-23T16:45:58.580030+0000","last_became_active":"2024-04-23T16:43:08.106076+0000","last_became_peered":"2024-04-23T16:43:08.106076+0000","last_unstale":"2024-04-23T16:45:58.580030+0000","last_undegraded":"2024-04-23T16:45:58.580030+0000","last_fullsized":"2024-04-23T16:45:58.580030+0000","mapping_epoch":563,"log_start":"554'4700","ondisk_log_start":"554'4700","created":540,"last_epoch_clean":595,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'13379","last_scrub_stamp":"2024-04-23T16:45:37.206679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:37.206679+0000","objects_scrubbed":1381,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:49.712976+0000","scrub_duration":350,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14746,"num_write_kb":471872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5714,"num_bytes_recovered":374472704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1524,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-04-23T16:46:02.142687+0000","last_change":"2024-04-23T16:45:33.395526+0000","last_active":"2024-04-23T16:46:02.142687+0000","last_peered":"2024-04-23T16:46:02.142687+0000","last_clean":"2024-04-23T16:46:02.142687+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:46:02.142687+0000","last_undegraded":"2024-04-23T16:46:02.142687+0000","last_fullsized":"2024-04-23T16:46:02.142687+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:33.395406+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:45:33.395406+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:01.585504+0000","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"599'14926","reported_seq":15194,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-04-23T16:46:02.142754+0000","last_change":"2024-04-23T16:45:54.352092+0000","last_active":"2024-04-23T16:46:02.142754+0000","last_peered":"2024-04-23T16:46:02.142754+0000","last_clean":"2024-04-23T16:46:02.142754+0000","last_became_active":"2024-04-23T16:41:57.311753+0000","last_became_peered":"2024-04-23T16:41:57.311753+0000","last_unstale":"2024-04-23T16:46:02.142754+0000","last_undegraded":"2024-04-23T16:46:02.142754+0000","last_fullsized":"2024-04-23T16:46:02.142754+0000","mapping_epoch":540,"log_start":"555'4900","ondisk_log_start":"555'4900","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"597'14638","last_scrub_stamp":"2024-04-23T16:45:54.351934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:54.351934+0000","objects_scrubbed":289,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:01.005355+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14926,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1522,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-04-23T16:46:02.142530+0000","last_change":"2024-04-23T16:44:45.850797+0000","last_active":"2024-04-23T16:46:02.142530+0000","last_peered":"2024-04-23T16:46:02.142530+0000","last_clean":"2024-04-23T16:46:02.142530+0000","last_became_active":"2024-04-23T16:43:07.962092+0000","last_became_peered":"2024-04-23T16:43:07.962092+0000","last_unstale":"2024-04-23T16:46:02.142530+0000","last_undegraded":"2024-04-23T16:46:02.142530+0000","last_fullsized":"2024-04-23T16:46:02.142530+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:44:45.850682+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:44:45.850682+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:12.889710+0000","scrub_duration":234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"599'14848","reported_seq":15107,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-04-23T16:46:02.142512+0000","last_change":"2024-04-23T16:44:44.578218+0000","last_active":"2024-04-23T16:46:02.142512+0000","last_peered":"2024-04-23T16:46:02.142512+0000","last_clean":"2024-04-23T16:46:02.142512+0000","last_became_active":"2024-04-23T16:41:57.305944+0000","last_became_peered":"2024-04-23T16:41:57.305944+0000","last_unstale":"2024-04-23T16:46:02.142512+0000","last_undegraded":"2024-04-23T16:46:02.142512+0000","last_fullsized":"2024-04-23T16:46:02.142512+0000","mapping_epoch":540,"log_start":"555'4800","ondisk_log_start":"555'4800","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'10405","last_scrub_stamp":"2024-04-23T16:44:44.578065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:44:44.578065+0000","objects_scrubbed":4454,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":505,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14848,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1495,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-04-23T16:45:57.974464+0000","last_change":"2024-04-23T16:45:38.633704+0000","last_active":"2024-04-23T16:45:57.974464+0000","last_peered":"2024-04-23T16:45:57.974464+0000","last_clean":"2024-04-23T16:45:57.974464+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:45:57.974464+0000","last_undegraded":"2024-04-23T16:45:57.974464+0000","last_fullsized":"2024-04-23T16:45:57.974464+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:38.633554+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:45:38.633554+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:50.365278+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"599'15166","reported_seq":15437,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-04-23T16:46:02.142317+0000","last_change":"2024-04-23T16:45:54.360285+0000","last_active":"2024-04-23T16:46:02.142317+0000","last_peered":"2024-04-23T16:46:02.142317+0000","last_clean":"2024-04-23T16:46:02.142317+0000","last_became_active":"2024-04-23T16:41:57.309858+0000","last_became_peered":"2024-04-23T16:41:57.309858+0000","last_unstale":"2024-04-23T16:46:02.142317+0000","last_undegraded":"2024-04-23T16:46:02.142317+0000","last_fullsized":"2024-04-23T16:46:02.142317+0000","mapping_epoch":540,"log_start":"555'5100","ondisk_log_start":"555'5100","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"597'14858","last_scrub_stamp":"2024-04-23T16:45:54.360119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:54.360119+0000","objects_scrubbed":315,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:55.562395+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15166,"num_write_kb":485312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1523,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-04-23T16:46:02.142444+0000","last_change":"2024-04-23T16:45:30.314968+0000","last_active":"2024-04-23T16:46:02.142444+0000","last_peered":"2024-04-23T16:46:02.142444+0000","last_clean":"2024-04-23T16:46:02.142444+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:46:02.142444+0000","last_undegraded":"2024-04-23T16:46:02.142444+0000","last_fullsized":"2024-04-23T16:46:02.142444+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:30.314881+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:45:30.314881+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:54.455465+0000","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"599'14698","reported_seq":14852,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-04-23T16:46:00.992790+0000","last_change":"2024-04-23T16:45:36.077763+0000","last_active":"2024-04-23T16:46:00.992790+0000","last_peered":"2024-04-23T16:46:00.992790+0000","last_clean":"2024-04-23T16:46:00.992790+0000","last_became_active":"2024-04-23T16:41:57.310400+0000","last_became_peered":"2024-04-23T16:41:57.310400+0000","last_unstale":"2024-04-23T16:46:00.992790+0000","last_undegraded":"2024-04-23T16:46:00.992790+0000","last_fullsized":"2024-04-23T16:46:00.992790+0000","mapping_epoch":540,"log_start":"555'4600","ondisk_log_start":"555'4600","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'13167","last_scrub_stamp":"2024-04-23T16:45:36.077623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:36.077623+0000","objects_scrubbed":1539,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:59.580518+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14698,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1093,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-04-23T16:45:57.972632+0000","last_change":"2024-04-23T16:45:37.184028+0000","last_active":"2024-04-23T16:45:57.972632+0000","last_peered":"2024-04-23T16:45:57.972632+0000","last_clean":"2024-04-23T16:45:57.972632+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:45:57.972632+0000","last_undegraded":"2024-04-23T16:45:57.972632+0000","last_fullsized":"2024-04-23T16:45:57.972632+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:37.183866+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:45:37.183866+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:04.176221+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"599'14704","reported_seq":14851,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-04-23T16:45:58.615404+0000","last_change":"2024-04-23T16:45:42.270579+0000","last_active":"2024-04-23T16:45:58.615404+0000","last_peered":"2024-04-23T16:45:58.615404+0000","last_clean":"2024-04-23T16:45:58.615404+0000","last_became_active":"2024-04-23T16:41:57.306229+0000","last_became_peered":"2024-04-23T16:41:57.306229+0000","last_unstale":"2024-04-23T16:45:58.615404+0000","last_undegraded":"2024-04-23T16:45:58.615404+0000","last_fullsized":"2024-04-23T16:45:58.615404+0000","mapping_epoch":540,"log_start":"555'4700","ondisk_log_start":"555'4700","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'13614","last_scrub_stamp":"2024-04-23T16:45:42.270426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:42.270426+0000","objects_scrubbed":1112,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:53.144483+0000","scrub_duration":405,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14704,"num_write_kb":470528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1051,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-04-23T16:46:01.999742+0000","last_change":"2024-04-23T16:45:33.205702+0000","last_active":"2024-04-23T16:46:01.999742+0000","last_peered":"2024-04-23T16:46:01.999742+0000","last_clean":"2024-04-23T16:46:01.999742+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:46:01.999742+0000","last_undegraded":"2024-04-23T16:46:01.999742+0000","last_fullsized":"2024-04-23T16:46:01.999742+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:33.205593+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:45:33.205593+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:58.254203+0000","scrub_duration":121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.0","version":"599'14700","reported_seq":14953,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-04-23T16:46:02.142579+0000","last_change":"2024-04-23T16:45:00.368292+0000","last_active":"2024-04-23T16:46:02.142579+0000","last_peered":"2024-04-23T16:46:02.142579+0000","last_clean":"2024-04-23T16:46:02.142579+0000","last_became_active":"2024-04-23T16:41:57.311479+0000","last_became_peered":"2024-04-23T16:41:57.311479+0000","last_unstale":"2024-04-23T16:46:02.142579+0000","last_undegraded":"2024-04-23T16:46:02.142579+0000","last_fullsized":"2024-04-23T16:46:02.142579+0000","mapping_epoch":540,"log_start":"554'4600","ondisk_log_start":"554'4600","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"591'11135","last_scrub_stamp":"2024-04-23T16:45:00.368005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:00.368005+0000","objects_scrubbed":3585,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:06.315750+0000","scrub_duration":778,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14700,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1532,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-04-23T16:46:02.142835+0000","last_change":"2024-04-23T16:45:30.258925+0000","last_active":"2024-04-23T16:46:02.142835+0000","last_peered":"2024-04-23T16:46:02.142835+0000","last_clean":"2024-04-23T16:46:02.142835+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:46:02.142835+0000","last_undegraded":"2024-04-23T16:46:02.142835+0000","last_fullsized":"2024-04-23T16:46:02.142835+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:30.258804+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:45:30.258804+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:51.589576+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.1","version":"599'14636","reported_seq":25977,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-04-23T16:46:03.076362+0000","last_change":"2024-04-23T16:45:34.213681+0000","last_active":"2024-04-23T16:46:03.076362+0000","last_peered":"2024-04-23T16:46:03.076362+0000","last_clean":"2024-04-23T16:46:03.076362+0000","last_became_active":"2024-04-23T16:43:08.067887+0000","last_became_peered":"2024-04-23T16:43:08.067887+0000","last_unstale":"2024-04-23T16:46:03.076362+0000","last_undegraded":"2024-04-23T16:46:03.076362+0000","last_fullsized":"2024-04-23T16:46:03.076362+0000","mapping_epoch":563,"log_start":"555'4600","ondisk_log_start":"555'4600","created":540,"last_epoch_clean":595,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'12978","last_scrub_stamp":"2024-04-23T16:45:34.213502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:34.213502+0000","objects_scrubbed":1669,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:40.877992+0000","scrub_duration":431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14636,"num_write_kb":468352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5588,"num_bytes_recovered":366215168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1512,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-04-23T16:45:57.973076+0000","last_change":"2024-04-23T16:45:45.059609+0000","last_active":"2024-04-23T16:45:57.973076+0000","last_peered":"2024-04-23T16:45:57.973076+0000","last_clean":"2024-04-23T16:45:57.973076+0000","last_became_active":"2024-04-23T16:43:07.918042+0000","last_became_peered":"2024-04-23T16:43:07.918042+0000","last_unstale":"2024-04-23T16:45:57.973076+0000","last_undegraded":"2024-04-23T16:45:57.973076+0000","last_fullsized":"2024-04-23T16:45:57.973076+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:45.059503+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:45:45.059503+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:59.782775+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"599'14818","reported_seq":26448,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-04-23T16:45:58.574164+0000","last_change":"2024-04-23T16:45:38.202729+0000","last_active":"2024-04-23T16:45:58.574164+0000","last_peered":"2024-04-23T16:45:58.574164+0000","last_clean":"2024-04-23T16:45:58.574164+0000","last_became_active":"2024-04-23T16:43:08.333793+0000","last_became_peered":"2024-04-23T16:43:08.333793+0000","last_unstale":"2024-04-23T16:45:58.574164+0000","last_undegraded":"2024-04-23T16:45:58.574164+0000","last_fullsized":"2024-04-23T16:45:58.574164+0000","mapping_epoch":563,"log_start":"555'4800","ondisk_log_start":"555'4800","created":540,"last_epoch_clean":595,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'13496","last_scrub_stamp":"2024-04-23T16:45:38.202648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:38.202648+0000","objects_scrubbed":1328,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:04.203602+0000","scrub_duration":339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14818,"num_write_kb":474176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5737,"num_bytes_recovered":375980032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1406,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-04-23T16:46:00.992817+0000","last_change":"2024-04-23T16:45:28.948302+0000","last_active":"2024-04-23T16:46:00.992817+0000","last_peered":"2024-04-23T16:46:00.992817+0000","last_clean":"2024-04-23T16:46:00.992817+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:46:00.992817+0000","last_undegraded":"2024-04-23T16:46:00.992817+0000","last_fullsized":"2024-04-23T16:46:00.992817+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:28.948185+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:45:28.948185+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:44.902931+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"599'14924","reported_seq":26401,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-04-23T16:46:03.076362+0000","last_change":"2024-04-23T16:45:36.018343+0000","last_active":"2024-04-23T16:46:03.076362+0000","last_peered":"2024-04-23T16:46:03.076362+0000","last_clean":"2024-04-23T16:46:03.076362+0000","last_became_active":"2024-04-23T16:43:08.109240+0000","last_became_peered":"2024-04-23T16:43:08.109240+0000","last_unstale":"2024-04-23T16:46:03.076362+0000","last_undegraded":"2024-04-23T16:46:03.076362+0000","last_fullsized":"2024-04-23T16:46:03.076362+0000","mapping_epoch":563,"log_start":"555'4900","ondisk_log_start":"555'4900","created":540,"last_epoch_clean":594,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'13296","last_scrub_stamp":"2024-04-23T16:45:36.018091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:41:56.284274+0000","last_clean_scrub_stamp":"2024-04-23T16:45:36.018091+0000","objects_scrubbed":1632,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:36.440846+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14924,"num_write_kb":477568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5656,"num_bytes_recovered":370671616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1326,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-04-23T16:45:57.975214+0000","last_change":"2024-04-23T16:45:33.917246+0000","last_active":"2024-04-23T16:45:57.975214+0000","last_peered":"2024-04-23T16:45:57.975214+0000","last_clean":"2024-04-23T16:45:57.975214+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:45:57.975214+0000","last_undegraded":"2024-04-23T16:45:57.975214+0000","last_fullsized":"2024-04-23T16:45:57.975214+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:33.917127+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:45:33.917127+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:57.447552+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"599'15248","reported_seq":15541,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-04-23T16:46:02.142317+0000","last_change":"2024-04-23T16:45:51.785542+0000","last_active":"2024-04-23T16:46:02.142317+0000","last_peered":"2024-04-23T16:46:02.142317+0000","last_clean":"2024-04-23T16:46:02.142317+0000","last_became_active":"2024-04-23T16:41:57.309349+0000","last_became_peered":"2024-04-23T16:41:57.309349+0000","last_unstale":"2024-04-23T16:46:02.142317+0000","last_undegraded":"2024-04-23T16:46:02.142317+0000","last_fullsized":"2024-04-23T16:46:02.142317+0000","mapping_epoch":540,"log_start":"556'5201","ondisk_log_start":"556'5201","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"597'14772","last_scrub_stamp":"2024-04-23T16:45:51.785406+0000","last_deep_scrub":"597'14772","last_deep_scrub_stamp":"2024-04-23T16:45:51.785406+0000","last_clean_scrub_stamp":"2024-04-23T16:45:51.785406+0000","objects_scrubbed":604,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:51.799605+0000","scrub_duration":4584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14644,"num_write_kb":468608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1495,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-04-23T16:45:57.974763+0000","last_change":"2024-04-23T16:45:37.584810+0000","last_active":"2024-04-23T16:45:57.974763+0000","last_peered":"2024-04-23T16:45:57.974763+0000","last_clean":"2024-04-23T16:45:57.974763+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:45:57.974763+0000","last_undegraded":"2024-04-23T16:45:57.974763+0000","last_fullsized":"2024-04-23T16:45:57.974763+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:37.584692+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:45:37.584692+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:05.236607+0000","scrub_duration":121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":237358,"num_write_kb":7595393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22695,"num_bytes_recovered":1487339520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1411842048,"data_stored":1411842048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160761,"ondisk_log_size":160761,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1474560,"data_stored":1460352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057009,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":726364,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":93018788,"statfs":{"total":95995035648,"available":95251238912,"internally_reserved":0,"allocated":1069056,"data_stored":694373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":742717892},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351352,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612989,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545596,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199556,"statfs":{"total":95995035648,"available":95436345344,"internally_reserved":0,"allocated":765952,"data_stored":379261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":557906368},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841921,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":375360,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93369792,"statfs":{"total":95995035648,"available":95610667008,"internally_reserved":0,"allocated":1298432,"data_stored":908189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168966,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":609440,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93135712,"statfs":{"total":95995035648,"available":95370969088,"internally_reserved":0,"allocated":741376,"data_stored":366693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":623311306},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463307,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545564,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199588,"statfs":{"total":95995035648,"available":95436378112,"internally_reserved":0,"allocated":741376,"data_stored":366693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137448,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":235492,"kb_used_data":724,"kb_used_omap":231,"kb_used_meta":234520,"kb_avail":93509660,"statfs":{"total":95995035648,"available":95753891840,"internally_reserved":0,"allocated":741376,"data_stored":366693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":240149392},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738794,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":581692,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93163460,"statfs":{"total":95995035648,"available":95399383040,"internally_reserved":0,"allocated":765952,"data_stored":379261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269464,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921172,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":739964,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93005188,"statfs":{"total":95995035648,"available":95237312512,"internally_reserved":0,"allocated":765952,"data_stored":379261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756939205},"hb_peers":[0,1,2,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757647,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":641604,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93103548,"statfs":{"total":95995035648,"available":95338033152,"internally_reserved":0,"allocated":970752,"data_stored":580509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1411842048,"data_stored":1411842048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:46:08.360 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-23T16:46:08.360 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-04-23T16:46:08.450 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:46:08.451 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:46:08.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:46:08.451 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-23T16:46:08.451 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:46:09.222 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:46:09.222 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:46:09.236 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:46:09.257 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:46:09.277 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:46:09.296 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:46:09.297 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:46:09.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:09.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:46:09.317 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:46:09.337 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:46:09.367 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:46:09.387 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:46:09.402 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:46:09.407 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:46:09.407 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:46:09.407 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:46:09.407 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:46:09.407 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:46:09.408 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:46:09.408 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:46:09.408 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:46:09.408 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:46:09.408 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:46:09.408 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:46:09.408 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:46:09.409 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:46:09.434 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:09.458 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:46:09.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:09.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:46:09.561 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:46:09.591 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:09.831 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-23T16:46:09.840 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:46:09.842 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:09 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:46:09.842 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:09 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:46:09.842 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:09 smithi103 ceph-mon[39629]: osdmap e602: 12 total, 11 up, 9 in 2024-04-23T16:46:09.842 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:09 smithi103 ceph-mon[39629]: pgmap v1699: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 24 op/s 2024-04-23T16:46:09.842 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:09 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/108175424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:46:09.842 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:09.842 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:09 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:09.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:09.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:46:09.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:09 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:46:09.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:09 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:46:09.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:09 smithi148 ceph-mon[44520]: osdmap e602: 12 total, 11 up, 9 in 2024-04-23T16:46:09.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:09 smithi148 ceph-mon[44520]: pgmap v1699: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 24 op/s 2024-04-23T16:46:09.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:09 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/108175424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:46:09.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:09.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:09.944 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:46:09.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:09 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:46:09.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:09 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:46:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:09 smithi188 ceph-mon[44101]: osdmap e602: 12 total, 11 up, 9 in 2024-04-23T16:46:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:09 smithi188 ceph-mon[44101]: pgmap v1699: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 24 op/s 2024-04-23T16:46:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:09 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/108175424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:46:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:09.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:09.976 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:09.998 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:46:10.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:10.001 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:46:10.102 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:46:10.108 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:10.134 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:10.156 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:46:10.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:10.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:46:10.261 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:46:10.291 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:10.312 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:46:10.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:10.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:46:10.416 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:46:10.445 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:10.588 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.588 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.589 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.589 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.589 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.592 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:46:10.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:10.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:46:10.614 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:46:10.615 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:46:10.674 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:10.700 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:10.733 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:10.755 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:10.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:10.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:10.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.860 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:46:10.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.890 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:10.910 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:46:10.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:10.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:46:10.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:10.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:11.015 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:46:11.043 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:11.063 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:46:11.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:11.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:46:11.168 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:46:11.201 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:11.221 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:46:11.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:11.224 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:46:11.325 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:46:11.354 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:11.374 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:46:11.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:11.377 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:46:11.478 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:46:11.507 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:11.526 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:46:11.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:11.529 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:46:11.630 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:46:11.659 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:11.677 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:46:11.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:11.680 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:46:11.781 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:11.813 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:11.835 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:11.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:11.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:11.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:11 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:46:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:11 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:46:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:11 smithi103 ceph-mon[39629]: osdmap e603: 12 total, 11 up, 9 in 2024-04-23T16:46:11.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:11 smithi103 ceph-mon[39629]: pgmap v1701: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:11.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:11 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:46:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:11 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:46:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:11 smithi148 ceph-mon[44520]: osdmap e603: 12 total, 11 up, 9 in 2024-04-23T16:46:11.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:11 smithi148 ceph-mon[44520]: pgmap v1701: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:11.940 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:46:11.971 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:11.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:11 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:46:11.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:11 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:46:11.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:11 smithi188 ceph-mon[44101]: osdmap e603: 12 total, 11 up, 9 in 2024-04-23T16:46:11.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:11 smithi188 ceph-mon[44101]: pgmap v1701: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:12.023 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:46:12.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:12.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:46:12.170 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:46:12.198 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:12.547 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:46:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:12.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:46:12.616 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:46:12.653 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:46:12.686 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:12.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:12 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:46:12.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:12 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:46:12.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:12 smithi103 ceph-mon[39629]: osdmap e604: 12 total, 11 up, 9 in 2024-04-23T16:46:12.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:12 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:46:12.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:12 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:46:12.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:12 smithi148 ceph-mon[44520]: osdmap e604: 12 total, 11 up, 9 in 2024-04-23T16:46:12.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:12 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:46:12.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:12 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:46:12.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:12 smithi188 ceph-mon[44101]: osdmap e604: 12 total, 11 up, 9 in 2024-04-23T16:46:13.135 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:46:13.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:13.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:46:13.238 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:46:13.271 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:13.293 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:46:13.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:13.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:46:13.398 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:46:13.430 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:13.454 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:46:13.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:13.458 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:46:13.559 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:46:13.590 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:13.612 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:46:13.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:13.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:46:13.717 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:46:13.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:13 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:46:13.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:13 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:46:13.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:13 smithi188 ceph-mon[44101]: osdmap e605: 12 total, 11 up, 9 in 2024-04-23T16:46:13.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:13 smithi188 ceph-mon[44101]: pgmap v1704: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:13.750 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:13.771 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:46:13.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:13.775 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:46:13.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:13 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:46:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:13 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:46:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:13 smithi148 ceph-mon[44520]: osdmap e605: 12 total, 11 up, 9 in 2024-04-23T16:46:13.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:13 smithi148 ceph-mon[44520]: pgmap v1704: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:13.875 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:46:13.907 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:13.929 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:46:13.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:13.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:46:14.008 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:13 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:46:14.009 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:13 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:46:14.009 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:13 smithi103 ceph-mon[39629]: osdmap e605: 12 total, 11 up, 9 in 2024-04-23T16:46:14.009 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:13 smithi103 ceph-mon[39629]: pgmap v1704: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:14.032 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:46:14.064 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:14.083 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:46:14.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:14.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:46:14.187 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:46:14.215 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:14.235 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:46:14.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:14.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:46:14.339 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:46:14.370 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:14.390 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:14.391 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:46:14.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:14.394 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:46:14.424 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:14.494 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:46:14.524 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:14.621 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:46:14.621 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'unique_pool_5' removed 2024-04-23T16:46:14.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:14 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/903598760' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:46:14.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:14 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/855093188' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:46:14.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:14 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/903598760' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:46:14.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:14 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/855093188' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:46:14.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:14 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/903598760' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:46:14.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:14 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/855093188' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:46:15.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:15 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:46:15.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:15 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/903598760' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:46:15.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:15 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/855093188' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:46:15.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:15 smithi148 ceph-mon[44520]: osdmap e606: 12 total, 11 up, 9 in 2024-04-23T16:46:15.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:15 smithi148 ceph-mon[44520]: pgmap v1706: 11 pgs: 11 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:15.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:15 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:46:15.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:15 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/903598760' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:46:15.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:15 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/855093188' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:46:15.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:15 smithi188 ceph-mon[44101]: osdmap e606: 12 total, 11 up, 9 in 2024-04-23T16:46:15.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:15 smithi188 ceph-mon[44101]: pgmap v1706: 11 pgs: 11 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:16.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:15 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:46:16.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:15 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/903598760' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:46:16.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:15 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/855093188' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:46:16.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:15 smithi103 ceph-mon[39629]: osdmap e606: 12 total, 11 up, 9 in 2024-04-23T16:46:16.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:15 smithi103 ceph-mon[39629]: pgmap v1706: 11 pgs: 11 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:16.074 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:46:16.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:16.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:46:16.097 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-23T16:46:16.098 INFO:tasks.radosbench:Beginning radosbench... 2024-04-23T16:46:16.098 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-04-23T16:46:16.098 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_6 16 2024-04-23T16:46:16.178 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:16.210 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:16.230 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:16.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:16.334 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:46:16.364 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:16.387 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:46:16.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:16.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:46:16.490 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:46:16.517 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:16.865 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:16.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:16.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:16.969 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:46:16.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:16 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:46:16.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:16 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:46:16.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:16 smithi188 ceph-mon[44101]: osdmap e607: 12 total, 11 up, 9 in 2024-04-23T16:46:17.002 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:17.015 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:16 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:46:17.016 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:16 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:46:17.016 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:16 smithi103 ceph-mon[39629]: osdmap e607: 12 total, 11 up, 9 in 2024-04-23T16:46:17.023 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:46:17.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:17.027 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:46:17.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:16 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:46:17.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:16 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:46:17.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:16 smithi148 ceph-mon[44520]: osdmap e607: 12 total, 11 up, 9 in 2024-04-23T16:46:17.127 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:17.159 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:17.182 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:17.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:17.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:17.286 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:46:17.315 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:17.336 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:46:17.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:17.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:46:17.441 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:17.469 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:17.488 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:17.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:17.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:17.594 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:17.620 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:17.641 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:17.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:17.746 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:46:17.749 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:17 smithi103 ceph-mon[39629]: pgmap v1708: 11 pgs: 11 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:17.749 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:17 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:46:17.749 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:17 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:46:17.778 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:17.799 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:46:17.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:17.803 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:46:17.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:17 smithi148 ceph-mon[44520]: pgmap v1708: 11 pgs: 11 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:17.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:17 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:46:17.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:17 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:46:17.903 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:46:17.930 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:17.949 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:46:17.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:17.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:46:17.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:17 smithi188 ceph-mon[44101]: pgmap v1708: 11 pgs: 11 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:17.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:17 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:46:17.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:17 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:46:18.052 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:46:18.084 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:18.105 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:46:18.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:18.108 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:46:18.209 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:18.237 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:18.256 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:18.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:18.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:18.295 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:18.360 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:46:18.392 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:18.413 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:46:18.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:18.416 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:46:18.517 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:46:18.546 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:18.567 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:46:18.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:18.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:46:18.671 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:46:18.699 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:18.719 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:46:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:18.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:46:18.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:18 smithi188 ceph-mon[44101]: osdmap e608: 12 total, 11 up, 9 in 2024-04-23T16:46:18.823 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:46:18.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:18 smithi103 ceph-mon[39629]: osdmap e608: 12 total, 11 up, 9 in 2024-04-23T16:46:18.853 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:18.871 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:46:19.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:18 smithi148 ceph-mon[44520]: osdmap e608: 12 total, 11 up, 9 in 2024-04-23T16:46:19.302 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:46:19.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:19.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:46:19.405 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:46:19.438 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:19.459 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:46:19.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:19.462 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:46:19.564 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:46:19.591 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:46:19.592 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:19.597 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:19.621 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:46:19.641 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:46:19.663 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:46:19.682 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:46:19.702 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:46:19.722 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:46:19.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:19 smithi188 ceph-mon[44101]: pgmap v1710: 11 pgs: 11 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:19.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:19 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3349256266' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:46:19.744 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:46:19.765 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:46:19.785 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:46:19.805 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:46:19.805 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:46:19.805 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:46:19.805 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:46:19.806 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:46:19.806 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:46:19.806 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:46:19.806 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:46:19.806 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:46:19.806 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:46:19.806 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:46:19.806 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:46:19.806 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:46:19.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:19 smithi103 ceph-mon[39629]: pgmap v1710: 11 pgs: 11 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:19.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:19 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3349256266' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:46:20.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:19 smithi148 ceph-mon[44520]: pgmap v1710: 11 pgs: 11 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:20.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:19 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3349256266' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:46:20.253 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:20.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:20.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:20.358 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:46:20.391 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:20.412 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:46:20.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:20.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:46:20.516 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:46:20.551 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:20.787 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:46:20.787 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:46:20.787 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-23T16:46:20.788 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:46:20.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:20.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:21.001 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:46:21.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:21.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:46:21.105 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:46:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:21.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:21.137 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:21.159 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:46:21.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:21.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:46:21.264 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:46:21.293 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:21.313 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:46:21.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:21.318 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:46:21.419 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:46:21.449 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:21.489 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:21.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:21.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:21.515 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-23T16:46:21.595 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:46:21.625 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:21.646 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:46:21.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:21.650 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:46:21.751 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:46:21.783 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:21.806 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:46:21.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:21.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:46:21.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:21 smithi103 ceph-mon[39629]: pgmap v1711: 11 pgs: 11 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:21.909 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:46:21.937 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:21.957 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:21.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:21.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:21 smithi188 ceph-mon[44101]: pgmap v1711: 11 pgs: 11 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:22.061 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:46:22.092 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:21 smithi148 ceph-mon[44520]: pgmap v1711: 11 pgs: 11 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:22.093 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:22.115 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:46:22.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:22.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:46:22.220 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:22.250 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:22.269 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:22.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:22.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:22.373 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:46:22.406 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:22.426 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:46:22.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:22.430 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:46:22.531 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:46:22.559 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:22.612 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:22.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:22.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:22.728 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:46:22.758 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:23.768 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:46:23.768 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:46:24.005 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:46:24.006 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:23 smithi103 ceph-mon[39629]: pgmap v1712: 11 pgs: 11 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:24.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:24.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:46:24.109 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:46:24.141 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:24.163 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:46:24.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:24.166 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:46:24.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:23 smithi188 ceph-mon[44101]: pgmap v1712: 11 pgs: 11 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:24.267 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:46:24.297 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:23 smithi148 ceph-mon[44520]: pgmap v1712: 11 pgs: 11 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:24.299 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:24.321 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:46:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:24.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:46:24.425 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:46:24.455 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:24.475 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:46:24.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:24.478 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:46:24.580 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:24.610 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:24.623 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:24.629 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:24.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:24.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:24.734 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:24.762 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:24.781 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:24.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:24.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:24.886 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:46:24.916 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:24.935 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:46:24.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:24.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:46:25.039 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:46:25.065 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:25.093 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:46:25.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:25.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:46:25.197 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:46:25.227 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:25.248 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:46:25.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:25.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:46:25.352 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:46:25.359 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:46:25.360 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'unique_pool_6' created 2024-04-23T16:46:25.385 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:25.406 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:46:25.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:25.409 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:46:25.510 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:46:25.538 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:25.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:25 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1965464382' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-04-23T16:46:25.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:25 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3126504541' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:46:25.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:25 smithi103 ceph-mon[39629]: pgmap v1713: 11 pgs: 11 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:25.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:25 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1965464382' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-04-23T16:46:25.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:25 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3126504541' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:46:25.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:25 smithi148 ceph-mon[44520]: pgmap v1713: 11 pgs: 11 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:25.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:25 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1965464382' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-04-23T16:46:25.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:25 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3126504541' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:46:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:25 smithi188 ceph-mon[44101]: pgmap v1713: 11 pgs: 11 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:25.769 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:46:26.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:26 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:46:26.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:26 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1965464382' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-04-23T16:46:26.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:26 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3126504541' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:46:26.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:26 smithi148 ceph-mon[44520]: osdmap e609: 12 total, 11 up, 9 in 2024-04-23T16:46:26.710 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:26 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:46:26.710 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:26 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1965464382' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-04-23T16:46:26.711 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:26 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3126504541' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:46:26.711 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:26 smithi103 ceph-mon[39629]: osdmap e609: 12 total, 11 up, 9 in 2024-04-23T16:46:26.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:26 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:46:26.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:26 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1965464382' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-04-23T16:46:26.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:26 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3126504541' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:46:26.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:26 smithi188 ceph-mon[44101]: osdmap e609: 12 total, 11 up, 9 in 2024-04-23T16:46:26.742 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-04-23T16:46:26.788 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:26.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:26.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:26.892 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:46:26.924 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:26.946 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:46:26.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:26.949 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:46:27.050 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:46:27.082 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:27.102 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:46:27.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:27.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:46:27.206 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:46:27.232 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:27.251 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:46:27.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:27.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:46:27.257 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:27.357 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:46:27.386 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:27.406 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:46:27.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:27.409 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:46:27.509 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:46:27.542 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:27.564 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:46:27.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:27.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:46:27.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:27 smithi103 ceph-mon[39629]: osdmap e610: 12 total, 11 up, 9 in 2024-04-23T16:46:27.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:27 smithi103 ceph-mon[39629]: pgmap v1716: 27 pgs: 2 creating+peering, 9 unknown, 16 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:27.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:27 smithi148 ceph-mon[44520]: osdmap e610: 12 total, 11 up, 9 in 2024-04-23T16:46:27.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:27 smithi148 ceph-mon[44520]: pgmap v1716: 27 pgs: 2 creating+peering, 9 unknown, 16 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:27.668 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:46:27.697 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:27.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:27 smithi188 ceph-mon[44101]: osdmap e610: 12 total, 11 up, 9 in 2024-04-23T16:46:27.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:27 smithi188 ceph-mon[44101]: pgmap v1716: 27 pgs: 2 creating+peering, 9 unknown, 16 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:27.878 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:46:27.878 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:46:27.878 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-23T16:46:27.878 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-23T16:46:27.878 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:46:27.895 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:46:27.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:27.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:46:27.999 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:28.032 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:28.054 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:28.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:28.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:28.158 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:28.188 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:28.208 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:28.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:28.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:28.311 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:46:28.344 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:28.366 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:46:28.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:28.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:46:28.387 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:46:28.470 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:46:28.500 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:28.521 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:46:28.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:28.524 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:46:28.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:28 smithi148 ceph-mon[44520]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:46:28.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:28 smithi148 ceph-mon[44520]: osdmap e611: 12 total, 11 up, 9 in 2024-04-23T16:46:28.625 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:28.657 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:28.679 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:28.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:28.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:28.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:28 smithi188 ceph-mon[44101]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:46:28.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:28 smithi188 ceph-mon[44101]: osdmap e611: 12 total, 11 up, 9 in 2024-04-23T16:46:28.783 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:46:28.812 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:28.822 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:28 smithi103 ceph-mon[39629]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:46:28.822 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:28 smithi103 ceph-mon[39629]: osdmap e611: 12 total, 11 up, 9 in 2024-04-23T16:46:28.831 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:46:28.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:28.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:46:28.847 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-23T16:46:28.935 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:46:28.967 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:28.989 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:46:28.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:28.992 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:46:29.093 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:29.125 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:29.144 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:29.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:29.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:29.249 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:46:29.275 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:29.295 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:29.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:29.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:29.399 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:29.428 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:29.450 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:29.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:29.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:29.552 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:46:29.584 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:29.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:29 smithi103 ceph-mon[39629]: pgmap v1718: 27 pgs: 2 creating+peering, 3 unknown, 22 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:29.605 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:46:29.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:29.609 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:46:29.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:29 smithi148 ceph-mon[44520]: pgmap v1718: 27 pgs: 2 creating+peering, 3 unknown, 22 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:29.703 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:29.707 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:29.710 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:46:29.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:29 smithi188 ceph-mon[44101]: pgmap v1718: 27 pgs: 2 creating+peering, 3 unknown, 22 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:29.739 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:29.758 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:46:29.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:29.761 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:46:29.862 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:29.896 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:29.916 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:29.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:29.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:30.020 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:46:30.049 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:31.985 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:31 smithi103 ceph-mon[39629]: pgmap v1719: 27 pgs: 2 creating+peering, 25 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:32.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:31 smithi188 ceph-mon[44101]: pgmap v1719: 27 pgs: 2 creating+peering, 25 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:32.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:31 smithi148 ceph-mon[44520]: pgmap v1719: 27 pgs: 2 creating+peering, 25 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:32.400 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:46:32.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:32.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:46:32.416 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:32.505 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:46:32.601 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:32.622 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:46:32.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:32.625 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:46:32.726 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:46:32.756 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:32.954 INFO:teuthology.orchestra.run.smithi103.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-04-23T16:46:33.088 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:32 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2355642541' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:46:33.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:32 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2355642541' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:46:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:32 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2355642541' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:46:33.504 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:46:33.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:33.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:46:33.608 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:46:33.640 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:33.662 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:46:33.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:33.665 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:46:33.766 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:33.799 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:33.820 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:33.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:33.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:33.926 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:46:33.960 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:33.964 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:46:33.964 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:46:33.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:46:33.964 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:46:33.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.964 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:161911 6.9 6:90448c89:::benchmark_data_smithi188_130221_object7450:head [delete] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-23T16:46:33.964 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:39:08.519032+0000", 2024-04-23T16:46:33.964 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 445.24111905799998, 2024-04-23T16:46:33.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.181464707, 2024-04-23T16:46:33.964 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:33.964 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:33.964 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:33.965 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:33.965 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:46:33.965 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:46:33.965 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 161911 2024-04-23T16:46:33.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.965 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:33.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:33.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.519032+0000", 2024-04-23T16:46:33.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:33.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:33.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.519032+0000", 2024-04-23T16:46:33.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:33.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:33.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.519035+0000", 2024-04-23T16:46:33.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.379e-06 2024-04-23T16:46:33.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:33.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.519041+0000", 2024-04-23T16:46:33.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5649999999999998e-06 2024-04-23T16:46:33.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:33.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.519055+0000", 2024-04-23T16:46:33.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4079e-05 2024-04-23T16:46:33.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:33.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.519063+0000", 2024-04-23T16:46:33.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0299999999999996e-06 2024-04-23T16:46:33.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:33.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.519127+0000", 2024-04-23T16:46:33.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4770000000000005e-05 2024-04-23T16:46:33.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:33.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.519256+0000", 2024-04-23T16:46:33.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000128961 2024-04-23T16:46:33.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 9", 2024-04-23T16:46:33.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.519378+0000", 2024-04-23T16:46:33.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000121719 2024-04-23T16:46:33.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:33.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.523830+0000", 2024-04-23T16:46:33.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044523330000000002 2024-04-23T16:46:33.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.969 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:33.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.700390+0000", 2024-04-23T16:46:33.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.176559306 2024-04-23T16:46:33.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.969 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:33.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.700442+0000", 2024-04-23T16:46:33.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1992e-05 2024-04-23T16:46:33.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.969 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:33.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.700497+0000", 2024-04-23T16:46:33.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5572999999999998e-05 2024-04-23T16:46:33.970 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:33.970 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:33.970 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:33.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.970 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:161947 6.9 6:968d66de:::benchmark_data_smithi188_130221_object7486:head [delete] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-23T16:46:33.970 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:39:08.559025+0000", 2024-04-23T16:46:33.970 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 445.201126053, 2024-04-23T16:46:33.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.141566362, 2024-04-23T16:46:33.970 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:33.970 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:33.970 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:33.971 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:33.971 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:46:33.971 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:46:33.971 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 161947 2024-04-23T16:46:33.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.971 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:33.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:33.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.559025+0000", 2024-04-23T16:46:33.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:33.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:33.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.559025+0000", 2024-04-23T16:46:33.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:33.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:33.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.559028+0000", 2024-04-23T16:46:33.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.305e-06 2024-04-23T16:46:33.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:33.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.559035+0000", 2024-04-23T16:46:33.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2860000000000002e-06 2024-04-23T16:46:33.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:33.973 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.559048+0000", 2024-04-23T16:46:33.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3165e-05 2024-04-23T16:46:33.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.973 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:33.973 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.559057+0000", 2024-04-23T16:46:33.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4950000000000008e-06 2024-04-23T16:46:33.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.973 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:33.973 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.559123+0000", 2024-04-23T16:46:33.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5803999999999995e-05 2024-04-23T16:46:33.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.973 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:33.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.559265+0000", 2024-04-23T16:46:33.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000142461 2024-04-23T16:46:33.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 9", 2024-04-23T16:46:33.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.559390+0000", 2024-04-23T16:46:33.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000125105 2024-04-23T16:46:33.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:33.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.564253+0000", 2024-04-23T16:46:33.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0048625179999999997 2024-04-23T16:46:33.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:33.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.700535+0000", 2024-04-23T16:46:33.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13628216400000001 2024-04-23T16:46:33.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:33.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.700565+0000", 2024-04-23T16:46:33.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0040999999999999e-05 2024-04-23T16:46:33.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:33.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.700592+0000", 2024-04-23T16:46:33.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7018000000000001e-05 2024-04-23T16:46:33.975 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:33.975 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:33.976 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:33.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.976 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:161952 6.9 6:975064cb:::benchmark_data_smithi188_130221_object7491:head [delete] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-23T16:46:33.976 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:39:08.565539+0000", 2024-04-23T16:46:33.976 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 445.19461250400002, 2024-04-23T16:46:33.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13518182100000001, 2024-04-23T16:46:33.976 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:33.976 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:33.976 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:33.976 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:33.976 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:46:33.976 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:46:33.976 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 161952 2024-04-23T16:46:33.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.977 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:33.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:33.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.565539+0000", 2024-04-23T16:46:33.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:33.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:33.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.565539+0000", 2024-04-23T16:46:33.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:33.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:33.978 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.565541+0000", 2024-04-23T16:46:33.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3740000000000001e-06 2024-04-23T16:46:33.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.978 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:33.978 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.565550+0000", 2024-04-23T16:46:33.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0229999999999998e-06 2024-04-23T16:46:33.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:33.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.565562+0000", 2024-04-23T16:46:33.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1148999999999999e-05 2024-04-23T16:46:33.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:33.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.565570+0000", 2024-04-23T16:46:33.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1370000000000002e-06 2024-04-23T16:46:33.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:33.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.565621+0000", 2024-04-23T16:46:33.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1366000000000003e-05 2024-04-23T16:46:33.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.980 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:33.980 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.565721+0000", 2024-04-23T16:46:33.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100284 2024-04-23T16:46:33.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.980 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 9", 2024-04-23T16:46:33.980 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.565807+0000", 2024-04-23T16:46:33.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5907000000000003e-05 2024-04-23T16:46:33.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.981 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:33.981 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.573721+0000", 2024-04-23T16:46:33.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079139759999999996 2024-04-23T16:46:33.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.981 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.981 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:33.981 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.700640+0000", 2024-04-23T16:46:33.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.12691902999999999 2024-04-23T16:46:33.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.981 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.981 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:33.981 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.700685+0000", 2024-04-23T16:46:33.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4308000000000003e-05 2024-04-23T16:46:33.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.982 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.982 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:33.982 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.700721+0000", 2024-04-23T16:46:33.982 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6266999999999997e-05 2024-04-23T16:46:33.982 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:33.982 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:33.982 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:33.982 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.982 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.982 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:161956 6.9 6:9cc2279d:::benchmark_data_smithi188_130221_object7495:head [delete] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-23T16:46:33.982 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:39:08.566895+0000", 2024-04-23T16:46:33.982 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 445.19325605, 2024-04-23T16:46:33.982 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13389725599999999, 2024-04-23T16:46:33.983 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:33.983 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:33.983 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:33.983 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:33.983 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:46:33.983 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:46:33.983 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 161956 2024-04-23T16:46:33.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.983 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:33.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.983 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:33.983 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.566895+0000", 2024-04-23T16:46:33.983 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:33.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.984 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:33.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.566895+0000", 2024-04-23T16:46:33.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:33.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.984 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.984 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:33.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.566897+0000", 2024-04-23T16:46:33.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1689999999999999e-06 2024-04-23T16:46:33.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.984 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.985 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:33.985 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.566902+0000", 2024-04-23T16:46:33.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9509999999999997e-06 2024-04-23T16:46:33.985 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.986 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.986 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:33.986 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.566913+0000", 2024-04-23T16:46:33.986 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1178999999999999e-05 2024-04-23T16:46:33.986 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.986 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.986 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:33.986 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.566919+0000", 2024-04-23T16:46:33.986 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8329999999999999e-06 2024-04-23T16:46:33.986 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.986 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.986 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:33.986 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.569640+0000", 2024-04-23T16:46:33.986 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027217640000000002 2024-04-23T16:46:33.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.987 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:33.987 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.569761+0000", 2024-04-23T16:46:33.987 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000120698 2024-04-23T16:46:33.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.987 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 9", 2024-04-23T16:46:33.987 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.569885+0000", 2024-04-23T16:46:33.987 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012345300000000001 2024-04-23T16:46:33.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.987 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:33.987 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.579480+0000", 2024-04-23T16:46:33.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0095954939999999996 2024-04-23T16:46:33.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:33.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.700746+0000", 2024-04-23T16:46:33.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.121265779 2024-04-23T16:46:33.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:33.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.700772+0000", 2024-04-23T16:46:33.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6546e-05 2024-04-23T16:46:33.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:33.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.700793+0000", 2024-04-23T16:46:33.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.039e-05 2024-04-23T16:46:33.989 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:33.989 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:33.989 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:33.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.989 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:161962 6.9 6:9aa885fa:::benchmark_data_smithi188_130221_object7501:head [delete] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-23T16:46:33.989 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:39:08.572775+0000", 2024-04-23T16:46:33.989 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 445.18737615200001, 2024-04-23T16:46:33.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13283278800000001, 2024-04-23T16:46:33.989 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:33.989 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:33.989 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:33.989 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:33.990 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:46:33.990 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:46:33.990 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 161962 2024-04-23T16:46:33.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.990 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:33.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:33.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.572775+0000", 2024-04-23T16:46:33.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:33.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:33.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.572775+0000", 2024-04-23T16:46:33.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:33.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.991 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:33.991 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.572778+0000", 2024-04-23T16:46:33.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.249e-06 2024-04-23T16:46:33.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.991 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:33.991 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.572785+0000", 2024-04-23T16:46:33.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8909999999999998e-06 2024-04-23T16:46:33.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.991 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:33.991 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.572797+0000", 2024-04-23T16:46:33.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2282e-05 2024-04-23T16:46:33.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:33.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.572805+0000", 2024-04-23T16:46:33.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8909999999999995e-06 2024-04-23T16:46:33.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:33.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.573751+0000", 2024-04-23T16:46:33.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00094609399999999997 2024-04-23T16:46:33.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:33.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.573885+0000", 2024-04-23T16:46:33.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013388199999999999 2024-04-23T16:46:33.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 9", 2024-04-23T16:46:33.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.574005+0000", 2024-04-23T16:46:33.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000120321 2024-04-23T16:46:33.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:33.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.579488+0000", 2024-04-23T16:46:33.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054833440000000002 2024-04-23T16:46:33.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:33.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.705545+0000", 2024-04-23T16:46:33.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.12605680999999999 2024-04-23T16:46:33.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:33.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.705579+0000", 2024-04-23T16:46:33.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4091999999999998e-05 2024-04-23T16:46:33.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:33.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.705608+0000", 2024-04-23T16:46:33.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8932e-05 2024-04-23T16:46:33.994 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:33.994 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:33.994 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:33.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.995 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:161980 6.9 6:990e71ee:::benchmark_data_smithi188_130221_object7519:head [delete] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-23T16:46:33.995 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:39:08.599509+0000", 2024-04-23T16:46:33.995 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 445.160642725, 2024-04-23T16:46:33.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.106231644, 2024-04-23T16:46:33.995 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:33.995 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:33.995 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:33.995 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:33.995 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:46:33.995 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:46:33.995 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 161980 2024-04-23T16:46:33.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.996 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:33.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:33.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.599509+0000", 2024-04-23T16:46:33.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:33.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:33.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.599509+0000", 2024-04-23T16:46:33.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:33.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:33.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.599511+0000", 2024-04-23T16:46:33.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1260000000000001e-06 2024-04-23T16:46:33.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.997 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.997 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:33.997 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.599516+0000", 2024-04-23T16:46:33.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7690000000000004e-06 2024-04-23T16:46:33.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.997 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.997 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:33.997 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.599527+0000", 2024-04-23T16:46:33.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0815000000000001e-05 2024-04-23T16:46:33.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.997 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.997 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:33.997 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.599533+0000", 2024-04-23T16:46:33.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9859999999999997e-06 2024-04-23T16:46:33.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:33.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.599572+0000", 2024-04-23T16:46:33.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8274e-05 2024-04-23T16:46:33.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:33.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.599659+0000", 2024-04-23T16:46:33.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6833e-05 2024-04-23T16:46:33.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 9", 2024-04-23T16:46:33.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.599734+0000", 2024-04-23T16:46:33.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5506999999999994e-05 2024-04-23T16:46:33.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:33.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.603418+0000", 2024-04-23T16:46:33.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0036839949999999998 2024-04-23T16:46:33.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:33.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:33.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.705650+0000", 2024-04-23T16:46:33.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.102231473 2024-04-23T16:46:33.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:33.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:34.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.705700+0000", 2024-04-23T16:46:34.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0012000000000002e-05 2024-04-23T16:46:34.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:34.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:39:08.705740+0000", 2024-04-23T16:46:34.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0853999999999999e-05 2024-04-23T16:46:34.000 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.000 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:34.000 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.000 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.26584.0:34120 7.9 7:9aa2be71:::benchmark_data_smithi188_141483_object34119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e549)", 2024-04-23T16:46:34.001 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:42:19.008592+0000", 2024-04-23T16:46:34.001 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 254.751559649, 2024-04-23T16:46:34.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.092520477000000004, 2024-04-23T16:46:34.001 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:34.001 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:34.001 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:34.001 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:34.001 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.26584", 2024-04-23T16:46:34.001 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:34.001 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 34120 2024-04-23T16:46:34.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.001 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:34.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:34.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.008592+0000", 2024-04-23T16:46:34.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:34.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.008592+0000", 2024-04-23T16:46:34.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:34.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.008593+0000", 2024-04-23T16:46:34.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8100000000000002e-07 2024-04-23T16:46:34.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:34.003 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.008631+0000", 2024-04-23T16:46:34.003 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7922e-05 2024-04-23T16:46:34.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.003 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:34.003 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.008637+0000", 2024-04-23T16:46:34.003 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8619999999999997e-06 2024-04-23T16:46:34.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.003 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:34.003 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.008643+0000", 2024-04-23T16:46:34.003 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9619999999999998e-06 2024-04-23T16:46:34.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:34.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.008691+0000", 2024-04-23T16:46:34.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7632000000000001e-05 2024-04-23T16:46:34.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:34.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.008804+0000", 2024-04-23T16:46:34.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112697 2024-04-23T16:46:34.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 11", 2024-04-23T16:46:34.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.009976+0000", 2024-04-23T16:46:34.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0011720999999999999 2024-04-23T16:46:34.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:34.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.012333+0000", 2024-04-23T16:46:34.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023574350000000002 2024-04-23T16:46:34.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:34.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.101010+0000", 2024-04-23T16:46:34.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.088676560000000001 2024-04-23T16:46:34.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:34.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.101064+0000", 2024-04-23T16:46:34.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3878999999999997e-05 2024-04-23T16:46:34.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:34.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.101112+0000", 2024-04-23T16:46:34.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8646999999999997e-05 2024-04-23T16:46:34.006 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.006 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:34.006 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.006 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.26584.0:34130 7.4 7:269cdfca:::benchmark_data_smithi188_141483_object34129:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e549)", 2024-04-23T16:46:34.006 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:42:19.011397+0000", 2024-04-23T16:46:34.006 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 254.74875424300001, 2024-04-23T16:46:34.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.094595981999999995, 2024-04-23T16:46:34.006 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:34.007 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:34.007 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:34.007 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:34.007 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.26584", 2024-04-23T16:46:34.007 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:34.007 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 34130 2024-04-23T16:46:34.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.007 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:34.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.007 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:34.007 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.011397+0000", 2024-04-23T16:46:34.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.008 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:34.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.011397+0000", 2024-04-23T16:46:34.008 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.008 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:34.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.011398+0000", 2024-04-23T16:46:34.008 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2399999999999996e-07 2024-04-23T16:46:34.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.008 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:34.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.011497+0000", 2024-04-23T16:46:34.008 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8382999999999996e-05 2024-04-23T16:46:34.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.009 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.009 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:34.009 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.011504+0000", 2024-04-23T16:46:34.009 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2250000000000003e-06 2024-04-23T16:46:34.009 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.009 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.009 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:34.009 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.011510+0000", 2024-04-23T16:46:34.009 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3260000000000001e-06 2024-04-23T16:46:34.009 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.009 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.009 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:34.009 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.099543+0000", 2024-04-23T16:46:34.009 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.088032885000000005 2024-04-23T16:46:34.009 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.010 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:34.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.099687+0000", 2024-04-23T16:46:34.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014401600000000001 2024-04-23T16:46:34.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.010 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:46:34.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.099841+0000", 2024-04-23T16:46:34.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000154148 2024-04-23T16:46:34.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.010 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:34.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.104474+0000", 2024-04-23T16:46:34.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046330399999999997 2024-04-23T16:46:34.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:34.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.105904+0000", 2024-04-23T16:46:34.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001430204 2024-04-23T16:46:34.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:34.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.105956+0000", 2024-04-23T16:46:34.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1347999999999997e-05 2024-04-23T16:46:34.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:34.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.105993+0000", 2024-04-23T16:46:34.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7483e-05 2024-04-23T16:46:34.012 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.012 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:34.012 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.012 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.26584.0:34131 7.4 7:256b8a92:::benchmark_data_smithi188_141483_object34130:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e549)", 2024-04-23T16:46:34.012 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:42:19.011523+0000", 2024-04-23T16:46:34.012 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 254.74862819099999, 2024-04-23T16:46:34.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.096114388999999995, 2024-04-23T16:46:34.012 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:34.012 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:34.012 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:34.012 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:34.013 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.26584", 2024-04-23T16:46:34.013 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:34.013 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 34131 2024-04-23T16:46:34.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.013 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:34.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.013 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:34.013 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.011523+0000", 2024-04-23T16:46:34.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.013 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:34.013 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.011523+0000", 2024-04-23T16:46:34.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.014 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.014 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:34.014 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.011524+0000", 2024-04-23T16:46:34.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6000000000000003e-07 2024-04-23T16:46:34.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.014 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.014 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:34.014 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.011691+0000", 2024-04-23T16:46:34.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016676899999999999 2024-04-23T16:46:34.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.014 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.014 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:34.014 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.011703+0000", 2024-04-23T16:46:34.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2150000000000001e-05 2024-04-23T16:46:34.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.015 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:34.015 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.011708+0000", 2024-04-23T16:46:34.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1739999999999999e-06 2024-04-23T16:46:34.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.015 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:34.015 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.101411+0000", 2024-04-23T16:46:34.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.089702760000000006 2024-04-23T16:46:34.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.015 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:34.015 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.101544+0000", 2024-04-23T16:46:34.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000133208 2024-04-23T16:46:34.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:46:34.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.101688+0000", 2024-04-23T16:46:34.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014382699999999999 2024-04-23T16:46:34.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:34.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.105863+0000", 2024-04-23T16:46:34.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041748209999999996 2024-04-23T16:46:34.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:34.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.107540+0000", 2024-04-23T16:46:34.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001676711 2024-04-23T16:46:34.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:34.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.107600+0000", 2024-04-23T16:46:34.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0584999999999997e-05 2024-04-23T16:46:34.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:34.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.107638+0000", 2024-04-23T16:46:34.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7623999999999998e-05 2024-04-23T16:46:34.017 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.017 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:34.017 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.018 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.26584.0:34132 7.9 7:9ff97874:::benchmark_data_smithi188_141483_object34131:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e549)", 2024-04-23T16:46:34.018 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:42:19.011722+0000", 2024-04-23T16:46:34.018 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 254.748429415, 2024-04-23T16:46:34.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.092557667999999996, 2024-04-23T16:46:34.018 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:34.018 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:34.018 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:34.018 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:34.018 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.26584", 2024-04-23T16:46:34.018 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:34.018 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 34132 2024-04-23T16:46:34.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.019 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:34.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.019 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:34.019 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.011722+0000", 2024-04-23T16:46:34.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.019 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:34.019 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.011722+0000", 2024-04-23T16:46:34.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.019 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:34.019 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.011723+0000", 2024-04-23T16:46:34.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.85e-07 2024-04-23T16:46:34.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:34.020 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.011829+0000", 2024-04-23T16:46:34.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000106295 2024-04-23T16:46:34.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:34.020 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.011838+0000", 2024-04-23T16:46:34.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8629999999999997e-06 2024-04-23T16:46:34.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:34.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.011845+0000", 2024-04-23T16:46:34.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9650000000000002e-06 2024-04-23T16:46:34.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:34.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.012380+0000", 2024-04-23T16:46:34.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053524300000000003 2024-04-23T16:46:34.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:34.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.099489+0000", 2024-04-23T16:46:34.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.087108463999999997 2024-04-23T16:46:34.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 11", 2024-04-23T16:46:34.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.099613+0000", 2024-04-23T16:46:34.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000123876 2024-04-23T16:46:34.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:34.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.102869+0000", 2024-04-23T16:46:34.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0032559709999999999 2024-04-23T16:46:34.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:34.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.104174+0000", 2024-04-23T16:46:34.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013052070000000001 2024-04-23T16:46:34.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:34.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.104234+0000", 2024-04-23T16:46:34.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9935000000000001e-05 2024-04-23T16:46:34.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:34.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.104280+0000", 2024-04-23T16:46:34.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6063999999999998e-05 2024-04-23T16:46:34.023 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.023 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:34.023 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.024 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.26584.0:34135 7.6 7:677d6aeb:::benchmark_data_smithi188_141483_object34134:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e549)", 2024-04-23T16:46:34.024 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:42:19.012177+0000", 2024-04-23T16:46:34.024 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 254.747974067, 2024-04-23T16:46:34.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.093310884999999996, 2024-04-23T16:46:34.024 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:34.024 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:34.024 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:34.024 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:34.024 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.26584", 2024-04-23T16:46:34.024 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:34.024 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 34135 2024-04-23T16:46:34.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.024 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:34.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.024 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:34.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.012177+0000", 2024-04-23T16:46:34.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:34.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.012177+0000", 2024-04-23T16:46:34.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:34.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.012178+0000", 2024-04-23T16:46:34.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1299999999999998e-07 2024-04-23T16:46:34.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:34.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.012273+0000", 2024-04-23T16:46:34.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4221000000000005e-05 2024-04-23T16:46:34.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:34.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.012279+0000", 2024-04-23T16:46:34.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8709999999999999e-06 2024-04-23T16:46:34.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:34.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.012284+0000", 2024-04-23T16:46:34.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4270000000000001e-06 2024-04-23T16:46:34.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:34.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.012344+0000", 2024-04-23T16:46:34.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9908000000000003e-05 2024-04-23T16:46:34.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:34.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.012480+0000", 2024-04-23T16:46:34.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000135959 2024-04-23T16:46:34.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:46:34.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.099605+0000", 2024-04-23T16:46:34.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.087125196000000002 2024-04-23T16:46:34.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:34.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.104019+0000", 2024-04-23T16:46:34.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044140539999999997 2024-04-23T16:46:34.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:34.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.105421+0000", 2024-04-23T16:46:34.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0014024230000000001 2024-04-23T16:46:34.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:34.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.105459+0000", 2024-04-23T16:46:34.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7499000000000001e-05 2024-04-23T16:46:34.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:34.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.105488+0000", 2024-04-23T16:46:34.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9414e-05 2024-04-23T16:46:34.029 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.029 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:34.029 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.029 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.26584.0:34142 7.6 7:6d54edff:::benchmark_data_smithi188_141483_object34141:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e549)", 2024-04-23T16:46:34.029 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:42:19.014588+0000", 2024-04-23T16:46:34.029 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 254.74556319600001, 2024-04-23T16:46:34.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.093663115000000005, 2024-04-23T16:46:34.030 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:34.030 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:34.030 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:34.030 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:34.030 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.26584", 2024-04-23T16:46:34.030 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:34.030 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 34142 2024-04-23T16:46:34.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.030 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:34.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:34.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.014588+0000", 2024-04-23T16:46:34.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:34.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.014588+0000", 2024-04-23T16:46:34.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:34.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.014589+0000", 2024-04-23T16:46:34.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9199999999999999e-07 2024-04-23T16:46:34.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:34.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.014675+0000", 2024-04-23T16:46:34.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6046999999999998e-05 2024-04-23T16:46:34.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:34.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.099507+0000", 2024-04-23T16:46:34.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.084831163000000001 2024-04-23T16:46:34.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:34.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.099517+0000", 2024-04-23T16:46:34.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0394000000000001e-05 2024-04-23T16:46:34.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:34.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.101037+0000", 2024-04-23T16:46:34.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015205240000000001 2024-04-23T16:46:34.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:34.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.101158+0000", 2024-04-23T16:46:34.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000120325 2024-04-23T16:46:34.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:46:34.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.101315+0000", 2024-04-23T16:46:34.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000156754 2024-04-23T16:46:34.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:34.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.104022+0000", 2024-04-23T16:46:34.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002707877 2024-04-23T16:46:34.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:34.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.108142+0000", 2024-04-23T16:46:34.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041193260000000004 2024-04-23T16:46:34.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:34.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.108206+0000", 2024-04-23T16:46:34.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4128999999999995e-05 2024-04-23T16:46:34.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:34.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:42:19.108251+0000", 2024-04-23T16:46:34.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5584000000000003e-05 2024-04-23T16:46:34.035 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.035 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:34.035 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.035 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.26584.0:130132 7.6 7:6a9c3344:::benchmark_data_smithi188_141483_object11451:head [delete] snapc 0=[] ondisk+write+known_if_redirected e577)", 2024-04-23T16:46:34.035 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:43:55.965827+0000", 2024-04-23T16:46:34.035 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 157.79432486600001, 2024-04-23T16:46:34.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.075590599999999994, 2024-04-23T16:46:34.035 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:34.036 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:34.036 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:34.036 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:34.036 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.26584", 2024-04-23T16:46:34.036 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:34.036 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 130132 2024-04-23T16:46:34.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.036 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:34.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.036 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:34.036 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.965827+0000", 2024-04-23T16:46:34.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:34.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.965827+0000", 2024-04-23T16:46:34.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:34.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.965828+0000", 2024-04-23T16:46:34.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.094e-06 2024-04-23T16:46:34.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:34.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.965832+0000", 2024-04-23T16:46:34.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2089999999999999e-06 2024-04-23T16:46:34.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:34.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.965839+0000", 2024-04-23T16:46:34.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4309999999999997e-06 2024-04-23T16:46:34.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:34.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.965847+0000", 2024-04-23T16:46:34.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9039999999999992e-06 2024-04-23T16:46:34.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:34.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.965905+0000", 2024-04-23T16:46:34.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7368999999999999e-05 2024-04-23T16:46:34.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:34.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.966025+0000", 2024-04-23T16:46:34.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000120671 2024-04-23T16:46:34.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:46:34.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.966095+0000", 2024-04-23T16:46:34.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0100999999999998e-05 2024-04-23T16:46:34.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:34.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.971376+0000", 2024-04-23T16:46:34.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052804890000000002 2024-04-23T16:46:34.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:34.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:56.041349+0000", 2024-04-23T16:46:34.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.069972732999999995 2024-04-23T16:46:34.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:34.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:56.041383+0000", 2024-04-23T16:46:34.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4314999999999999e-05 2024-04-23T16:46:34.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:34.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:56.041417+0000", 2024-04-23T16:46:34.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4283999999999999e-05 2024-04-23T16:46:34.041 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.041 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:34.041 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.041 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.26584.0:130134 7.6 7:6fb62f3d:::benchmark_data_smithi188_141483_object11453:head [delete] snapc 0=[] ondisk+write+known_if_redirected e577)", 2024-04-23T16:46:34.041 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:43:55.966518+0000", 2024-04-23T16:46:34.041 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 157.79363305800001, 2024-04-23T16:46:34.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.074989185999999999, 2024-04-23T16:46:34.041 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:34.041 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:34.041 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:34.042 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:34.042 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.26584", 2024-04-23T16:46:34.042 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:34.042 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 130134 2024-04-23T16:46:34.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.042 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:34.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.042 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:34.042 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.966518+0000", 2024-04-23T16:46:34.042 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.042 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:34.042 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.966518+0000", 2024-04-23T16:46:34.042 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:34.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.966519+0000", 2024-04-23T16:46:34.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2399999999999996e-07 2024-04-23T16:46:34.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:34.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.966523+0000", 2024-04-23T16:46:34.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3249999999999999e-06 2024-04-23T16:46:34.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:34.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.966530+0000", 2024-04-23T16:46:34.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6629999999999993e-06 2024-04-23T16:46:34.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.044 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:34.044 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.966535+0000", 2024-04-23T16:46:34.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7600000000000002e-06 2024-04-23T16:46:34.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.044 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:34.044 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.968992+0000", 2024-04-23T16:46:34.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002456555 2024-04-23T16:46:34.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.044 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:34.044 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.969056+0000", 2024-04-23T16:46:34.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4559000000000003e-05 2024-04-23T16:46:34.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:46:34.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.969115+0000", 2024-04-23T16:46:34.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8511000000000003e-05 2024-04-23T16:46:34.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:34.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.973090+0000", 2024-04-23T16:46:34.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039756319999999998 2024-04-23T16:46:34.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:34.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:56.041445+0000", 2024-04-23T16:46:34.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.068355071000000003 2024-04-23T16:46:34.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:34.046 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:56.041484+0000", 2024-04-23T16:46:34.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8557999999999999e-05 2024-04-23T16:46:34.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:34.046 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:56.041508+0000", 2024-04-23T16:46:34.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3628000000000001e-05 2024-04-23T16:46:34.046 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.046 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:34.047 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.047 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.26584.0:130136 7.6 7:63cc51da:::benchmark_data_smithi188_141483_object11455:head [delete] snapc 0=[] ondisk+write+known_if_redirected e577)", 2024-04-23T16:46:34.047 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:43:55.967782+0000", 2024-04-23T16:46:34.047 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 157.79236977100001, 2024-04-23T16:46:34.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.082161224000000005, 2024-04-23T16:46:34.047 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:34.047 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:34.047 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:34.047 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:34.047 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.26584", 2024-04-23T16:46:34.047 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:34.047 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 130136 2024-04-23T16:46:34.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.048 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:34.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:34.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.967782+0000", 2024-04-23T16:46:34.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:34.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.967782+0000", 2024-04-23T16:46:34.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:34.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.967783+0000", 2024-04-23T16:46:34.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6099999999999999e-07 2024-04-23T16:46:34.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:34.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.967786+0000", 2024-04-23T16:46:34.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1099999999999999e-06 2024-04-23T16:46:34.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:34.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.967792+0000", 2024-04-23T16:46:34.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3230000000000003e-06 2024-04-23T16:46:34.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:34.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.967796+0000", 2024-04-23T16:46:34.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.861e-06 2024-04-23T16:46:34.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.050 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:34.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.971394+0000", 2024-04-23T16:46:34.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003598572 2024-04-23T16:46:34.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.050 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:34.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.971470+0000", 2024-04-23T16:46:34.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5036000000000005e-05 2024-04-23T16:46:34.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:46:34.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.971542+0000", 2024-04-23T16:46:34.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2667000000000001e-05 2024-04-23T16:46:34.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:34.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.973199+0000", 2024-04-23T16:46:34.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001656341 2024-04-23T16:46:34.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:34.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:56.049822+0000", 2024-04-23T16:46:34.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.076623298000000006 2024-04-23T16:46:34.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.052 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:34.052 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:56.049878+0000", 2024-04-23T16:46:34.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6215999999999998e-05 2024-04-23T16:46:34.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.052 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:34.052 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:56.049943+0000", 2024-04-23T16:46:34.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4938999999999996e-05 2024-04-23T16:46:34.052 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.052 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:34.052 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.053 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.26584.0:130142 7.4 7:206f0e3d:::benchmark_data_smithi188_141483_object11461:head [delete] snapc 0=[] ondisk+write+known_if_redirected e577)", 2024-04-23T16:46:34.053 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:43:55.983668+0000", 2024-04-23T16:46:34.053 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 157.776483985, 2024-04-23T16:46:34.053 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.070967585999999999, 2024-04-23T16:46:34.053 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:34.053 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:34.053 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:34.053 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:34.053 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.26584", 2024-04-23T16:46:34.053 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:34.053 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 130142 2024-04-23T16:46:34.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.053 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:34.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:34.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.983668+0000", 2024-04-23T16:46:34.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:34.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.983668+0000", 2024-04-23T16:46:34.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:34.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.983671+0000", 2024-04-23T16:46:34.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0529999999999998e-06 2024-04-23T16:46:34.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:34.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.983681+0000", 2024-04-23T16:46:34.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0463e-05 2024-04-23T16:46:34.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:34.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.983706+0000", 2024-04-23T16:46:34.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4803999999999998e-05 2024-04-23T16:46:34.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:34.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.983722+0000", 2024-04-23T16:46:34.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5869e-05 2024-04-23T16:46:34.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:34.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.983823+0000", 2024-04-23T16:46:34.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100851 2024-04-23T16:46:34.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:34.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.983898+0000", 2024-04-23T16:46:34.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5572000000000001e-05 2024-04-23T16:46:34.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:46:34.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.983970+0000", 2024-04-23T16:46:34.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1694999999999997e-05 2024-04-23T16:46:34.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:34.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:55.987071+0000", 2024-04-23T16:46:34.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0031010769999999998 2024-04-23T16:46:34.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:34.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:56.054546+0000", 2024-04-23T16:46:34.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.067474749000000001 2024-04-23T16:46:34.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:34.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:56.054579+0000", 2024-04-23T16:46:34.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3507999999999998e-05 2024-04-23T16:46:34.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:34.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:56.054635+0000", 2024-04-23T16:46:34.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5945000000000001e-05 2024-04-23T16:46:34.058 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.058 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:34.058 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.058 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.26584.0:141174 7.6 7:623fd8d8:::benchmark_data_smithi188_141483_object22493:head [delete] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-04-23T16:46:34.059 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:44:09.430264+0000", 2024-04-23T16:46:34.059 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 144.32988790799999, 2024-04-23T16:46:34.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.072959343999999995, 2024-04-23T16:46:34.059 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:34.059 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:34.059 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:34.059 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:34.059 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.26584", 2024-04-23T16:46:34.059 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:34.059 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141174 2024-04-23T16:46:34.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.059 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:34.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:34.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.430264+0000", 2024-04-23T16:46:34.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:34.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.430264+0000", 2024-04-23T16:46:34.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:34.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.430265+0000", 2024-04-23T16:46:34.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4279999999999999e-06 2024-04-23T16:46:34.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:34.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.430269+0000", 2024-04-23T16:46:34.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9720000000000003e-06 2024-04-23T16:46:34.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:34.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.430278+0000", 2024-04-23T16:46:34.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8219999999999996e-06 2024-04-23T16:46:34.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:34.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.430283+0000", 2024-04-23T16:46:34.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4060000000000004e-06 2024-04-23T16:46:34.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:34.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.430328+0000", 2024-04-23T16:46:34.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4381000000000001e-05 2024-04-23T16:46:34.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:34.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.430445+0000", 2024-04-23T16:46:34.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117457 2024-04-23T16:46:34.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:46:34.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.430560+0000", 2024-04-23T16:46:34.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114522 2024-04-23T16:46:34.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:34.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.434470+0000", 2024-04-23T16:46:34.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039101279999999997 2024-04-23T16:46:34.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:34.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.503150+0000", 2024-04-23T16:46:34.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.068680419000000006 2024-04-23T16:46:34.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:34.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.503188+0000", 2024-04-23T16:46:34.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8056999999999998e-05 2024-04-23T16:46:34.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:34.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.503223+0000", 2024-04-23T16:46:34.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4752000000000002e-05 2024-04-23T16:46:34.064 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.064 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:34.064 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.064 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.26584.0:141178 7.4 7:26882c4a:::benchmark_data_smithi188_141483_object22497:head [delete] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-04-23T16:46:34.064 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:44:09.435275+0000", 2024-04-23T16:46:34.065 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 144.32487680599999, 2024-04-23T16:46:34.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.076081276000000003, 2024-04-23T16:46:34.065 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:34.065 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:34.065 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:34.065 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:34.065 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.26584", 2024-04-23T16:46:34.065 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:34.065 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141178 2024-04-23T16:46:34.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.065 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:34.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.065 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:34.065 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.435275+0000", 2024-04-23T16:46:34.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:34.066 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.435275+0000", 2024-04-23T16:46:34.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:34.066 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.435276+0000", 2024-04-23T16:46:34.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.325e-06 2024-04-23T16:46:34.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:34.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.435281+0000", 2024-04-23T16:46:34.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0780000000000003e-06 2024-04-23T16:46:34.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:34.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.435291+0000", 2024-04-23T16:46:34.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8160000000000005e-06 2024-04-23T16:46:34.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:34.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.435297+0000", 2024-04-23T16:46:34.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1859999999999997e-06 2024-04-23T16:46:34.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:34.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.435347+0000", 2024-04-23T16:46:34.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9907000000000002e-05 2024-04-23T16:46:34.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:34.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.435471+0000", 2024-04-23T16:46:34.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012414500000000001 2024-04-23T16:46:34.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:46:34.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.435579+0000", 2024-04-23T16:46:34.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010736899999999999 2024-04-23T16:46:34.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:34.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.439030+0000", 2024-04-23T16:46:34.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034510890000000001 2024-04-23T16:46:34.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:34.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.511246+0000", 2024-04-23T16:46:34.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.072216066999999995 2024-04-23T16:46:34.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:34.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.511305+0000", 2024-04-23T16:46:34.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8959000000000003e-05 2024-04-23T16:46:34.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:34.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:44:09.511356+0000", 2024-04-23T16:46:34.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1335000000000002e-05 2024-04-23T16:46:34.070 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.070 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:34.070 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.070 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.26584.0:212424 7.f 7:fc43dd98:::benchmark_data_smithi188_141483_object93743:head [delete] snapc 0=[] ondisk+write+known_if_redirected e595)", 2024-04-23T16:46:34.070 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:45:35.628446+0000", 2024-04-23T16:46:34.070 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 58.131705293000003, 2024-04-23T16:46:34.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.078521675999999999, 2024-04-23T16:46:34.071 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:34.071 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:34.071 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:34.071 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:34.071 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.26584", 2024-04-23T16:46:34.071 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:34.071 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 212424 2024-04-23T16:46:34.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.071 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:34.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:34.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.628446+0000", 2024-04-23T16:46:34.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:34.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.628446+0000", 2024-04-23T16:46:34.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:34.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.628448+0000", 2024-04-23T16:46:34.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7570000000000001e-06 2024-04-23T16:46:34.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:34.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.628480+0000", 2024-04-23T16:46:34.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1798999999999998e-05 2024-04-23T16:46:34.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:34.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.628488+0000", 2024-04-23T16:46:34.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9349999999999994e-06 2024-04-23T16:46:34.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:34.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.628492+0000", 2024-04-23T16:46:34.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8809999999999998e-06 2024-04-23T16:46:34.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:34.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.678207+0000", 2024-04-23T16:46:34.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.049715257999999998 2024-04-23T16:46:34.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:34.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.678369+0000", 2024-04-23T16:46:34.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016261100000000001 2024-04-23T16:46:34.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 8", 2024-04-23T16:46:34.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.678496+0000", 2024-04-23T16:46:34.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012635000000000001 2024-04-23T16:46:34.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:34.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.691328+0000", 2024-04-23T16:46:34.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012832005 2024-04-23T16:46:34.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:34.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.691341+0000", 2024-04-23T16:46:34.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3488e-05 2024-04-23T16:46:34.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:34.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.691386+0000", 2024-04-23T16:46:34.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4438999999999999e-05 2024-04-23T16:46:34.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:34.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.706968+0000", 2024-04-23T16:46:34.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015582153 2024-04-23T16:46:34.076 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.076 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:34.076 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.076 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.26584.0:212429 7.4 7:2441f206:::benchmark_data_smithi188_141483_object93748:head [delete] snapc 0=[] ondisk+write+known_if_redirected e595)", 2024-04-23T16:46:34.076 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:45:35.633007+0000", 2024-04-23T16:46:34.076 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 58.127144850999997, 2024-04-23T16:46:34.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.074073583999999998, 2024-04-23T16:46:34.076 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:46:34.076 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:46:34.077 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:34.077 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:46:34.077 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.26584", 2024-04-23T16:46:34.077 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:34.077 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 212429 2024-04-23T16:46:34.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.077 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:46:34.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:46:34.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.633007+0000", 2024-04-23T16:46:34.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:46:34.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.633007+0000", 2024-04-23T16:46:34.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:46:34.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:46:34.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.633009+0000", 2024-04-23T16:46:34.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9470000000000002e-06 2024-04-23T16:46:34.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:46:34.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.633014+0000", 2024-04-23T16:46:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8649999999999998e-06 2024-04-23T16:46:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:46:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.678087+0000", 2024-04-23T16:46:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045072291 2024-04-23T16:46:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:46:34.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.678102+0000", 2024-04-23T16:46:34.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5241e-05 2024-04-23T16:46:34.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:46:34.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.683118+0000", 2024-04-23T16:46:34.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050159239999999997 2024-04-23T16:46:34.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:46:34.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.683232+0000", 2024-04-23T16:46:34.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000113998 2024-04-23T16:46:34.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:46:34.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.683335+0000", 2024-04-23T16:46:34.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103076 2024-04-23T16:46:34.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:34.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.687813+0000", 2024-04-23T16:46:34.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004477516 2024-04-23T16:46:34.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:46:34.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.706993+0000", 2024-04-23T16:46:34.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019180675000000001 2024-04-23T16:46:34.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:46:34.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.707039+0000", 2024-04-23T16:46:34.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6279000000000002e-05 2024-04-23T16:46:34.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:46:34.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:46:34.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:46:34.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:45:35.707080+0000", 2024-04-23T16:46:34.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0772000000000002e-05 2024-04-23T16:46:34.082 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.082 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:34.082 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.082 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:46:34.082 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:46:34.083 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:46:34.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:34.084 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:46:34.084 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:46:34.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:33 smithi103 ceph-mon[39629]: pgmap v1720: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:33 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2355642541' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:46:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:33 smithi103 ceph-mon[39629]: osdmap e612: 12 total, 11 up, 9 in 2024-04-23T16:46:34.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:33 smithi148 ceph-mon[44520]: pgmap v1720: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:34.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:33 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2355642541' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:46:34.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:33 smithi148 ceph-mon[44520]: osdmap e612: 12 total, 11 up, 9 in 2024-04-23T16:46:34.184 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:46:34.215 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:34.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:33 smithi188 ceph-mon[44101]: pgmap v1720: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:34.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:33 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2355642541' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:46:34.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:33 smithi188 ceph-mon[44101]: osdmap e612: 12 total, 11 up, 9 in 2024-04-23T16:46:34.234 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:46:34.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:34.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:46:34.343 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:46:34.376 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:34.398 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:46:34.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:34.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:46:34.502 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:46:34.529 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:34.678 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:46:34.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:34.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:46:34.690 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-23T16:46:34.782 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:46:34.810 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:34.963 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:46:35.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:35 smithi103 ceph-mon[39629]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:46:35.264 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:35 smithi148 ceph-mon[44520]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:46:35.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:35 smithi188 ceph-mon[44101]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:46:36.038 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:36.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:36.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:36.142 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:46:36.158 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:46:36.158 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:46:36.158 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:46:36.158 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:46:36.158 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:46:36.158 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:46:36.159 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:46:36.159 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:46:36.159 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:46:36.159 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:46:36.159 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:46:36.159 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:46:36.159 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:46:36.159 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:46:36.159 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:46:36.159 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:46:36.159 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:46:36.160 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:46:36.160 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:46:36.160 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:46:36.160 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:46:36.160 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:46:36.160 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:46:36.169 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:36.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:36 smithi103 ceph-mon[39629]: pgmap v1722: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/68352935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:46:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:36 smithi148 ceph-mon[44520]: pgmap v1722: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/68352935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:46:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:36.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:36 smithi188 ceph-mon[44101]: pgmap v1722: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/68352935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:46:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:36.786 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:46:36.787 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:46:37.019 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:37.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:37.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:37.026 DEBUG:teuthology.orchestra.run.smithi188:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-04-23T16:46:37.029 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-23T16:46:37.029 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-23T16:46:37.039 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:46:37.039 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:46:37.123 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:46:37.140 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:hints = 1 2024-04-23T16:46:37.140 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-23T16:46:37.141 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Object prefix: benchmark_data_smithi188_155546 2024-04-23T16:46:37.141 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:46:37.141 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 0 0 0 0 0 0 - 0 2024-04-23T16:46:37.155 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:37.258 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:37.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:37.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:37 smithi103 ceph-mon[39629]: 1.8 scrub starts 2024-04-23T16:46:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:37 smithi103 ceph-mon[39629]: 1.8 scrub ok 2024-04-23T16:46:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.362 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:46:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:37 smithi148 ceph-mon[44520]: 1.8 scrub starts 2024-04-23T16:46:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:37 smithi148 ceph-mon[44520]: 1.8 scrub ok 2024-04-23T16:46:37.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.409 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:37.440 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:46:37.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:37.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:46:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:37 smithi188 ceph-mon[44101]: 1.8 scrub starts 2024-04-23T16:46:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:37 smithi188 ceph-mon[44101]: 1.8 scrub ok 2024-04-23T16:46:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:37.548 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:46:37.583 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:37.741 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:46:37.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:37.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:46:37.777 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-23T16:46:37.849 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:46:37.901 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:37.927 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:46:37.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:37.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:46:38.016 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:46:38.032 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:46:38.068 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:38.093 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:46:38.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:38.096 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:46:38.141 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 1 16 5327 5311 331.919 331.938 0.00341022 0.00300019 2024-04-23T16:46:38.187 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:46:38.197 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:46:38.240 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:38.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:38 smithi103 ceph-mon[39629]: pgmap v1723: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:38 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4196479305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:46:38.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:38 smithi148 ceph-mon[44520]: pgmap v1723: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:38 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4196479305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:46:38.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:38 smithi188 ceph-mon[44101]: pgmap v1723: 27 pgs: 27 active+clean; 577 KiB data, 4.8 GiB used, 800 GiB / 805 GiB avail 2024-04-23T16:46:38.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:38 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4196479305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:46:38.496 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:46:38.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:38.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:46:38.601 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:46:38.638 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:38.788 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:46:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:39 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:46:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:39 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:46:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:39 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4196479305' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:46:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:39 smithi148 ceph-mon[44520]: osdmap e613: 12 total, 11 up, 9 in 2024-04-23T16:46:39.142 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 2 16 10105 10089 315.059 298.625 0.00563377 0.00317101 2024-04-23T16:46:39.198 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:46:39.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:39.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:46:39.303 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:46:39.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:39 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:46:39.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:39 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:46:39.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:39 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4196479305' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:46:39.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:39 smithi103 ceph-mon[39629]: osdmap e613: 12 total, 11 up, 9 in 2024-04-23T16:46:39.335 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:39.361 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:39.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:39.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:39.466 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:46:39.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:39 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:46:39.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:39 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:46:39.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:39 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4196479305' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:46:39.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:39 smithi188 ceph-mon[44101]: osdmap e613: 12 total, 11 up, 9 in 2024-04-23T16:46:39.504 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:39.763 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:39.860 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:46:39.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:39.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:46:39.964 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:46:39.995 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:40.020 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:46:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:40.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:46:40.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:40 smithi148 ceph-mon[44520]: pgmap v1725: 27 pgs: 27 active+clean; 125 MiB data, 5.0 GiB used, 800 GiB / 805 GiB avail; 16 MiB/s wr, 249 op/s 2024-04-23T16:46:40.125 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:46:40.142 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 3 16 14567 14551 302.991 278.875 0.00395801 0.00329818 2024-04-23T16:46:40.159 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:40.272 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:40 smithi103 ceph-mon[39629]: pgmap v1725: 27 pgs: 27 active+clean; 125 MiB data, 5.0 GiB used, 800 GiB / 805 GiB avail; 16 MiB/s wr, 249 op/s 2024-04-23T16:46:40.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:40 smithi188 ceph-mon[44101]: pgmap v1725: 27 pgs: 27 active+clean; 125 MiB data, 5.0 GiB used, 800 GiB / 805 GiB avail; 16 MiB/s wr, 249 op/s 2024-04-23T16:46:41.142 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 4 16 18177 18161 283.651 225.625 0.00387955 0.00352265 2024-04-23T16:46:41.415 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:46:41.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:41.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:46:41.522 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:41.563 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:41.650 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:41.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:41.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:41.741 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:46:41.741 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:46:41.752 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:46:41.756 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:46:41.781 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:46:41.792 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:41.810 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:46:41.811 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:46:41.821 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:46:41.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:41.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:46:41.838 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:46:41.865 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:46:41.891 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:46:41.919 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:46:41.926 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:41.948 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:46:41.963 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:41.976 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:46:41.993 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:41.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:41.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:42.005 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:46:42.033 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:46:42.034 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:46:42.034 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:46:42.034 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:46:42.034 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:46:42.034 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:46:42.034 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:46:42.034 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:46:42.034 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:46:42.034 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:46:42.034 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:46:42.035 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:46:42.035 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:46:42.048 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:42 smithi103 ceph-mon[39629]: pgmap v1726: 27 pgs: 27 active+clean; 267 MiB data, 5.4 GiB used, 799 GiB / 805 GiB avail; 33 MiB/s wr, 532 op/s 2024-04-23T16:46:42.048 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:42 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1803301641' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:46:42.048 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:42.048 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:42.048 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:42.099 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:46:42.141 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:42.142 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 5 16 21918 21902 273.682 233.812 0.00431477 0.00364911 2024-04-23T16:46:42.170 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:46:42.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:42.173 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:46:42.274 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:46:42.308 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:42 smithi148 ceph-mon[44520]: pgmap v1726: 27 pgs: 27 active+clean; 267 MiB data, 5.4 GiB used, 799 GiB / 805 GiB avail; 33 MiB/s wr, 532 op/s 2024-04-23T16:46:42.308 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:42 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1803301641' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:46:42.308 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:42.309 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:42.309 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:42.313 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:42.471 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:46:42.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:42 smithi188 ceph-mon[44101]: pgmap v1726: 27 pgs: 27 active+clean; 267 MiB data, 5.4 GiB used, 799 GiB / 805 GiB avail; 33 MiB/s wr, 532 op/s 2024-04-23T16:46:42.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:42 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1803301641' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:46:42.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:42.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:42.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:42.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:42.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:46:42.506 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-23T16:46:42.575 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:46:42.637 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:42.662 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:46:42.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:42.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:46:42.767 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:46:42.802 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:42.921 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:46:43.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.143 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 6 16 25394 25378 264.274 217.25 0.00136501 0.00376302 2024-04-23T16:46:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:44.043 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:44 smithi103 ceph-mon[39629]: pgmap v1727: 27 pgs: 27 active+clean; 1.0 GiB data, 6.4 GiB used, 798 GiB / 805 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-04-23T16:46:44.142 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 7 16 28760 28744 256.573 210.375 0.00454433 0.00389493 2024-04-23T16:46:44.257 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:46:44.258 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:44.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:46:44.300 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:46:44.301 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:46:44.362 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:46:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:44 smithi148 ceph-mon[44520]: pgmap v1727: 27 pgs: 27 active+clean; 1.0 GiB data, 6.4 GiB used, 798 GiB / 805 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-04-23T16:46:44.400 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:44.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:44 smithi188 ceph-mon[44101]: pgmap v1727: 27 pgs: 27 active+clean; 1.0 GiB data, 6.4 GiB used, 798 GiB / 805 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-04-23T16:46:44.754 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:46:44.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:44.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:46:44.859 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:46:44.906 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:44.931 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:46:44.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:44.935 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:46:45.036 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:46:45.069 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:45.126 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:46:45.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:45.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:46:45.143 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 8 16 31892 31876 248.968 195.75 0.00139397 0.00400636 2024-04-23T16:46:45.230 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:45.264 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:45.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:45 smithi148 ceph-mon[44520]: pgmap v1728: 27 pgs: 27 active+clean; 1.4 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-04-23T16:46:45.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:45 smithi188 ceph-mon[44101]: pgmap v1728: 27 pgs: 27 active+clean; 1.4 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-04-23T16:46:45.728 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:45 smithi103 ceph-mon[39629]: pgmap v1728: 27 pgs: 27 active+clean; 1.4 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-04-23T16:46:46.143 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 9 16 34904 34888 242.22 188.25 0.0102403 0.00412611 2024-04-23T16:46:46.256 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:46.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:46.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:46.301 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:46:46.361 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:46:46.399 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:46.426 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:46:46.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:46.431 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:46:46.531 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:46.572 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:46.663 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:46.686 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:46:46.689 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:46 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/293532759' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:46:46.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:46 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/293532759' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:46:46.927 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:46.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:46.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:47.034 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:46:47.036 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:46:47.036 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2024-04-23T16:46:47.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2024-04-23T16:46:47.037 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:46:47.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.037 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:173387 6.e 6:7cc917ba:::benchmark_data_smithi188_130221_object18926:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e489)", 2024-04-23T16:46:47.037 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:39:21.935652+0000", 2024-04-23T16:46:47.037 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 445.09640637799998, 2024-04-23T16:46:47.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20711752999999999, 2024-04-23T16:46:47.037 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.037 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.037 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.037 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.037 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:46:47.038 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:46:47.038 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 173387 2024-04-23T16:46:47.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.038 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935652+0000", 2024-04-23T16:46:47.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935652+0000", 2024-04-23T16:46:47.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935655+0000", 2024-04-23T16:46:47.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2060000000000001e-06 2024-04-23T16:46:47.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935660+0000", 2024-04-23T16:46:47.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7960000000000001e-06 2024-04-23T16:46:47.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935673+0000", 2024-04-23T16:46:47.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2741999999999999e-05 2024-04-23T16:46:47.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935683+0000", 2024-04-23T16:46:47.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0002e-05 2024-04-23T16:46:47.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935735+0000", 2024-04-23T16:46:47.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2249000000000001e-05 2024-04-23T16:46:47.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:46:47.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935745+0000", 2024-04-23T16:46:47.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.73e-06 2024-04-23T16:46:47.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.128395+0000", 2024-04-23T16:46:47.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19265043400000001 2024-04-23T16:46:47.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.128503+0000", 2024-04-23T16:46:47.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107508 2024-04-23T16:46:47.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 1,5", 2024-04-23T16:46:47.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.128598+0000", 2024-04-23T16:46:47.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.4862000000000001e-05 2024-04-23T16:46:47.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.043 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.132780+0000", 2024-04-23T16:46:47.043 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0041825990000000004 2024-04-23T16:46:47.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.043 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.044 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142388+0000", 2024-04-23T16:46:47.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0096078719999999999 2024-04-23T16:46:47.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.045 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142628+0000", 2024-04-23T16:46:47.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000239886 2024-04-23T16:46:47.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.045 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142692+0000", 2024-04-23T16:46:47.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3414e-05 2024-04-23T16:46:47.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.046 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142770+0000", 2024-04-23T16:46:47.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8230000000000001e-05 2024-04-23T16:46:47.046 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.046 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.046 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.046 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:173393 6.e 6:73e3fd60:::benchmark_data_smithi188_130221_object18932:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e489)", 2024-04-23T16:46:47.046 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:39:21.935705+0000", 2024-04-23T16:46:47.047 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 445.09635353700003, 2024-04-23T16:46:47.047 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20725584, 2024-04-23T16:46:47.047 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.047 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.047 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.047 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.047 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:46:47.047 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:46:47.047 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 173393 2024-04-23T16:46:47.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.047 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.048 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935705+0000", 2024-04-23T16:46:47.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.048 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.048 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.048 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935705+0000", 2024-04-23T16:46:47.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.048 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.048 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.048 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935706+0000", 2024-04-23T16:46:47.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6300000000000004e-07 2024-04-23T16:46:47.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.048 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.049 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.049 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935710+0000", 2024-04-23T16:46:47.049 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6270000000000002e-06 2024-04-23T16:46:47.049 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.049 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.049 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.049 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935716+0000", 2024-04-23T16:46:47.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6590000000000001e-06 2024-04-23T16:46:47.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.050 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935723+0000", 2024-04-23T16:46:47.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3829999999999997e-06 2024-04-23T16:46:47.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.050 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935789+0000", 2024-04-23T16:46:47.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6511000000000001e-05 2024-04-23T16:46:47.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:46:47.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935799+0000", 2024-04-23T16:46:47.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8560000000000002e-06 2024-04-23T16:46:47.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.052 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.052 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.132835+0000", 2024-04-23T16:46:47.053 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19703553500000001 2024-04-23T16:46:47.053 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.054 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.054 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.132984+0000", 2024-04-23T16:46:47.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000149319 2024-04-23T16:46:47.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.054 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 1,5", 2024-04-23T16:46:47.054 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.133139+0000", 2024-04-23T16:46:47.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015512800000000001 2024-04-23T16:46:47.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.054 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.054 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142778+0000", 2024-04-23T16:46:47.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.009639356 2024-04-23T16:46:47.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142816+0000", 2024-04-23T16:46:47.055 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7710999999999999e-05 2024-04-23T16:46:47.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142864+0000", 2024-04-23T16:46:47.055 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8038000000000003e-05 2024-04-23T16:46:47.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142895+0000", 2024-04-23T16:46:47.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1092e-05 2024-04-23T16:46:47.056 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.056 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.056 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.056 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142961+0000", 2024-04-23T16:46:47.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5761999999999998e-05 2024-04-23T16:46:47.056 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.056 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.056 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.056 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.056 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.056 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:173402 6.e 6:71c0b42e:::benchmark_data_smithi188_130221_object18941:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e489)", 2024-04-23T16:46:47.056 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:39:21.935731+0000", 2024-04-23T16:46:47.056 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 445.09632814399998, 2024-04-23T16:46:47.057 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.207581344, 2024-04-23T16:46:47.057 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.057 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.057 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.057 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.057 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:46:47.057 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:46:47.057 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 173402 2024-04-23T16:46:47.057 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.057 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.057 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.057 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.057 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935731+0000", 2024-04-23T16:46:47.057 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.058 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.058 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.058 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.058 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935731+0000", 2024-04-23T16:46:47.058 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.058 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.058 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.058 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.058 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935731+0000", 2024-04-23T16:46:47.058 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4600000000000005e-07 2024-04-23T16:46:47.058 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.058 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.058 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.058 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935734+0000", 2024-04-23T16:46:47.059 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7800000000000001e-06 2024-04-23T16:46:47.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.059 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.059 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.059 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935740+0000", 2024-04-23T16:46:47.059 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6840000000000001e-06 2024-04-23T16:46:47.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.059 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.059 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.059 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935746+0000", 2024-04-23T16:46:47.059 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0549999999999996e-06 2024-04-23T16:46:47.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.059 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.059 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.060 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935857+0000", 2024-04-23T16:46:47.060 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000111638 2024-04-23T16:46:47.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.060 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:46:47.060 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935867+0000", 2024-04-23T16:46:47.060 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8309999999999996e-06 2024-04-23T16:46:47.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.060 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.060 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.138168+0000", 2024-04-23T16:46:47.060 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20230051600000001 2024-04-23T16:46:47.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.061 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.061 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.138324+0000", 2024-04-23T16:46:47.061 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015627699999999999 2024-04-23T16:46:47.061 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.061 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.061 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 1,5", 2024-04-23T16:46:47.061 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.138498+0000", 2024-04-23T16:46:47.061 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000173616 2024-04-23T16:46:47.061 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.061 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.061 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.061 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.142970+0000", 2024-04-23T16:46:47.061 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0044722119999999997 2024-04-23T16:46:47.061 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.062 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.062 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.062 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.143014+0000", 2024-04-23T16:46:47.062 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3940000000000003e-05 2024-04-23T16:46:47.062 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.062 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.062 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.062 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.143159+0000", 2024-04-23T16:46:47.062 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014494700000000001 2024-04-23T16:46:47.062 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.062 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.062 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.062 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.143216+0000", 2024-04-23T16:46:47.062 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7194000000000002e-05 2024-04-23T16:46:47.063 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.063 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.063 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.063 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.143312+0000", 2024-04-23T16:46:47.063 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6107999999999995e-05 2024-04-23T16:46:47.063 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.064 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.064 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.064 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.064 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.064 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:173404 6.e 6:767f53bf:::benchmark_data_smithi188_130221_object18943:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e489)", 2024-04-23T16:46:47.064 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:39:21.935780+0000", 2024-04-23T16:46:47.064 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 445.096278235, 2024-04-23T16:46:47.064 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.22186477900000001, 2024-04-23T16:46:47.064 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.064 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.064 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.064 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.065 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:46:47.065 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:46:47.065 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 173404 2024-04-23T16:46:47.065 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.065 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.065 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.065 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.065 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935780+0000", 2024-04-23T16:46:47.065 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.065 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.065 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.065 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.065 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935780+0000", 2024-04-23T16:46:47.065 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.065 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.066 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.066 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.066 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935782+0000", 2024-04-23T16:46:47.066 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1599999999999999e-06 2024-04-23T16:46:47.066 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.066 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.066 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.066 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935786+0000", 2024-04-23T16:46:47.066 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7099999999999998e-06 2024-04-23T16:46:47.066 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.066 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.066 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.066 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935794+0000", 2024-04-23T16:46:47.066 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0460000000000006e-06 2024-04-23T16:46:47.067 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.067 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.067 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.067 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935800+0000", 2024-04-23T16:46:47.067 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1630000000000004e-06 2024-04-23T16:46:47.067 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.067 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.067 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.067 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935935+0000", 2024-04-23T16:46:47.067 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013570699999999999 2024-04-23T16:46:47.067 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.067 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.067 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:46:47.067 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935942+0000", 2024-04-23T16:46:47.068 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.866e-06 2024-04-23T16:46:47.068 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.068 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.068 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.068 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.143493+0000", 2024-04-23T16:46:47.068 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20755101000000001 2024-04-23T16:46:47.068 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.068 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.069 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.069 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.143625+0000", 2024-04-23T16:46:47.069 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013207200000000001 2024-04-23T16:46:47.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.069 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.069 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 1,5", 2024-04-23T16:46:47.069 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.143709+0000", 2024-04-23T16:46:47.069 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3735999999999999e-05 2024-04-23T16:46:47.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.069 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.069 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.070 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157306+0000", 2024-04-23T16:46:47.070 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013596924 2024-04-23T16:46:47.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.070 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.070 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.070 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157383+0000", 2024-04-23T16:46:47.070 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7142000000000001e-05 2024-04-23T16:46:47.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.070 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.070 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.070 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157471+0000", 2024-04-23T16:46:47.070 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8399000000000005e-05 2024-04-23T16:46:47.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.070 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.071 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.071 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157532+0000", 2024-04-23T16:46:47.071 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0378999999999999e-05 2024-04-23T16:46:47.071 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.071 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.071 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.071 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157645+0000", 2024-04-23T16:46:47.071 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000113465 2024-04-23T16:46:47.071 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.071 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.071 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.071 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.071 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.071 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16308.0:173424 6.e 6:721bacbd:::benchmark_data_smithi188_130221_object18963:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e489)", 2024-04-23T16:46:47.072 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:39:21.935810+0000", 2024-04-23T16:46:47.072 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 445.096248898, 2024-04-23T16:46:47.072 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.221983126, 2024-04-23T16:46:47.072 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.072 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.072 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.072 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.072 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16308", 2024-04-23T16:46:47.072 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:46:47.072 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 173424 2024-04-23T16:46:47.072 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.072 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.072 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.072 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.073 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935810+0000", 2024-04-23T16:46:47.073 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.073 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.073 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.073 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935810+0000", 2024-04-23T16:46:47.073 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.073 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.073 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.073 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935810+0000", 2024-04-23T16:46:47.073 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.68e-07 2024-04-23T16:46:47.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.073 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.074 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.074 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935813+0000", 2024-04-23T16:46:47.074 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.559e-06 2024-04-23T16:46:47.074 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.074 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.074 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.074 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935817+0000", 2024-04-23T16:46:47.074 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.822e-06 2024-04-23T16:46:47.074 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.074 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.074 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.074 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935822+0000", 2024-04-23T16:46:47.074 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2089999999999996e-06 2024-04-23T16:46:47.074 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935964+0000", 2024-04-23T16:46:47.075 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014171 2024-04-23T16:46:47.075 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:46:47.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:21.935970+0000", 2024-04-23T16:46:47.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6959999999999999e-06 2024-04-23T16:46:47.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.076 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.076 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.076 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.147821+0000", 2024-04-23T16:46:47.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.211850487 2024-04-23T16:46:47.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.076 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.076 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.076 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.147950+0000", 2024-04-23T16:46:47.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000129469 2024-04-23T16:46:47.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.076 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.076 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 1,5", 2024-04-23T16:46:47.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.148069+0000", 2024-04-23T16:46:47.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000118879 2024-04-23T16:46:47.077 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157455+0000", 2024-04-23T16:46:47.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00938604 2024-04-23T16:46:47.077 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157655+0000", 2024-04-23T16:46:47.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00019966600000000001 2024-04-23T16:46:47.077 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.078 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.078 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157690+0000", 2024-04-23T16:46:47.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4844e-05 2024-04-23T16:46:47.078 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.078 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.078 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.078 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157726+0000", 2024-04-23T16:46:47.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5672e-05 2024-04-23T16:46:47.078 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.078 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.078 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.078 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:39:22.157793+0000", 2024-04-23T16:46:47.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7404999999999996e-05 2024-04-23T16:46:47.079 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.079 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.079 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.079 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.079 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.079 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90970 7.3 7:c5362e43:::benchmark_data_smithi188_141483_object90969:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e564)", 2024-04-23T16:46:47.079 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.955423+0000", 2024-04-23T16:46:47.079 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 219.07663611699999, 2024-04-23T16:46:47.079 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.17896795700000001, 2024-04-23T16:46:47.079 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.079 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.079 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.079 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.079 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:46:47.079 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:47.080 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90970 2024-04-23T16:46:47.080 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.080 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.080 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.080 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.080 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955423+0000", 2024-04-23T16:46:47.080 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.080 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.080 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.080 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.080 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955423+0000", 2024-04-23T16:46:47.080 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.080 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.080 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.081 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.081 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955424+0000", 2024-04-23T16:46:47.081 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.378e-06 2024-04-23T16:46:47.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.081 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.081 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955518+0000", 2024-04-23T16:46:47.081 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3611999999999998e-05 2024-04-23T16:46:47.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.081 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.081 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955527+0000", 2024-04-23T16:46:47.081 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6099999999999995e-06 2024-04-23T16:46:47.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955540+0000", 2024-04-23T16:46:47.082 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2911999999999999e-05 2024-04-23T16:46:47.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.082 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.957217+0000", 2024-04-23T16:46:47.082 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001676528 2024-04-23T16:46:47.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.082 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:46:47.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.957224+0000", 2024-04-23T16:46:47.082 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0129999999999996e-06 2024-04-23T16:46:47.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.117685+0000", 2024-04-23T16:46:47.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.16046112500000001 2024-04-23T16:46:47.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.120209+0000", 2024-04-23T16:46:47.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025241790000000001 2024-04-23T16:46:47.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:46:47.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.120376+0000", 2024-04-23T16:46:47.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00016709200000000001 2024-04-23T16:46:47.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.084 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.084 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.127692+0000", 2024-04-23T16:46:47.084 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0073160459999999997 2024-04-23T16:46:47.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.084 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.084 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.132152+0000", 2024-04-23T16:46:47.084 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0044600830000000001 2024-04-23T16:46:47.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.084 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.084 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.132194+0000", 2024-04-23T16:46:47.084 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1566999999999997e-05 2024-04-23T16:46:47.085 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.085 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.085 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.085 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.134391+0000", 2024-04-23T16:46:47.085 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021968119999999998 2024-04-23T16:46:47.085 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.085 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.085 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.085 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.085 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.085 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90985 7.3 7:cd402d81:::benchmark_data_smithi188_141483_object90984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e564)", 2024-04-23T16:46:47.085 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.955551+0000", 2024-04-23T16:46:47.085 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 219.076507757, 2024-04-23T16:46:47.085 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.180068492, 2024-04-23T16:46:47.086 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.086 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.086 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.086 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.086 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:46:47.086 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:47.086 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90985 2024-04-23T16:46:47.086 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.086 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.086 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.086 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.086 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955551+0000", 2024-04-23T16:46:47.086 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.086 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.087 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.087 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.087 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955551+0000", 2024-04-23T16:46:47.087 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.087 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.087 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.087 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.087 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955552+0000", 2024-04-23T16:46:47.087 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5700000000000002e-07 2024-04-23T16:46:47.087 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.087 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.087 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.087 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955640+0000", 2024-04-23T16:46:47.087 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8832000000000006e-05 2024-04-23T16:46:47.088 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.088 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.088 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.088 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955647+0000", 2024-04-23T16:46:47.088 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3210000000000001e-06 2024-04-23T16:46:47.088 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.088 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.088 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.088 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955651+0000", 2024-04-23T16:46:47.088 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5070000000000002e-06 2024-04-23T16:46:47.088 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.088 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.088 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.088 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.957253+0000", 2024-04-23T16:46:47.088 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0016021449999999999 2024-04-23T16:46:47.089 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.089 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.089 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:46:47.089 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.957262+0000", 2024-04-23T16:46:47.089 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.054e-06 2024-04-23T16:46:47.089 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.089 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.089 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.089 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.124930+0000", 2024-04-23T16:46:47.089 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.167667447 2024-04-23T16:46:47.089 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.089 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.089 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.089 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.125079+0000", 2024-04-23T16:46:47.090 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014880399999999999 2024-04-23T16:46:47.090 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.090 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.090 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:46:47.090 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.125192+0000", 2024-04-23T16:46:47.090 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000113248 2024-04-23T16:46:47.090 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.090 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.090 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.090 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.132090+0000", 2024-04-23T16:46:47.090 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0068983550000000001 2024-04-23T16:46:47.090 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.090 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.090 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.134404+0000", 2024-04-23T16:46:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0023141669999999998 2024-04-23T16:46:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.134439+0000", 2024-04-23T16:46:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4937000000000001e-05 2024-04-23T16:46:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.135619+0000", 2024-04-23T16:46:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001180118 2024-04-23T16:46:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.091 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.092 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.092 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.092 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.092 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:142822 7.1 7:84205ba3:::benchmark_data_smithi188_141483_object24141:head [delete] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-04-23T16:46:47.092 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:44:11.533044+0000", 2024-04-23T16:46:47.092 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 155.49901484399999, 2024-04-23T16:46:47.092 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19764346999999999, 2024-04-23T16:46:47.092 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.092 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.092 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.092 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.092 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:46:47.092 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:47.092 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 142822 2024-04-23T16:46:47.093 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.093 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.093 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.093 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.093 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.533044+0000", 2024-04-23T16:46:47.093 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.093 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.093 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.093 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.093 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.533044+0000", 2024-04-23T16:46:47.093 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.093 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.093 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.093 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.093 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.533047+0000", 2024-04-23T16:46:47.094 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8380000000000002e-06 2024-04-23T16:46:47.094 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.094 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.094 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.094 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.533053+0000", 2024-04-23T16:46:47.094 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.071e-06 2024-04-23T16:46:47.094 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.094 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.094 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.094 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.533064+0000", 2024-04-23T16:46:47.094 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.08e-05 2024-04-23T16:46:47.094 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.094 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.094 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.095 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.533072+0000", 2024-04-23T16:46:47.095 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2349999999999999e-06 2024-04-23T16:46:47.095 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.095 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.095 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.095 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.538813+0000", 2024-04-23T16:46:47.095 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0057408390000000002 2024-04-23T16:46:47.095 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.095 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.095 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:46:47.095 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.538872+0000", 2024-04-23T16:46:47.095 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8941999999999999e-05 2024-04-23T16:46:47.095 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.095 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.095 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.096 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.663242+0000", 2024-04-23T16:46:47.096 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.12437045100000001 2024-04-23T16:46:47.096 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.096 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.096 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.096 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.663339+0000", 2024-04-23T16:46:47.096 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6487e-05 2024-04-23T16:46:47.096 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.096 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.096 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:46:47.096 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.663476+0000", 2024-04-23T16:46:47.096 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013769699999999999 2024-04-23T16:46:47.096 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.096 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.096 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.097 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.706593+0000", 2024-04-23T16:46:47.097 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.043116301000000003 2024-04-23T16:46:47.097 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.097 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.097 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.097 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.728302+0000", 2024-04-23T16:46:47.097 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021709513999999999 2024-04-23T16:46:47.097 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.097 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.097 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.097 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.728353+0000", 2024-04-23T16:46:47.097 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.058e-05 2024-04-23T16:46:47.097 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.097 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.098 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.098 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.730687+0000", 2024-04-23T16:46:47.098 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0023347149999999998 2024-04-23T16:46:47.098 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.098 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.098 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.098 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.098 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.098 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:142839 7.1 7:8d7b9cc0:::benchmark_data_smithi188_141483_object24158:head [delete] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-04-23T16:46:47.098 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:44:11.559611+0000", 2024-04-23T16:46:47.098 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 155.47244797299999, 2024-04-23T16:46:47.098 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.16867516299999999, 2024-04-23T16:46:47.098 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.098 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.099 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.099 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.099 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:46:47.099 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:47.099 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 142839 2024-04-23T16:46:47.099 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.099 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.099 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.099 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.099 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.559611+0000", 2024-04-23T16:46:47.099 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.099 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.099 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.099 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.099 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.559611+0000", 2024-04-23T16:46:47.100 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.100 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.100 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.100 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.100 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.559615+0000", 2024-04-23T16:46:47.100 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2420000000000002e-06 2024-04-23T16:46:47.100 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.100 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.100 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.100 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.559627+0000", 2024-04-23T16:46:47.100 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1681e-05 2024-04-23T16:46:47.100 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.100 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.101 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.101 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.559646+0000", 2024-04-23T16:46:47.101 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.978e-05 2024-04-23T16:46:47.101 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.101 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.101 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.101 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.559659+0000", 2024-04-23T16:46:47.101 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2310000000000001e-05 2024-04-23T16:46:47.101 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.101 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.101 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.101 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.571315+0000", 2024-04-23T16:46:47.101 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01165657 2024-04-23T16:46:47.101 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.101 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.102 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:46:47.102 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.571995+0000", 2024-04-23T16:46:47.102 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00067957200000000003 2024-04-23T16:46:47.102 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.102 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.102 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.102 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.640823+0000", 2024-04-23T16:46:47.102 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.068827837000000003 2024-04-23T16:46:47.102 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.102 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.102 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.102 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.640918+0000", 2024-04-23T16:46:47.102 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5236999999999996e-05 2024-04-23T16:46:47.103 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.103 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.103 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:46:47.103 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.641045+0000", 2024-04-23T16:46:47.103 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012659999999999999 2024-04-23T16:46:47.103 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.103 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.103 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.103 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.682283+0000", 2024-04-23T16:46:47.103 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.041238896999999997 2024-04-23T16:46:47.103 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.103 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.103 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.103 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.725628+0000", 2024-04-23T16:46:47.104 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.043344383 2024-04-23T16:46:47.104 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.104 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.104 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.104 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.725697+0000", 2024-04-23T16:46:47.104 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8770999999999996e-05 2024-04-23T16:46:47.104 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.104 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.104 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.104 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:11.728286+0000", 2024-04-23T16:46:47.104 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025892829999999999 2024-04-23T16:46:47.104 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.104 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.104 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.105 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.105 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.105 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:143561 7.1 7:88132120:::benchmark_data_smithi188_141483_object24880:head [delete] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-04-23T16:46:47.105 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:44:12.565217+0000", 2024-04-23T16:46:47.105 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 154.466841389, 2024-04-23T16:46:47.105 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.171397467, 2024-04-23T16:46:47.105 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.105 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.105 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.105 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.105 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:46:47.105 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:47.105 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 143561 2024-04-23T16:46:47.105 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.106 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.106 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.106 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.106 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.565217+0000", 2024-04-23T16:46:47.106 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.106 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.106 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.106 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.106 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.565217+0000", 2024-04-23T16:46:47.106 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.106 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.106 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.106 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.106 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.565219+0000", 2024-04-23T16:46:47.107 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0619999999999999e-06 2024-04-23T16:46:47.107 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.107 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.107 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.107 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.565226+0000", 2024-04-23T16:46:47.107 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2310000000000001e-06 2024-04-23T16:46:47.107 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.107 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.107 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.107 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.565237+0000", 2024-04-23T16:46:47.107 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.173e-05 2024-04-23T16:46:47.107 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.107 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.107 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.107 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.565246+0000", 2024-04-23T16:46:47.108 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0329999999999997e-06 2024-04-23T16:46:47.108 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.108 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.108 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.108 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.585510+0000", 2024-04-23T16:46:47.108 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020263757 2024-04-23T16:46:47.108 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.108 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.108 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:46:47.108 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.585554+0000", 2024-04-23T16:46:47.108 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4168999999999997e-05 2024-04-23T16:46:47.108 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.108 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.108 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.109 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.685228+0000", 2024-04-23T16:46:47.109 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.099674174000000004 2024-04-23T16:46:47.109 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.109 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.109 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.109 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.685285+0000", 2024-04-23T16:46:47.109 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6953000000000002e-05 2024-04-23T16:46:47.109 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.109 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.109 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:46:47.109 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.685356+0000", 2024-04-23T16:46:47.109 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0559e-05 2024-04-23T16:46:47.109 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.109 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.110 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.110 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.693401+0000", 2024-04-23T16:46:47.110 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0080455070000000004 2024-04-23T16:46:47.110 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.110 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.110 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.110 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.734307+0000", 2024-04-23T16:46:47.110 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.040905946999999998 2024-04-23T16:46:47.110 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.110 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.110 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.110 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.734369+0000", 2024-04-23T16:46:47.110 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1326000000000002e-05 2024-04-23T16:46:47.110 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.111 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.111 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.111 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.736615+0000", 2024-04-23T16:46:47.111 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0022460190000000001 2024-04-23T16:46:47.111 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.111 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.111 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.111 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.111 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.111 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:143591 7.1 7:82f43a44:::benchmark_data_smithi188_141483_object24910:head [delete] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-04-23T16:46:47.111 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:44:12.605683+0000", 2024-04-23T16:46:47.111 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 154.42637595100001, 2024-04-23T16:46:47.111 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.17438996200000001, 2024-04-23T16:46:47.111 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.112 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.112 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.112 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.112 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:46:47.112 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:47.112 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 143591 2024-04-23T16:46:47.112 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.112 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.112 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.112 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.112 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.605683+0000", 2024-04-23T16:46:47.112 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.112 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.112 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.112 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.113 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.605683+0000", 2024-04-23T16:46:47.113 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.113 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.113 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.113 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.113 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.605687+0000", 2024-04-23T16:46:47.113 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8600000000000003e-06 2024-04-23T16:46:47.113 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.113 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.113 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.113 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.605696+0000", 2024-04-23T16:46:47.113 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2150000000000007e-06 2024-04-23T16:46:47.113 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.113 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.113 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.114 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.605711+0000", 2024-04-23T16:46:47.114 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5464999999999998e-05 2024-04-23T16:46:47.114 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.114 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.114 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.114 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.605721+0000", 2024-04-23T16:46:47.114 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0125e-05 2024-04-23T16:46:47.114 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.114 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.115 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.115 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.609520+0000", 2024-04-23T16:46:47.115 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0037989019999999998 2024-04-23T16:46:47.115 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.115 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.115 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:46:47.115 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.609559+0000", 2024-04-23T16:46:47.115 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9155e-05 2024-04-23T16:46:47.115 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.115 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.115 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.115 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.715853+0000", 2024-04-23T16:46:47.115 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.10629387999999999 2024-04-23T16:46:47.115 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.116 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.116 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.116 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.715958+0000", 2024-04-23T16:46:47.116 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000105105 2024-04-23T16:46:47.116 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.116 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.116 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:46:47.116 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.716095+0000", 2024-04-23T16:46:47.116 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000136985 2024-04-23T16:46:47.116 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.116 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.116 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.116 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.743394+0000", 2024-04-23T16:46:47.117 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.027298354 2024-04-23T16:46:47.117 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.117 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.117 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.117 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.777670+0000", 2024-04-23T16:46:47.117 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.034276607000000001 2024-04-23T16:46:47.117 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.117 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.117 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.117 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.777748+0000", 2024-04-23T16:46:47.118 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7853000000000002e-05 2024-04-23T16:46:47.118 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.118 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.118 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.118 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:12.780073+0000", 2024-04-23T16:46:47.118 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0023244559999999999 2024-04-23T16:46:47.118 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.118 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.118 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.118 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.118 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.118 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:145495 7.1 7:813d9471:::benchmark_data_smithi188_141483_object26814:head [delete] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-04-23T16:46:47.118 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:44:15.225218+0000", 2024-04-23T16:46:47.119 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 151.806840835, 2024-04-23T16:46:47.119 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.15784366599999999, 2024-04-23T16:46:47.119 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.119 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.119 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.119 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.119 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:46:47.119 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:47.119 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 145495 2024-04-23T16:46:47.119 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.119 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.119 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.119 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.119 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:15.225218+0000", 2024-04-23T16:46:47.120 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.120 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.120 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.120 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.120 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:15.225218+0000", 2024-04-23T16:46:47.120 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.120 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.120 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.120 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.120 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:15.225221+0000", 2024-04-23T16:46:47.120 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.016e-06 2024-04-23T16:46:47.120 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.120 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.121 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.121 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:15.225228+0000", 2024-04-23T16:46:47.121 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9580000000000001e-06 2024-04-23T16:46:47.121 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.121 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.121 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.121 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:15.225239+0000", 2024-04-23T16:46:47.121 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0767999999999999e-05 2024-04-23T16:46:47.121 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.121 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.121 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.121 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:15.225246+0000", 2024-04-23T16:46:47.121 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7190000000000001e-06 2024-04-23T16:46:47.121 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.122 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.122 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.122 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:15.240696+0000", 2024-04-23T16:46:47.122 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015450101000000001 2024-04-23T16:46:47.122 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.122 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.122 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:46:47.122 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:15.240768+0000", 2024-04-23T16:46:47.122 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1185000000000004e-05 2024-04-23T16:46:47.122 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.122 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.122 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.122 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:15.304205+0000", 2024-04-23T16:46:47.123 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.063437438999999998 2024-04-23T16:46:47.123 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.123 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.123 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.123 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:15.304301+0000", 2024-04-23T16:46:47.123 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5911999999999999e-05 2024-04-23T16:46:47.123 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.123 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.123 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:46:47.123 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:15.304432+0000", 2024-04-23T16:46:47.123 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013061400000000001 2024-04-23T16:46:47.123 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.123 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.123 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.123 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:15.354207+0000", 2024-04-23T16:46:47.124 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.049775231000000003 2024-04-23T16:46:47.124 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.124 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.124 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.124 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:15.381138+0000", 2024-04-23T16:46:47.124 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026930826000000001 2024-04-23T16:46:47.124 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.124 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.124 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.125 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:15.381171+0000", 2024-04-23T16:46:47.125 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2954000000000003e-05 2024-04-23T16:46:47.125 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.125 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.125 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.125 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:15.383062+0000", 2024-04-23T16:46:47.125 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001890943 2024-04-23T16:46:47.125 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.125 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.125 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.125 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.125 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.125 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:146883 7.1 7:80b3f15b:::benchmark_data_smithi188_141483_object28202:head [delete] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-04-23T16:46:47.125 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:44:17.038504+0000", 2024-04-23T16:46:47.125 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 149.99355438800001, 2024-04-23T16:46:47.126 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.16464025600000001, 2024-04-23T16:46:47.126 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.126 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.126 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.126 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.126 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:46:47.126 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:47.126 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 146883 2024-04-23T16:46:47.126 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.126 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.126 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.126 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.126 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:17.038504+0000", 2024-04-23T16:46:47.127 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.127 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.127 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.127 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.127 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:17.038504+0000", 2024-04-23T16:46:47.127 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.127 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.127 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.127 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.127 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:17.038505+0000", 2024-04-23T16:46:47.127 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.17e-06 2024-04-23T16:46:47.127 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.127 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.128 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.128 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:17.038509+0000", 2024-04-23T16:46:47.128 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8240000000000001e-06 2024-04-23T16:46:47.128 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.128 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.128 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.128 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:17.038515+0000", 2024-04-23T16:46:47.128 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9349999999999999e-06 2024-04-23T16:46:47.128 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.128 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.128 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.128 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:17.038521+0000", 2024-04-23T16:46:47.128 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8220000000000003e-06 2024-04-23T16:46:47.128 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.129 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.129 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.129 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:17.086992+0000", 2024-04-23T16:46:47.129 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048470882 2024-04-23T16:46:47.129 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.129 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.129 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:46:47.129 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:17.088034+0000", 2024-04-23T16:46:47.129 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0010425199999999999 2024-04-23T16:46:47.129 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.129 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.129 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.129 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:17.148245+0000", 2024-04-23T16:46:47.129 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.060210605 2024-04-23T16:46:47.129 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.130 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.130 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.130 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:17.148343+0000", 2024-04-23T16:46:47.130 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8171999999999995e-05 2024-04-23T16:46:47.130 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.130 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.130 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:46:47.130 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:17.148484+0000", 2024-04-23T16:46:47.130 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014112200000000001 2024-04-23T16:46:47.130 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.130 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.130 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.130 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:17.185147+0000", 2024-04-23T16:46:47.131 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.036662721000000002 2024-04-23T16:46:47.131 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.131 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.131 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.131 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:17.200821+0000", 2024-04-23T16:46:47.131 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015673646999999999 2024-04-23T16:46:47.131 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.131 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.131 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.131 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:17.200873+0000", 2024-04-23T16:46:47.131 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2411999999999999e-05 2024-04-23T16:46:47.131 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.131 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.131 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.132 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:17.203145+0000", 2024-04-23T16:46:47.132 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0022714240000000002 2024-04-23T16:46:47.132 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.132 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.132 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.132 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.132 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.132 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:162056 7.1 7:8227e60a:::benchmark_data_smithi188_141483_object43375:head [delete] snapc 0=[] ondisk+write+known_if_redirected e582)", 2024-04-23T16:46:47.132 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:44:38.057950+0000", 2024-04-23T16:46:47.132 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 128.974108376, 2024-04-23T16:46:47.132 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.159124659, 2024-04-23T16:46:47.132 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.132 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.132 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.133 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.133 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:46:47.133 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:47.133 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 162056 2024-04-23T16:46:47.133 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.133 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.133 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.133 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.133 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.057950+0000", 2024-04-23T16:46:47.133 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.133 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.133 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.133 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.133 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.057950+0000", 2024-04-23T16:46:47.134 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.134 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.134 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.134 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.134 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.057952+0000", 2024-04-23T16:46:47.134 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1280000000000002e-06 2024-04-23T16:46:47.134 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.134 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.134 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.134 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.057958+0000", 2024-04-23T16:46:47.134 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9270000000000002e-06 2024-04-23T16:46:47.134 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.134 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.134 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.135 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.057969+0000", 2024-04-23T16:46:47.135 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0355e-05 2024-04-23T16:46:47.135 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.135 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.135 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.135 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.057976+0000", 2024-04-23T16:46:47.135 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3889999999999999e-06 2024-04-23T16:46:47.135 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.135 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.135 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.135 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.089233+0000", 2024-04-23T16:46:47.135 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.031256752999999998 2024-04-23T16:46:47.135 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.135 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.136 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:46:47.136 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.089281+0000", 2024-04-23T16:46:47.136 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8041000000000003e-05 2024-04-23T16:46:47.136 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.136 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.136 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.136 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.172867+0000", 2024-04-23T16:46:47.136 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.083586400000000005 2024-04-23T16:46:47.136 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.136 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.136 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.137 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.172969+0000", 2024-04-23T16:46:47.137 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000102097 2024-04-23T16:46:47.137 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.137 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.137 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:46:47.137 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.173099+0000", 2024-04-23T16:46:47.137 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012912899999999999 2024-04-23T16:46:47.137 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.137 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.137 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.137 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.207659+0000", 2024-04-23T16:46:47.137 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.034560232000000003 2024-04-23T16:46:47.137 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.137 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.138 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.138 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.214949+0000", 2024-04-23T16:46:47.138 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.007290403 2024-04-23T16:46:47.138 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.138 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.138 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.138 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.215007+0000", 2024-04-23T16:46:47.138 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8165000000000003e-05 2024-04-23T16:46:47.138 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.138 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.138 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.138 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.217075+0000", 2024-04-23T16:46:47.138 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0020676399999999999 2024-04-23T16:46:47.139 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.139 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.139 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.139 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.139 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.139 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:162231 7.1 7:88727f99:::benchmark_data_smithi188_141483_object43550:head [delete] snapc 0=[] ondisk+write+known_if_redirected e582)", 2024-04-23T16:46:47.139 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:44:38.311869+0000", 2024-04-23T16:46:47.139 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 128.72018927400001, 2024-04-23T16:46:47.139 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.16640809200000001, 2024-04-23T16:46:47.139 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.139 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.139 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.139 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.139 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:46:47.139 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:47.140 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 162231 2024-04-23T16:46:47.140 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.140 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.140 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.140 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.140 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.311869+0000", 2024-04-23T16:46:47.140 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.140 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.140 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.140 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.140 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.311869+0000", 2024-04-23T16:46:47.140 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.141 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.141 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.141 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.141 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.311870+0000", 2024-04-23T16:46:47.141 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8599999999999998e-07 2024-04-23T16:46:47.141 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.141 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.141 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.141 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.311873+0000", 2024-04-23T16:46:47.141 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.666e-06 2024-04-23T16:46:47.141 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.141 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.142 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.142 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.311877+0000", 2024-04-23T16:46:47.142 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7759999999999997e-06 2024-04-23T16:46:47.142 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.142 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.142 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.142 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.311883+0000", 2024-04-23T16:46:47.142 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7339999999999996e-06 2024-04-23T16:46:47.142 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.142 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.142 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.142 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.336861+0000", 2024-04-23T16:46:47.143 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.024977841000000001 2024-04-23T16:46:47.143 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.143 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.143 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:46:47.143 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.336929+0000", 2024-04-23T16:46:47.145 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8017999999999998e-05 2024-04-23T16:46:47.145 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.145 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.145 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.146 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.363816+0000", 2024-04-23T16:46:47.146 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026887019000000002 2024-04-23T16:46:47.146 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.146 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.146 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.146 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.363904+0000", 2024-04-23T16:46:47.146 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8218000000000001e-05 2024-04-23T16:46:47.146 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.146 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.146 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:46:47.146 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.364025+0000", 2024-04-23T16:46:47.146 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000120332 2024-04-23T16:46:47.146 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.147 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.147 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.147 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.396710+0000", 2024-04-23T16:46:47.147 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.032685436999999998 2024-04-23T16:46:47.147 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.147 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.147 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.147 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.476320+0000", 2024-04-23T16:46:47.147 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.079610091999999993 2024-04-23T16:46:47.147 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.147 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.147 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.147 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.476379+0000", 2024-04-23T16:46:47.148 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8656999999999998e-05 2024-04-23T16:46:47.148 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.148 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.148 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.148 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.478278+0000", 2024-04-23T16:46:47.148 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001898816 2024-04-23T16:46:47.148 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.148 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.148 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.148 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.148 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.148 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:162242 7.1 7:834e25fa:::benchmark_data_smithi188_141483_object43561:head [delete] snapc 0=[] ondisk+write+known_if_redirected e582)", 2024-04-23T16:46:47.148 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:44:38.336154+0000", 2024-04-23T16:46:47.148 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 128.69590452, 2024-04-23T16:46:47.149 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.160488661, 2024-04-23T16:46:47.149 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.149 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.149 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.149 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.149 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:46:47.149 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:47.149 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 162242 2024-04-23T16:46:47.149 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.149 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.149 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.149 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.149 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.336154+0000", 2024-04-23T16:46:47.149 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.150 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.150 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.150 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.150 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.336154+0000", 2024-04-23T16:46:47.150 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.150 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.150 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.150 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.150 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.336156+0000", 2024-04-23T16:46:47.150 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0609999999999998e-06 2024-04-23T16:46:47.150 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.150 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.150 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.150 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.336161+0000", 2024-04-23T16:46:47.151 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9690000000000002e-06 2024-04-23T16:46:47.151 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.151 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.151 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.151 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.336174+0000", 2024-04-23T16:46:47.151 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2566e-05 2024-04-23T16:46:47.151 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.151 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.151 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.151 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.336181+0000", 2024-04-23T16:46:47.151 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0160000000000003e-06 2024-04-23T16:46:47.151 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.151 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.151 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.152 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.358651+0000", 2024-04-23T16:46:47.152 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022470475 2024-04-23T16:46:47.152 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.152 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.152 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:46:47.152 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.359243+0000", 2024-04-23T16:46:47.152 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00059179299999999996 2024-04-23T16:46:47.152 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.152 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.152 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.152 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.440325+0000", 2024-04-23T16:46:47.152 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.081081462000000007 2024-04-23T16:46:47.152 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.153 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.153 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.153 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.440420+0000", 2024-04-23T16:46:47.153 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5351000000000002e-05 2024-04-23T16:46:47.153 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.153 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.153 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:46:47.153 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.440555+0000", 2024-04-23T16:46:47.153 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013476300000000001 2024-04-23T16:46:47.153 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.153 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.153 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.153 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.452500+0000", 2024-04-23T16:46:47.153 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011945267000000001 2024-04-23T16:46:47.154 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.154 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.154 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.154 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.494597+0000", 2024-04-23T16:46:47.154 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.042096702 2024-04-23T16:46:47.154 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.154 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.154 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.154 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.494655+0000", 2024-04-23T16:46:47.154 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7944e-05 2024-04-23T16:46:47.154 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.154 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.154 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.154 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.496643+0000", 2024-04-23T16:46:47.155 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001988292 2024-04-23T16:46:47.155 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.155 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.155 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.155 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.155 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.155 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:162244 7.1 7:8d913904:::benchmark_data_smithi188_141483_object43563:head [delete] snapc 0=[] ondisk+write+known_if_redirected e582)", 2024-04-23T16:46:47.155 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:44:38.336198+0000", 2024-04-23T16:46:47.155 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 128.69586103, 2024-04-23T16:46:47.155 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.201851999, 2024-04-23T16:46:47.155 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.155 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.155 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.155 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.156 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:46:47.156 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:47.156 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 162244 2024-04-23T16:46:47.156 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.156 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.156 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.156 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.156 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.336198+0000", 2024-04-23T16:46:47.156 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.156 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.156 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.156 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.156 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.336198+0000", 2024-04-23T16:46:47.157 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.157 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.157 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.157 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.157 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.336199+0000", 2024-04-23T16:46:47.157 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8100000000000001e-07 2024-04-23T16:46:47.157 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.157 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.157 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.157 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.336202+0000", 2024-04-23T16:46:47.157 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.038e-06 2024-04-23T16:46:47.157 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.157 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.158 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.158 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.336208+0000", 2024-04-23T16:46:47.158 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9020000000000004e-06 2024-04-23T16:46:47.158 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.158 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.158 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.159 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.336214+0000", 2024-04-23T16:46:47.159 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0140000000000004e-06 2024-04-23T16:46:47.159 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.159 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.159 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.159 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.359278+0000", 2024-04-23T16:46:47.159 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023064068 2024-04-23T16:46:47.159 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.159 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.159 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:46:47.159 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.359727+0000", 2024-04-23T16:46:47.160 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00044905599999999998 2024-04-23T16:46:47.160 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.160 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.160 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.160 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.496817+0000", 2024-04-23T16:46:47.160 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.137089978 2024-04-23T16:46:47.160 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.160 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.160 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.160 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.496880+0000", 2024-04-23T16:46:47.160 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3740999999999998e-05 2024-04-23T16:46:47.161 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.161 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.161 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:46:47.161 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.496960+0000", 2024-04-23T16:46:47.161 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.9961000000000002e-05 2024-04-23T16:46:47.161 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.161 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.161 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.161 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.527865+0000", 2024-04-23T16:46:47.162 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.030904805 2024-04-23T16:46:47.162 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.162 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.162 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.162 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.536046+0000", 2024-04-23T16:46:47.162 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0081806299999999995 2024-04-23T16:46:47.162 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.162 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.162 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.162 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.536097+0000", 2024-04-23T16:46:47.162 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1363000000000003e-05 2024-04-23T16:46:47.162 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.162 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.163 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.163 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.538050+0000", 2024-04-23T16:46:47.163 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0019524620000000001 2024-04-23T16:46:47.163 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.163 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.163 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.163 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.163 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.163 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:162254 7.1 7:87afd614:::benchmark_data_smithi188_141483_object43573:head [delete] snapc 0=[] ondisk+write+known_if_redirected e582)", 2024-04-23T16:46:47.164 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:44:38.352470+0000", 2024-04-23T16:46:47.164 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 128.679588545, 2024-04-23T16:46:47.164 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.172789886, 2024-04-23T16:46:47.164 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.164 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.164 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.164 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.164 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:46:47.164 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:47.164 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 162254 2024-04-23T16:46:47.164 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.164 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.164 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.165 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.165 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.352470+0000", 2024-04-23T16:46:47.165 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.165 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.165 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.165 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.165 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.352470+0000", 2024-04-23T16:46:47.165 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.165 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.165 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.165 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.165 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.352473+0000", 2024-04-23T16:46:47.165 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.615e-06 2024-04-23T16:46:47.165 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.165 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.166 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.166 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.352478+0000", 2024-04-23T16:46:47.166 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2249999999999999e-06 2024-04-23T16:46:47.166 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.166 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.166 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.166 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.352487+0000", 2024-04-23T16:46:47.166 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.4280000000000002e-06 2024-04-23T16:46:47.166 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.166 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.166 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.166 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.352494+0000", 2024-04-23T16:46:47.166 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5570000000000001e-06 2024-04-23T16:46:47.166 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.167 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.167 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.167 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.359760+0000", 2024-04-23T16:46:47.167 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0072663750000000003 2024-04-23T16:46:47.167 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.167 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.167 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:46:47.167 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.360284+0000", 2024-04-23T16:46:47.167 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00052409699999999995 2024-04-23T16:46:47.167 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.167 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.167 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.167 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.462288+0000", 2024-04-23T16:46:47.167 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.102003677 2024-04-23T16:46:47.168 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.168 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.168 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.168 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.462395+0000", 2024-04-23T16:46:47.168 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107149 2024-04-23T16:46:47.168 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.168 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.168 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:46:47.168 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.462545+0000", 2024-04-23T16:46:47.168 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014974399999999999 2024-04-23T16:46:47.168 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.168 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.168 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.168 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.496698+0000", 2024-04-23T16:46:47.168 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.034153416999999998 2024-04-23T16:46:47.169 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.169 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.169 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.169 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.523340+0000", 2024-04-23T16:46:47.169 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026641933 2024-04-23T16:46:47.169 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.169 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.169 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.169 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.523391+0000", 2024-04-23T16:46:47.169 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1093e-05 2024-04-23T16:46:47.169 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.169 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.169 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.169 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.525260+0000", 2024-04-23T16:46:47.170 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001868576 2024-04-23T16:46:47.170 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.170 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.170 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.170 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.170 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.170 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:162265 7.1 7:810814d2:::benchmark_data_smithi188_141483_object43584:head [delete] snapc 0=[] ondisk+write+known_if_redirected e582)", 2024-04-23T16:46:47.170 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:44:38.358121+0000", 2024-04-23T16:46:47.170 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 128.673937309, 2024-04-23T16:46:47.170 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.16823956900000001, 2024-04-23T16:46:47.170 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.170 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.171 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.171 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.171 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:46:47.171 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:47.171 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 162265 2024-04-23T16:46:47.172 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.172 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.172 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.172 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.172 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.358121+0000", 2024-04-23T16:46:47.172 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.172 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.172 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.172 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.172 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.358121+0000", 2024-04-23T16:46:47.172 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.172 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.172 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.172 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.173 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.358123+0000", 2024-04-23T16:46:47.173 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9700000000000002e-06 2024-04-23T16:46:47.173 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.173 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.173 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.173 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.358129+0000", 2024-04-23T16:46:47.173 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5500000000000002e-06 2024-04-23T16:46:47.173 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.173 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.173 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.173 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.358139+0000", 2024-04-23T16:46:47.173 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6670000000000008e-06 2024-04-23T16:46:47.174 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.174 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.174 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.174 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.358147+0000", 2024-04-23T16:46:47.174 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1319999999999994e-06 2024-04-23T16:46:47.174 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.174 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.174 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.174 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.360320+0000", 2024-04-23T16:46:47.174 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021736630000000002 2024-04-23T16:46:47.174 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.175 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.175 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:46:47.175 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.360832+0000", 2024-04-23T16:46:47.175 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00051133099999999996 2024-04-23T16:46:47.175 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.175 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.175 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.175 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.470671+0000", 2024-04-23T16:46:47.175 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.10983942100000001 2024-04-23T16:46:47.175 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.175 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.175 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.175 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.470753+0000", 2024-04-23T16:46:47.175 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1848000000000006e-05 2024-04-23T16:46:47.176 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.176 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.176 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:46:47.176 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.470823+0000", 2024-04-23T16:46:47.176 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0318e-05 2024-04-23T16:46:47.176 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.176 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.176 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.176 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.496742+0000", 2024-04-23T16:46:47.176 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.025918502999999999 2024-04-23T16:46:47.176 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.176 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.177 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.177 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.525273+0000", 2024-04-23T16:46:47.177 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028531007000000001 2024-04-23T16:46:47.177 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.177 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.177 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.177 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.525322+0000", 2024-04-23T16:46:47.177 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9248e-05 2024-04-23T16:46:47.177 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.177 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.177 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.177 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.526361+0000", 2024-04-23T16:46:47.177 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001038911 2024-04-23T16:46:47.177 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.177 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.178 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.178 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.178 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.178 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:162266 7.1 7:8842e472:::benchmark_data_smithi188_141483_object43585:head [delete] snapc 0=[] ondisk+write+known_if_redirected e582)", 2024-04-23T16:46:47.178 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:44:38.364157+0000", 2024-04-23T16:46:47.178 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 128.66790147200001, 2024-04-23T16:46:47.178 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.192253598, 2024-04-23T16:46:47.178 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:47.178 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:47.178 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:46:47.178 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:47.178 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:46:47.178 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:46:47.179 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 162266 2024-04-23T16:46:47.179 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.179 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:47.179 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.179 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:47.179 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.364157+0000", 2024-04-23T16:46:47.179 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.179 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.179 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.179 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:47.179 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.364157+0000", 2024-04-23T16:46:47.179 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:47.180 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.180 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.180 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:47.180 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.364159+0000", 2024-04-23T16:46:47.180 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.182e-06 2024-04-23T16:46:47.180 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.180 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.180 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:47.180 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.364166+0000", 2024-04-23T16:46:47.180 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3879999999999997e-06 2024-04-23T16:46:47.180 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.180 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.180 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:47.180 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.364177+0000", 2024-04-23T16:46:47.181 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0823e-05 2024-04-23T16:46:47.181 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.181 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.181 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:47.181 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.364184+0000", 2024-04-23T16:46:47.181 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.712e-06 2024-04-23T16:46:47.181 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.181 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.181 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.181 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.396817+0000", 2024-04-23T16:46:47.181 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.032632886 2024-04-23T16:46:47.181 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.181 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.181 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:46:47.182 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.396884+0000", 2024-04-23T16:46:47.182 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7106000000000004e-05 2024-04-23T16:46:47.182 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.182 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.182 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:47.182 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.512580+0000", 2024-04-23T16:46:47.182 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.11569542100000001 2024-04-23T16:46:47.182 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.182 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.182 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:47.182 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.512648+0000", 2024-04-23T16:46:47.182 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8338999999999997e-05 2024-04-23T16:46:47.182 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.183 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.183 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 6", 2024-04-23T16:46:47.183 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.512734+0000", 2024-04-23T16:46:47.183 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5575000000000007e-05 2024-04-23T16:46:47.183 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.183 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.183 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:47.183 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.546077+0000", 2024-04-23T16:46:47.183 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033343042000000003 2024-04-23T16:46:47.183 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.183 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.183 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:46:47.183 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.554539+0000", 2024-04-23T16:46:47.184 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0084623779999999996 2024-04-23T16:46:47.184 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.184 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.184 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:46:47.184 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.554589+0000", 2024-04-23T16:46:47.184 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9929999999999998e-05 2024-04-23T16:46:47.184 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:47.184 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:47.184 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:46:47.184 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:44:38.556411+0000", 2024-04-23T16:46:47.184 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001821816 2024-04-23T16:46:47.184 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.184 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.185 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.185 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:47.185 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:47.185 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:46:47.186 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 10 16 37795 37779 236.066 180.688 0.00411081 0.00423431 2024-04-23T16:46:47.186 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:46 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/293532759' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:46:47.186 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:47.619 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:47.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:47.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:47.723 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:46:47.759 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:47.786 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-23T16:46:47.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:47 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:46:47.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:47 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/293532759' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:46:47.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:47 smithi188 ceph-mon[44101]: osdmap e614: 12 total, 11 up, 9 in 2024-04-23T16:46:47.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:47 smithi188 ceph-mon[44101]: pgmap v1730: 27 pgs: 27 active+clean; 1.9 GiB data, 8.4 GiB used, 796 GiB / 805 GiB avail; 212 MiB/s wr, 3.39k op/s 2024-04-23T16:46:47.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:47 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:46:48.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:47 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:46:48.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:47 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/293532759' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:46:48.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:47 smithi103 ceph-mon[39629]: osdmap e614: 12 total, 11 up, 9 in 2024-04-23T16:46:48.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:47 smithi103 ceph-mon[39629]: pgmap v1730: 27 pgs: 27 active+clean; 1.9 GiB data, 8.4 GiB used, 796 GiB / 805 GiB avail; 212 MiB/s wr, 3.39k op/s 2024-04-23T16:46:48.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:47 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:46:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:47 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:46:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:47 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/293532759' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:46:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:47 smithi148 ceph-mon[44520]: osdmap e614: 12 total, 11 up, 9 in 2024-04-23T16:46:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:47 smithi148 ceph-mon[44520]: pgmap v1730: 27 pgs: 27 active+clean; 1.9 GiB data, 8.4 GiB used, 796 GiB / 805 GiB avail; 212 MiB/s wr, 3.39k op/s 2024-04-23T16:46:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:47 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:46:48.144 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 11 16 40263 40247 228.609 154.25 0.0054937 0.00437222 2024-04-23T16:46:48.154 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:46:48.361 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-04-23T16:46:48.362 INFO:tasks.cephadm.osd.3:Restarting osd.3 (starting--it wasn't running)... 2024-04-23T16:46:48.362 DEBUG:teuthology.orchestra.run.smithi103:osd.3> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.3.service 2024-04-23T16:46:48.364 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl start ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.3 2024-04-23T16:46:48.395 INFO:journalctl@ceph.osd.3.smithi103.stdout:-- Logs begin at Tue 2024-04-23 15:57:24 UTC. -- 2024-04-23T16:46:48.600 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:46:48 smithi103 systemd[1]: Starting Ceph osd.3 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T16:46:49.144 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 12 16 42756 42740 222.542 155.812 0.00193778 0.00448849 2024-04-23T16:46:49.314 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:46:49.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:49.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:46:49.417 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:46:49.461 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:49.947 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:49 smithi103 ceph-mon[39629]: pgmap v1731: 27 pgs: 27 active+clean; 2.2 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 215 MiB/s wr, 3.43k op/s 2024-04-23T16:46:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:49 smithi148 ceph-mon[44520]: pgmap v1731: 27 pgs: 27 active+clean; 2.2 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 215 MiB/s wr, 3.43k op/s 2024-04-23T16:46:50.145 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 13 16 45177 45161 217.063 151.312 0.00462782 0.00460591 2024-04-23T16:46:50.181 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:46:50.181 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:46:50.181 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.181 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:45253 8.0 8:05c03d4d:::benchmark_data_smithi188_155546_object45252:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e614)", 2024-04-23T16:46:50.181 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:50.177428+0000", 2024-04-23T16:46:50.181 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.002682921, 2024-04-23T16:46:50.181 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0027152019999999999, 2024-04-23T16:46:50.181 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:50.182 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:50.182 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:46:50.182 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:50.182 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:46:50.182 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:46:50.182 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 45253 2024-04-23T16:46:50.182 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.182 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:50.182 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.182 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:50.182 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.177428+0000", 2024-04-23T16:46:50.182 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:50.182 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.182 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.183 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:50.183 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.177428+0000", 2024-04-23T16:46:50.183 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:50.183 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.183 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.183 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:50.183 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.177429+0000", 2024-04-23T16:46:50.183 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.322e-06 2024-04-23T16:46:50.183 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.183 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.183 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:50.183 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.177484+0000", 2024-04-23T16:46:50.183 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4811000000000001e-05 2024-04-23T16:46:50.183 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.184 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.184 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:50.184 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.177493+0000", 2024-04-23T16:46:50.184 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.4979999999999995e-06 2024-04-23T16:46:50.184 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.184 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.184 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:50.184 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.177499+0000", 2024-04-23T16:46:50.184 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6540000000000004e-06 2024-04-23T16:46:50.184 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.184 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.184 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:50.184 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.177547+0000", 2024-04-23T16:46:50.185 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8301999999999999e-05 2024-04-23T16:46:50.185 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.185 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.185 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:50.185 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.177701+0000", 2024-04-23T16:46:50.185 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000153907 2024-04-23T16:46:50.185 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.185 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.185 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:46:50.185 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.177838+0000", 2024-04-23T16:46:50.185 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000137174 2024-04-23T16:46:50.185 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.185 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.185 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:46:50.185 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.179961+0000", 2024-04-23T16:46:50.186 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021222749999999999 2024-04-23T16:46:50.186 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:50.186 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:50.186 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:50.186 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.186 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.186 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:45254 8.0 8:08fe6383:::benchmark_data_smithi188_155546_object45253:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e614)", 2024-04-23T16:46:50.186 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:50.177511+0000", 2024-04-23T16:46:50.186 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.0025997709999999999, 2024-04-23T16:46:50.186 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0027294900000000002, 2024-04-23T16:46:50.186 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:50.186 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:50.187 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:46:50.187 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:50.187 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:46:50.187 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:46:50.187 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 45254 2024-04-23T16:46:50.187 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.187 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:50.187 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.187 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:50.187 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.177511+0000", 2024-04-23T16:46:50.187 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:50.187 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.187 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.187 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:50.187 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.177511+0000", 2024-04-23T16:46:50.188 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:50.188 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.188 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.188 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:50.188 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.177511+0000", 2024-04-23T16:46:50.188 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7899999999999998e-07 2024-04-23T16:46:50.188 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.188 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.188 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:50.188 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.177547+0000", 2024-04-23T16:46:50.188 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5865999999999998e-05 2024-04-23T16:46:50.188 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.188 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.188 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:50.188 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.177559+0000", 2024-04-23T16:46:50.189 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1902999999999999e-05 2024-04-23T16:46:50.189 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.189 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.189 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:50.189 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.177566+0000", 2024-04-23T16:46:50.189 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5729999999999996e-06 2024-04-23T16:46:50.189 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.189 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.189 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:50.189 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.179039+0000", 2024-04-23T16:46:50.189 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0014732269999999999 2024-04-23T16:46:50.189 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.189 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.189 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:50.189 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.179130+0000", 2024-04-23T16:46:50.190 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1491999999999997e-05 2024-04-23T16:46:50.190 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.190 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.190 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:46:50.190 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.179215+0000", 2024-04-23T16:46:50.190 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4818e-05 2024-04-23T16:46:50.190 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:50.190 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:50.190 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:50.190 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.190 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.190 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:45259 8.8 8:14003470:::benchmark_data_smithi188_155546_object45258:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e614)", 2024-04-23T16:46:50.190 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:50.179807+0000", 2024-04-23T16:46:50.190 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.00030336300000000002, 2024-04-23T16:46:50.191 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00051344200000000002, 2024-04-23T16:46:50.191 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:50.191 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:50.191 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:46:50.191 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:50.191 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:46:50.191 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:46:50.191 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 45259 2024-04-23T16:46:50.191 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.191 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:50.191 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.191 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:50.191 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.179807+0000", 2024-04-23T16:46:50.191 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:50.191 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.192 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.192 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:50.192 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.179807+0000", 2024-04-23T16:46:50.192 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:50.192 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.192 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.192 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:50.192 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.179816+0000", 2024-04-23T16:46:50.192 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7390000000000005e-06 2024-04-23T16:46:50.192 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.192 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.192 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:50.192 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.179871+0000", 2024-04-23T16:46:50.192 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5486999999999999e-05 2024-04-23T16:46:50.192 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.193 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.193 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:50.193 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.179890+0000", 2024-04-23T16:46:50.193 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8510999999999999e-05 2024-04-23T16:46:50.193 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.193 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.193 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:50.193 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.179898+0000", 2024-04-23T16:46:50.193 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6190000000000002e-06 2024-04-23T16:46:50.193 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.193 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.193 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:50.193 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.179960+0000", 2024-04-23T16:46:50.193 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2088000000000006e-05 2024-04-23T16:46:50.193 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.194 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.194 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:50.194 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.180072+0000", 2024-04-23T16:46:50.194 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000112089 2024-04-23T16:46:50.194 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.194 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.194 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:46:50.194 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.180159+0000", 2024-04-23T16:46:50.194 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6802e-05 2024-04-23T16:46:50.194 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:50.194 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:50.194 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:50.194 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.194 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.195 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:45260 8.4 8:2a8584d1:::benchmark_data_smithi188_155546_object45259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e614)", 2024-04-23T16:46:50.195 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:50.179924+0000", 2024-04-23T16:46:50.195 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.000186126, 2024-04-23T16:46:50.195 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00047214799999999998, 2024-04-23T16:46:50.195 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:46:50.195 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:46:50.195 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:46:50.195 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:46:50.195 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:46:50.195 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:46:50.195 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 45260 2024-04-23T16:46:50.195 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.195 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:46:50.195 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.196 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:46:50.196 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.179924+0000", 2024-04-23T16:46:50.196 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:50.196 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.196 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.196 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:46:50.196 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.179924+0000", 2024-04-23T16:46:50.196 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:46:50.196 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.196 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.196 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:46:50.196 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.179925+0000", 2024-04-23T16:46:50.196 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7199999999999997e-07 2024-04-23T16:46:50.196 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.197 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.197 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:46:50.197 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.179958+0000", 2024-04-23T16:46:50.197 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.243e-05 2024-04-23T16:46:50.197 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.197 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.197 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:46:50.197 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.179966+0000", 2024-04-23T16:46:50.197 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8089999999999993e-06 2024-04-23T16:46:50.197 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.197 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.197 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:46:50.197 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.179972+0000", 2024-04-23T16:46:50.197 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5060000000000001e-06 2024-04-23T16:46:50.197 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.198 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.198 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:46:50.198 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.180020+0000", 2024-04-23T16:46:50.198 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8182e-05 2024-04-23T16:46:50.198 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.198 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.198 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:46:50.198 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.180117+0000", 2024-04-23T16:46:50.198 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6478999999999996e-05 2024-04-23T16:46:50.198 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:46:50.198 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:46:50.198 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:46:50.198 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:50.180210+0000", 2024-04-23T16:46:50.198 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3200000000000002e-05 2024-04-23T16:46:50.198 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:50.199 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:46:50.199 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:50.199 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:46:50.199 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-04-23T16:46:50.199 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 4 2024-04-23T16:46:50.199 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:46:50.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:49 smithi188 ceph-mon[44101]: pgmap v1731: 27 pgs: 27 active+clean; 2.2 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 215 MiB/s wr, 3.43k op/s 2024-04-23T16:46:50.420 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:46:50.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:50.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:46:50.525 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:46:50.563 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:50.606 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:50.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:50.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:50.711 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:46:50.759 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:50.840 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:50.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:50.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:50.896 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-23T16:46:50.945 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:46:50.987 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:51.037 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:51.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:51.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:51.127 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:46:51.140 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:46:51.144 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 14 16 47664 47648 212.661 155.438 0.00682911 0.00470006 2024-04-23T16:46:51.155 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:46:51.178 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:51.186 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:46:51.208 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:46:51.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:51.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:46:51.217 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:46:51.243 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:46:51.245 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:46:51.269 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:46:51.291 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:46:51.312 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:46:51.318 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:46:51.353 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:46:51.373 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:51.375 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:46:51.402 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:46:51.402 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:46:51.402 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:46:51.402 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:46:51.402 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-23T16:46:51.402 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:46:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:46:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:46:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:46:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:46:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:46:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:46:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:46:51.475 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:46:51.600 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:46:51 smithi103 bash[268322]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T16:46:51.600 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:46:51 smithi103 bash[268322]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-23T16:46:51.600 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:46:51 smithi103 bash[268322]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-23T16:46:51.600 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:46:51 smithi103 bash[268322]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-23T16:46:51.600 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:46:51 smithi103 bash[268322]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-23T16:46:51.600 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:46:51 smithi103 bash[268322]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-23T16:46:51.600 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:46:51 smithi103 bash[268322]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-23T16:46:51.600 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:46:51 smithi103 bash[268322]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-23T16:46:51.738 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:46:51.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:51.742 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:46:51.843 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:46:51.882 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:51.913 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:46:51.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:51.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:46:52.018 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:46:52.051 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:52.094 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:52.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:52.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:51 smithi148 ceph-mon[44520]: pgmap v1732: 27 pgs: 27 active+clean; 2.4 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 216 MiB/s wr, 3.46k op/s 2024-04-23T16:46:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:51 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/81244419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:46:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:52.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:52.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:52.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:52.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:52.144 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 15 16 49839 49823 207.546 135.938 0.00346788 0.00481586 2024-04-23T16:46:52.198 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:46:52.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:51 smithi188 ceph-mon[44101]: pgmap v1732: 27 pgs: 27 active+clean; 2.4 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 216 MiB/s wr, 3.46k op/s 2024-04-23T16:46:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:51 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/81244419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:46:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:52.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:52.226 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:51 smithi103 ceph-mon[39629]: pgmap v1732: 27 pgs: 27 active+clean; 2.4 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 216 MiB/s wr, 3.46k op/s 2024-04-23T16:46:52.226 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:51 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/81244419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:46:52.226 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:52.226 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:52.226 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:52.226 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:52.227 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:52.227 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:52.229 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:52.354 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:52.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:52.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:52.458 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:46:52.488 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:46:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:46:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:46:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:46:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:46:53.146 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 16 16 52063 52047 203.243 139 0.00831659 0.00491827 2024-04-23T16:46:53.173 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:46:53.173 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:46:53.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:46:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:46:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:46:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:46:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:46:53.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:53.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:53.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:53.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:53.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:46:53.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:46:53.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:46:53.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:46:53.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:46:53.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:46:53.881 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:46:53.881 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:46:53.953 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-23T16:46:54.039 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:46:54.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:54.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:46:54.066 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:54.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:53 smithi103 ceph-mon[39629]: pgmap v1733: 27 pgs: 27 active+clean; 2.9 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 188 MiB/s wr, 3.02k op/s 2024-04-23T16:46:54.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:53 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:46:54.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:53 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:46:54.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:53 smithi103 ceph-mon[39629]: osdmap e615: 12 total, 11 up, 9 in 2024-04-23T16:46:54.146 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 17 16 54421 54405 199.957 147.375 0.00769879 0.00499939 2024-04-23T16:46:54.146 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:46:54.192 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:54.224 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:46:54.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:54.229 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:46:54.304 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:46:54.330 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:46:54.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:53 smithi148 ceph-mon[44520]: pgmap v1733: 27 pgs: 27 active+clean; 2.9 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 188 MiB/s wr, 3.02k op/s 2024-04-23T16:46:54.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:53 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:46:54.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:53 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:46:54.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:53 smithi148 ceph-mon[44520]: osdmap e615: 12 total, 11 up, 9 in 2024-04-23T16:46:54.367 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:53 smithi188 ceph-mon[44101]: pgmap v1733: 27 pgs: 27 active+clean; 2.9 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 188 MiB/s wr, 3.02k op/s 2024-04-23T16:46:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:53 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:46:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:53 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:46:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:53 smithi188 ceph-mon[44101]: osdmap e615: 12 total, 11 up, 9 in 2024-04-23T16:46:54.575 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:46:54.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:54.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:46:54.680 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:46:54.721 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:54.939 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-23T16:46:54.974 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:46:54.976 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:46:54 smithi103 bash[269046]: bd66260900de617b534ba06e0032c2c1b56f67ffa490903e66b5780292e79629 2024-04-23T16:46:54.976 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:46:54 smithi103 systemd[1]: Started Ceph osd.3 for 8e3594f2-018b-11ef-bc93-c7b262605968. 2024-04-23T16:46:54.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:54.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:46:55.078 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:46:55.123 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:55.146 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 18 15 56672 56657 196.667 140.75 0.00743639 0.00508302 2024-04-23T16:46:55.174 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:46:55.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:55.177 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:46:55.278 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:46:55.330 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:55.361 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:46:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:55.366 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:46:55.466 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:46:55.518 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:55.543 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:46:55.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:55.551 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:46:55.652 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:46:55.698 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:55.727 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:46:55.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:55.733 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:46:55.834 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:46:55.845 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:46:55 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-3[269072]: 2024-04-23T16:46:55.385+0000 7f6c0433c7c0 -1 Falling back to public interface 2024-04-23T16:46:55.882 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:46:55.891 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:55.913 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:46:55.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:55.918 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:46:55.940 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:56.019 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:46:56.073 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:56.098 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:46:56.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:55 smithi103 ceph-mon[39629]: pgmap v1735: 27 pgs: 27 active+clean; 3.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-04-23T16:46:56.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:56.102 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:46:56.146 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 19 16 58831 58815 193.414 134.875 0.00409372 0.00516872 2024-04-23T16:46:56.203 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:46:56.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:55 smithi188 ceph-mon[44101]: pgmap v1735: 27 pgs: 27 active+clean; 3.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-04-23T16:46:56.253 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:56.282 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:46:56.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:56.287 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:46:56.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:55 smithi148 ceph-mon[44520]: pgmap v1735: 27 pgs: 27 active+clean; 3.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-04-23T16:46:56.388 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:46:56.427 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:56.458 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:46:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:56.463 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:46:56.564 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:46:56.600 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:57.146 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:46:57.146525 min lat: 0.00111068 max lat: 0.10535 avg lat: 0.00524743 2024-04-23T16:46:57.146 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:46:57.146 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 20 16 60996 60980 190.509 135.312 0.00535672 0.00524743 2024-04-23T16:46:58.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:58 smithi148 ceph-mon[44520]: pgmap v1736: 27 pgs: 27 active+clean; 3.4 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 157 MiB/s wr, 2.51k op/s 2024-04-23T16:46:58.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:58 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3343362044' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:46:58.146 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 21 16 62836 62820 186.914 115 0.0111845 0.00534808 2024-04-23T16:46:58.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:58 smithi103 ceph-mon[39629]: pgmap v1736: 27 pgs: 27 active+clean; 3.4 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 157 MiB/s wr, 2.51k op/s 2024-04-23T16:46:58.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:58 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3343362044' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:46:58.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:58 smithi188 ceph-mon[44101]: pgmap v1736: 27 pgs: 27 active+clean; 3.4 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 157 MiB/s wr, 2.51k op/s 2024-04-23T16:46:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:58 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3343362044' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:46:58.474 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:46:58.475 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:46:58.515 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:46:58.754 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:46:58.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:58.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:46:58.849 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:46:58 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-3[269072]: 2024-04-23T16:46:58.507+0000 7f6c0433c7c0 -1 osd.3 559 log_to_monitors true 2024-04-23T16:46:58.860 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:46:58.893 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:58.936 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:46:58.936 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:46:58.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:58.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:46:59.042 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:46:59.076 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:59.146 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 22 16 64700 64684 183.713 116.5 0.00246122 0.00543597 2024-04-23T16:46:59.280 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:46:59.282 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:59 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:46:59.282 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:59 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3343362044' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:46:59.282 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:59 smithi148 ceph-mon[44520]: osdmap e616: 12 total, 11 up, 9 in 2024-04-23T16:46:59.282 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:46:59 smithi148 ceph-mon[44520]: from='osd.3 v1:172.21.15.103:6813/995549078' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T16:46:59.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:59.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:46:59.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:59 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:46:59.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:59 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3343362044' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:46:59.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:59 smithi103 ceph-mon[39629]: osdmap e616: 12 total, 11 up, 9 in 2024-04-23T16:46:59.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:46:59 smithi103 ceph-mon[39629]: from='osd.3 v1:172.21.15.103:6813/995549078' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T16:46:59.384 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:46:59.417 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:59 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:46:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:59 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3343362044' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:46:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:59 smithi188 ceph-mon[44101]: osdmap e616: 12 total, 11 up, 9 in 2024-04-23T16:46:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:46:59 smithi188 ceph-mon[44101]: from='osd.3 v1:172.21.15.103:6813/995549078' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T16:46:59.770 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:46:59.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:59.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:46:59.814 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-23T16:46:59.832 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 16:46:59 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-3[269072]: 2024-04-23T16:46:59.548+0000 7f6bf178c700 -1 osd.3 618 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:46:59.879 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:46:59.928 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:46:59.954 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:46:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:46:59.958 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:47:00.061 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:47:00.095 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:00 smithi103 ceph-mon[39629]: pgmap v1738: 27 pgs: 27 active+clean; 3.7 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 165 MiB/s wr, 2.64k op/s 2024-04-23T16:47:00.095 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:00 smithi103 ceph-mon[39629]: from='osd.3 v1:172.21.15.103:6813/995549078' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T16:47:00.096 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:00 smithi103 ceph-mon[39629]: osdmap e617: 12 total, 11 up, 9 in 2024-04-23T16:47:00.096 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:00 smithi103 ceph-mon[39629]: from='osd.3 v1:172.21.15.103:6813/995549078' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:47:00.096 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:00 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:47:00.096 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:00 smithi103 ceph-mon[39629]: osdmap e618: 12 total, 11 up, 9 in 2024-04-23T16:47:00.096 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:00 smithi103 ceph-mon[39629]: from='osd.3 v1:172.21.15.103:6813/995549078' entity='osd.3' 2024-04-23T16:47:00.098 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:00.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:00 smithi148 ceph-mon[44520]: pgmap v1738: 27 pgs: 27 active+clean; 3.7 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 165 MiB/s wr, 2.64k op/s 2024-04-23T16:47:00.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:00 smithi148 ceph-mon[44520]: from='osd.3 v1:172.21.15.103:6813/995549078' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T16:47:00.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:00 smithi148 ceph-mon[44520]: osdmap e617: 12 total, 11 up, 9 in 2024-04-23T16:47:00.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:00 smithi148 ceph-mon[44520]: from='osd.3 v1:172.21.15.103:6813/995549078' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:47:00.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:00 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:47:00.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:00 smithi148 ceph-mon[44520]: osdmap e618: 12 total, 11 up, 9 in 2024-04-23T16:47:00.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:00 smithi148 ceph-mon[44520]: from='osd.3 v1:172.21.15.103:6813/995549078' entity='osd.3' 2024-04-23T16:47:00.146 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 23 16 66353 66337 180.218 103.312 0.00842313 0.00554712 2024-04-23T16:47:00.212 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:47:00.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:00 smithi188 ceph-mon[44101]: pgmap v1738: 27 pgs: 27 active+clean; 3.7 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 165 MiB/s wr, 2.64k op/s 2024-04-23T16:47:00.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:00 smithi188 ceph-mon[44101]: from='osd.3 v1:172.21.15.103:6813/995549078' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T16:47:00.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:00 smithi188 ceph-mon[44101]: osdmap e617: 12 total, 11 up, 9 in 2024-04-23T16:47:00.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:00 smithi188 ceph-mon[44101]: from='osd.3 v1:172.21.15.103:6813/995549078' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:47:00.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:00 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:47:00.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:00 smithi188 ceph-mon[44101]: osdmap e618: 12 total, 11 up, 9 in 2024-04-23T16:47:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:00 smithi188 ceph-mon[44101]: from='osd.3 v1:172.21.15.103:6813/995549078' entity='osd.3' 2024-04-23T16:47:00.650 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:47:00.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:00.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:47:00.695 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:47:00.695 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:47:00.695 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-23T16:47:00.695 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:47:00.756 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:47:00.796 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:01.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:01 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:47:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:01 smithi148 ceph-mon[44520]: osd.3 v1:172.21.15.103:6813/995549078 boot 2024-04-23T16:47:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:01 smithi148 ceph-mon[44520]: osdmap e619: 12 total, 12 up, 9 in 2024-04-23T16:47:01.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:01 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:47:01.147 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 24 16 68413 68397 178.072 128.75 0.00794301 0.00561396 2024-04-23T16:47:01.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:01 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:47:01.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:01 smithi103 ceph-mon[39629]: osd.3 v1:172.21.15.103:6813/995549078 boot 2024-04-23T16:47:01.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:01 smithi103 ceph-mon[39629]: osdmap e619: 12 total, 12 up, 9 in 2024-04-23T16:47:01.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:01 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:47:01.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:01 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:47:01.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:01 smithi188 ceph-mon[44101]: osd.3 v1:172.21.15.103:6813/995549078 boot 2024-04-23T16:47:01.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:01 smithi188 ceph-mon[44101]: osdmap e619: 12 total, 12 up, 9 in 2024-04-23T16:47:01.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:01 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T16:47:01.500 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:47:01.552 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:47:01.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:01.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:47:01.657 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:47:01.688 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:02.147 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 25 16 70470 70454 176.092 128.562 0.00663938 0.00567686 2024-04-23T16:47:02.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:02.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:02.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:02 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:47:02.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:02 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:47:02.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:02 smithi103 ceph-mon[39629]: pgmap v1742: 27 pgs: 27 active+clean; 3.8 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-23T16:47:02.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:02 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:47:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:02 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:47:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:02 smithi148 ceph-mon[44520]: pgmap v1742: 27 pgs: 27 active+clean; 3.8 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-23T16:47:02.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:02.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:02.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:02 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:47:02.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:02 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:47:02.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:02 smithi188 ceph-mon[44101]: pgmap v1742: 27 pgs: 27 active+clean; 3.8 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-23T16:47:02.747 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:47:02.747 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:47:03.035 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:03 smithi148 ceph-mon[44520]: osdmap e620: 12 total, 12 up, 9 in 2024-04-23T16:47:03.138 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:47:03.138 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 4, 5, 6, 7, 8, 9, 11, 1] 2024-04-23T16:47:03.139 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd out 5 2024-04-23T16:47:03.147 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 26 16 72477 72461 174.143 125.438 0.00911523 0.0057414 2024-04-23T16:47:03.336 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:03 smithi103 ceph-mon[39629]: osdmap e620: 12 total, 12 up, 9 in 2024-04-23T16:47:03.341 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:47:03.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:03.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:47:03.446 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:47:03.447 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-23T16:47:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:03 smithi188 ceph-mon[44101]: osdmap e620: 12 total, 12 up, 9 in 2024-04-23T16:47:03.482 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:03.568 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:47:03.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:03.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:47:03.673 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:47:03.722 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:03.727 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:47:03.733 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:03.737 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:03.752 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:47:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:03.758 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:47:03.758 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:47:03.787 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:47:03.789 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:47:03.814 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:47:03.837 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:47:03.858 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:47:03.859 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:47:03.884 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:47:03.897 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:03.908 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:47:03.934 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:47:03.966 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:47:03.992 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:47:04.019 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:47:04.021 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:47:04.021 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:47:04.021 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:47:04.021 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:47:04.022 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:47:04.022 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:47:04.022 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:47:04.022 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:47:04.022 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:47:04.022 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:47:04.022 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:47:04.022 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:47:04.054 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:47:04.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:04.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:47:04.147 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 27 15 74311 74296 171.941 114.688 0.00432354 0.00581382 2024-04-23T16:47:04.159 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:47:04.196 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:04 smithi148 ceph-mon[44520]: pgmap v1744: 27 pgs: 27 active+clean; 4.1 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-04-23T16:47:04.196 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:04 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2951517105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:04.196 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:04.196 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:04.196 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:04 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:04.200 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:04.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:04 smithi103 ceph-mon[39629]: pgmap v1744: 27 pgs: 27 active+clean; 4.1 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-04-23T16:47:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:04 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2951517105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:04 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:04.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:04 smithi188 ceph-mon[44101]: pgmap v1744: 27 pgs: 27 active+clean; 4.1 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-04-23T16:47:04.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:04 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2951517105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:04.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:04.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:04.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:04 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:47:04.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:04.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:47:04.768 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:47:04.799 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:05.059 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:47:05.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:05.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:47:05.148 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 28 16 76220 76204 170.056 119.25 0.00359698 0.00587865 2024-04-23T16:47:05.163 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:47:05.191 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:05.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:05 smithi103 ceph-mon[39629]: osdmap e621: 12 total, 12 up, 9 in 2024-04-23T16:47:05.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.376 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:05 smithi148 ceph-mon[44520]: osdmap e621: 12 total, 12 up, 9 in 2024-04-23T16:47:05.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:05.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:05 smithi188 ceph-mon[44101]: osdmap e621: 12 total, 12 up, 9 in 2024-04-23T16:47:05.748 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:47:05.748 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:47:05.748 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2024-04-23T16:47:05.748 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2024-04-23T16:47:05.748 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:47:06.142 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:47:06.147 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 29 16 78102 78086 168.248 117.625 0.00534234 0.00594222 2024-04-23T16:47:06.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:06.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:47:06.249 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:47:06.304 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:06 smithi188 ceph-mon[44101]: pgmap v1746: 27 pgs: 27 active+clean; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-04-23T16:47:06.307 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:06.335 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:47:06.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:06.339 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:47:06.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:06 smithi103 ceph-mon[39629]: pgmap v1746: 27 pgs: 27 active+clean; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-04-23T16:47:06.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:06 smithi148 ceph-mon[44520]: pgmap v1746: 27 pgs: 27 active+clean; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-04-23T16:47:06.439 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:47:06.485 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:06.497 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-23T16:47:06.515 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:47:06.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:06.519 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:47:06.620 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:47:06.654 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:06.781 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:47:06.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:06.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:47:06.840 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:47:06.890 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:47:06.932 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:06.963 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:47:06.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:06.968 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:47:07.069 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:47:07.103 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:07.129 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:47:07.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:07.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:47:07.148 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 30 16 79869 79853 166.321 110.438 0.00635278 0.00601038 2024-04-23T16:47:07.232 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:47:07.266 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:07.921 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:47:07.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:07.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:47:08.026 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:47:08.048 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:08 smithi148 ceph-mon[44520]: pgmap v1747: 27 pgs: 27 active+clean; 4.6 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 135 MiB/s wr, 2.16k op/s 2024-04-23T16:47:08.048 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:08 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2441965792' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:47:08.062 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:47:08.063 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:08.097 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:47:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:08.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:47:08.148 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 31 16 81493 81477 164.23 101.5 0.00739719 0.0060873 2024-04-23T16:47:08.200 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:47:08.232 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:08 smithi103 ceph-mon[39629]: pgmap v1747: 27 pgs: 27 active+clean; 4.6 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 135 MiB/s wr, 2.16k op/s 2024-04-23T16:47:08.232 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:08 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2441965792' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:47:08.233 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:08 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T16:47:08.229+0000 7f135b1c7700 -1 mon.a@0(leader).osd e622 definitely_dead 0 2024-04-23T16:47:08.265 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:08.314 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:47:08.317 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:08 smithi188 ceph-mon[44101]: pgmap v1747: 27 pgs: 27 active+clean; 4.6 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 135 MiB/s wr, 2.16k op/s 2024-04-23T16:47:08.317 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:08 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2441965792' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:47:08.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:08.321 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:47:08.422 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:47:08.471 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:08.493 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:47:08.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:08.498 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:47:08.599 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:47:08.641 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:08.698 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:47:08.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:08.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:47:08.805 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:47:08.835 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:08.989 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:47:08.990 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2024-04-23T16:47:08.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2024-04-23T16:47:08.990 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:47:08.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.990 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90919 7.2 7:408d099e:::benchmark_data_smithi188_141483_object90918:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:47:08.990 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.952953+0000", 2024-04-23T16:47:08.990 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 241.03285852799999, 2024-04-23T16:47:08.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.40145672100000002, 2024-04-23T16:47:08.990 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:08.990 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:08.990 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2024-04-23T16:47:08.990 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:08.990 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:08.991 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:08.991 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90919 2024-04-23T16:47:08.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.991 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:08.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.991 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:08.991 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.952953+0000", 2024-04-23T16:47:08.991 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:08.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.991 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:08.991 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.952953+0000", 2024-04-23T16:47:08.991 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:08.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:08.992 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.952956+0000", 2024-04-23T16:47:08.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6849999999999999e-06 2024-04-23T16:47:08.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:08.992 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953158+0000", 2024-04-23T16:47:08.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000201986 2024-04-23T16:47:08.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:08.993 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953174+0000", 2024-04-23T16:47:08.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5914e-05 2024-04-23T16:47:08.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.993 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:08.993 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953189+0000", 2024-04-23T16:47:08.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4158999999999999e-05 2024-04-23T16:47:08.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.993 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:08.994 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953687+0000", 2024-04-23T16:47:08.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00049814399999999995 2024-04-23T16:47:08.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.994 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:47:08.994 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953710+0000", 2024-04-23T16:47:08.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3178999999999999e-05 2024-04-23T16:47:08.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.994 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:08.994 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.343767+0000", 2024-04-23T16:47:08.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.39005708300000003 2024-04-23T16:47:08.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:47:08.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.344502+0000", 2024-04-23T16:47:08.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00073475700000000005 2024-04-23T16:47:08.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:08.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.352972+0000", 2024-04-23T16:47:08.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0084703320000000006 2024-04-23T16:47:08.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:08.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.354409+0000", 2024-04-23T16:47:08.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0014374819999999999 2024-04-23T16:47:08.996 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:08.996 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:08.996 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:08.996 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.996 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.996 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90920 7.2 7:45ca754c:::benchmark_data_smithi188_141483_object90919:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:47:08.996 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.953221+0000", 2024-04-23T16:47:08.996 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 241.03259041300001, 2024-04-23T16:47:08.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.40592639699999999, 2024-04-23T16:47:08.996 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:08.996 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:08.996 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2024-04-23T16:47:08.996 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:08.997 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:08.997 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:08.997 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90920 2024-04-23T16:47:08.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.997 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:08.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.997 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:08.997 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953221+0000", 2024-04-23T16:47:08.997 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:08.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.997 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:08.997 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953221+0000", 2024-04-23T16:47:08.997 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:08.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.998 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:08.998 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953222+0000", 2024-04-23T16:47:08.998 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9199999999999999e-07 2024-04-23T16:47:08.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.998 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:08.998 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953284+0000", 2024-04-23T16:47:08.998 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1965999999999996e-05 2024-04-23T16:47:08.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.998 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:08.998 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953291+0000", 2024-04-23T16:47:08.998 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4739999999999999e-06 2024-04-23T16:47:08.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.999 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:08.999 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953297+0000", 2024-04-23T16:47:08.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8390000000000003e-06 2024-04-23T16:47:08.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.999 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:08.999 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953757+0000", 2024-04-23T16:47:08.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000460381 2024-04-23T16:47:08.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:08.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:08.999 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:47:08.999 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953776+0000", 2024-04-23T16:47:08.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8097999999999999e-05 2024-04-23T16:47:09.000 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.000 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.000 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.347099+0000", 2024-04-23T16:47:09.000 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.39332325699999998 2024-04-23T16:47:09.000 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.000 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:47:09.000 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.347593+0000", 2024-04-23T16:47:09.000 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00049373400000000001 2024-04-23T16:47:09.000 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.000 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.000 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.359068+0000", 2024-04-23T16:47:09.000 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011475378 2024-04-23T16:47:09.001 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.001 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.001 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:09.001 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.359147+0000", 2024-04-23T16:47:09.001 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.9377999999999998e-05 2024-04-23T16:47:09.001 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.001 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.001 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.001 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.001 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.001 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90926 7.2 7:42975870:::benchmark_data_smithi188_141483_object90925:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:47:09.001 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.953305+0000", 2024-04-23T16:47:09.001 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 241.032506213, 2024-04-23T16:47:09.001 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.40994392299999999, 2024-04-23T16:47:09.001 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:09.002 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:09.002 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2024-04-23T16:47:09.002 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:09.002 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:09.002 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:09.002 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90926 2024-04-23T16:47:09.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.002 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:09.002 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.002 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:09.002 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953305+0000", 2024-04-23T16:47:09.002 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.002 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:09.003 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953305+0000", 2024-04-23T16:47:09.003 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.003 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:09.003 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953306+0000", 2024-04-23T16:47:09.003 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6899999999999997e-07 2024-04-23T16:47:09.003 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:09.003 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953354+0000", 2024-04-23T16:47:09.003 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8729000000000001e-05 2024-04-23T16:47:09.003 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.004 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:09.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953361+0000", 2024-04-23T16:47:09.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2539999999999998e-06 2024-04-23T16:47:09.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.004 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:09.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953366+0000", 2024-04-23T16:47:09.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1839999999999998e-06 2024-04-23T16:47:09.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953840+0000", 2024-04-23T16:47:09.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00047402499999999999 2024-04-23T16:47:09.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:47:09.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953848+0000", 2024-04-23T16:47:09.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7069999999999992e-06 2024-04-23T16:47:09.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.347658+0000", 2024-04-23T16:47:09.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.393810098 2024-04-23T16:47:09.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.006 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.006 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:47:09.006 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.348210+0000", 2024-04-23T16:47:09.006 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00055245100000000003 2024-04-23T16:47:09.006 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.006 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.006 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.006 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.363193+0000", 2024-04-23T16:47:09.006 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014983273 2024-04-23T16:47:09.006 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.006 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.006 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:09.006 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.363249+0000", 2024-04-23T16:47:09.007 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5633000000000001e-05 2024-04-23T16:47:09.007 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.007 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.007 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.007 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.007 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90929 7.2 7:4dfdd413:::benchmark_data_smithi188_141483_object90928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:47:09.007 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.953372+0000", 2024-04-23T16:47:09.007 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 241.032438824, 2024-04-23T16:47:09.007 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.46898921199999999, 2024-04-23T16:47:09.007 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:09.007 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:09.007 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2024-04-23T16:47:09.007 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:09.008 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:09.008 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:09.008 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90929 2024-04-23T16:47:09.008 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.008 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:09.008 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.008 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:09.008 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953372+0000", 2024-04-23T16:47:09.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.008 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.008 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.008 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:09.008 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953372+0000", 2024-04-23T16:47:09.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.009 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.009 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.009 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:09.009 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953373+0000", 2024-04-23T16:47:09.009 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8100000000000003e-07 2024-04-23T16:47:09.009 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.009 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.009 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:09.009 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953486+0000", 2024-04-23T16:47:09.009 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000113242 2024-04-23T16:47:09.009 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.009 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.009 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:09.009 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953496+0000", 2024-04-23T16:47:09.010 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6800000000000005e-06 2024-04-23T16:47:09.010 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.010 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:09.010 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953504+0000", 2024-04-23T16:47:09.010 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1939999999999998e-06 2024-04-23T16:47:09.010 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.010 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.010 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953881+0000", 2024-04-23T16:47:09.010 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00037709000000000001 2024-04-23T16:47:09.010 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.011 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:47:09.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953888+0000", 2024-04-23T16:47:09.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0539999999999997e-06 2024-04-23T16:47:09.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.011 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.011 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.365555+0000", 2024-04-23T16:47:09.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.41166677200000001 2024-04-23T16:47:09.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.011 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.011 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:47:09.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.366044+0000", 2024-04-23T16:47:09.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00048844999999999997 2024-04-23T16:47:09.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.012 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.422260+0000", 2024-04-23T16:47:09.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.056216751000000002 2024-04-23T16:47:09.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:09.012 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.422362+0000", 2024-04-23T16:47:09.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010139799999999999 2024-04-23T16:47:09.012 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.012 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.012 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.012 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90941 7.2 7:492610cd:::benchmark_data_smithi188_141483_object90940:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:47:09.013 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.953517+0000", 2024-04-23T16:47:09.013 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 241.03229387499999, 2024-04-23T16:47:09.013 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.48637596500000002, 2024-04-23T16:47:09.013 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:09.013 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:09.013 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:47:09.013 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:09.013 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:09.013 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:09.013 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90941 2024-04-23T16:47:09.013 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.013 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:09.013 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.013 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:09.014 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953517+0000", 2024-04-23T16:47:09.014 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.014 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.014 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.014 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:09.014 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953517+0000", 2024-04-23T16:47:09.014 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.014 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.014 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.014 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:09.014 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953518+0000", 2024-04-23T16:47:09.014 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.33e-07 2024-04-23T16:47:09.014 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.014 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.015 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:09.015 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953567+0000", 2024-04-23T16:47:09.015 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9076000000000002e-05 2024-04-23T16:47:09.015 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.015 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.015 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:09.015 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953573+0000", 2024-04-23T16:47:09.015 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7939999999999999e-06 2024-04-23T16:47:09.015 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.015 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.015 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:09.015 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953577+0000", 2024-04-23T16:47:09.015 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.053e-06 2024-04-23T16:47:09.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.016 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.016 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953919+0000", 2024-04-23T16:47:09.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00034173100000000001 2024-04-23T16:47:09.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.016 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:47:09.016 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953926+0000", 2024-04-23T16:47:09.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.058e-06 2024-04-23T16:47:09.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.016 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.017 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.366082+0000", 2024-04-23T16:47:09.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.412156409 2024-04-23T16:47:09.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.017 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:47:09.017 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.366789+0000", 2024-04-23T16:47:09.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000706377 2024-04-23T16:47:09.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.017 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:47:09.017 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.366970+0000", 2024-04-23T16:47:09.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00018143900000000001 2024-04-23T16:47:09.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.018 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:47:09.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.376052+0000", 2024-04-23T16:47:09.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0090815070000000008 2024-04-23T16:47:09.018 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.018 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:47:09.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.436876+0000", 2024-04-23T16:47:09.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.06082427 2024-04-23T16:47:09.018 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.018 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:47:09.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.436922+0000", 2024-04-23T16:47:09.019 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6088999999999999e-05 2024-04-23T16:47:09.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.019 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.019 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:09.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.439893+0000", 2024-04-23T16:47:09.019 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0029712290000000001 2024-04-23T16:47:09.019 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.019 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.019 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.019 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.019 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90944 7.2 7:45278a20:::benchmark_data_smithi188_141483_object90943:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:47:09.019 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.953583+0000", 2024-04-23T16:47:09.019 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 241.032228041, 2024-04-23T16:47:09.020 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.48914311599999999, 2024-04-23T16:47:09.020 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:09.020 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:09.020 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:47:09.020 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:09.020 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:09.020 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:09.020 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90944 2024-04-23T16:47:09.020 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.020 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:09.020 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.020 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:09.020 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953583+0000", 2024-04-23T16:47:09.021 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.021 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.021 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.021 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:09.021 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953583+0000", 2024-04-23T16:47:09.021 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.021 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.021 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.021 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:09.021 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953584+0000", 2024-04-23T16:47:09.021 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5200000000000002e-07 2024-04-23T16:47:09.021 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.021 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.021 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:09.022 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953666+0000", 2024-04-23T16:47:09.022 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2119000000000004e-05 2024-04-23T16:47:09.022 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.022 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.022 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:09.022 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953675+0000", 2024-04-23T16:47:09.022 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5510000000000005e-06 2024-04-23T16:47:09.022 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.022 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.022 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:09.022 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953683+0000", 2024-04-23T16:47:09.022 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0399999999999993e-06 2024-04-23T16:47:09.022 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.023 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.023 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953951+0000", 2024-04-23T16:47:09.023 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00026800499999999998 2024-04-23T16:47:09.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.023 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:47:09.023 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953958+0000", 2024-04-23T16:47:09.023 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2559999999999999e-06 2024-04-23T16:47:09.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.023 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.023 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.369565+0000", 2024-04-23T16:47:09.024 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.41560729000000002 2024-04-23T16:47:09.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.024 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:47:09.024 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.369774+0000", 2024-04-23T16:47:09.024 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00020933599999999999 2024-04-23T16:47:09.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.024 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:47:09.024 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.369961+0000", 2024-04-23T16:47:09.024 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0001872 2024-04-23T16:47:09.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.025 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:47:09.025 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.378828+0000", 2024-04-23T16:47:09.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0088667429999999998 2024-04-23T16:47:09.025 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.025 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:47:09.025 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.439955+0000", 2024-04-23T16:47:09.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.061126416000000003 2024-04-23T16:47:09.025 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.025 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:47:09.025 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.440015+0000", 2024-04-23T16:47:09.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0238000000000002e-05 2024-04-23T16:47:09.025 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.026 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:09.026 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.442726+0000", 2024-04-23T16:47:09.026 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0027114700000000001 2024-04-23T16:47:09.026 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.026 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.026 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.026 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.026 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90951 7.2 7:4e3203d8:::benchmark_data_smithi188_141483_object90950:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:47:09.026 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.953695+0000", 2024-04-23T16:47:09.026 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 241.03211657200001, 2024-04-23T16:47:09.026 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.49130570800000001, 2024-04-23T16:47:09.026 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:09.027 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:09.027 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:47:09.027 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:09.027 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:09.027 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:09.027 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90951 2024-04-23T16:47:09.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.027 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:09.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:09.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953695+0000", 2024-04-23T16:47:09.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:09.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953695+0000", 2024-04-23T16:47:09.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.028 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.028 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:09.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953696+0000", 2024-04-23T16:47:09.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5899999999999995e-07 2024-04-23T16:47:09.028 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.028 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:09.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953747+0000", 2024-04-23T16:47:09.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1547e-05 2024-04-23T16:47:09.028 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.029 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:09.029 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953753+0000", 2024-04-23T16:47:09.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6659999999999996e-06 2024-04-23T16:47:09.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.029 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:09.029 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953757+0000", 2024-04-23T16:47:09.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9269999999999998e-06 2024-04-23T16:47:09.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.029 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.029 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953980+0000", 2024-04-23T16:47:09.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000223773 2024-04-23T16:47:09.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:47:09.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953984+0000", 2024-04-23T16:47:09.030 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6380000000000002e-06 2024-04-23T16:47:09.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.372732+0000", 2024-04-23T16:47:09.030 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.41874784799999998 2024-04-23T16:47:09.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:47:09.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.372875+0000", 2024-04-23T16:47:09.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000143273 2024-04-23T16:47:09.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.031 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.031 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:47:09.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.373022+0000", 2024-04-23T16:47:09.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014640899999999999 2024-04-23T16:47:09.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.031 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.031 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:47:09.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.381656+0000", 2024-04-23T16:47:09.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0086342160000000001 2024-04-23T16:47:09.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.032 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:47:09.032 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.442769+0000", 2024-04-23T16:47:09.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.061113155000000002 2024-04-23T16:47:09.032 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.032 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:47:09.032 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.442803+0000", 2024-04-23T16:47:09.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3815000000000001e-05 2024-04-23T16:47:09.032 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.032 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:09.032 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.445000+0000", 2024-04-23T16:47:09.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002197582 2024-04-23T16:47:09.033 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.033 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.033 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.033 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90965 7.2 7:403ee5c1:::benchmark_data_smithi188_141483_object90964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:47:09.033 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.953762+0000", 2024-04-23T16:47:09.033 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 241.03204891300001, 2024-04-23T16:47:09.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.49351197600000002, 2024-04-23T16:47:09.033 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:09.033 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:09.033 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:47:09.033 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:09.033 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:09.033 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:09.034 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90965 2024-04-23T16:47:09.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.034 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:09.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.034 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:09.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953762+0000", 2024-04-23T16:47:09.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.034 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:09.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953762+0000", 2024-04-23T16:47:09.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:09.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953763+0000", 2024-04-23T16:47:09.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1699999999999998e-07 2024-04-23T16:47:09.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:09.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953881+0000", 2024-04-23T16:47:09.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000118267 2024-04-23T16:47:09.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:09.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953890+0000", 2024-04-23T16:47:09.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7649999999999999e-06 2024-04-23T16:47:09.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:09.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953896+0000", 2024-04-23T16:47:09.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5699999999999998e-06 2024-04-23T16:47:09.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954051+0000", 2024-04-23T16:47:09.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000154623 2024-04-23T16:47:09.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:47:09.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954060+0000", 2024-04-23T16:47:09.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9299999999999992e-06 2024-04-23T16:47:09.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.037 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.376075+0000", 2024-04-23T16:47:09.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.42201528399999999 2024-04-23T16:47:09.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:47:09.037 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.376223+0000", 2024-04-23T16:47:09.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000147606 2024-04-23T16:47:09.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:47:09.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.376331+0000", 2024-04-23T16:47:09.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107767 2024-04-23T16:47:09.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:47:09.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.388216+0000", 2024-04-23T16:47:09.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011885269 2024-04-23T16:47:09.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:47:09.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.445040+0000", 2024-04-23T16:47:09.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.056823783000000003 2024-04-23T16:47:09.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:47:09.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.445094+0000", 2024-04-23T16:47:09.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4716999999999998e-05 2024-04-23T16:47:09.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:09.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.447274+0000", 2024-04-23T16:47:09.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021798780000000001 2024-04-23T16:47:09.039 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.039 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.039 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.039 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90945 7.c 7:3e4f21cd:::benchmark_data_smithi188_141483_object90944:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:47:09.040 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.954061+0000", 2024-04-23T16:47:09.040 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 241.03174997900001, 2024-04-23T16:47:09.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.18484241200000001, 2024-04-23T16:47:09.040 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:09.040 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:09.040 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:47:09.040 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:09.040 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:09.040 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:09.040 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90945 2024-04-23T16:47:09.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.040 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:09.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:09.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954061+0000", 2024-04-23T16:47:09.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:09.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954061+0000", 2024-04-23T16:47:09.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:09.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954062+0000", 2024-04-23T16:47:09.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9999999999999997e-07 2024-04-23T16:47:09.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:09.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954123+0000", 2024-04-23T16:47:09.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1469000000000004e-05 2024-04-23T16:47:09.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:09.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954130+0000", 2024-04-23T16:47:09.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3879999999999997e-06 2024-04-23T16:47:09.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:09.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954136+0000", 2024-04-23T16:47:09.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7370000000000003e-06 2024-04-23T16:47:09.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.043 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955319+0000", 2024-04-23T16:47:09.043 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00118315 2024-04-23T16:47:09.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.043 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:47:09.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955326+0000", 2024-04-23T16:47:09.043 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9249999999999998e-06 2024-04-23T16:47:09.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.043 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.119609+0000", 2024-04-23T16:47:09.043 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.16428289800000001 2024-04-23T16:47:09.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:47:09.044 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.119782+0000", 2024-04-23T16:47:09.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000173711 2024-04-23T16:47:09.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:47:09.044 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.119997+0000", 2024-04-23T16:47:09.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000214489 2024-04-23T16:47:09.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:47:09.044 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.130219+0000", 2024-04-23T16:47:09.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01022229 2024-04-23T16:47:09.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:47:09.045 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.135874+0000", 2024-04-23T16:47:09.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0056549449999999998 2024-04-23T16:47:09.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:47:09.045 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.135936+0000", 2024-04-23T16:47:09.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2121999999999999e-05 2024-04-23T16:47:09.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:09.045 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.138904+0000", 2024-04-23T16:47:09.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0029675880000000002 2024-04-23T16:47:09.045 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.046 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.046 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.046 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90962 7.c 7:32187531:::benchmark_data_smithi188_141483_object90961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:47:09.046 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.954146+0000", 2024-04-23T16:47:09.046 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 241.03166526000001, 2024-04-23T16:47:09.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.188198055, 2024-04-23T16:47:09.046 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:09.046 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:09.046 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:47:09.046 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:09.046 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:09.046 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:09.047 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90962 2024-04-23T16:47:09.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.047 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:09.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:09.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954146+0000", 2024-04-23T16:47:09.047 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:09.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954146+0000", 2024-04-23T16:47:09.047 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:09.048 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954147+0000", 2024-04-23T16:47:09.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9999999999999997e-07 2024-04-23T16:47:09.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.048 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.048 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:09.048 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954221+0000", 2024-04-23T16:47:09.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3998000000000005e-05 2024-04-23T16:47:09.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.048 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.048 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:09.048 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954226+0000", 2024-04-23T16:47:09.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5280000000000003e-06 2024-04-23T16:47:09.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.048 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.049 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:09.049 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954232+0000", 2024-04-23T16:47:09.049 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0669999999999997e-06 2024-04-23T16:47:09.049 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.049 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.049 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.049 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955364+0000", 2024-04-23T16:47:09.049 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0011320880000000001 2024-04-23T16:47:09.049 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.049 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.049 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:47:09.049 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955371+0000", 2024-04-23T16:47:09.049 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0829999999999998e-06 2024-04-23T16:47:09.049 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.049 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.050 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.122427+0000", 2024-04-23T16:47:09.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.16705524799999999 2024-04-23T16:47:09.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:47:09.050 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.122536+0000", 2024-04-23T16:47:09.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000109279 2024-04-23T16:47:09.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:47:09.050 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.122631+0000", 2024-04-23T16:47:09.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.4969000000000005e-05 2024-04-23T16:47:09.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:47:09.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.130227+0000", 2024-04-23T16:47:09.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0075958079999999999 2024-04-23T16:47:09.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:47:09.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.138964+0000", 2024-04-23T16:47:09.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0087376569999999994 2024-04-23T16:47:09.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:47:09.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.139020+0000", 2024-04-23T16:47:09.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5680000000000002e-05 2024-04-23T16:47:09.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.052 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:09.052 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.142344+0000", 2024-04-23T16:47:09.052 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00332395 2024-04-23T16:47:09.052 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.052 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.052 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.052 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.052 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90969 7.c 7:35415fb1:::benchmark_data_smithi188_141483_object90968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:47:09.052 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.954241+0000", 2024-04-23T16:47:09.052 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 241.03157052099999, 2024-04-23T16:47:09.052 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19155655399999999, 2024-04-23T16:47:09.052 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:09.053 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:09.053 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:47:09.053 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:09.053 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:09.053 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:09.053 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90969 2024-04-23T16:47:09.053 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.053 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:09.053 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.053 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:09.053 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954241+0000", 2024-04-23T16:47:09.053 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.053 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.053 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.054 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:09.054 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954241+0000", 2024-04-23T16:47:09.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.054 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:09.054 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954241+0000", 2024-04-23T16:47:09.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8599999999999998e-07 2024-04-23T16:47:09.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.054 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:09.054 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954327+0000", 2024-04-23T16:47:09.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5081999999999996e-05 2024-04-23T16:47:09.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:09.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954332+0000", 2024-04-23T16:47:09.055 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2870000000000004e-06 2024-04-23T16:47:09.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:09.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954335+0000", 2024-04-23T16:47:09.055 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6729999999999998e-06 2024-04-23T16:47:09.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955414+0000", 2024-04-23T16:47:09.055 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001078332 2024-04-23T16:47:09.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.056 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.056 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:47:09.056 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955420+0000", 2024-04-23T16:47:09.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1650000000000003e-06 2024-04-23T16:47:09.056 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.056 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.056 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.056 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.127524+0000", 2024-04-23T16:47:09.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.17210402 2024-04-23T16:47:09.056 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.056 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.056 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:47:09.056 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.127638+0000", 2024-04-23T16:47:09.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000113719 2024-04-23T16:47:09.056 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.057 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.057 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:47:09.057 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.127806+0000", 2024-04-23T16:47:09.057 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000168102 2024-04-23T16:47:09.057 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.057 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.057 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:47:09.057 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.132574+0000", 2024-04-23T16:47:09.057 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0047678249999999998 2024-04-23T16:47:09.057 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.057 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.057 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:47:09.057 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.142402+0000", 2024-04-23T16:47:09.058 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0098287519999999996 2024-04-23T16:47:09.058 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.058 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.058 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:47:09.058 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.142463+0000", 2024-04-23T16:47:09.058 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0618999999999997e-05 2024-04-23T16:47:09.058 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.058 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.058 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:09.058 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.145797+0000", 2024-04-23T16:47:09.058 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003334292 2024-04-23T16:47:09.058 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.058 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.058 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.059 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.059 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90972 7.c 7:3f3cd386:::benchmark_data_smithi188_141483_object90971:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:47:09.059 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.954341+0000", 2024-04-23T16:47:09.059 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 241.031470154, 2024-04-23T16:47:09.059 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19740545100000001, 2024-04-23T16:47:09.059 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:09.059 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:09.059 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:47:09.059 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:09.059 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:09.059 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:09.059 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90972 2024-04-23T16:47:09.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.059 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:09.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.060 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:09.060 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954341+0000", 2024-04-23T16:47:09.060 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.060 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:09.060 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954341+0000", 2024-04-23T16:47:09.060 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.060 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:09.060 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954342+0000", 2024-04-23T16:47:09.060 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9999999999999998e-07 2024-04-23T16:47:09.061 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.061 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.061 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:09.061 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954472+0000", 2024-04-23T16:47:09.061 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013074399999999999 2024-04-23T16:47:09.061 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.061 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.061 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:09.061 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954478+0000", 2024-04-23T16:47:09.061 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2499999999999997e-06 2024-04-23T16:47:09.061 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.061 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.061 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:09.061 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954482+0000", 2024-04-23T16:47:09.061 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9879999999999998e-06 2024-04-23T16:47:09.062 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.062 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.062 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.062 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955457+0000", 2024-04-23T16:47:09.062 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00097560500000000005 2024-04-23T16:47:09.062 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.062 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.062 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:47:09.062 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955465+0000", 2024-04-23T16:47:09.062 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7340000000000008e-06 2024-04-23T16:47:09.062 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.062 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.062 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.062 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.130257+0000", 2024-04-23T16:47:09.063 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.17479178100000001 2024-04-23T16:47:09.063 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.063 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.063 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:47:09.063 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.130385+0000", 2024-04-23T16:47:09.063 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012878499999999999 2024-04-23T16:47:09.063 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.063 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.063 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:47:09.063 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.130499+0000", 2024-04-23T16:47:09.063 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011311000000000001 2024-04-23T16:47:09.063 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.063 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.063 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:47:09.064 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.132767+0000", 2024-04-23T16:47:09.064 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0022680090000000001 2024-04-23T16:47:09.064 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.064 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.064 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:47:09.064 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.148612+0000", 2024-04-23T16:47:09.064 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015844982000000001 2024-04-23T16:47:09.064 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.064 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.064 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:47:09.064 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.148675+0000", 2024-04-23T16:47:09.064 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3193000000000003e-05 2024-04-23T16:47:09.064 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.064 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.064 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:09.065 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.151747+0000", 2024-04-23T16:47:09.065 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00307187 2024-04-23T16:47:09.065 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.065 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.065 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.065 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.065 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.065 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90988 7.2 7:41f48dd3:::benchmark_data_smithi188_141483_object90987:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-23T16:47:09.065 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:08.087481+0000", 2024-04-23T16:47:09.065 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 240.89833057999999, 2024-04-23T16:47:09.065 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.36209360899999998, 2024-04-23T16:47:09.065 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:09.065 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:09.066 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:47:09.066 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:09.066 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:09.066 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:09.066 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90988 2024-04-23T16:47:09.066 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.066 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:09.066 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.066 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:09.067 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.087481+0000", 2024-04-23T16:47:09.067 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.067 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.067 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.067 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:09.067 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.087481+0000", 2024-04-23T16:47:09.067 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.067 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.067 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.067 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:09.067 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.087483+0000", 2024-04-23T16:47:09.067 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.826e-06 2024-04-23T16:47:09.067 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.067 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.067 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:09.068 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.087532+0000", 2024-04-23T16:47:09.068 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9818999999999999e-05 2024-04-23T16:47:09.068 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.068 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.068 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:09.068 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.087543+0000", 2024-04-23T16:47:09.068 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0331e-05 2024-04-23T16:47:09.068 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.068 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.068 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:09.068 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.087548+0000", 2024-04-23T16:47:09.068 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7509999999999997e-06 2024-04-23T16:47:09.068 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.068 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.068 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.069 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.087601+0000", 2024-04-23T16:47:09.069 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2181000000000001e-05 2024-04-23T16:47:09.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.069 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.069 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:47:09.069 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.087607+0000", 2024-04-23T16:47:09.069 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4489999999999996e-06 2024-04-23T16:47:09.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.069 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.069 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.069 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.378860+0000", 2024-04-23T16:47:09.069 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.291253137 2024-04-23T16:47:09.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.069 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.070 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:47:09.070 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.378976+0000", 2024-04-23T16:47:09.070 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000115454 2024-04-23T16:47:09.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.070 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.070 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:47:09.070 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.379112+0000", 2024-04-23T16:47:09.070 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000136668 2024-04-23T16:47:09.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.070 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.070 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:47:09.070 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.395111+0000", 2024-04-23T16:47:09.070 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015998366999999999 2024-04-23T16:47:09.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.070 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:47:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.447324+0000", 2024-04-23T16:47:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052213183000000003 2024-04-23T16:47:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:47:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.447379+0000", 2024-04-23T16:47:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5078999999999999e-05 2024-04-23T16:47:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:09.071 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.449574+0000", 2024-04-23T16:47:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002195364 2024-04-23T16:47:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90993 7.2 7:44f8e9e0:::benchmark_data_smithi188_141483_object90992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-23T16:47:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:08.138947+0000", 2024-04-23T16:47:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 240.84686444100001, 2024-04-23T16:47:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.31274553900000002, 2024-04-23T16:47:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:47:09.072 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90993 2024-04-23T16:47:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.138947+0000", 2024-04-23T16:47:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.138947+0000", 2024-04-23T16:47:09.073 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.138948+0000", 2024-04-23T16:47:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5659999999999999e-06 2024-04-23T16:47:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.139005+0000", 2024-04-23T16:47:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6983e-05 2024-04-23T16:47:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.074 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.139014+0000", 2024-04-23T16:47:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2709999999999992e-06 2024-04-23T16:47:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.139020+0000", 2024-04-23T16:47:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.93e-06 2024-04-23T16:47:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.139080+0000", 2024-04-23T16:47:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0785000000000002e-05 2024-04-23T16:47:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:47:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.139092+0000", 2024-04-23T16:47:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1473e-05 2024-04-23T16:47:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.381704+0000", 2024-04-23T16:47:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.24261231799999999 2024-04-23T16:47:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:47:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.381868+0000", 2024-04-23T16:47:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00016402100000000001 2024-04-23T16:47:09.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:47:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.382051+0000", 2024-04-23T16:47:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00018281499999999999 2024-04-23T16:47:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:47:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.395119+0000", 2024-04-23T16:47:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01306789 2024-04-23T16:47:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:47:09.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.449634+0000", 2024-04-23T16:47:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.054515317000000001 2024-04-23T16:47:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:47:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.449684+0000", 2024-04-23T16:47:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9802000000000002e-05 2024-04-23T16:47:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.451692+0000", 2024-04-23T16:47:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002008368 2024-04-23T16:47:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.078 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.079 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.079 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.079 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.081 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:91001 7.2 7:483e89c4:::benchmark_data_smithi188_141483_object91000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-23T16:47:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:08.150731+0000", 2024-04-23T16:47:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 240.835080267, 2024-04-23T16:47:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.30912154400000003, 2024-04-23T16:47:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:47:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 91001 2024-04-23T16:47:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.082 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.150731+0000", 2024-04-23T16:47:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.150731+0000", 2024-04-23T16:47:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:09.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.150734+0000", 2024-04-23T16:47:09.084 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6520000000000002e-06 2024-04-23T16:47:09.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.084 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:09.084 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.150789+0000", 2024-04-23T16:47:09.084 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5597999999999999e-05 2024-04-23T16:47:09.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.084 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:09.084 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.150803+0000", 2024-04-23T16:47:09.084 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3278000000000001e-05 2024-04-23T16:47:09.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.085 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:09.085 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.150811+0000", 2024-04-23T16:47:09.085 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5920000000000001e-06 2024-04-23T16:47:09.085 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.085 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.085 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.085 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.150881+0000", 2024-04-23T16:47:09.085 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0037999999999996e-05 2024-04-23T16:47:09.085 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.085 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.085 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:47:09.085 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.150893+0000", 2024-04-23T16:47:09.085 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1942e-05 2024-04-23T16:47:09.085 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.086 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.086 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.086 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.385185+0000", 2024-04-23T16:47:09.086 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.234292324 2024-04-23T16:47:09.086 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.086 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.086 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:47:09.086 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.385327+0000", 2024-04-23T16:47:09.086 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014171499999999999 2024-04-23T16:47:09.086 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.086 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.086 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:47:09.086 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.385502+0000", 2024-04-23T16:47:09.086 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000174778 2024-04-23T16:47:09.087 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.087 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.087 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:47:09.087 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.401156+0000", 2024-04-23T16:47:09.087 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015653923 2024-04-23T16:47:09.087 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.087 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.087 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:47:09.087 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.456669+0000", 2024-04-23T16:47:09.087 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.055513593999999999 2024-04-23T16:47:09.087 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.087 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.087 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:47:09.087 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.456727+0000", 2024-04-23T16:47:09.088 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7475000000000001e-05 2024-04-23T16:47:09.088 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.088 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.088 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:09.088 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.459853+0000", 2024-04-23T16:47:09.088 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0031256349999999999 2024-04-23T16:47:09.088 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.088 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.088 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.088 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.088 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.088 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:91022 7.2 7:46f7851e:::benchmark_data_smithi188_141483_object91021:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-23T16:47:09.089 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:08.215689+0000", 2024-04-23T16:47:09.089 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 240.77012264699999, 2024-04-23T16:47:09.089 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.24714624399999999, 2024-04-23T16:47:09.089 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:09.089 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:09.089 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:47:09.089 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:09.089 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:09.089 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:09.089 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 91022 2024-04-23T16:47:09.089 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.089 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:09.089 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.089 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:09.090 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.215689+0000", 2024-04-23T16:47:09.090 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.090 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.090 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.090 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:09.090 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.215689+0000", 2024-04-23T16:47:09.090 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.090 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.090 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.090 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:09.090 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.215691+0000", 2024-04-23T16:47:09.090 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6240000000000002e-06 2024-04-23T16:47:09.090 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.090 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.091 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:09.091 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.215745+0000", 2024-04-23T16:47:09.091 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4035e-05 2024-04-23T16:47:09.091 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.091 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.091 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:09.091 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.215759+0000", 2024-04-23T16:47:09.091 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3451999999999999e-05 2024-04-23T16:47:09.091 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.091 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.091 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:09.091 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.215769+0000", 2024-04-23T16:47:09.091 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9690000000000006e-06 2024-04-23T16:47:09.092 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.092 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.092 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.092 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.215833+0000", 2024-04-23T16:47:09.092 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3972000000000003e-05 2024-04-23T16:47:09.092 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.092 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.092 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:47:09.092 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.215844+0000", 2024-04-23T16:47:09.092 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0875e-05 2024-04-23T16:47:09.092 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.092 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.092 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.093 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.388269+0000", 2024-04-23T16:47:09.093 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.17242563999999999 2024-04-23T16:47:09.093 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.093 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.093 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:47:09.093 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.388427+0000", 2024-04-23T16:47:09.093 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015798799999999999 2024-04-23T16:47:09.093 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.093 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.093 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:47:09.093 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.388609+0000", 2024-04-23T16:47:09.093 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00018175600000000001 2024-04-23T16:47:09.093 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.094 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.094 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:47:09.094 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.401160+0000", 2024-04-23T16:47:09.094 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012551306 2024-04-23T16:47:09.094 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.094 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.094 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:47:09.094 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.459916+0000", 2024-04-23T16:47:09.094 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.058755325999999997 2024-04-23T16:47:09.094 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.095 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.095 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:47:09.095 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.459969+0000", 2024-04-23T16:47:09.095 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3513000000000001e-05 2024-04-23T16:47:09.095 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.095 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.095 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:09.095 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.462835+0000", 2024-04-23T16:47:09.095 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0028657880000000002 2024-04-23T16:47:09.095 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.095 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.095 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.095 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.095 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.095 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:91042 7.2 7:4e7ae021:::benchmark_data_smithi188_141483_object91041:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-23T16:47:09.096 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:08.279824+0000", 2024-04-23T16:47:09.096 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 240.70598755200001, 2024-04-23T16:47:09.096 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.185784803, 2024-04-23T16:47:09.096 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:09.096 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:09.096 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:47:09.096 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:09.096 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:09.096 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:09.096 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 91042 2024-04-23T16:47:09.096 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.096 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:09.097 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.097 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:09.097 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.279824+0000", 2024-04-23T16:47:09.097 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.097 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.097 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.097 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:09.097 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.279824+0000", 2024-04-23T16:47:09.097 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.097 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.097 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.097 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:09.097 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.279826+0000", 2024-04-23T16:47:09.097 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4760000000000001e-06 2024-04-23T16:47:09.098 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.098 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.098 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:09.098 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.279880+0000", 2024-04-23T16:47:09.098 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4113000000000002e-05 2024-04-23T16:47:09.098 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.098 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.098 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:09.098 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.279892+0000", 2024-04-23T16:47:09.098 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1477999999999999e-05 2024-04-23T16:47:09.098 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.098 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.099 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:09.099 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.279900+0000", 2024-04-23T16:47:09.099 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.055e-06 2024-04-23T16:47:09.099 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.099 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.099 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.099 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.279955+0000", 2024-04-23T16:47:09.099 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5603999999999998e-05 2024-04-23T16:47:09.099 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.099 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.099 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:47:09.099 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.279962+0000", 2024-04-23T16:47:09.099 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0010000000000004e-06 2024-04-23T16:47:09.099 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.100 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.100 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.100 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.391777+0000", 2024-04-23T16:47:09.100 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.11181430000000001 2024-04-23T16:47:09.100 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.100 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.100 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:47:09.100 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.391914+0000", 2024-04-23T16:47:09.100 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000137553 2024-04-23T16:47:09.100 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.100 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.100 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:47:09.100 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.392022+0000", 2024-04-23T16:47:09.100 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107668 2024-04-23T16:47:09.100 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.101 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.101 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:47:09.101 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.403995+0000", 2024-04-23T16:47:09.101 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011972554999999999 2024-04-23T16:47:09.101 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.101 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.101 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:47:09.101 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.462906+0000", 2024-04-23T16:47:09.101 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.058911223999999998 2024-04-23T16:47:09.101 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.101 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.101 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:47:09.101 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.462955+0000", 2024-04-23T16:47:09.101 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9363999999999997e-05 2024-04-23T16:47:09.102 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.102 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.102 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:09.102 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.465609+0000", 2024-04-23T16:47:09.102 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002653412 2024-04-23T16:47:09.102 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.102 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.102 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.102 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.102 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.102 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:212497 7.d 7:bd34efcb:::benchmark_data_smithi188_141483_object93816:head [delete] snapc 0=[] ondisk+write+known_if_redirected e595)", 2024-04-23T16:47:09.103 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:45:35.717520+0000", 2024-04-23T16:47:09.103 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 93.268291493999996, 2024-04-23T16:47:09.103 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.211223097, 2024-04-23T16:47:09.103 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:09.103 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:09.103 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:47:09.103 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:09.103 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:09.103 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:09.103 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 212497 2024-04-23T16:47:09.103 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.103 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:09.103 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.104 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:09.104 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.717520+0000", 2024-04-23T16:47:09.104 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.104 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.104 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.104 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:09.104 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.717520+0000", 2024-04-23T16:47:09.104 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.104 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.104 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.104 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:09.104 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.717521+0000", 2024-04-23T16:47:09.104 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.187e-06 2024-04-23T16:47:09.104 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.105 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.105 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:09.105 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.717525+0000", 2024-04-23T16:47:09.105 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1779999999999996e-06 2024-04-23T16:47:09.105 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.105 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.105 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:09.105 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.717535+0000", 2024-04-23T16:47:09.105 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5449999999999992e-06 2024-04-23T16:47:09.105 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.105 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.105 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:09.105 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.717543+0000", 2024-04-23T16:47:09.105 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8849999999999999e-06 2024-04-23T16:47:09.106 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.106 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.106 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.106 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.717592+0000", 2024-04-23T16:47:09.106 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9314000000000003e-05 2024-04-23T16:47:09.106 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.106 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.106 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:47:09.106 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.717669+0000", 2024-04-23T16:47:09.106 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6867999999999997e-05 2024-04-23T16:47:09.106 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.106 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.106 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 1", 2024-04-23T16:47:09.107 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.717737+0000", 2024-04-23T16:47:09.107 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8164999999999995e-05 2024-04-23T16:47:09.107 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.107 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.107 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:47:09.107 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.721609+0000", 2024-04-23T16:47:09.107 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0038720389999999999 2024-04-23T16:47:09.107 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.107 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.107 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:47:09.107 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.928636+0000", 2024-04-23T16:47:09.107 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20702727300000001 2024-04-23T16:47:09.108 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.108 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.108 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:47:09.108 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.928693+0000", 2024-04-23T16:47:09.108 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6764000000000001e-05 2024-04-23T16:47:09.108 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.108 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.108 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:09.108 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.928743+0000", 2024-04-23T16:47:09.108 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9879000000000001e-05 2024-04-23T16:47:09.108 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.108 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.108 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.108 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.109 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.109 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:212519 7.d 7:bb623d9c:::benchmark_data_smithi188_141483_object93838:head [delete] snapc 0=[] ondisk+write+known_if_redirected e595)", 2024-04-23T16:47:09.109 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:45:35.744077+0000", 2024-04-23T16:47:09.109 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 93.241734336999997, 2024-04-23T16:47:09.109 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19108017899999999, 2024-04-23T16:47:09.109 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:09.109 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:09.109 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:47:09.109 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:09.109 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:09.109 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:09.109 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 212519 2024-04-23T16:47:09.109 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.110 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:09.110 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.110 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:09.110 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744077+0000", 2024-04-23T16:47:09.110 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.110 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.110 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.110 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:09.110 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744077+0000", 2024-04-23T16:47:09.110 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.110 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.110 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.110 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:09.110 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744078+0000", 2024-04-23T16:47:09.111 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9999999999999995e-07 2024-04-23T16:47:09.111 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.111 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.111 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:09.111 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744082+0000", 2024-04-23T16:47:09.111 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.929e-06 2024-04-23T16:47:09.111 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.111 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.111 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:09.111 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744089+0000", 2024-04-23T16:47:09.111 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6149999999999999e-06 2024-04-23T16:47:09.111 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.111 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.111 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:09.112 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744095+0000", 2024-04-23T16:47:09.112 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1159999999999998e-06 2024-04-23T16:47:09.112 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.112 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.112 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.112 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744135+0000", 2024-04-23T16:47:09.112 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0763000000000003e-05 2024-04-23T16:47:09.112 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.112 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.112 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:47:09.112 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744229+0000", 2024-04-23T16:47:09.112 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.4123000000000007e-05 2024-04-23T16:47:09.113 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.113 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.113 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 1", 2024-04-23T16:47:09.113 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744297+0000", 2024-04-23T16:47:09.113 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7549e-05 2024-04-23T16:47:09.113 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.113 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.113 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:47:09.113 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.753199+0000", 2024-04-23T16:47:09.113 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0089017230000000003 2024-04-23T16:47:09.113 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.113 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.113 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:47:09.113 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.935049+0000", 2024-04-23T16:47:09.113 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.18185061399999999 2024-04-23T16:47:09.114 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.114 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.114 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:47:09.114 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.935109+0000", 2024-04-23T16:47:09.114 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9227000000000001e-05 2024-04-23T16:47:09.114 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.114 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.114 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:09.114 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.935157+0000", 2024-04-23T16:47:09.114 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8520000000000003e-05 2024-04-23T16:47:09.114 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.114 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.114 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.114 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.115 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.115 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:212520 7.d 7:b66cbaa7:::benchmark_data_smithi188_141483_object93839:head [delete] snapc 0=[] ondisk+write+known_if_redirected e595)", 2024-04-23T16:47:09.115 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:45:35.744178+0000", 2024-04-23T16:47:09.115 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 93.241632834000001, 2024-04-23T16:47:09.115 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.191085055, 2024-04-23T16:47:09.115 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:47:09.115 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:47:09.115 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:47:09.115 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:47:09.115 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:47:09.115 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:47:09.115 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 212520 2024-04-23T16:47:09.115 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.116 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:47:09.116 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.116 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:47:09.116 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744178+0000", 2024-04-23T16:47:09.116 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.116 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.116 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.116 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:47:09.116 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744178+0000", 2024-04-23T16:47:09.116 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:47:09.116 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.116 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.116 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:47:09.116 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744179+0000", 2024-04-23T16:47:09.117 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5900000000000005e-07 2024-04-23T16:47:09.117 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.117 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.117 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:47:09.117 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744183+0000", 2024-04-23T16:47:09.117 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8739999999999997e-06 2024-04-23T16:47:09.117 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.117 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.117 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:47:09.117 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744192+0000", 2024-04-23T16:47:09.117 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.541e-06 2024-04-23T16:47:09.117 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.117 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.118 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:47:09.118 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744198+0000", 2024-04-23T16:47:09.118 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1600000000000003e-06 2024-04-23T16:47:09.118 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.118 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.118 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:47:09.118 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.748787+0000", 2024-04-23T16:47:09.118 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045890490000000004 2024-04-23T16:47:09.118 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.118 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.118 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:47:09.118 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.748917+0000", 2024-04-23T16:47:09.118 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012969500000000001 2024-04-23T16:47:09.118 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.119 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.119 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 1", 2024-04-23T16:47:09.119 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.749037+0000", 2024-04-23T16:47:09.119 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000120419 2024-04-23T16:47:09.119 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.119 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.119 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:47:09.119 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.753343+0000", 2024-04-23T16:47:09.119 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0043057740000000001 2024-04-23T16:47:09.119 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.119 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.119 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:47:09.119 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.935194+0000", 2024-04-23T16:47:09.119 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.18185153000000001 2024-04-23T16:47:09.120 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.120 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.120 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:47:09.120 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.935226+0000", 2024-04-23T16:47:09.120 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1449999999999999e-05 2024-04-23T16:47:09.120 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:47:09.120 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:47:09.120 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:47:09.120 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.935264+0000", 2024-04-23T16:47:09.120 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7604000000000001e-05 2024-04-23T16:47:09.120 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.120 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.120 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.120 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:47:09.120 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:47:09.121 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:47:09.121 INFO:teuthology.orchestra.run.smithi103.stderr:marked out osd.5. 2024-04-23T16:47:09.122 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:47:09.122 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:09 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:47:09.122 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:09 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2441965792' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:47:09.122 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:09 smithi103 ceph-mon[39629]: osdmap e622: 12 total, 12 up, 9 in 2024-04-23T16:47:09.122 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:09 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:47:09.122 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:09 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:47:09.122 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:09 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3478378459' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-04-23T16:47:09.122 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:09 smithi103 ceph-mon[39629]: Client client.admin marked osd.5 out, while it was still marked up 2024-04-23T16:47:09.122 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:09 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:47:09.122 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:09 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:47:09.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:09.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:47:09.148 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 32 16 83103 83087 162.242 100.625 0.0031031 0.00615076 2024-04-23T16:47:09.224 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:47:09.235 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:47:09.250 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:09.269 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:47:09.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:09.272 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:47:09.373 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:47:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:09 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:47:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:09 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2441965792' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:47:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:09 smithi148 ceph-mon[44520]: osdmap e622: 12 total, 12 up, 9 in 2024-04-23T16:47:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:09 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:47:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:09 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:47:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:09 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3478378459' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-04-23T16:47:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:09 smithi148 ceph-mon[44520]: Client client.admin marked osd.5 out, while it was still marked up 2024-04-23T16:47:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:09 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:47:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:09 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:47:09.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:09 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:47:09.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:09 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2441965792' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:47:09.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:09 smithi188 ceph-mon[44101]: osdmap e622: 12 total, 12 up, 9 in 2024-04-23T16:47:09.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:09 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:47:09.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:09 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:47:09.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:09 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3478378459' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-04-23T16:47:09.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:09 smithi188 ceph-mon[44101]: Client client.admin marked osd.5 out, while it was still marked up 2024-04-23T16:47:09.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:09 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:47:09.400 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:09 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:47:09.400 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:09.420 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:47:09.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:09.422 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:47:09.523 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:47:09.552 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:09.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:09.902 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:47:09.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:47:10.004 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:47:10.031 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:10.155 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 33 16 83103 83087 157.294 0 - 0.00615076 2024-04-23T16:47:10.268 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:10 smithi103 ceph-mon[39629]: pgmap v1749: 27 pgs: 27 active+clean; 4.8 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-04-23T16:47:10.268 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:10 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3478378459' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-04-23T16:47:10.268 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:10 smithi103 ceph-mon[39629]: osdmap e623: 12 total, 12 up, 8 in 2024-04-23T16:47:10.279 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:47:10.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:10.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:47:10.304 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:10.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:10 smithi148 ceph-mon[44520]: pgmap v1749: 27 pgs: 27 active+clean; 4.8 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-04-23T16:47:10.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:10 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3478378459' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-04-23T16:47:10.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:10 smithi148 ceph-mon[44520]: osdmap e623: 12 total, 12 up, 8 in 2024-04-23T16:47:10.383 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:47:10.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:10 smithi188 ceph-mon[44101]: pgmap v1749: 27 pgs: 27 active+clean; 4.8 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-04-23T16:47:10.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:10 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3478378459' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-04-23T16:47:10.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:10 smithi188 ceph-mon[44101]: osdmap e623: 12 total, 12 up, 8 in 2024-04-23T16:47:10.433 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:10.459 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:47:10.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:10.464 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:47:10.565 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:47:10.595 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:10.615 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:47:10.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:10.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:47:10.719 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:47:10.753 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:10.808 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:47:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:10.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:47:10.913 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:47:10.967 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:11.007 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:47:11.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:11.008 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:47:11.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:11 smithi103 ceph-mon[39629]: 1.4 scrub starts 2024-04-23T16:47:11.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:11 smithi103 ceph-mon[39629]: 1.4 scrub ok 2024-04-23T16:47:11.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:11 smithi103 ceph-mon[39629]: osdmap e624: 12 total, 12 up, 8 in 2024-04-23T16:47:11.109 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:47:11.124 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-23T16:47:11.139 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:11.158 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 34 16 83765 83749 153.873 20.6875 0.019094 0.00649581 2024-04-23T16:47:11.162 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:47:11.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:11.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:47:11.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:11 smithi188 ceph-mon[44101]: 1.4 scrub starts 2024-04-23T16:47:11.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:11 smithi188 ceph-mon[44101]: 1.4 scrub ok 2024-04-23T16:47:11.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:11 smithi188 ceph-mon[44101]: osdmap e624: 12 total, 12 up, 8 in 2024-04-23T16:47:11.265 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:47:11.294 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:11.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:11 smithi148 ceph-mon[44520]: 1.4 scrub starts 2024-04-23T16:47:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:11 smithi148 ceph-mon[44520]: 1.4 scrub ok 2024-04-23T16:47:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:11 smithi148 ceph-mon[44520]: osdmap e624: 12 total, 12 up, 8 in 2024-04-23T16:47:11.430 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:47:11.544 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:47:11.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:11.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:47:11.648 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:47:11.676 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:11.829 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:47:11.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:11.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:47:11.883 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:47:11.933 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:47:12.005 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:12.048 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:47:12.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:12.051 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:47:12.086 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:12 smithi103 ceph-mon[39629]: pgmap v1752: 27 pgs: 27 active+clean; 4.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 95 MiB/s wr, 1.51k op/s 2024-04-23T16:47:12.148 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:47:12.152 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:47:12.158 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 35 16 84569 84553 150.913 50.25 0.00993131 0.00662363 2024-04-23T16:47:12.176 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:47:12.200 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:47:12.200 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:12.222 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:47:12.230 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:12.244 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:47:12.261 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:47:12.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:12.265 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:47:12.265 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:12 smithi188 ceph-mon[44101]: pgmap v1752: 27 pgs: 27 active+clean; 4.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 95 MiB/s wr, 1.51k op/s 2024-04-23T16:47:12.267 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:47:12.292 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:47:12.314 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:47:12.339 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:47:12.366 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:47:12.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:12 smithi148 ceph-mon[44520]: pgmap v1752: 27 pgs: 27 active+clean; 4.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 95 MiB/s wr, 1.51k op/s 2024-04-23T16:47:12.375 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:47:12.401 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:12.404 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:47:12.430 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:47:12.432 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:47:12.432 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:47:12.432 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:47:12.432 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:47:12.432 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:47:12.432 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:47:12.432 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:47:12.432 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:47:12.432 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:47:12.432 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:47:12.433 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:47:12.433 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:47:12.448 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:47:12.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:12.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:47:12.553 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:47:12.583 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:12.784 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:47:12.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:12.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:47:12.888 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:47:12.960 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:12.999 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:47:13.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:13.008 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:47:13.099 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:13 smithi148 ceph-mon[44520]: 1.5 scrub starts 2024-04-23T16:47:13.099 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:13 smithi148 ceph-mon[44520]: 1.5 scrub ok 2024-04-23T16:47:13.109 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:47:13.139 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:13.158 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 36 16 85401 85385 148.167 52 0.0123051 0.00674619 2024-04-23T16:47:13.173 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:47:13.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:13.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:47:13.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:13 smithi188 ceph-mon[44101]: 1.5 scrub starts 2024-04-23T16:47:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:13 smithi188 ceph-mon[44101]: 1.5 scrub ok 2024-04-23T16:47:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:13 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4159058746' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:13 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:47:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:13 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:47:13.278 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:47:13.309 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:13.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:13 smithi103 ceph-mon[39629]: 1.5 scrub starts 2024-04-23T16:47:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:13 smithi103 ceph-mon[39629]: 1.5 scrub ok 2024-04-23T16:47:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:13 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4159058746' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:13 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:47:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:13 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:47:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:13 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4159058746' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:13 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:47:13.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:13.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:13 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:47:13.461 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:47:13.511 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:47:13.639 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:47:13.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:13.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:47:13.744 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:47:13.820 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:13.873 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:47:13.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:13.883 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:47:13.984 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:47:14.015 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:14.101 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:14.101 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:14.102 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:14.102 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:14.102 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:14 smithi148 ceph-mon[44520]: pgmap v1753: 27 pgs: 1 active+recovering, 26 active+clean; 5.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s; 5260/165828 objects misplaced (3.172%); 288 KiB/s, 4 objects/s recovering 2024-04-23T16:47:14.102 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:14.103 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:14.103 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:14.103 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:14.103 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:14 smithi188 ceph-mon[44101]: pgmap v1753: 27 pgs: 1 active+recovering, 26 active+clean; 5.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s; 5260/165828 objects misplaced (3.172%); 288 KiB/s, 4 objects/s recovering 2024-04-23T16:47:14.159 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 37 16 86209 86193 145.527 50.5 0.0189974 0.00686892 2024-04-23T16:47:14.171 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-23T16:47:14.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:14.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:14.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:14.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:14.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:14 smithi103 ceph-mon[39629]: pgmap v1753: 27 pgs: 1 active+recovering, 26 active+clean; 5.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s; 5260/165828 objects misplaced (3.172%); 288 KiB/s, 4 objects/s recovering 2024-04-23T16:47:14.216 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:47:14.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:14.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:47:14.322 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:47:14.382 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:14.415 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:47:14.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:14.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:47:14.465 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:47:14.520 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:47:14.548 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:14.748 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:47:14.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:14.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:47:14.853 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:47:14.885 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:14.906 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:47:14.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:14.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:47:15.011 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:47:15.044 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:15.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:15 smithi103 ceph-mon[39629]: 1.9 scrub starts 2024-04-23T16:47:15.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:15 smithi103 ceph-mon[39629]: osdmap e625: 12 total, 12 up, 8 in 2024-04-23T16:47:15.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:15 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:47:15.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:15 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:47:15.104 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:15 smithi103 ceph-mon[39629]: pgmap v1755: 27 pgs: 4 active+recovering, 23 active+clean; 5.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 20398/170272 objects misplaced (11.980%); 3.4 MiB/s, 54 objects/s recovering 2024-04-23T16:47:15.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:15 smithi103 ceph-mon[39629]: from='client.17058 v1:172.21.15.103:0/203122140' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:47:15.158 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 38 16 87034 87018 143.056 51.5625 0.0174515 0.00698718 2024-04-23T16:47:15.305 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 6, 7, 8, 9, 11, 1] out_osds: [10, 2, 3, 5] dead_osds: [] live_osds: [0, 4, 5, 6, 7, 8, 9, 10, 11, 1, 2, 3] 2024-04-23T16:47:15.306 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-23T16:47:15.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:15 smithi103 ceph-mon[39629]: from='client.17058 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:47:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:15 smithi148 ceph-mon[44520]: 1.9 scrub starts 2024-04-23T16:47:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:15 smithi148 ceph-mon[44520]: osdmap e625: 12 total, 12 up, 8 in 2024-04-23T16:47:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:15 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:47:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:15 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:47:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:15 smithi148 ceph-mon[44520]: pgmap v1755: 27 pgs: 4 active+recovering, 23 active+clean; 5.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 20398/170272 objects misplaced (11.980%); 3.4 MiB/s, 54 objects/s recovering 2024-04-23T16:47:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:15 smithi148 ceph-mon[44520]: from='client.17058 v1:172.21.15.103:0/203122140' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:47:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:15 smithi148 ceph-mon[44520]: from='client.17058 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:47:15.411 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:47:15.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:15 smithi188 ceph-mon[44101]: 1.9 scrub starts 2024-04-23T16:47:15.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:15 smithi188 ceph-mon[44101]: osdmap e625: 12 total, 12 up, 8 in 2024-04-23T16:47:15.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:15 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:47:15.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:15 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:47:15.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:15 smithi188 ceph-mon[44101]: pgmap v1755: 27 pgs: 4 active+recovering, 23 active+clean; 5.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 20398/170272 objects misplaced (11.980%); 3.4 MiB/s, 54 objects/s recovering 2024-04-23T16:47:15.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:15 smithi188 ceph-mon[44101]: from='client.17058 v1:172.21.15.103:0/203122140' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:47:15.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:15 smithi188 ceph-mon[44101]: from='client.17058 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:47:15.729 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:47:15.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:15.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:47:15.837 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:47:15.911 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:15.968 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:47:15.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:15.972 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:47:16.073 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:47:16.105 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:16.158 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 39 16 87810 87794 140.632 48.5 0.0119517 0.00710821 2024-04-23T16:47:16.164 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:16.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:16 smithi188 ceph-mon[44101]: 1.9 scrub ok 2024-04-23T16:47:16.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:16 smithi188 ceph-mon[44101]: 1.7 scrub starts 2024-04-23T16:47:16.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:16 smithi188 ceph-mon[44101]: 1.7 scrub ok 2024-04-23T16:47:16.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:16 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:47:16.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:16 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:47:16.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:16 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:47:16.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:16 smithi188 ceph-mon[44101]: from='client.17058 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:47:16.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:16 smithi188 ceph-mon[44101]: osdmap e626: 12 total, 12 up, 8 in 2024-04-23T16:47:16.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:16 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:47:16.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:16 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:47:16.244 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:47:16.330 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:47:16.330 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:47:16.351 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:16 smithi148 ceph-mon[44520]: 1.9 scrub ok 2024-04-23T16:47:16.351 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:16 smithi148 ceph-mon[44520]: 1.7 scrub starts 2024-04-23T16:47:16.352 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:16 smithi148 ceph-mon[44520]: 1.7 scrub ok 2024-04-23T16:47:16.352 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:16 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:47:16.352 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:16 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:47:16.352 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:16 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:47:16.352 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:16 smithi148 ceph-mon[44520]: from='client.17058 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:47:16.352 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:16 smithi148 ceph-mon[44520]: osdmap e626: 12 total, 12 up, 8 in 2024-04-23T16:47:16.352 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:16 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:47:16.353 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:16 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:47:16.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:16 smithi103 ceph-mon[39629]: 1.9 scrub ok 2024-04-23T16:47:16.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:16 smithi103 ceph-mon[39629]: 1.7 scrub starts 2024-04-23T16:47:16.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:16 smithi103 ceph-mon[39629]: 1.7 scrub ok 2024-04-23T16:47:16.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:16 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:47:16.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:16 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:47:16.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:16 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:47:16.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:16 smithi103 ceph-mon[39629]: from='client.17058 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:47:16.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:16 smithi103 ceph-mon[39629]: osdmap e626: 12 total, 12 up, 8 in 2024-04-23T16:47:16.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:16 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:47:16.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:16 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:47:16.355 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:47:16.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:16.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:47:16.463 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:47:16.521 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:16.552 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:47:16.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:16.556 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:47:16.658 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:47:16.758 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:16.793 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:47:16.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:16.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:47:16.898 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:47:16.961 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:17.008 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:47:17.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:17.017 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:47:17.099 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-23T16:47:17.118 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:47:17.158 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:47:17.158706 min lat: 0.00111068 max lat: 1.40272 avg lat: 0.00722515 2024-04-23T16:47:17.158 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:47:17.158 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 40 16 88588 88572 138.332 48.625 0.0136625 0.00722515 2024-04-23T16:47:17.186 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:17.220 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:47:17.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:17.231 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:47:17.332 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:47:17.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:17 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:47:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:17 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:47:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:17 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:47:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:17 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:47:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:17 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:47:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:17 smithi103 ceph-mon[39629]: osdmap e627: 12 total, 12 up, 8 in 2024-04-23T16:47:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:17 smithi103 ceph-mon[39629]: pgmap v1758: 27 pgs: 4 active+recovering, 23 active+clean; 5.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 20260/173842 objects misplaced (11.654%); 4.8 MiB/s, 77 objects/s recovering 2024-04-23T16:47:17.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:17 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:47:17.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:17 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:47:17.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:17 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:47:17.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:17 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:47:17.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:17 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:47:17.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:17 smithi148 ceph-mon[44520]: osdmap e627: 12 total, 12 up, 8 in 2024-04-23T16:47:17.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:17 smithi148 ceph-mon[44520]: pgmap v1758: 27 pgs: 4 active+recovering, 23 active+clean; 5.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 20260/173842 objects misplaced (11.654%); 4.8 MiB/s, 77 objects/s recovering 2024-04-23T16:47:17.367 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:17.393 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:47:17.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:17.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:47:17.404 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:47:17.443 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:17.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:17 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:47:17.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:17 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:47:17.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:17 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:47:17.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:17 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:47:17.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:17 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:47:17.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:17 smithi188 ceph-mon[44101]: osdmap e627: 12 total, 12 up, 8 in 2024-04-23T16:47:17.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:17 smithi188 ceph-mon[44101]: pgmap v1758: 27 pgs: 4 active+recovering, 23 active+clean; 5.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 20260/173842 objects misplaced (11.654%); 4.8 MiB/s, 77 objects/s recovering 2024-04-23T16:47:17.498 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:47:17.525 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:17.553 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:47:17.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:17.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:47:17.661 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:47:17.691 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:17.714 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:47:17.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:17.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:47:17.819 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:47:17.846 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:17.922 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:47:17.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:17.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:47:18.026 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:47:18.077 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:18.110 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:47:18.114 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:47:18.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:18.119 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:47:18.158 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 41 16 89373 89357 136.156 49.0625 0.00468032 0.00734216 2024-04-23T16:47:18.219 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:47:18.248 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:18 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:47:18.248 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:18 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:47:18.248 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:18 smithi148 ceph-mon[44520]: osdmap e628: 12 total, 12 up, 8 in 2024-04-23T16:47:18.251 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:18 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:47:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:18 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:47:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:18 smithi103 ceph-mon[39629]: osdmap e628: 12 total, 12 up, 8 in 2024-04-23T16:47:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:18 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:47:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:18 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:47:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:18 smithi188 ceph-mon[44101]: osdmap e628: 12 total, 12 up, 8 in 2024-04-23T16:47:18.700 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:47:18.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:18.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:47:18.805 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:47:18.837 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:19.165 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 42 16 90180 90164 134.095 50.4375 0.0253367 0.00745466 2024-04-23T16:47:19.288 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:47:19.289 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:47:19.311 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:19 smithi148 ceph-mon[44520]: pgmap v1760: 27 pgs: 4 active+recovering, 23 active+clean; 5.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 54 MiB/s wr, 870 op/s; 19847/176892 objects misplaced (11.220%); 7.6 MiB/s, 121 objects/s recovering 2024-04-23T16:47:19.416 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:19 smithi103 ceph-mon[39629]: pgmap v1760: 27 pgs: 4 active+recovering, 23 active+clean; 5.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 54 MiB/s wr, 870 op/s; 19847/176892 objects misplaced (11.220%); 7.6 MiB/s, 121 objects/s recovering 2024-04-23T16:47:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:19 smithi188 ceph-mon[44101]: pgmap v1760: 27 pgs: 4 active+recovering, 23 active+clean; 5.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 54 MiB/s wr, 870 op/s; 19847/176892 objects misplaced (11.220%); 7.6 MiB/s, 121 objects/s recovering 2024-04-23T16:47:20.068 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.0 to 1 (10000) 2024-04-23T16:47:20.072 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-23T16:47:20.166 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 43 16 90952 90936 132.095 48.25 0.0397677 0.00756529 2024-04-23T16:47:20.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:20 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3333773178' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:47:20.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:20 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3333773178' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:47:20.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:20 smithi103 ceph-mon[39629]: osdmap e629: 12 total, 12 up, 8 in 2024-04-23T16:47:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:20 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3333773178' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:47:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:20 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3333773178' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:47:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:20 smithi148 ceph-mon[44520]: osdmap e629: 12 total, 12 up, 8 in 2024-04-23T16:47:20.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:20 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3333773178' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:47:20.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:20 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3333773178' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:47:20.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:20 smithi188 ceph-mon[44101]: osdmap e629: 12 total, 12 up, 8 in 2024-04-23T16:47:20.539 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:47:20.797 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:47:20.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:20.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:47:20.829 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-23T16:47:20.861 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:20.905 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:47:20.975 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:21.166 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 44 16 91745 91729 130.22 49.5625 0.00525095 0.0076766 2024-04-23T16:47:21.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:21 smithi188 ceph-mon[44101]: pgmap v1762: 27 pgs: 4 active+recovering, 23 active+clean; 5.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 45 MiB/s wr, 725 op/s; 19847/178248 objects misplaced (11.134%); 6.3 MiB/s, 100 objects/s recovering 2024-04-23T16:47:21.378 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:21 smithi103 ceph-mon[39629]: pgmap v1762: 27 pgs: 4 active+recovering, 23 active+clean; 5.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 45 MiB/s wr, 725 op/s; 19847/178248 objects misplaced (11.134%); 6.3 MiB/s, 100 objects/s recovering 2024-04-23T16:47:21.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:21 smithi148 ceph-mon[44520]: pgmap v1762: 27 pgs: 4 active+recovering, 23 active+clean; 5.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 45 MiB/s wr, 725 op/s; 19847/178248 objects misplaced (11.134%); 6.3 MiB/s, 100 objects/s recovering 2024-04-23T16:47:22.167 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 45 16 92557 92541 128.455 50.75 0.0116382 0.0077825 2024-04-23T16:47:22.299 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:22 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2904510865' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:22.299 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:22.299 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:22.299 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:22.299 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:22.299 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:22.299 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:22 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2904510865' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:22.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:22.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:22.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:22.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:22 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2904510865' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:22.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:22.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:22.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:22.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:22.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:22.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:22.663 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:47:22.663 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:47:22.663 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:47:22.663 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:47:22.663 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:47:22.663 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:47:22.663 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:47:22.663 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:47:22.664 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:47:22.664 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:47:22.664 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:47:22.664 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:47:22.664 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:47:22.664 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:47:22.664 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:47:22.664 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:47:22.664 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:47:22.664 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:47:22.664 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:47:22.665 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:47:22.665 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:47:22.665 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:47:22.665 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:47:22.665 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:47:23.169 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 46 16 93382 93366 126.777 51.5625 0.0121421 0.00788468 2024-04-23T16:47:23.378 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:47:23.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:23.383 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:47:23.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:23.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:23.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:23.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:23.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:23.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:23.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:23 smithi188 ceph-mon[44101]: pgmap v1763: 27 pgs: 4 active+recovering, 23 active+clean; 5.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 51 MiB/s wr, 810 op/s; 19719/181806 objects misplaced (10.846%); 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:47:23.483 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:47:23.571 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:23.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:23.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:23.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:23.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:23.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:23.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:23.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:23 smithi103 ceph-mon[39629]: pgmap v1763: 27 pgs: 4 active+recovering, 23 active+clean; 5.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 51 MiB/s wr, 810 op/s; 19719/181806 objects misplaced (10.846%); 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:47:23.612 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:47:23.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:23.617 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:47:23.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:23.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:23.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:23.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:23.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:23.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:23.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:23 smithi148 ceph-mon[44520]: pgmap v1763: 27 pgs: 4 active+recovering, 23 active+clean; 5.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 51 MiB/s wr, 810 op/s; 19719/181806 objects misplaced (10.846%); 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:47:23.686 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:23.718 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:47:23.778 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:23.930 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:47:23.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:23.932 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:47:24.033 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:47:24.066 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:24.160 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2574299247' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:47:24.160 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:24 smithi188 ceph-mon[44101]: from='client.36512 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:47:24.169 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 47 16 94172 94156 125.131 49.375 0.00928805 0.00798885 2024-04-23T16:47:24.170 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:47:24.194 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:47:24.196 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2574299247' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:47:24.196 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:24 smithi103 ceph-mon[39629]: from='client.36512 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:47:24.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:24.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:47:24.298 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:47:24.330 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:24.350 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:47:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:24.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:47:24.455 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:47:24.486 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:24.525 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:47:24.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:24.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:47:24.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:24 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2574299247' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:47:24.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:24 smithi148 ceph-mon[44520]: from='client.36512 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:47:24.629 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:47:24.656 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:24.680 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:47:24.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:24.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:47:24.785 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:47:24.813 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:24.833 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:47:24.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:24.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:47:24.937 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:47:24.965 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:25.166 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:25 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:47:25.167 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:25 smithi103 ceph-mon[39629]: from='client.36512 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:47:25.167 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:25 smithi103 ceph-mon[39629]: osdmap e630: 12 total, 12 up, 8 in 2024-04-23T16:47:25.167 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:25 smithi103 ceph-mon[39629]: osdmap e631: 12 total, 12 up, 8 in 2024-04-23T16:47:25.167 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:25 smithi103 ceph-mon[39629]: pgmap v1766: 27 pgs: 4 active+recovering, 23 active+clean; 5.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 49 MiB/s wr, 784 op/s; 19298/186312 objects misplaced (10.358%); 5.7 MiB/s, 91 objects/s recovering 2024-04-23T16:47:25.172 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 48 16 94930 94914 123.505 47.375 0.0497891 0.00809243 2024-04-23T16:47:25.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:25 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:47:25.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:25 smithi188 ceph-mon[44101]: from='client.36512 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:47:25.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:25 smithi188 ceph-mon[44101]: osdmap e630: 12 total, 12 up, 8 in 2024-04-23T16:47:25.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:25 smithi188 ceph-mon[44101]: osdmap e631: 12 total, 12 up, 8 in 2024-04-23T16:47:25.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:25 smithi188 ceph-mon[44101]: pgmap v1766: 27 pgs: 4 active+recovering, 23 active+clean; 5.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 49 MiB/s wr, 784 op/s; 19298/186312 objects misplaced (10.358%); 5.7 MiB/s, 91 objects/s recovering 2024-04-23T16:47:25.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:25 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:47:25.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:25 smithi148 ceph-mon[44520]: from='client.36512 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:47:25.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:25 smithi148 ceph-mon[44520]: osdmap e630: 12 total, 12 up, 8 in 2024-04-23T16:47:25.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:25 smithi148 ceph-mon[44520]: osdmap e631: 12 total, 12 up, 8 in 2024-04-23T16:47:25.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:25 smithi148 ceph-mon[44520]: pgmap v1766: 27 pgs: 4 active+recovering, 23 active+clean; 5.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 49 MiB/s wr, 784 op/s; 19298/186312 objects misplaced (10.358%); 5.7 MiB/s, 91 objects/s recovering 2024-04-23T16:47:25.715 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:47:25.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:25.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:47:25.724 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:47:25.820 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:47:25.905 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:25.969 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:47:25.969 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:47:25.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.969 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:95467 8.a 8:5d9999f3:::benchmark_data_smithi188_155546_object95466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-23T16:47:25.969 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:47:25.938854+0000", 2024-04-23T16:47:25.969 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.021700746999999999, 2024-04-23T16:47:25.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021731637000000002, 2024-04-23T16:47:25.969 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:47:25.969 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:47:25.970 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:47:25.970 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:47:25.970 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:47:25.970 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:47:25.970 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 95467 2024-04-23T16:47:25.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.970 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:47:25.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:47:25.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.938854+0000", 2024-04-23T16:47:25.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:47:25.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:47:25.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.938854+0000", 2024-04-23T16:47:25.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:47:25.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:47:25.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.938856+0000", 2024-04-23T16:47:25.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4420000000000001e-06 2024-04-23T16:47:25.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:47:25.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.938884+0000", 2024-04-23T16:47:25.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7710000000000001e-05 2024-04-23T16:47:25.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:47:25.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.938903+0000", 2024-04-23T16:47:25.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9476000000000001e-05 2024-04-23T16:47:25.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:47:25.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.938913+0000", 2024-04-23T16:47:25.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0144e-05 2024-04-23T16:47:25.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:47:25.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.939031+0000", 2024-04-23T16:47:25.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000117964 2024-04-23T16:47:25.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:47:25.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.939201+0000", 2024-04-23T16:47:25.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017006500000000001 2024-04-23T16:47:25.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:47:25.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.939348+0000", 2024-04-23T16:47:25.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014692399999999999 2024-04-23T16:47:25.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:47:25.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.946549+0000", 2024-04-23T16:47:25.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0072008150000000002 2024-04-23T16:47:25.974 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:47:25.974 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:47:25.974 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:47:25.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.974 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:95478 8.b 8:dffc3bc9:::benchmark_data_smithi188_155546_object95477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-23T16:47:25.974 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:47:25.952074+0000", 2024-04-23T16:47:25.974 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0084802049999999993, 2024-04-23T16:47:25.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085693090000000006, 2024-04-23T16:47:25.975 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:47:25.975 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:47:25.975 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:47:25.975 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:47:25.975 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:47:25.975 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:47:25.975 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 95478 2024-04-23T16:47:25.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.975 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:47:25.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:47:25.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952074+0000", 2024-04-23T16:47:25.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:47:25.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:47:25.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952074+0000", 2024-04-23T16:47:25.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:47:25.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:47:25.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952078+0000", 2024-04-23T16:47:25.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0890000000000002e-06 2024-04-23T16:47:25.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:47:25.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952126+0000", 2024-04-23T16:47:25.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7678e-05 2024-04-23T16:47:25.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:47:25.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952142+0000", 2024-04-23T16:47:25.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5911000000000001e-05 2024-04-23T16:47:25.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:47:25.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952151+0000", 2024-04-23T16:47:25.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5319999999999995e-06 2024-04-23T16:47:25.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:47:25.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952347+0000", 2024-04-23T16:47:25.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019574699999999999 2024-04-23T16:47:25.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:47:25.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952475+0000", 2024-04-23T16:47:25.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012795499999999999 2024-04-23T16:47:25.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:47:25.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952591+0000", 2024-04-23T16:47:25.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000116068 2024-04-23T16:47:25.979 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:47:25.979 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:47:25.979 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:47:25.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.979 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:95480 8.b 8:d5eade0f:::benchmark_data_smithi188_155546_object95479:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-23T16:47:25.979 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:47:25.952174+0000", 2024-04-23T16:47:25.979 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0083803750000000007, 2024-04-23T16:47:25.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085160380000000001, 2024-04-23T16:47:25.980 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:47:25.980 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:47:25.980 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:47:25.980 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:47:25.980 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:47:25.980 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:47:25.980 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 95480 2024-04-23T16:47:25.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.980 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:47:25.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:47:25.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952174+0000", 2024-04-23T16:47:25.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:47:25.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:47:25.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952174+0000", 2024-04-23T16:47:25.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:47:25.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:47:25.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952175+0000", 2024-04-23T16:47:25.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1900000000000001e-07 2024-04-23T16:47:25.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:47:25.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952208+0000", 2024-04-23T16:47:25.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3283e-05 2024-04-23T16:47:25.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.997 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:47:25.997 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952216+0000", 2024-04-23T16:47:25.997 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8739999999999995e-06 2024-04-23T16:47:25.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.997 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:47:25.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952222+0000", 2024-04-23T16:47:25.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6239999999999999e-06 2024-04-23T16:47:25.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:47:25.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.955727+0000", 2024-04-23T16:47:25.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0035054439999999999 2024-04-23T16:47:25.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:47:25.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.955872+0000", 2024-04-23T16:47:25.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014488200000000001 2024-04-23T16:47:25.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.999 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 4", 2024-04-23T16:47:25.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.956013+0000", 2024-04-23T16:47:25.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014128900000000001 2024-04-23T16:47:25.999 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:47:25.999 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:47:25.999 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:47:25.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:25.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:25.999 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.16995.0:95465 8.c e631/609 8:31762c3f:::benchmark_data_smithi188_155546_object95464:head v 631'5958, mlcod=631'5956)", 2024-04-23T16:47:25.999 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:47:25.952237+0000", 2024-04-23T16:47:26.000 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0083170099999999997, 2024-04-23T16:47:26.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0084976110000000004, 2024-04-23T16:47:26.000 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:47:26.000 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:47:26.000 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-04-23T16:47:26.000 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:47:26.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:26.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:47:26.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952237+0000", 2024-04-23T16:47:26.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:47:26.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:26.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:26.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:47:26.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952237+0000", 2024-04-23T16:47:26.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:47:26.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:26.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:26.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:47:26.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952239+0000", 2024-04-23T16:47:26.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1909999999999999e-06 2024-04-23T16:47:26.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:26.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:26.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:47:26.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952263+0000", 2024-04-23T16:47:26.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4553e-05 2024-04-23T16:47:26.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:26.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:26.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:47:26.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952271+0000", 2024-04-23T16:47:26.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9319999999999996e-06 2024-04-23T16:47:26.003 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:26.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:26.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:47:26.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.952276+0000", 2024-04-23T16:47:26.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.814e-06 2024-04-23T16:47:26.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:26.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:26.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:47:26.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.959825+0000", 2024-04-23T16:47:26.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007549427 2024-04-23T16:47:26.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:47:26.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:47:26.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:47:26.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:25.959850+0000", 2024-04-23T16:47:26.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.461e-05 2024-04-23T16:47:26.004 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:47:26.004 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:47:26.005 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:47:26.005 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:47:26.005 INFO:teuthology.orchestra.run.smithi188.stdout: ], 2024-04-23T16:47:26.005 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 4 2024-04-23T16:47:26.005 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:47:26.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:26.007 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:47:26.007 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:47:26.108 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:47:26.135 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:26.157 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:47:26.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:26.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:47:26.172 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 49 16 95633 95617 121.882 43.9375 0.0075689 0.00820204 2024-04-23T16:47:26.260 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:47:26.287 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:26.306 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:47:26.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:26.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:47:26.315 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:26.410 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:47:26.438 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:26.458 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:47:26.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:26.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:47:26.562 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:47:26.610 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.4 to 1 (10000) 2024-04-23T16:47:26.632 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:26 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2220212617' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-23T16:47:26.662 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:26.725 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:47:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:26.729 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:47:26.762 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-23T16:47:26.830 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:47:26.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:26 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2220212617' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-23T16:47:26.856 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:26 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2220212617' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-23T16:47:26.858 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:26.881 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:47:26.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:26.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:47:26.986 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:47:27.018 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:27.184 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 50 16 96387 96371 120.372 47.125 0.0330799 0.00830363 2024-04-23T16:47:27.235 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:47:27.288 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:47:27.367 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:47:27.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:27.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:47:27.473 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:47:27.503 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:27.803 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:27 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2220212617' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-23T16:47:27.803 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:27 smithi103 ceph-mon[39629]: osdmap e632: 12 total, 12 up, 8 in 2024-04-23T16:47:27.803 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:27 smithi103 ceph-mon[39629]: pgmap v1768: 27 pgs: 4 active+recovering, 23 active+clean; 5.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 59 MiB/s wr, 947 op/s; 19160/189622 objects misplaced (10.104%); 7.2 MiB/s, 114 objects/s recovering 2024-04-23T16:47:27.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:27 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2220212617' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-23T16:47:27.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:27 smithi188 ceph-mon[44101]: osdmap e632: 12 total, 12 up, 8 in 2024-04-23T16:47:27.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:27 smithi188 ceph-mon[44101]: pgmap v1768: 27 pgs: 4 active+recovering, 23 active+clean; 5.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 59 MiB/s wr, 947 op/s; 19160/189622 objects misplaced (10.104%); 7.2 MiB/s, 114 objects/s recovering 2024-04-23T16:47:28.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:27 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2220212617' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-23T16:47:28.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:27 smithi148 ceph-mon[44520]: osdmap e632: 12 total, 12 up, 8 in 2024-04-23T16:47:28.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:27 smithi148 ceph-mon[44520]: pgmap v1768: 27 pgs: 4 active+recovering, 23 active+clean; 5.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 59 MiB/s wr, 947 op/s; 19160/189622 objects misplaced (10.104%); 7.2 MiB/s, 114 objects/s recovering 2024-04-23T16:47:28.180 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 51 16 97208 97192 119.015 51.3125 0.0238633 0.00839988 2024-04-23T16:47:28.364 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:47:28.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:28.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:47:28.372 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-23T16:47:28.372 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:28.471 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:47:28.499 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:28.522 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:47:28.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:28.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:47:28.626 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:47:28.655 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:28.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:28.679 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:47:28.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:47:28.780 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:47:28.808 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:28.832 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:47:28.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:28.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:47:28.936 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:47:28.964 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:28.985 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:47:28.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:28.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:47:29.090 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:47:29.117 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:29.139 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:47:29.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:29.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:47:29.180 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 52 16 97998 97982 117.677 49.375 0.00929863 0.00849464 2024-04-23T16:47:29.242 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:47:29.272 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:29.526 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:47:29.527 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:47:29.527 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:47:29.527 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:47:29.527 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:47:29.527 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:47:29.527 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:47:29.527 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:47:29.527 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:47:29.527 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:47:29.527 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:47:29.527 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:47:29.528 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:47:29.528 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:47:29.528 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:47:29.528 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:47:29.528 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:47:29.528 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:47:29.528 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:47:29.528 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:47:29.528 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:47:29.528 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:47:29.528 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:47:29.528 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:47:29.846 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:47:29.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:29.851 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:47:29.952 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:47:29.980 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:30.001 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:47:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:30.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:47:30.107 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:47:30.134 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:30.155 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:47:30.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:30.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:47:30.180 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 53 16 98694 98678 116.278 43.5 0.012873 0.00859749 2024-04-23T16:47:30.234 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:29 smithi188 ceph-mon[44101]: pgmap v1769: 27 pgs: 4 active+recovering, 23 active+clean; 5.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 56 MiB/s wr, 896 op/s; 18739/192562 objects misplaced (9.731%); 10 MiB/s, 163 objects/s recovering 2024-04-23T16:47:30.234 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:29 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1588224585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:30.234 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.234 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.234 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.234 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.234 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.234 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:29 smithi188 ceph-mon[44101]: osdmap e633: 12 total, 12 up, 8 in 2024-04-23T16:47:30.235 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.235 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.259 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:47:30.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:29 smithi148 ceph-mon[44520]: pgmap v1769: 27 pgs: 4 active+recovering, 23 active+clean; 5.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 56 MiB/s wr, 896 op/s; 18739/192562 objects misplaced (9.731%); 10 MiB/s, 163 objects/s recovering 2024-04-23T16:47:30.285 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:29 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1588224585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:30.285 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.285 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.285 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.285 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.285 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.285 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:29 smithi148 ceph-mon[44520]: osdmap e633: 12 total, 12 up, 8 in 2024-04-23T16:47:30.285 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.285 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.286 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:30.287 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:29 smithi103 ceph-mon[39629]: pgmap v1769: 27 pgs: 4 active+recovering, 23 active+clean; 5.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 56 MiB/s wr, 896 op/s; 18739/192562 objects misplaced (9.731%); 10 MiB/s, 163 objects/s recovering 2024-04-23T16:47:30.287 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:29 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1588224585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:30.287 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.287 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.287 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.287 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.287 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.288 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:29 smithi103 ceph-mon[39629]: osdmap e633: 12 total, 12 up, 8 in 2024-04-23T16:47:30.288 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.288 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:30.305 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:47:30.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:30.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:47:30.412 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:47:30.444 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:30.798 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:30.798 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:47:30.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:30.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:47:30.902 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:47:30.984 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:31.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:31.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:31.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:31.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:31.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:31.180 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 54 16 99529 99513 115.092 52.1875 0.00713415 0.00868576 2024-04-23T16:47:31.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:31.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:31.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:31.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:31.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:31.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:31.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:31.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:31.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:31.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:32.181 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 55 16 100263 100247 113.834 45.875 0.0287683 0.00878242 2024-04-23T16:47:32.220 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:31 smithi103 ceph-mon[39629]: pgmap v1771: 27 pgs: 4 active+recovering, 23 active+clean; 5.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 44 MiB/s wr, 706 op/s; 18739/194018 objects misplaced (9.658%); 5.4 MiB/s, 85 objects/s recovering 2024-04-23T16:47:32.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:31 smithi148 ceph-mon[44520]: pgmap v1771: 27 pgs: 4 active+recovering, 23 active+clean; 5.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 44 MiB/s wr, 706 op/s; 18739/194018 objects misplaced (9.658%); 5.4 MiB/s, 85 objects/s recovering 2024-04-23T16:47:32.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:31 smithi188 ceph-mon[44101]: pgmap v1771: 27 pgs: 4 active+recovering, 23 active+clean; 5.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 44 MiB/s wr, 706 op/s; 18739/194018 objects misplaced (9.658%); 5.4 MiB/s, 85 objects/s recovering 2024-04-23T16:47:32.840 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:47:33.013 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:47:33.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:32 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:47:33.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:32 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:47:33.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:32 smithi103 ceph-mon[39629]: osdmap e634: 12 total, 12 up, 8 in 2024-04-23T16:47:33.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:32 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/171264666' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:47:33.181 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 56 16 101051 101035 112.681 49.25 0.0285516 0.00887194 2024-04-23T16:47:33.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:32 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:47:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:32 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:47:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:32 smithi148 ceph-mon[44520]: osdmap e634: 12 total, 12 up, 8 in 2024-04-23T16:47:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:32 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/171264666' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:47:33.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:32 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:47:33.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:32 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:47:33.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:32 smithi188 ceph-mon[44101]: osdmap e634: 12 total, 12 up, 8 in 2024-04-23T16:47:33.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:32 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/171264666' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:47:33.568 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:47:33.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:33.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:47:33.673 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:47:33.702 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:33.912 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:34.017 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.6 to 1 (10000) 2024-04-23T16:47:34.181 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 57 16 101822 101806 111.551 48.1875 0.00672931 0.00896145 2024-04-23T16:47:34.206 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:34 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:47:34.206 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:34 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:47:34.206 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:34 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:47:34.206 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:34 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:47:34.206 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:34 smithi103 ceph-mon[39629]: pgmap v1773: 27 pgs: 4 active+recovering, 23 active+clean; 6.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 50 MiB/s wr, 802 op/s; 18604/197364 objects misplaced (9.426%); 6.9 MiB/s, 110 objects/s recovering 2024-04-23T16:47:34.206 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:34 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:47:34.206 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:34 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/171264666' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:47:34.207 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:34 smithi103 ceph-mon[39629]: osdmap e635: 12 total, 12 up, 8 in 2024-04-23T16:47:34.207 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:34 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1243115849' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:47:34.207 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:34 smithi103 ceph-mon[39629]: from='client.36536 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:47:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:34 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:47:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:34 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:47:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:34 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:47:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:34 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:47:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:34 smithi148 ceph-mon[44520]: pgmap v1773: 27 pgs: 4 active+recovering, 23 active+clean; 6.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 50 MiB/s wr, 802 op/s; 18604/197364 objects misplaced (9.426%); 6.9 MiB/s, 110 objects/s recovering 2024-04-23T16:47:34.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:34 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:47:34.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:34 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/171264666' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:47:34.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:34 smithi148 ceph-mon[44520]: osdmap e635: 12 total, 12 up, 8 in 2024-04-23T16:47:34.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:34 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1243115849' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:47:34.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:34 smithi148 ceph-mon[44520]: from='client.36536 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:47:34.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:34 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:47:34.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:34 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:47:34.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:34 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:47:34.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:34 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:47:34.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:34 smithi188 ceph-mon[44101]: pgmap v1773: 27 pgs: 4 active+recovering, 23 active+clean; 6.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 50 MiB/s wr, 802 op/s; 18604/197364 objects misplaced (9.426%); 6.9 MiB/s, 110 objects/s recovering 2024-04-23T16:47:34.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:34 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:47:34.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:34 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/171264666' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:47:34.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:34 smithi188 ceph-mon[44101]: osdmap e635: 12 total, 12 up, 8 in 2024-04-23T16:47:34.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:34 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1243115849' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:47:34.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:34 smithi188 ceph-mon[44101]: from='client.36536 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:47:34.955 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:47:34.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:34.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:47:35.017 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:35 smithi188 ceph-mon[44101]: from='client.36536 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:47:35.017 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:35 smithi188 ceph-mon[44101]: osdmap e636: 12 total, 12 up, 8 in 2024-04-23T16:47:35.017 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:35 smithi188 ceph-mon[44101]: osdmap e637: 12 total, 12 up, 8 in 2024-04-23T16:47:35.059 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:47:35.126 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:35.181 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 58 16 102529 102513 110.39 44.1875 0.0295465 0.00905525 2024-04-23T16:47:35.262 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:35 smithi103 ceph-mon[39629]: from='client.36536 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:47:35.262 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:35 smithi103 ceph-mon[39629]: osdmap e636: 12 total, 12 up, 8 in 2024-04-23T16:47:35.263 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:35 smithi103 ceph-mon[39629]: osdmap e637: 12 total, 12 up, 8 in 2024-04-23T16:47:35.287 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-23T16:47:35.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:35 smithi148 ceph-mon[44520]: from='client.36536 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:47:35.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:35 smithi148 ceph-mon[44520]: osdmap e636: 12 total, 12 up, 8 in 2024-04-23T16:47:35.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:35 smithi148 ceph-mon[44520]: osdmap e637: 12 total, 12 up, 8 in 2024-04-23T16:47:35.958 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:47:36.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:36 smithi103 ceph-mon[39629]: pgmap v1777: 27 pgs: 4 active+recovering, 23 active+clean; 6.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 60 MiB/s wr, 964 op/s; 18208/201732 objects misplaced (9.026%); 8.3 MiB/s, 132 objects/s recovering 2024-04-23T16:47:36.181 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 59 16 103292 103276 109.328 47.6875 0.0203024 0.00914484 2024-04-23T16:47:36.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:36 smithi148 ceph-mon[44520]: pgmap v1777: 27 pgs: 4 active+recovering, 23 active+clean; 6.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 60 MiB/s wr, 964 op/s; 18208/201732 objects misplaced (9.026%); 8.3 MiB/s, 132 objects/s recovering 2024-04-23T16:47:36.377 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:36.447 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:47:36.470 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:47:36.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:36 smithi188 ceph-mon[44101]: pgmap v1777: 27 pgs: 4 active+recovering, 23 active+clean; 6.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 60 MiB/s wr, 964 op/s; 18208/201732 objects misplaced (9.026%); 8.3 MiB/s, 132 objects/s recovering 2024-04-23T16:47:36.495 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:47:36.518 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:47:36.543 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:47:36.568 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:47:36.590 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:47:36.611 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:47:36.636 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:47:36.672 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:47:36.699 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:47:36.724 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:47:36.725 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:47:36.725 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:47:36.726 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:47:36.726 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:47:36.726 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:47:36.726 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:47:36.726 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:47:36.727 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:47:36.727 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:47:36.727 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:47:36.727 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:47:36.727 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:47:37.181 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:47:37.181446 min lat: 0.00111068 max lat: 1.40272 avg lat: 0.00923178 2024-04-23T16:47:37.181 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:47:37.181 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 60 16 104036 104020 108.281 46.5 0.0249612 0.00923178 2024-04-23T16:47:37.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:37 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:47:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:37 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:47:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:37 smithi103 ceph-mon[39629]: osdmap e638: 12 total, 12 up, 8 in 2024-04-23T16:47:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:37 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3743909050' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:37 smithi103 ceph-mon[39629]: pgmap v1779: 27 pgs: 4 active+recovering, 23 active+clean; 6.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 60 MiB/s wr, 952 op/s; 18074/204988 objects misplaced (8.817%); 8.3 MiB/s, 132 objects/s recovering 2024-04-23T16:47:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:37 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:47:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:37 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:47:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:37 smithi188 ceph-mon[44101]: osdmap e638: 12 total, 12 up, 8 in 2024-04-23T16:47:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:37 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3743909050' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:37 smithi188 ceph-mon[44101]: pgmap v1779: 27 pgs: 4 active+recovering, 23 active+clean; 6.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 60 MiB/s wr, 952 op/s; 18074/204988 objects misplaced (8.817%); 8.3 MiB/s, 132 objects/s recovering 2024-04-23T16:47:37.824 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:37.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:37 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:47:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:37 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:47:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:37 smithi148 ceph-mon[44520]: osdmap e638: 12 total, 12 up, 8 in 2024-04-23T16:47:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:37 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3743909050' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:37.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:37 smithi148 ceph-mon[44520]: pgmap v1779: 27 pgs: 4 active+recovering, 23 active+clean; 6.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 60 MiB/s wr, 952 op/s; 18074/204988 objects misplaced (8.817%); 8.3 MiB/s, 132 objects/s recovering 2024-04-23T16:47:38.181 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:47:38.181 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 61 16 104759 104743 107.247 45.1875 0.0121269 0.00932104 2024-04-23T16:47:38.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:38.184 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:47:38.285 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:47:38.318 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:38.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:38.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:38.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:38.499 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:38.499 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:38.499 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:38.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:38.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:38.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:39.181 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 62 16 105467 105451 106.231 44.25 0.0129692 0.00940806 2024-04-23T16:47:39.571 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:47:39.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:39.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:47:39.675 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:47:39.702 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:39 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:47:39.702 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:39 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:47:39.702 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:39 smithi148 ceph-mon[44520]: osdmap e639: 12 total, 12 up, 8 in 2024-04-23T16:47:39.702 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:39 smithi148 ceph-mon[44520]: pgmap v1781: 27 pgs: 4 active+recovering, 23 active+clean; 6.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 36 MiB/s wr, 569 op/s; 18074/205872 objects misplaced (8.779%); 1.7 MiB/s, 27 objects/s recovering 2024-04-23T16:47:39.703 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:39.726 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:47:39.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:39.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:47:39.729 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:39 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:47:39.729 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:39 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:47:39.729 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:39 smithi188 ceph-mon[44101]: osdmap e639: 12 total, 12 up, 8 in 2024-04-23T16:47:39.729 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:39 smithi188 ceph-mon[44101]: pgmap v1781: 27 pgs: 4 active+recovering, 23 active+clean; 6.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 36 MiB/s wr, 569 op/s; 18074/205872 objects misplaced (8.779%); 1.7 MiB/s, 27 objects/s recovering 2024-04-23T16:47:39.820 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:39 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:47:39.821 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:39 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:47:39.821 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:39 smithi103 ceph-mon[39629]: osdmap e639: 12 total, 12 up, 8 in 2024-04-23T16:47:39.821 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:39 smithi103 ceph-mon[39629]: pgmap v1781: 27 pgs: 4 active+recovering, 23 active+clean; 6.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 36 MiB/s wr, 569 op/s; 18074/205872 objects misplaced (8.779%); 1.7 MiB/s, 27 objects/s recovering 2024-04-23T16:47:39.829 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:47:39.856 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:39.877 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:47:39.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:39.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:47:39.982 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:47:40.048 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:40.184 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 63 15 106221 106206 105.291 47.1875 0.0149601 0.00949373 2024-04-23T16:47:40.421 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:40 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/952165008' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:47:40.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:40 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/952165008' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:47:40.432 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.7 to 1 (10000) 2024-04-23T16:47:40.625 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:47:40.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:40.635 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:47:40.736 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:47:40.762 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:40 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/952165008' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:47:40.763 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:40.785 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:47:40.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:40.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:47:40.889 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:47:40.922 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:41.184 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 64 16 106914 106898 104.322 43.25 0.0188118 0.00958221 2024-04-23T16:47:41.336 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 8 1 2024-04-23T16:47:41.376 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:47:41.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:41.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:47:41.440 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:47:41.479 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:47:41.479 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:47:41.479 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:47:41.479 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:47:41.479 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:47:41.480 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:47:41.485 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:47:41.509 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:41.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:41 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/952165008' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:47:41.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:41 smithi103 ceph-mon[39629]: osdmap e640: 12 total, 12 up, 8 in 2024-04-23T16:47:41.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:41 smithi103 ceph-mon[39629]: pgmap v1783: 27 pgs: 4 active+recovering, 23 active+clean; 6.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 38 MiB/s wr, 615 op/s; 17652/209122 objects misplaced (8.441%); 5.8 MiB/s, 92 objects/s recovering 2024-04-23T16:47:41.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:41 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3465894161' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:47:41.728 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:41 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/952165008' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:47:41.728 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:41 smithi188 ceph-mon[44101]: osdmap e640: 12 total, 12 up, 8 in 2024-04-23T16:47:41.728 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:41 smithi188 ceph-mon[44101]: pgmap v1783: 27 pgs: 4 active+recovering, 23 active+clean; 6.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 38 MiB/s wr, 615 op/s; 17652/209122 objects misplaced (8.441%); 5.8 MiB/s, 92 objects/s recovering 2024-04-23T16:47:41.728 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:41 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3465894161' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:47:41.768 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:47:41.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:41.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:47:41.872 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:47:41.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:41 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/952165008' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:47:41.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:41 smithi148 ceph-mon[44520]: osdmap e640: 12 total, 12 up, 8 in 2024-04-23T16:47:41.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:41 smithi148 ceph-mon[44520]: pgmap v1783: 27 pgs: 4 active+recovering, 23 active+clean; 6.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 38 MiB/s wr, 615 op/s; 17652/209122 objects misplaced (8.441%); 5.8 MiB/s, 92 objects/s recovering 2024-04-23T16:47:41.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:41 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3465894161' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:47:41.899 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:41.919 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:47:41.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:41.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:47:42.022 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:47:42.092 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:42.185 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 65 16 107629 107613 103.404 44.6875 0.00844159 0.00966844 2024-04-23T16:47:42.268 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:47:42.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:42.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:47:42.329 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-23T16:47:42.378 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:47:42.436 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:42.461 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:42.461 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:42.492 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:47:42.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:42.508 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:47:42.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:42 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:47:42.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:42 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3465894161' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:47:42.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:42 smithi103 ceph-mon[39629]: osdmap e641: 12 total, 12 up, 8 in 2024-04-23T16:47:42.609 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:47:42.638 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:42.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:42 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:47:42.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:42 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3465894161' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:47:42.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:42 smithi188 ceph-mon[44101]: osdmap e641: 12 total, 12 up, 8 in 2024-04-23T16:47:42.764 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:47:42.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:42.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:47:42.868 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:47:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:42 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:47:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:42 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3465894161' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:47:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:42 smithi148 ceph-mon[44520]: osdmap e641: 12 total, 12 up, 8 in 2024-04-23T16:47:42.875 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:47:42.901 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:42.923 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:47:42.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:42.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:47:43.029 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:47:43.056 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:43.074 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:47:43.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:43.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:47:43.178 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:47:43.185 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 66 16 108334 108318 102.504 44.0625 0.00682131 0.00975134 2024-04-23T16:47:43.205 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:43.226 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:47:43.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:43.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:47:43.330 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:47:43.359 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:43.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:43 smithi188 ceph-mon[44101]: pgmap v1785: 27 pgs: 4 active+recovering, 23 active+clean; 6.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 38 MiB/s wr, 606 op/s; 17507/212270 objects misplaced (8.248%); 5.9 MiB/s, 94 objects/s recovering 2024-04-23T16:47:43.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:43 smithi103 ceph-mon[39629]: pgmap v1785: 27 pgs: 4 active+recovering, 23 active+clean; 6.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 38 MiB/s wr, 606 op/s; 17507/212270 objects misplaced (8.248%); 5.9 MiB/s, 94 objects/s recovering 2024-04-23T16:47:43.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:43 smithi148 ceph-mon[44520]: pgmap v1785: 27 pgs: 4 active+recovering, 23 active+clean; 6.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 38 MiB/s wr, 606 op/s; 17507/212270 objects misplaced (8.248%); 5.9 MiB/s, 94 objects/s recovering 2024-04-23T16:47:44.186 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 67 16 109055 109039 101.646 45.0625 0.00926934 0.00983466 2024-04-23T16:47:44.381 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:47:44.542 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:47:44.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:44.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:47:44.651 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:47:44.686 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:45.046 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:47:45.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:45.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:47:45.151 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:47:45.185 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:45.189 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 68 15 109784 109769 100.821 45.625 0.0203185 0.00991604 2024-04-23T16:47:45.286 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:47:45.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:45.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:47:45.390 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:47:45.417 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:45.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:45 smithi188 ceph-mon[44101]: 8.7 scrub starts 2024-04-23T16:47:45.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:45 smithi188 ceph-mon[44101]: pgmap v1786: 27 pgs: 4 active+recovering, 23 active+clean; 6.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 50 MiB/s wr, 806 op/s; 17099/216356 objects misplaced (7.903%); 9.4 MiB/s, 150 objects/s recovering 2024-04-23T16:47:45.437 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:47:45.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:45.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:47:45.540 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:47:45.568 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:45 smithi103 ceph-mon[39629]: 8.7 scrub starts 2024-04-23T16:47:45.568 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:45 smithi103 ceph-mon[39629]: pgmap v1786: 27 pgs: 4 active+recovering, 23 active+clean; 6.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 50 MiB/s wr, 806 op/s; 17099/216356 objects misplaced (7.903%); 9.4 MiB/s, 150 objects/s recovering 2024-04-23T16:47:45.696 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:45.747 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:47:45.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:45.752 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:47:45.853 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:47:45.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:45 smithi148 ceph-mon[44520]: 8.7 scrub starts 2024-04-23T16:47:45.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:45 smithi148 ceph-mon[44520]: pgmap v1786: 27 pgs: 4 active+recovering, 23 active+clean; 6.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 50 MiB/s wr, 806 op/s; 17099/216356 objects misplaced (7.903%); 9.4 MiB/s, 150 objects/s recovering 2024-04-23T16:47:45.961 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:46.026 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:47:46.026 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:47:46.026 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:47:46.040 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:47:46.062 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:46.067 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:47:46.093 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:47:46.121 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:47:46.142 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:47:46.179 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:47:46.188 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 69 15 110416 110401 99.9322 39.5 0.021809 0.0100037 2024-04-23T16:47:46.209 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:47:46.232 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:47:46.271 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:47:46.273 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:47:46.273 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:47:46.273 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:47:46.273 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:47:46.273 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:47:46.273 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:47:46.273 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:47:46.273 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:47:46.273 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:47:46.273 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:47:46.273 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:47:46.273 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:47:46.425 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.8 to 1 (10000) 2024-04-23T16:47:46.446 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:47:46.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:46.458 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:47:46.561 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:47:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:46 smithi103 ceph-mon[39629]: 8.a scrub starts 2024-04-23T16:47:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:46 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1968470918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:46 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2474849028' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:47:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:46 smithi103 ceph-mon[39629]: from='client.36560 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:47:46.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:46.606 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:46.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:46 smithi188 ceph-mon[44101]: 8.a scrub starts 2024-04-23T16:47:46.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:46 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1968470918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:46.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:46.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:46.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:46 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2474849028' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:47:46.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:46 smithi188 ceph-mon[44101]: from='client.36560 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:47:46.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:46 smithi148 ceph-mon[44520]: 8.a scrub starts 2024-04-23T16:47:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:46 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1968470918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:46 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2474849028' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:47:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:46 smithi148 ceph-mon[44520]: from='client.36560 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:47:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.188 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 70 16 111157 111141 99.1657 46.25 0.00393764 0.0100806 2024-04-23T16:47:47.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:47 smithi188 ceph-mon[44101]: 8.7 scrub ok 2024-04-23T16:47:47.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:47 smithi188 ceph-mon[44101]: from='client.36560 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:47:47.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:47 smithi188 ceph-mon[44101]: osdmap e642: 12 total, 12 up, 8 in 2024-04-23T16:47:47.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:47 smithi188 ceph-mon[44101]: 8.a scrub ok 2024-04-23T16:47:47.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:47 smithi188 ceph-mon[44101]: pgmap v1788: 27 pgs: 4 active+recovering, 23 active+clean; 6.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 50 MiB/s wr, 796 op/s; 16956/219436 objects misplaced (7.727%); 6.7 MiB/s, 107 objects/s recovering 2024-04-23T16:47:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:47 smithi103 ceph-mon[39629]: 8.7 scrub ok 2024-04-23T16:47:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:47 smithi103 ceph-mon[39629]: from='client.36560 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:47:47.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:47 smithi103 ceph-mon[39629]: osdmap e642: 12 total, 12 up, 8 in 2024-04-23T16:47:47.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:47 smithi103 ceph-mon[39629]: 8.a scrub ok 2024-04-23T16:47:47.853 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:47 smithi103 ceph-mon[39629]: pgmap v1788: 27 pgs: 4 active+recovering, 23 active+clean; 6.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 50 MiB/s wr, 796 op/s; 16956/219436 objects misplaced (7.727%); 6.7 MiB/s, 107 objects/s recovering 2024-04-23T16:47:47.866 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:47:47.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:47.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:47:47.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:47 smithi148 ceph-mon[44520]: 8.7 scrub ok 2024-04-23T16:47:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:47 smithi148 ceph-mon[44520]: from='client.36560 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:47:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:47 smithi148 ceph-mon[44520]: osdmap e642: 12 total, 12 up, 8 in 2024-04-23T16:47:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:47 smithi148 ceph-mon[44520]: 8.a scrub ok 2024-04-23T16:47:47.875 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:47 smithi148 ceph-mon[44520]: pgmap v1788: 27 pgs: 4 active+recovering, 23 active+clean; 6.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 50 MiB/s wr, 796 op/s; 16956/219436 objects misplaced (7.727%); 6.7 MiB/s, 107 objects/s recovering 2024-04-23T16:47:47.969 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:47:48.038 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:48.188 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 71 16 111895 111879 98.419 46.125 0.0116012 0.0101554 2024-04-23T16:47:49.188 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 72 16 112650 112634 97.7078 47.1875 0.0147469 0.0102318 2024-04-23T16:47:50.188 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 73 16 113414 113398 97.0238 47.75 0.0178015 0.0103042 2024-04-23T16:47:50.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:49 smithi103 ceph-mon[39629]: pgmap v1789: 27 pgs: 4 active+recovering, 23 active+clean; 6.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 44 MiB/s wr, 700 op/s; 16956/220340 objects misplaced (7.695%); 5.4 MiB/s, 86 objects/s recovering 2024-04-23T16:47:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:49 smithi103 ceph-mon[39629]: 8.1 scrub starts 2024-04-23T16:47:50.358 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:47:50.359 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:47:50.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:49 smithi148 ceph-mon[44520]: pgmap v1789: 27 pgs: 4 active+recovering, 23 active+clean; 6.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 44 MiB/s wr, 700 op/s; 16956/220340 objects misplaced (7.695%); 5.4 MiB/s, 86 objects/s recovering 2024-04-23T16:47:50.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:49 smithi148 ceph-mon[44520]: 8.1 scrub starts 2024-04-23T16:47:50.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:49 smithi188 ceph-mon[44101]: pgmap v1789: 27 pgs: 4 active+recovering, 23 active+clean; 6.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 44 MiB/s wr, 700 op/s; 16956/220340 objects misplaced (7.695%); 5.4 MiB/s, 86 objects/s recovering 2024-04-23T16:47:50.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:50 smithi188 ceph-mon[44101]: 8.1 scrub starts 2024-04-23T16:47:50.550 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 9 1 2024-04-23T16:47:51.189 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 74 16 114125 114109 96.3132 44.4375 0.00900406 0.0103778 2024-04-23T16:47:51.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:50 smithi103 ceph-mon[39629]: 8.1 scrub ok 2024-04-23T16:47:51.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:50 smithi148 ceph-mon[44520]: 8.1 scrub ok 2024-04-23T16:47:51.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:50 smithi188 ceph-mon[44101]: 8.1 scrub ok 2024-04-23T16:47:51.491 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:47:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:51.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:47:51.595 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:47:51.623 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:51.643 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:47:51.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:51.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:47:51.748 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:47:51.967 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:47:52.060 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:52.080 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:52.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:51 smithi103 ceph-mon[39629]: pgmap v1790: 27 pgs: 4 active+recovering, 23 active+clean; 6.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 44 MiB/s wr, 710 op/s; 16555/223472 objects misplaced (7.408%); 7.2 MiB/s, 115 objects/s recovering 2024-04-23T16:47:52.189 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 75 16 114869 114853 95.6493 46.5 0.0558343 0.0104532 2024-04-23T16:47:52.214 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:47:52.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:52.220 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:47:52.321 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:47:52.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:51 smithi148 ceph-mon[44520]: pgmap v1790: 27 pgs: 4 active+recovering, 23 active+clean; 6.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 44 MiB/s wr, 710 op/s; 16555/223472 objects misplaced (7.408%); 7.2 MiB/s, 115 objects/s recovering 2024-04-23T16:47:52.400 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:51 smithi188 ceph-mon[44101]: pgmap v1790: 27 pgs: 4 active+recovering, 23 active+clean; 6.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 44 MiB/s wr, 710 op/s; 16555/223472 objects misplaced (7.408%); 7.2 MiB/s, 115 objects/s recovering 2024-04-23T16:47:52.404 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:53.067 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:47:53.189 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 76 16 115548 115532 94.9493 42.4375 0.0174673 0.0105294 2024-04-23T16:47:53.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:52 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4105454283' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:47:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:52 smithi103 ceph-mon[39629]: from='client.36572 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:47:53.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:52 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:47:53.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:52 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4105454283' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:47:53.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:52 smithi148 ceph-mon[44520]: from='client.36572 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:47:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:52 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:47:53.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:52 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4105454283' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:47:53.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:52 smithi188 ceph-mon[44101]: from='client.36572 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:47:53.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:52 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:47:54.189 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 77 16 116164 116148 94.2165 38.5 0.0216621 0.0106124 2024-04-23T16:47:54.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:54 smithi103 ceph-mon[39629]: pgmap v1791: 27 pgs: 4 active+recovering, 23 active+clean; 6.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 46 MiB/s wr, 731 op/s; 16405/226912 objects misplaced (7.230%); 6.9 MiB/s, 110 objects/s recovering 2024-04-23T16:47:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:54 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:47:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:54 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:47:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:54 smithi103 ceph-mon[39629]: from='client.36572 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:47:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:54 smithi103 ceph-mon[39629]: osdmap e643: 12 total, 12 up, 8 in 2024-04-23T16:47:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:54 smithi103 ceph-mon[39629]: 8.b scrub starts 2024-04-23T16:47:54.364 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:54.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:54 smithi148 ceph-mon[44520]: pgmap v1791: 27 pgs: 4 active+recovering, 23 active+clean; 6.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 46 MiB/s wr, 731 op/s; 16405/226912 objects misplaced (7.230%); 6.9 MiB/s, 110 objects/s recovering 2024-04-23T16:47:54.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:54 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:47:54.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:54 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:47:54.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:54 smithi148 ceph-mon[44520]: from='client.36572 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:47:54.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:54 smithi148 ceph-mon[44520]: osdmap e643: 12 total, 12 up, 8 in 2024-04-23T16:47:54.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:54 smithi148 ceph-mon[44520]: 8.b scrub starts 2024-04-23T16:47:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:54 smithi188 ceph-mon[44101]: pgmap v1791: 27 pgs: 4 active+recovering, 23 active+clean; 6.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 46 MiB/s wr, 731 op/s; 16405/226912 objects misplaced (7.230%); 6.9 MiB/s, 110 objects/s recovering 2024-04-23T16:47:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:54 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:47:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:54 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:47:54.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:54 smithi188 ceph-mon[44101]: from='client.36572 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:47:54.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:54 smithi188 ceph-mon[44101]: osdmap e643: 12 total, 12 up, 8 in 2024-04-23T16:47:54.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:54 smithi188 ceph-mon[44101]: 8.b scrub starts 2024-04-23T16:47:55.189 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 78 16 116852 116836 93.5602 43 0.00566333 0.0106855 2024-04-23T16:47:55.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:55 smithi188 ceph-mon[44101]: 8.b scrub ok 2024-04-23T16:47:55.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:55 smithi188 ceph-mon[44101]: pgmap v1793: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering, 22 active+clean; 7.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 50 MiB/s wr, 796 op/s; 15992/230760 objects misplaced (6.930%); 8.2 MiB/s, 130 objects/s recovering 2024-04-23T16:47:55.822 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:55 smithi103 ceph-mon[39629]: 8.b scrub ok 2024-04-23T16:47:55.823 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:55 smithi103 ceph-mon[39629]: pgmap v1793: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering, 22 active+clean; 7.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 50 MiB/s wr, 796 op/s; 15992/230760 objects misplaced (6.930%); 8.2 MiB/s, 130 objects/s recovering 2024-04-23T16:47:55.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:55 smithi148 ceph-mon[44520]: 8.b scrub ok 2024-04-23T16:47:55.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:55 smithi148 ceph-mon[44520]: pgmap v1793: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering, 22 active+clean; 7.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 50 MiB/s wr, 796 op/s; 15992/230760 objects misplaced (6.930%); 8.2 MiB/s, 130 objects/s recovering 2024-04-23T16:47:56.189 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 79 16 117603 117587 92.9703 46.9375 0.0133024 0.0107546 2024-04-23T16:47:56.531 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:56 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:47:56.531 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:56 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:47:56.531 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:56 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:47:56.728 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:56 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:47:56.728 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:56 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:47:56.728 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:56 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:47:56.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:56 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:47:56.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:56 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:47:56.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:56 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:47:56.956 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:47:56.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:56.968 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:47:56.989 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-23T16:47:57.069 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:47:57.101 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:57.160 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:47:57.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:57.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:47:57.181 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:47:57.189 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:47:57.189823 min lat: 0.00111068 max lat: 1.40272 avg lat: 0.0108242 2024-04-23T16:47:57.190 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:47:57.190 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 80 16 118306 118290 92.3576 43.9375 0.032328 0.0108242 2024-04-23T16:47:57.265 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:47:57.315 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:47:57.337 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:47:57.343 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:57.369 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:47:57.384 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:47:57.391 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:47:57.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:57.392 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:47:57.417 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:47:57.421 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.9 to 1 (10000) 2024-04-23T16:47:57.455 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:47:57.477 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:47:57.493 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:47:57.505 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:47:57.520 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:57.540 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:47:57.541 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:47:57.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:57.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:47:57.556 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:47:57.566 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:47:57.594 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:47:57.629 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:47:57.631 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:47:57.631 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:47:57.631 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:47:57.631 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:47:57.631 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:47:57.631 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:47:57.631 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:47:57.631 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:47:57.631 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:47:57.631 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:47:57.631 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:47:57.631 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:47:57.644 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:47:57.669 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:57 smithi148 ceph-mon[44520]: pgmap v1794: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering, 22 active+clean; 7.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 38 MiB/s wr, 609 op/s; 15992/231632 objects misplaced (6.904%); 6.0 MiB/s, 96 objects/s recovering 2024-04-23T16:47:57.669 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:57 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1291799829' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:57.669 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:57 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2405947605' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:47:57.674 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:57.691 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:47:57.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:57.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:47:57.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:57 smithi188 ceph-mon[44101]: pgmap v1794: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering, 22 active+clean; 7.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 38 MiB/s wr, 609 op/s; 15992/231632 objects misplaced (6.904%); 6.0 MiB/s, 96 objects/s recovering 2024-04-23T16:47:57.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:57 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1291799829' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:57.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:57 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2405947605' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:47:57.796 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:47:57.822 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:57 smithi103 ceph-mon[39629]: pgmap v1794: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering, 22 active+clean; 7.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 38 MiB/s wr, 609 op/s; 15992/231632 objects misplaced (6.904%); 6.0 MiB/s, 96 objects/s recovering 2024-04-23T16:47:57.823 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:57 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1291799829' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:47:57.823 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:57 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2405947605' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:47:57.825 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:58.030 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:47:58.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:58.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:47:58.135 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:47:58.176 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:58.190 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 81 16 119025 119009 91.7724 44.9375 0.00713051 0.0108943 2024-04-23T16:47:58.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:58 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2405947605' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:47:58.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:58 smithi103 ceph-mon[39629]: osdmap e644: 12 total, 12 up, 8 in 2024-04-23T16:47:58.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:58 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:58 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2405947605' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:47:58.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:58 smithi188 ceph-mon[44101]: osdmap e644: 12 total, 12 up, 8 in 2024-04-23T16:47:58.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:58 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:58 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2405947605' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:47:58.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:58 smithi148 ceph-mon[44520]: osdmap e644: 12 total, 12 up, 8 in 2024-04-23T16:47:58.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:58.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:58 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:59.192 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 82 16 119751 119735 91.2065 45.375 0.0747878 0.0109617 2024-04-23T16:47:59.628 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:47:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:59.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:47:59.647 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 11 1 2024-04-23T16:47:59.652 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:47:59.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:47:59 smithi188 ceph-mon[44101]: pgmap v1796: 27 pgs: 4 active+recovering, 23 active+clean; 7.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 44 MiB/s wr, 700 op/s; 15846/234688 objects misplaced (6.752%); 5.5 MiB/s, 88 objects/s recovering 2024-04-23T16:47:59.734 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:47:59.760 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:59.760 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:47:59 smithi148 ceph-mon[44520]: pgmap v1796: 27 pgs: 4 active+recovering, 23 active+clean; 7.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 44 MiB/s wr, 700 op/s; 15846/234688 objects misplaced (6.752%); 5.5 MiB/s, 88 objects/s recovering 2024-04-23T16:47:59.762 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:59.787 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:47:59.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:59.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:47:59.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:47:59.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:47:59 smithi103 ceph-mon[39629]: pgmap v1796: 27 pgs: 4 active+recovering, 23 active+clean; 7.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 44 MiB/s wr, 700 op/s; 15846/234688 objects misplaced (6.752%); 5.5 MiB/s, 88 objects/s recovering 2024-04-23T16:47:59.891 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:47:59.925 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:47:59.945 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:47:59.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:47:59.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:48:00.049 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:48:00.192 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 83 16 120530 120514 90.6921 48.6875 0.0212787 0.0110238 2024-04-23T16:48:00.279 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:00.318 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:48:00.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:00.327 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:48:00.428 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:48:00.503 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:00.581 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:48:00.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:00.585 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:48:00.686 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:48:00.717 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:01.001 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:01.112 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:48:01.192 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 84 16 121224 121208 90.1291 43.375 0.0293421 0.0110931 2024-04-23T16:48:01.231 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:00 smithi188 ceph-mon[44101]: 1.8 scrub starts 2024-04-23T16:48:01.231 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:00 smithi188 ceph-mon[44101]: 1.8 scrub ok 2024-04-23T16:48:01.264 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:00 smithi103 ceph-mon[39629]: 1.8 scrub starts 2024-04-23T16:48:01.264 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:00 smithi103 ceph-mon[39629]: 1.8 scrub ok 2024-04-23T16:48:01.265 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:48:01.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:01.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:48:01.369 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:48:01.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:00 smithi148 ceph-mon[44520]: 1.8 scrub starts 2024-04-23T16:48:01.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:00 smithi148 ceph-mon[44520]: 1.8 scrub ok 2024-04-23T16:48:01.442 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:02.192 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 85 16 121948 121932 89.6013 45.25 0.0287202 0.0111583 2024-04-23T16:48:02.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:01 smithi103 ceph-mon[39629]: pgmap v1797: 27 pgs: 4 active+recovering, 23 active+clean; 7.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 43 MiB/s wr, 681 op/s; 15419/237816 objects misplaced (6.484%); 7.7 MiB/s, 123 objects/s recovering 2024-04-23T16:48:02.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:01 smithi148 ceph-mon[44520]: pgmap v1797: 27 pgs: 4 active+recovering, 23 active+clean; 7.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 43 MiB/s wr, 681 op/s; 15419/237816 objects misplaced (6.484%); 7.7 MiB/s, 123 objects/s recovering 2024-04-23T16:48:02.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:01 smithi188 ceph-mon[44101]: pgmap v1797: 27 pgs: 4 active+recovering, 23 active+clean; 7.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 43 MiB/s wr, 681 op/s; 15419/237816 objects misplaced (6.484%); 7.7 MiB/s, 123 objects/s recovering 2024-04-23T16:48:02.929 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:03.007 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:48:03.131 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:48:03.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:03.143 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:48:03.192 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 86 16 122672 122656 89.0858 45.25 0.0341028 0.0112238 2024-04-23T16:48:03.218 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:02 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2954150685' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:48:03.244 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:48:03.310 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:02 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2954150685' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:48:03.317 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:03.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:02 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2954150685' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:48:04.192 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 87 16 123389 123373 88.5772 44.8125 0.0082028 0.0112866 2024-04-23T16:48:04.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:03 smithi103 ceph-mon[39629]: pgmap v1798: 27 pgs: 4 active+recovering, 23 active+clean; 7.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 45 MiB/s wr, 714 op/s; 15266/241062 objects misplaced (6.333%); 7.2 MiB/s, 115 objects/s recovering 2024-04-23T16:48:04.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:03 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:48:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:03 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2954150685' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:48:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:03 smithi103 ceph-mon[39629]: osdmap e645: 12 total, 12 up, 8 in 2024-04-23T16:48:04.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:03 smithi148 ceph-mon[44520]: pgmap v1798: 27 pgs: 4 active+recovering, 23 active+clean; 7.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 45 MiB/s wr, 714 op/s; 15266/241062 objects misplaced (6.333%); 7.2 MiB/s, 115 objects/s recovering 2024-04-23T16:48:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:03 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:48:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:03 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2954150685' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:48:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:03 smithi148 ceph-mon[44520]: osdmap e645: 12 total, 12 up, 8 in 2024-04-23T16:48:04.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:03 smithi188 ceph-mon[44101]: pgmap v1798: 27 pgs: 4 active+recovering, 23 active+clean; 7.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 45 MiB/s wr, 714 op/s; 15266/241062 objects misplaced (6.333%); 7.2 MiB/s, 115 objects/s recovering 2024-04-23T16:48:04.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:03 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:48:04.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:03 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2954150685' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:48:04.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:03 smithi188 ceph-mon[44101]: osdmap e645: 12 total, 12 up, 8 in 2024-04-23T16:48:05.195 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 88 16 124092 124076 88.0675 43.9375 0.0049073 0.0113486 2024-04-23T16:48:05.414 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.11 to 1 (10000) 2024-04-23T16:48:05.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:05 smithi188 ceph-mon[44101]: pgmap v1800: 27 pgs: 4 active+recovering, 23 active+clean; 7.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 53 MiB/s wr, 841 op/s; 14830/245102 objects misplaced (6.051%); 9.1 MiB/s, 145 objects/s recovering 2024-04-23T16:48:05.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:05 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2136958654' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:48:05.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:05 smithi103 ceph-mon[39629]: pgmap v1800: 27 pgs: 4 active+recovering, 23 active+clean; 7.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 53 MiB/s wr, 841 op/s; 14830/245102 objects misplaced (6.051%); 9.1 MiB/s, 145 objects/s recovering 2024-04-23T16:48:05.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:05 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2136958654' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:48:05.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:05 smithi148 ceph-mon[44520]: pgmap v1800: 27 pgs: 4 active+recovering, 23 active+clean; 7.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 53 MiB/s wr, 841 op/s; 14830/245102 objects misplaced (6.051%); 9.1 MiB/s, 145 objects/s recovering 2024-04-23T16:48:05.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:05 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2136958654' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:48:06.195 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 89 16 124815 124799 87.5859 45.1875 0.02963 0.011415 2024-04-23T16:48:06.433 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-23T16:48:06.517 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:48:06.517 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:48:06.517 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:48:06.517 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:48:06.517 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:48:06.517 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:48:06.517 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:48:06.517 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:48:06.517 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:48:06.517 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:48:06.517 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:48:06.517 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:48:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:48:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:48:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:48:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:48:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:48:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:48:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:48:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:48:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:48:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:48:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:48:06.519 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:48:06.729 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:06 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2136958654' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:48:06.729 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:06 smithi188 ceph-mon[44101]: osdmap e646: 12 total, 12 up, 8 in 2024-04-23T16:48:06.729 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:06 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4060380542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:06.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:06 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2136958654' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:48:06.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:06 smithi103 ceph-mon[39629]: osdmap e646: 12 total, 12 up, 8 in 2024-04-23T16:48:06.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:06 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4060380542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:06.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:06 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2136958654' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:48:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:06 smithi148 ceph-mon[44520]: osdmap e646: 12 total, 12 up, 8 in 2024-04-23T16:48:06.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:06 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4060380542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:07.196 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Total time run: 90.0278 2024-04-23T16:48:07.196 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Total writes made: 125506 2024-04-23T16:48:07.196 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Write size: 65536 2024-04-23T16:48:07.196 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Object size: 65536 2024-04-23T16:48:07.196 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Bandwidth (MB/sec): 87.13 2024-04-23T16:48:07.196 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev Bandwidth: 66.7685 2024-04-23T16:48:07.196 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max bandwidth (MB/sec): 331.938 2024-04-23T16:48:07.196 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min bandwidth (MB/sec): 0 2024-04-23T16:48:07.196 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Average IOPS: 1394 2024-04-23T16:48:07.196 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev IOPS: 1068.3 2024-04-23T16:48:07.196 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max IOPS: 5311 2024-04-23T16:48:07.196 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min IOPS: 0 2024-04-23T16:48:07.197 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Average Latency(s): 0.0114757 2024-04-23T16:48:07.197 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev Latency(s): 0.0187573 2024-04-23T16:48:07.197 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max latency(s): 1.40272 2024-04-23T16:48:07.197 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min latency(s): 0.00111068 2024-04-23T16:48:07.203 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Cleaning up (deleting benchmark objects) 2024-04-23T16:48:07.238 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:07.507 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.507 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.507 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.508 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.508 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.508 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.508 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.508 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.508 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.508 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.508 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.508 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.508 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:07 smithi103 ceph-mon[39629]: pgmap v1802: 27 pgs: 4 active+recovering, 23 active+clean; 7.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 44 MiB/s wr, 708 op/s; 14830/246020 objects misplaced (6.028%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:48:07.568 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:48:07.685 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:48:07.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:07.695 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:48:07.695 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.695 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.695 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.695 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.695 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.695 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.695 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.695 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.695 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.695 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.696 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.696 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.696 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:07 smithi188 ceph-mon[44101]: pgmap v1802: 27 pgs: 4 active+recovering, 23 active+clean; 7.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 44 MiB/s wr, 708 op/s; 14830/246020 objects misplaced (6.028%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:48:07.796 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:48:07.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:07 smithi148 ceph-mon[44520]: pgmap v1802: 27 pgs: 4 active+recovering, 23 active+clean; 7.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 44 MiB/s wr, 708 op/s; 14830/246020 objects misplaced (6.028%); 7.9 MiB/s, 126 objects/s recovering 2024-04-23T16:48:07.885 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:08.003 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:48:08.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:08.013 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:48:08.114 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:48:08.201 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:08.222 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:09.227 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:48:10.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:09 smithi188 ceph-mon[44101]: pgmap v1803: 27 pgs: 4 active+recovering, 23 active+clean; 7.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 44 MiB/s wr, 713 op/s; 14673/248778 objects misplaced (5.898%); 5.8 MiB/s, 93 objects/s recovering 2024-04-23T16:48:10.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:09 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3916802247' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:48:10.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:09 smithi188 ceph-mon[44101]: from='client.36608 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:48:10.345 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:48:10.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:09 smithi103 ceph-mon[39629]: pgmap v1803: 27 pgs: 4 active+recovering, 23 active+clean; 7.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 44 MiB/s wr, 713 op/s; 14673/248778 objects misplaced (5.898%); 5.8 MiB/s, 93 objects/s recovering 2024-04-23T16:48:10.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:09 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3916802247' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:48:10.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:09 smithi103 ceph-mon[39629]: from='client.36608 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:48:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:10.353 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:48:10.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:09 smithi148 ceph-mon[44520]: pgmap v1803: 27 pgs: 4 active+recovering, 23 active+clean; 7.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 44 MiB/s wr, 713 op/s; 14673/248778 objects misplaced (5.898%); 5.8 MiB/s, 93 objects/s recovering 2024-04-23T16:48:10.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:09 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3916802247' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:48:10.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:09 smithi148 ceph-mon[44520]: from='client.36608 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:48:10.407 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.1 to 1 (10000) 2024-04-23T16:48:10.454 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:48:10.534 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:10.570 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:48:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:10.575 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:48:10.676 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:48:10.704 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:10.727 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:48:10.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:10.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:48:10.831 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:48:10.858 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:10.882 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:48:10.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:10.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:48:10.988 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:48:11.018 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:11.057 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:48:11.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:11.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:48:11.162 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:48:11.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:10 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:48:11.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:10 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:48:11.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:10 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:48:11.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:10 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:48:11.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:10 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:48:11.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:10 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:48:11.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:10 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:48:11.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:10 smithi188 ceph-mon[44101]: from='client.36608 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-23T16:48:11.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:10 smithi188 ceph-mon[44101]: osdmap e647: 12 total, 12 up, 8 in 2024-04-23T16:48:11.237 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:11.288 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:48:11.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:11.292 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:48:11.307 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:10 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:48:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:10 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:48:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:10 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:48:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:10 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:48:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:10 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:48:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:10 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:48:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:10 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:48:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:10 smithi103 ceph-mon[39629]: from='client.36608 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-23T16:48:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:10 smithi103 ceph-mon[39629]: osdmap e647: 12 total, 12 up, 8 in 2024-04-23T16:48:11.363 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:48:11.363 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:48:11.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:10 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:48:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:10 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:48:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:10 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:48:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:10 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:48:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:10 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:48:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:10 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:48:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:10 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:48:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:10 smithi148 ceph-mon[44520]: from='client.36608 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-23T16:48:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:10 smithi148 ceph-mon[44520]: osdmap e647: 12 total, 12 up, 8 in 2024-04-23T16:48:11.394 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:48:11.422 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:12.135 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:11 smithi188 ceph-mon[44101]: pgmap v1805: 27 pgs: 4 active+recovering, 23 active+clean; 7.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 38 MiB/s wr, 716 op/s; 14223/249088 objects misplaced (5.710%); 8.2 MiB/s, 131 objects/s recovering 2024-04-23T16:48:12.135 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:11 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:48:12.136 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:11 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:48:12.172 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:48:12.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:12.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:48:12.187 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-23T16:48:12.194 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-04-23T16:48:12.194 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-04-23T16:48:12.194 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-23T16:48:12.194 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T16:48:12.265 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:11 smithi103 ceph-mon[39629]: pgmap v1805: 27 pgs: 4 active+recovering, 23 active+clean; 7.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 38 MiB/s wr, 716 op/s; 14223/249088 objects misplaced (5.710%); 8.2 MiB/s, 131 objects/s recovering 2024-04-23T16:48:12.265 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:11 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:48:12.265 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:11 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:48:12.275 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:48:12.313 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:12.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:11 smithi148 ceph-mon[44520]: pgmap v1805: 27 pgs: 4 active+recovering, 23 active+clean; 7.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 38 MiB/s wr, 716 op/s; 14223/249088 objects misplaced (5.710%); 8.2 MiB/s, 131 objects/s recovering 2024-04-23T16:48:12.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:11 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:48:12.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:11 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:48:12.662 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:48:12.773 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:48:12.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:12.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:48:12.877 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:48:12.956 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:13.004 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:48:13.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:13.013 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:48:13.114 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:48:13.146 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:13.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:12 smithi188 ceph-mon[44101]: osdmap e648: 12 total, 12 up, 8 in 2024-04-23T16:48:13.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:12 smithi103 ceph-mon[39629]: osdmap e648: 12 total, 12 up, 8 in 2024-04-23T16:48:13.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:12 smithi148 ceph-mon[44520]: osdmap e648: 12 total, 12 up, 8 in 2024-04-23T16:48:14.068 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:14.097 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:48:14.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:13 smithi103 ceph-mon[39629]: pgmap v1807: 27 pgs: 4 active+recovering, 23 active+clean; 7.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 136 B/s rd, 25 MiB/s wr, 703 op/s; 14051/246382 objects misplaced (5.703%); 6.5 MiB/s, 103 objects/s recovering 2024-04-23T16:48:14.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:14.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:48:14.204 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:48:14.232 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:14.361 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:48:14.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:14.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:48:14.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:13 smithi148 ceph-mon[44520]: pgmap v1807: 27 pgs: 4 active+recovering, 23 active+clean; 7.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 136 B/s rd, 25 MiB/s wr, 703 op/s; 14051/246382 objects misplaced (5.703%); 6.5 MiB/s, 103 objects/s recovering 2024-04-23T16:48:14.466 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:48:14.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:13 smithi188 ceph-mon[44101]: pgmap v1807: 27 pgs: 4 active+recovering, 23 active+clean; 7.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 136 B/s rd, 25 MiB/s wr, 703 op/s; 14051/246382 objects misplaced (5.703%); 6.5 MiB/s, 103 objects/s recovering 2024-04-23T16:48:14.538 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:15.019 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:48:15.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:14 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1501526728' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:48:15.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:14 smithi103 ceph-mon[39629]: from='client.36614 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:48:15.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:14 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/574314064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:15.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:15.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:15.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:14 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1501526728' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:48:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:14 smithi148 ceph-mon[44520]: from='client.36614 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:48:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:14 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/574314064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:15.442 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:48:15.442 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:48:15.442 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:48:15.442 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:48:15.442 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:48:15.442 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:48:15.442 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:48:15.442 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:48:15.442 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:48:15.443 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:48:15.443 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:48:15.443 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:48:15.443 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:48:15.443 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:48:15.443 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:48:15.443 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:48:15.443 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:48:15.443 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:48:15.443 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:48:15.443 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:48:15.444 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:48:15.444 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:48:15.444 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:48:15.444 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:48:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:14 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1501526728' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:48:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:14 smithi188 ceph-mon[44101]: from='client.36614 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:48:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:14 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/574314064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:15.632 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:48:15.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:15.636 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:48:15.737 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:48:15.771 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:16.005 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.006 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.006 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:16 smithi103 ceph-mon[39629]: pgmap v1808: 27 pgs: 4 active+recovering, 23 active+clean; 7.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 127 B/s rd, 20 MiB/s wr, 857 op/s; 13605/242290 objects misplaced (5.615%); 9.6 MiB/s, 153 objects/s recovering 2024-04-23T16:48:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:16 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:48:16.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:16 smithi103 ceph-mon[39629]: from='client.36614 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:48:16.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:16 smithi103 ceph-mon[39629]: osdmap e649: 12 total, 12 up, 8 in 2024-04-23T16:48:16.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:16 smithi148 ceph-mon[44520]: pgmap v1808: 27 pgs: 4 active+recovering, 23 active+clean; 7.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 127 B/s rd, 20 MiB/s wr, 857 op/s; 13605/242290 objects misplaced (5.615%); 9.6 MiB/s, 153 objects/s recovering 2024-04-23T16:48:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:16 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:48:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:16 smithi148 ceph-mon[44520]: from='client.36614 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:48:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:16 smithi148 ceph-mon[44520]: osdmap e649: 12 total, 12 up, 8 in 2024-04-23T16:48:16.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:16 smithi188 ceph-mon[44101]: pgmap v1808: 27 pgs: 4 active+recovering, 23 active+clean; 7.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 127 B/s rd, 20 MiB/s wr, 857 op/s; 13605/242290 objects misplaced (5.615%); 9.6 MiB/s, 153 objects/s recovering 2024-04-23T16:48:16.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:16.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:16 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:48:16.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:16 smithi188 ceph-mon[44101]: from='client.36614 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:48:16.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:16 smithi188 ceph-mon[44101]: osdmap e649: 12 total, 12 up, 8 in 2024-04-23T16:48:18.224 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:48:18.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:18.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:48:18.328 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:48:18.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:18 smithi103 ceph-mon[39629]: pgmap v1810: 27 pgs: 4 active+recovering, 23 active+clean; 7.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 157 B/s rd, 1.3 MiB/s wr, 631 op/s; 13605/241438 objects misplaced (5.635%); 5.9 MiB/s, 94 objects/s recovering 2024-04-23T16:48:18.369 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:18.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:18 smithi148 ceph-mon[44520]: pgmap v1810: 27 pgs: 4 active+recovering, 23 active+clean; 7.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 157 B/s rd, 1.3 MiB/s wr, 631 op/s; 13605/241438 objects misplaced (5.635%); 5.9 MiB/s, 94 objects/s recovering 2024-04-23T16:48:18.395 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:48:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:18.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:48:18.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:18 smithi188 ceph-mon[44101]: pgmap v1810: 27 pgs: 4 active+recovering, 23 active+clean; 7.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 157 B/s rd, 1.3 MiB/s wr, 631 op/s; 13605/241438 objects misplaced (5.635%); 5.9 MiB/s, 94 objects/s recovering 2024-04-23T16:48:18.499 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:48:18.582 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:18.601 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:48:18.602 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":650,"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","created":"2024-04-23T16:08:13.477590+0000","modified":"2024-04-23T16:48:18.005520+0000","last_up_change":"2024-04-23T16:47:00.523429+0000","last_in_change":"2024-04-23T16:47:09.056375+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":32,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T16:11:22.416777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"303","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"301","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1600000858306885,"score_stable":2.1600000858306885,"optimal_score":0.67000001668930054,"raw_score_acting":1.4500000476837158,"raw_score_stable":1.4500000476837158,"primary_affinity_weighted":0.98181778192520142,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}},{"pool":8,"pool_name":"unique_pool_6","create_time":"2024-04-23T16:46:24.469390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"612","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.7799999713897705,"score_stable":3.7799999713897705,"optimal_score":0.57999998331069946,"raw_score_acting":2.190000057220459,"raw_score_stable":2.190000057220459,"primary_affinity_weighted":0.98181778192520142,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}}],"osds":[{"osd":0,"uuid":"4c084047-8f01-4b86-a60c-5592735b5862","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":609,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6801","nonce":4274040097}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6802","nonce":4274040097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6804","nonce":4274040097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6803","nonce":4274040097}]},"public_addr":"172.21.15.103:6801/4274040097","cluster_addr":"172.21.15.103:6802/4274040097","heartbeat_back_addr":"172.21.15.103:6804/4274040097","heartbeat_front_addr":"172.21.15.103:6803/4274040097","state":["exists","up"]},{"osd":1,"uuid":"3438f2bd-5fb7-432c-b383-29e18c442f20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":232,"up_from":271,"up_thru":609,"down_at":233,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6805","nonce":859282187}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6806","nonce":859282187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6808","nonce":859282187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6807","nonce":859282187}]},"public_addr":"172.21.15.103:6805/859282187","cluster_addr":"172.21.15.103:6806/859282187","heartbeat_back_addr":"172.21.15.103:6808/859282187","heartbeat_front_addr":"172.21.15.103:6807/859282187","state":["exists","up"]},{"osd":2,"uuid":"c17fffb4-8d17-4546-ba24-e5951a2a37fe","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":386,"up_from":481,"up_thru":359,"down_at":387,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6809","nonce":3324226049}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6810","nonce":3324226049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6812","nonce":3324226049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6811","nonce":3324226049}]},"public_addr":"172.21.15.103:6809/3324226049","cluster_addr":"172.21.15.103:6810/3324226049","heartbeat_back_addr":"172.21.15.103:6812/3324226049","heartbeat_front_addr":"172.21.15.103:6811/3324226049","state":["exists","up"]},{"osd":3,"uuid":"cee91a3d-55dc-4e89-b568-86821632bb1c","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":559,"up_from":619,"up_thru":540,"down_at":560,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6813","nonce":995549078}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6814","nonce":995549078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6816","nonce":995549078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6815","nonce":995549078}]},"public_addr":"172.21.15.103:6813/995549078","cluster_addr":"172.21.15.103:6814/995549078","heartbeat_back_addr":"172.21.15.103:6816/995549078","heartbeat_front_addr":"172.21.15.103:6815/995549078","state":["exists","up"]},{"osd":4,"uuid":"61d5385b-2c82-42f8-93c3-8f1baf71c65e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":609,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6800","nonce":794945071}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6801","nonce":794945071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6803","nonce":794945071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6802","nonce":794945071}]},"public_addr":"172.21.15.148:6800/794945071","cluster_addr":"172.21.15.148:6801/794945071","heartbeat_back_addr":"172.21.15.148:6803/794945071","heartbeat_front_addr":"172.21.15.148:6802/794945071","state":["exists","up"]},{"osd":5,"uuid":"5d492189-c3cd-47ad-a12a-ed4d275bcfe3","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":609,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6804","nonce":3547200711}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6805","nonce":3547200711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6807","nonce":3547200711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6806","nonce":3547200711}]},"public_addr":"172.21.15.148:6804/3547200711","cluster_addr":"172.21.15.148:6805/3547200711","heartbeat_back_addr":"172.21.15.148:6807/3547200711","heartbeat_front_addr":"172.21.15.148:6806/3547200711","state":["exists","up"]},{"osd":6,"uuid":"89f2379e-cf14-4e69-8747-7535db81b70b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":623,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6808","nonce":366919758}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6809","nonce":366919758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6811","nonce":366919758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6810","nonce":366919758}]},"public_addr":"172.21.15.148:6808/366919758","cluster_addr":"172.21.15.148:6809/366919758","heartbeat_back_addr":"172.21.15.148:6811/366919758","heartbeat_front_addr":"172.21.15.148:6810/366919758","state":["exists","up"]},{"osd":7,"uuid":"f27b035a-558f-4528-80ea-2a00d03f6539","up":1,"in":1,"weight":1,"primary_affinity":0.7818145751953125,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":540,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6812","nonce":3719550888}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6813","nonce":3719550888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6815","nonce":3719550888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6814","nonce":3719550888}]},"public_addr":"172.21.15.148:6812/3719550888","cluster_addr":"172.21.15.148:6813/3719550888","heartbeat_back_addr":"172.21.15.148:6815/3719550888","heartbeat_front_addr":"172.21.15.148:6814/3719550888","state":["exists","up"]},{"osd":8,"uuid":"a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":623,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6800","nonce":1151408595}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6801","nonce":1151408595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6803","nonce":1151408595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6802","nonce":1151408595}]},"public_addr":"172.21.15.188:6800/1151408595","cluster_addr":"172.21.15.188:6801/1151408595","heartbeat_back_addr":"172.21.15.188:6803/1151408595","heartbeat_front_addr":"172.21.15.188:6802/1151408595","state":["exists","up"]},{"osd":9,"uuid":"124d9b05-11eb-4622-b6a2-c4d5bd60b7e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":623,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6804","nonce":306501862}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6805","nonce":306501862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6807","nonce":306501862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6806","nonce":306501862}]},"public_addr":"172.21.15.188:6804/306501862","cluster_addr":"172.21.15.188:6805/306501862","heartbeat_back_addr":"172.21.15.188:6807/306501862","heartbeat_front_addr":"172.21.15.188:6806/306501862","state":["exists","up"]},{"osd":10,"uuid":"3dfb9070-e534-4ea4-a25e-ee1c51297447","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6808","nonce":1231263138}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6809","nonce":1231263138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6811","nonce":1231263138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6810","nonce":1231263138}]},"public_addr":"172.21.15.188:6808/1231263138","cluster_addr":"172.21.15.188:6809/1231263138","heartbeat_back_addr":"172.21.15.188:6811/1231263138","heartbeat_front_addr":"172.21.15.188:6810/1231263138","state":["exists","up"]},{"osd":11,"uuid":"bc163925-f623-4f63-ab89-35e2d511a730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":623,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6812","nonce":457992661}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6813","nonce":457992661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6815","nonce":457992661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6814","nonce":457992661}]},"public_addr":"172.21.15.188:6812/457992661","cluster_addr":"172.21.15.188:6813/457992661","heartbeat_back_addr":"172.21.15.188:6815/457992661","heartbeat_front_addr":"172.21.15.188:6814/457992661","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:47:38.388271+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:47:32.791559+0000","dead_epoch":232},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-23T16:47:35.858867+0000","dead_epoch":386},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-23T16:48:17.817909+0000","dead_epoch":559},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:47:16.779276+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-23T16:47:15.583868+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:47:31.631089+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:47:15.840541+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:47:14.570847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:47:32.310824+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-23T16:48:10.972639+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:47:15.565407+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6800/3535470771":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1188058303":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1510553721":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/281182801":"2024-04-24T16:08:44.450632+0000","172.21.15.103:6800/2049124518":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/466241963":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/2726838698":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/596230573":"2024-04-24T16:09:14.324448+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T16:48:18.791 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:48:19.167 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:48:19.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:19.181 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:48:19.282 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:48:19.309 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:19 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:48:19.309 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:19 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:48:19.309 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:19 smithi148 ceph-mon[44520]: osdmap e650: 12 total, 12 up, 8 in 2024-04-23T16:48:19.309 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:19 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:48:19.309 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:19 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:48:19.309 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:19 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3007336091' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:48:19.311 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:19.331 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:48:19.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:19.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:48:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:19 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:48:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:19 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:48:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:19 smithi103 ceph-mon[39629]: osdmap e650: 12 total, 12 up, 8 in 2024-04-23T16:48:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:19 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:48:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:19 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:48:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:19 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3007336091' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:48:19.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:48:19.471 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:19 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:48:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:19 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:48:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:19 smithi188 ceph-mon[44101]: osdmap e650: 12 total, 12 up, 8 in 2024-04-23T16:48:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:19 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:48:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:19 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:48:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:19 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3007336091' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:48:19.493 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:48:19.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:19.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:48:19.506 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-23T16:48:19.556 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:19.607 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:48:19.635 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:19.776 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:48:19.833 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:48:19.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:19.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:48:19.938 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:48:20.008 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:20.064 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:20 smithi103 ceph-mon[39629]: pgmap v1812: 27 pgs: 4 active+recovering, 23 active+clean; 7.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 147 B/s rd, 314 KiB/s wr, 756 op/s; 13447/238234 objects misplaced (5.644%); 7.0 MiB/s, 111 objects/s recovering 2024-04-23T16:48:20.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:20 smithi103 ceph-mon[39629]: osdmap e651: 12 total, 12 up, 8 in 2024-04-23T16:48:20.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:20 smithi148 ceph-mon[44520]: pgmap v1812: 27 pgs: 4 active+recovering, 23 active+clean; 7.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 147 B/s rd, 314 KiB/s wr, 756 op/s; 13447/238234 objects misplaced (5.644%); 7.0 MiB/s, 111 objects/s recovering 2024-04-23T16:48:20.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:20 smithi148 ceph-mon[44520]: osdmap e651: 12 total, 12 up, 8 in 2024-04-23T16:48:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:20 smithi188 ceph-mon[44101]: pgmap v1812: 27 pgs: 4 active+recovering, 23 active+clean; 7.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 147 B/s rd, 314 KiB/s wr, 756 op/s; 13447/238234 objects misplaced (5.644%); 7.0 MiB/s, 111 objects/s recovering 2024-04-23T16:48:20.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:20 smithi188 ceph-mon[44101]: osdmap e651: 12 total, 12 up, 8 in 2024-04-23T16:48:20.983 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:48:20.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:20.990 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:48:21.058 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:21.091 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:48:21.151 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:21.201 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:48:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:21.204 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:48:21.305 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:48:21.384 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:21.440 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:48:21.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:21.447 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:48:21.548 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:48:21.587 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:21.663 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:48:21.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:21.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:48:21.771 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:48:21.798 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:21.818 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:48:21.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:21.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:48:21.922 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:48:21.950 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:21.970 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:48:21.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:21.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:48:22.074 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:48:22.106 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:22.208 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:48:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:48:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:48:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:48:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:48:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:48:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:48:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:48:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:48:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:48:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:48:22.211 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:48:22.213 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:48:22.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:22.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:48:22.215 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:48:22.215 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:48:22.215 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:48:22.215 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:48:22.215 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:48:22.215 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:48:22.215 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:48:22.215 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:48:22.215 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:48:22.215 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:48:22.216 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:48:22.216 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:48:22.316 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:48:22.342 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:22 smithi148 ceph-mon[44520]: pgmap v1814: 27 pgs: 4 active+recovering, 23 active+clean; 7.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 596 op/s; 12960/235128 objects misplaced (5.512%); 6.7 MiB/s, 107 objects/s recovering 2024-04-23T16:48:22.342 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:22 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1543358618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:22.344 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:22.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:22 smithi103 ceph-mon[39629]: pgmap v1814: 27 pgs: 4 active+recovering, 23 active+clean; 7.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 596 op/s; 12960/235128 objects misplaced (5.512%); 6.7 MiB/s, 107 objects/s recovering 2024-04-23T16:48:22.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:22 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1543358618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:22.364 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:48:22.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:22.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:48:22.431 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:48:22.432 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:48:22.457 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:22 smithi188 ceph-mon[44101]: pgmap v1814: 27 pgs: 4 active+recovering, 23 active+clean; 7.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 596 op/s; 12960/235128 objects misplaced (5.512%); 6.7 MiB/s, 107 objects/s recovering 2024-04-23T16:48:22.457 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:22 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1543358618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:22.469 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:48:22.547 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:22.600 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:48:22.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:22.602 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:48:22.703 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:48:22.732 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:23.084 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:48:23.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:23.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:48:23.188 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:48:23.215 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:23.236 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:48:23.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:23.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:48:23.339 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:48:23.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.368 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:23.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:23.520 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:48:23.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:23.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:48:23.627 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:48:23.654 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:23.663 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-23T16:48:23.854 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:48:23.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:23.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:48:23.959 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:48:23.987 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:24.006 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:48:24.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:24.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:48:24.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:24.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:24 smithi103 ceph-mon[39629]: pgmap v1815: 27 pgs: 4 active+recovering, 23 active+clean; 7.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 12783/231904 objects misplaced (5.512%); 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:48:24.109 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:48:24.122 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:48:24.137 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:24.157 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:48:24.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:24.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:48:24.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:48:24.294 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:24.319 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:48:24.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:24.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:48:24.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:24.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:24 smithi148 ceph-mon[44520]: pgmap v1815: 27 pgs: 4 active+recovering, 23 active+clean; 7.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 12783/231904 objects misplaced (5.512%); 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:48:24.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:24.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:24 smithi188 ceph-mon[44101]: pgmap v1815: 27 pgs: 4 active+recovering, 23 active+clean; 7.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 12783/231904 objects misplaced (5.512%); 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:48:24.428 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:48:24.455 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:24.475 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:48:24.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:24.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:48:24.579 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:48:24.666 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:24.735 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:48:24.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:24.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:48:24.840 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:48:24.913 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:25.087 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:48:25.102 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:48:25.120 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:48:25.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:25.129 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:48:25.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:25 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3670962553' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:48:25.229 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:48:25.270 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:25.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:25 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3670962553' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:48:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:25 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3670962553' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:48:25.744 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:48:25.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:25.748 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:48:25.842 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:25.848 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:48:25.884 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:26.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:26 smithi103 ceph-mon[39629]: pgmap v1816: 27 pgs: 4 active+recovering, 23 active+clean; 6.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 861 op/s; 12306/227650 objects misplaced (5.406%); 10 MiB/s, 162 objects/s recovering 2024-04-23T16:48:26.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:26 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:48:26.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:26 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3670962553' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:48:26.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:26 smithi103 ceph-mon[39629]: osdmap e652: 12 total, 12 up, 8 in 2024-04-23T16:48:26.136 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:48:26.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:26.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:48:26.241 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:48:26.268 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:26.287 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:48:26.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:26.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:48:26.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:26 smithi148 ceph-mon[44520]: pgmap v1816: 27 pgs: 4 active+recovering, 23 active+clean; 6.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 861 op/s; 12306/227650 objects misplaced (5.406%); 10 MiB/s, 162 objects/s recovering 2024-04-23T16:48:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:26 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:48:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:26 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3670962553' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:48:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:26 smithi148 ceph-mon[44520]: osdmap e652: 12 total, 12 up, 8 in 2024-04-23T16:48:26.390 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:48:26.468 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:26 smithi188 ceph-mon[44101]: pgmap v1816: 27 pgs: 4 active+recovering, 23 active+clean; 6.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 861 op/s; 12306/227650 objects misplaced (5.406%); 10 MiB/s, 162 objects/s recovering 2024-04-23T16:48:26.468 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:26 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:48:26.468 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:26 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3670962553' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:48:26.468 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:26 smithi188 ceph-mon[44101]: osdmap e652: 12 total, 12 up, 8 in 2024-04-23T16:48:26.487 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:26.539 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:48:26.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:26.542 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:48:26.643 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:48:26.673 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:26.693 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:48:26.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:26.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:48:26.796 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:48:26.825 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:26.852 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:48:26.852 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:48:27.274 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:48:27.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:27.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:48:27.383 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:48:27.468 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:27.702 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:48:27.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:27.707 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:48:27.720 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-23T16:48:27.808 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:48:27.811 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:48:27.830 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:48:27.852 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:48:27.873 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:48:27.883 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:27.885 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:48:27.897 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:48:27.919 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:48:27.943 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:48:27.964 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:48:27.965 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:48:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:27.974 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:48:27.995 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:48:28.042 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:48:28.075 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:48:28.083 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:48:28.123 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:48:28.124 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:48:28.125 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:48:28.125 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:48:28.125 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:48:28.125 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:48:28.125 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:48:28.125 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:48:28.125 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:48:28.125 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:48:28.125 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:48:28.125 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:48:28.126 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:48:28.139 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:28.187 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:48:28.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:28.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:48:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:28 smithi188 ceph-mon[44101]: pgmap v1818: 27 pgs: 4 active+recovering, 23 active+clean; 6.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 12306/226730 objects misplaced (5.428%); 8.9 MiB/s, 142 objects/s recovering 2024-04-23T16:48:28.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:28 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4079905852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:28.267 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:48:28.291 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:48:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:28 smithi103 ceph-mon[39629]: pgmap v1818: 27 pgs: 4 active+recovering, 23 active+clean; 6.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 12306/226730 objects misplaced (5.428%); 8.9 MiB/s, 142 objects/s recovering 2024-04-23T16:48:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:28 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4079905852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:28.361 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:28.373 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:28.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:28 smithi148 ceph-mon[44520]: pgmap v1818: 27 pgs: 4 active+recovering, 23 active+clean; 6.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 12306/226730 objects misplaced (5.428%); 8.9 MiB/s, 142 objects/s recovering 2024-04-23T16:48:28.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:28 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4079905852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:28.533 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:48:28.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:28.543 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:48:28.644 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:48:28.674 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:28.697 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:48:28.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:28.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:48:28.802 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:48:28.831 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:28.851 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:48:28.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:28.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:48:28.959 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:48:28.986 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:29.005 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:48:29.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:29.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:48:29.107 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.108 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.109 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:48:29.137 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:29.157 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:48:29.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:29.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:48:29.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:29 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:48:29.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:29 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:48:29.261 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:48:29.325 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:29 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:48:29.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:29 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:48:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:29 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:48:29.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:29 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:48:29.834 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:48:29.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:29.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:48:29.939 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:48:29.968 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:29.988 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:48:29.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:29.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:48:30.092 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:48:30.108 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:30 smithi188 ceph-mon[44101]: 1.9 scrub starts 2024-04-23T16:48:30.108 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:30 smithi188 ceph-mon[44101]: 1.9 scrub ok 2024-04-23T16:48:30.124 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:30.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:30 smithi103 ceph-mon[39629]: 1.9 scrub starts 2024-04-23T16:48:30.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:30 smithi103 ceph-mon[39629]: 1.9 scrub ok 2024-04-23T16:48:30.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:30 smithi103 ceph-mon[39629]: pgmap v1819: 27 pgs: 4 active+recovering, 23 active+clean; 6.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 12117/223456 objects misplaced (5.423%); 8.4 MiB/s, 134 objects/s recovering 2024-04-23T16:48:30.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:30 smithi148 ceph-mon[44520]: 1.9 scrub starts 2024-04-23T16:48:30.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:30 smithi148 ceph-mon[44520]: 1.9 scrub ok 2024-04-23T16:48:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:30 smithi148 ceph-mon[44520]: pgmap v1819: 27 pgs: 4 active+recovering, 23 active+clean; 6.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 12117/223456 objects misplaced (5.423%); 8.4 MiB/s, 134 objects/s recovering 2024-04-23T16:48:30.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:30 smithi188 ceph-mon[44101]: pgmap v1819: 27 pgs: 4 active+recovering, 23 active+clean; 6.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 12117/223456 objects misplaced (5.423%); 8.4 MiB/s, 134 objects/s recovering 2024-04-23T16:48:30.975 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:48:30.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:30.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:48:31.079 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:48:31.109 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:31.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:31 smithi188 ceph-mon[44101]: pgmap v1820: 27 pgs: 4 active+recovering, 23 active+clean; 6.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 11637/220332 objects misplaced (5.282%); 8.3 MiB/s, 132 objects/s recovering 2024-04-23T16:48:31.131 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:48:31.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:31.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:48:31.239 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:48:31.265 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:31.285 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:48:31.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:31.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:48:31.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:31 smithi103 ceph-mon[39629]: pgmap v1820: 27 pgs: 4 active+recovering, 23 active+clean; 6.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 11637/220332 objects misplaced (5.282%); 8.3 MiB/s, 132 objects/s recovering 2024-04-23T16:48:31.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:31 smithi148 ceph-mon[44520]: pgmap v1820: 27 pgs: 4 active+recovering, 23 active+clean; 6.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 11637/220332 objects misplaced (5.282%); 8.3 MiB/s, 132 objects/s recovering 2024-04-23T16:48:31.389 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:48:31.418 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:31.438 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:48:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:31.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:48:31.536 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:48:31.542 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:48:31.570 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:31.591 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:48:31.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:31.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:48:31.695 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:48:31.726 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:31.756 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:48:31.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:31.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:48:31.858 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:31.870 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:48:31.896 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:31.917 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:48:31.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:31.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:48:31.955 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:48:31.955 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:48:31.955 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-04-23T16:48:31.955 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:48:32.022 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:48:32.132 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:48:32.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:32 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3186412349' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:48:32.413 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:32 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3186412349' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:48:32.419 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:32.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:32 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3186412349' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:48:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:33 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:48:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:33 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:48:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:33 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3186412349' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:48:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:33 smithi103 ceph-mon[39629]: osdmap e653: 12 total, 12 up, 8 in 2024-04-23T16:48:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:33 smithi103 ceph-mon[39629]: pgmap v1822: 27 pgs: 4 active+recovering, 23 active+clean; 6.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 645 op/s; 11441/217314 objects misplaced (5.265%); 6.8 MiB/s, 108 objects/s recovering 2024-04-23T16:48:33.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:33 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:48:33.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:33 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:48:33.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:33 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3186412349' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:48:33.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:33 smithi188 ceph-mon[44101]: osdmap e653: 12 total, 12 up, 8 in 2024-04-23T16:48:33.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:33 smithi188 ceph-mon[44101]: pgmap v1822: 27 pgs: 4 active+recovering, 23 active+clean; 6.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 645 op/s; 11441/217314 objects misplaced (5.265%); 6.8 MiB/s, 108 objects/s recovering 2024-04-23T16:48:33.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:33 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:48:33.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:33 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:48:33.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:33 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3186412349' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:48:33.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:33 smithi148 ceph-mon[44520]: osdmap e653: 12 total, 12 up, 8 in 2024-04-23T16:48:33.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:33 smithi148 ceph-mon[44520]: pgmap v1822: 27 pgs: 4 active+recovering, 23 active+clean; 6.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 645 op/s; 11441/217314 objects misplaced (5.265%); 6.8 MiB/s, 108 objects/s recovering 2024-04-23T16:48:33.770 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:48:33.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:33.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:48:33.876 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:48:33.939 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:34.507 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:48:34.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:34.517 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:48:34.618 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:48:34.650 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:34.723 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:48:34.729 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:34 smithi103 ceph-mon[39629]: 1.5 scrub starts 2024-04-23T16:48:34.729 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:34 smithi103 ceph-mon[39629]: 1.5 scrub ok 2024-04-23T16:48:34.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:34.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:48:34.831 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:48:34.856 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:34 smithi148 ceph-mon[44520]: 1.5 scrub starts 2024-04-23T16:48:34.856 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:34 smithi148 ceph-mon[44520]: 1.5 scrub ok 2024-04-23T16:48:34.858 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:34.879 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:48:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:34.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:48:34.982 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:48:34.983 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:34 smithi188 ceph-mon[44101]: 1.5 scrub starts 2024-04-23T16:48:34.984 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:34 smithi188 ceph-mon[44101]: 1.5 scrub ok 2024-04-23T16:48:35.033 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:35.343 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:48:35.366 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:48:35.394 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:48:35.415 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:48:35.435 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:48:35.457 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:48:35.482 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:48:35.504 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:48:35.532 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:48:35.600 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:48:35.634 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:48:35.662 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:48:35.664 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:48:35.664 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:48:35.664 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:48:35.664 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:48:35.664 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:48:35.664 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:48:35.664 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:48:35.664 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:48:35.664 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:48:35.664 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:48:35.664 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:48:35.664 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:48:35.823 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:35 smithi188 ceph-mon[44101]: pgmap v1823: 27 pgs: 4 active+recovering, 23 active+clean; 6.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 10950/213264 objects misplaced (5.134%); 8.6 MiB/s, 137 objects/s recovering 2024-04-23T16:48:35.823 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:35 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/322339862' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:35.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:35 smithi103 ceph-mon[39629]: pgmap v1823: 27 pgs: 4 active+recovering, 23 active+clean; 6.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 10950/213264 objects misplaced (5.134%); 8.6 MiB/s, 137 objects/s recovering 2024-04-23T16:48:35.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:35 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/322339862' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:35.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:35 smithi148 ceph-mon[44520]: pgmap v1823: 27 pgs: 4 active+recovering, 23 active+clean; 6.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 10950/213264 objects misplaced (5.134%); 8.6 MiB/s, 137 objects/s recovering 2024-04-23T16:48:35.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:35 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/322339862' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:36.211 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:48:36.534 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:36.792 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.792 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.792 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.792 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.792 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.792 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.792 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.792 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.792 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.792 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.793 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.793 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:36.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:37 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:48:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:37 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:48:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:37 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:48:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:37 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:48:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:37 smithi103 ceph-mon[39629]: pgmap v1824: 27 pgs: 4 active+recovering, 23 active+clean; 6.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 10950/212318 objects misplaced (5.157%); 8.5 MiB/s, 135 objects/s recovering 2024-04-23T16:48:37.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:37 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:48:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:37 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:48:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:37 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:48:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:37 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:48:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:37 smithi148 ceph-mon[44520]: pgmap v1824: 27 pgs: 4 active+recovering, 23 active+clean; 6.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 10950/212318 objects misplaced (5.157%); 8.5 MiB/s, 135 objects/s recovering 2024-04-23T16:48:37.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:37 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:48:37.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:37 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:48:37.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:37 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:48:37.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:37 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:48:37.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:37 smithi188 ceph-mon[44101]: pgmap v1824: 27 pgs: 4 active+recovering, 23 active+clean; 6.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 10950/212318 objects misplaced (5.157%); 8.5 MiB/s, 135 objects/s recovering 2024-04-23T16:48:38.483 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:48:38.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:38 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:48:38.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:38 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:48:38.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:38 smithi103 ceph-mon[39629]: osdmap e654: 12 total, 12 up, 8 in 2024-04-23T16:48:38.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:38 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:48:38.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:38 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:48:38.824 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:38.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:38 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:48:38.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:38 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:48:38.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:38 smithi148 ceph-mon[44520]: osdmap e654: 12 total, 12 up, 8 in 2024-04-23T16:48:38.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:38 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:48:38.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:38 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:48:38.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:38 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:48:38.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:38 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:48:38.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:38 smithi188 ceph-mon[44101]: osdmap e654: 12 total, 12 up, 8 in 2024-04-23T16:48:38.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:38 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:48:38.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:38 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:48:39.507 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-23T16:48:39.507 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:48:39.514 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:48:39.520 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-23T16:48:39.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:39.524 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:48:39.552 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:48:39.625 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:48:39.654 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:39.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:39 smithi188 ceph-mon[44101]: osdmap e655: 12 total, 12 up, 8 in 2024-04-23T16:48:39.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:39 smithi188 ceph-mon[44101]: 1.7 scrub starts 2024-04-23T16:48:39.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:39 smithi188 ceph-mon[44101]: 1.7 scrub ok 2024-04-23T16:48:39.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:39 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/487530377' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:48:39.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:39 smithi188 ceph-mon[44101]: from='client.36653 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:48:39.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:39 smithi188 ceph-mon[44101]: pgmap v1827: 27 pgs: 4 active+recovering, 23 active+clean; 6.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 595 op/s; 10750/209226 objects misplaced (5.138%); 6.4 MiB/s, 101 objects/s recovering 2024-04-23T16:48:39.806 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:48:39.808 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:39 smithi103 ceph-mon[39629]: osdmap e655: 12 total, 12 up, 8 in 2024-04-23T16:48:39.808 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:39 smithi103 ceph-mon[39629]: 1.7 scrub starts 2024-04-23T16:48:39.808 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:39 smithi103 ceph-mon[39629]: 1.7 scrub ok 2024-04-23T16:48:39.808 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:39 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/487530377' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:48:39.808 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:39 smithi103 ceph-mon[39629]: from='client.36653 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:48:39.808 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:39 smithi103 ceph-mon[39629]: pgmap v1827: 27 pgs: 4 active+recovering, 23 active+clean; 6.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 595 op/s; 10750/209226 objects misplaced (5.138%); 6.4 MiB/s, 101 objects/s recovering 2024-04-23T16:48:39.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:39.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:48:39.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:39 smithi148 ceph-mon[44520]: osdmap e655: 12 total, 12 up, 8 in 2024-04-23T16:48:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:39 smithi148 ceph-mon[44520]: 1.7 scrub starts 2024-04-23T16:48:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:39 smithi148 ceph-mon[44520]: 1.7 scrub ok 2024-04-23T16:48:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:39 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/487530377' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:48:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:39 smithi148 ceph-mon[44520]: from='client.36653 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:48:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:39 smithi148 ceph-mon[44520]: pgmap v1827: 27 pgs: 4 active+recovering, 23 active+clean; 6.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 595 op/s; 10750/209226 objects misplaced (5.138%); 6.4 MiB/s, 101 objects/s recovering 2024-04-23T16:48:39.910 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:48:39.938 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:40.011 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:48:40.037 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:48:40.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:40.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:48:40.141 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:48:40.209 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:40.332 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:48:40.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:40.335 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:48:40.436 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:48:40.463 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:40.482 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:48:40.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:40.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:48:40.634 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:48:40.665 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:40.808 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:40 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:48:40.808 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:40 smithi188 ceph-mon[44101]: from='client.36653 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:48:40.808 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:40 smithi188 ceph-mon[44101]: osdmap e656: 12 total, 12 up, 8 in 2024-04-23T16:48:40.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:40 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:48:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:40 smithi103 ceph-mon[39629]: from='client.36653 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:48:40.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:40 smithi103 ceph-mon[39629]: osdmap e656: 12 total, 12 up, 8 in 2024-04-23T16:48:40.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:40 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:48:40.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:40 smithi148 ceph-mon[44520]: from='client.36653 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:48:40.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:40 smithi148 ceph-mon[44520]: osdmap e656: 12 total, 12 up, 8 in 2024-04-23T16:48:41.115 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:48:41.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:41.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:48:41.220 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:48:41.252 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:41.276 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:48:41.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:41.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:48:41.383 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:48:41.412 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:41.431 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:48:41.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:41.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:48:41.536 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:48:41.569 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:41.633 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:41.644 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:48:41.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:41.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:48:41.653 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:41 smithi103 ceph-mon[39629]: pgmap v1829: 27 pgs: 4 active+recovering, 23 active+clean; 6.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 10265/206272 objects misplaced (4.976%); 7.1 MiB/s, 114 objects/s recovering 2024-04-23T16:48:41.682 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:48:41.705 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:48:41.729 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:48:41.752 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:48:41.754 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:48:41.774 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:48:41.795 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:48:41.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:41 smithi188 ceph-mon[44101]: pgmap v1829: 27 pgs: 4 active+recovering, 23 active+clean; 6.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 10265/206272 objects misplaced (4.976%); 7.1 MiB/s, 114 objects/s recovering 2024-04-23T16:48:41.818 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:48:41.822 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:41.838 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:48:41.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:41 smithi148 ceph-mon[44520]: pgmap v1829: 27 pgs: 4 active+recovering, 23 active+clean; 6.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 10265/206272 objects misplaced (4.976%); 7.1 MiB/s, 114 objects/s recovering 2024-04-23T16:48:41.879 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:48:41.916 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:48:41.923 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:48:41.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:41.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:48:41.954 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:48:41.985 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:48:41.987 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:48:41.987 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:48:41.987 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:48:41.987 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:48:41.987 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:48:41.987 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:48:41.987 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:48:41.987 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:48:41.988 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:48:41.988 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:48:41.988 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:48:41.988 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:48:42.029 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:48:42.062 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:42.313 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:48:42.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:42.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:48:42.417 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:48:42.498 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:42.597 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:48:42.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:42 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:48:42.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:42 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:48:42.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:42 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2735160165' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:42.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:42 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:42.601 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:48:42.617 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:48:42.703 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:48:42.730 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:42.752 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:48:42.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:42.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:48:42.856 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:48:42.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:42 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:48:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:42 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:48:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:42 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2735160165' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:42 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.884 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:48:42.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:42.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:48:42.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:42 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:48:42.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:42 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:48:42.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:42 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2735160165' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:42.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:42.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:42 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:43.011 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:48:43.039 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:43.058 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:48:43.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:43.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:48:43.162 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:48:43.192 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:43.319 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:48:43.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:43.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:48:43.424 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:48:43.475 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:43.522 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:48:43.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:43.528 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:48:43.630 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:48:43.656 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:43.680 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:48:43.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:43.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:48:43.784 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:48:43.814 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:43.833 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:48:43.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:43.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:48:43.836 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:43.836 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:43 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:43.836 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:43 smithi148 ceph-mon[44520]: osdmap e657: 12 total, 12 up, 8 in 2024-04-23T16:48:43.836 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:43 smithi148 ceph-mon[44520]: pgmap v1831: 27 pgs: 4 active+recovering, 23 active+clean; 6.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 643 op/s; 10059/203114 objects misplaced (4.952%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:48:43.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:43.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:43 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:43.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:43 smithi103 ceph-mon[39629]: osdmap e657: 12 total, 12 up, 8 in 2024-04-23T16:48:43.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:43 smithi103 ceph-mon[39629]: pgmap v1831: 27 pgs: 4 active+recovering, 23 active+clean; 6.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 643 op/s; 10059/203114 objects misplaced (4.952%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:48:43.937 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:48:43.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:43.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:43 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:43.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:43 smithi188 ceph-mon[44101]: osdmap e657: 12 total, 12 up, 8 in 2024-04-23T16:48:43.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:43 smithi188 ceph-mon[44101]: pgmap v1831: 27 pgs: 4 active+recovering, 23 active+clean; 6.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 643 op/s; 10059/203114 objects misplaced (4.952%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:48:44.008 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:44.044 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:48:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:44.047 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:48:44.148 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:48:44.180 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:44.211 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:48:44.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:44.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:48:44.221 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:44.316 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:48:44.345 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:44.459 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:48:44.461 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:48:44.499 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:48:44.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:44.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:48:44.531 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:48:44.532 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:48:44.532 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:48:44.532 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:48:44.532 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:48:44.605 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:48:44.632 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:44.654 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:48:44.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:44.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:48:44.758 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:48:44.785 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:44.807 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:48:44.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:44.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:48:44.911 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:48:44.981 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:45.354 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:48:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:45 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:48:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:45 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:48:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:45 smithi103 ceph-mon[39629]: from='client.36665 v1:172.21.15.103:0/336352729' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:48:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:45 smithi103 ceph-mon[39629]: osdmap e658: 12 total, 12 up, 8 in 2024-04-23T16:48:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:45 smithi103 ceph-mon[39629]: pgmap v1833: 27 pgs: 4 active+recovering, 23 active+clean; 6.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 9542/199068 objects misplaced (4.793%); 13 MiB/s, 201 objects/s recovering 2024-04-23T16:48:45.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:45 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:48:45.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:45 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:48:45.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:45 smithi148 ceph-mon[44520]: from='client.36665 v1:172.21.15.103:0/336352729' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:48:45.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:45 smithi148 ceph-mon[44520]: osdmap e658: 12 total, 12 up, 8 in 2024-04-23T16:48:45.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:45 smithi148 ceph-mon[44520]: pgmap v1833: 27 pgs: 4 active+recovering, 23 active+clean; 6.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 9542/199068 objects misplaced (4.793%); 13 MiB/s, 201 objects/s recovering 2024-04-23T16:48:45.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:45 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:48:45.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:45 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:48:45.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:45 smithi188 ceph-mon[44101]: from='client.36665 v1:172.21.15.103:0/336352729' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:48:45.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:45 smithi188 ceph-mon[44101]: osdmap e658: 12 total, 12 up, 8 in 2024-04-23T16:48:45.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:45 smithi188 ceph-mon[44101]: pgmap v1833: 27 pgs: 4 active+recovering, 23 active+clean; 6.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 9542/199068 objects misplaced (4.793%); 13 MiB/s, 201 objects/s recovering 2024-04-23T16:48:46.117 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1831,"stamp":"2024-04-23T16:48:42.914965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6656033336,"num_objects":101557,"num_object_clones":0,"num_object_copies":203114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10059,"num_objects_unfound":0,"num_objects_dirty":101557,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149459,"num_write_kb":8032385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10668,"num_bytes_recovered":699409472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150988,"ondisk_log_size":150988,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17276100,"kb_used_data":12410972,"kb_used_omap":242,"kb_used_meta":4864717,"kb_avail":732685116,"statfs":{"total":767960285184,"available":750269558784,"internally_reserved":0,"allocated":12708835328,"data_stored":12705923540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":247821,"internal_metadata":4981471219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-227540992,"num_objects":-3472,"num_object_clones":0,"num_object_copies":-6944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-691,"num_objects_unfound":0,"num_objects_dirty":-3472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3472,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":691,"num_bytes_recovered":45285376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.395707"},"pg_stats":[{"pgid":"8.7","version":"656'9295","reported_seq":9474,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-04-23T16:48:40.936960+0000","last_change":"2024-04-23T16:47:45.637166+0000","last_active":"2024-04-23T16:48:40.936960+0000","last_peered":"2024-04-23T16:48:40.936960+0000","last_clean":"2024-04-23T16:48:40.936960+0000","last_became_active":"2024-04-23T16:46:26.363422+0000","last_became_peered":"2024-04-23T16:46:26.363422+0000","last_unstale":"2024-04-23T16:48:40.936960+0000","last_undegraded":"2024-04-23T16:48:40.936960+0000","last_fullsized":"2024-04-23T16:48:40.936960+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"641'6798","last_scrub_stamp":"2024-04-23T16:47:45.637054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:47:45.637054+0000","objects_scrubbed":6770,"log_size":9295,"log_dups_size":0,"ondisk_log_size":9295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:01.884159+0000","scrub_duration":1366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409927680,"num_objects":6255,"num_object_clones":0,"num_object_copies":12510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9295,"num_write_kb":497600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"656'9551","reported_seq":9655,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-04-23T16:48:42.082251+0000","last_change":"2024-04-23T16:46:26.365698+0000","last_active":"2024-04-23T16:48:42.082251+0000","last_peered":"2024-04-23T16:48:42.082251+0000","last_clean":"2024-04-23T16:48:42.082251+0000","last_became_active":"2024-04-23T16:46:26.365309+0000","last_became_peered":"2024-04-23T16:46:26.365309+0000","last_unstale":"2024-04-23T16:48:42.082251+0000","last_undegraded":"2024-04-23T16:48:42.082251+0000","last_fullsized":"2024-04-23T16:48:42.082251+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9551,"log_dups_size":0,"ondisk_log_size":9551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411369472,"num_objects":6277,"num_object_clones":0,"num_object_copies":12554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9551,"num_write_kb":506496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.5","version":"656'9135","reported_seq":9239,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-04-23T16:48:39.613085+0000","last_change":"2024-04-23T16:46:26.365075+0000","last_active":"2024-04-23T16:48:39.613085+0000","last_peered":"2024-04-23T16:48:39.613085+0000","last_clean":"2024-04-23T16:48:39.613085+0000","last_became_active":"2024-04-23T16:46:26.364780+0000","last_became_peered":"2024-04-23T16:46:26.364780+0000","last_unstale":"2024-04-23T16:48:39.613085+0000","last_undegraded":"2024-04-23T16:48:39.613085+0000","last_fullsized":"2024-04-23T16:48:39.613085+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9135,"log_dups_size":0,"ondisk_log_size":9135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414253056,"num_objects":6321,"num_object_clones":0,"num_object_copies":12642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9135,"num_write_kb":494592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"655'9555","reported_seq":14518,"reported_epoch":655,"state":"active+recovering","last_fresh":"2024-04-23T16:48:39.422006+0000","last_change":"2024-04-23T16:47:10.280064+0000","last_active":"2024-04-23T16:48:39.422006+0000","last_peered":"2024-04-23T16:48:39.422006+0000","last_clean":"2024-04-23T16:47:09.049801+0000","last_became_active":"2024-04-23T16:47:10.244722+0000","last_became_peered":"2024-04-23T16:47:10.244722+0000","last_unstale":"2024-04-23T16:48:39.422006+0000","last_undegraded":"2024-04-23T16:48:39.422006+0000","last_fullsized":"2024-04-23T16:48:39.422006+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9555,"log_dups_size":0,"ondisk_log_size":9555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429326336,"num_objects":6551,"num_object_clones":0,"num_object_copies":13102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2906,"num_objects_unfound":0,"num_objects_dirty":6551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9555,"num_write_kb":515392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2429,"num_bytes_recovered":159186944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"5,9","objects":2906},{"shards":"9,11","objects":3645}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"656'9210","reported_seq":9314,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-04-23T16:48:39.543658+0000","last_change":"2024-04-23T16:46:26.365492+0000","last_active":"2024-04-23T16:48:39.543658+0000","last_peered":"2024-04-23T16:48:39.543658+0000","last_clean":"2024-04-23T16:48:39.543658+0000","last_became_active":"2024-04-23T16:46:26.365000+0000","last_became_peered":"2024-04-23T16:46:26.365000+0000","last_unstale":"2024-04-23T16:48:39.543658+0000","last_undegraded":"2024-04-23T16:48:39.543658+0000","last_fullsized":"2024-04-23T16:48:39.543658+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9210,"log_dups_size":0,"ondisk_log_size":9210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417464320,"num_objects":6370,"num_object_clones":0,"num_object_copies":12740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9210,"num_write_kb":498560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"656'9411","reported_seq":9515,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-04-23T16:48:42.162198+0000","last_change":"2024-04-23T16:46:26.365668+0000","last_active":"2024-04-23T16:48:42.162198+0000","last_peered":"2024-04-23T16:48:42.162198+0000","last_clean":"2024-04-23T16:48:42.162198+0000","last_became_active":"2024-04-23T16:46:26.364349+0000","last_became_peered":"2024-04-23T16:46:26.364349+0000","last_unstale":"2024-04-23T16:48:42.162198+0000","last_undegraded":"2024-04-23T16:48:42.162198+0000","last_fullsized":"2024-04-23T16:48:42.162198+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9411,"log_dups_size":0,"ondisk_log_size":9411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418185216,"num_objects":6381,"num_object_clones":0,"num_object_copies":12762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9411,"num_write_kb":505344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1632,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-04-23T16:48:38.550395+0000","last_change":"2024-04-23T16:45:33.395526+0000","last_active":"2024-04-23T16:48:38.550395+0000","last_peered":"2024-04-23T16:48:38.550395+0000","last_clean":"2024-04-23T16:48:38.550395+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:48:38.550395+0000","last_undegraded":"2024-04-23T16:48:38.550395+0000","last_fullsized":"2024-04-23T16:48:38.550395+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:33.395406+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:45:33.395406+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.1","version":"656'9433","reported_seq":9617,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-04-23T16:48:40.926342+0000","last_change":"2024-04-23T16:47:50.555018+0000","last_active":"2024-04-23T16:48:40.926342+0000","last_peered":"2024-04-23T16:48:40.926342+0000","last_clean":"2024-04-23T16:48:40.926342+0000","last_became_active":"2024-04-23T16:46:26.362670+0000","last_became_peered":"2024-04-23T16:46:26.362670+0000","last_unstale":"2024-04-23T16:48:40.926342+0000","last_undegraded":"2024-04-23T16:48:40.926342+0000","last_fullsized":"2024-04-23T16:48:40.926342+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'7115","last_scrub_stamp":"2024-04-23T16:47:50.554855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:47:50.554855+0000","objects_scrubbed":7087,"log_size":9433,"log_dups_size":0,"ondisk_log_size":9433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:19.584547+0000","scrub_duration":1213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414384128,"num_objects":6323,"num_object_clones":0,"num_object_copies":12646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9433,"num_write_kb":504192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1501,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-04-23T16:48:37.535370+0000","last_change":"2024-04-23T16:48:00.248930+0000","last_active":"2024-04-23T16:48:37.535370+0000","last_peered":"2024-04-23T16:48:37.535370+0000","last_clean":"2024-04-23T16:48:37.535370+0000","last_became_active":"2024-04-23T16:47:10.310322+0000","last_became_peered":"2024-04-23T16:47:10.310322+0000","last_unstale":"2024-04-23T16:48:37.535370+0000","last_undegraded":"2024-04-23T16:48:37.535370+0000","last_fullsized":"2024-04-23T16:48:37.535370+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:00.248799+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:00.248799+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:24.559886+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.0","version":"655'9033","reported_seq":15502,"reported_epoch":655,"state":"active+recovering","last_fresh":"2024-04-23T16:48:39.423492+0000","last_change":"2024-04-23T16:47:10.359661+0000","last_active":"2024-04-23T16:48:39.423492+0000","last_peered":"2024-04-23T16:48:39.423492+0000","last_clean":"2024-04-23T16:47:09.045399+0000","last_became_active":"2024-04-23T16:47:10.314835+0000","last_became_peered":"2024-04-23T16:47:10.314835+0000","last_unstale":"2024-04-23T16:48:39.423492+0000","last_undegraded":"2024-04-23T16:48:39.423492+0000","last_fullsized":"2024-04-23T16:48:39.423492+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9033,"log_dups_size":0,"ondisk_log_size":9033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412811264,"num_objects":6299,"num_object_clones":0,"num_object_copies":12598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1767,"num_objects_unfound":0,"num_objects_dirty":6299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9033,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3183,"num_bytes_recovered":208601088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"4,9","objects":4532},{"shards":"5,9","objects":1767}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1626,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-04-23T16:48:39.564523+0000","last_change":"2024-04-23T16:48:28.280886+0000","last_active":"2024-04-23T16:48:39.564523+0000","last_peered":"2024-04-23T16:48:39.564523+0000","last_clean":"2024-04-23T16:48:39.564523+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:48:39.564523+0000","last_undegraded":"2024-04-23T16:48:39.564523+0000","last_fullsized":"2024-04-23T16:48:39.564523+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:28.280751+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:28.280751+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:56.638217+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"654'9356","reported_seq":9537,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-04-23T16:48:38.397157+0000","last_change":"2024-04-23T16:47:54.493670+0000","last_active":"2024-04-23T16:48:38.397157+0000","last_peered":"2024-04-23T16:48:38.397157+0000","last_clean":"2024-04-23T16:48:38.397157+0000","last_became_active":"2024-04-23T16:46:26.366359+0000","last_became_peered":"2024-04-23T16:46:26.366359+0000","last_unstale":"2024-04-23T16:48:38.397157+0000","last_undegraded":"2024-04-23T16:48:38.397157+0000","last_fullsized":"2024-04-23T16:48:38.397157+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"643'7308","last_scrub_stamp":"2024-04-23T16:47:54.493495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:47:54.493495+0000","objects_scrubbed":7285,"log_size":9356,"log_dups_size":0,"ondisk_log_size":9356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:59.465079+0000","scrub_duration":1445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425590784,"num_objects":6494,"num_object_clones":0,"num_object_copies":12988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9356,"num_write_kb":507200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1640,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-04-23T16:48:39.543695+0000","last_change":"2024-04-23T16:47:08.281155+0000","last_active":"2024-04-23T16:48:39.543695+0000","last_peered":"2024-04-23T16:48:39.543695+0000","last_clean":"2024-04-23T16:48:39.543695+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:48:39.543695+0000","last_undegraded":"2024-04-23T16:48:39.543695+0000","last_fullsized":"2024-04-23T16:48:39.543695+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:47:08.281053+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:47:08.281053+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"654'9197","reported_seq":9373,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-04-23T16:48:38.402273+0000","last_change":"2024-04-23T16:47:46.611253+0000","last_active":"2024-04-23T16:48:38.402273+0000","last_peered":"2024-04-23T16:48:38.402273+0000","last_clean":"2024-04-23T16:48:38.402273+0000","last_became_active":"2024-04-23T16:46:26.366686+0000","last_became_peered":"2024-04-23T16:46:26.366686+0000","last_unstale":"2024-04-23T16:48:38.402273+0000","last_undegraded":"2024-04-23T16:48:38.402273+0000","last_fullsized":"2024-04-23T16:48:38.402273+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'6905","last_scrub_stamp":"2024-04-23T16:47:46.611007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:47:46.611007+0000","objects_scrubbed":6867,"log_size":9197,"log_dups_size":0,"ondisk_log_size":9197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:13.325560+0000","scrub_duration":1577,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421068800,"num_objects":6425,"num_object_clones":0,"num_object_copies":12850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9197,"num_write_kb":499904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1224,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-04-23T16:48:39.550365+0000","last_change":"2024-04-23T16:48:28.293172+0000","last_active":"2024-04-23T16:48:39.550365+0000","last_peered":"2024-04-23T16:48:39.550365+0000","last_clean":"2024-04-23T16:48:39.550365+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:48:39.550365+0000","last_undegraded":"2024-04-23T16:48:39.550365+0000","last_fullsized":"2024-04-23T16:48:39.550365+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:28.293038+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:48:28.293038+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:50.003967+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"655'9190","reported_seq":13619,"reported_epoch":655,"state":"active+recovering","last_fresh":"2024-04-23T16:48:39.423477+0000","last_change":"2024-04-23T16:47:10.314968+0000","last_active":"2024-04-23T16:48:39.423477+0000","last_peered":"2024-04-23T16:48:39.423477+0000","last_clean":"2024-04-23T16:47:09.038944+0000","last_became_active":"2024-04-23T16:47:10.251499+0000","last_became_peered":"2024-04-23T16:47:10.251499+0000","last_unstale":"2024-04-23T16:48:39.423477+0000","last_undegraded":"2024-04-23T16:48:39.423477+0000","last_fullsized":"2024-04-23T16:48:39.423477+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9190,"log_dups_size":0,"ondisk_log_size":9190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411959296,"num_objects":6286,"num_object_clones":0,"num_object_copies":12572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2991,"num_objects_unfound":0,"num_objects_dirty":6286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9190,"num_write_kb":495232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2162,"num_bytes_recovered":141688832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"5,9","objects":2991},{"shards":"9,11","objects":3295}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1175,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-04-23T16:48:37.535381+0000","last_change":"2024-04-23T16:48:36.291394+0000","last_active":"2024-04-23T16:48:37.535381+0000","last_peered":"2024-04-23T16:48:37.535381+0000","last_clean":"2024-04-23T16:48:37.535381+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:48:37.535381+0000","last_undegraded":"2024-04-23T16:48:37.535381+0000","last_fullsized":"2024-04-23T16:48:37.535381+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:36.291278+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:48:36.291278+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:58.136232+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.f","version":"656'9416","reported_seq":9519,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-04-23T16:48:42.185598+0000","last_change":"2024-04-23T16:46:26.364838+0000","last_active":"2024-04-23T16:48:42.185598+0000","last_peered":"2024-04-23T16:48:42.185598+0000","last_clean":"2024-04-23T16:48:42.185598+0000","last_became_active":"2024-04-23T16:46:26.364570+0000","last_became_peered":"2024-04-23T16:46:26.364570+0000","last_unstale":"2024-04-23T16:48:42.185598+0000","last_undegraded":"2024-04-23T16:48:42.185598+0000","last_fullsized":"2024-04-23T16:48:42.185598+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9416,"log_dups_size":0,"ondisk_log_size":9416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410255360,"num_objects":6260,"num_object_clones":0,"num_object_copies":12520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9416,"num_write_kb":501632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1640,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-04-23T16:48:38.550423+0000","last_change":"2024-04-23T16:45:30.258925+0000","last_active":"2024-04-23T16:48:38.550423+0000","last_peered":"2024-04-23T16:48:38.550423+0000","last_clean":"2024-04-23T16:48:38.550423+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:48:38.550423+0000","last_undegraded":"2024-04-23T16:48:38.550423+0000","last_fullsized":"2024-04-23T16:48:38.550423+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:30.258804+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:45:30.258804+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.e","version":"656'9219","reported_seq":9322,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-04-23T16:48:39.543728+0000","last_change":"2024-04-23T16:46:26.365834+0000","last_active":"2024-04-23T16:48:39.543728+0000","last_peered":"2024-04-23T16:48:39.543728+0000","last_clean":"2024-04-23T16:48:39.543728+0000","last_became_active":"2024-04-23T16:46:26.365596+0000","last_became_peered":"2024-04-23T16:46:26.365596+0000","last_unstale":"2024-04-23T16:48:39.543728+0000","last_undegraded":"2024-04-23T16:48:39.543728+0000","last_fullsized":"2024-04-23T16:48:39.543728+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9219,"log_dups_size":0,"ondisk_log_size":9219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408223744,"num_objects":6229,"num_object_clones":0,"num_object_copies":12458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9219,"num_write_kb":494336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1648,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-04-23T16:48:39.685210+0000","last_change":"2024-04-23T16:48:38.816478+0000","last_active":"2024-04-23T16:48:39.685210+0000","last_peered":"2024-04-23T16:48:39.685210+0000","last_clean":"2024-04-23T16:48:39.685210+0000","last_became_active":"2024-04-23T16:47:10.266014+0000","last_became_peered":"2024-04-23T16:47:10.266014+0000","last_unstale":"2024-04-23T16:48:39.685210+0000","last_undegraded":"2024-04-23T16:48:39.685210+0000","last_fullsized":"2024-04-23T16:48:39.685210+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:38.816369+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:38.816369+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:57.044042+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"656'9512","reported_seq":9615,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-04-23T16:48:42.158288+0000","last_change":"2024-04-23T16:46:26.364586+0000","last_active":"2024-04-23T16:48:42.158288+0000","last_peered":"2024-04-23T16:48:42.158288+0000","last_clean":"2024-04-23T16:48:42.158288+0000","last_became_active":"2024-04-23T16:46:26.364302+0000","last_became_peered":"2024-04-23T16:46:26.364302+0000","last_unstale":"2024-04-23T16:48:42.158288+0000","last_undegraded":"2024-04-23T16:48:42.158288+0000","last_fullsized":"2024-04-23T16:48:42.158288+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9512,"log_dups_size":0,"ondisk_log_size":9512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414384152,"num_objects":6324,"num_object_clones":0,"num_object_copies":12648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9512,"num_write_kb":506689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1526,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-04-23T16:48:39.554037+0000","last_change":"2024-04-23T16:47:10.037769+0000","last_active":"2024-04-23T16:48:39.554037+0000","last_peered":"2024-04-23T16:48:39.554037+0000","last_clean":"2024-04-23T16:48:39.554037+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:48:39.554037+0000","last_undegraded":"2024-04-23T16:48:39.554037+0000","last_fullsized":"2024-04-23T16:48:39.554037+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:47:10.037665+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:47:10.037665+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"655'9303","reported_seq":9404,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-04-23T16:48:39.421504+0000","last_change":"2024-04-23T16:46:26.364583+0000","last_active":"2024-04-23T16:48:39.421504+0000","last_peered":"2024-04-23T16:48:39.421504+0000","last_clean":"2024-04-23T16:48:39.421504+0000","last_became_active":"2024-04-23T16:46:26.364307+0000","last_became_peered":"2024-04-23T16:46:26.364307+0000","last_unstale":"2024-04-23T16:48:39.421504+0000","last_undegraded":"2024-04-23T16:48:39.421504+0000","last_fullsized":"2024-04-23T16:48:39.421504+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9303,"log_dups_size":0,"ondisk_log_size":9303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417660928,"num_objects":6373,"num_object_clones":0,"num_object_copies":12746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9303,"num_write_kb":501632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1457,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-04-23T16:48:39.685211+0000","last_change":"2024-04-23T16:48:33.921605+0000","last_active":"2024-04-23T16:48:39.685211+0000","last_peered":"2024-04-23T16:48:39.685211+0000","last_clean":"2024-04-23T16:48:39.685211+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:48:39.685211+0000","last_undegraded":"2024-04-23T16:48:39.685211+0000","last_fullsized":"2024-04-23T16:48:39.685211+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:33.921450+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:48:33.921450+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:48.394938+0000","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.9","version":"656'9643","reported_seq":15535,"reported_epoch":656,"state":"active+recovering","last_fresh":"2024-04-23T16:48:42.166237+0000","last_change":"2024-04-23T16:47:10.511786+0000","last_active":"2024-04-23T16:48:42.166237+0000","last_peered":"2024-04-23T16:48:42.166237+0000","last_clean":"2024-04-23T16:47:09.051081+0000","last_became_active":"2024-04-23T16:47:10.431200+0000","last_became_peered":"2024-04-23T16:47:10.431200+0000","last_unstale":"2024-04-23T16:48:42.166237+0000","last_undegraded":"2024-04-23T16:48:42.166237+0000","last_fullsized":"2024-04-23T16:48:42.166237+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9643,"log_dups_size":0,"ondisk_log_size":9643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418578432,"num_objects":6387,"num_object_clones":0,"num_object_copies":12774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2395,"num_objects_unfound":0,"num_objects_dirty":6387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9643,"num_write_kb":512960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2892,"num_bytes_recovered":189530112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":["11","5"],"object_location_counts":[{"shards":"0,11","objects":3992},{"shards":"5,11","objects":2395}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1626,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-04-23T16:48:39.564523+0000","last_change":"2024-04-23T16:48:37.224501+0000","last_active":"2024-04-23T16:48:39.564523+0000","last_peered":"2024-04-23T16:48:39.564523+0000","last_clean":"2024-04-23T16:48:39.564523+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:48:39.564523+0000","last_undegraded":"2024-04-23T16:48:39.564523+0000","last_fullsized":"2024-04-23T16:48:39.564523+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:37.224360+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:48:37.224360+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:50.365513+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":6655442968,"num_objects":101555,"num_object_clones":0,"num_object_copies":203110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10059,"num_objects_unfound":0,"num_objects_dirty":101555,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149459,"num_write_kb":8032385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10666,"num_bytes_recovered":699006976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14010294272,"data_stored":14010286128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149459,"ondisk_log_size":149459,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1679360,"data_stored":1661600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057041,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3699084,"kb_used_data":2877956,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":90046068,"statfs":{"total":95995035648,"available":92207173632,"internally_reserved":0,"allocated":2947026944,"data_stored":2946657027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":840825284},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351383,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613020,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3084608,"kb_used_data":2457844,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":90660544,"statfs":{"total":95995035648,"available":92836397056,"internally_reserved":0,"allocated":2516832256,"data_stored":2516472465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":641792448},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841952,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1651580,"kb_used_data":1241328,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92093572,"statfs":{"total":95995035648,"available":94303817728,"internally_reserved":0,"allocated":1271119872,"data_stored":1270762533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168998,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1455372,"kb_used_data":820292,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92289780,"statfs":{"total":95995035648,"available":94504734720,"internally_reserved":0,"allocated":839979008,"data_stored":839609091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":650312138},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463339,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545540,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199612,"statfs":{"total":95995035648,"available":95436402688,"internally_reserved":0,"allocated":708608,"data_stored":348817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137480,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2342412,"kb_used_data":2031296,"kb_used_omap":231,"kb_used_meta":310872,"kb_avail":91402740,"statfs":{"total":95995035648,"available":93596405760,"internally_reserved":0,"allocated":2080047104,"data_stored":2079681259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":318333840},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738825,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1744324,"kb_used_data":1067572,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":92000828,"statfs":{"total":95995035648,"available":94208847872,"internally_reserved":0,"allocated":1093193728,"data_stored":1092833937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269495,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2658584756246,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921203,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2753180,"kb_used_data":1913992,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":90991972,"statfs":{"total":95995035648,"available":93175779328,"internally_reserved":0,"allocated":1959927808,"data_stored":1959558411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":859306437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757679,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1087832064,"data_stored":1087832064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2070675456,"data_stored":2070675456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1957363712,"data_stored":1957363712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1358233600,"data_stored":1358233600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":834670592,"data_stored":834666520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1267007488,"data_stored":1267007488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2508455936,"data_stored":2508455936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2926055424,"data_stored":2926051352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:48:46.120 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:48:46.638 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:46.680 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:46.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:46 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:48:46.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:46 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:48:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:46 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:48:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:46 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:48:46.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:46 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:48:46.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:46 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:48:47.052 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:48:47.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:47.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:48:47.160 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:48:47.224 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:47.621 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:48:47.651 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:48:47.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:47.668 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:48:47.708 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-23T16:48:47.770 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:48:47.796 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:47.816 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:48:47.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:47.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:48:47.840 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:47 smithi103 ceph-mon[39629]: osdmap e659: 12 total, 12 up, 8 in 2024-04-23T16:48:47.840 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:47 smithi103 ceph-mon[39629]: pgmap v1835: 27 pgs: 4 active+recovering, 23 active+clean; 6.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 677 op/s; 9542/198136 objects misplaced (4.816%); 7.5 MiB/s, 120 objects/s recovering 2024-04-23T16:48:47.840 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:47 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2337225480' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:48:47.840 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:47 smithi103 ceph-mon[39629]: from='client.36671 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:48:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:47 smithi148 ceph-mon[44520]: osdmap e659: 12 total, 12 up, 8 in 2024-04-23T16:48:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:47 smithi148 ceph-mon[44520]: pgmap v1835: 27 pgs: 4 active+recovering, 23 active+clean; 6.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 677 op/s; 9542/198136 objects misplaced (4.816%); 7.5 MiB/s, 120 objects/s recovering 2024-04-23T16:48:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:47 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2337225480' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:48:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:47 smithi148 ceph-mon[44520]: from='client.36671 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:48:47.922 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:48:47.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:47 smithi188 ceph-mon[44101]: osdmap e659: 12 total, 12 up, 8 in 2024-04-23T16:48:47.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:47 smithi188 ceph-mon[44101]: pgmap v1835: 27 pgs: 4 active+recovering, 23 active+clean; 6.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 677 op/s; 9542/198136 objects misplaced (4.816%); 7.5 MiB/s, 120 objects/s recovering 2024-04-23T16:48:47.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:47 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2337225480' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:48:47.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:47 smithi188 ceph-mon[44101]: from='client.36671 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:48:48.026 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:48.062 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:48:48.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:48.065 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:48:48.167 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:48:48.194 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:48.214 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:48:48.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:48.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:48:48.231 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:48:48.318 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:48:48.347 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:48.368 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:48:48.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:48.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:48:48.473 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:48:48.506 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:48.582 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:48:48.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:48.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:48:48.614 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:48 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:48:48.614 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:48 smithi148 ceph-mon[44520]: from='client.36671 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:48:48.614 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:48 smithi148 ceph-mon[44520]: osdmap e660: 12 total, 12 up, 8 in 2024-04-23T16:48:48.688 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:48:48.715 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:48.734 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:48:48.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:48.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:48:48.839 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:48:48.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:48 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:48:48.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:48 smithi103 ceph-mon[39629]: from='client.36671 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:48:48.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:48 smithi103 ceph-mon[39629]: osdmap e660: 12 total, 12 up, 8 in 2024-04-23T16:48:48.867 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:48.888 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:48:48.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:48.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:48:48.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:48 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:48:48.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:48 smithi188 ceph-mon[44101]: from='client.36671 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:48:48.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:48 smithi188 ceph-mon[44101]: osdmap e660: 12 total, 12 up, 8 in 2024-04-23T16:48:48.994 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:48:49.023 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:49.079 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:48:49.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:49.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:48:49.183 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:48:49.212 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:49.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:49 smithi103 ceph-mon[39629]: pgmap v1837: 27 pgs: 4 active+recovering, 23 active+clean; 5.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 9312/194810 objects misplaced (4.780%); 7.8 MiB/s, 124 objects/s recovering 2024-04-23T16:48:49.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:49 smithi148 ceph-mon[44520]: pgmap v1837: 27 pgs: 4 active+recovering, 23 active+clean; 5.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 9312/194810 objects misplaced (4.780%); 7.8 MiB/s, 124 objects/s recovering 2024-04-23T16:48:49.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:49 smithi188 ceph-mon[44101]: pgmap v1837: 27 pgs: 4 active+recovering, 23 active+clean; 5.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 9312/194810 objects misplaced (4.780%); 7.8 MiB/s, 124 objects/s recovering 2024-04-23T16:48:50.468 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:48:50.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:50.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:48:50.574 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:48:50.605 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:50.734 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:48:50.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:50.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:48:50.839 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:48:50.866 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:50.884 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:48:50.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:50.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:48:50.988 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:48:51.078 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:51.236 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:48:51.258 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:48:51.279 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:48:51.300 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:48:51.322 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:48:51.343 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:48:51.368 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:48:51.391 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:48:51.431 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:48:51.474 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:48:51.534 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:48:51.560 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:48:51.562 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:48:51.562 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:48:51.562 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:48:51.562 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:48:51.562 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:48:51.562 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:48:51.562 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:48:51.563 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:48:51.563 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:48:51.563 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:48:51.563 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:48:51.563 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:48:51.632 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:48:51.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:51.639 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:48:51.674 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:48:51.677 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:48:51.740 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:48:51.768 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:51.792 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:48:51.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:51.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:48:51.895 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:48:51.931 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:51.953 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:48:51.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:51.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:48:51.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:51 smithi188 ceph-mon[44101]: pgmap v1838: 27 pgs: 4 active+recovering, 23 active+clean; 5.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 8783/191604 objects misplaced (4.584%); 7.5 MiB/s, 119 objects/s recovering 2024-04-23T16:48:51.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:51 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3453896534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:51.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:51.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:51.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:51.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:51.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:51.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:51 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:52.066 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:48:52.091 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:51 smithi148 ceph-mon[44520]: pgmap v1838: 27 pgs: 4 active+recovering, 23 active+clean; 5.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 8783/191604 objects misplaced (4.584%); 7.5 MiB/s, 119 objects/s recovering 2024-04-23T16:48:52.091 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:51 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3453896534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:52.091 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:52.091 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:52.091 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:52.091 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:52.092 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:52.092 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:51 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:52.093 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:52.112 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:48:52.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:52.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:48:52.221 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:48:52.249 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:52.269 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:48:52.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:52.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:48:52.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:51 smithi103 ceph-mon[39629]: pgmap v1838: 27 pgs: 4 active+recovering, 23 active+clean; 5.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 8783/191604 objects misplaced (4.584%); 7.5 MiB/s, 119 objects/s recovering 2024-04-23T16:48:52.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:51 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3453896534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:52.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:52.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:52.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:52.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:52.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:52.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:51 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:52.375 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:48:52.443 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:52.470 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:48:52.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:52.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:48:52.580 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:48:52.658 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:52.913 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:48:52.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:52.917 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:48:52.978 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1838,"stamp":"2024-04-23T16:48:50.916254+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6278873656,"num_objects":95802,"num_object_clones":0,"num_object_copies":191604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8783,"num_objects_unfound":0,"num_objects_dirty":95802,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155214,"num_write_kb":8032385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11944,"num_bytes_recovered":783033408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156743,"ondisk_log_size":156743,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16659468,"kb_used_data":11723508,"kb_used_omap":242,"kb_used_meta":4935501,"kb_avail":733301748,"statfs":{"total":767960285184,"available":750900989952,"internally_reserved":0,"allocated":12004872192,"data_stored":12001755768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":247821,"internal_metadata":5053954035},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,6,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-287506432,"num_objects":-4387,"num_object_clones":0,"num_object_copies":-8774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-759,"num_objects_unfound":0,"num_objects_dirty":-4387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4387,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":759,"num_bytes_recovered":49741824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.334545"},"pg_stats":[{"pgid":"8.7","version":"658'9536","reported_seq":9719,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-04-23T16:48:45.904528+0000","last_change":"2024-04-23T16:47:45.637166+0000","last_active":"2024-04-23T16:48:45.904528+0000","last_peered":"2024-04-23T16:48:45.904528+0000","last_clean":"2024-04-23T16:48:45.904528+0000","last_became_active":"2024-04-23T16:46:26.363422+0000","last_became_peered":"2024-04-23T16:46:26.363422+0000","last_unstale":"2024-04-23T16:48:45.904528+0000","last_undegraded":"2024-04-23T16:48:45.904528+0000","last_fullsized":"2024-04-23T16:48:45.904528+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"641'6798","last_scrub_stamp":"2024-04-23T16:47:45.637054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:47:45.637054+0000","objects_scrubbed":6770,"log_size":9536,"log_dups_size":0,"ondisk_log_size":9536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:01.884159+0000","scrub_duration":1366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394133504,"num_objects":6014,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9536,"num_write_kb":497600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"659'9782","reported_seq":9892,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.166212+0000","last_change":"2024-04-23T16:46:26.365698+0000","last_active":"2024-04-23T16:48:47.166212+0000","last_peered":"2024-04-23T16:48:47.166212+0000","last_clean":"2024-04-23T16:48:47.166212+0000","last_became_active":"2024-04-23T16:46:26.365309+0000","last_became_peered":"2024-04-23T16:46:26.365309+0000","last_unstale":"2024-04-23T16:48:47.166212+0000","last_undegraded":"2024-04-23T16:48:47.166212+0000","last_fullsized":"2024-04-23T16:48:47.166212+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9782,"log_dups_size":0,"ondisk_log_size":9782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396230656,"num_objects":6046,"num_object_clones":0,"num_object_copies":12092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9782,"num_write_kb":506496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.5","version":"660'9588","reported_seq":9700,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:49.611060+0000","last_change":"2024-04-23T16:46:26.365075+0000","last_active":"2024-04-23T16:48:49.611060+0000","last_peered":"2024-04-23T16:48:49.611060+0000","last_clean":"2024-04-23T16:48:49.611060+0000","last_became_active":"2024-04-23T16:46:26.364780+0000","last_became_peered":"2024-04-23T16:46:26.364780+0000","last_unstale":"2024-04-23T16:48:49.611060+0000","last_undegraded":"2024-04-23T16:48:49.611060+0000","last_fullsized":"2024-04-23T16:48:49.611060+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9588,"log_dups_size":0,"ondisk_log_size":9588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384565248,"num_objects":5868,"num_object_clones":0,"num_object_copies":11736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9588,"num_write_kb":494592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"660'10015","reported_seq":15580,"reported_epoch":660,"state":"active+recovering","last_fresh":"2024-04-23T16:48:49.430371+0000","last_change":"2024-04-23T16:47:10.280064+0000","last_active":"2024-04-23T16:48:49.430371+0000","last_peered":"2024-04-23T16:48:49.430371+0000","last_clean":"2024-04-23T16:47:09.049801+0000","last_became_active":"2024-04-23T16:47:10.244722+0000","last_became_peered":"2024-04-23T16:47:10.244722+0000","last_unstale":"2024-04-23T16:48:49.430371+0000","last_undegraded":"2024-04-23T16:48:49.430371+0000","last_fullsized":"2024-04-23T16:48:49.430371+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399179776,"num_objects":6091,"num_object_clones":0,"num_object_copies":12182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2610,"num_objects_unfound":0,"num_objects_dirty":6091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10015,"num_write_kb":515392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2725,"num_bytes_recovered":178585600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"5,9","objects":2610},{"shards":"9,11","objects":3481}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"660'9696","reported_seq":9808,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:49.574224+0000","last_change":"2024-04-23T16:46:26.365492+0000","last_active":"2024-04-23T16:48:49.574224+0000","last_peered":"2024-04-23T16:48:49.574224+0000","last_clean":"2024-04-23T16:48:49.574224+0000","last_became_active":"2024-04-23T16:46:26.365000+0000","last_became_peered":"2024-04-23T16:46:26.365000+0000","last_unstale":"2024-04-23T16:48:49.574224+0000","last_undegraded":"2024-04-23T16:48:49.574224+0000","last_fullsized":"2024-04-23T16:48:49.574224+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9696,"log_dups_size":0,"ondisk_log_size":9696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385613824,"num_objects":5884,"num_object_clones":0,"num_object_copies":11768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9696,"num_write_kb":498560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"659'9650","reported_seq":9759,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.166333+0000","last_change":"2024-04-23T16:46:26.365668+0000","last_active":"2024-04-23T16:48:47.166333+0000","last_peered":"2024-04-23T16:48:47.166333+0000","last_clean":"2024-04-23T16:48:47.166333+0000","last_became_active":"2024-04-23T16:46:26.364349+0000","last_became_peered":"2024-04-23T16:46:26.364349+0000","last_unstale":"2024-04-23T16:48:47.166333+0000","last_undegraded":"2024-04-23T16:48:47.166333+0000","last_fullsized":"2024-04-23T16:48:47.166333+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9650,"log_dups_size":0,"ondisk_log_size":9650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402522112,"num_objects":6142,"num_object_clones":0,"num_object_copies":12284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9650,"num_write_kb":505344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1642,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.623529+0000","last_change":"2024-04-23T16:45:33.395526+0000","last_active":"2024-04-23T16:48:47.623529+0000","last_peered":"2024-04-23T16:48:47.623529+0000","last_clean":"2024-04-23T16:48:47.623529+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:48:47.623529+0000","last_undegraded":"2024-04-23T16:48:47.623529+0000","last_fullsized":"2024-04-23T16:48:47.623529+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:33.395406+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:45:33.395406+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.1","version":"658'9658","reported_seq":9846,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-04-23T16:48:45.884310+0000","last_change":"2024-04-23T16:47:50.555018+0000","last_active":"2024-04-23T16:48:45.884310+0000","last_peered":"2024-04-23T16:48:45.884310+0000","last_clean":"2024-04-23T16:48:45.884310+0000","last_became_active":"2024-04-23T16:46:26.362670+0000","last_became_peered":"2024-04-23T16:46:26.362670+0000","last_unstale":"2024-04-23T16:48:45.884310+0000","last_undegraded":"2024-04-23T16:48:45.884310+0000","last_fullsized":"2024-04-23T16:48:45.884310+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'7115","last_scrub_stamp":"2024-04-23T16:47:50.554855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:47:50.554855+0000","objects_scrubbed":7087,"log_size":9658,"log_dups_size":0,"ondisk_log_size":9658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:19.584547+0000","scrub_duration":1213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399638528,"num_objects":6098,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9658,"num_write_kb":504192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1513,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.623777+0000","last_change":"2024-04-23T16:48:00.248930+0000","last_active":"2024-04-23T16:48:47.623777+0000","last_peered":"2024-04-23T16:48:47.623777+0000","last_clean":"2024-04-23T16:48:47.623777+0000","last_became_active":"2024-04-23T16:47:10.310322+0000","last_became_peered":"2024-04-23T16:47:10.310322+0000","last_unstale":"2024-04-23T16:48:47.623777+0000","last_undegraded":"2024-04-23T16:48:47.623777+0000","last_fullsized":"2024-04-23T16:48:47.623777+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:00.248799+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:00.248799+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:24.559886+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.0","version":"660'9487","reported_seq":16912,"reported_epoch":660,"state":"active+recovering","last_fresh":"2024-04-23T16:48:49.447425+0000","last_change":"2024-04-23T16:47:10.359661+0000","last_active":"2024-04-23T16:48:49.447425+0000","last_peered":"2024-04-23T16:48:49.447425+0000","last_clean":"2024-04-23T16:47:09.045399+0000","last_became_active":"2024-04-23T16:47:10.314835+0000","last_became_peered":"2024-04-23T16:47:10.314835+0000","last_unstale":"2024-04-23T16:48:49.447425+0000","last_undegraded":"2024-04-23T16:48:49.447425+0000","last_fullsized":"2024-04-23T16:48:49.447425+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9487,"log_dups_size":0,"ondisk_log_size":9487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383057920,"num_objects":5845,"num_object_clones":0,"num_object_copies":11690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1294,"num_objects_unfound":0,"num_objects_dirty":5845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9487,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3656,"num_bytes_recovered":239599616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"4,9","objects":4551},{"shards":"5,9","objects":1294}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1634,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.631710+0000","last_change":"2024-04-23T16:48:28.280886+0000","last_active":"2024-04-23T16:48:47.631710+0000","last_peered":"2024-04-23T16:48:47.631710+0000","last_clean":"2024-04-23T16:48:47.631710+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:48:47.631710+0000","last_undegraded":"2024-04-23T16:48:47.631710+0000","last_fullsized":"2024-04-23T16:48:47.631710+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:28.280751+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:28.280751+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:56.638217+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"660'9821","reported_seq":10013,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:48.450961+0000","last_change":"2024-04-23T16:47:54.493670+0000","last_active":"2024-04-23T16:48:48.450961+0000","last_peered":"2024-04-23T16:48:48.450961+0000","last_clean":"2024-04-23T16:48:48.450961+0000","last_became_active":"2024-04-23T16:46:26.366359+0000","last_became_peered":"2024-04-23T16:46:26.366359+0000","last_unstale":"2024-04-23T16:48:48.450961+0000","last_undegraded":"2024-04-23T16:48:48.450961+0000","last_fullsized":"2024-04-23T16:48:48.450961+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"643'7308","last_scrub_stamp":"2024-04-23T16:47:54.493495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:47:54.493495+0000","objects_scrubbed":7285,"log_size":9821,"log_dups_size":0,"ondisk_log_size":9821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:59.465079+0000","scrub_duration":1445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395116544,"num_objects":6029,"num_object_clones":0,"num_object_copies":12058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9821,"num_write_kb":507200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1648,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.621864+0000","last_change":"2024-04-23T16:47:08.281155+0000","last_active":"2024-04-23T16:48:47.621864+0000","last_peered":"2024-04-23T16:48:47.621864+0000","last_clean":"2024-04-23T16:48:47.621864+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:48:47.621864+0000","last_undegraded":"2024-04-23T16:48:47.621864+0000","last_fullsized":"2024-04-23T16:48:47.621864+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:47:08.281053+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:47:08.281053+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"660'9657","reported_seq":9845,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:48.432943+0000","last_change":"2024-04-23T16:47:46.611253+0000","last_active":"2024-04-23T16:48:48.432943+0000","last_peered":"2024-04-23T16:48:48.432943+0000","last_clean":"2024-04-23T16:48:48.432943+0000","last_became_active":"2024-04-23T16:46:26.366686+0000","last_became_peered":"2024-04-23T16:46:26.366686+0000","last_unstale":"2024-04-23T16:48:48.432943+0000","last_undegraded":"2024-04-23T16:48:48.432943+0000","last_fullsized":"2024-04-23T16:48:48.432943+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'6905","last_scrub_stamp":"2024-04-23T16:47:46.611007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:47:46.611007+0000","objects_scrubbed":6867,"log_size":9657,"log_dups_size":0,"ondisk_log_size":9657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:13.325560+0000","scrub_duration":1577,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390922240,"num_objects":5965,"num_object_clones":0,"num_object_copies":11930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9657,"num_write_kb":499904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1228,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-04-23T16:48:44.595336+0000","last_change":"2024-04-23T16:48:28.293172+0000","last_active":"2024-04-23T16:48:44.595336+0000","last_peered":"2024-04-23T16:48:44.595336+0000","last_clean":"2024-04-23T16:48:44.595336+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:48:44.595336+0000","last_undegraded":"2024-04-23T16:48:44.595336+0000","last_fullsized":"2024-04-23T16:48:44.595336+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:28.293038+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:48:28.293038+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:50.003967+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"660'9630","reported_seq":14623,"reported_epoch":660,"state":"active+recovering","last_fresh":"2024-04-23T16:48:49.425208+0000","last_change":"2024-04-23T16:47:10.314968+0000","last_active":"2024-04-23T16:48:49.425208+0000","last_peered":"2024-04-23T16:48:49.425208+0000","last_clean":"2024-04-23T16:47:09.038944+0000","last_became_active":"2024-04-23T16:47:10.251499+0000","last_became_peered":"2024-04-23T16:47:10.251499+0000","last_unstale":"2024-04-23T16:48:49.425208+0000","last_undegraded":"2024-04-23T16:48:49.425208+0000","last_fullsized":"2024-04-23T16:48:49.425208+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9630,"log_dups_size":0,"ondisk_log_size":9630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383123456,"num_objects":5846,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2714,"num_objects_unfound":0,"num_objects_dirty":5846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9630,"num_write_kb":495232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2439,"num_bytes_recovered":159842304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"5,9","objects":2714},{"shards":"9,11","objects":3132}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1187,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.624017+0000","last_change":"2024-04-23T16:48:36.291394+0000","last_active":"2024-04-23T16:48:47.624017+0000","last_peered":"2024-04-23T16:48:47.624017+0000","last_clean":"2024-04-23T16:48:47.624017+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:48:47.624017+0000","last_undegraded":"2024-04-23T16:48:47.624017+0000","last_fullsized":"2024-04-23T16:48:47.624017+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:36.291278+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:48:36.291278+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:58.136232+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.f","version":"659'9649","reported_seq":9759,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.140726+0000","last_change":"2024-04-23T16:46:26.364838+0000","last_active":"2024-04-23T16:48:47.140726+0000","last_peered":"2024-04-23T16:48:47.140726+0000","last_clean":"2024-04-23T16:48:47.140726+0000","last_became_active":"2024-04-23T16:46:26.364570+0000","last_became_peered":"2024-04-23T16:46:26.364570+0000","last_unstale":"2024-04-23T16:48:47.140726+0000","last_undegraded":"2024-04-23T16:48:47.140726+0000","last_fullsized":"2024-04-23T16:48:47.140726+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9649,"log_dups_size":0,"ondisk_log_size":9649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394985472,"num_objects":6027,"num_object_clones":0,"num_object_copies":12054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9649,"num_write_kb":501632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1650,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.623453+0000","last_change":"2024-04-23T16:45:30.258925+0000","last_active":"2024-04-23T16:48:47.623453+0000","last_peered":"2024-04-23T16:48:47.623453+0000","last_clean":"2024-04-23T16:48:47.623453+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:48:47.623453+0000","last_undegraded":"2024-04-23T16:48:47.623453+0000","last_fullsized":"2024-04-23T16:48:47.623453+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:30.258804+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:45:30.258804+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.e","version":"660'9650","reported_seq":9761,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:49.562719+0000","last_change":"2024-04-23T16:46:26.365834+0000","last_active":"2024-04-23T16:48:49.562719+0000","last_peered":"2024-04-23T16:48:49.562719+0000","last_clean":"2024-04-23T16:48:49.562719+0000","last_became_active":"2024-04-23T16:46:26.365596+0000","last_became_peered":"2024-04-23T16:46:26.365596+0000","last_unstale":"2024-04-23T16:48:49.562719+0000","last_undegraded":"2024-04-23T16:48:49.562719+0000","last_fullsized":"2024-04-23T16:48:49.562719+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9650,"log_dups_size":0,"ondisk_log_size":9650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379977728,"num_objects":5798,"num_object_clones":0,"num_object_copies":11596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9650,"num_write_kb":494336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1656,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:48.122246+0000","last_change":"2024-04-23T16:48:38.816478+0000","last_active":"2024-04-23T16:48:48.122246+0000","last_peered":"2024-04-23T16:48:48.122246+0000","last_clean":"2024-04-23T16:48:48.122246+0000","last_became_active":"2024-04-23T16:47:10.266014+0000","last_became_peered":"2024-04-23T16:47:10.266014+0000","last_unstale":"2024-04-23T16:48:48.122246+0000","last_undegraded":"2024-04-23T16:48:48.122246+0000","last_fullsized":"2024-04-23T16:48:48.122246+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:38.816369+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:38.816369+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:57.044042+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"659'9761","reported_seq":9872,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.200466+0000","last_change":"2024-04-23T16:46:26.364586+0000","last_active":"2024-04-23T16:48:47.200466+0000","last_peered":"2024-04-23T16:48:47.200466+0000","last_clean":"2024-04-23T16:48:47.200466+0000","last_became_active":"2024-04-23T16:46:26.364302+0000","last_became_peered":"2024-04-23T16:46:26.364302+0000","last_unstale":"2024-04-23T16:48:47.200466+0000","last_undegraded":"2024-04-23T16:48:47.200466+0000","last_fullsized":"2024-04-23T16:48:47.200466+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9761,"log_dups_size":0,"ondisk_log_size":9761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398065688,"num_objects":6075,"num_object_clones":0,"num_object_copies":12150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6075,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9761,"num_write_kb":506689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1532,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-04-23T16:48:46.616191+0000","last_change":"2024-04-23T16:47:10.037769+0000","last_active":"2024-04-23T16:48:46.616191+0000","last_peered":"2024-04-23T16:48:46.616191+0000","last_clean":"2024-04-23T16:48:46.616191+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:48:46.616191+0000","last_undegraded":"2024-04-23T16:48:46.616191+0000","last_fullsized":"2024-04-23T16:48:46.616191+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:47:10.037665+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:47:10.037665+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"660'9749","reported_seq":9861,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:49.408785+0000","last_change":"2024-04-23T16:46:26.364583+0000","last_active":"2024-04-23T16:48:49.408785+0000","last_peered":"2024-04-23T16:48:49.408785+0000","last_clean":"2024-04-23T16:48:49.408785+0000","last_became_active":"2024-04-23T16:46:26.364307+0000","last_became_peered":"2024-04-23T16:46:26.364307+0000","last_unstale":"2024-04-23T16:48:49.408785+0000","last_undegraded":"2024-04-23T16:48:49.408785+0000","last_fullsized":"2024-04-23T16:48:49.408785+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9749,"log_dups_size":0,"ondisk_log_size":9749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388431872,"num_objects":5927,"num_object_clones":0,"num_object_copies":11854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9749,"num_write_kb":501632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1465,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:48.122265+0000","last_change":"2024-04-23T16:48:33.921605+0000","last_active":"2024-04-23T16:48:48.122265+0000","last_peered":"2024-04-23T16:48:48.122265+0000","last_clean":"2024-04-23T16:48:48.122265+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:48:48.122265+0000","last_undegraded":"2024-04-23T16:48:48.122265+0000","last_fullsized":"2024-04-23T16:48:48.122265+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:33.921450+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:48:33.921450+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:48.394938+0000","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.9","version":"659'9885","reported_seq":16242,"reported_epoch":659,"state":"active+recovering","last_fresh":"2024-04-23T16:48:47.200436+0000","last_change":"2024-04-23T16:47:10.511786+0000","last_active":"2024-04-23T16:48:47.200436+0000","last_peered":"2024-04-23T16:48:47.200436+0000","last_clean":"2024-04-23T16:47:09.051081+0000","last_became_active":"2024-04-23T16:47:10.431200+0000","last_became_peered":"2024-04-23T16:47:10.431200+0000","last_unstale":"2024-04-23T16:48:47.200436+0000","last_undegraded":"2024-04-23T16:48:47.200436+0000","last_fullsized":"2024-04-23T16:48:47.200436+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9885,"log_dups_size":0,"ondisk_log_size":9885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402718720,"num_objects":6145,"num_object_clones":0,"num_object_copies":12290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2165,"num_objects_unfound":0,"num_objects_dirty":6145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9885,"num_write_kb":512960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3122,"num_bytes_recovered":204603392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":["11","5"],"object_location_counts":[{"shards":"0,11","objects":3980},{"shards":"5,11","objects":2165}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1634,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.631708+0000","last_change":"2024-04-23T16:48:37.224501+0000","last_active":"2024-04-23T16:48:47.631708+0000","last_peered":"2024-04-23T16:48:47.631708+0000","last_clean":"2024-04-23T16:48:47.631708+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:48:47.631708+0000","last_undegraded":"2024-04-23T16:48:47.631708+0000","last_fullsized":"2024-04-23T16:48:47.631708+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:37.224360+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:48:37.224360+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:50.365513+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":6278283288,"num_objects":95800,"num_object_clones":0,"num_object_copies":191600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8783,"num_objects_unfound":0,"num_objects_dirty":95800,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155214,"num_write_kb":8032385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11942,"num_bytes_recovered":782630912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13322166272,"data_stored":13322158128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155214,"ondisk_log_size":155214,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1679360,"data_stored":1661600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057042,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3607668,"kb_used_data":2760040,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":90137484,"statfs":{"total":95995035648,"available":92300783616,"internally_reserved":0,"allocated":2826280960,"data_stored":2825893031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":867826116},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351385,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613022,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2938940,"kb_used_data":2285808,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":90806212,"statfs":{"total":95995035648,"available":92985561088,"internally_reserved":0,"allocated":2340667392,"data_stored":2340277481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":668793280},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841954,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1569288,"kb_used_data":1154936,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92175864,"statfs":{"total":95995035648,"available":94388084736,"internally_reserved":0,"allocated":1182654464,"data_stored":1182261001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169001,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1381564,"kb_used_data":758964,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92363588,"statfs":{"total":95995035648,"available":94580314112,"internally_reserved":0,"allocated":777179136,"data_stored":776785153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":637532618},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463341,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545600,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199552,"statfs":{"total":95995035648,"available":95436341248,"internally_reserved":0,"allocated":770048,"data_stored":380137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137481,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2253280,"kb_used_data":1954648,"kb_used_omap":231,"kb_used_meta":298392,"kb_avail":91491872,"statfs":{"total":95995035648,"available":93687676928,"internally_reserved":0,"allocated":2001559552,"data_stored":2001181699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":305554320},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738827,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1693048,"kb_used_data":1007984,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":92052104,"statfs":{"total":95995035648,"available":94261354496,"internally_reserved":0,"allocated":1032175616,"data_stored":1031785705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269497,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2658584756247,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921205,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2670080,"kb_used_data":1800376,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":91075072,"statfs":{"total":95995035648,"available":93260873728,"internally_reserved":0,"allocated":1843585024,"data_stored":1843191561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":890567109},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757680,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1025245184,"data_stored":1025245184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1994129408,"data_stored":1994129408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1840119808,"data_stored":1840119808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1358233600,"data_stored":1358233600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":771690496,"data_stored":771686424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1177944064,"data_stored":1177944064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2330918912,"data_stored":2330918912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2823884800,"data_stored":2823880728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:48:52.979 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:48:53.019 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:48:53.120 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:53.161 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:48:53.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:53.170 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:48:53.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:53.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:53.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:53.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:53.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:53.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:52 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:53.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:52 smithi188 ceph-mon[44101]: from='client.36683 v1:172.21.15.103:0/1809417425' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:48:53.271 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:48:53.298 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:53.298 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:53.298 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:53.298 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:53.298 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:53.298 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:52 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:53.298 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:52 smithi103 ceph-mon[39629]: from='client.36683 v1:172.21.15.103:0/1809417425' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:48:53.300 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:52 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:52 smithi148 ceph-mon[44520]: from='client.36683 v1:172.21.15.103:0/1809417425' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:48:53.450 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:48:53.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:53.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:48:53.497 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:48:53.497 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:48:53.554 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:48:53.582 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:53.758 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:48:53.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:53.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:48:53.864 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:48:53.894 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:53.920 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:48:53.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:53.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:48:53.956 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:54.029 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:48:54.091 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:54.282 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:53 smithi103 ceph-mon[39629]: pgmap v1839: 27 pgs: 4 active+recovering, 23 active+clean; 5.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 660 op/s; 8553/188504 objects misplaced (4.537%); 7.7 MiB/s, 123 objects/s recovering 2024-04-23T16:48:54.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:53 smithi148 ceph-mon[44520]: pgmap v1839: 27 pgs: 4 active+recovering, 23 active+clean; 5.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 660 op/s; 8553/188504 objects misplaced (4.537%); 7.7 MiB/s, 123 objects/s recovering 2024-04-23T16:48:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:54 smithi188 ceph-mon[44101]: pgmap v1839: 27 pgs: 4 active+recovering, 23 active+clean; 5.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 660 op/s; 8553/188504 objects misplaced (4.537%); 7.7 MiB/s, 123 objects/s recovering 2024-04-23T16:48:54.872 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:48:54.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:54.883 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:48:54.984 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:48:55.011 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:55.031 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:48:55.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:55.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:48:55.136 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:48:55.167 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:55.417 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-23T16:48:55.518 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:48:55.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:55.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:48:55.623 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:48:55.650 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:55.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:55 smithi188 ceph-mon[44101]: pgmap v1840: 27 pgs: 4 active+recovering, 23 active+clean; 5.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 815 op/s; 7993/184572 objects misplaced (4.331%); 12 MiB/s, 186 objects/s recovering 2024-04-23T16:48:55.792 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:48:55.793 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:55 smithi103 ceph-mon[39629]: pgmap v1840: 27 pgs: 4 active+recovering, 23 active+clean; 5.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 815 op/s; 7993/184572 objects misplaced (4.331%); 12 MiB/s, 186 objects/s recovering 2024-04-23T16:48:55.799 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:48:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:55.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:48:55.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:55 smithi148 ceph-mon[44520]: pgmap v1840: 27 pgs: 4 active+recovering, 23 active+clean; 5.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 815 op/s; 7993/184572 objects misplaced (4.331%); 12 MiB/s, 186 objects/s recovering 2024-04-23T16:48:55.887 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:48:55.903 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:48:55.931 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:48:55.933 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:55.974 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:48:55.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:55.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:48:56.084 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:48:56.112 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:56.134 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:48:56.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:56.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:48:56.168 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:56.239 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:48:56.314 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:56.350 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:48:56.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:56.355 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:48:56.383 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:48:56.386 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:48:56.455 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:48:56.482 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:56.501 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:48:56.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:56.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:48:56.606 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:48:56.636 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:57.441 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:57 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:48:57.442 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:57 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:48:57.442 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:57 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:48:57.442 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:57 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:48:57.442 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:57 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:48:57.442 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:57 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:48:57.442 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:57 smithi103 ceph-mon[39629]: from='client.36689 v1:172.21.15.103:0/2135990564' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:48:57.443 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:57 smithi103 ceph-mon[39629]: osdmap e661: 12 total, 12 up, 8 in 2024-04-23T16:48:57.443 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:57 smithi103 ceph-mon[39629]: pgmap v1842: 27 pgs: 4 active+recovering, 23 active+clean; 5.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 648 op/s; 7993/183684 objects misplaced (4.351%); 8.9 MiB/s, 141 objects/s recovering 2024-04-23T16:48:57.589 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:48:57.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:57.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:48:57.693 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:48:57.720 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:57 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:48:57.720 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:57 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:48:57.720 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:57 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:48:57.720 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:57 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:48:57.720 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:57 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:48:57.721 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:57 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:48:57.721 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:57 smithi148 ceph-mon[44520]: from='client.36689 v1:172.21.15.103:0/2135990564' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:48:57.721 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:57 smithi148 ceph-mon[44520]: osdmap e661: 12 total, 12 up, 8 in 2024-04-23T16:48:57.721 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:57 smithi148 ceph-mon[44520]: pgmap v1842: 27 pgs: 4 active+recovering, 23 active+clean; 5.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 648 op/s; 7993/183684 objects misplaced (4.351%); 8.9 MiB/s, 141 objects/s recovering 2024-04-23T16:48:57.722 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:57.728 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:57 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:48:57.728 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:57 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:48:57.728 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:57 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:48:57.729 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:57 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:48:57.729 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:57 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:48:57.729 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:57 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:48:57.729 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:57 smithi188 ceph-mon[44101]: from='client.36689 v1:172.21.15.103:0/2135990564' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:48:57.729 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:57 smithi188 ceph-mon[44101]: osdmap e661: 12 total, 12 up, 8 in 2024-04-23T16:48:57.729 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:57 smithi188 ceph-mon[44101]: pgmap v1842: 27 pgs: 4 active+recovering, 23 active+clean; 5.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 648 op/s; 7993/183684 objects misplaced (4.351%); 8.9 MiB/s, 141 objects/s recovering 2024-04-23T16:48:57.741 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:48:57.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:57.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:48:57.846 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:48:57.873 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:57.898 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:48:57.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:57.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:48:58.002 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:48:58.072 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:58.122 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1840,"stamp":"2024-04-23T16:48:54.916928+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6048449080,"num_objects":92286,"num_object_clones":0,"num_object_copies":184572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7993,"num_objects_unfound":0,"num_objects_dirty":92286,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158730,"num_write_kb":8032385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12734,"num_bytes_recovered":834806848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159959,"ondisk_log_size":159959,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16294560,"kb_used_data":11348824,"kb_used_omap":242,"kb_used_meta":4945229,"kb_avail":733666656,"statfs":{"total":767960285184,"available":751274655744,"internally_reserved":0,"allocated":11621195776,"data_stored":11618061240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":247821,"internal_metadata":5063915507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-444465152,"num_objects":-6782,"num_object_clones":0,"num_object_copies":-13564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1549,"num_objects_unfound":0,"num_objects_dirty":-6782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6782,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1549,"num_bytes_recovered":101515264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.313857"},"pg_stats":[{"pgid":"8.7","version":"660'9731","reported_seq":9918,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:50.859219+0000","last_change":"2024-04-23T16:47:45.637166+0000","last_active":"2024-04-23T16:48:50.859219+0000","last_peered":"2024-04-23T16:48:50.859219+0000","last_clean":"2024-04-23T16:48:50.859219+0000","last_became_active":"2024-04-23T16:46:26.363422+0000","last_became_peered":"2024-04-23T16:46:26.363422+0000","last_unstale":"2024-04-23T16:48:50.859219+0000","last_undegraded":"2024-04-23T16:48:50.859219+0000","last_fullsized":"2024-04-23T16:48:50.859219+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"641'6798","last_scrub_stamp":"2024-04-23T16:47:45.637054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:47:45.637054+0000","objects_scrubbed":6770,"log_size":9731,"log_dups_size":0,"ondisk_log_size":9731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:01.884159+0000","scrub_duration":1366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381353984,"num_objects":5819,"num_object_clones":0,"num_object_copies":11638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9731,"num_write_kb":497600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"660'10020","reported_seq":10132,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:52.237629+0000","last_change":"2024-04-23T16:46:26.365698+0000","last_active":"2024-04-23T16:48:52.237629+0000","last_peered":"2024-04-23T16:48:52.237629+0000","last_clean":"2024-04-23T16:48:52.237629+0000","last_became_active":"2024-04-23T16:46:26.365309+0000","last_became_peered":"2024-04-23T16:46:26.365309+0000","last_unstale":"2024-04-23T16:48:52.237629+0000","last_undegraded":"2024-04-23T16:48:52.237629+0000","last_fullsized":"2024-04-23T16:48:52.237629+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380633088,"num_objects":5808,"num_object_clones":0,"num_object_copies":11616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10020,"num_write_kb":506496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.5","version":"660'9793","reported_seq":9905,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:54.650725+0000","last_change":"2024-04-23T16:46:26.365075+0000","last_active":"2024-04-23T16:48:54.650725+0000","last_peered":"2024-04-23T16:48:54.650725+0000","last_clean":"2024-04-23T16:48:54.650725+0000","last_became_active":"2024-04-23T16:46:26.364780+0000","last_became_peered":"2024-04-23T16:46:26.364780+0000","last_unstale":"2024-04-23T16:48:54.650725+0000","last_undegraded":"2024-04-23T16:48:54.650725+0000","last_fullsized":"2024-04-23T16:48:54.650725+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9793,"log_dups_size":0,"ondisk_log_size":9793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371130368,"num_objects":5663,"num_object_clones":0,"num_object_copies":11326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9793,"num_write_kb":494592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"660'10248","reported_seq":16085,"reported_epoch":660,"state":"active+recovering","last_fresh":"2024-04-23T16:48:54.423747+0000","last_change":"2024-04-23T16:47:10.280064+0000","last_active":"2024-04-23T16:48:54.423747+0000","last_peered":"2024-04-23T16:48:54.423747+0000","last_clean":"2024-04-23T16:47:09.049801+0000","last_became_active":"2024-04-23T16:47:10.244722+0000","last_became_peered":"2024-04-23T16:47:10.244722+0000","last_unstale":"2024-04-23T16:48:54.423747+0000","last_undegraded":"2024-04-23T16:48:54.423747+0000","last_fullsized":"2024-04-23T16:48:54.423747+0000","mapping_epoch":623,"log_start":"612'200","ondisk_log_start":"612'200","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383909888,"num_objects":5858,"num_object_clones":0,"num_object_copies":11716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2474,"num_objects_unfound":0,"num_objects_dirty":5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10248,"num_write_kb":515392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2861,"num_bytes_recovered":187498496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"5,9","objects":2474},{"shards":"9,11","objects":3384}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"660'9901","reported_seq":10013,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:54.593327+0000","last_change":"2024-04-23T16:46:26.365492+0000","last_active":"2024-04-23T16:48:54.593327+0000","last_peered":"2024-04-23T16:48:54.593327+0000","last_clean":"2024-04-23T16:48:54.593327+0000","last_became_active":"2024-04-23T16:46:26.365000+0000","last_became_peered":"2024-04-23T16:46:26.365000+0000","last_unstale":"2024-04-23T16:48:54.593327+0000","last_undegraded":"2024-04-23T16:48:54.593327+0000","last_fullsized":"2024-04-23T16:48:54.593327+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9901,"log_dups_size":0,"ondisk_log_size":9901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372178944,"num_objects":5679,"num_object_clones":0,"num_object_copies":11358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9901,"num_write_kb":498560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"660'9890","reported_seq":10002,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:52.209447+0000","last_change":"2024-04-23T16:46:26.365668+0000","last_active":"2024-04-23T16:48:52.209447+0000","last_peered":"2024-04-23T16:48:52.209447+0000","last_clean":"2024-04-23T16:48:52.209447+0000","last_became_active":"2024-04-23T16:46:26.364349+0000","last_became_peered":"2024-04-23T16:46:26.364349+0000","last_unstale":"2024-04-23T16:48:52.209447+0000","last_undegraded":"2024-04-23T16:48:52.209447+0000","last_fullsized":"2024-04-23T16:48:52.209447+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9890,"log_dups_size":0,"ondisk_log_size":9890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386793472,"num_objects":5902,"num_object_clones":0,"num_object_copies":11804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9890,"num_write_kb":505344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1642,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.623529+0000","last_change":"2024-04-23T16:45:33.395526+0000","last_active":"2024-04-23T16:48:47.623529+0000","last_peered":"2024-04-23T16:48:47.623529+0000","last_clean":"2024-04-23T16:48:47.623529+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:48:47.623529+0000","last_undegraded":"2024-04-23T16:48:47.623529+0000","last_fullsized":"2024-04-23T16:48:47.623529+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:33.395406+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:45:33.395406+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.1","version":"660'9860","reported_seq":10052,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:50.816671+0000","last_change":"2024-04-23T16:47:50.555018+0000","last_active":"2024-04-23T16:48:50.816671+0000","last_peered":"2024-04-23T16:48:50.816671+0000","last_clean":"2024-04-23T16:48:50.816671+0000","last_became_active":"2024-04-23T16:46:26.362670+0000","last_became_peered":"2024-04-23T16:46:26.362670+0000","last_unstale":"2024-04-23T16:48:50.816671+0000","last_undegraded":"2024-04-23T16:48:50.816671+0000","last_fullsized":"2024-04-23T16:48:50.816671+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'7115","last_scrub_stamp":"2024-04-23T16:47:50.554855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:47:50.554855+0000","objects_scrubbed":7087,"log_size":9860,"log_dups_size":0,"ondisk_log_size":9860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:19.584547+0000","scrub_duration":1213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386400256,"num_objects":5896,"num_object_clones":0,"num_object_copies":11792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9860,"num_write_kb":504192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1513,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.623777+0000","last_change":"2024-04-23T16:48:00.248930+0000","last_active":"2024-04-23T16:48:47.623777+0000","last_peered":"2024-04-23T16:48:47.623777+0000","last_clean":"2024-04-23T16:48:47.623777+0000","last_became_active":"2024-04-23T16:47:10.310322+0000","last_became_peered":"2024-04-23T16:47:10.310322+0000","last_unstale":"2024-04-23T16:48:47.623777+0000","last_undegraded":"2024-04-23T16:48:47.623777+0000","last_fullsized":"2024-04-23T16:48:47.623777+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:00.248799+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:00.248799+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:24.559886+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.0","version":"660'9704","reported_seq":17678,"reported_epoch":660,"state":"active+recovering","last_fresh":"2024-04-23T16:48:54.460943+0000","last_change":"2024-04-23T16:47:10.359661+0000","last_active":"2024-04-23T16:48:54.460943+0000","last_peered":"2024-04-23T16:48:54.460943+0000","last_clean":"2024-04-23T16:47:09.045399+0000","last_became_active":"2024-04-23T16:47:10.314835+0000","last_became_peered":"2024-04-23T16:47:10.314835+0000","last_unstale":"2024-04-23T16:48:54.460943+0000","last_undegraded":"2024-04-23T16:48:54.460943+0000","last_fullsized":"2024-04-23T16:48:54.460943+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9704,"log_dups_size":0,"ondisk_log_size":9704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368836608,"num_objects":5628,"num_object_clones":0,"num_object_copies":11256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1019,"num_objects_unfound":0,"num_objects_dirty":5628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9704,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3931,"num_bytes_recovered":257622016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"4,9","objects":4609},{"shards":"5,9","objects":1019}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1634,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.631710+0000","last_change":"2024-04-23T16:48:28.280886+0000","last_active":"2024-04-23T16:48:47.631710+0000","last_peered":"2024-04-23T16:48:47.631710+0000","last_clean":"2024-04-23T16:48:47.631710+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:48:47.631710+0000","last_undegraded":"2024-04-23T16:48:47.631710+0000","last_fullsized":"2024-04-23T16:48:47.631710+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:28.280751+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:28.280751+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:56.638217+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"660'10023","reported_seq":10216,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:53.394900+0000","last_change":"2024-04-23T16:47:54.493670+0000","last_active":"2024-04-23T16:48:53.394900+0000","last_peered":"2024-04-23T16:48:53.394900+0000","last_clean":"2024-04-23T16:48:53.394900+0000","last_became_active":"2024-04-23T16:46:26.366359+0000","last_became_peered":"2024-04-23T16:46:26.366359+0000","last_unstale":"2024-04-23T16:48:53.394900+0000","last_undegraded":"2024-04-23T16:48:53.394900+0000","last_fullsized":"2024-04-23T16:48:53.394900+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"643'7308","last_scrub_stamp":"2024-04-23T16:47:54.493495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:47:54.493495+0000","objects_scrubbed":7285,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:59.465079+0000","scrub_duration":1445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381878272,"num_objects":5827,"num_object_clones":0,"num_object_copies":11654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10023,"num_write_kb":507200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1648,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.621864+0000","last_change":"2024-04-23T16:47:08.281155+0000","last_active":"2024-04-23T16:48:47.621864+0000","last_peered":"2024-04-23T16:48:47.621864+0000","last_clean":"2024-04-23T16:48:47.621864+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:48:47.621864+0000","last_undegraded":"2024-04-23T16:48:47.621864+0000","last_fullsized":"2024-04-23T16:48:47.621864+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:47:08.281053+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:47:08.281053+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"660'9882","reported_seq":10070,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:53.421273+0000","last_change":"2024-04-23T16:47:46.611253+0000","last_active":"2024-04-23T16:48:53.421273+0000","last_peered":"2024-04-23T16:48:53.421273+0000","last_clean":"2024-04-23T16:48:53.421273+0000","last_became_active":"2024-04-23T16:46:26.366686+0000","last_became_peered":"2024-04-23T16:46:26.366686+0000","last_unstale":"2024-04-23T16:48:53.421273+0000","last_undegraded":"2024-04-23T16:48:53.421273+0000","last_fullsized":"2024-04-23T16:48:53.421273+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'6905","last_scrub_stamp":"2024-04-23T16:47:46.611007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:47:46.611007+0000","objects_scrubbed":6867,"log_size":9882,"log_dups_size":0,"ondisk_log_size":9882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:13.325560+0000","scrub_duration":1577,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376176640,"num_objects":5740,"num_object_clones":0,"num_object_copies":11480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9882,"num_write_kb":499904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1232,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.616337+0000","last_change":"2024-04-23T16:48:28.293172+0000","last_active":"2024-04-23T16:48:47.616337+0000","last_peered":"2024-04-23T16:48:47.616337+0000","last_clean":"2024-04-23T16:48:47.616337+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:48:47.616337+0000","last_undegraded":"2024-04-23T16:48:47.616337+0000","last_fullsized":"2024-04-23T16:48:47.616337+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:28.293038+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:48:28.293038+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:50.003967+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"660'9837","reported_seq":15128,"reported_epoch":660,"state":"active+recovering","last_fresh":"2024-04-23T16:48:54.414849+0000","last_change":"2024-04-23T16:47:10.314968+0000","last_active":"2024-04-23T16:48:54.414849+0000","last_peered":"2024-04-23T16:48:54.414849+0000","last_clean":"2024-04-23T16:47:09.038944+0000","last_became_active":"2024-04-23T16:47:10.251499+0000","last_became_peered":"2024-04-23T16:47:10.251499+0000","last_unstale":"2024-04-23T16:48:54.414849+0000","last_undegraded":"2024-04-23T16:48:54.414849+0000","last_fullsized":"2024-04-23T16:48:54.414849+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9837,"log_dups_size":0,"ondisk_log_size":9837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369557504,"num_objects":5639,"num_object_clones":0,"num_object_copies":11278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2565,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9837,"num_write_kb":495232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2588,"num_bytes_recovered":169607168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"5,9","objects":2565},{"shards":"9,11","objects":3074}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1187,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.624017+0000","last_change":"2024-04-23T16:48:36.291394+0000","last_active":"2024-04-23T16:48:47.624017+0000","last_peered":"2024-04-23T16:48:47.624017+0000","last_clean":"2024-04-23T16:48:47.624017+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:48:47.624017+0000","last_undegraded":"2024-04-23T16:48:47.624017+0000","last_fullsized":"2024-04-23T16:48:47.624017+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:36.291278+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:48:36.291278+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:58.136232+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.f","version":"660'9854","reported_seq":9966,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:52.192912+0000","last_change":"2024-04-23T16:46:26.364838+0000","last_active":"2024-04-23T16:48:52.192912+0000","last_peered":"2024-04-23T16:48:52.192912+0000","last_clean":"2024-04-23T16:48:52.192912+0000","last_became_active":"2024-04-23T16:46:26.364570+0000","last_became_peered":"2024-04-23T16:46:26.364570+0000","last_unstale":"2024-04-23T16:48:52.192912+0000","last_undegraded":"2024-04-23T16:48:52.192912+0000","last_fullsized":"2024-04-23T16:48:52.192912+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9854,"log_dups_size":0,"ondisk_log_size":9854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381550592,"num_objects":5822,"num_object_clones":0,"num_object_copies":11644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9854,"num_write_kb":501632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1650,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.623453+0000","last_change":"2024-04-23T16:45:30.258925+0000","last_active":"2024-04-23T16:48:47.623453+0000","last_peered":"2024-04-23T16:48:47.623453+0000","last_clean":"2024-04-23T16:48:47.623453+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:48:47.623453+0000","last_undegraded":"2024-04-23T16:48:47.623453+0000","last_fullsized":"2024-04-23T16:48:47.623453+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:30.258804+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:45:30.258804+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.e","version":"660'9880","reported_seq":9992,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:54.531098+0000","last_change":"2024-04-23T16:46:26.365834+0000","last_active":"2024-04-23T16:48:54.531098+0000","last_peered":"2024-04-23T16:48:54.531098+0000","last_clean":"2024-04-23T16:48:54.531098+0000","last_became_active":"2024-04-23T16:46:26.365596+0000","last_became_peered":"2024-04-23T16:46:26.365596+0000","last_unstale":"2024-04-23T16:48:54.531098+0000","last_undegraded":"2024-04-23T16:48:54.531098+0000","last_fullsized":"2024-04-23T16:48:54.531098+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9880,"log_dups_size":0,"ondisk_log_size":9880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364904448,"num_objects":5568,"num_object_clones":0,"num_object_copies":11136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9880,"num_write_kb":494336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1656,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:48.122246+0000","last_change":"2024-04-23T16:48:38.816478+0000","last_active":"2024-04-23T16:48:48.122246+0000","last_peered":"2024-04-23T16:48:48.122246+0000","last_clean":"2024-04-23T16:48:48.122246+0000","last_became_active":"2024-04-23T16:47:10.266014+0000","last_became_peered":"2024-04-23T16:47:10.266014+0000","last_unstale":"2024-04-23T16:48:48.122246+0000","last_undegraded":"2024-04-23T16:48:48.122246+0000","last_fullsized":"2024-04-23T16:48:48.122246+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:38.816369+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:38.816369+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:57.044042+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"660'9988","reported_seq":10101,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:52.185265+0000","last_change":"2024-04-23T16:46:26.364586+0000","last_active":"2024-04-23T16:48:52.185265+0000","last_peered":"2024-04-23T16:48:52.185265+0000","last_clean":"2024-04-23T16:48:52.185265+0000","last_became_active":"2024-04-23T16:46:26.364302+0000","last_became_peered":"2024-04-23T16:46:26.364302+0000","last_unstale":"2024-04-23T16:48:52.185265+0000","last_undegraded":"2024-04-23T16:48:52.185265+0000","last_fullsized":"2024-04-23T16:48:52.185265+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9988,"log_dups_size":0,"ondisk_log_size":9988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383189016,"num_objects":5848,"num_object_clones":0,"num_object_copies":11696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9988,"num_write_kb":506689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1534,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.628169+0000","last_change":"2024-04-23T16:47:10.037769+0000","last_active":"2024-04-23T16:48:47.628169+0000","last_peered":"2024-04-23T16:48:47.628169+0000","last_clean":"2024-04-23T16:48:47.628169+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:48:47.628169+0000","last_undegraded":"2024-04-23T16:48:47.628169+0000","last_fullsized":"2024-04-23T16:48:47.628169+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:47:10.037665+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:47:10.037665+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"660'9991","reported_seq":10103,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:54.464704+0000","last_change":"2024-04-23T16:46:26.364583+0000","last_active":"2024-04-23T16:48:54.464704+0000","last_peered":"2024-04-23T16:48:54.464704+0000","last_clean":"2024-04-23T16:48:54.464704+0000","last_became_active":"2024-04-23T16:46:26.364307+0000","last_became_peered":"2024-04-23T16:46:26.364307+0000","last_unstale":"2024-04-23T16:48:54.464704+0000","last_undegraded":"2024-04-23T16:48:54.464704+0000","last_fullsized":"2024-04-23T16:48:54.464704+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":9991,"log_dups_size":0,"ondisk_log_size":9991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372572160,"num_objects":5685,"num_object_clones":0,"num_object_copies":11370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9991,"num_write_kb":501632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1465,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:48.122265+0000","last_change":"2024-04-23T16:48:33.921605+0000","last_active":"2024-04-23T16:48:48.122265+0000","last_peered":"2024-04-23T16:48:48.122265+0000","last_clean":"2024-04-23T16:48:48.122265+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:48:48.122265+0000","last_undegraded":"2024-04-23T16:48:48.122265+0000","last_fullsized":"2024-04-23T16:48:48.122265+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:33.921450+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:48:33.921450+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:48.394938+0000","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.9","version":"660'10128","reported_seq":16948,"reported_epoch":660,"state":"active+recovering","last_fresh":"2024-04-23T16:48:52.240121+0000","last_change":"2024-04-23T16:47:10.511786+0000","last_active":"2024-04-23T16:48:52.240121+0000","last_peered":"2024-04-23T16:48:52.240121+0000","last_clean":"2024-04-23T16:47:09.051081+0000","last_became_active":"2024-04-23T16:47:10.431200+0000","last_became_peered":"2024-04-23T16:47:10.431200+0000","last_unstale":"2024-04-23T16:48:52.240121+0000","last_undegraded":"2024-04-23T16:48:52.240121+0000","last_fullsized":"2024-04-23T16:48:52.240121+0000","mapping_epoch":623,"log_start":"612'100","ondisk_log_start":"612'100","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386793472,"num_objects":5902,"num_object_clones":0,"num_object_copies":11804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1935,"num_objects_unfound":0,"num_objects_dirty":5902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10128,"num_write_kb":512960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3352,"num_bytes_recovered":219676672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":["11","5"],"object_location_counts":[{"shards":"0,11","objects":3967},{"shards":"5,11","objects":1935}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1634,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-04-23T16:48:47.631708+0000","last_change":"2024-04-23T16:48:37.224501+0000","last_active":"2024-04-23T16:48:47.631708+0000","last_peered":"2024-04-23T16:48:47.631708+0000","last_clean":"2024-04-23T16:48:47.631708+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:48:47.631708+0000","last_undegraded":"2024-04-23T16:48:47.631708+0000","last_fullsized":"2024-04-23T16:48:47.631708+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:37.224360+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:48:37.224360+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:50.365513+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":6047858712,"num_objects":92284,"num_object_clones":0,"num_object_copies":184568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7993,"num_objects_unfound":0,"num_objects_dirty":92284,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158730,"num_write_kb":8032385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12732,"num_bytes_recovered":834404352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12939436032,"data_stored":12939427888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158430,"ondisk_log_size":158430,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1679360,"data_stored":1661600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057043,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3534396,"kb_used_data":2664628,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":90210756,"statfs":{"total":95995035648,"available":92375814144,"internally_reserved":0,"allocated":2728579072,"data_stored":2728185089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":890567108},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351386,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613023,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2839928,"kb_used_data":2199280,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":90905224,"statfs":{"total":95995035648,"available":93086949376,"internally_reserved":0,"allocated":2252062720,"data_stored":2251672809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":656013760},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841955,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1529984,"kb_used_data":1111352,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92215168,"statfs":{"total":95995035648,"available":94428332032,"internally_reserved":0,"allocated":1138024448,"data_stored":1137630985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169001,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1381564,"kb_used_data":758964,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92363588,"statfs":{"total":95995035648,"available":94580314112,"internally_reserved":0,"allocated":777179136,"data_stored":776785153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":637532618},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463341,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545600,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199552,"statfs":{"total":95995035648,"available":95436341248,"internally_reserved":0,"allocated":770048,"data_stored":380137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137482,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2188024,"kb_used_data":1885232,"kb_used_omap":231,"kb_used_meta":302552,"kb_avail":91557128,"statfs":{"total":95995035648,"available":93754499072,"internally_reserved":0,"allocated":1930477568,"data_stored":1930087657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":309814160},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738828,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1667832,"kb_used_data":978608,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":92077320,"statfs":{"total":95995035648,"available":94287175680,"internally_reserved":0,"allocated":1002094592,"data_stored":1001704681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269498,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2658584756248,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921206,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2607232,"kb_used_data":1750008,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":91137920,"statfs":{"total":95995035648,"available":93325230080,"internally_reserved":0,"allocated":1792008192,"data_stored":1791614729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":877787589},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757681,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":996999168,"data_stored":996999168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1923547136,"data_stored":1923547136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1787101184,"data_stored":1787101184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1358233600,"data_stored":1358233600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":771690496,"data_stored":771686424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1133314048,"data_stored":1133314048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2243756032,"data_stored":2243756032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2724794368,"data_stored":2724790296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:48:58.124 INFO:tasks.cephadm.ceph_manager.ceph:PG 8.4 is not active+clean 2024-04-23T16:48:58.124 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '8.4', 'version': "660'10248", 'reported_seq': 16085, 'reported_epoch': 660, 'state': 'active+recovering', 'last_fresh': '2024-04-23T16:48:54.423747+0000', 'last_change': '2024-04-23T16:47:10.280064+0000', 'last_active': '2024-04-23T16:48:54.423747+0000', 'last_peered': '2024-04-23T16:48:54.423747+0000', 'last_clean': '2024-04-23T16:47:09.049801+0000', 'last_became_active': '2024-04-23T16:47:10.244722+0000', 'last_became_peered': '2024-04-23T16:47:10.244722+0000', 'last_unstale': '2024-04-23T16:48:54.423747+0000', 'last_undegraded': '2024-04-23T16:48:54.423747+0000', 'last_fullsized': '2024-04-23T16:48:54.423747+0000', 'mapping_epoch': 623, 'log_start': "612'200", 'ondisk_log_start': "612'200", 'created': 609, 'last_epoch_clean': 610, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T16:46:25.347995+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:46:25.347995+0000', 'last_clean_scrub_stamp': '2024-04-23T16:46:25.347995+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 383909888, 'num_objects': 5858, 'num_object_clones': 0, 'num_object_copies': 11716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2474, 'num_objects_unfound': 0, 'num_objects_dirty': 5858, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10248, 'num_write_kb': 515392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2861, 'num_bytes_recovered': 187498496, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9, 11], 'acting': [9, 11], 'avail_no_missing': ['9', '5'], 'object_location_counts': [{'shards': '5,9', 'objects': 2474}, {'shards': '9,11', 'objects': 3384}], 'blocked_by': [], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2024-04-23T16:48:58.124 INFO:tasks.cephadm.ceph_manager.ceph:PG 8.0 is not active+clean 2024-04-23T16:48:58.124 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '8.0', 'version': "660'9704", 'reported_seq': 17678, 'reported_epoch': 660, 'state': 'active+recovering', 'last_fresh': '2024-04-23T16:48:54.460943+0000', 'last_change': '2024-04-23T16:47:10.359661+0000', 'last_active': '2024-04-23T16:48:54.460943+0000', 'last_peered': '2024-04-23T16:48:54.460943+0000', 'last_clean': '2024-04-23T16:47:09.045399+0000', 'last_became_active': '2024-04-23T16:47:10.314835+0000', 'last_became_peered': '2024-04-23T16:47:10.314835+0000', 'last_unstale': '2024-04-23T16:48:54.460943+0000', 'last_undegraded': '2024-04-23T16:48:54.460943+0000', 'last_fullsized': '2024-04-23T16:48:54.460943+0000', 'mapping_epoch': 623, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 609, 'last_epoch_clean': 610, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T16:46:25.347995+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:46:25.347995+0000', 'last_clean_scrub_stamp': '2024-04-23T16:46:25.347995+0000', 'objects_scrubbed': 0, 'log_size': 9704, 'log_dups_size': 0, 'ondisk_log_size': 9704, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 368836608, 'num_objects': 5628, 'num_object_clones': 0, 'num_object_copies': 11256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1019, 'num_objects_unfound': 0, 'num_objects_dirty': 5628, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9704, 'num_write_kb': 490624, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3931, 'num_bytes_recovered': 257622016, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9, 4], 'acting': [9, 4], 'avail_no_missing': ['9', '5'], 'object_location_counts': [{'shards': '4,9', 'objects': 4609}, {'shards': '5,9', 'objects': 1019}], 'blocked_by': [], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2024-04-23T16:48:58.124 INFO:tasks.cephadm.ceph_manager.ceph:PG 8.8 is not active+clean 2024-04-23T16:48:58.125 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '8.8', 'version': "660'9837", 'reported_seq': 15128, 'reported_epoch': 660, 'state': 'active+recovering', 'last_fresh': '2024-04-23T16:48:54.414849+0000', 'last_change': '2024-04-23T16:47:10.314968+0000', 'last_active': '2024-04-23T16:48:54.414849+0000', 'last_peered': '2024-04-23T16:48:54.414849+0000', 'last_clean': '2024-04-23T16:47:09.038944+0000', 'last_became_active': '2024-04-23T16:47:10.251499+0000', 'last_became_peered': '2024-04-23T16:47:10.251499+0000', 'last_unstale': '2024-04-23T16:48:54.414849+0000', 'last_undegraded': '2024-04-23T16:48:54.414849+0000', 'last_fullsized': '2024-04-23T16:48:54.414849+0000', 'mapping_epoch': 623, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 609, 'last_epoch_clean': 610, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T16:46:25.347995+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:46:25.347995+0000', 'last_clean_scrub_stamp': '2024-04-23T16:46:25.347995+0000', 'objects_scrubbed': 0, 'log_size': 9837, 'log_dups_size': 0, 'ondisk_log_size': 9837, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 369557504, 'num_objects': 5639, 'num_object_clones': 0, 'num_object_copies': 11278, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2565, 'num_objects_unfound': 0, 'num_objects_dirty': 5639, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9837, 'num_write_kb': 495232, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2588, 'num_bytes_recovered': 169607168, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9, 11], 'acting': [9, 11], 'avail_no_missing': ['9', '5'], 'object_location_counts': [{'shards': '5,9', 'objects': 2565}, {'shards': '9,11', 'objects': 3074}], 'blocked_by': [], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2024-04-23T16:48:58.125 INFO:tasks.cephadm.ceph_manager.ceph:PG 8.9 is not active+clean 2024-04-23T16:48:58.125 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '8.9', 'version': "660'10128", 'reported_seq': 16948, 'reported_epoch': 660, 'state': 'active+recovering', 'last_fresh': '2024-04-23T16:48:52.240121+0000', 'last_change': '2024-04-23T16:47:10.511786+0000', 'last_active': '2024-04-23T16:48:52.240121+0000', 'last_peered': '2024-04-23T16:48:52.240121+0000', 'last_clean': '2024-04-23T16:47:09.051081+0000', 'last_became_active': '2024-04-23T16:47:10.431200+0000', 'last_became_peered': '2024-04-23T16:47:10.431200+0000', 'last_unstale': '2024-04-23T16:48:52.240121+0000', 'last_undegraded': '2024-04-23T16:48:52.240121+0000', 'last_fullsized': '2024-04-23T16:48:52.240121+0000', 'mapping_epoch': 623, 'log_start': "612'100", 'ondisk_log_start': "612'100", 'created': 609, 'last_epoch_clean': 610, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T16:46:25.347995+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:46:25.347995+0000', 'last_clean_scrub_stamp': '2024-04-23T16:46:25.347995+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 386793472, 'num_objects': 5902, 'num_object_clones': 0, 'num_object_copies': 11804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1935, 'num_objects_unfound': 0, 'num_objects_dirty': 5902, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10128, 'num_write_kb': 512960, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3352, 'num_bytes_recovered': 219676672, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [11, 0], 'acting': [11, 0], 'avail_no_missing': ['11', '5'], 'object_location_counts': [{'shards': '0,11', 'objects': 3967}, {'shards': '5,11', 'objects': 1935}], 'blocked_by': [], 'up_primary': 11, 'acting_primary': 11, 'purged_snaps': []} 2024-04-23T16:48:58.125 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-23T16:48:58.175 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:48:58.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:58.180 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:48:58.281 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:48:58.308 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:58.328 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:48:58.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:58.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:48:58.429 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:48:58.429 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:48:58.434 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:48:58.461 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:58.487 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:48:58.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:58.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:48:58.592 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:48:58.620 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:58.641 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:48:58.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:58.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:48:58.666 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:58.670 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:48:58.691 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:48:58.716 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:58 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/843415565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:58.719 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:58 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/843415565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:58.722 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:48:58.745 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:48:58.747 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:48:58.770 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:48:58.793 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:48:58.800 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:58.815 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:48:58.842 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:48:58.866 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:48:58.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:58 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/843415565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:48:58.895 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:48:58.916 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:48:58.943 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:48:58.982 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:48:58.989 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:48:58.990 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:48:58.990 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:48:58.990 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:48:58.990 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:48:58.990 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:48:58.990 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:48:58.990 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:48:58.990 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:48:58.990 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:48:58.990 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:48:58.990 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:48:59.288 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:48:59.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:59.294 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:48:59.319 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-23T16:48:59.398 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:48:59.458 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:59.508 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:48:59.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:59.517 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:48:59.617 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:48:59.691 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:59.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:59.726 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:48:59.726 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:48:59.728 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:59.728 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:59.827 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:48:59.854 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:48:59.867 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:48:59.873 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:48:59.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:48:59.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:48:59.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:59.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:59.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:59.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:59.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:59.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:59.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:59.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:59.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:59.979 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:59.979 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:59 smithi188 ceph-mon[44101]: pgmap v1843: 27 pgs: 4 active+recovering, 23 active+clean; 5.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 7744/180486 objects misplaced (4.291%); 9.8 MiB/s, 156 objects/s recovering 2024-04-23T16:48:59.979 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:48:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:48:59.979 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:49:00.010 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.010 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.010 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.010 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.010 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.010 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.010 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.010 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.011 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:59 smithi103 ceph-mon[39629]: pgmap v1843: 27 pgs: 4 active+recovering, 23 active+clean; 5.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 7744/180486 objects misplaced (4.291%); 9.8 MiB/s, 156 objects/s recovering 2024-04-23T16:49:00.011 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:48:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.014 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:00.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:59 smithi148 ceph-mon[44520]: pgmap v1843: 27 pgs: 4 active+recovering, 23 active+clean; 5.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 7744/180486 objects misplaced (4.291%); 9.8 MiB/s, 156 objects/s recovering 2024-04-23T16:49:00.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:48:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.468 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:49:00.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:00.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:49:00.572 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:49:00.599 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:00.621 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:49:00.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:00.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:49:00.724 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:49:00.754 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:00.807 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:49:00.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:00.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:49:00.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:00.911 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:49:00.939 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:00.961 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:49:01.022 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:01.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:01.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:49:01.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:01.123 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:49:01.151 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:01.172 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:49:01.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:01.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:49:01.276 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:49:01.343 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:01.386 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:49:01.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:01.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:49:01.488 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:49:01.517 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:01.866 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:49:01.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:01.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:49:01.971 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:49:02.000 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:02.023 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:49:02.025 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:01 smithi148 ceph-mon[44520]: pgmap v1844: 27 pgs: 4 active+recovering, 23 active+clean; 5.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 714 op/s; 7174/177308 objects misplaced (4.046%); 10 MiB/s, 160 objects/s recovering 2024-04-23T16:49:02.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:02.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:49:02.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:01 smithi103 ceph-mon[39629]: pgmap v1844: 27 pgs: 4 active+recovering, 23 active+clean; 5.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 714 op/s; 7174/177308 objects misplaced (4.046%); 10 MiB/s, 160 objects/s recovering 2024-04-23T16:49:02.128 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:49:02.165 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:02.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:01 smithi188 ceph-mon[44101]: pgmap v1844: 27 pgs: 4 active+recovering, 23 active+clean; 5.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 714 op/s; 7174/177308 objects misplaced (4.046%); 10 MiB/s, 160 objects/s recovering 2024-04-23T16:49:02.316 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:49:02.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:02.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:49:02.419 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:49:02.419 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:49:02.419 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:49:02.419 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:49:02.419 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:49:02.419 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:49:02.486 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:49:02.498 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:02.547 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:49:02.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:02.551 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:49:02.652 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:49:02.679 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:02.703 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:49:02.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:02.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:49:02.807 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:49:02.835 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:02 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:49:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:02 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:49:02.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:02 smithi103 ceph-mon[39629]: osdmap e662: 12 total, 12 up, 8 in 2024-04-23T16:49:02.871 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:03.046 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:49:03.047 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2325,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":662,"num_osds":12,"num_up_osds":12,"osd_up_since":1713890820,"num_in_osds":8,"osd_in_since":1713890829,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":23},{"state_name":"active+recovering","count":4}],"num_pgs":27,"num_pools":2,"num_objects":88654,"data_bytes":5810422328,"bytes_used":16263344128,"bytes_avail":751696941056,"bytes_total":767960285184,"misplaced_objects":7174,"misplaced_total":177308,"misplaced_ratio":0.040460667313375588,"recovering_objects_per_sec":160,"recovering_bytes_per_sec":10542965,"recovering_keys_per_sec":0,"num_objects_recovered":1609,"num_bytes_recovered":105447424,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":714},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.103:8443/"}},"servicemap":{"epoch":49,"modified":"2024-04-23T16:48:02.907113+0000","services":{}},"progress_events":{}} 2024-04-23T16:49:03.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:02 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:49:03.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:02 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:49:03.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:02 smithi148 ceph-mon[44520]: osdmap e662: 12 total, 12 up, 8 in 2024-04-23T16:49:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:02 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:49:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:02 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:49:03.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:02 smithi188 ceph-mon[44101]: osdmap e662: 12 total, 12 up, 8 in 2024-04-23T16:49:03.756 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:03 smithi103 ceph-mon[39629]: pgmap v1846: 27 pgs: 4 active+recovering, 23 active+clean; 5.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 659 op/s; 6905/174026 objects misplaced (3.968%); 8.5 MiB/s, 135 objects/s recovering 2024-04-23T16:49:03.756 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:03 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3655008535' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:49:03.756 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:03 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3247636625' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:49:03.793 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:49:04.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:03 smithi148 ceph-mon[44520]: pgmap v1846: 27 pgs: 4 active+recovering, 23 active+clean; 5.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 659 op/s; 6905/174026 objects misplaced (3.968%); 8.5 MiB/s, 135 objects/s recovering 2024-04-23T16:49:04.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:03 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3655008535' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:49:04.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:03 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3247636625' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:49:04.231 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:03 smithi188 ceph-mon[44101]: pgmap v1846: 27 pgs: 4 active+recovering, 23 active+clean; 5.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 659 op/s; 6905/174026 objects misplaced (3.968%); 8.5 MiB/s, 135 objects/s recovering 2024-04-23T16:49:04.231 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:03 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3655008535' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:49:04.231 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:03 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3247636625' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:49:05.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:04 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:49:05.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:04 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3247636625' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:49:05.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:04 smithi103 ceph-mon[39629]: osdmap e663: 12 total, 12 up, 8 in 2024-04-23T16:49:05.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:04 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:49:05.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:04 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3247636625' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:49:05.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:04 smithi148 ceph-mon[44520]: osdmap e663: 12 total, 12 up, 8 in 2024-04-23T16:49:05.211 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:49:05.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:05.217 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:49:05.219 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:04 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:49:05.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:04 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3247636625' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:49:05.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:04 smithi188 ceph-mon[44101]: osdmap e663: 12 total, 12 up, 8 in 2024-04-23T16:49:05.317 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:49:05.351 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:06.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:05 smithi103 ceph-mon[39629]: pgmap v1848: 27 pgs: 4 active+recovering, 23 active+clean; 5.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 853 op/s; 6299/170024 objects misplaced (3.705%); 13 MiB/s, 211 objects/s recovering 2024-04-23T16:49:06.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:05 smithi148 ceph-mon[44520]: pgmap v1848: 27 pgs: 4 active+recovering, 23 active+clean; 5.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 853 op/s; 6299/170024 objects misplaced (3.705%); 13 MiB/s, 211 objects/s recovering 2024-04-23T16:49:06.221 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-23T16:49:06.221 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:49:06.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:05 smithi188 ceph-mon[44101]: pgmap v1848: 27 pgs: 4 active+recovering, 23 active+clean; 5.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 853 op/s; 6299/170024 objects misplaced (3.705%); 13 MiB/s, 211 objects/s recovering 2024-04-23T16:49:06.306 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:49:06.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:06.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:49:06.413 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:49:06.442 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:06.643 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:49:06.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:06.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:49:06.724 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:06.747 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:49:06.805 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:07.499 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:49:07.499 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:49:07.499 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:49:07.508 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:49:07.539 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:49:07.568 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:49:07.587 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:49:07.607 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:49:07.657 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:49:07.699 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:49:07.725 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:49:07.742 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:49:07.762 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:49:07.764 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:49:07.764 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:49:07.764 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:49:07.764 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:49:07.764 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:49:07.764 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:49:07.765 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:49:07.765 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:49:07.765 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:49:07.765 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:49:07.765 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:49:07.765 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:49:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:07 smithi103 ceph-mon[39629]: pgmap v1849: 27 pgs: 4 active+recovering, 23 active+clean; 5.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 712 op/s; 6299/169088 objects misplaced (3.725%); 11 MiB/s, 180 objects/s recovering 2024-04-23T16:49:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:07 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2309981600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:49:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:07 smithi103 ceph-mon[39629]: 8.7 scrub starts 2024-04-23T16:49:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:08.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:07 smithi148 ceph-mon[44520]: pgmap v1849: 27 pgs: 4 active+recovering, 23 active+clean; 5.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 712 op/s; 6299/169088 objects misplaced (3.725%); 11 MiB/s, 180 objects/s recovering 2024-04-23T16:49:08.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:07 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2309981600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:49:08.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:08.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:07 smithi148 ceph-mon[44520]: 8.7 scrub starts 2024-04-23T16:49:08.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:08.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:08.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:08.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:08.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:07 smithi188 ceph-mon[44101]: pgmap v1849: 27 pgs: 4 active+recovering, 23 active+clean; 5.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 712 op/s; 6299/169088 objects misplaced (3.725%); 11 MiB/s, 180 objects/s recovering 2024-04-23T16:49:08.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:07 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2309981600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:49:08.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:08.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:07 smithi188 ceph-mon[44101]: 8.7 scrub starts 2024-04-23T16:49:08.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:08.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:08.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:08.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:08.605 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:49:08.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:08.613 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:49:08.635 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-23T16:49:08.714 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:49:08.741 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:08.763 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:49:08.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:08.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:49:08.867 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:49:08.896 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:08.917 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:49:08.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:08.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:49:09.021 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:49:09.084 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:09.105 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:09.152 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:49:09.153 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:49:09.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:09.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:49:09.263 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:49:09.324 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:08 smithi188 ceph-mon[44101]: 8.b scrub starts 2024-04-23T16:49:09.324 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:08 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.324 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:08 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.324 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:08 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.325 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:08 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.325 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:08 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.325 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:08 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.325 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:08 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.325 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:08 smithi188 ceph-mon[44101]: 8.7 scrub ok 2024-04-23T16:49:09.332 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:09.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:08 smithi103 ceph-mon[39629]: 8.b scrub starts 2024-04-23T16:49:09.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:08 smithi103 ceph-mon[39629]: 8.7 scrub ok 2024-04-23T16:49:09.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:08 smithi148 ceph-mon[44520]: 8.b scrub starts 2024-04-23T16:49:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:08 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:08 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:08 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:08 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:08 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:08 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:08 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:08 smithi148 ceph-mon[44520]: 8.7 scrub ok 2024-04-23T16:49:09.922 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:49:09.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:09.930 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:49:10.031 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:49:10.062 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:10.116 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:10 smithi103 ceph-mon[39629]: 8.b scrub ok 2024-04-23T16:49:10.117 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:10 smithi103 ceph-mon[39629]: pgmap v1850: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering, 22 active+clean; 5.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 6011/166038 objects misplaced (3.620%); 9.1 MiB/s, 145 objects/s recovering 2024-04-23T16:49:10.119 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:49:10.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:10.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:49:10.195 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:10 smithi188 ceph-mon[44101]: 8.b scrub ok 2024-04-23T16:49:10.195 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:10 smithi188 ceph-mon[44101]: pgmap v1850: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering, 22 active+clean; 5.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 6011/166038 objects misplaced (3.620%); 9.1 MiB/s, 145 objects/s recovering 2024-04-23T16:49:10.224 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:49:10.252 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:10.274 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:49:10.278 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:10 smithi148 ceph-mon[44520]: 8.b scrub ok 2024-04-23T16:49:10.278 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:10 smithi148 ceph-mon[44520]: pgmap v1850: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering, 22 active+clean; 5.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 6011/166038 objects misplaced (3.620%); 9.1 MiB/s, 145 objects/s recovering 2024-04-23T16:49:10.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:10.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:49:10.380 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:49:10.408 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:10.429 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:49:10.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:10.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:49:10.533 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:49:10.562 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:10.587 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:49:10.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:10.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:49:10.692 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:49:10.736 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:11.027 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:49:11.028 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:49:11.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:11 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:49:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:11 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:49:11.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:11 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:49:11.799 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1850,"stamp":"2024-04-23T16:49:08.919364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5441126968,"num_objects":83019,"num_object_clones":0,"num_object_copies":166038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6011,"num_objects_unfound":0,"num_objects_dirty":83019,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167997,"num_write_kb":8032385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14716,"num_bytes_recovered":964699200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162026,"ondisk_log_size":162026,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15266572,"kb_used_data":10298680,"kb_used_omap":242,"kb_used_meta":4967437,"kb_avail":734694644,"statfs":{"total":767960285184,"available":752327315456,"internally_reserved":0,"allocated":10545848320,"data_stored":10542570088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":247821,"internal_metadata":5086656499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,5,3,3,4,6,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-369295360,"num_objects":-5635,"num_object_clones":0,"num_object_copies":-11270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1163,"num_objects_unfound":0,"num_objects_dirty":-5635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5635,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1163,"num_bytes_recovered":76218368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001424"},"pg_stats":[{"pgid":"8.7","version":"663'10416","reported_seq":10609,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:05.769490+0000","last_change":"2024-04-23T16:47:45.637166+0000","last_active":"2024-04-23T16:49:05.769490+0000","last_peered":"2024-04-23T16:49:05.769490+0000","last_clean":"2024-04-23T16:49:05.769490+0000","last_became_active":"2024-04-23T16:46:26.363422+0000","last_became_peered":"2024-04-23T16:46:26.363422+0000","last_unstale":"2024-04-23T16:49:05.769490+0000","last_undegraded":"2024-04-23T16:49:05.769490+0000","last_fullsized":"2024-04-23T16:49:05.769490+0000","mapping_epoch":609,"log_start":"613'400","ondisk_log_start":"613'400","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"641'6798","last_scrub_stamp":"2024-04-23T16:47:45.637054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:47:45.637054+0000","objects_scrubbed":6770,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:07.334271+0000","scrub_duration":1366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336461824,"num_objects":5134,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10416,"num_write_kb":497600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"663'10715","reported_seq":10833,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:07.230963+0000","last_change":"2024-04-23T16:46:26.365698+0000","last_active":"2024-04-23T16:49:07.230963+0000","last_peered":"2024-04-23T16:49:07.230963+0000","last_clean":"2024-04-23T16:49:07.230963+0000","last_became_active":"2024-04-23T16:46:26.365309+0000","last_became_peered":"2024-04-23T16:46:26.365309+0000","last_unstale":"2024-04-23T16:49:07.230963+0000","last_undegraded":"2024-04-23T16:49:07.230963+0000","last_fullsized":"2024-04-23T16:49:07.230963+0000","mapping_epoch":609,"log_start":"613'700","ondisk_log_start":"613'700","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335085568,"num_objects":5113,"num_object_clones":0,"num_object_copies":10226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10715,"num_write_kb":506496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.5","version":"663'10237","reported_seq":10355,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:04.648111+0000","last_change":"2024-04-23T16:46:26.365075+0000","last_active":"2024-04-23T16:49:04.648111+0000","last_peered":"2024-04-23T16:49:04.648111+0000","last_clean":"2024-04-23T16:49:04.648111+0000","last_became_active":"2024-04-23T16:46:26.364780+0000","last_became_peered":"2024-04-23T16:46:26.364780+0000","last_unstale":"2024-04-23T16:49:04.648111+0000","last_undegraded":"2024-04-23T16:49:04.648111+0000","last_fullsized":"2024-04-23T16:49:04.648111+0000","mapping_epoch":609,"log_start":"612'200","ondisk_log_start":"612'200","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342032384,"num_objects":5219,"num_object_clones":0,"num_object_copies":10438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10237,"num_write_kb":494592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"663'10719","reported_seq":17174,"reported_epoch":663,"state":"active+recovering","last_fresh":"2024-04-23T16:49:04.519205+0000","last_change":"2024-04-23T16:47:10.280064+0000","last_active":"2024-04-23T16:49:04.519205+0000","last_peered":"2024-04-23T16:49:04.519205+0000","last_clean":"2024-04-23T16:47:09.049801+0000","last_became_active":"2024-04-23T16:47:10.244722+0000","last_became_peered":"2024-04-23T16:47:10.244722+0000","last_unstale":"2024-04-23T16:49:04.519205+0000","last_undegraded":"2024-04-23T16:49:04.519205+0000","last_fullsized":"2024-04-23T16:49:04.519205+0000","mapping_epoch":623,"log_start":"613'700","ondisk_log_start":"613'700","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353042432,"num_objects":5387,"num_object_clones":0,"num_object_copies":10774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2168,"num_objects_unfound":0,"num_objects_dirty":5387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10719,"num_write_kb":515392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3167,"num_bytes_recovered":207552512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"5,9","objects":2168},{"shards":"9,11","objects":3219}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"663'10364","reported_seq":10482,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:04.595851+0000","last_change":"2024-04-23T16:46:26.365492+0000","last_active":"2024-04-23T16:49:04.595851+0000","last_peered":"2024-04-23T16:49:04.595851+0000","last_clean":"2024-04-23T16:49:04.595851+0000","last_became_active":"2024-04-23T16:46:26.365000+0000","last_became_peered":"2024-04-23T16:46:26.365000+0000","last_unstale":"2024-04-23T16:49:04.595851+0000","last_undegraded":"2024-04-23T16:49:04.595851+0000","last_fullsized":"2024-04-23T16:49:04.595851+0000","mapping_epoch":609,"log_start":"613'300","ondisk_log_start":"613'300","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341835776,"num_objects":5216,"num_object_clones":0,"num_object_copies":10432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10364,"num_write_kb":498560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"663'10579","reported_seq":10697,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:07.216590+0000","last_change":"2024-04-23T16:46:26.365668+0000","last_active":"2024-04-23T16:49:07.216590+0000","last_peered":"2024-04-23T16:49:07.216590+0000","last_clean":"2024-04-23T16:49:07.216590+0000","last_became_active":"2024-04-23T16:46:26.364349+0000","last_became_peered":"2024-04-23T16:46:26.364349+0000","last_unstale":"2024-04-23T16:49:07.216590+0000","last_undegraded":"2024-04-23T16:49:07.216590+0000","last_fullsized":"2024-04-23T16:49:07.216590+0000","mapping_epoch":609,"log_start":"613'500","ondisk_log_start":"613'500","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341639168,"num_objects":5213,"num_object_clones":0,"num_object_copies":10426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10579,"num_write_kb":505344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1648,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:03.757968+0000","last_change":"2024-04-23T16:45:33.395526+0000","last_active":"2024-04-23T16:49:03.757968+0000","last_peered":"2024-04-23T16:49:03.757968+0000","last_clean":"2024-04-23T16:49:03.757968+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:49:03.757968+0000","last_undegraded":"2024-04-23T16:49:03.757968+0000","last_fullsized":"2024-04-23T16:49:03.757968+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:33.395406+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:45:33.395406+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.1","version":"663'10528","reported_seq":10726,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:05.913020+0000","last_change":"2024-04-23T16:47:50.555018+0000","last_active":"2024-04-23T16:49:05.913020+0000","last_peered":"2024-04-23T16:49:05.913020+0000","last_clean":"2024-04-23T16:49:05.913020+0000","last_became_active":"2024-04-23T16:46:26.362670+0000","last_became_peered":"2024-04-23T16:46:26.362670+0000","last_unstale":"2024-04-23T16:49:05.913020+0000","last_undegraded":"2024-04-23T16:49:05.913020+0000","last_fullsized":"2024-04-23T16:49:05.913020+0000","mapping_epoch":609,"log_start":"613'500","ondisk_log_start":"613'500","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'7115","last_scrub_stamp":"2024-04-23T16:47:50.554855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:47:50.554855+0000","objects_scrubbed":7087,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:19.584547+0000","scrub_duration":1213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342622208,"num_objects":5228,"num_object_clones":0,"num_object_copies":10456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10528,"num_write_kb":504192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1519,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:03.756832+0000","last_change":"2024-04-23T16:48:00.248930+0000","last_active":"2024-04-23T16:49:03.756832+0000","last_peered":"2024-04-23T16:49:03.756832+0000","last_clean":"2024-04-23T16:49:03.756832+0000","last_became_active":"2024-04-23T16:47:10.310322+0000","last_became_peered":"2024-04-23T16:47:10.310322+0000","last_unstale":"2024-04-23T16:49:03.756832+0000","last_undegraded":"2024-04-23T16:49:03.756832+0000","last_fullsized":"2024-04-23T16:49:03.756832+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:00.248799+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:00.248799+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:24.559886+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.0","version":"663'10109","reported_seq":19246,"reported_epoch":663,"state":"active+recovering","last_fresh":"2024-04-23T16:49:04.524262+0000","last_change":"2024-04-23T16:47:10.359661+0000","last_active":"2024-04-23T16:49:04.524262+0000","last_peered":"2024-04-23T16:49:04.524262+0000","last_clean":"2024-04-23T16:47:09.045399+0000","last_became_active":"2024-04-23T16:47:10.314835+0000","last_became_peered":"2024-04-23T16:47:10.314835+0000","last_unstale":"2024-04-23T16:49:04.524262+0000","last_undegraded":"2024-04-23T16:49:04.524262+0000","last_fullsized":"2024-04-23T16:49:04.524262+0000","mapping_epoch":623,"log_start":"612'100","ondisk_log_start":"612'100","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342294528,"num_objects":5223,"num_object_clones":0,"num_object_copies":10446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":441,"num_objects_unfound":0,"num_objects_dirty":5223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10109,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4509,"num_bytes_recovered":295501824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"4,9","objects":4782},{"shards":"5,9","objects":441}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1640,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:03.751615+0000","last_change":"2024-04-23T16:48:28.280886+0000","last_active":"2024-04-23T16:49:03.751615+0000","last_peered":"2024-04-23T16:49:03.751615+0000","last_clean":"2024-04-23T16:49:03.751615+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:49:03.751615+0000","last_undegraded":"2024-04-23T16:49:03.751615+0000","last_fullsized":"2024-04-23T16:49:03.751615+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:28.280751+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:28.280751+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:56.638217+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"663'10676","reported_seq":10915,"reported_epoch":663,"state":"active+clean+scrubbing","last_fresh":"2024-04-23T16:49:07.899092+0000","last_change":"2024-04-23T16:49:07.170877+0000","last_active":"2024-04-23T16:49:07.899092+0000","last_peered":"2024-04-23T16:49:07.899092+0000","last_clean":"2024-04-23T16:49:07.899092+0000","last_became_active":"2024-04-23T16:46:26.366359+0000","last_became_peered":"2024-04-23T16:46:26.366359+0000","last_unstale":"2024-04-23T16:49:07.899092+0000","last_undegraded":"2024-04-23T16:49:07.899092+0000","last_fullsized":"2024-04-23T16:49:07.899092+0000","mapping_epoch":609,"log_start":"613'600","ondisk_log_start":"613'600","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"643'7308","last_scrub_stamp":"2024-04-23T16:47:54.493495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:47:54.493495+0000","objects_scrubbed":3549,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"scrubbing for 0s","scrub_duration":1445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339083264,"num_objects":5174,"num_object_clones":0,"num_object_copies":10348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10676,"num_write_kb":507200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1654,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:03.780083+0000","last_change":"2024-04-23T16:47:08.281155+0000","last_active":"2024-04-23T16:49:03.780083+0000","last_peered":"2024-04-23T16:49:03.780083+0000","last_clean":"2024-04-23T16:49:03.780083+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:49:03.780083+0000","last_undegraded":"2024-04-23T16:49:03.780083+0000","last_fullsized":"2024-04-23T16:49:03.780083+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:47:08.281053+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:47:08.281053+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"663'10543","reported_seq":10737,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:07.856597+0000","last_change":"2024-04-23T16:47:46.611253+0000","last_active":"2024-04-23T16:49:07.856597+0000","last_peered":"2024-04-23T16:49:07.856597+0000","last_clean":"2024-04-23T16:49:07.856597+0000","last_became_active":"2024-04-23T16:46:26.366686+0000","last_became_peered":"2024-04-23T16:46:26.366686+0000","last_unstale":"2024-04-23T16:49:07.856597+0000","last_undegraded":"2024-04-23T16:49:07.856597+0000","last_fullsized":"2024-04-23T16:49:07.856597+0000","mapping_epoch":609,"log_start":"613'500","ondisk_log_start":"613'500","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'6905","last_scrub_stamp":"2024-04-23T16:47:46.611007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:47:46.611007+0000","objects_scrubbed":6867,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:13.325560+0000","scrub_duration":1577,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332857344,"num_objects":5079,"num_object_clones":0,"num_object_copies":10158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10543,"num_write_kb":499904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1238,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:03.754346+0000","last_change":"2024-04-23T16:48:28.293172+0000","last_active":"2024-04-23T16:49:03.754346+0000","last_peered":"2024-04-23T16:49:03.754346+0000","last_clean":"2024-04-23T16:49:03.754346+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:49:03.754346+0000","last_undegraded":"2024-04-23T16:49:03.754346+0000","last_fullsized":"2024-04-23T16:49:03.754346+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:28.293038+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:48:28.293038+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:50.003967+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"663'10305","reported_seq":16186,"reported_epoch":663,"state":"active+recovering","last_fresh":"2024-04-23T16:49:04.526499+0000","last_change":"2024-04-23T16:47:10.314968+0000","last_active":"2024-04-23T16:49:04.526499+0000","last_peered":"2024-04-23T16:49:04.526499+0000","last_clean":"2024-04-23T16:47:09.038944+0000","last_became_active":"2024-04-23T16:47:10.251499+0000","last_became_peered":"2024-04-23T16:47:10.251499+0000","last_unstale":"2024-04-23T16:49:04.526499+0000","last_undegraded":"2024-04-23T16:49:04.526499+0000","last_fullsized":"2024-04-23T16:49:04.526499+0000","mapping_epoch":623,"log_start":"612'300","ondisk_log_start":"612'300","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338886656,"num_objects":5171,"num_object_clones":0,"num_object_copies":10342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2273,"num_objects_unfound":0,"num_objects_dirty":5171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10305,"num_write_kb":495232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2880,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"5,9","objects":2273},{"shards":"9,11","objects":2898}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1193,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:03.756862+0000","last_change":"2024-04-23T16:48:36.291394+0000","last_active":"2024-04-23T16:49:03.756862+0000","last_peered":"2024-04-23T16:49:03.756862+0000","last_clean":"2024-04-23T16:49:03.756862+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:49:03.756862+0000","last_undegraded":"2024-04-23T16:49:03.756862+0000","last_fullsized":"2024-04-23T16:49:03.756862+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:36.291278+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:48:36.291278+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:58.136232+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.f","version":"663'10518","reported_seq":10636,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:07.201508+0000","last_change":"2024-04-23T16:46:26.364838+0000","last_active":"2024-04-23T16:49:07.201508+0000","last_peered":"2024-04-23T16:49:07.201508+0000","last_clean":"2024-04-23T16:49:07.201508+0000","last_became_active":"2024-04-23T16:46:26.364570+0000","last_became_peered":"2024-04-23T16:46:26.364570+0000","last_unstale":"2024-04-23T16:49:07.201508+0000","last_undegraded":"2024-04-23T16:49:07.201508+0000","last_fullsized":"2024-04-23T16:49:07.201508+0000","mapping_epoch":609,"log_start":"613'500","ondisk_log_start":"613'500","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338034688,"num_objects":5158,"num_object_clones":0,"num_object_copies":10316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10518,"num_write_kb":501632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1656,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:03.758088+0000","last_change":"2024-04-23T16:45:30.258925+0000","last_active":"2024-04-23T16:49:03.758088+0000","last_peered":"2024-04-23T16:49:03.758088+0000","last_clean":"2024-04-23T16:49:03.758088+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:49:03.758088+0000","last_undegraded":"2024-04-23T16:49:03.758088+0000","last_fullsized":"2024-04-23T16:49:03.758088+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:30.258804+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:45:30.258804+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.e","version":"663'10317","reported_seq":10435,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:04.528078+0000","last_change":"2024-04-23T16:46:26.365834+0000","last_active":"2024-04-23T16:49:04.528078+0000","last_peered":"2024-04-23T16:49:04.528078+0000","last_clean":"2024-04-23T16:49:04.528078+0000","last_became_active":"2024-04-23T16:46:26.365596+0000","last_became_peered":"2024-04-23T16:46:26.365596+0000","last_unstale":"2024-04-23T16:49:04.528078+0000","last_undegraded":"2024-04-23T16:49:04.528078+0000","last_fullsized":"2024-04-23T16:49:04.528078+0000","mapping_epoch":609,"log_start":"612'300","ondisk_log_start":"612'300","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336265216,"num_objects":5131,"num_object_clones":0,"num_object_copies":10262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10317,"num_write_kb":494336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1662,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:03.752008+0000","last_change":"2024-04-23T16:48:38.816478+0000","last_active":"2024-04-23T16:49:03.752008+0000","last_peered":"2024-04-23T16:49:03.752008+0000","last_clean":"2024-04-23T16:49:03.752008+0000","last_became_active":"2024-04-23T16:47:10.266014+0000","last_became_peered":"2024-04-23T16:47:10.266014+0000","last_unstale":"2024-04-23T16:49:03.752008+0000","last_undegraded":"2024-04-23T16:49:03.752008+0000","last_fullsized":"2024-04-23T16:49:03.752008+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:38.816369+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:38.816369+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:57.044042+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"663'10712","reported_seq":10831,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:07.255473+0000","last_change":"2024-04-23T16:46:26.364586+0000","last_active":"2024-04-23T16:49:07.255473+0000","last_peered":"2024-04-23T16:49:07.255473+0000","last_clean":"2024-04-23T16:49:07.255473+0000","last_became_active":"2024-04-23T16:46:26.364302+0000","last_became_peered":"2024-04-23T16:46:26.364302+0000","last_unstale":"2024-04-23T16:49:07.255473+0000","last_undegraded":"2024-04-23T16:49:07.255473+0000","last_fullsized":"2024-04-23T16:49:07.255473+0000","mapping_epoch":609,"log_start":"613'700","ondisk_log_start":"613'700","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335740952,"num_objects":5124,"num_object_clones":0,"num_object_copies":10248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10712,"num_write_kb":506689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1540,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:03.781011+0000","last_change":"2024-04-23T16:47:10.037769+0000","last_active":"2024-04-23T16:49:03.781011+0000","last_peered":"2024-04-23T16:49:03.781011+0000","last_clean":"2024-04-23T16:49:03.781011+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:49:03.781011+0000","last_undegraded":"2024-04-23T16:49:03.781011+0000","last_fullsized":"2024-04-23T16:49:03.781011+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:47:10.037665+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:47:10.037665+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"663'10456","reported_seq":10574,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:04.452200+0000","last_change":"2024-04-23T16:46:26.364583+0000","last_active":"2024-04-23T16:49:04.452200+0000","last_peered":"2024-04-23T16:49:04.452200+0000","last_clean":"2024-04-23T16:49:04.452200+0000","last_became_active":"2024-04-23T16:46:26.364307+0000","last_became_peered":"2024-04-23T16:46:26.364307+0000","last_unstale":"2024-04-23T16:49:04.452200+0000","last_undegraded":"2024-04-23T16:49:04.452200+0000","last_fullsized":"2024-04-23T16:49:04.452200+0000","mapping_epoch":609,"log_start":"613'400","ondisk_log_start":"613'400","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342097920,"num_objects":5220,"num_object_clones":0,"num_object_copies":10440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10456,"num_write_kb":501632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1471,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:03.751832+0000","last_change":"2024-04-23T16:48:33.921605+0000","last_active":"2024-04-23T16:49:03.751832+0000","last_peered":"2024-04-23T16:49:03.751832+0000","last_clean":"2024-04-23T16:49:03.751832+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:49:03.751832+0000","last_undegraded":"2024-04-23T16:49:03.751832+0000","last_fullsized":"2024-04-23T16:49:03.751832+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:33.921450+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:48:33.921450+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:48.394938+0000","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.9","version":"663'10803","reported_seq":19241,"reported_epoch":663,"state":"active+recovering","last_fresh":"2024-04-23T16:49:07.256211+0000","last_change":"2024-04-23T16:47:10.511786+0000","last_active":"2024-04-23T16:49:07.256211+0000","last_peered":"2024-04-23T16:49:07.256211+0000","last_clean":"2024-04-23T16:47:09.051081+0000","last_became_active":"2024-04-23T16:47:10.431200+0000","last_became_peered":"2024-04-23T16:47:10.431200+0000","last_unstale":"2024-04-23T16:49:07.256211+0000","last_undegraded":"2024-04-23T16:49:07.256211+0000","last_fullsized":"2024-04-23T16:49:07.256211+0000","mapping_epoch":623,"log_start":"613'800","ondisk_log_start":"613'800","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342556672,"num_objects":5227,"num_object_clones":0,"num_object_copies":10454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1129,"num_objects_unfound":0,"num_objects_dirty":5227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10803,"num_write_kb":512960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4158,"num_bytes_recovered":272498688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":["11","5"],"object_location_counts":[{"shards":"0,11","objects":4098},{"shards":"5,11","objects":1129}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1640,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-04-23T16:49:03.751653+0000","last_change":"2024-04-23T16:48:37.224501+0000","last_active":"2024-04-23T16:49:03.751653+0000","last_peered":"2024-04-23T16:49:03.751653+0000","last_clean":"2024-04-23T16:49:03.751653+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:49:03.751653+0000","last_undegraded":"2024-04-23T16:49:03.751653+0000","last_fullsized":"2024-04-23T16:49:03.751653+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:37.224360+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:48:37.224360+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:50.365513+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5440536600,"num_objects":83017,"num_object_clones":0,"num_object_copies":166034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6011,"num_objects_unfound":0,"num_objects_dirty":83017,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167997,"num_write_kb":8032385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14714,"num_bytes_recovered":964296704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11859337216,"data_stored":11859329072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160497,"ondisk_log_size":160497,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1679360,"data_stored":1661600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057046,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3237344,"kb_used_data":2367512,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":90507808,"statfs":{"total":95995035648,"available":92679995392,"internally_reserved":0,"allocated":2424332288,"data_stored":2423920343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":890567108},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,0,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351388,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613025,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2676896,"kb_used_data":2027924,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":91068256,"statfs":{"total":95995035648,"available":93253894144,"internally_reserved":0,"allocated":2076594176,"data_stored":2076186303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":664533440},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,2,2,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841958,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1398244,"kb_used_data":988060,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92346908,"statfs":{"total":95995035648,"available":94563233792,"internally_reserved":0,"allocated":1011773440,"data_stored":1011362015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169004,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1299744,"kb_used_data":668824,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92445408,"statfs":{"total":95995035648,"available":94664097792,"internally_reserved":0,"allocated":684875776,"data_stored":684463831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":646052298},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463344,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545636,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199516,"statfs":{"total":95995035648,"available":95436304384,"internally_reserved":0,"allocated":806912,"data_stored":399039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137485,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1971424,"kb_used_data":1672788,"kb_used_omap":231,"kb_used_meta":298392,"kb_avail":91773728,"statfs":{"total":95995035648,"available":93976297472,"internally_reserved":0,"allocated":1712934912,"data_stored":1712527039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":305554320},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738830,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1603936,"kb_used_data":927188,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":92141216,"statfs":{"total":95995035648,"available":94352605184,"internally_reserved":0,"allocated":949440512,"data_stored":949032639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269501,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2658584756251,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921208,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2533348,"kb_used_data":1645596,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":91211804,"statfs":{"total":95995035648,"available":93400887296,"internally_reserved":0,"allocated":1685090304,"data_stored":1684678879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":909048261},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757684,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":944439296,"data_stored":944439296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1704460288,"data_stored":1704460288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1679425536,"data_stored":1679425536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1358233600,"data_stored":1358233600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":678629376,"data_stored":678625304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1004470272,"data_stored":1004470272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2068119552,"data_stored":2068119552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2421559296,"data_stored":2421555224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:49:11.890 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:49:12.027 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:12 smithi103 ceph-mon[39629]: from='client.17265 v1:172.21.15.103:0/3353522439' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:49:12.027 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:12 smithi103 ceph-mon[39629]: pgmap v1851: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering, 22 active+clean; 5.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 5420/162912 objects misplaced (3.327%); 12 MiB/s, 191 objects/s recovering 2024-04-23T16:49:12.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:49:12.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:49:12.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:12 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:49:12.196 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:49:12.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:12.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:49:12.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:12 smithi188 ceph-mon[44101]: from='client.17265 v1:172.21.15.103:0/3353522439' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:49:12.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:12 smithi188 ceph-mon[44101]: pgmap v1851: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering, 22 active+clean; 5.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 5420/162912 objects misplaced (3.327%); 12 MiB/s, 191 objects/s recovering 2024-04-23T16:49:12.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:49:12.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:49:12.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:12 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:49:12.264 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:12.304 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:49:12.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:12 smithi148 ceph-mon[44520]: from='client.17265 v1:172.21.15.103:0/3353522439' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:49:12.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:12 smithi148 ceph-mon[44520]: pgmap v1851: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering, 22 active+clean; 5.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 5420/162912 objects misplaced (3.327%); 12 MiB/s, 191 objects/s recovering 2024-04-23T16:49:12.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:49:12.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:49:12.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:12 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:49:12.389 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:12.424 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:49:12.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:12.428 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:49:12.463 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:49:12.464 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:49:12.530 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:49:12.570 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:12.753 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:49:12.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:12.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:49:12.857 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:49:12.886 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:12.909 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:49:12.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:12.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:49:13.015 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:49:13.044 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:13.065 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:49:13.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:13.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:49:13.163 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:13 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:49:13.163 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:13 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:49:13.169 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:49:13.216 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:13 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:49:13.216 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:13 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:49:13.221 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:13.266 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:49:13.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:13.276 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:49:13.313 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-23T16:49:13.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:13 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:49:13.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:13 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:49:13.377 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:49:13.405 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:13.425 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:49:13.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:13.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:49:13.530 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:49:13.558 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:13.579 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:49:13.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:13.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:49:13.684 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:49:13.712 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:13.733 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:49:13.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:13.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:49:13.836 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:49:13.866 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:14.075 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:49:14.296 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:14 smithi103 ceph-mon[39629]: pgmap v1852: 27 pgs: 4 active+recovering, 23 active+clean; 4.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 5105/158764 objects misplaced (3.215%); 11 MiB/s, 179 objects/s recovering 2024-04-23T16:49:14.296 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:14 smithi103 ceph-mon[39629]: osdmap e664: 12 total, 12 up, 8 in 2024-04-23T16:49:14.296 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:14 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1006216789' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:49:14.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:14 smithi148 ceph-mon[44520]: pgmap v1852: 27 pgs: 4 active+recovering, 23 active+clean; 4.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 5105/158764 objects misplaced (3.215%); 11 MiB/s, 179 objects/s recovering 2024-04-23T16:49:14.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:14 smithi148 ceph-mon[44520]: osdmap e664: 12 total, 12 up, 8 in 2024-04-23T16:49:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:14 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1006216789' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:49:14.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:14 smithi188 ceph-mon[44101]: pgmap v1852: 27 pgs: 4 active+recovering, 23 active+clean; 4.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 5105/158764 objects misplaced (3.215%); 11 MiB/s, 179 objects/s recovering 2024-04-23T16:49:14.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:14 smithi188 ceph-mon[44101]: osdmap e664: 12 total, 12 up, 8 in 2024-04-23T16:49:14.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:14 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1006216789' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:49:14.802 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:49:15.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:15 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:49:15.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:15 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:49:15.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:15 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1006216789' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:49:15.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:15 smithi103 ceph-mon[39629]: osdmap e665: 12 total, 12 up, 8 in 2024-04-23T16:49:15.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:15 smithi103 ceph-mon[39629]: 8.a scrub starts 2024-04-23T16:49:15.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:15 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:49:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:15 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:49:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:15 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1006216789' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:49:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:15 smithi148 ceph-mon[44520]: osdmap e665: 12 total, 12 up, 8 in 2024-04-23T16:49:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:15 smithi148 ceph-mon[44520]: 8.a scrub starts 2024-04-23T16:49:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:15 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:49:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:15 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:49:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:15 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1006216789' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:49:15.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:15 smithi188 ceph-mon[44101]: osdmap e665: 12 total, 12 up, 8 in 2024-04-23T16:49:15.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:15 smithi188 ceph-mon[44101]: 8.a scrub starts 2024-04-23T16:49:15.825 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:49:15.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:15.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:49:15.930 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:49:15.958 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:15.963 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:49:15.963 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:49:15.964 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:49:15.964 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:49:15.964 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:49:15.964 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:49:15.964 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:49:15.964 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:49:15.964 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:49:15.964 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:49:15.964 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:49:15.970 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:49:15.971 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:15.974 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:49:15.974 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:49:15.974 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:49:15.974 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:49:15.975 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:49:15.975 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:49:15.975 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:49:15.975 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:49:15.975 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:49:15.975 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:49:15.975 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:49:15.975 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:49:15.977 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:49:15.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:15.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:49:16.082 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:49:16.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:16 smithi103 ceph-mon[39629]: pgmap v1855: 27 pgs: 3 active+recovering, 24 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 4624/155540 objects misplaced (2.973%); 13 MiB/s, 209 objects/s recovering 2024-04-23T16:49:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:16 smithi103 ceph-mon[39629]: 8.a scrub ok 2024-04-23T16:49:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:16 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4257608465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:49:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:16.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:16.117 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:16.236 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:16.242 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:49:16.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:16.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:49:16.349 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:49:16.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:16 smithi148 ceph-mon[44520]: pgmap v1855: 27 pgs: 3 active+recovering, 24 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 4624/155540 objects misplaced (2.973%); 13 MiB/s, 209 objects/s recovering 2024-04-23T16:49:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:16 smithi148 ceph-mon[44520]: 8.a scrub ok 2024-04-23T16:49:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:16 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4257608465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:49:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:16.404 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:16.406 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:16 smithi188 ceph-mon[44101]: pgmap v1855: 27 pgs: 3 active+recovering, 24 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 4624/155540 objects misplaced (2.973%); 13 MiB/s, 209 objects/s recovering 2024-04-23T16:49:16.406 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:16 smithi188 ceph-mon[44101]: 8.a scrub ok 2024-04-23T16:49:16.406 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:16 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4257608465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:49:16.406 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:16.406 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:16.406 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:16.444 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:49:16.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:16.451 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:49:16.552 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:49:16.583 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:17.137 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:49:17.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:17.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:49:17.243 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:49:17.273 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:17.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.374 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:49:17.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:17.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:49:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:17.478 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:49:17.505 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:17.525 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:49:17.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:17.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:49:17.629 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:49:17.660 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:17.762 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:49:17.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:17.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:49:17.858 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:49:17.868 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:49:17.933 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:18.195 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:49:18.198 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:18 smithi103 ceph-mon[39629]: pgmap v1856: 27 pgs: 3 active+recovering, 24 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 4624/154592 objects misplaced (2.991%); 11 MiB/s, 173 objects/s recovering 2024-04-23T16:49:18.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:18.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:49:18.301 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:49:18.329 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:18.350 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:49:18.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:18.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:49:18.354 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:18 smithi148 ceph-mon[44520]: pgmap v1856: 27 pgs: 3 active+recovering, 24 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 4624/154592 objects misplaced (2.991%); 11 MiB/s, 173 objects/s recovering 2024-04-23T16:49:18.455 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:49:18.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:18 smithi188 ceph-mon[44101]: pgmap v1856: 27 pgs: 3 active+recovering, 24 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 4624/154592 objects misplaced (2.991%); 11 MiB/s, 173 objects/s recovering 2024-04-23T16:49:18.484 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:18.506 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:49:18.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:18.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:49:18.611 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:49:18.675 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:18.714 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:49:18.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:18.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:49:18.824 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:49:18.855 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:19.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:19 smithi103 ceph-mon[39629]: 8.5 scrub starts 2024-04-23T16:49:19.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:19 smithi103 ceph-mon[39629]: 8.5 scrub ok 2024-04-23T16:49:19.163 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:49:19.208 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:49:19.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:19.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:49:19.313 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:49:19.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:19 smithi188 ceph-mon[44101]: 8.5 scrub starts 2024-04-23T16:49:19.361 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:19 smithi188 ceph-mon[44101]: 8.5 scrub ok 2024-04-23T16:49:19.364 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:19.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:19 smithi148 ceph-mon[44520]: 8.5 scrub starts 2024-04-23T16:49:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:19 smithi148 ceph-mon[44520]: 8.5 scrub ok 2024-04-23T16:49:19.402 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:49:19.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:19.405 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:49:19.507 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:49:19.541 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:19.744 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:49:19.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:19.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:49:19.848 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:49:19.878 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:20.028 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:49:20.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:20.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:49:20.136 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:49:20.187 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:20.246 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:49:20.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:20.251 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:49:20.297 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:49:20.298 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:20.300 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:49:20.300 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:49:20.300 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:20.301 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.16995.0:178988 8.3 e665/609 8:c4c1e9ac:::benchmark_data_smithi188_155546_object53479:head v 665'11190, mlcod=665'11188)", 2024-04-23T16:49:20.301 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:49:20.297877+0000", 2024-04-23T16:49:20.302 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.001360743, 2024-04-23T16:49:20.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013981270000000001, 2024-04-23T16:49:20.302 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:20.302 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:20.302 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:49:20.302 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:20.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:20.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:20.302 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:20.297877+0000", 2024-04-23T16:49:20.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:20.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:20.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:20.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:20.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:20.297877+0000", 2024-04-23T16:49:20.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:20.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:20.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:20.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:20.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:20.297885+0000", 2024-04-23T16:49:20.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5209999999999997e-06 2024-04-23T16:49:20.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:20.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:20.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:20.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:20.297904+0000", 2024-04-23T16:49:20.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9290999999999999e-05 2024-04-23T16:49:20.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:20.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:20.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:20.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:20.297928+0000", 2024-04-23T16:49:20.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3822999999999999e-05 2024-04-23T16:49:20.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:20.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:20.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:20.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:20.297941+0000", 2024-04-23T16:49:20.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3363000000000001e-05 2024-04-23T16:49:20.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:20.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:20.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:20.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:20.298083+0000", 2024-04-23T16:49:20.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014118999999999999 2024-04-23T16:49:20.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:20.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:20.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:20.305 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:20.298109+0000", 2024-04-23T16:49:20.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6081e-05 2024-04-23T16:49:20.305 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:20.305 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:20.305 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:20.305 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:20.305 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:49:20.305 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 1 2024-04-23T16:49:20.305 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:49:20.306 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:49:20.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:20 smithi103 ceph-mon[39629]: pgmap v1857: 27 pgs: 3 active+recovering, 24 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 746 op/s; 4258/150958 objects misplaced (2.821%); 9.1 MiB/s, 145 objects/s recovering 2024-04-23T16:49:20.339 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:20.352 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:49:20.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:20 smithi148 ceph-mon[44520]: pgmap v1857: 27 pgs: 3 active+recovering, 24 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 746 op/s; 4258/150958 objects misplaced (2.821%); 9.1 MiB/s, 145 objects/s recovering 2024-04-23T16:49:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:20 smithi188 ceph-mon[44101]: pgmap v1857: 27 pgs: 3 active+recovering, 24 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 746 op/s; 4258/150958 objects misplaced (2.821%); 9.1 MiB/s, 145 objects/s recovering 2024-04-23T16:49:20.420 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:20.472 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:49:20.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:20.476 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:49:20.577 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:49:20.604 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:20.624 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:49:20.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:20.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:49:20.729 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:49:20.759 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:20.782 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:49:20.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:20.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:49:20.887 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:49:20.914 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:20.933 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:49:20.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:20.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:49:21.037 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:49:21.065 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:21.086 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:49:21.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:21.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:49:21.190 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:49:21.224 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:21.474 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:49:21.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:21 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:49:21.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:21 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:49:21.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:21 smithi103 ceph-mon[39629]: from='client.17283 v1:172.21.15.103:0/3581536273' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:49:21.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:21 smithi103 ceph-mon[39629]: osdmap e666: 12 total, 12 up, 8 in 2024-04-23T16:49:21.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:21 smithi103 ceph-mon[39629]: pgmap v1859: 27 pgs: 3 active+recovering, 24 active+clean; 4.5 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 721 op/s; 3899/147420 objects misplaced (2.645%); 9.6 MiB/s, 153 objects/s recovering 2024-04-23T16:49:21.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:21.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:49:21.708 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:49:21.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:21 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:49:21.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:21 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:49:21.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:21 smithi188 ceph-mon[44101]: from='client.17283 v1:172.21.15.103:0/3581536273' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:49:21.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:21 smithi188 ceph-mon[44101]: osdmap e666: 12 total, 12 up, 8 in 2024-04-23T16:49:21.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:21 smithi188 ceph-mon[44101]: pgmap v1859: 27 pgs: 3 active+recovering, 24 active+clean; 4.5 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 721 op/s; 3899/147420 objects misplaced (2.645%); 9.6 MiB/s, 153 objects/s recovering 2024-04-23T16:49:21.733 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:21 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:49:21.733 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:21 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:49:21.733 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:21 smithi148 ceph-mon[44520]: from='client.17283 v1:172.21.15.103:0/3581536273' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:49:21.733 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:21 smithi148 ceph-mon[44520]: osdmap e666: 12 total, 12 up, 8 in 2024-04-23T16:49:21.733 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:21 smithi148 ceph-mon[44520]: pgmap v1859: 27 pgs: 3 active+recovering, 24 active+clean; 4.5 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 721 op/s; 3899/147420 objects misplaced (2.645%); 9.6 MiB/s, 153 objects/s recovering 2024-04-23T16:49:21.735 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:21.755 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:49:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:21.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:49:21.859 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:49:21.907 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:21.952 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:49:21.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:21.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:49:22.061 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:49:22.089 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:22.109 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:49:22.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:22.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:49:22.213 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:49:22.248 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:22.980 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:22 smithi188 ceph-mon[44101]: 8.1 scrub starts 2024-04-23T16:49:22.980 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:22 smithi188 ceph-mon[44101]: 8.1 scrub ok 2024-04-23T16:49:23.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:22 smithi103 ceph-mon[39629]: 8.1 scrub starts 2024-04-23T16:49:23.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:22 smithi103 ceph-mon[39629]: 8.1 scrub ok 2024-04-23T16:49:23.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:22 smithi148 ceph-mon[44520]: 8.1 scrub starts 2024-04-23T16:49:23.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:22 smithi148 ceph-mon[44520]: 8.1 scrub ok 2024-04-23T16:49:23.201 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:49:23.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:23.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:49:23.306 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:49:23.341 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:23.973 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:23 smithi103 ceph-mon[39629]: pgmap v1860: 27 pgs: 3 active+recovering, 24 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 901 op/s; 3460/142794 objects misplaced (2.423%); 12 MiB/s, 185 objects/s recovering 2024-04-23T16:49:23.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:23 smithi188 ceph-mon[44101]: pgmap v1860: 27 pgs: 3 active+recovering, 24 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 901 op/s; 3460/142794 objects misplaced (2.423%); 12 MiB/s, 185 objects/s recovering 2024-04-23T16:49:24.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:23 smithi148 ceph-mon[44520]: pgmap v1860: 27 pgs: 3 active+recovering, 24 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 901 op/s; 3460/142794 objects misplaced (2.423%); 12 MiB/s, 185 objects/s recovering 2024-04-23T16:49:24.292 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:49:24.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:24.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:49:24.305 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-23T16:49:24.395 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:49:24.425 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:24.450 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:49:24.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:24.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:49:24.554 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:49:24.591 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:24.630 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:49:24.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:24.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:49:24.669 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1857,"stamp":"2024-04-23T16:49:18.921315+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4946985528,"num_objects":75479,"num_object_clones":0,"num_object_copies":150958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4258,"num_objects_unfound":0,"num_objects_dirty":75479,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175537,"num_write_kb":8032385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16469,"num_bytes_recovered":1079583808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162166,"ondisk_log_size":162166,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14430636,"kb_used_data":9454484,"kb_used_omap":242,"kb_used_meta":4975757,"kb_avail":735530580,"statfs":{"total":767960285184,"available":753183313920,"internally_reserved":0,"allocated":9681391616,"data_stored":9678035082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":247821,"internal_metadata":5095176179},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-391708672,"num_objects":-5977,"num_object_clones":0,"num_object_copies":-11954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1162,"num_objects_unfound":0,"num_objects_dirty":-5977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5977,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1162,"num_bytes_recovered":76152832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001588"},"pg_stats":[{"pgid":"8.7","version":"665'10845","reported_seq":11100,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:15.950333+0000","last_change":"2024-04-23T16:49:08.424575+0000","last_active":"2024-04-23T16:49:15.950333+0000","last_peered":"2024-04-23T16:49:15.950333+0000","last_clean":"2024-04-23T16:49:15.950333+0000","last_became_active":"2024-04-23T16:46:26.363422+0000","last_became_peered":"2024-04-23T16:46:26.363422+0000","last_unstale":"2024-04-23T16:49:15.950333+0000","last_undegraded":"2024-04-23T16:49:15.950333+0000","last_fullsized":"2024-04-23T16:49:15.950333+0000","mapping_epoch":609,"log_start":"613'800","ondisk_log_start":"613'800","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"663'10506","last_scrub_stamp":"2024-04-23T16:49:08.424442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:08.424442+0000","objects_scrubbed":5063,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:20.445332+0000","scrub_duration":991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308346880,"num_objects":4705,"num_object_clones":0,"num_object_copies":9410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10845,"num_write_kb":497600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"665'11229","reported_seq":11351,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:17.266944+0000","last_change":"2024-04-23T16:46:26.365698+0000","last_active":"2024-04-23T16:49:17.266944+0000","last_peered":"2024-04-23T16:49:17.266944+0000","last_clean":"2024-04-23T16:49:17.266944+0000","last_became_active":"2024-04-23T16:46:26.365309+0000","last_became_peered":"2024-04-23T16:46:26.365309+0000","last_unstale":"2024-04-23T16:49:17.266944+0000","last_undegraded":"2024-04-23T16:49:17.266944+0000","last_fullsized":"2024-04-23T16:49:17.266944+0000","mapping_epoch":609,"log_start":"613'1200","ondisk_log_start":"613'1200","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301400064,"num_objects":4599,"num_object_clones":0,"num_object_copies":9198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11229,"num_write_kb":506496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.5","version":"665'10701","reported_seq":10823,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:14.678608+0000","last_change":"2024-04-23T16:46:26.365075+0000","last_active":"2024-04-23T16:49:14.678608+0000","last_peered":"2024-04-23T16:49:14.678608+0000","last_clean":"2024-04-23T16:49:14.678608+0000","last_became_active":"2024-04-23T16:46:26.364780+0000","last_became_peered":"2024-04-23T16:46:26.364780+0000","last_unstale":"2024-04-23T16:49:14.678608+0000","last_undegraded":"2024-04-23T16:49:14.678608+0000","last_fullsized":"2024-04-23T16:49:14.678608+0000","mapping_epoch":609,"log_start":"613'700","ondisk_log_start":"613'700","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311623680,"num_objects":4755,"num_object_clones":0,"num_object_copies":9510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10701,"num_write_kb":494592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"665'11169","reported_seq":18277,"reported_epoch":665,"state":"active+recovering","last_fresh":"2024-04-23T16:49:14.543349+0000","last_change":"2024-04-23T16:47:10.280064+0000","last_active":"2024-04-23T16:49:14.543349+0000","last_peered":"2024-04-23T16:49:14.543349+0000","last_clean":"2024-04-23T16:47:09.049801+0000","last_became_active":"2024-04-23T16:47:10.244722+0000","last_became_peered":"2024-04-23T16:47:10.244722+0000","last_unstale":"2024-04-23T16:49:14.543349+0000","last_undegraded":"2024-04-23T16:49:14.543349+0000","last_fullsized":"2024-04-23T16:49:14.543349+0000","mapping_epoch":623,"log_start":"613'1100","ondisk_log_start":"613'1100","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323551232,"num_objects":4937,"num_object_clones":0,"num_object_copies":9874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1841,"num_objects_unfound":0,"num_objects_dirty":4937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11169,"num_write_kb":515392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3494,"num_bytes_recovered":228982784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"5,9","objects":1841},{"shards":"9,11","objects":3096}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"665'10801","reported_seq":10923,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:14.513518+0000","last_change":"2024-04-23T16:46:26.365492+0000","last_active":"2024-04-23T16:49:14.513518+0000","last_peered":"2024-04-23T16:49:14.513518+0000","last_clean":"2024-04-23T16:49:14.513518+0000","last_became_active":"2024-04-23T16:46:26.365000+0000","last_became_peered":"2024-04-23T16:46:26.365000+0000","last_unstale":"2024-04-23T16:49:14.513518+0000","last_undegraded":"2024-04-23T16:49:14.513518+0000","last_fullsized":"2024-04-23T16:49:14.513518+0000","mapping_epoch":609,"log_start":"613'800","ondisk_log_start":"613'800","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313196544,"num_objects":4779,"num_object_clones":0,"num_object_copies":9558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10801,"num_write_kb":498560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"665'11039","reported_seq":11161,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:17.309770+0000","last_change":"2024-04-23T16:46:26.365668+0000","last_active":"2024-04-23T16:49:17.309770+0000","last_peered":"2024-04-23T16:49:17.309770+0000","last_clean":"2024-04-23T16:49:17.309770+0000","last_became_active":"2024-04-23T16:46:26.364349+0000","last_became_peered":"2024-04-23T16:46:26.364349+0000","last_unstale":"2024-04-23T16:49:17.309770+0000","last_undegraded":"2024-04-23T16:49:17.309770+0000","last_fullsized":"2024-04-23T16:49:17.309770+0000","mapping_epoch":609,"log_start":"613'1000","ondisk_log_start":"613'1000","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311492608,"num_objects":4753,"num_object_clones":0,"num_object_copies":9506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11039,"num_write_kb":505344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1652,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:14.078058+0000","last_change":"2024-04-23T16:45:33.395526+0000","last_active":"2024-04-23T16:49:14.078058+0000","last_peered":"2024-04-23T16:49:14.078058+0000","last_clean":"2024-04-23T16:49:14.078058+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:49:14.078058+0000","last_undegraded":"2024-04-23T16:49:14.078058+0000","last_fullsized":"2024-04-23T16:49:14.078058+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:33.395406+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:45:33.395406+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.1","version":"665'11017","reported_seq":11219,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:15.886218+0000","last_change":"2024-04-23T16:47:50.555018+0000","last_active":"2024-04-23T16:49:15.886218+0000","last_peered":"2024-04-23T16:49:15.886218+0000","last_clean":"2024-04-23T16:49:15.886218+0000","last_became_active":"2024-04-23T16:46:26.362670+0000","last_became_peered":"2024-04-23T16:46:26.362670+0000","last_unstale":"2024-04-23T16:49:15.886218+0000","last_undegraded":"2024-04-23T16:49:15.886218+0000","last_fullsized":"2024-04-23T16:49:15.886218+0000","mapping_epoch":609,"log_start":"613'1000","ondisk_log_start":"613'1000","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'7115","last_scrub_stamp":"2024-04-23T16:47:50.554855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:47:50.554855+0000","objects_scrubbed":7087,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:19.584547+0000","scrub_duration":1213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310575104,"num_objects":4739,"num_object_clones":0,"num_object_copies":9478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11017,"num_write_kb":504192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1523,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:14.098057+0000","last_change":"2024-04-23T16:48:00.248930+0000","last_active":"2024-04-23T16:49:14.098057+0000","last_peered":"2024-04-23T16:49:14.098057+0000","last_clean":"2024-04-23T16:49:14.098057+0000","last_became_active":"2024-04-23T16:47:10.310322+0000","last_became_peered":"2024-04-23T16:47:10.310322+0000","last_unstale":"2024-04-23T16:49:14.098057+0000","last_undegraded":"2024-04-23T16:49:14.098057+0000","last_fullsized":"2024-04-23T16:49:14.098057+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:00.248799+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:00.248799+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:24.559886+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.0","version":"665'10598","reported_seq":20625,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:14.527297+0000","last_change":"2024-04-23T16:49:11.451708+0000","last_active":"2024-04-23T16:49:14.527297+0000","last_peered":"2024-04-23T16:49:14.527297+0000","last_clean":"2024-04-23T16:49:14.527297+0000","last_became_active":"2024-04-23T16:47:10.314835+0000","last_became_peered":"2024-04-23T16:47:10.314835+0000","last_unstale":"2024-04-23T16:49:14.527297+0000","last_undegraded":"2024-04-23T16:49:14.527297+0000","last_fullsized":"2024-04-23T16:49:14.527297+0000","mapping_epoch":623,"log_start":"613'500","ondisk_log_start":"613'500","created":609,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310247424,"num_objects":4734,"num_object_clones":0,"num_object_copies":9468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10598,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4950,"num_bytes_recovered":324403200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1644,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:14.082242+0000","last_change":"2024-04-23T16:48:28.280886+0000","last_active":"2024-04-23T16:49:14.082242+0000","last_peered":"2024-04-23T16:49:14.082242+0000","last_clean":"2024-04-23T16:49:14.082242+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:49:14.082242+0000","last_undegraded":"2024-04-23T16:49:14.082242+0000","last_fullsized":"2024-04-23T16:49:14.082242+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:28.280751+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:28.280751+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:56.638217+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"665'11151","reported_seq":11415,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:17.889715+0000","last_change":"2024-04-23T16:49:08.214265+0000","last_active":"2024-04-23T16:49:17.889715+0000","last_peered":"2024-04-23T16:49:17.889715+0000","last_clean":"2024-04-23T16:49:17.889715+0000","last_became_active":"2024-04-23T16:46:26.366359+0000","last_became_peered":"2024-04-23T16:46:26.366359+0000","last_unstale":"2024-04-23T16:49:17.889715+0000","last_undegraded":"2024-04-23T16:49:17.889715+0000","last_fullsized":"2024-04-23T16:49:17.889715+0000","mapping_epoch":609,"log_start":"613'1100","ondisk_log_start":"613'1100","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"663'10692","last_scrub_stamp":"2024-04-23T16:49:08.214052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:08.214052+0000","objects_scrubbed":5183,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:11.250982+0000","scrub_duration":1042,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307953664,"num_objects":4699,"num_object_clones":0,"num_object_copies":9398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11151,"num_write_kb":507200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1658,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:14.073948+0000","last_change":"2024-04-23T16:47:08.281155+0000","last_active":"2024-04-23T16:49:14.073948+0000","last_peered":"2024-04-23T16:49:14.073948+0000","last_clean":"2024-04-23T16:49:14.073948+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:49:14.073948+0000","last_undegraded":"2024-04-23T16:49:14.073948+0000","last_fullsized":"2024-04-23T16:49:14.073948+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:47:08.281053+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:47:08.281053+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"665'11049","reported_seq":11301,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:17.897101+0000","last_change":"2024-04-23T16:49:15.382664+0000","last_active":"2024-04-23T16:49:17.897101+0000","last_peered":"2024-04-23T16:49:17.897101+0000","last_clean":"2024-04-23T16:49:17.897101+0000","last_became_active":"2024-04-23T16:46:26.366686+0000","last_became_peered":"2024-04-23T16:46:26.366686+0000","last_unstale":"2024-04-23T16:49:17.897101+0000","last_undegraded":"2024-04-23T16:49:17.897101+0000","last_fullsized":"2024-04-23T16:49:17.897101+0000","mapping_epoch":609,"log_start":"613'1000","ondisk_log_start":"613'1000","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"665'10905","last_scrub_stamp":"2024-04-23T16:49:15.382180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:15.382180+0000","objects_scrubbed":4744,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:32.221200+0000","scrub_duration":1175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299696128,"num_objects":4573,"num_object_clones":0,"num_object_copies":9146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11049,"num_write_kb":499904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1242,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:14.071990+0000","last_change":"2024-04-23T16:48:28.293172+0000","last_active":"2024-04-23T16:49:14.071990+0000","last_peered":"2024-04-23T16:49:14.071990+0000","last_clean":"2024-04-23T16:49:14.071990+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:49:14.071990+0000","last_undegraded":"2024-04-23T16:49:14.071990+0000","last_fullsized":"2024-04-23T16:49:14.071990+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:28.293038+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:48:28.293038+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:50.003967+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"665'10767","reported_seq":17260,"reported_epoch":665,"state":"active+recovering","last_fresh":"2024-04-23T16:49:14.540538+0000","last_change":"2024-04-23T16:47:10.314968+0000","last_active":"2024-04-23T16:49:14.540538+0000","last_peered":"2024-04-23T16:49:14.540538+0000","last_clean":"2024-04-23T16:47:09.038944+0000","last_became_active":"2024-04-23T16:47:10.251499+0000","last_became_peered":"2024-04-23T16:47:10.251499+0000","last_unstale":"2024-04-23T16:49:14.540538+0000","last_undegraded":"2024-04-23T16:49:14.540538+0000","last_fullsized":"2024-04-23T16:49:14.540538+0000","mapping_epoch":623,"log_start":"613'700","ondisk_log_start":"613'700","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308609024,"num_objects":4709,"num_object_clones":0,"num_object_copies":9418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1969,"num_objects_unfound":0,"num_objects_dirty":4709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10767,"num_write_kb":495232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3184,"num_bytes_recovered":208666624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"5,9","objects":1969},{"shards":"9,11","objects":2740}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1197,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:14.097232+0000","last_change":"2024-04-23T16:48:36.291394+0000","last_active":"2024-04-23T16:49:14.097232+0000","last_peered":"2024-04-23T16:49:14.097232+0000","last_clean":"2024-04-23T16:49:14.097232+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:49:14.097232+0000","last_undegraded":"2024-04-23T16:49:14.097232+0000","last_fullsized":"2024-04-23T16:49:14.097232+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:36.291278+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:48:36.291278+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:58.136232+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.f","version":"665'11014","reported_seq":11136,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:17.310895+0000","last_change":"2024-04-23T16:46:26.364838+0000","last_active":"2024-04-23T16:49:17.310895+0000","last_peered":"2024-04-23T16:49:17.310895+0000","last_clean":"2024-04-23T16:49:17.310895+0000","last_became_active":"2024-04-23T16:46:26.364570+0000","last_became_peered":"2024-04-23T16:46:26.364570+0000","last_unstale":"2024-04-23T16:49:17.310895+0000","last_undegraded":"2024-04-23T16:49:17.310895+0000","last_fullsized":"2024-04-23T16:49:17.310895+0000","mapping_epoch":609,"log_start":"613'1000","ondisk_log_start":"613'1000","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305528832,"num_objects":4662,"num_object_clones":0,"num_object_copies":9324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11014,"num_write_kb":501632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1660,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:14.072888+0000","last_change":"2024-04-23T16:45:30.258925+0000","last_active":"2024-04-23T16:49:14.072888+0000","last_peered":"2024-04-23T16:49:14.072888+0000","last_clean":"2024-04-23T16:49:14.072888+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:49:14.072888+0000","last_undegraded":"2024-04-23T16:49:14.072888+0000","last_fullsized":"2024-04-23T16:49:14.072888+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:30.258804+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:45:30.258804+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.e","version":"665'10743","reported_seq":10865,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:14.602816+0000","last_change":"2024-04-23T16:46:26.365834+0000","last_active":"2024-04-23T16:49:14.602816+0000","last_peered":"2024-04-23T16:49:14.602816+0000","last_clean":"2024-04-23T16:49:14.602816+0000","last_became_active":"2024-04-23T16:46:26.365596+0000","last_became_peered":"2024-04-23T16:46:26.365596+0000","last_unstale":"2024-04-23T16:49:14.602816+0000","last_undegraded":"2024-04-23T16:49:14.602816+0000","last_fullsized":"2024-04-23T16:49:14.602816+0000","mapping_epoch":609,"log_start":"613'700","ondisk_log_start":"613'700","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308346880,"num_objects":4705,"num_object_clones":0,"num_object_copies":9410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10743,"num_write_kb":494336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1666,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:14.824784+0000","last_change":"2024-04-23T16:48:38.816478+0000","last_active":"2024-04-23T16:49:14.824784+0000","last_peered":"2024-04-23T16:49:14.824784+0000","last_clean":"2024-04-23T16:49:14.824784+0000","last_became_active":"2024-04-23T16:47:10.266014+0000","last_became_peered":"2024-04-23T16:47:10.266014+0000","last_unstale":"2024-04-23T16:49:14.824784+0000","last_undegraded":"2024-04-23T16:49:14.824784+0000","last_fullsized":"2024-04-23T16:49:14.824784+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:38.816369+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:38.816369+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:57.044042+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"665'11213","reported_seq":11334,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:17.315582+0000","last_change":"2024-04-23T16:46:26.364586+0000","last_active":"2024-04-23T16:49:17.315582+0000","last_peered":"2024-04-23T16:49:17.315582+0000","last_clean":"2024-04-23T16:49:17.315582+0000","last_became_active":"2024-04-23T16:46:26.364302+0000","last_became_peered":"2024-04-23T16:46:26.364302+0000","last_unstale":"2024-04-23T16:49:17.315582+0000","last_undegraded":"2024-04-23T16:49:17.315582+0000","last_fullsized":"2024-04-23T16:49:17.315582+0000","mapping_epoch":609,"log_start":"613'1200","ondisk_log_start":"613'1200","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302907416,"num_objects":4623,"num_object_clones":0,"num_object_copies":9246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4623,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11213,"num_write_kb":506689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1545,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:14.079551+0000","last_change":"2024-04-23T16:47:10.037769+0000","last_active":"2024-04-23T16:49:14.079551+0000","last_peered":"2024-04-23T16:49:14.079551+0000","last_clean":"2024-04-23T16:49:14.079551+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:49:14.079551+0000","last_undegraded":"2024-04-23T16:49:14.079551+0000","last_fullsized":"2024-04-23T16:49:14.079551+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:47:10.037665+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:47:10.037665+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"665'10903","reported_seq":11024,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:14.471427+0000","last_change":"2024-04-23T16:46:26.364583+0000","last_active":"2024-04-23T16:49:14.471427+0000","last_peered":"2024-04-23T16:49:14.471427+0000","last_clean":"2024-04-23T16:49:14.471427+0000","last_became_active":"2024-04-23T16:46:26.364307+0000","last_became_peered":"2024-04-23T16:46:26.364307+0000","last_unstale":"2024-04-23T16:49:14.471427+0000","last_undegraded":"2024-04-23T16:49:14.471427+0000","last_fullsized":"2024-04-23T16:49:14.471427+0000","mapping_epoch":609,"log_start":"613'900","ondisk_log_start":"613'900","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312803328,"num_objects":4773,"num_object_clones":0,"num_object_copies":9546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10903,"num_write_kb":501632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1475,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:14.824784+0000","last_change":"2024-04-23T16:48:33.921605+0000","last_active":"2024-04-23T16:49:14.824784+0000","last_peered":"2024-04-23T16:49:14.824784+0000","last_clean":"2024-04-23T16:49:14.824784+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:49:14.824784+0000","last_undegraded":"2024-04-23T16:49:14.824784+0000","last_fullsized":"2024-04-23T16:49:14.824784+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:33.921450+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:48:33.921450+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:48.394938+0000","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.9","version":"665'11298","reported_seq":21102,"reported_epoch":665,"state":"active+recovering","last_fresh":"2024-04-23T16:49:17.323054+0000","last_change":"2024-04-23T16:47:10.511786+0000","last_active":"2024-04-23T16:49:17.323054+0000","last_peered":"2024-04-23T16:49:17.323054+0000","last_clean":"2024-04-23T16:47:09.051081+0000","last_became_active":"2024-04-23T16:47:10.431200+0000","last_became_peered":"2024-04-23T16:47:10.431200+0000","last_unstale":"2024-04-23T16:49:17.323054+0000","last_undegraded":"2024-04-23T16:49:17.323054+0000","last_fullsized":"2024-04-23T16:49:17.323054+0000","mapping_epoch":623,"log_start":"613'1200","ondisk_log_start":"613'1200","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310116352,"num_objects":4732,"num_object_clones":0,"num_object_copies":9464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":448,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11298,"num_write_kb":512960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4839,"num_bytes_recovered":317128704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":["11","5"],"object_location_counts":[{"shards":"0,11","objects":4284},{"shards":"5,11","objects":448}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1644,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-23T16:49:14.082242+0000","last_change":"2024-04-23T16:48:37.224501+0000","last_active":"2024-04-23T16:49:14.082242+0000","last_peered":"2024-04-23T16:49:14.082242+0000","last_clean":"2024-04-23T16:49:14.082242+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:49:14.082242+0000","last_undegraded":"2024-04-23T16:49:14.082242+0000","last_fullsized":"2024-04-23T16:49:14.082242+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:37.224360+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:48:37.224360+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:50.365513+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4946395160,"num_objects":75477,"num_object_clones":0,"num_object_copies":150954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4258,"num_objects_unfound":0,"num_objects_dirty":75477,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175537,"num_write_kb":8032385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16467,"num_bytes_recovered":1079181312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10972897280,"data_stored":10972889136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160637,"ondisk_log_size":160637,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1679360,"data_stored":1661600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057048,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3031032,"kb_used_data":2161328,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":90714120,"statfs":{"total":95995035648,"available":92891258880,"internally_reserved":0,"allocated":2213199872,"data_stored":2212775869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":890567108},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351390,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613027,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2491756,"kb_used_data":1851104,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":91253396,"statfs":{"total":95995035648,"available":93443477504,"internally_reserved":0,"allocated":1895530496,"data_stored":1895116619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":656013760},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841960,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1312444,"kb_used_data":898036,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92432708,"statfs":{"total":95995035648,"available":94651092992,"internally_reserved":0,"allocated":919588864,"data_stored":919165381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169006,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1242304,"kb_used_data":607216,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92502848,"statfs":{"total":95995035648,"available":94722916352,"internally_reserved":0,"allocated":621789184,"data_stored":621365181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":650312138},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463346,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545644,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199508,"statfs":{"total":95995035648,"available":95436296192,"internally_reserved":0,"allocated":819200,"data_stored":405323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137487,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1829428,"kb_used_data":1522476,"kb_used_omap":231,"kb_used_meta":306712,"kb_avail":91915724,"statfs":{"total":95995035648,"available":94121701376,"internally_reserved":0,"allocated":1559015424,"data_stored":1558595493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":314074000},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738832,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1567020,"kb_used_data":881952,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":92178132,"statfs":{"total":95995035648,"available":94390407168,"internally_reserved":0,"allocated":903118848,"data_stored":902704971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269503,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2658584756253,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921210,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2411008,"kb_used_data":1531572,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":91334144,"statfs":{"total":95995035648,"available":93526163456,"internally_reserved":0,"allocated":1568329728,"data_stored":1567906245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757686,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":898957312,"data_stored":898957312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1552023552,"data_stored":1552023552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1560346624,"data_stored":1560346624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1344536576,"data_stored":1344536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":614535168,"data_stored":614531096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":909836288,"data_stored":909836288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1882390528,"data_stored":1882390528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2210271232,"data_stored":2210267160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:49:24.671 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:49:24.733 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:49:24.791 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:24.851 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:49:24.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:24.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:49:24.958 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:49:25.016 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:25.061 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:49:25.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:25.065 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:49:25.166 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:49:25.197 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:25.558 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:49:25.562 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:25 smithi103 ceph-mon[39629]: osdmap e667: 12 total, 12 up, 8 in 2024-04-23T16:49:25.562 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:25 smithi103 ceph-mon[39629]: pgmap v1862: 27 pgs: 3 active+recovering, 24 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 961 op/s; 3065/139198 objects misplaced (2.202%); 12 MiB/s, 194 objects/s recovering 2024-04-23T16:49:25.562 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:25 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2597322011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:49:25.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:25.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:49:25.654 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:49:25.654 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:49:25.654 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:49:25.655 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:49:25.655 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:49:25.655 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:49:25.655 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:49:25.655 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:49:25.655 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:49:25.655 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:49:25.655 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:49:25.655 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:49:25.655 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:49:25.655 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:49:25.655 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:49:25.656 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:49:25.656 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:49:25.656 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:49:25.656 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:49:25.656 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:49:25.656 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:49:25.656 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:49:25.656 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:49:25.656 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:49:25.665 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:49:25.690 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:25 smithi148 ceph-mon[44520]: osdmap e667: 12 total, 12 up, 8 in 2024-04-23T16:49:25.690 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:25 smithi148 ceph-mon[44520]: pgmap v1862: 27 pgs: 3 active+recovering, 24 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 961 op/s; 3065/139198 objects misplaced (2.202%); 12 MiB/s, 194 objects/s recovering 2024-04-23T16:49:25.690 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:25 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2597322011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:49:25.691 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:25.712 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:49:25.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:25.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:49:25.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:25 smithi188 ceph-mon[44101]: osdmap e667: 12 total, 12 up, 8 in 2024-04-23T16:49:25.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:25 smithi188 ceph-mon[44101]: pgmap v1862: 27 pgs: 3 active+recovering, 24 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 961 op/s; 3065/139198 objects misplaced (2.202%); 12 MiB/s, 194 objects/s recovering 2024-04-23T16:49:25.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:25 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2597322011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:49:25.816 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:49:25.847 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:25.920 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:49:25.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:25.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:49:26.024 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:49:26.056 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:26.129 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:49:26.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:26.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:49:26.234 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:49:26.261 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:26.283 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:49:26.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:26.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:49:26.386 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:49:26.413 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:26.435 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:49:26.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:26.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:49:26.539 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:49:26.563 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:26 smithi148 ceph-mon[44520]: 8.2 scrub starts 2024-04-23T16:49:26.563 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.564 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.564 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:26 smithi148 ceph-mon[44520]: 1.8 scrub starts 2024-04-23T16:49:26.564 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.564 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:26 smithi148 ceph-mon[44520]: 8.2 scrub ok 2024-04-23T16:49:26.564 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.564 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.564 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.564 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.564 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:26 smithi148 ceph-mon[44520]: 1.8 scrub ok 2024-04-23T16:49:26.564 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.564 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.564 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.564 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.564 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.565 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:26.587 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:49:26.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:26.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:49:26.690 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:49:26.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:26 smithi188 ceph-mon[44101]: 8.2 scrub starts 2024-04-23T16:49:26.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:26 smithi188 ceph-mon[44101]: 1.8 scrub starts 2024-04-23T16:49:26.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:26 smithi188 ceph-mon[44101]: 8.2 scrub ok 2024-04-23T16:49:26.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:26 smithi188 ceph-mon[44101]: 1.8 scrub ok 2024-04-23T16:49:26.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.728 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.728 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.764 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:26.794 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:49:26.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:26.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:49:26.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:26 smithi103 ceph-mon[39629]: 8.2 scrub starts 2024-04-23T16:49:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:26 smithi103 ceph-mon[39629]: 1.8 scrub starts 2024-04-23T16:49:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:26 smithi103 ceph-mon[39629]: 8.2 scrub ok 2024-04-23T16:49:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:26 smithi103 ceph-mon[39629]: 1.8 scrub ok 2024-04-23T16:49:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:26.897 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:49:26.923 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:26.939 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:26.969 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:49:26.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:26.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:49:26.978 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:27.075 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:49:27.102 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:27.126 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:49:27.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:27.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:49:27.230 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:49:27.268 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:27.463 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:49:27.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:27 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/970335628' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:49:27.728 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:27 smithi188 ceph-mon[44101]: pgmap v1863: 27 pgs: 3 active+recovering, 24 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 800 op/s; 3065/138142 objects misplaced (2.219%); 9.3 MiB/s, 149 objects/s recovering 2024-04-23T16:49:27.806 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:27 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/970335628' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:49:27.807 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:27 smithi103 ceph-mon[39629]: pgmap v1863: 27 pgs: 3 active+recovering, 24 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 800 op/s; 3065/138142 objects misplaced (2.219%); 9.3 MiB/s, 149 objects/s recovering 2024-04-23T16:49:27.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:27 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/970335628' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:49:27.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:27 smithi148 ceph-mon[44520]: pgmap v1863: 27 pgs: 3 active+recovering, 24 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 800 op/s; 3065/138142 objects misplaced (2.219%); 9.3 MiB/s, 149 objects/s recovering 2024-04-23T16:49:28.518 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:49:28.520 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:28 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:49:28.521 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:28 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/970335628' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:49:28.521 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:28 smithi103 ceph-mon[39629]: osdmap e668: 12 total, 12 up, 8 in 2024-04-23T16:49:28.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:28.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:49:28.623 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:49:28.650 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:28.670 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:49:28.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:28.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:49:28.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:28 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:49:28.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:28 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/970335628' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:49:28.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:28 smithi188 ceph-mon[44101]: osdmap e668: 12 total, 12 up, 8 in 2024-04-23T16:49:28.774 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:49:28.800 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:28 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:49:28.800 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:28 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/970335628' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:49:28.800 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:28 smithi148 ceph-mon[44520]: osdmap e668: 12 total, 12 up, 8 in 2024-04-23T16:49:28.802 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:28.822 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:49:28.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:28.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:49:28.926 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:49:28.952 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:49:28.993 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:29.049 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:49:29.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:29.052 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:49:29.153 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:49:29.209 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:29.242 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:49:29.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:29.247 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:49:29.348 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:49:29.378 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:29.728 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:29 smithi188 ceph-mon[44101]: pgmap v1865: 27 pgs: 2 active+recovering, 25 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 816 op/s; 3056/134348 objects misplaced (2.275%); 6.6 MiB/s, 105 objects/s recovering 2024-04-23T16:49:29.728 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:29 smithi188 ceph-mon[44101]: osdmap e669: 12 total, 12 up, 8 in 2024-04-23T16:49:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:29 smithi103 ceph-mon[39629]: pgmap v1865: 27 pgs: 2 active+recovering, 25 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 816 op/s; 3056/134348 objects misplaced (2.275%); 6.6 MiB/s, 105 objects/s recovering 2024-04-23T16:49:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:29 smithi103 ceph-mon[39629]: osdmap e669: 12 total, 12 up, 8 in 2024-04-23T16:49:29.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:29 smithi148 ceph-mon[44520]: pgmap v1865: 27 pgs: 2 active+recovering, 25 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 816 op/s; 3056/134348 objects misplaced (2.275%); 6.6 MiB/s, 105 objects/s recovering 2024-04-23T16:49:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:29 smithi148 ceph-mon[44520]: osdmap e669: 12 total, 12 up, 8 in 2024-04-23T16:49:30.431 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:49:30.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:30.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:49:30.537 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:49:30.565 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:30.669 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:49:30.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:30.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:49:30.775 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:49:30.806 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:30.826 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:49:30.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:30.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:49:30.930 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:49:30.961 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:31.414 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:49:31.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:31.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:49:31.519 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:49:31.539 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:49:31.539 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:49:31.572 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:31.609 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:49:31.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:31.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:49:31.718 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:49:31.745 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:31.768 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:31.810 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:49:31.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:31.815 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:49:31.916 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:49:31.978 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:32.009 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:49:32.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:32.016 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:49:32.117 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:49:32.173 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:32.213 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:49:32.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:31 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:49:32.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:31 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:49:32.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:31 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:49:32.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:31 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:49:32.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:31 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:49:32.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:31 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:49:32.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:31 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:49:32.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:32.221 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:49:32.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:31 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:49:32.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:31 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:49:32.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:31 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:49:32.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:31 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:49:32.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:31 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:49:32.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:31 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:49:32.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:31 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:49:32.322 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:49:32.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:31 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:49:32.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:31 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:49:32.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:31 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:49:32.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:31 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:49:32.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:31 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:49:32.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:31 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:49:32.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:31 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:49:32.384 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:32.422 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:49:32.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:32.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:49:32.461 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:49:32.531 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:49:32.578 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:32.625 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:49:32.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:32.634 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:49:32.735 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:49:32.785 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:32.815 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:49:32.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:32.822 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:49:32.923 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:49:32.954 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:33.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:32 smithi103 ceph-mon[39629]: from='client.17298 v1:172.21.15.103:0/2275889437' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:49:33.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:32 smithi148 ceph-mon[44520]: from='client.17298 v1:172.21.15.103:0/2275889437' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:49:33.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:32 smithi188 ceph-mon[44101]: from='client.17298 v1:172.21.15.103:0/2275889437' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:49:34.046 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:49:34.046 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:49:34.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:34 smithi103 ceph-mon[39629]: pgmap v1868: 27 pgs: 2 active+recovering, 25 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 829 op/s; 2619/125930 objects misplaced (2.080%); 3.5 MiB/s, 55 objects/s recovering 2024-04-23T16:49:34.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:34 smithi148 ceph-mon[44520]: pgmap v1868: 27 pgs: 2 active+recovering, 25 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 829 op/s; 2619/125930 objects misplaced (2.080%); 3.5 MiB/s, 55 objects/s recovering 2024-04-23T16:49:34.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:34 smithi188 ceph-mon[44101]: pgmap v1868: 27 pgs: 2 active+recovering, 25 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 829 op/s; 2619/125930 objects misplaced (2.080%); 3.5 MiB/s, 55 objects/s recovering 2024-04-23T16:49:34.862 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-23T16:49:35.194 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1867,"stamp":"2024-04-23T16:49:30.925527+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4281467448,"num_objects":65324,"num_object_clones":0,"num_object_copies":130648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2619,"num_objects_unfound":0,"num_objects_dirty":65324,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185692,"num_write_kb":8032385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18108,"num_bytes_recovered":1186997312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162321,"ondisk_log_size":162321,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13192848,"kb_used_data":8237440,"kb_used_omap":242,"kb_used_meta":4954957,"kb_avail":736768368,"statfs":{"total":767960285184,"available":754450808832,"internally_reserved":0,"allocated":8435138560,"data_stored":8431631076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":247821,"internal_metadata":5073876979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-398000128,"num_objects":-6073,"num_object_clones":0,"num_object_copies":-12146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-841,"num_objects_unfound":0,"num_objects_dirty":-6073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6073,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":841,"num_bytes_recovered":55115776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.510586"},"pg_stats":[{"pgid":"8.7","version":"667'11348","reported_seq":11607,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-04-23T16:49:25.943226+0000","last_change":"2024-04-23T16:49:08.424575+0000","last_active":"2024-04-23T16:49:25.943226+0000","last_peered":"2024-04-23T16:49:25.943226+0000","last_clean":"2024-04-23T16:49:25.943226+0000","last_became_active":"2024-04-23T16:46:26.363422+0000","last_became_peered":"2024-04-23T16:46:26.363422+0000","last_unstale":"2024-04-23T16:49:25.943226+0000","last_undegraded":"2024-04-23T16:49:25.943226+0000","last_fullsized":"2024-04-23T16:49:25.943226+0000","mapping_epoch":609,"log_start":"613'1300","ondisk_log_start":"613'1300","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"663'10506","last_scrub_stamp":"2024-04-23T16:49:08.424442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:08.424442+0000","objects_scrubbed":5063,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:20.445332+0000","scrub_duration":991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275382272,"num_objects":4202,"num_object_clones":0,"num_object_copies":8404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11348,"num_write_kb":497600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"667'11756","reported_seq":11882,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-04-23T16:49:27.355431+0000","last_change":"2024-04-23T16:46:26.365698+0000","last_active":"2024-04-23T16:49:27.355431+0000","last_peered":"2024-04-23T16:49:27.355431+0000","last_clean":"2024-04-23T16:49:27.355431+0000","last_became_active":"2024-04-23T16:46:26.365309+0000","last_became_peered":"2024-04-23T16:46:26.365309+0000","last_unstale":"2024-04-23T16:49:27.355431+0000","last_undegraded":"2024-04-23T16:49:27.355431+0000","last_fullsized":"2024-04-23T16:49:27.355431+0000","mapping_epoch":609,"log_start":"613'1700","ondisk_log_start":"613'1700","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266862592,"num_objects":4072,"num_object_clones":0,"num_object_copies":8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11756,"num_write_kb":506496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.5","version":"669'11428","reported_seq":11612,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-04-23T16:49:29.608071+0000","last_change":"2024-04-23T16:49:17.828862+0000","last_active":"2024-04-23T16:49:29.608071+0000","last_peered":"2024-04-23T16:49:29.608071+0000","last_clean":"2024-04-23T16:49:29.608071+0000","last_became_active":"2024-04-23T16:46:26.364780+0000","last_became_peered":"2024-04-23T16:46:26.364780+0000","last_unstale":"2024-04-23T16:49:29.608071+0000","last_undegraded":"2024-04-23T16:49:29.608071+0000","last_fullsized":"2024-04-23T16:49:29.608071+0000","mapping_epoch":609,"log_start":"613'1400","ondisk_log_start":"613'1400","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"665'10858","last_scrub_stamp":"2024-04-23T16:49:17.828773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:17.828773+0000","objects_scrubbed":4612,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:37.125054+0000","scrub_duration":521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263979008,"num_objects":4028,"num_object_clones":0,"num_object_copies":8056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11428,"num_write_kb":494592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"669'11957","reported_seq":20223,"reported_epoch":669,"state":"active+recovering","last_fresh":"2024-04-23T16:49:29.572240+0000","last_change":"2024-04-23T16:47:10.280064+0000","last_active":"2024-04-23T16:49:29.572240+0000","last_peered":"2024-04-23T16:49:29.572240+0000","last_clean":"2024-04-23T16:47:09.049801+0000","last_became_active":"2024-04-23T16:47:10.244722+0000","last_became_peered":"2024-04-23T16:47:10.244722+0000","last_unstale":"2024-04-23T16:49:29.572240+0000","last_undegraded":"2024-04-23T16:49:29.572240+0000","last_fullsized":"2024-04-23T16:49:29.572240+0000","mapping_epoch":623,"log_start":"613'1900","ondisk_log_start":"613'1900","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271908864,"num_objects":4149,"num_object_clones":0,"num_object_copies":8298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1269,"num_objects_unfound":0,"num_objects_dirty":4149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11957,"num_write_kb":515392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4066,"num_bytes_recovered":266469376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"5,9","objects":1269},{"shards":"9,11","objects":2880}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"669'11594","reported_seq":11772,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-04-23T16:49:29.608718+0000","last_change":"2024-04-23T16:49:25.388473+0000","last_active":"2024-04-23T16:49:29.608718+0000","last_peered":"2024-04-23T16:49:29.608718+0000","last_clean":"2024-04-23T16:49:29.608718+0000","last_became_active":"2024-04-23T16:46:26.365000+0000","last_became_peered":"2024-04-23T16:46:26.365000+0000","last_unstale":"2024-04-23T16:49:29.608718+0000","last_undegraded":"2024-04-23T16:49:29.608718+0000","last_fullsized":"2024-04-23T16:49:29.608718+0000","mapping_epoch":609,"log_start":"613'1500","ondisk_log_start":"613'1500","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"667'11355","last_scrub_stamp":"2024-04-23T16:49:25.388270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:25.388270+0000","objects_scrubbed":4236,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:40.837654+0000","scrub_duration":563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261226496,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11594,"num_write_kb":498560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"667'11565","reported_seq":11691,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-04-23T16:49:27.261700+0000","last_change":"2024-04-23T16:46:26.365668+0000","last_active":"2024-04-23T16:49:27.261700+0000","last_peered":"2024-04-23T16:49:27.261700+0000","last_clean":"2024-04-23T16:49:27.261700+0000","last_became_active":"2024-04-23T16:46:26.364349+0000","last_became_peered":"2024-04-23T16:46:26.364349+0000","last_unstale":"2024-04-23T16:49:27.261700+0000","last_undegraded":"2024-04-23T16:49:27.261700+0000","last_fullsized":"2024-04-23T16:49:27.261700+0000","mapping_epoch":609,"log_start":"613'1500","ondisk_log_start":"613'1500","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277020672,"num_objects":4227,"num_object_clones":0,"num_object_copies":8454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11565,"num_write_kb":505344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1660,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-04-23T16:49:29.430735+0000","last_change":"2024-04-23T16:45:33.395526+0000","last_active":"2024-04-23T16:49:29.430735+0000","last_peered":"2024-04-23T16:49:29.430735+0000","last_clean":"2024-04-23T16:49:29.430735+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:49:29.430735+0000","last_undegraded":"2024-04-23T16:49:29.430735+0000","last_fullsized":"2024-04-23T16:49:29.430735+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:33.395406+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:45:33.395406+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.1","version":"667'11518","reported_seq":11776,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-04-23T16:49:25.883992+0000","last_change":"2024-04-23T16:49:22.266220+0000","last_active":"2024-04-23T16:49:25.883992+0000","last_peered":"2024-04-23T16:49:25.883992+0000","last_clean":"2024-04-23T16:49:25.883992+0000","last_became_active":"2024-04-23T16:46:26.362670+0000","last_became_peered":"2024-04-23T16:46:26.362670+0000","last_unstale":"2024-04-23T16:49:25.883992+0000","last_undegraded":"2024-04-23T16:49:25.883992+0000","last_fullsized":"2024-04-23T16:49:25.883992+0000","mapping_epoch":609,"log_start":"613'1500","ondisk_log_start":"613'1500","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"666'11314","last_scrub_stamp":"2024-04-23T16:49:22.266039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:22.266039+0000","objects_scrubbed":4452,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:38.560321+0000","scrub_duration":641,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277741568,"num_objects":4238,"num_object_clones":0,"num_object_copies":8476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11518,"num_write_kb":504192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1538,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-04-23T16:49:27.468633+0000","last_change":"2024-04-23T16:49:25.478623+0000","last_active":"2024-04-23T16:49:27.468633+0000","last_peered":"2024-04-23T16:49:27.468633+0000","last_clean":"2024-04-23T16:49:27.468633+0000","last_became_active":"2024-04-23T16:47:10.310322+0000","last_became_peered":"2024-04-23T16:47:10.310322+0000","last_unstale":"2024-04-23T16:49:27.468633+0000","last_undegraded":"2024-04-23T16:49:27.468633+0000","last_fullsized":"2024-04-23T16:49:27.468633+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:49:25.478506+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:49:25.478506+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:27.187808+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.0","version":"669'11353","reported_seq":21388,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-04-23T16:49:29.557533+0000","last_change":"2024-04-23T16:49:11.451708+0000","last_active":"2024-04-23T16:49:29.557533+0000","last_peered":"2024-04-23T16:49:29.557533+0000","last_clean":"2024-04-23T16:49:29.557533+0000","last_became_active":"2024-04-23T16:47:10.314835+0000","last_became_peered":"2024-04-23T16:47:10.314835+0000","last_unstale":"2024-04-23T16:49:29.557533+0000","last_undegraded":"2024-04-23T16:49:29.557533+0000","last_fullsized":"2024-04-23T16:49:29.557533+0000","mapping_epoch":623,"log_start":"613'1300","ondisk_log_start":"613'1300","created":609,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260767744,"num_objects":3979,"num_object_clones":0,"num_object_copies":7958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11353,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4950,"num_bytes_recovered":324403200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1652,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-04-23T16:49:29.426431+0000","last_change":"2024-04-23T16:48:28.280886+0000","last_active":"2024-04-23T16:49:29.426431+0000","last_peered":"2024-04-23T16:49:29.426431+0000","last_clean":"2024-04-23T16:49:29.426431+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:49:29.426431+0000","last_undegraded":"2024-04-23T16:49:29.426431+0000","last_fullsized":"2024-04-23T16:49:29.426431+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:28.280751+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:28.280751+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:56.638217+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"668'11701","reported_seq":11970,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-04-23T16:49:27.906597+0000","last_change":"2024-04-23T16:49:08.214265+0000","last_active":"2024-04-23T16:49:27.906597+0000","last_peered":"2024-04-23T16:49:27.906597+0000","last_clean":"2024-04-23T16:49:27.906597+0000","last_became_active":"2024-04-23T16:46:26.366359+0000","last_became_peered":"2024-04-23T16:46:26.366359+0000","last_unstale":"2024-04-23T16:49:27.906597+0000","last_undegraded":"2024-04-23T16:49:27.906597+0000","last_fullsized":"2024-04-23T16:49:27.906597+0000","mapping_epoch":609,"log_start":"613'1700","ondisk_log_start":"613'1700","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"663'10692","last_scrub_stamp":"2024-04-23T16:49:08.214052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:08.214052+0000","objects_scrubbed":5183,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:11.250982+0000","scrub_duration":1042,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271908864,"num_objects":4149,"num_object_clones":0,"num_object_copies":8298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11701,"num_write_kb":507200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1667,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-04-23T16:49:29.429594+0000","last_change":"2024-04-23T16:47:08.281155+0000","last_active":"2024-04-23T16:49:29.429594+0000","last_peered":"2024-04-23T16:49:29.429594+0000","last_clean":"2024-04-23T16:49:29.429594+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:49:29.429594+0000","last_undegraded":"2024-04-23T16:49:29.429594+0000","last_fullsized":"2024-04-23T16:49:29.429594+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:47:08.281053+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:47:08.281053+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:32.694415+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"668'11572","reported_seq":11831,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-04-23T16:49:27.905435+0000","last_change":"2024-04-23T16:49:15.382664+0000","last_active":"2024-04-23T16:49:27.905435+0000","last_peered":"2024-04-23T16:49:27.905435+0000","last_clean":"2024-04-23T16:49:27.905435+0000","last_became_active":"2024-04-23T16:46:26.366686+0000","last_became_peered":"2024-04-23T16:46:26.366686+0000","last_unstale":"2024-04-23T16:49:27.905435+0000","last_undegraded":"2024-04-23T16:49:27.905435+0000","last_fullsized":"2024-04-23T16:49:27.905435+0000","mapping_epoch":609,"log_start":"613'1500","ondisk_log_start":"613'1500","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"665'10905","last_scrub_stamp":"2024-04-23T16:49:15.382180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:15.382180+0000","objects_scrubbed":4744,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:32.221200+0000","scrub_duration":1175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265420800,"num_objects":4050,"num_object_clones":0,"num_object_copies":8100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11572,"num_write_kb":499904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1246,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-04-23T16:49:24.422581+0000","last_change":"2024-04-23T16:48:28.293172+0000","last_active":"2024-04-23T16:49:24.422581+0000","last_peered":"2024-04-23T16:49:24.422581+0000","last_clean":"2024-04-23T16:49:24.422581+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:49:24.422581+0000","last_undegraded":"2024-04-23T16:49:24.422581+0000","last_fullsized":"2024-04-23T16:49:24.422581+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:28.293038+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:48:28.293038+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:50.003967+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"669'11540","reported_seq":19280,"reported_epoch":669,"state":"active+recovering","last_fresh":"2024-04-23T16:49:29.576514+0000","last_change":"2024-04-23T16:47:10.314968+0000","last_active":"2024-04-23T16:49:29.576514+0000","last_peered":"2024-04-23T16:49:29.576514+0000","last_clean":"2024-04-23T16:47:09.038944+0000","last_became_active":"2024-04-23T16:47:10.251499+0000","last_became_peered":"2024-04-23T16:47:10.251499+0000","last_unstale":"2024-04-23T16:49:29.576514+0000","last_undegraded":"2024-04-23T16:49:29.576514+0000","last_fullsized":"2024-04-23T16:49:29.576514+0000","mapping_epoch":623,"log_start":"613'1500","ondisk_log_start":"613'1500","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257949696,"num_objects":3936,"num_object_clones":0,"num_object_copies":7872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1350,"num_objects_unfound":0,"num_objects_dirty":3936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11540,"num_write_kb":495232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3803,"num_bytes_recovered":249233408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"5,9","objects":1350},{"shards":"9,11","objects":2586}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1203,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-04-23T16:49:27.468660+0000","last_change":"2024-04-23T16:48:36.291394+0000","last_active":"2024-04-23T16:49:27.468660+0000","last_peered":"2024-04-23T16:49:27.468660+0000","last_clean":"2024-04-23T16:49:27.468660+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:49:27.468660+0000","last_undegraded":"2024-04-23T16:49:27.468660+0000","last_fullsized":"2024-04-23T16:49:27.468660+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:36.291278+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:48:36.291278+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:58.136232+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.f","version":"667'11575","reported_seq":11701,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-04-23T16:49:27.340852+0000","last_change":"2024-04-23T16:46:26.364838+0000","last_active":"2024-04-23T16:49:27.340852+0000","last_peered":"2024-04-23T16:49:27.340852+0000","last_clean":"2024-04-23T16:49:27.340852+0000","last_became_active":"2024-04-23T16:46:26.364570+0000","last_became_peered":"2024-04-23T16:46:26.364570+0000","last_unstale":"2024-04-23T16:49:27.340852+0000","last_undegraded":"2024-04-23T16:49:27.340852+0000","last_fullsized":"2024-04-23T16:49:27.340852+0000","mapping_epoch":609,"log_start":"613'1500","ondisk_log_start":"613'1500","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268763136,"num_objects":4101,"num_object_clones":0,"num_object_copies":8202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11575,"num_write_kb":501632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1668,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-04-23T16:49:29.430562+0000","last_change":"2024-04-23T16:45:30.258925+0000","last_active":"2024-04-23T16:49:29.430562+0000","last_peered":"2024-04-23T16:49:29.430562+0000","last_clean":"2024-04-23T16:49:29.430562+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:49:29.430562+0000","last_undegraded":"2024-04-23T16:49:29.430562+0000","last_fullsized":"2024-04-23T16:49:29.430562+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:30.258804+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:45:30.258804+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.e","version":"669'11551","reported_seq":11681,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-04-23T16:49:29.575803+0000","last_change":"2024-04-23T16:46:26.365834+0000","last_active":"2024-04-23T16:49:29.575803+0000","last_peered":"2024-04-23T16:49:29.575803+0000","last_clean":"2024-04-23T16:49:29.575803+0000","last_became_active":"2024-04-23T16:46:26.365596+0000","last_became_peered":"2024-04-23T16:46:26.365596+0000","last_unstale":"2024-04-23T16:49:29.575803+0000","last_undegraded":"2024-04-23T16:49:29.575803+0000","last_fullsized":"2024-04-23T16:49:29.575803+0000","mapping_epoch":609,"log_start":"613'1500","ondisk_log_start":"613'1500","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:32.694395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255393792,"num_objects":3897,"num_object_clones":0,"num_object_copies":7794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11551,"num_write_kb":494336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1674,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-04-23T16:49:29.533826+0000","last_change":"2024-04-23T16:48:38.816478+0000","last_active":"2024-04-23T16:49:29.533826+0000","last_peered":"2024-04-23T16:49:29.533826+0000","last_clean":"2024-04-23T16:49:29.533826+0000","last_became_active":"2024-04-23T16:47:10.266014+0000","last_became_peered":"2024-04-23T16:47:10.266014+0000","last_unstale":"2024-04-23T16:49:29.533826+0000","last_undegraded":"2024-04-23T16:49:29.533826+0000","last_fullsized":"2024-04-23T16:49:29.533826+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:38.816369+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:38.816369+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:57.044042+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"667'11722","reported_seq":11849,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-04-23T16:49:27.346669+0000","last_change":"2024-04-23T16:46:26.364586+0000","last_active":"2024-04-23T16:49:27.346669+0000","last_peered":"2024-04-23T16:49:27.346669+0000","last_clean":"2024-04-23T16:49:27.346669+0000","last_became_active":"2024-04-23T16:46:26.364302+0000","last_became_peered":"2024-04-23T16:46:26.364302+0000","last_unstale":"2024-04-23T16:49:27.346669+0000","last_undegraded":"2024-04-23T16:49:27.346669+0000","last_fullsized":"2024-04-23T16:49:27.346669+0000","mapping_epoch":609,"log_start":"613'1700","ondisk_log_start":"613'1700","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269549592,"num_objects":4114,"num_object_clones":0,"num_object_copies":8228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4114,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11722,"num_write_kb":506689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1549,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-04-23T16:49:24.443881+0000","last_change":"2024-04-23T16:47:10.037769+0000","last_active":"2024-04-23T16:49:24.443881+0000","last_peered":"2024-04-23T16:49:24.443881+0000","last_clean":"2024-04-23T16:49:24.443881+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:49:24.443881+0000","last_undegraded":"2024-04-23T16:49:24.443881+0000","last_fullsized":"2024-04-23T16:49:24.443881+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:47:10.037665+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:47:10.037665+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"669'11676","reported_seq":11806,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-04-23T16:49:29.581510+0000","last_change":"2024-04-23T16:46:26.364583+0000","last_active":"2024-04-23T16:49:29.581510+0000","last_peered":"2024-04-23T16:49:29.581510+0000","last_clean":"2024-04-23T16:49:29.581510+0000","last_became_active":"2024-04-23T16:46:26.364307+0000","last_became_peered":"2024-04-23T16:46:26.364307+0000","last_unstale":"2024-04-23T16:49:29.581510+0000","last_undegraded":"2024-04-23T16:49:29.581510+0000","last_fullsized":"2024-04-23T16:49:29.581510+0000","mapping_epoch":609,"log_start":"613'1600","ondisk_log_start":"613'1600","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144000,"num_objects":4000,"num_object_clones":0,"num_object_copies":8000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11676,"num_write_kb":501632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1483,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-04-23T16:49:29.533744+0000","last_change":"2024-04-23T16:48:33.921605+0000","last_active":"2024-04-23T16:49:29.533744+0000","last_peered":"2024-04-23T16:49:29.533744+0000","last_clean":"2024-04-23T16:49:29.533744+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:49:29.533744+0000","last_undegraded":"2024-04-23T16:49:29.533744+0000","last_fullsized":"2024-04-23T16:49:29.533744+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:33.921450+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:48:33.921450+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:48.394938+0000","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.9","version":"667'11836","reported_seq":22542,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-04-23T16:49:27.347798+0000","last_change":"2024-04-23T16:49:22.445082+0000","last_active":"2024-04-23T16:49:27.347798+0000","last_peered":"2024-04-23T16:49:27.347798+0000","last_clean":"2024-04-23T16:49:27.347798+0000","last_became_active":"2024-04-23T16:47:10.431200+0000","last_became_peered":"2024-04-23T16:47:10.431200+0000","last_unstale":"2024-04-23T16:49:27.347798+0000","last_undegraded":"2024-04-23T16:49:27.347798+0000","last_fullsized":"2024-04-23T16:49:27.347798+0000","mapping_epoch":623,"log_start":"613'1800","ondisk_log_start":"613'1800","created":609,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274857984,"num_objects":4194,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11836,"num_write_kb":512960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5287,"num_bytes_recovered":346488832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1652,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-04-23T16:49:29.426431+0000","last_change":"2024-04-23T16:48:37.224501+0000","last_active":"2024-04-23T16:49:29.426431+0000","last_peered":"2024-04-23T16:49:29.426431+0000","last_clean":"2024-04-23T16:49:29.426431+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:49:29.426431+0000","last_undegraded":"2024-04-23T16:49:29.426431+0000","last_fullsized":"2024-04-23T16:49:29.426431+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:37.224360+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:48:37.224360+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:50.365513+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4280877080,"num_objects":65322,"num_object_clones":0,"num_object_copies":130644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2619,"num_objects_unfound":0,"num_objects_dirty":65322,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185692,"num_write_kb":8032385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18106,"num_bytes_recovered":1186594816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9671155712,"data_stored":9671147568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160792,"ondisk_log_size":160792,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1679360,"data_stored":1661600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057050,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2831184,"kb_used_data":1965640,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":90913968,"statfs":{"total":95995035648,"available":93095903232,"internally_reserved":0,"allocated":2012815360,"data_stored":2012379299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":886307268},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351393,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613030,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2193756,"kb_used_data":1552976,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":91551396,"statfs":{"total":95995035648,"available":93748629504,"internally_reserved":0,"allocated":1590247424,"data_stored":1589809381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":656013760},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841962,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1212884,"kb_used_data":794380,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92532268,"statfs":{"total":95995035648,"available":94753042432,"internally_reserved":0,"allocated":813445120,"data_stored":813009579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169009,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1125292,"kb_used_data":502688,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92619860,"statfs":{"total":95995035648,"available":94842736640,"internally_reserved":0,"allocated":514752512,"data_stored":514304343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":637532618},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463349,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545692,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199460,"statfs":{"total":95995035648,"available":95436247040,"internally_reserved":0,"allocated":868352,"data_stored":430309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137489,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1662416,"kb_used_data":1359620,"kb_used_omap":231,"kb_used_meta":302552,"kb_avail":92082736,"statfs":{"total":95995035648,"available":94292721664,"internally_reserved":0,"allocated":1392250880,"data_stored":1391818891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":309814160},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738835,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1457560,"kb_used_data":776656,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":92287592,"statfs":{"total":95995035648,"available":94502494208,"internally_reserved":0,"allocated":795295744,"data_stored":794857701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269505,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2658584756255,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921213,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2164064,"kb_used_data":1284632,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":91581088,"statfs":{"total":95995035648,"available":93779034112,"internally_reserved":0,"allocated":1315463168,"data_stored":1315021573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757688,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":783351808,"data_stored":783351808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1384775680,"data_stored":1384775680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1306066944,"data_stored":1306066944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1313144832,"data_stored":1313144832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":513544192,"data_stored":513540120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":805961728,"data_stored":805961728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1577320448,"data_stored":1577320448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1986990080,"data_stored":1986986008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:49:35.196 INFO:tasks.cephadm.ceph_manager.ceph:PG 8.4 is not active+clean 2024-04-23T16:49:35.196 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '8.4', 'version': "669'11957", 'reported_seq': 20223, 'reported_epoch': 669, 'state': 'active+recovering', 'last_fresh': '2024-04-23T16:49:29.572240+0000', 'last_change': '2024-04-23T16:47:10.280064+0000', 'last_active': '2024-04-23T16:49:29.572240+0000', 'last_peered': '2024-04-23T16:49:29.572240+0000', 'last_clean': '2024-04-23T16:47:09.049801+0000', 'last_became_active': '2024-04-23T16:47:10.244722+0000', 'last_became_peered': '2024-04-23T16:47:10.244722+0000', 'last_unstale': '2024-04-23T16:49:29.572240+0000', 'last_undegraded': '2024-04-23T16:49:29.572240+0000', 'last_fullsized': '2024-04-23T16:49:29.572240+0000', 'mapping_epoch': 623, 'log_start': "613'1900", 'ondisk_log_start': "613'1900", 'created': 609, 'last_epoch_clean': 610, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T16:46:25.347995+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:46:25.347995+0000', 'last_clean_scrub_stamp': '2024-04-23T16:46:25.347995+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 271908864, 'num_objects': 4149, 'num_object_clones': 0, 'num_object_copies': 8298, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1269, 'num_objects_unfound': 0, 'num_objects_dirty': 4149, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11957, 'num_write_kb': 515392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4066, 'num_bytes_recovered': 266469376, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9, 11], 'acting': [9, 11], 'avail_no_missing': ['9', '5'], 'object_location_counts': [{'shards': '5,9', 'objects': 1269}, {'shards': '9,11', 'objects': 2880}], 'blocked_by': [], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2024-04-23T16:49:35.196 INFO:tasks.cephadm.ceph_manager.ceph:PG 8.8 is not active+clean 2024-04-23T16:49:35.196 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '8.8', 'version': "669'11540", 'reported_seq': 19280, 'reported_epoch': 669, 'state': 'active+recovering', 'last_fresh': '2024-04-23T16:49:29.576514+0000', 'last_change': '2024-04-23T16:47:10.314968+0000', 'last_active': '2024-04-23T16:49:29.576514+0000', 'last_peered': '2024-04-23T16:49:29.576514+0000', 'last_clean': '2024-04-23T16:47:09.038944+0000', 'last_became_active': '2024-04-23T16:47:10.251499+0000', 'last_became_peered': '2024-04-23T16:47:10.251499+0000', 'last_unstale': '2024-04-23T16:49:29.576514+0000', 'last_undegraded': '2024-04-23T16:49:29.576514+0000', 'last_fullsized': '2024-04-23T16:49:29.576514+0000', 'mapping_epoch': 623, 'log_start': "613'1500", 'ondisk_log_start': "613'1500", 'created': 609, 'last_epoch_clean': 610, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T16:46:25.347995+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:46:25.347995+0000', 'last_clean_scrub_stamp': '2024-04-23T16:46:25.347995+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 257949696, 'num_objects': 3936, 'num_object_clones': 0, 'num_object_copies': 7872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1350, 'num_objects_unfound': 0, 'num_objects_dirty': 3936, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11540, 'num_write_kb': 495232, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3803, 'num_bytes_recovered': 249233408, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9, 11], 'acting': [9, 11], 'avail_no_missing': ['9', '5'], 'object_location_counts': [{'shards': '5,9', 'objects': 1350}, {'shards': '9,11', 'objects': 2586}], 'blocked_by': [], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2024-04-23T16:49:35.197 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-23T16:49:35.204 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:35.207 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:49:35.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:35.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:49:35.249 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:35.312 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:49:35.338 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:35.359 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:49:35.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:35.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:49:35.428 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:35 smithi103 ceph-mon[39629]: osdmap e670: 12 total, 12 up, 8 in 2024-04-23T16:49:35.429 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:35 smithi103 ceph-mon[39629]: pgmap v1870: 27 pgs: 2 active+recovering, 25 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 882 op/s; 2126/122174 objects misplaced (1.740%); 7.8 MiB/s, 124 objects/s recovering 2024-04-23T16:49:35.463 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:49:35.489 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:35.509 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:49:35.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:35.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:49:35.613 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:49:35.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:35 smithi148 ceph-mon[44520]: osdmap e670: 12 total, 12 up, 8 in 2024-04-23T16:49:35.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:35 smithi148 ceph-mon[44520]: pgmap v1870: 27 pgs: 2 active+recovering, 25 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 882 op/s; 2126/122174 objects misplaced (1.740%); 7.8 MiB/s, 124 objects/s recovering 2024-04-23T16:49:35.642 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:35.659 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:49:35.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:35.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:49:35.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:35 smithi188 ceph-mon[44101]: osdmap e670: 12 total, 12 up, 8 in 2024-04-23T16:49:35.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:35 smithi188 ceph-mon[44101]: pgmap v1870: 27 pgs: 2 active+recovering, 25 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 882 op/s; 2126/122174 objects misplaced (1.740%); 7.8 MiB/s, 124 objects/s recovering 2024-04-23T16:49:35.764 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:49:35.825 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:35.862 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:49:35.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:35.868 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:49:35.969 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:49:35.997 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:36.019 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:49:36.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:36.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:49:36.124 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:49:36.193 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:36.226 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:49:36.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:36.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:49:36.332 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:49:36.359 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:36.379 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:49:36.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:36.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:49:36.484 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:49:36.511 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:36.531 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:49:36.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:36.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:49:36.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3020592087' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:49:36.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:36 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:36.635 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:49:36.666 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:36.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3020592087' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:49:36.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:36 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:36.814 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:49:36.816 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3020592087' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:49:36.817 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:36 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:36.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:36.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:49:36.919 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:49:36.952 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:37.240 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:49:37.241 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:49:37.241 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:49:37.241 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:49:37.241 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:49:37.241 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:49:37.241 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:49:37.241 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:49:37.241 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:49:37.241 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:49:37.241 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:49:37.241 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:49:37.242 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:49:37.242 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:49:37.242 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:49:37.242 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:49:37.242 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:49:37.243 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:49:37.243 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:49:37.243 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:49:37.243 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:49:37.243 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:49:37.243 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:49:37.243 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:49:37.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:37 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:37 smithi188 ceph-mon[44101]: pgmap v1871: 27 pgs: 2 active+recovering, 25 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 821 op/s; 2126/121208 objects misplaced (1.754%); 7.3 MiB/s, 116 objects/s recovering 2024-04-23T16:49:37.728 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.728 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.728 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.728 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.728 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.728 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.729 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.729 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.729 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.729 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.729 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:37 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.729 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:37 smithi103 ceph-mon[39629]: pgmap v1871: 27 pgs: 2 active+recovering, 25 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 821 op/s; 2126/121208 objects misplaced (1.754%); 7.3 MiB/s, 116 objects/s recovering 2024-04-23T16:49:37.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:37 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:37 smithi148 ceph-mon[44520]: pgmap v1871: 27 pgs: 2 active+recovering, 25 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 821 op/s; 2126/121208 objects misplaced (1.754%); 7.3 MiB/s, 116 objects/s recovering 2024-04-23T16:49:38.210 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:49:38.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:38.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:49:38.315 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:49:38.344 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:38.374 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:49:38.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:38.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:49:38.482 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:49:38.509 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:38.530 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:49:38.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:38.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:49:38.634 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:49:38.660 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:38.680 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:49:38.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:38.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:49:38.784 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:49:38.817 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:39.267 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:49:39.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:39.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:49:39.317 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:49:39.317 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:49:39.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:49:39.317 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:49:39.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.317 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:291492 6.4 6:2161f0f3:::benchmark_data_smithi188_130221_object137031:head [delete] snapc 0=[] ondisk+write+known_if_redirected e532)", 2024-04-23T16:49:39.317 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:41:19.288911+0000", 2024-04-23T16:49:39.317 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 500.02504807399998, 2024-04-23T16:49:39.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.10719031499999999, 2024-04-23T16:49:39.317 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.317 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.317 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.318 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:49:39.318 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:49:39.318 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:49:39.318 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 291492 2024-04-23T16:49:39.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.318 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.318 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.318 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.288911+0000", 2024-04-23T16:49:39.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.318 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.318 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.288911+0000", 2024-04-23T16:49:39.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.288912+0000", 2024-04-23T16:49:39.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.891e-06 2024-04-23T16:49:39.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.288918+0000", 2024-04-23T16:49:39.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6359999999999999e-06 2024-04-23T16:49:39.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.288928+0000", 2024-04-23T16:49:39.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.431e-06 2024-04-23T16:49:39.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.288934+0000", 2024-04-23T16:49:39.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.489e-06 2024-04-23T16:49:39.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.288985+0000", 2024-04-23T16:49:39.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0929e-05 2024-04-23T16:49:39.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.289064+0000", 2024-04-23T16:49:39.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9529000000000004e-05 2024-04-23T16:49:39.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 5", 2024-04-23T16:49:39.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.289129+0000", 2024-04-23T16:49:39.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4898000000000001e-05 2024-04-23T16:49:39.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:49:39.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.292319+0000", 2024-04-23T16:49:39.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0031894779999999999 2024-04-23T16:49:39.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:49:39.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.396033+0000", 2024-04-23T16:49:39.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.10371372299999999 2024-04-23T16:49:39.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.396068+0000", 2024-04-23T16:49:39.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4986e-05 2024-04-23T16:49:39.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.396101+0000", 2024-04-23T16:49:39.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3324999999999997e-05 2024-04-23T16:49:39.323 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.323 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.323 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.323 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16308.0:291496 6.4 6:2e9d1974:::benchmark_data_smithi188_130221_object137035:head [delete] snapc 0=[] ondisk+write+known_if_redirected e532)", 2024-04-23T16:49:39.323 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:41:19.293393+0000", 2024-04-23T16:49:39.324 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 500.02056535299999, 2024-04-23T16:49:39.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.102865559, 2024-04-23T16:49:39.324 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.324 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.324 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.324 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:49:39.324 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16308", 2024-04-23T16:49:39.324 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/444587747", 2024-04-23T16:49:39.324 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 291496 2024-04-23T16:49:39.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.324 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.293393+0000", 2024-04-23T16:49:39.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.293393+0000", 2024-04-23T16:49:39.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.293395+0000", 2024-04-23T16:49:39.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.392e-06 2024-04-23T16:49:39.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.293399+0000", 2024-04-23T16:49:39.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9169999999999999e-06 2024-04-23T16:49:39.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.293406+0000", 2024-04-23T16:49:39.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9809999999999997e-06 2024-04-23T16:49:39.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.293410+0000", 2024-04-23T16:49:39.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7249999999999997e-06 2024-04-23T16:49:39.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.293470+0000", 2024-04-23T16:49:39.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9361000000000003e-05 2024-04-23T16:49:39.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.389392+0000", 2024-04-23T16:49:39.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.095922293000000006 2024-04-23T16:49:39.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 5", 2024-04-23T16:49:39.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.389507+0000", 2024-04-23T16:49:39.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011497699999999999 2024-04-23T16:49:39.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:49:39.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.396142+0000", 2024-04-23T16:49:39.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066347419999999999 2024-04-23T16:49:39.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:49:39.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.396203+0000", 2024-04-23T16:49:39.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1032000000000001e-05 2024-04-23T16:49:39.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.396230+0000", 2024-04-23T16:49:39.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6826e-05 2024-04-23T16:49:39.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.329 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.396259+0000", 2024-04-23T16:49:39.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9312999999999999e-05 2024-04-23T16:49:39.329 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.329 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.329 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.329 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.16308.0:291497 6.d e532/510 6:bcda3e19:::benchmark_data_smithi188_130221_object137036:head v 532'17913, mlcod=532'17911)", 2024-04-23T16:49:39.329 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:41:19.296543+0000", 2024-04-23T16:49:39.330 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 500.01741549799999, 2024-04-23T16:49:39.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.1058099, 2024-04-23T16:49:39.330 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.330 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.330 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.330 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.296543+0000", 2024-04-23T16:49:39.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.296543+0000", 2024-04-23T16:49:39.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.296546+0000", 2024-04-23T16:49:39.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6929999999999999e-06 2024-04-23T16:49:39.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.296559+0000", 2024-04-23T16:49:39.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3107e-05 2024-04-23T16:49:39.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.389339+0000", 2024-04-23T16:49:39.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.092780043000000006 2024-04-23T16:49:39.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.389351+0000", 2024-04-23T16:49:39.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1911e-05 2024-04-23T16:49:39.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.389406+0000", 2024-04-23T16:49:39.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4818000000000003e-05 2024-04-23T16:49:39.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.389422+0000", 2024-04-23T16:49:39.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5993000000000001e-05 2024-04-23T16:49:39.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.402303+0000", 2024-04-23T16:49:39.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012881737000000001 2024-04-23T16:49:39.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:41:19.402353+0000", 2024-04-23T16:49:39.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9598000000000002e-05 2024-04-23T16:49:39.333 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.333 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.334 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.334 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.26584.0:92576 7.b 7:dd33b1c1:::benchmark_data_smithi188_141483_object92575:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e565)", 2024-04-23T16:49:39.334 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:43:10.196932+0000", 2024-04-23T16:49:39.334 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 389.11702666299999, 2024-04-23T16:49:39.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13238688000000001, 2024-04-23T16:49:39.334 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.334 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.334 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.334 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:49:39.334 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.26584", 2024-04-23T16:49:39.334 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:49:39.334 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 92576 2024-04-23T16:49:39.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.335 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.196932+0000", 2024-04-23T16:49:39.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.196932+0000", 2024-04-23T16:49:39.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.196935+0000", 2024-04-23T16:49:39.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.762e-06 2024-04-23T16:49:39.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.196961+0000", 2024-04-23T16:49:39.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6517999999999999e-05 2024-04-23T16:49:39.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.196975+0000", 2024-04-23T16:49:39.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3828999999999999e-05 2024-04-23T16:49:39.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.196982+0000", 2024-04-23T16:49:39.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0520000000000004e-06 2024-04-23T16:49:39.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.197032+0000", 2024-04-23T16:49:39.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9877999999999999e-05 2024-04-23T16:49:39.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.197125+0000", 2024-04-23T16:49:39.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2520000000000002e-05 2024-04-23T16:49:39.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 5", 2024-04-23T16:49:39.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.197221+0000", 2024-04-23T16:49:39.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6673000000000001e-05 2024-04-23T16:49:39.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:49:39.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.199430+0000", 2024-04-23T16:49:39.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002208935 2024-04-23T16:49:39.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:49:39.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.329253+0000", 2024-04-23T16:49:39.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.12982253399999999 2024-04-23T16:49:39.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.329289+0000", 2024-04-23T16:49:39.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6365000000000002e-05 2024-04-23T16:49:39.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.329319+0000", 2024-04-23T16:49:39.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9813999999999999e-05 2024-04-23T16:49:39.339 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.339 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.339 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.339 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.26584.0:92612 7.b 7:d78996e3:::benchmark_data_smithi188_141483_object92611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e565)", 2024-04-23T16:49:39.340 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:43:10.239851+0000", 2024-04-23T16:49:39.340 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 389.07410730499998, 2024-04-23T16:49:39.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.089628843999999999, 2024-04-23T16:49:39.340 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.340 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.340 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.340 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:49:39.340 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.26584", 2024-04-23T16:49:39.340 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:49:39.340 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 92612 2024-04-23T16:49:39.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.340 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.239851+0000", 2024-04-23T16:49:39.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.239851+0000", 2024-04-23T16:49:39.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.239854+0000", 2024-04-23T16:49:39.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.401e-06 2024-04-23T16:49:39.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.239894+0000", 2024-04-23T16:49:39.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0568999999999997e-05 2024-04-23T16:49:39.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.239910+0000", 2024-04-23T16:49:39.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6178e-05 2024-04-23T16:49:39.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.239921+0000", 2024-04-23T16:49:39.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0524e-05 2024-04-23T16:49:39.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.239999+0000", 2024-04-23T16:49:39.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8449999999999996e-05 2024-04-23T16:49:39.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.240133+0000", 2024-04-23T16:49:39.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000133829 2024-04-23T16:49:39.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 5", 2024-04-23T16:49:39.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.240245+0000", 2024-04-23T16:49:39.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112074 2024-04-23T16:49:39.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:49:39.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.242495+0000", 2024-04-23T16:49:39.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002249764 2024-04-23T16:49:39.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:49:39.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.329372+0000", 2024-04-23T16:49:39.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.086877264999999995 2024-04-23T16:49:39.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.329429+0000", 2024-04-23T16:49:39.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6399e-05 2024-04-23T16:49:39.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:43:10.329480+0000", 2024-04-23T16:49:39.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1391000000000003e-05 2024-04-23T16:49:39.345 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.345 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.345 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.345 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:131081 8.7 8:efb7ae5f:::benchmark_data_smithi188_155546_object5572:head [delete] snapc 0=[] ondisk+write+known_if_redirected e648)", 2024-04-23T16:49:39.345 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:48:14.801667+0000", 2024-04-23T16:49:39.345 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 84.51229189, 2024-04-23T16:49:39.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.087288621999999996, 2024-04-23T16:49:39.345 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.345 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.346 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.346 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:49:39.346 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:49:39.346 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:49:39.346 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 131081 2024-04-23T16:49:39.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.346 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:14.801667+0000", 2024-04-23T16:49:39.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:14.801667+0000", 2024-04-23T16:49:39.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:14.801670+0000", 2024-04-23T16:49:39.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5250000000000001e-06 2024-04-23T16:49:39.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:14.801683+0000", 2024-04-23T16:49:39.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.225e-05 2024-04-23T16:49:39.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:14.801703+0000", 2024-04-23T16:49:39.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0000000000000002e-05 2024-04-23T16:49:39.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:14.801715+0000", 2024-04-23T16:49:39.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2745000000000001e-05 2024-04-23T16:49:39.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:14.801781+0000", 2024-04-23T16:49:39.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5648000000000006e-05 2024-04-23T16:49:39.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:14.801881+0000", 2024-04-23T16:49:39.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9802000000000004e-05 2024-04-23T16:49:39.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 11", 2024-04-23T16:49:39.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:14.801959+0000", 2024-04-23T16:49:39.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8595000000000002e-05 2024-04-23T16:49:39.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:49:39.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:14.805200+0000", 2024-04-23T16:49:39.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0032405870000000001 2024-04-23T16:49:39.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:49:39.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:14.888826+0000", 2024-04-23T16:49:39.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.083626026000000006 2024-04-23T16:49:39.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:14.888897+0000", 2024-04-23T16:49:39.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0630000000000006e-05 2024-04-23T16:49:39.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:14.888955+0000", 2024-04-23T16:49:39.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8814000000000003e-05 2024-04-23T16:49:39.351 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.351 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.351 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.351 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:132599 8.7 8:e5fe2c14:::benchmark_data_smithi188_155546_object7090:head [delete] snapc 0=[] ondisk+write+known_if_redirected e649)", 2024-04-23T16:49:39.351 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:48:16.885143+0000", 2024-04-23T16:49:39.351 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.428816005000002, 2024-04-23T16:49:39.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.104241074, 2024-04-23T16:49:39.351 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.351 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.351 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.351 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:49:39.351 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:49:39.351 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:49:39.352 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 132599 2024-04-23T16:49:39.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.352 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.885143+0000", 2024-04-23T16:49:39.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.885143+0000", 2024-04-23T16:49:39.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.885144+0000", 2024-04-23T16:49:39.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2160000000000001e-06 2024-04-23T16:49:39.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.885148+0000", 2024-04-23T16:49:39.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4100000000000001e-06 2024-04-23T16:49:39.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.885156+0000", 2024-04-23T16:49:39.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.323e-06 2024-04-23T16:49:39.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.885162+0000", 2024-04-23T16:49:39.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8560000000000001e-06 2024-04-23T16:49:39.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.885213+0000", 2024-04-23T16:49:39.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0680999999999998e-05 2024-04-23T16:49:39.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.885328+0000", 2024-04-23T16:49:39.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011523099999999999 2024-04-23T16:49:39.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 11", 2024-04-23T16:49:39.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.885431+0000", 2024-04-23T16:49:39.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000102166 2024-04-23T16:49:39.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:49:39.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.888651+0000", 2024-04-23T16:49:39.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0032201590000000002 2024-04-23T16:49:39.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:49:39.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.989322+0000", 2024-04-23T16:49:39.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.10067145800000001 2024-04-23T16:49:39.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.989352+0000", 2024-04-23T16:49:39.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9747000000000002e-05 2024-04-23T16:49:39.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.989384+0000", 2024-04-23T16:49:39.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1826999999999999e-05 2024-04-23T16:49:39.357 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.357 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.357 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.357 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:132606 8.7 8:e0395511:::benchmark_data_smithi188_155546_object7097:head [delete] snapc 0=[] ondisk+write+known_if_redirected e649)", 2024-04-23T16:49:39.357 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:48:16.896254+0000", 2024-04-23T16:49:39.357 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.417704254, 2024-04-23T16:49:39.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.096002271, 2024-04-23T16:49:39.357 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.357 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.357 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.357 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:49:39.357 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:49:39.358 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:49:39.358 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 132606 2024-04-23T16:49:39.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.358 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.896254+0000", 2024-04-23T16:49:39.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.896254+0000", 2024-04-23T16:49:39.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.896255+0000", 2024-04-23T16:49:39.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8199999999999999e-07 2024-04-23T16:49:39.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.896257+0000", 2024-04-23T16:49:39.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3190000000000002e-06 2024-04-23T16:49:39.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.896264+0000", 2024-04-23T16:49:39.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5309999999999998e-06 2024-04-23T16:49:39.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.896270+0000", 2024-04-23T16:49:39.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2999999999999998e-06 2024-04-23T16:49:39.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.896322+0000", 2024-04-23T16:49:39.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1956999999999997e-05 2024-04-23T16:49:39.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.896437+0000", 2024-04-23T16:49:39.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114636 2024-04-23T16:49:39.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 11", 2024-04-23T16:49:39.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.896551+0000", 2024-04-23T16:49:39.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011410599999999999 2024-04-23T16:49:39.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:49:39.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.899667+0000", 2024-04-23T16:49:39.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003116237 2024-04-23T16:49:39.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:49:39.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.992203+0000", 2024-04-23T16:49:39.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.092536068999999999 2024-04-23T16:49:39.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.992233+0000", 2024-04-23T16:49:39.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.923e-05 2024-04-23T16:49:39.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:16.992257+0000", 2024-04-23T16:49:39.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4104000000000002e-05 2024-04-23T16:49:39.362 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.362 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.362 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.363 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:133090 8.1 8:81d494c0:::benchmark_data_smithi188_155546_object7581:head [delete] snapc 0=[] ondisk+write+known_if_redirected e649)", 2024-04-23T16:49:39.363 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:48:17.585443+0000", 2024-04-23T16:49:39.363 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.728515256999998, 2024-04-23T16:49:39.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.085201162999999996, 2024-04-23T16:49:39.363 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.363 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.363 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.363 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:49:39.363 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:49:39.363 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:49:39.363 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 133090 2024-04-23T16:49:39.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.364 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.585443+0000", 2024-04-23T16:49:39.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.585443+0000", 2024-04-23T16:49:39.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.585444+0000", 2024-04-23T16:49:39.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9599999999999999e-07 2024-04-23T16:49:39.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.585446+0000", 2024-04-23T16:49:39.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9659999999999999e-06 2024-04-23T16:49:39.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.585450+0000", 2024-04-23T16:49:39.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0489999999999997e-06 2024-04-23T16:49:39.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.585458+0000", 2024-04-23T16:49:39.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7789999999999996e-06 2024-04-23T16:49:39.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.589468+0000", 2024-04-23T16:49:39.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0040102530000000001 2024-04-23T16:49:39.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.589561+0000", 2024-04-23T16:49:39.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3579000000000006e-05 2024-04-23T16:49:39.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 9", 2024-04-23T16:49:39.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.589639+0000", 2024-04-23T16:49:39.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7317000000000005e-05 2024-04-23T16:49:39.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:49:39.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.596184+0000", 2024-04-23T16:49:39.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0065453500000000001 2024-04-23T16:49:39.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:49:39.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.670558+0000", 2024-04-23T16:49:39.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.074374006000000006 2024-04-23T16:49:39.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.670606+0000", 2024-04-23T16:49:39.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7573e-05 2024-04-23T16:49:39.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.670645+0000", 2024-04-23T16:49:39.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8794999999999997e-05 2024-04-23T16:49:39.368 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.368 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.368 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.368 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:133296 8.1 8:88ddb81b:::benchmark_data_smithi188_155546_object7787:head [delete] snapc 0=[] ondisk+write+known_if_redirected e649)", 2024-04-23T16:49:39.368 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:48:17.865495+0000", 2024-04-23T16:49:39.368 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.448464053999999, 2024-04-23T16:49:39.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.10388114800000001, 2024-04-23T16:49:39.369 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.369 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.369 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.369 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:49:39.369 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:49:39.369 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:49:39.369 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 133296 2024-04-23T16:49:39.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.369 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.865495+0000", 2024-04-23T16:49:39.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.865495+0000", 2024-04-23T16:49:39.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.865497+0000", 2024-04-23T16:49:39.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0770000000000002e-06 2024-04-23T16:49:39.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.865503+0000", 2024-04-23T16:49:39.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1260000000000003e-06 2024-04-23T16:49:39.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.865515+0000", 2024-04-23T16:49:39.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2556e-05 2024-04-23T16:49:39.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.865522+0000", 2024-04-23T16:49:39.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1990000000000003e-06 2024-04-23T16:49:39.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.865572+0000", 2024-04-23T16:49:39.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0529999999999999e-05 2024-04-23T16:49:39.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.865655+0000", 2024-04-23T16:49:39.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3202999999999996e-05 2024-04-23T16:49:39.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 9", 2024-04-23T16:49:39.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.865728+0000", 2024-04-23T16:49:39.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2414000000000002e-05 2024-04-23T16:49:39.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:49:39.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.868738+0000", 2024-04-23T16:49:39.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030099549999999999 2024-04-23T16:49:39.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:49:39.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.969303+0000", 2024-04-23T16:49:39.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.100564919 2024-04-23T16:49:39.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.969339+0000", 2024-04-23T16:49:39.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6109000000000003e-05 2024-04-23T16:49:39.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.969376+0000", 2024-04-23T16:49:39.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7060000000000001e-05 2024-04-23T16:49:39.374 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.374 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.374 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.374 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:133302 8.1 8:86764668:::benchmark_data_smithi188_155546_object7793:head [delete] snapc 0=[] ondisk+write+known_if_redirected e649)", 2024-04-23T16:49:39.374 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:48:17.871328+0000", 2024-04-23T16:49:39.374 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.442630346000001, 2024-04-23T16:49:39.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.10698818, 2024-04-23T16:49:39.374 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.374 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.374 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.375 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:49:39.375 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:49:39.375 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:49:39.375 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 133302 2024-04-23T16:49:39.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.375 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.871328+0000", 2024-04-23T16:49:39.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.871328+0000", 2024-04-23T16:49:39.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.871330+0000", 2024-04-23T16:49:39.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3510000000000001e-06 2024-04-23T16:49:39.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.871333+0000", 2024-04-23T16:49:39.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8360000000000002e-06 2024-04-23T16:49:39.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.871343+0000", 2024-04-23T16:49:39.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2539999999999998e-06 2024-04-23T16:49:39.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.871348+0000", 2024-04-23T16:49:39.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3120000000000001e-06 2024-04-23T16:49:39.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.871399+0000", 2024-04-23T16:49:39.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0717000000000002e-05 2024-04-23T16:49:39.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.871521+0000", 2024-04-23T16:49:39.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012215699999999999 2024-04-23T16:49:39.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 9", 2024-04-23T16:49:39.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.871629+0000", 2024-04-23T16:49:39.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010780700000000001 2024-04-23T16:49:39.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:49:39.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.874450+0000", 2024-04-23T16:49:39.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028208119999999998 2024-04-23T16:49:39.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:49:39.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.978221+0000", 2024-04-23T16:49:39.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.10377152000000001 2024-04-23T16:49:39.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.978275+0000", 2024-04-23T16:49:39.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4398000000000003e-05 2024-04-23T16:49:39.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.978316+0000", 2024-04-23T16:49:39.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1016000000000002e-05 2024-04-23T16:49:39.379 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.379 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.380 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.380 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:133314 8.1 8:8f5f23a8:::benchmark_data_smithi188_155546_object7805:head [delete] snapc 0=[] ondisk+write+known_if_redirected e649)", 2024-04-23T16:49:39.380 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:48:17.881718+0000", 2024-04-23T16:49:39.380 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.432241013999999, 2024-04-23T16:49:39.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.096734086999999996, 2024-04-23T16:49:39.380 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.380 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.380 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.380 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:49:39.380 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:49:39.380 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:49:39.380 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 133314 2024-04-23T16:49:39.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.381 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.881718+0000", 2024-04-23T16:49:39.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.881718+0000", 2024-04-23T16:49:39.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.881720+0000", 2024-04-23T16:49:39.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9810000000000002e-06 2024-04-23T16:49:39.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.881726+0000", 2024-04-23T16:49:39.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9490000000000001e-06 2024-04-23T16:49:39.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.881738+0000", 2024-04-23T16:49:39.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2602000000000001e-05 2024-04-23T16:49:39.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.881746+0000", 2024-04-23T16:49:39.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5159999999999997e-06 2024-04-23T16:49:39.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.881800+0000", 2024-04-23T16:49:39.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.414e-05 2024-04-23T16:49:39.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.881919+0000", 2024-04-23T16:49:39.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011904 2024-04-23T16:49:39.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 9", 2024-04-23T16:49:39.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.882032+0000", 2024-04-23T16:49:39.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011269600000000001 2024-04-23T16:49:39.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:49:39.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.884922+0000", 2024-04-23T16:49:39.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002890685 2024-04-23T16:49:39.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:49:39.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.978360+0000", 2024-04-23T16:49:39.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.093438189000000005 2024-04-23T16:49:39.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.978413+0000", 2024-04-23T16:49:39.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2899000000000003e-05 2024-04-23T16:49:39.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:17.978452+0000", 2024-04-23T16:49:39.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8389999999999997e-05 2024-04-23T16:49:39.385 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.385 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.385 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.385 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:153891 8.7 8:eceebe53:::benchmark_data_smithi188_155546_object28382:head [delete] snapc 0=[] ondisk+write+known_if_redirected e658)", 2024-04-23T16:49:39.385 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:48:46.430237+0000", 2024-04-23T16:49:39.386 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 52.883721999999999, 2024-04-23T16:49:39.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.086335560000000006, 2024-04-23T16:49:39.386 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.386 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.386 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.386 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:49:39.386 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:49:39.386 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:49:39.386 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 153891 2024-04-23T16:49:39.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.386 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.430237+0000", 2024-04-23T16:49:39.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.430237+0000", 2024-04-23T16:49:39.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.430238+0000", 2024-04-23T16:49:39.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.514e-06 2024-04-23T16:49:39.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.430243+0000", 2024-04-23T16:49:39.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6879999999999998e-06 2024-04-23T16:49:39.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.430251+0000", 2024-04-23T16:49:39.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6370000000000005e-06 2024-04-23T16:49:39.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.430259+0000", 2024-04-23T16:49:39.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7649999999999993e-06 2024-04-23T16:49:39.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.430316+0000", 2024-04-23T16:49:39.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6396e-05 2024-04-23T16:49:39.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.430449+0000", 2024-04-23T16:49:39.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013336700000000001 2024-04-23T16:49:39.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 11", 2024-04-23T16:49:39.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.430580+0000", 2024-04-23T16:49:39.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013117599999999999 2024-04-23T16:49:39.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:49:39.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.437189+0000", 2024-04-23T16:49:39.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066086870000000002 2024-04-23T16:49:39.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:49:39.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.516500+0000", 2024-04-23T16:49:39.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.079311066999999999 2024-04-23T16:49:39.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.516534+0000", 2024-04-23T16:49:39.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3946000000000003e-05 2024-04-23T16:49:39.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.516572+0000", 2024-04-23T16:49:39.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8316999999999999e-05 2024-04-23T16:49:39.391 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.391 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.391 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.391 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:153894 8.7 8:ecb36c79:::benchmark_data_smithi188_155546_object28385:head [delete] snapc 0=[] ondisk+write+known_if_redirected e658)", 2024-04-23T16:49:39.391 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:48:46.431238+0000", 2024-04-23T16:49:39.391 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 52.882720325000001, 2024-04-23T16:49:39.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.085421929999999993, 2024-04-23T16:49:39.391 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.391 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.391 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:49:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:49:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:49:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 153894 2024-04-23T16:49:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.431238+0000", 2024-04-23T16:49:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.431238+0000", 2024-04-23T16:49:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.431239+0000", 2024-04-23T16:49:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6599999999999994e-07 2024-04-23T16:49:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.431243+0000", 2024-04-23T16:49:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1219999999999997e-06 2024-04-23T16:49:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.431252+0000", 2024-04-23T16:49:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6840000000000002e-06 2024-04-23T16:49:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.431258+0000", 2024-04-23T16:49:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.507e-06 2024-04-23T16:49:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.434222+0000", 2024-04-23T16:49:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0029642179999999998 2024-04-23T16:49:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.434300+0000", 2024-04-23T16:49:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8020000000000002e-05 2024-04-23T16:49:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 11", 2024-04-23T16:49:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.434363+0000", 2024-04-23T16:49:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2941000000000006e-05 2024-04-23T16:49:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:49:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.437319+0000", 2024-04-23T16:49:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0029563160000000001 2024-04-23T16:49:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:49:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.516605+0000", 2024-04-23T16:49:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.079286014000000002 2024-04-23T16:49:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.516633+0000", 2024-04-23T16:49:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7895999999999999e-05 2024-04-23T16:49:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:46.516660+0000", 2024-04-23T16:49:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7246e-05 2024-04-23T16:49:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:157007 8.7 8:e8f57ef5:::benchmark_data_smithi188_155546_object31498:head [delete] snapc 0=[] ondisk+write+known_if_redirected e660)", 2024-04-23T16:49:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:48:50.711703+0000", 2024-04-23T16:49:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.602255382999999, 2024-04-23T16:49:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.081529779999999996, 2024-04-23T16:49:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:49:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:49:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:49:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 157007 2024-04-23T16:49:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:50.711703+0000", 2024-04-23T16:49:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:50.711703+0000", 2024-04-23T16:49:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:50.711704+0000", 2024-04-23T16:49:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3600000000000003e-07 2024-04-23T16:49:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:50.711707+0000", 2024-04-23T16:49:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2100000000000002e-06 2024-04-23T16:49:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:50.711712+0000", 2024-04-23T16:49:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0880000000000002e-06 2024-04-23T16:49:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:50.711718+0000", 2024-04-23T16:49:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7699999999999998e-06 2024-04-23T16:49:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:50.714710+0000", 2024-04-23T16:49:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0029922310000000001 2024-04-23T16:49:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:50.714788+0000", 2024-04-23T16:49:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8004999999999996e-05 2024-04-23T16:49:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 11", 2024-04-23T16:49:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:50.714877+0000", 2024-04-23T16:49:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8632000000000001e-05 2024-04-23T16:49:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:49:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:50.717365+0000", 2024-04-23T16:49:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024883169999999999 2024-04-23T16:49:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:49:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:50.793177+0000", 2024-04-23T16:49:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.075812266000000003 2024-04-23T16:49:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:50.793207+0000", 2024-04-23T16:49:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.955e-05 2024-04-23T16:49:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:48:50.793233+0000", 2024-04-23T16:49:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5975000000000001e-05 2024-04-23T16:49:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:168538 8.7 8:eecba0a1:::benchmark_data_smithi188_155546_object43029:head [delete] snapc 0=[] ondisk+write+known_if_redirected e663)", 2024-04-23T16:49:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:49:06.613782+0000", 2024-04-23T16:49:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.700176128999999, 2024-04-23T16:49:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.091573801999999996, 2024-04-23T16:49:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:49:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:49:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:49:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 168538 2024-04-23T16:49:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:06.613782+0000", 2024-04-23T16:49:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:06.613782+0000", 2024-04-23T16:49:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:06.613784+0000", 2024-04-23T16:49:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0270000000000001e-06 2024-04-23T16:49:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:06.613788+0000", 2024-04-23T16:49:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6970000000000001e-06 2024-04-23T16:49:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:06.613797+0000", 2024-04-23T16:49:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0299999999999999e-06 2024-04-23T16:49:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:06.613805+0000", 2024-04-23T16:49:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5299999999999999e-06 2024-04-23T16:49:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:06.618100+0000", 2024-04-23T16:49:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042947899999999997 2024-04-23T16:49:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:06.618195+0000", 2024-04-23T16:49:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5354999999999998e-05 2024-04-23T16:49:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 11", 2024-04-23T16:49:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:06.618282+0000", 2024-04-23T16:49:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7012e-05 2024-04-23T16:49:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:49:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:06.621975+0000", 2024-04-23T16:49:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0036928479999999999 2024-04-23T16:49:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:49:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:06.705294+0000", 2024-04-23T16:49:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.083319330999999996 2024-04-23T16:49:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:06.705323+0000", 2024-04-23T16:49:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8398000000000002e-05 2024-04-23T16:49:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:06.705356+0000", 2024-04-23T16:49:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3784e-05 2024-04-23T16:49:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.16995.0:192746 8.f e670/609 8:fd9eac50:::benchmark_data_smithi188_155546_object67237:head v 670'12056, mlcod=670'12053)", 2024-04-23T16:49:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:49:36.614493+0000", 2024-04-23T16:49:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 2.699465987, 2024-04-23T16:49:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.118752438, 2024-04-23T16:49:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.614493+0000", 2024-04-23T16:49:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.614493+0000", 2024-04-23T16:49:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.614494+0000", 2024-04-23T16:49:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.243e-06 2024-04-23T16:49:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.614500+0000", 2024-04-23T16:49:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.246e-06 2024-04-23T16:49:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.614508+0000", 2024-04-23T16:49:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5610000000000001e-06 2024-04-23T16:49:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.614513+0000", 2024-04-23T16:49:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0579999999999997e-06 2024-04-23T16:49:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.614562+0000", 2024-04-23T16:49:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9289000000000002e-05 2024-04-23T16:49:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.614575+0000", 2024-04-23T16:49:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2711000000000001e-05 2024-04-23T16:49:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.733199+0000", 2024-04-23T16:49:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.118623861 2024-04-23T16:49:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.733245+0000", 2024-04-23T16:49:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6468999999999998e-05 2024-04-23T16:49:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.16995.0:192758 8.5 e670/609 8:a94bc997:::benchmark_data_smithi188_155546_object67249:head v 670'11829, mlcod=670'11827)", 2024-04-23T16:49:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:49:36.615631+0000", 2024-04-23T16:49:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 2.698327736, 2024-04-23T16:49:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13628989899999999, 2024-04-23T16:49:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.615631+0000", 2024-04-23T16:49:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.615631+0000", 2024-04-23T16:49:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.615632+0000", 2024-04-23T16:49:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3430000000000001e-06 2024-04-23T16:49:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.615640+0000", 2024-04-23T16:49:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7239999999999992e-06 2024-04-23T16:49:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.615649+0000", 2024-04-23T16:49:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1549999999999996e-06 2024-04-23T16:49:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.615656+0000", 2024-04-23T16:49:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7290000000000003e-06 2024-04-23T16:49:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.733198+0000", 2024-04-23T16:49:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.117542541 2024-04-23T16:49:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.733226+0000", 2024-04-23T16:49:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7537000000000001e-05 2024-04-23T16:49:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.751893+0000", 2024-04-23T16:49:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018667065 2024-04-23T16:49:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.751921+0000", 2024-04-23T16:49:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7804999999999999e-05 2024-04-23T16:49:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.16995.0:192759 8.3 e670/609 8:c9627483:::benchmark_data_smithi188_155546_object67250:head v 670'12083, mlcod=670'12081)", 2024-04-23T16:49:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:49:36.619210+0000", 2024-04-23T16:49:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 2.6947481280000001, 2024-04-23T16:49:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.12931895299999999, 2024-04-23T16:49:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.619210+0000", 2024-04-23T16:49:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.619210+0000", 2024-04-23T16:49:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.619212+0000", 2024-04-23T16:49:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2699999999999999e-06 2024-04-23T16:49:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.619218+0000", 2024-04-23T16:49:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5520000000000001e-06 2024-04-23T16:49:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.733180+0000", 2024-04-23T16:49:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.113961307 2024-04-23T16:49:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.733199+0000", 2024-04-23T16:49:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9361999999999999e-05 2024-04-23T16:49:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.733272+0000", 2024-04-23T16:49:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3301999999999997e-05 2024-04-23T16:49:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.733291+0000", 2024-04-23T16:49:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9114e-05 2024-04-23T16:49:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.748491+0000", 2024-04-23T16:49:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015200094000000001 2024-04-23T16:49:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.748529+0000", 2024-04-23T16:49:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7951999999999998e-05 2024-04-23T16:49:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.16995.0:192769 8.5 e670/609 8:ab08c879:::benchmark_data_smithi188_155546_object67260:head v 670'11830, mlcod=670'11827)", 2024-04-23T16:49:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:49:36.626815+0000", 2024-04-23T16:49:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 2.6871436600000003, 2024-04-23T16:49:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.12512695400000001, 2024-04-23T16:49:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:49:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:49:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:49:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:49:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:49:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.626815+0000", 2024-04-23T16:49:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:49:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.626815+0000", 2024-04-23T16:49:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:49:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:49:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.626816+0000", 2024-04-23T16:49:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3009999999999999e-06 2024-04-23T16:49:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:49:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.626823+0000", 2024-04-23T16:49:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0029999999999997e-06 2024-04-23T16:49:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:49:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.733184+0000", 2024-04-23T16:49:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.106360677 2024-04-23T16:49:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:49:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.733199+0000", 2024-04-23T16:49:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5364000000000001e-05 2024-04-23T16:49:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:49:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.737829+0000", 2024-04-23T16:49:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046293510000000003 2024-04-23T16:49:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:49:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.737849+0000", 2024-04-23T16:49:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0777000000000001e-05 2024-04-23T16:49:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:49:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.751925+0000", 2024-04-23T16:49:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014075560000000001 2024-04-23T16:49:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:49:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:49:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:49:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.751942+0000", 2024-04-23T16:49:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6920999999999999e-05 2024-04-23T16:49:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:49:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:49:39.425 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:49:39.425 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:49:39.453 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:39.474 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:49:39.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:39.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:49:39.578 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:49:39.629 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:39.662 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:49:39.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:39.670 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:49:39.771 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:49:39.799 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:39.822 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:49:39.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:39.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:49:39.926 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:49:39.954 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:39.974 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:49:39.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:39.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:49:40.078 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:49:40.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:39 smithi103 ceph-mon[39629]: pgmap v1872: 27 pgs: 2 active+recovering, 25 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 886 op/s; 2126/117488 objects misplaced (1.810%); 6.1 MiB/s, 97 objects/s recovering 2024-04-23T16:49:40.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:39 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:49:40.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:39 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:49:40.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:39 smithi103 ceph-mon[39629]: osdmap e671: 12 total, 12 up, 8 in 2024-04-23T16:49:40.103 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:39 smithi148 ceph-mon[44520]: pgmap v1872: 27 pgs: 2 active+recovering, 25 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 886 op/s; 2126/117488 objects misplaced (1.810%); 6.1 MiB/s, 97 objects/s recovering 2024-04-23T16:49:40.103 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:39 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:49:40.103 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:39 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:49:40.103 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:39 smithi148 ceph-mon[44520]: osdmap e671: 12 total, 12 up, 8 in 2024-04-23T16:49:40.105 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:40.127 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:49:40.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:40.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:49:40.231 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:49:40.258 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:40.278 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:49:40.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:40.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:49:40.382 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:49:40.421 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:40.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:39 smithi188 ceph-mon[44101]: pgmap v1872: 27 pgs: 2 active+recovering, 25 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 886 op/s; 2126/117488 objects misplaced (1.810%); 6.1 MiB/s, 97 objects/s recovering 2024-04-23T16:49:40.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:39 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:49:40.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:39 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:49:40.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:39 smithi188 ceph-mon[44101]: osdmap e671: 12 total, 12 up, 8 in 2024-04-23T16:49:40.508 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:49:40.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:40.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:49:40.612 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:49:40.653 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:40.679 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:49:40.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:40.683 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:49:40.784 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:49:40.825 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:40.864 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:49:40.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:40.868 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:49:40.943 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-23T16:49:40.948 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:40.969 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:49:40.996 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:41.015 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:49:41.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:41.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:49:41.087 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:49:41.120 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:49:41.147 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:41.167 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:49:41.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:41.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:49:41.272 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:49:41.309 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:41.460 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:49:41.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:41.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:49:41.568 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:49:41.597 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:41.619 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:49:41.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:41.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:49:41.723 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:49:41.752 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:41.773 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:49:41.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:41.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:49:41.880 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:49:41.908 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:41.929 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:49:41.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:41.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:49:41.984 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:41 smithi103 ceph-mon[39629]: pgmap v1874: 27 pgs: 2 active+recovering, 25 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 1592/113730 objects misplaced (1.400%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:49:42.035 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:49:42.065 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:42.140 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:49:42.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:42.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:49:42.177 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:42.190 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:42.246 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:49:42.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:41 smithi188 ceph-mon[44101]: pgmap v1874: 27 pgs: 2 active+recovering, 25 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 1592/113730 objects misplaced (1.400%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:49:42.305 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:42.340 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:49:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:42.343 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:49:42.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:41 smithi148 ceph-mon[44520]: pgmap v1874: 27 pgs: 2 active+recovering, 25 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 1592/113730 objects misplaced (1.400%); 8.0 MiB/s, 128 objects/s recovering 2024-04-23T16:49:42.444 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:49:42.472 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:42.823 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:49:42.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:42.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:49:42.929 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:49:42.956 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:42.976 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:49:42.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:42.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:49:43.009 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:49:43.035 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:42 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:49:43.035 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:42 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:49:43.035 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:42 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/185698406' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:49:43.035 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:42 smithi103 ceph-mon[39629]: osdmap e672: 12 total, 12 up, 8 in 2024-04-23T16:49:43.081 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:49:43.118 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:43.244 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:49:43.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:43.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:49:43.348 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:49:43.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:42 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:49:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:42 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:49:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:42 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/185698406' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:49:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:42 smithi148 ceph-mon[44520]: osdmap e672: 12 total, 12 up, 8 in 2024-04-23T16:49:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:42 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:49:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:42 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:49:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:42 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/185698406' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:49:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:42 smithi188 ceph-mon[44101]: osdmap e672: 12 total, 12 up, 8 in 2024-04-23T16:49:43.415 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:43.450 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:49:43.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:43.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:49:43.556 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:49:43.615 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:43.659 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:49:43.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:43.662 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:49:43.763 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:49:43.826 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:43.858 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:49:43.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:43.870 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:49:43.971 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:49:44.001 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:43 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:49:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:43 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:49:44.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:43 smithi188 ceph-mon[44101]: pgmap v1876: 27 pgs: 2 active+recovering, 25 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 835 op/s; 1592/108796 objects misplaced (1.463%); 4.2 MiB/s, 66 objects/s recovering 2024-04-23T16:49:44.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:43 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:49:44.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:43 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/185698406' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:49:44.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:43 smithi188 ceph-mon[44101]: osdmap e673: 12 total, 12 up, 8 in 2024-04-23T16:49:44.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:43 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:49:44.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:43 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:49:44.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:43 smithi103 ceph-mon[39629]: pgmap v1876: 27 pgs: 2 active+recovering, 25 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 835 op/s; 1592/108796 objects misplaced (1.463%); 4.2 MiB/s, 66 objects/s recovering 2024-04-23T16:49:44.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:43 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:49:44.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:43 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/185698406' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:49:44.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:43 smithi103 ceph-mon[39629]: osdmap e673: 12 total, 12 up, 8 in 2024-04-23T16:49:44.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:43 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:49:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:43 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:49:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:43 smithi148 ceph-mon[44520]: pgmap v1876: 27 pgs: 2 active+recovering, 25 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 835 op/s; 1592/108796 objects misplaced (1.463%); 4.2 MiB/s, 66 objects/s recovering 2024-04-23T16:49:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:43 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:49:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:43 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/185698406' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:49:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:43 smithi148 ceph-mon[44520]: osdmap e673: 12 total, 12 up, 8 in 2024-04-23T16:49:45.158 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:49:45.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:45.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:49:45.275 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:49:45.302 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:49:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:45.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:49:45.361 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:49:45.361 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2367,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":674,"num_osds":12,"num_up_osds":12,"osd_up_since":1713890820,"num_in_osds":8,"osd_in_since":1713890829,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":25},{"state_name":"active+recovering","count":2}],"num_pgs":27,"num_pools":2,"num_objects":54398,"data_bytes":3565421112,"bytes_used":12322713600,"bytes_avail":755637571584,"bytes_total":767960285184,"misplaced_objects":1592,"misplaced_total":108796,"misplaced_ratio":0.014632890915107173,"recovering_objects_per_sec":66,"recovering_bytes_per_sec":4373701,"recovering_keys_per_sec":0,"num_objects_recovered":534,"num_bytes_recovered":34996224,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":835},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.103:8443/"}},"servicemap":{"epoch":50,"modified":"2024-04-23T16:49:26.924853+0000","services":{"osd":{"daemons":{"summary":"","7":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"8":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-23T16:49:45.426 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:49:45.466 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:45.503 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:49:45.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:45.512 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:49:45.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:45 smithi103 ceph-mon[39629]: osdmap e674: 12 total, 12 up, 8 in 2024-04-23T16:49:45.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:45 smithi103 ceph-mon[39629]: pgmap v1879: 27 pgs: 2 active+recovering, 25 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 1060/105108 objects misplaced (1.008%); 12 MiB/s, 193 objects/s recovering 2024-04-23T16:49:45.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:45 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2727938882' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:49:45.613 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:49:45.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:45 smithi148 ceph-mon[44520]: osdmap e674: 12 total, 12 up, 8 in 2024-04-23T16:49:45.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:45 smithi148 ceph-mon[44520]: pgmap v1879: 27 pgs: 2 active+recovering, 25 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 1060/105108 objects misplaced (1.008%); 12 MiB/s, 193 objects/s recovering 2024-04-23T16:49:45.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:45 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2727938882' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:49:45.641 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:45.661 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:49:45.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:45.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:49:45.728 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:45 smithi188 ceph-mon[44101]: osdmap e674: 12 total, 12 up, 8 in 2024-04-23T16:49:45.728 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:45 smithi188 ceph-mon[44101]: pgmap v1879: 27 pgs: 2 active+recovering, 25 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 1060/105108 objects misplaced (1.008%); 12 MiB/s, 193 objects/s recovering 2024-04-23T16:49:45.728 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:45 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2727938882' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:49:45.764 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:49:45.800 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:46.003 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:49:46.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:46.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:49:46.110 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:49:46.137 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:46.160 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:49:46.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:46.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:49:46.264 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:49:46.291 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:46.316 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:49:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:46.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:49:46.419 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:49:46.449 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:46.471 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:49:46.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:46.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:49:46.563 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:49:46.563 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:49:46.563 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-23T16:49:46.563 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:49:46.577 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:49:46.604 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:46.623 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:49:46.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:46.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:49:46.727 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:49:46.758 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:46.836 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:49:46.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:46.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:49:46.944 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:49:46.971 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:46.990 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:49:46.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:46.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:49:47.096 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:49:47.123 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:47.141 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:49:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:47.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:49:47.246 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:49:47.272 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:47.295 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:49:47.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:47.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:49:47.399 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:49:47.428 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:48.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:47 smithi103 ceph-mon[39629]: pgmap v1880: 27 pgs: 2 active+recovering, 25 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 1060/103994 objects misplaced (1.019%); 5.5 MiB/s, 88 objects/s recovering 2024-04-23T16:49:48.287 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-23T16:49:48.287 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:49:48.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:47 smithi148 ceph-mon[44520]: pgmap v1880: 27 pgs: 2 active+recovering, 25 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 1060/103994 objects misplaced (1.019%); 5.5 MiB/s, 88 objects/s recovering 2024-04-23T16:49:48.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:47 smithi188 ceph-mon[44101]: pgmap v1880: 27 pgs: 2 active+recovering, 25 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 1060/103994 objects misplaced (1.019%); 5.5 MiB/s, 88 objects/s recovering 2024-04-23T16:49:49.234 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:49:49.235 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:49:49.235 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:49:49.235 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:49:49.235 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:49:49.235 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:49:49.235 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:49:49.235 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:49:49.236 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:49:49.236 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:49:49.236 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:49:49.236 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:49:49.236 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:49:49.236 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:49:49.237 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:49:49.237 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:49:49.237 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:49:49.237 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:49:49.237 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:49:49.237 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:49:49.237 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:49:49.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:49:49.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:49:49.238 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:49:49.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:48 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/468880581' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:49:49.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:48 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/468880581' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:49:49.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:48 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/468880581' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:49:49.516 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:49:49.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:49.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:49:49.621 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:49:49.652 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:50.002 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:49:50.003 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-23T16:49:50.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:50.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:49:50.106 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:49:50.134 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:50.154 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:49:50.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:50.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:49:50.257 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:49:50.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:49 smithi148 ceph-mon[44520]: pgmap v1881: 27 pgs: 2 active+recovering, 25 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 894 op/s; 1060/100214 objects misplaced (1.058%); 4.8 MiB/s, 76 objects/s recovering 2024-04-23T16:49:50.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.285 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.285 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.285 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.286 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:50.305 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:49:50.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:50.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:49:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:49 smithi103 ceph-mon[39629]: pgmap v1881: 27 pgs: 2 active+recovering, 25 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 894 op/s; 1060/100214 objects misplaced (1.058%); 4.8 MiB/s, 76 objects/s recovering 2024-04-23T16:49:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.412 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:49:50.439 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:50.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:49 smithi188 ceph-mon[44101]: pgmap v1881: 27 pgs: 2 active+recovering, 25 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 894 op/s; 1060/100214 objects misplaced (1.058%); 4.8 MiB/s, 76 objects/s recovering 2024-04-23T16:49:50.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:49:50.538 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:49:50.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:50.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:49:50.626 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:50.644 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:49:50.676 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:50.687 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:50.940 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:49:50.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:50.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:49:51.049 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:49:51.111 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:51.131 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:49:51.144 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:49:51.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:51.147 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:49:51.248 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:49:51.278 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:52.034 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:49:52.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:52.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:49:52.138 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:49:52.204 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:52.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:52.249 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:49:52.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:49:52.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:52 smithi103 ceph-mon[39629]: pgmap v1882: 27 pgs: 2 active+recovering, 25 active+clean; 2.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 554/96186 objects misplaced (0.576%); 8.1 MiB/s, 129 objects/s recovering 2024-04-23T16:49:52.350 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:49:52.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:52 smithi148 ceph-mon[44520]: pgmap v1882: 27 pgs: 2 active+recovering, 25 active+clean; 2.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 554/96186 objects misplaced (0.576%); 8.1 MiB/s, 129 objects/s recovering 2024-04-23T16:49:52.380 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:52.455 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:49:52.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:52.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:49:52.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:52 smithi188 ceph-mon[44101]: pgmap v1882: 27 pgs: 2 active+recovering, 25 active+clean; 2.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 554/96186 objects misplaced (0.576%); 8.1 MiB/s, 129 objects/s recovering 2024-04-23T16:49:52.558 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:49:52.568 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:49:52.609 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:52.645 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:49:52.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:52.651 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:49:52.752 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:49:52.791 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:52.918 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:49:52.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:52.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:49:53.027 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:49:53.074 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:53.124 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:49:53.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:53.127 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:49:53.228 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:49:53.277 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:53.315 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:49:53.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:53.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:49:53.421 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:49:53.452 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:53.579 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:49:53.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:53.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:49:53.679 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:53.683 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:49:53.727 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:53.970 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:49:53.970 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:49:53.987 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:49:53.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:53.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:49:53.999 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:49:54.012 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:49:54.048 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:54.091 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:49:54.149 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:54.188 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:49:54.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:54.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:49:54.293 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:49:54.323 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:54 smithi103 ceph-mon[39629]: pgmap v1883: 27 pgs: 2 active+recovering, 25 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 554/92282 objects misplaced (0.600%); 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:49:54.323 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:54.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:54 smithi148 ceph-mon[44520]: pgmap v1883: 27 pgs: 2 active+recovering, 25 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 554/92282 objects misplaced (0.600%); 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:49:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:54 smithi188 ceph-mon[44101]: pgmap v1883: 27 pgs: 2 active+recovering, 25 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 554/92282 objects misplaced (0.600%); 6.5 MiB/s, 104 objects/s recovering 2024-04-23T16:49:55.278 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:49:55.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:55.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:49:55.282 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:55 smithi103 ceph-mon[39629]: from='client.36770 v1:172.21.15.103:0/3684827647' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:49:55.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:55 smithi148 ceph-mon[44520]: from='client.36770 v1:172.21.15.103:0/3684827647' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:49:55.382 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:49:55.414 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:55 smithi188 ceph-mon[44101]: from='client.36770 v1:172.21.15.103:0/3684827647' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:49:56.286 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:56 smithi103 ceph-mon[39629]: pgmap v1884: 27 pgs: 27 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 9.5 MiB/s, 151 objects/s recovering 2024-04-23T16:49:56.286 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:56 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:49:56.286 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:56 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:49:56.317 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-23T16:49:56.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:56 smithi148 ceph-mon[44520]: pgmap v1884: 27 pgs: 27 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 9.5 MiB/s, 151 objects/s recovering 2024-04-23T16:49:56.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:56 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:49:56.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:56 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:49:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:56 smithi188 ceph-mon[44101]: pgmap v1884: 27 pgs: 27 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 9.5 MiB/s, 151 objects/s recovering 2024-04-23T16:49:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:56 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:49:56.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:56 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:49:57.200 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:49:57.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:57.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:49:57.305 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:49:57.332 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:57 smithi148 ceph-mon[44520]: osdmap e675: 12 total, 12 up, 8 in 2024-04-23T16:49:57.334 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:57.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:57 smithi103 ceph-mon[39629]: osdmap e675: 12 total, 12 up, 8 in 2024-04-23T16:49:57.353 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:49:57.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:57.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:49:57.459 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:49:57.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:57 smithi188 ceph-mon[44101]: osdmap e675: 12 total, 12 up, 8 in 2024-04-23T16:49:57.492 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:57.713 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1883,"stamp":"2024-04-23T16:49:52.929671+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3024290360,"num_objects":46141,"num_object_clones":0,"num_object_copies":92282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":554,"num_objects_unfound":0,"num_objects_dirty":46141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204875,"num_write_kb":8032385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20173,"num_bytes_recovered":1322329152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162404,"ondisk_log_size":162404,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11000988,"kb_used_data":5947160,"kb_used_omap":242,"kb_used_meta":5053517,"kb_avail":738960228,"statfs":{"total":767960285184,"available":756695273472,"internally_reserved":0,"allocated":6089891840,"data_stored":6086082856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":247821,"internal_metadata":5174802419},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,3,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-541130752,"num_objects":-8257,"num_object_clones":0,"num_object_copies":-16514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1038,"num_objects_unfound":0,"num_objects_dirty":-8257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8257,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1038,"num_bytes_recovered":68026368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.923573"},"pg_stats":[{"pgid":"8.7","version":"674'12709","reported_seq":12980,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:50.998302+0000","last_change":"2024-04-23T16:49:08.424575+0000","last_active":"2024-04-23T16:49:50.998302+0000","last_peered":"2024-04-23T16:49:50.998302+0000","last_clean":"2024-04-23T16:49:50.998302+0000","last_became_active":"2024-04-23T16:46:26.363422+0000","last_became_peered":"2024-04-23T16:46:26.363422+0000","last_unstale":"2024-04-23T16:49:50.998302+0000","last_undegraded":"2024-04-23T16:49:50.998302+0000","last_fullsized":"2024-04-23T16:49:50.998302+0000","mapping_epoch":609,"log_start":"614'2700","ondisk_log_start":"614'2700","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"663'10506","last_scrub_stamp":"2024-04-23T16:49:08.424442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:08.424442+0000","objects_scrubbed":5063,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:20.445332+0000","scrub_duration":991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186187776,"num_objects":2841,"num_object_clones":0,"num_object_copies":5682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12709,"num_write_kb":497600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"674'13100","reported_seq":13240,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:52.414514+0000","last_change":"2024-04-23T16:46:26.365698+0000","last_active":"2024-04-23T16:49:52.414514+0000","last_peered":"2024-04-23T16:49:52.414514+0000","last_clean":"2024-04-23T16:49:52.414514+0000","last_became_active":"2024-04-23T16:46:26.365309+0000","last_became_peered":"2024-04-23T16:46:26.365309+0000","last_unstale":"2024-04-23T16:49:52.414514+0000","last_undegraded":"2024-04-23T16:49:52.414514+0000","last_fullsized":"2024-04-23T16:49:52.414514+0000","mapping_epoch":609,"log_start":"614'3000","ondisk_log_start":"614'3000","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178782208,"num_objects":2728,"num_object_clones":0,"num_object_copies":5456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13100,"num_write_kb":506496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.5","version":"674'12605","reported_seq":12799,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:49.697254+0000","last_change":"2024-04-23T16:49:17.828862+0000","last_active":"2024-04-23T16:49:49.697254+0000","last_peered":"2024-04-23T16:49:49.697254+0000","last_clean":"2024-04-23T16:49:49.697254+0000","last_became_active":"2024-04-23T16:46:26.364780+0000","last_became_peered":"2024-04-23T16:46:26.364780+0000","last_unstale":"2024-04-23T16:49:49.697254+0000","last_undegraded":"2024-04-23T16:49:49.697254+0000","last_fullsized":"2024-04-23T16:49:49.697254+0000","mapping_epoch":609,"log_start":"614'2600","ondisk_log_start":"614'2600","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"665'10858","last_scrub_stamp":"2024-04-23T16:49:17.828773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:17.828773+0000","objects_scrubbed":4612,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:37.125054+0000","scrub_duration":521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186843136,"num_objects":2851,"num_object_clones":0,"num_object_copies":5702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12605,"num_write_kb":494592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"674'13068","reported_seq":23248,"reported_epoch":674,"state":"active+recovering","last_fresh":"2024-04-23T16:49:49.634775+0000","last_change":"2024-04-23T16:47:10.280064+0000","last_active":"2024-04-23T16:49:49.634775+0000","last_peered":"2024-04-23T16:49:49.634775+0000","last_clean":"2024-04-23T16:47:09.049801+0000","last_became_active":"2024-04-23T16:47:10.244722+0000","last_became_peered":"2024-04-23T16:47:10.244722+0000","last_unstale":"2024-04-23T16:49:49.634775+0000","last_undegraded":"2024-04-23T16:49:49.634775+0000","last_fullsized":"2024-04-23T16:49:49.634775+0000","mapping_epoch":623,"log_start":"614'3000","ondisk_log_start":"614'3000","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199098368,"num_objects":3038,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":317,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13068,"num_write_kb":515392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5018,"num_bytes_recovered":328859648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"5,9","objects":317},{"shards":"9,11","objects":2721}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"674'12681","reported_seq":12871,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:49.631556+0000","last_change":"2024-04-23T16:49:25.388473+0000","last_active":"2024-04-23T16:49:49.631556+0000","last_peered":"2024-04-23T16:49:49.631556+0000","last_clean":"2024-04-23T16:49:49.631556+0000","last_became_active":"2024-04-23T16:46:26.365000+0000","last_became_peered":"2024-04-23T16:46:26.365000+0000","last_unstale":"2024-04-23T16:49:49.631556+0000","last_undegraded":"2024-04-23T16:49:49.631556+0000","last_fullsized":"2024-04-23T16:49:49.631556+0000","mapping_epoch":609,"log_start":"614'2600","ondisk_log_start":"614'2600","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"667'11355","last_scrub_stamp":"2024-04-23T16:49:25.388270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:25.388270+0000","objects_scrubbed":4236,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:40.837654+0000","scrub_duration":563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189988864,"num_objects":2899,"num_object_clones":0,"num_object_copies":5798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12681,"num_write_kb":498560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"674'12971","reported_seq":13111,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:52.408415+0000","last_change":"2024-04-23T16:46:26.365668+0000","last_active":"2024-04-23T16:49:52.408415+0000","last_peered":"2024-04-23T16:49:52.408415+0000","last_clean":"2024-04-23T16:49:52.408415+0000","last_became_active":"2024-04-23T16:46:26.364349+0000","last_became_peered":"2024-04-23T16:46:26.364349+0000","last_unstale":"2024-04-23T16:49:52.408415+0000","last_undegraded":"2024-04-23T16:49:52.408415+0000","last_fullsized":"2024-04-23T16:49:52.408415+0000","mapping_epoch":609,"log_start":"614'2900","ondisk_log_start":"614'2900","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184877056,"num_objects":2821,"num_object_clones":0,"num_object_copies":5642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12971,"num_write_kb":505344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1670,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:44.433577+0000","last_change":"2024-04-23T16:45:33.395526+0000","last_active":"2024-04-23T16:49:44.433577+0000","last_peered":"2024-04-23T16:49:44.433577+0000","last_clean":"2024-04-23T16:49:44.433577+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:49:44.433577+0000","last_undegraded":"2024-04-23T16:49:44.433577+0000","last_fullsized":"2024-04-23T16:49:44.433577+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:33.395406+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:45:33.395406+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.1","version":"674'12846","reported_seq":13118,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:51.006572+0000","last_change":"2024-04-23T16:49:22.266220+0000","last_active":"2024-04-23T16:49:51.006572+0000","last_peered":"2024-04-23T16:49:51.006572+0000","last_clean":"2024-04-23T16:49:51.006572+0000","last_became_active":"2024-04-23T16:46:26.362670+0000","last_became_peered":"2024-04-23T16:46:26.362670+0000","last_unstale":"2024-04-23T16:49:51.006572+0000","last_undegraded":"2024-04-23T16:49:51.006572+0000","last_fullsized":"2024-04-23T16:49:51.006572+0000","mapping_epoch":609,"log_start":"614'2800","ondisk_log_start":"614'2800","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"666'11314","last_scrub_stamp":"2024-04-23T16:49:22.266039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:22.266039+0000","objects_scrubbed":4452,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:38.560321+0000","scrub_duration":641,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190709760,"num_objects":2910,"num_object_clones":0,"num_object_copies":5820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12846,"num_write_kb":504192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1550,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:44.438579+0000","last_change":"2024-04-23T16:49:25.478623+0000","last_active":"2024-04-23T16:49:44.438579+0000","last_peered":"2024-04-23T16:49:44.438579+0000","last_clean":"2024-04-23T16:49:44.438579+0000","last_became_active":"2024-04-23T16:47:10.310322+0000","last_became_peered":"2024-04-23T16:47:10.310322+0000","last_unstale":"2024-04-23T16:49:44.438579+0000","last_undegraded":"2024-04-23T16:49:44.438579+0000","last_fullsized":"2024-04-23T16:49:44.438579+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:49:25.478506+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:49:25.478506+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:27.187808+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.0","version":"674'12354","reported_seq":22399,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:49.610745+0000","last_change":"2024-04-23T16:49:11.451708+0000","last_active":"2024-04-23T16:49:49.610745+0000","last_peered":"2024-04-23T16:49:49.610745+0000","last_clean":"2024-04-23T16:49:49.610745+0000","last_became_active":"2024-04-23T16:47:10.314835+0000","last_became_peered":"2024-04-23T16:47:10.314835+0000","last_unstale":"2024-04-23T16:49:49.610745+0000","last_undegraded":"2024-04-23T16:49:49.610745+0000","last_fullsized":"2024-04-23T16:49:49.610745+0000","mapping_epoch":623,"log_start":"614'2300","ondisk_log_start":"614'2300","created":609,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195166208,"num_objects":2978,"num_object_clones":0,"num_object_copies":5956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12354,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4950,"num_bytes_recovered":324403200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1662,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:44.422101+0000","last_change":"2024-04-23T16:48:28.280886+0000","last_active":"2024-04-23T16:49:44.422101+0000","last_peered":"2024-04-23T16:49:44.422101+0000","last_clean":"2024-04-23T16:49:44.422101+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:49:44.422101+0000","last_undegraded":"2024-04-23T16:49:44.422101+0000","last_fullsized":"2024-04-23T16:49:44.422101+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:28.280751+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:28.280751+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:56.638217+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"674'12783","reported_seq":13065,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:47.922446+0000","last_change":"2024-04-23T16:49:08.214265+0000","last_active":"2024-04-23T16:49:47.922446+0000","last_peered":"2024-04-23T16:49:47.922446+0000","last_clean":"2024-04-23T16:49:47.922446+0000","last_became_active":"2024-04-23T16:46:26.366359+0000","last_became_peered":"2024-04-23T16:46:26.366359+0000","last_unstale":"2024-04-23T16:49:47.922446+0000","last_undegraded":"2024-04-23T16:49:47.922446+0000","last_fullsized":"2024-04-23T16:49:47.922446+0000","mapping_epoch":609,"log_start":"614'2700","ondisk_log_start":"614'2700","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"663'10692","last_scrub_stamp":"2024-04-23T16:49:08.214052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:08.214052+0000","objects_scrubbed":5183,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:11.250982+0000","scrub_duration":1042,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200998912,"num_objects":3067,"num_object_clones":0,"num_object_copies":6134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12783,"num_write_kb":507200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1677,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:44.635499+0000","last_change":"2024-04-23T16:47:08.281155+0000","last_active":"2024-04-23T16:49:44.635499+0000","last_peered":"2024-04-23T16:49:44.635499+0000","last_clean":"2024-04-23T16:49:44.635499+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:49:44.635499+0000","last_undegraded":"2024-04-23T16:49:44.635499+0000","last_fullsized":"2024-04-23T16:49:44.635499+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:47:08.281053+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:47:08.281053+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:44.659630+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"674'12593","reported_seq":12862,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:47.925026+0000","last_change":"2024-04-23T16:49:15.382664+0000","last_active":"2024-04-23T16:49:47.925026+0000","last_peered":"2024-04-23T16:49:47.925026+0000","last_clean":"2024-04-23T16:49:47.925026+0000","last_became_active":"2024-04-23T16:46:26.366686+0000","last_became_peered":"2024-04-23T16:46:26.366686+0000","last_unstale":"2024-04-23T16:49:47.925026+0000","last_undegraded":"2024-04-23T16:49:47.925026+0000","last_fullsized":"2024-04-23T16:49:47.925026+0000","mapping_epoch":609,"log_start":"614'2500","ondisk_log_start":"614'2500","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"665'10905","last_scrub_stamp":"2024-04-23T16:49:15.382180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:15.382180+0000","objects_scrubbed":4744,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:32.221200+0000","scrub_duration":1175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198508544,"num_objects":3029,"num_object_clones":0,"num_object_copies":6058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12593,"num_write_kb":499904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1261,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:44.426530+0000","last_change":"2024-04-23T16:48:28.293172+0000","last_active":"2024-04-23T16:49:44.426530+0000","last_peered":"2024-04-23T16:49:44.426530+0000","last_clean":"2024-04-23T16:49:44.426530+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:49:44.426530+0000","last_undegraded":"2024-04-23T16:49:44.426530+0000","last_fullsized":"2024-04-23T16:49:44.426530+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:28.293038+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:48:28.293038+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:50.003967+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"674'12627","reported_seq":22603,"reported_epoch":674,"state":"active+recovering","last_fresh":"2024-04-23T16:49:49.626069+0000","last_change":"2024-04-23T16:47:10.314968+0000","last_active":"2024-04-23T16:49:49.626069+0000","last_peered":"2024-04-23T16:49:49.626069+0000","last_clean":"2024-04-23T16:47:09.038944+0000","last_became_active":"2024-04-23T16:47:10.251499+0000","last_became_peered":"2024-04-23T16:47:10.251499+0000","last_unstale":"2024-04-23T16:49:49.626069+0000","last_undegraded":"2024-04-23T16:49:49.626069+0000","last_fullsized":"2024-04-23T16:49:49.626069+0000","mapping_epoch":623,"log_start":"614'2600","ondisk_log_start":"614'2600","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186712064,"num_objects":2849,"num_object_clones":0,"num_object_copies":5698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":237,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12627,"num_write_kb":495232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4916,"num_bytes_recovered":322174976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"5,9","objects":237},{"shards":"9,11","objects":2612}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1215,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:44.438594+0000","last_change":"2024-04-23T16:48:36.291394+0000","last_active":"2024-04-23T16:49:44.438594+0000","last_peered":"2024-04-23T16:49:44.438594+0000","last_clean":"2024-04-23T16:49:44.438594+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:49:44.438594+0000","last_undegraded":"2024-04-23T16:49:44.438594+0000","last_fullsized":"2024-04-23T16:49:44.438594+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:36.291278+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:48:36.291278+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:58.136232+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.f","version":"674'12912","reported_seq":13052,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:52.424003+0000","last_change":"2024-04-23T16:46:26.364838+0000","last_active":"2024-04-23T16:49:52.424003+0000","last_peered":"2024-04-23T16:49:52.424003+0000","last_clean":"2024-04-23T16:49:52.424003+0000","last_became_active":"2024-04-23T16:46:26.364570+0000","last_became_peered":"2024-04-23T16:46:26.364570+0000","last_unstale":"2024-04-23T16:49:52.424003+0000","last_undegraded":"2024-04-23T16:49:52.424003+0000","last_fullsized":"2024-04-23T16:49:52.424003+0000","mapping_epoch":609,"log_start":"614'2900","ondisk_log_start":"614'2900","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181141504,"num_objects":2764,"num_object_clones":0,"num_object_copies":5528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12912,"num_write_kb":501632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1678,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:44.433493+0000","last_change":"2024-04-23T16:45:30.258925+0000","last_active":"2024-04-23T16:49:44.433493+0000","last_peered":"2024-04-23T16:49:44.433493+0000","last_clean":"2024-04-23T16:49:44.433493+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:49:44.433493+0000","last_undegraded":"2024-04-23T16:49:44.433493+0000","last_fullsized":"2024-04-23T16:49:44.433493+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:45:30.258804+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:45:30.258804+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.e","version":"674'12642","reported_seq":12782,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:49.632993+0000","last_change":"2024-04-23T16:46:26.365834+0000","last_active":"2024-04-23T16:49:49.632993+0000","last_peered":"2024-04-23T16:49:49.632993+0000","last_clean":"2024-04-23T16:49:49.632993+0000","last_became_active":"2024-04-23T16:46:26.365596+0000","last_became_peered":"2024-04-23T16:46:26.365596+0000","last_unstale":"2024-04-23T16:49:49.632993+0000","last_undegraded":"2024-04-23T16:49:49.632993+0000","last_fullsized":"2024-04-23T16:49:49.632993+0000","mapping_epoch":609,"log_start":"614'2600","ondisk_log_start":"614'2600","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:50.750225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183894016,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12642,"num_write_kb":494336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1684,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:44.600250+0000","last_change":"2024-04-23T16:48:38.816478+0000","last_active":"2024-04-23T16:49:44.600250+0000","last_peered":"2024-04-23T16:49:44.600250+0000","last_clean":"2024-04-23T16:49:44.600250+0000","last_became_active":"2024-04-23T16:47:10.266014+0000","last_became_peered":"2024-04-23T16:47:10.266014+0000","last_unstale":"2024-04-23T16:49:44.600250+0000","last_undegraded":"2024-04-23T16:49:44.600250+0000","last_fullsized":"2024-04-23T16:49:44.600250+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:38.816369+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:38.816369+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:57.044042+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"674'13071","reported_seq":13211,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:52.440234+0000","last_change":"2024-04-23T16:46:26.364586+0000","last_active":"2024-04-23T16:49:52.440234+0000","last_peered":"2024-04-23T16:49:52.440234+0000","last_clean":"2024-04-23T16:49:52.440234+0000","last_became_active":"2024-04-23T16:46:26.364302+0000","last_became_peered":"2024-04-23T16:46:26.364302+0000","last_unstale":"2024-04-23T16:49:52.440234+0000","last_undegraded":"2024-04-23T16:49:52.440234+0000","last_fullsized":"2024-04-23T16:49:52.440234+0000","mapping_epoch":609,"log_start":"614'3000","ondisk_log_start":"614'3000","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181141528,"num_objects":2765,"num_object_clones":0,"num_object_copies":5530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13071,"num_write_kb":506689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1563,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:44.433667+0000","last_change":"2024-04-23T16:47:10.037769+0000","last_active":"2024-04-23T16:49:44.433667+0000","last_peered":"2024-04-23T16:49:44.433667+0000","last_clean":"2024-04-23T16:49:44.433667+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:49:44.433667+0000","last_undegraded":"2024-04-23T16:49:44.433667+0000","last_fullsized":"2024-04-23T16:49:44.433667+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:47:10.037665+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:47:10.037665+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"674'12737","reported_seq":12877,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:49.639887+0000","last_change":"2024-04-23T16:46:26.364583+0000","last_active":"2024-04-23T16:49:49.639887+0000","last_peered":"2024-04-23T16:49:49.639887+0000","last_clean":"2024-04-23T16:49:49.639887+0000","last_became_active":"2024-04-23T16:46:26.364307+0000","last_became_peered":"2024-04-23T16:46:26.364307+0000","last_unstale":"2024-04-23T16:49:49.639887+0000","last_undegraded":"2024-04-23T16:49:49.639887+0000","last_fullsized":"2024-04-23T16:49:49.639887+0000","mapping_epoch":609,"log_start":"614'2700","ondisk_log_start":"614'2700","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192610304,"num_objects":2939,"num_object_clones":0,"num_object_copies":5878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":501632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1494,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:44.600340+0000","last_change":"2024-04-23T16:48:33.921605+0000","last_active":"2024-04-23T16:49:44.600340+0000","last_peered":"2024-04-23T16:49:44.600340+0000","last_clean":"2024-04-23T16:49:44.600340+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:49:44.600340+0000","last_undegraded":"2024-04-23T16:49:44.600340+0000","last_fullsized":"2024-04-23T16:49:44.600340+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:33.921450+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:48:33.921450+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:54.276555+0000","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.9","version":"674'13176","reported_seq":23897,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:52.434282+0000","last_change":"2024-04-23T16:49:22.445082+0000","last_active":"2024-04-23T16:49:52.434282+0000","last_peered":"2024-04-23T16:49:52.434282+0000","last_clean":"2024-04-23T16:49:52.434282+0000","last_became_active":"2024-04-23T16:47:10.431200+0000","last_became_peered":"2024-04-23T16:47:10.431200+0000","last_unstale":"2024-04-23T16:49:52.434282+0000","last_undegraded":"2024-04-23T16:49:52.434282+0000","last_fullsized":"2024-04-23T16:49:52.434282+0000","mapping_epoch":623,"log_start":"614'3100","ondisk_log_start":"614'3100","created":609,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187039744,"num_objects":2854,"num_object_clones":0,"num_object_copies":5708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13176,"num_write_kb":512960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5287,"num_bytes_recovered":346488832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1662,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-04-23T16:49:44.422101+0000","last_change":"2024-04-23T16:48:37.224501+0000","last_active":"2024-04-23T16:49:44.422101+0000","last_peered":"2024-04-23T16:49:44.422101+0000","last_clean":"2024-04-23T16:49:44.422101+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:49:44.422101+0000","last_undegraded":"2024-04-23T16:49:44.422101+0000","last_fullsized":"2024-04-23T16:49:44.422101+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:37.224360+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:48:37.224360+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:50.365513+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3023699992,"num_objects":46139,"num_object_clones":0,"num_object_copies":92278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":554,"num_objects_unfound":0,"num_objects_dirty":46139,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204875,"num_write_kb":8032385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20171,"num_bytes_recovered":1321926656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7246782464,"data_stored":7246774320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160875,"ondisk_log_size":160875,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1679360,"data_stored":1661600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057055,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2307044,"kb_used_data":1413724,"kb_used_omap":1,"kb_used_meta":893310,"kb_avail":91438108,"statfs":{"total":95995035648,"available":93632622592,"internally_reserved":0,"allocated":1447653376,"data_stored":1447175087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":914749892},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351397,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613034,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1788832,"kb_used_data":1144024,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":91956320,"statfs":{"total":95995035648,"available":94163271680,"internally_reserved":0,"allocated":1171480576,"data_stored":1171006359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":660273600},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841966,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1004520,"kb_used_data":590176,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92740632,"statfs":{"total":95995035648,"available":94966407168,"internally_reserved":0,"allocated":604340224,"data_stored":603862455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169013,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1019812,"kb_used_data":370844,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92725340,"statfs":{"total":95995035648,"available":94950748160,"internally_reserved":0,"allocated":379744256,"data_stored":379265967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":664533450},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463353,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545764,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199388,"statfs":{"total":95995035648,"available":95436173312,"internally_reserved":0,"allocated":942080,"data_stored":467863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137494,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1233888,"kb_used_data":922776,"kb_used_omap":231,"kb_used_meta":310872,"kb_avail":92511264,"statfs":{"total":95995035648,"available":94731534336,"internally_reserved":0,"allocated":944922624,"data_stored":944448407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":318333840},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738839,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1282400,"kb_used_data":569560,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":92462752,"statfs":{"total":95995035648,"available":94681858048,"internally_reserved":0,"allocated":583229440,"data_stored":582755223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269509,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2658584756260,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921217,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1818728,"kb_used_data":935136,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":91926424,"statfs":{"total":95995035648,"available":94132658176,"internally_reserved":0,"allocated":957579264,"data_stored":957101495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904788421},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757693,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":571539456,"data_stored":571539456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":934084608,"data_stored":934084608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":949944320,"data_stored":949944320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1214840832,"data_stored":1214840832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":371920896,"data_stored":371916824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":598278144,"data_stored":598278144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1160904704,"data_stored":1160904704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1445269504,"data_stored":1445265432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:49:58.028 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:49:58 smithi103 ceph-mon[39629]: pgmap v1886: 27 pgs: 27 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 891 op/s; 6.6 MiB/s, 105 objects/s recovering 2024-04-23T16:49:58.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:49:58 smithi148 ceph-mon[44520]: pgmap v1886: 27 pgs: 27 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 891 op/s; 6.6 MiB/s, 105 objects/s recovering 2024-04-23T16:49:58.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:49:58 smithi188 ceph-mon[44101]: pgmap v1886: 27 pgs: 27 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 891 op/s; 6.6 MiB/s, 105 objects/s recovering 2024-04-23T16:49:58.949 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:49:58.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:58.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:49:59.054 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:49:59.084 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:59.138 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:49:59.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:59.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:49:59.247 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:49:59.277 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:59.297 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:49:59.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:59.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:49:59.401 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:49:59.430 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:59.582 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:49:59.587 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:49:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:59.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:49:59.689 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:49:59.745 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:49:59.782 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:49:59.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:49:59.792 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:49:59.893 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:49:59.928 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:00.029 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:00.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:00.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:00.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:00 smithi103 ceph-mon[39629]: pgmap v1887: 27 pgs: 27 active+clean; 2.5 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 915 op/s; 6.6 MiB/s, 105 objects/s recovering 2024-04-23T16:50:00.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:00 smithi103 ceph-mon[39629]: osdmap e676: 12 total, 12 up, 8 in 2024-04-23T16:50:00.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:00 smithi103 ceph-mon[39629]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-04-23T16:50:00.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:00 smithi103 ceph-mon[39629]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-04-23T16:50:00.134 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:50:00.162 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:00.185 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:50:00.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:00.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:50:00.289 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:00.331 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:00.369 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:00.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:00.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:00.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:00 smithi148 ceph-mon[44520]: pgmap v1887: 27 pgs: 27 active+clean; 2.5 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 915 op/s; 6.6 MiB/s, 105 objects/s recovering 2024-04-23T16:50:00.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:00 smithi148 ceph-mon[44520]: osdmap e676: 12 total, 12 up, 8 in 2024-04-23T16:50:00.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:00 smithi148 ceph-mon[44520]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-04-23T16:50:00.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:00 smithi148 ceph-mon[44520]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-04-23T16:50:00.472 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:50:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:00 smithi188 ceph-mon[44101]: pgmap v1887: 27 pgs: 27 active+clean; 2.5 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 915 op/s; 6.6 MiB/s, 105 objects/s recovering 2024-04-23T16:50:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:00 smithi188 ceph-mon[44101]: osdmap e676: 12 total, 12 up, 8 in 2024-04-23T16:50:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:00 smithi188 ceph-mon[44101]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-04-23T16:50:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:00 smithi188 ceph-mon[44101]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-04-23T16:50:00.513 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:00.542 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:50:00.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:00.546 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:50:00.647 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:50:00.714 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:00.716 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:50:00.760 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:50:00.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:00.768 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:50:00.869 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:50:00.933 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:00.959 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:50:00.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:00.966 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:50:01.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:01 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1111242705' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:50:01.055 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:50:01.067 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:01.099 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:01.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:01 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1111242705' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:50:01.361 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:50:01.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:01 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1111242705' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:50:01.387 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:50:01.408 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:50:01.433 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:50:01.457 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:50:01.479 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:50:01.507 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:50:01.533 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:50:01.564 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:50:01.594 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:50:01.624 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:50:01.648 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:50:01.649 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:50:01.649 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:50:01.649 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:50:01.649 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:50:01.649 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:50:01.649 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:50:01.649 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:50:01.650 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:50:01.650 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:50:01.650 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:50:01.650 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:50:01.650 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:50:02.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:02 smithi103 ceph-mon[39629]: pgmap v1889: 27 pgs: 27 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 924 op/s; 4.3 MiB/s, 69 objects/s recovering 2024-04-23T16:50:02.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:02 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:50:02.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:02 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1111242705' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:50:02.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:02 smithi103 ceph-mon[39629]: osdmap e677: 12 total, 12 up, 8 in 2024-04-23T16:50:02.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:02 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2264730152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:50:02.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:02.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:02.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:02.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:02.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:02.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:02 smithi148 ceph-mon[44520]: pgmap v1889: 27 pgs: 27 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 924 op/s; 4.3 MiB/s, 69 objects/s recovering 2024-04-23T16:50:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:02 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:50:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:02 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1111242705' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:50:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:02 smithi148 ceph-mon[44520]: osdmap e677: 12 total, 12 up, 8 in 2024-04-23T16:50:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:02 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2264730152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:50:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:02.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:02 smithi188 ceph-mon[44101]: pgmap v1889: 27 pgs: 27 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 924 op/s; 4.3 MiB/s, 69 objects/s recovering 2024-04-23T16:50:02.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:02 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:50:02.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:02 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1111242705' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:50:02.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:02 smithi188 ceph-mon[44101]: osdmap e677: 12 total, 12 up, 8 in 2024-04-23T16:50:02.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:02 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2264730152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:50:02.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:02.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:02.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:02.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:02.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:03 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.354 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:03.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:03.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:03.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:03 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.397 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:50:03.398 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:50:03.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:50:03.398 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-23T16:50:03.398 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:50:03.461 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:50:03.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:03 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:03.494 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:04.058 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:04 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T16:50:04.058 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:04 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T16:50:04.059 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:04 smithi103 ceph-mon[39629]: 1.9 scrub starts 2024-04-23T16:50:04.059 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:04 smithi103 ceph-mon[39629]: 1.9 scrub ok 2024-04-23T16:50:04.059 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:04 smithi103 ceph-mon[39629]: pgmap v1891: 27 pgs: 27 active+clean; 2.2 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 943 op/s 2024-04-23T16:50:04.232 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-23T16:50:04.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:04 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T16:50:04.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:04 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T16:50:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:04 smithi148 ceph-mon[44520]: 1.9 scrub starts 2024-04-23T16:50:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:04 smithi148 ceph-mon[44520]: 1.9 scrub ok 2024-04-23T16:50:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:04 smithi148 ceph-mon[44520]: pgmap v1891: 27 pgs: 27 active+clean; 2.2 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 943 op/s 2024-04-23T16:50:04.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:04 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T16:50:04.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:04 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T16:50:04.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:04 smithi188 ceph-mon[44101]: 1.9 scrub starts 2024-04-23T16:50:04.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:04 smithi188 ceph-mon[44101]: 1.9 scrub ok 2024-04-23T16:50:04.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:04 smithi188 ceph-mon[44101]: pgmap v1891: 27 pgs: 27 active+clean; 2.2 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 943 op/s 2024-04-23T16:50:04.564 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:50:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:04.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:50:04.675 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:04.704 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:04.726 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:04.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:04.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:04.842 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:50:04.871 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:04.892 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:50:04.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:04.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:50:04.996 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:50:05.023 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:05.044 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:50:05.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:05.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:50:05.082 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:05.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:05 smithi148 ceph-mon[44520]: 1.a scrub starts 2024-04-23T16:50:05.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:05 smithi148 ceph-mon[44520]: osdmap e678: 12 total, 12 up, 8 in 2024-04-23T16:50:05.125 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:05.148 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:50:05.197 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:05.223 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:50:05.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:05.230 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:50:05.331 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:05.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:05 smithi103 ceph-mon[39629]: 1.a scrub starts 2024-04-23T16:50:05.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:05 smithi103 ceph-mon[39629]: osdmap e678: 12 total, 12 up, 8 in 2024-04-23T16:50:05.360 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:05.383 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:05.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:05.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:05.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:05 smithi188 ceph-mon[44101]: 1.a scrub starts 2024-04-23T16:50:05.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:05 smithi188 ceph-mon[44101]: osdmap e678: 12 total, 12 up, 8 in 2024-04-23T16:50:05.494 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:50:05.591 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:05.624 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:50:05.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:05.632 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:50:05.733 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:50:05.759 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:05.783 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:50:05.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:05.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:50:05.888 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:05.918 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:06.198 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:06 smithi103 ceph-mon[39629]: 1.a scrub ok 2024-04-23T16:50:06.198 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:06 smithi103 ceph-mon[39629]: pgmap v1893: 27 pgs: 27 active+clean; 2.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:50:06.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:06 smithi148 ceph-mon[44520]: 1.a scrub ok 2024-04-23T16:50:06.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:06 smithi148 ceph-mon[44520]: pgmap v1893: 27 pgs: 27 active+clean; 2.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:50:06.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:06 smithi188 ceph-mon[44101]: 1.a scrub ok 2024-04-23T16:50:06.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:06 smithi188 ceph-mon[44101]: pgmap v1893: 27 pgs: 27 active+clean; 2.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-23T16:50:06.566 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:50:07.073 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:07.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:07.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:07.120 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:50:07.177 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:50:07.203 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:07.226 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:50:07.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:07.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:50:07.330 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:50:07.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:07 smithi103 ceph-mon[39629]: 8.c scrub starts 2024-04-23T16:50:07.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:07 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:50:07.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:07 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:50:07.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:07 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:50:07.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:07 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:50:07.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:07 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:50:07.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:07 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:50:07.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:07 smithi103 ceph-mon[39629]: 8.c scrub ok 2024-04-23T16:50:07.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:07 smithi103 ceph-mon[39629]: osdmap e679: 12 total, 12 up, 8 in 2024-04-23T16:50:07.360 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:07.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:07 smithi148 ceph-mon[44520]: 8.c scrub starts 2024-04-23T16:50:07.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:07 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:50:07.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:07 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:50:07.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:07 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:50:07.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:07 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:50:07.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:07 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:50:07.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:07 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:50:07.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:07 smithi148 ceph-mon[44520]: 8.c scrub ok 2024-04-23T16:50:07.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:07 smithi148 ceph-mon[44520]: osdmap e679: 12 total, 12 up, 8 in 2024-04-23T16:50:07.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:07 smithi188 ceph-mon[44101]: 8.c scrub starts 2024-04-23T16:50:07.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:07 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:50:07.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:07 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:50:07.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:07 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:50:07.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:07 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:50:07.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:07 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:50:07.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:07 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:50:07.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:07 smithi188 ceph-mon[44101]: 8.c scrub ok 2024-04-23T16:50:07.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:07 smithi188 ceph-mon[44101]: osdmap e679: 12 total, 12 up, 8 in 2024-04-23T16:50:07.563 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:50:07.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:07.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:50:07.667 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:50:07.714 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:07.750 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:50:07.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:07.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:50:07.857 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:07.900 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:08.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:08 smithi103 ceph-mon[39629]: pgmap v1895: 27 pgs: 27 active+clean; 2.0 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 913 op/s 2024-04-23T16:50:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:08 smithi103 ceph-mon[39629]: 8.d scrub starts 2024-04-23T16:50:08.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:08 smithi103 ceph-mon[39629]: 8.d scrub ok 2024-04-23T16:50:08.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:08 smithi148 ceph-mon[44520]: pgmap v1895: 27 pgs: 27 active+clean; 2.0 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 913 op/s 2024-04-23T16:50:08.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:08 smithi148 ceph-mon[44520]: 8.d scrub starts 2024-04-23T16:50:08.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:08 smithi148 ceph-mon[44520]: 8.d scrub ok 2024-04-23T16:50:08.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:08 smithi188 ceph-mon[44101]: pgmap v1895: 27 pgs: 27 active+clean; 2.0 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 913 op/s 2024-04-23T16:50:08.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:08 smithi188 ceph-mon[44101]: 8.d scrub starts 2024-04-23T16:50:08.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:08 smithi188 ceph-mon[44101]: 8.d scrub ok 2024-04-23T16:50:08.650 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:50:08.654 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:50:08.732 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:50:08.732 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:50:08.732 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-23T16:50:08.732 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:50:09.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:09 smithi103 ceph-mon[39629]: 8.e scrub starts 2024-04-23T16:50:09.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:09 smithi103 ceph-mon[39629]: 1.5 scrub starts 2024-04-23T16:50:09.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:09 smithi103 ceph-mon[39629]: 8.e scrub ok 2024-04-23T16:50:09.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:09 smithi103 ceph-mon[39629]: 1.5 scrub ok 2024-04-23T16:50:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:09 smithi148 ceph-mon[44520]: 8.e scrub starts 2024-04-23T16:50:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:09 smithi148 ceph-mon[44520]: 1.5 scrub starts 2024-04-23T16:50:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:09 smithi148 ceph-mon[44520]: 8.e scrub ok 2024-04-23T16:50:09.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:09 smithi148 ceph-mon[44520]: 1.5 scrub ok 2024-04-23T16:50:09.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:09 smithi188 ceph-mon[44101]: 8.e scrub starts 2024-04-23T16:50:09.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:09 smithi188 ceph-mon[44101]: 1.5 scrub starts 2024-04-23T16:50:09.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:09 smithi188 ceph-mon[44101]: 8.e scrub ok 2024-04-23T16:50:09.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:09 smithi188 ceph-mon[44101]: 1.5 scrub ok 2024-04-23T16:50:09.758 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:09.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:09.798 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:09.863 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:50:09.890 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:09.912 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:50:09.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:09.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:50:10.016 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:50:10.059 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:10.105 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:50:10.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:10.112 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:50:10.213 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:50:10.240 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:10 smithi148 ceph-mon[44520]: from='client.17343 v1:172.21.15.103:0/558748827' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:50:10.240 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:10 smithi148 ceph-mon[44520]: pgmap v1896: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.9 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 893 op/s 2024-04-23T16:50:10.241 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:10 smithi148 ceph-mon[44520]: 8.0 scrub starts 2024-04-23T16:50:10.241 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:10 smithi148 ceph-mon[44520]: 8.3 scrub starts 2024-04-23T16:50:10.241 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:10 smithi148 ceph-mon[44520]: osdmap e680: 12 total, 12 up, 8 in 2024-04-23T16:50:10.241 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:10 smithi148 ceph-mon[44520]: 8.3 scrub ok 2024-04-23T16:50:10.241 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:10.269 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:50:10.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:10.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:50:10.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:10 smithi103 ceph-mon[39629]: from='client.17343 v1:172.21.15.103:0/558748827' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:50:10.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:10 smithi103 ceph-mon[39629]: pgmap v1896: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.9 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 893 op/s 2024-04-23T16:50:10.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:10 smithi103 ceph-mon[39629]: 8.0 scrub starts 2024-04-23T16:50:10.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:10 smithi103 ceph-mon[39629]: 8.3 scrub starts 2024-04-23T16:50:10.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:10 smithi103 ceph-mon[39629]: osdmap e680: 12 total, 12 up, 8 in 2024-04-23T16:50:10.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:10 smithi103 ceph-mon[39629]: 8.3 scrub ok 2024-04-23T16:50:10.373 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:50:10.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:10 smithi188 ceph-mon[44101]: from='client.17343 v1:172.21.15.103:0/558748827' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:50:10.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:10 smithi188 ceph-mon[44101]: pgmap v1896: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.9 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 893 op/s 2024-04-23T16:50:10.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:10 smithi188 ceph-mon[44101]: 8.0 scrub starts 2024-04-23T16:50:10.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:10 smithi188 ceph-mon[44101]: 8.3 scrub starts 2024-04-23T16:50:10.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:10 smithi188 ceph-mon[44101]: osdmap e680: 12 total, 12 up, 8 in 2024-04-23T16:50:10.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:10 smithi188 ceph-mon[44101]: 8.3 scrub ok 2024-04-23T16:50:10.435 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:10.502 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:50:10.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:10.505 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:50:10.605 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:50:10.634 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:10.654 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:50:10.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:10.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:50:10.760 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:50:10.827 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:10.864 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:50:10.867 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:10.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:10.871 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:50:10.903 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-23T16:50:10.972 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:11.018 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:11.086 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:11 smithi148 ceph-mon[44520]: 8.0 scrub ok 2024-04-23T16:50:11.087 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:11 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:50:11.090 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:11 smithi188 ceph-mon[44101]: 8.0 scrub ok 2024-04-23T16:50:11.090 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:11 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:50:11.090 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:11 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:50:11.090 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:11 smithi188 ceph-mon[44101]: 1.7 scrub starts 2024-04-23T16:50:11.090 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:11 smithi188 ceph-mon[44101]: 1.7 scrub ok 2024-04-23T16:50:11.090 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:11 smithi188 ceph-mon[44101]: 8.9 scrub starts 2024-04-23T16:50:11.091 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:11 smithi188 ceph-mon[44101]: osdmap e681: 12 total, 12 up, 8 in 2024-04-23T16:50:11.091 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:11 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:50:11.091 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:11 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:50:11.091 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:11 smithi188 ceph-mon[44101]: 8.9 scrub ok 2024-04-23T16:50:11.194 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:11 smithi103 ceph-mon[39629]: 8.0 scrub ok 2024-04-23T16:50:11.194 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:11 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:50:11.194 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:11 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:50:11.195 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:11 smithi103 ceph-mon[39629]: 1.7 scrub starts 2024-04-23T16:50:11.195 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:11 smithi103 ceph-mon[39629]: 1.7 scrub ok 2024-04-23T16:50:11.195 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:11 smithi103 ceph-mon[39629]: 8.9 scrub starts 2024-04-23T16:50:11.195 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:11 smithi103 ceph-mon[39629]: osdmap e681: 12 total, 12 up, 8 in 2024-04-23T16:50:11.195 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:11 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:50:11.195 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:11 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:50:11.195 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:11 smithi103 ceph-mon[39629]: 8.9 scrub ok 2024-04-23T16:50:11.223 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1895,"stamp":"2024-04-23T16:50:06.932609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2179990072,"num_objects":33258,"num_object_clones":0,"num_object_copies":66516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33258,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217758,"num_write_kb":8032385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20727,"num_bytes_recovered":1358636096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162187,"ondisk_log_size":162187,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9363440,"kb_used_data":4267804,"kb_used_omap":242,"kb_used_meta":5095181,"kb_avail":740597776,"statfs":{"total":767960285184,"available":758372122624,"internally_reserved":0,"allocated":4370231296,"data_stored":4366259254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":247821,"internal_metadata":5217466355},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-359399424,"num_objects":-5484,"num_object_clones":0,"num_object_copies":-10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5484,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001172"},"pg_stats":[{"pgid":"8.7","version":"678'13609","reported_seq":13890,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-04-23T16:50:06.007871+0000","last_change":"2024-04-23T16:49:08.424575+0000","last_active":"2024-04-23T16:50:06.007871+0000","last_peered":"2024-04-23T16:50:06.007871+0000","last_clean":"2024-04-23T16:50:06.007871+0000","last_became_active":"2024-04-23T16:46:26.363422+0000","last_became_peered":"2024-04-23T16:46:26.363422+0000","last_unstale":"2024-04-23T16:50:06.007871+0000","last_undegraded":"2024-04-23T16:50:06.007871+0000","last_fullsized":"2024-04-23T16:50:06.007871+0000","mapping_epoch":609,"log_start":"615'3600","ondisk_log_start":"615'3600","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"663'10506","last_scrub_stamp":"2024-04-23T16:49:08.424442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:08.424442+0000","objects_scrubbed":5063,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:20.445332+0000","scrub_duration":991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127205376,"num_objects":1941,"num_object_clones":0,"num_object_copies":3882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13609,"num_write_kb":497600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"677'13661","reported_seq":13807,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-04-23T16:50:02.404192+0000","last_change":"2024-04-23T16:46:26.365698+0000","last_active":"2024-04-23T16:50:02.404192+0000","last_peered":"2024-04-23T16:50:02.404192+0000","last_clean":"2024-04-23T16:50:02.404192+0000","last_became_active":"2024-04-23T16:46:26.365309+0000","last_became_peered":"2024-04-23T16:46:26.365309+0000","last_unstale":"2024-04-23T16:50:02.404192+0000","last_undegraded":"2024-04-23T16:50:02.404192+0000","last_fullsized":"2024-04-23T16:50:02.404192+0000","mapping_epoch":609,"log_start":"615'3600","ondisk_log_start":"615'3600","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142016512,"num_objects":2167,"num_object_clones":0,"num_object_copies":4334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13661,"num_write_kb":506496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.5","version":"678'13463","reported_seq":13665,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-04-23T16:50:04.688970+0000","last_change":"2024-04-23T16:49:17.828862+0000","last_active":"2024-04-23T16:50:04.688970+0000","last_peered":"2024-04-23T16:50:04.688970+0000","last_clean":"2024-04-23T16:50:04.688970+0000","last_became_active":"2024-04-23T16:46:26.364780+0000","last_became_peered":"2024-04-23T16:46:26.364780+0000","last_unstale":"2024-04-23T16:50:04.688970+0000","last_undegraded":"2024-04-23T16:50:04.688970+0000","last_fullsized":"2024-04-23T16:50:04.688970+0000","mapping_epoch":609,"log_start":"615'3400","ondisk_log_start":"615'3400","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"665'10858","last_scrub_stamp":"2024-04-23T16:49:17.828773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:17.828773+0000","objects_scrubbed":4612,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:37.125054+0000","scrub_duration":521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130613248,"num_objects":1993,"num_object_clones":0,"num_object_copies":3986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13463,"num_write_kb":494592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"678'14062","reported_seq":24887,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-04-23T16:50:04.641382+0000","last_change":"2024-04-23T16:49:54.611980+0000","last_active":"2024-04-23T16:50:04.641382+0000","last_peered":"2024-04-23T16:50:04.641382+0000","last_clean":"2024-04-23T16:50:04.641382+0000","last_became_active":"2024-04-23T16:47:10.244722+0000","last_became_peered":"2024-04-23T16:47:10.244722+0000","last_unstale":"2024-04-23T16:50:04.641382+0000","last_undegraded":"2024-04-23T16:50:04.641382+0000","last_fullsized":"2024-04-23T16:50:04.641382+0000","mapping_epoch":623,"log_start":"616'4000","ondisk_log_start":"616'4000","created":609,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133955584,"num_objects":2044,"num_object_clones":0,"num_object_copies":4088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14062,"num_write_kb":515392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5335,"num_bytes_recovered":349634560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"678'13535","reported_seq":13733,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-04-23T16:50:04.644348+0000","last_change":"2024-04-23T16:49:25.388473+0000","last_active":"2024-04-23T16:50:04.644348+0000","last_peered":"2024-04-23T16:50:04.644348+0000","last_clean":"2024-04-23T16:50:04.644348+0000","last_became_active":"2024-04-23T16:46:26.365000+0000","last_became_peered":"2024-04-23T16:46:26.365000+0000","last_unstale":"2024-04-23T16:50:04.644348+0000","last_undegraded":"2024-04-23T16:50:04.644348+0000","last_fullsized":"2024-04-23T16:50:04.644348+0000","mapping_epoch":609,"log_start":"615'3500","ondisk_log_start":"615'3500","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"667'11355","last_scrub_stamp":"2024-04-23T16:49:25.388270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:25.388270+0000","objects_scrubbed":4236,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:40.837654+0000","scrub_duration":563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134021120,"num_objects":2045,"num_object_clones":0,"num_object_copies":4090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13535,"num_write_kb":498560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"677'13584","reported_seq":13730,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-04-23T16:50:02.462140+0000","last_change":"2024-04-23T16:46:26.365668+0000","last_active":"2024-04-23T16:50:02.462140+0000","last_peered":"2024-04-23T16:50:02.462140+0000","last_clean":"2024-04-23T16:50:02.462140+0000","last_became_active":"2024-04-23T16:46:26.364349+0000","last_became_peered":"2024-04-23T16:46:26.364349+0000","last_unstale":"2024-04-23T16:50:02.462140+0000","last_undegraded":"2024-04-23T16:50:02.462140+0000","last_fullsized":"2024-04-23T16:50:02.462140+0000","mapping_epoch":609,"log_start":"615'3500","ondisk_log_start":"615'3500","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144703488,"num_objects":2208,"num_object_clones":0,"num_object_copies":4416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13584,"num_write_kb":505344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1686,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-04-23T16:50:04.463259+0000","last_change":"2024-04-23T16:50:04.463296+0000","last_active":"2024-04-23T16:50:04.463259+0000","last_peered":"2024-04-23T16:50:04.463259+0000","last_clean":"2024-04-23T16:50:04.463259+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:50:04.463259+0000","last_undegraded":"2024-04-23T16:50:04.463259+0000","last_fullsized":"2024-04-23T16:50:04.463259+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:50:04.463176+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:50:04.463176+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:51:18.372129+0000","scrub_duration":205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.1","version":"678'13724","reported_seq":14004,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-04-23T16:50:05.971878+0000","last_change":"2024-04-23T16:49:22.266220+0000","last_active":"2024-04-23T16:50:05.971878+0000","last_peered":"2024-04-23T16:50:05.971878+0000","last_clean":"2024-04-23T16:50:05.971878+0000","last_became_active":"2024-04-23T16:46:26.362670+0000","last_became_peered":"2024-04-23T16:46:26.362670+0000","last_unstale":"2024-04-23T16:50:05.971878+0000","last_undegraded":"2024-04-23T16:50:05.971878+0000","last_fullsized":"2024-04-23T16:50:05.971878+0000","mapping_epoch":609,"log_start":"615'3700","ondisk_log_start":"615'3700","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"666'11314","last_scrub_stamp":"2024-04-23T16:49:22.266039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:22.266039+0000","objects_scrubbed":4452,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:38.560321+0000","scrub_duration":641,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133169152,"num_objects":2032,"num_object_clones":0,"num_object_copies":4064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13724,"num_write_kb":504192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1556,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-04-23T16:50:01.062106+0000","last_change":"2024-04-23T16:49:25.478623+0000","last_active":"2024-04-23T16:50:01.062106+0000","last_peered":"2024-04-23T16:50:01.062106+0000","last_clean":"2024-04-23T16:50:01.062106+0000","last_became_active":"2024-04-23T16:47:10.310322+0000","last_became_peered":"2024-04-23T16:47:10.310322+0000","last_unstale":"2024-04-23T16:50:01.062106+0000","last_undegraded":"2024-04-23T16:50:01.062106+0000","last_fullsized":"2024-04-23T16:50:01.062106+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:49:25.478506+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:49:25.478506+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:27.187808+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.0","version":"678'13258","reported_seq":23311,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-04-23T16:50:04.596146+0000","last_change":"2024-04-23T16:49:11.451708+0000","last_active":"2024-04-23T16:50:04.596146+0000","last_peered":"2024-04-23T16:50:04.596146+0000","last_clean":"2024-04-23T16:50:04.596146+0000","last_became_active":"2024-04-23T16:47:10.314835+0000","last_became_peered":"2024-04-23T16:47:10.314835+0000","last_unstale":"2024-04-23T16:50:04.596146+0000","last_undegraded":"2024-04-23T16:50:04.596146+0000","last_fullsized":"2024-04-23T16:50:04.596146+0000","mapping_epoch":623,"log_start":"614'3200","ondisk_log_start":"614'3200","created":609,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135921664,"num_objects":2074,"num_object_clones":0,"num_object_copies":4148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13258,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4950,"num_bytes_recovered":324403200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1679,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-04-23T16:50:04.470485+0000","last_change":"2024-04-23T16:50:02.829266+0000","last_active":"2024-04-23T16:50:04.470485+0000","last_peered":"2024-04-23T16:50:04.470485+0000","last_clean":"2024-04-23T16:50:04.470485+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:50:04.470485+0000","last_undegraded":"2024-04-23T16:50:04.470485+0000","last_fullsized":"2024-04-23T16:50:04.470485+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:50:02.829128+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:50:02.829128+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:51:05.108860+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"677'13639","reported_seq":13927,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-04-23T16:50:02.940265+0000","last_change":"2024-04-23T16:49:08.214265+0000","last_active":"2024-04-23T16:50:02.940265+0000","last_peered":"2024-04-23T16:50:02.940265+0000","last_clean":"2024-04-23T16:50:02.940265+0000","last_became_active":"2024-04-23T16:46:26.366359+0000","last_became_peered":"2024-04-23T16:46:26.366359+0000","last_unstale":"2024-04-23T16:50:02.940265+0000","last_undegraded":"2024-04-23T16:50:02.940265+0000","last_fullsized":"2024-04-23T16:50:02.940265+0000","mapping_epoch":609,"log_start":"615'3600","ondisk_log_start":"615'3600","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"663'10692","last_scrub_stamp":"2024-04-23T16:49:08.214052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:08.214052+0000","objects_scrubbed":5183,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:11.250982+0000","scrub_duration":1042,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144900096,"num_objects":2211,"num_object_clones":0,"num_object_copies":4422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13639,"num_write_kb":507200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1685,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-04-23T16:50:04.428884+0000","last_change":"2024-04-23T16:47:08.281155+0000","last_active":"2024-04-23T16:50:04.428884+0000","last_peered":"2024-04-23T16:50:04.428884+0000","last_clean":"2024-04-23T16:50:04.428884+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:50:04.428884+0000","last_undegraded":"2024-04-23T16:50:04.428884+0000","last_fullsized":"2024-04-23T16:50:04.428884+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:47:08.281053+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:47:08.281053+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"677'13504","reported_seq":13781,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-04-23T16:50:02.895503+0000","last_change":"2024-04-23T16:49:15.382664+0000","last_active":"2024-04-23T16:50:02.895503+0000","last_peered":"2024-04-23T16:50:02.895503+0000","last_clean":"2024-04-23T16:50:02.895503+0000","last_became_active":"2024-04-23T16:46:26.366686+0000","last_became_peered":"2024-04-23T16:46:26.366686+0000","last_unstale":"2024-04-23T16:50:02.895503+0000","last_undegraded":"2024-04-23T16:50:02.895503+0000","last_fullsized":"2024-04-23T16:50:02.895503+0000","mapping_epoch":609,"log_start":"615'3500","ondisk_log_start":"615'3500","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"665'10905","last_scrub_stamp":"2024-04-23T16:49:15.382180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:49:15.382180+0000","objects_scrubbed":4744,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:32.221200+0000","scrub_duration":1175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138805248,"num_objects":2118,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13504,"num_write_kb":499904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1269,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-04-23T16:50:04.430976+0000","last_change":"2024-04-23T16:48:28.293172+0000","last_active":"2024-04-23T16:50:04.430976+0000","last_peered":"2024-04-23T16:50:04.430976+0000","last_clean":"2024-04-23T16:50:04.430976+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:50:04.430976+0000","last_undegraded":"2024-04-23T16:50:04.430976+0000","last_fullsized":"2024-04-23T16:50:04.430976+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:28.293038+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:48:28.293038+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"678'13500","reported_seq":23961,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-04-23T16:50:04.632440+0000","last_change":"2024-04-23T16:49:53.819327+0000","last_active":"2024-04-23T16:50:04.632440+0000","last_peered":"2024-04-23T16:50:04.632440+0000","last_clean":"2024-04-23T16:50:04.632440+0000","last_became_active":"2024-04-23T16:47:10.251499+0000","last_became_peered":"2024-04-23T16:47:10.251499+0000","last_unstale":"2024-04-23T16:50:04.632440+0000","last_undegraded":"2024-04-23T16:50:04.632440+0000","last_fullsized":"2024-04-23T16:50:04.632440+0000","mapping_epoch":623,"log_start":"615'3400","ondisk_log_start":"615'3400","created":609,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129499136,"num_objects":1976,"num_object_clones":0,"num_object_copies":3952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13500,"num_write_kb":495232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5153,"num_bytes_recovered":337707008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1221,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-04-23T16:50:01.062213+0000","last_change":"2024-04-23T16:48:36.291394+0000","last_active":"2024-04-23T16:50:01.062213+0000","last_peered":"2024-04-23T16:50:01.062213+0000","last_clean":"2024-04-23T16:50:01.062213+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:50:01.062213+0000","last_undegraded":"2024-04-23T16:50:01.062213+0000","last_fullsized":"2024-04-23T16:50:01.062213+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:36.291278+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:48:36.291278+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.f","version":"677'13536","reported_seq":13682,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-04-23T16:50:02.426019+0000","last_change":"2024-04-23T16:46:26.364838+0000","last_active":"2024-04-23T16:50:02.426019+0000","last_peered":"2024-04-23T16:50:02.426019+0000","last_clean":"2024-04-23T16:50:02.426019+0000","last_became_active":"2024-04-23T16:46:26.364570+0000","last_became_peered":"2024-04-23T16:46:26.364570+0000","last_unstale":"2024-04-23T16:50:02.426019+0000","last_undegraded":"2024-04-23T16:50:02.426019+0000","last_fullsized":"2024-04-23T16:50:02.426019+0000","mapping_epoch":609,"log_start":"615'3500","ondisk_log_start":"615'3500","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140247040,"num_objects":2140,"num_object_clones":0,"num_object_copies":4280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13536,"num_write_kb":501632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1694,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-04-23T16:50:04.430073+0000","last_change":"2024-04-23T16:50:02.490277+0000","last_active":"2024-04-23T16:50:04.430073+0000","last_peered":"2024-04-23T16:50:04.430073+0000","last_clean":"2024-04-23T16:50:04.430073+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:50:04.430073+0000","last_undegraded":"2024-04-23T16:50:04.430073+0000","last_fullsized":"2024-04-23T16:50:04.430073+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:50:02.490166+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:50:02.490166+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:51:10.319881+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.e","version":"678'13502","reported_seq":13650,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-04-23T16:50:04.632931+0000","last_change":"2024-04-23T16:46:26.365834+0000","last_active":"2024-04-23T16:50:04.632931+0000","last_peered":"2024-04-23T16:50:04.632931+0000","last_clean":"2024-04-23T16:50:04.632931+0000","last_became_active":"2024-04-23T16:46:26.365596+0000","last_became_peered":"2024-04-23T16:46:26.365596+0000","last_unstale":"2024-04-23T16:50:04.632931+0000","last_undegraded":"2024-04-23T16:50:04.632931+0000","last_fullsized":"2024-04-23T16:50:04.632931+0000","mapping_epoch":609,"log_start":"615'3500","ondisk_log_start":"615'3500","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127533056,"num_objects":1946,"num_object_clones":0,"num_object_copies":3892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13502,"num_write_kb":494336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1693,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-04-23T16:50:04.464543+0000","last_change":"2024-04-23T16:48:38.816478+0000","last_active":"2024-04-23T16:50:04.464543+0000","last_peered":"2024-04-23T16:50:04.464543+0000","last_clean":"2024-04-23T16:50:04.464543+0000","last_became_active":"2024-04-23T16:47:10.266014+0000","last_became_peered":"2024-04-23T16:47:10.266014+0000","last_unstale":"2024-04-23T16:50:04.464543+0000","last_undegraded":"2024-04-23T16:50:04.464543+0000","last_fullsized":"2024-04-23T16:50:04.464543+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:38.816369+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:48:38.816369+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:06.233199+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"677'13717","reported_seq":13864,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-04-23T16:50:02.452299+0000","last_change":"2024-04-23T16:46:26.364586+0000","last_active":"2024-04-23T16:50:02.452299+0000","last_peered":"2024-04-23T16:50:02.452299+0000","last_clean":"2024-04-23T16:50:02.452299+0000","last_became_active":"2024-04-23T16:46:26.364302+0000","last_became_peered":"2024-04-23T16:46:26.364302+0000","last_unstale":"2024-04-23T16:50:02.452299+0000","last_undegraded":"2024-04-23T16:50:02.452299+0000","last_fullsized":"2024-04-23T16:50:02.452299+0000","mapping_epoch":609,"log_start":"615'3700","ondisk_log_start":"615'3700","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138805272,"num_objects":2119,"num_object_clones":0,"num_object_copies":4238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2119,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13717,"num_write_kb":506689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1569,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-04-23T16:50:01.061215+0000","last_change":"2024-04-23T16:47:10.037769+0000","last_active":"2024-04-23T16:50:01.061215+0000","last_peered":"2024-04-23T16:50:01.061215+0000","last_clean":"2024-04-23T16:50:01.061215+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:50:01.061215+0000","last_undegraded":"2024-04-23T16:50:01.061215+0000","last_fullsized":"2024-04-23T16:50:01.061215+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:47:10.037665+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:47:10.037665+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:02.407021+0000","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"678'13629","reported_seq":13777,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-04-23T16:50:04.633076+0000","last_change":"2024-04-23T16:46:26.364583+0000","last_active":"2024-04-23T16:50:04.633076+0000","last_peered":"2024-04-23T16:50:04.633076+0000","last_clean":"2024-04-23T16:50:04.633076+0000","last_became_active":"2024-04-23T16:46:26.364307+0000","last_became_peered":"2024-04-23T16:46:26.364307+0000","last_unstale":"2024-04-23T16:50:04.633076+0000","last_undegraded":"2024-04-23T16:50:04.633076+0000","last_fullsized":"2024-04-23T16:50:04.633076+0000","mapping_epoch":609,"log_start":"615'3600","ondisk_log_start":"615'3600","created":609,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134152192,"num_objects":2047,"num_object_clones":0,"num_object_copies":4094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13629,"num_write_kb":501632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1502,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-04-23T16:50:04.464477+0000","last_change":"2024-04-23T16:48:33.921605+0000","last_active":"2024-04-23T16:50:04.464477+0000","last_peered":"2024-04-23T16:50:04.464477+0000","last_clean":"2024-04-23T16:50:04.464477+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:50:04.464477+0000","last_undegraded":"2024-04-23T16:50:04.464477+0000","last_fullsized":"2024-04-23T16:50:04.464477+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:48:33.921450+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:48:33.921450+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:06.233180+0000","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.9","version":"677'13835","reported_seq":24562,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-04-23T16:50:02.432563+0000","last_change":"2024-04-23T16:49:22.445082+0000","last_active":"2024-04-23T16:50:02.432563+0000","last_peered":"2024-04-23T16:50:02.432563+0000","last_clean":"2024-04-23T16:50:02.432563+0000","last_became_active":"2024-04-23T16:47:10.431200+0000","last_became_peered":"2024-04-23T16:47:10.431200+0000","last_unstale":"2024-04-23T16:50:02.432563+0000","last_undegraded":"2024-04-23T16:50:02.432563+0000","last_fullsized":"2024-04-23T16:50:02.432563+0000","mapping_epoch":623,"log_start":"615'3800","ondisk_log_start":"615'3800","created":609,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:46:25.347995+0000","last_clean_scrub_stamp":"2024-04-23T16:46:25.347995+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143851520,"num_objects":2195,"num_object_clones":0,"num_object_copies":4390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13835,"num_write_kb":512960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5287,"num_bytes_recovered":346488832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1679,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-04-23T16:50:05.743061+0000","last_change":"2024-04-23T16:50:05.743103+0000","last_active":"2024-04-23T16:50:05.743061+0000","last_peered":"2024-04-23T16:50:05.743061+0000","last_clean":"2024-04-23T16:50:05.743061+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:50:05.743061+0000","last_undegraded":"2024-04-23T16:50:05.743061+0000","last_fullsized":"2024-04-23T16:50:05.743061+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:50:05.742975+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:50:05.742975+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:51:29.665460+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2179399704,"num_objects":33256,"num_object_clones":0,"num_object_copies":66512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33256,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217758,"num_write_kb":8032385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20725,"num_bytes_recovered":1358233600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5452341248,"data_stored":5452333104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160658,"ondisk_log_size":160658,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1679360,"data_stored":1661600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057057,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2003340,"kb_used_data":1110016,"kb_used_omap":1,"kb_used_meta":893310,"kb_avail":91741812,"statfs":{"total":95995035648,"available":93943615488,"internally_reserved":0,"allocated":1136656384,"data_stored":1136159983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":914749892},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351400,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613037,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1457284,"kb_used_data":794364,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":92287868,"statfs":{"total":95995035648,"available":94502776832,"internally_reserved":0,"allocated":813428736,"data_stored":812936407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":678754752},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841969,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":827088,"kb_used_data":420996,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92918064,"statfs":{"total":95995035648,"available":95148097536,"internally_reserved":0,"allocated":431099904,"data_stored":430604023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169016,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":890200,"kb_used_data":249548,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92854952,"statfs":{"total":95995035648,"available":95083470848,"internally_reserved":0,"allocated":255537152,"data_stored":255034697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":656013770},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463356,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545812,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199340,"statfs":{"total":95995035648,"available":95436124160,"internally_reserved":0,"allocated":991232,"data_stored":492849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137497,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":972180,"kb_used_data":637448,"kb_used_omap":231,"kb_used_meta":334488,"kb_avail":92772972,"statfs":{"total":95995035648,"available":94999523328,"internally_reserved":0,"allocated":652746752,"data_stored":652248369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":342516624},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738842,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1122692,"kb_used_data":397372,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92622460,"statfs":{"total":95995035648,"available":94845399040,"internally_reserved":0,"allocated":406908928,"data_stored":406416599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269512,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2658584756263,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921220,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1544844,"kb_used_data":657092,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92200308,"statfs":{"total":95995035648,"available":94413115392,"internally_reserved":0,"allocated":672862208,"data_stored":672366327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":909048261},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757695,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":395968512,"data_stored":395968512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":640876544,"data_stored":640876544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":664666112,"data_stored":664666112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1138294784,"data_stored":1138294784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":253169664,"data_stored":253165592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":424148992,"data_stored":424148992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":803012608,"data_stored":803012608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1132204032,"data_stored":1132199960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:50:11.224 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-23T16:50:11.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:11 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:50:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:11 smithi148 ceph-mon[44520]: 1.7 scrub starts 2024-04-23T16:50:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:11 smithi148 ceph-mon[44520]: 1.7 scrub ok 2024-04-23T16:50:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:11 smithi148 ceph-mon[44520]: 8.9 scrub starts 2024-04-23T16:50:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:11 smithi148 ceph-mon[44520]: osdmap e681: 12 total, 12 up, 8 in 2024-04-23T16:50:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:11 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:50:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:11 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:50:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:11 smithi148 ceph-mon[44520]: 8.9 scrub ok 2024-04-23T16:50:11.474 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:11.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:11.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:11.579 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:50:11.608 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:11.813 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:50:11.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:11.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:50:11.918 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:50:11.965 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:11.994 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:50:11.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:11.998 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:50:12.091 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:12 smithi103 ceph-mon[39629]: pgmap v1899: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 1.8 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 818 op/s 2024-04-23T16:50:12.092 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:12 smithi103 ceph-mon[39629]: 8.b scrub starts 2024-04-23T16:50:12.092 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:12 smithi103 ceph-mon[39629]: 8.b scrub ok 2024-04-23T16:50:12.101 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:50:12.130 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:12.150 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:50:12.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:12.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:50:12.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:12 smithi188 ceph-mon[44101]: pgmap v1899: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 1.8 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 818 op/s 2024-04-23T16:50:12.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:12 smithi188 ceph-mon[44101]: 8.b scrub starts 2024-04-23T16:50:12.226 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:12 smithi188 ceph-mon[44101]: 8.b scrub ok 2024-04-23T16:50:12.257 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:50:12.316 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:12.347 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:50:12.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:12.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:50:12.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:12 smithi148 ceph-mon[44520]: pgmap v1899: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 1.8 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 818 op/s 2024-04-23T16:50:12.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:12 smithi148 ceph-mon[44520]: 8.b scrub starts 2024-04-23T16:50:12.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:12 smithi148 ceph-mon[44520]: 8.b scrub ok 2024-04-23T16:50:12.452 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:50:12.479 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:12.499 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:50:12.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:12.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:50:12.607 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:50:12.636 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:12.657 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:50:12.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:12.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:50:12.761 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:50:12.816 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:12.844 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:50:12.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:12.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:50:12.954 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:50:12.982 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:13.008 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:50:13.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:13.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:50:13.101 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:13 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:50:13.101 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:13 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:50:13.117 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:50:13.165 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:13.169 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:13 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:50:13.169 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:13 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:50:13.212 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:50:13.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:13.219 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:50:13.320 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:13.357 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:13.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:13 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:50:13.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:13 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:50:13.558 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:13.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:13.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:13.662 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:50:13.717 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:13.785 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:50:13.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:13.790 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:50:13.823 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:13.848 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:50:13.873 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:50:13.891 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:50:13.894 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:50:13.914 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:50:13.918 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:13.936 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:50:13.937 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:50:13.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:13.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:50:13.956 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:50:13.978 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:50:14.005 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:50:14.029 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:50:14.042 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:14.082 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:50:14.084 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:14.114 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:50:14.152 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:50:14.153 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:50:14.153 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:50:14.153 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:50:14.154 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:50:14.154 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:50:14.154 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:50:14.154 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:50:14.154 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:50:14.154 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:50:14.154 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:50:14.154 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:50:14.154 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:50:14.345 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:14.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:14.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:14 smithi103 ceph-mon[39629]: pgmap v1900: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 978 op/s 2024-04-23T16:50:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:14 smithi103 ceph-mon[39629]: 8.6 scrub starts 2024-04-23T16:50:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:14 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1292620124' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:50:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:14 smithi103 ceph-mon[39629]: 8.6 scrub ok 2024-04-23T16:50:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:14.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:14 smithi148 ceph-mon[44520]: pgmap v1900: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 978 op/s 2024-04-23T16:50:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:14 smithi148 ceph-mon[44520]: 8.6 scrub starts 2024-04-23T16:50:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:14 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1292620124' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:50:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:14 smithi148 ceph-mon[44520]: 8.6 scrub ok 2024-04-23T16:50:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:14.450 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:50:14.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:14 smithi188 ceph-mon[44101]: pgmap v1900: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 978 op/s 2024-04-23T16:50:14.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:14 smithi188 ceph-mon[44101]: 8.6 scrub starts 2024-04-23T16:50:14.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:14 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1292620124' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:50:14.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:14 smithi188 ceph-mon[44101]: 8.6 scrub ok 2024-04-23T16:50:14.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:14.480 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:14.505 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:50:14.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:14.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:50:14.610 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:14.643 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:15.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:15 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:50:15.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:15 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:50:15.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:15 smithi103 ceph-mon[39629]: osdmap e682: 12 total, 12 up, 8 in 2024-04-23T16:50:15.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:15 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:15 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/204576709' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:50:15.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:15 smithi103 ceph-mon[39629]: pgmap v1902: 27 pgs: 27 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:50:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:15 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:50:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:15 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:50:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:15 smithi148 ceph-mon[44520]: osdmap e682: 12 total, 12 up, 8 in 2024-04-23T16:50:15.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:15 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:15 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/204576709' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:50:15.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:15 smithi148 ceph-mon[44520]: pgmap v1902: 27 pgs: 27 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:50:15.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:15 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:50:15.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:15 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:50:15.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:15 smithi188 ceph-mon[44101]: osdmap e682: 12 total, 12 up, 8 in 2024-04-23T16:50:15.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:15 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:15.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:15 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/204576709' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:50:15.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:15 smithi188 ceph-mon[44101]: pgmap v1902: 27 pgs: 27 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:50:15.510 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:50:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:16 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:50:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:16 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:50:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:16 smithi148 ceph-mon[44520]: 8.4 scrub starts 2024-04-23T16:50:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:16 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:50:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:16 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:50:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:16 smithi148 ceph-mon[44520]: 8.f scrub starts 2024-04-23T16:50:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:16 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/204576709' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:50:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:16 smithi148 ceph-mon[44520]: osdmap e683: 12 total, 12 up, 8 in 2024-04-23T16:50:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:16 smithi148 ceph-mon[44520]: 8.f scrub ok 2024-04-23T16:50:16.462 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:16 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:50:16.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:16 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:50:16.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:16 smithi103 ceph-mon[39629]: 8.4 scrub starts 2024-04-23T16:50:16.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:16 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:50:16.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:16 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:50:16.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:16 smithi103 ceph-mon[39629]: 8.f scrub starts 2024-04-23T16:50:16.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:16 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/204576709' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:50:16.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:16 smithi103 ceph-mon[39629]: osdmap e683: 12 total, 12 up, 8 in 2024-04-23T16:50:16.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:16 smithi103 ceph-mon[39629]: 8.f scrub ok 2024-04-23T16:50:16.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:16 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:50:16.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:16 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:50:16.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:16 smithi188 ceph-mon[44101]: 8.4 scrub starts 2024-04-23T16:50:16.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:16 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:50:16.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:16 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:50:16.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:16 smithi188 ceph-mon[44101]: 8.f scrub starts 2024-04-23T16:50:16.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:16 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/204576709' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:50:16.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:16 smithi188 ceph-mon[44101]: osdmap e683: 12 total, 12 up, 8 in 2024-04-23T16:50:16.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:16 smithi188 ceph-mon[44101]: 8.f scrub ok 2024-04-23T16:50:16.631 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:16.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:16.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:16.736 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:50:16.777 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:16.814 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:50:16.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:16.827 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:50:16.929 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:50:16.961 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:50:16.961 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:50:16.979 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:17.008 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:50:17.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:17.020 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:50:17.121 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:50:17.154 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:17 smithi188 ceph-mon[44101]: 8.4 scrub ok 2024-04-23T16:50:17.154 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:17 smithi188 ceph-mon[44101]: pgmap v1904: 27 pgs: 27 active+clean; 1.4 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 830 op/s 2024-04-23T16:50:17.184 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:17.214 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:50:17.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:17.219 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:50:17.320 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:50:17.357 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:17.377 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:50:17.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:17.380 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:50:17.481 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:17.509 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:17 smithi103 ceph-mon[39629]: 8.4 scrub ok 2024-04-23T16:50:17.509 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:17 smithi103 ceph-mon[39629]: pgmap v1904: 27 pgs: 27 active+clean; 1.4 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 830 op/s 2024-04-23T16:50:17.513 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:17.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:17 smithi148 ceph-mon[44520]: 8.4 scrub ok 2024-04-23T16:50:17.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:17 smithi148 ceph-mon[44520]: pgmap v1904: 27 pgs: 27 active+clean; 1.4 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 830 op/s 2024-04-23T16:50:17.755 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-23T16:50:17.765 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:17.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:17.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:17.926 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:50:17.997 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:18.042 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:50:18.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:18.048 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:50:18.150 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:50:18.206 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:50:18.214 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:18.246 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:50:18.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:18.251 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:18 smithi188 ceph-mon[44101]: 8.8 scrub starts 2024-04-23T16:50:18.251 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:18 smithi188 ceph-mon[44101]: 8.8 scrub ok 2024-04-23T16:50:18.251 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:50:18.341 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:18.352 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:50:18.402 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:18 smithi103 ceph-mon[39629]: 8.8 scrub starts 2024-04-23T16:50:18.403 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:18 smithi103 ceph-mon[39629]: 8.8 scrub ok 2024-04-23T16:50:18.404 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:50:18.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:18.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:50:18.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:18 smithi148 ceph-mon[44520]: 8.8 scrub starts 2024-04-23T16:50:18.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:18 smithi148 ceph-mon[44520]: 8.8 scrub ok 2024-04-23T16:50:18.723 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:50:18.751 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:18.771 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:50:18.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:18.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:50:18.875 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:50:18.902 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:18.921 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:50:18.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:18.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:50:19.026 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:19.054 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:19.204 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:19.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:19.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:19.244 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:19.309 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:50:19.336 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:19.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=54,logm=5} crc {auth=3528038337,config=1157536788,health=3897446751,kv=3952156989,logm=2520571032}) 2024-04-23T16:50:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4038983281}) 2024-04-23T16:50:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4165405392}) 2024-04-23T16:50:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3951797301}) 2024-04-23T16:50:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=217586413}) 2024-04-23T16:50:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2556922759}) 2024-04-23T16:50:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3908539132}) 2024-04-23T16:50:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1828462091}) 2024-04-23T16:50:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=836453571}) 2024-04-23T16:50:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2673119142}) 2024-04-23T16:50:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3366575248}) 2024-04-23T16:50:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1890824960}) 2024-04-23T16:50:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3982847333}) 2024-04-23T16:50:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2707758754}) 2024-04-23T16:50:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1429216795}) 2024-04-23T16:50:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2205256993}) 2024-04-23T16:50:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4120746046}) 2024-04-23T16:50:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2532778319}) 2024-04-23T16:50:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=996823104}) 2024-04-23T16:50:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2185061029}) 2024-04-23T16:50:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3360594257}) 2024-04-23T16:50:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1719762473}) 2024-04-23T16:50:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1856552360}) 2024-04-23T16:50:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1538739832}) 2024-04-23T16:50:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1421552180}) 2024-04-23T16:50:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=92269414}) 2024-04-23T16:50:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2323296776}) 2024-04-23T16:50:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2975602039}) 2024-04-23T16:50:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=465395356}) 2024-04-23T16:50:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3089635592}) 2024-04-23T16:50:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2634743880}) 2024-04-23T16:50:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2515208622}) 2024-04-23T16:50:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1194852437}) 2024-04-23T16:50:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3651746934}) 2024-04-23T16:50:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=503403446}) 2024-04-23T16:50:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=835402123}) 2024-04-23T16:50:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=329512260}) 2024-04-23T16:50:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=584417029}) 2024-04-23T16:50:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1743484160}) 2024-04-23T16:50:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=441470396}) 2024-04-23T16:50:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1925657108}) 2024-04-23T16:50:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2808737745}) 2024-04-23T16:50:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2013040899}) 2024-04-23T16:50:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=840019682}) 2024-04-23T16:50:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=982496427}) 2024-04-23T16:50:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3706115099}) 2024-04-23T16:50:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1429013189}) 2024-04-23T16:50:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=739947265}) 2024-04-23T16:50:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=689006243}) 2024-04-23T16:50:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=619965460}) 2024-04-23T16:50:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1606095505}) 2024-04-23T16:50:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1644529013}) 2024-04-23T16:50:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1985732407}) 2024-04-23T16:50:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2889668072}) 2024-04-23T16:50:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=737985308}) 2024-04-23T16:50:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4182862782}) 2024-04-23T16:50:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3815252923}) 2024-04-23T16:50:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2125581442}) 2024-04-23T16:50:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2901833679}) 2024-04-23T16:50:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=439920079}) 2024-04-23T16:50:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=556460180}) 2024-04-23T16:50:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1498897453}) 2024-04-23T16:50:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=352882301}) 2024-04-23T16:50:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2769986457}) 2024-04-23T16:50:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=361268846}) 2024-04-23T16:50:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1274851794}) 2024-04-23T16:50:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=529945396}) 2024-04-23T16:50:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2364660629}) 2024-04-23T16:50:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2041562747}) 2024-04-23T16:50:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2464283913}) 2024-04-23T16:50:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3632068837}) 2024-04-23T16:50:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2380547247}) 2024-04-23T16:50:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4266697125}) 2024-04-23T16:50:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2281627193}) 2024-04-23T16:50:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=33031462}) 2024-04-23T16:50:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1633652571}) 2024-04-23T16:50:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=254092423}) 2024-04-23T16:50:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3588609190}) 2024-04-23T16:50:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1562864147}) 2024-04-23T16:50:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=219214070}) 2024-04-23T16:50:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1304000507}) 2024-04-23T16:50:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=232801363}) 2024-04-23T16:50:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=709084693}) 2024-04-23T16:50:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3280750781}) 2024-04-23T16:50:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3195839642}) 2024-04-23T16:50:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2690688791}) 2024-04-23T16:50:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=815803702}) 2024-04-23T16:50:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=88,mdsmap=3,mgr=9} crc {logm=3511517721,mdsmap=2840952586,mgr=1749075989}) 2024-04-23T16:50:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mgr=10,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=72} crc {mgr=2880060479,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=1355922433,mon_config_key=2448238741}) 2024-04-23T16:50:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2366448680}) 2024-04-23T16:50:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3588877728}) 2024-04-23T16:50:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=18,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=64} crc {mon_config_key=3721157688,monmap=2133572193,osd_metadata=3822402344,osd_pg_creating=3883549161,osdmap=737360640}) 2024-04-23T16:50:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=43} crc {osdmap=131643909}) 2024-04-23T16:50:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:19 smithi103 ceph-mon[39629]: pgmap v1905: 27 pgs: 27 active+clean; 1.3 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 933 op/s 2024-04-23T16:50:19.359 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:50:19.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:19.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:50:19.465 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:50:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=54,logm=5} crc {auth=3528038337,config=1157536788,health=3897446751,kv=3952156989,logm=2520571032}) 2024-04-23T16:50:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4038983281}) 2024-04-23T16:50:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4165405392}) 2024-04-23T16:50:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3951797301}) 2024-04-23T16:50:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=217586413}) 2024-04-23T16:50:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2556922759}) 2024-04-23T16:50:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3908539132}) 2024-04-23T16:50:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1828462091}) 2024-04-23T16:50:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=836453571}) 2024-04-23T16:50:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2673119142}) 2024-04-23T16:50:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3366575248}) 2024-04-23T16:50:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1890824960}) 2024-04-23T16:50:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3982847333}) 2024-04-23T16:50:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2707758754}) 2024-04-23T16:50:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1429216795}) 2024-04-23T16:50:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2205256993}) 2024-04-23T16:50:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4120746046}) 2024-04-23T16:50:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2532778319}) 2024-04-23T16:50:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=996823104}) 2024-04-23T16:50:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2185061029}) 2024-04-23T16:50:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3360594257}) 2024-04-23T16:50:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1719762473}) 2024-04-23T16:50:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1856552360}) 2024-04-23T16:50:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1538739832}) 2024-04-23T16:50:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1421552180}) 2024-04-23T16:50:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=92269414}) 2024-04-23T16:50:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2323296776}) 2024-04-23T16:50:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2975602039}) 2024-04-23T16:50:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=465395356}) 2024-04-23T16:50:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3089635592}) 2024-04-23T16:50:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2634743880}) 2024-04-23T16:50:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2515208622}) 2024-04-23T16:50:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1194852437}) 2024-04-23T16:50:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3651746934}) 2024-04-23T16:50:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=503403446}) 2024-04-23T16:50:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=835402123}) 2024-04-23T16:50:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=329512260}) 2024-04-23T16:50:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=584417029}) 2024-04-23T16:50:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1743484160}) 2024-04-23T16:50:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=441470396}) 2024-04-23T16:50:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1925657108}) 2024-04-23T16:50:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2808737745}) 2024-04-23T16:50:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2013040899}) 2024-04-23T16:50:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=840019682}) 2024-04-23T16:50:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=982496427}) 2024-04-23T16:50:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3706115099}) 2024-04-23T16:50:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1429013189}) 2024-04-23T16:50:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=739947265}) 2024-04-23T16:50:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=689006243}) 2024-04-23T16:50:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=619965460}) 2024-04-23T16:50:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1606095505}) 2024-04-23T16:50:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1644529013}) 2024-04-23T16:50:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1985732407}) 2024-04-23T16:50:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2889668072}) 2024-04-23T16:50:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=737985308}) 2024-04-23T16:50:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4182862782}) 2024-04-23T16:50:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3815252923}) 2024-04-23T16:50:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2125581442}) 2024-04-23T16:50:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2901833679}) 2024-04-23T16:50:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=439920079}) 2024-04-23T16:50:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=556460180}) 2024-04-23T16:50:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1498897453}) 2024-04-23T16:50:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=352882301}) 2024-04-23T16:50:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2769986457}) 2024-04-23T16:50:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=361268846}) 2024-04-23T16:50:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1274851794}) 2024-04-23T16:50:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=529945396}) 2024-04-23T16:50:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2364660629}) 2024-04-23T16:50:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2041562747}) 2024-04-23T16:50:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2464283913}) 2024-04-23T16:50:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3632068837}) 2024-04-23T16:50:19.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2380547247}) 2024-04-23T16:50:19.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4266697125}) 2024-04-23T16:50:19.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2281627193}) 2024-04-23T16:50:19.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=33031462}) 2024-04-23T16:50:19.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1633652571}) 2024-04-23T16:50:19.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=254092423}) 2024-04-23T16:50:19.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3588609190}) 2024-04-23T16:50:19.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1562864147}) 2024-04-23T16:50:19.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=219214070}) 2024-04-23T16:50:19.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1304000507}) 2024-04-23T16:50:19.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=232801363}) 2024-04-23T16:50:19.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=709084693}) 2024-04-23T16:50:19.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3280750781}) 2024-04-23T16:50:19.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3195839642}) 2024-04-23T16:50:19.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2690688791}) 2024-04-23T16:50:19.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=815803702}) 2024-04-23T16:50:19.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=88,mdsmap=3,mgr=9} crc {logm=3511517721,mdsmap=2840952586,mgr=1749075989}) 2024-04-23T16:50:19.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mgr=10,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=72} crc {mgr=2880060479,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=1355922433,mon_config_key=2448238741}) 2024-04-23T16:50:19.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2366448680}) 2024-04-23T16:50:19.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3588877728}) 2024-04-23T16:50:19.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=18,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=64} crc {mon_config_key=3721157688,monmap=2133572193,osd_metadata=3822402344,osd_pg_creating=3883549161,osdmap=737360640}) 2024-04-23T16:50:19.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=43} crc {osdmap=131643909}) 2024-04-23T16:50:19.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:19 smithi188 ceph-mon[44101]: pgmap v1905: 27 pgs: 27 active+clean; 1.3 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 933 op/s 2024-04-23T16:50:19.491 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=54,logm=5} crc {auth=3528038337,config=1157536788,health=3897446751,kv=3952156989,logm=2520571032}) 2024-04-23T16:50:19.492 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4038983281}) 2024-04-23T16:50:19.492 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4165405392}) 2024-04-23T16:50:19.492 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3951797301}) 2024-04-23T16:50:19.492 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=217586413}) 2024-04-23T16:50:19.492 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2556922759}) 2024-04-23T16:50:19.492 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3908539132}) 2024-04-23T16:50:19.493 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1828462091}) 2024-04-23T16:50:19.493 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=836453571}) 2024-04-23T16:50:19.493 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2673119142}) 2024-04-23T16:50:19.493 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3366575248}) 2024-04-23T16:50:19.493 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1890824960}) 2024-04-23T16:50:19.493 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3982847333}) 2024-04-23T16:50:19.494 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2707758754}) 2024-04-23T16:50:19.494 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1429216795}) 2024-04-23T16:50:19.494 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2205256993}) 2024-04-23T16:50:19.494 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4120746046}) 2024-04-23T16:50:19.494 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2532778319}) 2024-04-23T16:50:19.495 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=996823104}) 2024-04-23T16:50:19.495 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2185061029}) 2024-04-23T16:50:19.495 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3360594257}) 2024-04-23T16:50:19.495 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1719762473}) 2024-04-23T16:50:19.495 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1856552360}) 2024-04-23T16:50:19.495 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1538739832}) 2024-04-23T16:50:19.496 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1421552180}) 2024-04-23T16:50:19.496 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=92269414}) 2024-04-23T16:50:19.496 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2323296776}) 2024-04-23T16:50:19.496 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2975602039}) 2024-04-23T16:50:19.496 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=465395356}) 2024-04-23T16:50:19.496 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3089635592}) 2024-04-23T16:50:19.497 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2634743880}) 2024-04-23T16:50:19.497 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2515208622}) 2024-04-23T16:50:19.497 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1194852437}) 2024-04-23T16:50:19.497 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3651746934}) 2024-04-23T16:50:19.497 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=503403446}) 2024-04-23T16:50:19.497 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=835402123}) 2024-04-23T16:50:19.498 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=329512260}) 2024-04-23T16:50:19.498 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=584417029}) 2024-04-23T16:50:19.498 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1743484160}) 2024-04-23T16:50:19.498 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=441470396}) 2024-04-23T16:50:19.498 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1925657108}) 2024-04-23T16:50:19.498 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2808737745}) 2024-04-23T16:50:19.499 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2013040899}) 2024-04-23T16:50:19.499 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=840019682}) 2024-04-23T16:50:19.499 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=982496427}) 2024-04-23T16:50:19.499 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3706115099}) 2024-04-23T16:50:19.499 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1429013189}) 2024-04-23T16:50:19.499 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=739947265}) 2024-04-23T16:50:19.500 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=689006243}) 2024-04-23T16:50:19.500 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=619965460}) 2024-04-23T16:50:19.500 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1606095505}) 2024-04-23T16:50:19.500 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1644529013}) 2024-04-23T16:50:19.500 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1985732407}) 2024-04-23T16:50:19.500 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2889668072}) 2024-04-23T16:50:19.501 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=737985308}) 2024-04-23T16:50:19.501 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4182862782}) 2024-04-23T16:50:19.501 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3815252923}) 2024-04-23T16:50:19.501 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2125581442}) 2024-04-23T16:50:19.501 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2901833679}) 2024-04-23T16:50:19.501 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=439920079}) 2024-04-23T16:50:19.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=556460180}) 2024-04-23T16:50:19.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1498897453}) 2024-04-23T16:50:19.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=352882301}) 2024-04-23T16:50:19.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2769986457}) 2024-04-23T16:50:19.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=361268846}) 2024-04-23T16:50:19.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1274851794}) 2024-04-23T16:50:19.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=529945396}) 2024-04-23T16:50:19.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2364660629}) 2024-04-23T16:50:19.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2041562747}) 2024-04-23T16:50:19.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2464283913}) 2024-04-23T16:50:19.502 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3632068837}) 2024-04-23T16:50:19.503 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2380547247}) 2024-04-23T16:50:19.503 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4266697125}) 2024-04-23T16:50:19.503 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2281627193}) 2024-04-23T16:50:19.503 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=33031462}) 2024-04-23T16:50:19.503 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1633652571}) 2024-04-23T16:50:19.503 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=254092423}) 2024-04-23T16:50:19.503 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3588609190}) 2024-04-23T16:50:19.503 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1562864147}) 2024-04-23T16:50:19.503 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=219214070}) 2024-04-23T16:50:19.503 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1304000507}) 2024-04-23T16:50:19.503 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=232801363}) 2024-04-23T16:50:19.503 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=709084693}) 2024-04-23T16:50:19.503 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3280750781}) 2024-04-23T16:50:19.503 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3195839642}) 2024-04-23T16:50:19.503 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2690688791}) 2024-04-23T16:50:19.504 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=815803702}) 2024-04-23T16:50:19.504 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=88,mdsmap=3,mgr=9} crc {logm=3511517721,mdsmap=2840952586,mgr=1749075989}) 2024-04-23T16:50:19.504 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mgr=10,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=72} crc {mgr=2880060479,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=1355922433,mon_config_key=2448238741}) 2024-04-23T16:50:19.504 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2366448680}) 2024-04-23T16:50:19.504 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3588877728}) 2024-04-23T16:50:19.504 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=18,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=64} crc {mon_config_key=3721157688,monmap=2133572193,osd_metadata=3822402344,osd_pg_creating=3883549161,osdmap=737360640}) 2024-04-23T16:50:19.504 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=43} crc {osdmap=131643909}) 2024-04-23T16:50:19.504 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:19 smithi148 ceph-mon[44520]: pgmap v1905: 27 pgs: 27 active+clean; 1.3 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 933 op/s 2024-04-23T16:50:19.505 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:19.514 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:50:19.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:19.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:50:19.621 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:19.654 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:20.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:20 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:50:20.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:20 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:50:20.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:20 smithi103 ceph-mon[39629]: osdmap e684: 12 total, 12 up, 8 in 2024-04-23T16:50:20.278 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:50:20.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:20 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:50:20.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:20 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:50:20.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:20 smithi188 ceph-mon[44101]: osdmap e684: 12 total, 12 up, 8 in 2024-04-23T16:50:20.605 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:20.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:20.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:20.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:20 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:50:20.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:20 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:50:20.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:20 smithi148 ceph-mon[44520]: osdmap e684: 12 total, 12 up, 8 in 2024-04-23T16:50:20.710 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:20.739 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:20.850 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:20.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:20.954 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:50:21.010 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:21.040 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:50:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:21.045 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:50:21.146 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:50:21.179 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:21.429 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:50:21.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:21.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:50:21.435 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:21 smithi103 ceph-mon[39629]: pgmap v1907: 27 pgs: 27 active+clean; 1.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-04-23T16:50:21.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:21 smithi188 ceph-mon[44101]: pgmap v1907: 27 pgs: 27 active+clean; 1.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-04-23T16:50:21.535 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:21.572 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:21.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:21 smithi148 ceph-mon[44520]: pgmap v1907: 27 pgs: 27 active+clean; 1.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-04-23T16:50:21.649 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:21.649 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:21.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:21.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:21.689 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:50:21.689 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:50:21.756 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:50:21.790 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:21.917 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:50:21.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:21.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:50:22.022 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:22.054 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:22.133 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:22.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:22.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:22.237 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:50:22.291 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:22.326 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:50:22.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:22.333 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:50:22.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:22 smithi103 ceph-mon[39629]: 1.4 scrub starts 2024-04-23T16:50:22.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:22 smithi103 ceph-mon[39629]: 1.4 scrub ok 2024-04-23T16:50:22.434 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:50:22.460 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:22 smithi148 ceph-mon[44520]: 1.4 scrub starts 2024-04-23T16:50:22.460 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:22 smithi148 ceph-mon[44520]: 1.4 scrub ok 2024-04-23T16:50:22.462 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:22 smithi188 ceph-mon[44101]: 1.4 scrub starts 2024-04-23T16:50:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:22 smithi188 ceph-mon[44101]: 1.4 scrub ok 2024-04-23T16:50:22.485 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:50:22.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:22.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:50:22.589 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:50:22.653 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:22.682 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:50:22.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:22.687 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:50:22.788 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:22.820 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:23.177 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:50:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:23 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/166325380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:50:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:23 smithi188 ceph-mon[44101]: pgmap v1908: 27 pgs: 27 active+clean; 1.1 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 857 op/s 2024-04-23T16:50:23.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:23 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/166325380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:50:23.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:23 smithi103 ceph-mon[39629]: pgmap v1908: 27 pgs: 27 active+clean; 1.1 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 857 op/s 2024-04-23T16:50:23.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:23 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/166325380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:50:23.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:23 smithi148 ceph-mon[44520]: pgmap v1908: 27 pgs: 27 active+clean; 1.1 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 857 op/s 2024-04-23T16:50:24.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:24 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:50:24.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/166325380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:50:24.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:24 smithi188 ceph-mon[44101]: osdmap e685: 12 total, 12 up, 8 in 2024-04-23T16:50:24.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:24 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:50:24.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/166325380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:50:24.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:24 smithi103 ceph-mon[39629]: osdmap e685: 12 total, 12 up, 8 in 2024-04-23T16:50:24.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:24 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:50:24.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:24 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/166325380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:50:24.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:24 smithi148 ceph-mon[44520]: osdmap e685: 12 total, 12 up, 8 in 2024-04-23T16:50:24.702 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:24.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:24.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:24.733 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-23T16:50:24.809 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:24.851 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:24.985 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:24.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:24.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:25.089 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:50:25.146 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:25.177 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:50:25.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:25.180 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:50:25.281 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:50:25.311 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:25.410 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:50:25.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:25.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:50:25.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:25 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:50:25.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:25 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:50:25.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:25 smithi188 ceph-mon[44101]: pgmap v1910: 27 pgs: 27 active+clean; 908 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-23T16:50:25.513 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:25.545 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:25.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:25 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:50:25.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:25 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:50:25.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:25 smithi103 ceph-mon[39629]: pgmap v1910: 27 pgs: 27 active+clean; 908 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-23T16:50:25.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:25 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:50:25.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:25 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:50:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:25 smithi148 ceph-mon[44520]: pgmap v1910: 27 pgs: 27 active+clean; 908 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-23T16:50:26.196 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:26.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:26.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:26.300 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:50:26.327 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:26.350 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:50:26.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:26.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:50:26.454 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:26.487 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:26.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:26 smithi148 ceph-mon[44520]: osdmap e686: 12 total, 12 up, 8 in 2024-04-23T16:50:26.691 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:26.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:26.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:26.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:26 smithi188 ceph-mon[44101]: osdmap e686: 12 total, 12 up, 8 in 2024-04-23T16:50:26.724 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:26 smithi103 ceph-mon[39629]: osdmap e686: 12 total, 12 up, 8 in 2024-04-23T16:50:26.742 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:26.796 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:26.833 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:27.484 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:27.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:27.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:27.591 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:50:27.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:27 smithi103 ceph-mon[39629]: pgmap v1912: 27 pgs: 27 active+clean; 867 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 977 op/s 2024-04-23T16:50:27.629 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:50:27.650 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:50:27.663 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:27.673 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:50:27.693 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:50:27.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:27.695 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:50:27.695 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:50:27.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:27 smithi188 ceph-mon[44101]: pgmap v1912: 27 pgs: 27 active+clean; 867 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 977 op/s 2024-04-23T16:50:27.724 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:50:27.747 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:50:27.768 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:50:27.789 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:50:27.796 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:50:27.815 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:50:27.821 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:27 smithi148 ceph-mon[44520]: pgmap v1912: 27 pgs: 27 active+clean; 867 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 977 op/s 2024-04-23T16:50:27.823 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:27.842 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:50:27.844 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:50:27.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:27.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:50:27.868 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:50:27.907 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:50:27.909 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:50:27.909 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:50:27.909 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:50:27.909 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:50:27.909 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:50:27.910 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:50:27.910 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:50:27.910 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:50:27.910 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:50:27.910 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:50:27.910 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:50:27.910 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:50:27.950 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:50:28.010 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:28.044 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:50:28.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:28.049 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:50:28.150 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:50:28.199 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:28.230 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:50:28.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:28.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:50:28.335 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:28.367 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:28.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:28 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3707459306' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:50:28.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.480 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:28.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:28.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:28.588 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:28.616 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:28 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3707459306' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:50:28.616 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.616 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.616 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.616 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.616 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.616 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.616 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.616 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.617 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.617 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.617 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:28.726 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:50:28.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:28 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3707459306' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:50:28.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:28.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:29.168 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:29.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:29.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:29.272 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:29.302 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:29.431 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:29.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:29.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:29.539 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:50:29.589 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:29.619 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:50:29.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:29.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:50:29.706 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:29.726 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:50:29.765 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:50:29.765 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:50:29.765 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:50:29.765 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:242940 8.2 8:4f3d8bf1:::benchmark_data_smithi188_155546_object117431:head [delete] snapc 0=[] ondisk+write+known_if_redirected e686)", 2024-04-23T16:50:29.765 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:50:29.760243+0000", 2024-04-23T16:50:29.765 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.0041919089999999997, 2024-04-23T16:50:29.765 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042230210000000004, 2024-04-23T16:50:29.765 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:50:29.765 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:50:29.765 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:50:29.765 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:50:29.765 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:50:29.766 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:50:29.766 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 242940 2024-04-23T16:50:29.766 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:50:29.766 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:50:29.766 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:50:29.766 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:50:29.766 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:29.760243+0000", 2024-04-23T16:50:29.766 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:50:29.766 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:50:29.766 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:50:29.766 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:50:29.766 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:29.760243+0000", 2024-04-23T16:50:29.766 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:50:29.767 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:50:29.767 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:50:29.767 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:50:29.767 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:29.760245+0000", 2024-04-23T16:50:29.767 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8959999999999999e-06 2024-04-23T16:50:29.767 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:50:29.767 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:50:29.767 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:50:29.767 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:29.760251+0000", 2024-04-23T16:50:29.767 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6390000000000003e-06 2024-04-23T16:50:29.767 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:50:29.767 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:50:29.767 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:50:29.767 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:29.760264+0000", 2024-04-23T16:50:29.768 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3209e-05 2024-04-23T16:50:29.768 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:50:29.768 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:50:29.768 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:50:29.768 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:29.760291+0000", 2024-04-23T16:50:29.768 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6239000000000001e-05 2024-04-23T16:50:29.768 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:50:29.768 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:50:29.768 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:50:29.768 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:29.760356+0000", 2024-04-23T16:50:29.768 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5172999999999994e-05 2024-04-23T16:50:29.768 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:50:29.768 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:50:29.768 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:50:29.769 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:29.760495+0000", 2024-04-23T16:50:29.769 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013924299999999999 2024-04-23T16:50:29.769 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:50:29.769 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:50:29.769 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:50:29.769 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:29.760622+0000", 2024-04-23T16:50:29.769 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012729899999999999 2024-04-23T16:50:29.769 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:50:29.769 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:50:29.769 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:50:29.769 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:29.764380+0000", 2024-04-23T16:50:29.769 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037576110000000001 2024-04-23T16:50:29.769 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:50:29.770 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:50:29.770 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:50:29.770 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:29.764437+0000", 2024-04-23T16:50:29.770 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7420000000000003e-05 2024-04-23T16:50:29.770 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:50:29.770 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:50:29.770 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:50:29.770 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:29.764475+0000", 2024-04-23T16:50:29.770 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8098e-05 2024-04-23T16:50:29.770 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:50:29.770 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:50:29.770 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:50:29.770 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:50:29.770 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:50:29.771 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 1 2024-04-23T16:50:29.771 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:50:29.774 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:29.774 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:29.774 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:29 smithi188 ceph-mon[44101]: pgmap v1913: 27 pgs: 27 active+clean; 728 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 972 op/s 2024-04-23T16:50:29.774 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:29.786 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:29.787 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:29.787 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:29 smithi103 ceph-mon[39629]: pgmap v1913: 27 pgs: 27 active+clean; 728 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 972 op/s 2024-04-23T16:50:29.809 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:50:29.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:29.814 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:50:29.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:29.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:29 smithi148 ceph-mon[44520]: pgmap v1913: 27 pgs: 27 active+clean; 728 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 972 op/s 2024-04-23T16:50:29.915 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:50:29.973 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:30.002 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:50:30.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:30.010 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:50:30.111 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:30.146 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:30.235 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:30.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:30.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:30.340 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:30.370 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:31.182 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:50:31.221 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:31.225 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-04-23T16:50:31.225 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 4, 5, 6, 7, 8, 9, 10, 11, 1, 2, 3] 2024-04-23T16:50:31.225 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-23T16:50:31.225 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.4 2024-04-23T16:50:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:31.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:31.243 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-23T16:50:31.327 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:50:31.362 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:31.386 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:50:31.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:31.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:50:31.490 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:50:31.517 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:50:31 smithi148 systemd[1]: Stopping Ceph osd.4 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T16:50:31.519 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:31.621 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:50:31.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:31.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:50:31.726 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:50:31.776 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:31.805 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:50:31.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:31.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:50:31.873 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:50:31 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[49826]: 2024-04-23T16:50:31.618+0000 7f98bab67700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T16:50:31.874 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:50:31 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[49826]: 2024-04-23T16:50:31.618+0000 7f98bab67700 -1 osd.4 686 *** Got signal Terminated *** 2024-04-23T16:50:31.874 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:50:31 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[49826]: 2024-04-23T16:50:31.618+0000 7f98bab67700 -1 osd.4 686 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T16:50:31.911 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:50:31.983 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:32.013 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:50:32.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:32.020 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:50:32.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:31 smithi103 ceph-mon[39629]: pgmap v1914: 27 pgs: 27 active+clean; 593 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 976 op/s 2024-04-23T16:50:32.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:31 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:50:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:31 smithi103 ceph-mon[39629]: osd.4 marked itself down and dead 2024-04-23T16:50:32.121 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:50:32.145 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:31 smithi188 ceph-mon[44101]: pgmap v1914: 27 pgs: 27 active+clean; 593 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 976 op/s 2024-04-23T16:50:32.145 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:31 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:50:32.145 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:31 smithi188 ceph-mon[44101]: osd.4 marked itself down and dead 2024-04-23T16:50:32.147 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:32.166 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:50:32.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:32.169 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:50:32.270 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:50:32.296 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:31 smithi148 ceph-mon[44520]: pgmap v1914: 27 pgs: 27 active+clean; 593 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 976 op/s 2024-04-23T16:50:32.296 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:31 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:50:32.296 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:31 smithi148 ceph-mon[44520]: osd.4 marked itself down and dead 2024-04-23T16:50:32.297 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:32.318 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:50:32.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:32.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:50:32.421 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:50:32.448 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:32.466 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:50:32.467 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:32.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:32.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:50:32.570 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:32.600 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:32.750 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:32.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:32.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:32.854 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:50:32.887 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:32.908 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:50:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:32.911 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:50:33.007 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:50:33.012 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:50:33.042 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:33.065 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:50:33.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:33.069 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:50:33.170 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:50:33.198 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:33.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:32 smithi188 ceph-mon[44101]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T16:50:33.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:32 smithi188 ceph-mon[44101]: osdmap e687: 12 total, 11 up, 8 in 2024-04-23T16:50:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:32 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:50:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:32 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:50:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:32 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:50:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:32 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:50:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:32 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:50:33.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:32 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4219743692' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:50:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:32 smithi103 ceph-mon[39629]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T16:50:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:32 smithi103 ceph-mon[39629]: osdmap e687: 12 total, 11 up, 8 in 2024-04-23T16:50:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:32 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:50:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:32 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:50:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:50:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:50:33.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:50:33.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:32 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4219743692' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:50:33.374 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:50:33 smithi148 bash[166154]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4 2024-04-23T16:50:33.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:32 smithi148 ceph-mon[44520]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T16:50:33.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:32 smithi148 ceph-mon[44520]: osdmap e687: 12 total, 11 up, 8 in 2024-04-23T16:50:33.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:32 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:50:33.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:32 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:50:33.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:32 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:50:33.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:32 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:50:33.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:32 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:50:33.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:32 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4219743692' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:50:33.390 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:50:33.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:33.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:50:33.495 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:50:33.527 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:33.548 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:50:33.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:33.552 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:50:33.653 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:50:33.682 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:33.702 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:50:33.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:33.705 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:50:33.806 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:50:33.838 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:33.864 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:50:33.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:33.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:50:33.919 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:33.967 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:50:33.994 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:34.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:33 smithi103 ceph-mon[39629]: pgmap v1916: 27 pgs: 1 stale+active+clean, 26 active+clean; 465 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 884 op/s 2024-04-23T16:50:34.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:33 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:50:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:33 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4219743692' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:50:34.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:33 smithi103 ceph-mon[39629]: osdmap e688: 12 total, 11 up, 8 in 2024-04-23T16:50:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:33 smithi148 ceph-mon[44520]: pgmap v1916: 27 pgs: 1 stale+active+clean, 26 active+clean; 465 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 884 op/s 2024-04-23T16:50:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:33 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:50:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:33 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4219743692' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:50:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:33 smithi148 ceph-mon[44520]: osdmap e688: 12 total, 11 up, 8 in 2024-04-23T16:50:34.442 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:50:34.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:34.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:50:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:33 smithi188 ceph-mon[44101]: pgmap v1916: 27 pgs: 1 stale+active+clean, 26 active+clean; 465 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 884 op/s 2024-04-23T16:50:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:33 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:50:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:33 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4219743692' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:50:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:33 smithi188 ceph-mon[44101]: osdmap e688: 12 total, 11 up, 8 in 2024-04-23T16:50:34.546 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:50:34.578 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:34.599 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:50:34.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:34.603 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:50:34.704 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:50:34.734 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:34.753 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:50:34.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:34.757 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:50:34.858 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:50:34.886 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:34.905 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:50:34.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:34.908 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:50:35.008 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:50:35.037 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:35.057 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:50:35.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:35.060 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:50:35.161 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:35.167 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:35 smithi148 ceph-mon[44520]: Health check failed: Degraded data redundancy: 1459/12222 objects degraded (11.937%), 5 pgs degraded (PG_DEGRADED) 2024-04-23T16:50:35.183 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.4.service' 2024-04-23T16:50:35.189 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:35.209 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:35.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:35.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:35.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:35 smithi188 ceph-mon[44101]: Health check failed: Degraded data redundancy: 1459/12222 objects degraded (11.937%), 5 pgs degraded (PG_DEGRADED) 2024-04-23T16:50:35.239 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T16:50:35.239 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-23T16:50:35.239 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd down 4 2024-04-23T16:50:35.312 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:50:35.341 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:35 smithi103 ceph-mon[39629]: Health check failed: Degraded data redundancy: 1459/12222 objects degraded (11.937%), 5 pgs degraded (PG_DEGRADED) 2024-04-23T16:50:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:35.362 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:50:35.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:35.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:50:35.466 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:50:35.495 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:35.515 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:50:35.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:35.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:50:35.620 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:50:35.647 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:35.666 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:50:35.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:35.669 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:50:35.673 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:50:35.673 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:50:35.673 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-23T16:50:35.673 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-23T16:50:35.673 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:50:35.770 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:50:35.799 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:35.818 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:50:35.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:35.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:50:35.863 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:50:35.922 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:50:35.950 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:36.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:36 smithi148 ceph-mon[44520]: pgmap v1918: 27 pgs: 5 undersized+degraded+peered, 1 stale+active+clean, 21 active+clean; 382 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 969 op/s; 1459/12222 objects degraded (11.937%) 2024-04-23T16:50:36.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:36 smithi103 ceph-mon[39629]: pgmap v1918: 27 pgs: 5 undersized+degraded+peered, 1 stale+active+clean, 21 active+clean; 382 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 969 op/s; 1459/12222 objects degraded (11.937%) 2024-04-23T16:50:36.399 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:50:36.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:36.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:50:36.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:36 smithi188 ceph-mon[44101]: pgmap v1918: 27 pgs: 5 undersized+degraded+peered, 1 stale+active+clean, 21 active+clean; 382 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 969 op/s; 1459/12222 objects degraded (11.937%) 2024-04-23T16:50:36.502 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:50:36.528 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:37.278 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:50:37.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:37.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:50:37.382 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:37.410 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:37.609 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:37.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:37.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:37.726 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:50:37.759 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:37.781 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:50:37.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:37.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:50:37.885 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:37.913 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:38.038 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:38.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:38.043 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:38.078 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-23T16:50:38.136 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:50:38.137 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:50:38.137 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:50:38.137 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:50:38.137 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:50:38.137 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:50:38.137 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:50:38.137 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:50:38.137 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:50:38.137 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:50:38.137 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:50:38.138 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:50:38.138 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:50:38.138 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:50:38.138 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:50:38.138 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-23T16:50:38.138 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:50:38.138 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:50:38.138 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:50:38.138 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:50:38.138 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:50:38.138 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:50:38.138 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:50:38.144 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:50:38.175 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:38.197 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:50:38.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:38.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:50:38.301 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:38.326 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:38 smithi103 ceph-mon[39629]: pgmap v1919: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 723 op/s; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:38.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:38 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1551165597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:50:38.328 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:38.365 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:38.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:38.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:38.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:38 smithi148 ceph-mon[44520]: pgmap v1919: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 723 op/s; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:38.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:38 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1551165597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:50:38.468 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:50:38.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:38 smithi188 ceph-mon[44101]: pgmap v1919: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 723 op/s; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:38.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:38 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1551165597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:50:38.500 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:38.521 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:50:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:38.525 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:50:38.626 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:50:38.655 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:38.657 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:38.678 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:50:38.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:38.681 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:50:38.782 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:50:38.809 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:39.326 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.326 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.326 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.326 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:39.709 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:40.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:40 smithi103 ceph-mon[39629]: pgmap v1920: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 453 op/s; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:40.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:40 smithi148 ceph-mon[44520]: pgmap v1920: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 453 op/s; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:40.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:40 smithi188 ceph-mon[44101]: pgmap v1920: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 453 op/s; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:41.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:41 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:50:41.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:41 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:50:41.360 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:50:41.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:41.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:50:41.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:41 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:50:41.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:41 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:50:41.464 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:50:41.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:41 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:50:41.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:41 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:50:41.497 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:41.519 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:50:41.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:41.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:50:41.623 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:41.653 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:41.775 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:41.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:41.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:41.878 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:50:41.905 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:42.191 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:42 smithi103 ceph-mon[39629]: pgmap v1921: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 333 op/s; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:42.192 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:42 smithi103 ceph-mon[39629]: osdmap e689: 12 total, 11 up, 8 in 2024-04-23T16:50:42.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:42 smithi148 ceph-mon[44520]: pgmap v1921: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 333 op/s; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:42 smithi148 ceph-mon[44520]: osdmap e689: 12 total, 11 up, 8 in 2024-04-23T16:50:42.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:42 smithi188 ceph-mon[44101]: pgmap v1921: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 333 op/s; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:42.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:42 smithi188 ceph-mon[44101]: osdmap e689: 12 total, 11 up, 8 in 2024-04-23T16:50:42.852 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:50:42.852 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:50:42.852 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:50:42.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:42.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:50:42.956 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:42.985 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:43.083 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:43.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:43.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:43.189 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:43.217 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:43.291 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:43.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:43.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:43.395 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:50:43.427 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:43.449 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:50:43.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:43.452 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:50:43.553 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:50:43.586 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:43.607 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:50:43.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:43.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:50:43.711 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:50:43.743 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:43.749 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:50:43.749 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:50:43.764 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:50:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:43.769 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:50:43.870 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:43.900 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:43.937 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:43.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:43.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:44.041 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:50:44.065 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:50:44.074 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:44.095 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:50:44.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:44.099 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:50:44.199 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:50:44.219 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4 is already down. 2024-04-23T16:50:44.227 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:44.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:44 smithi103 ceph-mon[39629]: pgmap v1923: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 159 op/s; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:44.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:44 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1038920179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:50:44.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:44 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T16:50:44.215+0000 7f135b1c7700 -1 mon.a@0(leader).osd e690 definitely_dead 0 2024-04-23T16:50:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:44 smithi148 ceph-mon[44520]: pgmap v1923: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 159 op/s; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:44 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1038920179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:50:44.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:44 smithi188 ceph-mon[44101]: pgmap v1923: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 159 op/s; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:44 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1038920179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:50:44.576 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:50:44.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:44.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:50:44.679 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:50:44.712 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:44.733 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:50:44.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:44.736 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:50:44.837 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:50:44.853 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:50:44.867 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:44.887 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:50:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:44.890 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:50:44.991 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:50:45.018 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:45.117 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:50:45.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:45.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:50:45.208 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:45 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:50:45.209 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:45 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1038920179' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:50:45.209 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:45 smithi103 ceph-mon[39629]: osdmap e690: 12 total, 11 up, 8 in 2024-04-23T16:50:45.209 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:45 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/195479500' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["4"]}]: dispatch 2024-04-23T16:50:45.209 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:45 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 1796/11718 objects degraded (15.327%), 6 pgs degraded (PG_DEGRADED) 2024-04-23T16:50:45.221 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:50:45.252 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:45.273 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:50:45.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:45.276 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:50:45.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:45 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:50:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:45 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1038920179' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:50:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:45 smithi148 ceph-mon[44520]: osdmap e690: 12 total, 11 up, 8 in 2024-04-23T16:50:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:45 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/195479500' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["4"]}]: dispatch 2024-04-23T16:50:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:45 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 1796/11718 objects degraded (15.327%), 6 pgs degraded (PG_DEGRADED) 2024-04-23T16:50:45.377 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:50:45.409 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:45.430 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:50:45.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:45.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:50:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:45 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:50:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:45 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1038920179' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:50:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:45 smithi188 ceph-mon[44101]: osdmap e690: 12 total, 11 up, 8 in 2024-04-23T16:50:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:45 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/195479500' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["4"]}]: dispatch 2024-04-23T16:50:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:45 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 1796/11718 objects degraded (15.327%), 6 pgs degraded (PG_DEGRADED) 2024-04-23T16:50:45.535 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:50:45.564 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:45.572 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-23T16:50:45.572 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:45.584 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:50:45.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:45.587 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:50:45.688 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:50:45.716 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:45.736 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:50:45.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:45.740 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:50:45.841 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:50:45.842 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:50:45.871 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:45.890 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:50:45.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:45.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:50:45.994 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:46.023 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:46.051 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 4, 6, 7, 8, 9, 11, 1] 2024-04-23T16:50:46.052 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd out 4 2024-04-23T16:50:46.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:46 smithi148 ceph-mon[44520]: pgmap v1925: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:46.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:46 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:50:46.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:46 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:50:46.225 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:46.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:46.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:46.279 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:46 smithi103 ceph-mon[39629]: pgmap v1925: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:46.279 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:46 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:50:46.279 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:46 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:50:46.329 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:50:46.357 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:46.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:46 smithi188 ceph-mon[44101]: pgmap v1925: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:46.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:46 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:50:46.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:46 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:50:46.859 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:50:46.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:46.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:50:46.964 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:50:46.994 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:47.014 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:50:47.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:47.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:50:47.118 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:50:47.128 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:50:47.146 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:47.152 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:47.174 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:50:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:47.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:50:47.279 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:47.428 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:47.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:47.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:47.532 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:50:47.540 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:50:47.541 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:50:47.560 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:47.567 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:47 smithi148 ceph-mon[44520]: 8.1 scrub starts 2024-04-23T16:50:47.567 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:47 smithi148 ceph-mon[44520]: 8.1 scrub ok 2024-04-23T16:50:47.567 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:47 smithi148 ceph-mon[44520]: osdmap e691: 12 total, 11 up, 8 in 2024-04-23T16:50:47.567 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:47 smithi148 ceph-mon[44520]: pgmap v1927: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:47.720 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:47 smithi103 ceph-mon[39629]: 8.1 scrub starts 2024-04-23T16:50:47.720 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:47 smithi103 ceph-mon[39629]: 8.1 scrub ok 2024-04-23T16:50:47.720 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:47 smithi103 ceph-mon[39629]: osdmap e691: 12 total, 11 up, 8 in 2024-04-23T16:50:47.720 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:47 smithi103 ceph-mon[39629]: pgmap v1927: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:47.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:47 smithi188 ceph-mon[44101]: 8.1 scrub starts 2024-04-23T16:50:47.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:47 smithi188 ceph-mon[44101]: 8.1 scrub ok 2024-04-23T16:50:47.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:47 smithi188 ceph-mon[44101]: osdmap e691: 12 total, 11 up, 8 in 2024-04-23T16:50:47.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:47 smithi188 ceph-mon[44101]: pgmap v1927: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:48.208 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:50:48.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:48.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:50:48.228 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-23T16:50:48.313 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:50:48.345 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:48.367 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:50:48.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:48.370 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:50:48.472 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:50:48.491 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:50:48.501 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:48.521 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:50:48.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:48.524 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:50:48.534 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:48.560 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:50:48.579 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:50:48.599 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:50:48.619 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:50:48.625 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:50:48.638 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:50:48.652 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:48.658 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:50:48.678 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:50:48.698 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:50:48.717 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:50:48.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:48 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1203659528' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:50:48.737 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:50:48.751 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:50:48.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:48.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:50:48.756 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:50:48.756 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:50:48.756 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:50:48.757 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:50:48.757 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:50:48.757 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-23T16:50:48.757 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:50:48.757 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:50:48.757 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:50:48.757 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:50:48.757 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:50:48.757 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:50:48.757 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:50:48.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:48 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1203659528' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:50:48.855 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:50:48.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:48 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1203659528' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:50:48.883 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:48.902 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:50:48.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:48.905 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:50:49.006 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:50:49.037 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:49.235 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:50:49.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:49.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:50:49.340 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:50:49.369 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:49.389 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:50:49.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:49.392 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:50:49.493 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:49.521 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:49.574 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:49.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:49.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:49.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:49 smithi148 ceph-mon[44520]: 8.a scrub starts 2024-04-23T16:50:49.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:49 smithi148 ceph-mon[44520]: 8.a scrub ok 2024-04-23T16:50:49.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:49 smithi148 ceph-mon[44520]: pgmap v1928: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:49.691 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:50:49.721 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.721 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.721 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.721 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.722 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:49 smithi188 ceph-mon[44101]: 8.a scrub starts 2024-04-23T16:50:49.722 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.722 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:49 smithi188 ceph-mon[44101]: 8.a scrub ok 2024-04-23T16:50:49.722 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.722 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.722 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.722 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.722 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.722 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.722 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:49 smithi188 ceph-mon[44101]: pgmap v1928: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:49.723 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:49.745 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:50:49.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:49.748 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:50:49.849 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:49.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:49 smithi103 ceph-mon[39629]: 8.a scrub starts 2024-04-23T16:50:49.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:49 smithi103 ceph-mon[39629]: 8.a scrub ok 2024-04-23T16:50:49.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:49.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:49 smithi103 ceph-mon[39629]: pgmap v1928: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:49.879 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:50.005 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:50.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:50.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:50.026 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:50:50.026 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:50:50.112 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:50:50.146 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:50.248 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:50:50.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:50.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:50:50.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:50 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T16:50:50.024+0000 7f135b1c7700 -1 mon.a@0(leader).osd e691 definitely_dead 0 2024-04-23T16:50:50.352 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:50:50.380 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:50.400 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:50:50.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:50.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:50:50.504 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:50:50.537 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:50.558 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:50:50.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:50.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:50:50.605 INFO:teuthology.orchestra.run.smithi103.stderr:marked out osd.4. 2024-04-23T16:50:50.662 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:50:50.674 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:50 smithi103 ceph-mon[39629]: 8.7 scrub starts 2024-04-23T16:50:50.674 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:50 smithi103 ceph-mon[39629]: 8.7 scrub ok 2024-04-23T16:50:50.675 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:50 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3695887736' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-04-23T16:50:50.675 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:50 smithi103 ceph-mon[39629]: Client client.admin marked osd.4 out, after it was down for 18 seconds 2024-04-23T16:50:50.691 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:50.710 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:50:50.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:50.714 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:50:50.815 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:50:50.841 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:50 smithi188 ceph-mon[44101]: 8.7 scrub starts 2024-04-23T16:50:50.841 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:50 smithi188 ceph-mon[44101]: 8.7 scrub ok 2024-04-23T16:50:50.842 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:50 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3695887736' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-04-23T16:50:50.842 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:50 smithi188 ceph-mon[44101]: Client client.admin marked osd.4 out, after it was down for 18 seconds 2024-04-23T16:50:50.843 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:50.863 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:50:50.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:50.866 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:50:50.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:50 smithi148 ceph-mon[44520]: 8.7 scrub starts 2024-04-23T16:50:50.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:50 smithi148 ceph-mon[44520]: 8.7 scrub ok 2024-04-23T16:50:50.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:50 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3695887736' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-04-23T16:50:50.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:50 smithi148 ceph-mon[44520]: Client client.admin marked osd.4 out, after it was down for 18 seconds 2024-04-23T16:50:50.966 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:50:50.967 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:50:50.967 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2024-04-23T16:50:50.967 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2024-04-23T16:50:50.967 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:50:50.967 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:50:50.997 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:51.600 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-23T16:50:51.647 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:50:51.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:51.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:50:51.751 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:51.780 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:51.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:51 smithi103 ceph-mon[39629]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T16:50:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:51 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3695887736' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-04-23T16:50:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:51 smithi103 ceph-mon[39629]: osdmap e692: 12 total, 11 up, 7 in 2024-04-23T16:50:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:51 smithi103 ceph-mon[39629]: pgmap v1930: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:51.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:51 smithi148 ceph-mon[44520]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T16:50:51.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:51 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3695887736' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-04-23T16:50:51.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:51 smithi148 ceph-mon[44520]: osdmap e692: 12 total, 11 up, 7 in 2024-04-23T16:50:51.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:51 smithi148 ceph-mon[44520]: pgmap v1930: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:51.904 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:50:51.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:51 smithi188 ceph-mon[44101]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T16:50:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:51 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3695887736' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-04-23T16:50:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:51 smithi188 ceph-mon[44101]: osdmap e692: 12 total, 11 up, 7 in 2024-04-23T16:50:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:51 smithi188 ceph-mon[44101]: pgmap v1930: 27 pgs: 6 undersized+degraded+peered, 21 active+clean; 367 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1796/11718 objects degraded (15.327%) 2024-04-23T16:50:51.977 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:51.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:51.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:52.006 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.4 2024-04-23T16:50:52.006 INFO:tasks.thrashosds.thrasher:remote for osd 4 is ubuntu@smithi148.front.sepia.ceph.com 2024-04-23T16:50:52.006 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.4 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op list-pgs 2024-04-23T16:50:52.027 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:50:52.081 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:52.116 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:52.218 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.4/config 2024-04-23T16:50:52.314 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:52.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:52.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:52.418 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:52.446 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:52.499 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:52.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:52.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:52.603 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:50:52.606 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:52.631 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:52 smithi148 ceph-mon[44520]: osdmap e693: 12 total, 11 up, 7 in 2024-04-23T16:50:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:52 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:50:52.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:52 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:50:52.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:52 smithi188 ceph-mon[44101]: osdmap e693: 12 total, 11 up, 7 in 2024-04-23T16:50:52.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:52 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:50:52.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:52 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:50:53.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:52 smithi103 ceph-mon[39629]: osdmap e693: 12 total, 11 up, 7 in 2024-04-23T16:50:53.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:52 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:50:53.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:52 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:50:53.282 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:50:53.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:53.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:50:53.386 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:50:53.419 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:53.441 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:50:53.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:53.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:50:53.547 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:53.574 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:53.626 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:50:53.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:53 smithi188 ceph-mon[44101]: 8.b starting backfill to osd.1 from (0'0,0'0] MAX to 686'15450 2024-04-23T16:50:53.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:53 smithi188 ceph-mon[44101]: 8.0 starting backfill to osd.6 from (0'0,0'0] MAX to 686'14978 2024-04-23T16:50:53.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:53 smithi188 ceph-mon[44101]: 8.2 starting backfill to osd.9 from (0'0,0'0] MAX to 686'15231 2024-04-23T16:50:53.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:53 smithi188 ceph-mon[44101]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 686'15093 2024-04-23T16:50:53.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:53 smithi188 ceph-mon[44101]: osdmap e694: 12 total, 11 up, 7 in 2024-04-23T16:50:53.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:53 smithi188 ceph-mon[44101]: 8.5 starting backfill to osd.11 from (0'0,0'0] MAX to 686'15119 2024-04-23T16:50:53.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:53 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:50:53.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:53 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:50:53.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:53 smithi188 ceph-mon[44101]: from='client.36848 v1:172.21.15.103:0/13991202' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:50:53.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:53 smithi188 ceph-mon[44101]: from='client.36848 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:50:53.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:53 smithi188 ceph-mon[44101]: pgmap v1933: 27 pgs: 1 peering, 5 undersized+degraded+peered, 21 active+clean; 367 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1459/11718 objects degraded (12.451%) 2024-04-23T16:50:53.845 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:53 smithi103 ceph-mon[39629]: 8.b starting backfill to osd.1 from (0'0,0'0] MAX to 686'15450 2024-04-23T16:50:53.845 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:53 smithi103 ceph-mon[39629]: 8.0 starting backfill to osd.6 from (0'0,0'0] MAX to 686'14978 2024-04-23T16:50:53.845 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:53 smithi103 ceph-mon[39629]: 8.2 starting backfill to osd.9 from (0'0,0'0] MAX to 686'15231 2024-04-23T16:50:53.846 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:53 smithi103 ceph-mon[39629]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 686'15093 2024-04-23T16:50:53.846 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:53 smithi103 ceph-mon[39629]: osdmap e694: 12 total, 11 up, 7 in 2024-04-23T16:50:53.846 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:53 smithi103 ceph-mon[39629]: 8.5 starting backfill to osd.11 from (0'0,0'0] MAX to 686'15119 2024-04-23T16:50:53.846 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:53 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:50:53.846 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:53 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:50:53.846 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:53 smithi103 ceph-mon[39629]: from='client.36848 v1:172.21.15.103:0/13991202' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:50:53.846 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:53 smithi103 ceph-mon[39629]: from='client.36848 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:50:53.846 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:53 smithi103 ceph-mon[39629]: pgmap v1933: 27 pgs: 1 peering, 5 undersized+degraded+peered, 21 active+clean; 367 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1459/11718 objects degraded (12.451%) 2024-04-23T16:50:53.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:53 smithi148 ceph-mon[44520]: 8.b starting backfill to osd.1 from (0'0,0'0] MAX to 686'15450 2024-04-23T16:50:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:53 smithi148 ceph-mon[44520]: 8.0 starting backfill to osd.6 from (0'0,0'0] MAX to 686'14978 2024-04-23T16:50:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:53 smithi148 ceph-mon[44520]: 8.2 starting backfill to osd.9 from (0'0,0'0] MAX to 686'15231 2024-04-23T16:50:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:53 smithi148 ceph-mon[44520]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 686'15093 2024-04-23T16:50:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:53 smithi148 ceph-mon[44520]: osdmap e694: 12 total, 11 up, 7 in 2024-04-23T16:50:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:53 smithi148 ceph-mon[44520]: 8.5 starting backfill to osd.11 from (0'0,0'0] MAX to 686'15119 2024-04-23T16:50:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:53 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:50:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:53 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:50:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:53 smithi148 ceph-mon[44520]: from='client.36848 v1:172.21.15.103:0/13991202' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:50:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:53 smithi148 ceph-mon[44520]: from='client.36848 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:50:53.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:53 smithi148 ceph-mon[44520]: pgmap v1933: 27 pgs: 1 peering, 5 undersized+degraded+peered, 21 active+clean; 367 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1459/11718 objects degraded (12.451%) 2024-04-23T16:50:54.493 INFO:teuthology.orchestra.run.smithi148.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:50:54.772 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:50:54.772 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2024-04-23T16:50:54.772 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2024-04-23T16:50:54.772 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:50:54.773 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.773 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:180899 7.0 7:048b8431:::benchmark_data_smithi188_141483_object62218:head [delete] snapc 0=[] ondisk+write+known_if_redirected e591)", 2024-04-23T16:50:54.773 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:45:02.306963+0000", 2024-04-23T16:50:54.773 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 352.46316148599999, 2024-04-23T16:50:54.773 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.049749896000000002, 2024-04-23T16:50:54.773 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:50:54.773 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:50:54.773 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:50:54.773 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:50:54.773 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:50:54.773 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:50:54.773 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 180899 2024-04-23T16:50:54.773 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.774 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:50:54.774 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.774 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:50:54.774 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.306963+0000", 2024-04-23T16:50:54.774 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.774 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.774 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.774 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:50:54.774 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.306963+0000", 2024-04-23T16:50:54.774 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.774 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.774 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.774 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:50:54.775 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.306965+0000", 2024-04-23T16:50:54.775 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5510000000000001e-06 2024-04-23T16:50:54.775 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.775 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.775 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:50:54.775 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.306970+0000", 2024-04-23T16:50:54.775 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3890000000000004e-06 2024-04-23T16:50:54.775 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.775 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.775 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:50:54.775 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.306980+0000", 2024-04-23T16:50:54.775 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0064000000000001e-05 2024-04-23T16:50:54.775 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.776 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.776 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:50:54.776 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.306988+0000", 2024-04-23T16:50:54.776 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6939999999999995e-06 2024-04-23T16:50:54.776 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.776 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.776 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:50:54.776 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.311897+0000", 2024-04-23T16:50:54.776 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.004909353 2024-04-23T16:50:54.776 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.776 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.776 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:50:54.776 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.312018+0000", 2024-04-23T16:50:54.776 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000121011 2024-04-23T16:50:54.777 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.777 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.777 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:50:54.777 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.312135+0000", 2024-04-23T16:50:54.777 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011671300000000001 2024-04-23T16:50:54.777 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.777 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.777 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:50:54.777 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.322363+0000", 2024-04-23T16:50:54.777 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010228113 2024-04-23T16:50:54.777 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.777 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.777 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:50:54.778 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.356615+0000", 2024-04-23T16:50:54.778 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.034251860000000002 2024-04-23T16:50:54.778 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.778 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.778 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:50:54.778 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.356670+0000", 2024-04-23T16:50:54.778 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4763999999999999e-05 2024-04-23T16:50:54.778 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.778 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.778 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:50:54.778 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.356713+0000", 2024-04-23T16:50:54.778 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3383999999999997e-05 2024-04-23T16:50:54.778 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.779 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:50:54.779 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.779 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.779 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.779 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "rep_scrubmap(1.8 e563 from shard 8)", 2024-04-23T16:50:54.779 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:35.776826+0000", 2024-04-23T16:50:54.779 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 258.99329811600001, 2024-04-23T16:50:54.779 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.056699045000000003, 2024-04-23T16:50:54.779 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:50:54.779 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:50:54.779 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:50:54.779 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:50:54.779 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.779 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:50:54.780 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:35.776826+0000", 2024-04-23T16:50:54.780 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.780 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.780 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.780 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:50:54.780 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:35.776826+0000", 2024-04-23T16:50:54.780 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.780 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.780 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.780 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:50:54.780 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:35.776829+0000", 2024-04-23T16:50:54.780 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3190000000000002e-06 2024-04-23T16:50:54.780 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.780 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.781 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:50:54.781 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:35.776857+0000", 2024-04-23T16:50:54.781 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.853e-05 2024-04-23T16:50:54.781 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.781 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.781 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:50:54.781 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:35.776872+0000", 2024-04-23T16:50:54.781 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4584e-05 2024-04-23T16:50:54.781 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.781 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.781 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:50:54.781 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:35.776881+0000", 2024-04-23T16:50:54.781 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3090000000000001e-06 2024-04-23T16:50:54.781 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.782 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.782 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:50:54.782 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:35.833496+0000", 2024-04-23T16:50:54.782 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.056614386000000003 2024-04-23T16:50:54.782 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.782 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.782 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:50:54.782 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:35.833506+0000", 2024-04-23T16:50:54.782 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0883999999999999e-05 2024-04-23T16:50:54.782 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.782 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.782 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:50:54.782 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:35.833526+0000", 2024-04-23T16:50:54.782 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9032999999999999e-05 2024-04-23T16:50:54.783 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.783 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:50:54.783 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.783 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.783 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.783 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:31845 8.4 8:2d8f1695:::benchmark_data_smithi188_155546_object31844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e613)", 2024-04-23T16:50:54.783 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:45.116080+0000", 2024-04-23T16:50:54.783 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 249.65404482599999, 2024-04-23T16:50:54.783 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.050014955, 2024-04-23T16:50:54.783 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:50:54.783 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:50:54.783 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:50:54.783 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:50:54.783 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:50:54.784 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:50:54.784 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 31845 2024-04-23T16:50:54.784 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.784 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:50:54.784 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.784 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:50:54.784 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116080+0000", 2024-04-23T16:50:54.784 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.784 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.784 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.784 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:50:54.784 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116080+0000", 2024-04-23T16:50:54.784 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.784 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.784 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.785 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:50:54.785 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116081+0000", 2024-04-23T16:50:54.785 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1399999999999995e-07 2024-04-23T16:50:54.785 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.785 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.785 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:50:54.785 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116160+0000", 2024-04-23T16:50:54.785 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.9260999999999999e-05 2024-04-23T16:50:54.785 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.785 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.785 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:50:54.785 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116168+0000", 2024-04-23T16:50:54.785 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6029999999999999e-06 2024-04-23T16:50:54.786 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.786 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.786 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:50:54.786 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116174+0000", 2024-04-23T16:50:54.786 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1419999999999998e-06 2024-04-23T16:50:54.786 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.786 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.786 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:50:54.786 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116213+0000", 2024-04-23T16:50:54.786 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9607000000000002e-05 2024-04-23T16:50:54.786 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.786 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.786 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:50:54.787 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116353+0000", 2024-04-23T16:50:54.787 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013967600000000001 2024-04-23T16:50:54.787 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.787 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.787 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:50:54.787 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116485+0000", 2024-04-23T16:50:54.787 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013235499999999999 2024-04-23T16:50:54.787 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.787 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.787 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:50:54.787 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118878+0000", 2024-04-23T16:50:54.787 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002392693 2024-04-23T16:50:54.787 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.787 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.788 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:50:54.788 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.166002+0000", 2024-04-23T16:50:54.788 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.047124302999999999 2024-04-23T16:50:54.788 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.788 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.788 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:50:54.788 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.166057+0000", 2024-04-23T16:50:54.788 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4505000000000001e-05 2024-04-23T16:50:54.788 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.788 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.788 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:50:54.788 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.166095+0000", 2024-04-23T16:50:54.788 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7895999999999998e-05 2024-04-23T16:50:54.788 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.789 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:50:54.789 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.789 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.789 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.789 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:31847 8.0 8:00f95cbe:::benchmark_data_smithi188_155546_object31846:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e613)", 2024-04-23T16:50:54.789 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:45.116184+0000", 2024-04-23T16:50:54.789 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 249.65394022500001, 2024-04-23T16:50:54.789 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048306495999999997, 2024-04-23T16:50:54.789 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:50:54.789 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:50:54.789 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:50:54.789 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:50:54.789 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:50:54.789 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:50:54.790 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 31847 2024-04-23T16:50:54.790 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.790 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:50:54.790 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.790 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:50:54.790 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116184+0000", 2024-04-23T16:50:54.790 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.790 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.790 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.790 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:50:54.790 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116184+0000", 2024-04-23T16:50:54.790 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.791 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.791 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.791 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:50:54.791 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116185+0000", 2024-04-23T16:50:54.791 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0279999999999999e-06 2024-04-23T16:50:54.791 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.791 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.791 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:50:54.791 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116251+0000", 2024-04-23T16:50:54.791 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5147999999999994e-05 2024-04-23T16:50:54.791 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.791 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.791 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:50:54.791 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116257+0000", 2024-04-23T16:50:54.792 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8809999999999998e-06 2024-04-23T16:50:54.792 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.792 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.792 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:50:54.792 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116263+0000", 2024-04-23T16:50:54.792 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5799999999999999e-06 2024-04-23T16:50:54.792 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.792 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.792 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:50:54.792 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.117936+0000", 2024-04-23T16:50:54.792 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00167289 2024-04-23T16:50:54.792 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.792 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.792 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:50:54.793 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118056+0000", 2024-04-23T16:50:54.793 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000120318 2024-04-23T16:50:54.793 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.793 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.793 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:50:54.793 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118150+0000", 2024-04-23T16:50:54.793 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3642999999999998e-05 2024-04-23T16:50:54.793 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.793 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.793 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:50:54.793 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.119649+0000", 2024-04-23T16:50:54.794 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0014991480000000001 2024-04-23T16:50:54.794 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.794 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.794 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:50:54.794 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164347+0000", 2024-04-23T16:50:54.794 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.044698030999999999 2024-04-23T16:50:54.794 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.794 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.794 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:50:54.794 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164403+0000", 2024-04-23T16:50:54.794 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6314999999999999e-05 2024-04-23T16:50:54.794 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.794 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.794 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:50:54.795 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164491+0000", 2024-04-23T16:50:54.795 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7514000000000003e-05 2024-04-23T16:50:54.795 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.795 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:50:54.795 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.795 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.795 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.795 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:31849 8.4 8:24fbb2b9:::benchmark_data_smithi188_155546_object31848:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e613)", 2024-04-23T16:50:54.795 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:45.116272+0000", 2024-04-23T16:50:54.795 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 249.65385289100001, 2024-04-23T16:50:54.795 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.049955200999999998, 2024-04-23T16:50:54.795 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:50:54.795 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:50:54.795 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:50:54.796 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:50:54.796 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:50:54.796 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:50:54.796 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 31849 2024-04-23T16:50:54.796 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.796 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:50:54.796 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.796 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:50:54.796 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116272+0000", 2024-04-23T16:50:54.796 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.796 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.796 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.796 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:50:54.796 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116272+0000", 2024-04-23T16:50:54.796 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.797 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.797 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.797 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:50:54.797 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116273+0000", 2024-04-23T16:50:54.797 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6300000000000004e-07 2024-04-23T16:50:54.797 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.797 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.797 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:50:54.797 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116376+0000", 2024-04-23T16:50:54.797 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000103206 2024-04-23T16:50:54.797 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.797 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.797 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:50:54.797 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116383+0000", 2024-04-23T16:50:54.798 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4529999999999996e-06 2024-04-23T16:50:54.798 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.798 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.798 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:50:54.798 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116390+0000", 2024-04-23T16:50:54.798 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3430000000000001e-06 2024-04-23T16:50:54.798 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.798 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.798 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:50:54.798 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.117596+0000", 2024-04-23T16:50:54.798 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001206307 2024-04-23T16:50:54.798 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.798 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.798 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:50:54.799 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.117726+0000", 2024-04-23T16:50:54.799 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012993999999999999 2024-04-23T16:50:54.799 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.799 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.799 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:50:54.799 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.117814+0000", 2024-04-23T16:50:54.799 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8517999999999995e-05 2024-04-23T16:50:54.799 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.799 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.799 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:50:54.799 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118987+0000", 2024-04-23T16:50:54.799 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0011723720000000001 2024-04-23T16:50:54.799 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.799 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.799 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:50:54.800 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.166143+0000", 2024-04-23T16:50:54.800 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.047155860000000001 2024-04-23T16:50:54.800 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.800 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.800 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:50:54.800 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.166193+0000", 2024-04-23T16:50:54.800 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0800999999999997e-05 2024-04-23T16:50:54.800 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.800 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.800 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:50:54.800 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.166227+0000", 2024-04-23T16:50:54.800 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3538000000000003e-05 2024-04-23T16:50:54.800 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.800 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:50:54.801 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.801 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.801 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.801 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:31856 8.0 8:071e21bd:::benchmark_data_smithi188_155546_object31855:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e613)", 2024-04-23T16:50:54.801 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:45.118606+0000", 2024-04-23T16:50:54.801 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 249.65151895700001, 2024-04-23T16:50:54.801 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.046233700000000003, 2024-04-23T16:50:54.801 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:50:54.801 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:50:54.801 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:50:54.801 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:50:54.801 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:50:54.801 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:50:54.801 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 31856 2024-04-23T16:50:54.802 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.802 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:50:54.802 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.802 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:50:54.802 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118606+0000", 2024-04-23T16:50:54.802 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.802 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.802 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.802 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:50:54.802 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118606+0000", 2024-04-23T16:50:54.802 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.802 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.802 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.803 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:50:54.803 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118607+0000", 2024-04-23T16:50:54.803 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5060000000000001e-06 2024-04-23T16:50:54.803 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.803 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.803 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:50:54.803 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118684+0000", 2024-04-23T16:50:54.803 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6756999999999997e-05 2024-04-23T16:50:54.803 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.803 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.803 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:50:54.803 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118693+0000", 2024-04-23T16:50:54.803 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7150000000000004e-06 2024-04-23T16:50:54.803 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.804 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.804 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:50:54.804 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118699+0000", 2024-04-23T16:50:54.804 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9630000000000003e-06 2024-04-23T16:50:54.804 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.804 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.804 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:50:54.804 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118906+0000", 2024-04-23T16:50:54.804 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00020748199999999999 2024-04-23T16:50:54.804 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.804 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.804 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:50:54.804 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118994+0000", 2024-04-23T16:50:54.804 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7503000000000006e-05 2024-04-23T16:50:54.805 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.805 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.805 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:50:54.805 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.119071+0000", 2024-04-23T16:50:54.805 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7849000000000006e-05 2024-04-23T16:50:54.805 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.805 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.805 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:50:54.805 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.119784+0000", 2024-04-23T16:50:54.805 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00071264400000000002 2024-04-23T16:50:54.805 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.805 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.805 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:50:54.806 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164739+0000", 2024-04-23T16:50:54.806 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.044954820999999999 2024-04-23T16:50:54.806 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.806 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.806 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:50:54.806 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164798+0000", 2024-04-23T16:50:54.806 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9419000000000001e-05 2024-04-23T16:50:54.806 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.806 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.806 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:50:54.806 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164839+0000", 2024-04-23T16:50:54.806 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1041000000000002e-05 2024-04-23T16:50:54.806 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.806 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:50:54.807 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.807 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.807 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.807 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:31864 8.4 8:2d0e3829:::benchmark_data_smithi188_155546_object31863:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e613)", 2024-04-23T16:50:54.807 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:45.121165+0000", 2024-04-23T16:50:54.807 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 249.64895980599999, 2024-04-23T16:50:54.807 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.047666467999999997, 2024-04-23T16:50:54.807 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:50:54.807 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:50:54.807 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:50:54.807 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:50:54.807 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:50:54.807 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:50:54.807 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 31864 2024-04-23T16:50:54.808 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.808 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:50:54.808 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.808 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:50:54.808 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.121165+0000", 2024-04-23T16:50:54.808 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.808 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.808 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.808 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:50:54.808 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.121165+0000", 2024-04-23T16:50:54.808 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.808 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.808 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.809 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:50:54.809 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.121166+0000", 2024-04-23T16:50:54.809 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0899999999999999e-06 2024-04-23T16:50:54.809 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.809 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.809 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:50:54.809 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.121212+0000", 2024-04-23T16:50:54.809 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5917000000000001e-05 2024-04-23T16:50:54.809 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.809 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.809 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:50:54.809 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164332+0000", 2024-04-23T16:50:54.809 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.043120612000000003 2024-04-23T16:50:54.809 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.809 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.810 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:50:54.810 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164346+0000", 2024-04-23T16:50:54.810 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3345e-05 2024-04-23T16:50:54.810 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.810 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.810 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:50:54.810 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164419+0000", 2024-04-23T16:50:54.810 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3740999999999998e-05 2024-04-23T16:50:54.810 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.810 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.810 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:50:54.810 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164596+0000", 2024-04-23T16:50:54.810 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000176968 2024-04-23T16:50:54.811 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.811 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.811 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:50:54.811 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164747+0000", 2024-04-23T16:50:54.811 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015088600000000001 2024-04-23T16:50:54.811 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.811 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.811 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:50:54.811 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.167602+0000", 2024-04-23T16:50:54.811 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0028541859999999999 2024-04-23T16:50:54.811 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.811 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.811 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:50:54.811 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.168772+0000", 2024-04-23T16:50:54.811 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0011700230000000001 2024-04-23T16:50:54.812 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.812 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.812 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:50:54.812 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.168808+0000", 2024-04-23T16:50:54.812 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6628000000000002e-05 2024-04-23T16:50:54.812 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.812 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.812 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:50:54.812 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.168831+0000", 2024-04-23T16:50:54.812 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3071999999999999e-05 2024-04-23T16:50:54.812 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.812 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:50:54.812 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.812 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.813 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.813 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:43834 8.4 8:2ee18187:::benchmark_data_smithi188_155546_object43833:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e614)", 2024-04-23T16:50:54.813 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:49.576241+0000", 2024-04-23T16:50:54.813 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 245.19388351800001, 2024-04-23T16:50:54.813 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.056131107, 2024-04-23T16:50:54.813 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:50:54.813 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:50:54.813 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:50:54.813 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:50:54.813 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:50:54.813 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:50:54.813 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 43834 2024-04-23T16:50:54.813 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.814 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:50:54.814 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.814 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:50:54.814 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576241+0000", 2024-04-23T16:50:54.814 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.814 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.815 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.815 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:50:54.815 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576241+0000", 2024-04-23T16:50:54.815 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.815 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.815 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.815 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:50:54.815 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576243+0000", 2024-04-23T16:50:54.815 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2010000000000002e-06 2024-04-23T16:50:54.815 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.815 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.815 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:50:54.816 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576312+0000", 2024-04-23T16:50:54.816 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9154999999999998e-05 2024-04-23T16:50:54.816 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.816 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.816 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:50:54.816 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576326+0000", 2024-04-23T16:50:54.816 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3505e-05 2024-04-23T16:50:54.816 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.816 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.816 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:50:54.816 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576335+0000", 2024-04-23T16:50:54.816 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0580000000000004e-06 2024-04-23T16:50:54.816 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.817 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.817 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:50:54.817 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576390+0000", 2024-04-23T16:50:54.817 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5346000000000002e-05 2024-04-23T16:50:54.817 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.817 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.817 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:50:54.817 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576542+0000", 2024-04-23T16:50:54.817 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015117099999999999 2024-04-23T16:50:54.817 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.817 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.817 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:50:54.817 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576694+0000", 2024-04-23T16:50:54.817 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015268700000000001 2024-04-23T16:50:54.818 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.818 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.818 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:50:54.818 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.579902+0000", 2024-04-23T16:50:54.818 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032082399999999998 2024-04-23T16:50:54.818 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.818 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.818 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:50:54.818 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.632245+0000", 2024-04-23T16:50:54.818 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052342872999999998 2024-04-23T16:50:54.818 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.818 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.818 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:50:54.819 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.632313+0000", 2024-04-23T16:50:54.819 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7423999999999997e-05 2024-04-23T16:50:54.819 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.819 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.819 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:50:54.819 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.632372+0000", 2024-04-23T16:50:54.819 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9447000000000002e-05 2024-04-23T16:50:54.819 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.819 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:50:54.819 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.819 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.819 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.819 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:43838 8.4 8:2122428a:::benchmark_data_smithi188_155546_object43837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e614)", 2024-04-23T16:50:54.820 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:49.576429+0000", 2024-04-23T16:50:54.820 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 245.19369579900001, 2024-04-23T16:50:54.820 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.057184003999999997, 2024-04-23T16:50:54.820 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:50:54.820 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:50:54.820 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:50:54.820 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:50:54.820 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:50:54.820 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:50:54.820 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 43838 2024-04-23T16:50:54.820 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.820 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:50:54.820 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.821 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:50:54.821 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576429+0000", 2024-04-23T16:50:54.821 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.821 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.821 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.821 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:50:54.821 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576429+0000", 2024-04-23T16:50:54.821 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.821 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.821 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.821 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:50:54.821 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576430+0000", 2024-04-23T16:50:54.821 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2449999999999999e-06 2024-04-23T16:50:54.821 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.821 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.822 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:50:54.822 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576497+0000", 2024-04-23T16:50:54.822 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7193999999999994e-05 2024-04-23T16:50:54.822 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.822 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.822 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:50:54.822 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576507+0000", 2024-04-23T16:50:54.822 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3689999999999996e-06 2024-04-23T16:50:54.822 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.822 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.822 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:50:54.822 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576514+0000", 2024-04-23T16:50:54.822 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2150000000000004e-06 2024-04-23T16:50:54.823 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.823 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.823 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:50:54.823 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578392+0000", 2024-04-23T16:50:54.823 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00187791 2024-04-23T16:50:54.823 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.823 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.823 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:50:54.823 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578517+0000", 2024-04-23T16:50:54.823 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012536200000000001 2024-04-23T16:50:54.823 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.823 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.823 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:50:54.823 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578639+0000", 2024-04-23T16:50:54.824 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000122228 2024-04-23T16:50:54.824 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.824 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.824 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:50:54.824 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.580100+0000", 2024-04-23T16:50:54.824 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0014611660000000001 2024-04-23T16:50:54.824 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.824 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.824 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:50:54.824 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.633547+0000", 2024-04-23T16:50:54.824 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.053446395000000001 2024-04-23T16:50:54.824 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.824 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.824 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:50:54.824 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.633583+0000", 2024-04-23T16:50:54.825 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6047999999999998e-05 2024-04-23T16:50:54.825 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.825 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.825 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:50:54.825 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.633613+0000", 2024-04-23T16:50:54.825 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9872000000000001e-05 2024-04-23T16:50:54.825 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.825 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:50:54.825 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.825 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.825 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.825 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:43843 8.0 8:0e72c17a:::benchmark_data_smithi188_155546_object43842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e614)", 2024-04-23T16:50:54.825 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:49.576723+0000", 2024-04-23T16:50:54.825 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 245.19340167799999, 2024-04-23T16:50:54.826 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.057769413999999998, 2024-04-23T16:50:54.826 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:50:54.826 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:50:54.826 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:50:54.826 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:50:54.826 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:50:54.826 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:50:54.826 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 43843 2024-04-23T16:50:54.826 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.826 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:50:54.826 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.826 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:50:54.826 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576723+0000", 2024-04-23T16:50:54.827 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.827 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.827 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.827 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:50:54.827 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576723+0000", 2024-04-23T16:50:54.827 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.827 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.827 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.827 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:50:54.827 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576724+0000", 2024-04-23T16:50:54.827 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.47e-07 2024-04-23T16:50:54.827 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.827 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.827 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:50:54.827 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576753+0000", 2024-04-23T16:50:54.828 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9153e-05 2024-04-23T16:50:54.828 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.828 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.828 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:50:54.828 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576760+0000", 2024-04-23T16:50:54.828 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8619999999999997e-06 2024-04-23T16:50:54.828 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.828 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.828 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:50:54.828 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576765+0000", 2024-04-23T16:50:54.828 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3870000000000003e-06 2024-04-23T16:50:54.828 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.828 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.829 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:50:54.829 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578311+0000", 2024-04-23T16:50:54.829 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0015454690000000001 2024-04-23T16:50:54.829 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.829 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.829 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:50:54.829 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578466+0000", 2024-04-23T16:50:54.829 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000155298 2024-04-23T16:50:54.829 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.829 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.829 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:50:54.829 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578584+0000", 2024-04-23T16:50:54.829 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000118146 2024-04-23T16:50:54.829 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.830 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.830 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:50:54.830 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.581785+0000", 2024-04-23T16:50:54.830 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032011600000000002 2024-04-23T16:50:54.830 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.830 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.830 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:50:54.830 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.634392+0000", 2024-04-23T16:50:54.830 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052606762000000001 2024-04-23T16:50:54.830 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.830 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.830 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:50:54.830 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.634448+0000", 2024-04-23T16:50:54.831 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6174000000000001e-05 2024-04-23T16:50:54.831 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.831 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.831 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:50:54.831 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.634492+0000", 2024-04-23T16:50:54.831 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4156000000000002e-05 2024-04-23T16:50:54.831 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.831 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:50:54.831 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.831 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.831 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.831 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:43847 8.0 8:0211add4:::benchmark_data_smithi188_155546_object43846:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e614)", 2024-04-23T16:50:54.831 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:49.578901+0000", 2024-04-23T16:50:54.831 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 245.19122311300001, 2024-04-23T16:50:54.832 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.055695753000000001, 2024-04-23T16:50:54.832 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:50:54.832 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:50:54.832 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:50:54.832 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:50:54.832 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:50:54.832 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:50:54.832 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 43847 2024-04-23T16:50:54.832 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.832 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:50:54.832 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.832 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:50:54.832 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578901+0000", 2024-04-23T16:50:54.832 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.832 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.833 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.833 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:50:54.833 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578901+0000", 2024-04-23T16:50:54.833 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.833 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.833 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.833 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:50:54.833 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578903+0000", 2024-04-23T16:50:54.833 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0269999999999998e-06 2024-04-23T16:50:54.833 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.833 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.833 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:50:54.833 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578940+0000", 2024-04-23T16:50:54.833 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6461000000000002e-05 2024-04-23T16:50:54.834 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.834 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.834 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:50:54.834 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578952+0000", 2024-04-23T16:50:54.834 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.173e-05 2024-04-23T16:50:54.834 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.834 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.834 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:50:54.834 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578960+0000", 2024-04-23T16:50:54.834 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1319999999999994e-06 2024-04-23T16:50:54.834 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.834 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.835 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:50:54.835 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.579571+0000", 2024-04-23T16:50:54.835 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00061083999999999997 2024-04-23T16:50:54.835 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.835 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.835 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:50:54.835 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.579694+0000", 2024-04-23T16:50:54.835 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012373000000000001 2024-04-23T16:50:54.835 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.835 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.835 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:50:54.835 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.579843+0000", 2024-04-23T16:50:54.835 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014876000000000001 2024-04-23T16:50:54.835 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.836 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.836 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:50:54.836 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.581796+0000", 2024-04-23T16:50:54.836 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0019525650000000001 2024-04-23T16:50:54.836 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.836 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.836 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:50:54.836 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.634539+0000", 2024-04-23T16:50:54.836 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052743341999999999 2024-04-23T16:50:54.836 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.836 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.836 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:50:54.836 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.634576+0000", 2024-04-23T16:50:54.836 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7169999999999998e-05 2024-04-23T16:50:54.837 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.837 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.837 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:50:54.837 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.634597+0000", 2024-04-23T16:50:54.837 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0996e-05 2024-04-23T16:50:54.837 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.837 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:50:54.837 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.837 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.837 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.837 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:43848 8.0 8:0e8e0095:::benchmark_data_smithi188_155546_object43847:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e614)", 2024-04-23T16:50:54.837 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:49.578973+0000", 2024-04-23T16:50:54.837 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 245.191151179, 2024-04-23T16:50:54.838 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.055746616999999998, 2024-04-23T16:50:54.838 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:50:54.838 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:50:54.838 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:50:54.838 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:50:54.838 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:50:54.838 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:50:54.838 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 43848 2024-04-23T16:50:54.838 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.838 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:50:54.838 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.838 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:50:54.838 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578973+0000", 2024-04-23T16:50:54.838 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.839 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.839 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.839 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:50:54.839 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578973+0000", 2024-04-23T16:50:54.839 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:50:54.839 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.839 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.839 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:50:54.839 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578974+0000", 2024-04-23T16:50:54.839 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4099999999999997e-07 2024-04-23T16:50:54.839 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.839 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.839 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:50:54.839 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.579006+0000", 2024-04-23T16:50:54.840 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1572000000000002e-05 2024-04-23T16:50:54.840 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.840 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.840 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:50:54.840 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.579013+0000", 2024-04-23T16:50:54.840 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2370000000000003e-06 2024-04-23T16:50:54.840 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.840 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.840 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:50:54.840 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.579019+0000", 2024-04-23T16:50:54.840 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0610000000000001e-06 2024-04-23T16:50:54.840 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.840 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.840 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:50:54.840 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.580808+0000", 2024-04-23T16:50:54.841 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0017892139999999999 2024-04-23T16:50:54.841 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.841 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.841 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:50:54.841 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.580931+0000", 2024-04-23T16:50:54.841 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012230300000000001 2024-04-23T16:50:54.841 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.841 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.841 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:50:54.841 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.581069+0000", 2024-04-23T16:50:54.841 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013858300000000001 2024-04-23T16:50:54.841 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.841 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.841 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:50:54.841 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.581927+0000", 2024-04-23T16:50:54.842 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00085782000000000002 2024-04-23T16:50:54.842 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.842 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.842 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:50:54.842 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.634651+0000", 2024-04-23T16:50:54.842 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052723999000000001 2024-04-23T16:50:54.842 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.842 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.842 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:50:54.842 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.634685+0000", 2024-04-23T16:50:54.842 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3766999999999997e-05 2024-04-23T16:50:54.842 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:50:54.842 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:50:54.843 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:50:54.843 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.634720+0000", 2024-04-23T16:50:54.843 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5219999999999998e-05 2024-04-23T16:50:54.843 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.843 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:50:54.843 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.843 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:50:54.843 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:50:54.843 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:50:54.844 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:54 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 1459/11718 objects degraded (12.451%), 5 pgs degraded (PG_DEGRADED) 2024-04-23T16:50:54.844 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:54 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:50:54.844 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:54 smithi148 ceph-mon[44520]: from='client.36848 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:50:54.844 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:54 smithi148 ceph-mon[44520]: osdmap e695: 12 total, 11 up, 7 in 2024-04-23T16:50:54.844 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:54 smithi148 ceph-mon[44520]: 1.8 scrub starts 2024-04-23T16:50:54.844 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:54 smithi148 ceph-mon[44520]: 1.8 scrub ok 2024-04-23T16:50:54.925 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:54.927 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:54 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 1459/11718 objects degraded (12.451%), 5 pgs degraded (PG_DEGRADED) 2024-04-23T16:50:54.927 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:54 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:50:54.927 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:54 smithi103 ceph-mon[39629]: from='client.36848 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:50:54.927 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:54 smithi103 ceph-mon[39629]: osdmap e695: 12 total, 11 up, 7 in 2024-04-23T16:50:54.927 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:54 smithi103 ceph-mon[39629]: 1.8 scrub starts 2024-04-23T16:50:54.927 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:54 smithi103 ceph-mon[39629]: 1.8 scrub ok 2024-04-23T16:50:54.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:54.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:54.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:54 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 1459/11718 objects degraded (12.451%), 5 pgs degraded (PG_DEGRADED) 2024-04-23T16:50:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:54 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:50:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:54 smithi188 ceph-mon[44101]: from='client.36848 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:50:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:54 smithi188 ceph-mon[44101]: osdmap e695: 12 total, 11 up, 7 in 2024-04-23T16:50:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:54 smithi188 ceph-mon[44101]: 1.8 scrub starts 2024-04-23T16:50:54.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:54 smithi188 ceph-mon[44101]: 1.8 scrub ok 2024-04-23T16:50:55.028 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:50:55.085 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:55.130 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:50:55.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:55.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:50:55.243 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:55.273 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:55.529 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:55.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:55.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:55.635 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:50:55.665 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:55 smithi103 ceph-mon[39629]: osdmap e696: 12 total, 11 up, 7 in 2024-04-23T16:50:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:55 smithi103 ceph-mon[39629]: pgmap v1936: 27 pgs: 1 peering, 1 undersized+degraded+remapped+backfilling+peered, 25 active+clean; 366 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 2 op/s; 244/11696 objects degraded (2.086%); 19 MiB/s, 303 objects/s recovering 2024-04-23T16:50:55.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:55 smithi103 ceph-mon[39629]: 8.5 deep-scrub starts 2024-04-23T16:50:55.919 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:50:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:55.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:50:55.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:55 smithi188 ceph-mon[44101]: osdmap e696: 12 total, 11 up, 7 in 2024-04-23T16:50:55.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:55 smithi188 ceph-mon[44101]: pgmap v1936: 27 pgs: 1 peering, 1 undersized+degraded+remapped+backfilling+peered, 25 active+clean; 366 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 2 op/s; 244/11696 objects degraded (2.086%); 19 MiB/s, 303 objects/s recovering 2024-04-23T16:50:55.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:55 smithi188 ceph-mon[44101]: 8.5 deep-scrub starts 2024-04-23T16:50:56.025 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:50:56.073 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:56.093 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:55 smithi148 ceph-mon[44520]: osdmap e696: 12 total, 11 up, 7 in 2024-04-23T16:50:56.093 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:55 smithi148 ceph-mon[44520]: pgmap v1936: 27 pgs: 1 peering, 1 undersized+degraded+remapped+backfilling+peered, 25 active+clean; 366 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 2 op/s; 244/11696 objects degraded (2.086%); 19 MiB/s, 303 objects/s recovering 2024-04-23T16:50:56.093 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:55 smithi148 ceph-mon[44520]: 8.5 deep-scrub starts 2024-04-23T16:50:56.126 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:50:56.131 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-23T16:50:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:56.132 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:50:56.221 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:50:56.233 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:50:56.243 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:50:56.277 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:50:56.300 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:56.301 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:50:56.323 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:50:56.331 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:50:56.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:56.336 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:50:56.347 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:50:56.369 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:50:56.397 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:50:56.421 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:50:56.436 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:50:56.437 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:50:56.453 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:50:56.466 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:56.487 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:50:56.487 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:50:56.487 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:50:56.487 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:50:56.487 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:50:56.488 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-23T16:50:56.488 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:50:56.488 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:50:56.488 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:50:56.488 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:50:56.488 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:50:56.488 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:50:56.488 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:50:56.641 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:50:56.642 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:56 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3666034690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:50:56.642 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:56.642 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:56.642 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:56.672 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:50:56.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:56.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:50:56.729 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:56 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3666034690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:50:56.730 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:56.730 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:56.730 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:56.775 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:50:56.806 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:56.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:56 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3666034690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:50:56.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:56.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:56.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:57.256 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:50:57.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:57.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:50:57.333 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:50:57.334 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:50:57.359 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:50:57.388 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:57.409 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:57.430 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:50:57.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:57.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:50:57.540 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:50:57.596 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:57.644 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:50:57.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:57.651 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:50:57.752 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:57.784 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:57.815 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:57.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:57.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:57.926 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:57.955 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:57.955 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:57.955 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:57.955 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:57.955 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:57.956 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:57.956 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:57.956 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:57.956 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:57 smithi103 ceph-mon[39629]: pgmap v1937: 27 pgs: 1 active+clean+scrubbing+deep, 1 undersized+degraded+remapped+backfilling+peered, 25 active+clean; 350 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 48 op/s; 244/11200 objects degraded (2.179%); 18 MiB/s, 290 objects/s recovering 2024-04-23T16:50:57.956 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:57 smithi103 ceph-mon[39629]: 8.5 deep-scrub ok 2024-04-23T16:50:57.957 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:57.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:57.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:57.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:57.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:57.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:57.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:57.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:57.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:57.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:57 smithi188 ceph-mon[44101]: pgmap v1937: 27 pgs: 1 active+clean+scrubbing+deep, 1 undersized+degraded+remapped+backfilling+peered, 25 active+clean; 350 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 48 op/s; 244/11200 objects degraded (2.179%); 18 MiB/s, 290 objects/s recovering 2024-04-23T16:50:57.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:57 smithi188 ceph-mon[44101]: 8.5 deep-scrub ok 2024-04-23T16:50:58.086 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:58.087 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:58.087 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:58.087 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:58.087 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:58.087 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:58.087 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:58.087 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:50:58.087 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:57 smithi148 ceph-mon[44520]: pgmap v1937: 27 pgs: 1 active+clean+scrubbing+deep, 1 undersized+degraded+remapped+backfilling+peered, 25 active+clean; 350 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 48 op/s; 244/11200 objects degraded (2.179%); 18 MiB/s, 290 objects/s recovering 2024-04-23T16:50:58.087 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:57 smithi148 ceph-mon[44520]: 8.5 deep-scrub ok 2024-04-23T16:50:58.316 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:58.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:58.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:58.371 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:50:58.371 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:50:58.420 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:58.451 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:58.474 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:58.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:58.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:58.580 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:58.610 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:58.883 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:58 smithi103 ceph-mon[39629]: 8.2 deep-scrub starts 2024-04-23T16:50:58.962 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:58.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:58.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:58.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:58 smithi188 ceph-mon[44101]: 8.2 deep-scrub starts 2024-04-23T16:50:59.066 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:50:59.097 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:58 smithi148 ceph-mon[44520]: 8.2 deep-scrub starts 2024-04-23T16:50:59.099 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:59.121 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:50:59.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:59.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:50:59.226 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:50:59.270 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:59.335 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:50:59.398 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:50:59.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:59.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:50:59.502 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:50:59.542 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:59.577 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:50:59.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:59.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:50:59.683 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:50:59.686 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:50:59.720 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:50:59.850 INFO:teuthology.orchestra.run.smithi148.stdout:8.0 2024-04-23T16:50:59.850 INFO:teuthology.orchestra.run.smithi148.stdout:1.8 2024-04-23T16:50:59.851 INFO:teuthology.orchestra.run.smithi148.stdout:8.5 2024-04-23T16:50:59.851 INFO:teuthology.orchestra.run.smithi148.stdout:8.2 2024-04-23T16:50:59.851 INFO:teuthology.orchestra.run.smithi148.stdout:8.b 2024-04-23T16:50:59.851 INFO:teuthology.orchestra.run.smithi148.stdout:8.e 2024-04-23T16:50:59.867 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-23T16:50:59.930 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:59 smithi103 ceph-mon[39629]: 8.2 deep-scrub ok 2024-04-23T16:50:59.930 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:59 smithi103 ceph-mon[39629]: pgmap v1938: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 280 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 219 op/s; 18 MiB/s, 283 objects/s recovering 2024-04-23T16:50:59.930 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:50:59 smithi103 ceph-mon[39629]: Health check update: Degraded data redundancy: 244/11200 objects degraded (2.179%), 1 pg degraded (PG_DEGRADED) 2024-04-23T16:50:59.934 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:50:59.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:50:59.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:50:59.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:59 smithi188 ceph-mon[44101]: 8.2 deep-scrub ok 2024-04-23T16:50:59.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:59 smithi188 ceph-mon[44101]: pgmap v1938: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 280 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 219 op/s; 18 MiB/s, 283 objects/s recovering 2024-04-23T16:50:59.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:50:59 smithi188 ceph-mon[44101]: Health check update: Degraded data redundancy: 244/11200 objects degraded (2.179%), 1 pg degraded (PG_DEGRADED) 2024-04-23T16:51:00.040 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:00.094 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:00.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:59 smithi148 ceph-mon[44520]: 8.2 deep-scrub ok 2024-04-23T16:51:00.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:59 smithi148 ceph-mon[44520]: pgmap v1938: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 280 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 219 op/s; 18 MiB/s, 283 objects/s recovering 2024-04-23T16:51:00.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:50:59 smithi148 ceph-mon[44520]: Health check update: Degraded data redundancy: 244/11200 objects degraded (2.179%), 1 pg degraded (PG_DEGRADED) 2024-04-23T16:51:00.126 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:51:00.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:00.131 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:51:00.150 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:51:00.232 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:51:00.262 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:00.687 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:51:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:00 smithi103 ceph-mon[39629]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 244/11200 objects degraded (2.179%), 1 pg degraded) 2024-04-23T16:51:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:00 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:51:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:00 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2916323908' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:51:00.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:00 smithi188 ceph-mon[44101]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 244/11200 objects degraded (2.179%), 1 pg degraded) 2024-04-23T16:51:00.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:00 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:51:00.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:00 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2916323908' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:51:00.993 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:00 smithi148 ceph-mon[44520]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 244/11200 objects degraded (2.179%), 1 pg degraded) 2024-04-23T16:51:00.993 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:00 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:51:00.993 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:00 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2916323908' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:51:01.345 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Removed 125506 objects 2024-04-23T16:51:01.345 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Clean up completed and total clean up time :174.141 2024-04-23T16:51:01.517 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:51:01.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:01.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:51:01.635 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:01.657 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.4 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op export --pgid 8.5 --file /var/log/ceph/exp.8.5.4 2024-04-23T16:51:01.662 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:01.671 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:01 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:51:01.671 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:01 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2916323908' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:51:01.671 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:01 smithi103 ceph-mon[39629]: osdmap e697: 12 total, 11 up, 7 in 2024-04-23T16:51:01.671 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:01 smithi103 ceph-mon[39629]: pgmap v1940: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 177 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 413 op/s; 14 MiB/s, 223 objects/s recovering 2024-04-23T16:51:01.681 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:51:01.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:01.683 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:51:01.755 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:51:01.774 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:51:01.784 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:51:01.793 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:51:01.812 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:01.812 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:51:01.832 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:51:01.851 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:51:01.871 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:51:01.891 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:51:01.910 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:51:01.930 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:51:01.949 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:51:01.949 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:51:01.949 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:51:01.949 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:51:01.949 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:51:01.950 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-23T16:51:01.950 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:51:01.950 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:51:01.950 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:51:01.950 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:51:01.950 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:51:01.950 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:51:01.950 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:51:01.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:01 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:51:01.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:01 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2916323908' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:51:01.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:01 smithi188 ceph-mon[44101]: osdmap e697: 12 total, 11 up, 7 in 2024-04-23T16:51:01.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:01 smithi188 ceph-mon[44101]: pgmap v1940: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 177 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 413 op/s; 14 MiB/s, 223 objects/s recovering 2024-04-23T16:51:02.095 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:01 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:51:02.095 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:01 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2916323908' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:51:02.095 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:01 smithi148 ceph-mon[44520]: osdmap e697: 12 total, 11 up, 7 in 2024-04-23T16:51:02.095 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:01 smithi148 ceph-mon[44520]: pgmap v1940: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 177 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 413 op/s; 14 MiB/s, 223 objects/s recovering 2024-04-23T16:51:02.102 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.4/config 2024-04-23T16:51:02.361 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:51:02.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:02.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:51:02.465 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:02.493 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:02.511 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:51:02.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:02.514 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:51:02.615 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:51:02.644 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:02.756 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:51:02.756 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:51:02.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:02 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3209734852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:02.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:02 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.090 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:02 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3209734852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:03.090 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.090 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.090 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.090 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.090 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.091 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.091 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.091 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.091 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.091 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.091 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:02 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:02 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3209734852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:03.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:02 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:03.435 INFO:teuthology.orchestra.run.smithi148.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:51:03.673 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:51:03.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:03.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:51:03.705 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:51:03.705 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:51:03.705 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2024-04-23T16:51:03.705 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2024-04-23T16:51:03.705 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:51:03.778 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:51:03.810 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:03.831 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:51:03.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:03.835 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:51:03.938 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:51:03.943 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:03 smithi148 ceph-mon[44520]: pgmap v1941: 27 pgs: 27 active+clean; 61 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 587 op/s; 12 MiB/s, 196 objects/s recovering 2024-04-23T16:51:03.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:03 smithi188 ceph-mon[44101]: pgmap v1941: 27 pgs: 27 active+clean; 61 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 587 op/s; 12 MiB/s, 196 objects/s recovering 2024-04-23T16:51:03.967 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:03.988 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:51:03.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:03.991 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:51:04.092 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:51:04.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:03 smithi103 ceph-mon[39629]: pgmap v1941: 27 pgs: 27 active+clean; 61 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 587 op/s; 12 MiB/s, 196 objects/s recovering 2024-04-23T16:51:04.121 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:04.134 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:51:04.141 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:51:04.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:04.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:51:04.245 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:51:04.273 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:04.371 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:51:04.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:04.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:51:04.414 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:51:04.475 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:51:04.505 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:04.525 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:51:04.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:04.528 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:51:04.629 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:51:04.656 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:04.757 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:51:04.854 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:51:04.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:04.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:51:04.876 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-23T16:51:04.958 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:51:04.991 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:05.012 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:51:05.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:05.015 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:51:05.116 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:51:05.146 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:05.167 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:51:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:05.170 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:51:05.196 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:51:05.271 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:51:05.298 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:05.421 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:51:05.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:05.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:51:05.495 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-04-23T16:51:05.495 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-04-23T16:51:05.525 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:51:05.550 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:05 smithi148 ceph-mon[44520]: pgmap v1942: 27 pgs: 27 active+clean; 3.5 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 579 op/s; 3.6 MiB/s, 57 objects/s recovering 2024-04-23T16:51:05.553 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:05.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:05 smithi103 ceph-mon[39629]: pgmap v1942: 27 pgs: 27 active+clean; 3.5 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 579 op/s; 3.6 MiB/s, 57 objects/s recovering 2024-04-23T16:51:05.673 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:51:05.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:05 smithi188 ceph-mon[44101]: pgmap v1942: 27 pgs: 27 active+clean; 3.5 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 579 op/s; 3.6 MiB/s, 57 objects/s recovering 2024-04-23T16:51:06.402 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:51:06.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:06.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:51:06.506 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:51:06.534 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:06.632 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:51:06.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:06.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:51:06.735 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:06.768 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:06.789 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:51:06.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:06.794 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:51:06.894 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:51:06.927 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:06.948 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:51:06.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:06.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:51:06.960 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:51:06.960 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2024-04-23T16:51:06.960 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2024-04-23T16:51:06.960 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:51:06.960 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.960 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90919 7.2 7:408d099e:::benchmark_data_smithi188_141483_object90918:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:51:06.960 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.952953+0000", 2024-04-23T16:51:06.960 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 479.00264514899999, 2024-04-23T16:51:06.960 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.40145672100000002, 2024-04-23T16:51:06.960 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:06.961 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:06.961 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2024-04-23T16:51:06.961 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:06.961 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:06.961 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:06.961 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90919 2024-04-23T16:51:06.961 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.961 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:06.961 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.961 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:06.961 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.952953+0000", 2024-04-23T16:51:06.961 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:06.962 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.962 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.962 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:06.962 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.952953+0000", 2024-04-23T16:51:06.962 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:06.962 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.962 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.962 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:06.962 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.952956+0000", 2024-04-23T16:51:06.963 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6849999999999999e-06 2024-04-23T16:51:06.963 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.963 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.963 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:06.963 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953158+0000", 2024-04-23T16:51:06.963 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000201986 2024-04-23T16:51:06.963 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.963 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.963 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:06.963 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953174+0000", 2024-04-23T16:51:06.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5914e-05 2024-04-23T16:51:06.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.964 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:06.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953189+0000", 2024-04-23T16:51:06.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4158999999999999e-05 2024-04-23T16:51:06.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.964 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:06.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953687+0000", 2024-04-23T16:51:06.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00049814399999999995 2024-04-23T16:51:06.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.965 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:51:06.965 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953710+0000", 2024-04-23T16:51:06.965 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3178999999999999e-05 2024-04-23T16:51:06.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.965 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:06.966 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.343767+0000", 2024-04-23T16:51:06.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.39005708300000003 2024-04-23T16:51:06.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.966 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:51:06.966 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.344502+0000", 2024-04-23T16:51:06.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00073475700000000005 2024-04-23T16:51:06.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.966 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:06.966 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.352972+0000", 2024-04-23T16:51:06.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0084703320000000006 2024-04-23T16:51:06.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.967 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:06.967 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.354409+0000", 2024-04-23T16:51:06.967 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0014374819999999999 2024-04-23T16:51:06.967 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:06.967 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:06.967 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:06.967 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.967 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.967 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90920 7.2 7:45ca754c:::benchmark_data_smithi188_141483_object90919:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:51:06.967 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.953221+0000", 2024-04-23T16:51:06.967 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 479.00237703400001, 2024-04-23T16:51:06.967 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.40592639699999999, 2024-04-23T16:51:06.967 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:06.968 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:06.968 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2024-04-23T16:51:06.968 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:06.968 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:06.968 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:06.968 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90920 2024-04-23T16:51:06.968 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.968 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:06.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.968 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:06.968 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953221+0000", 2024-04-23T16:51:06.968 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:06.968 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.968 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:06.969 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953221+0000", 2024-04-23T16:51:06.969 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:06.969 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.969 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.969 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:06.969 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953222+0000", 2024-04-23T16:51:06.969 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9199999999999999e-07 2024-04-23T16:51:06.969 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.969 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.969 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:06.969 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953284+0000", 2024-04-23T16:51:06.969 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1965999999999996e-05 2024-04-23T16:51:06.969 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:06.970 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953291+0000", 2024-04-23T16:51:06.970 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4739999999999999e-06 2024-04-23T16:51:06.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:06.970 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953297+0000", 2024-04-23T16:51:06.970 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8390000000000003e-06 2024-04-23T16:51:06.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:06.970 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953757+0000", 2024-04-23T16:51:06.970 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000460381 2024-04-23T16:51:06.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:51:06.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953776+0000", 2024-04-23T16:51:06.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8097999999999999e-05 2024-04-23T16:51:06.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:06.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.347099+0000", 2024-04-23T16:51:06.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.39332325699999998 2024-04-23T16:51:06.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:51:06.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.347593+0000", 2024-04-23T16:51:06.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00049373400000000001 2024-04-23T16:51:06.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.972 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:06.972 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.359068+0000", 2024-04-23T16:51:06.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011475378 2024-04-23T16:51:06.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.972 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:06.972 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.359147+0000", 2024-04-23T16:51:06.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.9377999999999998e-05 2024-04-23T16:51:06.972 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:06.972 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:06.972 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:06.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.973 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90926 7.2 7:42975870:::benchmark_data_smithi188_141483_object90925:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:51:06.973 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.953305+0000", 2024-04-23T16:51:06.973 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 479.002292834, 2024-04-23T16:51:06.974 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.40994392299999999, 2024-04-23T16:51:06.974 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:06.974 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:06.974 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2024-04-23T16:51:06.974 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:06.974 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:06.974 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:06.974 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90926 2024-04-23T16:51:06.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.974 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:06.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:06.974 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953305+0000", 2024-04-23T16:51:06.974 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:06.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.975 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:06.975 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953305+0000", 2024-04-23T16:51:06.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:06.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.975 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:06.975 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953306+0000", 2024-04-23T16:51:06.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6899999999999997e-07 2024-04-23T16:51:06.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:06.976 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953354+0000", 2024-04-23T16:51:06.976 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8729000000000001e-05 2024-04-23T16:51:06.976 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:06.976 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953361+0000", 2024-04-23T16:51:06.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2539999999999998e-06 2024-04-23T16:51:06.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.977 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:06.977 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953366+0000", 2024-04-23T16:51:06.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1839999999999998e-06 2024-04-23T16:51:06.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.977 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:06.977 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953840+0000", 2024-04-23T16:51:06.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00047402499999999999 2024-04-23T16:51:06.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.978 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:51:06.978 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953848+0000", 2024-04-23T16:51:06.978 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7069999999999992e-06 2024-04-23T16:51:06.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.978 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.978 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:06.978 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.347658+0000", 2024-04-23T16:51:06.978 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.393810098 2024-04-23T16:51:06.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.978 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.978 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:51:06.978 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.348210+0000", 2024-04-23T16:51:06.979 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00055245100000000003 2024-04-23T16:51:06.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.979 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.979 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:06.979 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.363193+0000", 2024-04-23T16:51:06.979 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014983273 2024-04-23T16:51:06.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.979 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.979 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:06.979 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.363249+0000", 2024-04-23T16:51:06.979 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5633000000000001e-05 2024-04-23T16:51:06.979 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:06.979 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:06.979 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:06.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.980 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.980 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90929 7.2 7:4dfdd413:::benchmark_data_smithi188_141483_object90928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:51:06.980 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.953372+0000", 2024-04-23T16:51:06.980 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 479.00222544500002, 2024-04-23T16:51:06.980 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.46898921199999999, 2024-04-23T16:51:06.980 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:06.980 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:06.981 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2024-04-23T16:51:06.981 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:06.981 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:06.981 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:06.981 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90929 2024-04-23T16:51:06.981 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.981 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:06.981 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.981 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:06.981 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953372+0000", 2024-04-23T16:51:06.981 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:06.981 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.981 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.982 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:06.982 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953372+0000", 2024-04-23T16:51:06.982 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:06.982 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.982 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.982 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:06.982 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953373+0000", 2024-04-23T16:51:06.982 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8100000000000003e-07 2024-04-23T16:51:06.982 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.982 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.982 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:06.982 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953486+0000", 2024-04-23T16:51:06.982 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000113242 2024-04-23T16:51:06.982 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.983 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.983 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:06.983 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953496+0000", 2024-04-23T16:51:06.983 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6800000000000005e-06 2024-04-23T16:51:06.983 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.983 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.983 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:06.983 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953504+0000", 2024-04-23T16:51:06.983 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1939999999999998e-06 2024-04-23T16:51:06.983 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.983 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.983 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:06.983 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953881+0000", 2024-04-23T16:51:06.983 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00037709000000000001 2024-04-23T16:51:06.984 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.984 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:51:06.984 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953888+0000", 2024-04-23T16:51:06.984 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0539999999999997e-06 2024-04-23T16:51:06.984 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.984 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:06.984 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.365555+0000", 2024-04-23T16:51:06.984 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.41166677200000001 2024-04-23T16:51:06.984 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.984 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for degraded object", 2024-04-23T16:51:06.984 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.366044+0000", 2024-04-23T16:51:06.985 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00048844999999999997 2024-04-23T16:51:06.985 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.985 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.985 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:06.985 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.422260+0000", 2024-04-23T16:51:06.985 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.056216751000000002 2024-04-23T16:51:06.985 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.985 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.985 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:06.985 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.422362+0000", 2024-04-23T16:51:06.985 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010139799999999999 2024-04-23T16:51:06.985 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:06.985 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:06.985 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:06.985 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.986 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.986 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90941 7.2 7:492610cd:::benchmark_data_smithi188_141483_object90940:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:51:06.986 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.953517+0000", 2024-04-23T16:51:06.986 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 479.00208049600002, 2024-04-23T16:51:06.986 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.48637596500000002, 2024-04-23T16:51:06.986 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:06.986 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:06.986 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:06.986 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:06.987 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:06.987 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:06.987 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90941 2024-04-23T16:51:06.987 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.987 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:06.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:06.987 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953517+0000", 2024-04-23T16:51:06.987 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:06.987 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:06.987 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953517+0000", 2024-04-23T16:51:06.987 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:06.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.988 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:06.988 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953518+0000", 2024-04-23T16:51:06.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.33e-07 2024-04-23T16:51:06.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.988 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:06.988 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953567+0000", 2024-04-23T16:51:06.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9076000000000002e-05 2024-04-23T16:51:06.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.988 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:06.988 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953573+0000", 2024-04-23T16:51:06.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7939999999999999e-06 2024-04-23T16:51:06.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.989 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:06.989 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953577+0000", 2024-04-23T16:51:06.989 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.053e-06 2024-04-23T16:51:06.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.989 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:06.989 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953919+0000", 2024-04-23T16:51:06.989 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00034173100000000001 2024-04-23T16:51:06.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.989 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:51:06.989 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953926+0000", 2024-04-23T16:51:06.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.058e-06 2024-04-23T16:51:06.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:06.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.366082+0000", 2024-04-23T16:51:06.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.412156409 2024-04-23T16:51:06.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:51:06.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.366789+0000", 2024-04-23T16:51:06.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000706377 2024-04-23T16:51:06.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:51:06.991 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.366970+0000", 2024-04-23T16:51:06.991 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00018143900000000001 2024-04-23T16:51:06.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.991 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:51:06.991 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.376052+0000", 2024-04-23T16:51:06.991 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0090815070000000008 2024-04-23T16:51:06.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.991 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:06.991 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.436876+0000", 2024-04-23T16:51:06.991 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.06082427 2024-04-23T16:51:06.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:51:06.992 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.436922+0000", 2024-04-23T16:51:06.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6088999999999999e-05 2024-04-23T16:51:06.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:06.992 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.439893+0000", 2024-04-23T16:51:06.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0029712290000000001 2024-04-23T16:51:06.992 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:06.992 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:06.992 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:06.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.992 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90944 7.2 7:45278a20:::benchmark_data_smithi188_141483_object90943:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:51:06.993 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.953583+0000", 2024-04-23T16:51:06.993 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 479.00201466200002, 2024-04-23T16:51:06.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.48914311599999999, 2024-04-23T16:51:06.993 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:06.993 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:06.993 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:06.993 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:06.993 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:06.993 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:06.993 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90944 2024-04-23T16:51:06.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.993 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:06.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.993 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:06.994 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953583+0000", 2024-04-23T16:51:06.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:06.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.994 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:06.994 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953583+0000", 2024-04-23T16:51:06.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:06.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.994 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:06.994 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953584+0000", 2024-04-23T16:51:06.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5200000000000002e-07 2024-04-23T16:51:06.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:06.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953666+0000", 2024-04-23T16:51:06.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2119000000000004e-05 2024-04-23T16:51:06.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:06.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953675+0000", 2024-04-23T16:51:06.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5510000000000005e-06 2024-04-23T16:51:06.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:06.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953683+0000", 2024-04-23T16:51:06.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0399999999999993e-06 2024-04-23T16:51:06.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.996 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.996 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:06.996 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953951+0000", 2024-04-23T16:51:06.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00026800499999999998 2024-04-23T16:51:06.996 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.996 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.996 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:51:06.996 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953958+0000", 2024-04-23T16:51:06.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2559999999999999e-06 2024-04-23T16:51:06.996 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.996 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.996 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:06.996 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.369565+0000", 2024-04-23T16:51:06.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.41560729000000002 2024-04-23T16:51:06.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.997 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:51:06.997 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.369774+0000", 2024-04-23T16:51:06.997 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00020933599999999999 2024-04-23T16:51:06.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.997 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:51:06.997 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.369961+0000", 2024-04-23T16:51:06.997 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0001872 2024-04-23T16:51:06.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.997 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:51:06.997 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.378828+0000", 2024-04-23T16:51:06.998 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0088667429999999998 2024-04-23T16:51:06.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.998 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:06.998 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.439955+0000", 2024-04-23T16:51:06.998 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.061126416000000003 2024-04-23T16:51:06.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.998 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:51:06.998 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.440015+0000", 2024-04-23T16:51:06.998 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0238000000000002e-05 2024-04-23T16:51:06.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:06.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:06.998 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:06.999 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.442726+0000", 2024-04-23T16:51:06.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0027114700000000001 2024-04-23T16:51:06.999 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.002 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:07.002 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.002 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.002 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90951 7.2 7:4e3203d8:::benchmark_data_smithi188_141483_object90950:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:51:07.002 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.953695+0000", 2024-04-23T16:51:07.002 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 479.00190319299998, 2024-04-23T16:51:07.002 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.49130570800000001, 2024-04-23T16:51:07.002 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:07.002 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:07.002 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:07.002 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:07.002 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:07.002 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:07.003 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90951 2024-04-23T16:51:07.003 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.003 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:07.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:07.003 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953695+0000", 2024-04-23T16:51:07.003 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.003 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:07.003 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953695+0000", 2024-04-23T16:51:07.003 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.003 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:07.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953696+0000", 2024-04-23T16:51:07.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5899999999999995e-07 2024-04-23T16:51:07.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.004 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:07.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953747+0000", 2024-04-23T16:51:07.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1547e-05 2024-04-23T16:51:07.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.004 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:07.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953753+0000", 2024-04-23T16:51:07.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6659999999999996e-06 2024-04-23T16:51:07.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:07.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953757+0000", 2024-04-23T16:51:07.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9269999999999998e-06 2024-04-23T16:51:07.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953980+0000", 2024-04-23T16:51:07.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000223773 2024-04-23T16:51:07.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:51:07.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953984+0000", 2024-04-23T16:51:07.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6380000000000002e-06 2024-04-23T16:51:07.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.006 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.006 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.372732+0000", 2024-04-23T16:51:07.006 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.41874784799999998 2024-04-23T16:51:07.006 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.006 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.006 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:51:07.006 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.372875+0000", 2024-04-23T16:51:07.006 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000143273 2024-04-23T16:51:07.006 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.006 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.006 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:51:07.006 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.373022+0000", 2024-04-23T16:51:07.006 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014640899999999999 2024-04-23T16:51:07.006 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.007 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:51:07.007 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.381656+0000", 2024-04-23T16:51:07.007 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0086342160000000001 2024-04-23T16:51:07.007 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.007 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:07.007 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.442769+0000", 2024-04-23T16:51:07.007 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.061113155000000002 2024-04-23T16:51:07.007 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.007 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:51:07.007 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.442803+0000", 2024-04-23T16:51:07.007 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3815000000000001e-05 2024-04-23T16:51:07.008 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.008 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.008 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:07.008 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.445000+0000", 2024-04-23T16:51:07.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002197582 2024-04-23T16:51:07.008 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.008 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:07.008 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.008 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.008 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.008 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90965 7.2 7:403ee5c1:::benchmark_data_smithi188_141483_object90964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:51:07.008 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.953762+0000", 2024-04-23T16:51:07.008 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 479.00183553400001, 2024-04-23T16:51:07.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.49351197600000002, 2024-04-23T16:51:07.009 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:07.009 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:07.009 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:07.009 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:07.009 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:07.009 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:07.009 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90965 2024-04-23T16:51:07.009 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.009 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:07.009 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.009 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:07.009 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953762+0000", 2024-04-23T16:51:07.009 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.009 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.010 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:07.010 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953762+0000", 2024-04-23T16:51:07.010 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.010 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.010 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:07.010 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953763+0000", 2024-04-23T16:51:07.010 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1699999999999998e-07 2024-04-23T16:51:07.010 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.010 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:07.010 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953881+0000", 2024-04-23T16:51:07.010 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000118267 2024-04-23T16:51:07.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.011 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.011 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:07.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953890+0000", 2024-04-23T16:51:07.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7649999999999999e-06 2024-04-23T16:51:07.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.011 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.011 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:07.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.953896+0000", 2024-04-23T16:51:07.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5699999999999998e-06 2024-04-23T16:51:07.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.011 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.011 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954051+0000", 2024-04-23T16:51:07.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000154623 2024-04-23T16:51:07.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:51:07.012 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954060+0000", 2024-04-23T16:51:07.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9299999999999992e-06 2024-04-23T16:51:07.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.012 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.376075+0000", 2024-04-23T16:51:07.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.42201528399999999 2024-04-23T16:51:07.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:51:07.013 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.376223+0000", 2024-04-23T16:51:07.013 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000147606 2024-04-23T16:51:07.013 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.013 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.013 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:51:07.013 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.376331+0000", 2024-04-23T16:51:07.013 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107767 2024-04-23T16:51:07.013 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.013 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.013 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:51:07.013 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.388216+0000", 2024-04-23T16:51:07.013 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011885269 2024-04-23T16:51:07.013 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.013 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.013 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:07.014 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.445040+0000", 2024-04-23T16:51:07.014 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.056823783000000003 2024-04-23T16:51:07.014 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.014 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.014 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:51:07.014 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.445094+0000", 2024-04-23T16:51:07.014 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4716999999999998e-05 2024-04-23T16:51:07.014 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.014 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.014 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:07.014 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.447274+0000", 2024-04-23T16:51:07.014 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021798780000000001 2024-04-23T16:51:07.014 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.015 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:07.015 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.015 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.015 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.015 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90945 7.c 7:3e4f21cd:::benchmark_data_smithi188_141483_object90944:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:51:07.015 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.954061+0000", 2024-04-23T16:51:07.015 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 479.00153660000001, 2024-04-23T16:51:07.015 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.18484241200000001, 2024-04-23T16:51:07.015 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:07.015 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:07.015 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:07.015 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:07.015 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:07.015 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:07.016 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90945 2024-04-23T16:51:07.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.016 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:07.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.016 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:07.016 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954061+0000", 2024-04-23T16:51:07.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.016 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:07.016 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954061+0000", 2024-04-23T16:51:07.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.016 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:07.017 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954062+0000", 2024-04-23T16:51:07.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9999999999999997e-07 2024-04-23T16:51:07.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.017 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:07.017 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954123+0000", 2024-04-23T16:51:07.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1469000000000004e-05 2024-04-23T16:51:07.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.017 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:07.017 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954130+0000", 2024-04-23T16:51:07.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3879999999999997e-06 2024-04-23T16:51:07.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.018 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:07.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954136+0000", 2024-04-23T16:51:07.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7370000000000003e-06 2024-04-23T16:51:07.018 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.018 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955319+0000", 2024-04-23T16:51:07.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00118315 2024-04-23T16:51:07.018 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.018 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:51:07.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955326+0000", 2024-04-23T16:51:07.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9249999999999998e-06 2024-04-23T16:51:07.018 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.019 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.019 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.119609+0000", 2024-04-23T16:51:07.019 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.16428289800000001 2024-04-23T16:51:07.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.019 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.019 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:51:07.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.119782+0000", 2024-04-23T16:51:07.019 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000173711 2024-04-23T16:51:07.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.019 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.019 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:51:07.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.119997+0000", 2024-04-23T16:51:07.019 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000214489 2024-04-23T16:51:07.020 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.020 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.020 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:51:07.020 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.130219+0000", 2024-04-23T16:51:07.020 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01022229 2024-04-23T16:51:07.020 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.020 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.020 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:07.020 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.135874+0000", 2024-04-23T16:51:07.020 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0056549449999999998 2024-04-23T16:51:07.020 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.020 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.020 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:51:07.020 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.135936+0000", 2024-04-23T16:51:07.021 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2121999999999999e-05 2024-04-23T16:51:07.021 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.021 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.021 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:07.021 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.138904+0000", 2024-04-23T16:51:07.021 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0029675880000000002 2024-04-23T16:51:07.021 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.021 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:07.021 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.021 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.021 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.021 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90962 7.c 7:32187531:::benchmark_data_smithi188_141483_object90961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:51:07.021 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.954146+0000", 2024-04-23T16:51:07.021 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 479.00145188099998, 2024-04-23T16:51:07.022 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.188198055, 2024-04-23T16:51:07.022 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:07.022 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:07.022 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:07.022 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:07.022 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:07.022 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:07.022 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90962 2024-04-23T16:51:07.022 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.022 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:07.022 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.022 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:07.022 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954146+0000", 2024-04-23T16:51:07.022 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.023 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:07.023 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954146+0000", 2024-04-23T16:51:07.023 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.023 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:07.023 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954147+0000", 2024-04-23T16:51:07.023 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9999999999999997e-07 2024-04-23T16:51:07.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.023 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:07.023 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954221+0000", 2024-04-23T16:51:07.024 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3998000000000005e-05 2024-04-23T16:51:07.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.024 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:07.024 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954226+0000", 2024-04-23T16:51:07.024 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5280000000000003e-06 2024-04-23T16:51:07.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.024 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:07.024 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954232+0000", 2024-04-23T16:51:07.024 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0669999999999997e-06 2024-04-23T16:51:07.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.024 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.025 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955364+0000", 2024-04-23T16:51:07.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0011320880000000001 2024-04-23T16:51:07.025 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.025 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:51:07.025 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955371+0000", 2024-04-23T16:51:07.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0829999999999998e-06 2024-04-23T16:51:07.025 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.025 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.025 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.122427+0000", 2024-04-23T16:51:07.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.16705524799999999 2024-04-23T16:51:07.025 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.026 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:51:07.026 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.122536+0000", 2024-04-23T16:51:07.026 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000109279 2024-04-23T16:51:07.026 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.026 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:51:07.026 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.122631+0000", 2024-04-23T16:51:07.026 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.4969000000000005e-05 2024-04-23T16:51:07.026 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.026 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:51:07.026 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.130227+0000", 2024-04-23T16:51:07.026 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0075958079999999999 2024-04-23T16:51:07.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:07.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.138964+0000", 2024-04-23T16:51:07.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0087376569999999994 2024-04-23T16:51:07.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:51:07.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.139020+0000", 2024-04-23T16:51:07.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5680000000000002e-05 2024-04-23T16:51:07.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:07.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.142344+0000", 2024-04-23T16:51:07.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00332395 2024-04-23T16:51:07.028 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.028 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:07.028 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.028 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.028 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.028 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90969 7.c 7:35415fb1:::benchmark_data_smithi188_141483_object90968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:51:07.028 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.954241+0000", 2024-04-23T16:51:07.028 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 479.00135714200002, 2024-04-23T16:51:07.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19155655399999999, 2024-04-23T16:51:07.028 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:07.028 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:07.028 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:07.028 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:07.028 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:07.029 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:07.029 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90969 2024-04-23T16:51:07.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.029 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:07.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.029 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:07.029 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954241+0000", 2024-04-23T16:51:07.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.029 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:07.029 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954241+0000", 2024-04-23T16:51:07.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:07.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954241+0000", 2024-04-23T16:51:07.030 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8599999999999998e-07 2024-04-23T16:51:07.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:07.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954327+0000", 2024-04-23T16:51:07.030 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5081999999999996e-05 2024-04-23T16:51:07.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:07.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954332+0000", 2024-04-23T16:51:07.030 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2870000000000004e-06 2024-04-23T16:51:07.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.031 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.031 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:07.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954335+0000", 2024-04-23T16:51:07.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6729999999999998e-06 2024-04-23T16:51:07.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.031 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.031 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955414+0000", 2024-04-23T16:51:07.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001078332 2024-04-23T16:51:07.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.031 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.031 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:51:07.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955420+0000", 2024-04-23T16:51:07.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1650000000000003e-06 2024-04-23T16:51:07.032 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.032 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.032 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.127524+0000", 2024-04-23T16:51:07.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.17210402 2024-04-23T16:51:07.032 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.032 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:51:07.032 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.127638+0000", 2024-04-23T16:51:07.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000113719 2024-04-23T16:51:07.032 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.032 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:51:07.032 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.127806+0000", 2024-04-23T16:51:07.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000168102 2024-04-23T16:51:07.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:51:07.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.132574+0000", 2024-04-23T16:51:07.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0047678249999999998 2024-04-23T16:51:07.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:07.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.142402+0000", 2024-04-23T16:51:07.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0098287519999999996 2024-04-23T16:51:07.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:51:07.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.142463+0000", 2024-04-23T16:51:07.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0618999999999997e-05 2024-04-23T16:51:07.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.034 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:07.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.145797+0000", 2024-04-23T16:51:07.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003334292 2024-04-23T16:51:07.034 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.034 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:07.034 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.034 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90972 7.c 7:3f3cd386:::benchmark_data_smithi188_141483_object90971:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e563)", 2024-04-23T16:51:07.034 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:07.954341+0000", 2024-04-23T16:51:07.035 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 479.001256775, 2024-04-23T16:51:07.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19740545100000001, 2024-04-23T16:51:07.035 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:07.035 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:07.035 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:07.035 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:07.035 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:07.035 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:07.035 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90972 2024-04-23T16:51:07.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.035 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:07.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:07.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954341+0000", 2024-04-23T16:51:07.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:07.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954341+0000", 2024-04-23T16:51:07.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:07.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954342+0000", 2024-04-23T16:51:07.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9999999999999998e-07 2024-04-23T16:51:07.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:07.037 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954472+0000", 2024-04-23T16:51:07.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013074399999999999 2024-04-23T16:51:07.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:07.037 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954478+0000", 2024-04-23T16:51:07.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2499999999999997e-06 2024-04-23T16:51:07.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:07.037 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.954482+0000", 2024-04-23T16:51:07.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9879999999999998e-06 2024-04-23T16:51:07.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955457+0000", 2024-04-23T16:51:07.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00097560500000000005 2024-04-23T16:51:07.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:51:07.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:07.955465+0000", 2024-04-23T16:51:07.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7340000000000008e-06 2024-04-23T16:51:07.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.130257+0000", 2024-04-23T16:51:07.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.17479178100000001 2024-04-23T16:51:07.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:51:07.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.130385+0000", 2024-04-23T16:51:07.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012878499999999999 2024-04-23T16:51:07.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 11", 2024-04-23T16:51:07.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.130499+0000", 2024-04-23T16:51:07.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011311000000000001 2024-04-23T16:51:07.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:51:07.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.132767+0000", 2024-04-23T16:51:07.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0022680090000000001 2024-04-23T16:51:07.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:07.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.148612+0000", 2024-04-23T16:51:07.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015844982000000001 2024-04-23T16:51:07.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:51:07.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.148675+0000", 2024-04-23T16:51:07.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3193000000000003e-05 2024-04-23T16:51:07.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:07.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.151747+0000", 2024-04-23T16:51:07.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00307187 2024-04-23T16:51:07.041 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.041 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:07.041 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.041 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90988 7.2 7:41f48dd3:::benchmark_data_smithi188_141483_object90987:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-23T16:51:07.041 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:08.087481+0000", 2024-04-23T16:51:07.041 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 478.86811720100002, 2024-04-23T16:51:07.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.36209360899999998, 2024-04-23T16:51:07.041 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:07.041 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:07.041 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:07.041 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:07.042 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:07.042 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:07.042 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90988 2024-04-23T16:51:07.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.042 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:07.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:07.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.087481+0000", 2024-04-23T16:51:07.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:07.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.087481+0000", 2024-04-23T16:51:07.043 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.043 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:07.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.087483+0000", 2024-04-23T16:51:07.043 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.826e-06 2024-04-23T16:51:07.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.043 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:07.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.087532+0000", 2024-04-23T16:51:07.043 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9818999999999999e-05 2024-04-23T16:51:07.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.043 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:07.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.087543+0000", 2024-04-23T16:51:07.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0331e-05 2024-04-23T16:51:07.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:07.044 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.087548+0000", 2024-04-23T16:51:07.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7509999999999997e-06 2024-04-23T16:51:07.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.044 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.087601+0000", 2024-04-23T16:51:07.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2181000000000001e-05 2024-04-23T16:51:07.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:51:07.045 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.087607+0000", 2024-04-23T16:51:07.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4489999999999996e-06 2024-04-23T16:51:07.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.045 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.378860+0000", 2024-04-23T16:51:07.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.291253137 2024-04-23T16:51:07.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:51:07.045 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.378976+0000", 2024-04-23T16:51:07.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000115454 2024-04-23T16:51:07.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:51:07.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.379112+0000", 2024-04-23T16:51:07.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000136668 2024-04-23T16:51:07.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.046 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:51:07.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.395111+0000", 2024-04-23T16:51:07.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015998366999999999 2024-04-23T16:51:07.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.046 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:07.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.447324+0000", 2024-04-23T16:51:07.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052213183000000003 2024-04-23T16:51:07.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:51:07.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.447379+0000", 2024-04-23T16:51:07.047 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5078999999999999e-05 2024-04-23T16:51:07.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:07.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.449574+0000", 2024-04-23T16:51:07.047 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002195364 2024-04-23T16:51:07.047 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.047 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:07.047 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.047 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:90993 7.2 7:44f8e9e0:::benchmark_data_smithi188_141483_object90992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-23T16:51:07.048 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:08.138947+0000", 2024-04-23T16:51:07.048 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 478.81665106200001, 2024-04-23T16:51:07.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.31274553900000002, 2024-04-23T16:51:07.048 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:07.048 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:07.048 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:07.048 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:07.048 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:07.048 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:07.048 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 90993 2024-04-23T16:51:07.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.048 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:07.048 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.048 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:07.049 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.138947+0000", 2024-04-23T16:51:07.049 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.049 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.049 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.049 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:07.049 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.138947+0000", 2024-04-23T16:51:07.049 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.049 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.049 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.049 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:07.049 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.138948+0000", 2024-04-23T16:51:07.049 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5659999999999999e-06 2024-04-23T16:51:07.049 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.049 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.049 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:07.050 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.139005+0000", 2024-04-23T16:51:07.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6983e-05 2024-04-23T16:51:07.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:07.050 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.139014+0000", 2024-04-23T16:51:07.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2709999999999992e-06 2024-04-23T16:51:07.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:07.050 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.139020+0000", 2024-04-23T16:51:07.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.93e-06 2024-04-23T16:51:07.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.139080+0000", 2024-04-23T16:51:07.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0785000000000002e-05 2024-04-23T16:51:07.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:51:07.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.139092+0000", 2024-04-23T16:51:07.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1473e-05 2024-04-23T16:51:07.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.381704+0000", 2024-04-23T16:51:07.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.24261231799999999 2024-04-23T16:51:07.052 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.052 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:51:07.052 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.381868+0000", 2024-04-23T16:51:07.052 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00016402100000000001 2024-04-23T16:51:07.052 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.052 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:51:07.052 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.382051+0000", 2024-04-23T16:51:07.052 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00018281499999999999 2024-04-23T16:51:07.052 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.052 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:51:07.053 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.395119+0000", 2024-04-23T16:51:07.053 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01306789 2024-04-23T16:51:07.053 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.053 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.053 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:07.053 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.449634+0000", 2024-04-23T16:51:07.053 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.054515317000000001 2024-04-23T16:51:07.053 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.053 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.053 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:51:07.053 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.449684+0000", 2024-04-23T16:51:07.053 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9802000000000002e-05 2024-04-23T16:51:07.053 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.053 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.054 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:07.054 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.451692+0000", 2024-04-23T16:51:07.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002008368 2024-04-23T16:51:07.054 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.054 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:07.054 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.054 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:91001 7.2 7:483e89c4:::benchmark_data_smithi188_141483_object91000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-23T16:51:07.054 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:08.150731+0000", 2024-04-23T16:51:07.054 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 478.80486688799999, 2024-04-23T16:51:07.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.30912154400000003, 2024-04-23T16:51:07.054 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:07.054 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:07.055 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:07.055 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:07.055 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:07.055 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:07.055 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 91001 2024-04-23T16:51:07.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.055 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:07.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:07.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.150731+0000", 2024-04-23T16:51:07.055 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:07.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.150731+0000", 2024-04-23T16:51:07.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.056 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.056 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.056 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:07.056 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.150734+0000", 2024-04-23T16:51:07.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6520000000000002e-06 2024-04-23T16:51:07.056 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.056 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.056 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:07.056 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.150789+0000", 2024-04-23T16:51:07.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5597999999999999e-05 2024-04-23T16:51:07.056 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.056 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.056 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:07.056 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.150803+0000", 2024-04-23T16:51:07.057 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3278000000000001e-05 2024-04-23T16:51:07.057 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.057 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.057 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:07.057 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.150811+0000", 2024-04-23T16:51:07.057 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5920000000000001e-06 2024-04-23T16:51:07.057 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.057 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.057 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.057 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.150881+0000", 2024-04-23T16:51:07.057 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0037999999999996e-05 2024-04-23T16:51:07.057 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.057 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.057 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:51:07.058 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.150893+0000", 2024-04-23T16:51:07.058 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1942e-05 2024-04-23T16:51:07.058 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.058 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.058 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.058 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.385185+0000", 2024-04-23T16:51:07.058 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.234292324 2024-04-23T16:51:07.058 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.058 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.058 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:51:07.058 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.385327+0000", 2024-04-23T16:51:07.058 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014171499999999999 2024-04-23T16:51:07.058 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.058 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.059 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:51:07.059 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.385502+0000", 2024-04-23T16:51:07.059 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000174778 2024-04-23T16:51:07.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.059 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.059 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:51:07.059 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.401156+0000", 2024-04-23T16:51:07.059 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015653923 2024-04-23T16:51:07.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.059 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.059 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:07.059 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.456669+0000", 2024-04-23T16:51:07.059 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.055513593999999999 2024-04-23T16:51:07.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.060 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:51:07.060 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.456727+0000", 2024-04-23T16:51:07.060 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7475000000000001e-05 2024-04-23T16:51:07.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.060 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:07.060 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.459853+0000", 2024-04-23T16:51:07.060 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0031256349999999999 2024-04-23T16:51:07.060 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.060 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:07.060 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.060 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:91022 7.2 7:46f7851e:::benchmark_data_smithi188_141483_object91021:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-23T16:51:07.061 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:08.215689+0000", 2024-04-23T16:51:07.061 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 478.73990926800002, 2024-04-23T16:51:07.061 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.24714624399999999, 2024-04-23T16:51:07.061 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:07.061 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:07.061 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:07.061 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:07.061 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:07.061 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:07.061 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 91022 2024-04-23T16:51:07.061 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.061 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:07.061 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.061 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:07.062 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.215689+0000", 2024-04-23T16:51:07.062 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.062 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.062 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.062 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:07.062 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.215689+0000", 2024-04-23T16:51:07.062 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.062 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.062 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.062 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:07.062 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.215691+0000", 2024-04-23T16:51:07.062 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6240000000000002e-06 2024-04-23T16:51:07.062 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.062 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.063 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:07.063 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.215745+0000", 2024-04-23T16:51:07.063 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4035e-05 2024-04-23T16:51:07.063 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.063 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.063 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:07.063 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.215759+0000", 2024-04-23T16:51:07.063 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3451999999999999e-05 2024-04-23T16:51:07.063 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.063 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.063 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:07.063 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.215769+0000", 2024-04-23T16:51:07.063 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9690000000000006e-06 2024-04-23T16:51:07.063 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.064 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.064 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.064 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.215833+0000", 2024-04-23T16:51:07.064 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3972000000000003e-05 2024-04-23T16:51:07.064 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.064 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.064 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:51:07.064 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.215844+0000", 2024-04-23T16:51:07.064 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0875e-05 2024-04-23T16:51:07.064 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.064 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.064 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.064 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.388269+0000", 2024-04-23T16:51:07.064 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.17242563999999999 2024-04-23T16:51:07.065 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.065 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.065 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:51:07.065 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.388427+0000", 2024-04-23T16:51:07.065 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015798799999999999 2024-04-23T16:51:07.065 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.065 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.065 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:51:07.065 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.388609+0000", 2024-04-23T16:51:07.065 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00018175600000000001 2024-04-23T16:51:07.065 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.065 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.065 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:51:07.065 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.401160+0000", 2024-04-23T16:51:07.066 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012551306 2024-04-23T16:51:07.066 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.066 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.066 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:07.066 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.459916+0000", 2024-04-23T16:51:07.066 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.058755325999999997 2024-04-23T16:51:07.066 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.066 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.066 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:51:07.066 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.459969+0000", 2024-04-23T16:51:07.066 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3513000000000001e-05 2024-04-23T16:51:07.066 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.066 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.066 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:07.067 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.462835+0000", 2024-04-23T16:51:07.067 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0028657880000000002 2024-04-23T16:51:07.067 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.067 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:07.067 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.067 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.067 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.067 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:91042 7.2 7:4e7ae021:::benchmark_data_smithi188_141483_object91041:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-23T16:51:07.067 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:43:08.279824+0000", 2024-04-23T16:51:07.067 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 478.67577417299998, 2024-04-23T16:51:07.067 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.185784803, 2024-04-23T16:51:07.067 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:07.067 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:07.067 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:07.068 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:07.068 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:07.068 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:07.068 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 91042 2024-04-23T16:51:07.068 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.068 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:07.068 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.068 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:07.068 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.279824+0000", 2024-04-23T16:51:07.068 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.068 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.068 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.068 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:07.068 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.279824+0000", 2024-04-23T16:51:07.068 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.069 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.069 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:07.069 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.279826+0000", 2024-04-23T16:51:07.069 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4760000000000001e-06 2024-04-23T16:51:07.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.069 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.069 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:07.069 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.279880+0000", 2024-04-23T16:51:07.069 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4113000000000002e-05 2024-04-23T16:51:07.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.069 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.069 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:07.069 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.279892+0000", 2024-04-23T16:51:07.070 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1477999999999999e-05 2024-04-23T16:51:07.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.070 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.070 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:07.070 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.279900+0000", 2024-04-23T16:51:07.070 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.055e-06 2024-04-23T16:51:07.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.070 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.070 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.070 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.279955+0000", 2024-04-23T16:51:07.070 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5603999999999998e-05 2024-04-23T16:51:07.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.070 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.070 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2024-04-23T16:51:07.071 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.279962+0000", 2024-04-23T16:51:07.071 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0010000000000004e-06 2024-04-23T16:51:07.071 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.071 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.071 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.071 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.391777+0000", 2024-04-23T16:51:07.071 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.11181430000000001 2024-04-23T16:51:07.071 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.071 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.071 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:51:07.071 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.391914+0000", 2024-04-23T16:51:07.071 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000137553 2024-04-23T16:51:07.071 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.071 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.071 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 7", 2024-04-23T16:51:07.072 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.392022+0000", 2024-04-23T16:51:07.072 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107668 2024-04-23T16:51:07.072 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.072 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.072 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:51:07.072 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.403995+0000", 2024-04-23T16:51:07.072 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011972554999999999 2024-04-23T16:51:07.072 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.072 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.072 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:07.072 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.462906+0000", 2024-04-23T16:51:07.072 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.058911223999999998 2024-04-23T16:51:07.072 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.072 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.072 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:51:07.073 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.462955+0000", 2024-04-23T16:51:07.073 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9363999999999997e-05 2024-04-23T16:51:07.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.073 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.073 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:07.073 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:43:08.465609+0000", 2024-04-23T16:51:07.073 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002653412 2024-04-23T16:51:07.073 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.073 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:07.073 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.073 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.073 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:212497 7.d 7:bd34efcb:::benchmark_data_smithi188_141483_object93816:head [delete] snapc 0=[] ondisk+write+known_if_redirected e595)", 2024-04-23T16:51:07.073 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:45:35.717520+0000", 2024-04-23T16:51:07.074 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 331.23807811500001, 2024-04-23T16:51:07.074 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.211223097, 2024-04-23T16:51:07.074 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:07.074 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:07.074 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:07.074 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:07.074 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:07.074 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:07.074 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 212497 2024-04-23T16:51:07.074 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.074 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:07.074 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.074 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:07.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.717520+0000", 2024-04-23T16:51:07.075 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.075 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:07.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.717520+0000", 2024-04-23T16:51:07.075 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.075 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:07.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.717521+0000", 2024-04-23T16:51:07.075 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.187e-06 2024-04-23T16:51:07.075 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.076 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:07.076 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.717525+0000", 2024-04-23T16:51:07.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1779999999999996e-06 2024-04-23T16:51:07.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.076 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.076 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:07.076 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.717535+0000", 2024-04-23T16:51:07.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5449999999999992e-06 2024-04-23T16:51:07.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.076 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.076 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:07.076 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.717543+0000", 2024-04-23T16:51:07.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8849999999999999e-06 2024-04-23T16:51:07.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.076 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.717592+0000", 2024-04-23T16:51:07.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9314000000000003e-05 2024-04-23T16:51:07.077 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:51:07.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.717669+0000", 2024-04-23T16:51:07.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6867999999999997e-05 2024-04-23T16:51:07.077 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 1", 2024-04-23T16:51:07.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.717737+0000", 2024-04-23T16:51:07.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8164999999999995e-05 2024-04-23T16:51:07.077 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.078 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.078 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:51:07.078 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.721609+0000", 2024-04-23T16:51:07.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0038720389999999999 2024-04-23T16:51:07.078 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.078 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.078 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:07.078 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.928636+0000", 2024-04-23T16:51:07.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.20702727300000001 2024-04-23T16:51:07.078 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.078 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.078 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:51:07.078 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.928693+0000", 2024-04-23T16:51:07.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6764000000000001e-05 2024-04-23T16:51:07.079 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.079 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.079 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:07.079 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.928743+0000", 2024-04-23T16:51:07.079 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9879000000000001e-05 2024-04-23T16:51:07.079 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.079 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:07.079 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.079 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.079 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.079 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:212519 7.d 7:bb623d9c:::benchmark_data_smithi188_141483_object93838:head [delete] snapc 0=[] ondisk+write+known_if_redirected e595)", 2024-04-23T16:51:07.079 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:45:35.744077+0000", 2024-04-23T16:51:07.079 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 331.21152095799999, 2024-04-23T16:51:07.079 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.19108017899999999, 2024-04-23T16:51:07.080 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:07.080 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:07.080 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:07.080 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:07.080 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:07.080 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:07.080 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 212519 2024-04-23T16:51:07.080 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.080 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:07.080 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.080 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:07.080 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744077+0000", 2024-04-23T16:51:07.080 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.080 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.081 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:07.081 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744077+0000", 2024-04-23T16:51:07.081 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.081 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:07.081 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744078+0000", 2024-04-23T16:51:07.081 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9999999999999995e-07 2024-04-23T16:51:07.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.081 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:07.081 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744082+0000", 2024-04-23T16:51:07.081 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.929e-06 2024-04-23T16:51:07.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.082 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:07.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744089+0000", 2024-04-23T16:51:07.082 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6149999999999999e-06 2024-04-23T16:51:07.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.082 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:07.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744095+0000", 2024-04-23T16:51:07.082 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1159999999999998e-06 2024-04-23T16:51:07.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.082 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744135+0000", 2024-04-23T16:51:07.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0763000000000003e-05 2024-04-23T16:51:07.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:51:07.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744229+0000", 2024-04-23T16:51:07.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.4123000000000007e-05 2024-04-23T16:51:07.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 1", 2024-04-23T16:51:07.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744297+0000", 2024-04-23T16:51:07.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7549e-05 2024-04-23T16:51:07.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:51:07.084 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.753199+0000", 2024-04-23T16:51:07.084 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0089017230000000003 2024-04-23T16:51:07.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.084 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:07.084 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.935049+0000", 2024-04-23T16:51:07.084 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.18185061399999999 2024-04-23T16:51:07.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.084 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:51:07.084 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.935109+0000", 2024-04-23T16:51:07.084 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9227000000000001e-05 2024-04-23T16:51:07.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.085 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:07.085 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.935157+0000", 2024-04-23T16:51:07.085 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8520000000000003e-05 2024-04-23T16:51:07.085 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.085 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:07.085 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.085 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.085 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.085 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:212520 7.d 7:b66cbaa7:::benchmark_data_smithi188_141483_object93839:head [delete] snapc 0=[] ondisk+write+known_if_redirected e595)", 2024-04-23T16:51:07.085 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:45:35.744178+0000", 2024-04-23T16:51:07.085 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 331.211419455, 2024-04-23T16:51:07.085 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.191085055, 2024-04-23T16:51:07.085 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:51:07.085 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:51:07.086 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:07.086 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:51:07.086 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:51:07.086 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:51:07.086 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 212520 2024-04-23T16:51:07.086 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.086 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:51:07.086 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.086 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:51:07.086 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744178+0000", 2024-04-23T16:51:07.086 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.086 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.086 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.086 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:51:07.087 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744178+0000", 2024-04-23T16:51:07.087 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:51:07.087 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.087 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.087 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:51:07.087 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744179+0000", 2024-04-23T16:51:07.087 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5900000000000005e-07 2024-04-23T16:51:07.087 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.087 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.087 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:51:07.087 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744183+0000", 2024-04-23T16:51:07.087 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8739999999999997e-06 2024-04-23T16:51:07.087 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.087 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.088 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:51:07.088 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744192+0000", 2024-04-23T16:51:07.088 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.541e-06 2024-04-23T16:51:07.088 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.088 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.088 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:51:07.088 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.744198+0000", 2024-04-23T16:51:07.088 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1600000000000003e-06 2024-04-23T16:51:07.088 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.088 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.088 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:51:07.088 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.748787+0000", 2024-04-23T16:51:07.088 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045890490000000004 2024-04-23T16:51:07.088 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.089 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.089 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:51:07.089 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.748917+0000", 2024-04-23T16:51:07.089 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012969500000000001 2024-04-23T16:51:07.089 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.089 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.089 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 1", 2024-04-23T16:51:07.089 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.749037+0000", 2024-04-23T16:51:07.089 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000120419 2024-04-23T16:51:07.089 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.089 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.089 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:51:07.089 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.753343+0000", 2024-04-23T16:51:07.089 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0043057740000000001 2024-04-23T16:51:07.090 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.090 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.090 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:07.090 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.935194+0000", 2024-04-23T16:51:07.090 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.18185153000000001 2024-04-23T16:51:07.090 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.090 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.090 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:51:07.090 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.935226+0000", 2024-04-23T16:51:07.090 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1449999999999999e-05 2024-04-23T16:51:07.090 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:51:07.090 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:51:07.090 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:51:07.090 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:35.935264+0000", 2024-04-23T16:51:07.091 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7604000000000001e-05 2024-04-23T16:51:07.091 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.091 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:07.091 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.091 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:51:07.091 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:51:07.091 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:51:07.091 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:51:07.119 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:07.375 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:51:07.732 INFO:teuthology.orchestra.run.smithi148.stderr:Exporting 8.5 info 8.5( v 686'15119 (622'5100,686'15119] local-lis/les=609/610 n=337 ec=609/609 lis/c=609/609 les/c/f=610/610/0 sis=609) 2024-04-23T16:51:07.780 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a010b253:::benchmark_data_smithi188_155546_object122864:head# 2024-04-23T16:51:07.781 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a02ae0ac:::benchmark_data_smithi188_155546_object120330:head# 2024-04-23T16:51:07.781 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a02ee131:::benchmark_data_smithi188_155546_object121579:head# 2024-04-23T16:51:07.781 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a04fc086:::benchmark_data_smithi188_155546_object123752:head# 2024-04-23T16:51:07.782 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a057387b:::benchmark_data_smithi188_155546_object123860:head# 2024-04-23T16:51:07.782 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a06eb75a:::benchmark_data_smithi188_155546_object120177:head# 2024-04-23T16:51:07.782 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a07b990c:::benchmark_data_smithi188_155546_object123914:head# 2024-04-23T16:51:07.783 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a0802f6a:::benchmark_data_smithi188_155546_object124986:head# 2024-04-23T16:51:07.783 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a080daca:::benchmark_data_smithi188_155546_object120734:head# 2024-04-23T16:51:07.784 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a086bb45:::benchmark_data_smithi188_155546_object124127:head# 2024-04-23T16:51:07.784 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a0900182:::benchmark_data_smithi188_155546_object122698:head# 2024-04-23T16:51:07.784 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a0981296:::benchmark_data_smithi188_155546_object121158:head# 2024-04-23T16:51:07.785 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a0a3c42f:::benchmark_data_smithi188_155546_object125052:head# 2024-04-23T16:51:07.785 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a0abf622:::benchmark_data_smithi188_155546_object123049:head# 2024-04-23T16:51:07.786 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a0cccd85:::benchmark_data_smithi188_155546_object123425:head# 2024-04-23T16:51:07.786 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a0d1bc10:::benchmark_data_smithi188_155546_object122934:head# 2024-04-23T16:51:07.786 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a0ec5cd2:::benchmark_data_smithi188_155546_object123649:head# 2024-04-23T16:51:07.787 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a0f45659:::benchmark_data_smithi188_155546_object119812:head# 2024-04-23T16:51:07.788 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a0f4654b:::benchmark_data_smithi188_155546_object123351:head# 2024-04-23T16:51:07.788 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a0ff3eda:::benchmark_data_smithi188_155546_object122301:head# 2024-04-23T16:51:07.789 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a10443e6:::benchmark_data_smithi188_155546_object123256:head# 2024-04-23T16:51:07.789 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a127bc8c:::benchmark_data_smithi188_155546_object120285:head# 2024-04-23T16:51:07.790 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a12996f4:::benchmark_data_smithi188_155546_object121208:head# 2024-04-23T16:51:07.790 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a1460e7f:::benchmark_data_smithi188_155546_object123171:head# 2024-04-23T16:51:07.790 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a14a66d1:::benchmark_data_smithi188_155546_object124965:head# 2024-04-23T16:51:07.791 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a16d1690:::benchmark_data_smithi188_155546_object124875:head# 2024-04-23T16:51:07.791 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a1725ca4:::benchmark_data_smithi188_155546_object121669:head# 2024-04-23T16:51:07.792 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a17ee3e2:::benchmark_data_smithi188_155546_object123892:head# 2024-04-23T16:51:07.792 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a18232cd:::benchmark_data_smithi188_155546_object123908:head# 2024-04-23T16:51:07.793 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a1a59fd6:::benchmark_data_smithi188_155546_object120665:head# 2024-04-23T16:51:07.793 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a1b59251:::benchmark_data_smithi188_155546_object123165:head# 2024-04-23T16:51:07.793 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a1bb51cd:::benchmark_data_smithi188_155546_object119869:head# 2024-04-23T16:51:07.794 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a1cdc045:::benchmark_data_smithi188_155546_object120569:head# 2024-04-23T16:51:07.794 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a1d5858b:::benchmark_data_smithi188_155546_object125430:head# 2024-04-23T16:51:07.795 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a1d73841:::benchmark_data_smithi188_155546_object121387:head# 2024-04-23T16:51:07.795 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a1e8693c:::benchmark_data_smithi188_155546_object125494:head# 2024-04-23T16:51:07.796 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a1f9a61c:::benchmark_data_smithi188_155546_object121692:head# 2024-04-23T16:51:07.796 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a22631ac:::benchmark_data_smithi188_155546_object119829:head# 2024-04-23T16:51:07.798 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a22f8662:::benchmark_data_smithi188_155546_object120716:head# 2024-04-23T16:51:07.799 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a22f8d74:::benchmark_data_smithi188_155546_object124226:head# 2024-04-23T16:51:07.799 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a234813d:::benchmark_data_smithi188_155546_object120712:head# 2024-04-23T16:51:07.799 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a24bf6ed:::benchmark_data_smithi188_155546_object120417:head# 2024-04-23T16:51:07.799 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a2690a27:::benchmark_data_smithi188_155546_object120823:head# 2024-04-23T16:51:07.799 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a29387de:::benchmark_data_smithi188_155546_object122335:head# 2024-04-23T16:51:07.800 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a2a14a20:::benchmark_data_smithi188_155546_object124052:head# 2024-04-23T16:51:07.800 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a2a28a8f:::benchmark_data_smithi188_155546_object124179:head# 2024-04-23T16:51:07.800 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a2c8df32:::benchmark_data_smithi188_155546_object122051:head# 2024-04-23T16:51:07.800 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a2f5dd01:::benchmark_data_smithi188_155546_object125176:head# 2024-04-23T16:51:07.800 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a2fe209d:::benchmark_data_smithi188_155546_object124630:head# 2024-04-23T16:51:07.801 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a2feeb32:::benchmark_data_smithi188_155546_object122985:head# 2024-04-23T16:51:07.802 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a30b6ed6:::benchmark_data_smithi188_155546_object125177:head# 2024-04-23T16:51:07.802 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a3179e07:::benchmark_data_smithi188_155546_object119886:head# 2024-04-23T16:51:07.802 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a31c0bff:::benchmark_data_smithi188_155546_object124672:head# 2024-04-23T16:51:07.803 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a31e9f1e:::benchmark_data_smithi188_155546_object124075:head# 2024-04-23T16:51:07.803 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a32ccd06:::benchmark_data_smithi188_155546_object120718:head# 2024-04-23T16:51:07.804 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a338f55b:::benchmark_data_smithi188_155546_object119687:head# 2024-04-23T16:51:07.804 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a33d1717:::benchmark_data_smithi188_155546_object120608:head# 2024-04-23T16:51:07.805 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a3499455:::benchmark_data_smithi188_155546_object123972:head# 2024-04-23T16:51:07.806 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a3562320:::benchmark_data_smithi188_155546_object121773:head# 2024-04-23T16:51:07.806 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a35b7459:::benchmark_data_smithi188_155546_object125192:head# 2024-04-23T16:51:07.806 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a37bca0a:::benchmark_data_smithi188_155546_object123224:head# 2024-04-23T16:51:07.807 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a37efe38:::benchmark_data_smithi188_155546_object120484:head# 2024-04-23T16:51:07.807 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a384c988:::benchmark_data_smithi188_155546_object123597:head# 2024-04-23T16:51:07.808 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a3875196:::benchmark_data_smithi188_155546_object119612:head# 2024-04-23T16:51:07.808 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a39b7e91:::benchmark_data_smithi188_155546_object124483:head# 2024-04-23T16:51:07.809 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a3a10be2:::benchmark_data_smithi188_155546_object123388:head# 2024-04-23T16:51:07.809 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a3a75561:::benchmark_data_smithi188_155546_object124991:head# 2024-04-23T16:51:07.809 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a3a889ab:::benchmark_data_smithi188_155546_object124170:head# 2024-04-23T16:51:07.810 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a3ac4fdc:::benchmark_data_smithi188_155546_object121742:head# 2024-04-23T16:51:07.810 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a3b0ee73:::benchmark_data_smithi188_155546_object121125:head# 2024-04-23T16:51:07.811 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a3b4fb05:::benchmark_data_smithi188_155546_object120880:head# 2024-04-23T16:51:07.811 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a3bdc21c:::benchmark_data_smithi188_155546_object120017:head# 2024-04-23T16:51:07.811 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a3c70747:::benchmark_data_smithi188_155546_object125308:head# 2024-04-23T16:51:07.812 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a3e79407:::benchmark_data_smithi188_155546_object120214:head# 2024-04-23T16:51:07.812 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a40fbf03:::benchmark_data_smithi188_155546_object120385:head# 2024-04-23T16:51:07.812 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a4221b2e:::benchmark_data_smithi188_155546_object124030:head# 2024-04-23T16:51:07.813 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a434bc03:::benchmark_data_smithi188_155546_object122318:head# 2024-04-23T16:51:07.814 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a43983fa:::benchmark_data_smithi188_155546_object123577:head# 2024-04-23T16:51:07.814 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a43c1850:::benchmark_data_smithi188_155546_object119793:head# 2024-04-23T16:51:07.814 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a4416623:::benchmark_data_smithi188_155546_object121273:head# 2024-04-23T16:51:07.815 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a4515d25:::benchmark_data_smithi188_155546_object125472:head# 2024-04-23T16:51:07.815 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a45e1097:::benchmark_data_smithi188_155546_object122019:head# 2024-04-23T16:51:07.816 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a46a1894:::benchmark_data_smithi188_155546_object121501:head# 2024-04-23T16:51:07.816 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a47b7454:::benchmark_data_smithi188_155546_object122667:head# 2024-04-23T16:51:07.816 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a47e3af8:::benchmark_data_smithi188_155546_object121253:head# 2024-04-23T16:51:07.817 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a48c1601:::benchmark_data_smithi188_155546_object120542:head# 2024-04-23T16:51:07.817 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a4b01365:::benchmark_data_smithi188_155546_object121530:head# 2024-04-23T16:51:07.818 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a4b7917b:::benchmark_data_smithi188_155546_object123557:head# 2024-04-23T16:51:07.818 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a4bf9d61:::benchmark_data_smithi188_155546_object120302:head# 2024-04-23T16:51:07.819 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a4cc79bf:::benchmark_data_smithi188_155546_object124019:head# 2024-04-23T16:51:07.819 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a4d2e5be:::benchmark_data_smithi188_155546_object119889:head# 2024-04-23T16:51:07.819 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a4efde96:::benchmark_data_smithi188_155546_object122310:head# 2024-04-23T16:51:07.820 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a4fd2b64:::benchmark_data_smithi188_155546_object123729:head# 2024-04-23T16:51:07.820 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a5023974:::benchmark_data_smithi188_155546_object123045:head# 2024-04-23T16:51:07.820 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a50fde17:::benchmark_data_smithi188_155546_object125282:head# 2024-04-23T16:51:07.821 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a5119228:::benchmark_data_smithi188_155546_object123475:head# 2024-04-23T16:51:07.821 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a51f4eea:::benchmark_data_smithi188_155546_object123264:head# 2024-04-23T16:51:07.822 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a5210cc7:::benchmark_data_smithi188_155546_object120362:head# 2024-04-23T16:51:07.822 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a52e2bbe:::benchmark_data_smithi188_155546_object123839:head# 2024-04-23T16:51:07.823 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a5486cb6:::benchmark_data_smithi188_155546_object122433:head# 2024-04-23T16:51:07.823 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a5542243:::benchmark_data_smithi188_155546_object123113:head# 2024-04-23T16:51:07.824 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a55b3379:::benchmark_data_smithi188_155546_object122487:head# 2024-04-23T16:51:07.824 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a55d607a:::benchmark_data_smithi188_155546_object122589:head# 2024-04-23T16:51:07.824 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a55f4a6f:::benchmark_data_smithi188_155546_object124902:head# 2024-04-23T16:51:07.825 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a577510d:::benchmark_data_smithi188_155546_object120239:head# 2024-04-23T16:51:07.826 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a57fbd4c:::benchmark_data_smithi188_155546_object122141:head# 2024-04-23T16:51:07.826 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a58ae4ff:::benchmark_data_smithi188_155546_object124951:head# 2024-04-23T16:51:07.826 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a58f284d:::benchmark_data_smithi188_155546_object120467:head# 2024-04-23T16:51:07.827 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a597e64e:::benchmark_data_smithi188_155546_object122499:head# 2024-04-23T16:51:07.827 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a5aefa24:::benchmark_data_smithi188_155546_object120922:head# 2024-04-23T16:51:07.828 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a5b6849d:::benchmark_data_smithi188_155546_object121661:head# 2024-04-23T16:51:07.828 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a5c19da9:::benchmark_data_smithi188_155546_object123036:head# 2024-04-23T16:51:07.829 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a5c6a168:::benchmark_data_smithi188_155546_object122219:head# 2024-04-23T16:51:07.829 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a5caecb5:::benchmark_data_smithi188_155546_object124350:head# 2024-04-23T16:51:07.831 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a5ceff17:::benchmark_data_smithi188_155546_object120828:head# 2024-04-23T16:51:07.832 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a5d87991:::benchmark_data_smithi188_155546_object124310:head# 2024-04-23T16:51:07.832 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a5e07be2:::benchmark_data_smithi188_155546_object120518:head# 2024-04-23T16:51:07.833 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a5e963ed:::benchmark_data_smithi188_155546_object120453:head# 2024-04-23T16:51:07.833 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a60cfbab:::benchmark_data_smithi188_155546_object120451:head# 2024-04-23T16:51:07.834 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a6115565:::benchmark_data_smithi188_155546_object125501:head# 2024-04-23T16:51:07.835 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a6471537:::benchmark_data_smithi188_155546_object120354:head# 2024-04-23T16:51:07.835 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a65075ef:::benchmark_data_smithi188_155546_object125247:head# 2024-04-23T16:51:07.835 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a6522ce6:::benchmark_data_smithi188_155546_object120274:head# 2024-04-23T16:51:07.836 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a65ca50b:::benchmark_data_smithi188_155546_object120752:head# 2024-04-23T16:51:07.836 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a6602f03:::benchmark_data_smithi188_155546_object123146:head# 2024-04-23T16:51:07.836 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a663adf1:::benchmark_data_smithi188_155546_object125268:head# 2024-04-23T16:51:07.836 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a663dc20:::benchmark_data_smithi188_155546_object122248:head# 2024-04-23T16:51:07.836 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a678bd21:::benchmark_data_smithi188_155546_object125159:head# 2024-04-23T16:51:07.837 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a68287be:::benchmark_data_smithi188_155546_object125236:head# 2024-04-23T16:51:07.837 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a694170d:::benchmark_data_smithi188_155546_object125380:head# 2024-04-23T16:51:07.838 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a6981e93:::benchmark_data_smithi188_155546_object122459:head# 2024-04-23T16:51:07.838 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a6aad106:::benchmark_data_smithi188_155546_object119880:head# 2024-04-23T16:51:07.839 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a6b8b2c0:::benchmark_data_smithi188_155546_object124351:head# 2024-04-23T16:51:07.839 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a6c9b63b:::benchmark_data_smithi188_155546_object124287:head# 2024-04-23T16:51:07.840 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a6cbb757:::benchmark_data_smithi188_155546_object124493:head# 2024-04-23T16:51:07.840 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a6d1bc7c:::benchmark_data_smithi188_155546_object120847:head# 2024-04-23T16:51:07.841 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a6d2b095:::benchmark_data_smithi188_155546_object121244:head# 2024-04-23T16:51:07.841 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a6d47c17:::benchmark_data_smithi188_155546_object125033:head# 2024-04-23T16:51:07.841 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a6dd1e3b:::benchmark_data_smithi188_155546_object122067:head# 2024-04-23T16:51:07.842 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a6e2a766:::benchmark_data_smithi188_155546_object121602:head# 2024-04-23T16:51:07.842 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a6f04630:::benchmark_data_smithi188_155546_object120216:head# 2024-04-23T16:51:07.843 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a70210a2:::benchmark_data_smithi188_155546_object122771:head# 2024-04-23T16:51:07.843 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a7068b14:::benchmark_data_smithi188_155546_object123160:head# 2024-04-23T16:51:07.843 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a713cb4e:::benchmark_data_smithi188_155546_object124463:head# 2024-04-23T16:51:07.844 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a718a683:::benchmark_data_smithi188_155546_object120145:head# 2024-04-23T16:51:07.844 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a7296ab1:::benchmark_data_smithi188_155546_object121984:head# 2024-04-23T16:51:07.845 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a7332781:::benchmark_data_smithi188_155546_object120654:head# 2024-04-23T16:51:07.845 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a734ee88:::benchmark_data_smithi188_155546_object122467:head# 2024-04-23T16:51:07.846 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a747b93a:::benchmark_data_smithi188_155546_object120942:head# 2024-04-23T16:51:07.846 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a7496e54:::benchmark_data_smithi188_155546_object125388:head# 2024-04-23T16:51:07.846 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a74f0f60:::benchmark_data_smithi188_155546_object119990:head# 2024-04-23T16:51:07.847 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a77b773c:::benchmark_data_smithi188_155546_object120472:head# 2024-04-23T16:51:07.848 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a785cea4:::benchmark_data_smithi188_155546_object123819:head# 2024-04-23T16:51:07.848 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a786f82c:::benchmark_data_smithi188_155546_object120193:head# 2024-04-23T16:51:07.849 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a789c6ea:::benchmark_data_smithi188_155546_object123488:head# 2024-04-23T16:51:07.849 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a79b6c99:::benchmark_data_smithi188_155546_object124036:head# 2024-04-23T16:51:07.849 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a7a0e359:::benchmark_data_smithi188_155546_object120644:head# 2024-04-23T16:51:07.849 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a7abb061:::benchmark_data_smithi188_155546_object123487:head# 2024-04-23T16:51:07.850 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a7ba9df8:::benchmark_data_smithi188_155546_object121826:head# 2024-04-23T16:51:07.850 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a7c2cfb3:::benchmark_data_smithi188_155546_object120759:head# 2024-04-23T16:51:07.851 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a7c3b8a4:::benchmark_data_smithi188_155546_object120020:head# 2024-04-23T16:51:07.851 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a7db6ece:::benchmark_data_smithi188_155546_object124866:head# 2024-04-23T16:51:07.852 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a7e3e7cd:::benchmark_data_smithi188_155546_object120474:head# 2024-04-23T16:51:07.853 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a7ff2ea0:::benchmark_data_smithi188_155546_object122398:head# 2024-04-23T16:51:07.853 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a80d81c2:::benchmark_data_smithi188_155546_object123046:head# 2024-04-23T16:51:07.853 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a854df89:::benchmark_data_smithi188_155546_object123638:head# 2024-04-23T16:51:07.854 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a858c814:::benchmark_data_smithi188_155546_object120456:head# 2024-04-23T16:51:07.854 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a85b1ace:::benchmark_data_smithi188_155546_object121117:head# 2024-04-23T16:51:07.855 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a85d628d:::benchmark_data_smithi188_155546_object120967:head# 2024-04-23T16:51:07.855 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a862caf8:::benchmark_data_smithi188_155546_object120180:head# 2024-04-23T16:51:07.856 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a86e7fe2:::benchmark_data_smithi188_155546_object120697:head# 2024-04-23T16:51:07.856 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a87661c3:::benchmark_data_smithi188_155546_object120725:head# 2024-04-23T16:51:07.857 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a8a17ccc:::benchmark_data_smithi188_155546_object121013:head# 2024-04-23T16:51:07.857 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a8b5d02f:::benchmark_data_smithi188_155546_object119923:head# 2024-04-23T16:51:07.858 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a8c008ce:::benchmark_data_smithi188_155546_object123705:head# 2024-04-23T16:51:07.859 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a8d21a66:::benchmark_data_smithi188_155546_object124204:head# 2024-04-23T16:51:07.859 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a8d71dd6:::benchmark_data_smithi188_155546_object122703:head# 2024-04-23T16:51:07.859 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a8df47f3:::benchmark_data_smithi188_155546_object124624:head# 2024-04-23T16:51:07.860 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a8e5bf29:::benchmark_data_smithi188_155546_object124833:head# 2024-04-23T16:51:07.861 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a9272355:::benchmark_data_smithi188_155546_object121600:head# 2024-04-23T16:51:07.861 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a933108c:::benchmark_data_smithi188_155546_object122602:head# 2024-04-23T16:51:07.861 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a94064c6:::benchmark_data_smithi188_155546_object123738:head# 2024-04-23T16:51:07.862 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a9477d0e:::benchmark_data_smithi188_155546_object121085:head# 2024-04-23T16:51:07.862 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a9541e1f:::benchmark_data_smithi188_155546_object123921:head# 2024-04-23T16:51:07.862 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a9554b0e:::benchmark_data_smithi188_155546_object120516:head# 2024-04-23T16:51:07.863 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a95fa950:::benchmark_data_smithi188_155546_object121256:head# 2024-04-23T16:51:07.863 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a960c10c:::benchmark_data_smithi188_155546_object121324:head# 2024-04-23T16:51:07.864 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a96b033b:::benchmark_data_smithi188_155546_object122760:head# 2024-04-23T16:51:07.865 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a977943b:::benchmark_data_smithi188_155546_object123928:head# 2024-04-23T16:51:07.865 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a98890c5:::benchmark_data_smithi188_155546_object122726:head# 2024-04-23T16:51:07.866 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a988c59f:::benchmark_data_smithi188_155546_object123774:head# 2024-04-23T16:51:07.866 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a988c66b:::benchmark_data_smithi188_155546_object124192:head# 2024-04-23T16:51:07.867 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a99a210d:::benchmark_data_smithi188_155546_object123728:head# 2024-04-23T16:51:07.867 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a9a940c2:::benchmark_data_smithi188_155546_object123799:head# 2024-04-23T16:51:07.868 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a9b40d63:::benchmark_data_smithi188_155546_object123909:head# 2024-04-23T16:51:07.868 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a9b7e442:::benchmark_data_smithi188_155546_object123485:head# 2024-04-23T16:51:07.869 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a9b97fac:::benchmark_data_smithi188_155546_object123787:head# 2024-04-23T16:51:07.869 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a9c2be40:::benchmark_data_smithi188_155546_object121369:head# 2024-04-23T16:51:07.869 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a9c4fab8:::benchmark_data_smithi188_155546_object124334:head# 2024-04-23T16:51:07.870 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a9d0e209:::benchmark_data_smithi188_155546_object123759:head# 2024-04-23T16:51:07.871 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a9d872a1:::benchmark_data_smithi188_155546_object119956:head# 2024-04-23T16:51:07.871 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a9d98110:::benchmark_data_smithi188_155546_object122475:head# 2024-04-23T16:51:07.871 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a9d9df1d:::benchmark_data_smithi188_155546_object124621:head# 2024-04-23T16:51:07.872 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a9dac822:::benchmark_data_smithi188_155546_object122749:head# 2024-04-23T16:51:07.872 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a9e5ffaa:::benchmark_data_smithi188_155546_object124381:head# 2024-04-23T16:51:07.873 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a9e9dc9e:::benchmark_data_smithi188_155546_object122523:head# 2024-04-23T16:51:07.873 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a9f08464:::benchmark_data_smithi188_155546_object119790:head# 2024-04-23T16:51:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:a9f740ff:::benchmark_data_smithi188_155546_object121662:head# 2024-04-23T16:51:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aa2306b4:::benchmark_data_smithi188_155546_object122133:head# 2024-04-23T16:51:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aa29a284:::benchmark_data_smithi188_155546_object121308:head# 2024-04-23T16:51:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aa2f6347:::benchmark_data_smithi188_155546_object124993:head# 2024-04-23T16:51:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aa5b1e6f:::benchmark_data_smithi188_155546_object121001:head# 2024-04-23T16:51:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aa64d5c7:::benchmark_data_smithi188_155546_object124683:head# 2024-04-23T16:51:07.877 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aa8aad9c:::benchmark_data_smithi188_155546_object124682:head# 2024-04-23T16:51:07.877 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aa8beaf7:::benchmark_data_smithi188_155546_object121658:head# 2024-04-23T16:51:07.877 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aa96296f:::benchmark_data_smithi188_155546_object124257:head# 2024-04-23T16:51:07.878 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aad13d4e:::benchmark_data_smithi188_155546_object125424:head# 2024-04-23T16:51:07.878 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aada83ab:::benchmark_data_smithi188_155546_object123064:head# 2024-04-23T16:51:07.880 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aadc7b00:::benchmark_data_smithi188_155546_object121410:head# 2024-04-23T16:51:07.880 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aae65f16:::benchmark_data_smithi188_155546_object119924:head# 2024-04-23T16:51:07.880 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aaf68894:::benchmark_data_smithi188_155546_object123087:head# 2024-04-23T16:51:07.880 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab020dbb:::benchmark_data_smithi188_155546_object124312:head# 2024-04-23T16:51:07.880 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab04bef1:::benchmark_data_smithi188_155546_object119915:head# 2024-04-23T16:51:07.881 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab19e7dc:::benchmark_data_smithi188_155546_object120217:head# 2024-04-23T16:51:07.882 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab1c8dff:::benchmark_data_smithi188_155546_object125217:head# 2024-04-23T16:51:07.882 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab1ec853:::benchmark_data_smithi188_155546_object121400:head# 2024-04-23T16:51:07.882 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab298846:::benchmark_data_smithi188_155546_object124990:head# 2024-04-23T16:51:07.883 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab29b8a3:::benchmark_data_smithi188_155546_object120454:head# 2024-04-23T16:51:07.883 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab2b27ca:::benchmark_data_smithi188_155546_object120332:head# 2024-04-23T16:51:07.884 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab2f6801:::benchmark_data_smithi188_155546_object124367:head# 2024-04-23T16:51:07.884 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab344a99:::benchmark_data_smithi188_155546_object123163:head# 2024-04-23T16:51:07.885 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab38dcbd:::benchmark_data_smithi188_155546_object121265:head# 2024-04-23T16:51:07.885 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab4317f3:::benchmark_data_smithi188_155546_object120793:head# 2024-04-23T16:51:07.885 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab4d5cb1:::benchmark_data_smithi188_155546_object123135:head# 2024-04-23T16:51:07.886 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab4f6298:::benchmark_data_smithi188_155546_object124262:head# 2024-04-23T16:51:07.887 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab5600d5:::benchmark_data_smithi188_155546_object121946:head# 2024-04-23T16:51:07.887 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab5a5225:::benchmark_data_smithi188_155546_object124664:head# 2024-04-23T16:51:07.887 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab71f68d:::benchmark_data_smithi188_155546_object121488:head# 2024-04-23T16:51:07.887 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab71fbbe:::benchmark_data_smithi188_155546_object121598:head# 2024-04-23T16:51:07.889 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab7a3453:::benchmark_data_smithi188_155546_object119890:head# 2024-04-23T16:51:07.889 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab899e53:::benchmark_data_smithi188_155546_object124700:head# 2024-04-23T16:51:07.889 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab8a4b51:::benchmark_data_smithi188_155546_object120795:head# 2024-04-23T16:51:07.889 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ab9c9515:::benchmark_data_smithi188_155546_object121631:head# 2024-04-23T16:51:07.889 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:abb4b75e:::benchmark_data_smithi188_155546_object120309:head# 2024-04-23T16:51:07.891 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:abb6fe31:::benchmark_data_smithi188_155546_object122772:head# 2024-04-23T16:51:07.891 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:abca7314:::benchmark_data_smithi188_155546_object121760:head# 2024-04-23T16:51:07.892 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:abcd2ad7:::benchmark_data_smithi188_155546_object122967:head# 2024-04-23T16:51:07.892 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:abd2115b:::benchmark_data_smithi188_155546_object124020:head# 2024-04-23T16:51:07.892 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:abdc268f:::benchmark_data_smithi188_155546_object121680:head# 2024-04-23T16:51:07.892 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:abdcd0fb:::benchmark_data_smithi188_155546_object119725:head# 2024-04-23T16:51:07.894 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:abe59634:::benchmark_data_smithi188_155546_object123674:head# 2024-04-23T16:51:07.894 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:abe82858:::benchmark_data_smithi188_155546_object125242:head# 2024-04-23T16:51:07.895 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ac02d5e4:::benchmark_data_smithi188_155546_object120408:head# 2024-04-23T16:51:07.895 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ac08f6cc:::benchmark_data_smithi188_155546_object119701:head# 2024-04-23T16:51:07.895 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ac1742ec:::benchmark_data_smithi188_155546_object121367:head# 2024-04-23T16:51:07.895 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ac2132b8:::benchmark_data_smithi188_155546_object124345:head# 2024-04-23T16:51:07.895 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ac2e376f:::benchmark_data_smithi188_155546_object122722:head# 2024-04-23T16:51:07.895 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ac3ada00:::benchmark_data_smithi188_155546_object121643:head# 2024-04-23T16:51:07.897 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ac4f152d:::benchmark_data_smithi188_155546_object122264:head# 2024-04-23T16:51:07.898 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ac6060e4:::benchmark_data_smithi188_155546_object125466:head# 2024-04-23T16:51:07.898 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ac6a2201:::benchmark_data_smithi188_155546_object121014:head# 2024-04-23T16:51:07.898 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ac71a317:::benchmark_data_smithi188_155546_object122193:head# 2024-04-23T16:51:07.898 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ac72026c:::benchmark_data_smithi188_155546_object119820:head# 2024-04-23T16:51:07.898 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ac769816:::benchmark_data_smithi188_155546_object121493:head# 2024-04-23T16:51:07.899 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ac7d22a0:::benchmark_data_smithi188_155546_object121220:head# 2024-04-23T16:51:07.901 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ac906305:::benchmark_data_smithi188_155546_object121455:head# 2024-04-23T16:51:07.901 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ac99ad76:::benchmark_data_smithi188_155546_object119810:head# 2024-04-23T16:51:07.901 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ac9d035b:::benchmark_data_smithi188_155546_object125056:head# 2024-04-23T16:51:07.902 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:acafca62:::benchmark_data_smithi188_155546_object123878:head# 2024-04-23T16:51:07.902 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:acb0c1c4:::benchmark_data_smithi188_155546_object120577:head# 2024-04-23T16:51:07.902 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:acba7d0f:::benchmark_data_smithi188_155546_object124336:head# 2024-04-23T16:51:07.902 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:acc0e2ec:::benchmark_data_smithi188_155546_object122270:head# 2024-04-23T16:51:07.902 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:acc20528:::benchmark_data_smithi188_155546_object121171:head# 2024-04-23T16:51:07.902 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:acc3295a:::benchmark_data_smithi188_155546_object120689:head# 2024-04-23T16:51:07.903 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:acc80e3d:::benchmark_data_smithi188_155546_object120550:head# 2024-04-23T16:51:07.903 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:acceb0f9:::benchmark_data_smithi188_155546_object120920:head# 2024-04-23T16:51:07.903 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:acda7db5:::benchmark_data_smithi188_155546_object122319:head# 2024-04-23T16:51:07.903 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:acfffd02:::benchmark_data_smithi188_155546_object123440:head# 2024-04-23T16:51:07.904 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ad00cb16:::benchmark_data_smithi188_155546_object125043:head# 2024-04-23T16:51:07.904 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ad016c28:::benchmark_data_smithi188_155546_object122826:head# 2024-04-23T16:51:07.904 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ad018937:::benchmark_data_smithi188_155546_object123335:head# 2024-04-23T16:51:07.904 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ad110119:::benchmark_data_smithi188_155546_object121973:head# 2024-04-23T16:51:07.905 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ad1b5ba0:::benchmark_data_smithi188_155546_object125251:head# 2024-04-23T16:51:07.905 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ad1c6046:::benchmark_data_smithi188_155546_object124592:head# 2024-04-23T16:51:07.905 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ad20dace:::benchmark_data_smithi188_155546_object122085:head# 2024-04-23T16:51:07.906 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ad20eb4b:::benchmark_data_smithi188_155546_object121504:head# 2024-04-23T16:51:07.906 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ad3c3d86:::benchmark_data_smithi188_155546_object120899:head# 2024-04-23T16:51:07.907 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ad3d6922:::benchmark_data_smithi188_155546_object123806:head# 2024-04-23T16:51:07.907 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ad416d37:::benchmark_data_smithi188_155546_object120012:head# 2024-04-23T16:51:07.908 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ad4a2f6c:::benchmark_data_smithi188_155546_object121779:head# 2024-04-23T16:51:07.908 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ad5d462c:::benchmark_data_smithi188_155546_object124853:head# 2024-04-23T16:51:07.909 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ad78ddf3:::benchmark_data_smithi188_155546_object122196:head# 2024-04-23T16:51:07.909 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ad86fb1f:::benchmark_data_smithi188_155546_object119933:head# 2024-04-23T16:51:07.909 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:adb65a2e:::benchmark_data_smithi188_155546_object121549:head# 2024-04-23T16:51:07.910 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:adc32291:::benchmark_data_smithi188_155546_object119815:head# 2024-04-23T16:51:07.910 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:adc56a4a:::benchmark_data_smithi188_155546_object120024:head# 2024-04-23T16:51:07.910 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:adde2c25:::benchmark_data_smithi188_155546_object122350:head# 2024-04-23T16:51:07.911 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ae1c2ec2:::benchmark_data_smithi188_155546_object123513:head# 2024-04-23T16:51:07.912 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ae1df1c1:::benchmark_data_smithi188_155546_object122208:head# 2024-04-23T16:51:07.913 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ae26c4df:::benchmark_data_smithi188_155546_object120172:head# 2024-04-23T16:51:07.913 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ae2eb1a3:::benchmark_data_smithi188_155546_object123260:head# 2024-04-23T16:51:07.913 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ae414d44:::benchmark_data_smithi188_155546_object122816:head# 2024-04-23T16:51:07.914 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ae5ffb07:::benchmark_data_smithi188_155546_object122862:head# 2024-04-23T16:51:07.914 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ae72b275:::benchmark_data_smithi188_155546_object120052:head# 2024-04-23T16:51:07.914 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ae97cba2:::benchmark_data_smithi188_155546_object119928:head# 2024-04-23T16:51:07.915 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ae9b8bb3:::benchmark_data_smithi188_155546_object124985:head# 2024-04-23T16:51:07.915 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:ae9bfd19:::benchmark_data_smithi188_155546_object119818:head# 2024-04-23T16:51:07.916 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aea9902e:::benchmark_data_smithi188_155546_object122525:head# 2024-04-23T16:51:07.916 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aeaba9af:::benchmark_data_smithi188_155546_object119969:head# 2024-04-23T16:51:07.917 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aeace135:::benchmark_data_smithi188_155546_object119909:head# 2024-04-23T16:51:07.917 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aec2c9aa:::benchmark_data_smithi188_155546_object124309:head# 2024-04-23T16:51:07.917 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aec71284:::benchmark_data_smithi188_155546_object124906:head# 2024-04-23T16:51:07.918 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aee32dfa:::benchmark_data_smithi188_155546_object125096:head# 2024-04-23T16:51:07.918 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aee4e238:::benchmark_data_smithi188_155546_object124794:head# 2024-04-23T16:51:07.919 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aeea2c2a:::benchmark_data_smithi188_155546_object122593:head# 2024-04-23T16:51:07.919 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aef6d058:::benchmark_data_smithi188_155546_object122391:head# 2024-04-23T16:51:07.919 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aef79b68:::benchmark_data_smithi188_155546_object123862:head# 2024-04-23T16:51:07.920 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:af08a30f:::benchmark_data_smithi188_155546_object122655:head# 2024-04-23T16:51:07.920 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:af13179e:::benchmark_data_smithi188_155546_object125261:head# 2024-04-23T16:51:07.921 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:af187b09:::benchmark_data_smithi188_155546_object119663:head# 2024-04-23T16:51:07.921 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:af1d22ac:::benchmark_data_smithi188_155546_object125068:head# 2024-04-23T16:51:07.921 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:af1fe105:::benchmark_data_smithi188_155546_object125423:head# 2024-04-23T16:51:07.922 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:af3a2f0c:::benchmark_data_smithi188_155546_object123404:head# 2024-04-23T16:51:07.922 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:af55ad56:::benchmark_data_smithi188_155546_object119757:head# 2024-04-23T16:51:07.922 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:af5e1315:::benchmark_data_smithi188_155546_object120605:head# 2024-04-23T16:51:07.923 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:af5f6758:::benchmark_data_smithi188_155546_object122935:head# 2024-04-23T16:51:07.923 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:af6bedb1:::benchmark_data_smithi188_155546_object121571:head# 2024-04-23T16:51:07.923 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:af7b2bbd:::benchmark_data_smithi188_155546_object123285:head# 2024-04-23T16:51:07.924 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:af7caa7a:::benchmark_data_smithi188_155546_object122561:head# 2024-04-23T16:51:07.924 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:af899c58:::benchmark_data_smithi188_155546_object123499:head# 2024-04-23T16:51:07.925 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:af8c0c22:::benchmark_data_smithi188_155546_object123881:head# 2024-04-23T16:51:07.925 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:afaebb47:::benchmark_data_smithi188_155546_object122341:head# 2024-04-23T16:51:07.925 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:afc38a4a:::benchmark_data_smithi188_155546_object119678:head# 2024-04-23T16:51:07.926 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:afc6daa0:::benchmark_data_smithi188_155546_object122343:head# 2024-04-23T16:51:07.926 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:afdb39bc:::benchmark_data_smithi188_155546_object123309:head# 2024-04-23T16:51:07.927 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:afdd71b0:::benchmark_data_smithi188_155546_object119737:head# 2024-04-23T16:51:07.927 INFO:teuthology.orchestra.run.smithi148.stderr:Read #8:aff38299:::benchmark_data_smithi188_155546_object122738:head# 2024-04-23T16:51:07.929 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-23T16:51:07.932 INFO:teuthology.orchestra.run.smithi148.stderr:Export successful 2024-04-23T16:51:07.969 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:51:07.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:07.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:51:08.007 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:51:08.072 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:51:08.105 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:08.125 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:51:08.126 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:51:08.126 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:51:08.126 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:51:08.126 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:51:08.126 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:51:08.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:08.129 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:51:08.136 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:51:08.155 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:51:08.175 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:51:08.195 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:51:08.198 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:51:08.214 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:51:08.229 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:51:08.234 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:51:08.234 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:51:08.234 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:51:08.234 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:51:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:51:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-23T16:51:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:51:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:51:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:51:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:51:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:51:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:51:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:51:08.252 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:07 smithi103 ceph-mon[39629]: pgmap v1943: 27 pgs: 27 active+clean; 577 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 559 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-04-23T16:51:08.252 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:07 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3170883481' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:51:08.255 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:07 smithi148 ceph-mon[44520]: pgmap v1943: 27 pgs: 27 active+clean; 577 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 559 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-04-23T16:51:08.255 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:07 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3170883481' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:51:08.258 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:08.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:07 smithi188 ceph-mon[44101]: pgmap v1943: 27 pgs: 27 active+clean; 577 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 559 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-04-23T16:51:08.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:07 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3170883481' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:51:09.108 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:51:09.110 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:09 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:51:09.110 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:09 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3170883481' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:51:09.110 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:09 smithi148 ceph-mon[44520]: osdmap e698: 12 total, 11 up, 7 in 2024-04-23T16:51:09.110 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:09 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4036224436' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:09.110 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.110 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.110 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.111 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.111 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.111 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.111 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.111 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.111 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:09 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:09.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:51:09.212 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:51:09.245 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:09.266 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:51:09.268 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:09 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:51:09.268 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:09 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3170883481' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:51:09.268 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:09 smithi188 ceph-mon[44101]: osdmap e698: 12 total, 11 up, 7 in 2024-04-23T16:51:09.268 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:09 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4036224436' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:09.268 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.268 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.268 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.269 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.269 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.269 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.269 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.269 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.269 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:09 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:09.270 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:51:09.298 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:08 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:51:09.298 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:08 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3170883481' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:51:09.298 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:08 smithi103 ceph-mon[39629]: osdmap e698: 12 total, 11 up, 7 in 2024-04-23T16:51:09.298 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:08 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4036224436' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:09.298 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.299 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.299 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.299 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.299 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.299 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.299 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.299 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.299 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:09.371 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:51:09.403 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:09.520 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.4 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --force --op remove --pgid 8.5 2024-04-23T16:51:09.753 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:51:09.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:09.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:51:09.857 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:51:09.889 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:09.911 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:51:09.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:09.914 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:51:10.015 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:51:10.045 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:10.065 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:51:10.066 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.4/config 2024-04-23T16:51:10.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:10.068 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:51:10.169 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:10.196 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:10.196 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:10 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:10.196 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:10 smithi188 ceph-mon[44101]: pgmap v1945: 27 pgs: 27 active+clean; 577 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 341 op/s 2024-04-23T16:51:10.198 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:10.218 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:51:10.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:10.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:51:10.271 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:10.271 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:10 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:10.271 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:10 smithi148 ceph-mon[44520]: pgmap v1945: 27 pgs: 27 active+clean; 577 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 341 op/s 2024-04-23T16:51:10.283 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:10.283 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:10 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:10.283 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:10 smithi103 ceph-mon[39629]: pgmap v1945: 27 pgs: 27 active+clean; 577 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 341 op/s 2024-04-23T16:51:10.321 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:51:10.349 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:10.369 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:51:10.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:10.372 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:51:10.473 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:51:10.505 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:10.528 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:51:10.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:10.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:51:10.633 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:51:10.652 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:51:10.653 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:51:10.661 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:10.860 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:51:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:10.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:51:10.964 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:51:10.991 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:11.369 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:51:11.370 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:51:11.374 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-23T16:51:11.440 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:51:11.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:11.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:51:11.543 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:51:11.570 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:11.720 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:51:11.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:11.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:51:11.824 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:51:11.857 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:11.878 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:51:11.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:11.881 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:51:11.982 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:12.016 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:12.037 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:51:12.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:12.041 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:51:12.142 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:51:12.153 INFO:teuthology.orchestra.run.smithi148.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:51:12.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:12 smithi188 ceph-mon[44101]: pgmap v1946: 27 pgs: 27 active+clean; 577 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 282 op/s 2024-04-23T16:51:12.172 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:12.192 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:51:12.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:12.195 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:51:12.285 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:51:12.296 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:51:12.314 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:12 smithi103 ceph-mon[39629]: pgmap v1946: 27 pgs: 27 active+clean; 577 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 282 op/s 2024-04-23T16:51:12.341 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:12.360 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:51:12.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:12.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:51:12.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:12 smithi148 ceph-mon[44520]: pgmap v1946: 27 pgs: 27 active+clean; 577 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 282 op/s 2024-04-23T16:51:12.465 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:51:12.492 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:12.514 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:51:12.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:12.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:51:12.564 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:51:12.618 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:51:12.649 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:51:12.650 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:12.672 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:51:12.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:12.675 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:51:12.776 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:51:12.805 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:12.825 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:51:12.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:12.828 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:51:12.929 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:51:12.958 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:12.982 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:51:12.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:12.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:51:13.022 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'unique_pool_6' removed 2024-04-23T16:51:13.086 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:51:13.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:13 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3114145759' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:51:13.113 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:13.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:13 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3114145759' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:51:13.131 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:51:13.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:13.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:51:13.235 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:51:13.264 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:13.362 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:51:13.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:13.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:51:13.371 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:51:13.465 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:51:13.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:13 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3114145759' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:51:13.493 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:13.512 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:51:13.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:13.516 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:51:13.617 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:51:13.646 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:13.845 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:51:13.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:13.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:51:13.949 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:51:13.976 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:13.995 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:51:13.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:13.998 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:51:14.099 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:14.121 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:14 smithi148 ceph-mon[44520]: pgmap v1947: 27 pgs: 27 active+clean; 577 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 96 op/s 2024-04-23T16:51:14.121 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:14 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3114145759' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:51:14.121 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:14 smithi148 ceph-mon[44520]: osdmap e699: 12 total, 11 up, 7 in 2024-04-23T16:51:14.121 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:14 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:51:14.122 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:14 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:51:14.126 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:14.145 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:51:14.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:14.147 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:51:14.248 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:51:14.283 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:14.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:14 smithi103 ceph-mon[39629]: pgmap v1947: 27 pgs: 27 active+clean; 577 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 96 op/s 2024-04-23T16:51:14.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:14 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3114145759' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:51:14.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:14 smithi103 ceph-mon[39629]: osdmap e699: 12 total, 11 up, 7 in 2024-04-23T16:51:14.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:14 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:51:14.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:14 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:51:14.381 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:51:14.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:14.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:51:14.399 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:51:14.399 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:51:14.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:14 smithi188 ceph-mon[44101]: pgmap v1947: 27 pgs: 27 active+clean; 577 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 96 op/s 2024-04-23T16:51:14.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:14 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3114145759' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:51:14.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:14 smithi188 ceph-mon[44101]: osdmap e699: 12 total, 11 up, 7 in 2024-04-23T16:51:14.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:14 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:51:14.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:14 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:51:14.485 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:51:14.515 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:14.589 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:51:14.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:14.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:51:14.693 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:51:14.720 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:15.031 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:51:15.039 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:15 smithi148 ceph-mon[44520]: osdmap e700: 12 total, 11 up, 7 in 2024-04-23T16:51:15.039 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:15 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3625320123' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:51:15.044 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-23T16:51:15.044 INFO:tasks.radosbench:Beginning radosbench... 2024-04-23T16:51:15.045 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_7 2024-04-23T16:51:15.045 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_7 16 2024-04-23T16:51:15.065 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-23T16:51:15.075 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:51:15.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:15.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:51:15.178 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:51:15.210 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:15.233 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:51:15.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:15.235 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:51:15.336 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:51:15.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:15 smithi103 ceph-mon[39629]: osdmap e700: 12 total, 11 up, 7 in 2024-04-23T16:51:15.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:15 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3625320123' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:51:15.364 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:15.386 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:51:15.387 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:51:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:15.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:51:15.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:15 smithi188 ceph-mon[44101]: osdmap e700: 12 total, 11 up, 7 in 2024-04-23T16:51:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:15 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3625320123' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:51:15.491 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:51:15.523 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:15.544 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:51:15.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:15.547 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:51:15.648 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:51:15.676 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:15.748 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:51:15.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:15.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:51:15.821 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:51:15.852 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:51:15.860 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:51:15.885 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:15.905 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:51:15.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:15.909 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:51:16.010 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:51:16.029 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:16 smithi103 ceph-mon[39629]: pgmap v1950: 11 pgs: 11 active+clean; 577 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:16.040 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:16.060 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:51:16.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:16.063 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:51:16.165 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:51:16.192 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:16 smithi188 ceph-mon[44101]: pgmap v1950: 11 pgs: 11 active+clean; 577 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:16.193 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:16 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:51:16.193 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:16 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3625320123' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:51:16.193 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:16 smithi188 ceph-mon[44101]: osdmap e701: 12 total, 11 up, 7 in 2024-04-23T16:51:16.194 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:16.214 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:51:16.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:16.217 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:51:16.318 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:51:16.342 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:16 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:51:16.342 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:16 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3625320123' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:51:16.342 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:16 smithi103 ceph-mon[39629]: osdmap e701: 12 total, 11 up, 7 in 2024-04-23T16:51:16.348 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:16 smithi148 ceph-mon[44520]: pgmap v1950: 11 pgs: 11 active+clean; 577 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:16.348 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:16 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:51:16.348 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:16 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3625320123' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:51:16.348 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:16 smithi148 ceph-mon[44520]: osdmap e701: 12 total, 11 up, 7 in 2024-04-23T16:51:16.351 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:16.700 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:51:16.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:16.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:51:16.805 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:51:16.833 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:17.270 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:51:17.270 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:51:17.415 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:51:17.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:17.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:51:17.519 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:51:17.547 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:17.770 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:51:17.896 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-23T16:51:18.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:18 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:51:18.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:18 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:51:18.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:18 smithi103 ceph-mon[39629]: pgmap v1952: 11 pgs: 11 active+clean; 577 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:18.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:18 smithi103 ceph-mon[39629]: osdmap e702: 12 total, 11 up, 7 in 2024-04-23T16:51:18.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:18 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:51:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:18 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:51:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:18 smithi148 ceph-mon[44520]: pgmap v1952: 11 pgs: 11 active+clean; 577 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:18 smithi148 ceph-mon[44520]: osdmap e702: 12 total, 11 up, 7 in 2024-04-23T16:51:18.176 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:51:18.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:18 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:51:18.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:18 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:51:18.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:18 smithi188 ceph-mon[44101]: pgmap v1952: 11 pgs: 11 active+clean; 577 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:18.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:18 smithi188 ceph-mon[44101]: osdmap e702: 12 total, 11 up, 7 in 2024-04-23T16:51:18.296 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:51:18.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:18.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:51:18.372 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:51:18.401 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:51:18.430 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:18.450 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:51:18.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:18.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:51:18.554 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:51:18.584 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:18.604 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:51:18.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:18.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:51:18.708 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:51:18.736 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:18.755 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:51:18.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:18.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:51:18.859 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:51:18.888 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:18.908 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:51:18.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:18.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:51:19.012 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:51:19.044 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:19.089 INFO:teuthology.orchestra.run.smithi148.stdout: marking collection for removal 2024-04-23T16:51:19.116 INFO:teuthology.orchestra.run.smithi148.stdout:setting '_remove' omap key 2024-04-23T16:51:19.117 INFO:teuthology.orchestra.run.smithi148.stdout:finish_remove_pgs 8.5_head removing 8.5 2024-04-23T16:51:19.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:19 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T16:51:19.141 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:19 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T16:51:19.147 INFO:teuthology.orchestra.run.smithi148.stdout:Remove successful 2024-04-23T16:51:19.175 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:51:19.176 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:51:19.176 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:51:19.176 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:51:19.176 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:51:19.194 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:51:19.200 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:51:19.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:19.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:51:19.214 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:51:19.235 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:51:19.254 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:51:19.274 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:51:19.293 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:51:19.294 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:51:19.294 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:51:19.294 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:51:19.294 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:51:19.294 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-23T16:51:19.294 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:51:19.294 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:51:19.294 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:51:19.294 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:51:19.294 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:51:19.295 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:51:19.295 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:51:19.304 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:51:19.313 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:19 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T16:51:19.314 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:19 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T16:51:19.333 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:19.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:19 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T16:51:19.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:19 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T16:51:19.683 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:51:19.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:19.687 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:51:19.788 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:51:19.816 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:20.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:20 smithi103 ceph-mon[39629]: pgmap v1954: 11 pgs: 11 active+clean; 577 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:20.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:20 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3574121883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:20.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:20 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4267711675' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-04-23T16:51:20.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:20 smithi103 ceph-mon[39629]: 1.5 scrub starts 2024-04-23T16:51:20.046 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.053 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'unique_pool_7' created 2024-04-23T16:51:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:20 smithi188 ceph-mon[44101]: pgmap v1954: 11 pgs: 11 active+clean; 577 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:20 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3574121883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:20 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4267711675' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-04-23T16:51:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:20 smithi188 ceph-mon[44101]: 1.5 scrub starts 2024-04-23T16:51:20.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:20 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:20 smithi188 ceph-mon[44101]: 1.5 scrub ok 2024-04-23T16:51:20.298 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:20 smithi148 ceph-mon[44520]: pgmap v1954: 11 pgs: 11 active+clean; 577 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:20.298 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:20 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3574121883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:20.298 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.298 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.298 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.298 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.299 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.299 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:20 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4267711675' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-04-23T16:51:20.299 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:20 smithi148 ceph-mon[44520]: 1.5 scrub starts 2024-04-23T16:51:20.299 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.299 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.299 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.299 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:20 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.299 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:20 smithi148 ceph-mon[44520]: 1.5 scrub ok 2024-04-23T16:51:20.321 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.4 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.8.5.4 2024-04-23T16:51:20.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:20 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:20.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:20 smithi103 ceph-mon[39629]: 1.5 scrub ok 2024-04-23T16:51:20.441 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:51:20.441 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:51:20.441 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-04-23T16:51:20.441 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:51:20.527 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.4/config 2024-04-23T16:51:20.925 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-04-23T16:51:21.139 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-23T16:51:21.221 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:21.221 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:21 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:21.221 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:21 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4267711675' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-04-23T16:51:21.221 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:21 smithi103 ceph-mon[39629]: osdmap e703: 12 total, 11 up, 7 in 2024-04-23T16:51:21.221 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:21 smithi103 ceph-mon[39629]: 1.a scrub starts 2024-04-23T16:51:21.222 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:21 smithi103 ceph-mon[39629]: 1.a scrub ok 2024-04-23T16:51:21.249 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:51:21.250 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:51:21.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:21.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:21 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:21.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:21 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4267711675' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-04-23T16:51:21.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:21 smithi148 ceph-mon[44520]: osdmap e703: 12 total, 11 up, 7 in 2024-04-23T16:51:21.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:21 smithi148 ceph-mon[44520]: 1.a scrub starts 2024-04-23T16:51:21.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:21 smithi148 ceph-mon[44520]: 1.a scrub ok 2024-04-23T16:51:21.403 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:51:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:21 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:21 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4267711675' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-04-23T16:51:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:21 smithi188 ceph-mon[44101]: osdmap e703: 12 total, 11 up, 7 in 2024-04-23T16:51:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:21 smithi188 ceph-mon[44101]: 1.a scrub starts 2024-04-23T16:51:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:21 smithi188 ceph-mon[44101]: 1.a scrub ok 2024-04-23T16:51:21.667 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:51:21.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:21.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:51:21.671 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:51:21.771 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:51:21.798 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:21.848 INFO:teuthology.orchestra.run.smithi148.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:51:21.921 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:51:21.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:21.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:51:22.024 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:51:22.052 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:22.175 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:51:22.179 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:22 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:51:22.179 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:22 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:51:22.179 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:22 smithi103 ceph-mon[39629]: pgmap v1956: 27 pgs: 3 creating+peering, 13 unknown, 11 active+clean; 577 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:22.179 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:22 smithi103 ceph-mon[39629]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:51:22.179 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:22 smithi103 ceph-mon[39629]: osdmap e704: 12 total, 11 up, 7 in 2024-04-23T16:51:22.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:22.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:51:22.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:22 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:51:22.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:22 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:51:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:22 smithi188 ceph-mon[44101]: pgmap v1956: 27 pgs: 3 creating+peering, 13 unknown, 11 active+clean; 577 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:22 smithi188 ceph-mon[44101]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:51:22.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:22 smithi188 ceph-mon[44101]: osdmap e704: 12 total, 11 up, 7 in 2024-04-23T16:51:22.281 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:51:22.311 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:22 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:51:22.311 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:22 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:51:22.311 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:22 smithi148 ceph-mon[44520]: pgmap v1956: 27 pgs: 3 creating+peering, 13 unknown, 11 active+clean; 577 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:22.311 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:22 smithi148 ceph-mon[44520]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T16:51:22.311 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:22 smithi148 ceph-mon[44520]: osdmap e704: 12 total, 11 up, 7 in 2024-04-23T16:51:22.314 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:22.414 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:51:22.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:22.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:51:22.518 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:22.548 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:22.627 INFO:teuthology.orchestra.run.smithi148.stdout:get_pg_num_history pg_num_history pg_num_history(e686 pg_nums {1={301=11},8={609=16}} deleted_pools ) 2024-04-23T16:51:22.628 INFO:teuthology.orchestra.run.smithi148.stdout:Importing pgid 8.5 2024-04-23T16:51:22.637 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a010b253:::benchmark_data_smithi188_155546_object122864:head# 2024-04-23T16:51:22.637 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.638 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a02ae0ac:::benchmark_data_smithi188_155546_object120330:head# 2024-04-23T16:51:22.638 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.638 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a02ee131:::benchmark_data_smithi188_155546_object121579:head# 2024-04-23T16:51:22.638 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.639 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a04fc086:::benchmark_data_smithi188_155546_object123752:head# 2024-04-23T16:51:22.639 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.639 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a057387b:::benchmark_data_smithi188_155546_object123860:head# 2024-04-23T16:51:22.639 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.639 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a06eb75a:::benchmark_data_smithi188_155546_object120177:head# 2024-04-23T16:51:22.639 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.639 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a07b990c:::benchmark_data_smithi188_155546_object123914:head# 2024-04-23T16:51:22.639 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.640 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a0802f6a:::benchmark_data_smithi188_155546_object124986:head# 2024-04-23T16:51:22.640 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.641 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a080daca:::benchmark_data_smithi188_155546_object120734:head# 2024-04-23T16:51:22.641 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.641 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a086bb45:::benchmark_data_smithi188_155546_object124127:head# 2024-04-23T16:51:22.641 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.642 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a0900182:::benchmark_data_smithi188_155546_object122698:head# 2024-04-23T16:51:22.642 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.642 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a0981296:::benchmark_data_smithi188_155546_object121158:head# 2024-04-23T16:51:22.642 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.642 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a0a3c42f:::benchmark_data_smithi188_155546_object125052:head# 2024-04-23T16:51:22.642 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.642 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a0abf622:::benchmark_data_smithi188_155546_object123049:head# 2024-04-23T16:51:22.642 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.642 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a0cccd85:::benchmark_data_smithi188_155546_object123425:head# 2024-04-23T16:51:22.642 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.642 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a0d1bc10:::benchmark_data_smithi188_155546_object122934:head# 2024-04-23T16:51:22.642 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.645 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a0ec5cd2:::benchmark_data_smithi188_155546_object123649:head# 2024-04-23T16:51:22.646 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.646 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a0f45659:::benchmark_data_smithi188_155546_object119812:head# 2024-04-23T16:51:22.646 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.646 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a0f4654b:::benchmark_data_smithi188_155546_object123351:head# 2024-04-23T16:51:22.646 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.646 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a0ff3eda:::benchmark_data_smithi188_155546_object122301:head# 2024-04-23T16:51:22.646 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.646 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a10443e6:::benchmark_data_smithi188_155546_object123256:head# 2024-04-23T16:51:22.646 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.646 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a127bc8c:::benchmark_data_smithi188_155546_object120285:head# 2024-04-23T16:51:22.646 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.646 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a12996f4:::benchmark_data_smithi188_155546_object121208:head# 2024-04-23T16:51:22.647 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.647 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a1460e7f:::benchmark_data_smithi188_155546_object123171:head# 2024-04-23T16:51:22.647 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.647 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a14a66d1:::benchmark_data_smithi188_155546_object124965:head# 2024-04-23T16:51:22.647 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.647 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a16d1690:::benchmark_data_smithi188_155546_object124875:head# 2024-04-23T16:51:22.647 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.647 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a1725ca4:::benchmark_data_smithi188_155546_object121669:head# 2024-04-23T16:51:22.647 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.647 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a17ee3e2:::benchmark_data_smithi188_155546_object123892:head# 2024-04-23T16:51:22.647 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.647 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a18232cd:::benchmark_data_smithi188_155546_object123908:head# 2024-04-23T16:51:22.647 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.647 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a1a59fd6:::benchmark_data_smithi188_155546_object120665:head# 2024-04-23T16:51:22.647 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.648 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a1b59251:::benchmark_data_smithi188_155546_object123165:head# 2024-04-23T16:51:22.648 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.648 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a1bb51cd:::benchmark_data_smithi188_155546_object119869:head# 2024-04-23T16:51:22.648 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.648 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a1cdc045:::benchmark_data_smithi188_155546_object120569:head# 2024-04-23T16:51:22.648 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.648 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a1d5858b:::benchmark_data_smithi188_155546_object125430:head# 2024-04-23T16:51:22.648 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.648 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a1d73841:::benchmark_data_smithi188_155546_object121387:head# 2024-04-23T16:51:22.648 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.654 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a1e8693c:::benchmark_data_smithi188_155546_object125494:head# 2024-04-23T16:51:22.655 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.655 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a1f9a61c:::benchmark_data_smithi188_155546_object121692:head# 2024-04-23T16:51:22.655 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.655 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a22631ac:::benchmark_data_smithi188_155546_object119829:head# 2024-04-23T16:51:22.655 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.655 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a22f8662:::benchmark_data_smithi188_155546_object120716:head# 2024-04-23T16:51:22.655 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.655 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a22f8d74:::benchmark_data_smithi188_155546_object124226:head# 2024-04-23T16:51:22.655 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.655 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a234813d:::benchmark_data_smithi188_155546_object120712:head# 2024-04-23T16:51:22.655 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.655 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a24bf6ed:::benchmark_data_smithi188_155546_object120417:head# 2024-04-23T16:51:22.656 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.656 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a2690a27:::benchmark_data_smithi188_155546_object120823:head# 2024-04-23T16:51:22.656 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.656 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a29387de:::benchmark_data_smithi188_155546_object122335:head# 2024-04-23T16:51:22.656 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.656 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a2a14a20:::benchmark_data_smithi188_155546_object124052:head# 2024-04-23T16:51:22.656 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.656 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a2a28a8f:::benchmark_data_smithi188_155546_object124179:head# 2024-04-23T16:51:22.656 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.656 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a2c8df32:::benchmark_data_smithi188_155546_object122051:head# 2024-04-23T16:51:22.656 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.656 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a2f5dd01:::benchmark_data_smithi188_155546_object125176:head# 2024-04-23T16:51:22.656 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.656 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a2fe209d:::benchmark_data_smithi188_155546_object124630:head# 2024-04-23T16:51:22.656 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.657 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a2feeb32:::benchmark_data_smithi188_155546_object122985:head# 2024-04-23T16:51:22.657 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.657 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a30b6ed6:::benchmark_data_smithi188_155546_object125177:head# 2024-04-23T16:51:22.657 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.657 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a3179e07:::benchmark_data_smithi188_155546_object119886:head# 2024-04-23T16:51:22.657 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.657 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a31c0bff:::benchmark_data_smithi188_155546_object124672:head# 2024-04-23T16:51:22.657 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.657 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a31e9f1e:::benchmark_data_smithi188_155546_object124075:head# 2024-04-23T16:51:22.657 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.657 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a32ccd06:::benchmark_data_smithi188_155546_object120718:head# 2024-04-23T16:51:22.658 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.658 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a338f55b:::benchmark_data_smithi188_155546_object119687:head# 2024-04-23T16:51:22.658 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.658 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a33d1717:::benchmark_data_smithi188_155546_object120608:head# 2024-04-23T16:51:22.658 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.658 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a3499455:::benchmark_data_smithi188_155546_object123972:head# 2024-04-23T16:51:22.658 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.658 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a3562320:::benchmark_data_smithi188_155546_object121773:head# 2024-04-23T16:51:22.658 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.658 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a35b7459:::benchmark_data_smithi188_155546_object125192:head# 2024-04-23T16:51:22.658 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.658 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a37bca0a:::benchmark_data_smithi188_155546_object123224:head# 2024-04-23T16:51:22.658 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.658 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a37efe38:::benchmark_data_smithi188_155546_object120484:head# 2024-04-23T16:51:22.659 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.659 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a384c988:::benchmark_data_smithi188_155546_object123597:head# 2024-04-23T16:51:22.659 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.659 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a3875196:::benchmark_data_smithi188_155546_object119612:head# 2024-04-23T16:51:22.659 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.659 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a39b7e91:::benchmark_data_smithi188_155546_object124483:head# 2024-04-23T16:51:22.659 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.659 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a3a10be2:::benchmark_data_smithi188_155546_object123388:head# 2024-04-23T16:51:22.659 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.659 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a3a75561:::benchmark_data_smithi188_155546_object124991:head# 2024-04-23T16:51:22.659 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.659 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a3a889ab:::benchmark_data_smithi188_155546_object124170:head# 2024-04-23T16:51:22.659 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.659 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a3ac4fdc:::benchmark_data_smithi188_155546_object121742:head# 2024-04-23T16:51:22.660 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.660 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a3b0ee73:::benchmark_data_smithi188_155546_object121125:head# 2024-04-23T16:51:22.660 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.660 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a3b4fb05:::benchmark_data_smithi188_155546_object120880:head# 2024-04-23T16:51:22.660 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.660 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a3bdc21c:::benchmark_data_smithi188_155546_object120017:head# 2024-04-23T16:51:22.660 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.660 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a3c70747:::benchmark_data_smithi188_155546_object125308:head# 2024-04-23T16:51:22.660 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.664 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a3e79407:::benchmark_data_smithi188_155546_object120214:head# 2024-04-23T16:51:22.665 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.665 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a40fbf03:::benchmark_data_smithi188_155546_object120385:head# 2024-04-23T16:51:22.666 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.666 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a4221b2e:::benchmark_data_smithi188_155546_object124030:head# 2024-04-23T16:51:22.666 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.666 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a434bc03:::benchmark_data_smithi188_155546_object122318:head# 2024-04-23T16:51:22.666 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.666 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a43983fa:::benchmark_data_smithi188_155546_object123577:head# 2024-04-23T16:51:22.667 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.667 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a43c1850:::benchmark_data_smithi188_155546_object119793:head# 2024-04-23T16:51:22.667 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.667 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a4416623:::benchmark_data_smithi188_155546_object121273:head# 2024-04-23T16:51:22.667 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.667 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a4515d25:::benchmark_data_smithi188_155546_object125472:head# 2024-04-23T16:51:22.667 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.668 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a45e1097:::benchmark_data_smithi188_155546_object122019:head# 2024-04-23T16:51:22.668 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.668 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a46a1894:::benchmark_data_smithi188_155546_object121501:head# 2024-04-23T16:51:22.668 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.668 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a47b7454:::benchmark_data_smithi188_155546_object122667:head# 2024-04-23T16:51:22.668 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.668 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a47e3af8:::benchmark_data_smithi188_155546_object121253:head# 2024-04-23T16:51:22.668 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.668 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a48c1601:::benchmark_data_smithi188_155546_object120542:head# 2024-04-23T16:51:22.668 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.669 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a4b01365:::benchmark_data_smithi188_155546_object121530:head# 2024-04-23T16:51:22.669 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.669 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a4b7917b:::benchmark_data_smithi188_155546_object123557:head# 2024-04-23T16:51:22.669 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.669 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a4bf9d61:::benchmark_data_smithi188_155546_object120302:head# 2024-04-23T16:51:22.669 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.669 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a4cc79bf:::benchmark_data_smithi188_155546_object124019:head# 2024-04-23T16:51:22.669 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.669 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a4d2e5be:::benchmark_data_smithi188_155546_object119889:head# 2024-04-23T16:51:22.669 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.669 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a4efde96:::benchmark_data_smithi188_155546_object122310:head# 2024-04-23T16:51:22.669 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.670 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a4fd2b64:::benchmark_data_smithi188_155546_object123729:head# 2024-04-23T16:51:22.670 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.670 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a5023974:::benchmark_data_smithi188_155546_object123045:head# 2024-04-23T16:51:22.670 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.670 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a50fde17:::benchmark_data_smithi188_155546_object125282:head# 2024-04-23T16:51:22.670 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.670 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a5119228:::benchmark_data_smithi188_155546_object123475:head# 2024-04-23T16:51:22.670 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.670 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a51f4eea:::benchmark_data_smithi188_155546_object123264:head# 2024-04-23T16:51:22.670 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.670 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a5210cc7:::benchmark_data_smithi188_155546_object120362:head# 2024-04-23T16:51:22.670 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.670 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a52e2bbe:::benchmark_data_smithi188_155546_object123839:head# 2024-04-23T16:51:22.670 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.671 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a5486cb6:::benchmark_data_smithi188_155546_object122433:head# 2024-04-23T16:51:22.671 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.671 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a5542243:::benchmark_data_smithi188_155546_object123113:head# 2024-04-23T16:51:22.671 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.671 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a55b3379:::benchmark_data_smithi188_155546_object122487:head# 2024-04-23T16:51:22.671 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.693 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a55d607a:::benchmark_data_smithi188_155546_object122589:head# 2024-04-23T16:51:22.693 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.693 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a55f4a6f:::benchmark_data_smithi188_155546_object124902:head# 2024-04-23T16:51:22.693 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.693 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a577510d:::benchmark_data_smithi188_155546_object120239:head# 2024-04-23T16:51:22.693 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.693 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a57fbd4c:::benchmark_data_smithi188_155546_object122141:head# 2024-04-23T16:51:22.693 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.693 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a58ae4ff:::benchmark_data_smithi188_155546_object124951:head# 2024-04-23T16:51:22.693 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.693 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a58f284d:::benchmark_data_smithi188_155546_object120467:head# 2024-04-23T16:51:22.694 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.694 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a597e64e:::benchmark_data_smithi188_155546_object122499:head# 2024-04-23T16:51:22.694 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.694 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a5aefa24:::benchmark_data_smithi188_155546_object120922:head# 2024-04-23T16:51:22.694 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.694 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a5b6849d:::benchmark_data_smithi188_155546_object121661:head# 2024-04-23T16:51:22.694 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.694 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a5c19da9:::benchmark_data_smithi188_155546_object123036:head# 2024-04-23T16:51:22.694 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.694 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a5c6a168:::benchmark_data_smithi188_155546_object122219:head# 2024-04-23T16:51:22.694 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.694 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a5caecb5:::benchmark_data_smithi188_155546_object124350:head# 2024-04-23T16:51:22.695 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.695 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a5ceff17:::benchmark_data_smithi188_155546_object120828:head# 2024-04-23T16:51:22.695 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.695 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a5d87991:::benchmark_data_smithi188_155546_object124310:head# 2024-04-23T16:51:22.695 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.695 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a5e07be2:::benchmark_data_smithi188_155546_object120518:head# 2024-04-23T16:51:22.695 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.695 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a5e963ed:::benchmark_data_smithi188_155546_object120453:head# 2024-04-23T16:51:22.695 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.695 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a60cfbab:::benchmark_data_smithi188_155546_object120451:head# 2024-04-23T16:51:22.695 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.695 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a6115565:::benchmark_data_smithi188_155546_object125501:head# 2024-04-23T16:51:22.695 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.696 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a6471537:::benchmark_data_smithi188_155546_object120354:head# 2024-04-23T16:51:22.696 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.696 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a65075ef:::benchmark_data_smithi188_155546_object125247:head# 2024-04-23T16:51:22.696 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.696 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a6522ce6:::benchmark_data_smithi188_155546_object120274:head# 2024-04-23T16:51:22.696 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.696 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a65ca50b:::benchmark_data_smithi188_155546_object120752:head# 2024-04-23T16:51:22.696 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.696 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a6602f03:::benchmark_data_smithi188_155546_object123146:head# 2024-04-23T16:51:22.696 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.696 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a663adf1:::benchmark_data_smithi188_155546_object125268:head# 2024-04-23T16:51:22.697 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.697 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a663dc20:::benchmark_data_smithi188_155546_object122248:head# 2024-04-23T16:51:22.697 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.697 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a678bd21:::benchmark_data_smithi188_155546_object125159:head# 2024-04-23T16:51:22.697 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.697 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a68287be:::benchmark_data_smithi188_155546_object125236:head# 2024-04-23T16:51:22.697 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.697 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a694170d:::benchmark_data_smithi188_155546_object125380:head# 2024-04-23T16:51:22.697 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.697 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a6981e93:::benchmark_data_smithi188_155546_object122459:head# 2024-04-23T16:51:22.697 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.697 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a6aad106:::benchmark_data_smithi188_155546_object119880:head# 2024-04-23T16:51:22.698 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.698 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a6b8b2c0:::benchmark_data_smithi188_155546_object124351:head# 2024-04-23T16:51:22.698 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.698 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a6c9b63b:::benchmark_data_smithi188_155546_object124287:head# 2024-04-23T16:51:22.698 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.698 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a6cbb757:::benchmark_data_smithi188_155546_object124493:head# 2024-04-23T16:51:22.698 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.698 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a6d1bc7c:::benchmark_data_smithi188_155546_object120847:head# 2024-04-23T16:51:22.698 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.698 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a6d2b095:::benchmark_data_smithi188_155546_object121244:head# 2024-04-23T16:51:22.698 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.698 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a6d47c17:::benchmark_data_smithi188_155546_object125033:head# 2024-04-23T16:51:22.698 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.699 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a6dd1e3b:::benchmark_data_smithi188_155546_object122067:head# 2024-04-23T16:51:22.699 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.699 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a6e2a766:::benchmark_data_smithi188_155546_object121602:head# 2024-04-23T16:51:22.699 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.699 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a6f04630:::benchmark_data_smithi188_155546_object120216:head# 2024-04-23T16:51:22.699 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.699 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a70210a2:::benchmark_data_smithi188_155546_object122771:head# 2024-04-23T16:51:22.699 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.699 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a7068b14:::benchmark_data_smithi188_155546_object123160:head# 2024-04-23T16:51:22.699 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.699 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a713cb4e:::benchmark_data_smithi188_155546_object124463:head# 2024-04-23T16:51:22.699 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.699 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a718a683:::benchmark_data_smithi188_155546_object120145:head# 2024-04-23T16:51:22.700 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.700 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a7296ab1:::benchmark_data_smithi188_155546_object121984:head# 2024-04-23T16:51:22.700 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.700 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a7332781:::benchmark_data_smithi188_155546_object120654:head# 2024-04-23T16:51:22.700 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.700 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a734ee88:::benchmark_data_smithi188_155546_object122467:head# 2024-04-23T16:51:22.700 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.700 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a747b93a:::benchmark_data_smithi188_155546_object120942:head# 2024-04-23T16:51:22.700 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.700 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a7496e54:::benchmark_data_smithi188_155546_object125388:head# 2024-04-23T16:51:22.700 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.700 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a74f0f60:::benchmark_data_smithi188_155546_object119990:head# 2024-04-23T16:51:22.700 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.701 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a77b773c:::benchmark_data_smithi188_155546_object120472:head# 2024-04-23T16:51:22.701 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.701 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a785cea4:::benchmark_data_smithi188_155546_object123819:head# 2024-04-23T16:51:22.701 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.701 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a786f82c:::benchmark_data_smithi188_155546_object120193:head# 2024-04-23T16:51:22.701 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.701 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a789c6ea:::benchmark_data_smithi188_155546_object123488:head# 2024-04-23T16:51:22.701 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.701 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a79b6c99:::benchmark_data_smithi188_155546_object124036:head# 2024-04-23T16:51:22.701 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.701 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a7a0e359:::benchmark_data_smithi188_155546_object120644:head# 2024-04-23T16:51:22.701 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.702 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a7abb061:::benchmark_data_smithi188_155546_object123487:head# 2024-04-23T16:51:22.702 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.702 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a7ba9df8:::benchmark_data_smithi188_155546_object121826:head# 2024-04-23T16:51:22.702 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.702 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a7c2cfb3:::benchmark_data_smithi188_155546_object120759:head# 2024-04-23T16:51:22.702 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.702 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a7c3b8a4:::benchmark_data_smithi188_155546_object120020:head# 2024-04-23T16:51:22.702 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.702 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a7db6ece:::benchmark_data_smithi188_155546_object124866:head# 2024-04-23T16:51:22.702 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.702 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a7e3e7cd:::benchmark_data_smithi188_155546_object120474:head# 2024-04-23T16:51:22.702 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.702 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a7ff2ea0:::benchmark_data_smithi188_155546_object122398:head# 2024-04-23T16:51:22.703 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.703 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a80d81c2:::benchmark_data_smithi188_155546_object123046:head# 2024-04-23T16:51:22.703 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.703 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a854df89:::benchmark_data_smithi188_155546_object123638:head# 2024-04-23T16:51:22.703 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.703 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a858c814:::benchmark_data_smithi188_155546_object120456:head# 2024-04-23T16:51:22.703 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.703 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a85b1ace:::benchmark_data_smithi188_155546_object121117:head# 2024-04-23T16:51:22.703 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.703 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a85d628d:::benchmark_data_smithi188_155546_object120967:head# 2024-04-23T16:51:22.703 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.703 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a862caf8:::benchmark_data_smithi188_155546_object120180:head# 2024-04-23T16:51:22.703 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.703 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a86e7fe2:::benchmark_data_smithi188_155546_object120697:head# 2024-04-23T16:51:22.704 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.704 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a87661c3:::benchmark_data_smithi188_155546_object120725:head# 2024-04-23T16:51:22.704 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.704 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a8a17ccc:::benchmark_data_smithi188_155546_object121013:head# 2024-04-23T16:51:22.704 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.704 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a8b5d02f:::benchmark_data_smithi188_155546_object119923:head# 2024-04-23T16:51:22.704 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.704 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a8c008ce:::benchmark_data_smithi188_155546_object123705:head# 2024-04-23T16:51:22.704 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.704 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a8d21a66:::benchmark_data_smithi188_155546_object124204:head# 2024-04-23T16:51:22.704 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.704 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a8d71dd6:::benchmark_data_smithi188_155546_object122703:head# 2024-04-23T16:51:22.704 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.704 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a8df47f3:::benchmark_data_smithi188_155546_object124624:head# 2024-04-23T16:51:22.705 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.705 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a8e5bf29:::benchmark_data_smithi188_155546_object124833:head# 2024-04-23T16:51:22.705 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.705 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a9272355:::benchmark_data_smithi188_155546_object121600:head# 2024-04-23T16:51:22.705 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.705 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a933108c:::benchmark_data_smithi188_155546_object122602:head# 2024-04-23T16:51:22.705 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.705 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a94064c6:::benchmark_data_smithi188_155546_object123738:head# 2024-04-23T16:51:22.705 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.705 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a9477d0e:::benchmark_data_smithi188_155546_object121085:head# 2024-04-23T16:51:22.705 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.705 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a9541e1f:::benchmark_data_smithi188_155546_object123921:head# 2024-04-23T16:51:22.705 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.706 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a9554b0e:::benchmark_data_smithi188_155546_object120516:head# 2024-04-23T16:51:22.706 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.706 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a95fa950:::benchmark_data_smithi188_155546_object121256:head# 2024-04-23T16:51:22.706 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.706 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a960c10c:::benchmark_data_smithi188_155546_object121324:head# 2024-04-23T16:51:22.706 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.706 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a96b033b:::benchmark_data_smithi188_155546_object122760:head# 2024-04-23T16:51:22.706 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.706 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a977943b:::benchmark_data_smithi188_155546_object123928:head# 2024-04-23T16:51:22.706 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.706 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a98890c5:::benchmark_data_smithi188_155546_object122726:head# 2024-04-23T16:51:22.706 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.706 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a988c59f:::benchmark_data_smithi188_155546_object123774:head# 2024-04-23T16:51:22.706 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.707 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a988c66b:::benchmark_data_smithi188_155546_object124192:head# 2024-04-23T16:51:22.707 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.707 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a99a210d:::benchmark_data_smithi188_155546_object123728:head# 2024-04-23T16:51:22.707 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.707 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a9a940c2:::benchmark_data_smithi188_155546_object123799:head# 2024-04-23T16:51:22.707 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.707 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a9b40d63:::benchmark_data_smithi188_155546_object123909:head# 2024-04-23T16:51:22.707 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.707 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a9b7e442:::benchmark_data_smithi188_155546_object123485:head# 2024-04-23T16:51:22.707 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.707 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a9b97fac:::benchmark_data_smithi188_155546_object123787:head# 2024-04-23T16:51:22.707 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.707 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a9c2be40:::benchmark_data_smithi188_155546_object121369:head# 2024-04-23T16:51:22.708 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.708 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a9c4fab8:::benchmark_data_smithi188_155546_object124334:head# 2024-04-23T16:51:22.708 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.708 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a9d0e209:::benchmark_data_smithi188_155546_object123759:head# 2024-04-23T16:51:22.708 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.708 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a9d872a1:::benchmark_data_smithi188_155546_object119956:head# 2024-04-23T16:51:22.708 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.708 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a9d98110:::benchmark_data_smithi188_155546_object122475:head# 2024-04-23T16:51:22.708 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.708 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a9d9df1d:::benchmark_data_smithi188_155546_object124621:head# 2024-04-23T16:51:22.708 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.708 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a9dac822:::benchmark_data_smithi188_155546_object122749:head# 2024-04-23T16:51:22.708 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.709 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a9e5ffaa:::benchmark_data_smithi188_155546_object124381:head# 2024-04-23T16:51:22.709 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.709 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a9e9dc9e:::benchmark_data_smithi188_155546_object122523:head# 2024-04-23T16:51:22.709 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.709 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a9f08464:::benchmark_data_smithi188_155546_object119790:head# 2024-04-23T16:51:22.709 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.709 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:a9f740ff:::benchmark_data_smithi188_155546_object121662:head# 2024-04-23T16:51:22.709 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.709 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aa2306b4:::benchmark_data_smithi188_155546_object122133:head# 2024-04-23T16:51:22.709 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.709 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aa29a284:::benchmark_data_smithi188_155546_object121308:head# 2024-04-23T16:51:22.709 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.731 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aa2f6347:::benchmark_data_smithi188_155546_object124993:head# 2024-04-23T16:51:22.731 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.731 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aa5b1e6f:::benchmark_data_smithi188_155546_object121001:head# 2024-04-23T16:51:22.731 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.732 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aa64d5c7:::benchmark_data_smithi188_155546_object124683:head# 2024-04-23T16:51:22.732 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.732 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aa8aad9c:::benchmark_data_smithi188_155546_object124682:head# 2024-04-23T16:51:22.732 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.732 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aa8beaf7:::benchmark_data_smithi188_155546_object121658:head# 2024-04-23T16:51:22.732 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.732 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aa96296f:::benchmark_data_smithi188_155546_object124257:head# 2024-04-23T16:51:22.732 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.732 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aad13d4e:::benchmark_data_smithi188_155546_object125424:head# 2024-04-23T16:51:22.732 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.732 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aada83ab:::benchmark_data_smithi188_155546_object123064:head# 2024-04-23T16:51:22.732 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.732 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aadc7b00:::benchmark_data_smithi188_155546_object121410:head# 2024-04-23T16:51:22.733 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.733 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aae65f16:::benchmark_data_smithi188_155546_object119924:head# 2024-04-23T16:51:22.733 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.733 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aaf68894:::benchmark_data_smithi188_155546_object123087:head# 2024-04-23T16:51:22.733 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.733 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab020dbb:::benchmark_data_smithi188_155546_object124312:head# 2024-04-23T16:51:22.733 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.733 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab04bef1:::benchmark_data_smithi188_155546_object119915:head# 2024-04-23T16:51:22.733 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.733 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab19e7dc:::benchmark_data_smithi188_155546_object120217:head# 2024-04-23T16:51:22.733 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.733 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab1c8dff:::benchmark_data_smithi188_155546_object125217:head# 2024-04-23T16:51:22.733 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.733 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab1ec853:::benchmark_data_smithi188_155546_object121400:head# 2024-04-23T16:51:22.734 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.734 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab298846:::benchmark_data_smithi188_155546_object124990:head# 2024-04-23T16:51:22.734 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.734 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab29b8a3:::benchmark_data_smithi188_155546_object120454:head# 2024-04-23T16:51:22.734 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.734 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab2b27ca:::benchmark_data_smithi188_155546_object120332:head# 2024-04-23T16:51:22.734 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.734 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab2f6801:::benchmark_data_smithi188_155546_object124367:head# 2024-04-23T16:51:22.734 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.734 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab344a99:::benchmark_data_smithi188_155546_object123163:head# 2024-04-23T16:51:22.734 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.734 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab38dcbd:::benchmark_data_smithi188_155546_object121265:head# 2024-04-23T16:51:22.734 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.735 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab4317f3:::benchmark_data_smithi188_155546_object120793:head# 2024-04-23T16:51:22.735 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.735 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab4d5cb1:::benchmark_data_smithi188_155546_object123135:head# 2024-04-23T16:51:22.735 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.735 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab4f6298:::benchmark_data_smithi188_155546_object124262:head# 2024-04-23T16:51:22.735 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.735 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab5600d5:::benchmark_data_smithi188_155546_object121946:head# 2024-04-23T16:51:22.735 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.735 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab5a5225:::benchmark_data_smithi188_155546_object124664:head# 2024-04-23T16:51:22.735 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.735 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab71f68d:::benchmark_data_smithi188_155546_object121488:head# 2024-04-23T16:51:22.735 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.735 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab71fbbe:::benchmark_data_smithi188_155546_object121598:head# 2024-04-23T16:51:22.736 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.736 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab7a3453:::benchmark_data_smithi188_155546_object119890:head# 2024-04-23T16:51:22.736 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.736 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab899e53:::benchmark_data_smithi188_155546_object124700:head# 2024-04-23T16:51:22.736 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.736 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab8a4b51:::benchmark_data_smithi188_155546_object120795:head# 2024-04-23T16:51:22.736 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.736 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ab9c9515:::benchmark_data_smithi188_155546_object121631:head# 2024-04-23T16:51:22.736 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.736 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:abb4b75e:::benchmark_data_smithi188_155546_object120309:head# 2024-04-23T16:51:22.736 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.736 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:abb6fe31:::benchmark_data_smithi188_155546_object122772:head# 2024-04-23T16:51:22.737 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.737 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:abca7314:::benchmark_data_smithi188_155546_object121760:head# 2024-04-23T16:51:22.737 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.737 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:abcd2ad7:::benchmark_data_smithi188_155546_object122967:head# 2024-04-23T16:51:22.737 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.737 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:abd2115b:::benchmark_data_smithi188_155546_object124020:head# 2024-04-23T16:51:22.737 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.737 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:abdc268f:::benchmark_data_smithi188_155546_object121680:head# 2024-04-23T16:51:22.737 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.737 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:abdcd0fb:::benchmark_data_smithi188_155546_object119725:head# 2024-04-23T16:51:22.737 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.737 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:abe59634:::benchmark_data_smithi188_155546_object123674:head# 2024-04-23T16:51:22.737 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.737 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:abe82858:::benchmark_data_smithi188_155546_object125242:head# 2024-04-23T16:51:22.738 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.738 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ac02d5e4:::benchmark_data_smithi188_155546_object120408:head# 2024-04-23T16:51:22.738 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.738 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ac08f6cc:::benchmark_data_smithi188_155546_object119701:head# 2024-04-23T16:51:22.738 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.738 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ac1742ec:::benchmark_data_smithi188_155546_object121367:head# 2024-04-23T16:51:22.738 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.738 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ac2132b8:::benchmark_data_smithi188_155546_object124345:head# 2024-04-23T16:51:22.738 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.738 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ac2e376f:::benchmark_data_smithi188_155546_object122722:head# 2024-04-23T16:51:22.738 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.738 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ac3ada00:::benchmark_data_smithi188_155546_object121643:head# 2024-04-23T16:51:22.738 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.739 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ac4f152d:::benchmark_data_smithi188_155546_object122264:head# 2024-04-23T16:51:22.739 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.739 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ac6060e4:::benchmark_data_smithi188_155546_object125466:head# 2024-04-23T16:51:22.739 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.739 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ac6a2201:::benchmark_data_smithi188_155546_object121014:head# 2024-04-23T16:51:22.739 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.739 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ac71a317:::benchmark_data_smithi188_155546_object122193:head# 2024-04-23T16:51:22.739 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.739 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ac72026c:::benchmark_data_smithi188_155546_object119820:head# 2024-04-23T16:51:22.739 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.739 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ac769816:::benchmark_data_smithi188_155546_object121493:head# 2024-04-23T16:51:22.739 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.739 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ac7d22a0:::benchmark_data_smithi188_155546_object121220:head# 2024-04-23T16:51:22.740 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.740 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ac906305:::benchmark_data_smithi188_155546_object121455:head# 2024-04-23T16:51:22.740 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.740 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ac99ad76:::benchmark_data_smithi188_155546_object119810:head# 2024-04-23T16:51:22.740 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.740 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ac9d035b:::benchmark_data_smithi188_155546_object125056:head# 2024-04-23T16:51:22.740 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.740 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:acafca62:::benchmark_data_smithi188_155546_object123878:head# 2024-04-23T16:51:22.740 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.740 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:acb0c1c4:::benchmark_data_smithi188_155546_object120577:head# 2024-04-23T16:51:22.740 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.740 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:acba7d0f:::benchmark_data_smithi188_155546_object124336:head# 2024-04-23T16:51:22.740 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.741 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:acc0e2ec:::benchmark_data_smithi188_155546_object122270:head# 2024-04-23T16:51:22.741 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.741 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:acc20528:::benchmark_data_smithi188_155546_object121171:head# 2024-04-23T16:51:22.741 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.741 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:acc3295a:::benchmark_data_smithi188_155546_object120689:head# 2024-04-23T16:51:22.741 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.741 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:acc80e3d:::benchmark_data_smithi188_155546_object120550:head# 2024-04-23T16:51:22.741 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.741 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:acceb0f9:::benchmark_data_smithi188_155546_object120920:head# 2024-04-23T16:51:22.741 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.741 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:acda7db5:::benchmark_data_smithi188_155546_object122319:head# 2024-04-23T16:51:22.741 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.741 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:acfffd02:::benchmark_data_smithi188_155546_object123440:head# 2024-04-23T16:51:22.741 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.741 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ad00cb16:::benchmark_data_smithi188_155546_object125043:head# 2024-04-23T16:51:22.742 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.742 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ad016c28:::benchmark_data_smithi188_155546_object122826:head# 2024-04-23T16:51:22.742 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.742 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ad018937:::benchmark_data_smithi188_155546_object123335:head# 2024-04-23T16:51:22.742 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.742 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ad110119:::benchmark_data_smithi188_155546_object121973:head# 2024-04-23T16:51:22.742 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.742 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ad1b5ba0:::benchmark_data_smithi188_155546_object125251:head# 2024-04-23T16:51:22.742 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.742 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ad1c6046:::benchmark_data_smithi188_155546_object124592:head# 2024-04-23T16:51:22.742 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.743 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ad20dace:::benchmark_data_smithi188_155546_object122085:head# 2024-04-23T16:51:22.743 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.743 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ad20eb4b:::benchmark_data_smithi188_155546_object121504:head# 2024-04-23T16:51:22.743 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.743 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ad3c3d86:::benchmark_data_smithi188_155546_object120899:head# 2024-04-23T16:51:22.743 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.743 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ad3d6922:::benchmark_data_smithi188_155546_object123806:head# 2024-04-23T16:51:22.743 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.743 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ad416d37:::benchmark_data_smithi188_155546_object120012:head# 2024-04-23T16:51:22.743 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.743 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ad4a2f6c:::benchmark_data_smithi188_155546_object121779:head# 2024-04-23T16:51:22.744 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.744 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ad5d462c:::benchmark_data_smithi188_155546_object124853:head# 2024-04-23T16:51:22.744 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.744 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ad78ddf3:::benchmark_data_smithi188_155546_object122196:head# 2024-04-23T16:51:22.744 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.744 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ad86fb1f:::benchmark_data_smithi188_155546_object119933:head# 2024-04-23T16:51:22.744 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.744 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:adb65a2e:::benchmark_data_smithi188_155546_object121549:head# 2024-04-23T16:51:22.744 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.744 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:adc32291:::benchmark_data_smithi188_155546_object119815:head# 2024-04-23T16:51:22.744 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.744 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:adc56a4a:::benchmark_data_smithi188_155546_object120024:head# 2024-04-23T16:51:22.744 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.745 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:adde2c25:::benchmark_data_smithi188_155546_object122350:head# 2024-04-23T16:51:22.745 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.745 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ae1c2ec2:::benchmark_data_smithi188_155546_object123513:head# 2024-04-23T16:51:22.745 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.745 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ae1df1c1:::benchmark_data_smithi188_155546_object122208:head# 2024-04-23T16:51:22.745 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.745 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ae26c4df:::benchmark_data_smithi188_155546_object120172:head# 2024-04-23T16:51:22.745 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.745 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ae2eb1a3:::benchmark_data_smithi188_155546_object123260:head# 2024-04-23T16:51:22.745 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.745 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ae414d44:::benchmark_data_smithi188_155546_object122816:head# 2024-04-23T16:51:22.745 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.745 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ae5ffb07:::benchmark_data_smithi188_155546_object122862:head# 2024-04-23T16:51:22.745 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.746 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ae72b275:::benchmark_data_smithi188_155546_object120052:head# 2024-04-23T16:51:22.746 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.746 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ae97cba2:::benchmark_data_smithi188_155546_object119928:head# 2024-04-23T16:51:22.746 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.746 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ae9b8bb3:::benchmark_data_smithi188_155546_object124985:head# 2024-04-23T16:51:22.746 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.746 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:ae9bfd19:::benchmark_data_smithi188_155546_object119818:head# 2024-04-23T16:51:22.746 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.746 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aea9902e:::benchmark_data_smithi188_155546_object122525:head# 2024-04-23T16:51:22.746 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.746 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aeaba9af:::benchmark_data_smithi188_155546_object119969:head# 2024-04-23T16:51:22.746 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.747 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aeace135:::benchmark_data_smithi188_155546_object119909:head# 2024-04-23T16:51:22.747 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.747 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aec2c9aa:::benchmark_data_smithi188_155546_object124309:head# 2024-04-23T16:51:22.747 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.747 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aec71284:::benchmark_data_smithi188_155546_object124906:head# 2024-04-23T16:51:22.747 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.747 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aee32dfa:::benchmark_data_smithi188_155546_object125096:head# 2024-04-23T16:51:22.747 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.747 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aee4e238:::benchmark_data_smithi188_155546_object124794:head# 2024-04-23T16:51:22.747 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.747 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aeea2c2a:::benchmark_data_smithi188_155546_object122593:head# 2024-04-23T16:51:22.747 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.747 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aef6d058:::benchmark_data_smithi188_155546_object122391:head# 2024-04-23T16:51:22.747 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.747 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aef79b68:::benchmark_data_smithi188_155546_object123862:head# 2024-04-23T16:51:22.748 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.748 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:af08a30f:::benchmark_data_smithi188_155546_object122655:head# 2024-04-23T16:51:22.748 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.748 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:af13179e:::benchmark_data_smithi188_155546_object125261:head# 2024-04-23T16:51:22.748 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.748 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:af187b09:::benchmark_data_smithi188_155546_object119663:head# 2024-04-23T16:51:22.748 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.748 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:af1d22ac:::benchmark_data_smithi188_155546_object125068:head# 2024-04-23T16:51:22.748 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.748 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:af1fe105:::benchmark_data_smithi188_155546_object125423:head# 2024-04-23T16:51:22.748 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.748 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:af3a2f0c:::benchmark_data_smithi188_155546_object123404:head# 2024-04-23T16:51:22.748 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.749 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:af55ad56:::benchmark_data_smithi188_155546_object119757:head# 2024-04-23T16:51:22.749 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.749 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:af5e1315:::benchmark_data_smithi188_155546_object120605:head# 2024-04-23T16:51:22.749 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.749 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:af5f6758:::benchmark_data_smithi188_155546_object122935:head# 2024-04-23T16:51:22.749 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.749 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:af6bedb1:::benchmark_data_smithi188_155546_object121571:head# 2024-04-23T16:51:22.749 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.749 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:af7b2bbd:::benchmark_data_smithi188_155546_object123285:head# 2024-04-23T16:51:22.749 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.749 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:af7caa7a:::benchmark_data_smithi188_155546_object122561:head# 2024-04-23T16:51:22.749 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.749 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:af899c58:::benchmark_data_smithi188_155546_object123499:head# 2024-04-23T16:51:22.749 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.750 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:af8c0c22:::benchmark_data_smithi188_155546_object123881:head# 2024-04-23T16:51:22.750 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.750 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:afaebb47:::benchmark_data_smithi188_155546_object122341:head# 2024-04-23T16:51:22.750 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.750 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:afc38a4a:::benchmark_data_smithi188_155546_object119678:head# 2024-04-23T16:51:22.750 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.750 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:afc6daa0:::benchmark_data_smithi188_155546_object122343:head# 2024-04-23T16:51:22.750 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.750 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:afdb39bc:::benchmark_data_smithi188_155546_object123309:head# 2024-04-23T16:51:22.750 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.750 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:afdd71b0:::benchmark_data_smithi188_155546_object119737:head# 2024-04-23T16:51:22.750 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.750 INFO:teuthology.orchestra.run.smithi148.stdout:Write #8:aff38299:::benchmark_data_smithi188_155546_object122738:head# 2024-04-23T16:51:22.751 INFO:teuthology.orchestra.run.smithi148.stdout:snapset 0=[]:{} 2024-04-23T16:51:22.751 INFO:teuthology.orchestra.run.smithi148.stdout:write_pg epoch 686 info 8.5( v 686'15119 (622'5100,686'15119] local-lis/les=609/610 n=337 ec=609/609 lis/c=609/609 les/c/f=610/610/0 sis=609) 2024-04-23T16:51:22.811 INFO:teuthology.orchestra.run.smithi148.stdout:Import successful 2024-04-23T16:51:23.069 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:51:23.250 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:51:23.277 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:23 smithi103 ceph-mon[39629]: osdmap e705: 12 total, 11 up, 7 in 2024-04-23T16:51:23.277 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:23 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3996466814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:51:23.298 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:51:23.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:23.301 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:51:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:23 smithi148 ceph-mon[44520]: osdmap e705: 12 total, 11 up, 7 in 2024-04-23T16:51:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:23 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3996466814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:51:23.402 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:51:23.432 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:23 smithi188 ceph-mon[44101]: osdmap e705: 12 total, 11 up, 7 in 2024-04-23T16:51:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:23 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3996466814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:51:23.486 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:51:23.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:23.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:51:23.591 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:51:23.619 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:23.871 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/exp.8.5.4 2024-04-23T16:51:23.935 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-04-23T16:51:23.936 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-23T16:51:23.936 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:51:24.069 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:51:24.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:24.074 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:51:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:24 smithi148 ceph-mon[44520]: pgmap v1959: 27 pgs: 8 creating+peering, 6 unknown, 13 active+clean; 577 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:24 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:51:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:24 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3996466814' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:51:24.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:24 smithi148 ceph-mon[44520]: osdmap e706: 12 total, 11 up, 7 in 2024-04-23T16:51:24.175 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:51:24.205 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:24.224 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:51:24.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:24.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:51:24.328 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:51:24.401 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:24 smithi103 ceph-mon[39629]: pgmap v1959: 27 pgs: 8 creating+peering, 6 unknown, 13 active+clean; 577 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:24.402 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:24 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:51:24.402 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3996466814' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:51:24.402 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:24 smithi103 ceph-mon[39629]: osdmap e706: 12 total, 11 up, 7 in 2024-04-23T16:51:24.402 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:51:24.402 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:51:24.402 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:51:24.402 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:51:24.402 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:51:24.402 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:24 smithi188 ceph-mon[44101]: pgmap v1959: 27 pgs: 8 creating+peering, 6 unknown, 13 active+clean; 577 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:24.402 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:24 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:51:24.403 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3996466814' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:51:24.403 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:24 smithi188 ceph-mon[44101]: osdmap e706: 12 total, 11 up, 7 in 2024-04-23T16:51:24.430 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:24.449 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:51:24.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:24.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:51:24.553 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:51:24.580 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:24.598 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:51:24.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:24.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:51:24.631 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:51:24.637 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:51:24.702 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:51:24.729 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:24.748 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:51:24.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:24.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:51:24.852 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:51:24.882 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:25.007 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:51:25.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:25.011 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:51:25.021 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-23T16:51:25.112 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:51:25.123 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:25 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:51:25.123 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:25 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:51:25.123 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:25 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1099678202' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:51:25.140 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:25.322 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:51:25.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:25 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:51:25.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:25 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:51:25.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:25 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1099678202' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:51:25.456 INFO:teuthology.orchestra.run.smithi103.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-04-23T16:51:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:25 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:51:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:25 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:51:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:25 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1099678202' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T16:51:26.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:26 smithi188 ceph-mon[44101]: pgmap v1961: 27 pgs: 8 creating+peering, 19 active+clean; 577 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:26.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:26 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1099678202' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:51:26.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:26 smithi188 ceph-mon[44101]: osdmap e707: 12 total, 11 up, 7 in 2024-04-23T16:51:26.304 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:51:26.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:26 smithi103 ceph-mon[39629]: pgmap v1961: 27 pgs: 8 creating+peering, 19 active+clean; 577 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:26.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:26 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1099678202' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:51:26.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:26 smithi103 ceph-mon[39629]: osdmap e707: 12 total, 11 up, 7 in 2024-04-23T16:51:26.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:26 smithi148 ceph-mon[44520]: pgmap v1961: 27 pgs: 8 creating+peering, 19 active+clean; 577 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:26.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:26 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1099678202' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-23T16:51:26.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:26 smithi148 ceph-mon[44520]: osdmap e707: 12 total, 11 up, 7 in 2024-04-23T16:51:27.053 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:51:27.053 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-04-23T16:51:27.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-04-23T16:51:27.053 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:51:27.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.053 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:105434 8.a 8:5e91d54b:::benchmark_data_smithi188_155546_object105433:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e639)", 2024-04-23T16:51:27.053 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:47:39.088404+0000", 2024-04-23T16:51:27.053 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 227.96113956900001, 2024-04-23T16:51:27.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.084440646999999994, 2024-04-23T16:51:27.053 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.054 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.054 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:27.054 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:51:27.054 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:51:27.054 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:51:27.054 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 105434 2024-04-23T16:51:27.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.054 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:39.088404+0000", 2024-04-23T16:51:27.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:39.088404+0000", 2024-04-23T16:51:27.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:39.088404+0000", 2024-04-23T16:51:27.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9100000000000004e-07 2024-04-23T16:51:27.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:39.088422+0000", 2024-04-23T16:51:27.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8062000000000001e-05 2024-04-23T16:51:27.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:39.088428+0000", 2024-04-23T16:51:27.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7180000000000001e-06 2024-04-23T16:51:27.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:39.088433+0000", 2024-04-23T16:51:27.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9599999999999999e-06 2024-04-23T16:51:27.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:39.092124+0000", 2024-04-23T16:51:27.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036907519999999998 2024-04-23T16:51:27.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:51:27.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:39.092254+0000", 2024-04-23T16:51:27.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013009600000000001 2024-04-23T16:51:27.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:51:27.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:39.092398+0000", 2024-04-23T16:51:27.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014457599999999999 2024-04-23T16:51:27.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:51:27.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:39.096495+0000", 2024-04-23T16:51:27.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0040963529999999996 2024-04-23T16:51:27.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:27.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:39.172730+0000", 2024-04-23T16:51:27.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.076235577999999998 2024-04-23T16:51:27.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:51:27.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:39.172793+0000", 2024-04-23T16:51:27.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3199000000000003e-05 2024-04-23T16:51:27.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:39.172844+0000", 2024-04-23T16:51:27.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0762e-05 2024-04-23T16:51:27.059 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.059 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.059 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.059 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:106222 8.a 8:50ff8611:::benchmark_data_smithi188_155546_object106221:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e639)", 2024-04-23T16:51:27.059 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:47:40.184409+0000", 2024-04-23T16:51:27.059 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 226.865134297, 2024-04-23T16:51:27.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.080644824000000004, 2024-04-23T16:51:27.060 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.060 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.060 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:27.060 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:51:27.060 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:51:27.060 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:51:27.060 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 106222 2024-04-23T16:51:27.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.060 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.184409+0000", 2024-04-23T16:51:27.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.184409+0000", 2024-04-23T16:51:27.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.184411+0000", 2024-04-23T16:51:27.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5679999999999998e-06 2024-04-23T16:51:27.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.184434+0000", 2024-04-23T16:51:27.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2819e-05 2024-04-23T16:51:27.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.184449+0000", 2024-04-23T16:51:27.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4816e-05 2024-04-23T16:51:27.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.184460+0000", 2024-04-23T16:51:27.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0453e-05 2024-04-23T16:51:27.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.184515+0000", 2024-04-23T16:51:27.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5936000000000001e-05 2024-04-23T16:51:27.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:51:27.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.184627+0000", 2024-04-23T16:51:27.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000112049 2024-04-23T16:51:27.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:51:27.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.184725+0000", 2024-04-23T16:51:27.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7490000000000003e-05 2024-04-23T16:51:27.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:51:27.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.187471+0000", 2024-04-23T16:51:27.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027457850000000002 2024-04-23T16:51:27.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:27.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.264944+0000", 2024-04-23T16:51:27.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.077473252000000006 2024-04-23T16:51:27.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:51:27.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.265005+0000", 2024-04-23T16:51:27.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0993999999999999e-05 2024-04-23T16:51:27.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.265054+0000", 2024-04-23T16:51:27.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8662000000000003e-05 2024-04-23T16:51:27.065 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.065 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.065 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.065 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:106225 8.a 8:5b4d27e0:::benchmark_data_smithi188_155546_object106224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e639)", 2024-04-23T16:51:27.065 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:47:40.187432+0000", 2024-04-23T16:51:27.065 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 226.86211125899999, 2024-04-23T16:51:27.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.079509406000000005, 2024-04-23T16:51:27.066 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.066 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.066 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:27.066 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:51:27.066 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:51:27.066 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:51:27.066 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 106225 2024-04-23T16:51:27.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.066 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.187432+0000", 2024-04-23T16:51:27.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.187432+0000", 2024-04-23T16:51:27.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.187434+0000", 2024-04-23T16:51:27.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.849e-06 2024-04-23T16:51:27.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.187459+0000", 2024-04-23T16:51:27.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5248e-05 2024-04-23T16:51:27.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.187469+0000", 2024-04-23T16:51:27.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7089999999999997e-06 2024-04-23T16:51:27.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.187475+0000", 2024-04-23T16:51:27.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3949999999999998e-06 2024-04-23T16:51:27.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.187513+0000", 2024-04-23T16:51:27.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.773e-05 2024-04-23T16:51:27.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:51:27.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.187607+0000", 2024-04-23T16:51:27.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3845999999999996e-05 2024-04-23T16:51:27.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:51:27.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.187695+0000", 2024-04-23T16:51:27.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8819000000000005e-05 2024-04-23T16:51:27.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:51:27.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.190058+0000", 2024-04-23T16:51:27.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023629639999999999 2024-04-23T16:51:27.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:27.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.266846+0000", 2024-04-23T16:51:27.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.076787639000000005 2024-04-23T16:51:27.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:51:27.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.266900+0000", 2024-04-23T16:51:27.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4128000000000001e-05 2024-04-23T16:51:27.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:47:40.266941+0000", 2024-04-23T16:51:27.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1078999999999998e-05 2024-04-23T16:51:27.071 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.072 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.072 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.072 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:124273 8.a 8:5f88b1cf:::benchmark_data_smithi188_155546_object124272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-04-23T16:51:27.072 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:48:05.458100+0000", 2024-04-23T16:51:27.072 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 201.59144307599999, 2024-04-23T16:51:27.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.076634463, 2024-04-23T16:51:27.072 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.072 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.072 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:27.072 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:51:27.072 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:51:27.073 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:51:27.073 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 124273 2024-04-23T16:51:27.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.073 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:05.458100+0000", 2024-04-23T16:51:27.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:05.458100+0000", 2024-04-23T16:51:27.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:05.458104+0000", 2024-04-23T16:51:27.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7050000000000001e-06 2024-04-23T16:51:27.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:05.458143+0000", 2024-04-23T16:51:27.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9357000000000003e-05 2024-04-23T16:51:27.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:05.458162+0000", 2024-04-23T16:51:27.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9321e-05 2024-04-23T16:51:27.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:05.458175+0000", 2024-04-23T16:51:27.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2641e-05 2024-04-23T16:51:27.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:05.458302+0000", 2024-04-23T16:51:27.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012706699999999999 2024-04-23T16:51:27.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:51:27.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:05.458482+0000", 2024-04-23T16:51:27.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017959200000000001 2024-04-23T16:51:27.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:51:27.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:05.458622+0000", 2024-04-23T16:51:27.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013978800000000001 2024-04-23T16:51:27.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:51:27.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:05.467400+0000", 2024-04-23T16:51:27.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087782099999999998 2024-04-23T16:51:27.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:27.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:05.534636+0000", 2024-04-23T16:51:27.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067235838000000006 2024-04-23T16:51:27.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:51:27.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:05.534689+0000", 2024-04-23T16:51:27.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3656000000000003e-05 2024-04-23T16:51:27.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:05.534735+0000", 2024-04-23T16:51:27.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5287999999999997e-05 2024-04-23T16:51:27.078 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.078 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.078 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.078 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:133294 8.a 8:5389bae5:::benchmark_data_smithi188_155546_object7785:head [delete] snapc 0=[] ondisk+write+known_if_redirected e649)", 2024-04-23T16:51:27.078 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:48:17.865339+0000", 2024-04-23T16:51:27.078 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 189.18420367300001, 2024-04-23T16:51:27.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.11017893300000001, 2024-04-23T16:51:27.078 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.078 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.079 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:27.079 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:51:27.079 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:51:27.079 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:51:27.079 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 133294 2024-04-23T16:51:27.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.079 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.865339+0000", 2024-04-23T16:51:27.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.865339+0000", 2024-04-23T16:51:27.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.865342+0000", 2024-04-23T16:51:27.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.368e-06 2024-04-23T16:51:27.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.865349+0000", 2024-04-23T16:51:27.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8210000000000004e-06 2024-04-23T16:51:27.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.865361+0000", 2024-04-23T16:51:27.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2682e-05 2024-04-23T16:51:27.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.865370+0000", 2024-04-23T16:51:27.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9029999999999993e-06 2024-04-23T16:51:27.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.867149+0000", 2024-04-23T16:51:27.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017787739999999999 2024-04-23T16:51:27.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:51:27.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.867264+0000", 2024-04-23T16:51:27.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000114688 2024-04-23T16:51:27.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:51:27.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.867367+0000", 2024-04-23T16:51:27.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010330100000000001 2024-04-23T16:51:27.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:51:27.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.874443+0000", 2024-04-23T16:51:27.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0070760140000000003 2024-04-23T16:51:27.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:27.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.975402+0000", 2024-04-23T16:51:27.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.100959457 2024-04-23T16:51:27.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:51:27.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.975463+0000", 2024-04-23T16:51:27.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0986000000000002e-05 2024-04-23T16:51:27.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.975518+0000", 2024-04-23T16:51:27.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4938999999999997e-05 2024-04-23T16:51:27.084 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.084 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.084 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.084 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:133311 8.a 8:5152764a:::benchmark_data_smithi188_155546_object7802:head [delete] snapc 0=[] ondisk+write+known_if_redirected e649)", 2024-04-23T16:51:27.084 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:48:17.877972+0000", 2024-04-23T16:51:27.085 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 189.17157159499999, 2024-04-23T16:51:27.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.097657944999999996, 2024-04-23T16:51:27.085 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.085 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.085 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:27.085 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:51:27.085 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:51:27.085 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:51:27.085 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 133311 2024-04-23T16:51:27.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.085 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.877972+0000", 2024-04-23T16:51:27.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.877972+0000", 2024-04-23T16:51:27.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.877974+0000", 2024-04-23T16:51:27.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.475e-06 2024-04-23T16:51:27.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.877982+0000", 2024-04-23T16:51:27.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6329999999999996e-06 2024-04-23T16:51:27.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.877995+0000", 2024-04-23T16:51:27.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3419e-05 2024-04-23T16:51:27.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.878004+0000", 2024-04-23T16:51:27.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0720000000000006e-06 2024-04-23T16:51:27.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.878065+0000", 2024-04-23T16:51:27.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0965999999999999e-05 2024-04-23T16:51:27.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:51:27.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.878171+0000", 2024-04-23T16:51:27.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010597100000000001 2024-04-23T16:51:27.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:51:27.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.878244+0000", 2024-04-23T16:51:27.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2607000000000005e-05 2024-04-23T16:51:27.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:51:27.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.884295+0000", 2024-04-23T16:51:27.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0060512400000000003 2024-04-23T16:51:27.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:27.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.975554+0000", 2024-04-23T16:51:27.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.091259187000000005 2024-04-23T16:51:27.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:51:27.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.975596+0000", 2024-04-23T16:51:27.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1573000000000003e-05 2024-04-23T16:51:27.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.975629+0000", 2024-04-23T16:51:27.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3801999999999999e-05 2024-04-23T16:51:27.090 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.090 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.090 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.091 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:133313 8.a 8:5858210a:::benchmark_data_smithi188_155546_object7804:head [delete] snapc 0=[] ondisk+write+known_if_redirected e649)", 2024-04-23T16:51:27.091 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:48:17.881286+0000", 2024-04-23T16:51:27.091 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 189.16825761300001, 2024-04-23T16:51:27.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.096694506999999999, 2024-04-23T16:51:27.091 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.091 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.091 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:27.091 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:51:27.091 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:51:27.091 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:51:27.091 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 133313 2024-04-23T16:51:27.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.091 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.881286+0000", 2024-04-23T16:51:27.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.881286+0000", 2024-04-23T16:51:27.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.881288+0000", 2024-04-23T16:51:27.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1689999999999999e-06 2024-04-23T16:51:27.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.881295+0000", 2024-04-23T16:51:27.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.917e-06 2024-04-23T16:51:27.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.881306+0000", 2024-04-23T16:51:27.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1667e-05 2024-04-23T16:51:27.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.881315+0000", 2024-04-23T16:51:27.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.636e-06 2024-04-23T16:51:27.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.881353+0000", 2024-04-23T16:51:27.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7700000000000002e-05 2024-04-23T16:51:27.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:51:27.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.881429+0000", 2024-04-23T16:51:27.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6556000000000004e-05 2024-04-23T16:51:27.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:51:27.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.881487+0000", 2024-04-23T16:51:27.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7661000000000002e-05 2024-04-23T16:51:27.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:51:27.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.884437+0000", 2024-04-23T16:51:27.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029501420000000002 2024-04-23T16:51:27.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:27.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.977881+0000", 2024-04-23T16:51:27.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.093444490000000005 2024-04-23T16:51:27.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:51:27.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.977933+0000", 2024-04-23T16:51:27.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1975999999999998e-05 2024-04-23T16:51:27.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.977980+0000", 2024-04-23T16:51:27.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6592999999999999e-05 2024-04-23T16:51:27.096 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.096 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.097 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.097 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:166122 8.a 8:5942b20f:::benchmark_data_smithi188_155546_object40613:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-23T16:51:27.097 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:49:03.366318+0000", 2024-04-23T16:51:27.097 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 143.68322509399999, 2024-04-23T16:51:27.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.086988368999999996, 2024-04-23T16:51:27.097 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.097 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.097 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:27.097 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:51:27.097 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:51:27.097 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:51:27.097 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 166122 2024-04-23T16:51:27.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.098 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.366318+0000", 2024-04-23T16:51:27.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.366318+0000", 2024-04-23T16:51:27.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.366320+0000", 2024-04-23T16:51:27.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.869e-06 2024-04-23T16:51:27.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.366326+0000", 2024-04-23T16:51:27.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.806e-06 2024-04-23T16:51:27.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.366336+0000", 2024-04-23T16:51:27.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0738e-05 2024-04-23T16:51:27.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.366344+0000", 2024-04-23T16:51:27.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8380000000000002e-06 2024-04-23T16:51:27.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.366401+0000", 2024-04-23T16:51:27.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6338000000000002e-05 2024-04-23T16:51:27.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:51:27.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.366477+0000", 2024-04-23T16:51:27.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6778000000000002e-05 2024-04-23T16:51:27.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:51:27.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.366543+0000", 2024-04-23T16:51:27.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5549000000000005e-05 2024-04-23T16:51:27.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:51:27.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.371878+0000", 2024-04-23T16:51:27.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053354229999999997 2024-04-23T16:51:27.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:27.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.453188+0000", 2024-04-23T16:51:27.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.081309598999999996 2024-04-23T16:51:27.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:51:27.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.453262+0000", 2024-04-23T16:51:27.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3620000000000003e-05 2024-04-23T16:51:27.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.453306+0000", 2024-04-23T16:51:27.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4811000000000002e-05 2024-04-23T16:51:27.103 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.103 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.103 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:166314 8.a 8:512dbd4d:::benchmark_data_smithi188_155546_object40805:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-23T16:51:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:49:03.654898+0000", 2024-04-23T16:51:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 143.39464488199999, 2024-04-23T16:51:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.073803167000000003, 2024-04-23T16:51:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:51:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:51:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:51:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 166314 2024-04-23T16:51:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.654898+0000", 2024-04-23T16:51:27.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.654898+0000", 2024-04-23T16:51:27.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.654900+0000", 2024-04-23T16:51:27.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.962e-06 2024-04-23T16:51:27.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.654905+0000", 2024-04-23T16:51:27.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.78e-06 2024-04-23T16:51:27.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.654915+0000", 2024-04-23T16:51:27.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7499999999999998e-06 2024-04-23T16:51:27.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.654922+0000", 2024-04-23T16:51:27.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9480000000000002e-06 2024-04-23T16:51:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.656204+0000", 2024-04-23T16:51:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001282643 2024-04-23T16:51:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:51:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.656325+0000", 2024-04-23T16:51:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000120467 2024-04-23T16:51:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:51:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.656428+0000", 2024-04-23T16:51:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000103477 2024-04-23T16:51:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:51:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.662380+0000", 2024-04-23T16:51:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059521840000000001 2024-04-23T16:51:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.728593+0000", 2024-04-23T16:51:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.066212971999999995 2024-04-23T16:51:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:51:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.728645+0000", 2024-04-23T16:51:27.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2034999999999999e-05 2024-04-23T16:51:27.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.728701+0000", 2024-04-23T16:51:27.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5949000000000003e-05 2024-04-23T16:51:27.109 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.109 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.109 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.109 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.16995.0:169126 8.c e663/609 8:3dc973af:::benchmark_data_smithi188_155546_object43617:head v 663'10586, mlcod=663'10584)", 2024-04-23T16:51:27.109 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:49:07.546171+0000", 2024-04-23T16:51:27.109 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 139.503372329, 2024-04-23T16:51:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.085263928000000003, 2024-04-23T16:51:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.546171+0000", 2024-04-23T16:51:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.546171+0000", 2024-04-23T16:51:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.546171+0000", 2024-04-23T16:51:27.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4900000000000001e-07 2024-04-23T16:51:27.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.546177+0000", 2024-04-23T16:51:27.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7470000000000002e-06 2024-04-23T16:51:27.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.546182+0000", 2024-04-23T16:51:27.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2190000000000003e-06 2024-04-23T16:51:27.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.546187+0000", 2024-04-23T16:51:27.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6720000000000003e-06 2024-04-23T16:51:27.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.546253+0000", 2024-04-23T16:51:27.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5801000000000002e-05 2024-04-23T16:51:27.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:51:27.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.546272+0000", 2024-04-23T16:51:27.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9287999999999999e-05 2024-04-23T16:51:27.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:51:27.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.631394+0000", 2024-04-23T16:51:27.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.085121727999999994 2024-04-23T16:51:27.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.631435+0000", 2024-04-23T16:51:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1023999999999999e-05 2024-04-23T16:51:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:169139 8.a 8:5e8e7fbe:::benchmark_data_smithi188_155546_object43630:head [delete] snapc 0=[] ondisk+write+known_if_redirected e663)", 2024-04-23T16:51:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:49:07.554266+0000", 2024-04-23T16:51:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 139.49527665100001, 2024-04-23T16:51:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.077354949000000006, 2024-04-23T16:51:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.115 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:27.115 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:51:27.115 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:51:27.115 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:51:27.115 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 169139 2024-04-23T16:51:27.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.115 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.554266+0000", 2024-04-23T16:51:27.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.554266+0000", 2024-04-23T16:51:27.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.554269+0000", 2024-04-23T16:51:27.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0260000000000002e-06 2024-04-23T16:51:27.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.554275+0000", 2024-04-23T16:51:27.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.511e-06 2024-04-23T16:51:27.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.620544+0000", 2024-04-23T16:51:27.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.066268937 2024-04-23T16:51:27.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.620556+0000", 2024-04-23T16:51:27.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1783000000000001e-05 2024-04-23T16:51:27.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.620822+0000", 2024-04-23T16:51:27.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000266323 2024-04-23T16:51:27.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:51:27.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.620954+0000", 2024-04-23T16:51:27.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013155400000000001 2024-04-23T16:51:27.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:51:27.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.621069+0000", 2024-04-23T16:51:27.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000115617 2024-04-23T16:51:27.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:51:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.631385+0000", 2024-04-23T16:51:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010315349999999999 2024-04-23T16:51:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.631535+0000", 2024-04-23T16:51:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00015021199999999999 2024-04-23T16:51:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:51:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.631579+0000", 2024-04-23T16:51:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4688000000000003e-05 2024-04-23T16:51:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:07.631621+0000", 2024-04-23T16:51:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1947999999999999e-05 2024-04-23T16:51:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:218218 8.a 8:5770b466:::benchmark_data_smithi188_155546_object92709:head [delete] snapc 0=[] ondisk+write+known_if_redirected e677)", 2024-04-23T16:51:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:50:04.344434+0000", 2024-04-23T16:51:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 82.705109350000001, 2024-04-23T16:51:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.097772399999999995, 2024-04-23T16:51:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:27.121 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:51:27.121 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:51:27.121 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:51:27.121 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 218218 2024-04-23T16:51:27.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.121 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.344434+0000", 2024-04-23T16:51:27.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.344434+0000", 2024-04-23T16:51:27.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.344436+0000", 2024-04-23T16:51:27.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.43e-06 2024-04-23T16:51:27.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.344443+0000", 2024-04-23T16:51:27.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0439999999999998e-06 2024-04-23T16:51:27.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.344458+0000", 2024-04-23T16:51:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4704e-05 2024-04-23T16:51:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.344467+0000", 2024-04-23T16:51:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3749999999999992e-06 2024-04-23T16:51:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.349154+0000", 2024-04-23T16:51:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046870829999999999 2024-04-23T16:51:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:51:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.349288+0000", 2024-04-23T16:51:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000133602 2024-04-23T16:51:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:51:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.349399+0000", 2024-04-23T16:51:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000111376 2024-04-23T16:51:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:51:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.359249+0000", 2024-04-23T16:51:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0098499739999999992 2024-04-23T16:51:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.442108+0000", 2024-04-23T16:51:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.082858672999999994 2024-04-23T16:51:27.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:51:27.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.442164+0000", 2024-04-23T16:51:27.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6242000000000001e-05 2024-04-23T16:51:27.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.442206+0000", 2024-04-23T16:51:27.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1897000000000002e-05 2024-04-23T16:51:27.125 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.125 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.125 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:218236 8.a 8:54b6fb57:::benchmark_data_smithi188_155546_object92727:head [delete] snapc 0=[] ondisk+write+known_if_redirected e677)", 2024-04-23T16:51:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:50:04.362539+0000", 2024-04-23T16:51:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 82.687003801000003, 2024-04-23T16:51:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.079806109, 2024-04-23T16:51:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:51:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:51:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:51:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 218236 2024-04-23T16:51:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.362539+0000", 2024-04-23T16:51:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.362539+0000", 2024-04-23T16:51:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.362546+0000", 2024-04-23T16:51:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0620000000000003e-06 2024-04-23T16:51:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.362554+0000", 2024-04-23T16:51:27.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1969999999999999e-06 2024-04-23T16:51:27.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.362566+0000", 2024-04-23T16:51:27.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2344000000000001e-05 2024-04-23T16:51:27.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.362575+0000", 2024-04-23T16:51:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1949999999999992e-06 2024-04-23T16:51:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.362641+0000", 2024-04-23T16:51:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5493000000000005e-05 2024-04-23T16:51:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:51:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.362787+0000", 2024-04-23T16:51:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014641599999999999 2024-04-23T16:51:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:51:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.362908+0000", 2024-04-23T16:51:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000120808 2024-04-23T16:51:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:51:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.367327+0000", 2024-04-23T16:51:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00441932 2024-04-23T16:51:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.442242+0000", 2024-04-23T16:51:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.074914526999999995 2024-04-23T16:51:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:51:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.442283+0000", 2024-04-23T16:51:27.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1229000000000001e-05 2024-04-23T16:51:27.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.442345+0000", 2024-04-23T16:51:27.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2518e-05 2024-04-23T16:51:27.131 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.131 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.131 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.131 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:218239 8.a 8:5ae69382:::benchmark_data_smithi188_155546_object92730:head [delete] snapc 0=[] ondisk+write+known_if_redirected e677)", 2024-04-23T16:51:27.131 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:50:04.368257+0000", 2024-04-23T16:51:27.131 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 82.681286444999998, 2024-04-23T16:51:27.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.091060946000000004, 2024-04-23T16:51:27.132 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.132 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.132 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:27.132 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:51:27.132 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:51:27.132 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:51:27.132 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 218239 2024-04-23T16:51:27.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.132 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.368257+0000", 2024-04-23T16:51:27.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.368257+0000", 2024-04-23T16:51:27.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.368259+0000", 2024-04-23T16:51:27.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3149999999999999e-06 2024-04-23T16:51:27.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.368264+0000", 2024-04-23T16:51:27.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.135e-06 2024-04-23T16:51:27.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.368276+0000", 2024-04-23T16:51:27.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2140999999999999e-05 2024-04-23T16:51:27.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.368284+0000", 2024-04-23T16:51:27.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8180000000000004e-06 2024-04-23T16:51:27.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.368345+0000", 2024-04-23T16:51:27.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0543e-05 2024-04-23T16:51:27.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:51:27.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.368448+0000", 2024-04-23T16:51:27.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000103761 2024-04-23T16:51:27.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:51:27.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.368556+0000", 2024-04-23T16:51:27.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000108076 2024-04-23T16:51:27.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:51:27.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.377660+0000", 2024-04-23T16:51:27.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091033409999999992 2024-04-23T16:51:27.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:27.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.459218+0000", 2024-04-23T16:51:27.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.081558418999999993 2024-04-23T16:51:27.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:51:27.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.459271+0000", 2024-04-23T16:51:27.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3121999999999997e-05 2024-04-23T16:51:27.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.459318+0000", 2024-04-23T16:51:27.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6275e-05 2024-04-23T16:51:27.137 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.137 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.137 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.138 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:218243 8.a 8:55986bd0:::benchmark_data_smithi188_155546_object92734:head [delete] snapc 0=[] ondisk+write+known_if_redirected e677)", 2024-04-23T16:51:27.138 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:50:04.372705+0000", 2024-04-23T16:51:27.138 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 82.676838114999995, 2024-04-23T16:51:27.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.086727655000000001, 2024-04-23T16:51:27.138 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.138 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.138 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:51:27.138 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:51:27.138 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:51:27.138 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:51:27.138 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 218243 2024-04-23T16:51:27.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.138 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.372705+0000", 2024-04-23T16:51:27.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.372705+0000", 2024-04-23T16:51:27.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.372708+0000", 2024-04-23T16:51:27.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7580000000000001e-06 2024-04-23T16:51:27.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.372716+0000", 2024-04-23T16:51:27.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3319999999999992e-06 2024-04-23T16:51:27.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.372727+0000", 2024-04-23T16:51:27.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1342000000000001e-05 2024-04-23T16:51:27.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.372736+0000", 2024-04-23T16:51:27.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8750000000000006e-06 2024-04-23T16:51:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.372815+0000", 2024-04-23T16:51:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8734999999999997e-05 2024-04-23T16:51:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:51:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.372928+0000", 2024-04-23T16:51:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000112454 2024-04-23T16:51:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:51:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.373042+0000", 2024-04-23T16:51:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000114657 2024-04-23T16:51:27.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:51:27.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.378390+0000", 2024-04-23T16:51:27.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053481529999999996 2024-04-23T16:51:27.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:51:27.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.459353+0000", 2024-04-23T16:51:27.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.080962440999999996 2024-04-23T16:51:27.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:51:27.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.459399+0000", 2024-04-23T16:51:27.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6479e-05 2024-04-23T16:51:27.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.459433+0000", 2024-04-23T16:51:27.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3429000000000001e-05 2024-04-23T16:51:27.143 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.143 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.143 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.144 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:245129 8.b 8.3b61954b (undecoded) ondisk+retry+write+known_if_redirected e693)", 2024-04-23T16:51:27.144 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:50:52.629359+0000", 2024-04-23T16:51:27.144 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.420184257999999, 2024-04-23T16:51:27.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.147168771, 2024-04-23T16:51:27.144 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.144 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.144 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:51:27.144 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:51:27.144 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:51:27.144 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:51:27.144 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 245129 2024-04-23T16:51:27.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.145 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629359+0000", 2024-04-23T16:51:27.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629359+0000", 2024-04-23T16:51:27.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629362+0000", 2024-04-23T16:51:27.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3400000000000002e-06 2024-04-23T16:51:27.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629388+0000", 2024-04-23T16:51:27.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5666000000000001e-05 2024-04-23T16:51:27.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629404+0000", 2024-04-23T16:51:27.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5747999999999999e-05 2024-04-23T16:51:27.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629416+0000", 2024-04-23T16:51:27.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2317000000000001e-05 2024-04-23T16:51:27.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629556+0000", 2024-04-23T16:51:27.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013970200000000001 2024-04-23T16:51:27.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:51:27.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629577+0000", 2024-04-23T16:51:27.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1243e-05 2024-04-23T16:51:27.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776495+0000", 2024-04-23T16:51:27.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14691843399999999 2024-04-23T16:51:27.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776528+0000", 2024-04-23T16:51:27.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2320999999999998e-05 2024-04-23T16:51:27.148 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.148 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.148 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.148 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:245137 8.b 8.7c0e014b (undecoded) ondisk+retry+write+known_if_redirected e693)", 2024-04-23T16:51:27.149 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:50:52.629445+0000", 2024-04-23T16:51:27.149 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.420098617000001, 2024-04-23T16:51:27.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14719322100000001, 2024-04-23T16:51:27.149 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.149 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.149 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:51:27.149 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:51:27.149 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:51:27.149 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:51:27.149 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 245137 2024-04-23T16:51:27.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.149 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629445+0000", 2024-04-23T16:51:27.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629445+0000", 2024-04-23T16:51:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629445+0000", 2024-04-23T16:51:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3200000000000003e-07 2024-04-23T16:51:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629449+0000", 2024-04-23T16:51:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8700000000000002e-06 2024-04-23T16:51:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629455+0000", 2024-04-23T16:51:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1079999999999998e-06 2024-04-23T16:51:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629461+0000", 2024-04-23T16:51:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8420000000000001e-06 2024-04-23T16:51:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629673+0000", 2024-04-23T16:51:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000211664 2024-04-23T16:51:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:51:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629684+0000", 2024-04-23T16:51:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0628999999999999e-05 2024-04-23T16:51:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776631+0000", 2024-04-23T16:51:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14694732299999999 2024-04-23T16:51:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776638+0000", 2024-04-23T16:51:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8530000000000003e-06 2024-04-23T16:51:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:245145 8.b 8.8ae4058b (undecoded) ondisk+retry+write+known_if_redirected e693)", 2024-04-23T16:51:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:50:52.629482+0000", 2024-04-23T16:51:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.420060876999997, 2024-04-23T16:51:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14720270899999999, 2024-04-23T16:51:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:51:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:51:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:51:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:51:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 245145 2024-04-23T16:51:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.154 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629482+0000", 2024-04-23T16:51:27.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629482+0000", 2024-04-23T16:51:27.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629483+0000", 2024-04-23T16:51:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6899999999999996e-07 2024-04-23T16:51:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629486+0000", 2024-04-23T16:51:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3550000000000001e-06 2024-04-23T16:51:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629497+0000", 2024-04-23T16:51:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0896999999999999e-05 2024-04-23T16:51:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629503+0000", 2024-04-23T16:51:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9959999999999999e-06 2024-04-23T16:51:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629753+0000", 2024-04-23T16:51:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00025012899999999998 2024-04-23T16:51:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:51:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629775+0000", 2024-04-23T16:51:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1433e-05 2024-04-23T16:51:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776675+0000", 2024-04-23T16:51:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14690025100000001 2024-04-23T16:51:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776685+0000", 2024-04-23T16:51:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8789999999999998e-06 2024-04-23T16:51:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:245150 8.b 8.40b84c3b (undecoded) ondisk+retry+write+known_if_redirected e693)", 2024-04-23T16:51:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:50:52.629519+0000", 2024-04-23T16:51:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.420024578000003, 2024-04-23T16:51:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14719273099999999, 2024-04-23T16:51:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.158 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.158 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:51:27.158 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:51:27.158 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:51:27.158 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:51:27.158 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 245150 2024-04-23T16:51:27.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.158 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629519+0000", 2024-04-23T16:51:27.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629519+0000", 2024-04-23T16:51:27.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629519+0000", 2024-04-23T16:51:27.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1299999999999999e-07 2024-04-23T16:51:27.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629523+0000", 2024-04-23T16:51:27.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3780000000000001e-06 2024-04-23T16:51:27.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629530+0000", 2024-04-23T16:51:27.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1620000000000002e-06 2024-04-23T16:51:27.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629535+0000", 2024-04-23T16:51:27.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3299999999999998e-06 2024-04-23T16:51:27.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629804+0000", 2024-04-23T16:51:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000268671 2024-04-23T16:51:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:51:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629812+0000", 2024-04-23T16:51:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0749999999999998e-06 2024-04-23T16:51:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776706+0000", 2024-04-23T16:51:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14689437699999999 2024-04-23T16:51:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776711+0000", 2024-04-23T16:51:27.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0250000000000002e-06 2024-04-23T16:51:27.162 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.162 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.162 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.162 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:245162 8.b 8.2eb619eb (undecoded) ondisk+retry+write+known_if_redirected e693)", 2024-04-23T16:51:27.162 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:50:52.629543+0000", 2024-04-23T16:51:27.162 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.420000453999997, 2024-04-23T16:51:27.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14728439099999999, 2024-04-23T16:51:27.162 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:51:27.162 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:51:27.162 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:51:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:51:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:51:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:51:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 245162 2024-04-23T16:51:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:51:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:51:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629543+0000", 2024-04-23T16:51:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:51:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629543+0000", 2024-04-23T16:51:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:51:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:51:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629543+0000", 2024-04-23T16:51:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8500000000000001e-07 2024-04-23T16:51:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:51:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629546+0000", 2024-04-23T16:51:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8679999999999999e-06 2024-04-23T16:51:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:51:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629551+0000", 2024-04-23T16:51:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.904e-06 2024-04-23T16:51:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:51:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629556+0000", 2024-04-23T16:51:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1889999999999998e-06 2024-04-23T16:51:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629870+0000", 2024-04-23T16:51:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031422699999999999 2024-04-23T16:51:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:51:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629882+0000", 2024-04-23T16:51:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1571e-05 2024-04-23T16:51:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:51:27.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776822+0000", 2024-04-23T16:51:27.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14694010699999999 2024-04-23T16:51:27.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:51:27.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:51:27.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:51:27.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776827+0000", 2024-04-23T16:51:27.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9400000000000001e-06 2024-04-23T16:51:27.166 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.166 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.166 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.166 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:51:27.166 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:51:27.167 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:51:27.168 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:51:27.168 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:51:27.168 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:51:27.168 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:51:27.168 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:51:27.168 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:51:27.186 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:51:27.205 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:51:27.225 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:51:27.244 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:51:27.264 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:51:27.265 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:51:27.265 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:51:27.265 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:51:27.265 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:51:27.265 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-23T16:51:27.265 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:51:27.265 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:51:27.265 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:51:27.265 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:51:27.265 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:51:27.265 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:51:27.266 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:51:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:27 smithi103 ceph-mon[39629]: 1.9 deep-scrub starts 2024-04-23T16:51:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:27 smithi103 ceph-mon[39629]: 1.9 deep-scrub ok 2024-04-23T16:51:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:27 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2872663820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:27.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:27 smithi148 ceph-mon[44520]: 1.9 deep-scrub starts 2024-04-23T16:51:27.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:27 smithi148 ceph-mon[44520]: 1.9 deep-scrub ok 2024-04-23T16:51:27.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:27 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2872663820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:27.377 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:27 smithi188 ceph-mon[44101]: 1.9 deep-scrub starts 2024-04-23T16:51:27.377 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:27 smithi188 ceph-mon[44101]: 1.9 deep-scrub ok 2024-04-23T16:51:27.377 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:27 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2872663820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:27.592 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:51:27.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:27.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:51:27.697 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:51:27.701 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-23T16:51:27.726 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:27.964 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:51:28.177 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:51:28.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:28.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:51:28.282 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:51:28.311 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:28.330 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:51:28.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:28 smithi188 ceph-mon[44101]: pgmap v1963: 27 pgs: 27 active+clean; 577 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:28.333 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.333 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:28 smithi188 ceph-mon[44101]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:51:28.333 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:28 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:51:28.333 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.333 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.333 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.333 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.333 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.333 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.333 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.333 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:28 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:28.334 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:51:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:28 smithi103 ceph-mon[39629]: pgmap v1963: 27 pgs: 27 active+clean; 577 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:28 smithi103 ceph-mon[39629]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:51:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:28 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:51:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:28 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:28 smithi148 ceph-mon[44520]: pgmap v1963: 27 pgs: 27 active+clean; 577 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:28.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:28 smithi148 ceph-mon[44520]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T16:51:28.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:28 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:51:28.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:28 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:28.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:51:28.462 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:51:28.955 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:51:28.969 DEBUG:teuthology.orchestra.run.smithi188:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-04-23T16:51:28.972 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-23T16:51:28.972 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-23T16:51:29.064 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:hints = 1 2024-04-23T16:51:29.064 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-23T16:51:29.064 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Object prefix: benchmark_data_smithi188_167360 2024-04-23T16:51:29.064 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:51:29.064 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 0 0 0 0 0 0 - 0 2024-04-23T16:51:29.081 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:51:29.082 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:51:29.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:29.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:29 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:29.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:29 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:51:29.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:29 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:51:29.272 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:51:29.312 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:51:29.315 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:29.316 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:29 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:29.316 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:29 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:51:29.316 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:29 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:51:29.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:29.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:51:29.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:29 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:29 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:51:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:29 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:51:29.417 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:51:29.456 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:29.535 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:51:29.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:29.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:51:29.641 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:51:29.687 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:29.827 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:51:29.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:29.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:51:29.863 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:51:29.863 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:51:29.939 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:51:29.976 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:30.064 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 1 16 4787 4771 298.115 298.188 0.00178776 0.00334904 2024-04-23T16:51:30.082 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:51:30.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:30.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:51:30.187 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:51:30.227 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:30 smithi103 ceph-mon[39629]: from='client.17487 v1:172.21.15.103:0/246295118' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:51:30.227 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:30 smithi103 ceph-mon[39629]: pgmap v1964: 27 pgs: 27 active+clean; 577 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:30.234 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:30.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:30 smithi148 ceph-mon[44520]: from='client.17487 v1:172.21.15.103:0/246295118' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:51:30.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:30 smithi148 ceph-mon[44520]: pgmap v1964: 27 pgs: 27 active+clean; 577 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:30.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:30 smithi188 ceph-mon[44101]: from='client.17487 v1:172.21.15.103:0/246295118' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:51:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:30 smithi188 ceph-mon[44101]: pgmap v1964: 27 pgs: 27 active+clean; 577 KiB data, 4.2 GiB used, 622 GiB / 626 GiB avail 2024-04-23T16:51:30.605 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-23T16:51:30.668 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1963,"stamp":"2024-04-23T16:51:26.948830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":4357216,"kb_used_data":6464,"kb_used_omap":240,"kb_used_meta":4350415,"kb_avail":651858848,"statfs":{"total":671965249536,"available":667503460352,"internally_reserved":0,"allocated":6619136,"data_stored":3778582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":246226,"internal_metadata":4454825518},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.892927"},"pg_stats":[{"pgid":"9.7","version":"0'0","reported_seq":17,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-23T16:51:25.456887+0000","last_change":"2024-04-23T16:51:21.064703+0000","last_active":"2024-04-23T16:51:25.456887+0000","last_peered":"2024-04-23T16:51:25.456887+0000","last_clean":"2024-04-23T16:51:25.456887+0000","last_became_active":"2024-04-23T16:51:21.064432+0000","last_became_peered":"2024-04-23T16:51:21.064432+0000","last_unstale":"2024-04-23T16:51:25.456887+0000","last_undegraded":"2024-04-23T16:51:25.456887+0000","last_fullsized":"2024-04-23T16:51:25.456887+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:41.570514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":14,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-04-23T16:51:23.129793+0000","last_change":"2024-04-23T16:51:21.063927+0000","last_active":"2024-04-23T16:51:23.129793+0000","last_peered":"2024-04-23T16:51:23.129793+0000","last_clean":"2024-04-23T16:51:23.129793+0000","last_became_active":"2024-04-23T16:51:21.063642+0000","last_became_peered":"2024-04-23T16:51:21.063642+0000","last_unstale":"2024-04-23T16:51:23.129793+0000","last_undegraded":"2024-04-23T16:51:23.129793+0000","last_fullsized":"2024-04-23T16:51:23.129793+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:31.855738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-04-23T16:51:22.062747+0000","last_change":"2024-04-23T16:51:21.064490+0000","last_active":"2024-04-23T16:51:22.062747+0000","last_peered":"2024-04-23T16:51:22.062747+0000","last_clean":"2024-04-23T16:51:22.062747+0000","last_became_active":"2024-04-23T16:51:21.063776+0000","last_became_peered":"2024-04-23T16:51:21.063776+0000","last_unstale":"2024-04-23T16:51:22.062747+0000","last_undegraded":"2024-04-23T16:51:22.062747+0000","last_fullsized":"2024-04-23T16:51:22.062747+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:30.226366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-04-23T16:51:23.129663+0000","last_change":"2024-04-23T16:51:21.063360+0000","last_active":"2024-04-23T16:51:23.129663+0000","last_peered":"2024-04-23T16:51:23.129663+0000","last_clean":"2024-04-23T16:51:23.129663+0000","last_became_active":"2024-04-23T16:51:21.063114+0000","last_became_peered":"2024-04-23T16:51:21.063114+0000","last_unstale":"2024-04-23T16:51:23.129663+0000","last_undegraded":"2024-04-23T16:51:23.129663+0000","last_fullsized":"2024-04-23T16:51:23.129663+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:28.626485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":13,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-04-23T16:51:23.069208+0000","last_change":"2024-04-23T16:51:21.063893+0000","last_active":"2024-04-23T16:51:23.069208+0000","last_peered":"2024-04-23T16:51:23.069208+0000","last_clean":"2024-04-23T16:51:23.069208+0000","last_became_active":"2024-04-23T16:51:21.063652+0000","last_became_peered":"2024-04-23T16:51:21.063652+0000","last_unstale":"2024-04-23T16:51:23.069208+0000","last_undegraded":"2024-04-23T16:51:23.069208+0000","last_fullsized":"2024-04-23T16:51:23.069208+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:46.384986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":13,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-04-23T16:51:23.067950+0000","last_change":"2024-04-23T16:51:21.062805+0000","last_active":"2024-04-23T16:51:23.067950+0000","last_peered":"2024-04-23T16:51:23.067950+0000","last_clean":"2024-04-23T16:51:23.067950+0000","last_became_active":"2024-04-23T16:51:21.062561+0000","last_became_peered":"2024-04-23T16:51:21.062561+0000","last_unstale":"2024-04-23T16:51:23.067950+0000","last_undegraded":"2024-04-23T16:51:23.067950+0000","last_fullsized":"2024-04-23T16:51:23.067950+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:33.520364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1751,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-04-23T16:51:23.130173+0000","last_change":"2024-04-23T16:51:20.224584+0000","last_active":"2024-04-23T16:51:23.130173+0000","last_peered":"2024-04-23T16:51:23.130173+0000","last_clean":"2024-04-23T16:51:23.130173+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:51:23.130173+0000","last_undegraded":"2024-04-23T16:51:23.130173+0000","last_fullsized":"2024-04-23T16:51:23.130173+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:51:20.224480+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.224480+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:47.325219+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-04-23T16:51:23.129615+0000","last_change":"2024-04-23T16:51:21.063626+0000","last_active":"2024-04-23T16:51:23.129615+0000","last_peered":"2024-04-23T16:51:23.129615+0000","last_clean":"2024-04-23T16:51:23.129615+0000","last_became_active":"2024-04-23T16:51:21.063395+0000","last_became_peered":"2024-04-23T16:51:21.063395+0000","last_unstale":"2024-04-23T16:51:23.129615+0000","last_undegraded":"2024-04-23T16:51:23.129615+0000","last_fullsized":"2024-04-23T16:51:23.129615+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:23.448353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1631,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-04-23T16:51:23.067797+0000","last_change":"2024-04-23T16:50:53.723122+0000","last_active":"2024-04-23T16:51:23.067797+0000","last_peered":"2024-04-23T16:51:23.067797+0000","last_clean":"2024-04-23T16:51:23.067797+0000","last_became_active":"2024-04-23T16:50:51.611397+0000","last_became_peered":"2024-04-23T16:50:51.611397+0000","last_unstale":"2024-04-23T16:51:23.067797+0000","last_undegraded":"2024-04-23T16:51:23.067797+0000","last_fullsized":"2024-04-23T16:51:23.067797+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:50:53.722996+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:50:53.722996+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:02.512304+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-04-23T16:51:23.130215+0000","last_change":"2024-04-23T16:51:21.246684+0000","last_active":"2024-04-23T16:51:23.130215+0000","last_peered":"2024-04-23T16:51:23.130215+0000","last_clean":"2024-04-23T16:51:23.130215+0000","last_became_active":"2024-04-23T16:51:21.246391+0000","last_became_peered":"2024-04-23T16:51:21.246391+0000","last_unstale":"2024-04-23T16:51:23.130215+0000","last_undegraded":"2024-04-23T16:51:23.130215+0000","last_fullsized":"2024-04-23T16:51:23.130215+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:49.163071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1745,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-23T16:51:25.456766+0000","last_change":"2024-04-23T16:51:25.290906+0000","last_active":"2024-04-23T16:51:25.456766+0000","last_peered":"2024-04-23T16:51:25.456766+0000","last_clean":"2024-04-23T16:51:25.456766+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:51:25.456766+0000","last_undegraded":"2024-04-23T16:51:25.456766+0000","last_fullsized":"2024-04-23T16:51:25.456766+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:51:25.290774+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:51:25.290774+0000","last_clean_scrub_stamp":"2024-04-23T16:51:25.290774+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:30.207942+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-04-23T16:51:23.129628+0000","last_change":"2024-04-23T16:51:21.063902+0000","last_active":"2024-04-23T16:51:23.129628+0000","last_peered":"2024-04-23T16:51:23.129628+0000","last_clean":"2024-04-23T16:51:23.129628+0000","last_became_active":"2024-04-23T16:51:21.063669+0000","last_became_peered":"2024-04-23T16:51:21.063669+0000","last_unstale":"2024-04-23T16:51:23.129628+0000","last_undegraded":"2024-04-23T16:51:23.129628+0000","last_fullsized":"2024-04-23T16:51:23.129628+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:49.396373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1750,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-04-23T16:51:23.067007+0000","last_change":"2024-04-23T16:50:12.233877+0000","last_active":"2024-04-23T16:51:23.067007+0000","last_peered":"2024-04-23T16:51:23.067007+0000","last_clean":"2024-04-23T16:51:23.067007+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:51:23.067007+0000","last_undegraded":"2024-04-23T16:51:23.067007+0000","last_fullsized":"2024-04-23T16:51:23.067007+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:50:12.233741+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:50:12.233741+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:51:29.028698+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":13,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-04-23T16:51:23.067929+0000","last_change":"2024-04-23T16:51:21.063044+0000","last_active":"2024-04-23T16:51:23.067929+0000","last_peered":"2024-04-23T16:51:23.067929+0000","last_clean":"2024-04-23T16:51:23.067929+0000","last_became_active":"2024-04-23T16:51:21.062760+0000","last_became_peered":"2024-04-23T16:51:21.062760+0000","last_unstale":"2024-04-23T16:51:23.067929+0000","last_undegraded":"2024-04-23T16:51:23.067929+0000","last_fullsized":"2024-04-23T16:51:23.067929+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:37.155174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1333,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-04-23T16:51:23.129661+0000","last_change":"2024-04-23T16:50:10.581817+0000","last_active":"2024-04-23T16:51:23.129661+0000","last_peered":"2024-04-23T16:51:23.129661+0000","last_clean":"2024-04-23T16:51:23.129661+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:51:23.129661+0000","last_undegraded":"2024-04-23T16:51:23.129661+0000","last_fullsized":"2024-04-23T16:51:23.129661+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:50:10.581707+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:50:10.581707+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:51:24.942124+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-04-23T16:51:22.062725+0000","last_change":"2024-04-23T16:51:21.246813+0000","last_active":"2024-04-23T16:51:22.062725+0000","last_peered":"2024-04-23T16:51:22.062725+0000","last_clean":"2024-04-23T16:51:22.062725+0000","last_became_active":"2024-04-23T16:51:21.246519+0000","last_became_peered":"2024-04-23T16:51:21.246519+0000","last_unstale":"2024-04-23T16:51:22.062725+0000","last_undegraded":"2024-04-23T16:51:22.062725+0000","last_fullsized":"2024-04-23T16:51:22.062725+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:43.259608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1286,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-04-23T16:51:23.068143+0000","last_change":"2024-04-23T16:50:05.645859+0000","last_active":"2024-04-23T16:51:23.068143+0000","last_peered":"2024-04-23T16:51:23.068143+0000","last_clean":"2024-04-23T16:51:23.068143+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:51:23.068143+0000","last_undegraded":"2024-04-23T16:51:23.068143+0000","last_fullsized":"2024-04-23T16:51:23.068143+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:50:05.645764+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:50:05.645764+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":17,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-23T16:51:25.456869+0000","last_change":"2024-04-23T16:51:21.064696+0000","last_active":"2024-04-23T16:51:25.456869+0000","last_peered":"2024-04-23T16:51:25.456869+0000","last_clean":"2024-04-23T16:51:25.456869+0000","last_became_active":"2024-04-23T16:51:21.064486+0000","last_became_peered":"2024-04-23T16:51:21.064486+0000","last_unstale":"2024-04-23T16:51:25.456869+0000","last_undegraded":"2024-04-23T16:51:25.456869+0000","last_fullsized":"2024-04-23T16:51:25.456869+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:35.126146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1759,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-04-23T16:51:23.130369+0000","last_change":"2024-04-23T16:51:18.204631+0000","last_active":"2024-04-23T16:51:23.130369+0000","last_peered":"2024-04-23T16:51:23.130369+0000","last_clean":"2024-04-23T16:51:23.130369+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:51:23.130369+0000","last_undegraded":"2024-04-23T16:51:23.130369+0000","last_fullsized":"2024-04-23T16:51:23.130369+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:51:18.204524+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:51:18.204524+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:24.446712+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":14,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-04-23T16:51:23.129763+0000","last_change":"2024-04-23T16:51:21.063851+0000","last_active":"2024-04-23T16:51:23.129763+0000","last_peered":"2024-04-23T16:51:23.129763+0000","last_clean":"2024-04-23T16:51:23.129763+0000","last_became_active":"2024-04-23T16:51:21.063588+0000","last_became_peered":"2024-04-23T16:51:21.063588+0000","last_unstale":"2024-04-23T16:51:23.129763+0000","last_undegraded":"2024-04-23T16:51:23.129763+0000","last_fullsized":"2024-04-23T16:51:23.129763+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:23.541067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1757,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-04-23T16:51:23.130336+0000","last_change":"2024-04-23T16:50:10.300939+0000","last_active":"2024-04-23T16:51:23.130336+0000","last_peered":"2024-04-23T16:51:23.130336+0000","last_clean":"2024-04-23T16:51:23.130336+0000","last_became_active":"2024-04-23T16:47:10.266014+0000","last_became_peered":"2024-04-23T16:47:10.266014+0000","last_unstale":"2024-04-23T16:51:23.130336+0000","last_undegraded":"2024-04-23T16:51:23.130336+0000","last_fullsized":"2024-04-23T16:51:23.130336+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:50:10.300793+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:50:10.300793+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:51:32.846335+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-04-23T16:51:23.130322+0000","last_change":"2024-04-23T16:51:21.080034+0000","last_active":"2024-04-23T16:51:23.130322+0000","last_peered":"2024-04-23T16:51:23.130322+0000","last_clean":"2024-04-23T16:51:23.130322+0000","last_became_active":"2024-04-23T16:51:21.079798+0000","last_became_peered":"2024-04-23T16:51:21.079798+0000","last_unstale":"2024-04-23T16:51:23.130322+0000","last_undegraded":"2024-04-23T16:51:23.130322+0000","last_fullsized":"2024-04-23T16:51:23.130322+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:47.411506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1634,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-04-23T16:51:22.062611+0000","last_change":"2024-04-23T16:50:21.753563+0000","last_active":"2024-04-23T16:51:22.062611+0000","last_peered":"2024-04-23T16:51:22.062611+0000","last_clean":"2024-04-23T16:51:22.062611+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:51:22.062611+0000","last_undegraded":"2024-04-23T16:51:22.062611+0000","last_fullsized":"2024-04-23T16:51:22.062611+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:50:21.753450+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:50:21.753450+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:51:41.041517+0000","scrub_duration":277,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":17,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-23T16:51:25.456906+0000","last_change":"2024-04-23T16:51:21.064751+0000","last_active":"2024-04-23T16:51:25.456906+0000","last_peered":"2024-04-23T16:51:25.456906+0000","last_clean":"2024-04-23T16:51:25.456906+0000","last_became_active":"2024-04-23T16:51:21.064506+0000","last_became_peered":"2024-04-23T16:51:21.064506+0000","last_unstale":"2024-04-23T16:51:25.456906+0000","last_undegraded":"2024-04-23T16:51:25.456906+0000","last_fullsized":"2024-04-23T16:51:25.456906+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:37.912311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1574,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-04-23T16:51:23.130304+0000","last_change":"2024-04-23T16:51:19.334794+0000","last_active":"2024-04-23T16:51:23.130304+0000","last_peered":"2024-04-23T16:51:23.130304+0000","last_clean":"2024-04-23T16:51:23.130304+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:51:23.130304+0000","last_undegraded":"2024-04-23T16:51:23.130304+0000","last_fullsized":"2024-04-23T16:51:23.130304+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:51:19.334693+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:51:19.334693+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:41.467169+0000","scrub_duration":146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-04-23T16:51:23.130130+0000","last_change":"2024-04-23T16:51:21.063679+0000","last_active":"2024-04-23T16:51:23.130130+0000","last_peered":"2024-04-23T16:51:23.130130+0000","last_clean":"2024-04-23T16:51:23.130130+0000","last_became_active":"2024-04-23T16:51:21.063420+0000","last_became_peered":"2024-04-23T16:51:21.063420+0000","last_unstale":"2024-04-23T16:51:23.130130+0000","last_undegraded":"2024-04-23T16:51:23.130130+0000","last_fullsized":"2024-04-23T16:51:23.130130+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:32.658159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1737,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-23T16:51:25.456766+0000","last_change":"2024-04-23T16:50:05.743103+0000","last_active":"2024-04-23T16:51:25.456766+0000","last_peered":"2024-04-23T16:51:25.456766+0000","last_clean":"2024-04-23T16:51:25.456766+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:51:25.456766+0000","last_undegraded":"2024-04-23T16:51:25.456766+0000","last_fullsized":"2024-04-23T16:51:25.456766+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:50:05.742975+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:54.636376+0000","last_clean_scrub_stamp":"2024-04-23T16:50:05.742975+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:51:29.665460+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":1862848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057073,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":961036,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":92784116,"statfs":{"total":95995035648,"available":95010934784,"internally_reserved":0,"allocated":1114112,"data_stored":712847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":982972868},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351416,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613053,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":681688,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93063464,"statfs":{"total":95995035648,"available":95296987136,"internally_reserved":0,"allocated":798720,"data_stored":391388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":697235904},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841985,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":451724,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":93293428,"statfs":{"total":95995035648,"available":95532470272,"internally_reserved":0,"allocated":1306624,"data_stored":907547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169032,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":645592,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93099560,"statfs":{"total":95995035648,"available":95333949440,"internally_reserved":0,"allocated":798720,"data_stored":391388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":660273610},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":533144,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93212008,"statfs":{"total":95995035648,"available":95449096192,"internally_reserved":0,"allocated":798720,"data_stored":391388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":545126854},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137513,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":366044,"kb_used_data":780,"kb_used_omap":231,"kb_used_meta":365016,"kb_avail":93379108,"statfs":{"total":95995035648,"available":95620206592,"internally_reserved":0,"allocated":798720,"data_stored":391388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":373777296},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738858,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":717988,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":93027164,"statfs":{"total":95995035648,"available":95259815936,"internally_reserved":0,"allocated":1003520,"data_stored":592636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269528,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2658584756279,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757711,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:51:30.670 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:51:30.955 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:51:30.991 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:51:30.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:30.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:51:31.065 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 2 16 9216 9200 287.453 276.812 0.00247494 0.00347401 2024-04-23T16:51:31.066 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:51:31.096 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:51:31.149 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:31.324 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:51:31.367 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:51:31.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:31.370 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:51:31.471 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:51:31.513 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:31.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:31 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:51:31.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:31 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:51:31.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:31 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:51:31.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:31 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:51:31.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:31 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:51:31.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:31 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:51:31.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:31 smithi103 ceph-mon[39629]: osdmap e708: 12 total, 11 up, 7 in 2024-04-23T16:51:31.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:31 smithi103 ceph-mon[39629]: pgmap v1966: 27 pgs: 27 active+clean; 138 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 17 MiB/s wr, 279 op/s 2024-04-23T16:51:31.649 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:51:31.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:31.654 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:51:31.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:31 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:51:31.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:31 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:51:31.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:31 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:51:31.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:31 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:51:31.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:31 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:51:31.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:31 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:51:31.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:31 smithi188 ceph-mon[44101]: osdmap e708: 12 total, 11 up, 7 in 2024-04-23T16:51:31.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:31 smithi188 ceph-mon[44101]: pgmap v1966: 27 pgs: 27 active+clean; 138 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 17 MiB/s wr, 279 op/s 2024-04-23T16:51:31.755 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:51:31.794 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:31.824 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:51:31.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:31.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:51:31.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:31 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:51:31.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:31 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:51:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:31 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:51:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:31 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:51:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:31 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:51:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:31 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:51:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:31 smithi148 ceph-mon[44520]: osdmap e708: 12 total, 11 up, 7 in 2024-04-23T16:51:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:31 smithi148 ceph-mon[44520]: pgmap v1966: 27 pgs: 27 active+clean; 138 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 17 MiB/s wr, 279 op/s 2024-04-23T16:51:31.929 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:51:31.959 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:31.981 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:51:31.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:31.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:51:32.065 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 3 16 13177 13161 274.15 247.562 0.00267341 0.00364441 2024-04-23T16:51:32.085 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:51:32.122 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:32.227 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:51:32.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:32.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:51:32.333 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:51:32.363 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:32.389 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:51:32.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:32.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:51:32.494 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:51:32.523 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:32.532 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:51:32.545 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:51:32.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:32.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:51:32.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:32 smithi148 ceph-mon[44520]: 1.0 deep-scrub starts 2024-04-23T16:51:32.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:32 smithi148 ceph-mon[44520]: 1.0 deep-scrub ok 2024-04-23T16:51:32.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:32 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1431098896' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:51:32.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:32 smithi148 ceph-mon[44520]: from='client.36944 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:51:32.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:32 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:51:32.649 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:51:32.683 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:32.705 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:51:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:32.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:51:32.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:32 smithi188 ceph-mon[44101]: 1.0 deep-scrub starts 2024-04-23T16:51:32.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:32 smithi188 ceph-mon[44101]: 1.0 deep-scrub ok 2024-04-23T16:51:32.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:32 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1431098896' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:51:32.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:32 smithi188 ceph-mon[44101]: from='client.36944 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:51:32.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:32 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:51:32.809 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:51:32.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:32 smithi103 ceph-mon[39629]: 1.0 deep-scrub starts 2024-04-23T16:51:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:32 smithi103 ceph-mon[39629]: 1.0 deep-scrub ok 2024-04-23T16:51:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:32 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1431098896' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:51:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:32 smithi103 ceph-mon[39629]: from='client.36944 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:51:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:51:32.859 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:32.887 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:51:32.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:32.893 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:51:32.994 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:33.040 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:33.065 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 4 16 16486 16470 257.312 206.812 0.00455268 0.00388082 2024-04-23T16:51:33.179 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:51:33.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:33.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:51:33.262 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:51:33.262 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:51:33.286 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:51:33.316 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:33.338 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:51:33.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:33.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:51:33.442 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:51:33.472 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:33.495 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:51:33.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:33.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:51:33.599 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:51:33.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:33 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:51:33.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:33 smithi148 ceph-mon[44520]: from='client.36944 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:51:33.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:33 smithi148 ceph-mon[44520]: osdmap e709: 12 total, 11 up, 7 in 2024-04-23T16:51:33.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:33 smithi148 ceph-mon[44520]: pgmap v1968: 27 pgs: 27 active+clean; 430 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 57 MiB/s wr, 916 op/s 2024-04-23T16:51:33.628 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:33.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:33 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:51:33.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:33 smithi188 ceph-mon[44101]: from='client.36944 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:51:33.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:33 smithi188 ceph-mon[44101]: osdmap e709: 12 total, 11 up, 7 in 2024-04-23T16:51:33.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:33 smithi188 ceph-mon[44101]: pgmap v1968: 27 pgs: 27 active+clean; 430 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 57 MiB/s wr, 916 op/s 2024-04-23T16:51:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:33 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:51:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:33 smithi103 ceph-mon[39629]: from='client.36944 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:51:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:33 smithi103 ceph-mon[39629]: osdmap e709: 12 total, 11 up, 7 in 2024-04-23T16:51:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:33 smithi103 ceph-mon[39629]: pgmap v1968: 27 pgs: 27 active+clean; 430 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 57 MiB/s wr, 916 op/s 2024-04-23T16:51:33.978 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-23T16:51:34.065 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 5 16 19746 19730 246.596 203.75 0.00686125 0.00405218 2024-04-23T16:51:34.371 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:51:34.376 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:51:34.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:34.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:51:34.480 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:34.532 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:34.600 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:51:34.644 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:51:34.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:34.648 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:51:34.749 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:51:34.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:34 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:51:34.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:34 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:51:34.801 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:34.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:34 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:51:34.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:34 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:51:34.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:34 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:51:34.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:34 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:51:34.889 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:51:34.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:34.893 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:51:34.994 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:51:35.033 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:35.065 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 6 16 22780 22764 237.098 189.625 0.00324959 0.00421507 2024-04-23T16:51:35.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:35 smithi188 ceph-mon[44101]: osdmap e710: 12 total, 11 up, 7 in 2024-04-23T16:51:35.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:35 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:51:35.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:35 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:51:35.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:35 smithi188 ceph-mon[44101]: pgmap v1970: 27 pgs: 27 active+clean; 840 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-04-23T16:51:35.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:35 smithi103 ceph-mon[39629]: osdmap e710: 12 total, 11 up, 7 in 2024-04-23T16:51:35.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:35 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:51:35.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:35 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:51:35.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:35 smithi103 ceph-mon[39629]: pgmap v1970: 27 pgs: 27 active+clean; 840 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-04-23T16:51:35.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:35 smithi148 ceph-mon[44520]: osdmap e710: 12 total, 11 up, 7 in 2024-04-23T16:51:35.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:35 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:51:35.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:35 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:51:35.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:35 smithi148 ceph-mon[44520]: pgmap v1970: 27 pgs: 27 active+clean; 840 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-04-23T16:51:36.066 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 7 16 26140 26124 233.202 210 0.00538361 0.00428613 2024-04-23T16:51:36.484 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:51:36.484 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:51:36.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:36 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:51:36.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:36 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:51:36.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:36 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:51:36.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:36 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:51:36.893 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:36 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:51:36.893 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:36 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:51:36.937 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:51:36.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:36.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:51:37.042 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:51:37.067 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 8 16 29230 29214 228.161 193.125 0.00937877 0.00437888 2024-04-23T16:51:37.090 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:37.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:36 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:51:37.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:36 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:51:37.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:36 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:51:37.244 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:51:37.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:37.252 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:51:37.287 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-23T16:51:37.352 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:51:37.396 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:37.739 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:51:37.854 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:51:37.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:37.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:51:37.962 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:51:38.000 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:37 smithi103 ceph-mon[39629]: pgmap v1971: 27 pgs: 27 active+clean; 1.3 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 205 MiB/s wr, 3.29k op/s 2024-04-23T16:51:38.003 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:38.012 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:37 smithi188 ceph-mon[44101]: pgmap v1971: 27 pgs: 27 active+clean; 1.3 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 205 MiB/s wr, 3.29k op/s 2024-04-23T16:51:38.025 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:51:38.056 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:51:38.067 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 9 16 32027 32011 222.233 174.812 0.00872659 0.00449701 2024-04-23T16:51:38.080 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:51:38.109 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:51:38.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:38.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:51:38.117 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:51:38.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:37 smithi148 ceph-mon[44520]: pgmap v1971: 27 pgs: 27 active+clean; 1.3 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 205 MiB/s wr, 3.29k op/s 2024-04-23T16:51:38.147 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:51:38.172 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:51:38.199 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:51:38.214 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:38.219 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:51:38.234 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:51:38.258 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:38.279 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:51:38.290 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:51:38.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:38.294 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:51:38.311 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:51:38.346 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:51:38.346 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:51:38.346 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:51:38.347 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:51:38.347 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:51:38.347 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-23T16:51:38.347 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:51:38.347 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:51:38.347 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:51:38.347 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:51:38.347 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:51:38.347 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:51:38.347 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:51:38.395 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:51:38.445 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:38.732 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:51:38.742 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:51:39.067 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 10 16 34788 34772 217.265 172.562 0.00255074 0.00459979 2024-04-23T16:51:39.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:38 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2053097979' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:39.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:39.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:39.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:39.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:39.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:39.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:39.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:38 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2053097979' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:39.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:39.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:38 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2053097979' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:39.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:39.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:39.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:39.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:39.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:39.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:39.921 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:51:39.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:39.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:51:40.025 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:51:40.042 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:40.042 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:40.042 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:40.042 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:40.042 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:40.042 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:39 smithi188 ceph-mon[44101]: from='client.17511 v1:172.21.15.103:0/1715350325' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:51:40.042 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:39 smithi188 ceph-mon[44101]: pgmap v1972: 27 pgs: 27 active+clean; 1.6 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 192 MiB/s wr, 3.07k op/s 2024-04-23T16:51:40.042 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:39 smithi188 ceph-mon[44101]: osdmap e711: 12 total, 11 up, 7 in 2024-04-23T16:51:40.067 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 11 15 37416 37401 212.451 164.312 0.0078136 0.00470541 2024-04-23T16:51:40.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:40.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:40.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:40.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:40.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:40.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:39 smithi103 ceph-mon[39629]: from='client.17511 v1:172.21.15.103:0/1715350325' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:51:40.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:39 smithi103 ceph-mon[39629]: pgmap v1972: 27 pgs: 27 active+clean; 1.6 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 192 MiB/s wr, 3.07k op/s 2024-04-23T16:51:40.071 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:39 smithi103 ceph-mon[39629]: osdmap e711: 12 total, 11 up, 7 in 2024-04-23T16:51:40.073 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:40.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:40.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:40.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:40.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:40.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:40.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:39 smithi148 ceph-mon[44520]: from='client.17511 v1:172.21.15.103:0/1715350325' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:51:40.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:39 smithi148 ceph-mon[44520]: pgmap v1972: 27 pgs: 27 active+clean; 1.6 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 192 MiB/s wr, 3.07k op/s 2024-04-23T16:51:40.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:39 smithi148 ceph-mon[44520]: osdmap e711: 12 total, 11 up, 7 in 2024-04-23T16:51:40.226 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:51:40.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:40.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:51:40.332 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:51:40.370 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:40.810 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1971,"stamp":"2024-04-23T16:51:36.950774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1437794848,"num_objects":21932,"num_object_clones":0,"num_object_copies":43864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21930,"num_write_kb":1403520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23459,"ondisk_log_size":23459,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7050244,"kb_used_data":2624568,"kb_used_omap":240,"kb_used_meta":4425359,"kb_avail":649165820,"statfs":{"total":671965249536,"available":664745799680,"internally_reserved":0,"allocated":2687557632,"data_stored":2684563192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":246226,"internal_metadata":4531568174},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1399848960,"num_objects":21360,"num_object_clones":0,"num_object_copies":42720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21360,"num_write_kb":1367040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.498801"},"pg_stats":[{"pgid":"9.7","version":"710'1654","reported_seq":1674,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-04-23T16:51:35.884625+0000","last_change":"2024-04-23T16:51:21.064703+0000","last_active":"2024-04-23T16:51:35.884625+0000","last_peered":"2024-04-23T16:51:35.884625+0000","last_clean":"2024-04-23T16:51:35.884625+0000","last_became_active":"2024-04-23T16:51:21.064432+0000","last_became_peered":"2024-04-23T16:51:21.064432+0000","last_unstale":"2024-04-23T16:51:35.884625+0000","last_undegraded":"2024-04-23T16:51:35.884625+0000","last_fullsized":"2024-04-23T16:51:35.884625+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":1654,"log_dups_size":0,"ondisk_log_size":1654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:41.570514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108396544,"num_objects":1654,"num_object_clones":0,"num_object_copies":3308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1654,"num_write_kb":105856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"710'1646","reported_seq":1669,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-04-23T16:51:36.049316+0000","last_change":"2024-04-23T16:51:21.063927+0000","last_active":"2024-04-23T16:51:36.049316+0000","last_peered":"2024-04-23T16:51:36.049316+0000","last_clean":"2024-04-23T16:51:36.049316+0000","last_became_active":"2024-04-23T16:51:21.063642+0000","last_became_peered":"2024-04-23T16:51:21.063642+0000","last_unstale":"2024-04-23T16:51:36.049316+0000","last_undegraded":"2024-04-23T16:51:36.049316+0000","last_fullsized":"2024-04-23T16:51:36.049316+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":1646,"log_dups_size":0,"ondisk_log_size":1646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:49.079291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107872256,"num_objects":1646,"num_object_clones":0,"num_object_copies":3292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1646,"num_write_kb":105344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"709'921","reported_seq":942,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-04-23T16:51:32.528572+0000","last_change":"2024-04-23T16:51:21.064490+0000","last_active":"2024-04-23T16:51:32.528572+0000","last_peered":"2024-04-23T16:51:32.528572+0000","last_clean":"2024-04-23T16:51:32.528572+0000","last_became_active":"2024-04-23T16:51:21.063776+0000","last_became_peered":"2024-04-23T16:51:21.063776+0000","last_unstale":"2024-04-23T16:51:32.528572+0000","last_undegraded":"2024-04-23T16:51:32.528572+0000","last_fullsized":"2024-04-23T16:51:32.528572+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":921,"log_dups_size":0,"ondisk_log_size":921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:21.532298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60358656,"num_objects":921,"num_object_clones":0,"num_object_copies":1842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":921,"num_write_kb":58944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"710'1613","reported_seq":1636,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-04-23T16:51:36.043743+0000","last_change":"2024-04-23T16:51:21.063360+0000","last_active":"2024-04-23T16:51:36.043743+0000","last_peered":"2024-04-23T16:51:36.043743+0000","last_clean":"2024-04-23T16:51:36.043743+0000","last_became_active":"2024-04-23T16:51:21.063114+0000","last_became_peered":"2024-04-23T16:51:21.063114+0000","last_unstale":"2024-04-23T16:51:36.043743+0000","last_undegraded":"2024-04-23T16:51:36.043743+0000","last_fullsized":"2024-04-23T16:51:36.043743+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":1613,"log_dups_size":0,"ondisk_log_size":1613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:42.055107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105709568,"num_objects":1613,"num_object_clones":0,"num_object_copies":3226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1613,"num_write_kb":103232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"709'1019","reported_seq":1039,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-04-23T16:51:33.054627+0000","last_change":"2024-04-23T16:51:21.063893+0000","last_active":"2024-04-23T16:51:33.054627+0000","last_peered":"2024-04-23T16:51:33.054627+0000","last_clean":"2024-04-23T16:51:33.054627+0000","last_became_active":"2024-04-23T16:51:21.063652+0000","last_became_peered":"2024-04-23T16:51:21.063652+0000","last_unstale":"2024-04-23T16:51:33.054627+0000","last_undegraded":"2024-04-23T16:51:33.054627+0000","last_fullsized":"2024-04-23T16:51:33.054627+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":1019,"log_dups_size":0,"ondisk_log_size":1019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:48.971239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66781184,"num_objects":1019,"num_object_clones":0,"num_object_copies":2038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1019,"num_write_kb":65216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.2","version":"709'1056","reported_seq":1076,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-04-23T16:51:33.059532+0000","last_change":"2024-04-23T16:51:21.062805+0000","last_active":"2024-04-23T16:51:33.059532+0000","last_peered":"2024-04-23T16:51:33.059532+0000","last_clean":"2024-04-23T16:51:33.059532+0000","last_became_active":"2024-04-23T16:51:21.062561+0000","last_became_peered":"2024-04-23T16:51:21.062561+0000","last_unstale":"2024-04-23T16:51:33.059532+0000","last_undegraded":"2024-04-23T16:51:33.059532+0000","last_fullsized":"2024-04-23T16:51:33.059532+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":1056,"log_dups_size":0,"ondisk_log_size":1056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:32.737026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69206016,"num_objects":1056,"num_object_clones":0,"num_object_copies":2112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1056,"num_write_kb":67584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1759,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-04-23T16:51:34.477081+0000","last_change":"2024-04-23T16:51:20.224584+0000","last_active":"2024-04-23T16:51:34.477081+0000","last_peered":"2024-04-23T16:51:34.477081+0000","last_clean":"2024-04-23T16:51:34.477081+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:51:34.477081+0000","last_undegraded":"2024-04-23T16:51:34.477081+0000","last_fullsized":"2024-04-23T16:51:34.477081+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:51:20.224480+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:10.222378+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.224480+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:47.325219+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"710'1641","reported_seq":1664,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-04-23T16:51:36.061812+0000","last_change":"2024-04-23T16:51:21.063626+0000","last_active":"2024-04-23T16:51:36.061812+0000","last_peered":"2024-04-23T16:51:36.061812+0000","last_clean":"2024-04-23T16:51:36.061812+0000","last_became_active":"2024-04-23T16:51:21.063395+0000","last_became_peered":"2024-04-23T16:51:21.063395+0000","last_unstale":"2024-04-23T16:51:36.061812+0000","last_undegraded":"2024-04-23T16:51:36.061812+0000","last_fullsized":"2024-04-23T16:51:36.061812+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":1641,"log_dups_size":0,"ondisk_log_size":1641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:45.074171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107544576,"num_objects":1641,"num_object_clones":0,"num_object_copies":3282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1641,"num_write_kb":105024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1638,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-04-23T16:51:32.469956+0000","last_change":"2024-04-23T16:50:53.723122+0000","last_active":"2024-04-23T16:51:32.469956+0000","last_peered":"2024-04-23T16:51:32.469956+0000","last_clean":"2024-04-23T16:51:32.469956+0000","last_became_active":"2024-04-23T16:50:51.611397+0000","last_became_peered":"2024-04-23T16:50:51.611397+0000","last_unstale":"2024-04-23T16:51:32.469956+0000","last_undegraded":"2024-04-23T16:51:32.469956+0000","last_fullsized":"2024-04-23T16:51:32.469956+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:50:53.722996+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:50:53.722996+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:02.512304+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.1","version":"710'1334","reported_seq":1357,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-04-23T16:51:34.726830+0000","last_change":"2024-04-23T16:51:21.246684+0000","last_active":"2024-04-23T16:51:34.726830+0000","last_peered":"2024-04-23T16:51:34.726830+0000","last_clean":"2024-04-23T16:51:34.726830+0000","last_became_active":"2024-04-23T16:51:21.246391+0000","last_became_peered":"2024-04-23T16:51:21.246391+0000","last_unstale":"2024-04-23T16:51:34.726830+0000","last_undegraded":"2024-04-23T16:51:34.726830+0000","last_fullsized":"2024-04-23T16:51:34.726830+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":1334,"log_dups_size":0,"ondisk_log_size":1334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:26.694086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87425024,"num_objects":1334,"num_object_clones":0,"num_object_copies":2668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1334,"num_write_kb":85376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1751,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-04-23T16:51:34.479272+0000","last_change":"2024-04-23T16:51:25.290906+0000","last_active":"2024-04-23T16:51:34.479272+0000","last_peered":"2024-04-23T16:51:34.479272+0000","last_clean":"2024-04-23T16:51:34.479272+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:51:34.479272+0000","last_undegraded":"2024-04-23T16:51:34.479272+0000","last_fullsized":"2024-04-23T16:51:34.479272+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:51:25.290774+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:51:25.290774+0000","last_clean_scrub_stamp":"2024-04-23T16:51:25.290774+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:30.207942+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"710'1617","reported_seq":1639,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-04-23T16:51:36.051048+0000","last_change":"2024-04-23T16:51:21.063902+0000","last_active":"2024-04-23T16:51:36.051048+0000","last_peered":"2024-04-23T16:51:36.051048+0000","last_clean":"2024-04-23T16:51:36.051048+0000","last_became_active":"2024-04-23T16:51:21.063669+0000","last_became_peered":"2024-04-23T16:51:21.063669+0000","last_unstale":"2024-04-23T16:51:36.051048+0000","last_undegraded":"2024-04-23T16:51:36.051048+0000","last_fullsized":"2024-04-23T16:51:36.051048+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":1617,"log_dups_size":0,"ondisk_log_size":1617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:35.088553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105971712,"num_objects":1617,"num_object_clones":0,"num_object_copies":3234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1617,"num_write_kb":103488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1766,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-04-23T16:51:34.480048+0000","last_change":"2024-04-23T16:51:30.393027+0000","last_active":"2024-04-23T16:51:34.480048+0000","last_peered":"2024-04-23T16:51:34.480048+0000","last_clean":"2024-04-23T16:51:34.480048+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:51:34.480048+0000","last_undegraded":"2024-04-23T16:51:34.480048+0000","last_fullsized":"2024-04-23T16:51:34.480048+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:51:30.392898+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:51:30.392898+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:54.377839+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"709'1068","reported_seq":1088,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-04-23T16:51:33.061703+0000","last_change":"2024-04-23T16:51:21.063044+0000","last_active":"2024-04-23T16:51:33.061703+0000","last_peered":"2024-04-23T16:51:33.061703+0000","last_clean":"2024-04-23T16:51:33.061703+0000","last_became_active":"2024-04-23T16:51:21.062760+0000","last_became_peered":"2024-04-23T16:51:21.062760+0000","last_unstale":"2024-04-23T16:51:33.061703+0000","last_undegraded":"2024-04-23T16:51:33.061703+0000","last_fullsized":"2024-04-23T16:51:33.061703+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":1068,"log_dups_size":0,"ondisk_log_size":1068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:33.943628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69992448,"num_objects":1068,"num_object_clones":0,"num_object_copies":2136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1068,"num_write_kb":68352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1349,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-04-23T16:51:34.477017+0000","last_change":"2024-04-23T16:51:27.836760+0000","last_active":"2024-04-23T16:51:34.477017+0000","last_peered":"2024-04-23T16:51:34.477017+0000","last_clean":"2024-04-23T16:51:34.477017+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:51:34.477017+0000","last_undegraded":"2024-04-23T16:51:34.477017+0000","last_fullsized":"2024-04-23T16:51:34.477017+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:51:27.836628+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:51:27.836628+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:50.428310+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"709'883","reported_seq":904,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-04-23T16:51:32.535685+0000","last_change":"2024-04-23T16:51:21.246813+0000","last_active":"2024-04-23T16:51:32.535685+0000","last_peered":"2024-04-23T16:51:32.535685+0000","last_clean":"2024-04-23T16:51:32.535685+0000","last_became_active":"2024-04-23T16:51:21.246519+0000","last_became_peered":"2024-04-23T16:51:21.246519+0000","last_unstale":"2024-04-23T16:51:32.535685+0000","last_undegraded":"2024-04-23T16:51:32.535685+0000","last_fullsized":"2024-04-23T16:51:32.535685+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":883,"log_dups_size":0,"ondisk_log_size":883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:38.940744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57868288,"num_objects":883,"num_object_clones":0,"num_object_copies":1766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":883,"num_write_kb":56512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1300,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-04-23T16:51:32.469909+0000","last_change":"2024-04-23T16:51:23.676325+0000","last_active":"2024-04-23T16:51:32.469909+0000","last_peered":"2024-04-23T16:51:32.469909+0000","last_clean":"2024-04-23T16:51:32.469909+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:51:32.469909+0000","last_undegraded":"2024-04-23T16:51:32.469909+0000","last_fullsized":"2024-04-23T16:51:32.469909+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:51:23.676218+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:51:23.676218+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:25.153352+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.e","version":"710'1616","reported_seq":1639,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-04-23T16:51:35.884659+0000","last_change":"2024-04-23T16:51:21.064696+0000","last_active":"2024-04-23T16:51:35.884659+0000","last_peered":"2024-04-23T16:51:35.884659+0000","last_clean":"2024-04-23T16:51:35.884659+0000","last_became_active":"2024-04-23T16:51:21.064486+0000","last_became_peered":"2024-04-23T16:51:21.064486+0000","last_unstale":"2024-04-23T16:51:35.884659+0000","last_undegraded":"2024-04-23T16:51:35.884659+0000","last_fullsized":"2024-04-23T16:51:35.884659+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":1616,"log_dups_size":0,"ondisk_log_size":1616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:35.126146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105906176,"num_objects":1616,"num_object_clones":0,"num_object_copies":3232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1616,"num_write_kb":103424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1767,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-04-23T16:51:34.477239+0000","last_change":"2024-04-23T16:51:18.204631+0000","last_active":"2024-04-23T16:51:34.477239+0000","last_peered":"2024-04-23T16:51:34.477239+0000","last_clean":"2024-04-23T16:51:34.477239+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:51:34.477239+0000","last_undegraded":"2024-04-23T16:51:34.477239+0000","last_fullsized":"2024-04-23T16:51:34.477239+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:51:18.204524+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:51:18.204524+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:24.446712+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.f","version":"710'1671","reported_seq":1693,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-04-23T16:51:36.061297+0000","last_change":"2024-04-23T16:51:21.063851+0000","last_active":"2024-04-23T16:51:36.061297+0000","last_peered":"2024-04-23T16:51:36.061297+0000","last_clean":"2024-04-23T16:51:36.061297+0000","last_became_active":"2024-04-23T16:51:21.063588+0000","last_became_peered":"2024-04-23T16:51:21.063588+0000","last_unstale":"2024-04-23T16:51:36.061297+0000","last_undegraded":"2024-04-23T16:51:36.061297+0000","last_fullsized":"2024-04-23T16:51:36.061297+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":1671,"log_dups_size":0,"ondisk_log_size":1671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:32.557471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109510656,"num_objects":1671,"num_object_clones":0,"num_object_copies":3342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1671,"num_write_kb":106944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1765,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-04-23T16:51:34.480310+0000","last_change":"2024-04-23T16:50:10.300939+0000","last_active":"2024-04-23T16:51:34.480310+0000","last_peered":"2024-04-23T16:51:34.480310+0000","last_clean":"2024-04-23T16:51:34.480310+0000","last_became_active":"2024-04-23T16:47:10.266014+0000","last_became_peered":"2024-04-23T16:47:10.266014+0000","last_unstale":"2024-04-23T16:51:34.480310+0000","last_undegraded":"2024-04-23T16:51:34.480310+0000","last_fullsized":"2024-04-23T16:51:34.480310+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:50:10.300793+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:50:10.300793+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:51:38.266525+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"710'1323","reported_seq":1346,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-04-23T16:51:34.728627+0000","last_change":"2024-04-23T16:51:21.080034+0000","last_active":"2024-04-23T16:51:34.728627+0000","last_peered":"2024-04-23T16:51:34.728627+0000","last_clean":"2024-04-23T16:51:34.728627+0000","last_became_active":"2024-04-23T16:51:21.079798+0000","last_became_peered":"2024-04-23T16:51:21.079798+0000","last_unstale":"2024-04-23T16:51:34.728627+0000","last_undegraded":"2024-04-23T16:51:34.728627+0000","last_fullsized":"2024-04-23T16:51:34.728627+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":1323,"log_dups_size":0,"ondisk_log_size":1323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:32.988866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86704128,"num_objects":1323,"num_object_clones":0,"num_object_copies":2646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1323,"num_write_kb":84672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1642,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-04-23T16:51:32.477208+0000","last_change":"2024-04-23T16:50:21.753563+0000","last_active":"2024-04-23T16:51:32.477208+0000","last_peered":"2024-04-23T16:51:32.477208+0000","last_clean":"2024-04-23T16:51:32.477208+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:51:32.477208+0000","last_undegraded":"2024-04-23T16:51:32.477208+0000","last_fullsized":"2024-04-23T16:51:32.477208+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:50:21.753450+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:50:21.753450+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:51:41.041517+0000","scrub_duration":277,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"710'1537","reported_seq":1560,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-04-23T16:51:35.878080+0000","last_change":"2024-04-23T16:51:21.064751+0000","last_active":"2024-04-23T16:51:35.878080+0000","last_peered":"2024-04-23T16:51:35.878080+0000","last_clean":"2024-04-23T16:51:35.878080+0000","last_became_active":"2024-04-23T16:51:21.064506+0000","last_became_peered":"2024-04-23T16:51:21.064506+0000","last_unstale":"2024-04-23T16:51:35.878080+0000","last_undegraded":"2024-04-23T16:51:35.878080+0000","last_fullsized":"2024-04-23T16:51:35.878080+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":1537,"log_dups_size":0,"ondisk_log_size":1537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:37.912311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100728832,"num_objects":1537,"num_object_clones":0,"num_object_copies":3074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1537,"num_write_kb":98368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1582,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-04-23T16:51:34.480216+0000","last_change":"2024-04-23T16:51:19.334794+0000","last_active":"2024-04-23T16:51:34.480216+0000","last_peered":"2024-04-23T16:51:34.480216+0000","last_clean":"2024-04-23T16:51:34.480216+0000","last_became_active":"2024-04-23T16:33:25.848708+0000","last_became_peered":"2024-04-23T16:33:25.848708+0000","last_unstale":"2024-04-23T16:51:34.480216+0000","last_undegraded":"2024-04-23T16:51:34.480216+0000","last_fullsized":"2024-04-23T16:51:34.480216+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:51:19.334693+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:51:19.334693+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:41.467169+0000","scrub_duration":146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"710'1331","reported_seq":1354,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-04-23T16:51:34.718209+0000","last_change":"2024-04-23T16:51:21.063679+0000","last_active":"2024-04-23T16:51:34.718209+0000","last_peered":"2024-04-23T16:51:34.718209+0000","last_clean":"2024-04-23T16:51:34.718209+0000","last_became_active":"2024-04-23T16:51:21.063420+0000","last_became_peered":"2024-04-23T16:51:21.063420+0000","last_unstale":"2024-04-23T16:51:34.718209+0000","last_undegraded":"2024-04-23T16:51:34.718209+0000","last_fullsized":"2024-04-23T16:51:34.718209+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":1331,"log_dups_size":0,"ondisk_log_size":1331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:29.481317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87228416,"num_objects":1331,"num_object_clones":0,"num_object_copies":2662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1331,"num_write_kb":85184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1751,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-04-23T16:51:34.479272+0000","last_change":"2024-04-23T16:51:31.339866+0000","last_active":"2024-04-23T16:51:34.479272+0000","last_peered":"2024-04-23T16:51:34.479272+0000","last_clean":"2024-04-23T16:51:34.479272+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:51:34.479272+0000","last_undegraded":"2024-04-23T16:51:34.479272+0000","last_fullsized":"2024-04-23T16:51:34.479272+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:51:31.339728+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:51:31.339728+0000","last_clean_scrub_stamp":"2024-04-23T16:51:31.339728+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:52:59.842945+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1437204480,"num_objects":21930,"num_object_clones":0,"num_object_copies":43860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21930,"num_write_kb":1403520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2885025792,"data_stored":2885025792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21930,"ondisk_log_size":21930,"up":32,"acting":32,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":1862848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057075,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1180208,"kb_used_data":224420,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":92564944,"statfs":{"total":95995035648,"available":94786502656,"internally_reserved":0,"allocated":229806080,"data_stored":229387080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":978713028},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351418,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613055,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1158588,"kb_used_data":473520,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":92586564,"statfs":{"total":95995035648,"available":94808641536,"internally_reserved":0,"allocated":484884480,"data_stored":484459426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":701495744},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841987,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":749764,"kb_used_data":299320,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":92995388,"statfs":{"total":95995035648,"available":95227277312,"internally_reserved":0,"allocated":306503680,"data_stored":306075074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169034,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1023304,"kb_used_data":374332,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92721848,"statfs":{"total":95995035648,"available":94947172352,"internally_reserved":0,"allocated":383315968,"data_stored":382884910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":664533450},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":725508,"kb_used_data":180668,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93019644,"statfs":{"total":95995035648,"available":95252115456,"internally_reserved":0,"allocated":185004032,"data_stored":184572974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137515,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1412936,"kb_used_data":993532,"kb_used_omap":231,"kb_used_meta":419160,"kb_avail":92332216,"statfs":{"total":95995035648,"available":94548189184,"internally_reserved":0,"allocated":1017376768,"data_stored":1016945710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":429220752},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738860,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":799936,"kb_used_data":78776,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":92945216,"statfs":{"total":95995035648,"available":95175901184,"internally_reserved":0,"allocated":80666624,"data_stored":80238018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2065879269530,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2658584756281,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757713,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":85852160,"data_stored":85852160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1080754176,"data_stored":1080754176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":191823872,"data_stored":191823872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":416874496,"data_stored":416874496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":339214336,"data_stored":339214336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":531038208,"data_stored":531038208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":239468544,"data_stored":239468544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:51:40.812 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-23T16:51:40.812 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-04-23T16:51:41.068 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 12 16 40114 40098 208.777 168.562 0.00288042 0.00477055 2024-04-23T16:51:41.224 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:51:41.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:41.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:51:41.252 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:51:41.253 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:51:41.253 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:51:41.330 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:51:41.363 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:41.569 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:51:41.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:41.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:51:41.675 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:51:41.711 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:41.770 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:51:41.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:41.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:51:41.875 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:51:41.914 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:42.068 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 13 15 42561 42546 204.487 153 0.00620234 0.00488827 2024-04-23T16:51:42.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:42 smithi188 ceph-mon[44101]: pgmap v1974: 27 pgs: 27 active+clean; 2.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 198 MiB/s wr, 3.17k op/s 2024-04-23T16:51:42.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:42 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:51:42.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:42 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:51:42.296 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:42 smithi103 ceph-mon[39629]: pgmap v1974: 27 pgs: 27 active+clean; 2.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 198 MiB/s wr, 3.17k op/s 2024-04-23T16:51:42.297 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:42 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:51:42.297 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:42 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:51:42.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:42 smithi148 ceph-mon[44520]: pgmap v1974: 27 pgs: 27 active+clean; 2.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 198 MiB/s wr, 3.17k op/s 2024-04-23T16:51:42.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:42 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:51:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:42 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:51:43.021 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:43 smithi103 ceph-mon[39629]: osdmap e712: 12 total, 11 up, 7 in 2024-04-23T16:51:43.069 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 14 16 44998 44982 200.742 152.25 0.00393539 0.00497882 2024-04-23T16:51:43.253 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:51:43.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:43 smithi148 ceph-mon[44520]: osdmap e712: 12 total, 11 up, 7 in 2024-04-23T16:51:43.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:43 smithi188 ceph-mon[44101]: osdmap e712: 12 total, 11 up, 7 in 2024-04-23T16:51:43.574 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:51:43.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:43.578 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:51:43.642 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:51:43.679 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:51:43.724 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:43.860 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:51:43.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:43.886 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:51:43.987 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:44.027 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:44.029 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:51:44.069 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 15 16 47310 47294 196.993 144.5 0.0106488 0.00507443 2024-04-23T16:51:44.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:44 smithi188 ceph-mon[44101]: pgmap v1976: 27 pgs: 27 active+clean; 2.3 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 194 MiB/s wr, 3.10k op/s 2024-04-23T16:51:44.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:44 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2951060773' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:51:44.236 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:44 smithi103 ceph-mon[39629]: pgmap v1976: 27 pgs: 27 active+clean; 2.3 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 194 MiB/s wr, 3.10k op/s 2024-04-23T16:51:44.236 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:44 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2951060773' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:51:44.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:44 smithi148 ceph-mon[44520]: pgmap v1976: 27 pgs: 27 active+clean; 2.3 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 194 MiB/s wr, 3.10k op/s 2024-04-23T16:51:44.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:44 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2951060773' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:51:45.069 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 16 16 49355 49339 192.669 127.812 0.00222689 0.00518101 2024-04-23T16:51:45.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:45 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:51:45.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:45 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2951060773' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:51:45.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:45 smithi103 ceph-mon[39629]: osdmap e713: 12 total, 11 up, 7 in 2024-04-23T16:51:45.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:45 smithi103 ceph-mon[39629]: osdmap e714: 12 total, 11 up, 7 in 2024-04-23T16:51:45.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:45 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:51:45.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:45 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2951060773' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:51:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:45 smithi148 ceph-mon[44520]: osdmap e713: 12 total, 11 up, 7 in 2024-04-23T16:51:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:45 smithi148 ceph-mon[44520]: osdmap e714: 12 total, 11 up, 7 in 2024-04-23T16:51:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:45 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:51:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:45 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2951060773' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:51:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:45 smithi188 ceph-mon[44101]: osdmap e713: 12 total, 11 up, 7 in 2024-04-23T16:51:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:45 smithi188 ceph-mon[44101]: osdmap e714: 12 total, 11 up, 7 in 2024-04-23T16:51:45.796 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:51:45.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:45.801 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:51:45.901 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:51:45.947 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:45.975 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:51:45.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:45.979 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:51:46.028 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:46 smithi188 ceph-mon[44101]: pgmap v1979: 27 pgs: 27 active+clean; 2.6 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 184 MiB/s wr, 2.95k op/s 2024-04-23T16:51:46.028 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:46 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:51:46.028 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:46 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:51:46.072 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 17 16 51071 51055 187.615 107.25 0.00422047 0.00532726 2024-04-23T16:51:46.082 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:51:46.127 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:46.158 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:51:46.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:46.165 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:51:46.266 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:51:46.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:46 smithi103 ceph-mon[39629]: pgmap v1979: 27 pgs: 27 active+clean; 2.6 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 184 MiB/s wr, 2.95k op/s 2024-04-23T16:51:46.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:46 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:51:46.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:46 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:51:46.335 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:46.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:46 smithi148 ceph-mon[44520]: pgmap v1979: 27 pgs: 27 active+clean; 2.6 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 184 MiB/s wr, 2.95k op/s 2024-04-23T16:51:46.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:46 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:51:46.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:46 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:51:46.858 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:51:46.859 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:51:46.859 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-04-23T16:51:46.859 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:51:47.072 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 18 16 53250 53234 184.757 136.188 0.00430448 0.0054106 2024-04-23T16:51:47.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:47 smithi103 ceph-mon[39629]: osdmap e715: 12 total, 11 up, 7 in 2024-04-23T16:51:47.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:47 smithi188 ceph-mon[44101]: osdmap e715: 12 total, 11 up, 7 in 2024-04-23T16:51:47.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:47 smithi148 ceph-mon[44520]: osdmap e715: 12 total, 11 up, 7 in 2024-04-23T16:51:47.442 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:51:47.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:47.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:51:47.547 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:51:47.578 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:47.599 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:51:47.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:47.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:51:47.649 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-23T16:51:47.704 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:51:47.742 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:47.947 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:51:47.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:47.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:51:48.000 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:51:48.028 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:51:48.052 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:51:48.054 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:51:48.072 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 19 16 55407 55391 182.129 134.812 0.00870359 0.00548856 2024-04-23T16:51:48.080 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:51:48.087 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:51:48.092 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:48.106 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:51:48.125 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:51:48.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:48.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:51:48.131 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:51:48.157 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:51:48.188 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:51:48.218 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:51:48.228 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:51:48.251 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:51:48.265 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:48 smithi103 ceph-mon[39629]: pgmap v1981: 27 pgs: 27 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-04-23T16:51:48.265 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:48 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2149698217' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:48.286 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:51:48.286 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:51:48.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:51:48.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:51:48.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:51:48.287 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-23T16:51:48.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:51:48.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:51:48.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:51:48.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:51:48.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:51:48.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:51:48.287 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:51:48.301 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:51:48.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:48.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:51:48.365 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:48 smithi188 ceph-mon[44101]: pgmap v1981: 27 pgs: 27 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-04-23T16:51:48.365 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:48 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2149698217' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:48.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:48 smithi148 ceph-mon[44520]: pgmap v1981: 27 pgs: 27 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-04-23T16:51:48.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:48 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2149698217' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:48.405 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:51:48.436 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:48.537 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:51:48.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:48.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:51:48.644 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:51:48.696 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:49.028 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:51:49.029 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:51:49.058 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:51:49.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:49.064 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:51:49.074 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:51:49.074416 min lat: 0.0010724 max lat: 0.139989 avg lat: 0.00556493 2024-04-23T16:51:49.074 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:51:49.074 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 20 15 57528 57513 179.64 132.625 0.00731782 0.00556493 2024-04-23T16:51:49.165 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:51:49.202 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:49.228 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:51:49.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:49.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:51:49.333 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:51:49.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:49 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.370 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:49.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:49 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.394 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:51:49.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:49.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:51:49.419 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:51:49.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:49 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:49.498 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:51:49.527 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:49.549 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:51:49.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:49.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:51:49.657 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:51:49.708 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:49.786 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:51:49.810 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:51:49.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:49.814 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:51:49.915 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:51:49.948 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:50.074 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 21 16 59586 59570 177.208 128.562 0.00771444 0.00564111 2024-04-23T16:51:50.081 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:51:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:50.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:51:50.186 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:51:50.189 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:51:50.189 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:51:50.218 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:50.218 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:50 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:50.219 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:50 smithi188 ceph-mon[44101]: pgmap v1982: 27 pgs: 27 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-04-23T16:51:50.219 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:50 smithi188 ceph-mon[44101]: osdmap e716: 12 total, 11 up, 7 in 2024-04-23T16:51:50.254 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:50 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:50 smithi103 ceph-mon[39629]: pgmap v1982: 27 pgs: 27 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-04-23T16:51:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:50 smithi103 ceph-mon[39629]: osdmap e716: 12 total, 11 up, 7 in 2024-04-23T16:51:50.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:50.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:50 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:50.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:50 smithi148 ceph-mon[44520]: pgmap v1982: 27 pgs: 27 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-04-23T16:51:50.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:50 smithi148 ceph-mon[44520]: osdmap e716: 12 total, 11 up, 7 in 2024-04-23T16:51:50.461 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:51:50.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:50.466 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:51:50.567 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:50.616 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:50.648 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:51:50.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:50.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:51:50.755 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:51:50.808 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:51.022 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:51:51.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:51.029 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:51:51.030 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:51:51.037 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-23T16:51:51.074 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 22 16 61609 61593 174.901 126.438 0.00783167 0.00571648 2024-04-23T16:51:51.129 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:51:51.175 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:51.210 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:51:51.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:51.217 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:51:51.318 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:51:51.347 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:51.366 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:51:51.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:51.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:51:51.471 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:51:51.481 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:51:51.499 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:51.520 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:51:51.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:51.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:51:51.625 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:51:51.665 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:51.773 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:51:51.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:51.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:51:51.878 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:51:51.880 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:51:51.921 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:52.053 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:51:52.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:52.057 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:51:52.074 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 23 16 63439 63423 172.27 114.375 0.00443745 0.00580335 2024-04-23T16:51:52.157 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:51:52.207 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:52.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:52 smithi188 ceph-mon[44101]: pgmap v1984: 27 pgs: 27 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-04-23T16:51:52.234 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:51:52.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:52.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:51:52.341 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:52.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:52 smithi103 ceph-mon[39629]: pgmap v1984: 27 pgs: 27 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-04-23T16:51:52.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:52 smithi148 ceph-mon[44520]: pgmap v1984: 27 pgs: 27 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-04-23T16:51:52.383 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:52.413 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:51:52.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:52.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:51:52.518 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:52.561 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:52.927 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:51:52.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:52.931 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:51:53.033 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:51:53.074 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 24 16 65372 65356 170.126 120.812 0.00788858 0.00587651 2024-04-23T16:51:53.075 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:51:53.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:53 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/552297258' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:51:53.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:53 smithi103 ceph-mon[39629]: from='client.36971 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:51:53.080 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:53.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:53 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/552297258' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:51:53.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:53 smithi188 ceph-mon[44101]: from='client.36971 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:51:53.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:53 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/552297258' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:51:53.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:53 smithi148 ceph-mon[44520]: from='client.36971 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:51:53.673 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:51:53.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:53.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:51:53.778 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:51:53.806 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:53.829 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:51:53.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:53.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:51:53.840 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:51:53.841 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:51:53.933 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:53.976 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:54.006 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:51:54.034 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:51:54.061 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:51:54.064 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:54 smithi148 ceph-mon[44520]: pgmap v1985: 27 pgs: 27 active+clean; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 134 MiB/s wr, 2.15k op/s 2024-04-23T16:51:54.064 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:54 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:51:54.064 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:54 smithi148 ceph-mon[44520]: from='client.36971 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:51:54.064 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:54 smithi148 ceph-mon[44520]: osdmap e717: 12 total, 11 up, 7 in 2024-04-23T16:51:54.064 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:54 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:51:54.064 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:54 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:51:54.064 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:54 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1865643368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:54.074 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 25 16 67236 67220 167.981 116.5 0.00717396 0.00595094 2024-04-23T16:51:54.093 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:51:54.096 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:51:54.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:54.101 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:51:54.121 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:51:54.147 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:51:54.173 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:51:54.200 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:51:54.201 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:51:54.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:54 smithi188 ceph-mon[44101]: pgmap v1985: 27 pgs: 27 active+clean; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 134 MiB/s wr, 2.15k op/s 2024-04-23T16:51:54.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:54 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:51:54.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:54 smithi188 ceph-mon[44101]: from='client.36971 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:51:54.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:54 smithi188 ceph-mon[44101]: osdmap e717: 12 total, 11 up, 7 in 2024-04-23T16:51:54.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:54 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:51:54.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:54 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:51:54.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:54 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1865643368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:54.227 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:51:54.230 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:54.252 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:51:54.253 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:51:54.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:54.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:51:54.283 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:51:54.284 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:51:54.284 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:51:54.284 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:51:54.284 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:51:54.284 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-23T16:51:54.284 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:51:54.284 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:51:54.284 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:51:54.284 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:51:54.284 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:51:54.284 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:51:54.284 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:51:54.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:54 smithi103 ceph-mon[39629]: pgmap v1985: 27 pgs: 27 active+clean; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 134 MiB/s wr, 2.15k op/s 2024-04-23T16:51:54.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:54 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:51:54.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:54 smithi103 ceph-mon[39629]: from='client.36971 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:51:54.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:54 smithi103 ceph-mon[39629]: osdmap e717: 12 total, 11 up, 7 in 2024-04-23T16:51:54.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:54 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:51:54.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:54 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:51:54.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:54 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1865643368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:51:54.356 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:54.405 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:54.601 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:51:54.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:54.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:51:54.639 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-23T16:51:54.718 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:54.767 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:54.805 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:51:54.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:54.811 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:51:54.912 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:51:54.955 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:54.987 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:51:54.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:54.992 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:51:55.074 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 26 16 68980 68964 165.713 109 0.0107418 0.00603165 2024-04-23T16:51:55.091 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:51:55.091 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:51:55.093 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:51:55.113 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:51:55.121 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:55.141 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:51:55.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:55.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:51:55.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:55 smithi188 ceph-mon[44101]: osdmap e718: 12 total, 11 up, 7 in 2024-04-23T16:51:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.245 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:51:55.301 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:55.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:55 smithi103 ceph-mon[39629]: osdmap e718: 12 total, 11 up, 7 in 2024-04-23T16:51:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:55 smithi148 ceph-mon[44520]: osdmap e718: 12 total, 11 up, 7 in 2024-04-23T16:51:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:55.416 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:51:55.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:55.422 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:51:55.523 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:55.580 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:51:56.075 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 27 16 70819 70803 163.833 114.938 0.00579101 0.0061019 2024-04-23T16:51:56.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:56 smithi103 ceph-mon[39629]: pgmap v1988: 27 pgs: 27 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 125 MiB/s wr, 2.01k op/s 2024-04-23T16:51:56.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:56 smithi148 ceph-mon[44520]: pgmap v1988: 27 pgs: 27 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 125 MiB/s wr, 2.01k op/s 2024-04-23T16:51:56.449 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:51:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:56.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:51:56.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:56 smithi188 ceph-mon[44101]: pgmap v1988: 27 pgs: 27 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 125 MiB/s wr, 2.01k op/s 2024-04-23T16:51:56.795 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:51:57.075 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 28 16 72595 72579 161.946 111 0.00254131 0.00616882 2024-04-23T16:51:57.093 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:51:58.076 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 29 16 74384 74368 160.212 111.812 0.00513611 0.00623985 2024-04-23T16:51:58.296 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:58 smithi103 ceph-mon[39629]: pgmap v1989: 27 pgs: 27 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-04-23T16:51:58.296 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:58 smithi103 ceph-mon[39629]: 1.7 scrub starts 2024-04-23T16:51:58.297 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:58 smithi103 ceph-mon[39629]: 1.7 scrub ok 2024-04-23T16:51:58.348 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:51:58.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:58 smithi148 ceph-mon[44520]: pgmap v1989: 27 pgs: 27 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-04-23T16:51:58.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:58 smithi148 ceph-mon[44520]: 1.7 scrub starts 2024-04-23T16:51:58.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:58 smithi148 ceph-mon[44520]: 1.7 scrub ok 2024-04-23T16:51:58.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:58 smithi188 ceph-mon[44101]: pgmap v1989: 27 pgs: 27 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-04-23T16:51:58.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:58 smithi188 ceph-mon[44101]: 1.7 scrub starts 2024-04-23T16:51:58.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:58 smithi188 ceph-mon[44101]: 1.7 scrub ok 2024-04-23T16:51:59.078 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 30 16 75849 75833 157.914 91.5625 0.00418455 0.00633087 2024-04-23T16:51:59.344 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:51:59.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:51:59.348 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:51:59.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:59 smithi188 ceph-mon[44101]: 1.4 scrub starts 2024-04-23T16:51:59.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:51:59 smithi188 ceph-mon[44101]: 1.4 scrub ok 2024-04-23T16:51:59.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:59 smithi103 ceph-mon[39629]: 1.4 scrub starts 2024-04-23T16:51:59.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:51:59 smithi103 ceph-mon[39629]: 1.4 scrub ok 2024-04-23T16:51:59.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:59 smithi148 ceph-mon[44520]: 1.4 scrub starts 2024-04-23T16:51:59.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:51:59 smithi148 ceph-mon[44520]: 1.4 scrub ok 2024-04-23T16:51:59.449 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:51:59.495 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:00.078 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 31 16 77282 77266 155.71 89.5625 0.00345206 0.00641754 2024-04-23T16:52:00.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:00 smithi103 ceph-mon[39629]: pgmap v1990: 27 pgs: 27 active+clean; 4.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-04-23T16:52:00.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:00 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3171315228' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:52:00.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:00 smithi148 ceph-mon[44520]: pgmap v1990: 27 pgs: 27 active+clean; 4.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-04-23T16:52:00.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:00 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3171315228' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:52:00.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:00 smithi188 ceph-mon[44101]: pgmap v1990: 27 pgs: 27 active+clean; 4.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-04-23T16:52:00.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:00 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3171315228' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:52:00.468 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:52:00.772 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:52:00.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:00.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:52:00.879 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:52:00.917 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:01.078 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 32 16 78830 78814 153.868 96.75 0.00672521 0.00649665 2024-04-23T16:52:01.100 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:01 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:52:01.100 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:01 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:52:01.100 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:01 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3171315228' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:52:01.100 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:01 smithi188 ceph-mon[44101]: osdmap e719: 12 total, 11 up, 7 in 2024-04-23T16:52:01.127 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:52:01.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:01.135 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:52:01.236 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:52:01.282 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:01.344 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:52:01.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:01.348 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:52:01.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:01 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:52:01.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:01 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:52:01.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:01 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3171315228' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:52:01.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:01 smithi103 ceph-mon[39629]: osdmap e719: 12 total, 11 up, 7 in 2024-04-23T16:52:01.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:01 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:52:01.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:01 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:52:01.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:01 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3171315228' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:52:01.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:01 smithi148 ceph-mon[44520]: osdmap e719: 12 total, 11 up, 7 in 2024-04-23T16:52:01.448 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:52:01.486 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:02.078 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 33 16 80391 80375 152.162 97.5625 0.00932704 0.00657035 2024-04-23T16:52:02.150 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:52:02.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:02.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:52:02.255 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:52:02.314 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:02.327 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:52:02.327 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:52:02.327 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:52:02.327 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.27202.0:80683 9.0 e719/703 9:0637efd4:::benchmark_data_smithi188_167360_object80682:head v 719'5094, mlcod=719'5091)", 2024-04-23T16:52:02.327 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:52:02.306269+0000", 2024-04-23T16:52:02.327 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0085464349999999998, 2024-04-23T16:52:02.327 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.008580631, 2024-04-23T16:52:02.327 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:52:02.327 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:52:02.328 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:52:02.328 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:52:02.328 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:52:02.328 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:52:02.328 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:02.306269+0000", 2024-04-23T16:52:02.328 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:52:02.328 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:52:02.328 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:52:02.328 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:52:02.328 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:02.306269+0000", 2024-04-23T16:52:02.328 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:52:02.328 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:52:02.328 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:52:02.328 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:52:02.329 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:02.306270+0000", 2024-04-23T16:52:02.329 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7689999999999999e-06 2024-04-23T16:52:02.329 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:52:02.329 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:52:02.329 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:52:02.329 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:02.306323+0000", 2024-04-23T16:52:02.329 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2701999999999998e-05 2024-04-23T16:52:02.329 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:52:02.329 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:52:02.329 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:52:02.329 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:02.306340+0000", 2024-04-23T16:52:02.329 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7110999999999999e-05 2024-04-23T16:52:02.329 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:52:02.329 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:52:02.330 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:52:02.330 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:02.306349+0000", 2024-04-23T16:52:02.330 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5839999999999995e-06 2024-04-23T16:52:02.330 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:52:02.330 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:52:02.330 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:52:02.330 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:02.307147+0000", 2024-04-23T16:52:02.330 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00079863199999999997 2024-04-23T16:52:02.330 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:52:02.330 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:52:02.330 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:52:02.330 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:02.307169+0000", 2024-04-23T16:52:02.330 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1331999999999998e-05 2024-04-23T16:52:02.330 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:52:02.331 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:52:02.331 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:52:02.331 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:02.314790+0000", 2024-04-23T16:52:02.331 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076210430000000001 2024-04-23T16:52:02.331 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:52:02.331 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:52:02.331 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:52:02.331 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:02.314852+0000", 2024-04-23T16:52:02.331 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2099000000000003e-05 2024-04-23T16:52:02.331 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:52:02.331 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:52:02.331 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:52:02.331 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:52:02.332 INFO:teuthology.orchestra.run.smithi188.stdout: ], 2024-04-23T16:52:02.332 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 1 2024-04-23T16:52:02.332 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:52:02.343 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:52:02.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:02.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:52:02.354 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:02 smithi188 ceph-mon[44101]: pgmap v1992: 27 pgs: 27 active+clean; 4.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-04-23T16:52:02.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:02 smithi148 ceph-mon[44520]: pgmap v1992: 27 pgs: 27 active+clean; 4.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-04-23T16:52:02.450 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:52:02.487 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:02 smithi103 ceph-mon[39629]: pgmap v1992: 27 pgs: 27 active+clean; 4.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-04-23T16:52:02.490 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:02.981 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:52:02.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:02.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:52:03.079 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 34 16 81992 81976 150.625 100.062 0.00748979 0.00663705 2024-04-23T16:52:03.090 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:52:03.128 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:03.263 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:52:03.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:03.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:52:03.346 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-23T16:52:03.368 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:52:03.398 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:03.505 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:52:03.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:03.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:52:03.611 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:52:03.640 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:03.660 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:52:03.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:03.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:52:03.764 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:52:03.799 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:03.819 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:52:04.082 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 35 16 83589 83573 149.171 99.8125 0.00562244 0.00670198 2024-04-23T16:52:04.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:04 smithi148 ceph-mon[44520]: pgmap v1993: 27 pgs: 27 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-04-23T16:52:04.403 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:04 smithi188 ceph-mon[44101]: pgmap v1993: 27 pgs: 27 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-04-23T16:52:04.541 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:04 smithi103 ceph-mon[39629]: pgmap v1993: 27 pgs: 27 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-04-23T16:52:04.554 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:52:04.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:04.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:52:04.659 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:52:04.707 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:04.881 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:52:04.907 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:52:04.935 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:52:04.962 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:52:04.986 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:52:05.013 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:52:05.040 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:52:05.069 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:52:05.080 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 36 16 85163 85147 147.758 98.375 0.0101618 0.00676671 2024-04-23T16:52:05.095 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:52:05.123 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:52:05.148 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:52:05.148 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:52:05.148 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:52:05.148 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:52:05.148 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:52:05.148 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-23T16:52:05.148 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:52:05.149 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:52:05.149 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:52:05.149 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:52:05.149 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:52:05.149 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:52:05.149 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:52:05.462 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:05 smithi103 ceph-mon[39629]: osdmap e720: 12 total, 11 up, 7 in 2024-04-23T16:52:05.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:05 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3249845803' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:52:05.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:05 smithi103 ceph-mon[39629]: pgmap v1995: 27 pgs: 27 active+clean; 4.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-04-23T16:52:05.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:05.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:05.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:05.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:05.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:05 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:05.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:05 smithi188 ceph-mon[44101]: osdmap e720: 12 total, 11 up, 7 in 2024-04-23T16:52:05.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:05 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3249845803' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:52:05.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:05 smithi188 ceph-mon[44101]: pgmap v1995: 27 pgs: 27 active+clean; 4.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-04-23T16:52:05.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:05.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:05.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:05.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:05.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:05 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:05.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:05 smithi148 ceph-mon[44520]: osdmap e720: 12 total, 11 up, 7 in 2024-04-23T16:52:05.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:05 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3249845803' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:52:05.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:05 smithi148 ceph-mon[44520]: pgmap v1995: 27 pgs: 27 active+clean; 4.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-04-23T16:52:05.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:05.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:05.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:05.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:05.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:05 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:05.979 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:52:05.979 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:52:06.070 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:52:06.080 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 37 16 86775 86759 146.488 100.75 0.00453548 0.0068247 2024-04-23T16:52:06.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:06.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:52:06.182 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:52:06.213 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:06.236 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:52:06.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:06.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:52:06.340 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:52:06.398 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:06.425 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:52:06.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:06.433 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:52:06.534 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:52:06.564 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:06.575 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:52:06.585 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:52:06.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:06.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:52:06.689 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:52:06.717 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:06.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:06.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:06.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:06.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:06.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:06.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:06 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:06.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:06 smithi188 ceph-mon[44101]: 1.8 deep-scrub starts 2024-04-23T16:52:06.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:06 smithi188 ceph-mon[44101]: 1.8 deep-scrub ok 2024-04-23T16:52:06.740 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:52:06.743 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:06.743 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:06.743 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:06.743 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:06.743 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:06.743 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:06 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:06.743 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:06 smithi148 ceph-mon[44520]: 1.8 deep-scrub starts 2024-04-23T16:52:06.743 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:06 smithi148 ceph-mon[44520]: 1.8 deep-scrub ok 2024-04-23T16:52:06.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:06.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:52:06.846 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:52:06.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:06.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:06.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:06.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:06.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:06.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:06 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:06.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:06 smithi103 ceph-mon[39629]: 1.8 deep-scrub starts 2024-04-23T16:52:06.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:06 smithi103 ceph-mon[39629]: 1.8 deep-scrub ok 2024-04-23T16:52:06.903 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:06.903 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:52:06.904 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:52:06.908 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:52:06.931 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:52:06.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:06.935 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:52:07.036 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:52:07.065 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:07.080 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 38 16 88331 88315 145.193 97.25 0.0101957 0.00688587 2024-04-23T16:52:07.089 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:52:07.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:07.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:52:07.195 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:52:07.244 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:07.271 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:52:07.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:07.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:52:07.373 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:52:07.423 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:07.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:07 smithi188 ceph-mon[44101]: pgmap v1996: 27 pgs: 27 active+clean; 5.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-04-23T16:52:07.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:07 smithi103 ceph-mon[39629]: pgmap v1996: 27 pgs: 27 active+clean; 5.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-04-23T16:52:07.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:07 smithi148 ceph-mon[44520]: pgmap v1996: 27 pgs: 27 active+clean; 5.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-04-23T16:52:07.981 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:52:08.081 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 39 16 89788 89772 143.805 91.0625 0.00317937 0.00693877 2024-04-23T16:52:08.428 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:52:08.436 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:52:08.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:08.441 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:52:08.457 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-23T16:52:08.541 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:52:08.570 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:08.590 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:52:08.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:08.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:52:08.695 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:52:08.722 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:08.743 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:52:08.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:08.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:52:08.847 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:52:08.903 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:08.915 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:52:08.934 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:52:08.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:08.938 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:52:09.039 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:52:09.085 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:52:09.082957 min lat: 0.0010724 max lat: 0.210834 avg lat: 0.00701796 2024-04-23T16:52:09.085 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:52:09.085 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 40 16 91229 91213 142.455 90.0625 0.0111558 0.00701796 2024-04-23T16:52:09.088 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:09.203 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:52:09.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:09.209 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:52:09.310 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:52:09.360 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:10.018 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:52:10.083 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 41 15 92757 92742 141.311 95.5625 0.0117744 0.00707487 2024-04-23T16:52:10.344 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:10 smithi188 ceph-mon[44101]: pgmap v1997: 27 pgs: 27 active+clean; 5.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-04-23T16:52:10.344 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:10 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1203388074' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:52:10.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:10 smithi103 ceph-mon[39629]: pgmap v1997: 27 pgs: 27 active+clean; 5.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-04-23T16:52:10.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:10 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1203388074' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:52:10.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:10 smithi148 ceph-mon[44520]: pgmap v1997: 27 pgs: 27 active+clean; 5.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-04-23T16:52:10.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:10 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1203388074' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:52:11.083 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 42 16 94322 94306 140.275 97.75 0.00560746 0.00712731 2024-04-23T16:52:11.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:11 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:52:11.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:11 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1203388074' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:52:11.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:11 smithi103 ceph-mon[39629]: osdmap e721: 12 total, 11 up, 7 in 2024-04-23T16:52:11.327 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:52:11.328 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:52:11.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:11 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:52:11.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:11 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1203388074' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:52:11.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:11 smithi188 ceph-mon[44101]: osdmap e721: 12 total, 11 up, 7 in 2024-04-23T16:52:11.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:11 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:52:11.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:11 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1203388074' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:52:11.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:11 smithi148 ceph-mon[44520]: osdmap e721: 12 total, 11 up, 7 in 2024-04-23T16:52:11.425 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:52:11.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:11.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:52:11.536 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:52:11.564 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:11.584 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:52:11.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:11.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:52:11.691 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:52:11.737 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:12.083 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-23T16:52:12.084 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 43 16 95861 95845 139.249 96.1875 0.00567431 0.00717955 2024-04-23T16:52:12.295 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:52:12.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:12.298 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:12 smithi103 ceph-mon[39629]: pgmap v1999: 27 pgs: 27 active+clean; 5.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-04-23T16:52:12.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:52:12.369 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:52:12.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:12 smithi148 ceph-mon[44520]: pgmap v1999: 27 pgs: 27 active+clean; 5.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-04-23T16:52:12.400 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:52:12.404 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:52:12.433 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:52:12.458 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:12 smithi188 ceph-mon[44101]: pgmap v1999: 27 pgs: 27 active+clean; 5.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-04-23T16:52:12.462 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:52:12.470 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:12.487 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:52:12.498 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:52:12.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:12.505 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:52:12.510 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:52:12.538 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:52:12.567 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:52:12.597 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:52:12.605 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:52:12.626 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:52:12.644 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:12.665 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:52:12.665 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:52:12.665 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:52:12.665 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:52:12.665 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:52:12.666 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-23T16:52:12.666 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:52:12.666 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:52:12.666 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:52:12.666 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:52:12.666 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:52:12.666 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:52:12.666 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:52:12.677 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:52:12.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:12.684 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:52:12.785 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:52:12.813 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:12.834 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:52:12.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:12.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:52:12.939 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:52:12.974 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:12.999 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:52:13.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:13.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:52:13.083 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 44 16 97390 97374 138.257 95.5625 0.0108919 0.00723117 2024-04-23T16:52:13.105 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:52:13.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:13 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1582017092' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:52:13.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:13.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:13.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:13.151 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:13.188 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:52:13.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:13.192 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:52:13.293 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:52:13.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:13 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1582017092' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:52:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:13.354 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:13 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1582017092' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:52:13.354 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:13.354 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:13.354 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:13.355 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:13.355 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:13.357 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:13.387 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:52:13.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:13.393 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:52:13.490 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:52:13.493 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:52:13.493 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:52:13.533 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:52:13.544 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:13.574 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:52:13.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:13.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:52:13.679 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:52:13.717 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:13.756 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:52:13.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:13.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:52:13.866 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:52:13.913 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:13.950 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:52:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:13.954 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:52:14.055 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:52:14.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:14.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:14.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:14.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:14.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:14.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:14.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:14 smithi103 ceph-mon[39629]: pgmap v2000: 27 pgs: 27 active+clean; 5.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-04-23T16:52:14.085 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 45 16 98904 98888 137.282 94.625 0.00793646 0.00728201 2024-04-23T16:52:14.103 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:14.132 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:52:14.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:14.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:52:14.238 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:52:14.273 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:14.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:14 smithi148 ceph-mon[44520]: pgmap v2000: 27 pgs: 27 active+clean; 5.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-04-23T16:52:14.378 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:52:14.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:14.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:52:14.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:14.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:14.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:14.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:14.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:14.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:14.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:14 smithi188 ceph-mon[44101]: pgmap v2000: 27 pgs: 27 active+clean; 5.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-04-23T16:52:14.483 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:52:14.529 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:14.582 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:52:14.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:14.584 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:52:14.685 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:52:14.714 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:14.735 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:52:14.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:14.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:52:14.801 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:52:14.839 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:52:14.891 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:14.921 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:52:14.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:14.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:52:15.037 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:52:15.078 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:15.088 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 46 16 100405 100389 136.325 93.8125 0.00451829 0.00733367 2024-04-23T16:52:15.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:15 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:52:15.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:15 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:52:15.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:15 smithi103 ceph-mon[39629]: osdmap e722: 12 total, 11 up, 7 in 2024-04-23T16:52:15.286 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:52:15.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:15.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:52:15.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:15 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:52:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:15 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:52:15.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:15 smithi148 ceph-mon[44520]: osdmap e722: 12 total, 11 up, 7 in 2024-04-23T16:52:15.398 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:52:15.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:15 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:52:15.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:15 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:52:15.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:15 smithi188 ceph-mon[44101]: osdmap e722: 12 total, 11 up, 7 in 2024-04-23T16:52:15.445 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:15.475 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:52:15.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:15.480 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:52:15.494 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:52:15.582 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:52:15.609 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:15.620 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:52:15.630 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:52:15.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:15.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:52:15.647 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:52:15.648 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:52:15.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.648 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.27202.0:101186 9.2 e722/703 9:4ec1a0c3:::benchmark_data_smithi188_167360_object101185:head v 722'6422, mlcod=722'6419)", 2024-04-23T16:52:15.648 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:52:15.629124+0000", 2024-04-23T16:52:15.648 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.013768757, 2024-04-23T16:52:15.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013799511, 2024-04-23T16:52:15.648 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:52:15.648 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:52:15.648 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:52:15.648 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:52:15.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:52:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.629124+0000", 2024-04-23T16:52:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:52:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.629124+0000", 2024-04-23T16:52:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:52:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.629125+0000", 2024-04-23T16:52:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9399999999999993e-07 2024-04-23T16:52:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:52:15.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.629203+0000", 2024-04-23T16:52:15.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8349999999999994e-05 2024-04-23T16:52:15.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:52:15.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.629215+0000", 2024-04-23T16:52:15.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.22e-05 2024-04-23T16:52:15.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:52:15.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.629224+0000", 2024-04-23T16:52:15.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5429999999999994e-06 2024-04-23T16:52:15.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:52:15.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.630752+0000", 2024-04-23T16:52:15.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001527624 2024-04-23T16:52:15.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:52:15.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.630770+0000", 2024-04-23T16:52:15.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8413000000000001e-05 2024-04-23T16:52:15.651 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.651 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:52:15.651 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.652 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.27202.0:101187 9.7 e722/703 9:e8d3c818:::benchmark_data_smithi188_167360_object101186:head v 722'6440, mlcod=722'6438)", 2024-04-23T16:52:15.652 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:52:15.629323+0000", 2024-04-23T16:52:15.652 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.013569695999999999, 2024-04-23T16:52:15.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013667288, 2024-04-23T16:52:15.652 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:52:15.652 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:52:15.652 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:52:15.652 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:52:15.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:52:15.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.629323+0000", 2024-04-23T16:52:15.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:52:15.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.629323+0000", 2024-04-23T16:52:15.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:52:15.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.629324+0000", 2024-04-23T16:52:15.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4600000000000003e-07 2024-04-23T16:52:15.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:52:15.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.629422+0000", 2024-04-23T16:52:15.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7658999999999995e-05 2024-04-23T16:52:15.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.654 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:52:15.654 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.629428+0000", 2024-04-23T16:52:15.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8789999999999997e-06 2024-04-23T16:52:15.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.654 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:52:15.654 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.629434+0000", 2024-04-23T16:52:15.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2850000000000002e-06 2024-04-23T16:52:15.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:52:15.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.629514+0000", 2024-04-23T16:52:15.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0144999999999999e-05 2024-04-23T16:52:15.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:52:15.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.629535+0000", 2024-04-23T16:52:15.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1342e-05 2024-04-23T16:52:15.655 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.655 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:52:15.655 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.655 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.27202.0:101189 9.6 9:6c951824:::benchmark_data_smithi188_167360_object101188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e722)", 2024-04-23T16:52:15.655 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:52:15.631307+0000", 2024-04-23T16:52:15.656 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.011585164, 2024-04-23T16:52:15.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01174995, 2024-04-23T16:52:15.656 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:52:15.656 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:52:15.656 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:52:15.656 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:52:15.656 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.27202", 2024-04-23T16:52:15.656 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:52:15.656 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 101189 2024-04-23T16:52:15.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.656 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:52:15.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:52:15.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631307+0000", 2024-04-23T16:52:15.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:52:15.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631307+0000", 2024-04-23T16:52:15.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:52:15.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631309+0000", 2024-04-23T16:52:15.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9190000000000002e-06 2024-04-23T16:52:15.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:52:15.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631341+0000", 2024-04-23T16:52:15.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1612999999999997e-05 2024-04-23T16:52:15.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:52:15.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631355+0000", 2024-04-23T16:52:15.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3609e-05 2024-04-23T16:52:15.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:52:15.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631362+0000", 2024-04-23T16:52:15.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.311e-06 2024-04-23T16:52:15.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:52:15.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631414+0000", 2024-04-23T16:52:15.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2006000000000003e-05 2024-04-23T16:52:15.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:52:15.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631543+0000", 2024-04-23T16:52:15.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012877099999999999 2024-04-23T16:52:15.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 9", 2024-04-23T16:52:15.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631637+0000", 2024-04-23T16:52:15.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4538999999999997e-05 2024-04-23T16:52:15.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.660 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:52:15.660 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.638086+0000", 2024-04-23T16:52:15.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0064484549999999996 2024-04-23T16:52:15.660 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.660 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:52:15.660 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.660 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.27202.0:101190 9.a 9:5ceccc2f:::benchmark_data_smithi188_167360_object101189:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e722)", 2024-04-23T16:52:15.660 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:52:15.631379+0000", 2024-04-23T16:52:15.660 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.011513825, 2024-04-23T16:52:15.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011760789000000001, 2024-04-23T16:52:15.660 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:52:15.660 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:52:15.661 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:52:15.661 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:52:15.661 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.27202", 2024-04-23T16:52:15.661 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:52:15.661 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 101190 2024-04-23T16:52:15.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.661 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:52:15.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:52:15.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631379+0000", 2024-04-23T16:52:15.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:52:15.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631379+0000", 2024-04-23T16:52:15.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:52:15.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631379+0000", 2024-04-23T16:52:15.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2399999999999997e-07 2024-04-23T16:52:15.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:52:15.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631459+0000", 2024-04-23T16:52:15.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9963999999999995e-05 2024-04-23T16:52:15.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:52:15.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631467+0000", 2024-04-23T16:52:15.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8299999999999996e-06 2024-04-23T16:52:15.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:52:15.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631473+0000", 2024-04-23T16:52:15.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.156e-06 2024-04-23T16:52:15.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:52:15.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.633956+0000", 2024-04-23T16:52:15.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024822860000000002 2024-04-23T16:52:15.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:52:15.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.634074+0000", 2024-04-23T16:52:15.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118566 2024-04-23T16:52:15.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 8", 2024-04-23T16:52:15.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.634202+0000", 2024-04-23T16:52:15.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012746 2024-04-23T16:52:15.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:52:15.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.640146+0000", 2024-04-23T16:52:15.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0059445740000000002 2024-04-23T16:52:15.664 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.665 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:52:15.665 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.665 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.27202.0:101191 9.a 9:5f9d5393:::benchmark_data_smithi188_167360_object101190:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e722)", 2024-04-23T16:52:15.665 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:52:15.631486+0000", 2024-04-23T16:52:15.665 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.011406642999999999, 2024-04-23T16:52:15.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011735120999999999, 2024-04-23T16:52:15.665 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:52:15.665 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:52:15.665 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:52:15.665 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:52:15.665 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.27202", 2024-04-23T16:52:15.665 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:52:15.666 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 101191 2024-04-23T16:52:15.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.666 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:52:15.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:52:15.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631486+0000", 2024-04-23T16:52:15.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:52:15.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631486+0000", 2024-04-23T16:52:15.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:52:15.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631487+0000", 2024-04-23T16:52:15.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4700000000000001e-07 2024-04-23T16:52:15.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:52:15.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631571+0000", 2024-04-23T16:52:15.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4349000000000002e-05 2024-04-23T16:52:15.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:52:15.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631578+0000", 2024-04-23T16:52:15.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2969999999999998e-06 2024-04-23T16:52:15.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:52:15.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631587+0000", 2024-04-23T16:52:15.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9940000000000006e-06 2024-04-23T16:52:15.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:52:15.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.637067+0000", 2024-04-23T16:52:15.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054796230000000003 2024-04-23T16:52:15.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:52:15.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.637154+0000", 2024-04-23T16:52:15.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7318000000000007e-05 2024-04-23T16:52:15.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 8", 2024-04-23T16:52:15.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.637246+0000", 2024-04-23T16:52:15.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1634999999999999e-05 2024-04-23T16:52:15.669 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.669 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:52:15.669 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.669 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.27202.0:101193 9.6 9:6673d829:::benchmark_data_smithi188_167360_object101192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e722)", 2024-04-23T16:52:15.669 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:52:15.631596+0000", 2024-04-23T16:52:15.669 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.011296914, 2024-04-23T16:52:15.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011701506, 2024-04-23T16:52:15.670 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:52:15.670 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:52:15.670 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:52:15.670 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:52:15.670 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.27202", 2024-04-23T16:52:15.670 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:52:15.670 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 101193 2024-04-23T16:52:15.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.670 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:52:15.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.670 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:52:15.670 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631596+0000", 2024-04-23T16:52:15.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.671 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:52:15.671 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631596+0000", 2024-04-23T16:52:15.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.671 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:52:15.671 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631596+0000", 2024-04-23T16:52:15.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6700000000000003e-07 2024-04-23T16:52:15.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.671 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:52:15.671 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631719+0000", 2024-04-23T16:52:15.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012296599999999999 2024-04-23T16:52:15.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:52:15.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631727+0000", 2024-04-23T16:52:15.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.464e-06 2024-04-23T16:52:15.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:52:15.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631733+0000", 2024-04-23T16:52:15.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.161e-06 2024-04-23T16:52:15.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:52:15.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.634647+0000", 2024-04-23T16:52:15.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0029137759999999999 2024-04-23T16:52:15.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:52:15.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.634760+0000", 2024-04-23T16:52:15.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112821 2024-04-23T16:52:15.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 9", 2024-04-23T16:52:15.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.634883+0000", 2024-04-23T16:52:15.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012304800000000001 2024-04-23T16:52:15.673 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.673 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:52:15.673 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.674 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.27202.0:101194 9.0 9:0ac89c17:::benchmark_data_smithi188_167360_object101193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e722)", 2024-04-23T16:52:15.674 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:52:15.631741+0000", 2024-04-23T16:52:15.674 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.011151322, 2024-04-23T16:52:15.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011629618, 2024-04-23T16:52:15.674 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:52:15.674 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:52:15.674 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:52:15.674 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:52:15.674 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.27202", 2024-04-23T16:52:15.674 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:52:15.674 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 101194 2024-04-23T16:52:15.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.674 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:52:15.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:52:15.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631741+0000", 2024-04-23T16:52:15.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:52:15.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631741+0000", 2024-04-23T16:52:15.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:52:15.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631742+0000", 2024-04-23T16:52:15.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5099999999999999e-07 2024-04-23T16:52:15.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:52:15.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631807+0000", 2024-04-23T16:52:15.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5357999999999993e-05 2024-04-23T16:52:15.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:52:15.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631814+0000", 2024-04-23T16:52:15.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0129999999999996e-06 2024-04-23T16:52:15.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:52:15.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631820+0000", 2024-04-23T16:52:15.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8089999999999998e-06 2024-04-23T16:52:15.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:52:15.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631865+0000", 2024-04-23T16:52:15.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4931000000000001e-05 2024-04-23T16:52:15.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:52:15.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631976+0000", 2024-04-23T16:52:15.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110747 2024-04-23T16:52:15.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 11", 2024-04-23T16:52:15.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.632099+0000", 2024-04-23T16:52:15.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012306 2024-04-23T16:52:15.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:52:15.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.638693+0000", 2024-04-23T16:52:15.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0065941740000000004 2024-04-23T16:52:15.678 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.678 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:52:15.678 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.678 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.27202.0:101195 9.0 9:08131edb:::benchmark_data_smithi188_167360_object101194:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e722)", 2024-04-23T16:52:15.678 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:52:15.631832+0000", 2024-04-23T16:52:15.678 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.011060708000000001, 2024-04-23T16:52:15.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011633866, 2024-04-23T16:52:15.679 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:52:15.679 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:52:15.679 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:52:15.679 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:52:15.679 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.27202", 2024-04-23T16:52:15.679 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:52:15.679 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 101195 2024-04-23T16:52:15.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.679 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:52:15.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:52:15.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631832+0000", 2024-04-23T16:52:15.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:52:15.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631832+0000", 2024-04-23T16:52:15.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:52:15.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631833+0000", 2024-04-23T16:52:15.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6700000000000003e-07 2024-04-23T16:52:15.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:52:15.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631899+0000", 2024-04-23T16:52:15.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6736000000000006e-05 2024-04-23T16:52:15.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:52:15.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631906+0000", 2024-04-23T16:52:15.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8709999999999999e-06 2024-04-23T16:52:15.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:52:15.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.631912+0000", 2024-04-23T16:52:15.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0820000000000004e-06 2024-04-23T16:52:15.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:52:15.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635378+0000", 2024-04-23T16:52:15.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003465858 2024-04-23T16:52:15.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:52:15.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635510+0000", 2024-04-23T16:52:15.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013159099999999999 2024-04-23T16:52:15.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 11", 2024-04-23T16:52:15.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635646+0000", 2024-04-23T16:52:15.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013598000000000001 2024-04-23T16:52:15.682 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.682 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:52:15.682 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.683 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.683 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.27202.0:101197 9.4 9:208d024f:::benchmark_data_smithi188_167360_object101196:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e722)", 2024-04-23T16:52:15.683 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:52:15.632067+0000", 2024-04-23T16:52:15.683 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.010825292, 2024-04-23T16:52:15.683 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011475393, 2024-04-23T16:52:15.683 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:52:15.683 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:52:15.683 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:52:15.683 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:52:15.683 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.27202", 2024-04-23T16:52:15.683 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:52:15.683 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 101197 2024-04-23T16:52:15.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.684 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:52:15.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:52:15.684 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.632067+0000", 2024-04-23T16:52:15.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.684 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:52:15.684 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.632067+0000", 2024-04-23T16:52:15.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.684 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:52:15.684 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.632068+0000", 2024-04-23T16:52:15.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0160000000000001e-06 2024-04-23T16:52:15.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.685 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:52:15.685 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.632198+0000", 2024-04-23T16:52:15.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012929 2024-04-23T16:52:15.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.685 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:52:15.685 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.632206+0000", 2024-04-23T16:52:15.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9650000000000008e-06 2024-04-23T16:52:15.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.685 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:52:15.685 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.632212+0000", 2024-04-23T16:52:15.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1920000000000002e-06 2024-04-23T16:52:15.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:52:15.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.632259+0000", 2024-04-23T16:52:15.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7448999999999999e-05 2024-04-23T16:52:15.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:52:15.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.632368+0000", 2024-04-23T16:52:15.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010900699999999999 2024-04-23T16:52:15.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 8", 2024-04-23T16:52:15.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.632505+0000", 2024-04-23T16:52:15.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136568 2024-04-23T16:52:15.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:52:15.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.639068+0000", 2024-04-23T16:52:15.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0065628780000000003 2024-04-23T16:52:15.687 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.687 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:52:15.687 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.687 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.27202.0:101196 9.d e722/703 9:ba796ae1:::benchmark_data_smithi188_167360_object101195:head v 722'6150, mlcod=722'6146)", 2024-04-23T16:52:15.687 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:52:15.632371+0000", 2024-04-23T16:52:15.687 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.010522091000000001, 2024-04-23T16:52:15.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011251063, 2024-04-23T16:52:15.688 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:52:15.688 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:52:15.688 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:52:15.688 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:52:15.688 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.688 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:52:15.688 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.632371+0000", 2024-04-23T16:52:15.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.688 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.689 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.689 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:52:15.689 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.632371+0000", 2024-04-23T16:52:15.689 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.689 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.689 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:52:15.689 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.632372+0000", 2024-04-23T16:52:15.689 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3680000000000001e-06 2024-04-23T16:52:15.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.689 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.689 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:52:15.689 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.632428+0000", 2024-04-23T16:52:15.689 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5875000000000003e-05 2024-04-23T16:52:15.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:52:15.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.632436+0000", 2024-04-23T16:52:15.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7239999999999998e-06 2024-04-23T16:52:15.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:52:15.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.632445+0000", 2024-04-23T16:52:15.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4649999999999994e-06 2024-04-23T16:52:15.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:52:15.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.632516+0000", 2024-04-23T16:52:15.691 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0593999999999995e-05 2024-04-23T16:52:15.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:52:15.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.632534+0000", 2024-04-23T16:52:15.691 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8105999999999999e-05 2024-04-23T16:52:15.691 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.691 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:52:15.691 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.691 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.27202.0:101199 9.d e722/703 9:bf6a58bb:::benchmark_data_smithi188_167360_object101198:head v 722'6151, mlcod=722'6148)", 2024-04-23T16:52:15.691 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:52:15.635380+0000", 2024-04-23T16:52:15.691 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.0075123799999999999, 2024-04-23T16:52:15.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083039670000000006, 2024-04-23T16:52:15.692 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:52:15.692 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:52:15.692 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:52:15.692 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:52:15.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:52:15.692 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635380+0000", 2024-04-23T16:52:15.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:52:15.692 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635380+0000", 2024-04-23T16:52:15.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:52:15.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635382+0000", 2024-04-23T16:52:15.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.485e-06 2024-04-23T16:52:15.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:52:15.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635417+0000", 2024-04-23T16:52:15.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5281000000000003e-05 2024-04-23T16:52:15.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:52:15.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635426+0000", 2024-04-23T16:52:15.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7229999999999993e-06 2024-04-23T16:52:15.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.694 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:52:15.694 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635434+0000", 2024-04-23T16:52:15.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7919999999999993e-06 2024-04-23T16:52:15.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.694 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:52:15.694 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635583+0000", 2024-04-23T16:52:15.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014983299999999999 2024-04-23T16:52:15.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.695 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:52:15.695 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635603+0000", 2024-04-23T16:52:15.695 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9349000000000001e-05 2024-04-23T16:52:15.695 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.695 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:52:15.695 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.695 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.695 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.695 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.27202.0:101198 9.6 9:64cb8f39:::benchmark_data_smithi188_167360_object101197:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e722)", 2024-04-23T16:52:15.695 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:52:15.635558+0000", 2024-04-23T16:52:15.695 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.0073346740000000002, 2024-04-23T16:52:15.695 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0082073130000000008, 2024-04-23T16:52:15.695 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:52:15.695 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:52:15.696 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:52:15.696 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:52:15.696 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.27202", 2024-04-23T16:52:15.696 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:52:15.696 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 101198 2024-04-23T16:52:15.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.696 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:52:15.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:52:15.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635558+0000", 2024-04-23T16:52:15.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:52:15.697 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635558+0000", 2024-04-23T16:52:15.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:52:15.697 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635559+0000", 2024-04-23T16:52:15.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3230000000000001e-06 2024-04-23T16:52:15.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:52:15.697 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635591+0000", 2024-04-23T16:52:15.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1470000000000002e-05 2024-04-23T16:52:15.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:52:15.698 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635598+0000", 2024-04-23T16:52:15.698 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4200000000000001e-06 2024-04-23T16:52:15.698 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.698 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.698 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:52:15.698 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635604+0000", 2024-04-23T16:52:15.698 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3230000000000003e-06 2024-04-23T16:52:15.698 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.698 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.698 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:52:15.698 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.638116+0000", 2024-04-23T16:52:15.698 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002511101 2024-04-23T16:52:15.698 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.698 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:52:15.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.638233+0000", 2024-04-23T16:52:15.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117175 2024-04-23T16:52:15.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 9", 2024-04-23T16:52:15.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.638359+0000", 2024-04-23T16:52:15.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012662700000000001 2024-04-23T16:52:15.699 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.699 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:52:15.699 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.699 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.27202.0:101200 9.6 9:6d63244f:::benchmark_data_smithi188_167360_object101199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e722)", 2024-04-23T16:52:15.700 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:52:15.635614+0000", 2024-04-23T16:52:15.700 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.0072783819999999999, 2024-04-23T16:52:15.700 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0082463859999999996, 2024-04-23T16:52:15.700 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:52:15.700 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:52:15.700 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:52:15.700 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:52:15.700 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.27202", 2024-04-23T16:52:15.700 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:52:15.700 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 101200 2024-04-23T16:52:15.700 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.700 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:52:15.700 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.700 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:52:15.700 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635614+0000", 2024-04-23T16:52:15.701 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.701 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.701 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.701 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:52:15.701 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635614+0000", 2024-04-23T16:52:15.701 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.701 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.701 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.701 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:52:15.701 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635615+0000", 2024-04-23T16:52:15.701 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1799999999999995e-07 2024-04-23T16:52:15.701 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.701 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.701 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:52:15.702 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635643+0000", 2024-04-23T16:52:15.702 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8096999999999999e-05 2024-04-23T16:52:15.702 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.702 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.702 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:52:15.702 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635649+0000", 2024-04-23T16:52:15.702 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7819999999999999e-06 2024-04-23T16:52:15.702 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.702 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.702 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:52:15.702 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.635654+0000", 2024-04-23T16:52:15.702 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1869999999999996e-06 2024-04-23T16:52:15.702 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.702 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.703 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:52:15.703 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.641401+0000", 2024-04-23T16:52:15.703 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0057473680000000001 2024-04-23T16:52:15.703 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.703 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:52:15.703 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.641542+0000", 2024-04-23T16:52:15.703 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014067600000000001 2024-04-23T16:52:15.703 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.703 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 9", 2024-04-23T16:52:15.703 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.641685+0000", 2024-04-23T16:52:15.703 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014352200000000001 2024-04-23T16:52:15.703 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.704 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:52:15.704 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.704 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.704 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.704 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.27202.0:101204 9.2 e722/703 9:4795eb12:::benchmark_data_smithi188_167360_object101203:head v 722'6423, mlcod=722'6420)", 2024-04-23T16:52:15.704 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:52:15.640856+0000", 2024-04-23T16:52:15.704 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.0020365919999999998, 2024-04-23T16:52:15.704 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030922219999999999, 2024-04-23T16:52:15.704 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:52:15.704 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:52:15.704 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:52:15.704 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:52:15.704 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.704 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:52:15.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.640856+0000", 2024-04-23T16:52:15.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.705 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.705 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.705 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:52:15.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.640856+0000", 2024-04-23T16:52:15.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.705 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.705 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.705 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:52:15.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.640857+0000", 2024-04-23T16:52:15.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2900000000000002e-07 2024-04-23T16:52:15.705 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.705 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.706 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:52:15.706 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.640965+0000", 2024-04-23T16:52:15.706 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108556 2024-04-23T16:52:15.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.706 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.706 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:52:15.706 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.640975+0000", 2024-04-23T16:52:15.706 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0680000000000003e-06 2024-04-23T16:52:15.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.706 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.706 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:52:15.706 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.640980+0000", 2024-04-23T16:52:15.706 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4519999999999996e-06 2024-04-23T16:52:15.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.707 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:52:15.707 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.641021+0000", 2024-04-23T16:52:15.707 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1009e-05 2024-04-23T16:52:15.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.707 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:52:15.707 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.641042+0000", 2024-04-23T16:52:15.707 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1311e-05 2024-04-23T16:52:15.707 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.707 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:52:15.707 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.708 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.27202.0:101206 9.2 e722/703 9:457d5db8:::benchmark_data_smithi188_167360_object101205:head v 722'6424, mlcod=722'6420)", 2024-04-23T16:52:15.708 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:52:15.643535+0000", 2024-04-23T16:52:15.708 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4294967295.9993577, 2024-04-23T16:52:15.708 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000476074, 2024-04-23T16:52:15.708 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:52:15.708 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:52:15.708 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:52:15.708 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:52:15.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.708 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:52:15.708 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643535+0000", 2024-04-23T16:52:15.708 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.708 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:52:15.709 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643535+0000", 2024-04-23T16:52:15.709 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:52:15.709 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643537+0000", 2024-04-23T16:52:15.709 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1789999999999998e-06 2024-04-23T16:52:15.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:52:15.709 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643624+0000", 2024-04-23T16:52:15.709 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6656999999999994e-05 2024-04-23T16:52:15.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:52:15.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643640+0000", 2024-04-23T16:52:15.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6300999999999999e-05 2024-04-23T16:52:15.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:52:15.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643649+0000", 2024-04-23T16:52:15.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0189999999999995e-06 2024-04-23T16:52:15.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:52:15.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643817+0000", 2024-04-23T16:52:15.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016756799999999999 2024-04-23T16:52:15.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.711 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:52:15.711 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643837+0000", 2024-04-23T16:52:15.711 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9921000000000001e-05 2024-04-23T16:52:15.711 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.711 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:52:15.711 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.711 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop_reply(client.27202.0:101193 9.6 e722/703)", 2024-04-23T16:52:15.711 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:52:15.643614+0000", 2024-04-23T16:52:15.711 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4294967295.9992785, 2024-04-23T16:52:15.711 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00045469100000000001, 2024-04-23T16:52:15.711 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:52:15.712 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:52:15.712 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "queued for pg", 2024-04-23T16:52:15.712 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:52:15.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.712 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:52:15.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643614+0000", 2024-04-23T16:52:15.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.712 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:52:15.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643614+0000", 2024-04-23T16:52:15.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.712 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:52:15.713 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643615+0000", 2024-04-23T16:52:15.713 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.415e-06 2024-04-23T16:52:15.713 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.713 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.713 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:52:15.713 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643625+0000", 2024-04-23T16:52:15.713 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0076e-05 2024-04-23T16:52:15.713 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.713 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.713 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:52:15.713 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643697+0000", 2024-04-23T16:52:15.713 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1812999999999998e-05 2024-04-23T16:52:15.713 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.713 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.714 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:52:15.714 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643704+0000", 2024-04-23T16:52:15.714 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1439999999999997e-06 2024-04-23T16:52:15.714 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.714 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:52:15.714 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.714 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.714 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.714 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.27202.0:101205 9.4 9:236e1bcd:::benchmark_data_smithi188_167360_object101204:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e722)", 2024-04-23T16:52:15.714 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:52:15.643767+0000", 2024-04-23T16:52:15.714 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4294967295.999126, 2024-04-23T16:52:15.714 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000368709, 2024-04-23T16:52:15.714 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:52:15.714 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:52:15.715 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:52:15.715 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:52:15.715 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.27202", 2024-04-23T16:52:15.715 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:52:15.715 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 101205 2024-04-23T16:52:15.715 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.715 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:52:15.715 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.715 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:52:15.715 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643767+0000", 2024-04-23T16:52:15.715 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.715 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.715 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.715 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:52:15.716 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643767+0000", 2024-04-23T16:52:15.716 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:52:15.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.716 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.716 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:52:15.716 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643768+0000", 2024-04-23T16:52:15.716 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6500000000000001e-06 2024-04-23T16:52:15.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.716 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.716 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:52:15.716 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643801+0000", 2024-04-23T16:52:15.716 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2694999999999998e-05 2024-04-23T16:52:15.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.716 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.717 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:52:15.717 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643809+0000", 2024-04-23T16:52:15.717 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0190000000000006e-06 2024-04-23T16:52:15.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.717 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.717 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:52:15.717 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643816+0000", 2024-04-23T16:52:15.717 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.725e-06 2024-04-23T16:52:15.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.717 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.717 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:52:15.717 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.643888+0000", 2024-04-23T16:52:15.717 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2673e-05 2024-04-23T16:52:15.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:52:15.717 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:52:15.718 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:52:15.718 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:15.644019+0000", 2024-04-23T16:52:15.718 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000130765 2024-04-23T16:52:15.718 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.718 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:52:15.718 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.718 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:52:15.718 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:52:15.718 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 17 2024-04-23T16:52:15.718 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:52:15.736 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:52:15.780 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:15.941 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:52:15.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:15.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:52:16.048 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:52:16.079 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:16.089 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 47 16 101867 101851 135.369 91.375 0.00939761 0.00738526 2024-04-23T16:52:16.099 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:52:16.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:16.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:52:16.204 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:52:16.238 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:16 smithi103 ceph-mon[39629]: pgmap v2002: 27 pgs: 27 active+clean; 5.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-23T16:52:16.241 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:16.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:16 smithi148 ceph-mon[44520]: pgmap v2002: 27 pgs: 27 active+clean; 5.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-23T16:52:16.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:16 smithi188 ceph-mon[44101]: pgmap v2002: 27 pgs: 27 active+clean; 5.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-23T16:52:16.693 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:52:16.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:16.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:52:16.724 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-23T16:52:16.798 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:52:16.858 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:17.089 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 48 16 103339 103323 134.466 92 0.0129698 0.00743596 2024-04-23T16:52:18.089 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 49 16 104754 104738 133.526 88.4375 0.0147839 0.0074868 2024-04-23T16:52:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:18 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:52:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:18 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:52:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:18 smithi103 ceph-mon[39629]: pgmap v2003: 27 pgs: 27 active+clean; 6.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-04-23T16:52:18.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:18 smithi103 ceph-mon[39629]: osdmap e723: 12 total, 11 up, 7 in 2024-04-23T16:52:18.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:18 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:52:18.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:18 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:52:18.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:18 smithi148 ceph-mon[44520]: pgmap v2003: 27 pgs: 27 active+clean; 6.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-04-23T16:52:18.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:18 smithi148 ceph-mon[44520]: osdmap e723: 12 total, 11 up, 7 in 2024-04-23T16:52:18.450 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:52:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:18 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:52:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:18 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:52:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:18 smithi188 ceph-mon[44101]: pgmap v2003: 27 pgs: 27 active+clean; 6.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-04-23T16:52:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:18 smithi188 ceph-mon[44101]: osdmap e723: 12 total, 11 up, 7 in 2024-04-23T16:52:18.596 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:52:18.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:18.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:52:18.718 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:52:18.770 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:19.089 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 50 16 106161 106145 132.615 87.9375 0.0126505 0.00753926 2024-04-23T16:52:19.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:19 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:52:19.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:19 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:52:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:19 smithi103 ceph-mon[39629]: osdmap e724: 12 total, 11 up, 7 in 2024-04-23T16:52:19.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:19 smithi103 ceph-mon[39629]: pgmap v2006: 27 pgs: 27 active+clean; 6.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-23T16:52:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:19 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:52:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:19 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:52:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:19 smithi188 ceph-mon[44101]: osdmap e724: 12 total, 11 up, 7 in 2024-04-23T16:52:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:19 smithi188 ceph-mon[44101]: pgmap v2006: 27 pgs: 27 active+clean; 6.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-23T16:52:19.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:19 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:52:19.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:19 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:52:19.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:19 smithi148 ceph-mon[44520]: osdmap e724: 12 total, 11 up, 7 in 2024-04-23T16:52:19.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:19 smithi148 ceph-mon[44520]: pgmap v2006: 27 pgs: 27 active+clean; 6.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-23T16:52:20.089 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 51 16 107633 107617 131.819 92 0.00442812 0.00758434 2024-04-23T16:52:20.256 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:52:20.257 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:20 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3744660802' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:52:20.257 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:20 smithi103 ceph-mon[39629]: from='client.37004 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:52:20.257 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:20 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:52:20.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:20 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3744660802' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:52:20.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:20 smithi148 ceph-mon[44520]: from='client.37004 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:52:20.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:20 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:52:20.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:20 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3744660802' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:52:20.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:20 smithi188 ceph-mon[44101]: from='client.37004 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:52:20.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:20 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:52:20.751 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:52:20.813 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-04-23T16:52:20.814 INFO:tasks.cephadm.osd.4:Restarting osd.4 (starting--it wasn't running)... 2024-04-23T16:52:20.814 DEBUG:teuthology.orchestra.run.smithi148:osd.4> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.4.service 2024-04-23T16:52:20.816 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl start ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.4 2024-04-23T16:52:20.845 INFO:journalctl@ceph.osd.4.smithi148.stdout:-- Logs begin at Tue 2024-04-23 15:57:31 UTC. -- 2024-04-23T16:52:21.090 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 52 16 109083 109067 131.026 90.625 0.00367052 0.00762996 2024-04-23T16:52:21.124 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:52:20 smithi148 systemd[1]: Starting Ceph osd.4 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T16:52:21.333 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:52:21.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:21.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:52:21.439 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:52:21.478 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:21.535 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:52:21.536 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:21 smithi103 ceph-mon[39629]: from='client.37004 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:52:21.536 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:21 smithi103 ceph-mon[39629]: osdmap e725: 12 total, 11 up, 7 in 2024-04-23T16:52:21.537 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:21 smithi103 ceph-mon[39629]: pgmap v2008: 27 pgs: 27 active+clean; 6.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-23T16:52:21.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:21.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:52:21.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:21 smithi148 ceph-mon[44520]: from='client.37004 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:52:21.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:21 smithi148 ceph-mon[44520]: osdmap e725: 12 total, 11 up, 7 in 2024-04-23T16:52:21.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:21 smithi148 ceph-mon[44520]: pgmap v2008: 27 pgs: 27 active+clean; 6.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-23T16:52:21.640 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:52:21.673 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:21.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:21 smithi188 ceph-mon[44101]: from='client.37004 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:52:21.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:21 smithi188 ceph-mon[44101]: osdmap e725: 12 total, 11 up, 7 in 2024-04-23T16:52:21.725 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:21 smithi188 ceph-mon[44101]: pgmap v2008: 27 pgs: 27 active+clean; 6.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-23T16:52:22.089 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 53 16 110507 110491 130.234 89 0.011267 0.00767668 2024-04-23T16:52:22.623 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:52:22 smithi148 bash[172772]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T16:52:22.624 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:52:22 smithi148 bash[172772]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-23T16:52:22.624 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:52:22 smithi148 bash[172772]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-23T16:52:22.624 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:52:22 smithi148 bash[172772]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-23T16:52:22.624 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:52:22 smithi148 bash[172772]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-23T16:52:22.624 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:52:22 smithi148 bash[172772]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-23T16:52:22.624 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:52:22 smithi148 bash[172772]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-23T16:52:22.624 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:52:22 smithi148 bash[172772]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-23T16:52:22.931 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:52:22.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:22.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:52:23.035 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:52:23.090 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 54 16 111939 111923 129.48 89.5 0.00749454 0.00772174 2024-04-23T16:52:23.093 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:23.123 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:52:23.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:23.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:52:23.227 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:52:23.263 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:23.309 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:52:23.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:23.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:52:23.373 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:52:23.414 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:52:23.450 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:23.473 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:52:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:23.481 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:52:23.582 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:52:23.631 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:23.654 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:52:23.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:23.658 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:52:23.671 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-23T16:52:23.711 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:52:23.747 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:52:23.759 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:52:23.779 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:52:23.799 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:23.815 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:52:23.827 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:52:23.828 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:52:23.829 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:52:23.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:23.831 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:52:23.838 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:52:23.877 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:52:23.903 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:52:23.932 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:52:23.935 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:52:23.969 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:52:23.973 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:23.989 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:52:23.996 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:52:24.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:24.001 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:52:24.003 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:52:24.017 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:52:23 smithi148 bash[173023]: d4f545da7afea84243aec13b06b0a74a2249ca7d14d73f5c6789379f58993f8d 2024-04-23T16:52:24.018 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:52:23 smithi148 systemd[1]: Started Ceph osd.4 for 8e3594f2-018b-11ef-bc93-c7b262605968. 2024-04-23T16:52:24.035 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:52:24.036 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:52:24.036 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:52:24.036 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:52:24.036 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:52:24.036 INFO:teuthology.orchestra.run.smithi103.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-23T16:52:24.036 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:52:24.036 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:52:24.036 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:52:24.037 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:52:24.037 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:52:24.037 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:52:24.037 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:52:24.090 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 55 16 113245 113229 128.61 81.625 0.00879971 0.00777427 2024-04-23T16:52:24.102 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:52:24.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:24 smithi103 ceph-mon[39629]: pgmap v2009: 27 pgs: 27 active+clean; 6.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-04-23T16:52:24.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1421416176' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:52:24.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:24.105 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:24 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:24.143 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:24.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:24 smithi188 ceph-mon[44101]: pgmap v2009: 27 pgs: 27 active+clean; 6.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-04-23T16:52:24.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1421416176' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:52:24.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:24.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:24 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:24.289 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:52:24 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[173037]: 2024-04-23T16:52:24.120+0000 7f94536db7c0 -1 Falling back to public interface 2024-04-23T16:52:24.290 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:24 smithi148 ceph-mon[44520]: pgmap v2009: 27 pgs: 27 active+clean; 6.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-04-23T16:52:24.290 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:24 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1421416176' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:52:24.290 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:24.290 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:24 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:24.306 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:52:24.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:24.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:52:24.410 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:52:24.445 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:24.922 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-23T16:52:25.090 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 56 15 114633 114618 127.863 86.8125 0.0118811 0.00781968 2024-04-23T16:52:25.321 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.321 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.321 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.321 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.321 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.322 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.322 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.322 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.322 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.322 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:25 smithi148 ceph-mon[44520]: osdmap e726: 12 total, 11 up, 7 in 2024-04-23T16:52:25.327 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:52:25.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:25 smithi103 ceph-mon[39629]: osdmap e726: 12 total, 11 up, 7 in 2024-04-23T16:52:25.370 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T16:52:25.371 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T16:52:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:25.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:25 smithi188 ceph-mon[44101]: osdmap e726: 12 total, 11 up, 7 in 2024-04-23T16:52:25.702 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:52:25.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:25.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:52:25.806 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:52:25.851 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:25.881 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:52:25.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:25.889 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:52:25.984 INFO:teuthology.orchestra.run.smithi148.stderr:no valid command found; 10 closest matches: 2024-04-23T16:52:25.985 INFO:teuthology.orchestra.run.smithi148.stderr:0 2024-04-23T16:52:25.985 INFO:teuthology.orchestra.run.smithi148.stderr:1 2024-04-23T16:52:25.985 INFO:teuthology.orchestra.run.smithi148.stderr:2 2024-04-23T16:52:25.985 INFO:teuthology.orchestra.run.smithi148.stderr:abort 2024-04-23T16:52:25.985 INFO:teuthology.orchestra.run.smithi148.stderr:assert 2024-04-23T16:52:25.985 INFO:teuthology.orchestra.run.smithi148.stderr:config diff 2024-04-23T16:52:25.985 INFO:teuthology.orchestra.run.smithi148.stderr:config diff get 2024-04-23T16:52:25.985 INFO:teuthology.orchestra.run.smithi148.stderr:config get 2024-04-23T16:52:25.985 INFO:teuthology.orchestra.run.smithi148.stderr:config help [] 2024-04-23T16:52:25.985 INFO:teuthology.orchestra.run.smithi148.stderr:config set ... 2024-04-23T16:52:25.985 INFO:teuthology.orchestra.run.smithi148.stderr:admin_socket: invalid command 2024-04-23T16:52:25.990 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:52:26.034 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:26.059 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:52:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:26.063 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:52:26.092 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 57 16 115982 115966 127.094 84.25 0.0122118 0.00786647 2024-04-23T16:52:26.164 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:52:26.217 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:26 smithi188 ceph-mon[44101]: pgmap v2011: 27 pgs: 27 active+clean; 6.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-23T16:52:26.220 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:26.244 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:52:26.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:26.252 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:52:26.336 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:26 smithi103 ceph-mon[39629]: pgmap v2011: 27 pgs: 27 active+clean; 6.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-23T16:52:26.340 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:26 smithi148 ceph-mon[44520]: pgmap v2011: 27 pgs: 27 active+clean; 6.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-23T16:52:26.353 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:52:26.388 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T16:52:26.730 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-04-23T16:52:26.784 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:52:27.092 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 58 16 117311 117295 126.335 83.0625 0.0166138 0.00791389 2024-04-23T16:52:27.371 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:52:27.644 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:52:27.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:27.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:52:27.749 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:52:27.805 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:27.847 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:52:27.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:27.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:52:27.874 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:52:27 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[173037]: 2024-04-23T16:52:27.532+0000 7f94536db7c0 -1 osd.4 686 log_to_monitors true 2024-04-23T16:52:27.954 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:52:28.005 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:28.039 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:52:28.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:28.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:52:28.093 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 59 16 118624 118608 125.582 82.0625 0.00437374 0.00796063 2024-04-23T16:52:28.144 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:52:28.176 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:28 smithi103 ceph-mon[39629]: pgmap v2012: 27 pgs: 27 active+clean; 7.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 93 MiB/s wr, 1.50k op/s 2024-04-23T16:52:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:28 smithi103 ceph-mon[39629]: from='osd.4 v1:172.21.15.148:6800/4230909898' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T16:52:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:28 smithi103 ceph-mon[39629]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T16:52:28.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:28 smithi148 ceph-mon[44520]: pgmap v2012: 27 pgs: 27 active+clean; 7.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 93 MiB/s wr, 1.50k op/s 2024-04-23T16:52:28.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:28 smithi148 ceph-mon[44520]: from='osd.4 v1:172.21.15.148:6800/4230909898' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T16:52:28.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:28 smithi148 ceph-mon[44520]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T16:52:28.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:28 smithi188 ceph-mon[44101]: pgmap v2012: 27 pgs: 27 active+clean; 7.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 93 MiB/s wr, 1.50k op/s 2024-04-23T16:52:28.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:28 smithi188 ceph-mon[44101]: from='osd.4 v1:172.21.15.148:6800/4230909898' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T16:52:28.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:28 smithi188 ceph-mon[44101]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T16:52:28.532 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:52:28.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:28.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:52:28.638 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:52:28.671 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:28.696 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:52:28.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:28.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:52:28.802 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:52:28.833 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 16:52:28 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[173037]: 2024-04-23T16:52:28.529+0000 7f9440b2b700 -1 osd.4 727 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:52:28.836 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:28.857 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:52:28.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:28.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:52:28.960 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:52:29.003 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:29.090 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:52:29.093 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:52:29.093801 min lat: 0.0010724 max lat: 0.210834 avg lat: 0.00801694 2024-04-23T16:52:29.093 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:52:29.093 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 60 15 119796 119781 124.711 73.3125 0.0128164 0.00801694 2024-04-23T16:52:29.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:29.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:52:29.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:29 smithi148 ceph-mon[44520]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T16:52:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:29 smithi148 ceph-mon[44520]: osdmap e727: 12 total, 11 up, 7 in 2024-04-23T16:52:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:29 smithi148 ceph-mon[44520]: from='osd.4 v1:172.21.15.148:6800/4230909898' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:52:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:29 smithi148 ceph-mon[44520]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:52:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:29 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:52:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:29 smithi148 ceph-mon[44520]: from='osd.4 ' entity='osd.4' 2024-04-23T16:52:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:29 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/481489041' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:52:29.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:29 smithi148 ceph-mon[44520]: from='client.27262 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:52:29.197 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:52:29.214 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:52:29.224 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:29.248 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:52:29.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:29.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:52:29.257 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:52:29.257 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:52:29.292 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:29 smithi103 ceph-mon[39629]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T16:52:29.292 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:29 smithi103 ceph-mon[39629]: osdmap e727: 12 total, 11 up, 7 in 2024-04-23T16:52:29.292 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:29 smithi103 ceph-mon[39629]: from='osd.4 v1:172.21.15.148:6800/4230909898' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:52:29.292 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:29 smithi103 ceph-mon[39629]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:52:29.292 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:29 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:52:29.292 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:29 smithi103 ceph-mon[39629]: from='osd.4 ' entity='osd.4' 2024-04-23T16:52:29.292 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:29 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/481489041' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:52:29.292 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:29 smithi103 ceph-mon[39629]: from='client.27262 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:52:29.352 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:52:29.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:29 smithi188 ceph-mon[44101]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T16:52:29.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:29 smithi188 ceph-mon[44101]: osdmap e727: 12 total, 11 up, 7 in 2024-04-23T16:52:29.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:29 smithi188 ceph-mon[44101]: from='osd.4 v1:172.21.15.148:6800/4230909898' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:52:29.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:29 smithi188 ceph-mon[44101]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-04-23T16:52:29.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:29 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:52:29.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:29 smithi188 ceph-mon[44101]: from='osd.4 ' entity='osd.4' 2024-04-23T16:52:29.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:29 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/481489041' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:52:29.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:29 smithi188 ceph-mon[44101]: from='client.27262 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:52:29.396 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:29.420 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:52:29.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:29.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:52:29.489 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:52:29.524 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:52:29.565 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:29.821 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:52:29.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:29.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:52:29.925 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:52:29.954 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:29.975 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:52:29.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:29.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:52:30.079 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:52:30.094 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 61 16 121117 121101 124.019 82.5 0.00719354 0.00806155 2024-04-23T16:52:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:30 smithi103 ceph-mon[39629]: pgmap v2014: 27 pgs: 27 active+clean; 7.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-23T16:52:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:30 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:52:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:30 smithi103 ceph-mon[39629]: from='client.27262 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:52:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:30 smithi103 ceph-mon[39629]: osd.4 v1:172.21.15.148:6800/4230909898 boot 2024-04-23T16:52:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:30 smithi103 ceph-mon[39629]: osdmap e728: 12 total, 12 up, 7 in 2024-04-23T16:52:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:30 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:52:30.118 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:30.151 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:52:30.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:30.160 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:52:30.261 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:52:30.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:30 smithi188 ceph-mon[44101]: pgmap v2014: 27 pgs: 27 active+clean; 7.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-23T16:52:30.296 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:30 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:52:30.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:30 smithi188 ceph-mon[44101]: from='client.27262 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:52:30.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:30 smithi188 ceph-mon[44101]: osd.4 v1:172.21.15.148:6800/4230909898 boot 2024-04-23T16:52:30.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:30 smithi188 ceph-mon[44101]: osdmap e728: 12 total, 12 up, 7 in 2024-04-23T16:52:30.297 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:30 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:52:30.305 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:30.333 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:52:30.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:30.336 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:52:30.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:30 smithi148 ceph-mon[44520]: pgmap v2014: 27 pgs: 27 active+clean; 7.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-23T16:52:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:30 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:52:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:30 smithi148 ceph-mon[44520]: from='client.27262 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:52:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:30 smithi148 ceph-mon[44520]: osd.4 v1:172.21.15.148:6800/4230909898 boot 2024-04-23T16:52:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:30 smithi148 ceph-mon[44520]: osdmap e728: 12 total, 12 up, 7 in 2024-04-23T16:52:30.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:30 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T16:52:30.438 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:52:30.469 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:31.094 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 62 16 122396 122380 123.308 79.9375 0.0215427 0.00810796 2024-04-23T16:52:31.466 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:52:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:31.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:52:31.497 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-23T16:52:31.571 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:52:31.622 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:31.731 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-23T16:52:31.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:31.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:31.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:31 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:52:31.851 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:31 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:52:31.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:31 smithi103 ceph-mon[39629]: osdmap e729: 12 total, 12 up, 7 in 2024-04-23T16:52:31.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:31 smithi103 ceph-mon[39629]: 9.8 scrub starts 2024-04-23T16:52:31.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:31 smithi103 ceph-mon[39629]: pgmap v2017: 27 pgs: 27 active+clean; 7.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-04-23T16:52:31.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:31 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:52:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:31 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:52:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:31 smithi148 ceph-mon[44520]: osdmap e729: 12 total, 12 up, 7 in 2024-04-23T16:52:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:31 smithi148 ceph-mon[44520]: 9.8 scrub starts 2024-04-23T16:52:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:31 smithi148 ceph-mon[44520]: pgmap v2017: 27 pgs: 27 active+clean; 7.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-04-23T16:52:31.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:31.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:31.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:31 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:52:31.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:31 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:52:31.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:31 smithi188 ceph-mon[44101]: osdmap e729: 12 total, 12 up, 7 in 2024-04-23T16:52:31.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:31 smithi188 ceph-mon[44101]: 9.8 scrub starts 2024-04-23T16:52:31.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:31 smithi188 ceph-mon[44101]: pgmap v2017: 27 pgs: 27 active+clean; 7.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-04-23T16:52:32.023 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:52:32.078 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:52:32.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:32.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:52:32.093 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 63 16 123549 123533 122.495 72.0625 0.0264182 0.00816126 2024-04-23T16:52:32.186 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:52:32.242 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:32.272 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:52:32.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:32.275 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:52:32.377 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:52:32.406 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:32.429 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:52:32.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:32.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:52:32.534 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:52:32.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:32 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T16:52:32.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:32 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T16:52:32.565 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:32.565 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:32 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T16:52:32.565 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:32 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T16:52:32.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:32 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T16:52:32.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:32 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T16:52:32.626 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:52:32.691 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:52:32.715 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:52:32.744 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:52:32.769 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:52:32.801 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:52:32.822 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:52:32.845 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:52:32.875 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:52:32.900 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:52:32.945 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:52:32.971 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:52:32.999 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:52:33.000 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:52:33.000 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:52:33.000 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:52:33.000 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:52:33.000 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:52:33.001 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:52:33.001 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:52:33.001 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:52:33.001 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:52:33.001 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:52:33.001 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:52:33.001 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:52:33.094 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 64 16 124707 124691 121.712 72.375 0.0123436 0.00821414 2024-04-23T16:52:33.096 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:52:33.416 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:52:33.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:33.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:52:33.522 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:52:33.557 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:33.731 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:33 smithi148 ceph-mon[44520]: 9.8 scrub ok 2024-04-23T16:52:33.731 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:33 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3442818971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:52:33.731 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:33 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.732 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:33 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.732 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:33 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.732 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:33 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.732 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:33 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.732 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:33 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.732 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:33 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.732 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:33 smithi148 ceph-mon[44520]: pgmap v2018: 27 pgs: 27 active+clean; 7.4 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-23T16:52:33.732 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:33 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.732 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:33 smithi148 ceph-mon[44520]: 9.1 scrub starts 2024-04-23T16:52:33.732 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:33 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:33 smithi103 ceph-mon[39629]: 9.8 scrub ok 2024-04-23T16:52:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:33 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3442818971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:52:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:33 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:33 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:33 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:33 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:33 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:33 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:33 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:33 smithi103 ceph-mon[39629]: pgmap v2018: 27 pgs: 27 active+clean; 7.4 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-23T16:52:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:33 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:33 smithi103 ceph-mon[39629]: 9.1 scrub starts 2024-04-23T16:52:33.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:33 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:33 smithi188 ceph-mon[44101]: 9.8 scrub ok 2024-04-23T16:52:33.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:33 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3442818971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:52:33.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:33 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:33 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:33 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:33 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:33 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:33 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:33 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:33 smithi188 ceph-mon[44101]: pgmap v2018: 27 pgs: 27 active+clean; 7.4 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-23T16:52:33.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:33 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:33.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:33 smithi188 ceph-mon[44101]: 9.1 scrub starts 2024-04-23T16:52:33.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:33 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:34.066 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:52:34.066 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:52:34.066 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2024-04-23T16:52:34.066 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:52:34.094 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 65 16 125843 125827 120.932 71 0.0049727 0.008267 2024-04-23T16:52:34.703 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:34 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:34.703 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:34 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:34.703 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:34 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:34.704 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:34 smithi148 ceph-mon[44520]: 9.1 scrub ok 2024-04-23T16:52:34.704 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:34 smithi148 ceph-mon[44520]: osdmap e730: 12 total, 12 up, 7 in 2024-04-23T16:52:34.721 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:52:34.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:34.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:52:34.733 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:52:34.734 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-04-23T16:52:34.734 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd in 2 2024-04-23T16:52:34.812 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:34 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:34.812 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:34 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:34.812 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:34 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:34.813 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:34 smithi103 ceph-mon[39629]: 9.1 scrub ok 2024-04-23T16:52:34.813 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:34 smithi103 ceph-mon[39629]: osdmap e730: 12 total, 12 up, 7 in 2024-04-23T16:52:34.828 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:52:34.877 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:34.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:34 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:34.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:34 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:34.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:34 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:34.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:34 smithi188 ceph-mon[44101]: 9.1 scrub ok 2024-04-23T16:52:34.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:34 smithi188 ceph-mon[44101]: osdmap e730: 12 total, 12 up, 7 in 2024-04-23T16:52:35.094 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 66 16 126985 126969 120.182 71.375 0.00528204 0.00831902 2024-04-23T16:52:35.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:35 smithi103 ceph-mon[39629]: pgmap v2020: 27 pgs: 27 active+clean; 7.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-04-23T16:52:35.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:35 smithi148 ceph-mon[44520]: pgmap v2020: 27 pgs: 27 active+clean; 7.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-04-23T16:52:35.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:35 smithi188 ceph-mon[44101]: pgmap v2020: 27 pgs: 27 active+clean; 7.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-04-23T16:52:36.095 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 67 16 128086 128070 119.413 68.8125 0.0163432 0.00837277 2024-04-23T16:52:36.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:36 smithi103 ceph-mon[39629]: 9.f scrub starts 2024-04-23T16:52:36.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:36 smithi103 ceph-mon[39629]: 9.5 scrub starts 2024-04-23T16:52:36.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:36 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:52:36.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:36 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:52:36.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:36 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:52:36.880 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:36 smithi148 ceph-mon[44520]: 9.f scrub starts 2024-04-23T16:52:36.880 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:36 smithi148 ceph-mon[44520]: 9.5 scrub starts 2024-04-23T16:52:36.880 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:36 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:52:36.880 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:36 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:52:36.880 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:36 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:52:36.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:36 smithi188 ceph-mon[44101]: 9.f scrub starts 2024-04-23T16:52:36.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:36 smithi188 ceph-mon[44101]: 9.5 scrub starts 2024-04-23T16:52:36.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:36 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:52:36.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:36 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:52:36.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:36 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:52:37.096 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 68 16 129093 129077 118.582 62.9375 0.0135118 0.00843176 2024-04-23T16:52:37.349 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:52:37.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:37.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:52:37.461 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:52:37.492 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:37.513 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:52:37.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:37.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:52:37.617 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:52:37.645 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:37.666 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:52:37.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:37.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:52:37.672 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:52:37.732 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:52:37.732 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:52:37.732 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-23T16:52:37.732 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:52:37.733 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:52:37.772 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:52:37.808 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:37 smithi103 ceph-mon[39629]: 9.a scrub starts 2024-04-23T16:52:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:37 smithi103 ceph-mon[39629]: 9.f scrub ok 2024-04-23T16:52:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:37 smithi103 ceph-mon[39629]: 9.2 scrub starts 2024-04-23T16:52:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:37 smithi103 ceph-mon[39629]: 9.5 scrub ok 2024-04-23T16:52:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:37 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:52:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:37 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:52:37.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:37 smithi103 ceph-mon[39629]: pgmap v2021: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 7.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-23T16:52:37.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:37 smithi148 ceph-mon[44520]: 9.a scrub starts 2024-04-23T16:52:37.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:37 smithi148 ceph-mon[44520]: 9.f scrub ok 2024-04-23T16:52:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:37 smithi148 ceph-mon[44520]: 9.2 scrub starts 2024-04-23T16:52:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:37 smithi148 ceph-mon[44520]: 9.5 scrub ok 2024-04-23T16:52:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:37 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:52:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:37 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:52:37.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:37 smithi148 ceph-mon[44520]: pgmap v2021: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 7.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-23T16:52:37.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:37 smithi188 ceph-mon[44101]: 9.a scrub starts 2024-04-23T16:52:37.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:37 smithi188 ceph-mon[44101]: 9.f scrub ok 2024-04-23T16:52:37.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:37 smithi188 ceph-mon[44101]: 9.2 scrub starts 2024-04-23T16:52:37.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:37 smithi188 ceph-mon[44101]: 9.5 scrub ok 2024-04-23T16:52:37.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:37 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:52:37.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:37 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:52:37.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:37 smithi188 ceph-mon[44101]: pgmap v2021: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 7.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-23T16:52:38.099 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 69 16 130130 130114 117.797 64.8125 0.0166729 0.00848735 2024-04-23T16:52:38.588 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:38 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:52:38.588 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:38 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:52:38.588 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:38 smithi103 ceph-mon[39629]: 1.9 scrub starts 2024-04-23T16:52:38.588 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:38 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:52:38.589 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:38 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:52:38.589 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:38 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:52:38.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:38 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:52:38.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:38 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:52:38.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:38 smithi148 ceph-mon[44520]: 1.9 scrub starts 2024-04-23T16:52:38.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:38 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:52:38.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:38 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:52:38.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:38 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:52:38.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:38 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:52:38.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:38 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:52:38.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:38 smithi188 ceph-mon[44101]: 1.9 scrub starts 2024-04-23T16:52:38.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:38 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:52:38.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:38 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:52:38.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:38 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:52:39.103 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 70 16 131293 131277 117.15 72.6875 0.0081827 0.00853456 2024-04-23T16:52:39.106 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:52:39.368 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:52:39.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:39.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:52:39.389 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-23T16:52:39.481 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:52:39.519 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:39.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:39 smithi148 ceph-mon[44520]: 9.a scrub ok 2024-04-23T16:52:39.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:39 smithi148 ceph-mon[44520]: 1.9 scrub ok 2024-04-23T16:52:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:39 smithi148 ceph-mon[44520]: 9.2 scrub ok 2024-04-23T16:52:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:39 smithi148 ceph-mon[44520]: pgmap v2022: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 7.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-23T16:52:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:39 smithi148 ceph-mon[44520]: 9.c scrub starts 2024-04-23T16:52:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:39 smithi148 ceph-mon[44520]: 9.e scrub starts 2024-04-23T16:52:39.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:39 smithi188 ceph-mon[44101]: 9.a scrub ok 2024-04-23T16:52:39.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:39 smithi188 ceph-mon[44101]: 1.9 scrub ok 2024-04-23T16:52:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:39 smithi188 ceph-mon[44101]: 9.2 scrub ok 2024-04-23T16:52:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:39 smithi188 ceph-mon[44101]: pgmap v2022: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 7.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-23T16:52:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:39 smithi188 ceph-mon[44101]: 9.c scrub starts 2024-04-23T16:52:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:39 smithi188 ceph-mon[44101]: 9.e scrub starts 2024-04-23T16:52:40.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:39 smithi103 ceph-mon[39629]: 9.a scrub ok 2024-04-23T16:52:40.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:39 smithi103 ceph-mon[39629]: 1.9 scrub ok 2024-04-23T16:52:40.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:39 smithi103 ceph-mon[39629]: 9.2 scrub ok 2024-04-23T16:52:40.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:39 smithi103 ceph-mon[39629]: pgmap v2022: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 7.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-23T16:52:40.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:39 smithi103 ceph-mon[39629]: 9.c scrub starts 2024-04-23T16:52:40.102 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:39 smithi103 ceph-mon[39629]: 9.e scrub starts 2024-04-23T16:52:40.102 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 71 16 132205 132189 116.302 57 0.010229 0.00859705 2024-04-23T16:52:40.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:40 smithi148 ceph-mon[44520]: 9.9 scrub starts 2024-04-23T16:52:40.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:40 smithi148 ceph-mon[44520]: 9.b scrub starts 2024-04-23T16:52:40.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:40 smithi148 ceph-mon[44520]: 9.e scrub ok 2024-04-23T16:52:40.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:40 smithi148 ceph-mon[44520]: 9.e scrub starts 2024-04-23T16:52:40.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:40 smithi188 ceph-mon[44101]: 9.9 scrub starts 2024-04-23T16:52:40.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:40 smithi188 ceph-mon[44101]: 9.b scrub starts 2024-04-23T16:52:40.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:40 smithi188 ceph-mon[44101]: 9.e scrub ok 2024-04-23T16:52:40.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:40 smithi188 ceph-mon[44101]: 9.e scrub starts 2024-04-23T16:52:41.085 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:52:41.087 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:40 smithi103 ceph-mon[39629]: 9.9 scrub starts 2024-04-23T16:52:41.087 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:40 smithi103 ceph-mon[39629]: 9.b scrub starts 2024-04-23T16:52:41.087 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:40 smithi103 ceph-mon[39629]: 9.e scrub ok 2024-04-23T16:52:41.087 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:40 smithi103 ceph-mon[39629]: 9.e scrub starts 2024-04-23T16:52:41.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:41.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:52:41.102 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 72 16 133134 133118 115.494 58.0625 0.00632607 0.00865641 2024-04-23T16:52:41.189 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:52:41.217 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:41.240 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:52:41.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:41.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:52:41.344 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:52:41.392 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:41.664 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:41 smithi103 ceph-mon[39629]: 9.c scrub ok 2024-04-23T16:52:41.665 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:41 smithi103 ceph-mon[39629]: 9.9 scrub ok 2024-04-23T16:52:41.665 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:41 smithi103 ceph-mon[39629]: pgmap v2023: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 8.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-23T16:52:41.665 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:41 smithi103 ceph-mon[39629]: 9.e scrub ok 2024-04-23T16:52:41.672 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:52:41.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:41.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:52:41.782 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:52:41.810 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:52:41.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:41.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:52:41.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:41 smithi148 ceph-mon[44520]: 9.c scrub ok 2024-04-23T16:52:41.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:41 smithi148 ceph-mon[44520]: 9.9 scrub ok 2024-04-23T16:52:41.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:41 smithi148 ceph-mon[44520]: pgmap v2023: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 8.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-23T16:52:41.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:41 smithi148 ceph-mon[44520]: 9.e scrub ok 2024-04-23T16:52:41.935 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:52:41.962 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:41.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:41 smithi188 ceph-mon[44101]: 9.c scrub ok 2024-04-23T16:52:41.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:41 smithi188 ceph-mon[44101]: 9.9 scrub ok 2024-04-23T16:52:41.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:41 smithi188 ceph-mon[44101]: pgmap v2023: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 8.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-23T16:52:41.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:41 smithi188 ceph-mon[44101]: 9.e scrub ok 2024-04-23T16:52:41.982 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:52:41.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:41.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:52:42.087 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:52:42.102 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 73 16 134297 134281 114.907 72.6875 0.00624122 0.00870022 2024-04-23T16:52:42.115 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:42.121 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:52:42.137 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:52:42.141 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:52:42.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:42.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:52:42.249 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:52:42.284 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:42.314 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:52:42.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:42.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:52:42.418 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:52:42.448 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:42.471 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:52:42.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:42.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:52:42.577 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:52:42.625 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:42.636 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:42 smithi103 ceph-mon[39629]: 9.b scrub ok 2024-04-23T16:52:42.636 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:42 smithi103 ceph-mon[39629]: 9.d scrub starts 2024-04-23T16:52:42.636 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:42 smithi103 ceph-mon[39629]: 1.5 scrub starts 2024-04-23T16:52:42.636 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:42 smithi103 ceph-mon[39629]: 1.5 scrub ok 2024-04-23T16:52:42.666 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:52:42.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:42.670 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:52:42.771 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:52:42.815 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:42.857 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:52:42.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:42.861 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:52:42.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:42 smithi148 ceph-mon[44520]: 9.b scrub ok 2024-04-23T16:52:42.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:42 smithi148 ceph-mon[44520]: 9.d scrub starts 2024-04-23T16:52:42.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:42 smithi148 ceph-mon[44520]: 1.5 scrub starts 2024-04-23T16:52:42.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:42 smithi148 ceph-mon[44520]: 1.5 scrub ok 2024-04-23T16:52:42.963 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:52:42.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:42 smithi188 ceph-mon[44101]: 9.b scrub ok 2024-04-23T16:52:42.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:42 smithi188 ceph-mon[44101]: 9.d scrub starts 2024-04-23T16:52:42.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:42 smithi188 ceph-mon[44101]: 1.5 scrub starts 2024-04-23T16:52:42.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:42 smithi188 ceph-mon[44101]: 1.5 scrub ok 2024-04-23T16:52:43.020 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:43.103 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 74 16 135402 135386 114.286 69.0625 0.018388 0.00874816 2024-04-23T16:52:43.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:43 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T16:52:43.189+0000 7f135b1c7700 -1 mon.a@0(leader).osd e731 definitely_dead 0 2024-04-23T16:52:43.636 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:52:43.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:43 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:52:43.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:43 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:52:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:43 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:52:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:43 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:52:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:43 smithi188 ceph-mon[44101]: osdmap e731: 12 total, 12 up, 7 in 2024-04-23T16:52:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:43 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:52:43.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:43 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:52:43.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:43 smithi188 ceph-mon[44101]: pgmap v2025: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 8.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-04-23T16:52:43.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:43 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1701621348' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-23T16:52:43.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:43 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3051612002' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:52:43.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:43 smithi188 ceph-mon[44101]: from='client.37040 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:52:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:43 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:52:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:43 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:52:44.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:43 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:52:44.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:43 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:52:44.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:43 smithi103 ceph-mon[39629]: osdmap e731: 12 total, 12 up, 7 in 2024-04-23T16:52:44.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:43 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:52:44.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:43 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:52:44.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:43 smithi103 ceph-mon[39629]: pgmap v2025: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 8.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-04-23T16:52:44.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:43 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1701621348' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-23T16:52:44.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:43 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3051612002' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:52:44.101 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:43 smithi103 ceph-mon[39629]: from='client.37040 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:52:44.104 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 75 16 136201 136185 113.428 49.9375 0.00415205 0.00877021 2024-04-23T16:52:44.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:43 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:52:44.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:43 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:52:44.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:43 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:52:44.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:43 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:52:44.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:43 smithi148 ceph-mon[44520]: osdmap e731: 12 total, 12 up, 7 in 2024-04-23T16:52:44.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:43 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:52:44.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:43 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:52:44.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:43 smithi148 ceph-mon[44520]: pgmap v2025: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 8.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-04-23T16:52:44.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:43 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1701621348' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-23T16:52:44.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:43 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3051612002' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:52:44.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:43 smithi148 ceph-mon[44520]: from='client.37040 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:52:44.899 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:44 smithi103 ceph-mon[39629]: 9.d scrub ok 2024-04-23T16:52:44.900 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:44 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:52:44.900 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:44 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:52:44.900 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:44 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1701621348' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-04-23T16:52:44.900 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:44 smithi103 ceph-mon[39629]: from='client.37040 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:52:44.900 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:44 smithi103 ceph-mon[39629]: osdmap e732: 12 total, 12 up, 8 in 2024-04-23T16:52:44.900 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:44 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:52:44.900 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:44 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:52:44.900 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:44 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:52:44.900 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:44 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:52:44.900 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:44 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:52:44.900 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:44 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:52:44.901 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:44 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:52:44.901 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:44 smithi103 ceph-mon[39629]: osdmap e733: 12 total, 12 up, 8 in 2024-04-23T16:52:44.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:44 smithi188 ceph-mon[44101]: 9.d scrub ok 2024-04-23T16:52:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:44 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:52:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:44 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:52:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:44 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1701621348' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-04-23T16:52:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:44 smithi188 ceph-mon[44101]: from='client.37040 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:52:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:44 smithi188 ceph-mon[44101]: osdmap e732: 12 total, 12 up, 8 in 2024-04-23T16:52:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:44 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:52:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:44 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:52:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:44 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:52:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:44 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:52:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:44 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:52:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:44 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:52:44.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:44 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:52:44.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:44 smithi188 ceph-mon[44101]: osdmap e733: 12 total, 12 up, 8 in 2024-04-23T16:52:45.104 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 76 16 136621 136605 112.281 26.25 0.00668571 0.00890451 2024-04-23T16:52:45.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:44 smithi148 ceph-mon[44520]: 9.d scrub ok 2024-04-23T16:52:45.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:44 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:52:45.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:44 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:52:45.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:44 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1701621348' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-04-23T16:52:45.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:44 smithi148 ceph-mon[44520]: from='client.37040 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:52:45.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:44 smithi148 ceph-mon[44520]: osdmap e732: 12 total, 12 up, 8 in 2024-04-23T16:52:45.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:44 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:52:45.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:44 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:52:45.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:44 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:52:45.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:44 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:52:45.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:44 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:52:45.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:44 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:52:45.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:44 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:52:45.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:44 smithi148 ceph-mon[44520]: osdmap e733: 12 total, 12 up, 8 in 2024-04-23T16:52:45.481 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:52:45.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:45.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:52:45.594 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:52:45.629 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:45.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:45 smithi103 ceph-mon[39629]: pgmap v2028: 27 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 25 active+clean; 8.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 62 MiB/s wr, 998 op/s 2024-04-23T16:52:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:45 smithi103 ceph-mon[39629]: osdmap e734: 12 total, 12 up, 8 in 2024-04-23T16:52:45.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:45 smithi188 ceph-mon[44101]: pgmap v2028: 27 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 25 active+clean; 8.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 62 MiB/s wr, 998 op/s 2024-04-23T16:52:45.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:45 smithi188 ceph-mon[44101]: osdmap e734: 12 total, 12 up, 8 in 2024-04-23T16:52:46.104 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 77 16 137615 137599 111.63 62.125 0.00711253 0.00895618 2024-04-23T16:52:46.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:45 smithi148 ceph-mon[44520]: pgmap v2028: 27 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 25 active+clean; 8.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 62 MiB/s wr, 998 op/s 2024-04-23T16:52:46.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:45 smithi148 ceph-mon[44520]: osdmap e734: 12 total, 12 up, 8 in 2024-04-23T16:52:46.824 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:52:46.848 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:52:46.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:46 smithi103 ceph-mon[39629]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-23T16:52:46.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:46 smithi103 ceph-mon[39629]: osdmap e735: 12 total, 12 up, 8 in 2024-04-23T16:52:46.870 INFO:teuthology.orchestra.run.smithi103.stderr:marked in osd.2. 2024-04-23T16:52:46.883 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:52:46.908 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:52:46.935 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:52:46.971 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:52:46.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:46 smithi188 ceph-mon[44101]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-23T16:52:46.979 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:46 smithi188 ceph-mon[44101]: osdmap e735: 12 total, 12 up, 8 in 2024-04-23T16:52:46.996 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:52:47.026 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:52:47.054 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:52:47.086 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:52:47.086 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:52:47.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:47.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:52:47.104 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 78 15 138723 138708 111.088 69.3125 0.015172 0.00900011 2024-04-23T16:52:47.114 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:52:47.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:46 smithi148 ceph-mon[44520]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-23T16:52:47.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:46 smithi148 ceph-mon[44520]: osdmap e735: 12 total, 12 up, 8 in 2024-04-23T16:52:47.144 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:52:47.147 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:52:47.147 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:52:47.147 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:52:47.148 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:52:47.148 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:52:47.148 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:52:47.148 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:52:47.148 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:52:47.148 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:52:47.148 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:52:47.148 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:52:47.148 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:52:47.194 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:52:47.242 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:47.798 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:52:47.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:47.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:52:47.829 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-04-23T16:52:47.904 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:52:47.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:47 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:52:47.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:47 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:52:47.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:47 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1592670712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:52:47.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:47 smithi103 ceph-mon[39629]: pgmap v2031: 27 pgs: 2 remapped+peering, 1 active+recovering, 24 active+clean; 8.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 62 MiB/s wr, 995 op/s; 8561/273606 objects misplaced (3.129%); 319 KiB/s, 5 objects/s recovering 2024-04-23T16:52:47.939 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.940 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.940 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.940 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.940 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.940 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.940 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:47 smithi103 ceph-mon[39629]: osdmap e736: 12 total, 12 up, 8 in 2024-04-23T16:52:47.940 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:47 smithi103 ceph-mon[39629]: 9.4 scrub starts 2024-04-23T16:52:47.946 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:47 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:52:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:47 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:52:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:47 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1592670712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:52:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:47 smithi188 ceph-mon[44101]: pgmap v2031: 27 pgs: 2 remapped+peering, 1 active+recovering, 24 active+clean; 8.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 62 MiB/s wr, 995 op/s; 8561/273606 objects misplaced (3.129%); 319 KiB/s, 5 objects/s recovering 2024-04-23T16:52:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:47 smithi188 ceph-mon[44101]: osdmap e736: 12 total, 12 up, 8 in 2024-04-23T16:52:47.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:47 smithi188 ceph-mon[44101]: 9.4 scrub starts 2024-04-23T16:52:47.980 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:52:47.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:47.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:52:48.090 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:52:48.104 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 79 16 139770 139754 110.509 65.375 0.01108 0.00904696 2024-04-23T16:52:48.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:47 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:52:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:47 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:52:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:47 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1592670712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:52:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:47 smithi148 ceph-mon[44520]: pgmap v2031: 27 pgs: 2 remapped+peering, 1 active+recovering, 24 active+clean; 8.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 62 MiB/s wr, 995 op/s; 8561/273606 objects misplaced (3.129%); 319 KiB/s, 5 objects/s recovering 2024-04-23T16:52:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:48.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:47 smithi148 ceph-mon[44520]: osdmap e736: 12 total, 12 up, 8 in 2024-04-23T16:52:48.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:47 smithi148 ceph-mon[44520]: 9.4 scrub starts 2024-04-23T16:52:48.136 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:48.167 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:52:48.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:48.170 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:52:48.271 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:52:48.307 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:48.335 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:52:48.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:48.341 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:52:48.441 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:52:48.470 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:48.494 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:52:48.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:48.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:52:48.584 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:52:48.597 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:52:48.634 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:48.670 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:52:48.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:48.677 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:52:48.778 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:52:48.836 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:48.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:48.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:48 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:48.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:48.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:48 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:49.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:48 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:49.104 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:2024-04-23 16:52:49.104332 min lat: 0.0010724 max lat: 1.08629 avg lat: 0.00909433 2024-04-23T16:52:49.104 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-23T16:52:49.104 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 80 16 140798 140782 109.931 64.25 0.0137376 0.00909433 2024-04-23T16:52:49.124 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:52:49.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:49.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:52:49.231 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:52:49.286 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:49.320 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:52:49.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:49.325 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:52:49.426 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:52:49.450 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:52:49.461 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:49.470 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:52:49.470 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:52:49.487 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:52:49.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:49.496 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:52:49.597 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:52:49.647 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:49.671 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:52:49.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:49.677 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:52:49.766 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:49 smithi103 ceph-mon[39629]: pgmap v2033: 27 pgs: 1 active+recovering, 26 active+clean; 8.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 61 MiB/s wr, 973 op/s; 8561/276384 objects misplaced (3.098%); 327 KiB/s, 5 objects/s recovering 2024-04-23T16:52:49.766 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:49 smithi103 ceph-mon[39629]: 9.4 scrub ok 2024-04-23T16:52:49.766 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:49 smithi103 ceph-mon[39629]: 9.0 scrub starts 2024-04-23T16:52:49.777 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:52:49.816 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:49.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:49 smithi188 ceph-mon[44101]: pgmap v2033: 27 pgs: 1 active+recovering, 26 active+clean; 8.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 61 MiB/s wr, 973 op/s; 8561/276384 objects misplaced (3.098%); 327 KiB/s, 5 objects/s recovering 2024-04-23T16:52:49.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:49 smithi188 ceph-mon[44101]: 9.4 scrub ok 2024-04-23T16:52:49.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:49 smithi188 ceph-mon[44101]: 9.0 scrub starts 2024-04-23T16:52:50.040 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:52:50.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:50.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:52:50.104 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 81 16 141822 141806 109.364 64 0.0100464 0.00914188 2024-04-23T16:52:50.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:49 smithi148 ceph-mon[44520]: pgmap v2033: 27 pgs: 1 active+recovering, 26 active+clean; 8.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 61 MiB/s wr, 973 op/s; 8561/276384 objects misplaced (3.098%); 327 KiB/s, 5 objects/s recovering 2024-04-23T16:52:50.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:49 smithi148 ceph-mon[44520]: 9.4 scrub ok 2024-04-23T16:52:50.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:49 smithi148 ceph-mon[44520]: 9.0 scrub starts 2024-04-23T16:52:50.146 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:52:50.194 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:50.343 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:52:50.402 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:52:50.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:50.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:52:50.508 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:52:50.545 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:50.572 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:52:50.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:50.578 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:52:50.663 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:50 smithi103 ceph-mon[39629]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-23T16:52:50.663 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:50 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:52:50.663 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:50 smithi103 ceph-mon[39629]: 1.a scrub starts 2024-04-23T16:52:50.663 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:50 smithi103 ceph-mon[39629]: 1.a scrub ok 2024-04-23T16:52:50.679 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:52:50.724 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:50.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:50 smithi188 ceph-mon[44101]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-23T16:52:50.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:50 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:52:50.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:50 smithi188 ceph-mon[44101]: 1.a scrub starts 2024-04-23T16:52:50.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:50 smithi188 ceph-mon[44101]: 1.a scrub ok 2024-04-23T16:52:50.980 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:52:50.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:50.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:52:51.026 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-23T16:52:51.089 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:52:51.104 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 82 15 142867 142852 108.828 65.375 0.00491127 0.00918653 2024-04-23T16:52:51.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:50 smithi148 ceph-mon[44520]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-23T16:52:51.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:50 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:52:51.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:50 smithi148 ceph-mon[44520]: 1.a scrub starts 2024-04-23T16:52:51.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:50 smithi148 ceph-mon[44520]: 1.a scrub ok 2024-04-23T16:52:51.150 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:51.519 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:52:51.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:51.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:52:51.560 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:52:51.561 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:52:51.623 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:52:51.673 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:51.699 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:52:51.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:51.703 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:52:51.804 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:52:51.826 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:51 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:52:51.826 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:51 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:52:51.826 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:51 smithi103 ceph-mon[39629]: pgmap v2034: 27 pgs: 1 active+recovering, 26 active+clean; 8.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 58 MiB/s wr, 927 op/s; 8453/280356 objects misplaced (3.015%); 1.4 MiB/s, 21 objects/s recovering 2024-04-23T16:52:51.826 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:51 smithi103 ceph-mon[39629]: 9.0 scrub ok 2024-04-23T16:52:51.833 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:51.853 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:52:51.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:51.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:52:51.958 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:52:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:51 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:52:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:51 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:52:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:51 smithi188 ceph-mon[44101]: pgmap v2034: 27 pgs: 1 active+recovering, 26 active+clean; 8.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 58 MiB/s wr, 927 op/s; 8453/280356 objects misplaced (3.015%); 1.4 MiB/s, 21 objects/s recovering 2024-04-23T16:52:51.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:51 smithi188 ceph-mon[44101]: 9.0 scrub ok 2024-04-23T16:52:51.986 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:51 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:52:51.986 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:51 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:52:51.986 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:51 smithi148 ceph-mon[44520]: pgmap v2034: 27 pgs: 1 active+recovering, 26 active+clean; 8.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 58 MiB/s wr, 927 op/s; 8453/280356 objects misplaced (3.015%); 1.4 MiB/s, 21 objects/s recovering 2024-04-23T16:52:51.986 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:51 smithi148 ceph-mon[44520]: 9.0 scrub ok 2024-04-23T16:52:51.988 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:52.008 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:52:52.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:52.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:52:52.104 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 83 16 143922 143906 108.311 65.875 0.0154099 0.00923092 2024-04-23T16:52:52.115 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:52:52.146 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:52.169 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:52:52.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:52.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:52:52.275 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:52:52.304 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:52.324 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:52:52.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:52.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:52:52.428 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:52:52.479 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:52.829 INFO:tasks.thrashosds.thrasher:in_osds: [0, 6, 7, 8, 9, 11, 1, 2] out_osds: [10, 3, 5, 4] dead_osds: [] live_osds: [0, 5, 6, 7, 8, 9, 10, 11, 1, 2, 3, 4] 2024-04-23T16:52:52.829 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-23T16:52:52.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:52 smithi103 ceph-mon[39629]: osdmap e737: 12 total, 12 up, 8 in 2024-04-23T16:52:52.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:52 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:52:52.852 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:52 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:52:52.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:52 smithi188 ceph-mon[44101]: osdmap e737: 12 total, 12 up, 8 in 2024-04-23T16:52:52.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:52 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:52:52.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:52 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:52:53.104 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 84 16 144979 144963 107.808 66.0625 0.0164536 0.00927378 2024-04-23T16:52:53.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:52 smithi148 ceph-mon[44520]: osdmap e737: 12 total, 12 up, 8 in 2024-04-23T16:52:53.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:52 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:52:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:52 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:52:53.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:53 smithi188 ceph-mon[44101]: osdmap e738: 12 total, 12 up, 8 in 2024-04-23T16:52:53.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:53 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4046442104' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:52:53.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:53 smithi188 ceph-mon[44101]: from='client.27283 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:52:53.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:53 smithi188 ceph-mon[44101]: pgmap v2037: 27 pgs: 1 active+recovering, 26 active+clean; 8.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 59 MiB/s wr, 941 op/s; 8453/284918 objects misplaced (2.967%); 1.2 MiB/s, 18 objects/s recovering 2024-04-23T16:52:54.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:53 smithi103 ceph-mon[39629]: osdmap e738: 12 total, 12 up, 8 in 2024-04-23T16:52:54.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:53 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4046442104' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:52:54.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:53 smithi103 ceph-mon[39629]: from='client.27283 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:52:54.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:53 smithi103 ceph-mon[39629]: pgmap v2037: 27 pgs: 1 active+recovering, 26 active+clean; 8.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 59 MiB/s wr, 941 op/s; 8453/284918 objects misplaced (2.967%); 1.2 MiB/s, 18 objects/s recovering 2024-04-23T16:52:54.104 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 85 16 146047 146031 107.325 66.75 0.00525294 0.00931446 2024-04-23T16:52:54.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:53 smithi148 ceph-mon[44520]: osdmap e738: 12 total, 12 up, 8 in 2024-04-23T16:52:54.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:53 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4046442104' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:52:54.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:53 smithi148 ceph-mon[44520]: from='client.27283 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:52:54.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:53 smithi148 ceph-mon[44520]: pgmap v2037: 27 pgs: 1 active+recovering, 26 active+clean; 8.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 59 MiB/s wr, 941 op/s; 8453/284918 objects misplaced (2.967%); 1.2 MiB/s, 18 objects/s recovering 2024-04-23T16:52:54.314 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:52:54.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:54 smithi188 ceph-mon[44101]: 9.3 scrub starts 2024-04-23T16:52:54.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:54 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:52:54.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:54 smithi188 ceph-mon[44101]: from='client.27283 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:52:54.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:54 smithi188 ceph-mon[44101]: osdmap e739: 12 total, 12 up, 8 in 2024-04-23T16:52:55.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:54 smithi103 ceph-mon[39629]: 9.3 scrub starts 2024-04-23T16:52:55.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:54 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:52:55.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:54 smithi103 ceph-mon[39629]: from='client.27283 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:52:55.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:54 smithi103 ceph-mon[39629]: osdmap e739: 12 total, 12 up, 8 in 2024-04-23T16:52:55.104 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 86 16 147034 147018 106.795 61.6875 0.00809251 0.00936245 2024-04-23T16:52:55.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:54 smithi148 ceph-mon[44520]: 9.3 scrub starts 2024-04-23T16:52:55.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:54 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:52:55.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:54 smithi148 ceph-mon[44520]: from='client.27283 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:52:55.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:54 smithi148 ceph-mon[44520]: osdmap e739: 12 total, 12 up, 8 in 2024-04-23T16:52:55.348 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:52:55.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:55.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:52:55.452 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:52:55.508 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:55.541 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:52:55.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:55.547 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:52:55.648 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:52:55.675 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:55.699 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:52:55.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:55.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:52:55.804 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:52:55.850 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:55.851 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:55 smithi188 ceph-mon[44101]: pgmap v2039: 27 pgs: 1 active+recovering, 26 active+clean; 8.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 8453/288752 objects misplaced (2.927%); 1.1 MiB/s, 17 objects/s recovering 2024-04-23T16:52:55.876 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:52:55.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:55.881 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:52:55.982 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:52:56.034 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:56.056 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:52:56.057 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:55 smithi103 ceph-mon[39629]: pgmap v2039: 27 pgs: 1 active+recovering, 26 active+clean; 8.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 8453/288752 objects misplaced (2.927%); 1.1 MiB/s, 17 objects/s recovering 2024-04-23T16:52:56.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:56.059 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:52:56.104 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 87 16 148074 148058 106.314 65 0.0103735 0.00940448 2024-04-23T16:52:56.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:55 smithi148 ceph-mon[44520]: pgmap v2039: 27 pgs: 1 active+recovering, 26 active+clean; 8.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 8453/288752 objects misplaced (2.927%); 1.1 MiB/s, 17 objects/s recovering 2024-04-23T16:52:56.161 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:52:56.209 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:56.233 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:52:56.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:56.236 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:52:56.337 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:52:56.384 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:56.419 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:52:56.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:56.426 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:52:56.527 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:52:56.571 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:56.598 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:52:56.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:56.603 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:52:56.704 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:52:56.746 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:56.793 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:52:56.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:56.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:52:56.899 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:52:56.926 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:56.952 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:52:56.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:56.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:52:57.060 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:52:57.095 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:57.105 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 88 16 149102 149086 105.836 64.25 0.00575185 0.00944627 2024-04-23T16:52:57.133 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:52:57.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:57.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:52:57.241 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:52:57.281 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:57.360 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:52:57.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:57.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:52:57.464 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:52:57.524 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:57.567 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:52:57.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:57.576 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:52:57.677 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:52:57.719 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:57.928 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:52:57.997 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:52:58.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:58.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:52:58.095 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:52:58.106 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:52:58.107 INFO:tasks.radosbench.radosbench.2.smithi188.stdout: 89 16 150132 150116 105.371 64.375 0.00598141 0.00948916 2024-04-23T16:52:58.107 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:52:58.107 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:52:58.134 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:52:58.163 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:58.176 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:52:58.206 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:52:58.242 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:52:58.281 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:52:58.314 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:52:58.342 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:52:58.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:58 smithi103 ceph-mon[39629]: pgmap v2040: 27 pgs: 1 active+recovering, 26 active+clean; 9.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s; 8358/294004 objects misplaced (2.843%); 1012 KiB/s, 15 objects/s recovering 2024-04-23T16:52:58.368 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:52:58.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:58 smithi148 ceph-mon[44520]: pgmap v2040: 27 pgs: 1 active+recovering, 26 active+clean; 9.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s; 8358/294004 objects misplaced (2.843%); 1012 KiB/s, 15 objects/s recovering 2024-04-23T16:52:58.399 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:52:58.432 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:52:58.451 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:52:58.467 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:52:58.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:58 smithi188 ceph-mon[44101]: pgmap v2040: 27 pgs: 1 active+recovering, 26 active+clean; 9.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s; 8358/294004 objects misplaced (2.843%); 1012 KiB/s, 15 objects/s recovering 2024-04-23T16:52:58.489 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:52:58.489 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:52:58.489 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:52:58.489 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:52:58.489 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:52:58.489 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:52:58.490 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:52:58.490 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:52:58.490 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:52:58.490 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:52:58.490 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:52:58.490 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:52:58.541 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:52:58.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:58.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:52:58.647 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:52:58.699 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:58.724 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:52:58.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:52:58.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:52:58.832 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:52:58.867 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:52:59.105 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Total time run: 90.0358 2024-04-23T16:52:59.105 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Total writes made: 151130 2024-04-23T16:52:59.106 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Write size: 65536 2024-04-23T16:52:59.106 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Object size: 65536 2024-04-23T16:52:59.106 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Bandwidth (MB/sec): 104.91 2024-04-23T16:52:59.106 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev Bandwidth: 49.5669 2024-04-23T16:52:59.106 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max bandwidth (MB/sec): 298.188 2024-04-23T16:52:59.106 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min bandwidth (MB/sec): 26.25 2024-04-23T16:52:59.106 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Average IOPS: 1678 2024-04-23T16:52:59.106 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev IOPS: 793.07 2024-04-23T16:52:59.106 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max IOPS: 4771 2024-04-23T16:52:59.106 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min IOPS: 420 2024-04-23T16:52:59.106 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Average Latency(s): 0.00953035 2024-04-23T16:52:59.106 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Stddev Latency(s): 0.0134032 2024-04-23T16:52:59.106 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Max latency(s): 1.08629 2024-04-23T16:52:59.107 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Min latency(s): 0.0010724 2024-04-23T16:52:59.113 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Cleaning up (deleting benchmark objects) 2024-04-23T16:52:59.128 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:59 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1753542713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:52:59.128 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.128 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.128 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.129 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.129 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.129 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.129 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:59 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.129 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:59 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:52:59.129 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:52:59 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:52:59.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:59 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1753542713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:52:59.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:59 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:59 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:52:59.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:52:59 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:52:59.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:59 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1753542713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:52:59.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:59 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:52:59.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:59 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:52:59.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:52:59 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:52:59.598 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-23T16:53:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:00 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:00 smithi103 ceph-mon[39629]: pgmap v2041: 27 pgs: 1 active+recovering, 26 active+clean; 9.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 8358/297114 objects misplaced (2.813%); 834 KiB/s, 13 objects/s recovering 2024-04-23T16:53:00.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:00 smithi103 ceph-mon[39629]: osdmap e740: 12 total, 12 up, 8 in 2024-04-23T16:53:00.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:00.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:00.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:00.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:00.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:00 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:00.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:00 smithi148 ceph-mon[44520]: pgmap v2041: 27 pgs: 1 active+recovering, 26 active+clean; 9.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 8358/297114 objects misplaced (2.813%); 834 KiB/s, 13 objects/s recovering 2024-04-23T16:53:00.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:00 smithi148 ceph-mon[44520]: osdmap e740: 12 total, 12 up, 8 in 2024-04-23T16:53:00.427 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:53:00.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:00.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:53:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:00 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:00 smithi188 ceph-mon[44101]: pgmap v2041: 27 pgs: 1 active+recovering, 26 active+clean; 9.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 8358/297114 objects misplaced (2.813%); 834 KiB/s, 13 objects/s recovering 2024-04-23T16:53:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:00 smithi188 ceph-mon[44101]: osdmap e740: 12 total, 12 up, 8 in 2024-04-23T16:53:00.531 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:53:00.574 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:00.604 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:53:00.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:00.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:53:00.713 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:53:00.756 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:01.019 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:53:01.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:01.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:53:01.124 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:53:01.161 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:01.191 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:53:01.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:01.197 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:53:01.298 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:53:01.356 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:01.387 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:53:01.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:01.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:53:01.497 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:53:01.552 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:01.580 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:53:01.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:01.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:53:01.614 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:01.686 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:53:01.742 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:01.764 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:53:01.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:01.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:53:01.875 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:53:01.912 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:01.936 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:53:01.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:01.941 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:53:02.042 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:53:02.084 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:02.130 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:53:02.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:02.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:53:02.158 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:02 smithi103 ceph-mon[39629]: pgmap v2043: 27 pgs: 1 active+recovering, 26 active+clean; 9.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 55 MiB/s wr, 893 op/s; 8358/298616 objects misplaced (2.799%); 760 KiB/s, 11 objects/s recovering 2024-04-23T16:53:02.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:02 smithi188 ceph-mon[44101]: pgmap v2043: 27 pgs: 1 active+recovering, 26 active+clean; 9.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 55 MiB/s wr, 893 op/s; 8358/298616 objects misplaced (2.799%); 760 KiB/s, 11 objects/s recovering 2024-04-23T16:53:02.239 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:53:02.272 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:02.299 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:53:02.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:02.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:53:02.353 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:02.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:02 smithi148 ceph-mon[44520]: pgmap v2043: 27 pgs: 1 active+recovering, 26 active+clean; 9.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 55 MiB/s wr, 893 op/s; 8358/298616 objects misplaced (2.799%); 760 KiB/s, 11 objects/s recovering 2024-04-23T16:53:02.405 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:53:02.472 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:03.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:03 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:53:03.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:03 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:53:03.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:03 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/416618052' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:53:03.087 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.0 to 1 (10000) 2024-04-23T16:53:03.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:03 smithi103 ceph-mon[39629]: from='client.37064 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:53:03.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:03 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:53:03.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:03 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:53:03.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:03 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/416618052' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:53:03.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:03 smithi148 ceph-mon[44520]: from='client.37064 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:53:03.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:03 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:53:03.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:03 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:53:03.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:03 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/416618052' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:53:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:03 smithi188 ceph-mon[44101]: from='client.37064 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:53:03.661 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:53:04.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:04 smithi103 ceph-mon[39629]: pgmap v2044: 27 pgs: 1 active+recovering, 26 active+clean; 9.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 110 B/s rd, 51 MiB/s wr, 1.02k op/s; 8250/298078 objects misplaced (2.768%); 1.4 MiB/s, 21 objects/s recovering 2024-04-23T16:53:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:04 smithi103 ceph-mon[39629]: from='client.37064 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:53:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:04 smithi103 ceph-mon[39629]: osdmap e741: 12 total, 12 up, 8 in 2024-04-23T16:53:04.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:04 smithi148 ceph-mon[44520]: pgmap v2044: 27 pgs: 1 active+recovering, 26 active+clean; 9.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 110 B/s rd, 51 MiB/s wr, 1.02k op/s; 8250/298078 objects misplaced (2.768%); 1.4 MiB/s, 21 objects/s recovering 2024-04-23T16:53:04.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:04 smithi148 ceph-mon[44520]: from='client.37064 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:53:04.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:04 smithi148 ceph-mon[44520]: osdmap e741: 12 total, 12 up, 8 in 2024-04-23T16:53:04.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:04 smithi188 ceph-mon[44101]: pgmap v2044: 27 pgs: 1 active+recovering, 26 active+clean; 9.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 110 B/s rd, 51 MiB/s wr, 1.02k op/s; 8250/298078 objects misplaced (2.768%); 1.4 MiB/s, 21 objects/s recovering 2024-04-23T16:53:04.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:04 smithi188 ceph-mon[44101]: from='client.37064 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:53:04.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:04 smithi188 ceph-mon[44101]: osdmap e741: 12 total, 12 up, 8 in 2024-04-23T16:53:05.040 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:53:05.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:05.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:53:05.144 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:53:05.183 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:05.212 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:53:05.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:05.216 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:53:05.317 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:53:05.365 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:05.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:05 smithi103 ceph-mon[39629]: pgmap v2046: 27 pgs: 1 active+recovering, 26 active+clean; 9.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 127 B/s rd, 32 MiB/s wr, 991 op/s; 8250/294654 objects misplaced (2.800%); 863 KiB/s, 13 objects/s recovering 2024-04-23T16:53:05.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:05 smithi148 ceph-mon[44520]: pgmap v2046: 27 pgs: 1 active+recovering, 26 active+clean; 9.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 127 B/s rd, 32 MiB/s wr, 991 op/s; 8250/294654 objects misplaced (2.800%); 863 KiB/s, 13 objects/s recovering 2024-04-23T16:53:05.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:05 smithi188 ceph-mon[44101]: pgmap v2046: 27 pgs: 1 active+recovering, 26 active+clean; 9.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 127 B/s rd, 32 MiB/s wr, 991 op/s; 8250/294654 objects misplaced (2.800%); 863 KiB/s, 13 objects/s recovering 2024-04-23T16:53:06.530 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:53:06.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:06.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:53:06.562 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-23T16:53:06.634 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:53:06.692 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:07.266 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:53:07.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:07.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:53:07.333 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:07.333 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:53:07.333 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:53:07.333 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-23T16:53:07.333 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:53:07.371 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:53:07.380 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:07.399 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:07.432 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:53:07.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:07.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:53:07.543 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:53:07.576 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:07.605 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:53:07.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:07.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:53:07.712 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:53:07.751 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:08.051 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:53:08.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:08 smithi103 ceph-mon[39629]: pgmap v2047: 27 pgs: 1 active+recovering, 26 active+clean; 8.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 127 B/s rd, 20 MiB/s wr, 1.12k op/s; 8118/289546 objects misplaced (2.804%); 1.9 MiB/s, 29 objects/s recovering 2024-04-23T16:53:08.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:08 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/594733140' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:53:08.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:08 smithi103 ceph-mon[39629]: from='client.27295 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:53:08.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:08 smithi148 ceph-mon[44520]: pgmap v2047: 27 pgs: 1 active+recovering, 26 active+clean; 8.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 127 B/s rd, 20 MiB/s wr, 1.12k op/s; 8118/289546 objects misplaced (2.804%); 1.9 MiB/s, 29 objects/s recovering 2024-04-23T16:53:08.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:08 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/594733140' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:53:08.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:08 smithi148 ceph-mon[44520]: from='client.27295 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:53:08.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:08 smithi188 ceph-mon[44101]: pgmap v2047: 27 pgs: 1 active+recovering, 26 active+clean; 8.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 127 B/s rd, 20 MiB/s wr, 1.12k op/s; 8118/289546 objects misplaced (2.804%); 1.9 MiB/s, 29 objects/s recovering 2024-04-23T16:53:08.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:08 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/594733140' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:53:08.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:08 smithi188 ceph-mon[44101]: from='client.27295 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:53:09.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:09 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:53:09.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:09 smithi103 ceph-mon[39629]: from='client.27295 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:53:09.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:09 smithi103 ceph-mon[39629]: osdmap e742: 12 total, 12 up, 8 in 2024-04-23T16:53:09.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:09 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:53:09.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:09 smithi148 ceph-mon[44520]: from='client.27295 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:53:09.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:09 smithi148 ceph-mon[44520]: osdmap e742: 12 total, 12 up, 8 in 2024-04-23T16:53:09.410 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:53:09.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:09.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:53:09.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:09 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:53:09.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:09 smithi188 ceph-mon[44101]: from='client.27295 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:53:09.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:09 smithi188 ceph-mon[44101]: osdmap e742: 12 total, 12 up, 8 in 2024-04-23T16:53:09.515 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:53:09.551 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:10.175 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:10 smithi103 ceph-mon[39629]: pgmap v2049: 27 pgs: 1 active+recovering, 26 active+clean; 8.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 127 B/s rd, 13 MiB/s wr, 1.17k op/s; 8118/286522 objects misplaced (2.833%); 1.9 MiB/s, 29 objects/s recovering 2024-04-23T16:53:10.205 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-23T16:53:10.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:10 smithi148 ceph-mon[44520]: pgmap v2049: 27 pgs: 1 active+recovering, 26 active+clean; 8.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 127 B/s rd, 13 MiB/s wr, 1.17k op/s; 8118/286522 objects misplaced (2.833%); 1.9 MiB/s, 29 objects/s recovering 2024-04-23T16:53:10.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:10 smithi188 ceph-mon[44101]: pgmap v2049: 27 pgs: 1 active+recovering, 26 active+clean; 8.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 127 B/s rd, 13 MiB/s wr, 1.17k op/s; 8118/286522 objects misplaced (2.833%); 1.9 MiB/s, 29 objects/s recovering 2024-04-23T16:53:11.122 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:53:11.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:11.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:53:11.232 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:53:11.260 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:11.289 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:53:11.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:11.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:53:11.395 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:53:11.423 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:11.448 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:53:11.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:11.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:53:11.553 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:53:11.586 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:11.751 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:11.761 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:53:11.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:11.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:53:11.869 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:53:11.921 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:11.994 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:53:12.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:12.002 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:53:12.103 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:53:12.158 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:12 smithi188 ceph-mon[44101]: pgmap v2050: 27 pgs: 1 active+recovering, 26 active+clean; 8.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 MiB/s wr, 854 op/s; 8118/284974 objects misplaced (2.849%); 1.0 MiB/s, 16 objects/s recovering 2024-04-23T16:53:12.160 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:12.162 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:12 smithi103 ceph-mon[39629]: pgmap v2050: 27 pgs: 1 active+recovering, 26 active+clean; 8.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 MiB/s wr, 854 op/s; 8118/284974 objects misplaced (2.849%); 1.0 MiB/s, 16 objects/s recovering 2024-04-23T16:53:12.187 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:53:12.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:12.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:53:12.291 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:53:12.319 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:12 smithi148 ceph-mon[44520]: pgmap v2050: 27 pgs: 1 active+recovering, 26 active+clean; 8.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 MiB/s wr, 854 op/s; 8118/284974 objects misplaced (2.849%); 1.0 MiB/s, 16 objects/s recovering 2024-04-23T16:53:12.320 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:12.343 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:53:12.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:12.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:53:12.447 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:53:12.486 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:12.794 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:53:12.794 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:53:12.794 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:53:12.794 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:53:12.794 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:53:12.794 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:53:12.794 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:53:12.794 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:53:12.794 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:53:12.794 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:53:12.794 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:53:12.794 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:53:12.795 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:53:12.795 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:53:12.795 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:53:12.795 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:53:12.795 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:53:12.795 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:53:12.795 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:53:12.795 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:53:12.795 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:53:12.795 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:53:12.795 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:53:12.795 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:53:13.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:13 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2056375187' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:53:13.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:13.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:13.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:13.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:13.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:13.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:13 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:13.071 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:53:13.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:13 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2056375187' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:53:13.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:13.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:13.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:13 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:13.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:13 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2056375187' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:53:13.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:13.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:13.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:13.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:13.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:13.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:13 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:14.250 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:53:14.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:14.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:53:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:14 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:14 smithi103 ceph-mon[39629]: pgmap v2051: 27 pgs: 1 active+recovering, 26 active+clean; 8.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 937 KiB/s wr, 955 op/s; 7812/279718 objects misplaced (2.793%); 2.8 MiB/s, 44 objects/s recovering 2024-04-23T16:53:14.361 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:53:14.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:14 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:14 smithi148 ceph-mon[44520]: pgmap v2051: 27 pgs: 1 active+recovering, 26 active+clean; 8.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 937 KiB/s wr, 955 op/s; 7812/279718 objects misplaced (2.793%); 2.8 MiB/s, 44 objects/s recovering 2024-04-23T16:53:14.389 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:14.413 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:53:14.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:14.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:53:14.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:14.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:14.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:14.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:14.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:14 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:14.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:14 smithi188 ceph-mon[44101]: pgmap v2051: 27 pgs: 1 active+recovering, 26 active+clean; 8.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 937 KiB/s wr, 955 op/s; 7812/279718 objects misplaced (2.793%); 2.8 MiB/s, 44 objects/s recovering 2024-04-23T16:53:14.518 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:53:14.549 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:14.580 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:53:14.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:14.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:53:14.689 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:53:14.720 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:14.798 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:53:14.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:14.805 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:53:14.906 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:53:14.936 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:14.960 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:53:14.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:14.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:53:15.065 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:53:15.108 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:15.153 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:53:15.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:15.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:53:15.261 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:53:15.288 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:15.293 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:15.323 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:53:15.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:15.329 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:53:15.430 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:53:15.457 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:15.477 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:53:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:15.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:53:15.584 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:53:15.614 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:15.636 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:53:15.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:15.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:53:15.741 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:53:15.777 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:15.807 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:53:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:15.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:53:15.917 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:53:15.947 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:15.972 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:53:15.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:15.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:53:16.076 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:53:16.076 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:53:16.077 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:53:16.085 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:16 smithi188 ceph-mon[44101]: pgmap v2052: 27 pgs: 1 active+recovering, 26 active+clean; 8.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 912 op/s; 7812/276396 objects misplaced (2.826%); 2.7 MiB/s, 43 objects/s recovering 2024-04-23T16:53:16.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:16 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1360291886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:53:16.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:16 smithi188 ceph-mon[44101]: from='client.37073 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:53:16.088 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.6 to 1 (10000) 2024-04-23T16:53:16.128 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:16.169 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:53:16.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:16.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:53:16.274 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:53:16.281 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:53:16.311 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:16 smithi103 ceph-mon[39629]: pgmap v2052: 27 pgs: 1 active+recovering, 26 active+clean; 8.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 912 op/s; 7812/276396 objects misplaced (2.826%); 2.7 MiB/s, 43 objects/s recovering 2024-04-23T16:53:16.311 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:16 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1360291886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:53:16.311 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:16 smithi103 ceph-mon[39629]: from='client.37073 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:53:16.318 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:16 smithi148 ceph-mon[44520]: pgmap v2052: 27 pgs: 1 active+recovering, 26 active+clean; 8.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 912 op/s; 7812/276396 objects misplaced (2.826%); 2.7 MiB/s, 43 objects/s recovering 2024-04-23T16:53:16.318 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:16 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1360291886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:53:16.318 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:16 smithi148 ceph-mon[44520]: from='client.37073 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:53:16.320 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:16.343 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:53:16.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:16.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:53:16.449 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:53:16.477 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:16.500 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:53:16.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:16.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:53:16.605 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:53:16.647 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:16.679 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:53:16.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:16.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:53:16.788 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:53:16.856 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:16.887 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:53:16.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:16.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:53:17.002 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:53:17.030 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:17.053 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:53:17.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:17.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:53:17.157 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:53:17.205 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:17 smithi188 ceph-mon[44101]: from='client.37073 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:53:17.206 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:17 smithi188 ceph-mon[44101]: osdmap e743: 12 total, 12 up, 8 in 2024-04-23T16:53:17.208 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:17.240 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:53:17.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:17.243 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:53:17.344 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:53:17.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:17 smithi103 ceph-mon[39629]: from='client.37073 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:53:17.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:17 smithi103 ceph-mon[39629]: osdmap e743: 12 total, 12 up, 8 in 2024-04-23T16:53:17.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:17 smithi148 ceph-mon[44520]: from='client.37073 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:53:17.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:17 smithi148 ceph-mon[44520]: osdmap e743: 12 total, 12 up, 8 in 2024-04-23T16:53:17.380 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:17.416 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:53:17.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:17.422 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:53:17.523 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:53:17.563 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:17.821 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:53:17.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:17.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:53:17.866 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-23T16:53:17.966 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:53:18.000 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:18.017 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:53:18.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:18.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:53:18.092 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:18 smithi103 ceph-mon[39629]: pgmap v2054: 27 pgs: 1 active+recovering, 26 active+clean; 8.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 924 op/s; 7525/271746 objects misplaced (2.769%); 4.1 MiB/s, 66 objects/s recovering 2024-04-23T16:53:18.125 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:53:18.170 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:18.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:18 smithi148 ceph-mon[44520]: pgmap v2054: 27 pgs: 1 active+recovering, 26 active+clean; 8.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 924 op/s; 7525/271746 objects misplaced (2.769%); 4.1 MiB/s, 66 objects/s recovering 2024-04-23T16:53:18.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:18 smithi188 ceph-mon[44101]: pgmap v2054: 27 pgs: 1 active+recovering, 26 active+clean; 8.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 924 op/s; 7525/271746 objects misplaced (2.769%); 4.1 MiB/s, 66 objects/s recovering 2024-04-23T16:53:18.795 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-23T16:53:19.451 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:19.462 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:53:19.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:19.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:53:19.517 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:19.569 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:53:19.598 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:19.620 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:53:19.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:19.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:53:19.724 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:53:19.751 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:19.773 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:53:19.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:19.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:53:19.876 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:53:19.922 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:20.120 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:53:20.190 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:53:20.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:20.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:53:20.311 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:53:20.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:20 smithi103 ceph-mon[39629]: pgmap v2055: 27 pgs: 1 active+recovering, 26 active+clean; 8.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 877 op/s; 7525/268966 objects misplaced (2.798%); 3.7 MiB/s, 59 objects/s recovering 2024-04-23T16:53:20.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:20 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1874515962' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:53:20.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:20 smithi103 ceph-mon[39629]: from='client.37079 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:53:20.346 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:20.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:20 smithi148 ceph-mon[44520]: pgmap v2055: 27 pgs: 1 active+recovering, 26 active+clean; 8.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 877 op/s; 7525/268966 objects misplaced (2.798%); 3.7 MiB/s, 59 objects/s recovering 2024-04-23T16:53:20.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:20 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1874515962' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:53:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:20 smithi148 ceph-mon[44520]: from='client.37079 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:53:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:20 smithi188 ceph-mon[44101]: pgmap v2055: 27 pgs: 1 active+recovering, 26 active+clean; 8.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 877 op/s; 7525/268966 objects misplaced (2.798%); 3.7 MiB/s, 59 objects/s recovering 2024-04-23T16:53:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:20 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1874515962' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:53:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:20 smithi188 ceph-mon[44101]: from='client.37079 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:53:21.097 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:21 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:53:21.097 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:21 smithi103 ceph-mon[39629]: from='client.37079 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:53:21.097 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:21 smithi103 ceph-mon[39629]: osdmap e744: 12 total, 12 up, 8 in 2024-04-23T16:53:21.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:21 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:53:21.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:21 smithi148 ceph-mon[44520]: from='client.37079 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:53:21.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:21 smithi148 ceph-mon[44520]: osdmap e744: 12 total, 12 up, 8 in 2024-04-23T16:53:21.420 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:53:21.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:21.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:53:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:21 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:53:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:21 smithi188 ceph-mon[44101]: from='client.37079 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:53:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:21 smithi188 ceph-mon[44101]: osdmap e744: 12 total, 12 up, 8 in 2024-04-23T16:53:21.527 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:53:21.563 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:22.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:22 smithi103 ceph-mon[39629]: pgmap v2057: 27 pgs: 1 active+recovering, 26 active+clean; 8.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 775 op/s; 7525/267306 objects misplaced (2.815%); 2.2 MiB/s, 35 objects/s recovering 2024-04-23T16:53:22.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:22 smithi148 ceph-mon[44520]: pgmap v2057: 27 pgs: 1 active+recovering, 26 active+clean; 8.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 775 op/s; 7525/267306 objects misplaced (2.815%); 2.2 MiB/s, 35 objects/s recovering 2024-04-23T16:53:22.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:22 smithi188 ceph-mon[44101]: pgmap v2057: 27 pgs: 1 active+recovering, 26 active+clean; 8.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 775 op/s; 7525/267306 objects misplaced (2.815%); 2.2 MiB/s, 35 objects/s recovering 2024-04-23T16:53:22.927 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:53:22.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:22.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:53:23.034 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:53:23.066 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:23.089 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:53:23.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:23.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:53:23.193 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:53:23.222 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:23.242 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:53:23.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:23.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:53:23.246 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:23.346 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:53:23.375 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:23.397 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:53:23.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:23.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:53:23.502 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:53:23.528 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:23.549 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:53:23.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:23.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:53:23.655 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:53:23.699 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:23.727 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:53:23.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:23.733 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:53:23.834 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:53:23.862 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:23.885 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:53:23.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:23.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:53:23.989 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:53:24.029 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:24.053 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:53:24.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:24.059 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:53:24.081 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:53:24.081 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:53:24.105 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:53:24.111 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:24 smithi148 ceph-mon[44520]: pgmap v2058: 27 pgs: 1 active+recovering, 26 active+clean; 8.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 929 op/s; 7222/261526 objects misplaced (2.761%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:53:24.111 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:24 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1748187142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:53:24.138 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:53:24.160 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:53:24.169 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:53:24.194 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:53:24.197 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:24.223 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:53:24.224 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:53:24.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:24.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:53:24.255 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:53:24.281 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:53:24.314 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:53:24.329 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:53:24.342 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:53:24.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:24 smithi103 ceph-mon[39629]: pgmap v2058: 27 pgs: 1 active+recovering, 26 active+clean; 8.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 929 op/s; 7222/261526 objects misplaced (2.761%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:53:24.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1748187142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:53:24.373 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:24.376 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:53:24.383 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:53:24.383 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:53:24.383 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:53:24.383 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:53:24.383 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:53:24.383 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:53:24.383 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:53:24.384 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:53:24.384 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:53:24.384 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:53:24.384 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:53:24.384 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:53:24.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:24 smithi188 ceph-mon[44101]: pgmap v2058: 27 pgs: 1 active+recovering, 26 active+clean; 8.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 929 op/s; 7222/261526 objects misplaced (2.761%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:53:24.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1748187142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:53:24.513 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:53:24.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:24.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:53:24.628 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:53:24.682 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:24.711 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:53:24.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:24.718 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:53:24.819 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:53:24.865 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:24.904 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:53:24.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:24.912 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:53:25.013 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:53:25.064 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:25.096 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:53:25.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:25.104 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:53:25.205 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:53:25.249 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:25.294 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:53:25.328 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:53:25.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:25.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:53:25.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:25 smithi148 ceph-mon[44520]: 9.6 scrub starts 2024-04-23T16:53:25.441 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:53:25.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:25 smithi188 ceph-mon[44101]: 9.6 scrub starts 2024-04-23T16:53:25.481 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:25.506 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:53:25.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:25.510 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:53:25.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:25.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:25 smithi103 ceph-mon[39629]: 9.6 scrub starts 2024-04-23T16:53:25.611 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:53:25.654 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:25.791 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:53:25.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:25.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:53:25.898 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:53:25.944 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:25.969 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:53:25.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:25.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:53:26.079 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:53:26.113 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:26.157 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:53:26.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:26.163 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:53:26.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:26.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:26.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:26 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:53:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:26 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:53:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:26 smithi188 ceph-mon[44101]: pgmap v2059: 27 pgs: 1 active+recovering, 26 active+clean; 7.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 942 op/s; 7222/257856 objects misplaced (2.801%); 2.1 MiB/s, 34 objects/s recovering 2024-04-23T16:53:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:26 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:53:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:26 smithi188 ceph-mon[44101]: osdmap e745: 12 total, 12 up, 8 in 2024-04-23T16:53:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:26 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:53:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:26 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:53:26.264 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:53:26.293 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:26.322 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:53:26.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:26.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:53:26.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:26.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:26.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:26 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:53:26.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:26 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:53:26.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:26 smithi103 ceph-mon[39629]: pgmap v2059: 27 pgs: 1 active+recovering, 26 active+clean; 7.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 942 op/s; 7222/257856 objects misplaced (2.801%); 2.1 MiB/s, 34 objects/s recovering 2024-04-23T16:53:26.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:26 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:53:26.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:26 smithi103 ceph-mon[39629]: osdmap e745: 12 total, 12 up, 8 in 2024-04-23T16:53:26.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:26 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:53:26.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:26 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:53:26.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:26.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:26 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:53:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:26 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:53:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:26 smithi148 ceph-mon[44520]: pgmap v2059: 27 pgs: 1 active+recovering, 26 active+clean; 7.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 942 op/s; 7222/257856 objects misplaced (2.801%); 2.1 MiB/s, 34 objects/s recovering 2024-04-23T16:53:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:26 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:53:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:26 smithi148 ceph-mon[44520]: osdmap e745: 12 total, 12 up, 8 in 2024-04-23T16:53:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:26 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:53:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:26 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:53:26.427 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:53:26.454 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:26.475 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:53:26.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:26.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:53:26.528 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:53:26.528 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:53:26.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:53:26.528 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-23T16:53:26.528 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:53:26.580 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:53:26.619 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:26.648 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:53:26.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:26.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:53:26.756 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:53:26.787 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:26.817 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:53:26.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:26.822 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:53:26.923 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:53:26.958 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:26.990 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:53:26.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:26.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:53:27.094 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:53:27.145 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:27.163 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.7 to 1 (10000) 2024-04-23T16:53:27.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:27 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:53:27.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:27 smithi188 ceph-mon[44101]: 9.6 scrub ok 2024-04-23T16:53:27.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:27 smithi188 ceph-mon[44101]: 1.4 scrub starts 2024-04-23T16:53:27.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:27 smithi188 ceph-mon[44101]: 1.4 scrub ok 2024-04-23T16:53:27.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:27 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/335361451' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:53:27.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:27 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:53:27.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:27 smithi103 ceph-mon[39629]: 9.6 scrub ok 2024-04-23T16:53:27.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:27 smithi103 ceph-mon[39629]: 1.4 scrub starts 2024-04-23T16:53:27.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:27 smithi103 ceph-mon[39629]: 1.4 scrub ok 2024-04-23T16:53:27.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:27 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/335361451' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:53:27.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:27 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:53:27.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:27 smithi148 ceph-mon[44520]: 9.6 scrub ok 2024-04-23T16:53:27.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:27 smithi148 ceph-mon[44520]: 1.4 scrub starts 2024-04-23T16:53:27.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:27 smithi148 ceph-mon[44520]: 1.4 scrub ok 2024-04-23T16:53:27.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:27 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/335361451' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:53:27.705 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-23T16:53:27.819 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:53:27.906 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:53:27.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:27.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:53:28.013 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:53:28.022 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:53:28.053 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:28 smithi103 ceph-mon[39629]: pgmap v2061: 27 pgs: 1 active+recovering, 26 active+clean; 7.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 985 op/s; 6939/253198 objects misplaced (2.741%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:53:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:28 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/335361451' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:53:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:28 smithi103 ceph-mon[39629]: osdmap e746: 12 total, 12 up, 8 in 2024-04-23T16:53:28.414 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:53:28.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:28.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:53:28.447 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 8 1 2024-04-23T16:53:28.466 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:28.468 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:28.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:28 smithi188 ceph-mon[44101]: pgmap v2061: 27 pgs: 1 active+recovering, 26 active+clean; 7.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 985 op/s; 6939/253198 objects misplaced (2.741%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:53:28.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:28 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/335361451' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:53:28.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:28 smithi188 ceph-mon[44101]: osdmap e746: 12 total, 12 up, 8 in 2024-04-23T16:53:28.518 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:53:28.558 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:28.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:28 smithi148 ceph-mon[44520]: pgmap v2061: 27 pgs: 1 active+recovering, 26 active+clean; 7.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 985 op/s; 6939/253198 objects misplaced (2.741%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:53:28.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:28 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/335361451' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:53:28.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:28 smithi148 ceph-mon[44520]: osdmap e746: 12 total, 12 up, 8 in 2024-04-23T16:53:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:29 smithi103 ceph-mon[39629]: 1.7 scrub starts 2024-04-23T16:53:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:29 smithi103 ceph-mon[39629]: 1.7 scrub ok 2024-04-23T16:53:29.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:29 smithi148 ceph-mon[44520]: 1.7 scrub starts 2024-04-23T16:53:29.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:29 smithi148 ceph-mon[44520]: 1.7 scrub ok 2024-04-23T16:53:29.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:29 smithi188 ceph-mon[44101]: 1.7 scrub starts 2024-04-23T16:53:29.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:29 smithi188 ceph-mon[44101]: 1.7 scrub ok 2024-04-23T16:53:30.133 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:53:30.133 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:53:30.133 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2024-04-23T16:53:30.133 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:53:30.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:30 smithi188 ceph-mon[44101]: 9.3 scrub starts 2024-04-23T16:53:30.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:30 smithi188 ceph-mon[44101]: pgmap v2063: 27 pgs: 1 active+recovering, 26 active+clean; 7.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 6939/250252 objects misplaced (2.773%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:53:30.527 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:30 smithi148 ceph-mon[44520]: 9.3 scrub starts 2024-04-23T16:53:30.527 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:30 smithi148 ceph-mon[44520]: pgmap v2063: 27 pgs: 1 active+recovering, 26 active+clean; 7.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 6939/250252 objects misplaced (2.773%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:53:30.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:30 smithi103 ceph-mon[39629]: 9.3 scrub starts 2024-04-23T16:53:30.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:30 smithi103 ceph-mon[39629]: pgmap v2063: 27 pgs: 1 active+recovering, 26 active+clean; 7.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 6939/250252 objects misplaced (2.773%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:53:31.168 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:31 smithi103 ceph-mon[39629]: pgmap v2064: 27 pgs: 1 active+recovering, 26 active+clean; 7.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 815 op/s; 6939/248476 objects misplaced (2.793%); 2.2 MiB/s, 35 objects/s recovering 2024-04-23T16:53:31.223 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:53:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:31.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:53:31.327 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:53:31.359 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:31.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:31 smithi188 ceph-mon[44101]: pgmap v2064: 27 pgs: 1 active+recovering, 26 active+clean; 7.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 815 op/s; 6939/248476 objects misplaced (2.793%); 2.2 MiB/s, 35 objects/s recovering 2024-04-23T16:53:31.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:31 smithi148 ceph-mon[44520]: pgmap v2064: 27 pgs: 1 active+recovering, 26 active+clean; 7.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 815 op/s; 6939/248476 objects misplaced (2.793%); 2.2 MiB/s, 35 objects/s recovering 2024-04-23T16:53:31.913 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:53:31.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:31.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:53:31.930 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-23T16:53:32.018 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:53:32.049 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:32.076 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:53:32.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:32.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:53:32.179 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:53:32.211 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:32.235 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:53:32.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:32.239 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:53:32.262 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:32.289 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:53:32.339 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:53:32.372 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:32.527 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:53:32.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:32.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:53:32.635 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:53:32.675 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:32.734 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:53:32.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:32.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:53:32.843 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:53:32.886 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:33.443 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:53:33.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:33.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:53:33.547 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:53:33.576 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:33.857 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:53:33.887 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:53:33.911 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:53:33.942 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:53:33.968 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:53:33.991 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:53:34.020 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:53:34.045 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:53:34.075 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:53:34.104 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:53:34.138 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:53:34.165 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:53:34.169 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:53:34.169 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:53:34.169 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:53:34.169 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:53:34.169 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:53:34.169 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:53:34.170 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:53:34.170 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:53:34.170 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:53:34.170 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:53:34.170 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:53:34.170 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:53:34.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:34 smithi103 ceph-mon[39629]: pgmap v2065: 27 pgs: 1 active+recovering, 26 active+clean; 7.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 960 op/s; 6645/242488 objects misplaced (2.740%); 4.5 MiB/s, 72 objects/s recovering 2024-04-23T16:53:34.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:34 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1095756842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:53:34.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:34 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:34.366 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:34 smithi148 ceph-mon[44520]: pgmap v2065: 27 pgs: 1 active+recovering, 26 active+clean; 7.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 960 op/s; 6645/242488 objects misplaced (2.740%); 4.5 MiB/s, 72 objects/s recovering 2024-04-23T16:53:34.367 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:34 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1095756842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:53:34.367 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:34 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:34.428 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:53:34.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:34.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:53:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:34 smithi188 ceph-mon[44101]: pgmap v2065: 27 pgs: 1 active+recovering, 26 active+clean; 7.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 960 op/s; 6645/242488 objects misplaced (2.740%); 4.5 MiB/s, 72 objects/s recovering 2024-04-23T16:53:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:34 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1095756842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:53:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:34 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:34.533 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:53:34.569 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:34.695 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:53:34.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:34.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:53:34.799 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:53:34.840 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:35.057 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:35 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:53:35.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:35 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:53:35.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:35 smithi148 ceph-mon[44520]: osdmap e747: 12 total, 12 up, 8 in 2024-04-23T16:53:35.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.058 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:35 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.197 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:35 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:53:35.198 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:35 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:53:35.198 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.198 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.198 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.198 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.198 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.198 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.198 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:35 smithi103 ceph-mon[39629]: osdmap e747: 12 total, 12 up, 8 in 2024-04-23T16:53:35.198 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.198 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.198 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.199 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.199 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:35 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:35 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:53:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:35 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:53:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:35 smithi188 ceph-mon[44101]: osdmap e747: 12 total, 12 up, 8 in 2024-04-23T16:53:35.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:35 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:35.501 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:53:35.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:35.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:53:35.607 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:53:35.668 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:35.729 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:53:35.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:35.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:53:35.835 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:53:35.865 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:35.920 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:53:35.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:35.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:53:36.025 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:53:36.067 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:36.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:36 smithi103 ceph-mon[39629]: pgmap v2067: 27 pgs: 1 active+recovering, 26 active+clean; 7.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 886 op/s; 6645/239012 objects misplaced (2.780%); 2.3 MiB/s, 36 objects/s recovering 2024-04-23T16:53:36.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:36 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2753790626' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:53:36.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:36 smithi103 ceph-mon[39629]: from='client.27331 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:53:36.078 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.8 to 1 (10000) 2024-04-23T16:53:36.102 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:53:36.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:36.105 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:53:36.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:36 smithi148 ceph-mon[44520]: pgmap v2067: 27 pgs: 1 active+recovering, 26 active+clean; 7.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 886 op/s; 6645/239012 objects misplaced (2.780%); 2.3 MiB/s, 36 objects/s recovering 2024-04-23T16:53:36.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:36 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2753790626' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:53:36.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:36 smithi148 ceph-mon[44520]: from='client.27331 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:53:36.206 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:53:36.247 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:36.335 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:53:36.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:36.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:53:36.440 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:53:36.469 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:36.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:36 smithi188 ceph-mon[44101]: pgmap v2067: 27 pgs: 1 active+recovering, 26 active+clean; 7.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 886 op/s; 6645/239012 objects misplaced (2.780%); 2.3 MiB/s, 36 objects/s recovering 2024-04-23T16:53:36.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:36 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2753790626' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:53:36.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:36 smithi188 ceph-mon[44101]: from='client.27331 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:53:36.874 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 9 1 2024-04-23T16:53:36.983 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:53:37.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:37 smithi103 ceph-mon[39629]: from='client.27331 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:53:37.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:37 smithi103 ceph-mon[39629]: osdmap e748: 12 total, 12 up, 8 in 2024-04-23T16:53:37.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:37 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3658641092' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:53:37.124 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:53:37.125 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:53:37.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:37.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:53:37.229 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:53:37.261 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:37.289 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:53:37.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:37.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:53:37.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:37 smithi148 ceph-mon[44520]: from='client.27331 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:53:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:37 smithi148 ceph-mon[44520]: osdmap e748: 12 total, 12 up, 8 in 2024-04-23T16:53:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:37 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3658641092' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:53:37.396 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:53:37.435 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:37.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:37 smithi188 ceph-mon[44101]: from='client.27331 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:53:37.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:37 smithi188 ceph-mon[44101]: osdmap e748: 12 total, 12 up, 8 in 2024-04-23T16:53:37.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:37 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3658641092' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:53:37.563 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:53:37.563 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:53:38.320 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:38 smithi148 ceph-mon[44520]: pgmap v2069: 27 pgs: 1 active+recovering, 26 active+clean; 7.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 6347/234180 objects misplaced (2.710%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:53:38.321 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:38 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:53:38.321 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:38 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:53:38.321 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:38 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3658641092' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:53:38.321 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:38 smithi148 ceph-mon[44520]: osdmap e749: 12 total, 12 up, 8 in 2024-04-23T16:53:38.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:38 smithi103 ceph-mon[39629]: pgmap v2069: 27 pgs: 1 active+recovering, 26 active+clean; 7.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 6347/234180 objects misplaced (2.710%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:53:38.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:38 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:53:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:38 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:53:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:38 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3658641092' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:53:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:38 smithi103 ceph-mon[39629]: osdmap e749: 12 total, 12 up, 8 in 2024-04-23T16:53:38.352 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-23T16:53:38.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:38 smithi188 ceph-mon[44101]: pgmap v2069: 27 pgs: 1 active+recovering, 26 active+clean; 7.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 6347/234180 objects misplaced (2.710%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:53:38.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:38 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:53:38.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:38 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:53:38.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:38 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3658641092' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:53:38.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:38 smithi188 ceph-mon[44101]: osdmap e749: 12 total, 12 up, 8 in 2024-04-23T16:53:38.667 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:53:38.829 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:53:38.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:38.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:53:38.843 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:38.843 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:38.933 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:53:38.977 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:39.008 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:53:39.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:39.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:53:39.112 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:53:39.145 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:40.296 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:40 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:53:40.296 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:40 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:53:40.296 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:40 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:53:40.296 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:40 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:53:40.296 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:40 smithi148 ceph-mon[44520]: pgmap v2071: 27 pgs: 1 active+recovering, 26 active+clean; 7.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 6347/231200 objects misplaced (2.745%); 3.1 MiB/s, 49 objects/s recovering 2024-04-23T16:53:40.296 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:40 smithi148 ceph-mon[44520]: osdmap e750: 12 total, 12 up, 8 in 2024-04-23T16:53:40.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:40 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:53:40.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:40 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:53:40.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:40 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:53:40.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:40 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:53:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:40 smithi103 ceph-mon[39629]: pgmap v2071: 27 pgs: 1 active+recovering, 26 active+clean; 7.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 6347/231200 objects misplaced (2.745%); 3.1 MiB/s, 49 objects/s recovering 2024-04-23T16:53:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:40 smithi103 ceph-mon[39629]: osdmap e750: 12 total, 12 up, 8 in 2024-04-23T16:53:40.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:40 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:53:40.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:40 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:53:40.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:40 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:53:40.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:40 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:53:40.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:40 smithi188 ceph-mon[44101]: pgmap v2071: 27 pgs: 1 active+recovering, 26 active+clean; 7.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 6347/231200 objects misplaced (2.745%); 3.1 MiB/s, 49 objects/s recovering 2024-04-23T16:53:40.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:40 smithi188 ceph-mon[44101]: osdmap e750: 12 total, 12 up, 8 in 2024-04-23T16:53:40.815 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:53:40.923 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:53:40.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:40.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:53:41.025 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:53:41.025 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:53:41.029 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:53:41.065 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:42.233 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:42 smithi103 ceph-mon[39629]: 9.c scrub starts 2024-04-23T16:53:42.234 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:42 smithi103 ceph-mon[39629]: pgmap v2073: 27 pgs: 1 active+recovering, 26 active+clean; 7.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 801 op/s; 6347/229398 objects misplaced (2.767%); 3.1 MiB/s, 49 objects/s recovering 2024-04-23T16:53:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:42 smithi148 ceph-mon[44520]: 9.c scrub starts 2024-04-23T16:53:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:42 smithi148 ceph-mon[44520]: pgmap v2073: 27 pgs: 1 active+recovering, 26 active+clean; 7.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 801 op/s; 6347/229398 objects misplaced (2.767%); 3.1 MiB/s, 49 objects/s recovering 2024-04-23T16:53:42.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:42 smithi188 ceph-mon[44101]: 9.c scrub starts 2024-04-23T16:53:42.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:42 smithi188 ceph-mon[44101]: pgmap v2073: 27 pgs: 1 active+recovering, 26 active+clean; 7.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 801 op/s; 6347/229398 objects misplaced (2.767%); 3.1 MiB/s, 49 objects/s recovering 2024-04-23T16:53:42.964 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:53:42.989 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:53:43.016 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:53:43.044 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:53:43.074 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:53:43.108 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:53:43.136 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:53:43.169 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:53:43.206 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:53:43.236 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:53:43.259 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:53:43.269 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-23T16:53:43.290 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:53:43.292 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:53:43.292 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:53:43.292 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:53:43.292 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:53:43.293 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:53:43.293 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:53:43.293 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:53:43.293 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:53:43.293 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:53:43.293 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:53:43.293 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:53:43.293 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:53:43.320 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:53:43.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:43.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:53:43.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:43 smithi148 ceph-mon[44520]: 9.c scrub ok 2024-04-23T16:53:43.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:43 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2466815997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:53:43.426 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:53:43.454 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:43 smithi103 ceph-mon[39629]: 9.c scrub ok 2024-04-23T16:53:43.454 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:43 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2466815997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:53:43.456 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:43.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:43 smithi188 ceph-mon[44101]: 9.c scrub ok 2024-04-23T16:53:43.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:43 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2466815997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:53:43.531 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:53:43.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:43.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:53:43.639 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:53:43.669 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:43.692 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:53:43.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:43.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:53:43.798 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:53:43.828 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:43.852 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:53:43.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:43.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:53:43.960 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:53:44.040 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:44.090 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:53:44.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:44.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:53:44.113 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:44.123 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.9 to 1 (10000) 2024-04-23T16:53:44.198 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:53:44.230 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:44.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:44 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:53:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:44 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:53:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:44 smithi148 ceph-mon[44520]: pgmap v2074: 27 pgs: 1 active+recovering, 26 active+clean; 6.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 6045/223666 objects misplaced (2.703%); 5.4 MiB/s, 86 objects/s recovering 2024-04-23T16:53:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:44 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2733321387' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:53:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:44 smithi148 ceph-mon[44520]: from='client.37106 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:53:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:44 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.433 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:53:44.438 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:44 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:53:44.438 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:44 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:53:44.438 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.438 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:44 smithi103 ceph-mon[39629]: pgmap v2074: 27 pgs: 1 active+recovering, 26 active+clean; 6.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 6045/223666 objects misplaced (2.703%); 5.4 MiB/s, 86 objects/s recovering 2024-04-23T16:53:44.438 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.439 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.439 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.439 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.439 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.439 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:44 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2733321387' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:53:44.439 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:44 smithi103 ceph-mon[39629]: from='client.37106 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:53:44.439 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.439 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.439 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.439 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:44 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:44.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:53:44.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:44 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:53:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:44 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:53:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:44 smithi188 ceph-mon[44101]: pgmap v2074: 27 pgs: 1 active+recovering, 26 active+clean; 6.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 6045/223666 objects misplaced (2.703%); 5.4 MiB/s, 86 objects/s recovering 2024-04-23T16:53:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:44 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2733321387' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:53:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:44 smithi188 ceph-mon[44101]: from='client.37106 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:53:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:44 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:44.541 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:53:44.575 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:44.657 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:53:44.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:44.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:53:44.761 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:53:44.789 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:44.811 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:53:44.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:44.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:53:44.915 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:53:44.953 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:45.079 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:53:45.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:45.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:53:45.118 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:45 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:45.119 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:45 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:45.119 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:45 smithi148 ceph-mon[44520]: 9.8 deep-scrub starts 2024-04-23T16:53:45.119 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:45 smithi148 ceph-mon[44520]: from='client.37106 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:53:45.119 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:45 smithi148 ceph-mon[44520]: osdmap e751: 12 total, 12 up, 8 in 2024-04-23T16:53:45.119 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:45 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:53:45.185 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:53:45.231 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:45.260 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:53:45.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:45.268 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:53:45.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:45 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:45 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:45 smithi103 ceph-mon[39629]: 9.8 deep-scrub starts 2024-04-23T16:53:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:45 smithi103 ceph-mon[39629]: from='client.37106 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:53:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:45 smithi103 ceph-mon[39629]: osdmap e751: 12 total, 12 up, 8 in 2024-04-23T16:53:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:45 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:53:45.368 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:53:45.397 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:45.416 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:53:45.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:45.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:53:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:45 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:45 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:45 smithi188 ceph-mon[44101]: 9.8 deep-scrub starts 2024-04-23T16:53:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:45 smithi188 ceph-mon[44101]: from='client.37106 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:53:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:45 smithi188 ceph-mon[44101]: osdmap e751: 12 total, 12 up, 8 in 2024-04-23T16:53:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:45 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:53:45.519 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:53:45.550 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:45.613 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:53:45.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:45.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:53:45.717 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:53:45.745 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:45.769 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:53:45.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:45.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:53:45.873 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:53:45.901 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:45.923 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:53:45.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:45.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:53:46.026 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:53:46.053 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:46.075 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 11 1 2024-04-23T16:53:46.076 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:53:46.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:46.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:53:46.170 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:53:46.179 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:53:46.206 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:46.226 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:53:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:46.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:53:46.297 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:46 smithi103 ceph-mon[39629]: pgmap v2076: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 895 op/s; 6045/220020 objects misplaced (2.747%); 2.4 MiB/s, 38 objects/s recovering 2024-04-23T16:53:46.331 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:53:46.356 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:46 smithi148 ceph-mon[44520]: pgmap v2076: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 895 op/s; 6045/220020 objects misplaced (2.747%); 2.4 MiB/s, 38 objects/s recovering 2024-04-23T16:53:46.357 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:46.379 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:53:46.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:46.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:53:46.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:46 smithi188 ceph-mon[44101]: pgmap v2076: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 895 op/s; 6045/220020 objects misplaced (2.747%); 2.4 MiB/s, 38 objects/s recovering 2024-04-23T16:53:46.483 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:53:46.510 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:46.532 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:53:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:46.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:53:46.606 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:46.638 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:53:46.666 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:46.686 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:53:46.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:46.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:53:46.794 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:53:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:46.843 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:53:46.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:46.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:53:46.947 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:53:46.990 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:47.121 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:47 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:53:47.121 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:47 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:53:47.121 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:47 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:53:47.121 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:47 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:53:47.121 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:47 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:53:47.121 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:47 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:53:47.121 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:47 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/693559877' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:53:47.122 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:47 smithi103 ceph-mon[39629]: from='client.27349 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:53:47.333 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:53:47.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:47 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/693559877' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:53:47.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:47 smithi148 ceph-mon[44520]: from='client.27349 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:53:47.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:47 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:53:47.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:47 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:53:47.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:47 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:53:47.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:47 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/693559877' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:53:47.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:47 smithi188 ceph-mon[44101]: from='client.27349 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:53:48.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:48 smithi148 ceph-mon[44520]: pgmap v2077: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 953 op/s; 5773/215936 objects misplaced (2.673%); 4.5 MiB/s, 71 objects/s recovering 2024-04-23T16:53:48.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:48 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:53:48.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:48 smithi148 ceph-mon[44520]: from='client.27349 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:53:48.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:48 smithi148 ceph-mon[44520]: osdmap e752: 12 total, 12 up, 8 in 2024-04-23T16:53:48.431 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:48 smithi103 ceph-mon[39629]: pgmap v2077: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 953 op/s; 5773/215936 objects misplaced (2.673%); 4.5 MiB/s, 71 objects/s recovering 2024-04-23T16:53:48.432 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:48 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:53:48.432 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:48 smithi103 ceph-mon[39629]: from='client.27349 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:53:48.432 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:48 smithi103 ceph-mon[39629]: osdmap e752: 12 total, 12 up, 8 in 2024-04-23T16:53:48.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:48 smithi188 ceph-mon[44101]: pgmap v2077: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 953 op/s; 5773/215936 objects misplaced (2.673%); 4.5 MiB/s, 71 objects/s recovering 2024-04-23T16:53:48.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:48 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:53:48.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:48 smithi188 ceph-mon[44101]: from='client.27349 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:53:48.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:48 smithi188 ceph-mon[44101]: osdmap e752: 12 total, 12 up, 8 in 2024-04-23T16:53:48.982 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:48.983 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:50.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:50 smithi103 ceph-mon[39629]: pgmap v2079: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 993 op/s; 5773/213506 objects misplaced (2.704%); 4.5 MiB/s, 71 objects/s recovering 2024-04-23T16:53:50.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:50 smithi188 ceph-mon[44101]: pgmap v2079: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 993 op/s; 5773/213506 objects misplaced (2.704%); 4.5 MiB/s, 71 objects/s recovering 2024-04-23T16:53:50.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:50 smithi148 ceph-mon[44520]: pgmap v2079: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 993 op/s; 5773/213506 objects misplaced (2.704%); 4.5 MiB/s, 71 objects/s recovering 2024-04-23T16:53:51.031 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:53:52.164 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:53:52.164 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:53:52.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:53:52.164 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.27202.0:199795 9.c e752/703 9:3b47aebc:::benchmark_data_smithi188_167360_object48662:head v 752'12596, mlcod=752'12593)", 2024-04-23T16:53:52.164 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:53:52.141350+0000", 2024-04-23T16:53:52.164 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.001416713, 2024-04-23T16:53:52.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0014475549999999999, 2024-04-23T16:53:52.164 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:53:52.164 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:53:52.164 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:53:52.165 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:53:52.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:53:52.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:53:52.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:52.141350+0000", 2024-04-23T16:53:52.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:53:52.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:53:52.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:53:52.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:53:52.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:52.141350+0000", 2024-04-23T16:53:52.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:53:52.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:53:52.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:53:52.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:53:52.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:52.141352+0000", 2024-04-23T16:53:52.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.644e-06 2024-04-23T16:53:52.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:53:52.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:53:52.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:53:52.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:52.141365+0000", 2024-04-23T16:53:52.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2792999999999999e-05 2024-04-23T16:53:52.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:53:52.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:53:52.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:53:52.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:52.141383+0000", 2024-04-23T16:53:52.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7697999999999999e-05 2024-04-23T16:53:52.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:53:52.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:53:52.167 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:53:52.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:52.141392+0000", 2024-04-23T16:53:52.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9239999999999996e-06 2024-04-23T16:53:52.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:53:52.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:53:52.167 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:53:52.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:52.141466+0000", 2024-04-23T16:53:52.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4610000000000006e-05 2024-04-23T16:53:52.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:53:52.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:53:52.167 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:53:52.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:52.141493+0000", 2024-04-23T16:53:52.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6554999999999999e-05 2024-04-23T16:53:52.167 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:53:52.167 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:53:52.168 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:53:52.168 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:53:52.168 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:53:52.168 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 1 2024-04-23T16:53:52.168 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:53:52.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:52 smithi103 ceph-mon[39629]: pgmap v2080: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 5773/211956 objects misplaced (2.724%); 2.1 MiB/s, 33 objects/s recovering 2024-04-23T16:53:52.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:52 smithi188 ceph-mon[44101]: pgmap v2080: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 5773/211956 objects misplaced (2.724%); 2.1 MiB/s, 33 objects/s recovering 2024-04-23T16:53:52.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:52 smithi148 ceph-mon[44520]: pgmap v2080: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 5773/211956 objects misplaced (2.724%); 2.1 MiB/s, 33 objects/s recovering 2024-04-23T16:53:53.289 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:53:53.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:53.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:53:53.334 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:53:53.361 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:53:53.385 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:53:53.394 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:53:53.416 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:53:53.431 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:53.438 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:53:53.461 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:53:53.462 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:53:53.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:53.465 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:53:53.488 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:53:53.518 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:53:53.551 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:53:53.566 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:53:53.579 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:53:53.606 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:53.612 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:53:53.626 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:53:53.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:53.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:53:53.639 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:53:53.641 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:53:53.641 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:53:53.641 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:53:53.641 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:53:53.641 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:53:53.641 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:53:53.641 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:53:53.642 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:53:53.642 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:53:53.642 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:53:53.642 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:53:53.642 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:53:53.730 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:53:53.775 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:53.799 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:53:53.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:53.806 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:53:53.908 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:53:53.923 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:53.947 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:53.977 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:53:53.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:53.982 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:53:54.083 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:53:54.119 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:54.154 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.11 to 1 (10000) 2024-04-23T16:53:54.169 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:53:54.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:54.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:53:54.278 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:53:54.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:54 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:53:54.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:54 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:53:54.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:54 smithi103 ceph-mon[39629]: pgmap v2081: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 826 op/s; 5486/207064 objects misplaced (2.649%); 3.9 MiB/s, 62 objects/s recovering 2024-04-23T16:53:54.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:54 smithi103 ceph-mon[39629]: osdmap e753: 12 total, 12 up, 8 in 2024-04-23T16:53:54.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:54 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/430076470' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:53:54.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:54.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:54.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:54.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:54.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:54.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:54 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:54.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:54 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/589538842' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:53:54.309 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:54 smithi103 ceph-mon[39629]: from='client.37127 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:53:54.311 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:54.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:54 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:53:54.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:54 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:53:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:54 smithi188 ceph-mon[44101]: pgmap v2081: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 826 op/s; 5486/207064 objects misplaced (2.649%); 3.9 MiB/s, 62 objects/s recovering 2024-04-23T16:53:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:54 smithi188 ceph-mon[44101]: osdmap e753: 12 total, 12 up, 8 in 2024-04-23T16:53:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:54 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/430076470' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:53:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:54 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:54 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/589538842' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:53:54.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:54 smithi188 ceph-mon[44101]: from='client.37127 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:53:54.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:54 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:53:54.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:54 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:53:54.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:54 smithi148 ceph-mon[44520]: pgmap v2081: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 826 op/s; 5486/207064 objects misplaced (2.649%); 3.9 MiB/s, 62 objects/s recovering 2024-04-23T16:53:54.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:54 smithi148 ceph-mon[44520]: osdmap e753: 12 total, 12 up, 8 in 2024-04-23T16:53:54.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:54 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/430076470' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:53:54.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:54.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:54.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:54.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:54.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:54.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:54 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:54.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:54 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/589538842' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:53:54.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:54 smithi148 ceph-mon[44520]: from='client.37127 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:53:54.934 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-23T16:53:55.446 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:55.446 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:55.446 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:55.446 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:55.446 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:55.446 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:55 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:55.447 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:55 smithi103 ceph-mon[39629]: from='client.37127 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:53:55.447 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:55 smithi103 ceph-mon[39629]: osdmap e754: 12 total, 12 up, 8 in 2024-04-23T16:53:55.447 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:55 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:53:55.447 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:55 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:53:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:55 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:55 smithi188 ceph-mon[44101]: from='client.37127 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:53:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:55 smithi188 ceph-mon[44101]: osdmap e754: 12 total, 12 up, 8 in 2024-04-23T16:53:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:55 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:53:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:55 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:53:55.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:55.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:55.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:55.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:55.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:55.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:55 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:53:55.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:55 smithi148 ceph-mon[44520]: from='client.37127 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:53:55.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:55 smithi148 ceph-mon[44520]: osdmap e754: 12 total, 12 up, 8 in 2024-04-23T16:53:55.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:55 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:53:55.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:55 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:53:55.819 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-23T16:53:56.276 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:53:56.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:56.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:53:56.381 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:53:56.408 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:56 smithi148 ceph-mon[44520]: pgmap v2084: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 683 op/s; 5486/205456 objects misplaced (2.670%); 2.3 MiB/s, 37 objects/s recovering 2024-04-23T16:53:56.408 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:56 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:53:56.408 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:56 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:53:56.408 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:56 smithi148 ceph-mon[44520]: osdmap e755: 12 total, 12 up, 8 in 2024-04-23T16:53:56.410 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:56.441 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:53:56.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:56.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:53:56.459 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:56 smithi103 ceph-mon[39629]: pgmap v2084: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 683 op/s; 5486/205456 objects misplaced (2.670%); 2.3 MiB/s, 37 objects/s recovering 2024-04-23T16:53:56.460 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:56 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:53:56.460 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:56 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:53:56.460 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:56 smithi103 ceph-mon[39629]: osdmap e755: 12 total, 12 up, 8 in 2024-04-23T16:53:56.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:56 smithi188 ceph-mon[44101]: pgmap v2084: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 683 op/s; 5486/205456 objects misplaced (2.670%); 2.3 MiB/s, 37 objects/s recovering 2024-04-23T16:53:56.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:56 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:53:56.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:56 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:53:56.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:56 smithi188 ceph-mon[44101]: osdmap e755: 12 total, 12 up, 8 in 2024-04-23T16:53:56.544 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:53:56.545 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:53:56.573 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:56.595 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:53:56.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:56.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:53:56.699 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:53:56.739 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:56.771 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:53:56.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:56.784 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:53:56.885 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:53:56.886 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:56.902 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:53:56.913 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:56.934 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:53:56.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:56.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:53:57.038 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:53:57.077 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:57.161 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:53:57.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:57.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:53:57.265 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:53:57.293 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:57.316 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:53:57.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:57.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:53:57.420 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:53:57.457 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:58.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:58 smithi188 ceph-mon[44101]: pgmap v2086: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 989 op/s; 5239/200076 objects misplaced (2.619%); 5.6 MiB/s, 88 objects/s recovering 2024-04-23T16:53:58.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:58 smithi103 ceph-mon[39629]: pgmap v2086: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 989 op/s; 5239/200076 objects misplaced (2.619%); 5.6 MiB/s, 88 objects/s recovering 2024-04-23T16:53:58.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:58 smithi148 ceph-mon[44520]: pgmap v2086: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 989 op/s; 5239/200076 objects misplaced (2.619%); 5.6 MiB/s, 88 objects/s recovering 2024-04-23T16:53:59.189 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:53:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:59 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:53:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:59 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:53:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:59 smithi188 ceph-mon[44101]: osdmap e756: 12 total, 12 up, 8 in 2024-04-23T16:53:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:59 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:53:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:59 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:53:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:59 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2376592103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:53:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:53:59 smithi188 ceph-mon[44101]: pgmap v2088: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 5239/197574 objects misplaced (2.652%); 2.6 MiB/s, 42 objects/s recovering 2024-04-23T16:53:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:59 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:53:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:59 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:53:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:59 smithi103 ceph-mon[39629]: osdmap e756: 12 total, 12 up, 8 in 2024-04-23T16:53:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:59 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:53:59.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:59 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:53:59.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:59 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2376592103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:53:59.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:53:59 smithi103 ceph-mon[39629]: pgmap v2088: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 5239/197574 objects misplaced (2.652%); 2.6 MiB/s, 42 objects/s recovering 2024-04-23T16:53:59.620 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:53:59.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:59 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:53:59.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:59 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:53:59.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:59 smithi148 ceph-mon[44520]: osdmap e756: 12 total, 12 up, 8 in 2024-04-23T16:53:59.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:59 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:53:59.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:59 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:53:59.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:59 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2376592103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:53:59.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:53:59 smithi148 ceph-mon[44520]: pgmap v2088: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 25 active+clean; 6.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 5239/197574 objects misplaced (2.652%); 2.6 MiB/s, 42 objects/s recovering 2024-04-23T16:53:59.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:59.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:53:59.726 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:53:59.754 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:59.776 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:53:59.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:59.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:53:59.882 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:53:59.910 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:53:59.930 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:53:59.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:53:59.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:54:00.034 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:54:00.083 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:00.171 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:54:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:00.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:54:00.276 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:54:00.307 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:00.335 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:54:00.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:00.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:54:00.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:00 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:54:00.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:00 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2376592103' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:54:00.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:00 smithi103 ceph-mon[39629]: osdmap e757: 12 total, 12 up, 8 in 2024-04-23T16:54:00.441 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:54:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:00 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:54:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:00 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2376592103' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:54:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:00 smithi188 ceph-mon[44101]: osdmap e757: 12 total, 12 up, 8 in 2024-04-23T16:54:00.479 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:00.525 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:54:00.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:00.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:54:00.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:00 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:54:00.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:00 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2376592103' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:54:00.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:00 smithi148 ceph-mon[44520]: osdmap e757: 12 total, 12 up, 8 in 2024-04-23T16:54:00.632 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:54:00.661 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:00.686 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:54:00.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:00.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:54:00.789 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:54:00.830 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:00.870 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:54:00.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:00.873 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:54:00.974 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:54:01.004 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:01.026 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:54:01.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:01.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:54:01.047 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:54:01.130 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:54:01.157 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:01.179 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:54:01.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:01.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:54:01.198 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:01 smithi188 ceph-mon[44101]: pgmap v2090: 27 pgs: 1 active+recovering, 26 active+clean; 6.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 785 op/s; 5239/196024 objects misplaced (2.673%); 2.6 MiB/s, 41 objects/s recovering 2024-04-23T16:54:01.284 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:54:01.339 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:01.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:01 smithi103 ceph-mon[39629]: pgmap v2090: 27 pgs: 1 active+recovering, 26 active+clean; 6.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 785 op/s; 5239/196024 objects misplaced (2.673%); 2.6 MiB/s, 41 objects/s recovering 2024-04-23T16:54:01.366 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:54:01.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:01.373 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:54:01.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:01 smithi148 ceph-mon[44520]: pgmap v2090: 27 pgs: 1 active+recovering, 26 active+clean; 6.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 785 op/s; 5239/196024 objects misplaced (2.673%); 2.6 MiB/s, 41 objects/s recovering 2024-04-23T16:54:01.473 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:54:01.509 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:01.538 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:54:01.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:01.545 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:54:01.647 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:54:01.676 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:01.699 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:54:01.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:01.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:54:01.803 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:54:01.832 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:01.855 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:54:01.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:01.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:54:01.960 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:54:02.007 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:02.043 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:54:02.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:02.054 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:54:02.155 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:54:02.185 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:02.208 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:54:02.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:02.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:54:02.312 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:54:02.341 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:02.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:02 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:54:02.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:02 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:54:02.365 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:54:02.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:02.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:54:02.473 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:54:02.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:02 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:54:02.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:02 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:54:02.480 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-23T16:54:02.480 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:54:02.498 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:02 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:54:02.499 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:02 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:54:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:02.521 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:54:02.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:02.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:54:02.625 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:54:02.680 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:02.719 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:54:02.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:02.724 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:54:02.825 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:54:02.854 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:02.876 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:54:02.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:02.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:54:02.981 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:54:03.010 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:03.031 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:54:03.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:03.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:54:03.065 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:54:03.137 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:54:03.176 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:03.206 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:54:03.210 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-23T16:54:03.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:03.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:54:03.237 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.1 to 1 (10000) 2024-04-23T16:54:03.312 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:54:03.342 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:03.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:03 smithi103 ceph-mon[39629]: osdmap e758: 12 total, 12 up, 8 in 2024-04-23T16:54:03.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:03 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/163356415' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:54:03.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:03 smithi103 ceph-mon[39629]: from='client.37139 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:54:03.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:03 smithi103 ceph-mon[39629]: pgmap v2092: 27 pgs: 1 active+recovering, 26 active+clean; 5.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 815 op/s; 4934/190284 objects misplaced (2.593%); 3.2 MiB/s, 50 objects/s recovering 2024-04-23T16:54:03.362 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:54:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:03.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:54:03.467 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:54:03.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:03 smithi188 ceph-mon[44101]: osdmap e758: 12 total, 12 up, 8 in 2024-04-23T16:54:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:03 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/163356415' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:54:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:03 smithi188 ceph-mon[44101]: from='client.37139 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:54:03.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:03 smithi188 ceph-mon[44101]: pgmap v2092: 27 pgs: 1 active+recovering, 26 active+clean; 5.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 815 op/s; 4934/190284 objects misplaced (2.593%); 3.2 MiB/s, 50 objects/s recovering 2024-04-23T16:54:03.506 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:03.535 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:54:03.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:03.540 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:54:03.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:03 smithi148 ceph-mon[44520]: osdmap e758: 12 total, 12 up, 8 in 2024-04-23T16:54:03.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:03 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/163356415' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:54:03.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:03 smithi148 ceph-mon[44520]: from='client.37139 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:54:03.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:03 smithi148 ceph-mon[44520]: pgmap v2092: 27 pgs: 1 active+recovering, 26 active+clean; 5.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 815 op/s; 4934/190284 objects misplaced (2.593%); 3.2 MiB/s, 50 objects/s recovering 2024-04-23T16:54:03.641 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:54:03.669 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:03.694 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:54:03.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:03.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:54:03.799 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:54:03.827 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:03.848 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:54:03.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:03.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:54:03.952 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:54:03.995 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:04.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:04 smithi188 ceph-mon[44101]: from='client.37139 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-23T16:54:04.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:04 smithi188 ceph-mon[44101]: osdmap e759: 12 total, 12 up, 8 in 2024-04-23T16:54:04.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:04 smithi103 ceph-mon[39629]: from='client.37139 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-23T16:54:04.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:04 smithi103 ceph-mon[39629]: osdmap e759: 12 total, 12 up, 8 in 2024-04-23T16:54:04.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:04 smithi148 ceph-mon[44520]: from='client.37139 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-23T16:54:04.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:04 smithi148 ceph-mon[44520]: osdmap e759: 12 total, 12 up, 8 in 2024-04-23T16:54:05.057 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:54:05.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:05.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:54:05.173 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:54:05.208 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:05.235 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:54:05.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:05.238 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:54:05.339 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:54:05.387 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:05.424 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:54:05.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:05.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:54:05.529 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:54:05.565 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:05.652 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:54:05.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:05.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:54:05.757 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:54:05.785 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:05 smithi148 ceph-mon[44520]: pgmap v2094: 27 pgs: 1 active+recovering, 26 active+clean; 5.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 761 op/s; 4934/188436 objects misplaced (2.618%); 3.2 MiB/s, 50 objects/s recovering 2024-04-23T16:54:05.787 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:05.808 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:54:05.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:05.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:54:05.819 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:05 smithi103 ceph-mon[39629]: pgmap v2094: 27 pgs: 1 active+recovering, 26 active+clean; 5.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 761 op/s; 4934/188436 objects misplaced (2.618%); 3.2 MiB/s, 50 objects/s recovering 2024-04-23T16:54:05.912 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:54:05.941 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:05.963 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:54:05.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:05.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:54:05.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:05 smithi188 ceph-mon[44101]: pgmap v2094: 27 pgs: 1 active+recovering, 26 active+clean; 5.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 761 op/s; 4934/188436 objects misplaced (2.618%); 3.2 MiB/s, 50 objects/s recovering 2024-04-23T16:54:06.068 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:54:06.111 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:06.191 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:54:06.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:06.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:54:06.218 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-23T16:54:06.306 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:54:06.348 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:06.433 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:54:06.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:06.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:54:06.540 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:54:06.576 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:06.633 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:54:06.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:06.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:54:06.745 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:54:06.757 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:54:06.758 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:54:06.786 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:06.817 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:54:06.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:06.820 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:54:06.844 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:54:06.873 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:54:06.910 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:54:06.921 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:54:06.950 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:54:06.971 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:06.974 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:54:06.999 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:54:07.026 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:54:07.048 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:54:07.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:07.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:54:07.060 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:54:07.077 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:54:07.092 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:54:07.128 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:54:07.151 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:54:07.157 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:54:07.190 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:54:07.193 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:07.212 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:54:07.212 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:54:07.212 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:54:07.212 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:54:07.212 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:54:07.212 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:54:07.212 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:54:07.213 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:54:07.213 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:54:07.213 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:54:07.213 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:54:07.213 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:54:07.529 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:07 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:54:07.529 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:07 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:54:07.529 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:07 smithi103 ceph-mon[39629]: osdmap e760: 12 total, 12 up, 8 in 2024-04-23T16:54:07.529 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:07 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/960282684' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:54:07.529 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:07.529 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:07 smithi103 ceph-mon[39629]: pgmap v2096: 27 pgs: 1 active+recovering, 26 active+clean; 5.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 4595/181640 objects misplaced (2.530%); 6.7 MiB/s, 107 objects/s recovering 2024-04-23T16:54:07.529 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:07.529 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:07.529 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:07.530 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:07.530 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:07.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:07 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:54:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:07 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:54:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:07 smithi148 ceph-mon[44520]: osdmap e760: 12 total, 12 up, 8 in 2024-04-23T16:54:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:07 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/960282684' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:54:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:07 smithi148 ceph-mon[44520]: pgmap v2096: 27 pgs: 1 active+recovering, 26 active+clean; 5.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 4595/181640 objects misplaced (2.530%); 6.7 MiB/s, 107 objects/s recovering 2024-04-23T16:54:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:07.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:07 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:54:07.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:07 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:54:07.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:07 smithi188 ceph-mon[44101]: osdmap e760: 12 total, 12 up, 8 in 2024-04-23T16:54:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:07 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/960282684' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:54:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:07 smithi188 ceph-mon[44101]: pgmap v2096: 27 pgs: 1 active+recovering, 26 active+clean; 5.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 4595/181640 objects misplaced (2.530%); 6.7 MiB/s, 107 objects/s recovering 2024-04-23T16:54:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:08.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:08.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:08.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:08.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:08.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:08 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:08.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:08 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:08.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:08 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:08.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:08 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:08.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:08 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:08.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:08 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:08.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:08 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:08.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:08 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:08.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:08 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:08.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:08 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:08.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:08 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:09.539 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:09 smithi103 ceph-mon[39629]: pgmap v2097: 27 pgs: 1 active+recovering, 26 active+clean; 5.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 954 op/s; 4595/178558 objects misplaced (2.573%); 3.1 MiB/s, 50 objects/s recovering 2024-04-23T16:54:09.667 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:54:09.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:09.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:54:09.776 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:54:09.801 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:09 smithi148 ceph-mon[44520]: pgmap v2097: 27 pgs: 1 active+recovering, 26 active+clean; 5.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 954 op/s; 4595/178558 objects misplaced (2.573%); 3.1 MiB/s, 50 objects/s recovering 2024-04-23T16:54:09.803 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:09.828 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:54:09.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:09.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:54:09.934 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:54:09.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:09 smithi188 ceph-mon[44101]: pgmap v2097: 27 pgs: 1 active+recovering, 26 active+clean; 5.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 954 op/s; 4595/178558 objects misplaced (2.573%); 3.1 MiB/s, 50 objects/s recovering 2024-04-23T16:54:09.986 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:10.028 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:54:10.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:10.035 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:54:10.136 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:54:10.163 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:10.189 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:54:10.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:10.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:54:10.296 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:54:10.324 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:10.349 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:54:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:10.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:54:10.455 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:54:10.482 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:10.505 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:54:10.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:10.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:54:10.609 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:54:10.637 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:10.657 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:54:10.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:10.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:54:10.761 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:54:10.794 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:10.856 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:54:10.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:10.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:54:10.963 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:54:11.007 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:11.706 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:54:12.273 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:54:12.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:12.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:54:12.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:12 smithi103 ceph-mon[39629]: pgmap v2098: 27 pgs: 1 active+recovering, 26 active+clean; 5.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 844 op/s; 4595/176768 objects misplaced (2.599%); 2.6 MiB/s, 42 objects/s recovering 2024-04-23T16:54:12.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:12 smithi148 ceph-mon[44520]: pgmap v2098: 27 pgs: 1 active+recovering, 26 active+clean; 5.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 844 op/s; 4595/176768 objects misplaced (2.599%); 2.6 MiB/s, 42 objects/s recovering 2024-04-23T16:54:12.380 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:54:12.408 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:12.427 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:54:12.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:12.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:54:12.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:12 smithi188 ceph-mon[44101]: pgmap v2098: 27 pgs: 1 active+recovering, 26 active+clean; 5.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 844 op/s; 4595/176768 objects misplaced (2.599%); 2.6 MiB/s, 42 objects/s recovering 2024-04-23T16:54:12.531 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:54:12.569 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:12.771 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:54:12.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:12.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:54:12.876 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:54:12.921 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:12.970 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:54:12.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:12.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:54:13.081 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:54:13.144 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:14.081 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:54:14.101 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.2 to 1 (10000) 2024-04-23T16:54:14.225 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:54:14.226 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:54:14.226 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:54:14.226 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:54:14.226 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.226 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:199513 8.e 8:7084a617:::benchmark_data_smithi188_155546_object74004:head [delete] snapc 0=[] ondisk+write+known_if_redirected e673)", 2024-04-23T16:54:14.226 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:49:44.361320+0000", 2024-04-23T16:54:14.226 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 269.86024527900003, 2024-04-23T16:54:14.226 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.26868059700000002, 2024-04-23T16:54:14.226 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.227 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.227 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:14.227 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:54:14.227 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:54:14.227 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:14.227 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 199513 2024-04-23T16:54:14.227 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.227 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.227 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.227 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.227 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.361320+0000", 2024-04-23T16:54:14.227 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.228 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.228 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.228 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.228 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.361320+0000", 2024-04-23T16:54:14.228 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.228 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.228 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.228 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.228 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.361320+0000", 2024-04-23T16:54:14.228 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2300000000000002e-07 2024-04-23T16:54:14.228 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.229 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.229 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.229 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.361322+0000", 2024-04-23T16:54:14.229 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.869e-06 2024-04-23T16:54:14.229 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.229 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.229 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.229 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.361325+0000", 2024-04-23T16:54:14.229 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.676e-06 2024-04-23T16:54:14.229 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.229 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.229 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.229 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.361329+0000", 2024-04-23T16:54:14.230 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5889999999999999e-06 2024-04-23T16:54:14.230 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.230 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.230 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.230 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.365158+0000", 2024-04-23T16:54:14.230 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038293300000000001 2024-04-23T16:54:14.230 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.230 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.230 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:54:14.230 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.365235+0000", 2024-04-23T16:54:14.230 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6512000000000002e-05 2024-04-23T16:54:14.230 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.230 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.231 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:54:14.231 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.365310+0000", 2024-04-23T16:54:14.231 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5340999999999996e-05 2024-04-23T16:54:14.231 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.231 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.231 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:54:14.231 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.368418+0000", 2024-04-23T16:54:14.231 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0031074480000000001 2024-04-23T16:54:14.231 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.231 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.231 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:54:14.231 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.368680+0000", 2024-04-23T16:54:14.232 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026259900000000002 2024-04-23T16:54:14.232 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.232 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.232 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:54:14.232 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.368704+0000", 2024-04-23T16:54:14.232 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3802999999999999e-05 2024-04-23T16:54:14.232 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.232 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.232 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.232 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.630000+0000", 2024-04-23T16:54:14.232 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.261296007 2024-04-23T16:54:14.232 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.233 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.233 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.233 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.233 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.233 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop_reply(client.16995.0:199513 8.e e673/609 ondisk, result = 0)", 2024-04-23T16:54:14.233 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:49:44.368634+0000", 2024-04-23T16:54:14.233 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 269.85293117700002, 2024-04-23T16:54:14.233 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.26137463599999999, 2024-04-23T16:54:14.233 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.233 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.233 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:54:14.233 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.233 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.234 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.234 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.368634+0000", 2024-04-23T16:54:14.234 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.234 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.234 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.234 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.234 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.368634+0000", 2024-04-23T16:54:14.234 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.234 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.234 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.234 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.234 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.368634+0000", 2024-04-23T16:54:14.234 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4899999999999995e-07 2024-04-23T16:54:14.235 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.235 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.235 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.235 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.368637+0000", 2024-04-23T16:54:14.235 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5040000000000001e-06 2024-04-23T16:54:14.235 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.235 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.235 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.235 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.368642+0000", 2024-04-23T16:54:14.235 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8189999999999999e-06 2024-04-23T16:54:14.235 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.235 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.236 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.236 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.368645+0000", 2024-04-23T16:54:14.236 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.473e-06 2024-04-23T16:54:14.236 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.236 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.236 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.236 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.368674+0000", 2024-04-23T16:54:14.236 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9215999999999999e-05 2024-04-23T16:54:14.236 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.236 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.236 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:54:14.236 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.368678+0000", 2024-04-23T16:54:14.236 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5039999999999998e-06 2024-04-23T16:54:14.237 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.237 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.237 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.237 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.630008+0000", 2024-04-23T16:54:14.237 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.26133047100000001 2024-04-23T16:54:14.237 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.237 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.237 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.237 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.237 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.237 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:199533 8.e 8:789d84c9:::benchmark_data_smithi188_155546_object74024:head [delete] snapc 0=[] ondisk+write+known_if_redirected e673)", 2024-04-23T16:54:14.237 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:49:44.383441+0000", 2024-04-23T16:54:14.237 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 269.83812345400003, 2024-04-23T16:54:14.238 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.255368031, 2024-04-23T16:54:14.238 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.238 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.238 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:14.238 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:54:14.238 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:54:14.238 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:14.238 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 199533 2024-04-23T16:54:14.238 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.238 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.238 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.238 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.239 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.383441+0000", 2024-04-23T16:54:14.239 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.239 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.239 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.239 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.239 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.383441+0000", 2024-04-23T16:54:14.239 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.239 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.239 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.239 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.239 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.383443+0000", 2024-04-23T16:54:14.239 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7749999999999999e-06 2024-04-23T16:54:14.239 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.240 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.240 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.240 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.383449+0000", 2024-04-23T16:54:14.240 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7230000000000001e-06 2024-04-23T16:54:14.240 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.240 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.240 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.240 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.629988+0000", 2024-04-23T16:54:14.240 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.246539341 2024-04-23T16:54:14.240 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.240 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.240 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.240 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.630001+0000", 2024-04-23T16:54:14.240 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2353e-05 2024-04-23T16:54:14.240 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.241 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.241 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.241 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.630130+0000", 2024-04-23T16:54:14.241 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000129073 2024-04-23T16:54:14.241 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.241 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.241 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:54:14.241 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.630262+0000", 2024-04-23T16:54:14.241 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013241899999999999 2024-04-23T16:54:14.241 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.241 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.241 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 4", 2024-04-23T16:54:14.241 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.630385+0000", 2024-04-23T16:54:14.242 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012259 2024-04-23T16:54:14.242 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.242 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.242 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:54:14.242 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.638738+0000", 2024-04-23T16:54:14.242 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083536170000000007 2024-04-23T16:54:14.242 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.242 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.242 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:54:14.242 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.638750+0000", 2024-04-23T16:54:14.242 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1416999999999999e-05 2024-04-23T16:54:14.242 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.242 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.243 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:54:14.243 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.638781+0000", 2024-04-23T16:54:14.243 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1559000000000001e-05 2024-04-23T16:54:14.243 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.243 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.243 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.243 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.638809+0000", 2024-04-23T16:54:14.243 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8164e-05 2024-04-23T16:54:14.243 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.243 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.243 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.243 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.243 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.243 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.16995.0:199532 8.9 e673/623 8:988418fc:::benchmark_data_smithi188_155546_object74023:head v 673'12741, mlcod=673'12739)", 2024-04-23T16:54:14.244 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:49:44.384094+0000", 2024-04-23T16:54:14.244 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 269.83747084499998, 2024-04-23T16:54:14.244 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.26875693899999997, 2024-04-23T16:54:14.244 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.244 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.244 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:14.244 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.244 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.244 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.244 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.384094+0000", 2024-04-23T16:54:14.244 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.244 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.244 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.244 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.245 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.384094+0000", 2024-04-23T16:54:14.245 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.245 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.245 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.245 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.245 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.384095+0000", 2024-04-23T16:54:14.245 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.53e-06 2024-04-23T16:54:14.245 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.245 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.245 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.245 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.384104+0000", 2024-04-23T16:54:14.245 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6440000000000006e-06 2024-04-23T16:54:14.245 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.245 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.246 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.246 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.629990+0000", 2024-04-23T16:54:14.246 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.24588633500000001 2024-04-23T16:54:14.246 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.246 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.246 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.246 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.630000+0000", 2024-04-23T16:54:14.246 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4760000000000004e-06 2024-04-23T16:54:14.246 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.246 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.246 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.246 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.630051+0000", 2024-04-23T16:54:14.246 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1542000000000002e-05 2024-04-23T16:54:14.246 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.247 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:54:14.247 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.630073+0000", 2024-04-23T16:54:14.247 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1846000000000001e-05 2024-04-23T16:54:14.247 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.247 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:54:14.247 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.652815+0000", 2024-04-23T16:54:14.247 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.022741939999999999 2024-04-23T16:54:14.247 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.247 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.247 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:44.652851+0000", 2024-04-23T16:54:14.247 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5626000000000001e-05 2024-04-23T16:54:14.248 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.248 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.248 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.248 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.248 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.248 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.16995.0:219366 8.9 e678/623 8:9f30d106:::benchmark_data_smithi188_155546_object93857:head v 678'14026, mlcod=678'14024)", 2024-04-23T16:54:14.248 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:50:05.551366+0000", 2024-04-23T16:54:14.248 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 248.67019882, 2024-04-23T16:54:14.248 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.094025423999999996, 2024-04-23T16:54:14.248 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.248 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.248 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:14.248 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.248 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.249 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.249 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:05.551366+0000", 2024-04-23T16:54:14.249 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.249 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.249 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.249 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:05.551366+0000", 2024-04-23T16:54:14.249 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.249 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.249 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.249 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:05.551369+0000", 2024-04-23T16:54:14.249 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9069999999999999e-06 2024-04-23T16:54:14.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.250 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.250 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.250 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:05.551392+0000", 2024-04-23T16:54:14.250 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2849000000000001e-05 2024-04-23T16:54:14.250 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.250 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.250 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.250 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:05.551401+0000", 2024-04-23T16:54:14.250 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9749999999999996e-06 2024-04-23T16:54:14.250 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.250 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.250 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.251 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:05.551406+0000", 2024-04-23T16:54:14.251 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4129999999999997e-06 2024-04-23T16:54:14.251 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.251 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.251 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.251 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:05.551452+0000", 2024-04-23T16:54:14.251 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5667999999999997e-05 2024-04-23T16:54:14.251 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.251 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.251 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:54:14.251 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:05.551484+0000", 2024-04-23T16:54:14.251 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2668e-05 2024-04-23T16:54:14.251 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.251 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.252 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:54:14.252 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:05.645362+0000", 2024-04-23T16:54:14.252 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.093877523000000004 2024-04-23T16:54:14.252 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.252 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.252 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.252 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:05.645391+0000", 2024-04-23T16:54:14.252 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9420999999999998e-05 2024-04-23T16:54:14.252 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.252 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.252 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.252 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.252 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.252 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:245133 8.2 8.6acb39c2 (undecoded) ondisk+retry+write+known_if_redirected e692)", 2024-04-23T16:54:14.252 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:50:50.605178+0000", 2024-04-23T16:54:14.253 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 203.616386715, 2024-04-23T16:54:14.253 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0013688810000001, 2024-04-23T16:54:14.253 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.253 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.253 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-23T16:54:14.253 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:54:14.253 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:54:14.253 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:14.253 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 245133 2024-04-23T16:54:14.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.253 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.253 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.253 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.253 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605178+0000", 2024-04-23T16:54:14.254 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.254 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.254 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.254 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.254 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605178+0000", 2024-04-23T16:54:14.254 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.254 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.254 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.254 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.254 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605180+0000", 2024-04-23T16:54:14.254 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7120000000000001e-06 2024-04-23T16:54:14.254 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.254 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.254 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.255 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605184+0000", 2024-04-23T16:54:14.255 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2889999999999999e-06 2024-04-23T16:54:14.255 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.255 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.255 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.255 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605194+0000", 2024-04-23T16:54:14.255 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.012e-05 2024-04-23T16:54:14.255 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.255 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.255 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.255 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605201+0000", 2024-04-23T16:54:14.255 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8499999999999996e-06 2024-04-23T16:54:14.255 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.255 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.256 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.256 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605277+0000", 2024-04-23T16:54:14.256 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6327000000000003e-05 2024-04-23T16:54:14.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.256 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:54:14.256 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605288+0000", 2024-04-23T16:54:14.256 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.043e-05 2024-04-23T16:54:14.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.256 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.256 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:51.606517+0000", 2024-04-23T16:54:14.256 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0012294509999999 2024-04-23T16:54:14.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.257 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.257 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:51.606547+0000", 2024-04-23T16:54:14.257 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9702000000000001e-05 2024-04-23T16:54:14.257 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.257 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.257 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.257 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.257 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.257 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:245147 8.2 8.4e1183d2 (undecoded) ondisk+retry+write+known_if_redirected e692)", 2024-04-23T16:54:14.257 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:50:50.605213+0000", 2024-04-23T16:54:14.257 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 203.61635133300001, 2024-04-23T16:54:14.257 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.001399785, 2024-04-23T16:54:14.257 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.258 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.258 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-23T16:54:14.258 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:54:14.258 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:54:14.258 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:14.258 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 245147 2024-04-23T16:54:14.258 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.258 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.258 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.258 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.258 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605213+0000", 2024-04-23T16:54:14.258 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.258 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.259 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.259 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.259 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605213+0000", 2024-04-23T16:54:14.259 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.259 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.259 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.259 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.259 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605214+0000", 2024-04-23T16:54:14.259 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9700000000000002e-07 2024-04-23T16:54:14.259 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.259 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.259 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.259 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605216+0000", 2024-04-23T16:54:14.259 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9999999999999999e-06 2024-04-23T16:54:14.259 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.260 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.260 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.260 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605219+0000", 2024-04-23T16:54:14.260 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6109999999999998e-06 2024-04-23T16:54:14.260 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.260 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.260 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.260 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605225+0000", 2024-04-23T16:54:14.260 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0560000000000004e-06 2024-04-23T16:54:14.260 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.260 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.260 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.260 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605331+0000", 2024-04-23T16:54:14.260 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000106011 2024-04-23T16:54:14.260 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.261 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.261 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:54:14.261 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605339+0000", 2024-04-23T16:54:14.261 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0930000000000003e-06 2024-04-23T16:54:14.261 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.261 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.261 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.261 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:51.606601+0000", 2024-04-23T16:54:14.261 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.001262798 2024-04-23T16:54:14.261 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.261 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.261 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.261 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:51.606613+0000", 2024-04-23T16:54:14.261 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1819e-05 2024-04-23T16:54:14.261 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.262 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.262 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.262 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.262 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:245166 8.2 8.850d8c52 (undecoded) ondisk+retry+write+known_if_redirected e692)", 2024-04-23T16:54:14.262 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:50:50.605232+0000", 2024-04-23T16:54:14.262 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 203.616333013, 2024-04-23T16:54:14.262 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0014255359999999, 2024-04-23T16:54:14.262 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.262 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.262 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-23T16:54:14.262 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:54:14.262 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:54:14.262 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:14.263 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 245166 2024-04-23T16:54:14.263 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.263 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.263 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.263 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.263 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605232+0000", 2024-04-23T16:54:14.263 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.263 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.263 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.263 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.263 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605232+0000", 2024-04-23T16:54:14.263 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.263 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.263 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.263 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.263 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605232+0000", 2024-04-23T16:54:14.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6800000000000001e-07 2024-04-23T16:54:14.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.264 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605235+0000", 2024-04-23T16:54:14.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5399999999999998e-06 2024-04-23T16:54:14.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.264 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605241+0000", 2024-04-23T16:54:14.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.1959999999999996e-06 2024-04-23T16:54:14.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.264 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605255+0000", 2024-04-23T16:54:14.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3838e-05 2024-04-23T16:54:14.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.265 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605361+0000", 2024-04-23T16:54:14.265 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105769 2024-04-23T16:54:14.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.265 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:54:14.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:50.605367+0000", 2024-04-23T16:54:14.265 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.46e-06 2024-04-23T16:54:14.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.265 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:51.606645+0000", 2024-04-23T16:54:14.266 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.001278412 2024-04-23T16:54:14.266 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.266 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.266 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.266 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:51.606657+0000", 2024-04-23T16:54:14.266 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1853e-05 2024-04-23T16:54:14.266 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.266 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.266 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.266 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.266 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.266 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:245142 8.e 8.69305a9e (undecoded) ondisk+retry+write+known_if_redirected e694)", 2024-04-23T16:54:14.266 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:50:52.645764+0000", 2024-04-23T16:54:14.266 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 201.575801195, 2024-04-23T16:54:14.266 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.30644914899999998, 2024-04-23T16:54:14.267 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.267 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.267 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-23T16:54:14.267 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:54:14.267 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:54:14.267 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:14.267 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 245142 2024-04-23T16:54:14.267 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.267 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.267 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.267 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.267 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.645764+0000", 2024-04-23T16:54:14.267 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.268 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.268 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.645764+0000", 2024-04-23T16:54:14.268 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.268 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.268 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.645767+0000", 2024-04-23T16:54:14.268 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3699999999999999e-06 2024-04-23T16:54:14.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.268 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.268 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.645778+0000", 2024-04-23T16:54:14.269 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.129e-05 2024-04-23T16:54:14.269 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.269 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.269 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.269 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.645792+0000", 2024-04-23T16:54:14.269 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3883000000000001e-05 2024-04-23T16:54:14.269 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.269 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.269 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.269 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.645806+0000", 2024-04-23T16:54:14.269 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3966e-05 2024-04-23T16:54:14.269 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.269 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.269 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.269 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646017+0000", 2024-04-23T16:54:14.270 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000210428 2024-04-23T16:54:14.270 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.270 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.270 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:54:14.270 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646028+0000", 2024-04-23T16:54:14.270 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.189e-05 2024-04-23T16:54:14.270 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.270 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.270 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.270 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.952187+0000", 2024-04-23T16:54:14.270 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.30615880400000001 2024-04-23T16:54:14.270 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.271 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.271 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.271 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.952213+0000", 2024-04-23T16:54:14.271 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5517999999999999e-05 2024-04-23T16:54:14.271 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.271 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.271 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.271 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.271 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:245146 8.e 8.54ab1a0e (undecoded) ondisk+retry+write+known_if_redirected e694)", 2024-04-23T16:54:14.271 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:50:52.645839+0000", 2024-04-23T16:54:14.271 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 201.57572560099999, 2024-04-23T16:54:14.271 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.30654451399999999, 2024-04-23T16:54:14.272 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.272 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.272 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-23T16:54:14.272 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:54:14.272 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:54:14.272 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:14.272 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 245146 2024-04-23T16:54:14.272 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.272 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.272 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.272 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.272 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.645839+0000", 2024-04-23T16:54:14.272 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.272 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.273 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.273 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.273 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.645839+0000", 2024-04-23T16:54:14.273 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.273 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.273 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.273 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.273 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.645840+0000", 2024-04-23T16:54:14.273 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9899999999999999e-07 2024-04-23T16:54:14.273 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.273 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.273 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.273 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.645843+0000", 2024-04-23T16:54:14.273 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4699999999999998e-06 2024-04-23T16:54:14.274 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.274 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.274 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.274 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.645850+0000", 2024-04-23T16:54:14.274 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0079999999999996e-06 2024-04-23T16:54:14.274 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.274 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.274 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.274 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.645857+0000", 2024-04-23T16:54:14.274 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2960000000000004e-06 2024-04-23T16:54:14.274 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.274 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.274 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.274 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646055+0000", 2024-04-23T16:54:14.275 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019835799999999999 2024-04-23T16:54:14.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.275 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:54:14.275 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646063+0000", 2024-04-23T16:54:14.275 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.4850000000000003e-06 2024-04-23T16:54:14.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.275 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.275 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.952369+0000", 2024-04-23T16:54:14.275 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.30630618399999998 2024-04-23T16:54:14.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.275 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.276 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.952384+0000", 2024-04-23T16:54:14.276 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4914e-05 2024-04-23T16:54:14.276 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.276 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.276 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.276 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.276 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.276 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:245148 8.e 8.887c9b0e (undecoded) ondisk+retry+write+known_if_redirected e694)", 2024-04-23T16:54:14.276 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:50:52.645865+0000", 2024-04-23T16:54:14.276 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 201.575699864, 2024-04-23T16:54:14.276 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.30658556199999998, 2024-04-23T16:54:14.276 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.276 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.276 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-23T16:54:14.277 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:54:14.277 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:54:14.277 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:14.277 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 245148 2024-04-23T16:54:14.277 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.277 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.277 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.277 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.277 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.645865+0000", 2024-04-23T16:54:14.277 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.277 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.277 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.277 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.277 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.645865+0000", 2024-04-23T16:54:14.277 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.278 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.278 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.278 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.278 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.645866+0000", 2024-04-23T16:54:14.278 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7599999999999997e-07 2024-04-23T16:54:14.278 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.278 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.278 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.278 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.645868+0000", 2024-04-23T16:54:14.278 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9629999999999998e-06 2024-04-23T16:54:14.278 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.278 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.278 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.278 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.645876+0000", 2024-04-23T16:54:14.279 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0339999999999997e-06 2024-04-23T16:54:14.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.279 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.645883+0000", 2024-04-23T16:54:14.279 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.956e-06 2024-04-23T16:54:14.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.279 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646086+0000", 2024-04-23T16:54:14.279 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020230700000000001 2024-04-23T16:54:14.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:54:14.280 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646093+0000", 2024-04-23T16:54:14.280 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0659999999999998e-06 2024-04-23T16:54:14.280 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.280 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.280 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.280 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.952440+0000", 2024-04-23T16:54:14.280 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.30634745400000002 2024-04-23T16:54:14.280 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.280 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.280 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.280 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.952450+0000", 2024-04-23T16:54:14.280 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0206e-05 2024-04-23T16:54:14.280 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.280 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.280 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.281 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.281 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:245133 8.2 8.6acb39c2 (undecoded) ondisk+retry+write+known_if_redirected e694)", 2024-04-23T16:54:14.281 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:50:52.646334+0000", 2024-04-23T16:54:14.281 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 201.57523119999999, 2024-04-23T16:54:14.281 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.148010324, 2024-04-23T16:54:14.281 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.281 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.281 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-23T16:54:14.281 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:54:14.281 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:54:14.281 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:14.281 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 245133 2024-04-23T16:54:14.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.282 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.282 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.282 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.282 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646334+0000", 2024-04-23T16:54:14.282 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.282 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.282 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.282 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646334+0000", 2024-04-23T16:54:14.282 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.282 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.282 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.283 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646335+0000", 2024-04-23T16:54:14.283 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.079e-06 2024-04-23T16:54:14.283 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.283 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.283 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.283 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646338+0000", 2024-04-23T16:54:14.283 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.585e-06 2024-04-23T16:54:14.283 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.283 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.283 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.283 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646346+0000", 2024-04-23T16:54:14.283 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6450000000000005e-06 2024-04-23T16:54:14.283 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.283 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.284 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.284 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646353+0000", 2024-04-23T16:54:14.284 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8410000000000002e-06 2024-04-23T16:54:14.284 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.284 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.284 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.284 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646402+0000", 2024-04-23T16:54:14.284 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9188999999999999e-05 2024-04-23T16:54:14.284 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.284 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.284 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:54:14.284 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646411+0000", 2024-04-23T16:54:14.284 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0710000000000001e-06 2024-04-23T16:54:14.284 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.285 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.285 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.794320+0000", 2024-04-23T16:54:14.285 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.147909441 2024-04-23T16:54:14.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.285 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.285 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.794344+0000", 2024-04-23T16:54:14.285 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3473e-05 2024-04-23T16:54:14.285 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.285 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.285 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.286 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:245147 8.2 8.4e1183d2 (undecoded) ondisk+retry+write+known_if_redirected e694)", 2024-04-23T16:54:14.286 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:50:52.646366+0000", 2024-04-23T16:54:14.286 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 201.575198895, 2024-04-23T16:54:14.286 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.14811496399999999, 2024-04-23T16:54:14.286 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.286 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.286 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-23T16:54:14.286 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:54:14.286 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:54:14.286 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:14.286 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 245147 2024-04-23T16:54:14.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.286 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.286 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.287 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.287 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646366+0000", 2024-04-23T16:54:14.287 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.287 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.287 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.287 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.287 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646366+0000", 2024-04-23T16:54:14.287 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.287 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.287 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.287 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.287 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646367+0000", 2024-04-23T16:54:14.287 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6000000000000003e-07 2024-04-23T16:54:14.287 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.288 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.288 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.288 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646370+0000", 2024-04-23T16:54:14.288 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3009999999999998e-06 2024-04-23T16:54:14.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.288 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.288 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.288 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646376+0000", 2024-04-23T16:54:14.288 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5930000000000002e-06 2024-04-23T16:54:14.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.288 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.288 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.288 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646382+0000", 2024-04-23T16:54:14.288 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7060000000000001e-06 2024-04-23T16:54:14.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.289 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.289 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.289 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646451+0000", 2024-04-23T16:54:14.289 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8757999999999995e-05 2024-04-23T16:54:14.289 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.289 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.289 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:54:14.289 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646473+0000", 2024-04-23T16:54:14.289 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2269e-05 2024-04-23T16:54:14.289 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.289 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.289 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.289 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.794475+0000", 2024-04-23T16:54:14.289 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.148001681 2024-04-23T16:54:14.290 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.290 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.290 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.290 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.794481+0000", 2024-04-23T16:54:14.290 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9959999999999999e-06 2024-04-23T16:54:14.290 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.290 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.290 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.290 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.290 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.290 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:245166 8.2 8.850d8c52 (undecoded) ondisk+retry+write+known_if_redirected e694)", 2024-04-23T16:54:14.290 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:50:52.646391+0000", 2024-04-23T16:54:14.290 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 201.57517343999999, 2024-04-23T16:54:14.291 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.14813042000000001, 2024-04-23T16:54:14.291 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.291 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.291 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-23T16:54:14.291 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:54:14.291 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:54:14.291 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:14.291 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 245166 2024-04-23T16:54:14.291 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.291 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.291 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.291 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.291 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646391+0000", 2024-04-23T16:54:14.292 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.292 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646391+0000", 2024-04-23T16:54:14.292 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.292 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646392+0000", 2024-04-23T16:54:14.292 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5799999999999999e-07 2024-04-23T16:54:14.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646395+0000", 2024-04-23T16:54:14.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.745e-06 2024-04-23T16:54:14.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.293 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646400+0000", 2024-04-23T16:54:14.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5099999999999998e-06 2024-04-23T16:54:14.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.293 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646406+0000", 2024-04-23T16:54:14.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8560000000000003e-06 2024-04-23T16:54:14.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.294 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.294 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646500+0000", 2024-04-23T16:54:14.294 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4073000000000005e-05 2024-04-23T16:54:14.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.294 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.294 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:54:14.294 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.646508+0000", 2024-04-23T16:54:14.294 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.593e-06 2024-04-23T16:54:14.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.294 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.294 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.294 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.794515+0000", 2024-04-23T16:54:14.294 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.14800712399999999 2024-04-23T16:54:14.295 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.295 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.295 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.295 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.794522+0000", 2024-04-23T16:54:14.295 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9609999999999999e-06 2024-04-23T16:54:14.295 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.295 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.295 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.295 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.295 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.295 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.27202.0:151947 9.c e740/703 9:3d4fba10:::benchmark_data_smithi188_167360_object814:head v 740'9554, mlcod=740'9552)", 2024-04-23T16:54:14.295 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:52:59.840706+0000", 2024-04-23T16:54:14.295 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.380858513999996, 2024-04-23T16:54:14.295 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.022448837999999999, 2024-04-23T16:54:14.296 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.296 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.296 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:14.296 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.296 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.296 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.296 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:59.840706+0000", 2024-04-23T16:54:14.296 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.296 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.296 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.296 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.296 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:59.840706+0000", 2024-04-23T16:54:14.296 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.296 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.296 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.297 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.297 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:59.840716+0000", 2024-04-23T16:54:14.297 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6670000000000008e-06 2024-04-23T16:54:14.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.297 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.297 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.297 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:59.840729+0000", 2024-04-23T16:54:14.297 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2565000000000001e-05 2024-04-23T16:54:14.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.297 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.297 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.297 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:59.840749+0000", 2024-04-23T16:54:14.297 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0018999999999999e-05 2024-04-23T16:54:14.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.298 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:59.840765+0000", 2024-04-23T16:54:14.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6555999999999999e-05 2024-04-23T16:54:14.298 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.298 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:59.840837+0000", 2024-04-23T16:54:14.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1419000000000002e-05 2024-04-23T16:54:14.298 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.298 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:54:14.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:59.840860+0000", 2024-04-23T16:54:14.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3431e-05 2024-04-23T16:54:14.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.299 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.299 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:54:14.299 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:59.863110+0000", 2024-04-23T16:54:14.299 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.02225042 2024-04-23T16:54:14.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.299 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.299 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.299 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:52:59.863155+0000", 2024-04-23T16:54:14.299 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4761000000000001e-05 2024-04-23T16:54:14.299 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.299 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.299 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.300 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.300 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.27202.0:154987 9.c e740/703 9:3a4fc15b:::benchmark_data_smithi188_167360_object3854:head v 740'9732, mlcod=740'9730)", 2024-04-23T16:54:14.300 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:53:02.683823+0000", 2024-04-23T16:54:14.300 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 71.537741401999995, 2024-04-23T16:54:14.300 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.023142539, 2024-04-23T16:54:14.300 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.300 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.300 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:14.300 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.300 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.300 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.300 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:02.683823+0000", 2024-04-23T16:54:14.300 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.300 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.301 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.301 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.301 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:02.683823+0000", 2024-04-23T16:54:14.301 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.301 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.301 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.301 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.301 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:02.683826+0000", 2024-04-23T16:54:14.301 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.526e-06 2024-04-23T16:54:14.301 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.301 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.301 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.301 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:02.683837+0000", 2024-04-23T16:54:14.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1036000000000001e-05 2024-04-23T16:54:14.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.302 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:02.683854+0000", 2024-04-23T16:54:14.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7436999999999999e-05 2024-04-23T16:54:14.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.302 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:02.683864+0000", 2024-04-23T16:54:14.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2080000000000006e-06 2024-04-23T16:54:14.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:02.683932+0000", 2024-04-23T16:54:14.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8300999999999995e-05 2024-04-23T16:54:14.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:54:14.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:02.683952+0000", 2024-04-23T16:54:14.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0499000000000001e-05 2024-04-23T16:54:14.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:54:14.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:02.706912+0000", 2024-04-23T16:54:14.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.022959192 2024-04-23T16:54:14.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:02.706966+0000", 2024-04-23T16:54:14.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4339999999999998e-05 2024-04-23T16:54:14.304 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.304 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.304 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.304 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.27202.0:174922 9.c e744/703 9:375ac669:::benchmark_data_smithi188_167360_object23789:head v 744'10970, mlcod=744'10966)", 2024-04-23T16:54:14.304 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:53:24.317194+0000", 2024-04-23T16:54:14.304 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 49.904371157, 2024-04-23T16:54:14.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.023029113, 2024-04-23T16:54:14.304 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.304 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.304 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:14.305 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.305 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:24.317194+0000", 2024-04-23T16:54:14.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.305 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:24.317194+0000", 2024-04-23T16:54:14.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.306 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:24.317196+0000", 2024-04-23T16:54:14.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1789999999999998e-06 2024-04-23T16:54:14.306 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.306 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.306 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.306 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:24.317205+0000", 2024-04-23T16:54:14.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3670000000000003e-06 2024-04-23T16:54:14.306 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.306 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.306 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.306 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:24.317225+0000", 2024-04-23T16:54:14.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9383999999999999e-05 2024-04-23T16:54:14.306 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.306 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:24.317233+0000", 2024-04-23T16:54:14.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6980000000000004e-06 2024-04-23T16:54:14.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:24.317289+0000", 2024-04-23T16:54:14.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6047000000000001e-05 2024-04-23T16:54:14.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:54:14.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:24.317307+0000", 2024-04-23T16:54:14.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7689999999999998e-05 2024-04-23T16:54:14.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:54:14.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:24.340157+0000", 2024-04-23T16:54:14.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.022850321 2024-04-23T16:54:14.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:24.340223+0000", 2024-04-23T16:54:14.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5426999999999995e-05 2024-04-23T16:54:14.308 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.308 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.308 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.308 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "rep_scrubmap(1.2 e306 from shard 7)", 2024-04-23T16:54:14.309 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:53:25.497638+0000", 2024-04-23T16:54:14.309 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.723926571, 2024-04-23T16:54:14.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.065583773999999997, 2024-04-23T16:54:14.309 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.309 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.309 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:54:14.309 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.309 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:25.497638+0000", 2024-04-23T16:54:14.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.309 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:25.497638+0000", 2024-04-23T16:54:14.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.310 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.310 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.310 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:25.497641+0000", 2024-04-23T16:54:14.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3910000000000001e-06 2024-04-23T16:54:14.310 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.310 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.310 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:25.497651+0000", 2024-04-23T16:54:14.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9569999999999997e-06 2024-04-23T16:54:14.310 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.310 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.311 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:25.497672+0000", 2024-04-23T16:54:14.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0968e-05 2024-04-23T16:54:14.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.311 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.311 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:25.497683+0000", 2024-04-23T16:54:14.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.113e-05 2024-04-23T16:54:14.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.311 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.311 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:25.563180+0000", 2024-04-23T16:54:14.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.065497110999999997 2024-04-23T16:54:14.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.311 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:54:14.312 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:25.563191+0000", 2024-04-23T16:54:14.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1311999999999999e-05 2024-04-23T16:54:14.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.312 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.312 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.312 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:25.563222+0000", 2024-04-23T16:54:14.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0905000000000003e-05 2024-04-23T16:54:14.312 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.312 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.312 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.312 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.312 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.27202.0:195605 9.c e751/703 9:3a930728:::benchmark_data_smithi188_167360_object44472:head v 751'12329, mlcod=751'12327)", 2024-04-23T16:54:14.312 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:53:46.859416+0000", 2024-04-23T16:54:14.313 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.362148301000001, 2024-04-23T16:54:14.313 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.022172984, 2024-04-23T16:54:14.313 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.313 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.313 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:14.313 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.313 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.313 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.313 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:46.859416+0000", 2024-04-23T16:54:14.313 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.313 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.313 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.313 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.313 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:46.859416+0000", 2024-04-23T16:54:14.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.314 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.314 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.314 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:46.859419+0000", 2024-04-23T16:54:14.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6989999999999999e-06 2024-04-23T16:54:14.314 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.314 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.314 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:46.859432+0000", 2024-04-23T16:54:14.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2821e-05 2024-04-23T16:54:14.314 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.314 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:46.859458+0000", 2024-04-23T16:54:14.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5582999999999999e-05 2024-04-23T16:54:14.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:46.859469+0000", 2024-04-23T16:54:14.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1143e-05 2024-04-23T16:54:14.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:46.863533+0000", 2024-04-23T16:54:14.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0040639450000000002 2024-04-23T16:54:14.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:54:14.316 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:46.863555+0000", 2024-04-23T16:54:14.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2708e-05 2024-04-23T16:54:14.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.316 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:54:14.316 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:46.881546+0000", 2024-04-23T16:54:14.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017990697999999999 2024-04-23T16:54:14.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.316 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.316 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:53:46.881589+0000", 2024-04-23T16:54:14.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3387000000000003e-05 2024-04-23T16:54:14.316 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.316 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.316 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.317 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.27202.0:219472 9.c e760/703 9:3e861296:::benchmark_data_smithi188_167360_object68339:head v 760'13784, mlcod=760'13782)", 2024-04-23T16:54:14.317 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:54:13.216020+0000", 2024-04-23T16:54:14.317 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.0055447289999999, 2024-04-23T16:54:14.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.022286733999999999, 2024-04-23T16:54:14.317 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:54:14.317 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:54:14.317 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:14.317 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:54:14.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:54:14.318 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:13.216020+0000", 2024-04-23T16:54:14.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.318 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:54:14.318 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:13.216020+0000", 2024-04-23T16:54:14.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:54:14.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.318 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:54:14.318 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:13.216023+0000", 2024-04-23T16:54:14.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4789999999999999e-06 2024-04-23T16:54:14.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:54:14.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:13.216034+0000", 2024-04-23T16:54:14.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1019000000000001e-05 2024-04-23T16:54:14.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:54:14.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:13.216056+0000", 2024-04-23T16:54:14.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2150999999999998e-05 2024-04-23T16:54:14.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:54:14.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:13.216065+0000", 2024-04-23T16:54:14.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4919999999999999e-06 2024-04-23T16:54:14.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:54:14.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:13.219473+0000", 2024-04-23T16:54:14.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034075440000000002 2024-04-23T16:54:14.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:54:14.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:13.219500+0000", 2024-04-23T16:54:14.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7127999999999998e-05 2024-04-23T16:54:14.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:54:14.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:13.238242+0000", 2024-04-23T16:54:14.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018742183999999999 2024-04-23T16:54:14.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:54:14.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:54:14.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:54:14.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:13.238307+0000", 2024-04-23T16:54:14.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4737e-05 2024-04-23T16:54:14.321 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.321 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.321 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.321 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:54:14.321 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:54:14.321 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:54:14.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:14 smithi103 ceph-mon[39629]: pgmap v2099: 27 pgs: 1 active+recovering, 26 active+clean; 5.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 4253/170438 objects misplaced (2.495%); 4.4 MiB/s, 69 objects/s recovering 2024-04-23T16:54:14.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:14 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1020299682' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:54:14.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:14 smithi103 ceph-mon[39629]: from='client.37151 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:54:14.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:14 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2741635768' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:54:14.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:14 smithi103 ceph-mon[39629]: from='client.37154 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:54:14.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:14 smithi148 ceph-mon[44520]: pgmap v2099: 27 pgs: 1 active+recovering, 26 active+clean; 5.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 4253/170438 objects misplaced (2.495%); 4.4 MiB/s, 69 objects/s recovering 2024-04-23T16:54:14.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:14 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1020299682' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:54:14.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:14 smithi148 ceph-mon[44520]: from='client.37151 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:54:14.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:14 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2741635768' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:54:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:14 smithi148 ceph-mon[44520]: from='client.37154 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:54:14.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:14 smithi188 ceph-mon[44101]: pgmap v2099: 27 pgs: 1 active+recovering, 26 active+clean; 5.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 4253/170438 objects misplaced (2.495%); 4.4 MiB/s, 69 objects/s recovering 2024-04-23T16:54:14.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:14 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1020299682' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:54:14.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:14 smithi188 ceph-mon[44101]: from='client.37151 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:54:14.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:14 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2741635768' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:54:14.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:14 smithi188 ceph-mon[44101]: from='client.37154 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:54:15.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:15 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:54:15.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:15 smithi103 ceph-mon[39629]: from='client.37151 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:54:15.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:15 smithi103 ceph-mon[39629]: from='client.37154 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-23T16:54:15.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:15 smithi103 ceph-mon[39629]: osdmap e761: 12 total, 12 up, 8 in 2024-04-23T16:54:15.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:15 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:54:15.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:15 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:54:15.212 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:54:15.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:15.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:54:15.320 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:54:15.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:15 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:54:15.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:15 smithi148 ceph-mon[44520]: from='client.37151 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:54:15.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:15 smithi148 ceph-mon[44520]: from='client.37154 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-23T16:54:15.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:15 smithi148 ceph-mon[44520]: osdmap e761: 12 total, 12 up, 8 in 2024-04-23T16:54:15.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:15 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:54:15.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:15 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:54:15.351 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:15.372 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:54:15.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:15.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:54:15.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:15 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:54:15.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:15 smithi188 ceph-mon[44101]: from='client.37151 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:54:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:15 smithi188 ceph-mon[44101]: from='client.37154 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-23T16:54:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:15 smithi188 ceph-mon[44101]: osdmap e761: 12 total, 12 up, 8 in 2024-04-23T16:54:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:15 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:54:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:15 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:54:15.476 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:54:15.505 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:15.527 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:54:15.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:15.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:54:15.635 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:54:15.679 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:15.696 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-04-23T16:54:15.696 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-04-23T16:54:15.697 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-23T16:54:15.697 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T16:54:15.719 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:54:15.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:15.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:54:15.826 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:54:15.857 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:15.879 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:54:15.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:15.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:54:15.963 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:54:15.983 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:54:16.035 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:16.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:16 smithi148 ceph-mon[44520]: pgmap v2101: 27 pgs: 1 active+recovering, 26 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 967 op/s; 4253/168370 objects misplaced (2.526%); 2.5 MiB/s, 40 objects/s recovering 2024-04-23T16:54:16.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:16 smithi148 ceph-mon[44520]: osdmap e762: 12 total, 12 up, 8 in 2024-04-23T16:54:16.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:16 smithi103 ceph-mon[39629]: pgmap v2101: 27 pgs: 1 active+recovering, 26 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 967 op/s; 4253/168370 objects misplaced (2.526%); 2.5 MiB/s, 40 objects/s recovering 2024-04-23T16:54:16.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:16 smithi103 ceph-mon[39629]: osdmap e762: 12 total, 12 up, 8 in 2024-04-23T16:54:16.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:16 smithi188 ceph-mon[44101]: pgmap v2101: 27 pgs: 1 active+recovering, 26 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 967 op/s; 4253/168370 objects misplaced (2.526%); 2.5 MiB/s, 40 objects/s recovering 2024-04-23T16:54:16.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:16 smithi188 ceph-mon[44101]: osdmap e762: 12 total, 12 up, 8 in 2024-04-23T16:54:16.527 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:54:16.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:16.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:54:16.554 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-23T16:54:16.632 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:54:16.686 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:16.719 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:54:16.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:16.722 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:54:16.823 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:54:16.870 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:54:16.879 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:16.912 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:54:16.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:16.919 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:54:17.020 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:54:17.069 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:17.104 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:54:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:17.107 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:54:17.208 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:54:17.238 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:17.255 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:54:17.262 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:54:17.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:17.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:54:17.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:17 smithi103 ceph-mon[39629]: 9.1 scrub starts 2024-04-23T16:54:17.366 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:54:17.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:17 smithi148 ceph-mon[44520]: 9.1 scrub starts 2024-04-23T16:54:17.402 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:54:17.404 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:17 smithi188 ceph-mon[44101]: 9.1 scrub starts 2024-04-23T16:54:17.413 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:17.445 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:54:17.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:17.450 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:54:17.551 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:54:17.591 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:18.252 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:18 smithi103 ceph-mon[39629]: 9.f scrub starts 2024-04-23T16:54:18.252 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:18 smithi103 ceph-mon[39629]: pgmap v2103: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering, 25 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 3930/161464 objects misplaced (2.434%); 5.2 MiB/s, 83 objects/s recovering 2024-04-23T16:54:18.252 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:18 smithi103 ceph-mon[39629]: 9.1 scrub ok 2024-04-23T16:54:18.252 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:18 smithi103 ceph-mon[39629]: 9.f scrub ok 2024-04-23T16:54:18.254 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:54:18.255 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:18 smithi148 ceph-mon[44520]: 9.f scrub starts 2024-04-23T16:54:18.255 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:18 smithi148 ceph-mon[44520]: pgmap v2103: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering, 25 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 3930/161464 objects misplaced (2.434%); 5.2 MiB/s, 83 objects/s recovering 2024-04-23T16:54:18.255 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:18 smithi148 ceph-mon[44520]: 9.1 scrub ok 2024-04-23T16:54:18.255 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:18 smithi148 ceph-mon[44520]: 9.f scrub ok 2024-04-23T16:54:18.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:18.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:54:18.358 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:54:18.402 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:18.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:18 smithi188 ceph-mon[44101]: 9.f scrub starts 2024-04-23T16:54:18.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:18 smithi188 ceph-mon[44101]: pgmap v2103: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering, 25 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 3930/161464 objects misplaced (2.434%); 5.2 MiB/s, 83 objects/s recovering 2024-04-23T16:54:18.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:18 smithi188 ceph-mon[44101]: 9.1 scrub ok 2024-04-23T16:54:18.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:18 smithi188 ceph-mon[44101]: 9.f scrub ok 2024-04-23T16:54:18.527 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:54:18.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:18.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:54:18.636 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:54:18.669 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:19.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:19 smithi148 ceph-mon[44520]: 9.a scrub starts 2024-04-23T16:54:19.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:19 smithi148 ceph-mon[44520]: 9.9 scrub starts 2024-04-23T16:54:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:19 smithi188 ceph-mon[44101]: 9.a scrub starts 2024-04-23T16:54:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:19 smithi188 ceph-mon[44101]: 9.9 scrub starts 2024-04-23T16:54:19.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:19 smithi103 ceph-mon[39629]: 9.a scrub starts 2024-04-23T16:54:19.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:19 smithi103 ceph-mon[39629]: 9.9 scrub starts 2024-04-23T16:54:19.792 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:54:19.792 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:54:19.792 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2024-04-23T16:54:19.792 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:54:20.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:20 smithi148 ceph-mon[44520]: 9.b scrub starts 2024-04-23T16:54:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:20 smithi148 ceph-mon[44520]: 9.a scrub ok 2024-04-23T16:54:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:20 smithi148 ceph-mon[44520]: pgmap v2104: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering, 25 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 3930/158332 objects misplaced (2.482%); 5.2 MiB/s, 83 objects/s recovering 2024-04-23T16:54:20.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:20 smithi148 ceph-mon[44520]: 9.9 scrub ok 2024-04-23T16:54:20.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:20 smithi188 ceph-mon[44101]: 9.b scrub starts 2024-04-23T16:54:20.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:20 smithi188 ceph-mon[44101]: 9.a scrub ok 2024-04-23T16:54:20.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:20 smithi188 ceph-mon[44101]: pgmap v2104: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering, 25 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 3930/158332 objects misplaced (2.482%); 5.2 MiB/s, 83 objects/s recovering 2024-04-23T16:54:20.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:20 smithi188 ceph-mon[44101]: 9.9 scrub ok 2024-04-23T16:54:20.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:20 smithi103 ceph-mon[39629]: 9.b scrub starts 2024-04-23T16:54:20.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:20 smithi103 ceph-mon[39629]: 9.a scrub ok 2024-04-23T16:54:20.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:20 smithi103 ceph-mon[39629]: pgmap v2104: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering, 25 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 3930/158332 objects misplaced (2.482%); 5.2 MiB/s, 83 objects/s recovering 2024-04-23T16:54:20.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:20 smithi103 ceph-mon[39629]: 9.9 scrub ok 2024-04-23T16:54:21.044 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:54:21.092 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:54:21.123 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:54:21.147 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:54:21.170 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:54:21.202 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:54:21.228 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:54:21.258 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:54:21.285 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:54:21.322 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:54:21.350 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:54:21.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:21 smithi148 ceph-mon[44520]: 9.b scrub ok 2024-04-23T16:54:21.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:21 smithi148 ceph-mon[44520]: 9.5 scrub starts 2024-04-23T16:54:21.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:21 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/8542395' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:54:21.386 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:54:21.413 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:54:21.419 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:54:21.419 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:54:21.419 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:54:21.419 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:54:21.420 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:54:21.420 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:54:21.420 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:54:21.420 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:54:21.420 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:54:21.420 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:54:21.420 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:54:21.420 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:54:21.453 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:21 smithi103 ceph-mon[39629]: 9.b scrub ok 2024-04-23T16:54:21.454 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:21 smithi103 ceph-mon[39629]: 9.5 scrub starts 2024-04-23T16:54:21.454 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:21 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/8542395' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:54:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:21 smithi188 ceph-mon[44101]: 9.b scrub ok 2024-04-23T16:54:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:21 smithi188 ceph-mon[44101]: 9.5 scrub starts 2024-04-23T16:54:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:21 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/8542395' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:54:21.654 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:54:21.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:21.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:54:21.769 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:54:21.805 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:21.818 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-23T16:54:21.829 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:54:21.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:21.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:54:21.933 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:54:21.979 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:22.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:22 smithi148 ceph-mon[44520]: pgmap v2105: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering, 25 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 872 op/s; 3930/156474 objects misplaced (2.512%); 2.5 MiB/s, 40 objects/s recovering 2024-04-23T16:54:22.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:22 smithi148 ceph-mon[44520]: 9.5 scrub ok 2024-04-23T16:54:22.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:22 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.143 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:54:22.267 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:54:22.267 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":762,"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","created":"2024-04-23T16:08:13.477590+0000","modified":"2024-04-23T16:54:15.075444+0000","last_up_change":"2024-04-23T16:52:29.462756+0000","last_in_change":"2024-04-23T16:52:43.617607+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":33,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T16:11:22.416777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"303","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"301","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1600000858306885,"score_stable":2.1600000858306885,"optimal_score":0.67000001668930054,"raw_score_acting":1.4500000476837158,"raw_score_stable":1.4500000476837158,"primary_affinity_weighted":0.98181778192520142,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}},{"pool":9,"pool_name":"unique_pool_7","create_time":"2024-04-23T16:51:19.192533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"707","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.7300000190734863,"score_stable":3.7300000190734863,"optimal_score":0.67000001668930054,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":0.98181778192520142,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}}],"osds":[{"osd":0,"uuid":"4c084047-8f01-4b86-a60c-5592735b5862","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":734,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6801","nonce":4274040097}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6802","nonce":4274040097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6804","nonce":4274040097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6803","nonce":4274040097}]},"public_addr":"172.21.15.103:6801/4274040097","cluster_addr":"172.21.15.103:6802/4274040097","heartbeat_back_addr":"172.21.15.103:6804/4274040097","heartbeat_front_addr":"172.21.15.103:6803/4274040097","state":["exists","up"]},{"osd":1,"uuid":"3438f2bd-5fb7-432c-b383-29e18c442f20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":232,"up_from":271,"up_thru":703,"down_at":233,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6805","nonce":859282187}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6806","nonce":859282187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6808","nonce":859282187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6807","nonce":859282187}]},"public_addr":"172.21.15.103:6805/859282187","cluster_addr":"172.21.15.103:6806/859282187","heartbeat_back_addr":"172.21.15.103:6808/859282187","heartbeat_front_addr":"172.21.15.103:6807/859282187","state":["exists","up"]},{"osd":2,"uuid":"c17fffb4-8d17-4546-ba24-e5951a2a37fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":386,"up_from":481,"up_thru":735,"down_at":387,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6809","nonce":3324226049}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6810","nonce":3324226049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6812","nonce":3324226049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6811","nonce":3324226049}]},"public_addr":"172.21.15.103:6809/3324226049","cluster_addr":"172.21.15.103:6810/3324226049","heartbeat_back_addr":"172.21.15.103:6812/3324226049","heartbeat_front_addr":"172.21.15.103:6811/3324226049","state":["exists","up"]},{"osd":3,"uuid":"cee91a3d-55dc-4e89-b568-86821632bb1c","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":559,"up_from":619,"up_thru":540,"down_at":560,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6813","nonce":995549078}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6814","nonce":995549078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6816","nonce":995549078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6815","nonce":995549078}]},"public_addr":"172.21.15.103:6813/995549078","cluster_addr":"172.21.15.103:6814/995549078","heartbeat_back_addr":"172.21.15.103:6816/995549078","heartbeat_front_addr":"172.21.15.103:6815/995549078","state":["exists","up"]},{"osd":4,"uuid":"61d5385b-2c82-42f8-93c3-8f1baf71c65e","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":33,"last_clean_end":686,"up_from":728,"up_thru":609,"down_at":687,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6800","nonce":4230909898}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6801","nonce":4230909898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6803","nonce":4230909898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6802","nonce":4230909898}]},"public_addr":"172.21.15.148:6800/4230909898","cluster_addr":"172.21.15.148:6801/4230909898","heartbeat_back_addr":"172.21.15.148:6803/4230909898","heartbeat_front_addr":"172.21.15.148:6802/4230909898","state":["exists","up"]},{"osd":5,"uuid":"5d492189-c3cd-47ad-a12a-ed4d275bcfe3","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":609,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6804","nonce":3547200711}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6805","nonce":3547200711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6807","nonce":3547200711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6806","nonce":3547200711}]},"public_addr":"172.21.15.148:6804/3547200711","cluster_addr":"172.21.15.148:6805/3547200711","heartbeat_back_addr":"172.21.15.148:6807/3547200711","heartbeat_front_addr":"172.21.15.148:6806/3547200711","state":["exists","up"]},{"osd":6,"uuid":"89f2379e-cf14-4e69-8747-7535db81b70b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":734,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6808","nonce":366919758}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6809","nonce":366919758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6811","nonce":366919758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6810","nonce":366919758}]},"public_addr":"172.21.15.148:6808/366919758","cluster_addr":"172.21.15.148:6809/366919758","heartbeat_back_addr":"172.21.15.148:6811/366919758","heartbeat_front_addr":"172.21.15.148:6810/366919758","state":["exists","up"]},{"osd":7,"uuid":"f27b035a-558f-4528-80ea-2a00d03f6539","up":1,"in":1,"weight":1,"primary_affinity":0.7818145751953125,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":732,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6812","nonce":3719550888}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6813","nonce":3719550888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6815","nonce":3719550888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6814","nonce":3719550888}]},"public_addr":"172.21.15.148:6812/3719550888","cluster_addr":"172.21.15.148:6813/3719550888","heartbeat_back_addr":"172.21.15.148:6815/3719550888","heartbeat_front_addr":"172.21.15.148:6814/3719550888","state":["exists","up"]},{"osd":8,"uuid":"a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":703,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6800","nonce":1151408595}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6801","nonce":1151408595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6803","nonce":1151408595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6802","nonce":1151408595}]},"public_addr":"172.21.15.188:6800/1151408595","cluster_addr":"172.21.15.188:6801/1151408595","heartbeat_back_addr":"172.21.15.188:6803/1151408595","heartbeat_front_addr":"172.21.15.188:6802/1151408595","state":["exists","up"]},{"osd":9,"uuid":"124d9b05-11eb-4622-b6a2-c4d5bd60b7e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":703,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6804","nonce":306501862}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6805","nonce":306501862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6807","nonce":306501862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6806","nonce":306501862}]},"public_addr":"172.21.15.188:6804/306501862","cluster_addr":"172.21.15.188:6805/306501862","heartbeat_back_addr":"172.21.15.188:6807/306501862","heartbeat_front_addr":"172.21.15.188:6806/306501862","state":["exists","up"]},{"osd":10,"uuid":"3dfb9070-e534-4ea4-a25e-ee1c51297447","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6808","nonce":1231263138}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6809","nonce":1231263138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6811","nonce":1231263138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6810","nonce":1231263138}]},"public_addr":"172.21.15.188:6808/1231263138","cluster_addr":"172.21.15.188:6809/1231263138","heartbeat_back_addr":"172.21.15.188:6811/1231263138","heartbeat_front_addr":"172.21.15.188:6810/1231263138","state":["exists","up"]},{"osd":11,"uuid":"bc163925-f623-4f63-ab89-35e2d511a730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":703,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6812","nonce":457992661}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6813","nonce":457992661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6815","nonce":457992661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6814","nonce":457992661}]},"public_addr":"172.21.15.188:6812/457992661","cluster_addr":"172.21.15.188:6813/457992661","heartbeat_back_addr":"172.21.15.188:6815/457992661","heartbeat_front_addr":"172.21.15.188:6814/457992661","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:54:01.606994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:53:55.050074+0000","dead_epoch":232},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:54:06.441473+0000","dead_epoch":386},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-23T16:53:54.428509+0000","dead_epoch":559},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-23T16:53:33.615314+0000","dead_epoch":686},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-23T16:53:57.240792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:53:24.867263+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:53:38.921482+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:53:38.288918+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:53:52.663753+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-23T16:53:58.529659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:54:14.593200+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6800/3535470771":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1188058303":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1510553721":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/281182801":"2024-04-24T16:08:44.450632+0000","172.21.15.103:6800/2049124518":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/466241963":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/2726838698":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/596230573":"2024-04-24T16:09:14.324448+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T16:54:22.294 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:22 smithi103 ceph-mon[39629]: pgmap v2105: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering, 25 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 872 op/s; 3930/156474 objects misplaced (2.512%); 2.5 MiB/s, 40 objects/s recovering 2024-04-23T16:54:22.294 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.294 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.294 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.294 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.295 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.295 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.295 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:22 smithi103 ceph-mon[39629]: 9.5 scrub ok 2024-04-23T16:54:22.295 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.295 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.295 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:22 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:22 smithi188 ceph-mon[44101]: pgmap v2105: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering, 25 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 872 op/s; 3930/156474 objects misplaced (2.512%); 2.5 MiB/s, 40 objects/s recovering 2024-04-23T16:54:22.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:22 smithi188 ceph-mon[44101]: 9.5 scrub ok 2024-04-23T16:54:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:22 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:23 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1337428826' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:54:23.571 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:23.571 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:23.571 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:23.572 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:23 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1337428826' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:54:23.578 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:23.578 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:23.578 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:23.578 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:23 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1337428826' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:54:23.648 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:54:23.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:23.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:54:23.756 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:54:23.792 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:24.134 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:24 smithi103 ceph-mon[39629]: pgmap v2106: 27 pgs: 1 active+recovering, 26 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 3666/150392 objects misplaced (2.438%); 4.1 MiB/s, 65 objects/s recovering 2024-04-23T16:54:24.135 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:24 smithi103 ceph-mon[39629]: 9.8 scrub starts 2024-04-23T16:54:24.198 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:24 smithi148 ceph-mon[44520]: pgmap v2106: 27 pgs: 1 active+recovering, 26 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 3666/150392 objects misplaced (2.438%); 4.1 MiB/s, 65 objects/s recovering 2024-04-23T16:54:24.198 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:24 smithi148 ceph-mon[44520]: 9.8 scrub starts 2024-04-23T16:54:24.230 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-23T16:54:24.402 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:54:24.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:24.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:54:24.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:24 smithi188 ceph-mon[44101]: pgmap v2106: 27 pgs: 1 active+recovering, 26 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 3666/150392 objects misplaced (2.438%); 4.1 MiB/s, 65 objects/s recovering 2024-04-23T16:54:24.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:24 smithi188 ceph-mon[44101]: 9.8 scrub starts 2024-04-23T16:54:24.508 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:54:24.549 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:24.589 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:54:24.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:24.597 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:54:24.671 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:54:24.671 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:54:24.697 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:54:24.741 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:24.948 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:54:24.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:24.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:54:25.053 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:54:25.105 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:25.141 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:54:25.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:25.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:54:25.248 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:54:25.291 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:25.316 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:54:25.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:25.324 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:54:25.425 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:54:25.472 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:25.474 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:25 smithi103 ceph-mon[39629]: 9.2 scrub starts 2024-04-23T16:54:25.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:25 smithi188 ceph-mon[44101]: 9.2 scrub starts 2024-04-23T16:54:25.506 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:54:25.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:25.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:54:25.612 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:54:25.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:25 smithi148 ceph-mon[44520]: 9.2 scrub starts 2024-04-23T16:54:25.642 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:26.147 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:54:26.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:26 smithi148 ceph-mon[44520]: 9.8 scrub ok 2024-04-23T16:54:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:26 smithi148 ceph-mon[44520]: 9.2 scrub ok 2024-04-23T16:54:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:26 smithi148 ceph-mon[44520]: pgmap v2107: 27 pgs: 1 active+recovering, 26 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 987 op/s; 3666/148616 objects misplaced (2.467%); 3.7 MiB/s, 58 objects/s recovering 2024-04-23T16:54:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:26 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/578809355' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:54:26.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:26 smithi188 ceph-mon[44101]: 9.8 scrub ok 2024-04-23T16:54:26.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:26 smithi188 ceph-mon[44101]: 9.2 scrub ok 2024-04-23T16:54:26.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:26 smithi188 ceph-mon[44101]: pgmap v2107: 27 pgs: 1 active+recovering, 26 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 987 op/s; 3666/148616 objects misplaced (2.467%); 3.7 MiB/s, 58 objects/s recovering 2024-04-23T16:54:26.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:26 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/578809355' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:54:26.560 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:54:26.591 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:26 smithi103 ceph-mon[39629]: 9.8 scrub ok 2024-04-23T16:54:26.591 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:26 smithi103 ceph-mon[39629]: 9.2 scrub ok 2024-04-23T16:54:26.592 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:26 smithi103 ceph-mon[39629]: pgmap v2107: 27 pgs: 1 active+recovering, 26 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 987 op/s; 3666/148616 objects misplaced (2.467%); 3.7 MiB/s, 58 objects/s recovering 2024-04-23T16:54:26.592 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:26 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/578809355' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:54:26.694 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:54:26.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:26.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:54:26.705 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-23T16:54:26.799 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:54:26.835 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:26.877 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:54:26.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:26.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:54:26.983 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:54:27.013 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:27.038 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:54:27.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:27.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:54:27.071 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:54:27.146 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:54:27.147 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:54:27.176 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:27 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:54:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:27 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:54:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:27 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/578809355' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:54:27.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:27 smithi103 ceph-mon[39629]: osdmap e763: 12 total, 12 up, 8 in 2024-04-23T16:54:27.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:27 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T16:54:27.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:27 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T16:54:27.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:27 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:54:27.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:27 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:54:27.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:27 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/578809355' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:54:27.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:27 smithi148 ceph-mon[44520]: osdmap e763: 12 total, 12 up, 8 in 2024-04-23T16:54:27.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:27 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T16:54:27.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:27 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T16:54:27.377 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:54:27.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:27.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:54:27.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:27 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:54:27.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:27 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:54:27.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:27 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/578809355' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:54:27.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:27 smithi188 ceph-mon[44101]: osdmap e763: 12 total, 12 up, 8 in 2024-04-23T16:54:27.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:27 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T16:54:27.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:27 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T16:54:27.483 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:54:27.521 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:27.548 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:54:27.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:27.553 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:54:27.654 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:54:27.693 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:28 smithi103 ceph-mon[39629]: pgmap v2109: 27 pgs: 1 active+recovering, 26 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 987 op/s; 3369/141712 objects misplaced (2.377%); 3.5 MiB/s, 56 objects/s recovering 2024-04-23T16:54:28.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:28 smithi188 ceph-mon[44101]: pgmap v2109: 27 pgs: 1 active+recovering, 26 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 987 op/s; 3369/141712 objects misplaced (2.377%); 3.5 MiB/s, 56 objects/s recovering 2024-04-23T16:54:28.552 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:54:28.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:28.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:54:28.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:28 smithi148 ceph-mon[44520]: pgmap v2109: 27 pgs: 1 active+recovering, 26 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 987 op/s; 3369/141712 objects misplaced (2.377%); 3.5 MiB/s, 56 objects/s recovering 2024-04-23T16:54:28.657 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:54:28.698 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:28.851 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:54:28.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:28.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:54:28.956 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:54:28.993 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:29.032 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:54:29.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:29.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:54:29.057 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:54:29.137 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:54:29.167 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:29.247 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:54:29.274 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:54:29.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:29.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:54:29.279 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:54:29.313 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:54:29.336 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:54:29.372 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:54:29.379 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:54:29.395 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:54:29.415 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:29.427 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:54:29.455 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:54:29.459 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:54:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:29.462 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:54:29.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:29 smithi188 ceph-mon[44101]: 1.a deep-scrub starts 2024-04-23T16:54:29.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:29 smithi188 ceph-mon[44101]: 1.a deep-scrub ok 2024-04-23T16:54:29.483 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:54:29.484 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:54:29.516 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:54:29.545 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:54:29.563 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:54:29.588 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:54:29.590 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:54:29.590 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:54:29.590 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:54:29.590 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:54:29.590 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:54:29.590 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:54:29.590 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:54:29.590 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:54:29.591 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:54:29.591 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:54:29.591 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:54:29.591 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:54:29.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:29 smithi103 ceph-mon[39629]: 1.a deep-scrub starts 2024-04-23T16:54:29.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:29 smithi103 ceph-mon[39629]: 1.a deep-scrub ok 2024-04-23T16:54:29.614 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:29.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:29 smithi148 ceph-mon[44520]: 1.a deep-scrub starts 2024-04-23T16:54:29.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:29 smithi148 ceph-mon[44520]: 1.a deep-scrub ok 2024-04-23T16:54:30.174 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:54:30.174 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2024-04-23T16:54:30.174 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2024-04-23T16:54:30.174 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:54:30.174 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.174 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.26584.0:180899 7.0 7:048b8431:::benchmark_data_smithi188_141483_object62218:head [delete] snapc 0=[] ondisk+write+known_if_redirected e591)", 2024-04-23T16:54:30.174 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:45:02.306963+0000", 2024-04-23T16:54:30.175 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 567.86534030500002, 2024-04-23T16:54:30.175 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.049749896000000002, 2024-04-23T16:54:30.175 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:54:30.175 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:54:30.175 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:30.175 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:54:30.175 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.26584", 2024-04-23T16:54:30.175 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1061291304", 2024-04-23T16:54:30.175 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 180899 2024-04-23T16:54:30.175 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.175 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:54:30.175 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.175 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:54:30.175 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.306963+0000", 2024-04-23T16:54:30.175 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.176 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.176 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.176 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:54:30.176 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.306963+0000", 2024-04-23T16:54:30.176 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.176 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.176 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.176 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:54:30.176 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.306965+0000", 2024-04-23T16:54:30.176 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5510000000000001e-06 2024-04-23T16:54:30.176 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.176 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.176 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:54:30.176 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.306970+0000", 2024-04-23T16:54:30.177 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3890000000000004e-06 2024-04-23T16:54:30.177 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.177 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.177 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:54:30.177 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.306980+0000", 2024-04-23T16:54:30.177 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0064000000000001e-05 2024-04-23T16:54:30.177 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.177 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.177 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:54:30.177 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.306988+0000", 2024-04-23T16:54:30.177 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6939999999999995e-06 2024-04-23T16:54:30.177 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.177 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.177 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:54:30.178 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.311897+0000", 2024-04-23T16:54:30.178 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.004909353 2024-04-23T16:54:30.178 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.178 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.178 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:54:30.178 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.312018+0000", 2024-04-23T16:54:30.178 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000121011 2024-04-23T16:54:30.178 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.178 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.178 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:54:30.178 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.312135+0000", 2024-04-23T16:54:30.178 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011671300000000001 2024-04-23T16:54:30.178 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.179 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.179 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:54:30.179 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.322363+0000", 2024-04-23T16:54:30.179 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010228113 2024-04-23T16:54:30.179 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.179 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.179 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:54:30.179 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.356615+0000", 2024-04-23T16:54:30.179 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.034251860000000002 2024-04-23T16:54:30.179 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.179 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.179 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:54:30.179 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.356670+0000", 2024-04-23T16:54:30.179 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4763999999999999e-05 2024-04-23T16:54:30.179 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.180 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.180 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:54:30.180 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:45:02.356713+0000", 2024-04-23T16:54:30.180 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3383999999999997e-05 2024-04-23T16:54:30.180 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.180 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:54:30.180 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.180 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.180 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.180 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "rep_scrubmap(1.8 e563 from shard 8)", 2024-04-23T16:54:30.180 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:35.776826+0000", 2024-04-23T16:54:30.180 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 474.39547693499998, 2024-04-23T16:54:30.180 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.056699045000000003, 2024-04-23T16:54:30.181 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:54:30.181 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:54:30.181 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:54:30.181 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:54:30.181 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.181 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:54:30.181 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:35.776826+0000", 2024-04-23T16:54:30.181 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.181 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.181 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.181 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:54:30.181 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:35.776826+0000", 2024-04-23T16:54:30.181 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.181 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.181 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.182 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:54:30.182 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:35.776829+0000", 2024-04-23T16:54:30.182 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3190000000000002e-06 2024-04-23T16:54:30.182 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.182 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.182 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:54:30.182 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:35.776857+0000", 2024-04-23T16:54:30.182 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.853e-05 2024-04-23T16:54:30.182 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.182 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.182 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:54:30.182 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:35.776872+0000", 2024-04-23T16:54:30.182 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4584e-05 2024-04-23T16:54:30.182 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.183 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.183 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:54:30.183 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:35.776881+0000", 2024-04-23T16:54:30.183 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3090000000000001e-06 2024-04-23T16:54:30.183 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.183 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.183 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:54:30.183 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:35.833496+0000", 2024-04-23T16:54:30.183 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.056614386000000003 2024-04-23T16:54:30.183 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.183 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.183 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:54:30.183 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:35.833506+0000", 2024-04-23T16:54:30.183 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0883999999999999e-05 2024-04-23T16:54:30.184 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.184 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.184 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:54:30.184 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:35.833526+0000", 2024-04-23T16:54:30.184 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9032999999999999e-05 2024-04-23T16:54:30.184 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.184 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:54:30.184 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.184 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.184 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.184 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:31845 8.4 8:2d8f1695:::benchmark_data_smithi188_155546_object31844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e613)", 2024-04-23T16:54:30.184 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:45.116080+0000", 2024-04-23T16:54:30.184 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 465.05622364499999, 2024-04-23T16:54:30.184 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.050014955, 2024-04-23T16:54:30.184 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:54:30.185 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:54:30.185 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:30.185 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:54:30.185 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:54:30.185 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:30.185 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 31845 2024-04-23T16:54:30.185 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.185 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:54:30.185 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.185 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:54:30.185 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116080+0000", 2024-04-23T16:54:30.185 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.185 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.185 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.186 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:54:30.186 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116080+0000", 2024-04-23T16:54:30.186 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.186 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.186 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.186 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:54:30.186 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116081+0000", 2024-04-23T16:54:30.186 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1399999999999995e-07 2024-04-23T16:54:30.186 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.186 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.186 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:54:30.186 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116160+0000", 2024-04-23T16:54:30.186 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.9260999999999999e-05 2024-04-23T16:54:30.186 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.187 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.187 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:54:30.187 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116168+0000", 2024-04-23T16:54:30.187 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6029999999999999e-06 2024-04-23T16:54:30.187 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.187 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.187 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:54:30.187 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116174+0000", 2024-04-23T16:54:30.187 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1419999999999998e-06 2024-04-23T16:54:30.187 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.187 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.187 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:54:30.187 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116213+0000", 2024-04-23T16:54:30.187 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9607000000000002e-05 2024-04-23T16:54:30.187 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.188 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.188 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:54:30.188 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116353+0000", 2024-04-23T16:54:30.188 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013967600000000001 2024-04-23T16:54:30.188 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.188 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.188 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:54:30.188 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116485+0000", 2024-04-23T16:54:30.188 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013235499999999999 2024-04-23T16:54:30.188 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.188 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.188 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:54:30.188 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118878+0000", 2024-04-23T16:54:30.189 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002392693 2024-04-23T16:54:30.189 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.189 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.189 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:54:30.189 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.166002+0000", 2024-04-23T16:54:30.189 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.047124302999999999 2024-04-23T16:54:30.189 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.189 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.189 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:54:30.189 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.166057+0000", 2024-04-23T16:54:30.189 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4505000000000001e-05 2024-04-23T16:54:30.189 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.189 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.189 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:54:30.190 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.166095+0000", 2024-04-23T16:54:30.190 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7895999999999998e-05 2024-04-23T16:54:30.190 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.190 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:54:30.190 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.190 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.190 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.190 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:31847 8.0 8:00f95cbe:::benchmark_data_smithi188_155546_object31846:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e613)", 2024-04-23T16:54:30.190 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:45.116184+0000", 2024-04-23T16:54:30.190 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 465.05611904400001, 2024-04-23T16:54:30.190 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048306495999999997, 2024-04-23T16:54:30.190 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:54:30.190 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:54:30.190 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:30.191 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:54:30.191 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:54:30.191 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:30.191 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 31847 2024-04-23T16:54:30.191 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.191 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:54:30.191 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.191 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:54:30.191 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116184+0000", 2024-04-23T16:54:30.191 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.191 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.191 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.191 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:54:30.191 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116184+0000", 2024-04-23T16:54:30.192 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.192 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.192 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.192 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:54:30.192 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116185+0000", 2024-04-23T16:54:30.192 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0279999999999999e-06 2024-04-23T16:54:30.192 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.192 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.192 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:54:30.192 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116251+0000", 2024-04-23T16:54:30.192 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5147999999999994e-05 2024-04-23T16:54:30.192 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.192 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.192 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:54:30.192 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116257+0000", 2024-04-23T16:54:30.193 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8809999999999998e-06 2024-04-23T16:54:30.193 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.193 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.193 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:54:30.193 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116263+0000", 2024-04-23T16:54:30.193 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5799999999999999e-06 2024-04-23T16:54:30.193 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.193 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.193 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:54:30.193 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.117936+0000", 2024-04-23T16:54:30.193 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00167289 2024-04-23T16:54:30.193 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.193 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.193 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:54:30.194 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118056+0000", 2024-04-23T16:54:30.194 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000120318 2024-04-23T16:54:30.194 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.194 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.194 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:54:30.194 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118150+0000", 2024-04-23T16:54:30.194 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3642999999999998e-05 2024-04-23T16:54:30.194 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.194 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.194 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:54:30.194 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.119649+0000", 2024-04-23T16:54:30.194 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0014991480000000001 2024-04-23T16:54:30.194 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.194 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.195 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:54:30.195 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164347+0000", 2024-04-23T16:54:30.195 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.044698030999999999 2024-04-23T16:54:30.195 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.195 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.195 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:54:30.195 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164403+0000", 2024-04-23T16:54:30.195 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6314999999999999e-05 2024-04-23T16:54:30.195 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.195 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.195 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:54:30.195 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164491+0000", 2024-04-23T16:54:30.195 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7514000000000003e-05 2024-04-23T16:54:30.195 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.196 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:54:30.196 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.196 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.196 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.196 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:31849 8.4 8:24fbb2b9:::benchmark_data_smithi188_155546_object31848:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e613)", 2024-04-23T16:54:30.196 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:45.116272+0000", 2024-04-23T16:54:30.196 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 465.05603171000001, 2024-04-23T16:54:30.196 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.049955200999999998, 2024-04-23T16:54:30.196 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:54:30.196 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:54:30.196 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:30.196 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:54:30.196 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:54:30.196 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:30.197 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 31849 2024-04-23T16:54:30.197 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.197 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:54:30.197 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.197 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:54:30.197 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116272+0000", 2024-04-23T16:54:30.197 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.197 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.197 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.197 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:54:30.197 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116272+0000", 2024-04-23T16:54:30.197 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.197 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.197 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.197 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:54:30.198 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116273+0000", 2024-04-23T16:54:30.198 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6300000000000004e-07 2024-04-23T16:54:30.198 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.198 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.198 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:54:30.198 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116376+0000", 2024-04-23T16:54:30.198 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000103206 2024-04-23T16:54:30.198 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.198 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.198 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:54:30.198 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116383+0000", 2024-04-23T16:54:30.198 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4529999999999996e-06 2024-04-23T16:54:30.198 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.198 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.199 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:54:30.199 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.116390+0000", 2024-04-23T16:54:30.199 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3430000000000001e-06 2024-04-23T16:54:30.199 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.199 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.199 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:54:30.199 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.117596+0000", 2024-04-23T16:54:30.199 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001206307 2024-04-23T16:54:30.199 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.199 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.199 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:54:30.199 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.117726+0000", 2024-04-23T16:54:30.199 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012993999999999999 2024-04-23T16:54:30.199 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.199 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.200 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:54:30.200 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.117814+0000", 2024-04-23T16:54:30.200 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8517999999999995e-05 2024-04-23T16:54:30.200 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.200 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.200 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:54:30.200 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118987+0000", 2024-04-23T16:54:30.200 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0011723720000000001 2024-04-23T16:54:30.200 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.200 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.200 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:54:30.200 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.166143+0000", 2024-04-23T16:54:30.201 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.047155860000000001 2024-04-23T16:54:30.201 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.201 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.201 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:54:30.201 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.166193+0000", 2024-04-23T16:54:30.201 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0800999999999997e-05 2024-04-23T16:54:30.201 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.201 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.201 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:54:30.201 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.166227+0000", 2024-04-23T16:54:30.201 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3538000000000003e-05 2024-04-23T16:54:30.201 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.201 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:54:30.202 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.202 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.202 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.202 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:31856 8.0 8:071e21bd:::benchmark_data_smithi188_155546_object31855:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e613)", 2024-04-23T16:54:30.202 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:45.118606+0000", 2024-04-23T16:54:30.202 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 465.05369777599998, 2024-04-23T16:54:30.202 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.046233700000000003, 2024-04-23T16:54:30.202 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:54:30.202 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:54:30.202 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:30.202 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:54:30.202 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:54:30.202 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:30.203 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 31856 2024-04-23T16:54:30.203 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.203 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:54:30.203 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.203 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:54:30.203 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118606+0000", 2024-04-23T16:54:30.203 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.203 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.203 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.203 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:54:30.203 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118606+0000", 2024-04-23T16:54:30.203 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.203 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.203 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.204 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:54:30.204 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118607+0000", 2024-04-23T16:54:30.204 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5060000000000001e-06 2024-04-23T16:54:30.204 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.204 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.204 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:54:30.204 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118684+0000", 2024-04-23T16:54:30.204 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6756999999999997e-05 2024-04-23T16:54:30.204 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.204 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.204 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:54:30.204 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118693+0000", 2024-04-23T16:54:30.204 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7150000000000004e-06 2024-04-23T16:54:30.204 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.205 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.205 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:54:30.205 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118699+0000", 2024-04-23T16:54:30.205 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9630000000000003e-06 2024-04-23T16:54:30.205 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.205 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.205 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:54:30.205 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118906+0000", 2024-04-23T16:54:30.205 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00020748199999999999 2024-04-23T16:54:30.205 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.205 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.205 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:54:30.205 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.118994+0000", 2024-04-23T16:54:30.205 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7503000000000006e-05 2024-04-23T16:54:30.206 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.206 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.206 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:54:30.206 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.119071+0000", 2024-04-23T16:54:30.206 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7849000000000006e-05 2024-04-23T16:54:30.206 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.206 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.206 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:54:30.206 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.119784+0000", 2024-04-23T16:54:30.206 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00071264400000000002 2024-04-23T16:54:30.206 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.206 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.206 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:54:30.206 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164739+0000", 2024-04-23T16:54:30.207 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.044954820999999999 2024-04-23T16:54:30.207 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.207 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.207 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:54:30.207 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164798+0000", 2024-04-23T16:54:30.207 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9419000000000001e-05 2024-04-23T16:54:30.207 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.207 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.207 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:54:30.207 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164839+0000", 2024-04-23T16:54:30.207 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1041000000000002e-05 2024-04-23T16:54:30.207 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.207 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:54:30.207 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.208 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.208 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.208 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:31864 8.4 8:2d0e3829:::benchmark_data_smithi188_155546_object31863:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e613)", 2024-04-23T16:54:30.208 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:45.121165+0000", 2024-04-23T16:54:30.208 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 465.05113862500002, 2024-04-23T16:54:30.208 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.047666467999999997, 2024-04-23T16:54:30.208 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:54:30.208 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:54:30.208 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:30.208 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:54:30.209 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:54:30.209 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:30.209 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 31864 2024-04-23T16:54:30.209 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.209 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:54:30.209 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.209 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:54:30.209 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.121165+0000", 2024-04-23T16:54:30.209 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.209 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.209 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.209 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:54:30.209 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.121165+0000", 2024-04-23T16:54:30.210 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.210 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.210 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.210 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:54:30.210 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.121166+0000", 2024-04-23T16:54:30.210 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0899999999999999e-06 2024-04-23T16:54:30.210 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.210 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.210 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:54:30.210 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.121212+0000", 2024-04-23T16:54:30.210 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5917000000000001e-05 2024-04-23T16:54:30.210 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.210 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.210 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:54:30.211 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164332+0000", 2024-04-23T16:54:30.211 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.043120612000000003 2024-04-23T16:54:30.211 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.211 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.211 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:54:30.211 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164346+0000", 2024-04-23T16:54:30.211 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3345e-05 2024-04-23T16:54:30.211 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.211 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.211 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:54:30.211 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164419+0000", 2024-04-23T16:54:30.211 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3740999999999998e-05 2024-04-23T16:54:30.211 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.211 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.212 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:54:30.212 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164596+0000", 2024-04-23T16:54:30.212 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000176968 2024-04-23T16:54:30.212 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.212 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.212 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:54:30.212 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.164747+0000", 2024-04-23T16:54:30.212 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015088600000000001 2024-04-23T16:54:30.212 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.212 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.212 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:54:30.212 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.167602+0000", 2024-04-23T16:54:30.212 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0028541859999999999 2024-04-23T16:54:30.212 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.213 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.213 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:54:30.213 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.168772+0000", 2024-04-23T16:54:30.213 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0011700230000000001 2024-04-23T16:54:30.213 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.213 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.213 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:54:30.213 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.168808+0000", 2024-04-23T16:54:30.213 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6628000000000002e-05 2024-04-23T16:54:30.213 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.213 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.213 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:54:30.213 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:45.168831+0000", 2024-04-23T16:54:30.213 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3071999999999999e-05 2024-04-23T16:54:30.213 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.214 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:54:30.214 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.214 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.214 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.214 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:43834 8.4 8:2ee18187:::benchmark_data_smithi188_155546_object43833:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e614)", 2024-04-23T16:54:30.214 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:49.576241+0000", 2024-04-23T16:54:30.214 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 460.59606233699998, 2024-04-23T16:54:30.214 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.056131107, 2024-04-23T16:54:30.214 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:54:30.214 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:54:30.214 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:30.214 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:54:30.214 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:54:30.214 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:30.215 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 43834 2024-04-23T16:54:30.215 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.215 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:54:30.215 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.215 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:54:30.215 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576241+0000", 2024-04-23T16:54:30.215 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.215 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.215 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.215 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:54:30.215 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576241+0000", 2024-04-23T16:54:30.215 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.215 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.215 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.216 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:54:30.216 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576243+0000", 2024-04-23T16:54:30.216 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2010000000000002e-06 2024-04-23T16:54:30.216 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.216 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.216 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:54:30.216 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576312+0000", 2024-04-23T16:54:30.216 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9154999999999998e-05 2024-04-23T16:54:30.216 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.216 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.216 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:54:30.216 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576326+0000", 2024-04-23T16:54:30.216 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3505e-05 2024-04-23T16:54:30.217 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.217 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.217 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:54:30.217 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576335+0000", 2024-04-23T16:54:30.217 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0580000000000004e-06 2024-04-23T16:54:30.217 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.217 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.217 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:54:30.217 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576390+0000", 2024-04-23T16:54:30.217 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5346000000000002e-05 2024-04-23T16:54:30.217 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.217 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.217 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:54:30.217 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576542+0000", 2024-04-23T16:54:30.217 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015117099999999999 2024-04-23T16:54:30.218 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.218 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.218 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:54:30.218 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576694+0000", 2024-04-23T16:54:30.218 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015268700000000001 2024-04-23T16:54:30.218 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.218 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.218 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:54:30.218 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.579902+0000", 2024-04-23T16:54:30.218 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032082399999999998 2024-04-23T16:54:30.218 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.218 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.218 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:54:30.219 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.632245+0000", 2024-04-23T16:54:30.219 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052342872999999998 2024-04-23T16:54:30.219 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.219 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.219 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:54:30.219 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.632313+0000", 2024-04-23T16:54:30.219 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7423999999999997e-05 2024-04-23T16:54:30.219 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.219 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.219 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:54:30.219 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.632372+0000", 2024-04-23T16:54:30.219 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9447000000000002e-05 2024-04-23T16:54:30.219 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.219 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:54:30.220 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.220 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.220 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.220 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:43838 8.4 8:2122428a:::benchmark_data_smithi188_155546_object43837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e614)", 2024-04-23T16:54:30.220 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:49.576429+0000", 2024-04-23T16:54:30.220 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 460.59587461799998, 2024-04-23T16:54:30.220 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.057184003999999997, 2024-04-23T16:54:30.220 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:54:30.220 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:54:30.220 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:30.220 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:54:30.220 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:54:30.220 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:30.221 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 43838 2024-04-23T16:54:30.221 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.221 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:54:30.221 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.221 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:54:30.221 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576429+0000", 2024-04-23T16:54:30.221 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.221 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.221 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.221 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:54:30.221 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576429+0000", 2024-04-23T16:54:30.221 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.221 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.221 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.222 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:54:30.222 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576430+0000", 2024-04-23T16:54:30.222 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2449999999999999e-06 2024-04-23T16:54:30.222 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.222 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.222 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:54:30.222 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576497+0000", 2024-04-23T16:54:30.222 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7193999999999994e-05 2024-04-23T16:54:30.222 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.222 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.222 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:54:30.222 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576507+0000", 2024-04-23T16:54:30.222 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3689999999999996e-06 2024-04-23T16:54:30.222 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.223 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.223 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:54:30.223 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576514+0000", 2024-04-23T16:54:30.223 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2150000000000004e-06 2024-04-23T16:54:30.223 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.223 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.223 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:54:30.223 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578392+0000", 2024-04-23T16:54:30.223 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00187791 2024-04-23T16:54:30.223 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.223 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.223 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:54:30.223 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578517+0000", 2024-04-23T16:54:30.223 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012536200000000001 2024-04-23T16:54:30.224 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.224 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.224 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:54:30.224 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578639+0000", 2024-04-23T16:54:30.224 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000122228 2024-04-23T16:54:30.224 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.224 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.224 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:54:30.224 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.580100+0000", 2024-04-23T16:54:30.224 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0014611660000000001 2024-04-23T16:54:30.224 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.224 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.224 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:54:30.224 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.633547+0000", 2024-04-23T16:54:30.224 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.053446395000000001 2024-04-23T16:54:30.225 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.225 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.225 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:54:30.225 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.633583+0000", 2024-04-23T16:54:30.225 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6047999999999998e-05 2024-04-23T16:54:30.225 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.225 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.225 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:54:30.225 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.633613+0000", 2024-04-23T16:54:30.225 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9872000000000001e-05 2024-04-23T16:54:30.225 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.225 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:54:30.226 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.226 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.226 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.226 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:43843 8.0 8:0e72c17a:::benchmark_data_smithi188_155546_object43842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e614)", 2024-04-23T16:54:30.226 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:49.576723+0000", 2024-04-23T16:54:30.226 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 460.59558049700001, 2024-04-23T16:54:30.226 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.057769413999999998, 2024-04-23T16:54:30.226 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:54:30.226 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:54:30.226 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:30.226 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:54:30.227 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:54:30.227 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:30.227 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 43843 2024-04-23T16:54:30.227 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.227 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:54:30.227 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.227 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:54:30.227 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576723+0000", 2024-04-23T16:54:30.227 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.227 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.227 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.227 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:54:30.227 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576723+0000", 2024-04-23T16:54:30.227 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.227 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.228 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.228 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:54:30.228 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576724+0000", 2024-04-23T16:54:30.228 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.47e-07 2024-04-23T16:54:30.228 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.228 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.228 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:54:30.228 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576753+0000", 2024-04-23T16:54:30.228 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9153e-05 2024-04-23T16:54:30.228 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.228 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.228 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:54:30.228 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576760+0000", 2024-04-23T16:54:30.228 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8619999999999997e-06 2024-04-23T16:54:30.229 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.229 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.229 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:54:30.229 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.576765+0000", 2024-04-23T16:54:30.229 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3870000000000003e-06 2024-04-23T16:54:30.229 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.229 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.229 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:54:30.229 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578311+0000", 2024-04-23T16:54:30.229 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0015454690000000001 2024-04-23T16:54:30.229 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.229 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.229 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:54:30.229 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578466+0000", 2024-04-23T16:54:30.230 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000155298 2024-04-23T16:54:30.230 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.230 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.230 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:54:30.230 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578584+0000", 2024-04-23T16:54:30.230 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000118146 2024-04-23T16:54:30.230 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.230 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.230 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:54:30.230 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.581785+0000", 2024-04-23T16:54:30.230 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032011600000000002 2024-04-23T16:54:30.230 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.230 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.231 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:54:30.231 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.634392+0000", 2024-04-23T16:54:30.231 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052606762000000001 2024-04-23T16:54:30.231 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.231 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.231 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:54:30.231 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.634448+0000", 2024-04-23T16:54:30.231 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6174000000000001e-05 2024-04-23T16:54:30.231 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.231 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.231 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:54:30.231 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.634492+0000", 2024-04-23T16:54:30.231 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4156000000000002e-05 2024-04-23T16:54:30.231 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.231 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:54:30.232 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.232 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.232 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.232 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:43847 8.0 8:0211add4:::benchmark_data_smithi188_155546_object43846:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e614)", 2024-04-23T16:54:30.232 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:49.578901+0000", 2024-04-23T16:54:30.232 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 460.59340193200001, 2024-04-23T16:54:30.232 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.055695753000000001, 2024-04-23T16:54:30.232 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:54:30.232 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:54:30.232 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:30.232 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:54:30.232 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:54:30.232 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:30.232 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 43847 2024-04-23T16:54:30.233 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.233 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:54:30.233 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.233 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:54:30.233 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578901+0000", 2024-04-23T16:54:30.233 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.233 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.233 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.233 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:54:30.233 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578901+0000", 2024-04-23T16:54:30.233 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.233 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.233 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.233 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:54:30.234 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578903+0000", 2024-04-23T16:54:30.234 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0269999999999998e-06 2024-04-23T16:54:30.234 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.234 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.234 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:54:30.234 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578940+0000", 2024-04-23T16:54:30.234 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6461000000000002e-05 2024-04-23T16:54:30.234 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.234 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.234 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:54:30.234 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578952+0000", 2024-04-23T16:54:30.234 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.173e-05 2024-04-23T16:54:30.234 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.234 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.235 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:54:30.235 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578960+0000", 2024-04-23T16:54:30.235 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1319999999999994e-06 2024-04-23T16:54:30.235 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.235 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.235 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:54:30.235 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.579571+0000", 2024-04-23T16:54:30.235 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00061083999999999997 2024-04-23T16:54:30.235 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.235 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.235 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:54:30.235 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.579694+0000", 2024-04-23T16:54:30.235 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012373000000000001 2024-04-23T16:54:30.235 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.236 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.236 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:54:30.236 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.579843+0000", 2024-04-23T16:54:30.236 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014876000000000001 2024-04-23T16:54:30.236 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.236 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.236 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:54:30.236 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.581796+0000", 2024-04-23T16:54:30.236 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0019525650000000001 2024-04-23T16:54:30.236 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.236 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.236 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:54:30.236 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.634539+0000", 2024-04-23T16:54:30.237 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052743341999999999 2024-04-23T16:54:30.237 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.237 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.237 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:54:30.237 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.634576+0000", 2024-04-23T16:54:30.237 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7169999999999998e-05 2024-04-23T16:54:30.237 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.237 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.237 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:54:30.237 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.634597+0000", 2024-04-23T16:54:30.237 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0996e-05 2024-04-23T16:54:30.237 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.237 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:54:30.237 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.238 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.238 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.238 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.16995.0:43848 8.0 8:0e8e0095:::benchmark_data_smithi188_155546_object43847:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e614)", 2024-04-23T16:54:30.238 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:46:49.578973+0000", 2024-04-23T16:54:30.238 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 460.593329998, 2024-04-23T16:54:30.238 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.055746616999999998, 2024-04-23T16:54:30.238 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:54:30.238 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:54:30.238 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:54:30.238 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:54:30.239 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.16995", 2024-04-23T16:54:30.239 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:54:30.239 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 43848 2024-04-23T16:54:30.239 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.239 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:54:30.239 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.239 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:54:30.239 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578973+0000", 2024-04-23T16:54:30.239 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.239 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.239 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.239 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:54:30.239 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578973+0000", 2024-04-23T16:54:30.239 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:30.239 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.240 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.240 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:54:30.240 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.578974+0000", 2024-04-23T16:54:30.240 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4099999999999997e-07 2024-04-23T16:54:30.240 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.240 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.240 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:54:30.240 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.579006+0000", 2024-04-23T16:54:30.240 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1572000000000002e-05 2024-04-23T16:54:30.240 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.240 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.240 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:54:30.240 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.579013+0000", 2024-04-23T16:54:30.240 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2370000000000003e-06 2024-04-23T16:54:30.241 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.241 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.241 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:54:30.241 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.579019+0000", 2024-04-23T16:54:30.241 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0610000000000001e-06 2024-04-23T16:54:30.241 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.241 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.241 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:54:30.241 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.580808+0000", 2024-04-23T16:54:30.241 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0017892139999999999 2024-04-23T16:54:30.241 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.241 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.241 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:54:30.241 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.580931+0000", 2024-04-23T16:54:30.241 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012230300000000001 2024-04-23T16:54:30.242 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.242 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.242 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 9", 2024-04-23T16:54:30.242 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.581069+0000", 2024-04-23T16:54:30.242 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013858300000000001 2024-04-23T16:54:30.242 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.242 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.242 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:54:30.242 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.581927+0000", 2024-04-23T16:54:30.242 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00085782000000000002 2024-04-23T16:54:30.242 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.242 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.242 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:54:30.243 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.634651+0000", 2024-04-23T16:54:30.243 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.052723999000000001 2024-04-23T16:54:30.243 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.243 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.243 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2024-04-23T16:54:30.243 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.634685+0000", 2024-04-23T16:54:30.243 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3766999999999997e-05 2024-04-23T16:54:30.243 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:30.243 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:30.243 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2024-04-23T16:54:30.243 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:46:49.634720+0000", 2024-04-23T16:54:30.243 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5219999999999998e-05 2024-04-23T16:54:30.243 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.243 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:54:30.244 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.244 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:30.244 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:54:30.244 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:54:30.391 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:54:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:30.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:54:30.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:30 smithi188 ceph-mon[44101]: pgmap v2110: 27 pgs: 1 active+recovering, 26 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 988 op/s; 3369/138562 objects misplaced (2.431%); 3.5 MiB/s, 56 objects/s recovering 2024-04-23T16:54:30.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:30 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2188366759' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:54:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:30 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.498 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:54:30.528 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:30 smithi148 ceph-mon[44520]: pgmap v2110: 27 pgs: 1 active+recovering, 26 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 988 op/s; 3369/138562 objects misplaced (2.431%); 3.5 MiB/s, 56 objects/s recovering 2024-04-23T16:54:30.528 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:30 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2188366759' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:54:30.528 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.528 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.528 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.528 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.528 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.528 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.528 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.528 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.528 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:30 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.529 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:30.555 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:54:30.560 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:30 smithi103 ceph-mon[39629]: pgmap v2110: 27 pgs: 1 active+recovering, 26 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 988 op/s; 3369/138562 objects misplaced (2.431%); 3.5 MiB/s, 56 objects/s recovering 2024-04-23T16:54:30.561 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:30 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2188366759' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:54:30.561 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.561 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.561 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.561 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.561 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.561 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.561 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.561 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.561 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:30 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:30.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:30.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:54:30.669 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:54:30.717 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:30.747 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:54:30.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:30.753 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:54:30.854 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:54:30.884 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:30.916 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:54:30.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:30.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:54:30.957 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-23T16:54:31.021 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:54:31.061 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:31.094 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:54:31.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:31.102 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:54:31.186 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:31.186 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:31.187 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:31 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:31.187 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:31 smithi103 ceph-mon[39629]: pgmap v2111: 27 pgs: 1 active+recovering, 26 active+clean; 4.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 993 op/s; 3369/136608 objects misplaced (2.466%); 3.5 MiB/s, 56 objects/s recovering 2024-04-23T16:54:31.202 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:54:31.248 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:31.393 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:54:31.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:31.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:31.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:31 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:31.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:31 smithi188 ceph-mon[44101]: pgmap v2111: 27 pgs: 1 active+recovering, 26 active+clean; 4.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 993 op/s; 3369/136608 objects misplaced (2.466%); 3.5 MiB/s, 56 objects/s recovering 2024-04-23T16:54:31.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:31.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:31.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:31 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:31.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:31 smithi148 ceph-mon[44520]: pgmap v2111: 27 pgs: 1 active+recovering, 26 active+clean; 4.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 993 op/s; 3369/136608 objects misplaced (2.466%); 3.5 MiB/s, 56 objects/s recovering 2024-04-23T16:54:31.807 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:54:31.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:31.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:54:31.913 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:54:31.948 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:32.152 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:54:32.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:32.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:54:32.263 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:54:32.305 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:32.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:32 smithi188 ceph-mon[44101]: 9.0 scrub starts 2024-04-23T16:54:32.514 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:54:32.516 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:32 smithi103 ceph-mon[39629]: 9.0 scrub starts 2024-04-23T16:54:32.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:32.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:54:32.619 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:54:32.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:32 smithi148 ceph-mon[44520]: 9.0 scrub starts 2024-04-23T16:54:32.659 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:32.721 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:54:32.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:32.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:54:32.829 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:54:32.865 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:33.124 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:54:33.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:33.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:54:33.237 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:54:33.265 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:33 smithi148 ceph-mon[44520]: 9.0 scrub ok 2024-04-23T16:54:33.265 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:33 smithi148 ceph-mon[44520]: 1.4 scrub starts 2024-04-23T16:54:33.266 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:33 smithi148 ceph-mon[44520]: 1.4 scrub ok 2024-04-23T16:54:33.266 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:33 smithi148 ceph-mon[44520]: pgmap v2112: 27 pgs: 1 active+recovering, 26 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 3066/130276 objects misplaced (2.353%); 3.7 MiB/s, 59 objects/s recovering 2024-04-23T16:54:33.268 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:33.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:33 smithi188 ceph-mon[44101]: 9.0 scrub ok 2024-04-23T16:54:33.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:33 smithi188 ceph-mon[44101]: 1.4 scrub starts 2024-04-23T16:54:33.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:33 smithi188 ceph-mon[44101]: 1.4 scrub ok 2024-04-23T16:54:33.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:33 smithi188 ceph-mon[44101]: pgmap v2112: 27 pgs: 1 active+recovering, 26 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 3066/130276 objects misplaced (2.353%); 3.7 MiB/s, 59 objects/s recovering 2024-04-23T16:54:33.484 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:33 smithi103 ceph-mon[39629]: 9.0 scrub ok 2024-04-23T16:54:33.484 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:33 smithi103 ceph-mon[39629]: 1.4 scrub starts 2024-04-23T16:54:33.484 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:33 smithi103 ceph-mon[39629]: 1.4 scrub ok 2024-04-23T16:54:33.484 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:33 smithi103 ceph-mon[39629]: pgmap v2112: 27 pgs: 1 active+recovering, 26 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 3066/130276 objects misplaced (2.353%); 3.7 MiB/s, 59 objects/s recovering 2024-04-23T16:54:33.722 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:54:33.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:33.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:54:33.806 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:54:33.827 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:54:33.912 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:33.952 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:54:33.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:33.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:54:34.056 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:54:34.117 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:34.152 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:54:34.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:34.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:54:34.170 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:54:34.170 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:54:34.170 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:34.170 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.27202.0:240519 9.1 e763/703 9:8e3a4aa0:::benchmark_data_smithi188_167360_object89386:head v 763'14728, mlcod=763'14726)", 2024-04-23T16:54:34.170 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:54:34.166429+0000", 2024-04-23T16:54:34.170 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.0031587640000000001, 2024-04-23T16:54:34.170 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0031882099999999999, 2024-04-23T16:54:34.170 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:54:34.171 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:54:34.171 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2024-04-23T16:54:34.171 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:54:34.171 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:34.171 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:54:34.171 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:34.166429+0000", 2024-04-23T16:54:34.171 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:34.171 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:34.171 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:34.171 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:54:34.171 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:34.166429+0000", 2024-04-23T16:54:34.171 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:34.171 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:34.171 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:34.171 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:54:34.172 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:34.166431+0000", 2024-04-23T16:54:34.172 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6390000000000001e-06 2024-04-23T16:54:34.172 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:34.172 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:34.172 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:54:34.172 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:34.166439+0000", 2024-04-23T16:54:34.172 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8550000000000002e-06 2024-04-23T16:54:34.172 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:34.172 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:34.172 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:54:34.172 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:34.166459+0000", 2024-04-23T16:54:34.172 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9964000000000001e-05 2024-04-23T16:54:34.172 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:34.172 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:34.173 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:54:34.173 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:34.166467+0000", 2024-04-23T16:54:34.173 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6950000000000006e-06 2024-04-23T16:54:34.173 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:34.173 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:34.173 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:54:34.173 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:34.166518+0000", 2024-04-23T16:54:34.173 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0668999999999999e-05 2024-04-23T16:54:34.173 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:34.173 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:34.173 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:54:34.173 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:34.166534+0000", 2024-04-23T16:54:34.173 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5996999999999999e-05 2024-04-23T16:54:34.173 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:34.173 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:54:34.173 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:34.174 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:34.174 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-04-23T16:54:34.174 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 1 2024-04-23T16:54:34.174 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:54:34.224 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:54:34.224 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:54:34.258 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:54:34.305 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:34.335 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:54:34.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:34.339 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:54:34.440 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:54:34.472 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:34 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/482911176' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:54:34.474 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:34.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:34 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/482911176' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:54:34.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:34 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/482911176' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:54:35.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:35 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:54:35.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:35 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/482911176' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:54:35.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:35 smithi188 ceph-mon[44101]: osdmap e764: 12 total, 12 up, 8 in 2024-04-23T16:54:35.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:35 smithi188 ceph-mon[44101]: pgmap v2114: 27 pgs: 1 active+recovering, 26 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 3066/128284 objects misplaced (2.390%); 4.2 MiB/s, 67 objects/s recovering 2024-04-23T16:54:35.564 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:35 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:54:35.564 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:35 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/482911176' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:54:35.564 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:35 smithi148 ceph-mon[44520]: osdmap e764: 12 total, 12 up, 8 in 2024-04-23T16:54:35.565 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:35 smithi148 ceph-mon[44520]: pgmap v2114: 27 pgs: 1 active+recovering, 26 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 3066/128284 objects misplaced (2.390%); 4.2 MiB/s, 67 objects/s recovering 2024-04-23T16:54:35.567 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:54:35.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:35.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:54:35.600 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-23T16:54:35.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:35 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:54:35.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:35 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/482911176' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:54:35.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:35 smithi103 ceph-mon[39629]: osdmap e764: 12 total, 12 up, 8 in 2024-04-23T16:54:35.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:35 smithi103 ceph-mon[39629]: pgmap v2114: 27 pgs: 1 active+recovering, 26 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 3066/128284 objects misplaced (2.390%); 4.2 MiB/s, 67 objects/s recovering 2024-04-23T16:54:35.676 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:54:35.720 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:35.751 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:54:35.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:35.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:54:35.861 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:54:35.900 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:35.959 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:54:36.064 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:54:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:36.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:54:36.172 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:54:36.202 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:36.225 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:54:36.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:36.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:54:36.330 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:54:36.366 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:36.578 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:54:36.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:36.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:54:36.686 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:54:36.734 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:36.758 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:54:36.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:36.768 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:54:36.869 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:54:36.906 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:36.946 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:54:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:36.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:54:37.054 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:54:37.106 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:37.138 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:54:37.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:37.146 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:54:37.247 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:54:37.280 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:54:37.289 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:37.324 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:54:37.336 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:54:37.350 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:54:37.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:37.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:54:37.378 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:54:37.403 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:54:37.428 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:54:37.452 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:54:37.455 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:54:37.483 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:37.484 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:54:37.507 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:54:37.534 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:54:37.569 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:54:37.598 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:54:37.603 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:54:37.603 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:54:37.603 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:54:37.603 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:54:37.603 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:54:37.603 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:54:37.604 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:54:37.604 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:54:37.604 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:54:37.604 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:54:37.604 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:54:37.604 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:54:37.735 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:54:37.737 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:54:37.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:37.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:54:37.839 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:54:37.894 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:38.059 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:54:38.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:38.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:54:38.169 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:54:38.202 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:38.227 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:54:38.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:38.234 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:54:38.334 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:54:38.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:38 smithi103 ceph-mon[39629]: pgmap v2115: 27 pgs: 1 active+recovering, 26 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 2744/121050 objects misplaced (2.267%); 3.9 MiB/s, 62 objects/s recovering 2024-04-23T16:54:38.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:38 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3325694343' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:54:38.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:38.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:38 smithi188 ceph-mon[44101]: pgmap v2115: 27 pgs: 1 active+recovering, 26 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 2744/121050 objects misplaced (2.267%); 3.9 MiB/s, 62 objects/s recovering 2024-04-23T16:54:38.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:38 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3325694343' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:54:38.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:38.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:38.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:38.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:38.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:38.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:38 smithi148 ceph-mon[44520]: pgmap v2115: 27 pgs: 1 active+recovering, 26 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 2744/121050 objects misplaced (2.267%); 3.9 MiB/s, 62 objects/s recovering 2024-04-23T16:54:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:38 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3325694343' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:54:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:38.377 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:38.398 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:54:38.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:38.407 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:54:38.507 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:54:38.536 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:38.581 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:54:38.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:38.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:54:38.690 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:54:38.749 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:38.784 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:54:38.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:38.791 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:54:38.892 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:54:38.927 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:39.023 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:54:39.023 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:54:39.087 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:54:39.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:39.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:54:39.192 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:54:39.225 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:39.240 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:54:39.253 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:54:39.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:39.259 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:54:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:39.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:39.361 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:54:39.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:39.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:39.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:39.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:39.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:39.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:39.392 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:39.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:39.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:39.646 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:54:39.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:39.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:54:39.751 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:54:39.798 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:39.833 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:54:39.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:39.843 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:54:39.944 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:54:39.988 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:40.117 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-23T16:54:40.193 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:54:40.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:40.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:54:40.200 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:40 smithi103 ceph-mon[39629]: pgmap v2116: 27 pgs: 1 active+recovering, 26 active+clean; 3.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 2744/117760 objects misplaced (2.330%); 3.9 MiB/s, 62 objects/s recovering 2024-04-23T16:54:40.298 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:54:40.330 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:40 smithi148 ceph-mon[44520]: pgmap v2116: 27 pgs: 1 active+recovering, 26 active+clean; 3.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 2744/117760 objects misplaced (2.330%); 3.9 MiB/s, 62 objects/s recovering 2024-04-23T16:54:40.332 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:40.368 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:54:40.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:40.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:54:40.460 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:54:40.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:40 smithi188 ceph-mon[44101]: pgmap v2116: 27 pgs: 1 active+recovering, 26 active+clean; 3.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 2744/117760 objects misplaced (2.330%); 3.9 MiB/s, 62 objects/s recovering 2024-04-23T16:54:40.480 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:54:40.513 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:40.774 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:54:40.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:40.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:54:40.879 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:54:40.908 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:41.009 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:54:41.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:41.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:54:41.113 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:54:41.117 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:54:41.174 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:41.202 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:54:41.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:41.209 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:54:41.310 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:54:41.365 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:41.403 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:54:41.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:41.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:54:41.507 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:54:41.539 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:54:41.550 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:41.585 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:54:41.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:41.589 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:54:41.689 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:54:41.718 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:41.746 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:54:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:41.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:54:41.853 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:54:41.882 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:42.107 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:54:42.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:42 smithi103 ceph-mon[39629]: pgmap v2117: 27 pgs: 1 active+recovering, 26 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 2744/115878 objects misplaced (2.368%); 3.9 MiB/s, 62 objects/s recovering 2024-04-23T16:54:42.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:42 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4131147739' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:54:42.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:42 smithi148 ceph-mon[44520]: pgmap v2117: 27 pgs: 1 active+recovering, 26 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 2744/115878 objects misplaced (2.368%); 3.9 MiB/s, 62 objects/s recovering 2024-04-23T16:54:42.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:42 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4131147739' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:54:42.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:42 smithi188 ceph-mon[44101]: pgmap v2117: 27 pgs: 1 active+recovering, 26 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 2744/115878 objects misplaced (2.368%); 3.9 MiB/s, 62 objects/s recovering 2024-04-23T16:54:42.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:42 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4131147739' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:54:42.934 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:54:42.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:42.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:54:43.038 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:54:43.066 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:43.175 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:54:43.179 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:43 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:54:43.179 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:43 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:54:43.179 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:43 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:54:43.179 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:43 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4131147739' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:54:43.179 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:43 smithi148 ceph-mon[44520]: osdmap e765: 12 total, 12 up, 8 in 2024-04-23T16:54:43.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:43.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:54:43.282 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:54:43.313 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:43.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:43 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:54:43.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:43 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:54:43.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:43 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:54:43.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:43 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4131147739' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:54:43.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:43 smithi103 ceph-mon[39629]: osdmap e765: 12 total, 12 up, 8 in 2024-04-23T16:54:43.371 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:54:43.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:43.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:54:43.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:43 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:54:43.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:43 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:54:43.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:43 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:54:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:43 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4131147739' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:54:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:43 smithi188 ceph-mon[44101]: osdmap e765: 12 total, 12 up, 8 in 2024-04-23T16:54:43.477 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:54:43.520 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:43.551 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:54:43.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:43.558 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:54:43.561 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:54:43.561 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:54:43.659 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:54:43.702 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:43.728 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:54:43.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:43.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:54:43.836 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:54:43.878 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:43.901 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:54:43.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:43.904 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:54:44.005 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:54:44.047 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:44.182 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:54:44.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:44.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:54:44.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:44 smithi188 ceph-mon[44101]: pgmap v2119: 27 pgs: 1 active+recovering, 26 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 2416/109188 objects misplaced (2.213%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:54:44.231 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-23T16:54:44.231 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:54:44.287 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:54:44.346 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:44.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:44 smithi103 ceph-mon[39629]: pgmap v2119: 27 pgs: 1 active+recovering, 26 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 2416/109188 objects misplaced (2.213%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:54:44.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:44 smithi148 ceph-mon[44520]: pgmap v2119: 27 pgs: 1 active+recovering, 26 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 2416/109188 objects misplaced (2.213%); 4.6 MiB/s, 73 objects/s recovering 2024-04-23T16:54:44.377 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:54:44.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:44.385 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:54:44.487 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:54:44.550 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:54:44.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:44.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:54:44.696 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:54:44.744 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:44.778 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:54:44.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:44.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:54:44.881 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:54:44.883 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:54:44.914 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:54:44.935 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:44.949 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:54:44.977 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:54:45.008 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:54:45.029 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:54:45.057 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:54:45.092 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:54:45.113 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:54:45.151 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:54:45.189 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:54:45.218 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:54:45.236 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-23T16:54:45.244 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:54:45.245 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:54:45.246 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:54:45.246 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:54:45.246 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:54:45.246 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:54:45.246 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:54:45.246 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:54:45.246 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:54:45.246 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:54:45.246 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:54:45.246 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:54:45.246 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:54:45.357 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:54:45.532 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:45 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2336236658' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:54:45.532 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:45 smithi148 ceph-mon[44520]: pgmap v2120: 27 pgs: 1 active+recovering, 26 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 2416/107170 objects misplaced (2.254%); 4.1 MiB/s, 64 objects/s recovering 2024-04-23T16:54:45.532 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:45 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:45.532 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:45 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:45.532 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:45 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:45.532 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:45 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:45.532 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:45 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:45.573 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:54:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:45 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2336236658' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:54:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:45 smithi103 ceph-mon[39629]: pgmap v2120: 27 pgs: 1 active+recovering, 26 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 2416/107170 objects misplaced (2.254%); 4.1 MiB/s, 64 objects/s recovering 2024-04-23T16:54:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:45 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:45 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:45 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:45 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:45 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:45.904 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:54:45.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:45.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:54:45.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:45 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2336236658' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:54:45.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:45 smithi188 ceph-mon[44101]: pgmap v2120: 27 pgs: 1 active+recovering, 26 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 2416/107170 objects misplaced (2.254%); 4.1 MiB/s, 64 objects/s recovering 2024-04-23T16:54:45.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:45 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:45.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:45 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:45.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:45 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:45.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:45 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:45.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:45 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.016 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:54:46.047 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:46.247 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:54:46.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:46.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:54:46.353 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:54:46.390 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:46.746 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.747 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.747 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.747 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.747 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.747 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.748 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.754 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:54:46.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:46.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:54:46.858 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:54:46.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.913 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:46.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:46.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:47.172 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:54:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:47.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:54:47.279 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:54:47.319 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:47.342 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:54:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:47.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:54:47.449 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:54:47.484 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:47.552 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:47 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:54:47.553 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:47 smithi148 ceph-mon[44520]: pgmap v2121: 27 pgs: 1 active+recovering, 26 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 886 op/s; 2068/103306 objects misplaced (2.002%); 4.2 MiB/s, 67 objects/s recovering 2024-04-23T16:54:47.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:47 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:54:47.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:47 smithi188 ceph-mon[44101]: pgmap v2121: 27 pgs: 1 active+recovering, 26 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 886 op/s; 2068/103306 objects misplaced (2.002%); 4.2 MiB/s, 67 objects/s recovering 2024-04-23T16:54:47.726 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:54:47.737 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:54:47.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:47.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:54:47.748 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:47 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:54:47.748 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:47 smithi103 ceph-mon[39629]: pgmap v2121: 27 pgs: 1 active+recovering, 26 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 886 op/s; 2068/103306 objects misplaced (2.002%); 4.2 MiB/s, 67 objects/s recovering 2024-04-23T16:54:47.849 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:54:47.878 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:48.030 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:54:48.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:48.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:54:48.136 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:54:48.190 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:48.224 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:54:48.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:48.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:54:48.330 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:54:48.382 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:48.383 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:54:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2024-04-23T16:54:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.27202.0:255414 9.d 9:bfa08164:::benchmark_data_smithi188_167360_object104281:head [delete] snapc 0=[] ondisk+write+known_if_redirected e765)", 2024-04-23T16:54:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:54:48.373350+0000", 2024-04-23T16:54:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.0092161069999999994, 2024-04-23T16:54:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0092446019999999993, 2024-04-23T16:54:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:54:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:54:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:54:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:54:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.27202", 2024-04-23T16:54:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:54:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 255414 2024-04-23T16:54:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:54:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:54:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.373350+0000", 2024-04-23T16:54:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:54:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.373350+0000", 2024-04-23T16:54:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:54:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.373353+0000", 2024-04-23T16:54:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5000000000000002e-06 2024-04-23T16:54:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:54:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.373360+0000", 2024-04-23T16:54:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7940000000000003e-06 2024-04-23T16:54:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:54:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.373376+0000", 2024-04-23T16:54:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5913000000000002e-05 2024-04-23T16:54:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:54:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.373384+0000", 2024-04-23T16:54:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0879999999999995e-06 2024-04-23T16:54:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:54:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.373435+0000", 2024-04-23T16:54:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0708000000000003e-05 2024-04-23T16:54:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:54:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.373564+0000", 2024-04-23T16:54:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000129061 2024-04-23T16:54:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 1", 2024-04-23T16:54:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.373688+0000", 2024-04-23T16:54:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012422600000000001 2024-04-23T16:54:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:54:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.378766+0000", 2024-04-23T16:54:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00507786 2024-04-23T16:54:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:54:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.27202.0:255417 9.e 9:770d76c8:::benchmark_data_smithi188_167360_object104284:head [delete] snapc 0=[] ondisk+write+known_if_redirected e765)", 2024-04-23T16:54:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:54:48.373450+0000", 2024-04-23T16:54:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.0091158109999999997, 2024-04-23T16:54:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0092274390000000005, 2024-04-23T16:54:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:54:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:54:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:54:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:54:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.27202", 2024-04-23T16:54:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:54:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 255417 2024-04-23T16:54:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:54:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:54:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.373450+0000", 2024-04-23T16:54:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:54:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.373450+0000", 2024-04-23T16:54:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:54:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.373452+0000", 2024-04-23T16:54:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5990000000000001e-06 2024-04-23T16:54:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:54:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.373455+0000", 2024-04-23T16:54:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5240000000000001e-06 2024-04-23T16:54:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:54:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.373461+0000", 2024-04-23T16:54:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3090000000000003e-06 2024-04-23T16:54:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:54:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.373465+0000", 2024-04-23T16:54:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0640000000000004e-06 2024-04-23T16:54:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:54:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.373507+0000", 2024-04-23T16:54:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2033000000000002e-05 2024-04-23T16:54:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:54:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.373618+0000", 2024-04-23T16:54:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011063100000000001 2024-04-23T16:54:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 1", 2024-04-23T16:54:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.373713+0000", 2024-04-23T16:54:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5332000000000001e-05 2024-04-23T16:54:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:54:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.378652+0000", 2024-04-23T16:54:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0049393129999999999 2024-04-23T16:54:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:54:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.27202.0:255420 9.7 9:ee899a25:::benchmark_data_smithi188_167360_object104287:head [delete] snapc 0=[] ondisk+write+known_if_redirected e765)", 2024-04-23T16:54:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:54:48.374557+0000", 2024-04-23T16:54:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.0080095129999999994, 2024-04-23T16:54:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0081954149999999993, 2024-04-23T16:54:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:54:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:54:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "waiting for sub ops", 2024-04-23T16:54:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2024-04-23T16:54:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.27202", 2024-04-23T16:54:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:54:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 255420 2024-04-23T16:54:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:54:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:54:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.374557+0000", 2024-04-23T16:54:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:54:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.374557+0000", 2024-04-23T16:54:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:54:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.374558+0000", 2024-04-23T16:54:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1069999999999999e-06 2024-04-23T16:54:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:54:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.374562+0000", 2024-04-23T16:54:48.396 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1690000000000002e-06 2024-04-23T16:54:48.396 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.396 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.396 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:54:48.396 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.374571+0000", 2024-04-23T16:54:48.396 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6940000000000001e-06 2024-04-23T16:54:48.396 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.396 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.396 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:54:48.396 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.374578+0000", 2024-04-23T16:54:48.396 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2010000000000002e-06 2024-04-23T16:54:48.396 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.396 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.396 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:54:48.397 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.374622+0000", 2024-04-23T16:54:48.397 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4354000000000003e-05 2024-04-23T16:54:48.397 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.397 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.397 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2024-04-23T16:54:48.397 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.374712+0000", 2024-04-23T16:54:48.397 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9770000000000003e-05 2024-04-23T16:54:48.397 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.397 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.397 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for subops from 2", 2024-04-23T16:54:48.397 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.374788+0000", 2024-04-23T16:54:48.397 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6339000000000002e-05 2024-04-23T16:54:48.397 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.397 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.398 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op_commit", 2024-04-23T16:54:48.398 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.378543+0000", 2024-04-23T16:54:48.398 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0037546160000000001 2024-04-23T16:54:48.398 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:48.398 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:54:48.398 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:48.398 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.398 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.398 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPushReply(9.7 765/732 [PushReplyOp(9:e75e362b:::benchmark_data_smithi188_167360_object104997:head)])", 2024-04-23T16:54:48.398 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:54:48.381694+0000", 2024-04-23T16:54:48.398 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.00087210100000000002, 2024-04-23T16:54:48.398 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0011304399999999999, 2024-04-23T16:54:48.398 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:54:48.398 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:54:48.399 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2024-04-23T16:54:48.399 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:54:48.399 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.399 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:54:48.399 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.381694+0000", 2024-04-23T16:54:48.399 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:48.399 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.399 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.399 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:54:48.399 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.381694+0000", 2024-04-23T16:54:48.399 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:48.399 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.400 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.400 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:54:48.400 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.381696+0000", 2024-04-23T16:54:48.400 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.699e-06 2024-04-23T16:54:48.400 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.400 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.400 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:54:48.400 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.381701+0000", 2024-04-23T16:54:48.400 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8760000000000004e-06 2024-04-23T16:54:48.400 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.400 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.400 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:54:48.400 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.381711+0000", 2024-04-23T16:54:48.400 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0396e-05 2024-04-23T16:54:48.401 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.401 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.401 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:54:48.401 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.381717+0000", 2024-04-23T16:54:48.401 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0800000000000002e-06 2024-04-23T16:54:48.401 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.401 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.401 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2024-04-23T16:54:48.401 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.381763+0000", 2024-04-23T16:54:48.401 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5558e-05 2024-04-23T16:54:48.401 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:48.401 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:54:48.401 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:48.401 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.401 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.402 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop_reply(client.27202.0:255420 9.7 e765/732)", 2024-04-23T16:54:48.402 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2024-04-23T16:54:48.382326+0000", 2024-04-23T16:54:48.402 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.00023988199999999999, 2024-04-23T16:54:48.402 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00054552699999999997, 2024-04-23T16:54:48.402 INFO:teuthology.orchestra.run.smithi148.stdout: "continuous": false, 2024-04-23T16:54:48.402 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2024-04-23T16:54:48.402 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "queued for pg", 2024-04-23T16:54:48.402 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2024-04-23T16:54:48.402 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.402 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2024-04-23T16:54:48.402 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.382326+0000", 2024-04-23T16:54:48.402 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:48.402 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.403 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.403 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2024-04-23T16:54:48.403 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.382326+0000", 2024-04-23T16:54:48.403 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2024-04-23T16:54:48.403 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.403 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.403 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2024-04-23T16:54:48.403 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.382327+0000", 2024-04-23T16:54:48.403 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1699999999999997e-07 2024-04-23T16:54:48.403 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.403 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.403 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2024-04-23T16:54:48.403 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.382330+0000", 2024-04-23T16:54:48.403 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4450000000000001e-06 2024-04-23T16:54:48.404 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.404 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.404 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2024-04-23T16:54:48.404 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.382336+0000", 2024-04-23T16:54:48.404 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9699999999999996e-06 2024-04-23T16:54:48.404 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-04-23T16:54:48.404 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-04-23T16:54:48.404 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2024-04-23T16:54:48.404 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2024-04-23T16:54:48.382341+0000", 2024-04-23T16:54:48.404 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2579999999999997e-06 2024-04-23T16:54:48.404 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:48.404 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-04-23T16:54:48.404 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:48.404 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-04-23T16:54:48.404 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-04-23T16:54:48.405 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 5 2024-04-23T16:54:48.405 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:54:48.416 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:54:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:48.423 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:54:48.524 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:54:48.572 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:48.599 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:54:48.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:48.602 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:54:48.703 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:54:48.739 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:48.779 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:54:49.464 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:54:49.465 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:54:49.691 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:54:49.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:49.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:54:49.717 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-23T16:54:49.799 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:54:49.845 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:49.886 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:54:49.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:49.890 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:54:49.991 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:54:50.025 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:50.051 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:54:50.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:50.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:54:50.078 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:54:50.154 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:54:50.200 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:50 smithi188 ceph-mon[44101]: from='client.37199 v1:172.21.15.103:0/509194858' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:54:50.200 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:50 smithi188 ceph-mon[44101]: pgmap v2122: 27 pgs: 1 active+recovering, 26 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 2068/96836 objects misplaced (2.136%); 4.2 MiB/s, 67 objects/s recovering 2024-04-23T16:54:50.202 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:50.239 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:54:50.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:50.247 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:54:50.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:50 smithi103 ceph-mon[39629]: from='client.37199 v1:172.21.15.103:0/509194858' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:54:50.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:50 smithi103 ceph-mon[39629]: pgmap v2122: 27 pgs: 1 active+recovering, 26 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 2068/96836 objects misplaced (2.136%); 4.2 MiB/s, 67 objects/s recovering 2024-04-23T16:54:50.346 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:50 smithi148 ceph-mon[44520]: from='client.37199 v1:172.21.15.103:0/509194858' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:54:50.346 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:50 smithi148 ceph-mon[44520]: pgmap v2122: 27 pgs: 1 active+recovering, 26 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 2068/96836 objects misplaced (2.136%); 4.2 MiB/s, 67 objects/s recovering 2024-04-23T16:54:50.348 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:54:50.395 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:50.434 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:54:50.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:50.438 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:54:50.539 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:54:50.571 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:50.629 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:54:50.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:50.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:54:50.736 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:54:50.771 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:51.137 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":2121,"stamp":"2024-04-23T16:54:47.002136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3385524792,"num_objects":51653,"num_object_clones":0,"num_object_copies":103306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2068,"num_objects_unfound":0,"num_objects_dirty":51653,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":250611,"num_write_kb":9672321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6519,"num_bytes_recovered":427636320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162454,"ondisk_log_size":162454,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":12319772,"kb_used_data":7061644,"kb_used_omap":522,"kb_used_meta":5257205,"kb_avail":737641444,"statfs":{"total":767960285184,"available":755344838656,"internally_reserved":0,"allocated":7231123456,"data_stored":7228019540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":534579,"internal_metadata":5383378893},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-581435392,"num_objects":-8872,"num_object_clones":0,"num_object_copies":-17744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-676,"num_objects_unfound":0,"num_objects_dirty":-8872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8872,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":676,"num_bytes_recovered":44302336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003768"},"pg_stats":[{"pgid":"9.7","version":"765'16026","reported_seq":29196,"reported_epoch":765,"state":"active+recovering","last_fresh":"2024-04-23T16:54:46.321933+0000","last_change":"2024-04-23T16:52:44.746944+0000","last_active":"2024-04-23T16:54:46.321933+0000","last_peered":"2024-04-23T16:54:46.321933+0000","last_clean":"2024-04-23T16:52:43.617275+0000","last_became_active":"2024-04-23T16:52:44.683405+0000","last_became_peered":"2024-04-23T16:52:44.683405+0000","last_unstale":"2024-04-23T16:54:46.321933+0000","last_undegraded":"2024-04-23T16:54:46.321933+0000","last_fullsized":"2024-04-23T16:54:46.321933+0000","mapping_epoch":732,"log_start":"721'6000","ondisk_log_start":"721'6000","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199491584,"num_objects":3044,"num_object_clones":0,"num_object_copies":6088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2068,"num_objects_unfound":0,"num_objects_dirty":3044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16026,"num_write_kb":610240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6516,"num_bytes_recovered":427032576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":2068},{"shards":"2,7","objects":976}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"764'15188","reported_seq":15405,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-04-23T16:54:41.971854+0000","last_change":"2024-04-23T16:53:26.127957+0000","last_active":"2024-04-23T16:54:41.971854+0000","last_peered":"2024-04-23T16:54:41.971854+0000","last_clean":"2024-04-23T16:54:41.971854+0000","last_became_active":"2024-04-23T16:51:21.063642+0000","last_became_peered":"2024-04-23T16:51:21.063642+0000","last_unstale":"2024-04-23T16:54:41.971854+0000","last_undegraded":"2024-04-23T16:54:41.971854+0000","last_fullsized":"2024-04-23T16:54:41.971854+0000","mapping_epoch":703,"log_start":"719'5100","ondisk_log_start":"719'5100","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"745'10860","last_scrub_stamp":"2024-04-23T16:53:26.127859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:53:26.127859+0000","objects_scrubbed":7678,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:54:43.531872+0000","scrub_duration":1541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216530944,"num_objects":3304,"num_object_clones":0,"num_object_copies":6608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15188,"num_write_kb":591744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"765'15459","reported_seq":15735,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:42.774635+0000","last_change":"2024-04-23T16:54:21.272785+0000","last_active":"2024-04-23T16:54:42.774635+0000","last_peered":"2024-04-23T16:54:42.774635+0000","last_clean":"2024-04-23T16:54:42.774635+0000","last_became_active":"2024-04-23T16:51:21.063776+0000","last_became_peered":"2024-04-23T16:51:21.063776+0000","last_unstale":"2024-04-23T16:54:42.774635+0000","last_undegraded":"2024-04-23T16:54:42.774635+0000","last_fullsized":"2024-04-23T16:54:42.774635+0000","mapping_epoch":703,"log_start":"720'5400","ondisk_log_start":"720'5400","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14080","last_scrub_stamp":"2024-04-23T16:54:21.272505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:21.272505+0000","objects_scrubbed":4691,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:50.192674+0000","scrub_duration":969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215023616,"num_objects":3281,"num_object_clones":0,"num_object_copies":6562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15459,"num_write_kb":599680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"764'15459","reported_seq":15686,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-04-23T16:54:41.956672+0000","last_change":"2024-04-23T16:52:48.980259+0000","last_active":"2024-04-23T16:54:41.956672+0000","last_peered":"2024-04-23T16:54:41.956672+0000","last_clean":"2024-04-23T16:54:41.956672+0000","last_became_active":"2024-04-23T16:51:21.063114+0000","last_became_peered":"2024-04-23T16:51:21.063114+0000","last_unstale":"2024-04-23T16:54:41.956672+0000","last_undegraded":"2024-04-23T16:54:41.956672+0000","last_fullsized":"2024-04-23T16:54:41.956672+0000","mapping_epoch":703,"log_start":"720'5400","ondisk_log_start":"720'5400","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"736'8752","last_scrub_stamp":"2024-04-23T16:52:48.980138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:52:48.980138+0000","objects_scrubbed":8709,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:54:46.779024+0000","scrub_duration":1449,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221052928,"num_objects":3373,"num_object_clones":0,"num_object_copies":6746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15459,"num_write_kb":602624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"765'15735","reported_seq":15968,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:43.342471+0000","last_change":"2024-04-23T16:53:29.947031+0000","last_active":"2024-04-23T16:54:43.342471+0000","last_peered":"2024-04-23T16:54:43.342471+0000","last_clean":"2024-04-23T16:54:43.342471+0000","last_became_active":"2024-04-23T16:51:21.063652+0000","last_became_peered":"2024-04-23T16:51:21.063652+0000","last_unstale":"2024-04-23T16:54:43.342471+0000","last_undegraded":"2024-04-23T16:54:43.342471+0000","last_fullsized":"2024-04-23T16:54:43.342471+0000","mapping_epoch":703,"log_start":"720'5700","ondisk_log_start":"720'5700","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"746'11271","last_scrub_stamp":"2024-04-23T16:53:29.946887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:53:29.946887+0000","objects_scrubbed":7717,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:54:45.136918+0000","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210436096,"num_objects":3211,"num_object_clones":0,"num_object_copies":6422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15735,"num_write_kb":606272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.2","version":"765'15872","reported_seq":16148,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:43.321776+0000","last_change":"2024-04-23T16:54:24.962245+0000","last_active":"2024-04-23T16:54:43.321776+0000","last_peered":"2024-04-23T16:54:43.321776+0000","last_clean":"2024-04-23T16:54:43.321776+0000","last_became_active":"2024-04-23T16:51:21.062561+0000","last_became_peered":"2024-04-23T16:51:21.062561+0000","last_unstale":"2024-04-23T16:54:43.321776+0000","last_undegraded":"2024-04-23T16:54:43.321776+0000","last_fullsized":"2024-04-23T16:54:43.321776+0000","mapping_epoch":703,"log_start":"720'5800","ondisk_log_start":"720'5800","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14622","last_scrub_stamp":"2024-04-23T16:54:24.962060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:24.962060+0000","objects_scrubbed":4531,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:49.051124+0000","scrub_duration":1162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212598784,"num_objects":3244,"num_object_clones":0,"num_object_copies":6488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15872,"num_write_kb":611712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1887,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:42.086699+0000","last_change":"2024-04-23T16:54:28.362501+0000","last_active":"2024-04-23T16:54:42.086699+0000","last_peered":"2024-04-23T16:54:42.086699+0000","last_clean":"2024-04-23T16:54:42.086699+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:54:42.086699+0000","last_undegraded":"2024-04-23T16:54:42.086699+0000","last_fullsized":"2024-04-23T16:54:42.086699+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:54:28.362406+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:54:28.362406+0000","last_clean_scrub_stamp":"2024-04-23T16:54:28.362406+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:28.652451+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"764'15709","reported_seq":15988,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-04-23T16:54:41.971641+0000","last_change":"2024-04-23T16:54:31.665323+0000","last_active":"2024-04-23T16:54:41.971641+0000","last_peered":"2024-04-23T16:54:41.971641+0000","last_clean":"2024-04-23T16:54:41.971641+0000","last_became_active":"2024-04-23T16:51:21.063395+0000","last_became_peered":"2024-04-23T16:51:21.063395+0000","last_unstale":"2024-04-23T16:54:41.971641+0000","last_undegraded":"2024-04-23T16:54:41.971641+0000","last_fullsized":"2024-04-23T16:54:41.971641+0000","mapping_epoch":703,"log_start":"720'5700","ondisk_log_start":"720'5700","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"763'15015","last_scrub_stamp":"2024-04-23T16:54:31.665200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:31.665200+0000","objects_scrubbed":4086,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:39.674179+0000","scrub_duration":876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220790784,"num_objects":3369,"num_object_clones":0,"num_object_copies":6738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15709,"num_write_kb":610496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1660,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:42.101754+0000","last_change":"2024-04-23T16:52:47.483911+0000","last_active":"2024-04-23T16:54:42.101754+0000","last_peered":"2024-04-23T16:54:42.101754+0000","last_clean":"2024-04-23T16:54:42.101754+0000","last_became_active":"2024-04-23T16:52:47.483615+0000","last_became_peered":"2024-04-23T16:52:47.483615+0000","last_unstale":"2024-04-23T16:54:42.101754+0000","last_undegraded":"2024-04-23T16:54:42.101754+0000","last_fullsized":"2024-04-23T16:54:42.101754+0000","mapping_epoch":735,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":736,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:52:05.504325+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:52:05.504325+0000","last_clean_scrub_stamp":"2024-04-23T16:52:05.504325+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"765'15362","reported_seq":15636,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:45.071829+0000","last_change":"2024-04-23T16:54:17.136201+0000","last_active":"2024-04-23T16:54:45.071829+0000","last_peered":"2024-04-23T16:54:45.071829+0000","last_clean":"2024-04-23T16:54:45.071829+0000","last_became_active":"2024-04-23T16:51:21.246391+0000","last_became_peered":"2024-04-23T16:51:21.246391+0000","last_unstale":"2024-04-23T16:54:45.071829+0000","last_undegraded":"2024-04-23T16:54:45.071829+0000","last_fullsized":"2024-04-23T16:54:45.071829+0000","mapping_epoch":703,"log_start":"720'5300","ondisk_log_start":"720'5300","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'13668","last_scrub_stamp":"2024-04-23T16:54:17.136038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:17.136038+0000","objects_scrubbed":4818,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:23.005433+0000","scrub_duration":963,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202637312,"num_objects":3092,"num_object_clones":0,"num_object_copies":6184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15362,"num_write_kb":590528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1871,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:42.090143+0000","last_change":"2024-04-23T16:52:38.199979+0000","last_active":"2024-04-23T16:54:42.090143+0000","last_peered":"2024-04-23T16:54:42.090143+0000","last_clean":"2024-04-23T16:54:42.090143+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:54:42.090143+0000","last_undegraded":"2024-04-23T16:54:42.090143+0000","last_fullsized":"2024-04-23T16:54:42.090143+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:52:38.199895+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:51:25.290774+0000","last_clean_scrub_stamp":"2024-04-23T16:52:38.199895+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"764'15673","reported_seq":15951,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-04-23T16:54:41.911529+0000","last_change":"2024-04-23T16:54:18.910760+0000","last_active":"2024-04-23T16:54:41.911529+0000","last_peered":"2024-04-23T16:54:41.911529+0000","last_clean":"2024-04-23T16:54:41.911529+0000","last_became_active":"2024-04-23T16:51:21.063669+0000","last_became_peered":"2024-04-23T16:51:21.063669+0000","last_unstale":"2024-04-23T16:54:41.911529+0000","last_undegraded":"2024-04-23T16:54:41.911529+0000","last_fullsized":"2024-04-23T16:54:41.911529+0000","mapping_epoch":703,"log_start":"720'5600","ondisk_log_start":"720'5600","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14143","last_scrub_stamp":"2024-04-23T16:54:18.910481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:18.910481+0000","objects_scrubbed":4947,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:29.963181+0000","scrub_duration":1208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221708288,"num_objects":3383,"num_object_clones":0,"num_object_copies":6766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15673,"num_write_kb":609792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1885,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:42.168827+0000","last_change":"2024-04-23T16:53:25.567832+0000","last_active":"2024-04-23T16:54:42.168827+0000","last_peered":"2024-04-23T16:54:42.168827+0000","last_clean":"2024-04-23T16:54:42.168827+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:54:42.168827+0000","last_undegraded":"2024-04-23T16:54:42.168827+0000","last_fullsized":"2024-04-23T16:54:42.168827+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:53:25.567686+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:53:25.567686+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:54:43.744211+0000","scrub_duration":275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"765'15866","reported_seq":16146,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:43.362821+0000","last_change":"2024-04-23T16:54:19.829779+0000","last_active":"2024-04-23T16:54:43.362821+0000","last_peered":"2024-04-23T16:54:43.362821+0000","last_clean":"2024-04-23T16:54:43.362821+0000","last_became_active":"2024-04-23T16:51:21.062760+0000","last_became_peered":"2024-04-23T16:51:21.062760+0000","last_unstale":"2024-04-23T16:54:43.362821+0000","last_undegraded":"2024-04-23T16:54:43.362821+0000","last_fullsized":"2024-04-23T16:54:43.362821+0000","mapping_epoch":703,"log_start":"720'5800","ondisk_log_start":"720'5800","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14356","last_scrub_stamp":"2024-04-23T16:54:19.829518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:19.829518+0000","objects_scrubbed":4782,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:22.348945+0000","scrub_duration":1029,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212860928,"num_objects":3248,"num_object_clones":0,"num_object_copies":6496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15866,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1467,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-04-23T16:54:34.214913+0000","last_change":"2024-04-23T16:53:25.755305+0000","last_active":"2024-04-23T16:54:34.214913+0000","last_peered":"2024-04-23T16:54:34.214913+0000","last_clean":"2024-04-23T16:54:34.214913+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:54:34.214913+0000","last_undegraded":"2024-04-23T16:54:34.214913+0000","last_fullsized":"2024-04-23T16:54:34.214913+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:53:25.755173+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:53:25.755173+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:54:48.864398+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"765'15537","reported_seq":15818,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:42.782630+0000","last_change":"2024-04-23T16:54:19.146832+0000","last_active":"2024-04-23T16:54:42.782630+0000","last_peered":"2024-04-23T16:54:42.782630+0000","last_clean":"2024-04-23T16:54:42.782630+0000","last_became_active":"2024-04-23T16:51:21.246519+0000","last_became_peered":"2024-04-23T16:51:21.246519+0000","last_unstale":"2024-04-23T16:54:42.782630+0000","last_undegraded":"2024-04-23T16:54:42.782630+0000","last_fullsized":"2024-04-23T16:54:42.782630+0000","mapping_epoch":703,"log_start":"720'5500","ondisk_log_start":"720'5500","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'13985","last_scrub_stamp":"2024-04-23T16:54:19.146171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:19.146171+0000","objects_scrubbed":4796,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:28.380772+0000","scrub_duration":863,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210829312,"num_objects":3217,"num_object_clones":0,"num_object_copies":6434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15537,"num_write_kb":600128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1429,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:42.095778+0000","last_change":"2024-04-23T16:53:42.723708+0000","last_active":"2024-04-23T16:54:42.095778+0000","last_peered":"2024-04-23T16:54:42.095778+0000","last_clean":"2024-04-23T16:54:42.095778+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:54:42.095778+0000","last_undegraded":"2024-04-23T16:54:42.095778+0000","last_fullsized":"2024-04-23T16:54:42.095778+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:53:42.723618+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:53:42.723618+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:54:49.698863+0000","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.e","version":"765'15867","reported_seq":16181,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:46.259931+0000","last_change":"2024-04-23T16:52:41.066866+0000","last_active":"2024-04-23T16:54:46.259931+0000","last_peered":"2024-04-23T16:54:46.259931+0000","last_clean":"2024-04-23T16:54:46.259931+0000","last_became_active":"2024-04-23T16:51:21.064486+0000","last_became_peered":"2024-04-23T16:51:21.064486+0000","last_unstale":"2024-04-23T16:54:46.259931+0000","last_undegraded":"2024-04-23T16:54:46.259931+0000","last_fullsized":"2024-04-23T16:54:46.259931+0000","mapping_epoch":703,"log_start":"721'5800","ondisk_log_start":"721'5800","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"730'8312","last_scrub_stamp":"2024-04-23T16:52:41.066729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:52:41.066729+0000","objects_scrubbed":8276,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203882496,"num_objects":3111,"num_object_clones":0,"num_object_copies":6222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15867,"num_write_kb":607296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1895,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:42.086802+0000","last_change":"2024-04-23T16:54:26.409871+0000","last_active":"2024-04-23T16:54:42.086802+0000","last_peered":"2024-04-23T16:54:42.086802+0000","last_clean":"2024-04-23T16:54:42.086802+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:54:42.086802+0000","last_undegraded":"2024-04-23T16:54:42.086802+0000","last_fullsized":"2024-04-23T16:54:42.086802+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:54:26.409705+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:54:26.409705+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:33.984886+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.f","version":"764'15734","reported_seq":16014,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-04-23T16:54:41.981676+0000","last_change":"2024-04-23T16:54:17.546594+0000","last_active":"2024-04-23T16:54:41.981676+0000","last_peered":"2024-04-23T16:54:41.981676+0000","last_clean":"2024-04-23T16:54:41.981676+0000","last_became_active":"2024-04-23T16:51:21.063588+0000","last_became_peered":"2024-04-23T16:51:21.063588+0000","last_unstale":"2024-04-23T16:54:41.981676+0000","last_undegraded":"2024-04-23T16:54:41.981676+0000","last_fullsized":"2024-04-23T16:54:41.981676+0000","mapping_epoch":703,"log_start":"720'5700","ondisk_log_start":"720'5700","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14153","last_scrub_stamp":"2024-04-23T16:54:17.546419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:17.546419+0000","objects_scrubbed":4978,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:23.570053+0000","scrub_duration":812,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221380608,"num_objects":3378,"num_object_clones":0,"num_object_copies":6756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15734,"num_write_kb":611584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1898,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:42.092455+0000","last_change":"2024-04-23T16:53:28.261207+0000","last_active":"2024-04-23T16:54:42.092455+0000","last_peered":"2024-04-23T16:54:42.092455+0000","last_clean":"2024-04-23T16:54:42.092455+0000","last_became_active":"2024-04-23T16:52:44.474580+0000","last_became_peered":"2024-04-23T16:52:44.474580+0000","last_unstale":"2024-04-23T16:54:42.092455+0000","last_undegraded":"2024-04-23T16:54:42.092455+0000","last_fullsized":"2024-04-23T16:54:42.092455+0000","mapping_epoch":732,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":733,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:53:28.261070+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:53:28.261070+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:54:43.027937+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"765'15849","reported_seq":16151,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:45.064945+0000","last_change":"2024-04-23T16:53:42.243957+0000","last_active":"2024-04-23T16:54:45.064945+0000","last_peered":"2024-04-23T16:54:45.064945+0000","last_clean":"2024-04-23T16:54:45.064945+0000","last_became_active":"2024-04-23T16:51:21.079798+0000","last_became_peered":"2024-04-23T16:51:21.079798+0000","last_unstale":"2024-04-23T16:54:45.064945+0000","last_undegraded":"2024-04-23T16:54:45.064945+0000","last_fullsized":"2024-04-23T16:54:45.064945+0000","mapping_epoch":703,"log_start":"720'5800","ondisk_log_start":"720'5800","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"750'12096","last_scrub_stamp":"2024-04-23T16:53:42.243776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:53:42.243776+0000","objects_scrubbed":6933,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:54:50.054687+0000","scrub_duration":1286,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206110720,"num_objects":3145,"num_object_clones":0,"num_object_copies":6290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15849,"num_write_kb":607808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1779,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:42.089335+0000","last_change":"2024-04-23T16:54:32.619720+0000","last_active":"2024-04-23T16:54:42.089335+0000","last_peered":"2024-04-23T16:54:42.089335+0000","last_clean":"2024-04-23T16:54:42.089335+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:54:42.089335+0000","last_undegraded":"2024-04-23T16:54:42.089335+0000","last_fullsized":"2024-04-23T16:54:42.089335+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:54:32.619503+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:54:32.619503+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:40.217222+0000","scrub_duration":309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"765'15497","reported_seq":15722,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:46.292621+0000","last_change":"2024-04-23T16:52:43.250377+0000","last_active":"2024-04-23T16:54:46.292621+0000","last_peered":"2024-04-23T16:54:46.292621+0000","last_clean":"2024-04-23T16:54:46.292621+0000","last_became_active":"2024-04-23T16:51:21.064506+0000","last_became_peered":"2024-04-23T16:51:21.064506+0000","last_unstale":"2024-04-23T16:54:46.292621+0000","last_undegraded":"2024-04-23T16:54:46.292621+0000","last_fullsized":"2024-04-23T16:54:46.292621+0000","mapping_epoch":703,"log_start":"720'5400","ondisk_log_start":"720'5400","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"731'8308","last_scrub_stamp":"2024-04-23T16:52:43.250180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:52:43.250180+0000","objects_scrubbed":8271,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201981976,"num_objects":3083,"num_object_clones":0,"num_object_copies":6166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3083,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15497,"num_write_kb":594497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1703,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:42.101765+0000","last_change":"2024-04-23T16:52:47.480919+0000","last_active":"2024-04-23T16:54:42.101765+0000","last_peered":"2024-04-23T16:54:42.101765+0000","last_clean":"2024-04-23T16:54:42.101765+0000","last_became_active":"2024-04-23T16:52:47.480630+0000","last_became_peered":"2024-04-23T16:52:47.480630+0000","last_unstale":"2024-04-23T16:54:42.101765+0000","last_undegraded":"2024-04-23T16:54:42.101765+0000","last_fullsized":"2024-04-23T16:54:42.101765+0000","mapping_epoch":735,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":736,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:52:42.428791+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:52:42.428791+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"765'17794","reported_seq":18155,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:45.064881+0000","last_change":"2024-04-23T16:54:24.434938+0000","last_active":"2024-04-23T16:54:45.064881+0000","last_peered":"2024-04-23T16:54:45.064881+0000","last_clean":"2024-04-23T16:54:45.064881+0000","last_became_active":"2024-04-23T16:51:21.063420+0000","last_became_peered":"2024-04-23T16:51:21.063420+0000","last_unstale":"2024-04-23T16:54:45.064881+0000","last_undegraded":"2024-04-23T16:54:45.064881+0000","last_fullsized":"2024-04-23T16:54:45.064881+0000","mapping_epoch":703,"log_start":"729'7702","ondisk_log_start":"729'7702","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'16466","last_scrub_stamp":"2024-04-23T16:54:24.434705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:24.434705+0000","objects_scrubbed":4533,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:43.605780+0000","scrub_duration":1184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207618048,"num_objects":3168,"num_object_clones":0,"num_object_copies":6336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15778,"num_write_kb":606272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1863,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:42.089982+0000","last_change":"2024-04-23T16:51:31.339866+0000","last_active":"2024-04-23T16:54:42.089982+0000","last_peered":"2024-04-23T16:54:42.089982+0000","last_clean":"2024-04-23T16:54:42.089982+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:54:42.089982+0000","last_undegraded":"2024-04-23T16:54:42.089982+0000","last_fullsized":"2024-04-23T16:54:42.089982+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:51:31.339728+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:51:31.339728+0000","last_clean_scrub_stamp":"2024-04-23T16:51:31.339728+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3384934424,"num_objects":51651,"num_object_clones":0,"num_object_copies":103302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2068,"num_objects_unfound":0,"num_objects_dirty":51651,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":250611,"num_write_kb":9672321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6516,"num_bytes_recovered":427032576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7179542528,"data_stored":7179534384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160925,"ondisk_log_size":160925,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1679360,"data_stored":1661600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057113,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1939060,"kb_used_data":852776,"kb_used_omap":1,"kb_used_meta":1086270,"kb_avail":91806092,"statfs":{"total":95995035648,"available":94009438208,"internally_reserved":0,"allocated":873242624,"data_stored":872855789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1112340932},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351456,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613093,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2036016,"kb_used_data":1202344,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":91709136,"statfs":{"total":95995035648,"available":93910155264,"internally_reserved":0,"allocated":1231200256,"data_stored":1230813421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":853604800},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":66,"seq":283467842025,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1623932,"kb_used_data":1055344,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92121220,"statfs":{"total":95995035648,"available":94332129280,"internally_reserved":0,"allocated":1080672256,"data_stored":1080281869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169072,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1526580,"kb_used_data":787372,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":92218572,"statfs":{"total":95995035648,"available":94431817728,"internally_reserved":0,"allocated":806268928,"data_stored":805878021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":756939210},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463412,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":992560,"kb_used_data":397736,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92752592,"statfs":{"total":95995035648,"available":94978654208,"internally_reserved":0,"allocated":407281664,"data_stored":406894829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":609089990},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137552,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3087400,"kb_used_data":2495712,"kb_used_omap":231,"kb_used_meta":591320,"kb_avail":90657752,"statfs":{"total":95995035648,"available":92833538048,"internally_reserved":0,"allocated":2555609088,"data_stored":2555224185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":605512592},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":961396,"kb_used_data":208296,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":92783756,"statfs":{"total":95995035648,"available":95010566144,"internally_reserved":0,"allocated":213295104,"data_stored":212908269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":771160517},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":481,"seq":2065879269568,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":152828,"kb_used_data":62064,"kb_used_omap":281,"kb_used_meta":90470,"kb_avail":93592324,"statfs":{"total":95995035648,"available":95838539776,"internally_reserved":0,"allocated":63553536,"data_stored":63163157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":288353,"internal_metadata":92641695},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2658584756319,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":3126736191517,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757751,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208863232,"data_stored":208863232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2545422336,"data_stored":2545418264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":63373312,"data_stored":63373312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":400293888,"data_stored":400293888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":804065280,"data_stored":804061208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1066598400,"data_stored":1066598400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1229783040,"data_stored":1229783040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":861143040,"data_stored":861143040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:54:51.139 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:54:51.691 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:54:52.129 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:54:52.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:52.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:54:52.235 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:54:52.275 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:52 smithi103 ceph-mon[39629]: pgmap v2123: 27 pgs: 1 active+recovering, 26 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 2068/94906 objects misplaced (2.179%); 4.2 MiB/s, 67 objects/s recovering 2024-04-23T16:54:52.275 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:52.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:52 smithi148 ceph-mon[44520]: pgmap v2123: 27 pgs: 1 active+recovering, 26 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 2068/94906 objects misplaced (2.179%); 4.2 MiB/s, 67 objects/s recovering 2024-04-23T16:54:52.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:52 smithi188 ceph-mon[44101]: pgmap v2123: 27 pgs: 1 active+recovering, 26 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 2068/94906 objects misplaced (2.179%); 4.2 MiB/s, 67 objects/s recovering 2024-04-23T16:54:52.592 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:54:52.821 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:54:52.821 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:54:52.821 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2024-04-23T16:54:52.821 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2024-04-23T16:54:52.821 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:54:53.083 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:53 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:54:53.083 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:53 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:54:53.084 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:53 smithi103 ceph-mon[39629]: osdmap e766: 12 total, 12 up, 8 in 2024-04-23T16:54:53.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:53 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:54:53.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:53 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:54:53.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:53 smithi148 ceph-mon[44520]: osdmap e766: 12 total, 12 up, 8 in 2024-04-23T16:54:53.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:53 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:54:53.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:53 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:54:53.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:53 smithi188 ceph-mon[44101]: osdmap e766: 12 total, 12 up, 8 in 2024-04-23T16:54:53.754 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:54:53.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:53.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:54:53.861 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:54:53.908 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:53.945 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:54:53.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:53.949 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:54:54.050 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:54:54.085 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:54.191 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:54:54.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:54.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:54:54.218 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-23T16:54:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:54 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:54:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:54 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:54:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:54 smithi188 ceph-mon[44101]: pgmap v2125: 27 pgs: 1 active+recovering, 26 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 1693/88690 objects misplaced (1.909%); 4.5 MiB/s, 72 objects/s recovering 2024-04-23T16:54:54.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:54 smithi188 ceph-mon[44101]: osdmap e767: 12 total, 12 up, 8 in 2024-04-23T16:54:54.298 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:54:54.330 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:54.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:54 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:54:54.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:54 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:54:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:54 smithi103 ceph-mon[39629]: pgmap v2125: 27 pgs: 1 active+recovering, 26 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 1693/88690 objects misplaced (1.909%); 4.5 MiB/s, 72 objects/s recovering 2024-04-23T16:54:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:54 smithi103 ceph-mon[39629]: osdmap e767: 12 total, 12 up, 8 in 2024-04-23T16:54:54.354 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:54:54.358 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:54 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:54:54.358 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:54 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:54:54.358 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:54 smithi148 ceph-mon[44520]: pgmap v2125: 27 pgs: 1 active+recovering, 26 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 1693/88690 objects misplaced (1.909%); 4.5 MiB/s, 72 objects/s recovering 2024-04-23T16:54:54.358 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:54 smithi148 ceph-mon[44520]: osdmap e767: 12 total, 12 up, 8 in 2024-04-23T16:54:54.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:54.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:54:54.461 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:54:54.499 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:54.567 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:54:55.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:55 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:54:55.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:55 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:54:55.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:55 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4147896600' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:54:55.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:55 smithi148 ceph-mon[44520]: from='client.37205 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:54:55.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:55 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:54:55.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:55 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:54:55.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:55 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:54:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:55 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:54:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:55 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:54:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:55 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4147896600' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:54:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:55 smithi188 ceph-mon[44101]: from='client.37205 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:54:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:55 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:54:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:55 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:54:55.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:55 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:54:55.503 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:55 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:54:55.503 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:55 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:54:55.503 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:55 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4147896600' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:54:55.503 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:55 smithi103 ceph-mon[39629]: from='client.37205 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:54:55.503 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:55 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:54:55.503 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:55 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:54:55.504 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:55 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:54:55.528 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:54:55.982 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:54:55.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:55.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:54:56.092 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:54:56.117 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:54:56.117 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:54:56.117 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:54:56.117 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:54:56.117 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:54:56.117 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:54:56.118 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:54:56.118 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:54:56.118 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:54:56.118 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:54:56.118 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:54:56.118 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:54:56.118 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:54:56.118 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:54:56.118 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:54:56.118 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:54:56.118 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:54:56.119 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:54:56.119 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:54:56.119 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:54:56.119 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:54:56.119 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:54:56.119 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:54:56.119 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:54:56.125 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:56.156 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:54:56.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:56.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:54:56.262 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:54:56.280 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:56 smithi148 ceph-mon[44520]: pgmap v2127: 27 pgs: 1 active+recovering, 26 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 1693/86772 objects misplaced (1.951%); 2.9 MiB/s, 46 objects/s recovering 2024-04-23T16:54:56.280 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:56 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2672277338' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:54:56.280 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.280 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.280 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.280 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.280 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.280 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.280 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.280 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.280 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:56 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.280 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:56 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:54:56.280 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:56 smithi148 ceph-mon[44520]: from='client.37205 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:54:56.281 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:56 smithi148 ceph-mon[44520]: osdmap e768: 12 total, 12 up, 8 in 2024-04-23T16:54:56.301 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:56.335 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:54:56.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:56.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:54:56.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:56 smithi103 ceph-mon[39629]: pgmap v2127: 27 pgs: 1 active+recovering, 26 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 1693/86772 objects misplaced (1.951%); 2.9 MiB/s, 46 objects/s recovering 2024-04-23T16:54:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:56 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2672277338' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:54:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:56 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:56 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:54:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:56 smithi103 ceph-mon[39629]: from='client.37205 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:54:56.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:56 smithi103 ceph-mon[39629]: osdmap e768: 12 total, 12 up, 8 in 2024-04-23T16:54:56.438 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:54:56.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:56 smithi188 ceph-mon[44101]: pgmap v2127: 27 pgs: 1 active+recovering, 26 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 1693/86772 objects misplaced (1.951%); 2.9 MiB/s, 46 objects/s recovering 2024-04-23T16:54:56.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:56 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2672277338' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:54:56.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:56 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:56.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:56 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:54:56.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:56 smithi188 ceph-mon[44101]: from='client.37205 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:54:56.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:56 smithi188 ceph-mon[44101]: osdmap e768: 12 total, 12 up, 8 in 2024-04-23T16:54:56.478 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:56.526 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:54:56.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:56.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:54:56.632 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:54:56.683 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:56.723 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:54:56.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:56.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:54:56.828 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:54:56.872 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:57.108 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:57.108 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:57.108 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:57 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:57.108 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:57 smithi188 ceph-mon[44101]: 1.7 scrub starts 2024-04-23T16:54:57.108 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:57 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:54:57.108 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:57 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:54:57.108 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:57 smithi188 ceph-mon[44101]: 1.7 scrub ok 2024-04-23T16:54:57.108 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:57 smithi188 ceph-mon[44101]: 9.c scrub starts 2024-04-23T16:54:57.108 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:57 smithi188 ceph-mon[44101]: osdmap e769: 12 total, 12 up, 8 in 2024-04-23T16:54:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:57 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:57 smithi148 ceph-mon[44520]: 1.7 scrub starts 2024-04-23T16:54:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:57 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:54:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:57 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:54:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:57 smithi148 ceph-mon[44520]: 1.7 scrub ok 2024-04-23T16:54:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:57 smithi148 ceph-mon[44520]: 9.c scrub starts 2024-04-23T16:54:57.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:57 smithi148 ceph-mon[44520]: osdmap e769: 12 total, 12 up, 8 in 2024-04-23T16:54:57.412 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:57.412 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:57.412 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:57 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:54:57.412 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:57 smithi103 ceph-mon[39629]: 1.7 scrub starts 2024-04-23T16:54:57.412 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:57 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:54:57.412 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:57 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:54:57.413 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:57 smithi103 ceph-mon[39629]: 1.7 scrub ok 2024-04-23T16:54:57.413 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:57 smithi103 ceph-mon[39629]: 9.c scrub starts 2024-04-23T16:54:57.413 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:57 smithi103 ceph-mon[39629]: osdmap e769: 12 total, 12 up, 8 in 2024-04-23T16:54:57.507 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2024-04-23T16:54:57.508 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:54:57.734 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:54:57.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:57.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:54:57.839 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:54:57.873 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:58.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:58 smithi148 ceph-mon[44520]: 9.c scrub ok 2024-04-23T16:54:58.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:54:58 smithi148 ceph-mon[44520]: pgmap v2130: 27 pgs: 1 active+recovering, 26 active+clean; 2.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 1300/82884 objects misplaced (1.568%); 5.0 MiB/s, 79 objects/s recovering 2024-04-23T16:54:58.224 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:54:58.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:58.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:54:58.245 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-23T16:54:58.327 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:54:58.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:58 smithi103 ceph-mon[39629]: 9.c scrub ok 2024-04-23T16:54:58.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:54:58 smithi103 ceph-mon[39629]: pgmap v2130: 27 pgs: 1 active+recovering, 26 active+clean; 2.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 1300/82884 objects misplaced (1.568%); 5.0 MiB/s, 79 objects/s recovering 2024-04-23T16:54:58.376 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:58.414 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:54:58.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:58.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:54:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:58 smithi188 ceph-mon[44101]: 9.c scrub ok 2024-04-23T16:54:58.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:54:58 smithi188 ceph-mon[44101]: pgmap v2130: 27 pgs: 1 active+recovering, 26 active+clean; 2.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 1300/82884 objects misplaced (1.568%); 5.0 MiB/s, 79 objects/s recovering 2024-04-23T16:54:58.518 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:54:58.547 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:58.568 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:54:58.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:58.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:54:58.640 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:54:58.671 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:54:58.680 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:54:58.683 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:54:58.698 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:58.718 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:54:58.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:58.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:54:58.825 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:54:58.852 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:58.874 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:54:58.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:58.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:54:58.883 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:54:58.980 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:54:59.013 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:59.035 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:54:59.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:59.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:54:59.139 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:54:59.167 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:59.188 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:54:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:54:59.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:54:59.292 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:54:59.346 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:54:59.590 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":2130,"stamp":"2024-04-23T16:54:57.004182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2716336696,"num_objects":41442,"num_object_clones":0,"num_object_copies":82884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1300,"num_objects_unfound":0,"num_objects_dirty":41442,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":260822,"num_write_kb":9672321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7287,"num_bytes_recovered":477967968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162265,"ondisk_log_size":162265,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11108680,"kb_used_data":5819952,"kb_used_omap":522,"kb_used_meta":5287733,"kb_avail":738852536,"statfs":{"total":767960285184,"available":756584996864,"internally_reserved":0,"allocated":5959630848,"data_stored":5956508780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":534579,"internal_metadata":5414639565},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,4,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-231145472,"num_objects":-3527,"num_object_clones":0,"num_object_copies":-7054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-393,"num_objects_unfound":0,"num_objects_dirty":-3527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3527,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":393,"num_bytes_recovered":25755648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.924571"},"pg_stats":[{"pgid":"9.7","version":"768'16648","reported_seq":31360,"reported_epoch":768,"state":"active+recovering","last_fresh":"2024-04-23T16:54:56.330859+0000","last_change":"2024-04-23T16:52:44.746944+0000","last_active":"2024-04-23T16:54:56.330859+0000","last_peered":"2024-04-23T16:54:56.330859+0000","last_clean":"2024-04-23T16:52:43.617275+0000","last_became_active":"2024-04-23T16:52:44.683405+0000","last_became_peered":"2024-04-23T16:52:44.683405+0000","last_unstale":"2024-04-23T16:54:56.330859+0000","last_undegraded":"2024-04-23T16:54:56.330859+0000","last_fullsized":"2024-04-23T16:54:56.330859+0000","mapping_epoch":732,"log_start":"723'6600","ondisk_log_start":"723'6600","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158728192,"num_objects":2422,"num_object_clones":0,"num_object_copies":4844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1300,"num_objects_unfound":0,"num_objects_dirty":2422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16648,"num_write_kb":610240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7284,"num_bytes_recovered":477364224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1300},{"shards":"2,7","objects":1122}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"765'15866","reported_seq":16085,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:52.010020+0000","last_change":"2024-04-23T16:53:26.127957+0000","last_active":"2024-04-23T16:54:52.010020+0000","last_peered":"2024-04-23T16:54:52.010020+0000","last_clean":"2024-04-23T16:54:52.010020+0000","last_became_active":"2024-04-23T16:51:21.063642+0000","last_became_peered":"2024-04-23T16:51:21.063642+0000","last_unstale":"2024-04-23T16:54:52.010020+0000","last_undegraded":"2024-04-23T16:54:52.010020+0000","last_fullsized":"2024-04-23T16:54:52.010020+0000","mapping_epoch":703,"log_start":"721'5800","ondisk_log_start":"721'5800","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"745'10860","last_scrub_stamp":"2024-04-23T16:53:26.127859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:53:26.127859+0000","objects_scrubbed":7678,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:54:56.818537+0000","scrub_duration":1541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172097536,"num_objects":2626,"num_object_clones":0,"num_object_copies":5252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15866,"num_write_kb":591744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"766'16109","reported_seq":16387,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-04-23T16:54:52.771256+0000","last_change":"2024-04-23T16:54:21.272785+0000","last_active":"2024-04-23T16:54:52.771256+0000","last_peered":"2024-04-23T16:54:52.771256+0000","last_clean":"2024-04-23T16:54:52.771256+0000","last_became_active":"2024-04-23T16:51:21.063776+0000","last_became_peered":"2024-04-23T16:51:21.063776+0000","last_unstale":"2024-04-23T16:54:52.771256+0000","last_undegraded":"2024-04-23T16:54:52.771256+0000","last_fullsized":"2024-04-23T16:54:52.771256+0000","mapping_epoch":703,"log_start":"721'6100","ondisk_log_start":"721'6100","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14080","last_scrub_stamp":"2024-04-23T16:54:21.272505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:21.272505+0000","objects_scrubbed":4691,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:50.192674+0000","scrub_duration":969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172425216,"num_objects":2631,"num_object_clones":0,"num_object_copies":5262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16109,"num_write_kb":599680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"765'16105","reported_seq":16333,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:51.991648+0000","last_change":"2024-04-23T16:52:48.980259+0000","last_active":"2024-04-23T16:54:51.991648+0000","last_peered":"2024-04-23T16:54:51.991648+0000","last_clean":"2024-04-23T16:54:51.991648+0000","last_became_active":"2024-04-23T16:51:21.063114+0000","last_became_peered":"2024-04-23T16:51:21.063114+0000","last_unstale":"2024-04-23T16:54:51.991648+0000","last_undegraded":"2024-04-23T16:54:51.991648+0000","last_fullsized":"2024-04-23T16:54:51.991648+0000","mapping_epoch":703,"log_start":"721'6100","ondisk_log_start":"721'6100","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"736'8752","last_scrub_stamp":"2024-04-23T16:52:48.980138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:52:48.980138+0000","objects_scrubbed":8709,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:54:56.818517+0000","scrub_duration":1449,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178716672,"num_objects":2727,"num_object_clones":0,"num_object_copies":5454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16105,"num_write_kb":602624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"767'16348","reported_seq":16585,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-23T16:54:53.364269+0000","last_change":"2024-04-23T16:53:29.947031+0000","last_active":"2024-04-23T16:54:53.364269+0000","last_peered":"2024-04-23T16:54:53.364269+0000","last_clean":"2024-04-23T16:54:53.364269+0000","last_became_active":"2024-04-23T16:51:21.063652+0000","last_became_peered":"2024-04-23T16:51:21.063652+0000","last_unstale":"2024-04-23T16:54:53.364269+0000","last_undegraded":"2024-04-23T16:54:53.364269+0000","last_fullsized":"2024-04-23T16:54:53.364269+0000","mapping_epoch":703,"log_start":"722'6300","ondisk_log_start":"722'6300","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"746'11271","last_scrub_stamp":"2024-04-23T16:53:29.946887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:53:29.946887+0000","objects_scrubbed":7717,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170262528,"num_objects":2598,"num_object_clones":0,"num_object_copies":5196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16348,"num_write_kb":606272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.2","version":"767'16476","reported_seq":16758,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-23T16:54:53.316286+0000","last_change":"2024-04-23T16:54:24.962245+0000","last_active":"2024-04-23T16:54:53.316286+0000","last_peered":"2024-04-23T16:54:53.316286+0000","last_clean":"2024-04-23T16:54:53.316286+0000","last_became_active":"2024-04-23T16:51:21.062561+0000","last_became_peered":"2024-04-23T16:51:21.062561+0000","last_unstale":"2024-04-23T16:54:53.316286+0000","last_undegraded":"2024-04-23T16:54:53.316286+0000","last_fullsized":"2024-04-23T16:54:53.316286+0000","mapping_epoch":703,"log_start":"722'6400","ondisk_log_start":"722'6400","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14622","last_scrub_stamp":"2024-04-23T16:54:24.962060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:24.962060+0000","objects_scrubbed":4531,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:49.051124+0000","scrub_duration":1162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173015040,"num_objects":2640,"num_object_clones":0,"num_object_copies":5280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16476,"num_write_kb":611712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1891,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-23T16:54:53.085181+0000","last_change":"2024-04-23T16:54:28.362501+0000","last_active":"2024-04-23T16:54:53.085181+0000","last_peered":"2024-04-23T16:54:53.085181+0000","last_clean":"2024-04-23T16:54:53.085181+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:54:53.085181+0000","last_undegraded":"2024-04-23T16:54:53.085181+0000","last_fullsized":"2024-04-23T16:54:53.085181+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:54:28.362406+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:54:28.362406+0000","last_clean_scrub_stamp":"2024-04-23T16:54:28.362406+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:28.652451+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"765'16388","reported_seq":16669,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:51.976989+0000","last_change":"2024-04-23T16:54:31.665323+0000","last_active":"2024-04-23T16:54:51.976989+0000","last_peered":"2024-04-23T16:54:51.976989+0000","last_clean":"2024-04-23T16:54:51.976989+0000","last_became_active":"2024-04-23T16:51:21.063395+0000","last_became_peered":"2024-04-23T16:51:21.063395+0000","last_unstale":"2024-04-23T16:54:51.976989+0000","last_undegraded":"2024-04-23T16:54:51.976989+0000","last_fullsized":"2024-04-23T16:54:51.976989+0000","mapping_epoch":703,"log_start":"722'6300","ondisk_log_start":"722'6300","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"763'15015","last_scrub_stamp":"2024-04-23T16:54:31.665200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:31.665200+0000","objects_scrubbed":4086,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:39.674179+0000","scrub_duration":876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176291840,"num_objects":2690,"num_object_clones":0,"num_object_copies":5380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16388,"num_write_kb":610496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1664,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-23T16:54:53.094098+0000","last_change":"2024-04-23T16:52:47.483911+0000","last_active":"2024-04-23T16:54:53.094098+0000","last_peered":"2024-04-23T16:54:53.094098+0000","last_clean":"2024-04-23T16:54:53.094098+0000","last_became_active":"2024-04-23T16:52:47.483615+0000","last_became_peered":"2024-04-23T16:52:47.483615+0000","last_unstale":"2024-04-23T16:54:53.094098+0000","last_undegraded":"2024-04-23T16:54:53.094098+0000","last_fullsized":"2024-04-23T16:54:53.094098+0000","mapping_epoch":735,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":736,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:52:05.504325+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:52:05.504325+0000","last_clean_scrub_stamp":"2024-04-23T16:52:05.504325+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"767'15966","reported_seq":16244,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-23T16:54:55.096259+0000","last_change":"2024-04-23T16:54:17.136201+0000","last_active":"2024-04-23T16:54:55.096259+0000","last_peered":"2024-04-23T16:54:55.096259+0000","last_clean":"2024-04-23T16:54:55.096259+0000","last_became_active":"2024-04-23T16:51:21.246391+0000","last_became_peered":"2024-04-23T16:51:21.246391+0000","last_unstale":"2024-04-23T16:54:55.096259+0000","last_undegraded":"2024-04-23T16:54:55.096259+0000","last_fullsized":"2024-04-23T16:54:55.096259+0000","mapping_epoch":703,"log_start":"721'5900","ondisk_log_start":"721'5900","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'13668","last_scrub_stamp":"2024-04-23T16:54:17.136038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:17.136038+0000","objects_scrubbed":4818,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:23.005433+0000","scrub_duration":963,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163053568,"num_objects":2488,"num_object_clones":0,"num_object_copies":4976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15966,"num_write_kb":590528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1877,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-04-23T16:54:55.528549+0000","last_change":"2024-04-23T16:52:38.199979+0000","last_active":"2024-04-23T16:54:55.528549+0000","last_peered":"2024-04-23T16:54:55.528549+0000","last_clean":"2024-04-23T16:54:55.528549+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:54:55.528549+0000","last_undegraded":"2024-04-23T16:54:55.528549+0000","last_fullsized":"2024-04-23T16:54:55.528549+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:52:38.199895+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:51:25.290774+0000","last_clean_scrub_stamp":"2024-04-23T16:52:38.199895+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"765'16311","reported_seq":16592,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:51.985104+0000","last_change":"2024-04-23T16:54:18.910760+0000","last_active":"2024-04-23T16:54:51.985104+0000","last_peered":"2024-04-23T16:54:51.985104+0000","last_clean":"2024-04-23T16:54:51.985104+0000","last_became_active":"2024-04-23T16:51:21.063669+0000","last_became_peered":"2024-04-23T16:51:21.063669+0000","last_unstale":"2024-04-23T16:54:51.985104+0000","last_undegraded":"2024-04-23T16:54:51.985104+0000","last_fullsized":"2024-04-23T16:54:51.985104+0000","mapping_epoch":703,"log_start":"722'6300","ondisk_log_start":"722'6300","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14143","last_scrub_stamp":"2024-04-23T16:54:18.910481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:18.910481+0000","objects_scrubbed":4947,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:29.963181+0000","scrub_duration":1208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179896320,"num_objects":2745,"num_object_clones":0,"num_object_copies":5490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16311,"num_write_kb":609792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1889,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-23T16:54:53.092714+0000","last_change":"2024-04-23T16:53:25.567832+0000","last_active":"2024-04-23T16:54:53.092714+0000","last_peered":"2024-04-23T16:54:53.092714+0000","last_clean":"2024-04-23T16:54:53.092714+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:54:53.092714+0000","last_undegraded":"2024-04-23T16:54:53.092714+0000","last_fullsized":"2024-04-23T16:54:53.092714+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:53:25.567686+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:53:25.567686+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:54:55.349609+0000","scrub_duration":275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"767'16511","reported_seq":16793,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-23T16:54:53.339142+0000","last_change":"2024-04-23T16:54:19.829779+0000","last_active":"2024-04-23T16:54:53.339142+0000","last_peered":"2024-04-23T16:54:53.339142+0000","last_clean":"2024-04-23T16:54:53.339142+0000","last_became_active":"2024-04-23T16:51:21.062760+0000","last_became_peered":"2024-04-23T16:51:21.062760+0000","last_unstale":"2024-04-23T16:54:53.339142+0000","last_undegraded":"2024-04-23T16:54:53.339142+0000","last_fullsized":"2024-04-23T16:54:53.339142+0000","mapping_epoch":703,"log_start":"722'6500","ondisk_log_start":"722'6500","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14356","last_scrub_stamp":"2024-04-23T16:54:19.829518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:19.829518+0000","objects_scrubbed":4782,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:22.348945+0000","scrub_duration":1029,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170590208,"num_objects":2603,"num_object_clones":0,"num_object_copies":5206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16511,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1469,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:42.090802+0000","last_change":"2024-04-23T16:53:25.755305+0000","last_active":"2024-04-23T16:54:42.090802+0000","last_peered":"2024-04-23T16:54:42.090802+0000","last_clean":"2024-04-23T16:54:42.090802+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:54:42.090802+0000","last_undegraded":"2024-04-23T16:54:42.090802+0000","last_fullsized":"2024-04-23T16:54:42.090802+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:53:25.755173+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:53:25.755173+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:54:48.864398+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"766'16165","reported_seq":16447,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-04-23T16:54:52.802940+0000","last_change":"2024-04-23T16:54:19.146832+0000","last_active":"2024-04-23T16:54:52.802940+0000","last_peered":"2024-04-23T16:54:52.802940+0000","last_clean":"2024-04-23T16:54:52.802940+0000","last_became_active":"2024-04-23T16:51:21.246519+0000","last_became_peered":"2024-04-23T16:51:21.246519+0000","last_unstale":"2024-04-23T16:54:52.802940+0000","last_undegraded":"2024-04-23T16:54:52.802940+0000","last_fullsized":"2024-04-23T16:54:52.802940+0000","mapping_epoch":703,"log_start":"721'6100","ondisk_log_start":"721'6100","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'13985","last_scrub_stamp":"2024-04-23T16:54:19.146171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:19.146171+0000","objects_scrubbed":4796,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:28.380772+0000","scrub_duration":863,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169672704,"num_objects":2589,"num_object_clones":0,"num_object_copies":5178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16165,"num_write_kb":600128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1434,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-23T16:54:53.093467+0000","last_change":"2024-04-23T16:53:42.723708+0000","last_active":"2024-04-23T16:54:53.093467+0000","last_peered":"2024-04-23T16:54:53.093467+0000","last_clean":"2024-04-23T16:54:53.093467+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:54:53.093467+0000","last_undegraded":"2024-04-23T16:54:53.093467+0000","last_fullsized":"2024-04-23T16:54:53.093467+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:53:42.723618+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:53:42.723618+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:54:56.875882+0000","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.e","version":"768'16487","reported_seq":16807,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-04-23T16:54:56.335795+0000","last_change":"2024-04-23T16:52:41.066866+0000","last_active":"2024-04-23T16:54:56.335795+0000","last_peered":"2024-04-23T16:54:56.335795+0000","last_clean":"2024-04-23T16:54:56.335795+0000","last_became_active":"2024-04-23T16:51:21.064486+0000","last_became_peered":"2024-04-23T16:51:21.064486+0000","last_unstale":"2024-04-23T16:54:56.335795+0000","last_undegraded":"2024-04-23T16:54:56.335795+0000","last_fullsized":"2024-04-23T16:54:56.335795+0000","mapping_epoch":703,"log_start":"722'6400","ondisk_log_start":"722'6400","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"730'8312","last_scrub_stamp":"2024-04-23T16:52:41.066729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:52:41.066729+0000","objects_scrubbed":8276,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163250176,"num_objects":2491,"num_object_clones":0,"num_object_copies":4982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16487,"num_write_kb":607296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1899,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-23T16:54:53.084416+0000","last_change":"2024-04-23T16:54:26.409871+0000","last_active":"2024-04-23T16:54:53.084416+0000","last_peered":"2024-04-23T16:54:53.084416+0000","last_clean":"2024-04-23T16:54:53.084416+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:54:53.084416+0000","last_undegraded":"2024-04-23T16:54:53.084416+0000","last_fullsized":"2024-04-23T16:54:53.084416+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:54:26.409705+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:54:26.409705+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:33.984886+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.f","version":"765'16364","reported_seq":16646,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-23T16:54:51.992103+0000","last_change":"2024-04-23T16:54:17.546594+0000","last_active":"2024-04-23T16:54:51.992103+0000","last_peered":"2024-04-23T16:54:51.992103+0000","last_clean":"2024-04-23T16:54:51.992103+0000","last_became_active":"2024-04-23T16:51:21.063588+0000","last_became_peered":"2024-04-23T16:51:21.063588+0000","last_unstale":"2024-04-23T16:54:51.992103+0000","last_undegraded":"2024-04-23T16:54:51.992103+0000","last_fullsized":"2024-04-23T16:54:51.992103+0000","mapping_epoch":703,"log_start":"722'6300","ondisk_log_start":"722'6300","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14153","last_scrub_stamp":"2024-04-23T16:54:17.546419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:17.546419+0000","objects_scrubbed":4978,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:23.570053+0000","scrub_duration":812,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180092928,"num_objects":2748,"num_object_clones":0,"num_object_copies":5496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16364,"num_write_kb":611584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1902,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-23T16:54:53.088931+0000","last_change":"2024-04-23T16:53:28.261207+0000","last_active":"2024-04-23T16:54:53.088931+0000","last_peered":"2024-04-23T16:54:53.088931+0000","last_clean":"2024-04-23T16:54:53.088931+0000","last_became_active":"2024-04-23T16:52:44.474580+0000","last_became_peered":"2024-04-23T16:52:44.474580+0000","last_unstale":"2024-04-23T16:54:53.088931+0000","last_undegraded":"2024-04-23T16:54:53.088931+0000","last_fullsized":"2024-04-23T16:54:53.088931+0000","mapping_epoch":732,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":733,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:53:28.261070+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:53:28.261070+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:54:55.760687+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"767'16526","reported_seq":16833,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-23T16:54:55.099812+0000","last_change":"2024-04-23T16:53:42.243957+0000","last_active":"2024-04-23T16:54:55.099812+0000","last_peered":"2024-04-23T16:54:55.099812+0000","last_clean":"2024-04-23T16:54:55.099812+0000","last_became_active":"2024-04-23T16:51:21.079798+0000","last_became_peered":"2024-04-23T16:51:21.079798+0000","last_unstale":"2024-04-23T16:54:55.099812+0000","last_undegraded":"2024-04-23T16:54:55.099812+0000","last_fullsized":"2024-04-23T16:54:55.099812+0000","mapping_epoch":703,"log_start":"722'6500","ondisk_log_start":"722'6500","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"750'12096","last_scrub_stamp":"2024-04-23T16:53:42.243776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:53:42.243776+0000","objects_scrubbed":6933,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:54:55.149842+0000","scrub_duration":1286,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161742848,"num_objects":2468,"num_object_clones":0,"num_object_copies":4936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16526,"num_write_kb":607808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1781,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-04-23T16:54:52.079497+0000","last_change":"2024-04-23T16:54:32.619720+0000","last_active":"2024-04-23T16:54:52.079497+0000","last_peered":"2024-04-23T16:54:52.079497+0000","last_clean":"2024-04-23T16:54:52.079497+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:54:52.079497+0000","last_undegraded":"2024-04-23T16:54:52.079497+0000","last_fullsized":"2024-04-23T16:54:52.079497+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:54:32.619503+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:54:32.619503+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:40.217222+0000","scrub_duration":309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"768'16126","reported_seq":16357,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-04-23T16:54:56.293621+0000","last_change":"2024-04-23T16:52:43.250377+0000","last_active":"2024-04-23T16:54:56.293621+0000","last_peered":"2024-04-23T16:54:56.293621+0000","last_clean":"2024-04-23T16:54:56.293621+0000","last_became_active":"2024-04-23T16:51:21.064506+0000","last_became_peered":"2024-04-23T16:51:21.064506+0000","last_unstale":"2024-04-23T16:54:56.293621+0000","last_undegraded":"2024-04-23T16:54:56.293621+0000","last_fullsized":"2024-04-23T16:54:56.293621+0000","mapping_epoch":703,"log_start":"722'6100","ondisk_log_start":"722'6100","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"731'8308","last_scrub_stamp":"2024-04-23T16:52:43.250180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:52:43.250180+0000","objects_scrubbed":8271,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160759832,"num_objects":2454,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16126,"num_write_kb":594497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1708,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-23T16:54:53.095610+0000","last_change":"2024-04-23T16:52:47.480919+0000","last_active":"2024-04-23T16:54:53.095610+0000","last_peered":"2024-04-23T16:54:53.095610+0000","last_clean":"2024-04-23T16:54:53.095610+0000","last_became_active":"2024-04-23T16:52:47.480630+0000","last_became_peered":"2024-04-23T16:52:47.480630+0000","last_unstale":"2024-04-23T16:54:53.095610+0000","last_undegraded":"2024-04-23T16:54:53.095610+0000","last_fullsized":"2024-04-23T16:54:53.095610+0000","mapping_epoch":735,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":736,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:52:42.428791+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:52:42.428791+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"767'18442","reported_seq":18807,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-23T16:54:55.103198+0000","last_change":"2024-04-23T16:54:24.434938+0000","last_active":"2024-04-23T16:54:55.103198+0000","last_peered":"2024-04-23T16:54:55.103198+0000","last_clean":"2024-04-23T16:54:55.103198+0000","last_became_active":"2024-04-23T16:51:21.063420+0000","last_became_peered":"2024-04-23T16:51:21.063420+0000","last_unstale":"2024-04-23T16:54:55.103198+0000","last_undegraded":"2024-04-23T16:54:55.103198+0000","last_fullsized":"2024-04-23T16:54:55.103198+0000","mapping_epoch":703,"log_start":"730'8402","ondisk_log_start":"730'8402","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'16466","last_scrub_stamp":"2024-04-23T16:54:24.434705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:24.434705+0000","objects_scrubbed":4533,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:43.605780+0000","scrub_duration":1184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165150720,"num_objects":2520,"num_object_clones":0,"num_object_copies":5040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16426,"num_write_kb":606272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1869,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-04-23T16:54:55.528541+0000","last_change":"2024-04-23T16:51:31.339866+0000","last_active":"2024-04-23T16:54:55.528541+0000","last_peered":"2024-04-23T16:54:55.528541+0000","last_clean":"2024-04-23T16:54:55.528541+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:54:55.528541+0000","last_undegraded":"2024-04-23T16:54:55.528541+0000","last_fullsized":"2024-04-23T16:54:55.528541+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:51:31.339728+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:51:31.339728+0000","last_clean_scrub_stamp":"2024-04-23T16:51:31.339728+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2715746328,"num_objects":41440,"num_object_clones":0,"num_object_copies":82880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1300,"num_objects_unfound":0,"num_objects_dirty":41440,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":260822,"num_write_kb":9672321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7284,"num_bytes_recovered":477364224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5889400832,"data_stored":5889392688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1679360,"data_stored":1661600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057115,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1759936,"kb_used_data":681972,"kb_used_omap":1,"kb_used_meta":1077950,"kb_avail":91985216,"statfs":{"total":95995035648,"available":94192861184,"internally_reserved":0,"allocated":698339328,"data_stored":697946489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1103821252},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351458,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613095,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1808016,"kb_used_data":978368,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":91937136,"statfs":{"total":95995035648,"available":94143627264,"internally_reserved":0,"allocated":1001848832,"data_stored":1001449989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":849344960},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":66,"seq":283467842027,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1433864,"kb_used_data":852732,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92311288,"statfs":{"total":95995035648,"available":94526758912,"internally_reserved":0,"allocated":873197568,"data_stored":872801177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169074,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1363680,"kb_used_data":628632,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":92381472,"statfs":{"total":95995035648,"available":94598627328,"internally_reserved":0,"allocated":643719168,"data_stored":643370149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":752679370},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463414,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":918216,"kb_used_data":319232,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92826936,"statfs":{"total":95995035648,"available":95054782464,"internally_reserved":0,"allocated":326893568,"data_stored":326494725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":613349830},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137554,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2737716,"kb_used_data":2123948,"kb_used_omap":231,"kb_used_meta":613528,"kb_avail":91007436,"statfs":{"total":95995035648,"available":93191614464,"internally_reserved":0,"allocated":2174922752,"data_stored":2174531845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":628253584},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738900,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":920140,"kb_used_data":167040,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":92825012,"statfs":{"total":95995035648,"available":95052812288,"internally_reserved":0,"allocated":171048960,"data_stored":170650117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":771160517},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":481,"seq":2065879269570,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":167112,"kb_used_data":68028,"kb_used_omap":281,"kb_used_meta":98790,"kb_avail":93578040,"statfs":{"total":95995035648,"available":95823912960,"internally_reserved":0,"allocated":69660672,"data_stored":69264289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":288353,"internal_metadata":101161375},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2658584756321,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":3126736191519,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757753,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":165085184,"data_stored":165085184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2164985856,"data_stored":2164981784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69206016,"data_stored":69206016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":319422464,"data_stored":319422464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":642060288,"data_stored":642056216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":860291072,"data_stored":860291072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":979566592,"data_stored":979566592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":688783360,"data_stored":688783360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:54:59.592 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:54:59.693 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:54:59.766 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:55:00.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:00 smithi188 ceph-mon[44101]: from='client.17757 v1:172.21.15.103:0/530981213' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:55:00.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:00 smithi188 ceph-mon[44101]: pgmap v2131: 27 pgs: 1 active+recovering, 26 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1300/76536 objects misplaced (1.699%); 4.1 MiB/s, 65 objects/s recovering 2024-04-23T16:55:00.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:00 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:55:00.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:00 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:55:00.210 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:55:00.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:00.214 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:55:00.315 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:55:00.360 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:00.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:00 smithi148 ceph-mon[44520]: from='client.17757 v1:172.21.15.103:0/530981213' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:55:00.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:00 smithi148 ceph-mon[44520]: pgmap v2131: 27 pgs: 1 active+recovering, 26 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1300/76536 objects misplaced (1.699%); 4.1 MiB/s, 65 objects/s recovering 2024-04-23T16:55:00.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:00 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:55:00.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:00 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:55:00.580 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:55:00.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:00.585 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:55:00.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:00 smithi103 ceph-mon[39629]: from='client.17757 v1:172.21.15.103:0/530981213' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:55:00.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:00 smithi103 ceph-mon[39629]: pgmap v2131: 27 pgs: 1 active+recovering, 26 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1300/76536 objects misplaced (1.699%); 4.1 MiB/s, 65 objects/s recovering 2024-04-23T16:55:00.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:00 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:55:00.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:00 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:55:00.686 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:55:00.713 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:00.735 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:55:00.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:00.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:55:00.839 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:55:00.866 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:00.888 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:55:00.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:00.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:55:00.992 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:55:01.019 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:01.038 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:55:01.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:01.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:55:01.142 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:55:01.173 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:01.194 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:55:01.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:01.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:55:01.254 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:55:01.254 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:55:01.298 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:55:01.362 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:01.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:01 smithi148 ceph-mon[44520]: 9.4 scrub starts 2024-04-23T16:55:01.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:01 smithi148 ceph-mon[44520]: osdmap e770: 12 total, 12 up, 8 in 2024-04-23T16:55:01.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:01 smithi148 ceph-mon[44520]: 9.4 scrub ok 2024-04-23T16:55:01.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:01 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:55:01.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:01 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:55:01.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:01 smithi188 ceph-mon[44101]: 9.4 scrub starts 2024-04-23T16:55:01.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:01 smithi188 ceph-mon[44101]: osdmap e770: 12 total, 12 up, 8 in 2024-04-23T16:55:01.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:01 smithi188 ceph-mon[44101]: 9.4 scrub ok 2024-04-23T16:55:01.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:01 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:55:01.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:01 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:55:01.507 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:55:01.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:01.512 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:55:01.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:01 smithi103 ceph-mon[39629]: 9.4 scrub starts 2024-04-23T16:55:01.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:01 smithi103 ceph-mon[39629]: osdmap e770: 12 total, 12 up, 8 in 2024-04-23T16:55:01.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:01 smithi103 ceph-mon[39629]: 9.4 scrub ok 2024-04-23T16:55:01.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:01 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:55:01.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:01 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:55:01.612 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:55:01.658 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:01.681 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:55:01.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:01.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:55:01.734 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:01.738 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:01.785 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:55:01.843 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:02.029 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-23T16:55:02.453 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:55:02.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:02 smithi188 ceph-mon[44101]: 9.6 scrub starts 2024-04-23T16:55:02.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:02 smithi188 ceph-mon[44101]: pgmap v2133: 27 pgs: 1 active+recovering, 26 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1300/74590 objects misplaced (1.743%); 4.1 MiB/s, 65 objects/s recovering 2024-04-23T16:55:02.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:02 smithi188 ceph-mon[44101]: 9.3 scrub starts 2024-04-23T16:55:02.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:02 smithi188 ceph-mon[44101]: 9.6 scrub ok 2024-04-23T16:55:02.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:02 smithi103 ceph-mon[39629]: 9.6 scrub starts 2024-04-23T16:55:02.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:02 smithi103 ceph-mon[39629]: pgmap v2133: 27 pgs: 1 active+recovering, 26 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1300/74590 objects misplaced (1.743%); 4.1 MiB/s, 65 objects/s recovering 2024-04-23T16:55:02.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:02 smithi103 ceph-mon[39629]: 9.3 scrub starts 2024-04-23T16:55:02.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:02 smithi103 ceph-mon[39629]: 9.6 scrub ok 2024-04-23T16:55:02.608 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:55:02.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:02.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:55:02.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:02 smithi148 ceph-mon[44520]: 9.6 scrub starts 2024-04-23T16:55:02.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:02 smithi148 ceph-mon[44520]: pgmap v2133: 27 pgs: 1 active+recovering, 26 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1300/74590 objects misplaced (1.743%); 4.1 MiB/s, 65 objects/s recovering 2024-04-23T16:55:02.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:02 smithi148 ceph-mon[44520]: 9.3 scrub starts 2024-04-23T16:55:02.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:02 smithi148 ceph-mon[44520]: 9.6 scrub ok 2024-04-23T16:55:02.713 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:55:02.742 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:55:02.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:02.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:55:02.901 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:55:02.944 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:03.151 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:55:03.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:03.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:55:03.193 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:03 smithi103 ceph-mon[39629]: 9.3 scrub ok 2024-04-23T16:55:03.258 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:55:03.286 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:03.306 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:55:03.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:03.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:55:03.414 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:55:03.451 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:03 smithi188 ceph-mon[44101]: 9.3 scrub ok 2024-04-23T16:55:03.454 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:03.496 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:55:03.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:03.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:55:03.600 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:55:03.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:03 smithi148 ceph-mon[44520]: 9.3 scrub ok 2024-04-23T16:55:03.648 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:04.019 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:55:04.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:04.026 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:55:04.127 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:55:04.172 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:04.288 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:04 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:55:04.289 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:04 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:55:04.289 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:04 smithi103 ceph-mon[39629]: pgmap v2134: 27 pgs: 1 active+recovering, 26 active+clean; 2.1 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 828/68504 objects misplaced (1.209%); 7.2 MiB/s, 115 objects/s recovering 2024-04-23T16:55:04.289 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:04 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:55:04.379 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:55:04.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:04.384 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:55:04.384 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:04 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:55:04.384 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:04 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:55:04.385 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:04 smithi188 ceph-mon[44101]: pgmap v2134: 27 pgs: 1 active+recovering, 26 active+clean; 2.1 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 828/68504 objects misplaced (1.209%); 7.2 MiB/s, 115 objects/s recovering 2024-04-23T16:55:04.385 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:04 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:55:04.485 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:55:04.542 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:04.574 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:55:04.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:04.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:55:04.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:04 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:55:04.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:04 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:55:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:04 smithi148 ceph-mon[44520]: pgmap v2134: 27 pgs: 1 active+recovering, 26 active+clean; 2.1 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 828/68504 objects misplaced (1.209%); 7.2 MiB/s, 115 objects/s recovering 2024-04-23T16:55:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:04 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:55:04.680 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:55:04.736 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:04.776 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:55:04.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:04.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:55:04.884 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:55:04.889 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:55:04.889 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:55:04.932 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:05.150 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:55:05.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:05.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:55:05.255 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:55:05.285 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:05.306 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:55:05.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:05.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:55:05.410 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:55:05.454 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:05 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:55:05.458 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:05.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:05 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:55:05.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:05 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:55:05.676 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:55:05.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:05.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:55:05.712 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-23T16:55:05.781 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:55:05.809 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:05.830 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:55:05.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:05.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:55:05.933 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:55:05.960 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:05.980 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:55:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:05.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:55:06.085 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:55:06.114 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:06.129 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:55:06.138 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:55:06.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:06.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:55:06.242 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:55:06.285 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:06.318 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:55:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:06.321 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:55:06.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:06 smithi148 ceph-mon[44520]: pgmap v2135: 27 pgs: 1 active+recovering, 26 active+clean; 2.0 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 994 op/s; 828/66608 objects misplaced (1.243%); 3.6 MiB/s, 57 objects/s recovering 2024-04-23T16:55:06.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:06 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1865651365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:55:06.422 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:55:06.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:06 smithi188 ceph-mon[44101]: pgmap v2135: 27 pgs: 1 active+recovering, 26 active+clean; 2.0 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 994 op/s; 828/66608 objects misplaced (1.243%); 3.6 MiB/s, 57 objects/s recovering 2024-04-23T16:55:06.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:06 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1865651365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:55:06.459 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:06 smithi103 ceph-mon[39629]: pgmap v2135: 27 pgs: 1 active+recovering, 26 active+clean; 2.0 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 994 op/s; 828/66608 objects misplaced (1.243%); 3.6 MiB/s, 57 objects/s recovering 2024-04-23T16:55:06.459 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:06 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1865651365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:55:06.463 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:06.682 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:55:06.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:06.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:55:06.792 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:55:06.849 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:55:06.849 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:55:06.850 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:55:06.850 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:55:06.850 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:55:06.850 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:55:06.850 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:55:06.850 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:55:06.850 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:55:06.850 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:55:06.850 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:55:06.850 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:55:06.850 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:55:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:55:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:55:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:55:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:55:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:55:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:55:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:55:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:55:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:55:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:55:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:55:06.858 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:07.190 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:55:07.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:07 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1824250587' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:55:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:07 smithi188 ceph-mon[44101]: from='client.37226 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:55:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:07 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.546 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.546 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.547 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.547 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.547 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.547 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.547 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.547 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.547 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.547 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:07 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1824250587' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:55:07.547 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:07 smithi103 ceph-mon[39629]: from='client.37226 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:55:07.547 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.547 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.547 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:07 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:07 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1824250587' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:55:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:07 smithi148 ceph-mon[44520]: from='client.37226 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:55:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:07.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:07 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:08.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:08 smithi188 ceph-mon[44101]: pgmap v2136: 27 pgs: 1 active+recovering, 26 active+clean; 1.9 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 996 op/s; 269/62954 objects misplaced (0.427%); 6.4 MiB/s, 103 objects/s recovering 2024-04-23T16:55:08.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:08 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:55:08.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:08 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:55:08.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:08 smithi188 ceph-mon[44101]: from='client.37226 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:55:08.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:08 smithi188 ceph-mon[44101]: osdmap e771: 12 total, 12 up, 8 in 2024-04-23T16:55:08.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:08 smithi103 ceph-mon[39629]: pgmap v2136: 27 pgs: 1 active+recovering, 26 active+clean; 1.9 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 996 op/s; 269/62954 objects misplaced (0.427%); 6.4 MiB/s, 103 objects/s recovering 2024-04-23T16:55:08.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:08 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:55:08.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:08 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:55:08.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:08 smithi103 ceph-mon[39629]: from='client.37226 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:55:08.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:08 smithi103 ceph-mon[39629]: osdmap e771: 12 total, 12 up, 8 in 2024-04-23T16:55:08.605 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:55:08.613 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:55:08.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:08 smithi148 ceph-mon[44520]: pgmap v2136: 27 pgs: 1 active+recovering, 26 active+clean; 1.9 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 996 op/s; 269/62954 objects misplaced (0.427%); 6.4 MiB/s, 103 objects/s recovering 2024-04-23T16:55:08.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:08 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:55:08.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:08 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:55:08.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:08 smithi148 ceph-mon[44520]: from='client.37226 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:55:08.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:08 smithi148 ceph-mon[44520]: osdmap e771: 12 total, 12 up, 8 in 2024-04-23T16:55:09.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:09 smithi103 ceph-mon[39629]: from='client.37235 v1:172.21.15.103:0/1949368576' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:55:09.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:09 smithi103 ceph-mon[39629]: pgmap v2138: 27 pgs: 1 active+recovering, 26 active+clean; 1.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 269/56526 objects misplaced (0.476%); 7.3 MiB/s, 116 objects/s recovering 2024-04-23T16:55:09.461 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":2137,"stamp":"2024-04-23T16:55:07.184488+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1956708920,"num_objects":29851,"num_object_clones":0,"num_object_copies":59702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":269,"num_objects_unfound":0,"num_objects_dirty":29851,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272413,"num_write_kb":9672321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8318,"num_bytes_recovered":545535584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162456,"ondisk_log_size":162456,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9706872,"kb_used_data":4408760,"kb_used_omap":522,"kb_used_meta":5297461,"kb_avail":740254344,"statfs":{"total":767960285184,"available":758020448256,"internally_reserved":0,"allocated":4514570240,"data_stored":4511699360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":534579,"internal_metadata":5424601037},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-551616512,"num_objects":-8417,"num_object_clones":0,"num_object_copies":-16834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1031,"num_objects_unfound":0,"num_objects_dirty":-8417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8417,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1031,"num_bytes_recovered":67567616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.179968"},"pg_stats":[{"pgid":"9.7","version":"770'17296","reported_seq":34073,"reported_epoch":770,"state":"active+recovering","last_fresh":"2024-04-23T16:55:06.330224+0000","last_change":"2024-04-23T16:52:44.746944+0000","last_active":"2024-04-23T16:55:06.330224+0000","last_peered":"2024-04-23T16:55:06.330224+0000","last_clean":"2024-04-23T16:52:43.617275+0000","last_became_active":"2024-04-23T16:52:44.683405+0000","last_became_peered":"2024-04-23T16:52:44.683405+0000","last_unstale":"2024-04-23T16:55:06.330224+0000","last_undegraded":"2024-04-23T16:55:06.330224+0000","last_fullsized":"2024-04-23T16:55:06.330224+0000","mapping_epoch":732,"log_start":"725'7200","ondisk_log_start":"725'7200","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:51:20.043323+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116260864,"num_objects":1774,"num_object_clones":0,"num_object_copies":3548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":269,"num_objects_unfound":0,"num_objects_dirty":1774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17296,"num_write_kb":610240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8315,"num_bytes_recovered":544931840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":269},{"shards":"2,7","objects":1505}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"770'16778","reported_seq":17036,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:07.033691+0000","last_change":"2024-04-23T16:55:01.151246+0000","last_active":"2024-04-23T16:55:07.033691+0000","last_peered":"2024-04-23T16:55:07.033691+0000","last_clean":"2024-04-23T16:55:07.033691+0000","last_became_active":"2024-04-23T16:51:21.063642+0000","last_became_peered":"2024-04-23T16:51:21.063642+0000","last_unstale":"2024-04-23T16:55:07.033691+0000","last_undegraded":"2024-04-23T16:55:07.033691+0000","last_fullsized":"2024-04-23T16:55:07.033691+0000","mapping_epoch":703,"log_start":"725'6700","ondisk_log_start":"725'6700","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'16423","last_scrub_stamp":"2024-04-23T16:55:01.151049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:55:01.151049+0000","objects_scrubbed":2082,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:02.623292+0000","scrub_duration":380,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112328704,"num_objects":1714,"num_object_clones":0,"num_object_copies":3428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16778,"num_write_kb":591744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"770'16700","reported_seq":16986,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:02.714885+0000","last_change":"2024-04-23T16:54:21.272785+0000","last_active":"2024-04-23T16:55:02.714885+0000","last_peered":"2024-04-23T16:55:02.714885+0000","last_clean":"2024-04-23T16:55:02.714885+0000","last_became_active":"2024-04-23T16:51:21.063776+0000","last_became_peered":"2024-04-23T16:51:21.063776+0000","last_unstale":"2024-04-23T16:55:02.714885+0000","last_undegraded":"2024-04-23T16:55:02.714885+0000","last_fullsized":"2024-04-23T16:55:02.714885+0000","mapping_epoch":703,"log_start":"724'6600","ondisk_log_start":"724'6600","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14080","last_scrub_stamp":"2024-04-23T16:54:21.272505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:21.272505+0000","objects_scrubbed":4691,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:50.192674+0000","scrub_duration":969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133693440,"num_objects":2040,"num_object_clones":0,"num_object_copies":4080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16700,"num_write_kb":599680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"770'17059","reported_seq":17328,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:07.033477+0000","last_change":"2024-04-23T16:55:00.296822+0000","last_active":"2024-04-23T16:55:07.033477+0000","last_peered":"2024-04-23T16:55:07.033477+0000","last_clean":"2024-04-23T16:55:07.033477+0000","last_became_active":"2024-04-23T16:51:21.063114+0000","last_became_peered":"2024-04-23T16:51:21.063114+0000","last_unstale":"2024-04-23T16:55:07.033477+0000","last_undegraded":"2024-04-23T16:55:07.033477+0000","last_fullsized":"2024-04-23T16:55:07.033477+0000","mapping_epoch":703,"log_start":"725'7000","ondisk_log_start":"725'7000","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'16616","last_scrub_stamp":"2024-04-23T16:55:00.296670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:55:00.296670+0000","objects_scrubbed":2221,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:18.149351+0000","scrub_duration":479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116195328,"num_objects":1773,"num_object_clones":0,"num_object_copies":3546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17059,"num_write_kb":602624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"770'16943","reported_seq":17215,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:03.341236+0000","last_change":"2024-04-23T16:55:01.549355+0000","last_active":"2024-04-23T16:55:03.341236+0000","last_peered":"2024-04-23T16:55:03.341236+0000","last_clean":"2024-04-23T16:55:03.341236+0000","last_became_active":"2024-04-23T16:51:21.063652+0000","last_became_peered":"2024-04-23T16:51:21.063652+0000","last_unstale":"2024-04-23T16:55:03.341236+0000","last_undegraded":"2024-04-23T16:55:03.341236+0000","last_fullsized":"2024-04-23T16:55:03.341236+0000","mapping_epoch":703,"log_start":"725'6900","ondisk_log_start":"725'6900","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'16831","last_scrub_stamp":"2024-04-23T16:55:01.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:55:01.549165+0000","objects_scrubbed":2128,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:27.145270+0000","scrub_duration":536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131268608,"num_objects":2003,"num_object_clones":0,"num_object_copies":4006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16943,"num_write_kb":606272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.2","version":"770'17133","reported_seq":17421,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:03.384418+0000","last_change":"2024-04-23T16:54:24.962245+0000","last_active":"2024-04-23T16:55:03.384418+0000","last_peered":"2024-04-23T16:55:03.384418+0000","last_clean":"2024-04-23T16:55:03.384418+0000","last_became_active":"2024-04-23T16:51:21.062561+0000","last_became_peered":"2024-04-23T16:51:21.062561+0000","last_unstale":"2024-04-23T16:55:03.384418+0000","last_undegraded":"2024-04-23T16:55:03.384418+0000","last_fullsized":"2024-04-23T16:55:03.384418+0000","mapping_epoch":703,"log_start":"725'7100","ondisk_log_start":"725'7100","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14622","last_scrub_stamp":"2024-04-23T16:54:24.962060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:24.962060+0000","objects_scrubbed":4531,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:49.051124+0000","scrub_duration":1162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129957888,"num_objects":1983,"num_object_clones":0,"num_object_copies":3966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17133,"num_write_kb":611712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1897,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:00.118198+0000","last_change":"2024-04-23T16:54:28.362501+0000","last_active":"2024-04-23T16:55:00.118198+0000","last_peered":"2024-04-23T16:55:00.118198+0000","last_clean":"2024-04-23T16:55:00.118198+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:55:00.118198+0000","last_undegraded":"2024-04-23T16:55:00.118198+0000","last_fullsized":"2024-04-23T16:55:00.118198+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:54:28.362406+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:54:28.362406+0000","last_clean_scrub_stamp":"2024-04-23T16:54:28.362406+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:28.652451+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"770'17325","reported_seq":17616,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:07.047011+0000","last_change":"2024-04-23T16:54:31.665323+0000","last_active":"2024-04-23T16:55:07.047011+0000","last_peered":"2024-04-23T16:55:07.047011+0000","last_clean":"2024-04-23T16:55:07.047011+0000","last_became_active":"2024-04-23T16:51:21.063395+0000","last_became_peered":"2024-04-23T16:51:21.063395+0000","last_unstale":"2024-04-23T16:55:07.047011+0000","last_undegraded":"2024-04-23T16:55:07.047011+0000","last_fullsized":"2024-04-23T16:55:07.047011+0000","mapping_epoch":703,"log_start":"726'7300","ondisk_log_start":"726'7300","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"763'15015","last_scrub_stamp":"2024-04-23T16:54:31.665200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:31.665200+0000","objects_scrubbed":4086,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:39.674179+0000","scrub_duration":876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114884608,"num_objects":1753,"num_object_clones":0,"num_object_copies":3506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17325,"num_write_kb":610496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1671,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:00.127922+0000","last_change":"2024-04-23T16:52:47.483911+0000","last_active":"2024-04-23T16:55:00.127922+0000","last_peered":"2024-04-23T16:55:00.127922+0000","last_clean":"2024-04-23T16:55:00.127922+0000","last_became_active":"2024-04-23T16:52:47.483615+0000","last_became_peered":"2024-04-23T16:52:47.483615+0000","last_unstale":"2024-04-23T16:55:00.127922+0000","last_undegraded":"2024-04-23T16:55:00.127922+0000","last_fullsized":"2024-04-23T16:55:00.127922+0000","mapping_epoch":735,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":736,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:52:05.504325+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:52:05.504325+0000","last_clean_scrub_stamp":"2024-04-23T16:52:05.504325+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"770'16577","reported_seq":16861,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:05.096823+0000","last_change":"2024-04-23T16:54:17.136201+0000","last_active":"2024-04-23T16:55:05.096823+0000","last_peered":"2024-04-23T16:55:05.096823+0000","last_clean":"2024-04-23T16:55:05.096823+0000","last_became_active":"2024-04-23T16:51:21.246391+0000","last_became_peered":"2024-04-23T16:51:21.246391+0000","last_unstale":"2024-04-23T16:55:05.096823+0000","last_undegraded":"2024-04-23T16:55:05.096823+0000","last_fullsized":"2024-04-23T16:55:05.096823+0000","mapping_epoch":703,"log_start":"724'6500","ondisk_log_start":"724'6500","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'13668","last_scrub_stamp":"2024-04-23T16:54:17.136038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:17.136038+0000","objects_scrubbed":4818,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:23.005433+0000","scrub_duration":963,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123011072,"num_objects":1877,"num_object_clones":0,"num_object_copies":3754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16577,"num_write_kb":590528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1881,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:00.120119+0000","last_change":"2024-04-23T16:52:38.199979+0000","last_active":"2024-04-23T16:55:00.120119+0000","last_peered":"2024-04-23T16:55:00.120119+0000","last_clean":"2024-04-23T16:55:00.120119+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:55:00.120119+0000","last_undegraded":"2024-04-23T16:55:00.120119+0000","last_fullsized":"2024-04-23T16:55:00.120119+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:52:38.199895+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:51:25.290774+0000","last_clean_scrub_stamp":"2024-04-23T16:52:38.199895+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"770'17284","reported_seq":17575,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:07.014358+0000","last_change":"2024-04-23T16:54:18.910760+0000","last_active":"2024-04-23T16:55:07.014358+0000","last_peered":"2024-04-23T16:55:07.014358+0000","last_clean":"2024-04-23T16:55:07.014358+0000","last_became_active":"2024-04-23T16:51:21.063669+0000","last_became_peered":"2024-04-23T16:51:21.063669+0000","last_unstale":"2024-04-23T16:55:07.014358+0000","last_undegraded":"2024-04-23T16:55:07.014358+0000","last_fullsized":"2024-04-23T16:55:07.014358+0000","mapping_epoch":703,"log_start":"726'7200","ondisk_log_start":"726'7200","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14143","last_scrub_stamp":"2024-04-23T16:54:18.910481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:18.910481+0000","objects_scrubbed":4947,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:29.963181+0000","scrub_duration":1208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116129792,"num_objects":1772,"num_object_clones":0,"num_object_copies":3544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17284,"num_write_kb":609792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1903,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:00.540834+0000","last_change":"2024-04-23T16:55:00.540871+0000","last_active":"2024-04-23T16:55:00.540834+0000","last_peered":"2024-04-23T16:55:00.540834+0000","last_clean":"2024-04-23T16:55:00.540834+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:55:00.540834+0000","last_undegraded":"2024-04-23T16:55:00.540834+0000","last_fullsized":"2024-04-23T16:55:00.540834+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:00.540749+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:55:00.540749+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:15.651892+0000","scrub_duration":229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"770'17146","reported_seq":17436,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:03.375709+0000","last_change":"2024-04-23T16:54:19.829779+0000","last_active":"2024-04-23T16:55:03.375709+0000","last_peered":"2024-04-23T16:55:03.375709+0000","last_clean":"2024-04-23T16:55:03.375709+0000","last_became_active":"2024-04-23T16:51:21.062760+0000","last_became_peered":"2024-04-23T16:51:21.062760+0000","last_unstale":"2024-04-23T16:55:03.375709+0000","last_undegraded":"2024-04-23T16:55:03.375709+0000","last_fullsized":"2024-04-23T16:55:03.375709+0000","mapping_epoch":703,"log_start":"725'7100","ondisk_log_start":"725'7100","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14356","last_scrub_stamp":"2024-04-23T16:54:19.829518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:19.829518+0000","objects_scrubbed":4782,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:22.348945+0000","scrub_duration":1029,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128974848,"num_objects":1968,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17146,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1487,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:02.927560+0000","last_change":"2024-04-23T16:55:02.927584+0000","last_active":"2024-04-23T16:55:02.927560+0000","last_peered":"2024-04-23T16:55:02.927560+0000","last_clean":"2024-04-23T16:55:02.927560+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:55:02.927560+0000","last_undegraded":"2024-04-23T16:55:02.927560+0000","last_fullsized":"2024-04-23T16:55:02.927560+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:02.927497+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:55:02.927497+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:29.690298+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"770'16797","reported_seq":17088,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:02.793610+0000","last_change":"2024-04-23T16:54:19.146832+0000","last_active":"2024-04-23T16:55:02.793610+0000","last_peered":"2024-04-23T16:55:02.793610+0000","last_clean":"2024-04-23T16:55:02.793610+0000","last_became_active":"2024-04-23T16:51:21.246519+0000","last_became_peered":"2024-04-23T16:51:21.246519+0000","last_unstale":"2024-04-23T16:55:02.793610+0000","last_undegraded":"2024-04-23T16:55:02.793610+0000","last_fullsized":"2024-04-23T16:55:02.793610+0000","mapping_epoch":703,"log_start":"724'6700","ondisk_log_start":"724'6700","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'13985","last_scrub_stamp":"2024-04-23T16:54:19.146171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:19.146171+0000","objects_scrubbed":4796,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:28.380772+0000","scrub_duration":863,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128253952,"num_objects":1957,"num_object_clones":0,"num_object_copies":3914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16797,"num_write_kb":600128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1448,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:03.258975+0000","last_change":"2024-04-23T16:55:03.259007+0000","last_active":"2024-04-23T16:55:03.258975+0000","last_peered":"2024-04-23T16:55:03.258975+0000","last_clean":"2024-04-23T16:55:03.258975+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:55:03.258975+0000","last_undegraded":"2024-04-23T16:55:03.258975+0000","last_fullsized":"2024-04-23T16:55:03.258975+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:03.258896+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:55:03.258896+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:32.586872+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.e","version":"770'17119","reported_seq":17444,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:06.310977+0000","last_change":"2024-04-23T16:52:41.066866+0000","last_active":"2024-04-23T16:55:06.310977+0000","last_peered":"2024-04-23T16:55:06.310977+0000","last_clean":"2024-04-23T16:55:06.310977+0000","last_became_active":"2024-04-23T16:51:21.064486+0000","last_became_peered":"2024-04-23T16:51:21.064486+0000","last_unstale":"2024-04-23T16:55:06.310977+0000","last_undegraded":"2024-04-23T16:55:06.310977+0000","last_fullsized":"2024-04-23T16:55:06.310977+0000","mapping_epoch":703,"log_start":"726'7100","ondisk_log_start":"726'7100","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"730'8312","last_scrub_stamp":"2024-04-23T16:52:41.066729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:52:41.066729+0000","objects_scrubbed":8276,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121831424,"num_objects":1859,"num_object_clones":0,"num_object_copies":3718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17119,"num_write_kb":607296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1905,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:00.117236+0000","last_change":"2024-04-23T16:54:26.409871+0000","last_active":"2024-04-23T16:55:00.117236+0000","last_peered":"2024-04-23T16:55:00.117236+0000","last_clean":"2024-04-23T16:55:00.117236+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:55:00.117236+0000","last_undegraded":"2024-04-23T16:55:00.117236+0000","last_fullsized":"2024-04-23T16:55:00.117236+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:54:26.409705+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:54:26.409705+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:33.984886+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.f","version":"770'17329","reported_seq":17621,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:07.040959+0000","last_change":"2024-04-23T16:54:17.546594+0000","last_active":"2024-04-23T16:55:07.040959+0000","last_peered":"2024-04-23T16:55:07.040959+0000","last_clean":"2024-04-23T16:55:07.040959+0000","last_became_active":"2024-04-23T16:51:21.063588+0000","last_became_peered":"2024-04-23T16:51:21.063588+0000","last_unstale":"2024-04-23T16:55:07.040959+0000","last_undegraded":"2024-04-23T16:55:07.040959+0000","last_fullsized":"2024-04-23T16:55:07.040959+0000","mapping_epoch":703,"log_start":"726'7300","ondisk_log_start":"726'7300","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14153","last_scrub_stamp":"2024-04-23T16:54:17.546419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:17.546419+0000","objects_scrubbed":4978,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:23.570053+0000","scrub_duration":812,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116850688,"num_objects":1783,"num_object_clones":0,"num_object_copies":3566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17329,"num_write_kb":611584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1916,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:00.120680+0000","last_change":"2024-04-23T16:54:55.936994+0000","last_active":"2024-04-23T16:55:00.120680+0000","last_peered":"2024-04-23T16:55:00.120680+0000","last_clean":"2024-04-23T16:55:00.120680+0000","last_became_active":"2024-04-23T16:52:44.474580+0000","last_became_peered":"2024-04-23T16:52:44.474580+0000","last_unstale":"2024-04-23T16:55:00.120680+0000","last_undegraded":"2024-04-23T16:55:00.120680+0000","last_fullsized":"2024-04-23T16:55:00.120680+0000","mapping_epoch":732,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":733,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:54:55.936863+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:54:55.936863+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:10.224576+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"770'17119","reported_seq":17463,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:05.091752+0000","last_change":"2024-04-23T16:54:56.611840+0000","last_active":"2024-04-23T16:55:05.091752+0000","last_peered":"2024-04-23T16:55:05.091752+0000","last_clean":"2024-04-23T16:55:05.091752+0000","last_became_active":"2024-04-23T16:51:21.079798+0000","last_became_peered":"2024-04-23T16:51:21.079798+0000","last_unstale":"2024-04-23T16:55:05.091752+0000","last_undegraded":"2024-04-23T16:55:05.091752+0000","last_fullsized":"2024-04-23T16:55:05.091752+0000","mapping_epoch":703,"log_start":"725'7100","ondisk_log_start":"725'7100","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"768'16627","last_scrub_stamp":"2024-04-23T16:54:56.611621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:56.611621+0000","objects_scrubbed":2377,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:07.253674+0000","scrub_duration":457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122880000,"num_objects":1875,"num_object_clones":0,"num_object_copies":3750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17119,"num_write_kb":607808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1789,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:00.124064+0000","last_change":"2024-04-23T16:54:32.619720+0000","last_active":"2024-04-23T16:55:00.124064+0000","last_peered":"2024-04-23T16:55:00.124064+0000","last_clean":"2024-04-23T16:55:00.124064+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:55:00.124064+0000","last_undegraded":"2024-04-23T16:55:00.124064+0000","last_fullsized":"2024-04-23T16:55:00.124064+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:54:32.619503+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:54:32.619503+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:40.217222+0000","scrub_duration":309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"770'16756","reported_seq":16991,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:06.301858+0000","last_change":"2024-04-23T16:52:43.250377+0000","last_active":"2024-04-23T16:55:06.301858+0000","last_peered":"2024-04-23T16:55:06.301858+0000","last_clean":"2024-04-23T16:55:06.301858+0000","last_became_active":"2024-04-23T16:51:21.064506+0000","last_became_peered":"2024-04-23T16:51:21.064506+0000","last_unstale":"2024-04-23T16:55:06.301858+0000","last_undegraded":"2024-04-23T16:55:06.301858+0000","last_fullsized":"2024-04-23T16:55:06.301858+0000","mapping_epoch":703,"log_start":"725'6700","ondisk_log_start":"725'6700","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"731'8308","last_scrub_stamp":"2024-04-23T16:52:43.250180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:52:43.250180+0000","objects_scrubbed":8271,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119472152,"num_objects":1824,"num_object_clones":0,"num_object_copies":3648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1824,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16756,"num_write_kb":594497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1714,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:00.127916+0000","last_change":"2024-04-23T16:52:47.480919+0000","last_active":"2024-04-23T16:55:00.127916+0000","last_peered":"2024-04-23T16:55:00.127916+0000","last_clean":"2024-04-23T16:55:00.127916+0000","last_became_active":"2024-04-23T16:52:47.480630+0000","last_became_peered":"2024-04-23T16:52:47.480630+0000","last_unstale":"2024-04-23T16:55:00.127916+0000","last_undegraded":"2024-04-23T16:55:00.127916+0000","last_fullsized":"2024-04-23T16:55:00.127916+0000","mapping_epoch":735,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":736,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:52:42.428791+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:49.834374+0000","last_clean_scrub_stamp":"2024-04-23T16:52:42.428791+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"770'19068","reported_seq":19439,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:05.108302+0000","last_change":"2024-04-23T16:54:24.434938+0000","last_active":"2024-04-23T16:55:05.108302+0000","last_peered":"2024-04-23T16:55:05.108302+0000","last_clean":"2024-04-23T16:55:05.108302+0000","last_became_active":"2024-04-23T16:51:21.063420+0000","last_became_peered":"2024-04-23T16:51:21.063420+0000","last_unstale":"2024-04-23T16:55:05.108302+0000","last_undegraded":"2024-04-23T16:55:05.108302+0000","last_fullsized":"2024-04-23T16:55:05.108302+0000","mapping_epoch":703,"log_start":"737'9002","ondisk_log_start":"737'9002","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'16466","last_scrub_stamp":"2024-04-23T16:54:24.434705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:24.434705+0000","objects_scrubbed":4533,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:43.605780+0000","scrub_duration":1184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124125184,"num_objects":1894,"num_object_clones":0,"num_object_copies":3788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17052,"num_write_kb":606272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1873,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-23T16:55:00.120123+0000","last_change":"2024-04-23T16:51:31.339866+0000","last_active":"2024-04-23T16:55:00.120123+0000","last_peered":"2024-04-23T16:55:00.120123+0000","last_clean":"2024-04-23T16:55:00.120123+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:55:00.120123+0000","last_undegraded":"2024-04-23T16:55:00.120123+0000","last_fullsized":"2024-04-23T16:55:00.120123+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:51:31.339728+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:51:31.339728+0000","last_clean_scrub_stamp":"2024-04-23T16:51:31.339728+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1956118552,"num_objects":29849,"num_object_clones":0,"num_object_copies":59698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":269,"num_objects_unfound":0,"num_objects_dirty":29849,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272413,"num_write_kb":9672321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8315,"num_bytes_recovered":544931840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4453572608,"data_stored":4453564464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160927,"ondisk_log_size":160927,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1679360,"data_stored":1661600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057117,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1610548,"kb_used_data":524268,"kb_used_omap":1,"kb_used_meta":1086270,"kb_avail":92134604,"statfs":{"total":95995035648,"available":94345834496,"internally_reserved":0,"allocated":536850432,"data_stored":536493477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1112340932},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351460,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613097,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1574900,"kb_used_data":745452,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":92170252,"statfs":{"total":95995035648,"available":94382338048,"internally_reserved":0,"allocated":763342848,"data_stored":762985893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":849344960},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":66,"seq":283467842029,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1241212,"kb_used_data":646388,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92503940,"statfs":{"total":95995035648,"available":94724034560,"internally_reserved":0,"allocated":661901312,"data_stored":661540805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169076,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1212984,"kb_used_data":469616,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92532168,"statfs":{"total":95995035648,"available":94752940032,"internally_reserved":0,"allocated":480886784,"data_stored":480525757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":761199050},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463416,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":829684,"kb_used_data":239020,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92915468,"statfs":{"total":95995035648,"available":95145439232,"internally_reserved":0,"allocated":244756480,"data_stored":244399525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":604830150},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137557,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2180536,"kb_used_data":1570928,"kb_used_omap":231,"kb_used_meta":609368,"kb_avail":91564616,"statfs":{"total":95995035648,"available":93762166784,"internally_reserved":0,"allocated":1608630272,"data_stored":1608269245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":623993744},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738902,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":883508,"kb_used_data":126252,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92861644,"statfs":{"total":95995035648,"available":95090323456,"internally_reserved":0,"allocated":129282048,"data_stored":128925093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":481,"seq":2065879269572,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":173500,"kb_used_data":86836,"kb_used_omap":281,"kb_used_meta":86310,"kb_avail":93571652,"statfs":{"total":95995035648,"available":95817371648,"internally_reserved":0,"allocated":88920064,"data_stored":88559565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":288353,"internal_metadata":88381855},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2658584756323,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":3126736191521,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757755,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":125435904,"data_stored":125435904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1601966080,"data_stored":1601962008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89522176,"data_stored":89522176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":237764608,"data_stored":237764608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":478220288,"data_stored":478216216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":652673024,"data_stored":652673024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":740687872,"data_stored":740687872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":527302656,"data_stored":527302656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:55:09.463 INFO:tasks.cephadm.ceph_manager.ceph:PG 9.7 is not active+clean 2024-04-23T16:55:09.463 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '9.7', 'version': "770'17296", 'reported_seq': 34073, 'reported_epoch': 770, 'state': 'active+recovering', 'last_fresh': '2024-04-23T16:55:06.330224+0000', 'last_change': '2024-04-23T16:52:44.746944+0000', 'last_active': '2024-04-23T16:55:06.330224+0000', 'last_peered': '2024-04-23T16:55:06.330224+0000', 'last_clean': '2024-04-23T16:52:43.617275+0000', 'last_became_active': '2024-04-23T16:52:44.683405+0000', 'last_became_peered': '2024-04-23T16:52:44.683405+0000', 'last_unstale': '2024-04-23T16:55:06.330224+0000', 'last_undegraded': '2024-04-23T16:55:06.330224+0000', 'last_fullsized': '2024-04-23T16:55:06.330224+0000', 'mapping_epoch': 732, 'log_start': "725'7200", 'ondisk_log_start': "725'7200", 'created': 703, 'last_epoch_clean': 704, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T16:51:20.043323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T16:51:20.043323+0000', 'last_clean_scrub_stamp': '2024-04-23T16:51:20.043323+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 116260864, 'num_objects': 1774, 'num_object_clones': 0, 'num_object_copies': 3548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 269, 'num_objects_unfound': 0, 'num_objects_dirty': 1774, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17296, 'num_write_kb': 610240, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8315, 'num_bytes_recovered': 544931840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': ['7', '1'], 'object_location_counts': [{'shards': '1,7', 'objects': 269}, {'shards': '2,7', 'objects': 1505}], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-23T16:55:09.463 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-23T16:55:09.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:09 smithi188 ceph-mon[44101]: from='client.37235 v1:172.21.15.103:0/1949368576' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:55:09.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:09 smithi188 ceph-mon[44101]: pgmap v2138: 27 pgs: 1 active+recovering, 26 active+clean; 1.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 269/56526 objects misplaced (0.476%); 7.3 MiB/s, 116 objects/s recovering 2024-04-23T16:55:09.528 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:55:09.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:09.532 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:55:09.581 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:55:09.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:09 smithi148 ceph-mon[44520]: from='client.37235 v1:172.21.15.103:0/1949368576' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:55:09.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:09 smithi148 ceph-mon[44520]: pgmap v2138: 27 pgs: 1 active+recovering, 26 active+clean; 1.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 269/56526 objects misplaced (0.476%); 7.3 MiB/s, 116 objects/s recovering 2024-04-23T16:55:09.633 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:55:09.696 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:09.728 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:55:09.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:09.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:55:09.838 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:55:09.865 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:09.884 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:55:09.886 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:09.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:09.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:55:09.988 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:55:10.045 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:10.743 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-23T16:55:10.765 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:11.016 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:55:11.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:11.021 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:55:11.122 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:55:11.179 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:11.215 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:55:11.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:11.220 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:55:11.322 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:55:11.348 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:11.371 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:55:11.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:11.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:55:11.471 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:11 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:55:11.471 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:11 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:55:11.472 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:11 smithi103 ceph-mon[39629]: osdmap e772: 12 total, 12 up, 8 in 2024-04-23T16:55:11.472 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:11 smithi103 ceph-mon[39629]: 1.8 scrub starts 2024-04-23T16:55:11.472 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:11 smithi103 ceph-mon[39629]: 1.8 scrub ok 2024-04-23T16:55:11.472 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:11 smithi103 ceph-mon[39629]: pgmap v2140: 27 pgs: 1 active+recovering, 26 active+clean; 1.7 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 872 op/s; 269/54544 objects misplaced (0.493%); 4.4 MiB/s, 69 objects/s recovering 2024-04-23T16:55:11.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:11 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:55:11.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:11 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:55:11.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:11 smithi188 ceph-mon[44101]: osdmap e772: 12 total, 12 up, 8 in 2024-04-23T16:55:11.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:11 smithi188 ceph-mon[44101]: 1.8 scrub starts 2024-04-23T16:55:11.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:11 smithi188 ceph-mon[44101]: 1.8 scrub ok 2024-04-23T16:55:11.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:11 smithi188 ceph-mon[44101]: pgmap v2140: 27 pgs: 1 active+recovering, 26 active+clean; 1.7 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 872 op/s; 269/54544 objects misplaced (0.493%); 4.4 MiB/s, 69 objects/s recovering 2024-04-23T16:55:11.476 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:55:11.525 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:11.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:11 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:55:11.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:11 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:55:11.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:11 smithi148 ceph-mon[44520]: osdmap e772: 12 total, 12 up, 8 in 2024-04-23T16:55:11.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:11 smithi148 ceph-mon[44520]: 1.8 scrub starts 2024-04-23T16:55:11.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:11 smithi148 ceph-mon[44520]: 1.8 scrub ok 2024-04-23T16:55:11.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:11 smithi148 ceph-mon[44520]: pgmap v2140: 27 pgs: 1 active+recovering, 26 active+clean; 1.7 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 872 op/s; 269/54544 objects misplaced (0.493%); 4.4 MiB/s, 69 objects/s recovering 2024-04-23T16:55:11.891 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:55:11.891 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-04-23T16:55:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-04-23T16:55:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2024-04-23T16:55:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:133294 8.a 8:5389bae5:::benchmark_data_smithi188_155546_object7785:head [delete] snapc 0=[] ondisk+write+known_if_redirected e649)", 2024-04-23T16:55:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:48:17.865339+0000", 2024-04-23T16:55:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 414.02290842999997, 2024-04-23T16:55:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.11017893300000001, 2024-04-23T16:55:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:55:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:55:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:55:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 133294 2024-04-23T16:55:11.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.893 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:11.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.893 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:11.893 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.865339+0000", 2024-04-23T16:55:11.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.893 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:11.893 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.865339+0000", 2024-04-23T16:55:11.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.893 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:11.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.865342+0000", 2024-04-23T16:55:11.894 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.368e-06 2024-04-23T16:55:11.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.894 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:11.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.865349+0000", 2024-04-23T16:55:11.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8210000000000004e-06 2024-04-23T16:55:11.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:11.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.865361+0000", 2024-04-23T16:55:11.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2682e-05 2024-04-23T16:55:11.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:11.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.865370+0000", 2024-04-23T16:55:11.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9029999999999993e-06 2024-04-23T16:55:11.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.867149+0000", 2024-04-23T16:55:11.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017787739999999999 2024-04-23T16:55:11.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:55:11.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.867264+0000", 2024-04-23T16:55:11.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000114688 2024-04-23T16:55:11.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.899 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:55:11.899 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.867367+0000", 2024-04-23T16:55:11.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010330100000000001 2024-04-23T16:55:11.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:55:11.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.874443+0000", 2024-04-23T16:55:11.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0070760140000000003 2024-04-23T16:55:11.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:11.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.975402+0000", 2024-04-23T16:55:11.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.100959457 2024-04-23T16:55:11.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:55:11.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.975463+0000", 2024-04-23T16:55:11.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0986000000000002e-05 2024-04-23T16:55:11.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:11.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.975518+0000", 2024-04-23T16:55:11.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4938999999999997e-05 2024-04-23T16:55:11.903 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.903 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:11.903 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.904 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:133311 8.a 8:5152764a:::benchmark_data_smithi188_155546_object7802:head [delete] snapc 0=[] ondisk+write+known_if_redirected e649)", 2024-04-23T16:55:11.904 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:48:17.877972+0000", 2024-04-23T16:55:11.904 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 414.01027635200001, 2024-04-23T16:55:11.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.097657944999999996, 2024-04-23T16:55:11.904 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:11.905 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:11.905 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:11.905 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:55:11.905 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:55:11.905 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:55:11.905 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 133311 2024-04-23T16:55:11.905 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.906 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:11.906 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.906 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:11.906 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.877972+0000", 2024-04-23T16:55:11.906 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.906 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.907 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:11.907 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.877972+0000", 2024-04-23T16:55:11.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.907 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.907 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:11.907 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.877974+0000", 2024-04-23T16:55:11.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.475e-06 2024-04-23T16:55:11.907 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:11.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.877982+0000", 2024-04-23T16:55:11.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6329999999999996e-06 2024-04-23T16:55:11.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:11.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.877995+0000", 2024-04-23T16:55:11.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3419e-05 2024-04-23T16:55:11.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:11.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.878004+0000", 2024-04-23T16:55:11.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0720000000000006e-06 2024-04-23T16:55:11.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.878065+0000", 2024-04-23T16:55:11.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0965999999999999e-05 2024-04-23T16:55:11.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:55:11.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.878171+0000", 2024-04-23T16:55:11.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010597100000000001 2024-04-23T16:55:11.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:55:11.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.878244+0000", 2024-04-23T16:55:11.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2607000000000005e-05 2024-04-23T16:55:11.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:55:11.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.884295+0000", 2024-04-23T16:55:11.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0060512400000000003 2024-04-23T16:55:11.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:11.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.975554+0000", 2024-04-23T16:55:11.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.091259187000000005 2024-04-23T16:55:11.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:55:11.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.975596+0000", 2024-04-23T16:55:11.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1573000000000003e-05 2024-04-23T16:55:11.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:11.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.975629+0000", 2024-04-23T16:55:11.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3801999999999999e-05 2024-04-23T16:55:11.914 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.914 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:11.914 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.915 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:133313 8.a 8:5858210a:::benchmark_data_smithi188_155546_object7804:head [delete] snapc 0=[] ondisk+write+known_if_redirected e649)", 2024-04-23T16:55:11.915 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:48:17.881286+0000", 2024-04-23T16:55:11.915 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 414.00696237, 2024-04-23T16:55:11.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.096694506999999999, 2024-04-23T16:55:11.915 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:11.916 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:11.916 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:11.916 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:55:11.916 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:55:11.916 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:55:11.917 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 133313 2024-04-23T16:55:11.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.917 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:11.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:11.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.881286+0000", 2024-04-23T16:55:11.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:11.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.881286+0000", 2024-04-23T16:55:11.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:11.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.881288+0000", 2024-04-23T16:55:11.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1689999999999999e-06 2024-04-23T16:55:11.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:11.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.881295+0000", 2024-04-23T16:55:11.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.917e-06 2024-04-23T16:55:11.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:11.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.881306+0000", 2024-04-23T16:55:11.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1667e-05 2024-04-23T16:55:11.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:11.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.881315+0000", 2024-04-23T16:55:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.636e-06 2024-04-23T16:55:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.881353+0000", 2024-04-23T16:55:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7700000000000002e-05 2024-04-23T16:55:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:55:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.881429+0000", 2024-04-23T16:55:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6556000000000004e-05 2024-04-23T16:55:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:55:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.881487+0000", 2024-04-23T16:55:11.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7661000000000002e-05 2024-04-23T16:55:11.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:55:11.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.884437+0000", 2024-04-23T16:55:11.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029501420000000002 2024-04-23T16:55:11.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:11.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.977881+0000", 2024-04-23T16:55:11.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.093444490000000005 2024-04-23T16:55:11.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:55:11.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.977933+0000", 2024-04-23T16:55:11.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1975999999999998e-05 2024-04-23T16:55:11.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:11.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:48:17.977980+0000", 2024-04-23T16:55:11.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6592999999999999e-05 2024-04-23T16:55:11.922 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.922 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:11.922 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.922 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:166122 8.a 8:5942b20f:::benchmark_data_smithi188_155546_object40613:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-23T16:55:11.922 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:49:03.366318+0000", 2024-04-23T16:55:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 368.52192985099998, 2024-04-23T16:55:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.086988368999999996, 2024-04-23T16:55:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:55:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:55:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:55:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 166122 2024-04-23T16:55:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.366318+0000", 2024-04-23T16:55:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.366318+0000", 2024-04-23T16:55:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.366320+0000", 2024-04-23T16:55:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.869e-06 2024-04-23T16:55:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.366326+0000", 2024-04-23T16:55:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.806e-06 2024-04-23T16:55:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.366336+0000", 2024-04-23T16:55:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0738e-05 2024-04-23T16:55:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.366344+0000", 2024-04-23T16:55:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8380000000000002e-06 2024-04-23T16:55:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.366401+0000", 2024-04-23T16:55:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6338000000000002e-05 2024-04-23T16:55:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:55:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.366477+0000", 2024-04-23T16:55:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6778000000000002e-05 2024-04-23T16:55:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:55:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.366543+0000", 2024-04-23T16:55:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5549000000000005e-05 2024-04-23T16:55:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:55:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.371878+0000", 2024-04-23T16:55:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053354229999999997 2024-04-23T16:55:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.453188+0000", 2024-04-23T16:55:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.081309598999999996 2024-04-23T16:55:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:55:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.453262+0000", 2024-04-23T16:55:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3620000000000003e-05 2024-04-23T16:55:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:49:03.453306+0000", 2024-04-23T16:55:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4811000000000002e-05 2024-04-23T16:55:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:218218 8.a 8:5770b466:::benchmark_data_smithi188_155546_object92709:head [delete] snapc 0=[] ondisk+write+known_if_redirected e677)", 2024-04-23T16:55:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:50:04.344434+0000", 2024-04-23T16:55:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 307.543814107, 2024-04-23T16:55:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.097772399999999995, 2024-04-23T16:55:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:55:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:55:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:55:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 218218 2024-04-23T16:55:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.344434+0000", 2024-04-23T16:55:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.344434+0000", 2024-04-23T16:55:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:11.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.344436+0000", 2024-04-23T16:55:11.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.43e-06 2024-04-23T16:55:11.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:11.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.344443+0000", 2024-04-23T16:55:11.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0439999999999998e-06 2024-04-23T16:55:11.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:11.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.344458+0000", 2024-04-23T16:55:11.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4704e-05 2024-04-23T16:55:11.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:11.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.344467+0000", 2024-04-23T16:55:11.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3749999999999992e-06 2024-04-23T16:55:11.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.349154+0000", 2024-04-23T16:55:11.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046870829999999999 2024-04-23T16:55:11.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:55:11.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.349288+0000", 2024-04-23T16:55:11.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000133602 2024-04-23T16:55:11.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:55:11.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.349399+0000", 2024-04-23T16:55:11.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000111376 2024-04-23T16:55:11.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:55:11.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.359249+0000", 2024-04-23T16:55:11.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0098499739999999992 2024-04-23T16:55:11.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:11.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.442108+0000", 2024-04-23T16:55:11.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.082858672999999994 2024-04-23T16:55:11.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:55:11.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.442164+0000", 2024-04-23T16:55:11.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6242000000000001e-05 2024-04-23T16:55:11.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:11.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.442206+0000", 2024-04-23T16:55:11.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1897000000000002e-05 2024-04-23T16:55:11.933 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.933 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:11.934 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.934 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:218239 8.a 8:5ae69382:::benchmark_data_smithi188_155546_object92730:head [delete] snapc 0=[] ondisk+write+known_if_redirected e677)", 2024-04-23T16:55:11.934 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:50:04.368257+0000", 2024-04-23T16:55:11.934 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 307.51999120200003, 2024-04-23T16:55:11.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.091060946000000004, 2024-04-23T16:55:11.934 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:11.934 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:11.934 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:11.934 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:55:11.934 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:55:11.934 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:55:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 218239 2024-04-23T16:55:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.368257+0000", 2024-04-23T16:55:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.368257+0000", 2024-04-23T16:55:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.368259+0000", 2024-04-23T16:55:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3149999999999999e-06 2024-04-23T16:55:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.368264+0000", 2024-04-23T16:55:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.135e-06 2024-04-23T16:55:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.368276+0000", 2024-04-23T16:55:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2140999999999999e-05 2024-04-23T16:55:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:11.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.368284+0000", 2024-04-23T16:55:11.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8180000000000004e-06 2024-04-23T16:55:11.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.368345+0000", 2024-04-23T16:55:11.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0543e-05 2024-04-23T16:55:11.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:55:11.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.368448+0000", 2024-04-23T16:55:11.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000103761 2024-04-23T16:55:11.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:55:11.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.368556+0000", 2024-04-23T16:55:11.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000108076 2024-04-23T16:55:11.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:55:11.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.377660+0000", 2024-04-23T16:55:11.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091033409999999992 2024-04-23T16:55:11.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:11.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.459218+0000", 2024-04-23T16:55:11.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.081558418999999993 2024-04-23T16:55:11.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:55:11.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.459271+0000", 2024-04-23T16:55:11.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3121999999999997e-05 2024-04-23T16:55:11.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:11.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.459318+0000", 2024-04-23T16:55:11.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6275e-05 2024-04-23T16:55:11.939 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.939 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:11.939 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:218243 8.a 8:55986bd0:::benchmark_data_smithi188_155546_object92734:head [delete] snapc 0=[] ondisk+write+known_if_redirected e677)", 2024-04-23T16:55:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:50:04.372705+0000", 2024-04-23T16:55:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 307.51554287200003, 2024-04-23T16:55:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.086727655000000001, 2024-04-23T16:55:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:55:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:55:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:55:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 218243 2024-04-23T16:55:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.372705+0000", 2024-04-23T16:55:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.372705+0000", 2024-04-23T16:55:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.372708+0000", 2024-04-23T16:55:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7580000000000001e-06 2024-04-23T16:55:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:11.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.372716+0000", 2024-04-23T16:55:11.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3319999999999992e-06 2024-04-23T16:55:11.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:11.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.372727+0000", 2024-04-23T16:55:11.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1342000000000001e-05 2024-04-23T16:55:11.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:11.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.372736+0000", 2024-04-23T16:55:11.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8750000000000006e-06 2024-04-23T16:55:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.372815+0000", 2024-04-23T16:55:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8734999999999997e-05 2024-04-23T16:55:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:55:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.372928+0000", 2024-04-23T16:55:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000112454 2024-04-23T16:55:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:55:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.373042+0000", 2024-04-23T16:55:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000114657 2024-04-23T16:55:11.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:55:11.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.378390+0000", 2024-04-23T16:55:11.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053481529999999996 2024-04-23T16:55:11.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:11.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.459353+0000", 2024-04-23T16:55:11.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.080962440999999996 2024-04-23T16:55:11.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:55:11.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.459399+0000", 2024-04-23T16:55:11.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6479e-05 2024-04-23T16:55:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:04.459433+0000", 2024-04-23T16:55:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3429000000000001e-05 2024-04-23T16:55:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:245129 8.b 8.3b61954b (undecoded) ondisk+retry+write+known_if_redirected e693)", 2024-04-23T16:55:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:50:52.629359+0000", 2024-04-23T16:55:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 259.25888901500002, 2024-04-23T16:55:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.147168771, 2024-04-23T16:55:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:55:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:55:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:55:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:55:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 245129 2024-04-23T16:55:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629359+0000", 2024-04-23T16:55:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629359+0000", 2024-04-23T16:55:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629362+0000", 2024-04-23T16:55:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3400000000000002e-06 2024-04-23T16:55:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629388+0000", 2024-04-23T16:55:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5666000000000001e-05 2024-04-23T16:55:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629404+0000", 2024-04-23T16:55:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5747999999999999e-05 2024-04-23T16:55:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629416+0000", 2024-04-23T16:55:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2317000000000001e-05 2024-04-23T16:55:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629556+0000", 2024-04-23T16:55:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013970200000000001 2024-04-23T16:55:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:55:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629577+0000", 2024-04-23T16:55:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1243e-05 2024-04-23T16:55:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776495+0000", 2024-04-23T16:55:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14691843399999999 2024-04-23T16:55:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776528+0000", 2024-04-23T16:55:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2320999999999998e-05 2024-04-23T16:55:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:245137 8.b 8.7c0e014b (undecoded) ondisk+retry+write+known_if_redirected e693)", 2024-04-23T16:55:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:50:52.629445+0000", 2024-04-23T16:55:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 259.25880337400002, 2024-04-23T16:55:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14719322100000001, 2024-04-23T16:55:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:55:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:55:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:55:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:55:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 245137 2024-04-23T16:55:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629445+0000", 2024-04-23T16:55:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629445+0000", 2024-04-23T16:55:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629445+0000", 2024-04-23T16:55:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3200000000000003e-07 2024-04-23T16:55:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629449+0000", 2024-04-23T16:55:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8700000000000002e-06 2024-04-23T16:55:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629455+0000", 2024-04-23T16:55:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1079999999999998e-06 2024-04-23T16:55:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629461+0000", 2024-04-23T16:55:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8420000000000001e-06 2024-04-23T16:55:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629673+0000", 2024-04-23T16:55:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000211664 2024-04-23T16:55:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:55:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629684+0000", 2024-04-23T16:55:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0628999999999999e-05 2024-04-23T16:55:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776631+0000", 2024-04-23T16:55:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14694732299999999 2024-04-23T16:55:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776638+0000", 2024-04-23T16:55:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8530000000000003e-06 2024-04-23T16:55:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:245145 8.b 8.8ae4058b (undecoded) ondisk+retry+write+known_if_redirected e693)", 2024-04-23T16:55:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:50:52.629482+0000", 2024-04-23T16:55:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 259.25876563399999, 2024-04-23T16:55:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14720270899999999, 2024-04-23T16:55:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:55:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:55:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:55:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:55:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 245145 2024-04-23T16:55:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629482+0000", 2024-04-23T16:55:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629482+0000", 2024-04-23T16:55:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629483+0000", 2024-04-23T16:55:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6899999999999996e-07 2024-04-23T16:55:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629486+0000", 2024-04-23T16:55:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3550000000000001e-06 2024-04-23T16:55:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629497+0000", 2024-04-23T16:55:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0896999999999999e-05 2024-04-23T16:55:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629503+0000", 2024-04-23T16:55:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9959999999999999e-06 2024-04-23T16:55:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629753+0000", 2024-04-23T16:55:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00025012899999999998 2024-04-23T16:55:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:55:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629775+0000", 2024-04-23T16:55:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1433e-05 2024-04-23T16:55:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776675+0000", 2024-04-23T16:55:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14690025100000001 2024-04-23T16:55:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776685+0000", 2024-04-23T16:55:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8789999999999998e-06 2024-04-23T16:55:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:245150 8.b 8.40b84c3b (undecoded) ondisk+retry+write+known_if_redirected e693)", 2024-04-23T16:55:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:50:52.629519+0000", 2024-04-23T16:55:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 259.258729335, 2024-04-23T16:55:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14719273099999999, 2024-04-23T16:55:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:55:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:55:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:55:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:55:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 245150 2024-04-23T16:55:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629519+0000", 2024-04-23T16:55:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629519+0000", 2024-04-23T16:55:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629519+0000", 2024-04-23T16:55:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1299999999999999e-07 2024-04-23T16:55:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629523+0000", 2024-04-23T16:55:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3780000000000001e-06 2024-04-23T16:55:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629530+0000", 2024-04-23T16:55:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1620000000000002e-06 2024-04-23T16:55:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629535+0000", 2024-04-23T16:55:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3299999999999998e-06 2024-04-23T16:55:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629804+0000", 2024-04-23T16:55:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000268671 2024-04-23T16:55:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:55:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629812+0000", 2024-04-23T16:55:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0749999999999998e-06 2024-04-23T16:55:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776706+0000", 2024-04-23T16:55:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14689437699999999 2024-04-23T16:55:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776711+0000", 2024-04-23T16:55:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0250000000000002e-06 2024-04-23T16:55:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.16995.0:245162 8.b 8.2eb619eb (undecoded) ondisk+retry+write+known_if_redirected e693)", 2024-04-23T16:55:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:50:52.629543+0000", 2024-04-23T16:55:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 259.25870521100001, 2024-04-23T16:55:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14728439099999999, 2024-04-23T16:55:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2024-04-23T16:55:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:55:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.16995", 2024-04-23T16:55:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:55:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 245162 2024-04-23T16:55:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629543+0000", 2024-04-23T16:55:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629543+0000", 2024-04-23T16:55:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629543+0000", 2024-04-23T16:55:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8500000000000001e-07 2024-04-23T16:55:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629546+0000", 2024-04-23T16:55:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8679999999999999e-06 2024-04-23T16:55:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629551+0000", 2024-04-23T16:55:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.904e-06 2024-04-23T16:55:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629556+0000", 2024-04-23T16:55:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1889999999999998e-06 2024-04-23T16:55:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629870+0000", 2024-04-23T16:55:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031422699999999999 2024-04-23T16:55:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2024-04-23T16:55:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.629882+0000", 2024-04-23T16:55:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1571e-05 2024-04-23T16:55:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776822+0000", 2024-04-23T16:55:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.14694010699999999 2024-04-23T16:55:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:50:52.776827+0000", 2024-04-23T16:55:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9400000000000001e-06 2024-04-23T16:55:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.27202.0:50236 9.3 9:cab099b0:::benchmark_data_smithi188_167360_object50235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e714)", 2024-04-23T16:55:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:51:45.580775+0000", 2024-04-23T16:55:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 206.307472426, 2024-04-23T16:55:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.13267864900000001, 2024-04-23T16:55:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:55:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.27202", 2024-04-23T16:55:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:55:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 50236 2024-04-23T16:55:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.580775+0000", 2024-04-23T16:55:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.580775+0000", 2024-04-23T16:55:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.580777+0000", 2024-04-23T16:55:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7659999999999999e-06 2024-04-23T16:55:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.580810+0000", 2024-04-23T16:55:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2827000000000003e-05 2024-04-23T16:55:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.580821+0000", 2024-04-23T16:55:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0581000000000001e-05 2024-04-23T16:55:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.580828+0000", 2024-04-23T16:55:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1980000000000004e-06 2024-04-23T16:55:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.581295+0000", 2024-04-23T16:55:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00046699900000000001 2024-04-23T16:55:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:55:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.581383+0000", 2024-04-23T16:55:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8374000000000005e-05 2024-04-23T16:55:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:55:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.581461+0000", 2024-04-23T16:55:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7639000000000007e-05 2024-04-23T16:55:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:55:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.583043+0000", 2024-04-23T16:55:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001582176 2024-04-23T16:55:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.713361+0000", 2024-04-23T16:55:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.13031794699999999 2024-04-23T16:55:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:55:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.713413+0000", 2024-04-23T16:55:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2151999999999998e-05 2024-04-23T16:55:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.713454+0000", 2024-04-23T16:55:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0989999999999999e-05 2024-04-23T16:55:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop(client.27202.0:50239 9.4 e714/703 9:2eb04f24:::benchmark_data_smithi188_167360_object50238:head v 714'3100, mlcod=714'3098)", 2024-04-23T16:55:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:51:45.583042+0000", 2024-04-23T16:55:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 206.30520606300001, 2024-04-23T16:55:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.135581285, 2024-04-23T16:55:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.583042+0000", 2024-04-23T16:55:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.583042+0000", 2024-04-23T16:55:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.583043+0000", 2024-04-23T16:55:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4449999999999999e-06 2024-04-23T16:55:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.583090+0000", 2024-04-23T16:55:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.655e-05 2024-04-23T16:55:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.583108+0000", 2024-04-23T16:55:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8020999999999999e-05 2024-04-23T16:55:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.583113+0000", 2024-04-23T16:55:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9219999999999996e-06 2024-04-23T16:55:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.583156+0000", 2024-04-23T16:55:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2746e-05 2024-04-23T16:55:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:55:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.583174+0000", 2024-04-23T16:55:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8550999999999999e-05 2024-04-23T16:55:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:55:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.718577+0000", 2024-04-23T16:55:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.13540260200000001 2024-04-23T16:55:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.718623+0000", 2024-04-23T16:55:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6448e-05 2024-04-23T16:55:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.27202.0:50241 9.b 9:dbd7743c:::benchmark_data_smithi188_167360_object50240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e714)", 2024-04-23T16:55:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:51:45.583855+0000", 2024-04-23T16:55:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 206.30439337300001, 2024-04-23T16:55:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.133752918, 2024-04-23T16:55:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:55:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.27202", 2024-04-23T16:55:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:55:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 50241 2024-04-23T16:55:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.583855+0000", 2024-04-23T16:55:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.583855+0000", 2024-04-23T16:55:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.583856+0000", 2024-04-23T16:55:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3039999999999999e-06 2024-04-23T16:55:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.583875+0000", 2024-04-23T16:55:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9358999999999999e-05 2024-04-23T16:55:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.713242+0000", 2024-04-23T16:55:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.129366445 2024-04-23T16:55:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.713257+0000", 2024-04-23T16:55:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5401e-05 2024-04-23T16:55:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.713362+0000", 2024-04-23T16:55:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010515500000000001 2024-04-23T16:55:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:55:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.713510+0000", 2024-04-23T16:55:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014809400000000001 2024-04-23T16:55:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 1", 2024-04-23T16:55:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.713609+0000", 2024-04-23T16:55:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8548000000000006e-05 2024-04-23T16:55:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.717488+0000", 2024-04-23T16:55:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003878912 2024-04-23T16:55:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:55:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.717506+0000", 2024-04-23T16:55:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7952e-05 2024-04-23T16:55:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:55:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.717561+0000", 2024-04-23T16:55:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5237e-05 2024-04-23T16:55:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.717607+0000", 2024-04-23T16:55:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6511000000000002e-05 2024-04-23T16:55:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_repop_reply(client.27202.0:50236 9.3 e714/703 ondisk, result = 0)", 2024-04-23T16:55:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:51:45.584584+0000", 2024-04-23T16:55:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 206.303664113, 2024-04-23T16:55:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.128877995, 2024-04-23T16:55:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2024-04-23T16:55:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.584584+0000", 2024-04-23T16:55:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.584584+0000", 2024-04-23T16:55:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:11.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.584585+0000", 2024-04-23T16:55:11.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.088e-06 2024-04-23T16:55:11.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:11.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.584588+0000", 2024-04-23T16:55:11.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6049999999999998e-06 2024-04-23T16:55:11.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:11.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.713245+0000", 2024-04-23T16:55:11.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.12865659800000001 2024-04-23T16:55:11.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.713256+0000", 2024-04-23T16:55:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.079e-05 2024-04-23T16:55:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.713346+0000", 2024-04-23T16:55:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0295000000000002e-05 2024-04-23T16:55:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:55:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.713355+0000", 2024-04-23T16:55:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2709999999999998e-06 2024-04-23T16:55:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:51:45.713462+0000", 2024-04-23T16:55:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010634800000000001 2024-04-23T16:55:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.27202.0:79818 9.3 9:c33fd4fa:::benchmark_data_smithi188_167360_object79817:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e719)", 2024-04-23T16:55:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:52:01.684210+0000", 2024-04-23T16:55:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 190.204037852, 2024-04-23T16:55:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.089118033999999999, 2024-04-23T16:55:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:55:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.27202", 2024-04-23T16:55:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:55:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 79818 2024-04-23T16:55:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:01.684210+0000", 2024-04-23T16:55:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.990 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:11.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:01.684210+0000", 2024-04-23T16:55:11.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.990 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.990 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:11.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:01.684212+0000", 2024-04-23T16:55:11.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.463e-06 2024-04-23T16:55:11.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.990 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.990 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:11.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:01.684232+0000", 2024-04-23T16:55:11.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0936000000000001e-05 2024-04-23T16:55:11.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:01.684239+0000", 2024-04-23T16:55:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0430000000000002e-06 2024-04-23T16:55:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:01.684246+0000", 2024-04-23T16:55:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2899999999999999e-06 2024-04-23T16:55:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:01.684295+0000", 2024-04-23T16:55:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9398999999999999e-05 2024-04-23T16:55:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:55:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:01.684408+0000", 2024-04-23T16:55:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000113009 2024-04-23T16:55:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 9", 2024-04-23T16:55:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:01.684521+0000", 2024-04-23T16:55:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011294300000000001 2024-04-23T16:55:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:01.768188+0000", 2024-04-23T16:55:11.993 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.083666391000000007 2024-04-23T16:55:11.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.993 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:55:11.993 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:01.773228+0000", 2024-04-23T16:55:11.993 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050406619999999996 2024-04-23T16:55:11.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.993 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:55:11.993 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:01.773277+0000", 2024-04-23T16:55:11.993 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8838000000000002e-05 2024-04-23T16:55:11.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.993 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:52:01.773328+0000", 2024-04-23T16:55:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1060000000000002e-05 2024-04-23T16:55:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.27202.0:256092 9.b 9:d191289f:::benchmark_data_smithi188_167360_object104959:head [delete] snapc 0=[] ondisk+write+known_if_redirected e765)", 2024-04-23T16:55:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:54:49.048277+0000", 2024-04-23T16:55:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.839970853000001, 2024-04-23T16:55:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.131859436, 2024-04-23T16:55:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:11.995 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:55:11.995 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.27202", 2024-04-23T16:55:11.995 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:55:11.995 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 256092 2024-04-23T16:55:11.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.995 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:11.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.995 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:11.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.048277+0000", 2024-04-23T16:55:11.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.995 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:11.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.048277+0000", 2024-04-23T16:55:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.048278+0000", 2024-04-23T16:55:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3399999999999998e-07 2024-04-23T16:55:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.048281+0000", 2024-04-23T16:55:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4790000000000001e-06 2024-04-23T16:55:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:11.997 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.048288+0000", 2024-04-23T16:55:11.997 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4869999999999999e-06 2024-04-23T16:55:11.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.997 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:11.997 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.048293+0000", 2024-04-23T16:55:11.997 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3079999999999998e-06 2024-04-23T16:55:11.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.997 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:11.997 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.050145+0000", 2024-04-23T16:55:11.997 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001851852 2024-04-23T16:55:11.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:55:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.050261+0000", 2024-04-23T16:55:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011608900000000001 2024-04-23T16:55:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 1", 2024-04-23T16:55:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.050370+0000", 2024-04-23T16:55:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000109016 2024-04-23T16:55:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:55:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.054994+0000", 2024-04-23T16:55:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046240070000000003 2024-04-23T16:55:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.999 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:11.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.180011+0000", 2024-04-23T16:55:11.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.125017196 2024-04-23T16:55:11.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.999 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:55:11.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.180069+0000", 2024-04-23T16:55:11.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8062000000000001e-05 2024-04-23T16:55:11.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:11.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:11.999 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:11.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.180136+0000", 2024-04-23T16:55:11.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7106000000000004e-05 2024-04-23T16:55:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.27202.0:256105 9.2 9:4447e7f1:::benchmark_data_smithi188_167360_object104972:head [delete] snapc 0=[] ondisk+write+known_if_redirected e765)", 2024-04-23T16:55:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:54:49.061316+0000", 2024-04-23T16:55:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.82693158, 2024-04-23T16:55:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.13909565500000001, 2024-04-23T16:55:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:55:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.27202", 2024-04-23T16:55:12.001 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:55:12.001 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 256105 2024-04-23T16:55:12.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.001 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:12.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:12.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.061316+0000", 2024-04-23T16:55:12.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:12.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:12.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.061316+0000", 2024-04-23T16:55:12.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:12.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.061319+0000", 2024-04-23T16:55:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.796e-06 2024-04-23T16:55:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.061327+0000", 2024-04-23T16:55:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1130000000000001e-06 2024-04-23T16:55:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.061341+0000", 2024-04-23T16:55:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4031e-05 2024-04-23T16:55:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.061351+0000", 2024-04-23T16:55:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9010000000000006e-06 2024-04-23T16:55:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.061410+0000", 2024-04-23T16:55:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8678000000000003e-05 2024-04-23T16:55:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:55:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.061558+0000", 2024-04-23T16:55:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014767200000000001 2024-04-23T16:55:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 1", 2024-04-23T16:55:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.061686+0000", 2024-04-23T16:55:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012840799999999999 2024-04-23T16:55:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:55:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.065705+0000", 2024-04-23T16:55:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004019262 2024-04-23T16:55:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.200322+0000", 2024-04-23T16:55:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.134616876 2024-04-23T16:55:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:55:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.200376+0000", 2024-04-23T16:55:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3834e-05 2024-04-23T16:55:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.200412+0000", 2024-04-23T16:55:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6084000000000002e-05 2024-04-23T16:55:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.27202.0:256109 9.2 9:44afc45d:::benchmark_data_smithi188_167360_object104976:head [delete] snapc 0=[] ondisk+write+known_if_redirected e765)", 2024-04-23T16:55:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2024-04-23T16:54:49.066800+0000", 2024-04-23T16:55:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.821448059000002, 2024-04-23T16:55:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.13374540900000001, 2024-04-23T16:55:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: "continuous": false, 2024-04-23T16:55:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2024-04-23T16:55:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2024-04-23T16:55:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.27202", 2024-04-23T16:55:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:55:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 256109 2024-04-23T16:55:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2024-04-23T16:55:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2024-04-23T16:55:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.066800+0000", 2024-04-23T16:55:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2024-04-23T16:55:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.066800+0000", 2024-04-23T16:55:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2024-04-23T16:55:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2024-04-23T16:55:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.066802+0000", 2024-04-23T16:55:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6899999999999999e-06 2024-04-23T16:55:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2024-04-23T16:55:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.066807+0000", 2024-04-23T16:55:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4480000000000002e-06 2024-04-23T16:55:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2024-04-23T16:55:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.066817+0000", 2024-04-23T16:55:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7739999999999999e-06 2024-04-23T16:55:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2024-04-23T16:55:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.066823+0000", 2024-04-23T16:55:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6760000000000001e-06 2024-04-23T16:55:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2024-04-23T16:55:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.066863+0000", 2024-04-23T16:55:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9070999999999999e-05 2024-04-23T16:55:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2024-04-23T16:55:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.066943+0000", 2024-04-23T16:55:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0049999999999994e-05 2024-04-23T16:55:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for subops from 1", 2024-04-23T16:55:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.067012+0000", 2024-04-23T16:55:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9819000000000004e-05 2024-04-23T16:55:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op_commit", 2024-04-23T16:55:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.069757+0000", 2024-04-23T16:55:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027447029999999998 2024-04-23T16:55:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.200463+0000", 2024-04-23T16:55:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.13070556799999999 2024-04-23T16:55:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2024-04-23T16:55:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.200508+0000", 2024-04-23T16:55:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5673000000000001e-05 2024-04-23T16:55:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T16:55:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2024-04-23T16:55:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2024-04-23T16:55:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2024-04-23T16:54:49.200545+0000", 2024-04-23T16:55:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6937000000000002e-05 2024-04-23T16:55:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:12.015 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T16:55:12.015 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T16:55:12.015 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:55:12.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:12 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:55:12.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:12 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:55:12.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:12 smithi188 ceph-mon[44101]: 1.5 deep-scrub starts 2024-04-23T16:55:12.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:12 smithi188 ceph-mon[44101]: 1.5 deep-scrub ok 2024-04-23T16:55:12.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:12 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:55:12.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:12 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:55:12.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:12 smithi103 ceph-mon[39629]: 1.5 deep-scrub starts 2024-04-23T16:55:12.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:12 smithi103 ceph-mon[39629]: 1.5 deep-scrub ok 2024-04-23T16:55:12.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:12 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:55:12.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:12 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:55:12.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:12 smithi148 ceph-mon[44520]: 1.5 deep-scrub starts 2024-04-23T16:55:12.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:12 smithi148 ceph-mon[44520]: 1.5 deep-scrub ok 2024-04-23T16:55:12.682 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-23T16:55:13.119 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:55:13.182 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:55:13.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:13.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:55:13.294 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:55:13.337 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:13.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:13 smithi103 ceph-mon[39629]: 9.7 scrub starts 2024-04-23T16:55:13.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:13 smithi103 ceph-mon[39629]: osdmap e773: 12 total, 12 up, 8 in 2024-04-23T16:55:13.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:13 smithi103 ceph-mon[39629]: 9.7 scrub ok 2024-04-23T16:55:13.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:13 smithi103 ceph-mon[39629]: pgmap v2142: 27 pgs: 27 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.24k op/s; 2.8 MiB/s, 44 objects/s recovering 2024-04-23T16:55:13.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:13 smithi148 ceph-mon[44520]: 9.7 scrub starts 2024-04-23T16:55:13.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:13 smithi148 ceph-mon[44520]: osdmap e773: 12 total, 12 up, 8 in 2024-04-23T16:55:13.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:13 smithi148 ceph-mon[44520]: 9.7 scrub ok 2024-04-23T16:55:13.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:13 smithi148 ceph-mon[44520]: pgmap v2142: 27 pgs: 27 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.24k op/s; 2.8 MiB/s, 44 objects/s recovering 2024-04-23T16:55:13.645 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:55:13.645 INFO:teuthology.orchestra.run.smithi103.stdout:{"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2695,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":773,"num_osds":12,"num_up_osds":12,"osd_up_since":1713891149,"num_in_osds":8,"osd_in_since":1713891163,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":27}],"num_pgs":27,"num_pools":2,"num_objects":24065,"data_bytes":1577517624,"bytes_used":8618135552,"bytes_avail":759342149632,"bytes_total":767960285184,"recovering_objects_per_sec":44,"recovering_bytes_per_sec":2937636,"recovering_keys_per_sec":0,"num_objects_recovered":269,"num_bytes_recovered":17629184,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1235},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.103:8443/"}},"servicemap":{"epoch":55,"modified":"2024-04-23T16:53:50.989444+0000","services":{}},"progress_events":{}} 2024-04-23T16:55:13.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:13 smithi188 ceph-mon[44101]: 9.7 scrub starts 2024-04-23T16:55:13.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:13 smithi188 ceph-mon[44101]: osdmap e773: 12 total, 12 up, 8 in 2024-04-23T16:55:13.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:13 smithi188 ceph-mon[44101]: 9.7 scrub ok 2024-04-23T16:55:13.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:13 smithi188 ceph-mon[44101]: pgmap v2142: 27 pgs: 27 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.24k op/s; 2.8 MiB/s, 44 objects/s recovering 2024-04-23T16:55:14.002 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:55:14.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:14.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:55:14.108 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:55:14.140 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:14.218 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:55:14.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:14.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:55:14.328 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:55:14.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:14 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1870050142' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:55:14.371 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:14.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:14 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1870050142' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:55:14.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:14 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1870050142' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-23T16:55:15.093 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-23T16:55:15.093 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:55:15.317 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:15 smithi103 ceph-mon[39629]: pgmap v2143: 27 pgs: 27 active+clean; 1.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 862 op/s; 2.1 MiB/s, 34 objects/s recovering 2024-04-23T16:55:15.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:15 smithi188 ceph-mon[44101]: pgmap v2143: 27 pgs: 27 active+clean; 1.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 862 op/s; 2.1 MiB/s, 34 objects/s recovering 2024-04-23T16:55:15.539 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:55:15.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:15.547 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:55:15.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:15 smithi148 ceph-mon[44520]: pgmap v2143: 27 pgs: 27 active+clean; 1.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 862 op/s; 2.1 MiB/s, 34 objects/s recovering 2024-04-23T16:55:15.648 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:55:15.666 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:15.718 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:15.739 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:15.757 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:55:15.784 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:55:15.814 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:55:15.839 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:55:15.865 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:55:15.889 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:55:15.915 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:55:15.946 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:55:15.990 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:55:16.014 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:55:16.038 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:55:16.067 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:55:16.086 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:55:16.086 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:55:16.086 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:55:16.086 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:55:16.086 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:55:16.086 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:55:16.086 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:55:16.086 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:55:16.087 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:55:16.087 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:55:16.087 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:55:16.087 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:55:16.565 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:55:16.566 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:55:16.585 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:16 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2622258956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:55:16.585 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:16.586 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:16.586 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:16.586 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:16.586 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:16.586 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:16 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:16.589 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:16 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2622258956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:55:16.589 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:16.590 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:16.590 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:16.590 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:16.590 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:16.590 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:16 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:16.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:16 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2622258956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:55:16.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:16.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:16.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:16.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:16.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:16.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:16 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:17.300 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-23T16:55:17.489 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:55:17.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:17.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:55:17.594 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:55:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:17 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:17.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:17 smithi103 ceph-mon[39629]: pgmap v2144: 27 pgs: 27 active+clean; 1.3 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 885 op/s; 2.1 MiB/s, 33 objects/s recovering 2024-04-23T16:55:17.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:17.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:17.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:17.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:17.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:17.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:17 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:17.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:17 smithi148 ceph-mon[44520]: pgmap v2144: 27 pgs: 27 active+clean; 1.3 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 885 op/s; 2.1 MiB/s, 33 objects/s recovering 2024-04-23T16:55:17.630 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:17.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:17.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:17.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:17.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:17.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:17.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:17 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:17.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:17 smithi188 ceph-mon[44101]: pgmap v2144: 27 pgs: 27 active+clean; 1.3 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 885 op/s; 2.1 MiB/s, 33 objects/s recovering 2024-04-23T16:55:17.738 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:55:18.282 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:18 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:55:18.577 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:18 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:55:18.586 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:55:18.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:18.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:55:18.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:18 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:55:18.692 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:55:18.723 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:18.749 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:55:18.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:18.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:55:18.854 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:55:18.882 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:18.909 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:55:18.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:18.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:55:19.014 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:55:19.072 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:55:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=2977993281,config=230875443,health=3897446751,kv=1830957065}) 2024-04-23T16:55:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346459711}) 2024-04-23T16:55:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2563725561}) 2024-04-23T16:55:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=250184370}) 2024-04-23T16:55:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3939039327}) 2024-04-23T16:55:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1530647887}) 2024-04-23T16:55:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=973967549}) 2024-04-23T16:55:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3975948834}) 2024-04-23T16:55:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2291439121}) 2024-04-23T16:55:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=58651677}) 2024-04-23T16:55:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4035897428}) 2024-04-23T16:55:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=965846664}) 2024-04-23T16:55:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2235881393}) 2024-04-23T16:55:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2463676188}) 2024-04-23T16:55:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1884599868}) 2024-04-23T16:55:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1860323248}) 2024-04-23T16:55:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2194216160}) 2024-04-23T16:55:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=93379503}) 2024-04-23T16:55:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3139929892}) 2024-04-23T16:55:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=112524335}) 2024-04-23T16:55:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1521093066}) 2024-04-23T16:55:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3864162076}) 2024-04-23T16:55:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2117376133}) 2024-04-23T16:55:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3942043629}) 2024-04-23T16:55:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=506688512}) 2024-04-23T16:55:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1637737012}) 2024-04-23T16:55:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=561857522}) 2024-04-23T16:55:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2395432093}) 2024-04-23T16:55:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3579507559}) 2024-04-23T16:55:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3652861335}) 2024-04-23T16:55:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=664716838}) 2024-04-23T16:55:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1245905363}) 2024-04-23T16:55:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=82146705}) 2024-04-23T16:55:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=893385193}) 2024-04-23T16:55:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1311769949}) 2024-04-23T16:55:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2291620748}) 2024-04-23T16:55:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3969556948}) 2024-04-23T16:55:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350592957}) 2024-04-23T16:55:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2220759603}) 2024-04-23T16:55:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2228807271}) 2024-04-23T16:55:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2939587413}) 2024-04-23T16:55:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=170906641}) 2024-04-23T16:55:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=967846093}) 2024-04-23T16:55:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2601836803}) 2024-04-23T16:55:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3555205447}) 2024-04-23T16:55:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=223464242}) 2024-04-23T16:55:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3942785820}) 2024-04-23T16:55:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=904999756}) 2024-04-23T16:55:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4221497428}) 2024-04-23T16:55:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3679419661}) 2024-04-23T16:55:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3203809406}) 2024-04-23T16:55:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2413165765}) 2024-04-23T16:55:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=377780555}) 2024-04-23T16:55:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3256492542}) 2024-04-23T16:55:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2672343966}) 2024-04-23T16:55:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2631417696}) 2024-04-23T16:55:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=120532492}) 2024-04-23T16:55:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3922927175}) 2024-04-23T16:55:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2611118823}) 2024-04-23T16:55:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3380375412}) 2024-04-23T16:55:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2826714168}) 2024-04-23T16:55:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2367578107}) 2024-04-23T16:55:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=904860849}) 2024-04-23T16:55:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193024412}) 2024-04-23T16:55:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1374277148}) 2024-04-23T16:55:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3644671141}) 2024-04-23T16:55:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=221267837}) 2024-04-23T16:55:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=957440690}) 2024-04-23T16:55:19.483 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2971601277}) 2024-04-23T16:55:19.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2574491594}) 2024-04-23T16:55:19.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1804836172}) 2024-04-23T16:55:19.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1904254774}) 2024-04-23T16:55:19.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3472389220}) 2024-04-23T16:55:19.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3104954227}) 2024-04-23T16:55:19.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1049760739}) 2024-04-23T16:55:19.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4284108006}) 2024-04-23T16:55:19.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=184304786}) 2024-04-23T16:55:19.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2359583913}) 2024-04-23T16:55:19.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1549268475}) 2024-04-23T16:55:19.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=734629179}) 2024-04-23T16:55:19.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3430865004}) 2024-04-23T16:55:19.484 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2217200945}) 2024-04-23T16:55:19.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=345830155}) 2024-04-23T16:55:19.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=388127904}) 2024-04-23T16:55:19.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=732105007}) 2024-04-23T16:55:19.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1942192688}) 2024-04-23T16:55:19.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2412520900}) 2024-04-23T16:55:19.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3097654838}) 2024-04-23T16:55:19.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326279967}) 2024-04-23T16:55:19.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2468648299}) 2024-04-23T16:55:19.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3423314648}) 2024-04-23T16:55:19.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3125640079}) 2024-04-23T16:55:19.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2466331514}) 2024-04-23T16:55:19.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4243053342}) 2024-04-23T16:55:19.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3707680786}) 2024-04-23T16:55:19.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1196663414}) 2024-04-23T16:55:19.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=702507963}) 2024-04-23T16:55:19.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3088124237}) 2024-04-23T16:55:19.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091994056}) 2024-04-23T16:55:19.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=52,mdsmap=3,mgr=19,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=8} crc {logm=1783148079,mdsmap=2840952586,mgr=1765502803,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=1140223060,mon_config_key=3691066247}) 2024-04-23T16:55:19.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1135831861}) 2024-04-23T16:55:19.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3187337942}) 2024-04-23T16:55:19.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=85,monmap=5,osd_metadata=10} crc {mon_config_key=4246531204,monmap=2133572193,osd_metadata=2616290189}) 2024-04-23T16:55:19.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=2,osd_pg_creating=1,osdmap=97} crc {osd_metadata=2625464053,osd_pg_creating=3629790670,osdmap=52631800}) 2024-04-23T16:55:19.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=20} crc {osdmap=3344565392}) 2024-04-23T16:55:19.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: pgmap v2145: 27 pgs: 27 active+clean; 1.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 1.9 MiB/s, 30 objects/s recovering 2024-04-23T16:55:19.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:19 smithi188 ceph-mon[44101]: 9.e scrub starts 2024-04-23T16:55:19.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:55:19.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=2977993281,config=230875443,health=3897446751,kv=1830957065}) 2024-04-23T16:55:19.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346459711}) 2024-04-23T16:55:19.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2563725561}) 2024-04-23T16:55:19.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=250184370}) 2024-04-23T16:55:19.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3939039327}) 2024-04-23T16:55:19.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1530647887}) 2024-04-23T16:55:19.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=973967549}) 2024-04-23T16:55:19.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3975948834}) 2024-04-23T16:55:19.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2291439121}) 2024-04-23T16:55:19.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=58651677}) 2024-04-23T16:55:19.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4035897428}) 2024-04-23T16:55:19.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=965846664}) 2024-04-23T16:55:19.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2235881393}) 2024-04-23T16:55:19.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2463676188}) 2024-04-23T16:55:19.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1884599868}) 2024-04-23T16:55:19.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1860323248}) 2024-04-23T16:55:19.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2194216160}) 2024-04-23T16:55:19.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=93379503}) 2024-04-23T16:55:19.601 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3139929892}) 2024-04-23T16:55:19.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=112524335}) 2024-04-23T16:55:19.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1521093066}) 2024-04-23T16:55:19.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3864162076}) 2024-04-23T16:55:19.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2117376133}) 2024-04-23T16:55:19.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3942043629}) 2024-04-23T16:55:19.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=506688512}) 2024-04-23T16:55:19.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1637737012}) 2024-04-23T16:55:19.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=561857522}) 2024-04-23T16:55:19.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2395432093}) 2024-04-23T16:55:19.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3579507559}) 2024-04-23T16:55:19.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3652861335}) 2024-04-23T16:55:19.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=664716838}) 2024-04-23T16:55:19.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1245905363}) 2024-04-23T16:55:19.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=82146705}) 2024-04-23T16:55:19.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=893385193}) 2024-04-23T16:55:19.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1311769949}) 2024-04-23T16:55:19.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2291620748}) 2024-04-23T16:55:19.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3969556948}) 2024-04-23T16:55:19.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350592957}) 2024-04-23T16:55:19.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2220759603}) 2024-04-23T16:55:19.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2228807271}) 2024-04-23T16:55:19.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2939587413}) 2024-04-23T16:55:19.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=170906641}) 2024-04-23T16:55:19.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=967846093}) 2024-04-23T16:55:19.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2601836803}) 2024-04-23T16:55:19.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3555205447}) 2024-04-23T16:55:19.603 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=223464242}) 2024-04-23T16:55:19.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3942785820}) 2024-04-23T16:55:19.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=904999756}) 2024-04-23T16:55:19.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4221497428}) 2024-04-23T16:55:19.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3679419661}) 2024-04-23T16:55:19.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3203809406}) 2024-04-23T16:55:19.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2413165765}) 2024-04-23T16:55:19.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=377780555}) 2024-04-23T16:55:19.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3256492542}) 2024-04-23T16:55:19.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2672343966}) 2024-04-23T16:55:19.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2631417696}) 2024-04-23T16:55:19.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=120532492}) 2024-04-23T16:55:19.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3922927175}) 2024-04-23T16:55:19.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2611118823}) 2024-04-23T16:55:19.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3380375412}) 2024-04-23T16:55:19.604 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2826714168}) 2024-04-23T16:55:19.605 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2367578107}) 2024-04-23T16:55:19.605 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=904860849}) 2024-04-23T16:55:19.605 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193024412}) 2024-04-23T16:55:19.605 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1374277148}) 2024-04-23T16:55:19.605 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3644671141}) 2024-04-23T16:55:19.605 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=221267837}) 2024-04-23T16:55:19.605 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=957440690}) 2024-04-23T16:55:19.605 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2971601277}) 2024-04-23T16:55:19.605 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2574491594}) 2024-04-23T16:55:19.605 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1804836172}) 2024-04-23T16:55:19.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1904254774}) 2024-04-23T16:55:19.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3472389220}) 2024-04-23T16:55:19.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3104954227}) 2024-04-23T16:55:19.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1049760739}) 2024-04-23T16:55:19.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4284108006}) 2024-04-23T16:55:19.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=184304786}) 2024-04-23T16:55:19.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2359583913}) 2024-04-23T16:55:19.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1549268475}) 2024-04-23T16:55:19.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=734629179}) 2024-04-23T16:55:19.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3430865004}) 2024-04-23T16:55:19.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2217200945}) 2024-04-23T16:55:19.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=345830155}) 2024-04-23T16:55:19.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=388127904}) 2024-04-23T16:55:19.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=732105007}) 2024-04-23T16:55:19.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1942192688}) 2024-04-23T16:55:19.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2412520900}) 2024-04-23T16:55:19.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3097654838}) 2024-04-23T16:55:19.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326279967}) 2024-04-23T16:55:19.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2468648299}) 2024-04-23T16:55:19.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3423314648}) 2024-04-23T16:55:19.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3125640079}) 2024-04-23T16:55:19.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2466331514}) 2024-04-23T16:55:19.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4243053342}) 2024-04-23T16:55:19.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3707680786}) 2024-04-23T16:55:19.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1196663414}) 2024-04-23T16:55:19.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=702507963}) 2024-04-23T16:55:19.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3088124237}) 2024-04-23T16:55:19.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091994056}) 2024-04-23T16:55:19.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=52,mdsmap=3,mgr=19,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=8} crc {logm=1783148079,mdsmap=2840952586,mgr=1765502803,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=1140223060,mon_config_key=3691066247}) 2024-04-23T16:55:19.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1135831861}) 2024-04-23T16:55:19.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3187337942}) 2024-04-23T16:55:19.609 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=85,monmap=5,osd_metadata=10} crc {mon_config_key=4246531204,monmap=2133572193,osd_metadata=2616290189}) 2024-04-23T16:55:19.609 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=2,osd_pg_creating=1,osdmap=97} crc {osd_metadata=2625464053,osd_pg_creating=3629790670,osdmap=52631800}) 2024-04-23T16:55:19.609 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=20} crc {osdmap=3344565392}) 2024-04-23T16:55:19.609 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: pgmap v2145: 27 pgs: 27 active+clean; 1.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 1.9 MiB/s, 30 objects/s recovering 2024-04-23T16:55:19.609 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:19 smithi103 ceph-mon[39629]: 9.e scrub starts 2024-04-23T16:55:19.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:55:19.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=2977993281,config=230875443,health=3897446751,kv=1830957065}) 2024-04-23T16:55:19.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346459711}) 2024-04-23T16:55:19.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2563725561}) 2024-04-23T16:55:19.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=250184370}) 2024-04-23T16:55:19.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3939039327}) 2024-04-23T16:55:19.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1530647887}) 2024-04-23T16:55:19.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=973967549}) 2024-04-23T16:55:19.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3975948834}) 2024-04-23T16:55:19.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2291439121}) 2024-04-23T16:55:19.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=58651677}) 2024-04-23T16:55:19.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4035897428}) 2024-04-23T16:55:19.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=965846664}) 2024-04-23T16:55:19.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2235881393}) 2024-04-23T16:55:19.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2463676188}) 2024-04-23T16:55:19.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1884599868}) 2024-04-23T16:55:19.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1860323248}) 2024-04-23T16:55:19.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2194216160}) 2024-04-23T16:55:19.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=93379503}) 2024-04-23T16:55:19.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3139929892}) 2024-04-23T16:55:19.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=112524335}) 2024-04-23T16:55:19.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1521093066}) 2024-04-23T16:55:19.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3864162076}) 2024-04-23T16:55:19.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2117376133}) 2024-04-23T16:55:19.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3942043629}) 2024-04-23T16:55:19.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=506688512}) 2024-04-23T16:55:19.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1637737012}) 2024-04-23T16:55:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=561857522}) 2024-04-23T16:55:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2395432093}) 2024-04-23T16:55:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3579507559}) 2024-04-23T16:55:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3652861335}) 2024-04-23T16:55:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=664716838}) 2024-04-23T16:55:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1245905363}) 2024-04-23T16:55:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=82146705}) 2024-04-23T16:55:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=893385193}) 2024-04-23T16:55:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1311769949}) 2024-04-23T16:55:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2291620748}) 2024-04-23T16:55:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3969556948}) 2024-04-23T16:55:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350592957}) 2024-04-23T16:55:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2220759603}) 2024-04-23T16:55:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2228807271}) 2024-04-23T16:55:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2939587413}) 2024-04-23T16:55:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=170906641}) 2024-04-23T16:55:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=967846093}) 2024-04-23T16:55:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2601836803}) 2024-04-23T16:55:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3555205447}) 2024-04-23T16:55:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=223464242}) 2024-04-23T16:55:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3942785820}) 2024-04-23T16:55:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=904999756}) 2024-04-23T16:55:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4221497428}) 2024-04-23T16:55:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3679419661}) 2024-04-23T16:55:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3203809406}) 2024-04-23T16:55:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2413165765}) 2024-04-23T16:55:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=377780555}) 2024-04-23T16:55:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3256492542}) 2024-04-23T16:55:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2672343966}) 2024-04-23T16:55:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2631417696}) 2024-04-23T16:55:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=120532492}) 2024-04-23T16:55:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3922927175}) 2024-04-23T16:55:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2611118823}) 2024-04-23T16:55:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3380375412}) 2024-04-23T16:55:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2826714168}) 2024-04-23T16:55:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2367578107}) 2024-04-23T16:55:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=904860849}) 2024-04-23T16:55:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193024412}) 2024-04-23T16:55:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1374277148}) 2024-04-23T16:55:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3644671141}) 2024-04-23T16:55:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=221267837}) 2024-04-23T16:55:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=957440690}) 2024-04-23T16:55:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2971601277}) 2024-04-23T16:55:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2574491594}) 2024-04-23T16:55:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1804836172}) 2024-04-23T16:55:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1904254774}) 2024-04-23T16:55:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3472389220}) 2024-04-23T16:55:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3104954227}) 2024-04-23T16:55:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1049760739}) 2024-04-23T16:55:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4284108006}) 2024-04-23T16:55:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=184304786}) 2024-04-23T16:55:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2359583913}) 2024-04-23T16:55:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1549268475}) 2024-04-23T16:55:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=734629179}) 2024-04-23T16:55:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3430865004}) 2024-04-23T16:55:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2217200945}) 2024-04-23T16:55:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=345830155}) 2024-04-23T16:55:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=388127904}) 2024-04-23T16:55:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=732105007}) 2024-04-23T16:55:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1942192688}) 2024-04-23T16:55:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2412520900}) 2024-04-23T16:55:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3097654838}) 2024-04-23T16:55:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326279967}) 2024-04-23T16:55:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2468648299}) 2024-04-23T16:55:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3423314648}) 2024-04-23T16:55:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3125640079}) 2024-04-23T16:55:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2466331514}) 2024-04-23T16:55:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4243053342}) 2024-04-23T16:55:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3707680786}) 2024-04-23T16:55:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1196663414}) 2024-04-23T16:55:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=702507963}) 2024-04-23T16:55:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3088124237}) 2024-04-23T16:55:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091994056}) 2024-04-23T16:55:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=52,mdsmap=3,mgr=19,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=8} crc {logm=1783148079,mdsmap=2840952586,mgr=1765502803,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=1140223060,mon_config_key=3691066247}) 2024-04-23T16:55:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1135831861}) 2024-04-23T16:55:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3187337942}) 2024-04-23T16:55:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=85,monmap=5,osd_metadata=10} crc {mon_config_key=4246531204,monmap=2133572193,osd_metadata=2616290189}) 2024-04-23T16:55:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=2,osd_pg_creating=1,osdmap=97} crc {osd_metadata=2625464053,osd_pg_creating=3629790670,osdmap=52631800}) 2024-04-23T16:55:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=20} crc {osdmap=3344565392}) 2024-04-23T16:55:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: pgmap v2145: 27 pgs: 27 active+clean; 1.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 1.9 MiB/s, 30 objects/s recovering 2024-04-23T16:55:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:19 smithi148 ceph-mon[44520]: 9.e scrub starts 2024-04-23T16:55:20.089 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:55:20.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:20.099 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:55:20.200 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:55:20.239 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:20.366 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:55:20.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:20.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:55:20.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:20 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:55:20.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:20 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:55:20.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:20 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:55:20.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:20 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:55:20.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:20 smithi188 ceph-mon[44101]: 9.e scrub ok 2024-04-23T16:55:20.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:20 smithi188 ceph-mon[44101]: osdmap e774: 12 total, 12 up, 8 in 2024-04-23T16:55:20.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:20 smithi188 ceph-mon[44101]: 1.9 scrub starts 2024-04-23T16:55:20.456 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:20 smithi188 ceph-mon[44101]: 1.9 scrub ok 2024-04-23T16:55:20.470 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:55:20.524 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:20 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:55:20.524 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:20 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:55:20.524 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:20 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:55:20.524 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:20 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:55:20.524 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:20 smithi103 ceph-mon[39629]: 9.e scrub ok 2024-04-23T16:55:20.525 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:20 smithi103 ceph-mon[39629]: osdmap e774: 12 total, 12 up, 8 in 2024-04-23T16:55:20.525 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:20 smithi103 ceph-mon[39629]: 1.9 scrub starts 2024-04-23T16:55:20.525 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:20 smithi103 ceph-mon[39629]: 1.9 scrub ok 2024-04-23T16:55:20.526 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:20.583 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:55:20.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:20.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:55:20.616 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:55:20.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:20 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:55:20.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:20 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:55:20.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:20 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:55:20.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:20 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:55:20.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:20 smithi148 ceph-mon[44520]: 9.e scrub ok 2024-04-23T16:55:20.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:20 smithi148 ceph-mon[44520]: osdmap e774: 12 total, 12 up, 8 in 2024-04-23T16:55:20.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:20 smithi148 ceph-mon[44520]: 1.9 scrub starts 2024-04-23T16:55:20.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:20 smithi148 ceph-mon[44520]: 1.9 scrub ok 2024-04-23T16:55:20.687 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:55:20.714 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:20.739 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:55:20.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:20.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:55:20.844 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:55:20.893 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:20.930 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:55:20.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:20.938 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:55:20.970 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:55:20.984 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:55:21.014 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:55:21.014 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:55:21.035 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:21.039 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:55:21.079 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:21.296 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:21 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:55:21.296 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:21 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:55:21.296 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:21 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1668158655' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:55:21.297 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:21 smithi103 ceph-mon[39629]: from='client.37250 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:55:21.297 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:21 smithi103 ceph-mon[39629]: from='client.17784 v1:172.21.15.103:0/435551213' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:55:21.297 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:21 smithi103 ceph-mon[39629]: pgmap v2147: 27 pgs: 27 active+clean; 1.0 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 878 op/s 2024-04-23T16:55:21.307 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is set 2024-04-23T16:55:21.474 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:55:21.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:21.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:55:21.586 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:55:21.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:21 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:55:21.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:21 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:55:21.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:21 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1668158655' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:55:21.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:21 smithi148 ceph-mon[44520]: from='client.37250 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:55:21.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:21 smithi148 ceph-mon[44520]: from='client.17784 v1:172.21.15.103:0/435551213' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:55:21.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:21 smithi148 ceph-mon[44520]: pgmap v2147: 27 pgs: 27 active+clean; 1.0 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 878 op/s 2024-04-23T16:55:21.626 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:21.732 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:21 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:55:21.732 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:21 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:55:21.732 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:21 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1668158655' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:55:21.732 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:21 smithi188 ceph-mon[44101]: from='client.37250 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-23T16:55:21.732 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:21 smithi188 ceph-mon[44101]: from='client.17784 v1:172.21.15.103:0/435551213' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:55:21.732 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:21 smithi188 ceph-mon[44101]: pgmap v2147: 27 pgs: 27 active+clean; 1.0 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 878 op/s 2024-04-23T16:55:21.838 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-23T16:55:22.279 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:55:22.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:22 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:55:22.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:22 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:55:22.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:22 smithi103 ceph-mon[39629]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:55:22.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:22 smithi103 ceph-mon[39629]: from='client.37250 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:55:22.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:22 smithi103 ceph-mon[39629]: osdmap e775: 12 total, 12 up, 8 in 2024-04-23T16:55:22.555 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:22 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:55:22.555 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:22 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:55:22.555 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:22 smithi188 ceph-mon[44101]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:55:22.555 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:22 smithi188 ceph-mon[44101]: from='client.37250 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:55:22.556 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:22 smithi188 ceph-mon[44101]: osdmap e775: 12 total, 12 up, 8 in 2024-04-23T16:55:22.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:22 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:55:22.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:22 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:55:22.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:22 smithi148 ceph-mon[44520]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:55:22.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:22 smithi148 ceph-mon[44520]: from='client.37250 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-23T16:55:22.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:22 smithi148 ceph-mon[44520]: osdmap e775: 12 total, 12 up, 8 in 2024-04-23T16:55:22.684 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:55:22.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:22.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:55:22.793 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:55:22.835 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:23.200 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:55:23.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:23.203 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:55:23.305 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:55:23.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:23 smithi103 ceph-mon[39629]: pgmap v2149: 27 pgs: 27 active+clean; 869 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-23T16:55:23.336 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:23.462 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:55:23.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:23.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:55:23.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:23 smithi188 ceph-mon[44101]: pgmap v2149: 27 pgs: 27 active+clean; 869 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-23T16:55:23.572 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:55:23.605 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:23.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:23 smithi148 ceph-mon[44520]: pgmap v2149: 27 pgs: 27 active+clean; 869 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-23T16:55:24.060 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:55:24.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:24.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:55:24.165 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:55:24.200 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:24.492 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:55:24.492 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:55:24.495 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:55:24.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:24.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:55:24.527 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":2146,"stamp":"2024-04-23T16:55:19.286266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1177027128,"num_objects":17954,"num_object_clones":0,"num_object_copies":35908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":284310,"num_write_kb":9672321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8587,"num_bytes_recovered":563164768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162153,"ondisk_log_size":162153,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7674648,"kb_used_data":2359256,"kb_used_omap":522,"kb_used_meta":5314101,"kb_avail":742286568,"statfs":{"total":767960285184,"available":760101445632,"internally_reserved":0,"allocated":2415878144,"data_stored":2412862768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":534579,"internal_metadata":5441640397},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,8,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-610664448,"num_objects":-9318,"num_object_clones":0,"num_object_copies":-18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-269,"num_objects_unfound":0,"num_objects_dirty":-9318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9318,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":269,"num_bytes_recovered":17629184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.279382"},"pg_stats":[{"pgid":"9.7","version":"773'17918","reported_seq":35265,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:16.305121+0000","last_change":"2024-04-23T16:55:12.394650+0000","last_active":"2024-04-23T16:55:16.305121+0000","last_peered":"2024-04-23T16:55:16.305121+0000","last_clean":"2024-04-23T16:55:16.305121+0000","last_became_active":"2024-04-23T16:52:44.683405+0000","last_became_peered":"2024-04-23T16:52:44.683405+0000","last_unstale":"2024-04-23T16:55:16.305121+0000","last_undegraded":"2024-04-23T16:55:16.305121+0000","last_fullsized":"2024-04-23T16:55:16.305121+0000","mapping_epoch":732,"log_start":"729'7900","ondisk_log_start":"729'7900","created":703,"last_epoch_clean":771,"parent":"0.0","parent_split_bits":0,"last_scrub":"773'17677","last_scrub_stamp":"2024-04-23T16:55:12.394532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:55:12.394532+0000","objects_scrubbed":1399,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:13.017396+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":1152,"num_object_clones":0,"num_object_copies":2304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17918,"num_write_kb":610240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8584,"num_bytes_recovered":562561024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"773'17387","reported_seq":17651,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:17.063095+0000","last_change":"2024-04-23T16:55:01.151246+0000","last_active":"2024-04-23T16:55:17.063095+0000","last_peered":"2024-04-23T16:55:17.063095+0000","last_clean":"2024-04-23T16:55:17.063095+0000","last_became_active":"2024-04-23T16:51:21.063642+0000","last_became_peered":"2024-04-23T16:51:21.063642+0000","last_unstale":"2024-04-23T16:55:17.063095+0000","last_undegraded":"2024-04-23T16:55:17.063095+0000","last_fullsized":"2024-04-23T16:55:17.063095+0000","mapping_epoch":703,"log_start":"727'7300","ondisk_log_start":"727'7300","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'16423","last_scrub_stamp":"2024-04-23T16:55:01.151049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:55:01.151049+0000","objects_scrubbed":2082,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:02.623292+0000","scrub_duration":380,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72417280,"num_objects":1105,"num_object_clones":0,"num_object_copies":2210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17387,"num_write_kb":591744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"773'17662","reported_seq":17953,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:17.818369+0000","last_change":"2024-04-23T16:54:21.272785+0000","last_active":"2024-04-23T16:55:17.818369+0000","last_peered":"2024-04-23T16:55:17.818369+0000","last_clean":"2024-04-23T16:55:17.818369+0000","last_became_active":"2024-04-23T16:51:21.063776+0000","last_became_peered":"2024-04-23T16:51:21.063776+0000","last_unstale":"2024-04-23T16:55:17.818369+0000","last_undegraded":"2024-04-23T16:55:17.818369+0000","last_fullsized":"2024-04-23T16:55:17.818369+0000","mapping_epoch":703,"log_start":"729'7600","ondisk_log_start":"729'7600","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14080","last_scrub_stamp":"2024-04-23T16:54:21.272505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:21.272505+0000","objects_scrubbed":4691,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:50.192674+0000","scrub_duration":969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70647808,"num_objects":1078,"num_object_clones":0,"num_object_copies":2156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17662,"num_write_kb":599680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"773'17706","reported_seq":17981,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:17.075231+0000","last_change":"2024-04-23T16:55:00.296822+0000","last_active":"2024-04-23T16:55:17.075231+0000","last_peered":"2024-04-23T16:55:17.075231+0000","last_clean":"2024-04-23T16:55:17.075231+0000","last_became_active":"2024-04-23T16:51:21.063114+0000","last_became_peered":"2024-04-23T16:51:21.063114+0000","last_unstale":"2024-04-23T16:55:17.075231+0000","last_undegraded":"2024-04-23T16:55:17.075231+0000","last_fullsized":"2024-04-23T16:55:17.075231+0000","mapping_epoch":703,"log_start":"729'7700","ondisk_log_start":"729'7700","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'16616","last_scrub_stamp":"2024-04-23T16:55:00.296670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:55:00.296670+0000","objects_scrubbed":2221,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:18.149351+0000","scrub_duration":479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73793536,"num_objects":1126,"num_object_clones":0,"num_object_copies":2252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17706,"num_write_kb":602624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"773'17925","reported_seq":18203,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:18.420921+0000","last_change":"2024-04-23T16:55:01.549355+0000","last_active":"2024-04-23T16:55:18.420921+0000","last_peered":"2024-04-23T16:55:18.420921+0000","last_clean":"2024-04-23T16:55:18.420921+0000","last_became_active":"2024-04-23T16:51:21.063652+0000","last_became_peered":"2024-04-23T16:51:21.063652+0000","last_unstale":"2024-04-23T16:55:18.420921+0000","last_undegraded":"2024-04-23T16:55:18.420921+0000","last_fullsized":"2024-04-23T16:55:18.420921+0000","mapping_epoch":703,"log_start":"729'7900","ondisk_log_start":"729'7900","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'16831","last_scrub_stamp":"2024-04-23T16:55:01.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:55:01.549165+0000","objects_scrubbed":2128,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:27.145270+0000","scrub_duration":536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66912256,"num_objects":1021,"num_object_clones":0,"num_object_copies":2042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17925,"num_write_kb":606272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.2","version":"773'18114","reported_seq":18408,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:18.372216+0000","last_change":"2024-04-23T16:54:24.962245+0000","last_active":"2024-04-23T16:55:18.372216+0000","last_peered":"2024-04-23T16:55:18.372216+0000","last_clean":"2024-04-23T16:55:18.372216+0000","last_became_active":"2024-04-23T16:51:21.062561+0000","last_became_peered":"2024-04-23T16:51:21.062561+0000","last_unstale":"2024-04-23T16:55:18.372216+0000","last_undegraded":"2024-04-23T16:55:18.372216+0000","last_fullsized":"2024-04-23T16:55:18.372216+0000","mapping_epoch":703,"log_start":"730'8100","ondisk_log_start":"730'8100","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14622","last_scrub_stamp":"2024-04-23T16:54:24.962060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:24.962060+0000","objects_scrubbed":4531,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:49.051124+0000","scrub_duration":1162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65667072,"num_objects":1002,"num_object_clones":0,"num_object_copies":2004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18114,"num_write_kb":611712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1903,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:12.243432+0000","last_change":"2024-04-23T16:54:28.362501+0000","last_active":"2024-04-23T16:55:12.243432+0000","last_peered":"2024-04-23T16:55:12.243432+0000","last_clean":"2024-04-23T16:55:12.243432+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:55:12.243432+0000","last_undegraded":"2024-04-23T16:55:12.243432+0000","last_fullsized":"2024-04-23T16:55:12.243432+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:54:28.362406+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:54:28.362406+0000","last_clean_scrub_stamp":"2024-04-23T16:54:28.362406+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:28.652451+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"773'17980","reported_seq":18277,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:17.068385+0000","last_change":"2024-04-23T16:54:31.665323+0000","last_active":"2024-04-23T16:55:17.068385+0000","last_peered":"2024-04-23T16:55:17.068385+0000","last_clean":"2024-04-23T16:55:17.068385+0000","last_became_active":"2024-04-23T16:51:21.063395+0000","last_became_peered":"2024-04-23T16:51:21.063395+0000","last_unstale":"2024-04-23T16:55:17.068385+0000","last_undegraded":"2024-04-23T16:55:17.068385+0000","last_fullsized":"2024-04-23T16:55:17.068385+0000","mapping_epoch":703,"log_start":"729'7900","ondisk_log_start":"729'7900","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"763'15015","last_scrub_stamp":"2024-04-23T16:54:31.665200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:31.665200+0000","objects_scrubbed":4086,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:39.674179+0000","scrub_duration":876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71958528,"num_objects":1098,"num_object_clones":0,"num_object_copies":2196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17980,"num_write_kb":610496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1685,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:12.229490+0000","last_change":"2024-04-23T16:55:10.720904+0000","last_active":"2024-04-23T16:55:12.229490+0000","last_peered":"2024-04-23T16:55:12.229490+0000","last_clean":"2024-04-23T16:55:12.229490+0000","last_became_active":"2024-04-23T16:52:47.483615+0000","last_became_peered":"2024-04-23T16:52:47.483615+0000","last_unstale":"2024-04-23T16:55:12.229490+0000","last_undegraded":"2024-04-23T16:55:12.229490+0000","last_fullsized":"2024-04-23T16:55:12.229490+0000","mapping_epoch":735,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":736,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:10.720770+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:52:05.504325+0000","last_clean_scrub_stamp":"2024-04-23T16:55:10.720770+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:23.595719+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"773'17246","reported_seq":17536,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:15.083272+0000","last_change":"2024-04-23T16:54:17.136201+0000","last_active":"2024-04-23T16:55:15.083272+0000","last_peered":"2024-04-23T16:55:15.083272+0000","last_clean":"2024-04-23T16:55:15.083272+0000","last_became_active":"2024-04-23T16:51:21.246391+0000","last_became_peered":"2024-04-23T16:51:21.246391+0000","last_unstale":"2024-04-23T16:55:15.083272+0000","last_undegraded":"2024-04-23T16:55:15.083272+0000","last_fullsized":"2024-04-23T16:55:15.083272+0000","mapping_epoch":703,"log_start":"727'7200","ondisk_log_start":"727'7200","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'13668","last_scrub_stamp":"2024-04-23T16:54:17.136038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:17.136038+0000","objects_scrubbed":4818,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:23.005433+0000","scrub_duration":963,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79167488,"num_objects":1208,"num_object_clones":0,"num_object_copies":2416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17246,"num_write_kb":590528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1887,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:12.231143+0000","last_change":"2024-04-23T16:52:38.199979+0000","last_active":"2024-04-23T16:55:12.231143+0000","last_peered":"2024-04-23T16:55:12.231143+0000","last_clean":"2024-04-23T16:55:12.231143+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:55:12.231143+0000","last_undegraded":"2024-04-23T16:55:12.231143+0000","last_fullsized":"2024-04-23T16:55:12.231143+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:52:38.199895+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:51:25.290774+0000","last_clean_scrub_stamp":"2024-04-23T16:52:38.199895+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"773'17921","reported_seq":18218,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:17.068508+0000","last_change":"2024-04-23T16:54:18.910760+0000","last_active":"2024-04-23T16:55:17.068508+0000","last_peered":"2024-04-23T16:55:17.068508+0000","last_clean":"2024-04-23T16:55:17.068508+0000","last_became_active":"2024-04-23T16:51:21.063669+0000","last_became_peered":"2024-04-23T16:51:21.063669+0000","last_unstale":"2024-04-23T16:55:17.068508+0000","last_undegraded":"2024-04-23T16:55:17.068508+0000","last_fullsized":"2024-04-23T16:55:17.068508+0000","mapping_epoch":703,"log_start":"729'7900","ondisk_log_start":"729'7900","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14143","last_scrub_stamp":"2024-04-23T16:54:18.910481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:18.910481+0000","objects_scrubbed":4947,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:29.963181+0000","scrub_duration":1208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74383360,"num_objects":1135,"num_object_clones":0,"num_object_copies":2270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17921,"num_write_kb":609792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1909,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:12.289423+0000","last_change":"2024-04-23T16:55:00.540871+0000","last_active":"2024-04-23T16:55:12.289423+0000","last_peered":"2024-04-23T16:55:12.289423+0000","last_clean":"2024-04-23T16:55:12.289423+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:55:12.289423+0000","last_undegraded":"2024-04-23T16:55:12.289423+0000","last_fullsized":"2024-04-23T16:55:12.289423+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:00.540749+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:55:00.540749+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:15.651892+0000","scrub_duration":229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"773'18082","reported_seq":18378,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:18.418963+0000","last_change":"2024-04-23T16:54:19.829779+0000","last_active":"2024-04-23T16:55:18.418963+0000","last_peered":"2024-04-23T16:55:18.418963+0000","last_clean":"2024-04-23T16:55:18.418963+0000","last_became_active":"2024-04-23T16:51:21.062760+0000","last_became_peered":"2024-04-23T16:51:21.062760+0000","last_unstale":"2024-04-23T16:55:18.418963+0000","last_undegraded":"2024-04-23T16:55:18.418963+0000","last_fullsized":"2024-04-23T16:55:18.418963+0000","mapping_epoch":703,"log_start":"729'8000","ondisk_log_start":"729'8000","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14356","last_scrub_stamp":"2024-04-23T16:54:19.829518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:19.829518+0000","objects_scrubbed":4782,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:22.348945+0000","scrub_duration":1029,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67633152,"num_objects":1032,"num_object_clones":0,"num_object_copies":2064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18082,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1493,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:12.236757+0000","last_change":"2024-04-23T16:55:02.927584+0000","last_active":"2024-04-23T16:55:12.236757+0000","last_peered":"2024-04-23T16:55:12.236757+0000","last_clean":"2024-04-23T16:55:12.236757+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:55:12.236757+0000","last_undegraded":"2024-04-23T16:55:12.236757+0000","last_fullsized":"2024-04-23T16:55:12.236757+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:02.927497+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:55:02.927497+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:29.690298+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"773'17741","reported_seq":18038,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:17.832273+0000","last_change":"2024-04-23T16:54:19.146832+0000","last_active":"2024-04-23T16:55:17.832273+0000","last_peered":"2024-04-23T16:55:17.832273+0000","last_clean":"2024-04-23T16:55:17.832273+0000","last_became_active":"2024-04-23T16:51:21.246519+0000","last_became_peered":"2024-04-23T16:51:21.246519+0000","last_unstale":"2024-04-23T16:55:17.832273+0000","last_undegraded":"2024-04-23T16:55:17.832273+0000","last_fullsized":"2024-04-23T16:55:17.832273+0000","mapping_epoch":703,"log_start":"729'7700","ondisk_log_start":"729'7700","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'13985","last_scrub_stamp":"2024-04-23T16:54:19.146171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:19.146171+0000","objects_scrubbed":4796,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:28.380772+0000","scrub_duration":863,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66387968,"num_objects":1013,"num_object_clones":0,"num_object_copies":2026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17741,"num_write_kb":600128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1454,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:12.245719+0000","last_change":"2024-04-23T16:55:03.259007+0000","last_active":"2024-04-23T16:55:12.245719+0000","last_peered":"2024-04-23T16:55:12.245719+0000","last_clean":"2024-04-23T16:55:12.245719+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:55:12.245719+0000","last_undegraded":"2024-04-23T16:55:12.245719+0000","last_fullsized":"2024-04-23T16:55:12.245719+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:03.258896+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:55:03.258896+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:32.586872+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.e","version":"773'17771","reported_seq":18102,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:16.338043+0000","last_change":"2024-04-23T16:52:41.066866+0000","last_active":"2024-04-23T16:55:16.338043+0000","last_peered":"2024-04-23T16:55:16.338043+0000","last_clean":"2024-04-23T16:55:16.338043+0000","last_became_active":"2024-04-23T16:51:21.064486+0000","last_became_peered":"2024-04-23T16:51:21.064486+0000","last_unstale":"2024-04-23T16:55:16.338043+0000","last_undegraded":"2024-04-23T16:55:16.338043+0000","last_fullsized":"2024-04-23T16:55:16.338043+0000","mapping_epoch":703,"log_start":"729'7700","ondisk_log_start":"729'7700","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"730'8312","last_scrub_stamp":"2024-04-23T16:52:41.066729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:52:41.066729+0000","objects_scrubbed":8276,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79101952,"num_objects":1207,"num_object_clones":0,"num_object_copies":2414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17771,"num_write_kb":607296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1911,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:12.243518+0000","last_change":"2024-04-23T16:54:26.409871+0000","last_active":"2024-04-23T16:55:12.243518+0000","last_peered":"2024-04-23T16:55:12.243518+0000","last_clean":"2024-04-23T16:55:12.243518+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:55:12.243518+0000","last_undegraded":"2024-04-23T16:55:12.243518+0000","last_fullsized":"2024-04-23T16:55:12.243518+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:54:26.409705+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:54:26.409705+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:33.984886+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.f","version":"773'18017","reported_seq":18315,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:17.085144+0000","last_change":"2024-04-23T16:54:17.546594+0000","last_active":"2024-04-23T16:55:17.085144+0000","last_peered":"2024-04-23T16:55:17.085144+0000","last_clean":"2024-04-23T16:55:17.085144+0000","last_became_active":"2024-04-23T16:51:21.063588+0000","last_became_peered":"2024-04-23T16:51:21.063588+0000","last_unstale":"2024-04-23T16:55:17.085144+0000","last_undegraded":"2024-04-23T16:55:17.085144+0000","last_fullsized":"2024-04-23T16:55:17.085144+0000","mapping_epoch":703,"log_start":"730'8000","ondisk_log_start":"730'8000","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14153","last_scrub_stamp":"2024-04-23T16:54:17.546419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:17.546419+0000","objects_scrubbed":4978,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:23.570053+0000","scrub_duration":812,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71761920,"num_objects":1095,"num_object_clones":0,"num_object_copies":2190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18017,"num_write_kb":611584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1922,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:12.246049+0000","last_change":"2024-04-23T16:54:55.936994+0000","last_active":"2024-04-23T16:55:12.246049+0000","last_peered":"2024-04-23T16:55:12.246049+0000","last_clean":"2024-04-23T16:55:12.246049+0000","last_became_active":"2024-04-23T16:52:44.474580+0000","last_became_peered":"2024-04-23T16:52:44.474580+0000","last_unstale":"2024-04-23T16:55:12.246049+0000","last_undegraded":"2024-04-23T16:55:12.246049+0000","last_fullsized":"2024-04-23T16:55:12.246049+0000","mapping_epoch":732,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":733,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:54:55.936863+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:54:55.936863+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:10.224576+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"773'17714","reported_seq":18064,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:15.068493+0000","last_change":"2024-04-23T16:54:56.611840+0000","last_active":"2024-04-23T16:55:15.068493+0000","last_peered":"2024-04-23T16:55:15.068493+0000","last_clean":"2024-04-23T16:55:15.068493+0000","last_became_active":"2024-04-23T16:51:21.079798+0000","last_became_peered":"2024-04-23T16:51:21.079798+0000","last_unstale":"2024-04-23T16:55:15.068493+0000","last_undegraded":"2024-04-23T16:55:15.068493+0000","last_fullsized":"2024-04-23T16:55:15.068493+0000","mapping_epoch":703,"log_start":"729'7700","ondisk_log_start":"729'7700","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"768'16627","last_scrub_stamp":"2024-04-23T16:54:56.611621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:56.611621+0000","objects_scrubbed":2377,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:07.253674+0000","scrub_duration":457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":1280,"num_object_clones":0,"num_object_copies":2560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17714,"num_write_kb":607808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1795,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:12.234542+0000","last_change":"2024-04-23T16:54:32.619720+0000","last_active":"2024-04-23T16:55:12.234542+0000","last_peered":"2024-04-23T16:55:12.234542+0000","last_clean":"2024-04-23T16:55:12.234542+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:55:12.234542+0000","last_undegraded":"2024-04-23T16:55:12.234542+0000","last_fullsized":"2024-04-23T16:55:12.234542+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:54:32.619503+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:54:32.619503+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:40.217222+0000","scrub_duration":309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"773'17423","reported_seq":17664,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:16.307866+0000","last_change":"2024-04-23T16:52:43.250377+0000","last_active":"2024-04-23T16:55:16.307866+0000","last_peered":"2024-04-23T16:55:16.307866+0000","last_clean":"2024-04-23T16:55:16.307866+0000","last_became_active":"2024-04-23T16:51:21.064506+0000","last_became_peered":"2024-04-23T16:51:21.064506+0000","last_unstale":"2024-04-23T16:55:16.307866+0000","last_undegraded":"2024-04-23T16:55:16.307866+0000","last_fullsized":"2024-04-23T16:55:16.307866+0000","mapping_epoch":703,"log_start":"728'7400","ondisk_log_start":"728'7400","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"731'8308","last_scrub_stamp":"2024-04-23T16:52:43.250180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:52:43.250180+0000","objects_scrubbed":8271,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75759640,"num_objects":1157,"num_object_clones":0,"num_object_copies":2314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17423,"num_write_kb":594497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1728,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:12.229489+0000","last_change":"2024-04-23T16:55:11.687295+0000","last_active":"2024-04-23T16:55:12.229489+0000","last_peered":"2024-04-23T16:55:12.229489+0000","last_clean":"2024-04-23T16:55:12.229489+0000","last_became_active":"2024-04-23T16:52:47.480630+0000","last_became_peered":"2024-04-23T16:52:47.480630+0000","last_unstale":"2024-04-23T16:55:12.229489+0000","last_undegraded":"2024-04-23T16:55:12.229489+0000","last_fullsized":"2024-04-23T16:55:12.229489+0000","mapping_epoch":735,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":736,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:11.687173+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:55:11.687173+0000","last_clean_scrub_stamp":"2024-04-23T16:55:11.687173+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:24.332368+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"773'19719","reported_seq":20096,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:15.112903+0000","last_change":"2024-04-23T16:54:24.434938+0000","last_active":"2024-04-23T16:55:15.112903+0000","last_peered":"2024-04-23T16:55:15.112903+0000","last_clean":"2024-04-23T16:55:15.112903+0000","last_became_active":"2024-04-23T16:51:21.063420+0000","last_became_peered":"2024-04-23T16:51:21.063420+0000","last_unstale":"2024-04-23T16:55:15.112903+0000","last_undegraded":"2024-04-23T16:55:15.112903+0000","last_fullsized":"2024-04-23T16:55:15.112903+0000","mapping_epoch":703,"log_start":"740'9702","ondisk_log_start":"740'9702","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'16466","last_scrub_stamp":"2024-04-23T16:54:24.434705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:24.434705+0000","objects_scrubbed":4533,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:43.605780+0000","scrub_duration":1184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81461248,"num_objects":1243,"num_object_clones":0,"num_object_copies":2486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17703,"num_write_kb":606272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1879,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-04-23T16:55:12.231143+0000","last_change":"2024-04-23T16:51:31.339866+0000","last_active":"2024-04-23T16:55:12.231143+0000","last_peered":"2024-04-23T16:55:12.231143+0000","last_clean":"2024-04-23T16:55:12.231143+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:55:12.231143+0000","last_undegraded":"2024-04-23T16:55:12.231143+0000","last_fullsized":"2024-04-23T16:55:12.231143+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:51:31.339728+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:51:31.339728+0000","last_clean_scrub_stamp":"2024-04-23T16:51:31.339728+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1176436760,"num_objects":17952,"num_object_clones":0,"num_object_copies":35904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17952,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":284310,"num_write_kb":9672321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8584,"num_bytes_recovered":562561024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2348425216,"data_stored":2348417072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160624,"ondisk_log_size":160624,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1679360,"data_stored":1661600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057120,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1363416,"kb_used_data":277072,"kb_used_omap":1,"kb_used_meta":1086270,"kb_avail":92381736,"statfs":{"total":95995035648,"available":94598897664,"internally_reserved":0,"allocated":283721728,"data_stored":283346711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1112340932},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351462,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613099,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1323480,"kb_used_data":497744,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92421672,"statfs":{"total":95995035648,"available":94639792128,"internally_reserved":0,"allocated":509689856,"data_stored":509314839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":845085120},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":66,"seq":283467842032,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":935456,"kb_used_data":340440,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92809696,"statfs":{"total":95995035648,"available":95037128704,"internally_reserved":0,"allocated":348610560,"data_stored":348231991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169078,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1043740,"kb_used_data":304532,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":92701412,"statfs":{"total":95995035648,"available":94926245888,"internally_reserved":0,"allocated":311840768,"data_stored":311461679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":756939210},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463418,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":748440,"kb_used_data":153616,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92996712,"statfs":{"total":95995035648,"available":95228633088,"internally_reserved":0,"allocated":157302784,"data_stored":156927767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":609089990},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137559,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1265308,"kb_used_data":647380,"kb_used_omap":231,"kb_used_meta":617688,"kb_avail":92479844,"statfs":{"total":95995035648,"available":94699360256,"internally_reserved":0,"allocated":662917120,"data_stored":662538031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":632513424},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738905,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":830744,"kb_used_data":69328,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92914408,"statfs":{"total":95995035648,"available":95144353792,"internally_reserved":0,"allocated":70991872,"data_stored":70616855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":779680197},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":481,"seq":2065879269575,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":164064,"kb_used_data":69144,"kb_used_omap":281,"kb_used_meta":94630,"kb_avail":93581088,"statfs":{"total":95995035648,"available":95827034112,"internally_reserved":0,"allocated":70803456,"data_stored":70424895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":288353,"internal_metadata":96901535},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2658584756325,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":3126736191524,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757758,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":66912256,"data_stored":66912256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":655364096,"data_stored":655360024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":67633152,"data_stored":67633152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":151126016,"data_stored":151126016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":307040256,"data_stored":307036184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":338362368,"data_stored":338362368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":485097472,"data_stored":485097472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":276889600,"data_stored":276889600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:55:24.600 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:55:24.666 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:24.746 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:55:24.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:24.749 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:55:24.850 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:55:24.888 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:24.990 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:55:24.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:24.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:55:25.098 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:55:25.126 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:25.145 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:55:25.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:25.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:55:25.249 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:55:25.263 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:55:25.267 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-23T16:55:25.286 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:25.295 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:55:25.319 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:55:25.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:25.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:55:25.324 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:55:25.345 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:55:25.369 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:55:25.397 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:55:25.423 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:55:25.423 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:55:25.450 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:55:25.459 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:25.488 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:55:25.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:25.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:55:25.500 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:55:25.509 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-23T16:55:25.530 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:55:25.538 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:25 smithi188 ceph-mon[44101]: pgmap v2150: 27 pgs: 27 active+clean; 807 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-23T16:55:25.539 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:25 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2109684076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:55:25.539 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:25 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:25.567 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:55:25.594 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:55:25.614 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:55:25.617 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:55:25.617 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:55:25.617 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:55:25.617 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:55:25.617 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:55:25.617 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:55:25.617 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:55:25.617 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:55:25.617 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:55:25.617 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:55:25.617 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:55:25.618 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:55:25.637 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:25.669 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:55:25.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:25.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:55:25.687 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:55:25.775 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:55:25.809 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:25 smithi103 ceph-mon[39629]: pgmap v2150: 27 pgs: 27 active+clean; 807 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-23T16:55:25.809 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:25 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2109684076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:55:25.809 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:25 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:25.811 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:25.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:25 smithi148 ceph-mon[44520]: pgmap v2150: 27 pgs: 27 active+clean; 807 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-23T16:55:25.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:25 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2109684076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:55:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:25 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:25.891 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:55:25.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:25.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:55:25.995 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:55:26.024 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:26.055 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:55:26.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:26.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:55:26.166 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:55:26.200 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:26.445 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:26.465 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:55:26.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:26.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:55:26.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.543 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.573 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:55:26.599 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:26.619 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:55:26.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:26.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:55:26.724 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:55:26.751 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:26.760 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.760 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.760 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.760 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.761 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.761 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.761 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.761 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.761 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.761 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.761 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:26 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.779 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:55:26.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:26.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:55:26.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:26 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.883 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:55:26.946 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.946 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.946 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.946 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.946 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.946 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.946 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.947 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.947 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:26 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:26.956 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:27.530 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:55:27.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:27 smithi103 ceph-mon[39629]: pgmap v2151: 27 pgs: 27 active+clean; 688 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 868 op/s 2024-04-23T16:55:27.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:27 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:55:27.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:27 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:55:27.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:27 smithi148 ceph-mon[44520]: pgmap v2151: 27 pgs: 27 active+clean; 688 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 868 op/s 2024-04-23T16:55:27.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:27 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:55:27.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:27 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:55:27.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:27 smithi188 ceph-mon[44101]: pgmap v2151: 27 pgs: 27 active+clean; 688 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 868 op/s 2024-04-23T16:55:27.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:27 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:55:27.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:27 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:55:28.023 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:55:28.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:28.029 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:55:28.077 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:28.130 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:55:28.157 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:28.177 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:55:28.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:28.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:55:28.281 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:55:28.333 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:28.499 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T16:55:28.515 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:55:28.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:28.520 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:55:28.602 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:28 smithi103 ceph-mon[39629]: osdmap e776: 12 total, 12 up, 8 in 2024-04-23T16:55:28.621 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5 2024-04-23T16:55:28.647 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:28.666 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-23T16:55:28.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:28.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.5' 2024-04-23T16:55:28.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:28 smithi188 ceph-mon[44101]: osdmap e776: 12 total, 12 up, 8 in 2024-04-23T16:55:28.771 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:55:28.798 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:28.818 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:55:28.820 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:28 smithi148 ceph-mon[44520]: osdmap e776: 12 total, 12 up, 8 in 2024-04-23T16:55:28.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:28.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:55:28.922 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:55:28.967 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:29.568 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:55:29.568 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:55:29.568 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2024-04-23T16:55:29.568 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:55:29.574 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is set 2024-04-23T16:55:29.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:29 smithi103 ceph-mon[39629]: pgmap v2153: 27 pgs: 27 active+clean; 487 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-23T16:55:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:29 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2139793488' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:55:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:29 smithi103 ceph-mon[39629]: from='client.37265 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:55:29.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:29 smithi148 ceph-mon[44520]: pgmap v2153: 27 pgs: 27 active+clean; 487 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-23T16:55:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:29 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2139793488' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:55:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:29 smithi148 ceph-mon[44520]: from='client.37265 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:55:29.936 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:29 smithi188 ceph-mon[44101]: pgmap v2153: 27 pgs: 27 active+clean; 487 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-23T16:55:29.936 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:29 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2139793488' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:55:29.936 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:29 smithi188 ceph-mon[44101]: from='client.37265 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:55:30.301 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-23T16:55:30.563 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:30 smithi188 ceph-mon[44101]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:55:30.563 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:30 smithi188 ceph-mon[44101]: from='client.37265 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:55:30.564 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:30 smithi188 ceph-mon[44101]: osdmap e777: 12 total, 12 up, 8 in 2024-04-23T16:55:30.734 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:55:30.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:30 smithi103 ceph-mon[39629]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:55:30.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:30 smithi103 ceph-mon[39629]: from='client.37265 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:55:30.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:30 smithi103 ceph-mon[39629]: osdmap e777: 12 total, 12 up, 8 in 2024-04-23T16:55:30.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:30 smithi148 ceph-mon[44520]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:55:30.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:30 smithi148 ceph-mon[44520]: from='client.37265 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-23T16:55:30.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:30 smithi148 ceph-mon[44520]: osdmap e777: 12 total, 12 up, 8 in 2024-04-23T16:55:31.549 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:55:31.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:31.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:55:31.656 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:55:31.687 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:31.719 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:55:31.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:31 smithi188 ceph-mon[44101]: pgmap v2155: 27 pgs: 27 active+clean; 428 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 880 op/s 2024-04-23T16:55:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:31.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:55:31.829 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:55:31.837 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:31 smithi103 ceph-mon[39629]: pgmap v2155: 27 pgs: 27 active+clean; 428 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 880 op/s 2024-04-23T16:55:31.870 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:31.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:31 smithi148 ceph-mon[44520]: pgmap v2155: 27 pgs: 27 active+clean; 428 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 880 op/s 2024-04-23T16:55:31.897 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:55:31.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:31.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:55:32.009 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:55:32.053 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:32.178 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:32.191 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:55:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:32.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:55:32.297 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:55:32.323 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:32.345 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:55:32.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:32.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:55:32.449 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:55:32.499 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:32.534 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:55:32.537 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:55:32.539 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:55:32.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:32.540 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:55:32.642 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:55:32.681 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:32.820 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:55:32.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:32.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:55:32.927 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:55:32.934 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:55:32.934 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2024-04-23T16:55:32.934 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2024-04-23T16:55:32.935 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2024-04-23T16:55:32.935 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:55:32.997 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:33.164 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:55:33.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:33.170 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:55:33.271 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:55:33.299 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:33.321 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:55:33.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:33.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:55:33.428 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:55:33.460 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:33.556 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:55:33.655 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-23T16:55:34.103 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:55:34.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:34 smithi103 ceph-mon[39629]: from='client.17796 v1:172.21.15.103:0/705552323' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:55:34.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:34 smithi103 ceph-mon[39629]: pgmap v2156: 27 pgs: 27 active+clean; 225 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:55:34.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:34 smithi148 ceph-mon[44520]: from='client.17796 v1:172.21.15.103:0/705552323' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:55:34.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:34 smithi148 ceph-mon[44520]: pgmap v2156: 27 pgs: 27 active+clean; 225 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:55:34.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:34 smithi188 ceph-mon[44101]: from='client.17796 v1:172.21.15.103:0/705552323' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:55:34.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:34 smithi188 ceph-mon[44101]: pgmap v2156: 27 pgs: 27 active+clean; 225 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-23T16:55:34.510 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Removed 151130 objects 2024-04-23T16:55:34.511 INFO:tasks.radosbench.radosbench.2.smithi188.stdout:Clean up completed and total clean up time :155.397 2024-04-23T16:55:34.519 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:55:34.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:34.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:55:34.629 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:55:34.657 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:34.961 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":2155,"stamp":"2024-04-23T16:55:31.010491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":449249848,"num_objects":6849,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295415,"num_write_kb":9672321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8587,"num_bytes_recovered":563164768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162358,"ondisk_log_size":162358,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6217008,"kb_used_data":899424,"kb_used_omap":522,"kb_used_meta":5316917,"kb_avail":743744208,"statfs":{"total":767960285184,"available":761594068992,"internally_reserved":0,"allocated":921010176,"data_stored":917863462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":534579,"internal_metadata":5444523981},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,2,4,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-461701120,"num_objects":-7045,"num_object_clones":0,"num_object_copies":-14090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7045,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001366"},"pg_stats":[{"pgid":"9.7","version":"775'18532","reported_seq":35883,"reported_epoch":775,"state":"active+clean","last_fresh":"2024-04-23T16:55:26.301924+0000","last_change":"2024-04-23T16:55:12.394650+0000","last_active":"2024-04-23T16:55:26.301924+0000","last_peered":"2024-04-23T16:55:26.301924+0000","last_clean":"2024-04-23T16:55:26.301924+0000","last_became_active":"2024-04-23T16:52:44.683405+0000","last_became_peered":"2024-04-23T16:52:44.683405+0000","last_unstale":"2024-04-23T16:55:26.301924+0000","last_undegraded":"2024-04-23T16:55:26.301924+0000","last_fullsized":"2024-04-23T16:55:26.301924+0000","mapping_epoch":732,"log_start":"730'8500","ondisk_log_start":"730'8500","created":703,"last_epoch_clean":771,"parent":"0.0","parent_split_bits":0,"last_scrub":"773'17677","last_scrub_stamp":"2024-04-23T16:55:12.394532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:55:12.394532+0000","objects_scrubbed":1399,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:13.017396+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35258368,"num_objects":538,"num_object_clones":0,"num_object_copies":1076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18532,"num_write_kb":610240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8584,"num_bytes_recovered":562561024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"775'18037","reported_seq":18302,"reported_epoch":775,"state":"active+clean","last_fresh":"2024-04-23T16:55:27.071433+0000","last_change":"2024-04-23T16:55:01.151246+0000","last_active":"2024-04-23T16:55:27.071433+0000","last_peered":"2024-04-23T16:55:27.071433+0000","last_clean":"2024-04-23T16:55:27.071433+0000","last_became_active":"2024-04-23T16:51:21.063642+0000","last_became_peered":"2024-04-23T16:51:21.063642+0000","last_unstale":"2024-04-23T16:55:27.071433+0000","last_undegraded":"2024-04-23T16:55:27.071433+0000","last_fullsized":"2024-04-23T16:55:27.071433+0000","mapping_epoch":703,"log_start":"730'8000","ondisk_log_start":"730'8000","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'16423","last_scrub_stamp":"2024-04-23T16:55:01.151049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:55:01.151049+0000","objects_scrubbed":2082,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:02.623292+0000","scrub_duration":380,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29818880,"num_objects":455,"num_object_clones":0,"num_object_copies":910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18037,"num_write_kb":591744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"776'18312","reported_seq":18610,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-04-23T16:55:27.851483+0000","last_change":"2024-04-23T16:54:21.272785+0000","last_active":"2024-04-23T16:55:27.851483+0000","last_peered":"2024-04-23T16:55:27.851483+0000","last_clean":"2024-04-23T16:55:27.851483+0000","last_became_active":"2024-04-23T16:51:21.063776+0000","last_became_peered":"2024-04-23T16:51:21.063776+0000","last_unstale":"2024-04-23T16:55:27.851483+0000","last_undegraded":"2024-04-23T16:55:27.851483+0000","last_fullsized":"2024-04-23T16:55:27.851483+0000","mapping_epoch":703,"log_start":"730'8300","ondisk_log_start":"730'8300","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14080","last_scrub_stamp":"2024-04-23T16:54:21.272505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:21.272505+0000","objects_scrubbed":4691,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:50.192674+0000","scrub_duration":969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28049408,"num_objects":428,"num_object_clones":0,"num_object_copies":856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18312,"num_write_kb":599680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"775'18352","reported_seq":18631,"reported_epoch":775,"state":"active+clean","last_fresh":"2024-04-23T16:55:27.068397+0000","last_change":"2024-04-23T16:55:00.296822+0000","last_active":"2024-04-23T16:55:27.068397+0000","last_peered":"2024-04-23T16:55:27.068397+0000","last_clean":"2024-04-23T16:55:27.068397+0000","last_became_active":"2024-04-23T16:51:21.063114+0000","last_became_peered":"2024-04-23T16:51:21.063114+0000","last_unstale":"2024-04-23T16:55:27.068397+0000","last_undegraded":"2024-04-23T16:55:27.068397+0000","last_fullsized":"2024-04-23T16:55:27.068397+0000","mapping_epoch":703,"log_start":"730'8300","ondisk_log_start":"730'8300","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'16616","last_scrub_stamp":"2024-04-23T16:55:00.296670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:55:00.296670+0000","objects_scrubbed":2221,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:18.149351+0000","scrub_duration":479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31457280,"num_objects":480,"num_object_clones":0,"num_object_copies":960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18352,"num_write_kb":602624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"776'18578","reported_seq":18860,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-04-23T16:55:28.420226+0000","last_change":"2024-04-23T16:55:01.549355+0000","last_active":"2024-04-23T16:55:28.420226+0000","last_peered":"2024-04-23T16:55:28.420226+0000","last_clean":"2024-04-23T16:55:28.420226+0000","last_became_active":"2024-04-23T16:51:21.063652+0000","last_became_peered":"2024-04-23T16:51:21.063652+0000","last_unstale":"2024-04-23T16:55:28.420226+0000","last_undegraded":"2024-04-23T16:55:28.420226+0000","last_fullsized":"2024-04-23T16:55:28.420226+0000","mapping_epoch":703,"log_start":"731'8500","ondisk_log_start":"731'8500","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'16831","last_scrub_stamp":"2024-04-23T16:55:01.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:55:01.549165+0000","objects_scrubbed":2128,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:27.145270+0000","scrub_duration":536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24117248,"num_objects":368,"num_object_clones":0,"num_object_copies":736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18578,"num_write_kb":606272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.2","version":"776'18740","reported_seq":19040,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-04-23T16:55:28.388582+0000","last_change":"2024-04-23T16:54:24.962245+0000","last_active":"2024-04-23T16:55:28.388582+0000","last_peered":"2024-04-23T16:55:28.388582+0000","last_clean":"2024-04-23T16:55:28.388582+0000","last_became_active":"2024-04-23T16:51:21.062561+0000","last_became_peered":"2024-04-23T16:51:21.062561+0000","last_unstale":"2024-04-23T16:55:28.388582+0000","last_undegraded":"2024-04-23T16:55:28.388582+0000","last_fullsized":"2024-04-23T16:55:28.388582+0000","mapping_epoch":703,"log_start":"733'8700","ondisk_log_start":"733'8700","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14622","last_scrub_stamp":"2024-04-23T16:54:24.962060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:24.962060+0000","objects_scrubbed":4531,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:49.051124+0000","scrub_duration":1162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24641536,"num_objects":376,"num_object_clones":0,"num_object_copies":752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18740,"num_write_kb":611712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.a","version":"255'139","reported_seq":1911,"reported_epoch":777,"state":"active+clean","last_fresh":"2024-04-23T16:55:29.578140+0000","last_change":"2024-04-23T16:54:28.362501+0000","last_active":"2024-04-23T16:55:29.578140+0000","last_peered":"2024-04-23T16:55:29.578140+0000","last_clean":"2024-04-23T16:55:29.578140+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:55:29.578140+0000","last_undegraded":"2024-04-23T16:55:29.578140+0000","last_fullsized":"2024-04-23T16:55:29.578140+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:54:28.362406+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:54:28.362406+0000","last_clean_scrub_stamp":"2024-04-23T16:54:28.362406+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"775'18592","reported_seq":18894,"reported_epoch":775,"state":"active+clean","last_fresh":"2024-04-23T16:55:27.050188+0000","last_change":"2024-04-23T16:54:31.665323+0000","last_active":"2024-04-23T16:55:27.050188+0000","last_peered":"2024-04-23T16:55:27.050188+0000","last_clean":"2024-04-23T16:55:27.050188+0000","last_became_active":"2024-04-23T16:51:21.063395+0000","last_became_peered":"2024-04-23T16:51:21.063395+0000","last_unstale":"2024-04-23T16:55:27.050188+0000","last_undegraded":"2024-04-23T16:55:27.050188+0000","last_fullsized":"2024-04-23T16:55:27.050188+0000","mapping_epoch":703,"log_start":"730'8500","ondisk_log_start":"730'8500","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"763'15015","last_scrub_stamp":"2024-04-23T16:54:31.665200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:31.665200+0000","objects_scrubbed":4086,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:39.674179+0000","scrub_duration":876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31850496,"num_objects":486,"num_object_clones":0,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18592,"num_write_kb":610496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1691,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-04-23T16:55:27.558757+0000","last_change":"2024-04-23T16:55:10.720904+0000","last_active":"2024-04-23T16:55:27.558757+0000","last_peered":"2024-04-23T16:55:27.558757+0000","last_clean":"2024-04-23T16:55:27.558757+0000","last_became_active":"2024-04-23T16:52:47.483615+0000","last_became_peered":"2024-04-23T16:52:47.483615+0000","last_unstale":"2024-04-23T16:55:27.558757+0000","last_undegraded":"2024-04-23T16:55:27.558757+0000","last_fullsized":"2024-04-23T16:55:27.558757+0000","mapping_epoch":735,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":736,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:10.720770+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:52:05.504325+0000","last_clean_scrub_stamp":"2024-04-23T16:55:10.720770+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:23.595719+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"777'18174","reported_seq":18472,"reported_epoch":777,"state":"active+clean","last_fresh":"2024-04-23T16:55:30.072131+0000","last_change":"2024-04-23T16:54:17.136201+0000","last_active":"2024-04-23T16:55:30.072131+0000","last_peered":"2024-04-23T16:55:30.072131+0000","last_clean":"2024-04-23T16:55:30.072131+0000","last_became_active":"2024-04-23T16:51:21.246391+0000","last_became_peered":"2024-04-23T16:51:21.246391+0000","last_unstale":"2024-04-23T16:55:30.072131+0000","last_undegraded":"2024-04-23T16:55:30.072131+0000","last_fullsized":"2024-04-23T16:55:30.072131+0000","mapping_epoch":703,"log_start":"730'8100","ondisk_log_start":"730'8100","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'13668","last_scrub_stamp":"2024-04-23T16:54:17.136038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:17.136038+0000","objects_scrubbed":4818,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:30.222456+0000","scrub_duration":963,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18350080,"num_objects":280,"num_object_clones":0,"num_object_copies":560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18174,"num_write_kb":590528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1899,"reported_epoch":775,"state":"active+clean","last_fresh":"2024-04-23T16:55:21.306186+0000","last_change":"2024-04-23T16:55:20.164898+0000","last_active":"2024-04-23T16:55:21.306186+0000","last_peered":"2024-04-23T16:55:21.306186+0000","last_clean":"2024-04-23T16:55:21.306186+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:55:21.306186+0000","last_undegraded":"2024-04-23T16:55:21.306186+0000","last_fullsized":"2024-04-23T16:55:21.306186+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:20.164813+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:51:25.290774+0000","last_clean_scrub_stamp":"2024-04-23T16:55:20.164813+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:30.353917+0000","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"775'18561","reported_seq":18862,"reported_epoch":775,"state":"active+clean","last_fresh":"2024-04-23T16:55:27.095549+0000","last_change":"2024-04-23T16:54:18.910760+0000","last_active":"2024-04-23T16:55:27.095549+0000","last_peered":"2024-04-23T16:55:27.095549+0000","last_clean":"2024-04-23T16:55:27.095549+0000","last_became_active":"2024-04-23T16:51:21.063669+0000","last_became_peered":"2024-04-23T16:51:21.063669+0000","last_unstale":"2024-04-23T16:55:27.095549+0000","last_undegraded":"2024-04-23T16:55:27.095549+0000","last_fullsized":"2024-04-23T16:55:27.095549+0000","mapping_epoch":703,"log_start":"731'8500","ondisk_log_start":"731'8500","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14143","last_scrub_stamp":"2024-04-23T16:54:18.910481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:18.910481+0000","objects_scrubbed":4947,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:29.963181+0000","scrub_duration":1208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32440320,"num_objects":495,"num_object_clones":0,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18561,"num_write_kb":609792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1915,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-04-23T16:55:27.629997+0000","last_change":"2024-04-23T16:55:00.540871+0000","last_active":"2024-04-23T16:55:27.629997+0000","last_peered":"2024-04-23T16:55:27.629997+0000","last_clean":"2024-04-23T16:55:27.629997+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:55:27.629997+0000","last_undegraded":"2024-04-23T16:55:27.629997+0000","last_fullsized":"2024-04-23T16:55:27.629997+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:00.540749+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:55:00.540749+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:15.651892+0000","scrub_duration":229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"776'18761","reported_seq":19063,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-04-23T16:55:28.424320+0000","last_change":"2024-04-23T16:54:19.829779+0000","last_active":"2024-04-23T16:55:28.424320+0000","last_peered":"2024-04-23T16:55:28.424320+0000","last_clean":"2024-04-23T16:55:28.424320+0000","last_became_active":"2024-04-23T16:51:21.062760+0000","last_became_peered":"2024-04-23T16:51:21.062760+0000","last_unstale":"2024-04-23T16:55:28.424320+0000","last_undegraded":"2024-04-23T16:55:28.424320+0000","last_fullsized":"2024-04-23T16:55:28.424320+0000","mapping_epoch":703,"log_start":"733'8700","ondisk_log_start":"733'8700","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14356","last_scrub_stamp":"2024-04-23T16:54:19.829518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:19.829518+0000","objects_scrubbed":4782,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:28.874042+0000","scrub_duration":1029,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23134208,"num_objects":353,"num_object_clones":0,"num_object_copies":706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18761,"num_write_kb":611648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1497,"reported_epoch":775,"state":"active+clean","last_fresh":"2024-04-23T16:55:21.305869+0000","last_change":"2024-04-23T16:55:02.927584+0000","last_active":"2024-04-23T16:55:21.305869+0000","last_peered":"2024-04-23T16:55:21.305869+0000","last_clean":"2024-04-23T16:55:21.305869+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:55:21.305869+0000","last_undegraded":"2024-04-23T16:55:21.305869+0000","last_fullsized":"2024-04-23T16:55:21.305869+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:02.927497+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:55:02.927497+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:29.690298+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"776'18345","reported_seq":18648,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-04-23T16:55:27.851506+0000","last_change":"2024-04-23T16:54:19.146832+0000","last_active":"2024-04-23T16:55:27.851506+0000","last_peered":"2024-04-23T16:55:27.851506+0000","last_clean":"2024-04-23T16:55:27.851506+0000","last_became_active":"2024-04-23T16:51:21.246519+0000","last_became_peered":"2024-04-23T16:51:21.246519+0000","last_unstale":"2024-04-23T16:55:27.851506+0000","last_undegraded":"2024-04-23T16:55:27.851506+0000","last_fullsized":"2024-04-23T16:55:27.851506+0000","mapping_epoch":703,"log_start":"730'8300","ondisk_log_start":"730'8300","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'13985","last_scrub_stamp":"2024-04-23T16:54:19.146171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:19.146171+0000","objects_scrubbed":4796,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:28.380772+0000","scrub_duration":863,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26804224,"num_objects":409,"num_object_clones":0,"num_object_copies":818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18345,"num_write_kb":600128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1460,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-04-23T16:55:27.560525+0000","last_change":"2024-04-23T16:55:03.259007+0000","last_active":"2024-04-23T16:55:27.560525+0000","last_peered":"2024-04-23T16:55:27.560525+0000","last_clean":"2024-04-23T16:55:27.560525+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:55:27.560525+0000","last_undegraded":"2024-04-23T16:55:27.560525+0000","last_fullsized":"2024-04-23T16:55:27.560525+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:03.258896+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:55:03.258896+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:32.586872+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.e","version":"775'18432","reported_seq":18785,"reported_epoch":775,"state":"active+clean","last_fresh":"2024-04-23T16:55:26.331219+0000","last_change":"2024-04-23T16:55:19.205969+0000","last_active":"2024-04-23T16:55:26.331219+0000","last_peered":"2024-04-23T16:55:26.331219+0000","last_clean":"2024-04-23T16:55:26.331219+0000","last_became_active":"2024-04-23T16:51:21.064486+0000","last_became_peered":"2024-04-23T16:51:21.064486+0000","last_unstale":"2024-04-23T16:55:26.331219+0000","last_undegraded":"2024-04-23T16:55:26.331219+0000","last_fullsized":"2024-04-23T16:55:26.331219+0000","mapping_epoch":703,"log_start":"730'8400","ondisk_log_start":"730'8400","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"773'17967","last_scrub_stamp":"2024-04-23T16:55:19.205866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:55:19.205866+0000","objects_scrubbed":1017,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:31.674801+0000","scrub_duration":146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35782656,"num_objects":546,"num_object_clones":0,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18432,"num_write_kb":607296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1919,"reported_epoch":777,"state":"active+clean","last_fresh":"2024-04-23T16:55:29.581130+0000","last_change":"2024-04-23T16:54:26.409871+0000","last_active":"2024-04-23T16:55:29.581130+0000","last_peered":"2024-04-23T16:55:29.581130+0000","last_clean":"2024-04-23T16:55:29.581130+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:55:29.581130+0000","last_undegraded":"2024-04-23T16:55:29.581130+0000","last_fullsized":"2024-04-23T16:55:29.581130+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:54:26.409705+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:42:50.245710+0000","last_clean_scrub_stamp":"2024-04-23T16:54:26.409705+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:33.984886+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.f","version":"775'18630","reported_seq":18932,"reported_epoch":775,"state":"active+clean","last_fresh":"2024-04-23T16:55:27.075916+0000","last_change":"2024-04-23T16:54:17.546594+0000","last_active":"2024-04-23T16:55:27.075916+0000","last_peered":"2024-04-23T16:55:27.075916+0000","last_clean":"2024-04-23T16:55:27.075916+0000","last_became_active":"2024-04-23T16:51:21.063588+0000","last_became_peered":"2024-04-23T16:51:21.063588+0000","last_unstale":"2024-04-23T16:55:27.075916+0000","last_undegraded":"2024-04-23T16:55:27.075916+0000","last_fullsized":"2024-04-23T16:55:27.075916+0000","mapping_epoch":703,"log_start":"731'8600","ondisk_log_start":"731'8600","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'14153","last_scrub_stamp":"2024-04-23T16:54:17.546419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:17.546419+0000","objects_scrubbed":4978,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:28.891004+0000","scrub_duration":812,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31588352,"num_objects":482,"num_object_clones":0,"num_object_copies":964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18630,"num_write_kb":611584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1930,"reported_epoch":777,"state":"active+clean","last_fresh":"2024-04-23T16:55:29.569467+0000","last_change":"2024-04-23T16:54:55.936994+0000","last_active":"2024-04-23T16:55:29.569467+0000","last_peered":"2024-04-23T16:55:29.569467+0000","last_clean":"2024-04-23T16:55:29.569467+0000","last_became_active":"2024-04-23T16:52:44.474580+0000","last_became_peered":"2024-04-23T16:52:44.474580+0000","last_unstale":"2024-04-23T16:55:29.569467+0000","last_undegraded":"2024-04-23T16:55:29.569467+0000","last_fullsized":"2024-04-23T16:55:29.569467+0000","mapping_epoch":732,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":733,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:54:55.936863+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:54:55.936863+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:10.224576+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"777'18670","reported_seq":19028,"reported_epoch":777,"state":"active+clean","last_fresh":"2024-04-23T16:55:30.109721+0000","last_change":"2024-04-23T16:54:56.611840+0000","last_active":"2024-04-23T16:55:30.109721+0000","last_peered":"2024-04-23T16:55:30.109721+0000","last_clean":"2024-04-23T16:55:30.109721+0000","last_became_active":"2024-04-23T16:51:21.079798+0000","last_became_peered":"2024-04-23T16:51:21.079798+0000","last_unstale":"2024-04-23T16:55:30.109721+0000","last_undegraded":"2024-04-23T16:55:30.109721+0000","last_fullsized":"2024-04-23T16:55:30.109721+0000","mapping_epoch":703,"log_start":"734'8600","ondisk_log_start":"734'8600","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"768'16627","last_scrub_stamp":"2024-04-23T16:54:56.611621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:56.611621+0000","objects_scrubbed":2377,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:07.253674+0000","scrub_duration":457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21233664,"num_objects":324,"num_object_clones":0,"num_object_copies":648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18670,"num_write_kb":607808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1802,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-04-23T16:55:27.560696+0000","last_change":"2024-04-23T16:54:32.619720+0000","last_active":"2024-04-23T16:55:27.560696+0000","last_peered":"2024-04-23T16:55:27.560696+0000","last_clean":"2024-04-23T16:55:27.560696+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:55:27.560696+0000","last_undegraded":"2024-04-23T16:55:27.560696+0000","last_fullsized":"2024-04-23T16:55:27.560696+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:54:32.619503+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:54:32.619503+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:40.217222+0000","scrub_duration":309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"775'18051","reported_seq":18296,"reported_epoch":775,"state":"active+clean","last_fresh":"2024-04-23T16:55:26.322722+0000","last_change":"2024-04-23T16:52:43.250377+0000","last_active":"2024-04-23T16:55:26.322722+0000","last_peered":"2024-04-23T16:55:26.322722+0000","last_clean":"2024-04-23T16:55:26.322722+0000","last_became_active":"2024-04-23T16:51:21.064506+0000","last_became_peered":"2024-04-23T16:51:21.064506+0000","last_unstale":"2024-04-23T16:55:26.322722+0000","last_undegraded":"2024-04-23T16:55:26.322722+0000","last_fullsized":"2024-04-23T16:55:26.322722+0000","mapping_epoch":703,"log_start":"730'8000","ondisk_log_start":"730'8000","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"731'8308","last_scrub_stamp":"2024-04-23T16:52:43.250180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:52:43.250180+0000","objects_scrubbed":8271,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:26.937145+0000","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34603032,"num_objects":529,"num_object_clones":0,"num_object_copies":1058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":529,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18051,"num_write_kb":594497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1734,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-04-23T16:55:27.558766+0000","last_change":"2024-04-23T16:55:11.687295+0000","last_active":"2024-04-23T16:55:27.558766+0000","last_peered":"2024-04-23T16:55:27.558766+0000","last_clean":"2024-04-23T16:55:27.558766+0000","last_became_active":"2024-04-23T16:52:47.480630+0000","last_became_peered":"2024-04-23T16:52:47.480630+0000","last_unstale":"2024-04-23T16:55:27.558766+0000","last_undegraded":"2024-04-23T16:55:27.558766+0000","last_fullsized":"2024-04-23T16:55:27.558766+0000","mapping_epoch":735,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":736,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:11.687173+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:55:11.687173+0000","last_clean_scrub_stamp":"2024-04-23T16:55:11.687173+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:24.332368+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"777'20664","reported_seq":21049,"reported_epoch":777,"state":"active+clean","last_fresh":"2024-04-23T16:55:30.118436+0000","last_change":"2024-04-23T16:54:24.434938+0000","last_active":"2024-04-23T16:55:30.118436+0000","last_peered":"2024-04-23T16:55:30.118436+0000","last_clean":"2024-04-23T16:55:30.118436+0000","last_became_active":"2024-04-23T16:51:21.063420+0000","last_became_peered":"2024-04-23T16:51:21.063420+0000","last_unstale":"2024-04-23T16:55:30.118436+0000","last_undegraded":"2024-04-23T16:55:30.118436+0000","last_fullsized":"2024-04-23T16:55:30.118436+0000","mapping_epoch":703,"log_start":"743'10602","ondisk_log_start":"743'10602","created":703,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'16466","last_scrub_stamp":"2024-04-23T16:54:24.434705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T16:51:20.043323+0000","last_clean_scrub_stamp":"2024-04-23T16:54:24.434705+0000","objects_scrubbed":4533,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:43.605780+0000","scrub_duration":1184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19529728,"num_objects":298,"num_object_clones":0,"num_object_copies":596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18648,"num_write_kb":606272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1891,"reported_epoch":775,"state":"active+clean","last_fresh":"2024-04-23T16:55:21.306186+0000","last_change":"2024-04-23T16:55:18.211396+0000","last_active":"2024-04-23T16:55:21.306186+0000","last_peered":"2024-04-23T16:55:21.306186+0000","last_clean":"2024-04-23T16:55:21.306186+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:55:21.306186+0000","last_undegraded":"2024-04-23T16:55:21.306186+0000","last_fullsized":"2024-04-23T16:55:21.306186+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:18.211281+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:51:31.339728+0000","last_clean_scrub_stamp":"2024-04-23T16:55:18.211281+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:28.248187+0000","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":448659480,"num_objects":6847,"num_object_clones":0,"num_object_copies":13694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6847,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295415,"num_write_kb":9672321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8584,"num_bytes_recovered":562561024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":851714048,"data_stored":851705904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160829,"ondisk_log_size":160829,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1679360,"data_stored":1661600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057122,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1193200,"kb_used_data":115176,"kb_used_omap":1,"kb_used_meta":1077950,"kb_avail":92551952,"statfs":{"total":95995035648,"available":94773198848,"internally_reserved":0,"allocated":117940224,"data_stored":117553299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1103821252},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351465,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613102,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":978428,"kb_used_data":130932,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92766724,"statfs":{"total":95995035648,"available":94993125376,"internally_reserved":0,"allocated":134074368,"data_stored":133681439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":867826112},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":66,"seq":283467842034,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":724744,"kb_used_data":134076,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93020408,"statfs":{"total":95995035648,"available":95252897792,"internally_reserved":0,"allocated":137293824,"data_stored":136897343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169080,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":874036,"kb_used_data":143148,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":92871116,"statfs":{"total":95995035648,"available":95100022784,"internally_reserved":0,"allocated":146583552,"data_stored":146192555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":748419530},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463421,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":642252,"kb_used_data":37696,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93102900,"statfs":{"total":95995035648,"available":95337369600,"internally_reserved":0,"allocated":38600704,"data_stored":38201721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":619051462},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137561,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":886132,"kb_used_data":280684,"kb_used_omap":231,"kb_used_meta":605208,"kb_avail":92859020,"statfs":{"total":95995035648,"available":95087636480,"internally_reserved":0,"allocated":287420416,"data_stored":287029419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":619733904},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,3,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738907,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":793856,"kb_used_data":28276,"kb_used_omap":1,"kb_used_meta":765566,"kb_avail":92951296,"statfs":{"total":95995035648,"available":95182127104,"internally_reserved":0,"allocated":28954624,"data_stored":28561695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":783940037},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":481,"seq":2065879269577,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":124360,"kb_used_data":29436,"kb_used_omap":281,"kb_used_meta":94630,"kb_avail":93620792,"statfs":{"total":95995035648,"available":95867691008,"internally_reserved":0,"allocated":30142464,"data_stored":29745991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":288353,"internal_metadata":96901535},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2658584756327,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":3126736191526,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757760,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25100288,"data_stored":25100288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":279646208,"data_stored":279642136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":26673152,"data_stored":26673152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32964608,"data_stored":32964608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":140972032,"data_stored":140967960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":123404288,"data_stored":123404288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":111738880,"data_stored":111738880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":111214592,"data_stored":111214592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:55:34.963 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-23T16:55:35.005 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:55:35.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:35.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:55:35.024 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:35.109 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:55:35.138 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:35.386 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:55:35.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:35.388 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:55:35.490 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:55:35.517 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:35.536 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:35 smithi103 ceph-mon[39629]: pgmap v2157: 27 pgs: 27 active+clean; 167 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-23T16:55:35.641 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:55:35.642 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:35 smithi188 ceph-mon[44101]: pgmap v2157: 27 pgs: 27 active+clean; 167 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-23T16:55:35.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:35.644 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:55:35.745 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:55:35.775 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:35.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:35 smithi148 ceph-mon[44520]: pgmap v2157: 27 pgs: 27 active+clean; 167 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-23T16:55:35.948 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T16:55:35.948 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2024-04-23T16:55:35.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2024-04-23T16:55:35.948 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [] 2024-04-23T16:55:35.948 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T16:55:35.974 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:55:35.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:35.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:55:36.078 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:55:36.107 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:36.179 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:55:36.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:36.182 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:55:36.283 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:55:36.316 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:36.336 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:55:36.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:36.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:55:36.441 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7 2024-04-23T16:55:36.471 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:36.492 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-23T16:55:36.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:36.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.7' 2024-04-23T16:55:36.596 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:55:36.627 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:36.634 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-23T16:55:36.646 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:55:36.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:36.649 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:55:36.750 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:55:36.778 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:36.797 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:55:36.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:36.800 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:55:36.901 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6 2024-04-23T16:55:36.927 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:55:36.934 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:36.956 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-23T16:55:36.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:36.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.6' 2024-04-23T16:55:37.053 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:55:37.053 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:55:37.053 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:55:37.060 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:55:37.064 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:55:37.085 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:55:37.086 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:37.105 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:55:37.125 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:55:37.146 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:55:37.165 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:55:37.185 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:55:37.185 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:55:37.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:37.188 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:55:37.205 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:55:37.224 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:55:37.225 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:55:37.225 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:55:37.225 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:55:37.225 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:55:37.226 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:55:37.226 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:55:37.226 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:55:37.226 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:55:37.226 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:55:37.226 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:55:37.226 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:55:37.226 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:55:37.289 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:55:37.319 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:37 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1512378579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:55:37.321 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:37.342 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:55:37.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:37.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:55:37.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:37 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1512378579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:55:37.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:37 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1512378579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:55:37.446 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:55:37.477 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:37.498 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:55:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:37.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:55:37.602 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9 2024-04-23T16:55:37.631 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:37.704 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-23T16:55:37.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:37.707 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.9' 2024-04-23T16:55:37.808 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:55:37.839 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:37.962 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:55:37.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:37.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:55:38.065 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:55:38.069 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:55:38.097 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:38.119 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:55:38.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:38.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:55:38.223 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11 2024-04-23T16:55:38.250 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:38 smithi103 ceph-mon[39629]: pgmap v2158: 27 pgs: 27 active+clean; 74 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 870 op/s 2024-04-23T16:55:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:38 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:38 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/861897978' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:55:38.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:38 smithi148 ceph-mon[44520]: pgmap v2158: 27 pgs: 27 active+clean; 74 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 870 op/s 2024-04-23T16:55:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:38 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:38 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/861897978' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:55:38.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:38 smithi188 ceph-mon[44101]: pgmap v2158: 27 pgs: 27 active+clean; 74 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 870 op/s 2024-04-23T16:55:38.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:38 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:38.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:38 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/861897978' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:55:38.398 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-23T16:55:38.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:38.402 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.11' 2024-04-23T16:55:38.503 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:55:38.530 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:38.666 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:55:38.666 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:55:38.879 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:55:38.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:38.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:55:38.984 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:55:39.012 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:39.031 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:55:39.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:39.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:55:39.129 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_7 2024-04-23T16:55:39.129 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-04-23T16:55:39.135 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10 2024-04-23T16:55:39.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:39 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:39.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:39 smithi103 ceph-mon[39629]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:55:39.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:39 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/861897978' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:55:39.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:39 smithi103 ceph-mon[39629]: osdmap e778: 12 total, 12 up, 8 in 2024-04-23T16:55:39.164 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:39.334 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:39 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:39.334 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:39 smithi188 ceph-mon[44101]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:55:39.334 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:39 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/861897978' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:55:39.334 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:39 smithi188 ceph-mon[44101]: osdmap e778: 12 total, 12 up, 8 in 2024-04-23T16:55:39.359 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-23T16:55:39.363 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-23T16:55:39.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:39.367 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.10' 2024-04-23T16:55:39.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:39 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:39.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:39 smithi148 ceph-mon[44520]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-23T16:55:39.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:39 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/861897978' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:55:39.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:39 smithi148 ceph-mon[44520]: osdmap e778: 12 total, 12 up, 8 in 2024-04-23T16:55:39.390 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:39.468 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:55:39.495 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:39.521 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:55:39.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:39.524 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:55:39.625 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:55:39.635 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:55:39.656 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:39.675 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:55:39.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:39.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:55:39.779 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2 2024-04-23T16:55:39.806 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:39.878 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-23T16:55:39.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:39.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.2' 2024-04-23T16:55:39.978 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T16:55:39.982 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:55:40.050 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:40.198 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:55:40.200 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:40 smithi103 ceph-mon[39629]: pgmap v2160: 27 pgs: 27 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 824 op/s 2024-04-23T16:55:40.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:40.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:55:40.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:40 smithi188 ceph-mon[44101]: pgmap v2160: 27 pgs: 27 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 824 op/s 2024-04-23T16:55:40.303 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:55:40.330 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:40.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:40 smithi148 ceph-mon[44520]: pgmap v2160: 27 pgs: 27 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 824 op/s 2024-04-23T16:55:40.904 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:55:41.246 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:41.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:41 smithi103 ceph-mon[39629]: 9.0 scrub starts 2024-04-23T16:55:41.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:41 smithi103 ceph-mon[39629]: 9.0 scrub ok 2024-04-23T16:55:41.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:41 smithi103 ceph-mon[39629]: 9.b scrub starts 2024-04-23T16:55:41.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:41 smithi103 ceph-mon[39629]: 9.b scrub ok 2024-04-23T16:55:41.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:41 smithi148 ceph-mon[44520]: 9.0 scrub starts 2024-04-23T16:55:41.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:41 smithi148 ceph-mon[44520]: 9.0 scrub ok 2024-04-23T16:55:41.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:41 smithi148 ceph-mon[44520]: 9.b scrub starts 2024-04-23T16:55:41.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:41 smithi148 ceph-mon[44520]: 9.b scrub ok 2024-04-23T16:55:41.374 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:55:41.374 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:55:41.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:41 smithi188 ceph-mon[44101]: 9.0 scrub starts 2024-04-23T16:55:41.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:41 smithi188 ceph-mon[44101]: 9.0 scrub ok 2024-04-23T16:55:41.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:41 smithi188 ceph-mon[44101]: 9.b scrub starts 2024-04-23T16:55:41.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:41 smithi188 ceph-mon[44101]: 9.b scrub ok 2024-04-23T16:55:41.511 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:55:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:41.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:55:41.615 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1 2024-04-23T16:55:41.644 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:41.663 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-23T16:55:41.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:41.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.1' 2024-04-23T16:55:41.766 DEBUG:teuthology.orchestra.run.smithi188:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8 2024-04-23T16:55:41.797 INFO:teuthology.orchestra.run.smithi188.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:42.032 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-23T16:55:42.041 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:42.045 INFO:teuthology.orchestra.run.smithi188.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-23T16:55:42.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:42.048 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi188 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.8' 2024-04-23T16:55:42.093 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'unique_pool_7' removed 2024-04-23T16:55:42.148 DEBUG:teuthology.orchestra.run.smithi148:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4 2024-04-23T16:55:42.177 INFO:teuthology.orchestra.run.smithi148.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:42.196 INFO:teuthology.orchestra.run.smithi148.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-23T16:55:42.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:42.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi148 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.4' 2024-04-23T16:55:42.301 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3 2024-04-23T16:55:42.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:42 smithi103 ceph-mon[39629]: pgmap v2161: 27 pgs: 27 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 684 op/s 2024-04-23T16:55:42.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:42 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3745133844' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:55:42.329 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:42.330 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T16:55:42.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:42 smithi148 ceph-mon[44520]: pgmap v2161: 27 pgs: 27 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 684 op/s 2024-04-23T16:55:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:42 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3745133844' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:55:42.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:42 smithi188 ceph-mon[44101]: pgmap v2161: 27 pgs: 27 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 684 op/s 2024-04-23T16:55:42.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:42 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3745133844' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T16:55:42.477 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-23T16:55:42.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:42.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.3' 2024-04-23T16:55:42.581 DEBUG:teuthology.orchestra.run.smithi103:> sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0 2024-04-23T16:55:42.610 INFO:teuthology.orchestra.run.smithi103.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-23T16:55:42.958 INFO:teuthology.orchestra.run.smithi103.stderr:Error: no container with name or ID "ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-23T16:55:42.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-23T16:55:42.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi103 with status 125: 'sudo docker kill -s 1 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd.0' 2024-04-23T16:55:42.981 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-23T16:55:42.994 INFO:tasks.thrashosds:joining thrashosds 2024-04-23T16:55:43.224 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:43 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3745133844' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:55:43.224 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:43 smithi103 ceph-mon[39629]: osdmap e779: 12 total, 12 up, 8 in 2024-04-23T16:55:43.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:43 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3745133844' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:55:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:43 smithi148 ceph-mon[44520]: osdmap e779: 12 total, 12 up, 8 in 2024-04-23T16:55:43.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:43 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3745133844' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T16:55:43.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:43 smithi188 ceph-mon[44101]: osdmap e779: 12 total, 12 up, 8 in 2024-04-23T16:55:44.096 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2024-04-23T16:55:44.097 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T16:55:44.254 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:44 smithi103 ceph-mon[39629]: 1.4 scrub starts 2024-04-23T16:55:44.254 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:44 smithi103 ceph-mon[39629]: 1.4 scrub ok 2024-04-23T16:55:44.254 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:44 smithi103 ceph-mon[39629]: pgmap v2163: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:44.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:44 smithi103 ceph-mon[39629]: 1.a scrub starts 2024-04-23T16:55:44.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:44 smithi148 ceph-mon[44520]: 1.4 scrub starts 2024-04-23T16:55:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:44 smithi148 ceph-mon[44520]: 1.4 scrub ok 2024-04-23T16:55:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:44 smithi148 ceph-mon[44520]: pgmap v2163: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:44.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:44 smithi148 ceph-mon[44520]: 1.a scrub starts 2024-04-23T16:55:44.404 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:44 smithi188 ceph-mon[44101]: 1.4 scrub starts 2024-04-23T16:55:44.404 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:44 smithi188 ceph-mon[44101]: 1.4 scrub ok 2024-04-23T16:55:44.404 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:44 smithi188 ceph-mon[44101]: pgmap v2163: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:44.404 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:44 smithi188 ceph-mon[44101]: 1.a scrub starts 2024-04-23T16:55:44.769 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-23T16:55:45.330 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:45.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:45 smithi103 ceph-mon[39629]: 1.a scrub ok 2024-04-23T16:55:45.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:45 smithi148 ceph-mon[44520]: 1.a scrub ok 2024-04-23T16:55:45.453 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0: {} 2024-04-23T16:55:45.472 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1: {} 2024-04-23T16:55:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:45 smithi188 ceph-mon[44101]: 1.a scrub ok 2024-04-23T16:55:45.491 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2: {} 2024-04-23T16:55:45.510 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3: {} 2024-04-23T16:55:45.530 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4: {} 2024-04-23T16:55:45.552 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5: {} 2024-04-23T16:55:45.571 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6: {} 2024-04-23T16:55:45.591 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7: {} 2024-04-23T16:55:45.612 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8: {} 2024-04-23T16:55:45.638 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9: {} 2024-04-23T16:55:45.658 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10: {} 2024-04-23T16:55:45.679 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11: {} 2024-04-23T16:55:45.680 INFO:teuthology.orchestra.run.smithi103.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T16:55:45.680 INFO:teuthology.orchestra.run.smithi103.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T16:55:45.680 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T16:55:45.680 INFO:teuthology.orchestra.run.smithi103.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T16:55:45.680 INFO:teuthology.orchestra.run.smithi103.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T16:55:45.680 INFO:teuthology.orchestra.run.smithi103.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T16:55:45.681 INFO:teuthology.orchestra.run.smithi103.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T16:55:45.681 INFO:teuthology.orchestra.run.smithi103.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T16:55:45.681 INFO:teuthology.orchestra.run.smithi103.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-23T16:55:45.681 INFO:teuthology.orchestra.run.smithi103.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-23T16:55:45.681 INFO:teuthology.orchestra.run.smithi103.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-23T16:55:45.681 INFO:teuthology.orchestra.run.smithi103.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-23T16:55:46.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:46 smithi103 ceph-mon[39629]: pgmap v2164: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:46.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:46 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3668126660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:55:46.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:46.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:46.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:46.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:46.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:46 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:46.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:46 smithi148 ceph-mon[44520]: pgmap v2164: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:46.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:46 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3668126660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:55:46.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:46.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:46.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:46.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:46.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:46 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:46.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:46 smithi188 ceph-mon[44101]: pgmap v2164: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:46.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:46 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3668126660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-23T16:55:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:46 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.111 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:55:47.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:47 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:47 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/129959095' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:55:47.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:47 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:47 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/129959095' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:55:47.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:47 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:55:47.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:47 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/129959095' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:55:48.335 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:48 smithi103 ceph-mon[39629]: pgmap v2165: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:48.335 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:48 smithi103 ceph-mon[39629]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:55:48.335 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:48 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:55:48.335 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:48 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/129959095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:55:48.335 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:48 smithi103 ceph-mon[39629]: osdmap e780: 12 total, 12 up, 8 in 2024-04-23T16:55:48.336 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:48 smithi103 ceph-mon[39629]: 1.6 deep-scrub starts 2024-04-23T16:55:48.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:48 smithi148 ceph-mon[44520]: pgmap v2165: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:48.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:48 smithi148 ceph-mon[44520]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:55:48.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:48 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:55:48.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:48 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/129959095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:55:48.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:48 smithi148 ceph-mon[44520]: osdmap e780: 12 total, 12 up, 8 in 2024-04-23T16:55:48.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:48 smithi148 ceph-mon[44520]: 1.6 deep-scrub starts 2024-04-23T16:55:48.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:48 smithi188 ceph-mon[44101]: pgmap v2165: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:48.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:48 smithi188 ceph-mon[44101]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-23T16:55:48.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:48 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:55:48.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:48 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/129959095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:55:48.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:48 smithi188 ceph-mon[44101]: osdmap e780: 12 total, 12 up, 8 in 2024-04-23T16:55:48.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:48 smithi188 ceph-mon[44101]: 1.6 deep-scrub starts 2024-04-23T16:55:48.693 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:55:48.693 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:55:48.693 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-23T16:55:48.693 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:55:49.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:49 smithi103 ceph-mon[39629]: 1.6 deep-scrub ok 2024-04-23T16:55:49.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:49 smithi188 ceph-mon[44101]: 1.6 deep-scrub ok 2024-04-23T16:55:49.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:49 smithi148 ceph-mon[44520]: 1.6 deep-scrub ok 2024-04-23T16:55:49.803 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-23T16:55:50.067 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:50.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:50 smithi188 ceph-mon[44101]: pgmap v2167: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:50.495 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:50 smithi103 ceph-mon[39629]: pgmap v2167: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:50.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:50 smithi148 ceph-mon[44520]: pgmap v2167: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:51.022 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-23T16:55:51.304 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:52.111 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:55:52.111 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:55:52.111 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-23T16:55:52.111 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-23T16:55:52.111 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:55:52.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:52 smithi188 ceph-mon[44101]: pgmap v2168: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:52.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:52 smithi103 ceph-mon[39629]: pgmap v2168: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:52.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:52 smithi148 ceph-mon[44520]: pgmap v2168: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:53.692 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-23T16:55:54.297 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:54.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:54 smithi103 ceph-mon[39629]: pgmap v2169: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:54.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:54 smithi188 ceph-mon[44101]: pgmap v2169: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:54.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:54 smithi148 ceph-mon[44520]: pgmap v2169: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:54.963 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-04-23T16:55:54.964 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 5, 6, 7, 8, 9, 10, 11, 1, 2, 3, 4] 2024-04-23T16:55:54.964 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-23T16:55:54.964 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.2 2024-04-23T16:55:55.146 INFO:teuthology.orchestra.run.smithi103.stderr:noscrub is unset 2024-04-23T16:55:55.261 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:55:54 smithi103 systemd[1]: Stopping Ceph osd.2 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T16:55:55.261 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:55 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/476624323' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:55:55.261 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:55 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:55:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:55 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/476624323' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:55:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:55 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:55:55.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:55 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/476624323' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-23T16:55:55.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:55 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:55:56.035 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:55:55 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:55:55.790+0000 7f31332af700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T16:55:56.036 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:55:55 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:55:55.790+0000 7f31332af700 -1 osd.2 780 *** Got signal Terminated *** 2024-04-23T16:55:56.036 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:55:55 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[221104]: 2024-04-23T16:55:55.790+0000 7f31332af700 -1 osd.2 780 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T16:55:56.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:56 smithi103 ceph-mon[39629]: pgmap v2170: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:56.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:56 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/476624323' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:55:56.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:56 smithi103 ceph-mon[39629]: osdmap e781: 12 total, 12 up, 8 in 2024-04-23T16:55:56.335 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:56 smithi103 ceph-mon[39629]: osd.2 marked itself down and dead 2024-04-23T16:55:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:56 smithi188 ceph-mon[44101]: pgmap v2170: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:56.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:56 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/476624323' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:55:56.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:56 smithi188 ceph-mon[44101]: osdmap e781: 12 total, 12 up, 8 in 2024-04-23T16:55:56.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:56 smithi188 ceph-mon[44101]: osd.2 marked itself down and dead 2024-04-23T16:55:56.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:56 smithi148 ceph-mon[44520]: pgmap v2170: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:56.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:56 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/476624323' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-23T16:55:56.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:56 smithi148 ceph-mon[44520]: osdmap e781: 12 total, 12 up, 8 in 2024-04-23T16:55:56.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:56 smithi148 ceph-mon[44520]: osd.2 marked itself down and dead 2024-04-23T16:55:56.797 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-23T16:55:57.347 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:57 smithi103 ceph-mon[39629]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T16:55:57.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:57 smithi103 ceph-mon[39629]: osdmap e782: 12 total, 11 up, 8 in 2024-04-23T16:55:57.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:55:57.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:55:57.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:55:57.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:55:57.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:55:57.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:57 smithi188 ceph-mon[44101]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T16:55:57.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:57 smithi188 ceph-mon[44101]: osdmap e782: 12 total, 11 up, 8 in 2024-04-23T16:55:57.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:57 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:55:57.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:57 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:55:57.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:57 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:55:57.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:57 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:55:57.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:57 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:55:57.599 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:55:57 smithi103 bash[323669]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2 2024-04-23T16:55:57.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:57 smithi148 ceph-mon[44520]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T16:55:57.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:57 smithi148 ceph-mon[44520]: osdmap e782: 12 total, 11 up, 8 in 2024-04-23T16:55:57.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:57 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:55:57.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:57 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:55:57.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:57 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:55:57.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:57 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:55:57.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:57 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:55:57.650 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:55:57.650 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:55:57.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:55:57.650 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-23T16:55:57.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.650 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.16995.0:192758 8.5 e670/609 8:a94bc997:::benchmark_data_smithi188_155546_object67249:head v 670'11829, mlcod=670'11827)", 2024-04-23T16:55:57.650 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:49:36.615631+0000", 2024-04-23T16:55:57.650 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 380.45614537500001, 2024-04-23T16:55:57.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13628989899999999, 2024-04-23T16:55:57.651 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.651 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.651 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:57.651 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.615631+0000", 2024-04-23T16:55:57.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.615631+0000", 2024-04-23T16:55:57.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.615632+0000", 2024-04-23T16:55:57.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3430000000000001e-06 2024-04-23T16:55:57.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.615640+0000", 2024-04-23T16:55:57.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7239999999999992e-06 2024-04-23T16:55:57.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.615649+0000", 2024-04-23T16:55:57.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1549999999999996e-06 2024-04-23T16:55:57.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.615656+0000", 2024-04-23T16:55:57.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7290000000000003e-06 2024-04-23T16:55:57.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.733198+0000", 2024-04-23T16:55:57.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.117542541 2024-04-23T16:55:57.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.654 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:55:57.654 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.733226+0000", 2024-04-23T16:55:57.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7537000000000001e-05 2024-04-23T16:55:57.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.654 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:55:57.654 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.751893+0000", 2024-04-23T16:55:57.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018667065 2024-04-23T16:55:57.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.751921+0000", 2024-04-23T16:55:57.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7804999999999999e-05 2024-04-23T16:55:57.655 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.655 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.655 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.655 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.16995.0:192759 8.3 e670/609 8:c9627483:::benchmark_data_smithi188_155546_object67250:head v 670'12083, mlcod=670'12081)", 2024-04-23T16:55:57.655 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:49:36.619210+0000", 2024-04-23T16:55:57.656 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 380.45256576700001, 2024-04-23T16:55:57.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.12931895299999999, 2024-04-23T16:55:57.656 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.656 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.656 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:57.656 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.619210+0000", 2024-04-23T16:55:57.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.619210+0000", 2024-04-23T16:55:57.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.619212+0000", 2024-04-23T16:55:57.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2699999999999999e-06 2024-04-23T16:55:57.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.619218+0000", 2024-04-23T16:55:57.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5520000000000001e-06 2024-04-23T16:55:57.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.733180+0000", 2024-04-23T16:55:57.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.113961307 2024-04-23T16:55:57.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.733199+0000", 2024-04-23T16:55:57.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9361999999999999e-05 2024-04-23T16:55:57.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.733272+0000", 2024-04-23T16:55:57.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3301999999999997e-05 2024-04-23T16:55:57.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:55:57.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.733291+0000", 2024-04-23T16:55:57.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9114e-05 2024-04-23T16:55:57.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:55:57.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.748491+0000", 2024-04-23T16:55:57.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015200094000000001 2024-04-23T16:55:57.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.748529+0000", 2024-04-23T16:55:57.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7951999999999998e-05 2024-04-23T16:55:57.660 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.660 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.660 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.660 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.16995.0:192769 8.5 e670/609 8:ab08c879:::benchmark_data_smithi188_155546_object67260:head v 670'11830, mlcod=670'11827)", 2024-04-23T16:55:57.660 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:49:36.626815+0000", 2024-04-23T16:55:57.660 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 380.444961299, 2024-04-23T16:55:57.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.12512695400000001, 2024-04-23T16:55:57.660 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.660 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.660 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:57.660 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.626815+0000", 2024-04-23T16:55:57.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.626815+0000", 2024-04-23T16:55:57.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.626816+0000", 2024-04-23T16:55:57.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3009999999999999e-06 2024-04-23T16:55:57.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.626823+0000", 2024-04-23T16:55:57.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0029999999999997e-06 2024-04-23T16:55:57.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.733184+0000", 2024-04-23T16:55:57.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.106360677 2024-04-23T16:55:57.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.733199+0000", 2024-04-23T16:55:57.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5364000000000001e-05 2024-04-23T16:55:57.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.737829+0000", 2024-04-23T16:55:57.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046293510000000003 2024-04-23T16:55:57.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:55:57.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.737849+0000", 2024-04-23T16:55:57.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0777000000000001e-05 2024-04-23T16:55:57.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:55:57.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.751925+0000", 2024-04-23T16:55:57.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014075560000000001 2024-04-23T16:55:57.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:49:36.751942+0000", 2024-04-23T16:55:57.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6920999999999999e-05 2024-04-23T16:55:57.664 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.664 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.664 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.665 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:245103 8.5 8.917803a5 (undecoded) ondisk+retry+write+known_if_redirected e694)", 2024-04-23T16:55:57.665 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:50:52.647941+0000", 2024-04-23T16:55:57.665 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 304.42383495500002, 2024-04-23T16:55:57.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.29105898499999999, 2024-04-23T16:55:57.665 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.665 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.665 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-23T16:55:57.665 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:55:57.665 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:55:57.666 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:55:57.666 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 245103 2024-04-23T16:55:57.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.666 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.647941+0000", 2024-04-23T16:55:57.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.647941+0000", 2024-04-23T16:55:57.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.647945+0000", 2024-04-23T16:55:57.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.36e-06 2024-04-23T16:55:57.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.647953+0000", 2024-04-23T16:55:57.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.089e-06 2024-04-23T16:55:57.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.647964+0000", 2024-04-23T16:55:57.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1442e-05 2024-04-23T16:55:57.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.647975+0000", 2024-04-23T16:55:57.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0939e-05 2024-04-23T16:55:57.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648207+0000", 2024-04-23T16:55:57.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00023209000000000001 2024-04-23T16:55:57.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:55:57.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648219+0000", 2024-04-23T16:55:57.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1372000000000001e-05 2024-04-23T16:55:57.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.938971+0000", 2024-04-23T16:55:57.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.290752232 2024-04-23T16:55:57.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.670 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.670 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.939000+0000", 2024-04-23T16:55:57.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9461000000000001e-05 2024-04-23T16:55:57.670 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.670 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.670 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.670 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:245109 8.5 8.959aeb5 (undecoded) ondisk+retry+write+known_if_redirected e694)", 2024-04-23T16:55:57.670 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:50:52.647997+0000", 2024-04-23T16:55:57.670 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 304.42377921999997, 2024-04-23T16:55:57.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.29112418299999998, 2024-04-23T16:55:57.670 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.671 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.671 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-23T16:55:57.671 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:55:57.671 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:55:57.671 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:55:57.671 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 245109 2024-04-23T16:55:57.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.671 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.671 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.671 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.647997+0000", 2024-04-23T16:55:57.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.647997+0000", 2024-04-23T16:55:57.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.647998+0000", 2024-04-23T16:55:57.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9899999999999999e-07 2024-04-23T16:55:57.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648001+0000", 2024-04-23T16:55:57.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6200000000000001e-06 2024-04-23T16:55:57.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648009+0000", 2024-04-23T16:55:57.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2110000000000001e-06 2024-04-23T16:55:57.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648014+0000", 2024-04-23T16:55:57.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2129999999999999e-06 2024-04-23T16:55:57.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648243+0000", 2024-04-23T16:55:57.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00022882999999999999 2024-04-23T16:55:57.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:55:57.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648249+0000", 2024-04-23T16:55:57.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.942e-06 2024-04-23T16:55:57.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.939114+0000", 2024-04-23T16:55:57.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.29086487300000002 2024-04-23T16:55:57.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.939121+0000", 2024-04-23T16:55:57.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.695e-06 2024-04-23T16:55:57.675 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.675 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.675 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.675 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:245121 8.5 8.698ae1c5 (undecoded) ondisk+retry+write+known_if_redirected e694)", 2024-04-23T16:55:57.675 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:50:52.648023+0000", 2024-04-23T16:55:57.675 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 304.42375307200001, 2024-04-23T16:55:57.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.29113268399999997, 2024-04-23T16:55:57.675 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.675 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.675 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-23T16:55:57.676 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:55:57.676 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:55:57.676 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:55:57.676 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 245121 2024-04-23T16:55:57.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.676 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648023+0000", 2024-04-23T16:55:57.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648023+0000", 2024-04-23T16:55:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648024+0000", 2024-04-23T16:55:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8100000000000003e-07 2024-04-23T16:55:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648026+0000", 2024-04-23T16:55:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5440000000000001e-06 2024-04-23T16:55:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648033+0000", 2024-04-23T16:55:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3570000000000003e-06 2024-04-23T16:55:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648038+0000", 2024-04-23T16:55:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.006e-06 2024-04-23T16:55:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648286+0000", 2024-04-23T16:55:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00024821700000000001 2024-04-23T16:55:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:55:57.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648291+0000", 2024-04-23T16:55:57.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6520000000000003e-06 2024-04-23T16:55:57.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.939148+0000", 2024-04-23T16:55:57.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.29085675500000002 2024-04-23T16:55:57.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.939156+0000", 2024-04-23T16:55:57.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6720000000000004e-06 2024-04-23T16:55:57.680 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.680 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.680 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.680 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.16995.0:245172 8.5 8.90de18f5 (undecoded) ondisk+retry+write+known_if_redirected e694)", 2024-04-23T16:55:57.680 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:50:52.648043+0000", 2024-04-23T16:55:57.680 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 304.42373283500001, 2024-04-23T16:55:57.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.29113625100000001, 2024-04-23T16:55:57.680 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.680 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.680 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-23T16:55:57.680 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:55:57.681 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.16995", 2024-04-23T16:55:57.681 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/11649549", 2024-04-23T16:55:57.681 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 245172 2024-04-23T16:55:57.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.681 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648043+0000", 2024-04-23T16:55:57.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648043+0000", 2024-04-23T16:55:57.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648044+0000", 2024-04-23T16:55:57.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2199999999999999e-07 2024-04-23T16:55:57.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648046+0000", 2024-04-23T16:55:57.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2230000000000001e-06 2024-04-23T16:55:57.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648051+0000", 2024-04-23T16:55:57.683 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6059999999999996e-06 2024-04-23T16:55:57.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.683 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.683 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.683 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648055+0000", 2024-04-23T16:55:57.683 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2139999999999998e-06 2024-04-23T16:55:57.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.683 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.683 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.683 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648311+0000", 2024-04-23T16:55:57.683 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00025634399999999998 2024-04-23T16:55:57.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.683 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.683 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-23T16:55:57.684 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.648317+0000", 2024-04-23T16:55:57.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5060000000000003e-06 2024-04-23T16:55:57.684 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.684 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.939176+0000", 2024-04-23T16:55:57.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.290858797 2024-04-23T16:55:57.684 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.684 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:50:52.939180+0000", 2024-04-23T16:55:57.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1389999999999997e-06 2024-04-23T16:55:57.684 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.685 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.685 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.685 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.27202.0:50239 9.4 9:2eb04f24:::benchmark_data_smithi188_167360_object50238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e714)", 2024-04-23T16:55:57.685 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:51:45.582562+0000", 2024-04-23T16:55:57.685 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 251.489214726, 2024-04-23T16:55:57.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13753842899999999, 2024-04-23T16:55:57.685 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.685 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.685 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:57.685 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:55:57.685 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.27202", 2024-04-23T16:55:57.686 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:55:57.686 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 50239 2024-04-23T16:55:57.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.686 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:45.582562+0000", 2024-04-23T16:55:57.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:45.582562+0000", 2024-04-23T16:55:57.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:45.582562+0000", 2024-04-23T16:55:57.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0900000000000001e-07 2024-04-23T16:55:57.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:45.582674+0000", 2024-04-23T16:55:57.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112139 2024-04-23T16:55:57.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.688 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.688 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:45.582680+0000", 2024-04-23T16:55:57.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7579999999999998e-06 2024-04-23T16:55:57.688 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.688 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.688 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.688 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:45.582684+0000", 2024-04-23T16:55:57.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8650000000000003e-06 2024-04-23T16:55:57.688 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.688 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.688 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.689 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:45.582732+0000", 2024-04-23T16:55:57.689 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8352000000000001e-05 2024-04-23T16:55:57.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.689 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.689 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:55:57.689 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:45.582846+0000", 2024-04-23T16:55:57.689 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011398199999999999 2024-04-23T16:55:57.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.689 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.689 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 8", 2024-04-23T16:55:57.689 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:45.582977+0000", 2024-04-23T16:55:57.689 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000130961 2024-04-23T16:55:57.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:55:57.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:45.584053+0000", 2024-04-23T16:55:57.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001075398 2024-04-23T16:55:57.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:57.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:45.719858+0000", 2024-04-23T16:55:57.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13580524299999999 2024-04-23T16:55:57.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:55:57.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:45.720006+0000", 2024-04-23T16:55:57.691 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000147887 2024-04-23T16:55:57.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:45.720100+0000", 2024-04-23T16:55:57.691 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4135000000000006e-05 2024-04-23T16:55:57.691 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.691 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.691 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.691 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.27202.0:76312 9.0 9:03475e55:::benchmark_data_smithi188_167360_object76311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e718)", 2024-04-23T16:55:57.692 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:51:59.335350+0000", 2024-04-23T16:55:57.692 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 237.736426352, 2024-04-23T16:55:57.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.12931920499999999, 2024-04-23T16:55:57.692 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.692 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.692 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:57.692 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:55:57.692 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.27202", 2024-04-23T16:55:57.692 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:55:57.692 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 76312 2024-04-23T16:55:57.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.692 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.335350+0000", 2024-04-23T16:55:57.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.335350+0000", 2024-04-23T16:55:57.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.335351+0000", 2024-04-23T16:55:57.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.155e-06 2024-04-23T16:55:57.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.694 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.694 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.335415+0000", 2024-04-23T16:55:57.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3509000000000005e-05 2024-04-23T16:55:57.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.694 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.694 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.335424+0000", 2024-04-23T16:55:57.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2140000000000002e-06 2024-04-23T16:55:57.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.695 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.695 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.335430+0000", 2024-04-23T16:55:57.695 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9179999999999999e-06 2024-04-23T16:55:57.695 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.695 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.695 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.695 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.335485+0000", 2024-04-23T16:55:57.695 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5421999999999999e-05 2024-04-23T16:55:57.695 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.695 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.695 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:55:57.695 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.335621+0000", 2024-04-23T16:55:57.695 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013575499999999999 2024-04-23T16:55:57.695 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 11", 2024-04-23T16:55:57.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.335765+0000", 2024-04-23T16:55:57.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014461500000000001 2024-04-23T16:55:57.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:55:57.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.337610+0000", 2024-04-23T16:55:57.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001844853 2024-04-23T16:55:57.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:57.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.464579+0000", 2024-04-23T16:55:57.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.12696817599999999 2024-04-23T16:55:57.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:55:57.697 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.464630+0000", 2024-04-23T16:55:57.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1125000000000003e-05 2024-04-23T16:55:57.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.697 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.464669+0000", 2024-04-23T16:55:57.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9462999999999998e-05 2024-04-23T16:55:57.697 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.697 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.697 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.698 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.27202.0:76319 9.0 9:021ce701:::benchmark_data_smithi188_167360_object76318:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e718)", 2024-04-23T16:55:57.698 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:51:59.345357+0000", 2024-04-23T16:55:57.698 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 237.726419139, 2024-04-23T16:55:57.698 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.12686393400000001, 2024-04-23T16:55:57.698 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.698 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.698 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:57.698 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:55:57.698 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.27202", 2024-04-23T16:55:57.698 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:55:57.698 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 76319 2024-04-23T16:55:57.698 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.699 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.345357+0000", 2024-04-23T16:55:57.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.345357+0000", 2024-04-23T16:55:57.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.700 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.700 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.700 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.345359+0000", 2024-04-23T16:55:57.700 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.474e-06 2024-04-23T16:55:57.700 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.700 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.700 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.700 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.345392+0000", 2024-04-23T16:55:57.700 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3722e-05 2024-04-23T16:55:57.700 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.700 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.701 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.701 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.345403+0000", 2024-04-23T16:55:57.701 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0278000000000001e-05 2024-04-23T16:55:57.701 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.701 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.701 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.701 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.345427+0000", 2024-04-23T16:55:57.701 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4037999999999999e-05 2024-04-23T16:55:57.701 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.701 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.701 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.701 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.345474+0000", 2024-04-23T16:55:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7241999999999999e-05 2024-04-23T16:55:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:55:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.345561+0000", 2024-04-23T16:55:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7616999999999998e-05 2024-04-23T16:55:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 11", 2024-04-23T16:55:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.345650+0000", 2024-04-23T16:55:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8599999999999999e-05 2024-04-23T16:55:57.703 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.703 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:55:57.703 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.347445+0000", 2024-04-23T16:55:57.703 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001794786 2024-04-23T16:55:57.703 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.703 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:57.703 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.472117+0000", 2024-04-23T16:55:57.703 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.124672515 2024-04-23T16:55:57.703 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.703 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:55:57.704 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.472172+0000", 2024-04-23T16:55:57.704 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4231999999999998e-05 2024-04-23T16:55:57.704 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.704 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.704 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.704 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:51:59.472221+0000", 2024-04-23T16:55:57.704 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9429999999999999e-05 2024-04-23T16:55:57.704 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.704 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.704 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.704 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.705 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.705 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.27202.0:256079 9.0 9:0f1a8660:::benchmark_data_smithi188_167360_object104946:head [delete] snapc 0=[] ondisk+write+known_if_redirected e765)", 2024-04-23T16:55:57.705 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:54:49.034506+0000", 2024-04-23T16:55:57.705 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.037270555999996, 2024-04-23T16:55:57.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.14532616000000001, 2024-04-23T16:55:57.705 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.705 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.705 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:57.705 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:55:57.705 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.27202", 2024-04-23T16:55:57.705 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:55:57.705 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 256079 2024-04-23T16:55:57.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.706 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.706 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.706 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.706 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.034506+0000", 2024-04-23T16:55:57.706 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.706 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.706 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.706 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.034506+0000", 2024-04-23T16:55:57.706 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.707 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.707 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.034508+0000", 2024-04-23T16:55:57.707 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9300000000000002e-06 2024-04-23T16:55:57.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.707 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.707 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.034514+0000", 2024-04-23T16:55:57.707 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4969999999999998e-06 2024-04-23T16:55:57.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.708 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.708 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.034525+0000", 2024-04-23T16:55:57.708 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0933e-05 2024-04-23T16:55:57.708 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.708 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.708 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.034533+0000", 2024-04-23T16:55:57.708 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6699999999999994e-06 2024-04-23T16:55:57.708 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.709 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.034590+0000", 2024-04-23T16:55:57.709 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7419e-05 2024-04-23T16:55:57.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:55:57.709 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.034717+0000", 2024-04-23T16:55:57.709 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012669600000000001 2024-04-23T16:55:57.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 11", 2024-04-23T16:55:57.709 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.034835+0000", 2024-04-23T16:55:57.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011805499999999999 2024-04-23T16:55:57.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:55:57.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.040485+0000", 2024-04-23T16:55:57.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056498980000000004 2024-04-23T16:55:57.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:57.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179740+0000", 2024-04-23T16:55:57.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13925552899999999 2024-04-23T16:55:57.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.711 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:55:57.711 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179787+0000", 2024-04-23T16:55:57.711 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6748e-05 2024-04-23T16:55:57.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.711 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.711 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179832+0000", 2024-04-23T16:55:57.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4784999999999999e-05 2024-04-23T16:55:57.712 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.712 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.712 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.712 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.27202.0:256084 9.6 9:6674f8dd:::benchmark_data_smithi188_167360_object104951:head [delete] snapc 0=[] ondisk+write+known_if_redirected e765)", 2024-04-23T16:55:57.712 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:54:49.040283+0000", 2024-04-23T16:55:57.712 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.031493146000003, 2024-04-23T16:55:57.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13947969599999999, 2024-04-23T16:55:57.712 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.712 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.713 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:57.713 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:55:57.713 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.27202", 2024-04-23T16:55:57.713 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:55:57.713 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 256084 2024-04-23T16:55:57.713 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.713 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.713 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.713 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.713 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.040283+0000", 2024-04-23T16:55:57.713 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.713 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.713 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.713 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.713 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.040283+0000", 2024-04-23T16:55:57.714 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.714 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.714 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.714 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.714 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.040284+0000", 2024-04-23T16:55:57.714 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0100000000000004e-07 2024-04-23T16:55:57.714 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.714 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.714 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.714 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.040287+0000", 2024-04-23T16:55:57.714 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8210000000000002e-06 2024-04-23T16:55:57.714 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.714 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.715 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.715 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.040291+0000", 2024-04-23T16:55:57.715 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3610000000000003e-06 2024-04-23T16:55:57.715 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.715 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.715 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.715 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.040295+0000", 2024-04-23T16:55:57.715 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2309999999999998e-06 2024-04-23T16:55:57.715 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.715 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.715 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.715 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.040509+0000", 2024-04-23T16:55:57.715 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000213387 2024-04-23T16:55:57.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.716 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.716 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:55:57.716 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.040592+0000", 2024-04-23T16:55:57.716 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3238000000000005e-05 2024-04-23T16:55:57.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.716 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.716 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 9", 2024-04-23T16:55:57.716 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.040666+0000", 2024-04-23T16:55:57.716 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3955000000000006e-05 2024-04-23T16:55:57.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.717 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.717 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:55:57.717 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.049981+0000", 2024-04-23T16:55:57.717 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093148250000000005 2024-04-23T16:55:57.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.717 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.717 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:57.717 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179663+0000", 2024-04-23T16:55:57.717 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.129682086 2024-04-23T16:55:57.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.717 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.718 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:55:57.718 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179718+0000", 2024-04-23T16:55:57.718 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5132e-05 2024-04-23T16:55:57.718 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.718 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.718 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.718 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179763+0000", 2024-04-23T16:55:57.718 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4959000000000001e-05 2024-04-23T16:55:57.718 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.718 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.718 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.718 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.719 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.719 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.27202.0:256086 9.6 9:6bd9b2b3:::benchmark_data_smithi188_167360_object104953:head [delete] snapc 0=[] ondisk+write+known_if_redirected e765)", 2024-04-23T16:55:57.719 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:54:49.040901+0000", 2024-04-23T16:55:57.719 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.030874974, 2024-04-23T16:55:57.719 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13907019000000001, 2024-04-23T16:55:57.719 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.719 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.719 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:57.719 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:55:57.719 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.27202", 2024-04-23T16:55:57.719 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:55:57.719 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 256086 2024-04-23T16:55:57.719 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.719 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.720 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.720 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.720 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.040901+0000", 2024-04-23T16:55:57.720 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.720 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.720 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.720 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.720 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.040901+0000", 2024-04-23T16:55:57.720 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.720 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.720 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.720 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.720 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.040902+0000", 2024-04-23T16:55:57.720 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.097e-06 2024-04-23T16:55:57.721 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.721 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.721 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.721 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.040906+0000", 2024-04-23T16:55:57.721 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0380000000000001e-06 2024-04-23T16:55:57.721 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.721 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.721 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.721 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.040915+0000", 2024-04-23T16:55:57.721 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8310000000000007e-06 2024-04-23T16:55:57.721 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.721 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.721 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.721 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.040922+0000", 2024-04-23T16:55:57.721 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9029999999999998e-06 2024-04-23T16:55:57.722 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.722 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.722 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.722 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.044629+0000", 2024-04-23T16:55:57.722 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037071389999999999 2024-04-23T16:55:57.722 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.722 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.722 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:55:57.722 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.044769+0000", 2024-04-23T16:55:57.722 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013976900000000001 2024-04-23T16:55:57.722 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.722 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.722 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 9", 2024-04-23T16:55:57.723 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.044897+0000", 2024-04-23T16:55:57.723 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000127463 2024-04-23T16:55:57.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.723 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.723 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:55:57.723 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.050090+0000", 2024-04-23T16:55:57.723 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051938640000000003 2024-04-23T16:55:57.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.723 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.723 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:57.723 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179887+0000", 2024-04-23T16:55:57.723 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.12979639000000001 2024-04-23T16:55:57.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.723 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.724 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:55:57.724 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179933+0000", 2024-04-23T16:55:57.724 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6276999999999998e-05 2024-04-23T16:55:57.724 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.724 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.724 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.724 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179971+0000", 2024-04-23T16:55:57.724 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8418999999999999e-05 2024-04-23T16:55:57.724 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.724 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.724 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.724 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.724 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.724 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.27202.0:256089 9.4 9:2ecf6790:::benchmark_data_smithi188_167360_object104956:head [delete] snapc 0=[] ondisk+write+known_if_redirected e765)", 2024-04-23T16:55:57.724 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:54:49.042207+0000", 2024-04-23T16:55:57.725 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.029568917999995, 2024-04-23T16:55:57.725 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13745232900000001, 2024-04-23T16:55:57.725 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.725 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.725 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:57.725 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:55:57.725 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.27202", 2024-04-23T16:55:57.725 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:55:57.725 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 256089 2024-04-23T16:55:57.725 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.725 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.725 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.725 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.725 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.042207+0000", 2024-04-23T16:55:57.726 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.726 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.726 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.726 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.726 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.042207+0000", 2024-04-23T16:55:57.726 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.726 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.726 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.726 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.726 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.042209+0000", 2024-04-23T16:55:57.726 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2920000000000001e-06 2024-04-23T16:55:57.726 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.726 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.726 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.727 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.042213+0000", 2024-04-23T16:55:57.727 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4780000000000002e-06 2024-04-23T16:55:57.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.727 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.727 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.727 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.042223+0000", 2024-04-23T16:55:57.727 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3940000000000002e-06 2024-04-23T16:55:57.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.727 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.727 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.727 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.042229+0000", 2024-04-23T16:55:57.727 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6789999999999999e-06 2024-04-23T16:55:57.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.727 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.728 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.728 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.045471+0000", 2024-04-23T16:55:57.728 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003241664 2024-04-23T16:55:57.728 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.728 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.728 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:55:57.728 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.045574+0000", 2024-04-23T16:55:57.728 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103329 2024-04-23T16:55:57.728 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.728 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.728 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 8", 2024-04-23T16:55:57.728 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.045676+0000", 2024-04-23T16:55:57.729 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000102017 2024-04-23T16:55:57.729 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.729 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.729 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:55:57.729 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.049188+0000", 2024-04-23T16:55:57.729 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035118660000000002 2024-04-23T16:55:57.729 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.729 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.729 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:57.729 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.055036+0000", 2024-04-23T16:55:57.729 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058475560000000003 2024-04-23T16:55:57.729 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.730 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.730 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:55:57.730 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.055079+0000", 2024-04-23T16:55:57.730 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3847000000000002e-05 2024-04-23T16:55:57.730 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.730 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.730 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.730 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179660+0000", 2024-04-23T16:55:57.730 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.124580207 2024-04-23T16:55:57.730 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.730 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.731 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.731 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.731 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.731 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.27202.0:256092 9.b e765/703 9:d191289f:::benchmark_data_smithi188_167360_object104959:head v 765'16217, mlcod=765'16215)", 2024-04-23T16:55:57.731 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:54:49.051392+0000", 2024-04-23T16:55:57.731 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.020383832999997, 2024-04-23T16:55:57.731 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.12846100999999999, 2024-04-23T16:55:57.731 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.731 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.731 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:57.731 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.731 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.732 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.732 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.051392+0000", 2024-04-23T16:55:57.732 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.732 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.732 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.732 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.732 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.051392+0000", 2024-04-23T16:55:57.732 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.732 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.732 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.732 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.732 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.051393+0000", 2024-04-23T16:55:57.732 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3600000000000002e-07 2024-04-23T16:55:57.733 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.733 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.733 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.733 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.051400+0000", 2024-04-23T16:55:57.733 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9059999999999996e-06 2024-04-23T16:55:57.733 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.733 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.733 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.733 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.051407+0000", 2024-04-23T16:55:57.733 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0609999999999998e-06 2024-04-23T16:55:57.733 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.733 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.733 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.733 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.051412+0000", 2024-04-23T16:55:57.734 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3499999999999999e-06 2024-04-23T16:55:57.734 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.734 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.734 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.734 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.051451+0000", 2024-04-23T16:55:57.734 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9255999999999998e-05 2024-04-23T16:55:57.734 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.734 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.734 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:55:57.734 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.051475+0000", 2024-04-23T16:55:57.734 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4207e-05 2024-04-23T16:55:57.734 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.734 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.734 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:55:57.735 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.055169+0000", 2024-04-23T16:55:57.735 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003694251 2024-04-23T16:55:57.735 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.735 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.735 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.735 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179853+0000", 2024-04-23T16:55:57.735 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.12468404299999999 2024-04-23T16:55:57.735 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.735 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.735 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.735 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.735 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.736 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.27202.0:256094 9.e e765/703 9:71bcf9dd:::benchmark_data_smithi188_167360_object104961:head v 765'16024, mlcod=765'16022)", 2024-04-23T16:55:57.736 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:54:49.053925+0000", 2024-04-23T16:55:57.736 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.017851144000005, 2024-04-23T16:55:57.736 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.125880199, 2024-04-23T16:55:57.736 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.736 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.736 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:57.736 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.736 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.736 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.736 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.053925+0000", 2024-04-23T16:55:57.736 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.737 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.737 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.737 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.737 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.053925+0000", 2024-04-23T16:55:57.737 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.737 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.737 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.737 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.737 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.053927+0000", 2024-04-23T16:55:57.737 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.716e-06 2024-04-23T16:55:57.737 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.737 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.738 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.738 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.053935+0000", 2024-04-23T16:55:57.738 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9710000000000004e-06 2024-04-23T16:55:57.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.738 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.738 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.738 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.053944+0000", 2024-04-23T16:55:57.738 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5030000000000003e-06 2024-04-23T16:55:57.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.738 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.738 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.738 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.053951+0000", 2024-04-23T16:55:57.738 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5599999999999999e-06 2024-04-23T16:55:57.739 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.739 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.739 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.739 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.054008+0000", 2024-04-23T16:55:57.739 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6972999999999998e-05 2024-04-23T16:55:57.739 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.739 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.739 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:55:57.739 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.054026+0000", 2024-04-23T16:55:57.739 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7824000000000001e-05 2024-04-23T16:55:57.739 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.739 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.739 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:55:57.739 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179774+0000", 2024-04-23T16:55:57.740 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.125748427 2024-04-23T16:55:57.740 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.740 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.740 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.740 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179805+0000", 2024-04-23T16:55:57.740 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1225e-05 2024-04-23T16:55:57.740 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.740 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.740 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.740 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.740 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.740 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop_reply(client.27202.0:256084 9.6 e765/703 ondisk, result = 0)", 2024-04-23T16:55:57.740 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:54:49.055058+0000", 2024-04-23T16:55:57.740 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.016717818999993, 2024-04-23T16:55:57.740 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.124712511, 2024-04-23T16:55:57.741 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.741 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.741 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:55:57.741 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.741 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.741 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.741 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.055058+0000", 2024-04-23T16:55:57.741 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.741 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.741 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.741 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.741 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.055058+0000", 2024-04-23T16:55:57.741 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.742 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.742 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.742 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.055059+0000", 2024-04-23T16:55:57.742 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2399999999999996e-07 2024-04-23T16:55:57.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.742 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.742 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.742 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.055063+0000", 2024-04-23T16:55:57.742 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8510000000000001e-06 2024-04-23T16:55:57.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.742 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.742 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.742 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.055069+0000", 2024-04-23T16:55:57.743 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3789999999999997e-06 2024-04-23T16:55:57.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.743 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.743 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.743 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.055073+0000", 2024-04-23T16:55:57.743 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1010000000000002e-06 2024-04-23T16:55:57.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.743 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.743 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.743 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179648+0000", 2024-04-23T16:55:57.743 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.12457555400000001 2024-04-23T16:55:57.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.743 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.743 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:55:57.743 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179657+0000", 2024-04-23T16:55:57.744 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4470000000000006e-06 2024-04-23T16:55:57.744 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.744 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.744 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.744 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179771+0000", 2024-04-23T16:55:57.744 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114255 2024-04-23T16:55:57.744 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.744 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.744 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.744 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.744 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.744 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop_reply(client.27202.0:256086 9.6 e765/703 ondisk, result = 0)", 2024-04-23T16:55:57.744 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:54:49.055084+0000", 2024-04-23T16:55:57.744 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.016692168999995, 2024-04-23T16:55:57.745 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.124896912, 2024-04-23T16:55:57.745 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.745 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.745 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-23T16:55:57.745 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.745 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.745 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.745 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.055084+0000", 2024-04-23T16:55:57.745 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.745 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.745 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.745 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.055084+0000", 2024-04-23T16:55:57.745 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.746 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.746 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.746 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.055084+0000", 2024-04-23T16:55:57.746 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9499999999999998e-07 2024-04-23T16:55:57.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.746 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.746 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.746 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.055086+0000", 2024-04-23T16:55:57.746 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7099999999999999e-06 2024-04-23T16:55:57.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.746 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.746 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.746 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.055089+0000", 2024-04-23T16:55:57.747 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.954e-06 2024-04-23T16:55:57.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.747 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.747 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.747 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.055092+0000", 2024-04-23T16:55:57.747 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5330000000000001e-06 2024-04-23T16:55:57.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.747 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.747 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.747 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179874+0000", 2024-04-23T16:55:57.747 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.12478246 2024-04-23T16:55:57.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.747 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.747 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:55:57.748 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179881+0000", 2024-04-23T16:55:57.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9639999999999997e-06 2024-04-23T16:55:57.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.748 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.748 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.748 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179981+0000", 2024-04-23T16:55:57.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9995999999999996e-05 2024-04-23T16:55:57.748 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.748 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.748 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.748 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.748 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.27202.0:256095 9.f 9:fb7c9d9b:::benchmark_data_smithi188_167360_object104962:head [delete] snapc 0=[] ondisk+write+known_if_redirected e765)", 2024-04-23T16:55:57.748 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:54:49.055618+0000", 2024-04-23T16:55:57.748 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.016158442999995, 2024-04-23T16:55:57.749 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.13722469200000001, 2024-04-23T16:55:57.749 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.749 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.749 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:57.749 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-23T16:55:57.749 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.27202", 2024-04-23T16:55:57.749 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "v1:172.21.15.188:0/1541473698", 2024-04-23T16:55:57.749 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 256095 2024-04-23T16:55:57.749 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.749 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.749 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.749 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.749 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.055618+0000", 2024-04-23T16:55:57.749 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.750 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.750 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.750 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.750 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.055618+0000", 2024-04-23T16:55:57.750 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.750 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.750 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.750 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.750 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.055620+0000", 2024-04-23T16:55:57.750 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8509999999999999e-06 2024-04-23T16:55:57.750 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.750 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.750 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.750 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.055625+0000", 2024-04-23T16:55:57.750 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6080000000000003e-06 2024-04-23T16:55:57.751 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.751 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.751 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.751 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179616+0000", 2024-04-23T16:55:57.751 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.12399089200000001 2024-04-23T16:55:57.751 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.751 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.751 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.751 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179631+0000", 2024-04-23T16:55:57.751 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5044e-05 2024-04-23T16:55:57.751 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.751 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.751 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.751 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179729+0000", 2024-04-23T16:55:57.752 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7739999999999996e-05 2024-04-23T16:55:57.752 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.752 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.752 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:55:57.752 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179875+0000", 2024-04-23T16:55:57.752 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145694 2024-04-23T16:55:57.752 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.752 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.752 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 9", 2024-04-23T16:55:57.752 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.180016+0000", 2024-04-23T16:55:57.752 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000141736 2024-04-23T16:55:57.752 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.752 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.752 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-23T16:55:57.752 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.192769+0000", 2024-04-23T16:55:57.753 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012752753 2024-04-23T16:55:57.753 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.753 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.753 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-23T16:55:57.753 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.192779+0000", 2024-04-23T16:55:57.753 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4129999999999995e-06 2024-04-23T16:55:57.753 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.753 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.753 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:55:57.753 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.192806+0000", 2024-04-23T16:55:57.753 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7467999999999999e-05 2024-04-23T16:55:57.753 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.753 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.753 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.754 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.192843+0000", 2024-04-23T16:55:57.754 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6492999999999998e-05 2024-04-23T16:55:57.754 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.754 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.754 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.754 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.754 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.754 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.27202.0:256105 9.2 e765/703 9:4447e7f1:::benchmark_data_smithi188_167360_object104972:head v 765'16194, mlcod=765'16192)", 2024-04-23T16:55:57.754 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-23T16:54:49.061808+0000", 2024-04-23T16:55:57.754 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.009968178999998, 2024-04-23T16:55:57.754 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.136095142, 2024-04-23T16:55:57.754 INFO:teuthology.orchestra.run.smithi103.stdout: "continuous": false, 2024-04-23T16:55:57.754 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-23T16:55:57.754 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T16:55:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-23T16:55:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-23T16:55:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.061808+0000", 2024-04-23T16:55:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-23T16:55:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.061808+0000", 2024-04-23T16:55:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-23T16:55:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-23T16:55:57.755 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.061811+0000", 2024-04-23T16:55:57.756 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6800000000000002e-06 2024-04-23T16:55:57.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.756 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.756 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-23T16:55:57.756 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.061824+0000", 2024-04-23T16:55:57.756 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2795e-05 2024-04-23T16:55:57.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.756 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.756 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-23T16:55:57.756 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179642+0000", 2024-04-23T16:55:57.756 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.117818161 2024-04-23T16:55:57.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.756 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.756 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-23T16:55:57.756 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179652+0000", 2024-04-23T16:55:57.757 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.842e-06 2024-04-23T16:55:57.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.757 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.757 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-23T16:55:57.757 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179701+0000", 2024-04-23T16:55:57.757 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9869e-05 2024-04-23T16:55:57.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.757 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.757 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-23T16:55:57.757 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.179722+0000", 2024-04-23T16:55:57.757 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0220999999999998e-05 2024-04-23T16:55:57.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.757 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.757 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-23T16:55:57.757 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.197864+0000", 2024-04-23T16:55:57.758 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018142393999999999 2024-04-23T16:55:57.758 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:55:57.758 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:55:57.758 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-23T16:55:57.758 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-23T16:54:49.197903+0000", 2024-04-23T16:55:57.758 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9180000000000001e-05 2024-04-23T16:55:57.758 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.758 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.758 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.758 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:55:57.758 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:55:57.758 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:55:58.222 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-23T16:55:58.222 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:58.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:58 smithi103 ceph-mon[39629]: pgmap v2173: 11 pgs: 2 stale+active+clean, 9 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:58.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:58 smithi103 ceph-mon[39629]: osdmap e783: 12 total, 11 up, 8 in 2024-04-23T16:55:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:58 smithi188 ceph-mon[44101]: pgmap v2173: 11 pgs: 2 stale+active+clean, 9 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:58.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:58 smithi188 ceph-mon[44101]: osdmap e783: 12 total, 11 up, 8 in 2024-04-23T16:55:58.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:58 smithi148 ceph-mon[44520]: pgmap v2173: 11 pgs: 2 stale+active+clean, 9 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:55:58.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:58 smithi148 ceph-mon[44520]: osdmap e783: 12 total, 11 up, 8 in 2024-04-23T16:55:59.399 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:55:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:59 smithi188 ceph-mon[44101]: pgmap v2175: 11 pgs: 1 undersized+degraded+peered, 1 stale+active+clean, 9 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:55:59.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:55:59 smithi188 ceph-mon[44101]: Health check failed: Degraded data redundancy: 1/4 objects degraded (25.000%), 1 pg degraded (PG_DEGRADED) 2024-04-23T16:55:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:59 smithi103 ceph-mon[39629]: pgmap v2175: 11 pgs: 1 undersized+degraded+peered, 1 stale+active+clean, 9 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:55:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:55:59 smithi103 ceph-mon[39629]: Health check failed: Degraded data redundancy: 1/4 objects degraded (25.000%), 1 pg degraded (PG_DEGRADED) 2024-04-23T16:55:59.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:59 smithi148 ceph-mon[44520]: pgmap v2175: 11 pgs: 1 undersized+degraded+peered, 1 stale+active+clean, 9 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:55:59.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:55:59 smithi148 ceph-mon[44520]: Health check failed: Degraded data redundancy: 1/4 objects degraded (25.000%), 1 pg degraded (PG_DEGRADED) 2024-04-23T16:56:00.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:00 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:56:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:00 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:56:00.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:00 smithi103 ceph-mon[39629]: osdmap e784: 12 total, 11 up, 8 in 2024-04-23T16:56:00.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:00 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:56:00.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:00 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:56:00.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:00 smithi148 ceph-mon[44520]: osdmap e784: 12 total, 11 up, 8 in 2024-04-23T16:56:00.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:00 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:56:00.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:00 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:56:00.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:00 smithi188 ceph-mon[44101]: osdmap e784: 12 total, 11 up, 8 in 2024-04-23T16:56:01.801 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:01 smithi103 ceph-mon[39629]: pgmap v2177: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:01.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:01 smithi148 ceph-mon[44520]: pgmap v2177: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:01.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:01 smithi188 ceph-mon[44101]: pgmap v2177: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:02.126 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.2.service' 2024-04-23T16:56:02.167 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T16:56:02.167 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-23T16:56:02.167 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd down 2 2024-04-23T16:56:02.551 INFO:teuthology.orchestra.run.smithi103.stderr:nodeep-scrub is unset 2024-04-23T16:56:02.713 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:56:02.713 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:56:02.713 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-23T16:56:02.713 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:56:02.730 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:02 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3278947622' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:56:02.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:02 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3278947622' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:56:02.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:02 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3278947622' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-23T16:56:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:03 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3278947622' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:56:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:03 smithi103 ceph-mon[39629]: osdmap e785: 12 total, 11 up, 8 in 2024-04-23T16:56:03.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:03 smithi103 ceph-mon[39629]: pgmap v2179: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:03.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:03 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3278947622' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:56:03.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:03 smithi148 ceph-mon[44520]: osdmap e785: 12 total, 11 up, 8 in 2024-04-23T16:56:03.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:03 smithi148 ceph-mon[44520]: pgmap v2179: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:03.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:03 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3278947622' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-23T16:56:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:03 smithi188 ceph-mon[44101]: osdmap e785: 12 total, 11 up, 8 in 2024-04-23T16:56:03.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:03 smithi188 ceph-mon[44101]: pgmap v2179: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:04.001 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:56:04.035 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-23T16:56:05.215 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:56:05.741 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:05 smithi103 ceph-mon[39629]: osdmap e786: 12 total, 11 up, 8 in 2024-04-23T16:56:05.742 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:05 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:56:05.742 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:05 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:56:05.742 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:05 smithi103 ceph-mon[39629]: pgmap v2181: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:05.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:05 smithi148 ceph-mon[44520]: osdmap e786: 12 total, 11 up, 8 in 2024-04-23T16:56:05.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:05 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:56:05.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:05 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:56:05.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:05 smithi148 ceph-mon[44520]: pgmap v2181: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:05.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:05 smithi188 ceph-mon[44101]: osdmap e786: 12 total, 11 up, 8 in 2024-04-23T16:56:05.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:05 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:56:05.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:05 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:56:05.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:05 smithi188 ceph-mon[44101]: pgmap v2181: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:06.499 INFO:teuthology.orchestra.run.smithi103.stderr:osd.2 is already down. 2024-04-23T16:56:06.817 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:06 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T16:56:06.495+0000 7f135b1c7700 -1 mon.a@0(leader).osd e787 definitely_dead 0 2024-04-23T16:56:06.817 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:06 smithi103 ceph-mon[39629]: osdmap e787: 12 total, 11 up, 8 in 2024-04-23T16:56:06.817 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:06 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1114242809' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["2"]}]: dispatch 2024-04-23T16:56:06.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:06 smithi148 ceph-mon[44520]: osdmap e787: 12 total, 11 up, 8 in 2024-04-23T16:56:06.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:06 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1114242809' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["2"]}]: dispatch 2024-04-23T16:56:06.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:06 smithi188 ceph-mon[44101]: osdmap e787: 12 total, 11 up, 8 in 2024-04-23T16:56:06.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:06 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1114242809' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["2"]}]: dispatch 2024-04-23T16:56:07.523 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:56:07.524 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:56:07.524 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-23T16:56:07.524 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-23T16:56:07.524 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:56:07.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:07 smithi103 ceph-mon[39629]: pgmap v2183: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:07.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:07 smithi148 ceph-mon[44520]: pgmap v2183: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:07.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:07 smithi188 ceph-mon[44101]: pgmap v2183: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:08.183 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 6, 7, 8, 9, 11, 1, 2] 2024-04-23T16:56:08.184 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd out 2 2024-04-23T16:56:08.685 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-23T16:56:08.783 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:56:10.010 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:56:10.268 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:10 smithi103 ceph-mon[39629]: pgmap v2184: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:10.268 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:10 smithi103 ceph-mon[39629]: osdmap e788: 12 total, 11 up, 8 in 2024-04-23T16:56:10.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:10 smithi148 ceph-mon[44520]: pgmap v2184: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:10.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:10 smithi148 ceph-mon[44520]: osdmap e788: 12 total, 11 up, 8 in 2024-04-23T16:56:10.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:10 smithi188 ceph-mon[44101]: pgmap v2184: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:10.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:10 smithi188 ceph-mon[44101]: osdmap e788: 12 total, 11 up, 8 in 2024-04-23T16:56:11.081 INFO:teuthology.orchestra.run.smithi103.stderr:marked out osd.2. 2024-04-23T16:56:11.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:11 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T16:56:11.063+0000 7f135b1c7700 -1 mon.a@0(leader).osd e788 definitely_dead 0 2024-04-23T16:56:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:11 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:56:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:11 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:56:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:11 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1777973103' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-23T16:56:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:11 smithi103 ceph-mon[39629]: Client client.admin marked osd.2 out, after it was down for 14 seconds 2024-04-23T16:56:11.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:11 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:56:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:11 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:56:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:11 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1777973103' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-23T16:56:11.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:11 smithi148 ceph-mon[44520]: Client client.admin marked osd.2 out, after it was down for 14 seconds 2024-04-23T16:56:11.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:11 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:56:11.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:11 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:56:11.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:11 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1777973103' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-23T16:56:11.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:11 smithi188 ceph-mon[44101]: Client client.admin marked osd.2 out, after it was down for 14 seconds 2024-04-23T16:56:12.292 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:56:12.293 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-23T16:56:12.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-23T16:56:12.293 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-23T16:56:12.293 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:56:12.316 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:12 smithi103 ceph-mon[39629]: pgmap v2186: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:12.316 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:12 smithi103 ceph-mon[39629]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T16:56:12.316 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:12 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1777973103' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-04-23T16:56:12.316 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:12 smithi103 ceph-mon[39629]: osdmap e789: 12 total, 11 up, 7 in 2024-04-23T16:56:12.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:12 smithi148 ceph-mon[44520]: pgmap v2186: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:12.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:12 smithi148 ceph-mon[44520]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T16:56:12.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:12 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1777973103' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-04-23T16:56:12.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:12 smithi148 ceph-mon[44520]: osdmap e789: 12 total, 11 up, 7 in 2024-04-23T16:56:12.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:12 smithi188 ceph-mon[44101]: pgmap v2186: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:12.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:12 smithi188 ceph-mon[44101]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T16:56:12.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:12 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1777973103' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-04-23T16:56:12.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:12 smithi188 ceph-mon[44101]: osdmap e789: 12 total, 11 up, 7 in 2024-04-23T16:56:13.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:13 smithi103 ceph-mon[39629]: osdmap e790: 12 total, 11 up, 7 in 2024-04-23T16:56:13.329 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-23T16:56:13.361 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:13 smithi148 ceph-mon[44520]: osdmap e790: 12 total, 11 up, 7 in 2024-04-23T16:56:13.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:13 smithi188 ceph-mon[44101]: osdmap e790: 12 total, 11 up, 7 in 2024-04-23T16:56:13.606 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:56:13.639 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-04-23T16:56:13.639 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi103.front.sepia.ceph.com 2024-04-23T16:56:13.639 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.2 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-04-23T16:56:13.850 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.2/config 2024-04-23T16:56:14.308 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:14 smithi103 ceph-mon[39629]: pgmap v2189: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:14.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:14 smithi148 ceph-mon[44520]: pgmap v2189: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:14.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:14 smithi188 ceph-mon[44101]: pgmap v2189: 11 pgs: 1 undersized+degraded+peered, 2 undersized+peered, 8 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1/4 objects degraded (25.000%) 2024-04-23T16:56:15.057 INFO:teuthology.orchestra.run.smithi103.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:56:15.569 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:56:15.569 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:56:15.569 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2024-04-23T16:56:15.570 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:56:15.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:15 smithi103 ceph-mon[39629]: pgmap v2190: 11 pgs: 2 undersized+peered, 9 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 33 KiB/s, 0 objects/s recovering 2024-04-23T16:56:15.719 INFO:teuthology.orchestra.run.smithi103.stdout:1.5 2024-04-23T16:56:15.719 INFO:teuthology.orchestra.run.smithi103.stdout:1.8 2024-04-23T16:56:15.719 INFO:teuthology.orchestra.run.smithi103.stdout:1.7 2024-04-23T16:56:15.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:15 smithi148 ceph-mon[44520]: pgmap v2190: 11 pgs: 2 undersized+peered, 9 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 33 KiB/s, 0 objects/s recovering 2024-04-23T16:56:15.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:15 smithi188 ceph-mon[44101]: pgmap v2190: 11 pgs: 2 undersized+peered, 9 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 33 KiB/s, 0 objects/s recovering 2024-04-23T16:56:16.341 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-23T16:56:16.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:16 smithi148 ceph-mon[44520]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/4 objects degraded (25.000%), 1 pg degraded) 2024-04-23T16:56:16.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:16 smithi148 ceph-mon[44520]: Cluster is now healthy 2024-04-23T16:56:16.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:16 smithi148 ceph-mon[44520]: 1.7 deep-scrub starts 2024-04-23T16:56:16.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:16 smithi148 ceph-mon[44520]: 1.7 deep-scrub ok 2024-04-23T16:56:16.648 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:56:16.800 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:16 smithi103 ceph-mon[39629]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/4 objects degraded (25.000%), 1 pg degraded) 2024-04-23T16:56:16.801 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:16 smithi103 ceph-mon[39629]: Cluster is now healthy 2024-04-23T16:56:16.801 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:16 smithi103 ceph-mon[39629]: 1.7 deep-scrub starts 2024-04-23T16:56:16.801 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:16 smithi103 ceph-mon[39629]: 1.7 deep-scrub ok 2024-04-23T16:56:16.826 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.2 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 1.5 --file /var/log/ceph/exp.1.5.2 2024-04-23T16:56:16.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:16 smithi188 ceph-mon[44101]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/4 objects degraded (25.000%), 1 pg degraded) 2024-04-23T16:56:16.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:16 smithi188 ceph-mon[44101]: Cluster is now healthy 2024-04-23T16:56:16.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:16 smithi188 ceph-mon[44101]: 1.7 deep-scrub starts 2024-04-23T16:56:16.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:16 smithi188 ceph-mon[44101]: 1.7 deep-scrub ok 2024-04-23T16:56:17.037 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.2/config 2024-04-23T16:56:17.814 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:17 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:56:17.814 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:17 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:56:17.814 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:17 smithi148 ceph-mon[44520]: pgmap v2191: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 26 KiB/s, 0 objects/s recovering 2024-04-23T16:56:17.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:17 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:56:17.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:17 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:56:17.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:17 smithi103 ceph-mon[39629]: pgmap v2191: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 26 KiB/s, 0 objects/s recovering 2024-04-23T16:56:17.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:17 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:56:17.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:17 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:56:17.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:17 smithi188 ceph-mon[44101]: pgmap v2191: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 26 KiB/s, 0 objects/s recovering 2024-04-23T16:56:18.301 INFO:teuthology.orchestra.run.smithi103.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:56:18.391 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:56:18.391 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2024-04-23T16:56:18.391 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2024-04-23T16:56:18.391 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2024-04-23T16:56:18.391 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:56:18.917 INFO:teuthology.orchestra.run.smithi103.stderr:Exporting 1.5 info 1.5( v 255'139 (0'0,255'139] local-lis/les=735/736 n=0 ec=301/15 lis/c=735/735 les/c/f=736/736/0 sis=735) 2024-04-23T16:56:18.969 INFO:teuthology.orchestra.run.smithi103.stderr:Export successful 2024-04-23T16:56:19.014 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-23T16:56:19.280 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T16:56:20.287 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.2 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 1.5 2024-04-23T16:56:20.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:20 smithi103 ceph-mon[39629]: pgmap v2192: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 25 KiB/s, 0 objects/s recovering 2024-04-23T16:56:20.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:20 smithi148 ceph-mon[44520]: pgmap v2192: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 25 KiB/s, 0 objects/s recovering 2024-04-23T16:56:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:20 smithi188 ceph-mon[44101]: pgmap v2192: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 25 KiB/s, 0 objects/s recovering 2024-04-23T16:56:20.499 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.2/config 2024-04-23T16:56:21.027 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-04-23T16:56:21.027 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2024-04-23T16:56:21.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2024-04-23T16:56:21.027 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [] 2024-04-23T16:56:21.027 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-04-23T16:56:21.772 INFO:teuthology.orchestra.run.smithi103.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:56:22.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:22 smithi103 ceph-mon[39629]: pgmap v2193: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 20 KiB/s, 0 objects/s recovering 2024-04-23T16:56:22.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:22 smithi148 ceph-mon[44520]: pgmap v2193: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 20 KiB/s, 0 objects/s recovering 2024-04-23T16:56:22.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:22 smithi188 ceph-mon[44101]: pgmap v2193: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 20 KiB/s, 0 objects/s recovering 2024-04-23T16:56:22.676 INFO:teuthology.orchestra.run.smithi103.stdout: marking collection for removal 2024-04-23T16:56:22.676 INFO:teuthology.orchestra.run.smithi103.stdout:setting '_remove' omap key 2024-04-23T16:56:22.676 INFO:teuthology.orchestra.run.smithi103.stdout:finish_remove_pgs 1.5_head removing 1.5 2024-04-23T16:56:22.742 INFO:teuthology.orchestra.run.smithi103.stdout:Remove successful 2024-04-23T16:56:23.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:23 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:56:23.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:23 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:56:23.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:23 smithi103 ceph-mon[39629]: osdmap e791: 12 total, 11 up, 7 in 2024-04-23T16:56:23.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:23 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:56:23.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:23 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:56:23.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:23 smithi148 ceph-mon[44520]: osdmap e791: 12 total, 11 up, 7 in 2024-04-23T16:56:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:23 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:56:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:23 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:56:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:23 smithi188 ceph-mon[44101]: osdmap e791: 12 total, 11 up, 7 in 2024-04-23T16:56:23.974 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell -n osd.2 --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.1.5.2 2024-04-23T16:56:24.210 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/osd.2/config 2024-04-23T16:56:24.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:24 smithi103 ceph-mon[39629]: pgmap v2195: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 20 KiB/s, 0 objects/s recovering 2024-04-23T16:56:24.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:24 smithi148 ceph-mon[44520]: pgmap v2195: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 20 KiB/s, 0 objects/s recovering 2024-04-23T16:56:24.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:24 smithi188 ceph-mon[44101]: pgmap v2195: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 20 KiB/s, 0 objects/s recovering 2024-04-23T16:56:25.520 INFO:teuthology.orchestra.run.smithi103.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-23T16:56:25.780 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:25 smithi103 ceph-mon[39629]: osdmap e792: 12 total, 11 up, 7 in 2024-04-23T16:56:25.780 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:25 smithi103 ceph-mon[39629]: pgmap v2197: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:25.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:25 smithi148 ceph-mon[44520]: osdmap e792: 12 total, 11 up, 7 in 2024-04-23T16:56:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:25 smithi148 ceph-mon[44520]: pgmap v2197: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:25.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:25 smithi188 ceph-mon[44101]: osdmap e792: 12 total, 11 up, 7 in 2024-04-23T16:56:25.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:25 smithi188 ceph-mon[44101]: pgmap v2197: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:26.230 INFO:teuthology.orchestra.run.smithi103.stdout:get_pg_num_history pg_num_history pg_num_history(e780 pg_nums {1={301=11},9={703=16}} deleted_pools 779,9) 2024-04-23T16:56:26.230 INFO:teuthology.orchestra.run.smithi103.stdout:Importing pgid 1.5 2024-04-23T16:56:26.230 INFO:teuthology.orchestra.run.smithi103.stdout:write_pg epoch 772 info 1.5( v 255'139 (0'0,255'139] local-lis/les=735/736 n=0 ec=301/15 lis/c=735/735 les/c/f=736/736/0 sis=735) 2024-04-23T16:56:26.230 INFO:teuthology.orchestra.run.smithi103.stdout:Import successful 2024-04-23T16:56:26.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:26 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:56:26.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:26 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:56:26.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:26 smithi103 ceph-mon[39629]: osdmap e793: 12 total, 11 up, 7 in 2024-04-23T16:56:26.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:26 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:56:26.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:26 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:56:26.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:26 smithi148 ceph-mon[44520]: osdmap e793: 12 total, 11 up, 7 in 2024-04-23T16:56:26.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:26 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:56:26.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:26 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:56:26.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:26 smithi188 ceph-mon[44101]: osdmap e793: 12 total, 11 up, 7 in 2024-04-23T16:56:27.505 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/exp.1.5.2 2024-04-23T16:56:27.540 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-04-23T16:56:27.540 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-23T16:56:27.540 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:56:27.844 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:56:27.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:27 smithi103 ceph-mon[39629]: pgmap v2199: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:27.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:27 smithi148 ceph-mon[44520]: pgmap v2199: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:27.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:27 smithi188 ceph-mon[44101]: pgmap v2199: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:29.839 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:56:29.840 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:56:30.223 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:30 smithi103 ceph-mon[39629]: pgmap v2200: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:30.223 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:30 smithi103 ceph-mon[39629]: osdmap e794: 12 total, 11 up, 7 in 2024-04-23T16:56:30.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:30 smithi148 ceph-mon[44520]: pgmap v2200: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:30.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:30 smithi148 ceph-mon[44520]: osdmap e794: 12 total, 11 up, 7 in 2024-04-23T16:56:30.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:30 smithi188 ceph-mon[44101]: pgmap v2200: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:30 smithi188 ceph-mon[44101]: osdmap e794: 12 total, 11 up, 7 in 2024-04-23T16:56:30.672 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":2201,"stamp":"2024-04-23T16:56:29.547005+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":804992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":5256728,"kb_used_data":7740,"kb_used_omap":240,"kb_used_meta":5248655,"kb_avail":650959336,"statfs":{"total":671965249536,"available":666582360064,"internally_reserved":0,"allocated":7925760,"data_stored":4412144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":246226,"internal_metadata":5374623278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.525878"},"pg_stats":[{"pgid":"1.a","version":"255'139","reported_seq":1949,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-04-23T16:56:24.548231+0000","last_change":"2024-04-23T16:55:43.611889+0000","last_active":"2024-04-23T16:56:24.548231+0000","last_peered":"2024-04-23T16:56:24.548231+0000","last_clean":"2024-04-23T16:56:24.548231+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:56:24.548231+0000","last_undegraded":"2024-04-23T16:56:24.548231+0000","last_fullsized":"2024-04-23T16:56:24.548231+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:43.611781+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:54:28.362406+0000","last_clean_scrub_stamp":"2024-04-23T16:55:43.611781+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:57:01.354127+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1726,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-04-23T16:56:25.561302+0000","last_change":"2024-04-23T16:56:12.333729+0000","last_active":"2024-04-23T16:56:25.561302+0000","last_peered":"2024-04-23T16:56:25.561302+0000","last_clean":"2024-04-23T16:56:25.561302+0000","last_became_active":"2024-04-23T16:56:12.331280+0000","last_became_peered":"2024-04-23T16:56:12.331280+0000","last_unstale":"2024-04-23T16:56:25.561302+0000","last_undegraded":"2024-04-23T16:56:25.561302+0000","last_fullsized":"2024-04-23T16:56:25.561302+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:10.720770+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:52:05.504325+0000","last_clean_scrub_stamp":"2024-04-23T16:55:10.720770+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":804992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1934,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-04-23T16:56:25.765022+0000","last_change":"2024-04-23T16:55:20.164898+0000","last_active":"2024-04-23T16:56:25.765022+0000","last_peered":"2024-04-23T16:56:25.765022+0000","last_clean":"2024-04-23T16:56:25.765022+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:56:25.765022+0000","last_undegraded":"2024-04-23T16:56:25.765022+0000","last_fullsized":"2024-04-23T16:56:25.765022+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:20.164813+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:51:25.290774+0000","last_clean_scrub_stamp":"2024-04-23T16:55:20.164813+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:30.353917+0000","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1956,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-04-23T16:56:25.593422+0000","last_change":"2024-04-23T16:56:16.798938+0000","last_active":"2024-04-23T16:56:25.593422+0000","last_peered":"2024-04-23T16:56:25.593422+0000","last_clean":"2024-04-23T16:56:25.593422+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:56:25.593422+0000","last_undegraded":"2024-04-23T16:56:25.593422+0000","last_fullsized":"2024-04-23T16:56:25.593422+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:56:16.798777+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:56:16.798777+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:57:19.724583+0000","scrub_duration":173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1532,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-04-23T16:56:25.764548+0000","last_change":"2024-04-23T16:55:02.927584+0000","last_active":"2024-04-23T16:56:25.764548+0000","last_peered":"2024-04-23T16:56:25.764548+0000","last_clean":"2024-04-23T16:56:25.764548+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:56:25.764548+0000","last_undegraded":"2024-04-23T16:56:25.764548+0000","last_fullsized":"2024-04-23T16:56:25.764548+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:02.927497+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:55:02.927497+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:29.690298+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1494,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-04-23T16:56:25.561352+0000","last_change":"2024-04-23T16:55:03.259007+0000","last_active":"2024-04-23T16:56:25.561352+0000","last_peered":"2024-04-23T16:56:25.561352+0000","last_clean":"2024-04-23T16:56:25.561352+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:56:25.561352+0000","last_undegraded":"2024-04-23T16:56:25.561352+0000","last_fullsized":"2024-04-23T16:56:25.561352+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:03.258896+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:55:03.258896+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:32.586872+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1957,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-04-23T16:56:24.548239+0000","last_change":"2024-04-23T16:55:47.572854+0000","last_active":"2024-04-23T16:56:24.548239+0000","last_peered":"2024-04-23T16:56:24.548239+0000","last_clean":"2024-04-23T16:56:24.548239+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:56:24.548239+0000","last_undegraded":"2024-04-23T16:56:24.548239+0000","last_fullsized":"2024-04-23T16:56:24.548239+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:47.572718+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:55:47.572718+0000","last_clean_scrub_stamp":"2024-04-23T16:55:47.572718+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:50.613849+0000","scrub_duration":208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1974,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-04-23T16:56:25.194816+0000","last_change":"2024-04-23T16:56:16.211012+0000","last_active":"2024-04-23T16:56:25.194816+0000","last_peered":"2024-04-23T16:56:25.194816+0000","last_clean":"2024-04-23T16:56:25.194816+0000","last_became_active":"2024-04-23T16:56:12.330440+0000","last_became_peered":"2024-04-23T16:56:12.330440+0000","last_unstale":"2024-04-23T16:56:25.194816+0000","last_undegraded":"2024-04-23T16:56:25.194816+0000","last_fullsized":"2024-04-23T16:56:25.194816+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:56:16.210909+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:56:16.210909+0000","last_clean_scrub_stamp":"2024-04-23T16:56:16.210909+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:57:24.312789+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1842,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-04-23T16:56:25.765425+0000","last_change":"2024-04-23T16:55:42.647134+0000","last_active":"2024-04-23T16:56:25.765425+0000","last_peered":"2024-04-23T16:56:25.765425+0000","last_clean":"2024-04-23T16:56:25.765425+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:56:25.765425+0000","last_undegraded":"2024-04-23T16:56:25.765425+0000","last_fullsized":"2024-04-23T16:56:25.765425+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:42.646999+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:55:42.646999+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:57:00.484062+0000","scrub_duration":257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1671,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-04-23T16:56:25.194798+0000","last_change":"2024-04-23T16:56:12.305689+0000","last_active":"2024-04-23T16:56:25.194798+0000","last_peered":"2024-04-23T16:56:25.194798+0000","last_clean":"2024-04-23T16:56:25.194798+0000","last_became_active":"2024-04-23T16:56:12.304352+0000","last_became_peered":"2024-04-23T16:56:12.304352+0000","last_unstale":"2024-04-23T16:56:25.194798+0000","last_undegraded":"2024-04-23T16:56:25.194798+0000","last_fullsized":"2024-04-23T16:56:25.194798+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:11.687173+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:55:11.687173+0000","last_clean_scrub_stamp":"2024-04-23T16:55:11.687173+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:40.509488+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1927,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-04-23T16:56:25.765021+0000","last_change":"2024-04-23T16:55:18.211396+0000","last_active":"2024-04-23T16:56:25.765021+0000","last_peered":"2024-04-23T16:56:25.765021+0000","last_clean":"2024-04-23T16:56:25.765021+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:56:25.765021+0000","last_undegraded":"2024-04-23T16:56:25.765021+0000","last_fullsized":"2024-04-23T16:56:25.765021+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:18.211281+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:51:31.339728+0000","last_clean_scrub_stamp":"2024-04-23T16:55:18.211281+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:28.248187+0000","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":804992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":1862848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057134,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1087568,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":1086270,"kb_avail":92657584,"statfs":{"total":95995035648,"available":94881366016,"internally_reserved":0,"allocated":1314816,"data_stored":810232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1112340932},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351476,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613113,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852600,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892552,"statfs":{"total":95995035648,"available":95121973248,"internally_reserved":0,"allocated":962560,"data_stored":470764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":872085952},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467842046,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596316,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93148836,"statfs":{"total":95995035648,"available":95384408064,"internally_reserved":0,"allocated":1519616,"data_stored":1011480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169092,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":740180,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93004972,"statfs":{"total":95995035648,"available":95237091328,"internally_reserved":0,"allocated":987136,"data_stored":482552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":756939210},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463432,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":605496,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93139656,"statfs":{"total":95995035648,"available":95375007744,"internally_reserved":0,"allocated":962560,"data_stored":470764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":619051462},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137573,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":624464,"kb_used_data":964,"kb_used_omap":231,"kb_used_meta":623256,"kb_avail":93120688,"statfs":{"total":95995035648,"available":95355584512,"internally_reserved":0,"allocated":987136,"data_stored":482552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":638215056},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738919,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":750104,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":92995048,"statfs":{"total":95995035648,"available":95226929152,"internally_reserved":0,"allocated":1191936,"data_stored":683800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":766900677},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2658584756339,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":3126736191538,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757772,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:56:30.674 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T16:56:30.942 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:56:31.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:31 smithi103 ceph-mon[39629]: from='client.17859 v1:172.21.15.103:0/2275105533' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:56:31.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:31 smithi103 ceph-mon[39629]: 1.8 deep-scrub starts 2024-04-23T16:56:31.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:31 smithi103 ceph-mon[39629]: 1.8 deep-scrub ok 2024-04-23T16:56:31.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:31 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:56:31.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:31 smithi148 ceph-mon[44520]: from='client.17859 v1:172.21.15.103:0/2275105533' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:56:31.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:31 smithi148 ceph-mon[44520]: 1.8 deep-scrub starts 2024-04-23T16:56:31.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:31 smithi148 ceph-mon[44520]: 1.8 deep-scrub ok 2024-04-23T16:56:31.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:31 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:56:31.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:31 smithi188 ceph-mon[44101]: from='client.17859 v1:172.21.15.103:0/2275105533' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:56:31.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:31 smithi188 ceph-mon[44101]: 1.8 deep-scrub starts 2024-04-23T16:56:31.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:31 smithi188 ceph-mon[44101]: 1.8 deep-scrub ok 2024-04-23T16:56:31.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:31 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:56:32.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:32 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:56:32.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:32 smithi103 ceph-mon[39629]: pgmap v2202: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:32 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:56:32.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:32 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:56:32.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:32 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:56:32.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:32 smithi148 ceph-mon[44520]: pgmap v2202: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:32.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:32 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:56:32.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:32 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:56:32.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:32 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:56:32.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:32 smithi188 ceph-mon[44101]: pgmap v2202: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:32.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:32 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:56:32.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:32 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:56:32.938 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:56:32.940 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T16:56:33.707 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":2202,"stamp":"2024-04-23T16:56:31.022690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":804992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":5256780,"kb_used_data":7788,"kb_used_omap":240,"kb_used_meta":5248655,"kb_avail":650959284,"statfs":{"total":671965249536,"available":666582306816,"internally_reserved":0,"allocated":7974912,"data_stored":4435720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":246226,"internal_metadata":5374623278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.478047"},"pg_stats":[{"pgid":"1.a","version":"255'139","reported_seq":1953,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-04-23T16:56:29.548888+0000","last_change":"2024-04-23T16:55:43.611889+0000","last_active":"2024-04-23T16:56:29.548888+0000","last_peered":"2024-04-23T16:56:29.548888+0000","last_clean":"2024-04-23T16:56:29.548888+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T16:56:29.548888+0000","last_undegraded":"2024-04-23T16:56:29.548888+0000","last_fullsized":"2024-04-23T16:56:29.548888+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:43.611781+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:54:28.362406+0000","last_clean_scrub_stamp":"2024-04-23T16:55:43.611781+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:57:01.354127+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1726,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-04-23T16:56:25.561302+0000","last_change":"2024-04-23T16:56:12.333729+0000","last_active":"2024-04-23T16:56:25.561302+0000","last_peered":"2024-04-23T16:56:25.561302+0000","last_clean":"2024-04-23T16:56:25.561302+0000","last_became_active":"2024-04-23T16:56:12.331280+0000","last_became_peered":"2024-04-23T16:56:12.331280+0000","last_unstale":"2024-04-23T16:56:25.561302+0000","last_undegraded":"2024-04-23T16:56:25.561302+0000","last_fullsized":"2024-04-23T16:56:25.561302+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:10.720770+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:52:05.504325+0000","last_clean_scrub_stamp":"2024-04-23T16:55:10.720770+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":804992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":1934,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-04-23T16:56:25.765022+0000","last_change":"2024-04-23T16:55:20.164898+0000","last_active":"2024-04-23T16:56:25.765022+0000","last_peered":"2024-04-23T16:56:25.765022+0000","last_clean":"2024-04-23T16:56:25.765022+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T16:56:25.765022+0000","last_undegraded":"2024-04-23T16:56:25.765022+0000","last_fullsized":"2024-04-23T16:56:25.765022+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:20.164813+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:51:25.290774+0000","last_clean_scrub_stamp":"2024-04-23T16:55:20.164813+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:30.353917+0000","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":1956,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-04-23T16:56:25.593422+0000","last_change":"2024-04-23T16:56:16.798938+0000","last_active":"2024-04-23T16:56:25.593422+0000","last_peered":"2024-04-23T16:56:25.593422+0000","last_clean":"2024-04-23T16:56:25.593422+0000","last_became_active":"2024-04-23T16:28:25.061655+0000","last_became_peered":"2024-04-23T16:28:25.061655+0000","last_unstale":"2024-04-23T16:56:25.593422+0000","last_undegraded":"2024-04-23T16:56:25.593422+0000","last_fullsized":"2024-04-23T16:56:25.593422+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:56:16.798777+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:25:52.560053+0000","last_clean_scrub_stamp":"2024-04-23T16:56:16.798777+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:57:19.724583+0000","scrub_duration":173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1532,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-04-23T16:56:25.764548+0000","last_change":"2024-04-23T16:55:02.927584+0000","last_active":"2024-04-23T16:56:25.764548+0000","last_peered":"2024-04-23T16:56:25.764548+0000","last_clean":"2024-04-23T16:56:25.764548+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T16:56:25.764548+0000","last_undegraded":"2024-04-23T16:56:25.764548+0000","last_fullsized":"2024-04-23T16:56:25.764548+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:02.927497+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T16:55:02.927497+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:29.690298+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":1494,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-04-23T16:56:25.561352+0000","last_change":"2024-04-23T16:55:03.259007+0000","last_active":"2024-04-23T16:56:25.561352+0000","last_peered":"2024-04-23T16:56:25.561352+0000","last_clean":"2024-04-23T16:56:25.561352+0000","last_became_active":"2024-04-23T16:40:16.944778+0000","last_became_peered":"2024-04-23T16:40:16.944778+0000","last_unstale":"2024-04-23T16:56:25.561352+0000","last_undegraded":"2024-04-23T16:56:25.561352+0000","last_fullsized":"2024-04-23T16:56:25.561352+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:03.258896+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:55:03.258896+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:32.586872+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":1961,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-04-23T16:56:29.548888+0000","last_change":"2024-04-23T16:55:47.572854+0000","last_active":"2024-04-23T16:56:29.548888+0000","last_peered":"2024-04-23T16:56:29.548888+0000","last_clean":"2024-04-23T16:56:29.548888+0000","last_became_active":"2024-04-23T16:28:25.056950+0000","last_became_peered":"2024-04-23T16:28:25.056950+0000","last_unstale":"2024-04-23T16:56:29.548888+0000","last_undegraded":"2024-04-23T16:56:29.548888+0000","last_fullsized":"2024-04-23T16:56:29.548888+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:47.572718+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:55:47.572718+0000","last_clean_scrub_stamp":"2024-04-23T16:55:47.572718+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:50.613849+0000","scrub_duration":208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":1978,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-04-23T16:56:30.363891+0000","last_change":"2024-04-23T16:56:16.211012+0000","last_active":"2024-04-23T16:56:30.363891+0000","last_peered":"2024-04-23T16:56:30.363891+0000","last_clean":"2024-04-23T16:56:30.363891+0000","last_became_active":"2024-04-23T16:56:12.330440+0000","last_became_peered":"2024-04-23T16:56:12.330440+0000","last_unstale":"2024-04-23T16:56:30.363891+0000","last_undegraded":"2024-04-23T16:56:30.363891+0000","last_fullsized":"2024-04-23T16:56:30.363891+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:56:16.210909+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:56:16.210909+0000","last_clean_scrub_stamp":"2024-04-23T16:56:16.210909+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:57:24.312789+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":1842,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-04-23T16:56:25.765425+0000","last_change":"2024-04-23T16:55:42.647134+0000","last_active":"2024-04-23T16:56:25.765425+0000","last_peered":"2024-04-23T16:56:25.765425+0000","last_clean":"2024-04-23T16:56:25.765425+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T16:56:25.765425+0000","last_undegraded":"2024-04-23T16:56:25.765425+0000","last_fullsized":"2024-04-23T16:56:25.765425+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:42.646999+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:55:42.646999+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:57:00.484062+0000","scrub_duration":257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1675,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-04-23T16:56:30.363792+0000","last_change":"2024-04-23T16:56:12.305689+0000","last_active":"2024-04-23T16:56:30.363792+0000","last_peered":"2024-04-23T16:56:30.363792+0000","last_clean":"2024-04-23T16:56:30.363792+0000","last_became_active":"2024-04-23T16:56:12.304352+0000","last_became_peered":"2024-04-23T16:56:12.304352+0000","last_unstale":"2024-04-23T16:56:30.363792+0000","last_undegraded":"2024-04-23T16:56:30.363792+0000","last_fullsized":"2024-04-23T16:56:30.363792+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:11.687173+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:55:11.687173+0000","last_clean_scrub_stamp":"2024-04-23T16:55:11.687173+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:40.509488+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":1927,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-04-23T16:56:25.765021+0000","last_change":"2024-04-23T16:55:18.211396+0000","last_active":"2024-04-23T16:56:25.765021+0000","last_peered":"2024-04-23T16:56:25.765021+0000","last_clean":"2024-04-23T16:56:25.765021+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T16:56:25.765021+0000","last_undegraded":"2024-04-23T16:56:25.765021+0000","last_fullsized":"2024-04-23T16:56:25.765021+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:55:18.211281+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:51:31.339728+0000","last_clean_scrub_stamp":"2024-04-23T16:55:18.211281+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:28.248187+0000","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":804992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":1862848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057134,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1087568,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":1086270,"kb_avail":92657584,"statfs":{"total":95995035648,"available":94881366016,"internally_reserved":0,"allocated":1314816,"data_stored":810232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1112340932},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":347892351477,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613114,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852624,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892528,"statfs":{"total":95995035648,"available":95121948672,"internally_reserved":0,"allocated":987136,"data_stored":482552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":872085952},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467842046,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596316,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93148836,"statfs":{"total":95995035648,"available":95384408064,"internally_reserved":0,"allocated":1519616,"data_stored":1011480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169092,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":740180,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93004972,"statfs":{"total":95995035648,"available":95237091328,"internally_reserved":0,"allocated":987136,"data_stored":482552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":756939210},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463433,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":605524,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93139628,"statfs":{"total":95995035648,"available":95374979072,"internally_reserved":0,"allocated":987136,"data_stored":482552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":619051462},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137573,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":624464,"kb_used_data":964,"kb_used_omap":231,"kb_used_meta":623256,"kb_avail":93120688,"statfs":{"total":95995035648,"available":95355584512,"internally_reserved":0,"allocated":987136,"data_stored":482552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":638215056},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738919,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":750104,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":92995048,"statfs":{"total":95995035648,"available":95226929152,"internally_reserved":0,"allocated":1191936,"data_stored":683800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":766900677},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2658584756339,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":3126736191538,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757772,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T16:56:33.708 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-23T16:56:33.709 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-04-23T16:56:34.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:34 smithi103 ceph-mon[39629]: from='client.17865 v1:172.21.15.103:0/1870391048' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:56:34.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:34 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:56:34.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:34 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:56:34.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:34 smithi103 ceph-mon[39629]: pgmap v2203: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:34.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:34 smithi103 ceph-mon[39629]: osdmap e795: 12 total, 11 up, 7 in 2024-04-23T16:56:34.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:34 smithi103 ceph-mon[39629]: 1.9 deep-scrub starts 2024-04-23T16:56:34.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:34 smithi148 ceph-mon[44520]: from='client.17865 v1:172.21.15.103:0/1870391048' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:56:34.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:34 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:56:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:34 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:56:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:34 smithi148 ceph-mon[44520]: pgmap v2203: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:34 smithi148 ceph-mon[44520]: osdmap e795: 12 total, 11 up, 7 in 2024-04-23T16:56:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:34 smithi148 ceph-mon[44520]: 1.9 deep-scrub starts 2024-04-23T16:56:34.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:34 smithi188 ceph-mon[44101]: from='client.17865 v1:172.21.15.103:0/1870391048' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T16:56:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:34 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:56:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:34 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:56:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:34 smithi188 ceph-mon[44101]: pgmap v2203: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:34 smithi188 ceph-mon[44101]: osdmap e795: 12 total, 11 up, 7 in 2024-04-23T16:56:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:34 smithi188 ceph-mon[44101]: 1.9 deep-scrub starts 2024-04-23T16:56:35.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:35 smithi103 ceph-mon[39629]: 1.9 deep-scrub ok 2024-04-23T16:56:35.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:35 smithi103 ceph-mon[39629]: osdmap e796: 12 total, 11 up, 7 in 2024-04-23T16:56:35.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:35 smithi148 ceph-mon[44520]: 1.9 deep-scrub ok 2024-04-23T16:56:35.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:35 smithi148 ceph-mon[44520]: osdmap e796: 12 total, 11 up, 7 in 2024-04-23T16:56:35.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:35 smithi188 ceph-mon[44101]: 1.9 deep-scrub ok 2024-04-23T16:56:35.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:35 smithi188 ceph-mon[44101]: osdmap e796: 12 total, 11 up, 7 in 2024-04-23T16:56:36.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:36 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:56:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:36 smithi103 ceph-mon[39629]: pgmap v2206: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:36 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:56:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:36 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:56:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:36 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:56:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:36 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:56:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:36 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:56:36.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:36 smithi103 ceph-mon[39629]: osdmap e797: 12 total, 11 up, 7 in 2024-04-23T16:56:36.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:36 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:56:36.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:36 smithi148 ceph-mon[44520]: pgmap v2206: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:36 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:56:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:36 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:56:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:36 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:56:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:36 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:56:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:36 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:56:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:36 smithi148 ceph-mon[44520]: osdmap e797: 12 total, 11 up, 7 in 2024-04-23T16:56:36.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:36 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:56:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:36 smithi188 ceph-mon[44101]: pgmap v2206: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:36 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:56:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:36 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:56:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:36 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:56:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:36 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:56:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:36 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:56:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:36 smithi188 ceph-mon[44101]: osdmap e797: 12 total, 11 up, 7 in 2024-04-23T16:56:37.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:37 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:56:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:37 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:56:37.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:37 smithi103 ceph-mon[39629]: osdmap e798: 12 total, 11 up, 7 in 2024-04-23T16:56:37.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:37 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:56:37.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:37 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:56:37.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:37 smithi148 ceph-mon[44520]: osdmap e798: 12 total, 11 up, 7 in 2024-04-23T16:56:37.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:37 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:56:37.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:37 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:56:37.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:37 smithi188 ceph-mon[44101]: osdmap e798: 12 total, 11 up, 7 in 2024-04-23T16:56:38.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:38 smithi103 ceph-mon[39629]: pgmap v2209: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:38.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:38 smithi148 ceph-mon[44520]: pgmap v2209: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:38.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:38 smithi188 ceph-mon[44101]: pgmap v2209: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:40.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:40 smithi103 ceph-mon[39629]: pgmap v2210: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:40.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:40 smithi148 ceph-mon[44520]: pgmap v2210: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:40.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:40 smithi188 ceph-mon[44101]: pgmap v2210: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:42.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:42 smithi148 ceph-mon[44520]: pgmap v2211: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:42.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:42 smithi188 ceph-mon[44101]: pgmap v2211: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:42.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:42 smithi103 ceph-mon[39629]: pgmap v2211: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:43.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:43 smithi148 ceph-mon[44520]: 1.5 scrub starts 2024-04-23T16:56:43.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:43 smithi148 ceph-mon[44520]: 1.5 scrub ok 2024-04-23T16:56:43.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:43 smithi188 ceph-mon[44101]: 1.5 scrub starts 2024-04-23T16:56:43.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:43 smithi188 ceph-mon[44101]: 1.5 scrub ok 2024-04-23T16:56:43.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:43 smithi103 ceph-mon[39629]: 1.5 scrub starts 2024-04-23T16:56:43.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:43 smithi103 ceph-mon[39629]: 1.5 scrub ok 2024-04-23T16:56:44.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:44 smithi148 ceph-mon[44520]: pgmap v2212: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:44 smithi188 ceph-mon[44101]: pgmap v2212: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:44.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:44 smithi103 ceph-mon[39629]: pgmap v2212: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:45.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:45 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:56:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:45 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:56:45.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:45 smithi103 ceph-mon[39629]: pgmap v2213: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:45.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:45 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:56:45.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:45 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:56:45.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:45 smithi148 ceph-mon[44520]: pgmap v2213: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:45.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:45 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:56:45.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:45 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:56:45.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:45 smithi188 ceph-mon[44101]: pgmap v2213: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:46.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:46 smithi103 ceph-mon[39629]: osdmap e799: 12 total, 11 up, 7 in 2024-04-23T16:56:46.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:46 smithi148 ceph-mon[44520]: osdmap e799: 12 total, 11 up, 7 in 2024-04-23T16:56:46.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:46 smithi188 ceph-mon[44101]: osdmap e799: 12 total, 11 up, 7 in 2024-04-23T16:56:47.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:47 smithi103 ceph-mon[39629]: pgmap v2215: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:47.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:47 smithi148 ceph-mon[44520]: pgmap v2215: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:47 smithi188 ceph-mon[44101]: pgmap v2215: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:49.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:49 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:56:49.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:49 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:56:49.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:49 smithi103 ceph-mon[39629]: osdmap e800: 12 total, 11 up, 7 in 2024-04-23T16:56:49.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:49 smithi103 ceph-mon[39629]: pgmap v2217: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:49.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:49 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:56:49.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:49 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:56:49.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:49 smithi148 ceph-mon[44520]: osdmap e800: 12 total, 11 up, 7 in 2024-04-23T16:56:49.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:49 smithi148 ceph-mon[44520]: pgmap v2217: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:49.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:49 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:56:49.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:49 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:56:49.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:49 smithi188 ceph-mon[44101]: osdmap e800: 12 total, 11 up, 7 in 2024-04-23T16:56:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:49 smithi188 ceph-mon[44101]: pgmap v2217: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:52.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:52 smithi103 ceph-mon[39629]: pgmap v2218: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:52.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:52 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T16:56:52.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:52 smithi148 ceph-mon[44520]: pgmap v2218: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:52.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:52 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T16:56:52.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:52 smithi188 ceph-mon[44101]: pgmap v2218: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:52.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:52 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T16:56:53.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:53 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T16:56:53.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:53 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T16:56:53.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:53 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T16:56:54.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:54 smithi103 ceph-mon[39629]: pgmap v2219: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:54.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:54 smithi148 ceph-mon[44520]: pgmap v2219: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:54.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:54 smithi188 ceph-mon[44101]: pgmap v2219: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:56.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:56 smithi103 ceph-mon[39629]: pgmap v2220: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:56.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:56 smithi148 ceph-mon[44520]: pgmap v2220: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:56 smithi188 ceph-mon[44101]: pgmap v2220: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:57.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:57 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:56:57.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:57 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:56:57.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:57 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:56:58.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:58 smithi188 ceph-mon[44101]: pgmap v2221: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:58.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:58 smithi103 ceph-mon[39629]: pgmap v2221: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:58.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:58 smithi148 ceph-mon[44520]: pgmap v2221: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:59.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:59 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:56:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:59 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:56:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:59 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:56:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:59 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:56:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:59 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:56:59.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:56:59 smithi103 ceph-mon[39629]: pgmap v2222: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:59.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:59 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:56:59.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:59 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:56:59.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:59 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:56:59.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:59 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:56:59.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:59 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:56:59.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:56:59 smithi148 ceph-mon[44520]: pgmap v2222: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:56:59.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:59 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:56:59.723 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:59 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:56:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:59 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:56:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:59 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:56:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:59 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:56:59.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:56:59 smithi188 ceph-mon[44101]: pgmap v2222: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:02.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:02 smithi103 ceph-mon[39629]: pgmap v2223: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:02.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:02 smithi148 ceph-mon[44520]: pgmap v2223: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:02.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:02 smithi188 ceph-mon[44101]: pgmap v2223: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:03.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:03 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:57:03.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:03 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:57:03.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:03 smithi103 ceph-mon[39629]: osdmap e801: 12 total, 11 up, 7 in 2024-04-23T16:57:03.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:03 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:57:03.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:03 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:57:03.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:03 smithi148 ceph-mon[44520]: osdmap e801: 12 total, 11 up, 7 in 2024-04-23T16:57:03.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:03 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:57:03.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:03 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:57:03.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:03 smithi188 ceph-mon[44101]: osdmap e801: 12 total, 11 up, 7 in 2024-04-23T16:57:04.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:04 smithi103 ceph-mon[39629]: pgmap v2225: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:04 smithi103 ceph-mon[39629]: 1.a scrub starts 2024-04-23T16:57:04.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:04 smithi148 ceph-mon[44520]: pgmap v2225: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:04.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:04 smithi148 ceph-mon[44520]: 1.a scrub starts 2024-04-23T16:57:04.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:04 smithi188 ceph-mon[44101]: pgmap v2225: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:04.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:04 smithi188 ceph-mon[44101]: 1.a scrub starts 2024-04-23T16:57:05.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:05 smithi103 ceph-mon[39629]: 1.a scrub ok 2024-04-23T16:57:05.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:05 smithi103 ceph-mon[39629]: 1.4 scrub starts 2024-04-23T16:57:05.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:05 smithi148 ceph-mon[44520]: 1.a scrub ok 2024-04-23T16:57:05.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:05 smithi148 ceph-mon[44520]: 1.4 scrub starts 2024-04-23T16:57:05.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:05 smithi188 ceph-mon[44101]: 1.a scrub ok 2024-04-23T16:57:05.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:05 smithi188 ceph-mon[44101]: 1.4 scrub starts 2024-04-23T16:57:06.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:06 smithi148 ceph-mon[44520]: 1.4 scrub ok 2024-04-23T16:57:06.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:06 smithi148 ceph-mon[44520]: pgmap v2226: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:06.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:06 smithi188 ceph-mon[44101]: 1.4 scrub ok 2024-04-23T16:57:06.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:06 smithi188 ceph-mon[44101]: pgmap v2226: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:06.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:06 smithi103 ceph-mon[39629]: 1.4 scrub ok 2024-04-23T16:57:06.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:06 smithi103 ceph-mon[39629]: pgmap v2226: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:07.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:07 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:57:07.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:07 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:57:07.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:07 smithi148 ceph-mon[44520]: osdmap e802: 12 total, 11 up, 7 in 2024-04-23T16:57:07.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:07 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:57:07.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:07 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:57:07.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:07 smithi188 ceph-mon[44101]: osdmap e802: 12 total, 11 up, 7 in 2024-04-23T16:57:07.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:07 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:57:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:07 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:57:07.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:07 smithi103 ceph-mon[39629]: osdmap e802: 12 total, 11 up, 7 in 2024-04-23T16:57:08.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:08 smithi148 ceph-mon[44520]: pgmap v2228: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:08.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:08 smithi188 ceph-mon[44101]: pgmap v2228: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:08.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:08 smithi103 ceph-mon[39629]: pgmap v2228: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:10.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:10 smithi148 ceph-mon[44520]: pgmap v2229: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:10.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:10 smithi188 ceph-mon[44101]: pgmap v2229: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:10.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:10 smithi103 ceph-mon[39629]: pgmap v2229: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:12.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:12 smithi148 ceph-mon[44520]: pgmap v2230: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:12.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:12 smithi188 ceph-mon[44101]: pgmap v2230: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:12.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:12 smithi103 ceph-mon[39629]: pgmap v2230: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:13.710 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-04-23T16:57:13.710 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2024-04-23T16:57:13.711 DEBUG:teuthology.orchestra.run.smithi103:osd.2> sudo journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.2.service 2024-04-23T16:57:13.716 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl start ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.2 2024-04-23T16:57:13.754 INFO:journalctl@ceph.osd.2.smithi103.stdout:-- Logs begin at Tue 2024-04-23 15:57:24 UTC. -- 2024-04-23T16:57:14.099 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:57:13 smithi103 systemd[1]: Starting Ceph osd.2 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T16:57:14.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:14 smithi148 ceph-mon[44520]: pgmap v2231: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:14.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:14 smithi188 ceph-mon[44101]: pgmap v2231: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:14.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:14 smithi103 ceph-mon[39629]: pgmap v2231: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:15.099 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:57:14 smithi103 bash[327857]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T16:57:15.100 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:57:14 smithi103 bash[327857]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-23T16:57:15.100 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:57:14 smithi103 bash[327857]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-23T16:57:15.100 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:57:14 smithi103 bash[327857]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-23T16:57:15.100 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:57:14 smithi103 bash[327857]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-23T16:57:15.100 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:57:14 smithi103 bash[327857]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-23T16:57:15.100 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:57:14 smithi103 bash[327857]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-23T16:57:15.100 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:57:14 smithi103 bash[327857]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-23T16:57:16.080 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-23T16:57:16.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:16 smithi103 ceph-mon[39629]: pgmap v2232: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:16.350 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:57:16 smithi103 bash[328134]: c8edc68e1f37e0dc1c9291f85a3edbc4a8d9cd39148d959d7bdfdac61ab7866a 2024-04-23T16:57:16.350 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:57:16 smithi103 systemd[1]: Started Ceph osd.2 for 8e3594f2-018b-11ef-bc93-c7b262605968. 2024-04-23T16:57:16.350 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:57:16.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:16 smithi148 ceph-mon[44520]: pgmap v2232: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:16.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:16 smithi188 ceph-mon[44101]: pgmap v2232: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:16.820 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:57:16 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[328149]: 2024-04-23T16:57:16.512+0000 7f3475fd67c0 -1 Falling back to public interface 2024-04-23T16:57:18.115 INFO:teuthology.orchestra.run.smithi103.stderr:no valid command found; 10 closest matches: 2024-04-23T16:57:18.115 INFO:teuthology.orchestra.run.smithi103.stderr:0 2024-04-23T16:57:18.115 INFO:teuthology.orchestra.run.smithi103.stderr:1 2024-04-23T16:57:18.115 INFO:teuthology.orchestra.run.smithi103.stderr:2 2024-04-23T16:57:18.115 INFO:teuthology.orchestra.run.smithi103.stderr:abort 2024-04-23T16:57:18.115 INFO:teuthology.orchestra.run.smithi103.stderr:assert 2024-04-23T16:57:18.115 INFO:teuthology.orchestra.run.smithi103.stderr:config diff 2024-04-23T16:57:18.115 INFO:teuthology.orchestra.run.smithi103.stderr:config diff get 2024-04-23T16:57:18.115 INFO:teuthology.orchestra.run.smithi103.stderr:config get 2024-04-23T16:57:18.115 INFO:teuthology.orchestra.run.smithi103.stderr:config help [] 2024-04-23T16:57:18.115 INFO:teuthology.orchestra.run.smithi103.stderr:config set ... 2024-04-23T16:57:18.116 INFO:teuthology.orchestra.run.smithi103.stderr:admin_socket: invalid command 2024-04-23T16:57:18.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:18 smithi148 ceph-mon[44520]: pgmap v2233: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:18.421 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:18 smithi103 ceph-mon[39629]: pgmap v2233: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:18.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:18 smithi188 ceph-mon[44101]: pgmap v2233: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:18.721 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T16:57:18.722 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-23T16:57:19.599 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:57:19 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[328149]: 2024-04-23T16:57:19.296+0000 7f3475fd67c0 -1 osd.2 780 log_to_monitors true 2024-04-23T16:57:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:20 smithi188 ceph-mon[44101]: pgmap v2234: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:20 smithi188 ceph-mon[44101]: from='osd.2 v1:172.21.15.103:6809/1571438738' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T16:57:20.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:20 smithi103 ceph-mon[39629]: pgmap v2234: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:20.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:20 smithi103 ceph-mon[39629]: from='osd.2 v1:172.21.15.103:6809/1571438738' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T16:57:20.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:20 smithi148 ceph-mon[44520]: pgmap v2234: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:20.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:20 smithi148 ceph-mon[44520]: from='osd.2 v1:172.21.15.103:6809/1571438738' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T16:57:20.849 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 16:57:20 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[328149]: 2024-04-23T16:57:20.579+0000 7f3463426700 -1 osd.2 780 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T16:57:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:21 smithi188 ceph-mon[44101]: from='osd.2 v1:172.21.15.103:6809/1571438738' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T16:57:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:21 smithi188 ceph-mon[44101]: osdmap e803: 12 total, 11 up, 7 in 2024-04-23T16:57:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:21 smithi188 ceph-mon[44101]: from='osd.2 v1:172.21.15.103:6809/1571438738' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:57:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:21 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:57:21.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:21 smithi188 ceph-mon[44101]: from='osd.2 v1:172.21.15.103:6809/1571438738' entity='osd.2' 2024-04-23T16:57:21.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:21 smithi103 ceph-mon[39629]: from='osd.2 v1:172.21.15.103:6809/1571438738' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T16:57:21.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:21 smithi103 ceph-mon[39629]: osdmap e803: 12 total, 11 up, 7 in 2024-04-23T16:57:21.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:21 smithi103 ceph-mon[39629]: from='osd.2 v1:172.21.15.103:6809/1571438738' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:57:21.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:21 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:57:21.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:21 smithi103 ceph-mon[39629]: from='osd.2 v1:172.21.15.103:6809/1571438738' entity='osd.2' 2024-04-23T16:57:21.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:21 smithi148 ceph-mon[44520]: from='osd.2 v1:172.21.15.103:6809/1571438738' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T16:57:21.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:21 smithi148 ceph-mon[44520]: osdmap e803: 12 total, 11 up, 7 in 2024-04-23T16:57:21.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:21 smithi148 ceph-mon[44520]: from='osd.2 v1:172.21.15.103:6809/1571438738' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-23T16:57:21.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:21 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:57:21.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:21 smithi148 ceph-mon[44520]: from='osd.2 v1:172.21.15.103:6809/1571438738' entity='osd.2' 2024-04-23T16:57:22.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:22 smithi188 ceph-mon[44101]: pgmap v2236: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:22.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:22 smithi188 ceph-mon[44101]: osd.2 v1:172.21.15.103:6809/1571438738 boot 2024-04-23T16:57:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:22 smithi188 ceph-mon[44101]: osdmap e804: 12 total, 12 up, 7 in 2024-04-23T16:57:22.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:22 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:57:22.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:22 smithi103 ceph-mon[39629]: pgmap v2236: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:22.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:22 smithi103 ceph-mon[39629]: osd.2 v1:172.21.15.103:6809/1571438738 boot 2024-04-23T16:57:22.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:22 smithi103 ceph-mon[39629]: osdmap e804: 12 total, 12 up, 7 in 2024-04-23T16:57:22.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:22 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:57:22.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:22 smithi148 ceph-mon[44520]: pgmap v2236: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:22.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:22 smithi148 ceph-mon[44520]: osd.2 v1:172.21.15.103:6809/1571438738 boot 2024-04-23T16:57:22.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:22 smithi148 ceph-mon[44520]: osdmap e804: 12 total, 12 up, 7 in 2024-04-23T16:57:22.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:22 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T16:57:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:57:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:23 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:57:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:23 smithi188 ceph-mon[44101]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:57:23.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:23 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:57:23.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:57:23.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:23 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:57:23.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:23 smithi103 ceph-mon[39629]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:57:23.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:23 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:57:23.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:57:23.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:23 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:57:23.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:23 smithi148 ceph-mon[44520]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-23T16:57:23.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:23 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:57:23.723 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-23T16:57:24.006 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:57:24.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:24 smithi188 ceph-mon[44101]: pgmap v2238: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:24.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:24 smithi188 ceph-mon[44101]: osdmap e805: 12 total, 12 up, 7 in 2024-04-23T16:57:24.512 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:24 smithi103 ceph-mon[39629]: pgmap v2238: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:24.512 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:24 smithi103 ceph-mon[39629]: osdmap e805: 12 total, 12 up, 7 in 2024-04-23T16:57:24.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:24 smithi148 ceph-mon[44520]: pgmap v2238: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:24.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:24 smithi148 ceph-mon[44520]: osdmap e805: 12 total, 12 up, 7 in 2024-04-23T16:57:25.830 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:57:25.830 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-23T16:57:25.830 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-23T16:57:25.830 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:57:25.848 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:25 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:57:25.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:25 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:57:25.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:25 smithi103 ceph-mon[39629]: pgmap v2240: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:25 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:57:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:25 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:57:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:25 smithi148 ceph-mon[44520]: pgmap v2240: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:25.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:25 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:57:25.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:25 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:57:25.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:25 smithi188 ceph-mon[44101]: pgmap v2240: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:26.612 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T16:57:26.612 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-23T16:57:26.612 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-23T16:57:26.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:26 smithi103 ceph-mon[39629]: osdmap e806: 12 total, 12 up, 7 in 2024-04-23T16:57:26.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:26 smithi148 ceph-mon[44520]: osdmap e806: 12 total, 12 up, 7 in 2024-04-23T16:57:26.886 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:57:26.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:26 smithi188 ceph-mon[44101]: osdmap e806: 12 total, 12 up, 7 in 2024-04-23T16:57:27.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:27 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:57:27.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:27 smithi103 ceph-mon[39629]: pgmap v2242: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:27.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:27 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:57:27.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:27 smithi148 ceph-mon[44520]: pgmap v2242: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:27.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:27 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:57:27.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:27 smithi188 ceph-mon[44101]: pgmap v2242: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:28.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:28 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:57:28.785 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:57:28.785 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-23T16:57:28.785 INFO:teuthology.orchestra.run.smithi103.stdout: "epoch": 806, 2024-04-23T16:57:28.785 INFO:teuthology.orchestra.run.smithi103.stdout: "fsid": "8e3594f2-018b-11ef-bc93-c7b262605968", 2024-04-23T16:57:28.785 INFO:teuthology.orchestra.run.smithi103.stdout: "created": "2024-04-23T16:08:13.477590+0000", 2024-04-23T16:57:28.785 INFO:teuthology.orchestra.run.smithi103.stdout: "modified": "2024-04-23T16:57:25.566381+0000", 2024-04-23T16:57:28.785 INFO:teuthology.orchestra.run.smithi103.stdout: "last_up_change": "2024-04-23T16:57:21.581583+0000", 2024-04-23T16:57:28.785 INFO:teuthology.orchestra.run.smithi103.stdout: "last_in_change": "2024-04-23T16:56:11.072001+0000", 2024-04-23T16:57:28.785 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-23T16:57:28.786 INFO:teuthology.orchestra.run.smithi103.stdout: "flags_num": 5799936, 2024-04-23T16:57:28.786 INFO:teuthology.orchestra.run.smithi103.stdout: "flags_set": [ 2024-04-23T16:57:28.786 INFO:teuthology.orchestra.run.smithi103.stdout: "pglog_hardlimit", 2024-04-23T16:57:28.786 INFO:teuthology.orchestra.run.smithi103.stdout: "purged_snapdirs", 2024-04-23T16:57:28.786 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_deletes", 2024-04-23T16:57:28.786 INFO:teuthology.orchestra.run.smithi103.stdout: "sortbitwise" 2024-04-23T16:57:28.786 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:57:28.786 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_version": 34, 2024-04-23T16:57:28.786 INFO:teuthology.orchestra.run.smithi103.stdout: "full_ratio": 0.94999998807907104, 2024-04-23T16:57:28.786 INFO:teuthology.orchestra.run.smithi103.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-23T16:57:28.786 INFO:teuthology.orchestra.run.smithi103.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-23T16:57:28.786 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_snapshot": "", 2024-04-23T16:57:28.787 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_max": 9, 2024-04-23T16:57:28.787 INFO:teuthology.orchestra.run.smithi103.stdout: "max_osd": 12, 2024-04-23T16:57:28.787 INFO:teuthology.orchestra.run.smithi103.stdout: "require_min_compat_client": "luminous", 2024-04-23T16:57:28.787 INFO:teuthology.orchestra.run.smithi103.stdout: "min_compat_client": "jewel", 2024-04-23T16:57:28.787 INFO:teuthology.orchestra.run.smithi103.stdout: "require_osd_release": "squid", 2024-04-23T16:57:28.787 INFO:teuthology.orchestra.run.smithi103.stdout: "allow_crimson": false, 2024-04-23T16:57:28.787 INFO:teuthology.orchestra.run.smithi103.stdout: "pools": [ 2024-04-23T16:57:28.787 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.787 INFO:teuthology.orchestra.run.smithi103.stdout: "pool": 1, 2024-04-23T16:57:28.787 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_name": ".mgr", 2024-04-23T16:57:28.787 INFO:teuthology.orchestra.run.smithi103.stdout: "create_time": "2024-04-23T16:11:22.416777+0000", 2024-04-23T16:57:28.787 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-04-23T16:57:28.788 INFO:teuthology.orchestra.run.smithi103.stdout: "flags_names": "hashpspool", 2024-04-23T16:57:28.788 INFO:teuthology.orchestra.run.smithi103.stdout: "type": 1, 2024-04-23T16:57:28.788 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 2, 2024-04-23T16:57:28.788 INFO:teuthology.orchestra.run.smithi103.stdout: "min_size": 2, 2024-04-23T16:57:28.788 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_rule": 0, 2024-04-23T16:57:28.788 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_count": 0, 2024-04-23T16:57:28.788 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_target": 0, 2024-04-23T16:57:28.788 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T16:57:28.788 INFO:teuthology.orchestra.run.smithi103.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T16:57:28.788 INFO:teuthology.orchestra.run.smithi103.stdout: "object_hash": 2, 2024-04-23T16:57:28.788 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_autoscale_mode": "off", 2024-04-23T16:57:28.788 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num": 11, 2024-04-23T16:57:28.788 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_placement_num": 11, 2024-04-23T16:57:28.788 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_placement_num_target": 11, 2024-04-23T16:57:28.789 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_target": 11, 2024-04-23T16:57:28.789 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_pending": 11, 2024-04-23T16:57:28.789 INFO:teuthology.orchestra.run.smithi103.stdout: "last_pg_merge_meta": { 2024-04-23T16:57:28.789 INFO:teuthology.orchestra.run.smithi103.stdout: "source_pgid": "0.0", 2024-04-23T16:57:28.789 INFO:teuthology.orchestra.run.smithi103.stdout: "ready_epoch": 0, 2024-04-23T16:57:28.789 INFO:teuthology.orchestra.run.smithi103.stdout: "last_epoch_started": 0, 2024-04-23T16:57:28.789 INFO:teuthology.orchestra.run.smithi103.stdout: "last_epoch_clean": 0, 2024-04-23T16:57:28.789 INFO:teuthology.orchestra.run.smithi103.stdout: "source_version": "0'0", 2024-04-23T16:57:28.789 INFO:teuthology.orchestra.run.smithi103.stdout: "target_version": "0'0" 2024-04-23T16:57:28.789 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.789 INFO:teuthology.orchestra.run.smithi103.stdout: "last_change": "303", 2024-04-23T16:57:28.789 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend": "0", 2024-04-23T16:57:28.789 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T16:57:28.790 INFO:teuthology.orchestra.run.smithi103.stdout: "last_force_op_resend_preluminous": "301", 2024-04-23T16:57:28.790 INFO:teuthology.orchestra.run.smithi103.stdout: "auid": 0, 2024-04-23T16:57:28.790 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_mode": "selfmanaged", 2024-04-23T16:57:28.790 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_seq": 0, 2024-04-23T16:57:28.790 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_epoch": 0, 2024-04-23T16:57:28.790 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_snaps": [], 2024-04-23T16:57:28.790 INFO:teuthology.orchestra.run.smithi103.stdout: "removed_snaps": "[]", 2024-04-23T16:57:28.790 INFO:teuthology.orchestra.run.smithi103.stdout: "quota_max_bytes": 0, 2024-04-23T16:57:28.790 INFO:teuthology.orchestra.run.smithi103.stdout: "quota_max_objects": 0, 2024-04-23T16:57:28.790 INFO:teuthology.orchestra.run.smithi103.stdout: "tiers": [], 2024-04-23T16:57:28.790 INFO:teuthology.orchestra.run.smithi103.stdout: "tier_of": -1, 2024-04-23T16:57:28.790 INFO:teuthology.orchestra.run.smithi103.stdout: "read_tier": -1, 2024-04-23T16:57:28.790 INFO:teuthology.orchestra.run.smithi103.stdout: "write_tier": -1, 2024-04-23T16:57:28.790 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_mode": "none", 2024-04-23T16:57:28.791 INFO:teuthology.orchestra.run.smithi103.stdout: "target_max_bytes": 0, 2024-04-23T16:57:28.791 INFO:teuthology.orchestra.run.smithi103.stdout: "target_max_objects": 0, 2024-04-23T16:57:28.791 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T16:57:28.791 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T16:57:28.791 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T16:57:28.791 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_min_flush_age": 0, 2024-04-23T16:57:28.791 INFO:teuthology.orchestra.run.smithi103.stdout: "cache_min_evict_age": 0, 2024-04-23T16:57:28.791 INFO:teuthology.orchestra.run.smithi103.stdout: "erasure_code_profile": "", 2024-04-23T16:57:28.791 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_params": { 2024-04-23T16:57:28.791 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "none" 2024-04-23T16:57:28.791 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.791 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_period": 0, 2024-04-23T16:57:28.791 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_count": 0, 2024-04-23T16:57:28.791 INFO:teuthology.orchestra.run.smithi103.stdout: "use_gmt_hitset": true, 2024-04-23T16:57:28.792 INFO:teuthology.orchestra.run.smithi103.stdout: "min_read_recency_for_promote": 0, 2024-04-23T16:57:28.792 INFO:teuthology.orchestra.run.smithi103.stdout: "min_write_recency_for_promote": 0, 2024-04-23T16:57:28.792 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T16:57:28.792 INFO:teuthology.orchestra.run.smithi103.stdout: "hit_set_search_last_n": 0, 2024-04-23T16:57:28.792 INFO:teuthology.orchestra.run.smithi103.stdout: "grade_table": [], 2024-04-23T16:57:28.792 INFO:teuthology.orchestra.run.smithi103.stdout: "stripe_width": 0, 2024-04-23T16:57:28.792 INFO:teuthology.orchestra.run.smithi103.stdout: "expected_num_objects": 0, 2024-04-23T16:57:28.792 INFO:teuthology.orchestra.run.smithi103.stdout: "fast_read": false, 2024-04-23T16:57:28.792 INFO:teuthology.orchestra.run.smithi103.stdout: "options": { 2024-04-23T16:57:28.792 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_max": 32, 2024-04-23T16:57:28.792 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num_min": 1 2024-04-23T16:57:28.792 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.792 INFO:teuthology.orchestra.run.smithi103.stdout: "application_metadata": { 2024-04-23T16:57:28.793 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": {} 2024-04-23T16:57:28.793 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.793 INFO:teuthology.orchestra.run.smithi103.stdout: "read_balance": { 2024-04-23T16:57:28.793 INFO:teuthology.orchestra.run.smithi103.stdout: "score_type": "Fair distribution", 2024-04-23T16:57:28.793 INFO:teuthology.orchestra.run.smithi103.stdout: "score_acting": 2.190000057220459, 2024-04-23T16:57:28.793 INFO:teuthology.orchestra.run.smithi103.stdout: "score_stable": 2.190000057220459, 2024-04-23T16:57:28.793 INFO:teuthology.orchestra.run.smithi103.stdout: "optimal_score": 0.57999998331069946, 2024-04-23T16:57:28.793 INFO:teuthology.orchestra.run.smithi103.stdout: "raw_score_acting": 1.2699999809265137, 2024-04-23T16:57:28.793 INFO:teuthology.orchestra.run.smithi103.stdout: "raw_score_stable": 1.2699999809265137, 2024-04-23T16:57:28.793 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity_weighted": 0.98181778192520142, 2024-04-23T16:57:28.793 INFO:teuthology.orchestra.run.smithi103.stdout: "average_primary_affinity": 0.98000001907348633, 2024-04-23T16:57:28.793 INFO:teuthology.orchestra.run.smithi103.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-04-23T16:57:28.793 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.793 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.794 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:57:28.794 INFO:teuthology.orchestra.run.smithi103.stdout: "osds": [ 2024-04-23T16:57:28.794 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.794 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 0, 2024-04-23T16:57:28.794 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "4c084047-8f01-4b86-a60c-5592735b5862", 2024-04-23T16:57:28.794 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:57:28.794 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:57:28.794 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:57:28.794 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:57:28.794 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:57:28.794 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:57:28.794 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 8, 2024-04-23T16:57:28.794 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 734, 2024-04-23T16:57:28.794 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:57:28.795 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:57:28.795 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:57:28.795 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.795 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.795 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.795 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6801", 2024-04-23T16:57:28.795 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:57:28.795 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.795 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.795 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.795 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:57:28.795 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.795 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.795 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.795 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6802", 2024-04-23T16:57:28.796 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:57:28.796 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.796 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.796 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.796 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:57:28.796 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.796 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.796 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.796 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6804", 2024-04-23T16:57:28.796 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:57:28.796 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.796 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.796 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.796 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:57:28.796 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.797 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.797 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.797 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6803", 2024-04-23T16:57:28.797 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4274040097 2024-04-23T16:57:28.797 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.797 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.797 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.797 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6801/4274040097", 2024-04-23T16:57:28.797 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6802/4274040097", 2024-04-23T16:57:28.798 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6804/4274040097", 2024-04-23T16:57:28.798 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6803/4274040097", 2024-04-23T16:57:28.798 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:57:28.798 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:57:28.798 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:57:28.798 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.798 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.798 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.798 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 1, 2024-04-23T16:57:28.798 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "3438f2bd-5fb7-432c-b383-29e18c442f20", 2024-04-23T16:57:28.798 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:57:28.798 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:57:28.799 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:57:28.799 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:57:28.799 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 13, 2024-04-23T16:57:28.799 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 232, 2024-04-23T16:57:28.799 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 271, 2024-04-23T16:57:28.799 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 703, 2024-04-23T16:57:28.799 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 233, 2024-04-23T16:57:28.799 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:57:28.799 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:57:28.799 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.799 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.799 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.800 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6805", 2024-04-23T16:57:28.800 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 859282187 2024-04-23T16:57:28.800 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.800 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.800 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.800 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:57:28.800 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.800 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.800 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.800 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6806", 2024-04-23T16:57:28.800 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 859282187 2024-04-23T16:57:28.800 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.801 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.801 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.801 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:57:28.801 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.801 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.801 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.801 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6808", 2024-04-23T16:57:28.801 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 859282187 2024-04-23T16:57:28.801 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.801 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.801 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.801 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:57:28.801 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.802 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.802 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.802 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6807", 2024-04-23T16:57:28.802 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 859282187 2024-04-23T16:57:28.802 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.802 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.802 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.802 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6805/859282187", 2024-04-23T16:57:28.802 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6806/859282187", 2024-04-23T16:57:28.802 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6808/859282187", 2024-04-23T16:57:28.802 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6807/859282187", 2024-04-23T16:57:28.802 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:57:28.803 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:57:28.803 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:57:28.803 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.803 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.803 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.803 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 2, 2024-04-23T16:57:28.803 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "c17fffb4-8d17-4546-ba24-e5951a2a37fe", 2024-04-23T16:57:28.803 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:57:28.803 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 0, 2024-04-23T16:57:28.803 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 0, 2024-04-23T16:57:28.803 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:57:28.803 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 481, 2024-04-23T16:57:28.803 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 780, 2024-04-23T16:57:28.804 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 804, 2024-04-23T16:57:28.804 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 735, 2024-04-23T16:57:28.804 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 782, 2024-04-23T16:57:28.804 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:57:28.804 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:57:28.804 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.804 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.804 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.804 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6809", 2024-04-23T16:57:28.804 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1571438738 2024-04-23T16:57:28.804 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.805 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.805 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.805 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:57:28.805 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.805 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.805 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.805 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6810", 2024-04-23T16:57:28.805 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1571438738 2024-04-23T16:57:28.805 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.805 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.806 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.806 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:57:28.806 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.806 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.806 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.806 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6812", 2024-04-23T16:57:28.806 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1571438738 2024-04-23T16:57:28.806 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.806 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.806 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.806 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:57:28.807 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.807 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.807 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.807 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6811", 2024-04-23T16:57:28.807 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1571438738 2024-04-23T16:57:28.807 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.807 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.807 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.807 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6809/1571438738", 2024-04-23T16:57:28.807 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6810/1571438738", 2024-04-23T16:57:28.808 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6812/1571438738", 2024-04-23T16:57:28.808 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6811/1571438738", 2024-04-23T16:57:28.808 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:57:28.808 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:57:28.808 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:57:28.808 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.808 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.808 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.808 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 3, 2024-04-23T16:57:28.808 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "cee91a3d-55dc-4e89-b568-86821632bb1c", 2024-04-23T16:57:28.808 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:57:28.808 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 0, 2024-04-23T16:57:28.809 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 0, 2024-04-23T16:57:28.809 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:57:28.809 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 27, 2024-04-23T16:57:28.809 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 559, 2024-04-23T16:57:28.809 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 619, 2024-04-23T16:57:28.809 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 540, 2024-04-23T16:57:28.809 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 560, 2024-04-23T16:57:28.809 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:57:28.810 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:57:28.810 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.810 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.810 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.810 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6813", 2024-04-23T16:57:28.810 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 995549078 2024-04-23T16:57:28.810 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.810 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.810 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.810 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:57:28.810 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.810 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.811 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.811 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6814", 2024-04-23T16:57:28.811 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 995549078 2024-04-23T16:57:28.811 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.811 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.811 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.811 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:57:28.811 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.811 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.811 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.811 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6816", 2024-04-23T16:57:28.811 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 995549078 2024-04-23T16:57:28.812 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.812 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.812 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:57:28.812 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.812 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.812 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.812 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6815", 2024-04-23T16:57:28.812 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 995549078 2024-04-23T16:57:28.812 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.812 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.813 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.103:6813/995549078", 2024-04-23T16:57:28.813 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.103:6814/995549078", 2024-04-23T16:57:28.813 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.103:6816/995549078", 2024-04-23T16:57:28.813 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.103:6815/995549078", 2024-04-23T16:57:28.813 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:57:28.813 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:57:28.813 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:57:28.813 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.813 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.813 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 4, 2024-04-23T16:57:28.814 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "61d5385b-2c82-42f8-93c3-8f1baf71c65e", 2024-04-23T16:57:28.814 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:57:28.814 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 0, 2024-04-23T16:57:28.814 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 0, 2024-04-23T16:57:28.814 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:57:28.814 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 33, 2024-04-23T16:57:28.814 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 686, 2024-04-23T16:57:28.814 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 728, 2024-04-23T16:57:28.814 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 609, 2024-04-23T16:57:28.814 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 687, 2024-04-23T16:57:28.814 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:57:28.815 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:57:28.815 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.815 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.815 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.815 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6800", 2024-04-23T16:57:28.815 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4230909898 2024-04-23T16:57:28.815 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.815 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.815 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.815 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:57:28.815 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.815 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.816 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.816 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6801", 2024-04-23T16:57:28.816 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4230909898 2024-04-23T16:57:28.816 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.816 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.816 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.816 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:57:28.816 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.816 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.816 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.816 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6803", 2024-04-23T16:57:28.816 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4230909898 2024-04-23T16:57:28.817 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.817 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.817 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:57:28.817 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.817 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.817 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.817 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6802", 2024-04-23T16:57:28.817 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4230909898 2024-04-23T16:57:28.817 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.818 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.818 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6800/4230909898", 2024-04-23T16:57:28.818 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6801/4230909898", 2024-04-23T16:57:28.818 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6803/4230909898", 2024-04-23T16:57:28.818 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6802/4230909898", 2024-04-23T16:57:28.818 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:57:28.818 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:57:28.818 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:57:28.818 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.818 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.819 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 5, 2024-04-23T16:57:28.819 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "5d492189-c3cd-47ad-a12a-ed4d275bcfe3", 2024-04-23T16:57:28.819 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:57:28.819 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 0, 2024-04-23T16:57:28.819 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 0, 2024-04-23T16:57:28.819 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:57:28.819 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:57:28.819 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:57:28.819 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 38, 2024-04-23T16:57:28.819 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 609, 2024-04-23T16:57:28.819 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:57:28.819 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:57:28.820 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:57:28.820 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.820 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.820 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.820 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6804", 2024-04-23T16:57:28.820 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:57:28.820 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.820 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.820 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.820 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:57:28.820 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.821 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.821 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.821 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6805", 2024-04-23T16:57:28.821 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:57:28.821 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.821 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.821 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.821 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:57:28.821 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.821 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.821 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.821 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6807", 2024-04-23T16:57:28.822 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:57:28.822 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.822 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.822 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.822 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:57:28.822 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.822 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.822 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.822 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6806", 2024-04-23T16:57:28.822 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3547200711 2024-04-23T16:57:28.822 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.822 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.823 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.823 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6804/3547200711", 2024-04-23T16:57:28.823 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6805/3547200711", 2024-04-23T16:57:28.823 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6807/3547200711", 2024-04-23T16:57:28.823 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6806/3547200711", 2024-04-23T16:57:28.823 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:57:28.823 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:57:28.823 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:57:28.823 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.823 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.823 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.823 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 6, 2024-04-23T16:57:28.823 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "89f2379e-cf14-4e69-8747-7535db81b70b", 2024-04-23T16:57:28.824 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:57:28.824 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:57:28.824 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:57:28.824 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:57:28.824 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:57:28.824 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:57:28.824 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 47, 2024-04-23T16:57:28.824 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 789, 2024-04-23T16:57:28.824 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:57:28.824 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:57:28.824 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:57:28.825 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.825 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.825 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.825 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6808", 2024-04-23T16:57:28.825 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:57:28.825 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.825 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.825 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.825 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:57:28.825 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.825 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.825 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.826 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6809", 2024-04-23T16:57:28.826 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:57:28.826 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.826 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.826 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.826 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:57:28.826 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.826 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.826 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.826 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6811", 2024-04-23T16:57:28.826 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:57:28.826 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.827 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.827 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.827 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:57:28.827 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.827 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.827 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.827 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6810", 2024-04-23T16:57:28.827 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 366919758 2024-04-23T16:57:28.827 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.827 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.827 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.827 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6808/366919758", 2024-04-23T16:57:28.828 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6809/366919758", 2024-04-23T16:57:28.828 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6811/366919758", 2024-04-23T16:57:28.828 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6810/366919758", 2024-04-23T16:57:28.828 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:57:28.828 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:57:28.828 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:57:28.828 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.828 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.828 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.828 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 7, 2024-04-23T16:57:28.828 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "f27b035a-558f-4528-80ea-2a00d03f6539", 2024-04-23T16:57:28.828 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:57:28.829 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:57:28.829 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:57:28.829 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 0.7818145751953125, 2024-04-23T16:57:28.829 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:57:28.829 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:57:28.829 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 56, 2024-04-23T16:57:28.829 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 732, 2024-04-23T16:57:28.829 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:57:28.829 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:57:28.829 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:57:28.829 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.829 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.830 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.830 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6812", 2024-04-23T16:57:28.830 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:57:28.830 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.830 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.830 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.830 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:57:28.830 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.830 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.830 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.830 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6813", 2024-04-23T16:57:28.831 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:57:28.831 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.831 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.831 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.831 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:57:28.831 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.831 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.831 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.831 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6815", 2024-04-23T16:57:28.831 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:57:28.831 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.831 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.832 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.832 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:57:28.832 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.832 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.832 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.148:6814", 2024-04-23T16:57:28.832 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3719550888 2024-04-23T16:57:28.832 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.832 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.832 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.832 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.148:6812/3719550888", 2024-04-23T16:57:28.833 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.148:6813/3719550888", 2024-04-23T16:57:28.833 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.148:6815/3719550888", 2024-04-23T16:57:28.833 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.148:6814/3719550888", 2024-04-23T16:57:28.833 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:57:28.833 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:57:28.833 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:57:28.833 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.833 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.833 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 8, 2024-04-23T16:57:28.833 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed", 2024-04-23T16:57:28.833 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:57:28.834 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:57:28.834 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:57:28.834 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:57:28.834 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:57:28.834 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:57:28.834 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 66, 2024-04-23T16:57:28.834 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 789, 2024-04-23T16:57:28.834 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:57:28.834 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:57:28.834 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:57:28.834 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.835 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.835 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6800", 2024-04-23T16:57:28.835 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:57:28.835 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.835 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.835 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:57:28.835 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.835 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.836 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6801", 2024-04-23T16:57:28.836 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:57:28.836 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.836 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.836 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:57:28.836 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.836 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.836 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6803", 2024-04-23T16:57:28.836 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:57:28.836 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.837 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.837 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:57:28.837 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.837 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.837 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6802", 2024-04-23T16:57:28.837 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1151408595 2024-04-23T16:57:28.837 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.837 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.837 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6800/1151408595", 2024-04-23T16:57:28.838 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6801/1151408595", 2024-04-23T16:57:28.838 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6803/1151408595", 2024-04-23T16:57:28.838 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6802/1151408595", 2024-04-23T16:57:28.838 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:57:28.838 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:57:28.838 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:57:28.838 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.838 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 9, 2024-04-23T16:57:28.838 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "124d9b05-11eb-4622-b6a2-c4d5bd60b7e6", 2024-04-23T16:57:28.838 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:57:28.839 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:57:28.839 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:57:28.839 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:57:28.839 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:57:28.839 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:57:28.839 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 73, 2024-04-23T16:57:28.839 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 703, 2024-04-23T16:57:28.839 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:57:28.839 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:57:28.839 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:57:28.839 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.840 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.840 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6804", 2024-04-23T16:57:28.840 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:57:28.840 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.840 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.840 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:57:28.840 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.840 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.841 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6805", 2024-04-23T16:57:28.841 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:57:28.841 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.841 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.841 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:57:28.841 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.841 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.841 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6807", 2024-04-23T16:57:28.841 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:57:28.841 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.842 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.842 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:57:28.842 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.842 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.842 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6806", 2024-04-23T16:57:28.842 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 306501862 2024-04-23T16:57:28.842 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.842 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.842 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6804/306501862", 2024-04-23T16:57:28.843 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6805/306501862", 2024-04-23T16:57:28.843 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6807/306501862", 2024-04-23T16:57:28.843 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6806/306501862", 2024-04-23T16:57:28.843 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:57:28.843 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:57:28.843 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:57:28.843 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.843 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 10, 2024-04-23T16:57:28.843 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "3dfb9070-e534-4ea4-a25e-ee1c51297447", 2024-04-23T16:57:28.844 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:57:28.844 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 0, 2024-04-23T16:57:28.844 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 0, 2024-04-23T16:57:28.844 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:57:28.844 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:57:28.844 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:57:28.844 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 81, 2024-04-23T16:57:28.844 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 0, 2024-04-23T16:57:28.844 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:57:28.844 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:57:28.844 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:57:28.845 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.845 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.845 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6808", 2024-04-23T16:57:28.845 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:57:28.845 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.845 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.845 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:57:28.845 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.845 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.846 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6809", 2024-04-23T16:57:28.846 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:57:28.846 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.846 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.846 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.846 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:57:28.846 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.846 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.846 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.846 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6811", 2024-04-23T16:57:28.846 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:57:28.846 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.847 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.847 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:57:28.847 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.847 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.847 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6810", 2024-04-23T16:57:28.847 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1231263138 2024-04-23T16:57:28.847 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.847 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.847 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6808/1231263138", 2024-04-23T16:57:28.848 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6809/1231263138", 2024-04-23T16:57:28.848 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6811/1231263138", 2024-04-23T16:57:28.848 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6810/1231263138", 2024-04-23T16:57:28.848 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:57:28.848 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:57:28.848 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:57:28.848 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.848 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 11, 2024-04-23T16:57:28.848 INFO:teuthology.orchestra.run.smithi103.stdout: "uuid": "bc163925-f623-4f63-ab89-35e2d511a730", 2024-04-23T16:57:28.849 INFO:teuthology.orchestra.run.smithi103.stdout: "up": 1, 2024-04-23T16:57:28.849 INFO:teuthology.orchestra.run.smithi103.stdout: "in": 1, 2024-04-23T16:57:28.849 INFO:teuthology.orchestra.run.smithi103.stdout: "weight": 1, 2024-04-23T16:57:28.849 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_affinity": 1, 2024-04-23T16:57:28.849 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_begin": 0, 2024-04-23T16:57:28.849 INFO:teuthology.orchestra.run.smithi103.stdout: "last_clean_end": 0, 2024-04-23T16:57:28.849 INFO:teuthology.orchestra.run.smithi103.stdout: "up_from": 90, 2024-04-23T16:57:28.849 INFO:teuthology.orchestra.run.smithi103.stdout: "up_thru": 703, 2024-04-23T16:57:28.849 INFO:teuthology.orchestra.run.smithi103.stdout: "down_at": 0, 2024-04-23T16:57:28.849 INFO:teuthology.orchestra.run.smithi103.stdout: "lost_at": 0, 2024-04-23T16:57:28.849 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addrs": { 2024-04-23T16:57:28.850 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.850 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.850 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6812", 2024-04-23T16:57:28.850 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:57:28.850 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.850 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.850 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addrs": { 2024-04-23T16:57:28.850 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.851 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.851 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6813", 2024-04-23T16:57:28.851 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:57:28.851 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.851 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.851 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addrs": { 2024-04-23T16:57:28.851 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.851 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.851 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6815", 2024-04-23T16:57:28.851 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:57:28.851 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.852 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.852 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addrs": { 2024-04-23T16:57:28.852 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-04-23T16:57:28.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.852 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-04-23T16:57:28.852 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.188:6814", 2024-04-23T16:57:28.852 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 457992661 2024-04-23T16:57:28.852 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.852 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.852 INFO:teuthology.orchestra.run.smithi103.stdout: "public_addr": "172.21.15.188:6812/457992661", 2024-04-23T16:57:28.853 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_addr": "172.21.15.188:6813/457992661", 2024-04-23T16:57:28.853 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_back_addr": "172.21.15.188:6815/457992661", 2024-04-23T16:57:28.853 INFO:teuthology.orchestra.run.smithi103.stdout: "heartbeat_front_addr": "172.21.15.188:6814/457992661", 2024-04-23T16:57:28.853 INFO:teuthology.orchestra.run.smithi103.stdout: "state": [ 2024-04-23T16:57:28.853 INFO:teuthology.orchestra.run.smithi103.stdout: "exists", 2024-04-23T16:57:28.853 INFO:teuthology.orchestra.run.smithi103.stdout: "up" 2024-04-23T16:57:28.853 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-23T16:57:28.853 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.853 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:57:28.853 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_xinfo": [ 2024-04-23T16:57:28.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.853 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 0, 2024-04-23T16:57:28.854 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:57:28.854 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:57:28.854 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:57:28.854 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:57:28.854 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:57:28.854 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:56:44.847759+0000", 2024-04-23T16:57:28.854 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:57:28.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.854 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 1, 2024-04-23T16:57:28.854 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:57:28.855 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:57:28.855 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:57:28.855 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:57:28.855 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:57:28.855 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:56:35.075680+0000", 2024-04-23T16:57:28.855 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 232 2024-04-23T16:57:28.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.855 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 2, 2024-04-23T16:57:28.855 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:57:28.855 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:57:28.855 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:57:28.856 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:57:28.856 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 65536, 2024-04-23T16:57:28.856 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:57:20.594617+0000", 2024-04-23T16:57:28.856 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 780 2024-04-23T16:57:28.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.856 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.856 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 3, 2024-04-23T16:57:28.856 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:57:28.856 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:57:28.856 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:57:28.856 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:57:28.857 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 65536, 2024-04-23T16:57:28.857 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:56:25.474649+0000", 2024-04-23T16:57:28.857 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 559 2024-04-23T16:57:28.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.857 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 4, 2024-04-23T16:57:28.857 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:57:28.857 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:57:28.857 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:57:28.857 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:57:28.857 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 65536, 2024-04-23T16:57:28.857 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:57:05.867428+0000", 2024-04-23T16:57:28.858 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 686 2024-04-23T16:57:28.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.858 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 5, 2024-04-23T16:57:28.858 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:57:28.858 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:57:28.858 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:57:28.858 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:57:28.858 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 65536, 2024-04-23T16:57:28.858 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:56:48.301970+0000", 2024-04-23T16:57:28.858 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:57:28.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.859 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.859 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 6, 2024-04-23T16:57:28.859 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:57:28.859 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:57:28.859 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:57:28.859 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:57:28.859 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:57:28.859 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:57:24.746568+0000", 2024-04-23T16:57:28.859 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:57:28.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.859 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.860 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 7, 2024-04-23T16:57:28.860 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:57:28.860 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:57:28.860 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:57:28.860 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:57:28.860 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:57:28.860 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:56:22.011987+0000", 2024-04-23T16:57:28.860 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:57:28.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.860 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 8, 2024-04-23T16:57:28.860 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:57:28.861 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:57:28.861 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:57:28.861 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:57:28.861 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:57:28.861 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:57:02.045084+0000", 2024-04-23T16:57:28.861 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:57:28.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.861 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 9, 2024-04-23T16:57:28.861 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:57:28.861 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:57:28.861 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:57:28.862 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:57:28.862 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:57:28.862 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:56:35.674843+0000", 2024-04-23T16:57:28.862 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:57:28.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.862 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 10, 2024-04-23T16:57:28.862 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:57:28.862 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:57:28.862 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:57:28.862 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:57:28.862 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 65536, 2024-04-23T16:57:28.863 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:56:35.374566+0000", 2024-04-23T16:57:28.863 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:57:28.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-23T16:57:28.863 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": 11, 2024-04-23T16:57:28.863 INFO:teuthology.orchestra.run.smithi103.stdout: "down_stamp": "0.000000", 2024-04-23T16:57:28.863 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_probability": 0, 2024-04-23T16:57:28.863 INFO:teuthology.orchestra.run.smithi103.stdout: "laggy_interval": 0, 2024-04-23T16:57:28.863 INFO:teuthology.orchestra.run.smithi103.stdout: "features": 4540701547738038271, 2024-04-23T16:57:28.863 INFO:teuthology.orchestra.run.smithi103.stdout: "old_weight": 0, 2024-04-23T16:57:28.863 INFO:teuthology.orchestra.run.smithi103.stdout: "last_purged_snaps_scrub": "2024-04-23T16:56:33.000718+0000", 2024-04-23T16:57:28.863 INFO:teuthology.orchestra.run.smithi103.stdout: "dead_epoch": 0 2024-04-23T16:57:28.864 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.864 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-04-23T16:57:28.864 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_upmap": [], 2024-04-23T16:57:28.864 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_upmap_items": [], 2024-04-23T16:57:28.864 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_upmap_primaries": [], 2024-04-23T16:57:28.864 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_temp": [], 2024-04-23T16:57:28.864 INFO:teuthology.orchestra.run.smithi103.stdout: "primary_temp": [], 2024-04-23T16:57:28.864 INFO:teuthology.orchestra.run.smithi103.stdout: "blocklist": { 2024-04-23T16:57:28.864 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:6800/3535470771": "2024-04-24T16:09:14.324448+0000", 2024-04-23T16:57:28.864 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/1188058303": "2024-04-24T16:09:14.324448+0000", 2024-04-23T16:57:28.864 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/1510553721": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:57:28.864 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/281182801": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:57:28.864 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:6800/2049124518": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:57:28.865 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/466241963": "2024-04-24T16:08:44.450632+0000", 2024-04-23T16:57:28.865 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/2726838698": "2024-04-24T16:09:14.324448+0000", 2024-04-23T16:57:28.865 INFO:teuthology.orchestra.run.smithi103.stdout: "172.21.15.103:0/596230573": "2024-04-24T16:09:14.324448+0000" 2024-04-23T16:57:28.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.865 INFO:teuthology.orchestra.run.smithi103.stdout: "range_blocklist": {}, 2024-04-23T16:57:28.865 INFO:teuthology.orchestra.run.smithi103.stdout: "erasure_code_profiles": { 2024-04-23T16:57:28.865 INFO:teuthology.orchestra.run.smithi103.stdout: "default": { 2024-04-23T16:57:28.865 INFO:teuthology.orchestra.run.smithi103.stdout: "crush-failure-domain": "osd", 2024-04-23T16:57:28.865 INFO:teuthology.orchestra.run.smithi103.stdout: "k": "2", 2024-04-23T16:57:28.865 INFO:teuthology.orchestra.run.smithi103.stdout: "m": "1", 2024-04-23T16:57:28.865 INFO:teuthology.orchestra.run.smithi103.stdout: "plugin": "jerasure", 2024-04-23T16:57:28.865 INFO:teuthology.orchestra.run.smithi103.stdout: "technique": "reed_sol_van" 2024-04-23T16:57:28.866 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-23T16:57:28.866 INFO:teuthology.orchestra.run.smithi103.stdout: "removed_snaps_queue": [], 2024-04-23T16:57:28.866 INFO:teuthology.orchestra.run.smithi103.stdout: "new_removed_snaps": [], 2024-04-23T16:57:28.866 INFO:teuthology.orchestra.run.smithi103.stdout: "new_purged_snaps": [], 2024-04-23T16:57:28.866 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_node_flags": {}, 2024-04-23T16:57:28.866 INFO:teuthology.orchestra.run.smithi103.stdout: "device_class_flags": {}, 2024-04-23T16:57:28.866 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_mode": { 2024-04-23T16:57:28.866 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_mode_enabled": false, 2024-04-23T16:57:28.866 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_bucket_count": 0, 2024-04-23T16:57:28.866 INFO:teuthology.orchestra.run.smithi103.stdout: "degraded_stretch_mode": 0, 2024-04-23T16:57:28.866 INFO:teuthology.orchestra.run.smithi103.stdout: "recovering_stretch_mode": 0, 2024-04-23T16:57:28.867 INFO:teuthology.orchestra.run.smithi103.stdout: "stretch_mode_bucket": 0 2024-04-23T16:57:28.867 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-23T16:57:28.867 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-23T16:57:28.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:28 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:57:28.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:28 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:57:29.434 INFO:tasks.thrashosds.thrasher:j is {'epoch': 806, 'fsid': '8e3594f2-018b-11ef-bc93-c7b262605968', 'created': '2024-04-23T16:08:13.477590+0000', 'modified': '2024-04-23T16:57:25.566381+0000', 'last_up_change': '2024-04-23T16:57:21.581583+0000', 'last_in_change': '2024-04-23T16:56:11.072001+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 34, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T16:11:22.416777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '303', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '301', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 0.5799999833106995, 'raw_score_acting': 1.2699999809265137, 'raw_score_stable': 1.2699999809265137, 'primary_affinity_weighted': 0.9818177819252014, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}], 'osds': [{'osd': 0, 'uuid': '4c084047-8f01-4b86-a60c-5592735b5862', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 734, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6801', 'nonce': 4274040097}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6802', 'nonce': 4274040097}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6804', 'nonce': 4274040097}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6803', 'nonce': 4274040097}]}, 'public_addr': '172.21.15.103:6801/4274040097', 'cluster_addr': '172.21.15.103:6802/4274040097', 'heartbeat_back_addr': '172.21.15.103:6804/4274040097', 'heartbeat_front_addr': '172.21.15.103:6803/4274040097', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '3438f2bd-5fb7-432c-b383-29e18c442f20', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 232, 'up_from': 271, 'up_thru': 703, 'down_at': 233, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6805', 'nonce': 859282187}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6806', 'nonce': 859282187}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6808', 'nonce': 859282187}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6807', 'nonce': 859282187}]}, 'public_addr': '172.21.15.103:6805/859282187', 'cluster_addr': '172.21.15.103:6806/859282187', 'heartbeat_back_addr': '172.21.15.103:6808/859282187', 'heartbeat_front_addr': '172.21.15.103:6807/859282187', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c17fffb4-8d17-4546-ba24-e5951a2a37fe', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 481, 'last_clean_end': 780, 'up_from': 804, 'up_thru': 735, 'down_at': 782, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6809', 'nonce': 1571438738}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6810', 'nonce': 1571438738}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6812', 'nonce': 1571438738}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6811', 'nonce': 1571438738}]}, 'public_addr': '172.21.15.103:6809/1571438738', 'cluster_addr': '172.21.15.103:6810/1571438738', 'heartbeat_back_addr': '172.21.15.103:6812/1571438738', 'heartbeat_front_addr': '172.21.15.103:6811/1571438738', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'cee91a3d-55dc-4e89-b568-86821632bb1c', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 27, 'last_clean_end': 559, 'up_from': 619, 'up_thru': 540, 'down_at': 560, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6813', 'nonce': 995549078}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6814', 'nonce': 995549078}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6816', 'nonce': 995549078}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.103:6815', 'nonce': 995549078}]}, 'public_addr': '172.21.15.103:6813/995549078', 'cluster_addr': '172.21.15.103:6814/995549078', 'heartbeat_back_addr': '172.21.15.103:6816/995549078', 'heartbeat_front_addr': '172.21.15.103:6815/995549078', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '61d5385b-2c82-42f8-93c3-8f1baf71c65e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 33, 'last_clean_end': 686, 'up_from': 728, 'up_thru': 609, 'down_at': 687, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6800', 'nonce': 4230909898}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6801', 'nonce': 4230909898}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6803', 'nonce': 4230909898}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6802', 'nonce': 4230909898}]}, 'public_addr': '172.21.15.148:6800/4230909898', 'cluster_addr': '172.21.15.148:6801/4230909898', 'heartbeat_back_addr': '172.21.15.148:6803/4230909898', 'heartbeat_front_addr': '172.21.15.148:6802/4230909898', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '5d492189-c3cd-47ad-a12a-ed4d275bcfe3', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 609, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6804', 'nonce': 3547200711}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6805', 'nonce': 3547200711}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6807', 'nonce': 3547200711}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6806', 'nonce': 3547200711}]}, 'public_addr': '172.21.15.148:6804/3547200711', 'cluster_addr': '172.21.15.148:6805/3547200711', 'heartbeat_back_addr': '172.21.15.148:6807/3547200711', 'heartbeat_front_addr': '172.21.15.148:6806/3547200711', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '89f2379e-cf14-4e69-8747-7535db81b70b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 789, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6808', 'nonce': 366919758}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6809', 'nonce': 366919758}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6811', 'nonce': 366919758}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6810', 'nonce': 366919758}]}, 'public_addr': '172.21.15.148:6808/366919758', 'cluster_addr': '172.21.15.148:6809/366919758', 'heartbeat_back_addr': '172.21.15.148:6811/366919758', 'heartbeat_front_addr': '172.21.15.148:6810/366919758', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f27b035a-558f-4528-80ea-2a00d03f6539', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7818145751953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 732, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6812', 'nonce': 3719550888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6813', 'nonce': 3719550888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6815', 'nonce': 3719550888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.148:6814', 'nonce': 3719550888}]}, 'public_addr': '172.21.15.148:6812/3719550888', 'cluster_addr': '172.21.15.148:6813/3719550888', 'heartbeat_back_addr': '172.21.15.148:6815/3719550888', 'heartbeat_front_addr': '172.21.15.148:6814/3719550888', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 789, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6800', 'nonce': 1151408595}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6801', 'nonce': 1151408595}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6803', 'nonce': 1151408595}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6802', 'nonce': 1151408595}]}, 'public_addr': '172.21.15.188:6800/1151408595', 'cluster_addr': '172.21.15.188:6801/1151408595', 'heartbeat_back_addr': '172.21.15.188:6803/1151408595', 'heartbeat_front_addr': '172.21.15.188:6802/1151408595', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '124d9b05-11eb-4622-b6a2-c4d5bd60b7e6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 703, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6804', 'nonce': 306501862}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6805', 'nonce': 306501862}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6807', 'nonce': 306501862}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6806', 'nonce': 306501862}]}, 'public_addr': '172.21.15.188:6804/306501862', 'cluster_addr': '172.21.15.188:6805/306501862', 'heartbeat_back_addr': '172.21.15.188:6807/306501862', 'heartbeat_front_addr': '172.21.15.188:6806/306501862', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '3dfb9070-e534-4ea4-a25e-ee1c51297447', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6808', 'nonce': 1231263138}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6809', 'nonce': 1231263138}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6811', 'nonce': 1231263138}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6810', 'nonce': 1231263138}]}, 'public_addr': '172.21.15.188:6808/1231263138', 'cluster_addr': '172.21.15.188:6809/1231263138', 'heartbeat_back_addr': '172.21.15.188:6811/1231263138', 'heartbeat_front_addr': '172.21.15.188:6810/1231263138', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'bc163925-f623-4f63-ab89-35e2d511a730', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 703, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6812', 'nonce': 457992661}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6813', 'nonce': 457992661}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6815', 'nonce': 457992661}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.188:6814', 'nonce': 457992661}]}, 'public_addr': '172.21.15.188:6812/457992661', 'cluster_addr': '172.21.15.188:6813/457992661', 'heartbeat_back_addr': '172.21.15.188:6815/457992661', 'heartbeat_front_addr': '172.21.15.188:6814/457992661', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:56:44.847759+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:56:35.075680+0000', 'dead_epoch': 232}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-23T16:57:20.594617+0000', 'dead_epoch': 780}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-23T16:56:25.474649+0000', 'dead_epoch': 559}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-23T16:57:05.867428+0000', 'dead_epoch': 686}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-23T16:56:48.301970+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:57:24.746568+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:56:22.011987+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:57:02.045084+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:56:35.674843+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-23T16:56:35.374566+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T16:56:33.000718+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.103:6800/3535470771': '2024-04-24T16:09:14.324448+0000', '172.21.15.103:0/1188058303': '2024-04-24T16:09:14.324448+0000', '172.21.15.103:0/1510553721': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:0/281182801': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:6800/2049124518': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:0/466241963': '2024-04-24T16:08:44.450632+0000', '172.21.15.103:0/2726838698': '2024-04-24T16:09:14.324448+0000', '172.21.15.103:0/596230573': '2024-04-24T16:09:14.324448+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-23T16:57:29.434 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-23T16:57:29.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:29 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3051520845' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-23T16:57:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:29 smithi103 ceph-mon[39629]: pgmap v2243: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:29 smithi103 ceph-mon[39629]: 1.7 scrub starts 2024-04-23T16:57:29.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:29 smithi103 ceph-mon[39629]: 1.7 scrub ok 2024-04-23T16:57:29.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:29 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3051520845' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-23T16:57:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:29 smithi148 ceph-mon[44520]: pgmap v2243: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:29 smithi148 ceph-mon[44520]: 1.7 scrub starts 2024-04-23T16:57:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:29 smithi148 ceph-mon[44520]: 1.7 scrub ok 2024-04-23T16:57:29.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:29 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3051520845' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-23T16:57:29.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:29 smithi188 ceph-mon[44101]: pgmap v2243: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:29 smithi188 ceph-mon[44101]: 1.7 scrub starts 2024-04-23T16:57:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:29 smithi188 ceph-mon[44101]: 1.7 scrub ok 2024-04-23T16:57:32.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:32 smithi103 ceph-mon[39629]: pgmap v2244: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:32.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:32 smithi148 ceph-mon[44520]: pgmap v2244: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:32.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:32 smithi188 ceph-mon[44101]: pgmap v2244: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:34.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:34 smithi103 ceph-mon[39629]: pgmap v2245: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:34.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:34 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:57:34.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:34 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:57:34.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:34 smithi148 ceph-mon[44520]: pgmap v2245: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:34.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:34 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:57:34.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:34 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:57:34.435 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-23T16:57:34.435 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-04-23T16:57:34.436 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd in 2 2024-04-23T16:57:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:34 smithi188 ceph-mon[44101]: pgmap v2245: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:34 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:57:34.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:34 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:57:34.710 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:57:36.196 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:36 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:57:36.197 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:36 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:57:36.197 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:36 smithi103 ceph-mon[39629]: pgmap v2246: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:36.197 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:36 smithi103 ceph-mon[39629]: osdmap e807: 12 total, 12 up, 7 in 2024-04-23T16:57:36.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:36 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:57:36.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:36 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:57:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:36 smithi148 ceph-mon[44520]: pgmap v2246: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:36.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:36 smithi148 ceph-mon[44520]: osdmap e807: 12 total, 12 up, 7 in 2024-04-23T16:57:36.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:36 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:57:36.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:36 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:57:36.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:36 smithi188 ceph-mon[44101]: pgmap v2246: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:36.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:36 smithi188 ceph-mon[44101]: osdmap e807: 12 total, 12 up, 7 in 2024-04-23T16:57:37.098 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:36 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T16:57:36.693+0000 7f135b1c7700 -1 mon.a@0(leader).osd e807 definitely_dead 0 2024-04-23T16:57:37.108 INFO:teuthology.orchestra.run.smithi103.stderr:marked in osd.2. 2024-04-23T16:57:37.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:37 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2445989291' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-23T16:57:37.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:37 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2445989291' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-23T16:57:37.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:37 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2445989291' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-23T16:57:37.747 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-04-23T16:57:37.747 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-23T16:57:37.747 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-04-23T16:57:37.747 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd in 4 2024-04-23T16:57:38.016 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:57:38.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:38 smithi148 ceph-mon[44520]: pgmap v2248: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:38.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:38 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2445989291' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-04-23T16:57:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:38 smithi148 ceph-mon[44520]: osdmap e808: 12 total, 12 up, 8 in 2024-04-23T16:57:38.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:38 smithi188 ceph-mon[44101]: pgmap v2248: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:38.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:38 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2445989291' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-04-23T16:57:38.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:38 smithi188 ceph-mon[44101]: osdmap e808: 12 total, 12 up, 8 in 2024-04-23T16:57:38.529 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:38 smithi103 ceph-mon[39629]: pgmap v2248: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:38.529 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:38 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2445989291' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-04-23T16:57:38.529 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:38 smithi103 ceph-mon[39629]: osdmap e808: 12 total, 12 up, 8 in 2024-04-23T16:57:39.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:39 smithi103 ceph-mon[39629]: osdmap e809: 12 total, 12 up, 8 in 2024-04-23T16:57:39.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:39 smithi148 ceph-mon[44520]: osdmap e809: 12 total, 12 up, 8 in 2024-04-23T16:57:39.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:39 smithi188 ceph-mon[44101]: osdmap e809: 12 total, 12 up, 8 in 2024-04-23T16:57:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:40 smithi103 ceph-mon[39629]: pgmap v2251: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:40 smithi103 ceph-mon[39629]: osdmap e810: 12 total, 12 up, 8 in 2024-04-23T16:57:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:40 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:57:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:40 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:57:40.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:40 smithi103 ceph-mon[39629]: osdmap e811: 12 total, 12 up, 8 in 2024-04-23T16:57:40.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:40 smithi148 ceph-mon[44520]: pgmap v2251: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:40.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:40 smithi148 ceph-mon[44520]: osdmap e810: 12 total, 12 up, 8 in 2024-04-23T16:57:40.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:40 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:57:40.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:40 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:57:40.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:40 smithi148 ceph-mon[44520]: osdmap e811: 12 total, 12 up, 8 in 2024-04-23T16:57:40.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:40 smithi188 ceph-mon[44101]: pgmap v2251: 11 pgs: 11 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:40.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:40 smithi188 ceph-mon[44101]: osdmap e810: 12 total, 12 up, 8 in 2024-04-23T16:57:40.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:40 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:57:40.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:40 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:57:40.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:40 smithi188 ceph-mon[44101]: osdmap e811: 12 total, 12 up, 8 in 2024-04-23T16:57:40.568 INFO:teuthology.orchestra.run.smithi103.stderr:marked in osd.4. 2024-04-23T16:57:40.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:40 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T16:57:40.374+0000 7f135b1c7700 -1 mon.a@0(leader).osd e811 definitely_dead 0 2024-04-23T16:57:41.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:41 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2390407119' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-04-23T16:57:41.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:41 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2390407119' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-04-23T16:57:41.214 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:41 smithi103 ceph-mon[39629]: osdmap e812: 12 total, 12 up, 9 in 2024-04-23T16:57:41.240 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-04-23T16:57:41.240 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-23T16:57:41.240 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-04-23T16:57:41.241 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd in 10 2024-04-23T16:57:41.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:41 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2390407119' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-04-23T16:57:41.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:41 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2390407119' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-04-23T16:57:41.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:41 smithi148 ceph-mon[44520]: osdmap e812: 12 total, 12 up, 9 in 2024-04-23T16:57:41.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:41 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2390407119' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-04-23T16:57:41.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:41 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2390407119' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-04-23T16:57:41.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:41 smithi188 ceph-mon[44101]: osdmap e812: 12 total, 12 up, 9 in 2024-04-23T16:57:41.508 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:57:42.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:42 smithi103 ceph-mon[39629]: pgmap v2255: 11 pgs: 1 active+remapped, 10 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:42.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:42 smithi103 ceph-mon[39629]: 1.8 scrub starts 2024-04-23T16:57:42.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:42 smithi103 ceph-mon[39629]: 1.8 scrub ok 2024-04-23T16:57:42.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:42 smithi148 ceph-mon[44520]: pgmap v2255: 11 pgs: 1 active+remapped, 10 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:42 smithi148 ceph-mon[44520]: 1.8 scrub starts 2024-04-23T16:57:42.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:42 smithi148 ceph-mon[44520]: 1.8 scrub ok 2024-04-23T16:57:42.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:42 smithi188 ceph-mon[44101]: pgmap v2255: 11 pgs: 1 active+remapped, 10 active+clean; 577 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail 2024-04-23T16:57:42.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:42 smithi188 ceph-mon[44101]: 1.8 scrub starts 2024-04-23T16:57:42.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:42 smithi188 ceph-mon[44101]: 1.8 scrub ok 2024-04-23T16:57:43.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:43 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T16:57:43.379+0000 7f135b1c7700 -1 mon.a@0(leader).osd e812 definitely_dead 0 2024-04-23T16:57:44.127 INFO:teuthology.orchestra.run.smithi103.stderr:marked in osd.10. 2024-04-23T16:57:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:44 smithi188 ceph-mon[44101]: pgmap v2256: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:57:44.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:44 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3323685048' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-04-23T16:57:44.532 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:44 smithi103 ceph-mon[39629]: pgmap v2256: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:57:44.532 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:44 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3323685048' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-04-23T16:57:44.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:44 smithi148 ceph-mon[44520]: pgmap v2256: 11 pgs: 11 active+clean; 577 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-04-23T16:57:44.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:44 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3323685048' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-04-23T16:57:44.983 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-04-23T16:57:44.984 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-23T16:57:44.984 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-04-23T16:57:44.984 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd in 5 2024-04-23T16:57:45.253 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:57:45.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:45 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:57:45.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:45 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:57:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:45 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3323685048' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-04-23T16:57:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:45 smithi103 ceph-mon[39629]: osdmap e813: 12 total, 12 up, 10 in 2024-04-23T16:57:45.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:45 smithi103 ceph-mon[39629]: osdmap e814: 12 total, 12 up, 10 in 2024-04-23T16:57:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:45 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:57:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:45 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:57:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:45 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3323685048' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-04-23T16:57:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:45 smithi188 ceph-mon[44101]: osdmap e813: 12 total, 12 up, 10 in 2024-04-23T16:57:45.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:45 smithi188 ceph-mon[44101]: osdmap e814: 12 total, 12 up, 10 in 2024-04-23T16:57:45.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:45 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:57:45.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:45 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:57:45.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:45 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3323685048' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-04-23T16:57:45.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:45 smithi148 ceph-mon[44520]: osdmap e813: 12 total, 12 up, 10 in 2024-04-23T16:57:45.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:45 smithi148 ceph-mon[44520]: osdmap e814: 12 total, 12 up, 10 in 2024-04-23T16:57:46.454 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:46 smithi103 ceph-mon[39629]: pgmap v2259: 11 pgs: 11 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:57:46.454 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:46 smithi103 ceph-mon[39629]: osdmap e815: 12 total, 12 up, 10 in 2024-04-23T16:57:46.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:46 smithi188 ceph-mon[44101]: pgmap v2259: 11 pgs: 11 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:57:46.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:46 smithi188 ceph-mon[44101]: osdmap e815: 12 total, 12 up, 10 in 2024-04-23T16:57:46.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:46 smithi148 ceph-mon[44520]: pgmap v2259: 11 pgs: 11 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:57:46.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:46 smithi148 ceph-mon[44520]: osdmap e815: 12 total, 12 up, 10 in 2024-04-23T16:57:47.586 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:47 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T16:57:47.233+0000 7f135b1c7700 -1 mon.a@0(leader).osd e816 definitely_dead 0 2024-04-23T16:57:47.586 INFO:teuthology.orchestra.run.smithi103.stderr:marked in osd.5. 2024-04-23T16:57:47.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:47 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:57:47.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:47 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:57:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:47 smithi103 ceph-mon[39629]: osdmap e816: 12 total, 12 up, 10 in 2024-04-23T16:57:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:47 smithi103 ceph-mon[39629]: pgmap v2262: 11 pgs: 1 active+clean+remapped, 10 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:57:47.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:47 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1118423324' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-23T16:57:47.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:47 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:57:47.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:47 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:57:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:47 smithi148 ceph-mon[44520]: osdmap e816: 12 total, 12 up, 10 in 2024-04-23T16:57:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:47 smithi148 ceph-mon[44520]: pgmap v2262: 11 pgs: 1 active+clean+remapped, 10 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:57:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:47 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1118423324' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-23T16:57:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:47 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:57:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:47 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:57:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:47 smithi188 ceph-mon[44101]: osdmap e816: 12 total, 12 up, 10 in 2024-04-23T16:57:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:47 smithi188 ceph-mon[44101]: pgmap v2262: 11 pgs: 1 active+clean+remapped, 10 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-04-23T16:57:47.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:47 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1118423324' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-23T16:57:48.343 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-04-23T16:57:48.343 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-23T16:57:48.344 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-04-23T16:57:48.344 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd in 3 2024-04-23T16:57:48.610 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:57:48.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:48 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1118423324' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-04-23T16:57:48.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:48 smithi103 ceph-mon[39629]: osdmap e817: 12 total, 12 up, 11 in 2024-04-23T16:57:48.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:48 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1118423324' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-04-23T16:57:48.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:48 smithi148 ceph-mon[44520]: osdmap e817: 12 total, 12 up, 11 in 2024-04-23T16:57:48.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:48 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1118423324' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-04-23T16:57:48.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:48 smithi188 ceph-mon[44101]: osdmap e817: 12 total, 12 up, 11 in 2024-04-23T16:57:49.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:49 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:57:49.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:49 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:57:49.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:49 smithi103 ceph-mon[39629]: osdmap e818: 12 total, 12 up, 11 in 2024-04-23T16:57:49.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:49 smithi103 ceph-mon[39629]: pgmap v2265: 11 pgs: 1 unknown, 1 active+clean+remapped, 9 active+clean; 320 KiB data, 5.9 GiB used, 978 GiB / 983 GiB avail 2024-04-23T16:57:49.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:49 smithi103 ceph-mon[39629]: osdmap e819: 12 total, 12 up, 11 in 2024-04-23T16:57:49.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:49 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:57:49.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:49 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:57:49.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:49 smithi148 ceph-mon[44520]: osdmap e818: 12 total, 12 up, 11 in 2024-04-23T16:57:49.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:49 smithi148 ceph-mon[44520]: pgmap v2265: 11 pgs: 1 unknown, 1 active+clean+remapped, 9 active+clean; 320 KiB data, 5.9 GiB used, 978 GiB / 983 GiB avail 2024-04-23T16:57:49.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:49 smithi148 ceph-mon[44520]: osdmap e819: 12 total, 12 up, 11 in 2024-04-23T16:57:49.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:49 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:57:49.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:49 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:57:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:49 smithi188 ceph-mon[44101]: osdmap e818: 12 total, 12 up, 11 in 2024-04-23T16:57:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:49 smithi188 ceph-mon[44101]: pgmap v2265: 11 pgs: 1 unknown, 1 active+clean+remapped, 9 active+clean; 320 KiB data, 5.9 GiB used, 978 GiB / 983 GiB avail 2024-04-23T16:57:49.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:49 smithi188 ceph-mon[44101]: osdmap e819: 12 total, 12 up, 11 in 2024-04-23T16:57:50.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:50 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T16:57:50.587+0000 7f135b1c7700 -1 mon.a@0(leader).osd e820 definitely_dead 0 2024-04-23T16:57:50.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:50 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T16:57:50.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:50 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T16:57:50.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:50 smithi103 ceph-mon[39629]: osdmap e820: 12 total, 12 up, 11 in 2024-04-23T16:57:50.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:50 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/631706706' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-23T16:57:50.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:50 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T16:57:50.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:50 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T16:57:50.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:50 smithi148 ceph-mon[44520]: osdmap e820: 12 total, 12 up, 11 in 2024-04-23T16:57:50.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:50 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/631706706' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-23T16:57:50.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:50 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T16:57:50.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:50 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T16:57:50.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:50 smithi188 ceph-mon[44101]: osdmap e820: 12 total, 12 up, 11 in 2024-04-23T16:57:50.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:50 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/631706706' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-23T16:57:51.574 INFO:teuthology.orchestra.run.smithi103.stderr:marked in osd.3. 2024-04-23T16:57:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:51 smithi103 ceph-mon[39629]: pgmap v2268: 11 pgs: 1 unknown, 10 active+clean; 320 KiB data, 5.9 GiB used, 978 GiB / 983 GiB avail 2024-04-23T16:57:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:51 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/631706706' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-04-23T16:57:51.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:51 smithi103 ceph-mon[39629]: osdmap e821: 12 total, 12 up, 12 in 2024-04-23T16:57:51.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:51 smithi148 ceph-mon[44520]: pgmap v2268: 11 pgs: 1 unknown, 10 active+clean; 320 KiB data, 5.9 GiB used, 978 GiB / 983 GiB avail 2024-04-23T16:57:51.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:51 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/631706706' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-04-23T16:57:51.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:51 smithi148 ceph-mon[44520]: osdmap e821: 12 total, 12 up, 12 in 2024-04-23T16:57:51.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:51 smithi188 ceph-mon[44101]: pgmap v2268: 11 pgs: 1 unknown, 10 active+clean; 320 KiB data, 5.9 GiB used, 978 GiB / 983 GiB avail 2024-04-23T16:57:51.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:51 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/631706706' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-04-23T16:57:51.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:51 smithi188 ceph-mon[44101]: osdmap e821: 12 total, 12 up, 12 in 2024-04-23T16:57:52.240 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-04-23T16:57:52.240 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-04-23T16:57:52.240 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-04-23T16:57:52.511 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:57:54.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:54 smithi103 ceph-mon[39629]: pgmap v2270: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 177 KiB/s, 0 objects/s recovering 2024-04-23T16:57:54.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:54 smithi148 ceph-mon[44520]: pgmap v2270: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 177 KiB/s, 0 objects/s recovering 2024-04-23T16:57:54.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:54 smithi188 ceph-mon[44101]: pgmap v2270: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 177 KiB/s, 0 objects/s recovering 2024-04-23T16:57:55.098 INFO:teuthology.orchestra.run.smithi103.stderr:set pool 1 pgp_num to 11 2024-04-23T16:57:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:55 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:57:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:55 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:57:55.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:55 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3944005212' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-23T16:57:55.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:55 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:57:55.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:55 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:57:55.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:55 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3944005212' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-23T16:57:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:55 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:57:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:55 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:57:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:55 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3944005212' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-23T16:57:55.760 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-23T16:57:56.028 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:57:56.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:56 smithi148 ceph-mon[44520]: pgmap v2271: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-23T16:57:56.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:56 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3944005212' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-23T16:57:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:56 smithi148 ceph-mon[44520]: osdmap e822: 12 total, 12 up, 12 in 2024-04-23T16:57:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:56 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:57:56.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:56 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:57:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:56 smithi188 ceph-mon[44101]: pgmap v2271: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-23T16:57:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:56 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3944005212' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-23T16:57:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:56 smithi188 ceph-mon[44101]: osdmap e822: 12 total, 12 up, 12 in 2024-04-23T16:57:56.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:56 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:57:56.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:56 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:57:56.508 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:56 smithi103 ceph-mon[39629]: pgmap v2271: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-23T16:57:56.508 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:56 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3944005212' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-23T16:57:56.508 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:56 smithi103 ceph-mon[39629]: osdmap e822: 12 total, 12 up, 12 in 2024-04-23T16:57:56.508 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:56 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:57:56.508 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:56 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:57:57.263 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:57 smithi103 ceph-mon[39629]: osdmap e823: 12 total, 12 up, 12 in 2024-04-23T16:57:57.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:57 smithi148 ceph-mon[44520]: osdmap e823: 12 total, 12 up, 12 in 2024-04-23T16:57:57.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:57 smithi188 ceph-mon[44101]: osdmap e823: 12 total, 12 up, 12 in 2024-04-23T16:57:57.821 INFO:teuthology.orchestra.run.smithi103.stdout:mon.a: {} 2024-04-23T16:57:57.831 INFO:teuthology.orchestra.run.smithi103.stdout:mon.c: {} 2024-04-23T16:57:57.842 INFO:teuthology.orchestra.run.smithi103.stdout:mon.b: {} 2024-04-23T16:57:57.843 INFO:teuthology.orchestra.run.smithi103.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-23T16:57:57.843 INFO:teuthology.orchestra.run.smithi103.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-23T16:57:57.843 INFO:teuthology.orchestra.run.smithi103.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-23T16:57:58.190 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:58 smithi103 ceph-mon[39629]: pgmap v2274: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-23T16:57:58.190 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:58 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3186076100' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T16:57:58.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:58 smithi148 ceph-mon[44520]: pgmap v2274: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-23T16:57:58.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:58 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3186076100' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T16:57:58.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:58 smithi188 ceph-mon[44101]: pgmap v2274: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-23T16:57:58.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:58 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3186076100' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T16:57:58.626 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-23T16:57:58.892 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:57:59.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:59 smithi148 ceph-mon[44520]: 1.9 scrub starts 2024-04-23T16:57:59.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:59 smithi148 ceph-mon[44520]: 1.9 scrub ok 2024-04-23T16:57:59.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:57:59 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:57:59.437 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:59 smithi103 ceph-mon[39629]: 1.9 scrub starts 2024-04-23T16:57:59.437 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:59 smithi103 ceph-mon[39629]: 1.9 scrub ok 2024-04-23T16:57:59.437 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:57:59 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:57:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:59 smithi188 ceph-mon[44101]: 1.9 scrub starts 2024-04-23T16:57:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:59 smithi188 ceph-mon[44101]: 1.9 scrub ok 2024-04-23T16:57:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:57:59 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:58:00.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:00 smithi148 ceph-mon[44520]: pgmap v2275: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-23T16:58:00.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:00 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:58:00.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:00 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:58:00.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:00 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:58:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:00 smithi188 ceph-mon[44101]: pgmap v2275: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-23T16:58:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:00 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:58:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:00 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:58:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:00 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:58:00.519 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:00 smithi103 ceph-mon[39629]: pgmap v2275: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-23T16:58:00.520 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:00 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:58:00.520 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:00 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:58:00.520 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:00 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:58:01.125 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.0 to 1 (10000) 2024-04-23T16:58:01.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:01 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/717274333' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:58:01.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:01 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/717274333' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:58:01.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:01 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/717274333' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:58:01.879 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 0 1 2024-04-23T16:58:02.123 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:02 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:58:02.123 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:02 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:58:02.123 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:02 smithi103 ceph-mon[39629]: pgmap v2276: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:02.145 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:58:02.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:02 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:58:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:02 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:58:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:02 smithi148 ceph-mon[44520]: pgmap v2276: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:02 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/717274333' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:58:02.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:02 smithi148 ceph-mon[44520]: osdmap e824: 12 total, 12 up, 12 in 2024-04-23T16:58:02.382 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:02 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/717274333' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:58:02.382 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:02 smithi103 ceph-mon[39629]: osdmap e824: 12 total, 12 up, 12 in 2024-04-23T16:58:02.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:02 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:58:02.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:02 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:58:02.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:02 smithi188 ceph-mon[44101]: pgmap v2276: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:02.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:02 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/717274333' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:58:02.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:02 smithi188 ceph-mon[44101]: osdmap e824: 12 total, 12 up, 12 in 2024-04-23T16:58:04.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:04 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:58:04.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:04 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:58:04.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:04 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:58:04.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:04 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:58:04.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:04 smithi188 ceph-mon[44101]: pgmap v2278: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:04.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:04 smithi188 ceph-mon[44101]: osdmap e825: 12 total, 12 up, 12 in 2024-04-23T16:58:04.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:04 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:58:04.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:04 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:58:04.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:04 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:58:04.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:04 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:58:04.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:04 smithi103 ceph-mon[39629]: pgmap v2278: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:04.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:04 smithi103 ceph-mon[39629]: osdmap e825: 12 total, 12 up, 12 in 2024-04-23T16:58:04.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:04 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:58:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:04 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:58:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:04 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:58:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:04 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:58:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:04 smithi148 ceph-mon[44520]: pgmap v2278: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:04 smithi148 ceph-mon[44520]: osdmap e825: 12 total, 12 up, 12 in 2024-04-23T16:58:05.141 INFO:teuthology.orchestra.run.smithi103.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-23T16:58:05.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:05 smithi188 ceph-mon[44101]: 1.5 scrub starts 2024-04-23T16:58:05.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:05 smithi188 ceph-mon[44101]: 1.5 scrub ok 2024-04-23T16:58:05.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:05 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3834662452' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:58:05.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:05 smithi103 ceph-mon[39629]: 1.5 scrub starts 2024-04-23T16:58:05.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:05 smithi103 ceph-mon[39629]: 1.5 scrub ok 2024-04-23T16:58:05.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:05 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3834662452' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:58:05.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:05 smithi148 ceph-mon[44520]: 1.5 scrub starts 2024-04-23T16:58:05.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:05 smithi148 ceph-mon[44520]: 1.5 scrub ok 2024-04-23T16:58:05.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:05 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3834662452' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-23T16:58:05.986 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-23T16:58:06.256 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:58:06.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:06 smithi103 ceph-mon[39629]: pgmap v2280: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:06.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:06 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3834662452' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:58:06.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:06 smithi103 ceph-mon[39629]: osdmap e826: 12 total, 12 up, 12 in 2024-04-23T16:58:06.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:06 smithi188 ceph-mon[44101]: pgmap v2280: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:06.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:06 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3834662452' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:58:06.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:06 smithi188 ceph-mon[44101]: osdmap e826: 12 total, 12 up, 12 in 2024-04-23T16:58:06.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:06 smithi148 ceph-mon[44520]: pgmap v2280: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:06.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:06 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3834662452' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-23T16:58:06.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:06 smithi148 ceph-mon[44520]: osdmap e826: 12 total, 12 up, 12 in 2024-04-23T16:58:08.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:08 smithi188 ceph-mon[44101]: pgmap v2282: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:08.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:08 smithi103 ceph-mon[39629]: pgmap v2282: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:08.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:08 smithi148 ceph-mon[44520]: pgmap v2282: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:09.154 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.5 to 1 (10000) 2024-04-23T16:58:09.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:09 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3668667529' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-23T16:58:09.495 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:09 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3668667529' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-23T16:58:09.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:09 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3668667529' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-23T16:58:09.951 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 5 1 2024-04-23T16:58:10.222 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:58:10.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:10 smithi103 ceph-mon[39629]: pgmap v2283: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:10.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:10 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3668667529' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-23T16:58:10.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:10 smithi103 ceph-mon[39629]: osdmap e827: 12 total, 12 up, 12 in 2024-04-23T16:58:10.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:10 smithi188 ceph-mon[44101]: pgmap v2283: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:10.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:10 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3668667529' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-23T16:58:10.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:10 smithi188 ceph-mon[44101]: osdmap e827: 12 total, 12 up, 12 in 2024-04-23T16:58:10.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:10 smithi148 ceph-mon[44520]: pgmap v2283: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:10.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:10 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3668667529' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-23T16:58:10.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:10 smithi148 ceph-mon[44520]: osdmap e827: 12 total, 12 up, 12 in 2024-04-23T16:58:12.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:12 smithi188 ceph-mon[44101]: pgmap v2285: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:12.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:12 smithi103 ceph-mon[39629]: pgmap v2285: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:12.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:12 smithi148 ceph-mon[44520]: pgmap v2285: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:13.164 INFO:teuthology.orchestra.run.smithi103.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-23T16:58:13.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:13 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:58:13.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:13 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:58:13.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:13 smithi188 ceph-mon[44101]: osdmap e828: 12 total, 12 up, 12 in 2024-04-23T16:58:13.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:13 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/632703459' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-23T16:58:13.553 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:13 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:58:13.553 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:13 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:58:13.553 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:13 smithi103 ceph-mon[39629]: osdmap e828: 12 total, 12 up, 12 in 2024-04-23T16:58:13.553 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:13 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/632703459' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-23T16:58:13.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:13 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:58:13.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:13 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:58:13.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:13 smithi148 ceph-mon[44520]: osdmap e828: 12 total, 12 up, 12 in 2024-04-23T16:58:13.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:13 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/632703459' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-23T16:58:14.009 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-23T16:58:14.282 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:58:14.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:14 smithi103 ceph-mon[39629]: pgmap v2287: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:14.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:14 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/632703459' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-23T16:58:14.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:14 smithi103 ceph-mon[39629]: osdmap e829: 12 total, 12 up, 12 in 2024-04-23T16:58:14.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:14 smithi188 ceph-mon[44101]: pgmap v2287: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:14.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:14 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/632703459' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-23T16:58:14.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:14 smithi188 ceph-mon[44101]: osdmap e829: 12 total, 12 up, 12 in 2024-04-23T16:58:14.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:14 smithi148 ceph-mon[44520]: pgmap v2287: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:14.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:14 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/632703459' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-23T16:58:14.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:14 smithi148 ceph-mon[44520]: osdmap e829: 12 total, 12 up, 12 in 2024-04-23T16:58:16.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:16 smithi188 ceph-mon[44101]: pgmap v2289: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:16.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:16 smithi103 ceph-mon[39629]: pgmap v2289: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:16.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:16 smithi148 ceph-mon[44520]: pgmap v2289: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:17.183 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.6 to 1 (10000) 2024-04-23T16:58:17.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:17 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/196741061' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:58:17.551 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:17 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/196741061' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:58:17.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:17 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/196741061' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:58:17.973 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 6 1 2024-04-23T16:58:18.245 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:58:18.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:18 smithi103 ceph-mon[39629]: pgmap v2290: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:18.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:18 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/196741061' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:58:18.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:18 smithi103 ceph-mon[39629]: osdmap e830: 12 total, 12 up, 12 in 2024-04-23T16:58:18.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:18 smithi188 ceph-mon[44101]: pgmap v2290: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:18.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:18 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/196741061' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:58:18.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:18 smithi188 ceph-mon[44101]: osdmap e830: 12 total, 12 up, 12 in 2024-04-23T16:58:18.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:18 smithi148 ceph-mon[44520]: pgmap v2290: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:18 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/196741061' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:58:18.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:18 smithi148 ceph-mon[44520]: osdmap e830: 12 total, 12 up, 12 in 2024-04-23T16:58:20.186 INFO:teuthology.orchestra.run.smithi103.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-23T16:58:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:20 smithi188 ceph-mon[44101]: pgmap v2292: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:20 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1238287839' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:58:20.560 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:20 smithi103 ceph-mon[39629]: pgmap v2292: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:20.560 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:20 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1238287839' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:58:20.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:20 smithi148 ceph-mon[44520]: pgmap v2292: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:20.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:20 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1238287839' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-23T16:58:20.945 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-23T16:58:21.216 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:58:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:21 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1238287839' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:58:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:21 smithi188 ceph-mon[44101]: osdmap e831: 12 total, 12 up, 12 in 2024-04-23T16:58:21.476 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:21 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1238287839' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:58:21.476 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:21 smithi103 ceph-mon[39629]: osdmap e831: 12 total, 12 up, 12 in 2024-04-23T16:58:21.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:21 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1238287839' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-23T16:58:21.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:21 smithi148 ceph-mon[44520]: osdmap e831: 12 total, 12 up, 12 in 2024-04-23T16:58:22.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:22 smithi103 ceph-mon[39629]: pgmap v2294: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:22.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:22 smithi188 ceph-mon[44101]: pgmap v2294: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:22.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:22 smithi148 ceph-mon[44520]: pgmap v2294: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:23.213 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.7 to 1 (10000) 2024-04-23T16:58:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:23 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1909961717' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:58:23.556 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:23 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1909961717' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:58:23.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:23 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1909961717' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:58:23.856 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 7 1 2024-04-23T16:58:24.124 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:58:24.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:24 smithi188 ceph-mon[44101]: pgmap v2295: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:24.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1909961717' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:58:24.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:24 smithi188 ceph-mon[44101]: osdmap e832: 12 total, 12 up, 12 in 2024-04-23T16:58:24.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:24 smithi103 ceph-mon[39629]: pgmap v2295: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:24.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1909961717' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:58:24.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:24 smithi103 ceph-mon[39629]: osdmap e832: 12 total, 12 up, 12 in 2024-04-23T16:58:24.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:24 smithi148 ceph-mon[44520]: pgmap v2295: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:24.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:24 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1909961717' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:58:24.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:24 smithi148 ceph-mon[44520]: osdmap e832: 12 total, 12 up, 12 in 2024-04-23T16:58:25.660 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:25 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:58:25.661 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:25 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:58:25.661 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:25 smithi103 ceph-mon[39629]: pgmap v2297: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:25.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:25 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:58:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:25 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:58:25.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:25 smithi148 ceph-mon[44520]: pgmap v2297: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:25.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:25 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:58:25.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:25 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:58:25.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:25 smithi188 ceph-mon[44101]: pgmap v2297: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:26.592 INFO:teuthology.orchestra.run.smithi103.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-04-23T16:58:26.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:26 smithi103 ceph-mon[39629]: osdmap e833: 12 total, 12 up, 12 in 2024-04-23T16:58:26.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:26 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1753694972' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:58:26.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:26 smithi148 ceph-mon[44520]: osdmap e833: 12 total, 12 up, 12 in 2024-04-23T16:58:26.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:26 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1753694972' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:58:26.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:26 smithi188 ceph-mon[44101]: osdmap e833: 12 total, 12 up, 12 in 2024-04-23T16:58:26.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:26 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1753694972' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-23T16:58:27.363 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 8 1 2024-04-23T16:58:27.633 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:58:27.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:27 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1753694972' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:58:27.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:27 smithi103 ceph-mon[39629]: osdmap e834: 12 total, 12 up, 12 in 2024-04-23T16:58:27.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:27 smithi103 ceph-mon[39629]: pgmap v2300: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:27.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:27 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1753694972' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:58:27.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:27 smithi148 ceph-mon[44520]: osdmap e834: 12 total, 12 up, 12 in 2024-04-23T16:58:27.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:27 smithi148 ceph-mon[44520]: pgmap v2300: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:27.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:27 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1753694972' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-23T16:58:27.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:27 smithi188 ceph-mon[44101]: osdmap e834: 12 total, 12 up, 12 in 2024-04-23T16:58:27.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:27 smithi188 ceph-mon[44101]: pgmap v2300: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:28.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:28 smithi103 ceph-mon[39629]: 1.4 scrub starts 2024-04-23T16:58:28.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:28 smithi148 ceph-mon[44520]: 1.4 scrub starts 2024-04-23T16:58:28.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:28 smithi188 ceph-mon[44101]: 1.4 scrub starts 2024-04-23T16:58:29.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:29 smithi148 ceph-mon[44520]: 1.4 scrub ok 2024-04-23T16:58:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:29 smithi148 ceph-mon[44520]: osdmap e835: 12 total, 12 up, 12 in 2024-04-23T16:58:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:29 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:58:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:29 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:58:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:29 smithi148 ceph-mon[44520]: 1.2 scrub starts 2024-04-23T16:58:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:29 smithi148 ceph-mon[44520]: 1.2 scrub ok 2024-04-23T16:58:29.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:29 smithi148 ceph-mon[44520]: pgmap v2302: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:29.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:29 smithi188 ceph-mon[44101]: 1.4 scrub ok 2024-04-23T16:58:29.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:29 smithi188 ceph-mon[44101]: osdmap e835: 12 total, 12 up, 12 in 2024-04-23T16:58:29.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:29 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:58:29.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:29 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:58:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:29 smithi188 ceph-mon[44101]: 1.2 scrub starts 2024-04-23T16:58:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:29 smithi188 ceph-mon[44101]: 1.2 scrub ok 2024-04-23T16:58:29.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:29 smithi188 ceph-mon[44101]: pgmap v2302: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:29 smithi103 ceph-mon[39629]: 1.4 scrub ok 2024-04-23T16:58:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:29 smithi103 ceph-mon[39629]: osdmap e835: 12 total, 12 up, 12 in 2024-04-23T16:58:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:29 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:58:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:29 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:58:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:29 smithi103 ceph-mon[39629]: 1.2 scrub starts 2024-04-23T16:58:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:29 smithi103 ceph-mon[39629]: 1.2 scrub ok 2024-04-23T16:58:30.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:29 smithi103 ceph-mon[39629]: pgmap v2302: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:30.613 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.8 to 1 (10000) 2024-04-23T16:58:30.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:30 smithi148 ceph-mon[44520]: osdmap e836: 12 total, 12 up, 12 in 2024-04-23T16:58:30.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:30 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1757954144' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:58:30.970 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:30 smithi103 ceph-mon[39629]: osdmap e836: 12 total, 12 up, 12 in 2024-04-23T16:58:30.971 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:30 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1757954144' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:58:30.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:30 smithi188 ceph-mon[44101]: osdmap e836: 12 total, 12 up, 12 in 2024-04-23T16:58:30.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:30 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1757954144' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:58:31.318 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 8 1 2024-04-23T16:58:31.589 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:58:31.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:31 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1757954144' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:58:31.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:31 smithi148 ceph-mon[44520]: osdmap e837: 12 total, 12 up, 12 in 2024-04-23T16:58:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:31 smithi148 ceph-mon[44520]: 1.a scrub starts 2024-04-23T16:58:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:31 smithi148 ceph-mon[44520]: pgmap v2305: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:31.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:31 smithi148 ceph-mon[44520]: 1.a scrub ok 2024-04-23T16:58:31.883 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:31 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1757954144' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:58:31.883 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:31 smithi103 ceph-mon[39629]: osdmap e837: 12 total, 12 up, 12 in 2024-04-23T16:58:31.883 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:31 smithi103 ceph-mon[39629]: 1.a scrub starts 2024-04-23T16:58:31.883 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:31 smithi103 ceph-mon[39629]: pgmap v2305: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:31.883 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:31 smithi103 ceph-mon[39629]: 1.a scrub ok 2024-04-23T16:58:31.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:31 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1757954144' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:58:31.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:31 smithi188 ceph-mon[44101]: osdmap e837: 12 total, 12 up, 12 in 2024-04-23T16:58:31.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:31 smithi188 ceph-mon[44101]: 1.a scrub starts 2024-04-23T16:58:31.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:31 smithi188 ceph-mon[44101]: pgmap v2305: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:31.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:31 smithi188 ceph-mon[44101]: 1.a scrub ok 2024-04-23T16:58:34.104 INFO:teuthology.orchestra.run.smithi103.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-04-23T16:58:34.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:34 smithi148 ceph-mon[44520]: pgmap v2306: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:34.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:34 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3851616400' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:58:34.469 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:34 smithi103 ceph-mon[39629]: pgmap v2306: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:34.469 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:34 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3851616400' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:58:34.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:34 smithi188 ceph-mon[44101]: pgmap v2306: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:34.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:34 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3851616400' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-04-23T16:58:34.856 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 9 1 2024-04-23T16:58:35.128 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:58:35.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:35 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3851616400' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:58:35.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:35 smithi148 ceph-mon[44520]: osdmap e838: 12 total, 12 up, 12 in 2024-04-23T16:58:35.378 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:35 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3851616400' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:58:35.378 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:35 smithi103 ceph-mon[39629]: osdmap e838: 12 total, 12 up, 12 in 2024-04-23T16:58:35.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:35 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3851616400' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-04-23T16:58:35.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:35 smithi188 ceph-mon[44101]: osdmap e838: 12 total, 12 up, 12 in 2024-04-23T16:58:36.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:36 smithi103 ceph-mon[39629]: pgmap v2308: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:36.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:36 smithi148 ceph-mon[44520]: pgmap v2308: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:36.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:36 smithi188 ceph-mon[44101]: pgmap v2308: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:37.116 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.9 to 1 (10000) 2024-04-23T16:58:37.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:37 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1967059148' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:58:37.459 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:37 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1967059148' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:58:37.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:37 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1967059148' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:58:37.876 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 9 1 2024-04-23T16:58:38.142 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:58:38.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:38 smithi148 ceph-mon[44520]: pgmap v2309: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:38.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:38 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1967059148' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:58:38.375 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:38 smithi148 ceph-mon[44520]: osdmap e839: 12 total, 12 up, 12 in 2024-04-23T16:58:38.394 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:38 smithi103 ceph-mon[39629]: pgmap v2309: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:38.395 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:38 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1967059148' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:58:38.395 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:38 smithi103 ceph-mon[39629]: osdmap e839: 12 total, 12 up, 12 in 2024-04-23T16:58:38.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:38 smithi188 ceph-mon[44101]: pgmap v2309: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:38.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:38 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1967059148' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:58:38.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:38 smithi188 ceph-mon[44101]: osdmap e839: 12 total, 12 up, 12 in 2024-04-23T16:58:40.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:40 smithi148 ceph-mon[44520]: pgmap v2311: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:40.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:40 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3299829021' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:58:40.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:40 smithi188 ceph-mon[44101]: pgmap v2311: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:40.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:40 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3299829021' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:58:40.583 INFO:teuthology.orchestra.run.smithi103.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-04-23T16:58:40.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:40 smithi103 ceph-mon[39629]: pgmap v2311: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:40.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:40 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3299829021' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-04-23T16:58:41.546 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 10 1 2024-04-23T16:58:41.815 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:58:41.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:41 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3299829021' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:58:41.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:41 smithi103 ceph-mon[39629]: osdmap e840: 12 total, 12 up, 12 in 2024-04-23T16:58:41.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:41 smithi103 ceph-mon[39629]: pgmap v2313: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:41.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:41 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3299829021' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:58:41.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:41 smithi148 ceph-mon[44520]: osdmap e840: 12 total, 12 up, 12 in 2024-04-23T16:58:41.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:41 smithi148 ceph-mon[44520]: pgmap v2313: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:41.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:41 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3299829021' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-04-23T16:58:41.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:41 smithi188 ceph-mon[44101]: osdmap e840: 12 total, 12 up, 12 in 2024-04-23T16:58:41.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:41 smithi188 ceph-mon[44101]: pgmap v2313: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:44.105 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.10 to 1 (10000) 2024-04-23T16:58:44.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:44 smithi148 ceph-mon[44520]: pgmap v2314: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:44.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:44 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/481756534' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-23T16:58:44.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:44 smithi188 ceph-mon[44101]: pgmap v2314: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:44.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:44 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/481756534' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-23T16:58:44.478 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:44 smithi103 ceph-mon[39629]: pgmap v2314: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:44.478 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:44 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/481756534' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-23T16:58:44.879 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 10 1 2024-04-23T16:58:45.109 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:45 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/481756534' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-23T16:58:45.109 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:45 smithi103 ceph-mon[39629]: osdmap e841: 12 total, 12 up, 12 in 2024-04-23T16:58:45.149 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:58:45.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:45 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/481756534' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-23T16:58:45.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:45 smithi148 ceph-mon[44520]: osdmap e841: 12 total, 12 up, 12 in 2024-04-23T16:58:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:45 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/481756534' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-23T16:58:45.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:45 smithi188 ceph-mon[44101]: osdmap e841: 12 total, 12 up, 12 in 2024-04-23T16:58:46.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:46 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:58:46.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:46 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:58:46.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:46 smithi148 ceph-mon[44520]: pgmap v2316: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:46.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:46 smithi148 ceph-mon[44520]: osdmap e842: 12 total, 12 up, 12 in 2024-04-23T16:58:46.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:46 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:58:46.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:46 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:58:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:46 smithi188 ceph-mon[44101]: pgmap v2316: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:46.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:46 smithi188 ceph-mon[44101]: osdmap e842: 12 total, 12 up, 12 in 2024-04-23T16:58:46.474 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:46 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:58:46.474 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:46 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:58:46.474 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:46 smithi103 ceph-mon[39629]: pgmap v2316: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:46.474 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:46 smithi103 ceph-mon[39629]: osdmap e842: 12 total, 12 up, 12 in 2024-04-23T16:58:48.129 INFO:teuthology.orchestra.run.smithi103.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-04-23T16:58:48.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:48 smithi148 ceph-mon[44520]: pgmap v2318: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:48.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:48 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/47734186' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-04-23T16:58:48.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:48 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T16:58:48.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:48 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T16:58:48.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:48 smithi188 ceph-mon[44101]: pgmap v2318: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:48.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:48 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/47734186' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-04-23T16:58:48.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:48 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T16:58:48.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:48 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T16:58:48.492 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:48 smithi103 ceph-mon[39629]: pgmap v2318: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:48.492 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:48 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/47734186' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-04-23T16:58:48.492 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:48 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T16:58:48.492 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:48 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T16:58:48.871 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 11 1 2024-04-23T16:58:49.153 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:58:49.409 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:49 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/47734186' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-04-23T16:58:49.409 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:49 smithi103 ceph-mon[39629]: osdmap e843: 12 total, 12 up, 12 in 2024-04-23T16:58:49.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:49 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/47734186' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-04-23T16:58:49.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:49 smithi188 ceph-mon[44101]: osdmap e843: 12 total, 12 up, 12 in 2024-04-23T16:58:49.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:49 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/47734186' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-04-23T16:58:49.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:49 smithi148 ceph-mon[44520]: osdmap e843: 12 total, 12 up, 12 in 2024-04-23T16:58:50.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:50 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:58:50.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:50 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:58:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:50 smithi103 ceph-mon[39629]: pgmap v2320: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:50 smithi103 ceph-mon[39629]: osdmap e844: 12 total, 12 up, 12 in 2024-04-23T16:58:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:50 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T16:58:50.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:50 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T16:58:50.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:50 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:58:50.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:50 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:58:50.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:50 smithi188 ceph-mon[44101]: pgmap v2320: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:50.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:50 smithi188 ceph-mon[44101]: osdmap e844: 12 total, 12 up, 12 in 2024-04-23T16:58:50.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:50 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T16:58:50.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:50 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T16:58:50.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:50 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:58:50.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:50 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:58:50.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:50 smithi148 ceph-mon[44520]: pgmap v2320: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:50.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:50 smithi148 ceph-mon[44520]: osdmap e844: 12 total, 12 up, 12 in 2024-04-23T16:58:50.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:50 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T16:58:50.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:50 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T16:58:51.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:51 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:58:51.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:51 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:58:51.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:51 smithi188 ceph-mon[44101]: osdmap e845: 12 total, 12 up, 12 in 2024-04-23T16:58:51.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:51 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:58:51.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:51 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:58:51.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:51 smithi103 ceph-mon[39629]: osdmap e845: 12 total, 12 up, 12 in 2024-04-23T16:58:51.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:51 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:58:51.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:51 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:58:51.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:51 smithi148 ceph-mon[44520]: osdmap e845: 12 total, 12 up, 12 in 2024-04-23T16:58:52.140 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.11 to 1 (10000) 2024-04-23T16:58:52.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:52 smithi188 ceph-mon[44101]: pgmap v2323: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:52.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:52 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3565003116' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:58:52.523 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:52 smithi103 ceph-mon[39629]: pgmap v2323: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:52.524 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:52 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3565003116' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:58:52.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:52 smithi148 ceph-mon[44520]: pgmap v2323: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:52.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:52 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3565003116' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:58:52.842 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 11 1 2024-04-23T16:58:53.112 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:58:53.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:53 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3565003116' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:58:53.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:53 smithi188 ceph-mon[44101]: osdmap e846: 12 total, 12 up, 12 in 2024-04-23T16:58:53.592 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:53 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3565003116' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:58:53.593 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:53 smithi103 ceph-mon[39629]: osdmap e846: 12 total, 12 up, 12 in 2024-04-23T16:58:53.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:53 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3565003116' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:58:53.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:53 smithi148 ceph-mon[44520]: osdmap e846: 12 total, 12 up, 12 in 2024-04-23T16:58:54.311 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:54 smithi103 ceph-mon[39629]: pgmap v2325: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:54.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:54 smithi188 ceph-mon[44101]: pgmap v2325: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:54.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:54 smithi148 ceph-mon[44520]: pgmap v2325: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:55 smithi188 ceph-mon[44101]: 1.7 scrub starts 2024-04-23T16:58:55.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:55 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1086285517' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:58:55.589 INFO:teuthology.orchestra.run.smithi103.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-04-23T16:58:55.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:55 smithi103 ceph-mon[39629]: 1.7 scrub starts 2024-04-23T16:58:55.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:55 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1086285517' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:58:55.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:55 smithi148 ceph-mon[44520]: 1.7 scrub starts 2024-04-23T16:58:55.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:55 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1086285517' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-04-23T16:58:56.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:56 smithi103 ceph-mon[39629]: 1.7 scrub ok 2024-04-23T16:58:56.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:56 smithi103 ceph-mon[39629]: pgmap v2326: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:56.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:56 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1086285517' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:58:56.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:56 smithi103 ceph-mon[39629]: osdmap e847: 12 total, 12 up, 12 in 2024-04-23T16:58:56.375 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-23T16:58:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:56 smithi188 ceph-mon[44101]: 1.7 scrub ok 2024-04-23T16:58:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:56 smithi188 ceph-mon[44101]: pgmap v2326: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:56 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1086285517' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:58:56.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:56 smithi188 ceph-mon[44101]: osdmap e847: 12 total, 12 up, 12 in 2024-04-23T16:58:56.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:56 smithi148 ceph-mon[44520]: 1.7 scrub ok 2024-04-23T16:58:56.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:56 smithi148 ceph-mon[44520]: pgmap v2326: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:56.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:56 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1086285517' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-04-23T16:58:56.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:56 smithi148 ceph-mon[44520]: osdmap e847: 12 total, 12 up, 12 in 2024-04-23T16:58:56.644 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:58:58.258 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:58 smithi103 ceph-mon[39629]: pgmap v2328: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:58.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:58 smithi188 ceph-mon[44101]: pgmap v2328: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:58.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:58 smithi148 ceph-mon[44520]: pgmap v2328: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:58:59.161 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.1 to 1 (10000) 2024-04-23T16:58:59.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:58:59 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3562000049' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:58:59.517 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:58:59 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3562000049' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:58:59.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:58:59 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3562000049' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:58:59.951 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 1 1 2024-04-23T16:59:00.238 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:00.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:00 smithi103 ceph-mon[39629]: pgmap v2329: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:00.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:00 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3562000049' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-23T16:59:00.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:00 smithi103 ceph-mon[39629]: osdmap e848: 12 total, 12 up, 12 in 2024-04-23T16:59:00.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:00 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:59:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:00 smithi188 ceph-mon[44101]: pgmap v2329: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:00 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3562000049' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-23T16:59:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:00 smithi188 ceph-mon[44101]: osdmap e848: 12 total, 12 up, 12 in 2024-04-23T16:59:00.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:00 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:59:00.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:00 smithi148 ceph-mon[44520]: pgmap v2329: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:00.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:00 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3562000049' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-23T16:59:00.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:00 smithi148 ceph-mon[44520]: osdmap e848: 12 total, 12 up, 12 in 2024-04-23T16:59:00.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:00 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:59:01.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:01 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:59:01.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:01 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:59:01.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:01 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:59:01.498 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:01 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:59:01.498 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:01 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:59:01.498 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:01 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:59:01.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:01 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:59:01.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:01 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:59:01.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:01 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:59:02.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:02 smithi188 ceph-mon[44101]: pgmap v2331: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:02.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:02 smithi103 ceph-mon[39629]: pgmap v2331: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:02.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:02 smithi148 ceph-mon[44520]: pgmap v2331: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:03.189 INFO:teuthology.orchestra.run.smithi103.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-23T16:59:03.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:03 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1519876431' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:59:03.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:03 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1519876431' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:59:03.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:03 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1519876431' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-23T16:59:04.002 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-23T16:59:04.271 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:04.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:04 smithi103 ceph-mon[39629]: pgmap v2332: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:04.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:04 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1519876431' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-23T16:59:04.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:04 smithi103 ceph-mon[39629]: osdmap e849: 12 total, 12 up, 12 in 2024-04-23T16:59:04.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:04 smithi188 ceph-mon[44101]: pgmap v2332: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:04.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:04 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1519876431' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-23T16:59:04.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:04 smithi188 ceph-mon[44101]: osdmap e849: 12 total, 12 up, 12 in 2024-04-23T16:59:04.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:04 smithi148 ceph-mon[44520]: pgmap v2332: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:04 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1519876431' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-23T16:59:04.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:04 smithi148 ceph-mon[44520]: osdmap e849: 12 total, 12 up, 12 in 2024-04-23T16:59:05.785 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:05 smithi103 ceph-mon[39629]: pgmap v2334: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:05.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:05 smithi148 ceph-mon[44520]: pgmap v2334: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:05.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:05 smithi188 ceph-mon[44101]: pgmap v2334: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:06.595 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.3 to 1 (10000) 2024-04-23T16:59:06.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:06 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4218082498' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:59:06.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:06 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4218082498' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:59:06.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:06 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4218082498' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:59:07.301 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 3 1 2024-04-23T16:59:07.572 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:07.847 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:07 smithi103 ceph-mon[39629]: 1.8 scrub starts 2024-04-23T16:59:07.847 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:07 smithi103 ceph-mon[39629]: 1.1 scrub starts 2024-04-23T16:59:07.848 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:07 smithi103 ceph-mon[39629]: 1.1 scrub ok 2024-04-23T16:59:07.848 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:07 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4218082498' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-23T16:59:07.848 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:07 smithi103 ceph-mon[39629]: osdmap e850: 12 total, 12 up, 12 in 2024-04-23T16:59:07.848 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:07 smithi103 ceph-mon[39629]: 1.8 scrub ok 2024-04-23T16:59:07.848 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:07 smithi103 ceph-mon[39629]: pgmap v2336: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:07.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:07 smithi148 ceph-mon[44520]: 1.8 scrub starts 2024-04-23T16:59:07.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:07 smithi148 ceph-mon[44520]: 1.1 scrub starts 2024-04-23T16:59:07.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:07 smithi148 ceph-mon[44520]: 1.1 scrub ok 2024-04-23T16:59:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:07 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4218082498' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-23T16:59:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:07 smithi148 ceph-mon[44520]: osdmap e850: 12 total, 12 up, 12 in 2024-04-23T16:59:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:07 smithi148 ceph-mon[44520]: 1.8 scrub ok 2024-04-23T16:59:07.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:07 smithi148 ceph-mon[44520]: pgmap v2336: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:07.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:07 smithi188 ceph-mon[44101]: 1.8 scrub starts 2024-04-23T16:59:07.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:07 smithi188 ceph-mon[44101]: 1.1 scrub starts 2024-04-23T16:59:07.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:07 smithi188 ceph-mon[44101]: 1.1 scrub ok 2024-04-23T16:59:07.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:07 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4218082498' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-23T16:59:07.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:07 smithi188 ceph-mon[44101]: osdmap e850: 12 total, 12 up, 12 in 2024-04-23T16:59:07.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:07 smithi188 ceph-mon[44101]: 1.8 scrub ok 2024-04-23T16:59:07.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:07 smithi188 ceph-mon[44101]: pgmap v2336: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:08.792 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:08 smithi103 ceph-mon[39629]: 1.5 scrub starts 2024-04-23T16:59:08.792 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:08 smithi103 ceph-mon[39629]: 1.5 scrub ok 2024-04-23T16:59:08.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:08 smithi148 ceph-mon[44520]: 1.5 scrub starts 2024-04-23T16:59:08.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:08 smithi148 ceph-mon[44520]: 1.5 scrub ok 2024-04-23T16:59:08.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:08 smithi188 ceph-mon[44101]: 1.5 scrub starts 2024-04-23T16:59:08.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:08 smithi188 ceph-mon[44101]: 1.5 scrub ok 2024-04-23T16:59:09.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:09 smithi148 ceph-mon[44520]: pgmap v2337: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:09.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:09 smithi188 ceph-mon[44101]: pgmap v2337: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:10.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:09 smithi103 ceph-mon[39629]: pgmap v2337: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:10.621 INFO:teuthology.orchestra.run.smithi103.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-23T16:59:10.969 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:10 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:59:10.969 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:10 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:59:10.969 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:10 smithi103 ceph-mon[39629]: osdmap e851: 12 total, 12 up, 12 in 2024-04-23T16:59:10.969 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:10 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1187184903' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:59:10.969 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:10 smithi103 ceph-mon[39629]: 1.0 scrub starts 2024-04-23T16:59:10.969 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:10 smithi103 ceph-mon[39629]: 1.0 scrub ok 2024-04-23T16:59:10.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:10 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:59:10.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:10 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:59:10.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:10 smithi188 ceph-mon[44101]: osdmap e851: 12 total, 12 up, 12 in 2024-04-23T16:59:10.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:10 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1187184903' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:59:10.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:10 smithi188 ceph-mon[44101]: 1.0 scrub starts 2024-04-23T16:59:10.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:10 smithi188 ceph-mon[44101]: 1.0 scrub ok 2024-04-23T16:59:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:10 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:59:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:10 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:59:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:10 smithi148 ceph-mon[44520]: osdmap e851: 12 total, 12 up, 12 in 2024-04-23T16:59:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:10 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1187184903' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-23T16:59:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:10 smithi148 ceph-mon[44520]: 1.0 scrub starts 2024-04-23T16:59:11.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:10 smithi148 ceph-mon[44520]: 1.0 scrub ok 2024-04-23T16:59:11.358 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-23T16:59:11.631 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:11.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:11 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1187184903' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-23T16:59:11.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:11 smithi188 ceph-mon[44101]: osdmap e852: 12 total, 12 up, 12 in 2024-04-23T16:59:11.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:11 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:59:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:11 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:59:11.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:11 smithi188 ceph-mon[44101]: pgmap v2340: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:12.030 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:11 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1187184903' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-23T16:59:12.031 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:11 smithi103 ceph-mon[39629]: osdmap e852: 12 total, 12 up, 12 in 2024-04-23T16:59:12.031 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:11 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:59:12.031 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:11 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:59:12.031 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:11 smithi103 ceph-mon[39629]: pgmap v2340: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:12.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:11 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1187184903' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-23T16:59:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:11 smithi148 ceph-mon[44520]: osdmap e852: 12 total, 12 up, 12 in 2024-04-23T16:59:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:11 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:59:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:11 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:59:12.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:11 smithi148 ceph-mon[44520]: pgmap v2340: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:12.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:12 smithi188 ceph-mon[44101]: osdmap e853: 12 total, 12 up, 12 in 2024-04-23T16:59:13.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:12 smithi103 ceph-mon[39629]: osdmap e853: 12 total, 12 up, 12 in 2024-04-23T16:59:13.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:12 smithi148 ceph-mon[44520]: osdmap e853: 12 total, 12 up, 12 in 2024-04-23T16:59:13.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:13 smithi188 ceph-mon[44101]: pgmap v2342: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:14.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:13 smithi103 ceph-mon[39629]: pgmap v2342: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:14.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:13 smithi148 ceph-mon[44520]: pgmap v2342: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:14.646 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.4 to 1 (10000) 2024-04-23T16:59:14.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:14 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2232495384' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-23T16:59:14.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:14 smithi188 ceph-mon[44101]: 1.9 scrub starts 2024-04-23T16:59:14.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:14 smithi188 ceph-mon[44101]: 1.9 scrub ok 2024-04-23T16:59:15.025 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:14 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2232495384' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-23T16:59:15.025 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:14 smithi103 ceph-mon[39629]: 1.9 scrub starts 2024-04-23T16:59:15.026 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:14 smithi103 ceph-mon[39629]: 1.9 scrub ok 2024-04-23T16:59:15.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:14 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2232495384' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-23T16:59:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:14 smithi148 ceph-mon[44520]: 1.9 scrub starts 2024-04-23T16:59:15.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:14 smithi148 ceph-mon[44520]: 1.9 scrub ok 2024-04-23T16:59:15.455 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 4 1 2024-04-23T16:59:15.727 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:15.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:15 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2232495384' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-23T16:59:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:15 smithi103 ceph-mon[39629]: osdmap e854: 12 total, 12 up, 12 in 2024-04-23T16:59:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:15 smithi103 ceph-mon[39629]: pgmap v2344: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:15 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:59:15.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:15 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:59:15.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:15 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2232495384' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-23T16:59:15.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:15 smithi188 ceph-mon[44101]: osdmap e854: 12 total, 12 up, 12 in 2024-04-23T16:59:15.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:15 smithi188 ceph-mon[44101]: pgmap v2344: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:15.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:15 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:59:15.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:15 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:59:16.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:15 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2232495384' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-23T16:59:16.126 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:15 smithi148 ceph-mon[44520]: osdmap e854: 12 total, 12 up, 12 in 2024-04-23T16:59:16.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:15 smithi148 ceph-mon[44520]: pgmap v2344: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:16.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:15 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:59:16.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:15 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:59:16.803 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:16 smithi103 ceph-mon[39629]: osdmap e855: 12 total, 12 up, 12 in 2024-04-23T16:59:16.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:16 smithi188 ceph-mon[44101]: osdmap e855: 12 total, 12 up, 12 in 2024-04-23T16:59:17.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:16 smithi148 ceph-mon[44520]: osdmap e855: 12 total, 12 up, 12 in 2024-04-23T16:59:17.660 INFO:teuthology.orchestra.run.smithi103.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-23T16:59:17.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:17 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:59:17.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:17 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:59:17.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:17 smithi188 ceph-mon[44101]: osdmap e856: 12 total, 12 up, 12 in 2024-04-23T16:59:17.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:17 smithi188 ceph-mon[44101]: pgmap v2347: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:17.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:17 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2224258099' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-23T16:59:17.995 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:17 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:59:17.995 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:17 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:59:17.995 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:17 smithi103 ceph-mon[39629]: osdmap e856: 12 total, 12 up, 12 in 2024-04-23T16:59:17.996 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:17 smithi103 ceph-mon[39629]: pgmap v2347: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:17.996 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:17 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2224258099' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-23T16:59:18.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:17 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:59:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:17 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:59:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:17 smithi148 ceph-mon[44520]: osdmap e856: 12 total, 12 up, 12 in 2024-04-23T16:59:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:17 smithi148 ceph-mon[44520]: pgmap v2347: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:18.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:17 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2224258099' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-23T16:59:18.353 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-23T16:59:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:18.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:18 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2224258099' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-23T16:59:18.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:18 smithi188 ceph-mon[44101]: osdmap e857: 12 total, 12 up, 12 in 2024-04-23T16:59:19.097 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:18 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2224258099' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-23T16:59:19.097 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:18 smithi103 ceph-mon[39629]: osdmap e857: 12 total, 12 up, 12 in 2024-04-23T16:59:19.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:18 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2224258099' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-23T16:59:19.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:18 smithi148 ceph-mon[44520]: osdmap e857: 12 total, 12 up, 12 in 2024-04-23T16:59:19.783 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:19 smithi103 ceph-mon[39629]: pgmap v2349: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:19.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:19 smithi188 ceph-mon[44101]: pgmap v2349: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:20.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:19 smithi148 ceph-mon[44520]: pgmap v2349: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:20.677 INFO:teuthology.orchestra.run.smithi103.stderr:reweighted osd.2 to 1 (10000) 2024-04-23T16:59:20.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:20 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3866092009' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:59:21.017 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:20 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3866092009' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:59:21.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:20 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3866092009' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:59:21.452 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 2 1 2024-04-23T16:59:21.679 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:21 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3866092009' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-23T16:59:21.679 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:21 smithi103 ceph-mon[39629]: osdmap e858: 12 total, 12 up, 12 in 2024-04-23T16:59:21.679 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:21 smithi103 ceph-mon[39629]: pgmap v2351: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:21.717 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:21.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:21 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3866092009' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-23T16:59:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:21 smithi188 ceph-mon[44101]: osdmap e858: 12 total, 12 up, 12 in 2024-04-23T16:59:21.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:21 smithi188 ceph-mon[44101]: pgmap v2351: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:22.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:21 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3866092009' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-23T16:59:22.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:21 smithi148 ceph-mon[44520]: osdmap e858: 12 total, 12 up, 12 in 2024-04-23T16:59:22.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:21 smithi148 ceph-mon[44520]: pgmap v2351: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:24.114 INFO:teuthology.orchestra.run.smithi103.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-23T16:59:24.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:24 smithi148 ceph-mon[44520]: pgmap v2352: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:24.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:24 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1089551863' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:59:24.466 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:24 smithi103 ceph-mon[39629]: pgmap v2352: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:24.466 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:24 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1089551863' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:59:24.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:24 smithi188 ceph-mon[44101]: pgmap v2352: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:24.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:24 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1089551863' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-23T16:59:24.793 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-23T16:59:24.794 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-23T16:59:24.794 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-23T16:59:24.794 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-23T16:59:24.794 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-23T16:59:24.794 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T16:59:25.063 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:25.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:25 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1089551863' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-23T16:59:25.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:25 smithi148 ceph-mon[44520]: osdmap e859: 12 total, 12 up, 12 in 2024-04-23T16:59:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:25 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1089551863' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-23T16:59:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:25 smithi188 ceph-mon[44101]: osdmap e859: 12 total, 12 up, 12 in 2024-04-23T16:59:25.567 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:25 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1089551863' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-23T16:59:25.567 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:25 smithi103 ceph-mon[39629]: osdmap e859: 12 total, 12 up, 12 in 2024-04-23T16:59:26.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:26 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:59:26.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:26 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:59:26.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:26 smithi103 ceph-mon[39629]: pgmap v2354: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:26.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:26 smithi103 ceph-mon[39629]: osdmap e860: 12 total, 12 up, 12 in 2024-04-23T16:59:26.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:26 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:59:26.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:26 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:59:26.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:26 smithi148 ceph-mon[44520]: pgmap v2354: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:26.374 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:26 smithi148 ceph-mon[44520]: osdmap e860: 12 total, 12 up, 12 in 2024-04-23T16:59:26.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:26 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:59:26.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:26 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:59:26.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:26 smithi188 ceph-mon[44101]: pgmap v2354: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:26.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:26 smithi188 ceph-mon[44101]: osdmap e860: 12 total, 12 up, 12 in 2024-04-23T16:59:27.086 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:59:27.087 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":860,"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","created":"2024-04-23T16:08:13.477590+0000","modified":"2024-04-23T16:59:25.590372+0000","last_up_change":"2024-04-23T16:57:21.581583+0000","last_in_change":"2024-04-23T16:57:51.567223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":34,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T16:11:22.416777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"822","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"301","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.2899999618530273,"score_stable":3.2899999618530273,"optimal_score":0.82999998331069946,"raw_score_acting":2.7300000190734863,"raw_score_stable":2.7300000190734863,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4c084047-8f01-4b86-a60c-5592735b5862","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":815,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6801","nonce":4274040097}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6802","nonce":4274040097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6804","nonce":4274040097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6803","nonce":4274040097}]},"public_addr":"172.21.15.103:6801/4274040097","cluster_addr":"172.21.15.103:6802/4274040097","heartbeat_back_addr":"172.21.15.103:6804/4274040097","heartbeat_front_addr":"172.21.15.103:6803/4274040097","state":["exists","up"]},{"osd":1,"uuid":"3438f2bd-5fb7-432c-b383-29e18c442f20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":232,"up_from":271,"up_thru":703,"down_at":233,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6805","nonce":859282187}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6806","nonce":859282187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6808","nonce":859282187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6807","nonce":859282187}]},"public_addr":"172.21.15.103:6805/859282187","cluster_addr":"172.21.15.103:6806/859282187","heartbeat_back_addr":"172.21.15.103:6808/859282187","heartbeat_front_addr":"172.21.15.103:6807/859282187","state":["exists","up"]},{"osd":2,"uuid":"c17fffb4-8d17-4546-ba24-e5951a2a37fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":481,"last_clean_end":780,"up_from":804,"up_thru":819,"down_at":782,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6809","nonce":1571438738}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6810","nonce":1571438738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6812","nonce":1571438738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6811","nonce":1571438738}]},"public_addr":"172.21.15.103:6809/1571438738","cluster_addr":"172.21.15.103:6810/1571438738","heartbeat_back_addr":"172.21.15.103:6812/1571438738","heartbeat_front_addr":"172.21.15.103:6811/1571438738","state":["exists","up"]},{"osd":3,"uuid":"cee91a3d-55dc-4e89-b568-86821632bb1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":559,"up_from":619,"up_thru":540,"down_at":560,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6813","nonce":995549078}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6814","nonce":995549078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6816","nonce":995549078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6815","nonce":995549078}]},"public_addr":"172.21.15.103:6813/995549078","cluster_addr":"172.21.15.103:6814/995549078","heartbeat_back_addr":"172.21.15.103:6816/995549078","heartbeat_front_addr":"172.21.15.103:6815/995549078","state":["exists","up"]},{"osd":4,"uuid":"61d5385b-2c82-42f8-93c3-8f1baf71c65e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":33,"last_clean_end":686,"up_from":728,"up_thru":609,"down_at":687,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6800","nonce":4230909898}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6801","nonce":4230909898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6803","nonce":4230909898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6802","nonce":4230909898}]},"public_addr":"172.21.15.148:6800/4230909898","cluster_addr":"172.21.15.148:6801/4230909898","heartbeat_back_addr":"172.21.15.148:6803/4230909898","heartbeat_front_addr":"172.21.15.148:6802/4230909898","state":["exists","up"]},{"osd":5,"uuid":"5d492189-c3cd-47ad-a12a-ed4d275bcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":820,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6804","nonce":3547200711}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6805","nonce":3547200711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6807","nonce":3547200711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6806","nonce":3547200711}]},"public_addr":"172.21.15.148:6804/3547200711","cluster_addr":"172.21.15.148:6805/3547200711","heartbeat_back_addr":"172.21.15.148:6807/3547200711","heartbeat_front_addr":"172.21.15.148:6806/3547200711","state":["exists","up"]},{"osd":6,"uuid":"89f2379e-cf14-4e69-8747-7535db81b70b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":813,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6808","nonce":366919758}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6809","nonce":366919758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6811","nonce":366919758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6810","nonce":366919758}]},"public_addr":"172.21.15.148:6808/366919758","cluster_addr":"172.21.15.148:6809/366919758","heartbeat_back_addr":"172.21.15.148:6811/366919758","heartbeat_front_addr":"172.21.15.148:6810/366919758","state":["exists","up"]},{"osd":7,"uuid":"f27b035a-558f-4528-80ea-2a00d03f6539","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":834,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6812","nonce":3719550888}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6813","nonce":3719550888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6815","nonce":3719550888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6814","nonce":3719550888}]},"public_addr":"172.21.15.148:6812/3719550888","cluster_addr":"172.21.15.148:6813/3719550888","heartbeat_back_addr":"172.21.15.148:6815/3719550888","heartbeat_front_addr":"172.21.15.148:6814/3719550888","state":["exists","up"]},{"osd":8,"uuid":"a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":815,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6800","nonce":1151408595}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6801","nonce":1151408595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6803","nonce":1151408595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6802","nonce":1151408595}]},"public_addr":"172.21.15.188:6800/1151408595","cluster_addr":"172.21.15.188:6801/1151408595","heartbeat_back_addr":"172.21.15.188:6803/1151408595","heartbeat_front_addr":"172.21.15.188:6802/1151408595","state":["exists","up"]},{"osd":9,"uuid":"124d9b05-11eb-4622-b6a2-c4d5bd60b7e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":703,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6804","nonce":306501862}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6805","nonce":306501862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6807","nonce":306501862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6806","nonce":306501862}]},"public_addr":"172.21.15.188:6804/306501862","cluster_addr":"172.21.15.188:6805/306501862","heartbeat_back_addr":"172.21.15.188:6807/306501862","heartbeat_front_addr":"172.21.15.188:6806/306501862","state":["exists","up"]},{"osd":10,"uuid":"3dfb9070-e534-4ea4-a25e-ee1c51297447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":816,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6808","nonce":1231263138}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6809","nonce":1231263138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6811","nonce":1231263138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6810","nonce":1231263138}]},"public_addr":"172.21.15.188:6808/1231263138","cluster_addr":"172.21.15.188:6809/1231263138","heartbeat_back_addr":"172.21.15.188:6811/1231263138","heartbeat_front_addr":"172.21.15.188:6810/1231263138","state":["exists","up"]},{"osd":11,"uuid":"bc163925-f623-4f63-ab89-35e2d511a730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":703,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6812","nonce":457992661}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6813","nonce":457992661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6815","nonce":457992661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6814","nonce":457992661}]},"public_addr":"172.21.15.188:6812/457992661","cluster_addr":"172.21.15.188:6813/457992661","heartbeat_back_addr":"172.21.15.188:6815/457992661","heartbeat_front_addr":"172.21.15.188:6814/457992661","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:58:11.899076+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:59:09.191507+0000","dead_epoch":232},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:58:49.586345+0000","dead_epoch":780},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:58:48.731295+0000","dead_epoch":559},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:58:24.771654+0000","dead_epoch":686},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:59:15.086532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:58:28.684085+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:58:44.341862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:59:16.428277+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:59:11.051463+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:59:24.641219+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:58:00.280373+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6800/3535470771":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1188058303":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1510553721":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/281182801":"2024-04-24T16:08:44.450632+0000","172.21.15.103:6800/2049124518":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/466241963":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/2726838698":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/596230573":"2024-04-24T16:09:14.324448+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T16:59:27.471 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:27 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2306699489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:59:27.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:27 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2306699489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:59:27.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:27 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2306699489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:59:27.905 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-23T16:59:27.905 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T16:59:28.170 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:28 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:59:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:28 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:59:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:28 smithi103 ceph-mon[39629]: pgmap v2356: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:28 smithi103 ceph-mon[39629]: osdmap e861: 12 total, 12 up, 12 in 2024-04-23T16:59:28.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:28 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:59:28.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:28 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:59:28.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:28 smithi188 ceph-mon[44101]: pgmap v2356: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:28.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:28 smithi188 ceph-mon[44101]: osdmap e861: 12 total, 12 up, 12 in 2024-04-23T16:59:28.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:28 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:59:28.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:28 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:59:28.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:28 smithi148 ceph-mon[44520]: pgmap v2356: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:28.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:28 smithi148 ceph-mon[44520]: osdmap e861: 12 total, 12 up, 12 in 2024-04-23T16:59:30.029 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T16:59:30.030 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":861,"fsid":"8e3594f2-018b-11ef-bc93-c7b262605968","created":"2024-04-23T16:08:13.477590+0000","modified":"2024-04-23T16:59:27.119785+0000","last_up_change":"2024-04-23T16:57:21.581583+0000","last_in_change":"2024-04-23T16:57:51.567223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":34,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T16:11:22.416777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"822","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"301","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.2899999618530273,"score_stable":3.2899999618530273,"optimal_score":0.82999998331069946,"raw_score_acting":2.7300000190734863,"raw_score_stable":2.7300000190734863,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4c084047-8f01-4b86-a60c-5592735b5862","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":815,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6801","nonce":4274040097}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6802","nonce":4274040097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6804","nonce":4274040097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6803","nonce":4274040097}]},"public_addr":"172.21.15.103:6801/4274040097","cluster_addr":"172.21.15.103:6802/4274040097","heartbeat_back_addr":"172.21.15.103:6804/4274040097","heartbeat_front_addr":"172.21.15.103:6803/4274040097","state":["exists","up"]},{"osd":1,"uuid":"3438f2bd-5fb7-432c-b383-29e18c442f20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":232,"up_from":271,"up_thru":703,"down_at":233,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6805","nonce":859282187}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6806","nonce":859282187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6808","nonce":859282187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6807","nonce":859282187}]},"public_addr":"172.21.15.103:6805/859282187","cluster_addr":"172.21.15.103:6806/859282187","heartbeat_back_addr":"172.21.15.103:6808/859282187","heartbeat_front_addr":"172.21.15.103:6807/859282187","state":["exists","up"]},{"osd":2,"uuid":"c17fffb4-8d17-4546-ba24-e5951a2a37fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":481,"last_clean_end":780,"up_from":804,"up_thru":819,"down_at":782,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6809","nonce":1571438738}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6810","nonce":1571438738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6812","nonce":1571438738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6811","nonce":1571438738}]},"public_addr":"172.21.15.103:6809/1571438738","cluster_addr":"172.21.15.103:6810/1571438738","heartbeat_back_addr":"172.21.15.103:6812/1571438738","heartbeat_front_addr":"172.21.15.103:6811/1571438738","state":["exists","up"]},{"osd":3,"uuid":"cee91a3d-55dc-4e89-b568-86821632bb1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":559,"up_from":619,"up_thru":540,"down_at":560,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6813","nonce":995549078}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6814","nonce":995549078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6816","nonce":995549078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.103:6815","nonce":995549078}]},"public_addr":"172.21.15.103:6813/995549078","cluster_addr":"172.21.15.103:6814/995549078","heartbeat_back_addr":"172.21.15.103:6816/995549078","heartbeat_front_addr":"172.21.15.103:6815/995549078","state":["exists","up"]},{"osd":4,"uuid":"61d5385b-2c82-42f8-93c3-8f1baf71c65e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":33,"last_clean_end":686,"up_from":728,"up_thru":609,"down_at":687,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6800","nonce":4230909898}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6801","nonce":4230909898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6803","nonce":4230909898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6802","nonce":4230909898}]},"public_addr":"172.21.15.148:6800/4230909898","cluster_addr":"172.21.15.148:6801/4230909898","heartbeat_back_addr":"172.21.15.148:6803/4230909898","heartbeat_front_addr":"172.21.15.148:6802/4230909898","state":["exists","up"]},{"osd":5,"uuid":"5d492189-c3cd-47ad-a12a-ed4d275bcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":820,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6804","nonce":3547200711}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6805","nonce":3547200711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6807","nonce":3547200711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6806","nonce":3547200711}]},"public_addr":"172.21.15.148:6804/3547200711","cluster_addr":"172.21.15.148:6805/3547200711","heartbeat_back_addr":"172.21.15.148:6807/3547200711","heartbeat_front_addr":"172.21.15.148:6806/3547200711","state":["exists","up"]},{"osd":6,"uuid":"89f2379e-cf14-4e69-8747-7535db81b70b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":813,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6808","nonce":366919758}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6809","nonce":366919758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6811","nonce":366919758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6810","nonce":366919758}]},"public_addr":"172.21.15.148:6808/366919758","cluster_addr":"172.21.15.148:6809/366919758","heartbeat_back_addr":"172.21.15.148:6811/366919758","heartbeat_front_addr":"172.21.15.148:6810/366919758","state":["exists","up"]},{"osd":7,"uuid":"f27b035a-558f-4528-80ea-2a00d03f6539","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":834,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6812","nonce":3719550888}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6813","nonce":3719550888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6815","nonce":3719550888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6814","nonce":3719550888}]},"public_addr":"172.21.15.148:6812/3719550888","cluster_addr":"172.21.15.148:6813/3719550888","heartbeat_back_addr":"172.21.15.148:6815/3719550888","heartbeat_front_addr":"172.21.15.148:6814/3719550888","state":["exists","up"]},{"osd":8,"uuid":"a9ac86bd-5e83-4ed8-9304-7e0ad4ac48ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":815,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6800","nonce":1151408595}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6801","nonce":1151408595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6803","nonce":1151408595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6802","nonce":1151408595}]},"public_addr":"172.21.15.188:6800/1151408595","cluster_addr":"172.21.15.188:6801/1151408595","heartbeat_back_addr":"172.21.15.188:6803/1151408595","heartbeat_front_addr":"172.21.15.188:6802/1151408595","state":["exists","up"]},{"osd":9,"uuid":"124d9b05-11eb-4622-b6a2-c4d5bd60b7e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":703,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6804","nonce":306501862}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6805","nonce":306501862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6807","nonce":306501862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6806","nonce":306501862}]},"public_addr":"172.21.15.188:6804/306501862","cluster_addr":"172.21.15.188:6805/306501862","heartbeat_back_addr":"172.21.15.188:6807/306501862","heartbeat_front_addr":"172.21.15.188:6806/306501862","state":["exists","up"]},{"osd":10,"uuid":"3dfb9070-e534-4ea4-a25e-ee1c51297447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":816,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6808","nonce":1231263138}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6809","nonce":1231263138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6811","nonce":1231263138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6810","nonce":1231263138}]},"public_addr":"172.21.15.188:6808/1231263138","cluster_addr":"172.21.15.188:6809/1231263138","heartbeat_back_addr":"172.21.15.188:6811/1231263138","heartbeat_front_addr":"172.21.15.188:6810/1231263138","state":["exists","up"]},{"osd":11,"uuid":"bc163925-f623-4f63-ab89-35e2d511a730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":703,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6812","nonce":457992661}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6813","nonce":457992661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6815","nonce":457992661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.188:6814","nonce":457992661}]},"public_addr":"172.21.15.188:6812/457992661","cluster_addr":"172.21.15.188:6813/457992661","heartbeat_back_addr":"172.21.15.188:6815/457992661","heartbeat_front_addr":"172.21.15.188:6814/457992661","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:58:11.899076+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:59:09.191507+0000","dead_epoch":232},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:58:49.586345+0000","dead_epoch":780},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:58:48.731295+0000","dead_epoch":559},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:58:24.771654+0000","dead_epoch":686},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:59:15.086532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:58:28.684085+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:58:44.341862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:59:16.428277+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:59:11.051463+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:59:24.641219+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T16:59:26.937666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6800/3535470771":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1188058303":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/1510553721":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/281182801":"2024-04-24T16:08:44.450632+0000","172.21.15.103:6800/2049124518":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/466241963":"2024-04-24T16:08:44.450632+0000","172.21.15.103:0/2726838698":"2024-04-24T16:09:14.324448+0000","172.21.15.103:0/596230573":"2024-04-24T16:09:14.324448+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T16:59:30.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:30 smithi103 ceph-mon[39629]: pgmap v2358: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:30.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:30 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3837606410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:59:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:30 smithi188 ceph-mon[44101]: pgmap v2358: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:30 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3837606410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:59:30.633 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:30 smithi148 ceph-mon[44520]: pgmap v2358: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:30.633 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:30 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3837606410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T16:59:30.862 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-23T16:59:30.863 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-23T16:59:30.863 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-23T16:59:30.864 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-23T16:59:30.864 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-23T16:59:30.864 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-23T16:59:30.865 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-23T16:59:30.865 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-23T16:59:30.866 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-23T16:59:30.866 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-23T16:59:30.867 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-23T16:59:30.867 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-23T16:59:31.549 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:31.581 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:31.635 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:31.988 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:32.003 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:32.003 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:32.236 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:32 smithi103 ceph-mon[39629]: pgmap v2359: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:32.237 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:32.238 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:32.239 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:32.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:32 smithi188 ceph-mon[44101]: pgmap v2359: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:32.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:32 smithi148 ceph-mon[44520]: pgmap v2359: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:34.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:34 smithi188 ceph-mon[44101]: pgmap v2360: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:34.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:34 smithi103 ceph-mon[39629]: pgmap v2360: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:34.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:34 smithi148 ceph-mon[44520]: pgmap v2360: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:35.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:35 smithi103 ceph-mon[39629]: pgmap v2361: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:35.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:35 smithi148 ceph-mon[44520]: pgmap v2361: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:35.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:35 smithi188 ceph-mon[44101]: pgmap v2361: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:36.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:36 smithi103 ceph-mon[39629]: 1.4 scrub starts 2024-04-23T16:59:36.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:36 smithi148 ceph-mon[44520]: 1.4 scrub starts 2024-04-23T16:59:36.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:36 smithi188 ceph-mon[44101]: 1.4 scrub starts 2024-04-23T16:59:37.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:37 smithi148 ceph-mon[44520]: 1.4 scrub ok 2024-04-23T16:59:37.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:37 smithi148 ceph-mon[44520]: pgmap v2362: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:37.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:37 smithi188 ceph-mon[44101]: 1.4 scrub ok 2024-04-23T16:59:37.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:37 smithi188 ceph-mon[44101]: pgmap v2362: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:38.069 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:37 smithi103 ceph-mon[39629]: 1.4 scrub ok 2024-04-23T16:59:38.069 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:37 smithi103 ceph-mon[39629]: pgmap v2362: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:38.799 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:38 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:59:38.799 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:38 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:59:38.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:38 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:59:38.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:38 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:59:38.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:38 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:59:38.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:38 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:59:39.394 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:39.395 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:39.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:39 smithi148 ceph-mon[44520]: osdmap e862: 12 total, 12 up, 12 in 2024-04-23T16:59:39.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:39 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:59:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:39 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:59:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:39 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:59:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:39 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:59:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:39 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:59:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:39 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:59:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:39 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:59:39.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:39 smithi148 ceph-mon[44520]: pgmap v2364: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:39.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:39 smithi188 ceph-mon[44101]: osdmap e862: 12 total, 12 up, 12 in 2024-04-23T16:59:39.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:39 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:59:39.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:39 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:59:39.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:39 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:59:39.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:39 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:59:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:39 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:59:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:39 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:59:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:39 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:59:39.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:39 smithi188 ceph-mon[44101]: pgmap v2364: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:40.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:39 smithi103 ceph-mon[39629]: osdmap e862: 12 total, 12 up, 12 in 2024-04-23T16:59:40.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:39 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T16:59:40.099 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:39 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T16:59:40.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:39 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T16:59:40.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:39 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T16:59:40.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:39 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T16:59:40.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:39 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T16:59:40.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:39 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' 2024-04-23T16:59:40.100 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:39 smithi103 ceph-mon[39629]: pgmap v2364: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:40.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:40 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:59:40.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:40 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:59:40.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:40 smithi188 ceph-mon[44101]: osdmap e863: 12 total, 12 up, 12 in 2024-04-23T16:59:40.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:40 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:59:40.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:40 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:59:40.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:40 smithi103 ceph-mon[39629]: osdmap e863: 12 total, 12 up, 12 in 2024-04-23T16:59:41.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:40 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:59:41.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:40 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:59:41.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:40 smithi148 ceph-mon[44520]: osdmap e863: 12 total, 12 up, 12 in 2024-04-23T16:59:41.458 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:41.800 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:41 smithi103 ceph-mon[39629]: pgmap v2366: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:41.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:41 smithi188 ceph-mon[44101]: pgmap v2366: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:42.123 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:41 smithi148 ceph-mon[44520]: pgmap v2366: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:43.602 INFO:teuthology.orchestra.run.smithi103.stdout:201863463471 2024-04-23T16:59:43.602 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-23T16:59:44.173 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:44 smithi103 ceph-mon[39629]: pgmap v2367: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:44.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:44 smithi148 ceph-mon[44520]: pgmap v2367: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:44.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:44 smithi188 ceph-mon[44101]: pgmap v2367: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:45.027 INFO:teuthology.orchestra.run.smithi103.stdout:313532613153 2024-04-23T16:59:45.028 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-23T16:59:45.782 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:45 smithi103 ceph-mon[39629]: pgmap v2368: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:45.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:45 smithi148 ceph-mon[44520]: pgmap v2368: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:45.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:45 smithi188 ceph-mon[44101]: pgmap v2368: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:47.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:47 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:59:47.656 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:47 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:59:47.656 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:47 smithi103 ceph-mon[39629]: osdmap e864: 12 total, 12 up, 12 in 2024-04-23T16:59:47.656 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:47 smithi103 ceph-mon[39629]: pgmap v2370: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:47.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:47 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:59:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:47 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:59:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:47 smithi148 ceph-mon[44520]: osdmap e864: 12 total, 12 up, 12 in 2024-04-23T16:59:47.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:47 smithi148 ceph-mon[44520]: pgmap v2370: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:47 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:59:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:47 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:59:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:47 smithi188 ceph-mon[44101]: osdmap e864: 12 total, 12 up, 12 in 2024-04-23T16:59:47.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:47 smithi188 ceph-mon[44101]: pgmap v2370: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:48.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:48 smithi103 ceph-mon[39629]: 1.2 deep-scrub starts 2024-04-23T16:59:48.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:48 smithi103 ceph-mon[39629]: 1.a deep-scrub starts 2024-04-23T16:59:48.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:48 smithi103 ceph-mon[39629]: 1.a deep-scrub ok 2024-04-23T16:59:48.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:48 smithi148 ceph-mon[44520]: 1.2 deep-scrub starts 2024-04-23T16:59:48.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:48 smithi148 ceph-mon[44520]: 1.a deep-scrub starts 2024-04-23T16:59:48.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:48 smithi148 ceph-mon[44520]: 1.a deep-scrub ok 2024-04-23T16:59:48.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:48 smithi188 ceph-mon[44101]: 1.2 deep-scrub starts 2024-04-23T16:59:48.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:48 smithi188 ceph-mon[44101]: 1.a deep-scrub starts 2024-04-23T16:59:48.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:48 smithi188 ceph-mon[44101]: 1.a deep-scrub ok 2024-04-23T16:59:49.695 INFO:teuthology.orchestra.run.smithi103.stdout:3453153706015 2024-04-23T16:59:49.696 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-23T16:59:49.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:49 smithi103 ceph-mon[39629]: 1.2 deep-scrub ok 2024-04-23T16:59:49.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:49 smithi103 ceph-mon[39629]: pgmap v2371: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:49.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:49 smithi148 ceph-mon[44520]: 1.2 deep-scrub ok 2024-04-23T16:59:49.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:49 smithi148 ceph-mon[44520]: pgmap v2371: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:49.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:49 smithi188 ceph-mon[44101]: 1.2 deep-scrub ok 2024-04-23T16:59:49.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:49 smithi188 ceph-mon[44101]: pgmap v2371: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:51.005 INFO:teuthology.orchestra.run.smithi103.stdout:240518169134 2024-04-23T16:59:51.006 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-23T16:59:51.400 INFO:teuthology.orchestra.run.smithi103.stdout:347892351518 2024-04-23T16:59:51.401 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-23T16:59:52.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:52 smithi103 ceph-mon[39629]: pgmap v2372: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:52.423 INFO:teuthology.orchestra.run.smithi103.stdout:163208757812 2024-04-23T16:59:52.424 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-23T16:59:52.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:52 smithi188 ceph-mon[44101]: pgmap v2372: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:52.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:52 smithi148 ceph-mon[44520]: pgmap v2372: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:53.800 INFO:teuthology.orchestra.run.smithi103.stdout:283467842086 2024-04-23T16:59:53.801 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-23T16:59:54.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:54 smithi103 ceph-mon[39629]: pgmap v2373: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:54.419 INFO:teuthology.orchestra.run.smithi103.stdout:1163936137615 2024-04-23T16:59:54.419 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-23T16:59:54.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:54 smithi188 ceph-mon[44101]: pgmap v2373: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:54.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:54 smithi148 ceph-mon[44520]: pgmap v2373: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:54.956 INFO:teuthology.orchestra.run.smithi103.stdout:34359738959 2024-04-23T16:59:54.956 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-23T16:59:55.493 INFO:teuthology.orchestra.run.smithi103.stdout:3126736191578 2024-04-23T16:59:55.494 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-23T16:59:55.787 INFO:teuthology.orchestra.run.smithi103.stdout:2658584756381 2024-04-23T16:59:55.788 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-23T16:59:56.416 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:56 smithi103 ceph-mon[39629]: pgmap v2374: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:56.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:56 smithi188 ceph-mon[44101]: pgmap v2374: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:56.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:56 smithi148 ceph-mon[44520]: pgmap v2374: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:57.021 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:57.033 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:57.038 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:57.267 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:57.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:57 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T16:59:57.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:57 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T16:59:57.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:57 smithi103 ceph-mon[39629]: osdmap e865: 12 total, 12 up, 12 in 2024-04-23T16:59:57.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:57 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T16:59:57.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:57 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T16:59:57.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:57 smithi188 ceph-mon[44101]: osdmap e865: 12 total, 12 up, 12 in 2024-04-23T16:59:57.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:57 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T16:59:57.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:57 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T16:59:57.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:57 smithi148 ceph-mon[44520]: osdmap e865: 12 total, 12 up, 12 in 2024-04-23T16:59:58.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 16:59:58 smithi103 ceph-mon[39629]: pgmap v2376: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:58.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 16:59:58 smithi188 ceph-mon[44101]: pgmap v2376: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:58.571 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:58.572 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:58.609 INFO:teuthology.orchestra.run.smithi103.stdout:386547057177 2024-04-23T16:59:58.609 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-23T16:59:58.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 16:59:58 smithi148 ceph-mon[44520]: pgmap v2376: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T16:59:58.976 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T16:59:58.977 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T17:00:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:00 smithi188 ceph-mon[44101]: pgmap v2377: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:00.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:00 smithi188 ceph-mon[44101]: overall HEALTH_OK 2024-04-23T17:00:00.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:00 smithi103 ceph-mon[39629]: pgmap v2377: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:00.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:00 smithi103 ceph-mon[39629]: overall HEALTH_OK 2024-04-23T17:00:00.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:00 smithi148 ceph-mon[44520]: pgmap v2377: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:00.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:00 smithi148 ceph-mon[44520]: overall HEALTH_OK 2024-04-23T17:00:02.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:02 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T17:00:02.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:02 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T17:00:02.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:02 smithi103 ceph-mon[39629]: pgmap v2378: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:02.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:02 smithi103 ceph-mon[39629]: osdmap e866: 12 total, 12 up, 12 in 2024-04-23T17:00:02.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:02 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T17:00:02.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:02 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T17:00:02.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:02 smithi188 ceph-mon[44101]: pgmap v2378: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:02.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:02 smithi188 ceph-mon[44101]: osdmap e866: 12 total, 12 up, 12 in 2024-04-23T17:00:02.474 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T17:00:02.475 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T17:00:02.475 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T17:00:02.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:02 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T17:00:02.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:02 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T17:00:02.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:02 smithi148 ceph-mon[44520]: pgmap v2378: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:02.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:02 smithi148 ceph-mon[44520]: osdmap e866: 12 total, 12 up, 12 in 2024-04-23T17:00:03.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:03 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T17:00:03.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:03 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T17:00:03.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:03 smithi103 ceph-mon[39629]: osdmap e867: 12 total, 12 up, 12 in 2024-04-23T17:00:03.446 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T17:00:03.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:03 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T17:00:03.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:03 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T17:00:03.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:03 smithi188 ceph-mon[44101]: osdmap e867: 12 total, 12 up, 12 in 2024-04-23T17:00:03.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:03 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T17:00:03.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:03 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T17:00:03.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:03 smithi148 ceph-mon[44520]: osdmap e867: 12 total, 12 up, 12 in 2024-04-23T17:00:04.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:04 smithi188 ceph-mon[44101]: pgmap v2381: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:04.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:04 smithi103 ceph-mon[39629]: pgmap v2381: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:04.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:04 smithi148 ceph-mon[44520]: pgmap v2381: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:05.849 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:05 smithi103 ceph-mon[39629]: pgmap v2382: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:05.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:05 smithi148 ceph-mon[44520]: pgmap v2382: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:05.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:05 smithi188 ceph-mon[44101]: pgmap v2382: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:08.224 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:08 smithi188 ceph-mon[44101]: pgmap v2383: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:08.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:08 smithi103 ceph-mon[39629]: pgmap v2383: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:08.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:08 smithi148 ceph-mon[44520]: pgmap v2383: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:09.434 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:09 smithi103 ceph-mon[39629]: 1.3 scrub starts 2024-04-23T17:00:09.434 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:09 smithi103 ceph-mon[39629]: 1.3 scrub ok 2024-04-23T17:00:09.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:09 smithi188 ceph-mon[44101]: 1.3 scrub starts 2024-04-23T17:00:09.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:09 smithi188 ceph-mon[44101]: 1.3 scrub ok 2024-04-23T17:00:09.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:09 smithi148 ceph-mon[44520]: 1.3 scrub starts 2024-04-23T17:00:09.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:09 smithi148 ceph-mon[44520]: 1.3 scrub ok 2024-04-23T17:00:10.116 INFO:teuthology.orchestra.run.smithi103.stdout:240518169138 2024-04-23T17:00:10.235 INFO:teuthology.orchestra.run.smithi103.stdout:201863463477 2024-04-23T17:00:10.443 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:10 smithi103 ceph-mon[39629]: pgmap v2384: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:10.443 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:10 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2560315215' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T17:00:10.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:10 smithi188 ceph-mon[44101]: pgmap v2384: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:10.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:10 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2560315215' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T17:00:10.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:10 smithi148 ceph-mon[44520]: pgmap v2384: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:10.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:10 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2560315215' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T17:00:11.067 INFO:teuthology.orchestra.run.smithi103.stdout:1163936137618 2024-04-23T17:00:11.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:11 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2306872691' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T17:00:11.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:11 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/408613352' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T17:00:11.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:11 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2306872691' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T17:00:11.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:11 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/408613352' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T17:00:11.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:11 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2306872691' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T17:00:11.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:11 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/408613352' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T17:00:12.153 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863463471 got 201863463477 for osd.6 2024-04-23T17:00:12.154 DEBUG:teuthology.parallel:result is None 2024-04-23T17:00:12.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:12 smithi188 ceph-mon[44101]: pgmap v2385: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:12.528 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:12 smithi103 ceph-mon[39629]: pgmap v2385: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:12.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:12 smithi148 ceph-mon[44520]: pgmap v2385: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:12.766 INFO:teuthology.orchestra.run.smithi103.stdout:283467842091 2024-04-23T17:00:13.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:13 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1337902228' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T17:00:13.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:13 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/2123452553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T17:00:13.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:13 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1337902228' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T17:00:13.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:13 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/2123452553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T17:00:13.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:13 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1337902228' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T17:00:13.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:13 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/2123452553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T17:00:14.207 INFO:teuthology.orchestra.run.smithi103.stdout:2658584756385 2024-04-23T17:00:14.317 INFO:teuthology.orchestra.run.smithi103.stdout:3126736191583 2024-04-23T17:00:14.398 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:14 smithi103 ceph-mon[39629]: 1.6 scrub starts 2024-04-23T17:00:14.398 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:14 smithi103 ceph-mon[39629]: 1.6 scrub ok 2024-04-23T17:00:14.398 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:14 smithi103 ceph-mon[39629]: pgmap v2386: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:14.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:14 smithi188 ceph-mon[44101]: 1.6 scrub starts 2024-04-23T17:00:14.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:14 smithi188 ceph-mon[44101]: 1.6 scrub ok 2024-04-23T17:00:14.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:14 smithi188 ceph-mon[44101]: pgmap v2386: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:14.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:14 smithi148 ceph-mon[44520]: 1.6 scrub starts 2024-04-23T17:00:14.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:14 smithi148 ceph-mon[44520]: 1.6 scrub ok 2024-04-23T17:00:14.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:14 smithi148 ceph-mon[44520]: pgmap v2386: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:14.961 INFO:teuthology.orchestra.run.smithi103.stdout:3453153706020 2024-04-23T17:00:15.218 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1163936137615 got 1163936137618 for osd.1 2024-04-23T17:00:15.218 DEBUG:teuthology.parallel:result is None 2024-04-23T17:00:15.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:15 smithi188 ceph-mon[44101]: 1.0 deep-scrub starts 2024-04-23T17:00:15.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:15 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/624323305' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T17:00:15.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:15 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1620373311' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T17:00:15.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:15 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/559087954' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T17:00:15.474 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:15 smithi103 ceph-mon[39629]: 1.0 deep-scrub starts 2024-04-23T17:00:15.474 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:15 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/624323305' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T17:00:15.474 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:15 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1620373311' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T17:00:15.474 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:15 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/559087954' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T17:00:15.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:15 smithi148 ceph-mon[44520]: 1.0 deep-scrub starts 2024-04-23T17:00:15.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:15 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/624323305' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T17:00:15.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:15 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1620373311' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T17:00:15.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:15 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/559087954' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T17:00:15.804 INFO:tasks.cephadm.ceph_manager.ceph:need seq 3453153706015 got 3453153706020 for osd.2 2024-04-23T17:00:15.804 DEBUG:teuthology.parallel:result is None 2024-04-23T17:00:16.168 INFO:teuthology.orchestra.run.smithi103.stdout:313532613159 2024-04-23T17:00:16.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:16 smithi103 ceph-mon[39629]: 1.0 deep-scrub ok 2024-04-23T17:00:16.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:16 smithi103 ceph-mon[39629]: pgmap v2387: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:16.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:16 smithi188 ceph-mon[44101]: 1.0 deep-scrub ok 2024-04-23T17:00:16.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:16 smithi188 ceph-mon[44101]: pgmap v2387: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:16.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:16 smithi148 ceph-mon[44520]: 1.0 deep-scrub ok 2024-04-23T17:00:16.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:16 smithi148 ceph-mon[44520]: pgmap v2387: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:16.822 INFO:tasks.cephadm.ceph_manager.ceph:need seq 283467842086 got 283467842091 for osd.8 2024-04-23T17:00:16.822 DEBUG:teuthology.parallel:result is None 2024-04-23T17:00:17.784 INFO:teuthology.orchestra.run.smithi103.stdout:347892351523 2024-04-23T17:00:18.041 INFO:teuthology.orchestra.run.smithi103.stdout:163208757818 2024-04-23T17:00:18.229 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518169134 got 240518169138 for osd.7 2024-04-23T17:00:18.229 DEBUG:teuthology.parallel:result is None 2024-04-23T17:00:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:18 smithi103 ceph-mon[39629]: pgmap v2388: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:18 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1370374512' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T17:00:18.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:18 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/590706325' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T17:00:18.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:18 smithi188 ceph-mon[44101]: pgmap v2388: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:18.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:18 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1370374512' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T17:00:18.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:18 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/590706325' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T17:00:18.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:18 smithi148 ceph-mon[44520]: pgmap v2388: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:18.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:18 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1370374512' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T17:00:18.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:18 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/590706325' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T17:00:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T17:00:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T17:00:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=3153475483,config=1020693696,health=3897446751,kv=3639101421}) 2024-04-23T17:00:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=1946483614,logm=2473709081}) 2024-04-23T17:00:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3875431139}) 2024-04-23T17:00:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=23984538}) 2024-04-23T17:00:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3715988327}) 2024-04-23T17:00:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2399216684}) 2024-04-23T17:00:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1931407100}) 2024-04-23T17:00:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4207117881}) 2024-04-23T17:00:19.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=454428391}) 2024-04-23T17:00:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=795063541}) 2024-04-23T17:00:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3968081113}) 2024-04-23T17:00:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1119863315}) 2024-04-23T17:00:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2454627350}) 2024-04-23T17:00:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=962555964}) 2024-04-23T17:00:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2279827236}) 2024-04-23T17:00:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326754634}) 2024-04-23T17:00:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2843033377}) 2024-04-23T17:00:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2926511881}) 2024-04-23T17:00:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2153698373}) 2024-04-23T17:00:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3440259052}) 2024-04-23T17:00:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1092709004}) 2024-04-23T17:00:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=722985363}) 2024-04-23T17:00:19.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2856858419}) 2024-04-23T17:00:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1220973647}) 2024-04-23T17:00:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2104628049}) 2024-04-23T17:00:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=719003974}) 2024-04-23T17:00:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2560383462}) 2024-04-23T17:00:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3530242797}) 2024-04-23T17:00:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3181451130}) 2024-04-23T17:00:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1491412093}) 2024-04-23T17:00:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=739794977}) 2024-04-23T17:00:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=246382569}) 2024-04-23T17:00:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=333613058}) 2024-04-23T17:00:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4246025398}) 2024-04-23T17:00:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3142225749}) 2024-04-23T17:00:19.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4290098373}) 2024-04-23T17:00:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3121497655}) 2024-04-23T17:00:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3970188028}) 2024-04-23T17:00:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1231179758}) 2024-04-23T17:00:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3062301570}) 2024-04-23T17:00:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3811823569}) 2024-04-23T17:00:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2763617822}) 2024-04-23T17:00:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1303656436}) 2024-04-23T17:00:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=932326946}) 2024-04-23T17:00:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=92594683}) 2024-04-23T17:00:19.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2606665497}) 2024-04-23T17:00:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=811807650}) 2024-04-23T17:00:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1454401504}) 2024-04-23T17:00:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=47778674}) 2024-04-23T17:00:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1555139098}) 2024-04-23T17:00:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=862748010}) 2024-04-23T17:00:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1709925232}) 2024-04-23T17:00:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=299505349}) 2024-04-23T17:00:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4139573291}) 2024-04-23T17:00:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2958640488}) 2024-04-23T17:00:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=979023438}) 2024-04-23T17:00:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2339061918}) 2024-04-23T17:00:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2125655035}) 2024-04-23T17:00:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1424251916}) 2024-04-23T17:00:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1595128292}) 2024-04-23T17:00:19.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=698465310}) 2024-04-23T17:00:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2522418997}) 2024-04-23T17:00:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212045765}) 2024-04-23T17:00:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1955355462}) 2024-04-23T17:00:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2862901738}) 2024-04-23T17:00:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2152630805}) 2024-04-23T17:00:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1288688424}) 2024-04-23T17:00:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4089095786}) 2024-04-23T17:00:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3847213003}) 2024-04-23T17:00:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3571692720}) 2024-04-23T17:00:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=906386032}) 2024-04-23T17:00:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3538210125}) 2024-04-23T17:00:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3506406940}) 2024-04-23T17:00:19.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2321852902}) 2024-04-23T17:00:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1610343916}) 2024-04-23T17:00:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2803602982}) 2024-04-23T17:00:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2437725824}) 2024-04-23T17:00:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3131966741}) 2024-04-23T17:00:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=887114222}) 2024-04-23T17:00:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2058760787}) 2024-04-23T17:00:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1100611985}) 2024-04-23T17:00:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4078955366}) 2024-04-23T17:00:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2884754513}) 2024-04-23T17:00:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2270285645}) 2024-04-23T17:00:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4241172723}) 2024-04-23T17:00:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=455200917}) 2024-04-23T17:00:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3951673133}) 2024-04-23T17:00:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4047508314}) 2024-04-23T17:00:19.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=203891999}) 2024-04-23T17:00:19.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4223270103}) 2024-04-23T17:00:19.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1403325433}) 2024-04-23T17:00:19.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397489142}) 2024-04-23T17:00:19.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=123274370}) 2024-04-23T17:00:19.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=316535710}) 2024-04-23T17:00:19.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1139575311}) 2024-04-23T17:00:19.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1191818910}) 2024-04-23T17:00:19.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477467238}) 2024-04-23T17:00:19.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4002383901}) 2024-04-23T17:00:19.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131192587}) 2024-04-23T17:00:19.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3606024069}) 2024-04-23T17:00:19.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=586459843}) 2024-04-23T17:00:19.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3724618089}) 2024-04-23T17:00:19.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1130693516}) 2024-04-23T17:00:19.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=991955676}) 2024-04-23T17:00:19.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2940374020}) 2024-04-23T17:00:19.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=86526349}) 2024-04-23T17:00:19.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {logm=65,mdsmap=3,mgr=19,mgr_command_descs=1,mgr_metadata=2,mgrstat=10} crc {logm=3756730829,mdsmap=2840952586,mgr=1765502803,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=3634108332}) 2024-04-23T17:00:19.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=5,mon_config_key=95} crc {mgrstat=321900603,mon_config_key=257586141}) 2024-04-23T17:00:19.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1973464474}) 2024-04-23T17:00:19.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=501989154}) 2024-04-23T17:00:19.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=1,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=81} crc {mon_config_key=2207797493,monmap=2133572193,osd_metadata=3151045301,osd_pg_creating=4054887907,osdmap=2872017916}) 2024-04-23T17:00:19.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:19 smithi103 ceph-mon[39629]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=24} crc {osdmap=2419960085}) 2024-04-23T17:00:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T17:00:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T17:00:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=3153475483,config=1020693696,health=3897446751,kv=3639101421}) 2024-04-23T17:00:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=1946483614,logm=2473709081}) 2024-04-23T17:00:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3875431139}) 2024-04-23T17:00:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=23984538}) 2024-04-23T17:00:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3715988327}) 2024-04-23T17:00:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2399216684}) 2024-04-23T17:00:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1931407100}) 2024-04-23T17:00:19.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4207117881}) 2024-04-23T17:00:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=454428391}) 2024-04-23T17:00:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=795063541}) 2024-04-23T17:00:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3968081113}) 2024-04-23T17:00:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1119863315}) 2024-04-23T17:00:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2454627350}) 2024-04-23T17:00:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=962555964}) 2024-04-23T17:00:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2279827236}) 2024-04-23T17:00:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326754634}) 2024-04-23T17:00:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2843033377}) 2024-04-23T17:00:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2926511881}) 2024-04-23T17:00:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2153698373}) 2024-04-23T17:00:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3440259052}) 2024-04-23T17:00:19.475 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1092709004}) 2024-04-23T17:00:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=722985363}) 2024-04-23T17:00:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2856858419}) 2024-04-23T17:00:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1220973647}) 2024-04-23T17:00:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2104628049}) 2024-04-23T17:00:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=719003974}) 2024-04-23T17:00:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2560383462}) 2024-04-23T17:00:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3530242797}) 2024-04-23T17:00:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3181451130}) 2024-04-23T17:00:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1491412093}) 2024-04-23T17:00:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=739794977}) 2024-04-23T17:00:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=246382569}) 2024-04-23T17:00:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=333613058}) 2024-04-23T17:00:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4246025398}) 2024-04-23T17:00:19.476 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3142225749}) 2024-04-23T17:00:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4290098373}) 2024-04-23T17:00:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3121497655}) 2024-04-23T17:00:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3970188028}) 2024-04-23T17:00:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1231179758}) 2024-04-23T17:00:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3062301570}) 2024-04-23T17:00:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3811823569}) 2024-04-23T17:00:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2763617822}) 2024-04-23T17:00:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1303656436}) 2024-04-23T17:00:19.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=932326946}) 2024-04-23T17:00:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=92594683}) 2024-04-23T17:00:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2606665497}) 2024-04-23T17:00:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=811807650}) 2024-04-23T17:00:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1454401504}) 2024-04-23T17:00:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=47778674}) 2024-04-23T17:00:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1555139098}) 2024-04-23T17:00:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=862748010}) 2024-04-23T17:00:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1709925232}) 2024-04-23T17:00:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=299505349}) 2024-04-23T17:00:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4139573291}) 2024-04-23T17:00:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2958640488}) 2024-04-23T17:00:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=979023438}) 2024-04-23T17:00:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2339061918}) 2024-04-23T17:00:19.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2125655035}) 2024-04-23T17:00:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1424251916}) 2024-04-23T17:00:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1595128292}) 2024-04-23T17:00:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=698465310}) 2024-04-23T17:00:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2522418997}) 2024-04-23T17:00:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212045765}) 2024-04-23T17:00:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1955355462}) 2024-04-23T17:00:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2862901738}) 2024-04-23T17:00:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2152630805}) 2024-04-23T17:00:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1288688424}) 2024-04-23T17:00:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4089095786}) 2024-04-23T17:00:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3847213003}) 2024-04-23T17:00:19.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3571692720}) 2024-04-23T17:00:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=906386032}) 2024-04-23T17:00:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3538210125}) 2024-04-23T17:00:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3506406940}) 2024-04-23T17:00:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2321852902}) 2024-04-23T17:00:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1610343916}) 2024-04-23T17:00:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2803602982}) 2024-04-23T17:00:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2437725824}) 2024-04-23T17:00:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3131966741}) 2024-04-23T17:00:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=887114222}) 2024-04-23T17:00:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2058760787}) 2024-04-23T17:00:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1100611985}) 2024-04-23T17:00:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4078955366}) 2024-04-23T17:00:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2884754513}) 2024-04-23T17:00:19.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2270285645}) 2024-04-23T17:00:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4241172723}) 2024-04-23T17:00:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=455200917}) 2024-04-23T17:00:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3951673133}) 2024-04-23T17:00:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4047508314}) 2024-04-23T17:00:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=203891999}) 2024-04-23T17:00:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4223270103}) 2024-04-23T17:00:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1403325433}) 2024-04-23T17:00:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397489142}) 2024-04-23T17:00:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=123274370}) 2024-04-23T17:00:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=316535710}) 2024-04-23T17:00:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1139575311}) 2024-04-23T17:00:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1191818910}) 2024-04-23T17:00:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477467238}) 2024-04-23T17:00:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4002383901}) 2024-04-23T17:00:19.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131192587}) 2024-04-23T17:00:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3606024069}) 2024-04-23T17:00:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=586459843}) 2024-04-23T17:00:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3724618089}) 2024-04-23T17:00:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1130693516}) 2024-04-23T17:00:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=991955676}) 2024-04-23T17:00:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2940374020}) 2024-04-23T17:00:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=86526349}) 2024-04-23T17:00:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {logm=65,mdsmap=3,mgr=19,mgr_command_descs=1,mgr_metadata=2,mgrstat=10} crc {logm=3756730829,mdsmap=2840952586,mgr=1765502803,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=3634108332}) 2024-04-23T17:00:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=5,mon_config_key=95} crc {mgrstat=321900603,mon_config_key=257586141}) 2024-04-23T17:00:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1973464474}) 2024-04-23T17:00:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=501989154}) 2024-04-23T17:00:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=1,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=81} crc {mon_config_key=2207797493,monmap=2133572193,osd_metadata=3151045301,osd_pg_creating=4054887907,osdmap=2872017916}) 2024-04-23T17:00:19.482 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:19 smithi188 ceph-mon[44101]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=24} crc {osdmap=2419960085}) 2024-04-23T17:00:19.489 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2658584756381 got 2658584756385 for osd.3 2024-04-23T17:00:19.489 DEBUG:teuthology.parallel:result is None 2024-04-23T17:00:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T17:00:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T17:00:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=3153475483,config=1020693696,health=3897446751,kv=3639101421}) 2024-04-23T17:00:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=1946483614,logm=2473709081}) 2024-04-23T17:00:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3875431139}) 2024-04-23T17:00:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=23984538}) 2024-04-23T17:00:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3715988327}) 2024-04-23T17:00:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2399216684}) 2024-04-23T17:00:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1931407100}) 2024-04-23T17:00:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4207117881}) 2024-04-23T17:00:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=454428391}) 2024-04-23T17:00:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=795063541}) 2024-04-23T17:00:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3968081113}) 2024-04-23T17:00:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1119863315}) 2024-04-23T17:00:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2454627350}) 2024-04-23T17:00:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=962555964}) 2024-04-23T17:00:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2279827236}) 2024-04-23T17:00:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326754634}) 2024-04-23T17:00:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2843033377}) 2024-04-23T17:00:19.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2926511881}) 2024-04-23T17:00:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2153698373}) 2024-04-23T17:00:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3440259052}) 2024-04-23T17:00:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1092709004}) 2024-04-23T17:00:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=722985363}) 2024-04-23T17:00:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2856858419}) 2024-04-23T17:00:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1220973647}) 2024-04-23T17:00:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2104628049}) 2024-04-23T17:00:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=719003974}) 2024-04-23T17:00:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2560383462}) 2024-04-23T17:00:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3530242797}) 2024-04-23T17:00:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3181451130}) 2024-04-23T17:00:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1491412093}) 2024-04-23T17:00:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=739794977}) 2024-04-23T17:00:19.628 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=246382569}) 2024-04-23T17:00:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=333613058}) 2024-04-23T17:00:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4246025398}) 2024-04-23T17:00:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3142225749}) 2024-04-23T17:00:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4290098373}) 2024-04-23T17:00:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3121497655}) 2024-04-23T17:00:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3970188028}) 2024-04-23T17:00:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1231179758}) 2024-04-23T17:00:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3062301570}) 2024-04-23T17:00:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3811823569}) 2024-04-23T17:00:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2763617822}) 2024-04-23T17:00:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1303656436}) 2024-04-23T17:00:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=932326946}) 2024-04-23T17:00:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=92594683}) 2024-04-23T17:00:19.629 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2606665497}) 2024-04-23T17:00:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=811807650}) 2024-04-23T17:00:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1454401504}) 2024-04-23T17:00:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=47778674}) 2024-04-23T17:00:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1555139098}) 2024-04-23T17:00:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=862748010}) 2024-04-23T17:00:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1709925232}) 2024-04-23T17:00:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=299505349}) 2024-04-23T17:00:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4139573291}) 2024-04-23T17:00:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2958640488}) 2024-04-23T17:00:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=979023438}) 2024-04-23T17:00:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2339061918}) 2024-04-23T17:00:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2125655035}) 2024-04-23T17:00:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1424251916}) 2024-04-23T17:00:19.630 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1595128292}) 2024-04-23T17:00:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=698465310}) 2024-04-23T17:00:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2522418997}) 2024-04-23T17:00:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212045765}) 2024-04-23T17:00:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1955355462}) 2024-04-23T17:00:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2862901738}) 2024-04-23T17:00:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2152630805}) 2024-04-23T17:00:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1288688424}) 2024-04-23T17:00:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4089095786}) 2024-04-23T17:00:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3847213003}) 2024-04-23T17:00:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3571692720}) 2024-04-23T17:00:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=906386032}) 2024-04-23T17:00:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3538210125}) 2024-04-23T17:00:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3506406940}) 2024-04-23T17:00:19.631 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2321852902}) 2024-04-23T17:00:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1610343916}) 2024-04-23T17:00:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2803602982}) 2024-04-23T17:00:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2437725824}) 2024-04-23T17:00:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3131966741}) 2024-04-23T17:00:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=887114222}) 2024-04-23T17:00:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2058760787}) 2024-04-23T17:00:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1100611985}) 2024-04-23T17:00:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4078955366}) 2024-04-23T17:00:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2884754513}) 2024-04-23T17:00:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2270285645}) 2024-04-23T17:00:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4241172723}) 2024-04-23T17:00:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=455200917}) 2024-04-23T17:00:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3951673133}) 2024-04-23T17:00:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4047508314}) 2024-04-23T17:00:19.632 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=203891999}) 2024-04-23T17:00:19.633 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4223270103}) 2024-04-23T17:00:19.633 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1403325433}) 2024-04-23T17:00:19.633 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397489142}) 2024-04-23T17:00:19.633 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=123274370}) 2024-04-23T17:00:19.633 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=316535710}) 2024-04-23T17:00:19.633 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1139575311}) 2024-04-23T17:00:19.633 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1191818910}) 2024-04-23T17:00:19.633 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477467238}) 2024-04-23T17:00:19.633 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4002383901}) 2024-04-23T17:00:19.633 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131192587}) 2024-04-23T17:00:19.633 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3606024069}) 2024-04-23T17:00:19.633 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=586459843}) 2024-04-23T17:00:19.633 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3724618089}) 2024-04-23T17:00:19.633 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1130693516}) 2024-04-23T17:00:19.634 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=991955676}) 2024-04-23T17:00:19.634 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2940374020}) 2024-04-23T17:00:19.634 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=86526349}) 2024-04-23T17:00:19.634 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {logm=65,mdsmap=3,mgr=19,mgr_command_descs=1,mgr_metadata=2,mgrstat=10} crc {logm=3756730829,mdsmap=2840952586,mgr=1765502803,mgr_command_descs=4152233241,mgr_metadata=3768515308,mgrstat=3634108332}) 2024-04-23T17:00:19.634 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=5,mon_config_key=95} crc {mgrstat=321900603,mon_config_key=257586141}) 2024-04-23T17:00:19.634 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1973464474}) 2024-04-23T17:00:19.634 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=501989154}) 2024-04-23T17:00:19.634 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=1,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=81} crc {mon_config_key=2207797493,monmap=2133572193,osd_metadata=3151045301,osd_pg_creating=4054887907,osdmap=2872017916}) 2024-04-23T17:00:19.634 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:19 smithi148 ceph-mon[44520]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=24} crc {osdmap=2419960085}) 2024-04-23T17:00:19.883 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757812 got 163208757818 for osd.5 2024-04-23T17:00:19.883 DEBUG:teuthology.parallel:result is None 2024-04-23T17:00:20.395 INFO:tasks.cephadm.ceph_manager.ceph:need seq 3126736191578 got 3126736191583 for osd.4 2024-04-23T17:00:20.396 DEBUG:teuthology.parallel:result is None 2024-04-23T17:00:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:20 smithi188 ceph-mon[44101]: pgmap v2389: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:20.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:20 smithi188 ceph-mon[44101]: osdmap e868: 12 total, 12 up, 12 in 2024-04-23T17:00:20.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:20 smithi103 ceph-mon[39629]: pgmap v2389: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:20.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:20 smithi103 ceph-mon[39629]: osdmap e868: 12 total, 12 up, 12 in 2024-04-23T17:00:20.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:20 smithi148 ceph-mon[44520]: pgmap v2389: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:20.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:20 smithi148 ceph-mon[44520]: osdmap e868: 12 total, 12 up, 12 in 2024-04-23T17:00:20.884 INFO:teuthology.orchestra.run.smithi103.stdout:34359738965 2024-04-23T17:00:20.966 INFO:teuthology.orchestra.run.smithi103.stdout:386547057181 2024-04-23T17:00:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:21 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/1131709170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T17:00:21.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:21 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/4177183559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T17:00:21.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:21 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/1131709170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T17:00:21.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:21 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/4177183559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T17:00:21.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:21 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/1131709170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T17:00:21.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:21 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/4177183559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T17:00:21.732 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892351518 got 347892351523 for osd.10 2024-04-23T17:00:21.732 DEBUG:teuthology.parallel:result is None 2024-04-23T17:00:21.838 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532613153 got 313532613159 for osd.9 2024-04-23T17:00:21.838 DEBUG:teuthology.parallel:result is None 2024-04-23T17:00:22.267 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738959 got 34359738965 for osd.0 2024-04-23T17:00:22.267 DEBUG:teuthology.parallel:result is None 2024-04-23T17:00:22.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:22 smithi188 ceph-mon[44101]: pgmap v2391: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:22.573 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:22 smithi103 ceph-mon[39629]: pgmap v2391: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:22.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:22 smithi148 ceph-mon[44520]: pgmap v2391: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:22.855 INFO:tasks.cephadm.ceph_manager.ceph:need seq 386547057177 got 386547057181 for osd.11 2024-04-23T17:00:22.855 DEBUG:teuthology.parallel:result is None 2024-04-23T17:00:22.856 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-23T17:00:22.856 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T17:00:23.123 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T17:00:23.460 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:23 smithi103 ceph-mon[39629]: 1.5 scrub starts 2024-04-23T17:00:23.460 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:23 smithi103 ceph-mon[39629]: 1.5 scrub ok 2024-04-23T17:00:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:23 smithi188 ceph-mon[44101]: 1.5 scrub starts 2024-04-23T17:00:23.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:23 smithi188 ceph-mon[44101]: 1.5 scrub ok 2024-04-23T17:00:23.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:23 smithi148 ceph-mon[44520]: 1.5 scrub starts 2024-04-23T17:00:23.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:23 smithi148 ceph-mon[44520]: 1.5 scrub ok 2024-04-23T17:00:24.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:24 smithi188 ceph-mon[44101]: pgmap v2392: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:24.478 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:24 smithi103 ceph-mon[39629]: pgmap v2392: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:24.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:24 smithi148 ceph-mon[44520]: pgmap v2392: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:25.251 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T17:00:25.253 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T17:00:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:25 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T17:00:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:25 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T17:00:25.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:25 smithi188 ceph-mon[44101]: osdmap e869: 12 total, 12 up, 12 in 2024-04-23T17:00:25.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:25 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T17:00:25.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:25 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T17:00:25.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:25 smithi103 ceph-mon[39629]: osdmap e869: 12 total, 12 up, 12 in 2024-04-23T17:00:25.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:25 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T17:00:25.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:25 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T17:00:25.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:25 smithi148 ceph-mon[44520]: osdmap e869: 12 total, 12 up, 12 in 2024-04-23T17:00:26.055 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":2394,"stamp":"2024-04-23T17:00:25.076426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":804992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1124941824,"kb_used":6233108,"kb_used_data":9224,"kb_used_omap":989,"kb_used_meta":6221666,"kb_avail":1118708716,"statfs":{"total":1151940427776,"available":1145557725184,"internally_reserved":0,"allocated":9445376,"data_stored":4894200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1013120,"internal_metadata":6370986624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001341"},"pg_stats":[{"pgid":"1.a","version":"255'139","reported_seq":2128,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-04-23T17:00:19.161374+0000","last_change":"2024-04-23T16:59:48.468063+0000","last_active":"2024-04-23T17:00:19.161374+0000","last_peered":"2024-04-23T17:00:19.161374+0000","last_clean":"2024-04-23T17:00:19.161374+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T17:00:19.161374+0000","last_undegraded":"2024-04-23T17:00:19.161374+0000","last_fullsized":"2024-04-23T17:00:19.161374+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:59:48.467958+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:59:48.467958+0000","last_clean_scrub_stamp":"2024-04-23T16:59:48.467958+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:49.246592+0000","scrub_duration":205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1852,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-04-23T17:00:20.940725+0000","last_change":"2024-04-23T16:59:06.608130+0000","last_active":"2024-04-23T17:00:20.940725+0000","last_peered":"2024-04-23T17:00:20.940725+0000","last_clean":"2024-04-23T17:00:20.940725+0000","last_became_active":"2024-04-23T16:57:51.578086+0000","last_became_peered":"2024-04-23T16:57:51.578086+0000","last_unstale":"2024-04-23T17:00:20.940725+0000","last_undegraded":"2024-04-23T17:00:20.940725+0000","last_fullsized":"2024-04-23T17:00:20.940725+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:59:06.607994+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:56:30.007005+0000","last_clean_scrub_stamp":"2024-04-23T16:59:06.607994+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:24.411418+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":804992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":2109,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-04-23T17:00:20.879984+0000","last_change":"2024-04-23T16:59:14.146475+0000","last_active":"2024-04-23T17:00:20.879984+0000","last_peered":"2024-04-23T17:00:20.879984+0000","last_clean":"2024-04-23T17:00:20.879984+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T17:00:20.879984+0000","last_undegraded":"2024-04-23T17:00:20.879984+0000","last_fullsized":"2024-04-23T17:00:20.879984+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:59:14.146349+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:56:33.274411+0000","last_clean_scrub_stamp":"2024-04-23T16:59:14.146349+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:43.610603+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":920,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-04-23T17:00:20.879961+0000","last_change":"2024-04-23T16:59:48.269077+0000","last_active":"2024-04-23T17:00:20.879961+0000","last_peered":"2024-04-23T17:00:20.879961+0000","last_clean":"2024-04-23T17:00:20.879961+0000","last_became_active":"2024-04-23T16:58:28.608091+0000","last_became_peered":"2024-04-23T16:58:28.608091+0000","last_unstale":"2024-04-23T17:00:20.879961+0000","last_undegraded":"2024-04-23T17:00:20.879961+0000","last_fullsized":"2024-04-23T17:00:20.879961+0000","mapping_epoch":834,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:59:48.268977+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:59:48.268977+0000","last_clean_scrub_stamp":"2024-04-23T16:59:48.268977+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:03.397997+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1715,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-04-23T17:00:20.648003+0000","last_change":"2024-04-23T17:00:08.470978+0000","last_active":"2024-04-23T17:00:20.648003+0000","last_peered":"2024-04-23T17:00:20.648003+0000","last_clean":"2024-04-23T17:00:20.648003+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T17:00:20.648003+0000","last_undegraded":"2024-04-23T17:00:20.648003+0000","last_fullsized":"2024-04-23T17:00:20.648003+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T17:00:08.470877+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T17:00:08.470877+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:14.787650+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":964,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-04-23T17:00:19.751573+0000","last_change":"2024-04-23T16:59:06.575264+0000","last_active":"2024-04-23T17:00:19.751573+0000","last_peered":"2024-04-23T17:00:19.751573+0000","last_clean":"2024-04-23T17:00:19.751573+0000","last_became_active":"2024-04-23T16:57:47.588380+0000","last_became_peered":"2024-04-23T16:57:47.588380+0000","last_unstale":"2024-04-23T17:00:19.751573+0000","last_undegraded":"2024-04-23T17:00:19.751573+0000","last_fullsized":"2024-04-23T17:00:19.751573+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:59:06.575114+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:59:06.575114+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:34.944585+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":963,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-04-23T17:00:19.751504+0000","last_change":"2024-04-23T17:00:12.826497+0000","last_active":"2024-04-23T17:00:19.751504+0000","last_peered":"2024-04-23T17:00:19.751504+0000","last_clean":"2024-04-23T17:00:19.751504+0000","last_became_active":"2024-04-23T16:57:47.588492+0000","last_became_peered":"2024-04-23T16:57:47.588492+0000","last_unstale":"2024-04-23T17:00:19.751504+0000","last_undegraded":"2024-04-23T17:00:19.751504+0000","last_fullsized":"2024-04-23T17:00:19.751504+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T17:00:12.826375+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:55:47.572718+0000","last_clean_scrub_stamp":"2024-04-23T17:00:12.826375+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:28.579106+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":2154,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-04-23T17:00:20.336223+0000","last_change":"2024-04-23T16:58:54.530521+0000","last_active":"2024-04-23T17:00:20.336223+0000","last_peered":"2024-04-23T17:00:20.336223+0000","last_clean":"2024-04-23T17:00:20.336223+0000","last_became_active":"2024-04-23T16:57:44.571322+0000","last_became_peered":"2024-04-23T16:57:44.571322+0000","last_unstale":"2024-04-23T17:00:20.336223+0000","last_undegraded":"2024-04-23T17:00:20.336223+0000","last_fullsized":"2024-04-23T17:00:20.336223+0000","mapping_epoch":813,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":814,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:58:54.530405+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:56:16.210909+0000","last_clean_scrub_stamp":"2024-04-23T16:58:54.530405+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:23.687965+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":2016,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-04-23T17:00:20.671904+0000","last_change":"2024-04-23T16:59:36.231196+0000","last_active":"2024-04-23T17:00:20.671904+0000","last_peered":"2024-04-23T17:00:20.671904+0000","last_clean":"2024-04-23T17:00:20.671904+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T17:00:20.671904+0000","last_undegraded":"2024-04-23T17:00:20.671904+0000","last_fullsized":"2024-04-23T17:00:20.671904+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:59:36.231062+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:59:36.231062+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:48.388518+0000","scrub_duration":260,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1861,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-04-23T17:00:21.702751+0000","last_change":"2024-04-23T17:00:21.702781+0000","last_active":"2024-04-23T17:00:21.702751+0000","last_peered":"2024-04-23T17:00:21.702751+0000","last_clean":"2024-04-23T17:00:21.702751+0000","last_became_active":"2024-04-23T16:57:47.589840+0000","last_became_peered":"2024-04-23T16:57:47.589840+0000","last_unstale":"2024-04-23T17:00:21.702751+0000","last_undegraded":"2024-04-23T17:00:21.702751+0000","last_fullsized":"2024-04-23T17:00:21.702751+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T17:00:21.702666+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:55:11.687173+0000","last_clean_scrub_stamp":"2024-04-23T17:00:21.702666+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:51.372877+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":2110,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-04-23T17:00:20.879955+0000","last_change":"2024-04-23T17:00:14.290280+0000","last_active":"2024-04-23T17:00:20.879955+0000","last_peered":"2024-04-23T17:00:20.879955+0000","last_clean":"2024-04-23T17:00:20.879955+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T17:00:20.879955+0000","last_undegraded":"2024-04-23T17:00:20.879955+0000","last_fullsized":"2024-04-23T17:00:20.879955+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T17:00:14.290170+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T17:00:14.290170+0000","last_clean_scrub_stamp":"2024-04-23T17:00:14.290170+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:14.372346+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":804992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1474560,"data_stored":1460352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057182,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1087388,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":1086270,"kb_avail":92657764,"statfs":{"total":95995035648,"available":94881550336,"internally_reserved":0,"allocated":1007616,"data_stored":638391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1112340932},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351525,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76256,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93668896,"statfs":{"total":95995035648,"available":95916949504,"internally_reserved":0,"allocated":679936,"data_stored":310711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":77396416},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613162,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852444,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892708,"statfs":{"total":95995035648,"available":95122132992,"internally_reserved":0,"allocated":679936,"data_stored":310711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":872085952},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467842094,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":600092,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93145060,"statfs":{"total":95995035648,"available":95380541440,"internally_reserved":0,"allocated":1007616,"data_stored":638391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169141,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":739996,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93005156,"statfs":{"total":95995035648,"available":95237279744,"internally_reserved":0,"allocated":679936,"data_stored":310711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":756939210},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463480,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":605332,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93139820,"statfs":{"total":95995035648,"available":95375175680,"internally_reserved":0,"allocated":790528,"data_stored":364613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":619051462},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137621,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":624284,"kb_used_data":664,"kb_used_omap":231,"kb_used_meta":623256,"kb_avail":93120868,"statfs":{"total":95995035648,"available":95355768832,"internally_reserved":0,"allocated":679936,"data_stored":310711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":638215056},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738967,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":749724,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":92995428,"statfs":{"total":95995035648,"available":95227318272,"internally_reserved":0,"allocated":679936,"data_stored":310711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":766900677},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":804,"seq":3453153706022,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":47012,"kb_used_data":864,"kb_used_omap":559,"kb_used_meta":45456,"kb_avail":93698140,"statfs":{"total":95995035648,"available":95946895360,"internally_reserved":0,"allocated":884736,"data_stored":511967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":572908,"internal_metadata":46547476},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":619,"seq":2658584756387,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":82708,"kb_used_data":772,"kb_used_omap":169,"kb_used_meta":81750,"kb_avail":93662444,"statfs":{"total":95995035648,"available":95910342656,"internally_reserved":0,"allocated":790528,"data_stored":364613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":173193,"internal_metadata":83712887},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":728,"seq":3126736191586,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":85980,"kb_used_data":664,"kb_used_omap":17,"kb_used_meta":85166,"kb_avail":93659172,"statfs":{"total":95995035648,"available":95906992128,"internally_reserved":0,"allocated":679936,"data_stored":310711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17598,"internal_metadata":87210818},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":681892,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93063260,"statfs":{"total":95995035648,"available":95296778240,"internally_reserved":0,"allocated":884736,"data_stored":511959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T17:00:26.057 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T17:00:26.325 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T17:00:26.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:26 smithi103 ceph-mon[39629]: pgmap v2394: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:26.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:26 smithi103 ceph-mon[39629]: 1.7 scrub starts 2024-04-23T17:00:26.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:26 smithi103 ceph-mon[39629]: 1.7 scrub ok 2024-04-23T17:00:26.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:26 smithi188 ceph-mon[44101]: pgmap v2394: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:26.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:26 smithi188 ceph-mon[44101]: 1.7 scrub starts 2024-04-23T17:00:26.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:26 smithi188 ceph-mon[44101]: 1.7 scrub ok 2024-04-23T17:00:26.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:26 smithi148 ceph-mon[44520]: pgmap v2394: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:26.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:26 smithi148 ceph-mon[44520]: 1.7 scrub starts 2024-04-23T17:00:26.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:26 smithi148 ceph-mon[44520]: 1.7 scrub ok 2024-04-23T17:00:27.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:27 smithi188 ceph-mon[44101]: from='client.18228 v1:172.21.15.103:0/123653180' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T17:00:27.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:27 smithi103 ceph-mon[39629]: from='client.18228 v1:172.21.15.103:0/123653180' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T17:00:27.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:27 smithi148 ceph-mon[44520]: from='client.18228 v1:172.21.15.103:0/123653180' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T17:00:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:28 smithi103 ceph-mon[39629]: pgmap v2395: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:28.417 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-23T17:00:28.420 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-04-23T17:00:28.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:28 smithi188 ceph-mon[44101]: pgmap v2395: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:28.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:28 smithi148 ceph-mon[44520]: pgmap v2395: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:29.109 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":2395,"stamp":"2024-04-23T17:00:27.076758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":804992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1124941824,"kb_used":6232800,"kb_used_data":9044,"kb_used_omap":989,"kb_used_meta":6221666,"kb_avail":1118709024,"statfs":{"total":1151940427776,"available":1145558040576,"internally_reserved":0,"allocated":9261056,"data_stored":4804153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1013120,"internal_metadata":6370986624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001317"},"pg_stats":[{"pgid":"1.a","version":"255'139","reported_seq":2128,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-04-23T17:00:19.161374+0000","last_change":"2024-04-23T16:59:48.468063+0000","last_active":"2024-04-23T17:00:19.161374+0000","last_peered":"2024-04-23T17:00:19.161374+0000","last_clean":"2024-04-23T17:00:19.161374+0000","last_became_active":"2024-04-23T16:28:25.058534+0000","last_became_peered":"2024-04-23T16:28:25.058534+0000","last_unstale":"2024-04-23T17:00:19.161374+0000","last_undegraded":"2024-04-23T17:00:19.161374+0000","last_fullsized":"2024-04-23T17:00:19.161374+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:59:48.467958+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:59:48.467958+0000","last_clean_scrub_stamp":"2024-04-23T16:59:48.467958+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:49.246592+0000","scrub_duration":205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"255'139","reported_seq":1852,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-04-23T17:00:20.940725+0000","last_change":"2024-04-23T16:59:06.608130+0000","last_active":"2024-04-23T17:00:20.940725+0000","last_peered":"2024-04-23T17:00:20.940725+0000","last_clean":"2024-04-23T17:00:20.940725+0000","last_became_active":"2024-04-23T16:57:51.578086+0000","last_became_peered":"2024-04-23T16:57:51.578086+0000","last_unstale":"2024-04-23T17:00:20.940725+0000","last_undegraded":"2024-04-23T17:00:20.940725+0000","last_fullsized":"2024-04-23T17:00:20.940725+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:59:06.607994+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:56:30.007005+0000","last_clean_scrub_stamp":"2024-04-23T16:59:06.607994+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:24.411418+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":804992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"255'139","reported_seq":2111,"reported_epoch":869,"state":"active+clean","last_fresh":"2024-04-23T17:00:25.580088+0000","last_change":"2024-04-23T16:59:14.146475+0000","last_active":"2024-04-23T17:00:25.580088+0000","last_peered":"2024-04-23T17:00:25.580088+0000","last_clean":"2024-04-23T17:00:25.580088+0000","last_became_active":"2024-04-23T16:41:14.928714+0000","last_became_peered":"2024-04-23T16:41:14.928714+0000","last_unstale":"2024-04-23T17:00:25.580088+0000","last_undegraded":"2024-04-23T17:00:25.580088+0000","last_fullsized":"2024-04-23T17:00:25.580088+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:59:14.146349+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:56:33.274411+0000","last_clean_scrub_stamp":"2024-04-23T16:59:14.146349+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:43.610603+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"255'139","reported_seq":922,"reported_epoch":869,"state":"active+clean","last_fresh":"2024-04-23T17:00:25.580094+0000","last_change":"2024-04-23T16:59:48.269077+0000","last_active":"2024-04-23T17:00:25.580094+0000","last_peered":"2024-04-23T17:00:25.580094+0000","last_clean":"2024-04-23T17:00:25.580094+0000","last_became_active":"2024-04-23T16:58:28.608091+0000","last_became_peered":"2024-04-23T16:58:28.608091+0000","last_unstale":"2024-04-23T17:00:25.580094+0000","last_undegraded":"2024-04-23T17:00:25.580094+0000","last_fullsized":"2024-04-23T17:00:25.580094+0000","mapping_epoch":834,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:59:48.268977+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:59:48.268977+0000","last_clean_scrub_stamp":"2024-04-23T16:59:48.268977+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:03.397997+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"255'139","reported_seq":1715,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-04-23T17:00:20.648003+0000","last_change":"2024-04-23T17:00:08.470978+0000","last_active":"2024-04-23T17:00:20.648003+0000","last_peered":"2024-04-23T17:00:20.648003+0000","last_clean":"2024-04-23T17:00:20.648003+0000","last_became_active":"2024-04-23T16:39:23.985162+0000","last_became_peered":"2024-04-23T16:39:23.985162+0000","last_unstale":"2024-04-23T17:00:20.648003+0000","last_undegraded":"2024-04-23T17:00:20.648003+0000","last_fullsized":"2024-04-23T17:00:20.648003+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T17:00:08.470877+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:38:16.458700+0000","last_clean_scrub_stamp":"2024-04-23T17:00:08.470877+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:14.787650+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"255'139","reported_seq":964,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-04-23T17:00:19.751573+0000","last_change":"2024-04-23T16:59:06.575264+0000","last_active":"2024-04-23T17:00:19.751573+0000","last_peered":"2024-04-23T17:00:19.751573+0000","last_clean":"2024-04-23T17:00:19.751573+0000","last_became_active":"2024-04-23T16:57:47.588380+0000","last_became_peered":"2024-04-23T16:57:47.588380+0000","last_unstale":"2024-04-23T17:00:19.751573+0000","last_undegraded":"2024-04-23T17:00:19.751573+0000","last_fullsized":"2024-04-23T17:00:19.751573+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:59:06.575114+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:31:45.883736+0000","last_clean_scrub_stamp":"2024-04-23T16:59:06.575114+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:34.944585+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"255'139","reported_seq":963,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-04-23T17:00:19.751504+0000","last_change":"2024-04-23T17:00:12.826497+0000","last_active":"2024-04-23T17:00:19.751504+0000","last_peered":"2024-04-23T17:00:19.751504+0000","last_clean":"2024-04-23T17:00:19.751504+0000","last_became_active":"2024-04-23T16:57:47.588492+0000","last_became_peered":"2024-04-23T16:57:47.588492+0000","last_unstale":"2024-04-23T17:00:19.751504+0000","last_undegraded":"2024-04-23T17:00:19.751504+0000","last_fullsized":"2024-04-23T17:00:19.751504+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T17:00:12.826375+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:55:47.572718+0000","last_clean_scrub_stamp":"2024-04-23T17:00:12.826375+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:28.579106+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"255'139","reported_seq":2162,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-04-23T17:00:25.312639+0000","last_change":"2024-04-23T17:00:25.312683+0000","last_active":"2024-04-23T17:00:25.312639+0000","last_peered":"2024-04-23T17:00:25.312639+0000","last_clean":"2024-04-23T17:00:25.312639+0000","last_became_active":"2024-04-23T16:57:44.571322+0000","last_became_peered":"2024-04-23T16:57:44.571322+0000","last_unstale":"2024-04-23T17:00:25.312639+0000","last_undegraded":"2024-04-23T17:00:25.312639+0000","last_fullsized":"2024-04-23T17:00:25.312639+0000","mapping_epoch":813,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":814,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T17:00:25.312545+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:56:16.210909+0000","last_clean_scrub_stamp":"2024-04-23T17:00:25.312545+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:32.779669+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"255'139","reported_seq":2018,"reported_epoch":869,"state":"active+clean","last_fresh":"2024-04-23T17:00:25.771077+0000","last_change":"2024-04-23T16:59:36.231196+0000","last_active":"2024-04-23T17:00:25.771077+0000","last_peered":"2024-04-23T17:00:25.771077+0000","last_clean":"2024-04-23T17:00:25.771077+0000","last_became_active":"2024-04-23T16:28:22.049607+0000","last_became_peered":"2024-04-23T16:28:22.049607+0000","last_unstale":"2024-04-23T17:00:25.771077+0000","last_undegraded":"2024-04-23T17:00:25.771077+0000","last_fullsized":"2024-04-23T17:00:25.771077+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T16:59:36.231062+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:41:47.722762+0000","last_clean_scrub_stamp":"2024-04-23T16:59:36.231062+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:48.388518+0000","scrub_duration":260,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"255'139","reported_seq":1861,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-04-23T17:00:21.702751+0000","last_change":"2024-04-23T17:00:21.702781+0000","last_active":"2024-04-23T17:00:21.702751+0000","last_peered":"2024-04-23T17:00:21.702751+0000","last_clean":"2024-04-23T17:00:21.702751+0000","last_became_active":"2024-04-23T16:57:47.589840+0000","last_became_peered":"2024-04-23T16:57:47.589840+0000","last_unstale":"2024-04-23T17:00:21.702751+0000","last_undegraded":"2024-04-23T17:00:21.702751+0000","last_fullsized":"2024-04-23T17:00:21.702751+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":3,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T17:00:21.702666+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T16:55:11.687173+0000","last_clean_scrub_stamp":"2024-04-23T17:00:21.702666+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:51.372877+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"255'139","reported_seq":2112,"reported_epoch":869,"state":"active+clean","last_fresh":"2024-04-23T17:00:25.580088+0000","last_change":"2024-04-23T17:00:14.290280+0000","last_active":"2024-04-23T17:00:25.580088+0000","last_peered":"2024-04-23T17:00:25.580088+0000","last_clean":"2024-04-23T17:00:25.580088+0000","last_became_active":"2024-04-23T16:28:20.044915+0000","last_became_peered":"2024-04-23T16:28:20.044915+0000","last_unstale":"2024-04-23T17:00:25.580088+0000","last_undegraded":"2024-04-23T17:00:25.580088+0000","last_fullsized":"2024-04-23T17:00:25.580088+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'139","last_scrub_stamp":"2024-04-23T17:00:14.290170+0000","last_deep_scrub":"255'139","last_deep_scrub_stamp":"2024-04-23T17:00:14.290170+0000","last_clean_scrub_stamp":"2024-04-23T17:00:14.290170+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:14.372346+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":804992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1474560,"data_stored":1460352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057183,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1087280,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":1086270,"kb_avail":92657872,"statfs":{"total":95995035648,"available":94881660928,"internally_reserved":0,"allocated":1019904,"data_stored":644310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1112340932},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351525,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76256,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93668896,"statfs":{"total":95995035648,"available":95916949504,"internally_reserved":0,"allocated":679936,"data_stored":310711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":77396416},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613162,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852444,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892708,"statfs":{"total":95995035648,"available":95122132992,"internally_reserved":0,"allocated":679936,"data_stored":310711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":872085952},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467842094,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":600092,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93145060,"statfs":{"total":95995035648,"available":95380541440,"internally_reserved":0,"allocated":1007616,"data_stored":638391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169142,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":739888,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93005264,"statfs":{"total":95995035648,"available":95237390336,"internally_reserved":0,"allocated":692224,"data_stored":316630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":756939210},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463481,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":605340,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93139812,"statfs":{"total":95995035648,"available":95375167488,"internally_reserved":0,"allocated":679936,"data_stored":310711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":619051462},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":271,"seq":1163936137621,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":624284,"kb_used_data":664,"kb_used_omap":231,"kb_used_meta":623256,"kb_avail":93120868,"statfs":{"total":95995035648,"available":95355768832,"internally_reserved":0,"allocated":679936,"data_stored":310711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":236656,"internal_metadata":638215056},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738967,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":749724,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":92995428,"statfs":{"total":95995035648,"available":95227318272,"internally_reserved":0,"allocated":679936,"data_stored":310711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":766900677},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":804,"seq":3453153706023,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":46904,"kb_used_data":876,"kb_used_omap":559,"kb_used_meta":45456,"kb_avail":93698248,"statfs":{"total":95995035648,"available":95947005952,"internally_reserved":0,"allocated":897024,"data_stored":517886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":572908,"internal_metadata":46547476},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":619,"seq":2658584756388,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":82716,"kb_used_data":664,"kb_used_omap":169,"kb_used_meta":81750,"kb_avail":93662436,"statfs":{"total":95995035648,"available":95910334464,"internally_reserved":0,"allocated":679936,"data_stored":310711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":173193,"internal_metadata":83712887},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":728,"seq":3126736191586,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":85980,"kb_used_data":664,"kb_used_omap":17,"kb_used_meta":85166,"kb_avail":93659172,"statfs":{"total":95995035648,"available":95906992128,"internally_reserved":0,"allocated":679936,"data_stored":310711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17598,"internal_metadata":87210818},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":681892,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93063260,"statfs":{"total":95995035648,"available":95296778240,"internally_reserved":0,"allocated":884736,"data_stored":511959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T17:00:29.111 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-23T17:00:29.111 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5d349943c59c9485df060d6adb0594f3940ec0eb shell --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-04-23T17:00:29.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:29 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T17:00:29.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:29 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T17:00:29.378 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T17:00:29.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:29 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T17:00:29.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:29 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T17:00:29.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:29 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T17:00:29.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:29 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T17:00:30.195 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:30 smithi103 ceph-mon[39629]: from='client.18234 v1:172.21.15.103:0/22796102' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T17:00:30.195 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:30 smithi103 ceph-mon[39629]: 1.8 scrub starts 2024-04-23T17:00:30.195 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:30 smithi103 ceph-mon[39629]: 1.8 scrub ok 2024-04-23T17:00:30.195 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:30 smithi103 ceph-mon[39629]: pgmap v2396: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:30.195 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:30 smithi103 ceph-mon[39629]: osdmap e870: 12 total, 12 up, 12 in 2024-04-23T17:00:30.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:30 smithi188 ceph-mon[44101]: from='client.18234 v1:172.21.15.103:0/22796102' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T17:00:30.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:30 smithi188 ceph-mon[44101]: 1.8 scrub starts 2024-04-23T17:00:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:30 smithi188 ceph-mon[44101]: 1.8 scrub ok 2024-04-23T17:00:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:30 smithi188 ceph-mon[44101]: pgmap v2396: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:30.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:30 smithi188 ceph-mon[44101]: osdmap e870: 12 total, 12 up, 12 in 2024-04-23T17:00:30.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:30 smithi148 ceph-mon[44520]: from='client.18234 v1:172.21.15.103:0/22796102' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T17:00:30.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:30 smithi148 ceph-mon[44520]: 1.8 scrub starts 2024-04-23T17:00:30.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:30 smithi148 ceph-mon[44520]: 1.8 scrub ok 2024-04-23T17:00:30.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:30 smithi148 ceph-mon[44520]: pgmap v2396: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:30.627 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:30 smithi148 ceph-mon[44520]: osdmap e870: 12 total, 12 up, 12 in 2024-04-23T17:00:31.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:31 smithi103 ceph-mon[39629]: purged_snaps scrub starts 2024-04-23T17:00:31.350 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:31 smithi103 ceph-mon[39629]: purged_snaps scrub ok 2024-04-23T17:00:31.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:31 smithi188 ceph-mon[44101]: purged_snaps scrub starts 2024-04-23T17:00:31.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:31 smithi188 ceph-mon[44101]: purged_snaps scrub ok 2024-04-23T17:00:31.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:31 smithi148 ceph-mon[44520]: purged_snaps scrub starts 2024-04-23T17:00:31.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:31 smithi148 ceph-mon[44520]: purged_snaps scrub ok 2024-04-23T17:00:32.165 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-23T17:00:32.175 INFO:tasks.cephadm:Teardown begin 2024-04-23T17:00:32.177 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T17:00:32.223 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T17:00:32.267 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T17:00:32.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:32 smithi188 ceph-mon[44101]: pgmap v2398: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:32.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:32 smithi188 ceph-mon[44101]: osdmap e871: 12 total, 12 up, 12 in 2024-04-23T17:00:32.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:32 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3245965823' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-23T17:00:32.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:32 smithi188 ceph-mon[44101]: from='client.? v1:172.21.15.103:0/3245965823' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-23T17:00:32.474 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:32 smithi188 ceph-mon[44101]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T17:00:32.490 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-23T17:00:32.491 DEBUG:teuthology.orchestra.run.smithi103:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T17:00:32.525 DEBUG:teuthology.orchestra.run.smithi148:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T17:00:32.550 DEBUG:teuthology.orchestra.run.smithi188:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T17:00:32.569 INFO:tasks.cephadm:Stopping all daemons... 2024-04-23T17:00:32.569 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-23T17:00:32.569 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mon.a 2024-04-23T17:00:32.595 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:32 smithi103 ceph-mon[39629]: pgmap v2398: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:32.595 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:32 smithi103 ceph-mon[39629]: osdmap e871: 12 total, 12 up, 12 in 2024-04-23T17:00:32.595 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:32 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3245965823' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-23T17:00:32.595 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:32 smithi103 ceph-mon[39629]: from='client.? v1:172.21.15.103:0/3245965823' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-23T17:00:32.595 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:32 smithi103 ceph-mon[39629]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T17:00:32.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:32 smithi148 ceph-mon[44520]: pgmap v2398: 11 pgs: 11 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T17:00:32.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:32 smithi148 ceph-mon[44520]: osdmap e871: 12 total, 12 up, 12 in 2024-04-23T17:00:32.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:32 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3245965823' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-23T17:00:32.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:32 smithi148 ceph-mon[44520]: from='client.? v1:172.21.15.103:0/3245965823' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-23T17:00:32.624 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:32 smithi148 ceph-mon[44520]: from='mgr.14154 v1:172.21.15.103:0/2546616362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T17:00:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:32 smithi103 systemd[1]: Stopping Ceph mon.a for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T17:00:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:32 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T17:00:32.814+0000 7f13609d2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T17:00:32.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:32 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a[39606]: 2024-04-23T17:00:32.814+0000 7f13609d2700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-23T17:00:33.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Apr 23 17:00:33 smithi103 bash[342943]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-a 2024-04-23T17:00:33.941 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mon.a.service' 2024-04-23T17:00:34.021 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T17:00:34.021 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-23T17:00:34.021 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-23T17:00:34.022 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mon.b 2024-04-23T17:00:34.334 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:34 smithi148 systemd[1]: Stopping Ceph mon.b for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T17:00:34.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:34 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-b[44497]: 2024-04-23T17:00:34.332+0000 7fc2955fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T17:00:34.623 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:34 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-b[44497]: 2024-04-23T17:00:34.332+0000 7fc2955fb700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-23T17:00:35.100 INFO:journalctl@ceph.mon.b.smithi148.stdout:Apr 23 17:00:34 smithi148 bash[184084]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-b 2024-04-23T17:00:35.107 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mon.b.service' 2024-04-23T17:00:35.148 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T17:00:35.148 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-23T17:00:35.148 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-23T17:00:35.148 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mon.c 2024-04-23T17:00:35.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:35 smithi188 systemd[1]: Stopping Ceph mon.c for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T17:00:35.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:35 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-c[44076]: 2024-04-23T17:00:35.663+0000 7fa39a208700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T17:00:35.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:35 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-c[44076]: 2024-04-23T17:00:35.663+0000 7fa39a208700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-23T17:00:36.473 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 17:00:36 smithi188 bash[179605]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mon-c 2024-04-23T17:00:36.759 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mon.c.service' 2024-04-23T17:00:36.820 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T17:00:36.820 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-23T17:00:36.820 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-23T17:00:36.820 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mgr.y 2024-04-23T17:00:37.116 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 17:00:36 smithi103 systemd[1]: Stopping Ceph mgr.y for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T17:00:37.434 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 17:00:37 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y[39876]: 2024-04-23T17:00:37.114+0000 7fa2ed502700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T17:00:37.850 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 23 17:00:37 smithi103 bash[343087]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-y 2024-04-23T17:00:38.005 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-23T17:00:38.040 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T17:00:38.041 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-23T17:00:38.041 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-23T17:00:38.041 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mgr.x 2024-04-23T17:00:38.373 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 17:00:38 smithi148 systemd[1]: Stopping Ceph mgr.x for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T17:00:39.021 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 17:00:38 smithi148 bash[184248]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-mgr-x 2024-04-23T17:00:39.179 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 17:00:39 smithi148 systemd[1]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-23T17:00:39.179 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Apr 23 17:00:39 smithi148 systemd[1]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-04-23T17:00:39.181 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-23T17:00:39.216 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T17:00:39.216 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-23T17:00:39.217 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-23T17:00:39.217 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.0 2024-04-23T17:00:39.599 INFO:journalctl@ceph.osd.0.smithi103.stdout:Apr 23 17:00:39 smithi103 systemd[1]: Stopping Ceph osd.0 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T17:00:39.600 INFO:journalctl@ceph.osd.0.smithi103.stdout:Apr 23 17:00:39 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-0[58506]: 2024-04-23T17:00:39.494+0000 7f2489700700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T17:00:39.600 INFO:journalctl@ceph.osd.0.smithi103.stdout:Apr 23 17:00:39 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-0[58506]: 2024-04-23T17:00:39.494+0000 7f2489700700 -1 osd.0 871 *** Got signal Terminated *** 2024-04-23T17:00:39.600 INFO:journalctl@ceph.osd.0.smithi103.stdout:Apr 23 17:00:39 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-0[58506]: 2024-04-23T17:00:39.494+0000 7f2489700700 -1 osd.0 871 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T17:00:45.599 INFO:journalctl@ceph.osd.0.smithi103.stdout:Apr 23 17:00:45 smithi103 bash[343232]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-0 2024-04-23T17:00:47.375 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.0.service' 2024-04-23T17:00:47.419 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T17:00:47.420 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-23T17:00:47.420 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-23T17:00:47.420 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.1 2024-04-23T17:00:47.757 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 17:00:47 smithi103 systemd[1]: Stopping Ceph osd.1 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T17:00:48.099 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 17:00:47 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-1[149775]: 2024-04-23T17:00:47.756+0000 7fd06a08c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T17:00:48.100 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 17:00:47 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-1[149775]: 2024-04-23T17:00:47.756+0000 7fd06a08c700 -1 osd.1 871 *** Got signal Terminated *** 2024-04-23T17:00:48.100 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 17:00:47 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-1[149775]: 2024-04-23T17:00:47.756+0000 7fd06a08c700 -1 osd.1 871 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T17:00:54.100 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 23 17:00:53 smithi103 bash[343502]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-1 2024-04-23T17:00:55.695 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.1.service' 2024-04-23T17:00:55.735 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T17:00:55.735 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-23T17:00:55.736 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-23T17:00:55.736 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.2 2024-04-23T17:00:56.043 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 17:00:55 smithi103 systemd[1]: Stopping Ceph osd.2 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T17:00:56.349 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 17:00:56 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[328149]: 2024-04-23T17:00:56.042+0000 7f346c63d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T17:00:56.349 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 17:00:56 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[328149]: 2024-04-23T17:00:56.042+0000 7f346c63d700 -1 osd.2 871 *** Got signal Terminated *** 2024-04-23T17:00:56.349 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 17:00:56 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2[328149]: 2024-04-23T17:00:56.042+0000 7f346c63d700 -1 osd.2 871 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T17:01:02.599 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 17:01:02 smithi103 bash[343762]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-2 2024-04-23T17:01:03.946 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.2.service' 2024-04-23T17:01:03.973 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 17:01:03 smithi103 systemd[1]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.2.service: Succeeded. 2024-04-23T17:01:03.974 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 23 17:01:03 smithi103 systemd[1]: Stopped Ceph osd.2 for 8e3594f2-018b-11ef-bc93-c7b262605968. 2024-04-23T17:01:03.985 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T17:01:03.986 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-23T17:01:03.986 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-23T17:01:03.986 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.3 2024-04-23T17:01:04.288 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 17:01:04 smithi103 systemd[1]: Stopping Ceph osd.3 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T17:01:04.599 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 17:01:04 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-3[269072]: 2024-04-23T17:01:04.286+0000 7f6bfa9a3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T17:01:04.599 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 17:01:04 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-3[269072]: 2024-04-23T17:01:04.286+0000 7f6bfa9a3700 -1 osd.3 871 *** Got signal Terminated *** 2024-04-23T17:01:04.599 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 17:01:04 smithi103 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-3[269072]: 2024-04-23T17:01:04.286+0000 7f6bfa9a3700 -1 osd.3 871 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T17:01:05.973 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:05 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:05.700+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:06.973 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:06 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:06.724+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:06.973 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:06 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:06.506+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:07.872 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:07 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:07.542+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:07.873 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:07 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:07.696+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:08.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:07 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:07.872+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:08.586 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:08 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:08.251+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:08.878 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:08 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:08.585+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:08.878 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:08 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:08.731+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:09.212 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:08 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:08.877+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:09.374 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:08 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:08.876+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:09.473 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:09 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:09.212+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:09.885 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:09 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:09.570+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:09.885 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:09 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:09.722+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:10.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:09 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:09.885+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:10.349 INFO:journalctl@ceph.osd.3.smithi103.stdout:Apr 23 17:01:10 smithi103 bash[344038]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-3 2024-04-23T17:01:10.373 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 17:01:09 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[173037]: 2024-04-23T17:01:09.973+0000 7f944232e700 -1 osd.4 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.621275+0000 front 2024-04-23T17:00:43.621416+0000 (oldest deadline 2024-04-23T17:01:09.521264+0000) 2024-04-23T17:01:10.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:09 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:09.883+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:10.528 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:10 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:10.237+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:10.923 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:10 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:10.528+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:10.924 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:10 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:10.754+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:11.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:10 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:10.923+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:11.373 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 17:01:10 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[173037]: 2024-04-23T17:01:10.972+0000 7f944232e700 -1 osd.4 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.621275+0000 front 2024-04-23T17:00:43.621416+0000 (oldest deadline 2024-04-23T17:01:09.521264+0000) 2024-04-23T17:01:11.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:10 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:10.901+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:11.718 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:11 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:11.524+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:11.719 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:11 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:11.282+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:11.909 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.3.service' 2024-04-23T17:01:11.950 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T17:01:11.950 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-23T17:01:11.951 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-23T17:01:11.951 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.4 2024-04-23T17:01:11.973 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:11 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:11.904+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:11.973 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:11 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:11.718+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:12.123 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:11 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:11.853+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:12.124 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 17:01:11 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[173037]: 2024-04-23T17:01:11.952+0000 7f944232e700 -1 osd.4 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.621275+0000 front 2024-04-23T17:00:43.621416+0000 (oldest deadline 2024-04-23T17:01:09.521264+0000) 2024-04-23T17:01:12.124 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 17:01:11 smithi148 systemd[1]: Stopping Ceph osd.4 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T17:01:12.623 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 17:01:12 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[173037]: 2024-04-23T17:01:12.250+0000 7f9449d42700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T17:01:12.623 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 17:01:12 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[173037]: 2024-04-23T17:01:12.250+0000 7f9449d42700 -1 osd.4 871 *** Got signal Terminated *** 2024-04-23T17:01:12.623 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 17:01:12 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[173037]: 2024-04-23T17:01:12.250+0000 7f9449d42700 -1 osd.4 871 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T17:01:12.723 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:12 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:12.480+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:12.723 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:12 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:12.288+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:13.123 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 17:01:12 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[173037]: 2024-04-23T17:01:12.904+0000 7f944232e700 -1 osd.4 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.621275+0000 front 2024-04-23T17:00:43.621416+0000 (oldest deadline 2024-04-23T17:01:09.521264+0000) 2024-04-23T17:01:13.123 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:12 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:12.810+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:13.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:12 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:12.859+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:13.223 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:12 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:12.749+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:13.712 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:13 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:13.499+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:13.713 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:13 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:13.334+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:13.713 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:13 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:13.712+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:13.973 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:13 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:13.903+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:14.123 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 17:01:13 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[173037]: 2024-04-23T17:01:13.931+0000 7f944232e700 -1 osd.4 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.621275+0000 front 2024-04-23T17:00:43.621416+0000 (oldest deadline 2024-04-23T17:01:09.521264+0000) 2024-04-23T17:01:14.123 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:13 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:13.846+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:14.710 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:14 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:14.528+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:14.710 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:14 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:14.528+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:14.711 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:14 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:14.309+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:14.711 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:14 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:14.309+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:14.973 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:14 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:14.918+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:14.973 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:14 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:14.709+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:15.123 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 17:01:14 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[173037]: 2024-04-23T17:01:14.973+0000 7f944232e700 -1 osd.4 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.621275+0000 front 2024-04-23T17:00:43.621416+0000 (oldest deadline 2024-04-23T17:01:09.521264+0000) 2024-04-23T17:01:15.123 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:14 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:14.861+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:15.689 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:15 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:15.505+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:15.689 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:15 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:15.505+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:15.689 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:15 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:15.265+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:15.689 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:15 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:15.265+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:15.973 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:15 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:15.687+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:15.973 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:15 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:15.886+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:15.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:15 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:15.886+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:16.123 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:15 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:15.867+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:16.124 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 17:01:15 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[173037]: 2024-04-23T17:01:15.975+0000 7f944232e700 -1 osd.4 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.621275+0000 front 2024-04-23T17:00:43.621416+0000 (oldest deadline 2024-04-23T17:01:09.521264+0000) 2024-04-23T17:01:16.124 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 17:01:15 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[173037]: 2024-04-23T17:01:15.975+0000 7f944232e700 -1 osd.4 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:51.822619+0000 front 2024-04-23T17:00:51.822638+0000 (oldest deadline 2024-04-23T17:01:15.322608+0000) 2024-04-23T17:01:16.124 INFO:journalctl@ceph.osd.5.smithi148.stdout:Apr 23 17:01:15 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-5[56866]: 2024-04-23T17:01:15.783+0000 7f8e09e87700 -1 osd.5 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:51.712154+0000 front 2024-04-23T17:00:51.712134+0000 (oldest deadline 2024-04-23T17:01:15.212200+0000) 2024-04-23T17:01:16.549 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:16 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:16.270+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:16.549 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:16 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:16.270+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:16.623 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:16 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:16.306+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.139956+0000 front 2024-04-23T17:00:50.140150+0000 (oldest deadline 2024-04-23T17:01:16.040027+0000) 2024-04-23T17:01:16.923 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:16 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:16.730+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:16.923 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:16 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:16.548+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:16.923 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:16 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:16.548+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:17.123 INFO:journalctl@ceph.osd.5.smithi148.stdout:Apr 23 17:01:16 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-5[56866]: 2024-04-23T17:01:16.765+0000 7f8e09e87700 -1 osd.5 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:51.712154+0000 front 2024-04-23T17:00:51.712134+0000 (oldest deadline 2024-04-23T17:01:15.212200+0000) 2024-04-23T17:01:17.123 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 17:01:16 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[173037]: 2024-04-23T17:01:16.997+0000 7f944232e700 -1 osd.4 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.621275+0000 front 2024-04-23T17:00:43.621416+0000 (oldest deadline 2024-04-23T17:01:09.521264+0000) 2024-04-23T17:01:17.124 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 17:01:16 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4[173037]: 2024-04-23T17:01:16.997+0000 7f944232e700 -1 osd.4 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:51.822619+0000 front 2024-04-23T17:00:51.822638+0000 (oldest deadline 2024-04-23T17:01:15.322608+0000) 2024-04-23T17:01:17.124 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:16 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:16.821+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:17.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:16 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:16.922+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:17.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:16 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:16.922+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:17.623 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:17 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:17.277+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.139956+0000 front 2024-04-23T17:00:50.140150+0000 (oldest deadline 2024-04-23T17:01:16.040027+0000) 2024-04-23T17:01:17.723 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:17 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:17.557+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:17.723 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:17 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:17.557+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:17.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:17 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:17.309+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:17.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:17 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:17.309+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:18.119 INFO:journalctl@ceph.osd.5.smithi148.stdout:Apr 23 17:01:17 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-5[56866]: 2024-04-23T17:01:17.741+0000 7f8e09e87700 -1 osd.5 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:51.712154+0000 front 2024-04-23T17:00:51.712134+0000 (oldest deadline 2024-04-23T17:01:15.212200+0000) 2024-04-23T17:01:18.120 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:17 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:17.831+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:18.223 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:17 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:17.748+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:18.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:17 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:17.893+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:18.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:17 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:17.893+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:18.373 INFO:journalctl@ceph.osd.4.smithi148.stdout:Apr 23 17:01:18 smithi148 bash[184411]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-4 2024-04-23T17:01:18.373 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:18 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:18.283+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.139956+0000 front 2024-04-23T17:00:50.140150+0000 (oldest deadline 2024-04-23T17:01:16.040027+0000) 2024-04-23T17:01:18.723 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:18 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:18.553+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:18.723 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:18 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:18.553+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:18.723 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:18 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:18.316+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:18.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:18 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:18.316+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:19.123 INFO:journalctl@ceph.osd.5.smithi148.stdout:Apr 23 17:01:18 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-5[56866]: 2024-04-23T17:01:18.752+0000 7f8e09e87700 -1 osd.5 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:51.712154+0000 front 2024-04-23T17:00:51.712134+0000 (oldest deadline 2024-04-23T17:01:15.212200+0000) 2024-04-23T17:01:19.123 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:18 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:18.856+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:19.123 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:18 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:18.856+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:19.223 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:18 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:18.748+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:19.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:18 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:18.873+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:19.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:18 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:18.873+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:19.539 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:19 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:19.247+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.139956+0000 front 2024-04-23T17:00:50.140150+0000 (oldest deadline 2024-04-23T17:01:16.040027+0000) 2024-04-23T17:01:19.599 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:19 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:19.276+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:19.599 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:19 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:19.276+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:19.852 INFO:journalctl@ceph.osd.5.smithi148.stdout:Apr 23 17:01:19 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-5[56866]: 2024-04-23T17:01:19.744+0000 7f8e09e87700 -1 osd.5 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:51.712154+0000 front 2024-04-23T17:00:51.712134+0000 (oldest deadline 2024-04-23T17:01:15.212200+0000) 2024-04-23T17:01:19.867 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:19 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:19.699+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:19.867 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:19 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:19.594+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:19.867 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:19 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:19.594+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:20.123 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:19 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:19.889+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:20.123 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:19 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:19.889+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:20.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:19 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:19.866+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:20.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:19 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:19.866+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:20.263 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.4.service' 2024-04-23T17:01:20.305 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T17:01:20.305 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-23T17:01:20.305 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-23T17:01:20.305 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.5 2024-04-23T17:01:20.567 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:20 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:20.283+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:20.567 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:20 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:20.283+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:20.622 INFO:journalctl@ceph.osd.5.smithi148.stdout:Apr 23 17:01:20 smithi148 systemd[1]: Stopping Ceph osd.5 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T17:01:20.622 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:20 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:20.250+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.139956+0000 front 2024-04-23T17:00:50.140150+0000 (oldest deadline 2024-04-23T17:01:16.040027+0000) 2024-04-23T17:01:20.852 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:20 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:20.566+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:20.852 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:20 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:20.566+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:20.852 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:20 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:20.668+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:20.873 INFO:journalctl@ceph.osd.5.smithi148.stdout:Apr 23 17:01:20 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-5[56866]: 2024-04-23T17:01:20.619+0000 7f8e0d893700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T17:01:20.873 INFO:journalctl@ceph.osd.5.smithi148.stdout:Apr 23 17:01:20 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-5[56866]: 2024-04-23T17:01:20.619+0000 7f8e0d893700 -1 osd.5 871 *** Got signal Terminated *** 2024-04-23T17:01:20.873 INFO:journalctl@ceph.osd.5.smithi148.stdout:Apr 23 17:01:20 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-5[56866]: 2024-04-23T17:01:20.619+0000 7f8e0d893700 -1 osd.5 871 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T17:01:20.873 INFO:journalctl@ceph.osd.5.smithi148.stdout:Apr 23 17:01:20 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-5[56866]: 2024-04-23T17:01:20.791+0000 7f8e09e87700 -1 osd.5 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:51.712154+0000 front 2024-04-23T17:00:51.712134+0000 (oldest deadline 2024-04-23T17:01:15.212200+0000) 2024-04-23T17:01:21.218 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:20 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:20.887+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:21.219 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:20 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:20.887+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:21.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:20 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:20.850+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:21.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:20 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:20.850+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:21.594 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:21 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:21.294+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:21.594 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:21 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:21.294+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:21.623 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:21 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:21.218+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.139956+0000 front 2024-04-23T17:00:50.140150+0000 (oldest deadline 2024-04-23T17:01:16.040027+0000) 2024-04-23T17:01:21.870 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:21 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:21.644+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:21.870 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:21 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:21.593+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:21.870 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:21 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:21.593+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:22.123 INFO:journalctl@ceph.osd.5.smithi148.stdout:Apr 23 17:01:21 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-5[56866]: 2024-04-23T17:01:21.830+0000 7f8e09e87700 -1 osd.5 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:51.712154+0000 front 2024-04-23T17:00:51.712134+0000 (oldest deadline 2024-04-23T17:01:15.212200+0000) 2024-04-23T17:01:22.123 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:21 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:21.897+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:22.123 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:21 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:21.897+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:22.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:21 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:21.869+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:22.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:21 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:21.869+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:22.618 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:22 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:22.551+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:22.618 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:22 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:22.551+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:22.619 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:22 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:22.336+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:22.619 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:22 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:22.336+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:22.623 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:22 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:22.219+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.139956+0000 front 2024-04-23T17:00:50.140150+0000 (oldest deadline 2024-04-23T17:01:16.040027+0000) 2024-04-23T17:01:22.902 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:22 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:22.617+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:23.123 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:22 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:22.938+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:23.123 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:22 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:22.938+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:23.124 INFO:journalctl@ceph.osd.5.smithi148.stdout:Apr 23 17:01:22 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-5[56866]: 2024-04-23T17:01:22.870+0000 7f8e09e87700 -1 osd.5 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:51.712154+0000 front 2024-04-23T17:00:51.712134+0000 (oldest deadline 2024-04-23T17:01:15.212200+0000) 2024-04-23T17:01:23.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:22 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:22.901+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:23.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:22 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:22.901+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:23.591 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:23 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:23.319+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:23.591 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:23 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:23.319+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:23.623 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:23 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:23.204+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.139956+0000 front 2024-04-23T17:00:50.140150+0000 (oldest deadline 2024-04-23T17:01:16.040027+0000) 2024-04-23T17:01:23.623 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:23 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:23.204+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:00:58.341330+0000 front 2024-04-23T17:00:58.341406+0000 (oldest deadline 2024-04-23T17:01:22.441435+0000) 2024-04-23T17:01:23.903 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:23 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:23.590+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:23.903 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:23 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:23.590+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:23.903 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:23 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:23.650+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:24.124 INFO:journalctl@ceph.osd.5.smithi148.stdout:Apr 23 17:01:23 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-5[56866]: 2024-04-23T17:01:23.860+0000 7f8e09e87700 -1 osd.5 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:51.712154+0000 front 2024-04-23T17:00:51.712134+0000 (oldest deadline 2024-04-23T17:01:15.212200+0000) 2024-04-23T17:01:24.124 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:23 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:23.958+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:24.124 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:23 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:23.958+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:24.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:23 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:23.901+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:24.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:23 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:23.901+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:24.610 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:24 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:24.341+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:24.611 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:24 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:24.341+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:24.623 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:24 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:24.207+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.139956+0000 front 2024-04-23T17:00:50.140150+0000 (oldest deadline 2024-04-23T17:01:16.040027+0000) 2024-04-23T17:01:24.624 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:24 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:24.207+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:00:58.341330+0000 front 2024-04-23T17:00:58.341406+0000 (oldest deadline 2024-04-23T17:01:22.441435+0000) 2024-04-23T17:01:24.878 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:24 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:24.609+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:24.878 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:24 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:24.611+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:24.879 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:24 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:24.611+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:25.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:24 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:24.877+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:25.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:24 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:24.877+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:25.253 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:25 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:25.007+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:25.253 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:25 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:25.007+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:25.253 INFO:journalctl@ceph.osd.5.smithi148.stdout:Apr 23 17:01:24 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-5[56866]: 2024-04-23T17:01:24.892+0000 7f8e09e87700 -1 osd.5 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:51.712154+0000 front 2024-04-23T17:00:51.712134+0000 (oldest deadline 2024-04-23T17:01:15.212200+0000) 2024-04-23T17:01:25.623 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:25 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:25.253+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.139956+0000 front 2024-04-23T17:00:50.140150+0000 (oldest deadline 2024-04-23T17:01:16.040027+0000) 2024-04-23T17:01:25.624 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:25 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:25.253+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:00:58.341330+0000 front 2024-04-23T17:00:58.341406+0000 (oldest deadline 2024-04-23T17:01:22.441435+0000) 2024-04-23T17:01:25.628 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:25 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:25.572+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:25.628 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:25 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:25.572+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:25.628 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:25 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:25.336+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:25.628 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:25 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:25.336+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:25.973 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:25 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:25.626+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:25.973 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:25 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:25.871+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:25.973 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:25 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:25.871+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:26.367 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:26 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:26.249+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.139956+0000 front 2024-04-23T17:00:50.140150+0000 (oldest deadline 2024-04-23T17:01:16.040027+0000) 2024-04-23T17:01:26.367 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:26 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:26.249+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:00:58.341330+0000 front 2024-04-23T17:00:58.341406+0000 (oldest deadline 2024-04-23T17:01:22.441435+0000) 2024-04-23T17:01:26.367 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:26 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:26.040+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:26.367 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:26 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:26.040+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:26.595 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:26 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:26.544+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:26.596 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:26 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:26.544+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:26.596 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:26 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:26.298+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:26.596 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:26 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:26.298+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:26.623 INFO:journalctl@ceph.osd.5.smithi148.stdout:Apr 23 17:01:26 smithi148 bash[184687]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-5 2024-04-23T17:01:26.973 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:26 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:26.594+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:26.973 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:26 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:26.594+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:26.973 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:26 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:26.825+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:26.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:26 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:26.825+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:27.373 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:27 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:27.200+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.139956+0000 front 2024-04-23T17:00:50.140150+0000 (oldest deadline 2024-04-23T17:01:16.040027+0000) 2024-04-23T17:01:27.373 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:27 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:27.200+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:00:58.341330+0000 front 2024-04-23T17:00:58.341406+0000 (oldest deadline 2024-04-23T17:01:22.441435+0000) 2024-04-23T17:01:27.374 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:27 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:27.009+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:27.374 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:27 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:27.009+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:27.558 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:27 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:27.249+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:27.559 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:27 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:27.249+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:27.973 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:27 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:27.580+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:27.973 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:27 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:27.580+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:27.974 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:27 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:27.558+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:27.974 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:27 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:27.558+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:27.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:27 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:27.799+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:27.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:27 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:27.799+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:28.232 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:28 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:28.152+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.139956+0000 front 2024-04-23T17:00:50.140150+0000 (oldest deadline 2024-04-23T17:01:16.040027+0000) 2024-04-23T17:01:28.232 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:28 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:28.152+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:00:58.341330+0000 front 2024-04-23T17:00:58.341406+0000 (oldest deadline 2024-04-23T17:01:22.441435+0000) 2024-04-23T17:01:28.232 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:27 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:27.963+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:28.233 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:27 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:27.963+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:28.240 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.5.service' 2024-04-23T17:01:28.284 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T17:01:28.284 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-23T17:01:28.284 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-23T17:01:28.285 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.6 2024-04-23T17:01:28.551 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:28 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:28.244+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:28.551 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:28 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:28.244+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:28.597 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:28 smithi148 systemd[1]: Stopping Ceph osd.6 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T17:01:28.818 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:28 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:28.559+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:28.818 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:28 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:28.559+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:28.818 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:28 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:28.550+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:28.818 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:28 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:28.550+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:28.873 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:28 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:28.595+0000 7f5d5df6e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T17:01:28.873 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:28 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:28.595+0000 7f5d5df6e700 -1 osd.6 871 *** Got signal Terminated *** 2024-04-23T17:01:28.873 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:28 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:28.595+0000 7f5d5df6e700 -1 osd.6 871 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T17:01:29.196 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:28 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:28.817+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:29.197 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:28 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:28.817+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:29.373 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:29 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:29.129+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.139956+0000 front 2024-04-23T17:00:50.140150+0000 (oldest deadline 2024-04-23T17:01:16.040027+0000) 2024-04-23T17:01:29.373 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:29 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:29.129+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:00:58.341330+0000 front 2024-04-23T17:00:58.341406+0000 (oldest deadline 2024-04-23T17:01:22.441435+0000) 2024-04-23T17:01:29.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:28 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:28.936+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:29.374 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:28 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:28.936+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:29.473 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:29 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:29.196+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:29.473 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:29 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:29.196+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:29.821 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:29 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:29.561+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:29.822 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:29 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:29.561+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:29.822 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:29 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:29.557+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:29.822 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:29 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:29.557+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:30.214 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:29 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:29.820+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:30.214 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:29 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:29.820+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:30.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:29 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:29.963+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:30.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:29 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:29.963+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:30.373 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:30 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:30.157+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.139956+0000 front 2024-04-23T17:00:50.140150+0000 (oldest deadline 2024-04-23T17:01:16.040027+0000) 2024-04-23T17:01:30.374 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:30 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:30.157+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:00:58.341330+0000 front 2024-04-23T17:00:58.341406+0000 (oldest deadline 2024-04-23T17:01:22.441435+0000) 2024-04-23T17:01:30.374 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:30 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:30.157+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.843351+0000 front 2024-04-23T17:01:08.843438+0000 (oldest deadline 2024-04-23T17:01:29.943409+0000) 2024-04-23T17:01:30.473 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:30 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:30.213+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:30.473 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:30 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:30.213+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:30.973 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:30 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:30.586+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:30.973 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:30 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:30.586+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:30.974 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:30 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:30.559+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:30.974 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:30 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:30.559+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:30.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:30 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:30.802+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:30.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:30 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:30.802+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:31.373 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:31 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:31.168+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.139956+0000 front 2024-04-23T17:00:50.140150+0000 (oldest deadline 2024-04-23T17:01:16.040027+0000) 2024-04-23T17:01:31.373 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:31 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:31.168+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:00:58.341330+0000 front 2024-04-23T17:00:58.341406+0000 (oldest deadline 2024-04-23T17:01:22.441435+0000) 2024-04-23T17:01:31.373 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:31 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:31.168+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.843351+0000 front 2024-04-23T17:01:08.843438+0000 (oldest deadline 2024-04-23T17:01:29.943409+0000) 2024-04-23T17:01:31.374 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:30 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:30.946+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:31.374 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:30 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:30.946+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:31.550 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:31 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:31.251+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:31.550 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:31 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:31.251+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:31.815 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:31 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:31.589+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:31.815 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:31 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:31.589+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:31.815 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:31 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:31.549+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:31.815 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:31 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:31.549+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:32.179 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:31 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:31.925+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:32.179 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:31 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:31.925+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:32.179 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:31 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:31.925+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:06.986651+0000 front 2024-04-23T17:01:06.986687+0000 (oldest deadline 2024-04-23T17:01:31.086604+0000) 2024-04-23T17:01:32.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:31 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:31.813+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:32.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:31 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:31.813+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:32.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:31 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:31.813+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:32.571 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:32 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:32.242+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:32.571 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:32 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:32.242+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:32.623 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:32 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:32.177+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.139956+0000 front 2024-04-23T17:00:50.140150+0000 (oldest deadline 2024-04-23T17:01:16.040027+0000) 2024-04-23T17:01:32.623 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:32 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:32.177+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:00:58.341330+0000 front 2024-04-23T17:00:58.341406+0000 (oldest deadline 2024-04-23T17:01:22.441435+0000) 2024-04-23T17:01:32.623 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:32 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:32.177+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.843351+0000 front 2024-04-23T17:01:08.843438+0000 (oldest deadline 2024-04-23T17:01:29.943409+0000) 2024-04-23T17:01:32.844 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:32 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:32.582+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:32.844 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:32 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:32.582+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:32.844 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:32 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:32.570+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:32.845 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:32 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:32.570+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:32.845 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:32 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:32.570+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:33.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:32 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:32.843+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:33.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:32 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:32.843+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:33.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:32 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:32.843+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:33.373 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:33 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:33.148+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.139956+0000 front 2024-04-23T17:00:50.140150+0000 (oldest deadline 2024-04-23T17:01:16.040027+0000) 2024-04-23T17:01:33.373 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:33 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:33.148+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:00:58.341330+0000 front 2024-04-23T17:00:58.341406+0000 (oldest deadline 2024-04-23T17:01:22.441435+0000) 2024-04-23T17:01:33.373 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:33 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6[63958]: 2024-04-23T17:01:33.148+0000 7f5d5655a700 -1 osd.6 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.843351+0000 front 2024-04-23T17:01:08.843438+0000 (oldest deadline 2024-04-23T17:01:29.943409+0000) 2024-04-23T17:01:33.374 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:32 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:32.972+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:33.374 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:32 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:32.972+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:33.374 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:32 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:32.972+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:06.986651+0000 front 2024-04-23T17:01:06.986687+0000 (oldest deadline 2024-04-23T17:01:31.086604+0000) 2024-04-23T17:01:33.568 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:33 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:33.223+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:33.568 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:33 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:33.223+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:33.834 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:33 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:33.567+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:33.834 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:33 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:33.567+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:33.834 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:33 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:33.570+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:33.835 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:33 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:33.570+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:33.835 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:33 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:33.570+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:34.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:33 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:33.833+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:34.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:33 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:33.833+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:34.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:33 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:33.833+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:34.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:34 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:34.001+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:34.374 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:34 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:34.001+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:34.374 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:34 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:34.001+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:06.986651+0000 front 2024-04-23T17:01:06.986687+0000 (oldest deadline 2024-04-23T17:01:31.086604+0000) 2024-04-23T17:01:34.556 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:34 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:34.233+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:34.556 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:34 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:34.233+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:34.841 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:34 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:34.555+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:34.841 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:34 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:34.555+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:34.841 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:34 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:34.555+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:34.842 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:34 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:34.605+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:34.842 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:34 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:34.605+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:34.873 INFO:journalctl@ceph.osd.6.smithi148.stdout:Apr 23 17:01:34 smithi148 bash[184950]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-6 2024-04-23T17:01:35.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:34 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:34.840+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:35.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:34 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:34.840+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:35.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:34 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:34.840+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:35.229 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:34 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:34.969+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:35.229 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:34 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:34.969+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:35.229 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:34 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:34.969+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:06.986651+0000 front 2024-04-23T17:01:06.986687+0000 (oldest deadline 2024-04-23T17:01:31.086604+0000) 2024-04-23T17:01:35.572 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:35 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:35.228+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:35.573 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:35 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:35.228+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:35.879 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:35 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:35.572+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:35.879 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:35 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:35.572+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:35.879 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:35 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:35.572+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:35.879 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:35 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:35.623+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:35.879 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:35 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:35.623+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:35.879 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:35 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:35.623+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.877287+0000 front 2024-04-23T17:01:05.877308+0000 (oldest deadline 2024-04-23T17:01:35.277703+0000) 2024-04-23T17:01:36.208 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:35 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:35.878+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:36.208 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:35 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:35.878+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:36.208 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:35 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:35.878+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:36.275 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:36 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:36.018+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:36.275 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:36 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:36.018+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:36.275 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:36 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:36.018+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:06.986651+0000 front 2024-04-23T17:01:06.986687+0000 (oldest deadline 2024-04-23T17:01:31.086604+0000) 2024-04-23T17:01:36.282 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.6.service' 2024-04-23T17:01:36.321 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T17:01:36.322 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-23T17:01:36.322 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-23T17:01:36.322 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.7 2024-04-23T17:01:36.473 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:36 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:36.207+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:36.473 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:36 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:36.207+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:36.473 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:36 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:36.207+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:36.624 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:36 smithi148 systemd[1]: Stopping Ceph osd.7 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T17:01:36.830 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:36 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:36.592+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:36.831 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:36 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:36.592+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:36.831 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:36 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:36.592+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.877287+0000 front 2024-04-23T17:01:05.877308+0000 (oldest deadline 2024-04-23T17:01:35.277703+0000) 2024-04-23T17:01:36.831 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:36 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:36.560+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:36.831 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:36 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:36.560+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:36.831 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:36 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:36.560+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:36.986 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:36 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:36.632+0000 7f53a6492700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T17:01:36.986 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:36 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:36.632+0000 7f53a6492700 -1 osd.7 871 *** Got signal Terminated *** 2024-04-23T17:01:36.986 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:36 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:36.632+0000 7f53a6492700 -1 osd.7 871 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T17:01:36.986 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:36 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:36.985+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:36.986 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:36 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:36.985+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:37.188 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:36 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:36.829+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:37.188 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:36 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:36.829+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:37.189 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:36 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:36.829+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:37.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:36 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:36.985+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:06.986651+0000 front 2024-04-23T17:01:06.986687+0000 (oldest deadline 2024-04-23T17:01:31.086604+0000) 2024-04-23T17:01:37.473 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:37 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:37.188+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:37.473 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:37 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:37.188+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:37.474 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:37 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:37.188+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:37.879 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:37 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:37.523+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:37.879 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:37 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:37.523+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:37.879 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:37 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:37.523+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:37.879 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:37 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:37.581+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:37.879 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:37 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:37.581+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:37.880 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:37 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:37.581+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.877287+0000 front 2024-04-23T17:01:05.877308+0000 (oldest deadline 2024-04-23T17:01:35.277703+0000) 2024-04-23T17:01:38.192 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:37 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:37.878+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:38.193 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:37 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:37.878+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:38.193 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:37 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:37.878+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:38.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:37 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:37.940+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:38.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:37 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:37.940+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:38.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:37 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:37.940+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:06.986651+0000 front 2024-04-23T17:01:06.986687+0000 (oldest deadline 2024-04-23T17:01:31.086604+0000) 2024-04-23T17:01:38.473 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:38 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:38.192+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:38.473 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:38 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:38.192+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:38.473 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:38 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:38.192+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:38.473 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:38 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:38.192+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:38.860 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:38 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:38.500+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:38.860 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:38 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:38.500+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:38.860 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:38 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:38.500+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:38.860 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:38 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:38.584+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:38.860 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:38 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:38.584+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:38.861 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:38 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:38.584+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.877287+0000 front 2024-04-23T17:01:05.877308+0000 (oldest deadline 2024-04-23T17:01:35.277703+0000) 2024-04-23T17:01:39.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:38 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:38.859+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:39.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:38 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:38.859+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:39.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:38 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:38.859+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:39.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:38 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:38.951+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:39.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:38 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:38.951+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:39.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:38 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:38.951+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:06.986651+0000 front 2024-04-23T17:01:06.986687+0000 (oldest deadline 2024-04-23T17:01:31.086604+0000) 2024-04-23T17:01:39.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:38 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:38.951+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.088306+0000 front 2024-04-23T17:01:15.088279+0000 (oldest deadline 2024-04-23T17:01:37.988515+0000) 2024-04-23T17:01:39.532 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:39 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:39.241+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:39.532 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:39 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:39.241+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:39.532 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:39 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:39.241+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:39.532 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:39 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:39.241+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:39.879 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:39 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:39.531+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:39.879 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:39 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:39.531+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:39.879 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:39 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:39.531+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:39.879 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:39 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:39.531+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:39.879 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:39 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:39.580+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:39.879 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:39 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:39.580+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:39.879 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:39 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:39.580+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.877287+0000 front 2024-04-23T17:01:05.877308+0000 (oldest deadline 2024-04-23T17:01:35.277703+0000) 2024-04-23T17:01:40.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:39 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:39.878+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:40.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:39 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:39.878+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:40.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:39 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:39.878+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:40.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:39 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:39.878+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:40.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:39 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:39.954+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:40.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:39 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:39.954+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:40.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:39 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:39.954+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:06.986651+0000 front 2024-04-23T17:01:06.986687+0000 (oldest deadline 2024-04-23T17:01:31.086604+0000) 2024-04-23T17:01:40.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:39 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:39.954+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.088306+0000 front 2024-04-23T17:01:15.088279+0000 (oldest deadline 2024-04-23T17:01:37.988515+0000) 2024-04-23T17:01:40.485 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:40 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:40.228+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:40.485 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:40 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:40.228+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:40.486 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:40 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:40.228+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:40.486 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:40 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:40.228+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:40.863 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:40 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:40.586+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:40.863 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:40 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:40.586+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:40.863 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:40 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:40.586+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.877287+0000 front 2024-04-23T17:01:05.877308+0000 (oldest deadline 2024-04-23T17:01:35.277703+0000) 2024-04-23T17:01:40.863 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:40 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:40.586+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.278371+0000 front 2024-04-23T17:01:15.278414+0000 (oldest deadline 2024-04-23T17:01:39.978224+0000) 2024-04-23T17:01:40.863 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:40 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:40.484+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:40.863 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:40 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:40.484+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:40.863 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:40 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:40.484+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:40.864 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:40 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:40.484+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:41.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:40 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:40.862+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:41.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:40 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:40.862+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:41.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:40 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:40.862+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:41.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:40 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:40.862+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:41.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:40 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:40.912+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.582950+0000 front 2024-04-23T17:00:43.582918+0000 (oldest deadline 2024-04-23T17:01:08.282987+0000) 2024-04-23T17:01:41.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:40 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:40.912+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.384166+0000 front 2024-04-23T17:00:52.384150+0000 (oldest deadline 2024-04-23T17:01:18.284237+0000) 2024-04-23T17:01:41.373 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:40 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:40.912+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:06.986651+0000 front 2024-04-23T17:01:06.986687+0000 (oldest deadline 2024-04-23T17:01:31.086604+0000) 2024-04-23T17:01:41.374 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:40 smithi148 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7[71369]: 2024-04-23T17:01:40.912+0000 7f53a2a86700 -1 osd.7 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.088306+0000 front 2024-04-23T17:01:15.088279+0000 (oldest deadline 2024-04-23T17:01:37.988515+0000) 2024-04-23T17:01:41.584 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:41 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:41.237+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:41.584 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:41 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:41.237+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:41.584 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:41 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:41.237+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:41.584 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:41 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:41.237+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:41.585 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:41 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:41.449+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:41.585 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:41 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:41.449+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:41.585 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:41 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:41.449+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:41.585 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:41 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:41.449+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:41.873 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:41 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:41.583+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:41.874 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:41 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:41.583+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:41.874 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:41 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:41.583+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.877287+0000 front 2024-04-23T17:01:05.877308+0000 (oldest deadline 2024-04-23T17:01:35.277703+0000) 2024-04-23T17:01:41.874 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:41 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:41.583+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.278371+0000 front 2024-04-23T17:01:15.278414+0000 (oldest deadline 2024-04-23T17:01:39.978224+0000) 2024-04-23T17:01:42.195 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:41 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:41.873+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:42.195 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:41 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:41.873+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:42.195 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:41 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:41.873+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:42.195 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:41 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:41.873+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:42.473 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:42 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:42.419+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:42.473 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:42 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:42.419+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:42.473 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:42 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:42.419+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:42.474 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:42 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:42.419+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:42.474 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:42 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:42.194+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:42.474 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:42 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:42.194+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:42.474 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:42 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:42.194+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:42.474 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:42 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:42.194+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:42.810 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:42 smithi148 bash[185210]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-7 2024-04-23T17:01:42.888 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:42 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:42.611+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:42.888 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:42 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:42.611+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:42.888 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:42 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:42.611+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.877287+0000 front 2024-04-23T17:01:05.877308+0000 (oldest deadline 2024-04-23T17:01:35.277703+0000) 2024-04-23T17:01:42.888 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:42 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:42.611+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.278371+0000 front 2024-04-23T17:01:15.278414+0000 (oldest deadline 2024-04-23T17:01:39.978224+0000) 2024-04-23T17:01:43.215 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:42 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:42.886+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:43.215 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:42 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:42.886+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:43.215 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:42 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:42.886+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:43.215 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:42 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:42.886+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:43.473 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:43 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:43.214+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:43.473 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:43 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:43.214+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:43.473 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:43 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:43.214+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:43.473 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:43 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:43.214+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:43.474 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:43 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:43.449+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:43.474 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:43 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:43.449+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:43.474 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:43 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:43.449+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:43.474 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:43 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:43.449+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:43.973 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:43 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:43.622+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:43.973 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:43 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:43.622+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:43.974 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:43 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:43.622+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.877287+0000 front 2024-04-23T17:01:05.877308+0000 (oldest deadline 2024-04-23T17:01:35.277703+0000) 2024-04-23T17:01:43.974 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:43 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:43.622+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.278371+0000 front 2024-04-23T17:01:15.278414+0000 (oldest deadline 2024-04-23T17:01:39.978224+0000) 2024-04-23T17:01:43.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:43 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:43.845+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:43.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:43 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:43.845+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:43.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:43 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:43.845+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:43.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:43 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:43.845+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:44.180 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.7.service' 2024-04-23T17:01:44.207 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:44 smithi148 systemd[1]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.7.service: Succeeded. 2024-04-23T17:01:44.208 INFO:journalctl@ceph.osd.7.smithi148.stdout:Apr 23 17:01:44 smithi148 systemd[1]: Stopped Ceph osd.7 for 8e3594f2-018b-11ef-bc93-c7b262605968. 2024-04-23T17:01:44.219 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T17:01:44.220 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-23T17:01:44.220 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-23T17:01:44.220 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.8 2024-04-23T17:01:44.508 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:44 smithi188 systemd[1]: Stopping Ceph osd.8 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T17:01:44.509 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:44 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:44.246+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:44.509 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:44 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:44.246+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:44.509 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:44 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:44.246+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:44.509 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:44 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:44.246+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:44.509 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:44 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:44.471+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:44.509 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:44 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:44.471+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:44.509 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:44 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:44.471+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:44.509 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:44 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:44.471+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:44.892 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:44 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:44.507+0000 7fe9a546e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T17:01:44.892 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:44 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:44.507+0000 7fe9a546e700 -1 osd.8 871 *** Got signal Terminated *** 2024-04-23T17:01:44.892 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:44 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:44.507+0000 7fe9a546e700 -1 osd.8 871 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T17:01:44.892 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:44 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:44.646+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:44.892 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:44 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:44.646+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:44.892 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:44 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:44.646+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.877287+0000 front 2024-04-23T17:01:05.877308+0000 (oldest deadline 2024-04-23T17:01:35.277703+0000) 2024-04-23T17:01:44.892 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:44 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:44.646+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.278371+0000 front 2024-04-23T17:01:15.278414+0000 (oldest deadline 2024-04-23T17:01:39.978224+0000) 2024-04-23T17:01:45.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:44 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:44.891+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:45.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:44 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:44.891+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:45.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:44 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:44.891+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:45.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:44 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:44.891+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:45.614 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:45 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:45.294+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:45.614 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:45 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:45.294+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:45.614 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:45 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:45.294+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:45.614 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:45 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:45.294+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:45.615 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:45 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:45.422+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:45.615 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:45 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:45.422+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:45.615 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:45 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:45.422+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:45.615 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:45 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:45.422+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:45.973 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:45 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:45.613+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:45.974 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:45 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:45.613+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:45.974 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:45 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:45.613+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.877287+0000 front 2024-04-23T17:01:05.877308+0000 (oldest deadline 2024-04-23T17:01:35.277703+0000) 2024-04-23T17:01:45.974 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:45 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:45.613+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.278371+0000 front 2024-04-23T17:01:15.278414+0000 (oldest deadline 2024-04-23T17:01:39.978224+0000) 2024-04-23T17:01:45.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:45 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:45.852+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:45.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:45 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:45.852+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:45.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:45 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:45.852+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:45.975 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:45 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:45.852+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:46.633 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:46 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:46.420+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:46.633 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:46 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:46.420+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:46.633 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:46 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:46.420+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:46.633 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:46 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:46.420+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:46.634 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:46 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:46.286+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:46.634 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:46 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:46.286+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:46.634 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:46 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:46.286+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:46.634 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:46 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:46.286+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:46.890 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:46 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:46.632+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:46.890 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:46 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:46.632+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:46.890 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:46 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:46.632+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.877287+0000 front 2024-04-23T17:01:05.877308+0000 (oldest deadline 2024-04-23T17:01:35.277703+0000) 2024-04-23T17:01:46.890 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:46 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:46.632+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.278371+0000 front 2024-04-23T17:01:15.278414+0000 (oldest deadline 2024-04-23T17:01:39.978224+0000) 2024-04-23T17:01:47.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:46 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:46.889+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:47.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:46 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:46.889+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:47.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:46 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:46.889+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:47.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:46 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:46.889+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:47.599 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:47 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:47.333+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:47.599 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:47 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:47.333+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:47.599 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:47 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:47.333+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:47.599 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:47 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:47.333+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:47.599 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:47 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:47.397+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:47.599 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:47 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:47.397+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:47.599 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:47 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:47.397+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:47.599 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:47 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:47.397+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:47.932 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:47 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:47.597+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:47.932 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:47 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:47.597+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:47.932 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:47 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:47.597+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.877287+0000 front 2024-04-23T17:01:05.877308+0000 (oldest deadline 2024-04-23T17:01:35.277703+0000) 2024-04-23T17:01:47.932 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:47 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:47.597+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.278371+0000 front 2024-04-23T17:01:15.278414+0000 (oldest deadline 2024-04-23T17:01:39.978224+0000) 2024-04-23T17:01:48.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:47 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:47.931+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:48.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:47 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:47.931+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:48.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:47 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:47.931+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:48.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:47 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:47.931+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:48.604 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:48 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:48.347+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:48.605 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:48 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:48.347+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:48.605 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:48 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:48.347+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:48.605 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:48 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:48.347+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:48.605 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:48 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:48.390+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:48.605 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:48 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:48.390+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:48.605 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:48 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:48.390+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:48.605 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:48 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:48.390+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:48.926 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:48 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:48.604+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:44.472700+0000 front 2024-04-23T17:00:44.472749+0000 (oldest deadline 2024-04-23T17:01:06.172848+0000) 2024-04-23T17:01:48.926 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:48 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:48.604+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:50.274162+0000 front 2024-04-23T17:00:50.274275+0000 (oldest deadline 2024-04-23T17:01:13.774093+0000) 2024-04-23T17:01:48.926 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:48 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:48.604+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.877287+0000 front 2024-04-23T17:01:05.877308+0000 (oldest deadline 2024-04-23T17:01:35.277703+0000) 2024-04-23T17:01:48.926 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:48 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8[49813]: 2024-04-23T17:01:48.604+0000 7fe99da5a700 -1 osd.8 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.278371+0000 front 2024-04-23T17:01:15.278414+0000 (oldest deadline 2024-04-23T17:01:39.978224+0000) 2024-04-23T17:01:49.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:48 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:48.925+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:49.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:48 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:48.925+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:49.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:48 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:48.925+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:49.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:48 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:48.925+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:49.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:49 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:49.370+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:49.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:49 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:49.370+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:49.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:49 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:49.370+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:49.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:49 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:49.370+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:49.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:49 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:49.347+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:49.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:49 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:49.347+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:49.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:49 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:49.347+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:49.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:49 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:49.347+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:49.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:49 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:49.347+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.916776+0000 front 2024-04-23T17:01:24.916617+0000 (oldest deadline 2024-04-23T17:01:49.016611+0000) 2024-04-23T17:01:50.210 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:49 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:49.914+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:50.211 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:49 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:49.914+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:50.211 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:49 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:49.914+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:50.211 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:49 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:49.914+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:50.211 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:49 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:49.914+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:01:50.473 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 17:01:50 smithi188 bash[179794]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-8 2024-04-23T17:01:50.473 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:50 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:50.337+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:50.474 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:50 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:50.337+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:50.474 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:50 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:50.337+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:50.474 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:50 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:50.337+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:50.474 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:50 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:50.337+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.916776+0000 front 2024-04-23T17:01:24.916617+0000 (oldest deadline 2024-04-23T17:01:49.016611+0000) 2024-04-23T17:01:50.474 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:50 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:50.395+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:50.474 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:50 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:50.395+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:50.474 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:50 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:50.395+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:50.474 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:50 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:50.395+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:51.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:50 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:50.948+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:51.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:50 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:50.948+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:51.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:50 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:50.948+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:51.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:50 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:50.948+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:51.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:50 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:50.948+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:01:51.723 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:51 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:51.319+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:51.723 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:51 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:51.319+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:51.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:51 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:51.319+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:51.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:51 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:51.319+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:51.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:51 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:51.319+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.916776+0000 front 2024-04-23T17:01:24.916617+0000 (oldest deadline 2024-04-23T17:01:49.016611+0000) 2024-04-23T17:01:51.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:51 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:51.432+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:51.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:51 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:51.432+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:51.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:51 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:51.432+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:51.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:51 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:51.432+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:51.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:51 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:51.432+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.963161+0000 front 2024-04-23T17:01:24.963235+0000 (oldest deadline 2024-04-23T17:01:50.863060+0000) 2024-04-23T17:01:52.106 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:51 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:51.964+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:52.106 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:51 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:51.964+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:52.107 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:51 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:51.964+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:52.107 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:51 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:51.964+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:52.107 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:51 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:51.964+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:01:52.555 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.8.service' 2024-04-23T17:01:52.600 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T17:01:52.600 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-23T17:01:52.600 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-23T17:01:52.600 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.9 2024-04-23T17:01:52.667 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:52 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:52.364+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:52.667 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:52 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:52.364+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:52.667 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:52 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:52.364+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:52.667 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:52 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:52.364+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:52.667 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:52 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:52.364+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.916776+0000 front 2024-04-23T17:01:24.916617+0000 (oldest deadline 2024-04-23T17:01:49.016611+0000) 2024-04-23T17:01:52.667 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:52 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:52.469+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:52.667 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:52 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:52.469+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:52.667 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:52 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:52.469+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:52.667 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:52 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:52.470+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:52.667 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:52 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:52.470+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.963161+0000 front 2024-04-23T17:01:24.963235+0000 (oldest deadline 2024-04-23T17:01:50.863060+0000) 2024-04-23T17:01:52.948 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:52 smithi188 systemd[1]: Stopping Ceph osd.9 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T17:01:52.948 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:52 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:52.894+0000 7efcd5fd1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T17:01:52.948 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:52 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:52.894+0000 7efcd5fd1700 -1 osd.9 871 *** Got signal Terminated *** 2024-04-23T17:01:52.948 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:52 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:52.894+0000 7efcd5fd1700 -1 osd.9 871 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T17:01:53.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:52 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:52.947+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:53.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:52 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:52.947+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:53.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:52 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:52.947+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:53.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:52 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:52.947+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:53.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:52 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:52.947+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:01:53.723 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:53 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:53.350+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:53.723 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:53 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:53.350+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:53.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:53 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:53.350+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:53.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:53 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:53.350+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:53.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:53 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:53.350+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.916776+0000 front 2024-04-23T17:01:24.916617+0000 (oldest deadline 2024-04-23T17:01:49.016611+0000) 2024-04-23T17:01:53.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:53 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:53.454+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:53.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:53 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:53.454+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:53.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:53 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:53.454+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:53.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:53 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:53.454+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:53.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:53 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:53.454+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.963161+0000 front 2024-04-23T17:01:24.963235+0000 (oldest deadline 2024-04-23T17:01:50.863060+0000) 2024-04-23T17:01:54.349 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:53 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:53.983+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:54.349 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:53 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:53.983+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:54.349 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:53 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:53.983+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:54.349 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:53 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:53.983+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:54.349 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:53 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:53.983+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:01:54.723 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:54 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:54.348+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:54.723 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:54 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:54.348+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:54.723 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:54 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:54.348+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:54.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:54 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:54.348+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:54.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:54 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:54.348+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.916776+0000 front 2024-04-23T17:01:24.916617+0000 (oldest deadline 2024-04-23T17:01:49.016611+0000) 2024-04-23T17:01:54.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:54 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:54.411+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:54.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:54 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:54.411+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:54.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:54 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:54.411+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:54.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:54 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:54.411+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:54.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:54 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:54.411+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.963161+0000 front 2024-04-23T17:01:24.963235+0000 (oldest deadline 2024-04-23T17:01:50.863060+0000) 2024-04-23T17:01:55.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:54 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:54.959+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:55.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:54 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:54.959+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:55.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:54 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:54.959+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:55.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:54 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:54.959+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:55.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:54 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:54.959+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:01:55.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:55 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:55.392+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:55.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:55 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:55.392+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:55.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:55 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:55.392+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:55.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:55 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:55.392+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:55.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:55 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:55.392+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.916776+0000 front 2024-04-23T17:01:24.916617+0000 (oldest deadline 2024-04-23T17:01:49.016611+0000) 2024-04-23T17:01:55.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:55 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:55.426+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:55.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:55 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:55.426+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:55.725 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:55 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:55.426+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:55.725 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:55 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:55.426+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:55.725 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:55 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:55.426+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.963161+0000 front 2024-04-23T17:01:24.963235+0000 (oldest deadline 2024-04-23T17:01:50.863060+0000) 2024-04-23T17:01:56.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:55 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:55.954+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:56.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:55 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:55.954+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:56.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:55 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:55.954+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:56.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:55 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:55.954+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:56.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:55 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:55.954+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:01:56.723 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:56 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:56.352+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:56.723 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:56 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:56.352+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:56.723 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:56 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:56.352+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:56.723 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:56 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:56.352+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:56.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:56 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:56.352+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.916776+0000 front 2024-04-23T17:01:24.916617+0000 (oldest deadline 2024-04-23T17:01:49.016611+0000) 2024-04-23T17:01:56.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:56 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:56.400+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:56.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:56 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:56.400+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:56.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:56 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:56.400+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:56.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:56 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:56.400+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:56.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:56 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:56.400+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.963161+0000 front 2024-04-23T17:01:24.963235+0000 (oldest deadline 2024-04-23T17:01:50.863060+0000) 2024-04-23T17:01:57.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:56 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:56.923+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:57.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:56 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:56.924+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:57.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:56 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:56.924+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:57.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:56 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:56.924+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:57.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:56 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:56.924+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:01:57.723 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:57 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:57.316+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:42.810231+0000 front 2024-04-23T17:00:42.810305+0000 (oldest deadline 2024-04-23T17:01:08.110414+0000) 2024-04-23T17:01:57.723 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:57 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:57.316+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:48.110820+0000 front 2024-04-23T17:00:48.110851+0000 (oldest deadline 2024-04-23T17:01:14.011015+0000) 2024-04-23T17:01:57.723 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:57 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:57.316+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:05.613897+0000 front 2024-04-23T17:01:05.613925+0000 (oldest deadline 2024-04-23T17:01:35.614285+0000) 2024-04-23T17:01:57.723 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:57 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:57.316+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:15.614936+0000 front 2024-04-23T17:01:15.614846+0000 (oldest deadline 2024-04-23T17:01:37.914816+0000) 2024-04-23T17:01:57.723 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:57 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:57.316+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.916776+0000 front 2024-04-23T17:01:24.916617+0000 (oldest deadline 2024-04-23T17:01:49.016611+0000) 2024-04-23T17:01:57.724 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:57 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9[56824]: 2024-04-23T17:01:57.316+0000 7efcce5bd700 -1 osd.9 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:30.717656+0000 front 2024-04-23T17:01:30.717701+0000 (oldest deadline 2024-04-23T17:01:57.118152+0000) 2024-04-23T17:01:57.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:57 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:57.353+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:57.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:57 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:57.353+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:57.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:57 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:57.353+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:57.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:57 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:57.353+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:57.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:57 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:57.353+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.963161+0000 front 2024-04-23T17:01:24.963235+0000 (oldest deadline 2024-04-23T17:01:50.863060+0000) 2024-04-23T17:01:57.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:57 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:57.353+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:31.363969+0000 front 2024-04-23T17:01:31.363984+0000 (oldest deadline 2024-04-23T17:01:57.263888+0000) 2024-04-23T17:01:58.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:57 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:57.918+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:58.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:57 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:57.918+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:58.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:57 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:57.918+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:58.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:57 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:57.918+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:58.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:57 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:57.918+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:01:58.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:58 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:58.322+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:58.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:58 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:58.322+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:58.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:58 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:58.322+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:58.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:58 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:58.322+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:58.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:58 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:58.322+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.963161+0000 front 2024-04-23T17:01:24.963235+0000 (oldest deadline 2024-04-23T17:01:50.863060+0000) 2024-04-23T17:01:58.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:58 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:58.322+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:31.363969+0000 front 2024-04-23T17:01:31.363984+0000 (oldest deadline 2024-04-23T17:01:57.263888+0000) 2024-04-23T17:01:59.223 INFO:journalctl@ceph.osd.9.smithi188.stdout:Apr 23 17:01:58 smithi188 bash[180055]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-9 2024-04-23T17:01:59.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:58 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:58.922+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:01:59.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:58 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:58.922+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:01:59.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:58 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:58.922+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:01:59.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:58 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:58.922+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:01:59.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:58 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:58.922+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:01:59.589 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:59 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:59.299+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:01:59.590 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:59 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:59.299+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:01:59.590 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:59 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:59.299+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:01:59.590 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:59 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:59.299+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:01:59.590 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:59 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:59.299+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.963161+0000 front 2024-04-23T17:01:24.963235+0000 (oldest deadline 2024-04-23T17:01:50.863060+0000) 2024-04-23T17:01:59.590 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:01:59 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:01:59.299+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:31.363969+0000 front 2024-04-23T17:01:31.363984+0000 (oldest deadline 2024-04-23T17:01:57.263888+0000) 2024-04-23T17:02:00.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:59 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:59.888+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:02:00.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:59 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:59.888+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:02:00.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:59 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:59.888+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:02:00.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:59 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:59.888+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:02:00.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:59 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:59.888+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:02:00.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:01:59 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:01:59.888+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:33.382913+0000 front 2024-04-23T17:01:33.382955+0000 (oldest deadline 2024-04-23T17:01:59.282854+0000) 2024-04-23T17:02:00.656 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:00 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:00.256+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:02:00.656 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:00 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:00.256+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:02:00.656 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:00 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:00.256+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:02:00.656 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:00 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:00.256+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:02:00.656 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:00 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:00.256+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.963161+0000 front 2024-04-23T17:01:24.963235+0000 (oldest deadline 2024-04-23T17:01:50.863060+0000) 2024-04-23T17:02:00.656 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:00 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:00.256+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:31.363969+0000 front 2024-04-23T17:01:31.363984+0000 (oldest deadline 2024-04-23T17:01:57.263888+0000) 2024-04-23T17:02:00.663 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.9.service' 2024-04-23T17:02:00.741 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T17:02:00.742 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-23T17:02:00.742 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-23T17:02:00.742 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.10 2024-04-23T17:02:00.973 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:00 smithi188 systemd[1]: Stopping Ceph osd.10 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T17:02:00.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:00 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:00.842+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:02:00.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:00 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:00.842+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:02:00.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:00 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:00.842+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:02:00.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:00 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:00.842+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:02:00.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:00 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:00.842+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:02:00.975 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:00 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:00.842+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:33.382913+0000 front 2024-04-23T17:01:33.382955+0000 (oldest deadline 2024-04-23T17:01:59.282854+0000) 2024-04-23T17:02:01.473 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:01 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:01.029+0000 7f3dd5cde700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T17:02:01.473 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:01 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:01.029+0000 7f3dd5cde700 -1 osd.10 871 *** Got signal Terminated *** 2024-04-23T17:02:01.473 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:01 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:01.029+0000 7f3dd5cde700 -1 osd.10 871 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T17:02:01.473 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:01 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:01.253+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:02:01.473 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:01 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:01.253+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:02:01.474 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:01 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:01.253+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:02:01.474 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:01 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:01.253+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:02:01.474 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:01 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:01.253+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.963161+0000 front 2024-04-23T17:01:24.963235+0000 (oldest deadline 2024-04-23T17:01:50.863060+0000) 2024-04-23T17:02:01.474 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:01 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:01.253+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:31.363969+0000 front 2024-04-23T17:01:31.363984+0000 (oldest deadline 2024-04-23T17:01:57.263888+0000) 2024-04-23T17:02:02.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:01 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:01.844+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:02:02.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:01 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:01.844+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:02:02.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:01 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:01.844+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:02:02.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:01 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:01.844+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:02:02.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:01 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:01.844+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:02:02.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:01 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:01.844+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:33.382913+0000 front 2024-04-23T17:01:33.382955+0000 (oldest deadline 2024-04-23T17:01:59.282854+0000) 2024-04-23T17:02:02.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:02 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:02.282+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:02:02.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:02 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:02.282+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:02:02.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:02 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:02.282+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:02:02.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:02 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:02.282+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:02:02.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:02 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:02.282+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.963161+0000 front 2024-04-23T17:01:24.963235+0000 (oldest deadline 2024-04-23T17:01:50.863060+0000) 2024-04-23T17:02:02.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:02 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:02.282+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:31.363969+0000 front 2024-04-23T17:01:31.363984+0000 (oldest deadline 2024-04-23T17:01:57.263888+0000) 2024-04-23T17:02:03.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:02 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:02.860+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:02:03.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:02 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:02.860+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:02:03.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:02 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:02.860+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:02:03.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:02 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:02.860+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:02:03.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:02 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:02.860+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:02:03.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:02 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:02.860+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:33.382913+0000 front 2024-04-23T17:01:33.382955+0000 (oldest deadline 2024-04-23T17:01:59.282854+0000) 2024-04-23T17:02:03.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:02 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:02.860+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6814 osd.7 since back 2024-04-23T17:01:39.283457+0000 front 2024-04-23T17:01:39.283436+0000 (oldest deadline 2024-04-23T17:02:02.783307+0000) 2024-04-23T17:02:03.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:03 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:03.326+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:02:03.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:03 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:03.326+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:02:03.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:03 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:03.326+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:02:03.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:03 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:03.326+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:02:03.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:03 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:03.326+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.963161+0000 front 2024-04-23T17:01:24.963235+0000 (oldest deadline 2024-04-23T17:01:50.863060+0000) 2024-04-23T17:02:03.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:03 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:03.326+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:31.363969+0000 front 2024-04-23T17:01:31.363984+0000 (oldest deadline 2024-04-23T17:01:57.263888+0000) 2024-04-23T17:02:04.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:03 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:03.894+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:02:04.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:03 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:03.894+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:02:04.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:03 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:03.894+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:02:04.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:03 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:03.894+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:02:04.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:03 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:03.894+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:02:04.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:03 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:03.894+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:33.382913+0000 front 2024-04-23T17:01:33.382955+0000 (oldest deadline 2024-04-23T17:01:59.282854+0000) 2024-04-23T17:02:04.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:03 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:03.894+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6814 osd.7 since back 2024-04-23T17:01:39.283457+0000 front 2024-04-23T17:01:39.283436+0000 (oldest deadline 2024-04-23T17:02:02.783307+0000) 2024-04-23T17:02:04.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:04 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:04.348+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:02:04.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:04 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:04.348+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:02:04.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:04 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:04.348+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:02:04.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:04 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:04.348+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:02:04.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:04 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:04.348+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.963161+0000 front 2024-04-23T17:01:24.963235+0000 (oldest deadline 2024-04-23T17:01:50.863060+0000) 2024-04-23T17:02:04.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:04 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:04.348+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:31.363969+0000 front 2024-04-23T17:01:31.363984+0000 (oldest deadline 2024-04-23T17:01:57.263888+0000) 2024-04-23T17:02:05.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:04 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:04.857+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:02:05.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:04 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:04.857+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:02:05.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:04 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:04.857+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:02:05.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:04 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:04.857+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:02:05.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:04 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:04.857+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:02:05.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:04 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:04.857+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:33.382913+0000 front 2024-04-23T17:01:33.382955+0000 (oldest deadline 2024-04-23T17:01:59.282854+0000) 2024-04-23T17:02:05.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:04 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:04.857+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6814 osd.7 since back 2024-04-23T17:01:39.283457+0000 front 2024-04-23T17:01:39.283436+0000 (oldest deadline 2024-04-23T17:02:02.783307+0000) 2024-04-23T17:02:05.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:05 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:05.379+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.957081+0000 front 2024-04-23T17:00:43.957240+0000 (oldest deadline 2024-04-23T17:01:05.657185+0000) 2024-04-23T17:02:05.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:05 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:05.379+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6811 osd.2 since back 2024-04-23T17:01:00.359695+0000 front 2024-04-23T17:01:00.359586+0000 (oldest deadline 2024-04-23T17:01:26.259756+0000) 2024-04-23T17:02:05.723 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:05 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:05.379+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:07.960814+0000 front 2024-04-23T17:01:07.960755+0000 (oldest deadline 2024-04-23T17:01:32.060883+0000) 2024-04-23T17:02:05.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:05 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:05.379+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.761964+0000 front 2024-04-23T17:01:16.762103+0000 (oldest deadline 2024-04-23T17:01:39.061899+0000) 2024-04-23T17:02:05.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:05 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:05.379+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:24.963161+0000 front 2024-04-23T17:01:24.963235+0000 (oldest deadline 2024-04-23T17:01:50.863060+0000) 2024-04-23T17:02:05.724 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:05 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10[64014]: 2024-04-23T17:02:05.379+0000 7f3dd22d2700 -1 osd.10 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:31.363969+0000 front 2024-04-23T17:01:31.363984+0000 (oldest deadline 2024-04-23T17:01:57.263888+0000) 2024-04-23T17:02:06.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:05 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:05.860+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:02:06.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:05 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:05.860+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:02:06.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:05 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:05.860+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:02:06.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:05 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:05.860+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:02:06.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:05 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:05.860+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:02:06.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:05 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:05.860+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:33.382913+0000 front 2024-04-23T17:01:33.382955+0000 (oldest deadline 2024-04-23T17:01:59.282854+0000) 2024-04-23T17:02:06.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:05 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:05.860+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6814 osd.7 since back 2024-04-23T17:01:39.283457+0000 front 2024-04-23T17:01:39.283436+0000 (oldest deadline 2024-04-23T17:02:02.783307+0000) 2024-04-23T17:02:06.973 INFO:journalctl@ceph.osd.10.smithi188.stdout:Apr 23 17:02:06 smithi188 bash[180323]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-10 2024-04-23T17:02:06.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:06 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:06.894+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:02:06.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:06 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:06.894+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:02:06.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:06 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:06.894+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:02:06.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:06 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:06.894+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:02:06.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:06 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:06.894+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:02:06.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:06 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:06.894+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:33.382913+0000 front 2024-04-23T17:01:33.382955+0000 (oldest deadline 2024-04-23T17:01:59.282854+0000) 2024-04-23T17:02:06.974 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:06 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:06.894+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6814 osd.7 since back 2024-04-23T17:01:39.283457+0000 front 2024-04-23T17:01:39.283436+0000 (oldest deadline 2024-04-23T17:02:02.783307+0000) 2024-04-23T17:02:08.173 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:07 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:07.867+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:02:08.173 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:07 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:07.867+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:02:08.173 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:07 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:07.867+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:02:08.173 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:07 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:07.867+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:02:08.173 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:07 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:07.867+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:02:08.173 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:07 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:07.867+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:33.382913+0000 front 2024-04-23T17:01:33.382955+0000 (oldest deadline 2024-04-23T17:01:59.282854+0000) 2024-04-23T17:02:08.173 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:07 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:07.867+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6814 osd.7 since back 2024-04-23T17:01:39.283457+0000 front 2024-04-23T17:01:39.283436+0000 (oldest deadline 2024-04-23T17:02:02.783307+0000) 2024-04-23T17:02:08.475 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.10.service' 2024-04-23T17:02:08.514 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T17:02:08.514 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-23T17:02:08.515 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-23T17:02:08.515 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.11 2024-04-23T17:02:08.723 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:08 smithi188 systemd[1]: Stopping Ceph osd.11 for 8e3594f2-018b-11ef-bc93-c7b262605968... 2024-04-23T17:02:09.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:08 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:08.824+0000 7fa9fb7a9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T17:02:09.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:08 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:08.824+0000 7fa9fb7a9700 -1 osd.11 871 *** Got signal Terminated *** 2024-04-23T17:02:09.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:08 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:08.824+0000 7fa9fb7a9700 -1 osd.11 871 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T17:02:09.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:08 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:08.835+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:02:09.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:08 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:08.835+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:02:09.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:08 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:08.835+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:02:09.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:08 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:08.835+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:02:09.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:08 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:08.835+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:02:09.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:08 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:08.835+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:33.382913+0000 front 2024-04-23T17:01:33.382955+0000 (oldest deadline 2024-04-23T17:01:59.282854+0000) 2024-04-23T17:02:09.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:08 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:08.835+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6814 osd.7 since back 2024-04-23T17:01:39.283457+0000 front 2024-04-23T17:01:39.283436+0000 (oldest deadline 2024-04-23T17:02:02.783307+0000) 2024-04-23T17:02:10.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:09 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:09.850+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:02:10.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:09 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:09.850+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:02:10.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:09 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:09.850+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:02:10.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:09 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:09.850+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:02:10.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:09 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:09.850+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:02:10.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:09 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:09.850+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:33.382913+0000 front 2024-04-23T17:01:33.382955+0000 (oldest deadline 2024-04-23T17:01:59.282854+0000) 2024-04-23T17:02:10.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:09 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:09.850+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6814 osd.7 since back 2024-04-23T17:01:39.283457+0000 front 2024-04-23T17:01:39.283436+0000 (oldest deadline 2024-04-23T17:02:02.783307+0000) 2024-04-23T17:02:11.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:10 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:10.833+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:02:11.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:10 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:10.833+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:02:11.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:10 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:10.833+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:02:11.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:10 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:10.833+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:02:11.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:10 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:10.833+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:02:11.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:10 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:10.833+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:33.382913+0000 front 2024-04-23T17:01:33.382955+0000 (oldest deadline 2024-04-23T17:01:59.282854+0000) 2024-04-23T17:02:11.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:10 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:10.833+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6814 osd.7 since back 2024-04-23T17:01:39.283457+0000 front 2024-04-23T17:01:39.283436+0000 (oldest deadline 2024-04-23T17:02:02.783307+0000) 2024-04-23T17:02:12.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:11 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:11.823+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:02:12.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:11 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:11.823+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:02:12.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:11 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:11.823+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:02:12.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:11 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:11.823+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:02:12.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:11 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:11.823+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:02:12.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:11 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:11.823+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:33.382913+0000 front 2024-04-23T17:01:33.382955+0000 (oldest deadline 2024-04-23T17:01:59.282854+0000) 2024-04-23T17:02:12.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:11 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:11.823+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6814 osd.7 since back 2024-04-23T17:01:39.283457+0000 front 2024-04-23T17:01:39.283436+0000 (oldest deadline 2024-04-23T17:02:02.783307+0000) 2024-04-23T17:02:12.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:11 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:11.823+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.188:6802 osd.8 since back 2024-04-23T17:01:47.484258+0000 front 2024-04-23T17:01:47.484189+0000 (oldest deadline 2024-04-23T17:02:11.584170+0000) 2024-04-23T17:02:13.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:12 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:12.777+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:02:13.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:12 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:12.777+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:02:13.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:12 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:12.777+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:02:13.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:12 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:12.777+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:02:13.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:12 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:12.777+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:02:13.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:12 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:12.777+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:33.382913+0000 front 2024-04-23T17:01:33.382955+0000 (oldest deadline 2024-04-23T17:01:59.282854+0000) 2024-04-23T17:02:13.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:12 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:12.777+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6814 osd.7 since back 2024-04-23T17:01:39.283457+0000 front 2024-04-23T17:01:39.283436+0000 (oldest deadline 2024-04-23T17:02:02.783307+0000) 2024-04-23T17:02:13.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:12 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:12.777+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.188:6802 osd.8 since back 2024-04-23T17:01:47.484258+0000 front 2024-04-23T17:01:47.484189+0000 (oldest deadline 2024-04-23T17:02:11.584170+0000) 2024-04-23T17:02:14.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:13 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:13.744+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6803 osd.0 since back 2024-04-23T17:00:43.473919+0000 front 2024-04-23T17:00:43.474176+0000 (oldest deadline 2024-04-23T17:01:06.973984+0000) 2024-04-23T17:02:14.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:13 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:13.744+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6807 osd.1 since back 2024-04-23T17:00:52.175463+0000 front 2024-04-23T17:00:52.175542+0000 (oldest deadline 2024-04-23T17:01:15.675436+0000) 2024-04-23T17:02:14.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:13 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:13.744+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.103:6815 osd.3 since back 2024-04-23T17:01:08.378910+0000 front 2024-04-23T17:01:08.378841+0000 (oldest deadline 2024-04-23T17:01:31.278806+0000) 2024-04-23T17:02:14.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:13 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:13.744+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6802 osd.4 since back 2024-04-23T17:01:16.381122+0000 front 2024-04-23T17:01:16.381006+0000 (oldest deadline 2024-04-23T17:01:39.281000+0000) 2024-04-23T17:02:14.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:13 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:13.744+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6806 osd.5 since back 2024-04-23T17:01:25.182135+0000 front 2024-04-23T17:01:25.182067+0000 (oldest deadline 2024-04-23T17:01:49.881903+0000) 2024-04-23T17:02:14.223 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:13 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:13.744+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6810 osd.6 since back 2024-04-23T17:01:33.382913+0000 front 2024-04-23T17:01:33.382955+0000 (oldest deadline 2024-04-23T17:01:59.282854+0000) 2024-04-23T17:02:14.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:13 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:13.744+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.148:6814 osd.7 since back 2024-04-23T17:01:39.283457+0000 front 2024-04-23T17:01:39.283436+0000 (oldest deadline 2024-04-23T17:02:02.783307+0000) 2024-04-23T17:02:14.224 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:13 smithi188 ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11[71943]: 2024-04-23T17:02:13.744+0000 7fa9f3d95700 -1 osd.11 871 heartbeat_check: no reply from 172.21.15.188:6802 osd.8 since back 2024-04-23T17:01:47.484258+0000 front 2024-04-23T17:01:47.484189+0000 (oldest deadline 2024-04-23T17:02:11.584170+0000) 2024-04-23T17:02:14.918 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 17:02:14 smithi188 bash[180589]: ceph-8e3594f2-018b-11ef-bc93-c7b262605968-osd-11 2024-04-23T17:02:16.415 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3594f2-018b-11ef-bc93-c7b262605968@osd.11.service' 2024-04-23T17:02:16.454 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T17:02:16.454 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-23T17:02:16.455 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T17:02:16.685 INFO:teuthology.orchestra.run.smithi103.stdout:Deleting cluster with fsid: 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T17:02:17.827 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T17:02:18.059 INFO:teuthology.orchestra.run.smithi148.stdout:Deleting cluster with fsid: 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T17:02:19.173 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T17:02:19.405 INFO:teuthology.orchestra.run.smithi188.stdout:Deleting cluster with fsid: 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T17:02:20.437 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T17:02:20.474 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T17:02:20.510 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T17:02:20.540 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-23T17:02:20.542 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669729/remote/smithi103/crash 2024-04-23T17:02:20.543 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/crash -- . 2024-04-23T17:02:20.610 INFO:teuthology.orchestra.run.smithi103.stderr:tar: /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T17:02:20.611 INFO:teuthology.orchestra.run.smithi103.stderr:tar: Error is not recoverable: exiting now 2024-04-23T17:02:20.613 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669729/remote/smithi148/crash 2024-04-23T17:02:20.614 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/crash -- . 2024-04-23T17:02:20.682 INFO:teuthology.orchestra.run.smithi148.stderr:tar: /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T17:02:20.682 INFO:teuthology.orchestra.run.smithi148.stderr:tar: Error is not recoverable: exiting now 2024-04-23T17:02:20.684 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669729/remote/smithi188/crash 2024-04-23T17:02:20.684 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/crash -- . 2024-04-23T17:02:20.733 INFO:teuthology.orchestra.run.smithi188.stderr:tar: /var/lib/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T17:02:21.049 INFO:teuthology.orchestra.run.smithi188.stderr:tar: Error is not recoverable: exiting now 2024-04-23T17:02:21.051 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-23T17:02:21.052 DEBUG:teuthology.orchestra.run.smithi103:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-23T17:02:21.169 INFO:teuthology.orchestra.run.smithi103.stdout:2024-04-23T16:30:00.000089+0000 mon.a (mon.0) 1644 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-04-23T17:02:21.170 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-23T17:02:21.170 DEBUG:teuthology.orchestra.run.smithi103:> sudo egrep '\[SEC\]' /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-23T17:02:21.210 DEBUG:teuthology.orchestra.run.smithi103:> sudo egrep '\[ERR\]' /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-23T17:02:21.289 DEBUG:teuthology.orchestra.run.smithi103:> sudo egrep '\[WRN\]' /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-23T17:02:21.367 INFO:teuthology.orchestra.run.smithi103.stdout:2024-04-23T16:30:00.000089+0000 mon.a (mon.0) 1644 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-04-23T17:02:21.367 INFO:tasks.cephadm:Compressing logs... 2024-04-23T17:02:21.367 DEBUG:teuthology.orchestra.run.smithi103:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T17:02:21.410 DEBUG:teuthology.orchestra.run.smithi148:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T17:02:21.412 DEBUG:teuthology.orchestra.run.smithi188:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T17:02:21.449 INFO:teuthology.orchestra.run.smithi103.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-23T17:02:21.455 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T17:02:21.466 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T17:02:21.466 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-23T17:02:21.466 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-23T17:02:21.466 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T17:02:21.466 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T17:02:21.466 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-23T17:02:21.468 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-23T17:02:21.468 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/objectstore_tool.7.log 2024-04-23T17:02:21.468 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-23T17:02:21.468 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.log 2024-04-23T17:02:21.468 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-23T17:02:21.471 INFO:teuthology.orchestra.run.smithi148.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-23T17:02:21.471 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T17:02:21.472 INFO:teuthology.orchestra.run.smithi188.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- : No such file or directory/var/log/ceph/cephadm.log 2024-04-23T17:02:21.472 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-04-23T17:02:21.479 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-23T17:02:21.480 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T17:02:21.480 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T17:02:21.480 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.11.log: /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T17:02:21.483 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-23T17:02:21.483 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-23T17:02:21.483 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-23T17:02:21.483 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-23T17:02:21.483 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.log 2024-04-23T17:02:21.484 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-23T17:02:21.484 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T17:02:21.485 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T17:02:21.485 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.6.log: /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T17:02:21.485 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-23T17:02:21.485 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-23T17:02:21.485 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/objectstore_tool.7.log 2024-04-23T17:02:21.485 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-23T17:02:21.485 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.log 2024-04-23T17:02:21.485 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-23T17:02:21.485 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-23T17:02:21.490 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.4.log: /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.5.log: 93.1% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-23T17:02:21.494 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.1.log: 95.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T17:02:21.501 INFO:teuthology.orchestra.run.smithi103.stderr: 82.9% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T17:02:21.511 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.9.log: /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.log: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T17:02:21.579 INFO:teuthology.orchestra.run.smithi188.stderr: 82.4% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T17:02:21.584 INFO:teuthology.orchestra.run.smithi148.stderr: 89.7% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T17:02:21.744 INFO:teuthology.orchestra.run.smithi188.stderr: 89.7% 91.4% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-volume.log.gz -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T17:02:21.744 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-04-23T17:02:21.777 INFO:teuthology.orchestra.run.smithi148.stderr: 87.5% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T17:02:21.803 INFO:teuthology.orchestra.run.smithi103.stderr: 87.5% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T17:02:21.834 INFO:teuthology.orchestra.run.smithi103.stderr: 89.7% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T17:02:21.843 INFO:teuthology.orchestra.run.smithi148.stderr: 82.4% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T17:02:21.911 INFO:teuthology.orchestra.run.smithi103.stderr: 91.4% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T17:02:22.030 INFO:teuthology.orchestra.run.smithi148.stderr: 91.3% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T17:02:22.158 INFO:teuthology.orchestra.run.smithi188.stderr: 87.5% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T17:02:22.341 INFO:teuthology.orchestra.run.smithi103.stderr: 94.7% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/objectstore_tool.7.log.gz 2024-04-23T17:02:22.999 INFO:teuthology.orchestra.run.smithi148.stderr: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T17:02:23.071 INFO:teuthology.orchestra.run.smithi103.stderr: 91.1% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-23T17:02:23.348 INFO:teuthology.orchestra.run.smithi148.stderr: 94.1% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/objectstore_tool.7.log.gz 2024-04-23T17:02:24.838 INFO:teuthology.orchestra.run.smithi188.stderr: 90.8% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-23T17:02:26.185 INFO:teuthology.orchestra.run.smithi148.stderr: 90.7% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-23T17:02:26.786 INFO:teuthology.orchestra.run.smithi103.stderr: 89.7% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-23T17:02:31.435 INFO:teuthology.orchestra.run.smithi188.stderr: 94.4% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-23T17:03:56.839 INFO:teuthology.orchestra.run.smithi103.stderr: 93.2% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-23T17:04:43.262 INFO:teuthology.orchestra.run.smithi103.stderr: 93.7% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-23T17:04:53.093 INFO:teuthology.orchestra.run.smithi103.stderr: 94.0% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-23T17:05:06.939 INFO:teuthology.orchestra.run.smithi103.stderr: 94.0% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-23T17:05:06.952 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-04-23T17:05:06.952 INFO:teuthology.orchestra.run.smithi103.stderr:real 2m45.527s 2024-04-23T17:05:06.952 INFO:teuthology.orchestra.run.smithi103.stderr:user 4m34.241s 2024-04-23T17:05:06.952 INFO:teuthology.orchestra.run.smithi103.stderr:sys 0m11.471s 2024-04-23T17:06:26.176 INFO:teuthology.orchestra.run.smithi188.stderr: 92.6% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-23T17:07:02.497 INFO:teuthology.orchestra.run.smithi148.stderr: 92.5% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-23T17:07:07.605 INFO:teuthology.orchestra.run.smithi148.stderr: 93.0% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-23T17:07:27.955 INFO:teuthology.orchestra.run.smithi188.stderr: 92.6% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-23T17:07:30.423 INFO:teuthology.orchestra.run.smithi188.stderr: 93.3% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-23T17:07:30.426 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-04-23T17:07:30.426 INFO:teuthology.orchestra.run.smithi188.stderr:real 5m8.993s 2024-04-23T17:07:30.426 INFO:teuthology.orchestra.run.smithi188.stderr:user 6m33.380s 2024-04-23T17:07:30.426 INFO:teuthology.orchestra.run.smithi188.stderr:sys 0m16.505s 2024-04-23T17:07:55.677 INFO:teuthology.orchestra.run.smithi148.stderr: 92.7% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-23T17:08:50.984 INFO:teuthology.orchestra.run.smithi148.stderr: 92.1% -- replaced with /var/log/ceph/8e3594f2-018b-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-23T17:08:50.986 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-04-23T17:08:50.986 INFO:teuthology.orchestra.run.smithi148.stderr:real 6m29.555s 2024-04-23T17:08:50.986 INFO:teuthology.orchestra.run.smithi148.stderr:user 8m1.256s 2024-04-23T17:08:50.986 INFO:teuthology.orchestra.run.smithi148.stderr:sys 0m20.186s 2024-04-23T17:08:50.987 INFO:tasks.cephadm:Archiving logs... 2024-04-23T17:08:50.987 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669729/remote/smithi103/log 2024-04-23T17:08:50.988 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T17:09:34.124 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669729/remote/smithi148/log 2024-04-23T17:09:34.125 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T17:11:00.169 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669729/remote/smithi188/log 2024-04-23T17:11:00.170 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T17:12:09.388 INFO:tasks.cephadm:Removing cluster... 2024-04-23T17:12:09.389 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 --force 2024-04-23T17:12:09.880 INFO:teuthology.orchestra.run.smithi103.stdout:Deleting cluster with fsid: 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T17:12:10.477 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 --force 2024-04-23T17:12:10.868 INFO:teuthology.orchestra.run.smithi148.stdout:Deleting cluster with fsid: 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T17:12:11.677 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8e3594f2-018b-11ef-bc93-c7b262605968 --force 2024-04-23T17:12:12.224 INFO:teuthology.orchestra.run.smithi188.stdout:Deleting cluster with fsid: 8e3594f2-018b-11ef-bc93-c7b262605968 2024-04-23T17:12:12.895 INFO:tasks.cephadm:Removing cephadm ... 2024-04-23T17:12:12.896 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T17:12:12.919 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T17:12:12.954 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T17:12:12.990 INFO:tasks.cephadm:Teardown complete 2024-04-23T17:12:12.990 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-23T17:12:13.000 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-23T17:12:13.000 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T17:12:13.003 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T17:12:13.005 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T17:12:13.196 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-23T17:12:13.205 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-23T17:12:13.206 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T17:12:13.209 DEBUG:teuthology.orchestra.run.smithi148:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T17:12:13.211 DEBUG:teuthology.orchestra.run.smithi188:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T17:12:13.226 INFO:teuthology.orchestra.run.smithi103.stderr:bash: ntpq: command not found 2024-04-23T17:12:13.228 INFO:teuthology.orchestra.run.smithi188.stderr:bash: ntpq: command not found 2024-04-23T17:12:13.229 INFO:teuthology.orchestra.run.smithi148.stderr:bash: ntpq: command not found 2024-04-23T17:12:13.260 INFO:teuthology.orchestra.run.smithi148.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T17:12:13.260 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================== 2024-04-23T17:12:13.260 INFO:teuthology.orchestra.run.smithi148.stdout:^- hv01.front.sepia.ceph.com 2 6 377 60 +2936us[+2936us] +/- 578ms 2024-04-23T17:12:13.260 INFO:teuthology.orchestra.run.smithi148.stdout:^* hv02.front.sepia.ceph.com 3 6 377 63 -129us[ -149us] +/- 59ms 2024-04-23T17:12:13.260 INFO:teuthology.orchestra.run.smithi148.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 +208us[ +208us] +/- 88ms 2024-04-23T17:12:13.260 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T17:12:13.266 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T17:12:13.266 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-04-23T17:12:13.266 INFO:teuthology.orchestra.run.smithi188.stdout:^- hv01.front.sepia.ceph.com 2 6 377 58 +2927us[+2928us] +/- 578ms 2024-04-23T17:12:13.266 INFO:teuthology.orchestra.run.smithi188.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -143us[ -142us] +/- 59ms 2024-04-23T17:12:13.266 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 +216us[ +216us] +/- 88ms 2024-04-23T17:12:13.266 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T17:12:13.268 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T17:12:13.268 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-04-23T17:12:13.268 INFO:teuthology.orchestra.run.smithi103.stdout:^- hv01.front.sepia.ceph.com 2 7 377 123 +2971us[+2992us] +/- 577ms 2024-04-23T17:12:13.268 INFO:teuthology.orchestra.run.smithi103.stdout:^* hv02.front.sepia.ceph.com 3 6 377 65 -122us[ -99us] +/- 59ms 2024-04-23T17:12:13.268 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +244us[ +244us] +/- 88ms 2024-04-23T17:12:13.268 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T17:12:13.269 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-23T17:12:13.281 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-23T17:12:13.281 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-23T17:12:13.290 DEBUG:teuthology.orchestra.run.smithi103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T17:12:13.420 DEBUG:teuthology.orchestra.run.smithi148:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T17:12:13.508 DEBUG:teuthology.orchestra.run.smithi188:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T17:12:13.705 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-23T17:12:13.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T17:12:13.741 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-23T17:12:13.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T17:12:13.775 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-23T17:12:13.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T17:12:13.812 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-23T17:12:13.822 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-23T17:12:13.831 INFO:teuthology.task.internal:Duration was 4434.338830 seconds 2024-04-23T17:12:13.831 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-23T17:12:13.840 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-23T17:12:13.840 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T17:12:13.842 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T17:12:13.845 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T17:12:13.919 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T17:12:13.930 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T17:12:13.949 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T17:12:14.257 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-23T17:12:14.257 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2024-04-23T17:12:14.257 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T17:12:14.301 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi148.front.sepia.ceph.com 2024-04-23T17:12:14.301 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T17:12:14.343 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi188.front.sepia.ceph.com 2024-04-23T17:12:14.344 DEBUG:teuthology.orchestra.run.smithi188:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T17:12:14.391 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-23T17:12:14.391 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T17:12:14.394 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T17:12:14.397 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T17:12:14.546 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-23T17:12:14.546 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T17:12:14.548 DEBUG:teuthology.orchestra.run.smithi148:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T17:12:14.550 DEBUG:teuthology.orchestra.run.smithi188:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T17:12:14.812 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-23T17:12:14.822 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-23T17:12:14.823 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T17:12:14.867 DEBUG:teuthology.orchestra.run.smithi148:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T17:12:14.917 DEBUG:teuthology.orchestra.run.smithi188:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T17:12:14.972 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-23T17:12:14.982 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T17:12:14.984 DEBUG:teuthology.orchestra.run.smithi148:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T17:12:14.987 DEBUG:teuthology.orchestra.run.smithi188:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T17:12:15.014 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = core 2024-04-23T17:12:15.021 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2024-04-23T17:12:15.024 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = core 2024-04-23T17:12:15.038 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T17:12:15.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T17:12:15.093 DEBUG:teuthology.orchestra.run.smithi148:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T17:12:15.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T17:12:15.111 DEBUG:teuthology.orchestra.run.smithi188:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T17:12:15.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T17:12:15.130 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-23T17:12:15.142 INFO:teuthology.task.internal:Transferring archived files... 2024-04-23T17:12:15.144 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669729/remote/smithi103 2024-04-23T17:12:15.144 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T17:12:15.202 DEBUG:teuthology.misc:Transferring archived files from smithi148:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669729/remote/smithi148 2024-04-23T17:12:15.202 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T17:12:15.247 DEBUG:teuthology.misc:Transferring archived files from smithi188:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669729/remote/smithi188 2024-04-23T17:12:15.247 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T17:12:15.299 INFO:teuthology.task.internal:Removing archive directory... 2024-04-23T17:12:15.300 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T17:12:15.302 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T17:12:15.305 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T17:12:15.347 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-23T17:12:15.358 INFO:teuthology.task.internal:Not uploading archives. 2024-04-23T17:12:15.358 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-23T17:12:15.368 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-23T17:12:15.368 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T17:12:15.370 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T17:12:15.372 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T17:12:15.385 INFO:teuthology.orchestra.run.smithi103.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 17:12 /home/ubuntu/cephtest 2024-04-23T17:12:15.387 INFO:teuthology.orchestra.run.smithi148.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 17:12 /home/ubuntu/cephtest 2024-04-23T17:12:15.404 INFO:teuthology.orchestra.run.smithi188.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 17:12 /home/ubuntu/cephtest 2024-04-23T17:12:15.405 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-23T17:12:15.418 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-23T17:12:15.439 INFO:teuthology.nuke:Checking targets against current locks 2024-04-23T17:12:15.466 DEBUG:teuthology.nuke:shortname: smithi103 2024-04-23T17:12:15.466 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T17:12:15.492 DEBUG:teuthology.nuke:shortname: smithi148 2024-04-23T17:12:15.493 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T17:12:15.497 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669729', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 15:49:06.902165', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKcLq1XqhET9DWVFdAMWm2N9zEMTbTkRbDjXk5DXARzJ3Oud6Xg4iR1GIBZ7N5Pp3eEJurCAdxrPgVpFrYEKCXk='} 2024-04-23T17:12:15.516 DEBUG:teuthology.nuke:shortname: smithi188 2024-04-23T17:12:15.516 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T17:12:15.520 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669729', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 15:49:06.905872', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAJOtPzIJn6KzU2crEJ1/mEe1H325BE8DH6ASuMkeG+o9iUxizLCWs66qVk1YW6Hf+yWvUDSo7bX+vvx+94M3hE='} 2024-04-23T17:12:15.541 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669729', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 15:49:06.904114', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMHm2V0pB3G6ulATzdlCK83Lwo819iCQpzyiGN7s52sIdj7c32tT5nrv7NCAt9rgLnsX59tJAWsNyBRtS0pd008='} 2024-04-23T17:12:15.552 INFO:teuthology.orchestra.console.smithi103:Power off 2024-04-23T17:12:15.552 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T17:12:15.575 DEBUG:teuthology.orchestra.console.smithi103:power off output: Chassis Power Control: Down/Off 2024-04-23T17:12:15.576 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T17:12:15.594 INFO:teuthology.orchestra.console.smithi148:Power off 2024-04-23T17:12:15.594 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T17:12:15.606 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-04-23T17:12:15.623 DEBUG:teuthology.orchestra.console.smithi148:power off output: Chassis Power Control: Down/Off 2024-04-23T17:12:15.623 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T17:12:15.641 INFO:teuthology.orchestra.console.smithi188:Power off 2024-04-23T17:12:15.642 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T17:12:15.653 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2024-04-23T17:12:15.668 DEBUG:teuthology.orchestra.console.smithi188:power off output: Chassis Power Control: Down/Off 2024-04-23T17:12:15.668 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T17:12:15.693 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-04-23T17:12:19.606 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T17:12:19.655 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T17:12:19.695 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T17:12:19.719 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-04-23T17:12:19.765 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2024-04-23T17:12:19.806 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-04-23T17:12:23.722 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T17:12:23.766 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T17:12:23.807 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T17:12:23.833 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-04-23T17:12:23.877 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2024-04-23T17:12:23.921 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is off 2024-04-23T17:12:24.021 INFO:teuthology.orchestra.console.smithi188:Power off completed 2024-04-23T17:12:27.837 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T17:12:27.878 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T17:12:27.950 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is off 2024-04-23T17:12:27.988 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is off 2024-04-23T17:12:28.052 INFO:teuthology.orchestra.console.smithi103:Power off completed 2024-04-23T17:12:28.089 INFO:teuthology.orchestra.console.smithi148:Power off completed 2024-04-23T17:12:28.194 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} duration: 4434.338829994202 failure_reason: '"2024-04-23T16:30:00.000089+0000 mon.a (mon.0) 1644 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2024-04-23T17:12:28.194 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T17:12:28.276 INFO:teuthology.run:FAIL